diff --git a/channel/boston-meetup/index.html b/channel/boston-meetup/index.html new file mode 100644 index 0000000..28d3a36 --- /dev/null +++ b/channel/boston-meetup/index.html @@ -0,0 +1,459 @@ + + + + + + Slack Export - #boston-meetup + + + + + +
+ + + +
+ + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-04-28 12:49:23
 +
+

@Michael Robinson has joined the channel

+ + + +
+ ✅ Sheeri Cabral (Collibra) +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sheeri Cabral (Collibra) + (sheeri.cabral@collibra.com) +
+
2023-04-28 12:49:45
 +
+

@Sheeri Cabral (Collibra) has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harel Shein + (harel.shein@gmail.com) +
+
2023-04-28 12:49:45
 +
+

@Harel Shein has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-04-28 12:51:20
 +
+

Please join the meetup group: https://www.meetup.com/boston-data-lineage-meetup-group/

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Viraj Parekh + (vmpvmp94@gmail.com) +
+
2023-04-28 12:51:49
 +
+

@Viraj Parekh has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Eric Veleker + (eric@atlan.com) +
+
2023-05-02 15:16:45
 +
+

@Eric Veleker has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sheeri Cabral (Collibra) + (sheeri.cabral@collibra.com) +
+
2023-05-09 11:07:39
 +
+

I’m courting 2 orgs that would give us free space in Boston 😄

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-05-09 12:01:54
 +
+

Sweet! Thank you. Please let me know if/how I can help

+ + + +
+ ✅ Sheeri Cabral (Collibra) +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sheeri Cabral (Collibra) + (sheeri.cabral@collibra.com) +
+
2023-05-24 11:04:42
 +
+

I’m having no luck getting a venue, should we try CIC Boston?

+ +

[11:04 AM] It’s a short walk from South Station (train terminal) and close to the airport too

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Yuanli Wang + (yuanliw@bu.edu) +
+
2023-05-25 20:33:32
 +
+

@Yuanli Wang has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Nam Nguyen + (nam@astrafy.io) +
+
2023-07-14 05:37:43
 +
+

@Nam Nguyen has joined the channel

+ + + +
+
+
+
+ + + +
+
+ + + + \ No newline at end of file diff --git a/channel/dagster-integration/index.html b/channel/dagster-integration/index.html new file mode 100644 index 0000000..7aa61e4 --- /dev/null +++ b/channel/dagster-integration/index.html @@ -0,0 +1,2285 @@ + + + + + + Slack Export - #dagster-integration + + + + + +
+ + + +
+ + + +
+
+ + + + +
+ +
Dalin Kim + (dalinkim@northwesternmutual.com) +
+
2022-01-21 21:26:13
 +
+

@Dalin Kim has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kevin Mellott + (kevin.r.mellott@gmail.com) +
+
2022-01-21 21:28:55
 +
+

@Kevin Mellott has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Nafisah Islam + (nafisahislam@northwesternmutual.com) +
+
2022-01-21 21:28:56
 +
+

@Nafisah Islam has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Antonio Moctezuma + (antoniomoctezuma@northwesternmutual.com) +
+
2022-01-21 21:28:56
 +
+

@Antonio Moctezuma has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Joshua Wankowski + (joshuawankowski@northwesternmutual.com) +
+
2022-01-21 21:28:56
 +
+

@Joshua Wankowski has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-01-21 21:28:56
 +
+

@Maciej Obuchowski has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2022-01-21 21:28:56
 +
+

@Julien Le Dem has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Dalin Kim + (dalinkim@northwesternmutual.com) +
+
2022-01-21 22:17:56
 +
+

Hello, my team would like to contribute to the OpenLineage-Dagster integration work and wanted to start a public channel for general discussion on this topic.

+ +

Issue #489 is currently open for review and includes the proposal for the integration. As we proceed with the initial implementation, we’d appreciate feedback from the community to make sure the approach is reasonable and OpenLineage events are captured accurately.

+ +

Looking forward to more discussions. Thanks!

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/issues/489|#489 [INTEGRATION][Dagster] Openlineage-Dagster Integration>
+
+ <p>Hello, my team would like to contribute to the OpenLineage-Dagster integration work with the following proposal.</p> + +<p><strong>Integration Target:</strong></p> + +<p><a href="https://github.com/dagster-io/dagster">Dagster</a></p> + +<p><strong>Proposal:</strong></p> + +<p>New <em>openlineage-dagster</em> library that integrates Dagster with OpenLineage for automatic metadata collection</p> + +<p><strong>Incremental Support:</strong></p> + +<ol><li>Initial integration handling basic metadata (job runs)</li><li>Individual “extractor” handling additional metadata (input/output datasets, data quality metrics, and etc.)</li> +</ol> + +<p><strong>Proposed Implementation:</strong></p> + +<p>Dagster uses an <a href="https://docs.dagster.io/deployment/executors#executors"><code>Executor</code></a> to <a href="https://docs.dagster.io/_apidocs/internals#dagster.Executor.execute"><code>execute()</code></a> Dagster job (aka pipeline) runs, and each execution yields a stream of <a href="https://docs.dagster.io/_apidocs/execution#dagster.DagsterEvent"><code>DagsterEvents</code></a> from different stages of the run.</p> + +<p>As Dagster supports custom Executor, we can implement a custom <code>OpenLineageExecutor</code> that will:</p> + +<p>• act as a thin wrapper over an existing implementation of Executor +• translate given <a href="https://docs.dagster.io/_apidocs/internals#dagster.PipelineRun"><code>PipelineRun</code></a> during execution and emit parent job (pipeline/job-level) RunEvents +• execute steps using its helper (actual) executor +• intercept yielded DagsterEvents, translate, and emit child job (op/step-level) RunEvents +• yield original DagsterEvents</p> + +<p>Similar to openlienage-airflow, Dagster-specific <code>OpenLineageAdapter</code> can be implemented as a helper.</p> + +<p><strong>Generated OpenLineage Jobs &amp; RunEvents:</strong></p> + +<p>With the above initial work, the following jobs and events can be generated.</p> + +<p><strong>Dagster job (pipeline) run -&gt; OpenLineage parent job run</strong></p> + +<p>• During execution, <code>PipelineRun</code> is available from <code>PlanOrchestrationContext</code>, and we can get job/pipeline-level metadata from this PipelineRun and translate it to an OpenLineage event. +• As an example, given <code>pipeline_run</code>, a RunEvent may look like,</p> + +<pre><code>RunEvent( + eventType=event_type, + eventTime=datetime.now().isoformat(), + run=Run( + runId=pipeline_run.run_id, + ), + job=Job( + namespace=namespace, + name=pipeline_run.pipeline_name, + ), + producer=PRODUCER, +) +</code></pre> + +<p><strong>Dagster op (step) run -&gt; OpenLineage child job run</strong></p> + +<p>• Assuming "op" is equivalent to "step" at Dagster's low-level, we can identify op-related events from the returned DagsterEvents based on its <a href="https://docs.dagster.io/_apidocs/execution#dagster.DagsterEventType"><code>DagsterEventType</code></a>. For the initial implementation, the following types can be handled. + • DagsterEventType.STEP<em>START + • DagsterEventType.STEP</em>SUCCESS + • DagsterEventType.STEP_FAILURE +• Other types may be usable to extract additional metadata. +• As an example, given <code>dagster_event</code>, a RunEvent may look like,</p> + +<pre><code>RunEvent( + eventType=event_type, + eventTime=datetime.now().isoformat(), + run=self._build_run( + namespace=namespace, + run_id=run_id, # not associated with Dagster + parent_run_id=dagster_event.logging_tags.get("run_id"), # pipeline run id + parent_job_name=dagster_event.logging_tags.get("pipeline_name"), + ), + job=self._build_job( + namespace=namespace, + job_name=f"{dagster_event.pipeline_name}.{dagster_event.step_key}", + ), + producer=PRODUCER, +)) +</code></pre> + +<p><strong>How to Use in Dagster:</strong></p> + +<p>OpenLineageExecutor's ExecutorDefinition can be defined at the job-level like below.</p> + +<pre><code>from dagster import job, op +from openlineage.dagster.executor import openlineage_executor + +@job(executor_def=openlineage_executor) +def a_job(): + + @op + def an_op(): + pass + + an_op() +</code></pre> + +<p>And the executor is configurable like below as Dagster's "executor config".</p> + +<pre><code>execution: + config: + openlienage: + url: "https://..." + api_key: "" + namespace: "default" + executor: + multiprocess_executor: + max_concurrent: 2 +</code></pre> + +<hr /> + +<p>This proposal is based on limited research and understanding of Dagster, and there may be better alternatives as well that we are happy to discuss over and learn more so any feedback from the community is appreciated as we lay this initial block. Thank you for reviewing and looking forward to further discussions!</p> +
+ +
+
Comments
+ 2 +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+ 👏 Eric Veleker +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
firas + (firas.omrane.contact@gmail.com) +
+
2022-01-22 17:08:23
 +
+

@firas has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Laurent Paris + (laurent@datakin.com) +
+
2022-01-24 11:49:36
 +
+

@Laurent Paris has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2022-01-24 20:40:19
 +
+

FYI, I reached out to the Dagster community and they replied to @Dalin Kim’s ticket: https://github.com/OpenLineage/OpenLineage/issues/489#issuecomment-1020718071

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/issues/489#issuecomment-1020718071|Comment on #489 [INTEGRATION][Dagster] Openlineage-Dagster Integration>
+
+ <p>Hi <a href="https://github.com/dalinkim">@dalinkim</a> - I'm a Dagster engineer, here to answer any questions you have. It would be awesome to see a Dagster-OpenLineage integration.</p> + +<p>A challenge with accomplishing this via a custom executor is that there are already a number of executor implementations that dagster users rely on. E.g. some use the K8s executor, some use the in-process executor, and some use the multiprocess executor. If you wanted to maintain that flexibility and also integrate via the executor, you'd need to write an openlineage version of each of those executors.</p> + +<p>The way that we've recommended implementing similar integrations in the past is via a Dagster sensor that tails the Dagster event log. Something like:</p> + +<pre><code>from dagster import sensor +from dagster.core.storage.event_log.base import EventRecordsFilter + +@sensor +def openlineage_sensor(context): + event_records = context.instance.get_event_records( + EventRecordsFilter(after_cursor=context.cursor), + ascending=True, + limit=5, + ) + + for event_record in event_records: + event_log_entry = event_record.event_log_entry + storage_id = event_record.storage_id + # write relevant contents of the event log entry to OpenLineage + + context.update_cursor(storage_id) + +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2022-01-24 20:40:41
 +
+

Thank you for getting this going @Dalin Kim!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-02-04 15:53:24
 +
+

@Michael Robinson has joined the channel

+ + + +
+ 👋 Michael Robinson +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2022-02-04 15:59:17
 +
+

Let me intro @Michael Robinson who among other things is looking for topics to speak about in the OpenLineage monthly meeting

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-02-04 16:08:51
 +
+

Thanks, @Julien Le Dem. If anyone is interested in speaking at the next OL TSC meeting about their work on the Dagster integration, please reply here or message me. The integration is on the agenda for the upcoming https://wiki.lfaidata.foundation/display/OpenLineage/Monthly+TSC+meeting|meeting on 2/9 at 9 am PT. @Dalin Kim

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Dalin Kim + (dalinkim@northwesternmutual.com) +
+
2022-02-04 17:18:32
 +
+

*Thread Reply:* Hi Michael, While I’m currently going through internal review process before creating a PR, I can do a quick demo on the current OpenLineage sensor approach and get some initial feedback if that is okay.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2022-02-09 13:04:24
 +
+

*Thread Reply:* thanks for the demo!

+ + + +
+ 👍 Dalin Kim +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Dalin Kim + (dalinkim@northwesternmutual.com) +
+
2022-02-10 22:37:06
 +
+

*Thread Reply:* Thanks for the opportunity!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Dalin Kim + (dalinkim@northwesternmutual.com) +
+
2022-02-15 13:53:17
 +
+

Hello, I created a pull request for the OpenLineage sensor work here. This initial work handles the basic lifecycles of Dagster jobs & ops (pipelines & steps), and more discussion will be needed to define how we can handle datasets. Hopefully, this PR is acceptable as an initial groundwork, and all feedback is appreciated. Thanks!

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/pull/541|#541 [INTEGRATION][Dagster] OpenLineage sensor>
+
+ <p>Signed-off-by: Dalin Kim <a href="mailto:dalinkim@northwesternmutual.com">dalinkim@northwesternmutual.com</a></p> + +<p><strong>Problem</strong></p> + +<p>OpenLineage-Dagster integration is not available.</p> + +<p>Closes: <a href="https://github.com/OpenLineage/OpenLineage/issues/489">#489</a></p> + +<p><strong>Solution</strong></p> + +<p>Introduce an OpenLineage sensor that tails Dagster event logs for tracking metadata. On each sensor evaluation, the function processes a batch of event logs, converts Dagster events into OpenLineage events, and emits them to an OpenLineage backend.</p> + +<p>Dagster job is converted to an OpenLineage parent job with its individual steps converted to children jobs.</p> + +<p>Event type conversion:</p> + +<p>&gt; <strong>Note:</strong> All schema changes require discussion. Please <a href="https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue">link the issue</a> for context.</p> + +<p>☐ Your change modifies the <a href="https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.json">core</a> OpenLineage model +☐ Your change modifies one or more OpenLineage <a href="https://github.com/OpenLineage/OpenLineage/tree/main/spec/facets">facets</a></p> + +<p>If you're contributing a new integration, please specify the scope of the integration and how/where it has been tested (e.g., Apache Spark integration supports <code>S3</code> and <code>GCS</code> filesystem operations, tested with AWS EMR).</p> + +<p><strong>Checklist</strong></p> + +<p>☑︎ You've <a href="https://github.com/OpenLineage/OpenLineage/blob/main/why-the-dco.md">signed-off</a> your work +☑︎ Your pull request title follows our <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CONTRIBUTING.md#creating-pull-requests">guidelines</a> +☑︎ Your changes are accompanied by tests (<em>if relevant</em>) +☑︎ Your change contains a <a href="https://kurtisnusbaum.medium.com/stacked-diffs-keeping-phabricator-diffs-small-d9964f4dcfa6">small diff</a> and is self-contained +☑︎ You've updated any relevant documentation (<em>if relevant</em>) +☑︎ You've updated the <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md"><code>CHANGELOG.md</code></a> with details about your change under the "Unreleased" section (<em>if relevant, depending on the change, this may not be necessary</em>) +☐ You've versioned the core OpenLineage model or facets according to <a href="https://docs.snowplowanalytics.com/docs/pipeline-components-and-applications/iglu/common-architecture/schemaver">SchemaVer</a> (<em>if relevant</em>)</p> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-02-15 15:39:14
 +
+

*Thread Reply:* Thanks for the PR! I'll take a look tomorrow.

+ + + +
+ 👍 Dalin Kim +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-02-16 09:37:27
 +
+

*Thread Reply:* @Dalin Kim looks great! I approved it. One thing to do is to rebase on main and force-with-lease push: it appears that there are some conflicts.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Dalin Kim + (dalinkim@northwesternmutual.com) +
+
2022-02-16 11:59:49
 +
+

*Thread Reply:* @Maciej Obuchowski Thank you for the review. Just had a small conflict in CHANGELOG, which has been resolved. For integration test, do you suggest a similar approach like airflow integration using flask?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Dalin Kim + (dalinkim@northwesternmutual.com) +
+
2022-02-16 12:03:08
 +
+

*Thread Reply:* Also, I reached out to Sandy from Dagster for a review to make sure this is good from Dagster side of things.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-02-16 12:09:12
 +
+

*Thread Reply:* > For integration test, do you suggest a similar approach like airflow integration using flask? +Yes, I think we can reuse that part. The most important thing is that we have real Dagster running "real" workloads.

+ + + +
+ 👍 Dalin Kim +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Dalin Kim + (dalinkim@northwesternmutual.com) +
+
2022-02-17 00:45:37
 +
+

*Thread Reply:* Documentation has been updated based on feedback from Yuhan from Dagster team, and I believe everything is set from my end.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-02-17 05:25:16
 +
+

*Thread Reply:* Great! Let's just get rid of this linting error and I'll merge it then.

+ +

https://app.circleci.com/pipelines/github/OpenLineage/OpenLineage/2204/workflows/cdb412e6-b41a-4fab-bc8e-d8bee71d051d/jobs/18963

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Dalin Kim + (dalinkim@northwesternmutual.com) +
+
2022-02-17 11:27:41
 +
+

*Thread Reply:* Thank you. I overlooked this when updating docstring. All should be good now.

+ +

One final question - should we make the dagster unit test job “required” in the ci and how can that be configured?

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-02-17 11:54:08
 +
+

*Thread Reply:* @Willy Lulciuc I think you configured it, am I right?

+ +

@Dalin Kim one more rebase, please 🙏 +I've turned auto-merge on, but unfortunately I can't rebase your branch on fork.

+ + + +
+ 👍 Dalin Kim +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Dalin Kim + (dalinkim@northwesternmutual.com) +
+
2022-02-17 11:58:34
 +
+

*Thread Reply:* Rebased and pushed

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-02-17 12:23:44
 +
+

*Thread Reply:* @Dalin Kim merged!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Dalin Kim + (dalinkim@northwesternmutual.com) +
+
2022-02-17 12:24:28
 +
+

*Thread Reply:* @Maciej Obuchowski Awesome! Thank you so much for all your help!

+ + + +
+ 🙌 Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Dalin Kim + (dalinkim@northwesternmutual.com) +
+
2022-02-15 14:01:43
 +
+

One small question on ci - airflow integration test checks are stuck in the “expected” state. Is this expected or is there something I missed in the ci update?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-02-15 14:47:49
 +
+

*Thread Reply:* Sorry - we're not running integration tests for airflow on forks due to security reason. If you're not touching any Airflow files then it should not affect you at all 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-02-15 14:48:23
 +
+

*Thread Reply:* In other words, it's just as expected.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Dalin Kim + (dalinkim@northwesternmutual.com) +
+
2022-02-15 15:09:42
 +
+

*Thread Reply:* Thank you for the clarification

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2022-02-17 11:54:18
 +
+

@Willy Lulciuc has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Nicola Monger + (nicola.monger@moonpig.com) +
+
2022-02-17 16:52:44
 +
+

@Nicola Monger has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2022-02-18 13:03:29
 +
+

@John Thomas has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Dominique Tipton + (dominiquetipton@northwesternmutual.com) +
+
2022-03-01 17:21:49
 +
+

@Dominique Tipton has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Dominique Tipton + (dominiquetipton@northwesternmutual.com) +
+
2022-03-01 17:36:02
 +
+

Hi all 👋

+ +

I have opened up an issue/proposal on getting datasets incorporated with the dagster integration. I would love to get some feedback and conversations going with the community on the proposed approach. Thanks!

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/issues/583|#583 [INTEGRATION][DAGSTER] Add capability to include input and output datasets with the run event>
+
+ <p>With the initial work that has been done to integrate OpenLineage with Dagster (thanks <a href="https://github.com/dalinkim">@dalinkim</a>!), it seems like the next step should be to include a way to emit datasets along with the run events.</p> + +<p><strong>Proposal:</strong></p> + +<p>Introduce a way to transform Dagster assets or Outputs into InputDatasets and/or OutputDatasets to improve metadata collection.</p> + +<p><strong>Proposed Implementation:</strong></p> + +<p>I am no Dagster expert, but after working with the assets for a bit, there unfortunately doesn't seem to be a good way to automatically transform a generic <a href="https://docs.dagster.io/_apidocs/ops#dagster.AssetMaterialization">AssetMaterialization</a> or <a href="https://docs.dagster.io/_apidocs/ops#dagster.Output">Output</a> into an OpenLineage Dataset. However, specific OpenLineage assets can be manually created and yielded in a Dagster job/pipeline. Through the use of a wrapper/asset-generating function, a sufficiently unique AssetMaterialization with a defined structure can be yielded and later parsed into a dataset and added to the run event.</p> + +<p>&gt; Through the use of a wrapper/asset-generating function, a sufficiently unique AssetMaterialization with a defined structure can be created/yielded</p> + +<p>This function would take in an input/output dataset and create an AssetMaterialization—with an <a href="https://docs.dagster.io/_apidocs/ops#dagster.AssetKey">AssetKey</a> or unique structure that can be used to identify the asset as an OpenLineage dataset. That asset could then be yielded in a Dagster job/pipeline.</p> + +<p>&gt; and later parsed into a dataset</p> + +<p>An extra <code>elif</code> check can be added to the <a href="https://github.com/OpenLineage/OpenLineage/blob/8c15b9fe9dc88799d44fc3999f18a47909acb844/integration/dagster/openlineage/dagster/sensor.py#L145-L157">OpenLineage-Dagster sensor</a> to detect <a href="https://docs.dagster.io/_apidocs/execution#dagster.DagsterEventType.ASSET_MATERIALIZATION">when assets are yielded</a>. The asset can then be determined if it is a dataset by its AssetKey/structure and if it is, it can be parsed back into an input/output dataset by reversing the logic used in the above function.</p> + +<p>&gt; and added to the run event</p> + +<p>After the datasets have been parsed and are available in the sensor, they can be passed along to the emit methods so they can be added to the run event.</p> + +<p><strong>Example:</strong></p> + +<p>To emit a dataset, something similar to this would be added to a job/pipeline:</p> + +<pre><code>@op +def an_op(): + output_dataset = OutputDataset( + namespace="example_namespace", + name="example_name" + ) + + asset = asset_generating_function(output_dataset) + yield asset + + # or yield it directly in the function + # asset_yielding_function(output_dataset) +</code></pre> + +<p><strong>Limitations of this approach:</strong></p> + +<p>• In order to fit with other assets yielded by Dagster, the AssetKey should be 100% human readable (i.e. no UUIDs and have a hierarchy that makes sense). +• The created AssetKey should be idempotent. A dataset with the same namespace/name should produce the same key so dagster can relate the assets to one another. A dataset is identified by its namespace and name and the asset should be too. +• The AssetMaterialization (most likely the AssetKey) should contain a sufficiently unique identifier so other assets are not mistakenly identified as an OpenLineage dataset. + • A good option for the AssetKey limitations could be ["OpenLineage", "InputDataset <em>OR</em> OutputDataset <em>OR</em> Dataset", "&lt;dataset.namespace&gt;", "&lt;dataset.name&gt;"] <br /> + This approach is human readable, idempotent, and is a sufficiently unique identifier (assuming other assets beginning with OpenLineage &gt; Dataset are not being created elsewhere).</p> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+ 👍 Dalin Kim, Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-03-04 06:59:34
 +
+

FYI @Dalin Kim @Dominique Tipton Dagster 0.14.3 broke something, and unit tests started to fail. I've pinned version to 0.14.2 for now, but can you take a look?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Dalin Kim + (dalinkim@northwesternmutual.com) +
+
2022-03-04 09:21:58
 +
+

*Thread Reply:* Thanks for letting us know. We’ll take a look and follow up.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Dalin Kim + (dalinkim@northwesternmutual.com) +
+
2022-03-04 12:32:26
 +
+

*Thread Reply:* Just as an update on findings, it appears that this MR introduced a breaking change for the test helper function that creates a test EventLogRecord.

+
+ + + + +
<https://github.com/dagster-io/dagster/pull/6769|#6769 [events] remove message field>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/dagster-io/dagster">dagster-io/dagster</a> +
+ + +
+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/integration/dagster/tests/conftest.py | conftest.py>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-03-04 06:59:46
 +
+

test log: https://app.circleci.com/pipelines/github/OpenLineage/OpenLineage/2424/workflows/a426[…]5c1-b2f1-3a8e0cf61bd9/jobs/21719/parallel-runs/0/steps/0-105

+ + + +
+ 👀 Dalin Kim, Dominique Tipton +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Dalin Kim + (dalinkim@northwesternmutual.com) +
+
2022-03-04 14:38:59
 +
+

Here is PR to fix the failing tests with latest Dagster version.

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/pull/588|#588 [INTEGRATION][Dagster] Handle updated EventLogEntry in OpenLineage sensor unit test>
+
+ <p><strong>Problem</strong></p> + +<p>In Dagster 0.14.3, <code>message</code> field was removed from EventLogEntry per <a href="https://github.com/dagster-io/dagster/pull/6769">dagster-io/dagster#6769</a>. This is Dagster's internal change that should not affect the functionality of the OpenLineage sensor, but the unit test helper function that creates a mock EventLogEntry needs an update to handle this differently based on Dagster version.</p> + +<p><strong>Solution</strong></p> + +<p>Update unit test to handle removed <code>message</code> field in EventLogEntry based on Dagster version</p> + +<p><strong>Checklist</strong></p> + +<p>☑︎ You've <a href="https://github.com/OpenLineage/OpenLineage/blob/main/why-the-dco.md">signed-off</a> your work +☑︎ Your pull request title follows our <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CONTRIBUTING.md#creating-pull-requests">guidelines</a> +☑︎ Your changes are accompanied by tests (<em>if relevant</em>) +☑︎ Your change contains a <a href="https://kurtisnusbaum.medium.com/stacked-diffs-keeping-phabricator-diffs-small-d9964f4dcfa6">small diff</a> and is self-contained +☐ You've updated any relevant documentation (<em>if relevant</em>) +☐ You've updated the <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md"><code>CHANGELOG.md</code></a> with details about your change under the "Unreleased" section (<em>if relevant, depending on the change, this may not be necessary</em>) +☐ You've versioned the core OpenLineage model or facets according to <a href="https://docs.snowplowanalytics.com/docs/pipeline-components-and-applications/iglu/common-architecture/schemaver">SchemaVer</a> (<em>if relevant</em>)</p> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-03-04 15:35:01
 +
+

Thanks! Merged.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ofek Braunstein + (ofekbraunshtein@gmail.com) +
+
2022-03-06 12:35:36
 +
+

@Ofek Braunstein has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
David ROBERT + (david.robert.ext@louisvuitton.com) +
+
2022-03-18 11:19:18
 +
+

@David ROBERT has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-03-21 11:11:09
 +
+

Hello, Dagster tests are failing on main: https://app.circleci.com/pipelines/github/OpenLineage/OpenLineage/2675/workflows/fdaf[…]704-ad9b-e33d2b1e956c/jobs/24743/parallel-runs/0/steps/0-105

+ +

I think something broke again with 0.14.5

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Dominique Tipton + (dominiquetipton@northwesternmutual.com) +
+
2022-03-21 12:56:33
 +
+

*Thread Reply:* Thanks for the heads up. We’ll look into it and follow up

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Dominique Tipton + (dominiquetipton@northwesternmutual.com) +
+
2022-03-22 16:02:01
 +
+

*Thread Reply:* Here is the PR to fix the error with the latest Dagster version

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/pull/624|#624 [INTEGRATION][Dagster] Handle updated PipelineRun in OpenLineage sensor unit test>
+
+ <p><strong>Problem</strong></p> + +<p>In Dagster 0.14.4, <code>pipeline_name</code> was changed from an optional argument to a required argument of PipelineRun per <a href="https://github.com/dagster-io/dagster/pull/6917">dagster-io/dagster#6917</a>. This is Dagster's internal change that should not affect the functionality of the OpenLineage sensor, but the unit test helper function that creates a PipelineRun needs an update to include the required argument.</p> + +<p><strong>Solution</strong></p> + +<p>Update unit test helper method to include the required <code>pipeline_name</code> argument in PipelineRun.</p> + +<p><strong>Checklist</strong></p> + +<p>☑︎ You've <a href="https://github.com/OpenLineage/OpenLineage/blob/main/why-the-dco.md">signed-off</a> your work +☑︎ Your pull request title follows our <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CONTRIBUTING.md#creating-pull-requests">guidelines</a> +☑︎ Your changes are accompanied by tests (<em>if relevant</em>) +☑︎ Your change contains a <a href="https://kurtisnusbaum.medium.com/stacked-diffs-keeping-phabricator-diffs-small-d9964f4dcfa6">small diff</a> and is self-contained +☐ You've updated any relevant documentation (<em>if relevant</em>) +☐ You've updated the <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md"><code>CHANGELOG.md</code></a> with details about your change under the "Unreleased" section (<em>if relevant, depending on the change, this may not be necessary</em>) +☐ You've versioned the core OpenLineage model or facets according to <a href="https://docs.snowplowanalytics.com/docs/pipeline-components-and-applications/iglu/common-architecture/schemaver">SchemaVer</a> (<em>if relevant</em>)</p> +
+ +
+
Comments
+ 1 +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-03-23 08:20:41
 +
+

*Thread Reply:* Thanks! Merged.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Dominique Tipton + (dominiquetipton@northwesternmutual.com) +
+
2022-03-23 09:49:53
 +
+

*Thread Reply:* Thanks!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
marc_pan + (pxy0592@gmail.com) +
+
2022-03-28 23:03:45
 +
+

@marc_pan has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Nico Ritschel + (nico@antmoney.com) +
+
2022-03-30 20:23:13
 +
+

@Nico Ritschel has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Orbit + +
+
2022-03-31 13:21:47
 +
+

@Orbit has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Nico Ritschel + (nico@antmoney.com) +
+
2022-03-31 14:18:06
 +
+

I love the pattern of parsing logs in this integration, so much more flexible compared to the Airflow integration

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2022-03-31 15:59:33
 +
+

*Thread Reply:* That's definitely the advantage of the log-parsing method. The Airflow integration, especially the most recent version for Airflow 2.3+, has the advantage of being more robust when it comes to delivering lineage in real-time

+ + + +
+ 🙌 Nico Ritschel +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Nico Ritschel + (nico@antmoney.com) +
+
2022-03-31 19:18:29
 +
+

*Thread Reply:* Thanks for the heads up on this integration!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Nico Ritschel + (nico@antmoney.com) +
+
2022-03-31 19:20:10
 +
+

*Thread Reply:* I suspect future integrations will move towards this pattern as well? Sorry, off-topic for this channel, but this was the first place I've seen this metadata collection method in this project.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Nico Ritschel + (nico@antmoney.com) +
+
2022-03-31 19:22:14
 +
+

*Thread Reply:* I would be curious to explore similar external executor integrations for Airflow, say for Papermill or Kubernetes (via the corresponding operators). Suppose one would need to pass job metadata through to the respective platform where logs are actually collected.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sudhir Rao + (sudhir@zemosolabs.com) +
+
2022-04-01 13:25:58
 +
+

@Sudhir Rao has joined the channel

+ + + +
+
+
+
+ + + +
+
+ + + + \ No newline at end of file diff --git a/channel/data-council-meetup/index.html b/channel/data-council-meetup/index.html new file mode 100644 index 0000000..bf5cbd0 --- /dev/null +++ b/channel/data-council-meetup/index.html @@ -0,0 +1,423 @@ + + + + + + Slack Export - #data-council-meetup + + + + + +
+ + + +
+ + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-03-14 15:42:50
 +
+

@Michael Robinson has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2023-03-14 15:43:31
 +
+

@Ross Turk has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2023-03-14 15:43:31
 +
+

@John Thomas has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2023-03-14 15:43:32
 +
+

@Julien Le Dem has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2023-03-14 15:44:36
 +
+

👋

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2023-03-14 17:44:04
 +
+

I will be arriving Monday evening and leaving Friday morning.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john.thomas@astronomer.io) +
+
2023-03-20 14:37:23
 +
+

@John Thomas has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2023-03-23 12:16:54
 +
+

I’m arriving Wednesday afternoon and leaving Thursday afternoon.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Dev Jadhav + (dev.jadhav@loxsolution.com) +
+
2023-04-07 08:32:12
 +
+

@Dev Jadhav has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Nam Nguyen + (nam@astrafy.io) +
+
2023-07-14 05:37:46
 +
+

@Nam Nguyen has joined the channel

+ + + +
+
+
+
+ + + +
+
+ + + + \ No newline at end of file diff --git a/channel/dev-discuss/index.html b/channel/dev-discuss/index.html new file mode 100644 index 0000000..aee329f --- /dev/null +++ b/channel/dev-discuss/index.html @@ -0,0 +1,2454 @@ + + + + + + Slack Export - #dev-discuss + + + + + +
+ + + +
+ + + +
+
+ + + + +
+ +
Harel Shein + (harel.shein@gmail.com) +
+
2023-11-14 12:13:06
 +
+

@Harel Shein has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-11-14 12:13:10
 +
+

@Maciej Obuchowski has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2023-11-14 12:13:46
 +
+

@Julien Le Dem has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-11-14 12:13:46
 +
+

@Paweł Leszczyński has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-11-14 12:13:46
 +
+

@Jakub Dardziński has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-11-14 12:13:46
 +
+

@Michael Robinson has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2023-11-14 12:13:46
 +
+

@Willy Lulciuc has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Peter Hicks + (peter.hicks@astronomer.io) +
+
2023-11-14 12:13:46
 +
+

@Peter Hicks has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-11-14 12:13:57
 +
+

👋

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@rossturk.com) +
+
2023-11-14 12:14:02
 +
+

@Ross Turk has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-11-14 12:16:19
 +
+

👋

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2023-11-14 12:18:42
 +
+

👋

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2023-11-14 12:18:53
 +
+

👋

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@rossturk.com) +
+
2023-11-14 12:29:47
 +
+

🌊

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-11-14 13:53:08
 +
+

👋

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-11-14 18:30:48
 +
+

https://github.com/OpenLineage/OpenLineage/pull/2260 +fun PR incoming

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-11-15 04:35:37
 +
+

*Thread Reply:* hey look, more fun +https://github.com/OpenLineage/OpenLineage/pull/2263

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-11-15 05:03:58
 +
+

*Thread Reply:* nice to have fun with you Jakub

+ + + +
+ 🙂 Jakub Dardziński, Harel Shein, Willy Lulciuc +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-11-15 05:42:34
 +
+

*Thread Reply:* Can't wait to see it on the 1st January.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harel Shein + (harel.shein@gmail.com) +
+
2023-11-15 06:56:03
 +
+

*Thread Reply:* Ain’t no party like a dev ex improvement party

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-11-15 11:45:53
 +
+

*Thread Reply:* Gentoo installation party is in similar category of fun

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2023-11-15 03:32:27
 +
+

@Paweł Leszczyński approved PR #2661 with minor comments, I think the enum defined in the db layer is one comment we’ll need to address before merging; otherwise solid work dude 👌

+
+ + + + +
#2661 Runless events - consume job event
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/MarquezProject/marquez">MarquezProject/marquez</a> +
+ + +
+ + + +
+ 🙌 Paweł Leszczyński, Harel Shein +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-11-15 03:34:42
 +
+

_Minor_: We can consider defining a _run_state column and eventually dropping the event_type. That is, we can consider columns prefixed with _ to be "remappings" of OL properties to Marquez. -> didn't get this one. Is it for now or some future plans?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2023-11-15 03:36:02
 +
+

*Thread Reply:* future 😉

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-11-15 03:36:10
 +
+

*Thread Reply:* ok

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-11-15 03:36:23
 +
+

*Thread Reply:* I will then replace enum with string

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2023-11-15 03:36:10
 +
+

also, what about this PR? https://github.com/MarquezProject/marquez/pull/2654

+
+ + + + +
#2654 Runless events - refactor job_versions_io_mapping
+
+ <p><strong>Problem</strong></p> + +<p>This is currently a draft PR which is far from being merged. It is missing few tests related to schema changes which are marked with <code>todo</code> within the code. I've created a PR to have a better discussion on adding <code>job_id</code> to <code>job_versions_io_mapping</code>. This PR should be a follow-up of <a href="https://github.com/MarquezProject/marquez/pull/2641">#2641</a>.</p> + +<p>The assumption was that it should be helpful in optimising get-lineage query. I would like first to clarify how are we going to make benefit of this extra column.</p> + +<p><strong>Solution</strong></p> + +<p>Please describe your change as it relates to the problem, or bug fix, as well as any dependencies. If your change requires a database schema migration, please describe the schema modification(s) and whether it's a <em>backwards-incompatible</em> or <em>backwards-compatible</em> change.</p> + +<blockquote> + <p><strong>Note:</strong> All database schema changes require discussion. Please <a href="https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue">link the issue</a> for context.</p> +</blockquote> + +<p>One-line summary:</p> + +<p><strong>Checklist</strong></p> + +<p>☐ You've <a href="https://github.com/MarquezProject/marquez/blob/main/CONTRIBUTING.md#sign-your-work">signed-off</a> your work +☐ Your changes are accompanied by tests (<em>if relevant</em>) +☐ Your change contains a <a href="https://kurtisnusbaum.medium.com/stacked-diffs-keeping-phabricator-diffs-small-d9964f4dcfa6">small diff</a> and is self-contained +☐ You've updated any relevant documentation (<em>if relevant</em>) +☐ You've included a one-line summary of your change for the <a href="https://github.com/MarquezProject/marquez/blob/main/CHANGELOG.md#unreleased"><code>CHANGELOG.md</code></a> (<em>Depending on the change, this may not be necessary</em>). +☐ You've versioned your <code>.sql</code> database schema migration according to <a href="https://flywaydb.org/documentation/concepts/migrations#naming">Flyway's naming convention</a> (<em>if relevant</em>) +☐ You've included a <a href="https://github.com/MarquezProject/marquez/blob/main/CONTRIBUTING.md#copyright--license">header</a> in any source code files (<em>if relevant</em>)</p> +
+ +
+
Labels
+ docs, api +
+ +
+
Comments
+ 4 +
+ + + + + + + +
+ + <a href="https://github.com/MarquezProject/marquez">MarquezProject/marquez</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-11-15 03:36:33
 +
+

*Thread Reply:* this is the next to go

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-11-15 03:36:38
 +
+

*Thread Reply:* and i consider it ready

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-11-15 03:37:31
 +
+

*Thread Reply:* Then we have a draft one with streaming support https://github.com/MarquezProject/marquez/pull/2682/files -> which has an integration test of lineage endpoint working for streaming jobs

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-11-15 03:38:32
 +
+

*Thread Reply:* I still need to work on #2682 but you can review #2654. once you get some sleep, of course 😉

+ + + +
+ ❤️ Willy Lulciuc +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-11-15 11:44:44
 +
+

Got the doc + poc for hook-level coverage: https://docs.google.com/document/d/1q0shiUxopASO8glgMqjDn89xigJnGrQuBMbcRdolUdk/edit?usp=sharing

+ + + +
+ 👀 Jakub Dardziński +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-11-15 12:24:27
 +
+

*Thread Reply:* did you check if LineageCollector is instantiated once per process?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-11-15 12:26:37
 +
+

*Thread Reply:* Using it only via get_hook_lineage_collector

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-11-15 12:17:31
 +
+

is it time to support hudi?

+ +
+
image.png
+ + + + + + + + +
+ + +
+ 😂 Harel Shein +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-11-15 14:57:10
 +
+

Anyone have thoughts about how to address the question about “pain points” here? https://openlineage.slack.com/archives/C01CK9T7HKR/p1700064564825909. (Listing pros is easy — it’s the cons we don’t have boilerplate for)

+
+ + +
+ + + } + + Naresh reddy + (https://openlineage.slack.com/team/U066HKFCHUG) +
+ + + +
+
+ <p>Can anyone tell me why OL is better than other competitors if you can provide an analysis that would be great</p> +
+ + + + + + +
Original URL: https://openlineage.slack.com/archives/C01CK9T7HKR/p1700064564825909
+ + +
+ + Slack Conversation +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-11-15 14:58:08
 +
+

*Thread Reply:* Maybe something like “OL has many desirable integrations, including a best-in-class Spark integration, but it’s like any other open standard in that it requires contributions in order to approach total coverage. Thankfully, we have many active contributors, and integrations are being added or improved upon all the time.”

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-11-15 16:04:51
 +
+

*Thread Reply:* Maybe rephrase pain points to "something we're not actively focusing on"

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-11-15 14:59:19
 +
+

Apparently an admin can view a Slack archive at any time at this URL: https://openlineage.slack.com/services/export. Only public channels are available, though.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2023-11-15 16:53:09
 +
+

*Thread Reply:* you are now admin

+ + + +
+ 👍 Michael Robinson +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2023-11-15 17:32:26
 +
+

have we discussed adding column level lineage support to Airflow? https://marquezproject.slack.com/archives/C01E8MQGJP7/p1700087438599279?thread_ts=1700084629.245949&cid=C01E8MQGJP7

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-11-15 17:33:19
 +
+

*Thread Reply:* we have it in SQL operators

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2023-11-15 17:34:25
 +
+

*Thread Reply:* OOh any docs / code? or if you’d like to respond in the MQZ slack 🙏

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-11-15 17:35:19
 +
+

*Thread Reply:* I’ll reply there

+ + + +
+ ❤️ Willy Lulciuc, Harel Shein +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-11-15 17:50:23
 +
+

Any opinions about a free task management alternative to the free version of Notion (10-person limit)? Looking at Trello for keeping track of talks.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harel Shein + (harel.shein@gmail.com) +
+
2023-11-15 19:32:17
 +
+

*Thread Reply:* What about GitHub projects?

+ + + +
+ 👍 Michael Robinson +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-11-16 09:27:46
 +
+

*Thread Reply:* Projects is the way to go, thanks

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-11-16 10:23:34
 +
+

*Thread Reply:* Set up a Projects board. New projects are private by default. We could make it public. The one thing that’s missing that we could use is a built-in date field for alerting about upcoming deadlines…

+ + + +
+ 🙌 Harel Shein, Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-11-16 09:31:24
 +
+

worlds are colliding: 6point6 has been acquired by Accenture

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-11-16 09:31:59
 +
+

*Thread Reply:* https://newsroom.accenture.com/news/2023/accenture-to-expand-government-transformation-capabilities-in-the-uk-with-acquisition-of-6point6

+
+
newsroom.accenture.com
+ + + +
Accenture to Expand Government Transformation Capabilities in the U.K. with Acquisition of 6point6
+
+ Accenture has signed an agreement to acquire 6point6, a U.K. technology consultancy, specializing in cloud, data, and cybersecurity. +
+ + + + + + + + +
Original URL: https://newsroom.accenture.com/news/2023/accenture-to-expand-government-transformation-capabilities-in-the-uk-with-acquisition-of-6point6
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-11-16 10:03:27
 +
+

*Thread Reply:* We should sell OL to governments

+ + + +
+ 🙃 Harel Shein +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harel Shein + (harel.shein@gmail.com) +
+
2023-11-16 10:20:36
 +
+

*Thread Reply:* we may have to rebrand to ClosedLineage

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-11-16 10:23:37
 +
+

*Thread Reply:* not in this way; just emit any event second time to secret NSA endpoint

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-11-16 11:13:17
 +
+

*Thread Reply:* we would need to improve our stock photo game

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-11-16 12:17:22
 +
+

CFP for Berlin Buzzwords went up: https://2024.berlinbuzzwords.de/call-for-papers/ +Still over 3 months to submit 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-11-16 12:42:56
 +
+

*Thread Reply:* thanks, updated the talks board

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-11-16 12:43:10
 +
+

*Thread Reply:* https://github.com/orgs/OpenLineage/projects/4/views/1

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-11-16 15:19:53
 +
+

*Thread Reply:* I'm in, will think what to talk about and appreciate any advice 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2023-11-17 13:42:19
 +
+

just searching for OpenLineage in the Datahub code base. They have an “interesting” approach? https://github.com/datahub-project/datahub/blob/2b0811b9875d7d7ea11fb01d0157a21fdd[…]odules/airflow-plugin/src/datahubairflowplugin/_extractors.py

+
+ + + + +
<https://github.com/datahub-project/datahub/blob/2b0811b9875d7d7ea11fb01d0157a21fdd67f020/metadata-ingestion-modules/airflow-plugin/src/datahub_airflow_plugin/_extractors.py | _extractors.py>
+
+ <pre><code>from openlineage.airflow.extractors import BaseExtractor +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/datahub-project/datahub">datahub-project/datahub</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2023-11-17 13:47:21
 +
+

*Thread Reply:* It looks like the datahub airflow plugin uses OL. but turns it off +https://github.com/datahub-project/datahub/blob/2b0811b9875d7d7ea11fb01d0157a21fdd67f020/docs/lineage/airflow.md +disable_openlineage_plugin true Disable the OpenLineage plugin to avoid duplicative processing. +They reuse the extractors but then “patch” the behavior.

+
+ + + + +
<https://github.com/datahub-project/datahub/blob/2b0811b9875d7d7ea11fb01d0157a21fdd67f020/docs/lineage/airflow.md | airflow.md>
+
+ <p><strong>Airflow Integration</strong></p> + +<p>:::note</p> + +<p>If you're looking to schedule DataHub ingestion using Airflow, see the guide on &lt;../../metadata-ingestion/schedule_docs/airflow.md|scheduling ingestion with Airflow>.</p> + +<p>:::</p> + +<p>The DataHub Airflow plugin supports:</p> + +<p>• Automatic column-level lineage extraction from various operators e.g. <code>SqlOperator</code>s (including <code>MySqlOperator</code>, <code>PostgresOperator</code>, <code>SnowflakeOperator</code>, and more), <code>S3FileTransformOperator</code>, and a few others. +• Airflow DAG and tasks, including properties, ownership, and tags. +• Task run information, including task successes and failures. +• Manual lineage annotations using <code>inlets</code> and <code>outlets</code> on Airflow operators.</p> + +<p>There's two actively supported implementations of the plugin, with different Airflow version support.</p> + +<p>If you're using Airflow older than 2.1, it's possible to use the v1 plugin with older versions of <code>acryl-datahub-airflow-plugin</code>. See the <a href="https://github.com/datahub-project/datahub/blob/2b0811b9875d7d7ea11fb01d0157a21fdd67f020/docs/lineage/airflow.md#compatibility">compatibility section</a> for more details.</p> + +<p><strong>DataHub Plugin v2</strong> +<strong>Installation</strong></p> + +<p>The v2 plugin requires Airflow 2.3+ and Python 3.8+. If you don't meet these requirements, use the v1 plugin instead.</p> + +<pre><code>pip install 'acryl-datahub-airflow-plugin[plugin-v2]' +</code></pre> + +<p><strong>Configuration</strong></p> + +<p>Set up a DataHub connection in Airflow, either via command line or the Airflow UI.</p> + +<p><strong>Command Line</strong></p> + +<pre><code>airflow connections add --conn-type 'datahub-rest' 'datahub_rest_default' --conn-host '&lt;http://datahub-gms:8080&gt;' --conn-password '&lt;optional datahub auth token&gt;' +</code></pre> + +<p><strong>Airflow UI</strong></p> + +<p>On the Airflow UI, go to Admin -> Connections and click the "+" symbol to create a new connection. Select "DataHub REST Server" from the dropdown for "Connection Type" and enter the appropriate values.</p> + +<p><a href="https://raw.githubusercontent.com/datahub-project/static-assets/main/imgs/airflow/plugin_connection_setup.png">https://raw.githubusercontent.com/datahub-project/static-assets/main/imgs/airflow/plugin<em>connection</em>setup.png</a></p> + +<p><strong>Optional Configurations</strong></p> + +<p>No additional configuration is required to use the plugin. However, there are some optional configuration parameters that can be set in the <code>airflow.cfg</code> file.</p> + +<pre><code>[datahub] +# Optional - additional config here. +enabled = True # default +</code></pre> + +<p><strong>Automatic lineage extraction</strong></p> + +<p>To automatically extract lineage information, the v2 plugin builds on top of Airflow's built-in <a href="https://openlineage.io/docs/integrations/airflow/default-extractors">OpenLineage extractors</a>.</p> + +<p>The SQL-related extractors have been updated to use DataHub's SQL parser, which is more robust than the built-in one and uses DataHub's metadata information to generate column-level lineage. We discussed the DataHub SQL parser, including why schema-aware parsing works better and how it performs on benchmarks, during the <https://youtu.be/1QVcUmRQK5E?si=U27zygR7Gi_KdkzE&amp;t=2309|June 2023 community town hall>.</p> + +<p><strong>DataHub Plugin v1</strong> +<strong>Installation</strong></p> + +<p>The v1 plugin requires Airflow 2.1+ and Python 3.8+. If you're on older versions, it's still possible to use an older version of the plugin. See the <a href="https://github.com/datahub-project/datahub/blob/2b0811b9875d7d7ea11fb01d0157a21fdd67f020/docs/lineage/airflow.md#compatibility">compatibility section</a> for more details.</p> + +<p>If you're using Airflow 2.3+, we recommend using the v2 plugin instead. If you need to use the v1 plugin with Airflow 2.3+, you must also set the environment variable <code>DATAHUB_AIRFLOW_PLUGIN_USE_V1_PLUGIN=true</code>.</p> + +<pre><code>pip install 'acryl-datahub-airflow-plugin[plugin-v1]' + +# The DataHub rest connection type is included by default. +# To use the DataHub Kafka connection type, install the plugin with the kafka extras. +pip install 'acryl-datahub-airflow-plugin[plugin-v1,datahub-kafka]' +</code></pre> + +<p><strong>Configuration</strong> +<strong>Disable lazy plugin loading</strong></p> + +<pre><code>[core] +lazy_load_plugins = False +</code></pre> + +<p>On MWAA you should add this config to your <a href="https://docs.aws.amazon.com/mwaa/latest/userguide/configuring-env-variables.html#configuring-2.0-airflow-override">Apache Airflow configuration options</a>.</p> + +<p><strong>Setup a DataHub connection</strong></p> + +<p>You must configure an Airflow connection for Datahub. We support both a Datahub REST and a Kafka-based connections, but you only need one.</p> + +<pre><code># For REST-based: +airflow connections add --conn-type 'datahub_rest' 'datahub_rest_default' --conn-host '&lt;http://datahub-gms:8080&gt;' --conn-password '&lt;optional datahub auth token&gt;' +# For Kafka-based (standard Kafka sink config can be passed via extras): +airflow connections add --conn-type 'datahub_kafka' 'datahub_kafka_default' --conn-host 'broker:9092' --conn-extra '{}' +</code></pre> + +<p><strong>Configure the plugin</strong></p> + +<p>If your config doesn't align with the default values, you can configure the plugin in your <code>airflow.cfg</code> file.</p> + +<pre><code>[datahub] +enabled = true +conn_id = datahub_rest_default # or datahub_kafka_default +# etc. +</code></pre> + +<p><strong>Validate that the plugin is working</strong></p> + +<ol><li>Go and check in Airflow at Admin -> Plugins menu if you can see the DataHub plugin</li><li>Run an Airflow DAG. In the task logs, you should see Datahub related log messages like:</li> +</ol> + +<pre><code>Emitting DataHub ... +</code></pre> + +<p><strong>Manual Lineage Annotation</strong> +<strong>Using <code>inlets</code> and <code>outlets</code></strong></p> + +<p>You can manually annotate lineage by setting <code>inlets</code> and <code>outlets</code> on your Airflow operators. This is useful if you're using an operator that doesn't support automatic lineage extraction, or if you want to override the automatic lineage extraction.</p> + +<p>We have a few code samples that demonstrate how to use <code>inlets</code> and <code>outlets</code>:</p> + +<p>• &lt;../../metadata-ingestion-modules/airflow-plugin/src/datahub<em>airflow</em>plugin/example<em>dags/lineage</em>backend<em>demo.py|<code>lineage_backend_demo.py</code>&gt; +• &lt;../../metadata-ingestion-modules/airflow-plugin/src/datahub</em>airflow<em>plugin/example</em>dags/lineage<em>backend</em>taskflow_demo.py|<code>lineage_backend_taskflow_demo.py</code>&gt; - uses the <a href="https://airflow.apache.org/docs/apache-airflow/stable/concepts/taskflow.html">TaskFlow API</a></p> + +<p>For more information, take a look at the <a href="https://airflow.apache.org/docs/apache-airflow/stable/lineage.html">Airflow lineage docs</a>.</p> + +<p><strong>Custom Operators</strong></p> + +<p>If you have created a <a href="https://airflow.apache.org/docs/apache-airflow/stable/howto/custom-operator.html">custom Airflow operator</a> that inherits from the BaseOperator class, <br /> +when overriding the <code>execute</code> function, set inlets and outlets via <code>context['ti'].task.inlets</code> and <code>context['ti'].task.outlets</code>. <br /> +The DataHub Airflow plugin will then pick up those inlets and outlets after the task runs.</p> + +<pre><code>class DbtOperator(BaseOperator): + ... + + def execute(self, context): + # do something + inlets, outlets = self._get_lineage() + # inlets/outlets are lists of either datahub_airflow_plugin.entities.Dataset or datahub_airflow_plugin.entities.Urn + context['ti'].task.inlets = self.inlets + context['ti'].task.outlets = self.outlets + + def _get_lineage(self): + # Do some processing to get inlets/outlets + + return inlets, outlets +</code></pre> + +<p>If you override the <code>pre_execute</code> and <code>post_execute</code> function, ensure they include the <code>@prepare_lineage</code> and <code>@apply_lineage</code> decorators respectively. Reference the <a href="https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/lineage.html#lineage">Airflow docs</a> for more details.</p> + +<p><strong>Emit Lineage Directly</strong></p> + +<p>If you can't use the plugin or annotate inlets/outlets, you can also emit lineage using the <code>DatahubEmitterOperator</code>.</p> + +<p>Reference &lt;../../metadata-ingestion-modules/airflow-plugin/src/datahub<em>airflow</em>plugin/example<em>dags/lineage</em>emission_dag.py|<code>lineage_emission_dag.py</code>&gt; for a full example.</p> + +<p>In order to use this example, you must first configure the Datahub hook. Like in ingestion, we support a Datahub REST hook and a Kafka-based hook. See the plugin configuration for examples.</p> + +<p><strong>Debugging</strong> +<strong>Missing lineage</strong></p> + +<p>If you're not seeing lineage in DataHub, check the following:</p> + +<p>• Validate that the plugin is loaded in Airflow. Go to Admin -> Plugins and check that the DataHub plugin is listed. +• With the v2 plugin, it should also print a l…</p> +
+ + + + + + + + + +
+ + <a href="https://github.com/datahub-project/datahub">datahub-project/datahub</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2023-11-17 13:48:52
 +
+

*Thread Reply:* Of course this approach will need changing again with AF 2.7

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2023-11-17 13:49:02
 +
+

*Thread Reply:* It’s their choice 🤷

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2023-11-17 13:51:23
 +
+

*Thread Reply:* It looks like we can possibly learn from their approach in SQL parsing: https://datahubproject.io/docs/lineage/airflow/#automatic-lineage-extraction

+
+
datahubproject.io
+ + + +
Airflow Integration | DataHub
+
+ If you're looking to schedule DataHub ingestion using Airflow, see the guide on scheduling ingestion with Airflow. +
+ + + + + + +
Original URL: https://datahubproject.io/docs/lineage/airflow/#automatic-lineage-extraction
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-11-17 16:42:51
 +
+

*Thread Reply:* what's that approach? I only know they have been claiming best SQL parsing capabilities

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2023-11-17 20:54:48
 +
+

*Thread Reply:* I haven’t looked in the details but I’m assuming it is in this repo. (my comment is entirely based on the claim here)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-11-20 02:58:07
 +
+

*Thread Reply:* <https://www.acryldata.io/blog/extracting-column-level-lineage-from-sql> -> The interesting difference is that in order to find table schemas, they use their data catalog to evaluate column-level lineage instead of doing this on the client side.

+ +

My understanding by example is: If you do +create table x as select ** from y +you need to resolve ** to know column level lineage. Our approach is to do that on the client side, probably with an extra call to database. Their approach is to do that based on the data catalog information.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2023-11-17 20:56:54
 +
+

I’m off on vacation. See you in a week

+ + + +
+ ❤️ Jakub Dardziński, Maciej Obuchowski, Paweł Leszczyński, Harel Shein, Ross Turk +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-11-21 05:23:31
 +
+

Maybe move today's meeting earlier, since no one from west coast is joining? @Harel Shein

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harel Shein + (harel.shein@gmail.com) +
+
2023-11-21 09:27:22
 +
+

*Thread Reply:* Ah! That would have been a good idea, but I can’t :(

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harel Shein + (harel.shein@gmail.com) +
+
2023-11-21 09:27:44
 +
+

*Thread Reply:* Do you prefer an earlier meeting tomorrow?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-11-21 09:28:54
 +
+

*Thread Reply:* maybe let's keep today's meeting then

+ + + +
+ 👍 Harel Shein +
+ +
+
+
+
+ + + +
+
+ + + + \ No newline at end of file diff --git a/channel/general/index.html b/channel/general/index.html new file mode 100644 index 0000000..3323de9 --- /dev/null +++ b/channel/general/index.html @@ -0,0 +1,151812 @@ + + + + + + Slack Export - #general + + + + + +
+ + + +
+ + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2020-10-20 21:01:02
 +
+

@Julien Le Dem has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mars Lan + (mars.th.lan@gmail.com) +
+
2020-10-21 08:23:39
 +
+

@Mars Lan has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Wes McKinney + (wesmckinn@gmail.com) +
+
2020-10-21 11:39:13
 +
+

@Wes McKinney has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ryan Blue + (rblue@netflix.com) +
+
2020-10-21 12:46:39
 +
+

@Ryan Blue has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Drew Banin + (drew@fishtownanalytics.com) +
+
2020-10-21 12:53:42
 +
+

@Drew Banin has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2020-10-21 13:29:49
 +
+

@Willy Lulciuc has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Lewis Hemens + (lewis@dataform.co) +
+
2020-10-21 13:52:50
 +
+

@Lewis Hemens has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2020-10-21 14:15:41
 +
+

This is the official start of the OpenLineage initiative. Thank you all for joining. First item is to provide feedback on the doc: https://docs.google.com/document/d/1qL_mkd9lFfe_FMoLTyPIn80-fpvZUAdEIfrabn8bfLE/edit

+ + + +
+ 🎉 Willy Lulciuc, Abe Gong +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Abe Gong + (abe@superconductive.com) +
+
2020-10-21 23:22:03
 +
+

@Abe Gong has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Shirshanka Das + (sdas@linkedin.com) +
+
2020-10-22 13:50:35
 +
+

@Shirshanka Das has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
deleted_profile + (fengtao04@gmail.com) +
+
2020-10-23 15:03:44
 +
+

@deleted_profile has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Chris White + (chris@prefect.io) +
+
2020-10-23 19:30:36
 +
+

@Chris White has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2020-10-24 19:29:04
 +
+

Thanks all for joining. In addition to the google doc, I have opened a pull request with an initial openapi spec: https://github.com/OpenLineage/OpenLineage/pull/1 +The goal is to specify the initial model (just plain lineage) that will be extended with various facets. +It does not intend to restrict to HTTP. Those same PUT calls without output can be translated to any async protocol

+
+
GitHub
+ + + +
initial spec proposal by julienledem · Pull Request #1 · OpenLineage/OpenLineage
+
+ Signed-off-by: Julien Le Dem <a href="mailto:julien@apache.org">julien@apache.org</a> +
+ + + + + + + + +
Original URL: https://github.com/OpenLineage/OpenLineage/pull/1
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2020-10-24 19:31:09
 +
+

For reference, the slides of the kickoff meeting: https://docs.google.com/presentation/d/1bOnm4J7y1JRJBJtSImm-3vvXzvqqkL-UsCShAuub5oU/edit?usp=sharing

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Wes McKinney + (wesmckinn@gmail.com) +
+
2020-10-25 12:13:26
 +
+

Am I the only weirdo that would prefer a Google Group mailing list to Slack for communicating?

+ + + +
+ 👍 Ryan Blue +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2020-10-25 17:22:09
 +
+

*Thread Reply:* slack is the new email?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Wes McKinney + (wesmckinn@gmail.com) +
+
2020-10-25 17:40:19
 +
+

*Thread Reply:* :(

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ryan Blue + (rblue@netflix.com) +
+
2020-10-27 12:27:04
 +
+

*Thread Reply:* I'd prefer a google group as well

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ryan Blue + (rblue@netflix.com) +
+
2020-10-27 12:27:25
 +
+

*Thread Reply:* I think that is better for keeping people engaged, since it isn't just a ton of history to go through

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ryan Blue + (rblue@netflix.com) +
+
2020-10-27 12:27:38
 +
+

*Thread Reply:* And I think it is also better for having thoughtful design discussions

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2020-10-29 15:40:14
 +
+

*Thread Reply:* I’m happy to create a google group if that would help.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2020-10-29 15:45:23
 +
+

*Thread Reply:* Here it is: https://groups.google.com/g/openlineage

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2020-10-29 15:46:34
 +
+

*Thread Reply:* Slack is more of a way to nudge discussions along, we can use github issues or the mailing list to discuss specific points

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2020-11-03 17:34:53
 +
+

*Thread Reply:* @Ryan Blue and @Wes McKinney any recommendations on automating sending github issues update to that list?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ryan Blue + (rblue@netflix.com) +
+
2020-11-03 17:35:34
 +
+

*Thread Reply:* I don't really know how to do that

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ravi Suhag + (suhag.ravi@gmail.com) +
+
2021-04-02 07:18:25
 +
+

*Thread Reply:* @Julien Le Dem How about using Github discussions. They are specifically meant to solve this problem. Feature is still in beta, but it be enabled from repository settings. One positive side i see is that it will really easy to follow through and one separate place to go and look for discussions and ideas which are being discussed.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-04-02 19:51:55
 +
+

*Thread Reply:* I just enabled it: https://github.com/OpenLineage/OpenLineage/discussions

+ + + +
+ 🙌 Ravi Suhag +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Wes McKinney + (wesmckinn@gmail.com) +
+
2020-10-25 12:14:06
 +
+

Or GitHub Issues

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2020-10-25 17:21:44
 +
+

*Thread Reply:* the plan is to use github issues for discussions on the spec. This is to supplement

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Laurent Paris + (laurent@datakin.com) +
+
2020-10-26 19:28:17
 +
+

@Laurent Paris has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Josh Benamram + (josh@databand.ai) +
+
2020-10-27 21:17:30
 +
+

@Josh Benamram has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Victor Shafran + (victor.shafran@databand.ai) +
+
2020-10-28 04:07:27
 +
+

@Victor Shafran has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Victor Shafran + (victor.shafran@databand.ai) +
+
2020-10-28 04:09:00
 +
+

👋 Hi everyone!

+ + + +
+ 👋 Willy Lulciuc, Abe Gong, Drew Banin, Julien Le Dem +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Zhamak Dehghani + (zdehghan@thoughtworks.com) +
+
2020-10-29 17:59:31
 +
+

@Zhamak Dehghani has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2020-11-02 18:30:51
 +
+

I’ve opened a github issue to propose OpenAPI as the way to define the lineage metadata: https://github.com/OpenLineage/OpenLineage/issues/2 +I have also started a thread on the OpenLineage group: https://groups.google.com/g/openlineage/c/2i7ogPl1IP4 +Discussion should happen there: ^

+
+
GitHub
+ + + +
Proposal to use an openapi spec to define the metadata. · Issue #2 · OpenLineage/OpenLineage
+
+ I'm proposing to use openapi to formalize the core lineage model. This doesn't preclude having the ability to serialize in some other binary format (avro, protobuf,...) +
+ + + + + + + + +
Original URL: https://github.com/OpenLineage/OpenLineage/issues/2
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Evgeny Shulman + (evgeny.shulman@databand.ai) +
+
2020-11-04 10:56:00
 +
+

@Evgeny Shulman has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2020-11-05 20:51:22
 +
+

FYI I have updated the PR with a simple genrator: https://github.com/OpenLineage/OpenLineage/pull/1

+
+ + +
+ + + } + + julienledem + (https://github.com/julienledem) +
+ + + +
#1 initial spec proposal
+
+ <p>The goal is to specify the initial model (just plain lineage) that will be extended with various facets. <br /> +It does not intend to restrict to HTTP. Those same PUT calls without output can be translated to any async protocol</p> + +<p>Signed-off-by: Julien Le Dem <a href="mailto:julien@apache.org">julien@apache.org</a></p> +
+ +
+
Comments
+ 6 +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Daniel Henneberger + (danny@datakin.com) +
+
2020-11-11 15:05:46
 +
+

@Daniel Henneberger has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2020-12-08 17:27:57
 +
+

Please send me your github ids if you wish to be added to the github repo

+ + + +
+ 👍 Willy Lulciuc +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Fabrice Etanchaud + (fabrice.etanchaud@netc.fr) +
+
2020-12-10 02:10:35
 +
+

@Fabrice Etanchaud has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2020-12-10 17:04:29
 +
+

As mentioned on the mailing List, the initial spec is ready for a final review. Thanks for all who gave feedback so far.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2020-12-10 17:04:39
 +
+

*Thread Reply:* https://github.com/OpenLineage/OpenLineage/pull/1

+
+ + +
+ + + } + + julienledem + (https://github.com/julienledem) +
+ + + +
#1 initial spec proposal
+
+ <p>The goal is to specify the initial model (just plain lineage) that will be extended with various facets. <br /> +It does not intend to restrict to HTTP. Those same PUT calls without output can be translated to any async protocol</p> + +<p>Signed-off-by: Julien Le Dem <a href="mailto:julien@apache.org">julien@apache.org</a></p> +
+ +
+
Comments
+ 6 +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2020-12-10 17:04:51
 +
+

The next step will be to define individual facets

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2020-12-13 00:28:11
 +
+

I have opened a PR to update the ReadMe: https://openlineage.slack.com/archives/C01EB6DCLHX/p1607835827000100

+
+ + +
+ + + } + + julienledem + (https://github.com/julienledem) +
+ + +
Pull request opened by julienledem
+
#4 Adding README
+
+ <p>Signed-off-by: Julien Le Dem <a href="mailto:julien@apache.org">julien@apache.org</a></p> +
+ + + + + + +
Original URL: https://openlineage.slack.com/archives/C01EB6DCLHX/p1607835827000100
+ + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+ 👍 Willy Lulciuc +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2020-12-14 17:55:46
 +
+

*Thread Reply:* Looks great!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maxime Beauchemin + (max@preset.io) +
+
2020-12-13 17:45:49
 +
+

👋

+ + + +
+ 👋 Shirshanka Das, Julien Le Dem, Willy Lulciuc, Arthur Wiedmer, Mario Measic +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2020-12-14 20:19:57
 +
+

I’m planning to merge https://github.com/OpenLineage/OpenLineage/pull/1 soon. That will be the base that we can iterate on and will enable starting the discussion on individual facets

+
+ + +
+ + + } + + julienledem + (https://github.com/julienledem) +
+ + + +
#1 initial spec proposal
+
+ <p>The goal is to specify the initial model (just plain lineage) that will be extended with various facets. <br /> +It does not intend to restrict to HTTP. Those same PUT calls without output can be translated to any async protocol</p> + +<p>Signed-off-by: Julien Le Dem <a href="mailto:julien@apache.org">julien@apache.org</a></p> +
+ +
+
Comments
+ 6 +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2020-12-16 21:40:52
 +
+

Thank you all for the feedback. I have made an update to the initial spec adressing the final comments

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2020-12-16 21:41:16
 +
+

*Thread Reply:* https://github.com/OpenLineage/OpenLineage/pull/1

+
+ + +
+ + + } + + julienledem + (https://github.com/julienledem) +
+ + + +
#1 initial spec proposal
+
+ <p>The goal is to specify the initial model (just plain lineage) that will be extended with various facets. <br /> +It does not intend to restrict to HTTP. Those same PUT calls without output can be translated to any async protocol</p> + +<p>Signed-off-by: Julien Le Dem <a href="mailto:julien@apache.org">julien@apache.org</a></p> +
+ +
+
Comments
+ 7 +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2020-12-19 11:21:27
 +
+

The contributing guide is available here: https://github.com/OpenLineage/OpenLineage/blob/main/CONTRIBUTING.md +Here is an example proposal for adding a new facet: https://github.com/OpenLineage/OpenLineage/issues/9

+
+ + + + +
#9 [PROPOSAL] track custom state and transitions
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+ 👍 Josh Benamram, Victor Shafran +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2020-12-19 18:27:36
 +
+

Welcome to the newly joined members 🙂 👋

+ + + +
+ 👋 Chris Lambert, Ananth Packkildurai, Arthur Wiedmer, Abe Gong, ale, James Le, Ha Pham, David Krevitt, Harel Shein +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ash Berlin-Taylor + (ash@apache.org) +
+
2020-12-21 05:23:21
 +
+

Hello! Airflow PMC member here. Super interested in this effort

+ + + +
+ 👋 Willy Lulciuc +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2020-12-21 12:15:42
 +
+

*Thread Reply:* Welcome!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ash Berlin-Taylor + (ash@apache.org) +
+
2020-12-21 05:25:07
 +
+

I'm joining this slack now, but I'm basically done for the year, so will investigate proposals etc next year

+ + + +
+ 🙌 Willy Lulciuc +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Zachary Friedman + (zafriedman@gmail.com) +
+
2020-12-21 10:02:37
 +
+

Hey all 👋 Super curious what people's thoughts are on the best way for data quality tools i.e. Great Expectations to integrate with OpenLineage. Probably a Dataset level facet of some sort (from the 25 minutes of deep spec knowledge I have 😆), but curious if that's something being worked on? @Abe Gong

+ + + +
+ 👋 Abe Gong, Willy Lulciuc +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Abe Gong + (abe@superconductive.com) +
+
2020-12-21 10:30:51
 +
+

*Thread Reply:* Yes, that’s about right.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Abe Gong + (abe@superconductive.com) +
+
2020-12-21 10:31:45
 +
+

*Thread Reply:* There’s some subtlety here.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Abe Gong + (abe@superconductive.com) +
+
2020-12-21 10:32:02
 +
+

*Thread Reply:* The initial OpenLineage spec is pretty explicit about linking metadata primarily to execution of specific tasks, which is appropriate for ValidationResults in Great Expectations

+ + + +
+ ✅ Zachary Friedman +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Abe Gong + (abe@superconductive.com) +
+
2020-12-21 10:32:57
 +
+

*Thread Reply:* There isn’t as strong a concept of persistent data objects (e.g. a specific table, or batches of data from a specific table)

+ + + +
+ ✅ Zachary Friedman +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Abe Gong + (abe@superconductive.com) +
+
2020-12-21 10:33:20
 +
+

*Thread Reply:* (In the GE ecosystem, we call these DataAssets and Batches)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Abe Gong + (abe@superconductive.com) +
+
2020-12-21 10:33:56
 +
+

*Thread Reply:* This is also an important conceptual unit, since it’s the level of analysis where Expectations and data docs would typically attach.

+ + + +
+ ✅ Zachary Friedman +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Abe Gong + (abe@superconductive.com) +
+
2020-12-21 10:34:47
 +
+

*Thread Reply:* @James Campbell and I have had some productive conversations with @Julien Le Dem and others about this topic

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2020-12-21 12:20:53
 +
+

*Thread Reply:* Yep! The next step will be to open a few github issues with proposals to add to or amend the spec. We would probably start with a Descriptive Dataset facet of a dataset profile (or dataset update profile). There are other aspects to clarify as well as @Abe Gong is explaining above.

+ + + +
+ ✅ James Campbell +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Zachary Friedman + (zafriedman@gmail.com) +
+
2020-12-21 10:08:24
 +
+

Also interesting to see where this would hook into Dagster. Because one of the many great features of Dagster IMO is it let you do stuff like this (without a formal spec albeit). An OpenLineageMaterialization could be interesting

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2020-12-21 12:23:41
 +
+

*Thread Reply:* Totally! We had a quick discussion with Dagster. Looking forward to proposals along those lines.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harikiran Nayak + (hari@streamsets.com) +
+
2020-12-21 14:35:11
 +
+

Congrats @Julien Le Dem @Willy Lulciuc and team on launching OpenLineage!

+ + + +
+ 🙌 Willy Lulciuc +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2020-12-21 14:48:11
 +
+

*Thread Reply:* Thanks, @Harikiran Nayak! It’s amazing to see such interest in the community on defining a standard for lineage metadata collection.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harikiran Nayak + (hari@streamsets.com) +
+
2020-12-21 15:03:29
 +
+

*Thread Reply:* Yep! Its a validation that the problem is real!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kriti + (kathuriakritihp@gmail.com) +
+
2020-12-22 02:05:45
 +
+

Hey folks! +Worked on a variety of lineage problems across domains. Super excited about this initiative!

+ + + +
+ 👋 Willy Lulciuc +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2020-12-22 13:23:43
 +
+

*Thread Reply:* Welcome!

+ + + +
+ 👋 Kriti +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2020-12-30 22:30:23
 +
+

*Thread Reply:* What are you current use cases for lineage?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2020-12-22 19:54:33
 +
+

(for review) Proposal issue template: https://github.com/OpenLineage/OpenLineage/pull/11

+
+
GitHub
+ + + +
Add Proposal issue template by julienledem · Pull Request #11 · OpenLineage/OpenLineage
+
+ To formalize a bit more proposal submission +
+ + + + + + + + +
Original URL: https://github.com/OpenLineage/OpenLineage/pull/11
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2020-12-22 19:55:16
 +
+

for people interested, <#C01EB6DCLHX|github-notifications> has the github integration that will notify of new PRs …

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Martin Charrel + (martin.charrel@datadoghq.com) +
+
2020-12-29 09:39:46
 +
+

👋 Hello! I'm currently working on lineage systems @ Datadog. Super excited to learn more about this effort

+ + + +
+ 👋 Willy Lulciuc +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2020-12-30 22:28:54
 +
+

*Thread Reply:* Welcome!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2020-12-30 22:29:43
 +
+

*Thread Reply:* Would you mind sharing your main use cases for collecting lineage?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marko Jamedzija + (marko@popcore.com) +
+
2021-01-03 05:54:34
 +
+

Hi! I’m also working on a similar topic for some time. Really looking forward to having these ideas standardized 🙂

+ + + +
+ 👋 Willy Lulciuc +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Alexander Gilfillan + (agilfillan@dealerinspire.com) +
+
2021-01-05 11:29:31
 +
+

I would be interested to see how to extend this to dashboards/visualizations. If that still falls with the scope of this project.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-01-05 12:55:01
 +
+

*Thread Reply:* Definitely, each dashboard should become a node in the lineage graph. That way you can understand all the dependencies of a given dashboard. SOme example of interesting metadata around this: is the dashboard updated in a timely fashion (data freshness); is the data correct (data quality)? Observing changes upstream of the dashboard will provide insights to what’s hapening when freshness or quality suffer

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Alexander Gilfillan + (agilfillan@dealerinspire.com) +
+
2021-01-05 13:20:41
 +
+

*Thread Reply:* 100%. On a granular scale, the difference between a visualization and dashboard can be interesting. One visualization can be connected to multiple dashboards. But of course this depends on the BI tool, Redash would be an example in this case.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-01-05 15:15:23
 +
+

*Thread Reply:* We would need to decide how to model those things. Possibly as a Job type for dashboard and visualization.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Alexander Gilfillan + (agilfillan@dealerinspire.com) +
+
2021-01-06 18:20:06
 +
+

*Thread Reply:* It could be. Its interesting in Redash for example you create custom queries that run at certain intervals to produce the data you need to visualize. Pretty much equivalent to job. But you then build certain visualizations off of that “job”. Then you build dashboards off of visualizations. So you could model it as an job or it could make sense for it to be more modeled like an dataset.

+ +

Thats the hard part of this. How to you model a visualization/dashboard to all the possible ways they can be created since it differs depending on how the tool you use abstracts away creating an visualization.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason Reid + (reid.david.jason@gmail.com) +
+
2021-01-05 17:06:02
 +
+

👋 Hi everyone!

+ + + +
+ 🙌 Willy Lulciuc, Arthur Wiedmer +
+ +
+ 👋 Abe Gong +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason Reid + (reid.david.jason@gmail.com) +
+
2021-01-05 17:10:22
 +
+

*Thread Reply:* Part of my role at Netflix is to oversee our data lineage story so very interested in this effort and hope to be able to participate in its success

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-01-05 18:12:48
 +
+

*Thread Reply:* Hi Jason and welcome

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-01-05 18:15:12
 +
+

A reference implementation of the OpenLineage initial spec is in progress in Marquez: https://github.com/MarquezProject/marquez/pull/880

+
+ + +
+ + + } + + henneberger + (https://github.com/henneberger) +
+ + + +
#880 Add open lineage endpoint
+
+ <p>Signed-off-by: henneberger <a href="mailto:git@danielhenneberger.com">git@danielhenneberger.com</a></p> +
+ +
+
Comments
+ 1 +
+ + + + + + + +
+ + <a href="https://github.com/MarquezProject/marquez">MarquezProject/marquez</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-01-07 12:46:19
 +
+

*Thread Reply:* The OpenLineage reference implementation in Marquez will be presented this morning Thursday (01/07) at 10AM PST, at the Marquez Community meeting.

+ +

When: Thursday, January 7th at 10AM PST +Where: https://us02web.zoom.us/j/89344845719?pwd=Y09RZkxMZHc2U3pOTGZ6SnVMUUVoQT09

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-01-07 12:46:36
 +
+

*Thread Reply:* https://wiki.lfaidata.foundation/pages/viewpage.action?pageId=18481442

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-01-07 12:46:44
 +
+

*Thread Reply:* that’s in 15 min

+ + + +
+
+
+
+ + + + + +
+ +
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-01-12 17:10:23
 +
+

*Thread Reply:* And it’s merged!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-01-12 17:10:53
 +
+

*Thread Reply:* Marquez now has a reference implementation of the initial OpenLineage spec

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jon Loyens + (jon@data.world) +
+
2021-01-06 17:43:02
 +
+

👋 Hi everyone! I'm one of the co-founder at data.world and looking forward to hanging out here

+ + + +
+ 👋 Julien Le Dem, Willy Lulciuc +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Elena Goydina + (egoydina@provectus.com) +
+
2021-01-11 11:39:20
 +
+

👋 Hi everyone! I was looking for the roadmap and don't see any. Does it exist?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-01-13 19:06:34
 +
+

*Thread Reply:* There’s no explicit roadmap so far. With the initial spec defined and the reference implementation implemented, next steps are to define more facets (for example, data shape, dataset size, etc), provide clients to facilitate integrations (java, python, …), implement more integrations (Spark in the works). Members of the community are welcome to drive their own initiatives around the core spec. One of the design goals of the facet is to enable numerous and independant parallel efforts

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-01-13 19:06:48
 +
+

*Thread Reply:* Is there something you are interested about in particular?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-01-13 19:09:42
 +
+

I have opened a proposal to move the spec to JSONSchema, this will make it more focused and decouple from http: https://github.com/OpenLineage/OpenLineage/issues/15

+
+ + +
+ + + } + + julienledem + (https://github.com/julienledem) +
+ + + +
#15 [PROPOSAL] Move the OpenLineage spec to JonSchema (from OpenAPI)
+
+ <p><strong>Purpose:</strong> <br /> +OpenAPI and JsonSchema are mostly the same model with a goal of OpenAPI to be a strict superset of JSONSchema in the near future. <br /> +The goal is to make OpenLineage independent from the underlying protocol. OpenAPI assumes a REST endpoint which is not our intent here. This would still make possible to define an OpenLineage HTTP endpoint while decoupling the model and make it more obvious that it can be used with Kafka for example.</p> + +<p><strong>Proposed implementation</strong> <br /> +Convert the OpenAPI spec to a json schema.</p> +
+ +
+
Assignees
+ julienledem +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+ 👍 Willy Lulciuc +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-01-19 12:26:39
 +
+

Here is a PR with the corresponding change: https://github.com/OpenLineage/OpenLineage/pull/17

+
+ + +
+ + + } + + julienledem + (https://github.com/julienledem) +
+ + + +
#17 adding Json-Schema for OpenLineage
+
+ <p>resolves <a href="https://github.com/OpenLineage/OpenLineage/issues/15">#15</a> <br /> +Signed-off-by: Julien Le Dem <a href="mailto:julien@apache.org">julien@apache.org</a></p> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Xinbin Huang + (bin.huangxb@gmail.com) +
+
2021-02-01 17:07:50
 +
+

Really excited to see this project! I am curious what's the current state and the roadmap of it?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-02-01 17:55:59
 +
+

*Thread Reply:* You can find the initial spec here: https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.md +The process to contribute to the model is described here: https://github.com/OpenLineage/OpenLineage/blob/main/CONTRIBUTING.md +In particular, now we’d want to contribute more facets and integrations. +Marquez has a reference implementation: https://github.com/MarquezProject/marquez/pull/880 +On the roadmap: +• define more facets: data profile, etc +• more integrations +• java/python client +You can see current discussions here: https://github.com/OpenLineage/OpenLineage/issues

+ + + +
+ ✅ Xinbin Huang +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-02-01 17:56:43
 +
+

For people curious about following github activity you can subscribe to: <#C01EB6DCLHX|github-notifications>

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-02-01 17:57:05
 +
+

*Thread Reply:* It is not on general, as it can be a bit noisy

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Zachary Friedman + (zafriedman@gmail.com) +
+
2021-02-09 13:50:17
 +
+

Random-ish question: why is producer and schemaURL nested under nominalTime facet in the spec for postRunStateUpdate? It seems like the producer of its metadata isn’t related to the time of the lineage event?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-02-09 20:02:48
 +
+

*Thread Reply:* Hi @Zachary Friedman! I replied bellow. https://openlineage.slack.com/archives/C01CK9T7HKR/p1612918909009900

+
+ + +
+ + + } + + Julien Le Dem + (https://openlineage.slack.com/team/U01DCLP0GU9) +
+ + + +
+
+ <p><code>producer</code> and <code>schemaURL</code> are defined in the <code>BaseFacet</code> type and therefore all facets (including <code>nominalTime</code>) have it. +• The <code>producer</code> is an identifier for the code that produced the metadata. The idea is that different facets in the same event can be produced by different libraries. For example In a Spark integration, Iceberg could emit it’s own facet in addition to other facets. The producer identifies what produced what. +• The <code>_schemaURL</code> is the identifier of the version of the schema for a given facet. Similarly an event could contain a mixture of Core facets from the spec as well as custom facets. This makes explicit what the definition for this facet is.</p> +
+ + + + + + +
Original URL: https://openlineage.slack.com/archives/C01CK9T7HKR/p1612918909009900
+ + +
+ + Slack Conversation +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-02-09 20:01:49
 +
+

producer and schemaURL are defined in the BaseFacet type and therefore all facets (including nominalTime) have it. +• The producer is an identifier for the code that produced the metadata. The idea is that different facets in the same event can be produced by different libraries. For example In a Spark integration, Iceberg could emit it’s own facet in addition to other facets. The producer identifies what produced what. +• The _schemaURL is the identifier of the version of the schema for a given facet. Similarly an event could contain a mixture of Core facets from the spec as well as custom facets. This makes explicit what the definition for this facet is.

+ + + +
+ 👍 Zachary Friedman +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-02-09 21:27:05
 +
+

As discussed previously, I have separated a Json Schema spec for the OpenLineage events from the OpenAPI spec defining a HTTP endpoint: https://github.com/OpenLineage/OpenLineage/pull/17

+
+ + +
+ + + } + + julienledem + (https://github.com/julienledem) +
+ + + +
#17 adding Json-Schema for OpenLineage
+
+ <p>resolves <a href="https://github.com/OpenLineage/OpenLineage/issues/15">#15</a> <br /> +Signed-off-by: Julien Le Dem <a href="mailto:julien@apache.org">julien@apache.org</a></p> +
+ +
+
Reviewers
+ @wslulciuc, @henneberger +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-02-09 21:27:26
 +
+

*Thread Reply:* Feel free to comment, this is ready to merge

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2021-02-11 20:12:18
 +
+

*Thread Reply:* Thanks, Julien. The new spec format looks great 👍

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-02-09 21:34:31
 +
+

And the corresponding code generator to start the java (and other languages) client: https://github.com/OpenLineage/OpenLineage/pull/18

+
+ + +
+ + + } + + julienledem + (https://github.com/julienledem) +
+ + + +
#18 OpenLineage java model Generator
+
+ <p>This a generator of the OpenLineage model as java classes with Jackson serialization</p> +
+ +
+
Reviewers
+ @wslulciuc +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+ 👍 Willy Lulciuc +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-02-11 22:25:24
 +
+

those are merged, we now have a jsonschema, an openapi spec that extends it and a generated java model

+ + + +
+ 🎉 Willy Lulciuc +
+ +
+ 🙌 Willy Lulciuc +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-02-17 19:39:55
 +
+

Following up on a previous discussion: +This proposal and the accompanying PR add the notion of InputFacets and OutputFacets: https://github.com/OpenLineage/OpenLineage/issues/20 +In summary, we are collecting metadata about jobs and datasets. +At the Job level, when it’s fairly static metadata (not changing every run, like the current code version of the job) it goes in a JobFacet. When it is dynamic and changes every run (like the schedule time of the run), it goes in a RunFacet. +This proposal is adding the same notion at the Dataset level: when it is static and doesn’t change every run (like the dataset schema) it goes in a Dataset facet. When it is dynamic and changes every run (like the input time interval of the dataset being read, or the statistics of the dataset being written) it goes in an inputFacet or an outputFacet. +This enables Job and Dataset versioning logic, to keep track of what changes in the definition of something vs runtime changes

+
+ + +
+ + + } + + julienledem + (https://github.com/julienledem) +
+ + + +
#20 [PROPOSAL] Add InputFacets and OutputFacets
+
+ <p><strong>Purpose:</strong> <br /> +We currently have JobFacets for everything that does not change every run (example: the current version in source control like the git sha). We have RunFacets for everything that changes every Run (example: the schedule time). <br /> +We are missing the same notion for input and output datasets. <br /> +Currently we only have Dataset facets for things that don't change every run (example: Schema). I propose to add Input and Output facets to capture things that change in datasets every run (ex: number of row produced, etc)</p> + +<p><strong>Proposed implementation</strong> <br /> +Add InputFacets and OutputFacets in the core model. Directly in the Input or Output field.</p> + +<pre><code>"inputs": { + "description": "The set of ****input**** datasets.", + "type": "array", + "items": { + "allOf": [ + { "$ref": "#/definitions/Dataset" }, + { + "type": "object", + "properties": { + "inputFacets": { + ... + } + } + } + ] + } + } + +</code></pre> +
+ +
+
Labels
+ proposal +
+ +
+
Comments
+ 1 +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+ 👍 Kevin Mellott, Petr Šimeček +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-02-19 14:27:23
 +
+

*Thread Reply:* @Kevin Mellott and @Petr Šimeček Thanks for the confirmation on this slack message. To make your comment visible to the wider community, please chime in on the github issue as well: https://github.com/OpenLineage/OpenLineage/issues/20 +Thank you.

+
+ + +
+ + + } + + julienledem + (https://github.com/julienledem) +
+ + + +
#20 [PROPOSAL] Add InputFacets and OutputFacets
+
+ <p><strong>Purpose:</strong> <br /> +We currently have JobFacets for everything that does not change every run (example: the current version in source control like the git sha). We have RunFacets for everything that changes every Run (example: the schedule time). <br /> +We are missing the same notion for input and output datasets. <br /> +Currently we only have Dataset facets for things that don't change every run (example: Schema). I propose to add Input and Output facets to capture things that change in datasets every run (ex: number of row produced, etc)</p> + +<p><strong>Proposed implementation</strong> <br /> +Add InputFacets and OutputFacets in the core model. Directly in the Input or Output field.</p> + +<pre><code>"inputs": { + "description": "The set of ****input**** datasets.", + "type": "array", + "items": { + "allOf": [ + { "$ref": "#/definitions/Dataset" }, + { + "type": "object", + "properties": { + "inputFacets": { + ... + } + } + } + ] + } + } + +</code></pre> +
+ +
+
Labels
+ proposal +
+ +
+
Comments
+ 1 +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-02-19 14:27:46
 +
+

*Thread Reply:* The PR is out for this: https://github.com/OpenLineage/OpenLineage/pull/23

+
+ + +
+ + + } + + julienledem + (https://github.com/julienledem) +
+ + + +
#23 add input and output facets
+
+ <p>This PR resolves <a href="https://github.com/OpenLineage/OpenLineage/issues/20">#20</a> <br /> +Signed-off-by: Julien Le Dem <a href="mailto:julien@apache.org">julien@apache.org</a></p> +
+ +
+
Reviewers
+ @jcampbell, @abegong, @henneberger +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Weixi Li + (ashlee.happy@gmail.com) +
+
2021-02-19 04:14:59
 +
+

Hi, I am really interested in this project and Marquez. I am a bit not clear about the differences and relationship between those two projects. As my understanding, OpenLineage provides an api specification for other tools running jobs (e.g. Spark, Airflow) to send out an event to update the run state of the job, then for example Marquez can be the destination for those events and show the data lineage from those run state updates. When you are saying there is an reference implementation of the OpenLineage spec in Marquez, do you mean there is an /lineage endpoint implemented in the Marquez api https://github.com/MarquezProject/marquez/blob/main/api/src/main/java/marquez/api/OpenLineageResource.java? Then my question is what is next step after Marquez has this api? How does Marquez use that endpoint to integrate with airflow for example? I did not find the usage of that endpoint in Marquez project. The library marquez-airflow which integrates Airflow with Marquez seems like only use the other marquez apis to build the data lineage. Or did I misunderstand something? Thank you very much!

+
+ + + + +
api/src/main/java/marquez/api/OpenLineageResource.java
+
+ <pre><code>/** + ** Licensed under the Apache License, Version 2.0 (the "License"); + ** you may not use this file except in compliance with the License. + ** You may obtain a copy of the License at + ** + ** &lt;http://www.apache.org/licenses/LICENSE-2.0&gt; + ** + ** Unless required by applicable law or agreed to in writing, software + ** distributed under the License is distributed on an "AS IS" BASIS, + ** WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + ** See the License for the specific language governing permissions and + ** limitations under the License. + **/ + +package marquez.api; + +import static javax.ws.rs.core.MediaType.APPLICATION_JSON; +import static javax.ws.rs.core.Response.Status.INTERNAL_SERVER_ERROR; + +import com.codahale.metrics.annotation.ExceptionMetered; +import com.codahale.metrics.annotation.ResponseMetered; +import com.codahale.metrics.annotation.Timed; +import com.fasterxml.jackson.core.JsonProcessingException; +import java.sql.SQLException; +import javax.validation.Valid; +import javax.validation.constraints.NotNull; +import javax.ws.rs.Consumes; +import &lt;a href="http://javax.ws.rs.POST"&gt;javax.ws.rs.POST&lt;/a&gt;; +import javax.ws.rs.Path; +import javax.ws.rs.Produces; +import javax.ws.rs.container.AsyncResponse; +import javax.ws.rs.container.Suspended; +import javax.ws.rs.core.Response; +import lombok.extern.slf4j.Slf4j; +import marquez.service.OpenLineageService; +import marquez.service.models.LineageEvent; + +@Slf4j +@Path("/api/v1/lineage") +public class OpenLineageResource { + private final OpenLineageService openLineageService; + + public OpenLineageResource(OpenLineageService openLineageService) { + this.openLineageService = openLineageService; + } + + @Timed + @ResponseMetered + @ExceptionMetered + @POST + @Consumes(APPLICATION_JSON) + @Produces(APPLICATION_JSON) + public void create( + @Valid @NotNull LineageEvent event, @Suspended final AsyncResponse asyncResponse) + throws JsonProcessingException, SQLException { + openLineageService + .createAsync(event) + .whenComplete( + (result, err) -&amp;gt; { + if (err != null) { + log.error("Unexpected error while processing request", err); + asyncResponse.resume(Response.status(INTERNAL_SERVER_ERROR).build()); + } else { + asyncResponse.resume(Response.status(201).build()); + } + }); + } +} + +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/MarquezProject/marquez">MarquezProject/marquez</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Weixi Li + (ashlee.happy@gmail.com) +
+
2021-02-19 05:03:21
 +
+

*Thread Reply:* Okay, I found the spark integration in Marquez calls the /lineage endpoint. But I am still curious about the future plan to integrate with other tools, like airflow?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-02-19 12:41:23
 +
+

*Thread Reply:* Just restating some of my answers from teh marquez slack for the benefits of folks here.

+ +

• OpenLineage defines the schema to collect metadata +• Marquez has a /lineage endpoint implementing the OpenLineage spec to receive this metadata, implemented by the OpenLineageResource you pointed out +• In the future other projects will also have OpenLineage endpoints to receive this metadata +•  The Marquez Spark integration produces OpenLineage events: https://github.com/MarquezProject/marquez/tree/main/integrations/spark +• The Marquez airflow integration still uses the original marquez api but will be migrated to open lineage. +• All new integrations will use OpenLineage metadata

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Weixi Li + (ashlee.happy@gmail.com) +
+
2021-02-22 03:55:18
 +
+

*Thread Reply:* thank you! very clear answer🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ernie Ostic + (ernie.ostic@getmanta.com) +
+
2021-03-02 13:49:04
 +
+

Hi Everyone. Just got started with the Marquez REST API and a little bit into the Open Lineage aspects. Very easy to use. Great work on the curl examples for getting started. I'm working with Postman and am happy to share a collection I have once I finish testing. A question about tags --- are there plans for a "post new tag" call in the API? ...or maybe I missed it. Thx. --ernie

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-03-02 17:51:29
 +
+

*Thread Reply:* I forgot to reply in thread 🙂 https://openlineage.slack.com/archives/C01CK9T7HKR/p1614725462008300

+
+ + +
+ + + } + + Julien Le Dem + (https://openlineage.slack.com/team/U01DCLP0GU9) +
+ + + +
+
+ <p>OpenLineage doesn’t have a Tag facet yet (but tags are defined in the Marquez api). Feel free to open a proposal on the github repo. <a href="https://github.com/OpenLineage/OpenLineage/issues/new/choose">https://github.com/OpenLineage/OpenLineage/issues/new/choose</a></p> +
+ + + + + + +
Original URL: https://openlineage.slack.com/archives/C01CK9T7HKR/p1614725462008300
+ + +
+ + Slack Conversation +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-03-02 17:51:02
 +
+

OpenLineage doesn’t have a Tag facet yet (but tags are defined in the Marquez api). Feel free to open a proposal on the github repo. https://github.com/OpenLineage/OpenLineage/issues/new/choose

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-03-16 11:21:37
 +
+

Hey everyone. What's the story for stream processing (like Flink jobs) for OpenLineage? +It does not fit cleanly with runEvent model, which +It is required to issue 1 START event and 1 of [ COMPLETE, ABORT, FAIL ] event per run. +as unbounded stream jobs usually do not complete.

+ +

I'd imagine few "workarounds" that work for some cases - for example, imagine a job calculating hourly aggregations of transactions and dumpling them into parquet files for further analysis. The job could issue OTHER event type adding additional output dataset every hour. Another option would be to create new "run" every hour, just indicating the added data.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Adam Bellemare + (adam.bellemare@shopify.com) +
+
2021-03-16 15:07:04
 +
+

*Thread Reply:* Ha, I signed up just to ask this precise question!

+ + + +
+ 😀 Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Adam Bellemare + (adam.bellemare@shopify.com) +
+
2021-03-16 15:07:44
 +
+

*Thread Reply:* I’m still looking into the spec myself. Are we required to have 1 or more runs per Job? Or can a Job exist without a run event?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ravi Suhag + (suhag.ravi@gmail.com) +
+
2021-04-02 07:24:39
 +
+

*Thread Reply:* Run event can be emitted when it starts. and it can stay in RUNNING state unless something happens to the job. Additionally, you could send event periodically as state RUNNING to inform the system that job is healthy.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Adam Bellemare + (adam.bellemare@shopify.com) +
+
2021-03-16 15:09:31
 +
+

Similar to @Maciej Obuchowski question about Flink / Streaming jobs - what about Streaming sources (eg: a Kafka topic)? It does fit into the dataset model, more or less. But, has anyone used this yet for a set of streaming sources? Particularly with schema changes over time?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-03-16 18:30:46
 +
+

Hi @Maciej Obuchowski and @Adam Bellemare, streaming jobs are meant to be covered by the spec but I agree there are a few details to iron out.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-03-16 18:31:55
 +
+

In particular, streaming job still have runs. If they run continuously they do not run forever and you want to track that a job has been started at a point in time with a given version of the code, then stopped and started again after being upgraded for example.

+ + + +
+ 👍 Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-03-16 18:32:23
 +
+

I agree with @Maciej Obuchowski that we would also send OTHER events to keep track of progress.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-03-16 18:32:46
 +
+

For example one could track checkpointing this way.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-03-16 18:35:35
 +
+

For a Kafka topic you could have streaming dataset specific facets or even Kafka specific facets (ex: list of offsets we stopped reading at, schema id, etc )

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-03-17 10:05:53
 +
+

*Thread Reply:* That's good idea.

+ +

Now I'm wondering - let's say we want to track on which offset checkpoint ended processing. That would mean we want to expose checkpoint id, time, and offset. +I suppose we don't want to overwrite previous checkpoint info, so we want to have some collection of data in this facet.

+ +

Something like appendable facets would be nice, to just add new checkpoint info to the collection, instead of having to push all the checkpoint infos all the time we just want to add new data point.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-03-16 18:45:23
 +
+

Let me know if you have more thoughts

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Adam Bellemare + (adam.bellemare@shopify.com) +
+
2021-03-17 09:18:49
 +
+

*Thread Reply:* Thanks Julien! I will try to wrap my head around some use-cases and see how it maps to the current spec. From there, I can see if I can figure out any proposals

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-03-17 13:43:29
 +
+

*Thread Reply:* You can use the proposal issue template to propose a new facet for example: https://github.com/OpenLineage/OpenLineage/issues/new/choose

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Carlos Zubieta + (carlos.zubieta@wizeline.com) +
+
2021-03-16 18:49:00
 +
+

Hi everyone, I just hear about OpenLineage and would like to learn more about it. The talks in the repo explain nicely the purpose and general ideas but I have a couple of questions. Are there any working implementations to produce/consume the spec? Also, are there any discussions/guides standard information, naming conventions, etc. in the facets?

+ + + +
+
+
+
+ + + + + +
+ +
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-03-16 20:05:06
 +
+

Hi @Carlos Zubieta here are some pointers ^

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-03-16 20:06:51
 +
+

Marquez has a reference implementation of an OpenLineage endpoint. The Spark integration emits OpenLineage events.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Carlos Zubieta + (carlos.zubieta@wizeline.com) +
+
2021-03-16 20:56:37
 +
+

Thank you @Julien Le Dem!!! Will take a close look

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Adam Bellemare + (adam.bellemare@shopify.com) +
+
2021-03-17 15:41:50
 +
+

Q related to People/Teams/Stakeholders/Owners with regards to Jobs and Datasets (didn’t find anything in search): +Let’s say I have a dataset , and there are a number of other downstream jobs that ingest from it. In the case that the dataset is mutated in some way (or deleted, archived, etc), how would I go about notifying the stakeholders of that set about the changes?

+ +

Just to be clear, I’m not concerned about the mechanics of doing this, just that there is someone that needs to be notified, who has self-registered on this set. +Similarly, I want to manage the datasets I am concerned about , where I can grab a list of all the datasets I tagged myself on.

+ +

This seems to suggest that we could do with additional entities outside of Dataset, Run, Job. However, at the same time, I can see how this can lead to an explosion of other entities. Any thoughts on this particular domain? I think I could achieve something similar with aspects, but this would require that I update the aspect on each entity if I want to wholesale update the user contact, say their email address.

+ +

Has anyone else run into something like this? Have you any advice? Or is this something that may be upcoming in the spec?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Adam Bellemare + (adam.bellemare@shopify.com) +
+
2021-03-17 16:42:24
 +
+

*Thread Reply:* One thing we were considering is just adding these in as Facets ( Tags as per Marquez), and then plugging into some external people managing system. However, I think the question can be generalized to “should there be some sort of generic entity that can enable relationships between itself and Datasets, Jobs, Runs) as part of an integration element?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-03-18 16:03:55
 +
+

*Thread Reply:* That’s a great topic of discussion. I would definitely use the OpenLineage facets to capture what you describe as aspect above. The current Marquez model has a simple notion of ownership at the namespace model but this need to be extended to enable use cases you are describing (owning a dataset or a job) . Right now the owner is just a generic identifier as a string (a user id or a group id for example). Once things are tagged (in some way), you can use the lineage API to find all the downstream or upstream jobs and datasets. In OpenLineage I would start by being able to capture the owner identifier in a facet with contact info optional if it’s available at runtime. It will have the advantage of keeping track of how that changed over time. This definitely deserves its own discussion.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-03-18 17:52:13
 +
+

*Thread Reply:* And also to make sure I understand your use case, you want to be able to notify the consumers of a dataset that it is being discontinued/replaced/… ? What else are you thinking about?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Adam Bellemare + (adam.bellemare@shopify.com) +
+
2021-03-22 09:15:19
 +
+

*Thread Reply:* Let me pull in my colleagues

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Adam Bellemare + (adam.bellemare@shopify.com) +
+
2021-03-22 09:15:24
 +
+

*Thread Reply:* Standby

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Olessia D'Souza + (olessia.dsouza@shopify.com) +
+
2021-03-22 10:59:57
 +
+

*Thread Reply:* 👋 Hi Julien. I’m Olessia, I’m working on the metadata collection implementation with Adam. Some thought on this:

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Olessia D'Souza + (olessia.dsouza@shopify.com) +
+
2021-03-22 11:00:45
 +
+

*Thread Reply:* To start off, we’re thinking that there often isn’t a single owner, but rather a set of Stakeholders that evolve over time. So we’d like to be able to attach multiple entries, possibly of different types, to a Dataset. We’re also thinking that a dataset should have at least one owner. So a few things I’d like to confirm/discuss options:

  • If I were to stay true to the spec as it’s defined atm I wouldn’t be able to add a required facet. True/false?
  • According to the readme, “...emiting a new facet with the same name for the same entity replaces the previous facet instance for that entity entirely”. If we were to store multiple stakeholders, we’d have a field “stakeholders” and its value would be a list? This would make queries involving stakeholders not very straightforward. If the facet is overwritten every time, how do I a) add individuals to the list b) track changes to the list over time. Let me know what I’m missing, because based on what you said above tracking facet changes over time is possible.
  • Run events are issued by a scheduler. Why should it be in the domain of the scheduler to know the entire list of Stakeholders?
  • I noticed that Marquez has separate endpoints to capture information about Datasets, and some additional information beyond what’s described in the spec is required. In this context, we could add a required Stakeholder facets on a Dataset, and potentially even additional end points to add and remove Stakeholders. Is that a valid way to go about this, in your opinion?
    • +
+ +

Curious to hear your thoughts on all of this!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-03-24 17:06:50
 +
+

*Thread Reply:* > To start off, we’re thinking that there often isn’t a single owner, but rather a set of Stakeholders that evolve over time. So we’d like to be able to attach multiple entries, possibly of different types, to a Dataset. We’re also thinking > that a dataset should have at least one owner. So a few things I’d like to confirm/discuss options: +> -> If I were to stay true to the spec as it’s defined atm I wouldn’t be able to add a required facet. True/false? +Correct, The spec defines what facets looks like (and how you can make your own custom facets) but it does not make statements about whether facets are required. However, you can have your own validation and make certain things required if you wish to on the client side? +  +> - According to the readme, “...emiting a new facet with the same name for the same entity replaces the previous facet instance for that entity entirely”. If we were to store multiple stakeholders, we’d have a field “stakeholders” and its value would be a list?  +Yes, I would indeed consider such a facet on the dataset with the stakeholder.

+ +

> This would make queries involving stakeholders not very straightforward. If the facet is overwritten every time, how do I  +> a) add individuals to the list +You would provide the new list of stake holders. OpenLineage standardizes lineage collection and defines a format for expressing metadata. Marquez will keep track of how metadata has evolved over time.

+ +

> b) track changes to the list over time. Let me know what I’m missing, because based on what you said above tracking facet changes over time is possible. +Each event is an observation at a point in time. In a sense they are each immutable. There’s a “current” version but also all the previous ones stored in Marquez. +Marquez stores each version of a dataset it received through OpenLineage and exposes an API to see how that evolved over time.

+ +

> - Run events are issued by a scheduler. Why should it be in the domain of the scheduler to know the entire list of Stakeholders? +The scheduler emits the information that it knows about. For example: “I started this job and it’s reading from this dataset and is writing to this other dataset.” +It may or may not be in the domain of the scheduler to know the list of stakeholders. If not then you could emit different types of events to add a stakeholder facet to a dataset. We may want to refine the spec for that. Actually I would be curious to hear what you think should be the source of truth for stakeholders. It is not the intent to force everything coming from the scheduler.

  • example 1: stakeholders are people on call for the job, they are defined as part of the job and that also enables alerting
  • example 2: stakeholders are consumers of the jobs: they may be defined somewhere else
    • +
+ +

> - I noticed that Marquez has separate endpoints to capture information about Datasets, and some additional information beyond what’s described in the spec is required. In this context, we could add a required Stakeholder facets on a Dataset, and potentially even additional end points to add and remove Stakeholders. Is that a valid way to go about this, in your opinion?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-03-24 17:06:50
 +
+

*Thread Reply:* +Marquez existed before OpenLineage. In particular the /run end-point to create and update runs will be deprecated as the OpenLineage /lineage endpoint replaces it. At the moment we are mapping OpenLineage metadata to Marquez. Soon Marquez will have all the facets exposed in the Marquez API. (See: https://github.com/MarquezProject/marquez/pull/894/files) +We could make Marquez Configurable or Pluggable for validation purposes. There is already a notion of LineageListener for example. +Although Marquez collects the metadata. I feel like this validation would be better upstream or with some some other mechanism. The question is when do you create a dataset vs when do you become a stakeholder? What are the various stakeholder and what is the responsibility of the minimum one stakeholder? I would probably make it required to deploy the job that the stakeholder is defined. This would apply to the output dataset and would be collected in Marquez.

+ +

In general, you are very welcome to make suggestion on additional endpoints for Marquez and I’m happy to discuss this further as those ideas are progressing.

+ +

> Curious to hear your thoughts on all of this! +Thanks for taking the time!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-05-24 16:27:03
 +
+

*Thread Reply:* https://openlineage.slack.com/archives/C01CK9T7HKR/p1621887895004200

+
+ + +
+ + + } + + Julien Le Dem + (https://openlineage.slack.com/team/U01DCLP0GU9) +
+ + + +
+
+ <p>I have created a channel to discuss &lt;#C022MMLU31B|user-generated-metadata> since this came up in a few discussions.</p> +
+ + + + + + +
Original URL: https://openlineage.slack.com/archives/C01CK9T7HKR/p1621887895004200
+ + +
+ + Slack Conversation +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-03-24 18:58:00
 +
+

Thanks for the Python client submission @Maciej Obuchowski +https://github.com/OpenLineage/OpenLineage/pull/34

+
+ + +
+ + + } + + mobuchowski + (https://github.com/mobuchowski) +
+ + + +
#34 OpenLineage python client implementation
+
+ <p>Signed-off-by: Maciej Obuchowski <a href="mailto:maciej.obuchowski@getindata.com">maciej.obuchowski@getindata.com</a></p> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+ 🙌 Willy Lulciuc +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-03-24 18:59:50
 +
+

I also have added a spec to define a standard naming policy. Please review: https://github.com/OpenLineage/OpenLineage/pull/31/files

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-03-31 23:45:35
 +
+

We now have a python client! Thanks @Maciej Obuchowski

+ + + +
+ 👍 Maciej Obuchowski, Kevin Mellott, Ravi Suhag, Ross Turk, Willy Lulciuc, Mirko Raca +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Zachary Friedman + (zafriedman@gmail.com) +
+
2021-04-02 19:37:36
 +
+

Question, what do you folks see as the canonical mechanism for receiving OpenLineage events? Do you see an agent like statsd? Or do you see this as purely an API spec that services could implement? Do you see producers of lineage data writing code to send formatted OpenLineage payloads to arbitrary servers that implement receipt of these events? Curious what the long-term vision is here related to how an ecosystem of producers and consumers of payloads would interact?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-04-02 19:54:52
 +
+

*Thread Reply:* Marquez is the reference implementation for receiving events and tracking changes. But the definition of the API let’s other receive them (and also enables using openlineage events to sync between systems)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-04-02 19:55:32
 +
+

*Thread Reply:* In particular, Egeria is involved in enabling receiving and emitting openlineage

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Zachary Friedman + (zafriedman@gmail.com) +
+
2021-04-03 18:03:01
 +
+

*Thread Reply:* Thanks @Julien Le Dem. So to get specific, if dbt were to emit OpenLineage events, how would this work? Would dbt Cloud hypothetically allow users to configure an endpoint to send OpenLineage events to, similar in UI implementation to configuring a Stripe webhook perhaps? And then whatever server the user would input here would point to somewhere that implements receipt of OpenLineage payloads? This is all a very hypothetical example, but trying to ground it in something I have a solid mental model for.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2021-04-05 17:51:57
 +
+

*Thread Reply:* hypothetically speaking, that all sounds right. so a user, who, e.g., has a dbt pipeline and an AWS glue pipeline could configure both of those projects to point to the same open lineage service and get their entire lineage graph even if the two pipelines aren't connected.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2021-04-06 20:33:51
 +
+

*Thread Reply:* Yeah, OpenLineage events need to be published to a backend (can be Kafka, can be a graphDB, etc). Your Stripe webhook analogy is aligned with how events can be received. For example, in Marquez, we expose a /lineage endpoint that consumes OpenLineage events. We then map an OpenLineage event to the Marquez model (sources, datasets, jobs, runs) that’s persisted in postgres.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Zachary Friedman + (zafriedman@gmail.com) +
+
2021-04-07 10:47:06
 +
+

*Thread Reply:* Thanks both!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-04-13 20:52:53
 +
+

*Thread Reply:* sorry, I was away last week. Yes that sounds right.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Moravec + (jkb.moravec@gmail.com) +
+
2021-04-07 09:41:09
 +
+

Hi everyone, I just started discovering OpenLineage and Marquez, it looks great and the quick-start tutorial is very helpful! One question though, I pushed some metadata to Marquez using the Lineage POST endpoint, and when I try to confirm that everything was created using Marquez REST API, everything is there ... but I don't see these new objects in the Marquez UI... what is the best way how to investigate where the issue is?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2021-04-14 13:12:31
 +
+

*Thread Reply:* Welcome, @Jakub Moravec 👋 . Given that you're able to retrieve metadata using the marquezAPI, you should be able to also view dataset and job metadata in the UI. Mind using the search bar in the top right-hand corner in the UI to see if your metadata is searchable? The UI only renders jobs and datasets that are connected in the lineage graph. We're working towards a more general metadata exploration experience, but currently the lineage graph is the main experience.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakob Külzer + (jakob.kulzer@shopify.com) +
+
2021-04-08 11:23:18
 +
+

Hi friends, we're exploring OpenLineage and while building out integration for existing systems we realized there is no obvious way for an input to specify what "version" of that dataset is being consumed. For example, we have a job that rolls up a variable number of what OpenLineage calls dataset versions. By specifying only that dataset, we can't represent the specific instances of it that are actually rolled up. We think that would be a very important part of the lineage graph.

+ +

Are there any thoughts on how to address specific dataset versions? Is this where custom input facets would come to play?

+ +

Furthermore, based on the spec, it appears that events can provide dataset facets for both inputs and outputs and this seems to open the door to race conditions in which two runs concurrently create dataset versions of a dataset. Is this where the eventTime field is supposed to be used?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-04-13 20:56:42
 +
+

*Thread Reply:* Your intuition is right here. I think we should define an input facet that specifies which dataset version is being read. Similarly you would have an output facet that specifies what version is being produced. This would apply to storage layers like Deltalake and Iceberg as well.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-04-13 20:57:58
 +
+

*Thread Reply:* Regarding the race condition, input and output facets are attached to the run. The version of the dataset that was read is an attribute of a run and should not modify the dataset itself.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-04-13 21:01:34
 +
+

*Thread Reply:* See the Dataset description here: https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.md#core-lineage-model

+
+ + + + +
spec/OpenLineage.md
+
+ <pre><code># OpenLineage Spec + +## Specification + +The specification for OpenLineage is formalized as a JsonSchema [OpenLineage.json](OpenLineage.json). +An OpenAPI spec is also provided for HTTP based implementations: [OpenLineage.yml](OpenLineage.yml) +The documentation is published at: &lt;https://openlineage.github.io/&gt; +It allows extensions to the spec using `Custom Facets` as described in this document. + +## Core concepts + +### Core Lineage Model + +![Open Lineage model](OpenLineageModel.svg) + +- ****Run Event****: and event describing an observed state of a job run. It is required to at least send one event for a START transition and a COMPLETE/FAIL/ABORT transition. Aditional events are optional. + +- ****Job****: a process definition that consumes and produces datasets (defined as its inputs and outputs). It is identified by a unique name within a namespace (which is typicaly assigned to the scheduler starting the jobs). The **Job** evolves over time and this change is captured when the job runs. + +- ****Dataset****: an abstract representation of data. It has a unique name within a namespace derived from its physical location (for example db.host.database.schema.table). Typicaly, a **Dataset** changes when a job writing to it completes. Similarly to the **Job** and **Run** distinction, metadata that is more static from run to run is captured in a DatasetFacet (for example, the schema that does not change every run), what changes every **Run** is captured as an **InputFacet** or an **OutputFacet** (for example, what subset of the data set was read or written, like a time partition). + +- ****Run****: An instance of a running job with a start and completion (or failure) time. It is uniquely identified by an id relative to its job definition. + +- ****Facet****: A piece of metadata attached to one of the entities defined above. + +example: +Here is an example of a simple start run event not adding any facet information: +</code></pre> + +<p>{ + "transition": "START", + "eventTime": "2020-12-09T23:37:31.081Z", + "run": { + "runId": "345", + }, + "job": { + "namespace": "my-scheduler-namespace", + "name": "myjob.mytask", + }, + "inputs": [ + { + "namespace": "my-datasource-namespace", + "name": "instance.schema.table", + } + ], + "outputs": [ + { + "namespace": "my-datasource-namespace", + "name": "instance.schema.output_table", + } + ], + "producer": "<a href="https://github.com/OpenLineage/OpenLineage/blob/v1-0-0/client">https://github.com/OpenLineage/OpenLineage/blob/v1-0-0/client</a>" +}</p> + +<pre><code> +### Lifecycle + +The OpenLineage API defines events to capture the lifecycle of a **Run** for a given **Job**. +When a **job** is being **run**, we capture metadata by sending run events when the state of the job transitions to a different state. +We might observe different aspects of the job run at different stages. This means that different metadata might be collected in each event during the lyfecycle of a run. +All metadata is additive. for example, if more inputs or outputs are detected as the job is running we might send additional events specifically for those datasets without re-emiting previously observed inputs or outputs. +Example: + - When the run starts, we collect the following Metadata: + - Run Id + - Job id + - transition: START + - event time + - source location and version (ex: git sha) + - If known: Job inputs and outputs. (input schema, ...) + - When the run completes: + - Run Id + - Job id + - transition: COMPLETE + - event time + - Output datasets schema (and other metadata). + +### Facets + +Facets are pieces of metadata that can be attached to the core entities: +- Run +- Job +- Dataset (Inputs or Outputs) + +A facet is an atomic piece of metadata identified by its name. This means that emiting a new facet whith the same name for the same entity replaces the previous facet instance for that entity entirely). It is defined as a JSON object that can be either part of the spec or custom facets defined in a different project. + +Custom facets must use a distinct prefix named after the project defining them to avoid colision with standard facets defined in the [OpenLineage.json](OpenLineage.json) spec. +They have a \_schemaURL field pointing to the corresponding version of the facet schema (as a JSONPointer: [$ref URL location](&lt;https://swagger.io/docs/specification/using-ref/&gt;) ). + +Example: &lt;https://raw.githubusercontent.com/OpenLineage/OpenLineage/main/spec/OpenLineage.json#/definitions/MyCustomJobFacet&gt; + +The versioned URL must be an immutable pointer to the version of the facet schema. For example, it should include a tag of a git sha and not a branch name. This should also be a canonical URL. There should be only one URL used for a given version of a schema. + +Custom facets can be promoted to the standard by including them in the spec. + +### Standard Facets + +#### Run Facets + +- ****nominalTime****: Captures the time this run is scheduled for. This is a typical usage for time based scheduled job. The job has a nominal schedule time that will be different from the actual time it is running at. + +- ****parent****: Captures the parent job and Run when the run was spawn from a parent run. For example in the case of Airflow, there's a run for the DAG that then spawns runs for individual tasks that would refer to the parent run as the DAG run. Similarly when a SparkOperator starts a Spark job, this creates a separate run that refers to the task run as its parent. + +#### Job Facets + +- ****sourceCodeLocation****: Captures the source code location and version (example: git sha) of the job. + +- ****sql****: Capture the SQL query if this job is a SQL query. + +#### Dataset Facets + +- ****schema****: Captures the schema of the dataset + +- ****dataSource****: Captures the Database instance containing this datasets (ex: Database schema. Object store bucket, ...) + +#### Input Dataset Facets + +**Note: for now there no input facets defined** + +#### Output Dataset Facets + +**Note: for now there no output facets defined** + +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Stephen Pimentel + (stephenpiment@gmail.com) +
+
2021-04-14 18:20:42
 +
+

Hi everyone! I’m exploring what existing, open-source integrations are available, specifically for Spark, Airflow, and Trino (PrestoSQL). My team is looking both to use and contribute to these integrations. I’m aware of the integration in the Marquez repo: +• Spark: https://github.com/MarquezProject/marquez/tree/main/integrations/spark +• Airflow: https://github.com/MarquezProject/marquez/tree/main/integrations/airflow +Are there other efforts I should be aware of, whether for these two or for Trino? Thanks for any information!

+ + + +
+ 👋 Arthur Wiedmer, Maciej Obuchowski, Peter Hicks +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Zachary Friedman + (zafriedman@gmail.com) +
+
2021-04-19 16:17:06
 +
+

*Thread Reply:* I think for Trino integration you'd be looking at writing a Trino extractor if I'm not mistaken, yes?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Zachary Friedman + (zafriedman@gmail.com) +
+
2021-04-19 16:17:23
 +
+

*Thread Reply:* But extractor would obviously be at the Marquez layer not OpenLineage

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Zachary Friedman + (zafriedman@gmail.com) +
+
2021-04-19 16:19:00
 +
+

*Thread Reply:* And hopefully the metadata you'd be looking to extract from Trino wouldn't have any connector-specific syntax restrictions.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Antonio Moctezuma + (antoniomoctezuma@northwesternmutual.com) +
+
2021-04-16 15:37:24
 +
+

Hey all! Right now I am working on getting OpenLineage integrated with some microservices here at Northwestern Mutual and was looking for some advice. The current service I am trying to integrate it with moves files from one AWS S3 bucket to another so i was hoping to track that movement with OpenLineage. However by my understanding the inputs that would be passed along in a runEvent are meant to be datasets that have schema and other properties. But I wanted to have that input represent the file being moved. Is this a proper usage of Open Lineage? Or is this a use case that is still being developed? Any and all help is appreciated!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-04-19 21:42:14
 +
+

*Thread Reply:* This is a proper usage. That schema is optional if it’s not available.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-04-19 21:43:27
 +
+

*Thread Reply:* You would model it as a job reading from a folder (the input dataset) in the input bucket and writing to a folder (the output dataset) in the output bucket

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-04-19 21:43:58
 +
+

*Thread Reply:* This is similar to how this is modeled in the spark integration (spark job reading and writing to s3 buckets)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-04-19 21:47:06
 +
+

*Thread Reply:* for reference: getting the urls for the inputs: https://github.com/MarquezProject/marquez/blob/c5e5d7b8345e347164aa5aa173e8cf35062[…]marquez/spark/agent/lifecycle/plan/HadoopFsRelationVisitor.java

+
+ + + + +
<https://github.com/MarquezProject/marquez/blob/c5e5d7b8345e347164aa5aa173e8cf35062641e8/integrations/spark/src/main/java/marquez/spark/agent/lifecycle/plan/HadoopFsRelationVisitor.java | HadoopFsRelationVisitor.java>
+
+ <pre><code> public List&amp;lt;Dataset&amp;gt; apply(LogicalPlan x) { +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/MarquezProject/marquez">MarquezProject/marquez</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-04-19 21:47:54
 +
+

*Thread Reply:* getting the output URL: https://github.com/MarquezProject/marquez/blob/c5e5d7b8345e347164aa5aa173e8cf35062[…]ark/agent/lifecycle/plan/InsertIntoHadoopFsRelationVisitor.java

+
+ + + + +
<https://github.com/MarquezProject/marquez/blob/c5e5d7b8345e347164aa5aa173e8cf35062641e8/integrations/spark/src/main/java/marquez/spark/agent/lifecycle/plan/InsertIntoHadoopFsRelationVisitor.java | InsertIntoHadoopFsRelationVisitor.java>
+
+ <pre><code> return Collections.singletonList( +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/MarquezProject/marquez">MarquezProject/marquez</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-04-19 21:48:48
 +
+

*Thread Reply:* See the spec (comments welcome) for the naming of S3 datasets: https://github.com/OpenLineage/OpenLineage/pull/31/files#diff-e3a8184544e9bc70d8a12e76b58b109051c182a914f0b28529680e6ced0e2a1cR87

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Antonio Moctezuma + (antoniomoctezuma@northwesternmutual.com) +
+
2021-04-20 11:11:38
 +
+

*Thread Reply:* Hey Julien, thank you so much for getting back to me. I'll take a look at the documentation/implementations you've sent me and will reach out if I have anymore questions. Thanks again!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Antonio Moctezuma + (antoniomoctezuma@northwesternmutual.com) +
+
2021-04-20 17:39:24
 +
+

*Thread Reply:* @Julien Le Dem I left a quick comment on that spec PR you mentioned. Just wanted to let you know.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-04-20 17:49:15
 +
+

*Thread Reply:* thanks

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Josh Quintus + (josh.quintus@gmail.com) +
+
2021-04-28 09:41:45
 +
+

Hello all. I was reading through the OpenLineage documentation on GitHub and noticed a very minor typo (an instance where and should have been an). I was just about to create a PR for it but wanted to check with someone to see if that would be something that the team is interested in.

+ +

Thanks for the tool, I'm looking forward to learning more about it.

+ + + +
+ 👍 Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-04-28 20:56:53
 +
+

*Thread Reply:* Thank you! Please do fix typos, I’ll approve your PR.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Josh Quintus + (josh.quintus@gmail.com) +
+
2021-04-28 23:21:44
 +
+

*Thread Reply:* No problem. Here's the PR. https://github.com/OpenLineage/OpenLineage/pull/47

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Josh Quintus + (josh.quintus@gmail.com) +
+
2021-04-28 23:22:41
 +
+

*Thread Reply:* Once I fixed the ones I saw I figured "Why not just run it through a spell checker just in case... " and found a few additional ones.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2021-05-20 16:30:05
 +
+

For your enjoyment, @Julien Le Dem was on the Data Engineering Podcast talking about OpenLineage!

+ +

https://www.dataengineeringpodcast.com/openlineage-data-lineage-specification-episode-187/

+
+
Data Engineering Podcast
+ + + +
Unlocking The Power of Data Lineage In Your Platform with OpenLineage - Episode 187
+
+ Data lineage is the common thread that ties together all of your data pipelines, workflows, and systems. In order to get a holistic understanding of your data +
+ + + + + + + + +
Original URL: https://www.dataengineeringpodcast.com/openlineage-data-lineage-specification-episode-187/
+ + + +
+ + + +
+ 🙌 Willy Lulciuc, Maciej Obuchowski, Peter Hicks, Mario Measic +
+ +
+ ❤️ Willy Lulciuc, Maciej Obuchowski, Peter Hicks, Rogier Werschkull, A Pospiech, Kedar Rajwade, James Le +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2021-05-20 16:30:09
 +
+

share and enjoy 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-05-21 18:21:23
 +
+

Also happened yesterday: OpenLineage being accepted by the LFAI&Data.

+ + + +
+ 🎉 Abe Gong, Willy Lulciuc, Peter Hicks, Maciej Obuchowski, Daniel Henneberger, Harel Shein, Antonio Moctezuma, Josh Quintus, Mariusz Górski, James Le +
+ +
+ 👏 Matt Turck +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2021-05-21 19:20:55
 +
+

*Thread Reply:* Huge milestone! 🙌💯🎊

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-05-24 16:24:55
 +
+

I have created a channel to discuss <#C022MMLU31B|user-generated-metadata> since this came up in a few discussions.

+ + + +
+ 🙌 Willy Lulciuc +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jonathon Mitchal + (bigmit83@gmail.com) +
+
2021-05-31 01:28:35
 +
+

hey guys, does anyone have any sample openlineage schemas for S3 please? potentially including facets for attributes in a parquet file? that would help heaps thanks. i am trying to slowly bring in a common metadata interface and this will help shape some of the conversations 🙂 with a move to marquez/datahub et al over time

+ + + +
+ 🙌 Willy Lulciuc +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2021-06-01 17:56:16
 +
+

*Thread Reply:* We currently don’t have S3 (or distributed filesystem specific facets) at the moment, but such support would be a great addition! @Julien Le Dem would be best to answer if any work has been done in this area 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2021-06-01 17:57:19
 +
+

*Thread Reply:* Also, happy to answer any Marquez specific questions, @Jonathon Mitchal when you’re thinking of making the move. Marquez supports OpenLineage out of the box 🙌

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-06-01 19:58:21
 +
+

*Thread Reply:* @Jonathon Mitchal You can follow the naming strategy here for referring to a S3 dataset: https://github.com/OpenLineage/OpenLineage/blob/main/spec/Naming.md#s3

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/spec/Naming.md | Naming.md>
+
+ <p><strong>Naming</strong></p> + +<p>We define the unique name strategy per datasource to ensure it is followed uniformly independently from the type of job consuming and producing data. <br /> +The namespace for a dataset is the unique name for its datasource. <br /> +For jobs it is the scheduler.</p> + +<p><strong>Datasets:</strong></p> + +<p>The namespace and name of a datasource can be combined to form a URI (scheme:[//authority]path)</p> + +<p>• Namespace = scheme:[//authority] (the datasource) +• Name = path (the datasets)</p> + +<p><strong>Data warehouses/data bases.</strong></p> + +<p>Datasets are called tables. Tables are organised in databases and schemas.</p> + +<p><strong>Postgres:</strong></p> + +<p>Datasource hierarchy:</p> + +<p>• Host +• Port</p> + +<p>Naming hierarchy:</p> + +<p>• Database +• Schema +• Table</p> + +<p>Identifier:</p> + +<p>• Namespace: postgres://{host}:{port} of the service instance. + • Scheme = postgres + • Authority = {host}:{port} +• Unique name: {database}.{schema}.{table} + • URI = postgres://{host}:{port}/{database}.{schema}.{table}</p> + +<p><strong>Snowflake</strong></p> + +<p>See: <a href="https://docs.snowflake.com/en/sql-reference/identifiers.html">Object Identifiers — Snowflake Documentation</a></p> + +<p>Datasource hierarchy:</p> + +<p>• account name</p> + +<p>Naming hierarchy:</p> + +<p>• Database: {database name} =&gt; unique across the account +• Schema: {schema name} =&gt; unique within the database +• Table: {table name} =&gt; unique within the schema</p> + +<p>Identifier:</p> + +<p>• Namespace: snowflake://{account name} + • Scheme = snowflake + • Authority = {account name} +• Name: {database}.{schema}.{table} + • URI = snowflake://{account name}/{database}.{schema}.{table}</p> + +<p><strong>BigQuery</strong></p> + +<p>See: <br /> +<a href="https://cloud.google.com/resource-manager/docs/creating-managing-projects|Creating and managing projects | Resource Manager Documentation">https://cloud.google.com/resource-manager/docs/creating-managing-projects|Creating and managing projects | Resource Manager Documentation</a> <br /> +<a href="https://cloud.google.com/bigquery/docs/datasets-intro|Introduction to datasets | BigQuery">https://cloud.google.com/bigquery/docs/datasets-intro|Introduction to datasets | BigQuery</a> <br /> +<a href="https://cloud.google.com/bigquery/docs/tables-intro|Introduction to tables | BigQuery">https://cloud.google.com/bigquery/docs/tables-intro|Introduction to tables | BigQuery</a></p> + +<p>Datasource hierarchy:</p> + +<p>• bigquery</p> + +<p>Naming hierarchy:</p> + +<p>• Project Name: {project name} =&gt; is not unique +• Project number: {project number} =&gt; numeric: is unique across google cloud +• Project ID: {project id} =&gt; readable: is unique across google cloud +• dataset: {dataset name} =&gt; is unique within a project +• table: {table name} =&gt; is unique within a dataset</p> + +<p>Identifier :</p> + +<p>• Namespace: bigquery + • Scheme = bigquery + • Authority = +• Unique name: {project id}.{dataset name}.{table name} + • URI = bigquery:{project id}.{schema}.{table}</p> + +<p><strong>Distributed file systems/blob stores</strong> +<strong>GCS</strong></p> + +<p>Datasource hierarchy: none, naming is global</p> + +<p>Naming hierarchy:</p> + +<p>• bucket name =&gt; globally unique +• Path</p> + +<p>Identifier :</p> + +<p>• Namespace: gs://{bucket name} + • Scheme = gs + • Authority = {bucket name} +• Unique name: {path} + • URI = gs://{bucket name}{path}</p> + +<p><strong>S3</strong></p> + +<p>Naming hierarchy:</p> + +<p>• bucket name =&gt; globally unique +• Path</p> + +<p>Identifier :</p> + +<p>• Namespace: s3://{bucket name} + • Scheme = s3 + • Authority = {bucket name} +• Unique name: {path} + • URI = s3://{bucket name}{path}</p> + +<p><strong>HDFS</strong></p> + +<p>Naming hierarchy:</p> + +<p>• Namenode: host + port +• Path</p> + +<p>Identifier :</p> + +<p>• Namespace: hdfs://{namenode host}:{namenode port} + • Scheme = hdfs + • Authority = {namenode host}:{namenode port} +• Unique name: {path} + • URI = hdfs://{namenode host}:{namenode port}{path}</p> + +<p><strong>Schedulers</strong> +<strong>Airflow</strong></p> + +<p>Naming hierarchy:</p> + +<p>• job =&gt; workspace/DAG +• Task =&gt; unique within the job</p> + +<p>Identifier :</p> + +<p>• Namespace: {scheduler namespace} +• Unique name: {job}.{task}</p> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-06-01 19:59:30
 +
+

*Thread Reply:* There is no facet yet for the attributes of a Parquet file. I can give you feedback if you want to start defining one. https://github.com/OpenLineage/OpenLineage/blob/main/CONTRIBUTING.md#proposing-changes

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/CONTRIBUTING.md | CONTRIBUTING.md>
+
+ <p><strong>Contributing to OpenLineage</strong></p> + +<p>This project welcomes contributors from any organization or background, provided they are <br /> +willing to follow the simple processes outlined below, as well as adhere to the <br /> +&lt;CODE<em>OF</em>CONDUCT.md|Code of Conduct&gt;.</p> + +<p><strong>Joining the community</strong></p> + +<p>The community collaborates primarily through <code>GitHub</code> and the instance messaging tool, <code>Slack</code>. <br /> +There is also a mailing list. <br /> +See how to join <a href="https://github.com/OpenLineage/OpenLineage#community">here</a></p> + +<p><strong>Reporting an Issue</strong></p> + +<p>Please use the <a href="https://github.com/OpenLineage/OpenLineage/issues">issues</a> section of the OpenLineage repository and search for a similar problem. If you don't find it, submit your bug, question, proposal or feature request.</p> + +<p>In the case of bugs, please provide steps to reproduce it and tag your issue with "bug"</p> + +<p><strong>Contributing to the project</strong> +<strong>Creating Pull Requests</strong></p> + +<p>Before sending a Pull Request with significant changes, please use the <a href="https://github.com/OpenLineage/OpenLineage/issues">issue tracker</a> to discuss the potential improvements you want to make.</p> + +<p>OpenLineage uses <a href="https://help.github.com/articles/about-collaborative-development-models/">GitHub's fork and pull model</a> <br /> +to create a contribution.</p> + +<p>Every contribution is signed to say that the contributor has the rights to make the contribution and <br /> +agrees with the &lt;why-the-dco.md|Developer Certificate of Origin (DCO)&gt;</p> + +<p><strong>Proposing changes</strong></p> + +<p>Create an issue and prefix the description with "[PROPOSAL]"</p> + +<p>In the description provide the following sections:</p> + +<p>• Purpose (Why?): What is the use case this is for. +• Proposed implementation (How?): Quick description of how do you propose to implement it. Are you proposing a new facet?</p> + +<p>This can be just a couple paragraphs to start with.</p> + +<p><strong>First-Time Contributors</strong></p> + +<p>If this is your first contribution to open source, you can <a href="https://github.com/firstcontributions/first-contributions#first-contributions">follow this tutorial</a> or check <a href="https://egghead.io/courses/how-to-contribute-to-an-open-source-project-on-github">this video series</a> to learn about the contribution workflow with GitHub.</p> + +<p>Look tickets labeled <https://github.com/OpenLineage/OpenLineage/labels/good%20first%20issue|'good first issue'> and <https://github.com/OpenLineage/OpenLineage/labels/help%20wanted|'help wanted'>. These are a great starting point if you want to contribute. Don't hesitate to ask questions about the issue if you are not sure about the strategy to follow.</p> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-06-01 20:00:50
 +
+

*Thread Reply:* Adding Parquet metadata as a facet would make a lot of sense. It is mainly a matter of specifying what the json would look like

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-06-01 20:01:54
 +
+

*Thread Reply:* for reference the parquet metadata is defined here: https://github.com/apache/parquet-format/blob/master/src/main/thrift/parquet.thrift

+
+ + + + +
<https://github.com/apache/parquet-format/blob/master/src/main/thrift/parquet.thrift | parquet.thrift>
+
+ <p>``` +/<em>*</em>* + <em>* Licensed to the Apache Software Foundation (ASF) under one + *</em> or more contributor license agreements. See the NOTICE file + <em>* distributed with this work for additional information + *</em> regarding copyright ownership. The ASF licenses this file + <em>* to you under the Apache License, Version 2.0 (the + *</em> "License"); you may not use this file except in compliance + <em>* with the License. You may obtain a copy of the License at + *</em> + <em>* <a href="http://www.apache.org/licenses/LICENSE-2.0">http://www.apache.org/licenses/LICENSE-2.0</a> + *</em> + <em>* Unless required by applicable law or agreed to in writing, + *</em> software distributed under the License is distributed on an + <em>* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + *</em> KIND, either express or implied. See the License for the + <em>* specific language governing permissions and limitations + *</em> under the License. + **/</p> + +<p>/<em>*</em>* + <em>* File format description for the parquet file format + *</em>/ +namespace cpp parquet +namespace java org.apache.parquet.format</p> + +<p>/<em>*</em>* + <em>* Types supported by Parquet. These types are intended to be used in combination + *</em> with the encodings to control the on disk storage format. + <em>* For example INT16 is not included as a type since a good encoding of INT32 + *</em> would handle this. + **/ +enum Type { + BOOLEAN = 0; + INT32 = 1; + INT64 = 2; + INT96 = 3; // deprecated, only used by legacy implementations. + FLOAT = 4; + DOUBLE = 5; + BYTE<em>ARRAY = 6; + FIXED</em>LEN<em>BYTE</em>ARRAY = 7; +}</p> + +<p>/<strong><em>* + *</em> DEPRECATED: Common types used by frameworks(e.g. hive, pig) using parquet. + <em>* ConvertedType is superseded by LogicalType. This enum should not be extended. + *</em> + <em>* See LogicalTypes.md for conversion between ConvertedType and LogicalType. + *</em>/ +enum ConvertedType { + /<em>*</strong> a BYTE_ARRAY actually contains UTF8 encoded chars *</em>/ + UTF8 = 0;</p> + +<p>/<em>*</em>* a map is converted as an optional field containing a repeated key/value pair **/ + MAP = 1;</p> + +<p>/<em>*</em>* a key/value pair is converted into a group of two fields **/ + MAP<em>KEY</em>VALUE = 2;</p> + +<p>/<em>*</em>* a list is converted into an optional field containing a repeated field for its + <em>* values *</em>/ + LIST = 3;</p> + +<p>/<em>*</em>* an enum is converted into a binary field **/ + ENUM = 4;</p> + +<p>/<em>*</em>* + <em>* A decimal value. + *</em> + <em>* This may be used to annotate binary or fixed primitive types. The + *</em> underlying byte array stores the unscaled value encoded as two's + <em>* complement using big-endian byte order (the most significant byte is the + *</em> zeroth element). The value of the decimal is the value <em>* 10^{-scale}. + *</em> + <em>* This must be accompanied by a (maximum) precision and a scale in the + *</em> SchemaElement. The precision specifies the number of digits in the decimal + <em>* and the scale stores the location of the decimal point. For example 1.23 + *</em> would have precision 3 (3 total digits) and scale 2 (the decimal point is + <em>* 2 digits over). + *</em>/ + DECIMAL = 5;</p> + +<p>/<em>*</em>* + <em>* A Date + *</em> + <em>* Stored as days since Unix epoch, encoded as the INT32 physical type. + *</em> + **/ + DATE = 6;</p> + +<p>/<em>*</em>* + <em>* A time + *</em> + <em>* The total number of milliseconds since midnight. The value is stored + *</em> as an INT32 physical type. + **/ + TIME_MILLIS = 7;</p> + +<p>/<em>*</em>* + <em>* A time. + *</em> + <em>* The total number of microseconds since midnight. The value is stored as + *</em> an INT64 physical type. + **/ + TIME_MICROS = 8;</p> + +<p>/<em>*</em>* + <em>* A date/time combination + *</em> + <em>* Date and time recorded as milliseconds since the Unix epoch. Recorded as + *</em> a physical type of INT64. + **/ + TIMESTAMP_MILLIS = 9;</p> + +<p>/<em>*</em>* + <em>* A date/time combination + *</em> + <em>* Date and time recorded as microseconds since the Unix epoch. The value is + *</em> stored as an INT64 physical type. + **/ + TIMESTAMP_MICROS = 10;</p> + +<p>/<em>*</em>* + <em>* An unsigned integer value. + *</em> + <em>* The number describes the maximum number of meaningful data bits in + *</em> the stored value. 8, 16 and 32 bit values are stored using the + <em>* INT32 physical type. 64 bit values are stored using the INT64 + *</em> physical type. + <em>* + *</em>/ + UINT<em>8 = 11; + UINT</em>16 = 12; + UINT<em>32 = 13; + UINT</em>64 = 14;</p> + +<p>/<em>*</em>* + <em>* A signed integer value. + *</em> + <em>* The number describes the maximum number of meaningful data bits in + *</em> the stored value. 8, 16 and 32 bit values are stored using the + <em>* INT32 physical type. 64 bit values are stored using the INT64 + *</em> physical type. + <em>* + *</em>/ + INT<em>8 = 15; + INT</em>16 = 16; + INT<em>32 = 17; + INT</em>64 = 18;</p> + +<p>/<em>*</em>* + <em>* An embedded JSON document + *</em> + <em>* A JSON document embedded within a single UTF8 column. + *</em>/ + JSON = 19;</p> + +<p>/<em>*</em>* + <em>* An embedded BSON document + *</em> + <em>* A BSON document embedded within a single BINARY column. + *</em>/ + BSON = 20;</p> + +<p>/<em>*</em>* + <em>* An interval of time + *</em> + <em>* This type annotates data stored as a FIXED_LEN_BYTE_ARRAY of length 12 + *</em> This data is composed of three separate little endian unsigned + <em>* integers. Each stores a component of a duration of time. The first + *</em> integer identifies the number of months associated with the duration, + <em>* the second identifies the number of days associated with the duration + *</em> and the third identifies the number of milliseconds associated with + <em>* the provided duration. This duration of time is independent of any + *</em> particular timezone or date. + **/ + INTERVAL = 21; +}</p> + +<p>/<strong><em>* + *</em> Representation of Schemas + <em>*/ +enum FieldRepetitionType { + /</em><em></strong> This field is required (can not be null) and each record has exactly 1 value. *</em>/ + REQUIRED = 0;</p> + +<p>/<em>*</em>* The field is optional (can be null) and each record has 0 or 1 values. **/ + OPTIONAL = 1;</p> + +<p>/<em>*</em>* The field is repeated and can contain 0 or more values **/ + REPEATED = 2; +}</p> + +<p>/<strong><em>* + *</em> Statistics per row group and per page + <em>* All fields are optional. + *</em>/ +struct Statistics { + /<em>*</strong> + *</em> DEPRECATED: min and max value of the column. Use min<em>value and max</em>value. + <em>* + *</em> Values are encoded using PLAIN encoding, except that variable-length byte + <em>* arrays do not include a length prefix. + *</em> + <em>* These fields encode min and max values determined by signed comparison + *</em> only. New files should use the correct order for a column's logical type + <em>* and store the values in the min_value and max_value fields. + *</em> + <em>* To support older readers, these may be set when the column order is + *</em> signed. + <strong>/ + 1: optional binary max; + 2: optional binary min; + /<em>*</strong> count of null value in the column <strong>/ + 3: optional i64 null_count; + /</em><em></strong> count of distinct values occurring <strong>/ + 4: optional i64 distinct_count; + /</em><em></strong> + *</em> Min and max values for the column, determined by its ColumnOrder. + <em>* + *</em> Values are encoded using PLAIN encoding, except that variable-length byte + <em>* arrays do not include a length prefix. + *</em>/ + 5: optional binary max<em>value; + 6: optional binary min</em>value; +}</p> + +<p>/<em>*</em>* Empty structs to use as logical type annotations **/ +struct StringType {} // allowed for BINARY, must be encoded with UTF-8 +struct UUIDType {} // allowed for FIXED[16], must encoded raw UUID bytes +struct MapType {} // see LogicalTypes.md +struct ListType {} // see LogicalTypes.md +struct EnumType {} // allowed for BINARY, must be encoded with UTF-8 +struct DateType {} // allowed for INT32</p> + +<p>/<em>*</em>* + <em>* Logical type to annotate a column that is always null. + *</em> + <em>* Sometimes when discovering the schema of existing data, values are always + *</em> null and the physical type can't be determined. This annotation signals + <em>* the case where the physical type was guessed from all null values. + *</em>/ +struct NullType {} // allowed for any physical type, only null values stored</p> + +<p>/<em>*</em>* + <em>* Decimal logical type annotation + *</em> + <em>* To maintain forward-compatibility in v1, implementations using this logical + *</em> type must also set scale and precision on the annotated SchemaElement. + <em>* + *</em> Allowed for physical types: INT32, INT64, FIXED, and BINARY + **/ +struct DecimalType { + 1: required i32 scale + 2: required i32 precision +}</p> + +<p>/<em>*</em>* Time units for logical types **/ +struct MilliSeconds {} +struct MicroSeconds {} +struct NanoSeconds {} +union TimeUnit { + 1: MilliSeconds MILLIS + 2: MicroSeconds MICROS + 3: NanoSeconds NANOS +}</p> + +<p>/<em>*</em>* + <em>* Timestamp logical type annotation + *</em> + ** Allowed for physica…</p> +
+ + + + + + + +
+ + <a href="https://github.com/apache/parquet-format">apache/parquet-format</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jonathon Mitchal + (bigmit83@gmail.com) +
+
2021-06-01 23:20:50
 +
+

*Thread Reply:* Thats awesome, thanks for the guidance Willy and Julien ... will report back on how we get on

+ + + +
+ 🙏 Willy Lulciuc +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Pedram + (pedram@hightouch.io) +
+
2021-06-01 17:52:08
 +
+

hi all! just wanted to introduce myself, I'm the Head of Data at Hightouch.io, we build reverse etl pipelines from the warehouse into various destinations. I've been following OpenLineage for a while now and thought it would be nice to build and expose our runs via the standard and potentially save that back to the warehouse for analysis/alerting. Really interesting concept, looking forward to playing around with it

+ + + +
+ 👋 Willy Lulciuc, Ross Turk +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-06-01 20:02:34
 +
+

*Thread Reply:* Welcome! Let use know if you have any questions

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Leo + (leorobinovitch@gmail.com) +
+
2021-06-03 19:22:10
 +
+

Hi all! I have a noob question. As I understand it, one of the main purposes of OpenLineage is to avoid runaway proliferation of bespoke connectors for each data lineage/cataloging/provenance tool to each data source/job scheduler/query engine etc. as illustrated in the problem diagram from the main repo below.

+ +

My understanding is that instead, things push to OpenLineage which provides pollable endpoints for metadata tools.

+ +

I’m looking at Amundsen, and it seems to have bespoke connectors, but these are pull-based - I don’t need to instrument my data resources to push to Amundsen, I just need to configure Amundsen to poll my data resources (e.g. the Postgres metadata extractor here).

+ +

Can OpenLineage do something similar where I can just point it at something to extract metadata from it, rather than instrumenting that thing to push metadata to OpenLineage? If not, I’m wondering why?

+ +

Is it the case that Open Lineage defines the general framework but doesn’t actually enforce push or pull-based implementations, it just so happens that the reference implementation (Marquez) uses push?

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-06-04 04:45:15
 +
+

*Thread Reply:* > Is it the case that Open Lineage defines the general framework but doesn’t actually enforce push or pull-based implementations, it just so happens that the reference implementation (Marquez) uses push? +Yes, at core OpenLineage just enforces format of the event. We also aim to provide clients - REST, later Kafka, etc. and some reference implementations - which are now in Marquez repo. https://raw.githubusercontent.com/OpenLineage/OpenLineage/main/doc/Scope.png

+ +

There are several differences between push and poll models. Most important one is that with push model, latency between your job and emitting OpenLineage events is very low. With some systems, with internal, push based model you have more runtime metadata available than when looking from outside. Another one would be that naive poll implementation would need to "rebuild the world" on each change. There are also disadvantages, such as that usually, it's easier to write plugin that extracts data from outside the system than hooking up to the internals.

+ +

Integration with Amundsen specifically is planned. Although, right now it seems to me that way to do it is to bypass the databuilder framework and push directly to underlying database, such as Neo4j, or make Marquez backend for Metadata Service: https://raw.githubusercontent.com/amundsen-io/amundsen/master/docs/img/Amundsen_Architecture.png

+
+ + + + +
+
+ +
+ + + + + + + + +
Original URL: https://raw.githubusercontent.com/OpenLineage/OpenLineage/main/doc/Scope.png
+ + + +
+
+ + + + +
+
+ +
+ + + + + + + + +
Original URL: https://raw.githubusercontent.com/amundsen-io/amundsen/master/docs/img/Amundsen_Architecture.png
+ + + +
+ + + +
+ ❤️ Julien Le Dem +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Leo + (leorobinovitch@gmail.com) +
+
2021-06-04 10:39:51
 +
+

*Thread Reply:* This is really helpful, thank you @Maciej Obuchowski!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Leo + (leorobinovitch@gmail.com) +
+
2021-06-04 10:40:59
 +
+

*Thread Reply:* Similar to what you say about push vs pull, I found DataHub’s comment to be interesting yesterday: +> Push is better than pull: While pulling metadata directly from the source seems like the most straightforward way to gather metadata, developing and maintaining a centralized fleet of domain-specific crawlers quickly becomes a nightmare. It is more scalable to have individual metadata providers push the information to the central repository via APIs or messages. This push-based approach also ensures a more timely reflection of new and updated metadata.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-06-04 21:59:59
 +
+

*Thread Reply:* yes. You can also “pull-to-push” for things that don’t push.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mariusz Górski + (gorskimariusz13@gmail.com) +
+
2021-06-17 10:01:37
 +
+

*Thread Reply:* @Maciej Obuchowski any particular reason for bypassing databuilder and go directly to neo4j? By design databuilder is supposed to be very abstract so any kind of backend can be used with Amundsen. Currently there are at least 4 and neo4j is just one of them.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-06-17 10:28:52
 +
+

*Thread Reply:* Databuilder's pull model is very different than OpenLineage's push model, where the events are generated while the dataset itself is generated.

+ +

So, how would you see using it? Just to proxy the events to concrete search and metadata backend?

+ +

I'm definitely not an Amundsen expert, so feel free to correct me if I'm getting it wrong.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-07-07 19:59:28
 +
+

*Thread Reply:* @Mariusz Górski my slide that Maciej is referring to might be a bit misleading. The Amundsen integration does not exist yet. Please add your input in the ticket: https://github.com/OpenLineage/OpenLineage/issues/86

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/issues/86|#86 Add backend for sending lineage to Amundsen>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mariusz Górski + (gorskimariusz13@gmail.com) +
+
2021-07-09 02:22:06
 +
+

*Thread Reply:* thanks Julien! will take a look

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kedar Rajwade + (kedar@cloudzealous.com) +
+
2021-06-08 10:00:47
 +
+

@here Hello, My name is Kedar Rajwade. I happened to come across the OpenLineage project and it looks quite interesting. Is there some kind of getting start guide that I can follow. Also are there any weekly/bi-weekly calls that I can attend to know the current/future plans ?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-06-08 14:16:42
 +
+

*Thread Reply:* Welcome! You can look here: https://github.com/OpenLineage/OpenLineage/blob/main/CONTRIBUTING.md

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/CONTRIBUTING.md | CONTRIBUTING.md>
+
+ <p><strong>Contributing to OpenLineage</strong></p> + +<p>This project welcomes contributors from any organization or background, provided they are <br /> +willing to follow the simple processes outlined below, as well as adhere to the <br /> +&lt;CODE<em>OF</em>CONDUCT.md|Code of Conduct&gt;.</p> + +<p><strong>Joining the community</strong></p> + +<p>The community collaborates primarily through <code>GitHub</code> and the instance messaging tool, <code>Slack</code>. <br /> +There is also a mailing list. <br /> +See how to join <a href="https://github.com/OpenLineage/OpenLineage#community">here</a></p> + +<p><strong>Reporting an Issue</strong></p> + +<p>Please use the <a href="https://github.com/OpenLineage/OpenLineage/issues">issues</a> section of the OpenLineage repository and search for a similar problem. If you don't find it, submit your bug, question, proposal or feature request.</p> + +<p>In the case of bugs, please provide steps to reproduce it and tag your issue with "bug"</p> + +<p><strong>Contributing to the project</strong> +<strong>Creating Pull Requests</strong></p> + +<p>Before sending a Pull Request with significant changes, please use the <a href="https://github.com/OpenLineage/OpenLineage/issues">issue tracker</a> to discuss the potential improvements you want to make.</p> + +<p>OpenLineage uses <a href="https://help.github.com/articles/about-collaborative-development-models/">GitHub's fork and pull model</a> <br /> +to create a contribution.</p> + +<p>Every contribution is signed to say that the contributor has the rights to make the contribution and <br /> +agrees with the &lt;why-the-dco.md|Developer Certificate of Origin (DCO)&gt;</p> + +<p><strong>Proposing changes</strong></p> + +<p>Create an issue and prefix the description with "[PROPOSAL]"</p> + +<p>In the description provide the following sections:</p> + +<p>• Purpose (Why?): What is the use case this is for. +• Proposed implementation (How?): Quick description of how do you propose to implement it. Are you proposing a new facet?</p> + +<p>This can be just a couple paragraphs to start with.</p> + +<p><strong>First-Time Contributors</strong></p> + +<p>If this is your first contribution to open source, you can <a href="https://github.com/firstcontributions/first-contributions#first-contributions">follow this tutorial</a> or check <a href="https://egghead.io/courses/how-to-contribute-to-an-open-source-project-on-github">this video series</a> to learn about the contribution workflow with GitHub.</p> + +<p>Look tickets labeled <https://github.com/OpenLineage/OpenLineage/labels/good%20first%20issue|'good first issue'> and <https://github.com/OpenLineage/OpenLineage/labels/help%20wanted|'help wanted'>. These are a great starting point if you want to contribute. Don't hesitate to ask questions about the issue if you are not sure about the strategy to follow.</p> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-06-08 14:17:19
 +
+

*Thread Reply:* We’re starting a monthly call, I will publish more details here

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-06-08 14:17:48
 +
+

*Thread Reply:* Do you have a specific use case in mind?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kedar Rajwade + (kedar@cloudzealous.com) +
+
2021-06-08 21:32:02
 +
+

*Thread Reply:* Nothing specific yet

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-06-09 00:49:09
 +
+

The first instance of the OpenLineage Monthly meeting is tomorrow June 9 at 9am PT: https://calendar.google.com/event?action=TEMPLATE&tmeid=MDRubzk0cXAwZzA4bXRmY24yZjBkdTZzbDNfMjAyMTA2MDlUMTYwMDAwWiBqdWxpZW5AZGF0YWtpbi5jb20&tmsrc=julien%40datakin.com&scp=ALL|https://calendar.google.com/event?action=TEMPLATE&tmeid=MDRubzk0cXAwZzA4bXRmY24yZjBkdT[…]qdWxpZW5AZGF0YWtpbi5jb20&tmsrc=julien%40datakin.com&scp=ALL

+
+
accounts.google.com
+ + + +
Google Calendar
+
+ With Google's free online calendar, it’s easy to keep track of life’s important events all in one place. +
+ + + + + + +
Original URL: https://calendar.google.com/event?action=TEMPLATE&amp;tmeid=MDRubzk0cXAwZzA4bXRmY24yZjBkdTZzbDNfMjAyMTA2MDlUMTYwMDAwWiBqdWxpZW5AZGF0YWtpbi5jb20&amp;tmsrc=julien%40datakin.com&amp;scp=ALL
+ + + +
+ + + +
+ 🎉 Willy Lulciuc, Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Victor Shafran + (victor.shafran@databand.ai) +
+
2021-06-09 08:33:45
 +
+

*Thread Reply:* Hey @Julien Le Dem, I can’t add a link to my calendar… Can you send an invite?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Leo + (leorobinovitch@gmail.com) +
+
2021-06-09 11:00:05
 +
+

*Thread Reply:* Same!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-06-09 11:01:45
 +
+

*Thread Reply:* Will do. Also if you send your email in dm you can get added to the invite

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-06-09 11:59:22
 +
+

*Thread Reply:* You can find the invitation on the tsc mailing list: https://lists.lfaidata.foundation/g/openlineage-tsc/topic/invitation_openlineage/83423919?p=,,,20,0,0,0::recentpostdate%2Fsticky,,,20,2,0,83423919

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kedar Rajwade + (kedar@cloudzealous.com) +
+
2021-06-09 12:00:30
 +
+

*Thread Reply:* @Julien Le Dem Can't access the calendar.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kedar Rajwade + (kedar@cloudzealous.com) +
+
2021-06-09 12:00:43
 +
+

*Thread Reply:* Can you please share the meeting details

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-06-09 12:01:12
 +
+

*Thread Reply:*

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-06-09 12:01:24
 +
+

*Thread Reply:*

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2021-06-09 12:01:55
 +
+

*Thread Reply:* The calendar invite says 9am PDT, not 10am. Which is right?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kedar Rajwade + (kedar@cloudzealous.com) +
+
2021-06-09 12:01:58
 +
+

*Thread Reply:* Thanks

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-06-09 13:25:13
 +
+

*Thread Reply:* it is 9am,thanks

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-06-09 18:37:02
 +
+

*Thread Reply:* I have posted the notes on the wiki (includes link to recording) https://wiki.lfaidata.foundation/display/OpenLineage/Monthly+meeting+archive

+ + + +
+ 🙌 Willy Lulciuc, Victor Shafran +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Pedram + (pedram@hightouch.io) +
+
2021-06-10 13:53:18
 +
+

Hi! Are there some 'close-to-real' sample events available to build off and compare to? I'd like to make sure what I'm outputting makes sense but it's hard when only comparing to very synthetic data.

+ + + +
+ 👋 Willy Lulciuc +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2021-06-10 13:55:51
 +
+

*Thread Reply:* We’ve recently worked on a getting started guide for OpenLineage that we’d like to publish on the OpenLineage website. That should help with making things a bit more clear on usage. @Ross Turk / @Julien Le Dem might know of when that might become available. Otherwise, happy to answer any immediate questions you might have about posting/collecting OpenLineage events

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Pedram + (pedram@hightouch.io) +
+
2021-06-10 13:58:58
 +
+

*Thread Reply:* Here's a sample of what I'm producing, would appreciate any feedback if it's on the right track. One of our challenges is that 'dataset' is a little loosely defined for us as outputs since we take data from a warehouse/database and output to things like Salesforce, Airtable, Hubspot and even Slack.

+ +

{ + eventType: 'START', + eventTime: '2021-06-09T08:45:00.395+00:00', + run: { runId: '2821819' }, + job: { + namespace: '<hightouch://my-workspace>', + name: '<hightouch://my-workspace/sync/123>' + }, + inputs: [ + { + namespace: '<snowflake://abc1234>', + name: '<snowflake://abc1234/my_source_table>' + } + ], + outputs: [ + { + namespace: '<salesforce://mysf_instance.salesforce.com>', + name: 'accounts' + } + ], + producer: 'hightouch-event-producer-v.0.0.1' +} +{ + eventType: 'COMPLETE', + eventTime: '2021-06-09T08:45:30.519+00:00', + run: { runId: '2821819' }, + job: { + namespace: '<hightouch://my-workspace>', + name: '<hightouch://my-workspace/sync/123>' + }, + inputs: [ + { + namespace: '<snowflake://abc1234>', + name: '<snowflake://abc1234/my_source_table>' + } + ], + outputs: [ + { + namespace: '<salesforce://mysf_instance.salesforce.com>', + name: 'accounts' + } + ], + producer: 'hightouch-event-producer-v.0.0.1' +}

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Pedram + (pedram@hightouch.io) +
+
2021-06-10 14:02:59
 +
+

*Thread Reply:* One other question I have is really around how customers might take the metadata we emit at Hightouch and integrate that with OpenLineage metadata emitted from other tools like dbt, Airflow, and other integrations to create a true lineage of their data.

+ +

For example, if the data goes from S3 -&gt; Snowflake via Airflow and then from Snowflake -&gt; Salesforce via Hightouch, this would mean both Airflow/Hightouch would need to define the Snowflake dataset in exactly the same way to get the benefits of lineage?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2021-06-17 19:13:14
 +
+

*Thread Reply:* Hey, @Dejan Peretin! Sorry for the late replay here! Your OL events look solid and only have a few of suggestions:

+ +
  1. I would use a valid UUID for the run ID as the spec will standardize on that type, see https://github.com/OpenLineage/OpenLineage/pull/65
  2. You don’t need to provide the input dataset again on the COMPLETE event as the input datasets have already been associated with the run ID
  3. For the producer, I’d recommend using a link to the producer source code version to link the producer version with the OL event that was emitted.
    4. +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2021-06-17 19:13:59
 +
+

*Thread Reply:* You can now reference our OL getting started guide for a close-to-real example 🙂 , see http://openlineage.io/getting-started

+
+
openlineage.io
+ + + +
Getting Started
+
+ Data lineage is the foundation for a new generation of powerful, context-aware data tools and best practices. OpenLineage enables consistent collection of lineage metadata, creating a deeper understanding of how data is produced and used. +
+ + + + + + +
Original URL: http://openlineage.io/getting-started
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2021-06-17 19:18:19
 +
+

*Thread Reply:* > … this would mean both Airflow/Hightouch would need to define the Snowflake dataset in exactly the same way to get the benefits of lineage? +Yes, the dataset and the namespace that it was registered under would have to be the same to properly build the lineage graph. We’re working on defining unique dataset names and have made some good progress in this area. I’d suggest reviewing the OL naming conventions if you haven’t already: https://github.com/OpenLineage/OpenLineage/blob/main/spec/Naming.md

+ + + +
+ 🙌 Pedram +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Pedram + (pedram@hightouch.io) +
+
2021-06-19 01:09:27
 +
+

*Thread Reply:* Thanks! I'm really excited to see what the future holds, I think there are so many great possibilities here. Will be keeping a watchful eye. 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2021-06-22 15:14:39
 +
+

*Thread Reply:* 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Antonio Moctezuma + (antoniomoctezuma@northwesternmutual.com) +
+
2021-06-11 09:53:39
 +
+

Hey everyone! I've been running into a minor OpenLineage issue and I was curious if anyone had any advice. So according to OpenLineage specs its suggested that for a dataset coming from S3 that its namespace be in the form of s3://<bucket>. We have implemented our code to do so and RunEvents are published without issue but when trying to retrieve the information of this RunEvent (like the job) I am unable to retrieve it based on namespace from both /api/v1/namespaces/s3%3A%2F%2F<bucket name> (encoding since : and / are special characters in URL) and the beta endpoint of /api/v1-beta/lineage?nodeId=<dataset>:<namespace>:<name> and instead get a 400 error with a "Ambiguous Segment in URI" message.

+ +

Any and all advice would be super helpful! Thank you so much!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-06-11 10:16:41
 +
+

*Thread Reply:* Sounds like problem is with Marquez - might be worth to open issue here: https://github.com/MarquezProject/marquez/issues

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Antonio Moctezuma + (antoniomoctezuma@northwesternmutual.com) +
+
2021-06-11 10:25:58
 +
+

*Thread Reply:* Thank you! Will do.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-06-11 15:31:41
 +
+

*Thread Reply:* Thanks for reporting Antonio

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-06-16 19:01:52
 +
+

I have opened a proposal for versioning and publishing the spec: https://github.com/OpenLineage/OpenLineage/issues/63

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/issues/63|#63 [PROPOSAL] Define the OpenLineage spec versioning mechanism>
+
+ <p><strong>Purpose:</strong> <br /> +We need to define how the OpenLineage spec gets versioned and published. <br /> +Some requirements:</p> + +<p>• The OpenLineage spec and related libraries are in the OpenLineage repo. +• The OpenLineage spec version should change only when the spec itself changes. +• The libraries in the repo change more frequently than the spec (including when the spec changes). +• We want to version the OpenLineage spec independently of the api spec. +• The mechanism to version and publish the OpenLineage core spec, should apply to publishing custom facets.</p> + +<p><strong>Proposed implementation</strong> <br /> +I have started a google doc to make it easier to collaborate: <br /> +<a href="https://docs.google.com/document/d/1inhmb90SB62VyYf8nkkpjBDPpyr2wxkIXQ_AFIQiF9I/edit">https://docs.google.com/document/d/1inhmb90SB62VyYf8nkkpjBDPpyr2wxkIXQ_AFIQiF9I/edit</a> <br /> +please comment in the doc.</p> +
+ +
+
Labels
+ proposal +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-06-18 15:00:20
 +
+

We have a nice OpenLineage website now. https://openlineage.io/ +Thank you to contributors: @Ross Turk @Willy Lulciuc @Michael Collado!

+
+
openlineage.io
+ + + +
Home
+
+ Data lineage is the foundation for a new generation of powerful, context-aware data tools and best practices. OpenLineage enables consistent collection of lineage metadata, creating a deeper understanding of how data is produced and used. +
+ + + + + + +
Original URL: https://openlineage.io/
+ + + +
+ + + +
+ ❤️ Ross Turk, Kevin Mellott, Leo, Peter Hicks, Willy Lulciuc, Edgar Ramírez Mondragón, Maciej Obuchowski, Supratim Mukherjee +
+ +
+ 👍 Kedar Rajwade, Mukund +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Leo + (leorobinovitch@gmail.com) +
+
2021-06-18 15:09:18
 +
+

*Thread Reply:* Very nice!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Bruno Canal + (bcanal@gmail.com) +
+
2021-06-20 10:08:43
 +
+

Hi everyone! Im trying to run a spark job with openlineage and marquez...But Im getting some errors

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Bruno Canal + (bcanal@gmail.com) +
+
2021-06-20 10:09:28
 +
+

*Thread Reply:* Here is the error...

+ +

21/06/20 11:02:56 WARN ArgumentParser: missing jobs in [, api, v1, namespaces, spark_integration] at 5 +21/06/20 11:02:56 WARN ArgumentParser: missing runs in [, api, v1, namespaces, spark_integration] at 7 +21/06/20 11:03:01 ERROR AsyncEventQueue: Listener SparkListener threw an exception +java.lang.NullPointerException + at marquez.spark.agent.SparkListener.onJobEnd(SparkListener.java:165) + at org.apache.spark.scheduler.SparkListenerBus$class.doPostEvent(SparkListenerBus.scala:39) + at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37) + at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37) + at org.apache.spark.util.ListenerBus$class.postToAll(ListenerBus.scala:91) + at <a href="http://org.apache.spark.scheduler.AsyncEventQueue.org">org.apache.spark.scheduler.AsyncEventQueue.org</a>$apache$spark$scheduler$AsyncEventQueue$$super$postToAll(AsyncEventQueue.scala:92) + at org.apache.spark.scheduler.AsyncEventQueue$$anonfun$org$apache$spark$scheduler$AsyncEventQueue$$dispatch$1.apply$mcJ$sp(AsyncEventQueue.scala:92) + at org.apache.spark.scheduler.AsyncEventQueue$$anonfun$org$apache$spark$scheduler$AsyncEventQueue$$dispatch$1.apply(AsyncEventQueue.scala:87) + at org.apache.spark.scheduler.AsyncEventQueue$$anonfun$org$apache$spark$scheduler$AsyncEventQueue$$dispatch$1.apply(AsyncEventQueue.scala:87) + at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58) + at <a href="http://org.apache.spark.scheduler.AsyncEventQueue.org">org.apache.spark.scheduler.AsyncEventQueue.org</a>$apache$spark$scheduler$AsyncEventQueue$$dispatch(AsyncEventQueue.scala:87) + at org.apache.spark.scheduler.AsyncEventQueue$$anon$1$$anonfun$run$1.apply$mcV$sp(AsyncEventQueue.scala:83) + at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1302) + at org.apache.spark.scheduler.AsyncEventQueue$$anon$1.run(AsyncEventQueue.scala:82)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Bruno Canal + (bcanal@gmail.com) +
+
2021-06-20 10:10:41
 +
+

*Thread Reply:* Here is my code ...

+ +

```from pyspark.sql import SparkSession +from pyspark.sql.functions import lit

+ +

spark = SparkSession.builder \ + .master('local[1]') \ + .config('spark.jars.packages', 'io.github.marquezproject:marquezspark:0.15.2') \ + .config('spark.extraListeners', 'marquez.spark.agent.SparkListener') \ + .config('openlineage.url', 'http://localhost:5000/api/v1/namespaces/spark_integration/') \ + .config('openlineage.namespace', 'sparkintegration') \ + .getOrCreate()

+ +

Supress success

+ +

spark.sparkContext.jsc.hadoopConfiguration().set('mapreduce.fileoutputcommitter.marksuccessfuljobs', 'false') +spark.sparkContext.jsc.hadoopConfiguration().set('parquet.summary.metadata.level', 'NONE')

+ +

dfsourcetrip = spark.read \ + .option('inferSchema', True) \ + .option('header', True) \ + .option('delimiter', '|') \ + .csv('/Users/bcanal/Workspace/poc-marquez/pocspark/resources/data/source/trip.csv') \ + .createOrReplaceTempView('sourcetrip')

+ +

dfdrivers = spark.table('sourcetrip') \ + .select('driver') \ + .distinct() \ + .withColumn('drivername', lit('Bruno')) \ + .withColumnRenamed('driver', 'driverid') \ + .createOrReplaceTempView('source_driver')

+ +

df = spark.sql( + """ + SELECT d., t. + FROM sourcetrip t, sourcedriver d + WHERE t.driver = d.driver_id + """ +)

+ +

df.coalesce(1) \ + .drop('driverid') \ + .write.mode('overwrite') \ + .option('path', '/Users/bcanal/Workspace/poc-marquez/pocspark/resources/data/target') \ + .saveAsTable('trip')```

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Bruno Canal + (bcanal@gmail.com) +
+
2021-06-20 10:12:27
 +
+

*Thread Reply:* After this execution, I can see just the source from first dataframe called dfsourcetrip...

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Bruno Canal + (bcanal@gmail.com) +
+
2021-06-20 10:13:04
 +
+

*Thread Reply:*

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Bruno Canal + (bcanal@gmail.com) +
+
2021-06-20 10:13:45
 +
+

*Thread Reply:* I was expecting to see all source dataframes, target dataframes and the job

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Bruno Canal + (bcanal@gmail.com) +
+
2021-06-20 10:14:35
 +
+

*Thread Reply:* I`m running spark local on my laptop and I followed marquez getting start to up it

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Bruno Canal + (bcanal@gmail.com) +
+
2021-06-20 10:14:44
 +
+

*Thread Reply:* Can anyone help me?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2021-06-22 14:42:03
 +
+

*Thread Reply:* I think there's a race condition that causes the context to be missing when the job finishes too quickly. If I just add +spark.sparkContext.setLogLevel('info') +to the setup code, everything works reliably. Also works if you remove the master('local[1]') - at least when running in a notebook

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
anup agrawal + (anup.agrawal500@gmail.com) +
+
2021-06-22 13:48:34
 +
+

@here Hi everyone,

+ + + +
+ 👋 Willy Lulciuc +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
anup agrawal + (anup.agrawal500@gmail.com) +
+
2021-06-22 13:49:10
 +
+

i need to implement export functionality for my data lineage project.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
anup agrawal + (anup.agrawal500@gmail.com) +
+
2021-06-22 13:50:26
 +
+

as part of this i need to convert the information fetched from graph db (neo4j) to CSV format and send in response.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
anup agrawal + (anup.agrawal500@gmail.com) +
+
2021-06-22 13:51:21
 +
+

can someone please direct me to the CSV format of open lineage data

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2021-06-22 15:26:55
 +
+

*Thread Reply:* Hey, @anup agrawal. This is a great question! The OpenLineage spec is defined using the Json Schema format, and it’s mainly for the transport layer of OL events. In terms of how OL events are eventually stored, that’s determined by the backend consumer of the events. For example, Marquez stores the raw event in a lineage_events table, but that’s mainly for convenience and replayability of events . As for importing / exporting OL events from storage, as long as you can translate the CSV to an OL event, then HTTP backends like Marquez that support OL can consume them

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2021-06-22 15:27:29
 +
+

*Thread Reply:* > as part of this i need to convert the information fetched from graph db (neo4j) to CSV format and send in response. +Depending on the exported CSV, I would translate the CSV to an OL event, see https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.json

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2021-06-22 15:29:58
 +
+

*Thread Reply:* When you say “send in response”, who would be the consumer of the lineage metadata exported for the graph db?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
anup agrawal + (anup.agrawal500@gmail.com) +
+
2021-06-22 23:33:05
 +
+

*Thread Reply:* so far what i understood about my requirement is that. 1. my service will receive OL events

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
anup agrawal + (anup.agrawal500@gmail.com) +
+
2021-06-22 23:33:24
 +
+

*Thread Reply:* 2. store it in graph db (neo4j)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
anup agrawal + (anup.agrawal500@gmail.com) +
+
2021-06-22 23:38:28
 +
+

*Thread Reply:* 3. this lineage information will be displayed on ui, based on the request.

+ +
  1. now my part in that is to implement an Export functionality, so that someone can download it from UI. in UI there will be option to download the report.
  2. so i need to fetch data from storage and convert it into CSV format, send to UI
  3. they can download the report from UI.
    4. +
+ +

SO my question here is that i have never seen how that CSV report look like and how do i achieve that ? +when i had asked my team how should CSV look like they directed me to your website.

+ + + +
+ 👍 Willy Lulciuc +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2021-07-01 19:18:35
 +
+

*Thread Reply:* I see. @Julien Le Dem might have some thoughts on how an OL event would be represented in different formats like CSV (but, of course, there’s also avro, parquet, etc). The Json Schema is the recommended format for importing / exporting lineage metadata. And, for a file, each line would be an OL event. But, given that CSV is a requirement, I’m not sure how that would be structured. Or at least, it’s something we haven’t previously discussed

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
anup agrawal + (anup.agrawal500@gmail.com) +
+
2021-06-22 13:51:51
 +
+

i am very new to this .. sorry for any silly questions

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2021-06-22 20:29:22
 +
+

*Thread Reply:* There are no silly questions! 😉

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Abdulmalik AN + (lord.of.d1@gmail.com) +
+
2021-06-29 11:46:33
 +
+

Hello, I have read every topic and listened to 4 talks and the podcast episode about OpenLineage and Marquez due to my basic understanding for the data engineering field, I have a couple of questions which I did not understand: +1- What are events and facets and what are their purpose? +2- Can I implement the OpenLineage API to any software? or does the software needs to be integrated with the OpenLineage API? +3- Can I say that OpenLineage is about observability and Marquez is about collecting and storing the metadata? +Thank you all for being cooperative.

+ + + +
+ 👍 Stephen Pimentel, Kedar Rajwade +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2021-07-01 19:07:27
 +
+

*Thread Reply:* Welcome, @Abdulmalik AN 👋 Hopefully the talks / podcasts have been informative! And, sure, happy to clarify a few things:

+ +

> What are events and facets and what are their purpose? +An OpenLineage event is used to capture the lineage metadata at a point in time for a given run in execution. That is, the runs state transition, the inputs and outputs consumed/produced and the job associated with the run are part of the event. The metadata defined in the event can then be consumed by an HTTP backend (as well as other transport layers). Marquez is an HTTP backend implementation that consumes OL events via a REST API call. The OL core model only defines the metadata that should be captured in the context of a run, while the processing of the event is up to the backend implementation consuming the event (think consumer / producer model here). For Marquez, the end-to-end lineage metadata is stored for pipelines (composed of multiple jobs) with built-in metadata versioning support. Now, for the second part of your question: the OL core model is highly extensible via facets. A facet is user-defined metadata and enables entity enrichment. I’d recommend checking out the getting started guide for OL 🙂

+ +

> Can I implement the OpenLineage API to any software? or does the software needs to be integrated with the OpenLineage API? +Do you mean HTTP vs other protocols? Currently, OL defines an API spec for HTTP backends, that Marquez has adopted to ingest OL events. But there are also plans to support Kafka and many others.

+ +

> Can I say that OpenLineage is about observability and Marquez is about collecting and storing the metadata? +> Thank you all for being cooperative. +Yep! OL defines the metadata to collect for running jobs / pipelines that can later be used for root cause analysis / troubleshooting failing jobs, while Marquez is a metadata service that implements the OL standard to both consume and store lineage metadata while also exposing a REST API to query dataset, job and run metadata.

+
+ + + + +
<https://github.com/MarquezProject/marquez/blob/main/api/src/main/java/marquez/api/OpenLineageResource.java | OpenLineageResource.java>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/MarquezProject/marquez">MarquezProject/marquez</a> +
+ + +
+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.yml | OpenLineage.yml>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+
+
openlineage.io
+ + + +
Getting Started
+
+ Data lineage is the foundation for a new generation of powerful, context-aware data tools and best practices. OpenLineage enables consistent collection of lineage metadata, creating a deeper understanding of how data is produced and used. +
+ + + + + + +
Original URL: https://openlineage.io/getting-started/
+ + + +
+ + + +
+ 👍 Kedar Rajwade +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Nic Colley + (nic.colley@alation.com) +
+
2021-06-30 17:46:52
 +
+

Hi OpenLineage team! Has anyone got this working on databricks yet? I’ve been working on this for a few days and can’t get it to register lineage. I’ve attached my notebook in this thread.

+ +

silly question - does the jar file need be on the cluster? +Which versions of spark does OpenLineage support?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Nic Colley + (nic.colley@alation.com) +
+
2021-06-30 18:16:58
 +
+

*Thread Reply:* I based my code on this previous post https://openlineage.slack.com/archives/C01CK9T7HKR/p1624198123045800

+
+ + +
+ + + } + + Bruno Canal + (https://openlineage.slack.com/team/U025LV2BJUB) +
+ + + +
+
+ <p>Hi everyone! I<code>m trying to run a spark job with openlineage and marquez...But I</code>m getting some errors</p> +
+ + + + + + +
Original URL: https://openlineage.slack.com/archives/C01CK9T7HKR/p1624198123045800
+ + +
+ + Thread in Slack Conversation +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Nic Colley + (nic.colley@alation.com) +
+
2021-06-30 18:36:59
 +
+

*Thread Reply:*

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2021-07-01 13:45:42
 +
+

*Thread Reply:* In your first cell, you have +from pyspark.sql import SparkSession +from pyspark.sql.functions import lit +spark.sparkContext.setLogLevel('info') +unfortunately, the reference to sparkContext in the third line forces the initialization of the SparkContext so that in the next cell, your new configuration is ignored. In pyspark, you must initialize your SparkSession before any references to the SparkContext. It works if you remove the setLogInfo call from the first cell and make your 2nd cell +spark = SparkSession.builder \ + .config('spark.jars.packages', 'io.github.marquezproject:marquez_spark:0.15.2') \ + .config('spark.extraListeners', 'marquez.spark.agent.SparkListener') \ + .config('openlineage.url', '<https://domain.com>') \ + .config('openlineage.namespace', 'my-namespace') \ + .getOrCreate() +spark.sparkContext.setLogLevel('info')

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Samia Rahman + (srahman@thoughtworks.com) +
+
2021-06-30 19:26:42
 +
+

How would one capture lineage for job that's processing streaming data? Is that in scope for OpenLineage?

+ + + +
+ ➕ Josh Quintus, Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2021-07-01 16:32:18
 +
+

*Thread Reply:* It’s absolutely in scope! We’ve primarily focused on the batch use case (ETL jobs, etc), but the OpenLineage standard supports both batch and streaming jobs. You can check out our roadmap here, where you’ll find Flink and Beam on our list of future integrations.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2021-07-01 16:32:57
 +
+

*Thread Reply:* Is there a streaming framework you’d like to see added to our roadmap?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
mohamed chorfa + (chorfa672@gmail.com) +
+
2021-06-30 20:33:25
 +
+

👋 Hello everyone!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2021-07-01 16:24:16
 +
+

*Thread Reply:* Welcome, @mohamed chorfa 👋 . Let’s us know if you have any questions!

+ + + +
+ 👍 mohamed chorfa +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
mohamed chorfa + (chorfa672@gmail.com) +
+
2021-07-03 19:37:58
 +
+

*Thread Reply:* Really looking follow the evolution of the specification from RawData to the ML-Model

+ + + +
+ ❤️ Julien Le Dem, Willy Lulciuc +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-07-02 16:53:01
 +
+

Hello OpenLineage community, +We have been working on fleshing out the OpenLineage roadmap. +See on github on the currently prioritized effort: https://github.com/OpenLineage/OpenLineage/projects +Please add your feedback to the roadmap by either commenting on the github issues or opening new issues.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-07-02 17:04:13
 +
+

In particular, I have opened an issue to finalize our mission statement: https://github.com/OpenLineage/OpenLineage/issues/84

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/issues/84|#84 Finalize the OpenLineage mission statement>
+
+ <p>It is part of our technical charter. <br /> +Currently we have: "to enable the industry at-large to collect lineage metadata consistently and comprehensively across complex pipelines, creating a deeper understanding of data."</p> + +<p>We'd want to capture that this is specifically operational lineage, or lineage and metadata collected as the job is running.</p> + +<p>It is also something more widely applicable than just data pipelines, and should reflect that,</p> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+ ❤️ Ross Turk, Maciej Obuchowski, Peter Hicks +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-07-07 19:53:17
 +
+

*Thread Reply:* Based on community feedback, +The new proposed mission statement: “to enable the industry at-large to collect real-time lineage metadata consistently across complex ecosystems, creating a deeper understanding of how data is produced and used”

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-07-07 20:23:24
 +
+

I have updated the proposal for the spec versioning: https://github.com/OpenLineage/OpenLineage/issues/63

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/issues/63|#63 [PROPOSAL] Define the OpenLineage spec versioning mechanism>
+
+ <p><strong>Purpose:</strong> <br /> +We need to define how the OpenLineage spec gets versioned and published. <br /> +Some requirements:</p> + +<p>• The OpenLineage spec and related libraries are in the OpenLineage repo. +• The OpenLineage spec version should change only when the spec itself changes. +• The libraries in the repo change more frequently than the spec (including when the spec changes). +• We want to version the OpenLineage spec independently of the api spec. +• The mechanism to version and publish the OpenLineage core spec, should apply to publishing custom facets.</p> + +<p><strong>Proposed implementation</strong></p> + +<p>• The spec defines it’s current version using the “$id” field: <br /> + See: <br /> + <a href="https://json-schema.org/draft/2020-12/json-schema-core.html#rfc.section.8.2.1">json schema core doc</a> <br /> + <a href="https://json-schema.org/draft/2019-09/schema">Json schema spec $id</a> <a href="https://github.com/json-schema-org/json-schema-spec/blob/draft-next/meta/core.json">also on github</a> <br /> + Example: <br /> + "$id": "<a href="https://openlineage.io/spec/0.1.0/OpenLineage.json">https://openlineage.io/spec/0.1.0/OpenLineage.json</a>" <br /> + The URL in $id is resolvable and returns that version of the spec. <br /> + We use github pages to publish the spec to <a href="http://openlineage.io">openlineage.io</a> +• The $id urls uses a SEMVER compliant version, following the <a href="https://docs.snowplowanalytics.com/docs/pipeline-components-and-applications/iglu/common-architecture/schemaver/">SCHEMAVER semantics</a> <br /> + MODEL-REVISION-ADDITION + • MODEL when you make a breaking schema change which will prevent interaction with any historical data + • REVISION when you introduce a schema change which may prevent interaction with some historical data + • ADDITION when you make a schema change that is compatible with all historical data +• CI verification that: + • the version changes when the spec changed: <br /> + When resolving “$id”, the build fails if the spec is not exactly the same. + • The version does not change when the spec does not change. <br /> + TODO: define a mechanism to verify that the current version of the spec is not already published with a different. Possibly hash based. + • Libraries are generating event with current version + • Make sure the spec is backward compatible (only add optional fields) and consistent with the versioning semantics +• git pre commit: <br /> + Increments the versions automatically when the spec changes. +• CI publish (on tag): <br /> + The spec gets published automatically to the $id URL on <a href="http://OpenLineage.io">OpenLineage.io</a> <br /> + Check the $id field has the right domain prefix <br /> + Publish current OpenLineage.json to that URL</p> + +<p>The discussion is available on a google doc: <br /> +<a href="https://docs.google.com/document/d/1inhmb90SB62VyYf8nkkpjBDPpyr2wxkIXQ_AFIQiF9I/edit">https://docs.google.com/document/d/1inhmb90SB62VyYf8nkkpjBDPpyr2wxkIXQ_AFIQiF9I/edit</a></p> +
+ +
+
Assignees
+ julienledem +
+ +
+
Labels
+ proposal +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+ 🙌 Willy Lulciuc +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jorik + (jorik.blaas-sigmond@nn.nl) +
+
2021-07-08 07:06:53
 +
+

Hi all. I'm trying to get my bearings on openlineage. Love the concept. In our data transformation pipelines, output datasets are explicitly versioned (we have an incrementing snapshot id). Our storage layer (deltalake) allows us to also ingest 'older' versions of the same dataset, etc. If I understand it correctly I would have to add some inputFacets and outputFacets to run to store the actual version being referenced. Is that something that is currently available, or on the roadmap, or is it something I could extend myself?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-07-08 18:57:44
 +
+

*Thread Reply:* It is on the roadmap and there’s a ticket open but nobody is working on it at the moment. You are very welcome to contribute a spec and implementation

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-07-08 18:59:00
 +
+

*Thread Reply:* Please comment here and feel free to make a proposal: https://github.com/OpenLineage/OpenLineage/issues/35

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/issues/35|#35 [PROPOSAL] Dataset version facet>
+
+ <p><strong>Purpose:</strong> <br /> +Many databases have explicit versioning in place for datasets. It is highly desirable to track the specific version of the dataset if it is known at consumption/generation time. Rather than relying on the service to correctly tie job inputs to specific versions of a dataset, the client should be allowed to report that information if it is available.</p> + +<p><strong>Proposed implementation</strong> <br /> +A dataset version facet may be sufficient to report dataset version information, if it exists. Custom facets can be used to include any database specific version information that may exist, while the minimum required fields should include a versionName string field.</p> +
+ +
+
Labels
+ proposal +
+ +
+
Comments
+ 2 +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jorik + (jorik.blaas-sigmond@nn.nl) +
+
2021-07-08 07:07:29
 +
+

TL;DR: our database supports time-travel, and runs can be set up to use a specific point-in-time of an input. How do we make sure to keep that information within openlineage

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mariusz Górski + (gorskimariusz13@gmail.com) +
+
2021-07-09 02:23:29
 +
+

Hi, on a subject of spark integrations - I know that there is spark-marquez but was curious did you also consider https://github.com/AbsaOSS/spline-spark-agent ? It seems like this and spark-marquez are doing similar thing and maybe it would make sense to add openlineage support to spline spark agent?

+
+ + + + +
AbsaOSS/spline-spark-agent
+
+ Spline agent for Apache Spark +
+ +
+
Website
+ <https://absaoss.github.io/spline/> +
+ +
+
Stars
+ 36 +
+ + + + + + + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mariusz Górski + (gorskimariusz13@gmail.com) +
+
2021-07-09 02:23:42
 +
+

*Thread Reply:* cc @Julien Le Dem @Maciej Obuchowski

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-07-09 04:28:38
 +
+

*Thread Reply:* @Michael Collado

+ + + +
+ 👀 Michael Collado +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-07-12 21:17:12
 +
+

The OpenLineage Technical Steering Committee meetings are Monthly on the Second Wednesday 9:00am to 10:00am US Pacific and the link to join the meeting is https://us02web.zoom.us/j/81831865546?pwd=RTladlNpc0FTTDlFcWRkM2JyazM4Zz09 +The next meeting is this Wednesday +All are welcome. +•  Agenda: + ◦ Finalize the OpenLineage Mission Statement + ◦ Review OpenLineage 0.1 scope + ◦ Roadmap + ◦ Open discussion  + ◦ Slides: https://docs.google.com/presentation/d/1fD_TBUykuAbOqm51Idn7GeGqDnuhSd7f/edit#slide=id.ge4b57c6942_0_46 +notes are posted here: https://wiki.lfaidata.foundation/display/OpenLineage/Monthly+TSC+meeting.,.,_

+ + + +
+ 🙌 Willy Lulciuc, Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-07-12 21:18:04
 +
+

*Thread Reply:* Feel free to share your email with me if you want to be added to the gcal invite

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-07-14 12:03:31
 +
+

*Thread Reply:* It is starting now

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jiří Sedláček + (yirie.sedlahczech@gmail.com) +
+
2021-07-13 08:22:40
 +
+

Hello, is it possible to track lineage on column level? For example for SQL like this: +CREATE TABLE T2 AS SELECT c1,c2 FROM T1; +I would like to record this lineage: +T1.C1 -- job1 --&gt; T2.C1 +T1.C2 -- job1 --&gt; T2.C2 +Would that be possible to record in OL format?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jiří Sedláček + (yirie.sedlahczech@gmail.com) +
+
2021-07-13 08:29:52
 +
+

(the important thing for me is to be able to tell that T1.C1 has no effect on T2.C2)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-07-14 17:00:12
 +
+

I have updated the notes and added the link to the recording of the meeting this morning: https://wiki.lfaidata.foundation/display/OpenLineage/Monthly+TSC+meeting

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-07-14 17:04:18
 +
+

*Thread Reply:* In particular, please review the versioning proposal: https://github.com/OpenLineage/OpenLineage/issues/63

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/issues/63|#63 [PROPOSAL] Define the OpenLineage spec versioning mechanism>
+
+ <p><strong>Purpose:</strong> <br /> +We need to define how the OpenLineage spec gets versioned and published. <br /> +Some requirements:</p> + +<p>• The OpenLineage spec and related libraries are in the OpenLineage repo. +• The OpenLineage spec version should change only when the spec itself changes. +• The libraries in the repo change more frequently than the spec (including when the spec changes). +• We want to version the OpenLineage spec independently of the api spec. +• The mechanism to version and publish the OpenLineage core spec, should apply to publishing custom facets.</p> + +<p><strong>Proposed implementation</strong></p> + +<p>• The spec defines it’s current version using the <code>“$id”</code> field: + • See: + • <a href="https://json-schema.org/draft/2020-12/json-schema-core.html#rfc.section.8.2.1">json schema core doc</a> + • <a href="https://json-schema.org/draft/2019-09/schema">Json schema spec $id</a> + • <a href="https://github.com/json-schema-org/json-schema-spec/blob/draft-next/meta/core.json">also on github</a> + • Example: <br /> + <code>"$id": "&lt;https://openlineage.io/spec/0.1.0/OpenLineage.json&gt;"</code> <br /> + The URL in $id is resolvable and returns that version of the spec. <br /> + We use github pages to publish the spec to <a href="http://openlineage.io">openlineage.io</a> +• The $id urls uses a SEMVER compliant version, following the <a href="https://docs.snowplowanalytics.com/docs/pipeline-components-and-applications/iglu/common-architecture/schemaver/">SCHEMAVER semantics</a> <br /> + MODEL-REVISION-ADDITION + • MODEL when you make a breaking schema change which will prevent interaction with any historical data + • REVISION when you introduce a schema change which may prevent interaction with some historical data + • ADDITION when you make a schema change that is compatible with all historical data +• The spec in /spec/OpenLineage.json might be using a prerelease version (ex: 0.1.0-rc0) +• CI verifies that: + • the version changes when the spec changed: <br /> + When resolving “$id”, the build fails if the spec is not exactly the same. + • The version does not change when the spec does not change. <br /> + We can verify that the current version of the spec is not already published with a different version. + • Libraries are generating event with current version + • Make sure the spec is backward compatible (only add optional fields) and consistent with the versioning semantics +• git pre commit: <br /> + Increments the versions automatically when the spec changes. +• spec publication: + • update the $id to the correct value + • tag with spec-{version} + • The spec is published automatically to the $id URL on <a href="http://OpenLineage.io">OpenLineage.io</a> + • Validation that the $id field has the right domain prefix</p> + +<p>The discussion is available on a google doc: <br /> +<a href="https://docs.google.com/document/d/1inhmb90SB62VyYf8nkkpjBDPpyr2wxkIXQ_AFIQiF9I/edit">https://docs.google.com/document/d/1inhmb90SB62VyYf8nkkpjBDPpyr2wxkIXQ_AFIQiF9I/edit</a></p> +
+ +
+
Assignees
+ julienledem +
+ +
+
Labels
+ proposal +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-07-14 17:04:33
 +
+

*Thread Reply:* and the mission statement: https://github.com/OpenLineage/OpenLineage/issues/84

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/issues/84|#84 Finalize the OpenLineage mission statement>
+
+ <p>The mission statement is part of our technical charter. <br /> +We started from:</p> + +<p>&gt; "The mission of the Project is to enable the industry at-large to collect lineage metadata consistently and comprehensively across complex pipelines, creating a deeper understanding of data."</p> + +<p>We'd want to capture that this is specifically operational lineage, or lineage and metadata collected as the job is running. It is also something more widely applicable than just data pipelines, and should reflect that.</p> + +<p>The current proposal is:</p> + +<p>&gt; "The mission of the Project is to enable the industry at-large to collect real-time lineage metadata consistently across complex ecosystems, creating a deeper understanding of how data is produced and used"</p> +
+ +
+
Comments
+ 2 +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-07-14 17:05:02
 +
+

*Thread Reply:* for this one, please give explicit approval in the ticket

+ + + +
+ 👍 Willy Lulciuc +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-07-14 21:10:42
 +
+

*Thread Reply:* @Zhamak Dehghani @Daniel Henneberger @Drew Banin @James Campbell @Ryan Blue @Maciej Obuchowski @Willy Lulciuc ^

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-07-27 18:58:35
 +
+

*Thread Reply:* Per the votes in the github ticket, I have finalized the charter here: https://docs.google.com/document/d/11xo2cPtuYHmqRLnR-vt9ln4GToe0y60H/edit

+ + + +
+ 🙌 Willy Lulciuc +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jarek Potiuk + (jarek@potiuk.com) +
+
2021-07-16 01:25:56
 +
+

Hi Everyone. I am PMC member and committer of Apache Airflow. Watched the talk at the summit https://airflowsummit.org/sessions/2021/data-lineage-with-apache-airflow-using-openlineage/ and thought I might help (after the Summit is gone 🙂 with making OpenLineage/Marquez more seemlesly integrated in Airflow

+
+
airflowsummit.org
+ + + +
Data Lineage with Apache Airflow using OpenLineage
+
+ If you manage a lot of data, and you’re attending this summit, you likely rely on Apache Airflow to do a lot of the heavy lifting. Like any powerful tool, Apache Airflow allows you to accomplish what you couldn’t before… but also creates new challenges. As DAGs pile up, complexity layers on top of complexity and it becomes hard to grasp how a failed or delayed DAG will affect everything downstream. +
+ + + + + + + + +
Original URL: https://airflowsummit.org/sessions/2021/data-lineage-with-apache-airflow-using-openlineage/
+ + + +
+ + + +
+ ❤️ Abe Gong, WingCode, Maciej Obuchowski, Ross Turk, Julien Le Dem, Michael Collado, Samia Rahman, mohamed chorfa +
+ +
+ 🙌 Maciej Obuchowski +
+ +
+ 👍 Jorik +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Samia Rahman + (srahman@thoughtworks.com) +
+
2021-07-20 16:38:38
 +
+

*Thread Reply:* The demo in this does not really use the openlineage spec does it?

+ +

Did I miss something - the API that was should for lineage was that of Marquez, how does Marquest use the open lineage spec?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Samia Rahman + (srahman@thoughtworks.com) +
+
2021-07-20 18:09:01
 +
+

*Thread Reply:* I have a question about the SQLJobFacet in the job schema - isn't it better to call it the TransformationJob Facet or the ProjecessJobFacet such that any logic in the appropriate language and be described? Am I misinterpreting the intention of SQLJobFacet is to capture the logic that runs for a job?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2021-07-26 19:06:43
 +
+

*Thread Reply:* > The demo in this does not really use the openlineage spec does it? +@Samia Rahman In our Airflow talk, the demo used the marquez-airflow lib that sends OpenLineage events to Marquez’s . You can check out the how does Airflow works with OpenLineage + Marquez here https://openlineage.io/integration/apache-airflow/

+
+
openlineage.io
+ + + +
Apache Airflow
+
+ Airflow, a common tool used for workflow management, is the foundation of many data pipelines. Enabling OpenLineage in Apache Airflow automatically tracks metadata about jobs and datasets as DAGs execute. +
+ + + + + + +
Original URL: https://openlineage.io/integration/apache-airflow/
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2021-07-26 19:07:51
 +
+

*Thread Reply:* > Did I miss something - the API that was should for lineage was that of Marquez, how does Marquest use the open lineage spec? +Yes, Marquez ingests OpenLineage events that confirm to the spec via the . Hope this helps!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kenton (swiple.io) + (kknoxparton@gmail.com) +
+
2021-07-21 07:52:32
 +
+

Hi all, does OpenLineage intend on creating lineage off of query logs?

+ +

From what I have read, there are a number of supported integrations but none that cater to regular SQL based ETL. Is this on the OpenLineage roadmap?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2021-07-26 18:54:46
 +
+

*Thread Reply:* I would say this is more of an ingestion pattern, then something the OpenLineage spec would support directly. Though I completely agree, query logs are a great source of lineage metadata with minimal effort. On our roadmap, we have Kafka as a supported backend which would enable streaming lineage metadata from query logs into a topic. That said, confluent has some great blog posts on Change Data Capture: +• https://www.confluent.io/blog/no-more-silos-how-to-integrate-your-databases-with-apache-kafka-and-cdc/ +• https://www.confluent.io/blog/simplest-useful-kafka-connect-data-pipeline-world-thereabouts-part-1/

+
+
Confluent
+ + + +
No More Silos: How to Integrate your Databases with Apache Kafka and CDC | Confluent
+
+ This article shows you how Apache Kafka integrates with existing RDBMS seamlessly - and discusses Change Data Capture (CDC) options. +
+ + + + + + + + +
Original URL: https://www.confluent.io/blog/no-more-silos-how-to-integrate-your-databases-with-apache-kafka-and-cdc/
+ + + +
+
+
Confluent
+ + + +
The Simplest Useful Kafka Connect Data Pipeline in the World – Part 1
+
+ Learn how to stream data from a database (MySQL) into Apache Kafka and from Kafka into a text file with the Kafka Connect API. +
+ + + + + + + + +
Original URL: https://www.confluent.io/blog/simplest-useful-kafka-connect-data-pipeline-world-thereabouts-part-1/
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2021-07-26 18:57:59
 +
+

*Thread Reply:* Q: @Kenton (swiple.io) Are you planning on using Kafka connect? If so, I see 2 reasonable options:

+ +
  1. Stream query logs to a topic using the JDBC source connector, then have a consumer read the query logs off the topic, parse the logs, then stream the result of the query parsing to another topic as an OpenLineage event
  2. Add direct support for OpenLineage to the JDBC connector or any other application you planned to use to read the query logs.
    3. +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2021-07-26 19:01:31
 +
+

*Thread Reply:* Either way, I think this is a great question and a common ingestion pattern we should document or have best practices for. Also, more details on how you plan to ingestion the query logs would be help drive the discussion.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kenton (swiple.io) + (kknoxparton@gmail.com) +
+
2021-08-05 12:01:55
 +
+

*Thread Reply:* Using something like sqlflow could be a good starting point? Demo https://sqlflow.gudusoft.com/?utm_source=gspsite&utm_medium=blog&utm_campaign=support_article#/

+
+
sqlflow.gudusoft.com
+ + + +
SQLFlow: Visualize column impact and data lineage to track columns across transformations by analyzing SQL query.
+
+ SQLFlow: Visualize column impact and data lineage to track columns across transformations by analyzing SQL query. supported databases: bigquery, couchbase, dax, db2, greenplum, hana, hive, impala, informix, mdx, mysql, netezza, openedge, oracle, postgresql, redshift, snowflake, sqlserver, sybase, teradata, vertica +
+ + + + + + +
Original URL: https://sqlflow.gudusoft.com/?utm_source=gspsite&amp;utm_medium=blog&amp;utm_campaign=support_article#/
+ + + +
+
+ + + + +
sqlparser/sqlflow_public
+
+ Document, sample code and other materials for SQLFlow +
+ +
+
Stars
+ 79 +
+ +
+
Language
+ Python +
+ + + + + + + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2021-09-21 20:22:26
 +
+

*Thread Reply:* @Kenton (swiple.io) I haven’t heard of sqlflow but it does look promising. It’s not on our current roadmap, but I think there is a need to have support for parsing query logs as OpenLineage events. Do you mind opening an issue and outlining you thoughts? It’d be great to start the discussion if you’d like to drive this feature and help prioritize this 💯

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Samia Rahman + (srahman@thoughtworks.com) +
+
2021-07-21 08:49:23
 +
+

The openlineage implementation for airflow and spark code integration currently lives in Marquez repo, my understanding from the open lineage scope is that the the integration implementation is the scope of open lineage, are the spark code migrations going to be moved to open lineage?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2021-07-21 11:35:12
 +
+

@Samia Rahman Yes, that is the plan. For details you can see https://github.com/OpenLineage/OpenLineage/issues/73

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/issues/73|#73 Move Marquez integrations to OpenLineage>
+
+ <p>The <a href="https://github.com/MarquezProject/marquez">Marquez</a> community is excited to announce that the following integrations will be moved to the wider OpenLineage ecosystem:</p> + +<p>• <a href="https://github.com/MarquezProject/marquez/tree/main/integrations/airflow"><code>marquez-airflow</code></a>: A library that integrates Airflow DAGs to emit OpenLineage metadata. +• <a href="https://github.com/MarquezProject/marquez/tree/main/integrations/spark"><code>marquez-spark</code></a>: A spark agent using jvm instrumentation to emit OpenLineage metadata. +• <a href="https://github.com/MarquezProject/marquez/tree/main/integrations/common"><code>marquez-integration-common</code></a>: Shared code across integrations. +• <code>marquez-dbt-snowflake</code>: A library that integrates with DBT to emit OpenLineage metadata for Snowflake. +• <code>marquez-dbt-bigquery</code>: A library that integrates with DBT to emit OpenLineage metadata for BigQuery.</p> + +<p>&gt; Note: You can view the integrations in the Marquez repo under <a href="https://github.com/MarquezProject/marquez/tree/main/integrations"><code>marquez/integrations/</code></a>. Marquez is an official reference implementation of the OpenLineage API and all the integrations adopt the OpenLineage standard.</p> + +<p><strong>Why are the integrations being moved to OpenLineage?</strong></p> + +<p>The OpenLineage standard was influenced by Marquez's underlying data model (<em>datasets</em>, <em>jobs</em>, <em>runs</em>) while also drawing inspiration from the projects lineage ingestion API. With OpenLineage now a standalone project under <a href="https://lfaidata.foundation">LFAI &amp; Data Foundation</a>, we've decided that now would be the right time to move the integrations over as more and more integrations are being added to the <a href="https://github.com/OpenLineage/OpenLineage/projects/1">projects roadmap</a> (<em>Looker</em>, <em>Flink</em>, <em>Beam</em>, etc). This would also allow for the OpenLineage community to grow and learn from existing integration implementations, prioritize bug fixes, propose new integrations and changes to the spec, etc.</p> + +<p><strong>What is the timeline for moving the integrations to OpenLineage?</strong></p> + +<p>Below we outline a high-level timeline of moving the integrations to OpenLineage. We encourage everyone to provide feedback and express any concerns you may have by commenting in this issue:</p> + +<p>☑︎ Release Marquez <a href="https://github.com/MarquezProject/marquez/projects/12"><code>0.16.0</code></a> +☐ Move the following integrations from Marquez to OpenLineage: + ☐ <code>marquez-airflow</code> #93 + ☐ <code>marquez-spark</code> + ☑︎ <code>marquez-integration-common</code> #74 + ☐ <code>marquez-dbt-snowflake</code> + ☐ <code>marquez-dbt-bigquery</code> +☐ Deprecate all integrations in Marquez, then link to the new location in the OpenLineage repo</p> + +<p>Then, going forward, new integrations and improvements will be made by open pull requests against the OpenLineage repo. <strong>Note that during this migration phase, existing Marquez specific functionality and environment variables will be maintained to ensure backwards compatibility. The main difference is that integration packages will get a new name: <code>openlineage-**</code> instead of <code>marquez-**</code></strong>.</p> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+ 🙌 Samia Rahman, Willy Lulciuc +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Samia Rahman + (srahman@thoughtworks.com) +
+
2021-07-21 18:13:11
 +
+

I have a question about the SQLJobFacet in the job schema - isn't it better to call it the TransformationJob Facet or the ProjecessJobFacet such that any logic in the appropriate language and be described, can be scala or python code that runs in the job facet and processing streaming or batch data? Am I misinterpreting the intention of SQLJobFacet is to capture the logic that runs for a job?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2021-07-21 18:22:01
 +
+

*Thread Reply:* Hey, @Samia Rahman 👋. Yeah, great question! The SQLJobFacet is used only for SQL-based jobs. That is, it’s not intended to capture the code being executed, but rather the just the SQL if it’s present. The SQL fact can be used later for display purposes. For example, in Marquez, we use the SQLJobFacet to display the SQL executed by a given job to the user via the UI.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2021-07-21 18:23:03
 +
+

*Thread Reply:* To capture the logic of the job (meaning, the code being executed), the OpenLineage spec defines the SourceCodeLocationJobFacet that builds the link to source in version control

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.json | OpenLineage.json>
+
+ <pre><code> "SourceCodeLocationJobFacet": { +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-07-22 17:56:41
 +
+

The process started a few months back when the LF AI & Data voted to accept OpenLineage as part of the foundation. It is now official, OpenLineage joined the LFAI & data Foundation. + https://lfaidata.foundation/blog/2021/07/22/openlineage-joins-lf-ai-data-as-new-sandbox-project/

+
+
LF AI
+ + + +
OpenLineage Joins LF AI &amp; Data as New Sandbox Project - LF AI
+
+ LF AI &amp; Data Foundation—the organization building an ecosystem to sustain open source innovation in artificial intelligence (AI) and data open source projects, today is announcing OpenLineage as its latest... +
+ +
+
Written by
+ Jacqueline Z Cardoso +
+ +
+
Est. reading time
+ 3 minutes +
+ + + + + + + + +
Original URL: https://lfaidata.foundation/blog/2021/07/22/openlineage-joins-lf-ai-data-as-new-sandbox-project/
+ + + +
+ + + +
+ 🙌 Ross Turk, Luke Smith, Maciej Obuchowski, Gyan Kapur, Dr Daniel Smith, Jarek Potiuk, Peter Hicks, Kedar Rajwade, Abe Gong, Damian Warszawski, Willy Lulciuc +
+ +
+ ❤️ Ross Turk, Jarek Potiuk, Peter Hicks, Abe Gong, Willy Lulciuc +
+ +
+ 🎉 Laurent Paris, Rifa Achrinza, Minkyu Park, Peter Hicks, mohamed chorfa, Jarek Potiuk, Abe Gong, Damian Warszawski, Willy Lulciuc, James Le +
+ +
+ 👏 Matt Turck +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Namron + (ian.norman@avanade.com) +
+
2021-07-29 11:20:17
 +
+

Hi, I am trying to create lineage between two datasets. Following the Spec, I can see the syntax for declaring the input and output datasets, and for all creating the associated Job (which I take to be the process in the middle joining the two datasets together). What I can't see is where in the specification to relate the job to the inputs and outputs. Do you have an example of this?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2021-07-30 17:24:44
 +
+

*Thread Reply:* The run event is always tied to exactly one job. It's up to the backend to store the relationship between the job and its inputs/outputs. E.g., in marquez, this is where we associate the input datasets with the job- https://github.com/MarquezProject/marquez/blob/main/api/src/main/java/marquez/db/OpenLineageDao.java#L132-L143

+
+ + + + +
<https://github.com/MarquezProject/marquez/blob/main/api/src/main/java/marquez/db/OpenLineageDao.java | OpenLineageDao.java>
+
+ <pre><code> JobRow job = + jobDao.upsertJob( + UUID.randomUUID(), + getJobType(event.getJob()), + now, + namespace.getUuid(), + namespace.getName(), + event.getJob().getName(), + description, + jobContext.getUuid(), + location, + jobDao.toJson(toDatasetId(event.getInputs()), mapper)); +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/MarquezProject/marquez">MarquezProject/marquez</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-08-03 15:06:58
 +
+

the OuputStatistics facet PR is updated based on your comments @Michael Collado https://github.com/OpenLineage/OpenLineage/pull/114

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/pull/114|#114 add OutputStatistics facet>
+
+ <p>Signed-off-by: Julien Le Dem <a href="mailto:julien@apache.org">julien@apache.org</a></p> +
+ +
+
Comments
+ 1 +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+ 🙌 Michael Collado +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2021-08-03 15:11:56
 +
+

*Thread Reply:* /|~~~ + ///| + /////| + ///////| + /////////| + \==========|===/ +~~~~~~~~~~~~~~~~~~~~~

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-08-03 19:59:03
 +
+

*Thread Reply:* ⛵

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-08-03 19:59:38
 +
+

I have updated the DataQuality metrics proposal and the corresponding PR: https://github.com/OpenLineage/OpenLineage/issues/101 +https://github.com/OpenLineage/OpenLineage/pull/115

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/issues/101|#101 Add the DataQualityMetrics Dataset facet to the core spec>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/pull/115|#115 Data quality facet>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+ 🙌 Willy Lulciuc, Bruno González +
+ +
+ 💯 Willy Lulciuc, Dominique Tipton +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Oleksandr Dvornik + (oleksandr.dvornik@getindata.com) +
+
2021-08-04 10:42:48
 +
+

Guys, I've merged circleCI publish snapshot PR

+ +

Snapshots can be found bellow: +https://datakin.jfrog.io/artifactory/maven-public-libs-snapshot-local/io/openlineage/openlineage-java/0.0.1-SNAPSHOT/ +openlineage-java-0.0.1-20210804.142910-6.jar +https://datakin.jfrog.io/artifactory/maven-public-libs-snapshot-local/io/openlineage/openlineage-spark/0.1.0-SNAPSHOT/ +openlineage-spark-0.1.0-20210804.143452-5.jar

+ +

Build on main passed (edited)

+ +
+
+ + + + + + +
+ + +
+ 🎉 Julien Le Dem +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-08-04 23:08:08
 +
+

I added a mechanism to enforce spec versioning per: https://github.com/OpenLineage/OpenLineage/issues/63 +https://github.com/OpenLineage/OpenLineage/pull/140

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/issues/63|#63 [PROPOSAL] Define the OpenLineage spec versioning mechanism>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/pull/140|#140 Add mechanism to enforce spec versioning>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ben Teeuwen-Schuiringa + (ben.teeuwen@booking.com) +
+
2021-08-05 10:02:49
 +
+

Hi all, at Booking.com we’re using Spline to extract granular lineage information from spark jobs to be able to trace lineage on column-level and the operations in between. We wrote a custom python parser to create graph-like structure that is sent into arangodb. But tbh, the process is far from stable and is not able to quickly answer questions like ‘which root input columns are used to construct column x’.

+ +

My impression with openlineage thus far is it’s focusing on less granular, table input-output information. Is anyone here trying to accomplish something similar on a column-level?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Luke Smith + (luke.smith@kinandcarta.com) +
+
2021-08-05 12:56:48
 +
+

*Thread Reply:* Also interested in use case / implementation differences between Spline and OL. Watching this thread.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-08-05 14:46:44
 +
+

*Thread Reply:* It would be great to have the option to produce the spline lineage info as OpenLineage. +To capture the column level lineage, you would want to add a ColumnLineage facet to the Output dataset facets. +Which is something that is needed in the spec. +Here is a proposal, please chime in: https://github.com/OpenLineage/OpenLineage/issues/148 +Is this something you would be interested to do?

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/issues/148|#148 [PROPOSAL] column level lineage facet>
+
+ <p><strong>Purpose:</strong> <br /> +For transformations like SQL queries (but also in other cases), we can extract column level lineage. This allows answering questions like ‘which root input columns are used to construct column x?’. We need to be able to capture this information in the OpenLineage model</p> + +<p><strong>Proposed implementation</strong> <br /> +We propose to introduce a new dataset facet: <br /> +example:</p> + +<pre><code>{ + "eventType": "COMPLETE", + "eventTime": "2020-12-28T20:52:00.001+10:00", + "run" : { + "runId": "uuid" + }, + "job": { + "namespace": "scheduler", + "name": "myjob", + "facets": { + "sql": { + "query": "Insert into outputTable from select ** from inputTable" + } + } + }, + "inputs": [ + { + "namespace": "N1", + "name": "inputTable", + "facets": { + "schema": { + "fields": [ {"name": "col_a", "type": "VARCHAR"}, {"name": "col_b", "type": "int"}] + } + } + } + ], + "outputs": [ + { + "namespace": "N2", + "name": "outputTable", + "facets": { + "schema": { + "fields": [ {"name": "col_a", "type": "VARCHAR"}, {"name": "col_b", "type": "int"}] + }, + "columnLineage": { + "fields": { + "col_a": [ { "namespace": "N1", "name": "col_a" } ], + "col_b": [ { "namespace": "N1", "name": "col_b" } ] + } + } + } + } + ] +} +</code></pre> + +<p>Schema:</p> + +<pre><code> "columnLineage": { + "type": "object", + "properties": { + "fields": { + "type": "array", + "items": { + "type": "object", + "additionalProperties": { + "type": "object", + "properties": { + "namespace": { + "type": "string" + }, + "name": { + "type": "string" + }, + }, + "required": [ + "namespace", "name" + ] + } + } + } + } +</code></pre> +
+ +
+
Labels
+ proposal +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-08-09 19:49:51
 +
+

*Thread Reply:* regarding the difference of implementation, the OpenLineage spark integration focuses on extracting metadata and exposing it as a standard representation. (The OpenLineage LineageEvents described in the JSON-Schema spec). The goal is really to have a common language to express lineage and related metadata across everything. We’d be happy if Spline can produce or consume OpenLineage as well and be part of that ecosystem.

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.json | OpenLineage.json>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ben Teeuwen-Schuiringa + (ben.teeuwen@booking.com) +
+
2021-08-18 08:09:38
 +
+

*Thread Reply:* Does anyone know if the Spline developers are in this slack group?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ben Teeuwen-Schuiringa + (ben.teeuwen@booking.com) +
+
2022-08-03 03:07:56
 +
+

*Thread Reply:* @Luke Smith how have things progressed on your side the past year?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-08-09 19:39:28
 +
+

I have opened an issue to track the facet versioning discussion: +https://github.com/OpenLineage/OpenLineage/issues/153

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/issues/153|#153 [PROPOSAL] Version facets independently of the core spec>
+
+ <p><strong>Purpose:</strong> <br /> +The OpenLineage core spec is versioned using a version URL in the <code>$id</code> field: <br /> +See:</p> + +<p>• <a href="https://github.com/OpenLineage/OpenLineage/blob/main/spec/Versioning.md">https://github.com/OpenLineage/OpenLineage/blob/main/spec/Versioning.md</a> +• <a href="https://openlineage.io/spec/1-0-0/OpenLineage.json">https://openlineage.io/spec/1-0-0/OpenLineage.json</a></p> + +<pre><code>{ + "$schema": "&lt;https://json-schema.org/draft/2020-12/schema&gt;", + "$id": "&lt;https://openlineage.io/spec/1-0-0/OpenLineage.json&gt;", +</code></pre> + +<p>Currently all the core facets are also defined in that same core spec.</p> + +<p>This means that the core spec gets a new version whenever a facet is changed along with every other facet. <br /> +We'd rather have the core spec and each facet versioned independently so that their version changes only when they change. <br /> +This matches much better the use cases for using the version number.</p> + +<p>For example:</p> + +<p>• a consumer of OpenLineage would care only about the compatibility of the core spec, accept any version of any facet and handles specific facets independently. +• a producer of a facet cares what version of the core model and that specific facet (core or custom) it is generating. +• a consumer of a specific facet cares about the version of that specific facet and ignore changes to facets it does not consume.</p> + +<p><strong>Proposed implementation</strong></p> + +<p>This proposal is to move each core facet definition into its own file and use the same versioning mechanism (per file) as the core spec.</p> + +<p>This leaves the core spec to only contain the <code>LineageEvent</code> and related core entities (LineageEvent, Run, Job, Dataset) but none of the core facets.</p> + +<p>See for example: <br /> +<a href="https://github.com/OpenLineage/OpenLineage/blob/789b55fc6e6e07d0aedbe130f775721829e83618/spec/OpenLineage.json">https://github.com/OpenLineage/OpenLineage/blob/789b55fc6e6e07d0aedbe130f775721829e83618/spec/OpenLineage.json</a> <br /> +This would be the spec that a consumer would implement to accept any facet version. <br /> +This would ensure that the core spec version changes rarely, only when we add fields to the core model. <br /> +Each facet would be versioned independently following the same versioning mechanism. Each facet version will change if and only if this particular facet schema changes. It also makes core facets and custom facets work exactly the same, they are just maintained in different repositories.</p> + +<p>Example: <br /> +<a href="https://openlineage.io/spec/1-0-0/DatasetSchemaFacet.json">https://openlineage.io/spec/1-0-0/DatasetSchemaFacet.json</a></p> + +<pre><code>{ + "$schema": "&lt;https://json-schema.org/draft/2020-12/schema&gt;", + "$id": "&lt;https://openlineage.io/spec/1-0-0/DatasetSchemaFacet.json&gt;", + "type": "object", + "properties": { + "schema": { + "allOf": [ + { + "$ref": "&lt;https://openlineage.io/spec/1-0-0/OpenLineage.json#/definitions/DatasetFacet&gt;" + }, + { + "type": "object", + "properties": { + "fields": { + "description": "The fields of the table.", + "type": "array", + "items": { + "type": "object", + "properties": { + "name": { + "description": "The name of the field.", + "type": "string", + "example": "column1" + }, + "type": { + "description": "The type of the field.", + "type": "string", + "example": "VARCHAR|INT|..." + }, + "description": { + "description": "The description of the field.", + "type": "string" + } + }, + "required": [ + "name", + "type" + ] + } + } + } + } + ], + "type": "object" + }, + } + } +} +</code></pre> +
+ +
+
Labels
+ proposal +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-08-09 20:16:18
 +
+

I have updated the agenda to the OpenLineage monthly TSC meeting: +https://wiki.lfaidata.foundation/display/OpenLineage/Monthly+TSC+meeting +(meeting information bellow for reference, you can also DM me your email to get added to a google calendar invite)

+ +

The OpenLineage Technical Steering Committee meetings are Monthly on the Second Wednesday 9:00am to 10:00am US Pacific and the link to join the meeting is https://us02web.zoom.us/j/81831865546?pwd=RTladlNpc0FTTDlFcWRkM2JyazM4Zz09 +All are welcome.

+ +

Aug 11th 2021 +• Agenda: + ◦ Coming in OpenLineage 0.1 + ▪︎ OpenLineage spec versioning + ▪︎ Clients + ◦ Marquez integrations imported in OpenLineage + ▪︎ Apache Airflow: + • BigQuery  + • Postgres + • Snowflake + • Redshift + • Great Expectations + ▪︎ Apache Spark + ▪︎ dbt + ◦ OpenLineage 0.2 scope discussion + ▪︎ Facet versioning mechanism + ▪︎ OpenLineage Proxy Backend () + ▪︎ Kafka client + ◦ Roadmap + ◦ Open discussion +• Slides: https://docs.google.com/presentation/d/1Lxp2NB9xk8sTXOnT0_gTXicKX5FsktWa/edit#slide=id.ge80fbcb367_0_14

+ + + +
+ 🙌 Willy Lulciuc, Maciej Obuchowski, Dr Daniel Smith +
+ +
+ 💯 Willy Lulciuc, Dr Daniel Smith +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-08-11 10:05:27
 +
+

*Thread Reply:* Just a reminder that this is in 2 hours

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-08-11 18:50:32
 +
+

*Thread Reply:* I have added the notes to the meeting page: https://wiki.lfaidata.foundation/display/OpenLineage/Monthly+TSC+meeting

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-08-11 18:51:19
 +
+

*Thread Reply:* The recording of the meeting is linked there: +https://us02web.zoom.us/rec/share/2k4O-Rjmmd5TYXzT-pEQsbYXt6o4V6SnS6Vi7a27BPve9aoMmjm-bP8UzBBzsFzg.uY1je-PyT4qTgYLZ?startTime=1628697944000 +• Passcode: =RBUj01C

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Daniel Avancini + (dpavancini@gmail.com) +
+
2021-08-11 13:30:52
 +
+

Hi guys, great discussion today. Something we are particularly interested on is the integration with Airflow 2. I've been searching into Marquez and Openlineage repos and I couldn't find a clear answer on the status of that. I did some work locally to update the marquez-airflow package but I would like to know if someone else is working on this and maybe we could give it some help too.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-08-11 13:36:43
 +
+

*Thread Reply:* @Daniel Avancini I'm working on it. Some changes in airflow made current approach unfeasible, so slight change in a way how we capture events is needed. You can take a look at progress here: https://github.com/OpenLineage/OpenLineage/tree/airflow/2

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Daniel Avancini + (dpavancini@gmail.com) +
+
2021-08-11 13:48:36
 +
+

*Thread Reply:* Thank you Maciej. I'll take a look

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-08-11 20:37:09
 +
+

I have migrated the Marquez issues related to OpenLineage integrations to the OpenLineage repo

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-08-13 19:02:54
 +
+

And OpenLineage 0.1.0 is out ! https://github.com/OpenLineage/OpenLineage/releases/tag/0.1.0

+ + + +
+ 🙌 Peter Hicks, Maciej Obuchowski, Willy Lulciuc, Oleksandr Dvornik, Luke Smith, Daniel Avancini, Matt Gee +
+ +
+ ❤️ Willy Lulciuc, Matt Gee +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Oleksandr Dvornik + (oleksandr.dvornik@getindata.com) +
+
2021-08-16 11:42:24
 +
+

PR ready for review

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/pull/204|#204 Introduce UnknownFacet for spark>
+
+ <p>PR for <a href="https://github.com/OpenLineage/OpenLineage/issues/175">Issue:</a> <br /> +Adding new custom facet for spark with name"spark_unknown". <br /> +Holds information about unvisited nodes from spark plan.</p> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+ 👍 Willy Lulciuc +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Luke Smith + (luke.smith@kinandcarta.com) +
+
2021-08-20 13:54:08
 +
+

Anyone have experience parsing spark's logical plan to generate column-level lineage and DAGs with more human readable operations? I assume I could recreate a graph like the one below using the spark.logicalPlan facet. The analysts writing the SQL / spark queries aren't familiar with ShuffledRowRDD , MapPartitionsRDD, etc... It'd be better if I could convert this plan into spark SQL (or capture spark SQL as a facet at runtime).

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2021-08-26 16:46:53
 +
+

*Thread Reply:* The logicalPlan facet currently returns the Logical Plan, not the physical plan. This means you end up with expressions like Aggregate and Join rather than WholeStageCodegen and Exchange. I don't know if it's possible to reverse engineer the SQL- it's worth looking into the API and trying to find a way to generate that

+ + + +
+
+
+
+ + + + + + + +
+
+ + + + +
+ +
Erick Navarro + (Erick.Navarro@gt.ey.com) +
+
2021-08-31 14:26:35
 +
+

👋 Hi everyone!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Erick Navarro + (Erick.Navarro@gt.ey.com) +
+
2021-08-31 14:27:00
 +
+

Nice to e-meet you 🙂 +I want to use OpenLineage integration for spark in my Azure Databricks clusters, but I am having problems with the configuration of the listener in the cluster, I was wondering if you could help me, if you know any tutorial for the integration of spark with Azure Databricks that can help me, or some more specific guide for this scenario, I would really appreciate it.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Erick Navarro + (Erick.Navarro@gt.ey.com) +
+
2021-08-31 14:27:33
 +
+

I added this configuration to my cluster :

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Erick Navarro + (Erick.Navarro@gt.ey.com) +
+
2021-08-31 14:28:37
 +
+

I receive this error message:

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2021-08-31 14:30:00
 +
+

*Thread Reply:* Hey, @Erick Navarro 👋 . Are you using the openlineage-spark lib? (Note, the marquez-spark lib has been deprecated)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Luke Smith + (luke.smith@kinandcarta.com) +
+
2021-08-31 14:43:20
 +
+

*Thread Reply:* My team had this issue as well. Our read of the error is that Databricks attempts to register the listener before installing packages defined with either spark.jars or spark.jars.packages. Since the listener lib is not yet installed, the listener cannot be found. To solve the issue, we

+ +
  1. copy the OL JAR to a staging directory on DBFS (we use /dbfs/databricks/init/lineage)
  2. using an init script, copy the JAR from the staging directory to the default JAR location for the Databricks driver -- /mnt/driver-daemon/jars
  3. Within the same init script, write the spark config parameters to a .conf file in /databricks/driver/conf (we use open-lineage.conf) +The .conf file will be read by the driver on initialization. It should follow this format (lineagehosturl should point to your API): +[driver] { +"spark.jars" = "/mnt/driver-daemon/jars/openlineage-spark-0.1-SNAPSHOT.jar" +"spark.extraListeners" = "com.databricks.backend.daemon.driver.DBCEventLoggingListener,openlineage.spark.agent.OpenLineageSparkListener" +"spark.openlineage.url" = "$lineage_host_url" +} +Your cluster must be configured to call the init script (enabling lineage for entire cluster). OL is not friendly to notebook-level init as far as we can tell.
    4. +
+ +

@Willy Lulciuc -- I have some utils and init script templates that simplify this process. May be worth adding them to the OL repo along with a readme.

+ + + +
+ 🙏 Erick Navarro +
+ +
+ ❤️ Erick Navarro +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2021-08-31 14:51:46
 +
+

*Thread Reply:* Absolutely, thanks for elaborating on your spark + OL deployment process and I think that’d be great to document. @Michael Collado what are your thoughts?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2021-08-31 14:57:02
 +
+

*Thread Reply:* I haven't tried with Databricks specifically, but there should be no issue registering the OL listener in the Spark config as long as it's done before the Spark session is created- e.g., this example from the README works fine in a vanilla Jupyter notebook- https://github.com/OpenLineage/OpenLineage/tree/main/integration/spark#openlineagesparklistener-as-a-plain-spark-listener

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2021-08-31 15:11:37
 +
+

*Thread Reply:* Looks like Databricks' notebooks come with a Spark instance pre-configured- configuring lineage within the SparkSession configuration doesn't seem possible- https://docs.databricks.com/notebooks/notebooks-manage.html#attach-a-notebook-to-a-cluster 😞

+
+
docs.databricks.com
+ + + +
Manage notebooks | Databricks on AWS
+
+ Learn how to manage notebooks using the UI, the CLI, and by invoking the Workspace API. +
+ + + + + + +
Original URL: https://docs.databricks.com/notebooks/notebooks-manage.html#attach-a-notebook-to-a-cluster
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2021-08-31 15:11:53
 +
+

*Thread Reply:*

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Luke Smith + (luke.smith@kinandcarta.com) +
+
2021-08-31 15:59:38
 +
+

*Thread Reply:* Right, Databricks provides preconfigured spark context / session objects. With Spline, you can set some cluster level config (e.g. spark.spline.lineageDispatcher.http.producer.url ) and install the library on the cluster, but then enable tracking at a notebook level with:

+ +

%scala +import za.co.absa.spline.harvester.SparkLineageInitializer._ +sparkSession.enableLineageTracking() +In OL, it would be nice to install and config OL at a cluster level, but to enable it at a notebook level. This way, users could control whether all notebooks run on a cluster emit lineage or just those with lineage explicitly enabled.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2021-08-31 16:01:00
 +
+

*Thread Reply:* Seems, at the very least, we need to provide a way to specify the job name at the notebook level

+ + + +
+ 👍 Luke Smith +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Luke Smith + (luke.smith@kinandcarta.com) +
+
2021-08-31 16:03:50
 +
+

*Thread Reply:* Agreed. I'd like a default that uses the notebook name that can also be overridden in the notebook.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2021-08-31 16:10:42
 +
+

*Thread Reply:* if you have some insight into the available options, it would be great if you can open an issue on the OL project. I'll have to carve out some time to play with a databricks cluster and learn what options we have

+ + + +
+ 👍 Luke Smith +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Erick Navarro + (Erick.Navarro@gt.ey.com) +
+
2021-08-31 18:26:11
 +
+

*Thread Reply:* Thank you @Luke Smith, the method you recommend works for me, the cluster is running and apparently it fetch the configuration it was my first progress in over a week testing openlineage in azure databricks. Thank you!

+ +

Now I have this:

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Luke Smith + (luke.smith@kinandcarta.com) +
+
2021-08-31 18:52:15
 +
+

*Thread Reply:* Is this error thrown during init or job execution?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2021-08-31 18:55:30
 +
+

*Thread Reply:* this is likely a race condition- I've seen it happen for jobs that start and complete very quickly- things like defining temp views or similar

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Erick Navarro + (Erick.Navarro@gt.ey.com) +
+
2021-08-31 19:59:15
 +
+

*Thread Reply:* During the execution of the job @Luke Smith, thank you @Michael Collado, that was exactly the scenario, the job that I executed was empty, now the cluster is running ok, I don't have errors, I have run some jobs successfully, but I don't see any information in my datakin explorer

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2021-08-31 20:00:46
 +
+

*Thread Reply:* Awesome! Great to hear you’re up and running. For datakin specific questions, mind if we move the discussion to the datakin user slack channel?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Erick Navarro + (Erick.Navarro@gt.ey.com) +
+
2021-08-31 20:01:17
 +
+

*Thread Reply:* Yes Willy, thank you!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Erick Navarro + (Erick.Navarro@gt.ey.com) +
+
2021-09-02 10:06:00
 +
+

*Thread Reply:* Hi , @Luke Smith, thank you for your help, are you familiar with this error in azure databricks when you use OL?

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Erick Navarro + (Erick.Navarro@gt.ey.com) +
+
2021-09-02 10:07:07
 +
+

*Thread Reply:*

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Erick Navarro + (Erick.Navarro@gt.ey.com) +
+
2021-09-02 10:17:17
 +
+

*Thread Reply:* I found the solution here: +https://docs.microsoft.com/en-us/answers/questions/170730/handshake-fails-trying-to-connect-from-azure-datab.html

+
+
docs.microsoft.com
+ + + +
Handshake fails trying to connect from Azure Databricks to Azure PostgreSQL with SSL - Microsoft Q&amp;A
+
+ Microsoft Q&amp;A is the best place to get answers to all your technical questions on Microsoft products and services. Community. Forum. +
+ + + + + + +
Original URL: https://docs.microsoft.com/en-us/answers/questions/170730/handshake-fails-trying-to-connect-from-azure-datab.html
+ + + +
+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Erick Navarro + (Erick.Navarro@gt.ey.com) +
+
2021-09-02 10:17:28
 +
+

*Thread Reply:* It works now! 😄

+ + + +
+ 👍 Luke Smith, Maciej Obuchowski, Minkyu Park, Willy Lulciuc +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2021-09-02 16:33:01
 +
+

*Thread Reply:* @Erick Navarro This might be a helpful to add to our openlineage spark docs for others trying out openlineage-spark with Databricks. Let me know if that’s something you’d like to contribute 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Erick Navarro + (Erick.Navarro@gt.ey.com) +
+
2021-09-02 19:59:10
 +
+

*Thread Reply:* Yes of course @Willy Lulciuc, I will prepare a small tutorial for my colleagues and I will share it with you 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2021-09-02 20:44:36
 +
+

*Thread Reply:* Awesome. Thanks!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Thomas Fredriksen + (thomafred90@gmail.com) +
+
2021-09-02 03:47:35
 +
+

Hello everyone! I am currently evaluating OpenLineage and am finding it very interesting as Prefect is in the list of integrations. However, I am not seeing any documentation or code for this. How far are you from supporting Prefect?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-09-02 04:57:55
 +
+

*Thread Reply:* Hey! If you mean this picture, it provides concept of how OpenLineage works, not current state of integration. We don't have Prefect support yet; hovewer, it's on our roadmap.

+
+ + + + +
+
+ +
+ + + + + + + + +
Original URL: https://raw.githubusercontent.com/OpenLineage/OpenLineage/main/doc/problem.png
+ + + +
+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/issues/81|#81 [INTEGRATION] Add Prefect support>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Thomas Fredriksen + (thomafred90@gmail.com) +
+
2021-09-02 05:22:15
 +
+

*Thread Reply:* great, thanks 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-09-02 11:49:48
 +
+

*Thread Reply:* @Thomas Fredriksen Feel free to chime in the github issue Maciej linked if you want.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Luke Smith + (luke.smith@kinandcarta.com) +
+
2021-09-02 13:13:05
 +
+

What's the timeline to support spark 3.0 within OL? One breaking change we've found is within DatasetSourceVisitor.java -- the DataSourceV2 is deprecated in spark 3.0. There may be other issues we haven't found yet. Is there a good feel for the scope of work required to make OL spark 3.0 compatible?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-09-02 14:28:11
 +
+

*Thread Reply:* It is being worked on right now. @Oleksandr Dvornik is adding an integration test in the build so that we run test for both spark 2.4 and spark 3. Please open an issue with the stack trace if you can. From our perspective, it should be mostly compatible with a few exceptions like this one that we’d want to add test cases for.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-09-02 14:36:19
 +
+

*Thread Reply:* The goal is to be able to make a release in the next few weeks. The integration is being used with Spark 3 already.

+ + + +
+ 🙌 Luke Smith +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Luke Smith + (luke.smith@kinandcarta.com) +
+
2021-09-02 15:50:14
 +
+

*Thread Reply:* Great, I'll take some time to open an issue for this particular issue and a few others.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2021-09-02 17:33:08
 +
+

*Thread Reply:* are you actually using the DatasetSource interface in any capacity? Or are you just scanning the source code to find incompatibilities?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Luke Smith + (luke.smith@kinandcarta.com) +
+
2021-09-03 12:36:20
 +
+

*Thread Reply:* Turns out this has more to do with a how Databricks handles the delta format. It's related to https://github.com/AbsaOSS/spline-spark-agent/issues/96.

+
+ + + + +
<https://github.com/AbsaOSS/spline-spark-agent/issues/96|#96 Throwing runtime exception for delta commands>
+
+ <p>Current spline is not supported for delta commands such as MergeInto, Update, Delete</p> + +<p>I have added <code>io.delta</code> package with scope provided as databricks cluster by default has <code>io.delta</code> imported, and made changes to <code>WriteCommandExtractor.scala</code> file.</p> + +<pre><code>import org.apache.spark.sql.delta.commands._ +</code></pre> + +<p>and added below code to commandsToBeImplemented just to see if it throws custom exception when run in Required mode.</p> + +<pre><code>classOf[MergeIntoCommand] +</code></pre> + +<p>But it throws following exception when I run an application</p> + +<pre><code>Caused by: java.lang.ClassNotFoundException: org.apache.spark.sql.delta.commands.MergeIntoCommand + +</code></pre> + +<p>I am trying to debug the issue but your support on this is highly appreciated.</p> +
+ +
+
Labels
+ question +
+ +
+
Comments
+ 5 +
+ + + + + + + +
+ + <a href="https://github.com/AbsaOSS/spline-spark-agent">AbsaOSS/spline-spark-agent</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Luke Smith + (luke.smith@kinandcarta.com) +
+
2021-09-03 13:42:43
 +
+

*Thread Reply:* I haven't been chasing this issue down on my team -- turns out some things were lost in communication. There are really two problems here:

+ +
  1. When attempting to do delta I/O with Spark 3 on Databricks, e.g. +insert into . . . values . . . +We get an error related to DataSourceV2: +java.lang.NoSuchMethodError: org.apache.spark.sql.execution.datasources.v2.DataSourceV2Relation.source()Lorg/apache/spark/sql/sources/v2/DataSourceV2;
  2. Using Spline, which is Spark 3 compatible, we have issues with the way Databricks handles delta table io. This is related: https://github.com/AbsaOSS/spline-spark-agent/issues/96
    3. +
+ +

So there are two stacked issues related to spark 3 on Databricks with delta IO, not just one. Hope this clears things up.

+
+ + + + +
<https://github.com/AbsaOSS/spline-spark-agent/issues/96|#96 Throwing runtime exception for delta commands>
+
+ <p>Current spline is not supported for delta commands such as MergeInto, Update, Delete</p> + +<p>I have added <code>io.delta</code> package with scope provided as databricks cluster by default has <code>io.delta</code> imported, and made changes to <code>WriteCommandExtractor.scala</code> file.</p> + +<pre><code>import org.apache.spark.sql.delta.commands._ +</code></pre> + +<p>and added below code to commandsToBeImplemented just to see if it throws custom exception when run in Required mode.</p> + +<pre><code>classOf[MergeIntoCommand] +</code></pre> + +<p>But it throws following exception when I run an application</p> + +<pre><code>Caused by: java.lang.ClassNotFoundException: org.apache.spark.sql.delta.commands.MergeIntoCommand + +</code></pre> + +<p>I am trying to debug the issue but your support on this is highly appreciated.</p> +
+ +
+
Labels
+ question +
+ +
+
Comments
+ 5 +
+ + + + + + + +
+ + <a href="https://github.com/AbsaOSS/spline-spark-agent">AbsaOSS/spline-spark-agent</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2021-09-03 13:44:54
 +
+

*Thread Reply:* So, the first issue is OpenLineage related directly, and the second issue applies to both OpenLineage and Spline?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Luke Smith + (luke.smith@kinandcarta.com) +
+
2021-09-03 13:45:49
 +
+

*Thread Reply:* Yes, that's my read of what I'm getting from others on the team.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2021-09-03 13:46:56
 +
+

*Thread Reply:* For the first issue- can you give some details about the target of the INSERT INTO... ? Is it a data source defined in Databricks? a Hive table? a view on GCS?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2021-09-03 13:47:40
 +
+

*Thread Reply:* oh, it's a Delta table?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Luke Smith + (luke.smith@kinandcarta.com) +
+
2021-09-03 14:48:15
 +
+

*Thread Reply:* Yes, it's created via

+ +

CREATE TABLE . . . using DELTA location "/dbfs/mnt/ . . . "

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-09-02 14:28:53
 +
+

I have opened a PR to fix some outdated language in the spec: https://github.com/OpenLineage/OpenLineage/pull/241 Thank you @Mandy Chessell for the feedback

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/pull/241|#241 correct remaining use of transition>
+
+ <p>Signed-off-by: Julien Le Dem <a href="mailto:julien@apache.org">julien@apache.org</a></p> +
+ +
+
Comments
+ 1 +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-09-02 14:37:27
 +
+

The next OpenLineage monthly meeting is next week. Please chime in this thread if you’d like something added to the agenda

+ + + +
+ 🙌 Willy Lulciuc +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
marko + (marko.kristian.helin@gmail.com) +
+
2021-09-04 12:53:54
 +
+

*Thread Reply:* Apache Beam integration? I have a very crude integration at the moment. Maybe it’s better to integrate on the orchestration level (airflow, luigi). Thoughts?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-09-05 13:06:19
 +
+

*Thread Reply:* I think it makes a lot of sense to have a Beam level integration similar to the spark one. Feel free to post a draft PR if you want to share.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-09-07 21:04:09
 +
+

*Thread Reply:* I have added Beam as a topic for the roadmap discussion slide: https://docs.google.com/presentation/d/1fI0u8aE0iX9vG4GGrnQYAEcsJM9z7Rlv/edit#slide=id.ge7d4b64ef4_0_0

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-09-07 21:03:08
 +
+

I have prepared slides for the OpenLineage meeting tomorrow morning: https://docs.google.com/presentation/d/1fI0u8aE0iX9vG4GGrnQYAEcsJM9z7Rlv/edit#slide=id.ge7d4b64ef4_0_0

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-09-07 21:03:32
 +
+

*Thread Reply:* There will be a quick demo of the dbt integration (thanks @Willy Lulciuc!)

+ + + +
+ 🙌 Willy Lulciuc +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-09-07 21:05:13
 +
+

*Thread Reply:* Information to join and archive of previous meetings: https://wiki.lfaidata.foundation/display/OpenLineage/Monthly+TSC+meeting

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-09-08 14:49:52
 +
+

*Thread Reply:* The recording and notes are now available: https://wiki.lfaidata.foundation/display/OpenLineage/Monthly+TSC+meeting

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Venkatesh Tadinada + (venkat@mlacademy.io) +
+
2021-09-08 21:58:09
 +
+

*Thread Reply:* Good meeting today. @Julien Le Dem. Thanks

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Shreyas Kaushik + (shreyask@gmail.com) +
+
2021-09-08 04:03:29
 +
+

Hello, was looking to get some lineage out for BQ in my Airflow DAGs and saw that the BQ extractor here - https://github.com/OpenLineage/OpenLineage/blob/main/integration/airflow/openlineage/airflow/extractors/bigquery_extractor.py#L47 is using an operator that has been deprecated by Airflow - https://github.com/apache/airflow/blob/main/airflow/contrib/operators/bigquery_operator.py#L44 and most of my DAGs are using the operator BigQueryExecuteQueryOperator mentioned there. I presume with this lineage extraction wouldn’t work and some work is needed to support both these operators with the same ( or differnt) extractor. Is that correct or am I missing something ?

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/integration/airflow/openlineage/airflow/extractors/bigquery_extractor.py | bigquery_extractor.py>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+
+ + + + +
<https://github.com/apache/airflow/blob/main/airflow/contrib/operators/bigquery_operator.py | bigquery_operator.py>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/apache/airflow">apache/airflow</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-09-08 04:27:04
 +
+

*Thread Reply:* We're working on updating our integration to airflow 2. Some changes in airflow made current approach unfeasible, so slight change in a way how we capture events is needed. You can take a look at progress here: https://github.com/OpenLineage/OpenLineage/tree/airflow/2

+
+ + + + +
<https://github.com/apache/airflow/pull/10956|#10956 Fully support running more than one scheduler concurrently>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/apache/airflow">apache/airflow</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Shreyas Kaushik + (shreyask@gmail.com) +
+
2021-09-08 04:27:38
 +
+

*Thread Reply:* Thanks @Maciej Obuchowski When is this expected to land in a release ?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Daniel Zagales + (dzagales@gmail.com) +
+
2021-11-11 06:35:24
 +
+

*Thread Reply:* hi @Maciej Obuchowski I wanted to follow up on this to understand when the more recent BQ Operators will be supported, specifically BigQueryInsertJobOperator

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-09-11 22:30:31
 +
+

The PR to separate facets in their own file (and allowing versioning them independently) is now available: https://github.com/OpenLineage/OpenLineage/pull/118

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/pull/118|#118 separate facets to allow versioning them independently>
+
+ <p>This PR includes:</p> + +<p>• Splitting facet outside of the core spec into their own file. + • I wrote a separate automated program to generate each facet file from the original so you don't need to check that they have all the fields. + • You might to review what has been removed from OpenLineage.json and the general structure of each facet +• Updating code gen to deal with that + • Code generation merges all the definitions to be compatible with the existing logic. + • Adding more precise list of BaseFacets (for each entity type). Which in code gen allows for type checking adding the right type of facet in the right place + • It also adds a default implementation to the BaseFacets to allow deserializing custom facets.</p> + +<p>Signed-off-by: Julien Le Dem <a href="mailto:julien@apache.org">julien@apache.org</a></p> +
+ +
+
Comments
+ 1 +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jose Badeau + (jose.badeau@gmail.com) +
+
2021-09-13 03:46:20
 +
+

Hi, new to the channel but I think OL is a great initiative. Currently we are focused on beam/spark/delta but are moving to beam/flink/iceberg and I’m happy to help where I can.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2021-09-13 15:40:01
 +
+

*Thread Reply:* Welcome, @Jose Badeau 👋. That’s exciting to hear as we have Beam, Flink and Iceberg on our roadmap! Your welcome to join the discussion :)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-09-13 20:56:11
 +
+

Per the discussion last week, Ryan updated the metadata that would be available in Iceberg: https://github.com/OpenLineage/OpenLineage/issues/167#issuecomment-917237320

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/issues/167#issuecomment-917237320|Comment on #167 [INTEGRATION] Iceberg should be able to send OpenLineage events independently from where the library is used (including in a plain java process)>
+
+ <p>Here's an outline of what I talked about at the last open lineage meeting:</p> + +<p>OpenLineage and Iceberg</p> + +<p>What is Iceberg?</p> + +<p>• A table format that tracks the files in a table +• A library that implements the table format +• In the format, at "snapshot" is a version of a table +• Provides atomic operations to update the contents of a table +• Provides helper utilities for scanning data</p> + +<p>Iceberg provides notifications at certain points</p> + +<p>• For example, it plans a table scan, Iceberg summarizes that scan and sends a notification with + • The table that was scanned + • The snapshot that was used + • The filters that were pushed down + • The set of columns that were projected</p> + +<p>We want to do the same thing for metrics</p> + +<p>• Scan</p> + +<pre><code>• How many manifest files were read? Data and delete manifests? +• How many data files were read? How many total delete files? How many were reused? +• How long did each manifest read take? +• How long did the total scan planning take? +• How many files per second were located? What was the distribution of files/second over time? +</code></pre> + +<p>• Commit</p> + +<pre><code>• What snapshot was committed? +• What columns were written? +• What partitions were modified or appended to? +</code></pre> + +<p>• Commit metrics</p> + +<pre><code>• How many manifests were rewritten? +• How many manifests were created? +• How many data files were created? +• How many data files were removed? +• How many delete files were created? +• How much data (bytes) was written? +• How much data (bytes) was removed? +• How many retries were needed to complete the commit? +• How long did the first attempt take? +• What was the average length of a retry? +• For each retry, how many manifests were rewritten/created/removed? +• For each retry, how many data/delete files were created/removed? +</code></pre> + +<p>• Commits create a dataset facet</p> + +<pre><code>• Snapshot/version of the table +• Total number of rows +• Total number of data/delete files +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-09-13 21:00:54
 +
+

I have also created tickets for follow up discussions: (#269 and #270): https://wiki.lfaidata.foundation/display/OpenLineage/Monthly+TSC+meeting

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tomas Satka + (satka.tomas@gmail.com) +
+
2021-09-14 04:50:22
 +
+

Hello. I find OpenLineage an interesting tool however can someone help me with integration?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tomas Satka + (satka.tomas@gmail.com) +
+
2021-09-14 04:52:50
 +
+

I am trying to capture lineage from spark 3.1.1 but when executing i constantly get: java.lang.NoSuchMethodError: org.apache.spark.sql.execution.datasources.v2.WriteToDataSourceV2.writer()Lorg/apache/spark/sql/sources/v2/writer/DataSourceWriter; + at openlineage.spark.agent.lifecycle.plan.DatasetSourceVisitor.findDatasetSource(DatasetSourceVisitor.java:57) as if i would be using openlineage on wrong spark version (2.4) I have tried also spark jar from branch feature/itspark3. Is there any branch or release that works or can be tried with spark 3+?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Oleksandr Dvornik + (oleksandr.dvornik@getindata.com) +
+
2021-09-14 05:03:45
 +
+

*Thread Reply:* Hello Tomas. We are currently working on support for spark v3. Can you please raise an issue with stack trace, that would help us to track and solve it. We are currently adding integration tests. Next step would be fix changes in method signatures for v3 (that's what you have)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tomas Satka + (satka.tomas@gmail.com) +
+
2021-09-14 05:12:45
 +
+

*Thread Reply:* Hi @Oleksandr Dvornik i raised https://github.com/OpenLineage/OpenLineage/issues/272

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/issues/272|#272 spark 3 lineage>
+
+ <p>Hi Team</p> + +<p>I am trying to run openlineage with spark 3.1.1. I have tried the version that is released 0.2.2 but also build from main from branch feature/it<em>spark</em>3 but i always get the same exception.</p> + +<pre><code>java.lang.NoSuchMethodError: org.apache.spark.sql.execution.datasources.v2.WriteToDataSourceV2.writer()Lorg/apache/spark/sql/sources/v2/writer/DataSourceWriter; + at openlineage.spark.agent.lifecycle.plan.DatasetSourceVisitor.findDatasetSource(DatasetSourceVisitor.java:57) + at openlineage.spark.agent.lifecycle.plan.DatasetSourceVisitor.isDefinedAt(DatasetSourceVisitor.java:33) + at openlineage.spark.agent.lifecycle.plan.DatasetSourceVisitor.isDefinedAt(DatasetSourceVisitor.java:28) + at openlineage.spark.agent.lifecycle.plan.wrapper.OutputDatasetVisitor.isDefinedAt(OutputDatasetVisitor.java:27) + at openlineage.spark.agent.lifecycle.plan.wrapper.OutputDatasetVisitor.isDefinedAt(OutputDatasetVisitor.java:16) + at scala.PartialFunction$OrElse.isDefinedAt(PartialFunction.scala:170) + at scala.PartialFunction$OrElse.isDefinedAt(PartialFunction.scala:170) + at scala.PartialFunction$OrElse.isDefinedAt(PartialFunction.scala:170) + at scala.PartialFunction$OrElse.isDefinedAt(PartialFunction.scala:170) + at scala.PartialFunction$OrElse.isDefinedAt(PartialFunction.scala:170) + at openlineage.spark.agent.lifecycle.plan.PlanTraversal.isDefinedAt(PlanTraversal.java:32) + at openlineage.spark.agent.lifecycle.SparkSQLExecutionContext.start(SparkSQLExecutionContext.java:87) + at openlineage.spark.agent.OpenLineageSparkListener.lambda$onJobStart$3(OpenLineageSparkListener.java:154) + at java.util.Optional.ifPresent(Optional.java:159) + at openlineage.spark.agent.OpenLineageSparkListener.onJobStart(OpenLineageSparkListener.java:143) + at org.apache.spark.scheduler.SparkListenerBus.doPostEvent(SparkListenerBus.scala:37) + at org.apache.spark.scheduler.SparkListenerBus.doPostEvent$(SparkListenerBus.scala:28) + at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37) + at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37) + at org.apache.spark.util.ListenerBus.postToAll(ListenerBus.scala:117) + at org.apache.spark.util.ListenerBus.postToAll$(ListenerBus.scala:101) + at org.apache.spark.scheduler.AsyncEventQueue.super$postToAll(AsyncEventQueue.scala:105) + at org.apache.spark.scheduler.AsyncEventQueue.$anonfun$dispatch$1(AsyncEventQueue.scala:105) + at scala.runtime.java8.JFunction0$mcJ$sp.apply(JFunction0$mcJ$sp.java:23) + at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62) + at &lt;a href="http://org.apache.spark.scheduler.AsyncEventQueue.org"&gt;org.apache.spark.scheduler.AsyncEventQueue.org&lt;/a&gt;$apache$spark$scheduler$AsyncEventQueue$$dispatch(AsyncEventQueue.scala:100) + at org.apache.spark.scheduler.AsyncEventQueue$$anon$2.$anonfun$run$1(AsyncEventQueue.scala:96) + at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1381) + at org.apache.spark.scheduler.AsyncEventQueue$$anon$2.run(AsyncEventQueue.scala:96) + + +</code></pre> + +<p>Do i do something wrong or is spark 3 simply not yet supported? <br /> +Thanks</p> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+ 👍 Oleksandr Dvornik +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tomas Satka + (satka.tomas@gmail.com) +
+
2021-09-14 08:47:39
 +
+

I also tried to downgrade spark to 2.4.0 and retry with 0.2.2 but i also faced issue.. so my preferred way would be to push for spark 3.1.1 but depends a bit on when you plan to release version supporting it. As backup plan i would try spark 2.4.0 but this is blocking me also: https://github.com/OpenLineage/OpenLineage/issues/274

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/issues/274|#274 spark 2.4 ArrayIndexOutOfBoundsException>
+
+ <p>Hi Team</p> + +<p>I have tried to collect lineage on spark 2.4 with openlineage 0.2.2. <br /> +Here is my pom.xml;</p> + +<pre><code>&amp;lt;project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" + xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"&amp;gt; + &amp;lt;modelVersion&amp;gt;4.0.0&amp;lt;/modelVersion&amp;gt; + + &amp;lt;groupId&amp;gt;foo&amp;lt;/groupId&amp;gt; + &amp;lt;artifactId&amp;gt;bar&amp;lt;/artifactId&amp;gt; + &amp;lt;version&amp;gt;0.1.0-SNAPSHOT&amp;lt;/version&amp;gt; + + &amp;lt;properties&amp;gt; + &amp;lt;java.version&amp;gt;1.8&amp;lt;/java.version&amp;gt; + &amp;lt;maven.compiler.source&amp;gt;${java.version}&amp;lt;/maven.compiler.source&amp;gt; + &amp;lt;maven.compiler.target&amp;gt;${java.version}&amp;lt;/maven.compiler.target&amp;gt; + &amp;lt;project.build.sourceEncoding&amp;gt;UTF-8&amp;lt;/project.build.sourceEncoding&amp;gt; + &amp;lt;project.reporting.outputEncoding&amp;gt;UTF-8&amp;lt;/project.reporting.outputEncoding&amp;gt; + &amp;lt;spark2.version&amp;gt;2.4.0&amp;lt;/spark2.version&amp;gt; + &amp;lt;/properties&amp;gt; + + &amp;lt;build&amp;gt; + &amp;lt;plugins&amp;gt; + &amp;lt;plugin&amp;gt; + &amp;lt;groupId&amp;gt;org.apache.maven.plugins&amp;lt;/groupId&amp;gt; + &amp;lt;artifactId&amp;gt;maven-compiler-plugin&amp;lt;/artifactId&amp;gt; + &amp;lt;version&amp;gt;3.8.1&amp;lt;/version&amp;gt; + &amp;lt;/plugin&amp;gt; + &amp;lt;plugin&amp;gt; + &amp;lt;groupId&amp;gt;org.apache.maven.plugins&amp;lt;/groupId&amp;gt; + &amp;lt;artifactId&amp;gt;maven-jar-plugin&amp;lt;/artifactId&amp;gt; + &amp;lt;version&amp;gt;2.3.2&amp;lt;/version&amp;gt; + &amp;lt;configuration&amp;gt; + &amp;lt;finalName&amp;gt;${project.artifactId}&amp;lt;/finalName&amp;gt; + &amp;lt;/configuration&amp;gt; + &amp;lt;/plugin&amp;gt; + &amp;lt;plugin&amp;gt; + &amp;lt;groupId&amp;gt;org.apache.maven.plugins&amp;lt;/groupId&amp;gt; + &amp;lt;artifactId&amp;gt;maven-shade-plugin&amp;lt;/artifactId&amp;gt; + &amp;lt;version&amp;gt;3.2.4&amp;lt;/version&amp;gt; + &amp;lt;executions&amp;gt; + &amp;lt;execution&amp;gt; + &amp;lt;phase&amp;gt;package&amp;lt;/phase&amp;gt; + &amp;lt;goals&amp;gt; + &amp;lt;goal&amp;gt;shade&amp;lt;/goal&amp;gt; + &amp;lt;/goals&amp;gt; + &amp;lt;configuration&amp;gt; + &amp;lt;filters&amp;gt; + &amp;lt;filter&amp;gt; + &amp;lt;artifact&amp;gt;**:**&amp;lt;/artifact&amp;gt; + &amp;lt;excludes&amp;gt; + &amp;lt;exclude&amp;gt;META-INF/LICENSE&amp;lt;/exclude&amp;gt; + &amp;lt;exclude&amp;gt;META-INF/**.SF&amp;lt;/exclude&amp;gt; + &amp;lt;exclude&amp;gt;META-INF/**.DSA&amp;lt;/exclude&amp;gt; + &amp;lt;exclude&amp;gt;META-INF/**.RSA&amp;lt;/exclude&amp;gt; + &amp;lt;/excludes&amp;gt; + &amp;lt;/filter&amp;gt; + &amp;lt;/filters&amp;gt; + &amp;lt;transformers&amp;gt; + &amp;lt;transformer + implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"/&amp;gt; + &amp;lt;/transformers&amp;gt; + &amp;lt;/configuration&amp;gt; + &amp;lt;/execution&amp;gt; + &amp;lt;/executions&amp;gt; + &amp;lt;/plugin&amp;gt; + &amp;lt;/plugins&amp;gt; + &amp;lt;/build&amp;gt; + + &amp;lt;dependencies&amp;gt; + &amp;lt;dependency&amp;gt; + &amp;lt;groupId&amp;gt;org.apache.spark&amp;lt;/groupId&amp;gt; + &amp;lt;artifactId&amp;gt;spark-sql-kafka-0-10_2.12&amp;lt;/artifactId&amp;gt; + &amp;lt;version&amp;gt;${spark2.version}&amp;lt;/version&amp;gt; + &amp;lt;/dependency&amp;gt; + &amp;lt;dependency&amp;gt; + &amp;lt;groupId&amp;gt;org.apache.spark&amp;lt;/groupId&amp;gt; + &amp;lt;artifactId&amp;gt;spark-core_2.12&amp;lt;/artifactId&amp;gt; + &amp;lt;version&amp;gt;${spark2.version}&amp;lt;/version&amp;gt; + &amp;lt;scope&amp;gt;provided&amp;lt;/scope&amp;gt; + &amp;lt;/dependency&amp;gt; + &amp;lt;dependency&amp;gt; + &amp;lt;groupId&amp;gt;org.apache.spark&amp;lt;/groupId&amp;gt; + &amp;lt;artifactId&amp;gt;spark-sql_2.12&amp;lt;/artifactId&amp;gt; + &amp;lt;version&amp;gt;${spark2.version}&amp;lt;/version&amp;gt; + &amp;lt;scope&amp;gt;provided&amp;lt;/scope&amp;gt; + &amp;lt;/dependency&amp;gt; + &amp;lt;dependency&amp;gt; + &amp;lt;groupId&amp;gt;org.apache.spark&amp;lt;/groupId&amp;gt; + &amp;lt;artifactId&amp;gt;spark-streaming_2.12&amp;lt;/artifactId&amp;gt; + &amp;lt;version&amp;gt;${spark2.version}&amp;lt;/version&amp;gt; + &amp;lt;scope&amp;gt;provided&amp;lt;/scope&amp;gt; + &amp;lt;/dependency&amp;gt; + &amp;lt;dependency&amp;gt; + &amp;lt;groupId&amp;gt;io.openlineage&amp;lt;/groupId&amp;gt; + &amp;lt;artifactId&amp;gt;openlineage-spark&amp;lt;/artifactId&amp;gt; + &amp;lt;version&amp;gt;0.2.2&amp;lt;/version&amp;gt; + &amp;lt;/dependency&amp;gt; + + &amp;lt;/dependencies&amp;gt; + +&amp;lt;/project&amp;gt; +</code></pre> + +<p>This is my simple test that reads csv and write to csv:</p> + +<pre><code>import org.apache.spark.sql.SaveMode; +import org.apache.spark.sql.SparkSession; + +import java.time.Duration; + +public class Example { + public static void main(String[] args) throws InterruptedException { + SparkSession.Builder builder = SparkSession.builder(); + SparkSession session = builder + .appName("java example app") + .master("local[**]") + .config("spark.jars.packages", "io.openlineage:openlineage_spark:0.2.2") + .config("spark.extraListeners", "io.openlineage.spark.agent.OpenLineageSparkListener") + .config("spark.openlineage.url", "&lt;http://localhost:5000/api/v1/namespaces/spark_integration/&gt;") + .getOrCreate(); + + session.read() + .option("header", "true") + .option("inferSchema", "true") + .csv("src/main/resources/wikidata.csv") + .as("source") + .write() + .mode(SaveMode.Overwrite) + .csv("src/main/resources/java-sample.csv"); + Thread.sleep(Duration.ofMinutes(3).toMillis()); + } +} +</code></pre> + +<p>but when i run it i am getting following exception:</p> + +<p>``` +Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: Index 10582 out of bounds for length 249 + at com.thoughtworks.paranamer.BytecodeReadingParanamer$ClassReader.accept(BytecodeReadingParanamer.java:563) + at com.thoughtworks.paranamer.BytecodeReadingParanamer$ClassReader.access$200(BytecodeReadingParanamer.java:338) + at com.thoughtworks.paranamer.BytecodeReadingParanamer.lookupParameterNames(BytecodeReadingParanamer.java:103) + at com.thoughtworks.paranamer.CachingParanamer.lookupParameterNames(CachingParanamer.java:90) + at com.fasterxml.jackson.module.scala.introspect.BeanIntrospector$.getCtorParams(BeanIntrospector.scala:44) + at com.fasterxml.jackson.module.scala.introspect.BeanIntrospector$.$anonfun$apply$1(BeanIntrospector.scala:58) + at com.fasterxml.jackson.module.scala.introspect.BeanIntrospector$.$anonfun$apply$1$adapted(BeanIntrospector.scala:58) + at scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:240) + at scala.collection.Iterator.foreach(Iterator.scala:937) + at scala.collection.Iterator.foreach$(Iterator.scala:937) + at scala.collection.AbstractIterator.foreach(Iterator.scala:1425) + at scala.collection.IterableLike.foreach(IterableLike.scala:70) + at scala.collection.IterableLike.foreach$(IterableLike.scala:69) + at scala.collection.AbstractIterable.foreach(Iterable.scala:54) + at scala.collection.TraversableLike.flatMap(TraversableLike.scala:240) + at scala.collection.TraversableLike.flatMap$(TraversableLike.scala:237) + at scala.collection.AbstractTraversable.flatMap(Traversable.scala:104) + at com.fasterxml.jackson.module.scala.introspect.BeanIntrospector$.findConstructorParam$1(BeanIntrospector.scala:58) + at com.fasterxml.jackson.module.scala.introspect.BeanIntrospector$.$anonfun$apply$19(BeanIntrospector.scala:176) + at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:233) + at scala.collection.In…</p> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-09-14 08:55:44
 +
+

*Thread Reply:* I think this might be actually spark issue: https://stackoverflow.com/questions/53787624/spark-throwing-arrayindexoutofboundsexception-when-parallelizing-list/53787847

+
+
Stack Overflow
+ + + +
Spark throwing ArrayIndexOutOfBoundsException when parallelizing list
+
+ I've tried to create a Spark implementation of QuickSort to test against a serial implementation. I've got the serial implementation working, but the parallel implementation throws an +
+ + + + + + + + +
Original URL: https://stackoverflow.com/questions/53787624/spark-throwing-arrayindexoutofboundsexception-when-parallelizing-list/53787847
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-09-14 08:56:10
 +
+

*Thread Reply: Can you try newer version in 2.4.* line, like 2.4.7?

+ + + +
+ 👀 Tomas Satka +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-09-14 08:57:30
 +
+

*Thread Reply:* This might be also spark 2.4 with scala 2.12 issue - I'd recomment 2.11 versions.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tomas Satka + (satka.tomas@gmail.com) +
+
2021-09-14 09:04:26
 +
+

*Thread Reply:* @Maciej Obuchowski with 2.4.7 i get following exc:

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tomas Satka + (satka.tomas@gmail.com) +
+
2021-09-14 09:04:27
 +
+

*Thread Reply:* 21/09/14 15:03:25 WARN RddExecutionContext: Unable to access job conf from RDD +java.lang.NoSuchFieldException: config$1 + at java.base/java.lang.Class.getDeclaredField(Class.java:2411)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tomas Satka + (satka.tomas@gmail.com) +
+
2021-09-14 09:04:48
 +
+

*Thread Reply:* i can also try to switch to 2.11 scala

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tomas Satka + (satka.tomas@gmail.com) +
+
2021-09-14 09:05:37
 +
+

*Thread Reply:* or do you have some recommended setup that works for sure?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-09-14 09:09:58
 +
+

*Thread Reply:* One more check - you're using Java 8 with this, right?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-09-14 09:10:17
 +
+

*Thread Reply:* This is what works for me: +-&gt; % cat tools/spark-2.4/RELEASE +Spark 2.4.8 (git revision 4be4064) built for Hadoop 2.7.3 +Build flags: -B -Pmesos -Pyarn -Pkubernetes -Pflume -Psparkr -Pkafka-0-8 -Phadoop-2.7 -Phive -Phive-thriftserver -DzincPort=3036

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-09-14 09:11:23
 +
+

*Thread Reply:* spark-shell: +Using Scala version 2.11.12 (OpenJDK 64-Bit Server VM, Java 1.8.0_292)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tomas Satka + (satka.tomas@gmail.com) +
+
2021-09-14 09:12:05
 +
+

*Thread Reply:* awesome let me try 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tomas Satka + (satka.tomas@gmail.com) +
+
2021-09-14 09:26:00
 +
+

*Thread Reply:* data has been sent to marquez. coolio. however i noticed nullpointer being thrown: 21/09/14 15:23:53 ERROR AsyncEventQueue: Listener OpenLineageSparkListener threw an exception +java.lang.NullPointerException + at io.openlineage.spark.agent.OpenLineageSparkListener.onJobEnd(OpenLineageSparkListener.java:164) + at org.apache.spark.scheduler.SparkListenerBus$class.doPostEvent(SparkListenerBus.scala:39) + at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37) + at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37) + at org.apache.spark.util.ListenerBus$class.postToAll(ListenerBus.scala:91) + at <a href="http://org.apache.spark.scheduler.AsyncEventQueue.org">org.apache.spark.scheduler.AsyncEventQueue.org</a>$apache$spark$scheduler$AsyncEventQueue$$super$postToAll(AsyncEventQueue.scala:92) + at org.apache.spark.scheduler.AsyncEventQueue$$anonfun$org$apache$spark$scheduler$AsyncEventQueue$$dispatch$1.apply$mcJ$sp(AsyncEventQueue.scala:92) + at org.apache.spark.scheduler.AsyncEventQueue$$anonfun$org$apache$spark$scheduler$AsyncEventQueue$$dispatch$1.apply(AsyncEventQueue.scala:87) + at org.apache.spark.scheduler.AsyncEventQueue$$anonfun$org$apache$spark$scheduler$AsyncEventQueue$$dispatch$1.apply(AsyncEventQueue.scala:87) + at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58) + at <a href="http://org.apache.spark.scheduler.AsyncEventQueue.org">org.apache.spark.scheduler.AsyncEventQueue.org</a>$apache$spark$scheduler$AsyncEventQueue$$dispatch(AsyncEventQueue.scala:87) + at org.apache.spark.scheduler.AsyncEventQueue$$anon$1$$anonfun$run$1.apply$mcV$sp(AsyncEventQueue.scala:83) + at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1302) + at org.apache.spark.scheduler.AsyncEventQueue$$anon$1.run(AsyncEventQueue.scala:82)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tomas Satka + (satka.tomas@gmail.com) +
+
2021-09-14 10:59:45
 +
+

*Thread Reply:* closed related issue #274

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tomas Satka + (satka.tomas@gmail.com) +
+
2021-09-14 11:02:42
 +
+

does openlineage capture streaming in spark? as this example is not showing me anything unless i replace readStream() with batch read() and writeStream() with write() +```SparkSession.Builder builder = SparkSession.builder(); + SparkSession session = builder + .appName("quantweave") + .master("local[**]") + .config("spark.jars.packages", "io.openlineage:openlineage_spark:0.2.2") + .config("spark.extraListeners", "io.openlineage.spark.agent.OpenLineageSparkListener") + .config("spark.openlineage.url", "http://localhost:5000/api/v1/namespaces/spark_integration/") + .getOrCreate();

+ +
    Dataset&lt;Row&gt; df = session
+            .readStream()
+            .format("kafka")
+            .option("kafka.bootstrap.servers", "localhost:9092")
+            .option("subscribe", "topic1")
+            .option("startingOffsets", "earliest")
+            .load();
+
+    Dataset&lt;Row&gt; dff = df
+            .selectExpr("CAST(key AS STRING)", "CAST(value AS STRING)").as("data");
+
+    dff
+            .writeStream()
+            .format("kafka")
+            .option("kafka.bootstrap.servers", "localhost:9092")
+            .option("topic", "topic2")
+            .option("checkpointLocation", "/tmp/checkpoint")
+            .start();```
+
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-09-14 13:38:09
 +
+

*Thread Reply:* Not at the moment, but it is in scope. You are welcome to open an issue with your example to track this or even propose an implementation if you have the time.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Oleksandr Dvornik + (oleksandr.dvornik@getindata.com) +
+
2021-09-14 15:12:01
 +
+

*Thread Reply:* @Tomas Satka it would be great, if you can add an containerized integration test for kafka with your test case. You can take this as an example here

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/src/test/java/io/openlineage/spark/agent/SparkContainerIntegrationTest.java | SparkContainerIntegrationTest.java>
+
+ <p>``` +package io.openlineage.spark.agent;</p> + +<p>import static java.nio.file.Files.readAllBytes; +import static org.mockserver.model.HttpRequest.request; +import static org.mockserver.model.JsonBody.json;</p> + +<p>import io.openlineage.spark.agent.client.OpenLineageClient; +import java.io.IOException; +import java.nio.file.Path; +import java.nio.file.Paths; +import java.time.Duration; +import java.time.temporal.ChronoUnit; +import org.junit.jupiter.api.AfterAll; +import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.BeforeAll; +import org.junit.jupiter.api.Tag; +import org.junit.jupiter.api.Test; +import org.mockserver.client.MockServerClient; +import org.mockserver.matchers.MatchType; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import org.testcontainers.containers.GenericContainer; +import org.testcontainers.containers.MockServerContainer; +import org.testcontainers.containers.Network; +import org.testcontainers.containers.wait.strategy.Wait; +import org.testcontainers.junit.jupiter.Container; +import org.testcontainers.junit.jupiter.Testcontainers; +import org.testcontainers.utility.DockerImageName;</p> + +<p>@Tag("integration-test") +@Testcontainers +public class SparkContainerIntegrationTest {</p> + +<p>private static final Network network = Network.newNetwork();</p> + +<p>@Container + private static final MockServerContainer openLineageClientMockContainer = + makeMockServerContainer();</p> + +<p>private static GenericContainer&lt;?&gt; pyspark; + private static MockServerClient mockServerClient;</p> + +<p>@BeforeAll + public static void setup() { + mockServerClient = + new MockServerClient( + openLineageClientMockContainer.getHost(), + openLineageClientMockContainer.getServerPort()); + mockServerClient + .when(request("/api/v1/lineage")) + .respond(org.mockserver.model.HttpResponse.response().withStatusCode(201)); + }</p> + +<p>@AfterEach + public void cleanupSpark() { + pyspark.stop(); + }</p> + +<p>@AfterAll + public static void tearDown() { + Logger logger = LoggerFactory.getLogger(SparkContainerIntegrationTest.class); + try { + openLineageClientMockContainer.stop(); + } catch (Exception e2) { + logger.error("Unable to shut down openlineage client container", e2); + } + try { + pyspark.stop(); + } catch (Exception e2) { + logger.error("Unable to shut down pyspark container", e2); + } + network.close(); + }</p> + +<p>private static MockServerContainer makeMockServerContainer() { + return new MockServerContainer( + DockerImageName.parse("jamesdbloom/mockserver:mockserver-5.11.2")) + .withNetwork(network) + .withNetworkAliases("openlineageclient"); + }</p> + +<p>private static GenericContainer&lt;?&gt; makePysparkContainer(String... command) { + return new GenericContainer&lt;&gt;( + DockerImageName.parse("godatadriven/pyspark:" + System.getProperty("spark.version"))) + .withNetwork(network) + .withNetworkAliases("spark") + .withFileSystemBind("src/test/resources/test<em>data", "/test</em>data") + .withFileSystemBind("src/test/resources/spark<em>scripts", "/opt/spark</em>scripts") + .withFileSystemBind("build/libs", "/opt/libs") + .withLogConsumer(SparkContainerIntegrationTest::consumeOutput) + .withStartupTimeout(Duration.of(2, ChronoUnit.MINUTES)) + .dependsOn(openLineageClientMockContainer) + .withReuse(true) + .withCommand(command); + }</p> + +<p>private static void consumeOutput(org.testcontainers.containers.output.OutputFrame of) { + try { + switch (of.getType()) { + case STDOUT: + System.out.write(of.getBytes()); + break; + case STDERR: + System.err.write(of.getBytes()); + break; + case END: + System.out.println(of.getUtf8String()); + break; + } + } catch (IOException ioe) { + throw new RuntimeException(ioe); + } + }</p> + +<p>@Test + public void testPysparkWordCountWithCliArgs() throws IOException, InterruptedException { + pyspark = + makePysparkContainer( + "--master", + "local", + "--conf", + "spark.openlineage.url=" + + "<a href="http://openlineageclient:1080/api/v1/namespaces/testPysparkWordCountWithCliArgs">http://openlineageclient:1080/api/v1/namespaces/testPysparkWordCountWithCliArgs</a>", + "--conf", + "spark.extraListeners=" + OpenLineageSparkListener.class.getName(), + "--jars", + "/opt/libs/" + System.getProperty("openlineage.spark.jar"), + "/opt/spark<em>scripts/spark</em>word_count.py"); + pyspark.setWaitStrategy(Wait.forLogMessage(".<strong>ShutdownHookManager: Shutdown hook called.</strong>", 1)); + pyspark.start();</p> + +<pre><code>Path eventFolder = Paths.get("integrations/container/"); +String startEvent = + new String(readAllBytes(eventFolder.resolve("pysparkWordCountWithCliArgsStartEvent.json"))) + .replaceAll( + "&lt;https://github.com/OpenLineage/OpenLineage/tree/\\$VERSION/integration/spark&gt;", + OpenLineageClient.OPEN_LINEAGE_CLIENT_URI.toString()); + +String completeEvent = + new String( + readAllBytes(eventFolder.resolve("pysparkWordCountWithCliArgsCompleteEvent.json"))) + .replaceAll( + "&lt;https://github.com/OpenLineage/OpenLineage/tree/\\$VERSION/integration/spark&gt;", + OpenLineageClient.OPEN_LINEAGE_CLIENT_URI.toString()); +mockServerClient.verify( + request() + .withPath("/api/v1/lineage") + .withBody(json(startEvent, MatchType.ONLY_MATCHING_FIELDS)), + request() + .withPath("/api/v1/lineage") + .withBody(json(completeEvent, MatchType.ONLY_MATCHING_FIELDS))); +</code></pre> + +<p>}</p> + +<p>@Test + public void testPysparkRddToTable() throws IOException, InterruptedException { + pyspark = + makePysparkContainer( + "--master", + "local", + "--conf", + "spark.openlineage.host=" + "<a href="http://openlineageclient:1080">http://openlineageclient:1080</a>", + "--conf", + "spark.openlineage.namespace=testPysparkRddToTable", + "--conf", + "spark.extraListeners=" + OpenLineageSparkListener.class.getName(), + "--jars", + "/opt/libs/" + System.getProperty("openlineage.spark.jar"), + "/opt/spark<em>scripts/spark</em>rdd<em>to</em>table.py"); + pyspark.setWaitStrategy(Wait.forLogMessage(".<strong>ShutdownHookManager: Shutdown hook called.</strong>", 1)); + pyspark.start();</p> + +<pre><code>Path eventFolder = Paths.get("integrations/container/"); +String startCsvEvent = + new String(readAllBytes(eventFolder.resolve("pysparkRddToCsvStartEvent.json"))) + .replaceAll( + "&lt;https://github.com/OpenLineage/OpenLineage/tree/\\$VERSION/integration/spark&gt;", + OpenLineageClient.OPEN_LINEAGE_CLIENT_URI.toString()); +String completeCsvEvent = + new String(readAllBytes(eventFolder.resolve("pysparkRddToCsvCompleteEvent.json"))) + .replaceAll( + "&lt;https://github.com/OpenLineage/OpenLineage/tree/\\$VERSION/integration/spark&gt;", + OpenLineageClient.OPEN_LINEAGE_CLIENT_URI.toString()); + +String startTableEvent = + new String(readAllBytes(eventFolder.resolve("pysparkRddToTableStartEvent.json"))) + .replaceAll( + "&lt;https://github.com/OpenLineage/OpenLineage/tree/\\$VERSION/integration/spark&gt;", + OpenLineageClient.OPEN_LINEAGE_CLIENT_URI.toString()); + +String completeTableEvent = + new String(readAllBytes(eventFolder.resolve("pysparkRddToTableCompleteEvent.json"))) + .replaceAll( + "&lt;https://github.com/OpenLineage/OpenLineage/tree/\\$VERSION/integration/spark&gt;", + OpenLineageClient.OPEN_LINEAGE_CLIENT_URI.toString()); + +mockServerClient.verify( + request() + .withPath("/api/v1/lineage") + .withBody(json(startCsvEvent, MatchType.ONLY_MATCHING_FIELDS)), + request() + .withPath("/api/v1/lineage") + .withBody(json(completeCsvEvent, MatchType.ONLY_MATCHING_FIELDS)), + request() + … +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tomas Satka + (satka.tomas@gmail.com) +
+
2021-09-14 18:02:05
 +
+

*Thread Reply:* Hi @Oleksandr Dvornik i wrote a test for simple read/write from kafka topic using kafka testcontainer. However i discovered a bug. When writing to kafka topic getting java.lang.IllegalArgumentException: One of the following options must be specified for Kafka source: subscribe, subscribepattern, assign. See the docs for more details.

+ +

• How would you like me to add the test? Fork openlineage and create PR

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tomas Satka + (satka.tomas@gmail.com) +
+
2021-09-14 18:02:50
 +
+

*Thread Reply:* • Shall i raise bug for writing to kafka that should have only "topic" instead of "subscribe"

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tomas Satka + (satka.tomas@gmail.com) +
+
2021-09-14 18:03:42
 +
+

*Thread Reply:* • Since i dont know expected payload to openlineage mock server can somebody help me to create it?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Oleksandr Dvornik + (oleksandr.dvornik@getindata.com) +
+
2021-09-14 19:06:41
 +
+

*Thread Reply:* Hi @Tomas Satka, yes you should create a fork and raise a PR from that. For more details, please take a look at. Not sure about kafka, cause we don't have that integration yet. About expected payload, as a first step, I would suggest to leave that test without assertion for now. Second step would be investigation (what we can get from that plan node). Third step - implementation and asserting a payload. Basically we parse spark optimized plan, and get as much information as we can for specific implementation. You can take a look at recent PR for HIVE. We visit root node and leaves to get output datasets and input datasets accordingly.

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/pull/264|#264 [INTEGRATION[spark] add visitors for Hive>
+
+ <p>Implement visitors for <code>InsertIntoHiveTable</code> and <code>InsertIntoHiveDirCommand</code>.</p> + +<p>Solves <a href="https://github.com/OpenLineage/OpenLineage/issues/187">OpenLineage/OpenLineage#187</a>.</p> + +<p>Signed-off-by: Maciej Obuchowski <a href="mailto:maciej.obuchowski@getindata.com">maciej.obuchowski@getindata.com</a></p> +
+ +
+
Comments
+ 1 +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tomas Satka + (satka.tomas@gmail.com) +
+
2021-09-15 04:37:59
 +
+

*Thread Reply:* Hi @Oleksandr Dvornik PR for step one : https://github.com/OpenLineage/OpenLineage/pull/279

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/pull/279|#279 adding initial test for spark kafka integration>
+
+ <p>Added integration test using kafka testcontainer that is setup on same network. Intention is only to spin up kafka in independent container, run spark job to read from topic1 and write to topic2.</p> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+ 👍 Oleksandr Dvornik +
+ +
+ 🙌 Julien Le Dem +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Luke Smith + (luke.smith@kinandcarta.com) +
+
2021-09-14 15:52:41
 +
+

There may not be an answer to these questions yet, but I'm curious about the plan for Tableau lineage.

+ +

• How will this integration be packaged and attached to Tableau instances? + ◦ via Extensions API, REST API? +• What is the architecture? +https://github.com/OpenLineage/OpenLineage/issues/78

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/issues/78|#78 [INTEGRATION] Add support for Tableau lineage>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Thomas Fredriksen + (thomafred90@gmail.com) +
+
2021-09-15 01:58:37
 +
+

Hi everyone - Following up on my previous post on prefect. The technical integration does not seem very difficult, but I am wondering about how to structure the lineage logic. +Is it the case that each prefect task should be mapped to a lineage job? If so, how do we connect the jobs together? Does there have to be a dataset between each job? +I am OpenLineage with Marquez by the way

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-09-15 09:19:23
 +
+

*Thread Reply:* Hey Thomas!

+ +

Following what we do with Airflow, yes, I think that each task should be mapped to job.

+ +

You don't need datasets between each tasks. It's necessary only where you consume and produce datasets - and it does not matter where in uour job graph you've produced them.

+ +

To map tasks togther In Airflow, we use ParentRunFacet , and the same approach could be used here. In Prefect, I think using flow_run_id would work.

+ + + +
+ 👍 Julien Le Dem +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Thomas Fredriksen + (thomafred90@gmail.com) +
+
2021-09-15 09:26:21
 +
+

*Thread Reply:* this is very helpful, thank you

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Thomas Fredriksen + (thomafred90@gmail.com) +
+
2021-09-15 09:26:43
 +
+

*Thread Reply:* what would be the namespace used in the Job -definition of each task?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-09-15 09:31:34
 +
+

*Thread Reply:* In contrast to dataset namespaces - which we try to standardize, job namespaces should be provided by user, or operator of particular scheduler.

+ +

For example, it would be good if it helped you identify Prefect instance where the job was run.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-09-15 09:32:23
 +
+

*Thread Reply:* If you use openlineage-python client, you can provide namespace either in client constuctor, or via OPENLINEAGE_NAMESPACE env variable.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Thomas Fredriksen + (thomafred90@gmail.com) +
+
2021-09-15 09:32:55
 +
+

*Thread Reply:* awesome, thank you 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Brad + (bradley.mcelroy@live.com) +
+
2021-09-15 17:03:07
 +
+

*Thread Reply:* Hey @Thomas Fredriksen - just chiming in, I’m also keen for a prefect integration. Let me know if I can help out at all

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-09-15 17:27:20
 +
+

*Thread Reply:* Please chime in on https://github.com/OpenLineage/OpenLineage/issues/81

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/issues/81|#81 [INTEGRATION] Add Prefect support>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Brad + (bradley.mcelroy@live.com) +
+
2021-09-15 18:29:20
 +
+

*Thread Reply:* Done!

+ + + +
+ ❤️ Julien Le Dem +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Brad + (bradley.mcelroy@live.com) +
+
2021-09-16 00:06:41
 +
+

*Thread Reply:* For now I'm prototyping in a separate repo https://github.com/limx0/caching_flow_runner/tree/open_lineage

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Thomas Fredriksen + (thomafred90@gmail.com) +
+
2021-09-17 01:55:08
 +
+

*Thread Reply:* I really like your PR, @Brad. I think that using FlowRunner and TaskRunner may be a more "proper" way of doing this, as opposed as adding a state-handler to each task the way I do it.

+ +

How are you dealing with Prefect-library tasks such as the included BigQuery-tasks and such? Is it necessary to create DatasetTask for them to show up in the lineage graph?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Brad + (bradley.mcelroy@live.com) +
+
2021-09-17 02:04:19
 +
+

*Thread Reply:* Hey @Thomas Fredriksen! At the moment I'm not dealing with any task-specific things. The plan (in my head, and after speaking with another prefect user @davzucky) would be that we add a LineageTask subclass where you could define custom facets on a per task basis

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Brad + (bradley.mcelroy@live.com) +
+
2021-09-17 02:05:21
 +
+

*Thread Reply:* or some sort of other hook where basically you would define some lineage attribute or put something in the prefect.context that the TaskRunner would find and attach

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Brad + (bradley.mcelroy@live.com) +
+
2021-09-17 02:06:23
 +
+

*Thread Reply:* Sorry I misread your question - any tasks should be automatically tracked (I believe but have not tested yet!)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Thomas Fredriksen + (thomafred90@gmail.com) +
+
2021-09-17 02:16:02
 +
+

*Thread Reply:* @Brad Could you elaborate a bit on your ideas around adding custom context attributes?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Brad + (bradley.mcelroy@live.com) +
+
2021-09-17 02:21:57
 +
+

*Thread Reply:* yeah so basically we just need some hooks that you can easily access from the task decorator or somewhere else that we can pass through to the open lineage adapter to do things like custom facets

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Brad + (bradley.mcelroy@live.com) +
+
2021-09-17 02:24:31
 +
+

*Thread Reply:* like for your bigquery example - you might want to record some facets like in https://github.com/OpenLineage/OpenLineage/blob/main/integration/common/openlineage/common/provider/bigquery.py and we need a way to do that with the Prefect bigquery task

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Brad + (bradley.mcelroy@live.com) +
+
2021-09-17 02:28:28
 +
+

*Thread Reply:* @davzucky

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Thomas Fredriksen + (thomafred90@gmail.com) +
+
2021-09-17 02:29:12
 +
+

*Thread Reply:* I see. Is this supported by the airflow-integration?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Brad + (bradley.mcelroy@live.com) +
+
2021-09-17 02:29:32
 +
+

*Thread Reply:* I think so, yes

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Brad + (bradley.mcelroy@live.com) +
+
2021-09-17 02:30:51
 +
+

*Thread Reply:* The airflow code is here https://github.com/OpenLineage/OpenLineage/blob/main/integration/airflow/openlineage/airflow/extractors/bigquery_extractor.py

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Brad + (bradley.mcelroy@live.com) +
+
2021-09-17 02:31:54
 +
+

*Thread Reply:* (I don't actually use airflow or bigquery - but for my own use case I can see wanting to do thing like this)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Thomas Fredriksen + (thomafred90@gmail.com) +
+
2021-09-17 03:18:27
 +
+

*Thread Reply:* Interesting, I like how dynamic this is

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Chris Baynes + (chris@contiamo.com) +
+
2021-09-15 09:09:21
 +
+

HI all, I have a clarification question about dataset namespaces. What's the difference between a dataset namespace (in the input/output) and a dataSource name (in the dataSource facet)? +The dbt integration appears to set those to the same value (e.g. <snowflake://myprofile>), however it seems that Marquez assumes the dataset namespace to be a more generic concept (similar to a nice user provided name like the job namespace).

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-09-15 09:29:25
 +
+

*Thread Reply:* Hey. +Generally, dataSource name should be namespace of particular dataset.

+ +

In some cases, like Postgres, dataSource facet is used to provide additionally connection strings, with info like particular host and port that we're connected to.

+ +

In case of Snowflake - or Bigquery, or S3, or multiple systems where we have only "global" instance, so the dataSource facet does not carry any other additional information.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Chris Baynes + (chris@contiamo.com) +
+
2021-09-15 10:11:19
 +
+

*Thread Reply:* Thanks. So then perhaps marquez could differentiate a bit more between job & dataset namespaces. Right now it doesn't quite feel right to have a single global list of namespaces for jobs & datasets, especially as they also have a separate concept of sources (which are not in a namespace).

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-09-15 10:18:59
 +
+

*Thread Reply:* @Willy Lulciuc what do you think?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Chris Baynes + (chris@contiamo.com) +
+
2021-09-15 10:41:20
 +
+

*Thread Reply:* As an example, in marquez I have this list of namespaces (from some sample data): dbt-sales, default, <snowflake://my-account1>, <snowflake://my-account2>. +I think the new marquez UI with the nice namespace dropdown and job/dataset search is awesome, and I'd expect to be able to filter by job namespace everywhere, but how about being able to filter datasets by source (which would be populated by the OL dataset namespace) and not persist dataset namespaces in the global namespace table?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-09-15 18:38:03
 +
+

The dbt integration (https://github.com/OpenLineage/OpenLineage/tree/main/integration/dbt) is pretty awesome but there are still a few improvements we could make. +Here are a few thoughts. +• In dbt-ol if the configuration is wrong or missing we will fail silently. This one seems like a good first thing to fix by logging the error to stdout +• We need to wait until the end to know if it worked at all. It would be nice if we checked the config at the beginning and display an error right away. Possibly by adding a parent job/run with a start event at the beginning and an end event at the end when all is done. +• While we are sending events at the end the console will hang until it’s done. It’s not clear that progress is made. We could have a simple progress bar by printing a dot for every event sent. (ex: sending 10 OpenLineage events: .........) +• We could also write at the beginning that the OL events will be sent at the end so that the user knows what to expect. +What do you think? (@Maciej Obuchowski in particular, but anyone using dbt in general)

+ + + +
+ 👀 Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-09-15 18:43:18
 +
+

*Thread Reply:* Last point is that we should persist the configuration and not just have it in environment variables. What is the best way to do this in dbt?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-09-15 18:49:21
 +
+

*Thread Reply:* We could have something similar to https://docs.getdbt.com/dbt-cli/configure-your-profile - or even put our config in there

+ + + +
+ ❤️ Julien Le Dem +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-09-15 18:51:42
 +
+

*Thread Reply:* I think we should assume that variables/config should be set and valid - and fail the run if they aren't. After all, if someone wouldn't need lineage events, they wouldn't use our wrapper.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-09-15 18:56:36
 +
+

*Thread Reply:* 3rd point would be easy to address if we could send events async/in parallel. But there could be dataset version dependencies, and we don't want to get into needless complexity of recognizing that, building a dag etc.

+ +

We could batch events if the network roundtrips are responsible for majority of the slowdown. However, we can't assume any particular environment.

+ +

Maybe just notifying about the progress is the best thing we can do right now.

+ + + +
+ 👀 Mario Measic +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-09-15 18:58:22
 +
+

*Thread Reply:* About second point, I want to add recognizing if we already have a parent run - for example, if running via airflow. If not, creating run for this purpose is a good idea.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-09-15 21:31:35
 +
+

*Thread Reply:* @Maciej Obuchowski can you open github issues to propose those changes?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-09-16 09:11:31
 +
+

*Thread Reply:* Done

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2021-09-16 12:05:10
 +
+

*Thread Reply:* FWIW, I have been putting my config in ~/.openlineage/config so it can be mapped into a container

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-09-16 17:56:23
 +
+

*Thread Reply:* Makes sense, also, all clients could use that config

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mario Measic + (mario.measic.gavran@gmail.com) +
+
2021-10-18 04:47:08
 +
+

*Thread Reply:* if dbt could actually stream the events, that would be great.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-10-18 09:59:12
 +
+

*Thread Reply:* Unfortunately, this seems very unlikely for now, due to the fact that we rely on metadata files that dbt only produces after end of execution.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-09-15 22:52:09
 +
+

The split of facets in their own schemas is ready to be merged: https://github.com/OpenLineage/OpenLineage/pull/118

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/pull/118|#118 separate facets to allow versioning them independently>
+
+ <p>This PR includes:</p> + +<p>• Splitting facet outside of the core spec into their own file. + • I wrote a separate automated program to generate each facet file from the original so you don't need to check that they have all the fields. + • You might to review what has been removed from OpenLineage.json and the general structure of each facet +• Updating code gen to deal with that + • Code generation merges all the definitions to be compatible with the existing logic. + • Adding more precise list of BaseFacets (for each entity type). Which in code gen allows for type checking adding the right type of facet in the right place + • It also adds a default implementation to the BaseFacets to allow deserializing custom facets.</p> + +<p>Signed-off-by: Julien Le Dem <a href="mailto:julien@apache.org">julien@apache.org</a></p> +
+ +
+
Comments
+ 1 +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Brad + (bradley.mcelroy@live.com) +
+
2021-09-16 00:12:02
 +
+

Hey @Julien Le Dem I'm going to start a thread here for any issues I run into trying to build a prefect integration

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Brad + (bradley.mcelroy@live.com) +
+
2021-09-16 00:16:44
 +
+

*Thread Reply:* This might be useful to others https://github.com/OpenLineage/OpenLineage/pull/284

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Brad + (bradley.mcelroy@live.com) +
+
2021-09-16 00:18:44
 +
+

*Thread Reply:* So I'm trying to push a simple event to marquez, but getting the following response: +'{"code":400,"message":"Unable to process JSON"}' +The JSON I'm pushing:

+ +

{ + "eventTime": "2021-09-16T04:00:28.343702", + "eventType": "START", + "inputs": {}, + "job": { + "facets": {}, + "name": "prefect.core.parameter.p", + "namespace": "default" + }, + "producer": "<https://github.com/OpenLineage/OpenLineage/tree/0.0.0/integration/prefect>", + "run": { + "facets": {}, + "runId": "3bce33cb-9495-4c58-b326-6aac71634ace" + } +} +Does anything look obviously wrong here?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
marko + (marko.kristian.helin@gmail.com) +
+
2021-09-16 02:41:11
 +
+

*Thread Reply:* What I did previously when debugging something like this was to remove half of the payload until I found the culprit. Binary search essentially. I was running Marquez locally, so probably could’ve enabled better logging as well. +Aren’t inputs and facets arrays?

+ + + +
+ 👍 Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Brad + (bradley.mcelroy@live.com) +
+
2021-09-16 03:14:54
 +
+

*Thread Reply:* Thanks for the response @marko - this is a greatly reduced payload already (but I'll keep going). Yep they are supposed to be arrays (I've since fixed that)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Brad + (bradley.mcelroy@live.com) +
+
2021-09-16 03:46:01
 +
+

*Thread Reply:* okay it was my timestamp 🥲

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Brad + (bradley.mcelroy@live.com) +
+
2021-09-16 19:07:16
 +
+

*Thread Reply:* Okay - I've got a simply working example now https://github.com/limx0/caching_flow_runner/blob/open_lineage/caching_flow_runner/task_runner.py

+
+ + + + +
<https://github.com/limx0/caching_flow_runner/blob/open_lineage/caching_flow_runner/task_runner.py | task_runner.py>
+
+ <pre><code>import datetime +from typing import Dict, List + +import prefect +from openlineage.client.facet import DataSourceDatasetFacet +from openlineage.client.run import InputDataset +from openlineage.client.run import OutputDataset +from prefect import Parameter +from prefect.core import Edge +from prefect.engine import TaskRunner +from prefect.engine.result import Result +from prefect.engine.state import Failed +from prefect.engine.state import Pending +from prefect.engine.state import Running +from prefect.engine.state import State +from prefect.engine.state import Success + +from caching_flow_runner.adapter import OpenLineageAdapter +from caching_flow_runner.util import task_qualified_name + + +def utc_now() -&amp;gt; str: + """Create an openlineage compatible timestamp string""" + return datetime.datetime.now(datetime.timezone.utc).isoformat()[:-6] + "Z" + + +def flow_namespace(): + """Create a namespace from a flow (and optionally project) name""" + project = prefect.context.get("project_name") + prefix = f"{project}." if project else "" + return f"{prefix}{prefect.context.flow_name}" + + +def result_location(result: Result, ****raw_inputs) -&amp;gt; str: + """Determine results location using the same formatting kwargs prefect does in `task_runner.get_task_run_state`""" + formatting_kwargs = { + ****prefect.context.get("parameters", {}).copy(), + ****prefect.context, + ****raw_inputs, + } + clone = result.copy().format(****formatting_kwargs) + return clone.location + + +class OpenLineageTaskRunner(TaskRunner): + def __init__(self, **args, client: OpenLineageAdapter, ****kwargs): + super().__init__(**args, ****kwargs) + self._client = client + self.task_full_name = task_qualified_name(task=self.task) + self.state_handlers.append(self.on_state_changed) + self.task_inputs = None + self.inputs_to_tasks = {} + + def on_state_changed(self, _, old_state: State, new_state: State): + if isinstance(old_state, Running) and isinstance(new_state, Success): + self._on_success(new_state) + elif isinstance(old_state, (Pending, Running)) and isinstance(new_state, Failed): + self._on_failure(state=new_state) + + def _prefect_inputs_to_input_dataset(self, inputs: Dict[str, Result]) -&amp;gt; List[InputDataset]: + """Convert prefect inputs to input Datasets for OpenLineage""" + if not inputs: + return [] + return [ + InputDataset( + namespace=flow_namespace(), + name=self.inputs_to_tasks[k], + facets={}, + inputFacets={}, + ) + for k, v in inputs.items() + ] + + def prefect_result_to_output_dataset(self, result: Result) -&amp;gt; OutputDataset: + output_facets = {} + if not isinstance(self.task, Parameter): + output_facets["output-dataset"] = DataSourceDatasetFacet( + name=f"{self.task_full_name}-output", + uri=result_location(result, ****self.task_inputs), + ) + return OutputDataset( + namespace=flow_namespace(), + name=f"{self.task_full_name}", + facets={}, + outputFacets=output_facets, + ) + + def _task_description(self): + if isinstance(self.task, Parameter): + # Parameters don't have any doc / description at this stage, simply return the name + return self.task.name + else: + return self.task.__doc__ + + def _on_start(self, inputs: Dict[str, Result]): + context = prefect.context + run_id = self._client.start_task( + run_id=context.task_run_id, + job_name=self.task_full_name, + job_description=self._task_description(), + event_time=utc_now(), + parent_run_id=context.flow_run_id, + code_location=None, + inputs=self._prefect_inputs_to_input_dataset(inputs), + outputs=None, + ) + &lt;a href="http://self.logger.info"&gt;self.logger.info&lt;/a&gt;(f"Marquez run CREATED run_id: {run_id}") + + def _on_success(self, state: Success): + context = prefect.context + run_id = self._client.complete_task( + run_id=context.task_run_id, + job_name=self.task_full_name, + inputs=None, + end_time=utc_now(), + outputs=[self.prefect_result_to_output_dataset(result=state._result)], + ) + &lt;a href="http://self.logger.info"&gt;self.logger.info&lt;/a&gt;(f"Marquez run COMPLETE run_id: {run_id}") + + def _on_failure(self, state: Failed): + context = prefect.context + run_id = self._client.fail_task( + run_id=context.task_run_id, + job_name=self.task_full_name, + inputs=None, + outputs=None, + end_time=utc_now(), + ) + &lt;a href="http://self.logger.info"&gt;self.logger.info&lt;/a&gt;(f"Marquez run FAILED run_id: {run_id}") + + def get_task_inputs( + self, state: State, upstream_states: Dict[Edge, State] + ) -&amp;gt; Dict[str, Result]: + for upstream in upstream_states: + self.inputs_to_tasks[upstream.key] = task_qualified_name(task=upstream.upstream_task) + task_inputs = super().get_task_inputs(state=state, upstream_states=upstream_states) + return task_inputs + + def set_task_to_running(self, state: State, inputs: Dict[str, Result]) -&amp;gt; State: + self.task_inputs = inputs + state = super().set_task_to_running(state=state, inputs=inputs) + self._on_start(inputs=inputs) + return state + +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/limx0/caching_flow_runner">limx0/caching_flow_runner</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Brad + (bradley.mcelroy@live.com) +
+
2021-09-16 19:07:37
 +
+

*Thread Reply:* I might move this into a proper PR @Julien Le Dem

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Brad + (bradley.mcelroy@live.com) +
+
2021-09-16 19:08:12
 +
+

*Thread Reply:* Successfully got a basic prefect flow working

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Brad + (bradley.mcelroy@live.com) +
+
2021-09-16 02:11:53
 +
+

A question about DatasetType - is there a representation for a file-like type? For files stored in S3/FTP/NFS etc (assuming a fully resolvable url)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-09-16 09:53:24
 +
+

*Thread Reply:* I think there was some talk somewhere to actually drop the DatasetType concept; can't find where though.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-09-16 10:04:09
 +
+

*Thread Reply:* I've taken a look at your repo. Looks great so far!

+ +

One thing I've noticed I don't think you need to use any stuff from Marquez to emit events. It's lineage ingestion API is deprecated - you can just use openlineage-python client. If there's something you think it's missing from it, feel free to write that here or open issue.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Brad + (bradley.mcelroy@live.com) +
+
2021-09-16 17:12:31
 +
+

*Thread Reply:* And would that be replaced by just some Input/Output notion @Maciej Obuchowski?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Brad + (bradley.mcelroy@live.com) +
+
2021-09-16 17:13:26
 +
+

*Thread Reply:* Oh yeah I got a little confused by the single lineage endpoint - but I’ve realised how it all works now. I’m still using the marquez backend to view things but I’ll use the openlineage-client to talk to it

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-09-16 17:34:46
 +
+

*Thread Reply:* Yes 🙌

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tomas Satka + (satka.tomas@gmail.com) +
+
2021-09-16 06:04:30
 +
+

When trying to fix failing checks, i see integration-test-integration-airflow to fail +```#!/bin/bash -eo pipefail +if [[ GCLOUDSERVICEKEY,GOOGLEPROJECTID == "" ]]; then + echo "No required environment variables to check; moving on" +else + IFS="," read -ra PARAMS <<< "GCLOUDSERVICEKEY,GOOGLEPROJECTID"

+ +

for i in "${PARAMS[@]}"; do + if [[ -z "${!i}" ]]; then + echo "ERROR: Missing environment variable {i}" >&2

+ +
  if [[ -n "" ]]; then
+    echo "" &gt;&amp;2
+  fi
+
+  exit 1
+else
+  echo "Yes, ${i} is defined!"
+fi
+
+ +

done +fi

+ +

ERROR: Missing environment variable {i}

+ +

Exited with code exit status 1 +CircleCI received exit code 1``` +However i havent touch airflow at all.. can somebody help please?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-09-16 06:59:34
 +
+

*Thread Reply:* Hey, Airflow integration tests do not pass env variables to PRs from forks due to security reasons - everyone could create malicious PR and dump secrets

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-09-16 07:00:29
 +
+

*Thread Reply:* So, they will fail and there's nothing to do from your side 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-09-16 07:00:55
 +
+

*Thread Reply:* We probably should split those into ones that don't touch external systems, and run those for all PRs

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tomas Satka + (satka.tomas@gmail.com) +
+
2021-09-16 07:08:03
 +
+

*Thread Reply:* ah okie. good to know. +and in build-integration-spark Could not resolve all artifacts. Is that also known issue? Or something from my side that i could fix?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-09-16 07:11:12
 +
+

*Thread Reply:* Looks like gradle server problem? +&gt; Could not get resource '<https://plugins.gradle.org/m2/com/diffplug/spotless/spotless-lib/2.13.2/spotless-lib-2.13.2.module>'. + &gt; Could not GET '<https://plugins.gradle.org/m2/com/diffplug/spotless/spotless-lib/2.13.2/spotless-lib-2.13.2.module>'. Received status code 500 from server: Internal Server Error

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-09-16 07:34:44
 +
+

*Thread Reply:* After retry, there's spotless error:

+ +

+········.orElse(Collections.emptyList()).stream()

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-09-16 07:35:15
 +
+

*Thread Reply:* I think this is due to mismatch between behavior of spotless in Java 8 and Java 11+ - which you probably used 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tomas Satka + (satka.tomas@gmail.com) +
+
2021-09-16 07:40:01
 +
+

*Thread Reply:* ah.. i used java11. so shall i rerun something with java8 setup as sdk?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-09-16 07:44:31
 +
+

*Thread Reply:* For spotless, you can just fix this one line 🙂 +Though I don't guarantee that tests that run later will pass, so you might need Java 8 for later testing

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tomas Satka + (satka.tomas@gmail.com) +
+
2021-09-16 08:04:36
 +
+

*Thread Reply:* yup looks better now

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tomas Satka + (satka.tomas@gmail.com) +
+
2021-09-16 08:04:41
 +
+

*Thread Reply:* thanks

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tomas Satka + (satka.tomas@gmail.com) +
+
2021-09-16 14:27:02
 +
+

*Thread Reply:* will somebody please review my PR? had to already adjust due to updates on same test class 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Brad + (bradley.mcelroy@live.com) +
+
2021-09-16 20:36:28
 +
+

Hey team - I've opened https://github.com/OpenLineage/OpenLineage/pull/293 for a very WIP prefect integration

+ + + +
+ 🙌 Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Brad + (bradley.mcelroy@live.com) +
+
2021-09-16 20:37:27
 +
+

*Thread Reply:* @Thomas Fredriksen would love any feedback

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Thomas Fredriksen + (thomafred90@gmail.com) +
+
2021-09-17 04:21:13
 +
+

*Thread Reply:* nicely done! As we discussed in another thread - the way you have implemented lineage using FlowRunner and TaskRunner is likely the best way to do this. Let me know if you need any help, I would love to see this PR get merged!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-09-17 07:28:33
 +
+

*Thread Reply:* Hey @Brad, it looks great!

+ +

I've seen you're using task_qualified_name to name datasets and I don't think it's the right way. +I'd take a look at naming conventions here: https://github.com/OpenLineage/OpenLineage/blob/main/spec/Naming.md

+ +

Getting that right is key to making sure that lineage is properly tracked between systems - for example, if you use Prefect to schedule dbt runs or pyspark jobs, the unified naming makes sure that all those integrations properly refer to the same dataset.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Brad + (bradley.mcelroy@live.com) +
+
2021-09-17 08:12:50
 +
+

*Thread Reply:* Hey @Maciej Obuchowski thanks for the feedback. Yep the naming was a bit of a placeholder. Open to any recommendations.. I think things like dbt or pyspark are straight forward (we could add special handling for tasks like that) but what about regular transformation type tasks that run in a scheduler? Do you have any naming preference? Say I just had some pandas transform task in prefect for example

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-09-17 08:28:04
 +
+

*Thread Reply:* First of all, not all tasks are producing and consuming datasets. For example, I wouldn't expect any of the Github tasks to have any datasets.

+ +

Second, in Airflow we have a concept of Extractor where you can write specialized code to expose datasets. For example, for BigQuery we extract datasets from query plan. Now, I'm not sure if this concept would translate well to Prefect - but if yes, then we have some helpers inside openlineage common library that could be reused. Also, this way allows to emit additional facets, some of which are really useful - like query statistics for BigQuery, and data quality tests for dbt.

+ +

Third, if we're talking about generalized tasks like FunctionTask or ShellTask, then I think the right way is to expose functionality to user to expose lineage themselves. I'm not sure how exactly that would look in Prefect.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Brad + (bradley.mcelroy@live.com) +
+
2021-09-19 23:03:14
 +
+

*Thread Reply:* You've raised some good points @Maciej Obuchowski - I might have been thinking about this integration in slightly the wrong way. I think based on your comments I'll refactor some of the code to hook into the Results object in prefect (The Result object is the way in which data is serialized and persisted).

+ +

> Now, I'm not sure if this concept would translate well to Prefect - but if yes, then we have some helpers inside openlineage common library that could be reused +This definitely applies to prefect and the similar tasks exist in prefect and we should definitely leverage the common library in this case.

+ +

> Third, if we're talking about generalized tasks like FunctionTask or ShellTask, then I think the right way is to expose functionality to user to expose lineage themselves. I'm not sure how exactly that would look in Prefect. +Yeah I agree with this. I'd like to make it as easy a possible to opt-in, but I think you're right that there needs to be some hooks for user defined lineage. I'll think about this a little more.

+ +

> First of all, not all tasks are producing and consuming datasets. For example, I wouldn't expect any of the Github tasks to have any datasets. +My initial thoughts here were that it would still be good to have lineage as these tasks do have side effects, and downstream consumers of the lineage data might want to know about these tasks. However I don't have a good feeling yet how best to do this, so I'm going to park those thoughts for now.

+ + + +
+ 🙌 Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-09-20 06:30:51
 +
+

*Thread Reply:* > Yeah I agree with this. I'd like to make it as easy a possible to opt-in, but I think you're right that there needs to be some hooks for user defined lineage. I'll think about this a little more. +First version of an integration doesn't have to be perfect. in particular, not handling this use case would be okay, since it does not lock us into some particular way of doing it later.

+ +

> My initial thoughts here were that it would still be good to have lineage as these tasks do have side effects, and downstream consumers of the lineage data might want to know about these tasks. However I don't have a good feeling yet how best to do this, so I'm going to park those thoughts for now. +I'd think of two options first, before modeling it as a dataset: +Won't existence of a event be enough? After all, we'll still have it despite it not having any input and output datasets. +If not, then wouldn't custom run or job facet be a better fit?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Brad + (bradley.mcelroy@live.com) +
+
2021-09-23 17:27:49
 +
+

*Thread Reply:* > Won’t existence of a event be enough? After all, we’ll still have it despite it not having any input and output datasets. +Duh, yep you’re right @Maciej Obuchowski, I’m over thinking this. I’m going to clean this up based on your comments

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Thomas Fredriksen + (thomafred90@gmail.com) +
+
2021-10-06 03:39:28
 +
+

*Thread Reply:* Hi @Brad. How will this integration work for Prefect flows running in Prefect Cloud or on Prefect Server?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Brad + (bradley.mcelroy@live.com) +
+
2021-10-06 03:40:44
 +
+

*Thread Reply:* Hi @Thomas Fredriksen - it'll relate to the agent actually - you'll need to pass the flow runner class to the agent when running

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Thomas Fredriksen + (thomafred90@gmail.com) +
+
2021-10-06 03:48:14
 +
+

*Thread Reply:* nice!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Brad + (bradley.mcelroy@live.com) +
+
2021-10-06 03:48:54
 +
+

*Thread Reply:* Unfortunately I've been a little busy the past week, and I will be for the rest of this week

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Brad + (bradley.mcelroy@live.com) +
+
2021-10-06 03:49:09
 +
+

*Thread Reply:* but I do plan to pick this up next week

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Brad + (bradley.mcelroy@live.com) +
+
2021-10-06 03:49:23
 +
+

*Thread Reply:* (the additional changes I mention above)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Thomas Fredriksen + (thomafred90@gmail.com) +
+
2021-10-06 03:50:08
 +
+

*Thread Reply:* looking forward to it 🙂 let me know if you need any help!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Brad + (bradley.mcelroy@live.com) +
+
2021-10-06 03:50:34
 +
+

*Thread Reply:* yeah when I get this next lot of stuff in - I'd love for people to test it out

+ + + +
+ 🙌 Thomas Fredriksen, Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Adam Pocock + (adam.pocock@oracle.com) +
+
2021-09-20 17:38:51
 +
+

Is there a preferred academic citation for OpenLineage? I’m writing a paper on the provenance system in our machine learning library, and I’d like to cite OpenLineage as an example of future work on data lineage to integrate with.

+ + + +
+ 🙌 Willy Lulciuc +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-09-20 19:18:53
 +
+

*Thread Reply:* I think you can reffer to https://openlineage.io/

+
+
openlineage.io
+ + + +
Home
+
+ Data lineage is the foundation for a new generation of powerful, context-aware data tools and best practices. OpenLineage enables consistent collection of lineage metadata, creating a deeper understanding of how data is produced and used. +
+ + + + + + +
Original URL: https://openlineage.io/
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-09-20 19:31:30
 +
+

We’re starting to see the beginning of larger contributions (Spark streaming, prefect, …) and I think we need to define a way to accept those contributions incrementally. +If we take the example of Streaming (Spark streaming, Flink or Beam) support (but really this applies in general, sorry to pick on you Tomas, this is great!): +The first Spark streaming PR ( https://github.com/OpenLineage/OpenLineage/pull/279 ) lays the ground work for testing spark streaming but there’s more work to have a full feature. +I’m in favor of merging Spark streaming support into main once it’s working end to end (possibly with partial input/output coverage). +So I see 2 options:

+ +
  1. start a branch for spark streaming support. Have PRs like this one go into it until it’s completed (smaller reviews). Then merge the whole thing as a PR in main when it’s finished
  2. Keep working on that PR until it’s fully implemented, but it will get big, and make reviews difficult. +I have seen the model 1) work well. It’s easier to do multiple smaller reviews for larger projects.
    3. +
+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/pull/279|#279 adding initial test for spark kafka integration>
+
+ <p>Added integration test using kafka testcontainer that is setup on same network. Intention is only to spin up kafka in independent container, run spark job to read from topic1 and write to topic2.</p> +
+ +
+
Comments
+ 2 +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+ 👍 Ross Turk, Maciej Obuchowski, Faouzi +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Yannick Endrion + (yannick.endrion@gmail.com) +
+
2021-09-24 05:10:04
 +
+

Thank you @Ross Turk for this really useful article: https://openlineage.io/blog/dbt-with-marquez/?s=03 +Is anyone aware of additional environment being supported by the dbt<->OpenLineage<->Marquez integration ? I think only Snowflake and BigQuery are supported now. +I am really interested by SQLServer or even Dremio (which could be great because capable of read from multiples DB).

+ +

Thank you

+
+
openlineage.io
+ + + +
Using Marquez to Visualize dbt Models
+
+ Each time dbt runs it generates a trove of metadata about datasets and the work it performs with them. In this post, I’d like to show you how to harvest this metadata and put it to good use. +
+ + + + + + +
Original URL: https://openlineage.io/blog/dbt-with-marquez/?s=03
+ + + +
+ + + +
+ 🎉 Minkyu Park, Ross Turk +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-09-24 05:15:31
 +
+

*Thread Reply:* It should be really easy to add additional databases. Basically, we'd need to know how to get namespace for that database: https://github.com/OpenLineage/OpenLineage/blob/main/integration/common/openlineage/common/provider/dbt.py#L467

+ +

The first step should be to add SQLServer or Dremio to the dataset naming schema here https://github.com/OpenLineage/OpenLineage/blob/main/spec/Naming.md

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/integration/common/openlineage/common/provider/dbt.py | dbt.py>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/spec/Naming.md | Naming.md>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Yannick Endrion + (yannick.endrion@gmail.com) +
+
2021-10-04 16:22:59
 +
+

*Thread Reply:* Thank you @Maciej Obuchowski, +I tried to give it a try but without success yet. Not sure where I am suppose to add the sqlserver naming schema... +If you have any documentation that I could read I would be glad =) +Many thanks

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-10-07 15:13:43
 +
+

*Thread Reply:* This would be adding a paragraph similar to this one: https://github.com/OpenLineage/OpenLineage/blob/main/spec/Naming.md#snowflake

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/spec/Naming.md | Naming.md>
+
+ <p><strong>Naming</strong></p> + +<p>We define the unique name strategy per resource to ensure it is followed uniformly independently from who is producing metadata and we can connect lineage from various sources.</p> + +<p>Both Jobs and Datasets are in their own namespaces. Job namespaces are related to their scheduler. The namespace for a dataset is the unique name for its datasource.</p> + +<p><strong>Datasets</strong></p> + +<p>The namespace and name of a datasource can be combined to form a URI (scheme:[//authority]path)</p> + +<p>• Namespace = scheme:[//authority] (the datasource) +• Name = path (the datasets)</p> + +<p><strong>Data warehouses/data bases</strong></p> + +<p>Datasets are called tables. Tables are organised in databases and schemas.</p> + +<p><strong>Postgres:</strong></p> + +<p>Datasource hierarchy:</p> + +<p>• Host +• Port</p> + +<p>Naming hierarchy:</p> + +<p>• Database +• Schema +• Table</p> + +<p>Identifier:</p> + +<p>• Namespace: postgres://{host}:{port} of the service instance. + • Scheme = postgres + • Authority = {host}:{port} +• Unique name: {database}.{schema}.{table} + • URI = postgres://{host}:{port}/{database}.{schema}.{table}</p> + +<p><strong>Snowflake</strong></p> + +<p>See: <a href="https://docs.snowflake.com/en/sql-reference/identifiers.html">Object Identifiers — Snowflake Documentation</a></p> + +<p>Datasource hierarchy:</p> + +<p>• account name</p> + +<p>Naming hierarchy:</p> + +<p>• Database: {database name} =&gt; unique across the account +• Schema: {schema name} =&gt; unique within the database +• Table: {table name} =&gt; unique within the schema</p> + +<p>Identifier:</p> + +<p>• Namespace: snowflake://{account name} + • Scheme = snowflake + • Authority = {account name} +• Name: {database}.{schema}.{table} + • URI = snowflake://{account name}/{database}.{schema}.{table}</p> + +<p><strong>BigQuery</strong></p> + +<p>See: <br /> +<a href="https://cloud.google.com/resource-manager/docs/creating-managing-projects|Creating and managing projects | Resource Manager Documentation">https://cloud.google.com/resource-manager/docs/creating-managing-projects|Creating and managing projects | Resource Manager Documentation</a> <br /> +<a href="https://cloud.google.com/bigquery/docs/datasets-intro|Introduction to datasets | BigQuery">https://cloud.google.com/bigquery/docs/datasets-intro|Introduction to datasets | BigQuery</a> <br /> +<a href="https://cloud.google.com/bigquery/docs/tables-intro|Introduction to tables | BigQuery">https://cloud.google.com/bigquery/docs/tables-intro|Introduction to tables | BigQuery</a></p> + +<p>Datasource hierarchy:</p> + +<p>• bigquery</p> + +<p>Naming hierarchy:</p> + +<p>• Project Name: {project name} =&gt; is not unique +• Project number: {project number} =&gt; numeric: is unique across google cloud +• Project ID: {project id} =&gt; readable: is unique across google cloud +• dataset: {dataset name} =&gt; is unique within a project +• table: {table name} =&gt; is unique within a dataset</p> + +<p>Identifier :</p> + +<p>• Namespace: bigquery + • Scheme = bigquery + • Authority = +• Unique name: {project id}.{dataset name}.{table name} + • URI = bigquery:{project id}.{schema}.{table}</p> + +<p><strong>Distributed file systems/blob stores</strong> +<strong>GCS</strong></p> + +<p>Datasource hierarchy: none, naming is global</p> + +<p>Naming hierarchy:</p> + +<p>• bucket name =&gt; globally unique +• Path</p> + +<p>Identifier :</p> + +<p>• Namespace: gs://{bucket name} + • Scheme = gs + • Authority = {bucket name} +• Unique name: {path} + • URI = gs://{bucket name}{path}</p> + +<p><strong>S3</strong></p> + +<p>Naming hierarchy:</p> + +<p>• bucket name =&gt; globally unique +• Path</p> + +<p>Identifier :</p> + +<p>• Namespace: s3://{bucket name} + • Scheme = s3 + • Authority = {bucket name} +• Unique name: {path} + • URI = s3://{bucket name}{path}</p> + +<p><strong>HDFS</strong></p> + +<p>Naming hierarchy:</p> + +<p>• Namenode: host + port +• Path</p> + +<p>Identifier :</p> + +<p>• Namespace: hdfs://{namenode host}:{namenode port} + • Scheme = hdfs + • Authority = {namenode host}:{namenode port} +• Unique name: {path} + • URI = hdfs://{namenode host}:{namenode port}{path}</p> + +<p><strong>Jobs</strong> +<strong>Context</strong></p> + +<p>A <code>Job</code> is a recurring data transformation with Inputs and outputs. Each execution is captured as a <code>Run</code> with corresponding metadata. <br /> +A <code>Run</code> event identifies the <code>Job</code> it is an instance of by providing the job’s unique identifier. <br /> +The <code>Job</code> identifier is composed of a <code>Namespace</code> and a <code>Name</code>. The job name is unique within that namespace.</p> + +<p>The core property we want to identify about a <code>Job</code> is how it changes over time. Different schedules of the same logic applied to different datasets (possibly with different parameters) are different jobs. The notion of a <code>job</code> is tied to a recurring schedule with specific inputs and outputs. It could be an incremental update or a full reprocess or even a streaming job.</p> + +<p>If the same code artifact (for example a spark jar or a templated SQL query) is used in the context of different schedules with different input or outputs then they are different jobs. <br /> +We are interested first in how they affect the datasets they produce.</p> + +<p><strong>Job Namespace and constructing job names</strong></p> + +<p>Jobs have a <code>name</code> that is unique to them in their <code>namespace</code> by construction.</p> + +<p>The Namespace is the root of the naming hierarchy. The job name is constructed to identify the job within that namespace.</p> + +<p>Example:</p> + +<p>• Airflow: + • Namespace: the namespace is assigned to the airflow instance. Ex: airflow-staging, airflow-prod + • Job: each task in a DAG is a job. name: {dag name}.{task name} +• Spark: + • Namespace: the namespace is provided as a configuration parameter as in airflow. If there's a parent job, we use the same namespace, otherwise it is provided by configuration. + • Spark app job name: the spark.app.name + • Spark action job name: {spark.app.name}.{node.name}</p> + +<p><strong>Parent job run: a nested hierarchy of Jobs</strong></p> + +<p>It is often the case that jobs are part of a nested hierarchy. <br /> +For example an Airflow DAG contains tasks. An instance of the DAG is finished when all of the tasks are finished. Similarly a Spark job can spawn multiple actions each of them running independently. Additionally, a Spark job can be launched by an Airflow task within a DAG.</p> + +<p>Since what we care about is identifying the job as rooted in a recurring schedule, we want to capture that connection and make sure that we treat the same application logic triggered at different schedules as different jobs. For example: if an Airflow DAG runs individual tasks per partition (for example market segments) using the same underlying job logic, they will be tracked as separate jobs.</p> + +<p>To capture this, a run event provides <a href="https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.json#L282-L331">a <code>ParentRun</code> facet</a>, referring to the parent <code>Job</code> and <code>Run</code>. This allows tracking a recurring job from the root of the schedule it is running for. <br /> +If there's a parent job, we use the same namespace, otherwise it is provided by configuration.</p> + +<p>Example:</p> + +<pre><code>{ + "run": { + "runId": "run_uuid" + }, + "job": { + "namespace": "job_namespace", + "name": "job_name" + } +} + +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-10-07 15:14:30
 +
+

*Thread Reply:* Snowflake +See: Object Identifiers — Snowflake Documentation +Datasource hierarchy: +• account name +Naming hierarchy: +• Database: {database name} => unique across the account +• Schema: {schema name} => unique within the database +• Table: {table name} => unique within the schema +Identifier: +• Namespace: snowflake://{account name} + ◦ Scheme = snowflake + ◦ Authority = {account name} +• Name: {database}.{schema}.{table} + ◦ URI = snowflake://{account name}/{database}.{schema}.{table}

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marty Pitt + (martypitt@vyne.co) +
+
2021-09-24 06:53:05
 +
+

Hi all. I'm the Founder / CTO of a data discovery & transformation platform that captures very rich lineage information. We're interested in exposing / making our lineage data consumable via open standards, which is what lead me to this project. A couple of questions:

+ +

A) Am I right in considering that's the goal of this project? +B) Are you also considering provedance as well as lineage? +C) What's a good starting point to understand the models we should be exposing our data in, to make it consumable?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marty Pitt + (martypitt@vyne.co) +
+
2021-09-24 07:06:20
 +
+

*Thread Reply:* For clarity on the provedance vs lineage point (in case I'm using those terms incorrectly...)

+ +

Our platform performs automated enrichment and processing of data. In doing so, we often make calls to functions or out to other data services (such as APIs, or SELECTs against databases). We capture the inputs that pass to these, along with the outputs. (And, if the input is derived from other outputs, we capture the full chain, right back to the root).

+ +

That's the kinda stuff our customers are really interested in, and we feel like there's value in making is consumable.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-09-24 08:47:35
 +
+

*Thread Reply:* Not sure I understand you right, but are you interested in tracking individual API calls, and for example, values of some parameters passed for one call?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-09-24 08:51:16
 +
+

*Thread Reply:* I guess that's not in OpenLineage scope, as we're interested more in tracking metadata for whole datasets. But I might be wrong, some other people might chime in.

+ +

We could of course model this situation, but that would capture for example schema of those parameters. Not their values.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-09-24 08:52:16
 +
+

*Thread Reply:* I think this might be better suited for https://opentelemetry.io/

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marty Pitt + (martypitt@vyne.co) +
+
2021-09-24 10:55:54
 +
+

*Thread Reply:* Kinda, but not really. Telemetery data is metadata about the API calls. We have that, but it's not interesting to our customers. It's the metadata about the data that Vyne provides that we want to expose.

+ +

Our customers use Vyne to fetch data from lots of different sources. Eg:

+ +

> "Whenever a trade is booked, calculate it's compliance against these regulations, to report to the regulators". +or

+ +

> "Whenever a customer buys a $thing, capture the transaction data, client data, and account data, and store it in this table." +Providing answers to those questions involves fetching and transforming data, before storing it, or outputting it. We capture all that data, on a per-attribute basis, so we can answer the question "how did we get this value?" That's the lineage information we want to publish.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2021-09-30 15:10:51
 +
+

*Thread Reply:* The core OpenLineage model is documented at https://github.com/OpenLineage/OpenLineage/#core-model . The model is really focused on Jobs and Datasets. Jobs have Runs which have start and end times (typically scheduled start/end times as well) and read from and/or write to the target datasets. If your transformation chain fits within that model, then I think you can definitely record and share the lineage information with your customers. The existing implementations are all focused on batch data access, though streaming should be possible to capture as well

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Drew Bittenbender + (drew@salt.io) +
+
2021-09-29 11:10:29
 +
+

Hello. I am trying the openlineage-airflow integration with Marquez as the backend and have 3 questions.

+ +
  1. Does it only work for PostgresOperators?
  2. Which is the recommended integration: marquez-airflow or openlineage-airflow
  3. How do you enable more detailed logging? I tried OPENLINEAGELOGLEVEL and MARQUEZLOGLEVEL and neither seemed to affect logging. I assume this is logged to the airflow worker
    4. +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Faouzi + (faouzi@dataroots.io) +
+
2021-09-29 13:46:59
 +
+

*Thread Reply:* Hello @Drew Bittenbender!

+ +

For your two first questions:

+ +

• Yes right now only the PostgresOperator is integrated. I learnt it the hard way ^_^. Spent hours trying with MySQL. There were attempts to integrate with MySQL actually. If engineers do not integrate it I will allocate myself some time to try to implement other airflow db operators. +• Use the openlineage one. It is the recommended approach now.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Drew Bittenbender + (drew@salt.io) +
+
2021-09-29 13:49:41
 +
+

*Thread Reply:* Thank you @Faouzi. Is there any documentation/best practices to write your own extractor, or is it "read the code"? We use the Python, Docker and SSH operators a lot. Maybe those don't fit into the lineage paradigm well, but want to give it a shot

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Faouzi + (faouzi@dataroots.io) +
+
2021-09-29 13:52:16
 +
+

*Thread Reply:* To the best of my knowledge there is no documentation to guide through the design of your own extractor. So yes we need to read the code. Here a link where you can see how they did for postgre extractor and others. https://github.com/OpenLineage/OpenLineage/tree/main/integration/airflow/openlineage/airflow/extractors

+ + + +
+ 👍 Drew Bittenbender +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-09-30 05:08:53
 +
+

*Thread Reply:* I think in case of "bring your own code" operators like Python or Docker ones, it might be better to use lineage_run_id macro and use openlineage-python library inside, instead of implementing extractor.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2021-09-30 15:14:47
 +
+

*Thread Reply:* I think @Maciej Obuchowski is right here. The airflow integration will create the parent jobs, but to get the dataset input/output links, it's best to do that directly from the python/docker scripts. If you report the parent run id, Marquez will link the jobs together correctly

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-10-07 15:09:55
 +
+

*Thread Reply:* To clarify on what airflow operators are supported out of the box: +• postgres +• bigquery +• snowflake +• Great expectations (with extra config) +See: https://github.com/OpenLineage/OpenLineage/blob/3a1ccbd854bbf202bbe6437bf81786cb01[…]ntegration/airflow/openlineage/airflow/extractors/extractors.py +Mysql is not at the moment. We should track it as an issue

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/3a1ccbd854bbf202bbe6437bf81786cb01ed4d01/integration/airflow/openlineage/airflow/extractors/extractors.py | extractors.py>
+
+ <pre><code>_extractors = [ + PostgresExtractor, + BigQueryExtractor, + GreatExpectationsExtractor, + SnowflakeExtractor +] +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Yuki Tannai + (tannai-yuki@dmm.com) +
+
2021-09-30 09:21:35
 +
+

Hi there! +I’m trying to enhance the lineage functionality of a data infrastructure I’m working on. +All of the tools I found only visualize the relationships between tables before and after the transformation, but the DataHub RFC discusses Field Level Lineage, which I thought was close to the functionality I was looking for. +Does OpenLineage support the same functionality? +https://datahubproject.io/docs/rfc/active/1841-lineage/field_level_lineage/

+
+
datahubproject.io
+ + + +
RFC - Field Level Lineage | DataHub
+
+ - Start Date: 2020-08-28 +
+ + + + + + +
Original URL: https://datahubproject.io/docs/rfc/active/1841-lineage/field_level_lineage/
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-10-07 15:03:40
 +
+

*Thread Reply:* OpenLineage doesn’t have field level lineage yet. Here is the proposal for adding it: https://github.com/OpenLineage/OpenLineage/issues/148

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/issues/148|#148 [PROPOSAL] column level lineage facet>
+
+ <p><strong>Purpose:</strong> <br /> +For transformations like SQL queries (but also in other cases), we can extract column level lineage. This allows answering questions like ‘which root input columns are used to construct column x?’. We need to be able to capture this information in the OpenLineage model</p> + +<p><strong>Proposed implementation</strong> <br /> +We propose to introduce a new dataset facet: <br /> +example:</p> + +<pre><code>{ + "eventType": "COMPLETE", + "eventTime": "2020-12-28T20:52:00.001+10:00", + "run" : { + "runId": "uuid" + }, + "job": { + "namespace": "scheduler", + "name": "myjob", + "facets": { + "sql": { + "query": "Insert into outputTable from select ** from inputTable" + } + } + }, + "inputs": [ + { + "namespace": "N1", + "name": "inputTable", + "facets": { + "schema": { + "fields": [ {"name": "col_a", "type": "VARCHAR"}, {"name": "col_b", "type": "int"}] + } + } + } + ], + "outputs": [ + { + "namespace": "N2", + "name": "outputTable", + "facets": { + "schema": { + "fields": [ {"name": "col_a", "type": "VARCHAR"}, {"name": "col_b", "type": "int"}] + }, + "columnLineage": { + "fields": { + "col_a": [ { "namespace": "N1", "name": "inputTable", "field": "col_a"} ], + "col_b": [ { "namespace": "N1", "name": "inputTable", "field": "col_b" } ] + } + } + } + } + ] +} +</code></pre> + +<p>Schema:</p> + +<pre><code> "columnLineage": { + "type": "object", + "properties": { + "fields": { + "type": "array", + "items": { + "type": "object", + "additionalProperties": { + "type": "object", + "properties": { + "namespace": { + "type": "string", + "description": "the input dataset namespace" + }, + "name": { + "type": "string", + "description": "the input dataset name" + }, + "field": { + "type": "string", + "description": "the input field" + } + }, + "required": [ + "namespace", "name" + ] + } + } + } + } +</code></pre> +
+ +
+
Labels
+ proposal +
+ +
+
Comments
+ 2 +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+ 👀 Yuki Tannai, Ricardo Gaspar +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-10-07 15:04:36
 +
+

*Thread Reply:* Those two specs look compatible, so Datahub should be able to consume this lineage metadata in the future

+ + + +
+ 👍 Yuki Tannai +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
павел клопотюк + (klopotuk@gmail.com) +
+
2021-10-04 14:27:24
 +
+

Hello, everyone. I'm trying to work with OL and Airflow 2.1.4 and it doesn't work. I found that OL is supported for Airflow 1.10.12++. Does it support Airflow 2.X.Y?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2021-10-04 15:38:47
 +
+

*Thread Reply:* Hi! Airflow 2.x is currently in development - you can follow along with the progress here: +https://github.com/OpenLineage/OpenLineage/issues/205

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/issues/205|#205 Airflow 2.0 support>
+
+ <p>☑︎ Test suite for 2.0 +☑︎ Working LineageBackend PoC that passes existing tests +☑︎ <a href="https://github.com/apache/airflow/issues/17984">Issue in Airflow repo to create mechanism for reporting failures</a> +☑︎ <a href="https://github.com/OpenLineage/OpenLineage/issues/242">Way to add custom Extractors</a> +☑︎ <a href="https://github.com/apache/airflow/pull/18470">Airflow PR</a> +☑︎ Check if it's feasible to have both Airflows supported in one package +☐ Implement support in existing package</p> +
+ +
+
Assignees
+ mobuchowski +
+ +
+
Comments
+ 5 +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
павел клопотюк + (klopotuk@gmail.com) +
+
2021-10-05 03:01:54
 +
+

*Thread Reply:* Thank you for your reply!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-10-07 15:02:23
 +
+

*Thread Reply:* There should be a first version of Airflow 2.X support soon: https://github.com/OpenLineage/OpenLineage/pull/305 +We’re labelling it experimental because the config step might change as discussion in the airflow github evolve. It will track succesful jobs in its current state.

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/pull/305|#305 [INTEGRATION][airflow] integration working on both Airflow 2 and 1.10>
+
+ <p>Implement support for both Airflow versions on one python package.</p> + +<p>Airflow 2's support is done via Airflow's <code>LineageBackend</code> and is currently experimental. It has limitations - can't track job fails, and job starts are only tracked on job end.</p> + +<p>The end-user has only to specify <code>openlineage.airflow.backend.OpenLineageBackend</code> as their <code>LineageBackend</code> in the Airflow config or via env variable.</p> + +<p>To make sure this integration works on both versions, some helpers had to be implemented, including ones that correctly import based on version, like <code>safe_import_airflow</code>. Also, unit and integration tests are running on both versions of Airflow now.</p> + +<p>Closes <a href="https://github.com/OpenLineage/OpenLineage/issues/205">#205</a> <a href="https://github.com/OpenLineage/OpenLineage/issues/240">#240</a> <a href="https://github.com/OpenLineage/OpenLineage/issues/126">#126</a></p> + +<p>Signed-off-by: Maciej Obuchowski <a href="mailto:maciej.obuchowski@getindata.com">maciej.obuchowski@getindata.com</a></p> +
+ +
+
Comments
+ 2 +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
SAM + (skhettri@gmail.com) +
+
2021-10-04 23:14:26
 +
+

Hi All, I’m working on openlineage-dbt integration with Marquez as backend. I want to integrate OL with DBT cloud, would you please help to provide steps that I need to follow?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-10-05 04:18:42
 +
+

*Thread Reply:* Take a look at this: https://docs.getdbt.com/docs/dbt-cloud/dbt-cloud-api/metadata/metadata-overview

+
+
docs.getdbt.com
+ + + +
Metadata API Overview | dbt Docs
+
+ Every time that dbt Cloud runs a dbt project, it generates metadata which pertains to the accuracy, recency, configuration, and structure of the views and tables in the warehouse. dbt Cloud serves a GraphQL API which supports arbitrary queries over this metadata; the endpoint for this API is <https://metadata.cloud.getdbt.com/graphql>. This API is an incredibly rich resource for evaluating data health longitudinally or at a point in time. +
+ + + + + + + + +
Original URL: https://docs.getdbt.com/docs/dbt-cloud/dbt-cloud-api/metadata/metadata-overview
+ + + +
+ + + +
+ ✅ SAM +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-10-07 14:58:24
 +
+

*Thread Reply:* @SAM Let us know of your progress.

+ + + +
+ 👍 SAM +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ale + (alessandro.lollo@gmail.com) +
+
2021-10-05 16:23:41
 +
+

Hey folks 😊 +I’m trying to run dbt-ol with Redshift target, but I get the following error message +Traceback (most recent call last): + File "/usr/local/bin/dbt-ol", line 61, in &lt;module&gt; + main() + File "/usr/local/bin/dbt-ol", line 54, in main + events = processor.parse().events() + File "/usr/local/lib/python3.8/site-packages/openlineage/common/provider/dbt.py", line 97, in parse + self.extract_dataset_namespace(profile) + File "/usr/local/lib/python3.8/site-packages/openlineage/common/provider/dbt.py", line 368, in extract_dataset_namespace + self.dataset_namespace = self.extract_namespace(profile) + File "/usr/local/lib/python3.8/site-packages/openlineage/common/provider/dbt.py", line 382, in extract_namespace + raise NotImplementedError( +NotImplementedError: Only 'snowflake' and 'bigquery' adapters are supported right now. Passed redshift +I know that Redshift is not the best cloud DWH we can use… 😅 +But, still….do you have any plan to support it? +Thanks!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-10-05 16:41:30
 +
+

*Thread Reply:* Hey, can you create ticket in OpenLineage repository? FWIW Redshift is very similar to postgres, so supporting it won't be hard.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ale + (alessandro.lollo@gmail.com) +
+
2021-10-05 16:43:39
 +
+

*Thread Reply:* Hey @Maciej Obuchowski 😊 +Yep, will do now! Thanks!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ale + (alessandro.lollo@gmail.com) +
+
2021-10-05 16:46:26
 +
+

*Thread Reply:* Well...will do tomorrow morning 😅

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ale + (alessandro.lollo@gmail.com) +
+
2021-10-06 03:03:16
 +
+

*Thread Reply:* Here’s the issue: https://github.com/OpenLineage/OpenLineage/issues/318

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/issues/318|#318 Add Redshift support to dbt integration>
+
+ <p>As of <code>dbt-ol==0.2.2</code>, it is not possible to generate lineage if the target DWH is Redshift. <br /> +Try running <code>dbt-ol</code> and you get the following error:</p> + +<pre><code>Traceback (most recent call last): + File "/usr/local/bin/dbt-ol", line 61, in &amp;lt;module&amp;gt; + main() + File "/usr/local/bin/dbt-ol", line 54, in main + events = processor.parse().events() + File "/usr/local/lib/python3.8/site-packages/openlineage/common/provider/dbt.py", line 97, in parse + self.extract_dataset_namespace(profile) + File "/usr/local/lib/python3.8/site-packages/openlineage/common/provider/dbt.py", line 368, in extract_dataset_namespace + self.dataset_namespace = self.extract_namespace(profile) + File "/usr/local/lib/python3.8/site-packages/openlineage/common/provider/dbt.py", line 382, in extract_namespace + raise NotImplementedError( +NotImplementedError: Only 'snowflake' and 'bigquery' adapters are supported right now. Passed redshift + +</code></pre> + +<p>As <a href="https://github.com/mobuchowski">@mobuchowski</a> suggested <a href="https://openlineage.slack.com/archives/C01CK9T7HKR/p1633465421127100">here</a>, I'm opening this issue as it should be not hard to support Redshift since it is very similar to Postgres.</p> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-10-07 14:51:08
 +
+

*Thread Reply:* Thanks a lot. I pulled it in the current project.

+ + + +
+ 👍 ale +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ale + (alessandro.lollo@gmail.com) +
+
2021-10-08 05:48:28
 +
+

*Thread Reply:* @Julien Le Dem @Maciej Obuchowski I’m not familiar with dbt-ol codebase, but I’m willing to help on this if you guys can give me a bit of guidance 😅

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-10-08 05:53:05
 +
+

*Thread Reply:* @ale can you help us define naming schema for redshift, as we have for other databases? https://github.com/OpenLineage/OpenLineage/blob/main/spec/Naming.md

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ale + (alessandro.lollo@gmail.com) +
+
2021-10-08 05:53:21
 +
+

*Thread Reply:* Sure!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ale + (alessandro.lollo@gmail.com) +
+
2021-10-08 05:54:21
 +
+

*Thread Reply:* will work on this today and I’ll try to submit a PR by EOD

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ale + (alessandro.lollo@gmail.com) +
+
2021-10-08 06:36:12
 +
+

*Thread Reply:* There you go https://github.com/OpenLineage/OpenLineage/pull/324

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/pull/324|#324 Add redshift naming spec>
+
+ <p>This PR adds Redshift naming specs.</p> + +<p>Given that Redshift is based on Postgres, the proposed naming specs are almost identical to Postgres naming specs.</p> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-10-08 06:39:35
 +
+

*Thread Reply:* Host would be something like +examplecluster.&lt;XXXXXXXXXXXX&gt;.<a href="http://us-west-2.redshift.amazonaws.com">us-west-2.redshift.amazonaws.com</a> +right?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ale + (alessandro.lollo@gmail.com) +
+
2021-10-08 07:13:51
 +
+

*Thread Reply:* Yep, let me update the PR

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ale + (alessandro.lollo@gmail.com) +
+
2021-10-08 07:27:42
 +
+

*Thread Reply:* Done

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-10-08 07:31:40
 +
+

*Thread Reply:* 🙌

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-10-08 07:35:30
 +
+

*Thread Reply:* If you want to look at dbt integration itself, there are two things:

+ +

We need to determine how Redshift adapter reports metrics https://github.com/OpenLineage/OpenLineage/blob/610a687bf69df2b52ec4ac4da80b4a05580e8d32/integration/common/openlineage/common/provider/dbt.py#L412

+ +

And how we can create namespace and job name based on the job naming schema that you created: +https://github.com/OpenLineage/OpenLineage/blob/610a687bf69df2b52ec4ac4da80b4a05580e8d32/integration/common/openlineage/common/provider/dbt.py#L512

+ +

One thing how to get this info is to run the dbt yourself and look at resulting metadata files - in target dir of the dbt directory

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/610a687bf69df2b52ec4ac4da80b4a05580e8d32/integration/common/openlineage/common/provider/dbt.py | dbt.py>
+
+ <pre><code> def extract_namespace(self, profile: Dict) -&amp;gt; str: +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ale + (alessandro.lollo@gmail.com) +
+
2021-10-08 08:33:31
 +
+

*Thread Reply:* I figured out how to generate the namespace. +But I can’t understand which of the JSON files is inspected for metrics. Is it run_results.json ?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-10-08 09:48:50
 +
+

*Thread Reply:* yes, run_results.json - it's different in bigquery and snowflake, so I presume it's different in redshift too

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ale + (alessandro.lollo@gmail.com) +
+
2021-10-08 11:02:32
 +
+

*Thread Reply:* Ok thanks!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ale + (alessandro.lollo@gmail.com) +
+
2021-10-08 11:11:57
 +
+

*Thread Reply:* Should be stats:rows:value

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ale + (alessandro.lollo@gmail.com) +
+
2021-10-08 11:19:59
 +
+

*Thread Reply:* Regarding namespace: if env_var is used in profiles.yml , how is this handled now?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-10-08 11:44:50
 +
+

*Thread Reply:* Well, it isn't. This is relevant only if you passed cluster hostname this way, right?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ale + (alessandro.lollo@gmail.com) +
+
2021-10-08 11:53:52
 +
+

*Thread Reply:* Exactly

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ale + (alessandro.lollo@gmail.com) +
+
2021-10-11 07:10:38
 +
+

*Thread Reply:* If you think it make sense, I can submit a PR to handle dbt profile with env_var

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-10-11 07:18:01
 +
+

*Thread Reply:* Do you want to run jinja on the dbt profile?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-10-11 07:20:18
 +
+

*Thread Reply:* Theoretically, we'd need to run it also on dbt_project.yml , but we only take target path and profile name from it.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ale + (alessandro.lollo@gmail.com) +
+
2021-10-11 07:20:32
 +
+

*Thread Reply:* The env_var syntax in the profile is quite simple, I was thinking of extracting the env var name using re and then retrieving the value from os

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-10-11 07:23:59
 +
+

*Thread Reply:* It would work, but we can actually use jinja - if you're using dbt, it's already included. +The method is pretty simple: +``` @contextmember + @staticmethod + def envvar(var: str, default: Optional[str] = None) -> str: + """The envvar() function. Return the environment variable named 'var'. + If there is no such environment variable set, return the default.

+ +
    If the default is None, raise an exception for an undefined variable.
+    """
+    if var in os.environ:
+        return os.environ[var]
+    elif default is not None:
+        return default
+    else:
+        msg = f"Env var required but not provided: '{var}'"
+        undefined_error(msg)```
+
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ale + (alessandro.lollo@gmail.com) +
+
2021-10-11 07:25:07
 +
+

*Thread Reply:* Oh cool! +I will definitely use this one!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-10-11 07:25:09
 +
+

*Thread Reply:* We'd be sure that our implementation matches dbt's one, right? Also, you'd support default method for free

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ale + (alessandro.lollo@gmail.com) +
+
2021-10-11 07:26:34
 +
+

*Thread Reply:* So this env_varmethod is defined in dbt and not in OpenLineage codebase, right?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-10-11 07:27:01
 +
+

*Thread Reply:* yes

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-10-11 07:27:14
 +
+

*Thread Reply:* dbt is on Apache license 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ale + (alessandro.lollo@gmail.com) +
+
2021-10-11 07:28:06
 +
+

*Thread Reply:* Should we import dbt package and use the method or should we just copy/paste the method inside OpenLineage codebase?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ale + (alessandro.lollo@gmail.com) +
+
2021-10-11 07:28:28
 +
+

*Thread Reply:* I’m asking for guidance here 😊

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-10-11 07:34:44
 +
+

*Thread Reply:* I think we should just do basic jinja template rendering in our code like in the quick example: https://realpython.com/primer-on-jinja-templating/#quick-examples

+ +

just with the env_var method passed to the render method 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-10-11 07:37:05
 +
+

*Thread Reply:* basically, here in the code we should read the file, do the jinja render, and load yaml from string instead of straight from file +https://github.com/OpenLineage/OpenLineage/blob/610a687bf69df2b52ec4ac4da80b4a05580e8d32/integration/common/openlineage/common/provider/dbt.py#L176

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/610a687bf69df2b52ec4ac4da80b4a05580e8d32/integration/common/openlineage/common/provider/dbt.py | dbt.py>
+
+ <pre><code> def load_yaml(path: str) -&amp;gt; Dict: +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ale + (alessandro.lollo@gmail.com) +
+
2021-10-11 07:38:53
 +
+

*Thread Reply:* ok, got it. +Will try to implement following your suggestions. +Thanks @Maciej Obuchowski 🙌

+ + + +
+ 🙌 Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ale + (alessandro.lollo@gmail.com) +
+
2021-10-11 08:36:13
 +
+

*Thread Reply:* We need to:

+ +
  1. load the template profile from the profile.yml
  2. replace any env vars we found +For the first step, we can use jinja2.Template +However, to replace the env vars we find, we have to actually search for those env vars… 🤔
    3. +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-10-11 08:43:06
 +
+

*Thread Reply:* The dbt method implements that: +``` @contextmember + @staticmethod + def envvar(var: str, default: Optional[str] = None) -> str: + """The envvar() function. Return the environment variable named 'var'. + If there is no such environment variable set, return the default.

+ +
    If the default is None, raise an exception for an undefined variable.
+    """
+    if var in os.environ:
+        return os.environ[var]
+    elif default is not None:
+        return default
+    else:
+        msg = f"Env var required but not provided: '{var}'"
+        undefined_error(msg)```
+
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ale + (alessandro.lollo@gmail.com) +
+
2021-10-11 08:45:54
 +
+

*Thread Reply:* Ok, but I need to pass var to the env_var method. +And to pass the var value, I need to look into the loaded Template and search for env var names…

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-10-11 08:46:54
 +
+

*Thread Reply:* that's what jinja does - you're passing function to jinja render, and it's calling it itself

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-10-11 08:47:45
 +
+

*Thread Reply:* you can try the quick example from here, but just pass the env_var method (slightly adjusted - as a standalone function and without undefined error) and call it inside the template: https://realpython.com/primer-on-jinja-templating/#quick-examples

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ale + (alessandro.lollo@gmail.com) +
+
2021-10-11 08:51:19
 +
+

*Thread Reply:* Ok, will try

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ale + (alessandro.lollo@gmail.com) +
+
2021-10-11 09:37:49
 +
+

*Thread Reply:* I’m trying to run +pip install -e ".[dev]" +so that I can test my changes, but I get +ERROR: Could not find a version that satisfies the requirement openlineage-integration-common[dbt]==0.2.3 (from openlineage-dbt[dev]) (from versions: 0.0.1rc7, 0.0.1rc8, 0.0.1, 0.1.0rc5, 0.1.0, 0.2.0, 0.2.1, 0.2.2) +ERROR: No matching distribution found for openlineage-integration-common[dbt]==0.2.3 +I don’t understand what I’m doing wrong…

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-10-11 09:41:47
 +
+

*Thread Reply:* can you try installing it manually?

+ +

pip install openlineage-integration-common[dbt]==0.2.3

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-10-11 09:42:13
 +
+

*Thread Reply:* I mean, it exists in pypi: https://pypi.org/project/openlineage-integration-common/#files

+
+
PyPI
+ + + +
openlineage-integration-common
+
+ OpenLineage common python library for integrations +
+ + + + + + + + +
Original URL: https://pypi.org/project/openlineage-integration-common/#files
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ale + (alessandro.lollo@gmail.com) +
+
2021-10-11 09:44:57
 +
+

*Thread Reply:* Yep, maybe it’s our internal Pypi repo which is not synced. +Installing from the public pypi resolved the issue

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ale + (alessandro.lollo@gmail.com) +
+
2021-10-11 12:04:55
 +
+

*Thread Reply:* Can;’t seem to make env_var working as the render method of a Template 😅

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-10-11 12:57:07
 +
+

*Thread Reply:* try this:

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-10-11 12:57:09
 +
+

*Thread Reply:* ```import os +from typing import Optional +from jinja2 import Template

+ +

def envvar(var: str, default: Optional[str] = None) -> str: + """The envvar() function. Return the environment variable named 'var'. + If there is no such environment variable set, return the default.

+ +
If the default is None, raise an exception for an undefined variable.
+"""
+if var in os.environ:
+    return os.environ[var]
+elif default is not None:
+    return default
+else:
+    msg = f"Env var required but not provided: '{var}'"
+    raise Exception("")
+
+ +

if name == 'main': + t = Template("Hello {{ envvar('ENVVAR') }}!") + print(t.render(envvar=envvar))```

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-10-11 12:57:42
 +
+

*Thread Reply:* works for me: +mobuchowski@thinkpad [18:57:14] [~] +-&gt; % ENV_VAR=world python jinja_example.py +Hello world!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ale + (alessandro.lollo@gmail.com) +
+
2021-10-11 16:59:13
 +
+

*Thread Reply:* Finally 😅 +https://github.com/OpenLineage/OpenLineage/pull/328

+ +

There are minimal tests for Redshift and env vars. +Feedbacks and suggestions are welcome!

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/pull/328|#328 - Added support for Redshift profile>
+
+ <p>• Added support for Redshift profile +• Added support for handling env vars in profiles.yml</p> + +<p>Please note that I had to enclose <code>{{ config(...) }}</code> between <code>raw</code> Jinja tags, otherwise it complains about an unexpected '.'</p> + +<p>Signed-off-by: Alessandro Lollo <a href="mailto:alessandro.lollo@cloudacademy.com">alessandro.lollo@cloudacademy.com</a></p> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ale + (alessandro.lollo@gmail.com) +
+
2021-10-12 03:10:45
 +
+

*Thread Reply:* Hi @Maciej Obuchowski 😊 +Regarding this comment https://github.com/OpenLineage/OpenLineage/pull/328#discussion_r726586564

+ +

How can we distinguish between snowflake, bigquery and redshift in this method?

+ +

A simple, but not very clean solution, would be to split this +bytes = get_from_multiple_chains( + node.catalog_node, + [ + ['stats', 'num_bytes', 'value'], # bigquery + ['stats', 'bytes', 'value'], # snowflake + ['stats', 'size', 'value'] # redshift (Note: size = count of 1MB blocks) + ] + ) +into two pieces, one checking for snowflake and bigquery and the other checking for redshift.

+ +

A better solution would be to have the profile type inside method node_to_output_dataset , but I’m struggling understanding how to do that

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-10-12 05:35:00
 +
+

*Thread Reply:* Well, why not do something like

+ +

```bytes = getfrommultiple_chains(... rest of stuff)

+ +

if adapter == 'redshift': + bytes = 10241024```

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-10-12 05:36:49
 +
+

*Thread Reply:* we can store adapter type in the class

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-10-12 05:38:47
 +
+

*Thread Reply:* well, I've looked at last commit and that's exactly what you did 👍

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-10-12 05:40:35
 +
+

*Thread Reply:* Now, have you tested your branch on real redshift cluster? I don't think we 100% need automated tests for that now, but would be nice to have confirmation that it works.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ale + (alessandro.lollo@gmail.com) +
+
2021-10-12 06:35:04
 +
+

*Thread Reply:* Not yet, but I'll try to do that this afternoon. +Need to figure out how to build the lib locally, then I can use it to test with Redshift

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-10-12 06:40:58
 +
+

*Thread Reply:* I think pip install -e .[dbt] in common directory should be enough

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ale + (alessandro.lollo@gmail.com) +
+
2021-10-12 09:29:13
 +
+

*Thread Reply:* I was able to run my local branch with my Redshift cluster and metadata is pushed to Marquez. +However, I’m not sure about the namespace . +I also see exceptions in Marquez logs

+ +
+
+ + + + + + +
+
+
+ + + + + + +
+
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-10-12 09:33:26
 +
+

*Thread Reply:* namespace: well, if it matches what you put into your profile, there's not much we can do. I don't understand why you connect to redshift via host, maybe this is related to IAM?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-10-12 09:44:17
 +
+

*Thread Reply:* I think the marquez error is because we don't send SourceCodeLocationJobFacet

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ale + (alessandro.lollo@gmail.com) +
+
2021-10-12 09:46:17
 +
+

*Thread Reply:* Regarding the namespace, I will check it and figure it out 😊 +Regarding the error: in the context of this PR, is it something I should worry about or not?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-10-12 09:54:17
 +
+

*Thread Reply:* I think not in the context of the PR. It certainly deserves separate issue in Marquez repository.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ale + (alessandro.lollo@gmail.com) +
+
2021-10-12 10:24:38
 +
+

*Thread Reply:* 👍

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ale + (alessandro.lollo@gmail.com) +
+
2021-10-12 10:24:51
 +
+

*Thread Reply:* Is there anything else I can do to improve the PR?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-10-12 10:27:44
 +
+

*Thread Reply:* did you figure out the namespace stuff? +I think it's ready to be merged outside of that

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ale + (alessandro.lollo@gmail.com) +
+
2021-10-12 10:49:06
 +
+

*Thread Reply:* Not yet

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ale + (alessandro.lollo@gmail.com) +
+
2021-10-12 10:58:07
 +
+

*Thread Reply:* Ok i figured it out. +When running dbt locally, we connect to Redshift using an SSH tunnel. +dbt runs on Docker, hence it can access the tunnel using host.docker.internal

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ale + (alessandro.lollo@gmail.com) +
+
2021-10-12 10:58:16
 +
+

*Thread Reply:* So the namespace is correct

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-10-12 11:04:12
 +
+

*Thread Reply:* Makes sense. So, let's merge it, after DCO bot gets up again.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ale + (alessandro.lollo@gmail.com) +
+
2021-10-12 11:04:37
 +
+

*Thread Reply:* 👍

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-10-13 05:29:48
 +
+

*Thread Reply:* merged your PR 🙌

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ale + (alessandro.lollo@gmail.com) +
+
2021-10-13 10:54:09
 +
+

*Thread Reply:* 🎉

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-10-13 12:01:20
 +
+

*Thread Reply:* I think I'm going to change it up a bit. +The problem is that we can try to render jinja everywhere, including comments. +I tried to make it skip unknown methods and values here, but I think the right solution is to load the yaml, and then try to render jinja for values.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ale + (alessandro.lollo@gmail.com) +
+
2021-10-13 14:27:37
 +
+

*Thread Reply:* Ok sounds good to me!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
SAM + (skhettri@gmail.com) +
+
2021-10-06 10:50:43
 +
+

Hey there, I’m not sure why I’m getting below error, after I ran OPENLINEAGE_URL=<http://localhost:5000> dbt-ol run , although running this command dbt debug doesn’t show any error. Pls help.

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-10-06 10:54:32
 +
+

*Thread Reply:* Does it work with simply dbt run?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-10-06 10:55:51
 +
+

*Thread Reply:* also, do you have dbt-snowflake installed?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
SAM + (skhettri@gmail.com) +
+
2021-10-06 11:00:42
 +
+

*Thread Reply:* it works with dbt run

+ + + +
+ 👀 Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
SAM + (skhettri@gmail.com) +
+
2021-10-06 11:01:22
 +
+

*Thread Reply:* no i haven’t installed dbt-snowflake

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-10-06 12:04:19
 +
+

*Thread Reply:* what the dbt says - the snowflake profile with dev target - is that what you ment to run or was it something else?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-10-06 12:04:46
 +
+

*Thread Reply:* it feels very weird to me, since the dbt-ol script just runs dbt run underneath

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
SAM + (skhettri@gmail.com) +
+
2021-10-06 12:19:27
 +
+

*Thread Reply:* this is my profiles.yml file: +```snowflake: + target: dev + outputs: + dev: + type: snowflake + account: xxxxxxx

+ +
  # User/password auth
+  user: xxxxxx
+  password: xxxxx
+
+  role: poc_db_temp_fullaccess
+  database: POC_DB
+  warehouse: poc_wh
+  schema: temp
+  threads: 2
+  client_session_keep_alive: False
+  query_tag: dbt_ol```
+
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-10-06 12:26:39
 +
+

*Thread Reply:* Yes, it looks that everything is okay on your side...

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
SAM + (skhettri@gmail.com) +
+
2021-10-06 12:28:19
 +
+

*Thread Reply:* may be I’ll restart my machine and try again

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-10-06 12:30:25
 +
+

*Thread Reply:* can you try +OPENLINEAGE_URL=<http://localhost:5000> dbt-ol debug

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
SAM + (skhettri@gmail.com) +
+
2021-10-07 05:59:03
 +
+

*Thread Reply:* Actually i had to use venv that fixed above issue. However, i ran into another problem which is no jobs / datasets found in marquez:

+ +
+
+ + + + + + +
+
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-10-07 06:00:28
 +
+

*Thread Reply:* Good that you fixed that one 🙂 Regarding last one, I've found it independently yesterday and PR fixing it is already waiting for review: https://github.com/OpenLineage/OpenLineage/pull/322

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/pull/322|#322 dbt: support manifest and run-results v3>
+
+ <p>There are no major differences between v2 and v3 metadata, at least those impacting information that we're collecting. <br /> +This PR makes <code>dbt-ol</code> treat v3 metadata the same way we treated v2.</p> + +<p>Signed-off-by: Maciej Obuchowski <a href="mailto:maciej.obuchowski@getindata.com">maciej.obuchowski@getindata.com</a></p> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
SAM + (skhettri@gmail.com) +
+
2021-10-07 06:00:46
 +
+

*Thread Reply:* oh, thanks a lot

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-10-07 14:50:01
 +
+

*Thread Reply:* There will be a release soon: https://openlineage.slack.com/archives/C01CK9T7HKR/p1633631825147900

+
+ + +
+ + + } + + Willy Lulciuc + (https://openlineage.slack.com/team/U01DCMDFHBK) +
+ + + +
+
+ <p>@channel: We’ve recently become aware that our integration with dbt no longer works with the <em>latest</em> dbt manifest version (<code>v3</code>), see <a href="https://openlineage.slack.com/archives/C01CK9T7HKR/p1633531843132400">original</a> discussion. The manifest version change was introduced in dbt <code>0.21</code> , see <a href="https://github.com/dbt-labs/dbt/commit/ea07729bbf0a3481e220615644b7955659001529#diff-33c94e40283afdfeba411a98c0628fc610c287e41c3477d88559f1e0244259c3">diff</a>. That said, we do have a fix: <a href="https://github.com/OpenLineage/OpenLineage/pull/322">PR #322</a> contributed by @Maciej Obuchowski! Here’s our plan to rollout the <code>openlineage-dbt</code> hotfix for those using the <em>latest</em> version of dbt (<strong>NOTE: for those using an older dbt version, you will NOT not be affected by this bug</strong>):</p> + +<p><strong>Releasing OpenLineage <code>0.2.3</code></strong> <strong>with dbt <code>v3</code> manifest support:</strong></p> + +<ol><li>Branch off <code>0.2.2</code> tagged commit, and create a <code>openlineage-0.2.x</code> branch</li><li>Cherry pick the commit with the dbt manifest <code>v3</code> fix</li><li>Release <code>0.2.3</code> batch release +We will be releasing <code>0.2.3</code> <em>today</em>. Please reach out to us with any questions!</li> +</ol> +
+ + + + + + +
Original URL: https://openlineage.slack.com/archives/C01CK9T7HKR/p1633631825147900
+ + +
+ + Slack Conversation +
+ + +
+ + + +
+ 👍 SAM +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
SAM + (skhettri@gmail.com) +
+
2021-10-07 23:23:26
 +
+

*Thread Reply:* Hi, +openlineage-dbt==0.2.3 worked, thanks a lot for the quick fix.

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Alex P + (alexander.pelivan@scout24.com) +
+
2021-10-07 07:46:16
 +
+

Hi, I just started playing around with Marquez. When submitting some lineage data, after some experimenting, the visualisation becomes a bit cluttered with all the naive attempts of building a meaningful graph. Can I clear this up somehow? Or is there a tip, how to hide certain information?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Alex P + (alexander.pelivan@scout24.com) +
+
2021-10-07 07:46:59
 +
+

*Thread Reply:*

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Alex P + (alexander.pelivan@scout24.com) +
+
2021-10-07 09:51:40
 +
+

*Thread Reply:* So, as a quick fix, shutting down and re-starting the docker container resets everything. +./docker/up.sh

+ + + +
+ 👍 Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-10-07 12:28:25
 +
+

*Thread Reply:* I guess that it's the easiest way now. There should be API for that.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2021-10-07 14:09:50
 +
+

*Thread Reply:* @Alex P Yeah, we're realizing that being able to delete metadata is becoming very important. And, as @Maciej Obuchowski mentioned, dropping your entire database is the only way currently (not ideal!). We do have an issue in the Marquez backlog to expose delete APIs: https://github.com/MarquezProject/marquez/issues/754

+
+ + + + +
<https://github.com/MarquezProject/marquez/issues/754|#754 Delete endpoint for routes>
+
+ <p>It would be super convenient to have DELETE calls for endpoints.</p> + +<p>I.e</p> + +<p>As a user conducting testing of a data set I wish to DELETE it and then start a new <br /> +As a user creating a job I linked the wrong datasets, I wish to the DELETE the job</p> + +<p>Ideally for:</p> + +<p>• Sources +• Datasets +• Jobs +• Tags</p> +
+ +
+
Labels
+ feature, api +
+ + + + + + + +
+ + <a href="https://github.com/MarquezProject/marquez">MarquezProject/marquez</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2021-10-07 14:10:36
 +
+

*Thread Reply:* A bit more discussion is needed though. Like what if a dataset is deleted, but you still want to keep track that it existed at some point? (i.e. soft vs hard deletes). But, for the case that you just want to clear metadata because you were testing things out, then yeah, that's more obvious and requires little discussion of the API upfront.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2021-10-07 14:12:52
 +
+

*Thread Reply:* @Alex P I moved the delete APIs to the Marquez 0.20.0 release

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-10-07 14:39:03
 +
+

*Thread Reply:* Thanks Willy.

+ + + +
+ 🙌 Willy Lulciuc +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-10-07 14:48:37
 +
+

*Thread Reply:* I have also updated a corresponding issue to track this in OpenLineage: https://github.com/OpenLineage/OpenLineage/issues/323

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/issues/323|#323 [PROPOSAL] Ability to annotate datasets with metadata without a job run>
+
+ <p><strong>Purpose:</strong> <br /> +We need a way to make observations about a Dataset current state outside of a run reading or writing the data. <br /> +For example, this could be:</p> + +<p>• collecting schema information about a table from a warehouse +• informing that a table has been deleted +• etc</p> + +<p><strong>Proposed implementation</strong> <br /> +Adding Dataset event types that are not lineage events. This would be the Dataset entity with its facets: </p> + +<p><a href="https://github.com/OpenLineage/OpenLineage/blob/3a1ccbd854bbf202bbe6437bf81786cb01ed4d01/spec/OpenLineage.json#L178-L203">OpenLineage/spec/OpenLineage.json</a></p> + +<p>Lines 178 to 203 in &lt;/OpenLineage/OpenLineage/commit/3a1ccbd854bbf202bbe6437bf81786cb01ed4d01|3a1ccbd&gt;</p> + +<pre><code>"Dataset": { + "type": "object", + "properties": { + "namespace": { + "description": "The namespace containing that dataset", + "type": "string", + "example": "my-datasource-namespace" + }, + "name": { + "description": "The unique name for that dataset within that namespace", + "type": "string", + "example": "instance.schema.table" + }, + "facets": { + "description": "The facets for this dataset", + "type": "object", + "additionalProperties": { + "$ref": "#/$defs/DatasetFacet" + } + } + }, + "required": [ + "namespace", + "name" + ] + } + +</code></pre> +
+ +
+
Labels
+ proposal +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-10-07 13:36:48
 +
+

The next OpenLineage monthly meeting is on the 13th. https://wiki.lfaidata.foundation/display/OpenLineage/Monthly+TSC+meeting +please chime in here if you’d like a topic to be added to the agenda

+ + + +
+ 🙌 Willy Lulciuc, Maciej Obuchowski, Peter Hicks +
+ +
+ ❤️ Willy Lulciuc, Maciej Obuchowski, Peter Hicks +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-10-13 10:47:49
 +
+

*Thread Reply:* Reminder that the meeting is today. See you soon

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-10-13 19:49:21
 +
+

*Thread Reply:* The recording and notes of the meeting are now available: +https://wiki.lfaidata.foundation/display/OpenLineage/Monthly+TSC+meeting#MonthlyTSCmeeting-Oct13th2021

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2021-10-07 14:37:05
 +
+

@channel: We’ve recently become aware that our integration with dbt no longer works with the latest dbt manifest version (v3), see original discussion. The manifest version change was introduced in dbt 0.21 , see diff. That said, we do have a fix: PR #322 contributed by @Maciej Obuchowski! Here’s our plan to rollout the openlineage-dbt hotfix for those using the latest version of dbt (NOTE: for those using an older dbt version, you will NOT not be affected by this bug):

+ +

Releasing OpenLineage 0.2.3 with dbt v3 manifest support:

+ +
  1. Branch off 0.2.2 tagged commit, and create a openlineage-0.2.x branch
  2. Cherry pick the commit with the dbt manifest v3 fix
  3. Release 0.2.3 batch release +We will be releasing 0.2.3 today. Please reach out to us with any questions!
    4. +
+
+ + +
+ + + } + + Samjhana Khettri + (https://openlineage.slack.com/team/U02EYPQNU58) +
+ + + +
+
+ <p>Hey there, I’m not sure why I’m getting below error, after I ran <code>OPENLINEAGE_URL=&lt;http://localhost:5000&gt; dbt-ol run</code> , although running this command <code>dbt debug</code> doesn’t show any error. Pls help.</p> +
+ + + + + + +
Original URL: https://openlineage.slack.com/archives/C01CK9T7HKR/p1633531843132400
+ + +
+ + Thread in Slack Conversation +
+ + +
+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/pull/322|#322 dbt: support manifest and run-results v3>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+ 🙌 Mario Measic, Minkyu Park, Peter Hicks +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-10-07 14:55:35
 +
+

*Thread Reply:* For people following along, dbt changed the schema of its metadata which broke the openlineage integration. However we were a bit too stringent on validating the schema version (they increment it every time event if it’s backwards compatible, which it is in this case). We will fix that so that future compatible changes don’t prevent the ol integration to work.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mario Measic + (mario.measic.gavran@gmail.com) +
+
2021-10-07 16:44:28
 +
+

*Thread Reply:* As one of the main integrations, would be good to connect more within the dbt community for the next releases, by testing the release candidates 👍

+ +

Thanks for the PR

+ + + +
+ 💯 Willy Lulciuc +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2021-10-07 16:46:40
 +
+

*Thread Reply:* Yeah, I totally agree with you. We also should be more proactive and also be more aware in what’s coming in future dbt releases. Sorry if you were effected by this bug :ladybug:

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2021-10-07 18:12:22
 +
+

*Thread Reply:* We’ve release OpenLineage 0.2.3 with the hotfix for adding dbt v3 manifest support, see https://github.com/OpenLineage/OpenLineage/releases/tag/0.2.3

+ +

You can download and install openlineage-dbt 0.2.3 with the fix using:

+ +

$ pip3 install openlineage-dbt==0.2.3

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Drew Bittenbender + (drew@salt.io) +
+
2021-10-07 19:02:37
 +
+

Hello. I have a question about dbt-ol. I run dbt in a docker container and alias the dbt command to execute in that docker container. dbt-ol doesn't seem to use that alias. Do you know of a way to force it to use the alias?...or is there an alternative to getting the linage into Marquez?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-10-07 21:10:36
 +
+

*Thread Reply:* @Maciej Obuchowski might know

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-10-08 04:23:17
 +
+

*Thread Reply:* @Drew Bittenbender dbt-ol always calls dbt command now, without spawning shell - so it does not have access to bash aliases.

+ +

Can you elaborate about your use case? Do you mean that dbt in your path does docker run or something like this? It still might be a problem if we won't have access to artifacts generated by dbt in target directory.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Drew Bittenbender + (drew@salt.io) +
+
2021-10-08 10:59:32
 +
+

*Thread Reply:* I am running on a mac and I have aliased (.zshrc) dbt to execute docker run against the fishtownanalytics docker image rather than installing dbt natively (homebrew, etc). I am doing this so that the dbt configuration is portable and reusable by others.

+ +

It seems that by installing openlineage-dbt in a virtual environment, it pulls down it's own version of dbt which it calls inline rather than shelling out and executing the dbt setup resident in the host system. I understand that opening a shell is a security risk so that is understandable.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-10-08 11:05:00
 +
+

*Thread Reply:* It does not pull down, it just assumes that it's in the system. It would fail if it isn't.

+ +

For now I think you could build your own image based on official one, and install openlineage-dbt inside, something like:

+ +

FROM fishtownanalytics/dbt:0.21.0 +RUN pip install openlineage-dbt +ENTRYPOINT ["dbt-ol"]

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-10-08 11:05:15
 +
+

*Thread Reply:* and then pass OPENLINEAGE_URL in env while doing docker run

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-10-08 11:06:55
 +
+

*Thread Reply:* Also, to make sure that using shell would help in your case: do you bind mount your dbt directory to home? dbt-ol can't run without access to dbt's target directory, so if it's not visible in host, the only option is to have dbt-ol in container.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
SAM + (skhettri@gmail.com) +
+
2021-10-08 07:00:43
 +
+

Hi, I found below issues, not sure what is the root-cause:

+ +
  1. Marquez UI does not show any jobs/datasets, but if I search my table name then only it shows in search result section.
  2. After running dbt docs generate there is not schema information available in marquez?
    3. +
+ +
+
+ + + + + + +
+
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-10-08 08:16:37
 +
+

*Thread Reply:* Regarding 2), the data is only visible after next dbt-ol run - dbt docs generate does not emit events itself, but generates data that run take into account.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
SAM + (skhettri@gmail.com) +
+
2021-10-08 08:24:57
 +
+

*Thread Reply:* oh got it, since its in default, i need to click on it and choose my dbt profile’s account name. thnx

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
SAM + (skhettri@gmail.com) +
+
2021-10-08 11:25:22
 +
+

*Thread Reply:* May I know, why these highlighted ones dont have schema? FYI, I used sources in dbt.

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-10-08 11:26:18
 +
+

*Thread Reply:* Do they have it in dbt docs?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
SAM + (skhettri@gmail.com) +
+
2021-10-08 11:33:59
 +
+

*Thread Reply:* I prepared this yaml file, not sure this is what u asked

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ale + (alessandro.lollo@gmail.com) +
+
2021-10-12 04:14:08
 +
+

Hey folks 😊 +DCO checks on this PR https://github.com/OpenLineage/OpenLineage/pull/328 seem to be stuck. +Any suggestions on how to unblock it?

+ +

Thanks!

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/pull/328|#328 - Added support for Redshift profile>
+
+ <p>• Added support for Redshift profile +• Added support for handling env vars in profiles.yml</p> + +<p>Please note that I had to enclose <code>{{ config(...) }}</code> between <code>raw</code> Jinja tags, otherwise it complains about an unexpected '.'</p> + +<p>Signed-off-by: Alessandro Lollo <a href="mailto:alessandro.lollo@cloudacademy.com">alessandro.lollo@cloudacademy.com</a></p> +
+ +
+
Comments
+ 1 +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-10-12 07:21:33
 +
+

*Thread Reply:* I don't think anything is wrong with your branch. It's also not working on my one. Maybe it's globally stuck?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mark Taylor + (marktayl@microsoft.com) +
+
2021-10-12 15:17:02
 +
+

We are working on the hackathon and have a couple of questions about generating lineage information. @Willy Lulciuc would you have time to help answer a couple of questions?

+ +

• Is there a way to generate OpenLineage output that contains a mapping between input and output fields? +• In Azure Databricks sources often map to ADB mount points. We are looking for a way to translate this into source metadata in the OL output. Is there some configuration that would make this possible, or any other suggestions?

+ + + +
+ 👋 Willy Lulciuc +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2021-10-12 15:50:20
 +
+

*Thread Reply:* > Is there a way to generate OpenLineage output that contains a mapping between input and output fields? +OpenLineage defines discrete classes for both OpenLineage.InputDataset and OpenLineage.OutputDataset datasets. But, for clarification, are you asking:

+ +
  1. If a job reads / writes to the same dataset, how can OpenLineage track which fields were used in job’s logic as input and which fields were used to write back to the resulting output?
  2. Or, if a job reads / writes from two different dataset, how can OpenLineage track which input fields were used in the job’s logic for the resulting output dataset? (i.e. column-level lineage)
    3. +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2021-10-12 15:56:18
 +
+

*Thread Reply:* > In Azure Databricks sources often map to ADB mount points.  We are looking for a way to translate this into source metadata in the OL output.  Is there some configuration that would make this possible, or any other suggestions? +I would look into our OutputDatasetVisitors class (as a starting point) that extracts metadata from the spark logical plan to construct a mapping between a logic plan to one or more OpenLineage.Dataset for the spark job. But, I think @Michael Collado will have a more detailed suggestion / approach to what you’re asking

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2021-10-12 15:59:41
 +
+

*Thread Reply:* are the sources mounted like local filesystem mounts? are you ending up with datasources that point to the local filesystem rather than some dbfs url? (sorry, I'm not familiar with databricks or azure at this point)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mark Taylor + (marktayl@microsoft.com) +
+
2021-10-12 16:59:38
 +
+

*Thread Reply:* I think under the covers they are an os level fs mount, but it is using an ADB specific api, dbutils.fs.mount. It is using the ADB filesystem.

+
+
docs.microsoft.com
+ + + +
Databricks File System (DBFS) - Azure Databricks
+
+ Learn about Databricks File System (DBFS). +
+ + + + + + + + +
Original URL: https://docs.microsoft.com/en-us/azure/databricks/data/databricks-file-system
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2021-10-12 17:01:23
 +
+

*Thread Reply:* Do you use the dbfs scheme to access the files from Spark as in the example on that page? +df = spark.read.text("dbfs:/mymount/my_file.txt")

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mark Taylor + (marktayl@microsoft.com) +
+
2021-10-12 17:04:52
 +
+

*Thread Reply:* @Willy Lulciuc In our project, @Will Johnson had generated some sample OL output from just reading in and writing out a dataset to blob storage. In the resulting output, I see the columns represented as fields under the schema element with a set represented for output and another for input. I would need the mapping of in and out columns to generate column level lineage so wondering if it is possible to get or am I just missing it somewhere? Thanks for your help!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2021-10-12 17:26:35
 +
+

*Thread Reply:* Ahh, well currently, no, but it has been discussed and on the OpenLineage roadmap. Here’s a proposal opened by @Julien Le Dem, column level lineage facet, that starts the discussion to add the columnLineage face to the datasets model in order to support column-level lineage. Would be great to get your thoughts!

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/issues/148|#148 [PROPOSAL] column level lineage facet>
+
+ <p><strong>Purpose:</strong> <br /> +For transformations like SQL queries (but also in other cases), we can extract column level lineage. This allows answering questions like ‘which root input columns are used to construct column x?’. We need to be able to capture this information in the OpenLineage model</p> + +<p><strong>Proposed implementation</strong> <br /> +We propose to introduce a new dataset facet: <br /> +example:</p> + +<pre><code>{ + "eventType": "COMPLETE", + "eventTime": "2020-12-28T20:52:00.001+10:00", + "run" : { + "runId": "uuid" + }, + "job": { + "namespace": "scheduler", + "name": "myjob", + "facets": { + "sql": { + "query": "Insert into outputTable from select ** from inputTable" + } + } + }, + "inputs": [ + { + "namespace": "N1", + "name": "inputTable", + "facets": { + "schema": { + "fields": [ {"name": "col_a", "type": "VARCHAR"}, {"name": "col_b", "type": "int"}] + } + } + } + ], + "outputs": [ + { + "namespace": "N2", + "name": "outputTable", + "facets": { + "schema": { + "fields": [ {"name": "col_a", "type": "VARCHAR"}, {"name": "col_b", "type": "int"}] + }, + "columnLineage": { + "fields": { + "col_a": [ { "namespace": "N1", "name": "inputTable", "field": "col_a"} ], + "col_b": [ { "namespace": "N1", "name": "inputTable", "field": "col_b" } ] + } + } + } + } + ] +} +</code></pre> + +<p>Schema:</p> + +<pre><code> "columnLineage": { + "type": "object", + "properties": { + "fields": { + "type": "array", + "items": { + "type": "object", + "additionalProperties": { + "type": "object", + "properties": { + "namespace": { + "type": "string", + "description": "the input dataset namespace" + }, + "name": { + "type": "string", + "description": "the input dataset name" + }, + "field": { + "type": "string", + "description": "the input field" + } + }, + "required": [ + "namespace", "name" + ] + } + } + } + } +</code></pre> + +<p>References:</p> + +<p>• Datahub spec: <a href="https://datahubproject.io/docs/rfc/active/1841-lineage/field_level_lineage/">https://datahubproject.io/docs/rfc/active/1841-lineage/field<em>level</em>lineage/</a></p> +
+ +
+
Labels
+ proposal +
+ +
+
Comments
+ 3 +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2021-10-12 17:41:41
 +
+

*Thread Reply:* @Michael Collado - Databricks allows you to reference a file called /mnt/someMount/some/file/path The way you have referenced it would let you hit the file with local file system stuff like pandas / local python.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-10-12 17:49:37
 +
+

*Thread Reply:* For column level lineage, you can add your own custom facets: Here’s an example in the Spark integration: (LogicalPlanFacet) https://github.com/OpenLineage/OpenLineage/blob/5f189a94990dad715745506c0282e16fd8[…]openlineage/spark/agent/lifecycle/SparkSQLExecutionContext.java +Here is the paragraph about this in the spec: https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.md#custom-facet-naming

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/5f189a94990dad715745506c0282e16fd857b43a/integration/spark/src/main/java/io/openlineage/spark/agent/lifecycle/SparkSQLExecutionContext.java | SparkSQLExecutionContext.java>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.md | OpenLineage.md>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-10-12 17:51:24
 +
+

*Thread Reply:* This example adds facets to the run, but you can also add them to the job

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2021-10-12 17:52:46
 +
+

*Thread Reply:* unfortunately, there's not yet a way to add your own custom facets to the spark integration- there's some work on extensibility to be done

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2021-10-12 17:54:07
 +
+

*Thread Reply:* for the hackathon's sake, you can check out the package and just add in whatever you want

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2021-10-12 18:26:44
 +
+

*Thread Reply:* Thank you guys!!

+ + + +
+ 🙌 Willy Lulciuc +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2021-10-12 20:42:20
 +
+

Question on the Spark Integration and its SPARKCONFURL_KEY configuration variable.

+ +

https://github.com/OpenLineage/OpenLineage/blob/8afc4ff88b8dd8090cd9c45061a9f669fe[…]rk/src/main/java/io/openlineage/spark/agent/ArgumentParser.java

+ +

It looks like I can pass in any url but I'm not sure if I can pass in query parameters along with that URL. For example, if I had https://localhost/myendpoint?secret_code=123 I THINK that is used for the endpoint and it does not append /lineage to the end of the url. Is that a fair assessment of what happens when the url is provided?

+ +

Thank you for any guidance!

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/8afc4ff88b8dd8090cd9c45061a9f669fea2151e/integration/spark/src/main/java/io/openlineage/spark/agent/ArgumentParser.java | ArgumentParser.java>
+
+ <pre><code> public static ArgumentParser parse(String clientUrl) { +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-10-12 21:46:12
 +
+

*Thread Reply:* You can also pass the settings independently if you want something more flexible: https://github.com/OpenLineage/OpenLineage/blob/8afc4ff88b8dd8090cd9c45061a9f669fe[…]n/java/io/openlineage/spark/agent/OpenLineageSparkListener.java

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/8afc4ff88b8dd8090cd9c45061a9f669fea2151e/integration/spark/src/main/java/io/openlineage/spark/agent/OpenLineageSparkListener.java | OpenLineageSparkListener.java>
+
+ <pre><code> private ArgumentParser parseConf(SparkConf conf) { +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-10-12 21:47:36
 +
+

*Thread Reply:* SparkSession.builder() + .config("spark.jars.packages", "io.openlineage:openlineage_spark:0.2.+") + .config("spark.extraListeners", "io.openlineage.spark.agent.OpenLineageSparkListener") + .config("spark.openlineage.host", "<https://localhost>") + .config("spark.openlineage.apiKey", "your api key") + .config("spark.openlineage.namespace", "&lt;NAMESPACE_NAME&gt;") // Replace with the name of your Spark cluster. + .getOrCreate()

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-10-12 21:48:57
 +
+

*Thread Reply:* It is going to add /lineage in the end: https://github.com/OpenLineage/OpenLineage/blob/8afc4ff88b8dd8090cd9c45061a9f669fe[…]rc/main/java/io/openlineage/spark/agent/OpenLineageContext.java

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/8afc4ff88b8dd8090cd9c45061a9f669fea2151e/integration/spark/src/main/java/io/openlineage/spark/agent/OpenLineageContext.java | OpenLineageContext.java>
+
+ <pre><code> new URI(String.format("%s/api/%s/lineage", argument.getHost(), argument.getVersion())); +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-10-12 21:49:37
 +
+

*Thread Reply:* the apiKey setting is sent in an “Authorization” header

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-10-12 21:49:55
 +
+

*Thread Reply:* “Bearer $KEY”

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-10-12 21:51:09
 +
+

*Thread Reply:* https://github.com/OpenLineage/OpenLineage/blob/a6eea7a55fef444b6561005164869a9082[…]n/java/io/openlineage/spark/agent/client/OpenLineageClient.java

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/a6eea7a55fef444b6561005164869a90827cef41/integration/spark/src/main/java/io/openlineage/spark/agent/client/OpenLineageClient.java | OpenLineageClient.java>
+
+ <pre><code> request.addHeader(AUTHORIZATION, "Bearer " + apiKey.get()); +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2021-10-12 22:54:22
 +
+

*Thread Reply:* Thank you @Julien Le Dem it seems in both cases (defining the url endpoint with spark.openlineage.url and with the components: spark.openlineage.host / openlineage.version / openlineage.namespace / etc.) OpenLineage will strip out url parameters and rebuild the url endpoint with /lineage.

+ +

I think we might need to add in a url parameter configuration for our hackathon. We're using a bit of serverless code to shuttle open lineage events to a queue so that another job and/or serverless application can read that queue at its leisure.

+ +

Using the apiKey that feeds into the Authorization header as a Bearer token is great and would suffice but for our services we use OAuth tokens that would expire after two hours AND most of our customers wouldn't want to generate an access token themselves and feed it to Spark. ☹️

+ +

Would you guys entertain a proposal to support a spark.openlineage.urlParams configuration variable that lets you add url parameters to the derived lineage url?

+ +

Thank you for the detailed replies and deep links!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-10-13 10:46:22
 +
+

*Thread Reply:* Yes, please open an issue detailing the use case.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2021-10-13 13:02:06
 +
+

Quick question, is it expected, when using Spark SQL and the Spark Integration for Spark3 that we receive and INPUT but no OUTPUTS when doing a CREATE TABLE ... AS SELECT ... .

+ +

I'm reading from a Spark SQL table (underlying CSV) and then writing it to a DELTA lake table.

+ +

I get a COMPLETE event type with an INPUT but no OUTPUT and then I get an exception for the AsyncEvent Queue but I'm guessing it's unrelated 😅

+ +

21/10/13 15:38:15 INFO OpenLineageContext: Lineage completed successfully: ResponseMessage(responseCode=200, body=null, error=null) {"eventType":"COMPLETE","eventTime":"2021-10-13T15:38:15.878Z","run":{"runId":"2cfe52b3-e08f-4888-8813-ffcdd2b27c89","facets":{"spark_unknown":{"_producer":"<https://github.com/OpenLineage/OpenLineage/tree/0.2.3-SNAPSHOT/integration/spark>","_schemaURL":"<https://openlineage.io/spec/1-0-2/OpenLineage.json#/$defs/RunFacet>","output":{"description":{"@class":"org.apache.spark.sql.catalyst.plans.logical.Project","traceEnabled":false,"streaming":false,"cacheId":null,"canonicalizedPlan":false},"inputAttributes":[{"name":"id","type":"long","metadata":{}}],"outputAttributes":[{"name":"id","type":"long","metadata":{}},{"name":"action_date","type":"date","metadata":{}}]},"inputs":[{"description":{"@class":"org.apache.spark.sql.catalyst.plans.logical.Range","streaming":false,"traceEnabled":false,"cacheId":null,"canonicalizedPlan":false},"inputAttributes":[],"outputAttributes":[{"name":"id","type":"long","metadata":{}}]}]},"spark.logicalPlan":{"_producer":"<https://github.com/OpenLineage/OpenLineage/tree/0.2.3-SNAPSHOT/integration/spark>","_schemaURL":"<https://openlineage.io/spec/1-0-2/OpenLineage.json#/$defs/RunFacet>","plan":[{"class":"org.apache.spark.sql.catalyst.plans.logical.Project","num-children":1,"projectList":[[{"class":"org.apache.spark.sql.catalyst.expressions.AttributeReference","num_children":0,"name":"id","dataType":"long","nullable":false,"metadata":{},"exprId":{"product_class":"org.apache.spark.sql.catalyst.expressions.ExprId","id":111,"jvmId":"4bdfd808-97d5-455f-ad6a-a3b29855e85b"},"qualifier":[]}],[{"class":"org.apache.spark.sql.catalyst.expressions.Alias","num-children":1,"child":0,"name":"action_date","exprId":{"product-class":"org.apache.spark.sql.catalyst.expressions.ExprId","id":113,"jvmId":"4bdfd808_97d5_455f_ad6a_a3b29855e85b"},"qualifier":[],"explicitMetadata":{},"nonInheritableMetadataKeys":"[__dataset_id, __col_position]"},{"class":"org.apache.spark.sql.catalyst.expressions.CurrentDate","num_children":0,"timeZoneId":"Etc/UTC"}]],"child":0},{"class":"org.apache.spark.sql.catalyst.plans.logical.Range","num-children":0,"start":0,"end":5,"step":1,"numSlices":8,"output":[[{"class":"org.apache.spark.sql.catalyst.expressions.AttributeReference","num_children":0,"name":"id","dataType":"long","nullable":false,"metadata":{},"exprId":{"product_class":"org.apache.spark.sql.catalyst.expressions.ExprId","id":111,"jvmId":"4bdfd808-97d5-455f-ad6a-a3b29855e85b"},"qualifier":[]}]],"isStreaming":false}]}}},"job":{"namespace":"sparknamespace","name":"databricks_shell.project"},"inputs":[],"outputs":[],"producer":"<https://github.com/OpenLineage/OpenLineage/tree/0.2.3-SNAPSHOT/integration/spark>","schemaURL":"<https://openlineage.io/spec/1-0-2/OpenLineage.json#/$defs/RunEvent>"} +21/10/13 15:38:16 INFO FileSizeAutoTuner: File size tuning result: {"tuningType":"autoTuned","tunedConfs":{"spark.databricks.delta.optimize.minFileSize":"268435456","spark.databricks.delta.optimize.maxFileSize":"268435456"}} +21/10/13 15:38:16 INFO FileFormatWriter: Write Job e062f36c-8b9d-4252-8db9-73b58bd67b15 committed. +21/10/13 15:38:16 INFO FileFormatWriter: Finished processing stats for write job e062f36c-8b9d-4252-8db9-73b58bd67b15. +21/10/13 15:38:18 INFO CodeGenerator: Code generated in 253.294028 ms +21/10/13 15:38:18 INFO SparkContext: Starting job: collect at DataSkippingReader.scala:430 +21/10/13 15:38:18 INFO DAGScheduler: Job 1 finished: collect at DataSkippingReader.scala:430, took 0.000333 s +21/10/13 15:38:18 ERROR AsyncEventQueue: Listener OpenLineageSparkListener threw an exception +java.lang.NullPointerException + at io.openlineage.spark.agent.OpenLineageSparkListener.onJobEnd(OpenLineageSparkListener.java:167) + at org.apache.spark.scheduler.SparkListenerBus.doPostEvent(SparkListenerBus.scala:39) + at org.apache.spark.scheduler.SparkListenerBus.doPostEvent$(SparkListenerBus.scala:28) + at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37) + at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37) + at org.apache.spark.util.ListenerBus.postToAll(ListenerBus.scala:119) + at org.apache.spark.util.ListenerBus.postToAll$(ListenerBus.scala:103) + at org.apache.spark.scheduler.AsyncEventQueue.super$postToAll(AsyncEventQueue.scala:105) + at org.apache.spark.scheduler.AsyncEventQueue.$anonfun$dispatch$1(AsyncEventQueue.scala:105) + at scala.runtime.java8.JFunction0$mcJ$sp.apply(JFunction0$mcJ$sp.java:23) + at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62) + at <a href="http://org.apache.spark.scheduler.AsyncEventQueue.org">org.apache.spark.scheduler.AsyncEventQueue.org</a>$apache$spark$scheduler$AsyncEventQueue$$dispatch(AsyncEventQueue.scala:100) + at org.apache.spark.scheduler.AsyncEventQueue$$anon$2.$anonfun$run$1(AsyncEventQueue.scala:96) + at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1547) + at org.apache.spark.scheduler.AsyncEventQueue$$anon$2.run(AsyncEventQueue.scala:96)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-10-13 17:54:22
 +
+

*Thread Reply:* This is because this specific action is not covered yet. You can see the “spark_unknown” facet is describing things that are not understood yet +run": { +... + "facets": { + "spark_unknown": { +... + "output": { + "description": { + "@class": "org.apache.spark.sql.catalyst.plans.logical.Project", + "traceEnabled": false, + "streaming": false, + "cacheId": null, + "canonicalizedPlan": false + },

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-10-13 17:54:43
 +
+

*Thread Reply:* I think this is part of the Spark 3 gap

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-10-13 17:55:46
 +
+

*Thread Reply:* an unknown output will cause missing output lineage

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-10-13 18:05:57
 +
+

*Thread Reply:* Output handling is here: https://github.com/OpenLineage/OpenLineage/blob/e0f1852422f325dc019b0eab0e466dc905[…]io/openlineage/spark/agent/lifecycle/OutputDatasetVisitors.java

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/e0f1852422f325dc019b0eab0e466dc905876a48/integration/spark/src/main/java/io/openlineage/spark/agent/lifecycle/OutputDatasetVisitors.java | OutputDatasetVisitors.java>
+
+ <pre><code> public List&amp;lt;PartialFunction&amp;lt;LogicalPlan, List&amp;lt;OpenLineage.OutputDataset&amp;gt;&amp;gt;&amp;gt; get() { +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+ 🙌 Will Johnson +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2021-10-13 22:49:08
 +
+

*Thread Reply:* Ah! Thank you so much, Julien! This is very helpful to understand where that is set. This is a big gap that we want to help address after our hackathon. Thank you!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-10-13 20:09:17
 +
+

Following up on the meeting this morning, I have created an issue to formalize a design doc review process: https://github.com/OpenLineage/OpenLineage/issues/336 +If that sounds good I’ll create the first doc to describe this as a PR. (how meta!)

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/issues/336|#336 [PROPOSAL] Add a design doc process for proposals>
+
+ <p><strong>Purpose:</strong> <br /> +Add a design doc section for proposals. The process would be the following:</p> + +<p>• open an issue with the "PROPOSAL" template +• if the scope warrants it create a PR to add a proposal doc. +• have the PR approved +• Once the proposal is accepted you can open another issue for the implementation.</p> + +<p><strong>Proposed implementation</strong></p> + +<p>• Proposals live under OpenLineage/OpenLineage/proposals +• one folder per design (includes images and the proposal document) +• the proposal is keyed by the github issue id: "/proposals/{github issue id}/{short description}.md" +• each doc has a header with: + • Status: APPROVED | IN PROGRESS | RELEASED + • Author: + • Created: + • issue tracker: #xxx +• assignees for reviews will be in the issue itself +• community can vote on the issue</p> + +<pre><code>--- +Status: Proposal +Author: +Created: +Issue: +--- + +</code></pre> +
+ +
+
Labels
+ proposal +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-10-13 20:13:02
 +
+

*Thread Reply:* the github wiki is backed by a git repo but it does not allow PRs. (people do hacks but I’d rather avoid those)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-10-18 10:24:25
 +
+

We're discussing creating Transport abstraction for OpenLineage clients, that would allow us creating better experience for people that expect to be able to emit their events using something else than http interface. Please tell us what you think of proposed mechanism - encouraging emojis are helpful too 😉 +https://github.com/OpenLineage/OpenLineage/pull/344

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-10-18 20:57:04
 +
+

OpenLineage release 0.3 is coming. Please chiming if there’s anything blocker that should go in the release: https://github.com/OpenLineage/OpenLineage/projects/4

+ + + +
+ ❤️ Willy Lulciuc +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Carlos Quintas + (cdquintas@gmail.com) +
+
2021-10-19 06:36:05
 +
+

👋 Hi everyone!

+ + + +
+ 👋 Ross Turk, Willy Lulciuc, Michael Collado +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Carlos Quintas + (cdquintas@gmail.com) +
+
2021-10-22 05:38:14
 +
+

openlineage with DBT and Trino, is there any forecast?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-10-22 05:44:17
 +
+

*Thread Reply:* Maybe you want to contribute it? +It's not that hard, mostly testing, and figuring out what would be the naming of openlineage namespace for Trino, and how some additional statistics work.

+ +

For example, recently we had added support for Redshift by community member @ale

+ +

https://github.com/OpenLineage/OpenLineage/pull/328

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/pull/328|#328 - Added support for Redshift profile>
+
+ <p>• Added support for Redshift profile +• Added support for handling env vars in profiles.yml</p> + +<p>Please note that I had to enclose <code>{{ config(...) }}</code> between <code>raw</code> Jinja tags, otherwise it complains about an unexpected '.'</p> + +<p>Signed-off-by: Alessandro Lollo <a href="mailto:alessandro.lollo@cloudacademy.com">alessandro.lollo@cloudacademy.com</a></p> +
+ +
+
Comments
+ 1 +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Carlos Quintas + (cdquintas@gmail.com) +
+
2021-10-22 05:42:52
 +
+

Done. PASS=5 WARN=0 ERROR=0 SKIP=0 TOTAL=5 +Traceback (most recent call last): + File "/home/labuser/.local/bin/dbt-ol", line 61, in <module> + main() + File "/home/labuser/.local/bin/dbt-ol", line 54, in main + events = processor.parse().events() + File "/home/labuser/.local/lib/python3.8/site-packages/openlineage/common/provider/dbt.py", line 98, in parse + self.extractdatasetnamespace(profile) + File "/home/labuser/.local/lib/python3.8/site-packages/openlineage/common/provider/dbt.py", line 377, in extractdatasetnamespace + self.datasetnamespace = self.extractnamespace(profile) + File "/home/labuser/.local/lib/python3.8/site-packages/openlineage/common/provider/dbt.py", line 391, in extract_namespace + raise NotImplementedError( +NotImplementedError: Only 'snowflake' and 'bigquery' adapters are supported right now. Passed trino

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2021-10-22 12:41:08
 +
+

Hey folks, we've released OpenLineage 0.3.1. There are quite a few changes, including doc improvements, Redshift support in dbt, bugfixes, a new server-side client code base, but the real highlights are

+ +
  1. Official Spark 3 support- this is still a work in progress (the whole Spark integration is), but the big deal is we've split the source tree to support both Spark 2 and Spark 3 specific plan visitors. This will enable us to work with the Spark 3 API explicitly and to add support for those interfaces and classes that didn't exist in Spark 2. We're also running all integration tests against both Spark 2.4.7 and Spark 3.1.0
  2. Airflow 2 support- also a work in progress, but we have a new LineageBackend implementation that allows us to begin tracking lineage for successful Airflow 2 DAGs. We're working to support failure notifications so we can also trace failed jobs. The LineageBackend can also be enabled in Airflow 1.10.X to improve the reporting of task completion times. +Check the READMEs for more details and to get started with the new features. Thanks to @Maciej Obuchowski , @Oleksandr Dvornik, @ale, and @Willy Lulciuc for their contributions. See the full changelog
    3. +
+ + + +
+ 🎉 Willy Lulciuc, Maciej Obuchowski, Minkyu Park, Ross Turk, Peter Hicks, RamanD, Ry Walker +
+ +
+ 🙌 Willy Lulciuc, Maciej Obuchowski, Minkyu Park, Will Johnson, Ross Turk, Peter Hicks, Ry Walker +
+ +
+ 🔥 Ry Walker +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
David Virgil + (david.virgil.naranjo@googlemail.com) +
+
2021-10-28 07:27:12
 +
+

Hello community. I am starting using marquez. I try to connect dbt with Marquez, but the spark adapter is not yet available.

+ +

Are you planning to implement this spark dbt adapter in next openlineage versions?

+ +

NotImplementedError: Only 'snowflake', 'bigquery', and 'redshift' adapters are supported right now. Passed spark +In my company we are starting to use as well the athena dbt adapter. Are you planning to implement this integration? Thanks a lot community

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-10-28 12:20:27
 +
+

*Thread Reply:* That would make sense. I think you are the first person to request this. Is this something you would want to contribute to the project?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
David Virgil + (david.virgil.naranjo@googlemail.com) +
+
2021-10-28 17:37:53
 +
+

*Thread Reply:* I would like to Julien, but not sure how can I do it. Could you guide me how can i start? or show me other integration.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Matthew Mullins + (mmullins@aginity.com) +
+
2021-10-31 07:57:55
 +
+

*Thread Reply:* @David Virgil look at the pull request for the addition of Redshift as a starting guide. https://github.com/OpenLineage/OpenLineage/pull/328

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/pull/328|#328 - Added support for Redshift profile>
+
+ <p>• Added support for Redshift profile +• Added support for handling env vars in profiles.yml</p> + +<p>Please note that I had to enclose <code>{{ config(...) }}</code> between <code>raw</code> Jinja tags, otherwise it complains about an unexpected '.'</p> + +<p>Signed-off-by: Alessandro Lollo <a href="mailto:alessandro.lollo@cloudacademy.com">alessandro.lollo@cloudacademy.com</a></p> +
+ +
+
Comments
+ 1 +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
David Virgil + (david.virgil.naranjo@googlemail.com) +
+
2021-11-01 12:01:41
 +
+

*Thread Reply:* Thanks @Matthew Mullins I ll try to add dbt spark integration

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mario Measic + (mario.measic.gavran@gmail.com) +
+
2021-10-28 09:31:01
 +
+

Hey folks, quick question, are we able to run dbt-ol without providing OPENLINEAGE_URL? I find it quite limiting that I need to have a service set up in order to emit/generate OL events/messages. Is there a way to just output them to the console?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mario Measic + (mario.measic.gavran@gmail.com) +
+
2021-10-28 10:05:09
 +
+

*Thread Reply:* OK, was changed here: https://github.com/OpenLineage/OpenLineage/pull/286

+ +

Did you think about this?

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/pull/286|#286 [INTEGRATION][dbt] fail execution if OPENLINEAGE_URL is unset>
+
+ <p>Currently, <code>dbt-ol</code> will run even if <code>OPENLINEAGE_URL</code> is unset. This causes nomal dbt execution, after which user sees that events weren't emitted.</p> + +<p>We should assume that variables/config should be set and valid - and fail the run if they aren't. After all, if someone wouldn't need lineage events, they wouldn't use our wrapper.</p> + +<p>When <a href="https://github.com/OpenLineage/OpenLineage/issues/287">OpenLineage/OpenLineage#287</a> is done, we should use emitting dbt parent run as way to check if URL is valid and reachable.</p> + +<p>This will require change with introduction of client <code>Backend</code> or <code>Transport</code> - we should check if any transport is configured correctly, instead of just simple http one.</p> + +<p>Signed-off-by: Maciej Obuchowski <a href="mailto:maciej.obuchowski@getindata.com">maciej.obuchowski@getindata.com</a></p> +
+ +
+
Comments
+ 1 +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-10-28 12:19:27
 +
+

*Thread Reply:* In Marquez there was a mechanism to do that. Something like OPENLINEAGE_BACKEND=HTTP|LOG

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-10-28 13:56:42
 +
+

*Thread Reply:* @Mario Measic We're going to add Transport mechanism, that will address use cases like yours. Please comment on this PR what would you expect: https://github.com/OpenLineage/OpenLineage/pull/344

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/pull/344|#344 [PROPOSAL] Transport proposal document>
+
+ <p>This document summarizes existing discussion on <code>Transport</code> abstraction from issue <a href="https://github.com/OpenLineage/OpenLineage/issues/269">OpenLineage/OpenLineage#269</a>, following method proposed here: <a href="https://github.com/OpenLineage/OpenLineage/issues/336">OpenLineage/OpenLineage#336</a></p> + +<p>Draft implementation of <code>Transport</code> abstraction for Python is here: <a href="https://github.com/OpenLineage/OpenLineage/pull/329">OpenLineage/OpenLineage#329</a></p> + +<p>Signed-off-by: Maciej Obuchowski <a href="mailto:maciej.obuchowski@getindata.com">maciej.obuchowski@getindata.com</a></p> +
+ +
+
Comments
+ 1 +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+ 👀 Mario Measic +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mario Measic + (mario.measic.gavran@gmail.com) +
+
2021-10-28 15:29:50
 +
+

*Thread Reply:* Nice, thanks @Julien Le Dem and @Maciej Obuchowski.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mario Measic + (mario.measic.gavran@gmail.com) +
+
2021-10-28 15:46:45
 +
+

*Thread Reply:* Also, dbt build is not working which is kind of the biggest feature of the version 0.21.0, I will try testing the code with modifications to the https://github.com/OpenLineage/OpenLineage/blob/c3aa70e161244091969951d0da4f37619bcbe36f/integration/dbt/scripts/dbt-ol#L141

+ +

I guess there's a reason for it that I didn't see since you support v3 of the manifest.

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/c3aa70e161244091969951d0da4f37619bcbe36f/integration/dbt/scripts/dbt-ol | dbt-ol>
+
+ <pre><code> if len(sys.argv) &amp;lt; 2 or sys.argv[1] not in ['run', 'test']: +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mario Measic + (mario.measic.gavran@gmail.com) +
+
2021-10-29 03:45:27
 +
+

*Thread Reply:* Also, is it normal not to see the column descriptions for the model/table even though these are provided in the YAML file, persisted in Redshift and also dbt docs generate has been run before dbt-ol run?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mario Measic + (mario.measic.gavran@gmail.com) +
+
2021-10-29 04:26:22
 +
+

*Thread Reply:* Tried with dbt versions 0.20.2 and 0.21.0, openlineage-dbt==0.3.1

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-10-29 10:39:10
 +
+

*Thread Reply:* I'll take a look at that. Supporting descriptions might be simple, but dbt build might be a little larger task.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-11-01 19:12:01
 +
+

*Thread Reply:* I opened a ticket to track this: https://github.com/OpenLineage/OpenLineage/issues/376

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/issues/376|#376 [INTEGRATION][dbt] support dbt built>
+
+ <p>dbt build is a superset of the dbt run and dbt test commands. <br /> +We should apply the same logic there. <br /> +<a href="https://docs.getdbt.com/reference/commands/build">https://docs.getdbt.com/reference/commands/build</a></p> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+ 👀 Mario Measic +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-11-02 05:48:06
 +
+

*Thread Reply:* The column description issue should be fixed here: https://github.com/OpenLineage/OpenLineage/pull/383

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/pull/383|#383 dbt: use column description from metadata.json even if we have catalog>
+
+ <p>Column descriptions are properly set in <code>metadata.json</code> file. Fetch it from there, even if we have <code>catalog.json</code>. <br /> +Also, remove empty column descriptions from being checked in most tests.</p> + +<p>Signed-off-by: Maciej Obuchowski <a href="mailto:maciej.obuchowski@getindata.com">maciej.obuchowski@getindata.com</a></p> + +<p>Closes: <a href="https://github.com/OpenLineage/OpenLineage/issues/382">OpenLineage/OpenLineage#382</a></p> + +<p><strong>Checklist</strong></p> + +<p>☑︎ You've <a href="https://github.com/OpenLineage/OpenLineage/blob/main/why-the-dco.md">signed-off</a> your work +☑︎ Your pull request title follows our <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CONTRIBUTING.md#creating-pull-requests">guidelines</a> +☑︎ Your changes are accompanied by tests (<em>if relevant</em>) +☑︎ Your change contains a <a href="https://kurtisnusbaum.medium.com/stacked-diffs-keeping-phabricator-diffs-small-d9964f4dcfa6">small diff</a> and is self-contained +☐ You've updated any relevant documentation (<em>if relevant</em>) +☐ You've updated the <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md"><code>CHANGELOG.md</code></a> with details about your change under the "Unreleased" section (<em>if relevant, depending on the change, this may not be necessary</em>) +☐ You've versioned the core OpenLineage model or facets according to <a href="https://docs.snowplowanalytics.com/docs/pipeline-components-and-applications/iglu/common-architecture/schemaver">SchemaVer</a> (<em>if relevant</em>)</p> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-10-28 12:27:17
 +
+

I’m looking for feedback on my proposal to improve the proposal process ! https://github.com/OpenLineage/OpenLineage/issues/336

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/issues/336|#336 [PROPOSAL] Add a design doc process for proposals>
+
+ <p><strong>Purpose:</strong> <br /> +Add a design doc section for proposals. The process would be the following:</p> + +<p>• open an issue with the "PROPOSAL" template +• if the scope warrants it create a PR to add a proposal doc. +• have the PR approved +• Once the proposal is accepted you can open another issue for the implementation.</p> + +<p><strong>Proposed implementation</strong></p> + +<p>• Proposals live under OpenLineage/OpenLineage/proposals +• one folder per design (includes images and the proposal document) +• the proposal is keyed by the github issue id: "/proposals/{github issue id}/{short description}.md" +• each doc has a header with: + • Status: APPROVED | IN PROGRESS | RELEASED + • Author: + • Created: + • issue tracker: #xxx +• assignees for reviews will be in the issue itself +• community can vote on the issue</p> + +<pre><code>--- +Status: Proposal +Author: +Created: +Issue: +--- + +</code></pre> +
+ +
+
Assignees
+ wslulciuc, mobuchowski, mandy-chessell, collado-mike +
+ +
+
Labels
+ proposal +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Brad + (bradley.mcelroy@live.com) +
+
2021-10-28 18:49:12
 +
+

Hey guys - just an update on my prefect PR (https://github.com/OpenLineage/OpenLineage/pull/293) - there a little spiel on the ticket but I've closed that PR in favour of opening a new one. Prefect have just release a 2.0a technical preview, which they would like to make stable near the start of next year. I think it makes sense to target this release, and I've had one of the prefect team reach out and is keen to get some sort of lineage implemented in prefect.

+ + + +
+ 👍 Kevin Kho, Maciej Obuchowski, Willy Lulciuc, Michael Collado, Julien Le Dem, Thomas Fredriksen +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Brad + (bradley.mcelroy@live.com) +
+
2021-10-28 18:51:10
 +
+

*Thread Reply:* If anyone has any questions or comments - happy to discuss here

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Brad + (bradley.mcelroy@live.com) +
+
2021-10-28 18:51:15
 +
+

*Thread Reply:* @davzucky

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2021-10-28 23:01:29
 +
+

*Thread Reply:* Thanks for updating the community, Brad!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
davzucky + (davzucky@hotmail.com) +
+
2021-10-28 23:47:02
 +
+

*Thread Reply:* Than you Brad. Looking forward to see how to integrated that with v2

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kevin Kho + (kdykho@gmail.com) +
+
2021-10-28 18:53:23
 +
+

Hello, joining here from Prefect. Because of community requests from users like Brad above, we are looking to implement lineage for Prefect this quarter. Good to meet you all!

+ + + +
+ ❤️ Minkyu Park, Faouzi, John Thomas, Maciej Obuchowski, Kevin Mellott, Thomas Fredriksen +
+ +
+ 👍 Minkyu Park, Faouzi, John Thomas +
+ +
+ 🙌 Michael Collado, Faouzi, John Thomas +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2021-10-28 18:54:56
 +
+

*Thread Reply:* Welcome, @Kevin Kho 👋. Really excited to see this integration kick off! 💯🚀

+ + + +
+ 👍 Kevin Kho, Maciej Obuchowski, Peter Hicks, Faouzi +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
David Virgil + (david.virgil.naranjo@googlemail.com) +
+
2021-11-01 12:03:14
 +
+

Hello,

+ +

i am integratng openLineage with Airflow 2.2.0

+ +

Do you consider in the future airflow manual inlets and outlets?

+ +

Seeing the documentation I can see that is not possible.

+ +

OpenLineageBackend does not take into account manually configured inlets and outlets. +Thanks

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2021-11-01 12:23:11
 +
+

*Thread Reply:* While it’s not something we’re supporting at the moment, it’s definitely something that we’re considering!

+ +

If you can give me a little more detail on what your system infrastructure is like, it’ll help us set priority and design

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
David Virgil + (david.virgil.naranjo@googlemail.com) +
+
2021-11-01 13:57:34
 +
+

*Thread Reply:* So basic architecture of a datalake. We are using airflow to trigger jobs. Every job is a pipeline that runs a spark job (in our case it spin up an EMR). So the idea of lineage would be defining in the dags inlets and outlets based on the airflow lineage:

+ +

https://airflow.apache.org/docs/apache-airflow/stable/lineage.html

+ +

I think you need to be able to include these inlets and outlets in the picture of openlineage

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-11-01 14:01:24
 +
+

*Thread Reply:* Why not use spark integration? https://github.com/OpenLineage/OpenLineage/tree/main/integration/spark

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
David Virgil + (david.virgil.naranjo@googlemail.com) +
+
2021-11-01 14:05:02
 +
+

*Thread Reply:* because there are some other jobs that are not spark, some jobs they run in dbt, other jobs they run in redshift @Maciej Obuchowski

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-11-01 14:08:58
 +
+

*Thread Reply:* So, combo of https://github.com/OpenLineage/OpenLineage/tree/main/integration/dbt and PostgresExtractor from airflow integration should cover Redshift if you're using it from PostgresOperator 🙂

+ +

It's definitely interesting use case - you'd be using most of the existing integrations we have.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
David Virgil + (david.virgil.naranjo@googlemail.com) +
+
2021-11-01 15:04:44
 +
+

*Thread Reply:* @Maciej Obuchowski Do i need to define any extractor in the airflow startup?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Francis McGregor-Macdonald + (francis@mc-mac.com) +
+
2021-11-05 23:48:21
 +
+

*Thread Reply:* I am using Redshift with PostgresOperator and it is returning…

+ +

[2021-11-06 03:43:06,541] {{__init__.py:92}} ERROR - Failed to extract metadata 'NoneType' object has no attribute 'host' task_type=PostgresOperator airflow_dag_id=counter task_id=inc airflow_run_id=scheduled__2021-11-06T03:42:00+00:00 +Traceback (most recent call last): + File "/usr/local/airflow/.local/lib/python3.7/site-packages/openlineage/lineage_backend/__init__.py", line 83, in _extract_metadata + task_metadata = self._extract(extractor, task_instance) + File "/usr/local/airflow/.local/lib/python3.7/site-packages/openlineage/lineage_backend/__init__.py", line 104, in _extract + task_metadata = extractor.extract_on_complete(task_instance) + File "/usr/local/airflow/.local/lib/python3.7/site-packages/openlineage/airflow/extractors/base.py", line 61, in extract_on_complete + return self.extract() + File "/usr/local/airflow/.local/lib/python3.7/site-packages/openlineage/airflow/extractors/postgres_extractor.py", line 65, in extract + authority=self._get_authority(), + File "/usr/local/airflow/.local/lib/python3.7/site-packages/openlineage/airflow/extractors/postgres_extractor.py", line 120, in _get_authority + if self.conn.host and self.conn.port: +AttributeError: 'NoneType' object has no attribute 'host'

+ +

I can’t see this raised as an issue.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
David Virgil + (david.virgil.naranjo@googlemail.com) +
+
2021-11-01 13:57:54
 +
+

Hello, I am trying to integrate Airflow with openlineage.

+ +

It is not working for me.

+ +

What I tried:

+ +
  1. Adding openlineage-airflow to requirements.txt
  2. Adding +```- AIRFLOWLINEAGEBACKEND=openlineage.airflow.backend.OpenLineageBackend
    3. +
+ +

During handling of the above exception, another exception occurred:

+ +

Traceback (most recent call last): + File "/home/airflow/.local/bin/airflow", line 8, in <module> + sys.exit(main()) + File "/home/airflow/.local/lib/python3.8/site-packages/airflow/main.py", line 40, in main + args.func(args) + File "/home/airflow/.local/lib/python3.8/site-packages/airflow/cli/cliparser.py", line 47, in command + func = importstring(importpath) + File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/moduleloading.py", line 32, in importstring + module = importmodule(modulepath) + File "/usr/local/lib/python3.8/importlib/init.py", line 127, in importmodule + return bootstrap.gcdimport(name[level:], package, level) + File "<frozen importlib.bootstrap>", line 1014, in gcdimport + File "<frozen importlib.bootstrap>", line 991, in _findandload + File "<frozen importlib.bootstrap>", line 975, in findandloadunlocked + File "<frozen importlib.bootstrap>", line 671, in _loadunlocked + File "<frozen importlib.bootstrapexternal>", line 843, in execmodule + File "<frozen importlib.bootstrap>", line 219, in callwithframesremoved + File "/home/airflow/.local/lib/python3.8/site-packages/airflow/cli/commands/dbcommand.py", line 24, in <module> + from airflow.utils import cli as cliutils, db + File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/db.py", line 26, in <module> + from airflow.jobs.basejob import BaseJob # noqa: F401 + File "/home/airflow/.local/lib/python3.8/site-packages/airflow/jobs/init.py", line 19, in <module> + import airflow.jobs.backfilljob + File "/home/airflow/.local/lib/python3.8/site-packages/airflow/jobs/backfilljob.py", line 29, in <module> + from airflow import models + File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/init.py", line 20, in <module> + from airflow.models.baseoperator import BaseOperator, BaseOperatorLink + File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/baseoperator.py", line 196, in <module> + class BaseOperator(Operator, LoggingMixin, TaskMixin, metaclass=BaseOperatorMeta): + File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/baseoperator.py", line 941, in BaseOperator + def postexecute(self, context: Any, result: Any = None): + File "/home/airflow/.local/lib/python3.8/site-packages/airflow/lineage/init.py", line 103, in applylineage + _backend = getbackend() + File "/home/airflow/.local/lib/python3.8/site-packages/airflow/lineage/init.py", line 52, in get_backend + clazz = conf.getimport("lineage", "backend", fallback=None) + File "/home/airflow/.local/lib/python3.8/site-packages/airflow/configuration.py", line 469, in getimport + raise AirflowConfigException( +airflow.exceptions.AirflowConfigException: The object could not be loaded. Please check "backend" key in "lineage" section. Current value: "openlineage.airflow.backend.OpenLineageBackend".```

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-11-01 14:06:12
 +
+

*Thread Reply:* 1. Please use openlineage.lineage_backend.OpenLineageBackend as AIRFLOW__LINEAGE__BACKEND

+ +
  1. Please tell us where you've seen openlineage.airflow.backend.OpenLineageBackend, so we can fix the documentation 🙂
    2. +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-11-01 19:07:21
 +
+

*Thread Reply:* https://pypi.org/project/openlineage-airflow/

+
+
PyPI
+ + + +
openlineage-airflow
+
+ OpenLineage integration with Airflow +
+ + + + + + + + +
Original URL: https://pypi.org/project/openlineage-airflow/
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-11-01 19:08:03
 +
+

*Thread Reply:* (I googled it and found that page that seems to have an outdated doc)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
David Virgil + (david.virgil.naranjo@googlemail.com) +
+
2021-11-02 02:38:59
 +
+

*Thread Reply:* @Maciej Obuchowski +@Julien Le Dem that's the page i followed. Please guys revise the documentation, as it is very important

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-11-02 04:34:14
 +
+

*Thread Reply:* It should just copy actual readme

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/integration/airflow/setup.py | setup.py>
+
+ <pre><code> long_description=readme, +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2021-11-03 16:30:00
 +
+

*Thread Reply:* PyPi is using the README at the time of the release 0.3.1, rather than the current README, which is 0.4.0. If we send the new release to PyPi it should also update the README

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
David Virgil + (david.virgil.naranjo@googlemail.com) +
+
2021-11-01 15:09:54
 +
+

Related the Airflow integration. Is it required to install openlineage-airflow and setup the environment variables in both scheduler and webserver, or just in the scheduler?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
David Virgil + (david.virgil.naranjo@googlemail.com) +
+
2021-11-01 15:19:18
 +
+

*Thread Reply:* I set i up in the scheduler and it starts to log data to marquez. But it fails with this error:

+ +

Traceback (most recent call last): + File "/home/airflow/.local/lib/python3.8/site-packages/openlineage/client/client.py", line 49, in __init__ + raise ValueError(f"Need valid url for OpenLineageClient, passed {url}") +ValueError: Need valid url for OpenLineageClient, passed "<http://marquez-internal-eks.eu-west-1.dev.hbi.systems>"

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
David Virgil + (david.virgil.naranjo@googlemail.com) +
+
2021-11-01 15:19:26
 +
+

*Thread Reply:* why is it not a valid URL?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2021-11-01 18:39:58
 +
+

*Thread Reply:* Which version of the OpenLineage client are you using? On first check it should be fine

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
David Virgil + (david.virgil.naranjo@googlemail.com) +
+
2021-11-02 05:14:30
 +
+

*Thread Reply:* @John Thomas I was appending double quotes as part of the url. Forget about this error

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2021-11-02 10:35:28
 +
+

*Thread Reply:* aaaah, gotcha, good catch!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
David Virgil + (david.virgil.naranjo@googlemail.com) +
+
2021-11-02 05:15:52
 +
+

Hello, I am receiving this error today when I deployed openlineage in development environment (not using docker-compose locally).

+ +

I am running with KubernetesExecutor

+ +

airflow.exceptions.AirflowConfigException: The object could not be loaded. Please check "backend" key in "lineage" section. Current value: "openlineage.lineage_backend.OpenLineageBackend".

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-11-02 05:18:18
 +
+

*Thread Reply:* Are you sure that openlineage-airflow is present in the container?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
David Virgil + (david.virgil.naranjo@googlemail.com) +
+
2021-11-02 05:23:09
 +
+

So in this case in my template I am adding:

+ +

```env:
+ ADDITIONALPYTHONDEPS: "openpyxl==3.0.3 smartopen==2.0.0 apache-airflow-providers-http apache-airflow-providers-cncf-kubernetes apache-airflow-providers-amazon openlineage-airflow" + OPENLINEAGEURL: https://marquez-internal-eks.eu-west-1.dev.hbi.systems + OPENLINEAGENAMESPACE: dnsairflow + AIRFLOWKUBERNETESENVIRONMENTVARIABLESOPENLINEAGEURL: https://marquez-internal-eks.eu-west-1.dev.hbi.systems + AIRFLOWKUBERNETESENVIRONMENTVARIABLESOPENLINEAGENAMESPACE: dns_airflow

+ +

configmap: + mountPath: /var/airflow/config # mount path of the configmap + data: + airflow.cfg: | + [lineage] + backend = openlineage.lineage_backend.OpenLineageBackend

+ +
pod_template_file.yaml: |
+
+    containers:
+      - args: []
+        command: []
+        env:
+          - name: AIRFLOW__KUBERNETES_ENVIRONMENT_VARIABLES__OPENLINEAGE_URL
+            value: <https://marquez-internal-eks.eu-west-1.dev.hbi.systems>
+          - name: AIRFLOW__KUBERNETES_ENVIRONMENT_VARIABLES__OPENLINEAGE_NAMESPACE
+            value: dns_airflow
+          - name: AIRFLOW__LINEAGE__BACKEND
+            value: openlineage.lineage_backend.OpenLineageBackend```
+
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
David Virgil + (david.virgil.naranjo@googlemail.com) +
+
2021-11-02 05:23:31
 +
+

I am installing openlineage in the ADDITIONAL_PYTHON_DEPS

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-11-02 05:25:43
 +
+

*Thread Reply:* Maybe ADDITIONAL_PYTHON_DEPS are dependencies needed by the tasks, and are installed after Airflow tries to initialize LineageBackend?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
David Virgil + (david.virgil.naranjo@googlemail.com) +
+
2021-11-02 06:34:11
 +
+

*Thread Reply:* I am checking this accessing the Kubernetes pod

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
David Virgil + (david.virgil.naranjo@googlemail.com) +
+
2021-11-02 06:34:54
 +
+

I have a question related airflow and open lineage:

+ +

I have a dag that contains 2 tasks:

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
David Virgil + (david.virgil.naranjo@googlemail.com) +
+
2021-11-02 06:35:34
 +
+

I see that every task is displayed as a different job. I was expecting to see one job per dag.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
David Virgil + (david.virgil.naranjo@googlemail.com) +
+
2021-11-02 07:29:43
 +
+

Is this the expected behaviour??

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-11-02 07:34:47
 +
+

*Thread Reply:* Yes

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-11-02 07:35:53
 +
+

*Thread Reply:* Probably what you want is job hierarchy: https://github.com/MarquezProject/marquez/issues/1737

+
+ + + + +
<https://github.com/MarquezProject/marquez/issues/1737|#1737 Implement job hierarchy based on parent job>
+
+ <p>jobs should have a name that is relative to the parent job facet when created through OpenLineage events.</p> +
+ +
+
Assignees
+ collado-mike +
+ + + + + + + +
+ + <a href="https://github.com/MarquezProject/marquez">MarquezProject/marquez</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
David Virgil + (david.virgil.naranjo@googlemail.com) +
+
2021-11-02 07:46:02
 +
+

*Thread Reply:* I do not see any benefit of just having some airflow task metadata. I do not see relationship between tasks. Every task is a job. When I was thinking about lineage when i started working on my company integration with openlineage i though that openlineage would give me relationship between task or datasets and the only thing i see is some metadata of the history of airflow runs that is already provided by airflow

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
David Virgil + (david.virgil.naranjo@googlemail.com) +
+
2021-11-02 07:46:20
 +
+

*Thread Reply:* i was expecting to see a nice graph. I think it is missing some features

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
David Virgil + (david.virgil.naranjo@googlemail.com) +
+
2021-11-02 07:46:25
 +
+

*Thread Reply:* at this early stage

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-11-02 07:50:10
 +
+

*Thread Reply:* It probably depends on whether those tasks are covered by the extractors: https://github.com/OpenLineage/OpenLineage/tree/main/integration/airflow/openlineage/airflow/extractors

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
David Virgil + (david.virgil.naranjo@googlemail.com) +
+
2021-11-02 07:55:50
 +
+

*Thread Reply:* We are not using any of those operators: bigquery, postsgress or snowflake.

+ +

And what is it doing GreatExpectactions extractor?

+ +

It would be good if there is one extractor that relies in the inlets and outlets that you can define in any Airflow task, and that that can be the general way to make relationships between datasets

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
David Virgil + (david.virgil.naranjo@googlemail.com) +
+
2021-11-02 07:56:30
 +
+

*Thread Reply:* And that the same dag graph can be seen in marquez, and not one job per task.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-11-02 08:07:06
 +
+

*Thread Reply:* > It would be good if there is one extractor that relies in the inlets and outlets that you can define in any Airflow task +I think this is good idea. Overall, OpenLineage strongly focuses on automatic metadata collection. However, using them would be a nice fallback for not-covered-yet cases.

+ +

> And that the same dag graph can be seen in marquez, and not one job per task. +This currently depends on dataset hierarchy. If you're not using any of the covered extractors, then Marquez can't build dataset graph like in the demo: https://raw.githubusercontent.com/MarquezProject/marquez/main/web/docs/demo.gif

+ +

With the job hierarchy ticket, probably some graph could be generated using just the job data though.

+
+ + + + +
+
+ +
+ + + + + + + + +
Original URL: https://raw.githubusercontent.com/MarquezProject/marquez/main/web/docs/demo.gif
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-11-02 08:09:55
 +
+

*Thread Reply:* Created issue for the manual fallback: https://github.com/OpenLineage/OpenLineage/issues/384

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/issues/384|#384 [INTEGRATION][Airflow] Consider using manual inlets/outlets as a fallback for automatic extractor mechanism>
+
+ <p>There are way more operators than we can reasonably cover. Consider adding manual fallback mechanism that would cover cases where we don't have any extractors.</p> + +<p>Advantage of this approach is that it's generally easier for users to annotate dag than to modify it to use OpenLineage client manually.</p> +
+ +
+
Assignees
+ mobuchowski +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
David Virgil + (david.virgil.naranjo@googlemail.com) +
+
2021-11-02 08:28:29
 +
+

*Thread Reply:* @Maciej Obuchowski how many people are working full time in this library? I really would like to adopt it in my company, as we use airflow and spark, but i see that yet it does not have the features we would like to.

+ +

At the moment the same info we have in marquez related the tasks, is available in airflow UI or using airflow API.

+ +

The game changer for us would be that it could give us features/metadata that we cannot query directly from airflow. That's why if the airflow inlets/outlets could be used, then it really would make much more sense for us to adopt it.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-11-02 09:33:31
 +
+

*Thread Reply:* > how many people are working full time in this library? +On Airflow integration or on OpenLineage overall? 🙂

+ +

> The game changer for us would be that it could give us features/metadata that we cannot query directly from airflow. +I think there are three options there:

+ +
  1. Contribute relevant extractors for Airflow operators that you use
  2. Use those extractors as custom extractors: https://github.com/OpenLineage/OpenLineage/tree/main/integration/airflow#custom-extractors
  3. Create that manual fallback mechanism with Airflow inlets/outlets: https://github.com/OpenLineage/OpenLineage/issues/384
    4. +
+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/issues/384|#384 [INTEGRATION][Airflow] Consider using manual inlets/outlets as a fallback for automatic extractor mechanism>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-11-02 09:35:10
 +
+

*Thread Reply:* But first, before implementing last option, I'd like to get consensus about it - so feel free to comment there about your use case

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
David Virgil + (david.virgil.naranjo@googlemail.com) +
+
2021-11-02 09:19:14
 +
+

@Maciej Obuchowski even i can contribute or help with my ideas (from what i consider that should be lineage from a client side)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
David Virgil + (david.virgil.naranjo@googlemail.com) +
+
2021-11-03 07:58:56
 +
+

@Maciej Obuchowski I was able to put to work Airflow in Kubernetes pointing to Marquez using the openlineage library. I have a few problems I found that would be good to comment.

+ +

I see a warning +[2021-11-03 11:47:04,309] {great_expectations_extractor.py:27} WARNING - Did not find great_expectations_provider library or failed to import it +I couldnt find any information about GreatExpectationsExtractor. Could you tell me what is this extractor about?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-11-03 08:00:34
 +
+

*Thread Reply:* It should only affect you if you're using https://greatexpectations.io/

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Francis McGregor-Macdonald + (francis@mc-mac.com) +
+
2021-11-03 15:57:02
 +
+

*Thread Reply:* I have a similar message after installing openlineage into Amazon MWAA from the scheduler logs:

+ +

WARNING:/usr/local/airflow/.local/lib/python3.7/site-packages/openlineage/airflow/extractors/great_expectations_extractor.py:Did not find great_expectations_provider library or failed to import it

+ +

I am not using great expectations in the DAG.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
David Virgil + (david.virgil.naranjo@googlemail.com) +
+
2021-11-03 08:00:52
 +
+

I see a few priorities for Airflow integration:

+ +
  1. Direct relationship 1-1 between Dag && Job. At the moment every task is a different job in marquez. What i consider wrong.
  2. Airflow Inlets/outlets integration with marquez +When do you think you guys can have this? If you need any help I can happily contribute, but I would need some help
    3. +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-11-03 08:08:21
 +
+

*Thread Reply:* I don't think 1) is a good idea. You can have multiple tasks in one dag, processing different datasets and producing different datasets. If you want visual linking of jobs that produce disjoint datasets, then I think you want this: https://github.com/MarquezProject/marquez/issues/1737 +which wuill affect visual layer.

+ +

Regarding 2), I think we need to get along with Airflow maintainers regarding long term mechanism on which OL will work: https://github.com/apache/airflow/issues/17984

+ +

I think using inlets/outlets as a fallback mechanism when we're not doing automatic metadata extraction is a good idea, but we don't know if hypothetical future mechanism will have access to these. It's hard to commit to mechanism which might disappear soon.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
David Virgil + (david.virgil.naranjo@googlemail.com) +
+
2021-11-03 08:13:28
 +
+

Another option is that I build my own extractor, do you have any example of how to create a custom extractor? How I can apply that customExtractor to specific operators? Is there a way to link an extractor with an operator, so at runtime airflow knows which extractor to run?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-11-03 08:19:00
 +
+

*Thread Reply:* https://github.com/OpenLineage/OpenLineage/tree/main/integration/airflow#custom-extractors

+ +

I think you can base your code on any existing extractor, like PostgresExtractor: https://github.com/OpenLineage/OpenLineage/blob/main/integration/airflow/openlineage/airflow/extractors/postgres_extractor.py#L53

+ +

Custom extractors work just like buildin ones, just that you need to add bit of mapping between operator and extractor, like OPENLINEAGE_EXTRACTOR_PostgresOperator=openlineage.airflow.extractors.postgres_extractor.PostgresExtractor

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/integration/airflow/openlineage/airflow/extractors/postgres_extractor.py | postgres_extractor.py>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+ 👍 Francis McGregor-Macdonald +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
David Virgil + (david.virgil.naranjo@googlemail.com) +
+
2021-11-03 08:35:59
 +
+

*Thread Reply:* Thank you very much @Maciej Obuchowski

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
David Virgil + (david.virgil.naranjo@googlemail.com) +
+
2021-11-03 08:36:52
 +
+

Last question of the morning. Running one task that failed i could see that no information appeared in Marquez. Is this something that is expected to happen? I would like to see in Marquez all the history of runs, successful and unsucessful them.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-11-03 08:41:14
 +
+

*Thread Reply:* It worked like that in Airflow 1.10.

+ +

This is an unfortunate limitation of LineageBackend API that we're using for Airflow 2. We're trying to work out solution for this with Airflow maintainers: https://github.com/apache/airflow/issues/17984

+
+ + + + +
<https://github.com/apache/airflow/issues/17984|#17984 Add possibility to LineageBackend to be notified of task instance execution start and failure>
+
+ <p><strong>Description</strong></p> + +<p>Provide way for <code>LineageBackend</code> to be notified of task instance start and failed execution.</p> + +<p>Proposed implementation:</p> + +<ol><li>add start and fail methods to LineageBackend</li><li>prepare<em>lineage decorator should call LineageBackend's start method the same way that apply</em>lineage calls send_lineage.</li><li>call fail method on where on<em>failure</em>callback is called.</li> +</ol> + +<p><strong>Use case/motivation</strong></p> + +<p>I'm working on <a href="https://openlineage.io/">OpenLineage</a> integration for Airflow 2. <a href="https://github.com/OpenLineage/OpenLineage/blob/airflow/2/integration/airflow/openlineage/airflow/backend.py">I've created MVP of this.</a></p> + +<p>In contrast to our integration for Airflow 1.10 done via subclassing DAG, <code>LineageBackend</code> provides easy way to integrate OpenLineage just via configuration.</p> + +<p>However, <code>LineageBackend.send_lineage</code> is called just on successful task execution, while we also want to track newly started and failed tasks.</p> + +<p><strong>Related issues</strong></p> + +<p><em>No response</em></p> + +<p><strong>Are you willing to submit a PR?</strong></p> + +<p>☑︎ Yes I am willing to submit a PR!</p> + +<p><strong>Code of Conduct</strong></p> + +<p>☑︎ I agree to follow this project's <a href="https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md">Code of Conduct</a></p> +
+ +
+
Labels
+ kind:feature, area:lineage +
+ +
+
Comments
+ 23 +
+ + + + + + + +
+ + <a href="https://github.com/apache/airflow">apache/airflow</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
David Virgil + (david.virgil.naranjo@googlemail.com) +
+
2021-11-04 03:41:38
 +
+

Hello openlineage community.

+ +

Yesterday I tried the integration with spark.

+ +

The result was not satisfactory. This is what I did:

+ +
  1. Add openlineage-spark dependency
  2. Add these lines: +.config("spark.jars.packages", "io.openlineage:openlineage_spark:0.3.1") +.config("spark.extraListeners", "io.openlineage.spark.agent.OpenLineageSparkListener") +.config("spark.openlineage.url", "<https://marquez-internal-eks.eu-west-1.dev.hbi.systems/api/v1/namespaces/spark_integration/>" +This job was doing spark.read from 2 different json location. +It is doing spark write to 5 different parquet location in s3. +The job finished succesfully and the result in marquez is:
    3. +
+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
David Virgil + (david.virgil.naranjo@googlemail.com) +
+
2021-11-04 03:43:40
 +
+

It created 3 namespaces. One was the one that I point in the spark config property. The other 2 are the bucket that we are writing to () and the bucket where we are reading from ()

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
David Virgil + (david.virgil.naranjo@googlemail.com) +
+
2021-11-04 03:44:00
 +
+

If I enter in the bucket namespaces I see nowthing inside

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
David Virgil + (david.virgil.naranjo@googlemail.com) +
+
2021-11-04 03:48:35
 +
+

I can see if i enter in one of the weird jobs generated this:

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-11-04 18:47:41
 +
+

*Thread Reply:* This job with no output is a symptom of the output not being understood. you should be able to see the facets for that job. There will be a spark_unknown facet with more information about the problem. If you put that into an issue with some more details about this job we should be able to help.

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
David Virgil + (david.virgil.naranjo@googlemail.com) +
+
2021-11-05 04:36:30
 +
+

*Thread Reply:* I ll try to put all the info in a ticket, as it is not working as i would expect

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
David Virgil + (david.virgil.naranjo@googlemail.com) +
+
2021-11-04 03:52:24
 +
+

And i am seeing this as well

+ +

If I check the logs of marquez-web and marquez I can't see any error there

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
David Virgil + (david.virgil.naranjo@googlemail.com) +
+
2021-11-04 03:54:38
 +
+

When I try to open the job fulfilments.execute_insert_into_hadoop_fs_relation_command I see this window:

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
David Virgil + (david.virgil.naranjo@googlemail.com) +
+
2021-11-04 04:06:29
 +
+

The page froze and no link from the menu works. Apart from that I see that there are no messages in the logs

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-11-04 18:49:31
 +
+

*Thread Reply:* Is there an error in the browser javascript console? (example on chrome: View -> Developer -> Javascript console)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Alessandro Rizzo + (l.alessandrorizzo@gmail.com) +
+
2021-11-04 17:22:29
 +
+

Hi #general, I'm a data engineer for a UK-based insuretech (part of one of the biggest UK retail insurers). We run a series of tech meetups and we'd love to have someone from the OpenLineage project to give us a demo of the tool. Would anyone be interested (DM if so 🙂 ) ?

+ + + +
+ 👍 Ross Turk +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Taleb Zeghmi + (talebz@zillowgroup.com) +
+
2021-11-04 21:30:24
 +
+

Hi! Is there an example of tracking lineage when using Pandas to read/write and transform data?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2021-11-04 21:35:16
 +
+

*Thread Reply:* Hi Taleb - I don’t know of a generalized example of lineage tracking with Pandas, but you should be able to accomplish this by sending the runEvents manually to the OpenLineage API in your code: +https://openlineage.io/docs/openapi/

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Taleb Zeghmi + (talebz@zillowgroup.com) +
+
2021-11-04 21:38:25
 +
+

*Thread Reply:* Is this a work in progress, that we can investigate? Because I see it in this image https://github.com/OpenLineage/OpenLineage/blob/main/doc/Scope.png

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/doc/Scope.png | Scope.png>
+
+ +
+ + + + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2021-11-04 21:54:51
 +
+

*Thread Reply:* To my knowledge, while there are a few proposals around adding a wrapper on some Pandas methods to output runEvents, it’s not something that’s had work started on it yet

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2021-11-04 21:56:26
 +
+

*Thread Reply:* I sent some feelers out to get a little more context from folks who are more informed about this than I am, so I’ll get you more info about potential future plans and the considerations around them when I know more

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2021-11-04 23:04:47
 +
+

*Thread Reply:* So, Pandas is tricky because unlike Airflow, DBT, or Spark, Pandas doesn’t own the whole flow, and you might dip in and out of it to use other Python Packages (at least I did when I was doing more Data Science).

+ +

We have this issue open in OpenLineage that you should go +1 to help with our planning 🙂

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/issues/108|#108 [INTEGRATION] Add support for `pandas.read_sql`>
+
+ <p>Let's add support for the Panda wrapper method <a href="https://pandas.pydata.org/docs/reference/api/pandas.read_sql.html"><code>read_sql</code></a>:</p> + +<pre><code>pandas.read_sql(sql, con, index_col=None, coerce_float=True, + params=None, parse_dates=None, columns=None, chunksize=None) + +</code></pre> + +<p>Since we have access to the <code>sql</code> at time of method execution, we can easily parse the SQL to capture the in / out tables before sending an OpenLineage event. We also have access to <code>columns</code> (if present) on table reads and may also want to support <code>read_sql_table</code> and <code>read_sql_query</code> (for backward compatibility).</p> +
+ +
+
Labels
+ proposal +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Taleb Zeghmi + (talebz@zillowgroup.com) +
+
2021-11-05 15:08:09
 +
+

*Thread Reply:* interesting... what if it were instead on all the read_** to_** functions?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Lyndon Armitage + (lyndon.armitage@gmail.com) +
+
2021-11-05 12:00:57
 +
+

Hi! I am working alongside David at integrating OpenLineage into our Data Pipelines. I have a questions around Marquez and OpenLineage's divergent APIs: +That is to say, these 2 APIs differ: +https://openlineage.io/docs/openapi/ +https://marquezproject.github.io/marquez/openapi.html +This makes sense since they are at different layers of abstraction, but Marquez requires a few things that are absent from OpenLineage's API, for example the type in a data source, the distinctions between physicalName and sourceName in Datasets. Is that intentional? And can these be set using the OpenLineage API as some additional facets or keys? I noticed that the DatasourceDatasetFacet has a map of additionalProperties .

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2021-11-05 12:59:49
 +
+

*Thread Reply:* The Marquez write APIs are artifacts from before OpenLineage existed, and they’re already slated for deprecation soon.

+ +

If you POST an OpenLineage runEvent to the /lineage endpoint in Marquez, it’ll create any missing jobs or datasets that are relevant.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Lyndon Armitage + (lyndon.armitage@gmail.com) +
+
2021-11-05 13:06:06
 +
+

*Thread Reply:* Thanks for the response. That sounds good. Does this include the query interface e.g. +http://localhost:5000/api/v1/namespaces/testing_java/datasets/incremental_data +as that currently returns the Marquez version of a dataset including default set fields for type and the above mentioned properties.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2021-11-05 17:01:55
 +
+

*Thread Reply:* I believe the intention for type is to support a new facet- TBH, it hasn't been the most pressing concern for most users, as most people are only recording tables, not streams. However, there's been some recent work to support Kafka in Spark- maybe it's time to address that deficiency.

+ +

I don't actually know what happened to the datasource type field- maybe @Julien Le Dem can comment on whether that field was dropped intentionally or whether it was an oversight.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-11-05 18:18:06
 +
+

*Thread Reply:* It looks like an oversight, currently Marquez hard codes it to POSGRESQL: https://github.com/MarquezProject/marquez/blob/734bfd691636cb00212d7d22b1a489bd4870fb04/api/src/main/java/marquez/db/OpenLineageDao.java#L438

+
+ + + + +
<https://github.com/MarquezProject/marquez/blob/734bfd691636cb00212d7d22b1a489bd4870fb04/api/src/main/java/marquez/db/OpenLineageDao.java | OpenLineageDao.java>
+
+ <pre><code> default String getSourceType(Dataset ds) { +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/MarquezProject/marquez">MarquezProject/marquez</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-11-05 18:18:25
 +
+

*Thread Reply:* https://github.com/MarquezProject/marquez/blob/734bfd691636cb00212d7d22b1a489bd4870fb04/api/src/main/java/marquez/db/OpenLineageDao.java#L438-L440

+
+ + + + +
<https://github.com/MarquezProject/marquez/blob/734bfd691636cb00212d7d22b1a489bd4870fb04/api/src/main/java/marquez/db/OpenLineageDao.java | OpenLineageDao.java>
+
+ <pre><code> default String getSourceType(Dataset ds) { + return SourceType.of("POSTGRESQL").getValue(); + } +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/MarquezProject/marquez">MarquezProject/marquez</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-11-05 18:20:25
 +
+

*Thread Reply:* The source has a name though: https://github.com/OpenLineage/OpenLineage/blob/8afc4ff88b8dd8090cd9c45061a9f669fea2151e/spec/facets/DatasourceDatasetFacet.json#L12

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/8afc4ff88b8dd8090cd9c45061a9f669fea2151e/spec/facets/DatasourceDatasetFacet.json | DatasourceDatasetFacet.json>
+
+ <pre><code> "type" : "string" +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-11-05 18:07:16
 +
+

The next OpenLineage monthly meeting is this coming Wednesday at 9am PT +The tentative agenda is: +• OL Client use cases for Apache Iceberg [Ryan] +• OpenLineage and Azure Purview [Shrikanth] +• Proxy Backend and Egeria integration progress update (Issue #152) [Mandy] +• OpenLineage last release overview (0.3.1) + ◦ Facet versioning + ◦ Airflow 2 / Spark 3 support, dbt improvements +• OpenLineage 0.4 scope review + ◦ Proxy Backend (Issue #152) + ◦ Spark, Airflow, dbt improvements (documentation, coverage, ...) + ◦ improvements to the OpenLineage model +• Open discussion 

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/issues/152|#152 Proposal for a ProxyBackend>
+
+ <p>This issue is to track the discussion and development of a proxy backend. The diagram below shows the basic idea.</p> + +<p><a href="https://user-images.githubusercontent.com/12354195/128734055-2a08372e-8d1b-426e-9c25-c06aeb551e36.png">image</a></p> +
+ +
+
Assignees
+ mandy-chessell +
+ +
+
Comments
+ 3 +
+ + + + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+ 🙌 Maciej Obuchowski, Peter Hicks +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-11-05 18:07:57
 +
+

*Thread Reply:* If you want to add something please chime in this thread

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-11-05 19:27:44
 +
+

*Thread Reply:* https://wiki.lfaidata.foundation/display/OpenLineage/Monthly+TSC+meeting

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-11-09 19:47:26
 +
+

*Thread Reply:* The monthly meeting is happening tomorrow. +The purview team will present at the December meeting instead +See full agenda here: https://wiki.lfaidata.foundation/display/OpenLineage/Monthly+TSC+meeting +You are welcome to contribute

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-11-10 11:10:17
 +
+

*Thread Reply:* The slides for the meeting later today: https://docs.google.com/presentation/d/1z2NTkkL8hg_2typHRYhcFPyD5az-5-tl/edit#slide=id.ge7d4b64ef4_0_0

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-11-10 12:02:23
 +
+

*Thread Reply:* It’s happening now ^

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-11-16 19:57:23
 +
+

*Thread Reply:* I have posted the notes and the recording from the last instance of our monthly meeting: https://wiki.lfaidata.foundation/display/OpenLineage/Monthly+TSC+meeting#MonthlyTSCmeeting-Nov10th2021(9amPT) +I have a few TODOs to follow up on tickets

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-11-05 18:09:10
 +
+

The next release of OpenLineage is being scoped: https://github.com/OpenLineage/OpenLineage/projects/6 +Please chime in if you want to raise the priority of something or are planning to contribute

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anthony Ivanov + (anthvt@gmail.com) +
+
2021-11-09 08:18:11
 +
+

Hi, I have been looking at open lineage for some time. And I really like it. It is very simple specification that covers a lot of use-cases. You can create any provider or consumer in a very simple way. So that’s pretty powerful. +I have some questions about things that are not clear to me. I am not sure if this is the best place to ask. Please refer me to other place if this is not appropriate.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anthony Ivanov + (anthvt@gmail.com) +
+
2021-11-09 08:18:58
 +
+

*Thread Reply:* How do you model continuous process (not batch processes). For example a flume or spark job that does some real time processing on data.

+ +

Maybe it’s simply a “Job” But than what is run ?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anthony Ivanov + (anthvt@gmail.com) +
+
2021-11-09 08:19:44
 +
+

*Thread Reply:* How do you model consumers at the end - they can be reports? Data applications, ML model deployments, APIs, GUI consumed by end users ?

+ +

Have you considered having some examples of different use cases like those?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anthony Ivanov + (anthvt@gmail.com) +
+
2021-11-09 08:21:43
 +
+

*Thread Reply: By definition, Job is a process definition that consumes and produces datasets. It is many to many relations? I’ve been wondering about that.Shouldn’t be more restrictive? +For example important use-case for lineage is troubleshooting or error notifications (e.g mark report or job as temporarily in bad state if upstream data integration is broken). +In order to be able to that you need to be able to traverse the graph to find the original error. So having multiple inputs produce single output make sense (e.g insert into output_1 select * from x,y group by a,b) . +But what are the cases where you’d want to see multiple outputs ? You can have single process produce multiple tables (in above example) but they’d alway be separate queries. The actual inputs for each output would be different.

+ +

But having multiple outputs create ambiguity as now If x or y is broken but have multiple outputs I do not know which is really impacted?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-11-09 08:34:01
 +
+

*Thread Reply:* > How do you model continuous process (not batch processes). For example a flume or spark job that does some real time processing on data. +> +> Maybe it’s simply a “Job” But than what is run ? +Every continuous process eventually has end - for example, you can deploy new version of your Flink pipeline. The new version would be the next Run for the same Job.

+ +

Moreover, OTHER event type is useful to update metadata like amount of processed records. In this Flink example, it could be emitted per checkpoint.

+ +

I think more attention for streaming use cases will be given soon.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-11-09 08:43:09
 +
+

*Thread Reply:* > How do you model consumers at the end - they can be reports? Data applications, ML model deployments, APIs, GUI consumed by end users ? +Our reference implementation is an web application https://marquezproject.github.io/marquez/

+ +

We definitely do not exclude any of the things you're talking about - and it would make a lot of sense to talk more about potential usages.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-11-09 08:45:47
 +
+

*Thread Reply:* > By definition, Job is a process definition that consumes and produces datasets. It is many to many relations? I’ve been wondering about that.Shouldn’t be more restrictive? +I think this is too SQL-centric view 🙂

+ +

Not everything is a query. For example, those Flink streaming jobs can produce side outputs, or even push data to multiple sinks. We need to model those types of jobs too.

+ +

If your application does not do multiple outputs, then I don't see how specification allowing those would impact you.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anthony Ivanov + (anthvt@gmail.com) +
+
2021-11-17 12:11:37
 +
+

*Thread Reply:* > We definitely do not exclude any of the things you’re talking about - and it would make a lot of sense to talk more about potential usages. +Yes I think that would be great if we expand on potential usages. if Open Lineage documentation (perhaps) has all kind of examples for different use-cases or case studies. Financal or healthcase industry case study and how would someone doing integration with OpenLineage. It would be easier to understand the concepts and make sure things are modeled consistently.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anthony Ivanov + (anthvt@gmail.com) +
+
2021-11-17 14:19:19
 +
+

*Thread Reply:* > I think this is too SQL-centric view 🙂 +> +> Not everything is a query. For example, those Flink streaming jobs can produce side outputs, or even push data to multiple sinks. We need to model those types of jobs too. +Thanks for answering @Maciej Obuchowski

+ +

Even in SQL you can have multiple outputs if you look thing at transaction level. I was simply using it as an example.

+ +

Maybe it would be clear what I mean in another example . Let’s say we have those phases

+ +
  1. Ingest from sources
  2. Process/transform
  3. export to somewhere +(image/diagram) +https://mermaid.ink/img/eyJjb2RlIjoiXG5ncmFwaCBMUlxuICAgIHN1YmdyYXBoIFNvdXJjZXNcbi[…]yIjpmYWxzZSwiYXV0b1N5bmMiOnRydWUsInVwZGF0ZURpYWdyYW0iOmZhbHNlfQ
    4. +
+ +

Let’s look at those two cases:

+ +
  1. Within a single flink job and even task: Inventory & UI are both written to both S3, DB
  2. Within a single flink job and even task: Inventory is written only to S3, UI is written only to DB
    3. +
+ +

In 1. open lineage run event could look like {inputs: [ui, inventory], outputs: [s3, db] }

+ +

In 2. user can either do same as 1. (because data changes or copy-paste) which would be an error since both do not go to both +Likely accurate one would be +{inputs: [ui], outputs: [s3] } {inputs: [ui], outputs: [db] }

+ +

If the specification standard required single output then

+ +
  1. would be modelled like run event {inputs: [ui, inventory], outputs: [s3] } ; {inputs: [ui, inventory], outputs: [db] } which is still correct if more verbose.
  2. could only be modelled this way: +{inputs: [ui], outputs: [s3] }; {inputs: [ui], outputs: [db] }
    3. +
+ +

The more restrictive specification seems to lower the chance for an error doesn’t it?

+ +

Also if tools know spec guarantees single output , they’d be able to write tracing capabilities which are more precise because the structure would allow for less ambiguity. +Storage backends that implement the spec could be also written in more optimal ways perhaps I have not looked into those accuracy of those hypothesis though.

+ +

Those were the thoughts I was thinking when asking about that. I’d be curious if there’s document on the research of pros/cons and alternatives for the design of the current specifications

+
+ + + + +
+
+ +
+ + + + + + + + +
Original URL: https://mermaid.ink/img/eyJjb2RlIjoiXG5ncmFwaCBMUlxuICAgIHN1YmdyYXBoIFNvdXJjZXNcbiAgICAgICAgSW52ZW50b3J5XG4gICAgICAgIFVJIFxuICAgIGVuZFxuICAgIHN1YmdyYXBoIFByb2Nlc3NcbiAgICAgICAgRmxpbmtKb2IxXG4gICAgICAgIEludmVudG9yeSAtLT4gfCBldmVudHMgIHwgRmxpbmtKb2IxXG4gICAgICAgIFVJIC0tPiB8IGV2ZW50cyB8IEZsaW5rSm9iMVxuICAgIGVuZFxuICAgIHN1YmdyYXBoIERlc3RpbmF0aW9uIFxuICAgICAgICBTM1xuICAgICAgICBEQlxuICAgICAgICBGbGlua0pvYjEgLS0-IFMzXG4gICAgICAgIEZsaW5rSm9iMSAtLT4gREJcbiAgICBlbmRcbiIsIm1lcm1haWQiOnsidGhlbWUiOiJmb3Jlc3QifSwidXBkYXRlRWRpdG9yIjpmYWxzZSwiYXV0b1N5bmMiOnRydWUsInVwZGF0ZURpYWdyYW0iOmZhbHNlfQ
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-11-23 05:38:11
 +
+

*Thread Reply:* @Anthony Ivanov I see what you're trying to model. I think this could be solved by column level lineage though - when we'll have it. OL consumer could look at particular columns and derive which table contained particular error.

+ +

> 2. Within a single flink job and even task: Inventory is written only to S3, UI is written only to DB +Does that actually happen? I understand this in case of job, but having single operator write to two different systems seems like bad design. Wouldn't that leave the possibility of breaking exactly-once unless you're going full into two phase commit?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anthony Ivanov + (anthvt@gmail.com) +
+
2021-11-23 17:02:36
 +
+

*Thread Reply:* > Does that actually happen? I understand this in case of job, but having single operator write to two different systems seems like bad design +In a Spark or flink job it is less likely now that you mention it. But in a batch job (airflow python or kubernetes operator for example) users could do anything and then they’d need lineage to figure out what is wrong if even if what they did is suboptimal 🙂

+ +

> I see what you’re trying to model. +I am not trying to model something specific. I am trying to understand how would openlineage be used in different organisations/companies and use-cases.

+ +

> I think this could be solved by column level lineage though +There’s something specific planned ? I could not find a ticket in github. I thought you can use Dataset Facets - Schema for example could be subset of columns for a table …

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-11-24 04:55:41
 +
+

*Thread Reply:* @Anthony Ivanov take a look at this: https://github.com/OpenLineage/OpenLineage/issues/148

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Lyndon Armitage + (lyndon.armitage@gmail.com) +
+
2021-11-10 13:21:23
 +
+

How do you deleting jobs/runs from Marquez/OpenLineage?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2021-11-10 16:17:10
 +
+

*Thread Reply:* We’re adding APIs to delete metadata in Marquez 0.20.0. Here’s the related issue, https://github.com/MarquezProject/marquez/issues/1736

+
+ + + + +
<https://github.com/MarquezProject/marquez/issues/1736|#1736 Ability to DELETE jobs and datasets>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/MarquezProject/marquez">MarquezProject/marquez</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2021-11-10 16:17:37
 +
+

*Thread Reply:* Until then, you can connected to the DB directly and drop the rows from both the datasets and jobs tables (I know, not dieal)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Lyndon Armitage + (lyndon.armitage@gmail.com) +
+
2021-11-11 05:03:50
 +
+

*Thread Reply:* Thanks! I assume deleting information will remain a Marquez only feature rather than becoming part of OpenLineage itself?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2021-12-10 14:07:57
 +
+

*Thread Reply:* Yes! Delete operations will be an action supported by consumers of OpenLineage events

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Lyndon Armitage + (lyndon.armitage@gmail.com) +
+
2021-11-11 05:13:31
 +
+

Am I understanding namespaces correctly? A job namespace is different to a Dataset namespace. +And that job namespaces define a job environment, like Airflow, Spark or some other system that executes jobs. But Dataset namespace define data locations, like an S3 bucket, local file system or schema in a Database?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Lyndon Armitage + (lyndon.armitage@gmail.com) +
+
2021-11-11 05:14:39
 +
+

*Thread Reply:* I've been skimming this page: https://github.com/OpenLineage/OpenLineage/blob/main/spec/Naming.md

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/spec/Naming.md | Naming.md>
+
+ <p><strong>Naming</strong></p> + +<p>We define the unique name strategy per resource to ensure it is followed uniformly independently from who is producing metadata and we can connect lineage from various sources.</p> + +<p>Both Jobs and Datasets are in their own namespaces. Job namespaces are related to their scheduler. The namespace for a dataset is the unique name for its datasource.</p> + +<p><strong>Datasets</strong></p> + +<p>The namespace and name of a datasource can be combined to form a URI (scheme:[//authority]path)</p> + +<p>• Namespace = scheme:[//authority] (the datasource) +• Name = path (the datasets)</p> + +<p><strong>Data warehouses/data bases</strong></p> + +<p>Datasets are called tables. Tables are organised in databases and schemas.</p> + +<p><strong>Postgres:</strong></p> + +<p>Datasource hierarchy:</p> + +<p>• Host +• Port</p> + +<p>Naming hierarchy:</p> + +<p>• Database +• Schema +• Table</p> + +<p>Identifier:</p> + +<p>• Namespace: postgres://{host}:{port} of the service instance. + • Scheme = postgres + • Authority = {host}:{port} +• Unique name: {database}.{schema}.{table} + • URI = postgres://{host}:{port}/{database}.{schema}.{table}</p> + +<p><strong>Redshift:</strong></p> + +<p>Datasource hierarchy:</p> + +<p>• Host: examplecluster.&lt;XXXXXXXXXXXX&gt;.<a href="http://us-west-2.redshift.amazonaws.com">us-west-2.redshift.amazonaws.com</a> +• Port: 5439</p> + +<p>Naming hierarchy:</p> + +<p>• Database +• Schema +• Table</p> + +<p>Identifier:</p> + +<p>• Namespace: redshift://{host}:{port} of the cluster instance. + • Scheme = redshift + • Authority = {host}:{port} +• Unique name: {database}.{schema}.{table} + • URI = redshift://{host}:{port}/{database}.{schema}.{table}</p> + +<p><strong>Snowflake</strong></p> + +<p>See: <a href="https://docs.snowflake.com/en/sql-reference/identifiers.html">Object Identifiers — Snowflake Documentation</a></p> + +<p>Datasource hierarchy:</p> + +<p>• account name</p> + +<p>Naming hierarchy:</p> + +<p>• Database: {database name} =&gt; unique across the account +• Schema: {schema name} =&gt; unique within the database +• Table: {table name} =&gt; unique within the schema</p> + +<p>Identifier:</p> + +<p>• Namespace: snowflake://{account name} + • Scheme = snowflake + • Authority = {account name} +• Name: {database}.{schema}.{table} + • URI = snowflake://{account name}/{database}.{schema}.{table}</p> + +<p><strong>BigQuery</strong></p> + +<p>See: <br /> +<a href="https://cloud.google.com/resource-manager/docs/creating-managing-projects|Creating and managing projects | Resource Manager Documentation">https://cloud.google.com/resource-manager/docs/creating-managing-projects|Creating and managing projects | Resource Manager Documentation</a> <br /> +<a href="https://cloud.google.com/bigquery/docs/datasets-intro|Introduction to datasets | BigQuery">https://cloud.google.com/bigquery/docs/datasets-intro|Introduction to datasets | BigQuery</a> <br /> +<a href="https://cloud.google.com/bigquery/docs/tables-intro|Introduction to tables | BigQuery">https://cloud.google.com/bigquery/docs/tables-intro|Introduction to tables | BigQuery</a></p> + +<p>Datasource hierarchy:</p> + +<p>• bigquery</p> + +<p>Naming hierarchy:</p> + +<p>• Project Name: {project name} =&gt; is not unique +• Project number: {project number} =&gt; numeric: is unique across google cloud +• Project ID: {project id} =&gt; readable: is unique across google cloud +• dataset: {dataset name} =&gt; is unique within a project +• table: {table name} =&gt; is unique within a dataset</p> + +<p>Identifier :</p> + +<p>• Namespace: bigquery + • Scheme = bigquery + • Authority = +• Unique name: {project id}.{dataset name}.{table name} + • URI = bigquery:{project id}.{schema}.{table}</p> + +<p><strong>Distributed file systems/blob stores</strong> +<strong>GCS</strong></p> + +<p>Datasource hierarchy: none, naming is global</p> + +<p>Naming hierarchy:</p> + +<p>• bucket name =&gt; globally unique +• Path</p> + +<p>Identifier :</p> + +<p>• Namespace: gs://{bucket name} + • Scheme = gs + • Authority = {bucket name} +• Unique name: {path} + • URI = gs://{bucket name}{path}</p> + +<p><strong>S3</strong></p> + +<p>Naming hierarchy:</p> + +<p>• bucket name =&gt; globally unique +• Path</p> + +<p>Identifier :</p> + +<p>• Namespace: s3://{bucket name} + • Scheme = s3 + • Authority = {bucket name} +• Unique name: {path} + • URI = s3://{bucket name}{path}</p> + +<p><strong>HDFS</strong></p> + +<p>Naming hierarchy:</p> + +<p>• Namenode: host + port +• Path</p> + +<p>Identifier :</p> + +<p>• Namespace: hdfs://{namenode host}:{namenode port} + • Scheme = hdfs + • Authority = {namenode host}:{namenode port} +• Unique name: {path} + • URI = hdfs://{namenode host}:{namenode port}{path}</p> + +<p><strong>DBFS (Databricks File System)</strong></p> + +<p>Naming hierarchy:</p> + +<p>• workspace name: globally unique +• Path</p> + +<p>Identifier :</p> + +<p>• Namespace: hdfs://{workspace name} + • Scheme = hdfs + • Authority = workspace name +• Unique name: {path} + • URI = hdfs://{workspace name}{path}</p> + +<p><strong>Jobs</strong> +<strong>Context</strong></p> + +<p>A <code>Job</code> is a recurring data transformation with Inputs and outputs. Each execution is captured as a <code>Run</code> with corresponding metadata. <br /> +A <code>Run</code> event identifies the <code>Job</code> it is an instance of by providing the job’s unique identifier. <br /> +The <code>Job</code> identifier is composed of a <code>Namespace</code> and a <code>Name</code>. The job name is unique within that namespace.</p> + +<p>The core property we want to identify about a <code>Job</code> is how it changes over time. Different schedules of the same logic applied to different datasets (possibly with different parameters) are different jobs. The notion of a <code>job</code> is tied to a recurring schedule with specific inputs and outputs. It could be an incremental update or a full reprocess or even a streaming job.</p> + +<p>If the same code artifact (for example a spark jar or a templated SQL query) is used in the context of different schedules with different input or outputs then they are different jobs. <br /> +We are interested first in how they affect the datasets they produce.</p> + +<p><strong>Job Namespace and constructing job names</strong></p> + +<p>Jobs have a <code>name</code> that is unique to them in their <code>namespace</code> by construction.</p> + +<p>The Namespace is the root of the naming hierarchy. The job name is constructed to identify the job within that namespace.</p> + +<p>Example:</p> + +<p>• Airflow: + • Namespace: the namespace is assigned to the airflow instance. Ex: airflow-staging, airflow-prod + • Job: each task in a DAG is a job. name: {dag name}.{task name} +• Spark: + • Namespace: the namespace is provided as a configuration parameter as in airflow. If there's a parent job, we use the same namespace, otherwise it is provided by configuration. + • Spark app job name: the spark.app.name + • Spark action job name: {spark.app.name}.{node.name}</p> + +<p><strong>Parent job run: a nested hierarchy of Jobs</strong></p> + +<p>It is often the case that jobs are part of a nested hierarchy. <br /> +For example an Airflow DAG contains tasks. An instance of the DAG is finished when all of the tasks are finished. Similarly a Spark job can spawn multiple actions each of them running independently. Additionally, a Spark job can be launched by an Airflow task within a DAG.</p> + +<p>Since what we care about is identifying the job as rooted in a recurring schedule, we want to capture that connection and make sure that we treat the same application logic triggered at different schedules as different jobs. For example: if an Airflow DAG runs individual tasks per partition (for example market segments) using the same underlying job logic, they will be tracked as separate jobs.</p> + +<p>To capture this, a run event provides <a href="https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.json#L282-L331">a <code>ParentRun</code> facet</a>, referring to the parent <code>Job</code> and <code>Run</code>. This allows tracking a recurring job from the root of the schedule it is running for. <br /> +If there's a parent job, we use the same namespace, otherwise it is provided by configuration.</p> + +<p>Example:</p> + +<pre><code>{ + "run": { + "runId": "run_uuid" + }, + "job": { + "namespace": "job_namespace", + "name": "job_name" + } +} + +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-11-11 05:46:06
 +
+

*Thread Reply:* Yes!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Lyndon Armitage + (lyndon.armitage@gmail.com) +
+
2021-11-11 06:17:01
 +
+

*Thread Reply:* Excellent, I think I had mistakenly conflated the two originally. This document makes it a little clearer. +As an additional question: +When viewing a Dataset in Marquez will it cross the job namespace bounds? As in, will I see jobs from different job namespaces?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Lyndon Armitage + (lyndon.armitage@gmail.com) +
+
2021-11-11 09:20:14
 +
+

*Thread Reply:* In this example I have 1 job namespace and 2 dataset namespaces: +sql-runner-dev is the job namespace. +I cannot see a graph of my job now. Is this something to do with the namespace names?

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Lyndon Armitage + (lyndon.armitage@gmail.com) +
+
2021-11-11 09:21:46
 +
+

*Thread Reply:* The above document seems to have implied a namespace could be like a connection string for a database

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Lyndon Armitage + (lyndon.armitage@gmail.com) +
+
2021-11-11 09:22:25
 +
+

*Thread Reply:* Wait, it does work? Marquez was being temperamental

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Lyndon Armitage + (lyndon.armitage@gmail.com) +
+
2021-11-11 09:24:01
 +
+

*Thread Reply:* Yes, marquez is unable to fetch lineage for either dataset

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Lyndon Armitage + (lyndon.armitage@gmail.com) +
+
2021-11-11 09:32:19
 +
+

*Thread Reply:* Here's what I mean:

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-11-11 09:59:24
 +
+

*Thread Reply:* I think you might have hit this issue: https://github.com/MarquezProject/marquez/issues/1744

+
+ + + + +
<https://github.com/MarquezProject/marquez/issues/1744|#1744 UI doesn't urlencode dataset names>
+
+ <p>The UI doesn't urlencode dataset names, so any dataset that has a <code>/</code> or <code>:</code> in the name causes a 404. This is notable with any S3 or GCS dataset, as their structure is always</p> + +<pre><code>{"namespace": "&lt;gs://bucket&gt;", +"datasetName": "/path/to/data"} + +</code></pre> +
+ +
+
Labels
+ bug +
+ + + + + + + +
+ + <a href="https://github.com/MarquezProject/marquez">MarquezProject/marquez</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-11-11 10:00:29
 +
+

*Thread Reply:* or, maybe not? It was released already.

+ +

Can you create issue on github with those helpful gifs? @Lyndon Armitage

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Lyndon Armitage + (lyndon.armitage@gmail.com) +
+
2021-11-11 10:58:25
 +
+

*Thread Reply:* I think you are right Maciej

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Lyndon Armitage + (lyndon.armitage@gmail.com) +
+
2021-11-11 10:58:52
 +
+

*Thread Reply:* Was that patched in 0,19.1?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-11-11 11:06:06
 +
+

*Thread Reply:* As far as I see yes: https://github.com/MarquezProject/marquez/releases/tag/0.19.1

+ +

Haven't tested this myself unfortunately.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Lyndon Armitage + (lyndon.armitage@gmail.com) +
+
2021-11-11 11:07:07
 +
+

*Thread Reply:* Perhaps not. It is urlencoding them: +<http://localhost:3000/lineage/dataset/jdbc%3Ah2%3Amem%3Asql_tests_like/HBMOFA.ORDDETP> +But the error seems to be in marquez getting them.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Lyndon Armitage + (lyndon.armitage@gmail.com) +
+
2021-11-11 11:09:23
 +
+

*Thread Reply:* This is an example Lineage event JSON I am sending.

+ +
+
Example lineage event
+ + + + + + +
+ + +
+ 👀 Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Lyndon Armitage + (lyndon.armitage@gmail.com) +
+
2021-11-11 11:11:29
 +
+

*Thread Reply:* I did run into another issue with really long names not being supported due to Marquez's DB using a fixed size string for a column, but that is understandable and probably a non-issue (my test code was generating temporary folders with long names).

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Lyndon Armitage + (lyndon.armitage@gmail.com) +
+
2021-11-11 11:22:00
 +
+

*Thread Reply:* A 404 is returned for: http://localhost:3000/api/v1/lineage/?nodeId=dataset:jdbc%3Ah2%3Amem%3Asql_tests_like:HBMOFA.ORDDETP

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-11-11 11:36:01
 +
+

*Thread Reply:* @Lyndon Armitage can you create issue on the Marquez repo? https://github.com/MarquezProject/marquez/issues

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Lyndon Armitage + (lyndon.armitage@gmail.com) +
+
2021-11-11 11:52:36
 +
+

*Thread Reply:* https://github.com/MarquezProject/marquez/issues/1761 Is this sufficient?

+
+ + + + +
<https://github.com/MarquezProject/marquez/issues/1761|#1761 404 returned when querying nodes that contain URL Encoded characters>
+
+ <p>With an event similar to the attached scheme 3 namespaces are created. 2 of these are Dataset namespaces and 1 is a job namespace.</p> + +<p>When trying to access one of these datasets a 404 is encountered</p> + +<p><strong>URL:</strong> <a href="http://localhost:3000/api/v1/lineage/?nodeId=dataset:jdbc%3Ah2%3Amem%3Asql_tests_like:HBMOFA.ORDDETP|http://localhost:3000/api/v1/lineage/?nodeId=dataset:jdbc%3Ah2%3Amem%3Asql_tests_like:HBMOFA.ORDDETP">http://localhost:3000/api/v1/lineage/?nodeId=dataset:jdbc%3Ah2%3Amem%3Asql<em>tests</em>like:HBMOFA.ORDDETP|http://localhost:3000/api/v1/lineage/?nodeId=dataset:jdbc%3Ah2%3Amem%3Asql<em>tests</em>like:HBMOFA.ORDDETP</a></p> + +<pre><code>{ + "eventType": "COMPLETE", + "eventTime": 1636646662.687894, + "run": { + "runId": "d3968e5f-84ac-48c1-954c-f999ff27ef3a", + "facets": null + }, + "job": { + "namespace": "sql-runner-dev", + "name": "ORDDETP - ORDDETP.avro", + "facets": { + "sourceCodeLocation": null, + "sql": { + "_producer": "lyndon-thinkpad/127.0.1.1", + "_schemaURL": "&lt;https://openlineage.io/spec/facets/1-0-0/SQLJobFacet.json#/$defs/SQLJobFacet&gt;", + "query": "SELECT t.**, CURRENT_DATE AS ingest_date FROM HBMOFA.ORDDETP t WHERE (BAYY &amp;gt;= 93 AND BAMMDD &amp;gt;= 801 AND BAMMDD &amp;lt; 1111) OR (BAYY = 92 AND BAMMDD &amp;gt;= 1301)" + }, + "documentation": { + "_producer": "lyndon-thinkpad/127.0.1.1", + "_schemaURL": "&lt;https://openlineage.io/spec/facets/1-0-0/DocumentationJobFacet.json#/$defs/DocumentationJobFacet&gt;", + "description": "SQL Runner Job for /tmp/sql_runner_tests4560779590026189736/config.conf" + } + } + }, + "inputs": [ + { + "namespace": "jdbc:h2:mem:sql_tests_like", + "name": "HBMOFA.ORDDETP", + "facets": { + "documentation": null, + "dataSource": { + "_producer": "lyndon-thinkpad/127.0.1.1", + "_schemaURL": "&lt;https://openlineage.io/spec/facets/1-0-0/DatasourceDatasetFacet.json#/$defs/DatasourceDatasetFacet&gt;", + "name": "jdbc:h2:mem:sql_tests_like", + "uri": "jdbc:h2:mem:sql_tests_like" + }, + "schema": null + }, + "inputFacets": null + } + ], + "outputs": [ + { + "namespace": "&lt;s3://sql-runner&gt;", + "name": "2021-11-11/incremental/ORDDETP.avro", + "facets": { + "documentation": null, + "dataSource": { + "_producer": "lyndon-thinkpad/127.0.1.1", + "_schemaURL": "&lt;https://openlineage.io/spec/facets/1-0-0/DatasourceDatasetFacet.json#/$defs/DatasourceDatasetFacet&gt;", + "name": "&lt;s3://sql-runner&gt;", + "uri": "&lt;s3://sql-runner&gt;" + }, + "schema": null + }, + "outputFacets": null + } + ], + "producer": "lyndon-thinkpad/127.0.1.1", + "schemaURL": "&lt;https://openlineage.io/spec/1-0-2/OpenLineage.json#/$defs/RunEvent&gt;" +} +</code></pre> + +<p>A partial stack trace from the docker process looks as follows:</p> + +<pre><code>marquez-api | 172.21.0.4 - - [11/Nov/2021:16:49:56 +0000] "GET /api/v1/namespaces/jdbc%3Ah2%3Amem%3Asql_tests_like/datasets/HBMOFA.ORDDETP/versions?limit=100&amp;amp;offset=0 HTTP/1.1" 200 648 "&lt;http://localhost:3000/lineage/dataset/jdbc%3Ah2%3Amem%3Asql_tests_like/HBMOFA.ORDDETP&gt;" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.81 Safari/537.36" 27 +marquez-api | ERROR [2021-11-11 16:49:59,168] io.dropwizard.jersey.errors.IllegalStateExceptionMapper: Error handling a request: 7b955941cb928966 +marquez-api | ! java.lang.IllegalStateException: No match available +marquez-api | ! at java.base/java.util.regex.Matcher.start(Unknown Source) +marquez-api | ! at marquez.service.models.NodeId.parts(NodeId.java:190) +marquez-api | ! at marquez.service.models.NodeId.asDatasetId(NodeId.java:214) +marquez-api | ! at marquez.service.LineageService.getJobUuid(LineageService.java:191) +marquez-api | ! at marquez.service.LineageService.lineage(LineageService.java:40) +marquez-api | ! at marquez.api.OpenLineageResource.getLineage(OpenLineageResource.java:96) + +</code></pre> + +<p>Note that the original OpenLineage POST request succeeds.</p> +
+ + + + + + + +
+ + <a href="https://github.com/MarquezProject/marquez">MarquezProject/marquez</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-11-11 11:54:41
 +
+

*Thread Reply:* Yup, thanks!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Francis McGregor-Macdonald + (francis@mc-mac.com) +
+
2021-11-15 13:00:39
 +
+

I am looking at an AWS Glue Crawler lineage event. The glue crawler creates or updates a table schema, and I have a few questions on aligning to best practice.

+ +
  1. Is this a dataset create/update or…
  2. … a job with no dataset inputs and only dataset outputs or
  3. … is the path in S3 the input and the Glue table the output?
  4. Is there an example of the lineage even here I can clone or work from? +Thanks.
    5. +
+ + + +
+ 🚀 Willy Lulciuc +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2021-11-15 13:04:19
 +
+

*Thread Reply:* Hi Francis, for the event is it creating a new table with new data in glue / adding new data to an existing one or is it simply reformatting an existing table or making an empty one?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Francis McGregor-Macdonald + (francis@mc-mac.com) +
+
2021-11-15 13:35:00
 +
+

*Thread Reply:* The table does not exist in the Glue catalog until …

+ +

A Glue crawler connects to one or more data stores (in this case S3), determines the data structures, and writes tables into the Data Catalog.

+ +

The data/objects are in S3, the Glue catalog is a metadata representation (HIVE) as as table.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2021-11-15 13:41:14
 +
+

*Thread Reply:* Hmm, interesting, so the lineage of interest here would be of the metadata flow not of the data itself?

+ +

In that case I’d say that the glue Crawler is a job that outputs a dataset.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2021-11-15 15:03:36
 +
+

*Thread Reply:* The crawler is a job that discovers a dataset. It doesn't create it. If you're posting lineage yourself, I'd post it as an input event, not an output. The thing that actually wrote the data - generated the records and stored them in S3 - is the thing that would be outputting the dataset

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Francis McGregor-Macdonald + (francis@mc-mac.com) +
+
2021-11-15 15:23:23
 +
+

*Thread Reply:* @Michael Collado I agree the crawler discovers the S3 dataset. It also creates an event which creates/updates the HIVE/Glue table.

+ +

If the Glue table isn’t a distinct dataset from the S3 data, how does this compare to a view in a database on top of a table. Are they 2 datasets or just one?

+ +

Glue can discover data in remote databases too, in those cases does it make sense to have only the source dataset?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Francis McGregor-Macdonald + (francis@mc-mac.com) +
+
2021-11-15 15:24:39
 +
+

*Thread Reply:* @John Thomas yes, its the metadata flow.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2021-11-15 15:24:52
 +
+

*Thread Reply:* that's how the Spark integration currently treats Hive datasets- I'd like to add a facet to attach that indicates that it is being read as a Hive table, and include all the appropriate metadata, but it uses the dataset's location in S3 as the canonical dataset identifier

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2021-11-15 15:29:22
 +
+

*Thread Reply:* @Francis McGregor-Macdonald I think the way to represent this is predicated on what you’re looking to accomplish by sending a runEvent for the Glue crawler. What are your broader objectives in adding this?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Francis McGregor-Macdonald + (francis@mc-mac.com) +
+
2021-11-15 15:50:37
 +
+

*Thread Reply:* I am working through AWS native services seeing how they could, can, or do best integrate with openlineage (I’m an AWS SA). Hence the questions on best practice.

+ +

Aligning with the Spark integration sounds like it might make sense then. Is there an example I could build from?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2021-11-15 17:56:17
 +
+

*Thread Reply:* an example of reporting lineage? you can look at the Spark integration here - https://github.com/OpenLineage/OpenLineage/tree/main/integration/spark/

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2021-11-15 17:59:14
 +
+

*Thread Reply:* Ahh, in that case I would have to agree with Michael’s approach to things!

+ + + +
+ ✅ Diogo +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Francis McGregor-Macdonald + (francis@mc-mac.com) +
+
2021-11-19 03:30:03
 +
+

*Thread Reply:* @Michael Collado I am following the Spark integration you recommended (for a Glue job) and while everything appears to be set up correct, I am getting no lineage appear in marquez (a request.get from the pyspark script can reach the endpoint). Is there a way to enable a debug log so I can look to identify where the issue is? +Is there a specific place to look in the regular logs?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2021-11-19 13:39:01
 +
+

*Thread Reply:* listener output should be present in the driver logs. you can turn on debug logging in your log4j config (or whatever logging tool you use) for the package io.openlineage.spark.agent

+ + + +
+ ✅ Francis McGregor-Macdonald +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2021-11-19 19:44:06
 +
+

Woo hoo! Initial Spark <-> Kafka support has been merged 🙂 https://github.com/OpenLineage/OpenLineage/pull/387

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/pull/387|#387 [spark] support read/write to kafka>
+
+ <p><strong>Problem</strong></p> + +<p>On write to kafka in spark exception occurs.</p> + +<p>Closes: #280</p> + +<p><strong>Solution</strong></p> + +<p>Adds kafka read/write support</p> + +<p>☐ Your change modifies the <a href="https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.json">core</a> OpenLineage model +☐ Your change modifies one or more OpenLineage <a href="https://github.com/OpenLineage/OpenLineage/tree/main/spec/facets">facets</a></p> + +<p>If you're contributing a new integration, please specify the scope of the integration and how/where it has been tested (e.g., Apache Spark integration supports <code>S3</code> and <code>GCS</code> filesystem operations, tested with AWS EMR).</p> + +<p><strong>Checklist</strong></p> + +<p>☑︎ You've <a href="https://github.com/OpenLineage/OpenLineage/blob/main/why-the-dco.md">signed-off</a> your work +☑︎ Your pull request title follows our <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CONTRIBUTING.md#creating-pull-requests">guidelines</a> +☑︎ Your changes are accompanied by tests (<em>if relevant</em>) +☐ Your change contains a <a href="https://kurtisnusbaum.medium.com/stacked-diffs-keeping-phabricator-diffs-small-d9964f4dcfa6">small diff</a> and is self-contained +☐ You've updated any relevant documentation (<em>if relevant</em>) +☐ You've updated the <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md"><code>CHANGELOG.md</code></a> with details about your change under the "Unreleased" section (<em>if relevant, depending on the change, this may not be necessary</em>) +☐ You've versioned the core OpenLineage model or facets according to <a href="https://docs.snowplowanalytics.com/docs/pipeline-components-and-applications/iglu/common-architecture/schemaver">SchemaVer</a> (<em>if relevant</em>)</p> +
+ +
+
Comments
+ 1 +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+ 🎉 Willy Lulciuc, John Thomas, Peter Hicks, Maciej Obuchowski +
+ +
+ 🙌 Willy Lulciuc, John Thomas, Francis McGregor-Macdonald, Peter Hicks, Maciej Obuchowski +
+ +
+ 🚀 Willy Lulciuc, John Thomas, Peter Hicks, Francis McGregor-Macdonald, Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Francis McGregor-Macdonald + (francis@mc-mac.com) +
+
2021-11-22 13:32:57
 +
+

I am “successfully” exporting lineage to openlineage from AWS Glue using the listener. Only the source load is showing, not the transforms, or the sink

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Francis McGregor-Macdonald + (francis@mc-mac.com) +
+
2021-11-22 13:34:15
 +
+

*Thread Reply:* Output event:

+ +

2021-11-22 08:12:15,513 INFO [spark-listener-group-shared] agent.OpenLineageContext (OpenLineageContext.java:emit(50)): Lineage completed successfully: ResponseMessage(responseCode=201, body=, error=null) { + “eventType”: “COMPLETE”, + “eventTime”: “2021-11-22T08:12:15.478Z”, + “run”: { + “runId”: “03bfc770-2151-499e-9265-8457a38ceec3”, + “facets”: { + “sparkversion”: { + “producer”: “https://github.com/OpenLineage/OpenLineage/tree/0.3.1/integration/spark”, + “schemaURL”: “https://openlineage.io/spec/1-0-2/OpenLineage.json#/$defs/RunFacet”, + “spark-version”: “3.1.1-amzn-0”, + “openlineage-spark-version”: “0.3.1” + } + } + }, + “job”: { + “namespace”: “sparkintegration”, + “name”: “nyctaxirawstage.mappartitionsunionmappartitionsnew_hadoop” + }, + “inputs”: [ + { + “namespace”: “s3.cdkdl-dev-foundationstoragef3787fa8-raw1d6fb60a-171gwxf2sixt9”, + “name”: “” + } + ], + “outputs”: [], + “producer”: “https://github.com/OpenLineage/OpenLineage/tree/0.3.1/integration/spark”, + “schemaURL”: “https://openlineage.io/spec/1-0-2/OpenLineage.json#/$defs/RunEvent” +}

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Francis McGregor-Macdonald + (francis@mc-mac.com) +
+
2021-11-22 13:34:59
 +
+

*Thread Reply:* This sink record is missing details …

+ +

2021-11-22 08:12:15,481 INFO [Thread-7] sinks.HadoopDataSink (HadoopDataSink.scala:$anonfun$writeDynamicFrame$1(275)): nameSpace: , table:

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Francis McGregor-Macdonald + (francis@mc-mac.com) +
+
2021-11-22 13:40:30
 +
+

*Thread Reply:* I can also see multiple history events (presumably for each transform, each as above) emitted for the same Glue Job, with different RunId, with the same inputs and the same (null) output.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2021-11-22 14:31:06
 +
+

*Thread Reply:* Are you using the existing spark integration for the spark lineage?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Francis McGregor-Macdonald + (francis@mc-mac.com) +
+
2021-11-22 14:46:47
 +
+

*Thread Reply:* I followed: https://github.com/OpenLineage/OpenLineage/tree/main/integration/spark +In the Glue context I was not clear on the correct settings for “spark.openlineage.parentJobName” and “spark.openlineage.parentRunId”, I put in static values (which may be incorrect)? +I injected these via: "--conf": "spark.openlineage.parentJobName=nyc-taxi-raw-stage",

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Francis McGregor-Macdonald + (francis@mc-mac.com) +
+
2021-11-22 14:47:54
 +
+

*Thread Reply:* Happy to share what is working when I am done, I can’t seem to find an AWS Glue specific example to walk me through.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2021-11-22 15:03:31
 +
+

*Thread Reply:* yeah, We haven’t spent any significant time with AWS Glue, but we just released the Databricks integration, which might help guide the way you’re working a little bit more

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Francis McGregor-Macdonald + (francis@mc-mac.com) +
+
2021-11-22 15:12:15
 +
+

*Thread Reply:* from what I can see in the DBX integration (https://github.com/OpenLineage/OpenLineage/tree/main/integration/spark/databricks) all of what is being done here I am doing in Glue (upload the jar, embed the settings into the Glue spark job). +It is emitting the above for each transform in the Glue job, but does not seem to capture the output …

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Francis McGregor-Macdonald + (francis@mc-mac.com) +
+
2021-11-22 15:13:54
 +
+

*Thread Reply:* Is there a standard Spark test script in use with openlineage I could put into Glue to test without using any Glue specific functionality (without for example the GlueContext, or Glue dynamic frames)?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Francis McGregor-Macdonald + (francis@mc-mac.com) +
+
2021-11-22 15:25:30
 +
+

*Thread Reply:* The initialisation does appear to be working if I compare it to the DBX README +Mine from AWS Glue… +21/11/22 18:48:48 INFO SparkContext: Registered listener io.openlineage.spark.agent.OpenLineageSparkListener +21/11/22 18:48:49 INFO OpenLineageContext: Init OpenLineageContext: Args: ArgumentParser(host=<http://ec2>-….<a href="http://compute-1.amazonaws.com:5000">compute-1.amazonaws.com:5000</a>, version=v1, namespace=spark_integration, jobName=default, parentRunId=null, apiKey=Optional.empty) URI: <http://ec2>-….<a href="http://compute-1.amazonaws.com:5000/api/v1/lineage">compute-1.amazonaws.com:5000/api/v1/lineage</a> +21/11/22 18:48:49 INFO AsyncEventQueue: Process of event SparkListenerApplicationStart(nyc-taxi-raw-stage,Some(spark-application-1637606927106),1637606926281,spark,None,None,None) by listener OpenLineageSparkListener took 1.092252643s.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2021-11-22 16:12:40
 +
+

*Thread Reply:* We don’t have a test run, unfortunately, but you could follow this blog post’s processes in each and see what the differences are? https://openlineage.io/blog/openlineage-spark/

+
+
openlineage.io
+ + + +
Tracing Data Lineage with OpenLineage and Apache Spark
+
+ Spark ushered in a brand new age of data democratization... and left us with a mess of hidden dependencies, stale datasets, and failed jobs. +
+ + + + + + +
Original URL: https://openlineage.io/blog/openlineage-spark/
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Francis McGregor-Macdonald + (francis@mc-mac.com) +
+
2021-11-22 16:43:23
 +
+

*Thread Reply:* Thanks, I have been looking at that. I will create a Glue job aligned with that. What is the best way to pass feedback? Keep it here?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2021-11-22 16:49:50
 +
+

*Thread Reply:* yeah, this thread will work great 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ilya Davidov + (idavidov@marpaihealth.com) +
+
2022-07-18 11:37:02
 +
+

*Thread Reply:* @Francis McGregor-Macdonald are you managed to enable it?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Francis McGregor-Macdonald + (francis@mc-mac.com) +
+
2022-07-18 15:14:47
 +
+

*Thread Reply:* Just DM you the code I used a while back (app.py + CDK code). I haven’t used it in a while, and there is some duplication in it. I had openlineage enabled, but dynamic frames not working yet with lineage. Let me know how you go. +I haven’t had the space to look at it in a while, but happy to support if you are looking at it.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Dinakar Sundar + (dinakar_sundar@condenast.com) +
+
2021-11-23 08:48:51
 +
+

how to use the Open lineage with amundsen ?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-11-23 09:01:11
 +
+

*Thread Reply:* You can use this: https://github.com/amundsen-io/amundsen/pull/1444

+
+ + + + +
<https://github.com/amundsen-io/amundsen/pull/1444|#1444 feat: OpenLineage extractor for databuilder>
+
+ <p>Signed-off-by: dechoma <a href="mailto:dominik.choma@gmail.com">dominik.choma@gmail.com</a></p> + +<p><strong>Summary of Changes</strong></p> + +<p>Databulder extractor to extract lineage information from OpenLineage format <br /> +<a href="https://github.com/OpenLineage/OpenLineage">https://github.com/OpenLineage/OpenLineage</a></p> + +<p><strong>Tests</strong> +<strong>Documentation</strong> +<strong>CheckList</strong></p> + +<p>Make sure you have checked <strong>all</strong> steps below to ensure a timely review.</p> + +<p>☐ PR title addresses the issue accurately and concisely. Example: "Updates the version of Flask to v1.0.2" + • In case you are adding a dependency, check if the license complies with the <a href="https://www.apache.org/legal/resolved.html#category-x">ASF 3rd Party License Policy</a>. +☐ PR includes a summary of changes. +☐ PR adds unit tests, updates existing unit tests, <strong>OR</strong> documents why no test additions or modifications are needed. +☐ In case of new functionality, my PR adds documentation that describes how to use it. + • All the public functions and the classes in the PR contain docstrings that explain what it does</p> +
+ +
+
Labels
+ area:databuilder, area:dev-tools, area:docs +
+ +
+
Comments
+ 2 +
+ + + + + + + +
+ + <a href="https://github.com/amundsen-io/amundsen">amundsen-io/amundsen</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2021-11-23 09:38:44
 +
+

*Thread Reply:* you can also check out this section from the Amundsen Community Meeting in october: https://www.youtube.com/watch?v=7WgECcmLSRk

+
+
YouTube
+ +
+ + + } + + Amundsen + (https://www.youtube.com/channel/UCgOyzG0sEoolxuC9YXDYPeg) +
+ + + +
Pandas profiling with Amundsen &amp; OpenLineage integration with Amundsen - Oct 2021 community meeting
+
+ +
+ + + + + + + + +
Original URL: https://www.youtube.com/watch?v=7WgECcmLSRk
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Dinakar Sundar + (dinakar_sundar@condenast.com) +
+
2021-11-23 08:49:16
 +
+

do we need to use the Marquez ?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2021-11-23 12:45:34
 +
+

*Thread Reply:* No, I believe the databuilder OpenLineage extractor for Amundsen will continue to store lineage metadata in Atlas

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2021-11-23 12:47:01
 +
+

*Thread Reply:* We've spoken to the Amundsen team, and though using Marquez to store lineage metadata isn't an option, it's an integration that makes sense but hasn't yet been prioritized

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Dinakar Sundar + (dinakar_sundar@condenast.com) +
+
2021-11-23 13:51:00
 +
+

*Thread Reply:* Thanks , Right now amundsen has no support for lineage extraction from spark or airflow , if this case do we need to use marquez for open lineage implementation to capture the lineage from airflow & spark

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2021-11-23 13:57:13
 +
+

*Thread Reply:* Maybe, that would mean running the full Amundsen stack as well as the Marquez stack along side each other (not ideal). The OpenLineage integration for Amundsen is very recent, so haven't had a chance to look deeply into the implementation. But, briefly looking over the config for Openlineagetablelineageextractor, you can only send metadata to Atlas

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Dinakar Sundar + (dinakar_sundar@condenast.com) +
+
2021-11-24 00:36:56
 +
+

*Thread Reply:* @Willy Lulciuc thats our real concern , running the two stacks will make a mess environment , let me explain our amundsen setup , we are having neo4j as backend , (front end , search service , metadata service,elastic search & neo4j) . our requirement to capture lineage from spark and airflow , imported into amundsen

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Vinith Krishnan US + (vinithk@nvidia.com) +
+
2022-03-11 22:33:39
 +
+

*Thread Reply:* We are running into a similar issue. @Dinakar Sundar were you able to get the Amundsen OpenLineage integration to work with a neo4j backend?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
bitsofinfo + (bitsofinfo.g@gmail.com) +
+
2021-11-24 11:41:31
 +
+

Hi all - i just watched the presentation on this and Marquez from the Airflow 21 summit. I was pretty impressed with this. My question is what other open source players are in this space or are pretty much people consolidating around this? (which would be great). Was looking at the available datasource extractors for the airflow side and would hope to see more here, looking at the code doesn't seem like too huge of a deal. Is there a roadmap available?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-11-24 11:49:14
 +
+

*Thread Reply:* You can take a look at https://github.com/OpenLineage/OpenLineage/projects

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Martin Fiser + (fisa@keboola.com) +
+
2021-11-24 19:24:48
 +
+

Hi all, I was wondering what is the status of native support of openlineage for DataHub or Amundzen. re https://openlineage.slack.com/archives/C01CK9T7HKR/p1633633476151000?thread_ts=1633008095.115900&cid=C01CK9T7HKR +Many thanks!

+
+ + +
+ + + } + + Julien Le Dem + (https://openlineage.slack.com/team/U01DCLP0GU9) +
+ + + +
+
+ <p>Those two specs look compatible, so Datahub should be able to consume this lineage metadata in the future</p> +
+ + + + + + +
Original URL: https://openlineage.slack.com/archives/C01CK9T7HKR/p1633633476151000?thread_ts=1633008095.115900&amp;cid=C01CK9T7HKR
+ + +
+ + Thread in Slack Conversation +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Martin Fiser + (fisa@keboola.com) +
+
2021-12-01 16:35:17
 +
+

*Thread Reply:* Anyone? Thanks!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Dinakar Sundar + (dinakar_sundar@condenast.com) +
+
2021-11-25 01:42:26
 +
+

our amundsen setup , we are having neo4j as backend , (front end , search service , metadata service,elastic search & neo4j) . our requirement to capture lineage from spark and airflow , imported into amundsen ?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2021-11-29 23:30:12
 +
+

Hello, OpenLineage folks - I'm curious if anyone here has ran into an issue like we're running into as we look to extend OpenLineage's Spark integration into Databricks.

+ +

Has anyone ran into an issue where a scala class should exist (based on a decompiled jar, I see that it's a public class) but you keep getting an error like object SqlDWRelation in package sqldw cannot be accessed in package com.databricks.spark.sqldw?

+ +

Databricks has a Synapse SQL DW connector: https://docs.databricks.com/data/data-sources/azure/synapse-analytics.html

+ +

I want to extract the database URL, table, and schema from the logical plan but

+ +

I execute something like the below command that runs a SELECT ** on the given tableName ("borrower" in this case) in the Azure Synapse database.

+ +

val df = spark.read.format("com.databricks.spark.sqldw") +.option("url", sqlDwUrl) +.option("tempDir", tempDir) +.option("forwardSparkAzureStorageCredentials", "true") +.option("dbTable", tableName) +.load() +val logicalPlan = df.queryExecution.logical +val logicalRelation = logicalPlan.asInstanceOf[LogicalRelation] +val sqlBaseRelation = logicalRelation.relation +I end up with something like this, all good so far: +```logicalPlan: org.apache.spark.sql.catalyst.plans.logical.LogicalPlan = +Relation[memberId#97,residentialState#98,yearsEmployment#99,homeOwnership#100,annualIncome#101,incomeVerified#102,dtiRatio#103,lengthCreditHistory#104,numTotalCreditLines#105,numOpenCreditLines#106,numOpenCreditLines1Year#107,revolvingBalance#108,revolvingUtilizationRate#109,numDerogatoryRec#110,numDelinquency2Years#111,numChargeoff1year#112,numInquiries6Mon#113] SqlDWRelation("borrower")

+ +

logicalRelation: org.apache.spark.sql.execution.datasources.LogicalRelation = +Relation[memberId#97,residentialState#98,yearsEmployment#99,homeOwnership#100,annualIncome#101,incomeVerified#102,dtiRatio#103,lengthCreditHistory#104,numTotalCreditLines#105,numOpenCreditLines#106,numOpenCreditLines1Year#107,revolvingBalance#108,revolvingUtilizationRate#109,numDerogatoryRec#110,numDelinquency2Years#111,numChargeoff1year#112,numInquiries6Mon#113] SqlDWRelation("borrower")

+ +

sqlBaseRelation: org.apache.spark.sql.sources.BaseRelation = SqlDWRelation("borrower")`` +Schema, I can easily get withsqlBaseRelation.schema` but I cannot figure out:

+ +
  1. How I can get the database name from the logical relation
  2. How I can get the table name from the logical relation ("borrower" is the table name so I can always parse the string if necessary" +I know that Databricks has the SqlDWRelation class which I think I need to cast the BaseRelation to BUT it appears to be in a jar / package that is inaccessible during the execution of a notebook. Specifically import com.databricks.spark.sqldw.SqlDWRelation is the relation and it appears to have a few accessors that would help me answer some of these questions: params and JDBCWrapper
    3. +
+ +

Of course this is undocumented on the Databricks side 😰

+ +

If I could cast the BaseRelation into this SqlDWRelation, I'd be able to get this info. However, whenever I attempt to use the imported SqlDWRelation, I get an error object SqlDWRelation in package sqldw cannot be accessed in package com.databricks.spark.sqldw I'm hoping someone has run into something similar in the past on the Spark / Databricks / Scala side and might share some advice. Thank you for any guidance!

+
+
docs.databricks.com
+ + + +
Azure Synapse Analytics | Databricks on AWS
+
+ Learn how to read and write data to Azure Synapse Analytics (formerly SQL Data Warehouse) using Databricks. +
+ + + + + + +
Original URL: https://docs.databricks.com/data/data-sources/azure/synapse-analytics.html
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-11-30 07:03:30
 +
+

*Thread Reply:* Have you tried reflection? https://commons.apache.org/proper/commons-lang/apidocs/org/apache/commons/lang3/reflect/FieldUtils.html#getDeclaredField-jav[…].lang.String-boolean-

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2021-11-30 11:21:34
 +
+

*Thread Reply:* I have not! Will give it a try, Maciej! Thank you for the reply!

+ + + +
+ 🙌 Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2021-11-30 15:20:18
 +
+

*Thread Reply:* 🙏 @Maciej Obuchowski we're not worthy! That was the magic we needed. Seems like a hack since we're snooping in on private classes but if it works...

+ +

Thank you so much for pointing to those utilities!

+ + + +
+ ❤️ Julien Le Dem +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-11-30 15:48:25
 +
+

*Thread Reply:* Glad I could help!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Francis McGregor-Macdonald + (francis@mc-mac.com) +
+
2021-11-30 19:43:03
 +
+

A colleague pointed me at https://open-metadata.org/, is there anywhere a view or comparison of this and openlineage?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mario Measic + (mario.measic.gavran@gmail.com) +
+
2021-12-01 08:51:28
 +
+

*Thread Reply:* Different concepts. OL is focused on describing the lineage and metadata of the running jobs. So it keeps track of all the metadata (schema, ...) of inputs and outputs at the time transformation occurs + transformation metadata (code version, cost, etc.)

+ +

OM I am not an expert but it's a metadata model with clients and API around it.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
RamanD + (romantanzar@gmail.com) +
+
2021-12-01 12:33:51
 +
+

Hey! OpenLineage is a beautiful initiative, to be honest! We also try to accommodate it. One question, maybe it's already described somewhere then many apologies :) if we need to propagate run id from Airflow to a child task (AWS Batch job, for instance) what will be the best way to do it in the current realization (as we get run id only at post execute phase)?.. We use Airflow 2+ integration.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-12-01 12:40:53
 +
+

*Thread Reply:* Hey. For technical reasons, we can't automatically register macro that does this job, as we could in Airflow 1 integration. You could put it yourself:

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-12-01 12:41:02
 +
+

*Thread Reply:* ```def lineageparentid(run_id, task): + """ + Macro function which returns the generated job and run id for a given task. This + can be used to forward the ids from a task to a child run so the job + hierarchy is preserved. Child run can create ParentRunFacet from those ids. + Invoke as a jinja template, e.g.

+ +
PythonOperator(
+    task_id='render_template',
+    python_callable=my_task_function,
+    op_args=['{{ lineage_parent_id(run_id, task) }}'], # lineage_run_id macro invoked
+    provide_context=False,
+    dag=dag
+)
+
+:param run_id:
+:param task:
+:return:
+"""
+with create_session() as session:
+    job_name = openlineage_job_name(task.dag_id, task.task_id)
+    ids = JobIdMapping.get(job_name, run_id, session)
+    if ids is None:
+        return ""
+    elif isinstance(ids, list):
+        run_id = "" if len(ids) == 0 else ids[0]
+    else:
+        run_id = str(ids)
+    return f"{_DAG_NAMESPACE}/{job_name}/{run_id}"
+
+ +

def openlineagejobname(dagid: str, taskid: str) -> str: + return f'{dagid}.{taskid}'```

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-12-01 12:41:13
 +
+

*Thread Reply:* from here: https://github.com/OpenLineage/OpenLineage/blob/main/integration/airflow/openlineage/airflow/dag.py#L77

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/integration/airflow/openlineage/airflow/dag.py | dag.py>
+
+ <pre><code>def lineage_parent_id(run_id, task): +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
RamanD + (romantanzar@gmail.com) +
+
2021-12-01 12:53:27
 +
+

*Thread Reply:* the quickest response ever! And that works like a charm 🙌

+ + + +
+ 👍 Michael Collado +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-12-01 13:21:16
 +
+

*Thread Reply:* Glad I could help!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2021-12-01 14:14:23
 +
+

@Maciej Obuchowski and @Michael Collado given your work on the Spark Integration, what's the right way to explore the Write operations' logical plans? When doing a read, it's easy! In scala df.queryExecution.logical gives you exactly what you need but how do you guys interactively explore what sort of commands are being used during a write? We are exploring some of the DataSourceV2 data sources and are hoping to learn from you guys a bit more, please 😃

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-12-01 14:18:00
 +
+

*Thread Reply:* For SQL, EXPLAIN EXTENDED and show() in scala-shell is helpful:

+ +

spark.sql("EXPLAIN EXTENDED CREATE TABLE tbl USING delta LOCATION '/tmp/delta' AS SELECT ** FROM tmp").show(false) +```|== Parsed Logical Plan == +'CreateTableAsSelectStatement [tbl], delta, /tmp/delta, false, false ++- 'Project [**] + +- 'UnresolvedRelation [tmp], [], false

+ +

== Analyzed Logical Plan ==

+ +

CreateTableAsSelect org.apache.spark.sql.delta.catalog.DeltaCatalog@63c5b63a, default.tbl, [provider=delta, location=/tmp/delta], false ++- Project [x#12, y#13] + +- SubqueryAlias tmp + +- LocalRelation [x#12, y#13]

+ +

== Optimized Logical Plan == +CreateTableAsSelect org.apache.spark.sql.delta.catalog.DeltaCatalog@63c5b63a, default.tbl, [provider=delta, location=/tmp/delta], false ++- LocalRelation [x#12, y#13]

+ +

== Physical Plan == +AtomicCreateTableAsSelect org.apache.spark.sql.delta.catalog.DeltaCatalog@63c5b63a, default.tbl, LocalRelation [x#12, y#13], [provider=delta, location=/tmp/delta, owner=mobuchowski], [], false ++- LocalTableScan [x#12, y#13] +|```

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-12-01 14:27:25
 +
+

*Thread Reply:* For dataframe api, I'm usually just either logging plan to console from OpenLineage listener, or looking at sparklogicalPlan or sparkunknown facets send by listener - even when the particular write operation isn't supported by integration, those facets should have some relevant info.

+ + + +
+ 🙌 Will Johnson +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-12-01 14:27:40
 +
+

*Thread Reply:* For example, for the query I've send at comment above, the spark_logicalPlan facet looks like this:

+ +

"spark.logicalPlan": { + "_producer": "<https://github.com/OpenLineage/OpenLineage/tree/0.4.0-SNAPSHOT/integration/spark>", + "_schemaURL": "<https://openlineage.io/spec/1-0-2/OpenLineage.json#/$defs/RunFacet>", + "plan": [ + { + "allowExisting": false, + "child": [ + { + "class": "org.apache.spark.sql.catalyst.plans.logical.LocalRelation", + "data": null, + "isStreaming": false, + "num-children": 0, + "output": [ + [ + { + "class": "org.apache.spark.sql.catalyst.expressions.AttributeReference", + "dataType": "integer", + "exprId": { + "id": 2, + "jvmId": "e03e2860-a24b-41f5-addb-c35226173f7c", + "product-class": "org.apache.spark.sql.catalyst.expressions.ExprId" + }, + "metadata": {}, + "name": "x", + "nullable": false, + "num-children": 0, + "qualifier": [] + } + ], + [ + { + "class": "org.apache.spark.sql.catalyst.expressions.AttributeReference", + "dataType": "integer", + "exprId": { + "id": 3, + "jvmId": "e03e2860-a24b-41f5-addb-c35226173f7c", + "product-class": "org.apache.spark.sql.catalyst.expressions.ExprId" + }, + "metadata": {}, + "name": "y", + "nullable": false, + "num-children": 0, + "qualifier": [] + } + ] + ] + } + ], + "class": "org.apache.spark.sql.execution.command.CreateViewCommand", + "name": { + "product-class": "org.apache.spark.sql.catalyst.TableIdentifier", + "table": "tmp" + }, + "num-children": 0, + "properties": null, + "replace": true, + "userSpecifiedColumns": [], + "viewType": { + "object": "org.apache.spark.sql.catalyst.analysis.LocalTempView$" + } + } + ] + },

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2021-12-01 14:38:55
 +
+

*Thread Reply:* Okay! That is very helpful! I wasn't sure if there was a fancier trick but I can definitely do logging 🙂 Our challenge was that our proprietary packages were resulting in Null Pointer Exceptions when it tried to push to OpenLineage 😞

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2021-12-01 14:39:02
 +
+

*Thread Reply:* Thank you as usual!!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2021-12-01 14:40:25
 +
+

*Thread Reply:* You can always add test cases and add breakpoints to debug in your IDE. That doesn't work for the container tests, but it does work for the other ones

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2021-12-01 14:47:20
 +
+

*Thread Reply:* Ah! That's a great point! I definitely would appreciate being able to poke at the objects interactively in a debug mode. Thank you for the guidance as well!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ricardo Gaspar + (ricardogaspar2@gmail.com) +
+
2021-12-03 11:49:10
 +
+

hi everyone! 👋 +Very noob question here: I’ve been wanting to play with Marquez and open lineage for my company’s projects. I use mostly scala & spark, but also Airflow. +I’ve been reading and watching talks about OpenLineage and Marquez. +So far i didn’t quite discover if Marquez or OpenLineage does field-level lineage (with Spark), like spline tries to.

+ +

Any idea?

+ +

Other sources about this topic +• https://medium.com/cdapio/data-integration-with-field-level-lineage-5d9986524316 +• https://medium.com/cdapio/field-level-lineage-part-1-3cc5c9e1d8c6 +• https://medium.com/cdapio/designing-field-level-lineage-part-2-b6c7e6af5bf4 +• https://www.youtube.com/playlist?list=PL897MHVe_nHeEQC8UnCfXecmZdF0vka_T +• https://www.youtube.com/watch?v=gKYGKXIBcZ0 +• https://www.youtube.com/watch?v=eBep6rRh7ic

+
+
Medium
+ + + +
Data Integration with Field Level Lineage
+
+ Arguably, data is the most important asset that an organization owns, and in today’s climate of data security and stewardship it’s more… +
+ +
+
Reading time
+ 6 min read +
+ + + + + + + + +
Original URL: https://medium.com/cdapio/data-integration-with-field-level-lineage-5d9986524316
+ + + +
+
+
Medium
+ + + +
Field Level Lineage — part 1
+
+ How the CDAP team designed a tool to track the journey of data in enterprise production environments. +
+ +
+
Reading time
+ 6 min read +
+ + + + + + + + +
Original URL: https://medium.com/cdapio/field-level-lineage-part-1-3cc5c9e1d8c6
+ + + +
+
+
Medium
+ + + +
Designing field Level Lineage — part 2
+
+ How the CDAP team designed a tool to track the journey of data in enterprise production environments. +
+ +
+
Reading time
+ 10 min read +
+ + + + + + + + +
Original URL: https://medium.com/cdapio/designing-field-level-lineage-part-2-b6c7e6af5bf4
+ + + +
+
+
YouTube
+ + + +
CDAP How To’s
+
+ CDAP How To Videos +
+ + + + + + + + +
Original URL: https://www.youtube.com/playlist?list=PL897MHVe_nHeEQC8UnCfXecmZdF0vka_T
+ + + +
+
+
YouTube
+ +
+ + + } + + CDAP + (https://www.youtube.com/c/CDAPio) +
+ + + +
CDAP Community Meetup Maintaining full data lineage and governance across billions of data partition
+
+ +
+ + + + + + + + +
Original URL: https://www.youtube.com/watch?v=gKYGKXIBcZ0
+ + + +
+ + + +
+ 🙌 Francis McGregor-Macdonald +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2021-12-03 11:55:17
 +
+

*Thread Reply:* Hi Ricardo - OpenLineage doesn’t currently have support for field-level lineage, but it’s definitely something we’ve been looking into. This is a great collection of resources 🙂

+ +

We’ve to-date been working on our integrations library, making it as easy to set up as possible.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ricardo Gaspar + (ricardogaspar2@gmail.com) +
+
2021-12-03 12:01:25
 +
+

*Thread Reply:* Thanks John! I was checking the issues on github and other posts here. Just wanted to clarify that. +I’ll keep an eye on it

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-12-06 20:25:19
 +
+

The next OpenLineage monthly meeting is this Wednesday at 9am PT. (everybody is welcome to join) +The slides are here: https://docs.google.com/presentation/d/1q2Be7WTKlIhjLPgvH-eXAnf5p4w7To9v/edit#slide=id.ge4b57c6942_0_75 +tentative agenda: +• SPDX headers [Mandy Chessel] +• Azure Purview + OpenLineage [Will Johnson, Mark Taylor] +• Logging backend (OpenTelemetry, ...) [Julien Le Dem] +• Open discussion +Please chime in in this thread if you’d want to add something

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-12-06 20:28:09
 +
+

*Thread Reply:* The link to join the meeting is on the wiki: https://wiki.lfaidata.foundation/display/OpenLineage/Monthly+TSC+meeting

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-12-06 20:28:25
 +
+

*Thread Reply:* Please reach out to me if you’d like to be added to a gcal invite

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Dinakar Sundar + (dinakar_sundar@condenast.com) +
+
2021-12-06 22:37:29
 +
+

@John Thomas we in Condenast currently exploring the features of open lineage to integrate to databricks , https://github.com/OpenLineage/OpenLineage/tree/main/integration/spark/databricks , spark configuration not working ,

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2021-12-08 02:03:37
 +
+

*Thread Reply:* Hi Dinakar. Can you give some specifics regarding what kind of problem you're running into?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Dinakar Sundar + (dinakar_sundar@condenast.com) +
+
2021-12-09 10:15:50
 +
+

*Thread Reply:* Hi @Michael Collado, were able to set the spark configuration for spark extra listener & placed jars as well , wen i ran the sapark job , Lineage is not get tracked into the marquez

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Dinakar Sundar + (dinakar_sundar@condenast.com) +
+
2021-12-09 10:34:39
 +
+

*Thread Reply:* {"producer":"https://github.com/OpenLineage/OpenLineage/tree/0.3.1/integration/spark","schemaURL":"https://github.com/OpenLineage/OpenLineage/tree/0.3.1/integration/spark/facets/spark/v1/output-statistics-facet.json","rowCount":0,"size":-1,"status":"DEPRECATED"}},"outputFacets":{"outputStatistics":{"producer":"https://github.com/OpenLineage/OpenLineage/tree/0.3.1/integration/spark","schemaURL":"https://openlineage.io/spec/facets/1-0-0/OutputStatisticsOutputDatasetFacet.json#/$defs/OutputStatisticsOutputDatasetFacet","rowCount":0,"size":-1}}}],"producer":"https://github.com/OpenLineage/OpenLineage/tree/0.3.1/integration/spark","schemaURL":"https://openlineage.io/spec/1-0-2/OpenLineage.json#/$defs/RunEvent"} +OpenLineageHttpException(code=0, message=java.lang.IllegalArgumentException: Cannot construct instance of io.openlineage.spark.agent.client.HttpError (although at least one Creator exists): no String-argument constructor/factory method to deserialize from String value ('{"code":404,"message":"HTTP 404 Not Found"}') + at [Source: UNKNOWN; line: -1, column: -1], details=java.util.concurrent.CompletionException: java.lang.IllegalArgumentException: Cannot construct instance of io.openlineage.spark.agent.client.HttpError (although at least one Creator exists): no String-argument constructor/factory method to deserialize from String value ('{"code":404,"message":"HTTP 404 Not Found"}') + at [Source: UNKNOWN; line: -1, column: -1]) + at io.openlineage.spark.agent.OpenLineageContext.emit(OpenLineageContext.java:48) + at io.openlineage.spark.agent.lifecycle.SparkSQLExecutionContext.start(SparkSQLExecutionContext.java:122) + at io.openlineage.spark.agent.OpenLineageSparkListener.lambda$onJobStart$3(OpenLineageSparkListener.java:159) + at java.util.Optional.ifPresent(Optional.java:159) + at io.openlineage.spark.agent.OpenLineageSparkListener.onJobStart(OpenLineageSparkListener.java:148) + at org.apache.spark.scheduler.SparkListenerBus.doPostEvent(SparkListenerBus.scala:37) + at org.apache.spark.scheduler.SparkListenerBus.doPostEvent$(SparkListenerBus.scala:28) + at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37) + at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37) + at org.apache.spark.util.ListenerBus.postToAll(ListenerBus.scala:119) + at org.apache.spark.util.ListenerBus.postToAll$(ListenerBus.scala:103) + at org.apache.spark.scheduler.AsyncEventQueue.super$postToAll(AsyncEventQueue.scala:105) + at org.apache.spark.scheduler.AsyncEventQueue.$anonfun$dispatch$1(AsyncEventQueue.scala:105) + at scala.runtime.java8.JFunction0$mcJ$sp.apply(JFunction0$mcJ$sp.java:23) + at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62) + at org.apache.spark.scheduler.AsyncEventQueue.org$apache$spark$scheduler$AsyncEventQueue$$dispatch(AsyncEventQueue.scala:100) + at org.apache.spark.scheduler.AsyncEventQueue$$anon$2.$anonfun$run$1(AsyncEventQueue.scala:96) + at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1585) + at org.apache.spark.scheduler.AsyncEventQueue$$anon$2.run(AsyncEventQueue.scala:96)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Dinakar Sundar + (dinakar_sundar@condenast.com) +
+
2021-12-09 13:29:42
 +
+

*Thread Reply:* Issue solved , mentioned the version wrongly as 1 instead v1

+ + + +
+ 🙌 Michael Collado +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jitendra Sharma + (jitendra_sharma@condenast.com) +
+
2021-12-07 02:07:06
 +
+

👋 Hi everyone!

+ + + +
+ 👋 Willy Lulciuc, Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
kavuri raghavendra + (kavuri.raghavendra@gmail.com) +
+
2021-12-08 05:37:44
 +
+

Hello Everyone.. we are exploring Openlineage for capturing Spark lineage.. but form the GitHub(https://github.com/OpenLineage/OpenLineage/tree/main/integration/spark) ..I see that the output send to API (Marquez).. how can I send it to Kafka topic.. can some body please guide me on this.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Minkyu Park + (minkyu@datakin.com) +
+
2021-12-08 12:15:38
 +
+

*Thread Reply:* https://github.com/OpenLineage/OpenLineage/pull/400/files

+ +

there’s ongoing PR for proxy backend, which opens http API and redirects events to Kafka.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2021-12-08 12:17:38
 +
+

*Thread Reply:* Hi Kavuri, as minkyu said, there's currently work going on to simplify this process.

+ +

For now, you'll need to make something to capture the HTTP api events and send them to the Kafka topic. Changing the spark.openlineage.url parameter will send the runEvents wherever you like, but obviously you can't directly produce HTTP events to a topic

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
kavuri raghavendra + (kavuri.raghavendra@gmail.com) +
+
2021-12-08 22:13:09
 +
+

*Thread Reply:* Many Thanks for the Reply.. As I understand, currently pushing lineage to kafka topic is not yet there. it is under implementation. If you can help me out in understanding in which version it is going to be present, that will help me a lot. Thanks in advance.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Minkyu Park + (minkyu@datakin.com) +
+
2021-12-09 12:57:10
 +
+

*Thread Reply:* Not sure about the release plan, but the http endpoint is just regular RESTful API, and you will be able to write a super simple proxy for your own use case if you want.

+ + + +
+ 🙌 Will Johnson +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2021-12-12 00:13:54
 +
+

Hi, Open Lineage team - For the Spark Integration, I'm looking to extract information from a DataSourceV2 data source.

+ +

I'm working on the WRITE side of the data source and right now I'm touching the AppendData logical plan (I can't find the Java Doc): https://github.com/rdblue/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicLogicalOperators.scala#L446

+ +

I was able to extract out the table name (from the named relation) but I'm struggling getting out the schema next.

+ +

I noticed that the AppendData offers inputSet, schema, and outputSet. +• inputSet gives me an AttributeSet which does contain the names of my columns (https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/AttributeSet.scala#L69) +• schema returns an empty StructType +• outputSet is an empty AttributeSet +I thought I read in the Spark Internals book that outputSet would only be populated if there was some sort of change to the DataFrame columns but I cannot find that page and searching for spark outputSet turns up few relevant results.

+ +

Has anyone else worked with the AppendData plan and gotten the schema out of it? Am I going down the wrong path with this snippet of code below? Thank you for any guidance!

+ +

if (logical instanceof AppendData) { + AppendData appendOp = (AppendData) logical; + NamedRelation namedRel = appendOp.table(); + <a href="http://log.info">log.info</a>(namedRel.name()); // Works great! + <a href="http://log.info">log.info</a>(appendOp.inputSet().toString());// This will get you a rough schema + StructType schema = appendOp.schema(); // This is an empty StructType + <a href="http://log.info">log.info</a>(schema.json()); // Nothing useful here + }

+
+ + + + +
<https://github.com/rdblue/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicLogicalOperators.scala | basicLogicalOperators.scala>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/rdblue/spark">rdblue/spark</a> +
+ + +
+
+ + + + +
<https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/AttributeSet.scala | AttributeSet.scala>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/apache/spark">apache/spark</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-12-12 07:34:13
 +
+

*Thread Reply:* One thing, you're looking at Ryan's fork of Spark, which is few thousand commits behind head 🙂

+ +

This one should be good: +https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/v2Commands.scala#L72

+ +

About schema: looking at AppendData's query schema should work, if there's no change to columns, because to pass analysis, data being inserted have to match table's schema. I would test that though 🙂

+ +

On the other hand, current AppendDataVisitor just looks at AppendData's table and tries to extract dataset from it using list of common output visitors:

+ +

https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/src/main/co[…]o/openlineage/spark/agent/lifecycle/plan/AppendDataVisitor.java

+ +

In this case, the DataSourceV2RelationVisitor would look at it, provided we're using Spark 3:

+ +

https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/src/main/sp[…]ge/spark3/agent/lifecycle/plan/DataSourceV2RelationVisitor.java

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-12-12 07:37:04
 +
+

*Thread Reply:* In this case, we basically need more info about nature of this DataSourceV2Relation, because this is provider-dependent. We have Iceberg in main branch and Delta here: https://github.com/OpenLineage/OpenLineage/pull/393/files#diff-7b66a9bd5905f4ba42914b73a87d834c1321ebcf75137c1e2a2413c0d85d9db6

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2021-12-13 14:54:13
 +
+

*Thread Reply:* Ah! Maciej! As always, thank you! Looking through the DataSourceV2RelationVisitor you provided, it looks like the connector (Azure Cosmos Db) doesn't provide that Provider property 😞 😞 😞

+ +

Is there any other method for determining the type of DataSourceV2Relation?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2021-12-13 14:57:06
 +
+

*Thread Reply:* And, to make sure I close out on my original question, it was as simple as the code that Maciej was using:

+ +

I merely needed to use DataSourceV2Realtion rather than NamedRelation!

+ +

DataSourceV2Relation relation = (DataSourceV2Relation)appendOp.table(); + <a href="http://log.info">log.info</a>(relation.schema().toString()); + <a href="http://log.info">log.info</a>(relation.name());

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-12-15 06:20:31
 +
+

*Thread Reply:* Are we talking about this connector? https://github.com/Azure/azure-sdk-for-java/blob/934200f63dc5bc7d5502a95f8daeb8142[…]/src/main/scala/com/azure/cosmos/spark/ItemsReadOnlyTable.scala

+
+ + + + +
<https://github.com/Azure/azure-sdk-for-java/blob/934200f63dc5bc7d5502a95f8daeb81426bb600e/sdk/cosmos/azure-cosmos-spark_3-1_2-12/src/main/scala/com/azure/cosmos/spark/ItemsReadOnlyTable.scala | ItemsReadOnlyTable.scala>
+
+ <pre><code>private[spark] class ItemsReadOnlyTable(val sparkSession: SparkSession, +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/Azure/azure-sdk-for-java">Azure/azure-sdk-for-java</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-12-15 06:22:05
 +
+

*Thread Reply:* I guess you can use object.getClass.getCanonicalName() to find if the passed class matches the one that Cosmos provider uses.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2021-12-15 09:53:24
 +
+

*Thread Reply:* Yes! That's the one, Maciej! I will give getCanonicalName a try but also make a PR into that repo to get the provider property set up correctly 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2021-12-15 09:53:28
 +
+

*Thread Reply:* Thank you so much!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-12-15 10:09:39
 +
+

*Thread Reply:* Glad to help 😄

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-12-15 10:22:58
 +
+

*Thread Reply:* @Will Johnson could you tell on which commands from https://github.com/OpenLineage/OpenLineage/issues/368#issue-1038510649 you'll be working?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-12-15 10:24:14
 +
+

*Thread Reply:* If any, of course 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2021-12-15 10:49:31
 +
+

*Thread Reply:* From all of our tests on that Cosmos connector, it looks like it strictly uses athe AppendData operation. However @Harish Sune is looking at more of these commands from a Delta data source.

+ + + +
+ 👍 Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2021-12-22 22:43:34
 +
+

*Thread Reply:* Just to close the loop on this one - I submitted a PR for the work we've been doing. Looking forward to any feedback! https://github.com/OpenLineage/OpenLineage/pull/450

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/pull/450|#450 [INTEGRATION][SPARK] Support Azure Cosmos DB Spark Connector in Open Lineage>
+
+ <p><strong>Problem</strong></p> + +<p>We would like to extend OpenLineage's Spark integration to work with the <a href="https://github.com/Azure/azure-sdk-for-java/tree/main/sdk/cosmos/azure-cosmos-spark_3-1_2-12">Azure Cosmos DB Spark Connector for Spark3</a>.</p> + +<p>This would enable people using Spark, OpenLineage, and Cosmos DB to observe the input and output to their spark jobs.</p> + +<p>Closes: <a href="https://github.com/OpenLineage/OpenLineage/issues/449">#449</a></p> + +<p><strong>Solution</strong></p> + +<p>The solution requires extending the DataSourceV2RelationVisitor to support the Azure Cosmos DB as a provider.</p> + +<p>• Azure Cosmos DB Spark Connector does not use the provider property on the table but can be determined based on name. + • The Cosmos DB connector defines a Cosmos "table" as <code>com.azure.cosmos.spark.items.&amp;lt;serviceName&amp;gt;.&amp;lt;databaseName&amp;gt;.&amp;lt;collectionName&amp;gt;</code> +• The Azure Cosmos DB write operations are all <code>AppendData</code>. As such, the <code>Spark3VisitorFactoryImpl.java</code> must include <code>DataSourceV2RelationVisitor</code> as one of its common visitors. + • This gets fed into <code>AppendData</code> so the underlying relation can be tested against a <code>DataSourceV2RelationVisitor</code>. +• When reading Cosmos DB, Spark uses a <code>DataSourceV2ScanRelation</code>. + • This class has a <code>relation</code> property which is actually <code>DataSourceV2Relation</code>. + • Since it was that simple, rather than making an entirely new SCAN relation class, I hitched it onto the existing DataSourceV2Relation class. +• I followed the same process as Iceberg which does not check if you are using any Iceberg Jars (like the BigQuery integration).</p> + +<p>To support this addition, I added two tests similar to the Delta and Iceberge DataSourceV2Relation tests. I also modified <code>testIsDefinedAtForNonDefinedProvider</code> so that a mock table name was provided since the implementation now checks if the table name matches the pattern for Cosmos DB.</p> + +<p><strong>Scope</strong>: This solution has been tested locally on a Spark-Shell with the <code>--packages "com.azure.cosmos.spark:azure_cosmos_spark_3_1_2_12:4.5.0"</code> parameter along with in Azure Databricks on a Spark 3.1.2 cluster.</p> + +<p><strong>Checklist</strong></p> + +<p>☑︎ You've <a href="https://github.com/OpenLineage/OpenLineage/blob/main/why-the-dco.md">signed-off</a> your work +☑︎ Your pull request title follows our <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CONTRIBUTING.md#creating-pull-requests">guidelines</a> +☑︎ Your changes are accompanied by tests (<em>if relevant</em>) +☑︎ Your change contains a <a href="https://kurtisnusbaum.medium.com/stacked-diffs-keeping-phabricator-diffs-small-d9964f4dcfa6">small diff</a> and is self-contained +☐ You've updated any relevant documentation (<em>if relevant</em>) +☐ You've updated the <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md"><code>CHANGELOG.md</code></a> with details about your change under the "Unreleased" section (<em>if relevant, depending on the change, this may not be necessary</em>) +☐ You've versioned the core OpenLineage model or facets according to <a href="https://docs.snowplowanalytics.com/docs/pipeline-components-and-applications/iglu/common-architecture/schemaver">SchemaVer</a> (<em>if relevant</em>)</p> +
+ +
+
Comments
+ 1 +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-12-23 05:04:36
 +
+

*Thread Reply:* Thanks @Will Johnson! I added one question about dataset naming.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2021-12-14 19:45:59
 +
+

Finally got this doc posted - https://github.com/OpenLineage/OpenLineage/pull/437 (see the readable version here ) +Looking for feedback, @Willy Lulciuc @Maciej Obuchowski @Will Johnson

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/pull/437|#437 Add proposal doc for making Spark event handling code more extensible>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/90321b8b7e44c070efd1f6ba6b4bcafb3b78f5e9/proposals/168/making_spark_visitors_extensible.md | making_spark_visitors_extensible.md>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2021-12-15 10:54:41
 +
+

*Thread Reply:* Yes! This is awesome!! How might this work for an existing command like the DataSourceV2Visitor.

+ +

Right now, OpenLineage checks based on the provider property if it's an Iceberg or Delta provider.

+ +

Ideally, we'd be able to extend the list of providers or have a custom "CosmosDbDataSourceV2Visitor" that knew how to work with a custom DataSourceV2.

+ +

Would that cause any conflicts if the base class is already accounted for in OpenLineage?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-12-15 11:13:20
 +
+

*Thread Reply:* Resolving this would be nice addition to the doc (and, to the implementation) - currently, we're just returning result of first function for which isDefinedAt is satisfied.

+ +

This means, that we can depend on the order of the visitors...

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2021-12-15 13:59:12
 +
+

*Thread Reply:* great question. For posterity, I'd like to move this to the PR discussion. I'll address the question there.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2021-12-14 19:50:57
 +
+

Oh, and I forgot to post yesterday +OpenLineage 0.4.0 was released 🥳

+ +

This was a big one. +• Split tests for Spark 2 and Spark 3 +• Spark output metrics +• Databricks support with init scripts +• Initial Iceberg support for Spark +• Initial Kafka support for Spark +• dbt build support +• forward compatibility for dbt versions +• lots of bug fixes 🙂 +Check the full changelog for details

+ + + +
+ 🙌 Maciej Obuchowski, Will Johnson, Peter Hicks, Manuel, Peter Hanssens +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Dinakar Sundar + (dinakar_sundar@condenast.com) +
+
2021-12-14 21:42:40
 +
+

Hi @Michael Collado is there any documentation on using great expectations with open lineage

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2021-12-15 11:50:47
 +
+

*Thread Reply:* hmm, actually the only documentation we have right now is on the demo.datakin.com site https://demo.datakin.com/onboarding . The great expectations tab should be enough to get you started

+
+
demo.datakin.com
+ + + +
datakin
+
+ Observability, Lineage and Visualization for Data Pipelines +
+ + + + + + +
Original URL: https://demo.datakin.com/onboarding
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2021-12-15 11:51:04
 +
+

*Thread Reply:* I'll open a ticket to copy that documentation to the OpenLineage site repo

+ + + +
+ 👍 Madhu Maddikera, Dinakar Sundar +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Carlos Meza + (omar.m.8x@gmail.com) +
+
2021-12-15 09:52:51
 +
+

Hello ! I am new on OpenLineage , awesome project !! ; anybody knows about integration with Deequ ? Or a way to capture dataset stats with openlineage ? Thanks ! Appreciate the help !

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2021-12-15 19:01:50
 +
+

*Thread Reply:* Hi! We don't have any integration with deequ yet. We have a structure for recording data quality assertions and statistics, though - see https://github.com/OpenLineage/OpenLineage/blob/main/spec/facets/DataQualityAssertionsDatasetFacet.json and https://github.com/OpenLineage/OpenLineage/blob/main/spec/facets/DataQualityMetricsInputDatasetFacet.json for the specs.

+ +

Check the great expectations integration to see how those facets are being used

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Bruno González + (brugms2@gmail.com) +
+
2022-05-24 06:20:50
 +
+

*Thread Reply:* This is great. Thanks @Michael Collado!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anatoliy Zhyzhkevych + (Anatoliy.Zhyzhkevych@franklintempleton.com) +
+
2021-12-19 22:40:33
 +
+

Hi,

+ +

I am testing Open Lineage/Marquez 0.4.0 with dbt 1.0.0 using dbt-ol build +It seems 12 events were generated but UI shows only history of runs with "Nothing to show here" in detail section about datasets/tests failures in dbt namespace. +The warehouse namespace shows lineage but no details about dataset/test failures .

+ +

Please advice.

+ +

02:57:54 Done. PASS=4 WARN=0 ERROR=3 SKIP=2 TOTAL=9 +02:57:54 Error sending message, disabling tracking +Emitting OpenLineage events: 100%|██████████████████████████████████████████████████████| 12/12 [00:00<00:00, 12.50it/s]

+ +
+
+ + + + + + +
+
+
+ + + + + + +
+
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-12-20 04:15:51
 +
+

*Thread Reply:* This is nothing to show here when you click on test node, right? What about run node?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anatoliy Zhyzhkevych + (Anatoliy.Zhyzhkevych@franklintempleton.com) +
+
2021-12-20 12:28:21
 +
+

*Thread Reply:* There is no details about failure.

+ +

```dbt-ol build -t DEV --profile cdp --profiles-dir /c/Work/dbt/cdp100/profiles --project-dir /c/Work/dbt/cdp100 --select +riskrawmastersharedshareclass +Running OpenLineage dbt wrapper version 0.4.0 +This wrapper will send OpenLineage events at the end of dbt execution. +02:57:21 Running with dbt=1.0.0 +02:57:23 [WARNING]: Configuration paths exist in your dbtproject.yml file which do not apply to any resources. +There are 1 unused configuration paths:

  • models.cdp.risk.raw.liquidity.shared
    • +
+ +

02:57:23 Found 158 models, 181 tests, 0 snapshots, 0 analyses, 574 macros, 0 operations, 2 seed files, 56 sources, 1 exposure, 0 metrics +02:57:23 +02:57:35 Concurrency: 10 threads (target='DEV') +02:57:35 +02:57:35 1 of 9 START test dbtexpectationssourceexpectcompoundcolumnstobeuniquebsesharedpbshareclassEDMPORTFOLIOIDSHARECLASSCODEanyvalueismissingDELETEDFLAGFalse [RUN] +02:57:37 1 of 9 PASS dbtexpectationssourceexpectcompoundcolumnstobeuniquebsesharedpbshareclassEDMPORTFOLIOIDSHARECLASSCODEanyvalueismissingDELETEDFLAGFalse [PASS in 2.67s] +02:57:37 2 of 9 START view model REPL.SHARECLASSDIM.................................... [RUN] +02:57:39 2 of 9 OK created view model REPL.SHARECLASSDIM............................... [SUCCESS 1 in 2.12s] +02:57:39 3 of 9 START test dbtexpectationsexpectcompoundcolumnstobeuniquerawreplpbsharedshareclassRISKPORTFOLIOIDSHARECLASSCODEanyvalueismissingDELETEDFLAGFalse [RUN] +02:57:43 3 of 9 PASS dbtexpectationsexpectcompoundcolumnstobeuniquerawreplpbsharedshareclassRISKPORTFOLIOIDSHARECLASSCODEanyvalueismissingDELETEDFLAGFalse [PASS in 3.42s] +02:57:43 4 of 9 START view model RAWRISKDEV.STG.SHARECLASSDIM........................ [RUN] +02:57:46 4 of 9 OK created view model RAWRISKDEV.STG.SHARECLASSDIM................... [SUCCESS 1 in 3.44s] +02:57:46 5 of 9 START view model RAWRISKDEV.MASTER.SHARECLASSDIM..................... [RUN] +02:57:46 6 of 9 START test relationshipsriskrawstgsharedshareclassRISKINSTRUMENTIDRISKINSTRUMENTIDrefriskrawstgsharedsecurity_ [RUN] +02:57:46 7 of 9 START test relationshipsriskrawstgsharedshareclassRISKPORTFOLIOIDRISKPORTFOLIOIDrefriskrawstgsharedportfolio_ [RUN] +02:57:51 5 of 9 ERROR creating view model RAWRISKDEV.MASTER.SHARECLASSDIM............ [ERROR in 4.31s] +02:57:51 8 of 9 SKIP test relationshipsriskrawmastersharedshareclassRISKINSTRUMENTIDRISKINSTRUMENTIDrefriskrawmastersharedsecurity_ [SKIP] +02:57:51 9 of 9 SKIP test relationshipsriskrawmastersharedshareclassRISKPORTFOLIOIDRISKPORTFOLIOIDrefriskrawmastersharedportfolio_ [SKIP] +02:57:52 7 of 9 FAIL 7282 relationshipsriskrawstgsharedshareclassRISKPORTFOLIOIDRISKPORTFOLIOIDrefriskrawstgsharedportfolio_ [FAIL 7282 in 5.41s] +02:57:54 6 of 9 FAIL 6520 relationshipsriskrawstgsharedshareclassRISKINSTRUMENTIDRISKINSTRUMENTIDrefriskrawstgsharedsecurity_ [FAIL 6520 in 7.23s] +02:57:54 +02:57:54 Finished running 6 tests, 3 view models in 30.71s. +02:57:54 +02:57:54 Completed with 3 errors and 0 warnings: +02:57:54 +02:57:54 Database Error in model riskrawmastersharedshareclass (models/risk/raw/master/shared/riskrawmastersharedshareclass.sql) +02:57:54 002003 (42S02): SQL compilation error: +02:57:54 Object 'RAWRISKDEV.AUDIT.STGSHARECLASSDIMRELATIONSHIPRISKINSTRUMENTID' does not exist or not authorized. +02:57:54 compiled SQL at target/run/cdp/models/risk/raw/master/shared/riskrawmastersharedshareclass.sql +02:57:54 +02:57:54 Failure in test relationshipsriskrawstgsharedshareclassRISKPORTFOLIOIDRISKPORTFOLIOIDrefriskrawstgsharedportfolio (models/risk/raw/stg/shared/riskrawstgsharedschema.yml) +02:57:54 Got 7282 results, configured to fail if != 0 +02:57:54 +02:57:54 compiled SQL at target/compiled/cdp/models/risk/raw/stg/shared/riskrawstgsharedschema.yml/relationshipsriskrawstgsha19e10fb324f7d0cccf2aab512683f693.sql +02:57:54 +02:57:54 Failure in test relationshipsriskrawstgsharedshareclassRISKINSTRUMENTIDRISKINSTRUMENTID_refriskrawstgsharedsecurity_ (models/risk/raw/stg/shared/riskrawstgsharedschema.yml) +02:57:54 Got 6520 results, configured to fail if != 0 +02:57:54 +02:57:54 compiled SQL at target/compiled/cdp/models/risk/raw/stg/shared/riskrawstgsharedschema.yml/relationshipsriskrawstgsha_e3148a1627817f17f7f5a9eb841ef16f.sql +02:57:54 +02:57:54 See test failures:

+ +
+ +

select ** from RAWRISKDEV.AUDIT.STGSHARECLASSDIMrelationship_RISKINSTRUMENT_ID

+ +
+ +

02:57:54 +02:57:54 Done. PASS=4 WARN=0 ERROR=3 SKIP=2 TOTAL=9 +02:57:54 Error sending message, disabling tracking +Emitting OpenLineage events: 100%|██████████████████████████████████████████████████████| 12/12 [00:00<00:00, 12.50it/s]Emitted 14 openlineage events +(dbt) linux@dblnbk152371:/c/Work/dbt/cdp$```

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-12-20 12:30:20
 +
+

*Thread Reply:* I'm talking on clicking on non-test node in Marquez UI - the screenshots shared show you clicked on the one ending in test

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anatoliy Zhyzhkevych + (Anatoliy.Zhyzhkevych@franklintempleton.com) +
+
2021-12-20 16:46:11
 +
+

*Thread Reply:* There are two types of failures: tests failed on stage model (relationships) and physical error in master model (no table with such name). The stage test node in Marquez does not show any indication of failures and dataset node indicates failure but without number of failed records or table name for persistent test storage. The failed master model shows in red but no details of failure. Master model tests were skipped because of model failure but UI reports "Complete".

+ +
+
+ + + + + + +
+
+
+ + + + + + +
+
+
+ + + + + + +
+
+
+ + + + + + +
+
+
+ + + + + + +
+
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-12-20 18:11:50
 +
+

*Thread Reply:* If I understood correctly, for model you would like OpenLineage to capture message error, like this one +22:52:07 Database Error in model customers (models/customers.sql) +22:52:07 Syntax error: Expected "(" or keyword SELECT or keyword WITH but got identifier "PLEASE_REMOVE" at [56:12] +22:52:07 compiled SQL at target/run/jaffle_shop/models/customers.sql +And for dbt test failures, to visualize better that error is happening, for example like that:

+ +
+
+ + + + + + +
+
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-12-20 18:23:12
 +
+

*Thread Reply:* We actually do the first one for Airflow and Spark, I've missed it for dbt 😞

+ +

Created issue to add it to spec in a generic way: +https://github.com/OpenLineage/OpenLineage/issues/446

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/issues/446|#446 [PROPOSAL] Add generic ErrorRunFacet>
+
+ <p>On execution failure, multiple systems provide text-based error messages, like dbt:</p> + +<pre><code>22:52:07 Database Error in model customers (models/customers.sql) +22:52:07 Syntax error: Expected "(" or keyword SELECT or keyword WITH but got identifier "PLEASE_REMOVE" at [56:12] +22:52:07 compiled SQL at target/run/jaffle_shop/models/customers.sql + +</code></pre> + +<p>Airflow:</p> + +<pre><code>airflow.exceptions.AirflowException: Bash command failed. The command returned a non-zero exit code. + +</code></pre> + +<p>Spark:</p> + +<pre><code>org.apache.spark.sql.AnalysisException: Table or view not found: wrong_table_name; line 1 pos 14; + +</code></pre> + +<p>We should add simple <code>ErrorRunFacet</code> to capture human-readable messages. Currently, we have some custom facets for those for <code>BigQuery</code> and <code>Spark</code></p> + +<p>Proposed schema would look like that:</p> + +<pre><code>{ + "$schema" : "&lt;https://json-schema.org/draft/2020-12/schema&gt;", + "$id" : "&lt;https://openlineage.io/spec/facets/1-0-0/ErrorRunFacet.json&gt;", + "$defs" : { + "ErrorRunFacet" : { + "allOf" : [ { + "$ref" : "&lt;https://openlineage.io/spec/1-0-2/OpenLineage.json#/$defs/RunFacet&gt;" + }, { + "type" : "object", + "properties" : { + "message" : { + "description" : "A human-readable string representing error message generated by observed system", + "type" : "string", + "example" : "org.apache.spark.sql.AnalysisException: Table or view not found: wrong_table_name; line 1 pos 14" + }, + "stackTrace" : { + "description" : "A language-specific stack trace generated by observed system", + "type" : "string", + "example" : "Exception in thread \"main\" java.lang.RuntimeException: A test exception\nat io.openlineage.SomeClass.method(SomeClass.java:13)\nat io.openlineage.SomeClass.anotherMethod(SomeClass.java:9)" + } + }, + "required" : [ "message" ] + } ], + "type" : "object" + } + }, + "type" : "object", + "properties" : { + "error" : { + "$ref" : "#/$defs/ErrorRunFacet" + } + } +} +</code></pre> +
+ +
+
Labels
+ proposal +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anatoliy Zhyzhkevych + (Anatoliy.Zhyzhkevych@franklintempleton.com) +
+
2021-12-20 22:49:54
 +
+

*Thread Reply:* Sounds great. Failed/Skipped Tests/Models could be color-coded as well. Thanks.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jorge Reyes (Zenta Group) + (jorge.reyes@zentagroup.com) +
+
2021-12-22 12:37:00
 +
+

hello everyone , i'm learning Openlineage, I am trying to connect with airflow 2, is it possible? or that version is not yet released. this is currently throwing me airflow

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-12-22 12:38:26
 +
+

*Thread Reply:* Hey. If you're using Airflow 2, you should use LineageBackend method described here: https://github.com/OpenLineage/OpenLineage/tree/main/integration/airflow#airflow-21-experimental

+ + + +
+ 🙌 Jorge Reyes (Zenta Group) +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-12-22 12:39:06
 +
+

*Thread Reply:* You don't need to do anything with DAG import then.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jorge Reyes (Zenta Group) + (jorge.reyes@zentagroup.com) +
+
2021-12-22 12:40:30
 +
+

*Thread Reply:* Thanks!!!!! i'll try

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2021-12-27 16:49:20
 +
+

The PR at https://github.com/OpenLineage/OpenLineage/pull/451 should be everything needed to complete the implementation for https://github.com/OpenLineage/OpenLineage/pull/437 . The PR is in draft mode, as I still need ~1 day to update the integration test expectations to match the refactoring (there are some new events, but from my cursory look, the old events still match expected contents). But I think it's in a state that can be reviewed before the tests are updated.

+ +

There are two other PRs that this one is based on - broken up for easier reviewing +• https://github.com/OpenLineage/OpenLineage/pull/447 +• https://github.com/OpenLineage/OpenLineage/pull/448

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2021-12-27 16:49:56
 +
+

*Thread Reply:* @Will Johnson @Maciej Obuchowski FYI 👆

+ + + +
+ 🙌 Will Johnson, Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-01-07 15:25:11
 +
+

The next OpenLineage Technical Steering Committee meeting is Wednesday, January 12! Meetings are on the second Wednesday of each month from 9:00 to 10:00am PT.  +Join us on Zoom: https://us02web.zoom.us/j/81831865546?pwd=RTladlNpc0FTTDlFcWRkM2JyazM4Zz09 +All are welcome. +Agenda: +• OpenLineage 0.4 and 0.5 releases +• Egeria version 3.4 support for OpenLineage +• Airflow TaskListener to simplify OpenLineage integration [Maciej] +• Open Discussion +Notes: https://tinyurl.com/openlineagetsc

+ + + +
+ 🙌 Maciej Obuchowski, Ross Turk, John Thomas, Minkyu Park, Joshua Wankowski, Dalin Kim +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
David Virgil + (david.virgil.naranjo@googlemail.com) +
+
2022-01-11 12:16:09
 +
+

Hello community,

+ +

We are able to post this datasource in marquez. But then the information about the facet with the datasource is not displayed in the UI.

+ +

We want to display the S3 location (URI) where this datasource is pointing to. +{ + id: { + namespace: "<s3://hbi-dns-staging>", + name: "PCHG" + }, + type: "DB_TABLE", + name: "PCHG", + physicalName: "PCHG", + createdAt: "2022-01-11T16:15:54.887Z", + updatedAt: "2022-01-11T16:56:04.093153Z", + namespace: "<s3://hbi-dns-staging>", + sourceName: "<s3://hbi-dns-staging>", + fields: [], + tags: [], + lastModifiedAt: null, + description: null, + currentVersion: "c565864d-1a66-4cff-a5d9-2e43175cbf88", + facets: { + dataSource: { + uri: "<s3://hbi-dns-staging/sql-runner/2022-01-11/PCHG.avro>", + name: "<s3://hbi-dns-staging>", + _producer: "<a href="http://ip-172-25-23-163.dir.prod.aws.hollandandbarrett.comeu-west-1.com/172.25.23.163">ip-172-25-23-163.dir.prod.aws.hollandandbarrett.comeu-west-1.com/172.25.23.163</a>", + _schemaURL: "<https://openlineage.io/spec/facets/1-0-0/DatasourceDatasetFacet.json#/$defs/DatasourceDatasetFacet>" + } + } +}

+ + + +
+
+
+
+ + + + + + + +
+
+ + + + +
+ +
David Virgil + (david.virgil.naranjo@googlemail.com) +
+
2022-01-11 12:24:00
 +
+

As you see there is no much info in openlineage UI

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-01-11 13:02:16
 +
+

The OpenLineage TSC meeting is tomorrow! https://openlineage.slack.com/archives/C01CK9T7HKR/p1641587111000700

+
+ + +
+ + + } + + Michael Robinson + (https://openlineage.slack.com/team/U02LXF3HUN7) +
+ + + +
+
+ <p>The next OpenLineage Technical Steering Committee meeting is Wednesday, January 12! Meetings are on the second Wednesday of each month from 9:00 to 10:00am PT.  +Join us on Zoom: <a href="https://us02web.zoom.us/j/81831865546?pwd=RTladlNpc0FTTDlFcWRkM2JyazM4Zz09">https://us02web.zoom.us/j/81831865546?pwd=RTladlNpc0FTTDlFcWRkM2JyazM4Zz09</a> +All are welcome. +Agenda: +• OpenLineage 0.4 and 0.5 releases +• Egeria version 3.4 support for OpenLineage +• Airflow TaskListener to simplify OpenLineage integration [Maciej] +• Open Discussion +Notes: <a href="https://tinyurl.com/openlineagetsc">https://tinyurl.com/openlineagetsc</a></p> +
+ + + + + + +
Original URL: https://openlineage.slack.com/archives/C01CK9T7HKR/p1641587111000700
+ + +
+ + Slack Conversation +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2022-01-12 11:59:44
 +
+

*Thread Reply:* ^ It’s happening now!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
David Virgil + (david.virgil.naranjo@googlemail.com) +
+
2022-01-14 06:46:44
 +
+

any idea guys about the previous question?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Minkyu Park + (minkyu@datakin.com) +
+
2022-01-18 14:19:39
 +
+

*Thread Reply:* Just to be clear, were you able to get a datasource information from API but just now showing up in the UI? Or you weren’t able to get it from API too?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
SAM + (skhettri@gmail.com) +
+
2022-01-17 03:41:56
 +
+

Hi everyone !! I am doing POC of OpenLineage with Airflow version 2.1, before that would like to know, if this version is supported by OpenLineage?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Conor Beverland + (conorbev@gmail.com) +
+
2022-01-18 11:40:00
 +
+

*Thread Reply:* It does generally work, but, there's a known limitation in that only successful task runs are reported to the lineage backend. This is planned to be fixed in Airflow 2.3.

+ + + +
+ ✅ SAM +
+ +
+ ❤️ Julien Le Dem +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
SAM + (skhettri@gmail.com) +
+
2022-01-18 20:35:52
 +
+

*Thread Reply:* thank you. 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
SAM + (skhettri@gmail.com) +
+
2022-01-17 06:47:54
 +
+

Hello there, I’m using docker Airflow version 2.1.0 , below were the steps I performed but I encountered error, pls help:

+ +
  1. Inside requirements.txt file i added openlineage-airflow . Then ran pip install -r requirements.txt .
  2. Added environmental variable using this command +export AIRFLOW__LINEAGE__BACKEND = openlineage.lineage_backend.OpenLineageBackend
  3. Then configured HTTP Backend environment variables inside “airflow” folder: +export OPENLINEAGE_URL=<http://marquez:5000>
  4. Ran Marquez using ./docker/up.sh & open web frontend UI and saw below error msg:
    5. +
+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Conor Beverland + (conorbev@gmail.com) +
+
2022-01-18 11:30:38
 +
+

*Thread Reply:* hey, I'm aware of one small bug ( which will be fixed in the upcoming OpenLineage 0.5.0 ) which means you would also have to include google-cloud-bigquery in your requirements.txt. This is the bug: https://github.com/OpenLineage/OpenLineage/issues/438

+ + + +
+ ✅ SAM +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Conor Beverland + (conorbev@gmail.com) +
+
2022-01-18 11:31:51
 +
+

*Thread Reply:* The other thing I think you should check is, did you def define the AIRFLOW__LINEAGE__BACKEND variable correctly? What you pasted above looks a little odd with the 2 = signs

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Conor Beverland + (conorbev@gmail.com) +
+
2022-01-18 11:34:25
 +
+

*Thread Reply:* I'm looking a task log inside my own Airflow and I see msgs like: +INFO - Constructing openlineage client to send events to

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Conor Beverland + (conorbev@gmail.com) +
+
2022-01-18 11:34:47
 +
+

*Thread Reply:* ^ i.e. I think checking the task logs you can see if it's at least attempting to send data

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Conor Beverland + (conorbev@gmail.com) +
+
2022-01-18 11:34:52
 +
+

*Thread Reply:* hope this helps!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
SAM + (skhettri@gmail.com) +
+
2022-01-18 20:40:37
 +
+

*Thread Reply:* Thank you, will try again.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2022-01-18 20:10:25
 +
+

Just published OpenLineage 0.5.0 . Big items here are +• dbt-spark support +• New proxy message broker for forwarding OpenLineage messages to Kafka +• New extensibility API for Spark integration +Accompanying tweet thread on the latter two items here: https://twitter.com/PeladoCollado/status/1483607050953232385

+ + + +
+ 🙌 Maciej Obuchowski, Kevin Mellott +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2022-01-19 12:39:30
 +
+

*Thread Reply:* BTW, this was actually the 0.5.1 release. Because, pypi... 🤷‍♂️:skintone4:

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mario Measic + (mario.measic.gavran@gmail.com) +
+
2022-01-27 06:45:08
 +
+

*Thread Reply:* nice on the dbt-spark support 👍

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mohamed El IBRAHIMI + (mohamedelibrahimi700@gmail.com) +
+
2022-01-19 11:12:14
 +
+

HELLO everyone . I’ve been reading and watching talks about OpenLineage and Marquez . this solution is exactly what we been looking to lineage our etls . GREAT WORK . our etls based on postgres redshift and airflow. SO

+ +

I tried to implement the example respecting all the steps required. everything runs successfully (the two dags on airflow ) on host http://localhost:3000/ but nothing appeared on marquez ui . am i missing something ? .

+ +

I’am thinking about create a simple etl pandas to a pandas with some transformation . Like to have a poc to show it to my team . I REALLY NEED SOME HELP

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-01-19 11:13:35
 +
+

*Thread Reply:* Are you using docker on mac with "Use Docker Compose V2" enabled?

+ +

We've just found yesterday that it somehow breaks our example...

+ + + +
+ ✅ Mohamed El IBRAHIMI +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mohamed El IBRAHIMI + (mohamedelibrahimi700@gmail.com) +
+
2022-01-19 11:14:51
 +
+

*Thread Reply:* yes i just installed docker on mac

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mohamed El IBRAHIMI + (mohamedelibrahimi700@gmail.com) +
+
2022-01-19 11:15:02
 +
+

*Thread Reply:* and docker compose version 1.29.2

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-01-19 11:20:24
 +
+

*Thread Reply:* What you can do is to uncheck this, do docker system prune -a and try again.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mohamed El IBRAHIMI + (mohamedelibrahimi700@gmail.com) +
+
2022-01-19 11:21:56
 +
+

*Thread Reply:* done but i get this : Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-01-19 11:22:15
 +
+

*Thread Reply:* Try to restart docker for mac

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-01-19 11:23:00
 +
+

*Thread Reply:* It needs to show Docker Desktop is running :

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mohamed El IBRAHIMI + (mohamedelibrahimi700@gmail.com) +
+
2022-01-19 11:24:01
 +
+

*Thread Reply:* yeah done . I will try to implement the example again and see thank you very much

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mohamed El IBRAHIMI + (mohamedelibrahimi700@gmail.com) +
+
2022-01-19 11:32:55
 +
+

*Thread Reply:* i dont why i getting this when i $ docker-compose up :

+ +

WARNING: The TAG variable is not set. Defaulting to a blank string. +WARNING: The APIPORT variable is not set. Defaulting to a blank string. +WARNING: The APIADMINPORT variable is not set. Defaulting to a blank string. +WARNING: The WEBPORT variable is not set. Defaulting to a blank string. +ERROR: The Compose file ‘./../docker-compose.yml’ is invalid because: +services.api.ports contains an invalid type, it should be a number, or an object +services.api.ports contains an invalid type, it should be a number, or an object +services.web.ports contains an invalid type, it should be a number, or an object +services.api.ports value [‘:’, ‘:’] has non-unique elements

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-01-19 11:46:12
 +
+

*Thread Reply:* are you running it exactly like here, with respect to directories, etc?

+ +

https://github.com/MarquezProject/marquez/tree/main/examples/airflow

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mohamed El IBRAHIMI + (mohamedelibrahimi700@gmail.com) +
+
2022-01-19 11:59:36
 +
+

*Thread Reply:* yeah yeah my bad . every things work fine know . I see the graph in the ui

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mohamed El IBRAHIMI + (mohamedelibrahimi700@gmail.com) +
+
2022-01-19 12:04:01
 +
+

*Thread Reply:* one more question plz . As i said our etls based on postgres redshift and airflow . any advice you have for us to integrate OL to our pipeline ?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mohamed El IBRAHIMI + (mohamedelibrahimi700@gmail.com) +
+
2022-01-19 11:12:17
 +
+

thank you very much

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kevin Mellott + (kevin.r.mellott@gmail.com) +
+
2022-01-19 17:29:51
 +
+

I’m upgrading our OL Java client from an older version (0.2.3) and noticed that the ol.newCustomFacetBuilder() method to create custom facets no longer exists. I can see in this code diff that it might be replaced by simply adding to the additional properties of the standard element you are extending.

+ +

Can you please let me know if I’m understanding this change correctly? In other words, is the code in the diff functionally equivalent or is there a large change I should be understanding better?

+ +

https://github.com/OpenLineage/OpenLineage/compare/0.2.3...0.4.0#diff-f0381d7e68797d9ec60551c96897809072582350e1657d23425747358ec6e471L196

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2022-01-19 17:50:39
 +
+

*Thread Reply:* Hi Kevin - to my understanding that's correct. Do you guys have a custom extractor using this?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kevin Mellott + (kevin.r.mellott@gmail.com) +
+
2022-01-19 20:49:49
 +
+

*Thread Reply:* Thanks John! We have custom code emitting OL events within our ingestion pipeline and it includes a custom facet. I’ll refactor the code to the new format and should be good to go.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kevin Mellott + (kevin.r.mellott@gmail.com) +
+
2022-01-21 00:34:37
 +
+

*Thread Reply:* Just to follow up, this code update worked as expected and we are all good on the upgrade.

+ + + +
+ 👍 Minkyu Park, John Thomas, Julien Le Dem +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
SAM + (skhettri@gmail.com) +
+
2022-01-21 02:13:51
 +
+

I’m not sure what went wrong, with Airflow docker, version 2.1.0 , below were the steps I performed but Marquez UI is showing no jobs, pls help:

+ +
  1. requirements.txt i added openlineage-airflow==0.5.1 . Then ran pip install -r requirements.txt .
  2. Added environmental variable inside my airflow docker folder using this command: +export AIRFLOW__LINEAGE__BACKEND = openlineage.lineage_backend.OpenLineageBackend
  3. Then configured HTTP Backend environment variables inside same airflow docker folder: +export OPENLINEAGE_URL=<http://localhost:5000>
  4. Ran Marquez using ./docker/up.sh  which is in another folder, Front end UI is not showing any job, its empty:
  5. Attached in the airflow DAG log.
    6. +
+ +
+
+ + + + + + +
+
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-01-25 14:46:58
 +
+

*Thread Reply:* Hm, that is odd. Usually there are a few lines in the DAG log from the OpenLineage bits. I’d expect to see something about not having an extractor for the operator you are using.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-01-25 14:47:53
 +
+

*Thread Reply:* If you open a shell in your Airflow Scheduler container and check for the presence of AIRFLOW__LINEAGE__BACKEND is it properly set? Possible the env isn’t making it all the way there.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Lena Kullab + (Lena.Kullab@storable.com) +
+
2022-01-21 13:38:37
 +
+

Hi All,

+ +

I am working on a POC of OpenLineage-Airflow integration and was attempting to get it configured with Amundsen (also working on a POC). Reading through the tutorial here https://openlineage.io/integration/apache-airflow/, under the Prerequisites section it says: +To use the OpenLineage Airflow integration, you'll need a running Airflow instance. You'll also need an OpenLineage compatible HTTP backend. +The example uses Marquez, but I was trying to figure out how to get it to send metadata to the Amundsen graph db backend. Does the Airflow integration only support configuration with an HTTP compatible backend?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2022-01-21 14:03:29
 +
+

*Thread Reply:* Hi Lena! That’s correct, Openlineage is designed to send events to an HTTP backend. There’s a ticket on the future section of the roadmap to support pushing to Amundsen, but it’s not yet been worked on (Ref: Roadmap Issue #86)

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/issues/86|#86 Add backend for sending lineage to Amundsen>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Lena Kullab + (Lena.Kullab@storable.com) +
+
2022-01-21 14:08:35
 +
+

*Thread Reply:* Thank you for the info!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
naman shaundik + (namanshaundik@gmail.com) +
+
2022-01-30 11:01:42
 +
+

hi , i am completely new to openlineage and marquez, i have to integrate openlineage to my existing java project but i am completely confused on where to start, i have gone through documentation and all but i am not able to understand how to integrate openlineage using marquez http backend in my existing project. please someone help me. I may sound naive here but i am in dire need of help.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2022-01-30 12:37:39
 +
+

*Thread Reply:* what do you mean by “Integrate Openlineage”?

+ +

Can you give a little more information on what you’re trying to accomplish and what the existing project is?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
naman shaundik + (namanshaundik@gmail.com) +
+
2022-01-31 03:49:22
 +
+

*Thread Reply:* I work in a datalake team and we are trying to implement data lineage property in our project using openlineage. our project basically keeps track of datasets coming from different sources(hive, redshift, elasticsearch etc.) and jobs.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2022-01-31 15:01:31
 +
+

*Thread Reply:* Gotcha!

+ +

Broadly speaking, all an integration needs to do is to send runEvents to Marquez.

+ +

I'd start by understanding the OpenLineage data model, and then looking at your system to identify when / where runEvents should be sent from, and what information needs to be included.

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.md | OpenLineage.md>
+
+ <p><strong>OpenLineage Spec</strong> +<strong>Specification</strong></p> + +<p>The specification for OpenLineage is formalized as a JsonSchema &lt;OpenLineage.json|OpenLineage.json&gt;. <br /> +An OpenAPI spec is also provided for HTTP based implementations: &lt;OpenLineage.yml|OpenLineage.yml&gt; <br /> +The documentation is published at: <a href="https://openlineage.github.io/">https://openlineage.github.io/</a> <br /> +It allows extensions to the spec using <code>Custom Facets</code> as described in this document.</p> + +<p><strong>Core concepts</strong> +<strong>Core Lineage Model</strong></p> + +<p>&lt;OpenLineageModel.svg|Open Lineage model&gt;</p> + +<p>• <strong>Run Event</strong>: an event describing an observed state of a job run. It is required to at least send a START event and a COMPLETE/FAIL/ABORT event. Additional events are optional. +• <strong>Job</strong>: a process definition that consumes and produces datasets (defined as its inputs and outputs). It is &lt;Naming.md#Jobs|identified by a unique name within a namespace&gt; (which is assigned to the scheduler starting the jobs). The <em>Job</em> evolves over time and this change is captured when the job runs. +• <strong>Dataset</strong>: an abstract representation of data. It has a &lt;Naming.md#Datasets|unique name within the datasource namespace&gt; derived from its physical location (for example db.host.database.schema.table). Typically, a <em>Dataset</em> changes when a job writing to it completes. Similarly to the <em>Job</em> and <em>Run</em> distinction, metadata that is more static from run to run is captured in a DatasetFacet (for example, the schema that does not change every run), what changes every <em>Run</em> is captured as an <em>InputFacet</em> or an <em>OutputFacet</em> (for example, what subset of the data set was read or written, like a time partition). +• <strong>Run</strong>: An instance of a running job with a start and completion (or failure) time. A run is identified by a globally unique ID relative to its job definition. A run ID <strong>must</strong> be an <a href="https://datatracker.ietf.org/doc/html/rfc4122">UUID</a>. +• <strong>Facet</strong>: A piece of metadata attached to one of the entities defined above.</p> + +<p>example: <br /> +Here is an example of a simple start run event not adding any facet information:</p> + +<pre><code>{ + "eventType": "START", + "eventTime": "2020-12-09T23:37:31.081Z", + "run": { + "runId": "3b452093-782c-4ef2-9c0c-aafe2aa6f34d", + }, + "job": { + "namespace": "my-scheduler-namespace", + "name": "myjob.mytask", + }, + "inputs": [ + { + "namespace": "my-datasource-namespace", + "name": "instance.schema.table", + } + ], + "outputs": [ + { + "namespace": "my-datasource-namespace", + "name": "instance.schema.output_table", + } + ], + "producer": "&lt;https://github.com/OpenLineage/OpenLineage/blob/v1-0-0/client&gt;", + "schemaURL": "&lt;https://openlineage.io/spec/1-0-0/OpenLineage.json#/definitions/RunEvent&gt;" +} + +</code></pre> + +<p><strong>Lifecycle</strong></p> + +<p>The OpenLineage API defines events to capture the lifecycle of a <em>Run</em> for a given <em>Job</em>. <br /> +When a <em>job</em> is being <em>run</em>, we capture metadata by sending run events when the state of the job transitions to a different state. <br /> +We might observe different aspects of the job run at different stages. This means that different metadata might be collected in each event during the lifecycle of a run. <br /> +All metadata is additive. for example, if more inputs or outputs are detected as the job is running we might send additional events specifically for those datasets without re-emitting previously observed inputs or outputs. <br /> +Example:</p> + +<p>• When the run starts, we collect the following Metadata: + • Run Id + • Job id + • eventType: START + • event time + • source location and version (ex: git sha) + • If known: Job inputs and outputs. (input schema, ...) +• When the run completes: + • Run Id + • Job id + • eventType: COMPLETE + • event time + • Output datasets schema (and other metadata).</p> + +<p><strong>Facets</strong></p> + +<p>Facets are pieces of metadata that can be attached to the core entities:</p> + +<p>• Run +• Job +• Dataset (Inputs or Outputs)</p> + +<p>A facet is an atomic piece of metadata identified by its name. This means that emitting a new facet with the same name for the same entity replaces the previous facet instance for that entity entirely). It is defined as a JSON object that can be either part of the spec or custom facets defined in a different project.</p> + +<p>Custom facets must use a distinct prefix named after the project defining them to avoid collision with standard facets defined in the &lt;OpenLineage.json|OpenLineage.json&gt; spec. <br /> +They have a _schemaURL field pointing to the corresponding version of the facet schema (as a JSONPointer: <a href="https://swagger.io/docs/specification/using-ref/">$ref URL location</a> ).</p> + +<p>Example: <a href="https://raw.githubusercontent.com/OpenLineage/OpenLineage/main/spec/OpenLineage.json#/definitions/MyCustomJobFacet">https://raw.githubusercontent.com/OpenLineage/OpenLineage/main/spec/OpenLineage.json#/definitions/MyCustomJobFacet</a></p> + +<p>The versioned URL must be an immutable pointer to the version of the facet schema. For example, it should include a tag of a git sha and not a branch name. This should also be a canonical URL. There should be only one URL used for a given version of a schema.</p> + +<p>Custom facets can be promoted to the standard by including them in the spec.</p> + +<p><strong>Custom Facet Naming</strong></p> + +<p>Naming of custom facets should follow pattern <code>{prefix}{name}{entity}Facet</code> PascalCased. <br /> +Prefix must be distinct identifier named after the project defining them to avoid colision with standard facets defined in the &lt;OpenLineage.json|OpenLineage.json&gt; spec. <br /> +Entity is the core entity for which the facet is attached.</p> + +<p>When attached to core entity, the key should follow pattern <code>{prefix}_{name}</code>, where both prefix and name follow snakeCase pattern.</p> + +<p>Example of valid name is <code>BigQueryStatisticsJobFacet</code> and it's key <code>bigQuery_statistics</code>.</p> + +<p><strong>Standard Facets</strong> +<strong>Run Facets</strong></p> + +<p>• <strong>nominalTime</strong>: Captures the time this run is scheduled for. This is a typical usage for time based scheduled job. The job has a nominal schedule time that will be different from the actual time it is running at. +• <strong>parent</strong>: Captures the parent job and Run when the run was spawn from a parent run. For example in the case of Airflow, there's a run for the DAG that then spawns runs for individual tasks that would refer to the parent run as the DAG run. Similarly when a SparkOperator starts a Spark job, this creates a separate run that refers to the task run as its parent.</p> + +<p><strong>Job Facets</strong></p> + +<p>• <strong>sourceCodeLocation</strong>: Captures the source code location and version (example: git sha) of the job. +• <strong>sql</strong>: Capture the SQL query if this job is a SQL query.</p> + +<p><strong>Dataset Facets</strong></p> + +<p>• <strong>schema</strong>: Captures the schema of the dataset +• <strong>dataSource</strong>: Captures the Database instance containing this datasets (ex: Database schema. Object store bucket, ...)</p> + +<p><strong>Input Dataset Facets</strong></p> + +<p>• <strong>dataQualityMetrics</strong>: Captures dataset level and column level data quality metrics when scanning a dataset whith a DataQuality library (row count, byte size, null count, distinct count, average, min, max, quantiles). +• <strong>dataQualityAssertions</strong>: Captures the result of running data tests on dataset or its columns.</p> + +<p><strong>Output Dataset Facets</strong></p> + +<p>• <strong>outputStatistics</strong>: Captures the size of the output written to a dataset (row count and byte size).</p> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
TJ Tang + (tj@tapdata.io) +
+
2022-02-15 15:28:03
 +
+

*Thread Reply:* I suppose OpenLineage itself only defines the standard/protocol to design your data model. To be able to visualize/trace the lineage, you either have to implement your self with the standard data models or including Marquez in your project. You would need to use HTTP API to send lineage events from your Java project to Marquez in this case.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2022-02-16 11:17:13
 +
+

*Thread Reply:* Exactly! This project also includes connectors for more common data tools (Airflow, dbt, spark, etc), but at it's core OpenLineage is a standard and protocol

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-02-02 19:55:13
 +
+

The next OpenLineage Technical Steering Committee meeting is Wednesday, February 9. Meetings are on the second Wednesday of each month from 9:00 to 10:00am PT. +Join us on Zoom: https://us02web.zoom.us/j/81831865546?pwd=RTladlNpc0FTTDlFcWRkM2JyazM4Zz09 +All are welcome. Agenda items are always welcome, as well. Reply in thread with yours. +Current agenda: +• OpenLineage 0.5.1 release +• Apache Flink effort +• Dagster integration +• Open Discussion +Notes: https://tinyurl.com/openlineagetsc

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jensen Yap + (jensen@contxts.io) +
+
2022-02-03 00:33:45
 +
+

Hi everybody!

+ + + +
+ 👋 Maciej Obuchowski, John Thomas +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2022-02-03 12:39:57
 +
+

*Thread Reply:* Hello!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Albert Bikeev + (albert.bikeev@gmail.com) +
+
2022-02-04 09:36:46
 +
+

Hi everybody! +Very cool initiative, thank you! Is there any traction on Apache Atlas integration? Is there some way to help you there?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2022-02-04 15:07:07
 +
+

*Thread Reply:* Hey Albert! There aren't yet any issues or proposals around Apache Atlas yet, but that's definitely something you can help with!

+ +

I'm not super familiar with Atlas, were you thinking in terms of enabling Atlas to receive runEvents from OpenLineage connectors?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Albert Bikeev + (albert.bikeev@gmail.com) +
+
2022-02-07 05:49:16
 +
+

*Thread Reply:* Hi John! +Yes, exactly, it’d be nice to see Atlas as a receiver side of the OpenLineage events. Is there some guidelines on how to implement it? I guess we need OpenLineage-compatible server implementation so we could receive events and send them to Atlas, right?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2022-02-07 11:30:14
 +
+

*Thread Reply:* exactly - This would be a change on the Atlas side. I’d start by opening an issue in the atlas repo about making an API endpoint that can receive OpenLineage events. +Marquez is our reference implementation of OpenLineage, so I’d look around in that repo to see how it’s been implemented :)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Albert Bikeev + (albert.bikeev@gmail.com) +
+
2022-02-07 11:50:27
 +
+

*Thread Reply:* Got it, thanks! Did that: https://issues.apache.org/jira/browse/ATLAS-4550 +If it’d not get any traction we at New Work might contribute as well

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2022-02-07 11:56:09
 +
+

*Thread Reply:* awesome! if you guys have any questions, reach out and I can get you in touch with some of the engineers on our end

+ + + +
+ 👍 Albert Bikeev +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-02-08 11:20:47
 +
+

*Thread Reply:* @Albert Bikeev one minor thing that could be helpful: java OpenLineage library contains server model classes: https://github.com/OpenLineage/OpenLineage/pull/300#issuecomment-923489097

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/pull/300#issuecomment-923489097|Comment on #300 server model>
+
+ <p>the new classes look like this:</p> + +<p>``` +package io.openlineage.server;</p> + +<p>import com.fasterxml.jackson.annotation.JsonAnyGetter; +import com.fasterxml.jackson.annotation.JsonAnySetter; +import com.fasterxml.jackson.annotation.JsonCreator; +import com.fasterxml.jackson.annotation.JsonProperty; +import com.fasterxml.jackson.annotation.JsonPropertyOrder; +import java.lang.Object; +import java.lang.String; +import java.net.URI; +import java.time.ZonedDateTime; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; +import java.util.UUID;</p> + +<p>public final class OpenLineage { + @JsonPropertyOrder + public static final class RunFacets { + @JsonAnySetter + private final Map&lt;String, RunFacet&gt; additionalProperties;</p> + +<pre><code>@JsonCreator +public RunFacets() { + this.additionalProperties = new LinkedHashMap&amp;lt;&amp;gt;(); +} + +/**** + ** @return additional properties + **/ +@JsonAnyGetter +public Map&amp;lt;String, RunFacet&amp;gt; getAdditionalProperties() { + return additionalProperties; +} +</code></pre> + +<p>}</p> + +<p>@JsonPropertyOrder({ + "eventType", + "eventTime", + "run", + "job", + "inputs", + "outputs", + "producer", + "schemaURL" + }) + public static final class RunEvent { + private final String eventType;</p> + +<pre><code>private final ZonedDateTime eventTime; + +private final Run run; + +private final Job job; + +private final List&amp;lt;InputDataset&amp;gt; inputs; + +private final List&amp;lt;OutputDataset&amp;gt; outputs; + +private final URI producer; + +private final URI schemaURL; + +@JsonAnySetter +private final Map&amp;lt;String, Object&amp;gt; additionalProperties; + +/**** + ** @param eventType the current transition of the run state. It is required to issue 1 START event and 1 of [ COMPLETE, ABORT, FAIL ] event per run. Additional events with OTHER eventType can be added to the same run. For example to send additional metadata after the run is complete + ** @param eventTime the time the event occurred at + ** @param run the run + ** @param job the job + ** @param inputs The set of ****input**** datasets. + ** @param outputs The set of ****output**** datasets. + ** @param producer URI identifying the producer of this metadata. For example this could be a git url with a given tag or sha + ** @param schemaURL The JSON Pointer (&lt;https://tools.ietf.org/html/rfc6901&gt;) URL to the corresponding version of the schema definition for this RunEvent + **/ +@JsonCreator +public RunEvent(@JsonProperty("eventType") String eventType, + @JsonProperty("eventTime") ZonedDateTime eventTime, @JsonProperty("run") Run run, + @JsonProperty("job") Job job, @JsonProperty("inputs") List&amp;lt;InputDataset&amp;gt; inputs, + @JsonProperty("outputs") List&amp;lt;OutputDataset&amp;gt; outputs, + @JsonProperty("producer") URI producer, @JsonProperty("schemaURL") URI schemaURL) { + this.eventType = eventType; + this.eventTime = eventTime; + this.run = run; + this.job = job; + this.inputs = inputs; + this.outputs = outputs; + this.producer = producer; + this.schemaURL = schemaURL; + this.additionalProperties = new LinkedHashMap&amp;lt;&amp;gt;(); +} + +/**** + ** @return the current transition of the run state. It is required to issue 1 START event and 1 of [ COMPLETE, ABORT, FAIL ] event per run. Additional events with OTHER eventType can be added to the same run. For example to send additional metadata after the run is complete + **/ +public String getEventType() { + return eventType; +} + +/**** + ** @return the time the event occurred at + **/ +public ZonedDateTime getEventTime() { + return eventTime; +} + +public Run getRun() { + return run; +} + +public Job getJob() { + return job; +} + +/**** + ** @return The set of ****input**** datasets. + **/ +public List&amp;lt;InputDataset&amp;gt; getInputs() { + return inputs; +} + +/**** + ** @return The set of ****output**** datasets. + **/ +public List&amp;lt;OutputDataset&amp;gt; getOutputs() { + return outputs; +} + +/**** + ** @return URI identifying the producer of this metadata. For example this could be a git url with a given tag or sha + **/ +public URI getProducer() { + return producer; +} + +/**** + ** @return The JSON Pointer (&lt;https://tools.ietf.org/html/rfc6901&gt;) URL to the corresponding version of the schema definition for this RunEvent + **/ +public URI getSchemaURL() { + return schemaURL; +} + +/**** + ** @return additional properties + **/ +@JsonAnyGetter +public Map&amp;lt;String, Object&amp;gt; getAdditionalProperties() { + return additionalProperties; +} +</code></pre> + +<p>}</p> + +<p>@JsonPropertyOrder + public static final class JobFacets { + @JsonAnySetter + private final Map&lt;String, JobFacet&gt; additionalProperties;</p> + +<pre><code>@JsonCreator +public JobFacets() { + this.additionalProperties = new LinkedHashMap&amp;lt;&amp;gt;(); +} + +/**** + ** @return additional properties + **/ +@JsonAnyGetter +public Map&amp;lt;String, JobFacet&amp;gt; getAdditionalProperties() { + return additionalProperties; +} +</code></pre> + +<p>}</p> + +<p>@JsonPropertyOrder + public static final class InputDatasetInputFacets { + @JsonAnySetter + private final Map&lt;String, InputDatasetFacet&gt; additionalProperties;</p> + +<pre><code>@JsonCreator +public InputDatasetInputFacets() { + this.additionalProperties = new LinkedHashMap&amp;lt;&amp;gt;(); +} + +/**** + ** @return additional properties + **/ +@JsonAnyGetter +public Map&amp;lt;String, InputDatasetFacet&amp;gt; getAdditionalProperties() { + return additionalProperties; +} +</code></pre> + +<p>}</p> + +<p>@JsonPropertyOrder({ + "<em>producer", + "</em>schemaURL" + }) + public static final class DatasetFacet implements BaseFacet { + private final URI _producer;</p> + +<pre><code>private final URI _schemaURL; + +@JsonAnySetter +private final Map&amp;lt;String, Object&amp;gt; additionalProperties; + +/**** + ** @param _producer URI identifying the producer of this metadata. For example this could be a git url with a given tag or sha + ** @param _schemaURL The JSON Pointer (&lt;https://tools.ietf.org/html/rfc6901&gt;) URL to the corresponding version of the schema definition for this facet + **/ +@JsonCreator +public DatasetFacet(@JsonProperty("_producer") URI _producer, + @JsonProperty("_schemaURL") URI _schemaURL) { + this._producer = _producer; + this._schemaURL = _schemaURL; + this.additionalProperties = new LinkedHashMap&amp;lt;&amp;gt;(); +} + +/**** + ** @return URI identifying the producer of this metadata. For example this could be a git url with a given tag or sha + **/ +public URI get_producer() { + return _producer; +} + +/**** + ** @return The JSON Pointer (&lt;https://tools.ietf.org/html/rfc6901&gt;) URL to the corresponding version of the schema definition for this facet + **/ +public URI get_schemaURL() { + return _schemaURL; +} + +/**** + ** @return additional properties + **/ +@JsonAnyGetter +public Map&amp;lt;String, Object&amp;gt; getAdditionalProperties() { + return additionalProperties; +} +</code></pre> + +<p>}</p> + +<p>@JsonPropertyOrder({ + "<em>producer", + "</em>schemaURL" + }) + public static final class OutputDatasetFacet implements BaseFacet { + private final URI _producer;</p> + +<pre><code>private final URI _schemaURL; + +@JsonAnySetter +private final Map&amp;lt;String, Object&amp;gt; additionalProperties; + +/**** + ** @param _producer URI identifying the producer of this metadata. For example this could be a git url with a given tag or sha + ** @param _schemaURL The JSON Pointer (&lt;https://tools.ietf.org/html/rfc6901&gt;) URL to the corresponding version of the schema definition for this facet + **/ +@JsonCreator +public OutputDatasetFacet(@JsonProperty("_producer") URI _producer, + @JsonProperty("_schemaURL") URI _schemaURL) { + this._producer = _producer; + this._schemaURL = _schemaURL; + this.additionalProperties = new LinkedHashMap&amp;lt;&amp;gt;(); +} + +/**** + ** @return URI identifying the producer of this … +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Albert Bikeev + (albert.bikeev@gmail.com) +
+
2022-02-08 11:32:12
 +
+

*Thread Reply:* Got it, thank you!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Juan Carlos Fernández Rodríguez + (jcfernandez@keedio.com) +
+
2022-05-04 11:12:23
 +
+

*Thread Reply:* This is a quite old discussion, but isn't possible to use openlineage proxy to send json to kafka topic and let Atlas read that json without any modification? +It would be needed to create a new model for spark, other than https://github.com/apache/atlas/blob/release-2.1.0-rc3/addons/models/1000-Hadoop/1100-spark_model.json and upload it to atlas (what could be done with a call to the atlas Api) +Does it makes sense?

+
+ + + + +
<https://github.com/apache/atlas/blob/release-2.1.0-rc3/addons/models/1000-Hadoop/1100-spark_model.json | 1100-spark_model.json>
+
+ <p>``` +{ + "enumDefs": [], + "structDefs": [], + "classificationDefs": [], + "entityDefs": [ + { + "name": "spark<em>db", + "superTypes": [ + "DataSet" + ], + "serviceType": "spark", + "typeVersion": "1.0", + "attributeDefs": [ + { + "name": "location", + "typeName": "string", + "isOptional": true, + "cardinality": "SINGLE", + "isUnique": false, + "isIndexable": false, + "searchWeight": 5 + }, + { + "name": "clusterName", + "typeName": "string", + "isOptional": true, + "cardinality": "SINGLE", + "isUnique": false, + "isIndexable": false, + "searchWeight": 8 + }, + { + "name": "parameters", + "typeName": "map&lt;string,string&gt;", + "isOptional": true, + "cardinality": "SINGLE", + "isUnique": false, + "isIndexable": false + }, + { + "name": "ownerType", + "typeName": "string", + "isOptional": true, + "cardinality": "SINGLE", + "isUnique": false, + "isIndexable": false + } + ] + }, + { + "name": "spark</em>table", + "superTypes": [ + "DataSet" + ], + "serviceType": "spark", + "typeVersion": "1.0", + "attributeDefs": [ + { + "name": "tableType", + "typeName": "string", + "isOptional": true, + "cardinality": "SINGLE", + "isUnique": false, + "isIndexable": false + }, + { + "name": "provider", + "typeName": "string", + "isOptional": true, + "cardinality": "SINGLE", + "isUnique": false, + "isIndexable": false, + "searchWeight": 5 + }, + { + "name": "partitionColumnNames", + "typeName": "array&lt;string&gt;", + "isOptional": true, + "cardinality": "SINGLE", + "isUnique": false, + "isIndexable": false + }, + { + "name": "bucketSpec", + "typeName": "map&lt;string,string&gt;", + "isOptional": true, + "cardinality": "SINGLE", + "isUnique": false, + "isIndexable": false + }, + { + "name": "ownerType", + "typeName": "string", + "isOptional": true, + "cardinality": "SINGLE", + "isUnique": false, + "isIndexable": false + }, + { + "name": "createTime", + "typeName": "date", + "isOptional": true, + "cardinality": "SINGLE", + "isUnique": false, + "isIndexable": false + }, + { + "name": "parameters", + "typeName": "map&lt;string,string&gt;", + "isOptional": true, + "cardinality": "SINGLE", + "isUnique": false, + "isIndexable": false + }, + { + "name": "comment", + "typeName": "string", + "isOptional": true, + "cardinality": "SINGLE", + "isUnique": false, + "isIndexable": false, + "searchWeight": 9 + }, + { + "name": "unsupportedFeatures", + "typeName": "array&lt;string&gt;", + "isOptional": true, + "cardinality": "SINGLE", + "isUnique": false, + "isIndexable": false + }, + { + "name": "viewOriginalText", + "typeName": "string", + "isOptional": true, + "cardinality": "SINGLE", + "isUnique": false, + "isIndexable": false, + "searchWeight": 9 + }, + { + "name": "schemaDesc", + "typeName": "string", + "isOptional": true, + "cardinality": "SINGLE", + "isUnique": false, + "isIndexable": false, + "searchWeight": 5 + }, + { + "name": "partitionProvider", + "typeName": "string", + "isOptional": true, + "cardinality": "SINGLE", + "isUnique": false, + "isIndexable": false + } + ] + }, + { + "name": "spark<em>column", + "superTypes": [ + "DataSet" + ], + "serviceType": "spark", + "typeVersion": "1.0", + "attributeDefs": [ + { + "name": "type", + "typeName": "string", + "isOptional": false, + "cardinality": "SINGLE", + "isUnique": false, + "isIndexable": true + }, + { + "name": "nullable", + "typeName": "boolean", + "isOptional": true, + "cardinality": "SINGLE", + "isUnique": false, + "isIndexable": false + }, + { + "name": "metadata", + "typeName": "string", + "isOptional": true, + "cardinality": "SINGLE", + "isUnique": false, + "isIndexable": false + }, + { + "name": "comment", + "typeName": "string", + "isOptional": true, + "cardinality": "SINGLE", + "isUnique": false, + "isIndexable": false, + "searchWeight": 9 + } + ] + }, + { + "name": "spark</em>ml<em>directory", + "superTypes": [ + "DataSet" + ], + "serviceType": "spark", + "typeVersion": "1.0", + "attributeDefs": [ + { + "name": "uri", + "typeName": "string", + "isOptional": false, + "cardinality": "SINGLE", + "isUnique": false, + "isIndexable": true, + "searchWeight": 10 + }, + { + "name": "directory", + "typeName": "string", + "isOptional": false, + "cardinality": "SINGLE", + "isUnique": false, + "isIndexable": true, + "searchWeight": 10 + } + ] + }, + { + "name": "spark</em>storagedesc", + "superTypes": [ + "Referenceable" + ], + "serviceType": "spark", + "typeVersion": "1.0", + "attributeDefs": [ + { + "name": "location", + "typeName": "string", + "isOptional": true, + "cardinality": "SINGLE", + "isUnique": false, + "isIndexable": false, + "searchWeight": 10 + }, + { + "name": "inputFormat", + "typeName": "string", + "isOptional": true, + "cardinality": "SINGLE", + "isUnique": false, + "isIndexable": false + }, + { + "name": "outputFormat", + "typeName": "string", + "isOptional": true, + "cardinality": "SINGLE", + "isUnique": false, + "isIndexable": false + }, + { + "name": "serde", + "typeName": "string", + "isOptional": true, + "cardinality": "SINGLE", + "isUnique": false, + "isIndexable": false + }, + { + "name": "compressed", + "typeName": "boolean", + "isOptional": false, + "cardinality": "SINGLE", + "isUnique": false, + "isIndexable": true + }, + { + "name": "parameters", + "typeName": "map&lt;string,string&gt;", + "isOptional": true, + "cardinality": "SINGLE", + "isUnique": false, + "isIndexable": false + } + ] + }, + { + "name": "spark<em>ml</em>model", + "superTypes": [ + "DataSet" + ], + "serviceType": "spark", + "typeVersion": "1.0", + "attributeDefs": [ + { + "name": "extra", + "typeName": "string", + "isOptional": true, + "cardinality": "SINGLE", + "isUnique": false, + "isIndexable": false + } + ] + }, + { + "name": "spark_application", + "superTypes": [ + "Process" + ], + "serviceType": "spark", + "typeVersion": "1.0", + "attributeDefs": [ + { + "name": "currentUser", + "typeName": "string", + "isOptional": true, + "cardinality": "SINGLE", + "isUniq…</p> +
+ + + + + + + +
+ + <a href="https://github.com/apache/atlas">apache/atlas</a> +
+ + +
+ + + +
+ 👍 Albert Bikeev +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-05-04 11:24:02
 +
+

*Thread Reply:* @Juan Carlos Fernández Rodríguez - You still need to build a bridge between the OpenLineage Spec and the Apache Atlas entity JSON. So far, no one has contributed something like that to the open source community... yet!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Juan Carlos Fernández Rodríguez + (jcfernandez@keedio.com) +
+
2022-05-04 14:24:28
 +
+

*Thread Reply:* sorry for the ignorance, +But what is the purpose of the bridge?the comunicación with atlas should be done throw kafka, and that messages can be sent by the proxy. What are I missing?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2022-05-04 16:37:33
 +
+

*Thread Reply:* "bridge" in this case refers to a service of some sort that converts from OpenLineage run event to Atlas entity JSON, since there's currently nothing that will do that

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
xiang chen + (cdmikechen@hotmail.com) +
+
2022-05-19 09:08:23
 +
+

*Thread Reply:* If OpenLineage send an event to kafka, I think we can use kafka stream or kafka connect to rebuild message to atlas event.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
xiang chen + (cdmikechen@hotmail.com) +
+
2022-05-19 09:11:37
 +
+

*Thread Reply:* @John Thomas Our company used to use atlas as a metadata service. I just came into know this project. After I learned how openlineage works, I think I can create an issue to describe my design first.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
xiang chen + (cdmikechen@hotmail.com) +
+
2022-05-19 09:13:36
 +
+

*Thread Reply:* @Juan Carlos Fernández Rodríguez If you already have some experience and design, can you directly create an issue so that we can discuss it in more detail ?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Juan Carlos Fernández Rodríguez + (jcfernandez@keedio.com) +
+
2022-05-19 12:42:31
 +
+

*Thread Reply:* Hi @xiang chen we are discussing internally in my company if rewrite to atlas or another alternative. If we do this, we will share and could involve you in some way.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-02-04 15:02:29
 +
+

Who here is working with OpenLineage at Dagster or Flink? We would love to hear about your work at the next on February 9 at 9 a.m. PT. Please reply here or message me to coordinate. @Ziyoiddin Yusupov

+ + + +
+ 👍 Ziyoiddin Yusupov +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Luca Soato + (lucasoato@gmail.com) +
+
2022-02-04 19:18:24
 +
+

Hi everyone, +OpenLineage is wonderful, we really needed something like this! +Has anyone else used it with Databricks, Delta tables or Spark? If someone is interested into these technologies we can work together to get a POC and share some thoughts. +Thanks and have a nice weekend! :)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julius Rentergent + (julius.rentergent@thetradedesk.com) +
+
2022-02-25 13:06:16
 +
+

*Thread Reply:* Hi Luca, I agree this looks really promising. I’m working on getting it to run on Databricks, but I’m only just starting out 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-02-08 12:00:02
 +
+

Friendly reminder: this month’s OpenLineage TSC meeting is tomorrow! https://openlineage.slack.com/archives/C01CK9T7HKR/p1643849713216459

+
+ + +
+ + + } + + Michael Robinson + (https://openlineage.slack.com/team/U02LXF3HUN7) +
+ + + +
+
+ <p>The next OpenLineage Technical Steering Committee meeting is Wednesday, February 9. Meetings are on the second Wednesday of each month from 9:00 to 10:00am PT. +Join us on Zoom: <a href="https://us02web.zoom.us/j/81831865546?pwd=RTladlNpc0FTTDlFcWRkM2JyazM4Zz09">https://us02web.zoom.us/j/81831865546?pwd=RTladlNpc0FTTDlFcWRkM2JyazM4Zz09</a> +All are welcome. Agenda items are always welcome, as well. Reply in thread with yours. +Current agenda: +• OpenLineage 0.5.1 release +• Apache Flink effort +• Dagster integration +• Open Discussion +Notes: <a href="https://tinyurl.com/openlineagetsc">https://tinyurl.com/openlineagetsc</a></p> +
+ + + + + + +
Original URL: https://openlineage.slack.com/archives/C01CK9T7HKR/p1643849713216459
+ + +
+ + Slack Conversation +
+ + +
+ + + +
+ ❤️ Kevin Mellott, John Thomas +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Albert Bikeev + (albert.bikeev@gmail.com) +
+
2022-02-10 08:22:28
 +
+

Hi people, +One question regarding error reporting - what is the mechanism for that? E.g. if I send duplicated job to Openlineage, is there a way to notify me about that?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-02-10 09:05:39
 +
+

*Thread Reply:* By duplicated, you mean with the same runId?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Albert Bikeev + (albert.bikeev@gmail.com) +
+
2022-02-10 11:40:55
 +
+

*Thread Reply:* It’s only one example, could be also duplicated job name or anything else. The question is if there is mechanism to report that

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-02-14 17:21:20
 +
+

Reducing the Logging of Spark Integration

+ +

Hey, OpenLineage community! I'm curious if there are any quick tricks / fixes to reduce the amount of logging happening in the OpenLineage Spark Integration. Each job seems to print out the Logical Plan with INFO level logging. The default behavior of Databricks is to print out INFO level logs and so it gets pretty cluttered and noisy.

+ +

I'm hoping there's a feature flag that would help me shut off those kind of logs in OpenLineage's Spark integration 🤞

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-02-15 05:15:12
 +
+

*Thread Reply:* I think this log should be dropped to debug: https://github.com/OpenLineage/OpenLineage/blob/d66c41872f3cc7f7cd5c99664d401e070e[…]c/main/common/java/io/openlineage/spark/agent/EventEmitter.java

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/d66c41872f3cc7f7cd5c99664d401e070e60ff48/integration/spark/src/main/common/java/io/openlineage/spark/agent/EventEmitter.java | EventEmitter.java>
+
+ <pre><code> &lt;a href="http://log.info"&gt;log.info&lt;/a&gt;("Lineage completed successfully: {} {}", resp, mapper.writeValueAsString(event)); +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-02-15 23:27:07
 +
+

*Thread Reply:* @Maciej Obuchowski that is a good one! It would be nice to still have SOME logging in info to know that the event complete successfully but that response and event is very verbose.

+ +

I was also thinking about here: +https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/src/main/common/java/io/openlineage/spark/agent/lifecycle/OpenLineageRunEventBuilder.java#L337-L340

+ +

and here: +https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/src/main/common/java/io/openlineage/spark/agent/lifecycle/OpenLineageRunEventBuilder.java#L405-L408

+ +

These spots are where it's printing out the full logical plan for some reason.

+ +

Can I just open up a PR and switch these to log.debug instead?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-02-16 04:59:17
 +
+

*Thread Reply:* Yes, that would be good solution for now. Later would be nice to have some option to raise the log level - OL logs are absolutely drowning in logs from rest of Spark cluster when set to debug.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-02-16 13:35:15
 +
+

[SPARK][INTEGRATION] Need Brainstorming Ideas - How to Persist / Access Spark Configs in JobEnd

+ +

Hey, OL community! I'm working on PR#490 and I finally have all tests passing but now my desired behavior - display environment properties during COMPLETE / JobEnd events - is not happening 😭

+ +

The previous approach stored the spark properties in the OpenLineageContext with a properties attribute but that was part of all of the test failures I believe.

+ +

What are some other ways to store the jobStart's properties and make them accessible to the corresponding jobEnd? Hopefully it's okay to tag @Maciej Obuchowski, @Michael Collado, and @Paweł Leszczyński who have been extremely helpful in the past and brought great ideas to the table.

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/pull/490|#490 Added generic facet to collect environmental properties (EnvironmentFacet)>
+
+ <p><strong>Problem</strong></p> + +<p>👋 Introduce the ability to produce environmental properties using EnvironmentFacet, this facet can be used to work with any environment, based on the conditional check to verify environment EnvironmentFacet can be added.</p> + +<p>Closes: <a href="https://github.com/OpenLineage/OpenLineage/issues/464">#464</a></p> + +<p><strong>Solution</strong></p> + +<p>Generic implementation of EnvironmentFacet gives the ability to add environmental details. Since EnvironmentFacet produces HashMap it can handle properties dynamically.</p> + +<p>In the current implementation added ability to collect properties for Databricks environment including mount point details. <br /> +Below are some examples:</p> + +<p>• "orgId" +• "spark.databricks.clusterUsageTags.clusterOwnerOrgId" +• "spark.databricks.notebook.path" +• "spark.databricks.job.type" +• "spark.databricks.job.id" +• "spark.databricks.job.runId" +• "user" +• "userId" +• "spark.databricks.clusterUsageTags.clusterName" +• "spark.databricks.clusterUsageTags.azureSubscriptionId"</p> + +<p><strong>Checklist</strong></p> + +<p>☑︎ You've <a href="https://github.com/OpenLineage/OpenLineage/blob/main/why-the-dco.md">signed-off</a> your work +☑︎ Your pull request title follows our <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CONTRIBUTING.md#creating-pull-requests">guidelines</a> +☐ Your changes are accompanied by tests (<em>if relevant</em>) +☑︎ Your change contains a <a href="https://kurtisnusbaum.medium.com/stacked-diffs-keeping-phabricator-diffs-small-d9964f4dcfa6">small diff</a> and is self-contained +☐ You've updated any relevant documentation (<em>if relevant</em>) +☐ You've updated the <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md"><code>CHANGELOG.md</code></a> with details about your change under the "Unreleased" section (<em>if relevant, depending on the change, this may not be necessary</em>) +☐ You've versioned the core OpenLineage model or facets according to <a href="https://docs.snowplowanalytics.com/docs/pipeline-components-and-applications/iglu/common-architecture/schemaver">SchemaVer</a> (<em>if relevant</em>)</p> +
+ +
+
Comments
+ 6 +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2022-02-16 13:44:30
 +
+

*Thread Reply:* Hey, I responded on the issue, but just to make it clear for everyone, the OL events for a run are not expected to be an accumulation of all past events. Events should be treated as additive by the backend - each event can post what information it has about the run and the backend is responsible for constructing a holistic picture of the run

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/pull/490#issuecomment-1042011803|Comment on #490 Added generic facet to collect environmental properties (EnvironmentFacet)>
+
+ <p>&gt; how can I persist this information over to the COMPLETE events coming from OpenLineage?</p> + +<p>The different events are intended to be additive - it is not expected that each event is a cumulative representation of the previous events. If you send the environment facet on the <code>START</code> event, it will still be present for the run even if it's missing from the <code>COMPLETE</code>.</p> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2022-02-16 13:47:18
 +
+

*Thread Reply:* e.g., here is the marquez code that fetches the facets for a run. Note that all of the facets are included from all events with the requested run_uuid. If the env facet is present on any event, it will be returned by the API

+
+ + + + +
<https://github.com/MarquezProject/marquez/blob/main/api/src/main/java/marquez/db/RunDao.java | RunDao.java>
+
+ <pre><code> + "(\n" + + " SELECT le.run_uuid, JSON_AGG(event-&amp;gt;'run'-&amp;gt;'facets') AS facets\n" + + " FROM lineage_events le\n" + + " GROUP BY le.run_uuid\n" + + ") AS f ON r.uuid=f.run_uuid\n" +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/MarquezProject/marquez">MarquezProject/marquez</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-02-16 13:51:30
 +
+

*Thread Reply:* Ah! Thanks for that @Michael Collado it's good to understand the OpenLineage perspective.

+ +

So, we do need to maintain some state. That makes total sense, Mike.

+ +

How does Marquez handle failed jobs currently? Based on this issue (https://github.com/OpenLineage/OpenLineage/issues/436) I think Marquez would show a START but no COMPLETE event, right?

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/issues/436|#436 [INTEGRATION][Spark] use executionFailure on SparkListenerSQLExecutionEnd to indicate failed execution >
+
+ <p>In Spark 3, <code>SparkListenerSQLExecutionEnd</code> has private property that allows us to find out if job failed. We should use it if we're using Spark 3.</p> + +<p><a href="https://github.com/OpenLineage/OpenLineage/pull/393#discussion_r768049758">#393 (comment)</a></p> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2022-02-16 14:00:03
 +
+

*Thread Reply:* If I were building the backend, I would store events, then calculate the end state later, rather than trying to "maintain some state" (maybe we mean the same thing, but using different words here 😀). +Re: the failure events, I think job failures will currently result in one FAIL event and one COMPLETE event. The SparkListenerJobEnd event will trigger a FAIL event but the SparkListenerSQLExecutionEnd event will trigger the COMPLETE event.

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/src/main/common/java/io/openlineage/spark/agent/lifecycle/OpenLineageRunEventBuilder.java | OpenLineageRunEventBuilder.java>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/src/main/common/java/io/openlineage/spark/agent/lifecycle/OpenLineageRunEventBuilder.java | OpenLineageRunEventBuilder.java>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-02-16 15:16:27
 +
+

*Thread Reply:* Oooh! I did not know we already could get a FAIL event! That is super helpful to know, Mike! Thank you so much!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-02-21 10:04:18
 +
+

[SPARK] Connecting SparkListenerSQLExecutionStart to the various SparkListenerJobStarts

+ +

TL;DR: How can I connect the SparkListenerSQLExecutionStart to the SparkListenerJobStart events coming out of OpenLineage? The events appear to have two separate run ids and no link to indicate that the ExecutionStart event owns the subsequent JobStart events.

+ +

More Context:

+ +

Recently, I implemented a connector for Azure Synapse (data warehouse on the Microsoft cloud) for the Spark integration and now with https://github.com/OpenLineage/OpenLineage/pull/490, I realize now that the SparkListenerSQLExecutionStart events carries with it the necessary inputs and outputs to tell the "real" lineage. The way the Synapse in Databricks works is:

+ +

• SparkListenerSQLExecutionStart fires off an event with the end to end input and output (e.g. S3 as input and SQL table as output) +• SparkListenerJobStart events fire off that move content from one S3 location to a "staging" location controlled by Azure Synapse. OpenLineage records this event with INPUT S3 and output is a WASB "tempfolder" (which is a temporary locatio and not really useful for lineage since it will be destroyed at the end of the job) +• The final operation actually happens ALL in Synapse and OpenLineage does not fire off an event it seems. The Synapse database has a "COPY" command which moves the data from "tempfolder" in to the database. +• Finally a SparkListenerSQLExecutionEnd event happens and the query is complete. +Ideally, I could connect the SQLExecutionStart of SQLExecutionEnd with the SparkListenerJobStart so that I can get the JobStart properties. I see that ExecutionStart has an execution id and JobStart should have the same Execution Id BUT I think by the time I reach the ExecutionEND, all the JobStart events would have been removed from the HashMap that contains all of the events in OpenLineage.

+ +

Any guidance on how to reach a JobStart properties from an ExecutionStart or ExecutionEnd would be greatly appreciated!

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/pull/490|#490 Added generic facet to collect environmental properties (EnvironmentFacet)>
+
+ <p><strong>Problem</strong></p> + +<p>👋 Introduce the ability to produce environmental properties using EnvironmentFacet, this facet can be used to work with any environment, based on the conditional check to verify environment EnvironmentFacet can be added.</p> + +<p>Closes: <a href="https://github.com/OpenLineage/OpenLineage/issues/464">#464</a></p> + +<p><strong>Solution</strong></p> + +<p>Generic implementation of EnvironmentFacet gives the ability to add environmental details. Since EnvironmentFacet produces HashMap it can handle properties dynamically.</p> + +<p>In the current implementation added ability to collect properties for Databricks environment including mount point details. <br /> +Below are some examples:</p> + +<p>• "orgId" +• "spark.databricks.clusterUsageTags.clusterOwnerOrgId" +• "spark.databricks.notebook.path" +• "spark.databricks.job.type" +• "spark.databricks.job.id" +• "spark.databricks.job.runId" +• "user" +• "userId" +• "spark.databricks.clusterUsageTags.clusterName" +• "spark.databricks.clusterUsageTags.azureSubscriptionId"</p> + +<p><strong>Checklist</strong></p> + +<p>☑︎ You've <a href="https://github.com/OpenLineage/OpenLineage/blob/main/why-the-dco.md">signed-off</a> your work +☑︎ Your pull request title follows our <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CONTRIBUTING.md#creating-pull-requests">guidelines</a> +☐ Your changes are accompanied by tests (<em>if relevant</em>) +☑︎ Your change contains a <a href="https://kurtisnusbaum.medium.com/stacked-diffs-keeping-phabricator-diffs-small-d9964f4dcfa6">small diff</a> and is self-contained +☐ You've updated any relevant documentation (<em>if relevant</em>) +☐ You've updated the <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md"><code>CHANGELOG.md</code></a> with details about your change under the "Unreleased" section (<em>if relevant, depending on the change, this may not be necessary</em>) +☐ You've versioned the core OpenLineage model or facets according to <a href="https://docs.snowplowanalytics.com/docs/pipeline-components-and-applications/iglu/common-architecture/schemaver">SchemaVer</a> (<em>if relevant</em>)</p> +
+ +
+
Comments
+ 7 +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+ 🤔 Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-02-22 09:02:48
 +
+

*Thread Reply:* I think this scenario only happens when spark job spawns another "sub-job", right?

+ +

I think that maybe you can check sparkContext.getLocalProperty("spark.sql.execution.id")

+ +

> I see that ExecutionStart has an execution id and JobStart should have the same Execution Id BUT I think by the time I reach the ExecutionEND, all the JobStart events would have been removed from the HashMap that contains all of the events in OpenLineage. +But pairwise, those starts and ends should at least have the same runId as they were created with same OpenLineageContext, right?

+ +

Anyway, what @Michael Collado wrote on the issue is true: https://github.com/OpenLineage/OpenLineage/pull/490#issuecomment-1042011803 - you should not assume that we hold all the metadata somewhere in memory during whole execution of the run. The backend should be able to take care of it.

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/pull/490#issuecomment-1042011803|Comment on #490 Added generic facet to collect environmental properties (EnvironmentFacet)>
+
+ <p>&gt; how can I persist this information over to the COMPLETE events coming from OpenLineage?</p> + +<p>The different events are intended to be additive - it is not expected that each event is a cumulative representation of the previous events. If you send the environment facet on the <code>START</code> event, it will still be present for the run even if it's missing from the <code>COMPLETE</code>.</p> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-02-22 10:53:09
 +
+

*Thread Reply:* @Maciej Obuchowski - I was hoping they'd have the same run id as well but they do not 😞

+ +

But that is the expectation? A SparkSQLExecutionStart and JobStart SHOULD have the same execution ID, right?

+ +

I will take a look at sparkContext.getLocalProperty. Thank you so much for the reply Maciej!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-02-22 10:57:24
 +
+

*Thread Reply:* SparkSQLExecutionStart and SparkSQLExecutionEnd should have the same runId, as well as JobStart and JobEnd events. Beyond those it can get wild. For example, some jobs don't emit JobStart/JobEnd events. Some jobs, like Delta emit multiple, that aren't easily tied to SQL event.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-02-23 03:48:38
 +
+

*Thread Reply:* Okay, I dug into the Databricks Synapse Connector and it does the following:

+ +
  1. SparkSQLExecutionStart with execution id of 8 happens (so gets runid of abc123). It contains the real inputs and outputs that we want.
  2. The Synapse connector starts executing JDBC commands. These commands prepare the synapse database to connect with data that Spark will land in a staging area in the cloud. (I don't know how it' executing arbitrary commands before the official job start begins 😞 )
  3. SparkJobStart beings with execution id of 9 happens (so it gets runid of jkl456). This contains the inputs and an output to a temp folder (NOT the real output we want but a staging location) +a. There are four JobIds 0 - 3, all of which point back to execution id 9 with the same physical plan. +b. After job1, it runs more JDBC commands. +c. I think at Job2, it runs the actual Spark code to query and join my raw input data and land it in a cloud storage account "tempfolder"/ +d. After job3, it runs the final JDBC commands to actually move the data from "tempfolder/" to Synapse Db.
  4. Finally, the SparkSQLListenerEnd event occurs. +I can see this in the Spark UI as well.
    5. +
+ +

Because the Databricks Synapse connector somehow adds these additional JobStarts WITHOUT referencing the original SparkSQLExeuctionStart execution ID, we have to rely on heuristics to connect the /tempfolder to the real downstream table that was already provided in the ExecutionStart event 😞

+ +

I've attached the logs and a screenshot of what I'm seeing the Spark UI. If you had a chance to take a look, it's a bit verbose but I'd appreciate a second pair of eyes on my analysis. Hopefully I got something wrong 😅

+ +
+
log4j-SYNAPSE2-SUBSET.txt
+ + + + + + +
+
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-02-23 07:19:01
 +
+

*Thread Reply:* I think we've encountered the same stuff in Delta before 🙂

+ +

https://github.com/OpenLineage/OpenLineage/issues/388#issuecomment-964401860

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2022-02-23 14:13:18
 +
+

*Thread Reply:* @Will Johnson , am I reading your report correctly that the SparkListenerJobStart event is reported with a spark.sql.execution.id that differs from the execution id of the SparkSQLExecutionStart?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2022-02-23 14:18:04
 +
+

*Thread Reply:* WILLJ: We're deep inside this thing and have an executionid |9| +😂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-02-23 21:56:48
 +
+

*Thread Reply:* Hah @Michael Collado I see you found my method of debugging in Databricks 😅

+ +

But you're exactly right, there's a SparkSQLExecutionStart event with execution id 8 and then a set of JobStart events all with execution id 9!

+ +

I don't know enough about Spark internals on how you can just run arbitrary Scala code while making it look like a Spark Job but that's what it looks like. As if the SqlDwWriter somehow submits a new job without a ExecutionStart... maybe it's an RDD operation instead? This has given me another idea to add some more log.info statements to my jar 😅😬

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-02-28 14:00:23
 +
+

One of our own will be talking OpenLineage, Airflow and Spark at the Subsurface Conference this week. Register to attend @Michael Collado’s session on March 3rd at 11:45. You can register and learn more here: https://www.dremio.com/subsurface/live/winter2022/

+
+
Dremio
+ + + +
Subsurface Live Winter 2022, The Cloud Data Lake Conference
+
+ Subsurface LIVE is tailored for data architects and data engineers to discuss open source projects driving innovation in cloud data lakes +
+ + + + + + + + +
Original URL: https://www.dremio.com/subsurface/live/winter2022/
+ + + +
+ +
+
+ + + + + + +
+ + +
+ 🎉 Willy Lulciuc, Maciej Obuchowski +
+ +
+ 🙌 Will Johnson, Ziyoiddin Yusupov, Julien Le Dem +
+ +
+ 👍 Ziyoiddin Yusupov +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2022-02-28 14:00:56
 +
+

*Thread Reply:* You won’t want to miss this talk!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Martin Fiser + (fisa@keboola.com) +
+
2022-02-28 15:06:43
 +
+

I have a question about DataHub integration through OpenLineage standard. Is anyone working on it, or was it rather just an icon used in previous materials? We have build a openlineage API endpoint in our product and we were hoping OL will gain enough traction so it will be a native way to connect to variaty of data discovery/observability tools, such as datahub, amundzen, etc.

+ +

Many thanks!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2022-02-28 15:29:58
 +
+

*Thread Reply:* hi Martin - when you talk about a DataHub integration, did you mean a method to collect information from DataHub? I don't see a current issue open for that, but I recommend you make one and to kick off the discussion around it.

+ +

If you mean sending information to DataHub, that should already be possible if users pass a datahub api endpoint to the OPENLINEAGE_ENDPOINT variable

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Martin Fiser + (fisa@keboola.com) +
+
2022-02-28 16:29:54
 +
+

*Thread Reply:* Hi, thanks for a reply! I meant to emit Openlineage JSON structure to datahub.

+ +

Could you be please more specific, possibly link an article how to find the endpoint on the datahub side? Many thanks!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2022-02-28 17:15:31
 +
+

*Thread Reply:* ooooh, sorry I misread - I thought you meant that datahub had built an endpoint. Your integration should emit openlineage events to an endpoint, but datahub would have to build that support into their product likely? I'm not sure how to go about it

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2022-02-28 17:16:27
 +
+

*Thread Reply:* I'd reach out to datahub, potentially?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Martin Fiser + (fisa@keboola.com) +
+
2022-02-28 17:21:51
 +
+

*Thread Reply:* i see. ok, will do!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2022-03-02 18:15:21
 +
+

*Thread Reply:* It has been discussed in the past but I don’t think there is something yet. The Kafka transport PR that is in flight should facilitate this

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Martin Fiser + (fisa@keboola.com) +
+
2022-03-02 18:33:45
 +
+

*Thread Reply:* Thanks for the response! though dragging Kafka in just for data delivery bit is too much. I think the clearest way would be to push Datahub to make an API endpoint and parser for OL /lineage data structure.

+ +

I see this is more political think that would require join effort of DataHub team and OpenLineage with a common goal.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-02-28 17:22:47
 +
+

Is there a topic you think the community should discuss at the next OpenLineage TSC meeting? Reply or DM with your item, and we’ll add it to the agenda. Mark your calendars: the next TSC meeting is Wednesday, March 9 at 9 am PT on zoom.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-03-02 10:24:58
 +
+

The next OpenLineage Technical Steering Committee meeting is Wednesday, March 9! Meetings are on the second Wednesday of each month from 9:00 to 10:00am PT. +Join us on Zoom: https://us02web.zoom.us/j/81831865546?pwd=RTladlNpc0FTTDlFcWRkM2JyazM4Zz09 +All are welcome. +Agenda: +• New committers +• Release overview (0.6.0) +• New process for blog posts +• Retrospective: Spark integration +Notes: https://tinyurl.com/openlineagetsc

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2022-03-02 14:29:33
 +
+

FYI, there's a talk on OpenLineage at Subsurface live tomorrow - https://www.dremio.com/subsurface/live/winter2022/session/cross-platform-data-lineage-with-openlineage/

+
+
Dremio
+ + + +
Cross-Platform Data Lineage with OpenLineage | Subsurface Live Winter 2022
+
+ Data today is distributed and heterogeneous. Data lineage helps by tracing the relationship between datasets and placing them in context. +
+ +
+
Est. reading time
+ 1 minute +
+ + + + + + + + +
Original URL: https://www.dremio.com/subsurface/live/winter2022/session/cross-platform-data-lineage-with-openlineage/
+ + + +
+ + + +
+ 🙌 Maciej Obuchowski, John Thomas, Paweł Leszczyński, Francis McGregor-Macdonald +
+ +
+ 👍 Ziyoiddin Yusupov, Michael Robinson, Jac. +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-03-04 15:25:20
 +
+

@channel The latest release (0.6.0) of OpenLineage is now available, featuring a new Dagster integration, updates to the Airflow and Java integrations, a generic facet for env properties, bug fixes, and more. For more info, visit https://github.com/OpenLineage/OpenLineage/releases/tag/0.6.0

+ + + +
+ 🙌 Conor Beverland, Dalin Kim, Ziyoiddin Yusupov, Luca Soato +
+ +
+ 👍 Julien Le Dem +
+ +
+ 👀 William Angel, Francis McGregor-Macdonald +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-07 14:06:19
 +
+

Hello Guys,

+ +

Where do I find an example of building a custom extractor? We have several custom airflow operators that I need to integrate

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2022-03-07 14:56:58
 +
+

*Thread Reply:* Hi marco - we don't have documentation on that yet, but the Postgres extractor is a pretty good example of how they're implemented.

+ +

all the included extractors are here: https://github.com/OpenLineage/OpenLineage/tree/main/integration/airflow/openlineage/airflow/extractors

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-07 15:07:41
 +
+

*Thread Reply:* Thanks. I can follow that to build my own. Also I am installing this environment right now in Airflow 2. It seems I need Marquez and openlinegae-aiflow library. It seems that by this example I can put my extractors in any path as long as it is referenced in the environment variable. Is that correct? +OPENLINEAGE_EXTRACTOR_&lt;operator&gt;=full.path.to.ExtractorClass +Also do I need anything else other than Marquez and openlineage_airflow?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-03-07 15:30:45
 +
+

*Thread Reply:* Yes, as long as the extractors are in the python path.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-03-07 15:31:59
 +
+

*Thread Reply:* I built one a little while ago for a custom operator, I'd be happy to share what I did. I put it in the same file as the operator class for convenience.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-07 15:32:51
 +
+

*Thread Reply:* That will be great help. Thanks

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-03-08 20:38:27
 +
+

*Thread Reply:* This is the one I wrote:

+ +
+
http_to_bigquery.py
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-03-08 20:39:30
 +
+

*Thread Reply:* to make it work, I set this environment variable:

+ +

OPENLINEAGE_EXTRACTOR_HttpToBigQueryOperator=http_to_bigquery.HttpToBigQueryExtractor

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-03-08 20:40:57
 +
+

*Thread Reply:* the extractor starts at line 183, and the really important bits start at line 218

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-03-07 15:16:37
 +
+

@channel At the next OpenLineage TSC meeting, we’ll be reminiscing about the Spark integration. If you’ve had a hand in OL support for Spark, please join and share! The meeting will start at 9 am PT on Wednesday this week. @Maciej Obuchowski @Oleksandr Dvornik @Willy Lulciuc @Michael Collado https://wiki.lfaidata.foundation/display/OpenLineage/Monthly+TSC+meeting

+ + + +
+ 👍 Ross Turk, Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-07 18:44:26
 +
+

Would Marquez create some lineage for operators that don't have a custom extractor built yet?

+ + + +
+ ✅ Fuming Shih +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-03-08 12:05:25
 +
+

*Thread Reply:* You would see that job was run - but we couldn't extract dataset lineage from it.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-03-08 12:05:49
 +
+

*Thread Reply:* The good news is that we're working to solve this problem in general.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-08 12:15:52
 +
+

*Thread Reply:* I see, so i definitively will need the custom extractor built. I just need to understand where to set the path to the extractor. I can build one by following the postgress extractor you have built.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-03-08 12:50:00
 +
+

*Thread Reply:* That depends how you deploy Airflow. Our tests use environment in docker-compose: https://github.com/OpenLineage/OpenLineage/blob/main/integration/airflow/tests/integration/tests/docker-compose-2.yml#L34

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/integration/airflow/tests/integration/tests/docker-compose-2.yml | docker-compose-2.yml>
+
+ <pre><code> OPENLINEAGE_EXTRACTOR_CustomOperator: custom_extractor.CustomExtractor +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-08 13:19:37
 +
+

*Thread Reply:* Thanks for the example. I can show this to my infra support person for his reference.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-03-08 11:47:11
 +
+

This month’s OpenLineage TSC community meeting is tomorrow at 9am PT! It’s not too late to add an item to the agenda. Reply here or msg me with yours. https://openlineage.slack.com/archives/C01CK9T7HKR/p1646234698326859

+
+ + +
+ + + } + + Michael Robinson + (https://openlineage.slack.com/team/U02LXF3HUN7) +
+ + + +
+
+ <p>The next OpenLineage Technical Steering Committee meeting is Wednesday, March 9! Meetings are on the second Wednesday of each month from 9:00 to 10:00am PT. +Join us on Zoom: <a href="https://us02web.zoom.us/j/81831865546?pwd=RTladlNpc0FTTDlFcWRkM2JyazM4Zz09">https://us02web.zoom.us/j/81831865546?pwd=RTladlNpc0FTTDlFcWRkM2JyazM4Zz09</a> +All are welcome. +Agenda: +• New committers +• Release overview (0.6.0) +• New process for blog posts +• Retrospective: Spark integration +Notes: <a href="https://tinyurl.com/openlineagetsc">https://tinyurl.com/openlineagetsc</a></p> +
+ + + + + + +
Original URL: https://openlineage.slack.com/archives/C01CK9T7HKR/p1646234698326859
+ + +
+ + Slack Conversation +
+ + +
+ + + +
+ 👍 Ross Turk +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-09 19:31:23
 +
+

I am running the last command to install marquez in AWS +helm upgrade --install marquez . + --set marquez.db.host &lt;AWS-RDS-HOST&gt; + --set marquez.db.user &lt;AWS-RDS-USERNAME&gt; + --set marquez.db.password &lt;AWS-RDS-PASSWORD&gt; + --namespace marquez + --atomic + --wait +And I am receiving this error +Error: query: failed to query with labels: secrets is forbidden: User "xxx@xxx.xx" cannot list resource "secrets" in API group "" in the namespace "default"

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2022-03-10 12:46:18
 +
+

*Thread Reply:* Do you need to specify a namespace that is not « default »?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-09 19:31:48
 +
+

Can anyone let me know what is happening? My DI guy said it is a chart issue

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-03-10 07:40:13
 +
+

*Thread Reply:* @Kevin Mellott aren't you the chart wizard? Maybe you could help 🙂

+ + + +
+ 👀 Kevin Mellott +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-10 14:09:26
 +
+

*Thread Reply:* Ok so I had to update a chart dependency

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-10 14:10:39
 +
+

*Thread Reply:* Now I installed the service in amazon using this +helm install marquez . --dependency-update --set marquez.db.host=myhost --set marquez.db.user=myuser --set marquez.db.password=mypassword --namespace marquez --atomic --wait

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-10 14:11:31
 +
+

*Thread Reply:* i can see marquez-web running and marquez as well as the database i set up manually

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-10 14:12:27
 +
+

*Thread Reply:* however I can not fetch initial data when login into the endpoint

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kevin Mellott + (kevin.r.mellott@gmail.com) +
+
2022-03-10 14:52:06
 +
+

*Thread Reply:* 👋 @Marco Diaz happy to hear that the Helm install is completing without error! To help troubleshoot the error above, can you please let me know if this endpoint is available and working?

+ +

http://localhost:5000/api/v1/namespaces

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-10 15:13:16
 +
+

*Thread Reply:* i got this +{"namespaces":[{"name":"default","createdAt":"2022_03_10T18:05:55.780593Z","updatedAt":"2022-03-10T19:03:31.309713Z","ownerName":"anonymous","description":"The default global namespace for dataset, job, and run metadata not belonging to a user-specified namespace."}]}

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-10 15:13:34
 +
+

*Thread Reply:* i have to use the namespace marquez to redirect there +kubectl port-forward svc/marquez 5000:80 -n marquez

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-10 15:13:48
 +
+

*Thread Reply:* is there something i need to change in a config file?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-10 15:14:39
 +
+

*Thread Reply:* also how would i change the "localhost" address to something that is accessible in amazon without the need to redirect?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-10 15:14:59
 +
+

*Thread Reply:* Sorry for all the questions. I am not an infra guy and have had to do all this by myself

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kevin Mellott + (kevin.r.mellott@gmail.com) +
+
2022-03-10 15:39:23
 +
+

*Thread Reply:* No problem at all, I think there are a couple of things at play here. With the local setup, it appears that the web is attempting to access the API on the wrong port number (3000 instead of 5000). I’ll create an issue for that one so that we can fix it.

+ +

As to the EKS installation (or any non-local install), this is where you would need to use what’s called an ingress controller to expose the services outside of the Kubernetes cluster. There are different flavors of these (NGINX is popular), and I believe that AWS EKS has some built-in capabilities that might help as well.

+ +

https://www.eksworkshop.com/beginner/130_exposing-service/ingress/

+
+
Amazon EKS Workshop
+ + + +
EKSworkshop.com
+
+ Amazon EKS Workshop +
+ + + + + + + + +
Original URL: https://www.eksworkshop.com/beginner/130_exposing-service/ingress/
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-10 15:40:50
 +
+

*Thread Reply:* So how do i fix this issue?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kevin Mellott + (kevin.r.mellott@gmail.com) +
+
2022-03-10 15:46:56
 +
+

*Thread Reply:* If your goal is to deploy to AWS, then you would need to get the EKS ingress configured. It’s not a trivial task, but they do have a bit of a walkthrough at https://www.eksworkshop.com/beginner/130_exposing-service/.

+ +

However, if you are just seeking to explore Marquez and try things out, then I would highly recommend the “Open in Gitpod” functionality at https://github.com/MarquezProject/marquez#try-it. That will perform a full deployment for you in a temporary environment very quickly.

+
+
Amazon EKS Workshop
+ + + +
EKSworkshop.com
+
+ Amazon EKS Workshop +
+ + + + + + + + +
Original URL: https://www.eksworkshop.com/beginner/130_exposing-service/
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-10 16:02:05
 +
+

*Thread Reply:* i need to use it in aws for a POC

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-10 19:15:08
 +
+

*Thread Reply:* Is there a better guide on how to install and setup Marquez in AWS? +This guide is omitting many steps +https://marquezproject.github.io/marquez/running-on-aws.html

+
+
Marquez
+ + + +
Running Marquez on AWS
+
+ Collect, aggregate, and visualize a data ecosystem’s metadata +
+ + + + + + +
Original URL: https://marquezproject.github.io/marquez/running-on-aws.html
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-03-10 12:35:37
 +
+

We're trying to find best way to track upstream releases of projects we have integrations for, to support newer versions faster and with less bugs. If you have any opinions on this topic, please chime in here

+ +

https://github.com/OpenLineage/OpenLineage/issues/602

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-11 13:34:30
 +
+

@Kevin Mellott Hello Kevin I followed the tutorial you sent me and I have exposed my services. However I am still seeing the same errors (this comes from the api/namescape call) +{"namespaces":[{"name":"default","createdAt":"2022_03_10T18:05:55.780593Z","updatedAt":"2022-03-10T19:03:31.309713Z","ownerName":"anonymous","description":"The default global namespace for dataset, job, and run metadata not belonging to a user-specified namespace."}]}

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-11 13:35:08
 +
+

Is there something i need to change in the chart? I do not have access to the default namespace in kubernetes only marquez namescpace

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kevin Mellott + (kevin.r.mellott@gmail.com) +
+
2022-03-11 13:56:27
 +
+

@Marco Diaz that is actually a good response! This is the JSON returned back by the API to show some of the default Marquez data created by the install. Is there another error you are experiencing?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-11 13:59:28
 +
+

*Thread Reply:* I still see this +https://files.slack.com/files-pri/T01CWUYP5AR-F036JKN77EW/image.png

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-11 14:00:09
 +
+

*Thread Reply:* I created my own database and changed the values for host, user and password inside the chart.yml

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kevin Mellott + (kevin.r.mellott@gmail.com) +
+
2022-03-11 14:00:23
 +
+

*Thread Reply:* Does it show that within the AWS deployment? It looks to show localhost in your screenshot.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kevin Mellott + (kevin.r.mellott@gmail.com) +
+
2022-03-11 14:00:52
 +
+

*Thread Reply:* Or are you working through the local deploy right now?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-11 14:01:57
 +
+

*Thread Reply:* It shows the same using the exposed service

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-11 14:02:09
 +
+

*Thread Reply:* i just didnt do another screenshot

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-11 14:02:27
 +
+

*Thread Reply:* Could it be communication with the DB?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kevin Mellott + (kevin.r.mellott@gmail.com) +
+
2022-03-11 14:04:37
 +
+

*Thread Reply:* What do you see if you view the network traffic within your web browser (right click -> Inspect -> Network). Specifically, wondering what the response code from the Marquez API URL looks like.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-11 14:14:48
 +
+

*Thread Reply:* i see this error +Error occured while trying to proxy to: <a href="http://xxxxxxxxxxxxxxxxxxxxxxxxx.us-east-1.elb.amazonaws.com/api/v1/namespaces">xxxxxxxxxxxxxxxxxxxxxxxxx.us-east-1.elb.amazonaws.com/api/v1/namespaces</a>

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-11 14:16:00
 +
+

*Thread Reply:* it seems to be trying to use the same address to access the api endpoint

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-11 14:16:26
 +
+

*Thread Reply:* however the api service is in a different endpoint

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-11 14:18:24
 +
+

*Thread Reply:* The API resides here +<a href="http://Xxxxxxxxxxxxxxxxxxxxxx-2064419849.us-east-1.elb.amazonaws.com">Xxxxxxxxxxxxxxxxxxxxxx-2064419849.us-east-1.elb.amazonaws.com</a>

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-11 14:19:13
 +
+

*Thread Reply:* The web service resides here +<a href="http://xxxxxxxxxxxxxxxxxxxxxxxxxxx-335729662.us-east-1.elb.amazonaws.com">xxxxxxxxxxxxxxxxxxxxxxxxxxx-335729662.us-east-1.elb.amazonaws.com</a>

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-11 14:19:25
 +
+

*Thread Reply:* do they both need to be under the same LB?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-11 14:19:56
 +
+

*Thread Reply:* How would i do that is they install as separate services?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kevin Mellott + (kevin.r.mellott@gmail.com) +
+
2022-03-11 14:27:15
 +
+

*Thread Reply:* You are correct, both the website and API are expecting to be exposed on the same ALB. This will give you a single URL that can reach your Kubernetes cluster, and then the ALB will allow you to configure Ingress rules to route the traffic based on the request.

+ +

Here is an example from one of the AWS repos - in the ingress resource you can see the single rule setup to point traffic to a given service.

+ +

https://github.com/kubernetes-sigs/aws-load-balancer-controller/blob/main/docs/examples/2048/2048_full.yaml

+
+ + + + +
<https://github.com/kubernetes-sigs/aws-load-balancer-controller/blob/main/docs/examples/2048/2048_full.yaml | 2048_full.yaml>
+
+ <pre><code>--- +apiVersion: v1 +kind: Namespace +metadata: + name: game-2048 +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + namespace: game-2048 + name: deployment-2048 +spec: + selector: + matchLabels: + &lt;a href="http://app.kubernetes.io/name"&gt;app.kubernetes.io/name&lt;/a&gt;: app-2048 + replicas: 5 + template: + metadata: + labels: + &lt;a href="http://app.kubernetes.io/name"&gt;app.kubernetes.io/name&lt;/a&gt;: app-2048 + spec: + containers: + - image: public.ecr.aws/l6m2t8p7/docker-2048:latest + imagePullPolicy: Always + name: app-2048 + ports: + - containerPort: 80 +--- +apiVersion: v1 +kind: Service +metadata: + namespace: game-2048 + name: service-2048 +spec: + ports: + - port: 80 + targetPort: 80 + protocol: TCP + type: NodePort + selector: + &lt;a href="http://app.kubernetes.io/name"&gt;app.kubernetes.io/name&lt;/a&gt;: app-2048 +--- +apiVersion: &lt;a href="http://networking.k8s.io/v1"&gt;networking.k8s.io/v1&lt;/a&gt; +kind: Ingress +metadata: + namespace: game-2048 + name: ingress-2048 + annotations: + &lt;a href="http://alb.ingress.kubernetes.io/scheme"&gt;alb.ingress.kubernetes.io/scheme&lt;/a&gt;: internet-facing + &lt;a href="http://alb.ingress.kubernetes.io/target-type"&gt;alb.ingress.kubernetes.io/target-type&lt;/a&gt;: ip +spec: + ingressClassName: alb + rules: + - http: + paths: + - path: / + pathType: Prefix + backend: + service: + name: service-2048 + port: + number: 80 + +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/kubernetes-sigs/aws-load-balancer-controller">kubernetes-sigs/aws-load-balancer-controller</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-11 14:36:40
 +
+

*Thread Reply:* Thanks for the help. Now I know what the issue is

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kevin Mellott + (kevin.r.mellott@gmail.com) +
+
2022-03-11 14:51:34
 +
+

*Thread Reply:* Great to hear!!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sandeep Bhat + (bhatsandeep424@gmail.com) +
+
2022-03-16 00:55:36
 +
+

👋 Hi everyone! Our company is looking to adopt data lineage tool, so i have few queries on open lineage, so 1. Is this completey free.

+ +
  1. What are tha database it supports?
    2. +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-03-16 10:29:06
 +
+

*Thread Reply:* Hi! Yes, OpenLineage is free. It is an open source standard for collection, and it provides the agents that integrate with pipeline tools to capture lineage metadata. You also need a metadata server, and there is an open source one called Marquez that you can use.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-03-16 10:29:15
 +
+

*Thread Reply:* It supports the databases listed here: https://openlineage.io/integration

+
+
openlineage.io
+ + + +
Integrations
+
+ Data lineage is the foundation for a new generation of powerful, context-aware data tools and best practices. OpenLineage enables consistent collection of lineage metadata, creating a deeper understanding of how data is produced and used. +
+ + + + + + +
Original URL: https://openlineage.io/integration
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sandeep Bhat + (bhatsandeep424@gmail.com) +
+
2022-03-16 08:27:20
 +
+

and when i run the ./docker/up.sh --seed i got the result from java code(sample example) But how to get the same thing in python example?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-03-16 10:29:53
 +
+

*Thread Reply:* Not sure I understand - are you looking for example code in Python that shows how to make OpenLineage calls?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sandeep Bhat + (bhatsandeep424@gmail.com) +
+
2022-03-16 12:45:14
 +
+

*Thread Reply:* yup

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sandeep Bhat + (bhatsandeep424@gmail.com) +
+
2022-03-16 13:10:04
 +
+

*Thread Reply:* how to run

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-03-16 23:08:31
 +
+

*Thread Reply:* this is a good post for getting started with Marquez: https://openlineage.io/blog/explore-lineage-api/

+
+
openlineage.io
+ + + +
Exploring Lineage History via the Marquez API
+
+ Taking advantage of recent changes to the Marquez API, this post shows how to diagnose job failures and explore the impact of code changes on downstream dependents. +
+ + + + + + +
Original URL: https://openlineage.io/blog/explore-lineage-api/
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-03-16 23:08:51
 +
+

*Thread Reply:* once you have run ./docker/up.sh, you should be able to run through that and see how the system runs

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-03-16 23:09:45
 +
+

*Thread Reply:* There is a python client you can find here: https://github.com/OpenLineage/OpenLineage/tree/main/client/python

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sandeep Bhat + (bhatsandeep424@gmail.com) +
+
2022-03-17 00:05:58
 +
+

*Thread Reply:* Thank you

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-03-19 00:00:32
 +
+

*Thread Reply:* You are welcome 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mirko Raca + (racamirko@gmail.com) +
+
2022-04-19 09:28:50
 +
+

*Thread Reply:* Hey @Ross Turk, (and potentially @Maciej Obuchowski) - what are the plans for OL Python client? I'd like to use it, but without a pip package it's not really project-friendly.

+ +

Is there any work in that direction, is the current client code considered mature and just needs re-packaging, or is it just a thought sketch and some serious work is needed?

+ +

I'm trying to avoid re-inventing the wheel, so if there's already something in motion, I'd rather support than start (badly) from scratch?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-04-19 09:32:17
 +
+

*Thread Reply:* What do you mean without pip-package?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-04-19 09:32:18
 +
+

*Thread Reply:* https://pypi.org/project/openlineage-python/

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-04-19 09:35:08
 +
+

*Thread Reply:* It's still developed, for example next release will have pluggable backends - like Kafka +https://github.com/OpenLineage/OpenLineage/pull/530

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mirko Raca + (racamirko@gmail.com) +
+
2022-04-19 09:40:11
 +
+

*Thread Reply:* My apologies Maciej! +In my defense - looking for "open lineage" on pypi doesn't show this in the first 20 results. Still, should have checked setup.py. My bad, and thank you for the pointer!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-04-19 10:00:49
 +
+

*Thread Reply:* We might need to add some keywords to setup.py - right now we have only "openlineage" there 😉

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mirko Raca + (racamirko@gmail.com) +
+
2022-04-20 08:12:29
 +
+

*Thread Reply:* My mistake was that I was expecting a separate repo for the clients. But now I'm playing around with the package and trying to figure out the OL concepts. Thank you for your contribution, it's much nicer to experiment from ipynb than curl 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-03-16 12:00:01
 +
+

@Julien Le Dem and @Willy Lulciuc will be at Data Council Austin next week talking OpenLineage and Airflow https://www.datacouncil.ai/talks/data-lineage-with-apache-airflow-using-openlineage?hsLang=en

+
+
datacouncil.ai
+ + + +
Data Lineage with Apache Airflow using OpenLineage
+
+ Previous Data Council talks from past conferences +
+ + + + + + +
Original URL: https://www.datacouncil.ai/talks/data-lineage-with-apache-airflow-using-openlineage?hsLang=en
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sandeep Bhat + (bhatsandeep424@gmail.com) +
+
2022-03-16 12:50:20
 +
+

I couldn't figure out for the sample lineage flow (etldelivery7_days) when we ran the seed command after from which file its fetching data

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2022-03-16 14:35:14
 +
+

*Thread Reply:* the seed data is being inserted by this command here: https://github.com/MarquezProject/marquez/blob/main/api/src/main/java/marquez/cli/SeedCommand.java

+
+ + + + +
<https://github.com/MarquezProject/marquez/blob/main/api/src/main/java/marquez/cli/SeedCommand.java | SeedCommand.java>
+
+ <p>``` +/<em>* SPDX-License-Identifier: Apache-2.0 *</em>/</p> + +<p>package marquez.cli;</p> + +<p>import static com.google.common.base.Preconditions.checkArgument; +import static marquez.client.models.JobType.BATCH; +import static marquez.common.base.MorePreconditions.checkNotBlank;</p> + +<p>import com.google.common.collect.ImmutableList; +import com.google.common.collect.ImmutableMap; +import com.google.common.collect.ImmutableSet; +import com.google.common.collect.Iterables; +import com.google.common.collect.Lists; +import com.google.common.collect.Maps; +import com.google.common.collect.Sets; +import io.dropwizard.cli.ConfiguredCommand; +import io.dropwizard.setup.Bootstrap; +import java.net.URL; +import java.time.Duration; +import java.time.Instant; +import java.time.temporal.ChronoUnit; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; +import java.util.Optional; +import java.util.Random; +import javax.annotation.Nullable; +import lombok.EqualsAndHashCode; +import lombok.Getter; +import lombok.NonNull; +import lombok.ToString; +import lombok.extern.slf4j.Slf4j; +import marquez.MarquezConfig; +import marquez.client.MarquezClient; +import marquez.client.Utils; +import marquez.client.models.Dataset; +import marquez.client.models.DatasetId; +import marquez.client.models.DatasetMeta; +import marquez.client.models.DbTableMeta; +import marquez.client.models.Field; +import marquez.client.models.Job; +import marquez.client.models.JobMeta; +import marquez.client.models.Namespace; +import marquez.client.models.NamespaceMeta; +import marquez.client.models.Run; +import marquez.client.models.RunMeta; +import marquez.client.models.Source; +import marquez.client.models.SourceMeta;</p> + +<p>/<em>*</em>* + <em>* A command to seed the HTTP API with source, dataset, and job metadata. You can override the + *</em> default {@code host} and {@code port} using the command-line arguments {@code --host} and {@code + <em>* --port}. This command is meant to be used to explore the features of Marquez. For example, + *</em> lineage graph, dataset schemas, job run history, etc. + <em>* + *</em> &lt;h3&gt;Usage&lt;/h3&gt; + <em>* + *</em> For example, to override the {@code port}: + <em>* + *</em> &lt;pre&gt;{@code + <em>* java -jar marquez-api.jar seed --port 5001 marquez.yml + *</em> }&lt;/pre&gt; + <em>* + *</em> Note that all metadata is defined within this class and requires a running instance of Marquez. + **/ +@Slf4j +public final class SeedCommand extends ConfiguredCommand&lt;MarquezConfig&gt; { + static final String DEFAULT<em>MARQUEZ</em>HOST = "localhost"; + static final int DEFAULT<em>MARQUEZ</em>PORT = 8080;</p> + +<p>public static final String NAMESPACE<em>NAME = "food</em>delivery"; + static final String SOURCE<em>NAME = "analytics</em>db";</p> + +<p>static final int LINEAGE<em>GRAPH</em>24<em>HOUR</em>WINDOW = 24; + static final int RUN<em>TIME</em>IN<em>SEC</em>MIN = 120; + static final int RUN<em>TIME</em>IN<em>SEC</em>MAX = 240;</p> + +<p>public SeedCommand() { + super("seed", "seeds the HTTP API with metadata"); + }</p> + +<p>@Override + public void configure(@NonNull final net.sourceforge.argparse4j.inf.Subparser subparser) { + super.configure(subparser); + subparser + .addArgument("--host") + .dest("host") + .type(String.class) + .required(false) + .setDefault(DEFAULT<em>MARQUEZ</em>HOST) + .help("the HTTP API server host"); + subparser + .addArgument("--port") + .dest("port") + .type(Integer.class) + .required(false) + .setDefault(DEFAULT<em>MARQUEZ</em>PORT) + .help("the HTTP API server port"); + }</p> + +<p>@Override + protected void run( + @NonNull final Bootstrap&lt;MarquezConfig&gt; bootstrap, + @NonNull final net.sourceforge.argparse4j.inf.Namespace namespace, + @NonNull final MarquezConfig config) { + final String host = namespace.getString("host"); + final int port = namespace.getInt("port");</p> + +<pre><code>final URL baseUrl = Utils.toUrl(String.format("&lt;http://%s:%d&gt;", host, port)); +final MarquezClient client = MarquezClient.builder().baseUrl(baseUrl).build(); +seedApiWithMeta(client, LINEAGE_GRAPH_24_HOUR_WINDOW); +</code></pre> + +<p>}</p> + +<p>public void seedApiWithMeta(@NonNull MarquezClient client, int additionalIterations) { + // (1) Create namespace + final NamespaceMeta namespaceMeta = + NamespaceMeta.builder() + .ownerName("<a href="mailto:owner@food.com">owner@food.com</a>") + .description("Food delivery example!") + .build(); + final Namespace newNamespace = client.createNamespace(NAMESPACE_NAME, namespaceMeta); + <a href="http://log.info">log.info</a>("Created namespace: {}", newNamespace);</p> + +<pre><code>// (2) Create source +final SourceMeta sourceMeta = + SourceMeta.builder() + .type("POSTGRESQL") + .connectionUrl("jdbc:&lt;postgres://localhost:3306/deliveries%22|postgres://localhost:3306/deliveries"&gt;) + .description("Contains all food delivery orders.") + .build(); +final Source newSource = client.createSource(SOURCE_NAME, sourceMeta); +&lt;a href="http://log.info"&gt;log.info&lt;/a&gt;("Created source: {}", newSource); + +// (3) Seed dataset meta +DATASET_META.forEach( + (datasetName, datasetMeta) -&amp;gt; { + final Dataset newDataset = client.createDataset(NAMESPACE_NAME, datasetName, datasetMeta); + &lt;a href="http://log.info"&gt;log.info&lt;/a&gt;("Created dataset: {}", newDataset); + }); + +// (4) Seed job meta +JOB_META.forEach( + (jobName, jobMeta) -&amp;gt; { + final Job newJob = client.createJob(NAMESPACE_NAME, jobName, jobMeta); + &lt;a href="http://log.info"&gt;log.info&lt;/a&gt;("Created job: {}", newJob); + }); + +// (5) Define run start times for each graph level +final Instant startTimesGraphLevel0 = Instant.now(); +final Instant startTimesGraphLevel1 = startTimesGraphLevel0.plusSeconds(RUN_TIME_IN_SEC_MAX); +final Instant startTimesGraphLevel2 = startTimesGraphLevel1.plusSeconds(RUN_TIME_IN_SEC_MAX); +final Instant startTimesGraphLevel3 = startTimesGraphLevel2.plusSeconds(RUN_TIME_IN_SEC_MAX); +final Instant startTimesGraphLevel4 = startTimesGraphLevel3.plusSeconds(RUN_TIME_IN_SEC_MAX); + +final Instant[] startTimesByGraphLevel = { + startTimesGraphLevel0, + startTimesGraphLevel1, + startTimesGraphLevel2, + startTimesGraphLevel3, + startTimesGraphLevel4 +}; + +// (6) Seed run meta for jobs using graph level start times +for (int hourOfDay = additionalIterations; hourOfDay &amp;gt;= 0; hourOfDay--) { + for (final Map.Entry&amp;lt;String, JobMeta&amp;gt; entry : JOB_META.entrySet()) { + final String jobName = entry.getKey(); + final JobMeta jobMeta = entry.getValue(); + + if (hourOfDay &amp;gt;= ACTIVE_RUN_META.get(jobName).size()) { + continue; + } + + // On code change, create a new job version + final ActiveRunMeta activeRunMeta = ACTIVE_RUN_META.get(jobName).get(hourOfDay); + activeRunMeta + .getCodeChange() + .ifPresent( + codeChange -&amp;gt; { + client.createJob( + NAMESPACE_NAME, + jobName, + JobMeta.builder() + .type(jobMeta.getType()) + .inputs(jobMeta.getInputs()) + .outputs(jobMeta.getOutputs()) + .location(codeChange.getToUrl()) + .context(jobMeta.getContext()) + .description(jobMeta.getDescription().orElse(null)) + .build()); + }); + + // Set run start and end times + final Instant runStartedAt = + startTimesByGraphLevel[activeRunMeta.getLevelInGraph()].minus( + Duration.ofHours(hourOfDay)); + final Instant runEndedAt = runStartedAt.plusSeconds(secondsToAdd()); + + // Create run + final RunMeta runMeta = + RunMeta.builder() + .nominalStartTime(runStartedAt.truncatedTo(ChronoUnit.MINUTES)) + .nominalEndTime(runEndedAt.truncatedTo(ChronoUnit.MINUTES)) + .build(); + final Run run = client.createRun(NAMESPACE_NAME, jobName, runMeta); + &lt;a href="http://log.info"&gt;log.info&lt;/a&gt;("Create… +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/MarquezProject/marquez">MarquezProject/marquez</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sandeep Bhat + (bhatsandeep424@gmail.com) +
+
2022-03-17 00:06:53
 +
+

*Thread Reply:* Got it, but if i changed the code in this java file lets say i added another job here satisfying the syntax its not appearing in the lineage flow

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-22 18:18:22
 +
+

@Kevin Mellott Hello Kevin, sorry to bother you again. I was finally able to configure Marquez in AWS using an ALB. Now I am receiving this error when calling the API

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-22 18:18:32
 +
+

Is this an issue accessing the database?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-22 18:19:15
 +
+

I created the database and host manually and passed the parameters using helm --set

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-22 18:19:33
 +
+

Do the database services need to be exposed too through the ALB?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kevin Mellott + (kevin.r.mellott@gmail.com) +
+
2022-03-23 10:20:47
 +
+

*Thread Reply:* I’m not too familiar with the 504 error in ALB, but found a guide with troubleshooting steps. If this is an issue with connectivity to the Postgres database, then you should be able to see errors within the marquez pod in EKS (kubectl logs <marquez pod name>) to confirm.

+ +

I know that EKS needs to have connectivity established to the Postgres database, even in the case of RDS, so that could be the culprit.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-23 16:09:09
 +
+

*Thread Reply:* @Kevin Mellott This is the error I am seeing in the logs +[HPM] Proxy created: /api/v1 -&gt; <http://localhost:5000/> +App listening on port 3000! +[HPM] Error occurred while trying to proxy request /api/v1/namespaces from <a href="http://marquez-interface-test.di.rbx.com">marquez-interface-test.di.rbx.com</a> to <http://localhost:5000/> (ECONNREFUSED) (<https://nodejs.org/api/errors.html#errors_common_system_errors>)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kevin Mellott + (kevin.r.mellott@gmail.com) +
+
2022-03-23 16:22:13
 +
+

*Thread Reply:* It looks like the website is attempting to find the API on localhost. I believe this can be resolved by setting the following Helm chart value within your deployment.

+ +

marquez.hostname=marquez-interface-test.di.rbx.com

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kevin Mellott + (kevin.r.mellott@gmail.com) +
+
2022-03-23 16:22:54
 +
+

*Thread Reply:* assuming that is the DNS used by the website

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-23 16:48:53
 +
+

*Thread Reply:* thanks, that did it. I have a question regarding the database

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-23 16:50:01
 +
+

*Thread Reply:* I made my own database manually. Do the marquez tables should be created automatically when install marquez?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-23 16:56:10
 +
+

*Thread Reply:* Also could you put both the API and interface on the same port (3000)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-23 17:21:58
 +
+

*Thread Reply:* Seems I am still having the forwarding issue +[HPM] Proxy created: /api/v1 -&gt; <http://marquez-interface-test.di.rbx.com:5000/> +App listening on port 3000! +[HPM] Error occurred while trying to proxy request /api/v1/namespaces from <a href="http://marquez-interface-test.di.rbx.com">marquez-interface-test.di.rbx.com</a> to <http://marquez-interface-test.di.rbx.com:5000/> (ECONNRESET) (<https://nodejs.org/api/errors.html#errors_common_system_errors>)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-03-23 09:08:14
 +
+

Guidance on How / When a Spark SQL Execution event Controls JobStart Events?

+ +

@Maciej Obuchowski and @Paweł Leszczyński and @Michael Collado I'd really appreciate your thoughts on how / when JobStart events are triggered for a given execution. I've ran into two situations now where a SQLExecutionStart event fires with execution id X and then JobStart events fire with execution id Y.

+ +

• Spark 2 Delta SaveIntoDataSourceCommand on Databricks - I see it has a SparkSQLExecutionStart event but only on Spark 3 does it have JobStart events with the SaveIntoDataSourceCommand and the same execution id. +• Databricks Synapse Connector - A SparkSQLExecutionStart event occurs but then the job starts are different execution ids. +Is there any guidance / books / videos that dive deeper into how these events are triggered?

+ +

We need the JobStart event with the same execution id so that we can get some environment properties stored in the job start event.

+ +

Thanks you so much for any guidance!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-03-23 09:25:18
 +
+

*Thread Reply:* It's always Delta, isn't it?

+ +

When I originally worked on Delta support I tried to find answer on Delta slack and got an answer:

+ +

Hi Maciej, the main reason is that Delta will run queries on metadata to figure out what files should be read for a particular version of a Delta table and that's why you might see multiple jobs. In general Delta treats metadata as data and leverages Spark to handle them to make it scalable.

+ + + +
+ 🤣 Will Johnson +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-03-23 09:25:48
 +
+

*Thread Reply:* I haven't touched how it works in Spark 2 - wanted to make it work with Spark 3's new catalogs, so can't help you there.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-03-23 09:46:14
 +
+

*Thread Reply:* Argh!! It's always Databricks doing something 🙄

+ +

Thanks, Maciej!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-03-23 09:51:59
 +
+

*Thread Reply:* One last question for you, @Maciej Obuchowski, any thoughts on how I could identify WHY a particular JobStart event fired? Is it just stepping through every event? Was that your approach to getting Spark3 Delta working? Thank you so much for the insights!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-03-23 09:58:08
 +
+

*Thread Reply:* Before that, we were using just JobStart/JobEnd events and I couldn't find events that correspond to logical plan that has anything to do with what job was actually doing. I just found out that SQLExecution events have what I want, so I just started using them and stopped worrying about Projection or Aggregate, or other events that don't really matter here - and that's how filtering idea was born: https://github.com/OpenLineage/OpenLineage/issues/423

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-03-23 09:59:37
 +
+

*Thread Reply:* Are you trying to get environment info from those events, or do you actually get Job event with proper logical plans like SaveIntoDataSourceCommand?

+ +

Might be worth to just post here all the events + logical plans that are generated for particular job, as I've done in that issue

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-03-23 09:59:40
 +
+

*Thread Reply:* scala&gt; spark.sql("CREATE TABLE tbl USING delta AS SELECT ** FROM tmp") +21/11/09 19:01:46 WARN SparkSQLExecutionContext: SparkListenerSQLExecutionStart - executionId: 3 +21/11/09 19:01:46 WARN SparkSQLExecutionContext: org.apache.spark.sql.catalyst.plans.logical.CreateTableAsSelect +21/11/09 19:01:46 WARN SparkSQLExecutionContext: SparkListenerSQLExecutionStart - executionId: 4 +21/11/09 19:01:46 WARN SparkSQLExecutionContext: org.apache.spark.sql.catalyst.plans.logical.LocalRelation +21/11/09 19:01:46 WARN SparkSQLExecutionContext: SparkListenerJobStart - executionId: 4 +21/11/09 19:01:46 WARN SparkSQLExecutionContext: org.apache.spark.sql.catalyst.plans.logical.LocalRelation +21/11/09 19:01:47 WARN SparkSQLExecutionContext: SparkListenerJobEnd - executionId: 4 +21/11/09 19:01:47 WARN SparkSQLExecutionContext: org.apache.spark.sql.catalyst.plans.logical.LocalRelation +21/11/09 19:01:47 WARN SparkSQLExecutionContext: SparkListenerSQLExecutionEnd - executionId: 4 +21/11/09 19:01:47 WARN SparkSQLExecutionContext: org.apache.spark.sql.catalyst.plans.logical.LocalRelation +21/11/09 19:01:48 WARN SparkSQLExecutionContext: SparkListenerSQLExecutionStart - executionId: 5 +21/11/09 19:01:48 WARN SparkSQLExecutionContext: org.apache.spark.sql.catalyst.plans.logical.Aggregate +21/11/09 19:01:48 WARN SparkSQLExecutionContext: SparkListenerJobStart - executionId: 5 +21/11/09 19:01:48 WARN SparkSQLExecutionContext: org.apache.spark.sql.catalyst.plans.logical.Aggregate +21/11/09 19:01:49 WARN SparkSQLExecutionContext: SparkListenerJobEnd - executionId: 5 +21/11/09 19:01:49 WARN SparkSQLExecutionContext: org.apache.spark.sql.catalyst.plans.logical.Aggregate +21/11/09 19:01:49 WARN SparkSQLExecutionContext: SparkListenerSQLExecutionEnd - executionId: 5 +21/11/09 19:01:49 WARN SparkSQLExecutionContext: org.apache.spark.sql.catalyst.plans.logical.Aggregate +21/11/09 19:01:49 WARN SparkSQLExecutionContext: SparkListenerSQLExecutionEnd - executionId: 3 +21/11/09 19:01:49 WARN SparkSQLExecutionContext: org.apache.spark.sql.catalyst.plans.logical.CreateTableAsSelect

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-03-23 11:41:37
 +
+

*Thread Reply:* The JobStart event contains a Properties field and that contains a bunch of fields we want to extract to get more precise lineage information within Databricks.

+ +

As far as we know, the SQLExecutionStart event does not have any way to get these properties :(

+ +

https://github.com/OpenLineage/OpenLineage/blob/21b039b78bdcb5fb2e6c2489c4de840ebb[…]ark/agent/facets/builder/DatabricksEnvironmentFacetBuilder.java

+ +

As a result, I do have to care about the subsequent JobStart events coming from a given ExecutionStart 😢

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/21b039b78bdcb5fb2e6c2489c4de840ebb138759/integration/spark/src/main/common/java/io/openlineage/spark/agent/facets/builder/DatabricksEnvironmentFacetBuilder.java | DatabricksEnvironmentFacetBuilder.java>
+
+ <pre><code> private HashMap&amp;lt;String, Object&amp;gt; getDatabricksEnvironmentalAttributes( + SparkListenerJobStart jobStart) { + dbProperties = new HashMap&amp;lt;&amp;gt;(); + // These are useful properties to extract if they are available + + List&amp;lt;String&amp;gt; dbPropertiesKeys = + Arrays.asList( + "orgId", + "spark.databricks.clusterUsageTags.clusterOwnerOrgId", + "spark.databricks.notebook.path", + "spark.databricks.job.type", + "spark.databricks.job.id", + "spark.databricks.job.runId", + "user", + "userId", + "spark.databricks.clusterUsageTags.clusterName", + "spark.databricks.clusterUsageTags.azureSubscriptionId"); + dbPropertiesKeys.stream() + .forEach( + (p) -&amp;gt; { + dbProperties.put(p, jobStart.properties().getProperty(p)); + }); + + /**** + ** Azure Databricks makes available a dbutils mount point to list aliased paths to cloud + ** storage. However, that dbutils object is not available inside a spark listener. We must + ** access it via reflection. + **/ + try { + dbutilsClass = Class.forName("com.databricks.dbutils_v1.impl.DbfsUtilsImpl"); + dbutils = (DbfsUtils) dbutilsClass.getDeclaredConstructor().newInstance(); + dbProperties.put("mountPoints", getDatabricksMountpoints(dbutils)); + } catch (Exception e) { + log.warn("Failed to load dbutils in OpenLineageListener"); + dbProperties.put("mountPoints", new ArrayList&amp;lt;DatabricksMountpoint&amp;gt;()); + } + + return dbProperties; + } +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-03-23 11:42:33
 +
+

*Thread Reply:* I started down this path with the Project statement but I agree with @Michael Collado that a ProjectVisitor isn't a great idea.

+ +

https://github.com/OpenLineage/OpenLineage/issues/617

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/issues/617|#617 [INTEGRATION][SPARK] Support org.apache.spark.sql.catalyst.plans.logical.Project for Delta Tables>
+
+ <p>It seems that a <code>Project</code> logical plan is the terminal node for a Delta event. Currently OpenLineage does not support this as a Visitor and as a result, there are no output datasets. I'd like to propose adding a ProjectVisitor that extracts the appropriate output dataset contents.</p> + +<p>This example demonstrates the issue on Azure Databricks Runtime 6.4 (Spark 2.4.5)</p> + +<pre><code>val storageServiceName = "inputservice" +val storageContainerName = "inputcontainer" +val abfssRootPath = "abfss://"+storageContainerName+"@"+storageServiceName+".&lt;a href="http://dfs.core.windows.net"&gt;dfs.core.windows.net&lt;/a&gt;" +val outRootPath = "abfss://"+"outputdata"+"@"+storageServiceName+".&lt;a href="http://dfs.core.windows.net"&gt;dfs.core.windows.net&lt;/a&gt;" + +val exampleA = ( + spark.read.format("delta") + .load(abfssRootPath+"/examples/data/delta/exampleInputA") +) + +val exampleB = ( + spark.read.format("delta") + .load(abfssRootPath+"/examples/data/delta/exampleInputB") +) +val outputDf = exampleA.join(exampleB, exampleA("id") === exampleB("id"), "inner").drop(exampleB("id")) + +outputDf.write.format("delta").mode("append").save(outRootPath + "/outputdata/delta/delta-in-delta-out") +</code></pre> + +<p>Here's the truncated open lineage event that is produced:</p> + +<pre><code>{ + "eventType": "START", + "eventTime": "2022-03-18T12:58:53.112Z", + "run": { + "runId": "6068fde5-0e3e-4bd6-8bd9-fc3ebdb283ce", + "facets": { + "spark_unknown": { + "_producer": "&lt;https://github.com/OpenLineage/OpenLineage/tree/0.6.0-SNAPSHOT/integration/spark&gt;", + "_schemaURL": "&lt;https://openlineage.io/spec/1-0-2/OpenLineage.json#/$defs/RunFacet&gt;", + "output": { + "description": { + "@class": "org.apache.spark.sql.catalyst.plans.logical.Project", + "id": 1, + "projectList": ["some columns"], + "origin": 5, + "tags": {}, + "schema": { + "type": "struct", + "fields": ["the fields"] + } + }, + "spark_version": { + "_producer": "&lt;https://github.com/OpenLineage/OpenLineage/tree/0.6.0-SNAPSHOT/integration/spark&gt;", + "_schemaURL": "&lt;https://openlineage.io/spec/1-0-2/OpenLineage.json#/$defs/RunFacet&gt;", + "spark-version": "2.4.5", + "openlineage-spark-version": "0.6.0-SNAPSHOT" + } + } + }, + "job": { + "namespace": "adbpurviewol1", + "name": "databricks_shell.project", + "facets": {} + }, + "inputs": [ + { + "namespace": "&lt;abfss://CONTAINER@SERVICE.dfs.core.windows.net&gt;", + "name": "/examples/data/delta/exampleInputA", + "facets": { + "dataSource": { + "_producer": "&lt;https://github.com/OpenLineage/OpenLineage/tree/0.6.0-SNAPSHOT/integration/spark&gt;", + "_schemaURL": "&lt;https://openlineage.io/spec/facets/1-0-0/DatasourceDatasetFacet.json#/$defs/DatasourceDatasetFacet&gt;", + "name": "&lt;abfss://CONTAINER@SERVICE.dfs.core.windows.net&gt;", + "uri": "&lt;abfss://CONTAINER@SERVICE.dfs.core.windows.net&gt;" + }, + "schema": { + "_producer": "&lt;https://github.com/OpenLineage/OpenLineage/tree/0.6.0-SNAPSHOT/integration/spark&gt;", + "_schemaURL": "&lt;https://openlineage.io/spec/facets/1-0-0/SchemaDatasetFacet.json#/$defs/SchemaDatasetFacet&gt;", + "fields": [ + { + "name": "id", + "type": "integer" + }, + { + "name": "postalCode", + "type": "string" + }, + { + "name": "streetAddress", + "type": "string" + } + ] + } + }, + "inputFacets": {} + }, + { + "namespace": "&lt;abfss://CONTAINER@SERVICE.dfs.core.windows.net&gt;", + "name": "/examples/data/delta/exampleInputB", + "facets": { + "dataSource": { + "_producer": "&lt;https://github.com/OpenLineage/OpenLineage/tree/0.6.0-SNAPSHOT/integration/spark&gt;", + "_schemaURL": "&lt;https://openlineage.io/spec/facets/1-0-0/DatasourceDatasetFacet.json#/$defs/DatasourceDatasetFacet&gt;", + "name": "&lt;abfss://CONTAINER@SERVICE.dfs.core.windows.net&gt;", + "uri": "&lt;abfss://CONTAINER@SERVICE.dfs.core.windows.net&gt;" + }, + "schema": { + "_producer": "&lt;https://github.com/OpenLineage/OpenLineage/tree/0.6.0-SNAPSHOT/integration/spark&gt;", + "_schemaURL": "&lt;https://openlineage.io/spec/facets/1-0-0/SchemaDatasetFacet.json#/$defs/SchemaDatasetFacet&gt;", + "fields": [ + { + "name": "id", + "type": "integer" + }, + { + "name": "city", + "type": "string" + }, + { + "name": "stateAbbreviation", + "type": "string" + } + ] + } + }, + "inputFacets": {} + } + ], + "outputs": [], + "producer": "&lt;https://github.com/OpenLineage/OpenLineage/tree/0.6.0-SNAPSHOT/integration/spark&gt;", + "schemaURL": "&lt;https://openlineage.io/spec/1-0-2/OpenLineage.json#/$defs/RunEvent&gt;" +} +</code></pre> +
+ +
+
Comments
+ 2 +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-03-24 09:43:38
 +
+

Hey. I'm working on replacing current SQL parser - on which we rely for Postgres, Snowflake, Great Expectations - and I'd appreciate your opinion.

+ +

https://github.com/OpenLineage/OpenLineage/pull/627/files

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-25 19:30:29
 +
+

Am i supposed to see this when I open marquez fro the first time on an empty database?

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2022-03-25 20:33:02
 +
+

*Thread Reply:* Marquez and OpenLineage are job-focused lineage tools, so once you run a job in an OL-integrated instance of Airflow (or any other supported integration), you should see the jobs and DBs appear in the marquez ui

+ + + +
+ 👍 Marco Diaz +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-03-25 21:44:54
 +
+

*Thread Reply:* If you want to seed it with some data, just to try it out, you can run docker/up.sh -s and it will run a seeding job as it starts.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-25 19:31:09
 +
+

Would datasets be created when I send data from airflow?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2022-03-31 18:34:40
 +
+

*Thread Reply:* Yep! Marquez will register all in/out datasets present in the OL event as well as link them to the run

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2022-03-31 18:35:47
 +
+

*Thread Reply:* FYI, @Peter Hicks is working on displaying the dataset version to run relationship in the web UI, see https://github.com/MarquezProject/marquez/pull/1929

+
+ + + + +
<https://github.com/MarquezProject/marquez/pull/1929|#1929 Web: Created by view for dataset versions>
+
+ <p>Signed-off-by: Peter Hicks <a href="mailto:peter@datakin.com">peter@datakin.com</a></p> + +<p><strong>Problem</strong></p> + +<p>Some Marquez users were eager to see the run that created a dataset version.</p> + +<p><strong>Solution</strong></p> + +<p>We've added the <code>Created by Run</code> section on the dataset page as well as the dataset version list view. Also updated is syntax highlighting support for <code>sql</code> contexts and the empty states for context that is missing for us. <br /> +<a href="https://user-images.githubusercontent.com/7514204/160910381-644773d7-6ed6-499d-ad96-2ebdde14495b.png">image</a> <br /> +<a href="https://user-images.githubusercontent.com/7514204/160910466-5c62380f-0ad1-4e14-8292-c5c0b98b5ccb.png">image</a> <br /> +<a href="https://user-images.githubusercontent.com/7514204/160910773-4fbc6329-5ec3-483e-bf82-9269c0223906.png">image</a> <br /> +<a href="https://user-images.githubusercontent.com/7514204/160917522-b6a6c52f-e2bb-4590-85c1-6f6ed46b4eb7.png">image</a></p> + +<p>&gt; <strong>Note:</strong> All database schema changes require discussion. Please <a href="https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue">link the issue</a> for context.</p> + +<p><strong>Checklist</strong></p> + +<p>☑︎ You've <a href="https://github.com/MarquezProject/marquez/blob/main/CONTRIBUTING.md#sign-your-work">signed-off</a> your work +☐ Your changes are accompanied by tests (<em>if relevant</em>) +☑︎ Your change contains a <a href="https://kurtisnusbaum.medium.com/stacked-diffs-keeping-phabricator-diffs-small-d9964f4dcfa6">small diff</a> and is self-contained +☐ You've updated any relevant documentation (<em>if relevant</em>) +☐ You've updated the <a href="https://github.com/MarquezProject/marquez/blob/main/CHANGELOG.md#unreleased"><code>CHANGELOG.md</code></a> with details about your change under the "Unreleased" section (<em>if relevant, depending on the change, this may not be necessary</em>) +☐ You've versioned your <code>.sql</code> database schema migration according to <a href="https://flywaydb.org/documentation/concepts/migrations#naming">Flyway's naming convention</a> (<em>if relevant</em>)</p> +
+ +
+
Labels
+ feature, review, web, javascript +
+ +
+
Comments
+ 1 +
+ + + + + + + + + +
+ + <a href="https://github.com/MarquezProject/marquez">MarquezProject/marquez</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-28 14:31:32
 +
+

How is Datakin used in conjunction with Openlineage and Marquez?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2022-03-28 15:43:46
 +
+

*Thread Reply:* Hi Marco,

+ +

Datakin is a reporting tool built on the Marquez API, and therefore designed to take in Lineage using the OpenLineage specification.

+ +

Did you have a more specific question?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-28 15:47:53
 +
+

*Thread Reply:* No, that is it. Got it. So, i can install Datakin and still use openlineage and marquez?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2022-03-28 15:55:07
 +
+

*Thread Reply:* if you set up a datakin account, you'll have to change the environment variables used by your OpenLineage integrations, and the runEvents will be sent to Datakin rather than Marquez. You shouldn't have any loss of functionality, and you also won't have to keep manually hosting Marquez

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-28 16:10:25
 +
+

*Thread Reply:* Will I still be able to use facets for backfills?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2022-03-28 17:04:03
 +
+

*Thread Reply:* yeah it works in the same way - Datakin actually submodules the Marquez API

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-28 16:52:41
 +
+

Another question. I installed the open-lineage library and now I am trying to configure Airflow 2 to use it +Do I follow these steps?

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-28 16:53:20
 +
+

If I have marquez access via alb ingress what would i use the marquezurl variable or openlineageurl?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-28 16:54:53
 +
+

So, i don't need to modify my dags in Airflow 2 to use the library? Would this just allow me to start collecting data? +openlineage.lineage_backend.OpenLineageBackend

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-03-29 06:24:21
 +
+

*Thread Reply:* Yes, you don't need to modify dags in Airflow 2.1+

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-29 17:47:39
 +
+

*Thread Reply:* ok, I added that environment variable. Now my question is how do i configure my other variables. +I have marquez running in AWS with an ingress. +Do i use OpenLineageURL or Marquez_URL?

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-29 17:48:09
 +
+

*Thread Reply:* Also would a new namespace be created if i add the variable?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
data_fool + (data.fool.me@gmail.com) +
+
2022-03-29 02:12:30
 +
+

Hello! Are there any plans for openlineage to support dbt on trino?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2022-03-30 14:59:13
 +
+

*Thread Reply:* Hi Datafool - I'm not familiar with how trino works, but the DBT-OL integration works by wrapping the dbt run command with dtb-ol run , and capturing lineage data from the runresult file

+ +

These things don't necessarily preclude you from using OpenLineage on trino, so it may work already.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
data_fool + (data.fool.me@gmail.com) +
+
2022-03-30 18:34:38
 +
+

*Thread Reply:* hey @John Thomas yep, tried to use dbt-ol run command but it seems trino is not supported, only bigquery, redshift and few others.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2022-03-30 18:36:41
 +
+

*Thread Reply:* aaah I misunderstood what Trino is - yeah we don't currently support jobs that are running outside of those environments.

+ +

We don't currently have plans for this, but a great first step would be opening an issue in the OpenLineage repo.

+ +

If you're interested in implementing the support yourself I'm also happy to connect you to people that can help you get started.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
data_fool + (data.fool.me@gmail.com) +
+
2022-03-30 20:23:46
 +
+

*Thread Reply:* oh okay, got it, yes I can contribute, I'll see if I can get some time in the next few weeks. Thanks @John Thomas

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Francis McGregor-Macdonald + (francis@mc-mac.com) +
+
2022-03-30 16:08:39
 +
+

I can see 2 articles using Spline with BMW and Capital One. Could OpenLineage be doing the same job as Spline here? What would the differences be? +Are there any similar references for OpenLineage? I can see Northwestern Mutual but that article does not contain a lot of detail.

+
+
SpringerLink
+ + + +
Collecting and visualizing data lineage of Spark jobs
+
+ Datenbank-Spektrum - Metadata management constitutes a key prerequisite for enterprises as they engage in data analytics and governance. Today, however, the context of data is often only manually... +
+ + + + + + + + +
Original URL: https://link.springer.com/article/10.1007/s13222-021-00387-7
+ + + +
+
+
Capital One
+ + + +
Capturing &amp; Displaying Data Transformations with Spline | Capital One
+
+ Capturing data lineage manually--especially as an application’s logic changes over time--can be time consuming and prone to human error. As a data engineer, it’s advantageous to capture data lineage as a normal part of application flow, and to be able to display that data lineage in a format that is easily understood for documentation purposes. +
+ + + + + + + + +
Original URL: https://www.capitalone.com/tech/software-engineering/spline-spark-data-lineage/
+ + + +
+
+
openlineage.io
+ + + +
How Northwestern Mutual Simplified Data Observability with OpenLineage &amp; Marquez
+
+ Northwestern Mutual is building an Enterprise Data Platform. In this guest blog, learn about the experiences and decisions that led them to embrace the OpenLineage and Marquez communities. +
+ + + + + + +
Original URL: https://openlineage.io/blog/openlineage-at-northwestern-mutual/
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-31 12:47:59
 +
+

Could anyone help me wit this custom extractor. I am not sure what I am doing wrong. I added the variable to airflow2, but I still see this in the logs +[2022-03-31, 16:43:39 UTC] {__init__.py:97} WARNING - Unable to find an extractor. task_type=QueryOperator +Here is the code

+ +

```import logging +from typing import Optional, List +from openlineage.airflow.extractors.base import BaseExtractor,TaskMetadata +from openlineage.client.facet import SqlJobFacet, ExternalQueryRunFacet +from openlineage.common.sql import SqlMeta, SqlParser

+ +

logger = logging.getLogger(name)

+ +

class QueryOperatorExtractor(BaseExtractor):

+ +
def __init__(self, operator):
+    super().__init__(operator)
+
+@classmethod
+def get_operator_classnames(cls) -&gt; List[str]:
+    return ['QueryOperator']
+
+def extract(self) -&gt; Optional[TaskMetadata]:
+    # (1) Parse sql statement to obtain input / output tables.
+    sql_meta: SqlMeta = SqlParser.parse(self.operator.hql)
+    inputs = sql_meta.in_tables
+    outputs = sql_meta.out_tables
+    task_name = f"{self.operator.dag_id}.{self.operator.task_id}"
+    run_facets = {}
+    job_facets = {
+        'hql': SqlJobFacet(self.operator.hql)
+    }
+
+    return TaskMetadata(
+        name=task_name,
+        inputs=[inputs.to_openlineage_dataset()],
+        outputs=[outputs.to_openlineage_dataset()],
+        run_facets=run_facets,
+        job_facets=job_facets
+    )```
+
+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Orbit + +
+
2022-03-31 13:20:55
 +
+

@Orbit has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Orbit + +
+
2022-03-31 13:21:23
 +
+

@Orbit has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-31 14:07:24
 +
+

@Ross Turk Could you please take a look if you have a minute☝️? I know you have built one extractor before

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-03-31 14:11:35
 +
+

*Thread Reply:* Hmmmm. Are you running in Docker? Is it possible for you to shell into your scheduler container and make sure the ENV is properly set?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-03-31 14:11:57
 +
+

*Thread Reply:* looks to me like the value you posted is correct, and return ['QueryOperator'] seems right to me

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-31 14:33:00
 +
+

*Thread Reply:* It is in an EKS cluster +I checked and the variable is there +OPENLINEAGE_EXTRACTOR_QUERYOPERATOR=shared.plugins.ol_custom_extractors.QueryOperatorExtractor

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-31 14:33:56
 +
+

*Thread Reply:* I am wondering if it is an issue with my extractor code. Something not rendering well

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-03-31 14:40:17
 +
+

*Thread Reply:* I don’t think it’s even executing your extractor code. The error message traces back to here: +https://github.com/OpenLineage/OpenLineage/blob/249868fa9b97d218ee35c4a198bcdf231a9b874b/integration/airflow/openlineage/lineage_backend/__init__.py#L77

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/249868fa9b97d218ee35c4a198bcdf231a9b874b/integration/airflow/openlineage/lineage_backend/__init__.py | __init__.py>
+
+ <pre><code> extractor = self._get_extractor(task) +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-03-31 14:40:45
 +
+

*Thread Reply:* I am currently digging into _get_extractor to see where it might be missing yours 🤔

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-31 14:46:36
 +
+

*Thread Reply:* Thanks

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-03-31 14:47:19
 +
+

*Thread Reply:* silly idea, but you could add a log message to __init__ in your extractor.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-03-31 14:47:25
 +
+

*Thread Reply:* https://github.com/OpenLineage/OpenLineage/blob/249868fa9b97d218ee35c4a198bcdf231a[…]ntegration/airflow/openlineage/airflow/extractors/extractors.py

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/249868fa9b97d218ee35c4a198bcdf231a9b874b/integration/airflow/openlineage/airflow/extractors/extractors.py | extractors.py>
+
+ <pre><code> self.extractors[operator] = import_from_string(value) +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-03-31 14:48:20
 +
+

*Thread Reply:* the openlineage client actually tries to import the value of that env variable from pos 22. if that happens, but for some reason it fails to register the extractor, we can at least know that it’s importing

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-03-31 14:48:54
 +
+

*Thread Reply:* if you add a log line, you can verify that your PYTHONPATH and env are correct

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-31 14:49:23
 +
+

*Thread Reply:* will try that

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-31 14:49:29
 +
+

*Thread Reply:* and let you know

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-03-31 14:49:39
 +
+

*Thread Reply:* ok!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-03-31 15:04:05
 +
+

*Thread Reply:* @Marco Diaz can you try env variable OPENLINEAGE_EXTRACTOR_QueryOperator instead of full caps?

+ + + +
+ 👍 Ross Turk +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-31 15:13:37
 +
+

*Thread Reply:* Will try that too

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-31 15:13:44
 +
+

*Thread Reply:* Thanks for helping

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-31 16:58:24
 +
+

*Thread Reply:* @Maciej Obuchowski My setup does not allow me to submit environment variables with lowercases. Is the name of the variable used to register the extractor?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-03-31 17:15:57
 +
+

*Thread Reply:* yes, it's case sensitive...

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-31 17:18:42
 +
+

*Thread Reply:* i see

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-31 17:39:16
 +
+

*Thread Reply:* So it is definitively the name of the variable. I changed the name of the operator to capitals and now is being registered

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-31 17:39:44
 +
+

*Thread Reply:* Could there be a way not to make this case sensitive?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-03-31 18:31:27
 +
+

*Thread Reply:* yes - could you create issue on OpenLineage repository?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-04-01 10:46:59
 +
+

*Thread Reply:* sure

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-04-01 10:48:28
 +
+

I have another question. I have this query +INSERT OVERWRITE TABLE schema.daily_play_sessions_v2 + PARTITION (ds = '2022-03-30') + SELECT + platform_id, + universe_id, + pii_userid, + NULL as session_id, + NULL as session_start_ts, + COUNT(1) AS session_cnt, + SUM( + UNIX_TIMESTAMP(stopped) - UNIX_TIMESTAMP(joined) + ) AS time_spent_sec + FROM schema.fct_play_sessions_merged + WHERE ds = '2022-03-30' + AND UNIX_TIMESTAMP(stopped) - UNIX_TIMESTAMP(joined) BETWEEN 0 AND 28800 + GROUP BY + platform_id, + universe_id, + pii_userid +And I am seeing the following inputs +[DbTableName(None,'schema','fct_play_sessions_merged','schema.fct_play_sessions_merged')] +But the outputs are empty +Shouldn't this be an output table +schema.daily_play_sessions_v2

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-04-01 13:25:52
 +
+

*Thread Reply:* Yes, it should. This line is the likely culprit: +https://github.com/OpenLineage/OpenLineage/blob/431251d25f03302991905df2dc24357823d9c9c3/integration/common/openlineage/common/sql/parser.py#L30

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/431251d25f03302991905df2dc24357823d9c9c3/integration/common/openlineage/common/sql/parser.py | parser.py>
+
+ <pre><code> return _match_on(token, ['INTO']) +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-04-01 13:26:25
 +
+

*Thread Reply:* I bet if that said ['INTO','OVERWRITE'] it would work

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-04-01 13:27:23
 +
+

*Thread Reply:* @Maciej Obuchowski do you agree? should OVERWRITE be a token we look for? if so, I can submit a short PR.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-04-01 13:30:36
 +
+

*Thread Reply:* we have a better solution

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-04-01 13:30:37
 +
+

*Thread Reply:* https://github.com/OpenLineage/OpenLineage/pull/644

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/pull/644|#644 sql: implementation of parser API in Rust>
+
+ <p>Description in proposal doc: <a href="https://github.com/OpenLineage/OpenLineage/pull/627">#627</a></p> + +<p>Signed-off-by: Maciej Obuchowski <a href="mailto:obuchowski.maciej@gmail.com">obuchowski.maciej@gmail.com</a></p> + +<p>Closes: <a href="https://github.com/OpenLineage/OpenLineage/issues/582">#582</a></p> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-04-01 13:31:27
 +
+

*Thread Reply:* ah! I heard there was a new SQL parser, but did not know it was imminent!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-04-01 13:31:30
 +
+

*Thread Reply:* I've added this case as a test and it works: https://github.com/OpenLineage/OpenLineage/blob/764dfdb885112cd0840ebc7384ff958bf20d4a70/integration/sql/tests/tests_insert.rs

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/764dfdb885112cd0840ebc7384ff958bf20d4a70/integration/sql/tests/tests_insert.rs | tests_insert.rs>
+
+ <pre><code>use openlineage_sql::{parse_sql, SqlMeta}; +use sqlparser::dialect::PostgreSqlDialect; + +#[macro_use] +mod test_utils; +use test_utils::**; + +#[test] +fn insert_values() { + assert_eq!( + test_sql("INSERT INTO TEST VALUES(1)",), + SqlMeta { + in_tables: vec![], + out_tables: table("TEST") + } + ); +} + +#[test] +fn insert_cols_values() { + assert_eq!( + test_sql("INSERT INTO tbl(col1, col2) VALUES (1, 2), (2, 3)",), + SqlMeta { + in_tables: vec![], + out_tables: table("tbl") + } + ); +} + +#[test] +fn insert_select_table() { + assert_eq!( + test_sql("INSERT INTO TEST SELECT ** FROM TEMP",), + SqlMeta { + in_tables: table("TEMP"), + out_tables: table("TEST") + } + ); +} + +#[test] +fn insert_nested_select() { + assert_eq!(test_sql(" + INSERT INTO popular_orders_day_of_week (order_day_of_week, order_placed_on,orders_placed) + SELECT EXTRACT(ISODOW FROM order_placed_on) AS order_day_of_week, + order_placed_on, + COUNT(**) AS orders_placed + FROM top_delivery_times + GROUP BY order_placed_on; + ", + ), SqlMeta { + in_tables: table("top_delivery_times"), + out_tables: table("popular_orders_day_of_week") + }) +} + +#[test] +fn insert_cmpl() { + assert_eq!( + test_sql( + "\ + INSERT OVERWRITE TABLE schema.daily_play_sessions_v2 + PARTITION (ds = '2022-03-30') + SELECT + platform_id, + universe_id, + pii_userid, + NULL as session_id, + NULL as session_start_ts, + COUNT(1) AS session_cnt, + SUM( + UNIX_TIMESTAMP(stopped) - UNIX_TIMESTAMP(joined) + ) AS time_spent_sec + FROM schema.fct_play_sessions_merged + WHERE ds = '2022-03-30' + AND UNIX_TIMESTAMP(stopped) - UNIX_TIMESTAMP(joined) BETWEEN 0 AND 28800 + GROUP BY + platform_id, + universe_id, + pii_userid + " + ), + SqlMeta { + in_tables: table("schema.fct_play_sessions_merged"), + out_tables: table("schema.daily_play_sessions_v2") + } + ) +} + +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+ 👍 Ross Turk, Paweł Leszczyński +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-04-01 13:31:33
 +
+

*Thread Reply:* let me review this PR

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-04-01 13:36:32
 +
+

*Thread Reply:* Do i have to download a new version of the opelineage-airflow python library

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-04-01 13:36:41
 +
+

*Thread Reply:* If so which version?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-04-01 13:37:22
 +
+

*Thread Reply:* this PR isn’t merged yet 😞 so if you wanted to try this you’d have to build the python client from the sql/rust-parser-impl branch

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-04-01 13:38:17
 +
+

*Thread Reply:* ok, np. I am not in a hurry yet. Do you have an ETA for the merge?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-04-01 13:39:50
 +
+

*Thread Reply:* Hard to say, it’s currently in-review. Let me pull some strings, see if I can get eyes on it.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-04-01 13:40:34
 +
+

*Thread Reply:* I will check again next week don't worry. I still need to make some things in my extractor work

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-04-01 13:40:36
 +
+

*Thread Reply:* after it’s merged, we’ll have to do an OpenLineage release as well - perhaps next week?

+ + + +
+ 👍 Michael Robinson +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-04-01 13:40:41
 +
+

*Thread Reply:* 👍

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tien Nguyen + (tiennguyenhotel97@gmail.com) +
+
2022-04-01 12:25:48
 +
+

Hi everyone, I just started using openlineage to connect with DBT for my company. I work as data engineering. After the connection and run test on dbt-ol run, it gives me this error. I have looked up online to find the answer but couldn't see the answer anywhere. Can somebody please help me with? The error tells me that the correct version is DBT Schemajson version 2 instead of 3. I don't know where to change the schemajson version. Thank you everyone @channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-04-01 13:34:10
 +
+

*Thread Reply:* Hm - what version of dbt are you using?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-04-01 13:47:50
 +
+

*Thread Reply:* @Tien Nguyen The dbt schema version changes with different versions of dbt. If you have recently updated, you may have to make some changes: https://docs.getdbt.com/docs/guides/migration-guide/upgrading-to-v1.0

+
+
docs.getdbt.com
+ + + +
Upgrading to v1.0 | dbt Docs
+
+ Resources +
+ + + + + + + + +
Original URL: https://docs.getdbt.com/docs/guides/migration-guide/upgrading-to-v1.0
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-04-01 13:48:27
 +
+

*Thread Reply:* also make sure you are on the latest version of openlineage-dbt - I believe we have made it a bit more tolerant of dbt schema changes.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tien Nguyen + (tiennguyenhotel97@gmail.com) +
+
2022-04-01 13:52:46
 +
+

*Thread Reply:* @Ross Turk Thank you very much for your answer. I will update those and see if I can resolve the issues.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tien Nguyen + (tiennguyenhotel97@gmail.com) +
+
2022-04-01 14:20:00
 +
+

*Thread Reply:* @Ross Turk Thank you very much for your help. The latest version of dbt couldn't work. But version 0.20.0 works for this problem.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-04-01 14:22:42
 +
+

*Thread Reply:* Hmm. Interesting, I remember when dbt 1.0 came out we fixed a very similar issue: https://github.com/OpenLineage/OpenLineage/pull/397

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/pull/397|#397 [INTEGRATION][dbt] allow to parse unspecified, future version of artifacts>
+
+ <p>Currently, if <code>DbtMetadataArtifact</code> encounters artifact with schema version not exactly matching one that is supported by <code>dbt-ol</code>, it throws exception. This PR changes this behavior, when the metadata version is higher than currently supported (and tested with). If we encounter artifact with higher version, we warn the user and continue processing.</p> + +<p>Also, change raw <code>print()</code> calls to logger calls - this allows more granular control over output, and allows to test it easily.</p> + +<p>Signed-off-by: Maciej Obuchowski <a href="mailto:maciej.obuchowski@getindata.com">maciej.obuchowski@getindata.com</a></p> + +<p>Closes: <a href="https://github.com/OpenLineage/OpenLineage/issues/389">#389</a></p> + +<p><strong>Checklist</strong></p> + +<p>☑︎ You've <a href="https://github.com/OpenLineage/OpenLineage/blob/main/why-the-dco.md">signed-off</a> your work +☑︎ Your pull request title follows our <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CONTRIBUTING.md#creating-pull-requests">guidelines</a> +☑︎ Your changes are accompanied by tests (<em>if relevant</em>) +☑︎ Your change contains a <a href="https://kurtisnusbaum.medium.com/stacked-diffs-keeping-phabricator-diffs-small-d9964f4dcfa6">small diff</a> and is self-contained +☐ You've updated any relevant documentation (<em>if relevant</em>) +☐ You've updated the <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md"><code>CHANGELOG.md</code></a> with details about your change under the "Unreleased" section (<em>if relevant, depending on the change, this may not be necessary</em>) +☐ You've versioned the core OpenLineage model or facets according to <a href="https://docs.snowplowanalytics.com/docs/pipeline-components-and-applications/iglu/common-architecture/schemaver">SchemaVer</a> (<em>if relevant</em>)</p> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-04-01 14:25:17
 +
+

*Thread Reply:* if you run pip3 list | grep openlineage-dbt, what version does it show?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-04-01 14:26:26
 +
+

*Thread Reply:* I wonder if you have somehow ended up with an older version of the integration

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tien Nguyen + (tiennguyenhotel97@gmail.com) +
+
2022-04-01 14:33:43
 +
+

*Thread Reply:* it is 0.1.0

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tien Nguyen + (tiennguyenhotel97@gmail.com) +
+
2022-04-01 14:34:23
 +
+

*Thread Reply:* is it 0.1.0 the older version of openlineage ?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-04-01 14:43:14
 +
+

*Thread Reply:* ❯ pip3 list | grep openlineage-dbt +openlineage-dbt 0.6.2

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-04-01 14:43:26
 +
+

*Thread Reply:* the latest is 0.6.2 - that might be your issue

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-04-01 14:43:59
 +
+

*Thread Reply:* How are you going about installing it?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tien Nguyen + (tiennguyenhotel97@gmail.com) +
+
2022-04-01 18:35:26
 +
+

*Thread Reply:* @Ross Turk. I follow instruction from open lineage "pip3 install openlineage-dbt"

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-04-01 18:36:00
 +
+

*Thread Reply:* Hm! Interesting. I did the same thing to get 0.6.2.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tien Nguyen + (tiennguyenhotel97@gmail.com) +
+
2022-04-01 18:51:36
 +
+

*Thread Reply:* @Ross Turk Yes. I have tried to reinstall and clear cache but it still install 0.1.0

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tien Nguyen + (tiennguyenhotel97@gmail.com) +
+
2022-04-01 18:53:07
 +
+

*Thread Reply:* But thanks for the version. I reinstall 0.6.2 version by specify the version

+ + + +
+ 👍 Ross Turk +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-04-02 17:37:59
 +
+

@Ross Turk @Maciej Obuchowski FYI the sql parser also seems not to return any inputs or outpus for queries that have subqueries +Example +INSERT OVERWRITE TABLE mytable + PARTITION (ds = '2022-03-31') + SELECT + ** + FROM + (SELECT ** FROM table2) a +INSERT OVERWRITE TABLE mytable + PARTITION (ds = '2022-03-31') + SELECT + ** + FROM + (SELECT ** FROM table2 + UNION + SELECT ** FROM table3 + UNION ALL + SELECT ** FROM table4) a

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-04-03 15:07:09
 +
+

*Thread Reply:* they'll work with new parser - added test for those

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/2e8e51981483e17534445a751723dda787794739/integration/sql/tests/tests_insert.rs | tests_insert.rs>
+
+ <pre><code>fn insert_overwrite_multiple_subqueries() { +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-04-03 15:07:39
 +
+

*Thread Reply:* btw, thank you very much for notifying us about multiple bugs @Marco Diaz!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-04-03 15:20:55
 +
+

*Thread Reply:* @Maciej Obuchowski thank you for making sure these cases are taken into account. I am getting more familiar with the Open lineage code as i build my extractors. If I see anything else I will let you know. Any ETA on the new parser release date?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-04-03 15:55:28
 +
+

*Thread Reply:* it should be week-two, unless anything comes up

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-04-03 17:10:02
 +
+

*Thread Reply:* I see. Keeping my fingers crossed this is the only thing delaying me right now.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-04-02 20:27:37
 +
+

Also what would happen if someone uses a CTE in the SQL? Is the parser taken those cases in consideration?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-04-03 15:02:13
 +
+

*Thread Reply:* current one handles cases where you have one CTE (like this test) but not multiple - next one will handle arbitrary number of CTEs (like this test)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-04-04 10:54:47
 +
+

Agenda items are requested for the next OpenLineage Technical Steering Committee meeting on Wednesday, April 13. Please reply here or ping me with your items!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-04-04 11:11:53
 +
+

*Thread Reply:* I've mentioned it before but I want to talk a bit about new SQL parser

+ + + +
+ 🙌 Will Johnson, Ross Turk +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-04-04 13:25:17
 +
+

*Thread Reply:* Will the parser be released after the 13?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-04-08 11:47:05
 +
+

*Thread Reply:* @Michael Robinson added additional item to Agenda - client transports feature that we'll have in next release

+ + + +
+ 🙌 Michael Robinson +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-04-08 12:56:44
 +
+

*Thread Reply:* Thanks, Maciej

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sukanya Patra + (Sukanya_Patra@mckinsey.com) +
+
2022-04-05 02:39:59
 +
+

Hi Everyone,

+ +

I have come across OpenLineage at Data Council Austin, 2022 and am curious to try it out. I have reviewed the Getting Started section (https://openlineage.io/getting-started/) of OpenLineage docs but couldn't find clear reference documentation for using the API +• Are there any swagger API docs or equivalent dedicated for OpenLineage API? There is some reference docs of Marquez API: https://marquezproject.github.io/marquez/openapi.html#tag/Lineage +Secondly are there any means to use Open Lineage independent of Marquez? Any pointers would be appreciated.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Patrick Mol + (patrick.mol@prolin.com) +
+
2022-04-05 10:28:08
 +
+

*Thread Reply:* I had kind of the same question. +I found https://marquezproject.github.io/marquez/openapi.html#tag/Lineage +With some of the entries marked Deprecated, I am not sure how to proceed.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2022-04-05 11:55:35
 +
+

*Thread Reply:* Hey folks, are you looking for the OpenAPI specification found here?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2022-04-05 15:33:23
 +
+

*Thread Reply:* @Patrick Mol, Marquez's deprecated endpoints were the old methods for creating lineage (making jobs, dataset, and runs independently), they were deprecated because we moved over to using the OpenLineage spec for all lineage collection purposes.

+ +

The GET methods for jobs/datasets/etc are still functional

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sarat Chandra + (saratchandra9494@gmail.com) +
+
2022-04-05 21:10:39
 +
+

*Thread Reply:* Hey John,

+ +

Thanks for sharing the OpenAPI docs. Was wondering if there are any means to setup OpenLineage API that will receive events without a consumer like Marquez or is it essential to always pair with a consumer to receive the events?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2022-04-05 21:47:13
 +
+

*Thread Reply:* the OpenLineage integrations don’t have any way to recieve events, since they’re designed to send events to other apps - what were you expecting OpenLinege to do?

+ +

Marquez is our reference implementation of an OpenLineage consumer, but egeria also has a functional endpoint

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Patrick Mol + (patrick.mol@prolin.com) +
+
2022-04-06 09:53:31
 +
+

*Thread Reply:* Hi @John Thomas, +Would creation of Sources and Datasets have an equivalent in the OpenLineage specification ? +Sofar I only see the Inputs and Outputs in the Run Event spec.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2022-04-06 11:31:10
 +
+

*Thread Reply:* Inputs and outputs in the OL spec are Datasets in the old MZ spec, so they're equivalent

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-04-05 14:24:50
 +
+

Hey Guys,

+ +

The BaseExtractor is working fine with operators that are derived from Airflow BaseOperator. However for operators derived from LivyOperator the BaseExtractor does not seem to work. Is there a fix for this? We use livyoperator to run sparkjobs

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2022-04-05 15:16:34
 +
+

*Thread Reply:* Hi Marco - it looks like LivyOperator itself does derive from BaseOperator, have you seen any other errors around this problem?

+ +

@Maciej Obuchowski might be more help here

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-04-05 15:21:03
 +
+

*Thread Reply:* It is the operators that inherit from LivyOperator. It doesn't find the parameters like sql, connection etc

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-04-05 15:25:42
 +
+

*Thread Reply:* My guess is that operators that inherit from other operators (not baseoperator) will have the same problem

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2022-04-05 15:32:13
 +
+

*Thread Reply:* interesting! I'm not sure about that. I can look into it if I have time, but Maciej is definitely the person who would know the most.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-04-06 15:49:48
 +
+

*Thread Reply:* @Marco Diaz I wonder - perhaps it would be better to instrument spark with OpenLineage. It doesn’t seem that Airflow will know much about what’s happening underneath here. Have you looked into openlineage-spark?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-04-06 15:51:57
 +
+

*Thread Reply:* I have not tried that library yet. I need to see how it implement because we have several spark custom operators that use livy

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-04-06 15:52:59
 +
+

*Thread Reply:* Do you have any examples?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-04-06 15:54:01
 +
+

*Thread Reply:* there is a good blog post from @Michael Collado: https://openlineage.io/blog/openlineage-spark/

+
+
openlineage.io
+ + + +
Tracing Data Lineage with OpenLineage and Apache Spark
+
+ Spark ushered in a brand new age of data democratization... and left us with a mess of hidden dependencies, stale datasets, and failed jobs. +
+ + + + + + +
Original URL: https://openlineage.io/blog/openlineage-spark/
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-04-06 15:54:37
 +
+

*Thread Reply:* and the doc page here has a good overview: +https://openlineage.io/integration/apache-spark/

+
+
openlineage.io
+ + + +
Apache Spark
+
+ OpenLineage can automatically track lineage of jobs and datasets across Spark jobs. +
+ + + + + + +
Original URL: https://openlineage.io/integration/apache-spark/
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-04-06 16:38:15
 +
+

*Thread Reply:* is this all we need to pass? +spark-submit --conf "spark.extraListeners=io.openlineage.spark.agent.OpenLineageSparkListener" \ + --packages "io.openlineage:openlineage_spark:0.2.+" \ + --conf "spark.openlineage.host=http://&lt;your_ol_endpoint&gt;" \ + --conf "spark.openlineage.namespace=my_job_namespace" \ + --class com.mycompany.MySparkApp my_application.jar

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-04-06 16:38:49
 +
+

*Thread Reply:* If so, yes our operators have a way to pass configurations to spark and we may be able to implement it.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2022-04-06 16:41:27
 +
+

*Thread Reply:* Looks right to me

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-04-06 16:42:03
 +
+

*Thread Reply:* Will give it a try

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-04-06 16:42:50
 +
+

*Thread Reply:* Do we have to install the library on the spark side or the airflow side?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-04-06 16:42:58
 +
+

*Thread Reply:* I assume is the spark side

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2022-04-06 16:44:25
 +
+

*Thread Reply:* The —packages argument tells spark where to get the jar (you'll want to upgrade to 0.6.1)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-04-06 16:44:54
 +
+

*Thread Reply:* sounds good

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Varun Singh + (varuntestaz@outlook.com) +
+
2022-04-06 00:04:14
 +
+

Hi, I saw there was some work done for integrating OpenLineage with Azure Purview + + + +

+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-04-06 04:54:27
 +
+

*Thread Reply:* @Will Johnson

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-04-07 12:43:27
 +
+

*Thread Reply:* Hey @Varun Singh! We are building a github repository that deploys a few resources that will support a limited number of Azure data sources being pushed into Azure Purview. You can expect a public release near the end of the month! Feel free to direct message me if you'd like more details!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-04-06 15:05:39
 +
+

The next OpenLineage Technical Steering Committee meeting is Wednesday, April 13! Meetings are on the second Wednesday of each month from 9:00 to 10:00am PT. +Join us on Zoom: https://astronomer.zoom.us/j/87156607114?pwd=a3B0K210dnRaQmdkaFdGMytBREZEQT09 +All are welcome. +Agenda: +• OpenLineage 0.6.2 release overview +• Airflow integration update +• Dagster integration retrospective +• Open discussion +Notes: https://tinyurl.com/openlineagetsc

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
slackbot + +
+
2022-04-06 21:40:16
 +
+

This message was deleted.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-04-07 01:00:43
 +
+

*Thread Reply:* Are both airflow2 and Marquez installed locally on your computer?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jorge Reyes (Zenta Group) + (jorge.reyes@zentagroup.com) +
+
2022-04-07 09:04:19
 +
+

*Thread Reply:* yes Marco

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-04-07 15:00:18
 +
+

*Thread Reply:* can you open marquez on +<http://localhost:3000>

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-04-07 15:00:40
 +
+

*Thread Reply:* and get a response from +<http://localhost:5000/api/v1/namespaces>

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jorge Reyes (Zenta Group) + (jorge.reyes@zentagroup.com) +
+
2022-04-07 15:26:41
 +
+

*Thread Reply:* yes , i used this guide https://openlineage.io/getting-started and execute un post to marquez correctly

+
+
openlineage.io
+ + + +
Getting Started
+
+ Data lineage is the foundation for a new generation of powerful, context-aware data tools and best practices. OpenLineage enables consistent collection of lineage metadata, creating a deeper understanding of how data is produced and used. +
+ + + + + + +
Original URL: https://openlineage.io/getting-started
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-04-07 22:17:34
 +
+

*Thread Reply:* In theory you should receive events in jobs under airflow namespace

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tien Nguyen + (tiennguyenhotel97@gmail.com) +
+
2022-04-07 14:18:05
 +
+

Hi Everyone, Can someone please help me to debug this error ? Thank you very much all

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2022-04-07 14:59:06
 +
+

*Thread Reply:* It looks like you need to add a payment method to your DBT account

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tyler Farris + (tyler@kickstand.work) +
+
2022-04-11 12:46:41
 +
+

Hello. Does Airflow's TaskFlow API work with OpenLineage?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-04-11 12:50:48
 +
+

*Thread Reply:* It does, but admittedly not very well. It can't recognize what you're doing inside your tasks. The good news is that we're working on it and long term everything should work well.

+ + + +
+ 👍 Howard Yoo +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tyler Farris + (tyler@kickstand.work) +
+
2022-04-11 12:58:28
 +
+

*Thread Reply:* Thanks for the quick reply Maciej.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
sandeep + (sandeepgame07@gmail.com) +
+
2022-04-12 09:56:44
 +
+

Hi all, watched few of your demos with airflow(astronomer) recently, really liked them. +Thanks for doing those

+ +

Questions:

+ +
  1. Are there plans to have a hive listener similar to the open-lineage spark integration ?
  2. If not will the sql parser work with the HiveQL ?
  3. Maybe one for presto too ?
  4. Will the run version and dataset version come out of the box or do we need to define some facets ?
  5. I read the blog on facets, is there a tutorial on how to create a sample facet ? +Background: +We have hive, spark jobs and big query tasks running from airflow in GCP Dataproc
    6. +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2022-04-12 13:56:53
 +
+

*Thread Reply:* Hi Sandeep,

+ +

1&3: We don't currently have Hive or Presto on the roadmap! The best way to start the conversation around them would be to create a proposal in the OpenLineage repo, outlining your thoughts on implementation and benefits.

+ +

2: I'm not familiar enough with HiveQL, but you can read about the new SQL parser we're implementing here

+ +

  1. you can see the Standard Facets here - Dataset Version is included out of the box, but Run Version would have to be defined.

  2. the best place to start looking into making facets is the Spec doc here. We don't have a dedicated tutorial, but if you have more specific questions please feel free to reach out again on slack

    3. +
+ + + +
+ 👍 sandeep +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
sandeep + (sandeepgame07@gmail.com) +
+
2022-04-12 15:39:23
 +
+

*Thread Reply:* Thank you John +The standard facets links to the github issues currently

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2022-04-12 15:40:33
 +
+

*Thread Reply:* ah here -https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.md#standard-facets

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
sandeep + (sandeepgame07@gmail.com) +
+
2022-04-12 15:41:01
 +
+

*Thread Reply:* Will check it out thank you

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-04-12 10:37:58
 +
+

Reminder: this month’s OpenLineage TSC meeting is tomorrow, 4/13, at 9 am PT! https://openlineage.slack.com/archives/C01CK9T7HKR/p1649271939878419

+
+ + +
+ + + } + + Michael Robinson + (https://openlineage.slack.com/team/U02LXF3HUN7) +
+ + + +
+
+ <p>The next OpenLineage Technical Steering Committee meeting is Wednesday, April 13! Meetings are on the second Wednesday of each month from 9:00 to 10:00am PT. +Join us on Zoom: <a href="https://astronomer.zoom.us/j/87156607114?pwd=a3B0K210dnRaQmdkaFdGMytBREZEQT09">https://astronomer.zoom.us/j/87156607114?pwd=a3B0K210dnRaQmdkaFdGMytBREZEQT09</a> +All are welcome. +Agenda: +• OpenLineage 0.6.2 release overview +• Airflow integration update +• Dagster integration retrospective +• Open discussion +Notes: <a href="https://tinyurl.com/openlineagetsc">https://tinyurl.com/openlineagetsc</a></p> +
+ + + + + + +
Original URL: https://openlineage.slack.com/archives/C01CK9T7HKR/p1649271939878419
+ + +
+ + Slack Conversation +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
sandeep + (sandeepgame07@gmail.com) +
+
2022-04-12 15:43:29
 +
+

I setup the open-lineage spark integration for spark(dataproc) tasks from airflow. It’s able to post data to the marquez end point and I see the job information in Marquez UI.

+ +

I don’t see any dataset information in it, I see just the jobs ? Is there some setup I need to do or something else I need to configure ?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2022-04-12 16:08:30
 +
+

*Thread Reply:* is there anything in your marquez-api logs that might indicate issues?

+ +

What guide did you follow to setup the spark integration?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
sandeep + (sandeepgame07@gmail.com) +
+
2022-04-12 16:10:07
 +
+

*Thread Reply:* Followed this guide https://openlineage.io/integration/apache-spark/ and used the spark-defaults.conf approach

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
sandeep + (sandeepgame07@gmail.com) +
+
2022-04-12 16:11:04
 +
+

*Thread Reply:* The logs from dataproc side show no errors, let me check from the marquez api side +To confirm, we should be able to see the datasets from the marquez UI with the spark integration right ?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2022-04-12 16:11:50
 +
+

*Thread Reply:* I'm not super familiar with the spark integration, since I work more with airflow - I'd start with looking through the readme for the spark integration here

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
sandeep + (sandeepgame07@gmail.com) +
+
2022-04-12 16:14:44
 +
+

*Thread Reply:* Hmm, the readme says it aims to generate the input and output datasets

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-04-12 16:40:38
 +
+

*Thread Reply:* Are you looking at the same namespace?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
sandeep + (sandeepgame07@gmail.com) +
+
2022-04-12 16:40:51
 +
+

*Thread Reply:* Yes, the same one where I can see the job

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
sandeep + (sandeepgame07@gmail.com) +
+
2022-04-12 16:54:49
 +
+

*Thread Reply:* Tailing the API logs and rerunning the spark job now to hopefully catch errors if any, will ping back here

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
sandeep + (sandeepgame07@gmail.com) +
+
2022-04-12 17:01:10
 +
+

*Thread Reply:* Don’t see any failures in the logs, any suggestions on how to debug this ?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2022-04-12 17:08:24
 +
+

*Thread Reply:* I'd next set up a basic spark notebook and see if you can't get it to send dataset information on something simple in order to check if it's a setup issue or a problem with your spark job specifically

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
sandeep + (sandeepgame07@gmail.com) +
+
2022-04-12 17:14:43
 +
+

*Thread Reply:* ok, that sounds good, will try that

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
sandeep + (sandeepgame07@gmail.com) +
+
2022-04-12 17:16:06
 +
+

*Thread Reply:* before that, I see that spark-lineage integration posts lineage to the api +https://marquezproject.github.io/marquez/openapi.html#tag/Lineage/paths/~1lineage/post +We don’t seem to add a DataSet in this, does marquez internally create this “dataset” based on Output and fields ?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2022-04-12 17:16:34
 +
+

*Thread Reply:* yeah, you should be seeing "input" and "output" in the runEvents - that's where datasets come from

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2022-04-12 17:17:00
 +
+

*Thread Reply:* I'm not sure if it's a problem with your specific spark job or with the integration itself, however

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
sandeep + (sandeepgame07@gmail.com) +
+
2022-04-12 17:19:16
 +
+

*Thread Reply:* By runEvents, do you mean a job Object or lineage Object ? +The integration seems to be only POSTing lineage objects

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2022-04-12 17:20:34
 +
+

*Thread Reply:* yep, a runEvent is body that gets POSTed to the /lineage endpoint:

+ +

https://openlineage.io/docs/openapi/

+ + + +
+ 👍 sandeep +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-04-12 17:41:01
 +
+

*Thread Reply:* > Yes, the same one where I can see the job +I think you should look at other namespace, which name depends on what systems you're actually using

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
sandeep + (sandeepgame07@gmail.com) +
+
2022-04-12 17:48:24
 +
+

*Thread Reply:* Shouldn’t the dataset would be created in the same namespace we define in the spark properties?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
sandeep + (sandeepgame07@gmail.com) +
+
2022-04-15 10:19:06
 +
+

*Thread Reply:* I found few datasets in the table location, I ran it in a similar (hive metastore, gcs, sparksql and scala spark jobs) setup to the one mentioned in this post https://openlineage.slack.com/archives/C01CK9T7HKR/p1649967405659519

+
+ + +
+ + + } + + Will Johnson + (https://openlineage.slack.com/team/U02H4FF5M36) +
+ + + +
+
+ <p>Before I create an issue around it, maybe I'm just not seeing it in Databricks. In the Spark Integration, does OpenLineage report Hive Metastore tables or it ONLY reports the file path?</p> + +<p>For example, if I have a Hive table called default.myTable stored at LOCATION /usr/hive/warehouse/default/mytable.</p> + +<p>For a query that reads a CSV file and inserts into default.myTable, would I see an output of default.myTable or /usr/hive/warehoues/default/mytable?</p> + +<p>We want to include a link between the physical path and the hive metastore table but it seems that OpenLineage (at least on Databricks) only reports the physical path with the table name showing up in the catalog but not as a facet.</p> +
+ + + + + + +
Original URL: https://openlineage.slack.com/archives/C01CK9T7HKR/p1649967405659519
+ + +
+ + Thread in Slack Conversation +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
sandeep + (sandeepgame07@gmail.com) +
+
2022-04-12 15:49:46
 +
+

Is this the correct place for this Q or should I reach out to Marquez slack ? +I followed this post https://openlineage.io/integration/apache-spark/

+
+
openlineage.io
+ + + +
Apache Spark
+
+ OpenLineage can automatically track lineage of jobs and datasets across Spark jobs. +
+ + + + + + +
Original URL: https://openlineage.io/integration/apache-spark/
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-04-14 16:16:45
 +
+

Before I create an issue around it, maybe I'm just not seeing it in Databricks. In the Spark Integration, does OpenLineage report Hive Metastore tables or it ONLY reports the file path?

+ +

For example, if I have a Hive table called default.myTable stored at LOCATION /usr/hive/warehouse/default/mytable.

+ +

For a query that reads a CSV file and inserts into default.myTable, would I see an output of default.myTable or /usr/hive/warehoues/default/mytable?

+ +

We want to include a link between the physical path and the hive metastore table but it seems that OpenLineage (at least on Databricks) only reports the physical path with the table name showing up in the catalog but not as a facet.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
sandeep + (sandeepgame07@gmail.com) +
+
2022-04-15 10:17:55
 +
+

*Thread Reply:* This was my experience as well, I was under the impression we would see the table as a dataset. +Looking forward to understanding the expected behavior

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-04-15 10:39:34
 +
+

*Thread Reply:* relevant: https://github.com/OpenLineage/OpenLineage/issues/435

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/issues/435|#435 [INTEGRATION][Spark] provide physical location of a table facet in case we're using logical table name>
+
+ <p>When dealing with a logical Hive table, we've decided to use logical Hive table name as a dataset name. In some other cases, some other job - not necessarily Spark one - might deal with this dataset by directly knowing it's physical location. In this case we'll have the same dataset known by two different names in different namespaces.</p> + +<p>To make sure those datasets can be "linked" by OpenLineage consumers, when available, we should provide additional facet (<code>PhysicalLocationFacet</code>?) that indicates physical location at a storage system (like <code>hdfs</code>, <code>s3</code> etc) or other underlying storage mechanism.</p> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+ 👍 Howard Yoo +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-04-15 12:36:08
 +
+

*Thread Reply:* Ah! Thank you both for confirming this! And it's great to see the proposal, Maciej!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sharanya Santhanam + (santhanamsharanya@gmail.com) +
+
2022-06-10 12:37:41
 +
+

*Thread Reply:* Is there a timeline around when we can expect this fix ?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-06-10 12:46:47
 +
+

*Thread Reply:* Not a simple fix, but I guess we'll start working on this relatively soon.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sharanya Santhanam + (santhanamsharanya@gmail.com) +
+
2022-06-10 13:10:31
 +
+

*Thread Reply:* I see, thanks for the update ! We are very much interested in this feature.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-04-15 15:42:22
 +
+

@channel A significant number of us have a conflict with the current TSC meeting day/time, so, unfortunately, we need to reschedule the meeting. When you have a moment, please share your availability here: https://doodle.com/meeting/participate/id/ejRnMlPe. Thanks in advance for your input!

+
+
doodle.com
+ + + +
Doodle
+
+ Doodle is the simplest way to schedule meetings with clients, colleagues, or friends. Find the best time for one-to-ones and team meetings with our user-friendly calendar tool. Get started today! +
+ + + + + + +
Original URL: https://doodle.com/meeting/participate/id/ejRnMlPe
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Arturo + (ggrmos@gmail.com) +
+
2022-04-19 13:35:23
 +
+

Hello everyone, I'm learning Openlineage, I finally achieved the connection between Airflow 2+ and Openlineage+Marquez. The issue is that I don't see nothing on Marquez. Do I need to modify current airflow operators?

+ +
+
+ + + + + + +
+
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-04-19 13:40:54
 +
+

*Thread Reply:* You probably need to change dataset from default

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Arturo + (ggrmos@gmail.com) +
+
2022-04-19 13:47:04
 +
+

*Thread Reply:* I click it on everything 😕 I manually (joining to the pod and send curl to the marquez local endpoint) created a namespaces to check if there is a network issue I was ok, I created a namespaces called: data-dev . The airflow is mounted over k8s using helm chart. +``` config: + AIRFLOWWEBSERVERBASEURL: "http://airflow.dev.test.io" + PYTHONPATH: "/opt/airflow/dags/repo/config" + AIRFLOWAPIAUTHBACKEND: "airflow.api.auth.backend.basicauth" + AIRFLOWCOREPLUGINSFOLDER: "/opt/airflow/dags/repo/plugins" + AIRFLOWLINEAGEBACKEND: "openlineage.lineage_backend.OpenLineageBackend"

+ +

. +. +. +.

+ +

extraEnv: + - name: OPENLINEAGEURL + value: http://marquez-dev.data-dev.svc.cluster.local + - name: OPENLINEAGENAMESPACE + value: data-dev```

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-04-19 15:16:47
 +
+

*Thread Reply:* I think answer is somewhere in airflow logs 🙂 +For some reason, OpenLineage events aren't send to Marquez.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Arturo + (ggrmos@gmail.com) +
+
2022-04-20 11:08:09
 +
+

*Thread Reply:* Thanks, finally was my error .. I created a dummy dag to see if maybe it's an issue over the dag and now I can see something over Marquez

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mirko Raca + (racamirko@gmail.com) +
+
2022-04-20 08:15:32
 +
+

One really novice question - there doesn't seem to be a way of deleting lineage elements (any of them)? While I can imagine that in production system we want to keep history, it's not practical while testing/developing. I'm using throw-away namespaces to step around the issue. Is there a better way, or alternatively - did I miss an API somewhere?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-04-20 08:20:35
 +
+

*Thread Reply:* That's more of a Marquez question 🙂 +We have a long-standing issue to add that API https://github.com/MarquezProject/marquez/issues/1736

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mirko Raca + (racamirko@gmail.com) +
+
2022-04-20 09:32:19
 +
+

*Thread Reply:* I see it already got skipped for 2 releases, and my only conclusion is that people using Marquez don't make mistakes - ergo, API not needed 🙂 Lets see if I can stick around the project long enough to offer a bit of help, now I just need to showcase it and get interest in my org.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Dan Mahoney + (dan.mahoney@sphericalanalytics.io) +
+
2022-04-20 10:08:33
 +
+

Good day all. I’m trying out the openlineage-dagster plugin +• I’ve got dagit, dagster-daemon and marquez running locally +• The openlineagesensor is recognized in dagit and the daemon. +But, when I run a job, I see the following message in the daemon’s shell: +Sensor openlineage_sensor skipped: Last cursor: {"last_storage_id": 9, "running_pipelines": {"97e2efdf-9499-4ffd-8528-d7fea5b9362c": {"running_steps": {}, "repository_name": "hello_cereal_repository"}}} +I’ve attached my repos.py and serialjob.py. +Any thoughts?

+ +
+
serial_job.py
+ + + + + + +
+
+
repos.py
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
David + (drobin1437@gmail.com) +
+
2022-04-20 10:40:03
 +
+

Hi All, +I am walking through the curl examples on this page and have a question on the first curl example: +https://openlineage.io/getting-started/ +The curl command completes, and I can see the input file and job in the namespace, but the lineage graph does not show the input file connected as an input to the job. This only seems to happen after the job is marked complete.

+ +

Is there a way to have a running job show connections to its input files in the lineage? +Thanks!

+
+
openlineage.io
+ + + +
Getting Started
+
+ Data lineage is the foundation for a new generation of powerful, context-aware data tools and best practices. OpenLineage enables consistent collection of lineage metadata, creating a deeper understanding of how data is produced and used. +
+ + + + + + +
Original URL: https://openlineage.io/getting-started/
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
raghanag + (raghanag@gmail.com) +
+
2022-04-20 18:06:29
 +
+

Hi Team, we are using spark as a service, and we are planning to integrate open lineage spark listener and looking at the below params that we need to pass, we don't know the name of the spark cluster, is the spark.openlineage.namespace conf param mandatory? +spark-submit --conf "spark.extraListeners=io.openlineage.spark.agent.OpenLineageSparkListener" \ + --packages "io.openlineage:openlineage_spark:0.2.+" \ + --conf "spark.openlineage.host=http://&lt;your_ol_endpoint&gt;" \ + --conf "spark.openlineage.namespace=my_job_namespace" \ + --class com.mycompany.MySparkApp my_application.jar

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-04-20 18:11:19
 +
+

*Thread Reply:* Namespace is defined by you, it does not have to be name of the spark cluster.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-04-20 18:11:42
 +
+

*Thread Reply:* And I definitely recommend to use newer version than 0.2.+ 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
raghanag + (raghanag@gmail.com) +
+
2022-04-20 18:13:32
 +
+

*Thread Reply:* oh i see that someone mentioned that it has to be replaced with name of the spark clsuter

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
raghanag + (raghanag@gmail.com) +
+
2022-04-20 18:13:57
 +
+

*Thread Reply:* https://openlineage.slack.com/archives/C01CK9T7HKR/p1634089656188400?thread_ts=1634085740.187700&cid=C01CK9T7HKR

+
+ + +
+ + + } + + Julien Le Dem + (https://openlineage.slack.com/team/U01DCLP0GU9) +
+ + + +
+
+ <p><code>SparkSession.builder() + .config("spark.jars.packages", "io.openlineage:openlineage_spark:0.2.+") + .config("spark.extraListeners", "io.openlineage.spark.agent.OpenLineageSparkListener") + .config("spark.openlineage.host", "&lt;https://localhost&gt;") + .config("spark.openlineage.apiKey", "your api key") + .config("spark.openlineage.namespace", "&amp;lt;NAMESPACE_NAME&amp;gt;") // Replace with the name of your Spark cluster. + .getOrCreate()</code></p> +
+ + + + + + +
Original URL: https://openlineage.slack.com/archives/C01CK9T7HKR/p1634089656188400?thread_ts=1634085740.187700&amp;cid=C01CK9T7HKR
+ + +
+ + Thread in Slack Conversation +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
raghanag + (raghanag@gmail.com) +
+
2022-04-20 18:19:19
 +
+

*Thread Reply:* @Maciej Obuchowski may i know if i can add the --packages "io.openlineage:openlineage_spark:0.2.+" as part of the spark jar file, that meant as part of the pom.xml

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-04-21 03:54:25
 +
+

*Thread Reply:* I think it needs to run on the driver

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mirko Raca + (racamirko@gmail.com) +
+
2022-04-21 05:53:34
 +
+

Hello, +when looking through Marquez API it seems that most individual-element creation APIs are marked as deprecated and are going to be removed by 0.25, with a point of switching to open lineage. That makes POST to /api/v1/lineage the only creation point of elements, but OpenLineage API is very limited in attributes that can be passed.

+ +

Is that intended to stay that way? One practical question/example: how do we create a job of type STREAMING, when OL API only allows to pass name, namespace and facets. Do we now move all properties into facets?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-04-21 07:16:44
 +
+

*Thread Reply:* > OpenLineage API is very limited in attributes that can be passed. +Can you specify where do you think it's limited? The way to solve that problems would be to evolve OpenLineage.

+ +

> One practical question/example: how do we create a job of type STREAMING, +So, here I think the question is more how streaming jobs differ from batch jobs. One obvious difference is that output of the job is continuous (in practice, probably "microbatched" or commited on checkpoint). However, deprecated Marquez API didn't give us tools to properly indicate that. On the contrary, OpenLineage with different event types allows us to properly do that. +> Do we now move all properties into facets? +Basically, yes. Marquez should handle specific facets. For example, https://github.com/MarquezProject/marquez/pull/1847

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mirko Raca + (racamirko@gmail.com) +
+
2022-04-21 07:23:11
 +
+

*Thread Reply:* Hey Maciej

+ +

first off - thanks for being active on the channel!

+ +

> So, here I think the question is more how streaming jobs differ from batch jobs +Not really. I just gave an example of how would you express a specific job type creation which can be done with https://marquezproject.github.io/marquez/openapi.html#tag/Jobs/paths/~1namespaces~1{namespace}~1jobs~1{job}/put|/api/v1/namespaces/.../jobs/... , by passing the type field which is required. In the call to /api/v1/lineage the job field offers just to specify (namespace, name), but no other attributes.

+ +

> However, deprecated Marquez API didn't give us tools to properly indicate that. On the contrary, OpenLineage with different event types allows us to properly do that. +I have the feeling I'm still missing some key concepts on how OpenLineage is designed. I think I went over the API and documentation, but trying to use just OpenLineage failed to reproduce mildly complex chain-of-job scenarios, and when I took a look how Marquez seed demo is doing it - it was heavily based on deprecated API. So, I'm kinda lost on how to use OpenLineage.

+ +

I'm looking forward to some open-public meeting, as I don't think asking these long questions on chat really works. 😞 +Any pointers are welcome!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-04-21 07:53:59
 +
+

*Thread Reply:* > I just gave an example of how would you express a specific job type creation +Yes, but you're trying to achieve something by passing this parameter or creating a job in a certain way. We're trying to cover everything in OpenLineage API. Even if we don't have everything, the spec from the beginning is focused to allow emitting custom data by custom facet mechanism.

+ +

> I have the feeling I'm still missing some key concepts on how OpenLineage is designed. +This talk by @Julien Le Dem is a great place to start: https://www.youtube.com/watch?v=HEJFCQLwdtk

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-04-21 11:29:20
 +
+

*Thread Reply:* > Any pointers are welcome! +BTW: OpenLineage is an open standard. Everyone is welcome to contribute and discuss. Every feedback ultimately helps us build better systems.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mirko Raca + (racamirko@gmail.com) +
+
2022-04-22 03:32:48
 +
+

*Thread Reply:* I agree, but for now I'm more likely to be in the I didn't get it category, and not in the brilliant new idea category 🙂

+ +

My temporary goal is to go over the documentation and to write the gaps that confused me (and the solutions) and maybe publish that as an article for wider audience. So far I realized that: +• I don't get the naming convention - it became clearer that it's important with the Naming examples, but more info is needed +• I mis-interpret the namespaces. I was placing datasources and jobs in the same namespace which caused a lot of issues until I started using different ones. Not sure why... So now I'm interpreting namespaces=source as suggested by the naming convention +• JSON schema actually clarified things a lot, but that's not the most reader-friendly of resources, so surely there should be a better one +• I was questioning whether to move away from Marquez completely and go with DataHub, but for my scenario Marquez (with limitations outstanding) is still most suitable +• Marquez for some reason does not tolerate the datetimes if they're missing the 'T' delimiter in the ISO, which caused a lot of trial-and-error because the message is just "JSON parsing failed" +• Marquez doesn't give you (at least by default) meaningful OpenLineage parsing errors, so running examples against it is a very slow learning process

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Karatuğ Ozan BİRCAN + (karatugo@gmail.com) +
+
2022-04-21 10:20:55
 +
+

Hi everyone,

+ +

I'm running the Spark Listener on Databricks. It works fine for the event emit part for a basic Databricks SQL Create Table query. Nevertheless, it throws a NullPointerException exception after sending lineage successfully.

+ +

I tried to debug a bit. Looks like it's thrown at the line: +QueryExecution queryExecution = SQLExecution.getQueryExecution(executionId); +So, does this mean that the listener can't get the query exec from Spark SQL execution?

+ +

Please see the logs in the thread. Thanks.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Karatuğ Ozan BİRCAN + (karatugo@gmail.com) +
+
2022-04-21 10:21:33
 +
+

*Thread Reply:* Driver logs from Databricks:

+ +

```22/04/21 14:05:07 INFO EventEmitter: Lineage completed successfully: ResponseMessage(responseCode=200, body={}, error=null) {"eventType":"COMPLETE",[...], "schemaURL":"https://openlineage.io/spec/1-0-2/OpenLineage.json#/$defs/RunEvent"}

+ +

22/04/21 14:05:07 ERROR AsyncEventQueue: Listener OpenLineageSparkListener threw an exception +java.lang.NullPointerException + at io.openlineage.spark.agent.lifecycle.ContextFactory.createSparkSQLExecutionContext(ContextFactory.java:43) + at io.openlineage.spark.agent.OpenLineageSparkListener.lambda$getSparkSQLExecutionContext$8(OpenLineageSparkListener.java:221) + at java.util.HashMap.computeIfAbsent(HashMap.java:1127) + at java.util.Collections$SynchronizedMap.computeIfAbsent(Collections.java:2674) + at io.openlineage.spark.agent.OpenLineageSparkListener.getSparkSQLExecutionContext(OpenLineageSparkListener.java:220) + at io.openlineage.spark.agent.OpenLineageSparkListener.sparkSQLExecStart(OpenLineageSparkListener.java:143) + at io.openlineage.spark.agent.OpenLineageSparkListener.onOtherEvent(OpenLineageSparkListener.java:135) + at org.apache.spark.scheduler.SparkListenerBus.doPostEvent(SparkListenerBus.scala:102) + at org.apache.spark.scheduler.SparkListenerBus.doPostEvent$(SparkListenerBus.scala:28) + at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37) + at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37) + at org.apache.spark.util.ListenerBus.postToAll(ListenerBus.scala:119) + at org.apache.spark.util.ListenerBus.postToAll$(ListenerBus.scala:103) + at org.apache.spark.scheduler.AsyncEventQueue.super$postToAll(AsyncEventQueue.scala:105) + at org.apache.spark.scheduler.AsyncEventQueue.$anonfun$dispatch$1(AsyncEventQueue.scala:105) + at scala.runtime.java8.JFunction0$mcJ$sp.apply(JFunction0$mcJ$sp.java:23) + at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62) + at org.apache.spark.scheduler.AsyncEventQueue.org$apache$spark$scheduler$AsyncEventQueue$$dispatch(AsyncEventQueue.scala:100) + at org.apache.spark.scheduler.AsyncEventQueue$$anon$2.$anonfun$run$1(AsyncEventQueue.scala:96) + at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1588) + at org.apache.spark.scheduler.AsyncEventQueue$$anon$2.run(AsyncEventQueue.scala:96)```

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-04-21 11:32:37
 +
+

*Thread Reply:* @Karatuğ Ozan BİRCAN are you running on Spark 3.2? If yes, then new release should have fixed your problem: https://github.com/OpenLineage/OpenLineage/issues/609

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Karatuğ Ozan BİRCAN + (karatugo@gmail.com) +
+
2022-04-21 11:33:15
 +
+

*Thread Reply:* Spark 3.1.2 with Scala 2.12

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Karatuğ Ozan BİRCAN + (karatugo@gmail.com) +
+
2022-04-21 11:33:50
 +
+

*Thread Reply:* In fact, I couldn't make it work in Spark 3.2. But I'll test it again. Thanks for the info.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Vinith Krishnan US + (vinithk@nvidia.com) +
+
2022-05-20 16:15:47
 +
+

*Thread Reply:* Has this been resolved? +I am facing the same issue with spark 3.2.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ben + (ben@meridian.sh) +
+
2022-04-21 11:51:33
 +
+

Does anyone have thoughts on the difference between the sourceCode and sql job facets - and whether we’d expect to ever see both on a particular job?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2022-04-21 15:34:24
 +
+

*Thread Reply:* I don't think that the facets are particularly strongly defined, but I would expect that it could be possible to see both on a pythonOperator that's executing SQL queries, depending on how the extractor was written

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ben + (ben@meridian.sh) +
+
2022-04-21 15:34:45
 +
+

*Thread Reply:* ah sure, that makes sense

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Xiaoyong Zhu + (xiaoyzhu@outlook.com) +
+
2022-04-21 15:14:03
 +
+

Just get to know open lineage and it's really a great project! One question for the granularity on Spark + Openlineage - is it possible to track column level lineage (rather than the table lineage that's currently there)? Thanks!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-04-21 16:17:59
 +
+

*Thread Reply:* We're actively working on it - expect it in next OpenLineage release. https://github.com/OpenLineage/OpenLineage/pull/645

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/pull/645|#645 [Spark][Integration] column level lineage>
+
+ +
+ +
+
Labels
+ enhancement, integration/spark +
+ +
+
Milestone
+ <a href="https://github.com/OpenLineage/OpenLineage/milestone/4">0.8.0</a> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Xiaoyong Zhu + (xiaoyzhu@outlook.com) +
+
2022-04-21 16:24:16
 +
+

*Thread Reply:* nice -thanks!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Xiaoyong Zhu + (xiaoyzhu@outlook.com) +
+
2022-04-21 16:25:19
 +
+

*Thread Reply:* Assuming we don't need to do anything except using the next update? Or do you expect that we need to change quite a lot of configs?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-04-21 17:44:46
 +
+

*Thread Reply:* No, it should be automatic.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-04-24 14:37:33
 +
+

Hey, Team - We are starting to get requests for other, non Microsoft data sources (e.g. Teradata) for the Spark Integration. We (I) don't have a lot of bandwidth to fill every request but I DO want to help these people new to OpenLineage get started.

+ +

Has anyone on the team written up a blog post about extending open lineage or is this an area that we could collaborate on for the OpenLineage blog? Alternatively, is it a bad idea to write this down since the internals have changed a few times over the past six months?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mirko Raca + (racamirko@gmail.com) +
+
2022-04-25 03:52:20
 +
+

*Thread Reply:* Hey Will,

+ +

while I would not consider myself in the team, I'm dabbling in OL, hitting walls and learning as I go. If I don't have enough experience to contribute, I'd be happy to at least proof-read and point out things which are not clear from a novice perspective. Let me know!

+ + + +
+ 👍 Will Johnson +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-04-25 13:49:48
 +
+

*Thread Reply:* I'll hold you to that @Mirko Raca 😉

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-04-25 17:18:02
 +
+

*Thread Reply:* I will support! I’ve done a few recent presentations on the internals of OpenLineage that might also be useful - maybe some diagrams can be reused.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-04-25 17:56:44
 +
+

*Thread Reply:* Any chance you have links to those old presentations? Would be great to build off of an existing one and then update for some of the new naming conventions.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-04-25 18:00:26
 +
+

*Thread Reply:* the most recent one was an astronomer webinar

+ +

happy to share the slides with you if you want 👍 here’s a PDF:

+ +
+
+ + + + + + +
+ + +
+ 🙌 Will Johnson +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-04-25 18:00:44
 +
+

*Thread Reply:* the other ones have not been public, unfortunately 😕

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-04-25 18:02:24
 +
+

*Thread Reply:* architecture, object model, run lifecycle, naming conventions == the basics IMO

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-04-26 09:14:42
 +
+

*Thread Reply:* Thank you so much, Ross! This is a great base to work from.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-04-26 14:49:04
 +
+

Your periodical reminder that Github stars are one of those trivial things that make a significant difference for an OS project like ours. Have you starred us yet?

+ +
+
+ + + + + + +
+ + +
+ 👍 Ross Turk +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
raghanag + (raghanag@gmail.com) +
+
2022-04-26 15:02:10
 +
+

Hi All, I have a simple spark job from converting csv to parquet and I am using https://openlineage.io/integration/apache-spark/ to generate lineage events and posting to maquez but I see that both events (START & COMPLETE) have the same event except eventType, i thought we should see outputsarray in the complete event right?

+
+
openlineage.io
+ + + +
Apache Spark
+
+ OpenLineage can automatically track lineage of jobs and datasets across Spark jobs. +
+ + + + + + +
Original URL: https://openlineage.io/integration/apache-spark/
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-04-27 00:36:05
 +
+

*Thread Reply:* For a spark job like that, you'd have at least four events:

+ +
  1. START event - This represents the SparkSQLExecutionStart
  2. START event #2 - This represents a JobStart event
  3. COMPLET event - This represents a JobEnd event
  4. COMPLETE event #2 - This represents a SparkSQLExectionEnd event +For CSV to Parquet, you should be seeing inputs and outputs that match across each event. OpenLineage scans the logical plan and reports back the inputs / outputs / metadata across the different facets for each event BECAUSE each event might give you some different information.
    5. +
+ +

For example, the JobStart event might give you access to properties that weren't there before. The JobEnd event might give you information about how many rows were written.

+ +

Marquez / OpenLineage expects that you collect all of the resulting events and then aggregate the results.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
raghanag + (raghanag@gmail.com) +
+
2022-04-27 21:51:07
 +
+

*Thread Reply:* Hi @Will Johnson good evening. We are seeing an issue while using spark integaration and found that when we provide openlinegae.host property a value like <http://lineage.com/common/marquez> where my marquez api is running I see that the below line is modifying the host to become <http://lineage.com/api/v1/lineage> instead of <http://lineage.com/common/marquez/api/v1/lineage> which is causing the problem +https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/src/main/common/java/io/openlineage/spark/agent/EventEmitter.java#L49 +I see that it has been added 5 months ago and released it as part of 0.4.0, is there anyway that we can fix the line to be like below +this.lineageURI = + new URI( + hostURI.getScheme(), + hostURI.getAuthority(), + hostURI.getPath() + uriPath, + queryParams, + null);

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/src/main/common/java/io/openlineage/spark/agent/EventEmitter.java | EventEmitter.java>
+
+ <pre><code> new URI(hostURI.getScheme(), hostURI.getAuthority(), uriPath, queryParams, null); +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-04-28 14:31:42
 +
+

*Thread Reply:* Can you open up a Github issue for this? I had this same issue and so our implementation always has to feature the /api/v1/lineage. The host config is literally the host. You're specifying a host and path. I'd be happy to see greater flexibility with the api endpoint but the /v1/ is important to know which version of OpenLineage's specification you're communicating with.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Arturo + (ggrmos@gmail.com) +
+
2022-04-27 14:12:38
 +
+

Hi all, guys ... anyone have an example of a custom extractor with different source-destination, I'm trying to build an extractor from a custom operator like mysql_to_s3

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-04-27 15:10:24
 +
+

*Thread Reply:* @Michael Collado made one for a recent webinar:

+ +

https://gist.github.com/collado-mike/d1854958b7b1672f5a494933f80b8b58

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-04-27 15:11:38
 +
+

*Thread Reply:* it's not exactly for an operator that has source-destination, but it shows how to format lineage events for a few different kinds of datasets

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Arturo + (ggrmos@gmail.com) +
+
2022-04-27 15:51:32
 +
+

*Thread Reply:* Thanks! I'm going to take a look

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-04-27 23:04:18
 +
+

A release has been requested by @Howard Yoo and @Ross Turk pending the merging of PR 644. Are there any +1s?

+ + + +
+ 👍 Julien Le Dem, Maciej Obuchowski, Ross Turk, Conor Beverland +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-04-28 17:44:00
 +
+

*Thread Reply:* Thanks for your input. The release is authorized. Look for it tomorrow!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
raghanag + (raghanag@gmail.com) +
+
2022-04-28 14:29:13
 +
+

Hi All, We are seeing the below exception when we integrate the openlineage-spark into our spark job, can anyone share pointers +Exception uncaught: java.lang.NoSuchMethodError: com.fasterxml.jackson.databind.SerializationConfig.hasExplicitTimeZone()Z at openlineage.jackson.datatype.jsr310.ser.InstantSerializerBase.formatValue(InstantSerializerBase.java:144) at openlineage.jackson.datatype.jsr310.ser.InstantSerializerBase.serialize(InstantSerializerBase.java:103) at openlineage.jackson.datatype.jsr310.ser.ZonedDateTimeSerializer.serialize(ZonedDateTimeSerializer.java:79) at openlineage.jackson.datatype.jsr310.ser.ZonedDateTimeSerializer.serialize(ZonedDateTimeSerializer.java:13) at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:727) at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:719) at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:155) at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider._serialize(DefaultSerializerProvider.java:480) at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:319) at com.fasterxml.jackson.databind.ObjectMapper._configAndWriteValue(ObjectMapper.java:3906) at com.fasterxml.jackson.databind.ObjectMapper.writeValueAsString(ObjectMapper.java:3220) at io.openlineage.spark.agent.client.OpenLineageClient.executeAsync(OpenLineageClient.java:123) at io.openlineage.spark.agent.client.OpenLineageClient.executeSync(OpenLineageClient.java:85) at <a href="http://io.openlineage.spark.agent.client.OpenLineageClient.post">io.openlineage.spark.agent.client.OpenLineageClient.post</a>(OpenLineageClient.java:80) at <a href="http://io.openlineage.spark.agent.client.OpenLineageClient.post">io.openlineage.spark.agent.client.OpenLineageClient.post</a>(OpenLineageClient.java:75) at <a href="http://io.openlineage.spark.agent.client.OpenLineageClient.post">io.openlineage.spark.agent.client.OpenLineageClient.post</a>(OpenLineageClient.java:70) at io.openlineage.spark.agent.EventEmitter.emit(EventEmitter.java:67) at io.openlineage.spark.agent.lifecycle.SparkSQLExecutionContext.start(SparkSQLExecutionContext.java:69) at io.openlineage.spark.agent.OpenLineageSparkListener.lambda$sparkSQLExecStart$0(OpenLineageSparkListener.java:90) at java.util.Optional.ifPresent(Optional.java:159) at io.openlineage.spark.agent.OpenLineageSparkListener.sparkSQLExecStart(OpenLineageSparkListener.java:90) at io.openlineage.spark.agent.OpenLineageSparkListener.onOtherEvent(OpenLineageSparkListener.java:81) at org.apache.spark.scheduler.SparkListenerBus$class.doPostEvent(SparkListenerBus.scala:80) at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37) at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37) at org.apache.spark.util.ListenerBus$class.postToAll(ListenerBus.scala:91) at <a href="http://org.apache.spark.scheduler.AsyncEventQueue.org">org.apache.spark.scheduler.AsyncEventQueue.org</a>$apache$spark$scheduler$AsyncEventQueue$$super$postToAll(AsyncEventQueue.scala:92) at org.apache.spark.scheduler.AsyncEventQueue$$anonfun$org$apache$spark$scheduler$AsyncEventQueue$$dispatch$1.apply$mcJ$sp(AsyncEventQueue.scala:92) at org.apache.spark.scheduler.AsyncEventQueue$$anonfun$org$apache$spark$scheduler$AsyncEventQueue$$dispatch$1.apply(AsyncEventQueue.scala:87) at org.apache.spark.scheduler.AsyncEventQueue$$anonfun$org$apache$spark$scheduler$AsyncEventQueue$$dispatch$1.apply(AsyncEventQueue.scala:87) at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58) at <a href="http://org.apache.spark.scheduler.AsyncEventQueue.org">org.apache.spark.scheduler.AsyncEventQueue.org</a>$apache$spark$scheduler$AsyncEventQueue$$dispatch(AsyncEventQueue.scala:87) at org.apache.spark.scheduler.AsyncEventQueue$$anon$1$$anonfun$run$1.apply$mcV$sp(AsyncEventQueue.scala:83) at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1302) at org.apache.spark.scheduler.AsyncEventQueue$$anon$1.run(AsyncEventQueue.scala:82)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2022-04-28 14:41:10
 +
+

*Thread Reply:* What's the spark job that's running - this looks similar to an error that can happen when jobs have a very short lifecycle

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
raghanag + (raghanag@gmail.com) +
+
2022-04-28 14:47:27
 +
+

*Thread Reply:* nothing in spark job, its just a simple csv to parquet conversion file

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2022-04-28 14:48:50
 +
+

*Thread Reply:* ah yeah that's probably it - when the job is finished before the Openlineage integration can poll it for information this error is thrown. Since the job is very quick it creates a race condition

+ + + +
+ :gratitude_thank_you: raghanag +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
raghanag + (raghanag@gmail.com) +
+
2022-05-03 17:16:39
 +
+

*Thread Reply:* @John Thomas may i know how to solve this kind of issue?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2022-05-03 17:20:11
 +
+

*Thread Reply:* This is probably an issue with the integration - for now you can either open an issue, or see if you're still getting a subset of events and take it as is. I'm not sure what you could do on your end aside from adding a sleep call or similar

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
raghanag + (raghanag@gmail.com) +
+
2022-05-03 17:21:17
 +
+

*Thread Reply:* https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/src/main/common/java/io/openlineage/spark/agent/OpenLineageSparkListener.java#L151 you meant if we add a sleep in this method this will solve this

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/src/main/common/java/io/openlineage/spark/agent/OpenLineageSparkListener.java | OpenLineageSparkListener.java>
+
+ <pre><code> public void onJobEnd(SparkListenerJobEnd jobEnd) { +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2022-05-03 18:44:43
 +
+

*Thread Reply:* oh no I meant making sure your jobs don't close too quickly

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
raghanag + (raghanag@gmail.com) +
+
2022-05-06 00:14:15
 +
+

*Thread Reply:* Hi @John Thomas we figured out the error that it is indeed causing with conflicted versions and with shadowJar and shading, we are not seeing it anymore.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-04-29 18:40:41
 +
+

@channel The latest release (0.8.1) of OpenLineage is now available, featuring a new TaskInstance listener API for Airflow 2.3+, an HTTP client in the openlineage-java library for emitting run events, support for HiveTableRelation as an input source in the Spark integration, a new SQL parser used by multiple integrations, and bug fixes. For more info, visit https://github.com/OpenLineage/OpenLineage/releases/tag/0.8.1

+ + + +
+ 🚀 Willy Lulciuc, John Thomas, Minkyu Park, Ross Turk, Marco Diaz, Conor Beverland, Kevin Mellott, Howard Yoo, Peter Hicks, Maciej Obuchowski, Mario Measic +
+ +
+ 🙌 Francis McGregor-Macdonald, Ross Turk, Marco Diaz, Peter Hicks +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2022-04-29 18:41:37
 +
+

*Thread Reply:* Amazing work on the new sql parser @Maciej Obuchowski 💯 :firstplacemedal:

+ + + +
+ 👍 Ross Turk, Howard Yoo, Peter Hicks +
+ +
+ 🙌 Ross Turk, Howard Yoo, Peter Hicks, Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-04-30 07:54:48
 +
+

The May meeting of the TSC will be postponed because most of the TSC will be attending the Astronomer Spring Summit the week of May 9th. Details to follow along with a new meeting day/time for the meeting going forward (thanks to all who responded to the poll!).

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Hubert Dulay + (hubert.dulay@gmail.com) +
+
2022-05-01 09:25:23
 +
+

Are there examples of using openlineage with streaming data pipelines? Thanks

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mirko Raca + (racamirko@gmail.com) +
+
2022-05-03 04:12:09
 +
+

*Thread Reply:* Hi @Hubert Dulay,

+ +

while I'm not an expert, I can offer the following: +• Marquez has had the but what I got here - that API is not encouraged +• I personally don't find the run->job metaphor to work nicely with streaming transformation, but I'm using that in my current setup (until someone points me in a better direction 😉 ) +• I register each change of the stream processing as a new "run", which ends immediately - so duration information is lost, but current set of parameters is recorded. It's not pretty, I know. +Maybe stream processing is a scenario to be re-evaluated in OL meetings, or at least clarified?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Hubert Dulay + (hubert.dulay@gmail.com) +
+
2022-05-03 21:19:06
 +
+

*Thread Reply:* Thanks for the details

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kostikey Mustakas + (kostikey.mustakas@gmail.com) +
+
2022-05-02 09:32:23
 +
+

Hey OL! My company is in the process of migrating off of Palantir and into Databricks/Azure. There are a couple of business units not wanting to budge due to the built-in data lineage and code reference features Palantir has. I am tasked with researching an alternative data lineage solution and I quickly came across OL. I love what I have read and seen demos of so far and want to do a POC for my org of its capabilities. I was able to set up the Marquez server on a VM and get it talking to Databricks. I also have the iniit script installed on the cluster and I can see from the log4j logs it’s communicating fine (I think). However, I am embarrassed to admit I can’t figure out how the instrumentation works for the databricks notebooks. I ran a simple notebook that loads data, runs a simple transform, and saves the output somewhere but I don’t see any entries in my namespace I configured. I am sure I missed something very obvious somewhere, but are there examples of how to get a simple example into Marquez from databricks? Thanks so much for any guidance you can give!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2022-05-02 13:26:52
 +
+

*Thread Reply:* Hi Kostikey - this blog has an example with Spark and jupyter, which might be a good place to start!

+
+
openlineage.io
+ + + +
Tracing Data Lineage with OpenLineage and Apache Spark
+
+ Spark ushered in a brand new age of data democratization... and left us with a mess of hidden dependencies, stale datasets, and failed jobs. +
+ + + + + + +
Original URL: https://openlineage.io/blog/openlineage-spark/
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kostikey Mustakas + (kostikey.mustakas@gmail.com) +
+
2022-05-02 14:58:29
 +
+

*Thread Reply:* Hi @John Thomas, thanks for the reply. I think I am close but my cluster is unable to talk to the marquez server. After looking at log4j I see the following rows:

+ +

22/05/02 18:43:39 INFO SparkContext: Registered listener io.openlineage.spark.agent.OpenLineageSparkListener +22/05/02 18:43:40 INFO EventEmitter: Init OpenLineageContext: Args: ArgumentParser(host=<http://135.170.226.91:8400>, version=v1, namespace=gus-namespace, jobName=default, parentRunId=null, apiKey=Optional.empty, urlParams=Optional[{}]) URI: <http://135.170.226.91:8400/api/v1/lineage>? +22/05/02 18:46:21 ERROR EventEmitter: Could not emit lineage [responseCode=0]: {"eventType":"START","eventTime":"2022-05-02T18:44:08.36Z","run":{"runId":"91fd4e13-52ac-4175-8956-c06d7dee97fc","facets":{"spark_version":{"_producer":"<https://github.com/OpenLineage/OpenLineage/tree/0.8.1/integration/spark>","_schemaURL":"<https://openlineage.io/spec/1-0-2/OpenLineage.json#/$defs/RunFacet>","spark-version":"3.2.1","openlineage_spark_version":"0.8.1"},"spark.logicalPlan":{"_producer":"<https://github.com/OpenLineage/OpenLineage/tree/0.8.1/integration/spark>","_schemaURL":"<https://openlineage.io/spec/1-0-2/OpenLineage.json#/$defs/RunFacet>","plan":[{"class":"org.apache.spark.sql.catalyst.plans.logical.ShowNamespaces","num-children":1,"namespace":0,"output":[[{"class":"org.apache.spark.sql.catalyst.expressions.AttributeReference","num-children":0,"name":"databaseName","dataType":"string","nullable":false,"metadata":{},"exprId":{"product-class":"org.apache.spark.sql.catalyst.expressions.ExprId","id":4,"jvmId":"eaa0543b_5e04_4f5b_844b_0e4598f019a7"},"qualifier":[]}]]},{"class":"org.apache.spark.sql.catalyst.analysis.ResolvedNamespace","num_children":0,"catalog":null,"namespace":[]}]},"spark_unknown":{"_producer":"<https://github.com/OpenLineage/OpenLineage/tree/0.8.1/integration/spark>","_schemaURL":"<https://openlineage.io/spec/1-0-2/OpenLineage.json#/$defs/RunFacet>","output":{"description":"Unable to serialize logical plan due to: Infinite recursion (StackOverflowError) ... + OpenLineageHttpException(code=0, message=java.lang.RuntimeException: java.util.concurrent.ExecutionException: openlineage.hc.client5.http.ConnectTimeoutException: Connect to <http://135.170.226.91:8400> [/135.170.226.91] failed: Connection timed out, details=java.util.concurrent.CompletionException: java.lang.RuntimeException: java.util.concurrent.ExecutionException: openlineage.hc.client5.http.ConnectTimeoutException: Connect to <http://135.170.226.91:8400> [/135.170.226.91] failed: Connection timed out) + at io.openlineage.spark.agent.EventEmitter.emit(EventEmitter.java:68) + at io.openlineage.spark.agent.lifecycle.SparkSQLExecutionContext.start(SparkSQLExecutionContext.java:69) + at io.openlineage.spark.agent.OpenLineageSparkListener.lambda$sparkSQLExecStart$0(OpenLineageSparkListener.java:90) + at java.util.Optional.ifPresent(Optional.java:159) + at io.openlineage.spark.agent.OpenLineageSparkListener.sparkSQLExecStart(OpenLineageSparkListener.java:90) + at io.openlineage.spark.agent.OpenLineageSparkListener.onOtherEvent(OpenLineageSparkListener.java:81) + at org.apache.spark.scheduler.SparkListenerBus.doPostEvent(SparkListenerBus.scala:102) + at org.apache.spark.scheduler.SparkListenerBus.doPostEvent$(SparkListenerBus.scala:28) + at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37) + at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37) + at org.apache.spark.util.ListenerBus.postToAll(ListenerBus.scala:119) + at org.apache.spark.util.ListenerBus.postToAll$(ListenerBus.scala:103) + at org.apache.spark.scheduler.AsyncEventQueue.super$postToAll(AsyncEventQueue.scala:105) + at org.apache.spark.scheduler.AsyncEventQueue.$anonfun$dispatch$1(AsyncEventQueue.scala:105) + at scala.runtime.java8.JFunction0$mcJ$sp.apply(JFunction0$mcJ$sp.java:23) + at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62) + at <a href="http://org.apache.spark.scheduler.AsyncEventQueue.org">org.apache.spark.scheduler.AsyncEventQueue.org</a>$apache$spark$scheduler$AsyncEventQueue$$dispatch(AsyncEventQueue.scala:100) + at org.apache.spark.scheduler.AsyncEventQueue$$anon$2.$anonfun$run$1(AsyncEventQueue.scala:96) + at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1612) + at org.apache.spark.scheduler.AsyncEventQueue$$anon$2.run(AsyncEventQueue.scala:96) +the connection timeout is surprising because I can connect just fine using the example curl code from the same cluster:

+ +

%sh +curl -X POST <http://135.170.226.91:8400/api/v1/lineage> \ + -H 'Content-Type: application/json' \ + -d '{ + "eventType": "START", + "eventTime": "2020-12-28T19:52:00.001+10:00", + "run": { + "runId": "d46e465b-d358-4d32-83d4-df660ff614dd" + }, + "job": { + "namespace": "gus2~-namespace", + "name": "my-job" + }, + "inputs": [{ + "namespace": "gus2-namespace", + "name": "gus-input" + }], + "producer": "<https://github.com/OpenLineage/OpenLineage/blob/v1-0-0/client>" + }' +Spark config: +spark.openlineage.host <http://135.170.226.91:8400> +spark.openlineage.version v1 +spark.openlineage.namespace gus-namespace +Not sure what is going on, the EventEmitter init log looks like it's right but clearly something is off. Thanks so much for the help

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2022-05-02 15:03:40
 +
+

*Thread Reply:* hmmm, interesting - if it's easy could you spin both up locally and check that it's just a communication issue? It helps with diagnosis

+ +

It might also be a firewall issue, but your cURL should preclude that

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kostikey Mustakas + (kostikey.mustakas@gmail.com) +
+
2022-05-02 15:05:38
 +
+

*Thread Reply:* Since it's Databricks I was having a hard time figuring out how to try locally. Other than just using plain 'ol spark on my laptop and a localhost Marquez...

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2022-05-02 15:07:13
 +
+

*Thread Reply:* hmm, that could be an interesting test to see if it's a databricks issue - the databricks integration is pretty much the same as the spark integration, just with a little bit of a wrapper and the init script

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kostikey Mustakas + (kostikey.mustakas@gmail.com) +
+
2022-05-02 15:08:44
 +
+

*Thread Reply:* yeah, i was going to try that but it just didnt seem like helpful troubleshooting for exactly that reason... but i may just do that anyways just so i can see something working 🙂 (morale booster)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2022-05-02 15:09:22
 +
+

*Thread Reply:* oh totally! Network issues are a huge pain in the ass, and if you're still seeing issues locally with spark/mz then we'll know a lot more than we do now 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kostikey Mustakas + (kostikey.mustakas@gmail.com) +
+
2022-05-02 15:11:19
 +
+

*Thread Reply:* sounds good, i will give it a go!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-05-02 15:16:16
 +
+

*Thread Reply:* @Kostikey Mustakas - I think spark.openlineage.version should be equal to 1 not v1.

+ +

In addition, is http://135.170.226.91:8400 accessible to Databricks? Could you try doing a %sh command inside of a databricks notebook and see if you can ping that IP address (https://linux.die.net/man/8/ping)?

+ +

For your Databricks cluster did you VNET inject it into an existing VNET? If it's in an existing VNET, you should confirm that the VM running marquez can access it. If it's in a non-VNET injected VNET, you probably need to redeploy to a VNET that has that VM or has connectivity to that VM.

+
+
linux.die.net
+ + + +
ping(8) - Linux man page
+
+ ping uses the ICMP protocol's mandatory ECHO_REQUEST datagram to elicit an ICMP ECHO_RESPONSE from a host or gateway. ECHO_REQUEST datagrams (''pings'') ... +
+ + + + + + +
Original URL: https://linux.die.net/man/8/ping
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kostikey Mustakas + (kostikey.mustakas@gmail.com) +
+
2022-05-02 15:19:22
 +
+

*Thread Reply:* Ya, know i meant to ask about that. Docs say 1 like you mention: https://github.com/OpenLineage/OpenLineage/tree/main/integration/spark/databricks. I second guessed from this thread https://openlineage.slack.com/archives/C01CK9T7HKR/p1638848249159700.

+
+ + +
+ + + } + + Dinakar Sundar + (https://openlineage.slack.com/team/U02MQ8E22HF) +
+ + + +
+
+ <p>@John Thomas we in Condenast currently exploring the features of open lineage to integrate to databricks , <a href="https://github.com/OpenLineage/OpenLineage/tree/main/integration/spark/databricks">https://github.com/OpenLineage/OpenLineage/tree/main/integration/spark/databricks</a> , spark configuration not working ,</p> +
+ + + + + + +
Original URL: https://openlineage.slack.com/archives/C01CK9T7HKR/p1638848249159700
+ + +
+ + Thread in Slack Conversation +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kostikey Mustakas + (kostikey.mustakas@gmail.com) +
+
2022-05-02 15:23:42
 +
+

*Thread Reply:* @Will Johnson, ping fails... this is surprising as the curl command mentioned above works fine.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julius Rentergent + (julius.rentergent@thetradedesk.com) +
+
2022-05-02 15:37:00
 +
+

*Thread Reply:* I’m also trying to set up Databricks according to Running Marquez on AWS. Right now I’m stuck on the database part rather than the Marquez part — I can’t connect my EKS cluster to the RDS database which I described in more detail on the Marquez slack.

+ +

@Kostikey Mustakas Sorry for the distraction, but I’m curious how you have set up your networking to make the API requests work with Databricks. +Good luck with your issue!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kostikey Mustakas + (kostikey.mustakas@gmail.com) +
+
2022-05-02 15:47:17
 +
+

*Thread Reply:* @Julius Rentergent We are using Azure and leverage Private Endpoints to connect resources in separate subscriptions. There is a Bastion proxy in place that we can map http traffic through and I have a Load Balancer Inbound NAT rule I setup that maps one our whitelisted port ranges (8400) to 5000.

+ + + +
+ :gratitude_thank_you: Julius Rentergent +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kostikey Mustakas + (kostikey.mustakas@gmail.com) +
+
2022-05-02 20:15:01
 +
+

*Thread Reply:* @Will Johnson a little progress maybe... I created a private endpoint and updated dns to point to it. Now I get a 404 Not Found error instead of a timeout

+ + + +
+ 🙌 Will Johnson +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kostikey Mustakas + (kostikey.mustakas@gmail.com) +
+
2022-05-02 20:16:41
 +
+

*Thread Reply:* 22/05/03 00:09:24 ERROR EventEmitter: Could not emit lineage [responseCode=404]: {"eventType":"START","eventTime":"2022-05-03T00:09:22.498Z","run":{"runId":"f41575a0-e59d-4cbc-a401-9b52d2b020e0","facets":{"spark_version":{"_producer":"<https://github.com/OpenLineage/OpenLineage/tree/0.8.1/integration/spark>","_schemaURL":"<https://openlineage.io/spec/1-0-2/OpenLineage.json#/$defs/RunFacet>","spark-version":"3.2.1","openlineage_spark_version":"0.8.1"},"spark.logicalPlan":{"_producer":"<https://github.com/OpenLineage/OpenLineage/tree/0.8.1/integration/spark>","_schemaURL":"<https://openlineage.io/spec/1-0-2/OpenLineage.json#/$defs/RunFacet>","plan":[{"class":"org.apache.spark.sql.catalyst.plans.logical.ShowNamespaces","num-children":1,"namespace":0,"output":[[{"class":"org.apache.spark.sql.catalyst.expressions.AttributeReference","num-children":0,"name":"databaseName","dataType":"string","nullable":false,"metadata":{},"exprId":{"product-class":"org.apache.spark.sql.catalyst.expressions.ExprId","id":4,"jvmId":"aad3656d_8903_4db3_84f0_fe6d773d71c3"},"qualifier":[]}]]},{"class":"org.apache.spark.sql.catalyst.analysis.ResolvedNamespace","num_children":0,"catalog":null,"namespace":[]}]},"spark_unknown":{"_producer":"<https://github.com/OpenLineage/OpenLineage/tree/0.8.1/integration/spark>","_schemaURL":"<https://openlineage.io/spec/1-0-2/OpenLineage.json#/$defs/RunFacet>","output":{"description":"Unable to serialize logical plan due to: Infinite recursion (StackOverflowError) (through reference chain: org.apache.spark.sql.catalyst.expressions.AttributeReference[\"preCanonicalized\"] .... +OpenLineageHttpException(code=null, message={"code":404,"message":"HTTP 404 Not Found"}, details=null) + at io.openlineage.spark.agent.EventEmitter.emit(EventEmitter.java:68)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julius Rentergent + (julius.rentergent@thetradedesk.com) +
+
2022-05-27 00:03:30
 +
+

*Thread Reply:* Following up on this as I encounter the same issue with the Openlineage Databricks integration. This issue seems quite malicious as it crashes the Spark Context and requires a restart.

+ +

I have marquez running on AWS EKS; I’m using Openlineage 0.8.2 on Databricks 10.4 (Spark 3.2.1) and my Spark config looks like this: +spark.openlineage.host <https://internal-xxxxxxxxxxxxxxxxxxxxxxx-xxxxxxxxxxx.us-east-1.elb.amazonaws.com> +spark.openlineage.namespace default +spark.openlineage.version v1 &lt;- also tried "1" +I can run some simple read and write commands and successfully find the log4j events highlighted in the docs: +INFO SparkContext; +INFO OpenLineageContext; +INFO AsyncEventQueue for each time I run the cell +After doing this a few times I get The spark context has stopped and the driver is restarting. Your notebook will be automatically reattached. +stderr shows a bunch of things. log4j shows the same as for Kostikey: ERROR EventEmitter: [...] Unable to serialize logical plan due to: Infinite recursion (StackOverflowError)

+ +

I have one more piece of information which I can’t make much sense of, but hopefully someone else can; if I include the port in the host, I can very reliably crash the Spark Context on the first attempt. So: +<https://internal-xxxxxxxxxxxxxxxxxxxxxxx-xxxxxxxxxxx.us-east-1.elb.amazonaws.com> &lt;- crashes after a couple of attempts, sometimes it takes me a while to reproduce it while repeatedly reading/writing the same datasets +<https://internal-xxxxxxxxxxxxxxxxxxxxxxx-xxxxxxxxxxx.us-east-1.elb.amazonaws.com:80> &lt;- crashes on first try +Any insights would be greatly appreciated! 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julius Rentergent + (julius.rentergent@thetradedesk.com) +
+
2022-05-27 00:22:27
 +
+

*Thread Reply:* I tried two more things: +• curl works, ping fails, just like in the previous report +• Databricks allows providing spark configs without quotes, whereas quotes are generally required for Spark. So I added the quotes to the host name, but now I’m getting: ERROR OpenLineageSparkListener: Unable to parse open lineage endpoint. Lineage events will not be collected

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Martin Fiser + (fisa@keboola.com) +
+
2022-05-27 14:00:38
 +
+

*Thread Reply:* @Kostikey Mustakas May I ask what is the reason for migration from Palantir? Sorry for this off-topic question!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-05-30 05:46:27
 +
+

*Thread Reply:* @Julius Rentergent created issue on project github: https://github.com/OpenLineage/OpenLineage/issues/795

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/issues/795|#795 spark: port included in host causes crash of listener>
+
+ <p>Issue reported on Slack: <a href="https://openlineage.slack.com/archives/C01CK9T7HKR/p1653624210694359?thread_ts=1651498343.959749&amp;cid=C01CK9T7HKR|https://openlineage.slack.com/archives/C01CK9T7HKR/p1653624210694359?thread_ts=1651498343.959749&amp;cid=C01CK9T7HKR">https://openlineage.slack.com/archives/C01CK9T7HKR/p1653624210694359?thread<em>ts=1651498343.959749&amp;cid=C01CK9T7HKR|https://openlineage.slack.com/archives/C01CK9T7HKR/p1653624210694359?thread</em>ts=1651498343.959749&amp;cid=C01CK9T7HKR</a></p> + +<p>Following up on this as I encounter the same issue with the Openlineage Databricks integration. This issue seems quite malicious as it crashes the Spark Context and requires a restart. <br /> +I have marquez running on AWS EKS; I’m using Openlineage 0.8.2 on Databricks 10.4 (Spark 3.2.1) and my Spark config looks like this:</p> + +<pre><code>spark.openlineage.host &lt;https://internal-xxxxxxxxxxxxxxxxxxxxxxx-xxxxxxxxxxx.us-east-1.elb.amazonaws.com/&gt; +spark.openlineage.namespace default +spark.openlineage.version v1 &amp;lt;- also tried "1" + +</code></pre> + +<p>I can run some simple read and write commands and successfully find the log4j events <a href="https://github.com/OpenLineage/OpenLineage/tree/main/integration/spark/databricks">highlighted in the docs</a>:</p> + +<pre><code>INFO SparkContext; +INFO OpenLineageContext; +INFO AsyncEventQueue for each time I run the cell + +</code></pre> + +<p>After doing this a few times I get The spark context has stopped and the driver is restarting. Your notebook will be automatically reattached. <br /> +<a href="https://pastebin.com/9NbEjnGL">stderr shows a bunch of things</a>. log4j shows the same as for Kostikey: ERROR EventEmitter: [...] Unable to serialize logical plan due to: Infinite recursion (StackOverflowError) <br /> +I have one more piece of information which I can’t make much sense of, but hopefully someone else can; if I include the port in the host, I can very reliably crash the Spark Context on the first attempt. So:</p> + +<pre><code>&lt;https://internal-xxxxxxxxxxxxxxxxxxxxxxx-xxxxxxxxxxx.us-east-1.elb.amazonaws.com/&gt; &amp;lt;- crashes after a couple of attempts, sometimes it takes me a while to reproduce it while repeatedly reading/writing the same datasets +&lt;https://internal-xxxxxxxxxxxxxxxxxxxxxxx-xxxxxxxxxxx.us-east-1.elb.amazonaws.com:80/&gt; &amp;lt;- crashes on first try + +</code></pre> + +<p>Any insights would be greatly appreciated!</p> +
+ +
+
Labels
+ bug, integration/spark +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julius Rentergent + (julius.rentergent@thetradedesk.com) +
+
2022-06-01 11:15:26
 +
+

*Thread Reply:* Thank you @Maciej Obuchowski. +Just to clarify, the Spark Context crashes with and without port; it’s just that adding the port causes it to crash more quickly (on the 1st attempt).

+ +

I will run some more experiments when I have time, and add the results to the ticket.

+ +

Edit - added to issue:

+ +

I ran some more experiments, this time with a fake host and on OpenLineage 0.9.0, and was not able to reproduce the issue with regards to the port; instead, the new experiments show that Spark 3.2 looks to be involved.

+ +

On Spark 3.2.1 / Databricks 10.4 LTS: Using (fake) host http://ac7aca38330144df9.amazonaws.com:5000 crashes when the first notebook cell is evaluated with The spark context has stopped and the driver is restarting. + The same occurs when the port is removed.

+ +

On Spark 3.1.2 / Databricks 9.1 LTS: Using (fake) host http://ac7aca38330144df9.amazonaws.com:5000 does not impede the cluster but, reasonably, produces for each lineage event ERROR EventEmitter: Could not emit lineage w/ exception io.openlineage.client.OpenLineageClientException: java.net.UnknownHostException + The same occurs when the port is removed.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-05-02 14:52:09
 +
+

@channel The poll results are in, and the new day/time for the monthly TSC meeting is each second Thursday at 10 am PT. The next meeting will take place on Thursday, 5/19, at 10 am PT, due to a conflict with the Astronomer Spring Summit. Future meetings will take place on the second Thursday of each month. Calendar updates will be forthcoming. Thanks!

+ + + +
+ 🙌 Willy Lulciuc, Mynor Choc +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-05-02 15:09:42
 +
+

*Thread Reply:* @Michael Robinson - just to be sure, is the 5/19 meeting at 10 AM PT as well?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-05-02 15:14:11
 +
+

*Thread Reply:* Yes, and I’ll update the msg for others. Thank you

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-05-02 15:16:25
 +
+

*Thread Reply:* Thank you!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sandeep Bhat + (bhatsandeep424@gmail.com) +
+
2022-05-02 21:45:39
 +
+

Hii Team, as i saw marquez is building lineage by java code, from seed command, what should i do to connect with mysql (our database) with credentials and building a lineage for our data?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-05-03 12:40:55
 +
+

@here How do we clear old jobs, datasets and namespaces from Marquez?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mirko Raca + (racamirko@gmail.com) +
+
2022-05-04 07:04:48
 +
+

*Thread Reply:* It seems we can't for now. This was the same question I had last week:

+ +

https://github.com/MarquezProject/marquez/issues/1736

+
+ + + + +
<https://github.com/MarquezProject/marquez/issues/1736|#1736 Ability to DELETE jobs and datasets>
+
+ <p>When jobs or datasets are deleted (or renamed) we need a way to tell Marquez that they don't exist anymore so that they don't clutter the lineage view. <br /> +It would make sense to have <code>DELETE</code> support on <code>/api/v1/namespaces/{ns}/datasets/{name}</code> and <code>/api/v1/namespaces/{ns}/jobs/{name}</code> resources. <br /> +This is a soft delete in the internal marquez model. It creates a new <code>deleted</code> version. The resources will not show up in the "current" lineage view in <code>GET /api/v1/lineage</code></p> +
+ +
+
Comments
+ 1 +
+ + + + + + + +
+ + <a href="https://github.com/MarquezProject/marquez">MarquezProject/marquez</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-05-04 10:56:35
 +
+

*Thread Reply:* Seems that it's really popular request 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tyler Farris + (tyler@kickstand.work) +
+
2022-05-03 13:43:56
 +
+

Hello, +I'm sending lineage events to astrocloud.datakin DB with the Marquez API. The event is sent- but the metadata for inputs and outputs isn't coming through. Below is an example of the event I'm sending. Not sure if this is the place for this question. Cross-posting to Marquez Slack. +{ + "eventTime": "2022-05-03T17:20:04.151087+00:00", + "run": { + "runId": "2dfc6dcd4011d2a1c3dc1e5861127e5b" + }, + "job": { + "namespace": "from-airflow", + "name": "Postgres_1_to_Snowflake_2.extract" + }, + "producer": "<https://github.com/OpenLineage/OpenLineage/blob/v1-0-0/client>", + "inputs": [ + { + "name": "Postgres_1_to_Snowflake_2.extract", + "namespace": "from-airflow" + } + ] +} +Thanks.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tyler Farris + (tyler@kickstand.work) +
+
2022-05-04 11:28:48
 +
+

*Thread Reply:* @Mirko Raca pointed out that I was missing eventType.

+ +

Mirko Raca : +"From a quick glance - you're missing "eventType": "START", attribute. It's also worth noting that metadata typically shows up after the second event (type COMPLETE)"

+ +

thanks again.

+ + + +
+ 👍 Mirko Raca +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sandeep Bhat + (bhatsandeep424@gmail.com) +
+
2022-05-06 05:01:34
 +
+

Hii Team, could anyone tell me, to view lineage in marquez do we have to write metadata as a code, or does marquez has a feature to scan the sql code and build a lineage automatically?please clarify my doubt regarding this.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Juan Carlos Fernández Rodríguez + (jcfernandez@keedio.com) +
+
2022-05-06 05:26:16
 +
+

*Thread Reply:* As far as I understand, OpenLineage has tools to extract metadata from sources. Depend on your source, you could find an integration, if it doesn't exists you should write your own integration (and collaborate with the project)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-05-06 12:59:06
 +
+

*Thread Reply:* @Sandeep Bhat take a look at https://openlineage.io/integration - there is some info there on the different integrations that can be used to automatically pull metadata.

+
+
openlineage.io
+ + + +
Integrations
+
+ Data lineage is the foundation for a new generation of powerful, context-aware data tools and best practices. OpenLineage enables consistent collection of lineage metadata, creating a deeper understanding of how data is produced and used. +
+ + + + + + +
Original URL: https://openlineage.io/integration
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-05-06 13:00:39
 +
+

*Thread Reply:* The Airflow integration, in particular, uses a SQL parser to determine input/output tables (in cases where the data store can't be queried for that info)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jorik + (jorik@scivis.net) +
+
2022-05-12 05:13:01
 +
+

Hi all. We are looking at using OpenLineage for capturing some lineage in our custom processing system. I think we got the lineage events understood, but we have often datasets that get appended, or get overwritten by an operation. Is there anything in openlineage that would facilitate making this distinction? (ie. if a set gets overwritten we would be interested in the lineage events from the last overwrite, if it gets appended we would like to have all of these in the display)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mirko Raca + (racamirko@gmail.com) +
+
2022-05-12 05:48:43
 +
+

*Thread Reply:* To my understanding - datasets model the structure, not the content. So, as long as your table doesn't change number of columns, it's the same thing.

+ +

The catch-all would be to create a Dataset facet which would record the distinction between append/overwrite per run. But, while this is supported by the standard, Marquez does not handle custom facets at the moment (I'll happily be corrected).

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jorik + (jorik@scivis.net) +
+
2022-05-12 06:05:36
 +
+

*Thread Reply:* Thanks, that makes sense. We're looking for a way to get the lineage of table contents. We may have to opt for new names on overwrite, or indeed extend a facet to flag these.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jorik + (jorik@scivis.net) +
+
2022-05-12 06:06:44
 +
+

*Thread Reply:* Use case is compliancy, where we need to show how a certain delivered data product (at a given point in time) was constructed. We have all our transforms/transfers as code, but there are a few parts where datasets get recreated in the process after fixes have been made, and I wouldn't want to bother the auditors with those stray paths

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-05-12 06:12:09
 +
+

*Thread Reply:* We have LifecycleStateChangeDataset facet that captures this information. It's currently emitted when using Spark integration

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-05-12 06:13:25
 +
+

*Thread Reply:* > But, while this is supported by the standard, Marquez does not handle custom facets at the moment (I'll happily be corrected). +It displays this information when it exists

+ + + +
+ 🙌 Mirko Raca +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jorik + (jorik@scivis.net) +
+
2022-05-12 06:13:29
 +
+

*Thread Reply:* Oh that looks perfect! I completely missed that, thanks!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-05-12 15:46:04
 +
+

Are there any examples on how to use this facet ColumnLineageDatasetFacet.json?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-05-13 05:19:47
 +
+

*Thread Reply:* Work with Spark is not yet fully merged

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
raghanag + (raghanag@gmail.com) +
+
2022-05-12 17:49:23
 +
+

Hi All, I am trying to see where we can provide owner details when using openlineage-spark configuration, i see only namespace and other config parameters but not the owner. Can we add owner configuration also as part of openlineage-spark like spark.openlineage.owner? Owner will be used to even filter namespaces when showing the jobs or namespaces in Marquez UI.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-05-13 19:07:04
 +
+

@channel The next OpenLineage Technical Steering Committee meeting is next Thursday, 5/19, at 10 am PT! Going forward, meetings will take place on the second Thursday of each month at 10 am PT. +Join us on Zoom: +https://astronomer.zoom.us/j/87156607114?pwd=a3B0K210dnRaQmdkaFdGMytBREZEQT09 +All are welcome! +Agenda: +• releases 0.7.1 & 0.8.1 +• column-level lineage +• open lineage +For notes and the agenda visit the wiki: https://tinyurl.com/openlineagetsc

+ + + +
+ 🙌 Maciej Obuchowski, Ross Turk +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Yannick Libert + (yan@ioly.fr) +
+
2022-05-16 11:02:23
 +
+

Hi all, we are considering using OL to send lineage events from various jobs and places in our company. Since there will be multiple producers, we would like to use Kafka as our main hub for communication. One of our sources will be Airflow (more particularly MWAA, ie airflow in its 2.2.2 version). Is there a way to configure the Airflow lineage backend to send event to kafka instead of Marquez directly? So far, from what I've seen in the docs and in here, the only way would be to create a simple proxy to stream the http events to Kafka. Is it still the case?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-05-16 11:31:17
 +
+

*Thread Reply:* I think you can either use proxy backend: https://github.com/OpenLineage/OpenLineage/tree/main/proxy

+ +

or configure OL client to send data to kafka: +https://github.com/OpenLineage/OpenLineage/tree/main/client/python#kafka

+ + + +
+ 👍 Yannick Libert +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Yannick Libert + (yan@ioly.fr) +
+
2022-05-16 12:15:59
 +
+

*Thread Reply:* Thank you very much for the useful pointers. The proxy solutions could indeed work in our case but it implies creating another service in front of Kafka, and thus and another layer of complexity to the architecture. If there is another more "native" way of streaming event directly from the Airflow backend that'll be great to know

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-05-16 12:37:10
 +
+

*Thread Reply:* The second link 😉

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Yannick Libert + (yan@ioly.fr) +
+
2022-05-17 03:46:03
 +
+

*Thread Reply:* Sure, we already implemented the python client for jobs outside airflow and it works great 🙂 +You are saying that there is a way to use this python client in conjonction with the MWAA lineage backend to relay the job events that come with the airflow integration (without including it in the DAGs)? +Our strategy is to use both the airflow backend to collect automatic lineage events without modifying any existing DAGs, and the in-code implementation to allow our data engineers to send their own events if they want to. +The second option works perfectly but the first one is where we struggle a bit, especially with MWAA.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-05-17 05:24:30
 +
+

*Thread Reply:* If you can mount file to MWAA, then yes - it should work with config file option: https://github.com/OpenLineage/OpenLineage/tree/main/client/python#config-file

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Yannick Libert + (yan@ioly.fr) +
+
2022-05-17 05:40:45
 +
+

*Thread Reply:* Brilliant! I'm going to test that. Thank you Maciej!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-05-17 15:20:58
 +
+

A release has been requested. Are there any +1s? Three from committers will authorize. Thanks.

+ + + +
+ ➕ Maciej Obuchowski, Ross Turk, Willy Lulciuc, Michael Collado +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-05-18 10:33:03
 +
+

The OpenLineage TSC meeting is tomorrow at 10am PT! https://openlineage.slack.com/archives/C01CK9T7HKR/p1652483224119229

+
+ + +
+ + + } + + Michael Robinson + (https://openlineage.slack.com/team/U02LXF3HUN7) +
+ + + +
+
+ <p>@channel The next OpenLineage Technical Steering Committee meeting is next <strong>Thursday, 5/19, at 10 am PT</strong>! Going forward, meetings will take place on the second Thursday of each month at 10 am PT. +Join us on Zoom: +<a href="https://astronomer.zoom.us/j/87156607114?pwd=a3B0K210dnRaQmdkaFdGMytBREZEQT09">https://astronomer.zoom.us/j/87156607114?pwd=a3B0K210dnRaQmdkaFdGMytBREZEQT09</a> +All are welcome! +Agenda: +• releases 0.7.1 &amp; 0.8.1 +• column-level lineage +• open lineage +For notes and the agenda visit the wiki: <a href="https://tinyurl.com/openlineagetsc">https://tinyurl.com/openlineagetsc</a></p> +
+ + + + + + +
Original URL: https://openlineage.slack.com/archives/C01CK9T7HKR/p1652483224119229
+ + +
+ + Slack Conversation +
+ + +
+ + + +
+ 🙌 Willy Lulciuc +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tyler Farris + (tyler@kickstand.work) +
+
2022-05-18 16:23:56
 +
+

Hey all, +Do custom extractors work with the taskflow api?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2022-05-18 16:34:25
 +
+

*Thread Reply:* Hey Tyler - A custom extractor just needs to be able to assemble the runEvents and send the information out to the lineage backends.

+ +

If the things you're sending/receiving with TaskFlow are accessible in terms of metadata in the environment the DAG is running in, then you should be able to make one that would work!

+ +

This Webinar goes over creating custom extractors for reference.

+ +

Does that answer your question?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-05-18 16:41:16
 +
+

*Thread Reply:* Taskflow internally is just PythonOperator. If you'd write extractor that assumes something more than just it being PythonOperator then you'd probably make it work 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tyler Farris + (tyler@kickstand.work) +
+
2022-05-18 17:15:52
 +
+

*Thread Reply:* Thanks @John Thomas @Maciej Obuchowski, Your answers both make sense. I just keep running into this error in my logs: +[2022-05-18, 20:52:34 UTC] {__init__.py:97} WARNING - Unable to find an extractor. task_type=_PythonDecoratedOperator airflow_dag_id=Postgres_1_to_Snowflake_1_v3 task_id=Postgres_1 airflow_run_id=scheduled__2022-05-18T20:51:34.334045+00:00 +The picture is my custom extractor, it's not doing anything currently as this is just a test.

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tyler Farris + (tyler@kickstand.work) +
+
2022-05-18 17:16:05
 +
+

*Thread Reply:* thanks again for the help yall

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2022-05-18 17:16:34
 +
+

*Thread Reply:* did you set the environment variable with the path to your extractor?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tyler Farris + (tyler@kickstand.work) +
+
2022-05-18 17:16:46
 +
+

*Thread Reply:*

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tyler Farris + (tyler@kickstand.work) +
+
2022-05-18 17:17:13
 +
+

*Thread Reply:* i believe thats correct @John Thomas

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tyler Farris + (tyler@kickstand.work) +
+
2022-05-18 17:18:35
 +
+

*Thread Reply:* and the versions im using: +Astronomer Runtime 5.0.0 based on Airflow 2.3.0+astro.1

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2022-05-18 17:25:58
 +
+

*Thread Reply:* this might not be the problem, but you should have only one of extract and extract_on_complete - which one are you meaning to use?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tyler Farris + (tyler@kickstand.work) +
+
2022-05-18 17:32:26
 +
+

*Thread Reply:* ahh thanks John, as of right now extract_on_complete.

+ +

This is a similar setup as Michael had in the video.

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2022-05-18 17:33:31
 +
+

*Thread Reply:* if it's still not working I'm not really sure at this point - that's about what I had when I spun up my own custom extractor

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-05-18 17:39:44
 +
+

*Thread Reply:* is there anything in logs regarding extractors?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tyler Farris + (tyler@kickstand.work) +
+
2022-05-18 17:40:36
 +
+

*Thread Reply:* just this: +[2022-05-18, 21:36:59 UTC] {__init__.py:97} WARNING - Unable to find an extractor. task_type=_PythonDecoratedOperator airflow_dag_id=competitive_oss_projects_git_to_snowflake task_id=Transform_git_logs_to_S3 airflow_run_id=scheduled__2022-05-18T21:35:57.694690+00:00

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tyler Farris + (tyler@kickstand.work) +
+
2022-05-18 17:41:11
 +
+

*Thread Reply:* @John Thomas Thanks, I appreciate your help.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-05-19 06:01:52
 +
+

*Thread Reply:* No Failed to import messages?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tyler Farris + (tyler@kickstand.work) +
+
2022-05-19 11:26:34
 +
+

*Thread Reply: @Maciej Obuchowski None that I can see. Here is the full log: +``` Failed to verify remote log exists s3:///dag_id=Postgres_1_to_Snowflake_1_v3/run_id=scheduled2022-05-19T15:23:49.248097+00:00/task_id=Postgres_1/attempt=1.log. +Please provide a bucket_name instead of "s3:///dag_id=Postgres_1_to_Snowflake_1_v3/run_id=scheduled2022-05-19T15:23:49.248097+00:00/task_id=Postgres_1/attempt=1.log" + Falling back to local log +* Reading local file: /usr/local/airflow/logs/dagid=Postgres1toSnowflake1v3/runid=scheduled2022-05-19T15:23:49.248097+00:00/taskid=Postgres1/attempt=1.log +[2022-05-19, 15:24:50 UTC] {taskinstance.py:1158} INFO - Dependencies all met for <TaskInstance: Postgres1toSnowflake1v3.Postgres1 scheduled2022-05-19T15:23:49.248097+00:00 [queued]> +[2022-05-19, 15:24:50 UTC] {taskinstance.py:1158} INFO - Dependencies all met for <TaskInstance: Postgres1toSnowflake1v3.Postgres1 scheduled_2022-05-19T15:23:49.248097+00:00 [queued]>

+ +

[2022-05-19, 15:24:50 UTC] {taskinstance.py:1355} INFO -

+ +

[2022-05-19, 15:24:50 UTC] {taskinstance.py:1356} INFO - Starting attempt 1 of 1

+ +

[2022-05-19, 15:24:50 UTC] {taskinstance.py:1357} INFO -

+ +

[2022-05-19, 15:24:50 UTC] {taskinstance.py:1376} INFO - Executing <Task(PythonDecoratedOperator): Postgres1> on 2022-05-19 15:23:49.248097+00:00 +[2022-05-19, 15:24:50 UTC] {standardtaskrunner.py:52} INFO - Started process 3957 to run task +[2022-05-19, 15:24:50 UTC] {standardtaskrunner.py:79} INFO - Running: ['airflow', 'tasks', 'run', 'Postgres1toSnowflake1v3', 'Postgres1', 'scheduled2022-05-19T15:23:49.248097+00:00', '--job-id', '96473', '--raw', '--subdir', 'DAGSFOLDER/pgtosnow.py', '--cfg-path', '/tmp/tmp9n7u3i4t', '--error-file', '/tmp/tmp9a55v9b'] +[2022-05-19, 15:24:50 UTC] {standardtaskrunner.py:80} INFO - Job 96473: Subtask Postgres1 +[2022-05-19, 15:24:50 UTC] {loggingmixin.py:115} WARNING - /usr/local/lib/python3.9/site-packages/airflow/configuration.py:470 DeprecationWarning: The sqlalchemyconn option in [core] has been moved to the sqlalchemyconn option in [database] - the old setting has been used, but please update your config. +[2022-05-19, 15:24:50 UTC] {taskcommand.py:369} INFO - Running <TaskInstance: Postgres1toSnowflake1v3.Postgres1 scheduled2022-05-19T15:23:49.248097+00:00 [running]> on host 056ca0b6c7f5 +[2022-05-19, 15:24:50 UTC] {taskinstance.py:1568} INFO - Exporting the following env vars: +AIRFLOWCTXDAGOWNER=airflow +AIRFLOWCTXDAGID=Postgres1toSnowflake1v3 +AIRFLOWCTXTASKID=Postgres1 +AIRFLOWCTXEXECUTIONDATE=20220519T15:23:49.248097+00:00 +AIRFLOWCTXTRYNUMBER=1 +AIRFLOWCTXDAGRUNID=scheduled2022-05-19T15:23:49.248097+00:00 +[2022-05-19, 15:24:50 UTC] {loggingmixin.py:115} WARNING - /usr/local/lib/python3.9/site-packages/airflow/utils/context.py:202 AirflowContextDeprecationWarning: Accessing 'executiondate' from the template is deprecated and will be removed in a future version. Please use 'dataintervalstart' or 'logicaldate' instead. +[2022-05-19, 15:24:50 UTC] {loggingmixin.py:115} WARNING - /usr/local/lib/python3.9/site-packages/airflow/utils/context.py:202 AirflowContextDeprecationWarning: Accessing 'nextds' from the template is deprecated and will be removed in a future version. Please use '{{ dataintervalend | ds }}' instead. +[2022-05-19, 15:24:50 UTC] {loggingmixin.py:115} WARNING - /usr/local/lib/python3.9/site-packages/airflow/utils/context.py:202 AirflowContextDeprecationWarning: Accessing 'nextdsnodash' from the template is deprecated and will be removed in a future version. Please use '{{ dataintervalend | dsnodash }}' instead. +[2022-05-19, 15:24:50 UTC] {loggingmixin.py:115} WARNING - /usr/local/lib/python3.9/site-packages/airflow/utils/context.py:202 AirflowContextDeprecationWarning: Accessing 'nextexecutiondate' from the template is deprecated and will be removed in a future version. Please use 'dataintervalend' instead. +[2022-05-19, 15:24:50 UTC] {loggingmixin.py:115} WARNING - /usr/local/lib/python3.9/site-packages/airflow/utils/context.py:202 AirflowContextDeprecationWarning: Accessing 'prevds' from the template is deprecated and will be removed in a future version. +[2022-05-19, 15:24:50 UTC] {loggingmixin.py:115} WARNING - /usr/local/lib/python3.9/site-packages/airflow/utils/context.py:202 AirflowContextDeprecationWarning: Accessing 'prevdsnodash' from the template is deprecated and will be removed in a future version. +[2022-05-19, 15:24:50 UTC] {loggingmixin.py:115} WARNING - /usr/local/lib/python3.9/site-packages/airflow/utils/context.py:202 AirflowContextDeprecationWarning: Accessing 'prevexecutiondate' from the template is deprecated and will be removed in a future version. +[2022-05-19, 15:24:50 UTC] {loggingmixin.py:115} WARNING - /usr/local/lib/python3.9/site-packages/airflow/utils/context.py:202 AirflowContextDeprecationWarning: Accessing 'prevexecutiondatesuccess' from the template is deprecated and will be removed in a future version. Please use 'prevdataintervalstartsuccess' instead. +[2022-05-19, 15:24:50 UTC] {loggingmixin.py:115} WARNING - /usr/local/lib/python3.9/site-packages/airflow/utils/context.py:202 AirflowContextDeprecationWarning: Accessing 'tomorrowds' from the template is deprecated and will be removed in a future version. +[2022-05-19, 15:24:50 UTC] {loggingmixin.py:115} WARNING - /usr/local/lib/python3.9/site-packages/airflow/utils/context.py:202 AirflowContextDeprecationWarning: Accessing 'tomorrowdsnodash' from the template is deprecated and will be removed in a future version. +[2022-05-19, 15:24:50 UTC] {loggingmixin.py:115} WARNING - /usr/local/lib/python3.9/site-packages/airflow/utils/context.py:202 AirflowContextDeprecationWarning: Accessing 'yesterdayds' from the template is deprecated and will be removed in a future version. +[2022-05-19, 15:24:50 UTC] {loggingmixin.py:115} WARNING - /usr/local/lib/python3.9/site-packages/airflow/utils/context.py:202 AirflowContextDeprecationWarning: Accessing 'yesterdaydsnodash' from the template is deprecated and will be removed in a future version. +[2022-05-19, 15:24:50 UTC] {python.py:173} INFO - Done. Returned value was: extract +[2022-05-19, 15:24:50 UTC] {loggingmixin.py:115} WARNING - /usr/local/lib/python3.9/site-packages/airflow/models/baseoperator.py:1369 DeprecationWarning: Passing 'executiondate' to 'TaskInstance.xcompush()' is deprecated. +[2022-05-19, 15:24:50 UTC] {init.py:97} WARNING - Unable to find an extractor. tasktype=PythonDecoratedOperator airflowdagid=Postgres1toSnowflake1v3 taskid=Postgres1 airflowrunid=scheduled2022-05-19T15:23:49.248097+00:00 +[2022-05-19, 15:24:50 UTC] {client.py:74} INFO - Constructing openlineage client to send events to https://api.astro-livemaps.datakin.com/ +[2022-05-19, 15:24:50 UTC] {taskinstance.py:1394} INFO - Marking task as SUCCESS. dagid=Postgres1toSnowflake1v3, taskid=Postgres1, executiondate=20220519T152349, startdate=20220519T152450, enddate=20220519T152450 +[2022-05-19, 15:24:50 UTC] {localtaskjob.py:156} INFO - Task exited with return code 0 +[2022-05-19, 15:24:50 UTC] {localtask_job.py:273} INFO - 1 downstream tasks scheduled from follow-on schedule check```

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Josh Owens + (Josh@kickstand.work) +
+
2022-05-19 16:57:38
 +
+

*Thread Reply:* @Maciej Obuchowski is our ENV var wrong maybe? Do we need to mention the file to import somewhere else that we may have missed?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-05-20 10:26:01
 +
+

*Thread Reply:* @Josh Owens one thing I can think of is that you might have older openlineage integration version, as OPENLINEAGE_EXTRACTORS variable was added very recently: https://github.com/OpenLineage/OpenLineage/pull/694

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tyler Farris + (tyler@kickstand.work) +
+
2022-05-20 11:58:28
 +
+

*Thread Reply:* @Maciej Obuchowski, that was it! For some reason, my requirements.txt wasn't pulling the latest version of openlineage-airflow. Working now with 0.8.2

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-05-20 11:59:01
 +
+

*Thread Reply:* 🙌

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Raymond + (michael.raymond@cervest.earth) +
+
2022-05-19 05:32:06
 +
+

Hi 👋, I'm looking at OpenLineage as a solution for fine-grained data lineage tracking. Could I clarify a couple of points?

+ +

Where does one specify the version of an input dataset in the RunEvent? In the Marquez seed data I can see that it's recorded, but I'm not sure where it goes from looking at the OpenLineage schema. Or does it just assume the last version?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-05-19 05:59:59
 +
+

*Thread Reply:* Currently, it assumes latest version. +There's an effort with DatasetVersionDatasetFacet to be able to specify it manually - or extract this information from cases like Iceberg or Delta Lake tables.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Raymond + (michael.raymond@cervest.earth) +
+
2022-05-19 06:14:59
 +
+

*Thread Reply:* Ah ok. Is it Marquez assuming the latest version when it records the OpenLineage event?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-05-19 06:18:20
 +
+

*Thread Reply:* yes

+ + + +
+ ✅ Michael Raymond +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Raymond + (michael.raymond@cervest.earth) +
+
2022-05-19 06:54:40
 +
+

*Thread Reply:* Thanks, that's very helpful 👍

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Howard Yoo + (howardyoo@gmail.com) +
+
2022-05-19 15:23:33
 +
+

Hi all, +I was testing https://github.com/MarquezProject/marquez/tree/main/examples/airflow#step-21-create-dag-counter, and the following error was observed in my airflow env:

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Howard Yoo + (howardyoo@gmail.com) +
+
2022-05-19 15:23:52
 +
+

Anybody know why this is happening? Any comments would be welcomed.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tyler Farris + (tyler@kickstand.work) +
+
2022-05-19 15:27:35
 +
+

*Thread Reply:* @Howard Yoo What version of airflow?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Howard Yoo + (howardyoo@gmail.com) +
+
2022-05-19 15:27:51
 +
+

*Thread Reply:* it's 2.3

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Howard Yoo + (howardyoo@gmail.com) +
+
2022-05-19 15:28:42
 +
+

*Thread Reply:* (sorry, it's 2.4)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tyler Farris + (tyler@kickstand.work) +
+
2022-05-19 15:29:28
 +
+

*Thread Reply:* https://github.com/OpenLineage/OpenLineage/tree/main/integration/airflow Id refer to the docs again.

+ +

"Airflow 2.3+ +Integration automatically registers itself for Airflow 2.3 if it's installed on Airflow worker's python. This means you don't have to do anything besides configuring it, which is described in Configuration section."

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Howard Yoo + (howardyoo@gmail.com) +
+
2022-05-19 15:29:53
 +
+

*Thread Reply:* Right, configuring I don't see any issues

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tyler Farris + (tyler@kickstand.work) +
+
2022-05-19 15:30:56
 +
+

*Thread Reply:* so you dont need:

+ +

from openlineage.airflow import DAG

+ +

in your dag files

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Howard Yoo + (howardyoo@gmail.com) +
+
2022-05-19 15:31:41
 +
+

*Thread Reply:* Okay... that makes sense then

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tyler Farris + (tyler@kickstand.work) +
+
2022-05-19 15:32:47
 +
+

*Thread Reply:* so if you need to import DAG it would just be: +from airflow import DAG

+ + + +
+ 👍 Howard Yoo +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Howard Yoo + (howardyoo@gmail.com) +
+
2022-05-19 15:56:19
 +
+

*Thread Reply:* Thanks!

+ + + +
+ 👍 Tyler Farris +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-05-19 17:13:02
 +
+

@channel OpenLineage 0.8.2 is now available! The project now supports credentialing from the Airflow Secrets Backend and for the Azure Databricks Credential Passthrough, detection of datasets wrapped by ExternalRDDs, bug fixes, and more. For the details, see: https://github.com/OpenLineage/OpenLineage/releases/tag/0.8.2

+ + + +
+ 🎉 Marco Diaz, Howard Yoo, Willy Lulciuc, Michael Collado, Ross Turk, Francis McGregor-Macdonald, Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
xiang chen + (cdmikechen@hotmail.com) +
+
2022-05-19 22:18:42
 +
+

Hi~ everyone Is there possible to let openlineage to support camel pipeline?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-05-20 10:23:55
 +
+

*Thread Reply:* What changes do you mean by letting openlineage support? +Or, do you mean, to write Apache Camel integration?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
xiang chen + (cdmikechen@hotmail.com) +
+
2022-05-22 19:54:17
 +
+

*Thread Reply:* @Maciej Obuchowski Yes, let openlineage work as same as airflow

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
xiang chen + (cdmikechen@hotmail.com) +
+
2022-05-22 19:56:47
 +
+

*Thread Reply:* I think this is a very valuable thing. I wish openlineage can support some commonly used pipeline tools, and try to abstract out some general interfaces so that users can expand by themselves

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-05-23 05:20:30
 +
+

*Thread Reply:* For Python, we have OL client, common libraries (well, at least beginning of them) and SQL parser

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-05-23 05:20:44
 +
+

*Thread Reply:* As we support more systems, the general libraries will grow as well.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Conor Beverland + (conorbev@gmail.com) +
+
2022-05-20 13:50:53
 +
+

I see a change in the metadata collected from Airflow jobs which I think was introduced with the combination of Airflow 2.3/OpenLineage 0.8.1. There's an airflow_version facet that contains an operator attribute.

+ +

Previously that attribute had values such as: airflow.providers.postgres.operators.postgres.PostgresOperator but I now see that for the very same task the operator is now tracked as: airflow.models.taskinstance.TaskInstance

+ +

( fwiw there's also a taskInfo attribute in there containing a json string which itself has a operator that is still set to PostgresOperator )

+ +

Is this an already known issue?

+ + + +
+ 👀 Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2022-05-20 20:23:15
 +
+

*Thread Reply:* This looks like a bug. we are probably not looking at the right instance in the TaskInstanceListener

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Conor Beverland + (conorbev@gmail.com) +
+
2022-05-21 14:17:19
 +
+

*Thread Reply:* @Howard Yoo I filed: https://github.com/OpenLineage/OpenLineage/issues/767 for this

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/issues/767|#767 [BUG][INTEGRATION][Airflow] Incorrect Operator tracked in Airflow 2.3>
+
+ <p>I see a change in the metadata collected from Airflow jobs which I think was introduced with the combination of Airflow 2.3/OpenLineage 0.8.1. There's an <code>airflow_version</code> facet that contains an <code>operator</code> attribute that is added by the integration.</p> + +<p>In Airflow 2.2 for a <code>PostgresOperator</code> that attribute would have had a value: <code>airflow.providers.postgres.operators.postgres.PostgresOperator</code> but now in Airflow 2.3 that operator is tracked as: <code>airflow.models.taskinstance.TaskInstance</code> <br /> +( fwiw there's also a <code>taskInfo</code> attribute in there containing a json string which itself has a <code>operator</code> that is still set to <code>PostgresOperator</code> )</p> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-05-20 21:42:46
 +
+

Would anyone happen to have a link to the Technical Steering Committee meeting recordings?

+ +

I have quite a few people interested in seeing the overview of column lineage that Pawel provided during the Technical Steering Committee meeting on Thursday May 19th.

+ +

The wiki does not include a link to the recordings: https://wiki.lfaidata.foundation/display/OpenLineage/Monthly+TSC+meeting

+ +

Are the recordings made public? Thank you for any links and guidance!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2022-05-20 21:55:09
 +
+

That would be @Michael Robinson Yes the recordings are made public.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-05-20 22:05:27
 +
+

@Will Johnson I’ll put this on the https://wiki.lfaidata.foundation/display/OpenLineage/Monthly+TSC+meeting|wiki soon, but here is the link to the recording: https://astronomer.zoom.us/rec/share/xUBW-n6G4u1WS89tCSXStx8BMl99rCfCC6jGdXLnkN6gMGn5G-_BC7pxHKKeELhG.0JFl88isqb64xX-3 +PW: 1VJ=K5&X

+ + + +
+ 🙌 Will Johnson +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-05-21 09:42:21
 +
+

*Thread Reply:* Thank you so much, Michael!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tyler Farris + (tyler@kickstand.work) +
+
2022-05-23 15:00:10
 +
+

Is there documentation/examples around creating custom facets?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-05-24 06:41:11
 +
+

*Thread Reply:* In Python or Java?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-05-24 06:44:32
 +
+

*Thread Reply:* In python just inherit BaseFacet and add _get_schema static method that would point to some place where you have your json schema of a facet. For example our DbtVersionRunFacet

+ +

In Java you can take a look at Spark's custom facets.

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/integration/common/openlineage/common/provider/dbt.py | dbt.py>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/269107690da704ee19e3b4a0362c3fe8da155eaf/integration/spark/src/main/common/java/io/openlineage/spark/agent/facets/TableProviderFacet.java | TableProviderFacet.java>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tyler Farris + (tyler@kickstand.work) +
+
2022-05-24 16:40:00
 +
+

*Thread Reply:* Thanks, @Maciej Obuchowski, I was asking in regards to Python, sorry I should have clarified.

+ +

I'm not sure what the disconnect is, but the facets aren't showing up in the inputs and outputs. The Lineage event is sent successfully to my astrocloud.

+ +

below is the facet and extractor, any help is appreciated. Thanks!

+ +

```import logging +from openlineage.airflow.extractors.base import BaseExtractor, TaskMetadata +from openlineage.client.run import InputDataset, OutputDataset +from typing import List, Optional +from openlineage.client.facet import BaseFacet +import attr

+ +

log = logging.getLogger(name)

+ +

@attr.s +class ManualLineageFacet(BaseFacet): + database: Optional[str] = attr.ib(default=None) + cluster: Optional[str] = attr.ib(default=None) + connectionUrl: Optional[str] = attr.ib(default=None) + target: Optional[str] = attr.ib(default=None) + source: Optional[str] = attr.ib(default=None) + _producer: str = attr.ib(init=False) + _schemaURL: str = attr.ib(init=False)

+ +
@staticmethod
+def _get_schema() -&gt; str:
+    return {
+        "$schema": "<http://json-schema.org/schema#>",
+        "$defs": {
+            "ManualLineageFacet": {
+                "allOf": [
+                    {
+                        "type": "object",
+                        "properties": {
+                            "database": {
+                                "type": "string",
+                                "example": "Snowflake",
+                            },
+                            "cluster": {
+                                "type": "string",
+                                "example": "us-west-2",
+                            },
+                            "connectionUrl": {
+                                "type": "string",
+                                "example": "<http://snowflake>",
+                            },
+                            "target": {
+                                "type": "string",
+                                "example": "Postgres",
+                            },
+                            "source": {
+                                "type": "string",
+                                "example": "Stripe",
+                            },
+                            "description": {
+                                "type": "string",
+                                "example": "Description of inlet/outlet",
+                            },
+                            "_producer": {
+                                "type": "string",
+                            },
+                            "_schemaURL": {
+                                "type": "string",
+                            },
+                        },
+                    },
+                ],
+                "type": "object",
+            }
+        },
+    }
+
+ +

class ManualLineageExtractor(BaseExtractor): + @classmethod + def getoperatorclassnames(cls) -> List[str]: + return ["PythonOperator", "_PythonDecoratedOperator"]

+ +
def extract_on_complete(self, task_instance) -&gt; Optional[TaskMetadata]:
+
+    return TaskMetadata(
+        f"{task_instance.dag_run.dag_id}.{task_instance.task_id}",
+        inputs=[
+            InputDataset(
+                namespace="default",
+                name=self.operator.get_inlet_defs()[0]["name"],
+                inputFacets=ManualLineageFacet(
+                    database=self.operator.get_inlet_defs()[0]["database"],
+                    cluster=self.operator.get_inlet_defs()[0]["cluster"],
+                    connectionUrl=self.operator.get_inlet_defs()[0][
+                        "connectionUrl"
+                    ],
+                    target=self.operator.get_inlet_defs()[0]["target"],
+                    source=self.operator.get_inlet_defs()[0]["source"],
+                ),
+            )
+            if self.operator.get_inlet_defs()
+            else {},
+        ],
+        outputs=[
+            OutputDataset(
+                namespace="default",
+                name=self.operator.get_outlet_defs()[0]["name"],
+                outputFacets=ManualLineageFacet(
+                    database=self.operator.get_outlet_defs()[0]["database"],
+                    cluster=self.operator.get_outlet_defs()[0]["cluster"],
+                    connectionUrl=self.operator.get_outlet_defs()[0][
+                        "connectionUrl"
+                    ],
+                    target=self.operator.get_outlet_defs()[0]["target"],
+                    source=self.operator.get_outlet_defs()[0]["source"],
+                ),
+            )
+            if self.operator.get_outlet_defs()
+            else {},
+        ],
+        job_facets={},
+        run_facets={},
+    )
+
+def extract(self) -&gt; Optional[TaskMetadata]:
+    pass```
+
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-05-25 09:21:02
 +
+

*Thread Reply:* _get_schema should return address to the schema hosted somewhere else - afaik sending object field where server expects string field might cause some problems

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-05-25 09:21:59
 +
+

*Thread Reply:* can you register ManualLineageFacet as facets not as inputFacets or outputFacets?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tyler Farris + (tyler@kickstand.work) +
+
2022-05-25 13:15:30
 +
+

*Thread Reply:* Thanks for the advice @Maciej Obuchowski, I was able to get it working! +Also great talk today at the airflow summit.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-05-25 13:25:17
 +
+

*Thread Reply:* Thanks 🙇

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Bruno González + (brugms2@gmail.com) +
+
2022-05-24 06:26:25
 +
+

Hey guys! I'm pretty new with OL but would like to start using it for a combination of data lineage in Airflow + data quality metrics collection. I was wondering if that was possible, but Ross clarified that in the deeper dive webinar from some weeks ago (great one by the way!).

+ +

I'm referencing this comment from Julien to see if you have any updates or more examples apart from the one from great expectations. We have some custom operators and would like to push lineage and data quality metrics to Marquez using custom extractors. Any reference will be highly appreciated. Thanks in advance!

+
+ + +
+ + + } + + Julien Le Dem + (https://openlineage.slack.com/team/U01DCLP0GU9) +
+ + + +
+
+ <p>I have updated the DataQuality metrics proposal and the corresponding PR: <a href="https://github.com/OpenLineage/OpenLineage/issues/101">https://github.com/OpenLineage/OpenLineage/issues/101</a> +<a href="https://github.com/OpenLineage/OpenLineage/pull/115">https://github.com/OpenLineage/OpenLineage/pull/115</a></p> +
+ + + + + + + +
+ + Slack Conversation +
+ + +
+
+
YouTube
+ +
+ + + } + + Astronomer + (https://www.youtube.com/c/Astronomer) +
+ + + +
OpenLineage and Airflow: A Deeper Dive
+
+ +
+ + + + + + + + +
Original URL: https://www.youtube.com/watch?v=LRr-ja8_Wjs
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-05-24 06:35:05
 +
+

*Thread Reply:* We're also getting data quality from dbt if you're running dbt test or dbt build +https://github.com/OpenLineage/OpenLineage/blob/main/integration/common/openlineage/common/provider/dbt.py#L399

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/integration/common/openlineage/common/provider/dbt.py | dbt.py>
+
+ <pre><code> def parse_test( +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-05-24 06:37:15
 +
+

*Thread Reply:* Generally, you'd need to construct DataQualityAssertionsDatasetFacet and/or DataQualityMetricsInputDatasetFacet and attach it to tested dataset

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/2604990e5387d1b8fb73a61664e35e702f25eed5/client/python/openlineage/client/facet.py | facet.py>
+
+ <pre><code>class DataQualityAssertionsDatasetFacet(BaseFacet): +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Bruno González + (brugms2@gmail.com) +
+
2022-05-24 13:23:34
 +
+

*Thread Reply:* Thanks @Maciej Obuchowski!!!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Howard Yoo + (howardyoo@gmail.com) +
+
2022-05-24 16:55:08
 +
+

Hi all, https://github.com/OpenLineage/OpenLineage/tree/main/integration/airflow#development <-- does this still work? I did follow the instructions, but running pytest failed with error messages like +________________________________________________ ERROR collecting tests/extractors/test_bigquery_extractor.py ________________________________________________ +ImportError while importing test module '/Users/howardyoo/git/OpenLineage/integration/airflow/tests/extractors/test_bigquery_extractor.py'. +Hint: make sure your test modules/packages have valid Python names. +Traceback: +openlineage/airflow/utils.py:251: in import_from_string + module = importlib.import_module(module_path) +/opt/homebrew/Caskroom/miniconda/base/envs/airflow/lib/python3.9/importlib/__init__.py:127: in import_module + return _bootstrap._gcd_import(name[level:], package, level) +&lt;frozen importlib._bootstrap&gt;:1030: in _gcd_import + ??? +&lt;frozen importlib._bootstrap&gt;:1007: in _find_and_load + ??? +&lt;frozen importlib._bootstrap&gt;:986: in _find_and_load_unlocked + ??? +&lt;frozen importlib._bootstrap&gt;:680: in _load_unlocked + ??? +&lt;frozen importlib._bootstrap_external&gt;:850: in exec_module + ??? +&lt;frozen importlib._bootstrap&gt;:228: in _call_with_frames_removed + ??? +../../../airflow.master/airflow/providers/google/cloud/operators/bigquery.py:39: in &lt;module&gt; + from airflow.providers.google.cloud.hooks.bigquery import BigQueryHook, BigQueryJob +../../../airflow.master/airflow/providers/google/cloud/hooks/bigquery.py:46: in &lt;module&gt; + from googleapiclient.discovery import Resource, build +E ModuleNotFoundError: No module named 'googleapiclient'

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Howard Yoo + (howardyoo@gmail.com) +
+
2022-05-24 16:55:09
 +
+

...

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Howard Yoo + (howardyoo@gmail.com) +
+
2022-05-24 16:55:54
 +
+

looks like just running the pytest wouldn't be able to run all the tests - as some of these dag tests seems to be requiring connectivities to google's big query, databases, etc..

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mardaunt + (miostat@yandex.ru) +
+
2022-05-25 16:32:08
 +
+

👋 Hi everyone! +I didn't find this in the documentation. +Can open lineage show me which source columns the final DataFrame column came from? (Spark)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-05-25 16:59:47
 +
+

*Thread Reply:* We're working on this feature - should be in the next release from OpenLineage side

+ + + +
+ 🙌 Mardaunt +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mardaunt + (miostat@yandex.ru) +
+
2022-05-25 17:06:12
 +
+

*Thread Reply:* Thanks! I will keep an eye on updates.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Martin Fiser + (fisa@keboola.com) +
+
2022-05-25 21:08:39
 +
+

Hi all, showcase time:

+ +

We have implemented a native OpenLineage endpoint and metadata writer in our Keboola all-in-one data platform. +The reason was that for more complex data pipeline scenarios it is beneficial to display the lineage in more detail. Additionally, we hope that OpenLineage as a standard will catch up and open up the ability to push lineage data into other data governance tools than Marquez. +The implementation started as an internal POC of tweaking our metadata into OpenLineage /lineage format and resulted into a native API endpoint and later on an app within Keboola platform ecosystem - feeding platform job metadata in a regular cadence. +We furthermore use a namespace for each keboola project so users can observe the data through their whole data mesh setup (multi-project architecture). +Please reach me out if you have any questions!

+ +
+
+ + + + + + +
+
+
+ + + + + + +
+
+
+ + + + + + +
+
+
+ + + + + + +
+ + +
+ 🙌 Maciej Obuchowski, Michael Robinson +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-05-26 06:05:33
 +
+

*Thread Reply:* Looks great! Thanks for sharing!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Gopi Krishnan Rajbahadur + (gopikrishnanrajbahadur@gmail.com) +
+
2022-05-26 10:13:26
 +
+

Hi OpenLineage team,

+ +

I am Gopi Krishnan Rajbahadur, one of the core members of OpenDatalogy project (a project that we are currently trying to sandbox as a part of LF-AI). Our OpenDatalogy project focuses on providing a process that allows users of publicly available datasets (e.g., CIFAR-10) to ensure license compliance. In addition, we also aim to provide a public repo that documents the final rights and obligations associated with common publicly available datasets, so that users of these datasets can use them compliantly in their AI models and software.

+ +

One of the key aspects of conducting dataset license compliance analysis involves tracking the lineage and provenance of the dataset (as we highlight in this paper here: https://arxiv.org/abs/2111.02374). We think that in this regard, our projects (i.e., OpenLineage and OpenDatalogy) could work together to use the existing OpenLineage standard and also collaborate to adopt/modify/enhance and use OpenLineage to track and document the lineage of a publicly available dataset. On that note, we are also working with the SPDX community to make the lineage and provenance of a dataset be tracked as a part of the SPDX BOM that is in the works for representing AI software (AI SBOM).

+ +

We think our projects could mutually benefit from collaborating with each other. Our project's Github could be found here: https://github.com/OpenDataology/OpenDataology. Any feedback that you have about our project would be greatly appreciated. Also, as we are trying to sandbox our project, if you could also show us your support we would greatly appreciate it!

+ +

Look forward to hearing back from you

+ +

Sincerely, +Gopi

+
+
arXiv.org
+ + + +
Can I use this publicly available dataset to build commercial AI...
+
+ Publicly available datasets are one of the key drivers for commercial AI software. The use of publicly available datasets is governed by dataset licenses. These dataset licenses outline the rights... +
+ + + + + + + + +
Original URL: https://arxiv.org/abs/2111.02374
+ + + +
+
+ + + + +
OpenDataology/OpenDataology
+
+ Practice of AI dataset metadata and license compliance +
+ +
+
Stars
+ 3 +
+ +
+
Last updated
+ 3 days ago +
+ + + + + + + + +
+ + + +
+ 👀 Howard Yoo, Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ilqar Memmedov + (ccmilgar@gmail.com) +
+
2022-05-30 04:25:10
 +
+

Hi guys, sorry for basics. +I did some PoC for OpenLineage usage for gathering metrics on Spark job, especially for table creation, alter and drop +I detect that Drop/Alter table statements is not trigger listener to post lineage data, Is it normal behaviour?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-05-30 05:38:41
 +
+

*Thread Reply:* Might be that case if you're using Spark 3.2

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-05-30 05:38:54
 +
+

*Thread Reply:* There were some changes to those operators

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-05-30 05:39:09
 +
+

*Thread Reply:* If you're not using 3.2, please share more details 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ilqar Memmedov + (ccmilgar@gmail.com) +
+
2022-05-30 07:58:58
 +
+

*Thread Reply:* Yeap, im using spark version 3.2.1

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ilqar Memmedov + (ccmilgar@gmail.com) +
+
2022-05-30 07:59:35
 +
+

*Thread Reply:* is it open issue, or i have some option to force them to be sent?)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ilqar Memmedov + (ccmilgar@gmail.com) +
+
2022-05-30 07:59:58
 +
+

*Thread Reply:* btw thank you for quick response @Maciej Obuchowski

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-05-30 08:00:34
 +
+

*Thread Reply:* Yes, we have issue for AlterTable at least

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2022-06-01 02:52:14
 +
+

*Thread Reply:* https://github.com/OpenLineage/OpenLineage/issues/616 -> that’s the issue for altering tables in Spark 3.2. +@Ilqar Memmedov Did you mean drop table or drop columns? I am not aware of any drop table issue.

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/issues/616|#616 [INTEGRATION][SPARK] Implement dataset builders for alter table commands>
+
+ <p>Spark 3.2.1 introduced new alter table commands. In order to collect this information, new dataset builders need to be created.</p> + +<p><a href="https://github.com/apache/spark/blob/v3.2.1/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/v2AlterTableCommands.scala">https://github.com/apache/spark/blob/v3.2.1/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/v2AlterTableCommands.scala</a></p> +
+ +
+
Assignees
+ <a href="https://github.com/tnazarew">@tnazarew</a> +
+ +
+
Labels
+ enhancement, integration/spark +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ilqar Memmedov + (ccmilgar@gmail.com) +
+
2022-06-01 06:03:38
 +
+

*Thread Reply:* @Paweł Leszczyński drop table statement.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ilqar Memmedov + (ccmilgar@gmail.com) +
+
2022-06-01 06:05:58
 +
+

*Thread Reply:* For reproduce it, i just create simple spark job. +Create table as select from other, +Select data from table, and then drop entire table.

+ +

Lineage data was posted only for "Create table as select" part

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
xiang chen + (cdmikechen@hotmail.com) +
+
2022-06-01 05:16:01
 +
+

Hi~all, I have a question about lineage. I am now running airflow 2.3.1 and have started a latest marquez service by docker-compose. I found that using the example DAG of airflow can only see the job information, but not the lineage of the job. How can I configure it to see the lineage ?

+ +
+
+ + + + + + +
+
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-06-03 14:20:16
 +
+

*Thread Reply:* hi xiang 👋 lineage in airflow depends on the operator. some operators have extractors as part of the integration, but when they are missing you only see job information in Marquez.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-06-03 14:20:51
 +
+

*Thread Reply:* take a look at https://github.com/OpenLineage/OpenLineage/tree/main/integration/airflow#extractors--sending-the-correct-data-from-your-dags for a bit more detail

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
xiang chen + (cdmikechen@hotmail.com) +
+
2022-06-01 05:23:54
 +
+

Another problem is that if I declare a skip task(e.g. DummyOperator) in the DAG, it will never appear in the job list. I think this is a problem, because even if it can not run, it should be able to see it as a metadata object.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-06-01 10:19:33
 +
+

@channel The next OpenLineage Technical Steering Committee meeting is on Thursday, June 9 at 10 am PT. Join us on Zoom: https://us02web.zoom.us/j/81831865546?pwd=RTladlNpc0FTTDlFcWRkM2JyazM4Zz09 +All are welcome! +Agenda:

+ +
  1. a recent blog post about Snowflake
  2. the Great Expectations integration
  3. the dbt integration
  4. Open discussion +Notes: https://tinyurl.com/openlineagetsc +Is there a topic you think the community should discuss at this or a future meeting? DM me to add items to the agenda.
    5. +
+ + + +
+ 👀 Howard Yoo, Francis McGregor-Macdonald +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-06-04 09:45:41
 +
+

@channel OpenLineage 0.9.0 is now available, featuring column-level lineage in the Spark integration, bug fixes and more! For the details, see: https://github.com/OpenLineage/OpenLineage/releases/tag/0.9.0 and https://github.com/OpenLineage/OpenLineage/compare/0.8.2...0.9.0. Thanks to all the contributors who made this release possible, including @Paweł Leszczyński for authoring the column-level lineage PRs and new contributor @JDarDagran!

+ + + +
+ 👍 Howard Yoo, Jarek Potiuk, Maciej Obuchowski, Ross Turk, Minkyu Park, pankaj koti, Jorik, Li Ding, Faouzi, Howard Yoo, Mardaunt +
+ +
+ 🎉 pankaj koti, Faouzi, Howard Yoo, Sheeri Cabral (Collibra), Mardaunt +
+ +
+ ❤️ Faouzi, Howard Yoo, Mardaunt +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tyler Farris + (tyler@kickstand.work) +
+
2022-06-06 16:14:52
 +
+

Hey, all. Working on a PR to OpenLineage. I'm curious about file naming conventions for facets. Im noticing that there are two conventions being used:

+ +

• In OpenLineage.spec.facets; ex. ExampleFacet.json +• In OpenLineage.integration.common.openlineage.common.schema; ex. example-facet.json. +Thanks

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-06-08 08:02:58
 +
+

*Thread Reply:* I think internal naming is more important 🙂

+ +

I guess, for now, try to match what the local directory has.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tyler Farris + (tyler@kickstand.work) +
+
2022-06-08 10:59:39
 +
+

*Thread Reply:* Thanks @Maciej Obuchowski

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
raghanag + (raghanag@gmail.com) +
+
2022-06-07 03:24:03
 +
+

Hi Team, we are seeing DatasetName as the Custom query when we run a spark job which queries Oracle DB using JDBC with a Custom Query and the custom query is having newline syntax in it which is causing the NodeId ID_PATTERN match to fail. How to give custom dataset name when we use custom queries?

+ +

Marquez API regex ref: https://github.com/MarquezProject/marquez/blob/main/api/src/main/java/marquez/service/models/NodeId.java#L44 +ERROR [2022-06-07 06:11:49,592] io.dropwizard.jersey.errors.LoggingExceptionMapper: Error handling a request: 3648e87216d7815b +! java.lang.IllegalArgumentException: node ID (dataset:oracle:thin:_//&lt;host-name&gt;:1521:( +! SELECT +! RULE.RULE_ID, +! ASSG.ASSIGNED_OBJECT_ID, ASSG.ORG_ID, ASSG.SPLIT_PCT, +! PRTCP.PARTICIPANT_NAME, PRTCP.START_DATE, PRTCP.END_DATE +! FROM RULE RULE, +! ASSG ASSG, +! PRTCP PRTCP +! WHERE +! RULE.RULE_ID = ASSG.RULE_ID(+) +! --AND RULE.RULE_ID = 300100207891651 +! AND PRTCP.PARTICIPANT_ID = ASSG.ASSIGNED_OBJECT_ID +! -- and RULE.created_by = ' 1=1 ' +! and 1=1 +! )) must start with 'dataset', 'job', or 'run'

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
George Zachariah V + (manish.zack@gmail.com) +
+
2022-06-08 07:48:16
 +
+

Hi Team, +We have a spark job xyz that uses OpenLineageListener which posts Lineage events to Marquez server. But we are seeing some unknown jobs in the Marquez UI : +• xyz.collect_limit +• xyz.execute_insert_into_hadoop_fs_relation_command +What jobs are these (collect_limit, execute_insert_into_hadoop_fs_relation_command ) ? +How do we get the lineage listener to post only our job (xyz) ?

+ + + +
+ 👍 Pradeep S +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-06-08 11:00:41
 +
+

*Thread Reply:* Those jobs are actually what Spark does underneath 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-06-08 11:00:57
 +
+

*Thread Reply:* Are you using Delta Lake btw?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Moiz + (moiz.groups@gmail.com) +
+
2022-06-08 12:02:39
 +
+

*Thread Reply:* No, this is not Delta Lake. It is a normal Spark app .

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
raghanag + (raghanag@gmail.com) +
+
2022-06-08 13:58:05
 +
+

*Thread Reply:* @Maciej Obuchowski i think David posted about this before. https://openlineage.slack.com/archives/C01CK9T7HKR/p1636011698055200

+
+ + +
+ + + } + + David Virgil + (https://openlineage.slack.com/team/U02K9U58X7F) +
+ + + +
+
+ <p>Hello openlineage community.</p> + +<p>Yesterday I tried the integration with spark.</p> + +<p>The result was not satisfactory. This is what I did:</p> + +<ol><li>Add openlineage-spark dependency</li><li>Add these lines: +<code>.config("spark.jars.packages", "io.openlineage:openlineage_spark:0.3.1") +.config("spark.extraListeners", "io.openlineage.spark.agent.OpenLineageSparkListener") +.config("spark.openlineage.url", "&lt;https://marquez-internal-eks.eu-west-1.dev.hbi.systems/api/v1/namespaces/spark_integration/&gt;"</code> +This job was doing spark.read from 2 different json location. +It is doing spark write to 5 different parquet location in s3. +The job finished succesfully and the result in marquez is:</li> +</ol> +
+ + + + + + +
Original URL: https://openlineage.slack.com/archives/C01CK9T7HKR/p1636011698055200
+ + +
+ + Slack Conversation +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-06-08 14:27:46
 +
+

*Thread Reply:* I agree that it looks bad on UI, but I also think integration is going good job here. The eventual "aggregation" should be done by event consumer.

+ +

If anything, we should filter some 'useless' nodes like collect_limit since they add nothing.

+ +

We have an issue for doing this to specifically delta lake operations, as they are the biggest offenders: https://github.com/OpenLineage/OpenLineage/issues/628

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/issues/628|#628 [INTEGRATION][SPARK] Filter events for Delta operations>
+
+ <p>Delta operations in Spark SQL generate a ton of excess run events. There are two general causes for this.</p> + +<ol><li>Delta reads and writes Snapshot and delta files during execution - it access these files as DataFrames, which causes any access to be treated as a new SparkSQL execution - see the <a href="https://github.com/delta-io/delta/blob/master/core/src/main/scala/org/apache/spark/sql/delta/Snapshot.scala#L429-L457">source code</a>.</li><li>Delta writes start a new Spark SQLExecution separate from the original plan - this query references only the original query, not the write command. It converts the query to an RDD, then executes a new RDD job that writes to the target path.</li> +</ol> + +<p>Examples of the above two behaviors:</p> + +<p>The following events are generated by running the following Spark code, reading som JSON from a local file and writing it to a delta table</p> + +<pre><code> Dataset&amp;lt;Row&amp;gt; df = spark.read().json("file://" + testFile.toAbsolutePath()); + + df.filter("age &amp;gt; 100").write().format("delta").mode("append").save(deltaDir.toString()); +</code></pre> + +<hr /> + +<p><strong>Snapshot Operations</strong></p> + +<p>Abbreviated event for Snapshot query - note the input path includes <code>deltaTable/_delta_log</code>. All delta log file operations seem to generate a query similar to this.</p> + +<p>``` +{ + "eventType": "START", + "eventTime": "2021-01-01T00:00:00Z", + "run": { + "runId": "453ddea0-0f36-4eae-b420-d1b3eb3d232f", + "facets": { + "parent": { + "<em>producer": "<a href="https://github.com/OpenLineage/OpenLineage/tree/0.7.0-SNAPSHOT/integration/spark">https://github.com/OpenLineage/OpenLineage/tree/0.7.0-SNAPSHOT/integration/spark</a>", + "</em>schemaURL": "<a href="https://openlineage.io/spec/facets/1-0-0/ParentRunFacet.json#/$defs/ParentRunFacet">https://openlineage.io/spec/facets/1-0-0/ParentRunFacet.json#/$defs/ParentRunFacet</a>", + "run": { + "runId": "e6018ec7-a314-439c-888b-ddd63e927c05" + }, + "job": { + "namespace": "Namespace", + "name": "ParentJob" + } + }, + "spark<em>version": { + "</em>producer": "<a href="https://github.com/OpenLineage/OpenLineage/tree/0.7.0-SNAPSHOT/integration/spark">https://github.com/OpenLineage/OpenLineage/tree/0.7.0-SNAPSHOT/integration/spark</a>", + "<em>schemaURL": "<a href="https://openlineage.io/spec/1-0-2/OpenLineage.json#/$defs/RunFacet">https://openlineage.io/spec/1-0-2/OpenLineage.json#/$defs/RunFacet</a>", + "spark-version": "3.1.2" + }, + "spark.logicalPlan": { + "</em>producer": "<a href="https://github.com/OpenLineage/OpenLineage/tree/0.7.0-SNAPSHOT/integration/spark">https://github.com/OpenLineage/OpenLineage/tree/0.7.0-SNAPSHOT/integration/spark</a>", + "<em>schemaURL": "<a href="https://openlineage.io/spec/1-0-2/OpenLineage.json#/$defs/RunFacet">https://openlineage.io/spec/1-0-2/OpenLineage.json#/$defs/RunFacet</a>", + "plan": [ + { + "class": "org.apache.spark.sql.catalyst.plans.logical.Aggregate", + "num-children": 1, + "groupingExpressions": [], + "aggregateExpressions": [...], + "child": 0 + }, + { + "class": "org.apache.spark.sql.catalyst.plans.logical.Project", + "num-children": 1, + "projectList": [...], + "child": 0 + }, + { + "class": "org.apache.spark.sql.execution.LogicalRDD", + "num-children": 0, + "output": [...], + "rdd": null, + "outputPartitioning": { + "product-class": "org.apache.spark.sql.catalyst.plans.physical.UnknownPartitioning", + "numPartitions": 0 + }, + "outputOrdering": [], + "isStreaming": false, + "session": null + } + ] + }, + "spark</em>unknown": { + "<em>producer": "<a href="https://github.com/OpenLineage/OpenLineage/tree/0.7.0-SNAPSHOT/integration/spark">https://github.com/OpenLineage/OpenLineage/tree/0.7.0-SNAPSHOT/integration/spark</a>", + "</em>schemaURL": "<a href="https://openlineage.io/spec/1-0-2/OpenLineage.json#/$defs/RunFacet">https://openlineage.io/spec/1-0-2/OpenLineage.json#/$defs/RunFacet</a>", + "output": { + "description": { + "@class": "org.apache.spark.sql.catalyst.plans.logical.Aggregate", + "serializedId": 1, + "groupingExpressions": [], + "aggregateExpressions": [...], + "child": { + "@class": "org.apache.spark.sql.catalyst.plans.logical.Project", + "serializedId": 173, + "projectList": [ + 59, + 73, + 142, + 35, + 5 + ], + "child": { + "@class": "org.apache.spark.sql.execution.LogicalRDD", + "serializedId": 174, + "output": [...], + "rdd": { + "@class": "org.apache.spark.rdd.MapPartitionsRDD", + "serializedId": 179, + "prev": { + "@class": "org.apache.spark.sql.execution.SQLExecutionRDD", + "serializedId": 180, + "sqlRDD": { + "@class": "org.apache.spark.rdd.MapPartitionsRDD", + "serializedId": 181, + "prev": { + "@class": "org.apache.spark.rdd.MapPartitionsRDD", + "serializedId": 182, + "prev": { + "@class": "org.apache.spark.rdd.MapPartitionsRDD", + "serializedId": 183, + "prev": { + "@class": "org.apache.spark.rdd.MapPartitionsRDD", + "serializedId": 184, + "prev": { + "@class": "org.apache.spark.sql.execution.ShuffledRowRDD", + "serializedId": 185, + "dependency": { + ..., + "rdd": { + "@class": "org.apache.spark.rdd.MapPartitionsRDD", + "serializedId": 191, + "prev": { + "@class": "org.apache.spark.rdd.MapPartitionsRDD", + "serializedId": 192, + "prev": { + "@class": "org.apache.spark.rdd.MapPartitionsRDD", + "serializedId": 193, + "prev": { + "@class": "org.apache.spark.rdd.MapPartitionsRDD", + "serializedId": 194, + "prev": { + "@class": "org.apache.spark.rdd.MapPartitionsRDD", + "serializedId": 195, + "prev": { + "@class": "org.apache.spark.sql.execution.datasources.FileScanRDD", + "serializedId": 196, + "readFunction": { + "serializedId": 197 + }, + ... + }, + ... + }, + ... + }, + ... + }, + ... + }, + ... + } + }, + ... + }, + ... + }, + ... + }, + ... + }, + ... + }, + ... + }, + ... + }, + ... + }, + ... + }, + ... + }, + "inputAtt…</p> +
+ +
+
Milestone
+ <a href="https://github.com/OpenLineage/OpenLineage/milestone/6">0.10.0</a> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+ 👍 George Zachariah V +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
raghanag + (raghanag@gmail.com) +
+
2022-06-08 14:33:09
 +
+

*Thread Reply:* @Maciej Obuchowski but we only see these 2 jobs in the namespace, no other jobs were part of the lineage metadata, are we doing something wrong?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
raghanag + (raghanag@gmail.com) +
+
2022-06-08 16:09:15
 +
+

*Thread Reply:* @Michael Robinson On this note, may we know how to form a lineage if we have different set of API's before calling the spark job (already integrated with OpenLineageSparkListener), we want to see how the different set of params pass thru these components before landing into the spark job. If we use openlineage client to post the lineage events into the Marquez, do we need to mention the same Run UUID across the lineage events for the run or is there any other way to do this? Can you pls advise?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-06-08 22:51:38
 +
+

*Thread Reply:* I think I understand what you are asking -

+ +

The runID is used to correlate different state updates (i.e., start, fail, complete, abort) across the lifespan of a run. So if you are trying to add additional metadata to the same job run, you’d use the same runID.

+ +

So you’d generate a runID and send a START event, then in the various components you could send OTHER events containing the same runID + params you want to study in facets, then at the end you would send a COMPLETE.

+ +

(I think there should be an UPDATE event type in the spec for this sort of thing.)

+ + + +
+ 👍 George Zachariah V, raghanag +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
raghanag + (raghanag@gmail.com) +
+
2022-06-08 22:59:39
 +
+

*Thread Reply:* thanks @Ross Turk but what i am looking for is lets say for example, if we have 4 components in the system then we want to show the 4 components as job icons in the graph and the datasets between them would show the input/output parameters that these components use. +A(job) --> DS1(dataset) --> B(job) --> DS2(dataset) --> C(job) --> DS3(dataset) --> D(job)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-06-08 23:04:37
 +
+

*Thread Reply:* then you would need to have separate Jobs for each, with inputs and outputs defined

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-06-08 23:06:03
 +
+

*Thread Reply:* so there would be a Run of job B that shows DS1 as an input and DS2 as an output

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
raghanag + (raghanag@gmail.com) +
+
2022-06-08 23:06:18
 +
+

*Thread Reply:* got it

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-06-08 23:06:34
 +
+

*Thread Reply:* (fyi: I know openlineage but my understanding stops at spark 😄)

+ + + +
+ 👍 raghanag +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sharanya Santhanam + (santhanamsharanya@gmail.com) +
+
2022-06-10 12:27:58
 +
+

*Thread Reply:* > The eventual “aggregation” should be done by event consumer. +@Maciej Obuchowski Are there any known client side libraries that support this aggregation already ? In case of spark applications running as part of ETL pipelines, most of the times our end user is interested in seeing only the aggregated view where all jobs spawned as part of a single application are rolled up into 1 job.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-06-10 12:32:14
 +
+

*Thread Reply:* I believe Microsoft @Will Johnson has something similar to that, but it's probably proprietary.

+ +

We'd love to have something like it, but AFAIK it affects only some percentage of Spark jobs and we can only do so much.

+ +

With exception of Delta Lake/Databricks, where it affects every job, and we know some nodes that could be safely filtered client side.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-06-11 23:38:27
 +
+

*Thread Reply:* @Maciej Obuchowski Microsoft ❤️ OSS!

+ +

Apache Atlas doesn't have the same model as Marquez. It only knows of effectively one entity that represents the complete asset.

+ +

@Mark Taylor designed this solution available now on Github to consolidate OpenLineage messages

+ +

https://github.com/microsoft/Purview-ADB-Lineage-Solution-Accelerator/blob/d6514f2[…]/Function.Domain/Helpers/OlProcessing/OlMessageConsolodation.cs

+ +

In addition, we do some filtering only based on inputs and outputs to limit the messages AFTER it has been emitted.

+
+ + + + +
<https://github.com/microsoft/Purview-ADB-Lineage-Solution-Accelerator/blob/d6514f25c7f96151a197429efb9b7d5d425d0776/function-app/adb-to-purview/src/Function.Domain/Helpers/OlProcessing/OlMessageConsolodation.cs | OlMessageConsolodation.cs>
+
+ <pre><code> public class OlMessageConsolodation : IOlMessageConsolodation +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/microsoft/Purview-ADB-Lineage-Solution-Accelerator">microsoft/Purview-ADB-Lineage-Solution-Accelerator</a> +
+ + +
+ + + +
+ 🙌 Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sharanya Santhanam + (santhanamsharanya@gmail.com) +
+
2022-06-19 09:37:06
 +
+

*Thread Reply:* thank you !

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-06-08 10:54:32
 +
+

@channel The next OpenLineage TSC meeting is tomorrow! https://openlineage.slack.com/archives/C01CK9T7HKR/p1654093173961669

+
+ + +
+ + + } + + Michael Robinson + (https://openlineage.slack.com/team/U02LXF3HUN7) +
+ + + +
+
+ <p>@channel The next OpenLineage Technical Steering Committee meeting is on <strong>Thursday, June 9 at 10 am PT</strong>. Join us on Zoom: <a href="https://us02web.zoom.us/j/81831865546?pwd=RTladlNpc0FTTDlFcWRkM2JyazM4Zz09">https://us02web.zoom.us/j/81831865546?pwd=RTladlNpc0FTTDlFcWRkM2JyazM4Zz09</a> +All are welcome! +Agenda:</p> + +<ol><li>a recent blog post about Snowflake</li><li>the Great Expectations integration</li><li>the dbt integration</li><li>Open discussion +Notes: <a href="https://tinyurl.com/openlineagetsc">https://tinyurl.com/openlineagetsc</a> +Is there a topic you think the community should discuss at this or a future meeting? DM me to add items to the agenda.</li> +</ol> +
+ + + + + + +
Original URL: https://openlineage.slack.com/archives/C01CK9T7HKR/p1654093173961669
+ + +
+ + Slack Conversation +
+ + +
+ + + +
+ 👍 Maciej Obuchowski, Sheeri Cabral (Collibra), Willy Lulciuc, raghanag, Mardaunt +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Moravec + (jkb.moravec@gmail.com) +
+
2022-06-09 13:04:00
 +
+

*Thread Reply:* Hi, is the link correct? The meeting room is empty

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-06-09 16:04:23
 +
+

*Thread Reply:* sorry about that, thanks for letting us know

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mark Beebe + (mark_j_beebe@progressive.com) +
+
2022-06-13 15:13:59
 +
+

Hello all, after sending dbt openlineage events to Marquez, I am now looking to use the Marquez API to extract the lineage information. I am able to use python requests to call the Marquez API to get other information such as namespaces, datasets, etc., but I am a little bit confused about what I need to enter to get the lineage. I included screenshots for what the API reference shows regarding retrieving the lineage where it shows that a nodeId is required. However, this is where I seem to be having problems. It is not exactly clear where the nodeId needs to be set or what the nodeId needs to include. I would really appreciate any insights. Thank you!

+ +
+
+ + + + + + +
+
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-06-13 18:49:37
 +
+

*Thread Reply:* Hey @Mark Beebe!

+ +

In this case, nodeId is going to be either a dataset or a job. You need to tell Marquez where to start since there is likely to be more than one graph. So you need to get your hands on an identifier for that starting node.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-06-13 18:50:07
 +
+

*Thread Reply:* You can do this in a few ways (that I can think of). First, by looking for a namespace, then querying for the datasets in that namespace:

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-06-13 18:53:43
 +
+

*Thread Reply:* Or you can search, if you know the name of the dataset:

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-06-13 18:53:54
 +
+

*Thread Reply:* aaaaannnnd that’s actually all the ways I can think of.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mark Beebe + (mark_j_beebe@progressive.com) +
+
2022-06-14 08:11:30
 +
+

*Thread Reply:* That worked, thank you so much!

+ + + +
+ 👍 Ross Turk +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Varun Singh + (varuntestaz@outlook.com) +
+
2022-06-14 05:52:39
 +
+

Hi all, I need to send the lineage information from spark integration directly to a kafka topic. Java client seems to have a KafkaTransport, is it planned to have this support from inside the spark integration as well?

+ + + +
+ 👀 Francis McGregor-Macdonald +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-06-14 10:35:48
 +
+

Hi all, I’m working on a blog post about the Spark integration and would like to credit @tnazarew and @Sbargaoui for their contributions. Anyone know these contributors’ names? Are you on here? Thanks for any leads.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-06-14 10:37:01
 +
+

*Thread Reply:* tnazarew - Tomasz Nazarewicz

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-06-14 10:37:14
 +
+

*Thread Reply:* 🙌

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-06-15 12:46:45
 +
+

*Thread Reply:* 👍

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Conor Beverland + (conorbev@gmail.com) +
+
2022-06-14 13:58:07
 +
+

Has anyone tried getting the OpenLineage Spark integration working with GCP Dataproc ?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Peter Hanssens + (peter@cloudshuttle.com.au) +
+
2022-06-15 15:49:17
 +
+

Hi Folks, +DataEngBytes is a community data engineering conference here in Australia and will be hosted on the 27th and 29th of September. Our CFP is open for just under a month and tickets are on sale now: +Call for paper: https://sessionize.com/dataengbytes-2022/ +Tickets: https://www.tickettailor.com/events/dataengbytes/713307 +Promo video +https://youtu.be/1HE_XNLvHss

+
+
sessionize.com
+ + + +
DataEngBytes 2022: Call for Speakers/Papers
+
+ DataEngBytes! is a community conference centred on real world data engineering problems and solutions.We will have two days of in-person conferences i... +
+ + + + + + +
Original URL: https://sessionize.com/dataengbytes-2022/
+ + + +
+
+
tickettailor.com
+ + + +
Buy tickets – DataEngBytes 2022, Sydney! – Sydney, Thu 29 Sep 2022 9:00 AM - 5:00 PM
+
+ DataEngBytes 2022, Sydney! – Sydney, Thu 29 Sep 2022 - DataEngBytes! is a community conference centred on real world data engineering problems and solutions. We will have two days of in-person conferences in Melbourne (on the 27th) and Sydney (on the 29th) of September with both days broadcast online on our YouTube ... +
+ + + + + + + + +
Original URL: https://www.tickettailor.com/events/dataengbytes/713307
+ + + +
+
+
YouTube
+ +
+ + + } + + DataEngAU + (https://www.youtube.com/c/DataEngAU) +
+ + + +
DataEngBytes 2022 Announcement
+
+ +
+ + + + + + + + +
Original URL: https://youtu.be/1HE_XNLvHss
+ + + +
+ + + +
+ 👀 Ross Turk, Michael Collado +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-06-17 16:23:32
 +
+

A release of OpenLineage has been requested pending the merging of #856. Three +1s will authorize a release today. +@Willy Lulciuc @Michael Collado @Ross Turk @Maciej Obuchowski @Paweł Leszczyński @Mandy Chessell @Daniel Henneberger @Drew Banin @Julien Le Dem @Ryan Blue @Will Johnson @Zhamak Dehghani

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/pull/856|#856 Fix spark casting error and session catalog support for `iceberg`>
+
+ <p><strong>Problem</strong></p> + +<p>Class cast exception for <code>DataSourceV2ScanRelation</code>:</p> + +<pre><code>java.lang.ClassCastException: class org.apache.spark.sql.execution.datasources.v2.DataSourceV2ScanRelation cannot be cast to class org.apache.spark.sql.execution.datasources.v2.DataSourceV2Relation (org.apache.spark.sql.execution.datasources.v2.DataSourceV2ScanRelation and org.apache.spark.sql.execution.datasources.v2.DataSourceV2Relation are in unnamed module of loader 'app') + at io.openlineage.spark3.agent.lifecycle.plan.TableContentChangeDatasetBuilder.apply(TableContentChangeDatasetBuilder.java:83) ~[openlineage-spark-0.9.0.jar:0.9.0] + at io.openlineage.spark.api.AbstractQueryPlanDatasetBuilder.apply(AbstractQueryPlanDatasetBuilder.java:92) ~[openlineage-spark-0.9.0.jar:0.9.0] + at io.openlineage.spark.api.AbstractQueryPlanDatasetBuilder$1.apply(AbstractQueryPlanDatasetBuilder.java:86) ~[openlineage-spark-0.9.0.jar:0.9.0] + at io.openlineage.spark.api.AbstractQueryPlanDatasetBuilder$1.apply(AbstractQueryPlanDatasetBuilder.java:77) ~[openlineage-spark-0.9.0.jar:0.9.0] + at io.openlineage.spark.api.AbstractQueryPlanDatasetBuilder.lambda$apply$0(AbstractQueryPlanDatasetBuilder.java:67) ~[openlineage-spark-0.9.0.jar:0.9.0] + at java.util.Optional.map(Optional.java:265) ~[?:?] + . + . + +</code></pre> + +<p>Unsupported spark catalog session:</p> + +<pre><code>ERROR PlanUtils3: Catalog org.apache.iceberg.spark.SparkSessionCatalog is unsupported +io.openlineage.spark3.agent.lifecycle.plan.catalog.UnsupportedCatalogException: org.apache.iceberg.spark.SparkSessionCatalog + at io.openlineage.spark3.agent.lifecycle.plan.catalog.CatalogUtils3.lambda$getDatasetIdentifier$2(CatalogUtils3.java:50) ~[openlineage-spark-0.9.0.jar:0.9.0] + at java.util.Optional.orElseThrow(Optional.java:408) ~[?:?] + at io.openlineage.spark3.agent.lifecycle.plan.catalog.CatalogUtils3.getDatasetIdentifier(CatalogUtils3.java:50) ~[openlineage-spark-0.9.0.jar:0.9.0] + at io.openlineage.spark3.agent.lifecycle.plan.catalog.CatalogUtils3.getDatasetIdentifier(CatalogUtils3.java:36) ~[openlineage-spark-0.9.0.jar:0.9.0] + at io.openlineage.spark3.agent.utils.PlanUtils3.getDatasetIdentifier(PlanUtils3.java:67) ~[openlineage-spark-0.9.0.jar:0.9.0] + at io.openlineage.spark3.agent.utils.PlanUtils3.fromDataSourceV2Relation(PlanUtils3.java:110) ~[openlineage-spark-0.9.0.jar:0.9.0] + . + . + +</code></pre> + +<p><strong>Solution</strong></p> + +<p>Fix class cast exception for <code>DataSourceV2ScanRelation</code> and add support for <code>org.apache.iceberg.spark.SparkSessionCatalog</code>.</p> + +<p><strong>Checklist</strong></p> + +<p>☑︎ You've <a href="https://github.com/OpenLineage/OpenLineage/blob/main/why-the-dco.md">signed-off</a> your work +☑︎ Your pull request title follows our <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CONTRIBUTING.md#creating-pull-requests">guidelines</a> +☐ Your changes are accompanied by tests (<em>if relevant</em>) +☑︎ Your change contains a <a href="https://kurtisnusbaum.medium.com/stacked-diffs-keeping-phabricator-diffs-small-d9964f4dcfa6">small diff</a> and is self-contained +☐ You've updated any relevant documentation (<em>if relevant</em>) +☐ You've updated the <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md"><code>CHANGELOG.md</code></a> with details about your change under the "Unreleased" section (<em>if relevant, depending on the change, this may not be necessary</em>) +☐ You've versioned the core OpenLineage model or facets according to <a href="https://docs.snowplowanalytics.com/docs/pipeline-components-and-applications/iglu/common-architecture/schemaver">SchemaVer</a> (<em>if relevant</em>)</p> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+ ➕ Willy Lulciuc, Maciej Obuchowski, Michael Collado +
+ +
+ ✅ Michael Collado +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Chase Christensen + (christensenc3526@gmail.com) +
+
2022-06-22 17:09:18
 +
+

👋 Hi everyone!

+ + + +
+ 👋 Conor Beverland, Ross Turk, Maciej Obuchowski, Michael Robinson, George Zachariah V, Willy Lulciuc, Dinakar Sundar +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Lee + (chenzuoli709@gmail.com) +
+
2022-06-23 21:54:05
 +
+

hi

+ + + +
+ 👋 Maciej Obuchowski, Sheeri Cabral (Collibra), Willy Lulciuc, Michael Robinson, Dinakar Sundar +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-06-25 07:34:32
 +
+

@channel OpenLineage 0.10.0 is now available! We added SnowflakeOperatorAsync extractor support to the Airflow integration, an InMemoryRelationInputDatasetBuilder for InMemory datasets to the Spark integration, a static code analysis tool to run in CircleCI on Python modules, a copyright to all source files, and a debugger called PMD to the build process. +Changes we made include skipping FunctionRegistry.class serialization in the Spark integration, installing the new rust-based SQL parser by default in the Airflow integration, improving the integration tests for the Airflow integration, reducing event payload size by excluding local data and including an output node in start events, and splitting the Spark integration into submodules. +Thanks to all the contributors who made this release possible! +Release: https://github.com/OpenLineage/OpenLineage/releases/tag/0.10.0 +Changelog: https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md +Commit history: https://github.com/OpenLineage/OpenLineage/compare/0.9.0...0.10.0 +Maven: https://oss.sonatype.org/#nexus-search;quick~openlineage +PyPI: https://pypi.org/project/openlineage-python/

+ + + +
+ 🙌 Maciej Obuchowski, Filipe Comparini Vieira, Manuel, Dinakar Sundar, Ross Turk, Paweł Leszczyński, Willy Lulciuc, Adisesha Reddy G, Conor Beverland, Francis McGregor-Macdonald, Jam Car +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mike brenes + (brenesmi@gmail.com) +
+
2022-06-28 18:29:29
 +
+

Why has put dataset been deprecated? How do I add an initial data set via api?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2022-06-28 18:39:16
 +
+

*Thread Reply:* I think you’re reference the deprecation of the DatasetAPI in Marquez? A milestone for the Marquez is to only collect metadata via OpenLineage events. This includes metadata for datasets , jobs , and runs . The DatasetAPI won’t be removed until support for collecting dataset metadata via OpenLineage has been added, see https://github.com/OpenLineage/OpenLineage/issues/323

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/issues/323|#323 [PROPOSAL] Ability to annotate datasets with metadata without a job run>
+
+ <p><strong>Purpose:</strong> <br /> +We need a way to make observations about a Dataset current state outside of a run reading or writing the data. <br /> +For example, this could be:</p> + +<p>• collecting schema information about a table from a warehouse +• informing that a table has been deleted +• etc</p> + +<p><strong>Proposed implementation</strong> <br /> +Adding Dataset event types that are not lineage events. This would be the Dataset entity with its facets: </p> + +<p><a href="https://github.com/OpenLineage/OpenLineage/blob/3a1ccbd854bbf202bbe6437bf81786cb01ed4d01/spec/OpenLineage.json#L178-L203">OpenLineage/spec/OpenLineage.json</a></p> + +<p>Lines 178 to 203 in &lt;/OpenLineage/OpenLineage/commit/3a1ccbd854bbf202bbe6437bf81786cb01ed4d01|3a1ccbd&gt;</p> + +<pre><code>"Dataset": { + "type": "object", + "properties": { + "namespace": { + "description": "The namespace containing that dataset", + "type": "string", + "example": "my-datasource-namespace" + }, + "name": { + "description": "The unique name for that dataset within that namespace", + "type": "string", + "example": "instance.schema.table" + }, + "facets": { + "description": "The facets for this dataset", + "type": "object", + "additionalProperties": { + "$ref": "#/$defs/DatasetFacet" + } + } + }, + "required": [ + "namespace", + "name" + ] + } + +</code></pre> +
+ +
+
Assignees
+ <a href="https://github.com/mobuchowski">@mobuchowski</a> +
+ +
+
Labels
+ proposal +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2022-06-28 18:40:28
 +
+

*Thread Reply:* Once the spec supports dataset metadata, we’ll outline steps in the Marquez project to switch to using the new dataset event type

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2022-06-28 18:43:20
 +
+

*Thread Reply:* The DatasetAPI was also deprecated to avoid confusion around which API to use

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mike brenes + (brenesmi@gmail.com) +
+
2022-06-28 18:41:38
 +
+

🥺

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mike brenes + (brenesmi@gmail.com) +
+
2022-06-28 18:42:21
 +
+

So how would you propose I create the initial node if I am trying to do a POC?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2022-06-28 18:44:49
 +
+

*Thread Reply:* Do you want to register just datasets? Or are you extracting metadata for a job that would include input / output datasets? (outside of Airflow of course)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mike brenes + (brenesmi@gmail.com) +
+
2022-06-28 18:45:09
 +
+

*Thread Reply:* Sorry didn't notice you over here ! lol

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mike brenes + (brenesmi@gmail.com) +
+
2022-06-28 18:45:53
 +
+

*Thread Reply:* So ideally I would like to map out our current data flow from on prem to aws

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2022-06-28 18:47:39
 +
+

*Thread Reply:* What do you mean by mapping to AWS? Like send OL events to a service on AWS that would process the lineage metadata?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mike brenes + (brenesmi@gmail.com) +
+
2022-06-28 18:48:14
 +
+

*Thread Reply:* no, just visualize the current migration flow.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2022-06-28 18:48:53
 +
+

*Thread Reply:* Ah I see, youre doing a infra migration from on prem to AWS 👌

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mike brenes + (brenesmi@gmail.com) +
+
2022-06-28 18:49:08
 +
+

*Thread Reply:* really AWS is irrelevant. Source sink -> migration scriipts -> s3 -> additional processing -> final sink

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mike brenes + (brenesmi@gmail.com) +
+
2022-06-28 18:49:19
 +
+

*Thread Reply:* correct

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2022-06-28 18:49:45
 +
+

*Thread Reply:* right right. so you want to map out that flow and visualize it in Marquez? (or some other meta service)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mike brenes + (brenesmi@gmail.com) +
+
2022-06-28 18:50:05
 +
+

*Thread Reply:* yes

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mike brenes + (brenesmi@gmail.com) +
+
2022-06-28 18:50:26
 +
+

*Thread Reply:* which I think I can do once the first nodes exist

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mike brenes + (brenesmi@gmail.com) +
+
2022-06-28 18:51:18
 +
+

*Thread Reply:* But I don't know how to get that initial node. I tried using the input facet at job start , that didn't do it. I also can't get the sql context that is in these examples.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mike brenes + (brenesmi@gmail.com) +
+
2022-06-28 18:51:54
 +
+

*Thread Reply:* really just want to re-create food_devlivery using my own biz context

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2022-06-28 18:52:14
 +
+

*Thread Reply:* Have you looked over our workshops and this example? (assuming you’re using python?)

+
+ + + + +
<https://github.com/OpenLineage/workshops/blob/main/airflow/e2-lineage-api/python/generate-events.py | generate-events.py>
+
+ <pre><code>#!/usr/bin/env python3 + +from openlineage.client.run import RunEvent, RunState, Run, Job, Dataset +from openlineage.client.client import OpenLineageClient +from datetime import datetime +from uuid import uuid4 + +# Initialize the OpenLineage client +client = OpenLineageClient.from_environment() + +# Specify the producer of this lineage metadata +producer = "&lt;https://github.com/OpenLineage/workshops&gt;" + +# Create some basic Dataset objects for our fictional pipeline +monthly_summary = Dataset(namespace="&lt;postgres://workshop-db:None&gt;", name="workshop.public.monthly_summary") +commissions = Dataset(namespace="&lt;postgres://workshop-db:None&gt;", name="workshop.public.commissions") +taxes = Dataset(namespace="&lt;postgres://workshop-db:None&gt;", name="workshop.public.taxes") + +# Create a Job object +job = Job(namespace="workshop", name="monthly_accounting") + +# Create a Run object with a unique ID +run = Run(str(uuid4())) + +# Emit a START run event +client.emit( + RunEvent( + RunState.START, + datetime.now().isoformat(), + run, job, producer + ) +) + +# +# This is where our application would do the actual work :) +# + +# Emit a COMPLETE run event +client.emit( + RunEvent( + RunState.COMPLETE, + datetime.now().isoformat(), + run, job, producer, + inputs=[monthly_summary], + outputs=[commissions, taxes], + ) +) + +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/workshops">OpenLineage/workshops</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2022-06-28 18:53:49
 +
+

*Thread Reply:* that goes over the py client with some OL examples, but really calling openlineage.emit(...) method with RunEvents and specifying Marquez as the backend will get you up and running!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2022-06-28 18:54:32
 +
+

*Thread Reply:* Don’t forget to configure the transport for the client

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mike brenes + (brenesmi@gmail.com) +
+
2022-06-28 18:54:45
 +
+

*Thread Reply:* sweet. Thank you! I'll take a look. Also.. Just came across datakin for the first time. very nice 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2022-06-28 18:55:25
 +
+

*Thread Reply:* thanks! …. but we’re now part of astronomer.io 😉

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2022-06-28 18:55:48
 +
+

*Thread Reply:* making airflow oh-so-easy-to-use one DAG at a time

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mike brenes + (brenesmi@gmail.com) +
+
2022-06-28 18:55:52
 +
+

*Thread Reply:* saw that too !

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2022-06-28 18:56:03
 +
+

*Thread Reply:* you’re on top of it!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mike brenes + (brenesmi@gmail.com) +
+
2022-06-28 18:56:28
 +
+

*Thread Reply:* ha. Thanks again!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mike brenes + (brenesmi@gmail.com) +
+
2022-06-28 18:42:40
 +
+

This would be outside of Airflow

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Fenil Doshi + (fdoshi@salesforce.com) +
+
2022-06-28 18:43:22
 +
+

Hello, +Is OpenLineage planning to add support for inlets and outlets for Airflow integration? I am working on a project that relies on it and was hoping to contribute to this feature if its something that is in the talks. +I saw an open issue here

+ +

I am willing to work on it. My plan was to just support Files and Tables entities (for inlets and outlets). +Pass the inlets and outlets info into extract_metadata function here and then convert Airflow entities into TaskMetaData entities here.

+ +

Does this sound reasonable?

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/issues/384|#384 [INTEGRATION][Airflow] Consider using manual inlets/outlets as a fallback for automatic extractor mechanism>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/integration/airflow/openlineage/lineage_backend/__init__.py | __init__.py>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2022-06-28 18:59:38
 +
+

*Thread Reply:* Honestly, I’ve been a huge fan of using / falling back on inlets and outlets since day 1. AND if you’re willing to contribute this support, you get a +1 from me (I’ll add some minor comments to the issue) /cc @Julien Le Dem

+ + + +
+ 🙌 Fenil Doshi +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2022-06-28 18:59:59
 +
+

*Thread Reply:* would be great to get @Maciej Obuchowski thoughts on this as well

+ + + +
+ 👍 Fenil Doshi +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Fenil Doshi + (fdoshi@salesforce.com) +
+
2022-07-08 12:40:39
 +
+

*Thread Reply:* I have created a draft PR for this here. +Please let me know if the changes make sense.

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/pull/914|#914 [INTEGRATION][AIRFLOW] Allow lineage metadata to flow through inlets and outlets>
+
+ <p>Signed-off-by: Fenil Doshi <a href="mailto:fdoshi@slack-corp.com">fdoshi@slack-corp.com</a></p> + +<p><strong>Problem</strong></p> + +<p>Currently, in Airflow integration, lineage is only captured via a few extractors which greatly limits its use case. Airflow allows users to manually annotate lineage for the tasks using inlets and outlets. These can be a great fallback mechanism when extractors cannot automatically extract the input and output Datasets. <br /> +The PR closes <a href="https://github.com/OpenLineage/OpenLineage/issues/384">#384</a></p> + +<p>Addresses: <a href="https://github.com/OpenLineage/OpenLineage/issues/384">#384</a></p> + +<p><strong>Solution</strong></p> + +<p>Airflow sends inlets and outlets information to <code>LineageBackend</code> class. Inlets and Outlets can be one of <a href="https://github.com/apache/airflow/blob/main/airflow/lineage/entities.py">these</a> entities. The code only currently supports <code>Table</code> entity because OpenLineage does not track column-level lineage yet.</p> + +<p>• Check the TaskMetadata object that is created after corresponding extractors extract the lineage data and see whether inputs and outputs exist +• Check whether inlets and outlets are provided for the task +• If TaskMetadata object does not have inputs/outputs, use Airflow's inlets and outlets. +• Get all the Table entities and convert them to Dataset entities that are suitable for TaskMetaData object +Your change modifies the <a href="https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.json">core</a> OpenLineage model +Your change modifies one or more OpenLineage <a href="https://github.com/OpenLineage/OpenLineage/tree/main/spec/facets">facets</a></p> + +<p><strong>Checklist</strong></p> + +<p>☑︎ You've <a href="https://github.com/OpenLineage/OpenLineage/blob/main/why-the-dco.md">signed-off</a> your work +☑︎ Your pull request title follows our <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CONTRIBUTING.md#creating-pull-requests">guidelines</a> +☐ Your changes are accompanied by tests (<em>if relevant</em>) +☑︎ Your change contains a <a href="https://kurtisnusbaum.medium.com/stacked-diffs-keeping-phabricator-diffs-small-d9964f4dcfa6">small diff</a> and is self-contained +☐ You've updated any relevant documentation (<em>if relevant</em>) +☐ You've updated the <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md"><code>CHANGELOG.md</code></a> with details about your change under the "Unreleased" section (<em>if relevant, depending on the change, this may not be necessary</em>) +☐ You've versioned the core OpenLineage model or facets according to <a href="https://docs.snowplowanalytics.com/docs/pipeline-components-and-applications/iglu/common-architecture/schemaver">SchemaVer</a> (<em>if relevant</em>)</p> +
+ +
+
Comments
+ 1 +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-07-08 12:42:30
 +
+

*Thread Reply:* I think this effort: https://github.com/OpenLineage/OpenLineage/pull/904 ultimately makes more sense, since it will allow getting lineage on Airflow 2.3+ too

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/pull/904|#904 [INTEGRATION][AIRFLOW] Manual inlets/outlets issue#384>
+
+ <p><strong>Problem</strong></p> + +<p>Addressing issue <a href="https://github.com/OpenLineage/OpenLineage/issues/384">#384</a>.</p> + +<p>• We want the ability to set inlets and outlets manually</p> + +<p><strong>Solution</strong></p> + +<p>• expand the <code>postgres_extractor</code> to except <code>_PythonDecoratedOperator</code>. +• add env check to attach <code>inputs</code> and <code>outputs</code> to <code>TaskMetadata</code> when <code>OPENLINEAGE_COLLECT_MANUALLY</code> is set to <code>true</code>. +• added <code>extract_inlets_and_outlets_to_dataset</code> function. +Your change modifies the <a href="https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.json">core</a> OpenLineage model +Your change modifies one or more OpenLineage <a href="https://github.com/OpenLineage/OpenLineage/tree/main/spec/facets">facets</a></p> + +<p>If you're contributing a new integration, please specify the scope of the integration and how/where it has been tested (e.g., Apache Spark integration supports <code>S3</code> and <code>GCS</code> filesystem operations, tested with AWS EMR).</p> + +<p><strong>Checklist</strong></p> + +<p>• [ x] You've <a href="https://github.com/OpenLineage/OpenLineage/blob/main/why-the-dco.md">signed-off</a> your work +• [x ] Your pull request title follows our <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CONTRIBUTING.md#creating-pull-requests">guidelines</a> +☐ Your changes are accompanied by tests (<em>if relevant</em>) +☐ Your change contains a <a href="https://kurtisnusbaum.medium.com/stacked-diffs-keeping-phabricator-diffs-small-d9964f4dcfa6">small diff</a> and is self-contained +☐ You've updated any relevant documentation (<em>if relevant</em>) +☐ You've updated the <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md"><code>CHANGELOG.md</code></a> with details about your change under the "Unreleased" section (<em>if relevant, depending on the change, this may not be necessary</em>) +☐ You've versioned the core OpenLineage model or facets according to <a href="https://docs.snowplowanalytics.com/docs/pipeline-components-and-applications/iglu/common-architecture/schemaver">SchemaVer</a> (<em>if relevant</em>)</p> +
+ +
+
Comments
+ 2 +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+ ✅ Fenil Doshi +
+ +
+ 👀 Fenil Doshi +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Fenil Doshi + (fdoshi@salesforce.com) +
+
2022-07-08 18:12:47
 +
+

*Thread Reply:* I have made the changes in-line to the mentioned comments here. +Does this look good?

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/pull/914|#914 [INTEGRATION][AIRFLOW] Allow lineage metadata to flow through inlets and outlets>
+
+ <p>Signed-off-by: Fenil Doshi <a href="mailto:fdoshi@slack-corp.com">fdoshi@slack-corp.com</a></p> + +<p><strong>Problem</strong></p> + +<p>Currently, in Airflow integration, lineage is only captured via a few extractors which greatly limits its use case. Airflow allows users to manually annotate lineage for the tasks using inlets and outlets. These can be a great fallback mechanism when extractors cannot automatically extract the input and output Datasets. <br /> +The PR closes <a href="https://github.com/OpenLineage/OpenLineage/issues/384">#384</a></p> + +<p>Addresses: <a href="https://github.com/OpenLineage/OpenLineage/issues/384">#384</a></p> + +<p><strong>Solution</strong></p> + +<p>Airflow sends inlets and outlets information to <code>LineageBackend</code> class. Inlets and Outlets can be one of <a href="https://github.com/apache/airflow/blob/main/airflow/lineage/entities.py">these</a> entities. The code only currently supports <code>Table</code> entity because OpenLineage does not track column-level lineage yet.</p> + +<p>• Check the TaskMetadata object that is created after corresponding extractors extract the lineage data and see whether inputs and outputs exist +• Check whether inlets and outlets are provided for the task +• If TaskMetadata object does not have inputs/outputs, use Airflow's inlets and outlets. +• Get all the Table entities and convert them to Dataset entities that are suitable for TaskMetaData object +Your change modifies the <a href="https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.json">core</a> OpenLineage model +Your change modifies one or more OpenLineage <a href="https://github.com/OpenLineage/OpenLineage/tree/main/spec/facets">facets</a></p> + +<p><strong>Checklist</strong></p> + +<p>☑︎ You've <a href="https://github.com/OpenLineage/OpenLineage/blob/main/why-the-dco.md">signed-off</a> your work +☑︎ Your pull request title follows our <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CONTRIBUTING.md#creating-pull-requests">guidelines</a> +☐ Your changes are accompanied by tests (<em>if relevant</em>) +☑︎ Your change contains a <a href="https://kurtisnusbaum.medium.com/stacked-diffs-keeping-phabricator-diffs-small-d9964f4dcfa6">small diff</a> and is self-contained +☐ You've updated any relevant documentation (<em>if relevant</em>) +☐ You've updated the <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md"><code>CHANGELOG.md</code></a> with details about your change under the "Unreleased" section (<em>if relevant, depending on the change, this may not be necessary</em>) +☐ You've versioned the core OpenLineage model or facets according to <a href="https://docs.snowplowanalytics.com/docs/pipeline-components-and-applications/iglu/common-architecture/schemaver">SchemaVer</a> (<em>if relevant</em>)</p> +
+ +
+
Comments
+ 1 +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-07-12 09:35:22
 +
+

*Thread Reply:* I think it looks good! Would be great to have tests for this feature though.

+ + + +
+ 👍 Fenil Doshi, Julien Le Dem +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Fenil Doshi + (fdoshi@salesforce.com) +
+
2022-07-15 21:56:50
 +
+

*Thread Reply:* I have added the tests! Would really appreciate it if someone can take a look and let me know if anything else needs to be done. +Thank you for the support! 😄

+ + + +
+ 👀 Willy Lulciuc, Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-07-18 06:48:03
 +
+

*Thread Reply:* One change and I think it will be good for now.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-07-18 06:48:07
 +
+

*Thread Reply:* Have you tested it manually?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Fenil Doshi + (fdoshi@salesforce.com) +
+
2022-07-20 13:22:04
 +
+

*Thread Reply:* Thanks a lot for the review! Appreciate it 🙌 +Yes, I tested it manually (for Airflow versions 2.1.4 and 2.3.3) and it works 🎉

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Conor Beverland + (conorbev@gmail.com) +
+
2022-07-20 13:24:55
 +
+

*Thread Reply:* I think this is such a useful feature to have, thank you! Would you mind adding a little example to the PR of how to use it? Like a little example DAG or something? ( either in a comment or edit the PR description )

+ + + +
+ 👍 Fenil Doshi +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Fenil Doshi + (fdoshi@salesforce.com) +
+
2022-07-20 15:20:32
 +
+

*Thread Reply:* Yes, Sure! I will add it in the PR description

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-07-21 05:30:56
 +
+

*Thread Reply:* I think it would be easy to convert to integration test then if you provided example dag

+ + + +
+ 👍 Fenil Doshi +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Conor Beverland + (conorbev@gmail.com) +
+
2022-07-27 12:20:43
 +
+

*Thread Reply:* ping @Fenil Doshi if possible I would really love to see the example DAG on there 🙂 🙏

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Fenil Doshi + (fdoshi@salesforce.com) +
+
2022-07-27 12:26:22
 +
+

*Thread Reply:* Yes, I was going to but the PR got merged so did not update the description. Should I just update the description of merged PR? Or should I add it somewhere in the docs?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Conor Beverland + (conorbev@gmail.com) +
+
2022-07-27 12:42:29
 +
+

*Thread Reply:* ^ @Ross Turk is it easy for @Fenil Doshi to contribute doc for manual inlet definition on the new doc site?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-07-27 12:48:32
 +
+

*Thread Reply:* It is easy 🙂 it's just markdown: https://github.com/openlineage/docs/

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-07-27 12:49:23
 +
+

*Thread Reply:* @Fenil Doshi feel free to create new page here and don't sweat where to put it, we'll still figuring the structure of it out and will move it then

+ + + +
+ 👍 Ross Turk, Fenil Doshi +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-07-27 13:12:31
 +
+

*Thread Reply:* exactly, yes - don’t be worried about the doc quality right now, the doc site is still in a pre-release state. so whatever you write will be likely edited or moved before it becomes official 👍

+ + + +
+ 👍 Fenil Doshi +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Fenil Doshi + (fdoshi@salesforce.com) +
+
2022-07-27 20:37:34
 +
+

*Thread Reply:* I added documentations here - https://github.com/OpenLineage/docs/pull/16

+ +

Also, have added an example for it. 🙂 +Let me know if something is unclear and needs to be updated.

+
+ + + + +
<https://github.com/OpenLineage/docs/pull/16|#16 Add documentation for openlineage-airflow integration via inlets-outlets>
+
+ <p>Add documentation and example for making changes to the Airflow task via inlets and outlets in order to facilitate openlineage to collect lineage metadata.</p> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/docs">OpenLineage/docs</a> +
+ + +
+ + + +
+ ✅ Conor Beverland +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Conor Beverland + (conorbev@gmail.com) +
+
2022-07-28 12:50:54
 +
+

*Thread Reply:* Thanks! very cool.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Conor Beverland + (conorbev@gmail.com) +
+
2022-07-28 12:52:22
 +
+

*Thread Reply:* Does Airflow check the types of the inlets/outlets btw?

+ +

Like I wonder if a user could directly define an OpenLineage DataSet ( which might even have various other facets included on it ) and specify it in the inlets/outlets ?

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/client/python/openlineage/client/run.py | run.py>
+
+ <pre><code>class Dataset(RedactMixin): +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-07-28 12:54:56
 +
+

*Thread Reply:* Yeah, I was also curious about using the models from airflow.lineage.entities as opposed to openlineage.client.run.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-07-28 12:55:42
 +
+

*Thread Reply:* I am accustomed to creating OpenLineage entities like this:

+ +

taxes = Dataset(namespace="<postgres://foobar>", name="schema.table")

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-07-28 12:56:45
 +
+

*Thread Reply:* I don’t dislike the airflow.lineage.entities models especially, but if we only support one of them…

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Conor Beverland + (conorbev@gmail.com) +
+
2022-07-28 12:58:18
 +
+

*Thread Reply:* yeah, if Airflow allows that class within inlets/outlets it'd be nice to support both imo.

+ +

Like we would suggest users to use openlineage.client.run.Dataset but if a user already has DAGs that use Table then they'd still work in a best efforts way.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-07-28 13:03:07
 +
+

*Thread Reply:* either Airflow depends on OpenLineage or we can probably change those entities as part of AIP-48 overhaul to more openlineage-like ones

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-07-28 17:18:35
 +
+

*Thread Reply:* hm, not sure I understand the dependency issue. isn’t this extractor living in openlineage-airflow?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Conor Beverland + (conorbev@gmail.com) +
+
2022-08-15 09:49:02
 +
+

*Thread Reply:* I gave manual lineage a try with native OL Datasets specified in the Airflow inlets/outlets and it seems to work! Had to make some small tweaks which I have attempted here: https://github.com/OpenLineage/OpenLineage/pull/1015

+ +

( I left the support for converting the Airflow Table to Dataset because I think that's nice to have also )

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/pull/1015|#1015 [Integration][Airflow] Support OL Datasets in manual lineage inputs/outputs>
+
+ <p>Signed-off-by: Conor Beverland <a href="mailto:conorbev@gmail.com">conorbev@gmail.com</a></p> + +<p><strong>Problem</strong></p> + +<p>The current support for manual lineage definition requires a user to create an Airflow <code>airflow.lineage.entities.Table</code> ( which is then converted to an OpenLineage Dataset ).</p> + +<p>It would be good if users could create OpenLineage Dataset classes directly in their DAGs with no special conversion necessary.</p> + +<p><strong>Solution</strong></p> + +<p>This extends the current implementation to simply pass through Datasets which are specified in inlets or outlets without modification.</p> + +<p>In addition it makes the BashOperatorExtractor behave more similarly to the PythonOperator when source code collection is disabled which allows it to work with the manual lineage collection feature.</p> + +<p>☐ Your change modifies the <a href="https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.json">core</a> OpenLineage model +☐ Your change modifies one or more OpenLineage <a href="https://github.com/OpenLineage/OpenLineage/tree/main/spec/facets">facets</a></p> + +<p><strong>Checklist</strong></p> + +<p>• [ x] You've <a href="https://github.com/OpenLineage/OpenLineage/blob/main/why-the-dco.md">signed-off</a> your work +• [ x] Your pull request title follows our <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CONTRIBUTING.md#creating-pull-requests">guidelines</a> +• [x ] Your changes are accompanied by tests (<em>if relevant</em>) +• [x ] Your change contains a <a href="https://kurtisnusbaum.medium.com/stacked-diffs-keeping-phabricator-diffs-small-d9964f4dcfa6">small diff</a> and is self-contained</p> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mike brenes + (brenesmi@gmail.com) +
+
2022-06-28 18:44:24
 +
+

food_delivery example example.etl_categories node

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mike brenes + (brenesmi@gmail.com) +
+
2022-06-28 18:44:40
 +
+

how do I recreate that using Openlineage?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2022-06-28 18:45:52
 +
+

*Thread Reply:* Ahh great question! I actually just updated the seeding cmd for Marquez to do just this (but in java of course)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2022-06-28 18:46:15
 +
+

*Thread Reply:* Give me a sec to send you over the diff…

+ + + +
+ ❤️ Mike brenes +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2022-06-28 18:56:35
 +
+

*Thread Reply:* … continued here https://openlineage.slack.com/archives/C01CK9T7HKR/p1656456734272809?thread_ts=1656456141.097229&cid=C01CK9T7HKR

+
+ + +
+ + + } + + Willy Lulciuc + (https://openlineage.slack.com/team/U01DCMDFHBK) +
+ + + +
+
+ <p>Have you looked over our <a href="https://github.com/OpenLineage/workshops">workshops</a> and <a href="https://github.com/OpenLineage/workshops/blob/main/airflow/e2-lineage-api/python/generate-events.py">this</a> example? (assuming you’re using python?)</p> +
+ + + + + + +
Original URL: https://openlineage.slack.com/archives/C01CK9T7HKR/p1656456734272809?thread_ts=1656456141.097229&amp;cid=C01CK9T7HKR
+ + +
+ + Thread in Slack Conversation +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Conor Beverland + (conorbev@gmail.com) +
+
2022-06-28 20:05:33
 +
+

I'm very new to DBT but wanted to give it a try with OL. I had a couple of questions when going through the DBT tutorial here: https://docs.getdbt.com/guides/getting-started/learning-more/getting-started-dbt-core

+ +
  1. An earlier part of the tutorial has you build a model in a single sql file: https://docs.getdbt.com/guides/getting-started/learning-more/getting-started-dbt-core#build-your-first-model When I did this and ran dbt-ol I got a lineage graph like this:
    2. +
+ + + +
+ 👀 Maciej Obuchowski +
+ +
+
+
+
+ + + + + + + +
+
+ + + + +
+ +
Conor Beverland + (conorbev@gmail.com) +
+
2022-06-28 20:07:11
 +
+

then a later part of the tutorial has you split that same example into multiple models and when I run it again I get the graph like:

+ + + +
+
+
+
+ + + + + + + +
+
+ + + + +
+ +
Conor Beverland + (conorbev@gmail.com) +
+
2022-06-28 20:08:54
 +
+

^ I'm just kind of curious if it's working as expected? And/or could it be possible to parse the DBT .sql so that the lineage in the first case would still show those staging tables?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-06-29 10:04:14
 +
+

*Thread Reply:* I think you should declare those as sources? Or do you need something different?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Conor Beverland + (conorbev@gmail.com) +
+
2022-06-29 21:15:33
 +
+

*Thread Reply:* I'll try to experiment with this.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Conor Beverland + (conorbev@gmail.com) +
+
2022-06-28 20:09:19
 +
+
  1. I see that DBT has a concept of adding tests to your models. Could those add data quality facets in OL ?
    2. +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-06-29 10:02:17
 +
+

*Thread Reply:* this should already be working if you run dbt-ol test or dbt-ol build

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Conor Beverland + (conorbev@gmail.com) +
+
2022-06-29 21:15:25
 +
+

*Thread Reply:* oh, nice!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
shweta p + (shweta.pbs@gmail.com) +
+
2022-07-04 02:48:35
 +
+

Hi everyone, i am trying openlineage-dbt. It works perfectly on locally when i try to publish the events to Marquez...but when i run the same commands from mwaa...i dont see those events triggered..i amnt able to view any logs if there is any error. How do i debug the issue

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2022-07-06 14:26:59
 +
+

*Thread Reply:* Maybe @Maciej Obuchowski knows? You need to check, it's using the dbt-ol command and that the configuration is available. (environment variables or conf file)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-07-06 15:31:20
 +
+

*Thread Reply:* Maybe some aws networking stuff? I'm not really sure how mwaa works internally (or, at all - never used it)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-07-06 15:35:06
 +
+

*Thread Reply:* anyway, any logs/errors should be in the same space where your task logs are

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-07-06 05:32:28
 +
+

Agenda items are requested for the next OpenLineage Technical Steering Committee meeting on July 14. Reply in thread or ping me with your item(s)!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-07-06 10:21:50
 +
+

*Thread Reply:* What is the status on the Flink / Streaming decisions being made for OpenLineage / Marquez?

+ +

A few months ago, Flink was being introduced and it was said that more thought was needed around supporting streaming services in OpenLineage.

+ +

It would be very helpful to know where the community stands on how streaming data sources should work in OpenLineage.

+ + + +
+ 👍 Michael Robinson +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-07-06 11:08:01
 +
+

*Thread Reply:* @Will Johnson added your item

+ + + +
+ 👍 Will Johnson +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-07-06 10:19:44
 +
+

Request for Creating a New OpenLineage Release

+ +

Hello #general, as per the Governance guide (https://github.com/OpenLineage/OpenLineage/blob/main/GOVERNANCE.md#openlineage-project-releases), I am asking that we generate a new release based on the latest commit by @Maciej Obuchowski (c92a93cdf3df636a02984188563d019474904b2b) which fixes a critical issue running OpenLineage on Azure Databricks.

+ +

Having this release made available to the general public on Maven would allow us to enable the hundred+ users of the solution to run OpenLineage on the latest LTS versions of Databricks. In addition, it would enable the Microsoft team to integrate the amazing column level lineage feature contributed by @Paweł Leszczyński with our solution for Microsoft Purview.

+ + + +
+ 👍 Maciej Obuchowski, Jakub Dardziński, Ross Turk, Willy Lulciuc, Will Johnson, Julien Le Dem +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-07-07 10:33:41
 +
+

@channel The next OpenLineage Technical Steering Committee meeting is on Thursday, July 14 at 10 am PT. Join us on Zoom: https://bit.ly/OLzoom +All are welcome! +Agenda:

+ +
  1. Announcements/recent talks
  2. Release 0.10.0 overview
  3. Flink integration retrospective
  4. Discuss: streaming services in Flink integration
  5. Open discussion +Notes: https://bit.ly/OLwiki +Is there a topic you think the community should discuss at this or a future meeting? Reply or DM me to add items to the agenda.
    6. +
+
+
Zoom Video
+ + + +
Join our Cloud HD Video Meeting
+
+ Zoom is the leader in modern enterprise video communications, with an easy, reliable cloud platform for video and audio conferencing, chat, and webinars across mobile, desktop, and room systems. Zoom Rooms is the original software-based conference room solution used around the world in board, conference, huddle, and training rooms, as well as executive offices and classrooms. Founded in 2011, Zoom helps businesses and organizations bring their teams together in a frictionless environment to get more done. Zoom is a publicly traded company headquartered in San Jose, CA. +
+ + + + + + +
Original URL: https://bit.ly/OLzoom
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
David Cecchi + (david_cecchi@cargill.com) +
+
2022-07-11 10:30:34
 +
+

*Thread Reply:* would appreciate a TSC discussion on OL philosophy for Streaming in general and where/if it fits in the vision and strategy for OL. fully appreciate current maturity, moreso just validating how OL is being positioned from a vision perspective. as we consider aligning enterprise lineage solution around OL want to make sure we're not making bad assumptions. neat discussion might be "imagine that Confluent decided to make Stream Lineage OL compliant/capable - are we cool with that and what are the implications?".

+ + + +
+ 👍 Michael Robinson +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-07-12 12:36:17
 +
+

*Thread Reply:* @Michael Robinson could I also have a quick 5m to talk about plans for a documentation site?

+ + + +
+ 👍 Michael Robinson, Sheeri Cabral (Collibra) +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-07-12 12:46:29
 +
+

*Thread Reply:* @David Cecchi @Ross Turk Added your items to the agenda. Thanks and looking forward to the discussion!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
David Cecchi + (david_cecchi@cargill.com) +
+
2022-07-12 15:08:48
 +
+

*Thread Reply:* this is great - will keep an eye out for recording. if it got tabled due to lack of attendance will pick it up next TSC.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sheeri Cabral (Collibra) + (sheeri.cabral@collibra.com) +
+
2022-07-12 16:12:43
 +
+

*Thread Reply:* I think OpenLineage should have some representation at https://impactdatasummit.com/2022

+ +

I’m happy to help craft the abstract, look over slides, etc. (I could help present, but all I’ve done with OpenLineage is one tutorial, so I’m hardly an expert).

+ +

CfP closes 31 Aug so there’s plenty of time, but if you want a 2nd set of eyes on things, we can’t just wait until the last minute to submit 😄

+
+
impactdatasummit.com
+ + + +
IMPACT 2022 - The Data Observability Summit
+
+ Announcing Monte Carlo’s annual Data Observability event, IMPACT 2022: The Data Observability Summit. This year we will be spotlighting some of the industry’s most prominent voices, as well as the broader community of data leaders paving the way forward for reliable data. +
+ + + + + + +
Original URL: https://impactdatasummit.com/2022
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-07-07 12:04:09
 +
+

How to create custom facets without recompiling OpenLineage?

+ +

I have a customer who is interested in using OpenLineage but wants to extend the facets WITHOUT recompiling OL / maintaining a clone of OL with their changes.

+ +

Do we have any examples of how someone might create their own jar but using the OpenLineage CustomFacetBuilder and then have that jar's classes be injected into OpenLineage?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-07-07 12:04:55
 +
+

*Thread Reply:* @Michael Collado would you have any thoughts on how to extend the Facets without having to alter OpenLineage itself?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2022-07-07 15:16:45
 +
+

*Thread Reply:* This is described here. Notably: +> Custom implementations are registered by following Java's ServiceLoader conventions. A file called io.openlineage.spark.api.OpenLineageEventHandlerFactory must exist in the application or jar's META-INF/service directory. Each line of that file must be the fully qualified class name of a concrete implementation of OpenLineageEventHandlerFactory. More than one implementation can be present in a single file. This might be useful to separate extensions that are targeted toward different environments - e.g., one factory may contain Azure-specific extensions, while another factory may contain GCP extensions.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2022-07-07 15:17:55
 +
+

*Thread Reply:* This example is present in the test package - https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/app/src/tes[…]ervices/io.openlineage.spark.api.OpenLineageEventHandlerFactory

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/app/src/test/resources/META-INF/services/io.openlineage.spark.api.OpenLineageEventHandlerFactory | io.openlineage.spark.api.OpenLineageEventHandlerFactory>
+
+ <pre><code>io.openlineage.spark.agent.util.TestOpenLineageEventHandlerFactory +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-07-07 20:19:01
 +
+

*Thread Reply:* @Michael Collado you are amazing! Thank you so much for pointing me to the docs and example!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-07-07 19:27:47
 +
+

@channel @Will Johnson +OpenLineage 0.11.0 is now available! +We added: +• an HTTP option to override timeout and properly close connections in openlineage-java lib, +• dynamic mapped tasks support to the Airflow integration, +• a SqlExtractor to the Airflow integration, +• PMD to Java and Spark builds in CI. +We changed: +• when testing extractors in the Airflow integration, the extractor list length assertion is now dynamic, +• templates are rendered at the start of integration tests for the TaskListener in the Airflow integration. +Thanks to all the contributors who made this release possible! +For the bug fixes and more details, see: +Release: https://github.com/OpenLineage/OpenLineage/releases/tag/0.11.0 +Changelog: https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md +Commit history: https://github.com/OpenLineage/OpenLineage/compare/0.10.0...0.11.0 +Maven: https://oss.sonatype.org/#nexus-search;quick~openlineage +PyPI: https://pypi.org/project/openlineage-python/

+ + + +
+ 👍 Chandru TMBA, John Thomas, Maciej Obuchowski, Fenil Doshi +
+ +
+ 👏 John Thomas, Willy Lulciuc, Ricardo Gaspar +
+ +
+ 🙌 Will Johnson, Maciej Obuchowski, Sergio Sicre +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Varun Singh + (varuntestaz@outlook.com) +
+
2022-07-11 07:06:36
 +
+

Hi all, I am using openlineage-spark in my project where I lock the dependency versions in gradle.lockfile. After release 0.10.0, this is not working. Is this a known limitation of switching to splitting the integration into submodules?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-07-14 06:18:29
 +
+

*Thread Reply:* Can you expand on what's not working exactly?

+ +

This is not something we're aware of.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Varun Singh + (varuntestaz@outlook.com) +
+
2022-07-19 04:09:39
 +
+

*Thread Reply:* @Maciej Obuchowski Sure, I have my own library where I am creating a shadowJar. This includes the open lineage library into the new uber jar. This worked fine till 0.9.0 but now building the shadowJar gives this error +Could not determine the dependencies of task ':shadowJar'. +&gt; Could not resolve all dependencies for configuration ':runtimeClasspath'. + &gt; Could not find spark:app:0.10.0. + Searched in the following locations: + - <https://repo.maven.apache.org/maven2/spark/app/0.10.0/app-0.10.0.pom> + If the artifact you are trying to retrieve can be found in the repository but without metadata in 'Maven POM' format, you need to adjust the 'metadataSources { ... }' of the repository declaration. + Required by: + project : &gt; io.openlineage:openlineage_spark:0.10.0 + &gt; Could not find spark:shared:0.10.0. + Searched in the following locations: + - <https://repo.maven.apache.org/maven2/spark/shared/0.10.0/shared-0.10.0.pom> + If the artifact you are trying to retrieve can be found in the repository but without metadata in 'Maven POM' format, you need to adjust the 'metadataSources { ... }' of the repository declaration. + Required by: + project : &gt; io.openlineage:openlineage_spark:0.10.0 + &gt; Could not find spark:spark2:0.10.0. + Searched in the following locations: + - <https://repo.maven.apache.org/maven2/spark/spark2/0.10.0/spark2-0.10.0.pom> + If the artifact you are trying to retrieve can be found in the repository but without metadata in 'Maven POM' format, you need to adjust the 'metadataSources { ... }' of the repository declaration. + Required by: + project : &gt; io.openlineage:openlineage_spark:0.10.0 + &gt; Could not find spark:spark3:0.10.0. + Searched in the following locations: + - <https://repo.maven.apache.org/maven2/spark/spark3/0.10.0/spark3-0.10.0.pom> + If the artifact you are trying to retrieve can be found in the repository but without metadata in 'Maven POM' format, you need to adjust the 'metadataSources { ... }' of the repository declaration. + Required by: + project : &gt; io.openlineage:openlineage_spark:0.10.0

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-07-19 05:00:02
 +
+

*Thread Reply:* Can you try 0.11? I think we might already fixed that.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Varun Singh + (varuntestaz@outlook.com) +
+
2022-07-19 05:50:03
 +
+

*Thread Reply:* Tried with that as well. Doesn't work

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Varun Singh + (varuntestaz@outlook.com) +
+
2022-07-19 05:56:50
 +
+

*Thread Reply:* Same error with 0.11.0 as well

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-07-19 08:11:13
 +
+

*Thread Reply:* I think I see - we removed internal dependencies from maven's pom.xml but we also publish gradle metadata: https://repo1.maven.org/maven2/io/openlineage/openlineage-spark/0.11.0/openlineage-spark-0.11.0.module

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-07-19 08:11:34
 +
+

*Thread Reply:* we should remove the dependencies or disable the gradle metadata altogether, it's not required

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-07-19 08:16:18
 +
+

*Thread Reply:* @Varun Singh For now I think you can try ignoring gradle metadata: https://docs.gradle.org/current/userguide/declaring_repositories.html#sec:supported_metadata_sources

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Hanbing Wang + (doris.wang200902@gmail.com) +
+
2022-07-19 14:18:45
 +
+

*Thread Reply:* @Varun Singh did you find out how to build shadowJar successful with release 0.10.0. I can build shadowJar with 0.9.0, but not higher version. If your problem already resolved, could you share some suggestion. thanks ^^

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Varun Singh + (varuntestaz@outlook.com) +
+
2022-07-20 03:44:40
 +
+

*Thread Reply:* @Hanbing Wang I followed @Maciej Obuchowski's instructions (Thank you!) and added this to my build.gradle file: +repositories { + mavenCentral() { + metadataSources { + mavenPom() + ignoreGradleMetadataRedirection() + } + } +} +I am able to build the jar now. I am not proficient in gradle so don't know if this is the right way to do this. Please correct me if I am wrong.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Varun Singh + (varuntestaz@outlook.com) +
+
2022-07-20 05:26:04
 +
+

*Thread Reply:* Also, I am not able to see the 3rd party dependencies in the dependency lock file, but they are present in some folder inside the jar (relocated in subproject's build file). But this is a different problem ig

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Hanbing Wang + (doris.wang200902@gmail.com) +
+
2022-07-20 18:45:50
 +
+

*Thread Reply:* Thanks @Varun Singh for the very helpful info. I will also try update build.gradle and rebuild shadowJar again.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-07-13 01:10:01
 +
+

Java Question: Why Can't I Find a Class on the Class Path? / How the heck does the ClassLoader know where to find a class?

+ +

Are there any java pros that would be willing to share alternatives to searching if a given class exists or help explain what should change in the Kusto package to make it work for the behaviors as seen in Kafka and SQL DW relation visitors? +--- Details --- +@Hanna Moazam and I are trying to introduce two new Azure data sources into OpenLineage's Spark integration. The https://github.com/Azure/azure-kusto-spark package is nearly done but we're getting tripped up on some Java concepts. In order to know if we should add the KustoRelationVisitor to the input dataset visitors, we need to see if the Kusto jar is installed on the spark / databricks cluster. In this case, the com.microsoft.kusto.spark.datasource.DefaultSource is a public class but it cannot be found using the KustRelationVisitor.class.getClassLoader().loadClass("class name") methods as seen in:

+ +

• https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/shared/src/[…]nlineage/spark/agent/lifecycle/plan/SqlDWDatabricksVisitor.java +• https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/shared/src/[…]penlineage/spark/agent/lifecycle/plan/KafkaRelationVisitor.java +At first I thought it was the Azure packages but then I tried to do the same approach with a simple java library

+ +

I instantiate a spark-shell like this +spark-shell --master local[4] \ +--conf spark.driver.extraClassPath=/mnt/repos/SparkListener-Basic/lib/build/libs/custom-listener.jar \ +--conf spark.extraListeners=listener.MyListener +--jars /mnt/repos/wjtestlib/lib/build/libs/lib.jar +With lib.jar containing a class that looks like this: +```package wjtestlib;

+ +

public class WillLibrary { + public boolean someLibraryMethod() { + return true; + } +} +And the custom listener is very simple. +public class MyListener extends org.apache.spark.scheduler.SparkListener {

+ +

private static final Logger log = LoggerFactory.getLogger("MyLogger");

+ +

public MyListener() { + log.info("INITIALIZING"); + }

+ +

@Override + public void onJobStart(SparkListenerJobStart jobStart) { + log.info("MYLISTENER: ON JOB START"); + try{ + log.info("Trying wjtestlib.WillLibrary"); + MyListener.class.getClassLoader().loadClass("wjtestlib.WillLibrary"); + log.info("Got wjtestlib.WillLibrary"); + } catch(ClassNotFoundException e){ + log.info("Could not get wjtestlib.WillLibrary"); + }

+ +
try{
+  <a href="http://log.info">log.info</a>("Trying wjtestlib.WillLibrary using Class.forName");
+  Class.forName("wjtestlib.WillLibrary", false, this.getClass().getClassLoader());
+  <a href="http://log.info">log.info</a>("Got wjtestlib.WillLibrary using Class.forName");
+} catch(ClassNotFoundException e){
+  <a href="http://log.info">log.info</a>("Could not get wjtestlib.WillLibrary using Class.forName");
+}
+
+ +

} +} +And I still a result indicating it cannot find the class. +2022-07-12 23:58:22,048 INFO MyLogger: MYLISTENER: ON JOB START +2022-07-12 23:58:22,048 INFO MyLogger: Trying wjtestlib.WillLibrary +2022-07-12 23:58:22,057 INFO MyLogger: Could not get wjtestlib.WillLibrary +2022-07-12 23:58:22,058 INFO MyLogger: Trying wjtestlib.WillLibrary using Class.forName +2022-07-12 23:58:22,065 INFO MyLogger: Could not get wjtestlib.WillLibrary using Class.forName``` +Are there any java pros that would be willing to share alternatives to searching if a given class exists or help explain what should change in the Kusto package to make it work for the behaviors as seen in Kafka and SQL DW relation visitors?

+ +

Thank you for any guidance.!

+
+ + + + +
Azure/azure-kusto-spark
+
+ +
+ +
+
Stars
+ 58 +
+ +
+
Language
+ Scala +
+ + + + + + + + +
+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/shared/src/main/java/io/openlineage/spark/agent/lifecycle/plan/SqlDWDatabricksVisitor.java | SqlDWDatabricksVisitor.java>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2022-07-13 08:50:15
 +
+

*Thread Reply:* Could you unzip the created jar and verify that classes you’re trying to use are present? Perhaps there’s some relocate in shadowJar plugin, which renames the classes. Making sure the classes are present in jar good point to start.

+ +

Then you can try doing classForName just from the spark-shell without any listeners added. The classes should be available there.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-07-13 11:42:25
 +
+

*Thread Reply:* Thank you for the reply Pawel! Hanna and I just wrapped up some testing.

+ +

It looks like Databricks AND open source spark does some magic when you install a library OR use --jars on the spark-shell. In both Databricks and Apache Spark, the thread running the SparkListener cannot see the additional libraries installed unless they're on the original / main class path.

+ +

• Confirmed the uploaded jars are NOT shaded / renamed. +• The databricks class path ($CLASSPATH) is focused on /databricks/jars +• The added libraries are in /local_disk0/tmp and are not found in $CLASSPATH. +• The sparklistener only recognizes $CLASSPATH. +• Using a classloader with an object like spark does not find our installed class: spark.getClass().getClassLoader().getResource("com/microsoft/kusto/spark/datasource/KustoSourceOptions.class") +• When we use a classloader on a class we installed and imported, it DOES find the class. myImportedClass.getClass().getClassLoader().getResource("com/microsoft/kusto/spark/datasource/KustoSourceOptions.class") +@Michael Collado and @Maciej Obuchowski have you seen any challenges with using --jars on the spark-shell and detecting if the class is installed?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-07-13 12:02:05
 +
+

*Thread Reply:* We run tests using --packages for external stuff like Delta - which is the same as --jars , but getting them from maven central, not local disk, and it works, like in KafkaRelationVisitor.

+ +

What if you did it like it? By that I mean adding it to your code with compileOnly in gradle or provided in maven, compiling with it, then using static method to check if it loads?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-07-13 12:02:36
 +
+

*Thread Reply:* > • When we use a classloader on a class we installed and imported, it DOES find the class. myImportedClass.getClass().getClassLoader().getResource("com/microsoft/kusto/spark/datasource/KustoSourceOptions.class") +Isn't that this actual scenario?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-07-13 12:36:47
 +
+

*Thread Reply:* Thank you for the reply, Maciej!

+ +

I will try the compileOnly route tonight!

+ +

Re: myImportedClass.getClass().getClassLoader().getResource("com/microsoft/kusto/spark/datasource/KustoSourceOptions.class")

+ +

I failed to mention that this was only achieved in the interactive shell / Databricks notebook. It never worked inside the SparkListener UNLESS we installed the Kusto jar on the databricks class path.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2022-07-14 06:43:47
 +
+

*Thread Reply:* The difference between --jars and --packages is that for packages all transitive dependencies will be handled. But this does not seem to be the case here.

+ +

More doc can be found here: (https://spark.apache.org/docs/latest/submitting-applications.html#advanced-dependency-management)

+ +

When starting a SparkContext, all the jars available on the classpath should be listed and put into Spark logs. So that’s the place one can check if the jar is loaded or not.

+ +

If --conf spark.driver.extraClassPath is working, you can add multiple jar files there (they must be separated by commas).

+ +

Other examples of adding multiple jars to spark classpath can be found here -> https://sparkbyexamples.com/spark/add-multiple-jars-to-spark-submit-classpath/

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-07-14 11:20:02
 +
+

*Thread Reply:* @Paweł Leszczyński thank you for the reply! Hanna and I experimented with jars vs extraClassPath.

+ +

When using jars, the spark listener does NOT find the class using a classloader.

+ +

When using extraClassPath, the spark listener DOES find the class using a classloader.

+ +

When using --jars, we can see in the spark logs that after spark starts (and after the spark listener is already established?) there are Spark.AddJar commands being executed.

+ +

@Maciej Obuchowski we also experimented with doing a compileOnly on OpenLineage's spark listener, it did not change the behavior. OpenLineage still failed to identify that I had the kusto-spark-connector.

+ +

I'm going to reach out to Databricks to see if there is any guidance on letting the SparkListener be aware of classes added via their libraries / --jar method on the spark-shell.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-07-14 11:22:01
 +
+

*Thread Reply:* So, this is only relevant to Databricks now? Because I don't understand what do you do different than us with Kafka/Iceberg/Delta

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-07-14 11:22:48
 +
+

*Thread Reply:* I'm not the spark/classpath expert though - maybe @Michael Collado have something to add?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-07-14 11:24:12
 +
+

*Thread Reply:* @Maciej Obuchowski that's a super good question on Iceberg. How do you instantiate a spark job with Iceberg installed?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-07-14 11:26:04
 +
+

*Thread Reply:* It is still relevant to apache spark because I can't get OpenLineage to find the installed package UNLESS I use extraClassPath.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-07-14 11:29:13
 +
+

*Thread Reply:* Basically, by adding --packages org.apache.iceberg:iceberg_spark_runtime_3.1_2.12:0.13.0

+ +

https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/app/src/tes[…]a/io/openlineage/spark/agent/SparkContainerIntegrationTest.java

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/app/src/test/java/io/openlineage/spark/agent/SparkContainerIntegrationTest.java | SparkContainerIntegrationTest.java>
+
+ <pre><code> void testWriteIcebergTableVersion() { +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-07-14 11:29:51
 +
+

*Thread Reply:* Trying with --pacakges right now.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-07-14 11:54:37
 +
+

*Thread Reply:* Using --packages wouldn't let me find the Spark relation's default source:

+ +

Spark Shell command +spark-shell --master local[4] \ +--conf spark.driver.extraClassPath=/customListener-1.0-SNAPSHOT.jar \ +--conf spark.extraListeners=listener.MyListener \ +--jars /WillLibrary.jar \ +--packages com.microsoft.azure.kusto:kusto_spark_3.0_2.12:3.0.0 +Code inside customListener:

+ +

try{ + <a href="http://log.info">log.info</a>("Trying Kusto DefaultSource"); + MyListener.class.getClassLoader().loadClass("com.microsoft.kusto.spark.datasource.DefaultSource"); + <a href="http://log.info">log.info</a>("Got Kusto DefaultSource!!!!"); + } catch(ClassNotFoundException e){ + <a href="http://log.info">log.info</a>("Could not get Kusto DefaultSource"); + } +Logs indicating it still can't find the class when using --packages. +2022-07-14 10:47:35,997 INFO MyLogger: MYLISTENER: ON JOB START +2022-07-14 10:47:35,997 INFO MyLogger: Trying wjtestlib.WillLibrary +2022-07-14 10:47:36,000 INFO 2022-07-14 10:47:36,052 INFO MyLogger: Trying LogicalRelation +2022-07-14 10:47:36,053 INFO MyLogger: Got logical relation +2022-07-14 10:47:36,053 INFO MyLogger: Trying Kusto DefaultSource +2022-07-14 10:47:36,064 INFO MyLogger: Could not get Kusto DefaultSource +😢

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-07-14 11:59:07
 +
+

*Thread Reply:* what if you load your listener using also packages?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-07-14 12:00:38
 +
+

*Thread Reply:* That's how I'm doing it locally using spark.conf: +spark.jars.packages com.google.cloud.bigdataoss:gcs_connector:hadoop3-2.2.2,io.delta:delta_core_2.12:1.0.0,org.apache.iceberg:iceberg_spark3_runtime:0.12.1,io.openlineage:openlineage_spark:0.9.0

+ + + +
+ 👀 Will Johnson +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-07-14 12:20:47
 +
+

*Thread Reply:* @Maciej Obuchowski - You beautiful bearded man! +🙏 +2022-07-14 11:14:21,266 INFO MyLogger: Trying LogicalRelation +2022-07-14 11:14:21,266 INFO MyLogger: Got logical relation +2022-07-14 11:14:21,266 INFO MyLogger: Trying org.apache.iceberg.catalog.Catalog +2022-07-14 11:14:21,295 INFO MyLogger: Got org.apache.iceberg.catalog.Catalog!!!! +2022-07-14 11:14:21,295 INFO MyLogger: Trying Kusto DefaultSource +2022-07-14 11:14:21,361 INFO MyLogger: Got Kusto DefaultSource!!!! +I ended up setting my spark-shell like this (and used --jars for my custom spark listener since it's not on Maven).

+ +

spark-shell --master local[4] \ +--conf spark.extraListeners=listener.MyListener \ +--packages org.apache.iceberg:iceberg_spark_runtime_3.1_2.12:0.13.0,com.microsoft.azure.kusto:kusto_spark_3.0_2.12:3.0.0 \ +--jars customListener-1.0-SNAPSHOT.jar +So, now I just need to figure out how Databricks differs from this approach 😢

+ + + +
+ 😂 Maciej Obuchowski, Jakub Dardziński, Hanna Moazam +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2022-07-14 12:21:35
 +
+

*Thread Reply:* This is an annoying detail about Java ClassLoaders and the way Spark loads extra jars/packages

+ +

Remember Java's ClassLoaders are hierarchical - there are parent ClassLoaders and child ClassLoaders. Parents can't see their children's classes, but children can see their parent's classes.

+ +

When you use --spark.driver.extraClassPath , you're adding a jar to the main application ClassLoader. But when you use --jars or --packages, you're instructing the Spark application itself to load the extra jars into its own ClassLoader - a child of the main application ClassLoader that the Spark code creates and manages separately. Since your listener class is loaded by the main application ClassLoader, it can't see any classes that are loaded by the Spark child ClassLoader. Either both jars need to be on the driver classpath or both jars need to be loaded by the --jar or --packages configuration parameter

+ + + +
+ 🙌 Will Johnson, Paweł Leszczyński +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2022-07-14 12:26:15
 +
+

*Thread Reply:* In Databricks, we were not able to simply use the --packages argument to load the listener, which is why we have that init script that copies the jar into the classpath that Databricks uses for application startup (the main ClassLoader). You need to copy your visitor jar into the same location so that both jars are loaded by the same ClassLoader and can see each other

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/databricks/open-lineage-init-script.sh | open-lineage-init-script.sh>
+
+ <pre><code>cp -f $STAGE_DIR/openlineage-spark-**.jar /mnt/driver-daemon/jars || { echo "Error copying Spark Listener library file"; exit 1;} +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2022-07-14 12:29:09
 +
+

*Thread Reply:* (as an aside, this is one of the major drawbacks of the java agent approach and one reason why all the documentation recommends using the spark.jars.packages configuration parameter for loading the OL library - it guarantees that any DataSource nodes loaded by the Spark ClassLoader can be seen by the OL library and we don't have to use reflection for everything)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-07-14 12:30:25
 +
+

*Thread Reply:* @Michael Collado Thank you so much for the reply. The challenge is that Databricks has their own mechanism for installing libraries / packages.

+ +

https://docs.microsoft.com/en-us/azure/databricks/libraries/

+ +

These packages are installed on databricks AFTER spark is started and the physical files are located in a folder that is different than the main classpath.

+ +

I'm going to reach out to Databricks and see if we can get any guidance on this 😢

+
+
docs.microsoft.com
+ + + +
Libraries - Azure Databricks
+
+ Learn how to use and manage libraries in Azure Databricks. +
+ + + + + + + + +
Original URL: https://docs.microsoft.com/en-us/azure/databricks/libraries/
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-07-14 12:31:32
 +
+

*Thread Reply:* Unfortunately, I can't ask users to install their packages on Databricks in a non-standard way (e.g. via an init script) because no one will follow that recommendation.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2022-07-14 12:32:46
 +
+

*Thread Reply:* yeah, I'd prefer if we didn't need an init script to get OL on Databricks either 🤷‍♂️:skintone4:

+ + + +
+ 🤣 Will Johnson +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-07-17 01:03:02
 +
+

*Thread Reply:* Quick update: +• Turns out using a class loader from a Scala spark listener does not have this problem. +• https://stackoverflow.com/questions/7671888/scala-classloaders-confusion +• I'm trying to use URLClassLoader as recommended by a few MSFT folks and point it at the /local_disk0/tmp folder. +• https://stackoverflow.com/questions/17724481/set-classloader-different-directory +• I'm not having luck so far but hoping I can reason about it tomorrow and Monday. This is blocking us from adding additional data sources that are not pre-installed on databricks 😢

+
+
Stack Overflow
+ + + +
scala classloaders confusion
+
+ Please consider the following test program (using scala 2.9.0.1) object test { def main(args:Array[String]) = { println(ClassLoader.getSystemClassLoader.getResource("toto")) println(this. +
+ + + + + + + + +
Original URL: https://stackoverflow.com/questions/7671888/scala-classloaders-confusion
+ + + +
+
+
Stack Overflow
+ + + +
Set classloader different directory
+
+ I've searched some topics, but couldn't find the answer. What i need is just to set up an additional path for default ClassLoader. Right now i have such class: public class Loader extends ClassLo... +
+ + + + + + + + +
Original URL: https://stackoverflow.com/questions/17724481/set-classloader-different-directory
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-07-18 05:45:59
 +
+

*Thread Reply:* Can't help you now, but I'd love if you dumped the knowledge you've gained through this process into some doc on new OpenLineage doc site 🙏

+ + + +
+ 👍 Hanna Moazam +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Hanna Moazam + (hannamoazam@microsoft.com) +
+
2022-07-18 05:48:15
 +
+

*Thread Reply:* We'll definitely put all of it together as a reference for others, and hopefully have a solution by the end of it too

+ + + +
+ 🙌 Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-07-13 12:06:24
 +
+

@channel The next OpenLineage TSC meeting is tomorrow at 10 am PT! https://openlineage.slack.com/archives/C01CK9T7HKR/p1657204421157959

+
+ + +
+ + + } + + Michael Robinson + (https://openlineage.slack.com/team/U02LXF3HUN7) +
+ + + +
+
+ <p><strong>@channel</strong> The next OpenLineage Technical Steering Committee meeting is on <strong>Thursday, July 14 at 10 am PT</strong>. Join us on Zoom: <a href="https://bit.ly/OLzoom">https://bit.ly/OLzoom</a> +All are welcome! +Agenda:</p> + +<ol><li>Announcements/recent talks</li><li>Release 0.10.0 overview</li><li>Flink integration retrospective</li><li>Discuss: streaming services in Flink integration</li><li>Open discussion +Notes: <a href="https://bit.ly/OLwiki">https://bit.ly/OLwiki</a> +Is there a topic you think the community should discuss at this or a future meeting? Reply or DM me to add items to the agenda.</li> +</ol> +
+ + + + + + +
Original URL: https://openlineage.slack.com/archives/C01CK9T7HKR/p1657204421157959
+ + +
+ + Thread in Slack Conversation +
+ + +
+ + + +
+ 🙌 Willy Lulciuc, Maciej Obuchowski +
+ +
+ 💯 Willy Lulciuc, Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
David Cecchi + (david_cecchi@cargill.com) +
+
2022-07-13 16:32:12
 +
+

check this out folks - marklogic datahub flow lineage into OL/marquez with jobs and runs and more. i would guess this is a pretty narrow use case but it went together really smoothly and thought i'd share sometimes it's just cool to see what people are working on

+ +
+
+ + + + + + +
+ + +
+ 🍺 Willy Lulciuc, Conor Beverland, Maciej Obuchowski, Paweł Leszczyński +
+ +
+ ❤️ Willy Lulciuc, Conor Beverland, Julien Le Dem, Michael Robinson, Maciej Obuchowski, Minkyu Park +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2022-07-13 16:40:48
 +
+

*Thread Reply:* Soo cool, @David Cecchi 💯💯💯. I’m not familiar with marklogic, but pretty awesome ETL platform and the lineage graph looks 👌! Did you have to write any custom integration code? Or where you able to use our off the self integrations to get things working? (Also, thanks for sharing!)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
David Cecchi + (david_cecchi@cargill.com) +
+
2022-07-13 16:57:29
 +
+

*Thread Reply:* team had to write some custom stuff but it's all framework so it can be repurposed not rewritten over and over. i would see this as another "Platform" in the context of the integrations semantic OL uses, so no, we didn't start w/ an existing solution. just used internal hooks and then called lineage APIs.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2022-07-13 17:02:53
 +
+

*Thread Reply:* Ah totally make sense. Would you be open to a brief presentation and/or demo in a future OL community meeting? The community is always looking to hear how OL is used in the wild, and this seems aligned with that (assuming you can talk about the implementation at a high-level)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2022-07-13 17:05:35
 +
+

*Thread Reply:* No pressure, of course 😉

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
David Cecchi + (david_cecchi@cargill.com) +
+
2022-07-13 17:08:50
 +
+

*Thread Reply:* ha not feeling any pressure. familiar with the intentions and dynamic. let's keep that on radar - i don't keep tabs on community meetings but mid/late august would be workable. and to be clear, this is being used in the wild in a sandbox 🙂.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2022-07-13 17:12:55
 +
+

*Thread Reply:* Sounds great, and a reasonable timeline! (cc @Michael Robinson can follow up). Even if it’s in a sandbox, talking about the level of effort helps with improving our APIs or sharing with others how smooth it can be!

+ + + +
+ 👍 David Cecchi +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-07-13 17:18:27
 +
+

*Thread Reply:* chiming in as well to say this is really cool 👍

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2022-07-13 18:26:28
 +
+

*Thread Reply:* Nice! Would this become a product feature in Marklogic Data Hub?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mark Chiarelli + (mark.chiarelli@marklogic.com) +
+
2022-07-14 11:07:42
 +
+

*Thread Reply:* MarkLogic is a multi-model database and search engine. This implementation triggers off the MarkLogic Datahub Github batch records created when running the datahub flows. Just a toe in the water so far.

+
+ + + + +
MarkLogic
+
+ +
+ +
+
Location
+ San Carlos, CA USA +
+ +
+
URL
+ <http://developer.marklogic.com> +
+ +
+
Repositories
+ 23 +
+ + + + + + + + + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2022-07-14 20:31:18
 +
+

@Ross Turk, in the OL community meeting today, you presented the new doc site (awesome!) that isn’t up (yet!), but I’ve been talk with @Julien Le Dem about the usage of _producer and would like to add a section on the use / function of _producer in OL events. Feel like the new doc site would be a great place to add this! Let me know when’s a good time to start crowd sourcing content for the site

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-07-14 20:37:25
 +
+

*Thread Reply:* That sounds like a good idea to me. Be good to have some guidance on that.

+ +

The repo is open for business! Feel free to add the page where you think it fits.

+
+ + + + +
OpenLineage/docs
+
+ Documentation for OpenLineage +
+ +
+
Website
+ <https://docs.openlineage.io> +
+ +
+
Stars
+ 1 +
+ + + + + + + + +
+ + + +
+ ❤️ Willy Lulciuc +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2022-07-14 20:42:09
 +
+

*Thread Reply:* OK! Let’s do this!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2022-07-14 20:59:36
 +
+

*Thread Reply:* @Ross Turk, feel free to assign to me https://github.com/OpenLineage/docs/issues/1!

+
+ + + + +
<https://github.com/OpenLineage/docs/issues/1|#1 Doc usage of `_producer` and `_schemaURL`>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/docs">OpenLineage/docs</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-07-14 20:39:26
 +
+

Hey everyone! As Willy says, there is a new documentation site for OpenLineage in the works.

+ +

It’s not quite ready to be, uh, a proper reference yet. But it’s not too far away. Help us get there by submitting issues, making page stubs, and adding sections via PR.

+ +

https://github.com/openlineage/docs/

+
+ + + + +
OpenLineage/docs
+
+ Documentation for OpenLineage +
+ +
+
Website
+ <https://docs.openlineage.io> +
+ +
+
Stars
+ 1 +
+ + + + + + + + +
+ + + +
+ 🙌 Maciej Obuchowski, Michael Robinson +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2022-07-14 20:43:09
 +
+

*Thread Reply:* Thanks, @Ross Turk for finding a home for more technical / how-to docs… long overdue 💯

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-07-14 21:22:09
 +
+

*Thread Reply:* BTW you can see the current site at http://openlineage.io/docs/ - merges to main will ship a new site.

+
+
openlineage.io
+ + + +
About OpenLineage | OpenLineage Docs
+
+ This page has not yet been written! You're welcome to contribute using the Edit link at the bottom. +
+ + + + + + +
Original URL: http://openlineage.io/docs/
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2022-07-14 21:23:32
 +
+

*Thread Reply:* great, was using <a href="http://docs.openlineage.io">docs.openlineage.io</a> … we’ll eventually want the docs to live under the docs subdomain though?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-07-14 21:25:32
 +
+

*Thread Reply:* TBH I activated GitHub Pages on the repo expecting it to live at openlineage.github.io/docs, thinking we could look at it there before it's ready to be published and linked in to the website

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-07-14 21:25:39
 +
+

*Thread Reply:* and it came live at openlineage.io/docs 😄

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2022-07-14 21:26:06
 +
+

*Thread Reply:* nice and sounds good 👍

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-07-14 21:26:31
 +
+

*Thread Reply:* still do not understand why, but I'll take it as a happy accident. we can move to docs.openlineage.io easily - just need to add the A record in the LF infra + the CNAME file in the static dir of this repo

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
shweta p + (shweta.pbs@gmail.com) +
+
2022-07-15 09:10:46
 +
+

Hi #general, how do i link the tasks of airflow which may not have any input or output datasets as they are running some conditions. the dataset is generated only on the last task

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
shweta p + (shweta.pbs@gmail.com) +
+
2022-07-15 09:11:25
 +
+

In the lineage, though there is option to link the parent , it doesnt show up the lineage of job -> job

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
shweta p + (shweta.pbs@gmail.com) +
+
2022-07-15 09:11:43
 +
+

does it need to be job -> dataset -> job only ?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-07-15 14:41:30
 +
+

*Thread Reply:* yes - openlineage is job -> dataset -> job. particularly, the model is designed to observe the movement of data

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-07-15 14:43:41
 +
+

*Thread Reply:* the spec is based around run events, which are observed states of job runs. jobs are observed to see how they affect datasets, and that relationship is what OpenLineage traces

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ilya Davidov + (idavidov@marpaihealth.com) +
+
2022-07-18 11:32:06
 +
+

👋 Hi everyone!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ilya Davidov + (idavidov@marpaihealth.com) +
+
2022-07-18 11:32:51
 +
+

i am looking for some information regarding openlineage integration with AWS Glue jobs/workflows

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ilya Davidov + (idavidov@marpaihealth.com) +
+
2022-07-18 11:33:32
 +
+

i am wondering if it possible and someone already give a try and maybe documented it?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john.thomas@astronomer.io) +
+
2022-07-18 15:16:54
 +
+

*Thread Reply:* This thread covers glue in some detail: https://openlineage.slack.com/archives/C01CK9T7HKR/p1637605977118000?threadts=1637605977.118000&cid=C01CK9T7HKR|https://openlineage.slack.com/archives/C01CK9T7HKR/p1637605977118000?threadts=1637605977.118000&cid=C01CK9T7HKR

+
+ + +
+ + + } + + Francis McGregor-Macdonald + (https://openlineage.slack.com/team/U02K353H2KF) +
+ + + +
+
+ <p>I am “successfully” exporting lineage to openlineage from AWS Glue using the listener. Only the source load is showing, not the transforms, or the sink</p> +
+ + + + + + +
Original URL: https://openlineage.slack.com/archives/C01CK9T7HKR/p1637605977118000?thread_ts=1637605977.118000&amp;cid=C01CK9T7HKR
+ + +
+ + Thread in Slack Conversation +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john.thomas@astronomer.io) +
+
2022-07-18 15:17:49
 +
+

*Thread Reply:* TL;Dr: you can use the spark integration to capture some lineage, but it's not comprehensive

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
David Cecchi + (david_cecchi@cargill.com) +
+
2022-07-18 16:29:02
 +
+

*Thread Reply:* i suspect there will be opportunities to influence AWS to be a "fast follower" if OL adoption and buy-in starts to feel authentically real in non-aws portions of the stack. i discussed OL casually with AWS analytics leadership (Rahul Pathak) last winter and he seemed curious and open to this type of idea. to be clear, ~95% chance he's forgotten that conversation now but hey it's still something.

+ + + +
+ 👍 Ross Turk +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Francis McGregor-Macdonald + (francis@mc-mac.com) +
+
2022-07-18 19:34:32
 +
+

*Thread Reply:* There are a couple of aws people here (including me) following.

+ + + +
+ 👍 David Cecchi, Ross Turk +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mikkel Kringelbach + (mikkel@theoremlp.com) +
+
2022-07-19 18:01:46
 +
+

Hi all, I have been playing around with Marquez for a hackday. I have been able to get some lineage information loaded in (using the local docker version for now). I have been trying set the location (for the link) and description information for a job (the text saying "Nothing to show here") but I haven't been able to figure out how to do this using the /lineage api. Any help would be appreciated.

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-07-19 20:11:38
 +
+

*Thread Reply:* I believe what you want is the DocumentationJobFacet. It adds a description property to a job.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-07-19 20:13:03
 +
+

*Thread Reply:* You can see a Python example here, in the Airflow integration: https://github.com/OpenLineage/OpenLineage/blob/65a5f021a1ba3035d5198e759587737a05b242e1/integration/airflow/openlineage/airflow/adapter.py#L217

+ + + +
+ :gratitude_thank_you: Mikkel Kringelbach +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-07-19 20:13:18
 +
+

*Thread Reply:* (looking for a curl example…)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mikkel Kringelbach + (mikkel@theoremlp.com) +
+
2022-07-19 20:25:49
 +
+

*Thread Reply:* I see, so there are special facet keys which will get translated into something special in the ui, is that correct?

+ +

Are these documented anywhere?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-07-19 20:27:55
 +
+

*Thread Reply:* Correct - info from the various OpenLineage facets are used in the Marquez UI.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-07-19 20:28:28
 +
+

*Thread Reply:* I couldn’t find a curl example with a description field, but I did generate this one with a sql field:

+ +

{ + "job": { + "name": "order_analysis.find_popular_products", + "facets": { + "sql": { + "query": "DROP TABLE IF EXISTS top_products;\n\nCREATE TABLE top_products AS\nSELECT\n product,\n COUNT(order_id) AS num_orders,\n SUM(quantity) AS total_quantity,\n SUM(price ** quantity) AS total_value\nFROM\n orders\nGROUP BY\n product\nORDER BY\n total_value desc,\n num_orders desc;", + "_producer": "https: //github.com/OpenLineage/OpenLineage/tree/0.11.0/integration/airflow", + "_schemaURL": "<https://raw.githubusercontent.com/OpenLineage/OpenLineage/main/spec/OpenLineage.json#/definitions/SqlJobFacet>" + } + }, + "namespace": "workshop" + }, + "run": { + "runId": "13460e52-a829-4244-8c45-587192cfa009", + "facets": {} + }, + "inputs": [ + ... + ], + "outputs": [ + ... + ], + "producer": "<https://github.com/OpenLineage/OpenLineage/tree/0.11.0/integration/airflow>", + "eventTime": "2022-07-20T00: 23: 06.986998Z", + "eventType": "COMPLETE" +}

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-07-19 20:28:58
 +
+

*Thread Reply:* The facets (at least, those in the core spec) are here: https://github.com/OpenLineage/OpenLineage/tree/65a5f021a1ba3035d5198e759587737a05b242e1/spec/facets

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-07-19 20:29:19
 +
+

*Thread Reply:* it’s designed so that facets can exist outside the core, in other repos, as well

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mikkel Kringelbach + (mikkel@theoremlp.com) +
+
2022-07-19 22:25:39
 +
+

*Thread Reply:* Thank you for sharing these, I was able to get the sql query highlighting to work. But I failed to get the location link or the documentation to work. My facet attempt looked like: +{ + "facets": { + "description": "test-description-job", + "sql": { + "query": "SELECT QUERY", + "_schema": "<https://raw.githubusercontent.com/OpenLineage/OpenLineage/main/spec/OpenLineage.json#/definitions/SqlJobFacet>" + }, + "documentation": { + "documentation": "Test docs?", + "_schema": "<https://raw.githubusercontent.com/OpenLineage/OpenLineage/main/spec/OpenLineage.json#/definitions/DocumentationJobFacet>" + }, + "link": { + "type": "", + "url": "<a href="http://www.google.com/test_url">www.google.com/test_url</a>", + "_schema": "<https://raw.githubusercontent.com/OpenLineage/OpenLineage/main/spec/OpenLineage.json#/definitions/SourceCodeLocationJobFacet>" + } + } +}

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mikkel Kringelbach + (mikkel@theoremlp.com) +
+
2022-07-19 22:36:55
 +
+

*Thread Reply:* I got the documentation link to work by renaming the property from documentation -> description . I still haven't been able to get the external link to work

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-07-20 10:33:36
 +
+

Hey all. I've been doing a cleanup of issues on GitHub. If I've closed your issue that you think is still relevant, please reopen it and let us know.

+ + + +
+ 🙌 Jakub Dardziński, Michael Collado, Will Johnson, Ross Turk +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sheeri Cabral (Collibra) + (sheeri.cabral@collibra.com) +
+
2022-07-21 16:09:08
 +
+

Is https://databricks.com/blog/2022/06/08/announcing-the-availability-of-data-lineage-with-unity-catalog.html - are they using OpenLineage? I know there’s been a lot of work to make sure OpenLineage integrates with Databricks, even earlier this year.

+
+
Databricks
+ + + +
Automated and Real-time Data Lineage With Unity Catalog
+
+ Learn more about data lineage, its organizational importance and how Unity Catalog provides automated and real-time data lineage at a granular level for all workloads (SQL, R, Python, Scala) and across all asset types (notebooks, workflows, dashboards) +
+ + + + + + + + +
Original URL: https://databricks.com/blog/2022/06/08/announcing-the-availability-of-data-lineage-with-unity-catalog.html
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-07-21 16:25:47
 +
+

*Thread Reply:* There’s a good integration between OL and Databricks for pulling metadata out of running Spark clusters. But there’s not currently a connection between OL and the Unity Catalog.

+ +

I think it would be cool to see some discussions start to develop around it 👍

+ + + +
+ 👍 Sheeri Cabral (Collibra), Julius Rentergent +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sheeri Cabral (Collibra) + (sheeri.cabral@collibra.com) +
+
2022-07-21 16:26:44
 +
+

*Thread Reply:* Absolutely. I saw some mention of APIs and access, and was wondering if maybe they used OpenLineage as a framework, which would be awesome.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sheeri Cabral (Collibra) + (sheeri.cabral@collibra.com) +
+
2022-07-21 16:30:55
 +
+

*Thread Reply:* (and since Azure Databricks uses it - https://openlineage.io/blog/openlineage-microsoft-purview/ I wasn’t sure about Unity Catalog)

+
+
openlineage.io
+ + + +
Microsoft Purview Accelerates Lineage Extraction from Azure Databricks
+
+ A new collaboration between Microsoft and OpenLineage is making lineage extraction possible for Azure Databricks and Microsoft Purview users. +
+ + + + + + +
Original URL: https://openlineage.io/blog/openlineage-microsoft-purview/
+ + + +
+ + + +
+ 👍 Will Johnson +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2022-07-21 16:56:24
 +
+

*Thread Reply:* We're in the early stages of discussion regarding an OpenLineage integration for Unity. You showing interest would help increase the priority of that on the DB side.

+ + + +
+ 👍 Sheeri Cabral (Collibra), Will Johnson, Thijs Koot +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Thijs Koot + (thijs.koot@gmail.com) +
+
2022-07-27 11:41:48
 +
+

*Thread Reply:* I'm interested in Databricks enabling an openlineage endpoint, serving as a catalogue. Similar to how they provide hosted MLFlow. I can mention this to our Databricks reps as well

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Joao Vicente + (joao.diogo.vicente@gmail.com) +
+
2022-07-23 04:09:55
 +
+

Hi all +I am trying to find the state of columnLineage in OL +I see a proposal and some examples in https://github.com/OpenLineage/OpenLineage/search?q=columnLineage&type=|https://github.com/OpenLineage/OpenLineage/search?q=columnLineage&type= but I can't find it in the spec. +Can anyone shed any light why this would be the case?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Joao Vicente + (joao.diogo.vicente@gmail.com) +
+
2022-07-23 04:12:26
 +
+

*Thread Reply:* Link to spec where I looked https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.json

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.json | OpenLineage.json>
+
+ <pre><code>{ + "$schema": "&lt;https://json-schema.org/draft/2020-12/schema&gt;", + "$id": "&lt;https://openlineage.io/spec/1-0-2/OpenLineage.json&gt;", + "$defs": { + "RunEvent": { + "type": "object", + "properties": { + "eventType": { + "description": "the current transition of the run state. It is required to issue 1 START event and 1 of [ COMPLETE, ABORT, FAIL ] event per run. Additional events with OTHER eventType can be added to the same run. For example to send additional metadata after the run is complete", + "type": "string", + "enum": [ + "START", + "COMPLETE", + "ABORT", + "FAIL", + "OTHER" + ], + "example": "START|COMPLETE|ABORT|FAIL|OTHER" + }, + "eventTime": { + "description": "the time the event occurred at", + "type": "string", + "format": "date-time" + }, + "run": { + "$ref": "#/$defs/Run" + }, + "job": { + "$ref": "#/$defs/Job" + }, + "inputs": { + "description": "The set of ****input**** datasets.", + "type": "array", + "items": { + "$ref": "#/$defs/InputDataset" + } + }, + "outputs": { + "description": "The set of ****output**** datasets.", + "type": "array", + "items": { + "$ref": "#/$defs/OutputDataset" + } + }, + "producer": { + "description": "URI identifying the producer of this metadata. For example this could be a git url with a given tag or sha", + "type": "string", + "format": "uri", + "example": "&lt;https://github.com/OpenLineage/OpenLineage/blob/v1-0-0/client&gt;" + }, + "schemaURL": { + "description": "The JSON Pointer (&lt;https://tools.ietf.org/html/rfc6901&gt;) URL to the corresponding version of the schema definition for this RunEvent", + "type": "string", + "format": "uri", + "example": "&lt;https://openlineage.io/spec/0-0-1/OpenLineage.json&gt;" + } + }, + "required": [ + "run", + "job", + "eventTime", + "producer", + "schemaURL" + ] + }, + "Run": { + "type": "object", + "properties": { + "runId": { + "description": "The globally unique ID of the run associated with the job.", + "type": "string", + "format": "uuid" + }, + "facets": { + "description": "The run facets.", + "type": "object", + "additionalProperties": { + "$ref": "#/$defs/RunFacet" + } + } + }, + "required": [ + "runId" + ] + }, + "RunFacet": { + "description": "A Run Facet", + "type": "object", + "allOf": [ + { "$ref": "#/$defs/BaseFacet" } + ] + }, + "Job": { + "type": "object", + "properties": { + "namespace": { + "description": "The namespace containing that job", + "type": "string", + "example": "my-scheduler-namespace" + }, + "name": { + "description": "The unique name for that job within that namespace", + "type": "string", + "example": "myjob.mytask" + }, + "facets": { + "description": "The job facets.", + "type": "object", + "additionalProperties": { + "$ref": "#/$defs/JobFacet" + } + } + }, + "required": [ + "namespace", + "name" + ] + }, + "JobFacet": { + "description": "A Job Facet", + "type": "object", + "allOf": [ + { "$ref": "#/$defs/BaseFacet" } + ] + }, + "InputDataset": { + "description": "An input dataset", + "type": "object", + "allOf": [ + { "$ref": "#/$defs/Dataset" }, + { + "type": "object", + "properties": { + "inputFacets": { + "description": "The input facets for this dataset.", + "type": "object", + "additionalProperties": { + "$ref": "#/$defs/InputDatasetFacet" + } + } + } + } + ] + }, + "InputDatasetFacet": { + "description": "An Input Dataset Facet", + "type": "object", + "allOf": [ + { "$ref": "#/$defs/BaseFacet" } + ] + }, + "OutputDataset": { + "description": "An output dataset", + "type": "object", + "allOf": [ + { "$ref": "#/$defs/Dataset" }, + { + "type": "object", + "properties": { + "outputFacets": { + "description": "The output facets for this dataset", + "type": "object", + "additionalProperties": { + "$ref": "#/$defs/OutputDatasetFacet" + } + } + } + } + ] + }, + "OutputDatasetFacet": { + "description": "An Output Dataset Facet", + "type": "object", + "allOf": [ + { "$ref": "#/$defs/BaseFacet" } + ] + }, + "Dataset": { + "type": "object", + "properties": { + "namespace": { + "description": "The namespace containing that dataset", + "type": "string", + "example": "my-datasource-namespace" + }, + "name": { + "description": "The unique name for that dataset within that namespace", + "type": "string", + "example": "instance.schema.table" + }, + "facets": { + "description": "The facets for this dataset", + "type": "object", + "additionalProperties": { + "$ref": "#/$defs/DatasetFacet" + } + } + }, + "required": [ + "namespace", + "name" + ] + }, + "DatasetFacet": { + "description": "A Dataset Facet", + "type": "object", + "allOf": [ + { "$ref": "#/$defs/BaseFacet" } + ] + }, + "BaseFacet": { + "description": "all fields of the base facet are prefixed with _ to avoid name conflicts in facets", + "type": "object", + "properties": { + "_producer": { + "description": "URI identifying the producer of this metadata. For example this could be a git url with a given tag or sha", + "type": "string", + "format": "uri", + "example": "&lt;https://github.com/OpenLineage/OpenLineage/blob/v1-0-0/client&gt;" + }, + "_schemaURL": { + "description": "The JSON Pointer (&lt;https://tools.ietf.org/html/rfc6901&gt;) URL to the corresponding version of the schema definition for this facet", + "type": "string", + "format": "uri", + "example": "&lt;https://openlineage.io/spec/1-0-2/OpenLineage.json#/$defs/BaseFacet&gt;" + } + }, + "additionalProperties": true, + "required": [ + "_producer", + "_schemaURL" + ] + } + }, + "$ref": "#/$defs/RunEvent" +} + +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Joao Vicente + (joao.diogo.vicente@gmail.com) +
+
2022-07-23 04:37:11
 +
+

*Thread Reply:* My bad. I realize now that column lineage has been implemented as a facet, hence not visible in the main spec https://github.com/OpenLineage/OpenLineage/search?q=ColumnLineageDatasetFacet&type=|https://github.com/OpenLineage/OpenLineage/search?q=ColumnLineageDatasetFacet&type=

+ + + +
+ 👍 Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2022-07-26 19:37:54
 +
+

*Thread Reply:* It is supported in the Spark integration

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2022-07-26 19:39:13
 +
+

*Thread Reply:* @Paweł Leszczyński could you add the Column Lineage facet here in the spec? https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.md#standard-facets

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-07-24 16:24:15
 +
+

SundayFunday

+ +

Putting together some internal training for OpenLineage and highlighting some of the areas that have been useful to me on my journey with OpenLineage. Many thanks to @Michael Collado, @Maciej Obuchowski, and @Paweł Leszczyński for the continued technical support and guidance.

+ +
+
+ + + + + + +
+ + +
+ ❤️ Hanna Moazam, Ross Turk, Minkyu Park, Atif Tahir, Paweł Leszczyński +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-07-24 16:26:59
 +
+

*Thread Reply:* @Ross Turk I still want to contribute something like this to the OpenLineage docs / new site but the bar for an internal doc is lower in my mind 😅

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-07-25 11:49:54
 +
+

*Thread Reply:* 😄

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-07-25 11:50:54
 +
+

*Thread Reply:* @Will Johnson happy to help you with docs, when the time comes! sketching outline --> editing, whatever you need

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2022-07-26 19:39:56
 +
+

*Thread Reply:* This looks nice by the way.

+ + + +
+ ❤️ Will Johnson +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sylvia Seow + (sylviaseow@gmail.com) +
+
2022-07-26 09:06:28
 +
+

hi all, really appreciate if anyone could help. I have been trying to create a poc project with openlineage with dbt. attached will be the pip list of the openlineage packages that i have. However, when i run "dbt-ol"command, it prompted as öpen as file, instead of running as a command. the regular dbt run can be executed without issue. i would want i had done wrong or if any configuration that i have missed. Thanks a lot

+ +
+
+ + + + + + +
+
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-07-26 10:39:57
 +
+

*Thread Reply:* do you have proper execute permissions?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-07-26 10:41:09
 +
+

*Thread Reply:* not sure how that works on windows, but it just looks like it does not recognize dbt-ol as executable

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sylvia Seow + (sylviaseow@gmail.com) +
+
2022-07-26 10:43:00
 +
+

*Thread Reply:* yes i have admin rights. how to make this as executable?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sylvia Seow + (sylviaseow@gmail.com) +
+
2022-07-26 10:43:25
 +
+

*Thread Reply:* btw do we have a sample docker image where dbt-ol can run?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-07-26 17:33:08
 +
+

*Thread Reply:* I have also never tried on Windows 😕 but you might try python3 dbt-ol run?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sylvia Seow + (sylviaseow@gmail.com) +
+
2022-07-26 21:03:43
 +
+

*Thread Reply:* will try that

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-07-26 16:41:04
 +
+

Running a single unit test on the Spark Integration - How it works with the different modules?

+ +

Prior to splitting up the OpenLineage spark integration, I could run a command like the one below to test a single test or even a single test method. Now I get a failure and it's pointing to the app: module. Can anyone share the right syntax for running a unit test with the current package structure? Thank you!!

+ +

```wj@DESKTOP-ECF9QME:~/repos/OpenLineageWill/integration/spark$ ./gradlew test --tests io.openlineage.spark.agent.OpenLineageSparkListenerTest

+ +

> Task :app:test FAILED

+ +

SUCCESS: Executed 0 tests in 872ms

+ +

FAILURE: Build failed with an exception.

+ +

** What went wrong: +Execution failed for task ':app:test'. +> No tests found for given includes: io.openlineage.spark.agent.OpenLineageSparkListenerTest

+ +

** Try: +> Run with --stacktrace option to get the stack trace. +> Run with --info or --debug option to get more log output. +> Run with --scan to get full insights.

+ +

** Get more help at https://help.gradle.org

+ +

Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.

+ +

You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.

+ +

See https://docs.gradle.org/7.4/userguide/command_line_interface.html#sec:command_line_warnings

+ +

BUILD FAILED in 2s +18 actionable tasks: 4 executed, 14 up-to-date```

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2022-07-27 01:54:31
 +
+

*Thread Reply:* This may be a result of splitting Spark integration into multiple submodules: app, shared, spark2, spark3, spark32, etc. If the test case is from shared submodule (this one looks like that), you could try running: +./gradlew :shared:test --tests io.openlineage.spark.agent.OpenLineageSparkListenerTest

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Hanna Moazam + (hannamoazam@microsoft.com) +
+
2022-07-27 03:18:42
 +
+

*Thread Reply:* @Paweł Leszczyński, I tried running that command, and I get the following error:

+ +

```> Task :shared:test FAILED

+ +

FAILURE: Build failed with an exception.

+ +

** What went wrong: +Execution failed for task ':shared:test'. +> No tests found for given includes: io.openlineage.spark.agent.OpenLineageSparkListenerTest

+ +

** Try: +> Run with --stacktrace option to get the stack trace. +> Run with --info or --debug option to get more log output. +> Run with --scan to get full insights.

+ +

** Get more help at https://help.gradle.org

+ +

Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.

+ +

You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.

+ +

See https://docs.gradle.org/7.4/userguide/command_line_interface.html#sec:command_line_warnings

+ +

BUILD FAILED in 971ms +6 actionable tasks: 2 executed, 4 up-to-date```

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Hanna Moazam + (hannamoazam@microsoft.com) +
+
2022-07-27 03:24:41
 +
+

*Thread Reply:* When running build and test for all the submodules, I can see outputs for tests in different submodules (spark3, spark2 etc), but for some reason, I cannot find any indication that the tests in +OpenLineage/integration/spark/app/src/test/java/io/openlineage/spark/agent/lifecycle/plan +are being run at all.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2022-07-27 03:42:43
 +
+

*Thread Reply:* That’s interesting. Let’s ask @Tomasz Nazarewicz about that.

+ + + +
+ 👍 Hanna Moazam +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Hanna Moazam + (hannamoazam@microsoft.com) +
+
2022-07-27 03:57:08
 +
+

*Thread Reply:* For reference, I attached the stdout and stderr messages from running the following: +./gradlew :shared:spotlessApply &amp;&amp; ./gradlew :app:spotlessApply &amp;&amp; ./gradlew clean build test

+ +
+
testOutput.txt
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tomasz Nazarewicz + (tomasz.nazarewicz@getindata.com) +
+
2022-07-27 04:27:23
 +
+

*Thread Reply:* I'll look into it

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tomasz Nazarewicz + (tomasz.nazarewicz@getindata.com) +
+
2022-07-28 05:17:36
 +
+

*Thread Reply:* Update: some test appeared to not be visible after split, that's fixed but now I have to solevr some dependency issues

+ + + +
+ 🙌 Hanna Moazam, Will Johnson +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Hanna Moazam + (hannamoazam@microsoft.com) +
+
2022-07-28 05:19:16
 +
+

*Thread Reply:* That's great, thank you!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Hanna Moazam + (hannamoazam@microsoft.com) +
+
2022-07-29 06:05:55
 +
+

*Thread Reply:* Hi Tomasz, thanks so much for looking into this. Is this your PR (https://github.com/OpenLineage/OpenLineage/pull/953) that fixes the whole issue, or is there still some work to do to solve the dependency issues you mentioned?

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/pull/953|#953 making OpenLineageSparkListenerTest.java visible for tests and fixing them>
+
+ <p>… them</p> + +<p>Signed-off-by: tomasznazarewicz <a href="mailto:t.nazarewicz94@gmail.com">t.nazarewicz94@gmail.com</a></p> + +<p><strong>Problem</strong></p> + +<p><a href="https://openlineage.slack.com/archives/C01CK9T7HKR/p1658868064009839">https://openlineage.slack.com/archives/C01CK9T7HKR/p1658868064009839</a></p> + +<p>io.openlineage.spark.agent.OpenLineageSparkListenerTest not visible for test</p> + +<p>also one test is failing when its run</p> + +<p>Closes:</p> + +<p><strong>Solution</strong></p> + +<p>build script for <code>app</code> module changed so the tests are visible, failing test fixed</p> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tomasz Nazarewicz + (tomasz.nazarewicz@getindata.com) +
+
2022-07-29 06:07:58
 +
+

*Thread Reply:* I'm still testing it, should've changed it to draft, sorry

+ + + +
+ 👍 Hanna Moazam, Will Johnson +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Hanna Moazam + (hannamoazam@microsoft.com) +
+
2022-07-29 06:08:59
 +
+

*Thread Reply:* No worries! If I can help with testing or anything please let me know!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tomasz Nazarewicz + (tomasz.nazarewicz@getindata.com) +
+
2022-07-29 06:09:29
 +
+

*Thread Reply:* Will do! Thanks :)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Hanna Moazam + (hannamoazam@microsoft.com) +
+
2022-08-02 11:06:31
 +
+

*Thread Reply:* Hi @Tomasz Nazarewicz, if possible, could you please share an estimated timeline for resolving the issue? We have 3 PRs which we are either waiting to open or to update which are dependent on the tests.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tomasz Nazarewicz + (tomasz.nazarewicz@getindata.com) +
+
2022-08-02 13:45:34
 +
+

*Thread Reply:* @Hanna Moazam hi, it's quite difficult to do that because the issue is that all the tests are passing when I execute ./gradlew app:test +but one is failing with ./gradlew app:build

+ +

but if it fixes your problem I can disable this test for now and make a PR without it, then you can maybe unblock your stuff and I will have more time to investigate the issue.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Hanna Moazam + (hannamoazam@microsoft.com) +
+
2022-08-02 14:54:45
 +
+

*Thread Reply:* Oh that's a strange issue. Yes that would be really helpful if you can, because we have some tests we implemented which we need to make sure pass as expected.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Hanna Moazam + (hannamoazam@microsoft.com) +
+
2022-08-02 14:54:52
 +
+

*Thread Reply:* Thank you for your help Tomasz!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tomasz Nazarewicz + (tomasz.nazarewicz@getindata.com) +
+
2022-08-03 06:12:07
 +
+

*Thread Reply:* @Hanna Moazam https://github.com/OpenLineage/OpenLineage/pull/980 here is the pull request with the changes

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/pull/980|#980 enable tests for app, fix bugs, disable LibraryTest.testRdd>
+
+ <p>Signed-off-by: tomasznazarewicz <a href="mailto:t.nazarewicz94@gmail.com">t.nazarewicz94@gmail.com</a></p> + +<p><strong>Problem</strong></p> + +<p><code>app</code> module was only running integration tests</p> + +<p>Original solution assumed that the test will be enabled and all potential bugs fixed. That worked for all tests except <code>LibraryTest.testRdd</code>, which has two problems:</p> + +<ol><li><code>java.lang.NoSuchMethodError: io.openlineage.client.OpenLineageClientUtils.newObjectMapper()Lcom/fasterxml/jackson/databind/ObjectMapper;</code></li><li>when ObjectMapper was changed there is issue with serialization, if test is run with <code>./gradlew app:test</code> the tests are passing but if it's run with <code>./gradlew app:build</code> the <code>OpenLineage.RunFacets.additionalProperties</code> is serialized as additional field and test is failing</li> +</ol> + +<p>EDIT: <br /> +It turns out there are also issues with Delta.</p> + +<p>Closes: potentially -&gt; <a href="https://openlineage.slack.com/archives/C01CK9T7HKR/p1658868064009839">https://openlineage.slack.com/archives/C01CK9T7HKR/p1658868064009839</a></p> + +<p><strong>Solution</strong></p> + +<p>It seems to be a blocking issue so the proposed solution for now is to disable <code>LibraryTest.testRdd</code> and <code>DeltaDataSourceTest.testInsertIntoDeltaSource</code> so everything is passing and in the meantime investigate both issues</p> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+ 🙌 Hanna Moazam +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tomasz Nazarewicz + (tomasz.nazarewicz@getindata.com) +
+
2022-08-03 06:12:26
 +
+

*Thread Reply:* its waiting for review currently

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Hanna Moazam + (hannamoazam@microsoft.com) +
+
2022-08-03 06:20:41
 +
+

*Thread Reply:* Thank you!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Conor Beverland + (conorbev@gmail.com) +
+
2022-07-26 18:44:47
 +
+

Is there any doc yet about column level lineage? I see a spec for the facet here: https://github.com/openlineage/openlineage/issues/148

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/issues/148|#148 [PROPOSAL] column level lineage facet>
+
+ <p><strong>Purpose:</strong> <br /> +For transformations like SQL queries (but also in other cases), we can extract column level lineage. This allows answering questions like ‘which root input columns are used to construct column x?’. We need to be able to capture this information in the OpenLineage model</p> + +<p><strong>Proposed implementation</strong> <br /> +We propose to introduce a new dataset facet: <br /> +example:</p> + +<pre><code>{ + "eventType": "COMPLETE", + "eventTime": "2020-12-28T20:52:00.001+10:00", + "run" : { + "runId": "uuid" + }, + "job": { + "namespace": "scheduler", + "name": "myjob", + "facets": { + "sql": { + "query": "Insert into outputTable from select ** from inputTable" + } + } + }, + "inputs": [ + { + "namespace": "N1", + "name": "inputTable", + "facets": { + "schema": { + "fields": [ {"name": "col_a", "type": "VARCHAR"}, {"name": "col_b", "type": "int"}] + } + } + } + ], + "outputs": [ + { + "namespace": "N2", + "name": "outputTable", + "facets": { + "schema": { + "fields": [ {"name": "col_a", "type": "VARCHAR"}, {"name": "col_b", "type": "int"}] + }, + "columnLineage": { + "fields": { + "col_a": [ { "namespace": "N1", "name": "inputTable", "field": "col_a"} ], + "col_b": [ { "namespace": "N1", "name": "inputTable", "field": "col_b" } ] + } + } + } + } + ] +} +</code></pre> + +<p>Schema:</p> + +<pre><code> "columnLineage": { + "type": "object", + "properties": { + "fields": { + "type": "array", + "items": { + "type": "object", + "additionalProperties": { + "type": "object", + "properties": { + "namespace": { + "type": "string", + "description": "the input dataset namespace" + }, + "name": { + "type": "string", + "description": "the input dataset name" + }, + "field": { + "type": "string", + "description": "the input field" + } + }, + "required": [ + "namespace", "name" + ] + } + } + } + } +</code></pre> + +<p>References:</p> + +<p>• Datahub spec: <a href="https://datahubproject.io/docs/rfc/active/1841-lineage/field_level_lineage/">https://datahubproject.io/docs/rfc/active/1841-lineage/field<em>level</em>lineage/</a></p> +
+ +
+
Assignees
+ <a href="https://github.com/pawel-big-lebowski">@pawel-big-lebowski</a> +
+ +
+
Labels
+ proposal +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2022-07-26 19:41:13
 +
+

*Thread Reply:* The doc site would benefit from a page about it. Maybe @Paweł Leszczyński?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2022-07-27 01:59:27
 +
+

*Thread Reply:* Sure, it’s already on my list, will do

+ + + +
+ :gratitude_thank_you: Julien Le Dem +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2022-07-29 07:55:40
 +
+

*Thread Reply:* https://openlineage.io/docs/integrations/spark/spark_column_lineage

+
+
openlineage.io
+ + + +
Column Level Lineage | OpenLineage Docs
+
+ Column level lineage provides fine grained information on datasets' dependencies. Not only we know the dependency exist, but we are also able to understand which input columns are used to produce output columns. This allows answering questions like Which root input columns are used to construct column x? +
+ + + + + + +
Original URL: https://openlineage.io/docs/integrations/spark/spark_column_lineage
+ + + +
+ + + +
+ ✅ Conor Beverland +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Conor Beverland + (conorbev@gmail.com) +
+
2022-07-26 20:03:55
 +
+

maybe another question for @Paweł Leszczyński: I was watching the Airflow summit talk that you and @Maciej Obuchowski did ( very nice! ). How is this exposed? I'm wondering if it shows up as an edge on the graph in Marquez? ( I guess it may be tracked as a parent run and if so probably does not show on the graph directly at this time? )

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2022-07-27 04:08:18
 +
+

*Thread Reply:* To be honest, I have never seen that in action and would love to have that in our documentation.

+ +

@Michael Collado or @Maciej Obuchowski: are you able to create some doc? I think one of you was working on that.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-07-27 04:24:19
 +
+

*Thread Reply:* Yes, parent run

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
shweta p + (shweta.pbs@gmail.com) +
+
2022-07-27 01:29:05
 +
+

Hi #general, there has been a issue with airflow+dbt+openlineage. This was working fine with openlineage-dbt v0.11.0 but there has been some change to the typeextensions due to which i had to upgrade to latest dbt (from 1.0.0 to 1.1.0) and now the dbt-ol is failing with schema version support (the version generated is v5 vs dbt-ol supports only v4). Has anyone else been able to fix this

+ + + +
+ 👀 Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-07-27 04:47:18
 +
+

*Thread Reply:* Will take a look

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-07-27 04:47:40
 +
+

*Thread Reply:* But generally this support message is just a warning

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-07-27 10:04:20
 +
+

*Thread Reply:* @shweta p any actual error you've found? +I've tested it with dbt-bigquery on 1.1.0 and it works despite warning:

+ +

➜ small OPENLINEAGE_URL=<http://localhost:5050> dbt-ol build +Running OpenLineage dbt wrapper version 0.11.0 +This wrapper will send OpenLineage events at the end of dbt execution. +14:03:16 Running with dbt=1.1.0 +14:03:17 Found 2 models, 3 tests, 0 snapshots, 0 analyses, 191 macros, 0 operations, 0 seed files, 0 sources, 0 exposures, 0 metrics +14:03:17 +14:03:17 Concurrency: 2 threads (target='dev') +14:03:17 +14:03:17 1 of 5 START table model dbt_test1.my_first_dbt_model .......................... [RUN] +14:03:21 1 of 5 OK created table model dbt_test1.my_first_dbt_model ..................... [CREATE TABLE (2.0 rows, 0 processed) in 3.31s] +14:03:21 2 of 5 START test unique_my_first_dbt_model_id ................................. [RUN] +14:03:22 2 of 5 PASS unique_my_first_dbt_model_id ....................................... [PASS in 1.55s] +14:03:22 3 of 5 START view model dbt_test1.my_second_dbt_model .......................... [RUN] +14:03:24 3 of 5 OK created view model dbt_test1.my_second_dbt_model ..................... [OK in 1.38s] +14:03:24 4 of 5 START test not_null_my_second_dbt_model_id .............................. [RUN] +14:03:24 5 of 5 START test unique_my_second_dbt_model_id ................................ [RUN] +14:03:25 5 of 5 PASS unique_my_second_dbt_model_id ...................................... [PASS in 1.38s] +14:03:25 4 of 5 PASS not_null_my_second_dbt_model_id .................................... [PASS in 1.42s] +14:03:25 +14:03:25 Finished running 1 table model, 3 tests, 1 view model in 8.44s. +14:03:25 +14:03:25 Completed successfully +14:03:25 +14:03:25 Done. PASS=5 WARN=0 ERROR=0 SKIP=0 TOTAL=5 +Artifact schema version: <https://schemas.getdbt.com/dbt/manifest/v5.json> is above dbt-ol supported version 4. This might cause errors. +Emitting OpenLineage events: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:00&lt;00:00, 274.42it/s] +Emitted 10 openlineage events

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Fenil Doshi + (fdoshi@salesforce.com) +
+
2022-07-27 20:39:21
 +
+

When will the next version of OpenLineage be available tentatively?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-07-27 20:41:44
 +
+

*Thread Reply:* I think it's safe to say we'll see a release by the end of next week

+ + + +
+ :gratitude_thank_you: Fenil Doshi +
+ +
+ 👍 Fenil Doshi +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Yehuda Korotkin + (yehudak@elementor.com) +
+
2022-07-28 04:02:06
 +
+

👋 Hi everyone! +Yesterday was a great presentation by @Julien Le Dem that talked about OpenLineage and did grate comparison between OL and Open-Telemetry, (i wrote a small summary here: https://bit.ly/3z5caOI )

+ +

Julian’s charm sparked inside me curiosity especially regarding OL in streaming. +I saw the design/architecture of OL I got some questions/discussions that I would like to understand better.

+ +

In the context of streaming jobs reporting “start job” - “end job” might be more relevant in the context of a batch mode. +or do you mean reporting start job/end job should be processed each event?

  • and this will be equivalent to starting job each row in a table via UDF, for example.
    • +
+ +

Thank you in advance

+
+
linkedin.com
+ + + +
Yehuda Korotkin on LinkedIn: #new_tech #Kafka #Kafka
+
+ Summey of the meetup: / Kafka alternative / Open-Telemetry / Open-Lineage. was worth going. See details: <b>#new_tech</b> Memphis.dev Idan Asulin Talked about... +
+ + + + + + + + +
Original URL: https://bit.ly/3z5caOI
+ + + +
+ + + +
+ 🙌 Maciej Obuchowski, Michael Robinson, Paweł Leszczyński +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-07-28 08:50:44
 +
+

*Thread Reply:* Welcome to the community!

+ +

We talked about this exact topic in the most recent community call. +https://wiki.lfaidata.foundation/display/OpenLineage/Monthly+TSC+meeting#MonthlyTSCmeeting-Nextmeeting:Nov10th2021(9amPT)

+ +

Discussion: streaming in Flink integration +• Has there been any evolution in the thinking on support for streaming? + ◦ Julien: start event, complete event, snapshots in between limited to certain number per time interval + ◦ Paweł: we can make the snapshot volume configurable +• Does Flink support sending data to multiple tables like Spark? + ◦ Yes, multiple outputs supported by OpenLineage model + ◦ Marquez, the reference implementation of OL, combines the outputs

+ + + +
+ 🙏 Yehuda Korotkin +
+ +
+ ❤️ Julien Le Dem +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-07-28 09:56:05
 +
+

*Thread Reply:* > or do you mean reporting start job/end job should be processed each event? +We definitely want to avoid tracking every single event 🙂

+ +

One thing worth mentioning is that OpenLineage events are meant to be cumulative - the streaming jobs start, run, and eventually finish or restart. In the meantime, we capture additional events "in the middle" - for example, on Apache Flink checkpoint, or every few minutes - where we can emit additional information connected to the state of the job.

+ + + +
+ 🙏 Yehuda Korotkin +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Yehuda Korotkin + (yehudak@elementor.com) +
+
2022-07-28 11:11:17
 +
+

*Thread Reply:* @Will Johnson and @Maciej Obuchowski Thank you for your answer

+ +

jobs start, run, and eventually finish or restart

+ +

This is the perspective that I have a hard time understanding in the context of streaming.

+ +

The classic streaming job should always be on it should not be “finish” event (Except failure). +usually, streaming data is “dripping”.

+ +

It is possible to understand if the job starts/ends in the resolution of the running application and represents when the application begin and when it failed.

+ +

if you do start/stop events from the checkpoints on Flink it might be the wrong representation instead use the concept of event-driven for example reporting state.

+ +

What do you think?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Yehuda Korotkin + (yehudak@elementor.com) +
+
2022-07-28 11:11:36
 +
+

*Thread Reply:*

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-07-28 12:00:34
 +
+

*Thread Reply:* The idea is that jobs usually get upgraded - for example, you change Apache Flink version, increase resources, or change the structure of a job - that's the difference for us. The stop events make sense, because if you for example changed SQL of your Flink SQL job, you probably would want this to be captured - from X to Y job was running with older SQL version well, but after change, the second run started and throughput dropped to 10% of the previous one.

+ +

> if you do start/stop events from the checkpoints on Flink it might be the wrong representation instead use the concept of event-driven for example reporting state. +But this is an misunderstanding 🙂 +The information exposed from a checkpoints are in addition to start and stop events.

+ +

We want to get information from running job - I just argue that sometimes end of a streaming job is also relevant.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-07-28 12:01:16
 +
+

*Thread Reply:* The checkpoint would be captured as a new eventType: RUNNING - do I miss something why you want to add StateFacet?

+ + + +
+ 👍 Yehuda Korotkin +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Yehuda Korotkin + (yehudak@elementor.com) +
+
2022-07-28 14:24:03
 +
+

*Thread Reply:* About argue - it’s depends on what the definition of job in streaming mode, i agree that if you already have ‘job’ you want to know about the job more information.

+ +

each event that entering the sub process (job) should do REST call “Start job” and “End job” ?

+ +

Nope, I just represented two possible ways that i thought, + or StateFacet + or add new Event type eg. RUNNING 😉

+ + + +
+ 👍 Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-07-28 09:14:28
 +
+

Hi everyone, I’d like to request a release to publish the new Flink integration (thanks, @Maciej Obuchowski) and an important fix to the Spark integration (thanks, @Paweł Leszczyński). As per our policy here, 3 +1s from committers will authorize an immediate release. Thanks!

+ + + +
+ ➕ Maciej Obuchowski, Paweł Leszczyński, Willy Lulciuc, Will Johnson, Julien Le Dem +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-07-28 17:30:33
 +
+

*Thread Reply:* Thanks for the +1s. We will initiate the release by Tuesday.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Barak F + (fargoun@gmail.com) +
+
2022-07-28 10:30:15
 +
+

Static code annotations for OpenLineage: hi everyone, i heard yesterday a great lecture by @Julien Le Dem on OpenLineage, and as i'm very interested in this area, i wanted to raise a question: are there any plans to have OpenLineage-like annotations on actual code (e.g. Spark, AirFlow, arbitrary code) to allow deducing some of the lineage informtion from static code analysis?

+ +

The reason i'm asking this is because while OpenLineage does a great job of integrating with multiple platforms (AirFlow, Dbt, Spark), some companies still have a lot of legacy-related data processing stack that will probably not get full OpenLineage (as it's a one-off, and the companies themselves will probably won't implement OpenLineage support for their custom frameworks). +Having some standard way to annotate code with information like: "reads from X; writes to Y; Job name regexp: Z", may allow writing a "generic" OpenLineage colelctor that can go over the source code, collect this configuration information and then use it when constructing the lineage graph (even though it won't be as complete and full as the full OpenLineage info).

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-08-03 08:30:15
 +
+

*Thread Reply:* I think this is an interesting idea, however, just the static analysis does not convey any runtime information.

+ +

We're doing something similar within Airflow now, but as a fallback mechanism: https://github.com/OpenLineage/OpenLineage/pull/914

+ +

You can manually annotate DAG with information instead of writing extractor for your operator. This still gives you runtime information. Similar features might get added to other integrations, especially with such a vast scope as Airflow has - but I think it's unlikely we'd work on a feature for just statically traversing code without runtime context.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Barak F + (fargoun@gmail.com) +
+
2022-08-03 14:25:31
 +
+

*Thread Reply:* Thanks for the detailed response @Maciej Obuchowski! It seems like this solution is specific only to AirFlow, and i wonder why wouldn't we generalize this outside of just AirFlow? My thinking is that there are other areas where there is vast scope (e.g. arbitrary code that does data manipulations), and without such an option, the only path is to provide full runtime information via building your own extractor, which might be a bit hard/expensive to do. +If i understand your response correctly, then you assume that OpenLineage can get wide enough "native" support across the stack without resorting to a fallback like 'static code analysis'. Is that your base assumption?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Petr Hajek + (petr.hajek@profinit.eu) +
+
2022-07-29 04:36:03
 +
+

Hi all, does anybody have an experience extracting Airflow lineage using Marquez as documented here https://www.astronomer.io/guides/airflow-openlineage/#generating-and-viewing-lineage-data ? +We tested it on our Airflow instance with Marquez hoping to get the standard .json files describing lineage in accord with open-lineage model as described in https://json-schema.org/draft/2020-12/schema. +But there seems to be only one GET method related to lineage export in Marquez API library called "Get a lineage graph". This produces quite different .json structure than what we know from open-lineage. Could anybody help if there is a chance to get open-lineage .json structure from Marquez?

+
+
astronomer.io
+ + + +
OpenLineage and Airflow - Airflow Guides
+
+ Using OpenLineage and Marquez to get lineage data from your Airflow DAGs. +
+ + + + + + + + +
Original URL: https://www.astronomer.io/guides/airflow-openlineage/#generating-and-viewing-lineage-data
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-07-29 12:58:38
 +
+

*Thread Reply:* The query API has a different spec than the reporting API, so what you’d get from Marquez would look different from what Marquez receives.

+ +

Few ideas:

+ +
  1. you could send the lineage to a pipedream endpoint to inspect, if you’re just trying to experiment
  2. you could grab them from the lineage table in Marquez’s postgres
    3. +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Petr Hajek + (petr.hajek@profinit.eu) +
+
2022-07-30 16:29:24
 +
+

*Thread Reply:* ok, now I understand, thank you

+ + + +
+ 👍 Jan Kopic +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-08-03 08:25:57
 +
+

*Thread Reply:* FYI we want to have something like that too: https://github.com/MarquezProject/marquez/issues/1927

+ +

But if you need just the raw events endpoint, without UI, then Marquez might be overkill for your needs

+
+ + + + +
<https://github.com/MarquezProject/marquez/issues/1927|#1927 Option to dump OpenLineage events that correspond to dataset/namespace from web>
+
+ <p>Having this feature would make debugging, and replicating errors much faster.</p> +
+ +
+
Comments
+ 2 +
+ +
+
Milestone
+ <a href="https://github.com/MarquezProject/marquez/milestone/4">Roadmap</a> +
+ + + + + + + +
+ + <a href="https://github.com/MarquezProject/marquez">MarquezProject/marquez</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Dinakar Sundar + (dinakar_sundar@condenast.com) +
+
2022-07-30 13:44:13
 +
+

Hi @everyone , we are trying to extract lineage information and import into amundsen .please point us right direction to move - based on the documentation -> Databricks + marquez + amundsen is this the only way to move on ?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john.thomas@astronomer.io) +
+
2022-07-30 13:49:25
 +
+

*Thread Reply:* Short of implementing an open lineage endpoint in Amundsen, yes that's the right approach.

+ +

The Lineage endpoint in Marquez can output the whole graph centered on a node ID, and you can use the jobs/datasets apis to grab lists of each for reference

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Barak F + (fargoun@gmail.com) +
+
2022-07-31 00:35:06
 +
+

*Thread Reply:* Is your lineage information coming via OpenLineage? if so - you can quickly use the Amundsen scripts in order to load data into Amundsen, for example, see this script here: https://github.com/amundsen-io/amundsendatabuilder/blob/master/example/scripts/sample_data_loader.py

+ +

Where is your lineage coming from?

+
+ + + + +
<https://github.com/amundsen-io/amundsendatabuilder/blob/master/example/scripts/sample_data_loader.py | sample_data_loader.py>
+
+ <p>```</p> + +<h1>Copyright Contributors to the Amundsen project.</h1> + +<h1>SPDX-License-Identifier: Apache-2.0</h1> + +<p>""" +This is a example script demonstrating how to load data into Neo4j and +Elasticsearch without using an Airflow DAG.</p> + +<p>It contains several jobs:</p><ul><li><code>run_csv_job</code>: runs a job that extracts table data from a CSV, loads (writes) +this into a different local directory as a csv, then publishes this data to +neo4j.</li><li><code>run_table_column_job</code>: does the same thing as <code>run_csv_job</code>, but with a csv +containing column data.</li><li><code>create_last_updated_job</code>: creates a job that gets the current time, dumps it +into a predefined model schema, and publishes this to neo4j.</li><li><code>create_es_publisher_sample_job</code>: creates a job that extracts data from neo4j +and pubishes it into elasticsearch.</li> +</ul> + +<p>For other available extractors, please take a look at +<a href="https://github.com/amundsen-io/amundsendatabuilder#list-of-extractors">https://github.com/amundsen-io/amundsendatabuilder#list-of-extractors</a> +"""</p> + +<p>import logging +import os +import sys +import uuid</p> + +<p>from elasticsearch import Elasticsearch +from pyhocon import ConfigFactory +from sqlalchemy.ext.declarative import declarative_base</p> + +<p>from databuilder.extractor.csv<em>extractor import ( + CsvColumnLineageExtractor, CsvExtractor, CsvTableBadgeExtractor, CsvTableColumnExtractor, CsvTableLineageExtractor, +) +from databuilder.extractor.es</em>last<em>updated</em>extractor import EsLastUpdatedExtractor +from databuilder.extractor.neo4j<em>search</em>data<em>extractor import Neo4jSearchDataExtractor +from databuilder.job.job import DefaultJob +from databuilder.loader.file</em>system<em>elasticsearch</em>json<em>loader import FSElasticsearchJSONLoader +from databuilder.loader.file</em>system<em>neo4j</em>csv<em>loader import FsNeo4jCSVLoader +from databuilder.publisher.elasticsearch</em>constants import ( + DASHBOARD<em>ELASTICSEARCH</em>INDEX<em>MAPPING, USER</em>ELASTICSEARCH<em>INDEX</em>MAPPING, +) +from databuilder.publisher.elasticsearch<em>publisher import ElasticsearchPublisher +from databuilder.publisher.neo4j</em>csv<em>publisher import Neo4jCsvPublisher +from databuilder.task.task import DefaultTask +from databuilder.transformer.base</em>transformer import ChainedTransformer, NoopTransformer +from databuilder.transformer.dict<em>to</em>model import MODEL<em>CLASS, DictToModel +from databuilder.transformer.generic</em>transformer import ( + CALLBACK<em>FUNCTION, FIELD</em>NAME, GenericTransformer, +)</p> + +<p>es<em>host = os.getenv('CREDENTIALS</em>ELASTICSEARCH<em>PROXY</em>HOST', 'localhost') +neo<em>host = os.getenv('CREDENTIALS</em>NEO4J<em>PROXY</em>HOST', 'localhost')</p> + +<p>es<em>port = os.getenv('CREDENTIALS</em>ELASTICSEARCH<em>PROXY</em>PORT', 9200) +neo<em>port = os.getenv('CREDENTIALS</em>NEO4J<em>PROXY</em>PORT', 7687) +if len(sys.argv) &gt; 1: + es<em>host = sys.argv[1] +if len(sys.argv) &gt; 2: + neo</em>host = sys.argv[2]</p> + +<p>es = Elasticsearch([ + {'host': es<em>host, 'port': es</em>port}, +])</p> + +<p>Base = declarative_base()</p> + +<p>NEO4J<em>ENDPOINT = f'bolt://{neo</em>host}:{neo_port}'</p> + +<p>neo4j<em>endpoint = NEO4J</em>ENDPOINT</p> + +<p>neo4j<em>user = 'neo4j' +neo4j</em>password = 'test'</p> + +<p>LOGGER = logging.getLogger(<strong>name</strong>)</p> + +<p>def run<em>csv</em>job(file<em>loc, job</em>name, model): + tmp<em>folder = f'/var/tmp/amundsen/{job</em>name}' + node<em>files</em>folder = f'{tmp<em>folder}/nodes' + relationship</em>files<em>folder = f'{tmp</em>folder}/relationships'</p> + +<pre><code>csv_extractor = CsvExtractor() +csv_loader = FsNeo4jCSVLoader() + +task = DefaultTask(extractor=csv_extractor, + loader=csv_loader, + transformer=NoopTransformer()) + +job_config = ConfigFactory.from_dict({ + 'extractor.csv.file_location': file_loc, + 'extractor.csv.model_class': model, + 'loader.filesystem_csv_neo4j.node_dir_path': node_files_folder, + 'loader.filesystem_csv_neo4j.relationship_dir_path': relationship_files_folder, + 'loader.filesystem_csv_neo4j.delete_created_directories': True, + 'publisher.neo4j.node_files_directory': node_files_folder, + 'publisher.neo4j.relation_files_directory': relationship_files_folder, + 'publisher.neo4j.neo4j_endpoint': neo4j_endpoint, + 'publisher.neo4j.neo4j_user': neo4j_user, + 'publisher.neo4j.neo4j_password': neo4j_password, + 'publisher.neo4j.neo4j_encrypted': False, + 'publisher.neo4j.job_publish_tag': 'unique_tag', # should use unique tag here like {ds} +}) + +DefaultJob(conf=job_config, + task=task, + publisher=Neo4jCsvPublisher()).launch() +</code></pre> + +<p>def run<em>table</em>badge<em>job(table</em>path, badge<em>path): + tmp</em>folder = '/var/tmp/amundsen/table<em>badge' + node</em>files<em>folder = f'{tmp</em>folder}/nodes' + relationship<em>files</em>folder = f'{tmp<em>folder}/relationships' + extractor = CsvTableBadgeExtractor() + csv</em>loader = FsNeo4jCSVLoader() + task = DefaultTask(extractor=extractor, + loader=csv<em>loader, + transformer=NoopTransformer()) + job</em>config = ConfigFactory.from<em>dict({ + 'extractor.csvtablebadge.table</em>file<em>location': table</em>path, + 'extractor.csvtablebadge.badge<em>file</em>location': badge<em>path, + 'loader.filesystem</em>csv<em>neo4j.node</em>dir<em>path': node</em>files<em>folder, + 'loader.filesystem</em>csv<em>neo4j.relationship</em>dir<em>path': relationship</em>files<em>folder, + 'loader.filesystem</em>csv<em>neo4j.delete</em>created<em>directories': True, + 'publisher.neo4j.node</em>files<em>directory': node</em>files<em>folder, + 'publisher.neo4j.relation</em>files<em>directory': relationship</em>files<em>folder, + 'publisher.neo4j.neo4j</em>endpoint': neo4j<em>endpoint, + 'publisher.neo4j.neo4j</em>user': neo4j<em>user, + 'publisher.neo4j.neo4j</em>password': neo4j<em>password, + 'publisher.neo4j.neo4j</em>encrypted': False, + 'publisher.neo4j.job<em>publish</em>tag': 'unique<em>tag</em>b', # should use unique tag here like {ds} + }) + job = DefaultJob(conf=job_config, + task=task, + publisher=Neo4jCsvPublisher()) + job.launch()</p> + +<p>def run<em>table</em>column<em>job(table</em>path, column<em>path): + tmp</em>folder = '/var/tmp/amundsen/table<em>column' + node</em>files<em>folder = f'{tmp</em>folder}/nodes' + relationship<em>files</em>folder = f'{tmp<em>folder}/relationships' + extractor = CsvTableColumnExtractor() + csv</em>loader = FsNeo4jCSVLoader() + task = DefaultTask(extractor, + loader=csv<em>loader, + transformer=NoopTransformer()) + job</em>config = ConfigFactory.from<em>dict({ + 'extractor.csvtablecolumn.table</em>file<em>location': table</em>path, + 'extractor.csvtablecolumn.column<em>file</em>location': column<em>path, + 'loader.filesystem</em>csv<em>neo4j.node</em>dir<em>path': node</em>files<em>folder, + 'loader.filesystem</em>csv<em>neo4j.relationship</em>dir<em>path': relationship</em>files<em>folder, + 'loader.filesystem</em>csv<em>neo4j.delete</em>created<em>directories': True, + 'publisher.neo4j.node</em>files<em>directory': node</em>files<em>folder, + 'publisher.neo4j.relation</em>files<em>directory': relationship</em>files<em>folder, + 'publisher.neo4j.neo4j</em>endpoint': neo4j<em>endpoint, + 'publisher.neo4j.neo4j</em>user': neo4j<em>user, + 'publisher.neo4j.neo4j</em>password': neo4j<em>password, + 'publisher.neo4j.neo4j</em>encrypted': False, + 'publisher.neo4j.job<em>publish</em>tag': 'unique<em>tag', # should use unique tag here like {ds} + }) + job = DefaultJob(conf=job</em>config, + task=task, + publisher=Neo4jCsvPublisher()) + job.launch()</p> + +<p>def run<em>table</em>lineage<em>job(table</em>lineage<em>path): + tmp</em>folder = '/var/tmp/amundsen/table<em>column' + node</em>files<em>folder = f'{tmp</em>folder}/nodes' + relationship<em>files</em>folder = f'{tmp<em>folder}/relationships' + extractor = CsvTableLineageExtractor() + csv</em>loader = FsNeo4jCSVLoader() + task = DefaultTask(extractor, + loader=csv<em>loader, + transformer=NoopTransformer()) + job</em>config = ConfigFactory.from<em>dict({ + 'extractor.csvtablelineage.table</em>lineage<em>file</em>location': table<em>lineage</em>path, + 'loader.filesystem<em>csv</em>neo4j.node<em>dir</em>path': node<em>files</em>folder, + 'loader.filesystem<em>csv</em>neo4j.relationship<em>dir</em>path': relationship<em>files</em>folder, + 'loader.filesystem<em>csv</em>neo4j.delete<em>created</em>directories': True, + 'publisher.neo4j.node<em>files</em>directory': node<em>files</em>folder, + 'publisher.neo4j…</p> +
+ + + + + + + +
+ + <a href="https://github.com/amundsen-io/amundsendatabuilder">amundsen-io/amundsendatabuilder</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Dinakar Sundar + (dinakar_sundar@condenast.com) +
+
2022-08-01 20:17:22
 +
+

*Thread Reply:* yes @Barak F we are using open lineage

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Barak F + (fargoun@gmail.com) +
+
2022-08-02 01:26:18
 +
+

*Thread Reply:* So, have you tried using Amundsen data builder scripts to load the lineage information into Amundsen? (maybe you'll have to "play" with those a bit)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-08-03 08:24:58
 +
+

*Thread Reply:* AFAIK there is OpenLineage extractor: https://www.amundsen.io/amundsen/databuilder/#openlineagetablelineageextractor

+ +

Not sure it solves your issue though 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Dinakar Sundar + (dinakar_sundar@condenast.com) +
+
2022-08-05 04:46:45
 +
+

*Thread Reply:* thanks

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-08-01 17:08:46
 +
+

@channel +OpenLineage 0.12.0 is now available! +We added: +• an Apache Flink integration, +• support for Spark 3.3.0, +• the ability to extend column level lineage mechanism, +• an ErrorMessageRunFacet to the OpenLineage spec, +• SQLCheckExtractors, a RedshiftSQLExtractor & RedshiftDataExtractor to the Airflow integration, +• a dataset builder to the AlterTableCommand class in the Spark integration. +We changed: +• the filtering of Delta events to reduce noise, +• the flow of metadata in the Airflow integration to allow metadata from Airflow through inlets and outlets. +Thanks to all the contributors who made this release possible! +For the bug fixes and more details, see: +Release: https://github.com/OpenLineage/OpenLineage/releases/tag/0.12.0 +Changelog: https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md +Commit history: https://github.com/OpenLineage/OpenLineage/compare/0.11.0...0.12.0 +Maven: https://oss.sonatype.org/#nexus-search;quick~openlineage +PyPI: https://pypi.org/project/openlineage-python/ (edited)

+ + + +
+ ❤️ Minkyu Park, Harel Shein, Willy Lulciuc, Peter Hicks, Fenil Doshi, Maciej Obuchowski, Howard Yoo, Paul Wilson Villena, Jarek Potiuk, Dinakar Sundar, Shubham Mehta, Sharanya Santhanam, Sheeri Cabral (Collibra) +
+ +
+ 🎉 Minkyu Park, Peter Hicks, Fenil Doshi, Howard Yoo, Jarek Potiuk, Paweł Leszczyński, Ryan Peterson +
+ +
+ 🚀 Minkyu Park, Howard Yoo, Jarek Potiuk +
+ +
+ 🙌 Minkyu Park, Willy Lulciuc, Maciej Obuchowski, Howard Yoo, Jarek Potiuk +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sharanya Santhanam + (santhanamsharanya@gmail.com) +
+
2022-08-02 10:12:01
 +
+

What is the right way of handling/parsing facets on the server side?

+ +

I see the generated server side stubs are generic : https://github.com/OpenLineage/OpenLineage/blob/main/client/java/generator/src/main/java/io/openlineage/client/Generator.java#L131 and dont have any resolved facet information. +Marquez seems to have duplicated the OL model with https://github.com/MarquezProject/marquez/blob/main/api/src/main/java/marquez/service/models/LineageEvent.java#L71 and converts the incoming OL events to a “LineageEvent” for appropriate handling. Is there a cleaner approach where in the known facets can be generated in io.openlineage.server?

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/client/java/generator/src/main/java/io/openlineage/client/Generator.java | Generator.java>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+
+ + + + +
<https://github.com/MarquezProject/marquez/blob/main/api/src/main/java/marquez/service/models/LineageEvent.java | LineageEvent.java>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/MarquezProject/marquez">MarquezProject/marquez</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-08-02 12:28:11
 +
+

*Thread Reply:* I think the reason for server model being very generic is because new facets can be added later (also as custom facets) - and generally server wants to accept all valid events and get the facet information that it can actually use, rather than reject event because it has unknown field.

+ +

Server model was added here after some discussion in Marquez which is relevant - I think @Michael Collado @Willy Lulciuc can add to that

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/pull/300|#300 server model>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+
+ + + + +
<https://github.com/MarquezProject/marquez/issues/1650|#1650 Binary incompatibility after deleting LineageEvent classes>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/MarquezProject/marquez">MarquezProject/marquez</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sharanya Santhanam + (santhanamsharanya@gmail.com) +
+
2022-08-02 15:54:24
 +
+

*Thread Reply:* Thanks for the response. I realize the server stubs were created to support flexibility , but it also makes the parsing logic on server side a bit more complex as we need to maintain code on the server side to look for specific facets & their properties from maps or like maquez duplicate the OL model on our end with the facets we care about. Wanted to know whats the guidance around managing this server side. @Willy Lulciuc @Michael Collado Any suggestions ?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-08-02 18:27:27
 +
+

Agenda items are requested for the next OpenLineage Technical Steering Committee meeting on August 11 at 10am PT. Reply in thread or ping me with your item(s)!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Varun Singh + (varuntestaz@outlook.com) +
+
2022-08-03 04:16:22
 +
+

Hi all, +I am trying out the openlineage spark integration and can't find any column lineage information included with the events. I tried it out with an input dataset where I renamed one of the columns but the columnLineage facet was not present. Can anyone suggest some other examples where it might show up?

+ +

Thanks!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-08-03 04:45:36
 +
+

*Thread Reply:* @Paweł Leszczyński do we collect column level lineage on renames?

+ + + +
+
+
+
+ + + + + +
+ +
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2022-08-05 05:55:12
 +
+

*Thread Reply:* I’ve created an issue for column lineage in case of renaming: +https://github.com/OpenLineage/OpenLineage/issues/993

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/issues/993|#993 collect column level lineage for renames>
+
+ <p>In case of <code>alter table rename column</code> or <code>alter table rename</code> we do not collect column level lineage at the moment, which may be useful in some scenarios.</p> +
+ +
+
Labels
+ integration/spark +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Varun Singh + (varuntestaz@outlook.com) +
+
2022-08-08 09:37:43
 +
+

*Thread Reply:* Thanks @Paweł Leszczyński!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-08-03 12:58:44
 +
+

Hey everyone! I am looking into Fivetran a bit, and it occurs to me that the NAMING.md document does not have an opinion about how to deal with entire systems as datasets. More in 🧵.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-08-03 13:00:22
 +
+

*Thread Reply:* Fivetran is a tool that copies data from source systems to target databases. One of these source systems might be SalesForce, for example.

+ +

This copying results in thousands of SQL queries run against the target database for each sync. I don’t think each of these queries should map to an OpenLineage job, I think the entire synchronization should. Maybe I’m wrong here.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-08-03 13:01:00
 +
+

*Thread Reply:* But if I’m right, that means that there needs to be a way to specify “SalesForce Account #45123452233” as a dataset.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-08-03 13:01:44
 +
+

*Thread Reply:* or it ends up just being a job with outputs and no inputs…but that’s not very illuminating

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-08-03 13:02:27
 +
+

*Thread Reply:* or is that good enough?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-08-04 10:31:11
 +
+

*Thread Reply:* You are looking at a pretty big topic here 🙂

+ +

Basically you're asking what is a job in OpenLineage - and it's not fully answered yet.

+ +

I think the discussion is kinda relevant to this proposed facet and I kinda replied there: https://github.com/OpenLineage/OpenLineage/issues/812#issuecomment-1205337556

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harel Shein + (harel.shein@gmail.com) +
+
2022-08-04 15:50:22
 +
+

*Thread Reply:* my 2 cents on this is that in the Salesforce example, the system is to complex to capture as a single dataset. and so maybe different objects within a salesforce account (org/account/opportunity/etc…) could be treated as individual datasets. But as @Maciej Obuchowski pointed out, this is quite a large topic 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-08-08 13:46:31
 +
+

*Thread Reply:* I guess it depends on whether you actually care about the table/column level lineage for an operation like “copy salesforce to snowflake”.

+ +

I can see it being a nuisance having all of that on a lineage graph. OTOH, I can see it being useful to know that a datum can be traced back to a specific endpoint at SFDC.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-08-08 13:46:55
 +
+

*Thread Reply:* this is a design decision, IMO.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-08-04 11:30:00
 +
+

@channel The next OpenLineage Technical Steering Committee meeting is on Thursday, August 11 at 10 am PT. Join us on Zoom: https://bit.ly/OLzoom +All are welcome! +Agenda:

+ +
  1. Announcements
  2. Docs site update
  3. Release 0.11.0 and 0.12.0 overview
  4. Extractors: examples and how to write them
  5. Open discussion +Notes: https://bit.ly/OLwiki +Is there a topic you think the community should discuss at this or a future meeting? Reply or DM me to add items to the agenda. (edited)
    6. +
+
+
Zoom Video
+ + + +
Join our Cloud HD Video Meeting
+
+ Zoom is the leader in modern enterprise video communications, with an easy, reliable cloud platform for video and audio conferencing, chat, and webinars across mobile, desktop, and room systems. Zoom Rooms is the original software-based conference room solution used around the world in board, conference, huddle, and training rooms, as well as executive offices and classrooms. Founded in 2011, Zoom helps businesses and organizations bring their teams together in a frictionless environment to get more done. Zoom is a publicly traded company headquartered in San Jose, CA. +
+ + + + + + +
Original URL: https://bit.ly/OLzoom
+ + + +
+ + + +
+ 🙌 Maciej Obuchowski, Harel Shein, Paul Wilson Villena +
+ +
+ 👀 Francis McGregor-Macdonald +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Chris Coulthrust + (coulthrust@gmail.com) +
+
2022-08-06 12:06:47
 +
+

👋 Hi everyone!

+ + + +
+ 👋 Jakub Dardziński, Michael Robinson, Ross Turk, Harel Shein, Willy Lulciuc, Howard Yoo +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-08-10 11:00:01
 +
+

@channel The next OpenLineage TSC meeting is tomorrow! https://openlineage.slack.com/archives/C01CK9T7HKR/p1659627000308969

+
+ + +
+ + + } + + Michael Robinson + (https://openlineage.slack.com/team/U02LXF3HUN7) +
+ + + +
+
+ <p><strong>@channel</strong> The next OpenLineage Technical Steering Committee meeting is on <strong>Thursday, August 11 at 10 am PT</strong>. Join us on Zoom: <a href="https://bit.ly/OLzoom">https://bit.ly/OLzoom</a> +All are welcome! +Agenda:</p> + +<ol><li>Announcements</li><li>Docs site update</li><li>Release 0.11.0 and 0.12.0 overview</li><li>Extractors: examples and how to write them</li><li>Open discussion +Notes: <a href="https://bit.ly/OLwiki">https://bit.ly/OLwiki</a> +Is there a topic you think the community should discuss at this or a future meeting? Reply or DM me to add items to the agenda. (edited)</li> +</ol> +
+ + + + + + +
Original URL: https://openlineage.slack.com/archives/C01CK9T7HKR/p1659627000308969
+ + +
+ + Slack Conversation +
+ + +
+ + + +
+ 👀 Howard Yoo +
+ +
+ ❤️ Minkyu Park +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-08-10 22:34:29
 +
+

*Thread Reply:* I am so sad I'm going to miss this month's meeting 😰 Looking forward to the recording!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-08-12 06:19:58
 +
+

*Thread Reply:* We missed you too @Will Johnson 😉

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Raj Mishra + (hax0755@gmail.com) +
+
2022-08-11 18:50:18
 +
+

Hi everyone! I have a REST endpoint that I use for other pipelines that can POST their RunEvent and I forward that to marquez. I'm expecting a JSON which has the RunEvent details, which also has the input or output dataset depending upon the EventType. I can see the Run details always shows up on the marquez UI, but the dataset has issues. I can see the dataset listed but when I can click on it, just shows "something went wrong." I don't see any details of that dataset. +{ + "eventType": "START", + "eventTime": "2022-08-09T19:49:24.201361Z", + "run": { + "runId": "d46e465b-d358-4d32-83d4-df660ff614dd" + }, + "job": { + "namespace": "TEST-NAMESPACE", + "name": "test-job" + }, + "inputs": [ + { + "namespace": "TEST-NAMESPACE", + "name": "my-test-input", + "facets": { + "schema": { + "_producer": "<https://github.com/OpenLineage/OpenLineage/blob/v1-0-0/client>", + "_schemaURL": "<https://github.com/OpenLineage/OpenLineage/blob/v1-0-0/spec/OpenLineage.json#/definitions/SchemaDatasetFacet>", + "fields": [ + { + "name": "a", + "type": "INTEGER" + }, + { + "name": "b", + "type": "TIMESTAMP" + }, + { + "name": "c", + "type": "INTEGER" + }, + { + "name": "d", + "type": "INTEGER" + } + ] + } + } + } + ], + "producer": "<https://github.com/OpenLineage/OpenLineage/blob/v1-0-0/client>" +} +In above payload, the input data set is never created on marquez. I can only see the Run details, but input data set is just empty. Does the input data set needs to created first and then only the RunEvent can be created?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-08-12 06:09:57
 +
+

*Thread Reply:* From the first look, you're missing outputsfield in your event - this might break something

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-08-12 06:10:20
 +
+

*Thread Reply:* If not, then Marquez logs might help to see something

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Raj Mishra + (hax0755@gmail.com) +
+
2022-08-12 13:12:56
 +
+

*Thread Reply:* Does the START event needs to have an output?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-08-12 13:19:24
 +
+

*Thread Reply:* It can have empty output 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-08-12 13:32:43
 +
+

*Thread Reply:* well, in your case you need to send COMPLETE event

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-08-12 13:33:44
 +
+

*Thread Reply:* Internally, Marquez does not create dataset version until you complete event. It makes sense when your semantics are transactional - you can still read from previous dataset version until it's finished writing.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-08-12 13:34:06
 +
+

*Thread Reply:* After I send COMPLETE event with the same information I can see the dataset.

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Raj Mishra + (hax0755@gmail.com) +
+
2022-08-12 13:56:37
 +
+

*Thread Reply:* Thanks for the explanation @Maciej Obuchowski So, if I understand this correct. I won't see the my-test-input dataset till I have the COMPLETE event with input and output?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-08-12 14:34:51
 +
+

*Thread Reply:* @Raj Mishra Yes and no 🙂

+ +

Basically your COMPLETE event does not need to contain any input and output datasets at all - OpenLineage model is cumulative, so it's enough to have datasets on either start or complete. +That also means you can add different datasets in different moment of a run lifecycle - for example, you know inputs, but not outputs, so you emit inputs on START , but not COMPLETE.

+ +

Or, the job is modifying the same dataset it reads from (which happens surprisingly often), Then, you want to collect various input metadata from the dataset before modifying it - most likely you won't have them on COMPLETE 🙂

+ +

In this example I've added my-test-input on START and my-test-input2 on COMPLETE :

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Raj Mishra + (hax0755@gmail.com) +
+
2022-08-12 14:47:56
 +
+

*Thread Reply:* @Maciej Obuchowski Thank you so much! This is great explanation.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sharanya Santhanam + (santhanamsharanya@gmail.com) +
+
2022-08-11 20:28:40
 +
+

Effectively handling file datasets on server side. We have a common usecase where dataset of type is produced/consumed per day. On the Lineage UI/server side it would be ideal to treat all files of this pattern as 1 dataset Vs 1 dataset per daily file. Any suggestions ?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sharanya Santhanam + (santhanamsharanya@gmail.com) +
+
2022-08-11 20:35:33
 +
+

*Thread Reply:* Would adding support for alias/grouping as a config on OL client side be valuable to other users ? i.e OL client could pass down an Alias/grouping facet Or should this be treated purely a server side feature

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-08-12 06:11:21
 +
+

*Thread Reply:* Agreed 🙂

+ +

How do you produce this dataset? Spark integration? Are you using any system like Apache Iceberg/Delta Lake or just writing raw files?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sharanya Santhanam + (santhanamsharanya@gmail.com) +
+
2022-08-12 12:59:48
 +
+

*Thread Reply:* these are raw files written from Spark or map reduce jobs. And downstream Spark jobs read these raw files to produce tables

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-08-12 13:27:34
 +
+

*Thread Reply:* written using Spark dataframe API, like +df.write.format("parquet").save("/tmp/spark_output/parquet") + or RDD?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-08-12 13:27:59
 +
+

*Thread Reply:* the actual API used matters, because we're handling different cases separately

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sharanya Santhanam + (santhanamsharanya@gmail.com) +
+
2022-08-12 13:29:48
 +
+

*Thread Reply:* I see. Let me look that up to be absolutely sure

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sharanya Santhanam + (santhanamsharanya@gmail.com) +
+
2022-08-12 19:21:41
 +
+

*Thread Reply:* It is like. this : df.write.format("parquet").save("/tmp/spark_output/parquet")

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sharanya Santhanam + (santhanamsharanya@gmail.com) +
+
2022-08-15 12:43:45
 +
+

*Thread Reply:* @Maciej Obuchowski curious what you had in mind with respect to RDDs & Dataframes. Also what if we cannot integrate OL with the frameworks that produce this dataset , but only those that consume from the already produced datasets. Is there a way we could still capture the dataset appropriately ?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-08-16 05:30:57
 +
+

*Thread Reply:* @Sharanya Santhanam the naming should be consistent between reading and writing, so it wouldn't change much of you can't integrate OL into writers. For the rest, can you create an issue on OL GitHub so someone can pick it up? I'm at vacation now.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sharanya Santhanam + (santhanamsharanya@gmail.com) +
+
2022-08-16 15:08:41
 +
+

*Thread Reply:* Sounds good , Ty !

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Varun Singh + (varuntestaz@outlook.com) +
+
2022-08-12 06:02:00
 +
+

Hi, Minor Suggestion: +This line https://github.com/OpenLineage/OpenLineage/blob/46efab1e7c2a0aa5ebe8d11185fe8d5225[…]/app/src/main/java/io/openlineage/spark/agent/EventEmitter.java is printing variables like api key and other parameters in the logs. Wouldn't it be more appropriate to use log.debug instead? +I'll create an issue if others agree

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/46efab1e7c2a0aa5ebe8d11185fe8d5225c0f78b/integration/spark/app/src/main/java/io/openlineage/spark/agent/EventEmitter.java | EventEmitter.java>
+
+ <pre><code> "Init OpenLineageContext: Args: %s URI: %s", argument, lineageURI.toString())); +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-08-12 06:09:11
 +
+

*Thread Reply:* yes

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-08-12 06:09:32
 +
+

*Thread Reply:* please do create 🙂

+ + + +
+ ✅ Varun Singh +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Conor Beverland + (conorbev@gmail.com) +
+
2022-08-15 09:01:47
 +
+

dumb question but, is it easy to run all the OpenLineage tests locally? ( and if so how? 🙂 )

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2022-08-17 13:54:19
 +
+

*Thread Reply:* it's per project. +java based: ./gradlew test +python based: https://github.com/OpenLineage/OpenLineage/tree/main/integration/airflow#development

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-08-18 23:45:30
 +
+

Spark Integration: The Order of Processing Events in the Async Event Queue

+ +

Hey, OpenLineage team, I'm working on a PR (https://github.com/OpenLineage/OpenLineage/pull/849/) that is going to store information given in different spark events (e.g. SparkListenerSQLExecutionStart, SparkListenerJobStart).

+ +

However, I want to avoid holding all this data once the execution of the job is complete. As a result, I want to remove the data once I receive a SparkListenerSQLExecutionEnd.

+ +

However, can I be guaranteed that the ExecutionEnd event will be processed AFTER the JobStart event? Is it possible that I can take too long to process the the JobStart event that the ExecutionEnd executes prior to the JobStart finishing?

+ +

I know we do something similar to this with sparkSqlExecutionRegistry (https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/app/src/mai[…]n/java/io/openlineage/spark/agent/OpenLineageSparkListener.java) but do we have any docs to help explain how the AsyncEventQueue orders and consumes events for a listener?

+ +

Thank you so much for any insights

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/pull/849|#849 Feature/spark parent run facet execution>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/app/src/main/java/io/openlineage/spark/agent/OpenLineageSparkListener.java | OpenLineageSparkListener.java>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2022-08-19 18:38:10
 +
+

*Thread Reply:* Hey Will! A bunch of folks are on vacation or out this week. Sorry for the delay, I am personally not sure but if it's not too urgent you can have an answer when knowledgable folks are back.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-08-19 20:21:18
 +
+

*Thread Reply:* Hah! No worries, @Julien Le Dem! I can definitely wait for the lucky people who are enjoying the last few weeks of summer unlike the rest of us 😋

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-08-29 05:31:32
 +
+

*Thread Reply:* @Paweł Leszczyński might want to look at that

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Hanbing Wang + (doris.wang200902@gmail.com) +
+
2022-08-19 01:53:56
 +
+

Hi, +I try to find out if openLineage spark support pyspark (Non-sql) use cases? +Is there any doc I could get more details about non-sql openLineage support? +Thanks a lot

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2022-08-19 12:30:08
 +
+

*Thread Reply:* Hello Hanbing, the spark integration works for PySpark since pyspark is wrapped into regular spark operators.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Hanbing Wang + (doris.wang200902@gmail.com) +
+
2022-08-19 13:49:35
 +
+

*Thread Reply:* @Julien Le Dem Thanks a lot for your help. I searched around, but I couldn't find any doc introduce how pyspark supported in openLineage. +My company want to integrate with openLineage-spark, I am working on figure out what info does OpenLineage make available for non-sql and does it at least have support for logging the logical plan?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2022-08-19 18:26:48
 +
+

*Thread Reply:* Yes, it does send the logical plan as part of the event

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2022-08-19 18:27:32
 +
+

*Thread Reply:* This configuration here should work as well for pyspark https://openlineage.io/docs/integrations/spark/

+
+
openlineage.io
+ + + +
Apache Spark | OpenLineage Docs
+
+ Spark jobs typically run on clusters of machines. A single machine hosts the "driver" application, +
+ + + + + + +
Original URL: https://openlineage.io/docs/integrations/spark/
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2022-08-19 18:28:11
 +
+

*Thread Reply:* --conf "spark.extraListeners=io.openlineage.spark.agent.OpenLineageSparkListener"

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2022-08-19 18:28:26
 +
+

*Thread Reply:* you need to add the jar, set the listener and pass your OL config

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2022-08-19 18:31:11
 +
+

*Thread Reply:* Actually I'm demoing this at 27:10 right here 🙂 https://pretalx.com/bbuzz22/talk/FHEHAL/

+
+
pretalx
+ + + +
Cross-Platform Data Lineage with OpenLineage Berlin Buzzwords 2022
+
+ There are more data tools available than ever before, and it's easier to build a pipeline than it's ever been. This has resulted in an explosion of innovation, but it also means that data within today's organizations has become increasingly distributed. It can't be contained within a single brain, a single team, or a single platform. Data lineage can help by tracing the relationships between datasets and providing a map of your entire data universe. OpenLineage provides a standard for lineage collection that spans multiple platforms, including Apache Airflow, Apache Spark, Flink, and dbt. This empowers teams to diagnose and address widespread data quality and efficiency issues in real time. In this session, Julien Le Dem from Datakin will show how to trace data lineage across Apache Spark and Apache Airflow. He will walk through the OpenLineage architecture and provide a live demo of a running pipeline with real-time data lineage. +
+ + + + + + + + +
Original URL: https://pretalx.com/bbuzz22/talk/FHEHAL/
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2022-08-19 18:32:11
 +
+

*Thread Reply:* you can see the parameters I'm passing to the pyspark command line in the video

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Hanbing Wang + (doris.wang200902@gmail.com) +
+
2022-08-19 18:35:50
 +
+

*Thread Reply:* @Julien Le Dem Thanks for the info, Let me take a look at the video now.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2022-08-19 18:40:10
 +
+

*Thread Reply:* The full demo starts at 24:40. It shows lineage connected together in Marquez coming from 3 different sources: Airflow, Spark and a custom integration

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-08-22 14:32:53
 +
+

Hi everyone, a release has been requested by @Harel Shein. As per our policy here, 3 +1s from committers will authorize an immediate release. Thanks! +Unreleased commits: https://github.com/OpenLineage/OpenLineage/compare/0.12.0...HEAD

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/GOVERNANCE.md | GOVERNANCE.md>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+ ➕ Willy Lulciuc, Michael Robinson, Minkyu Park, Jakub Dardziński, Julien Le Dem +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2022-08-22 14:38:58
 +
+

*Thread Reply:* @Michael Robinson can we start posting the “Unreleased” section in the changelog along with the release request? That way, we / the community will know what will be in the upcoming release

+ + + +
+ 👍 Michael Robinson +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-08-22 15:00:37
 +
+

*Thread Reply:* The release is approved. Thanks @Willy Lulciuc, @Minkyu Park, @Harel Shein

+ + + +
+ 🙌 Willy Lulciuc, Harel Shein +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-08-22 16:18:30
 +
+

@channel +OpenLineage 0.13.0 is now available! +We added: +• BigQuery check support +• RUNNING EventType in the spec and Python client +• databases and schemas to SQL extractors +• an event forwarding feature via HTTP +• Azure Cosmos Handler to the Spark integration +• support for OL datasets in manual lineage inputs/outputs +• ownership facets. +We changed: +• use RUNNING EventType in Flink integration for currently running jobs +• convert task object into JSON encodable when creating Airflow version facet. +Thanks to all the contributors who made this release possible! +For the bug fixes and more details, see: +Release: https://github.com/OpenLineage/OpenLineage/releases/tag/0.13.0 +Changelog: https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md +Commit history: https://github.com/OpenLineage/OpenLineage/compare/0.12.0...0.13.0 +Maven: https://oss.sonatype.org/#nexus-search;quick~openlineage +PyPI: https://pypi.org/project/openlineage-python/ (edited)

+ + + +
+ 🎉 Harel Shein, Ross Turk, Jarek Potiuk, Sheeri Cabral (Collibra), Willy Lulciuc, Howard Yoo, Howard Yoo, Ernie Ostic, Francis McGregor-Macdonald +
+ +
+ ✅ Sheeri Cabral (Collibra), Howard Yoo +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Conor Beverland + (conorbev@gmail.com) +
+
2022-08-23 03:55:24
 +
+

*Thread Reply:* Cool! Are the new ownership facets populated by the Airflow integration ?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
AMRIT SARKAR + (sarkaramrit2@gmail.com) +
+
2022-08-24 08:23:35
 +
+

Hi everyone, excited to work with OpenLineage. I am new to both OpenLineage and Data Lineage in general. Are there working examples/blog posts around actually integrating OpenLineage with existing graph DBs like Neo4J, Neptune etc? (I understand the service layer in between) I understand we have Amundsen with sample open lineage sample data - databuilder/example/sample_data/openlineage/sample_openlineage_events.ndjson. Thanks in advance.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2022-08-25 18:15:59
 +
+

*Thread Reply:* There is not that I know of besides the Amundsen integration example you pointed at. +A basic idea to do such a thing would be to implement an OpenLineage endpoint (receive the lineage events through http posts) and convert them to a format the graph db understand. If others in the community have ideas, please chime in

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
AMRIT SARKAR + (sarkaramrit2@gmail.com) +
+
2022-09-01 13:48:09
 +
+

*Thread Reply:* Understood, thanks a lot Julien. Make sense.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harel Shein + (harel.shein@gmail.com) +
+
2022-08-25 17:30:46
 +
+

Hey all, can I ask for a release for OpenLineage?

+ + + +
+ 👍 Harel Shein, Minkyu Park, Michael Robinson, Michael Collado, Ross Turk, Julien Le Dem, Willy Lulciuc, Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2022-08-25 17:32:44
 +
+

*Thread Reply:* @Michael Robinson ^

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-08-25 17:34:04
 +
+

*Thread Reply:* Thanks, Harel. 3 +1s from committers is all we need to make this happen today.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Minkyu Park + (minkyu@datakin.com) +
+
2022-08-25 17:52:40
 +
+

*Thread Reply:* 🙏

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-08-25 18:09:51
 +
+

*Thread Reply:* Thanks, all. The release is authorized

+ + + +
+ 🎉 Willy Lulciuc +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2022-08-25 18:16:44
 +
+

*Thread Reply:* can you also state the main purpose for this release?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-08-25 18:25:49
 +
+

*Thread Reply:* I believe (correct me if wrong, @Harel Shein) that this is to make available a fix of a bug in the compare functionality

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Minkyu Park + (minkyu@datakin.com) +
+
2022-08-25 18:27:53
 +
+

*Thread Reply:* ParentRunFacet from the airflow integration is not compliant to OpenLineage spec and this release includes the fix of that so that the marquez can handle parent run/job information.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-08-25 18:49:30
 +
+

@channel +OpenLineage 0.13.1 is now available! +We fixed: +• Rename all parentRun occurrences to parent from Airflow integration #1037 @fm100 +• Do not change task instance during on_running event #1028 @JDarDagran +Release: https://github.com/OpenLineage/OpenLineage/releases/tag/0.13.1 +Changelog: https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md +Commit history: https://github.com/OpenLineage/OpenLineage/compare/0.13.0...0.13.1 +Maven: https://oss.sonatype.org/#nexus-search;quick~openlineage +PyPI: https://pypi.org/project/openlineage-python/

+ + + +
+ 🎉 Harel Shein, Minkyu Park, Ross Turk, Michael Collado, Howard Yoo +
+ +
+ ❤️ Minkyu Park, Ross Turk, Howard Yoo +
+ +
+ 🥳 Minkyu Park, Ross Turk, Howard Yoo +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason + (shzhan@coupang.com) +
+
2022-08-26 18:58:17
 +
+

Hi, I am new to openlineage. Any one know how to enable spark column level lineage? I saw the code comment, it said default is disabled, thanks

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harel Shein + (harel.shein@gmail.com) +
+
2022-08-26 19:26:22
 +
+

*Thread Reply:* What version of Spark are you using? it should be enabled by default for Spark 3 +https://openlineage.io/docs/integrations/spark/spark_column_lineage

+
+
openlineage.io
+ + + +
Column Level Lineage | OpenLineage Docs
+
+ Column level lineage for Spark is turned on by default and requires no additional work to be done. The following documentation describes its internals. +
+ + + + + + +
Original URL: https://openlineage.io/docs/integrations/spark/spark_column_lineage
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason + (shzhan@coupang.com) +
+
2022-08-26 20:21:12
 +
+

*Thread Reply:* Thanks. Good to here that. I am use 0.9.+ . I will try again

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason + (shzhan@coupang.com) +
+
2022-08-29 13:14:01
 +
+

*Thread Reply:* I tested 0.9.+ 0.12.+ with spark 3.0 and 3.2 version. There still do not have dataset facet columnlineage. This is strange. I saw the column lineage design proposals 148. It should support from 0.9.+ Do I miss something?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason + (shzhan@coupang.com) +
+
2022-08-29 13:14:41
 +
+

*Thread Reply:* @Harel Shein

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-08-30 00:56:18
 +
+

*Thread Reply:* @Jason it depends on the data source. What sort of data are you trying to read? Is it in a hive metastore? Is it on an S3 bucket? Is it a delta file format?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason + (shzhan@coupang.com) +
+
2022-08-30 13:51:03
 +
+

*Thread Reply:* I tried read hive megastore on s3 and cave file on local. All are miss the columnlineage

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-08-31 00:33:17
 +
+

*Thread Reply:* @Jason - Sorry, you'll have to translate a bit for me. Can you share a snippet of code you're using to do the read and write? Is it a special package you need to install or is it just using the hadoop standard for S3? https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/index.html

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason + (shzhan@coupang.com) +
+
2022-08-31 20:00:47
 +
+

*Thread Reply:* spark.read \ + .option("header", "true") \ + .option("inferschema", "true") \ + .csv("data/input/batch/wikidata.csv") \ + .write \ + .mode('overwrite') \ + .csv("data/output/batch/python-sample.csv")

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason + (shzhan@coupang.com) +
+
2022-08-31 20:01:21
 +
+

*Thread Reply:* This is simple code run on my local for testing

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-08-31 21:41:31
 +
+

*Thread Reply:* Which version of OpenLineage are you running? You might look at the code on the main branch. This looks like a HadoopFSRelation which I implemented for column lineage but the latest release (0.13.1) does not include it yet.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-08-31 21:42:05
 +
+

*Thread Reply:* Specifically this commit is what implemented it. +https://github.com/OpenLineage/OpenLineage/commit/ce30178cc81b63b9930be11ac7500ed34808edd3

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason + (shzhan@coupang.com) +
+
2022-08-31 22:02:16
 +
+

*Thread Reply:* I see. I use 0.13.0

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harel Shein + (harel.shein@gmail.com) +
+
2022-09-01 12:04:41
 +
+

*Thread Reply:* @Jason we have our monthly release coming up now, so it should be included in 0.14.0 when released today/tomorrow

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason + (shzhan@coupang.com) +
+
2022-09-01 12:52:52
 +
+

*Thread Reply:* Great. Thanks Harel.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Raj Mishra + (hax0755@gmail.com) +
+
2022-08-28 17:46:38
 +
+

Hi! I have ran into some issues and wanted to clarify my doubts. +• Why are input schema changes(column delete, new columns) doesn't show up on the UI. I have changed the input schema for the same job, but I'm not seeing getting updated on the UI. +• Why is there only ever 1 input schema version. Every change I make in input schema, I only see output schema has multiple versions but only 1 version for input schema. +• Is there a reason why can't we see the input schema till the COMPLETE event is posted? +I have used the examples from here. https://openlineage.io/getting-started/ +curl -X POST <http://localhost:5000/api/v1/lineage> \ + -H 'Content-Type: application/json' \ + -d '{ + "eventType": "START", + "eventTime": "2020-12-28T19:52:00.001+10:00", + "run": { + "runId": "d46e465b-d358-4d32-83d4-df660ff614dd" + }, + "job": { + "namespace": "my-namespace", + "name": "my-job" + }, + "inputs": [{ + "namespace": "my-namespace", + "name": "my-input" + }], + "producer": "<https://github.com/OpenLineage/OpenLineage/blob/v1-0-0/client>" + }' +curl -X POST <http://localhost:5000/api/v1/lineage> \ + -H 'Content-Type: application/json' \ + -d '{ + "eventType": "COMPLETE", + "eventTime": "2020-12-28T20:52:00.001+10:00", + "run": { + "runId": "d46e465b-d358-4d32-83d4-df660ff614dd" + }, + "job": { + "namespace": "my-namespace", + "name": "my-job" + }, + "outputs": [{ + "namespace": "my-namespace", + "name": "my-output", + "facets": { + "schema": { + "_producer": "<https://github.com/OpenLineage/OpenLineage/blob/v1-0-0/client>", + "_schemaURL": "<https://github.com/OpenLineage/OpenLineage/blob/v1-0-0/spec/OpenLineage.json#/definitions/SchemaDatasetFacet>", + "fields": [ + { "name": "a", "type": "VARCHAR"}, + { "name": "b", "type": "VARCHAR"} + ] + } + } + }], + "producer": "<https://github.com/OpenLineage/OpenLineage/blob/v1-0-0/client>" + }' +Changing the inputs schema for START doesn't change the schema input version and doesn't update the UI. +Thanks!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-08-29 05:29:52
 +
+

*Thread Reply:* Reading dataset - which input dataset implies - does not mutate the dataset 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-08-29 05:30:14
 +
+

*Thread Reply:* If you change the dataset, it would be represented as some other job with this datasets in the outputs list

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Raj Mishra + (hax0755@gmail.com) +
+
2022-08-29 12:42:55
 +
+

*Thread Reply:* So, changing the input dataset will always create new output data versions? Sorry I have trouble understanding this, but if the input is changing, shouldn't the input data set will have different versions?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-09-01 08:35:42
 +
+

*Thread Reply:* @Raj Mishra if input is changing, there should be something else in your data infrastructure that changes this dataset - and it should emit this dataset as output

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Iftach Schonbaum + (iftach.schonbaum@hunters.ai) +
+
2022-08-29 12:21:52
 +
+

Hi Everyone, new here. i went thourhg the docs and examples. cant seem to understand how can i model views on top of base tables if not from a data processing job but rather via modeling something static that is coming from some software internals. i.e. i want to issue the lineage my self rather it will learn it dynamically from some Airflow DAG or spark DAG

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-08-29 12:35:32
 +
+

*Thread Reply:* I think you want to emit raw events using python or java client: https://openlineage.io/docs/client/python

+
+
openlineage.io
+ + + +
Python | OpenLineage Docs
+
+ Overview +
+ + + + + + +
Original URL: https://openlineage.io/docs/client/python
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-08-29 12:35:46
 +
+

*Thread Reply:* (docs in progress 😉)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Iftach Schonbaum + (iftach.schonbaum@hunters.ai) +
+
2022-08-30 02:07:02
 +
+

*Thread Reply:* can you give a hind what should i look for for modeling a dataset on top of other dataset? potentially also map columns?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Iftach Schonbaum + (iftach.schonbaum@hunters.ai) +
+
2022-08-30 02:12:50
 +
+

*Thread Reply:* i can only see that i can have a dataset as input to a job run and not for another dataset

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-09-01 08:34:35
 +
+

*Thread Reply:* Not sure I understand - jobs process input datasets into output datasets. There is always something that can be modeled into a job that consumes input and produces output.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Iftach Schonbaum + (iftach.schonbaum@hunters.ai) +
+
2022-09-01 10:30:51
 +
+

*Thread Reply:* so openlineage force me to put a job between datasets? does not fit our use case

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Iftach Schonbaum + (iftach.schonbaum@hunters.ai) +
+
2022-09-01 10:31:09
 +
+

*Thread Reply:* unless we can some how easily hide the process that does that on the graph.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason + (shzhan@coupang.com) +
+
2022-08-29 20:41:19
 +
+

QQ, I saw that spark Column level lineage start with open lineage 0.9.+ version with spark 3.+, Does it mean it needs to run lower than open lineage 0.9 if our spark is 2.3 or 2.4?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-08-30 04:44:06
 +
+

*Thread Reply:* I don't think it will work for Spark 2.X.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason + (shzhan@coupang.com) +
+
2022-08-30 13:42:20
 +
+

*Thread Reply:* Is there have plan to support spark 2.x?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-08-30 14:00:38
 +
+

*Thread Reply:* Nope - on the other hand we plan to drop any support for it, as it's unmaintained for quite a bit and vendors are dropping support for it too - afaik Databricks in April 2023.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason + (shzhan@coupang.com) +
+
2022-08-30 17:19:43
 +
+

*Thread Reply:* I see. Thanks. Amazon Emr still support spark 2.x

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-08-30 01:15:10
 +
+

Spark Integration: Handling Data Source V2 API datasets

+ +

Is it expected that a DataSourceV2 relation has a start event with inputs and outputs but a complete event with only outputs? Based on @Michael Collado’s previous comments, I think it's fair to say YES this is expected and we just need to handle it. https://openlineage.slack.com/archives/C01CK9T7HKR/p1645037070719159?thread_ts=1645036515.163189&cid=C01CK9T7HKR

+ +

@Hanna Moazam and I noticed this behavior when we looked at the Cosmos Db visitor and then reproduced it for the Iceberg visitor. We traced it down to the fact that the AbstractQueryPlanInputDatasetBuilder (which is the parent of DataSourceV2RelationInputDatasetBuilder) has an isDefinedAt that only includes SparkListenerJobStart and SparkListenerSQLExecutionStart

+ +

This means an Iceberg COMPLETE event will NEVER contain inputs because the isDefinedAt will always be false (since COMPLETE only fires for JobEnd and ExecutionEnd events). Does that sound correct (@Paweł Leszczyński)?

+ +

It seems that Delta tables (or at least Delta on Databricks) does not follow this same code path and as a result our complete events includes outputs AND inputs.

+
+ + +
+ + + } + + Michael Collado + (https://openlineage.slack.com/team/U01NNCBCP6K) +
+ + + +
+
+ <p>Hey, I <a href="https://github.com/OpenLineage/OpenLineage/pull/490#issuecomment-1042011803">responded on the issue</a>, but just to make it clear for everyone, the OL events for a run are not expected to be an accumulation of all past events. Events should be treated as additive by the backend - each event can post what information it has about the run and the backend is responsible for constructing a holistic picture of the run</p> +
+ + + + + + +
Original URL: https://openlineage.slack.com/archives/C01CK9T7HKR/p1645037070719159?thread_ts=1645036515.163189&amp;cid=C01CK9T7HKR
+ + +
+ + Thread in Slack Conversation +
+ + +
+ + + +
+ 👀 Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-09-01 05:56:13
 +
+

*Thread Reply:* At least for Iceberg I've done it, since I want to emit DatasetVersionDatasetFacet for input dataset only at START - and after I finish writing the dataset might have different version than before writing.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-09-01 05:58:59
 +
+

*Thread Reply:* Same should be for output AFAIK - output version should be emitted only on COMPLETE, since the version changes after I finish writing.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-09-01 09:52:30
 +
+

*Thread Reply:* Ah! Okay, so this still requires us to truly combine START and COMPLETE to get a TOTAL picture of the entire run. Is that fair?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-09-01 10:30:41
 +
+

*Thread Reply:* Yes

+ + + +
+ 👍 Will Johnson +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-09-01 10:31:21
 +
+

*Thread Reply:* As usual, thank you Maciej for the responses and insights!

+ + + +
+ 🙌 Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason + (shzhan@coupang.com) +
+
2022-08-31 22:19:44
 +
+

QQ team, I use spark sql with openlineage namespace weblog: spark.sql(“select ** from weblog where dt=‘1’”).write.orc(“…”) there have two issues 1, there have no upstream dataset weblog on Marquez UI. 2, there have new namespace s3-cdp-prod-hive created. It should the bucket of s3. Am I missing something? Thanks

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason + (shzhan@coupang.com) +
+
2022-09-07 14:13:34
 +
+

*Thread Reply:* Anyone can help for it? Does I miss something

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason + (shzhan@coupang.com) +
+
2022-08-31 22:21:57
 +
+

Here is the Marquez UI

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-09-01 07:34:24
 +
+

Hi everyone, I’m opening up a vote on this month’s OpenLineage release. 3 +1s from committers will authorize. Additions include support for KustoRelationHandler in Kusto (Azure Data Explorer) and for ABFSS and Hadoop Logical Relation, both in the Spark integration. All commits can be found here: https://github.com/OpenLineage/OpenLineage/compare/0.13.1...HEAD. Thanks in advance!

+ + + +
+ ➕ Maciej Obuchowski, Ross Turk, Paweł Leszczyński, Will Johnson, Hanna Moazam +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-09-01 13:18:59
 +
+

*Thread Reply:* Thanks. The release is authorized. It will be initiated within 2 business days.

+ + + +
+ 🙌 Will Johnson, Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
srutikanta hota + (srutikanta.hota@gmail.com) +
+
2022-09-05 07:57:02
 +
+

Is there a reference on how to deploy openlineage on a Non AWS infrastructure ?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-09-08 10:31:44
 +
+

*Thread Reply:* Which integration are you looking to implement?

+ +

And what environment are you looking to deploy it on? The Cloud? On-Prem?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
srutikanta hota + (srutikanta.hota@gmail.com) +
+
2022-09-08 10:40:11
 +
+

*Thread Reply:* We are planning to deploy on premise with Kerberos as authentication for postgres

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-09-08 11:27:06
 +
+

*Thread Reply:* Ah! Are you planning on running Marquez as well and that is your main concern or are you planning on building your own store of OpenLineage Events and using the SQL integration to generate those events?

+ +

https://github.com/OpenLineage/OpenLineage/tree/main/integration

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
srutikanta hota + (srutikanta.hota@gmail.com) +
+
2022-09-08 11:33:44
 +
+

*Thread Reply:* I am looking to deploy Marquez on-prem with onprem postgres as back-end with Kerberos authentication.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
srutikanta hota + (srutikanta.hota@gmail.com) +
+
2022-09-08 11:34:32
 +
+

*Thread Reply:* Is the the right forum for Marquez as well or there is different slack channel for Marquez available

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-09-08 11:46:35
 +
+

*Thread Reply:* https://bit.ly/MarquezSlack

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-09-08 11:47:14
 +
+

*Thread Reply:* There is another slack channel just for Marquez! That might be a better spot with more dedicated Marquez developers.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-09-06 15:52:32
 +
+

@channel +OpenLineage 0.14.0 is now available! +We added: +• Support ABFSS and Hadoop Logical Relation in Column-level lineage #1008 @wjohnson +• Add Kusto relation visitor #939 @hmoazam +• Add ColumnLevelLineage facet doc #1020 @julienledem +• Include symlinks dataset facet #935 @pawel-big-lebowski +• Add support for dbt 1.3 beta’s metadata changes #1051 @mobuchowski +• Support Flink 1.15 #1009 @mzareba382 +• Add Redshift dialect to the SQL integration #1066 @mobuchowski +We changed: +• Make the timeout configurable in the Spark integration #1050 @tnazarew +We fixed: +• Add a dialect parameter to Great Expectations SQL parser calls #1049 @collado-mike +• Fix Delta 2.1.0 with Spark 3.3.0 #1065 @pawel-big-lebowski +Release: https://github.com/OpenLineage/OpenLineage/releases/tag/0.14.0 +Changelog: https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md +Commit history: https://github.com/OpenLineage/OpenLineage/compare/0.13.1...0.14.0 +Maven: https://oss.sonatype.org/#nexus-search;quick~openlineage +PyPI: https://pypi.org/project/openlineage-python/

+ + + +
+ ❤️ Willy Lulciuc, Howard Yoo, Alexander Wagner, Hanna Moazam, Minkyu Park, Grayson Stream, Paweł Leszczyński, Maciej Obuchowski, Conor Beverland, Jason +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2022-09-06 15:54:30
 +
+

*Thread Reply:* Thanks for breaking up the changes in the release! Love the new format 💯

+ + + +
+ 🙌 Michael Robinson +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-09-07 09:05:35
 +
+

Hello all, I’m requesting a patch release to fix a bug in the Spark integration. Currently, OpenlineageSparkListener fails when no openlineage.timeout is provided. PR #1069 by @Paweł Leszczyński, merged today, will fix it. As per our policy here, 3 +1s from committers will authorize an immediate release.

+ + + +
+ ➕ Paweł Leszczyński, Maciej Obuchowski, Howard Yoo, Willy Lulciuc, Ross Turk, Julien Le Dem +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2022-09-07 10:00:11
 +
+

*Thread Reply:* Is PR #1069 all that’s going in 0.14.1 ?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-09-07 10:27:39
 +
+

*Thread Reply:* There’s also 1058. 1069 is urgently needed. We can technically wait…

+ + + +
+ 🙌 Willy Lulciuc +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-09-07 10:30:31
 +
+

*Thread Reply:* (edited prior message because I’m not sure how accurately I was describing the issue)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2022-09-07 10:39:32
 +
+

*Thread Reply:* Thanks for clarifying!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-09-07 10:50:29
 +
+

*Thread Reply:* Thanks, all. The release is authorized.

+ + + +
+ ❤️ Willy Lulciuc +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-09-07 11:04:39
 +
+

*Thread Reply:* 1058 also fixes some bugs

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Iftach Schonbaum + (iftach.schonbaum@hunters.ai) +
+
2022-09-08 01:55:41
 +
+

Hello all, question: Views on top of base table is also a use case for lineage and there is no job in between. i dont seem to find a way to have a dataset on top of others to represent a view on top of tables. is there a way to do that without a job in between?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-09-08 04:41:07
 +
+

*Thread Reply:* Usually there is something creating the view, for example dbt materialization: https://docs.getdbt.com/docs/building-a-dbt-project/building-models/materializations

+ +

Besides that, there is this proposal that did not get enough love yet https://github.com/OpenLineage/OpenLineage/issues/323

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Iftach Schonbaum + (iftach.schonbaum@hunters.ai) +
+
2022-09-08 04:53:23
 +
+

*Thread Reply:* but we are not working iwth dbt. we try to model lineage of our internal view/tables hirarchy which is related to a propriety application of ours. so we like OpenLineage that lets me explicily model stuff and not only via scanning some DW. but in that case we dont want a job in between.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Iftach Schonbaum + (iftach.schonbaum@hunters.ai) +
+
2022-09-08 04:58:47
 +
+

*Thread Reply:* this PR does not seem to support lineage between datasets

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-09-08 12:49:48
 +
+

*Thread Reply:* This is something core to the OpenLineage design - the lineage relationships are defined as dataset-job-dataset, not dataset-dataset.

+ +

In OpenLineage, something observes the lineage relationship being created.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-09-08 12:50:13
 +
+

*Thread Reply:*

+ +
+
+ + + + + + +
+ + +
+ 🙌 Will Johnson, Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-09-08 12:51:15
 +
+

*Thread Reply:* It’s a bit different from some other lineage approaches, but OL is intended to be a push model. A job is observed as it runs, metadata is pushed to the backend.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-09-08 12:54:27
 +
+

*Thread Reply:* so in this case, according to openlineage 🙂, the job would be whatever runs within the pipeline that creates the view. very operational point of view.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Iftach Schonbaum + (iftach.schonbaum@hunters.ai) +
+
2022-09-11 12:27:42
 +
+

*Thread Reply:* but what about the view definition use case? u have lineage of columns in view/base table relation ships

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Iftach Schonbaum + (iftach.schonbaum@hunters.ai) +
+
2022-09-11 12:28:05
 +
+

*Thread Reply:* how would you model that in OpenLineage? would you create a dummy job ?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Iftach Schonbaum + (iftach.schonbaum@hunters.ai) +
+
2022-09-11 12:31:57
 +
+

*Thread Reply:* would you say that because this is my use case i might better choose some other lineage tool?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Iftach Schonbaum + (iftach.schonbaum@hunters.ai) +
+
2022-09-11 12:33:04
 +
+

*Thread Reply:* for the context: i am not talking about some view and table definitions in some warehouse e.g. SF but its internal data processing mechanism with propriety view/tables definition (in Flink SQL) and we want to push this metadata for visibility

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-09-12 17:20:13
 +
+

*Thread Reply:* Ah, gotcha. Yeah, I would say it’s probably best to create a job in this case. You can send the view definition using a sourcecodefacet, so it will be collected as well. You’d want to send START and STOP events for it.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-09-12 17:22:03
 +
+

*Thread Reply:* regarding the PR linked before, you are right - I wonder if someday the spec should have a way to express “the system was made aware that these datasets are related, but did not observe the relationship being created so it can’t tell you i.e. how long it took or whether it changed over time”

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-09-09 10:25:21
 +
+

@channel +OpenLineage 0.14.1 is now available! +We fixed: +• Fix Spark integration issues including error when no openlineage.timeout #1069 @pawel-big-lebowski +Bug fixes were also included in this release. +Release: https://github.com/OpenLineage/OpenLineage/releases/tag/0.14.1 +Changelog: https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md +Commit history: https://github.com/OpenLineage/OpenLineage/compare/0.14.0...0.14.1 +Maven: https://oss.sonatype.org/#nexus-search;quick~openlineage +PyPI: https://pypi.org/project/openlineage-python/

+ + + +
+ 🙌 Maciej Obuchowski, Willy Lulciuc, Howard Yoo, Francis McGregor-Macdonald, AMRIT SARKAR +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
data_fool + (data.fool.me@gmail.com) +
+
2022-09-09 13:52:39
 +
+

Hello, any future plans for integrating Airbyte with openlineage?

+ + + +
+ 👋 Willy Lulciuc, Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2022-09-09 14:01:13
 +
+

*Thread Reply:* Hey, @data_fool! Not in the near term. but of course we’d love to see this happen. We’re open to having an Airbyte integration driven by the community. Want to open an issue to start the discussion?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
data_fool + (data.fool.me@gmail.com) +
+
2022-09-09 15:36:20
 +
+

*Thread Reply:* hey @Willy Lulciuc, Yep, will open an issue. Thanks!

+ + + +
+ 🙌 Willy Lulciuc +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Hubert Dulay + (hubert.dulay@gmail.com) +
+
2022-09-10 22:00:10
 +
+

Hi can you create lineage across namespaces? Thanks

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2022-09-12 19:26:25
 +
+

*Thread Reply:* yes!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
srutikanta hota + (srutikanta.hota@gmail.com) +
+
2022-09-26 10:31:56
 +
+

*Thread Reply:* Any example or ticket on how to lineage across namespace

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Iftach Schonbaum + (iftach.schonbaum@hunters.ai) +
+
2022-09-12 02:27:49
 +
+

Hello, Does OpenLineage support column level lineage?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-09-12 04:56:13
 +
+

*Thread Reply:* Yes https://openlineage.io/blog/column-lineage/

+
+
openlineage.io
+ + + +
The Current State of Column-level Lineage
+
+ Column-level lineage helps organizations navigate a complex regulatory landscape. +
+ + + + + + +
Original URL: https://openlineage.io/blog/column-lineage/
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2022-09-22 02:18:45
 +
+

*Thread Reply:* • More details on Spark & Column level lineage integration: https://openlineage.io/docs/integrations/spark/spark_column_lineage +• Proposal on how to implement column level lineage in Marquez (implementation is currently work in progress): https://github.com/MarquezProject/marquez/blob/main/proposals/2045-column-lineage-endpoint.md +@Iftach Schonbaum let us know if you find the information useful.

+
+
openlineage.io
+ + + +
Column Level Lineage | OpenLineage Docs
+
+ Column level lineage for Spark is turned on by default and requires no additional work to be done. The following documentation describes its internals. +
+ + + + + + +
Original URL: https://openlineage.io/docs/integrations/spark/spark_column_lineage
+ + + +
+
+ + + + +
<https://github.com/MarquezProject/marquez/blob/main/proposals/2045-column-lineage-endpoint.md | 2045-column-lineage-endpoint.md>
+
+ <p><strong>Proposal: Column lineage endpoint proposal</strong></p> + +<p>Author(s): <a href="https://github.com/julienledem">@julienledem</a>, <a href="https://github.com/pawel-big-lebowski">@pawel-big-lebowski</a></p> + +<p>Created: 20022-08-18</p> + +<p>Dicussion: <a href="https://github.com/MarquezProject/marquez/issues/2045">column lineage endpoint issue #2045</a></p> + +<p><strong>Overview</strong> +<strong>Use cases</strong></p> + +<p>• Find the current upstream dependencies of a column. A column in a dataset is derived from columns in upstream datasets. +• See column-level lineage in the dataset level lineage when available. +• Retrieve point-in-time upstream lineage for a dataset or a column. What did the lineage look like yesterday compared to today?</p> + +<p><strong>Existing elements</strong></p> + +<p>• OpenLineage defines a [column-level lineage facet]- (<a href="https://github.com/OpenLineage/OpenLineage/blob/main/spec/facets/ColumnLineageDatasetFacet.json">https://github.com/OpenLineage/OpenLineage/blob/main/spec/facets/ColumnLineageDatasetFacet.json</a>). +• Marquez has a lineage endpoint <code>GET /api/v1/lineage</code> that returns the current lineage graph connected to a job or a dataset</p> + +<p><strong>Column lineage characteristics and general assumptions</strong></p> + +<p>Column level lineage is a different lineage graph due to a different node granularity - kind of zoomed-in view of existing lineage. Instead of datasets and jobs being lineage graph nodes, each dataset field becomes a node. Additionally, there are edges between dataset fields, instead of datasets itself. Thus, enriching existing lineage with column lineage information would not be sufficient. That’s why we propose another API endpoint with column lineage graph.</p> + +<p>Upstream and downstream edges do have different characteristics. An output dataset is always produced by a single version of input dataset (one upstream), while a single input datset version can have multiple output dataset versions. Lineage graph can be then easily flooded by downstream subgraph which blurs the overall view. That's why we consider an upstream column lineage as a default one. Downstream lineage will be returned only when requested explicitly.</p> + +<p><strong>New Elements</strong></p> + +<p>We propose the following changes:</p> + +<p>• Add column lineage to the dataset resource endpoint. Column lineage will NOT be added to existing <code>/lineage</code> endpoint as it may be a heavy database operation run on each lineage graph's node which we want to avoid. Based on that, column level lineage get be requested per dataset in separate requests when required. +• A new column-lineage endpoint leveraging the column lineage facet to retrieve lineage for a given column. +• Point-in-time upstream (dataset or column level) lineage given a version of a dataset.</p> + +<p><strong>Proposal</strong> +<strong>Add column lineage to existing datasets endpoint</strong></p> + +<p>In the <code>GET /api/v1/namespaces/{namespace}/datasets</code> api, add column lineage facet to returned dataset resource.</p> + +<p><strong>Add a column-level-lineage endpoint:</strong></p> + +<p>New endpoints to retrieve a column lineage of a single field or a whole dataset will be added:</p> + +<pre><code>GET /column-lineage?nodeId=dataset:{namespace}:{dataset} +GET /column-lineage?nodeId=datasetField:{namespace}:{dataset}:{field} +</code></pre> + +<p>For example:</p> + +<pre><code>GET /column-lineage?nodeId=dataset:food_delivery:public.delivery_7_days +GET /column-lineage?nodeId=datasetField:food_delivery:public.delivery_7_days:a +</code></pre> + +<p>Although creating a new endpoint, we would like to reuse existing data structures with a new <code>NodeType.FIELD</code> introduced.</p> + +<p>The logic returns dataset field node:</p> + +<pre><code>GET /column-lineage?nodeId=datasetField:db1:table1:a +... +{ + graph: [ + { + "id": "datasetField:db1:table1:a", + "type": "DATASET_FIELD", + "data": { + "namespace": "DB1", + "name": "table2", + "field": "a", + "type": "integer", + "transformationDescription": "identical", + "transformationType": "IDENTITY", + "inputFields": [ + { "namespace": "DBA", "name": "tableA", "field": "columnA"}, + { "namespace": "DBB", "name": "tableB", "field": "columnB"}, + { "namespace": "DBC", "name": "tableC", "field": "columnC"} + ] + "inEdges": [ + { + "origin": "datasetField:db1:table1:a", + "destination": "datasetField:DBA:tableA:columnA" + }, + { + "origin": "datasetField:db1:table1:a", + "destination": "datasetField:DBB:tableB:columnB" + }, + { + "origin": "datasetField:db1:table1:a", + "destination": "datasetField:DBB:tableB:columnC" + } + ], + }, + ... + # Input fields, present within "inEdges", can be also returned within a graph due to a `depth` parameter greate than 0. + } + ] +} +</code></pre> + +<p>The <code>depth</code> parameter controls how many edges, from a given dataset field, shall be returned. The default is set to <code>0</code>. In case of default equal <code>1</code>, each <code>inputField</code> will be returned as a separate node within a response graph with <code>inputFields</code> used to produce it. Please note that extending depth may increase the graph size and affect request performance.</p> + +<p>The endpoints above fetches upstream column-lineage for given dataset field or all fields within a dataset. Downstream column lineage is turned off by default. However, this can be turned on with an extra <code>withDownstream</code> parameter like:</p> + +<pre><code>GET /column-lineage?nodeId=datasetField:food_delivery:public.delivery_7_days:a&amp;amp;withDownstream=true + +</code></pre> + +<p>This will include <code>outEdges</code> within the returned node of the graph.</p> + +<p><strong>Point in time upstream lineage</strong></p> + +<p>Point in time lineage for newly proposed <code>/api/v1/column-lineage</code> endpoint:</p> + +<pre><code>GET /column-lineage?nodeId=dataset_field:food_delivery:public.delivery_7_days:a&amp;amp;datasetVersion=123e4567-e89b-12d3-a456-426614174000 +GET /column-lineage?nodeId=dataset_field:food_delivery:public.delivery_7_days:a&amp;amp;lineageAt=1661846242 +</code></pre> + +<p>Point in time can be controlled by:</p> + +<p>• <strong>datasetVersion</strong> - uuid of a specific dataset version, +• <strong>lineageAt</strong> - which contains a unix timestamp.</p> + +<p>When <strong>lineageAt</strong> specified, the latest dataset version before timestamp will be found. Regardles <strong>datasetVersion</strong> or <strong>lineageAt</strong> parameters applied, responses will be the same as below:</p> + +<pre><code>{ + graph: [ + { +&amp;lt; "id": "datasetField:db1:table1:a", +&amp;gt; "id": "datasetField:db1:table1:a#{VERSION UUID}", + "type": "DATASET_FIELD", + "data": { + .... +} +</code></pre> + +<p><strong>Implementation</strong> +<strong>columne lineage facet in dataset resource endpoint</strong></p> + +<p>Adding the columnLineage facet requires a formatting of existing facet data (work in progress).</p> + +<p><strong>column lineage endpoint</strong></p> + +<p>The <code>/column-lineage</code> endpoint leverages the <code>/lineage</code> endpoint and then filters down the payload to return the expected result.</p> + +<p><strong>point-in-time upstream lineage</strong></p> + +<p>The point-in-time upstream lineage leverages the run to dataset version relation to track back the lineage of a given dataset of job version. <br /> +Dataset version -&gt; run that produced it -&gt; consumed Dataset Versions.</p> + +<p><strong>Next Steps</strong></p> + +<p>Review of this proposal and production of detailed design for the implementation.:</p> +
+ + + + + + + +
+ + <a href="https://github.com/MarquezProject/marquez">MarquezProject/marquez</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paul Lee + (paullee@lyft.com) +
+
2022-09-12 15:29:12
 +
+

where can i find docs on just simply using extractors? without marquez. for example, a basic BashOperator on Airflow 1.10.15

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paul Lee + (paullee@lyft.com) +
+
2022-09-12 15:30:08
 +
+

*Thread Reply:* or is it automatic for anything that exists in extractors/?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Howard Yoo + (howard.yoo@astronomer.io) +
+
2022-09-12 15:30:16
 +
+

*Thread Reply:* Yes

+ + + +
+ 👍 Paul Lee +
+ +
+ :gratitude_thank_you: Paul Lee +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paul Lee + (paullee@lyft.com) +
+
2022-09-12 15:31:12
 +
+

*Thread Reply:* so anything i add to extractors directory with the same name as the operator will automatically extract the metadata from the operator is that correct?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Howard Yoo + (howard.yoo@astronomer.io) +
+
2022-09-12 15:31:31
 +
+

*Thread Reply:* Well, not entirely

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Howard Yoo + (howard.yoo@astronomer.io) +
+
2022-09-12 15:31:47
 +
+

*Thread Reply:* please take a look at the source code of one of the extractors

+ + + +
+ 👍 Paul Lee +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Howard Yoo + (howard.yoo@astronomer.io) +
+
2022-09-12 15:32:13
 +
+

*Thread Reply:* also, there are docs available at openlineage.io/docs

+ + + +
+ 🙏 Paul Lee +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paul Lee + (paullee@lyft.com) +
+
2022-09-12 15:33:45
 +
+

*Thread Reply:* ok, i'll take a look. i think one thing that would be helpful is having a custom setup without marquez. a lot of the docs or videos i found were integrated with marquez

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Howard Yoo + (howard.yoo@astronomer.io) +
+
2022-09-12 15:34:29
 +
+

*Thread Reply:* I see. Marquez is a openlineage backend that stores the lineage data, so many examples do need them.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Howard Yoo + (howard.yoo@astronomer.io) +
+
2022-09-12 15:34:47
 +
+

*Thread Reply:* If you do not want to run marquez but just test out the openlineage, you can also take a look at OpenLineage Proxy.

+ + + +
+ 👍 Paul Lee +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paul Lee + (paullee@lyft.com) +
+
2022-09-12 15:35:14
 +
+

*Thread Reply:* awesome thanks Howard! i'll take a look at these resources and come back around if i need to

+ + + +
+ 👍 Howard Yoo +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-09-12 16:01:45
 +
+

*Thread Reply:* http://openlineage.io/docs/integrations/airflow/extractor - this is the doc you might want to read

+
+
openlineage.io
+ + + +
Custom extractors | OpenLineage Docs
+
+ OpenLineage integration works by detecting which Airflow operators your dag is using, and extracting lineage +
+ + + + + + +
Original URL: http://openlineage.io/docs/integrations/airflow/extractor
+ + + +
+ + + +
+ 🎉 Paul Lee +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paul Lee + (paullee@lyft.com) +
+
2022-09-12 17:08:49
 +
+

*Thread Reply:* yeah, saw that doc earlier. thanks @Maciej Obuchowski appreciate it 🙏

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jay + (sanjay.sudhakaran@trovemoney.co.nz) +
+
2022-09-21 20:55:24
 +
+

Hey team! I’m pretty new to the field in general

+ +

In the real world, I would be running pyspark scripts on AWS EMR. Could you explain to me how the metadata is sent to Marquez from my pyspark script, and where it’s persisted?

+ +

Would I need to set up an S3 bucket to store the lineage data?

+ +

I’m also unsure about how I would run the Marquez UI on AWS - Would I need to have an EC2 instance running permanently in order to access that UI?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jay + (sanjay.sudhakaran@trovemoney.co.nz) +
+
2022-09-21 20:57:39
 +
+

*Thread Reply:* In my head, I have:

+ +

Pyspark script -> Store metadata in S3 -> Marquez UI gets data from S3 and displays it

+ +

I suspect this is incorrect?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2022-09-22 02:14:50
 +
+

*Thread Reply: It’s more like: you add openlineage jar to Spark job, configure it what to do with the events. Popular options are: + * sent to rest endpoint (like Marquez), + * send as an event onto Kafka, + * print it onto console +There is no S3 in between Spark & Marquez by default. +Marquez serves both as an API where events are sent and UI to investigate them.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jay + (sanjay.sudhakaran@trovemoney.co.nz) +
+
2022-09-22 17:36:10
 +
+

*Thread Reply:* Yeah S3 was just an example for a storage option.

+ +

I actually found the answer I was looking for, turns out I had to look at Marquez documentation: +https://marquezproject.ai/resources/deployment/

+ +

The answer is that Marquez uses a postgres instance to persist the metadata it is given. Thanks for your time though! I appreciate the effort 🙂

+ + + +
+ 👍 Kevin Adams +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Hanbing Wang + (doris.wang200902@gmail.com) +
+
2022-09-25 17:06:41
 +
+

Hello team, +For the OpenLineage Spark, even when I processed one Spark sql query (CTAS Create Table As Select), I will received multiple events back (2+ Start events, 2 Complete events). +I try to understand why OpenLineage need to send back that much events, and what is the primary difference between Start VS Start events, Start VS Complete events? +Do we have any doc can help me understand more on it? +Thanks

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-09-26 00:27:05
 +
+

*Thread Reply:* The Spark execution model follows:

+ +
  1. Spark SQL Execution Start event
  2. Spark Job Start event
  3. Spark Job End event
  4. Spark SQL Execution End event +As a result, OpenLineage tracks all of those execution and jobs. There is a proposed plan to distinguish between those events (e.g. you wouldn't get two starts but one Start and one Job Start or something like that).
    5. +
+ +

You should collect all of these events in order to be sure you are receiving all the data since each event may contain a subset of the complete facets that represent what occurred in the job.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Hanbing Wang + (doris.wang200902@gmail.com) +
+
2022-09-26 15:16:26
 +
+

*Thread Reply:* Thanks @Will Johnson +Can I get an example of how the proposed plan can be used to distinguish between start and job start events? +Because I compare the 2 starts events I got, only the event_time is different, all other information are the same.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Hanbing Wang + (doris.wang200902@gmail.com) +
+
2022-09-26 15:30:34
 +
+

*Thread Reply:* One followup question, if I process multiple queries in one command, for example (Drop + Create Table + Insert Overwrite), should I expected for +(1). 1 Spark SQL execution start event +(2). 3 Spark job start event (Each query has a job start event ) +(3). 3 Spark job end event (Each query has a job end event ) +(4). 1 Spark SQL execution end event

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-09-27 10:25:47
 +
+

*Thread Reply:* Re: Distinguish between start and job start events. There was a proposal to differentiate the two (https://github.com/OpenLineage/OpenLineage/issues/636) but the current discussion is here: https://github.com/OpenLineage/OpenLineage/issues/599 As it currently stands, there is not a way to tell which one is which (I believe). The design of OpenLineage is such that you should consume ALL events under the same run id and job name / namespace.

+ +

Re: Multiple Queries in One Command: This is where Spark's execution model comes into play. I believe each one of those commands are executed sequentially and as a result, you'd actually get three execution start and three execution end. If you chose DROP + Create Table As Select, that would be only two commands and thus only two execution start events.

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/issues/636|#636 [INTEGRATION][SPARK] Change event types for JobStart and JobEnd events>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/issues/599|#599 Spec: clarify event order lifecycle>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Hanbing Wang + (doris.wang200902@gmail.com) +
+
2022-09-27 16:49:37
 +
+

*Thread Reply:* Thanks a lot for your help 🙏 @Will Johnson, +For multiple queries in one command, I still have a confused place why Drop + CreateTable and Drop + CreateTableAsSelect act different.

+ +

When I test Drop + Create Table +Query: +DROP TABLE IF EXISTS shadow_test.test_sparklineage_4; CREATE TABLE IF NOT EXISTS shadow_test.test_sparklineage_4 (val INT, region STRING) PARTITIONED BY ( ds STRING ) STORED AS PARQUET; +I only received 1 start + 1 complete event +And the events only contains DropTableCommandVisitor/DropTableCommand. +I expected we should also received start and complete events for CreateTable query with CreateTableCommanVisitor/CreateTableComman .

+ +

But when I test Drop + Create Table As Select +Query: +DROP TABLE IF EXISTS shadow_test.test_sparklineage_5; CREATE TABLE IF NOT EXISTS shadow_test.test_sparklineage_5 AS SELECT ** from shadow_test.test_sparklineage where ds &gt; '2022-08-24'" +I received 1 start + 1 complete event with DropTableCommandVisitor/DropTableCommand +And 2 start + 2 complete events with CreateHiveTableAsSelectCommandVisitor/CreateHiveTableAsSelectCommand

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-09-27 22:03:38
 +
+

*Thread Reply:* @Hanbing Wang are you running this on Databricks with a hive metastore that is defaulting to Delta by any chance?

+ +

I THINK there are some gaps in OpenLineage because of the way Databricks Delta handles things and now there is Unity catalog that is causing some hiccups as well.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-09-28 09:18:48
 +
+

*Thread Reply:* > For multiple queries in one command, I still have a confused place why Drop + CreateTable and Drop + CreateTableAsSelect act different. +@Hanbing Wang That's basically why we capture all the events (SQL Execution, Job) instead of one of them. We're just inconsistently notified of them by Spark.

+ +

Some computations emit SQL Execution events, some emit Job events, I think majority emits both. This also differs by spark version.

+ +

The solution OpenLineage assumes is having cumulative model of job execution, where your backend deals with possible duplication of information.

+ +

> I THINK there are some gaps in OpenLineage because of the way Databricks Delta handles things and now there is Unity catalog that is causing some hiccups as well. +@Will Johnson would be great if you created issue with some complete examples

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Hanbing Wang + (doris.wang200902@gmail.com) +
+
2022-09-28 15:44:45
 +
+

*Thread Reply:* @Will Johnson and @Maciej Obuchowski Thanks a lot for your help +We are not running on Databricks. +We implemented the OpenLineage Spark listener, and custom the Event Transport which emitting the events to our own events pipeline with a hive metastore. +We are using Spark version 3.2.1 +OpenLineage version 0.14.1

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-09-29 15:16:28
 +
+

*Thread Reply:* Ooof! @Hanbing Wang then I'm not certain why you're not receiving the extra event 😞 You may need to run your spark cluster in debug mode to step through the Spark Listener.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-09-29 15:17:08
 +
+

*Thread Reply:* @Maciej Obuchowski - I'll add it to my list!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Hanbing Wang + (doris.wang200902@gmail.com) +
+
2022-09-30 15:34:01
 +
+

*Thread Reply:* @Will Johnson Thanks a lot for your help. Let us debug and continue investigating on this issue.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Yujia Yang + (yujia@tubi.tv) +
+
2022-09-26 03:46:19
 +
+

Hi team, I find Openlineage posts a lot for run events to the backend.

+ +

eg. I submit jar to Spark cluster with computations like

+ +
  1. count from table1. --> this will have more than one run events inputs:[table1], outputs:[]
  2. count from table2 --> this will have more than one run events inputs:[table2], outputs:[]
  3. write Seq[(t1, count1), (t2, count2)) to table3. --> this may give inputs:[] outputs [table3] +can I just get one post with a summary telling me, inputs:[table1, table2], outputs:[table3] alongside with a merged columnareLineage?
    4. +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2022-09-28 08:34:20
 +
+

*Thread Reply:* One of assumptions was to create a stateless integration model where multiple events can be sent for a single job run. This has several advantages like sending events for jobs which suddenly fail, sending events immediately, etc.

+ +

The events can be merged then at the backend side. The behavior, you describe, can be then achieved by using backends like Marquez and Marquez API to obtain combined data.

+ +

Currently, we’re developing column-lineage dedicated endpoint in Marquez according to the proposal: https://github.com/MarquezProject/marquez/blob/main/proposals/2045-column-lineage-endpoint.md +This will allow you to request whole column lineage graph based on multiple jobs.

+
+ + + + +
<https://github.com/MarquezProject/marquez/blob/main/proposals/2045-column-lineage-endpoint.md | 2045-column-lineage-endpoint.md>
+
+ <p><strong>Proposal: Column lineage endpoint proposal</strong></p> + +<p>Author(s): <a href="https://github.com/julienledem">@julienledem</a>, <a href="https://github.com/pawel-big-lebowski">@pawel-big-lebowski</a></p> + +<p>Created: 20022-08-18</p> + +<p>Dicussion: <a href="https://github.com/MarquezProject/marquez/issues/2045">column lineage endpoint issue #2045</a></p> + +<p><strong>Overview</strong> +<strong>Use cases</strong></p> + +<p>• Find the current upstream dependencies of a column. A column in a dataset is derived from columns in upstream datasets. +• See column-level lineage in the dataset level lineage when available. +• Retrieve point-in-time upstream lineage for a dataset or a column. What did the lineage look like yesterday compared to today?</p> + +<p><strong>Existing elements</strong></p> + +<p>• OpenLineage defines a [column-level lineage facet]- (<a href="https://github.com/OpenLineage/OpenLineage/blob/main/spec/facets/ColumnLineageDatasetFacet.json">https://github.com/OpenLineage/OpenLineage/blob/main/spec/facets/ColumnLineageDatasetFacet.json</a>). +• Marquez has a lineage endpoint <code>GET /api/v1/lineage</code> that returns the current lineage graph connected to a job or a dataset</p> + +<p><strong>Column lineage characteristics and general assumptions</strong></p> + +<p>Column level lineage is a different lineage graph due to a different node granularity - kind of zoomed-in view of existing lineage. Instead of datasets and jobs being lineage graph nodes, each dataset field becomes a node. Additionally, there are edges between dataset fields, instead of datasets itself. Thus, enriching existing lineage with column lineage information would not be sufficient. That’s why we propose another API endpoint with column lineage graph.</p> + +<p>Upstream and downstream edges do have different characteristics. An output dataset is always produced by a single version of input dataset (one upstream), while a single input datset version can have multiple output dataset versions. Lineage graph can be then easily flooded by downstream subgraph which blurs the overall view. That's why we consider an upstream column lineage as a default one. Downstream lineage will be returned only when requested explicitly.</p> + +<p><strong>New Elements</strong></p> + +<p>We propose the following changes:</p> + +<p>• Add column lineage to the dataset resource endpoint. Column lineage will NOT be added to existing <code>/lineage</code> endpoint as it may be a heavy database operation run on each lineage graph's node which we want to avoid. Based on that, column level lineage get be requested per dataset in separate requests when required. +• A new column-lineage endpoint leveraging the column lineage facet to retrieve lineage for a given column. +• Point-in-time upstream (dataset or column level) lineage given a version of a dataset.</p> + +<p><strong>Proposal</strong> +<strong>Add column lineage to existing datasets endpoint</strong></p> + +<p>In the <code>GET /api/v1/namespaces/{namespace}/datasets</code> api, add column lineage facet to returned dataset resource.</p> + +<p><strong>Add a column-level-lineage endpoint:</strong></p> + +<p>New endpoints to retrieve a column lineage of a single field or a whole dataset will be added:</p> + +<pre><code>GET /column-lineage?nodeId=dataset:{namespace}:{dataset} +GET /column-lineage?nodeId=datasetField:{namespace}:{dataset}:{field} +</code></pre> + +<p>For example:</p> + +<pre><code>GET /column-lineage?nodeId=dataset:food_delivery:public.delivery_7_days +GET /column-lineage?nodeId=datasetField:food_delivery:public.delivery_7_days:a +</code></pre> + +<p>Although creating a new endpoint, we would like to reuse existing data structures with a new <code>NodeType.FIELD</code> introduced.</p> + +<p>The logic returns dataset field node:</p> + +<pre><code>GET /column-lineage?nodeId=datasetField:db1:table1:a +... +{ + graph: [ + { + "id": "datasetField:db1:table1:a", + "type": "DATASET_FIELD", + "data": { + "namespace": "DB1", + "name": "table2", + "field": "a", + "type": "integer", + "transformationDescription": "identical", + "transformationType": "IDENTITY", + "inputFields": [ + { "namespace": "DBA", "name": "tableA", "field": "columnA"}, + { "namespace": "DBB", "name": "tableB", "field": "columnB"}, + { "namespace": "DBC", "name": "tableC", "field": "columnC"} + ] + "inEdges": [ + { + "origin": "datasetField:db1:table1:a", + "destination": "datasetField:DBA:tableA:columnA" + }, + { + "origin": "datasetField:db1:table1:a", + "destination": "datasetField:DBB:tableB:columnB" + }, + { + "origin": "datasetField:db1:table1:a", + "destination": "datasetField:DBB:tableB:columnC" + } + ], + }, + ... + # Input fields, present within "inEdges", can be also returned within a graph due to a `depth` parameter greate than 0. + } + ] +} +</code></pre> + +<p>The <code>depth</code> parameter controls how many edges, from a given dataset field, shall be returned. The default is set to <code>0</code>. In case of default equal <code>1</code>, each <code>inputField</code> will be returned as a separate node within a response graph with <code>inputFields</code> used to produce it. Please note that extending depth may increase the graph size and affect request performance.</p> + +<p>The endpoints above fetches upstream column-lineage for given dataset field or all fields within a dataset. Downstream column lineage is turned off by default. However, this can be turned on with an extra <code>withDownstream</code> parameter like:</p> + +<pre><code>GET /column-lineage?nodeId=datasetField:food_delivery:public.delivery_7_days:a&amp;amp;withDownstream=true + +</code></pre> + +<p>This will include <code>outEdges</code> within the returned node of the graph.</p> + +<p><strong>Point in time upstream lineage</strong></p> + +<p>Point in time lineage for newly proposed <code>/api/v1/column-lineage</code> endpoint:</p> + +<pre><code>GET /column-lineage?nodeId=dataset_field:food_delivery:public.delivery_7_days:a&amp;amp;datasetVersion=123e4567-e89b-12d3-a456-426614174000 +GET /column-lineage?nodeId=dataset_field:food_delivery:public.delivery_7_days:a&amp;amp;lineageAt=1661846242 +</code></pre> + +<p>Point in time can be controlled by:</p> + +<p>• <strong>datasetVersion</strong> - uuid of a specific dataset version, +• <strong>lineageAt</strong> - which contains a unix timestamp.</p> + +<p>When <strong>lineageAt</strong> specified, the latest dataset version before timestamp will be found. Regardles <strong>datasetVersion</strong> or <strong>lineageAt</strong> parameters applied, responses will be the same as below:</p> + +<pre><code>{ + graph: [ + { +&amp;lt; "id": "datasetField:db1:table1:a", +&amp;gt; "id": "datasetField:db1:table1:a#{VERSION UUID}", + "type": "DATASET_FIELD", + "data": { + .... +} +</code></pre> + +<p><strong>Implementation</strong> +<strong>columne lineage facet in dataset resource endpoint</strong></p> + +<p>Adding the columnLineage facet requires a formatting of existing facet data (work in progress).</p> + +<p><strong>column lineage endpoint</strong></p> + +<p>The <code>/column-lineage</code> endpoint leverages the <code>/lineage</code> endpoint and then filters down the payload to return the expected result.</p> + +<p><strong>point-in-time upstream lineage</strong></p> + +<p>The point-in-time upstream lineage leverages the run to dataset version relation to track back the lineage of a given dataset of job version. <br /> +Dataset version -&gt; run that produced it -&gt; consumed Dataset Versions.</p> + +<p><strong>Next Steps</strong></p> + +<p>Review of this proposal and production of detailed design for the implementation.:</p> +
+ + + + + + + +
+ + <a href="https://github.com/MarquezProject/marquez">MarquezProject/marquez</a> +
+ + +
+ + + +
+ :gratitude_thank_you: Yujia Yang +
+ +
+ 👀 Yujia Yang +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
srutikanta hota + (srutikanta.hota@gmail.com) +
+
2022-09-28 09:47:55
 +
+

Is there a provision to include additional MDC properties as part of openlineage ? +Or something like sparkSession.sparkContext().setLocalProperties("key","value")

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2022-09-29 14:30:37
 +
+

*Thread Reply:* Hello @srutikanta hota, could you elaborate a bit on your use case? I'm not sure what you are trying to achieve. Possibly @Paweł Leszczyński will know.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-09-29 15:24:26
 +
+

*Thread Reply:* @srutikanta hota - Not sure what MDC properties stands for but you might take inspiration from the DatabricksEnvironmentHandler Facet Builder: https://github.com/OpenLineage/OpenLineage/blob/65a5f021a1ba3035d5198e759587737a05[…]ark/agent/facets/builder/DatabricksEnvironmentFacetBuilder.java

+ +

You can create a facet that could extract out the properties that you might set from within the spark session.

+ +

I don't think OpenLineage / a Spark Listener can affect the SparkSession itself so you wouldn't be able to SET the properties in the listener.

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/65a5f021a1ba3035d5198e759587737a05b242e1/integration/spark/shared/src/main/java/io/openlineage/spark/agent/facets/builder/DatabricksEnvironmentFacetBuilder.java | DatabricksEnvironmentFacetBuilder.java>
+
+ <pre><code> private Map&amp;lt;String, Object&amp;gt; getDatabricksEnvironmentalAttributes(SparkListenerJobStart jobStart) { + dbProperties = new HashMap&amp;lt;&amp;gt;(); + // These are useful properties to extract if they are available + + List&amp;lt;String&amp;gt; dbPropertiesKeys = + Arrays.asList( + "orgId", + "spark.databricks.clusterUsageTags.clusterOwnerOrgId", + "spark.databricks.notebook.path", + "spark.databricks.job.type", + "spark.databricks.job.id", + "spark.databricks.job.runId", + "user", + "userId", + "spark.databricks.clusterUsageTags.clusterName", + "spark.databricks.clusterUsageTags.azureSubscriptionId"); + dbPropertiesKeys.stream() + .forEach( + (p) -&amp;gt; { + dbProperties.put(p, jobStart.properties().getProperty(p)); + }); + + /**** + ** Azure Databricks makes available a dbutils mount point to list aliased paths to cloud + ** storage. However, that dbutils object is not available inside a spark listener. We must + ** access it via reflection. + **/ + try { + dbutilsClass = Class.forName("com.databricks.dbutils_v1.impl.DbfsUtilsImpl"); + dbutils = (DbfsUtils) dbutilsClass.getDeclaredConstructor().newInstance(); + dbProperties.put("mountPoints", getDatabricksMountpoints(dbutils)); + } catch (Exception e) { + log.warn("Failed to load dbutils in OpenLineageListener"); + dbProperties.put("mountPoints", new ArrayList&amp;lt;DatabricksMountpoint&amp;gt;()); + } + + return dbProperties; + } +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
srutikanta hota + (srutikanta.hota@gmail.com) +
+
2022-09-30 04:56:25
 +
+

*Thread Reply:* Many thanks for the details. My usecase is simple, I like to default the sparkgroupjob Id as openlineage parent runid if there is no parent run Id set. +sc.setJobGroup("myjobgroupid", "job description goes here") +This set the value in spark as +setLocalProperty(SparkContext.SPARKJOBGROUPID, group_id)

+ +

I like to use myjobgroup_id as openlineage parent run id

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2022-09-30 05:01:08
 +
+

*Thread Reply:* MDC is an ability to add extra key -> value pairs to a log entry, while not doing this within message body. So the question here is (I believe): how to add custom entries / custom facets to OpenLineage events?

+ +

@srutikanta hota What information would you like to include? There is great chance we already have some fields for that. If not it’s still worth putting in in write place like: is this info job specific, run specific or relates to some of input / output datasets?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-09-30 05:04:34
 +
+

*Thread Reply:* @srutikanta hota sounds like you want to set up +spark.openlineage.parentJobName +spark.openlineage.parentRunId +https://openlineage.io/docs/integrations/spark/

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
srutikanta hota + (srutikanta.hota@gmail.com) +
+
2022-09-30 05:15:18
 +
+

*Thread Reply:* @… we are having a long-running spark context(the context may run for a week) where we submit jobs. Settings the parentrunid at beginning won't help. We are submitting the job with sparkgroupid. I like to use the group Id as parentRunId

+ +

https://spark.apache.org/docs/1.6.1/api/R/setJobGroup.html

+ + + +
+ 🤔 Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Trevor Swan + (trevor.swan@matillion.com) +
+
2022-09-29 13:59:20
 +
+

Hi team - I am from Matillion and we would like to build support for openlineage. Who would be best placed to move the conversation with my product team?

+ + + +
+ 🙌 Will Johnson, Maciej Obuchowski, Francis McGregor-Macdonald +
+ +
+ 🎉 Michael Robinson +
+ +
+ 👍 Ernie Ostic +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2022-09-29 14:22:06
 +
+

*Thread Reply:* Hi Trevor, thank you for reaching out. I’d be happy to discuss with you how we can help you support OpenLineage. Let me send you an email.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jarek Potiuk + (jarek@potiuk.com) +
+
2022-09-29 15:58:35
 +
+

cccccbctlvggfhvrcdlbbvtgeuredtbdjrdfttbnldcb

+ + + +
+ 🐈 Julien Le Dem, Jakub Dardziński, Maciej Obuchowski, Paweł Leszczyński +
+ +
+ 🐈‍⬛ Julien Le Dem, Maciej Obuchowski, Paweł Leszczyński +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Petr Hajek + (petr.hajek@profinit.eu) +
+
2022-09-30 02:52:51
 +
+

Hi Everyone! Would anybody be interested in participation in MANTA Open Lineage connector testing? We are specially looking for an environment with rich Airflow implementation but we will be happy to test on any other OL Producer technology. Send me a direct message for more information. Thanks, Petr

+ + + +
+ 🙌 Michael Robinson, Ross Turk +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sheeri Cabral (Collibra) + (sheeri.cabral@collibra.com) +
+
2022-09-30 14:34:45
 +
+

Question about Apache Airflow that I think folks here would know, because doing a web search has failed me:

+ +

Is there a way to interact with Apache Airflow to retrieve the contents of the files in the sql directory, but NOT to run them?

+ +

(the APIs all seem to run sql, and when I search I just get “how to use the airflow API to run queries”)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-09-30 14:38:34
 +
+

*Thread Reply:* Is this in the context of an OpenLineage extractor?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sheeri Cabral (Collibra) + (sheeri.cabral@collibra.com) +
+
2022-09-30 14:40:47
 +
+

*Thread Reply:* Yes! I was specifically looking at the PostgresOperator

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sheeri Cabral (Collibra) + (sheeri.cabral@collibra.com) +
+
2022-09-30 14:41:54
 +
+

*Thread Reply:* (as Snowflake lineage can be retrieved from their internal ACCESS_HISTORY tables, we wouldn’t need to use Airflow’s SnowflakeOperator to get lineage, we’d use the method on the openlineage blog)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-09-30 14:43:08
 +
+

*Thread Reply:* The extractor for the SQL operators gets the query like this: +https://github.com/OpenLineage/OpenLineage/blob/45fda47d8ef29dd6d25103bb491fb8c443[…]gration/airflow/openlineage/airflow/extractors/sql_extractor.py

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/45fda47d8ef29dd6d25103bb491fb8c44355eb3c/integration/airflow/openlineage/airflow/extractors/sql_extractor.py | sql_extractor.py>
+
+ <pre><code> job_facets = {"sql": SqlJobFacet(query=self.operator.sql)} +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+ 👍 Sheeri Cabral (Collibra) +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-09-30 14:43:48
 +
+

*Thread Reply:* let me see if I can find the corresponding part of the Airflow API docs...

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sheeri Cabral (Collibra) + (sheeri.cabral@collibra.com) +
+
2022-09-30 14:45:00
 +
+

*Thread Reply:* aha! I’m not so far behind the times, it was only put in during July https://github.com/OpenLineage/OpenLineage/pull/907

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/pull/907|#907 [INTEGRATION][Airflow][SQL] Add SqlExtractor class.>
+
+ <p>Signed-off-by: Jakub Dardzinski <a href="mailto:kuba0221@gmail.com">kuba0221@gmail.com</a></p> + +<p><strong>Problem</strong></p> + +<p>Current SQL based extractors have code duplicated.</p> + +<p><strong>Solution</strong></p> + +<p>Added common SqlExtractor class. Updated unit tests to match changes.</p> + +<p><strong>Checklist</strong></p> + +<p>☑︎ You've <a href="https://github.com/OpenLineage/OpenLineage/blob/main/why-the-dco.md">signed-off</a> your work +☑︎ Your pull request title follows our <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CONTRIBUTING.md#creating-pull-requests">guidelines</a> +☑︎ Your changes are accompanied by tests (<em>if relevant</em>) +☑︎ Your change contains a <a href="https://kurtisnusbaum.medium.com/stacked-diffs-keeping-phabricator-diffs-small-d9964f4dcfa6">small diff</a> and is self-contained</p> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-09-30 14:47:28
 +
+

*Thread Reply:* Hm. The PostgresOperator seems to extend BaseOperator directly: +https://github.com/apache/airflow/blob/029ebacd9cbbb5e307a03530bdaf111c2c3d4f51/airflow/providers/postgres/operators/postgres.py#L58

+
+ + + + +
<https://github.com/apache/airflow/blob/029ebacd9cbbb5e307a03530bdaf111c2c3d4f51/airflow/providers/postgres/operators/postgres.py | postgres.py>
+
+ <pre><code> sql: str | Iterable[str], +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/apache/airflow">apache/airflow</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sheeri Cabral (Collibra) + (sheeri.cabral@collibra.com) +
+
2022-09-30 14:48:01
 +
+

*Thread Reply:* yeah 😞 I couldn’t find a way to make that work as an end-user.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-09-30 14:48:08
 +
+

*Thread Reply:* perhaps that can't be assumed for all operators that deal with SQL. I know that @Maciej Obuchowski has spent a lot of time on this.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-09-30 14:49:14
 +
+

*Thread Reply:* I don't know enough about the airflow internals 😞

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sheeri Cabral (Collibra) + (sheeri.cabral@collibra.com) +
+
2022-09-30 14:50:00
 +
+

*Thread Reply:* No worries. In case it saves you work, I also had a look at https://github.com/apache/airflow/blob/029ebacd9cbbb5e307a03530bdaf111c2c3d4f51/airflow/providers/common/sql/operators/sql.py - which also extends BaseOperator but not with a way to just get the SQL.

+
+ + + + +
<https://github.com/apache/airflow/blob/029ebacd9cbbb5e307a03530bdaf111c2c3d4f51/airflow/providers/common/sql/operators/sql.py | sql.py>
+
+ <p>``` +#</p> + +<h1>Licensed to the Apache Software Foundation (ASF) under one</h1> + +<h1>or more contributor license agreements. See the NOTICE file</h1> + +<h1>distributed with this work for additional information</h1> + +<h1>regarding copyright ownership. The ASF licenses this file</h1> + +<h1>to you under the Apache License, Version 2.0 (the</h1> + +<h1>"License"); you may not use this file except in compliance</h1> + +<h1>with the License. You may obtain a copy of the License at</h1> + +<p>#</p> + +<h1><a href="http://www.apache.org/licenses/LICENSE-2.0">http://www.apache.org/licenses/LICENSE-2.0</a></h1> + +<p>#</p> + +<h1>Unless required by applicable law or agreed to in writing,</h1> + +<h1>software distributed under the License is distributed on an</h1> + +<h1>"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY</h1> + +<h1>KIND, either express or implied. See the License for the</h1> + +<h1>specific language governing permissions and limitations</h1> + +<h1>under the License.</h1> + +<p>from <strong>future</strong> import annotations</p> + +<p>import re +from typing import TYPE_CHECKING, Any, Iterable, Mapping, Sequence, SupportsAbs</p> + +<p>from packaging.version import Version</p> + +<p>from airflow.compat.functools import cached<em>property +from airflow.exceptions import AirflowException +from airflow.hooks.base import BaseHook +from airflow.models import BaseOperator, SkipMixin +from airflow.providers.common.sql.hooks.sql import DbApiHook, _backported</em>get_hook +from airflow.version import version</p> + +<p>if TYPE_CHECKING: + from airflow.utils.context import Context</p> + +<p>def parse_boolean(val: str) -&gt; str | bool: + """Try to parse a string into boolean.</p> + +<pre><code>Raises ValueError if the input is not a valid true- or false-like string value. +""" +val = val.lower() +if val in ('y', 'yes', 't', 'true', 'on', '1'): + return True +if val in ('n', 'no', 'f', 'false', 'off', '0'): + return False +raise ValueError(f"{val!r} is not a boolean-like string value") +</code></pre> + +<p>def <em>get</em>failed<em>checks(checks, col=None): + if col: + return [ + f"Column: {col}\nCheck: {check},\nCheck Values: {check</em>values}\n" + for check, check<em>values in checks.items() + if not check</em>values["success"] + ] + return [ + f"\tCheck: {check},\n\tCheck Values: {check<em>values}\n" + for check, check</em>values in checks.items() + if not check_values["success"] + ]</p> + +<p><em>PROVIDERS</em>MATCHER = re.compile(r'airflow.providers.(.<strong>).hooks.</strong>')</p> + +<p><em>MIN</em>SUPPORTED<em>PROVIDERS</em>VERSION = { + "amazon": "4.1.0", + "apache.drill": "2.1.0", + "apache.druid": "3.1.0", + "apache.hive": "3.1.0", + "apache.pinot": "3.1.0", + "databricks": "3.1.0", + "elasticsearch": "4.1.0", + "exasol": "3.1.0", + "google": "8.2.0", + "jdbc": "3.1.0", + "mssql": "3.1.0", + "mysql": "3.1.0", + "odbc": "3.1.0", + "oracle": "3.1.0", + "postgres": "5.1.0", + "presto": "3.1.0", + "qubole": "3.1.0", + "slack": "5.1.0", + "snowflake": "3.1.0", + "sqlite": "3.1.0", + "trino": "3.1.0", + "vertica": "3.1.0", +}</p> + +<p>class BaseSQLOperator(BaseOperator): + """ + This is a base class for generic SQL Operator to get a DB Hook</p> + +<pre><code>The provided method is .get_db_hook(). The default behavior will try to +retrieve the DB hook based on connection type. +You can custom the behavior by overriding the .get_db_hook() method. +""" + +def __init__( + self, + **, + conn_id: str | None = None, + database: str | None = None, + hook_params: dict | None = None, + ****kwargs, +): + super().__init__(****kwargs) + self.conn_id = conn_id + self.database = database + self.hook_params = {} if hook_params is None else hook_params + +@cached_property +def _hook(self): + """Get DB Hook based on connection type""" + self.log.debug("Get connection for %s", self.conn_id) + conn = BaseHook.get_connection(self.conn_id) + if Version(version) &amp;gt;= Version('2.3'): + # "hook_params" were introduced to into "get_hook()" only in Airflow 2.3. + hook = conn.get_hook(hook_params=self.hook_params) # ignore airflow compat check + else: + # For supporting Airflow versions &amp;lt; 2.3, we backport "get_hook()" method. This should be removed + # when "apache-airflow-providers-common-sql" will depend on Airflow &amp;gt;= 2.3. + hook = _backported_get_hook(conn, hook_params=self.hook_params) + if not isinstance(hook, DbApiHook): + from airflow.hooks.dbapi_hook import DbApiHook as _DbApiHook + + if isinstance(hook, _DbApiHook): + # This case might happen if user installed common.sql provider but did not upgrade the + # Other provider's versions to a version that supports common.sql provider + class_module = hook.__class__.__module__ + match = _PROVIDERS_MATCHER.match(class_module) + if match: + provider = match.group(1) + min_version = _MIN_SUPPORTED_PROVIDERS_VERSION.get(provider) + if min_version: + raise AirflowException( + f'You are trying to use common-sql with {hook.__class__.__name__},' + f' but the Hook class comes from provider {provider} that does not support it.' + f' Please upgrade provider {provider} to at least {min_version}.' + ) + raise AirflowException( + f'You are trying to use `common-sql` with {hook.__class__.__name__},' + ' but its provider does not support it. Please upgrade the provider to a version that' + ' supports `common-sql`. The hook class should be a subclass of' + ' `airflow.providers.common.sql.hooks.sql.DbApiHook`.' + f' Got {hook.__class__.__name__} Hook with class hierarchy: {hook.__class__.mro()}' + ) + + if self.database: + hook.schema = self.database + + return hook + +def get_db_hook(self) -&amp;gt; DbApiHook: + """ + Get the database hook for the connection. + + :return: the database hook object. + :rtype: DbApiHook + """ + return self._hook +</code></pre> + +<p>class SQLColumnCheckOperator(BaseSQLOperator): + """ + Performs one or more of the templated checks in the column<em>checks dictionary. + Checks are performed on a per-column basis specified by the column</em>mapping. + Each check can take one or more of the following options: + - equal<em>to: an exact value to equal, cannot be used with other comparison options + - greater</em>than: value that result should be strictly greater than + - less<em>than: value that results should be strictly less than + - geq</em>to: value that results should be greater than or equal to + - leq_to: value that results should be less than or equal to + - tolerance: the percentage that the result may be off from the expected value</p> + +<pre><code>:param table: the table to run checks on +:param column_mapping: the dictionary of columns and their associated checks, e.g. + +.. code-block:: python + + { + "col_name": { + "null_check": { + "equal_to": 0, + }, + "min": { + "greater_than": 5, + "leq_to": 10, + "tolerance": 0.2, + }, + "max": {"less_than": 1000, "geq_to": 10, "tolerance": 0.01}, + } + } + +:param partition_clause: a partial SQL statement that is added to a WHERE clause in the query built by + the operator that creates partition_clauses for the checks to run on, e.g. + +.. code-block:: python + + "date = '1970-01-01'" + +:param conn_id: the connection ID used to connect to the database +:param database: name of database which overwrite the defined one in connection + +.. seealso:: + For more information on how to use this operator, take a look at the guide: + :ref:`howto/operator:SQLColumnCheckOperator` +""" + +template_fields = ("partition_clause",) + +column_checks = { + "null_check": "SUM(CASE WH… +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/apache/airflow">apache/airflow</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2022-09-30 15:22:24
 +
+

*Thread Reply:* that's more of an Airflow question indeed. As far as I understand you need to read file with SQL statement within Airflow Operator and do something but run the query (like pass as an XCom)? SQLExtractors we have get same SQL that operators render and uses it to extract additional information like table schema straight from database

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sheeri Cabral (Collibra) + (sheeri.cabral@collibra.com) +
+
2022-09-30 14:36:18
 +
+

(I’m also ok with a way to get the SQL that has been run - but from Airflow, not the data source - I’m looking for a db-neutral way to do this, otherwise I can just parse query logs on any specific db system)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paul Lee + (paullee@lyft.com) +
+
2022-09-30 18:45:09
 +
+

👋 are there any docs on how the listener hooks in and gets run with openlineage-airflow? trying to write some unit tests but no docs seem to exist on the flow.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2022-09-30 19:06:47
 +
+

*Thread Reply:* There's a design doc linked from the PR: https://github.com/apache/airflow/pull/20443 +https://docs.google.com/document/d/1L3xfdlWVUrdnFXng1Di4nMQYQtzMfhvvWDR9K4wXnDU/edit

+
+ + + + +
<https://github.com/apache/airflow/pull/20443|#20443 Add Listener Plugin API that tracks TaskInstance state changes>
+
+ <p>This PR adds new Plugin API - listeners. It enables plugin authors to write <a href="https://pluggy.readthedocs.io/en/stable/"><code>pluggy</code> hook implementation</a> that will be called on certain formalized extension points. To differentiate between current Airflow extension points, like plugins, and current Airflow hooks, implementations of those hooks are called <code>listeners</code>.</p> + +<p>The API is ment to be called across all dags, and all operators - in contrast to current <code>on_success_callback</code>, <code>pre_execute</code> and related family which are meant to provide callbacks for particular dag authors, or operator creators.</p> + +<p><code>pluggy</code> mechanism enables us to execute multiple, or none, listeners that implement particular extension point, so that users can use multiple listeners seamlessly.</p> + +<p>In this PR, three such extension points are added. <br /> +When TaskInstance's state is changed to <code>RUNNING</code>, <code>on_task_instance_running</code> hook is called. On change to<code>SUCCESS</code> <code>on_task_instance_success</code> is called, similarly on <code>FAILED</code> <code>on_task_instance_failed</code> is called.</p> + +<p>Actual notification mechanism is be implemented using <a href="https://docs.sqlalchemy.org/en/13/orm/session_events.html#after-flush">SQLAlchemy’s events mechanism</a>. This ensures that plugins will get every change of state, regardless of where in the codebase it happened, and not require manual annotation of TI state changes across the codebase.</p> + +<p>To make sure that this change is not affecting performance, running this mechanism on scheduler is disabled by default. The SQLAlchemy event mechanism is also not affected by default - the event listener is only added if we have any plugin which actually provides any listener.</p> + +<p>Design doc: <br /> +<a href="https://docs.google.com/document/d/1L3xfdlWVUrdnFXng1Di4nMQYQtzMfhvvWDR9K4wXnDU/edit?usp=sharing">https://docs.google.com/document/d/1L3xfdlWVUrdnFXng1Di4nMQYQtzMfhvvWDR9K4wXnDU/edit?usp=sharing</a></p> + +<p>Background to this change: <br /> +When discussing changing LineageBackend api for OpenLineage plugin, we came to consensus that the changes proposed there would be better suited to the new API <a href="https://github.com/apache/airflow/issues/17984">#17984</a></p> + +<p>related: <a href="https://github.com/apache/airflow/issues/17984">#17984</a></p> +
+ +
+
Labels
+ area:scheduler/executor, area:dev-tools, area:plugins, type:new-feature, full tests needed +
+ +
+
Comments
+ 3 +
+ + + + + + + +
+ + <a href="https://github.com/apache/airflow">apache/airflow</a> +
+ + +
+ + + +
+ 👀 Paul Lee +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paul Lee + (paullee@lyft.com) +
+
2022-09-30 19:18:47
 +
+

*Thread Reply:* amazing thank you I will take a look

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-10-03 11:32:52
 +
+

@channel +Hello everyone, I’m opening up a vote on releasing OpenLineage 0.15.0, including +• an improved development experience in the Airflow integration +• updated proposal and integration templates +• a change to the BigQuery client in the Airflow integration +• plus bug fixes across the project. +3 +1s from committers will authorize an immediate release. For all the commits, see: https://github.com/OpenLineage/OpenLineage/compare/0.14.0...HEAD. Note: this will be the last release to support Airflow 1.x! +Thanks!

+ + + +
+ 🎉 Paul Lee, Howard Yoo, Minkyu Park, Michael Collado, Paweł Leszczyński, Maciej Obuchowski, Harel Shein +
+ +
+ 👍 Michael Collado, Julien Le Dem, Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-10-03 11:33:30
 +
+

*Thread Reply:* Hey @Michael Robinson. Removal of Airflow 1.x support is planned for next release after 0.15.0

+ + + +
+ 👍 Jakub Dardziński, Paul Lee +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-10-03 11:37:03
 +
+

*Thread Reply:* 0.15.0 would be the last release supporting Airflow 1.x

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-10-03 11:37:07
 +
+

*Thread Reply:* just caught this myself. I’ll make the change

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paul Lee + (paullee@lyft.com) +
+
2022-10-03 11:40:33
 +
+

*Thread Reply:* we’re still on 1.10.15 at the moment so i guess our team would have to rely on <=0.15.0?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-10-03 11:49:47
 +
+

*Thread Reply:* Is this something you want to continue doing or do you want to migrate relatively soon?

+ +

We want to remove 1.10 integration because for multiple PRs, maintaining compatibility with it takes a lot of time; the code is littered with checks like this. +if parse_version(AIRFLOW_VERSION) &gt;= parse_version("2.0.0"):

+ + + +
+ 👍 Paul Lee +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paul Lee + (paullee@lyft.com) +
+
2022-10-03 12:03:40
 +
+

*Thread Reply:* hey Maciej, we do have plans to migrate in the coming months but for right now we need to stay on 1.10.15.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-10-04 09:39:11
 +
+

*Thread Reply:* Thanks, all. The release is authorized, and you can expect it by Thursday.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paul Lee + (paullee@lyft.com) +
+
2022-10-03 17:56:08
 +
+

👋 what would be a possible reason for the built in airflow backend being utilized instead of a custom wrapper over airflow.lineage.Backend ? double checked the [lineage] key in our airflow.cfg

+ +

there doesn't seem to be any errors being thrown and the object loads 🤔

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paul Lee + (paullee@lyft.com) +
+
2022-10-03 17:56:36
 +
+

*Thread Reply:* running airflow 2.3.4 with openlineage-airflow 0.14.1

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-10-03 18:03:03
 +
+

*Thread Reply:* if you're talking about LineageBackend, it is used in Airflow 2.1-2.2. It did not have functionality where you can be notified on task start or failure, so we wanted to expand the functionality: https://github.com/apache/airflow/issues/17984

+ +

Consensus of Airflow maintainers wasn't positive about changing this interface, so we went with another direction: https://github.com/apache/airflow/pull/20443

+
+ + + + +
<https://github.com/apache/airflow/issues/17984|#17984 Add possibility to LineageBackend to be notified of task instance execution start and failure>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/apache/airflow">apache/airflow</a> +
+ + +
+
+ + + + +
<https://github.com/apache/airflow/pull/20443|#20443 Add Listener Plugin API that tracks TaskInstance state changes>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/apache/airflow">apache/airflow</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-10-03 18:06:58
 +
+

*Thread Reply:* Why nothing happens? https://github.com/OpenLineage/OpenLineage/blob/895160423643398348154a87e0682c3ab5c8704b/integration/airflow/openlineage/lineage_backend/__init__.py#L91

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/895160423643398348154a87e0682c3ab5c8704b/integration/airflow/openlineage/lineage_backend/__init__.py | __init__.py>
+
+ <pre><code> if parse_version(AIRFLOW_VERSION) &amp;gt;= parse_version("2.3.0.dev0"): +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paul Lee + (paullee@lyft.com) +
+
2022-10-03 18:30:32
 +
+

*Thread Reply:* ah hmm ok, i will double check. i commented that part out so technically it should run but maybe i missed something

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paul Lee + (paullee@lyft.com) +
+
2022-10-03 18:30:42
 +
+

*Thread Reply:* thank you for your fast response @Maciej Obuchowski ! i appreciate it

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paul Lee + (paullee@lyft.com) +
+
2022-10-03 18:31:13
 +
+

*Thread Reply:* it seems like it doesn't use my custom wrapper but instead uses the openlineage implementation.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paul Lee + (paullee@lyft.com) +
+
2022-10-03 20:11:15
 +
+

*Thread Reply:* @Maciej Obuchowski ok, after checking we are emitting events with our custom backend but an odd thing is an attempt is always made with the openlineage backend. is there something obvious i am perhaps missing 🤔

+ +

ends up with requests.exceptions.HTTPError: 401 Client Error: Unauthorized for url immediately after task start. but by the end on task success/failure it emits the event with our custom backend both RunState.COMPLETE and RunState.START into our own pipeline.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-10-04 06:19:06
 +
+

*Thread Reply:* If you're on 2.3 and trying to use some wrapped LineageBackend, what I think is happening is OpenLineagePlugin that automatically registers via setup.py entrypoint https://github.com/OpenLineage/OpenLineage/blob/65a5f021a1ba3035d5198e759587737a05b242e1/integration/airflow/openlineage/airflow/plugin.py#L30

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/65a5f021a1ba3035d5198e759587737a05b242e1/integration/airflow/openlineage/airflow/plugin.py | plugin.py>
+
+ <pre><code> listeners = [listener] +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-10-04 06:23:48
 +
+

*Thread Reply:* I think if you want to extend it with proprietary code there are two good options.

+ +

First, if your code only needs to touch HTTP client side - which I guess is the case due to 401 error - then you can create custom Transport.

+ +

Second, is that you fork OL code and create your own package, without entrypoint script or with adding your own if you decide to extend OpenLineagePlugin instead of LineageBackend

+ + + +
+ 👍 Paul Lee +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paul Lee + (paullee@lyft.com) +
+
2022-10-04 14:23:33
 +
+

*Thread Reply:* amazing thank you for your help. i will take a look

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paul Lee + (paullee@lyft.com) +
+
2022-10-04 14:49:47
 +
+

*Thread Reply:* @Maciej Obuchowski is there a way to extend the plugin like how we can wrap the custom backend with 2.2? or would it be necessary to fork it.

+ +

we're trying to not fork and instead opt with extending.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-10-05 04:55:05
 +
+

*Thread Reply:* I think it's best to fork, since it's getting loaded by Airflow as an entrypoint: https://github.com/OpenLineage/OpenLineage/blob/133110300e8ea4e42e3640608cfed459683d5a8d/integration/airflow/setup.py#L70

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/133110300e8ea4e42e3640608cfed459683d5a8d/integration/airflow/setup.py | setup.py>
+
+ <pre><code> "airflow.plugins": ["OpenLineagePlugin = openlineage.airflow.plugin:OpenLineagePlugin"] +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+ 🙏 Paul Lee +
+ +
+ :gratitude_thank_you: Paul Lee +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paul Lee + (paullee@lyft.com) +
+
2022-10-05 13:29:24
 +
+

*Thread Reply:* got it. and in terms of the openlineage.yml and defining a custom transport is there a way i can define where openlineage-python should look for the custom transport? e.g. different path

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paul Lee + (paullee@lyft.com) +
+
2022-10-05 13:30:04
 +
+

*Thread Reply:* because from the docs i. can't tell except for the file i'm supposed to copy and implement.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-10-05 14:18:19
 +
+

*Thread Reply:* @Paul Lee you should derive from Transport base class and register type as full python import path to your custom transport, for example https://github.com/OpenLineage/OpenLineage/blob/f8533266491acea2159f602f782a99a4f8a82cca/client/python/tests/openlineage.yml#L2

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/f8533266491acea2159f602f782a99a4f8a82cca/client/python/tests/openlineage.yml | openlineage.yml>
+
+ <pre><code> type: "tests.transport.AccumulatingTransport" +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-10-05 14:20:48
 +
+

*Thread Reply:* your custom transport should have also define custom class Config , and this class should implement from_dict method

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-10-05 14:20:56
 +
+

*Thread Reply:* the whole process is here: https://github.com/OpenLineage/OpenLineage/blob/a62484ec14359a985d283c639ac7e8b9cfc54c2e/client/python/openlineage/client/transport/factory.py#L47

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/a62484ec14359a985d283c639ac7e8b9cfc54c2e/client/python/openlineage/client/transport/factory.py | factory.py>
+
+ <pre><code> def _create_transport(self, config: dict): +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-10-05 14:21:09
 +
+

*Thread Reply:* and I know we need to document this better 🙂

+ + + +
+ 🙏 Paul Lee +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paul Lee + (paullee@lyft.com) +
+
2022-10-05 15:35:31
 +
+

*Thread Reply:* amazing, thanks for all your help 🙂 +1 to the docs, if i have some time when done i will push up some docs to document what i've done

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-10-05 15:50:29
 +
+

*Thread Reply:* https://github.com/openlineage/docs/ - let me know and I'll review 🙂

+
+ + + + +
OpenLineage/docs
+
+ Documentation for OpenLineage +
+ +
+
Website
+ <https://openlineage.io/docs> +
+ +
+
Stars
+ 4 +
+ + + + + + + + +
+ + + +
+ 🎉 Paul Lee +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-10-04 12:39:59
 +
+

@channel +Hi everyone, opening a vote on a release (0.15.1) to add #1131 to fix the release process on CI. 3 +1s from committers will authorize an immediate release. Thanks. More details are here: +https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/pull/1131|#1131 CI: build macos release package on medium resouce class>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md | CHANGELOG.md>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+ ✅ Michael Collado, Maciej Obuchowski, Julien Le Dem +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-10-04 14:25:49
 +
+

*Thread Reply:* Thanks, all. The release is authorized.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-10-05 10:46:46
 +
+

@channel +OpenLineage 0.15.1 is now available! +We added: +• Airflow: improve development experience #1101 @JDarDagran +• Documentation: update issue templates for proposal & add new integration template #1116 @rossturk +• Spark: add description for URL parameters in readme, change overwriteName to appName #1130 @tnazarew +We changed: +• Airflow: lazy load BigQuery client #1119 @mobuchowski +Many bug fixes were also included in this release. +Release: https://github.com/OpenLineage/OpenLineage/releases/tag/0.15.1 +Changelog: https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md +Commit history: https://github.com/OpenLineage/OpenLineage/compare/0.14.1...0.15.1 +Maven: https://oss.sonatype.org/#nexus-search;quick~openlineage +PyPI: https://pypi.org/project/openlineage-python/

+ + + +
+ 🙌 Maciej Obuchowski, Jakub Dardziński, Howard Yoo, Harel Shein, Paul Lee, Paweł Leszczyński +
+ +
+ 🎉 Howard Yoo, Harel Shein, Paul Lee +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-10-06 07:35:00
 +
+

Is there a topic you think the community should discuss at the next OpenLineage TSC meeting? Reply or DM with your item, and we’ll add it to the agenda.

+ + + +
+ 🌟 Paul Lee +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paul Lee + (paullee@lyft.com) +
+
2022-10-06 13:29:30
 +
+

*Thread Reply:* would love to add improvement in docs :) for newcomers

+ + + +
+ 👏 Jakub Dardziński +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paul Lee + (paullee@lyft.com) +
+
2022-10-06 13:31:07
 +
+

*Thread Reply:* also, what’s TSC?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-10-06 15:20:23
 +
+

*Thread Reply:* Technical Steering Committee, but it’s open to everyone

+ + + +
+ 👍 Paul Lee +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-10-06 15:20:45
 +
+

*Thread Reply:* and we encourage newcomers to attend

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paul Lee + (paullee@lyft.com) +
+
2022-10-06 13:49:00
 +
+

has anyone seen their COMPLETE/FAILED listeners not firing on Airflow 2.3.4 but START events do emit? using openlineage-airflow 0.14.1

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2022-10-06 14:39:27
 +
+

*Thread Reply:* is there any error/warn message logged maybe?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paul Lee + (paullee@lyft.com) +
+
2022-10-06 14:40:53
 +
+

*Thread Reply:* none that i'm seeing on our workers. i do see that our custom http transport is being utilized on START.

+ +

but on SUCCESS nothing fires.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paul Lee + (paullee@lyft.com) +
+
2022-10-06 14:41:21
 +
+

*Thread Reply:* which makes me believe the listeners themselves aren't being utilized? 🤔

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2022-10-06 16:37:54
 +
+

*Thread Reply:* uhm, any chance you're experiencing this with custom extractors?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2022-10-06 16:38:13
 +
+

*Thread Reply:* I'd be happy to jump on a quick call if you wish

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2022-10-06 16:38:40
 +
+

*Thread Reply:* but in more EU friendly hours 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paul Lee + (paullee@lyft.com) +
+
2022-10-07 16:19:47
 +
+

*Thread Reply:* no custom extractors, its usingt he base extractor. a call would be 👍. let me look at my calendar and EU hours.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-10-06 15:23:27
 +
+

@channel The next OpenLineage Technical Steering Committee meeting is on Thursday, October 13 at 10 am PT. Join us on Zoom: https://bit.ly/OLzoom +All are welcome! +Agenda:

+ +
  1. Announcements
  2. Recent Release 0.15.1
  3. Project roadmap review
  4. Open discussion +Notes: https://bit.ly/OLwiki +Is there a topic you think the community should discuss at this or a future meeting? Reply or DM me to add items to the agenda.
    5. +
+ + + +
+ 🙌 Paul Lee, Harel Shein +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Srinivasa Raghavan + (gsrinir@gmail.com) +
+
2022-10-07 06:52:42
 +
+

hello all. I am trying to run the airflow example from here +I changed the Marquez web port from 5000 to 15000 but when I start the docker images, it seems to always default to port 5000 and therefore when I go to localhost:3000, the jobs don't load up as they are not able to connect to the marquez app running in 15000. I've overriden the values in docker-compose.yml and in openLineage.env but it seems to be picking up the 5000 value from some other location. +This is what I see in the logs. Any pointers on this or please redirect me to the appropriate channel. Thanks! +INFO [2022-10-07 10:48:58,022] org.eclipse.jetty.server.AbstractConnector: Started application@782fd504{HTTP/1.1, (http/1.1)}{0.0.0.0:5000} +INFO [2022-10-07 10:48:58,034] org.eclipse.jetty.server.AbstractConnector: Started admin@1537c744{HTTP/1.1, (http/1.1)}{0.0.0.0:5001}

+ + + +
+ 👀 Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Srinivasa Raghavan + (gsrinir@gmail.com) +
+
2022-10-20 05:11:09
 +
+

*Thread Reply:* Apparently the value is hard coded in the code somewhere that I couldn't figure out but at-least learnt that in my Mac where this port 5000 is being held up can be freed by following the below simple step.

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Hanna Moazam + (hannamoazam@microsoft.com) +
+
2022-10-10 18:00:17
 +
+

Hi #general - @Will Johnson and I are working on adding support for Snowflake to OL, and as we were going to specify the package under the compileOnly dependencies in gradle, we had some doubts looking at the existing dependencies. Taking bigQuery as an example - we see it's included as a dependency in both the shared build.gradle file, and in the app build.gradle file. We're a bit confused about the following:

+ +
  1. Why do we need to have the bigQuery package in shared's dependencies? App of course contains the bigQueryNodeVisitor but we couldn't spot where it's being used within shared.
  2. For all the dependencies in the shared gradle file, the versions for Scala and Spark are fixed (Scala 2.11, Spark 2.4.8), whereas for app, the versionsMap allows for different combinations of spark and scala versions. Why is this so?
  3. How do the dependencies between app and shared interact? Does one or the other take precedence for which version of the bigQuery connector is compiled? +We'd appreciate any guidance!
    4. +
+ +

Thank you in advance!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2022-10-11 03:47:31
 +
+

*Thread Reply:* Hi @Hanna Moazam,

+ +

Within recent PR https://github.com/OpenLineage/OpenLineage/pull/1111, I removed BigQuery dependencies from spark2, spark32 and spark3 subprojects. It has to stay in sharedbecause of BigQueryNodeVisitor. The usage of BigQueryNodeVisitor is tricky as we never know if bigquery classes are available on runtime or not. The check is done in io.openlineage.spark.agent.lifecycle.BaseVisitorFactory +if (BigQueryNodeVisitor.hasBigQueryClasses()) { + list.add(new BigQueryNodeVisitor(context, factory)); + } +Regarding point 2, there were some Spark versions which allowed two Scala versions (2.11 and 2.12). Then it makes sense to make it configurable. On the other hand, for Spark 3.2 we only support 2.12 which is hardcoded in build.gradle.

+ +

The idea of app project is let's create a separate project to aggregate all the dependecies and run integration tests on it . Subprojects spark2, spark3, etc. do depend on shared . Putting integration tests in shared would create additional opposite-way dependency, which we wanted to avoid.

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/pull/1111|#1111 fix bigquery connector>
+
+ <p>Signed-off-by: Pawel Leszczynski <a href="mailto:leszczynski.pawel@gmail.com">leszczynski.pawel@gmail.com</a></p> + +<p><strong>Problem</strong></p> + +<p>Spark integration fails with spark-bigquery-connector &gt;=0.25.0</p> + +<p>Closes: <a href="https://github.com/OpenLineage/OpenLineage/issues/1105">#1105</a></p> + +<p><strong>Solution</strong></p> + +<p>• make Spark integration work with the latest connector +• provide a change in a way that the old version would work either</p> + +<p>&gt; <strong>Note:</strong> All schema changes require discussion. Please <a href="https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue">link the issue</a> for context.</p> + +<p>☐ Your change modifies the <a href="https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.json">core</a> OpenLineage model +☐ Your change modifies one or more OpenLineage <a href="https://github.com/OpenLineage/OpenLineage/tree/main/spec/facets">facets</a></p> + +<p>If you're contributing a new integration, please specify the scope of the integration and how/where it has been tested (e.g., Apache Spark integration supports <code>S3</code> and <code>GCS</code> filesystem operations, tested with AWS EMR).</p> + +<p><strong>Checklist</strong></p> + +<p>☑︎ You've <a href="https://github.com/OpenLineage/OpenLineage/blob/main/why-the-dco.md">signed-off</a> your work +☑︎ Your pull request title follows our <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CONTRIBUTING.md#creating-pull-requests">guidelines</a> +☐ Your changes are accompanied by tests (<em>if relevant</em>) +☑︎ Your change contains a <a href="https://kurtisnusbaum.medium.com/stacked-diffs-keeping-phabricator-diffs-small-d9964f4dcfa6">small diff</a> and is self-contained +☐ You've updated any relevant documentation (<em>if relevant</em>) +☐ You've updated the <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md"><code>CHANGELOG.md</code></a> with details about your change under the "Unreleased" section (<em>if relevant, depending on the change, this may not be necessary</em>) +☐ You've versioned the core OpenLineage model or facets according to <a href="https://docs.snowplowanalytics.com/docs/pipeline-components-and-applications/iglu/common-architecture/schemaver">SchemaVer</a> (<em>if relevant</em>) +☐ You've added a <a href="https://github.com/OpenLineage/OpenLineage/tree/main/.github/header_templates.md">header</a> to source files (<em>if relevant</em>)</p> +
+ +
+
Labels
+ bug, documentation, integration/spark, integration/bigquery +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-10-11 09:20:44
 +
+

*Thread Reply:* So, if we wanted to add Snowflake, we would need to:

+ +
  1. Pick a version of snowflake's spark library
  2. Pick a version of scala that we target (i.e. we are only going to support Snowflake in Spark 3.2 so scala 2.12 will be hard coded)
  3. Add the visitor code to Shared
  4. Add the dependencies to app (ONLY if there is an integration test in app?? This is the confusing part still)
    5. +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2022-10-12 03:51:54
 +
+

*Thread Reply:* Yes. Please note that snowflake library will not be included in target OpenLineage jar. So you may test it manually against multiple Snowflake library versions or even adjust code in case of minor differences.

+ + + +
+ 👍 Hanna Moazam, Will Johnson +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Hanna Moazam + (hannamoazam@microsoft.com) +
+
2022-10-12 05:20:17
 +
+

*Thread Reply:* Thank you Pawel!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-10-12 12:18:16
 +
+

*Thread Reply:* Basically the same pattern you've already done with Kusto 😉 +https://github.com/OpenLineage/OpenLineage/blob/a96ecdabe66567151e7739e25cd9dd03d6[…]va/io/openlineage/spark/agent/lifecycle/BaseVisitorFactory.java

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/a96ecdabe66567151e7739e25cd9dd03d67e03e8/integration/spark/shared/src/main/java/io/openlineage/spark/agent/lifecycle/BaseVisitorFactory.java | BaseVisitorFactory.java>
+
+ <pre><code> if (KustoRelationVisitor.hasKustoClasses()) { +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Hanna Moazam + (hannamoazam@microsoft.com) +
+
2022-10-12 12:26:35
 +
+

*Thread Reply:* We actually used only reflection for Kusto and were hoping to do it the 'better' way with the package itself for snowflake - if it's possible :)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Akash r + (akashrn25@gmail.com) +
+
2022-10-11 02:04:28
 +
+

Hi Community,

+ +

I was going through the code of dbt integration with Open lineage, Once the events has been emitted from client code , I wanted to check the server code where the events are read and the lineage is formed. Where can I find that code ?

+ +

Thanks

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-10-11 05:03:26
 +
+

*Thread Reply:* Reference implementation of OpenLineage consumer is Marquez: https://github.com/MarquezProject/marquez

+
+ + + + +
MarquezProject/marquez
+
+ Collect, aggregate, and visualize a data ecosystem's metadata +
+ +
+
Website
+ <https://marquezproject.ai> +
+ +
+
Stars
+ 1187 +
+ + + + + + + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-10-12 11:59:55
 +
+

This month’s OpenLineage TSC meeting is tomorrow at 10 am PT! https://openlineage.slack.com/archives/C01CK9T7HKR/p1665084207602369

+
+ + +
+ + + } + + Michael Robinson + (https://openlineage.slack.com/team/U02LXF3HUN7) +
+ + + +
+
+ <p><strong>@channel</strong> The next OpenLineage Technical Steering Committee meeting is on <strong>Thursday, October 13 at 10 am PT</strong>. Join us on Zoom: <a href="https://bit.ly/OLzoom">https://bit.ly/OLzoom</a> +All are welcome! +Agenda:</p> + +<ol><li>Announcements</li><li>Recent Release 0.15.1</li><li>Project roadmap review</li><li>Open discussion +Notes: <a href="https://bit.ly/OLwiki">https://bit.ly/OLwiki</a> +Is there a topic you think the community should discuss at this or a future meeting? Reply or DM me to add items to the agenda.</li> +</ol> +
+ + + + + + +
Original URL: https://openlineage.slack.com/archives/C01CK9T7HKR/p1665084207602369
+ + +
+ + Slack Conversation +
+ + +
+ + + +
+ 🙌 Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sheeri Cabral (Collibra) + (sheeri.cabral@collibra.com) +
+
2022-10-13 12:05:17
 +
+

Is there anyone in the Open Lineage community in San Diego? I’ll be there Nov 1-3 and would love to meet some of y’all in person

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paul Lee + (paullee@lyft.com) +
+
2022-10-20 13:49:39
 +
+

👋 is there a way to define a base extractor to be defaulted to? for example, i'd like to have all our operators (50+) default to my custom base extractor instead of having a list of 50+ operators in get_operator_classnames

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Howard Yoo + (howard.yoo@astronomer.io) +
+
2022-10-20 13:53:55
 +
+

I don't think that's possible yet, as the extractor checks are based on the class name... and it wouldn't check which parent operator has it inherited from.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paul Lee + (paullee@lyft.com) +
+
2022-10-20 14:05:38
 +
+

😢 ok, i would contribute upstream but unfortunately we're still on 1.10.15. looking like we might have to hardcode for a bit.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paul Lee + (paullee@lyft.com) +
+
2022-10-20 14:06:01
 +
+

is this the correct assumption? we're still on 0.14.1 ^

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-10-20 14:33:49
 +
+

If you'll move to 2.x series and OpenLineage 0.16, you could use this feature: https://github.com/OpenLineage/OpenLineage/pull/1162

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/pull/1162|#1162 airflow: add default extractor>
+
+ <p>As a way of extending OpenLineage airflow integration, we are providing a way to add custom extractors. This is great for people who want to add integration with external operators - for example, regular Airflow user who wants to support <code>GCSToS3Operator</code>.</p> + +<p>However, it's too cumbersome for people who own operators, and want to add default implementation of OpenLineage for their operators for external users. This PR adds support for <code>DefaultExtractor</code> which looks up particular method - <code>get_openlineage_facets</code> on Operator, and when it's present, uses it to get lineage data.</p> + +<p>Signed-off-by: Maciej Obuchowski <a href="mailto:obuchowski.maciej@gmail.com">obuchowski.maciej@gmail.com</a></p> +
+ +
+
Labels
+ integration/airflow, extractor +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+ 👍 Paul Lee +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paul Lee + (paullee@lyft.com) +
+
2022-10-20 14:46:36
 +
+

thanks @Maciej Obuchowski we're working on it. hoping we'll land on 2.3.4 in the coming month.

+ + + +
+ 🔥 Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Austin Poulton + (austin.poulton@equalexperts.com) +
+
2022-10-26 05:31:07
 +
+

👋 Hi everyone!

+ + + +
+ 👋 Jakub Dardziński, Maciej Obuchowski, Michael Robinson, Ross Turk, Willy Lulciuc, Paweł Leszczyński, Harel Shein +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harel Shein + (harel.shein@gmail.com) +
+
2022-10-26 15:22:22
 +
+

*Thread Reply:* Hey @Austin Poulton, welcome! 👋

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Austin Poulton + (austin.poulton@equalexperts.com) +
+
2022-10-31 06:09:41
 +
+

*Thread Reply:* thanks Harel 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-11-01 09:44:18
 +
+

@channel +Hi everyone, I’m opening a vote to release OpenLineage 0.16.0, featuring: +• support for boolean arguments in the DefaultExtractor +• a more efficient get_connection_uri method in the Airflow integration +• a reorganized, Rust-based SQL integration (easing the addition of language interfaces in the future) +• bug fixes and more. +3 +1s from committers will authorize an immediate release. Thanks. More details are here: +https://github.com/OpenLineage/OpenLineage/compare/0.15.1...HEAD

+ + + +
+ 🙌 Howard Yoo, Paweł Leszczyński, Maciej Obuchowski +
+ +
+ 👍 Ross Turk, Paweł Leszczyński, Maciej Obuchowski +
+ +
+ ➕ Willy Lulciuc, Mandy Chessell, Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-11-01 13:37:54
 +
+

*Thread Reply:* Thanks, all! The release is authorized. We will initiate it within 48 hours.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Iftach Schonbaum + (iftach.schonbaum@hunters.ai) +
+
2022-11-02 08:45:20
 +
+

Anybody with a success use-case of ingesting column-level lineage into amundsen?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-11-02 09:19:43
 +
+

*Thread Reply:* I think amundsen-openlineage dataloader precedes column-level lineage in OL by a bit, so I doubt this works

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harel Shein + (harel.shein@gmail.com) +
+
2022-11-02 15:54:31
 +
+

*Thread Reply:* do you want to open up an issue for it @Iftach Schonbaum?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-11-02 12:36:22
 +
+

Hi everyone, you might notice Dependabot opening PRs to update dependencies now that it’s been configured and turned on (https://github.com/OpenLineage/OpenLineage/pull/1182). There will probably be a large number of PRs to start with, but this shouldn’t always be the case and we can change the tool’s behavior, as well. (Some background: this will help us earn the OSSF Silver badge for the project, which will help us advance in the LFAI.)

+ + + +
+ 👍 Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-11-03 07:53:31
 +
+

@channel +I’m opening a vote to release OpenLineage 0.16.1 to fix an issue in the SQL integration. This release will also include all the commits announced for 0.16.0. +3 +1s from committers will authorize an immediate release. Thanks.

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/pull/1228|#1228 fix: sql macos build directory>
+
+ <p>Since package renaming, mac os CI release build was run in the wrong directory.</p> + +<p>Signed-off-by: Maciej Obuchowski <a href="mailto:obuchowski.maciej@gmail.com">obuchowski.maciej@gmail.com</a></p> +
+ +
+
Labels
+ integration/sql +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+ ➕ Maciej Obuchowski, Hanna Moazam, Jakub Dardziński, Ross Turk, Paweł Leszczyński, Jarek Potiuk, Willy Lulciuc +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-11-03 12:25:29
 +
+

*Thread Reply:* Thanks, all. The release is authorized and will be initiated shortly.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-11-03 13:46:58
 +
+

@channel +OpenLineage 0.16.1 is now available, featuring: +Additions: +• Airflow: add dag_run information to Airflow version run facet #1133 @fm100 +• Airflow: add LoggingMixin to extractors #1149 @JDarDagran +• Airflow: add default extractor #1162 @mobuchowski +• Airflow: add on_complete argument in DefaultExtractor #1188 @JDarDagran +• SQL: reorganize the library into multiple packages #1167 @StarostaGit @mobuchowski +Changes: +• Airflow: move get_connection_uri as extractor’s classmethod #1169 @JDarDagran +• Airflow: change get_openlineage_facets_on_start/complete behavior #1201 @JDarDagran +Bug fixes and more! +Release: https://github.com/OpenLineage/OpenLineage/releases/tag/0.16.1 +Changelog: https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md +Commit history: https://github.com/OpenLineage/OpenLineage/compare/0.15.1...0.16.1 +Maven: https://oss.sonatype.org/#nexus-search;quick~openlineage +PyPI: https://pypi.org/project/openlineage-python/

+ + + +
+ 🙌 Maciej Obuchowski, Francis McGregor-Macdonald, Eric Veleker +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Phil Chen + (phil@gpr.com) +
+
2022-11-03 13:59:29
 +
+

Are there any tutorial and documentation how to create an Openlinage connector. For example, what if we Argo workflow instead of Apache airflow for orchestrating ETL jobs? How are we going to create Openlinage Argo workflow connector? How much efforts, roughly? And can people contribute such connectors to the community if they create one?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-11-04 06:34:27
 +
+

*Thread Reply:* > Are there any tutorial and documentation how to create an Openlinage connector. +We have somewhat of a start of a doc: +https://openlineage.io/docs/development/developing/

+ +

Here we have an example of using Python OL client to emit OL events: https://openlineage.io/docs/client/python#start-docker-and-marquez

+ +

> How much efforts, roughly? +I'm not familiar with Argo workflows, but usually the effort needed depends on extensibility of the underlying system. From the first look, Argo looks like it has sufficient mechanisms for that: https://argoproj.github.io/argo-workflows/executor_plugins/#examples-and-community-contributed-plugins

+ +

Then, it depends if you can get the information that you need in that plugin. Basic need is to have information from which datasets the workflow/job is reading and to which datasets it's writing.

+ +

> And can people contribute such connectors to the community if they create one? +Definitely! And if you need help with anything OpenLineage feel free to write here on Slack

+
+
openlineage.io
+ + + +
Python | OpenLineage Docs
+
+ Overview +
+ + + + + + +
Original URL: https://openlineage.io/docs/client/python#start-docker-and-marquez
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-11-03 17:57:37
 +
+

Is there a topic you think the community should discuss at the next OpenLineage TSC meeting? Reply or DM with your item, and we’ll add it to the agenda.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-11-03 18:03:18
 +
+

@channel +This month’s OpenLineage TSC meeting is next Thursday, November 10th, at 10 am PT. Join us on Zoom: https://bit.ly/OLzoom. All are welcome! +On the tentative agenda:

+ +
  1. Recent release overview [Michael R.]
  2. Update on LFAI & Data Foundation progress [Michael R.]
  3. Proposal: Defining “implementing OpenLineage” [Julien]
  4. Update from MANTA on their OpenLineage integration [Eric and/or Petr from MANTA]
  5. Linking CMF (a common ML metadata framework) and OpenLineage [Suparna and AnnMary from HP Enterprise]
  6. Open discussion
    7. +
+ + + +
+ 👍 Luca Soato, Maciej Obuchowski, Paul Lee, Willy Lulciuc +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kenton (swiple.io) + (kknoxparton@gmail.com) +
+
2022-11-08 04:47:41
 +
+

Hi all 👋 I’m Kenton — a Software Engineer and founder of Swiple. I’m looking forward to working with OpenLineage and its community to integrate data lineage and data observability. +https://swiple.io

+
+
swiple.io
+ + + +
Build trust in every dataset | Build trust in every dataset
+
+ Swiple is an automated data monitoring platform that helps analytics and data engineering teams seamlessly monitor the quality of their data. With automated data analysis and profiling, scheduling and alerting, teams can resolve data quality issues before they impact mission critical resources. +
+ + + + + + +
Original URL: https://swiple.io
+ + + +
+ + + +
+ 🙌 Maciej Obuchowski, Jakub Dardziński, Michael Robinson, Ross Turk, John Thomas, Julien Le Dem, Willy Lulciuc, Varun Singh +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-11-08 10:22:15
 +
+

*Thread Reply:* Welcome Kenton! Happy to help 👍

+ + + +
+ 👍 Kenton (swiple.io) +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Deepika Prabha + (deepikaprabha@gmail.com) +
+
2022-11-08 05:35:03
 +
+

Hi everyone, +We wanted to pass some dynamic metadata from spark job that we can catch up in OpenLineage event and use it for processing. Presently I have seen that we have few conf parameters like openlineage params that we can send only with Spark conf. Is there any other option we have where we can send some information dynamically from the spark jobs.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-11-08 10:06:10
 +
+

*Thread Reply:* What kind of data? My first feeling is that you need to extend the Spark integration

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Deepika Prabha + (deepikaprabha@gmail.com) +
+
2022-11-09 00:35:29
 +
+

*Thread Reply:* Yes, we wanted to add information like user/job description that we can use later with rest of openlineage event fields in our system

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Deepika Prabha + (deepikaprabha@gmail.com) +
+
2022-11-09 00:41:35
 +
+

*Thread Reply:* I can see in this PR https://github.com/OpenLineage/OpenLineage/pull/490 that env values can be captured which we can use to add some custom metadata but it seems it is specific to Databricks only.

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/pull/490|#490 Added generic facet to collect environmental properties (EnvironmentFacet)>
+
+ <p><strong>Problem</strong></p> + +<p>👋 Introduce the ability to produce environmental properties using EnvironmentFacet, this facet can be used to work with any environment, based on the conditional check to verify environment EnvironmentFacet can be added.</p> + +<p>Closes: <a href="https://github.com/OpenLineage/OpenLineage/issues/464">#464</a></p> + +<p><strong>Solution</strong></p> + +<p>Generic implementation of EnvironmentFacet gives the ability to add environmental details. Since EnvironmentFacet produces HashMap it can handle properties dynamically.</p> + +<p>In the current implementation added ability to collect properties for Databricks environment including mount point details. <br /> +Below are some examples:</p> + +<p>• "orgId" +• "spark.databricks.clusterUsageTags.clusterOwnerOrgId" +• "spark.databricks.notebook.path" +• "spark.databricks.job.type" +• "spark.databricks.job.id" +• "spark.databricks.job.runId" +• "user" +• "userId" +• "spark.databricks.clusterUsageTags.clusterName" +• "spark.databricks.clusterUsageTags.azureSubscriptionId"</p> + +<p><strong>Checklist</strong></p> + +<p>☑︎ You've <a href="https://github.com/OpenLineage/OpenLineage/blob/main/why-the-dco.md">signed-off</a> your work +☑︎ Your pull request title follows our <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CONTRIBUTING.md#creating-pull-requests">guidelines</a> +☐ Your changes are accompanied by tests (<em>if relevant</em>) +☑︎ Your change contains a <a href="https://kurtisnusbaum.medium.com/stacked-diffs-keeping-phabricator-diffs-small-d9964f4dcfa6">small diff</a> and is self-contained +☐ You've updated any relevant documentation (<em>if relevant</em>) +☐ You've updated the <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md"><code>CHANGELOG.md</code></a> with details about your change under the "Unreleased" section (<em>if relevant, depending on the change, this may not be necessary</em>) +☐ You've versioned the core OpenLineage model or facets according to <a href="https://docs.snowplowanalytics.com/docs/pipeline-components-and-applications/iglu/common-architecture/schemaver">SchemaVer</a> (<em>if relevant</em>)</p> +
+ +
+
Comments
+ 8 +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-11-09 05:14:50
 +
+

*Thread Reply:* I think it makes sense to have something like that, but generic, if you want to contribute it

+ + + +
+ 👍 Will Johnson, Deepika Prabha +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Varun Singh + (varuntestaz@outlook.com) +
+
2022-11-14 03:28:35
 +
+

*Thread Reply:* @Maciej Obuchowski Do you mean adding something like +spark.openlineage.jobFacet.FacetName.Key=Value to the spark conf should add a new job facet like +"FacetName": { + "Key": "Value" +}

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-11-14 05:56:02
 +
+

*Thread Reply:* We can argue about name of that key, but yes, something like that. Just notice that while it's possible to attach something to run and job facets directly, it would be much harder to do this with datasets

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
slackbot + +
+
2022-11-09 11:15:49
 +
+

This message was deleted.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2022-11-10 02:22:18
 +
+

*Thread Reply:* Hi @Varun Singh, what version of openlineage-spark where you using? Are you able to copy lineage event here?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-11-09 12:31:10
 +
+

@channel +This month’s TSC meeting is tomorrow at 10 am PT! https://openlineage.slack.com/archives/C01CK9T7HKR/p1667512998061829

+
+ + +
+ + + } + + Michael Robinson + (https://openlineage.slack.com/team/U02LXF3HUN7) +
+ + + +
+
+ <p>@channel +This month’s OpenLineage TSC meeting is <strong>next Thursday, November 10th, at 10 am PT</strong>. Join us on Zoom: <a href="https://bit.ly/OLzoom">https://bit.ly/OLzoom</a>. All are welcome! +On the tentative agenda:</p> + +<ol><li>Recent release overview [Michael R.]</li><li>Update on LFAI &amp; Data Foundation progress [Michael R.]</li><li>Proposal: Defining “implementing OpenLineage” [Julien]</li><li>Update from MANTA on their OpenLineage integration [Eric and/or Petr from MANTA]</li><li>Linking CMF (a common ML metadata framework) and OpenLineage [Suparna and AnnMary from HP Enterprise]</li><li>Open discussion</li> +</ol> +
+ + + + + + +
Original URL: https://openlineage.slack.com/archives/C01CK9T7HKR/p1667512998061829
+ + +
+ + Slack Conversation +
+ + +
+ + + +
+ 💥 Willy Lulciuc, Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Hanna Moazam + (hannamoazam@microsoft.com) +
+
2022-11-11 11:32:54
 +
+

Hi #general, quick question: do we plan to disable spark 2 support in the near future?

+ +

Longer question: +I've recently made a PR (https://github.com/OpenLineage/OpenLineage/pull/1231) to support capturing lineage from Snowflake, but it fails at a specific integration test due to what we think is a dependency mismatch for guava. I've tried to exclude any transient dependencies which may cause the problem but no luck with that so far.

+ +

Just wondering if:

+ +
  1. It makes sense to spend more time trying to ensure that test passes? Especially if we plan to remove spark 2 support soon.
  2. Assuming we do want to make sure to pass the test, does anyone have any other ideas for where to look/modify to prevent the error? +Here's the test failure message: +```io.openlineage.spark.agent.lifecycle.LibraryTest testRdd(SparkSession) FAILED (16s)
    3. +
+ +

java.lang.IllegalAccessError: tried to access method com.google.common.base.Stopwatch.<init>()V from class org.apache.hadoop.mapred.FileInputFormat + at io.openlineage.spark.agent.lifecycle.LibraryTest.testRdd(LibraryTest.java:113) ``` +Thanks in advance!

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/pull/1231|#1231 spark: add Snowflake Relation Visitor>
+
+ <p><strong>Problem</strong></p> + +<p>OpenLineage's Spark integration does not currently support Snowflake.</p> + +<p><strong>Solution</strong></p> + +<p>Implemented the SnowflakeRelationVisitor to support lineage for the <a href="https://github.com/snowflakedb/spark-snowflake">Snowflake Spark connector</a>.</p> + +<p>There are two paths in OpenLineage to the SnowflakeRelationVisitor:</p> + +<p>• Write events reach the SaveIntoDataSourceCommandVisitor, which routes the command to the <code>createSnowflakeDatasets</code> method. +• Read events trigger SnowflakeRelationVisitor's apply method.</p> + +<p>The implementation has been tested locally and on Azure Databricks.</p> + +<p>☐ Your change modifies the <a href="https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.json">core</a> OpenLineage model +☐ Your change modifies one or more OpenLineage <a href="https://github.com/OpenLineage/OpenLineage/tree/main/spec/facets">facets</a></p> + +<p><strong>Checklist</strong></p> + +<p>☑︎ You've <a href="https://github.com/OpenLineage/OpenLineage/blob/main/why-the-dco.md">signed-off</a> your work +☑︎ Your pull request title follows our <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CONTRIBUTING.md#creating-pull-requests">guidelines</a> +☑︎ Your changes are accompanied by tests (<em>if relevant</em>) +☑︎ Your change contains a <a href="https://kurtisnusbaum.medium.com/stacked-diffs-keeping-phabricator-diffs-small-d9964f4dcfa6">small diff</a> and is self-contained +☐ You've updated any relevant documentation (<em>if relevant</em>) +☐ You've updated the <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md"><code>CHANGELOG.md</code></a> with details about your change under the "Unreleased" section (<em>if relevant, depending on the change, this may not be necessary</em>) +☐ You've versioned the core OpenLineage model or facets according to <a href="https://docs.snowplowanalytics.com/docs/pipeline-components-and-applications/iglu/common-architecture/schemaver">SchemaVer</a> (<em>if relevant</em>) +☑︎ You've added a <a href="https://github.com/OpenLineage/OpenLineage/tree/main/.github/header_templates.md">header</a> to source files (<em>if relevant</em>)</p> +
+ +
+
Labels
+ documentation, integration/spark, spec +
+ +
+
Comments
+ 4 +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-11-11 16:28:07
 +
+

*Thread Reply:* What if we just not include it in the BaseVisitorFactory but only in the Spark3 visitor factories?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paul Lee + (paullee@lyft.com) +
+
2022-11-11 14:52:19
 +
+

quick question: how do i get the &lt;&lt;non-serializable Time...to show in the extraction? or really any object that gets passed in.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-11-11 16:24:30
 +
+

*Thread Reply:* You might look here: https://github.com/OpenLineage/OpenLineage/blob/f7049c599a0b1416408860427f0759624326677d/client/python/openlineage/client/serde.py#L51

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/f7049c599a0b1416408860427f0759624326677d/client/python/openlineage/client/serde.py | serde.py>
+
+ <pre><code> default=lambda o: f"&lt;&lt;non-serializable: {type(o).__qualname__}&gt;&gt;" +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+ :gratitude_thank_you: Paul Lee +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
srutikanta hota + (srutikanta.hota@gmail.com) +
+
2022-11-14 01:12:45
 +
+

Is there a way I can update the detaset description and the column description. While generating the open lineage spark events and columns

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2022-11-15 02:09:25
 +
+

*Thread Reply:* I don’t think this is possible at the moment.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2022-11-15 15:47:49
 +
+

Hey all, I'd like to ask for a release for OpenLineage. #1256 fixes bug in DefaultExtractor. This blocks people from migrating code from custom extractors to get_openlineage_facets methods.

+ + + +
+ ➕ Michael Robinson, Howard Yoo, Maciej Obuchowski, Willy Lulciuc +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-11-16 09:13:17
 +
+

*Thread Reply:* Thanks, all. The release is authorized.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-11-16 10:41:07
 +
+

*Thread Reply:* The PR for the changelog updates: https://github.com/OpenLineage/OpenLineage/pull/1306

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Varun Singh + (varuntestaz@outlook.com) +
+
2022-11-16 03:34:01
 +
+

Hi, small question: Is it possible to disable the /api/{version}/lineage suffix that gets added to every url automatically? Thanks!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-11-16 12:27:12
 +
+

*Thread Reply:* I think we had similar request before, but nothing was implemented.

+ + + +
+ 👍 Varun Singh +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-11-16 12:23:54
 +
+

@channel +OpenLineage 0.17.0 is now available, featuring: +Additions: +• Spark: support latest Spark 3.3.1 #1183 @pawel-big-lebowski +• Spark: add Kinesis Transport and support config Kinesis in Spark integration #1200 @yogyang +• Spark: disable specified facets #1271 @pawel-big-lebowski +• Python: add facets implementation to Python client #1233 @pawel-big-lebowski +• SQL: add Rust parser interface #1172 @StarostaGit @mobuchowski +• Proxy: add helm chart for the proxy backend #1068 @wslulciuc +• Spec: include possible facets usage in spec #1249 @pawel-big-lebowski +• Website: publish YML version of spec to website #1300 @rossturk +• Docs: update language on nominating new committers #1270 @rossturk +Changes: +• Website: publish spec into new website repo location #1295 @rossturk +• Airflow: change how pip installs packages in tox environments #1302 @JDarDagran +Removals: +• Deprecate HttpTransport.Builder in favor of HttpConfig #1287 @collado-mike +Bug fixes and more! +Release: https://github.com/OpenLineage/OpenLineage/releases/tag/0.17.0 +Changelog: https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md +Commit history: https://github.com/OpenLineage/OpenLineage/compare/0.16.1...0.17.0 +Maven: https://oss.sonatype.org/#nexus-search;quick~openlineage +PyPI: https://pypi.org/project/openlineage-python/

+ + + +
+ 🙌 Howard Yoo, Maciej Obuchowski, Ross Turk, Aphra Bloomfield, Harel Shein, Kengo Seki, Paweł Leszczyński, pankaj koti, Varun Singh +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Diego Cesar + (dcesar@krakenrobotik.de) +
+
2022-11-18 05:40:53
 +
+

Hi everyone,

+ +

I'm trying to get the lineage of a dataset per version. I initially had something like

+ +

Dataset A -&gt; Dataset B -&gt; DataSet C (version 1)

+ +

then:

+ +

Dataset D -&gt; Dataset E -&gt; DataSet C (version 2)

+ +

I can get the graph for version 2 without problems, but I'm wondering if there's any way to retrieve the entire graph for DataSet C version 1.

+ +

Thanks

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-11-22 13:40:44
 +
+

*Thread Reply:* It's kind of a hard problem UI side. Backend can express that relationship

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Diego Cesar + (dcesar@krakenrobotik.de) +
+
2022-11-22 13:48:58
 +
+

*Thread Reply:* Thanks for replying. Could you please point me to the API that allows me to do that? I've been calling GET /lineage with dataset in the node ID, e g., nodeId=dataset:my_dataset . Where could I specify the version of my dataset?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paul Lee + (paullee@lyft.com) +
+
2022-11-18 17:55:24
 +
+

👋 how do we get the actual values from macros? e.g. a schema name is passed in with {{params.table_name}} and thats what shows in lineage instead of the actual table name

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2022-11-19 04:54:13
 +
+

*Thread Reply:* Templated fields are rendered before generating lineage data. Do you have some sample code or logs preferrably?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-11-22 13:40:11
 +
+

*Thread Reply:* If you're on 1.10 then I think it won't work

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paul Lee + (paullee@lyft.com) +
+
2022-11-28 12:50:39
 +
+

*Thread Reply:* @Maciej Obuchowski we are still on airflow 1.10.15 unfortunately.

+ +

cc. @Eli Schachar @Allison Suarez

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paul Lee + (paullee@lyft.com) +
+
2022-11-28 12:50:49
 +
+

*Thread Reply:* is there no workaround we can make work?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paul Lee + (paullee@lyft.com) +
+
2022-11-28 12:51:01
 +
+

*Thread Reply:* @Jakub Dardziński is this for airflow versions 2.0+?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Varun Singh + (varuntestaz@outlook.com) +
+
2022-11-21 07:07:10
 +
+

Hey, quick question: I see there is Kafka transport in the java client, but it's not supported in the spark integration, right?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-11-21 07:28:04
 +
+

*Thread Reply:* Yeah. However, to add it, just tiny bit of code would be required.

+ +

Either in the URL version https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/app/src/main/java/io/openlineage/spark/agent/EventEmitter.java#L48

+ +

Or as separate Spark config entry: https://github.com/OpenLineage/OpenLineage/blob/182d2e5a907e6602f7fe132e07ea569c7e[…]n/java/io/openlineage/spark/agent/OpenLineageSparkListener.java

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/app/src/main/java/io/openlineage/spark/agent/EventEmitter.java | EventEmitter.java>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/182d2e5a907e6602f7fe132e07ea569c7e923dff/integration/spark/app/src/main/java/io/openlineage/spark/agent/OpenLineageSparkListener.java | OpenLineageSparkListener.java>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
srutikanta hota + (srutikanta.hota@gmail.com) +
+
2022-11-22 13:03:41
 +
+

How can we auto instrument a dataset owner at Java agent level? Is there any spark property available?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
srutikanta hota + (srutikanta.hota@gmail.com) +
+
2022-11-22 16:47:37
 +
+

Is there a way if we are running a job with business day as yesterday to capture the information. Just think if I am running yesterday missing job today. Or Friday's file on Monday as we received file late from vendor etc..

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-11-22 18:45:48
 +
+

*Thread Reply:* I think that's what NominalTimeFacet covers

+
+
openlineage.io
+ + + +
Nominal Time Facet | OpenLineage Docs
+
+ The facet to describe the nominal start and end time of the run. The nominal usually means the time the job run was expected to run (like a scheduled time), and the actual time can be different. +
+ + + + + + +
Original URL: https://openlineage.io/docs/spec/facets/run-facets/nominal_time
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Rahul Sharma + (panditrahul151197@gmail.com) +
+
2022-11-24 09:15:45
 +
+

hello Team, i wanna to use data lineage using airflow but not getting understand from docs please let me know if someone have pretty docs

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harel Shein + (harel.shein@gmail.com) +
+
2022-11-28 10:29:58
 +
+

*Thread Reply:* Hey @Rahul Sharma, what version of Airflow are you running?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Rahul Sharma + (panditrahul151197@gmail.com) +
+
2022-11-28 10:30:14
 +
+

*Thread Reply:* i am using airflow 2.x

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Rahul Sharma + (panditrahul151197@gmail.com) +
+
2022-11-28 10:30:27
 +
+

*Thread Reply:* can we connect if you have time ?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harel Shein + (harel.shein@gmail.com) +
+
2022-11-28 11:11:58
 +
+

*Thread Reply:* did you see these docs before? https://openlineage.io/integration/apache-airflow/#airflow-20

+
+
openlineage.io
+ + + +
Apache Airflow
+
+ Enabling OpenLineage in Apache Airflow automatically tracks metadata about jobs and datasets as DAGs execute. +
+ + + + + + +
Original URL: https://openlineage.io/integration/apache-airflow/#airflow-20
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Rahul Sharma + (panditrahul151197@gmail.com) +
+
2022-11-28 11:12:22
 +
+

*Thread Reply:* yes

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Rahul Sharma + (panditrahul151197@gmail.com) +
+
2022-11-28 11:12:36
 +
+

*Thread Reply:* i already set configuration in airflow.cfg file

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harel Shein + (harel.shein@gmail.com) +
+
2022-11-28 11:12:57
 +
+

*Thread Reply:* where are you sending the events to?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Rahul Sharma + (panditrahul151197@gmail.com) +
+
2022-11-28 11:13:24
 +
+

*Thread Reply:* i have a docker machine on which marquez is working

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harel Shein + (harel.shein@gmail.com) +
+
2022-11-28 11:13:47
 +
+

*Thread Reply:* so, what is the issue you are seeing?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Rahul Sharma + (panditrahul151197@gmail.com) +
+
2022-11-28 11:15:37
 +
+

*Thread Reply:* there is no error

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Rahul Sharma + (panditrahul151197@gmail.com) +
+
2022-11-28 11:16:01
 +
+

*Thread Reply:* ```[lineage]

+ +

what lineage backend to use

+ +

backend =openlineage.lineage_backend.OpenLineageBackend

+ +

MARQUEZ_URL=http://10.36.37.178:3000

+ +

MARQUEZ_NAMESPACE=airflow

+ +

MARQUEZBACKEND=HTTP +MARQUEZURL=http://10.36.37.178:5000

+ +

MARQUEZAPIKEY=[YOURAPIKEY]

+ +

MARQUEZ_NAMESPACE=airflow```

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Rahul Sharma + (panditrahul151197@gmail.com) +
+
2022-11-28 11:16:09
 +
+

*Thread Reply:* above config i have set

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Rahul Sharma + (panditrahul151197@gmail.com) +
+
2022-11-28 11:16:22
 +
+

*Thread Reply:* please let me know any other thing need to do

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mohamed Nabil H + (m.nabil.hafez@gmail.com) +
+
2022-11-24 14:02:27
 +
+

hey i wonder if somebody can link me to the lineage ( table lineage ) event schema ?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2022-11-25 02:20:40
 +
+

*Thread Reply:* please have a look at openapi definition of the event: https://openlineage.io/apidocs/openapi/

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Murali Krishna + (vmurali.krishnaraju@genpact.com) +
+
2022-11-30 02:34:51
 +
+

Hello Team, I am from Genpact Data Analytics team, we are looking for demo of your product

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Conor Beverland + (conorbev@gmail.com) +
+
2022-11-30 14:10:10
 +
+

*Thread Reply:* hey, I'll DM you.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-12-01 15:00:28
 +
+

Hello all, I’m calling for a vote on releasing OpenLineage 0.18.0, including: +• improvements to the Spark integration, +• extractors for Sagemaker operators and SFTPOperator in the Airflow integration, +• a change to the Databricks integration to support Databricks Runtime 11.3, +• new governance docs, +• bug fixes, +• and more. +Three +1s from committers will authorize an immediate release.

+ + + +
+ ➕ Maciej Obuchowski, Will Johnson, Bramha Aelem +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-12-06 13:56:17
 +
+

*Thread Reply:* Thanks, all. The release is authorized will be initiated within two business days.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-12-01 15:11:10
 +
+

@channel +This month’s OpenLineage TSC meeting is next Thursday, December 8th, at 10 am PT. Join us on Zoom: https://bit.ly/OLzoom. All are welcome! +On the tentative agenda:

+ +
  1. an overview of the new Rust implementation of the SQL integration
  2. a pesentation/discussion of what it actually means to “implement” OpenLineage
  3. open discussion.
    4. +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Scott Anderson + (scott.anderson@alteryx.com) +
+
2022-12-02 13:57:07
 +
+

Hello everyone! General question here, aside from ‘consumer’ orgs/integrations (dbt/dagster/manta), is anyone aware of any enterprise organizations that are leveraging OpenLineage today? Example lighthouse brands?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-12-02 15:21:20
 +
+

*Thread Reply:* Microsoft https://openlineage.io/blog/openlineage-microsoft-purview/

+
+
openlineage.io
+ + + +
Microsoft Purview Accelerates Lineage Extraction from Azure Databricks
+
+ A new collaboration between Microsoft and OpenLineage is making lineage extraction possible for Azure Databricks and Microsoft Purview users. +
+ + + + + + +
Original URL: https://openlineage.io/blog/openlineage-microsoft-purview/
+ + + +
+ + + +
+ 🙌 Will Johnson +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-12-05 13:54:06
 +
+

*Thread Reply:* I think we can share that we have over 2,000 installs of that Microsoft solution accelerator using OpenLineage.

+ +

That means we have thousands of companies having experimented with OpenLineage and Microsoft Purview.

+ +

We can't name any customers at this point unfortunately.

+ + + +
+ 🎉 Conor Beverland, Kengo Seki +
+ +
+ 👍 Scott Anderson +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-12-07 12:03:06
 +
+

@channel +This month’s TSC meeting is tomorrow at 10 am PT. All are welcome! https://openlineage.slack.com/archives/C01CK9T7HKR/p1669925470878699

+
+ + +
+ + + } + + Michael Robinson + (https://openlineage.slack.com/team/U02LXF3HUN7) +
+ + + +
+
+ <p><strong>@channel</strong> +This month’s OpenLineage TSC meeting is <strong>next Thursday, December 8th, at 10 am PT</strong>. Join us on Zoom: <a href="https://bit.ly/OLzoom">https://bit.ly/OLzoom</a>. All are welcome! +On the tentative agenda:</p> + +<ol><li>an overview of the new Rust implementation of the SQL integration</li><li>a pesentation/discussion of what it actually means to “implement” OpenLineage</li><li>open discussion.</li> +</ol> +
+ + + + + + +
Original URL: https://openlineage.slack.com/archives/C01CK9T7HKR/p1669925470878699
+ + +
+ + Slack Conversation +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-12-07 14:22:58
 +
+

*Thread Reply:* For open discussion, I'd like to ask the team for an overview of how the different gradle files are working together for the Spark implementation. I'm terribly confused on where dependencies need to be added (whether it's in shared, app, or a spark version specific folder). Maybe @Maciej Obuchowski...?

+ + + +
+ 👍 Michael Robinson +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-12-07 14:25:12
 +
+

*Thread Reply:* Unfortunately I'll be unable to attend the meeting @Will Johnson 😞

+ + + +
+ 😭 Will Johnson +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2022-12-08 13:03:08
 +
+

*Thread Reply:* This is starting now. CC @Will Johnson

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2022-12-09 19:24:15
 +
+

*Thread Reply:* @Will Johnson Check the notes and the recording. @Michael Collado did a pass at explaining the relationship between shared, app and the versions

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2022-12-09 19:24:30
 +
+

*Thread Reply:* feel free to follow up here as well

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2022-12-09 19:39:37
 +
+

*Thread Reply:* ascii art to the rescue! (top “depends on” bottom)

+ +
              /   \
+             / / \ \
+            / /   \ \
+           / /     \ \
+          / /       \ \
+         / |         | \
+        /  |         |  \
+       /   |         |   \
+      /    |         |    \
+     /     |         |     \
+    /      |         |      \
+   /       |         |       \
+spark2   spark3   spark32   spark33
+   \        |        |       /
+    \       |        |      /
+     \      |        |     /
+      \     |        |    /
+       \    |        |   /
+        \   |        |  /
+         \  |        | /
+          \ |       / /
+           \ \     / /
+            \ \   / /
+             \ \ / /
+              \   /
+               \ /
+             share
+
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2022-12-09 19:40:05
 +
+

*Thread Reply:* 😍

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2022-12-09 19:41:13
 +
+

*Thread Reply:* (btw, we should have written datakin to output ascii art; it’s obviously the superior way to generate graphs 😜)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2022-12-14 05:18:53
 +
+

*Thread Reply:* Hi, is there a recording for this meeting?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Christian Lundgren + (christian@lunit.io) +
+
2022-12-07 20:33:19
 +
+

Hi! I have a basic question about the naming conventions for blob storage. The spec is not totally clear to me. Is the convention to use (1) namespace=bucket name=bucket+path or (2) namespace=bucket name=path?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2022-12-07 22:05:25
 +
+

*Thread Reply:* The namespace is the bucket and the dataset name is the path. Is there a blob storage provider in particular you are thinking of?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Christian Lundgren + (christian@lunit.io) +
+
2022-12-07 23:13:41
 +
+

*Thread Reply:* Thanks, that makes sense. We use GCS, so it is already covered by the naming conventions documented. I was just not sure if I was understanding the document correctly or not.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2022-12-07 23:34:33
 +
+

*Thread Reply:* No problem. Let us know if you have suggestions on the wording to make the doc clearer

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-12-08 11:44:49
 +
+

@channel +OpenLineage 0.18.0 is available now, featuring: +• Airflow: support SQLExecuteQueryOperator #1379 @JDarDagran +• Airflow: introduce a new extractor for SFTPOperator #1263 @sekikn +• Airflow: add Sagemaker extractors #1136 @fhoda +• Airflow: add S3 extractor for Airflow operators #1166 @fhoda +• Spec: add spec file for ExternalQueryRunFacet #1262 @howardyoo +• Docs: add a TSC doc #1303 @merobi-hub +• Plus bug fixes. +Thanks to all our contributors, including new contributor @Faisal Hoda! +Release: https://github.com/OpenLineage/OpenLineage/releases/tag/0.18.0 +Changelog: https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md +Commit history: https://github.com/OpenLineage/OpenLineage/compare/0.17.0...0.18.0 +Maven: https://oss.sonatype.org/#nexus-search;quick~openlineage +PyPI: https://pypi.org/project/openlineage-python/

+ + + +
+ 🚀 Willy Lulciuc, Minkyu Park, Kengo Seki, Enrico Rotundo, Faisal Hoda +
+ +
+ 🙌 Howard Yoo, Minkyu Park, Kengo Seki, Enrico Rotundo, Faisal Hoda +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
srutikanta hota + (srutikanta.hota@gmail.com) +
+
2022-12-09 01:42:59
 +
+

1) Is there a specifications to capture dataset dependency. ds1 is dependent on ds2

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-12-09 11:51:16
 +
+

*Thread Reply:* Dataset dependencies are represented through common relationship with a Job - e.g., the task that performed the transformation.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
srutikanta hota + (srutikanta.hota@gmail.com) +
+
2022-12-11 09:01:19
 +
+

*Thread Reply:* Is it possible to populate table level dependency without any transformation using open lineage specifications? Like to define dataset 1 is dependent of table 1 and table 2 which can be represented as separate datasets

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-12-13 15:24:20
 +
+

*Thread Reply:* Not explicitly, in today's spec. The guiding principle is that something created that dependency, and the dependency changes over time in a way that is important to study.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-12-13 15:25:12
 +
+

*Thread Reply:* I say this to explain why it is the way it is - but the spec can change over time to serve new uses cases, certainly!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2022-12-14 05:18:10
 +
+

Hi everyone, I'd like to use openlineage to capture column level lineage for spark. I would also like to capture a few custom environment variables along with the column lineage. May I know how this can be done? Thanks!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2022-12-14 09:56:22
 +
+

*Thread Reply:* Hi @Anirudh Shrinivason, you could start with column-lineage & spark workshop available here -> https://github.com/OpenLineage/workshops/tree/main/spark

+ + + +
+ ❤️ Ricardo Gaspar +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2022-12-14 10:05:54
 +
+

*Thread Reply:* Hi @Paweł Leszczyński Thanks for the link! But this does not really answer the concern.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2022-12-14 10:06:08
 +
+

*Thread Reply:* I am already able to capture column lineage

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2022-12-14 10:06:33
 +
+

*Thread Reply:* What I would like is to capture some extra environment variables, and send it to the server along with the lineage

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2022-12-14 11:22:59
 +
+

*Thread Reply:* i remember we already have a facet for that: https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/shared/src/main/java/io/openlineage/spark/agent/facets/EnvironmentFacet.java

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/shared/src/main/java/io/openlineage/spark/agent/facets/EnvironmentFacet.java | EnvironmentFacet.java>
+
+ <pre><code>/** +/** Copyright 2018-2022 contributors to the OpenLineage project +/** SPDX-License-Identifier: Apache-2.0 +**/ + +package io.openlineage.spark.agent.facets; + +import com.fasterxml.jackson.annotation.JsonProperty; +import io.openlineage.client.OpenLineage; +import io.openlineage.spark.agent.Versions; +import java.util.Map; + +/**** + ** Facet used to report environment specific properties. For example, reporting the name of the + ** cluster used, reporting certain environment variables, or resolving mount points. + **/ +public class EnvironmentFacet extends OpenLineage.DefaultRunFacet { + @JsonProperty("environment-properties") + @SuppressWarnings("PMD") + private Map&lt;String, Object&gt; properties; + + public EnvironmentFacet(Map&lt;String, Object&gt; environmentDetails) { + super(Versions.OPEN_LINEAGE_PRODUCER_URI); + this.properties = environmentDetails; + } +} + +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2022-12-14 11:24:07
 +
+

*Thread Reply:* but it is only used at the moment to capture some databricks environment attributes

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2022-12-14 11:28:29
 +
+

*Thread Reply:* so you can contribute to project and add a feature which adds specified/al environment variables to lineage event.

+ +

you can also have a look at extending section of spark integration docs (https://github.com/OpenLineage/OpenLineage/tree/main/integration/spark#extending) and create a class thats add run facet builder according to your needs.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2022-12-14 11:29:28
 +
+

*Thread Reply:* the third way is to create an issue related to this bcz being able to send selected/all environment variables in OL event seems to be really cool feature.

+ + + +
+ 👍 Anirudh Shrinivason +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2022-12-14 21:49:19
 +
+

*Thread Reply:* That is great! Thank you so much! This really helps!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2022-12-15 01:44:42
 +
+

*Thread Reply:* List&lt;String&gt; dbPropertiesKeys = + Arrays.asList( + "orgId", + "spark.databricks.clusterUsageTags.clusterOwnerOrgId", + "spark.databricks.notebook.path", + "spark.databricks.job.type", + "spark.databricks.job.id", + "spark.databricks.job.runId", + "user", + "userId", + "spark.databricks.clusterUsageTags.clusterName", + "spark.databricks.clusterUsageTags.azureSubscriptionId"); + dbPropertiesKeys.stream() + .forEach( + (p) -&gt; { + dbProperties.put(p, jobStart.properties().getProperty(p)); + }); +It seems like it is obtaining these env variable information from the jobStart obj, but not capturing from the env directly?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2022-12-15 01:57:05
 +
+

*Thread Reply:* I have opened an issue in the community here: https://github.com/OpenLineage/OpenLineage/issues/1419

+
+ + + + +
#1419 [PROPOSAL] Add custom environment variables to the RunEvent
+
+ <p>Hi, <br /> +I would like to capture some custom environment variables and send it along with the column lineage to the server. This feature will be of great use as there is a potential use case to capture and persist some environment-specific metadata from spark clusters. <br /> +Wondering if this feature can be supported, or is on the roadmap potentially? Thanks a lot! <br /> +cc: <a href="https://github.com/pawel-big-lebowski">@pawel-big-lebowski</a></p> +
+ +
+
Labels
+ proposal +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-02-01 02:24:39
 +
+

*Thread Reply:* Hi @Paweł Leszczyński I have opened a PR for helping to add this use case. Please do help to see if we can merge it in. Thanks! +https://github.com/OpenLineage/OpenLineage/pull/1545

+
+ + + + +
#1545 Add CustomEnvironmentFacetBuilder class
+
+ <p><strong>Problem</strong></p> + +<p>The main idea behind this pull request is to enable a way to capture custom environment variables from a spark integration. The flow defined here, as applicable to our use case is to have a CUSTOM_VARS="VAR1,VAR2,VAR3..." environment variable that indicates which environment variables should be captured. Then, we would want to capture those variables, ie VAR1, VAR2, VAR3, ... from the spark clusters.</p> + +<p>Closes: <a href="https://github.com/OpenLineage/OpenLineage/issues/1419">#1419</a></p> + +<p><strong>Solution</strong></p> + +<p>The solution proposed extends the EnvironmentFacets class, to build a CustomEnvironmentFacetBuilder class. This class then is used to get the values of the environment variables and pass on to the startEvent json.</p> + +<p>☐ Your change modifies the <a href="https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.json">core</a> OpenLineage model +☑︎ Your change modifies one or more OpenLineage <a href="https://github.com/OpenLineage/OpenLineage/tree/main/spec/facets">facets</a></p> + +<p>If you're contributing a new integration, please specify the scope of the integration and how/where it has been tested (e.g., Apache Spark integration supports <code>S3</code> and <code>GCS</code> filesystem operations, tested with AWS EMR).</p> + +<p><strong>Checklist</strong></p> + +<p>☑︎ You've <a href="https://github.com/OpenLineage/OpenLineage/blob/main/why-the-dco.md">signed-off</a> your work +☐ Your pull request title follows our <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CONTRIBUTING.md#creating-pull-requests">guidelines</a> +☐ Your changes are accompanied by tests (<em>if relevant</em>) +☐ Your change contains a <a href="https://kurtisnusbaum.medium.com/stacked-diffs-keeping-phabricator-diffs-small-d9964f4dcfa6">small diff</a> and is self-contained +☐ You've updated any relevant documentation (<em>if relevant</em>) +☐ You've updated the <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md"><code>CHANGELOG.md</code></a> with details about your change under the "Unreleased" section (<em>if relevant, depending on the change, this may not be necessary</em>) +☐ You've versioned the core OpenLineage model or facets according to <a href="https://docs.snowplowanalytics.com/docs/pipeline-components-and-applications/iglu/common-architecture/schemaver">SchemaVer</a> (<em>if relevant</em>) +☐ You've added a <a href="https://github.com/OpenLineage/OpenLineage/tree/main/.github/header_templates.md">header</a> to source files (<em>if relevant</em>)</p> + +<hr /> + +<p>SPDX-License-Identifier: Apache-2.0 <br /> +Copyright 2018-2023 contributors to the OpenLineage project</p> +
+ +
+
Labels
+ integration/spark +
+ +
+
Comments
+ 1 +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+ 👀 Maciej Obuchowski, Ross Turk +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-02-02 11:45:52
 +
+

*Thread Reply:* Hey @Anirudh Shrinivason, sorry for late reply, but I reviewed the PR.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-02-06 03:06:42
 +
+

*Thread Reply:* Hey thanks a lot! I have made the requested changes! Thanks!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-02-06 03:06:49
 +
+

*Thread Reply:* @Maciej Obuchowski ^ 🙂

+ + + +
+ 👀 Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-02-06 09:09:34
 +
+

*Thread Reply:* Hey @Anirudh Shrinivason, took a look at it but it unfortunately fails integration tests (throws NPE), can you take a look again?

+ +

23/02/06 12:18:39 ERROR AsyncEventQueue: Listener OpenLineageSparkListener threw an exception + java.lang.NullPointerException + at io.openlineage.spark.agent.EventEmitter.&lt;init&gt;(EventEmitter.java:39) + at io.openlineage.spark.agent.OpenLineageSparkListener.initializeContextFactoryIfNotInitialized(OpenLineageSparkListener.java:276) + at io.openlineage.spark.agent.OpenLineageSparkListener.onOtherEvent(OpenLineageSparkListener.java:80) + at org.apache.spark.scheduler.SparkListenerBus.doPostEvent(SparkListenerBus.scala:100) + at org.apache.spark.scheduler.SparkListenerBus.doPostEvent$(SparkListenerBus.scala:28) + at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37) + at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37) + at org.apache.spark.util.ListenerBus.postToAll(ListenerBus.scala:117) + at org.apache.spark.util.ListenerBus.postToAll$(ListenerBus.scala:101) + at org.apache.spark.scheduler.AsyncEventQueue.super$postToAll(AsyncEventQueue.scala:105) + at org.apache.spark.scheduler.AsyncEventQueue.$anonfun$dispatch$1(AsyncEventQueue.scala:105) + at scala.runtime.java8.JFunction0$mcJ$sp.apply(JFunction0$mcJ$sp.java:23) + at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62) + at <a href="http://org.apache.spark.scheduler.AsyncEventQueue.org">org.apache.spark.scheduler.AsyncEventQueue.org</a>$apache$spark$scheduler$AsyncEventQueue$$dispatch(AsyncEventQueue.scala:100) + at org.apache.spark.scheduler.AsyncEventQueue$$anon$2.$anonfun$run$1(AsyncEventQueue.scala:96) + at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1433) + at org.apache.spark.scheduler.AsyncEventQueue$$anon$2.run(AsyncEventQueue.scala:96)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-02-07 04:17:02
 +
+

*Thread Reply:* Hi yeah my bad. It should be fixed in the latest push. But I think the tests are not running in the CI because of some GCP environment issue? I am not really sure how to fix it...

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-02-07 04:18:46
 +
+

*Thread Reply:* I can make them run, it's just that running them on forks is disabled. We need to make it more clear I suppose

+ + + +
+ 👍 Anirudh Shrinivason +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-02-07 04:24:38
 +
+

*Thread Reply:* Ahh I see thanks! Also, some of the tests are failing on my local, such as https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/app/src/test/java/io/openlineage/spark/agent/lifecycle/DeltaDataSourceTest.java. Is this expected behaviour?

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/app/src/test/java/io/openlineage/spark/agent/lifecycle/DeltaDataSourceTest.java | DeltaDataSourceTest.java>
+
+ <pre><code>/** +/** Copyright 2018-2023 contributors to the OpenLineage project +/** SPDX-License-Identifier: Apache-2.0 +**/ + +package io.openlineage.spark.agent.lifecycle; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertTrue; + +import io.openlineage.client.OpenLineage; +import io.openlineage.client.OpenLineage.RunEvent; +import io.openlineage.client.OpenLineage.RunEvent.EventType; +import io.openlineage.spark.agent.SparkAgentTestExtension; +import java.io.IOException; +import java.nio.file.Path; +import java.util.Arrays; +import java.util.List; +import java.util.Optional; +import java.util.concurrent.TimeoutException; +import org.apache.spark.sql.Dataset; +import org.apache.spark.sql.Row; +import org.apache.spark.sql.SaveMode; +import org.apache.spark.sql.SparkSession; +import org.apache.spark.sql.catalyst.expressions.GenericRowWithSchema; +import org.apache.spark.sql.types.LongType$; +import org.apache.spark.sql.types.Metadata; +import org.apache.spark.sql.types.StringType$; +import org.apache.spark.sql.types.StructField; +import org.apache.spark.sql.types.StructType; +import org.junit.jupiter.api.Tag; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.extension.ExtendWith; +import org.junit.jupiter.api.io.TempDir; +import org.mockito.ArgumentCaptor; +import org.mockito.Mockito; + +@ExtendWith(SparkAgentTestExtension.class) +@Tag("delta") +class DeltaDataSourceTest { + + @Test + void testInsertIntoDeltaSource(@TempDir Path tempDir, SparkSession spark) + throws IOException, InterruptedException, TimeoutException { + StructType tableSchema = + new StructType( + new StructField[] { + new StructField("name", StringType$.MODULE$, false, Metadata.empty()), + new StructField("age", LongType$.MODULE$, false, Metadata.empty()) + }); + Dataset&lt;Row&gt; df = + spark.createDataFrame( + Arrays.asList( + new GenericRowWithSchema(new Object[] {"john", 25L}, tableSchema), + new GenericRowWithSchema(new Object[] {"sam", 22L}, tableSchema), + new GenericRowWithSchema(new Object[] {"alicia", 35L}, tableSchema), + new GenericRowWithSchema(new Object[] {"bob", 47L}, tableSchema), + new GenericRowWithSchema(new Object[] {"jordan", 52L}, tableSchema), + new GenericRowWithSchema(new Object[] {"liz", 19L}, tableSchema), + new GenericRowWithSchema(new Object[] {"marcia", 83L}, tableSchema), + new GenericRowWithSchema(new Object[] {"maria", 40L}, tableSchema), + new GenericRowWithSchema(new Object[] {"luis", 8L}, tableSchema), + new GenericRowWithSchema(new Object[] {"gabriel", 30L}, tableSchema)), + tableSchema); + + String deltaDir = tempDir.resolve("deltaData").toAbsolutePath().toString(); + df.write().format("delta").option("path", deltaDir).mode(SaveMode.Overwrite).save(); + // wait for event processing to complete + StaticExecutionContextFactory.waitForExecutionEnd(); + + ArgumentCaptor&lt;RunEvent&gt; lineageEvent = ArgumentCaptor.forClass(RunEvent.class); + + Mockito.verify(SparkAgentTestExtension.OPEN_LINEAGE_SPARK_CONTEXT, Mockito.atLeast(2)) + .emit(lineageEvent.capture()); + List&lt;RunEvent&gt; events = lineageEvent.getAllValues(); + Optional&lt;RunEvent&gt; completionEvent = + events.stream() + .filter(e -&gt; e.getEventType().equals(EventType.COMPLETE) &amp;&amp; !e.getOutputs().isEmpty()) + .findFirst(); + assertTrue(completionEvent.isPresent()); + RunEvent event = completionEvent.get(); + List&lt;OpenLineage.OutputDataset&gt; outputs = event.getOutputs(); + assertEquals(1, outputs.size()); + assertEquals("file", outputs.get(0).getNamespace()); + assertEquals(deltaDir, outputs.get(0).getName()); + } +} + +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-02-07 07:20:11
 +
+

*Thread Reply:* tests failing isn't expected behaviour 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-02-08 03:37:23
 +
+

*Thread Reply:* Ahh yeap it was a local ide issue on my side. I added some tests to verify the presence of env variables too.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-02-08 03:47:22
 +
+

*Thread Reply:* @Anirudh Shrinivason let me know then when you'll push fixed version, I can run full tests then

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-02-08 03:49:35
 +
+

*Thread Reply:* I have pushed just now

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-02-08 03:49:39
 +
+

*Thread Reply:* You can run the tests

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-02-08 04:13:07
 +
+

*Thread Reply:* @Maciej Obuchowski mb I pushed again rn. Missed out a closing bracket.

+ + + +
+ 👍 Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-02-10 00:47:04
 +
+

*Thread Reply:* @Maciej Obuchowski Hi, could we merge this PR in? I'd like to see if we can have these changes in the new release...

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Bramha Aelem + (bramhaaelem@gmail.com) +
+
2022-12-15 17:14:02
 +
+

Hi All- I am sending lineage from ADF for each activity which i am performing. But the individual activities are representing correctly. How can I represent task1 as a parent to task2. can someone please share the sample json request for it.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-12-16 13:29:44
 +
+

*Thread Reply:* Hi 👋 this would require a series of JSON calls:

+ +
  1. start the first task
  2. end the first task, specify output dataset
  3. start the second task, specify input dataset
  4. end the second task
    5. +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-12-16 13:32:08
 +
+

*Thread Reply:* in OpenLineage relationships are typically Job -> Dataset -> Job, so +• you create a relationship between datasets by referring to them in the same job - i.e., this task ran that read from these datasets and wrote to those datasets +• you create a relationship between tasks by referring to the same datasets across both of them - i.e., this task wrote that dataset and this other task read from it

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-12-16 13:35:06
 +
+

*Thread Reply:* @Bramha Aelem if you look in this directory, you can find example start/complete JSON calls that show how to specify input/output datasets.

+ +

(it’s an airflow workshop, but those examples are for a part of the workshop that doesn’t involve airflow)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-12-16 13:35:46
 +
+

*Thread Reply:* (these can also be found in the docs)

+
+
openlineage.io
+ + + +
Example Lineage Events | OpenLineage Docs
+
+ Simple Examples +
+ + + + + + +
Original URL: https://openlineage.io/docs/development/examples
+ + + +
+ + + +
+ 👍 Ross Turk +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Bramha Aelem + (bramhaaelem@gmail.com) +
+
2022-12-16 14:49:30
 +
+

*Thread Reply:* @Ross Turk - Thanks for the details. will try and get back to you on it

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Bramha Aelem + (bramhaaelem@gmail.com) +
+
2022-12-17 19:53:21
 +
+

*Thread Reply:* @Ross Turk - Good Evening, It worked as expected. I am able to replicate the scenarios which I am looking for.

+ + + +
+ 👍 Ross Turk +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Bramha Aelem + (bramhaaelem@gmail.com) +
+
2022-12-17 19:53:48
 +
+

*Thread Reply:* @Ross Turk - Thanks for your response.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Bramha Aelem + (bramhaaelem@gmail.com) +
+
2023-01-12 13:23:56
 +
+

*Thread Reply:* @Ross Turk - First activity : I am making HTTP Call to pull the lookup data and store it in ADLS. +Second Activity : After the completion of first activity I am making Azure databricks call to use the lookup file and generate the output tables. How I can refer the databricks generated tables facets as an input to the subsequent activities in the pipeline. +When I refer it's as an input the spark tables metadata is not showing up. How can this be achievable. +After the execution of each activity in ADF Pipeline I am sending start and complete/fail event lineage to Marquez.

+ +

Can someone please guide me on this.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Bramha Aelem + (bramhaaelem@gmail.com) +
+
2022-12-15 17:19:34
 +
+

I am not using airflow in my Process. pls suggest

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Bramha Aelem + (bramhaaelem@gmail.com) +
+
2022-12-19 12:40:26
 +
+

Hi All - Good Morning, how the column lineage of data source when it ran by different teams and jobs in openlineage.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Al (Koii) + (al@koii.network) +
+
2022-12-20 14:26:57
 +
+

Hey folks! I'm al from Koii.network, very happy to have heard about this project :)

+ + + +
+ 👋 Willy Lulciuc, Maciej Obuchowski, Julien Le Dem +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2022-12-20 14:27:59
 +
+

*Thread Reply:* welcome! let’s us know if you have any questions

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Matt Menzenski + (matt@payitgov.com) +
+
2022-12-29 08:22:26
 +
+

Hello! I found the OpenLineage project today after searching for “OpenTelemetry” in the dbt Slack.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harel Shein + (harel.shein@gmail.com) +
+
2022-12-29 10:47:00
 +
+

*Thread Reply:* Hey Matt! Happy to have you here! Feel free to reach out if you have any questions

+ + + +
+ :gratitude_thank_you: Matt Menzenski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Max + (maxime.broussard@gmail.com) +
+
2022-12-30 05:33:40
 +
+

Hi guys - I am really excited to test open lineage. +I had a quick question, sorry if this is not the right place for it. +We are testing dbt-ol with airflow and I was hoping this would by default push the number of rows updated/created in that dbt transformation to marquez. +It runs fine on airflow, but when I check in marquez there doesn't seem to be a 'dataset' created, only 'jobs' with job level metadata. +When i check here I see that the dataset facets should have it though https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.md +Does anyone know if creating a dataset & sending row counts to OL is out of the box on dbt-ol or if I need to build another script to get that number from my snowflake instance and push it to OL as another step in my process? +Thanks a lot!

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.md | OpenLineage.md>
+
+ <p><strong>OpenLineage Spec</strong> +<strong>Specification</strong></p> + +<p>The specification for OpenLineage is formalized as a JsonSchema <OpenLineage.json|OpenLineage.json>. <br /> +An OpenAPI spec is also provided for HTTP based implementations: <OpenLineage.yml|OpenLineage.yml> <br /> +The documentation is published at: <a href="https://openlineage.github.io/">https://openlineage.github.io/</a> <br /> +It allows extensions to the spec using <code>Custom Facets</code> as described in this document.</p> + +<p><strong>Core concepts</strong> +<strong>Core Lineage Model</strong></p> + +<p><OpenLineageModel.svg|Open Lineage model></p> + +<p>• <strong>Run Event</strong>: an event describing an observed state of a job run. It is required to at least send a START event and a COMPLETE/FAIL/ABORT event. Additional events are optional. +• <strong>Job</strong>: a process definition that consumes and produces datasets (defined as its inputs and outputs). It is <Naming.md#Jobs|identified by a unique name within a namespace> (which is assigned to the scheduler starting the jobs). The <em>Job</em> evolves over time and this change is captured when the job runs. +• <strong>Dataset</strong>: an abstract representation of data. It has a <Naming.md#Datasets|unique name within the datasource namespace> derived from its physical location (for example db.host.database.schema.table). Typically, a <em>Dataset</em> changes when a job writing to it completes. Similarly to the <em>Job</em> and <em>Run</em> distinction, metadata that is more static from run to run is captured in a DatasetFacet (for example, the schema that does not change every run), what changes every <em>Run</em> is captured as an <em>InputFacet</em> or an <em>OutputFacet</em> (for example, what subset of the data set was read or written, like a time partition). +• <strong>Run</strong>: An instance of a running job with a start and completion (or failure) time. A run is identified by a globally unique ID relative to its job definition. A run ID <strong>must</strong> be an <a href="https://datatracker.ietf.org/doc/html/rfc4122">UUID</a>. +• <strong>Facet</strong>: A piece of metadata attached to one of the entities defined above.</p> + +<p>example: <br /> +Here is an example of a simple start run event not adding any facet information:</p> + +<pre><code>{ + "eventType": "START", + "eventTime": "2020-12-09T23:37:31.081Z", + "run": { + "runId": "3b452093-782c-4ef2-9c0c-aafe2aa6f34d", + }, + "job": { + "namespace": "my-scheduler-namespace", + "name": "myjob.mytask", + }, + "inputs": [ + { + "namespace": "my-datasource-namespace", + "name": "instance.schema.table", + } + ], + "outputs": [ + { + "namespace": "my-datasource-namespace", + "name": "instance.schema.output_table", + } + ], + "producer": "&lt;https://github.com/OpenLineage/OpenLineage/blob/v1-0-0/client&gt;", + "schemaURL": "&lt;https://openlineage.io/spec/1-0-0/OpenLineage.json#/definitions/RunEvent&gt;" +} +</code></pre> + +<p><strong>Lifecycle</strong></p> + +<p>The OpenLineage API defines events to capture the lifecycle of a <em>Run</em> for a given <em>Job</em>. <br /> +When a <em>job</em> is being <em>run</em>, we capture metadata by sending run events when the state of the job transitions to a different state. <br /> +We might observe different aspects of the job run at different stages. This means that different metadata might be collected in each event during the lifecycle of a run. <br /> +All metadata is additive. for example, if more inputs or outputs are detected as the job is running we might send additional events specifically for those datasets without re-emitting previously observed inputs or outputs. <br /> +Example:</p> + +<p>• When the run starts, we collect the following Metadata: + • Run Id + • Job id + • eventType: START + • event time + • source location and version (ex: git sha) + • If known: Job inputs and outputs. (input schema, ...) +• When the run completes: + • Run Id + • Job id + • eventType: COMPLETE + • event time + • Output datasets schema (and other metadata).</p> + +<p><strong>Facets</strong></p> + +<p>Facets are pieces of metadata that can be attached to the core entities:</p> + +<p>• Run +• Job +• Dataset (Inputs or Outputs)</p> + +<p>A facet is an atomic piece of metadata identified by its name. This means that emitting a new facet with the same name for the same entity replaces the previous facet instance for that entity entirely). It is defined as a JSON object that can be either part of the spec or custom facets defined in a different project.</p> + +<p>Custom facets must use a distinct prefix named after the project defining them to avoid collision with standard facets defined in the <OpenLineage.json|OpenLineage.json> spec. <br /> +They have a _schemaURL field pointing to the corresponding version of the facet schema (as a JSONPointer: <a href="https://swagger.io/docs/specification/using-ref/">$ref URL location</a> ).</p> + +<p>Example: <a href="https://raw.githubusercontent.com/OpenLineage/OpenLineage/main/spec/OpenLineage.json#/definitions/MyCustomJobFacet">https://raw.githubusercontent.com/OpenLineage/OpenLineage/main/spec/OpenLineage.json#/definitions/MyCustomJobFacet</a></p> + +<p>The versioned URL must be an immutable pointer to the version of the facet schema. For example, it should include a tag of a git sha and not a branch name. This should also be a canonical URL. There should be only one URL used for a given version of a schema.</p> + +<p>Custom facets can be promoted to the standard by including them in the spec.</p> + +<p><strong>Custom Facet Naming</strong></p> + +<p>Naming of custom facets should follow pattern <code>{prefix}{name}{entity}Facet</code> PascalCased. <br /> +Prefix must be distinct identifier named after the project defining them to avoid colision with standard facets defined in the <OpenLineage.json|OpenLineage.json> spec. <br /> +Entity is the core entity for which the facet is attached.</p> + +<p>When attached to core entity, the key should follow pattern <code>{prefix}_{name}</code>, where both prefix and name follow snakeCase pattern.</p> + +<p>Example of valid name is <code>BigQueryStatisticsJobFacet</code> and it's key <code>bigQuery_statistics</code>.</p> + +<p><strong>Standard Facets</strong> +<strong>Run Facets</strong></p> + +<p>• <strong>nominalTime</strong>: Captures the time this run is scheduled for. This is a typical usage for time based scheduled job. The job has a nominal schedule time that will be different from the actual time it is running at. +• <strong>parent</strong>: Captures the parent job and Run when the run was spawn from a parent run. For example in the case of Airflow, there's a run for the DAG that then spawns runs for individual tasks that would refer to the parent run as the DAG run. Similarly when a SparkOperator starts a Spark job, this creates a separate run that refers to the task run as its parent. +• <strong>errorMessage</strong>: Captures potential error message, programming language - and optionally stack trace - with which the run failed.</p> + +<p><strong>Job Facets</strong></p> + +<p>• <strong>sourceCodeLocation</strong>: Captures the source code location and version (example: git sha) of the job. +• <strong>sourceCode</strong>: Captures language (ex. python) and actual source code of the job. +• <strong>sql</strong>: Capture the SQL query if this job is a SQL query. +• <strong>ownership</strong>: Captures the owners of the job</p> + +<p><strong>Dataset Facets</strong></p> + +<p>• <strong>schema</strong>: Captures the schema of the dataset +• <strong>dataSource</strong>: Captures the Database instance containing this datasets (ex: Database schema. Object store bucket, ...) +• <strong>lifecycleStateChange</strong>: Captures the lifecycle states of the dataset like: alter, create, drop, overwrite, rename, truncate. +• <strong>version</strong>: Captures the dataset version when versioning is defined by database (ex. Iceberg snapshot ID) +• <facets/ColumnLineageDatasetFacet.md|<strong>columnLineage</strong>&gt;: Captures the column-level lineage +• <strong>ownership</strong>: Captures the owners of the dataset</p> + +<p><strong>Input Dataset Facets</strong></p> + +<p>• <strong>dataQualityMetrics</strong>: Captures dataset level and column level data quality metrics when scanning a dataset whith a DataQuality library (row count, byte size, null count, distinct count, average, min, max, quantiles). +• <strong>dataQualityAssertions</strong>: Captures the result of running data tests on dataset or its columns.</p> + +<p><strong>Output Dataset Facets</strong></p> + +<p>• <strong>outputStatistics</strong>: Captures the size of the output written to a dataset (row count and byte size).</p> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Viraj Parekh + (vmpvmp94@gmail.com) +
+
2023-01-03 13:20:14
 +
+

*Thread Reply:* @Ross Turk maybe you can help with this?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2023-01-03 13:34:23
 +
+

*Thread Reply:* hmm, I believe the dbt-ol integration does capture bytes/rows, but only for some data sources: https://github.com/OpenLineage/OpenLineage/blob/6ae1fd5665d5fd539b05d044f9b6fb831ce9d475/integration/common/openlineage/common/provider/dbt.py#L567

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/6ae1fd5665d5fd539b05d044f9b6fb831ce9d475/integration/common/openlineage/common/provider/dbt.py | dbt.py>
+
+ <pre><code> def node_to_output_dataset( +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2023-01-03 13:34:58
 +
+

*Thread Reply:* I haven't personally tried it with Snowflake in a few versions, but the code suggests that it's one of them.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2023-01-03 13:35:42
 +
+

*Thread Reply:* @Max you say your dbt-ol run is resulting in only jobs and no datasets emitted, is that correct?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2023-01-03 13:38:06
 +
+

*Thread Reply:* if so, I'd say something rather strange is going on because in my experience each model should result in a Job and a Dataset.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kuldeep + (kuldeep.marathe@affirm.com) +
+
2023-01-03 00:41:09
 +
+

Hi All, Curious to see if there is an openlineage integration with luigi or any open source projects working on it.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kuldeep + (kuldeep.marathe@affirm.com) +
+
2023-01-03 01:53:10
 +
+

*Thread Reply:* I was looking for something similar to the airflow integration

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Viraj Parekh + (vmpvmp94@gmail.com) +
+
2023-01-03 13:21:18
 +
+

*Thread Reply:* hey @Kuldeep - i don't think there's something for Luigi right now - is that something you'd potentially be interested in?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kuldeep + (kuldeep.marathe@affirm.com) +
+
2023-01-03 13:23:53
 +
+

*Thread Reply:* @Viraj Parekh Yes this is something we are interested in! There are a lot of projects out there that use luigi

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-01-03 11:05:48
 +
+

Hello all, I’m opening a vote to release OpenLineage 0.19.0, including: +• new extractors for Trino and S3FileTransformOperator in the Airflow integration +• a new, standardized run facet in the Airflow integration +• a new NominalTimeRunFacet and OwnershipJobFacet in the Airflow integration +• Postgres support in the dbt integration +• a new client-side proxy (skeletal version) +• a new, improved mechanism for passing conf parameters to the OpenLineage client in the Spark integration +• a new ExtractionErrorRunFacet to reflect internal processing errors for the SQL parser +• testing improvements, bug fixes and more. +As always, three +1s from committers will authorize an immediate release. Thanks in advance!

+ + + +
+ ➕ Willy Lulciuc, Maciej Obuchowski, Paweł Leszczyński, Jakub Dardziński, Julien Le Dem +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-01-03 23:07:59
 +
+

*Thread Reply:* Hi @Michael Robinson a new, improved mechanism for passing conf parameters to the OpenLineage client in the Spark integration +Would it be possible to have more details on what this entails please? Thanks!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-01-04 09:21:46
 +
+

*Thread Reply:* @Tomasz Nazarewicz might explain this better

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tomasz Nazarewicz + (tomasz.nazarewicz@getindata.com) +
+
2023-01-04 10:04:22
 +
+

*Thread Reply:* @Anirudh Shrinivason until now If you wanted to add new property to OL client, you had to also implement it in the integration because it had to parse all properties, create appropriate objects etc. New implementation makes client properties transparent to integration, they are only passed through and parsing happens inside the client.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-01-04 13:02:39
 +
+

*Thread Reply:* Thanks, all. The release is authorized and will commence shortly 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-01-04 22:00:55
 +
+

*Thread Reply:* @Tomasz Nazarewicz Ahh I see. Okay thanks!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-01-05 10:37:09
 +
+

@channel +This month’s OpenLineage TSC meeting is next Thursday, January 12th, at 10 am PT. Join us on Zoom: https://bit.ly/OLzoom. All are welcome! +On the tentative agenda:

+ +
  1. Recent release overview @Michael Robinson
  2. Column lineage update @Maciej Obuchowski
  3. Airflow integration improvements @Jakub Dardziński
  4. Discussions: +• Real-world implementation of OpenLineage (What does it really mean?) @Sheeri Cabral (Collibra) +• Using namespaces @Michael Robinson
  5. Open discussion +Notes: https://bit.ly/OLwiki +Is there a topic you think the community should discuss at this or a future meeting? Reply or DM me to add items to the agenda.
    6. +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-01-05 23:45:38
 +
+

*Thread Reply:* @Michael Robinson Will there be a recording?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-01-06 09:10:50
 +
+

*Thread Reply:* @Anirudh Shrinivason Yes, and the recording will be here: https://wiki.lfaidata.foundation/display/OpenLineage/Monthly+TSC+meeting

+ + + +
+ :gratitude_thank_you: Anirudh Shrinivason +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-01-05 13:00:01
 +
+

OpenLineage 0.19.2 is available now, including: +• Airflow: add Trino extractor #1288 @sekikn +• Airflow: add S3FileTransformOperator extractor #1450 @sekikn +• Airflow: add standardized run facet #1413 @JDarDagran +• Airflow: add NominalTimeRunFacet and OwnershipJobFacet #1410 @JDarDagran +• dbt: add support for postgres datasources #1417 @julienledem +• Proxy: add client-side proxy (skeletal version) #1439 #1420 @fm100 +• Proxy: add CI job to publish Docker image #1086 @wslulciuc +• SQL: add ExtractionErrorRunFacet #1442 @mobuchowski +• SQL: add column-level lineage to SQL parser #1432 #1461 @mobuchowski @StarostaGit +• Spark: pass config parameters to the OL client #1383 @tnazarew +• Plus bug fixes and testing and CI improvements. +Thanks to all the contributors, including new contributor Saurabh (@versaurabh) +Release: https://github.com/OpenLineage/OpenLineage/releases/tag/0.19.2 +Changelog: https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md +Commit history: https://github.com/OpenLineage/OpenLineage/compare/0.18.0...0.19.2 +Maven: https://oss.sonatype.org/#nexus-search;quick~openlineage +PyPI: https://pypi.org/project/openlineage-python/

+ + + +
+ ❤️ Julien Le Dem, Howard Yoo, Willy Lulciuc, Maciej Obuchowski, Kengo Seki, Harel Shein, Jarek Potiuk, Varun Singh +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2023-01-06 01:07:18
 +
+

Question on Spark Integration and External Hive Metastores

+ +

@Hanna Moazam and I are working with a team using OpenLineage and wants to extract out the server name of the hive metastore they're using when writing to a Hive table through Spark.

+ +

For example, the hive metastore is an Azure SQL database and the table name is sales.transactions.

+ +

OpenLineage will give something like /usr/hive/warehouse/sales.db/transactions for the name.

+ +

However, this is not a complete picture since sales.db/transactions is defined like this for a given hive metastore. In Hive, you'd define the fully qualified name as sales.transactions@sqlservername.database.windows.net .

+ +

Has anyone else come across this before? If not, we plan on raising an issue and suggesting we extract out the spark.hadoop.javax.jdo.option.ConnectionURL in the DatabricksEnvironmentFacetBuilder but ideally there would be a better way of extracting this.

+ +

https://learn.microsoft.com/en-us/azure/databricks/data/metastores/external-hive-metastore#set-up-an-external-metastore-using-the-ui

+ +

There was an issue by @Maciej Obuchowski or @Paweł Leszczyński that talked about providing a facet of the alias of a path but I can't find it at this point :(

+
+
learn.microsoft.com
+ + + +
External Apache Hive metastore - Azure Databricks
+
+ Learn how to connect to external Apache Hive metastores in Azure Databricks. +
+ + + + + + + + +
Original URL: https://learn.microsoft.com/en-us/azure/databricks/data/metastores/external-hive-metastore#set-up-an-external-metastore-using-the-ui
+ + + +
+ + + +
+ 👀 Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-01-09 02:28:43
 +
+

*Thread Reply:* Hi @Hanna Moazam, we've written Jupyter notebook to demo dataset symlinks feature: +https://github.com/OpenLineage/workshops/blob/main/spark/dataset_symlinks.ipynb

+ +

For scenario you describe, there should be symlink facet sent similar to: +{ + "_producer": "<https://github.com/OpenLineage/OpenLineage/tree/0.15.1/integration/spark>", + "_schemaURL": "<https://openlineage.io/spec/facets/1-0-0/SymlinksDatasetFacet.json#/$defs/SymlinksDatasetFacet>", + "identifiers": [ + { + "namespace": "<hive://metastore>", + "name": "default.some_table", + "type": "TABLE" + } + ] +} +Within Openlineage Spark integration code, symlinks are included here: +https://github.com/OpenLineage/OpenLineage/blob/0.19.2/integration/spark/shared/src/main/java/io/openlineage/spark/agent/util/PathUtils.java#L75

+ +

and they are added only when spark catalog is hive and metastore URI in spark conf is present.

+
+ + + + +
<https://github.com/OpenLineage/workshops/blob/main/spark/dataset_symlinks.ipynb | dataset_symlinks.ipynb>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/workshops">OpenLineage/workshops</a> +
+ + +
+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/0.19.2/integration/spark/shared/src/main/java/io/openlineage/spark/agent/util/PathUtils.java | PathUtils.java>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+ ➕ Maciej Obuchowski +
+ +
+ 🤯 Will Johnson +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2023-01-09 14:21:10
 +
+

*Thread Reply:* This is so awesome, @Paweł Leszczyński - Thank you so much for sharing this! I'm wondering if we could extend this to capture the hive JDBC Connection URL. I will explore this and put in an issue and PR to try and extend it. Thank you for the insights!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-01-11 12:00:02
 +
+

@channel +Friendly reminder: this month’s OpenLineage TSC meeting is tomorrow at 10am, and all are welcome. https://openlineage.slack.com/archives/C01CK9T7HKR/p1672933029317449

+
+ + +
+ + + } + + Michael Robinson + (https://openlineage.slack.com/team/U02LXF3HUN7) +
+ + + +
+
+ <p><strong>@channel</strong> +This month’s OpenLineage TSC meeting is <strong>next Thursday, January 12th, at 10 am PT</strong>. Join us on Zoom: <a href="https://bit.ly/OLzoom">https://bit.ly/OLzoom</a>. All are welcome! +On the tentative agenda:</p> + +<ol><li>Recent release overview @Michael Robinson </li><li>Column lineage update @Maciej Obuchowski </li><li>Airflow integration improvements @Jakub Dardziński </li><li>Discussions: +• Real-world implementation of OpenLineage (What does it really mean?) @Sheeri Cabral (Collibra) +• Using namespaces @Michael Robinson </li><li>Open discussion +Notes: <a href="https://bit.ly/OLwiki">https://bit.ly/OLwiki</a> +Is there a topic you think the community should discuss at this or a future meeting? Reply or DM me to add items to the agenda.</li> +</ol> +
+ + + + + + +
Original URL: https://openlineage.slack.com/archives/C01CK9T7HKR/p1672933029317449
+ + +
+ + Thread in Slack Conversation +
+ + +
+ + + +
+ 🙌 Maciej Obuchowski, Will Johnson, John Bagnall, AnnMary Justine, Willy Lulciuc, Minkyu Park, Paweł Leszczyński, Varun Singh +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Varun Singh + (varuntestaz@outlook.com) +
+
2023-01-12 06:37:56
 +
+

Hi, are there any plans to add an Azure EventHub transport similar to the Kinesis one?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2023-01-12 17:31:12
 +
+

*Thread Reply:* @Varun Singh why not just use the KafkaTransport and the Event Hub's Kafka endpoint?

+ +

https://github.com/yogyang/OpenLineage/blob/2b7fa8bbd19a2207d54756e79aea7a542bf7bb[…]/main/java/io/openlineage/client/transports/KafkaTransport.java

+ +

https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-kafka-stream-analytics

+
+
learn.microsoft.com
+ + + +
Azure Event Hubs - Process Apache Kafka events - Azure Event Hubs
+
+ Tutorial: This article shows how to process Kafka events that are ingested through event hubs by using Azure Stream Analytics +
+ + + + + + + + +
Original URL: https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-kafka-stream-analytics
+ + + +
+
+ + + + +
<https://github.com/yogyang/OpenLineage/blob/2b7fa8bbd19a2207d54756e79aea7a542bf7bb4c/client/java/src/main/java/io/openlineage/client/transports/KafkaTransport.java | KafkaTransport.java>
+
+ <pre><code>/** +/** Copyright 2018-2022 contributors to the OpenLineage project +/** SPDX-License-Identifier: Apache-2.0 +**/ + +package io.openlineage.client.transports; + +import io.openlineage.client.OpenLineage; +import io.openlineage.client.OpenLineageClientUtils; +import lombok.NonNull; +import lombok.extern.slf4j.Slf4j; +import org.apache.kafka.clients.producer.KafkaProducer; +import org.apache.kafka.clients.producer.ProducerRecord; + +@Slf4j +public final class KafkaTransport extends Transport { + private final String topicName; + private final String localServerId; + private final KafkaProducer&lt;String, String&gt; producer; + + public KafkaTransport(@NonNull final KafkaConfig kafkaConfig) { + this(new KafkaProducer&lt;&gt;(kafkaConfig.getProperties()), kafkaConfig); + } + + public KafkaTransport( + @NonNull final KafkaProducer&lt;String, String&gt; kafkaProducer, + @NonNull final KafkaConfig kafkaConfig) { + super(Type.KAFKA); + this.topicName = kafkaConfig.getTopicName(); + this.localServerId = kafkaConfig.getLocalServerId(); + this.producer = kafkaProducer; + } + + @Override + public void emit(@NonNull OpenLineage.RunEvent runEvent) { + final String eventAsJson = OpenLineageClientUtils.toJson(runEvent); + log.debug("Received lineage event: {}", eventAsJson); + final ProducerRecord&lt;String, String&gt; record = + new ProducerRecord&lt;&gt;(topicName, localServerId, eventAsJson); + try { + producer.send(record); + } catch (Exception e) { + log.error("Failed to collect lineage event: {}", eventAsJson, e); + } + } +} + +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/yogyang/OpenLineage">yogyang/OpenLineage</a> +
+ + +
+ + + +
+ 👍 Varun Singh +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2023-01-12 09:01:24
 +
+

Following up on last month’s discussion (), I created the <#C04JPTTC876|spec-compliance> channel for further discussion

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2023-01-12 17:43:55
 +
+

*Thread Reply:* @Julien Le Dem is there a channel to discuss the community call / ask follow-up questions on the communiyt call topics? For example, I wanted to ask more about the AirflowFacet and if we expected to introduce more tool specific facets into the spec. Where's the right place to ask that question? On the PR?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2023-01-17 15:11:05
 +
+

*Thread Reply:* I think asking in #general is the right place. If there’s a specific github issue/PR, his is a good place as well. You can tag the relevant folks as well to get their attention

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Allison Suarez + (asuarezmiranda@lyft.com) +
+
2023-01-12 18:37:24
 +
+

@here I am using the Spark listener and whenever a query like INSERT OVERWRITE TABLE gets executed it looks like I can see some outputs, but there are no symlinks for the output table. The operation type being executed is InsertIntoHadoopFsRelationCommand . I am not sure why I cna see symlinks for all the input tables but not the output tables. Anyone know the reason behind this?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-01-13 02:30:37
 +
+

*Thread Reply:* Hello @Allison Suarez, in case of InsertIntoHadoopFsRelationCommand, Spark Openlineage implementation uses method: +DatasetIdentifier di = PathUtils.fromURI(command.outputPath().toUri(), "file"); +(https://github.com/OpenLineage/OpenLineage/blob/0.19.2/integration/spark/shared/sr[…]ark/agent/lifecycle/plan/InsertIntoHadoopFsRelationVisitor.java)

+ +

If the dataset identifier is constructed from a path, then no symlinks are added. That's the current behaviour.

+ +

Calling io.openlineage.spark.agent.util.DatasetIdentifier#withSymlink(io.openlineage.spark.agent.util.DatasetIdentifier.Symlink) on DatasretIdentifier in InsertIntoHadoopFsRelationVisitor +could be a remedy to that.

+ +

Do you have some Spark code snippet to reproduce this issue?

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/0.19.2/integration/spark/shared/src/main/java/io/openlineage/spark/agent/lifecycle/plan/InsertIntoHadoopFsRelationVisitor.java | InsertIntoHadoopFsRelationVisitor.java>
+
+ <pre><code> DatasetIdentifier di = PathUtils.fromURI(command.outputPath().toUri(), "file"); +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2023-01-22 10:04:56
 +
+

*Thread Reply:* @Allison Suarez it would also be good to know what compute engine you're using to run your code on? On-Prem Apache Spark? Azure/AWS/GCP Databricks?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Allison Suarez + (asuarezmiranda@lyft.com) +
+
2023-02-13 18:18:52
 +
+

*Thread Reply:* I created a custom visitor and fixed the issue that way, thank you!

+ + + +
+ 🙌 Will Johnson +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Varun Singh + (varuntestaz@outlook.com) +
+
2023-01-13 11:44:19
 +
+

Hi, I am trying to use kafka transport in spark for sending events to an EventHub but it requires me to set a property sasl.jaas.config which needs to have semicolons (;) in its value. But this gives an error about being unable to convert Array to a String. I think this is due to this line which splits property values into an array if they have a semicolon https://github.com/OpenLineage/OpenLineage/blob/92adbc877f0f4008928a420a1b8a93f394[…]pp/src/main/java/io/openlineage/spark/agent/ArgumentParser.java +Does this seem like a bug or is it intentional?

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/92adbc877f0f4008928a420a1b8a93f394530d04/integration/spark/app/src/main/java/io/openlineage/spark/agent/ArgumentParser.java | ArgumentParser.java>
+
+ <pre><code> Arrays.stream(value.split(DISABLED_FACETS_SEPARATOR)) +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harel Shein + (harel.shein@gmail.com) +
+
2023-01-13 14:39:51
 +
+

*Thread Reply:* seems like a bug to me, but tagging @Tomasz Nazarewicz / @Paweł Leszczyński

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tomasz Nazarewicz + (tomasz.nazarewicz@getindata.com) +
+
2023-01-13 15:22:19
 +
+

*Thread Reply:* So we needed a generic way of passing parameters to client and made an assumption that every field with ; will be treated as an array

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Varun Singh + (varuntestaz@outlook.com) +
+
2023-01-14 02:00:04
 +
+

*Thread Reply:* Thanks for the confirmation, should I add a condition to split only if it's a key that can have array values? We can have a list of such keys like facets.disabled

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tomasz Nazarewicz + (tomasz.nazarewicz@getindata.com) +
+
2023-01-14 02:28:41
 +
+

*Thread Reply:* We thought about this solution but it forces us to know the structure of each config and we wanted to avoid that as much as possible

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tomasz Nazarewicz + (tomasz.nazarewicz@getindata.com) +
+
2023-01-14 02:34:06
 +
+

*Thread Reply:* Maybe the condition could be having ; and [] in the value

+ + + +
+ 👍 Varun Singh +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Varun Singh + (varuntestaz@outlook.com) +
+
2023-01-15 08:14:14
 +
+

*Thread Reply:* Makes sense, I can add this check. Thanks @Tomasz Nazarewicz!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Varun Singh + (varuntestaz@outlook.com) +
+
2023-01-16 01:15:19
 +
+

*Thread Reply:* Created issue https://github.com/OpenLineage/OpenLineage/issues/1506 for this

+
+ + + + +
#1506 Spark configs with semicolons (;) in their value get split into an array
+
+ <p>Semicolon (;) is being used as the delimiter for array configs like <code>disabled.facets</code> but this also converts string configs which require a semicolon into arrays. For example, EventHub authentication requires a SAS key which has a semicolon in it but it doesn't need to be converted into an array.</p> + +<p>This conversion is being done in the following lines </p> + +<p><a href="https://github.com/OpenLineage/OpenLineage/blob/92adbc877f0f4008928a420a1b8a93f394530d04/integration/spark/app/src/main/java/io/openlineage/spark/agent/ArgumentParser.java#L150-L154">OpenLineage/integration/spark/app/src/main/java/io/openlineage/spark/agent/ArgumentParser.java</a></p> + +<p>Lines 150 to 154 in </OpenLineage/OpenLineage/commit/92adbc877f0f4008928a420a1b8a93f394530d04|92adbc8></p> +
+ +
+
Comments
+ 2 +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-01-17 12:00:02
 +
+

Hi everyone, I’m excited to share some good news about our progress in the LFAI & Data Foundation: we’ve achieved Incubation status! This required us to earn a Silver Badge from the OpenSSF, get 300+ stars on GitHub (which was NBD as we have over 1100 already), and win the approval of the LFAI & Data’s TAC. Now that we’ve cleared this hurdle, we have access to additional services from the foundation, including assistance with creative work, marketing and communication support, and event-planning assistance. Graduation from the program, which will earn us a voting seat on the TAC, is on the horizon. Stay tuned for updates on our progress with the foundation.

+ +

LF AI & Data is an umbrella foundation of the Linux Foundation that supports open source innovation in artificial intelligence (AI) and data. LF AI & Data was created to support open source AI and data, and to create a sustainable open source AI and data ecosystem that makes it easy to create AI and data products and services using open source technologies. They foster collaboration under a neutral environment with an open governance in support of the harmonization and acceleration of open source technical projects.

+ +

For more info about the foundation and other LFAI & Data projects, visit their website.

+ + + +
+ ❤️ Julien Le Dem, Paweł Leszczyński, Maciej Obuchowski, Ross Turk, Jakub Dardziński, Minkyu Park, Howard Yoo, Jarek Potiuk, Danilo Mota, Willy Lulciuc, Kengo Seki, Harel Shein +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2023-01-17 15:53:12
 +
+

if you want to share this news (and I hope you do!) there is a blog post here: https://openlineage.io/blog/incubation-stage-lfai/

+
+
openlineage.io
+ + + +
OpenLineage Advances to Incubation Stage with the LFAI &amp; Data
+
+ OpenLineage has achieved Incubation status with the LFAI &amp; Data. +
+ + + + + + +
Original URL: https://openlineage.io/blog/incubation-stage-lfai/
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2023-01-17 15:54:07
 +
+

and I'll add a quick shoutout of @Michael Robinson, who has done a whole lot of work to make this happen 🎉 thanks, man, you're awesome!

+ + + +
+ 🙌 Howard Yoo, Maciej Obuchowski, Jarek Potiuk, Minkyu Park, Willy Lulciuc, Kengo Seki, Paweł Leszczyński, Varun Singh +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-01-17 15:56:38
 +
+

*Thread Reply:* Thank you, Ross!! I appreciate it. I might have coordinated it, but it’s been a team effort. Lots of folks shared knowledge and time to help us check all the boxes, literally and figuratively (lots of boxes). ;)

+ + + +
+ ☑️ Willy Lulciuc, Paweł Leszczyński, Viraj Parekh +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jarek Potiuk + (jarek@potiuk.com) +
+
2023-01-17 16:03:36
 +
+

Congrats @Michael Robinson and @Ross Turk - > major step for Open Lineage!

+ + + +
+ 🙌 Michael Robinson, Maciej Obuchowski, Jakub Dardziński, Julien Le Dem, Ross Turk, Willy Lulciuc, Kengo Seki, Viraj Parekh, Paweł Leszczyński, Anirudh Shrinivason, Robert +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sudhir Nune + (sudhir.nune@kraftheinz.com) +
+
2023-01-18 11:15:02
 +
+

Hi all, I am new to the https://openlineage.io/integration/dbt/, I followed the steps on Windows Laptop. But the dbt-ol does not get executed.

+ +

'dbt-ol' is not recognized as an internal or external command, +operable program or batch file.

+ +

I see the following Packages installed too +openlineage-dbt==0.19.2 +openlineage-integration-common==0.19.2 +openlineage-python==0.19.2

+
+
openlineage.io
+ + + +
dbt
+
+ Enabling OpenLineage in dbt can capture lineage metadata for transformations running within your data warehouse. +
+ + + + + + +
Original URL: https://openlineage.io/integration/dbt/
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-01-18 11:17:14
 +
+

*Thread Reply:* What are the errors?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sudhir Nune + (sudhir.nune@kraftheinz.com) +
+
2023-01-18 11:18:09
 +
+

*Thread Reply:* 'dbt-ol' is not recognized as an internal or external command, +operable program or batch file.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-01-19 11:11:09
 +
+

*Thread Reply:* Hm, I think this is due to different windows conventions around scripts.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2023-01-19 14:26:35
 +
+

*Thread Reply:* I have not tried it on Windows before myself, but on mac/linux if you make a Python virtual environment in venv/ and run pip install openlineage-dbt, the script winds up in ./venv/bin/dbt-ol.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2023-01-19 14:27:04
 +
+

*Thread Reply:* (maybe that helps!)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-01-19 14:38:23
 +
+

*Thread Reply:* This might not work, but I think I have an idea that would allow it to run as python -m dbt-ol run ...

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-01-19 14:38:27
 +
+

*Thread Reply:* That needs one fix though

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sudhir Nune + (sudhir.nune@kraftheinz.com) +
+
2023-01-19 14:40:52
 +
+

*Thread Reply:* Hi @Maciej Obuchowski, thanks for the input, when I try to use python -m dbt-ol run, I see the below error :( +\python.exe: No module named dbt-ol

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-01-24 13:23:56
 +
+

*Thread Reply:* We’re seeing a similar issue with the Great Expectations integration at the moment. This is purely a guess, but what happens when you try with openlineage-dbt 0.18.0?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-01-24 13:24:36
 +
+

*Thread Reply:* @Michael Robinson GE issue is on Windows?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-01-24 13:24:49
 +
+

*Thread Reply:* No, not Windows

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-01-24 13:24:55
 +
+

*Thread Reply:* (that I know of)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sudhir Nune + (sudhir.nune@kraftheinz.com) +
+
2023-01-24 13:46:39
 +
+

*Thread Reply:* @Michael Robinson - I see the same error. I used 2 Combinations

+ +
  1. Python 3.8.10 with openlineage-dbt 0.18.0 & Latest
  2. Python 3.9.7 with openlineage-dbt 0.18.0 & Latest
    3. +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2023-01-24 13:49:19
 +
+

*Thread Reply:* Hm. You should be able to find the dbt-ol command wherever pip is installing the packages. In my case, that's usually in a virtual environment.

+ +

But if I am not in a virtual environment, it installs the packages in my PYTHONPATH. You might try this to see if the dbt-ol script can be found in one of the directories in sys.path.

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2023-01-24 13:58:38
 +
+

*Thread Reply:* this can help you verify that your PYTHONPATH and PATH are correct - installing an unrelated python command-line tool and seeing if you can execute it:

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-01-24 13:59:42
 +
+

*Thread Reply:* Again, I think this is windows issue

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2023-01-24 14:00:54
 +
+

*Thread Reply:* @Maciej Obuchowski you think even if dbt-ol could be found in the path, that might not be the issue?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sudhir Nune + (sudhir.nune@kraftheinz.com) +
+
2023-01-24 14:15:13
 +
+

*Thread Reply:* Hi @Ross Turk - I could not find the dbt-ol in the site-packages.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2023-01-24 14:16:48
 +
+

*Thread Reply:* Hm 😕 then perhaps @Maciej Obuchowski is right and there is a bigger issue here

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sudhir Nune + (sudhir.nune@kraftheinz.com) +
+
2023-01-24 14:31:15
 +
+

*Thread Reply:* @Ross Turk & @Maciej Obuchowski I see the issue event when I do the install using the https://pypi.org/project/openlineage-dbt/#files - openlineage-dbt-0.19.2.tar.gz.

+ +

For some reason, I see only the following folder created

+ +
  1. openlineage
  2. openlineage_dbt-0.19.2.dist-info
  3. openlineageintegrationcommon-0.19.2.dist-info
  4. openlineage_python-0.19.2.dist-info +and not brining in the openlineage-dbt-0.19.2, which has the scripts/dbt-ol
    5. +
+ +

If it helps I am using pip 21.2.4

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Francis McGregor-Macdonald + (francis@mc-mac.com) +
+
2023-01-18 18:40:32
 +
+

@Paul Villena @Stephen Said and Vishwanatha Nayak published an AWS blog Automate data lineage on Amazon MWAA with OpenLineage

+
+
Amazon Web Services
+ + + +
Automate data lineage on Amazon MWAA with OpenLineage | Amazon Web Services
+
+ In modern data architectures, datasets are combined across an organization using a variety of purpose-built services to unlock insights. As a result, data governance becomes a key component for data consumers and producers to know that their data-driven decisions are based on trusted and accurate datasets. One aspect of data governance is data lineage, which […] +
+ + + + + + + + +
Original URL: https://aws.amazon.com/blogs/big-data/automate-data-lineage-on-amazon-mwaa-with-openlineage/
+ + + +
+ + + +
+ 👀 Ross Turk, Peter Hicks, Willy Lulciuc +
+ +
+ 🔥 Ross Turk, Willy Lulciuc, Michael Collado, Peter Hicks, Minkyu Park, Julien Le Dem, Kengo Seki, Anirudh Shrinivason, Paweł Leszczyński, Maciej Obuchowski, Harel Shein, Paul Wilson Villena +
+ +
+ ❤️ Willy Lulciuc, Minkyu Park, Julien Le Dem, Kengo Seki, Paweł Leszczyński, Viraj Parekh +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2023-01-18 18:54:57
 +
+

*Thread Reply:* This is excellent! May we promote it on openlineage and marquez social channels?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2023-01-18 18:55:30
 +
+

*Thread Reply:* This is an amazing write up! 🔥 💯 🚀

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Francis McGregor-Macdonald + (francis@mc-mac.com) +
+
2023-01-18 19:49:46
 +
+

*Thread Reply:* Happy to have it promoted. 😄 +Vish posted on LinkedIn: https://www.linkedin.com/posts/vishwanatha-nayak-b8462054automate-data-lineage-on-amazon-mwaa-with-activity-7021589819763945473-yMHF?utmsource=share&utmmedium=memberios|https://www.linkedin.com/posts/vishwanatha-nayak-b8462054automate-data-lineage-on-amazon-mwaa-with-activity-7021589819763945473-yMHF?utmsource=share&utmmedium=memberios if you want something to repost there.

+
+
linkedin.com
+ + + +
Vishwanatha Nayak on LinkedIn: Automate data lineage on Amazon MWAA with OpenLineage | Amazon Web Services
+
+ We (Stephen Said, Paul Villena and I) are happy to start the year with a new blog on automating data lineage using Amazon Managed Workflows for Apache Airflow… +
+ + + + + + + + +
Original URL: https://www.linkedin.com/posts/vishwanatha-nayak-b8462054_automate-data-lineage-on-amazon-mwaa-with-activity-7021589819763945473-yMHF?utm_source=share&amp;utm_medium=member_ios
+ + + +
+ + + +
+ ❤️ Willy Lulciuc, Ross Turk +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-01-19 00:13:26
 +
+

Hi guys, I am trying to build the openlineage jar locally for spark. I ran ./gradlew shadowJar in the /integration/spark directory. However, I am getting this issue: +** What went wrong: +A problem occurred evaluating project ':app'. +&gt; Could not resolve all files for configuration ':app:spark33'. + &gt; Could not resolve io.openlineage:openlineage_java:0.20.0-SNAPSHOT. + Required by: + project :app &gt; project :shared + &gt; Could not resolve io.openlineage:openlineage_java:0.20.0-SNAPSHOT. + &gt; Unable to load Maven meta-data from <https://astronomer.jfrog.io/artifactory/maven-public-libs-snapshot/io/openlineage/openlineage-java/0.20.0-SNAPSHOT/maven-metadata.xml>. + &gt; Could not GET '<https://astronomer.jfrog.io/artifactory/maven-public-libs-snapshot/io/openlineage/openlineage-java/0.20.0-SNAPSHOT/maven-metadata.xml>'. Received status code 401 from server: Unauthorized +It used to work a few weeks ago...May I ask if anyone would know what the reason might be? Thanks! 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-01-19 03:58:42
 +
+

*Thread Reply:* Hello @Anirudh Shrinivason, you need to build your openlineage-java package first. Possibly you built in some time ao in different version

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-01-19 03:59:28
 +
+

*Thread Reply:* ./gradlew clean build publishToMavenLocal +in /client/java should help.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-01-19 04:34:33
 +
+

*Thread Reply:* Ahh yeap this works thanks! 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sheeri Cabral (Collibra) + (sheeri.cabral@collibra.com) +
+
2023-01-19 09:17:01
 +
+

Are there any resources to explain the differences between lineage with Apache Atlas vs. lineage using OpenLineage? we have discussions with customers and partners, and some of them are looking into which is more “ready for industry”.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-01-19 11:03:39
 +
+

*Thread Reply:* It's been a while since I looked at Atlas, but does it even now supports something else than very Java Apache-adjacent projects like Hive and HBase?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2023-01-19 13:10:11
 +
+

*Thread Reply:* To directly answer your question @Sheeri Cabral (Collibra): I am not aware of any resources currently that explain this 😞 but I would welcome the creation of one & pitch in where possible!

+ + + +
+ ✅ Sheeri Cabral (Collibra) +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sheeri Cabral (Collibra) + (sheeri.cabral@collibra.com) +
+
2023-01-20 17:00:25
 +
+

*Thread Reply:* I don’t know enough about Atlas to make that doc.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Justine Boulant + (justine.boulant@seenovate.com) +
+
2023-01-19 10:43:18
 +
+

Hi everyone, I am currently working on a project and we have some questions to use OpenLineage with Apache Airflow : +• How does it work : ux vs code/script? How can we implement it? a schema of its architecture for example +• What are the visual outputs available? +• Is the lineage done from A to Z? if there are multiple intermediary transformations for example? +• Is the lineage done horizontally across the organization or vertically on different system levels? or both? +• Can we upgrade it to industry-level? +• Does it work with Python and/or R? +• Does it read metadata or scripts? +Thanks a lot if you can help 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-01-19 11:00:54
 +
+

*Thread Reply:* I think most of your questions will be answered by this video: https://www.youtube.com/watch?v=LRr-ja8_Wjs

+
+
YouTube
+ +
+ + + } + + Astronomer + (https://www.youtube.com/@Astronomer) +
+ + + +
OpenLineage and Airflow: A Deeper Dive
+
+ +
+ + + + + + + + +
Original URL: https://www.youtube.com/watch?v=LRr-ja8_Wjs
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2023-01-19 13:10:58
 +
+

*Thread Reply:* I agree - a lot of the answers are in that overview video. You might also take a look at the docs, they do a pretty good job of explaining how it works.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2023-01-19 13:19:34
 +
+

*Thread Reply:* More explicitly: +• Airflow is an interesting platform to observe because it runs a large variety of workloads and lineage can only be automatically extracted for some of them +• In general, OpenLineage is essentially a standard and data model for lineage. There are integrations for various systems, including Airflow, that cause them to emit lineage events to an OpenLineage compatible backend. It's a push model. +• Marquez is one such backend, and the one I recommend for testing & development +• There are a few approaches for lineage in Airflow: + ◦ Extractors, which pair with Operators to extract and emit lineage + ◦ Manual inlets/outlets on a task, defined by a developer - useful for PythonOperator and other cases where an extractor can't do it auto + ◦ Orchestration of an underlying OpenLineage integration, like openlineage-dbt +• IDK about "A to Z", that depends on your environment. The goal is to capture every transformation. Depending on your pipeline, there may be a set of integrations that give you the coverage you need. We often find that there are gaps. +• It works with Python. You can use the openlineage-python client to emit lineage events to a backend. This is useful if there isn't an integration for something your pipeline does. +• It describes the pipeline by observing running jobs and the way they affect datasets, not the organization. I don't know what you mean by "industry-level". +• I am not aware of an integration that parses source code to determine lineage at this time. +• The openlineage-dbt integration consumes the various metadata that dbt leaves behind to construct lineage. Dunno if that's what you mean by "read metadata".

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2023-01-19 13:23:33
 +
+

*Thread Reply:* FWIW I did a workshop on openlineage and airflow a while back, and it's all in this repo. You can find slides + a quick Python example + a simple Airflow example in there.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Justine Boulant + (justine.boulant@seenovate.com) +
+
2023-01-20 03:44:22
 +
+

*Thread Reply:* Thanks a lot!! Very helpful!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2023-01-20 11:42:43
 +
+

*Thread Reply:* 👍

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Brad Paskewitz + (bradford.paskewitz@fivetran.com) +
+
2023-01-20 15:28:06
 +
+

Hey folks, my team is working on a solution that would support the OL standard with column level lineage. I'm working through the architecture now and I'm wondering if everyone uses the standard rest api backed by a db or if other teams found success using other technologies such as webhooks, streams, etc in order to capture and process lineage events. I'd be very curious to connect on the topic

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2023-01-20 19:45:55
 +
+

*Thread Reply:* Hello Brad, on top of my head:

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2023-01-20 19:47:15
 +
+

*Thread Reply:* • Marquez uses the API HTTP Post. so does Astro +• Egeria and Purview prefer consuming through a Kafka topic. There is a ProxyBackend that takes HTTP Posts and writes to Kafka. The client can also be configured to write to Kafka

+ + + +
+ 👍 Jakub Dardziński +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2023-01-20 19:48:09
 +
+

*Thread Reply:* @Will Johnson @Mandy Chessell might have opinions

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2023-01-20 19:49:10
 +
+

*Thread Reply:* The Microsoft Purview approach is documented here: https://learn.microsoft.com/en-us/samples/microsoft/purview-adb-lineage-solution-accelerator/azure-databricks-to-purview-lineage-connector/

+
+
learn.microsoft.com
+ + + +
Azure Databricks to Purview Lineage Connector - Code Samples
+
+ This solution accelerator, together with the [OpenLineage](<http://openlineage.io>) project, provides a connector that will transfer lineage metadata fr... +
+ + + + + + + + +
Original URL: https://learn.microsoft.com/en-us/samples/microsoft/purview-adb-lineage-solution-accelerator/azure-databricks-to-purview-lineage-connector/
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2023-01-20 19:49:47
 +
+

*Thread Reply:* There’s a blog post about Egeria here: https://openlineage.io/blog/openlineage-egeria/

+
+
openlineage.io
+ + + +
OpenLineage Support in Egeria
+
+ The Egeria project uses OpenLineage to enhance its production of holistic metadata about an organization's operations. +
+ + + + + + +
Original URL: https://openlineage.io/blog/openlineage-egeria/
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2023-01-22 10:00:56
 +
+

*Thread Reply:* @Brad Paskewitz at Microsoft, the solution that Julien linked above, we are using the HTTP Transport (REST API) as we are consuming the OpenLineage Events and transforming them to Apache Atlas / Microsoft Purview.

+ +

However, there is a good deal of interest in using the kafka transport instead and that's our future roadmap.

+ + + +
+ 👍 Ross Turk, Brad Paskewitz +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Quentin Nambot + (qnambot@gmail.com) +
+
2023-01-25 09:59:13
 +
+

❓ Hi everyone, I am trying to use openlineage with Databricks (using 11.3 LTS runtime, and openlineage 0.19.2) +Using this documentation I managed to install openlineage and send events to marquez +However marquez did not received all COMPLETE events, it seems like databricks cluster is shutdown immediatly at the end of the job. It is not the first time that i see this with databricks, last year I tried to use spline and we noticed that Databricks seems to not wait that spark session is nicely closed before shutting down instances (see this issue) +My question is: has anyone faced the same issue? Does somebody know a workaround? 🙏

+
+
spline
+ + + +
0.7
+
+ Data Lineage Tracking And Visualization Solution +
+ + + + + + +
Original URL: https://absaoss.github.io/spline/
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2023-01-25 12:04:48
 +
+

*Thread Reply:* Hmm, if Databricks is shutting the process down without waiting for the ListenerBus to clear, I don’t know that there’s a lot we can do. The best thing is to somehow delay the main application thread from exiting. One thing you could try is to subclass the OpenLineageSparkListener and generate a lock for each SparkListenerSQLExecutionStart and release it when the accompanying SparkListenerSQLExecutionEnd event is processed. Then, in the main application, block until all such locks are released. If you try it and it works, let us know!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Quentin Nambot + (qnambot@gmail.com) +
+
2023-01-26 05:46:35
 +
+

*Thread Reply:* Ok thanks for the idea! I'll tell you if I try this and if it works 🤞

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Petr Hajek + (petr.hajek@profinit.eu) +
+
2023-01-25 10:12:42
 +
+

Hi, would anybody be able and willing to help us configure S3 and Snowflake extractors within Airflow integration for one of our clients? Our trouble is that Airflow integration returns valid OpenLineage .json files but it lacks any information about input and output DataSets. Thanks in advance 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-01-25 10:38:03
 +
+

*Thread Reply:* Hey Petr. Please DM me or describe the issue here 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Susmitha Anandarao + (susmitha.anandarao@gmail.com) +
+
2023-01-27 15:24:47
 +
+

Hello.. I am trying to play with openlineage spark integration with Kafka and currently trying to just use the config as part of the spark submit command but I run into errors. Details in the 🧵

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Susmitha Anandarao + (susmitha.anandarao@gmail.com) +
+
2023-01-27 15:25:04
 +
+

*Thread Reply:* Command +spark-submit --packages "io.openlineage:openlineage_spark:0.19.+" \ + --conf "spark.extraListeners=io.openlineage.spark.agent.OpenLineageSparkListener" \ + --conf "spark.openlineage.transport.type=kafka" \ + --conf "spark.openlineage.transport.topicName=topicname" \ + --conf "spark.openlineage.transport.localServerId=Kafka_server" \ + file.py

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Susmitha Anandarao + (susmitha.anandarao@gmail.com) +
+
2023-01-27 15:25:14
 +
+

*Thread Reply:* 23/01/27 17:29:06 ERROR AsyncEventQueue: Listener OpenLineageSparkListener threw an exception +java.lang.NullPointerException + at io.openlineage.client.transports.TransportFactory.build(TransportFactory.java:44) + at io.openlineage.spark.agent.EventEmitter.&lt;init&gt;(EventEmitter.java:40) + at io.openlineage.spark.agent.OpenLineageSparkListener.initializeContextFactoryIfNotInitialized(OpenLineageSparkListener.java:278) + at io.openlineage.spark.agent.OpenLineageSparkListener.onApplicationStart(OpenLineageSparkListener.java:267) + at org.apache.spark.scheduler.SparkListenerBus.doPostEvent(SparkListenerBus.scala:55) + at org.apache.spark.scheduler.SparkListenerBus.doPostEvent$(SparkListenerBus.scala:28) + at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37) + at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37) + at org.apache.spark.util.ListenerBus.postToAll(ListenerBus.scala:117) + at org.apache.spark.util.ListenerBus.postToAll$(ListenerBus.scala:101) + at org.apache.spark.scheduler.AsyncEventQueue.super$postToAll(AsyncEventQueue.scala:105) + at org.apache.spark.scheduler.AsyncEventQueue.$anonfun$dispatch$1(AsyncEventQueue.scala:105) + at scala.runtime.java8.JFunction0$mcJ$sp.apply(JFunction0$mcJ$sp.java:23) + at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62) + at <a href="http://org.apache.spark.scheduler.AsyncEventQueue.org">org.apache.spark.scheduler.AsyncEventQueue.org</a>$apache$spark$scheduler$AsyncEventQueue$$dispatch(AsyncEventQueue.scala:100) + at org.apache.spark.scheduler.AsyncEventQueue$$anon$2.$anonfun$run$1(AsyncEventQueue.scala:96) + at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1446) + at org.apache.spark.scheduler.AsyncEventQueue$$anon$2.run(AsyncEventQueue.scala:96)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Susmitha Anandarao + (susmitha.anandarao@gmail.com) +
+
2023-01-27 15:25:31
 +
+

*Thread Reply:* I would appreciate any pointers on getting started with using openlineage-spark with Kafka.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Susmitha Anandarao + (susmitha.anandarao@gmail.com) +
+
2023-01-27 16:15:00
 +
+

*Thread Reply:* Also this might seem a little elementary but the kafka topic itself, should it be hosted on the spark cluster or could it be any kafka topic?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Susmitha Anandarao + (susmitha.anandarao@gmail.com) +
+
2023-01-30 08:37:07
 +
+

*Thread Reply:* 👀 Could I get some help on this, please?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-01-30 09:07:08
 +
+

*Thread Reply:* I think any NullPointerException is clearly our bug, can you open issue on OL GitHub?

+ + + +
+ 👍 Susmitha Anandarao +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Susmitha Anandarao + (susmitha.anandarao@gmail.com) +
+
2023-01-30 09:30:51
 +
+

*Thread Reply:* @Maciej Obuchowski Another interesting thing is if I use 0.19.2 version specifically, I get +23/01/30 14:28:33 INFO RddExecutionContext: RDDs are empty: skipping sending OpenLineage event

+ +

I am trying to print to console at the moment. I haven't been able to get Kafka transport type working though.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-01-30 09:41:12
 +
+

*Thread Reply:* Are you getting events printed on the console though? This log should not affect you if you're running, for example Spark SQL jobs

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Susmitha Anandarao + (susmitha.anandarao@gmail.com) +
+
2023-01-30 09:42:28
 +
+

*Thread Reply:* I am trying to run a python file using pyspark. 23/01/30 14:40:49 INFO RddExecutionContext: RDDs are empty: skipping sending OpenLineage event +I see this and don't see any events on the console.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-01-30 09:55:41
 +
+

*Thread Reply:* Any logs filling pattern +log.warn("Unable to access job conf from RDD", nfe); +or +<a href="http://log.info">log.info</a>("Found job conf from RDD {}", jc); +before?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Susmitha Anandarao + (susmitha.anandarao@gmail.com) +
+
2023-01-30 09:57:20
 +
+

*Thread Reply:* ```23/01/30 14:40:48 INFO DAGScheduler: Submitting ShuffleMapStage 0 (PairwiseRDD[2] at reduceByKey at /tmp/spark-20487725-f49b-4587-986d-e63a61890673/statusapidemo.py:47), which has no missing parents +23/01/30 14:40:49 WARN RddExecutionContext: Unable to access job conf from RDD +java.lang.NoSuchFieldException: Field is not instance of HadoopMapRedWriteConfigUtil + at io.openlineage.spark.agent.lifecycle.RddExecutionContext.lambda$setActiveJob$0(RddExecutionContext.java:117) + at java.util.Optional.orElseThrow(Optional.java:290) + at io.openlineage.spark.agent.lifecycle.RddExecutionContext.setActiveJob(RddExecutionContext.java:115) + at java.util.Optional.ifPresent(Optional.java:159) + at io.openlineage.spark.agent.OpenLineageSparkListener.lambda$onJobStart$9(OpenLineageSparkListener.java:148) + at java.util.Optional.ifPresent(Optional.java:159) + at io.openlineage.spark.agent.OpenLineageSparkListener.onJobStart(OpenLineageSparkListener.java:145) + at org.apache.spark.scheduler.SparkListenerBus.doPostEvent(SparkListenerBus.scala:37) + at org.apache.spark.scheduler.SparkListenerBus.doPostEvent$(SparkListenerBus.scala:28) + at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37) + at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37) + at org.apache.spark.util.ListenerBus.postToAll(ListenerBus.scala:117) + at org.apache.spark.util.ListenerBus.postToAll$(ListenerBus.scala:101) + at org.apache.spark.scheduler.AsyncEventQueue.super$postToAll(AsyncEventQueue.scala:105) + at org.apache.spark.scheduler.AsyncEventQueue.$anonfun$dispatch$1(AsyncEventQueue.scala:105) + at scala.runtime.java8.JFunction0$mcJ$sp.apply(JFunction0$mcJ$sp.java:23) + at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62) + at org.apache.spark.scheduler.AsyncEventQueue.org$apache$spark$scheduler$AsyncEventQueue$$dispatch(AsyncEventQueue.scala:100) + at org.apache.spark.scheduler.AsyncEventQueue$$anon$2.$anonfun$run$1(AsyncEventQueue.scala:96) + at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1446) + at org.apache.spark.scheduler.AsyncEventQueue$$anon$2.run(AsyncEventQueue.scala:96)

+ +

23/01/30 14:40:49 INFO RddExecutionContext: Found job conf from RDD Configuration: core-default.xml, core-site.xml, mapred-default.xml, mapred-site.xml, yarn-default.xml, yarn-site.xml, hdfs-default.xml, hdfs-rbf-default.xml, hdfs-site.xml, hdfs-rbf-site.xml, resource-types.xml

+ +

23/01/30 14:40:49 INFO RddExecutionContext: Found output path null from RDD PythonRDD[5] at collect at /tmp/spark-20487725-f49b-4587-986d-e63a61890673/statusapidemo.py:48 +23/01/30 14:40:49 INFO RddExecutionContext: RDDs are empty: skipping sending OpenLineage event``` +I see both actually.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-01-30 10:03:35
 +
+

*Thread Reply:* I think this is same problem as this: https://github.com/OpenLineage/OpenLineage/issues/1521

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-01-30 10:04:14
 +
+

*Thread Reply:* and I think I might have solution on a branch for it, just need to polish it up to release

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Susmitha Anandarao + (susmitha.anandarao@gmail.com) +
+
2023-01-30 10:13:37
 +
+

*Thread Reply:* Aah got it. I will give it a try with SQL and a jar.

+ +

Do you have a ETA on when the python issue would be fixed?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Susmitha Anandarao + (susmitha.anandarao@gmail.com) +
+
2023-01-30 10:37:51
 +
+

*Thread Reply:* @Maciej Obuchowski Well I run into the same errors if I run spark-submit on a jar.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-01-30 10:38:44
 +
+

*Thread Reply:* I think that has nothing to do with python

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-01-30 10:39:16
 +
+

*Thread Reply:* BTW, which Spark version are you using?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Susmitha Anandarao + (susmitha.anandarao@gmail.com) +
+
2023-01-30 10:41:22
 +
+

*Thread Reply:* We are on 3.3.1

+ + + +
+ 👍 Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Susmitha Anandarao + (susmitha.anandarao@gmail.com) +
+
2023-01-30 11:38:24
 +
+

*Thread Reply:* @Maciej Obuchowski Do you have a estimated release date for the fix. Our team is specifically interested in using the Emitter to write out to Kafka.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-01-30 11:46:30
 +
+

*Thread Reply:* I think we plan to release somewhere in the next week

+ + + +
+ :gratitude_thank_you: Susmitha Anandarao +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-02-06 09:21:25
 +
+

*Thread Reply:* @Susmitha Anandarao PR fixing this has been merged, release should be today

+ + + +
+ :gratitude_thank_you: Susmitha Anandarao +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paul Lee + (paullee@lyft.com) +
+
2023-01-27 16:31:45
 +
+

👋 +what would be the reason conn_id on something like SQLCheckOperator ends up being None when OpenLineage attempts to extract metadata but is fine on task execution?

+ +

i'm using OpenLineage for Airflow 0.14.1 on 2.3.4 and i'm getting an error about connid not being found. it's a SQLCheckOperator where the check runs fine but the task fails because when OpenLineage goes to extract task metadata it attempts to grab the connid but at that moment it finds it to be None.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2023-01-27 18:38:40
 +
+

*Thread Reply:* hmmm, I am not sure. perhaps @Benji Lampel can help, he’s very familiar with those operators.

+ + + +
+ 👍 Paul Lee +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paul Lee + (paullee@lyft.com) +
+
2023-01-27 18:46:15
 +
+

*Thread Reply:* @Benji Lampel any help would be appreciated!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Benji Lampel + (benjamin@astronomer.io) +
+
2023-01-30 09:01:34
 +
+

*Thread Reply:* Hey Paul, the SQLCheckExtractors were written with the intent that they would be used by a provider that inherits for them - they are all treated as a sort of base class. What is the exact error message you're getting? And what is the operator code? +Could you try this with a PostgresCheckOperator ? +(Also, only the SqlColumnCheckOperator and SqlTableCheckOperator will provide data quality facets in their output, those functions are not implementable in the other operators at this time)

+ + + +
+ 👀 Paul Lee +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paul Lee + (paullee@lyft.com) +
+
2023-01-31 14:36:07
 +
+

*Thread Reply:* @Benji Lampel here is the error message. i am not sure what the operator code is.

+ +

3-01-31, 00:32:38 UTC] {logging_mixin.py:115} WARNING - Traceback (most recent call last): +[2023-01-31, 00:32:38 UTC] {logging_mixin.py:115} WARNING - File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner +[2023-01-31, 00:32:38 UTC] {logging_mixin.py:115} WARNING - self.run() +[2023-01-31, 00:32:38 UTC] {logging_mixin.py:115} WARNING - File "/usr/lib/python3.8/threading.py", line 870, in run +[2023-01-31, 00:32:38 UTC] {logging_mixin.py:115} WARNING - self._target(**self._args, ****self._kwargs) +[2023-01-31, 00:32:38 UTC] {logging_mixin.py:115} WARNING - File "/code/venvs/venv/lib/python3.8/site-packages/openlineage/airflow/listener.py", line 99, in on_running +[2023-01-31, 00:32:38 UTC] {logging_mixin.py:115} WARNING - task_metadata = extractor_manager.extract_metadata(dagrun, task) +[2023-01-31, 00:32:38 UTC] {logging_mixin.py:115} WARNING - File "/code/venvs/venv/lib/python3.8/site-packages/openlineage/airflow/extractors/manager.py", line 28, in extract_metadata +[2023-01-31, 00:32:38 UTC] {logging_mixin.py:115} WARNING - extractor = self._get_extractor(task) +[2023-01-31, 00:32:38 UTC] {logging_mixin.py:115} WARNING - File "/code/venvs/venv/lib/python3.8/site-packages/openlineage/airflow/extractors/manager.py", line 96, in _get_extractor +[2023-01-31, 00:32:38 UTC] {logging_mixin.py:115} WARNING - self.task_to_extractor.instantiate_abstract_extractors(task) +[2023-01-31, 00:32:38 UTC] {logging_mixin.py:115} WARNING - File "/code/venvs/venv/lib/python3.8/site-packages/openlineage/airflow/extractors/extractors.py", line 118, in instantiate_abstract_extractors +[2023-01-31, 00:32:38 UTC] {logging_mixin.py:115} WARNING - task_conn_type = BaseHook.get_connection(task.conn_id).conn_type +[2023-01-31, 00:32:38 UTC] {logging_mixin.py:115} WARNING - File "/code/venvs/venv/lib/python3.8/site-packages/airflow/hooks/base.py", line 67, in get_connection +[2023-01-31, 00:32:38 UTC] {logging_mixin.py:115} WARNING - conn = Connection.get_connection_from_secrets(conn_id) +[2023-01-31, 00:32:38 UTC] {logging_mixin.py:115} WARNING - File "/code/venvs/venv/lib/python3.8/site-packages/airflow/models/connection.py", line 430, in get_connection_from_secrets +[2023-01-31, 00:32:38 UTC] {logging_mixin.py:115} WARNING - raise AirflowNotFoundException(f"The conn_id `{conn_id}` isn't defined") +[2023-01-31, 00:32:38 UTC] {logging_mixin.py:115} WARNING - airflow.exceptions.AirflowNotFoundException: The conn_id `None` isn't defined

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paul Lee + (paullee@lyft.com) +
+
2023-01-31 14:37:06
 +
+

*Thread Reply:* and above that

+ +

[2023-01-31, 00:32:38 UTC] {connection.py:424} ERROR - Unable to retrieve connection from secrets backend (EnvironmentVariablesBackend). Checking subsequent secrets backend. +Traceback (most recent call last): + File "/code/venvs/venv/lib/python3.8/site-packages/airflow/models/connection.py", line 420, in get_connection_from_secrets + conn = secrets_backend.get_connection(conn_id=conn_id) + File "/code/venvs/venv/lib/python3.8/site-packages/airflow/secrets/base_secrets.py", line 91, in get_connection + value = self.get_conn_value(conn_id=conn_id) + File "/code/venvs/venv/lib/python3.8/site-packages/airflow/secrets/environment_variables.py", line 48, in get_conn_value + return os.environ.get(CONN_ENV_PREFIX + conn_id.upper())

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paul Lee + (paullee@lyft.com) +
+
2023-01-31 14:39:31
 +
+

*Thread Reply:* sorry, i should mention we're wrapping over the CheckOperator as we're still migrating from 1.10.15 @Benji Lampel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Benji Lampel + (benjamin@astronomer.io) +
+
2023-01-31 15:09:51
 +
+

*Thread Reply:* What do you mean by wrapping the CheckOperator? Like how so, exactly? Can you show me the operator code you're using in the DAG?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paul Lee + (paullee@lyft.com) +
+
2023-01-31 17:38:45
 +
+

*Thread Reply:* like so

+ +

class CustomSQLCheckOperator(CheckOperator): +....

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paul Lee + (paullee@lyft.com) +
+
2023-01-31 17:39:30
 +
+

*Thread Reply:* i think i found the issue though, we have our own get_hook function and so we don't follow the traditional Airflow way of setting CONN_ID which is why CONN_ID is always None and that path only gets called through OpenLineage which doesn't ever get called with our custom wrapper

+ + + +
+ ✅ Benji Lampel +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-01-30 03:50:39
 +
+

Hi everyone, I am using openlineage to capture column level lineage from spark databricks. I noticed that the environment variables captured are only present in the start event, but are not present in the complete event. Is there a reason why it is implemented like this? It seems more intuitive that whatever variables are present in the start event should also be present in the complete event...

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Susmitha Anandarao + (susmitha.anandarao@gmail.com) +
+
2023-01-31 08:30:37
 +
+

Hi everyone.. Does the DBT integration provide an option to emit events to a Kafka topic similar to the Spark integration? I could not find anything regarding this in the documentation and I wanted to make sure if only http transport type is supported. Thank you!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2023-01-31 12:57:47
 +
+

*Thread Reply:* The dbt integration uses the python client, you should be able to do something similar than with the java client. See here: https://github.com/OpenLineage/OpenLineage/tree/main/client/python#kafka

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Susmitha Anandarao + (susmitha.anandarao@gmail.com) +
+
2023-01-31 13:26:33
 +
+

*Thread Reply:* Thank you for this!

+ +

I created a openlineage.yml file with the following data to test out the integration locally. +transport: + type: "kafka" + config: { 'bootstrap.servers': 'localhost:9092', } + topic: "ol_dbt_events" +However, I run into a no module named 'confluent_kafka' error from this code. +Running OpenLineage dbt wrapper version 0.19.2 +This wrapper will send OpenLineage events at the end of dbt execution. +Traceback (most recent call last): + File "/Users/susmithaanandarao/.pyenv/virtualenvs/dbt-examples-domain-repo/3.9.8/bin/dbt-ol", line 168, in &lt;module&gt; + main() + File "/Users/susmithaanandarao/.pyenv/virtualenvs/dbt-examples-domain-repo/3.9.8/bin/dbt-ol", line 94, in main + client = OpenLineageClient.from_environment() + File "/Users/susmithaanandarao/.pyenv/virtualenvs/dbt-examples-domain-repo/3.9.8/lib/python3.9/site-packages/openlineage/client/client.py", line 73, in from_environment + return cls(transport=get_default_factory().create()) + File "/Users/susmithaanandarao/.pyenv/virtualenvs/dbt-examples-domain-repo/3.9.8/lib/python3.9/site-packages/openlineage/client/transport/factory.py", line 37, in create + return self._create_transport(yml_config) + File "/Users/susmithaanandarao/.pyenv/virtualenvs/dbt-examples-domain-repo/3.9.8/lib/python3.9/site-packages/openlineage/client/transport/factory.py", line 69, in _create_transport + return transport_class(config_class.from_dict(config)) + File "/Users/susmithaanandarao/.pyenv/virtualenvs/dbt-examples-domain-repo/3.9.8/lib/python3.9/site-packages/openlineage/client/transport/kafka.py", line 43, in __init__ + import confluent_kafka as kafka +ModuleNotFoundError: No module named 'confluent_kafka' +Manually installing confluent-kafka worked. But I am curious why it was not automatically installed and if I am missing any config.

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/3d0fee9002a8d7a85f47c8a6e57446174f433e34/client/python/setup.py | setup.py>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-02-02 14:39:29
 +
+

*Thread Reply:* @Susmitha Anandarao It's not installed because it's large binary package. We don't want to install for every user something giant majority won't use, and it's 100x bigger than rest of the client.

+ +

We need to indicate this way better, and do not throw this error directly at user thought, both in docs and code.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Benji Lampel + (benjamin@astronomer.io) +
+
2023-01-31 11:28:53
 +
+

~Hey, would love to see a release of OpenLineage~

+ + + +
+ ➕ Michael Robinson, Jakub Dardziński, Ross Turk, Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2023-01-31 12:51:44
 +
+

Hello, I have been working on a proposal to bring an OpenLineage provider to +Airflow. I am currently looking for feedback on a draft AIP. See the thread here: https://lists.apache.org/thread/2brvl4ynkxcff86zlokkb47wb5gx8hw7

+ + + +
+ 🔥 Maciej Obuchowski, Viraj Parekh, Jakub Dardziński, Enrico Rotundo, Harel Shein, Paweł Leszczyński +
+ +
+ 👀 Enrico Rotundo +
+ +
+ 🙌 Will Johnson +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Bramha Aelem + (bramhaaelem@gmail.com) +
+
2023-01-31 14:02:21
 +
+

@Willy Lulciuc, - Any updates on - https://github.com/OpenLineage/OpenLineage/discussions/1494

+
+ + + + +
#1494 Usage Of databricks generated tables as an in/out in the Datapipeline Process
+
+ <p>Hi,</p> + +<p>First activity : Making HTTP Call to pull the lookup data and store it in datalake. <br /> +Second Activity : After the completion of first activity, invoking Azure databricks to use the lookup file and generate the output tables. What are all the steps should be follow to refer databricks generated tables facets as an output in the current activity &amp; input to the subsequent activities in the pipeline. <br /> +When I configure spark generated tables as output to the current activity the existing spark metadata is not showing up. <br /> +How can this be achievable. <br /> +Can someone please guide me on this. <br /> +Note: After the execution of each activity in ADF Pipeline I am sending start and complete/fail event lineage to Marquez.</p> + +<p>Thanks, <br /> +Brahma</p> +
+ +
+
Category
+ Ideas +
+ +
+
Comments
+ 3 +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Natalie Zeller + (natalie.zeller@naturalint.com) +
+
2023-02-02 08:26:38
 +
+

Hello, +While trying to use OpenLineage with spark, I've noticed that sometimes the query execution is missing or already got closed (here is the relevant code). As a result, some of the events are skipped. Is this a known issue? Is there a way to overcome it?

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/02a01a1ed60916424bff7755942e0f2e10ae8c67/integration/spark/app/src/main/java/io/openlineage/spark/agent/lifecycle/ContextFactory.java | ContextFactory.java>
+
+ <pre><code> return Optional.ofNullable(SQLExecution.getQueryExecution(executionId)) +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-02-02 08:39:34
 +
+

*Thread Reply:* https://github.com/OpenLineage/OpenLineage/issues/999#issuecomment-1209048556

+ +

Does this fit your experience?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-02-02 08:39:59
 +
+

*Thread Reply:* We sometimes experience this in context of very small, quick jobs

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Natalie Zeller + (natalie.zeller@naturalint.com) +
+
2023-02-02 08:43:24
 +
+

*Thread Reply:* Yes, my scenarios are dealing with quick jobs. +Good to know that we will be able to solve it with future spark versions. Thanks!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-02-02 11:09:13
 +
+

@channel +This month’s OpenLineage TSC meeting is next Thursday, February 9th, at 10 am PT. Join us on Zoom: https://bit.ly/OLzoom. All are welcome! +On the tentative agenda:

+ +
  1. Recent release overview @Michael Robinson
  2. AIP: OpenLineage in Airflow
  3. Discussions: +• Real-world implementation of OpenLineage (What does it really mean?) @Sheeri Cabral (Collibra) (continued) +• Using namespaces @Michael Robinson
  4. Open discussion +Notes: https://bit.ly/OLwiki +Is there a topic you think the community should discuss at this or a future meeting? Reply or DM me to add items to the agenda.
    5. +
+ + + +
+ 🔥 Maciej Obuchowski, Bramha Aelem, Viraj Parekh, Brad Paskewitz, Harel Shein +
+ +
+ 👍 Bramha Aelem, Viraj Parekh, Enrico Rotundo, Daniel Henneberger +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-02-03 13:22:51
 +
+

Hi folks, I’m opening a vote to release OpenLineage 0.20.0, featuring: +• Airflow: add new extractor for GCSToGCSOperator + Adds a new extractor for this operator. +• Proxy: implement lineage event validator for client proxy
+ Implements logic in the proxy (which is still in development) for validating and handling lineage events. +• A fix of a breaking change in the common integration and other bug fixes in the DBT, Airflow, Spark, and SQL integrations and in the Java and Python clients. +As per the policy here, three +1s from committers will authorize. Thanks in advance.

+ + + +
+ ➕ Willy Lulciuc, Maciej Obuchowski, Julien Le Dem, Jakub Dardziński, Howard Yoo +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2023-02-03 13:24:03
 +
+

*Thread Reply:* exciting to see the client proxy work being released by @Minkyu Park 💯

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-02-03 13:35:38
 +
+

*Thread Reply:* This was without a doubt among the fastest release votes we’ve ever had 😉 . Thank you! You can expect the release to happen on Monday.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Minkyu Park + (minkyu@datakin.com) +
+
2023-02-03 14:02:52
 +
+

*Thread Reply:* Lol the proxy is still in development and not ready for use

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2023-02-03 14:03:26
 +
+

*Thread Reply:* Good point! Let’s make that clear in the release / docs?

+ + + +
+ 👍 Michael Robinson, Minkyu Park +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Minkyu Park + (minkyu@datakin.com) +
+
2023-02-03 14:03:33
 +
+

*Thread Reply:* But it doesn’t block anything anyway, so happy to see the release

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Minkyu Park + (minkyu@datakin.com) +
+
2023-02-03 14:04:38
 +
+

*Thread Reply:* We can celebrate that the proposal for the proxy is merged. I’m happy with that 🥳

+ + + +
+ 🎊 Willy Lulciuc +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Daniel Joanes + (djoanes@gmail.com) +
+
2023-02-06 00:01:49
 +
+

Hey 👋 From what I gather, there's no solution to getting column level lineage from spark streaming jobs. Is there a issue I can follow to keep track?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2023-02-06 14:47:15
 +
+

*Thread Reply:* Hey @Daniel Joanes! thanks for the question.

+ +

I am not aware of an issue that captures this. Column-level lineage is a somewhat new facet in the spec, and implementations across the various integrations are in varying states of readiness.

+ +

I invite you to create the issue - that way it's attributed to you, which makes sense because you're the one who first raised it. But I'm happy to create it for you & give you the PR# if you'd rather, just let me know 👍

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Daniel Joanes + (djoanes@gmail.com) +
+
2023-02-06 14:50:59
 +
+

*Thread Reply:* Go for it, once it's created i'll add a watch

+ + + +
+ 👍 Ross Turk +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Daniel Joanes + (djoanes@gmail.com) +
+
2023-02-06 14:51:13
 +
+

*Thread Reply:* Thanks Ross!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2023-02-06 23:10:30
 +
+

*Thread Reply:* https://github.com/OpenLineage/OpenLineage/issues/1581

+
+ + + + +
#1581 Spark integration does not support column-lineage for streaming jobs
+
+ <p>There does not seem to be support for column-level lineage in the Spark integration for streaming jobs. Or, alternatively, it is insufficiently documented.</p> +
+ +
+
Labels
+ integration/spark, column-level-lineage +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-02-07 18:46:50
 +
+

@channel +OpenLineage 0.20.4 is now available, including: +Additions: +• Airflow: add new extractor for GCSToGCSOperator #1495 @sekikn +• Flink: resolve topic names from regex, support 1.16.0 #1522 @pawel-big-lebowski +• Proxy: implement lineage event validator for client proxy #1469 @fm100 +Changes: +• CI: use ruff instead of flake8, isort, etc., for linting and formatting #1526 @mobuchowski +Plus many bug fixes & doc changes. +Thank you to all our contributors! +Release: https://github.com/OpenLineage/OpenLineage/releases/tag/0.20.4 +Changelog: https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md +Commit history: https://github.com/OpenLineage/OpenLineage/compare/0.19.2...0.20.4 +Maven: https://oss.sonatype.org/#nexus-search;quick~openlineage +PyPI: https://pypi.org/project/openlineage-python/

+ + + +
+ 🎉 Kengo Seki, Harel Shein, Willy Lulciuc, Nadav Geva +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-02-08 15:31:32
 +
+

@channel +Friendly reminder: this month’s OpenLineage TSC meeting is tomorrow at 10am, and all are welcome. https://openlineage.slack.com/archives/C01CK9T7HKR/p1675354153489629

+ + + +
+ ❤️ Minkyu Park, Kengo Seki, Paweł Leszczyński, Harel Shein, Sheeri Cabral (Collibra), Enrico Rotundo +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harel Shein + (harel.shein@gmail.com) +
+
2023-02-09 10:50:07
 +
+

Hey, can we please schedule a release of OpenLineage? I would like to have a release that includes the latest fixes for Async Operator on Airflow and some dbt bug fixes.

+ + + +
+ ➕ Michael Robinson, Maciej Obuchowski, Benji Lampel, Willy Lulciuc +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-02-09 10:50:49
 +
+

*Thread Reply:* Thanks for requesting a release. 3 +1s from committers will authorize an immediate release.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-02-09 11:15:35
 +
+

*Thread Reply:* 0.20.5 ?

+ + + +
+ ➕ Harel Shein +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Benji Lampel + (benjamin@astronomer.io) +
+
2023-02-09 11:28:20
 +
+

*Thread Reply:* @Michael Robinson auth'd

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-02-09 11:32:06
 +
+

*Thread Reply:* 👍 the release is authorized

+ + + +
+ ❤️ Sheeri Cabral (Collibra), Willy Lulciuc, Paweł Leszczyński +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Avinash Pancham + (avinashpancham@outlook.com) +
+
2023-02-09 15:57:58
 +
+

Hi all, I have been experimenting with OpenLineage for a few days and it's great! I successfully setup the openlineage-spark listener on my Databricks cluster and that pushes openlineage data to our Marquez backend. That was all pretty easy to do 🙂

+ +

Now for my challenge: I would like to actually extend the metadata that my cluster pushes with custom values (you can think of spark config settings, commit hash of the executed code, or maybe even runtime defined values). I browsed through some documentation and found custom facets one can define. The link below describes how to use Python to push custom metadata to a backend, but I was actually hoping that there was a way to do this automatically in Spark. So ideally I would like to write my own OpenLineage.json (that has my custom facet) and tell Spark to use that Openlineage spec instead of the default one. In that way I hope my custom metadata will be forwarded automatically.

+ +

I just do not know how to do that (and whether that is even possible), since I could not find any tutorials on that topic. Any help on this would be greatly appreciated!

+ +

https://openlineage.io/docs/spec/facets/custom-facets

+
+
openlineage.io
+ + + +
Custom Facets | OpenLineage Docs
+
+ In addition to the existing facets mentioned in this documentation, users can extend the base facets and provide their own facet definition as part of the payload in OpenLineage event. For example, when OpenLineage event is emitted from the Apache Airflow using OpenLineage's Airflow integration, the following facets can be observed: +
+ + + + + + +
Original URL: https://openlineage.io/docs/spec/facets/custom-facets
+ + + +
+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.json | OpenLineage.json>
+
+ <p>``` +{ + "$schema": "<a href="https://json-schema.org/draft/2020-12/schema">https://json-schema.org/draft/2020-12/schema</a>", + "$id": "<a href="https://openlineage.io/spec/1-0-5/OpenLineage.json">https://openlineage.io/spec/1-0-5/OpenLineage.json</a>", + "$defs": { + "RunEvent": { + "type": "object", + "properties": { + "eventType": { + "description": "the current transition of the run state. It is required to issue 1 START event and 1 of [ COMPLETE, ABORT, FAIL ] event per run. Additional events with OTHER eventType can be added to the same run. For example to send additional metadata after the run is complete", + "type": "string", + "enum": [ + "START", + "RUNNING", + "COMPLETE", + "ABORT", + "FAIL", + "OTHER" + ], + "example": "START|RUNNING|COMPLETE|ABORT|FAIL|OTHER" + }, + "eventTime": { + "description": "the time the event occurred at", + "type": "string", + "format": "date-time" + }, + "run": { + "$ref": "#/$defs/Run" + }, + "job": { + "$ref": "#/$defs/Job" + }, + "inputs": { + "description": "The set of <strong><em>*input</em><em></strong> datasets.", + "type": "array", + "items": { + "$ref": "#/$defs/InputDataset" + } + }, + "outputs": { + "description": "The set of <strong></em><em>output</em>*</strong> datasets.", + "type": "array", + "items": { + "$ref": "#/$defs/OutputDataset" + } + }, + "producer": { + "description": "URI identifying the producer of this metadata. For example this could be a git url with a given tag or sha", + "type": "string", + "format": "uri", + "example": "<a href="https://github.com/OpenLineage/OpenLineage/blob/v1-0-0/client">https://github.com/OpenLineage/OpenLineage/blob/v1-0-0/client</a>" + }, + "schemaURL": { + "description": "The JSON Pointer (<a href="https://tools.ietf.org/html/rfc6901">https://tools.ietf.org/html/rfc6901</a>) URL to the corresponding version of the schema definition for this RunEvent", + "type": "string", + "format": "uri", + "example": "<a href="https://openlineage.io/spec/0-0-1/OpenLineage.json">https://openlineage.io/spec/0-0-1/OpenLineage.json</a>" + } + }, + "required": [ + "run", + "job", + "eventTime", + "producer", + "schemaURL" + ] + }, + "Run": { + "type": "object", + "properties": { + "runId": { + "description": "The globally unique ID of the run associated with the job.", + "type": "string", + "format": "uuid" + }, + "facets": { + "description": "The run facets.", + "type": "object", + "anyOf": [ + { + "type": "object", + "additionalProperties": { "$ref": "#/$defs/RunFacet" } + }, + { "$ref": "facets/ErrorMessageRunFacet.json" }, + { "$ref": "facets/ExternalQueryRunFacet.json" }, + { "$ref": "facets/NominalTimeRunFacet.json" }, + { "$ref": "facets/ParentRunFacet.json" } + ] + } + }, + "required": [ + "runId" + ] + }, + "RunFacet": { + "description": "A Run Facet", + "type": "object", + "allOf": [ + { "$ref": "#/$defs/BaseFacet" } + ] + }, + "Job": { + "type": "object", + "properties": { + "namespace": { + "description": "The namespace containing that job", + "type": "string", + "example": "my-scheduler-namespace" + }, + "name": { + "description": "The unique name for that job within that namespace", + "type": "string", + "example": "myjob.mytask" + }, + "facets": { + "description": "The job facets.", + "type": "object", + "anyOf": [ + { + "type": "object", + "additionalProperties": { "$ref": "#/$defs/JobFacet" } + }, + { "$ref": "facets/DocumentationJobFacet.json" }, + { "$ref": "facets/OwnershipJobFacet.json" }, + { "$ref": "facets/SourceCodeJobFacet.json" }, + { "$ref": "facets/SourceCodeLocationJobFacet.json" }, + { "$ref": "facets/SQLJobFacet.json" } + ] + } + }, + "required": [ + "namespace", + "name" + ] + }, + "JobFacet": { + "description": "A Job Facet", + "type": "object", + "allOf": [ + { "$ref": "#/$defs/BaseFacet" } + ] + }, + "InputDataset": { + "description": "An input dataset", + "type": "object", + "allOf": [ + { "$ref": "#/$defs/Dataset" }, + { + "type": "object", + "properties": { + "inputFacets": { + "description": "The input facets for this dataset.", + "type": "object", + "anyOf": [ + { + "type": "object", + "additionalProperties": { "$ref": "#/$defs/InputDatasetFacet" } + }, + { "$ref": "facets/DataQualityMetricsInputDatasetFacet.json"} + ] + } + } + } + ] + }, + "InputDatasetFacet": { + "description": "An Input Dataset Facet", + "type": "object", + "allOf": [ + { "$ref": "#/$defs/BaseFacet" } + ] + }, + "OutputDataset": { + "description": "An output dataset", + "type": "object", + "allOf": [ + { "$ref": "#/$defs/Dataset" }, + { + "type": "object", + "properties": { + "outputFacets": { + "description": "The output facets for this dataset", + "type": "object", + "anyOf": [ + { + "type": "object", + "additionalProperties": { "$ref": "#/$defs/OutputDatasetFacet" } + }, + { "$ref": "facets/OutputStatisticsOutputDatasetFacet.json"} + ] + } + } + } + ] + }, + "OutputDatasetFacet": { + "description": "An Output Dataset Facet", + "type": "object", + "allOf": [ + { "$ref": "#/$defs/BaseFacet" } + ] + }, + "Dataset": { + "type": "object", + "properties": { + "namespace": { + "description": "The namespace containing that dataset", + "type": "string", + "example": "my-datasource-namespace" + }, + "name": { + "description": "The unique name for that dataset within that namespace", + "type": "string", + "example": "instance.schema.table" + }, + "facets": { + "description": "The facets for this dataset", + "type": "object", + "anyOf": [ + { + "type": "object", + "additionalProperties": { "$ref": "#/$defs/DatasetFacet" } + }, + { "$ref": "facets/ColumnLineageDatasetFacet.json" }, + { "$ref": "facets/DatasourceDatasetFacet.json" }, + { "$ref": "facets/DataQualityAssertionsDatasetFacet.json" }, + { "$ref": "facets/LifecycleStateChangeDatasetFacet.json" }, + { "$ref": "facets/OwnershipDatasetFacet.json" }, + { "$ref": "facets/SchemaDatasetFacet.json" }, + { "$ref": "facets/StorageDatasetFacet.json" }, + { "$ref": "facets/SymlinksDatasetFacet.json" }, + { "$ref": "facets/DatasetVersionDatasetFacet.json" } + ] + } + }, + "required": [ + "namespace", + "name" + ] + }, + "DatasetFacet": { + "description": "A Dataset Facet", + "type": "object", + "allOf": [ + { "$ref": "#/$defs/BaseFacet" } + ] + }, + "BaseFacet": { + "description": "all fields of the base facet are prefixed with _ to avoid name conflicts in facets", + "type": "object", + "properties": { + "_producer": { + "description": "URI identifying the producer of this metadata. For example this could be a git url with a given tag or sha", + "type": "string", + "format": "uri", + "example": "<https://github.com/OpenLineage/OpenLineage/blob/v1-0-0/clie…</p> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Susmitha Anandarao + (susmitha.anandarao@gmail.com) +
+
2023-02-09 16:23:36
 +
+

*Thread Reply:* I am also exploring something similar, but writing to kafka, and would want to know more on how we could add custom metadata from spark.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-02-10 02:23:40
 +
+

*Thread Reply:* Hi @Avinash Pancham @Susmitha Anandarao, it's great to hear about successful experimenting on your side.

+ +

Although Openlineage spec provides some built-in facets definition, a facet object can be anything you want (https://openlineage.io/apidocs/openapi/#tag/OpenLineage/operation/postRunEvent). The example metadata provided in this chat could be put into job or run facets I believe.

+ +

There is also a way to extend Spark integration to collect custom metadata described here (https://github.com/OpenLineage/OpenLineage/tree/main/integration/spark#extending). One needs to create own JAR with DatasetFacetBuilders, RunFacetsBuilder (whatever is needed). openlineage-spark integration will make use of those bulders.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sheeri Cabral (Collibra) + (sheeri.cabral@collibra.com) +
+
2023-02-10 09:09:10
 +
+

*Thread Reply:* (I would love to see what your specs are! I’m not with Astronomer, just a community member, but I am finding that many of the customizations people are making to the spec are valuable ones that we should consider adding to core)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Susmitha Anandarao + (susmitha.anandarao@gmail.com) +
+
2023-02-14 16:51:28
 +
+

*Thread Reply:* Are there any examples out there of customizations already done in Spark? An example would definitely help!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-02-15 08:43:08
 +
+

*Thread Reply:* I think @Will Johnson might have something to add about customization

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2023-02-15 23:58:36
 +
+

*Thread Reply:* Oh man... Mike Collado did a nice write up on Slack of how many different ways there are to customize / extend OpenLineage. I know we all talked about doing a blog post at one point!

+ +

@Susmitha Anandarao - You might take a look at https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/shared/src/[…]ark/agent/facets/builder/DatabricksEnvironmentFacetBuilder.java which has a hard coded set of properties we are extracting.

+ +

It looks like Avinash's changes were accepted as well: https://github.com/OpenLineage/OpenLineage/pull/1545

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/shared/src/main/java/io/openlineage/spark/agent/facets/builder/DatabricksEnvironmentFacetBuilder.java | DatabricksEnvironmentFacetBuilder.java>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+
+ + + + +
#1545 Add CustomEnvironmentFacetBuilder class
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-02-10 12:42:24
 +
+

@channel +OpenLineage 0.20.6 is now available, including: +Additions +• Airflow: add new extractor for FTPFileTransmitOperator #1603 @sekikn +Changes +• Airflow: make extractors for async operators work #1601 @JDarDagran +Thanks to all our contributors! +For the bug fixes and details, see: +Release: https://github.com/OpenLineage/OpenLineage/releases/tag/0.20.6 +Changelog: https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md +Commit history: https://github.com/OpenLineage/OpenLineage/compare/0.20.4...0.20.6 +Maven: https://oss.sonatype.org/#nexus-search;quick~openlineage +PyPI: https://pypi.org/project/openlineage-python/

+ + + +
+ 🥳 Minkyu Park, Willy Lulciuc, Kengo Seki, Paweł Leszczyński, Anirudh Shrinivason, pankaj koti, Maciej Obuchowski +
+ +
+ ❤️ Minkyu Park, Ross Turk, Willy Lulciuc, Kengo Seki, Paweł Leszczyński, Anirudh Shrinivason, pankaj koti +
+ +
+ 🎉 Minkyu Park, Willy Lulciuc, Kengo Seki, Anirudh Shrinivason, pankaj koti +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-02-13 14:20:26
 +
+

Hi everyone, in case you missed the announcement at the most recent community meeting, our first-ever meetup will be held on March 9th in Providence, RI. Join us there to learn more about the present and future of OpenLineage, meet other members of the ecosystem, learn about the project’s goals and fundamental design, and participate in a discussion about the future of the project. +Food will be provided, and the meetup is open to all. Don’t miss this opportunity to influence the direction of this important new standard! We hope to see you there. +More information: https://openlineage.io/blog/data-lineage-meetup/

+
+
openlineage.io
+ + + +
Happening Soon - Our First Meetup!
+
+ The inaugural Data Lineage Meetup will take place on March 9th in Providence, RI. +
+ + + + + + +
Original URL: https://openlineage.io/blog/data-lineage-meetup/
+ + + +
+ + + +
+ 🎉 Harel Shein, Ross Turk, Maciej Obuchowski, Kengo Seki, Paweł Leszczyński, Willy Lulciuc, Sheeri Cabral (Collibra) +
+ +
+ 🔥 Harel Shein, Ross Turk, Maciej Obuchowski, Anirudh Shrinivason, Kengo Seki, Paweł Leszczyński, Willy Lulciuc, Sheeri Cabral (Collibra) +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Quentin Nambot + (qnambot@gmail.com) +
+
2023-02-15 04:52:27
 +
+

Hi, I opened a PR to fix the way that Athena extractor get the database, but spark integration tests failed. However I don't think that it is related to my PR, since I only updated the Airflow integration +Can anybody help me with that please? 🙏

+
+ + + + +
#1631 airflow: get table database in athena extractor
+
+ <p><strong>Problem</strong></p> + +<p>Athena extractor get database from operator. However it is possible to interact with tables from other databases.</p> + +<p>Closes: <a href="https://github.com/OpenLineage/OpenLineage/issues/1627">#ISSUE-NUMBER</a></p> + +<p><strong>Solution</strong></p> + +<p>If a table is used with a database (ex: <code>catalog.my_db.my_table</code> or <code>my_db.my_table</code>), the field <code>table.schema</code> is the database. However it is <code>None</code>. <br /> +So we can use this field to get database, or use operator default database if <code>table.schema is None</code></p> + +<p>☐ Your change modifies the <a href="https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.json">core</a> OpenLineage model +☐ Your change modifies one or more OpenLineage <a href="https://github.com/OpenLineage/OpenLineage/tree/main/spec/facets">facets</a></p> + +<p><strong>Checklist</strong></p> + +<p>☑︎ You've <a href="https://github.com/OpenLineage/OpenLineage/blob/main/why-the-dco.md">signed-off</a> your work +☑︎ Your pull request title follows our <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CONTRIBUTING.md#creating-pull-requests">guidelines</a> +☑︎ Your changes are accompanied by tests (<em>if relevant</em>) +☑︎ Your change contains a <a href="https://kurtisnusbaum.medium.com/stacked-diffs-keeping-phabricator-diffs-small-d9964f4dcfa6">small diff</a> and is self-contained +☐ You've updated any relevant documentation (<em>if relevant</em>) +☐ Your comment includes a one-liner for the changelog about the specific purpose of the change (<em>if necessary</em>) +☐ You've versioned the core OpenLineage model or facets according to <a href="https://docs.snowplowanalytics.com/docs/pipeline-components-and-applications/iglu/common-architecture/schemaver">SchemaVer</a> (<em>if relevant</em>) +☐ You've added a <a href="https://github.com/OpenLineage/OpenLineage/tree/main/.github/header_templates.md">header</a> to source files (<em>if relevant</em>)</p> + +<hr /> + +<p>SPDX-License-Identifier: Apache-2.0 <br /> +Copyright 2018-2023 contributors to the OpenLineage project</p> +
+ +
+
Labels
+ integration/airflow, extractor +
+ +
+
Comments
+ 2 +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Quentin Nambot + (qnambot@gmail.com) +
+
2023-02-15 04:52:59
 +
+

*Thread Reply:* https://app.circleci.com/pipelines/github/OpenLineage/OpenLineage/6398/workflows/9d2d19c8-f2d9-4148-a4f3-5dad3ba99eb1/jobs/97759 +ERROR: Missing environment variable {i}

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-02-15 07:19:39
 +
+

*Thread Reply:* @Quentin Nambot this happens because we run additional integration tests against real databases (like BigQuery) which aren't ever configured on forks, since we don't want to expose our secrets. We need to figure out how to make this experience better, but in the meantime we've pushed your code using git-push-fork-to-upstream-branch and it passes all the tests.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-02-15 07:21:49
 +
+

*Thread Reply:* Feel free to un-draft your PR if you think it's ready for review

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Quentin Nambot + (qnambot@gmail.com) +
+
2023-02-15 08:03:56
 +
+

*Thread Reply:* Ok nice thanks 👍

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Quentin Nambot + (qnambot@gmail.com) +
+
2023-02-15 08:04:49
 +
+

*Thread Reply:* I think it's ready, however should I update the version somewhere?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-02-15 08:42:39
 +
+

*Thread Reply:* @Quentin Nambot I don't think so - it's just that you opened PR as Draft , so I'm not sure if you want to add something else to it.

+ + + +
+ 👍 Quentin Nambot +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Quentin Nambot + (qnambot@gmail.com) +
+
2023-02-15 08:43:36
 +
+

*Thread Reply:* No I don't want to add anything so I opened it 👍

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Allison Suarez + (asuarezmiranda@lyft.com) +
+
2023-02-15 21:26:37
 +
+

@here I have a question about extending the spark integration. Is there a way to use a custom visitor factory? I am trying to see if I can add a visitor for a command that is not currently covered in this integration (AlterTableAddPartitionCommand). It seems that because its not in the base visitor factory I am unable to use the visitor I created.

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/shared/src/main/java/io/openlineage/spark/agent/lifecycle/BaseVisitorFactory.java | BaseVisitorFactory.java>
+
+ <pre><code> list.add(new AlterTableAddColumnsCommandVisitor(context)); +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2023-02-15 21:30:47
 +
+

*Thread Reply:* You can add your own EventHandlerFactory https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/shared/src/main/java/io/openlineage/spark/api/OpenLineageEventHandlerFactory.java . See the docs about extending here - https://github.com/OpenLineage/OpenLineage/tree/main/integration/spark#extending

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/shared/src/main/java/io/openlineage/spark/api/OpenLineageEventHandlerFactory.java | OpenLineageEventHandlerFactory.java>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Allison Suarez + (asuarezmiranda@lyft.com) +
+
2023-02-15 21:32:19
 +
+

*Thread Reply:* I have that set up already like this: +public class LyftOpenLineageEventHandlerFactory implements OpenLineageEventHandlerFactory { + @Override + public Collection&lt;PartialFunction&lt;LogicalPlan, List&lt;OutputDataset&gt;&gt;&gt; + createOutputDatasetQueryPlanVisitors(OpenLineageContext context) { + Collection&lt;PartialFunction&lt;LogicalPlan, List&lt;OutputDataset&gt;&gt;&gt; visitors = new ArrayList&lt;PartialFunction&lt;LogicalPlan, List&lt;OutputDataset&gt;&gt;&gt;(); + visitors.add(new LyftInsertIntoHadoopFsRelationVisitor(context)); + visitors.add(new AlterTableAddPartitionVisitor(context)); + visitors.add(new AlterTableDropPartitionVisitor(context)); + return visitors; + } +}

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Allison Suarez + (asuarezmiranda@lyft.com) +
+
2023-02-15 21:33:35
 +
+

*Thread Reply:* do I just add a constructor? the visitorFactory is private so I wasn't sure if that's something that was intended to change

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Allison Suarez + (asuarezmiranda@lyft.com) +
+
2023-02-15 21:34:30
 +
+

*Thread Reply:* .

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Allison Suarez + (asuarezmiranda@lyft.com) +
+
2023-02-15 21:34:49
 +
+

*Thread Reply:* @Michael Collado

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2023-02-15 21:35:14
 +
+

*Thread Reply:* The VisitorFactory is only used by the internal EventHandlerFactory. It shouldn’t be needed for your custom one

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2023-02-15 21:35:48
 +
+

*Thread Reply:* Have you added the file to the META-INF folder of your jar?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Allison Suarez + (asuarezmiranda@lyft.com) +
+
2023-02-16 11:01:56
 +
+

*Thread Reply:* yes, I am able to use my custom event handler factory with a list of visitors but for some reason I cant access the visitors for some commands (AlterTableAddPartitionCommand) is one

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Allison Suarez + (asuarezmiranda@lyft.com) +
+
2023-02-16 11:02:29
 +
+

*Thread Reply:* so even if I set up everything correctly I am unable to reach the code for that specific visitor

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Allison Suarez + (asuarezmiranda@lyft.com) +
+
2023-02-16 11:05:22
 +
+

*Thread Reply:* and my assumption is I can reach other commands but not this one because the command is not defined in the BaseVisitorFactory but maybe im wrong @Michael Collado

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2023-02-16 15:05:19
 +
+

*Thread Reply:* the VisitorFactory is loaded by the InternalEventHandlerFactory here. However, the createOutputDatasetQueryPlanVisitors should contain a union of everything defined by the VisitorFactory as well as your custom visitors: see this code.

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/app/src/main/java/io/openlineage/spark/agent/lifecycle/InternalEventHandlerFactory.java | InternalEventHandlerFactory.java>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/app/src/main/java/io/openlineage/spark/agent/lifecycle/InternalEventHandlerFactory.java | InternalEventHandlerFactory.java>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2023-02-16 15:09:21
 +
+

*Thread Reply:* there might be a conflict with another visitor that’s being matched against that command. Can you turn on debug logging and look for this line to see what visitor is being applied to that command?

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/shared/src/main/java/io/openlineage/spark/agent/util/PlanUtils.java | PlanUtils.java>
+
+ <pre><code> if (log.isDebugEnabled()) { + log.debug( + "Visitor {} visited {}, returned {}", + pfn.getClass().getCanonicalName(), + x.getClass().getCanonicalName(), + collection); + } +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Allison Suarez + (asuarezmiranda@lyft.com) +
+
2023-02-16 16:54:46
 +
+

*Thread Reply:* This was helpful, it works now, thank you so much Michael!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
slackbot + +
+
2023-02-16 19:08:26
 +
+

This message was deleted.

+ + + +
+ 👋 Willy Lulciuc +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2023-02-16 19:09:49
 +
+

*Thread Reply:* what is the curl cmd you are running? and what endpoint are you hitting? (assuming Marquez?)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
thebruuu + (bruno.c@inwind.it) +
+
2023-02-16 19:18:28
 +
+

*Thread Reply:* yep +I am running curl - X curl -X POST http://localhost:5000/api/v1/namespaces/test ^ + -H 'Content-Type: application/json' ^ + -d '{ownerName:"me", description:"no description"^ + }'

+ +

the weird thing is the log where I don't have a 0.0.0.0 IP (the log correspond to the equivament postman command)

+ +

marquez-api | WARN [2023-02-17 00:14:32,695] marquez.logging.LoggingMdcFilter: status: 405 +marquez-api | XXX.23.0.1 - - [17/Feb/2023:00:14:32 +0000] "POST /api/v1/namespaces/test HTTP/1.1" 405 52 "-" "PostmanRuntime/7.30.0" 2

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2023-02-16 19:23:08
 +
+

*Thread Reply:* Marquez logs all supported endpoints (and methods) on start up. For example, here are all the supported methods on /api/v1/namespaces/{namespace} : +marquez-api | DELETE /api/v1/namespaces/{namespace} (marquez.api.NamespaceResource) +marquez-api | GET /api/v1/namespaces/{namespace} (marquez.api.NamespaceResource) +marquez-api | PUT /api/v1/namespaces/{namespace} (marquez.api.NamespaceResource) +To ADD a namespace, you’ll want to use PUT (see API docs)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
thebruuu + (bruno.c@inwind.it) +
+
2023-02-16 19:26:23
 +
+

*Thread Reply:* 3rd stupid question of the night +Sorry kept on trying POST who knows why

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2023-02-16 19:26:56
 +
+

*Thread Reply:* no worries! keep the questions coming!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2023-02-16 19:29:46
 +
+

*Thread Reply:* well, maybe because it’s so late on your end! get some rest!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
thebruuu + (bruno.c@inwind.it) +
+
2023-02-16 19:36:25
 +
+

*Thread Reply:* Yeah but I want to see how it works +Right now I have a response 200 for the creation of the names ... but it seems that nothing occurred +nor on marquez front end (localhost:3000) +nor on the database

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2023-02-16 19:37:13
 +
+

*Thread Reply:* can you curl the list namespaces endpoint?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
thebruuu + (bruno.c@inwind.it) +
+
2023-02-16 19:38:14
 +
+

*Thread Reply:* yep : nothing changed +only default and food_delivery

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2023-02-16 19:38:47
 +
+

*Thread Reply:* can you post your server logs? you should see the request

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
thebruuu + (bruno.c@inwind.it) +
+
2023-02-16 19:40:41
 +
+

*Thread Reply:* marquez-api | XXX.23.0.4 - - [17/Feb/2023:00:30:38 +0000] "PUT /api/v1/namespaces/ciro HTTP/1.1" 500 110 "-" "-" 7 +marquez-api | INFO [2023-02-17 00:32:07,072] marquez.logging.LoggingMdcFilter: status: 200

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2023-02-16 19:41:12
 +
+

*Thread Reply:* the server is returning a 500 ?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2023-02-16 19:41:57
 +
+

*Thread Reply:* odd that LoggingMdcFilter is logging 200

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
thebruuu + (bruno.c@inwind.it) +
+
2023-02-16 19:43:24
 +
+

*Thread Reply:* Bit confused because now I realize that postman is returning bad request

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
thebruuu + (bruno.c@inwind.it) +
+
2023-02-16 19:43:51
 +
+

*Thread Reply:*

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
thebruuu + (bruno.c@inwind.it) +
+
2023-02-16 19:44:30
 +
+

*Thread Reply:* You'll notice that I go to use 3000 in the url +If I use 5000 I get No host

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2023-02-17 01:14:50
 +
+

*Thread Reply:* odd, the API should be using port 5000, have you followed our quickstart for Marquez?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
thebruuu + (bruno.c@inwind.it) +
+
2023-02-17 03:43:29
 +
+

*Thread Reply:* Hello Willy +I am starting from scratch followin instruction from https://openlineage.io/docs/getting-started/ +I am on Windows +Instead of +git clone git@github.com:MarquezProject/marquez.git && cd marquez +I run the +git clone

+ +

git clone <https://github.com/MarquezProject/marquez.git> +But before I had to clear the auto carriage return in git +git config --global core.autocrlf false +This avoid an error message on marquez-api when running wait-for-it.sh àt line 1 where +#!/usr/bin/env bash +is otherwise read as +#!/usr/bin/env bash\r'

+ +

It turns out that when switching off the autocr, this impacts some file containing marquez password ... and I get a fail on accessing the db +to overcome this I run notepad++ and replaced ALL the \r\n with \n +And in this way I managed to run +docker\up.sh and docker\down.sh +correctly (with or without seed ... with access to the db, via pgadmin)

+ + + +
+ 👍 Ernie Ostic +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
thebruuu + (bruno.c@inwind.it) +
+
2023-02-20 03:40:48
 +
+

*Thread Reply:* The issue is related to PostMan

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-02-17 03:39:07
 +
+

Hi, I'd like to capture column lineage from spark, but also capture how the columns are transformed, and any column operations that are done too. May I ask if this feature is supported currently, or will be supported in future based on current timeline? Thanks!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-02-17 03:54:47
 +
+

*Thread Reply:* Hi @Anirudh Shrinivason, this is a great question. We included extra fields in OpenLineage spec to contain that information: +"transformationDescription": { + "type": "string", + "description": "a string representation of the transformation applied" +}, +"transformationType": { + "type": "string", + "description": "IDENTITY|MASKED reflects a clearly defined behavior. IDENTITY: exact same as input; MASKED: no original data available (like a hash of PII for example)" +} +so the standard is ready to support it. We included two fields, so that one can contain human readable description of what is happening. However, we don't have this implemented in Spark integration.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-02-17 04:02:30
 +
+

*Thread Reply:* Thanks a lot! That is great. Is there a potential plan in the roadmap to support this for spark?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-02-17 04:08:16
 +
+

*Thread Reply:* I think there will be a growing interest in that. In general a dependency may really difficult to express if many Spark operators are used on input columns to produce output one. The simple version would be just to detect indetity operation or some kind of hashing.

+ +

To sum up, we don't have yet a proposal on that but this seems to be a natural next step in enriching column lineage features.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-02-17 04:40:04
 +
+

*Thread Reply:* Got it. Thanks! If this item potentially comes on the roadmap, then I'd be happy to work with other interested developers to help contribute! 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-02-17 04:43:00
 +
+

*Thread Reply:* Great to hear that. What you could perhaps start with, is come to our monthly OpenLineage meetings and ask @Michael Robinson to put this item on discussions' list. There are many strategies to address this issue and hearing your story, usage scenario and would are you trying to achieve, would be super helpful in design and implementation phase.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-02-17 04:44:18
 +
+

*Thread Reply:* Got it! The monthly meeting might be a bit hard for me to attend live, because of the time zone. But I'll try my best to make it to the next one! thanks!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-02-17 09:46:22
 +
+

*Thread Reply:* Thank you for bringing this up, @Anirudh Shrinivason. I’ll add it to the agenda of our next meeting because there might be interest from others in adding this to the roadmap.

+ + + +
+ 👍 Anirudh Shrinivason +
+ +
+ :gratitude_thank_you: Anirudh Shrinivason +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
thebruuu + (bruno.c@inwind.it) +
+
2023-02-17 15:12:57
 +
+

Hello +how can I improve the verbosity of the marquez-api? +Regards

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-02-20 02:10:13
 +
+

*Thread Reply:* Hi @thebruuu, pls take a look at logging documentation of Dropwizard (https://www.dropwizard.io/en/latest/manual/core.html#logging) - the framework Marquez is implemented in. The logging configuration section is present in marquez.yml .

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
thebruuu + (bruno.c@inwind.it) +
+
2023-02-20 03:29:07
 +
+

*Thread Reply:* Thank You Pavel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-02-21 02:23:40
 +
+

Hey, can we please schedule a release of OpenLineage? I would like to have the release that includes the feature to capture custom env variables from spark clusters... Thanks!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-02-21 09:12:17
 +
+

*Thread Reply:* We generally schedule a release every month, next one will be in the next week - is that okay @Anirudh Shrinivason?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-02-21 11:38:50
 +
+

*Thread Reply:* Yes, there’s one scheduled for next Wednesday, if that suits.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-02-21 21:45:58
 +
+

*Thread Reply:* Okay yeah sure that works. Thanks

+ + + +
+ 🙌 Michael Robinson +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-03-01 10:12:45
 +
+

*Thread Reply:* @Anirudh Shrinivason we’re expecting the release to happen today or tomorrow, FYI

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-03-01 21:22:40
 +
+

*Thread Reply:* Awesome thanks

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jingyi Chen + (jingyi@cloudshuttle.com.au) +
+
2023-02-23 23:43:23
 +
+

Hello team, we used OpenLineage and Great Expectations integrated. I want to use GE to verify the table in Snowflake. I found that the configuration I added OpenLineage into GE produced this error after running. Could someone please give me some answers? 👀 +File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/great_expectations/validation_operators/validation_operators.py", line 469, in _run_actions + action_result = self.actions[action["name"]].run( + File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/great_expectations/checkpoint/actions.py", line 106, in run + return self._run( + File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/openlineage/common/provider/great_expectations/action.py", line 156, in _run + datasets = self._fetch_datasets_from_sql_source( + File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/openlineage/common/provider/great_expectations/action.py", line 362, in _fetch_datasets_from_sql_source + self._get_sql_table( + File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/openlineage/common/provider/great_expectations/action.py", line 395, in _get_sql_table + if engine.connection_string: +AttributeError: 'Engine' object has no attribute 'connection_string' +'Engine' object has no attribute 'connection_string'

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jingyi Chen + (jingyi@cloudshuttle.com.au) +
+
2023-02-23 23:44:03
 +
+

*Thread Reply:* This is my checkponit configuration in GE. +```name: 'openlineagecheckpoint' +configversion: 1.0 +templatename: +modulename: greatexpectations.checkpoint +classname: Checkpoint +runnametemplate: '%Y%m%d-%H%M%S-mycheckpoint' +expectationsuitename: EMAILVALIDATION +batchrequest: +actionlist:

  • name: storevalidationresult +action: + class_name: StoreValidationResultAction
  • name: storeevaluationparams +action: + class_name: StoreEvaluationParametersAction
  • name: updatedatadocs +action: + classname: UpdateDataDocsAction + sitenames: []
  • name: openlineage +action: + classname: OpenLineageValidationAction + modulename: openlineage.common.provider.greatexpectations + openlineagehost: http://localhost:5000 + # openlineageapiKey: 12345 + openlineagenamespace: geexpectations # Replace with your job namespace; we recommend a meaningful namespace like dev or prod, etc. + jobname: gevalidation +evaluationparameters: {} +runtime_configuration: {} +validations:
  • batchrequest: + datasourcename: LANDINGDEV + dataconnectorname: defaultinferreddataconnectorname + dataassetname: 'snowpipe.pii' + dataconnectorquery: + index: -1 +expectationsuitename: EMAILVALIDATION
    • +
+ +

profilers: [] +gecloudid: +expectationsuitegecloudid:```

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Benji Lampel + (benjamin@astronomer.io) +
+
2023-02-24 11:31:05
 +
+

*Thread Reply:* What version of GX are you running? And is this being run directly through GX or through Airflow with the operator?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jingyi Chen + (jingyi@cloudshuttle.com.au) +
+
2023-02-26 20:05:12
 +
+

*Thread Reply:* I use the latest version of Great Expectations. This error occurs either directly through Great Expectations or airflow

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Benji Lampel + (benjamin@astronomer.io) +
+
2023-02-27 09:10:00
 +
+

*Thread Reply:* I noticed another issue in the latest version as well. Try dropping to GE version great-expectations==0.15.44 for now. That is the latest one that works for me.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Benji Lampel + (benjamin@astronomer.io) +
+
2023-02-27 09:11:34
 +
+

*Thread Reply:* You should definitely open an issue here, and you can tag me @denimalpaca in the comment

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jingyi Chen + (jingyi@cloudshuttle.com.au) +
+
2023-02-27 18:07:29
 +
+

*Thread Reply:* Thanks Benji, but I still have the same problem after I drop to great-expectations==0.15.44 , this is my requirement file

+ +
great_expectations==0.15.44
+sqlalchemy
+psycopg2-binary
+numpy
+pandas
+snowflake-connector-python
+snowflake-sqlalchem
+
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Benji Lampel + (benjamin@astronomer.io) +
+
2023-02-28 13:34:03
 +
+

*Thread Reply:* interesting... I do think this may be a GX issue so let's see if they say anything. I can also cross post this thread to their slack

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Saravanan + (saravanan@athivatech.com) +
+
2023-03-01 00:27:30
 +
+

Hello Team, I’m trying to use Open Lineage with AWS Glue and Marquez. Has anyone successfully integrated AWS Workflows/ Glue ETL jobs with Open Lineage?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sheeri Cabral (Collibra) + (sheeri.cabral@collibra.com) +
+
2023-05-01 11:47:40
 +
+

*Thread Reply:* I know I’m responding to an older post - I’m not sure if this would work in your environment? https://aws.amazon.com/blogs/big-data/build-data-lineage-for-data-lakes-using-aws-glue-amazon-neptune-and-spline/ +Are you using AWS Glue with Spark jobs?

+
+
Amazon Web Services
+ + + +
Build data lineage for data lakes using AWS Glue, Amazon Neptune, and Spline | Amazon Web Services
+
+ Data lineage is one of the most critical components of a data governance strategy for data lakes. Data lineage helps ensure that accurate, complete and trustworthy data is being used to drive business decisions. While a data catalog provides metadata management features and search capabilities, data lineage shows the full context of your data by […] +
+ + + + + + + + +
Original URL: https://aws.amazon.com/blogs/big-data/build-data-lineage-for-data-lakes-using-aws-glue-amazon-neptune-and-spline/
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Saravanan + (saravanan@athivatech.com) +
+
2023-05-02 15:16:14
 +
+

*Thread Reply:* This was proposed by our AWS Solution architect but we are not seeing much improvement compared to open lineage. Have you deployed the above solution to prod?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sheeri Cabral (Collibra) + (sheeri.cabral@collibra.com) +
+
2023-05-25 11:30:44
 +
+

*Thread Reply:* We are currently in the research phase, so we have not deployed to prod. We have customers with thousands of existing scripts that they don’t want to rewrite to add openlineage libraries - i would imagine that if you are already integrating OpenLineage in your code, the spark listener isn’t an improvement. Our research is on magically getting lineage from existing scripts 😄

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-03-01 09:42:23
 +
+

Hello everyone, I’m opening a vote to release OpenLineage 0.21.0, featuring: +• a new CustomEnvironmentFacetBuilder class and new output visitors AlterTableAddPartitionCommandVisitor and AlterTableSetLocationCommandVisitor in the Spark integration +• a Linux-ARM version of the SQL parser’s native library +• DEBUG logging of events in transports +• bug fixes and more. +Three +1s from committers will authorize an immediate release.

+ + + +
+ ➕ Maciej Obuchowski, Jakub Dardziński, Benji Lampel, Natalie Zeller, Paweł Leszczyński +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-03-01 10:26:22
 +
+

*Thread Reply:* Thanks, all. The release is authorized and will be initiated as soon as possible.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Nigel Jones + (nigel.l.jones@gmail.com) +
+
2023-03-02 03:52:03
 +
+

I’ve got some security related questions/observations. The main site suggests opening an issue to report vulnerabilities etc. I wanted to check if there is a private mailing list/DM channel to just check a few things first? I’m happy to use github issues otherwise. Thanks!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Moritz E. Beber + (midnighter@posteo.net) +
+
2023-03-02 05:15:55
 +
+

*Thread Reply:* GitHub has a new issue template for reporting vulnerabilities, actually. If you use a config that enables this issue template.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-03-02 10:21:16
 +
+

Reminder: our first meetup is one week from today in Providence, RI! You can find the details in the meetup blog post. And if you’re coming, it would be great if you could RSVP. Looking forward to seeing some of you there!

+
+
openlineage.io
+ + + +
Happening Soon - Our First Meetup!
+
+ The inaugural Data Lineage Meetup will take place on March 9th in Providence, RI. +
+ + + + + + +
Original URL: https://openlineage.io/blog/data-lineage-meetup/
+ + + +
+
+
Meetup
+ + + +
Data Lineage Meetup, Thu, Mar 9, 2023, 6:00 PM | Meetup
+
+ Join us on Thursday, March 9, 2023 from 6-8 pm in Providence, Rhode Island to learn more about the present and future of OpenLineage. Meet other members of the ecosystem, l +
+ + + + + + + + +
Original URL: https://www.meetup.com/providence-data-lineage-meetup/events/291847062/
+ + + +
+ + + +
+ 🎉 Kengo Seki +
+ +
+ 🚀 Kengo Seki +
+ +
+ ✅ Sheeri Cabral (Collibra) +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-03-02 16:52:50
 +
+

@channel +We released OpenLineage 0.21.1, including: +Additions +• Clients: add DEBUG logging of events to transports #1633 by @mobuchowski +• Spark: add CustomEnvironmentFacetBuilder class #1545 by New contributor @Anirudh181001 +• Spark: introduce the new output visitors AlterTableAddPartitionCommandVisitor and AlterTableSetLocationCommandVisitor #1629 by New contributor @nataliezeller1 +• Spark: add column lineage for JDBC relations #1636 by @tnazarew +• SQL: add linux-aarch64 native library to Java SQL parser #1664 by @mobuchowski +Changes +• Airflow: get table database in Athena extractor #1631 by New contributor @rinzool +Removals +• Airflow: remove JobIdMapping and update macros to better support Airflow version 2+ #1645 by @JDarDagran +Thanks to all our contributors! +For the bug fixes and details, see: +Release: https://github.com/OpenLineage/OpenLineage/releases/tag/0.21.1 +Changelog: https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md +Commit history: https://github.com/OpenLineage/OpenLineage/compare/0.20.6...0.21.1 +Maven: https://oss.sonatype.org/#nexus-search;quick~openlineage +PyPI: https://pypi.org/project/openlineage-python/

+ + + +
+ 🎉 Kengo Seki, Harel Shein, Maciej Obuchowski +
+ +
+ 🚀 Kengo Seki, Harel Shein, Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paul Lee + (paullee@lyft.com) +
+
2023-03-02 19:01:23
 +
+

how do you turn off the openlineage listener in airflow 2? for some reason we're seeing a Thread-2 and seeing it fire twice in tasks

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harel Shein + (harel.shein@gmail.com) +
+
2023-03-02 20:04:19
 +
+

*Thread Reply:* Hey @Paul Lee, are you seeing this happen for Async operators?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harel Shein + (harel.shein@gmail.com) +
+
2023-03-02 20:06:00
 +
+

*Thread Reply:* might be related to this issue https://github.com/OpenLineage/OpenLineage/pull/1601 +that was fixed in 0.20.6

+
+ + + + +
#1601 Airflow: make extractor for async operators work
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paul Lee + (paullee@lyft.com) +
+
2023-03-03 16:15:44
 +
+

*Thread Reply:* hmm perhaps.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paul Lee + (paullee@lyft.com) +
+
2023-03-03 16:15:55
 +
+

*Thread Reply:* @Harel Shein if i want to turn off openlineage listener how do i do that? do i just remove the package?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harel Shein + (harel.shein@gmail.com) +
+
2023-03-03 16:24:07
 +
+

*Thread Reply:* meaning, you don’t want openlineage to collect any information from your Airflow deployment?

+ + + +
+ 👍 Paul Lee +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harel Shein + (harel.shein@gmail.com) +
+
2023-03-03 16:24:50
 +
+

*Thread Reply:* in that case, you could either remove it from your requirements file, or set OPENLINEAGE_DISABLED=True in your Airflow env vars

+ + + +
+ 👍 Paul Lee +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paul Lee + (paullee@lyft.com) +
+
2023-03-06 14:43:56
 +
+

*Thread Reply:* removed it from requirements and also the backend key in airflow config. needed both

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-03-02 20:29:42
 +
+

@channel +This month’s OpenLineage TSC meeting is next Thursday, March 9th, at 10 am PT. Join us on Zoom: https://bit.ly/OLzoom. All are welcome! +On the tentative agenda:

+ +
  1. Recent release overview
  2. A new consumer
  3. Custom env variable support in Spark
  4. Async operator support in Airflow
  5. JDBC relations support in Spark
  6. Discussion topics: +• New feature idea: column transformations/operations in the Spark integration +• Using namespaces
  7. Open discussion +Notes: https://bit.ly/OLwiki +Is there a topic you think the community should discuss at this or a future meeting? Reply or DM me to add items to the agenda.
    8. +
+ + + +
+ 🙌 Willy Lulciuc, Paweł Leszczyński, Maciej Obuchowski, alexandre bergere +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-03-02 21:48:29
 +
+

Hi everyone, I noticed that Openlineage is sending each of the events twice for spark. Is this expected? Is there some way to disable this behaviour?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2023-03-02 23:46:08
 +
+

*Thread Reply:* Are you seeing duplicate START events or do you see two events one that is a START and one that is COMPLETE?

+ +

OpenLineage's events may send partial information. You should expect to collect all events for a given RunId and merge them together to get the complete events.

+ +

In addition, some data sources are really chatty like Delta tables. That may cause you to see many events that look very similar.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-03-03 00:45:19
 +
+

*Thread Reply:* Hmm...I'm seeing 2 start events for the same runnable command

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-03-03 00:45:27
 +
+

*Thread Reply:* And 2 complete

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-03-03 00:46:08
 +
+

*Thread Reply:* I am currently only testing on parquet tables...

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-03-03 02:31:28
 +
+

*Thread Reply:* One of openlineage assumptions is the ability to merge lineage events in the backend to make client integrations stateless. So, it is possible that Spark can emit multiple events for the same job. However, sometimes it does not make any sense to send or collect some events, which happened to us some time ago with delta. In that case we decided to filter them and created filtering mechanism (https://github.com/OpenLineage/OpenLineage/tree/main/integration/spark/shared/src/main/java/io/openlineage/spark/agent/filters) than can be extended in case of other unwanted events being generated and sent.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-03-05 22:59:06
 +
+

*Thread Reply:* Ahh I see...okay thanks!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Daniel Joanes + (djoanes@gmail.com) +
+
2023-03-07 00:05:48
 +
+

*Thread Reply:* in general , you should build any event consumer system with at least once semantics. Even if this issue is fixed, there is a possibility of duplicates for other valid scenarios

+ + + +
+ ➕ Maciej Obuchowski, Anirudh Shrinivason +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-03-09 14:10:47
 +
+

*Thread Reply:* Hi..I compared some duplicate 'START' events just now, and noticed that they are exactly the same, with the only exception of one of them having an 'environment-properties' field... Could I just quickly check if this is a bug or a feature haha?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-03-10 01:18:18
 +
+

*Thread Reply:* CC: @Paweł Leszczyński ^

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-03-08 11:15:48
 +
+

@channel +Reminder: this month’s OpenLineage TSC meeting is tomorrow at 10am PT. All are welcome. https://openlineage.slack.com/archives/C01CK9T7HKR/p1677806982084969

+
+ + +
+ + + } + + Michael Robinson + (https://openlineage.slack.com/team/U02LXF3HUN7) +
+ + + +
+
+ <p><strong>@channel</strong> +This month’s OpenLineage TSC meeting is <strong>next Thursday, March 9th, at 10 am PT</strong>. Join us on Zoom: <a href="https://bit.ly/OLzoom">https://bit.ly/OLzoom</a>. All are welcome! +On the tentative agenda:</p> + +<ol><li>Recent release overview </li><li>A new consumer</li><li>Discussion topics: +• New feature idea: column transformations/operations in the Spark integration +• Using namespaces</li><li>Open discussion +Notes: <a href="https://bit.ly/OLwiki">https://bit.ly/OLwiki</a> +Is there a topic you think the community should discuss at this or a future meeting? Reply or DM me to add items to the agenda.</li> +</ol> +
+ + + + + + +
Original URL: https://openlineage.slack.com/archives/C01CK9T7HKR/p1677806982084969
+ + +
+ + Slack Conversation +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Susmitha Anandarao + (susmitha.anandarao@gmail.com) +
+
2023-03-08 15:51:07
 +
+

Hi if we have OpenLineage listener configured as a default spark conf, is there an easy way to disable ol for a specific notebook?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-03-08 17:30:44
 +
+

*Thread Reply:* if you can set up env variables for particular notebooks, you can set OPENLINEAGE_DISABLED=true

+ + + +
+ :gratitude_thank_you: Susmitha Anandarao +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Benji Lampel + (benjamin@astronomer.io) +
+
2023-03-10 13:15:41
 +
+

Hey all,

+ +

I opened a PR (and corresponding issue) to change how naming works in OpenLineage. The idea generally is to move from Naming.md as the end-all-be-all of names for integrations, and towards JSON schemas per integration, with each schema defining very precisely what fields a name and namespace should contain, how they're connected, and how they're validated. Would really appreciate some feedback as this is a pretty big change!

+
+ + + + +
#1702 Add proposal for new naming schemas.
+
+ <p>Closes: <a href="https://github.com/OpenLineage/OpenLineage/issues/1681">#1681</a></p> + +<p><strong>Checklist</strong></p> + +<p>☑︎ You've <a href="https://github.com/OpenLineage/OpenLineage/blob/main/why-the-dco.md">signed-off</a> your work +☑︎ Your pull request title follows our <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CONTRIBUTING.md#creating-pull-requests">guidelines</a> +☐ Your changes are accompanied by tests (<em>if relevant</em>) +☐ Your change contains a <a href="https://kurtisnusbaum.medium.com/stacked-diffs-keeping-phabricator-diffs-small-d9964f4dcfa6">small diff</a> and is self-contained +☐ You've updated any relevant documentation (<em>if relevant</em>) +☐ Your comment includes a one-liner for the changelog about the specific purpose of the change (<em>if necessary</em>) +☐ You've versioned the core OpenLineage model or facets according to <a href="https://docs.snowplowanalytics.com/docs/pipeline-components-and-applications/iglu/common-architecture/schemaver">SchemaVer</a> (<em>if relevant</em>) +☐ You've added a <a href="https://github.com/OpenLineage/OpenLineage/tree/main/.github/header_templates.md">header</a> to source files (<em>if relevant</em>)</p> + +<hr /> + +<p>SPDX-License-Identifier: Apache-2.0 <br /> +Copyright 2018-2023 contributors to the OpenLineage project</p> +
+ +
+
Labels
+ documentation, proposal +
+ +
+
Comments
+ 1 +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sunil Patil + (spatil@twilio.com) +
+
2023-03-13 17:05:56
 +
+

What do i need to do to enable dag level metric capturing for airflow. I followed the instruction to install openlineage 0.21.1 on airflow 2.3.3. When i run a DAG i see metrics related to Task start, success/failure. But i dont see any metrics for Dag success/failure. Do i have to do something to enable DAG execution capturing ?

+
+
openlineage.io
+ + + +
Using OpenLineage with Airflow | OpenLineage Docs
+
+ This tutorial introduces you to using OpenLineage with Airflow. OpenLineage has various integrations that will enable airflow to emit OpenLineage events when using Airflow Integrations. In this tutorial, you will be running a local instance of Airflow using docker compose, and learn how to enable and setup OpenLineage to emit data lineage events. The tutorial will use two backends to check the data lineage, 1) using OpenLineage Proxy, and 2) using Marquez. +
+ + + + + + +
Original URL: https://openlineage.io/docs/guides/airflow
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sunil Patil + (spatil@twilio.com) +
+
2023-03-13 17:08:53
 +
+

*Thread Reply:* is DAG run capturing enabled starting airflow 2.5.1 ? https://github.com/apache/airflow/pull/27113

+
+ + + + +
#27113 notification: add dag run state notification system
+
+ <p>This PR expands <code>listeners</code> API to be notified about DagRun state changes.</p> + +<p>PR <a href="https://github.com/apache/airflow/pull/20443">#20443</a> introduced <code>Listener API</code> to Airflow, enabling users and developers to write <code>Pluggy</code> plugins. Here the same mechanism is used to provide notifications about <code>DagRun</code> states - whether <code>DagRun</code> started, succeeded or failed.</p> + +<p>Additionally, this PR adds lifecycle methods for plugins - when Airflow <code>Job</code> - whether <code>SchedulerJob</code>, <code>BackfillJob</code> or <code>LocalTaskJob</code> runs, it notifies plugin that it starts or finishes.</p> + +<p>Signed-off-by: Maciej Obuchowski <a href="mailto:obuchowski.maciej@gmail.com">obuchowski.maciej@gmail.com</a></p> +
+ +
+
Labels
+ area:scheduler/executor, type:new-feature +
+ +
+
Comments
+ 6 +
+ + + + + + + +
+ + <a href="https://github.com/apache/airflow">apache/airflow</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-03-13 17:11:47
 +
+

*Thread Reply:* you're right, only the change was included in 2.5.0

+ + + +
+ 🙏 Sunil Patil +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sunil Patil + (spatil@twilio.com) +
+
2023-03-13 17:43:15
 +
+

*Thread Reply:* Thanks Jakub

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-03-14 15:37:34
 +
+

Fresh on the heels of our first-ever in-person event, we’re meeting up again soon at Data Council Austin! Join us on March 30th (the same day as @Julien Le Dem’s talk) at 12:15 pm to discuss the project’s goals and design, meet other members of the data ecosystem, and help shape the future of the spec. For more info, check out the OpenLineage blog. If you haven’t registered for the conference yet, click and use promo code OpenLineage20 for a special rate. Hope to see you there!

+
+
openlineage.io
+ + + +
Meet Us at Data Council Austin
+
+ The OpenLineage community will be gathering on March 30th at Data Council Austin -- join us! +
+ + + + + + +
Original URL: https://openlineage.io/blog/data-council-meetup/
+ + + +
+
+
tickettailor.com
+ + + +
Buy tickets – Data Council Austin 2023 – AT&amp;T Hotel and Conference Center, Tue Mar 28, 2023 8:00 AM - Thu Mar 30, 2023 5:00 PM
+
+ Data Council Austin 2023 – AT&amp;T Hotel and Conference Center, Tue Mar 28, 2023 - Thu Mar 30, 2023 - Data Council US 2023 in Austin, TX is where data's most influential makers, doers, and thinkers gather to shape the future of data innovation at every layer of the stack. If you’re a technical professional who wants to learn about new tools... +
+ + + + + + + + +
Original URL: https://www.tickettailor.com/events/datacouncil/747883?hsCtaTracking=dc0af239-1ec5-4c9f-a175-f487bb858074%7Ca9aa7625-43fb-436d-b05d-9666bab6e414&amp;_ga=2.239870401.316757827.1677608801-1529956930.1676647160
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sheeri Cabral (Collibra) + (sheeri.cabral@collibra.com) +
+
2023-03-15 15:11:18
 +
+

If someone is using airflow and DAG-docs for lineage, can they export the lineage in, say, OL format?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harel Shein + (harel.shein@gmail.com) +
+
2023-03-15 15:18:22
 +
+

*Thread Reply:* I don’t see it currently on the AirflowRunFacet, but probably not a big deal to add it? @Benji Lampel wdyt?

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/integration/airflow/facets/AirflowRunFacet.json | AirflowRunFacet.json>
+
+ <pre><code>// TODO: that is Airflow specific facet that should be placed in new system-specific directory +{ + "$schema": "&lt;https://json-schema.org/draft/2020-12/schema&gt;", + "$id": "&lt;https://openlineage.io/spec/facets/1-0-0/AirflowRunFacet.json&gt;", + "$defs": { + "AirflowRunFacet": { + "allOf": [ + { + "$ref": "&lt;https://openlineage.io/spec/1-0-3/OpenLineage.json#/$defs/RunFacet&gt;" + }, + { + "type": "object", + "properties": { + "task": { + "$ref": "#/$defs/Task" + }, + "dag": { + "$ref": "#/$defs/DAG" + }, + "taskInstance": { + "$ref": "#/$defs/TaskInstance" + }, + "dagRun": { + "$ref": "#/$defs/DagRun" + }, + "taskUuid": { + "type": "string" + } + }, + "required": ["task", "dag", "taskInstance", "dagRun", "taskUuid"] + } + ] + }, + "Task": { + "type": "object", + "description": "An object containing task-related attributes", + "properties": { + "operator_class": { + "description": "Module + class name of the operator", + "type": "string" + }, + "email": { + "type": "string" + }, + "args": { + "type": "object", + "additionalProperties": true + }, + "downstream_group_ids": { + "type": "string" + }, + "downstream_task_ids": { + "type": "string" + }, + "upstream_group_ids": { + "type": "string" + }, + "upstream_task_ids": { + "type": "string" + }, + "task_group": { + "description": "Task group related information", + "type": "object", + "properties": { + "children": { + "type": "object", + "additionalProperties": true + }, + "downstream_group_ids": { + "type": "string" + }, + "downstream_task_ids": { + "type": "string" + }, + "prefix_group_id": { + "type": "boolean" + }, + "tooltip": { + "type": "string" + }, + "upstream_group_ids": { + "type": "string" + }, + "upstream_task_ids": { + "type": "string" + } + }, + "additionalProperties": true + } + }, + "required": ["operator_class"] + }, + "DAG": { + "type": "object", + "description": "A DAG-attributes representing object", + "properties": { + "dag_id": { + "type": "string" + }, + "schedule_interval": { + "type": "string" + }, + "tags": { + "type": "string" + }, + "start_date": { + "type": "string", + "format": "date-time" + }, + "timetable": { + "description": "Describes timetable (successor of schedule_interval)", + "type": "object", + "additionalProperties": true + } + }, + "additionalProperties": true, + "required": ["dag_id", "schedule_interval", "tags", "start_date"] + }, + "TaskInstance": { + "type": "object", + "properties": { + "duration": { + "type": "number" + }, + "try_number": { + "type": "integer" + }, + "map_index": { + "type": "integer" + } + }, + "additionalProperties": true, + "required": ["duration", "try_number"] + }, + "DagRun": { + "type": "object", + "properties": { + "conf": { + "type": "object", + "additionalProperties": true + }, + "dag_id": { + "type": "string" + }, + "data_interval_start": { + "type": "string", + "format": "date-time" + }, + "data_interval_end": { + "type": "string", + "format": "date-time" + }, + "run_id": { + "type": "string" + }, + "run_type": { + "type": "string" + }, + "external_trigger": { + "type": "boolean" + } + }, + "additionalProperties": true, + "required": ["dag_id", "external_trigger"] + } + }, + "type": "object", + "properties": { + "airflow": { + "$ref": "#/$defs/AirflowRunFacet" + } + } +} + +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+ ❤️ Sheeri Cabral (Collibra) +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Benji Lampel + (benjamin@astronomer.io) +
+
2023-03-15 15:22:00
 +
+

*Thread Reply:* Definitely could be a good thing to have--is there not some info facet that could hold this data already? I don't see an issue with adding to the AirflowRunFacet tho (full disclosure, I'm not super familiar with this facet)

+ + + +
+ ❤️ Sheeri Cabral (Collibra) +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2023-03-15 15:58:40
 +
+

*Thread Reply:* Perhaps DocumentationJobFacet or DocumentationDatasetFacet?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sheeri Cabral (Collibra) + (sheeri.cabral@collibra.com) +
+
2023-03-15 15:13:55
 +
+

(is it https://docs.astronomer.io/learn/airflow-openlineage ? )

+
+
docs.astronomer.io
+ + + +
Integrate OpenLineage and Airflow | Astronomer Documentation
+
+ Learn about OpenLineage concepts and benefits of integrating with Airflow. +
+ + + + + + + + +
Original URL: https://docs.astronomer.io/learn/airflow-openlineage
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Susmitha Anandarao + (susmitha.anandarao@gmail.com) +
+
2023-03-17 12:31:02
 +
+

Happy Friday 👋 I am looking for some help setting the parent information for a dbt run. I have set the namespace variable in the openlineage.yml but doesn't seem to take effect and ends up using the default value of dbt. Also using openlineage.yml to set the transport properties for emitting to kafka. Is there a way to set parent namespace, name and run id in the yml file? Thank you!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-03-18 12:09:23
 +
+

*Thread Reply:* dbt-ol does not read from openlineage.yml so you need to pass this information in OPENLINEAGE_NAMESPACE environment variable

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2023-03-20 15:17:03
 +
+

*Thread Reply:* Hmmm. Interesting! I thought that it used client = OpenLineageClient.from_environment(), I’ll do some testing with Kafka backends.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Susmitha Anandarao + (susmitha.anandarao@gmail.com) +
+
2023-03-20 15:22:07
 +
+

*Thread Reply:* Thank you for the hint. I was able to make it work with specifying the env OPENLINEAGE_CONFIGto specify the yml file holding transport info and OPENLINEAGE_NAMESPACE

+ + + +
+ 👍 Ross Turk +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2023-03-20 15:24:05
 +
+

*Thread Reply:* Awesome! That’s exactly what I was going to test.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2023-03-20 15:25:04
 +
+

*Thread Reply:* I think it also works if you put it in $HOME/.openlineage/openlineage.yml.

+ + + +
+ :gratitude_thank_you: Susmitha Anandarao +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-03-21 08:32:17
 +
+

*Thread Reply:* @Susmitha Anandarao I might have provided misleading information. I meant that dbt-ol does not read OL namespace from openlineage.yml but from OPENLINEAGE_NAMESPACE env var instead

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-03-21 13:48:28
 +
+

Data Council Austin, the host of our next meetup, is one week away: https://openlineage.slack.com/archives/C01CK9T7HKR/p1678822654288379

+
+ + +
+ + + } + + Michael Robinson + (https://openlineage.slack.com/team/U02LXF3HUN7) +
+ + + +
+
+ <p>Fresh on the heels of our first-ever in-person event, we’re meeting up again soon at <a href="https://www.datacouncil.ai/austin">Data Council Austin</a>! Join us on March 30th (the same day as @Julien Le Dem’s <a href="https://www.datacouncil.ai/talks/ten-years-of-building-open-source-standards?hsLang=en">talk</a>) at 12:15 pm to discuss the project’s goals and design, meet other members of the data ecosystem, and help shape the future of the spec. For more info, check out the OpenLineage <a href="https://openlineage.io/blog/data-council-meetup/">blog</a>. If you haven’t registered for the conference yet, click <https://www.tickettailor.com/events/datacouncil/747883?hsCtaTracking=dc0af239-1ec5-4c9f-a175-f487bb858074%7Ca9aa7625-43fb-436d-b05d-9666bab6e414&amp;_ga=2.239870401.316757827.1677608801-1529956930.1676647160|this link> and use promo code <code>OpenLineage20</code> for a special rate. Hope to see you there!</p> +
+ + + + + + +
Original URL: https://openlineage.slack.com/archives/C01CK9T7HKR/p1678822654288379
+ + +
+ + Slack Conversation +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-03-21 13:52:52
 +
+

In addition to Data Council Austin next week, the hybrid Big Data Technology Warsaw Summit will be taking place on March 28th-30th, featuring three of our committers: @Maciej Obuchowski, @Paweł Leszczyński and @Ross Turk ! There’s more info here: https://bigdatatechwarsaw.eu/

+
+
Big Data Technology Warsaw Summit
+ + + +
BDTWS2023 - Big Data Technology Warsaw Summit
+
+ Big Data Tech Warsaw 2023 is one of the largest Big Data conferences in Poland and Eastern Europe +
+ +
+
Estimated reading time
+ 6 minutes +
+ + + + + + + + +
Original URL: https://bigdatatechwarsaw.eu/
+ + + +
+ + + +
+ 🙌 Howard Yoo, Maciej Obuchowski, Jakub Dardziński, Ross Turk, Perttu Salonen +
+ +
+ 👍 thebruuu +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Brad Paskewitz + (bradford.paskewitz@fivetran.com) +
+
2023-03-22 22:38:26
 +
+

hey folks, is anyone capturing dataset metadata for multi-table schemas? I'm looking at the schema dataset facet: https://openlineage.io/docs/spec/facets/dataset-facets/schema but it looks like this only represents a single table so im wondering if I'll need to write a custom facet

+
+
openlineage.io
+ + + +
Schema Dataset Facet | OpenLineage Docs
+
+ The schema dataset facet contains the schema of a particular dataset. +
+ + + + + + +
Original URL: https://openlineage.io/docs/spec/facets/dataset-facets/schema
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-03-23 04:25:19
 +
+

*Thread Reply:* It should be represented by multiple datasets, unless I misunderstood what you mean by multi-table

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Brad Paskewitz + (bradford.paskewitz@fivetran.com) +
+
2023-03-23 10:55:58
 +
+

*Thread Reply:* here at Fivetran when we sync data it is generally 1 schema with multiple tables (sometimes many) so we would want to represent all of that

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-03-23 11:11:25
 +
+

*Thread Reply:* So what I understand:

+ +
  1. your single job represents synchronization of multiple tables
  2. you want to have precise input-output dataset lineage? +am I right?
    3. +
+ +

I would model that as multiple OL jobs that describe each dataset mappings. Additionally, I'd have one "wrapping" job that represents your definition of a job. Rest of those jobs would refer to it in ParentRunFacet.

+ +

This is a pattern we use for Airflow and dbt dags.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Brad Paskewitz + (bradford.paskewitz@fivetran.com) +
+
2023-03-23 12:57:15
 +
+

*Thread Reply:* Yes your statements are correct. Thanks for sharing that model, that makes sense to me

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Brad Paskewitz + (bradford.paskewitz@fivetran.com) +
+
2023-03-24 15:56:27
 +
+

has anyone had success creating custom facets using java? I'm following this guide: https://openlineage.io/docs/spec/facets/custom-facets and im wondering if it makes sense to manually create POJOs or if others are creating the json schema for the object and then automatically generating the java code?

+
+
openlineage.io
+ + + +
Custom Facets | OpenLineage Docs
+
+ In addition to the existing facets mentioned in this documentation, users can extend the base facets and provide their own facet definition as part of the payload in OpenLineage event. For example, when OpenLineage event is emitted from the Apache Airflow using OpenLineage's Airflow integration, the following facets can be observed: +
+ + + + + + +
Original URL: https://openlineage.io/docs/spec/facets/custom-facets
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-03-27 05:26:06
 +
+

*Thread Reply:* I think it's better to just create POJO. This is what we do in Spark integration, for example.

+ +

For now, JSON Schema generator isn't flexible enough to generate custom facets from whatever schema we give it, so it would be unnecessary complexity

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/ce5874ca86a5101720d4cb3b9bb195463b13438a/integration/spark/shared/src/main/java/io/openlineage/spark/agent/facets/EnvironmentFacet.java | EnvironmentFacet.java>
+
+ <pre><code>public class EnvironmentFacet extends OpenLineage.DefaultRunFacet { +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+ :gratitude_thank_you: Brad Paskewitz +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2023-03-27 12:29:57
 +
+

*Thread Reply:* Agreed, just a POJO would work. This is using Jackson, so you would use annotations as needed. You can also use a Jackson JSONNode or even Map.

+ + + +
+ :gratitude_thank_you: Brad Paskewitz +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Brad Paskewitz + (bradford.paskewitz@fivetran.com) +
+
2023-03-27 14:01:07
 +
+

One other question: I'm in the process of adding different types of facets to our base payloads and I'm wondering if we have any related guidelines / best practices / standards / conventions. For example if I add a full source schema as a schema dataset facet to every start event it seems like that could be inefficient compared to a 1-time full-source-schema followed by incremental diffs for each following sync. Curious how others are thinking about + solving these types of problems in practice

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-03-27 17:59:28
 +
+

*Thread Reply:* That depends on the OL consumer, but for something like SchemaDatasetFacet it seems to be okay to assume schema stays the same if not send.

+ +

For others, like OutputStatisticsOutputDatasetFacet you definitely can't assume that, as the data is unique to each run.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Brad Paskewitz + (bradford.paskewitz@fivetran.com) +
+
2023-03-27 19:05:14
 +
+

*Thread Reply:* ok great thanks, that makes sense to me

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Saravanan + (saravanan@athivatech.com) +
+
2023-03-27 21:42:20
 +
+

Hi Team, I’m seeing creating data source, dataset API’s marked as deprecated . Can anyone point me how to create datasets via API calls?

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-03-28 04:47:31
 +
+

*Thread Reply:* OpenLineage API: https://openlineage.io/docs/getting-started/

+
+
openlineage.io
+ + + +
Getting Started | OpenLineage Docs
+
+ This guide covers how you can quickly get started collecting dataset, job, and run metadata using OpenLineage. We'll show how to collect run-level metadata as OpenLineage events using Marquez as the HTTP backend, then explore lineage metadata via the Marquez UI. +
+ + + + + + +
Original URL: https://openlineage.io/docs/getting-started/
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-03-28 06:08:18
 +
+

Hi everyone, I recently encountered this error saying V2SessionCatalog is not supported by openlineage. May I ask if support for this will be added in near future? Thanks!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-03-28 08:05:30
 +
+

*Thread Reply:* I think it would be great to support V2SessionCatalog, and it would very much help if you created GitHub issue with more explanation and examples of it's use.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-03-29 02:53:37
 +
+

*Thread Reply:* Sure thanks!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-03-29 05:34:37
 +
+

*Thread Reply:* https://github.com/OpenLineage/OpenLineage/issues/1747 +I have opened an issue here. Thanks! 🙂

+
+ + + + +
#1747 [PROPOSAL] Support for V2SessionCatalog
+
+ <p>Delta catalog uses V2SessionCatalog for some operations. All catalog calls except the delta lake ones are delegated to V2SessionCatalog. Currently, openlineage does not support this spark catalog. <br /> +Adding support for the V2SessionCatalog will be of great use as we are currently unable to emit metadata from some important pipelines which use this spark catalog, to our backend. <br /> +Wondering if this feature can be supported, or is on the roadmap potentially? Thanks a lot!</p> +
+ +
+
Labels
+ proposal +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-04-17 11:53:52
 +
+

*Thread Reply:* Hi @Maciej Obuchowski Just curious, is this issue on the potential roadmap for the next Openlineage release?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tom van Eijk + (t.m.h.vaneijk@tilburguniversity.edu) +
+
2023-04-02 19:37:27
 +
+

Hi all! Can anyone provide me some advice on how to solve this error: +ValueError: `emit` only accepts RunEvent class +[2023-04-02, 23:22:00 UTC] {taskinstance.py:1326} INFO - Marking task as FAILED. dag_id=etl_openlineage, task_id=send_ol_events, execution_date=20230402T232112, start_date=20230402T232114, end_date=20230402T232200 +[2023-04-02, 23:22:00 UTC] {standard_task_runner.py:105} ERROR - Failed to execute job 400 for task send_ol_events (`emit` only accepts RunEvent class; 28020) +[2023-04-02, 23:22:00 UTC] {local_task_job.py:212} INFO - Task exited with return code 1 +[2023-04-02, 23:22:00 UTC] {taskinstance.py:2585} INFO - 0 downstream tasks scheduled from follow-on schedule check +I'm trying to follow this tutorial (https://openlineage.io/blog/openlineage-snowflake/) on connecting Snowflake to OpenLineage through Apache Airflow, however, the last step (sending the OpenLineage events) returns an error.

+
+
openlineage.io
+ + + +
Data Lineage with Snowflake | OpenLineage Docs
+
+ The OpenLineage Adapter offers Snowflake's enterprise users a powerful tool for analyzing their pipelines. +
+ + + + + + +
Original URL: https://openlineage.io/blog/openlineage-snowflake/
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-04-03 09:32:46
 +
+

*Thread Reply:* The blog post is a bit old and in the meantime there were changes in OpenLineage Python Client introduced. +May I ask if you want just to test the flow or looking for any viable Snowflake data lineage solution?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2023-04-03 10:47:57
 +
+

*Thread Reply:* I believe that this will work if you change the line to client.transport.emit()

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2023-04-03 10:49:05
 +
+

*Thread Reply:* (this would be in the dags/lineage folder, if memory serves)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-04-03 10:57:23
 +
+

*Thread Reply:* Ross is right, that should work

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tom van Eijk + (t.m.h.vaneijk@tilburguniversity.edu) +
+
2023-04-04 12:23:13
 +
+

*Thread Reply:* This works! Thank you so much!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tom van Eijk + (t.m.h.vaneijk@tilburguniversity.edu) +
+
2023-04-04 12:24:40
 +
+

*Thread Reply:* @Jakub Dardziński I want to use a viable Snowflake data lineage solution alongside a Amazon DataZone Catalog 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2023-04-04 13:03:58
 +
+

*Thread Reply:* I have been meaning to revisit that tutorial 👍

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-04-03 10:52:42
 +
+

Hello all, +I’d like to open a vote to release OpenLineage 0.22.0, including: +• a new properties facet in the Spark integration +• a new field in HttpConfig for passing custom headers in the Spark integration +• improved namespace generation for JDBC connections in the Spark integration +• removal of unnecessary warnings about column lineage in the Spark integration +• support for alter, truncate, and drop statements in the SQL parser +• typing hints in the SQL integration +• a new from_dict class method in the Python client to support creating it from a dictionary +• a case-insensitive env variable for disabling OpenLineage in the Python client and Airflow integration +• bug fixes, docs changes, and more. +Three +1s from committers will authorize an immediate release. For more details about the release process, see GOVERNANCE.md.

+ + + +
+ ➕ Maciej Obuchowski, Perttu Salonen, Jakub Dardziński, Ross Turk +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-04-03 15:39:46
 +
+

*Thread Reply:* Thanks, all. The release is authorized and will be initiated within 48 hours.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-04-03 16:55:44
 +
+

@channel +We released OpenLineage 0.22.0, including: +Additions: +• Spark: add properties facet #1717 by @tnazarew +• SQL: SQLParser supports alter, truncate and drop statements #1695 by @pawel-big-lebowski +• Common/SQL: provide public interface for openlineage_sql package #1727 by @JDarDagran +• Java client: add configurable headers to HTTP transport #1718 by @tnazarew +• Python client: create client from dictionary #1745 by @JDarDagran +Changes: +• Spark: remove URL parameters for JDBC namespaces #1708 by @tnazarew +• Make OPENLINEAGE_DISABLED case-insensitive #1705 by @jedcunningham +Removals: +• Spark: remove unnecessary warnings for column lineage #1700 by @pawel-big-lebowski +• Spark: remove deprecated configs #1711 by @tnazarew +Thanks to all the contributors! +For the bug fixes and details, see: +Release: https://github.com/OpenLineage/OpenLineage/releases/tag/0.22.0 +Changelog: https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md +Commit history: https://github.com/OpenLineage/OpenLineage/compare/0.21.1...0.22.0 +Maven: https://oss.sonatype.org/#nexus-search;quick~openlineage +PyPI: https://pypi.org/project/openlineage-python/

+ + + +
+ 🙌 Jakub Dardziński, Francis McGregor-Macdonald, Howard Yoo, 김형은, Kengo Seki, Anirudh Shrinivason, Perttu Salonen, Paweł Leszczyński, Maciej Obuchowski, Harel Shein +
+ +
+ 🎉 Ross Turk, 김형은, Kengo Seki, Anirudh Shrinivason, Perttu Salonen +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-04-04 01:49:37
 +
+

Hi everyone, if I set executors to 0, and bind address to localhost, and then if I want to use openlineage to capture metadata, I seem to run into an error where the executor tries to fetch the spark jar from the driver, even though there is no executor set. Then, it fails because a connection cannot be established. This is some of the error stack trace: +INFO Executor: Fetching spark://&lt;DRIVER_IP&gt;:44541/jars/io.openlineage_openlineage-spark-0.21.1.jar with timestamp 1680506544239 +ERROR Utils: Aborting task +java.io.IOException: Failed to connect to /&lt;DRIVER_IP&gt;:44541 + at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:287) + at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:218) + at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:230) + at org.apache.spark.rpc.netty.NettyRpcEnv.downloadClient(NettyRpcEnv.scala:399) + at org.apache.spark.rpc.netty.NettyRpcEnv.$anonfun$openChannel$4(NettyRpcEnv.scala:367) + at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) + at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1473) + at org.apache.spark.rpc.netty.NettyRpcEnv.openChannel(NettyRpcEnv.scala:366) + at org.apache.spark.util.Utils$.doFetchFile(Utils.scala:755) + at org.apache.spark.util.Utils$.fetchFile(Utils.scala:541) + at org.apache.spark.executor.Executor.$anonfun$updateDependencies$13(Executor.scala:953) + at org.apache.spark.executor.Executor.$anonfun$updateDependencies$13$adapted(Executor.scala:945) + at scala.collection.TraversableLike$WithFilter.$anonfun$foreach$1(TraversableLike.scala:877) + at scala.collection.mutable.HashMap.$anonfun$foreach$1(HashMap.scala:149) + at scala.collection.mutable.HashTable.foreachEntry(HashTable.scala:237) + at scala.collection.mutable.HashTable.foreachEntry$(HashTable.scala:230) + at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:44) + at scala.collection.mutable.HashMap.foreach(HashMap.scala:149) + at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:876) + at <a href="http://org.apache.spark.executor.Executor.org">org.apache.spark.executor.Executor.org</a>$apache$spark$executor$Executor$$updateDependencies(Executor.scala:945) + at org.apache.spark.executor.Executor.&lt;init&gt;(Executor.scala:247) + at org.apache.spark.scheduler.local.LocalEndpoint.&lt;init&gt;(LocalSchedulerBackend.scala:64) + at org.apache.spark.scheduler.local.LocalSchedulerBackend.start(LocalSchedulerBackend.scala:132) + at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:220) + at org.apache.spark.SparkContext.&lt;init&gt;(SparkContext.scala:579) + at org.apache.spark.api.java.JavaSparkContext.&lt;init&gt;(JavaSparkContext.scala:58) + at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) + at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source) + at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source) + at java.base/java.lang.reflect.Constructor.newInstance(Unknown Source) + at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:247) + at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357) + at py4j.Gateway.invoke(Gateway.java:238) + at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80) + at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69) + at py4j.GatewayConnection.run(GatewayConnection.java:238) + at java.base/java.lang.Thread.run(Unknown Source) +Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /&lt;DRIVER_IP&gt;:44541 +Caused by: java.net.ConnectException: Connection refused + at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) + at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source) + at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:330) + at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) + at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:702) + at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650) + at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576) + at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) + at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) + at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) + at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) + at java.base/java.lang.Thread.run(Unknown Source) +Just curious if anyone here has run into a similar problem before, and what the recommended way to resolve this would be...

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-04-04 13:39:19
 +
+

*Thread Reply:* Do you have small configuration and job to replicate this?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-04-04 22:21:35
 +
+

*Thread Reply:* Yeah. For configs: +spark.driver.bindAddress: "localhost" +spark.master: "local[**]" +spark.sql.catalogImplementation: "hive" + spark.openlineage.transport.endpoint: "&lt;endpoint&gt;" + spark.openlineage.transport.type: "http" + spark.sql.catalog.spark_catalog: "org.apache.spark.sql.delta.catalog.DeltaCatalog" + spark.openlineage.transport.url: "&lt;url&gt;" + spark.extraListeners: "io.openlineage.spark.agent.OpenLineageSparkListener" + and job is submitted via spark submit in client mode with number of executors set to 0. +The spark job by itself could be anything...I think the job fails before initializing the spark session itself.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-04-04 22:23:19
 +
+

*Thread Reply:* The issue is because of the spark.jars.packages config... spark.jars config also runs into the same issue. Because the executor tries to fetch the jar from driver for some reason even though there is no executors set...

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-04-05 05:38:55
 +
+

*Thread Reply:* TBH I'm not sure if we can do anything about it. Seems like just having any SparkListener which is not in Spark jars would fall under the same problems, right?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-04-10 06:07:11
 +
+

*Thread Reply:* Yeah... Actually, this was because of binding the driver ip to localhost. In that case, the executor was not able to get the jar from the driver. But yeah I don't think we could have done anything from openlienage end anyway for this. Was just an interesting error to encounter lol

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Lq Dodo + (tryopenmetadata@gmail.com) +
+
2023-04-04 12:07:21
 +
+

Hi, I am new to open lineage. I was able to follow https://openlineage.io/getting-started/ to create a lineage "my-input-->my-job-->my-output". I want to use "my-output" as an input dataset, and connect to the next job, thing like this "my-input-->my-job-->my-output-->my-job2-->my-final-output". How to do it? I have trouble to set eventType and runId, etc. Once the new lineages get massed up, the Marquez UI becomes blank (which is a separated issue).

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2023-04-04 13:02:21
 +
+

*Thread Reply:* In this case you would have four runevents:

+ +
  1. a START event on my-job where my-input is the input and my-output is the output, with a runId you generate on the client
  2. a COMPLETE event on my-job with the same runId from #1
  3. a START event on my-job2 where the input is my-output and the output is my-final-output, with a separate runId you generate
  4. a COMPLETE event on my-job2 with the same runId from #3
    5. +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Lq Dodo + (tryopenmetadata@gmail.com) +
+
2023-04-04 14:53:14
 +
+

*Thread Reply:* thanks for the response. I tried it but now the UI only shows like one second and then turn to blank. I has similar issue before. It seems to me every time when I added a bad lineage, the UI stops working. I have to delete the docker image:-( Not sure whether it is MacOS M1 related issue.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2023-04-04 16:07:06
 +
+

*Thread Reply:* Hmmm, that's interesting. Not sure I've seen that before. If you happen to catch it in that state again, perhaps capture the contents of the lineage_events table so it can be replicated.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Lq Dodo + (tryopenmetadata@gmail.com) +
+
2023-04-04 16:24:28
 +
+

*Thread Reply:* I can fairly easy to reproduce this blank UI issue. Apparently I used the same runId for two different jobs. If I use different unId (which I should), the lineage displays correctly. Thanks again!

+ + + +
+ 👍 Ross Turk +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Lq Dodo + (tryopenmetadata@gmail.com) +
+
2023-04-04 16:41:54
 +
+

Is it possible to add column level lineage via api? Let's say I have fields A,B,C from my-input, and A,B from my-output, and B,C from my-output-s3. I want to see, filter, or query by the column name.

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-04-05 05:35:02
 +
+

*Thread Reply:* You can add https://openlineage.io/docs/spec/facets/dataset-facets/column_lineage_facet/ to your datasets.

+ +

However, I don't think you can currently do any filtering over it

+
+
openlineage.io
+ + + +
Column Level Lineage Dataset Facet | OpenLineage Docs
+
+ Column level lineage provides fine grained information on datasets' dependencies. Not only we know the dependency exist, but we are also able to understand which input columns are used to produce output columns. This allows answering questions like Which root input columns are used to construct column x? +
+ + + + + + +
Original URL: https://openlineage.io/docs/spec/facets/dataset-facets/column_lineage_facet/
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2023-04-05 13:20:20
 +
+

*Thread Reply:* you can see a good example here, @Lq Dodo: https://github.com/MarquezProject/marquez/blob/289fa3eef967c8f7915b074325bb6f8f55480030/docker/metadata.json#L430

+
+ + + + +
<https://github.com/MarquezProject/marquez/blob/289fa3eef967c8f7915b074325bb6f8f55480030/docker/metadata.json | metadata.json>
+
+ <pre><code> "columnLineage": { +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/MarquezProject/marquez">MarquezProject/marquez</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Lq Dodo + (tryopenmetadata@gmail.com) +
+
2023-04-06 11:48:48
 +
+

*Thread Reply:* those examples really help. I can at least build the lineage with column level info using the apis. thanks a lot! Ideally I'd like select one column from the UI and then show me the column level graph. Seems not possible.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2023-04-06 12:46:54
 +
+

*Thread Reply:* correct, right now there isn't column-level metadata on the lineage graph 😞

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Pavani + (ylpavani@gmail.com) +
+
2023-04-05 22:01:33
 +
+

Is airflow mandatory, while integrating snowflake with openlineage?

+ +

I am currently looking for a solution which can capture lineage details from snowflake execution

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harel Shein + (harel.shein@gmail.com) +
+
2023-04-06 10:22:17
 +
+

*Thread Reply:* something needs to trigger lineage collection, are you using some sort of scheduler / execution engine?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Pavani + (ylpavani@gmail.com) +
+
2023-04-06 11:26:13
 +
+

*Thread Reply:* Nope... We currently don't have scheduling tool. Isn't it possible to use open lineage api and collect the details?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-04-06 13:12:44
 +
+

@channel +This month’s OpenLineage TSC meeting is on Thursday, April 20th, at 10 am PT. Meeting info: https://openlineage.io/meetings/. All are welcome! +On the tentative agenda:

+ +
  1. Announcements
  2. Updates (new!) +a. OpenLineage in Airflow AIP +b. Static lineage support +c. Reworking namespaces
  3. Recent release overview
  4. A new consumer
  5. Caching support for column lineage
  6. Discussion items +a. Snowflake tagging
  7. Open discussion +Notes: https://bit.ly/OLwiki +Is there a topic you think the community should discuss at this or a future meeting? Reply or DM me to add items to the agenda.
    8. +
+
+
openlineage.io
+ + + +
TSC Meetings | OpenLineage Docs
+
+ The OpenLineage Technical Steering Committee meets monthly, and is open to all. +
+ + + + + + +
Original URL: https://openlineage.io/meetings/
+ + + +
+ + + +
+ 🚀 alexandre bergere, Paweł Leszczyński +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tom van Eijk + (t.m.h.vaneijk@tilburguniversity.edu) +
+
2023-04-06 15:27:41
 +
+

Hi!

+ +

I have a specific question about how OpenLineage fits in between Amazon MWAA and Marquez on AWS EKS. I guess I need to change for example the etl_openlineage DAG in this Snowflake integration tutorial and the OPENLINEAGE_URL here. However, I'm wondering how to reproduce the Docker containers airflow, airflow_scheduler, and airflow_worker here.

+ +

I heard from @Ross Turk that @Willy Lulciuc and @Michael Collado are experts on the K8s integration for OpenLineage and Marquez. Could you provide me some recommendations on how to approach this integration? Or can anyone else help me?

+ +

Kind regards,

+ +

Tom

+
+
openlineage.io
+ + + +
Data Lineage with Snowflake | OpenLineage Docs
+
+ The OpenLineage Adapter offers Snowflake's enterprise users a powerful tool for analyzing their pipelines. +
+ + + + + + +
Original URL: https://openlineage.io/blog/openlineage-snowflake/
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Lukenoff + (john@jlukenoff.com) +
+
2023-04-07 12:47:18
 +
+

[RESOLVED]👋 Hi there, I’m doing a POC of OpenLineage for our airflow deployment. We have a ton of custom operators and I’m trying to test out extracting lineage using the get_openlineage_facets_on_start method. Currently when I’m testing I can see that the OpenLineage plugin is running via airflow plugins but am not able to see that the method is ever getting called. Do I need to do anything else to tell the default extractor to use get_openlineage_facets_on_start? This is the documentation I’m referencing: https://openlineage.io/docs/integrations/airflow/extractors/default-extractors

+
+
openlineage.io
+ + + +
Default extractors | OpenLineage Docs
+
+ Default extractors are a new way in OpenLineage 0.17.0+ to easily add lineage to your data pipelines by modifying your Airflow operators directly. This means custom operators—built in house or forked from another project—can provide you and your team with lineage data without having to modify the OpenLineage project directly, with data sent to your lineage backend of choice, most commonly using the OPENLINEAGE_URL environment variable. +
+ + + + + + +
Original URL: https://openlineage.io/docs/integrations/airflow/extractors/default-extractors
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Lukenoff + (john@jlukenoff.com) +
+
2023-04-07 12:50:14
 +
+

*Thread Reply:* E.g. do I need to update my custom operators to inherit from DefaultExtractor?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Lukenoff + (john@jlukenoff.com) +
+
2023-04-07 13:18:05
 +
+

*Thread Reply:* FWIW, I can tell some level of connectivity to my Marquez deployment is working since I can see it created the default namespace I defined in my OPENLINEAGE_NAMESPACE env var.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-04-07 18:37:44
 +
+

*Thread Reply:* hey John, it is enough to add the method to your custom operator. Perhaps something breaks inside the method. Did anything show up in the logs?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Lukenoff + (john@jlukenoff.com) +
+
2023-04-07 19:03:01
 +
+

*Thread Reply:* That’s the strange part. I’m not seeing anything to suggest that the method is ever getting called. I’m also expecting that the listener created by the plugin should at least be calling this log line when the task runs. However, I’m not seeing that either. I’m able to verify the plugin is registered using airflow plugins and have debug level logging enabled via AIRFLOW__LOGGING__LOGGING_LEVEL='DEBUG'. This is the output of airflow plugins

+ +

name | macros | listeners | source +==================+================================================+==============================+================================================= +OpenLineagePlugin | openlineage.airflow.macros.lineage_run_id,open | openlineage.airflow.listener | openlineage-airflow==0.22.0: + | lineage.airflow.macros.lineage_parent_id | | EntryPoint(name='OpenLineagePlugin', + | | | value='openlineage.airflow.plugin:OpenLineagePlu + | | | gin', group='airflow.plugins') +Appreciate any ideas you might have!

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/d2fa073347ece847c4b6e0ee75a533a95055eb18/integration/airflow/openlineage/airflow/listener.py | listener.py>
+
+ <pre><code> log.debug("OpenLineage listener got notification about task instance start") +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Lukenoff + (john@jlukenoff.com) +
+
2023-04-11 13:09:05
 +
+

*Thread Reply:* Figured this out. Just needed to run the airflow scheduler and trigger tasks through the DAGs vs. airflow tasks test …

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sheeri Cabral (Collibra) + (sheeri.cabral@collibra.com) +
+
2023-04-07 16:29:03
 +
+

I have a question that I believe will be very easy to answer, and I think I know the answer already, but I want to confirm my understanding of extracting OpenLineage with airflow python scripts.

+ +

Extractors extract lineage from operators, so they have to be using operators, right? If someone asks if I can get lineage from their Airflow-orchestrated python scripts, and they show me their scripts but they’re not importing anything starting with airflow.operators, then I can’t use extractors and therefore can’t get lineage. Is that accurate?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sheeri Cabral (Collibra) + (sheeri.cabral@collibra.com) +
+
2023-04-07 16:30:00
 +
+

*Thread Reply:* (they are importing dagkit sdk stuff like Job, JobContext, ExecutionContext, and NodeContext.)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-04-07 18:40:39
 +
+

*Thread Reply:* Do they run those scripts in PythonOperator? If so, they should receive some events but with no datasets extracted

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sheeri Cabral (Collibra) + (sheeri.cabral@collibra.com) +
+
2023-04-07 21:28:25
 +
+

*Thread Reply:* How can I know that? Would it be in the scripts or the airflow configuration or...

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sheeri Cabral (Collibra) + (sheeri.cabral@collibra.com) +
+
2023-04-08 07:13:56
 +
+

*Thread Reply:* And "with no datasets extracted" that means I wouldn't have the schema of the input and output datasets? (I need the db/schema/table/column names for my purposes)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-04-11 02:49:07
 +
+

*Thread Reply:* That really depends what is the current code but in general any custom code in Airflow does not extract any extra information, especially datasets. One can write their own extractors (more in the docs)

+
+
openlineage.io
+ + + +
Default extractors | OpenLineage Docs
+
+ Default extractors are a new way in OpenLineage 0.17.0+ to easily add lineage to your data pipelines by modifying your Airflow operators directly. This means custom operators—built in house or forked from another project—can provide you and your team with lineage data without having to modify the OpenLineage project directly, with data sent to your lineage backend of choice, most commonly using the OPENLINEAGE_URL environment variable. +
+ + + + + + +
Original URL: https://openlineage.io/docs/integrations/airflow/extractors/default-extractors
+ + + +
+ + + +
+ ✅ Sheeri Cabral (Collibra) +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sheeri Cabral (Collibra) + (sheeri.cabral@collibra.com) +
+
2023-04-12 16:52:04
 +
+

*Thread Reply:* Thanks! This is very helpful. Exactly what I needed.

+ + + +
+ 👍 Jakub Dardziński +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tushar Jain + (tujain@ivp.in) +
+
2023-04-09 12:48:04
 +
+

Hi. I was exploring OpenLineage and I want to know does OpenLineage integrate with MS-SQL (Microsoft SQL Server) ? If yes, how to generate OpenLineage events for MS-SQL Views/Tables/Queries?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-04-12 02:30:19
 +
+

*Thread Reply:* Currently there's no extractor implemented for MS-SQL. We try to update list of supported databases here: https://openlineage.io/docs/integrations/about/

+
+
openlineage.io
+ + + +
OpenLineage Integrations | OpenLineage Docs
+
+ Capability Matrix +
+ + + + + + +
Original URL: https://openlineage.io/docs/integrations/about/
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-04-10 12:00:03
 +
+

@channel +Save the date: the next OpenLineage meetup will be in New York on April 26th! More info is coming soon…

+ + + +
+ ✅ Sheeri Cabral (Collibra), Ross Turk, Minkyu Park +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-04-10 19:00:38
 +
+

@channel +Due to many TSC members being on vacation this week, this month’s TSC meeting will be moved to next Thursday, April 20th. All are welcome! https://openlineage.slack.com/archives/C01CK9T7HKR/p1680801164289949

+
+ + +
+ + + } + + Michael Robinson + (https://openlineage.slack.com/team/U02LXF3HUN7) +
+ + + +
+
+ <p>@channel +This month’s OpenLineage TSC meeting is <strong>on Thursday, April 20th, at 10 am PT</strong>. Meeting info: <a href="https://openlineage.io/meetings/">https://openlineage.io/meetings/</a>. All are welcome! +On the tentative agenda:</p> + +<ol><li>Announcements</li><li>Updates (new!) +a. OpenLineage in Airflow AIP +b. Static lineage support +c. Reworking namespaces</li><li>Recent release overview</li><li>A new consumer</li><li>Caching support for column lineage</li><li>Discussion items +a. Snowflake tagging</li><li>Open discussion +Notes: <a href="https://bit.ly/OLwiki">https://bit.ly/OLwiki</a> +Is there a topic you think the community should discuss at this or a future meeting? Reply or DM me to add items to the agenda.</li> +</ol> +
+ + + + + + +
Original URL: https://openlineage.slack.com/archives/C01CK9T7HKR/p1680801164289949
+ + +
+ + Slack Conversation +
+ + +
+ + + +
+ ✅ Sheeri Cabral (Collibra) +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tom van Eijk + (t.m.h.vaneijk@tilburguniversity.edu) +
+
2023-04-11 13:42:03
 +
+

Hi everyone!

+ +

I'm so sorry for all the messages but I'm trying to get Snowflake, OpenLineage and Marquez working for days now. Hopefully, this is my last question. +The snowflake.connector import connect package seems to be outdated here in extract_openlineage.py and is not working for airflow. Does anyone know how to rewrite this code (e.g., with SnowflakeOperator ) and extract the openlineage access history? You'd be my absolute hero!!!

+
+ + + + +
<https://github.com/Snowflake-Labs/OpenLineage-AccessHistory-Setup/blob/main/examples/airflow/dags/lineage/extract_openlineage.py | extract_openlineage.py>
+
+ <pre><code>import json +import os +from pendulum import datetime + +from airflow import DAG +from airflow.decorators import task +from openlineage.client import OpenLineageClient +from snowflake.connector import connect + +SNOWFLAKE_USER = os.getenv('SNOWFLAKE_USER') +SNOWFLAKE_PASSWORD = os.getenv('SNOWFLAKE_PASSWORD') +SNOWFLAKE_ACCOUNT = os.getenv('SNOWFLAKE_ACCOUNT') + + +@task +def send_ol_events(): + client = OpenLineageClient.from_environment() + + with connect( + user=SNOWFLAKE_USER, + password=SNOWFLAKE_PASSWORD, + account=SNOWFLAKE_ACCOUNT, + database='OPENLINEAGE', + schema='PUBLIC', + ) as conn: + with conn.cursor() as cursor: + ol_view = 'OPENLINEAGE_ACCESS_HISTORY' + ol_event_time_tag = 'OL_LATEST_EVENT_TIME' + + var_query = f''' + use warehouse {SNOWFLAKE_WAREHOUSE}; + ''' + + cursor.execute(var_query) + + var_query = f''' + set current_organization='{SNOWFLAKE_ACCOUNT}'; + ''' + + cursor.execute(var_query) + + ol_query = f''' + SELECT ** FROM {ol_view} + WHERE EVENT:eventTime &gt; system$get_tag('{ol_event_time_tag}', '{ol_view}', 'table') + ORDER BY EVENT:eventTime ASC; + ''' + + cursor.execute(ol_query) + ol_events = [json.loads(ol_event[0]) for ol_event in cursor.fetchall()] + + for ol_event in ol_events: + client.emit(ol_event) + + if len(ol_events) &gt; 0: + latest_event_time = ol_events[-1]['eventTime'] + cursor.execute(f''' + ALTER VIEW {ol_view} SET TAG {ol_event_time_tag} = '{latest_event_time}'; + ''') + + +with DAG( + 'etl_openlineage', + start_date=datetime(2022, 4, 12), + schedule_interval='@hourly', + catchup=False, + default_args={ + 'owner': 'openlineage', + 'depends_on_past': False, + 'email_on_failure': False, + 'email_on_retry': False, + 'email': ['&lt;a href="mailto:demo@openlineage.io"&gt;demo@openlineage.io&lt;/a&gt;'], + 'snowflake_conn_id': 'openlineage_snowflake' + }, + description='Send OL events every minutes.', + tags=["extract"], +) as dag: + send_ol_events() + +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/Snowflake-Labs/OpenLineage-AccessHistory-Setup">Snowflake-Labs/OpenLineage-AccessHistory-Setup</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-04-11 17:05:35
 +
+

*Thread Reply:* > The snowflake.connector import connect package seems to be outdated here in extract_openlineage.py and is not working for airflow. +What's the error?

+ +

> Does anyone know how to rewrite this code (e.g., with SnowflakeOperator ) +Current extractor for SnowflakeOperator extracts lineage for SQL executed in the task, in contrast to the method above with OPENLINEAGE_ACCESS_HISTORY view

+
+ + + + +
<https://github.com/Snowflake-Labs/OpenLineage-AccessHistory-Setup/blob/main/examples/airflow/dags/lineage/extract_openlineage.py | extract_openlineage.py>
+
+ <pre><code>import json +import os +from pendulum import datetime + +from airflow import DAG +from airflow.decorators import task +from openlineage.client import OpenLineageClient +from snowflake.connector import connect + +SNOWFLAKE_USER = os.getenv('SNOWFLAKE_USER') +SNOWFLAKE_PASSWORD = os.getenv('SNOWFLAKE_PASSWORD') +SNOWFLAKE_ACCOUNT = os.getenv('SNOWFLAKE_ACCOUNT') + + +@task +def send_ol_events(): + client = OpenLineageClient.from_environment() + + with connect( + user=SNOWFLAKE_USER, + password=SNOWFLAKE_PASSWORD, + account=SNOWFLAKE_ACCOUNT, + database='OPENLINEAGE', + schema='PUBLIC', + ) as conn: + with conn.cursor() as cursor: + ol_view = 'OPENLINEAGE_ACCESS_HISTORY' + ol_event_time_tag = 'OL_LATEST_EVENT_TIME' + + var_query = f''' + use warehouse {SNOWFLAKE_WAREHOUSE}; + ''' + + cursor.execute(var_query) + + var_query = f''' + set current_organization='{SNOWFLAKE_ACCOUNT}'; + ''' + + cursor.execute(var_query) + + ol_query = f''' + SELECT ** FROM {ol_view} + WHERE EVENT:eventTime &gt; system$get_tag('{ol_event_time_tag}', '{ol_view}', 'table') + ORDER BY EVENT:eventTime ASC; + ''' + + cursor.execute(ol_query) + ol_events = [json.loads(ol_event[0]) for ol_event in cursor.fetchall()] + + for ol_event in ol_events: + client.emit(ol_event) + + if len(ol_events) &gt; 0: + latest_event_time = ol_events[-1]['eventTime'] + cursor.execute(f''' + ALTER VIEW {ol_view} SET TAG {ol_event_time_tag} = '{latest_event_time}'; + ''') + + +with DAG( + 'etl_openlineage', + start_date=datetime(2022, 4, 12), + schedule_interval='@hourly', + catchup=False, + default_args={ + 'owner': 'openlineage', + 'depends_on_past': False, + 'email_on_failure': False, + 'email_on_retry': False, + 'email': ['&lt;a href="mailto:demo@openlineage.io"&gt;demo@openlineage.io&lt;/a&gt;'], + 'snowflake_conn_id': 'openlineage_snowflake' + }, + description='Send OL events every minutes.', + tags=["extract"], +) as dag: + send_ol_events() + +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/Snowflake-Labs/OpenLineage-AccessHistory-Setup">Snowflake-Labs/OpenLineage-AccessHistory-Setup</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tom van Eijk + (t.m.h.vaneijk@tilburguniversity.edu) +
+
2023-04-11 18:13:49
 +
+

*Thread Reply:* Hi Maciej!Thank you so much for the reply! I managed to generate a working combination on Windows between the airflow example in the marquez git and the snowflake openlineage git. The only error I still get is: +****** Log file does not exist: /opt/bitnami/airflow/logs/dag_id=etl_openlineage/run_id=manual__2023-04-10T14:12:53.764783+00:00/task_id=send_ol_events/attempt=1.log +****** Fetching from: <http://1c8bb4a78f14:8793/log/dag_id=etl_openlineage/run_id=manual__2023-04-10T14:12:53.764783+00:00/task_id=send_ol_events/attempt=1.log> +****** !!!! Please make sure that all your Airflow components (e.g. schedulers, webservers and workers) have the same 'secret_key' configured in 'webserver' section and time is synchronized on all your machines (for example with ntpd) !!!!! +************ See more at <https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#secret-key> +************ Failed to fetch log file from worker. Client error '403 FORBIDDEN' for url '<http://1c8bb4a78f14:8793/log/dag_id=etl_openlineage/run_id=manual__2023-04-10T14:12:53.764783+00:00/task_id=send_ol_events/attempt=1.log>' +For more information check: <https://httpstatuses.com/403> +This one doesn't make sense to me. I found a workaround for the ETL examples in the OpenLineage git by manually creating a Snowflake connector in Airflow, however, the error is still present for the extract_openlineage.py file. I noticed this file is the only one that uses snowflake.connector import connect and not airflow.providers.snowflake.operators.snowflake import SnowflakeOperator like the other ETL Dags.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-04-12 05:35:41
 +
+

*Thread Reply:* I think it's Airflow error related to getting logs from worker

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-04-12 05:36:07
 +
+

*Thread Reply:* snowflake.connector is a Snowflake connector library that SnowflakeOperator uses underneath to connect to Snowflake

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tom van Eijk + (t.m.h.vaneijk@tilburguniversity.edu) +
+
2023-04-12 10:15:21
 +
+

*Thread Reply:* Ah alright! Thanks for pointing that out! 🙂 Do you know how to solve it? Or do you have any recommendations on how to look for the solution?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-04-12 10:19:53
 +
+

*Thread Reply:* I have no experience with Windows, and I think it's the issue: https://github.com/apache/airflow/issues/10388

+ +

I would try running it in Docker TBH

+
+ + + + +
#10388 Windows support for Airflow
+
+ <p><strong>Description</strong></p> + +<p>Currently, the airflow project uses PEP-3143 style daemons to launch tasks (as implemented in <a href="https://pypi.org/project/python-daemon/">https://pypi.org/project/python-daemon/</a>), however this is targeted towards unix daemons. As a result, running airflow on windows requires multiple levels of abstraction each with their own problems. Would it be possible to use something like daemoniker (<a href="https://daemoniker.readthedocs.io/en/latest/">https://daemoniker.readthedocs.io/en/latest/</a>) to launch tasks? What are the challenges and issues?</p> + +<p>In machine learning workflows, with large datasets, it is a huge time-saver if the pipeline tasks can be run on the GPU. WSL 1 does not support GPU passthrough, docker through WSL 2 supports GPU passthrough only with the Insiders build, additionally it has issues with networking when connected to VPN (<a href="https://github.com/microsoft/WSL/issues/5068">microsoft/WSL#5068</a>).</p> + +<p><strong>Use case / motivation</strong></p> + +<p>Natively running airflow without WSL 1/2 or docker on Windows. This is helpful in cases where the company ecosystem is windows-based.</p> + +<p><strong>Possible implementation</strong></p> + +<p>The daemon module is only used to daemonize the scheduler and webserver. Here's a sample code that runs the scheduler (airflow origin/v1-10-stable) using daemoniker, comments are welcome:</p> + +<pre><code># airflow/bin/cli.py +from daemoniker import Daemonizer + +... + +if args.daemon: + with Daemonizer() as (is_setup, daemonizer): + if is_setup: + pid, stdout, stderr, log_file = setup_locations("scheduler", + args.pid, + args.stdout, + args.stderr, + args.log_file) + + _is_parent = daemonizer( + pid, + stdout_goto=stdout, + stderr_goto=stderr + ) + + job.run() +</code></pre> +
+ +
+
Labels
+ kind:feature +
+ +
+
Comments
+ 22 +
+ + + + + + + +
+ + <a href="https://github.com/apache/airflow">apache/airflow</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tom van Eijk + (t.m.h.vaneijk@tilburguniversity.edu) +
+
2023-04-12 11:47:41
 +
+

*Thread Reply:* Yeah I was running Airflow in Docker but this didn't work. I'll try to use my Macbook for now because I don't think there is a solution for this in the short time. Thank you so much for the support though!!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Peter Hanssens + (peter@cloudshuttle.com.au) +
+
2023-04-13 04:55:41
 +
+

Hi All, +My team and I have been building a status page based on open lineage and I did a talk about it… keen for feedback and thoughts: +https://youtu.be/nGh5_j3hXrE

+
+
YouTube
+ +
+ + + } + + DataEngAU + (https://www.youtube.com/@DataEngAU) +
+ + + +
Peter Hanssens - Sydney Data Engineering meetup, March Edition
+
+ +
+ + + + + + + + +
Original URL: https://youtu.be/nGh5_j3hXrE
+ + + +
+ + + +
+ 👀 Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-04-13 11:19:57
 +
+

*Thread Reply:* Very interesting!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2023-04-13 13:28:53
 +
+

*Thread Reply:* that’s awesome 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ernie Ostic + (ernie.ostic@getmanta.com) +
+
2023-04-13 08:22:50
 +
+

Hi Peter. Looks good. I like the way you introduced the premise of, and benefits of, using OpenLineage for your project. Have you also explored other integrations in addition to dbt?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Peter Hanssens + (peter@cloudshuttle.com.au) +
+
2023-04-13 08:36:01
 +
+

*Thread Reply:* Thanks Ernie, I’m looking at Airflow as well as GE and would like to contribute back to the project as well… we’re close to getting a public preview release of our product done and then we want to help build out open lineage

+ + + +
+ ❤️ Julien Le Dem, Harel Shein +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Lukenoff + (john@jlukenoff.com) +
+
2023-04-13 14:08:38
 +
+

[Resolved] Has anyone seen this error before where the openlineage-airflow plugin / listener fails to deepcopy the task instance? I’m using the native airflow DAG / BashOperator objects to do a basic test of static lineage tagging. More details in 🧵

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Lukenoff + (john@jlukenoff.com) +
+
2023-04-13 14:10:08
 +
+

*Thread Reply:* The dag is basically just: +```dag = DAG( + dagid="asanaexampledag", + defaultargs=defaultargs, + scheduleinterval=None, +)

+ +

samplelineagetask = BashOperator( + taskid="samplelineagetask", + bashcommand='echo $OPENLINEAGEURL', + dag=dag, + inlets=[Table(database="redshift", cluster="someschema", name="someinputtable")], + outlets=[Table(database="redshift", cluster="someotherschema", name="someoutputtable")] +)```

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Lukenoff + (john@jlukenoff.com) +
+
2023-04-13 14:11:02
 +
+

*Thread Reply:* This is the error I’m getting, seems to be coming from this line: +[2023-04-13, 17:45:33 UTC] {logging_mixin.py:115} WARNING - Exception in thread Thread-1: +Traceback (most recent call last): + File "/opt/conda/lib/python3.7/threading.py", line 926, in _bootstrap_inner + self.run() + File "/opt/conda/lib/python3.7/threading.py", line 870, in run + self._target(**self._args, ****self._kwargs) + File "/opt/conda/lib/python3.7/site-packages/openlineage/airflow/listener.py", line 89, in on_running + task_instance_copy = copy.deepcopy(task_instance) + File "/opt/conda/lib/python3.7/copy.py", line 180, in deepcopy + y = _reconstruct(x, memo, **rv) + File "/opt/conda/lib/python3.7/copy.py", line 281, in _reconstruct + state = deepcopy(state, memo) + File "/opt/conda/lib/python3.7/copy.py", line 150, in deepcopy + y = copier(x, memo) + File "/opt/conda/lib/python3.7/copy.py", line 241, in _deepcopy_dict + y[deepcopy(key, memo)] = deepcopy(value, memo) + File "/opt/conda/lib/python3.7/copy.py", line 161, in deepcopy + y = copier(memo) + File "/opt/conda/lib/python3.7/site-packages/airflow/models/baseoperator.py", line 1156, in __deepcopy__ + setattr(result, k, copy.deepcopy(v, memo)) + File "/opt/conda/lib/python3.7/copy.py", line 150, in deepcopy + y = copier(x, memo) + File "/opt/conda/lib/python3.7/copy.py", line 241, in _deepcopy_dict + y[deepcopy(key, memo)] = deepcopy(value, memo) + File "/opt/conda/lib/python3.7/copy.py", line 161, in deepcopy + y = copier(memo) + File "/opt/conda/lib/python3.7/site-packages/airflow/models/dag.py", line 1941, in __deepcopy__ + setattr(result, k, copy.deepcopy(v, memo)) + File "/opt/conda/lib/python3.7/copy.py", line 150, in deepcopy + y = copier(x, memo) + File "/opt/conda/lib/python3.7/copy.py", line 241, in _deepcopy_dict + y[deepcopy(key, memo)] = deepcopy(value, memo) + File "/opt/conda/lib/python3.7/copy.py", line 161, in deepcopy + y = copier(memo) + File "/opt/conda/lib/python3.7/site-packages/airflow/models/baseoperator.py", line 1156, in __deepcopy__ + setattr(result, k, copy.deepcopy(v, memo)) + File "/opt/conda/lib/python3.7/copy.py", line 180, in deepcopy + y = _reconstruct(x, memo, **rv) + File "/opt/conda/lib/python3.7/copy.py", line 281, in _reconstruct + state = deepcopy(state, memo) + File "/opt/conda/lib/python3.7/copy.py", line 150, in deepcopy + y = copier(x, memo) + File "/opt/conda/lib/python3.7/copy.py", line 241, in _deepcopy_dict + y[deepcopy(key, memo)] = deepcopy(value, memo) + File "/opt/conda/lib/python3.7/copy.py", line 150, in deepcopy + y = copier(x, memo) + File "/opt/conda/lib/python3.7/copy.py", line 241, in _deepcopy_dict + y[deepcopy(key, memo)] = deepcopy(value, memo) + File "/opt/conda/lib/python3.7/copy.py", line 161, in deepcopy + y = copier(memo) + File "/opt/conda/lib/python3.7/site-packages/airflow/models/baseoperator.py", line 1156, in __deepcopy__ + setattr(result, k, copy.deepcopy(v, memo)) + File "/opt/conda/lib/python3.7/site-packages/airflow/models/baseoperator.py", line 1000, in __setattr__ + self.set_xcomargs_dependencies() + File "/opt/conda/lib/python3.7/site-packages/airflow/models/baseoperator.py", line 1107, in set_xcomargs_dependencies + XComArg.apply_upstream_relationship(self, arg) + File "/opt/conda/lib/python3.7/site-packages/airflow/models/xcom_arg.py", line 186, in apply_upstream_relationship + op.set_upstream(ref.operator) + File "/opt/conda/lib/python3.7/site-packages/airflow/models/taskmixin.py", line 241, in set_upstream + self._set_relatives(task_or_task_list, upstream=True, edge_modifier=edge_modifier) + File "/opt/conda/lib/python3.7/site-packages/airflow/models/taskmixin.py", line 185, in _set_relatives + dags: Set["DAG"] = {task.dag for task in [**self.roots, **task_list] if task.has_dag() and task.dag} + File "/opt/conda/lib/python3.7/site-packages/airflow/models/taskmixin.py", line 185, in &lt;setcomp&gt; + dags: Set["DAG"] = {task.dag for task in [**self.roots, **task_list] if task.has_dag() and task.dag} + File "/opt/conda/lib/python3.7/site-packages/airflow/models/dag.py", line 508, in __hash__ + val = tuple(self.task_dict.keys()) +AttributeError: 'DAG' object has no attribute 'task_dict'

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/d2fa073347ece847c4b6e0ee75a533a95055eb18/integration/airflow/openlineage/airflow/listener.py | listener.py>
+
+ <pre><code> task_instance_copy = copy.deepcopy(task_instance) +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+ 👀 Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Lukenoff + (john@jlukenoff.com) +
+
2023-04-13 14:12:11
 +
+

*Thread Reply:* This is with Airflow 2.3.2 and openlineage-airflow 0.22.0

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Lukenoff + (john@jlukenoff.com) +
+
2023-04-13 14:13:34
 +
+

*Thread Reply:* Seems like it might be some issue like this with a circular structure? https://stackoverflow.com/questions/46283738/attributeerror-when-using-python-deepcopy

+
+
Stack Overflow
+ + + +
AttributeError when using python deepcopy
+
+ I have a class that has __eq__ and __hash__ overridden, to make its objects act as dictionary keys. Each object also carries a dictionary, keyed by other objects of the same class. I get a weird +
+ + + + + + + + +
Original URL: https://stackoverflow.com/questions/46283738/attributeerror-when-using-python-deepcopy
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-04-14 08:44:36
 +
+

*Thread Reply:* Just by quick look at it, it will definitely be fixed with Airflow 2.6, as it won't need to deepcopy anything.

+ + + +
+ 👍 John Lukenoff +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-04-14 08:47:16
 +
+

*Thread Reply:* I can't seem to reproduce the issue. I ran following example DAG with same Airflow and OL versions as yours: +```import datetime

+ +

from airflow.lineage.entities import Table +from airflow.models import DAG +from airflow.operators.bash import BashOperator

+ +

defaultargs = { + "startdate": datetime.datetime.now() +}

+ +

dag = DAG( + dagid="asanaexampledag", + defaultargs=defaultargs, + scheduleinterval=None, +)

+ +

samplelineagetask = BashOperator( + taskid="samplelineagetask", + bashcommand='echo $OPENLINEAGEURL', + dag=dag, + inlets=[Table(database="redshift", cluster="someschema", name="someinputtable")], + outlets=[Table(database="redshift", cluster="someotherschema", name="someoutputtable")] +)```

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-04-14 08:53:48
 +
+

*Thread Reply:* is there any extra configuration you made possibly?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-04-14 13:02:40
 +
+

*Thread Reply:* @John Lukenoff, I was finally able to reproduce this when passing xcom as task.output +looks like this was reported here and solved by this PR (not sure if this was released in 2.3.3 or later)

+
+ + + + +
#24525 mini-scheduler raises AttributeError: 'NoneType' object has no attribute 'keys'
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/apache/airflow">apache/airflow</a> +
+ + +
+
+ + + + +
#19965 Map and Partial DAG authoring interface for Dynamic Task Mapping
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/apache/airflow">apache/airflow</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Lukenoff + (john@jlukenoff.com) +
+
2023-04-14 13:06:59
 +
+

*Thread Reply:* Ah interesting. Let me see if bumping my Airflow version resolves this. Haven’t had a chance to tinker with it much since yesterday.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-04-14 13:13:21
 +
+

*Thread Reply:* I ran it against 2.4 and same dag works

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Lukenoff + (john@jlukenoff.com) +
+
2023-04-14 13:15:35
 +
+

*Thread Reply:* 👍 Looks like a fix for that issue was rolled out in 2.3.3. I’m gonna try that for now (my company has a notoriously difficult time with airflow major version updates 😅)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-04-14 13:17:06
 +
+

*Thread Reply:* 👍

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Lukenoff + (john@jlukenoff.com) +
+
2023-04-17 12:29:09
 +
+

*Thread Reply:* Got this working! We just monkey patched the __deepcopy__ method of the BaseOperator for now until we can get bandwidth for an airflow upgrade. Thanks for the help here!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-04-17 03:45:47
 +
+

Hi everyone, I am facing this null pointer error: +ERROR AsyncEventQueue: Listener OpenLineageSparkListener threw an exception +java.lang.NullPointerException +java.base/java.util.concurrent.ConcurrentHashMap.putVal(Unknown Source) +java.base/java.util.concurrent.ConcurrentHashMap.put(Unknown Source) +io.openlineage.spark.agent.JobMetricsHolder.addMetrics(JobMetricsHolder.java:40) +io.openlineage.spark.agent.OpenLineageSparkListener.onTaskEnd(OpenLineageSparkListener.java:179) +org.apache.spark.scheduler.SparkListenerBus.doPostEvent(SparkListenerBus.scala:45) +org.apache.spark.scheduler.SparkListenerBus.doPostEvent$(SparkListenerBus.scala:28) +org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37) +org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37) +org.apache.spark.util.ListenerBus.postToAll(ListenerBus.scala:117) +org.apache.spark.util.ListenerBus.postToAll$(ListenerBus.scala:101) +org.apache.spark.scheduler.AsyncEventQueue.super$postToAll(AsyncEventQueue.scala:105) +org.apache.spark.scheduler.AsyncEventQueue.$anonfun$dispatch$1(AsyncEventQueue.scala:105) +scala.runtime.java8.JFunction0$mcJ$sp.apply(JFunction0$mcJ$sp.java:23) +scala.util.DynamicVariable.withValue(DynamicVariable.scala:62) +<a href="http://org.apache.spark.scheduler.AsyncEventQueue.org">org.apache.spark.scheduler.AsyncEventQueue.org</a>$apache$spark$scheduler$AsyncEventQueue$$dispatch(AsyncEventQueue.scala:100) +org.apache.spark.scheduler.AsyncEventQueue$$anon$2.$anonfun$run$1(AsyncEventQueue.scala:96) +org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1381) +org.apache.spark.scheduler.AsyncEventQueue$$anon$2.run(AsyncEventQueue.scala:96) +Could I get some help on this pls 🙇

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-04-17 03:56:30
 +
+

*Thread Reply:* This is the spark submit command: +spark-submit --py-files /usr/local/lib/common_utils.zip,/usr/local/lib/team_utils.zip,/usr/local/lib/project_utils.zip + --conf spark.executor.cores=16 + --conf spark.hadoop.fs.s3a.connection.maximum=100 --conf spark.sql.shuffle.partitions=1000 + --conf spark.speculation=true --conf spark.sql.adaptive.advisoryPartitionSizeInBytes=256MB + --conf spark.hadoop.fs.s3a.multiobjectdelete.enable=false --conf spark.memory.fraction=0.7 --conf spark.kubernetes.executor.label.experiment=some_label --conf spark.kubernetes.executor.label.team=team_name --conf spark.driver.memory=26112m --conf <a href="http://spark.kubernetes.executor.label.app.kubernetes.io/managed-by=pipeline_name">spark.kubernetes.executor.label.app.kubernetes.io/managed-by=pipeline_name</a> --conf spark.kubernetes.executor.label.instance-type=4xlarge --conf spark.executor.instances=10 --conf spark.kubernetes.executor.label.env=prd --conf spark.kubernetes.executor.label.job-name=job_name --conf spark.kubernetes.executor.label.owner=owner --conf spark.kubernetes.executor.label.pipeline=pipeline --conf spark.kubernetes.executor.label.platform-name=platform_name --conf spark.speculation.multiplier=10 --conf spark.memory.storageFraction=0.4 --conf spark.driver.maxResultSize=26112m --conf spark.kubernetes.executor.request.cores=15000m --conf spark.speculation.interval=1s --conf spark.executor.memory=104g --conf spark.sql.catalogImplementation=hive --conf spark.eventLog.dir=file:///logs/spark-events --conf spark.hadoop.fs.s3a.threads.max=100 --conf spark.speculation.quantile=0.75 job.py

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-04-17 04:09:57
 +
+

*Thread Reply:* @Anirudh Shrinivason pls create an issue for this and I will look at it. Although it may be difficult to find the root cause, null pointer exception should be always avoided and this seems to be a bug.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-04-17 04:14:41
 +
+

*Thread Reply:* Hmm yeah sure. I'll create an issue on github for this issue. Thanks!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-04-17 05:13:54
 +
+

*Thread Reply:* https://github.com/OpenLineage/OpenLineage/issues/1784 +Opened an issue here

+
+ + + + +
#1784 Null Pointer Exception with OpenlineageSparkListener
+
+ <p>Hi everyone, <br /> +I am facing this null pointer exception in OpenlineageSparkListener for my job saving into delta table. <br /> +<code>ERROR AsyncEventQueue: Listener OpenLineageSparkListener threw an exception java.lang.NullPointerException java.base/java.util.concurrent.ConcurrentHashMap.putVal(Unknown Source) java.base/java.util.concurrent.ConcurrentHashMap.put(Unknown Source) io.openlineage.spark.agent.JobMetricsHolder.addMetrics(JobMetricsHolder.java:40) io.openlineage.spark.agent.OpenLineageSparkListener.onTaskEnd(OpenLineageSparkListener.java:179) org.apache.spark.scheduler.SparkListenerBus.doPostEvent(SparkListenerBus.scala:45) org.apache.spark.scheduler.SparkListenerBus.doPostEvent$(SparkListenerBus.scala:28) org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37) org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37) org.apache.spark.util.ListenerBus.postToAll(ListenerBus.scala:117) org.apache.spark.util.ListenerBus.postToAll$(ListenerBus.scala:101) org.apache.spark.scheduler.AsyncEventQueue.super$postToAll(AsyncEventQueue.scala:105) org.apache.spark.scheduler.AsyncEventQueue.$anonfun$dispatch$1(AsyncEventQueue.scala:105) scala.runtime.java8.JFunction0$mcJ$sp.apply(JFunction0$mcJ$sp.java:23) scala.util.DynamicVariable.withValue(DynamicVariable.scala:62) &lt;a href="http://org.apache.spark.scheduler.AsyncEventQueue.org"&gt;org.apache.spark.scheduler.AsyncEventQueue.org&lt;/a&gt;$apache$spark$scheduler$AsyncEventQueue$$dispatch(AsyncEventQueue.scala:100) org.apache.spark.scheduler.AsyncEventQueue$$anon$2.$anonfun$run$1(AsyncEventQueue.scala:96) org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1381) org.apache.spark.scheduler.AsyncEventQueue$$anon$2.run(AsyncEventQueue.scala:96)</code></p> + +<p>The spark submit command used is: <br /> +<code>spark-submit --py-files /usr/local/lib/common_utils.zip,/usr/local/lib/team_utils.zip,/usr/local/lib/project_utils.zip --conf spark.executor.cores=16 --conf spark.hadoop.fs.s3a.connection.maximum=100 --conf spark.sql.shuffle.partitions=1000 --conf spark.speculation=true --conf spark.sql.adaptive.advisoryPartitionSizeInBytes=256MB --conf spark.hadoop.fs.s3a.multiobjectdelete.enable=false --conf spark.memory.fraction=0.7 --conf spark.kubernetes.executor.label.experiment=some_label --conf spark.kubernetes.executor.label.team=team_name --conf spark.driver.memory=26112m --conf &lt;a href="http://spark.kubernetes.executor.label.app.kubernetes.io/managed-by=pipeline_name"&gt;spark.kubernetes.executor.label.app.kubernetes.io/managed-by=pipeline_name&lt;/a&gt; --conf spark.kubernetes.executor.label.instance-type=4xlarge --conf spark.executor.instances=10 --conf spark.kubernetes.executor.label.env=prd --conf spark.kubernetes.executor.label.job-name=job_name --conf spark.kubernetes.executor.label.owner=owner --conf spark.kubernetes.executor.label.pipeline=pipeline --conf spark.kubernetes.executor.label.platform-name=platform_name --conf spark.speculation.multiplier=10 --conf spark.memory.storageFraction=0.4 --conf spark.driver.maxResultSize=26112m --conf spark.kubernetes.executor.request.cores=15000m --conf spark.speculation.interval=1s --conf spark.executor.memory=104g --conf spark.sql.catalogImplementation=hive --conf spark.eventLog.dir=file:///logs/spark-events --conf spark.hadoop.fs.s3a.threads.max=100 --conf spark.speculation.quantile=0.75 job.py</code></p> + +<p>Would greatly appreciate any help on this! Thanks!</p> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Allison Suarez + (asuarezmiranda@lyft.com) +
+
2023-04-17 19:32:23
 +
+

Hey! Question about spark column lineage. What is the intended way to write custom code for getting column lineage? i am trying to implement CustomColumnLineageVisitor but when I try to do so I get: +io.openlineage.spark3.agent.lifecycle.plan.column.CustomColumnLineageVisitor is not public in io.openlineage.spark3.agent.lifecycle.plan.column; cannot be accessed from outside package

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-04-18 02:25:04
 +
+

*Thread Reply:* Hi @Allison Suarez, CustomColumnLineageVisitor should be definitely public. I'll prepare a fix PR for that. We do have a test for custom column lineage visitors (CustomColumnLineageVisitorTestImpl), but they're in the same package. Thanks for bringing this.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-04-18 03:07:11
 +
+

*Thread Reply:* This PR should resolve problem: +https://github.com/OpenLineage/OpenLineage/pull/1788

+
+ + + + +
#1788 [SPARK] fix custom column lineage visitors
+
+ <p><strong>Problem</strong></p> + +<p><code>CustomColumnLineageVisitor</code> interface is not public.</p> + +<p><strong>Solution</strong></p> + +<p>Make it public and rearrange test using it, so that visitor implementation is in a different package.</p> + +<blockquote> + <p><strong>Note:</strong> All schema changes require discussion. Please <a href="https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue">link the issue</a> for context.</p> +</blockquote> + +<p>☐ Your change modifies the <a href="https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.json">core</a> OpenLineage model +☐ Your change modifies one or more OpenLineage <a href="https://github.com/OpenLineage/OpenLineage/tree/main/spec/facets">facets</a></p> + +<p>If you're contributing a new integration, please specify the scope of the integration and how/where it has been tested (e.g., Apache Spark integration supports <code>S3</code> and <code>GCS</code> filesystem operations, tested with AWS EMR).</p> + +<p><strong>One-line summary:</strong> +<strong>Checklist</strong></p> + +<p>☑︎ You've <a href="https://github.com/OpenLineage/OpenLineage/blob/main/why-the-dco.md">signed-off</a> your work +☑︎ Your pull request title follows our <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CONTRIBUTING.md#creating-pull-requests">guidelines</a> +☑︎ Your changes are accompanied by tests (<em>if relevant</em>) +☐ Your change contains a <a href="https://kurtisnusbaum.medium.com/stacked-diffs-keeping-phabricator-diffs-small-d9964f4dcfa6">small diff</a> and is self-contained +☐ You've updated any relevant documentation (<em>if relevant</em>) +☐ Your comment includes a one-liner for the changelog about the specific purpose of the change (<em>if necessary</em>) +☐ You've versioned the core OpenLineage model or facets according to <a href="https://docs.snowplowanalytics.com/docs/pipeline-components-and-applications/iglu/common-architecture/schemaver">SchemaVer</a> (<em>if relevant</em>) +☐ You've added a <a href="https://github.com/OpenLineage/OpenLineage/tree/main/.github/header_templates.md">header</a> to source files (<em>if relevant</em>)</p> + +<hr /> + +<p>SPDX-License-Identifier: Apache-2.0 <br /> +Copyright 2018-2023 contributors to the OpenLineage project</p> +
+ +
+
Labels
+ integration/spark +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Allison Suarez + (asuarezmiranda@lyft.com) +
+
2023-04-18 13:34:43
 +
+

*Thread Reply:* Thank you so much @Paweł Leszczyński 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Allison Suarez + (asuarezmiranda@lyft.com) +
+
2023-04-18 13:35:46
 +
+

*Thread Reply:* How does the release process work for OL? Do we have to wait a certain amount of time to get this change in a new release?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Allison Suarez + (asuarezmiranda@lyft.com) +
+
2023-04-18 17:34:29
 +
+

*Thread Reply:* @Maciej Obuchowski ^

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-04-19 01:49:33
 +
+

*Thread Reply:* 0.22.0 was released two weeks ago, so the next schedule should be in next two weeks. We can ask @Michael Robinson his opinion on releasing 0.22.1 before that.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-04-19 09:08:58
 +
+

*Thread Reply:* Hi Allison 👋, +Anyone can request a release in the #general channel. I encourage you to go this route. You’ll need three +1s (there’s more info about the process here: https://github.com/OpenLineage/OpenLineage/blob/main/GOVERNANCE.md), but I don’t know of any reasons why we can’t do a mid-cycle release. 🙂

+ + + +
+ 🙏 Allison Suarez +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Allison Suarez + (asuarezmiranda@lyft.com) +
+
2023-04-19 16:23:20
 +
+

*Thread Reply:* seems like we got enough +1s

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-04-19 16:24:33
 +
+

*Thread Reply:* We need three committers to give a +1. I’ll reach out again to see if I can recruit a third

+ + + +
+ 🙌 Allison Suarez +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Allison Suarez + (asuarezmiranda@lyft.com) +
+
2023-04-19 16:24:55
 +
+

*Thread Reply:* oooh

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-04-19 16:32:47
 +
+

*Thread Reply:* Yeah, sorry I forgot to mention that!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-04-20 05:02:46
 +
+

*Thread Reply:* we have it now

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-04-19 09:52:02
 +
+

@channel +This month’s TSC meeting is tomorrow, 4/20, at 10 am PT: https://openlineage.slack.com/archives/C01CK9T7HKR/p1681167638153879

+
+ + +
+ + + } + + Michael Robinson + (https://openlineage.slack.com/team/U02LXF3HUN7) +
+ + + +
+
+ <p>@channel +Due to many TSC members being on vacation this week, this month’s TSC meeting will be moved to next <strong>Thursday, April 20th</strong>. All are welcome! <a href="https://openlineage.slack.com/archives/C01CK9T7HKR/p1680801164289949">https://openlineage.slack.com/archives/C01CK9T7HKR/p1680801164289949</a></p> +
+ + + + + + +
Original URL: https://openlineage.slack.com/archives/C01CK9T7HKR/p1681167638153879
+ + +
+ + Slack Conversation +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Allison Suarez + (asuarezmiranda@lyft.com) +
+
2023-04-19 13:40:31
 +
+

I would like to get a 0.22.1 patch release to get the issue described in this thread before the next scheduled release.

+
+ + +
+ + + } + + Allison Suarez + (https://openlineage.slack.com/team/U04BNREL8PM) +
+ + + +
+
+ <p>Hey! Question about spark column lineage. What is the intended way to write custom code for getting column lineage? i am trying to implement <code>CustomColumnLineageVisitor</code> but when I try to do so I get: +<code>io.openlineage.spark3.agent.lifecycle.plan.column.CustomColumnLineageVisitor is not public in io.openlineage.spark3.agent.lifecycle.plan.column; cannot be accessed from outside package</code></p> +
+ + + + + + +
Original URL: https://openlineage.slack.com/archives/C01CK9T7HKR/p1681774343143249
+ + +
+ + Thread in Slack Conversation +
+ + +
+ + + +
+ ➕ Michael Robinson, Paweł Leszczyński, Rohit Menon, Maciej Obuchowski, Julien Le Dem, Jakub Dardziński +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-04-20 09:46:06
 +
+

*Thread Reply:* The release is authorized and will be initiated within 2 business days (not including tomorrow).

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-04-19 15:19:38
 +
+

Here are the details about next week’s OpenLineage Meetup at Astronomer’s NY offices: https://openlineage.io/blog/nyc-meetup. Hope to see you there if you can make it!

+
+
openlineage.io
+ + + +
Meet Us in NYC Later This Month! | OpenLineage Docs
+
+ The next OpenLineage Meetup will take place on April 26th in NYC. +
+ + + + + + +
Original URL: https://openlineage.io/blog/nyc-meetup
+ + + +
+ + + +
+ 👍 Ernie Ostic +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sai + (saivenkatesh161@gmail.com) +
+
2023-04-20 07:38:55
 +
+

Hi Team, I tried integrating openLineage with spark databricks and followed the steps as per the documentation. Installation and all looks good as the listener is enabled, but no event is getting passed to Marquez. I can see below message in log4j logs. Am I missing any configuration to be set?

+ +

Running few spark commands in databricks notebook to create events.

+ +

23/04/20 11:10:34 INFO SparkSQLExecutionContext: OpenLineage received Spark event that is configured to be skipped: SparkListenerSQLExecutionStart +23/04/20 11:10:34 INFO SparkSQLExecutionContext: OpenLineage received Spark event that is configured to be skipped: SparkListenerSQLExecutionEnd

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-04-20 08:57:45
 +
+

*Thread Reply:* Hi Sai,

+ +

Perhaps you could try within printing OpenLineage events into logs. This can be achieved with Spark config parameter: +spark.openlineage.transport.type +equal to console .

+ +

This can help you determine if a problem is generating Openlineage events itself or emitting them into Marquez.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sai + (saivenkatesh161@gmail.com) +
+
2023-04-20 09:18:53
 +
+

*Thread Reply:* Hi @Paweł Leszczyński I passed this config as below, but could not see any changes in the logs. The events are getting generated sometimes like below:

+ +

23/04/20 10:00:15 INFO ConsoleTransport: {"eventType":"START","eventTime":"2023-04-20T10:00:15.085Z","run":{"runId":"ef4f46d1-d13a-420a-87c3-19fbf6ffa231","facets":{"spark.logicalPlan":{"producer":"https://github.com/OpenLineage/OpenLineage/tree/0.22.0/integration/spark","schemaURL":"https://openlineage.io/spec/1-0-5/OpenLineage.json#/$defs/RunFacet","plan":[{"class":"org.apache.spark.sql.catalyst.plans.logical.CreateTableAsSelect","num-children":2,"name":0,"partitioning":[],"query":1,"tableSpec":null,"writeOptions":null,"ignoreIfExists":false},{"class":"org.apache.spark.sql.catalyst.analysis.ResolvedTableName","num-children":0,"catalog":null,"ident":null},{"class":"org.apache.spark.sql.catalyst.plans.logical.Project","num-children":1,"projectList":[[{"class":"org.apache.spark.sql.catalyst.expressions.AttributeReference","num_children":0,"name":"workorderid","dataType":"integer","nullable":true,"metadata":{},"exprId":{"product-cl

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-04-20 09:19:37
 +
+

*Thread Reply:* Ok, great. This means the issue is related to Spark <-> Marquez connection

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-04-20 09:20:33
 +
+

*Thread Reply:* Some time ago Spark config has changed and here is the up-to-date-documentation: https://github.com/OpenLineage/OpenLineage/tree/main/integration/spark

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-04-20 09:21:10
 +
+

*Thread Reply:* please note that spark.openlineage.transport.url has to be used which is different from what you have on screenshot attached

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sai + (saivenkatesh161@gmail.com) +
+
2023-04-20 09:22:40
 +
+

*Thread Reply:* You mean instead of "spark.openlineage.host" I need to use "spark.openlineage.transport.url"?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-04-20 09:23:04
 +
+

*Thread Reply:* yes, please give it a try

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sai + (saivenkatesh161@gmail.com) +
+
2023-04-20 09:23:40
 +
+

*Thread Reply:* sure will give a try and let you know the outcome

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-04-20 09:23:48
 +
+

*Thread Reply:* and set spark.openlineage.transport.type to http

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sai + (saivenkatesh161@gmail.com) +
+
2023-04-20 09:24:04
 +
+

*Thread Reply:* okay

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sai + (saivenkatesh161@gmail.com) +
+
2023-04-20 09:26:42
 +
+

*Thread Reply:* does these configs suffice or I need to add anything else

+ +

spark.extraListeners io.openlineage.spark.agent.OpenLineageSparkListener +spark.openlineage.consoleTransport true +spark.openlineage.version v1 +spark.openlineage.transport.type http +spark.openlineage.transport.url http://<host>:5000/api/v1/namespaces/sparkintegrationpoc/

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-04-20 09:27:07
 +
+

*Thread Reply:* spark.openlineage.consoleTransport true this one can be removed

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-04-20 09:27:33
 +
+

*Thread Reply:* otherwise shall be OK

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sai + (saivenkatesh161@gmail.com) +
+
2023-04-20 10:01:30
 +
+

*Thread Reply:* I added these configs and run, but still same issue. Now I am not able to see the events in log file as well.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sai + (saivenkatesh161@gmail.com) +
+
2023-04-20 10:04:27
 +
+

*Thread Reply:* 23/04/20 13:51:22 INFO SparkSQLExecutionContext: OpenLineage received Spark event that is configured to be skipped: SparkListenerSQLExecutionStart +23/04/20 13:51:22 INFO SparkSQLExecutionContext: OpenLineage received Spark event that is configured to be skipped: SparkListenerSQLExecutionEnd

+ +

Does this need any changes in the config side?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sheeri Cabral (Collibra) + (sheeri.cabral@collibra.com) +
+
2023-04-20 13:02:23
 +
+

If you are trying to get into the OpenLineage Technical Steering Committee meeting, you have to RSVP to the specific event at https://www.addevent.com/calendar/pP575215 to get the password (in the invitation to add to your calendar)

+
+
addevent.com
+ + + +
OpenLineage TSC Meetings
+
+ This calendar contains upcoming meetings for the OpenLineage Technical Steering Committee. +
+ + + + + + + + +
Original URL: https://www.addevent.com/calendar/pP575215
+ + + +
+ + + +
+ 🙌 Michael Robinson +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-04-20 13:53:31
 +
+

Here is a nice article I found online that briefly explains about the spark catalogs just for some context: https://www.waitingforcode.com/apache-spark-sql/pluggable-catalog-api/read +In reference to the V2SessionCatalog use case brought up in the meeting just now

+
+
waitingforcode.com
+ + + +
Pluggable Catalog API
+
+ Despite working with Apache Spark for a while, I still have some undiscovered components. One of them crossed my path while I was writing the first blog post from the ACID file formats series. The lucky one is the Catalog API. +
+ + + + + + + + +
Original URL: https://www.waitingforcode.com/apache-spark-sql/pluggable-catalog-api/read
+ + + +
+ + + +
+ 🙌 Michael Robinson, Maciej Obuchowski, Paweł Leszczyński +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-04-24 06:49:43
 +
+

*Thread Reply:* @Anirudh Shrinivason Thanks for linking this as it contains a clear explanation on Spark catalogs. However, I am still unable to write a failing integration test that reproduces the scenario. Could you provide an example of Spark which is failing on V2SessionCatalog and provide more details how are you trying to read/write data?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-04-24 07:14:04
 +
+

*Thread Reply:* Hi @Paweł Leszczyński I noticed this issue on one of our pipelines before actually. I didn't note down which pipeline the issue was occuring in unfortunately. I'll keep checking from my end to identify the spark job that ran into this error. In the meantime, I'll also try to see for which cases deltaCatalog makes use of the V2SessionCatalog to understand this better. Thanks!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-04-26 03:44:15
 +
+

*Thread Reply:* Hi @Paweł Leszczyński +''' + CREATE TABLE IF NOT EXISTS TABLE_NAME ( + SOME COLUMNS + ) USING delta + PARTITIONED BY (col) + location 's3 location' + ''' +A spark sql like this actually triggers the V2SessionCatalog

+ + + +
+ ❤️ Paweł Leszczyński +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-04-26 03:44:48
 +
+

*Thread Reply:* Thanks @Anirudh Shrinivason, will look into that.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-04-26 05:06:05
 +
+

*Thread Reply:* which spark & delta versions are you using?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-04-27 02:35:50
 +
+

*Thread Reply:* I am not 100% sure if this is something you described, but this was an error I was able to replicate and fix. Please look at the exception stacktrace and let me know if it is same on your side. +https://github.com/OpenLineage/OpenLineage/pull/1798

+
+ + + + +
#1798 Spark/fix input dataset
+
+ <p><strong>Problem</strong></p> + +<p>Spark integration is trying to include output dataset's details on START event. This may lead to errors in case of creating tables as the table does not exist on <code>START</code>. This did not break lineage mechanics, although caused the following entry in logs:</p> + +<pre><code>org.apache.spark.sql.catalyst.analysis.NoSuchTableException: Table or view 'tbl' not found in database 'default' + at org.apache.spark.sql.catalyst.catalog.SessionCatalog.requireTableExists(SessionCatalog.scala:226) + at org.apache.spark.sql.catalyst.catalog.SessionCatalog.getTableRawMetadata(SessionCatalog.scala:514) + at org.apache.spark.sql.catalyst.catalog.SessionCatalog.getTableMetadata(SessionCatalog.scala:500) + at org.apache.spark.sql.execution.datasources.v2.V2SessionCatalog.loadTable(V2SessionCatalog.scala:66) + at org.apache.spark.sql.connector.catalog.DelegatingCatalogExtension.loadTable(DelegatingCatalogExtension.java:66) + at org.apache.spark.sql.delta.catalog.DeltaCatalog.loadTable(DeltaCatalog.scala:169) + at io.openlineage.spark3.agent.lifecycle.plan.catalog.DeltaHandler.getDatasetIdentifier(DeltaHandler.java:73) + at io.openlineage.spark3.agent.lifecycle.plan.catalog.CatalogUtils3.lambda$getDatasetIdentifier$2(CatalogUtils3.java:65) + at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) + at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175) + at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175) + at java.util.ArrayList$ArrayListSpliterator.tryAdvance(ArrayList.java:1361) + at java.util.stream.ReferencePipeline.forEachWithCancel(ReferencePipeline.java:126) + at java.util.stream.AbstractPipeline.copyIntoWithCancel(AbstractPipeline.java:499) + at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:486) + at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472) + at java.util.stream.FindOps$FindOp.evaluateSequential(FindOps.java:152) + at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) + at java.util.stream.ReferencePipeline.findAny(ReferencePipeline.java:536) + at io.openlineage.spark3.agent.lifecycle.plan.catalog.CatalogUtils3.getDatasetIdentifier(CatalogUtils3.java:67) + at io.openlineage.spark3.agent.lifecycle.plan.catalog.CatalogUtils3.getDatasetIdentifier(CatalogUtils3.java:48) + at io.openlineage.spark3.agent.utils.PlanUtils3.getDatasetIdentifier(PlanUtils3.java:78) + at io.openlineage.spark33.agent.lifecycle.plan.CreateReplaceDatasetBuilder.apply(CreateReplaceDatasetBuilder.java:143) + at io.openlineage.spark33.agent.lifecycle.plan.CreateReplaceDatasetBuilder.lambda$apply$1(CreateReplaceDatasetBuilder.java:86) + at java.util.Optional.map(Optional.java:215) +</code></pre> + +<p>Potentially, this was related to <a href="https://github.com/OpenLineage/OpenLineage/issues/1747">#1747</a></p> + +<p><strong>Solution</strong></p> + +<p>Catch <code>NoSuchTableException</code> and prevent an error with a stack trace being printed.</p> + +<blockquote> + <p><strong>Note:</strong> All schema changes require discussion. Please <a href="https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue">link the issue</a> for context.</p> +</blockquote> + +<p>• [] Your change modifies the <a href="https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.json">core</a> OpenLineage model +☐ Your change modifies one or more OpenLineage <a href="https://github.com/OpenLineage/OpenLineage/tree/main/spec/facets">facets</a></p> + +<p>If you're contributing a new integration, please specify the scope of the integration and how/where it has been tested (e.g., Apache Spark integration supports <code>S3</code> and <code>GCS</code> filesystem operations, tested with AWS EMR).</p> + +<p><strong>One-line summary:</strong> +<strong>Checklist</strong></p> + +<p>☑︎ You've <a href="https://github.com/OpenLineage/OpenLineage/blob/main/why-the-dco.md">signed-off</a> your work +☑︎ Your pull request title follows our <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CONTRIBUTING.md#creating-pull-requests">guidelines</a> +☑︎ Your changes are accompanied by tests (<em>if relevant</em>) +☑︎ Your change contains a <a href="https://kurtisnusbaum.medium.com/stacked-diffs-keeping-phabricator-diffs-small-d9964f4dcfa6">small diff</a> and is self-contained +☐ You've updated any relevant documentation (<em>if relevant</em>) +☐ Your comment includes a one-liner for the changelog about the specific purpose of the change (<em>if necessary</em>) +☐ You've versioned the core OpenLineage model or facets according to <a href="https://docs.snowplowanalytics.com/docs/pipeline-components-and-applications/iglu/common-architecture/schemaver">SchemaVer</a> (<em>if relevant</em>) +☐ You've added a <a href="https://github.com/OpenLineage/OpenLineage/tree/main/.github/header_templates.md">header</a> to source files (<em>if relevant</em>)</p> + +<hr /> + +<p>SPDX-License-Identifier: Apache-2.0 <br /> +Copyright 2018-2023 contributors to the OpenLineage project</p> +
+ +
+
Labels
+ documentation, integration/spark +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+ :gratitude_thank_you: Anirudh Shrinivason +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-04-27 02:36:20
 +
+

*Thread Reply:* Hi

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-04-27 02:36:45
 +
+

*Thread Reply:* Hmm actually I am noticing this error on my local

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-04-27 02:37:01
 +
+

*Thread Reply:* But on the prod job, I am seeing no such error in the logs...

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-04-27 02:37:28
 +
+

*Thread Reply:* Also, I was using spark 3.1.2

+ + + +
+ 👀 Paweł Leszczyński +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-04-27 02:37:39
 +
+

*Thread Reply:* then perhaps it's sth different :face_palm: will try to replicate on spark 3.1.2

+ + + +
+ :gratitude_thank_you: Anirudh Shrinivason +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-04-27 02:37:42
 +
+

*Thread Reply:* Not too sure which delta version the prod job was using...

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-04-28 03:30:49
 +
+

*Thread Reply:* I was running on Spark 3.1.2 the following command: +spark.sql( + "CREATE TABLE t_partitioned (a int, b int) USING delta " + + "PARTITIONED BY (a) LOCATION '/tmp/delta/tbl'" + ); +and I got Openlineage event emitted with t_partitioned output dataset.

+ + + +
+ :gratitude_thank_you: Anirudh Shrinivason +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-04-28 03:31:47
 +
+

*Thread Reply:* Oh... hmm... that is strange. Let me check more from my end too

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-04-28 03:33:01
 +
+

*Thread Reply:* for spark 3.1, we're using delta 1.0.0

+ + + +
+ 👀 Anirudh Shrinivason +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Cory Visi + (cvisi@amazon.com) +
+
2023-04-20 14:41:23
 +
+

Hi team! I have two Spark jobs chained together to process incoming data files, and I'm using openlineage-spark-0.22.0 with Marquez to visualize. +I'm struggling to figure out the best way to use spark.openlineage.parentRunId and spark.openlineage.parentJobName. Should these values be unique for each Spark job? Should they be unique for each execution of the chain of both spark jobs? Or should they be the same for all runs? +I'm setting them to be unique to the execution of the chain and I'm getting strange results (jobs are not showing completed, and not showing at all)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-04-24 05:38:09
 +
+

*Thread Reply:* Hi Cory, I think the definition of ParentRunFacet (https://openlineage.io/docs/spec/facets/run-facets/parent_run) contains answer to that: +Commonly, scheduler systems like Apache Airflow will trigger processes on remote systems, such as on Apache Spark or Apache Beam jobs. Those systems might have their own OpenLineage integration and report their own job runs and dataset inputs/outputs. The ParentRunFacet allows those downstream jobs to report which jobs spawned them to preserve job hierarchy. To do that, the scheduler system should have a way to pass its own job and run id to the child job. +For example, when airflow is used to run Spark job, we want Spark events to contain some information on what triggered the spark job and parameters, you ask about, are used to pass that information from airflow operator to spark job.

+
+
openlineage.io
+ + + +
Parent Run Facet | OpenLineage Docs
+
+ Commonly, scheduler systems like Apache Airflow will trigger processes on remote systems, such as on Apache Spark or Apache Beam jobs. +
+ + + + + + +
Original URL: https://openlineage.io/docs/spec/facets/run-facets/parent_run
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Cory Visi + (cvisi@amazon.com) +
+
2023-04-26 17:28:39
 +
+

*Thread Reply:* Thank you for pointing me at this documentation; I did not see it previously. In my setup, the calling system is AWS Step Functions, which have no integration with OpenLineage.

+ +

So I've been essentially passing non-existing parent job information to OpenLineage. It has been useful as a data point for searches and reporting though.

+ +

Is there any harm in doing what I am doing? Is it causing the jobs that I see never completing?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-04-27 04:59:39
 +
+

*Thread Reply:* I think parentRunId should be the same for Openlineage START and COMPLETE event. Is it like this in your case?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Cory Visi + (cvisi@amazon.com) +
+
2023-05-03 11:13:58
 +
+

*Thread Reply:* that makes sense, and based on my configuration, i would think that it would be. however, given that i am seeing incomplete jobs in Marquez, i'm wondering if somehow the parentrunID is changing. I need to investigate

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-04-20 15:44:39
 +
+

@channel +We released OpenLineage 0.23.0, including: +Additions: +• SQL: parser improvements to support: copy into, create stage, pivot #1742 @pawel-big-lebowski +• dbt: add support for snapshots #1787 @JDarDagran +Changes: +• Spark: change custom column lineage visitors #1788 @pawel-big-lebowski +Plus bug fixes, doc changes and more. +Thanks to all the contributors! +For the bug fixes and details, see: +Release: https://github.com/OpenLineage/OpenLineage/releases/tag/0.23.0 +Changelog: https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md +Commit history: https://github.com/OpenLineage/OpenLineage/compare/0.22.0...0.23.0 +Maven: https://oss.sonatype.org/#nexus-search;quick~openlineage +PyPI: https://pypi.org/project/openlineage-python/

+ + + +
+ 🎉 Harel Shein, Maciej Obuchowski, Anirudh Shrinivason, Kengo Seki, Paweł Leszczyński, Perttu Salonen +
+ +
+ 👍 Cory Visi, Maciej Obuchowski, Anirudh Shrinivason, Kengo Seki +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-04-21 05:07:30
 +
+

Just curious, how long before we can see 0.23.0 over here: https://mvnrepository.com/artifact/io.openlineage/openlineage-spark

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-04-21 09:06:06
 +
+

*Thread Reply:* I think @Michael Robinson has to manually promote artifacts

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-04-21 09:08:06
 +
+

*Thread Reply:* I promoted the artifacts, but there is a delay before they appear in Maven. A couple releases ago, the delay was about 24 hours long

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-04-21 09:26:09
 +
+

*Thread Reply:* Ahh I see... Thanks!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-04-21 10:10:38
 +
+

*Thread Reply:* @Anirudh Shrinivason are you using search.maven.org by chance? Version 0.23.0 is not appearing there yet, but I do see it on central.sonatype.com.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-04-21 10:15:00
 +
+

*Thread Reply:* Hmm I can see it now on search.maven.org actually. But I still cannot see it on https://mvnrepository.com/artifact/io.openlineage/openlineage-spark ...

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-04-21 10:19:38
 +
+

*Thread Reply:* Understood. I believe you can download the 0.23.0 jars from central.sonatype.com. For Spark, try going here: https://central.sonatype.com/artifact/io.openlineage/openlineage-spark/0.23.0/versions

+
+
Maven Central
+ + + +
Maven Central: io.openlineage:openlineage-spark:0.23.0
+
+ Maven Central: io.openlineage:openlineage_spark:0.23.0 +
+ + + + + + +
Original URL: https://central.sonatype.com/artifact/io.openlineage/openlineage-spark/0.23.0/versions
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-04-22 06:11:10
 +
+

*Thread Reply:* Yup. I can see it on all maven repos now haha. I think its just the delay.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-04-22 06:11:18
 +
+

*Thread Reply:* ~24 hours ig

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-04-24 16:49:15
 +
+

*Thread Reply:* 👍

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Doe + (adarsh.pansari@tigeranalytics.com) +
+
2023-04-21 08:49:54
 +
+

Hello Everyone, I am facing an issue while trying to integrate openlineage with Jupyter notebook. I am following the Docs. My containers are running and I am getting the URL for Jupyter notebook but when I try with the token in the terminal, I get invalid credentials error. Can someone please help resolve this ? Am I doing something wrong..

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Doe + (adarsh.pansari@tigeranalytics.com) +
+
2023-04-21 09:28:18
 +
+

*Thread Reply:* Good news, everyone! The login worked on the second attempt after starting the Docker containers. Although it's unclear why it failed the first time.

+ + + +
+ 👍 Maciej Obuchowski, Anirudh Shrinivason, Michael Robinson, Paweł Leszczyński +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Natalie Zeller + (natalie.zeller@naturalint.com) +
+
2023-04-23 23:52:34
 +
+

Hi team, +I have a question regarding the customization of transport types in OpenLineage. +At my company, we are using OpenLineage to report lineage from our Spark jobs to OpenMetadata. We have created a custom OpenMetadataTransport to send lineage to the OpenMetadata APIs, conforming to the OpenMetadata format. +Currently, we are using a fork of OpenLineage, as we needed to make some changes in the core to identify the new TransportConfig. +We believe it would be more optimal for OpenLineage to support custom transport types, which would allow us to use OpenLineage JAR alongside our own JAR containing the custom transport. +I noticed some comments in the code suggesting that customizations are possible. However, I couldn't make it work without modifying the TransportFactory and the TransportConfig interface, as the transport types are hardcoded. Am I missing something? 🤔 +If custom transport types are not currently supported, we would be more than happy to contribute a PR that enables custom transports. +What are your thoughts on this?

+ + + +
+ ❤️ Paweł Leszczyński +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-04-24 02:32:51
 +
+

*Thread Reply:* Hi Natalie, it's wonderful to hear you're planning to contribute. Yes, you're right about TransportFactory . What other transport type was in your mind? If it is something generic, then it is surely OK to include it within TransportFactory. If it is a custom feature, we could follow ServiceLoader pattern that we're using to allow including custom plan visitors and dataset builders.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Natalie Zeller + (natalie.zeller@naturalint.com) +
+
2023-04-24 02:54:40
 +
+

*Thread Reply:* Hi @Paweł Leszczyński +Yes, I was planning to change TransportFactory to support custom/generic transport types using ServiceLoader pattern. After this change is done, I will be able to use our custom OpenMetadataTransport without changing anything in OpenLineage core. For now I don't have other types in mind, but after we'll add the customization support anyone will be able to create their own transport type and report the lineage to different backends

+ + + +
+ 👍 Paweł Leszczyński +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-04-24 03:28:30
 +
+

*Thread Reply:* Perhaps it's not strictly related to this particular usecase, but you may also find interesting our recent PoC about Fluentd & Openlineage integration. This will bring some cool backend features like: copy event and send it to multiple backends, send it to backends supported by fluentd output plugins etc. https://github.com/OpenLineage/OpenLineage/pull/1757/files?short_path=4fc5534#diff-4fc55343748f353fa1def0e00c553caa735f9adcb0da18baad50a989c0f2e935

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Natalie Zeller + (natalie.zeller@naturalint.com) +
+
2023-04-24 05:36:24
 +
+

*Thread Reply:* Sounds interesting. Thanks, I will look into it

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-04-24 16:37:33
 +
+

Are you planning to come to the first New York OpenLineage Meetup this Wednesday at Astronomer’s offices in the Flatiron District? Don’t forget to RSVP so we know much food and drink to order!

+
+
Meetup
+ + + +
New York OpenLineage Meetup, Wed, Apr 26, 2023, 5:30 PM | Meetup
+
+ Join us on Wednesday, April 26th, 2023 from 5:30_8:30 pm at the Astronomer offices in New York, NY, to learn more about the present and future of OpenLineage. Meet other me +
+ + + + + + + + +
Original URL: https://www.meetup.com/data-lineage-meetup/events/292897496
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sudhar Balaji + (sudharshan.dataaces@gmail.com) +
+
2023-04-25 03:20:57
 +
+

Hi, I'm new to Open data lineage and I'm trying to connect snowflake database with marquez using airflow and getting the error in etl_openlineage while running the airflow dag on local ubuntu environment and unable to see the marquez UI once it etl_openlineage has ran completed as success.

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-04-25 08:07:36
 +
+

*Thread Reply:* What's the extract_openlineage.py file? Looks like your code?

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sudhar Balaji + (sudharshan.dataaces@gmail.com) +
+
2023-04-25 08:43:04
 +
+

*Thread Reply:* import json +import os +from pendulum import datetime

+ +

from airflow import DAG +from airflow.decorators import task +from openlineage.client import OpenLineageClient +from snowflake.connector import connect

+ +

SNOWFLAKEUSER = os.getenv('SNOWFLAKEUSER') +SNOWFLAKEPASSWORD = os.getenv('SNOWFLAKEPASSWORD') +SNOWFLAKEACCOUNT = os.getenv('SNOWFLAKEACCOUNT')

+ +

@task +def sendolevents(): + client = OpenLineageClient.from_environment()

+ +
with connect(
+    user=SNOWFLAKE_USER,
+    password=SNOWFLAKE_PASSWORD,
+    account=SNOWFLAKE_ACCOUNT,
+    database='OPENLINEAGE',
+    schema='PUBLIC',
+) as conn:
+    with conn.cursor() as cursor:
+        ol_view = 'OPENLINEAGE_ACCESS_HISTORY'
+        ol_event_time_tag = 'OL_LATEST_EVENT_TIME'
+
+        var_query = f'''
+            use warehouse {SNOWFLAKE_WAREHOUSE};
+        '''
+
+        cursor.execute(var_query)
+
+        var_query = f'''
+            set current_organization='{SNOWFLAKE_ACCOUNT}';
+        '''
+
+        cursor.execute(var_query)
+
+        ol_query = f'''
+            SELECT ** FROM {ol_view}
+            WHERE EVENT:eventTime &gt; system$get_tag('{ol_event_time_tag}', '{ol_view}', 'table')
+            ORDER BY EVENT:eventTime ASC;
+        '''
+
+        cursor.execute(ol_query)
+        ol_events = [json.loads(ol_event[0]) for ol_event in cursor.fetchall()]
+
+        for ol_event in ol_events:
+            client.emit(ol_event)
+
+        if len(ol_events) &gt; 0:
+            latest_event_time = ol_events[-1]['eventTime']
+            cursor.execute(f'''
+                ALTER VIEW {ol_view} SET TAG {ol_event_time_tag} = '{latest_event_time}';
+            ''')
+
+ +

with DAG( + 'etlopenlineage', + startdate=datetime(2022, 4, 12), + scheduleinterval='@hourly', + catchup=False, + defaultargs={ + 'owner': 'openlineage', + 'dependsonpast': False, + 'emailonfailure': False, + 'emailonretry': False, + 'email': ['demo@openlineage.io'], + 'snowflakeconnid': 'openlineagesnowflake' + }, + description='Send OL events every minutes.', + tags=["extract"], +) as dag: + sendol_events()

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-04-25 09:52:33
 +
+

*Thread Reply:* OpenLineageClient expects RunEvent classes and you're sending it raw json. I think at this point your options are either sending them by constructing your own HTTP client, using something like requests, or using something like https://github.com/python-attrs/cattrs to structure json to RunEvent

+
+ + + + +
python-attrs/cattrs
+
+ Complex custom class converters for attrs. +
+ +
+
Website
+ <https://catt.rs> +
+ +
+
Stars
+ 625 +
+ + + + + + + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-04-25 10:05:57
 +
+

*Thread Reply:* @Jakub Dardziński suggested that you can +change client.emit(ol_event) to client.transport.emit(ol_event) and it should work

+ + + +
+ 👍 Ross Turk, Sudhar Balaji +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@rossturk.com) +
+
2023-04-25 12:24:08
 +
+

*Thread Reply:* @Maciej Obuchowski I believe this is from https://github.com/Snowflake-Labs/OpenLineage-AccessHistory-Setup/blob/main/examples/airflow/dags/lineage/extract_openlineage.py

+
+ + + + +
<https://github.com/Snowflake-Labs/OpenLineage-AccessHistory-Setup/blob/main/examples/airflow/dags/lineage/extract_openlineage.py | extract_openlineage.py>
+
+ <pre><code>import json +import os +from pendulum import datetime + +from airflow import DAG +from airflow.decorators import task +from openlineage.client import OpenLineageClient +from snowflake.connector import connect + +SNOWFLAKE_USER = os.getenv('SNOWFLAKE_USER') +SNOWFLAKE_PASSWORD = os.getenv('SNOWFLAKE_PASSWORD') +SNOWFLAKE_ACCOUNT = os.getenv('SNOWFLAKE_ACCOUNT') + + +@task +def send_ol_events(): + client = OpenLineageClient.from_environment() + + with connect( + user=SNOWFLAKE_USER, + password=SNOWFLAKE_PASSWORD, + account=SNOWFLAKE_ACCOUNT, + database='OPENLINEAGE', + schema='PUBLIC', + ) as conn: + with conn.cursor() as cursor: + ol_view = 'OPENLINEAGE_ACCESS_HISTORY' + ol_event_time_tag = 'OL_LATEST_EVENT_TIME' + + var_query = f''' + use warehouse {SNOWFLAKE_WAREHOUSE}; + ''' + + cursor.execute(var_query) + + var_query = f''' + set current_organization='{SNOWFLAKE_ACCOUNT}'; + ''' + + cursor.execute(var_query) + + ol_query = f''' + SELECT ** FROM {ol_view} + WHERE EVENT:eventTime &gt; system$get_tag('{ol_event_time_tag}', '{ol_view}', 'table') + ORDER BY EVENT:eventTime ASC; + ''' + + cursor.execute(ol_query) + ol_events = [json.loads(ol_event[0]) for ol_event in cursor.fetchall()] + + for ol_event in ol_events: + client.emit(ol_event) + + if len(ol_events) &gt; 0: + latest_event_time = ol_events[-1]['eventTime'] + cursor.execute(f''' + ALTER VIEW {ol_view} SET TAG {ol_event_time_tag} = '{latest_event_time}'; + ''') + + +with DAG( + 'etl_openlineage', + start_date=datetime(2022, 4, 12), + schedule_interval='@hourly', + catchup=False, + default_args={ + 'owner': 'openlineage', + 'depends_on_past': False, + 'email_on_failure': False, + 'email_on_retry': False, + 'email': ['&lt;a href="mailto:demo@openlineage.io"&gt;demo@openlineage.io&lt;/a&gt;'], + 'snowflake_conn_id': 'openlineage_snowflake' + }, + description='Send OL events every minutes.', + tags=["extract"], +) as dag: + send_ol_events() + +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/Snowflake-Labs/OpenLineage-AccessHistory-Setup">Snowflake-Labs/OpenLineage-AccessHistory-Setup</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@rossturk.com) +
+
2023-04-25 12:25:26
 +
+

*Thread Reply:* I believe this example no longer works - perhaps a new access history pull/push example could be created that is simpler and doesn’t use airflow.

+ + + +
+ 👀 Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-04-26 08:34:02
 +
+

*Thread Reply:* I think separating the actual getting data from the view and Airflow DAG would make sense

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@rossturk.com) +
+
2023-04-26 13:57:34
 +
+

*Thread Reply:* Yeah - I also think that Airflow confuses the issue. You don’t need Airflow to get lineage from Snowflake Access History, the only reason Airflow is in the example is a) to simulate a pipeline that can be viewed in Marquez; b) to establish a mechanism that regularly pulls and emits lineage…

+ +

but most people will already have A, and the simplest example doesn’t need to accomplish B.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@rossturk.com) +
+
2023-04-26 13:58:59
 +
+

*Thread Reply:* just a few weeks ago 🙂 I was working on a script that you could run like SNOWFLAKE_USER=foo ./process_snowflake_lineage.py --from-date=xxxx-xx-xx --to-date=xxxx-xx-xx

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tom van Eijk + (t.m.h.vaneijk@tilburguniversity.edu) +
+
2023-04-27 11:13:58
 +
+

*Thread Reply:* Hi @Ross Turk! Do you have a link to this script? Perhaps this script can fix the connection issue 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@rossturk.com) +
+
2023-04-27 11:47:20
 +
+

*Thread Reply:* No, it never became functional before I stopped to take on another task 😕

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sudhar Balaji + (sudharshan.dataaces@gmail.com) +
+
2023-04-25 07:47:57
 +
+

Hi, +Currently, In the .env file, we have using the OPENLINEAGE_URL as <http://marquez-api:5000> and got the error +requests.exceptions.HTTPError: 422 Client Error: for url: <http://marquez-api:5000/api/v1/lineage> +we have tried using OPENLINEAGE_URL as <http://localhost:5000> and getting the error as +requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=5000): Max retries exceeded with url: /api/v1/lineage (Caused by NewConnectionError('&lt;urllib3.connection.HTTPConnection object at 0x7fc71edb9590&gt;: Failed to establish a new connection: [Errno 111] Connection refused')) +I'm not sure which variable value to use for OPENLINEAGE_URL, so please offer the correct variable value.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-04-25 09:54:07
 +
+

*Thread Reply:* Looks like the first URL is proper, but there's something wrong with entity - Marquez logs would help here.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sudhar Balaji + (sudharshan.dataaces@gmail.com) +
+
2023-04-25 09:57:36
 +
+

*Thread Reply:* This is my log in airflow, can you please prvide more info over it.

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-04-25 10:13:37
 +
+

*Thread Reply:* Airflow log does not tell us why Marquez rejected the event. Marquez logs would be more helpful

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sudhar Balaji + (sudharshan.dataaces@gmail.com) +
+
2023-04-26 05:48:08
 +
+

*Thread Reply:* We investigated the marquez container logs and were unable to locate the error. Could you please specify the log file that belongs to marquez while connecting the airflow or snowflake?

+ +

Is it correct that the marquez-web log points to <http://api:5000/>? +[HPM] Proxy created: /api/v1 -&gt; <http://api:5000/> +App listening on port 3000!

+ +
+
+ + + + + + +
+
+
+ + + + + + +
+ + +
+ 👀 Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tom van Eijk + (t.m.h.vaneijk@tilburguniversity.edu) +
+
2023-04-26 11:26:36
 +
+

*Thread Reply:* I've the same error at the moment but can provide some additional screenshots. The Event data in Snowflake seems fine and the data is being retrieved correctly by the Airflow DAG. However, there seems to be a warning in the Marquez API logs. Hopefully we can troubleshoot this together!

+ +
+
+ + + + + + +
+
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tom van Eijk + (t.m.h.vaneijk@tilburguniversity.edu) +
+
2023-04-26 11:33:35
 +
+

*Thread Reply:*

+ +
+
+ + + + + + +
+
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-04-26 13:06:30
 +
+

*Thread Reply:* Possibly the Python part between does some weird things, like double-jsonning the data? I can imagine it being wrapped in second, unnecessary JSON object

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-04-26 13:08:18
 +
+

*Thread Reply:* I guess only way to check is print one of those events - in the form they are send in Python part, not Snowflake - and see how they are like. For example using ConsoleTransport or setting DEBUG log level in Airflow

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tom van Eijk + (t.m.h.vaneijk@tilburguniversity.edu) +
+
2023-04-26 14:37:32
 +
+

*Thread Reply:* Here is a code snippet by using logging in DEBUG on the snowflake python connector:

+ +

[20230426T17:16:55.166+0000] {cursor.py:593} DEBUG - binding: [set currentorganization='[PRIVATE]';] with input=[None], processed=[{}] +[2023-04-26T17:16:55.166+0000] {cursor.py:800} INFO - query: [set currentorganization='[PRIVATE]';] +[2023-04-26T17:16:55.166+0000] {connection.py:1363} DEBUG - sequence counter: 2 +[2023-04-26T17:16:55.167+0000] {cursor.py:467} DEBUG - Request id: f7bca188-dda0-4fe6-8d5c-a92dc5f9c7ac +[2023-04-26T17:16:55.167+0000] {cursor.py:469} DEBUG - running query [set currentorganization='[PRIVATE]';] +[2023-04-26T17:16:55.168+0000] {cursor.py:476} DEBUG - isfiletransfer: True +[2023-04-26T17:16:55.168+0000] {connection.py:1035} DEBUG - _cmdquery +[2023-04-26T17:16:55.168+0000] {connection.py:1062} DEBUG - sql=[set currentorganization='[PRIVATE]';], sequenceid=[2], isfiletransfer=[False] +[2023-04-26T17:16:55.168+0000] {network.py:1162} DEBUG - Session status for SessionPool [PRIVATE]', SessionPool 1/1 active sessions +[2023-04-26T17:16:55.169+0000] {network.py:850} DEBUG - remaining request timeout: None, retry cnt: 1 +[2023-04-26T17:16:55.169+0000] {network.py:828} DEBUG - Request guid: 4acea1c3-6a68-4691-9af4-22f184e0f660 +[2023-04-26T17:16:55.169+0000] {network.py:1021} DEBUG - socket timeout: 60 +[2023-04-26T17:16:55.259+0000] {connectionpool.py:465} DEBUG - [PRIVATE]"POST /queries/v1/query-request?requestId=f7bca188-dda0-4fe6-8d5c-a92dc5f9c7ac&requestguid=4acea1c3-6a68-4691-9af4-22f184e0f660 HTTP/1.1" 200 1118 +[2023-04-26T17:16:55.261+0000] {network.py:1047} DEBUG - SUCCESS +[2023-04-26T17:16:55.261+0000] {network.py:1168} DEBUG - Session status for SessionPool [PRIVATE], SessionPool 0/1 active sessions +[2023-04-26T17:16:55.261+0000] {network.py:729} DEBUG - ret[code] = None, after post request +[2023-04-26T17:16:55.261+0000] {network.py:751} DEBUG - Query id: 01abe3ac-0603-4df4-0042-c78307975eb2 +[2023-04-26T17:16:55.262+0000] {cursor.py:807} DEBUG - sfqid: 01abe3ac-0603-4df4-0042-c78307975eb2 +[2023-04-26T17:16:55.262+0000] {cursor.py:813} INFO - query execution done +[2023-04-26T17:16:55.262+0000] {cursor.py:827} DEBUG - SUCCESS +[2023-04-26T17:16:55.262+0000] {cursor.py:846} DEBUG - PUT OR GET: False +[2023-04-26T17:16:55.263+0000] {cursor.py:941} DEBUG - Query result format: json +[2023-04-26T17:16:55.263+0000] {resultbatch.py:433} DEBUG - parsing for result batch id: 1 +[2023-04-26T17:16:55.263+0000] {cursor.py:956} INFO - Number of results in first chunk: 1 +[2023-04-26T17:16:55.263+0000] {cursor.py:735} DEBUG - executing SQL/command +[2023-04-26T17:16:55.263+0000] {cursor.py:593} DEBUG - binding: [SELECT * FROM OPENLINEAGE_ACCESS_HISTORY WHERE EVENT:eventTime > system$get_tag(...] with input=[None], processed=[{}] +[2023-04-26T17:16:55.264+0000] {cursor.py:800} INFO - query: [SELECT * FROM OPENLINEAGEACCESSHISTORY WHERE EVENT:eventTime > system$gettag(...] +[2023-04-26T17:16:55.264+0000] {connection.py:1363} DEBUG - sequence counter: 3 +[2023-04-26T17:16:55.264+0000] {cursor.py:467} DEBUG - Request id: 21e2ab85-4995-4010-865d-df06cf5ee5b5 +[2023-04-26T17:16:55.265+0000] {cursor.py:469} DEBUG - running query [SELECT ** FROM OPENLINEAGEACCESSHISTORY WHERE EVENT:eventTime > system$gettag(...] +[2023-04-26T17:16:55.265+0000] {cursor.py:476} DEBUG - isfiletransfer: True +[2023-04-26T17:16:55.265+0000] {connection.py:1035} DEBUG - cmdquery +[2023-04-26T17:16:55.265+0000] {connection.py:1062} DEBUG - sql=[SELECT ** FROM OPENLINEAGEACCESSHISTORY WHERE EVENT:eventTime > system$gettag(...], sequenceid=[3], isfiletransfer=[False] +[2023-04-26T17:16:55.266+0000] {network.py:1162} DEBUG - Session status for SessionPool '[PRIVATE}', SessionPool 1/1 active sessions +[2023-04-26T17:16:55.267+0000] {network.py:850} DEBUG - remaining request timeout: None, retry cnt: 1 +[2023-04-26T17:16:55.268+0000] {network.py:828} DEBUG - Request guid: aba82952-a5c2-4c6b-9c70-a10545b8772c +[2023-04-26T17:16:55.268+0000] {network.py:1021} DEBUG - socket timeout: 60 +[2023-04-26T17:17:21.844+0000] {connectionpool.py:465} DEBUG - [PRIVATE] "POST /queries/v1/query-request?requestId=21e2ab85-4995-4010-865d-df06cf5ee5b5&requestguid=aba82952-a5c2-4c6b-9c70-a10545b8772c HTTP/1.1" 200 None +[2023-04-26T17:17:21.879+0000] {network.py:1047} DEBUG - SUCCESS +[2023-04-26T17:17:21.881+0000] {network.py:1168} DEBUG - Session status for SessionPool '[PRIVATE}', SessionPool 0/1 active sessions +[2023-04-26T17:17:21.882+0000] {network.py:729} DEBUG - ret[code] = None, after post request +[2023-04-26T17:17:21.882+0000] {network.py:751} DEBUG - Query id: 01abe3ac-0603-4df4-0042-c78307975eb6 +[2023-04-26T17:17:21.882+0000] {cursor.py:807} DEBUG - sfqid: 01abe3ac-0603-4df4-0042-c78307975eb6 +[2023-04-26T17:17:21.882+0000] {cursor.py:813} INFO - query execution done +[2023-04-26T17:17:21.883+0000] {cursor.py:827} DEBUG - SUCCESS +[2023-04-26T17:17:21.883+0000] {cursor.py:846} DEBUG - PUT OR GET: False +[2023-04-26T17:17:21.883+0000] {cursor.py:941} DEBUG - Query result format: arrow +[2023-04-26T17:17:21.903+0000] {resultbatch.py:102} DEBUG - chunk size=256 +[2023-04-26T17:17:21.920+0000] {cursor.py:956} INFO - Number of results in first chunk: 112 +[2023-04-26T17:17:21.949+0000] {arrowiterator.cpython-37m-x8664-linux-gnu.so:0} DEBUG - Batches read: 1 +[2023-04-26T17:17:21.950+0000] {CArrowIterator.cpp:16} DEBUG - Arrow BatchSize: 1 +[2023-04-26T17:17:21.950+0000] {CArrowChunkIterator.cpp:50} DEBUG - Arrow chunk info: batchCount 1, columnCount 1, usenumpy: 0 +[2023-04-26T17:17:21.950+0000] {resultset.py:232} DEBUG - result batch 1 has id: data001 +[2023-04-26T17:17:21.951+0000] {resultset.py:232} DEBUG - result batch 2 has id: data002 +[2023-04-26T17:17:21.951+0000] {resultset.py:232} DEBUG - result batch 3 has id: data003 +[2023-04-26T17:17:21.951+0000] {resultset.py:232} DEBUG - result batch 4 has id: data010 +[2023-04-26T17:17:21.951+0000] {resultset.py:232} DEBUG - result batch 5 has id: data011 +[2023-04-26T17:17:21.951+0000] {resultset.py:232} DEBUG - result batch 6 has id: data012 +[2023-04-26T17:17:21.952+0000] {resultset.py:232} DEBUG - result batch 7 has id: data013 +[2023-04-26T17:17:21.952+0000] {resultset.py:232} DEBUG - result batch 8 has id: data020 +[2023-04-26T17:17:21.952+0000] {resultset.py:232} DEBUG - result batch 9 has id: data02_1

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-04-26 14:45:26
 +
+

*Thread Reply:* I don't see any Airflow standard logs here, but anyway I looked at it and debugging it would not work if you're bypassing OpenLineageClient.emit and going directly to transport - the logging is done on Client level https://github.com/OpenLineage/OpenLineage/blob/acc207d63e976db7c48384f04bc578409f08cc8a/client/python/openlineage/client/client.py#L73

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/acc207d63e976db7c48384f04bc578409f08cc8a/client/python/openlineage/client/client.py | client.py>
+
+ <pre><code> f"OpenLineageClient will emit event {Serde.to_json(event).encode('utf-8')}" +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tom van Eijk + (t.m.h.vaneijk@tilburguniversity.edu) +
+
2023-04-27 11:16:20
 +
+

*Thread Reply:* I'm sorry, do you have a code snippet on how to get these logs from https://github.com/Snowflake-Labs/OpenLineage-AccessHistory-Setup/blob/main/examples/airflow/dags/lineage/extract_openlineage.py? I still get the ValueError for OpenLineageClient.emit

+
+ + + + +
<https://github.com/Snowflake-Labs/OpenLineage-AccessHistory-Setup/blob/main/examples/airflow/dags/lineage/extract_openlineage.py | extract_openlineage.py>
+
+ <pre><code>import json +import os +from pendulum import datetime + +from airflow import DAG +from airflow.decorators import task +from openlineage.client import OpenLineageClient +from snowflake.connector import connect + +SNOWFLAKE_USER = os.getenv('SNOWFLAKE_USER') +SNOWFLAKE_PASSWORD = os.getenv('SNOWFLAKE_PASSWORD') +SNOWFLAKE_ACCOUNT = os.getenv('SNOWFLAKE_ACCOUNT') + + +@task +def send_ol_events(): + client = OpenLineageClient.from_environment() + + with connect( + user=SNOWFLAKE_USER, + password=SNOWFLAKE_PASSWORD, + account=SNOWFLAKE_ACCOUNT, + database='OPENLINEAGE', + schema='PUBLIC', + ) as conn: + with conn.cursor() as cursor: + ol_view = 'OPENLINEAGE_ACCESS_HISTORY' + ol_event_time_tag = 'OL_LATEST_EVENT_TIME' + + var_query = f''' + use warehouse {SNOWFLAKE_WAREHOUSE}; + ''' + + cursor.execute(var_query) + + var_query = f''' + set current_organization='{SNOWFLAKE_ACCOUNT}'; + ''' + + cursor.execute(var_query) + + ol_query = f''' + SELECT ** FROM {ol_view} + WHERE EVENT:eventTime &gt; system$get_tag('{ol_event_time_tag}', '{ol_view}', 'table') + ORDER BY EVENT:eventTime ASC; + ''' + + cursor.execute(ol_query) + ol_events = [json.loads(ol_event[0]) for ol_event in cursor.fetchall()] + + for ol_event in ol_events: + client.emit(ol_event) + + if len(ol_events) &gt; 0: + latest_event_time = ol_events[-1]['eventTime'] + cursor.execute(f''' + ALTER VIEW {ol_view} SET TAG {ol_event_time_tag} = '{latest_event_time}'; + ''') + + +with DAG( + 'etl_openlineage', + start_date=datetime(2022, 4, 12), + schedule_interval='@hourly', + catchup=False, + default_args={ + 'owner': 'openlineage', + 'depends_on_past': False, + 'email_on_failure': False, + 'email_on_retry': False, + 'email': ['&lt;a href="mailto:demo@openlineage.io"&gt;demo@openlineage.io&lt;/a&gt;'], + 'snowflake_conn_id': 'openlineage_snowflake' + }, + description='Send OL events every minutes.', + tags=["extract"], +) as dag: + send_ol_events() + +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/Snowflake-Labs/OpenLineage-AccessHistory-Setup">Snowflake-Labs/OpenLineage-AccessHistory-Setup</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tom van Eijk + (t.m.h.vaneijk@tilburguniversity.edu) +
+
2023-05-04 10:56:34
 +
+

*Thread Reply:* Hey does anyone have an idea on this? I'm still stuck on this issue 😞

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-05-05 08:58:49
 +
+

*Thread Reply:* I've found the root cause. It's because facets don't have _producer and _schemaURL set. I'll provide a fix soon

+ + + +
+ ♥️ Tom van Eijk, Michael Robinson +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-04-26 11:36:23
 +
+

The first New York OpenLineage Meetup is happening today at 5:30 pm ET at Astronomer’s offices in the Flatiron District! https://openlineage.slack.com/archives/C01CK9T7HKR/p1681931978353159

+
+ + +
+ + + } + + Michael Robinson + (https://openlineage.slack.com/team/U02LXF3HUN7) +
+ + + +
+
+ <p>Here are the details about next week’s OpenLineage Meetup at Astronomer’s NY offices: <a href="https://openlineage.io/blog/nyc-meetup">https://openlineage.io/blog/nyc-meetup</a>. Hope to see you there if you can make it!</p> +
+ + + + + + +
Original URL: https://openlineage.slack.com/archives/C01CK9T7HKR/p1681931978353159
+ + +
+ + Slack Conversation +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2023-04-26 11:36:57
 +
+

*Thread Reply:* I’ll be there! I’m looking forward to see you all.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2023-04-26 11:37:23
 +
+

*Thread Reply:* We’ll talk about the evolution of the spec.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-04-27 02:55:00
 +
+

delta_table = DeltaTable.forPath(spark, path) +delta_table.alias("source").merge(df.alias("update"),lookup_statement).whenMatchedUpdateAll().whenNotMatchedInsertAll().execute() +If I write based on df operations like this, I notice that OL does not emit any event. May I know whether these or similar cases can be supported too? 🙇

+ + + +
+ 👀 Paweł Leszczyński +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-04-28 04:23:24
 +
+

*Thread Reply:* I've created an integration test based on your example. The Openlineage event gets sent, however it does not contain output dataset. I will look deeper into that.

+ + + +
+ :gratitude_thank_you: Anirudh Shrinivason +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-04-28 08:55:43
 +
+

*Thread Reply:* Hey, sorry do you mean input dataset is empty? Or output dataset?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-04-28 08:55:51
 +
+

*Thread Reply:* I am seeing that input dataset is empty

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-04-28 08:56:05
 +
+

*Thread Reply:* ooh, I see input datasets

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-04-28 08:56:11
 +
+

*Thread Reply:* Hmm

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-04-28 08:56:12
 +
+

*Thread Reply:* I see

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-04-28 08:57:07
 +
+

*Thread Reply:* I create a test in SparkDeltaIntegrationTest class a test method: +```@Test + void testDeltaMergeInto() { + Dataset<Row> dataset = + spark + .createDataFrame( + ImmutableList.of( + RowFactory.create(1L, "bat"), + RowFactory.create(2L, "mouse"), + RowFactory.create(3L, "horse") + ), + new StructType( + new StructField[] { + new StructField("a", LongType$.MODULE$, false, Metadata.empty()), + new StructField("b", StringType$.MODULE$, false, Metadata.empty()) + })) + .repartition(1); + dataset.createOrReplaceTempView("temp");

+ +
spark.sql("CREATE TABLE t1 USING delta LOCATION '/tmp/delta/t1' AS SELECT ** FROM temp");
+spark.sql("CREATE TABLE t2 USING delta LOCATION '/tmp/delta/t2' AS SELECT ** FROM temp");
+
+DeltaTable.forName("t1")
+    .merge(spark.read().table("t2"),"t1.a = t2.a")
+    .whenMatched().updateAll()
+    .whenNotMatched().insertAll()
+    .execute();
+
+verifyEvents(mockServer, "pysparkDeltaMergeIntoCompleteEvent.json");
+
+ +

}```

+ + + +
+ 👍 Anirudh Shrinivason +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-04-28 08:59:14
 +
+

*Thread Reply:* Oh yeah my bad. I am seeing output dataset is empty.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-04-28 08:59:21
 +
+

*Thread Reply:* Checks out with your observation

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-05-03 23:23:36
 +
+

*Thread Reply:* Hi @Paweł Leszczyński just curious, has a fix for this been implemented alr?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-05-04 02:40:11
 +
+

*Thread Reply:* Hi @Anirudh Shrinivason, I had some days ooo. I will look into this soon.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-05-04 07:37:52
 +
+

*Thread Reply:* Ahh okie! Thanks so much! Hope you had a good rest!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-05-04 07:38:38
 +
+

*Thread Reply:* yeah. this was an amazing extended weekend 😉

+ + + +
+ 🎉 Anirudh Shrinivason +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-05-05 02:09:10
 +
+

*Thread Reply:* This should be it: https://github.com/OpenLineage/OpenLineage/pull/1823

+
+ + + + +
#1823 Spark/delta merge into
+
+ <p><strong>Problem</strong></p> + +<p><code>merge into</code> queries are not supported for delta tables.</p> + +<p><strong>Solution</strong></p> + +<p>MergeIntoCommand is transformed into following plan:</p> + +<pre><code> == Optimized Logical Plan == +SerializeFromObject ... ++- MapPartitions org.apache.spark.sql.delta.commands.MergeIntoCommand$$Lambda$ ... + +- Join FullOuter, (a#815L = a#819L) + :- Project [a#819L, b#820, UDF() AS _source_row_present_#1052] + : +- Relation[a#819L,b#820] parquet + +- Project [a#815L, b#816, true AS _target_row_present_#1058] + +- Relation[a#815L,b#816] parquet +</code></pre> + +<p>Within the implementation, we check if there is a full outer join with a joined projection containing field <code>_target_row_present_</code>. If so, it extracts output dataset from that projection.</p> + +<blockquote> + <p><strong>Note:</strong> All schema changes require discussion. Please <a href="https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue">link the issue</a> for context.</p> +</blockquote> + +<p>• [] Your change modifies the <a href="https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.json">core</a> OpenLineage model +☐ Your change modifies one or more OpenLineage <a href="https://github.com/OpenLineage/OpenLineage/tree/main/spec/facets">facets</a></p> + +<p>If you're contributing a new integration, please specify the scope of the integration and how/where it has been tested (e.g., Apache Spark integration supports <code>S3</code> and <code>GCS</code> filesystem operations, tested with AWS EMR).</p> + +<p><strong>One-line summary:</strong> +<strong>Checklist</strong></p> + +<p>☑︎ You've <a href="https://github.com/OpenLineage/OpenLineage/blob/main/why-the-dco.md">signed-off</a> your work +☑︎ Your pull request title follows our <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CONTRIBUTING.md#creating-pull-requests">guidelines</a> +☑︎ Your changes are accompanied by tests (<em>if relevant</em>) +☑︎ Your change contains a <a href="https://kurtisnusbaum.medium.com/stacked-diffs-keeping-phabricator-diffs-small-d9964f4dcfa6">small diff</a> and is self-contained +☐ You've updated any relevant documentation (<em>if relevant</em>) +☐ Your comment includes a one-liner for the changelog about the specific purpose of the change (<em>if necessary</em>) +☐ You've versioned the core OpenLineage model or facets according to <a href="https://docs.snowplowanalytics.com/docs/pipeline-components-and-applications/iglu/common-architecture/schemaver">SchemaVer</a> (<em>if relevant</em>) +☐ You've added a <a href="https://github.com/OpenLineage/OpenLineage/tree/main/.github/header_templates.md">header</a> to source files (<em>if relevant</em>)</p> + +<hr /> + +<p>SPDX-License-Identifier: Apache-2.0 <br /> +Copyright 2018-2023 contributors to the OpenLineage project</p> +
+ +
+
Labels
+ integration/spark +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-05-05 02:43:24
 +
+

*Thread Reply:* Hi @Anirudh Shrinivason, please let me know if there is still something to be done within #1747 PROPOSAL] Support for V2SessionCatalog. I could not reproduce exactly what you described but fixed some issue nearby.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-05-05 02:49:38
 +
+

*Thread Reply:* Hmm yeah sure let me find out the exact cause of the issue. The pipeline that was causing the issue is now inactive haha. So I'm trying to backtrace from the limited logs I captured last time. Let me get back by next week thanks! 🙇

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-05-05 09:35:00
 +
+

*Thread Reply:* Hi @Paweł Leszczyński I was trying to replicate the issue from my end, but couldn't do so. I think we can close the issue for now, and revisit later on if the issue resurfaces. Does that sound okay?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-05-05 09:40:33
 +
+

*Thread Reply:* sounds cool. we can surely create a new issue later on.

+ + + +
+ 👍 Anirudh Shrinivason +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harshini Devathi + (harshini.devathi@tigeranalytics.com) +
+
2023-05-09 23:34:04
 +
+

*Thread Reply:* @Paweł Leszczyński - I was trying to implement these new changes in databricks. I was wondering which java file should I use for building the jar file? Could you plese help me?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-06-09 00:46:34
 +
+

*Thread Reply:* .

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-06-09 02:37:49
 +
+

*Thread Reply:* Hi I found that these merge operations have no input datasets/col lineage: +```df.write.format(fileformat).mode(mode).option("mergeSchema", mergeschema).option("overwriteSchema", overwriteSchema).save(path)

+ +

df.write.format(fileformat).mode(mode).option("mergeSchema", mergeschema).option("overwriteSchema", overwriteSchema)\ + .partitionBy(**partitions).save(path)

+ +

df.write.format(fileformat).mode(mode).option("mergeSchema", mergeschema).option("overwriteSchema", overwriteSchema)\ + .partitionBy(**partitions).option("replaceWhere", where_clause).save(path)`` +I also noticed the same issue when using theMERGE INTO` command from spark sql. +Would it be possible to extend the support to these df operations. too please? Thanks! +CC: @Paweł Leszczyński

+ + + +
+ 👀 Paweł Leszczyński +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-06-09 02:41:24
 +
+

*Thread Reply:* Hi @Anirudh Shrinivason, great to hear from you. Could you create an issue out of this? I am working at the moment on Spark 3.4. Once this is ready, I will look at the spark issues. And this one seems to be nicely reproducible. Thanks for that.

+ + + +
+ 👍 Anirudh Shrinivason +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-06-09 02:49:56
 +
+

*Thread Reply:* Sure let me create an issue! Thanks!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-06-09 02:55:21
 +
+

*Thread Reply:* Created an issue here! https://github.com/OpenLineage/OpenLineage/issues/1919 +Thanks! 🙇

+
+ + + + +
#1919 Lineage not captured for MergeIntoCommands
+
+ <p>Hi, I found that these merge operations have no input datasets/col lineage:</p> + +<pre><code>df.write.format(file_format).mode(mode).option("mergeSchema", merge_schema).option("overwriteSchema", overwriteSchema).save(path) + +df.write.format(file_format).mode(mode).option("mergeSchema", merge_schema).option("overwriteSchema", overwriteSchema)\ + .partitionBy(**partitions).save(path) + +df.write.format(file_format).mode(mode).option("mergeSchema", merge_schema).option("overwriteSchema", overwriteSchema)\ + .partitionBy(**partitions).option("replaceWhere", where_clause).save(path) +</code></pre> + +<p>I also noticed the same issue when using the <code>MERGE INTO</code> command from spark sql. <br /> +Would it be possible to extend the support to these df operations. too please? Thanks!</p> +
+ +
+
Labels
+ proposal +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-06-15 10:39:50
 +
+

*Thread Reply:* Hi @Paweł Leszczyński I just realised, https://github.com/OpenLineage/OpenLineage/pull/1823/files +This PR doesn't actually capture column lineage for the MergeIntoCommand? It looks like there is no column lineage field in the events json.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-06-17 04:21:24
 +
+

*Thread Reply:* Hi @Paweł Leszczyński Is there a potential timeline in mind to support column lineage for the MergeIntoCommand? We're really excited for this feature and would be a huge help to overcome a current blocker. Thanks!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-04-28 14:11:34
 +
+

Thanks to everyone who came out to Wednesday night’s meetup in New York! In addition to great pizza from Grimaldi’s (thanks for the tip, @Harel Shein), we enjoyed a spirited discussion of: +• the state of observability tooling in the data space today +• the history and high-level architecture of the project courtesy of @Julien Le Dem +• exciting news of an OpenLineage Scanner being planned at MANTA courtesy of @Ernie Ostic +• updates on the project roadmap and some exciting proposals from @Julien Le Dem, @Harel Shein and @Willy Lulciuc +• an introduction to and demo of Marquez from project lead @Willy Lulciuc +• and more. +Be on the lookout for an announcement about the next meetup!

+ +
+
+ + + + + + +
+ + +
+ ❤️ Harel Shein, Maciej Obuchowski, Peter Hicks, Jakub Dardziński, Atif Tahir +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-04-28 16:02:22
 +
+

As discussed during the April TSC meeting, comments are sought from the community on a proposal to support RunEvent-less (AKA static) lineage metadata emission. This is currently a WIP. For details and to comment, please see: +• https://docs.google.com/document/d/1366bAPkk0OqKkNA4mFFt-41X0cFUQ6sOvhSWmh4Iydo/edit?usp=sharing +• https://docs.google.com/document/d/1gKJw3ITJHArTlE-Iinb4PLkm88moORR0xW7I7hKZIQA/edit?usp=sharing

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ernie Ostic + (ernie.ostic@getmanta.com) +
+
2023-04-30 21:35:47
 +
+

Hi all. Probably I just need to study the spec further, but what is the significance of _producer vs producer in the context of where they are used? (same question also for _schemaURL vs schemaURL)? Thx!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sheeri Cabral (Collibra) + (sheeri.cabral@collibra.com) +
+
2023-05-01 12:02:13
 +
+

*Thread Reply:* “producer” is an element of the event run itself - e.g. what produced the JSON packet you’re studying. There is only one of these per event run. You can think of it as a top-level property.

+ +

“producer” (and “schemaURL”) are elements of a facet. They are the 2 required elements for any customized facet (though I don’t agree they should be required, or at least I believe they should be able to be compatible with a blank value and a null value).

+ +

A packet sent to an API should only have one “producer” element, but can have many _producer elements in sub-objects (though, only one _producer per facet).

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ernie Ostic + (ernie.ostic@getmanta.com) +
+
2023-05-01 12:06:52
 +
+

*Thread Reply:* just curious --- is/was there any specific reason for the underscore prefix? If they are in a facet, they would already be qualified.......

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sheeri Cabral (Collibra) + (sheeri.cabral@collibra.com) +
+
2023-05-01 13:13:28
 +
+

*Thread Reply:* The facet “BaseFacet” that’s used for customization, has 2 required elements - _producer and _schemaURL. so I don’t believe it’s related to qualification.

+ + + +
+ 👍 Ernie Ostic +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-05-01 11:33:02
 +
+

I’m opening a vote to release OpenLineage 0.24.0, including: +• a new OpenLineage extractor for dbt Cloud +• a new interface - TransportBuilder - for creating custom transport types without modifying core components of OpenLineage +• a fix to the LogicalPlanSerializer in the Spark integration to make it operational again +• a new configuration parameter in the Spark integration for making dataset paths less verbose +• a fix to the Flink integration CI +• and more. + Three +1s from committers will authorize an immediate release.

+ + + +
+ ➕ Jakub Dardziński, Willy Lulciuc, Julien Le Dem +
+ +
+ ✅ Sheeri Cabral (Collibra) +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-05-02 19:43:12
 +
+

*Thread Reply:* Thanks for voting. The release will commence within 2 days.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sheeri Cabral (Collibra) + (sheeri.cabral@collibra.com) +
+
2023-05-01 12:03:19
 +
+

Does the Spark integration for OpenLineage also support ETL that uses the Apache Spark Structured Streaming framework?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-05-04 02:33:32
 +
+

*Thread Reply:* Although it is not documented, we do have an integration test for that: https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/app/src/test/resources/spark_scripts/spark_kafka.py

+ +

The test reads and writes data to Kafka and verifies if input/output datasets are collected.

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/app/src/test/resources/spark_scripts/spark_kafka.py | spark_kafka.py>
+
+ <pre><code># Copyright 2018-2023 contributors to the OpenLineage project +# SPDX-License-Identifier: Apache-2.0 + +from pyspark.sql import SparkSession + +appName = "Kafka Examples" +master = "local" + +spark = SparkSession.builder \ + .master(master) \ + .appName(appName) \ + .getOrCreate() + +kafka_servers = "kafka:9092" +checkpointLocation = '/test_data/checkpoint' + + +people = spark.createDataFrame([("Bilbo Baggins", 50), ("Gandalf", 1000), ("Thorin", 195), ("Balin", 178), ("Kili", 77), + ("Dwalin", 169), ("Oin", 167), ("Gloin", 158), ("Fili", 82), ("Bombur", None)], ["name", "age"]) + +ds = people \ + .selectExpr("CAST(name AS STRING) as key", "CAST(age AS STRING) as value") \ + .write \ + .format("kafka") \ + .option("kafka.bootstrap.servers", kafka_servers) \ + .option("topic", "topic1") \ + .option("checkpointLocation", checkpointLocation) \ + .save() + +df = spark \ + .read \ + .format("kafka") \ + .option("kafka.bootstrap.servers", kafka_servers) \ + .option("subscribe", "topic1") \ + .load() + +df.show() + +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+ ✅ Sheeri Cabral (Collibra) +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sheeri Cabral (Collibra) + (sheeri.cabral@collibra.com) +
+
2023-05-01 13:14:14
 +
+

Also, does it work for pyspark jobs? (Forgive me if Spark job = pyspark, I don’t have a lot of depth on how Spark works.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-05-01 22:37:25
 +
+

*Thread Reply:* From my experience, yeah it works for pyspark

+ + + +
+ 🙌 Paweł Leszczyński, Sheeri Cabral (Collibra) +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sheeri Cabral (Collibra) + (sheeri.cabral@collibra.com) +
+
2023-05-01 13:35:41
 +
+

(and in a less generic question, would it work on top of this Spline agent/lineage harvester, or is it a replacement for it?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-05-01 22:39:18
 +
+

*Thread Reply:* Also from my experience, I think we can only use one of them as we can only configure one spark listener... correct me if I'm wrong. But it seems like the latest releases of spline are already using openlineage to some capacity?

+ + + +
+ ✅ Sheeri Cabral (Collibra) +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-05-08 09:46:15
 +
+

*Thread Reply:* In spark.extraListeners you can configure multiple listeners by comma separating them - I think you can use multiple ones with OpenLineage without obvious problems. I think we do pretty similar things to Spline though

+ + + +
+ 👍 Anirudh Shrinivason +
+ +
+ ✅ Sheeri Cabral (Collibra) +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sheeri Cabral (Collibra) + (sheeri.cabral@collibra.com) +
+
2023-05-25 11:28:41
 +
+

*Thread Reply:* (I never said thank you for this, so, thank you!)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sai + (saivenkatesh161@gmail.com) +
+
2023-05-02 04:03:40
 +
+

Hi Team,

+ +

I have configured Open lineage with databricks and it is sending events to Marquez as expected. I have a notebook which joins 3 tables and write the result data frame to an azure adls location. Each time I run the notebook manually, it creates two start events and two complete events for one run as shown in the screenshot. Is this something expected or I am missing something?

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-05-02 10:45:37
 +
+

*Thread Reply:* Hello Sai, thanks for your question! A number of folks who could help with this are OOO, but someone will reply as soon as possible.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-05-04 02:44:46
 +
+

*Thread Reply:* That is interesting @Sai. Are you able to reproduce this with a simple code snippet? Which Openlineage version are you using?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sai + (saivenkatesh161@gmail.com) +
+
2023-05-05 01:16:20
 +
+

*Thread Reply:* Yes @Paweł Leszczyński. Each join query I run on top of delta tables have two start and two complete events. We are using below jar for openlineage.

+ +

openlineage-spark-0.22.0.jar

+ + + +
+ 👀 Paweł Leszczyński +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-05-05 02:41:26
 +
+

*Thread Reply:* https://github.com/OpenLineage/OpenLineage/issues/1828

+
+ + + + +
#1828 [Spark] duplicate events when for delta
+
+ <p>Each join query run on top of delta tables have two start and two complete events. Openlineage 0.22..00 <br /> +<a href="https://openlineage.slack.com/archives/C01CK9T7HKR/p1683014620909079">https://openlineage.slack.com/archives/C01CK9T7HKR/p1683014620909079</a></p> +
+ +
+
Assignees
+ <a href="https://github.com/pawel-big-lebowski">@pawel-big-lebowski</a> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sai + (saivenkatesh161@gmail.com) +
+
2023-05-08 04:05:26
 +
+

*Thread Reply:* Hi @Paweł Leszczyński any updates on this issue?

+ +

Also, OL is not giving column level lineage for group by operations on tables. Is this expected?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-05-08 04:07:04
 +
+

*Thread Reply:* Hi @Sai, https://github.com/OpenLineage/OpenLineage/pull/1830 should fix duplication issue

+
+ + + + +
#1830 [SPARK] filter delta adaptive plan events
+
+ <p><strong>Problem</strong></p> + +<p>Spark connector sends duplicate events when processing delta tables.</p> + +<p>Closes: <a href="https://github.com/OpenLineage/OpenLineage/issues/1828">#1828</a></p> + +<p><strong>Solution</strong></p> + +<p>The problem occurs for adaptive Spark plans when OL events are generated extra for adaptive plans when query is rewritten. As a solution, <code>DeltaEventFilter</code> class is extended to filter the events in such case.</p> + +<blockquote> + <p><strong>Note:</strong> All schema changes require discussion. Please <a href="https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue">link the issue</a> for context.</p> +</blockquote> + +<p>☐ Your change modifies the <a href="https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.json">core</a> OpenLineage model +☐ Your change modifies one or more OpenLineage <a href="https://github.com/OpenLineage/OpenLineage/tree/main/spec/facets">facets</a></p> + +<p>If you're contributing a new integration, please specify the scope of the integration and how/where it has been tested (e.g., Apache Spark integration supports <code>S3</code> and <code>GCS</code> filesystem operations, tested with AWS EMR).</p> + +<p><strong>One-line summary:</strong> +<strong>Checklist</strong></p> + +<p>☑︎ You've <a href="https://github.com/OpenLineage/OpenLineage/blob/main/why-the-dco.md">signed-off</a> your work +☑︎ Your pull request title follows our <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CONTRIBUTING.md#creating-pull-requests">guidelines</a> +☑︎ Your changes are accompanied by tests (<em>if relevant</em>) +☑︎ Your change contains a <a href="https://kurtisnusbaum.medium.com/stacked-diffs-keeping-phabricator-diffs-small-d9964f4dcfa6">small diff</a> and is self-contained +☐ You've updated any relevant documentation (<em>if relevant</em>) +☐ Your comment includes a one-liner for the changelog about the specific purpose of the change (<em>if necessary</em>) +☐ You've versioned the core OpenLineage model or facets according to <a href="https://docs.snowplowanalytics.com/docs/pipeline-components-and-applications/iglu/common-architecture/schemaver">SchemaVer</a> (<em>if relevant</em>) +☐ You've added a <a href="https://github.com/OpenLineage/OpenLineage/tree/main/.github/header_templates.md">header</a> to source files (<em>if relevant</em>)</p> + +<hr /> + +<p>SPDX-License-Identifier: Apache-2.0 <br /> +Copyright 2018-2023 contributors to the OpenLineage project</p> +
+ +
+
Labels
+ documentation, integration/spark +
+ +
+
Assignees
+ <a href="https://github.com/pawel-big-lebowski">@pawel-big-lebowski</a> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sai + (saivenkatesh161@gmail.com) +
+
2023-05-08 04:08:06
 +
+

*Thread Reply:* this would be part of next release?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-05-08 04:08:30
 +
+

*Thread Reply:* Regarding column lineage & group by issue, I think it's something on databricks side -> we do have an open issue for that #1821

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-05-08 04:09:24
 +
+

*Thread Reply:* once #1830 is reviewed and merged, it will be the part of the next relase

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sai + (saivenkatesh161@gmail.com) +
+
2023-05-08 04:11:01
 +
+

*Thread Reply:* sure.. thanks @Paweł Leszczyński

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sai + (saivenkatesh161@gmail.com) +
+
2023-05-16 03:27:01
 +
+

*Thread Reply:* @Paweł Leszczyński I have used the latest jar (0.25.0) and still this issue persists. I see two events for same input/output lineage.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Thomas + (xsist10@gmail.com) +
+
2023-05-03 03:55:44
 +
+

Has anyone used Open Lineage for application lineage? I'm particularly interested in how if/how you handled service boundaries like APIs and Kafka topics and what Dataset Naming (URI) you used.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Thomas + (xsist10@gmail.com) +
+
2023-05-03 04:06:37
 +
+

*Thread Reply:* For example, MySQL is stored as producer + host + port + database + table as something like <mysql://db.foo.com:6543/metrics.orders> +For an API (especially one following REST conditions), I was thinking something like method + host + port + path or GET <https://api.service.com:433/v1/users>

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-05-03 10:13:25
 +
+

*Thread Reply:* Hi Thomas, thanks for asking about this — it sounds cool! I don’t know of others working on this kind of thing, but I’ve been developing a SQLAlchemy integration and have been experimenting with job naming — which I realize isn’t exactly what you’re working on. Hopefully others will chime in here, but in the meantime, would you be willing to create an issue about this? It seems worth discussing how we could expand the spec for this kind of use case.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Thomas + (xsist10@gmail.com) +
+
2023-05-03 10:58:32
 +
+

*Thread Reply:* I suspect this will definitely be a bigger discussion. Let me ponder on the problem a bit more and come back with something a bit more concrete.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-05-03 10:59:21
 +
+

*Thread Reply:* Looking forward to hearing more!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Thomas + (xsist10@gmail.com) +
+
2023-05-03 11:05:47
 +
+

*Thread Reply:* On a tangential note, does OpenLineage's column level lineage have support for (I see it can be extended but want to know if someone had to map this before): +• Properties as a path in a structure (like a JSON structure, Avro schema, protobuf, etc) maybe using something like JSON Path or XPath notation. +• Fragments (when a column is a JSON blob, there is an entire sub-structure that needs to be described) +• Transformation description (how an input affects an output. Is it a direct copy of the value or is it part of a formula)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-05-03 11:22:21
 +
+

*Thread Reply:* I don’t know, but I’ll ping some folks who might.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-05-04 03:24:01
 +
+

*Thread Reply:* Hi @Thomas. Column-lineage support currently does not include json fields. We have included in specification fields like transformationDescription and transformationType to store a string representation of the transformation applied and its type like IDENTITY|MASKED. However, those fields aren't filled within Spark integration at the moment.

+ + + +
+ 🙌 Thomas, Michael Robinson +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-05-03 09:54:57
 +
+

@channel +We released OpenLineage 0.24.0, including: +Additions: +• Support custom transport types #1795 @nataliezeller1 +• Airflow: dbt Cloud integration #1418 @howardyoo +• Spark: support dataset name modification using regex #1796 @pawel-big-lebowski +Plus bug fixes and more. +Thanks to all the contributors! +For the bug fixes and details, see: +Release: https://github.com/OpenLineage/OpenLineage/releases/tag/0.24.0 +Changelog: https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md +Commit history: https://github.com/OpenLineage/OpenLineage/compare/0.23.0...0.24.0 +Maven: https://oss.sonatype.org/#nexus-search;quick~openlineage +PyPI: https://pypi.org/project/openlineage-python/

+ + + +
+ 🎉 Harel Shein, tati +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
GreetBot + +
+
2023-05-03 10:45:32
 +
+

@GreetBot has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-05-04 11:25:23
 +
+

@channel +This month’s TSC meeting is next Thursday, May 11th, at 10:00 am PT. The tentative agenda will be on the wiki. More info and the meeting link can be found on the website. All are welcome! Also, feel free to reply or DM me with discussion topics, agenda items, etc.

+
+
openlineage.io
+ + + +
TSC Meetings | OpenLineage Docs
+
+ The OpenLineage Technical Steering Committee meets monthly, and is open to all. +
+ + + + + + +
Original URL: https://openlineage.io/meetings/
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harshini Devathi + (harshini.devathi@tigeranalytics.com) +
+
2023-05-05 12:11:37
 +
+

Hello all, noticed that openlineage is not able to give column level lineage if there is a groupby operation on a spark dataframe. Has anyone else faced this issue and have any fixes or workarounds? Apache Spark 3.0.1 and Openlineage version 1 are being used. Also tried on Spark version 3.3.0

+ +

Log4j error details follow:

+ +

23/05/05 18:09:11 ERROR ColumnLevelLineageUtils: Error when invoking static method 'buildColumnLineageDatasetFacet' for Spark3 +java.lang.reflect.InvocationTargetException + at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) + at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) + at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) + at java.lang.reflect.Method.invoke(Method.java:498) + at io.openlineage.spark.agent.lifecycle.plan.column.ColumnLevelLineageUtils.buildColumnLineageDatasetFacet(ColumnLevelLineageUtils.java:35) + at io.openlineage.spark.agent.lifecycle.OpenLineageRunEventBuilder.lambda$buildOutputDatasets$21(OpenLineageRunEventBuilder.java:424) + at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) + at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) + at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) + at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472) + at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) + at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) + at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:566) + at io.openlineage.spark.agent.lifecycle.OpenLineageRunEventBuilder.buildOutputDatasets(OpenLineageRunEventBuilder.java:437) + at io.openlineage.spark.agent.lifecycle.OpenLineageRunEventBuilder.populateRun(OpenLineageRunEventBuilder.java:296) + at io.openlineage.spark.agent.lifecycle.OpenLineageRunEventBuilder.buildRun(OpenLineageRunEventBuilder.java:279) + at io.openlineage.spark.agent.lifecycle.OpenLineageRunEventBuilder.buildRun(OpenLineageRunEventBuilder.java:222) + at io.openlineage.spark.agent.lifecycle.SparkSQLExecutionContext.start(SparkSQLExecutionContext.java:70) + at io.openlineage.spark.agent.OpenLineageSparkListener.lambda$sparkSQLExecStart$0(OpenLineageSparkListener.java:91) + at java.util.Optional.ifPresent(Optional.java:159) + at io.openlineage.spark.agent.OpenLineageSparkListener.sparkSQLExecStart(OpenLineageSparkListener.java:91) + at io.openlineage.spark.agent.OpenLineageSparkListener.onOtherEvent(OpenLineageSparkListener.java:82) + at org.apache.spark.scheduler.SparkListenerBus.doPostEvent(SparkListenerBus.scala:102) + at org.apache.spark.scheduler.SparkListenerBus.doPostEvent$(SparkListenerBus.scala:28) + at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:39) + at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:39) + at org.apache.spark.util.ListenerBus.postToAll(ListenerBus.scala:118) + at org.apache.spark.util.ListenerBus.postToAll$(ListenerBus.scala:102) + at org.apache.spark.scheduler.AsyncEventQueue.super$postToAll(AsyncEventQueue.scala:107) + at org.apache.spark.scheduler.AsyncEventQueue.$anonfun$dispatch$1(AsyncEventQueue.scala:107) + at scala.runtime.java8.JFunction0$mcJ$sp.apply(JFunction0$mcJ$sp.java:23) + at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62) + at org.apache.spark.scheduler.AsyncEventQueue.org$apache$spark$scheduler$AsyncEventQueue$$dispatch(AsyncEventQueue.scala:102) + at org.apache.spark.scheduler.AsyncEventQueue$$anon$2.$anonfun$run$1(AsyncEventQueue.scala:98) + at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1639) + at org.apache.spark.scheduler.AsyncEventQueue$$anon$2.run(AsyncEventQueue.scala:98) +Caused by: java.lang.NoSuchMethodError: org.apache.spark.sql.catalyst.expressions.aggregate.AggregateExpression.resultId()Lorg/apache/spark/sql/catalyst/expressions/ExprId; + at io.openlineage.spark3.agent.lifecycle.plan.column.ExpressionDependencyCollector.traverseExpression(ExpressionDependencyCollector.java:79) + at io.openlineage.spark3.agent.lifecycle.plan.column.ExpressionDependencyCollector.lambda$traverseExpression$4(ExpressionDependencyCollector.java:74) + at java.util.Iterator.forEachRemaining(Iterator.java:116) + at scala.collection.convert.Wrappers$IteratorWrapper.forEachRemaining(Wrappers.scala:31) + at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801) + at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:647) + at io.openlineage.spark3.agent.lifecycle.plan.column.ExpressionDependencyCollector.traverseExpression(ExpressionDependencyCollector.java:74) + at io.openlineage.spark3.agent.lifecycle.plan.column.ExpressionDependencyCollector.lambda$null$2(ExpressionDependencyCollector.java:60) + at java.util.LinkedList$LLSpliterator.forEachRemaining(LinkedList.java:1235) + at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:647) + at io.openlineage.spark3.agent.lifecycle.plan.column.ExpressionDependencyCollector.lambda$collect$3(ExpressionDependencyCollector.java:60) + at org.apache.spark.sql.catalyst.trees.TreeNode.foreach(TreeNode.scala:285) + at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$foreach$1(TreeNode.scala:286) + at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$foreach$1$adapted(TreeNode.scala:286) + at scala.collection.Iterator.foreach(Iterator.scala:943) + at scala.collection.Iterator.foreach$(Iterator.scala:943) + at scala.collection.AbstractIterator.foreach(Iterator.scala:1431) + at scala.collection.IterableLike.foreach(IterableLike.scala:74) + at scala.collection.IterableLike.foreach$(IterableLike.scala:73) + at scala.collection.AbstractIterable.foreach(Iterable.scala:56) + at org.apache.spark.sql.catalyst.trees.TreeNode.foreach(TreeNode.scala:286) + at io.openlineage.spark3.agent.lifecycle.plan.column.ExpressionDependencyCollector.collect(ExpressionDependencyCollector.java:38) + at io.openlineage.spark3.agent.lifecycle.plan.column.ColumnLevelLineageUtils.collectInputsAndExpressionDependencies(ColumnLevelLineageUtils.java:70) + at io.openlineage.spark3.agent.lifecycle.plan.column.ColumnLevelLineageUtils.buildColumnLineageDatasetFacet(ColumnLevelLineageUtils.java:40) + ... 36 more

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-05-08 07:38:19
 +
+

*Thread Reply:* Hi @Harshini Devathi, I think this the same as issue: https://github.com/OpenLineage/OpenLineage/issues/1821

+
+ + + + +
#1821 Azure databricks use aggregate funcation can not buildColumnLineageDatasetFacet
+
+ <p>Enviroment: <br /> +databricks 12.1 (includes Apache Spark 3.3.1, Scala 2.12) <br /> +Openlineage 0.21.1 <br /> +sql: <br /> +INSERT INTO TABLE openliange<em>test <br /> +SELECT <br /> +union</em>id, <br /> +first(name) AS name <br /> +FROM tmp <br /> +GROUP BY union_id; <br /> +when we use 'GROUP BY' function, the program report err: <br /> +23/03/10 07:49:15 ERROR ColumnLevelLineageUtils: Error when invoking static method 'buildColumnLineageDatasetFacet' for Spark3 <br /> +java.lang.reflect.InvocationTargetException <br /> +at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) <br /> +at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) <br /> +at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) <br /> +at java.lang.reflect.Method.invoke(Method.java:498) <br /> +at io.openlineage.spark.agent.lifecycle.plan.column.ColumnLevelLineageUtils.buildColumnLineageDatasetFacet(ColumnLevelLineageUtils.java:35) <br /> +at io.openlineage.spark.agent.lifecycle.OpenLineageRunEventBuilder.lambda$buildOutputDatasets$21(OpenLineageRunEventBuilder.java:424) <br /> +at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) <br /> +at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) <br /> +at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) <br /> +at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472) <br /> +at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) <br /> +at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) <br /> +at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:566) <br /> +at io.openlineage.spark.agent.lifecycle.OpenLineageRunEventBuilder.buildOutputDatasets(OpenLineageRunEventBuilder.java:437) <br /> +at io.openlineage.spark.agent.lifecycle.OpenLineageRunEventBuilder.populateRun(OpenLineageRunEventBuilder.java:296) <br /> +at io.openlineage.spark.agent.lifecycle.OpenLineageRunEventBuilder.buildRun(OpenLineageRunEventBuilder.java:279) <br /> +at io.openlineage.spark.agent.lifecycle.OpenLineageRunEventBuilder.buildRun(OpenLineageRunEventBuilder.java:231) <br /> +at io.openlineage.spark.agent.lifecycle.SparkSQLExecutionContext.end(SparkSQLExecutionContext.java:95) <br /> +at io.openlineage.spark.agent.OpenLineageSparkListener.sparkSQLExecEnd(OpenLineageSparkListener.java:98) <br /> +at io.openlineage.spark.agent.OpenLineageSparkListener.onOtherEvent(OpenLineageSparkListener.java:84) <br /> +at org.apache.spark.scheduler.SparkListenerBus.doPostEvent(SparkListenerBus.scala:102) <br /> +at org.apache.spark.scheduler.SparkListenerBus.doPostEvent$(SparkListenerBus.scala:28) <br /> +at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:42) <br /> +at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:42) <br /> +at org.apache.spark.util.ListenerBus.postToAll(ListenerBus.scala:118) <br /> +at org.apache.spark.util.ListenerBus.postToAll$(ListenerBus.scala:102) <br /> +at org.apache.spark.scheduler.AsyncEventQueue.super$postToAll(AsyncEventQueue.scala:114) <br /> +at org.apache.spark.scheduler.AsyncEventQueue.$anonfun$dispatch$1(AsyncEventQueue.scala:114) <br /> +at scala.runtime.java8.JFunction0$mcJ$sp.apply(JFunction0$mcJ$sp.java:23) <br /> +at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62) <br /> +at <a href="http://org.apache.spark.scheduler.AsyncEventQueue.org">org.apache.spark.scheduler.AsyncEventQueue.org</a>$apache$spark$scheduler$AsyncEventQueue$$dispatch(AsyncEventQueue.scala:109) <br /> +at org.apache.spark.scheduler.AsyncEventQueue$$anon$2.$anonfun$run$1(AsyncEventQueue.scala:105) <br /> +at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1655) <br /> +at org.apache.spark.scheduler.AsyncEventQueue$$anon$2.run(AsyncEventQueue.scala:105) <br /> +Caused by: java.lang.NoSuchMethodError: org.apache.spark.sql.catalyst.expressions.aggregate.AggregateExpression.resultId()Lorg/apache/spark/sql/catalyst/expressions/ExprId; <br /> +at io.openlineage.spark3.agent.lifecycle.plan.column.ExpressionDependencyCollector.traverseExpression(ExpressionDependencyCollector.java:71) <br /> +at io.openlineage.spark3.agent.lifecycle.plan.column.ExpressionDependencyCollector.lambda$traverseExpression$4(ExpressionDependencyCollector.java:66) <br /> +at java.util.Iterator.forEachRemaining(Iterator.java:116) <br /> +at scala.collection.convert.Wrappers$IteratorWrapper.forEachRemaining(Wrappers.scala:31) <br /> +at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801) <br /> +at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:647) <br /> +at io.openlineage.spark3.agent.lifecycle.plan.column.ExpressionDependencyCollector.traverseExpression(ExpressionDependencyCollector.java:66) <br /> +at io.openlineage.spark3.agent.lifecycle.plan.column.ExpressionDependencyCollector.lambda$null$2(ExpressionDependencyCollector.java:52) <br /> +at java.util.LinkedList$LLSpliterator.forEachRemaining(LinkedList.java:1235) <br /> +at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:647) <br /> +at io.openlineage.spark3.agent.lifecycle.plan.column.ExpressionDependencyCollector.lambda$collect$3(ExpressionDependencyCollector.java:52) <br /> +at org.apache.spark.sql.catalyst.trees.TreeNode.foreach(TreeNode.scala:292) <br /> +at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$foreach$1(TreeNode.scala:293) <br /> +at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$foreach$1$adapted(TreeNode.scala:293) <br /> +at scala.collection.Iterator.foreach(Iterator.scala:943) <br /> +at scala.collection.Iterator.foreach$(Iterator.scala:943) <br /> +at scala.collection.AbstractIterator.foreach(Iterator.scala:1431) <br /> +at scala.collection.IterableLike.foreach(IterableLike.scala:74) <br /> +at scala.collection.IterableLike.foreach$(IterableLike.scala:73) <br /> +at scala.collection.AbstractIterable.foreach(Iterable.scala:56) <br /> +at org.apache.spark.sql.catalyst.trees.TreeNode.foreach(TreeNode.scala:293) <br /> +at io.openlineage.spark3.agent.lifecycle.plan.column.ExpressionDependencyCollector.collect(ExpressionDependencyCollector.java:35) <br /> +at io.openlineage.spark3.agent.lifecycle.plan.column.ColumnLevelLineageUtils.buildColumnLineageDatasetFacet(ColumnLevelLineageUtils.java:32) <br /> +... 34 more</p> +
+ +
+
Assignees
+ <a href="https://github.com/pawel-big-lebowski">@pawel-big-lebowski</a> +
+ +
+
Labels
+ integration/spark, integration/databricks +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harshini Devathi + (harshini.devathi@tigeranalytics.com) +
+
2023-05-08 19:44:26
 +
+

*Thread Reply:* Thank you @Paweł Leszczyński. So, is this an issue with databricks. The issue thread says that it was able to work on AWS Glue. If so, is there some kind of solution to make it work on Databricks?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harshini Devathi + (harshini.devathi@tigeranalytics.com) +
+
2023-05-05 12:22:06
 +
+

Hello all, is there a way to get lineage in azure synapse analytics with openlineage

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2023-05-09 20:17:38
 +
+

*Thread Reply:* maybe @Will Johnson knows?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sai + (saivenkatesh161@gmail.com) +
+
2023-05-08 07:06:37
 +
+

Hi Team,

+ +

I have a usecase where we are connecting to Azure sql database from databricks to extract, transform and load data to delta tables. I could see the lineage is getting build, but there is no column level lineage through its 1:1 mapping from source. Could you please check and update on this.

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-05-09 10:06:02
 +
+

*Thread Reply:* There are few possible issues:

+ +
  1. The column-level lineage is not implemented for particular part of Spark LogicalPlan
  2. Azure SQL or Databricks have their own implementations of some Spark class, which does not exactly match our extractor. We've seen that happen
  3. You're using SQL JDBC connection with SELECT ** - in which case we can't do anything for now, since we don't know the input columns.
  4. Possibly something else 🙂 @Paweł Leszczyński might have an idea +To fully understand the issue, we'd have to see logs, LogicalPlan of the Spark job, or the job code itself
    5. +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-05-10 02:35:32
 +
+

*Thread Reply:* @Sai, providing a short code snippet that is able to reproduce this would be super helpful in examining that.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sai + (saivenkatesh161@gmail.com) +
+
2023-05-10 02:59:24
 +
+

*Thread Reply:* sure Pawel +Will share the code I used in sometime

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sai + (saivenkatesh161@gmail.com) +
+
2023-05-10 03:37:54
 +
+

*Thread Reply:* Here is the code we use.

+ +
+
Azure SQL_DeltaLake.py
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sai + (saivenkatesh161@gmail.com) +
+
2023-05-16 03:23:13
 +
+

*Thread Reply:* Hi Team, Any updates on this?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sai + (saivenkatesh161@gmail.com) +
+
2023-05-16 03:23:37
 +
+

*Thread Reply:* I tried with putting a sql query having column names in it, still the lineage didn't show up..

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-05-09 10:00:39
 +
+

2023-05-09T13:37:48.526698281Z java.lang.ClassCastException: class org.apache.spark.scheduler.ShuffleMapStage cannot be cast to class java.lang.Boolean (org.apache.spark.scheduler.ShuffleMapStage is in unnamed module of loader 'app'; java.lang.Boolean is in module java.base of loader 'bootstrap') +2023-05-09T13:37:48.526703550Z at scala.runtime.BoxesRunTime.unboxToBoolean(BoxesRunTime.java:87) +2023_05_09T13:37:48.526707874Z at scala.collection.LinearSeqOptimized.forall(LinearSeqOptimized.scala:85) +2023_05_09T13:37:48.526712381Z at scala.collection.LinearSeqOptimized.forall$(LinearSeqOptimized.scala:82) +2023_05_09T13:37:48.526716848Z at scala.collection.immutable.List.forall(List.scala:91) +2023_05_09T13:37:48.526723183Z at io.openlineage.spark.agent.lifecycle.OpenLineageRunEventBuilder.registerJob(OpenLineageRunEventBuilder.java:181) +2023_05_09T13:37:48.526727604Z at io.openlineage.spark.agent.lifecycle.SparkSQLExecutionContext.setActiveJob(SparkSQLExecutionContext.java:152) +2023_05_09T13:37:48.526732292Z at java.base/java.util.Optional.ifPresent(Unknown Source) +2023-05-09T13:37:48.526736352Z at io.openlineage.spark.agent.OpenLineageSparkListener.lambda$onJobStart$10(OpenLineageSparkListener.java:150) +2023_05_09T13:37:48.526740471Z at java.base/java.util.Optional.ifPresent(Unknown Source) +2023-05-09T13:37:48.526744887Z at io.openlineage.spark.agent.OpenLineageSparkListener.onJobStart(OpenLineageSparkListener.java:147) +2023_05_09T13:37:48.526750258Z at org.apache.spark.scheduler.SparkListenerBus.doPostEvent(SparkListenerBus.scala:37) +2023_05_09T13:37:48.526753454Z at org.apache.spark.scheduler.SparkListenerBus.doPostEvent$(SparkListenerBus.scala:28) +2023_05_09T13:37:48.526756235Z at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37) +2023_05_09T13:37:48.526759315Z at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37) +2023_05_09T13:37:48.526762133Z at org.apache.spark.util.ListenerBus.postToAll(ListenerBus.scala:117) +2023_05_09T13:37:48.526764941Z at org.apache.spark.util.ListenerBus.postToAll$(ListenerBus.scala:101) +2023_05_09T13:37:48.526767739Z at org.apache.spark.scheduler.AsyncEventQueue.super$postToAll(AsyncEventQueue.scala:105) +2023_05_09T13:37:48.526776059Z at org.apache.spark.scheduler.AsyncEventQueue.$anonfun$dispatch$1(AsyncEventQueue.scala:105) +2023_05_09T13:37:48.526778937Z at scala.runtime.java8.JFunction0$mcJ$sp.apply(JFunction0$mcJ$sp.java:23) +2023_05_09T13:37:48.526781728Z at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62) +2023_05_09T13:37:48.526786986Z at <a href="http://org.apache.spark.scheduler.AsyncEventQueue.org">org.apache.spark.scheduler.AsyncEventQueue.org</a>$apache$spark$scheduler$AsyncEventQueue$$dispatch(AsyncEventQueue.scala:100) +2023_05_09T13:37:48.526789893Z at org.apache.spark.scheduler.AsyncEventQueue$$anon$2.$anonfun$run$1(AsyncEventQueue.scala:96) +2023_05_09T13:37:48.526792722Z at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1446) +2023_05_09T13:37:48.526795463Z at org.apache.spark.scheduler.AsyncEventQueue$$anon$2.run(AsyncEventQueue.scala:96) +Hi, noticing this error message from OL... anyone know why its happening?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-05-09 10:02:25
 +
+

*Thread Reply:* @Anirudh Shrinivason what's your OL and Spark version?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-05-09 10:03:29
 +
+

*Thread Reply:* Some example job would also help, or logs/LogicalPlan 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-05-09 10:05:54
 +
+

*Thread Reply:* OL version is 0.23.0 and spark version is 3.3.1

+ + + +
+ 👍 Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-05-09 11:00:22
 +
+

*Thread Reply:* Hmm actually, it seems like the error is intermittent actually. I ran the same job again, but did not notice any errors this time...

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-05-10 02:27:19
 +
+

*Thread Reply:* This is interesting and it happens within a line: +job.finalStage().parents().forall(toScalaFn(stage -&gt; stageMap.put(stage.id(), stage))); +The result of stageMap.put is Stage and for some reason which I don't undestand it tries doing unboxToBoolean . We could rewrite that to: +job.finalStage().parents().forall(toScalaFn(stage -&gt; { +stageMap.put(stage.id(), stage) +return true; +})); +but this is so weird that it is intermittent and I don't get why is it happening.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-05-11 02:22:25
 +
+

*Thread Reply:* @Anirudh Shrinivason, please let us know if it is still a valid issue. If so, we can create an issue for that.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-05-11 03:11:13
 +
+

*Thread Reply:* Hi @Paweł Leszczyński Sflr. Yeah, I think if we are able to fix this, it'll be better. If this is the dedicated fix, then I can create an issue and raise an MR.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-05-11 04:12:46
 +
+

*Thread Reply:* Opened an issue and PR. Do help check if its okay thanks!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-05-11 04:29:33
 +
+

*Thread Reply:* please run ./gradlew spotlessApply with Java 8

+ + + +
+ ✅ Anirudh Shrinivason +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Pietro Brunetti + (pietrobrunetti89@gmail.com) +
+
2023-05-10 05:49:00
 +
+

Hi all, +I’m new to openlineage (and marquez) so I’m trying to figure out if it could be the right option form a client usecase in which: +• a legacy custom data catalog (mongo backend + Java API backend for fronted in angular) +• AS-IS component lineage realations are retrieve in a custom way from the each component’s APIs +• the customer would like to bring in a basic data lineage feature based on already published metadata that represent custom workloads type (batch,streaming,interactive ones) + data access pattern (no direct relation with the datasources right now but only a abstraction layer upon them) +I’d like to exploit directly Marquez as the metastore to publish metadata about datasource, workload (the workload is the declaration + business logic code deployed into the customer platform) once the component is deployed (e.g. the service that exposes the specific access pattern, or the workload custom declaration), but I saw the openlinage spec is based on strictly coupling between run,job and datasource; I mean I want to be able to publish one item at a time and then (maybe in a future release of the customer product) be able to exploit runtime lineage also

+ +

Am I in the right place? +Thanks anyway :)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-05-10 07:36:33
 +
+

*Thread Reply:* > I mean I want to be able to publish one item at a time and then (maybe in a future release of the customer product) be able to exploit runtime lineage also +This is not something that we support yet - there are definitely a lot of plans and preliminary work for that.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Pietro Brunetti + (pietrobrunetti89@gmail.com) +
+
2023-05-10 07:57:44
 +
+

*Thread Reply:* Thanks for the response, btw I already took a look at the current capabilities provided by openlineage, so my “hidden” question is how do achieve what the customer want to in order to be integrated in some way with openalineage+marquez? +should I choose between make or buy (between already supported platforms) and then try to align “static” (aka declarative) lineage metadata within the openlinage conceptual model?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-05-10 11:04:20
 +
+

@channel +This month’s TSC meeting is tomorrow at 10am PT. All are welcome! https://openlineage.slack.com/archives/C01CK9T7HKR/p1683213923529529

+
+ + +
+ + + } + + Michael Robinson + (https://openlineage.slack.com/team/U02LXF3HUN7) +
+ + + +
+
+ <p>@channel +This month’s TSC meeting is next Thursday, May 11th, at 10:00 am PT. The tentative agenda will be on the <a href="https://bit.ly/OLwiki">wiki</a>. More info and the meeting link can be found on the <a href="https://openlineage.io/meetings/">website</a>. All are welcome! Also, feel free to reply or DM me with discussion topics, agenda items, etc.</p> +
+ + + + + + +
Original URL: https://openlineage.slack.com/archives/C01CK9T7HKR/p1683213923529529
+ + +
+ + Slack Conversation +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Lukenoff + (john@jlukenoff.com) +
+
2023-05-11 12:59:42
 +
+

Does anyone here have experience with vendors in this space like Atlan or Manta? I’m advocating pretty heavily for OpenLineage at my company and have a strong suspicion that the LoE of enabling an equivalent solution from a vendor is equal or greater than that of OL/Marquez. Curious if anyone has first-hand experience with these tools they might be willing to share?

+ + + +
+ 👋 Eric Veleker +
+ +
+ 👀 Pietro Brunetti +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ernie Ostic + (ernie.ostic@getmanta.com) +
+
2023-05-11 13:58:28
 +
+

*Thread Reply:* Hi John. Great question! [full disclosure, I am with Manta 🙂 ]. I'll let others answer as to their experience with ourselves or many other vendors that provide lineage, but want to mention that a variety of our customers are finding it beneficial to bring code based static lineage together with the event-based runtime lineage that OpenLineage provides. This gives them the best of both worlds, for analyzing the lineage of their existing systems, where rich parsers already exist (for everything from legacy ETL tools, reporting tools, rdbms, etc.), to newer or home-grown technologies where applying OpenLineage is a viable alternative.

+ + + +
+ 👍 John Lukenoff +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Brad Paskewitz + (bradford.paskewitz@fivetran.com) +
+
2023-05-11 14:12:04
 +
+

*Thread Reply:* @Ernie Ostic do you see a single front-runner in the static lineage space? The static/event-based situation you describe is exactly the product roadmap I'm seeing here at Fivetran and I'm wondering if there's an opportunity to drive consensus towards a best-practice solution. If I'm not mistaken weren't there plans to start supporting non-run-based events in OL as well?

+ + + +
+ 👋 Eric Veleker +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Lukenoff + (john@jlukenoff.com) +
+
2023-05-11 14:16:34
 +
+

*Thread Reply:* I definitely like the idea of a 3rd party solution being complementary to OSS tools we can maintain ourselves while allowing us to offload maintenance effort where possible. Currently I have strong opinions on both sides of the build vs. buy aisle and this seems like the best of both worlds.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harel Shein + (harel.shein@gmail.com) +
+
2023-05-11 14:52:40
 +
+

*Thread Reply:* @Brad Paskewitz that’s 100% our plan to extend the OL spec to support “run-less” events. We want to collect that static metadata for Datasets and Jobs outside of the context of a run through OpenLineage. +happy to get your feedback here as well: https://github.com/OpenLineage/OpenLineage/pull/1839

+ + + +
+ :gratitude_thank_you: Brad Paskewitz +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Eric Veleker + (eric@atlan.com) +
+
2023-05-11 14:57:46
 +
+

*Thread Reply:* Hi @John Lukenoff. Here at Atlan we've been working with the OpenLineage community for quite some time to unlock the use case you describe. These efforts are adjacent to our ongoing integration with Fivetran. Happy to connect and give you a demo of what we've built and dig into your use case specifics.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Lukenoff + (john@jlukenoff.com) +
+
2023-05-12 11:26:32
 +
+

*Thread Reply:* Thanks all! These comments are really informative, it’s exciting to hear about vendors leaning into the project to let us continue to benefit from the tremendous progress being made by the community. Had a great discussion with Atlan yesterday and plan to connect with Manta next week to discuss our use-cases.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ernie Ostic + (ernie.ostic@getmanta.com) +
+
2023-05-12 12:34:32
 +
+

*Thread Reply:* Reach out anytime, John. @John Lukenoff Looking forward to engaging further with you on these topics!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harshini Devathi + (harshini.devathi@tigeranalytics.com) +
+
2023-05-12 11:15:10
 +
+

Hello all, I would like to have a new release of Openlineage as the new code base seems to have some issues fixed. I need these fixes for my project.

+ + + +
+ ➕ Michael Robinson, Maciej Obuchowski, Julien Le Dem, Jakub Dardziński, Anirudh Shrinivason, Harshini Devathi, Paweł Leszczyński, pankaj koti +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-05-12 11:19:02
 +
+

*Thread Reply:* Thank you for requesting an OpenLineage release. As stated here, three +1s from committers will authorize an immediate release. Our policy is not to release on Fridays, so the earliest we could initiate would be Monday.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harshini Devathi + (harshini.devathi@tigeranalytics.com) +
+
2023-05-12 13:12:43
 +
+

*Thread Reply:* A release on Monday is totally fine @Michael Robinson.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-05-15 08:37:39
 +
+

*Thread Reply:* The release will be initiated today. Thanks @Harshini Devathi

+ + + +
+ 👍 Anirudh Shrinivason, Harshini Devathi +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harshini Devathi + (harshini.devathi@tigeranalytics.com) +
+
2023-05-16 20:16:07
 +
+

*Thread Reply:* Appreciate it @Michael Robinson and thanks to all the committers for the prompt response

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-05-15 12:09:24
 +
+

@channel +We released OpenLineage 0.25.0, including: +Additions: +• Spark: merge into query support #1823 @pawel-big-lebowski +Fixes: +• Spark: fix JDBC query handling #1808 @nataliezeller1 +• Spark: filter Delta adaptive plan events #1830 @pawel-big-lebowski +• Spark: fix Java class cast exception #1844 @Anirudh181001 +• Flink: include missing fields of Openlineage events #1840 @pawel-big-lebowski +Plus doc changes and more. +Thanks to all the contributors! +For the details, see: +Release: https://github.com/OpenLineage/OpenLineage/releases/tag/0.25.0 +Changelog: https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md +Commit history: https://github.com/OpenLineage/OpenLineage/compare/0.24.0...0.25.0 +Maven: https://oss.sonatype.org/#nexus-search;quick~openlineage +PyPI: https://pypi.org/project/openlineage-python/

+ + + +
+ 🙌 Jakub Dardziński, Sai, pankaj koti, Paweł Leszczyński, Perttu Salonen, Maciej Obuchowski, Fraser Marlow, Ross Turk, Harshini Devathi, tati +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-05-16 14:03:01
 +
+

@channel +If you’re planning on being in San Francisco at the end of June — perhaps for this year’s Data+AI Summit — please stop by Astronomer’s offices on California Street on 6/27 for the first SF OpenLineage Meetup. We’ll be discussing spec changes planned for OpenLineage v1.0.0, progress on Airflow AIP 53, and more. Plus, dinner will be provided! For more info and to sign up, check out the OL blog. Join us!

+
+
openlineage.io
+ + + +
Meet Us in San Francisco on June 27th! | OpenLineage Docs
+
+ The first SF OpenLineage Meetup will take place on June 27th. +
+ + + + + + +
Original URL: https://openlineage.io/blog/sf-meetup
+ + + +
+ + + +
+ 🙌 alexandre bergere, Anirudh Shrinivason, Harel Shein, Willy Lulciuc, Jarek Potiuk, Ross Turk, John Lukenoff, tati +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2023-05-16 14:13:16
 +
+

*Thread Reply:* Can’t wait! 💯

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-05-17 00:09:23
 +
+

Hi, I've been noticing this error that is intermittently popping up in some of the spark jobs: +AsyncEventQueue: Dropping event from queue appStatus. This likely means one of the listeners is too slow and cannot keep up with the rate at which tasks are being started by the scheduler. +spark.scheduler.listenerbus.eventqueue.size Increasing this spark config did not help either. +Any ideas on how to mitigate this issue? Seeing this in spark 3.1.2 btw

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-05-17 01:58:28
 +
+

*Thread Reply:* Hi @Anirudh Shrinivason, are you able to send the OL events to console. This would let us confirm if the issue is related with event generation or emitting it and waiting for the backend to repond.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-05-17 01:59:03
 +
+

*Thread Reply:* Ahh okay sure. Let me see if I can do that

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sai + (saivenkatesh161@gmail.com) +
+
2023-05-17 01:52:15
 +
+

Hi Team

+ +

We are seeing an issue with OL configured cluster where delta table merge is failing with below error. It is running fine when we run with other clusters where OL is not configured. I ran it multiple times assuming its intermittent issue with memory, but it keeps on failing with same error. Attached the code for reference. We are using the latest release (0.25.0)

+ +

org.apache.spark.SparkException: Job aborted due to stage failure: Task serialization failed: java.lang.StackOverflowError

+ +

@Paweł Leszczyński @Michael Robinson

+ +
+
merge-test.py
+ + + + + + +
+ + +
+ 👀 Paweł Leszczyński +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sai + (saivenkatesh161@gmail.com) +
+
2023-05-19 03:55:51
 +
+

*Thread Reply:* Hi @Paweł Leszczyński

+ +

Thanks for fixing the issue and with new release merge is working. But I could not see any input and output datasets for this. Let me know if you need any further details to look into this.

+ +
},
+"job": {
+    "namespace": "openlineage_poc",
+    "name": "spark_ol_integration_execute_merge_into_command_edge",
+    "facets": {}
+},
+"inputs": [],
+"outputs": [],
+
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-05-19 04:00:01
 +
+

*Thread Reply:* Oh man, it's just that vanilla spark differs from the one available in databricks platform. our integration tests do verify behaviour on vanilla spark which still leaves a possibility for inconsistency. will need to get back to it then at some time.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sai + (saivenkatesh161@gmail.com) +
+
2023-06-02 02:11:28
 +
+

*Thread Reply:* Hi @Paweł Leszczyński

+ +

Did you get chance to look into this issue?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-06-02 02:13:18
 +
+

*Thread Reply:* Hi Sai, I am going back to spark. I am working on support for Spark 3.4, which is going to add some event filtering on internal delta operations that trigger unncecessarly the events

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-06-02 02:13:28
 +
+

*Thread Reply:* this may be releated to issue you created

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-06-02 02:14:13
 +
+

*Thread Reply:* I do have planned creating integration test for databricks which will be helpful to tackle the issues you raised

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-06-02 02:14:27
 +
+

*Thread Reply:* so yes, I am looking at the Spark

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sai + (saivenkatesh161@gmail.com) +
+
2023-06-02 02:20:06
 +
+

*Thread Reply:* thanks much Pawel.. I am looking more into the merge part as first priority as we use is frequently.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-06-02 02:21:01
 +
+

*Thread Reply:* I know, this is important.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-06-02 02:21:14
 +
+

*Thread Reply:* It just need still some time.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-06-02 02:21:46
 +
+

*Thread Reply:* thank you for your patience and being so proactive on those issues.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sai + (saivenkatesh161@gmail.com) +
+
2023-06-02 02:22:12
 +
+

*Thread Reply:* no problem.. Please do keep us posted with updates..

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-05-17 10:47:27
 +
+

Our recent Openlineage release (0.25.0) proved there are many users that use Openlineage on databricks, which is incredible. I am super happy to know that, although we realised that as a side effect of a bug. Sorry for that.

+ +

I would like to opt for a new release which contains PR #1858 and should unblock databricks users.

+ + + +
+ ➕ Paweł Leszczyński, Maciej Obuchowski, Harshini Devathi, Jakub Dardziński, Sai, Anirudh Shrinivason, Anbarasi +
+ +
+ 👍 Michael Robinson +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-05-18 10:26:48
 +
+

*Thread Reply:* The release request has been approved and will be initiated shortly.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-05-17 22:49:41
 +
+

Actually, I noticed a few other stack overflow errors on 0.25.0. Let me raise an issue. Could we cut a release once this bug are fixed too please?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-05-18 02:29:55
 +
+

*Thread Reply:* Hi Anirudh, I saw your issue and I think it is the same one as solved within #1858. Are you able to reproduce it on a version built on the top of main?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-05-18 06:21:05
 +
+

*Thread Reply:* Hi I haven't managed to try with the main branch. But if its the same error then all's good! If the error resurfaces then we can look into it.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Lovenish Goyal + (lovenishgoyal@gmail.com) +
+
2023-05-18 02:21:13
 +
+

Hi All,

+ +

We are in POC phase OpenLineage integration with our core DBT, can anyone help me with document to start with.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-05-18 02:28:31
 +
+

*Thread Reply:* I know this one: https://openlineage.io/docs/integrations/dbt

+
+
openlineage.io
+ + + +
dbt | OpenLineage Docs
+
+ dbt (data build tool) is a powerful transformation engine. It operates on data already within a warehouse, making it easy for data engineers to build complex pipelines from the comfort of their laptops. While it doesn’t perform extraction and loading of data, it’s extremely powerful at transformations. +
+ + + + + + +
Original URL: https://openlineage.io/docs/integrations/dbt
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Lovenish Goyal + (lovenishgoyal@gmail.com) +
+
2023-05-18 02:41:39
 +
+

*Thread Reply:* Hi @Paweł Leszczyński Thanks for the revert, I tried same but facing below issue

+ +

requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=5000): Max retries exceeded with url:

+ +

Looks like I need to start the service

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-05-18 02:44:09
 +
+

*Thread Reply:* @Lovenish Goyal, exactly. You need to start Marquez. +More about it: https://marquezproject.ai/quickstart

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harel Shein + (harel.shein@gmail.com) +
+
2023-05-18 10:27:52
 +
+

*Thread Reply:* @Lovenish Goyal how are you running dbt core currently?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Lovenish Goyal + (lovenishgoyal@gmail.com) +
+
2023-05-19 01:55:20
 +
+

*Thread Reply:* Trying but facing issue while running marquezproject @Jakub Dardziński

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Lovenish Goyal + (lovenishgoyal@gmail.com) +
+
2023-05-19 01:56:03
 +
+

*Thread Reply:* @Harel Shein we have created custom docker image of DBT + Airflow and running it on an EC2

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harel Shein + (harel.shein@gmail.com) +
+
2023-05-19 09:05:31
 +
+

*Thread Reply:* for running dbt core on Airflow, we have a utility that helps develop dbt natively on Airflow. There’s also built in support for collecting lineage if you have the airflow-openlineage provider installed. +https://astronomer.github.io/astronomer-cosmos/#quickstart

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harel Shein + (harel.shein@gmail.com) +
+
2023-05-19 09:06:30
 +
+

*Thread Reply:* RE issues running Marquez, can you share what those are? I’m guessing that since you are running both of them in individual docker images, the airflow deployment might not be able to communicate with the Marquez endpoints?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-05-19 09:06:53
 +
+

*Thread Reply:* @Harel Shein I've already helped with running Marquez 🙂

+ + + +
+ :first_place_medal: Harel Shein, Paweł Leszczyński, Michael Robinson +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anbarasi + (anbujothi@gmail.com) +
+
2023-05-18 02:29:53
 +
+

@Paweł Leszczyński We are facing the following issue with Azure databricks. When we use aggregate functions in databricks notebooks, Open lineage is not able to provide column level lineage. I understand its an existing issue. Can you please let me know in which release this issue will be fixed ? It is one of the most needed feature for us to implement openlineage in our current project. Kindly let me know

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-05-18 02:34:35
 +
+

*Thread Reply:* I am not sure if this is the same. If you see OL events collected with column-lineage missing, then it's a different one.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-05-18 02:41:11
 +
+

*Thread Reply:* Please also be aware, that it is extremely helpful to investigate the issues on your own before creating them.

+ +

Our integration traverses spark's logical plans and extracts lineage events from plan nodes that it understands. Some plan nodes are not supported yet and, from my experience, when working on an issue, 80% of time is spent on reproducing the scenario.

+ +

So, if you are able to produce a minimal amount of spark code that reproduces an issue, this can be extremely helpful and significantly speed up resolution time.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anbarasi + (anbujothi@gmail.com) +
+
2023-05-18 03:52:30
 +
+

*Thread Reply:* @Paweł Leszczyński Thanks for the prompt response.

+ +

Provided sample codes with and without using aggregate functions and its respective lineage events for reference.

+ +

  1. Please find the code without using aggregate function: + finaldf=spark.sql(""" + select productid + ,OrderQty as TotalOrderQty + ,ReceivedQty as TotalReceivedQty + ,StockedQty as TotalStockedQty + ,RejectedQty as TotalRejectedQty + from openlineagepoc.purchaseorder + --group by productid + order by productid""")

    + +
       final_df.write.mode("overwrite").saveAsTable("openlineage_poc.productordertest1")
+
    2. +
+ +

Please find the Openlineage Events for the Input, Ouput datasets. We could find the column lineage in this.

+ +

"inputs": [ + { + "namespace": "dbfs", + "name": "/mnt/dlzones/warehouse/openlineagepoc/gold/purchaseorder", + "facets": { + "dataSource": { + "producer": "https://github.com/OpenLineage/OpenLineage/tree/0.25.0/integration/spark", + "schemaURL": "https://openlineage.io/spec/facets/1-0-0/DatasourceDatasetFacet.json#/$defs/DatasourceDatasetFacet", + "name": "dbfs", + "uri": "dbfs" + }, + "schema": { + "producer": "https://github.com/OpenLineage/OpenLineage/tree/0.25.0/integration/spark", + "schemaURL": "https://openlineage.io/spec/facets/1-0-0/SchemaDatasetFacet.json#/$defs/SchemaDatasetFacet", + "fields": [ + { + "name": "PurchaseOrderID", + "type": "integer" + }, + { + "name": "PurchaseOrderDetailID", + "type": "integer" + }, + { + "name": "DueDate", + "type": "timestamp" + }, + { + "name": "OrderQty", + "type": "short" + }, + { + "name": "ProductID", + "type": "integer" + }, + { + "name": "UnitPrice", + "type": "decimal(19,4)" + }, + { + "name": "LineTotal", + "type": "decimal(19,4)" + }, + { + "name": "ReceivedQty", + "type": "decimal(8,2)" + }, + { + "name": "RejectedQty", + "type": "decimal(8,2)" + }, + { + "name": "StockedQty", + "type": "decimal(9,2)" + }, + { + "name": "RevisionNumber", + "type": "integer" + }, + { + "name": "Status", + "type": "integer" + }, + { + "name": "EmployeeID", + "type": "integer" + }, + { + "name": "NationalIDNumber", + "type": "string" + }, + { + "name": "JobTitle", + "type": "string" + }, + { + "name": "Gender", + "type": "string" + }, + { + "name": "MaritalStatus", + "type": "string" + }, + { + "name": "VendorID", + "type": "integer" + }, + { + "name": "ShipMethodID", + "type": "integer" + }, + { + "name": "ShipMethodName", + "type": "string" + }, + { + "name": "ShipMethodrowguid", + "type": "string" + }, + { + "name": "OrderDate", + "type": "timestamp" + }, + { + "name": "ShipDate", + "type": "timestamp" + }, + { + "name": "SubTotal", + "type": "decimal(19,4)" + }, + { + "name": "TaxAmt", + "type": "decimal(19,4)" + }, + { + "name": "Freight", + "type": "decimal(19,4)" + }, + { + "name": "TotalDue", + "type": "decimal(19,4)" + } + ] + }, + "symlinks": { + "producer": "https://github.com/OpenLineage/OpenLineage/tree/0.25.0/integration/spark", + "schemaURL": "https://openlineage.io/spec/facets/1-0-0/SymlinksDatasetFacet.json#/$defs/SymlinksDatasetFacet", + "identifiers": [ + { + "namespace": "/mnt/dlzones/warehouse/openlineagepoc/gold", + "name": "openlineagepoc.purchaseorder", + "type": "TABLE" + } + ] + } + }, + "inputFacets": {} + } + ], + "outputs": [ + { + "namespace": "dbfs", + "name": "/mnt/dlzones/warehouse/openlineagepoc/productordertest1", + "facets": { + "dataSource": { + "producer": "https://github.com/OpenLineage/OpenLineage/tree/0.25.0/integration/spark", + "schemaURL": "https://openlineage.io/spec/facets/1-0-0/DatasourceDatasetFacet.json#/$defs/DatasourceDatasetFacet", + "name": "dbfs", + "uri": "dbfs" + }, + "schema": { + "producer": "https://github.com/OpenLineage/OpenLineage/tree/0.25.0/integration/spark", + "schemaURL": "https://openlineage.io/spec/facets/1-0-0/SchemaDatasetFacet.json#/$defs/SchemaDatasetFacet", + "fields": [ + { + "name": "productid", + "type": "integer" + }, + { + "name": "TotalOrderQty", + "type": "short" + }, + { + "name": "TotalReceivedQty", + "type": "decimal(8,2)" + }, + { + "name": "TotalStockedQty", + "type": "decimal(9,2)" + }, + { + "name": "TotalRejectedQty", + "type": "decimal(8,2)" + } + ] + }, + "storage": { + "producer": "https://github.com/OpenLineage/OpenLineage/tree/0.25.0/integration/spark", + "schemaURL": "https://openlineage.io/spec/facets/1-0-0/StorageDatasetFacet.json#/$defs/StorageDatasetFacet", + "storageLayer": "unity", + "fileFormat": "parquet" + }, + "columnLineage": { + "producer": "https://github.com/OpenLineage/OpenLineage/tree/0.25.0/integration/spark", + "schemaURL": "https://openlineage.io/spec/facets/1-0-1/ColumnLineageDatasetFacet.json#/$defs/ColumnLineageDatasetFacet", + "fields": { + "productid": { + "inputFields": [ + { + "namespace": "dbfs", + "name": "/mnt/dlzones/warehouse/openlineagepoc/gold/purchaseorder", + "field": "ProductID" + } + ] + }, + "TotalOrderQty": { + "inputFields": [ + { + "namespace": "dbfs", + "name": "/mnt/dlzones/warehouse/openlineagepoc/gold/purchaseorder", + "field": "OrderQty" + } + ] + }, + "TotalReceivedQty": { + "inputFields": [ + { + "namespace": "dbfs", + "name": "/mnt/dlzones/warehouse/openlineagepoc/gold/purchaseorder", + "field": "ReceivedQty" + } + ] + }, + "TotalStockedQty": { + "inputFields": [ + { + "namespace": "dbfs", + "name": "/mnt/dlzones/warehouse/openlineagepoc/gold/purchaseorder", + "field": "StockedQty" + } + ] + }, + "TotalRejectedQty": { + "inputFields": [ + { + "namespace": "dbfs", + "name": "/mnt/dlzones/warehouse/openlineagepoc/gold/purchaseorder", + "field": "RejectedQty" + } + ] + } + } + }, + "symlinks": { + "producer": "https://github.com/OpenLineage/OpenLineage/tree/0.25.0/integration/spark", + "schemaURL": "https://openlineage.io/spec/facets/1-0-0/SymlinksDatasetFacet.json#/$defs/SymlinksDatasetFacet", + "identifiers": [ + { + "namespace": "/mnt/dlzones/warehouse/openlineagepoc", + "name": "openlineagepoc.productordertest1", + "type": "TABLE" + } + ] + }, + "lifecycleStateChange": { + "producer": "https://github.com/OpenLineage/OpenLineage/tree/0.25.0/integration/spark", + "_schemaURL": "https://openlineage.io/spec/facets/1-0-0/LifecycleStateChangeDatasetFacet.json#/$defs/LifecycleStateChangeDatasetFacet", + "lifecycleStateChange": "OVERWRITE" + } + }, + "outputFacets": {} + } + ]

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anbarasi + (anbujothi@gmail.com) +
+
2023-05-18 03:55:04
 +
+

*Thread Reply:* 2. Please find the code using aggregate function:

+ +
    final_df=spark.sql("""
+    select productid
+    ,sum(OrderQty) as TotalOrderQty
+    ,sum(ReceivedQty) as TotalReceivedQty
+    ,sum(StockedQty) as TotalStockedQty
+    ,sum(RejectedQty) as TotalRejectedQty
+    from openlineage_poc.purchaseorder
+    group by productid
+    order by productid""")
+
+    final_df.write.mode("overwrite").saveAsTable("openlineage_poc.productordertest2")
+
+ +

Please find the Openlineage Events for the Input, Ouput datasets. We couldnt find the column lineage in output section. Please find the sample

+ +

"inputs": [ + { + "namespace": "dbfs", + "name": "/mnt/dlzones/warehouse/openlineagepoc/gold/purchaseorder", + "facets": { + "dataSource": { + "producer": "https://github.com/OpenLineage/OpenLineage/tree/0.25.0/integration/spark", + "schemaURL": "https://openlineage.io/spec/facets/1-0-0/DatasourceDatasetFacet.json#/$defs/DatasourceDatasetFacet", + "name": "dbfs", + "uri": "dbfs" + }, + "schema": { + "producer": "https://github.com/OpenLineage/OpenLineage/tree/0.25.0/integration/spark", + "schemaURL": "https://openlineage.io/spec/facets/1-0-0/SchemaDatasetFacet.json#/$defs/SchemaDatasetFacet", + "fields": [ + { + "name": "PurchaseOrderID", + "type": "integer" + }, + { + "name": "PurchaseOrderDetailID", + "type": "integer" + }, + { + "name": "DueDate", + "type": "timestamp" + }, + { + "name": "OrderQty", + "type": "short" + }, + { + "name": "ProductID", + "type": "integer" + }, + { + "name": "UnitPrice", + "type": "decimal(19,4)" + }, + { + "name": "LineTotal", + "type": "decimal(19,4)" + }, + { + "name": "ReceivedQty", + "type": "decimal(8,2)" + }, + { + "name": "RejectedQty", + "type": "decimal(8,2)" + }, + { + "name": "StockedQty", + "type": "decimal(9,2)" + }, + { + "name": "RevisionNumber", + "type": "integer" + }, + { + "name": "Status", + "type": "integer" + }, + { + "name": "EmployeeID", + "type": "integer" + }, + { + "name": "NationalIDNumber", + "type": "string" + }, + { + "name": "JobTitle", + "type": "string" + }, + { + "name": "Gender", + "type": "string" + }, + { + "name": "MaritalStatus", + "type": "string" + }, + { + "name": "VendorID", + "type": "integer" + }, + { + "name": "ShipMethodID", + "type": "integer" + }, + { + "name": "ShipMethodName", + "type": "string" + }, + { + "name": "ShipMethodrowguid", + "type": "string" + }, + { + "name": "OrderDate", + "type": "timestamp" + }, + { + "name": "ShipDate", + "type": "timestamp" + }, + { + "name": "SubTotal", + "type": "decimal(19,4)" + }, + { + "name": "TaxAmt", + "type": "decimal(19,4)" + }, + { + "name": "Freight", + "type": "decimal(19,4)" + }, + { + "name": "TotalDue", + "type": "decimal(19,4)" + } + ] + }, + "symlinks": { + "producer": "https://github.com/OpenLineage/OpenLineage/tree/0.25.0/integration/spark", + "schemaURL": "https://openlineage.io/spec/facets/1-0-0/SymlinksDatasetFacet.json#/$defs/SymlinksDatasetFacet", + "identifiers": [ + { + "namespace": "/mnt/dlzones/warehouse/openlineagepoc/gold", + "name": "openlineagepoc.purchaseorder", + "type": "TABLE" + } + ] + } + }, + "inputFacets": {} + } + ], + "outputs": [ + { + "namespace": "dbfs", + "name": "/mnt/dlzones/warehouse/openlineagepoc/productordertest2", + "facets": { + "dataSource": { + "producer": "https://github.com/OpenLineage/OpenLineage/tree/0.25.0/integration/spark", + "schemaURL": "https://openlineage.io/spec/facets/1-0-0/DatasourceDatasetFacet.json#/$defs/DatasourceDatasetFacet", + "name": "dbfs", + "uri": "dbfs" + }, + "schema": { + "producer": "https://github.com/OpenLineage/OpenLineage/tree/0.25.0/integration/spark", + "schemaURL": "https://openlineage.io/spec/facets/1-0-0/SchemaDatasetFacet.json#/$defs/SchemaDatasetFacet", + "fields": [ + { + "name": "productid", + "type": "integer" + }, + { + "name": "TotalOrderQty", + "type": "long" + }, + { + "name": "TotalReceivedQty", + "type": "decimal(18,2)" + }, + { + "name": "TotalStockedQty", + "type": "decimal(19,2)" + }, + { + "name": "TotalRejectedQty", + "type": "decimal(18,2)" + } + ] + }, + "storage": { + "producer": "https://github.com/OpenLineage/OpenLineage/tree/0.25.0/integration/spark", + "schemaURL": "https://openlineage.io/spec/facets/1-0-0/StorageDatasetFacet.json#/$defs/StorageDatasetFacet", + "storageLayer": "unity", + "fileFormat": "parquet" + }, + "symlinks": { + "producer": "https://github.com/OpenLineage/OpenLineage/tree/0.25.0/integration/spark", + "schemaURL": "https://openlineage.io/spec/facets/1-0-0/SymlinksDatasetFacet.json#/$defs/SymlinksDatasetFacet", + "identifiers": [ + { + "namespace": "/mnt/dlzones/warehouse/openlineagepoc", + "name": "openlineagepoc.productordertest2", + "type": "TABLE" + } + ] + }, + "lifecycleStateChange": { + "producer": "https://github.com/OpenLineage/OpenLineage/tree/0.25.0/integration/spark", + "schemaURL": "https://openlineage.io/spec/facets/1-0-0/LifecycleStateChangeDatasetFacet.json#/$defs/LifecycleStateChangeDatasetFacet", + "lifecycleStateChange": "OVERWRITE" + } + }, + "outputFacets": {} + } + ]

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-05-18 04:09:17
 +
+

*Thread Reply:* amazing. https://github.com/OpenLineage/OpenLineage/issues/1861

+
+ + + + +
#1861 Column lineage not working for aggregates on Databricks
+
+ <p>The code</p> + +<pre><code>final_df=spark.sql(""" + select productid + ,OrderQty as TotalOrderQty + ,ReceivedQty as TotalReceivedQty + ,StockedQty as TotalStockedQty + ,RejectedQty as TotalRejectedQty + from openlineage_poc.purchaseorder + --group by productid + order by productid""") + final_df.write.mode("overwrite").saveAsTable("openlineage_poc.productordertest1") +</code></pre> + +<p>generates OL event with column lineage. However, in case of group by enabled</p> + +<pre><code> final_df=spark.sql(""" + select productid + ,sum(OrderQty) as TotalOrderQty + ,sum(ReceivedQty) as TotalReceivedQty + ,sum(StockedQty) as TotalStockedQty + ,sum(RejectedQty) as TotalRejectedQty + from openlineage_poc.purchaseorder + group by productid + order by productid""") + final_df.write.mode("overwrite").saveAsTable("openlineage_poc.productordertest2") +</code></pre> + +<p>it's not present anymore. <br /> +<a href="https://github.com/OpenLineage/OpenLineage/files/11505766/lineage.txt">lineage.txt</a></p> + +<p>Requested on slack: <a href="https://openlineage.slack.com/archives/C01CK9T7HKR/p1684391393701839">https://openlineage.slack.com/archives/C01CK9T7HKR/p1684391393701839</a></p> +
+ +
+
Assignees
+ <a href="https://github.com/pawel-big-lebowski">@pawel-big-lebowski</a> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anbarasi + (anbujothi@gmail.com) +
+
2023-05-18 04:11:56
 +
+

*Thread Reply:* Thanks for considering the request and looking into it

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-05-18 13:12:35
 +
+

@channel +We released OpenLineage 0.26.0, including: +Additions: +• Proxy: Fluentd proxy support (experimental) #1757 @pawel-big-lebowski +Changes: +• Python client: use Hatchling over setuptools to orchestrate Python env setup #1856 @gaborbernat +Fixes: +• Spark: fix logicalPlan serialization issue on Databricks #1858 @pawel-big-lebowski +Plus an additional fix, doc changes and more. +Thanks to all the contributors, including new contributor @gaborbernat! +For the details, see: +Release: https://github.com/OpenLineage/OpenLineage/releases/tag/0.26.0 +Changelog: https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md +Commit history: https://github.com/OpenLineage/OpenLineage/compare/0.25.0...0.26.0 +Maven: https://oss.sonatype.org/#nexus-search;quick~openlineage +PyPI: https://pypi.org/project/openlineage-python/

+ + + +
+ ❤️ Paweł Leszczyński, Maciej Obuchowski, Anirudh Shrinivason, Peter Hicks, pankaj koti +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Bramha Aelem + (bramhaaelem@gmail.com) +
+
2023-05-18 14:42:49
 +
+

Hi Team , can someone please address https://github.com/OpenLineage/OpenLineage/issues/1866.

+
+ + + + +
#1866 Reg: Lineage Representation With Available Datasets Generated by the Jobs
+
+ <p><strong>Purpose:</strong> <br /> +As part of the data ingestion pipeline process. process#1 generating the tables using spark. How can refer process#1 as an input for my process#2. When we specify them in lineage process by refering spark namespace where the datasets resides it's not showling accurately on Marquez UI.</p> + +<p><strong>Proposed implementation</strong> <br /> +If N number of jobs generates the datasets under respective namespaces. We should refer the same in the lineage representation process as is insated of them craeting again and again under the job execution namespace.</p> +
+ +
+
Labels
+ proposal +
+ +
+
Comments
+ 1 +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2023-05-18 20:13:09
 +
+

*Thread Reply:* Hi @Bramha Aelem I replied in the ticket. Thank you for opening it.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Bramha Aelem + (bramhaaelem@gmail.com) +
+
2023-05-18 21:15:30
 +
+

*Thread Reply:* Hi @Julien Le Dem - Thanks for quick response. I replied in the ticket. Please let me know if you need any more details.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-05-19 02:13:57
 +
+

*Thread Reply:* Hi @Bramha Aelem - asked for more details in the ticket.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Bramha Aelem + (bramhaaelem@gmail.com) +
+
2023-05-22 11:08:58
 +
+

*Thread Reply:* Hi @Paweł Leszczyński - I replied with necessary details in the ticket. Please let me know if you need any more details.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Bramha Aelem + (bramhaaelem@gmail.com) +
+
2023-05-25 15:22:42
 +
+

*Thread Reply:* Hi @Paweł Leszczyński - any further updates on issue?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-05-26 01:56:47
 +
+

*Thread Reply:* hi @Bramha Aelem, i was out of office for a few days. will get back into this soon. thanks for update.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Bramha Aelem + (bramhaaelem@gmail.com) +
+
2023-05-27 18:46:27
 +
+

*Thread Reply:* Hi @Paweł Leszczyński -Thanks for your reply. will wait for your response to proceed further on the issue.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Bramha Aelem + (bramhaaelem@gmail.com) +
+
2023-06-02 19:29:08
 +
+

*Thread Reply:* Hi @Paweł Leszczyński -Hope you are doing well. Did you get a chance to look into the samples which are provided in the ticket. Kindly let me know your observations/recommendations.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Bramha Aelem + (bramhaaelem@gmail.com) +
+
2023-06-09 12:43:54
 +
+

*Thread Reply:* Hi @Paweł Leszczyński - Hope you are doing well. Did you get a chance to look into the samples which are provided in the ticket. Kindly let me know your observations/recommendations.

+ + + +
+ 👀 Paweł Leszczyński +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Bramha Aelem + (bramhaaelem@gmail.com) +
+
2023-07-06 10:29:01
 +
+

*Thread Reply:* Hi @Paweł Leszczyński - Good day. Did you get a chance to look into query which I have posted. can you please provide any thoughts on my observation/query.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Doe + (adarsh.pansari@tigeranalytics.com) +
+
2023-05-19 03:42:21
 +
+

Hello Everyone, I was trying to integrate openlineage with Jupyter Notebooks, I followed the docs but when I run the sample notebook I am getting an error +23/05/19 07:39:08 ERROR EventEmitter: Could not emit lineage w/ exception +Can someone Please help understand why am I getting this error and the resolution.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-05-19 03:49:27
 +
+

*Thread Reply:* Hello @John Doe, this mostly means there's somehting wrong with your transport config for emitting Openlineage events.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-05-19 03:49:41
 +
+

*Thread Reply:* what do you want to do with the events?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Doe + (adarsh.pansari@tigeranalytics.com) +
+
2023-05-19 04:10:24
 +
+

*Thread Reply:* Hi @Paweł Leszczyński, I am working on a PoC to understand the use cases of OL and how it build Lineages.

+ +

As for the transport config I am using the codes from the documentation to setup OL. +https://openlineage.io/docs/integrations/spark/quickstart_local

+ +

Apart from these I dont have anything else in my nb

+
+
openlineage.io
+ + + +
Quickstart with Jupyter | OpenLineage Docs
+
+ Trying out the Spark integration is super easy if you already have Docker Desktop and git installed. +
+ + + + + + +
Original URL: https://openlineage.io/docs/integrations/spark/quickstart_local
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-05-19 04:38:58
 +
+

*Thread Reply:* ok, I am wondering if what you experience isn't similar to issue #1860. Could you try openlineage 0.23.0 to see if get the same error?

+ +

<https://github.com/OpenLineage/OpenLineage/issues/1860>

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Doe + (adarsh.pansari@tigeranalytics.com) +
+
2023-05-19 10:05:59
 +
+

*Thread Reply:* I tried with 0.23.0 still getting the same error

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Doe + (adarsh.pansari@tigeranalytics.com) +
+
2023-05-23 02:34:52
 +
+

*Thread Reply:* @Paweł Leszczyński any other way I can try to setup. The issue still persists

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-05-29 03:53:04
 +
+

*Thread Reply:* hmyy, I've just redone steps from https://openlineage.io/docs/integrations/spark/quickstart_local with 0.26.0 and could not reproduce behaviour you encountered.

+
+
openlineage.io
+ + + +
Quickstart with Jupyter | OpenLineage Docs
+
+ Trying out the Spark integration is super easy if you already have Docker Desktop and git installed. +
+ + + + + + +
Original URL: https://openlineage.io/docs/integrations/spark/quickstart_local
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tom van Eijk + (t.m.h.vaneijk@tilburguniversity.edu) +
+
2023-05-22 09:41:55
 +
+

Hello Team!!! A part of my master thesis's case study was about data lineage in data mesh and how open-source initiatives such as OpenLineage and Marquez can realize this. Can you recommend me some material that can support the writing part of my thesis (more context: I tried to extract lineage events from Snowflake through Airlfow and used Docker Compose on EC2 to connect Airflow and the Marquez webserver)? We will divide the thesis into a few academic papers to make the content more digestible and publish one of them soon hopefully!

+ + + +
+ 👍 Ernie Ostic, Maciej Obuchowski, Ross Turk, Michael Robinson +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-05-22 16:34:00
 +
+

*Thread Reply:* Tom, thanks for your question. This is really exciting! I assume you’ve already started checking out the docs, but there are many other resources on the website, as well (on the blog and resources pages in particular). And don’t skip the YouTube channel, where we’ve recently started to upload short, more digestible excerpts from the community meetings. Please keep us updated as you make progress!

+ + + +
+ 👀 Tom van Eijk +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tom van Eijk + (t.m.h.vaneijk@tilburguniversity.edu) +
+
2023-05-22 16:48:06
 +
+

*Thread Reply:* Hi Michael! Thank you so much for sending these resources! I've been working on this thesis for quite some time already and it's almost finished. I just needed some additional information to help in accurately describing some of the processes in OpenLineage and Marquez. Will send you the case study chapter later this week to get some feedback if possible. Keep you posted on things such as publication! Perhaps it can make OpenLineage even more popular than it already is 😉

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-05-22 16:52:18
 +
+

*Thread Reply:* Yes, please share it! Looking forward to checking it out. Super cool!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ernie Ostic + (ernie.ostic@getmanta.com) +
+
2023-05-22 09:57:50
 +
+

Hi Tom. Good luck. Sounds like a great case study. You might want to compare and contrast various kinds of lineage solutions....all of which complement each other, as well as having their own pros and cons. (...code based lineage via parsing, data similarity lineage, run-time lineage reporting, etc.) ...and then focus on open source and OpenLineage with Marquez in particular.

+ + + +
+ 🙏 Tom van Eijk +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tom van Eijk + (t.m.h.vaneijk@tilburguniversity.edu) +
+
2023-05-22 10:04:44
 +
+

*Thread Reply:* Thank you so much Ernie! That sounds like a very interesting direction to keep in mind during research!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-05-22 16:37:44
 +
+

@channel +For an easily digestible recap of recent events, communications and releases in the community, please sign up for our new monthly newsletter! Look for it in your inbox soon.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Bernat Gabor + (gaborjbernat@gmail.com) +
+
2023-05-22 23:32:16
 +
+

looking here https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.json#L64 it show that the schemaURL must be set, but then the examples in https://openlineage.io/getting-started#step-1-start-a-run do not contain it, is this a bug, expected? 😄

+
+
openlineage.io
+ + + +
Getting Started | OpenLineage Docs
+
+ Data lineage is the foundation for a new generation of powerful, context-aware data tools and best practices. OpenLineage enables consistent collection of lineage metadata, creating a deeper understanding of how data is produced and used. +
+ + + + + + +
Original URL: https://openlineage.io/getting-started#step-1-start-a-run
+ + + +
+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.json | OpenLineage.json>
+
+ <pre><code> "schemaURL" +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-05-23 07:24:09
 +
+

*Thread Reply:* yeah, it's a bug

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Bernat Gabor + (gaborjbernat@gmail.com) +
+
2023-05-23 12:00:48
 +
+

*Thread Reply:* so it's optional then? 😄 or bug in the example?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Bernat Gabor + (gaborjbernat@gmail.com) +
+
2023-05-23 12:02:09
 +
+

I noticed that DataQualityAssertionsDatasetFacet inherits from InputDatasetFacet, https://github.com/OpenLineage/OpenLineage/blob/main/spec/facets/DataQualityAssertionsDatasetFacet.json though I think should do from DatasetFacet like all else 🤔

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-05-23 14:20:09
 +
+

@channel +Two years ago last Saturday, we released the first version of OpenLineage, a test release of the Python client. So it seemed like an appropriate time to share our first annual ecosystem survey, which is both a milestone in the project’s growth and an important effort to set our course. This survey has been designed to help us learn more about who is using OpenLineage, what your lineage needs are, and what new tools you hope the project will support. Thank you in advance for taking the time to share your opinions and vision for the project! (Please note: the survey might seem longer than it actually is due to the large number of optional questions. Not all questions apply to all use cases.)

+
+
Google Docs
+ + + +
2023 OpenLineage Ecosystem Survey
+
+ Our annual survey is designed to help us learn more about who is using OpenLineage, what your lineage needs are, and what new tools you hope the project will support. Your responses will help shape the future of the project. Thank you for participating! +
+ + + + + + + + +
Original URL: https://bit.ly/ecosystem_survey
+ + + +
+ + + +
+ 🙌 Harel Shein, Maciej Obuchowski, Atif Tahir, Peter Hicks, Tamara Fingerlin, Paweł Leszczyński +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sharanya Santhanam + (santhanamsharanya@gmail.com) +
+
2023-05-23 18:59:46
 +
+

Open Lineage Spark Integration our spark workloads on Spark 2.4 are correctly setting .config("spark.sql.catalogImplementation", "hive") however sql queries for CREATE/INSERT INTO dont recoognize the datasets as “Hive”. As per https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/supported-commands.md USING HIVE is needed for appropriate parsing. Why is that the case ? Why cant HQL format for CREATE/INSERT be supported?

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/supported-commands.md | supported-commands.md>
+
+ <p><strong>Supported Commands</strong></p> + +<p>The page lists Spark commands which are supported by OpenLineage.</p> + +<p><strong>V2 commands</strong></p> + +<p>Spark commands to integrate with external catalogs like <code>iceberg</code>, <code>delta</code>, etc.</p> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sharanya Santhanam + (santhanamsharanya@gmail.com) +
+
2023-05-23 19:01:43
 +
+

*Thread Reply:* @Michael Collado wondering if you could shed some light here

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-05-24 05:39:01
 +
+

*Thread Reply:* can you show logical plan of your Spark job? I think using hive is not the most important part, but whether job's LogicalPlan parses to CreateHiveTableAsSelectCommand or InsertIntoHiveTable

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sharanya Santhanam + (santhanamsharanya@gmail.com) +
+
2023-05-24 19:37:02
 +
+

*Thread Reply:* It parses into InsertIntoHadoopFsRelationCommand. example +== Optimized Logical Plan == +InsertIntoHadoopFsRelationCommand <s3a://uchmsdev03/default/sharanyaOutputTable>, false, [id#89], Parquet, [serialization.format=1, mergeSchema=false, partitionOverwriteMode=dynamic], Append, CatalogTable( +Database: default +Table: sharanyaoutputtable +Owner: 2700940971 +Created Time: Thu Jun 09 11:13:35 PDT 2022 +Last Access: UNKNOWN +Created By: Spark 3.2.0 +Type: EXTERNAL +Provider: hive +Table Properties: [transient_lastDdlTime=1654798415] +Location: <s3a://uchmsdev03/default/sharanyaOutputTable> +Serde Library: org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe +InputFormat: org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat +OutputFormat: org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat +Storage Properties: [serialization.format=1] +Partition Provider: Catalog +Partition Columns: [`id`] +Schema: root + |-- displayName: string (nullable = true) + |-- serialnum: string (nullable = true) + |-- osversion: string (nullable = true) + |-- productfamily: string (nullable = true) + |-- productmodel: string (nullable = true) + |-- id: string (nullable = true) +), org.apache.spark.sql.execution.datasources.CatalogFileIndex@5fe23214, [displayName, serialnum, osversion, productfamily, productmodel, id] ++- Union false, false + :- Relation default.tablea[displayName#84,serialnum#85,osversion#86,productfamily#87,productmodel#88,id#89] parquet + +- Relation default.tableb[displayName#90,serialnum#91,osversion#92,productfamily#93,productmodel#94,id#95] parquet

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sharanya Santhanam + (santhanamsharanya@gmail.com) +
+
2023-05-24 19:39:54
 +
+

*Thread Reply:* using spark 3.2 & this is the query +spark.sql(s"INSERT INTO default.sharanyaOutput select ** from (SELECT ** from default.tableA union all " + + s"select ** from default.tableB)")

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Rakesh Jain + (rakeshj@us.ibm.com) +
+
2023-05-24 01:09:58
 +
+

Is there any example of how sourceCodeLocation / git info can be used from a spark job? What do we need to set to be able to see that as part of metadata?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-05-24 05:37:06
 +
+

*Thread Reply:* I think we can't really get it from Spark context, as Spark jobs are submitted in compiled, jar form, instead of plain text like for example Airflow dags.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Rakesh Jain + (rakeshj@us.ibm.com) +
+
2023-05-25 02:15:35
 +
+

*Thread Reply:* How about Jupyter Notebook based spark job?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-05-25 08:44:18
 +
+

*Thread Reply:* I don't think it changes much - but maybe @Paweł Leszczyński knows more

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-05-25 11:24:21
 +
+

@channel +Deprecation notice: support for Airflow 2.1 will end in about two weeks, when it will be removed from testing. The exact date will be announced as we get closer to it — this is just a heads up. After that date, use 2.1 at your own risk! (Note: the next release, 0.27.0, will still support 2.1.)

+ + + +
+ 👍 Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sheeri Cabral (Collibra) + (sheeri.cabral@collibra.com) +
+
2023-05-25 11:27:39
 +
+

For the OpenLineageSparkListener, is there a way to configure it to send packets locally, e.g. save to a file? (instead of pushing to a URL destination)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
alexandre bergere + (alexandre.pro.bergere@gmail.com) +
+
2023-05-25 12:00:04
 +
+

*Thread Reply:* We developed a FileTransport class in order to save locally in json file our metrics if you interested in

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sheeri Cabral (Collibra) + (sheeri.cabral@collibra.com) +
+
2023-05-25 12:00:37
 +
+

*Thread Reply:* Does it also save the openlineage information, e.g. inputs/outputs?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
alexandre bergere + (alexandre.pro.bergere@gmail.com) +
+
2023-05-25 12:02:07
 +
+

*Thread Reply:* yes it save all json information, inputs / ouputs included

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sheeri Cabral (Collibra) + (sheeri.cabral@collibra.com) +
+
2023-05-25 12:03:03
 +
+

*Thread Reply:* Yes! then I am very interested. Is there guidance on how to use the FileTransport class?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-05-25 13:06:22
 +
+

*Thread Reply:* @alexandre bergere it would be pretty useful contribution if you can submit it 🙂

+ + + +
+ 🙌 alexandre bergere +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
alexandre bergere + (alexandre.pro.bergere@gmail.com) +
+
2023-05-25 13:08:28
 +
+

*Thread Reply:* We are using it on a transformed OpenLineage library we developed ! I'm going to make a PR in order to share it with you :)

+ + + +
+ 👍 Julien Le Dem, Anirudh Shrinivason +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-05-25 13:56:48
 +
+

*Thread Reply:* would be great to have. I had it in mind to implement as an enabler for databricks integration tests. great to hear that!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
alexandre bergere + (alexandre.pro.bergere@gmail.com) +
+
2023-05-29 08:19:46
 +
+

*Thread Reply:* PR sent: https://github.com/OpenLineage/OpenLineage/pull/1891 🙂 +@Maciej Obuchowski could you tell me how to update the documentation once approved please?

+
+ + + + +
#1891 Adding a file transport
+
+ <p><strong>Problem</strong></p> + +<p>We were missing a File transport for OpenLineage as we have a lot of log ingestion software depending on object storage/ (often fuse-mounted in our runtime environments). It was also asked on the Projects's Slack, so here's our take</p> + +<p><strong>Solution</strong></p> + +<p>A FileTransport and its configuration classes were created. This transport supports append mode or write-new-file mode (especially useful on object storage not supporting append mode e.g: Databricks DBFS fuse)</p> + +<p>Code written with <a href="https://github.com/TomRibuot">@TomRibuot</a> (<a href="https://github.com/TomRibuot">https://github.com/TomRibuot</a>).</p> + +<p><strong>One-line summary:</strong></p> + +<p>Adding a file transport</p> + +<p><strong>Checklist</strong></p> + +<p>☐ You've <a href="https://github.com/OpenLineage/OpenLineage/blob/main/why-the-dco.md">signed-off</a> your work +☐ Your pull request title follows our <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CONTRIBUTING.md#creating-pull-requests">guidelines</a> +☐ Your changes are accompanied by tests (<em>if relevant</em>) +☐ Your change contains a <a href="https://kurtisnusbaum.medium.com/stacked-diffs-keeping-phabricator-diffs-small-d9964f4dcfa6">small diff</a> and is self-contained +☐ You've updated any relevant documentation (<em>if relevant</em>) +☐ Your comment includes a one-liner for the changelog about the specific purpose of the change (<em>if necessary</em>) +☐ You've versioned the core OpenLineage model or facets according to <a href="https://docs.snowplowanalytics.com/docs/pipeline-components-and-applications/iglu/common-architecture/schemaver">SchemaVer</a> (<em>if relevant</em>) +☐ You've added a <a href="https://github.com/OpenLineage/OpenLineage/tree/main/.github/header_templates.md">header</a> to source files (<em>if relevant</em>)</p> + +<hr /> + +<p>SPDX-License-Identifier: Apache-2.0 <br /> +Copyright 2018-2023 contributors to the OpenLineage project</p> +
+ +
+
Labels
+ client/python +
+ +
+
Comments
+ 1 +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-05-29 08:36:21
 +
+

*Thread Reply:* @alexandre bergere we have separate repo for website + docs: https://github.com/OpenLineage/docs

+
+ + + + +
OpenLineage/docs
+
+ Documentation for OpenLineage +
+ +
+
Website
+ <https://openlineage.io/docs> +
+ +
+
Stars
+ 5 +
+ + + + + + + + +
+ + + +
+ 🙏 alexandre bergere +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Bramha Aelem + (bramhaaelem@gmail.com) +
+
2023-05-25 16:40:26
 +
+

Hi Team- When we run databricks job, lot of dbfs path namespaces are getting created. Can someone please let us know how to overwrite the symlink namespaces and link with the spark app name or openlineage namespace marquez UI.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harshini Devathi + (harshini.devathi@tigeranalytics.com) +
+
2023-05-26 09:09:09
 +
+

Hello,

+ +

I am looking to connect the common data model in postgres marquez database and Azure Purview (which uses Apache Atlas API's) lineage endpoint. Does anyone have any how-to on this or can point me to some useful links for this?

+ +

Thanks in advance.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-05-26 13:08:56
 +
+

*Thread Reply:* I wonder if this blog post might help? https://openlineage.io/blog/openlineage-microsoft-purview

+
+
openlineage.io
+ + + +
Microsoft Purview Accelerates Lineage Extraction from Azure Databricks | OpenLineage Docs
+
+ A new collaboration between Microsoft and OpenLineage is making lineage extraction possible for Azure Databricks and Microsoft Purview users. +
+ + + + + + +
Original URL: https://openlineage.io/blog/openlineage-microsoft-purview
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-05-26 16:13:38
 +
+

*Thread Reply:* This might not fully match your use case, either, but might help: https://learn.microsoft.com/en-us/samples/microsoft/purview-adb-lineage-solution-accelerator/azure-databricks-to-purview-lineage-connector/

+
+
learn.microsoft.com
+ + + +
Azure Databricks to Purview Lineage Connector - Code Samples
+
+ This solution accelerator, together with the [OpenLineage](<http://openlineage.io>) project, provides a connector that will transfer lineage metadata fr... +
+ + + + + + + + +
Original URL: https://learn.microsoft.com/en-us/samples/microsoft/purview-adb-lineage-solution-accelerator/azure-databricks-to-purview-lineage-connector/
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harshini Devathi + (harshini.devathi@tigeranalytics.com) +
+
2023-06-01 23:23:49
 +
+

*Thread Reply:* Thanks @Michael Robinson

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Bernat Gabor + (gaborjbernat@gmail.com) +
+
2023-05-26 12:44:09
 +
+

Are there any constraints on facets? Such as is reasonable to expect that a single job will have a single parent? The schema hints to this by making the parent a single entry; but then one can send different parents for the START and COMPLETE event? 🤔

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-05-29 05:04:32
 +
+

*Thread Reply:* I think, for now such thing is not defined other than by implementation of consumers.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Bernat Gabor + (gaborjbernat@gmail.com) +
+
2023-05-30 10:32:09
 +
+

*Thread Reply:* Any reason for that?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-06-01 10:25:33
 +
+

*Thread Reply:* The idea is that for particular run, facets can be attached to any event type.

+ +

This has advantages, for example, job that modifies dataset that it's also reading from, can get particular version of dataset it's reading from and attach it on start; it would not work if you tried to do it on complete as the dataset would change by then.

+ +

Similarly, if the job is creating dataset, we could not get additional metadata on it, so we can attach those information only on complete.

+ +

There are also cases where we want facets to be cumulative. The reason for this are streaming jobs. For example, with Apache Flink, we could emit metadata on each checkpoint (or every N checkpoints) that contain metadata that show us how the job is progressing.

+ +

Generally consumers should be agnostic for that, but we don't want to overspecify what consumers should do - as people might want to use OL data in different ways, or even ignore some data we're sending.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Bernat Gabor + (gaborjbernat@gmail.com) +
+
2023-05-30 17:49:54
 +
+

Any reason why the lifecycle state change facet is not just on the output? But is also allowed on the inputs? 🤔 https://openlineage.io/docs/spec/facets/dataset-facets/lifecycle_state_change I can't see how would it be interpreted for an input 🤔

+
+
openlineage.io
+ + + +
Lifecycle State Change Facet | OpenLineage Docs
+
+ Example: +
+ + + + + + +
Original URL: https://openlineage.io/docs/spec/facets/dataset-facets/lifecycle_state_change
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-06-01 10:18:48
 +
+

*Thread Reply:* I think it should be output-only, yes.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-06-01 10:19:14
 +
+

*Thread Reply:* @Paweł Leszczyński what do you think?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-06-02 08:35:13
 +
+

*Thread Reply:* yes, should be output only I think

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Bernat Gabor + (gaborjbernat@gmail.com) +
+
2023-06-05 13:39:07
 +
+

*Thread Reply:* should we move it over then? 😄

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Bernat Gabor + (gaborjbernat@gmail.com) +
+
2023-06-05 13:39:31
 +
+

*Thread Reply:* under Output Dataset Facets that is

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-06-01 12:30:00
 +
+

@channel +The first issue of OpenLineage News is now available. To get it directly in your inbox when it’s published, become a subscriber.

+ + + +
+ 🚀 Willy Lulciuc, Jakub Dardziński, Maciej Obuchowski, Bernat Gabor, Harel Shein, Laurent Paris, Tamara Fingerlin, Perttu Salonen +
+ +
+ 🔥 Willy Lulciuc, Natalie Zeller, Ernie Ostic, Laurent Paris +
+ +
+ 💯 Willy Lulciuc +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-06-01 14:23:17
 +
+

*Thread Reply:* Correction: Julien and Willy’s talk at Data+AI Summit will take place on June 28

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-06-01 13:50:23
 +
+

Hello all, I’m opening a vote to release 0.27.0, featuring: +• Spark: fixed column lineage from databricks in the case of aggregate queries +• Python client: configurable job-name filtering +• Airflow: fixed urllib.parse.urlparse in case of [] values +Three +1s from committers will authorize an immediate release.

+ + + +
+ ➕ Jakub Dardziński, Maciej Obuchowski, Willy Lulciuc, Paweł Leszczyński +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-06-02 10:30:39
 +
+

*Thread Reply:* Thanks, all. The release is authorized and will be initiated on Monday in accordance with our policy here.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-06-02 13:13:18
 +
+

@channel +This month’s TSC meeting is next Thursday, June 8th, at 10:00 am PT. On the tentative agenda: announcements, meetup updates, recent releases, static lineage progress, and open discussion. More info and the meeting link can be found on the website. All are welcome! Also, feel free to reply or DM me with discussion topics, agenda items, etc.

+
+
openlineage.io
+ + + +
TSC Meetings | OpenLineage Docs
+
+ The OpenLineage Technical Steering Committee meets monthly, and is open to all. +
+ + + + + + +
Original URL: https://openlineage.io/meetings/
+ + + +
+ + + +
+ 🙌 Sheeri Cabral (Collibra), Maciej Obuchowski, Harel Shein, alexandre bergere, Paweł Leszczyński, Willy Lulciuc +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-06-05 12:34:29
 +
+

@channel +We released OpenLineage 0.27.1, including: +Additions: +• Python client: add emission filtering mechanism and exact, regex filters #1878 @mobuchowski +Fixes: +• Spark: fix column lineage for aggregate queries on databricks #1867 @pawel-big-lebowski +• Airflow: fix unquoted [ and ] in Snowflake URIs #1883 @JDarDagran +Plus a CI fix and a proposal. +For the details, see: +Release: https://github.com/OpenLineage/OpenLineage/releases/tag/0.27.1 +Changelog: https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md +Commit history: https://github.com/OpenLineage/OpenLineage/compare/0.26.0...0.27.1 +Maven: https://oss.sonatype.org/#nexus-search;quick~openlineage +PyPI: https://pypi.org/project/openlineage-python/

+ + + +
+ 🙌 Sheeri Cabral (Collibra) +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Bernat Gabor + (gaborjbernat@gmail.com) +
+
2023-06-05 13:01:06
 +
+

Looking for a reviewer under: https://github.com/OpenLineage/OpenLineage/pull/1892 🙂

+
+ + + + +
#1892 Use prettier to format the spec
+
+ <p>Signed-off-by: Bernát Gábor <a href="mailto:bgabor8@bloomberg.net">bgabor8@bloomberg.net</a></p> +
+ +
+
Labels
+ documentation, spec +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+ 🙌 Sheeri Cabral (Collibra), Paweł Leszczyński, Michael Robinson +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-06-05 15:47:08
 +
+

*Thread Reply:* @Bernat Gabor thanks for the PR!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-06-06 08:17:47
 +
+

Hey, I request release 0.27.2 to fix potential breaking change in Python client in 0.27.1: https://github.com/OpenLineage/OpenLineage/pull/1908

+ + + +
+ ➕ Jakub Dardziński, Paweł Leszczyński, Michael Robinson, Willy Lulciuc +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-06-06 10:58:23
 +
+

*Thread Reply:* Thanks @Maciej Obuchowski. The release is authorized and will be initiated as soon as possible.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-06-06 12:33:55
 +
+

@channel +We released OpenLineage 0.27.2, including: +Fixes: +• Python client: deprecate client.from_environment, do not skip loading config #1908 @Maciej Obuchowski
+For the details, see: +Release: https://github.com/OpenLineage/OpenLineage/releases/tag/0.27.2 +Changelog: https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md +Commit history: https://github.com/OpenLineage/OpenLineage/compare/0.27.1...0.27.2 +Maven: https://oss.sonatype.org/#nexus-search;quick~openlineage +PyPI: https://pypi.org/project/openlineage-python/

+ + + +
+ 👍 Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Bernat Gabor + (gaborjbernat@gmail.com) +
+
2023-06-06 14:22:18
 +
+

Found a major bug in the python client - https://github.com/OpenLineage/OpenLineage/pull/1917, if someone can review

+
+ + + + +
#1917 Add schemaURL to run event
+
+ <p>Resolves <a href="https://github.com/OpenLineage/OpenLineage/issues/1909">#1909</a></p> +
+ +
+
Labels
+ client/python, common +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Bernat Gabor + (gaborjbernat@gmail.com) +
+
2023-06-06 14:54:47
 +
+

And also https://github.com/OpenLineage/OpenLineage/pull/1913 🙂 that fixes the type information not being packaged

+
+ + + + +
#1913 Fix type checked marker and packaging
+
+ <p>Resolves <a href="https://github.com/OpenLineage/OpenLineage/issues/1911">#1911</a></p> +
+ +
+
Labels
+ integration/airflow, integration/great-expectations, client/python, common, integration/dagster, extractor +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-06-07 09:48:58
 +
+

@channel +This month’s TSC meeting is tomorrow, and all are welcome! https://openlineage.slack.com/archives/C01CK9T7HKR/p1685725998982879

+
+ + +
+ + + } + + Michael Robinson + (https://openlineage.slack.com/team/U02LXF3HUN7) +
+ + + +
+
+ <p>@channel +This month’s TSC meeting is next Thursday, June 8th, at 10:00 am PT. On the tentative agenda: announcements, meetup updates, recent releases, static lineage progress, and open discussion. More info and the meeting link can be found on the <a href="https://openlineage.io/meetings/">website</a>. All are welcome! Also, feel free to reply or DM me with discussion topics, agenda items, etc.</p> +
+ + + + + + +
Original URL: https://openlineage.slack.com/archives/C01CK9T7HKR/p1685725998982879
+ + +
+ + Slack Conversation +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Rachana Gandhi + (rachana.gandhi410@gmail.com) +
+
2023-06-08 11:11:31
 +
+

Hi team,

+ +

I wanted a lineage of my data for my tables and column level. +I am using jupyter notebook and spark code.

+ +

spark = (SparkSession.builder.master('local') + .appName('samplespark') + .config('spark.extraListeners', 'io.openlineage.spark.agent.OpenLineageSparkListener') + .config('spark.jars.packages', 'io.openlineage:openlineagespark:0.12.0') + .config('spark.openlineage.host', 'http://marquez-api:5000') + .config('spark.openlineage.namespace', 'spark_integration') + .getOrCreate())

+ +

I used this and then opened the localhost:3000 for marquez

+ +

I can see my job there but when i click on the job when its supposed to show lineage, its just an empty screen

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Lukenoff + (john@jlukenoff.com) +
+
2023-06-08 12:39:20
 +
+

*Thread Reply:* Do you get any output in your devtools? I just ran into this yesterday and it looks like it’s related to this issue: https://github.com/MarquezProject/marquez/issues/2410

+
+ + + + +
#2410 UI error: max call stack size exceeded
+
+ <p><a href="https://user-images.githubusercontent.com/1553185/217618061-2a4c3f55-2b9d-4c41-9799-dfd035da5477.png">Screen Shot 2023-02-03 at 4 16 40 PM (1)</a></p> +
+ +
+
Labels
+ bug +
+ +
+
Comments
+ 2 +
+ + + + + + + + + +
+ + <a href="https://github.com/MarquezProject/marquez">MarquezProject/marquez</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Lukenoff + (john@jlukenoff.com) +
+
2023-06-08 12:40:01
 +
+

*Thread Reply:* Seems like more of a Marquez client-side issue than something with OL

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Rachana Gandhi + (rachana.gandhi410@gmail.com) +
+
2023-06-08 12:43:02
 +
+

*Thread Reply:* ohh but if i try using the console output, it throws ClientProtocolError

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Lukenoff + (john@jlukenoff.com) +
+
2023-06-08 12:43:41
 +
+

*Thread Reply:* Sorry I mean in the dev console of your web browser

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Rachana Gandhi + (rachana.gandhi410@gmail.com) +
+
2023-06-08 12:44:43
 +
+

*Thread Reply:* this is the dev console in browser

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Lukenoff + (john@jlukenoff.com) +
+
2023-06-08 12:47:59
 +
+

*Thread Reply:* Seems like it’s coming from this line. Are there any job facets defined when you fetch from the API directly? That seems like kind of an old version of OL so maybe the schema is incompatible with the version Marquez is expecting

+
+ + + + +
<https://github.com/MarquezProject/marquez/blob/9fece97cf4481bdffd3b71c7e46de2dcac74265f/web/src/components/jobs/RunInfo.tsx | RunInfo.tsx>
+
+ <pre><code> {&lt;MqCode code={(jobFacets?.sql as SqlFacet)?.query} language={'sql'}/&gt;} +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/MarquezProject/marquez">MarquezProject/marquez</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Rachana Gandhi + (rachana.gandhi410@gmail.com) +
+
2023-06-08 12:51:21
 +
+

*Thread Reply:* from pyspark.sql import SparkSession

+ +

spark = (SparkSession.builder.master('local') + .appName('sample_spark') + .config('spark.extraListeners', 'io.openlineage.spark.agent.OpenLineageSparkListener') + .config('spark.jars.packages', 'io.openlineage:openlineage_spark:0.12.0') + .config('spark.openlineage.host', '<http://marquez-api:5000>') + .config('spark.openlineage.namespace', 'spark_integration')
+ .getOrCreate())

+ +

spark.sparkContext.setLogLevel("INFO")

+ +

spark.createDataFrame([ + {'a': 1, 'b': 2}, + {'a': 3, 'b': 4} +]).write.mode("overwrite").saveAsTable("temp_table8")

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Rachana Gandhi + (rachana.gandhi410@gmail.com) +
+
2023-06-08 12:51:49
 +
+

*Thread Reply:* This is my only code, I havent done anything apart from this

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Lukenoff + (john@jlukenoff.com) +
+
2023-06-08 12:52:30
 +
+

*Thread Reply:* I would try a more recent version of OL. Looks like you’re using 0.12.0 and I think the project is on 0.27.x currently

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Rachana Gandhi + (rachana.gandhi410@gmail.com) +
+
2023-06-08 12:55:07
 +
+

*Thread Reply:* so i should change io.openlineage:openlineage_spark:0.12.0 to io.openlineage:openlineage_spark:0.27.1?

+ + + +
+ 👍 John Lukenoff, Julien Le Dem +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Rachana Gandhi + (rachana.gandhi410@gmail.com) +
+
2023-06-08 13:10:03
 +
+

*Thread Reply:* it executed well, unable to see it in marquez

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Rachana Gandhi + (rachana.gandhi410@gmail.com) +
+
2023-06-08 13:18:16
 +
+

*Thread Reply:* marquez didnt get updated

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Rachana Gandhi + (rachana.gandhi410@gmail.com) +
+
2023-06-08 13:20:44
 +
+

*Thread Reply:* I am actually doing a POC on OpenLineage to find table and column level lineage for my team at Amazon. +If this goes through, the team could use openlineage to track data lineage on a larger scale..

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Lukenoff + (john@jlukenoff.com) +
+
2023-06-08 13:24:49
 +
+

*Thread Reply:* Maybe marquez is still pulling the data from the previous run using the old OL version. Do you still get the same error in the browser console? Do you get the same result if you rebuild and start with a clean marquez db?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Rachana Gandhi + (rachana.gandhi410@gmail.com) +
+
2023-06-08 13:25:10
 +
+

*Thread Reply:* yes i did that as well

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Rachana Gandhi + (rachana.gandhi410@gmail.com) +
+
2023-06-08 13:25:49
 +
+

*Thread Reply:* the error was present only once you clicked on any of the jobs in marquez, +since my job isnt showing up i cant check for the error itself

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Rachana Gandhi + (rachana.gandhi410@gmail.com) +
+
2023-06-08 13:26:29
 +
+

*Thread Reply:* docker run --network sparkdefault -p 3000:3000 -e MARQUEZHOST=marquez-api -e MARQUEZ_PORT=5000 --link marquez-api:marquez-api marquezproject/marquez-web:0.19.1

+ +

used this to rebuild marquez

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Lukenoff + (john@jlukenoff.com) +
+
2023-06-08 13:26:54
 +
+

*Thread Reply:* That’s odd, sorry, that’s probably the most I can help, I’m kinda new to OL/Marquez as well 😅

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Rachana Gandhi + (rachana.gandhi410@gmail.com) +
+
2023-06-08 13:27:41
 +
+

*Thread Reply:* no problem, can you refer me to someone who would know, so that i can ask them?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Lukenoff + (john@jlukenoff.com) +
+
2023-06-08 13:29:25
 +
+

*Thread Reply:* Actually looking at in now I think you’re using a slightly outdated version of marquez-web too. I would update that tag to at least 0.33.0. that’s what I’m using

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Lukenoff + (john@jlukenoff.com) +
+
2023-06-08 13:30:10
 +
+

*Thread Reply:* Other than that I would ask in the marquez slack channel or raise an issue in github on that project. Seems like more of an issue with Marquez since some at least some data is rendering in the UI initially

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Rachana Gandhi + (rachana.gandhi410@gmail.com) +
+
2023-06-08 13:32:58
 +
+

*Thread Reply:* nope that version also didnt help

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Rachana Gandhi + (rachana.gandhi410@gmail.com) +
+
2023-06-08 13:33:19
 +
+

*Thread Reply:* can you share their slack link?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Lukenoff + (john@jlukenoff.com) +
+
2023-06-08 13:34:52
 +
+

*Thread Reply:* http://bit.ly/MarquezSlack

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Rachana Gandhi + (rachana.gandhi410@gmail.com) +
+
2023-06-08 13:35:08
 +
+

*Thread Reply:* that link is no longer active

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2023-06-09 18:44:25
 +
+

*Thread Reply:* Hello @Rachana Gandhi could you point to the doc where you found the example .config(‘spark.jars.packages’, ‘io.openlineage:openlineage_spark:0.12.0’) ? We should update it to have the latest version instead.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Rachana Gandhi + (rachana.gandhi410@gmail.com) +
+
2023-06-09 18:54:49
 +
+

*Thread Reply:* https://openlineage.io/docs/integrations/spark/quickstart_local/

+
+
openlineage.io
+ + + +
Quickstart with Jupyter | OpenLineage Docs
+
+ Trying out the Spark integration is super easy if you already have Docker Desktop and git installed. +
+ + + + + + +
Original URL: https://openlineage.io/docs/integrations/spark/quickstart_local/
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Rachana Gandhi + (rachana.gandhi410@gmail.com) +
+
2023-06-09 18:59:17
 +
+

*Thread Reply:* https://openlineage.io/docs/guides/spark

+ +

also the docker compose here has an earlier version of marquez

+
+
openlineage.io
+ + + +
Using OpenLineage with Spark | OpenLineage Docs
+
+ Adapted from a blog post by Michael Collado +
+ + + + + + +
Original URL: https://openlineage.io/docs/guides/spark
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harshit Soni + (harshit.soni@angelbroking.com) +
+
2023-07-13 17:00:54
 +
+

*Thread Reply:* Facing same issue with my initial POC. Did we get any solution for this?

+ + + +
+
+
+
+ + + + + + + +
+
+ + + + +
+ +
Bernat Gabor + (gaborjbernat@gmail.com) +
+
2023-06-08 14:36:38
 +
+

Approve a new release 🙂

+ + + +
+ ➕ Michael Robinson, Willy Lulciuc, Maciej Obuchowski, Jakub Dardziński +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-06-08 14:43:55
 +
+

*Thread Reply:* Requesting a release? 3 +1s from committers will authorize. More info here: https://github.com/OpenLineage/OpenLineage/blob/main/GOVERNANCE.md

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Bernat Gabor + (gaborjbernat@gmail.com) +
+
2023-06-08 14:44:14
 +
+

*Thread Reply:* Yeah, that one 😊

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Bernat Gabor + (gaborjbernat@gmail.com) +
+
2023-06-08 14:44:44
 +
+

*Thread Reply:* Because the python client is broken as is today without a new release

+ + + +
+ 👍 Michael Robinson +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-06-08 18:45:04
 +
+

*Thread Reply:* Thanks, all. The release is authorized and will be initiated by EOB next Tuesday, but in all likelihood well before then.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Bernat Gabor + (gaborjbernat@gmail.com) +
+
2023-06-08 19:06:34
 +
+

*Thread Reply:* cool

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-06-12 13:15:26
 +
+

@channel +We released OpenLineage 0.28.0, including: +Added +• dbt: add Databricks compatibility #1829 @Ines70 +Fixed +• Fix type-checked marker and packaging #1913 @gaborbernat +• Python client: add schemaURL to run event #1917 @gaborbernat +For the details, see: +Release: https://github.com/OpenLineage/OpenLineage/releases/tag/0.28.0 +Changelog: https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md +Commit history: https://github.com/OpenLineage/OpenLineage/compare/0.27.2...0.28.0 +Maven: https://oss.sonatype.org/#nexus-search;quick~openlineage +PyPI: https://pypi.org/project/openlineage-python/

+ + + +
+ 🚀 Maciej Obuchowski, Willy Lulciuc, Francis McGregor-Macdonald +
+ +
+ 👍 Ines DAHOUMANE -COWORKING PARIS- +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-06-12 14:35:56
 +
+

@channel +Meetup announcement: there’s another meetup happening soon! This one will be an evening event on 6/22 in New York at Collibra’s HQ. For details and to sign up, please join the meetup group: https://www.meetup.com/data-lineage-meetup/events/294065396/. Thanks to @Sheeri Cabral (Collibra) for cohosting and providing a space.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-06-12 23:27:16
 +
+

Hi, just curious, does openlineage have a log4j integration?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-06-13 04:44:28
 +
+

*Thread Reply:* Do you mean to just log events to logging backend?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-06-13 04:54:30
 +
+

*Thread Reply:* Hmm more like have a separate logging config for sending all the logs to a backend

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-06-13 04:54:38
 +
+

*Thread Reply:* Not the events itself

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-06-13 05:01:10
 +
+

*Thread Reply:* @Anirudh Shrinivason with Spark integration?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-06-13 05:01:59
 +
+

*Thread Reply:* It uses slf4j so you should be able to set up your log4j logger

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-06-13 05:10:55
 +
+

*Thread Reply:* Yeah with the spark integration. Ahh I see. Okay sure thanks!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-06-21 23:21:14
 +
+

*Thread Reply:* ~Hi @Maciej Obuchowski May I know what the class path I should be using for setting up the log4j if I want to set it up for OL related logs? Is there some guide or runbook to setting up the log4j with OL? Thanks!~ +Nvm lol found it! 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Vamshi krishna + (vnallamothu@cardinalcommerce.com) +
+
2023-06-13 12:19:01
 +
+

Hello all, we are just starting to use Marquez as part of our POC. We are following the getting started guide at https://openlineage.io/getting-started/ to set the environment on an AWS Ec2 instance. When we are running ./docker/up.sh, it is not bringing up marquez-web container. Also, we are not able to access Admin UI at 5000 and 5001 ports.

+ +

Docker version: 24.0.2 +Docker compose version: 2.18.1 +OS: Ubuntu_20.04

+ +

Can someone please let me know what I am missing? +Note: I had to modify docker-compose command in up.sh as per docker compose V2.

+ +

Also, we are seeing following log when our loadbalancer is checking for health:

+ +

WARN [2023-06-13 15:35:31,040] marquez.logging.LoggingMdcFilter: status: 404 +172.30.1.206 - - [13/Jun/2023:15:35:42 +0000] "GET / HTTP/1.1" 200 535 "-" "ELB-HealthChecker/2.0" 1 +172.30.1.206 - - [13/Jun/2023:15:35:42 +0000] "GET / HTTP/1.1" 404 43 "-" "ELB-HealthChecker/2.0" 2 +WARN [2023-06-13 15:35:42,866] marquez.logging.LoggingMdcFilter: status: 404

+
+
openlineage.io
+ + + +
Getting Started | OpenLineage Docs
+
+ Data lineage is the foundation for a new generation of powerful, context-aware data tools and best practices. OpenLineage enables consistent collection of lineage metadata, creating a deeper understanding of how data is produced and used. +
+ + + + + + +
Original URL: https://openlineage.io/getting-started/
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kavitha + (kkandaswamy@cardinalcommerce.com) +
+
2023-06-14 10:42:41
 +
+

*Thread Reply:* Hello, is anyone eho has recently installed latest version of marquez/open-lineage-spark using docker image available to help Vamshi and I or provide any pointers? Thank you

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-06-15 03:38:38
 +
+

*Thread Reply:* if you're working on mac, you can have an issue related to port 5000. Instructions here https://github.com/MarquezProject/marquez#quickstart provides a workaround for that ./docker/up.sh --api-port 9000

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kavitha + (kkandaswamy@cardinalcommerce.com) +
+
2023-06-15 08:43:33
 +
+

*Thread Reply:* @Paweł Leszczyński, thank you, we were using ubuntu on an EC2 instance and each time we are running into different errors and are never able to access the application page, web server, the admin interface, we have run out of ideas of what else to try differently to get this setup up and running

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Vamshi krishna + (vnallamothu@cardinalcommerce.com) +
+
2023-06-22 14:47:00
 +
+

*Thread Reply:* @Michael Robinson Can you please help us here?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-06-22 14:58:57
 +
+

*Thread Reply:* @Vamshi krishna I’m sorry you’re still blocked. Thanks for the information about your system. Would you please share some of the errors you are getting? More details would help us reproduce and diagnose.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kavitha + (kkandaswamy@cardinalcommerce.com) +
+
2023-06-22 16:35:00
 +
+

*Thread Reply:* @Michael Robinson, thank you, vamshi and i will share the errors that we are running into shortly

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Vamshi krishna + (vnallamothu@cardinalcommerce.com) +
+
2023-06-23 09:48:16
 +
+

*Thread Reply:* We are following https://openlineage.io/getting-started/ guide and trying to set up Marquez on a ubuntu ec2 instance. Following are versions of docker, docker compose and ubuntu

+
+
openlineage.io
+ + + +
Getting Started | OpenLineage Docs
+
+ Data lineage is the foundation for a new generation of powerful, context-aware data tools and best practices. OpenLineage enables consistent collection of lineage metadata, creating a deeper understanding of how data is produced and used. +
+ + + + + + +
Original URL: https://openlineage.io/getting-started/
+ + + +
+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Vamshi krishna + (vnallamothu@cardinalcommerce.com) +
+
2023-06-23 09:49:51
 +
+

*Thread Reply:* @Michael Robinson When we follow the documentation without changing anything and run sudo ./docker/up.sh we are seeing following errors:

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Vamshi krishna + (vnallamothu@cardinalcommerce.com) +
+
2023-06-23 10:00:38
 +
+

*Thread Reply:* So, I edited up.sh file and modified docker compose command by removing --log-level flag and ran sudo ./docker/up.sh and found following errors:

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Vamshi krishna + (vnallamothu@cardinalcommerce.com) +
+
2023-06-23 10:02:29
 +
+

*Thread Reply:* Then I copied .env.example to .env since compose needs .env file

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Vamshi krishna + (vnallamothu@cardinalcommerce.com) +
+
2023-06-23 10:05:04
 +
+

*Thread Reply:* I got this error:

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Vamshi krishna + (vnallamothu@cardinalcommerce.com) +
+
2023-06-23 10:09:24
 +
+

*Thread Reply:* since I am getting timeouts, I thought it might be an issue with proxy. So, I followed this doc: https://stackoverflow.com/questions/58841014/set-proxy-on-docker and added my outbound proxy and tried

+
+
Stack Overflow
+ + + +
Set proxy on docker
+
+ Im using centos 7.7 and docker 19.03. + +I cannot pull images and getting : + +Error response from daemon: Get <https://registry-1.docker.io.v2>./: dial tcp lookup : server misbehaving +I did what all gu... +
+ + + + + + + + +
Original URL: https://stackoverflow.com/questions/58841014/set-proxy-on-docker
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Vamshi krishna + (vnallamothu@cardinalcommerce.com) +
+
2023-06-23 10:23:46
 +
+

*Thread Reply:* @Michael Robinson Then it kind of worked but seeing following errors:

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Vamshi krishna + (vnallamothu@cardinalcommerce.com) +
+
2023-06-23 10:24:31
 +
+

*Thread Reply:*

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Vamshi krishna + (vnallamothu@cardinalcommerce.com) +
+
2023-06-23 10:25:29
 +
+

*Thread Reply:* @Michael Robinson @Paweł Leszczyński Can you please see above steps and let us know what are we missing/doing wrong? I appreciate your help and time.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-06-23 10:45:39
 +
+

*Thread Reply:* The latest errors look to me like they’re being caused by postgres and might reflect a port conflict. Are you using the default port for the API (5000)? You might try using a different port. More info about this in the Marquez readme: https://github.com/MarquezProject/marquez/blob/0.35.0/README.md.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Vamshi krishna + (vnallamothu@cardinalcommerce.com) +
+
2023-06-23 10:46:55
 +
+

*Thread Reply:* Yes we are using default ports: +APIPORT=5000 +APIADMINPORT=5001 +WEBPORT=3000 +TAG=0.35.0

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Vamshi krishna + (vnallamothu@cardinalcommerce.com) +
+
2023-06-23 10:47:40
 +
+

*Thread Reply:* We see these postgres permission issues only occasionally. Other times we only see db and api containers up but not the web

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-06-23 10:52:38
 +
+

*Thread Reply:* I would try running ./docker/up.sh --api-port 9000 (see Pawel’s message above for more context.)

+ + + +
+ 👍 Vamshi krishna +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Vamshi krishna + (vnallamothu@cardinalcommerce.com) +
+
2023-06-23 10:54:18
 +
+

*Thread Reply:* Still no luck. Seeing same errors.

+ +

2023-06-23 14:53:23.971 GMT [1] LOG: could not open configuration file "/etc/postgresql/postgresql.conf": Permission denied +marquez-db | 2023-06-23 14:53:23.971 GMT [1] FATAL: configuration file "/etc/postgresql/postgresql.conf" contains errors

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Vamshi krishna + (vnallamothu@cardinalcommerce.com) +
+
2023-06-23 10:54:43
 +
+

*Thread Reply:* ERROR [2023-06-23 14:53:42,269] org.apache.tomcat.jdbc.pool.ConnectionPool: Unable to create initial connections of pool. +marquez-api | ! java.net.UnknownHostException: postgres +marquez-api | ! at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:567) +marquez-api | ! at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327) +marquez-api | ! at java.base/java.net.Socket.connect(Socket.java:633) +marquez-api | ! at org.postgresql.core.PGStream.createSocket(PGStream.java:243) +marquez-api | ! at org.postgresql.core.PGStream.&lt;init&gt;(PGStream.java:98) +marquez-api | ! at org.postgresql.core.v3.ConnectionFactoryImpl.tryConnect(ConnectionFactoryImpl.java:132) +marquez-api | ! at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:258) +marquez-api | ! ... 26 common frames omitted +marquez-api | ! Causing: org.postgresql.util.PSQLException: The connection attempt failed. +marquez-api | ! at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:354) +marquez-api | ! at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:54) +marquez-api | ! at org.postgresql.jdbc.PgConnection.&lt;init&gt;(PgConnection.java:253) +marquez-api | ! at org.postgresql.Driver.makeConnection(Driver.java:434) +marquez-api | ! at org.postgresql.Driver.connect(Driver.java:291) +marquez-api | ! at org.apache.tomcat.jdbc.pool.PooledConnection.connectUsingDriver(PooledConnection.java:346) +marquez-api | ! at org.apache.tomcat.jdbc.pool.PooledConnection.connect(PooledConnection.java:227) +marquez-api | ! at org.apache.tomcat.jdbc.pool.ConnectionPool.createConnection(ConnectionPool.java:768) +marquez-api | ! at org.apache.tomcat.jdbc.pool.ConnectionPool.borrowConnection(ConnectionPool.java:696) +marquez-api | ! at org.apache.tomcat.jdbc.pool.ConnectionPool.init(ConnectionPool.java:495) +marquez-api | ! at org.apache.tomcat.jdbc.pool.ConnectionPool.&lt;init&gt;(ConnectionPool.java:153) +marquez-api | ! at org.apache.tomcat.jdbc.pool.DataSourceProxy.pCreatePool(DataSourceProxy.java:118) +marquez-api | ! at org.apache.tomcat.jdbc.pool.DataSourceProxy.createPool(DataSourceProxy.java:107) +marquez-api | ! at org.apache.tomcat.jdbc.pool.DataSourceProxy.getConnection(DataSourceProxy.java:131) +marquez-api | ! at org.flywaydb.core.internal.jdbc.JdbcUtils.openConnection(JdbcUtils.java:48) +marquez-api | ! at org.flywaydb.core.internal.jdbc.JdbcConnectionFactory.&lt;init&gt;(JdbcConnectionFactory.java:75) +marquez-api | ! at org.flywaydb.core.FlywayExecutor.execute(FlywayExecutor.java:147) +marquez-api | ! at <a href="http://org.flywaydb.core.Flyway.info">org.flywaydb.core.Flyway.info</a>(Flyway.java:190) +marquez-api | ! at marquez.db.DbMigration.hasPendingDbMigrations(DbMigration.java:73) +marquez-api | ! at marquez.db.DbMigration.migrateDbOrError(DbMigration.java:27) +marquez-api | ! at marquez.MarquezApp.run(MarquezApp.java:105) +marquez-api | ! at marquez.MarquezApp.run(MarquezApp.java:48) +marquez-api | ! at io.dropwizard.cli.EnvironmentCommand.run(EnvironmentCommand.java:67) +marquez-api | ! at io.dropwizard.cli.ConfiguredCommand.run(ConfiguredCommand.java:98) +marquez-api | ! at io.dropwizard.cli.Cli.run(Cli.java:78) +marquez-api | ! at io.dropwizard.Application.run(Application.java:94) +marquez-api | ! at marquez.MarquezApp.main(MarquezApp.java:60) +marquez-api | INFO [2023-06-23 14:53:42,274] marquez.MarquezApp: Stopping app...

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-06-23 11:06:32
 +
+

*Thread Reply:* Why do you run docker up with sudo? some of your screenshots suggest docker is not able to access docker registry. The last error java.net.UnknownHostException: postgres may be just a result of container being down. Could you verify if all the containers are up and running and if not what's the error? Are you able to test this docker.up in your laptop or other environment?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Vamshi krishna + (vnallamothu@cardinalcommerce.com) +
+
2023-06-23 11:08:34
 +
+

*Thread Reply:* Docker commands require sudo and cannot run with other user. +Postgres container is not coming up. It is failing with following errors:

+ +

2023-06-23 14:53:23.971 GMT [1] LOG: could not open configuration file "/etc/postgresql/postgresql.conf": Permission denied +marquez-db | 2023-06-23 14:53:23.971 GMT [1] FATAL: configuration file "/etc/postgresql/postgresql.conf" contains errors

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-06-23 11:10:19
 +
+

*Thread Reply:* and what does docker ps -a say about postgres container? why did it fail?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Vamshi krishna + (vnallamothu@cardinalcommerce.com) +
+
2023-06-23 11:11:36
 +
+

*Thread Reply:*

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-06-23 11:25:17
 +
+

*Thread Reply:* hmyy, no changes on our side have been done in postgresql.conf since August 2022. Did you apply any changes or have a clean clone of a repo?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Vamshi krishna + (vnallamothu@cardinalcommerce.com) +
+
2023-06-23 11:29:46
 +
+

*Thread Reply:* No we didn't make any changes

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-06-23 11:32:21
 +
+

*Thread Reply:* you did write earlier Note: I had to modify docker-compose command in up.sh as per docker compose V2.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Vamshi krishna + (vnallamothu@cardinalcommerce.com) +
+
2023-06-23 11:34:54
 +
+

*Thread Reply:* Yes all I did was modified this line: docker-compose --log-level ERROR $compose_files up $ARGS to +docker compose $compose_files up $ARGS since docker compose v2 doesn't support --log-level flag

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Vamshi krishna + (vnallamothu@cardinalcommerce.com) +
+
2023-06-23 11:37:03
 +
+

*Thread Reply:* Let me pull an older version and try

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Vamshi krishna + (vnallamothu@cardinalcommerce.com) +
+
2023-06-23 12:09:43
 +
+

*Thread Reply:* Still no luck same exact errors. Tried on a different ubuntu instance. Still seeing same errors with postgres

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Vamshi krishna + (vnallamothu@cardinalcommerce.com) +
+
2023-06-23 15:06:32
 +
+

*Thread Reply:* @Jeremy W

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-06-15 10:40:47
 +
+

Hi all, a general doubt. Would the column lineage associated with a job be present in both the start events and the complete events? Or could there be cases where the column lineage, and any output information is only present in one of the events, but not the other?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-06-15 10:49:42
 +
+

*Thread Reply:* > Or could there be cases where the column lineage, and any output information is only present in one of the events, but not the other? +Yes. Generally events regarding single run are cumulative

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-06-15 11:07:03
 +
+

*Thread Reply:* Ahh I see... Is it fair to assume that if I see column lineage in a start event, it's the full column lineage? Or could it be possible that half the lineage is in the start event, and half the lineage is in the complete event?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-06-15 22:50:51
 +
+

*Thread Reply:* Hi @Maciej Obuchowski just pinging in case you'd missed the above message. 🙇

+ + + +
+ 👀 Paweł Leszczyński +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-06-16 04:48:57
 +
+

*Thread Reply:* Actually, in this case this definitely should not happen. @Paweł Leszczyński am I right?

+ + + +
+ :gratitude_thank_you: Anirudh Shrinivason +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-06-16 04:50:16
 +
+

*Thread Reply:* @Maciej Obuchowski yes, you're

+ + + +
+ :gratitude_thank_you: Anirudh Shrinivason +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
nivethika R + (nivethikar8@gmail.com) +
+
2023-06-15 11:14:33
 +
+

Hi All.. Is JDBC supported for openLineage and marquez for columnlineage? I did some POC using tables in postgresdb and I am able to see all events but for columnLineage Iam getting it as NULL. Not sure where I am missing.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-06-16 02:14:19
 +
+

*Thread Reply:* ~No, we do have an open issue for that: https://github.com/OpenLineage/OpenLineage/issues/1758~

+
+ + + + +
#1758 Spark JDBC column lineage support
+
+ <p>Please add column lineage support also for Spark JDBC sources as discussed also in <a href="https://github.com/OpenLineage/OpenLineage/issues/1584">#1584</a></p> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-06-16 05:02:26
 +
+

*Thread Reply:* @nivethika R, I am sorry for misleading response, we've merged PR for that https://github.com/OpenLineage/OpenLineage/pull/1636. It does not support select ** but besides that, it should be operational.

+ +

Could you please try a query from our integration tests to verify if this is working for you or not: https://github.com/OpenLineage/OpenLineage/pull/1636/files#diff-137aa17091138b69681510e13e3b7d66aa9c9c7c81fe8fe13f09f0de76448dd5R46 ?

+
+ + + + +
#1636 Spark/jdbc relations
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Nagendra Kolisetty + (nkolisetty@geico.com) +
+
2023-06-16 12:12:00
 +
+

Hi There,

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Nagendra Kolisetty + (nkolisetty@geico.com) +
+
2023-06-16 12:12:43
 +
+

We are trying to install the image on the private AKS cluster and we ended up in below error

+ +

kubectl : pod marquez/pgsql-postgresql-client terminated (StartError) +At line:1 char:1

  • kubectl run pgsql-postgresql-client --rm --tty -i --restart='Never' `
  • ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +
    • CategoryInfo : NotSpecified: (pod marquez/pgs...ed (StartError):String) [], RemoteException
    • FullyQualifiedErrorId : NativeCommandError
      • +
    • +
+ +

failed to create containerd task: failed to create shim task: OCI runtime create failed: container_linux.go:380: starting container process caused: exec: "PGPASSWORD=macondo": executable file not found in $PATH: +unknown

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Nagendra Kolisetty + (nkolisetty@geico.com) +
+
2023-06-16 12:13:13
 +
+

We followed the below article to install Marquez in AKS (Azure). +By the way, we pulled the images from docker pushed it to our acr. +tried installing the postgresql via ACR and it failed with the error

+ +

https://github.com/MarquezProject/marquez/blob/main/docs/running-on-aws.md

+
+ + + + +
<https://github.com/MarquezProject/marquez/blob/main/docs/running-on-aws.md | running-on-aws.md>
+
+ <hr /> + +<p><strong>layout: running-on-aws</strong> +<strong>Running Marquez on AWS</strong></p> + +<p>This guide helps you deploy and manage Marquez on AWS <a href="https://aws.amazon.com/eks">EKS</a>.</p> + +<p><strong>PREREQUISITES</strong></p> + +<p>• <a href="https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html">AWS CLI</a> +• <a href="https://helm.sh/docs/helm/helm_install/">helm</a> +• <a href="https://kubernetes.io/docs/tasks/tools/">kubectl</a></p> + +<p><strong>AWS EKS Cluster</strong></p> + +<p>To create an AWS EKS cluster, please follow the steps outlined in the AWS EKS <a href="https://docs.aws.amazon.com/eks/latest/userguide/create-cluster.html">documentation</a>.</p> + +<p><strong>CONNECT TO AWS EKS CLUSTER</strong></p> + +<ol><li>Make sure you have configured your <a href="https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html">AWS CLI</a>, then create or update the <strong>kubeconfig</strong> file for your cluster: +<pre><code>$ aws eks --region &lt;AWS-REGION&gt; update-kubeconfig --name &lt;AWS-EKS-CLUSTER&gt; +</code></pre></li><li>Verify that the context has been switched: +<pre><code>$ kubectl config current-context +arn:aws:eks:&lt;AWS-REGION&gt;:&lt;AWS-ACCOUNT-ID&gt;:cluster/&lt;AWS-EKS-CLUSTER&gt; +</code></pre></li><li><p>Using <code>kubectl</code>, verify that you can connect to your cluster:</p> + +<pre><code>$ kubectl get svc +NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE +svc/kubernetes ClusterIP 10.100.0.1 &lt;none&gt; 443/TCP 1m +</code></pre> + +<blockquote> + <p>Note: If you're having issues connecting to your cluster, please see <a href="https://aws.amazon.com/premiumsupport/knowledge-center/eks-cluster-connection">Why can't I connect to my AWS EKS cluster?</a></p> +</blockquote></li> +</ol> + +<p><strong>AWS RDS</strong></p> + +<p>Next, create an AWS RDS instance as outlined in the AWS RDS <a href="https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_CreateDBInstance.html">documentation</a>. This database will be used to store dataset, job, and run metadata collected as <a href="https://openlineage.io">OpenLineage</a> events via the Marquez <a href="https://marquezproject.github.io/marquez/openapi.html">HTTP API</a>.</p> + +<p><strong>CREATE AWS RDS DATABASE</strong></p> + +<ol><li>Navigate to the AWS <a href="https://console.aws.amazon.com/rds/home">RDS</a> page and create a PostgreSQL database, leaving the database template as <strong>Production</strong>.</li><li>Use <code>marquez</code> as the database identifier and set the master username to <code>marquez</code>.</li><li>Choose a master password to use later in your Helm deployment (see <a href="https://github.com/MarquezProject/marquez/blob/main/chart/values.yaml#L32">password</a> in <code>values.yaml</code>).</li><li>Leave public access to the database <strong>off</strong>.</li><li>Choose the same VPC where your AWS EKS cluster resides.</li><li>In a separate tab, navigate to the AWS EKS cluster page and make note of the security group attached to your cluster.</li><li>Navigate back to the AWS RDS page and, in the security group section, add the AWS EKS cluster’s security group from <strong>step 6</strong>.</li><li>Next, under the <strong>Additional Configuration</strong> tab, enter <code>marquez</code> as the initial database name.</li><li>Finally, select <strong>Create Database</strong>.</li> +</ol> + +<p><strong>CONNECT TO AWS RDS DATABASE</strong></p> + +<ol><li>Create a <code>marquez</code> namespace: +<pre><code>$ kubectl create namespace marquez +</code></pre></li><li>Next, run the following command with your AWS RDS <code>host</code>, <code>username</code>, and <code>password</code>: +<pre><code>kubectl run pgsql-postgresql-client --rm --tty -i --restart='Never' \ + --namespace marquez \ + --image &lt;a href="http://docker.io/bitnami/postgresql:12-debian-10"&gt;docker.io/bitnami/postgresql:12-debian-10&lt;/a&gt; \ + --env="PGPASSWORD=&lt;AWS-RDS-PASSWORD&gt;" \ + --command -- psql marquez --host &lt;AWS-RDS-HOST&gt; -U &lt;AWS-RDS-USERNAME&gt; -d marquez -p 5432 +</code></pre></li> +</ol> + +<p><strong>Deploy Marquez on AWS EKS</strong> +<strong>INSTALLING MARQUEZ</strong></p> + +<ol><li>Get Marquez: +<pre><code>$ git clone git@github.com:MarquezProject/marquez.git &amp;&amp; cd chart +</code></pre></li><li><p>Install Marquez:</p> + +<pre><code>helm upgrade --install marquez . + --set marquez.db.host=&lt;AWS-RDS-HOST&gt; + --set marquez.db.user=&lt;AWS-RDS-USERNAME&gt; + --set marquez.db.password=&lt;AWS-RDS-PASSWORD&gt; + --namespace marquez + --atomic + --wait +</code></pre> + +<blockquote> + <p><strong>Note:</strong> To avoid overriding deployment settings via the command line, update the <a href="https://github.com/MarquezProject/marquez/blob/main/chart/values.yaml#L27">marquez.db</a> section of the Marquez Helm chart's <code>values.yaml</code> to include the AWS RDS <code>host</code>, <code>username</code>, and <code>password</code> in your deployment.</p> +</blockquote></li><li>Verify all the pods have come up correctly: +<pre><code>$ kubectl get pods --namespace marquez +</code></pre></li> +</ol> + +<p><strong>UNINSTALLING MARQUEZ</strong></p> + +<pre><code>helm uninstall marquez --namespace marquez +</code></pre> + +<hr /> + +<p>SPDX-License-Identifier: Apache-2.0 <br /> +Copyright 2018-2023 contributors to the Marquez project.</p> +
+ + + + + + + +
+ + <a href="https://github.com/MarquezProject/marquez">MarquezProject/marquez</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-06-21 11:07:04
 +
+

*Thread Reply:* Hi Nagendra, sorry you’re running into this error. We’re looking into it!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-06-18 09:53:19
 +
+

Hi, found this error in couple of the spark jobs: https://github.com/OpenLineage/OpenLineage/issues/1930 +Would request your help to kindly help patch thanks!

+
+ + + + +
#1930 Error in ColumnLevelLineageUtils
+
+ <p>`ERROR ColumnLevelLineageUtils: Error when invoking static method 'buildColumnLineageDatasetFacet' for Spark3 java.lang.reflect.InvocationTargetException at jdk.internal.reflect.GeneratedMethodAccessor476.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at io.openlineage.spark.agent.lifecycle.plan.column.ColumnLevelLineageUtils.buildColumnLineageDatasetFacet(ColumnLevelLineageUtils.java:35) at io.openlineage.spark.agent.lifecycle.OpenLineageRunEventBuilder.lambda$buildOutputDatasets$21(OpenLineageRunEventBuilder.java:431) at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195) at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1655) at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913) at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578) at io.openlineage.spark.agent.lifecycle.OpenLineageRunEventBuilder.buildOutputDatasets(OpenLineageRunEventBuilder.java:444) at io.openlineage.spark.agent.lifecycle.OpenLineageRunEventBuilder.populateRun(OpenLineageRunEventBuilder.java:303) at io.openlineage.spark.agent.lifecycle.OpenLineageRunEventBuilder.buildRun(OpenLineageRunEventBuilder.java:286) at io.openlineage.spark.agent.lifecycle.OpenLineageRunEventBuilder.buildRun(OpenLineageRunEventBuilder.java:238) at io.openlineage.spark.agent.lifecycle.SparkSQLExecutionContext.end(SparkSQLExecutionContext.java:95) at io.openlineage.spark.agent.OpenLineageSparkListener.sparkSQLExecEnd(OpenLineageSparkListener.java:98) at io.openlineage.spark.agent.OpenLineageSparkListener.onOtherEvent(OpenLineageSparkListener.java:84) at org.apache.spark.scheduler.SparkListenerBus.doPostEvent(SparkListenerBus.scala:100) at org.apache.spark.scheduler.SparkListenerBus.doPostEvent$(SparkListenerBus.scala:28) at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37) at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37) at org.apache.spark.util.ListenerBus.postToAll(ListenerBus.scala:117) at org.apache.spark.util.ListenerBus.postToAll$(ListenerBus.scala:101) at org.apache.spark.scheduler.AsyncEventQueue.super$postToAll(AsyncEventQueue.scala:105) at org.apache.spark.scheduler.AsyncEventQueue.$anonfun$dispatch$1(AsyncEventQueue.scala:105) at scala.runtime.java8.JFunction0$mcJ$sp.apply(JFunction0$mcJ$sp.java:23) at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62) at <a href="http://org.apache.spark.scheduler.AsyncEventQueue.org">org.apache.spark.scheduler.AsyncEventQueue.org</a>$apache$spark$scheduler$AsyncEventQueue$$dispatch(AsyncEventQueue.scala:100) at org.apache.spark.scheduler.AsyncEventQueue$$anon$2.$anonfun$run$1(AsyncEventQueue.scala:96) at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1381) at org.apache.spark.scheduler.AsyncEventQueue$$anon$2.run(AsyncEventQueue.scala:96) Caused by: java.lang.NullPointerException at io.openlineage.spark.agent.util.ScalaConversionUtils.fromSeq(ScalaConversionUtils.java:61) at io.openlineage.spark.agent.lifecycle.Rdds.findFileLikeRdds(Rdds.java:69) at io.openlineage.spark3.agent.lifecycle.plan.column.InputFieldsCollector.extractDatasetIdentifier(InputFieldsCollector.java:149) at io.openlineage.spark3.agent.lifecycle.plan.column.InputFieldsCollector.extractDatasetIdentifier(InputFieldsCollector.java:124) at io.openlineage.spark3.agent.lifecycle.plan.column.InputFieldsCollector.discoverInputsFromNode(InputFieldsCollector.java:70) at io.openlineage.spark3.agent.lifecycle.plan.column.InputFieldsCollector.collect(InputFieldsCollector.java:54) at io.openlineage.spark3.agent.lifecycle.plan.column.InputFieldsCollector.lambda$collect$0(InputFieldsCollector.java:64) at java.base/java.util.Iterator.forEachRemaining(Iterator.java:133) at scala.collection.convert.Wrappers$IteratorWrapper.forEachRemaining(Wrappers.scala:30) at java.base/java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801) at java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:658) at io.openlineage.spark3.agent.lifecycle.plan.column.InputFieldsCollector.collect(InputFieldsCollector.java:64) at io.openlineage.spark3.agent.lifecycle.plan.column.InputFieldsCollector.lambda$collect$0(InputFieldsCollector.java:64) at java.base/java.util.Iterator.forEachRemaining(Iterator.java:133) at scala.collection.convert.Wrappers$IteratorWrapper.forEachRemaining(Wrappers.scala:30) at java.base/java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801) at java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:658) at io.openlineage.spark3.agent.lifecycle.plan.column.InputFieldsCollector.collect(InputFieldsCollector.java:64) at io.openlineage.spark3.agent.lifecycle.plan.column.InputFieldsCollector.lambda$collect$0(InputFieldsCollector.java:64) at java.base/java.util.Iterator.forEachRemaining(Iterator.java:133) at scala.collection.convert.Wrappers$IteratorWrapper.forEachRemaining(Wrappers.scala:30) at java.base/java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801) at java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:658) at io.openlineage.spark3.agent.lifecycle.plan.column.InputFieldsCollector.collect(InputFieldsCollector.java:64) at io.openlineage.spark3.agent.lifecycle.plan.column.InputFieldsCollector.lambda$collect$0(InputFieldsCollector.java:64) at java.base/java.util.Iterator.forEachRemaining(Iterator.java:133) at scala.collection.convert.Wrappers$IteratorWrapper.forEachRemaining(Wrappers.scala:30) at java.base/java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801) at java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:658) at io.openlineage.spark3.agent.lifecycle.plan.column.InputFieldsCollector.collect(InputFieldsCollector.java:64) at io.openlineage.spark3.agent.lifecycle.plan.column.InputFieldsCollector.lambda$collect$0(InputFieldsCollector.java:64) at java.base/java.util.Iterator.forEachRemaining(Iterator.java:133) at scala.collection.convert.Wrappers$IteratorWrapper.forEachRemaining(Wrappers.scala:30) at java.base/java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801) at java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:658) at io.openlineage.spark3.agent.lifecycle.plan.column.InputFieldsCollector.collect(InputFieldsCollector.java:64) at io.openlineage.spark3.agent.lifecycle.plan.column.InputFieldsCollector.lambda$collect$0(InputFieldsCollector.java:64) at java.base/java.util.Iterator.forEachRemaining(Iterator.java:133) at scala.collection.convert.Wrappers$IteratorWrapper.forEachRemaining(Wrappers.scala:30) at java.base/java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801) at java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:658) at io.openlineage.spark3.agent.lifecycle.plan.column.InputFieldsCollector.collect(InputFieldsCollector.java:64) at io.openlineage.spark3.agent.lifecycle.plan.column.InputFieldsCollector.lambda$collect$0(InputFieldsCollector.java:64) at java.base/java.util.Iterator.forEachRemaining(Iterator.java:133) at scala.collection.convert.Wrappers$IteratorWrapper.forEachRemaining(Wrappers.scala:30) at java.base/java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801) at java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:658) at io.openlineage.spark3.agent.lifecycle.plan.column.InputFieldsCollector.co…</p> +
+ +
+
Labels
+ proposal +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-06-19 09:37:20
 +
+

*Thread Reply:* Hey @Anirudh Shrinivason, me and Paweł are at Berlin Buzzwords right now. Will definitely look at it later

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-06-19 10:47:06
 +
+

*Thread Reply:* Oh nice! Thanks!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ayush mittal + (ayushmittal057@gmail.com) +
+
2023-06-20 03:14:02
 +
+

Hi Team, we are not able to generate lineage for aggregate functions while joining two tables. below is the query +df2 = spark.sql("select th.ProductID as Pid, pd.Name as N, sum(th.quantity) as TotalQuantity, sum(th.ActualCost) as TotalCost from silveradventureworks.transactionhistory as th join productdescription_dim as pd on th.ProductID = pd.ProductID group by th.ProductID, pd.Name ")

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Rahul + (rahul812ry@gmail.com) +
+
2023-06-20 03:47:50
 +
+

*Thread Reply:* This is the event generated for above query.

+ +
+
log4j-active (11).txt
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ayush mittal + (ayushmittal057@gmail.com) +
+
2023-06-20 03:18:22
 +
+

and one more issue, we are not able to generate the open lineage events on top of view being created by joining multiple tables. +i have attached log events for your reference.

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ayush mittal + (ayushmittal057@gmail.com) +
+
2023-06-20 03:31:11
 +
+

this is event for view for which no lineage is being generated

+ +
+
view.txt
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Lukenoff + (john@jlukenoff.com) +
+
2023-06-20 13:59:00
 +
+

Has anyone here successfully implemented the Amundsen OpenLineage extractor? I’m a little confused on the best way to output my lineage events to ndjson files in a scalable way as the docs seem to suggest. Currently I’m pushing all my lineage events to Marquez via REST API. I suppose I could change my transports to Kinesis and write the events to s3 but that comes with the cost of having to build some new way of getting the events to Marquez.

+ +

In any case, this seems like a problem someone must have solved before?

+ +

Edit: looking at the source code for this Amundsen extractor, it seems like it should be pretty straightforward to just implement our own extractor that can pull these records from the Marquez backend. Will give that a shot and see about getting that merged into Amundsen later.

+ + + +
+ 👍 Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-06-20 17:34:08
 +
+

*Thread Reply:* Hi John, glad to hear you figured out a path forward on this! Please let us know what you learn 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-06-20 14:21:03
 +
+

Our New York meetup with Collibra is happening in just two days! https://openlineage.slack.com/archives/C01CK9T7HKR/p1686594956030059

+
+ + +
+ + + } + + Michael Robinson + (https://openlineage.slack.com/team/U02LXF3HUN7) +
+ + + +
+
+ <p>@channel +Meetup announcement: there’s another meetup happening soon! This one will be an evening event on 6/22 in New York at Collibra’s HQ. For details and to sign up, please join the meetup group: <a href="https://www.meetup.com/data-lineage-meetup/events/294065396/">https://www.meetup.com/data-lineage-meetup/events/294065396/</a>. Thanks to @Sheeri Cabral (Collibra) for cohosting and providing a space.</p> +
+ + + + + + +
Original URL: https://openlineage.slack.com/archives/C01CK9T7HKR/p1686594956030059
+ + +
+ + Slack Conversation +
+ + +
+ + + +
+ 👍 Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harshini Devathi + (harshini.devathi@tigeranalytics.com) +
+
2023-06-20 14:31:56
 +
+

Hello all, Do you know if we have th possibility of persisting column orders while creating lineage as it may be available in the table or data set from which it originates. Or, is there some way in which we can get the column order (id or something).

+ +

For example, if a dataset has columns xyz, abc, fgh, dec, I would like to know which column shows first in the dataset in the common data model. Please let me know. m

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-06-20 17:33:36
 +
+

*Thread Reply:* Hi Harshini, I’ve alerted our resident Spark and column-lineage expert about this. Hope to have an answer for you soon.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harshini Devathi + (harshini.devathi@tigeranalytics.com) +
+
2023-06-20 19:39:46
 +
+

*Thread Reply:* Thank you Michael, looking forward to it

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-06-21 02:58:41
 +
+

*Thread Reply:* Hello @Harshini Devathi. An interesting topic which I have never thought about. The ordering of the fields, we get for Spark Apps, comes from Spark logical plans we extract information from and we do not apply any sorting on them. So, if Spark plan contains columns a , b, c we trust it's the order of columns for a dataset and don't want to check it on our own.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-06-21 02:59:45
 +
+

*Thread Reply:* btw. please let us know how do you obtain your lineage: within a Spark app or from some SQL's scheduled by Airflow?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harshini Devathi + (harshini.devathi@tigeranalytics.com) +
+
2023-06-23 14:40:31
 +
+

*Thread Reply:* Hello @Paweł Leszczyński, thank you for the response. We do not need you to check the ordering specifically but I assume that the spark logical plan maintains the column order based on the input datasets. Can we retain that order by adding column id or some sequence number which helps to represent the lineage in the same order.

+ +

The lineage we are capturing using Spark openlineage connector, by posting custom lineage to Marquez through API calls, and also in process of leveraging SQL connector feature using Airflow.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-06-26 04:35:43
 +
+

*Thread Reply:* Hi @Harshini Devathi, are you asking about schema facet within a dataset? This should have an order from spark logical plans. Or, are you asking about columnLineage facet? Or Marquez API responses? It's not clear to me why do you need it. Each column, is identified by a dataset (dataset namespace + dataset name) and field name. You can, on your side, generate and column id based on that and order columns based on the id, but still I think I am missing some arguments behind doing so.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-06-21 17:41:48
 +
+

Attention all Bay-area data friends and Data+AI Summit attendees: our first San Francisco meetup is next Tuesday! https://www.meetup.com/meetup-group-bnfqymxe/events/293448130/

+
+
Meetup
+ + + +
OpenLineage Meetup @ Astronomer, Tue, Jun 27, 2023, 5:30 PM | Meetup
+
+ Data engineers and pipeline managers know that producing data lineage – end-to-end pipeline metadata instrumented at runtime or parsed at design time – is a heavy lift with +
+ + + + + + + + +
Original URL: https://www.meetup.com/meetup-group-bnfqymxe/events/293448130/
+ + + +
+ + + +
+ 🙌 alexandre bergere +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-06-23 16:41:29
 +
+

Last night in New York we held a meetup with Collibra at their lovely HQ in the Financial District! Many thanks to @Sheeri Cabral (Collibra) for inviting us. +Over a bunch of tasty snacks (thanks for the idea @Harel Shein), we discussed: +• the history and evolution of the spec, and trends in adoption +• progress on the OpenLineage Provider in Airflow (AIP 53) +• progress on “static” AKA design lineage support (expected soon in OpenLineage 1.0.0) +• progress in the LFAI program +• a proposal to add “jobless run” support for auditing use cases and similar edge cases +• an idea to throw a hackathon for creating validation tests and example payloads (would you be interested in participating? let us know!) +• and more. +Many thanks to: +• @Julien Le Dem for making the trip +• Sheeri & Collibra for hosting +• everyone for coming, including second-timer @Ernie Ostic and new member @Shirley Lu +It was great meeting/catching up with everyone. Hope to see you and more new faces at the next one!

+ +
+
+ + + + + + +
+ + +
+ 🎉 Harel Shein, Peter Hanssens, Ernie Ostic, Paweł Leszczyński, Maciej Obuchowski, Shirley Lu +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-06-26 10:59:08
 +
+

Our first San Francisco meetup is tomorrow at 5:30 PM at Astronomer’s offices in the Financial District. https://openlineage.slack.com/archives/C01CK9T7HKR/p1687383708927189

+
+ + +
+ + + } + + Michael Robinson + (https://openlineage.slack.com/team/U02LXF3HUN7) +
+ + + +
+
+ <p>Attention all Bay-area data friends and Data+AI Summit attendees: our first San Francisco meetup is next Tuesday! <a href="https://www.meetup.com/meetup-group-bnfqymxe/events/293448130/">https://www.meetup.com/meetup-group-bnfqymxe/events/293448130/</a></p> +
+ + + + + + +
Original URL: https://openlineage.slack.com/archives/C01CK9T7HKR/p1687383708927189
+ + +
+ + Slack Conversation +
+ + +
+ + + +
+ 🚀 alexandre bergere +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Rakesh Jain + (rakeshj@us.ibm.com) +
+
2023-06-27 03:43:10
 +
+

I can’t seem to get OL logging working with Spark. Any guidance please?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-06-27 03:45:31
 +
+

*Thread Reply:* Is it because the logLevel is set to WARN or ERROR?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Rakesh Jain + (rakeshj@us.ibm.com) +
+
2023-06-27 12:07:12
 +
+

*Thread Reply:* No, I set it to INFO, may be I need to add some jars?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-06-27 12:30:02
 +
+

*Thread Reply:* Hmm have you set the relevant spark configs?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Rakesh Jain + (rakeshj@us.ibm.com) +
+
2023-06-27 12:32:50
 +
+

*Thread Reply:* yep, I have http working. But not the console +spark.extraListeners=io.openlineage.spark.agent.OpenLineageSparkListener +spark.openlineage.transport.type=console

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-06-27 12:35:27
 +
+

*Thread Reply:* Oh wait http works but not console...

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-06-27 12:37:02
 +
+

*Thread Reply:* If you want to see the console events which are emitted, then need to set logLevel to DEBUG

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Rakesh Jain + (rakeshj@us.ibm.com) +
+
2023-06-27 12:37:44
 +
+

*Thread Reply:* tried that too, still nothing

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-06-27 12:38:54
 +
+

*Thread Reply:* Is the openlienage jar installed and added to config?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Rakesh Jain + (rakeshj@us.ibm.com) +
+
2023-06-27 12:39:09
 +
+

*Thread Reply:* yep, that’s why http works

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Rakesh Jain + (rakeshj@us.ibm.com) +
+
2023-06-27 12:39:26
 +
+

*Thread Reply:* the only thing I see in the logs is this: +23/06/27 07:39:11 INFO SparkSQLExecutionContext: OpenLineage received Spark event that is configured to be skipped: SparkListenerJobEnd

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-06-27 12:40:59
 +
+

*Thread Reply:* Hmm if an event is still emitted for this case, but logs not showing up then I'm not sure... Maybe someone with more knowledge on this can help

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Rakesh Jain + (rakeshj@us.ibm.com) +
+
2023-06-27 12:42:37
 +
+

*Thread Reply:* sure, thanks for trying @Anirudh Shrinivason

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-06-28 05:23:36
 +
+

*Thread Reply:* What job are you trying this on? If there's this message, then logging is working afaik

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-06-28 12:16:52
 +
+

*Thread Reply:* Hi @Maciej Obuchowski Actually I also noticed a similar issue... For some spark pipelines, the log level is set to debug, but I'm not seeing any events being logged. I am however receiving these events in the backend. Have any of the logging been removed from some places?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Rakesh Jain + (rakeshj@us.ibm.com) +
+
2023-06-28 20:57:45
 +
+

*Thread Reply:* yep, exactly same thing here also @Maciej Obuchowski, I can get the events on http, but changing to console gets me nothing from ConsoleTransport.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Lukenoff + (john@jlukenoff.com) +
+
2023-06-27 20:45:15
 +
+

@here A bunch of us are downstairs in the lobby at 8 California but no one is down here to let us up. Anyone here to help?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-06-29 03:36:36
 +
+

Hi guys, I noticed a few of the jobs getting OOMed while running with openlineage. Even increasing the number of executors and doubling the memory does not seem to fix it actually. This is observed especially when using the graphx libs. Is this a known issue? Just curious as to what the cause might be... The same jobs run fine once openlineage is disabled. Are there some rogue threads from the listener or any connections we aren't closing properly?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-06-29 05:57:59
 +
+

*Thread Reply:* Hi @Anirudh Shrinivason, could you disable serializing spark.logicalPlan to see if the behaviour is the same?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-06-29 05:58:28
 +
+

*Thread Reply:* https://github.com/OpenLineage/OpenLineage/tree/main/integration/spark -> spark.openlineage.facets.disabled -> [spark_unknown;spark.logicalPlan]

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-06-29 05:59:55
 +
+

*Thread Reply:* We do serialize logicalPlan because this is useful in many cases, but sometimes can lead to serializing things that shouldn't be serialized

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-06-29 15:49:35
 +
+

*Thread Reply:* Ahh I see. Yeah okay let me try that

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-06-30 08:01:34
 +
+

Hello all, I’m opening a vote to release OpenLineage 0.29.0, including: +• support for Spark 3.4 +• support for Flink 1.17.1 +• a fix in the Flink integration to enable dataset schema extraction for a KafkaSource when GenericRecord is used +• removal of the unused Golang proxy client (made redundant by the fluentd proxy) +• security vulnerability fixes, doc changes, test improvements, and more. +Three +1s from committers will authorize an immediate release.

+ + + +
+ ➕ Jakub Dardziński, Paweł Leszczyński, Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-06-30 08:05:53
 +
+

*Thread Reply:* Thanks, all. The release is authorized.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-06-30 13:27:35
 +
+

@channel +We released OpenLineage 0.29.2, including: +Added +• Flink: support Flink version 1.17.1 #1947 @pawel-big-lebowski +• Spark: support Spark version 3.4 #1790 @pawel-big-lebowski +Removed +• Proxy: remove unused Golang client approach #1926 @mobuchowski +• Req: bump minimum supported Python version to 3.8 #1950 @mobuchowski + ◦ Note: this removes support for Python 3.7, which is at EOL. +Plus test improvements, docs changes, bug fixes and more. +Thanks to all the contributors! +Release: https://github.com/OpenLineage/OpenLineage/releases/tag/0.29.2 +Changelog: https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md +Commit history: https://github.com/OpenLineage/OpenLineage/compare/0.28.0...0.29.2 +Maven: https://oss.sonatype.org/#nexus-search;quick~openlineage +PyPI: https://pypi.org/project/openlineage-python/

+ + + +
+ 👍 Shirley Lu, Maciej Obuchowski, Paweł Leszczyński, Tamara Fingerlin +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-06-30 17:23:04
 +
+

@channel +The latest issue of OpenLineage News is now available, featuring a recap of recent events, releases, and more. To get it directly in your inbox each month, sign up https://openlineage.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&id=e598962936&e=ef0563a7f8|here.

+
+
apache.us14.list-manage.com
+ + + +
OpenLineage Project
+
+ OpenLineage Project Email Forms +
+ + + + + + +
Original URL: https://openlineage.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&amp;id=e598962936&amp;e=ef0563a7f8
+ + + +
+ +
+
+ + + + + + +
+ + +
+ 👍 Maciej Obuchowski, Paweł Leszczyński, Tristan GUEZENNEC -CROIX-, Tamara Fingerlin, Jeremy W, Anirudh Shrinivason, Julien Le Dem, Sheeri Cabral (Collibra), alexandre bergere +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-07-06 13:36:44
 +
+

@channel +This month’s TSC meeting is next Thursday, 7/13, at a special time: 8 am PT. +All are welcome! +On the tentative agenda: +• announcements +• updates +• recent releases +• a new DataGalaxy integration +• open discussion

+ + + +
+ ✅ Sheeri Cabral (Collibra), Maciej Obuchowski, alexandre bergere, Paweł Leszczyński, Willy Lulciuc, Anirudh Shrinivason, Shirley Lu +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sheeri Cabral (Collibra) + (sheeri.cabral@collibra.com) +
+
2023-07-07 10:35:08
 +
+

Wow, I just got finished watching @Julien Le Dem and @Willy Lulciuc’s presentation of OpenLineage at databricks and it’s really fantastic! There isn’t a better 30 minutes of content on theory + practice than this, IMO. https://www.databricks.com/dataaisummit/session/cross-platform-data-lineage-openlineage/ (you can watch for free by making an account. I’m not affiliated with databricks…)

+
+
databricks.com
+ + + +
Cross-Platform Data Lineage with OpenLineage - Data + AI Summit 2023 | Databricks
+
+ +
+ + + + + + + + +
Original URL: https://www.databricks.com/dataaisummit/session/cross-platform-data-lineage-openlineage/
+ + + +
+ + + +
+ ❤️ Willy Lulciuc, Harel Shein, Yuanli Wang, Ross Turk, Michael Robinson, Jakub Dardziński, Conor Beverland, Maciej Obuchowski, Jarek Potiuk, Julien Le Dem, Chris Folkes, Anirudh Shrinivason, Shirley Lu +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2023-07-07 10:37:49
 +
+

*Thread Reply:* thanks for watching and sharing! the recording is also on youtube 😉 https://www.youtube.com/watch?v=rO3BPqUtWrI

+
+
YouTube
+ +
+ + + } + + Databricks + (https://www.youtube.com/@Databricks) +
+ + + +
Cross-Platform Data Lineage with OpenLineage
+
+ +
+ + + + + + + + +
Original URL: https://www.youtube.com/watch?v=rO3BPqUtWrI
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sheeri Cabral (Collibra) + (sheeri.cabral@collibra.com) +
+
2023-07-07 10:38:01
 +
+

*Thread Reply:* ❤️

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jarek Potiuk + (jarek@potiuk.com) +
+
2023-07-08 13:35:10
 +
+

*Thread Reply:* Very much agree. I’ve even forwarded to a few people here and there, those who I think should learn about it.

+ + + +
+ ❤️ Sheeri Cabral (Collibra) +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2023-07-08 13:47:17
 +
+

*Thread Reply:* You’re both too kind :) +Thank you for your support and being part of the community.

+ + + +
+ ❤️ Sheeri Cabral (Collibra), Jarek Potiuk +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-07-07 15:44:33
 +
+

@channel +If you registered for TSC meetings through AddEvent, first of all, thank you! Second of all, I’ve had to create a new event series there to enable the editing of individual events. When you have a moment, would you please register for next week’s meeting? Apologies for the inconvenience.

+
+
addevent.com
+ + + +
OpenLineage Monthly TSC Meeting
+
+ This is the monthly meeting for the OpenLineage maintainers.Agenda:The agenda is updated the week prior to the meetingArchive:   <https://wiki.lfaidata.foundation/display/OpenLineage/Monthly+TSC+meeting> - How to be part of the discussion:Slack: <https://bit.ly/lineageslackMailing> list: <https://lists.lfaidata.foundation/g/openlineage-tscConfluence>: <https://wiki.lfaidata.foundation/display/OpenLineage/OpenLineage+Home> +
+ + + + + + + + +
Original URL: https://evt.to/ahdseuidw
+ + + +
+ + + +
+ 👍 Kiran Hiremath, Willy Lulciuc, Shirley Lu +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Juan Manuel Cappi + (juancappi@gmail.com) +
+
2023-07-10 12:29:31
 +
+

Hi community, we are interested in capturing time-travel usage for Iceberg Spark sql in column lineage. For instance, INSERT INTO schema.table select ** from schema.another_table version as of 'some_version' . Column lineage is currently missing the version, if used, which it’s actually quite relevant. I’ve gone through the open issues and didn’t see anything similar. Does it look like a valid use case scenario? We started going through the OL, iceberg and Spark code in trying to capture/expose it, but so far we haven’t been able to. If anyone can give a hint/idea/pointer, we are willing to give it try a contribute back with the code

+ + + +
+ 👀 Rakesh Jain, Nitin Ramchandani +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2023-07-11 05:46:36
 +
+

*Thread Reply:* I think yes this is a great use case. @Paweł Leszczyński is more familiar with the spark integration code than I. +I think in this case, we would add the datasetVersion facet with the underlying Iceberg version: https://github.com/OpenLineage/OpenLineage/blob/main/spec/facets/DatasetVersionDatasetFacet.json +We extract this information in a few places: +https://github.com/search?q=repo%3AOpenLineage%2FOpenLineage%20%20DatasetVersionDatasetFacet&type=code

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/spec/facets/DatasetVersionDatasetFacet.json | DatasetVersionDatasetFacet.json>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-07-11 05:57:17
 +
+

*Thread Reply:* Yes, we do have datasetVersion which captures for output and input datasets their iceberg or delta version. Input versions are collected on START while output are collected on COMPLETE in case a job reads and writes to the same dataset. So, even though column-lineage facet is missing the version, it should be available within events related to a particular run.

+ +

If it is not, then perhaps the case here is the lack of support of as of syntax. As far as I remeber, we always get a current version of a dataset and this may be a missing part here.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-07-11 05:58:49
 +
+

*Thread Reply:* link to a method that gets dataset version for iceberg: https://github.com/OpenLineage/OpenLineage/blob/0.29.2/integration/spark/spark3/sr[…]lineage/spark3/agent/lifecycle/plan/catalog/IcebergHandler.java

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/0.29.2/integration/spark/spark3/src/main/java/io/openlineage/spark3/agent/lifecycle/plan/catalog/IcebergHandler.java | IcebergHandler.java>
+
+ <pre><code> public Optional&lt;String&gt; getDatasetVersion( +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Juan Manuel Cappi + (juancappi@gmail.com) +
+
2023-07-11 10:57:26
 +
+

*Thread Reply:* Thank you @Julien Le Dem and @Paweł Leszczyński +Based on what I’ve seen so far, indeed it seems that only the current snapshot is tracked. When IcebergHandler.getDatasetVersion() +Initially I was expecting to be able to obtain the snapshotId from the SparkTable which comes within getDatasetVersion() but now I realize that OL is using an older version of Iceberg runtime, (0.12.1) which does not support time travel (introduced in 0.14.1). +The evidence is: +• Iceberg documentation for release 0.14.1: https://iceberg.apache.org/docs/0.14.0/spark-queries/#sql +• Iceberg release notes https://iceberg.apache.org/releases/#0140-release +• Comparing the source code, I see the SparkTable from 0.14.1 onward does have a snapshotId instance variable, while previous versions don’t +https://github.com/apache/iceberg/blob/0.14.x/spark/v3.0/spark/src/main/java/org/apache/iceberg/spark/source/SparkTable.java#L82 +https://github.com/apache/iceberg/blob/0.12.x/spark3/src/main/java/org/apache/iceberg/spark/source/SparkTable.java#L78

+ +

I don’t see anyone complaining about the old version of Iceberg runtime being used and there is no open issue to upgrade so I’ll open the issue and please let me know if that seems reasonable as the immediate next step to take

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Juan Manuel Cappi + (juancappi@gmail.com) +
+
2023-07-11 15:48:53
 +
+

*Thread Reply:* Created issues: #1969 and #1970

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-07-12 07:15:14
 +
+

*Thread Reply:* Thanks @Juan Manuel Cappi. openlineage-spark jar contains modules like spark3 , spark32 , spark33 and spark34 that is going to be merged soon (we do have a ready PR for that). spark34 will be compiled against latest iceberg version. Once this is done #1969 can be closed. For 1970, one would need to implement datasetBuilder within spark34 module and visits node within spark's logical plan that is responsible for as of and creates dataset for OpenLineage event other way than getting latest snapshot version.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Juan Manuel Cappi + (juancappi@gmail.com) +
+
2023-07-13 12:51:19
 +
+

*Thread Reply:* @Paweł Leszczyński I’ve see PR #1971 and I see a new spark34 project with the latest iceberg-spark dependency version, but other versions (spark33, spark32, etc) have not being upgraded in that PR. Since the change is small and does not break any tests, I’ve created PR #1976 for to fix #1969. That alone is unlocking some time travel lineage (i.e. dataset identifier now becomes schema.table.version or schema.table.snapshot_id). Hope it makes sense

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-07-14 04:37:55
 +
+

*Thread Reply:* Hi @Juan Manuel Cappi, You're right and after discussion with you I realized we support some version of iceberg (for spark 3.3 it's still 0.14.0) but this is not the latest iceberg version matching spark version.

+ +

There's some tricky part here. Although we wan't our code to succeed with latest spark, we don't want it to fail in a nasty way (class not found exception) when a user is working with an old iceberg version. There are places in our code where we do check are iceberg classes on the classpath? We need to extend this to are iceberg classes on classpath is iceberg version above 0.14 or not For sure this is the case for merge into commands I am working on at the moment. Let's see if the other integration tests are affected in your PR

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Amod Bhalerao + (amod.bhalerao@gmail.com) +
+
2023-07-11 08:09:57
 +
+

HI Team, I Seen that Kafka lineage is not coming properly in for Spark streaming, Are we working on this?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-07-11 08:28:59
 +
+

*Thread Reply:* what do you mean by that? there is a pyspark & kafka integration test that verifies event being sent when reading or writing to kafka topic: https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/app/src/tes[…]a/io/openlineage/spark/agent/SparkContainerIntegrationTest.java

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/app/src/test/java/io/openlineage/spark/agent/SparkContainerIntegrationTest.java | SparkContainerIntegrationTest.java>
+
+ <pre><code> void testPysparkKafkaReadWrite() { +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-07-11 09:28:56
 +
+

*Thread Reply:* We do have an old issue https://github.com/OpenLineage/OpenLineage/issues/372 to support more spark plans that are stream related. But, if you had an example of streaming that is not working for you, this would have been really helpful.

+
+ + + + +
#372 spark: Cover streaming LogicalPlans
+
+ <p>Cover implementations of LogicalPlan in <br /> +org.apache.spark.sql.execution.streaming.continuous <br /> +WriteToContinuousDataSource <br /> +org.apache.spark.sql.execution.streaming.sources <br /> +MemoryPlan <br /> +WriteToMicroBatchDataSource</p> +
+ +
+
Labels
+ integration/spark, streaming +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Amod Bhalerao + (amod.bhalerao@gmail.com) +
+
2023-07-26 08:03:30
 +
+

*Thread Reply:* I have a pipeline Which reads from topic and send data to 3 HIVE tables and one postgres , Its not emitting any lineage for this pipeline

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Amod Bhalerao + (amod.bhalerao@gmail.com) +
+
2023-07-26 08:06:51
 +
+

*Thread Reply:* just one task is getting created

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-07-12 05:55:19
 +
+

Hi guys, I notice that with the below spark configs: +```from pyspark.sql import SparkSession +import os

+ +

os.environ["TEST_VAR"] = "1"

+ +

spark = (SparkSession.builder.master('local') + .appName('samplespark') + .config('spark.jars.packages', 'io.openlineage:openlineagespark:0.29.2,io.delta:deltacore2.12:1.0.1') + .config('spark.extraListeners', 'io.openlineage.spark.agent.OpenLineageSparkListener') + .config('spark.openlineage.transport.type', 'console') + .config('spark.sql.catalog.sparkcatalog', "org.apache.spark.sql.delta.catalog.DeltaCatalog") + .config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension") + .config("hive.metastore.schema.verification", False) + .config("spark.sql.warehouse.dir","/tmp/") + .config("hive.metastore.warehouse.dir","/tmp/") + .config("javax.jdo.option.ConnectionURL","jdbc:derby:;databaseName=/tmp/metastoredb;create=true") + .config("spark.openlineage.facets.customenvironmentvariables","[TESTVAR;]") + .config("spark.openlineage.facets.disabled","[sparkunknown;spark.logicalPlan]") + .config("spark.hadoop.fs.permissions.unmask-mode","000") + .enableHiveSupport() + .getOrCreate())``` +The custom environment variables facet is not kicking in. However, when all the delta related spark configs are removed, it is working fine. Is this a known issue? Are there any workarounds for it? Thanks!

+ + + +
+ 👀 Paweł Leszczyński +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Juan Manuel Cappi + (juancappi@gmail.com) +
+
2023-07-12 06:14:41
 +
+

*Thread Reply:* Hi @Anirudh Shrinivason, I’m not familiar with Delta, but enabling debugging helped me a lot to understand what’s going when things fail silently. Just add at the end: +spark.sparkContext.setLogLevel("DEBUG")

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-07-12 06:20:47
 +
+

*Thread Reply:* Yeah I checked on debug

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-07-12 06:20:50
 +
+

*Thread Reply:* There are no errors

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-07-12 06:21:10
 +
+

*Thread Reply:* Just that there is no environment-properties in the event that is being emitted

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-07-12 07:31:01
 +
+

*Thread Reply:* Hi @Anirudh Shrinivason, what spark version is that? i see you delta version is pretty old. Anyway, the observation is weird and don't know how come delta interferes with environment facet builder. These are so disjoint features. Are you sure you create a new session (there is getOrCreate) ?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Glen M + (glen_m@apple.com) +
+
2023-07-12 19:29:06
 +
+

*Thread Reply:* @Paweł Leszczyński its because of this line : https://github.com/OpenLineage/OpenLineage/blob/0.29.2/integration/spark/app/src/m[…]nlineage/spark/agent/lifecycle/InternalEventHandlerFactory.java

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/0.29.2/integration/spark/app/src/main/java/io/openlineage/spark/agent/lifecycle/InternalEventHandlerFactory.java | InternalEventHandlerFactory.java>
+
+ <pre><code> } else if (context.getCustomEnvironmentVariables() != null) { +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Glen M + (glen_m@apple.com) +
+
2023-07-12 19:32:44
 +
+

*Thread Reply:* Assuming this is https://learn.microsoft.com/en-us/azure/databricks/delta/ ... delta .. which is azure datbricks. @Anirudh Shrinivason

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-07-12 22:58:13
 +
+

*Thread Reply:* Hmm I wasn't using databricks

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-07-12 22:59:12
 +
+

*Thread Reply:* @Paweł Leszczyński I'm using spark 3.1 btw

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-07-13 08:05:49
 +
+

*Thread Reply:* @Anirudh Shrinivason This should resolve the issue https://github.com/OpenLineage/OpenLineage/pull/1973

+
+ + + + +
#1973 [SPARK] fix custom environment variables facet
+
+ <p><strong>Problem</strong></p> + +<p>Custom environment facet behaviour is non-deterministic: it's present in some case and it's not available when working on delta tables in Spark (<a href="https://openlineage.slack.com/archives/C01CK9T7HKR/p1689155719105569">https://openlineage.slack.com/archives/C01CK9T7HKR/p1689155719105569</a>). This is weird as the facet and delta format have nothin in common. However they do, as <code>CustomEnvironmentFacetBuilder</code> has been implemented to be triggered <code>SparkListenerJobStart</code>. We do have two types of start events: <code>SparkListenerJobStart</code> and <code>SparkListenerSQLExecutionStart</code> and Openlineage event is only sent for one of them. It seems as other event type got triggered with delta tables.</p> + +<p><strong>Solution</strong></p> + +<p>• Prepare an integration test to verify facet is there. +• Fix <code>CustomEnvironmentFacetBuilder</code> class.</p> + +<blockquote> + <p><strong>Note:</strong> All schema changes require discussion. Please <a href="https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue">link the issue</a> for context.</p> +</blockquote> + +<p>☐ Your change modifies the <a href="https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.json">core</a> OpenLineage model +☐ Your change modifies one or more OpenLineage <a href="https://github.com/OpenLineage/OpenLineage/tree/main/spec/facets">facets</a></p> + +<p>If you're contributing a new integration, please specify the scope of the integration and how/where it has been tested (e.g., Apache Spark integration supports <code>S3</code> and <code>GCS</code> filesystem operations, tested with AWS EMR).</p> + +<p><strong>One-line summary:</strong> +<strong>Checklist</strong></p> + +<p>☐ You've <a href="https://github.com/OpenLineage/OpenLineage/blob/main/why-the-dco.md">signed-off</a> your work +☐ Your pull request title follows our <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CONTRIBUTING.md#creating-pull-requests">guidelines</a> +☐ Your changes are accompanied by tests (<em>if relevant</em>) +☐ Your change contains a <a href="https://kurtisnusbaum.medium.com/stacked-diffs-keeping-phabricator-diffs-small-d9964f4dcfa6">small diff</a> and is self-contained +☐ You've updated any relevant documentation (<em>if relevant</em>) +☐ Your comment includes a one-liner for the changelog about the specific purpose of the change (<em>if necessary</em>) +☐ You've versioned the core OpenLineage model or facets according to <a href="https://docs.snowplowanalytics.com/docs/pipeline-components-and-applications/iglu/common-architecture/schemaver">SchemaVer</a> (<em>if relevant</em>) +☐ You've added a <a href="https://github.com/OpenLineage/OpenLineage/tree/main/.github/header_templates.md">header</a> to source files (<em>if relevant</em>)</p> + +<hr /> + +<p>SPDX-License-Identifier: Apache-2.0 <br /> +Copyright 2018-2023 contributors to the OpenLineage project</p> +
+ +
+
Labels
+ documentation, integration/spark +
+ +
+
Comments
+ 1 +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-07-13 08:06:11
 +
+

*Thread Reply:* PR description contains info on how come the observed behaviour was possible

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-07-13 08:07:47
 +
+

*Thread Reply:* As always, thank you @Anirudh Shrinivason for providing clear information on how to reproduce the issue 🚀 :medal: 👍

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-07-13 09:52:29
 +
+

*Thread Reply:* Ohh that is really great! Thankss so much for the help! 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-07-12 13:50:51
 +
+

@channel +A friendly reminder: this month’s TSC meeting — open to all — is tomorrow at 8 am PT. https://openlineage.slack.com/archives/C01CK9T7HKR/p1688665004736219

+
+ + +
+ + + } + + Michael Robinson + (https://openlineage.slack.com/team/U02LXF3HUN7) +
+ + + +
+
+ <p>@channel +This month’s TSC meeting is next Thursday, 7/13, at a special time: <a href="https://astronomer.zoom.us/j/87156607114?pwd=a3B0K210dnRaQmdkaFdGMytBREZEQT09">8 am PT</a>. +All are welcome! +On the tentative agenda: +• announcements +• updates +• recent releases +• a new DataGalaxy integration +• open discussion</p> +
+ + + + + + +
Original URL: https://openlineage.slack.com/archives/C01CK9T7HKR/p1688665004736219
+ + +
+ + Slack Conversation +
+ + +
+ + + +
+ 👍 Dongjin Seo +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
thebruuu + (bruno.c@inwind.it) +
+
2023-07-12 14:54:29
 +
+

Hi Team +How are you ? +Is there any chance to use airflow to run queries against Access file? +Sorry to bother with a question that is not directly related to openlineage ... but I am kind of stuck

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harel Shein + (harel.shein@gmail.com) +
+
2023-07-12 15:22:52
 +
+

*Thread Reply:* what do you mean by Access file?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
thebruuu + (bruno.c@inwind.it) +
+
2023-07-12 16:09:03
 +
+

*Thread Reply:* ... accdb file, Microsoft Access File: I am in a reverse engineering projet facing a spaghetti style development and would have loved to use, airflow and openlineage as a magic wand, to help me in this damn work

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harel Shein + (harel.shein@gmail.com) +
+
2023-07-12 21:44:21
 +
+

*Thread Reply:* oof.. I’d look into https://airflow.apache.org/docs/apache-airflow-providers-odbc/4.0.0/ +but I really have no clue..

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
thebruuu + (bruno.c@inwind.it) +
+
2023-07-13 09:47:02
 +
+

*Thread Reply:* Thank you Harel +I started from that too ... but it became foggy after the initial step

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Aaman Lamba + (aamanlamba@gmail.com) +
+
2023-07-12 16:30:41
 +
+

Hi folks, having an issue ingesting the seed metadata when starting the docker container. The output shows "seed-marquez-with-metadata exited with code 0" but no information is visible in marquez What can be the issue?

+ + + +
+ ✅ Aaman Lamba +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-07-12 16:55:00
 +
+

*Thread Reply:* Did you check the namespace menu in the top right for a food_delivery namespace?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-07-12 16:55:12
 +
+

*Thread Reply:* (Hi Aaman!)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Aaman Lamba + (aamanlamba@gmail.com) +
+
2023-07-12 16:55:45
 +
+

*Thread Reply:* Hi! Thank you that helped!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Aaman Lamba + (aamanlamba@gmail.com) +
+
2023-07-12 16:55:55
 +
+

*Thread Reply:* I think that should be added to the quickstart guide

+ + + +
+ 🙌 Michael Robinson +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-07-12 16:56:23
 +
+

*Thread Reply:* Great idea, thank you

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2023-07-13 12:09:29
 +
+

As discussed in the Monthly meeting, I have opened a PR to propose adding deletion to facets for static lineage metadata: https://github.com/OpenLineage/OpenLineage/pull/1975

+
+ + + + +
#1975 Add facet deletion
+
+ <p><strong>Problem</strong></p> + +<p>The <a href="https://github.com/OpenLineage/OpenLineage/blob/main/proposals/1837/static_lineage.md#semantics-of-facets-across-jobs-and-dataset-version">last section of the static lineage proposal</a> defines semantics of static facets attached to a job or dataset.</p> + +<p>“job and dataset facets apply to that entity from that moment on until they are replaced by a new version of the same facet”</p> + +<p>However this doesn’t allow deleting the facet altogether if the facet is no longer applicable. <br /> +For example: the dataset no longer has an owner.</p> + +<p>This proposal adds a mechanism for deleting job and dataset facets.</p> + +<p><strong>Solution</strong></p> + +<p>We propose to add a { _deleted: true } object that can take the place of any job or dataset facet (but not run or input/output facets which are valid only for a specific run)</p> + +<p>☑︎ Your change modifies the <a href="https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.json">core</a> OpenLineage model</p> + +<p><strong>One-line summary: add a mechanism for deleting job and dataset facets.</strong> +<strong>Checklist</strong></p> + +<p>☐ You've <a href="https://github.com/OpenLineage/OpenLineage/blob/main/why-the-dco.md">signed-off</a> your work +☐ Your pull request title follows our <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CONTRIBUTING.md#creating-pull-requests">guidelines</a> +☐ Your changes are accompanied by tests (<em>if relevant</em>) +☐ Your change contains a <a href="https://kurtisnusbaum.medium.com/stacked-diffs-keeping-phabricator-diffs-small-d9964f4dcfa6">small diff</a> and is self-contained +☐ You've updated any relevant documentation (<em>if relevant</em>) +☐ Your comment includes a one-liner for the changelog about the specific purpose of the change (<em>if necessary</em>) +☐ You've versioned the core OpenLineage model or facets according to <a href="https://docs.snowplowanalytics.com/docs/pipeline-components-and-applications/iglu/common-architecture/schemaver">SchemaVer</a> (<em>if relevant</em>) +☐ You've added a <a href="https://github.com/OpenLineage/OpenLineage/tree/main/.github/header_templates.md">header</a> to source files (<em>if relevant</em>)</p> + +<hr /> + +<p>SPDX-License-Identifier: Apache-2.0 <br /> +Copyright 2018-2023 contributors to the OpenLineage project</p> +
+ +
+
Labels
+ documentation, proposal +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Steven + (xli@zjuici.com) +
+
2023-07-13 23:21:29
 +
+

Hi, I'm using OL python client. +client.emit( + DatasetEvent( + _eventTime_=datetime.now().isoformat(), + _producer_=producer, + _schemaURL_="<https://openlineage.io/spec/1-0-5/OpenLineage.json#/definitions/DatasetEvent>", + _dataset_=Dataset(_namespace_=namespace, _name_=f"input-file"), + ) + ) +I want to send a dataset event once files been uploaded. But I received 422 from api/v1/lineage, saying that run and job must not be null. I don't have a job or run yet. How can I solve this?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-07-14 04:09:15
 +
+

*Thread Reply:* Hi @Steven, I assume you send your Openlineage events to Marquez. 422 http code is a response from backend and Marquez is still waiting for the PR https://github.com/MarquezProject/marquez/pull/2495 to be merged and released. This PR makes Marquez understand DatasetEvents. They won't be saved in Marquez database (this is to be implemented in future), but at least one will not experience error response code.

+ +

To sum up: what you do is correct. You are using a feature that is allowed on a client side but still not implemented on a backend.

+
+ + + + +
#2495 Ability to decode static metadata events
+
+ <p><strong>Problem</strong></p> + +<p>Static metadata events (known also as runless) where presented within in the proposal: <br /> +<a href="https://github.com/OpenLineage/OpenLineage/blob/main/proposals/1837/static_lineage.md">https://github.com/OpenLineage/OpenLineage/blob/main/proposals/1837/static_lineage.md</a></p> + +<p>and is being introduced to Openlineage specification within the PR: <a href="https://github.com/OpenLineage/OpenLineage/pull/1880">OpenLineage/OpenLineage#1880</a></p> + +<p>Of course, we would like to implement it within Marquez. We would like to achieve this ultimate goal gradually and this PR contains the first step.</p> + +<p>The changes introduced within the PR:</p> + +<p>• Don't affect the way <code>RunEvent</code> events are being collected. Events are saved into database and http status code <code>201 (created)</code> is returned. +• <code>DatasetEvent</code> and <code>JobEvent</code> are distinguished by <code>schemaURL</code> property. If distinction cannot be made, events are treated the default way, as <code>RunEvent</code>. +• <code>DatasetEvent</code> and <code>JobEvent</code>events are deserialised by standard Jackso library <code>DatasetEvent</code>/<code>JobEvent</code> classes. The response returns in such cases <code>200 OK</code> http status code but <strong>DOES NOT SAVE</strong> data to database (this will be introduced later). The great benefit of that is that it is a good integration test and a demo of of how to do events' distinction on the server side.</p> + +<p>Relates to: <a href="https://github.com/MarquezProject/marquez/issues/1868">#1868</a></p> + +<p><strong>Solution</strong></p> + +<p>• EventTypeResolver is introduced which decodes incoming <code>/create</code> requests to <code>LineageEvent</code>, <code>DatasetEvent</code> or <code>JobEvent</code> based on the <code>schemaURL</code> field. +• Nothing changes for <code>RunEvents</code>. +• In case of <code>DatasetEvent</code> or <code>JobEvent</code>, they get deserialised and distinguished, however no operation on database is performed.</p> + +<p>One-line summary:</p> + +<p><strong>Checklist</strong></p> + +<p>☑︎ You've <a href="https://github.com/MarquezProject/marquez/blob/main/CONTRIBUTING.md#sign-your-work">signed-off</a> your work +☑︎ Your changes are accompanied by tests (<em>if relevant</em>) +☐ Your change contains a <a href="https://kurtisnusbaum.medium.com/stacked-diffs-keeping-phabricator-diffs-small-d9964f4dcfa6">small diff</a> and is self-contained +☐ You've updated any relevant documentation (<em>if relevant</em>) +☐ You've included a one-line summary of your change for the <a href="https://github.com/MarquezProject/marquez/blob/main/CHANGELOG.md#unreleased"><code>CHANGELOG.md</code></a> (<em>Depending on the change, this may not be necessary</em>). +☐ You've versioned your <code>.sql</code> database schema migration according to <a href="https://flywaydb.org/documentation/concepts/migrations#naming">Flyway's naming convention</a> (<em>if relevant</em>) +☐ You've included a <a href="https://github.com/MarquezProject/marquez/blob/main/CONTRIBUTING.md#copyright--license">header</a> in any source code files (<em>if relevant</em>)</p> +
+ +
+
Labels
+ docs, api, client/java +
+ +
+
Comments
+ 1 +
+ + + + + + + +
+ + <a href="https://github.com/MarquezProject/marquez">MarquezProject/marquez</a> +
+ + +
+ + + +
+ ✅ Steven +
+ +
+ 🥳 Steven +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Steven + (xli@zjuici.com) +
+
2023-07-14 04:10:30
 +
+

*Thread Reply:* Thanks!!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harshit Soni + (harshit.soni@angelbroking.com) +
+
2023-07-14 08:36:23
 +
+

@here Hi Team, I am trying to run a spark application with openLineage +Spark :- 3.3.3 +Openlineage :- 0.29.2 +I am getting below error can you please me, what I could be doing wrong.

+ +

``` spark = (SparkSession + .builder + .config('spark.port.maxRetries', 100) + .appName(app_name) + .config("spark.openlineage.url","http://localhost/api/v1/namespaces/spark_integration/") + .config("spark.extraListeners","io.openlineage.spark.agent.OpenLineageSparkListener") + .getOrCreate())

+ +

23/07/14 18:04:01 ERROR Utils: uncaught error in thread spark-listener-group-shared, stopping SparkContext +java.lang.UnsatisfiedLinkError: /private/var/folders/z6/pl8p30z11v50zf6pv51p259m0000gp/T/native-lib4983292552717270883/libopenlineagesqljava.dylib: dlopen(/private/var/folders/z6/pl8p30z11v50zf6pv51p259m0000gp/T/native-lib4983292552717270883/libopenlineagesqljava.dylib, 0x0001): tried: '/private/var/folders/z6/pl8p30z11v50zf6pv51p259m0000gp/T/native-lib4983292552717270883/libopenlineagesqljava.dylib' (mach-o file, but is an incompatible architecture (have 'x8664', need 'arm64')), '/System/Volumes/Preboot/Cryptexes/OS/private/var/folders/z6/pl8p30z11v50zf6pv51p259m0000gp/T/native-lib4983292552717270883/libopenlineagesqljava.dylib' (no such file), '/private/var/folders/z6/pl8p30z11v50zf6pv51p259m0000gp/T/native-lib4983292552717270883/libopenlineagesqljava.dylib' (mach-o file, but is an incompatible architecture (have 'x8664', need 'arm64')) + at java.lang.ClassLoader$NativeLibrary.load(Native Method)```

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-07-18 02:35:18
 +
+

*Thread Reply:* Hi @Harshit Soni, where are you deploying your spark? locally or not? is it on mac? Calling @Maciej Obuchowski to help with ibopenlineage_sql_java architecture compilation issue

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harshit Soni + (harshit.soni@angelbroking.com) +
+
2023-07-18 02:38:03
 +
+

*Thread Reply:* Currently, was testing on local.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harshit Soni + (harshit.soni@angelbroking.com) +
+
2023-07-18 02:39:43
 +
+

*Thread Reply:* We have created a centralised utility for all data ingestion needs and want to see how lineage is created for same using Openlineage.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-07-18 05:16:55
 +
+

*Thread Reply:* 👀

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-07-14 13:00:29
 +
+

@channel +If you missed this month’s TSC meeting, the recording is now available on our YouTube channel: https://youtu.be/2vD6-Uwr7ZE. +A clip of Alexandre Bergere’s DataGalaxy integration demo is also available: https://youtu.be/l_HbEtpXphY.

+
+
YouTube
+ +
+ + + } + + OpenLineage Project + (https://www.youtube.com/@openlineageproject6897) +
+ + + +
OpenLineage Community Meeting | July 14, 2023
+
+ +
+ + + + + + + + +
Original URL: https://youtu.be/2vD6-Uwr7ZE
+ + + +
+
+
YouTube
+ +
+ + + } + + OpenLineage Project + (https://www.youtube.com/@openlineageproject6897) +
+ + + +
DataGalaxy Integration Demo | July 13, 2023
+
+ +
+ + + + + + + + +
Original URL: https://youtu.be/l_HbEtpXphY
+ + + +
+ + + +
+ 👍 Kiran Hiremath, alexandre bergere, Harel Shein, Paweł Leszczyński +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Robin Fehr + (robin.fehr@acosom.com) +
+
2023-07-16 17:39:26
 +
+

Hey guys - trying to get a grip on the ecosystem regarding flink lineage 🙂 as far as my research has revealed, the openlineage project is the only one that supports flink lineage with an out of the box library that can be integrated in jobs. at least as far as i've seen the for other toolings such as datahub we'd have to write our custom hooks that implement their api. as for my question - is my current assumption correct that an integration into the openlineage project of for example datahub/openmetadata would also require support from datahub/openmetadata itself so that they can work with the openlineage spec? or would it somewhat work to write a mapper in between to support their spec? (more of an architectural decision i assume but would be interested in knowing what the openlinage's approach is regarding that)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-07-17 08:13:49
 +
+

*Thread Reply:* > or would it somewhat work to write a mapper in between to support their spec? +I think yeah - maybe https://github.com/Natural-Intelligence/openLineage-openMetadata-transporter would work out of the box if I understand correctly?

+
+ + + + +
Natural-Intelligence/openLineage-openMetadata-transporter
+
+ Transporter for integrating OpenLineage with OpenMetadata +
+ +
+
Website
+ <https://www.top10.com> +
+ +
+
Language
+ Java +
+ + + + + + + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harel Shein + (harel.shein@gmail.com) +
+
2023-07-17 08:38:59
 +
+

*Thread Reply:* Tagging @Natalie Zeller in case you want to collaborate

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Natalie Zeller + (natalie.zeller@naturalint.com) +
+
2023-07-17 08:47:34
 +
+

*Thread Reply:* Hi, +We've implemented a transporter that transmits lineage from OpenLineage to OpenMetadata, you can find the github project here. +I've also published a blog post that explains this integration and how to use it. +I'll be happy to help if you have any question

+
+ + + + +
Natural-Intelligence/openLineage-openMetadata-transporter
+
+ Transporter for integrating OpenLineage with OpenMetadata +
+ +
+
Website
+ <https://www.top10.com> +
+ +
+
Language
+ Java +
+ + + + + + + + +
+ + + +
+ 🙌 Robin Fehr +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Robin Fehr + (robin.fehr@acosom.com) +
+
2023-07-17 09:49:30
 +
+

*Thread Reply:* very cool! thanks a lot for responding so quickly

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-07-17 18:23:53
 +
+

🚀 We recently hit the 1000-member mark on here! Thank you for joining the movement to establish an open standard for data lineage across the data ecosystem! Tell your friends 🙂! +💯💯💯💯💯💯💯💯💯💯 +https://bit.ly/lineageslack

+ + + +
+ 🎉 Juan Manuel Cappi, Harel Shein, Paweł Leszczyński, Maciej Obuchowski, Willy Lulciuc, Viraj Parekh +
+ +
+ 💯 Harel Shein, Anirudh Shrinivason, Paweł Leszczyński, Maciej Obuchowski, Willy Lulciuc, Robin Fehr, Viraj Parekh, Ernie Ostic +
+ +
+ 👏 thebruuu +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-07-18 04:58:14
 +
+

Btw, just curious what exactly does the runId correspond to in the OL spark integration? Is it possible to obtain the spark application id from the event too?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-07-18 05:10:31
 +
+

*Thread Reply:* runId is an UUID assigned per spark action (compute trigger within a spark job). A single spark script can result in multiple runs then

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-07-18 05:13:17
 +
+

*Thread Reply:* adding an extra facet with applicationId looks like a good idea to me: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/SparkContext.html#applicationId:String

+
+
spark.apache.org
+ + + +
Spark 3.4.1 ScalaDoc - org.apache.spark.SparkContext
+
+ Spark 3.4.1 ScalaDoc - org.apache.spark.SparkContext +
+ + + + + + +
Original URL: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/SparkContext.html#applicationId:String
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-07-18 23:06:01
 +
+

*Thread Reply:* Got it thanks!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Madhav Kakumani + (madhav.kakumani@6point6.co.uk) +
+
2023-07-18 09:47:47
 +
+

Hi, I have an usecase to integrate queries run in Jupyter notebook using pandas integrate with OpenLineage to get the Lineage in Marquez. Did anyone implemented this before? please let me know. Thanks

+ + + +
+ 🤩 thebruuu +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-07-20 06:48:54
 +
+

*Thread Reply:* I think we don't have pandas support so far. So, if one uses pandas to read local files on disk, then perhaps Openlineage (OL) has little sense to do. There is an old pandas issues in our backlog (over 2 years old) -> https://github.com/OpenLineage/OpenLineage/issues/108

+ +

Surely one can use use python OL client to create manully events and send them to MQZ, which may be less convenient (https://github.com/OpenLineage/OpenLineage/tree/main/client/python)

+ +

Anyway, we would like to know what's you usecase? this would be super helpful in understanding why OL & pandas integration may be useful.

+
+ + + + +
#108 [INTEGRATION] Add support for `pandas.read_sql`
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Madhav Kakumani + (madhav.kakumani@6point6.co.uk) +
+
2023-07-20 06:52:32
 +
+

*Thread Reply:* Thanks Pawel for responding

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-07-19 02:57:57
 +
+

Hi guys, when can we expect the next Openlineage release? Excited for MergeIntoCommand column lineage feature!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-07-19 03:40:20
 +
+

*Thread Reply:* Hi @Anirudh Shrinivason, I am still working on that. It's kind of complex because I want to refactor column level lineage so that it can work with multiple Spark versions and multiple delta jars as merge into implementation for delta differs for different delta releases. I thought it's ready, but this needs some extra work to be done in next days. I am excited about that too!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-07-19 03:54:37
 +
+

*Thread Reply:* Ahh I see... Got it! Is there a tentative timeline for when we can expect this? So sorry haha don't mean to rush you. Just curious to know thats all! 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-07-19 22:06:10
 +
+

*Thread Reply:* Can we author a release sometime soon? Would like to use the CustomEnvironmentFacetBuilder for delta catalog!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-07-20 05:28:43
 +
+

*Thread Reply:* we're pretty close i think with merge into delta which is under review. waiting for it would be nice. anyway, we're 3 weeks after the last release.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-07-20 06:50:56
 +
+

*Thread Reply:* @Anirudh Shrinivason releases are available basically on-demand using our process in GOVERNANCE.md. I recommend watching 1958 and then making a request in #general once it’s been merged. But, as Paweł suggested, we have a scheduled release coming soon, anyway. Thanks for your interest in the fix!

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/GOVERNANCE.md | GOVERNANCE.md>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+
+ + + + +
#1958 [SPARK] fix merge into for delta
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-07-20 11:01:14
 +
+

*Thread Reply:* Ahh I see. Got it. Thanks! @Michael Robinson @Paweł Leszczyński

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-07-21 03:12:22
 +
+

*Thread Reply:* @Anirudh Shrinivason it's merged -> https://github.com/OpenLineage/OpenLineage/pull/1958

+
+ + + + +
#1958 [SPARK] fix merge into for delta
+
+ <p><strong>Problem</strong></p> + +<p>Symptoms: <code>MERGE INTO</code> is not generating column level lineage for delta queries.</p> + +<p>This is connected with some other issues that should be resolved within this PR:</p> + +<p>• column level lineage is not tested for each supported spark version (its tests should be moved to <code>app</code> subproject) +• column level lineage does not support including visitors per Spark version</p> + +<p>Closes: <a href="https://github.com/OpenLineage/OpenLineage/issues/1919">#1919</a></p> + +<p><strong>Solution</strong></p> + +<p>• run column lineage tests on all supported versions, +• implement merge into per spark (per delta version)</p> + +<blockquote> + <p><strong>Note:</strong> All schema changes require discussion. Please <a href="https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue">link the issue</a> for context.</p> +</blockquote> + +<p>☐ Your change modifies the <a href="https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.json">core</a> OpenLineage model +☐ Your change modifies one or more OpenLineage <a href="https://github.com/OpenLineage/OpenLineage/tree/main/spec/facets">facets</a></p> + +<p>If you're contributing a new integration, please specify the scope of the integration and how/where it has been tested (e.g., Apache Spark integration supports <code>S3</code> and <code>GCS</code> filesystem operations, tested with AWS EMR).</p> + +<p><strong>One-line summary:</strong> +<strong>Checklist</strong></p> + +<p>☐ You've <a href="https://github.com/OpenLineage/OpenLineage/blob/main/why-the-dco.md">signed-off</a> your work +☐ Your pull request title follows our <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CONTRIBUTING.md#creating-pull-requests">guidelines</a> +☐ Your changes are accompanied by tests (<em>if relevant</em>) +☐ Your change contains a <a href="https://kurtisnusbaum.medium.com/stacked-diffs-keeping-phabricator-diffs-small-d9964f4dcfa6">small diff</a> and is self-contained +☐ You've updated any relevant documentation (<em>if relevant</em>) +☐ Your comment includes a one-liner for the changelog about the specific purpose of the change (<em>if necessary</em>) +☐ You've versioned the core OpenLineage model or facets according to <a href="https://docs.snowplowanalytics.com/docs/pipeline-components-and-applications/iglu/common-architecture/schemaver">SchemaVer</a> (<em>if relevant</em>) +☐ You've added a <a href="https://github.com/OpenLineage/OpenLineage/tree/main/.github/header_templates.md">header</a> to source files (<em>if relevant</em>)</p> + +<hr /> + +<p>SPDX-License-Identifier: Apache-2.0 <br /> +Copyright 2018-2023 contributors to the OpenLineage project</p> +
+ +
+
Labels
+ documentation, integration/spark +
+ +
+
Comments
+ 2 +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-07-21 04:19:15
 +
+

*Thread Reply:* Awesome thanks so much! @Paweł Leszczyński

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Juan Manuel Cappi + (juancappi@gmail.com) +
+
2023-07-19 06:59:31
 +
+

Hi there, related to my question a few days ago about usage of time travel in iceberg, currently only the alias used (i.e. tag, branch) is captured as part of the dataset identifier for lineage. If the tag is removed, or even worse, if it’s removed and re-created with the same name pointing to a difference snapshotid, the lineage will be capturing an inaccurate history. So, ideally, we’d like to capture the actual snapshotid behind the named reference, as part of the lineage. Anyone else thinking this is a reasonable scenario? => more in 🧵

+ + + +
+ 👀 Paweł Leszczyński, Dongjin Seo +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Juan Manuel Cappi + (juancappi@gmail.com) +
+
2023-07-19 07:14:54
 +
+

*Thread Reply:* One hacky approach would be to update the current dataset identifier to include the snapshot_id, so, for schema.table.tag we would have something like schema.table.tag-snapshot_id. The benefit is that it’s explicit and it doesn’t require a change in the OL schemas. The obvious downside (though not that serious in my opinion) is that impacts readability. Not sure though if there are other non-obvious side-effects.

+ +

Another alternative would be to add a dedicated property. For instance, the job > latestRun schema, the input/output dataset version objects could look like this: +"inputDatasetVersions": [ + { + "datasetVersionId": { + "namespace": "<s3a://warehouse>", + "name": "schema.table.tag", + "snapshot_id": "7056736771450556218", + "version": "1c634e18-e357-347b-b758-4337ac352d6d" + }, + "facets": {} + } +] +And column lineage could look like: +```"columnLineage": [ + { + "name": "somefield", + "inputFields": [ + { + "namespace": "s3a:warehouse", + "dataset": "schema.table.tag", + "snapshotid": "7056736771450556218", + "field": "some_field", + ... + }, + ...

+ +
  ],
+
+ +

...```

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-07-19 08:33:43
 +
+

*Thread Reply:* @Paweł Leszczyński what do you think?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-07-19 08:38:16
 +
+

*Thread Reply:* 1. How does snapshotId differ from version? Could one make OL version property to be a string concat of iceberg-snapshot-id.iceberg-version

+ +
  1. I don't think it's necessary (or don't understand why) to add snapshot-id within column-linegea. Each entry within inputFields of columnLineage is already available within inputs of the OL event related to this run.
    2. +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Juan Manuel Cappi + (juancappi@gmail.com) +
+
2023-07-19 18:43:31
 +
+

*Thread Reply:* Yes, I think follow the idea. The problem with that is the version is tied to the dataset name, i.e. my_namespace.table_A.tag_v1 which stays the same for the source dataset, which is the one being used with time travel. +Suppose the following sequence: +step 1 => +tableA.tagv1 has snapshot id 123-abc +run job: table_A.tag_v1 -> job x -> table_B +the inputDatasetVersions > datasetVersionId > version for table_B points to an object which represents table_A.tag_v1 with snapshot id 123-abc correctly captured within facets > version > datasetVersion

+ +

step 2 => +delete tag_v1, insert some data, create tag_v1 again +now table_A.tag_v1 has snapshot id 456-def +run job again: table_A.tag_v1 -> job x -> table_B +the inputDatasetVersions > datasetVersionId > version for table_B points to the same object which represents table_A.tag_v1 only now snapshot id has been replaced by 456-def within facets > version > datasetVersion which means I don’t have way to know which was the snapshot id used in the step 1

+ +

The “hack” I mentioned above though seems to solve the issue, since a new dataset is captured for each combination, so no information is overwritten/lost, i.e., the datasets referenced in inputDatasetVersions are now named: +table_A.tag_v1-123-abc +table_A.tag_v1-456-def

+ +

As a side effect, the column lineage also gets “fixed”, since the lineage for the step 1 and step 2 job runs, without the “hack” both referenced table_A.tag_v1 as the source of input field, though in each run the snapshot id was different. With the hack, one run references table_A.tag_v1-123-abc and the other one table_A.tag_v1-456-def

+ +

Hope it makes sense. If it helps, I can put together a few json files with the examples I’ve been using to experiment

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-07-20 06:35:22
 +
+

*Thread Reply:* So, my understanding of the problem is that icberg version is not unique. So, if you have version 3, revert to version 2, and then write something again, one ends up again with version 3.

+ +

I would not like to mess with dataset names because on the backend sides like Marquez, dataset names being the same in different jobs and runs allow creating lineage graph. If dataset names are different, then there is no way to build lineage graph across multiple jobs.

+ +

Adding snapshot_id to datasetVersion is one option to go. My concern here is that this is so iceberg specific while we're aiming to have a general solution to dataset versioning.

+ +

Some other options are: send concat of version+snapshotId as a version or send only snapshot_id as a version. The second ain't that bad as actually snapashotId is something we're aiming to get as a version, isn't it?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-07-21 04:21:26
 +
+

Hi guys, I’d like to open a vote to release the next OpenLineage version! We'd really like to use the fixed CustomEnvironmentFacetBuilder for delta catalogs, and column lineage for Merge Into command in the spark integration! Thanks! 🙂

+ + + +
+ ➕ Jakub Dardziński, Willy Lulciuc, Michael Robinson, Maciej Obuchowski, Anirudh Shrinivason +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-07-21 13:09:39
 +
+

*Thread Reply:* Thanks, all. The release is authorized and will be initiated within two business days per our policy here.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-07-25 13:44:47
 +
+

*Thread Reply:* @Anirudh Shrinivason and others waiting on this release: the release process isn’t working as expected due to security improvements recently made to the website, ironically enough, which is the source for the spec. But we’re working on a fix and hope to complete the release soon.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-07-25 15:19:49
 +
+

*Thread Reply:* @Anirudh Shrinivason the release (0.30.1) is out now. Thanks for your patience 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-07-25 23:21:14
 +
+

*Thread Reply:* Hi @Michael Robinson Thanks a lot!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-07-26 08:52:24
 +
+

*Thread Reply:* 👍

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Madhav Kakumani + (madhav.kakumani@6point6.co.uk) +
+
2023-07-21 06:38:16
 +
+

Hi, I am running a job in Marquez with 180 rows of metadata but it is running for more than an hour. Is there a way to check the log on Marquez? Below is the screenshot of the job:

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2023-07-21 08:10:58
 +
+

*Thread Reply:* > I am running a job in Marquez with 180 rows of metadata +Do you mean that you have +100 rows of metadata in the jobs table for Marquez? Or that the job never finishes?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2023-07-21 08:11:47
 +
+

*Thread Reply:* Also, yes, we have an even viewer that allows you to query the raw OL events

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2023-07-21 08:12:19
 +
+

*Thread Reply:* If you post a sample of your events, it’d be helpful to troubleshoot your issue

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Madhav Kakumani + (madhav.kakumani@6point6.co.uk) +
+
2023-07-21 08:53:25
 +
+

*Thread Reply:*

+ +
+
Untitled
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Madhav Kakumani + (madhav.kakumani@6point6.co.uk) +
+
2023-07-21 08:53:31
 +
+

*Thread Reply:* Sure Willy thanks for your response. The job is still running. This is the code I am running from jupyter notebook using Python client:

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Madhav Kakumani + (madhav.kakumani@6point6.co.uk) +
+
2023-07-21 08:54:33
 +
+

*Thread Reply:* as you can see my input and output datasets are just 1 row

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Madhav Kakumani + (madhav.kakumani@6point6.co.uk) +
+
2023-07-21 08:55:02
 +
+

*Thread Reply:* included column lineage but job keeps running so I don't know if it is working

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Madhav Kakumani + (madhav.kakumani@6point6.co.uk) +
+
2023-07-21 06:38:49
 +
+

Please ignore 'UPDATED AT' timestamp

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Madhav Kakumani + (madhav.kakumani@6point6.co.uk) +
+
2023-07-21 07:56:48
 +
+

@Paweł Leszczyński there is lot of interest in our organisation to implement Openlineage in several project and we might take the spark route so on that note a small question: Does open lineage works from extracting data from the Catalyst optimiser's Physical/Logical plans etc?

+ + + +
+ 👍 Paweł Leszczyński +
+ +
+ ❤️ Willy Lulciuc, Paweł Leszczyński, Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-07-21 08:20:33
 +
+

*Thread Reply:* spark integration is based on extracting lineage from optimized plans

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-07-21 08:25:35
 +
+

*Thread Reply:* https://youtu.be/rO3BPqUtWrI?t=1326 i recommend whole presentation but in case you're just interested in Spark integration, there few mins that explain how this is achieved (link points to 22:06 min of video)

+
+
YouTube
+ +
+ + + } + + Databricks + (https://www.youtube.com/@Databricks) +
+ + + +
Cross-Platform Data Lineage with OpenLineage
+
+ +
+ + + + + + + + +
Original URL: https://youtu.be/rO3BPqUtWrI?t=1326
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Madhav Kakumani + (madhav.kakumani@6point6.co.uk) +
+
2023-07-21 08:43:47
 +
+

*Thread Reply:* Thanks Pawel for sharing. I will take a look. Have a nice weekend.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jens Pfau + (jenspfau@google.com) +
+
2023-07-21 08:22:51
 +
+

Hello everyone!

+ + + +
+ 👋 Jakub Dardziński, Maciej Obuchowski, Willy Lulciuc, Michael Robinson, Harel Shein, Ross Turk, Robin Fehr, Julien Le Dem +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-07-21 09:57:51
 +
+

*Thread Reply:* Welcome, @Jens Pfau!

+ + + +
+ 😀 Jens Pfau +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
George Polychronopoulos + (george.polychronopoulos@6point6.co.uk) +
+
2023-07-23 08:36:38
 +
+

hello everyone! I am trying to follow your guide +https://openlineage.io/docs/integrations/spark/quickstart_local +and when i execute +spark.createDataFrame([ + {'a': 1, 'b': 2}, + {'a': 3, 'b': 4} +]).write.mode("overwrite").saveAsTable("temp1")

+ +

i not getting the expected result

+
+
openlineage.io
+ + + +
Quickstart with Jupyter | OpenLineage Docs
+
+ Trying out the Spark integration is super easy if you already have Docker Desktop and git installed. +
+ + + + + + +
Original URL: https://openlineage.io/docs/integrations/spark/quickstart_local
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
George Polychronopoulos + (george.polychronopoulos@6point6.co.uk) +
+
2023-07-23 08:37:55
 +
+

``23/07/23 12:35:20 INFO OpenLineageRunEventBuilder: Visiting query plan Optional[== Parsed Logical Plan == +'CreateTabletemp1`, Overwrite ++- LogicalRDD [a#6L, b#7L], false

+ +

== Analyzed Logical Plan ==

+ +

CreateDataSourceTableAsSelectCommand temp1, Overwrite, [a, b] ++- LogicalRDD [a#6L, b#7L], false

+ +

== Optimized Logical Plan == +CreateDataSourceTableAsSelectCommand temp1, Overwrite, [a, b] ++- LogicalRDD [a#6L, b#7L], false

+ +

== Physical Plan == +Execute CreateDataSourceTableAsSelectCommand temp1, Overwrite, [a, b] ++- **(1) Scan ExistingRDD[a#6L,b#7L] +] with input dataset builders [<function1>, <function1>, <function1>, <function1>, <function1>] +23/07/23 12:35:20 INFO OpenLineageRunEventBuilder: Visiting query plan Optional[== Parsed Logical Plan == +'CreateTable temp1, Overwrite ++- LogicalRDD [a#6L, b#7L], false

+ +

== Analyzed Logical Plan ==

+ +

CreateDataSourceTableAsSelectCommand temp1, Overwrite, [a, b] ++- LogicalRDD [a#6L, b#7L], false

+ +

== Optimized Logical Plan == +CreateDataSourceTableAsSelectCommand temp1, Overwrite, [a, b] ++- LogicalRDD [a#6L, b#7L], false

+ +

== Physical Plan == +Execute CreateDataSourceTableAsSelectCommand temp1, Overwrite, [a, b] ++- **(1) Scan ExistingRDD[a#6L,b#7L] +] with output dataset builders [<function1>, <function1>, <function1>, <function1>, <function1>, <function1>, <function1>] +23/07/23 12:35:20 INFO CreateDataSourceTableAsSelectCommandVisitor: Matched io.openlineage.spark.agent.lifecycle.plan.CreateDataSourceTableAsSelectCommandVisitor<org.apache.spark.sql.execution.command.CreateDataSourceTableAsSelectCommand,io.openlineage.client.OpenLineage$OutputDataset> to logical plan CreateDataSourceTableAsSelectCommand temp1, Overwrite, [a, b] ++- LogicalRDD [a#6L, b#7L], false

+ +

23/07/23 12:35:20 INFO CreateDataSourceTableAsSelectCommandVisitor: Matched io.openlineage.spark.agent.lifecycle.plan.CreateDataSourceTableAsSelectCommandVisitor<org.apache.spark.sql.execution.command.CreateDataSourceTableAsSelectCommand,io.openlineage.client.OpenLineage$OutputDataset> to logical plan CreateDataSourceTableAsSelectCommand temp1, Overwrite, [a, b] ++- LogicalRDD [a#6L, b#7L], false

+ +

23/07/23 12:35:20 ERROR EventEmitter: Could not emit lineage w/ exception +io.openlineage.client.OpenLineageClientException: io.openlineage.spark.shaded.org.apache.http.client.ClientProtocolException + at io.openlineage.client.transports.HttpTransport.emit(HttpTransport.java:105) + at io.openlineage.client.OpenLineageClient.emit(OpenLineageClient.java:34) + at io.openlineage.spark.agent.EventEmitter.emit(EventEmitter.java:71) + at io.openlineage.spark.agent.lifecycle.SparkSQLExecutionContext.start(SparkSQLExecutionContext.java:77) + at io.openlineage.spark.agent.OpenLineageSparkListener.lambda$sparkSQLExecStart$0(OpenLineageSparkListener.java:99) + at java.base/java.util.Optional.ifPresent(Optional.java:183) + at io.openlineage.spark.agent.OpenLineageSparkListener.sparkSQLExecStart(OpenLineageSparkListener.java:99) + at io.openlineage.spark.agent.OpenLineageSparkListener.onOtherEvent(OpenLineageSparkListener.java:90) + at org.apache.spark.scheduler.SparkListenerBus.doPostEvent(SparkListenerBus.scala:100) + at org.apache.spark.scheduler.SparkListenerBus.doPostEvent$(SparkListenerBus.scala:28) + at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37) + at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37) + at org.apache.spark.util.ListenerBus.postToAll(ListenerBus.scala:117) + at org.apache.spark.util.ListenerBus.postToAll$(ListenerBus.scala:101) + at org.apache.spark.scheduler.AsyncEventQueue.super$postToAll(AsyncEventQueue.scala:105) + at org.apache.spark.scheduler.AsyncEventQueue.$anonfun$dispatch$1(AsyncEventQueue.scala:105) + at scala.runtime.java8.JFunction0$mcJ$sp.apply(JFunction0$mcJ$sp.java:23) + at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62) + at org.apache.spark.scheduler.AsyncEventQueue.org$apache$spark$scheduler$AsyncEventQueue$$dispatch(AsyncEventQueue.scala:100) + at org.apache.spark.scheduler.AsyncEventQueue$$anon$2.$anonfun$run$1(AsyncEventQueue.scala:96) + at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1381) + at org.apache.spark.scheduler.AsyncEventQueue$$anon$2.run(AsyncEventQueue.scala:96) +Caused by: io.openlineage.spark.shaded.org.apache.http.client.ClientProtocolException + at io.openlineage.spark.shaded.org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:187) + at io.openlineage.spark.shaded.org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) + at io.openlineage.spark.shaded.org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108) + at io.openlineage.client.transports.HttpTransport.emit(HttpTransport.java:100) + ... 21 more +Caused by: io.openlineage.spark.shaded.org.apache.http.ProtocolException: Target host is not specified + at io.openlineage.spark.shaded.org.apache.http.impl.conn.DefaultRoutePlanner.determineRoute(DefaultRoutePlanner.java:71) + at io.openlineage.spark.shaded.org.apache.http.impl.client.InternalHttpClient.determineRoute(InternalHttpClient.java:125) + at io.openlineage.spark.shaded.org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184) + ... 24 more +23/07/23 12:35:20 INFO ParquetFileFormat: Using default output committer for Parquet: org.apache.parquet.hadoop.ParquetOutputCommitter +23/07/23 12:35:20 INFO FileOutputCommitter: File Output Committer Algorithm version is 1 +23/07/23 12:35:20 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false +23/07/23 12:35:20 INFO SQLHadoopMapReduceCommitProtocol: Using user defined output committer class org.apache.parquet.hadoop.ParquetOutputCommitter +23/07/23 12:35:20 INFO FileOutputCommitter: File Output Committer Algorithm version is 1 +23/07/23 12:35:20 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false +23/07/23 12:35:20 INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.parquet.hadoop.ParquetOutputCommitter +23/07/23 12:35:20 INFO CodeGenerator: Code generated in 120.989125 ms +23/07/23 12:35:21 INFO SparkContext: Starting job: saveAsTable at NativeMethodAccessorImpl.java:0 +23/07/23 12:35:21 INFO DAGScheduler: Got job 0 (saveAsTable at NativeMethodAccessorImpl.java:0) with 1 output partitions +23/07/23 12:35:21 INFO DAGScheduler: Final stage: ResultStage 0 (saveAsTable at NativeMethodAccessorImpl.java:0) +23/07/23 12:35:21 INFO DAGScheduler: Parents of final stage: List() +23/07/23 12:35:21 INFO DAGScheduler: Missing parents: List() +23/07/23 12:35:21 INFO OpenLineageRunEventBuilder: Visiting query plan Optional[== Parsed Logical Plan == +'CreateTable temp1, Overwrite ++- LogicalRDD [a#6L, b#7L], false

+ +

== Analyzed Logical Plan ==

+ +

CreateDataSourceTableAsSelectCommand temp1, Overwrite, [a, b] ++- LogicalRDD [a#6L, b#7L], false

+ +

== Optimized Logical Plan == +CreateDataSourceTableAsSelectCommand temp1, Overwrite, [a, b] ++- LogicalRDD [a#6L, b#7L], false

+ +

== Physical Plan == +Execute CreateDataSourceTableAsSelectCommand temp1, Overwrite, [a, b] ++- **(1) Scan ExistingRDD[a#6L,b#7L] +] with input dataset builders [<function1>, <function1>, <function1>, <function1>, <function1>] +23/07/23 12:35:21 INFO OpenLineageRunEventBuilder: Visiting query plan Optional[== Parsed Logical Plan == +'CreateTable temp1, Overwrite ++- LogicalRDD [a#6L, b#7L], false

+ +

== Analyzed Logical Plan ==

+ +

CreateDataSourceTableAsSelectCommand temp1, Overwrite, [a, b] ++- LogicalRDD [a#6L, b#7L], false

+ +

== Optimized Logical Plan == +CreateDataSourceTableAsSelectCommand temp1, Overwrite, [a, b] ++- LogicalRDD [a#6L, b#7L], false

+ +

== Physical Plan == +Execute CreateDataSourceTableAsSelectCommand temp1, Overwrite, [a, b] ++- **(1) Scan ExistingRDD[a#6L,b#7L] +] with output dataset builders [<function1>, <function1>, <function1>, <function1>, <function1>, <function1>, <function1>] +23/07/23 12:35:21 INFO CreateDataSourceTableAsSelectCommandVisitor: Matched io.openlineage.spark.agent.lifecycle.plan.CreateDataSourceTableAsSelectCommandVisitor<org.apache.spark.sql.execution.command.CreateDataSourceTableAsSelectCommand,io.openlineage.client.OpenLineage$OutputDataset> to logical plan CreateDataSourceTableAsSelectCommand temp1, Overwrite, [a, b] ++- LogicalRDD [a#6L, b#7L], false

+ +

23/07/23 12:35:21 INFO CreateDataSourceTableAsSelectCommandVisitor: Matched io.openlineage.spark.agent.lifecycle.plan.CreateDataSourceTableAsSelectCommandVisitor<org.apache.spark.sql.execution.command.CreateDataSourceTableAsSelectCommand,io.openlineage.client.OpenLineage$OutputDataset> to logical plan CreateDataSourceTableAsSelectCommand temp1, Overwrite, [a, b] ++- LogicalRDD [a#6L, b#7L], false

+ +

23/07/23 12:35:21 INFO DAGScheduler: Submitting ResultStage 0 (MapPartitionsRDD[10] at saveAsTable at NativeMethodAccessorImpl.java:0), which has no missing parents +23/07/23 12:35:21 ERROR EventEmitter: Could not emit lineage w/ exception +io.openlineage.client.OpenLineageClientException: io.openlineage.spark.shaded.org.apache.http.client.ClientProtocolException + at io.openlineage.client.transports.HttpTransport.emit(HttpTransport.java:105) + at io.openlineage.client.OpenLineageClient.emit(OpenLineageClient.java:34) + at io.openlineage.spark.agent.EventEmitter.emit(EventEmitter.java:71) + at io.openlineage.spark.agent.lifecycle.SparkSQLExecutionContext.start(SparkSQLExecutionContext.java:174) + at io.openlineage.spark.agent.OpenLineageSparkListener.lambda$onJobStart$9(OpenLineageSparkListener.java:153) + at java.base/java.util.Optional.ifPresent(Optional.java:183) + at io.openlineage.spark.agent.OpenLineageSparkListener.onJobStart(OpenLineageSparkListener.java:149) + at org.apache.spark.scheduler.SparkListenerBus.doPostEvent(SparkListenerBus.scala:37) + at org.apache.spark.scheduler.SparkListenerBus.doPostEvent$(SparkListenerBus.scala:28) + at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37) + at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37) + at org.apache.spark.util.ListenerBus.postToAll(ListenerBus.scala:117) + at org.apache.spark.util.ListenerBus.postToAll$(ListenerBus.scala:101) + at org.apache.spark.scheduler.AsyncEventQueue.super$postToAll(AsyncEventQueue.scala:105) + at org.apache.spark.scheduler.AsyncEventQueue.$anonfun$dispatch$1(AsyncEventQueue.scala:105) + at scala.runtime.java8.JFunction0$mcJ$sp.apply(JFunction0$mcJ$sp.java:23) + at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62) + at org.apache.spark.scheduler.AsyncEventQueue.org$apache$spark$scheduler$AsyncEventQueue$$dispatch(AsyncEventQueue.scala:100) + at org.apache.spark.scheduler.AsyncEventQueue$$anon$2.$anonfun$run$1(AsyncEventQueue.scala:96) + at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1381) + at org.apache.spark.scheduler.AsyncEventQueue$$anon$2.run(AsyncEventQueue.scala:96) +Caused by: io.openlineage.spark.shaded.org.apache.http.client.ClientProtocolException + at io.openlineage.spark.shaded.org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:187) + at io.openlineage.spark.shaded.org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) + at io.openlineage.spark.shaded.org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108) + at io.openlineage.client.transports.HttpTransport.emit(HttpTransport.java:100) + ... 20 more +Caused by: io.openlineage.spark.shaded.org.apache.http.ProtocolException: Target host is not specified + at io.openlineage.spark.shaded.org.apache.http.impl.conn.DefaultRoutePlanner.determineRoute(```

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
George Polychronopoulos + (george.polychronopoulos@6point6.co.uk) +
+
2023-07-23 08:38:46
 +
+

23/07/23 12:35:20 ERROR EventEmitter: Could not emit lineage w/ exception +io.openlineage.client.OpenLineageClientException: io.openlineage.spark.shaded.org.apache.http.client.ClientProtocolException + at io.openlineage.client.transports.HttpTransport.emit(HttpTransport.java:105) + at io.openlineage.client.OpenLineageClient.emit(OpenLineageClient.java:34) + at io.openlineage.spark.agent.EventEmitter.emit(EventEmitter.java:71) + at io.openlineage.spark.agent.lifecycle.SparkSQLExecutionContext.start(SparkSQLExecutionContext.java:77) + at io.openlineage.spark.agent.OpenLineageSparkListener.lambda$sparkSQLExecStart$0(OpenLineageSparkListener.java:99)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-07-23 13:31:53
 +
+

*Thread Reply:* That looks like your URL provided to OpenLineage is missing http:// or https:// in the front

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
George Polychronopoulos + (george.polychronopoulos@6point6.co.uk) +
+
2023-07-23 14:54:55
 +
+

*Thread Reply:* sorry how can i resolve this ? do i need to add this ? i just follow the guide step by step . You dont mention anywhere to add anything. You provide smth that

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
George Polychronopoulos + (george.polychronopoulos@6point6.co.uk) +
+
2023-07-23 14:55:05
 +
+

*Thread Reply:* really does not work out of the box

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
George Polychronopoulos + (george.polychronopoulos@6point6.co.uk) +
+
2023-07-23 14:55:13
 +
+

*Thread Reply:* anbd this is supposed to be a demo

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-07-23 17:07:49
 +
+

*Thread Reply:* bumping e.g. to io.openlineage:openlineage_spark:0.29.2 seems to be fixing the issue

+ +

not sure why it stopped working for 0.12.0 but we’ll take a look and fix accordingly

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-07-24 04:51:34
 +
+

*Thread Reply:* ...probably by bumping the version on this page 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
George Polychronopoulos + (george.polychronopoulos@6point6.co.uk) +
+
2023-07-24 05:00:28
 +
+

*Thread Reply:* thank you both for coming back to me , I bumped to 0.29 and i think that it now runs.Is this the expected output ? +23/07/24 08:43:55 INFO ConsoleTransport: {"eventTime":"2023_07_24T08:43:55.941Z","producer":"<https://github.com/OpenLineage/OpenLineage/tree/0.29.2/integration/spark>","schemaURL":"<https://openlineage.io/spec/2-0-0/OpenLineage.json#/$defs/RunEvent>","eventType":"COMPLETE","run":{"runId":"186c06c0_e79c_43cf_8bb7_08e1ab4c86a5","facets":{"spark.logicalPlan":{"_producer":"<https://github.com/OpenLineage/OpenLineage/tree/0.29.2/integration/spark>","_schemaURL":"<https://openlineage.io/spec/2-0-0/OpenLineage.json#/$defs/RunFacet>","plan":[{"class":"org.apache.spark.sql.execution.command.CreateDataSourceTableAsSelectCommand","num-children":1,"table":{"product-class":"org.apache.spark.sql.catalyst.catalog.CatalogTable","identifier":{"product-class":"org.apache.spark.sql.catalyst.TableIdentifier","table":"temp2","database":"default"},"tableType":{"product-class":"org.apache.spark.sql.catalyst.catalog.CatalogTableType","name":"MANAGED"},"storage":{"product_class":"org.apache.spark.sql.catalyst.catalog.CatalogStorageFormat","compressed":false,"properties":null},"schema":{"type":"struct","fields":[]},"provider":"parquet","partitionColumnNames":[],"owner":"","createTime":1690188235517,"lastAccessTime":-1,"createVersion":"","properties":null,"unsupportedFeatures":[],"tracksPartitionsInCatalog":false,"schemaPreservesCase":true,"ignoredProperties":null},"mode":null,"query":0,"outputColumnNames":"[a, b]"},{"class":"org.apache.spark.sql.execution.LogicalRDD","num_children":0,"output":[[{"class":"org.apache.spark.sql.catalyst.expressions.AttributeReference","num-children":0,"name":"a","dataType":"long","nullable":true,"metadata":{},"exprId":{"product-class":"org.apache.spark.sql.catalyst.expressions.ExprId","id":12,"jvmId":"173725f4_02c4_4174_9d18_3a61aa311d62"},"qualifier":[]}],[{"class":"org.apache.spark.sql.catalyst.expressions.AttributeReference","num_children":0,"name":"b","dataType":"long","nullable":true,"metadata":{},"exprId":{"product_class":"org.apache.spark.sql.catalyst.expressions.ExprId","id":13,"jvmId":"173725f4-02c4-4174-9d18-3a61aa311d62"},"qualifier":[]}]],"rdd":null,"outputPartitioning":{"product_class":"org.apache.spark.sql.catalyst.plans.physical.UnknownPartitioning","numPartitions":0},"outputOrdering":[],"isStreaming":false,"session":null}]},"spark_version":{"_producer":"<https://github.com/OpenLineage/OpenLineage/tree/0.29.2/integration/spark>","_schemaURL":"<https://openlineage.io/spec/2-0-0/OpenLineage.json#/$defs/RunFacet>","spark-version":"3.1.2","openlineage_spark_version":"0.29.2"}}},"job":{"namespace":"default","name":"sample_spark.execute_create_data_source_table_as_select_command","facets":{}},"inputs":[],"outputs":[{"namespace":"file","name":"/home/jovyan/spark-warehouse/temp2","facets":{"dataSource":{"_producer":"<https://github.com/OpenLineage/OpenLineage/tree/0.29.2/integration/spark>","_schemaURL":"<https://openlineage.io/spec/facets/1-0-0/DatasourceDatasetFacet.json#/$defs/DatasourceDatasetFacet>","name":"file","uri":"file"},"schema":{"_producer":"<https://github.com/OpenLineage/OpenLineage/tree/0.29.2/integration/spark>","_schemaURL":"<https://openlineage.io/spec/facets/1-0-0/SchemaDatasetFacet.json#/$defs/SchemaDatasetFacet>","fields":[{"name":"a","type":"long"},{"name":"b","type":"long"}]},"symlinks":{"_producer":"<https://github.com/OpenLineage/OpenLineage/tree/0.29.2/integration/spark>","_schemaURL":"<https://openlineage.io/spec/facets/1-0-0/SymlinksDatasetFacet.json#/$defs/SymlinksDatasetFacet>","identifiers":[{"namespace":"/home/jovyan/spark-warehouse","name":"default.temp2","type":"TABLE"}]},"lifecycleStateChange":{"_producer":"<https://github.com/OpenLineage/OpenLineage/tree/0.29.2/integration/spark>","_schemaURL":"<https://openlineage.io/spec/facets/1-0-0/LifecycleStateChangeDatasetFacet.json#/$defs/LifecycleStateChangeDatasetFacet>","lifecycleStateChange":"CREATE"}},"outputFacets":{}}]} +? Also i then proceeded to run +docker run --network spark_default -p 3000:3000 -e MARQUEZ_HOST=marquez-api -e MARQUEZ_PORT=5000 --link marquez-api:marquez-api marquezproject/marquez-web:0.19.1 +but the page is empty

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-07-24 11:11:08
 +
+

*Thread Reply:* You'd need to set up spark.openlineage.transport.url to send OpenLineage events to Marquez

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
George Polychronopoulos + (george.polychronopoulos@6point6.co.uk) +
+
2023-07-24 11:12:28
 +
+

*Thread Reply:* where n how can i do this ?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
George Polychronopoulos + (george.polychronopoulos@6point6.co.uk) +
+
2023-07-24 11:13:04
 +
+

*Thread Reply:* do i need to edit the conf ?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-07-24 11:37:09
 +
+

*Thread Reply:* yes, in the spark conf

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
George Polychronopoulos + (george.polychronopoulos@6point6.co.uk) +
+
2023-07-24 11:37:48
 +
+

*Thread Reply:* what this url should be ?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
George Polychronopoulos + (george.polychronopoulos@6point6.co.uk) +
+
2023-07-24 11:37:51
 +
+

*Thread Reply:* http://localhost:3000/ ?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-07-24 11:43:30
 +
+

*Thread Reply:* That depends how you ran Marquez, but looking at your screenshot UI is at 3000, I guess API would be at 5000

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-07-24 11:43:46
 +
+

*Thread Reply:* as that's default in Marquez docker-compose

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
George Polychronopoulos + (george.polychronopoulos@6point6.co.uk) +
+
2023-07-24 11:44:14
 +
+

*Thread Reply:* i cannot see spark conf

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
George Polychronopoulos + (george.polychronopoulos@6point6.co.uk) +
+
2023-07-24 11:44:23
 +
+

*Thread Reply:* is it in there or do i need to create it ?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-07-24 16:42:53
 +
+

*Thread Reply:* Is something like +```from pyspark.sql import SparkSession

+ +

spark = (SparkSession.builder.master('local') + .appName('samplespark') + .config('spark.extraListeners', 'io.openlineage.spark.agent.OpenLineageSparkListener') + .config('spark.jars.packages', 'io.openlineage:openlineagespark:0.29.2') + .config('spark.openlineage.transport.url', 'http://marquez:5000') + .config('spark.openlineage.transport.type', 'http') + .getOrCreate())``` +not working?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
George Polychronopoulos + (george.polychronopoulos@6point6.co.uk) +
+
2023-07-25 05:08:08
 +
+

*Thread Reply:* OK when i use the snippet you provided and then execute +docker run --network sparkdefault -p 3000:3000 -e MARQUEZHOST=marquez-api -e MARQUEZ_PORT=5000 --link marquez-api:marquez-api marquezproject/marquez-web:0.19.1

+ +

I can now see this

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
George Polychronopoulos + (george.polychronopoulos@6point6.co.uk) +
+
2023-07-25 05:08:52
 +
+

*Thread Reply:* but when i click on the job i then get this

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
George Polychronopoulos + (george.polychronopoulos@6point6.co.uk) +
+
2023-07-25 05:09:05
 +
+

*Thread Reply:* so i cannot see any details of the job

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sarwat Fatima + (sarwatfatimam@gmail.com) +
+
2023-09-05 05:54:50
 +
+

*Thread Reply:* @George Polychronopoulos Hi, I am facing the same issue. After adding spark conf and using the docker run command, marquez is still showing empty. Do I need to change something in the run command?

+ +
+
image.png
+ + + + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
George Polychronopoulos + (george.polychronopoulos@6point6.co.uk) +
+
2023-09-05 05:55:15
 +
+

*Thread Reply:* yes i will tell you

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sarwat Fatima + (sarwatfatimam@gmail.com) +
+
2023-09-05 07:36:41
 +
+

*Thread Reply:* For the docker command that I used, I updated the marquez-web version to 0.40.0 and I also updated the Marquez_host which I am not sure if I have to or not. The UI is running but not showing anything docker run --network spark_default -p 3000:3000 -e MARQUEZ_HOST=localhost -e MARQUEZ_PORT=5000 --link marquez-api:marquez-api marquez/marquez-web:0.40.0

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
George Polychronopoulos + (george.polychronopoulos@6point6.co.uk) +
+
2023-09-05 07:36:52
 +
+

*Thread Reply:* is because you are running this command right

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
George Polychronopoulos + (george.polychronopoulos@6point6.co.uk) +
+
2023-09-05 07:36:55
 +
+

*Thread Reply:* yes thats it

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
George Polychronopoulos + (george.polychronopoulos@6point6.co.uk) +
+
2023-09-05 07:36:58
 +
+

*Thread Reply:* you need 0.40

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
George Polychronopoulos + (george.polychronopoulos@6point6.co.uk) +
+
2023-09-05 07:37:03
 +
+

*Thread Reply:* and there is a lot of stuff

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
George Polychronopoulos + (george.polychronopoulos@6point6.co.uk) +
+
2023-09-05 07:37:07
 +
+

*Thread Reply:* you need rto chwange

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
George Polychronopoulos + (george.polychronopoulos@6point6.co.uk) +
+
2023-09-05 07:37:10
 +
+

*Thread Reply:* in the Docker

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
George Polychronopoulos + (george.polychronopoulos@6point6.co.uk) +
+
2023-09-05 07:37:24
 +
+

*Thread Reply:* so the spark

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
George Polychronopoulos + (george.polychronopoulos@6point6.co.uk) +
+
2023-09-05 07:37:25
 +
+

*Thread Reply:* version

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
George Polychronopoulos + (george.polychronopoulos@6point6.co.uk) +
+
2023-09-05 07:37:27
 +
+

*Thread Reply:* the python

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
George Polychronopoulos + (george.polychronopoulos@6point6.co.uk) +
+
2023-09-05 07:38:05
 +
+

*Thread Reply:* version: "3.10" +services: + notebook: + image: jupyter/pyspark-notebook:spark-3.4.1 + ports: + - "8888:8888" + volumes: + - ./docker/notebooks:/home/jovyan/notebooks + - ./build:/home/jovyan/openlineage + links: + - "api:marquez" + depends_on: + - api

+ +

Marquez as an OpenLineage Client

+ +

api: + image: marquezproject/marquez + containername: marquez-api + ports: + - "5000:5000" + - "5001:5001" + volumes: + - ./docker/wait-for-it.sh:/usr/src/app/wait-for-it.sh + links: + - "db:postgres" + dependson: + - db + entrypoint: [ "./wait-for-it.sh", "db:5432", "--", "./entrypoint.sh" ]

+ +

db: + image: postgres:12.1 + containername: marquez-db + ports: + - "5432:5432" + environment: + - POSTGRESUSER=postgres + - POSTGRESPASSWORD=password + - MARQUEZDB=marquez + - MARQUEZUSER=marquez + - MARQUEZPASSWORD=marquez + volumes: + - ./docker/init-db.sh:/docker-entrypoint-initdb.d/init-db.sh + # Enables SQL statement logging (see: https://www.postgresql.org/docs/12/runtime-config-logging.html#GUC-LOG-STATEMENT) + # command: ["postgres", "-c", "log_statement=all"]

+
+
PostgreSQL Documentation
+ + + +
19.8. Error Reporting and Logging
+
+ 19.8.&amp;nbsp;Error Reporting and Logging 19.8.1. Where to Log 19.8.2. When to Log 19.8.3. What to Log 19.8.4. Using CSV-Format Log … +
+ + + + + + + + +
Original URL: https://www.postgresql.org/docs/12/runtime-config-logging.html#GUC-LOG-STATEMENT
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
George Polychronopoulos + (george.polychronopoulos@6point6.co.uk) +
+
2023-09-05 07:38:10
 +
+

*Thread Reply:* this is hopw mine looks

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
George Polychronopoulos + (george.polychronopoulos@6point6.co.uk) +
+
2023-09-05 07:38:20
 +
+

*Thread Reply:* it is all tested and letest version

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
George Polychronopoulos + (george.polychronopoulos@6point6.co.uk) +
+
2023-09-05 07:38:31
 +
+

*Thread Reply:* postgres does not work beyond 12

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
George Polychronopoulos + (george.polychronopoulos@6point6.co.uk) +
+
2023-09-05 07:38:56
 +
+

*Thread Reply:* if you run this docker-compose up

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
George Polychronopoulos + (george.polychronopoulos@6point6.co.uk) +
+
2023-09-05 07:38:58
 +
+

*Thread Reply:* the notebooks

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
George Polychronopoulos + (george.polychronopoulos@6point6.co.uk) +
+
2023-09-05 07:39:02
 +
+

*Thread Reply:* are 10 faster

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
George Polychronopoulos + (george.polychronopoulos@6point6.co.uk) +
+
2023-09-05 07:39:06
 +
+

*Thread Reply:* and give no errors

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
George Polychronopoulos + (george.polychronopoulos@6point6.co.uk) +
+
2023-09-05 07:39:14
 +
+

*Thread Reply:* also you need to update other stuff

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
George Polychronopoulos + (george.polychronopoulos@6point6.co.uk) +
+
2023-09-05 07:39:18
 +
+

*Thread Reply:* such as

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
George Polychronopoulos + (george.polychronopoulos@6point6.co.uk) +
+
2023-09-05 07:39:26
 +
+

*Thread Reply:* dont run what is in the docs

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
George Polychronopoulos + (george.polychronopoulos@6point6.co.uk) +
+
2023-09-05 07:39:34
 +
+

*Thread Reply:* but run what is in github

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
George Polychronopoulos + (george.polychronopoulos@6point6.co.uk) +
+
2023-09-05 07:40:13
 +
+

*Thread Reply:* https://github.com/OpenLineage/OpenLineage/tree/main/integration/spark

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
George Polychronopoulos + (george.polychronopoulos@6point6.co.uk) +
+
2023-09-05 07:40:22
 +
+

*Thread Reply:* run in your notebooks what is in here

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
George Polychronopoulos + (george.polychronopoulos@6point6.co.uk) +
+
2023-09-05 07:40:32
 +
+

*Thread Reply:* ```from pyspark.sql import SparkSession

+ +

spark = (SparkSession.builder.master('local') + .appName('samplespark') + .config('spark.jars.packages', 'io.openlineage:openlineagespark:1.1.0') + .config('spark.extraListeners', 'io.openlineage.spark.agent.OpenLineageSparkListener') + .config('spark.openlineage.transport.url', 'http://{openlineage.client.host}/api/v1/namespaces/spark_integration/') + .getOrCreate())```

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
George Polychronopoulos + (george.polychronopoulos@6point6.co.uk) +
+
2023-09-05 07:40:38
 +
+

*Thread Reply:* the dont update documentation

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
George Polychronopoulos + (george.polychronopoulos@6point6.co.uk) +
+
2023-09-05 07:40:44
 +
+

*Thread Reply:* it took me 4 weeks to get here

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
George Polychronopoulos + (george.polychronopoulos@6point6.co.uk) +
+
2023-07-23 08:39:13
 +
+

is this a known error ? does anyone know how to debug this ?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Steven + (xli@zjuici.com) +
+
2023-07-23 23:57:43
 +
+

Hi, +Using Marquez. I tried to get the dataset version through two apis. +First: +http://host/api/v1/namespaces/{namespace}/datasets/{dataset} +It will include a currentVersion in the response. +Then: +http://host/api/v1/namespaces/{namespace}/datasets/{dataset}/versions/{currentVersion} +But the version used here refers to the "version" column in table dataset_versions. Not the primary key "uuid". Which leads to 404 not found. +I checked other apis but seemed that there are no other way to get the version through "currentVersion". +Any help?

+ + + +
+ 👀 Maciej Obuchowski, Willy Lulciuc +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Steven + (xli@zjuici.com) +
+
2023-07-24 00:14:43
 +
+

*Thread Reply:* Like I want to change the facets of a specific dataset.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-07-24 16:45:18
 +
+

*Thread Reply:* @Willy Lulciuc do you have any idea? 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Steven + (xli@zjuici.com) +
+
2023-07-25 05:02:47
 +
+

*Thread Reply:* I solved this by adding a new job which outputs to the same dataset. This ended up in a newer dataset version.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2023-07-25 06:20:58
 +
+

*Thread Reply:* @Steven great to hear that you solved the issue! but there are some minor logical inconsistencies that we’d like to address with versioning (for both datasets and jobs) in Marquez. The tl;dr is the version column wasn’t meant to be used externally, but internally within Marquez. The issue is “minor” as it’s more of a pointer thing. We’ll be addressing soon. For some background, you can look at: +• https://github.com/MarquezProject/marquez/issues/2071 +• https://github.com/MarquezProject/marquez/pull/2153

+
+ + + + +
#2071 Proposal: Update DatasetVersion versioning
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/MarquezProject/marquez">MarquezProject/marquez</a> +
+ + +
+
+ + + + +
#2153 Proposal/2071 update version versioning
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/MarquezProject/marquez">MarquezProject/marquez</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Steven + (xli@zjuici.com) +
+
2023-07-25 05:06:48
 +
+

Hi, +Are there any keys to set in marquez.yaml to skip db initialization and use existing db? I am deploying the marquez client on k8s client, which uses a cloud postgres. Every time I restart the marquez deployment I have to drop all those tables otherwise it will raise table already exists ERROR

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2023-07-25 06:43:32
 +
+

*Thread Reply:* @Steven ahh very good point, it’s technically not “error” in the true sense, but annoying nonetheless. I think you’re referencing the init container in the Marquez helm chart? https://github.com/MarquezProject/marquez/blob/main/chart/templates/marquez/deployment.yaml#L37

+
+ + + + +
<https://github.com/MarquezProject/marquez/blob/main/chart/templates/marquez/deployment.yaml | deployment.yaml>
+
+ <pre><code> - until pg_isready -h ${POSTGRES_HOST} -p ${POSTGRES_PORT} -U ${POSTGRES_USER}; do echo waiting for database; sleep 2; done; +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/MarquezProject/marquez">MarquezProject/marquez</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2023-07-25 06:45:24
 +
+

*Thread Reply:* hmm, actually what raises the error you’re referencing? the Maruez http server?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2023-07-25 06:49:08
 +
+

*Thread Reply:* > Every time I restart the marquez deployment I have to drop all those tables otherwise it will raise table already exists ERROR +This shouldn’t be an error. I’m trying to understand the scenario in which this error is thrown (any info is helpful). We use flyway to manage our db schema, but you may have gotten in an odd state somehow

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sheeri Cabral (Collibra) + (sheeri.cabral@collibra.com) +
+
2023-07-25 12:52:51
 +
+

For Databricks notebooks, does the Spark listener work without any notebook changes? (I see that Azure Databricks -> purview needs no changes, but I’m not sure if that applies to anywhere….e.g. if I have an existing databricks notebook, and I add a spark listener, can I get column-level lineage? or do I need to change my notebook to use openlineage libraries, like I do with an arbitrary Python script?)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-07-31 03:35:58
 +
+

*Thread Reply:* Nope, one should modify the cluster as per doc <https://openlineage.io/docs/integrations/spark/quickstart_databricks> but no changes in notebook are required.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sheeri Cabral (Collibra) + (sheeri.cabral@collibra.com) +
+
2023-08-02 10:59:00
 +
+

*Thread Reply:* Right, great, that’s exactly what I was hoping 😄

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-07-25 15:24:17
 +
+

@channel +We released OpenLineage 0.30.1, including: +Added +• Flink: support Iceberg sinks #1960 @pawel-big-lebowski +• Spark: column-level lineage for merge into on delta tables #1958 @pawel-big-lebowski +• Spark: column-level lineage for merge into on Iceberg tables #1971 @pawel-big-lebowski +• Spark: add supprt for Iceberg REST catalog #1963 @juancappi +• Airflow: add possibility to force direct-execution based on environment variable #1934 @mobuchowski +• SQL: add support for Apple Silicon to openlineage-sql-java #1981 @davidjgoss +• Spec: add facet deletion #1975 @julienledem +• Client: add a file transport #1891 @alexandre bergere +Changed +• Airflow: do not run plugin if OpenLineage provider is installed #1999 @JDarDagran +• Python: rename config to config_class #1998 @mobuchowski +Plus test improvements, docs changes, bug fixes and more. +Thanks to all the contributors, including new contributors @davidjgoss, @alexandre bergere and @Juan Manuel Cappi! +Release: https://github.com/OpenLineage/OpenLineage/releases/tag/0.30.1 +Changelog: https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md +Commit history: https://github.com/OpenLineage/OpenLineage/compare/0.29.2...0.30.1 +Maven: https://oss.sonatype.org/#nexus-search;quick~openlineage +PyPI: https://pypi.org/project/openlineage-python/

+ + + +
+ 👏 Julian Rossi, Bernat Gabor, Anirudh Shrinivason, Maciej Obuchowski, Jens Pfau, Sheeri Cabral (Collibra) +
+ +
+ 👍 Athitya Kumar, Sheeri Cabral (Collibra) +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Codrut Stoicescu + (codrut.stoicescu@gmail.com) +
+
2023-07-27 11:53:09
 +
+

Hello everyone! I’m part of a team trying to integrate OpenLineage and Marquez with multiple tools in our ecosystem. Integration with Spark and Iceberg was fairly easy with the listener you guys developed. We are now trying to integrate with Ray and we are having some trouble there. I was wondering if anybody has tried any work in that direction, so we can chat and exchange ideas. Thank you!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-07-27 14:47:18
 +
+

*Thread Reply:* This is the first I’ve heard of someone trying to do this, but others have tried getting lineage from pandas. There isn’t support for this currently, but this thread contains a link to an issue that might be helpful: https://openlineage.slack.com/archives/C01CK9T7HKR/p1689850134978429?thread_ts=1689688067.729469&cid=C01CK9T7HKR.

+
+ + +
+ + + } + + Paweł Leszczyński + (https://openlineage.slack.com/team/U02MK6YNAQ5) +
+ + + +
+
+ <p>I think we don't have pandas support so far. So, if one uses pandas to read local files on disk, then perhaps Openlineage (OL) has little sense to do. There is an old pandas issues in our backlog (over 2 years old) -&gt; <a href="https://github.com/OpenLineage/OpenLineage/issues/108">https://github.com/OpenLineage/OpenLineage/issues/108</a></p> + +<p>Surely one can use use python OL client to create manully events and send them to MQZ, which may be less convenient (<a href="https://github.com/OpenLineage/OpenLineage/tree/main/client/python">https://github.com/OpenLineage/OpenLineage/tree/main/client/python</a>)</p> + +<p>Anyway, we would like to know what's you usecase? this would be super helpful in understanding why OL &amp; pandas integration may be useful.</p> +
+ + + + + + +
Original URL: https://openlineage.slack.com/archives/C01CK9T7HKR/p1689850134978429?thread_ts=1689688067.729469&amp;cid=C01CK9T7HKR
+ + +
+ + Thread in Slack Conversation +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Codrut Stoicescu + (codrut.stoicescu@gmail.com) +
+
2023-07-28 02:10:14
 +
+

*Thread Reply:* Thank you for your response. We have implemented the “manual way” of emitting events with python OL client. We are now looking for a more automated way, so that updates to the scripts that run in Ray are minimal to none

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-07-28 13:03:43
 +
+

*Thread Reply:* If you're actively using Ray, then you know way more about it than me, or probably any other OL contributor 🙂 +I don't know how it works or is deployed, but I would recommend checking if there's robust way of being notified in the runtime about processing occuring there.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Codrut Stoicescu + (codrut.stoicescu@gmail.com) +
+
2023-07-31 12:17:07
 +
+

*Thread Reply:* Thank you for the tip. That’s the kind of details I’m looking for, but couldn’t find yet

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tereza Trojanová + (tereza.trojanova@revolt.bi) +
+
2023-07-28 09:20:34
 +
+

Hi, does anyone have experience integrating OpenLineage and Marquez with Keboola? I am new to OpenLineage and struggling with the KBC component configuration.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-07-28 10:53:35
 +
+

*Thread Reply:* @Martin Fiser can you share any resources or pointers that might be helpful?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Martin Fiser + (fisa@keboola.com) +
+
2023-08-21 19:17:17
 +
+

*Thread Reply:* Hi, apologies - vacation period has hit m. However here are the resources:

+ +

API endpoint: +https://app.swaggerhub.com/apis-docs/keboola/job-queue-api/1.3.4#/Jobs/getJobOpenApiLineage|job-queue-api | 1.3.4 | keboola | SwaggerHub +Dedicated component to push data into openlineage(Marquez instance): +https://components.keboola.com/components/keboola.wr-openlineage|OpenLineage data destination | Keboola Developer Portal

+ + + +
+ 🙌 Michael Robinson +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Damien Hawes + (damien.hawes@booking.com) +
+
2023-07-31 12:32:22
 +
+

Hi folks. I'm looking to find the complete spec in openapi format. For example, if I want to find the complete spec of 1.0.5 , where would I find that? I've looked here: https://openlineage.io/apidocs/openapi/ however when I download the spec, things are missing, specifically the facets. This makes it difficult to generate clients / backend interfaces from the (limited) openapi spec.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Silvia Pina + (silviampina@gmail.com) +
+
2023-08-01 05:14:58
 +
+

*Thread Reply:* +1, I could also really use this!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Silvia Pina + (silviampina@gmail.com) +
+
2023-08-01 05:27:34
 +
+

*Thread Reply:* Found a way: you download it as json in the above link (“Download OpenAPI specification”), then if you copy paste it to editor.swagger.io it asks f you want to convert to yaml :)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Damien Hawes + (damien.hawes@booking.com) +
+
2023-08-01 10:25:49
 +
+

*Thread Reply:* Whilst that works, it isn't complete. The issue is that the "facets" are not resolved. Exploring the website repository (https://github.com/OpenLineage/website/tree/main/static/spec) shows that facets aren't published alongside the spec, beyond 1.0.1 - which means its hard to know which revisions of the facets belong to which version of the spec.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Silvia Pina + (silviampina@gmail.com) +
+
2023-08-01 10:26:54
 +
+

*Thread Reply:* Good point! Would be good if we could clarify how to get the full spec, in that case

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Damien Hawes + (damien.hawes@booking.com) +
+
2023-08-01 10:30:57
 +
+

*Thread Reply:* Granted. If the spec follows backwards compatible evolution rules, then this shouldn't be a problem, i.e., new fields must be optional, you can not remove existing fields, you can not modify existing fields, etc.

+ + + +
+ 🙌 Silvia Pina +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-01 12:15:22
 +
+

*Thread Reply:* We don't have facets with newer version than 1.1.0

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-01 12:15:56
 +
+

*Thread Reply:* @Damien Hawes we've moved to merge docs and website repos here: https://github.com/OpenLineage/docs

+
+ + + + +
OpenLineage/docs
+
+ Documentation for OpenLineage +
+ +
+
Website
+ <https://openlineage.io/docs> +
+ +
+
Stars
+ 5 +
+ + + + + + + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-01 12:18:23
 +
+

*Thread Reply:* > Would be good if we could clarify how to get the full spec, in that case +Is using https://github.com/OpenLineage/OpenLineage/tree/main/spec not enough? We have separate files with facets definition to be able to evolve them separetely from main spec

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Damien Hawes + (damien.hawes@booking.com) +
+
2023-08-02 04:53:03
 +
+

*Thread Reply:* @Maciej Obuchowski - thanks for your input. I understand the desire to want to evolve the facets independently from the main spec, yet I keep running into a mental wall.

+ +

If I say, 'My application is compatible with OpenLineage 1.0.5' - what does that mean exactly? Does it mean that I am at least compatible with the base definition of RunEvent and its nested components, but not facets?

+ +

That's what I'm finding difficult to wrap my head around. Right now, I can not define (for my own sake and the sake of my org) what 'OpenLineage 1.0.5' means.

+ +

When I read the Marquez source code, I see that they state they implement 1.0.5, but again, it isn't clear what that completely entails.

+ +

I hope I am making sense.

+ + + +
+ 👍 Silvia Pina +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Damien Hawes + (damien.hawes@booking.com) +
+
2023-08-02 04:56:36
 +
+

*Thread Reply:* If I approach this from a conventional software engineering standpoint, where I provide a library to my consumers. The library has a version associated with it, and that version encompasses all the objects located within that particular library. If I release a new version of my library, it implies that some form of evolution has happened. Whether it is a bug fix, a documentation change, or evolving the API of my objects it means something has changed and the new version is there to indicate that.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-02 04:56:53
 +
+

*Thread Reply:* Yes - it means you can read and understand base spec. Facets are completely optional - reading them might provide you additional information, but you as a event consumer need to define what you do with them. Basically, the needs can be very different between consumers, spec should not define behavior of a consumer.

+ + + +
+ 🙌 Silvia Pina +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Damien Hawes + (damien.hawes@booking.com) +
+
2023-08-02 05:01:26
 +
+

*Thread Reply:* OK. Thanks for the clarification. That clears things up for me.

+ + + +
+ 👍 Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-07-31 16:42:48
 +
+

This month’s issue of OpenLineage News was just sent out. Please to get it directly in your inbox each month!

+
+
apache.us14.list-manage.com
+ + + +
OpenLineage Project
+
+ OpenLineage Project Email Forms +
+ + + + + + +
Original URL: https://openlineage.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&amp;id=785a308136&amp;e=ce16eef4ef
+ + + +
+ +
+
+ + + + + + +
+ + +
+ 👍 Ross Turk, Maciej Obuchowski, Shirley Lu +
+ +
+ 🎉 Harel Shein +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-01 12:35:22
 +
+

Hello, I request OpenLineage release, especially for two things: +• Snowflake/HTTP/Airflow bugfix: https://github.com/OpenLineage/OpenLineage/pull/2025 +• Spec: removing refs from core: https://github.com/OpenLineage/OpenLineage/pull/1997 +Three approvals from committers will authorize release. @Michael Robinson

+ + + +
+ ➕ Jakub Dardziński, Harel Shein, Michael Robinson, George Polychronopoulos, Willy Lulciuc, Shirley Lu +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-08-01 13:26:30
 +
+

*Thread Reply:* Thanks, @Maciej Obuchowski

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-08-01 15:43:00
 +
+

*Thread Reply:* Thanks, all. The release is authorized and will be initiated within two business days.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-08-01 16:42:32
 +
+

@channel +We released OpenLineage 1.0.0, featuring static lineage capability! +Added: +• Airflow: convert lineage from legacy File definition #2006 @Maciej Obuchowski +Removed: +• Spec: remove facet ref from core #1997 @JDarDagran +Changed +• Airflow: change log level to DEBUG when extractor isn’t found #2012 @kaxil +• Airflow: make sure we cannot fail in thread despite direct execution #2010 @Maciej Obuchowski +Plus test improvements, docs changes, bug fixes and more. +*See prior releases for additional changes related to static lineage. +Thanks to all the contributors, including new contributors @kaxil and @Mars Lan! +*Release: https://github.com/OpenLineage/OpenLineage/releases/tag/1.0.0 +Changelog: https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md +Commit history: https://github.com/OpenLineage/OpenLineage/compare/0.30.1...1.0.0 +Maven: https://oss.sonatype.org/#nexus-search;quick~openlineage +PyPI: https://pypi.org/project/openlineage-python/

+ + + +
+ 🙌 Julian LaNeve, Bernat Gabor, Maciej Obuchowski, Peter Hicks, Ross Turk, Harel Shein, Willy Lulciuc, Paweł Leszczyński, Peter Hicks +
+ +
+ 🥳 Julian LaNeve, alexandre bergere, Maciej Obuchowski, Peter Hicks, Juan Manuel Cappi, Ross Turk, Harel Shein, Paweł Leszczyński, Peter Hicks +
+ +
+ 🚀 alexandre bergere, Peter Hicks, Ross Turk, Harel Shein, Paweł Leszczyński, Peter Hicks +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Juan Luis Cano Rodríguez + (juan_luis_cano@mckinsey.com) +
+
2023-08-02 08:51:57
 +
+

hi folks! so happy to see that static lineage is making its way through OL. one question: is the OpenAPI spec up to date? https://openlineage.io/apidocs/openapi/ IIUC, proposal 1837 says that JobEvent and DatasetEvent can be emitted independently from RunEvents now, but it's not clear how this affected the spec.

+ +

I see the Python client https://pypi.org/project/openlineage-python/1.0.0/ includes these changes already, so I assume I can go ahead and use it already? (I'm also keeping tabs on https://github.com/MarquezProject/marquez/issues/2544)

+
+ + + + +
#2544 Proposal for static lineage support
+
+ <p>As of PR <a href="https://github.com/MarquezProject/marquez/pull/2495">#2495</a>, static lineage events are collected, but not stored. Let's write a proposal outlining on how static lineage events will:</p> + +<p>• be validated and stored in the <code>lineage_events</code> table +• change the core model logic; more specifically, the relationship between jobs, job versions, and runs +• how static lineage will be represented and queried via REST API</p> +
+ +
+
Assignees
+ <a href="https://github.com/wslulciuc">@wslulciuc</a> +
+ + + + + + + +
+ + <a href="https://github.com/MarquezProject/marquez">MarquezProject/marquez</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-02 10:09:33
 +
+

*Thread Reply:* I think the apidocs are not up to date 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-02 10:09:43
 +
+

*Thread Reply:* https://openlineage.io/spec/2-0-2/OpenLineage.json has the newest spec

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Juan Luis Cano Rodríguez + (juan_luis_cano@mckinsey.com) +
+
2023-08-02 10:44:23
 +
+

*Thread Reply:* thanks for the pointer @Maciej Obuchowski

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-08-02 10:49:17
 +
+

*Thread Reply:* Also working on updating the apidocs

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-08-02 11:21:14
 +
+

*Thread Reply:* The API docs are now up to date @Juan Luis Cano Rodríguez! Thank you for raising this issue.

+ + + +
+ 🙌:skin_tone_3: Juan Luis Cano Rodríguez +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-08-02 12:58:15
 +
+

@channel +If you can, please join us in San Francisco for a meetup at Astronomer on August 30th at 5:30 PM PT. +On the agenda: a presentation by special guest @John Lukenoff plus updates on the Airflow Provider, static lineage, and more. +Food will be provided, and all are welcome. +Please https://www.meetup.com/meetup-group-bnfqymxe/events/295195280/?utmmedium=referral&utmcampaign=share-btnsavedeventssharemodal&utmsource=link|RSVP to let us know you’re coming.

+
+
Meetup
+ + + +
OpenLineage Meetup @ Astronomer, Wed, Aug 30, 2023, 5:30 PM | Meetup
+
+ Data engineers and pipeline managers know that producing data lineage – end-to-end pipeline metadata instrumented at runtime or parsed at design time – is a heavy lift with +
+ + + + + + + + +
Original URL: https://www.meetup.com/meetup-group-bnfqymxe/events/295195280/?utm_medium=referral&amp;utm_campaign=share-btn_savedevents_share_modal&amp;utm_source=link
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Zahi Fail + (zahi.fail@gmail.com) +
+
2023-08-03 03:18:08
 +
+

Hey, I hope this is the right channel for this kind of question - I’m running a tests to integrate Airflow (2.4.3) with Marquez (Openlineage 0.30.1). Currently, I’m testing the postgres operator and for some reason queries like “Copy” and “Unload” are being sent as events, but doesn’t appear in the graph. Any idea how to solve it?

+ +

You can see attached

+ +
  1. The graph of an airflow DAG with all the tasks beside the copy and unload.
  2. The graph with the unload task that isn’t connected to the other flow.
    3. +
+ +
+
+ + + + + + +
+
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-03 05:36:04
 +
+

*Thread Reply:* I think our underlying SQL parser does not hancle the Postgres versions of those queries

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-03 05:36:14
 +
+

*Thread Reply:* Can you post the (anonymized?) queries?

+ + + +
+ 👍 Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Zahi Fail + (zahi.fail@gmail.com) +
+
2023-08-03 07:03:09
 +
+

*Thread Reply:* for example

+ +
copy bi.marquez_test_2 from '******' iam_role '**********' delimiter as '^' gzi
+
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-07 13:35:30
 +
+

*Thread Reply:* @Zahi Fail iam_role suggests you want redshift version of this supported, not Postgres one right?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Zahi Fail + (zahi.fail@gmail.com) +
+
2023-08-08 04:04:35
 +
+

*Thread Reply:* @Maciej Obuchowski hey, actually I tried both Postgres and Redshift to S3 operators. +Both of them sent a new event through OL to Marquez, and still wasn’t part of the entire flow.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Athitya Kumar + (athityakumar@gmail.com) +
+
2023-08-04 01:40:15
 +
+

Hey team! 👋

+ +

We were exploring open-lineage and had a couple of questions:

+ +
  1. Does open-lineage support presto-sql?
  2. Do we have any docs/benchmarks on query coverage (inner joins, subqueries, etc) & source/sink coverage (spark.read from JDBC, Files etc) for spark-sql?
  3. Can someone point to the code where we currently parse the input/output facets from the spark integration (like sql queries / transformations) and if it's extendable?
    4. +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-08-04 02:17:19
 +
+

*Thread Reply:* Hey @Athitya Kumar,

+ +
  1. For parsing SQL queries, we're using sqlparser-rs (https://github.com/sqlparser-rs/sqlparser-rs) which already has great coverage of sql syntax and supports different dialects. it's open source project and we already did contribute to it for snowflake dialect.
  2. We don't have such a benchmark, but if you like, you could contribute and help us providing such. We do support joins, subqueries, iceberg and delta tables, jdbc for Spark and much more. Everything we do support, is covered in our tests.
  3. Not sure if got it properly. Marquez is our reference backend implementation which parses all the facets and stores them in relational db in a relational manner (facets, jobs, datasets and runs in separate tables).
    4. +
+
+ + + + +
sqlparser-rs/sqlparser-rs
+
+ Extensible SQL Lexer and Parser for Rust +
+ +
+
Stars
+ 1956 +
+ +
+
Language
+ Rust +
+ + + + + + + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Athitya Kumar + (athityakumar@gmail.com) +
+
2023-08-04 02:29:53
 +
+

*Thread Reply:* For (3), I was referring to where we call the sqlparser-rs in our spark-openlineage event listener / integration; and how customising/improving them would look like

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-08-04 02:37:20
 +
+

*Thread Reply:* sqlparser-rs is a rust libary and we bundle it within iface-java (https://github.com/OpenLineage/OpenLineage/blob/main/integration/sql/iface-java/src/main/java/io/openlineage/sql/SqlMeta.java). It's capable of extracting input/output datasets, column lineage information from SQL

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/integration/sql/iface-java/src/main/java/io/openlineage/sql/SqlMeta.java | SqlMeta.java>
+
+ <pre><code>/** +/** Copyright 2018-2023 contributors to the OpenLineage project +/** SPDX-License-Identifier: Apache-2.0 +**/ + +package io.openlineage.sql; + +import java.util.Arrays; +import java.util.List; +import org.apache.commons.lang3.builder.HashCodeBuilder; + +public class SqlMeta { + private final List&lt;DbTableMeta&gt; inTables; + private final List&lt;DbTableMeta&gt; outTables; + private final List&lt;ColumnLineage&gt; columnLineage; + private final List&lt;ExtractionError&gt; errors; + + public SqlMeta( + List&lt;DbTableMeta&gt; in, + List&lt;DbTableMeta&gt; out, + List&lt;ColumnLineage&gt; columnLineage, + List&lt;ExtractionError&gt; errors) { + this.inTables = in; + this.outTables = out; + this.columnLineage = columnLineage; + this.errors = errors; + } + + public List&lt;DbTableMeta&gt; inTables() { + return inTables; + } + + public List&lt;DbTableMeta&gt; outTables() { + return outTables; + } + + public List&lt;ColumnLineage&gt; columnLineage() { + return columnLineage; + } + + public List&lt;ExtractionError&gt; errors() { + return errors; + } + + @Override + public String toString() { + return String.format( + "{\"inTables\": %s, \"outTables\": %s, \"columnLineage\": %s, \"errors\": %s}", + Arrays.toString(inTables.toArray()), + Arrays.toString(outTables.toArray()), + Arrays.toString(columnLineage.toArray()), + Arrays.toString(errors.toArray())); + } + + @Override + public boolean equals(Object o) { + if (o == this) { + return true; + } + + if (!(o instanceof SqlMeta)) { + return false; + } + + SqlMeta other = (SqlMeta) o; + return other.inTables.equals(inTables) + &amp;&amp; other.outTables.equals(outTables) + &amp;&amp; other.columnLineage.equals(columnLineage) + &amp;&amp; other.errors.equals(errors); + } + + @Override + public int hashCode() { + return new HashCodeBuilder() + .append(inTables) + .append(outTables) + .append(columnLineage) + .append(errors) + .toHashCode(); + } +} + +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-08-04 02:40:02
 +
+

*Thread Reply:* and this is Spark code that extracts it from JdbcRelation -> https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/shared/src/[…]ge/spark/agent/lifecycle/plan/handlers/JdbcRelationHandler.java

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/shared/src/main/java/io/openlineage/spark/agent/lifecycle/plan/handlers/JdbcRelationHandler.java | JdbcRelationHandler.java>
+
+ <pre><code>/** +/** Copyright 2018-2023 contributors to the OpenLineage project +/** SPDX-License-Identifier: Apache-2.0 +**/ + +package io.openlineage.spark.agent.lifecycle.plan.handlers; + +import io.openlineage.client.OpenLineage; +import io.openlineage.spark.agent.util.JdbcUtils; +import io.openlineage.spark.api.DatasetFactory; +import io.openlineage.sql.ColumnMeta; +import io.openlineage.sql.DbTableMeta; +import io.openlineage.sql.SqlMeta; +import java.util.Collections; +import java.util.List; +import java.util.Optional; +import java.util.stream.Collectors; +import org.apache.spark.sql.execution.datasources.LogicalRelation; +import org.apache.spark.sql.execution.datasources.jdbc.JDBCRelation; +import org.apache.spark.sql.types.StructField; +import org.apache.spark.sql.types.StructType; + +public class JdbcRelationHandler&lt;D extends OpenLineage.Dataset&gt; { + + private final DatasetFactory&lt;D&gt; datasetFactory; + + public JdbcRelationHandler(DatasetFactory&lt;D&gt; datasetFactory) { + this.datasetFactory = datasetFactory; + } + + public List&lt;D&gt; handleRelation(LogicalRelation x) { + // strip the jdbc: prefix from the url. this leaves us with a url like + // postgresql://&lt;hostname&gt;:&lt;port&gt;/&lt;database_name&gt;?params + // we don't parse the URI here because different drivers use different + // connection + // formats that aren't always amenable to how Java parses URIs. E.g., the oracle + // driver format looks like oracle:&lt;drivertype&gt;:&lt;user&gt;/&lt;password&gt;@&lt;database&gt; + // whereas postgres, mysql, and sqlserver use the &lt;scheme://hostname:port/db&gt; + // format. + JDBCRelation relation = (JDBCRelation) x.relation(); + String url = JdbcUtils.sanitizeJdbcUrl(relation.jdbcOptions().url()); + return getDatasets(relation, url); + } + + public List&lt;D&gt; getDatasets(JDBCRelation relation, String url) { + Optional&lt;SqlMeta&gt; sqlMeta = JdbcUtils.extractQueryFromSpark(relation); + if (!sqlMeta.isPresent()) { + return Collections.emptyList(); + } + if (sqlMeta.get().columnLineage().isEmpty()) { + return Collections.singletonList( + datasetFactory.getDataset( + sqlMeta.get().inTables().get(0).qualifiedName(), url, relation.schema())); + } + return sqlMeta.get().inTables().stream() + .map( + dbtm -&gt; + datasetFactory.getDataset( + dbtm.qualifiedName(), + url, + generateJDBCSchema(dbtm, relation.schema(), sqlMeta.get()))) + .collect(Collectors.toList()); + } + + private static StructType generateJDBCSchema( + DbTableMeta origin, StructType schema, SqlMeta sqlMeta) { + StructType originSchema = new StructType(); + for (StructField f : schema.fields()) { + List&lt;ColumnMeta&gt; fields = + sqlMeta.columnLineage().stream() + .filter(cl -&gt; cl.descendant().name().equals(f.name())) + .flatMap( + cl -&gt; + cl.lineage().stream() + .filter( + cm -&gt; cm.origin().isPresent() &amp;&amp; cm.origin().get().equals(origin))) + .collect(Collectors.toList()); + for (ColumnMeta cm : fields) { + originSchema = originSchema.add(cm.name(), f.dataType()); + } + } + return originSchema; + } +} + +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-04 04:08:53
 +
+

*Thread Reply:* I think 3 question relates generally to Spark SQL handling, rather than handling JDBC connections inside Spark, right?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Athitya Kumar + (athityakumar@gmail.com) +
+
2023-08-04 04:24:57
 +
+

*Thread Reply:* Yup, both actually. Related to getting the JDBC connection info in the input/output facet, as well as spark-sql queries we do on that JDBC connection

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-04 06:00:17
 +
+

*Thread Reply:* For Spark SQL - it's translated to Spark's internal query LogicalPlan. We take that plan, and process it's nodes. From root node we can take output dataset, from leaf nodes we can take input datasets, and inside internal nodes we track columns to extract column-level lineage. We express those (table-level) operations by implementing classes like QueryPlanVisitor

+ +

You can extend that, for example for additional types of nodes that we don't support by implementing your own QueryPlanVisitor, and then implementing OpenLineageEventHandlerFactory and packaging this into a .jar deployed alongside OpenLineage jar - this would be loaded by us using Java's ServiceLoader .

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/a9809c3e3ccb959c93af8edd6e3358514a8edd2e/integration/spark/shared/src/main/java/io/openlineage/spark/api/QueryPlanVisitor.java | QueryPlanVisitor.java>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/3e5ffee320fc52a8b287d99c678a89335c9c468c/integration/spark/shared/src/main/java/io/openlineage/spark/api/OpenLineageEventHandlerFactory.java | OpenLineageEventHandlerFactory.java>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+ 👍 Kiran Hiremath +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Athitya Kumar + (athityakumar@gmail.com) +
+
2023-08-08 05:06:07
 +
+

*Thread Reply:* @Maciej Obuchowski @Paweł Leszczyński - Thanks for your responses! I had a follow-up query regarding the sqlparser-rs that's used internally by open-lineage: we see that these are the SQL dialects supported by sqlparser-rs here doesn't include spark-sql / presto-sql dialects which means they'd fallback to generic dialect:

+ +

"--ansi" =&gt; Box::new(AnsiDialect {}), +"--bigquery" =&gt; Box::new(BigQueryDialect {}), +"--postgres" =&gt; Box::new(PostgreSqlDialect {}), +"--ms" =&gt; Box::new(MsSqlDialect {}), +"--mysql" =&gt; Box::new(MySqlDialect {}), +"--snowflake" =&gt; Box::new(SnowflakeDialect {}), +"--hive" =&gt; Box::new(HiveDialect {}), +"--redshift" =&gt; Box::new(RedshiftSqlDialect {}), +"--clickhouse" =&gt; Box::new(ClickHouseDialect {}), +"--duckdb" =&gt; Box::new(DuckDbDialect {}), +"--generic" | "" =&gt; Box::new(GenericDialect {}), +Any idea on how much coverage generic dialect provides for spark-sql / how different they are etc?

+
+ + + + +
<https://github.com/sqlparser-rs/sqlparser-rs/blob/173a6db8189a0f1bb9896117c86374c4e691f0f3/examples/cli.rs | cli.rs>
+
+ <pre><code> let dialect: Box&lt;dyn Dialect&gt; = match std::env::args().nth(2).unwrap_or_default().as_ref() { +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/sqlparser-rs/sqlparser-rs">sqlparser-rs/sqlparser-rs</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-08-08 05:21:32
 +
+

*Thread Reply:* spark-sql integration is based on spark LogicalPlan's tree. Extracting input/output datasets from tree nodes which is more detailed than sql parsing

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-08 07:04:52
 +
+

*Thread Reply:* I think presto/trino dialect is very standard - there shouldn't be any problems with regular queries

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Athitya Kumar + (athityakumar@gmail.com) +
+
2023-08-08 11:19:53
 +
+

*Thread Reply:* @Paweł Leszczyński - Got it, and would you be able to point me to where within the openlineage-spark integration do we:

+ +
  1. provide the Spark Logical Plan / query to sqlparser-rs
  2. get the output of sqlparser-rs (parsed query AST) & stitch back the inputs/outputs in the open-lineage events?
    3. +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Athitya Kumar + (athityakumar@gmail.com) +
+
2023-08-08 12:09:06
 +
+

*Thread Reply:* For example, we'd like to understand which dialectname of sqlparser-rs would be used in which scenario by open-lineage and what're the interactions b/w open-lineage & sqlparser-rs

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Athitya Kumar + (athityakumar@gmail.com) +
+
2023-08-09 12:18:47
 +
+

*Thread Reply:* @Paweł Leszczyński - Incase you missed the above messages ^

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-08-10 03:31:32
 +
+

*Thread Reply:* Sqlparser-rs is used within Spark integration only for spark jdbc queries (queries to external databases). That's the only scenario. For spark.sql(...) , instead of SQL parsing, we rely on a logical plan of a job and extract information from it. For jdbc queries, that user sqlparser-rs, dialect is extracted from url: +https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/shared/src/main/java/io/openlineage/spark/agent/util/JdbcUtils.java#L69

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/shared/src/main/java/io/openlineage/spark/agent/util/JdbcUtils.java | JdbcUtils.java>
+
+ <pre><code> private static String extractDialectFromJdbcUrl(String jdbcUrl) { +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+ 👍 Athitya Kumar +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
nivethika R + (nivethikar8@gmail.com) +
+
2023-08-06 07:16:53
 +
+

Hi.. Is column lineage available for spark version 2.4.0?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-06 17:25:31
 +
+

*Thread Reply:* No, it's not.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
nivethika R + (nivethikar8@gmail.com) +
+
2023-08-06 23:53:17
 +
+

*Thread Reply:* Is it only available for spark version 3+?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-07 04:53:41
 +
+

*Thread Reply:* Yes

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
GitHubOpenLineageIssues + (githubopenlineageissues@gmail.com) +
+
2023-08-07 11:18:25
 +
+

Hi, Will really appreciate if I can learn how community have been able to harness spark integration. In our testing where a spark application writes to S3 multiple times (different location), OL generates the same job name for all writes (namepsacename.executeinsertintohadoopfsrelation_command ) rendering the OL graph final output less helpful. Say for example if I have series of transformation/writes 5 times , in Lineage graph we are just seeing last 1. There is an open bug and hopefully will be resolved soon.

+ +

Curious how much is adoption of OL spark integration in presence of that bug, as generating same name for a job makes it less usable for anything other than trivial one output application.

+ +

Example from 2 write application +EXPECTED : first produce weather dataset and the subsequent produce weather40. (generated/mocked using 2 spark app). (1st image) +ACTUAL OL : weather40. see only last one. (2nd image)

+ +

Will really appreciate community guidance as in how successful they have been in utilizing spark integration (vanilla not Databricks) . Thank you

+ +

Expected. vs Actual.

+ +
+
+ + + + + + +
+
+
+ + + + + + +
+ + +
+
+
+
+ + + + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-08-07 11:30:00
 +
+

@channel +This month’s TSC meeting is this Thursday, August 10th at 10:00 a.m. PT. On the tentative agenda: +• announcements +• recent releases +• Airflow provider progress update +• OpenLineage 1.0 overview +• open discussion +• more (TBA) +More info and the meeting link can be found on the website. All are welcome! Also, feel free to reply or DM me with discussion topics, agenda items, etc.

+
+
openlineage.io
+ + + +
TSC Meetings | OpenLineage
+
+ The OpenLineage Technical Steering Committee meets monthly, and is open to all. +
+ + + + + + +
Original URL: https://openlineage.io/meetings/
+ + + +
+ + + +
+ 👍 Maciej Obuchowski, Athitya Kumar, Anirudh Shrinivason, Paweł Leszczyński +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
추호관 + (hogan.chu@toss.im) +
+
2023-08-08 04:39:45
 +
+

I can’t see output when saveAsTable 100+ columns in spark. Any help or ideas for issue? Really thanks.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-08 04:59:23
 +
+

*Thread Reply:* Does this work with similar jobs, but with small amount of columns?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
추호관 + (hogan.chu@toss.im) +
+
2023-08-08 05:12:52
 +
+

*Thread Reply:* thanks for reply @Maciej Obuchowski yes it works for small amount of columns +but not work in big amount of columns

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-08 05:14:04
 +
+

*Thread Reply:* one more question: how much data the jobs approximately process and how long does the execution take?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
추호관 + (hogan.chu@toss.im) +
+
2023-08-08 05:14:54
 +
+

*Thread Reply:* ah… it’s like 20 min ~ 30 min various +data size is like 2000,0000 rows with columns 100 ~ 1000

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-08-08 05:15:17
 +
+

*Thread Reply:* that's interesting. we could prepare integration test for that. 100 cols shouldn't make a difference

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
추호관 + (hogan.chu@toss.im) +
+
2023-08-08 05:15:37
 +
+

*Thread Reply:* honestly sorry for typo its 1000 columns

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
추호관 + (hogan.chu@toss.im) +
+
2023-08-08 05:15:44
 +
+

*Thread Reply:* pivoting features

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
추호관 + (hogan.chu@toss.im) +
+
2023-08-08 05:16:09
 +
+

*Thread Reply:* i check it works good for small numbers of columns

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-08 05:16:39
 +
+

*Thread Reply:* if it's 1000, then maybe we're over event size - event is too large and backend can't accept that

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-08 05:17:06
 +
+

*Thread Reply:* maybe debug logs could tell us something

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
추호관 + (hogan.chu@toss.im) +
+
2023-08-08 05:19:27
 +
+

*Thread Reply:* i’ll do spark.sparkContext.setLogLevel("DEBUG") ing

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-08-08 05:19:30
 +
+

*Thread Reply:* are there any errors in the logs? perhaps pivoting uses contains nodes in SparkPlan that we don't support yet

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-08-08 05:19:52
 +
+

*Thread Reply:* did you check pivoting that results in less columns?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-08 05:20:33
 +
+

*Thread Reply:* @추호관 would also be good to disable logicalPlan facet: +spark.openlineage.facets.disabled: [spark_unknown;spark.logicalPlan] +in spark conf

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
추호관 + (hogan.chu@toss.im) +
+
2023-08-08 05:23:40
 +
+

*Thread Reply:* got it can’t we do in python config +.config("spark.dynamicAllocation.enabled", "true") \ +.config("spark.dynamicAllocation.initialExecutors", "5") \ +.config("spark.openlineage.facets.disabled", [spark_unknown;spark.logicalPlan]

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-08-08 05:24:31
 +
+

*Thread Reply:* .config("spark.dynamicAllocation.enabled", "true") \ +.config("spark.dynamicAllocation.initialExecutors", "5") \ +.config("spark.openlineage.facets.disabled", "[spark_unknown;spark.logicalPlan]"

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
추호관 + (hogan.chu@toss.im) +
+
2023-08-08 05:24:42
 +
+

*Thread Reply:* ah.. string got it

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
추호관 + (hogan.chu@toss.im) +
+
2023-08-08 05:36:03
 +
+

*Thread Reply:* ah… there are no errors nor debug level issue successfully Registered listener ìo.openlineage.spark.agent.OpenLineageSparkListener

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
추호관 + (hogan.chu@toss.im) +
+
2023-08-08 05:39:40
 +
+

*Thread Reply: maybe df.groupBy(some column).pivot(some_column).agg(*agg_cols) is not supported

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
추호관 + (hogan.chu@toss.im) +
+
2023-08-08 05:43:44
 +
+

*Thread Reply:* oh.. interesting spark.openlineage.facets.disabled this option gives me output when eventType is START +“eventType”: “START” +“outputs”: [ +… +columns +…. +]

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
추호관 + (hogan.chu@toss.im) +
+
2023-08-08 05:54:13
 +
+

*Thread Reply:* Yes +"spark.openlineage.facets.disabled", "[spark_unknown;spark.logicalPlan]" <- this option gives output when eventType is START but not give output bunches of columns when that config is not set

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-08-08 05:55:18
 +
+

*Thread Reply:* this option prevents logicalPlan being serialized and sent as a part of Openlineage event which included in one of the facets

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-08-08 05:56:12
 +
+

*Thread Reply:* possibly, serializing logicalPlans, in case of pivots, leads to size of the events that are not acceptable

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
추호관 + (hogan.chu@toss.im) +
+
2023-08-08 05:57:56
 +
+

*Thread Reply:* Ah… so you mean pivot makes serializing logical plan not availble for generating event because of size. +and disable logical plan with not serializing make availabe to generate event cuz not serialiize logical plan made by pivot

+ +

Can we overcome this

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-08-08 05:58:48
 +
+

*Thread Reply:* we've seen such issues for some plans some time ago

+ + + +
+ 🙌 추호관 +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
추호관 + (hogan.chu@toss.im) +
+
2023-08-08 05:59:29
 +
+

*Thread Reply:* oh…. how did you solve it?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-08-08 05:59:32
 +
+

*Thread Reply:* https://github.com/OpenLineage/OpenLineage/blob/c3a5211f919c01870a7f79f48588177a9b[…]io/openlineage/spark/agent/lifecycle/LogicalPlanSerializer.java

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/c3a5211f919c01870a7f79f48588177a9b687fdf/integration/spark/shared/src/main/java/io/openlineage/spark/agent/lifecycle/LogicalPlanSerializer.java | LogicalPlanSerializer.java>
+
+ <pre><code> @JsonIgnoreProperties({ +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+ 🙌 추호관 +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-08-08 05:59:51
 +
+

*Thread Reply:* by excluding some properties from plan to be serialized

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-08-08 06:01:14
 +
+

*Thread Reply:* here https://github.com/OpenLineage/OpenLineage/blob/c3a5211f919c01870a7f79f48588177a9b[…]io/openlineage/spark/agent/lifecycle/LogicalPlanSerializer.java we exclude certain classes

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/c3a5211f919c01870a7f79f48588177a9b687fdf/integration/spark/shared/src/main/java/io/openlineage/spark/agent/lifecycle/LogicalPlanSerializer.java | LogicalPlanSerializer.java>
+
+ <pre><code> .put(PythonRDD.class, PythonRDDMixin.class) +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+ 🙌 추호관 +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
추호관 + (hogan.chu@toss.im) +
+
2023-08-08 06:02:00
 +
+

*Thread Reply:* AH…. excluded properties cause ignore logical plan’s of pivointing

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-08-08 06:08:25
 +
+

*Thread Reply:* you can start with writing a failing test here -> https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/app/src/tes[…]/openlineage/spark/agent/lifecycle/SparkReadWriteIntegTest.java

+ +

then you can try to debug logical plan trying to find out what should be excluded from it when it's being serialized. Even, if you find this difficult, a failing integration test is super helpful to let others help you in that.

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/app/src/test/java/io/openlineage/spark/agent/lifecycle/SparkReadWriteIntegTest.java | SparkReadWriteIntegTest.java>
+
+ <p>``` +/<em>* +/</em>* Copyright 2018-2023 contributors to the OpenLineage project +/<em>* SPDX-License-Identifier: Apache-2.0 +*</em>/</p> + +<p>package io.openlineage.spark.agent.lifecycle;</p> + +<p>import static io.openlineage.client.OpenLineage.RunEvent; +import static org.assertj.core.api.Assertions.assertThat; +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertNotNull; +import static org.junit.jupiter.api.Assertions.assertTrue; +import static org.mockito.ArgumentMatchers.eq; +import static org.mockito.Mockito.atLeast; +import static org.mockito.Mockito.reset; +import static org.mockito.Mockito.when; +import static org.mockito.internal.verification.VerificationModeFactory.times;</p> + +<p>import com.fasterxml.jackson.core.JsonGenerator; +import com.fasterxml.jackson.databind.ObjectMapper; +import com.google.cloud.bigquery.MockBigQueryRelationProvider; +import com.google.cloud.bigquery.connector.common.BigQueryUtil; +import com.google.cloud.spark.bigquery.repackaged.com.google.cloud.bigquery.BigQuery; +import com.google.cloud.spark.bigquery.repackaged.com.google.cloud.bigquery.Field; +import com.google.cloud.spark.bigquery.repackaged.com.google.cloud.bigquery.Schema; +import com.google.cloud.spark.bigquery.repackaged.com.google.cloud.bigquery.StandardSQLTypeName; +import com.google.cloud.spark.bigquery.repackaged.com.google.cloud.bigquery.StandardTableDefinition; +import com.google.cloud.spark.bigquery.repackaged.com.google.cloud.bigquery.TableId; +import com.google.cloud.spark.bigquery.repackaged.com.google.inject.Binder; +import com.google.cloud.spark.bigquery.repackaged.com.google.inject.Module; +import com.google.cloud.spark.bigquery.repackaged.com.google.inject.Provides; +import com.google.common.collect.ImmutableMap; +import io.openlineage.client.OpenLineage; +import io.openlineage.client.OpenLineage.DatasetFacets; +import io.openlineage.client.OpenLineage.InputDataset; +import io.openlineage.client.OpenLineage.OutputDataset; +import io.openlineage.client.OpenLineage.SchemaDatasetFacet; +import io.openlineage.client.OpenLineage.SchemaDatasetFacetFields; +import io.openlineage.spark.agent.SparkAgentTestExtension; +import io.openlineage.spark.agent.Versions; +import io.openlineage.spark.agent.util.PlanUtils; +import io.openlineage.spark.agent.util.SparkVersionUtils; +import io.openlineage.spark.agent.util.TestOpenLineageEventHandlerFactory; +import java.io.FileOutputStream; +import java.io.IOException; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.nio.file.Path; +import java.nio.file.Paths; +import java.util.Arrays; +import java.util.List; +import java.util.Optional; +import java.util.Properties; +import java.util.Random; +import java.util.UUID; +import java.util.concurrent.CompletableFuture; +import java.util.concurrent.ExecutionException; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.TimeoutException; +import lombok.extern.slf4j.Slf4j; +import org.apache.commons.lang3.tuple.Pair; +import org.apache.hadoop.io.LongWritable; +import org.apache.hadoop.io.Text; +import org.apache.hadoop.mapred.FileInputFormat; +import org.apache.hadoop.mapred.JobConf; +import org.apache.hadoop.mapred.TextInputFormat; +import org.apache.kafka.clients.producer.KafkaProducer; +import org.apache.kafka.clients.producer.ProducerRecord; +import org.apache.kafka.common.serialization.StringSerializer; +import org.apache.spark.api.java.JavaRDD; +import org.apache.spark.api.java.JavaSparkContext; +import org.apache.spark.sql.AnalysisException; +import org.apache.spark.sql.Dataset; +import org.apache.spark.sql.Row; +import org.apache.spark.sql.SaveMode; +import org.apache.spark.sql.SparkSession; +import org.apache.spark.sql.catalyst.expressions.GenericRow; +import org.apache.spark.sql.catalyst.expressions.GenericRowWithSchema; +import org.apache.spark.sql.types.BinaryType$; +import org.apache.spark.sql.types.IntegerType$; +import org.apache.spark.sql.types.LongType$; +import org.apache.spark.sql.types.Metadata; +import org.apache.spark.sql.types.StringType$; +import org.apache.spark.sql.types.StructField; +import org.apache.spark.sql.types.StructType; +import org.assertj.core.api.InstanceOfAssertFactories; +import org.assertj.core.api.ObjectAssert; +import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Tag; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.condition.EnabledIfSystemProperty; +import org.junit.jupiter.api.extension.ExtendWith; +import org.junit.jupiter.api.io.TempDir; +import org.mockito.ArgumentCaptor; +import org.mockito.Mockito; +import org.testcontainers.containers.KafkaContainer; +import org.testcontainers.utility.DockerImageName; +import scala.Tuple2; +import scala.collection.immutable.HashMap;</p> + +<p>@ExtendWith(SparkAgentTestExtension.class) +@Tag("integration-test") +@Slf4j +class SparkReadWriteIntegTest {</p> + +<p>private static final String EVENT<em>TYPE = "eventType"; + private static final String NAMESPACE = "namespace"; + private static final String FILE = "file"; + private static final String NAME = "name"; + private static final String AGE = "age"; + private static final String FILE</em>URI_PREFIX = "file://"; + private final KafkaContainer kafkaContainer = + new KafkaContainer(DockerImageName.parse("confluentinc/cp-kafka:7.0.0"));</p> + +<p>@BeforeEach + public void setUp() { + reset(MockBigQueryRelationProvider.BIG<em>QUERY); + when(SparkAgentTestExtension.OPEN</em>LINEAGE<em>SPARK</em>CONTEXT.getParentRunId()) + .thenReturn(Optional.of(UUID.randomUUID())); + when(SparkAgentTestExtension.OPEN<em>LINEAGE</em>SPARK<em>CONTEXT.getParentJobName()) + .thenReturn("ParentJob"); + when(SparkAgentTestExtension.OPEN</em>LINEAGE<em>SPARK</em>CONTEXT.getJobNamespace()) + .thenReturn("Namespace"); + }</p> + +<p>@AfterEach + public void tearDown() { + if (kafkaContainer.isCreated()) { + kafkaContainer.stop(); + } + }</p> + +<p>@Test + void testBigQueryReadWriteToFile(@TempDir Path writeDir, SparkSession spark) + throws InterruptedException, TimeoutException { + TableId tableId = TableId.of("testproject", "dataset", "MyTable"); + BigQuery bq = MockBigQueryRelationProvider.BIG_QUERY; + StructType tableSchema = + new StructType( + new StructField[] { + new StructField(NAME, StringType$.MODULE$, false, Metadata.empty()), + new StructField(AGE, LongType$.MODULE$, false, Metadata.empty()) + });</p> + +<pre><code>MockBigQueryRelationProvider.INJECTOR.setTestModule( + new Module() { + @Override + public void configure(Binder binder) {} + + @Provides + public Dataset&lt;Row&gt; testData() { + return spark.createDataFrame( + Arrays.asList( + new GenericRowWithSchema(new Object[] {"john", 25L}, tableSchema), + new GenericRowWithSchema(new Object[] {"sam", 22L}, tableSchema), + new GenericRowWithSchema(new Object[] {"alicia", 35L}, tableSchema), + new GenericRowWithSchema(new Object[] {"bob", 47L}, tableSchema), + new GenericRowWithSchema(new Object[] {"jordan", 52L}, tableSchema), + new GenericRowWithSchema(new Object[] {"liz", 19L}, tableSchema), + new GenericRowWithSchema(new Object[] {"marcia", 83L}, tableSchema), + new GenericRowWithSchema(new Object[] {"maria", 40L}, tableSchema), + new GenericRowWithSchema(new Object[] {"luis", 8L}, tableSchema), + new GenericRowWithSchema(new Object[] {"gabriel", 30L}, tableSchema)), + tableSchema); + } + }); +when(bq.getTable(eq(tableId))) + .thenAnswer( + invocation -&gt; + MockBigQueryRelationProvider.makeTable( + tableId, + StandardTableDefinition.newBuilder() + .setSchema( + Schema.of( + Field.of(NAME, StandardSQLTypeName.STRING), + … +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
추호관 + (hogan.chu@toss.im) +
+
2023-08-08 06:24:54
 +
+

*Thread Reply:* okay i would look into and maybe pr thanks

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
추호관 + (hogan.chu@toss.im) +
+
2023-08-08 06:38:45
 +
+

*Thread Reply:* Can I ask if there are any suspicious properties?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-08-08 06:39:25
 +
+

*Thread Reply:* sure

+ + + +
+ 👍 추호관 +
+ +
+ 🙂 추호관 +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
추호관 + (hogan.chu@toss.im) +
+
2023-08-08 07:10:40
 +
+

*Thread Reply:* Thanks I would also try to find the property too

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-08-08 05:34:46
 +
+

Hi guys, I've a generic sql-parsing doubt... what would be the recommended way (if any) to check for sql similarity? I understand that most sql parsers parse the query into an AST, but are there any well known ways to measure semantic similarities between 2 or more ASTs? Just curious lol... Any ideas appreciated! Thanks!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Guy Biecher + (guy.biecher21@gmail.com) +
+
2023-08-08 07:49:55
 +
+

*Thread Reply:* Hi @Anirudh Shrinivason, +I think I would take a look on this +https://sqlglot.com/sqlglot/diff.html

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-08-09 23:12:37
 +
+

*Thread Reply:* Hey @Guy Biecher Yeah I was looking at this... but it seems to calculate similarity from a more textual context, as opposed to a more semantic one... +eg: SELECT ** FROM TABLE_1 and SELECT col1,col2,col3 FROM TABLE_1 could be the same semantic query, but sqlglot would give diffs in the ast because its textual...

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Guy Biecher + (guy.biecher21@gmail.com) +
+
2023-08-10 02:26:51
 +
+

*Thread Reply:* I totally get you. In such cases without the metadata of the TABLE_1, it's impossible what I would do I would replace all ** before you use the diff function.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-08-10 07:04:37
 +
+

*Thread Reply:* Yeah I was thinking about the same... But the more nested and complex your queries get, the harder it'll become to accurately pre-process before running the ast diff too... +But yeah that's probably the approach I'd be taking haha... Happy to discuss and learn if there are better ways to doing this

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Luigi Scorzato + (luigi.scorzato@gmail.com) +
+
2023-08-08 08:36:46
 +
+

dear all, I have some novice questions. I put them in separate messages for clarity. 1st Question: I understand from the examples in the documentation that the main lineage events are RunEvent's, which can contain link to Run ID, Job ID, Dataset ID (I see they are RunEvent because they have EventType, correct?). However, the main openlineage json object contains also JobEvent and DatasetEvent. When are JobEvent and DatasetEvent supposed to be used in the workflow? Do you have relevant examples? thanks!

+
+
openlineage.io
+ + + +
Example Lineage Events | OpenLineage
+
+ Simple Examples +
+ + + + + + +
Original URL: https://openlineage.io/docs/development/examples
+ + + +
+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.json | OpenLineage.json>
+
+ <p>``` +{ + "$schema": "<a href="https://json-schema.org/draft/2020-12/schema">https://json-schema.org/draft/2020-12/schema</a>", + "$id": "<a href="https://openlineage.io/spec/2-0-2/OpenLineage.json">https://openlineage.io/spec/2-0-2/OpenLineage.json</a>", + "$defs": { + "BaseEvent": { + "type": "object", + "properties": { + "eventTime": { + "description": "the time the event occurred at", + "type": "string", + "format": "date-time" + }, + "producer": { + "description": "URI identifying the producer of this metadata. For example this could be a git url with a given tag or sha", + "type": "string", + "format": "uri", + "example": "<a href="https://github.com/OpenLineage/OpenLineage/blob/v1-0-0/client">https://github.com/OpenLineage/OpenLineage/blob/v1-0-0/client</a>" + }, + "schemaURL": { + "description": "The JSON Pointer (<a href="https://tools.ietf.org/html/rfc6901">https://tools.ietf.org/html/rfc6901</a>) URL to the corresponding version of the schema definition for this RunEvent", + "type": "string", + "format": "uri", + "example": "<a href="https://openlineage.io/spec/0-0-1/OpenLineage.json">https://openlineage.io/spec/0-0-1/OpenLineage.json</a>" + } + }, + "required": ["eventTime", "producer", "schemaURL"] + }, + "RunEvent": { + "allOf": [ + { "$ref": "#/$defs/BaseEvent" }, + { + "type": "object", + "properties": { + "eventType": { + "description": "the current transition of the run state. It is required to issue 1 START event and 1 of [ COMPLETE, ABORT, FAIL ] event per run. Additional events with OTHER eventType can be added to the same run. For example to send additional metadata after the run is complete", + "type": "string", + "enum": ["START", "RUNNING", "COMPLETE", "ABORT", "FAIL", "OTHER"], + "example": "START|RUNNING|COMPLETE|ABORT|FAIL|OTHER" + }, + "run": { + "$ref": "#/$defs/Run" + }, + "job": { + "$ref": "#/$defs/Job" + }, + "inputs": { + "description": "The set of <strong><em>*input</em><em></strong> datasets.", + "type": "array", + "items": { + "$ref": "#/$defs/InputDataset" + } + }, + "outputs": { + "description": "The set of <strong></em><em>output</em><em></strong> datasets.", + "type": "array", + "items": { + "$ref": "#/$defs/OutputDataset" + } + } + }, + "required": ["run", "job"] + } + ] + }, + "DatasetEvent": { + "allOf": [ + { "$ref": "#/$defs/BaseEvent" }, + { + "type": "object", + "properties": { + "dataset": { + "$ref": "#/$defs/StaticDataset" + } + }, + "required": ["dataset"], + "not": { "required": ["job", "run"] } + } + ] + }, + "JobEvent": { + "allOf": [ + { "$ref": "#/$defs/BaseEvent" }, + { + "type": "object", + "properties": { + "job": { + "$ref": "#/$defs/Job" + }, + "inputs": { + "description": "The set of <strong></em><em>input</em><em></strong> datasets.", + "type": "array", + "items": { + "$ref": "#/$defs/InputDataset" + } + }, + "outputs": { + "description": "The set of <strong></em><em>output</em>*</strong> datasets.", + "type": "array", + "items": { + "$ref": "#/$defs/OutputDataset" + } + } + }, + "required": ["job"], + "not": { "required": ["run"] } + } + ] + }, + "Run": { + "type": "object", + "properties": { + "runId": { + "description": "The globally unique ID of the run associated with the job.", + "type": "string", + "format": "uuid" + }, + "facets": { + "description": "The run facets.", + "type": "object", + "anyOf": [ + { + "type": "object", + "additionalProperties": { "$ref": "#/$defs/RunFacet" } + } + ] + } + }, + "required": ["runId"] + }, + "RunFacet": { + "description": "A Run Facet", + "type": "object", + "allOf": [{ "$ref": "#/$defs/BaseFacet" }] + }, + "Job": { + "type": "object", + "properties": { + "namespace": { + "description": "The namespace containing that job", + "type": "string", + "example": "my-scheduler-namespace" + }, + "name": { + "description": "The unique name for that job within that namespace", + "type": "string", + "example": "myjob.mytask" + }, + "facets": { + "description": "The job facets.", + "type": "object", + "anyOf": [ + { + "type": "object", + "additionalProperties": { "$ref": "#/$defs/JobFacet" } + } + ] + } + }, + "required": ["namespace", "name"] + }, + "JobFacet": { + "description": "A Job Facet", + "type": "object", + "allOf": [ + { "$ref": "#/$defs/BaseFacet" }, + { + "type": "object", + "properties": { + "<em>deleted": { + "description": "set to true to delete a facet", + "type": "boolean" + } + } + } + ] + }, + "InputDataset": { + "description": "An input dataset", + "type": "object", + "allOf": [ + { "$ref": "#/$defs/Dataset" }, + { + "type": "object", + "properties": { + "inputFacets": { + "description": "The input facets for this dataset.", + "type": "object", + "anyOf": [ + { + "type": "object", + "additionalProperties": { + "$ref": "#/$defs/InputDatasetFacet" + } + } + ] + } + } + } + ] + }, + "InputDatasetFacet": { + "description": "An Input Dataset Facet", + "type": "object", + "allOf": [{ "$ref": "#/$defs/BaseFacet" }] + }, + "OutputDataset": { + "description": "An output dataset", + "type": "object", + "allOf": [ + { "$ref": "#/$defs/Dataset" }, + { + "type": "object", + "properties": { + "outputFacets": { + "description": "The output facets for this dataset", + "type": "object", + "anyOf": [ + { + "type": "object", + "additionalProperties": { + "$ref": "#/$defs/OutputDatasetFacet" + } + } + ] + } + } + } + ] + }, + "OutputDatasetFacet": { + "description": "An Output Dataset Facet", + "type": "object", + "allOf": [{ "$ref": "#/$defs/BaseFacet" }] + }, + "Dataset": { + "type": "object", + "properties": { + "namespace": { + "description": "The namespace containing that dataset", + "type": "string", + "example": "my-datasource-namespace" + }, + "name": { + "description": "The unique name for that dataset within that namespace", + "type": "string", + "example": "instance.schema.table" + }, + "facets": { + "description": "The facets for this dataset", + "type": "object", + "anyOf": [ + { + "type": "object", + "additionalProperties": { "$ref": "#/$defs/DatasetFacet" } + } + ] + } + }, + "required": ["namespace", "name"] + }, + "StaticDataset": { + "description": "A Dataset sent within static metadata events", + "type": "object", + "allOf": [{ "$ref": "#/$defs/Dataset" }] + }, + "DatasetFacet": { + "description": "A Dataset Facet", + "type": "object", + "allOf": [ + { "$ref": "#/$defs/BaseFacet" }, + { + "type": "object", + "properties": { + "</em>deleted": { + …</p> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harel Shein + (harel.shein@gmail.com) +
+
2023-08-08 09:53:05
 +
+

*Thread Reply:* Hey @Luigi Scorzato! +You can read about these 2 event types in this blog post: https://openlineage.io/blog/static-lineage

+
+
openlineage.io
+ + + +
OpenLineage 1.0, Featuring Static Lineage, is Coming Soon | OpenLineage
+
+ The release of OpenLineage 1.0 will add static lineage support. +
+ + + + + + +
Original URL: https://openlineage.io/blog/static-lineage
+ + + +
+ + + +
+ 👍 Luigi Scorzato +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harel Shein + (harel.shein@gmail.com) +
+
2023-08-08 09:53:38
 +
+

*Thread Reply:* we’ll work on getting the documentation improved to clarify the expected use cases for each event type. this is a relatively new addition to the spec.

+ + + +
+ 👍 Luigi Scorzato +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Luigi Scorzato + (luigi.scorzato@gmail.com) +
+
2023-08-08 10:08:28
 +
+

*Thread Reply:* this sounds relevant for my 3rd question, doesn't it? But I do not see scheduling information among the use cases, am I wrong?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harel Shein + (harel.shein@gmail.com) +
+
2023-08-08 11:16:39
 +
+

*Thread Reply:* you’re not wrong, these 2 events were not designed for runtime lineage, but rather “static” lineage that gets emitted after the fact

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Luigi Scorzato + (luigi.scorzato@gmail.com) +
+
2023-08-08 08:46:39
 +
+

2nd Question. I see that the input dataset appears in the RunEvent with EventType=START, the output dataset appears in the RunEvent with EventType=COMPLETE only, the RunEvent with EventType=RUNNING has no dataset attached. This makes sense for ETL jobs, but for streaming (e.g. Flink), the run could run very long and never terminate with a COMPLETE. On the other hand, emitting all the info about the output dataset in every RUNNING event would be far too verbose. What is the recommended set up in this case? TLDR: what is the recommended configuration of the frequency and data model of the lineage events for streaming systems like Flink?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harel Shein + (harel.shein@gmail.com) +
+
2023-08-08 09:54:40
 +
+

*Thread Reply:* great question! did you get a chance to look at the current Flink integration?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Luigi Scorzato + (luigi.scorzato@gmail.com) +
+
2023-08-08 10:07:06
 +
+

*Thread Reply:* to be honest, I only quickly went through this and I did not identfy what I needed. Can you please point me to the relevant section?

+
+
openlineage.io
+ + + +
Apache Flink | OpenLineage
+
+ This integration is considered experimental: only specific workflows and use cases are supported. +
+ + + + + + +
Original URL: https://openlineage.io/docs/integrations/flink
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harel Shein + (harel.shein@gmail.com) +
+
2023-08-08 11:13:17
 +
+

*Thread Reply:* here’s an example START event for Flink: https://github.com/OpenLineage/OpenLineage/blob/main/integration/flink/src/test/resources/events/expected_kafka.json

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/integration/flink/src/test/resources/events/expected_kafka.json | expected_kafka.json>
+
+ <pre><code>{ + "eventType" : "START", + "eventTime": "${json-unit.any-string}", + "job" : { + "namespace" : "flink_job_namespace", + "name": "flink_examples_stateful" + }, + "run": { + "runId": "${json-unit.any-string}" + }, + "inputs": [ + { + "namespace": "kafka:9092", + "name": "io.openlineage.flink.kafka.input1", + "facets": { + "schema": { + "fields": [ + { + "name": "id", + "type": "string" + }, + { + "name": "version", + "type": "long" + } + ] + } + } + }, + { + "namespace": "kafka:9092", + "name": "io.openlineage.flink.kafka.input2", + "facets": { + "schema": { + "fields": [ + { + "name": "id", + "type": "string" + }, + { + "name": "version", + "type": "long" + } + ] + } + } + } + ], + "outputs": [ + { + "namespace": "kafka:9092", + "name": "io.openlineage.flink.kafka.output", + "facets": { + "schema": { + "fields": [ + { + "name": "id", + "type": "string" + }, + { + "name": "version", + "type": "long" + }, + { + "name": "counter", + "type": "long" + } + ] + } + } + } + ] +}, + +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harel Shein + (harel.shein@gmail.com) +
+
2023-08-08 11:13:26
 +
+

*Thread Reply:* or a checkpoint (RUNNING) event: https://github.com/OpenLineage/OpenLineage/blob/main/integration/flink/src/test/resources/events/expected_kafka_checkpoints.json

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/integration/flink/src/test/resources/events/expected_kafka_checkpoints.json | expected_kafka_checkpoints.json>
+
+ <pre><code>{ + "eventType" : "RUNNING", + "eventTime": "${json-unit.any-string}", + "job" : { + "namespace" : "flink_job_namespace", + "name": "flink_examples_stateful" + }, + "run": { + "runId": "${json-unit.any-string}", + "facets" : { + "checkpoints": { + "completed": "${json-unit.any-number}", + "failed": "${json-unit.any-number}", + "in-progress": "${json-unit.any-number}", + "restored": "${json-unit.any-number}", + "total": "${json-unit.any-number}" + } + } + }, + "inputs": [ + { + "namespace": "kafka:9092", + "name": "io.openlineage.flink.kafka.input1", + "facets": { + "schema": { + "fields": [ + { + "name": "id", + "type": "string" + }, + { + "name": "version", + "type": "long" + } + ] + } + } + }, + { + "namespace": "kafka:9092", + "name": "io.openlineage.flink.kafka.input2", + "facets": { + "schema": { + "fields": [ + { + "name": "id", + "type": "string" + }, + { + "name": "version", + "type": "long" + } + ] + } + } + } + ], + "outputs": [ + { + "namespace": "kafka:9092", + "name": "io.openlineage.flink.kafka.output", + "facets": { + "schema": { + "fields": [ + { + "name": "id", + "type": "string" + }, + { + "name": "version", + "type": "long" + }, + { + "name": "counter", + "type": "long" + } + ] + } + } + } + ] +} + +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harel Shein + (harel.shein@gmail.com) +
+
2023-08-08 11:15:55
 +
+

*Thread Reply:* generally speaking, you can see the execution contexts that invoke generation of OL events here: https://github.com/OpenLineage/OpenLineage/blob/main/integration/flink/src/main/ja[…]/openlineage/flink/visitor/lifecycle/FlinkExecutionContext.java

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/integration/flink/src/main/java/io/openlineage/flink/visitor/lifecycle/FlinkExecutionContext.java | FlinkExecutionContext.java>
+
+ <pre><code>/** +/** Copyright 2018-2023 contributors to the OpenLineage project +/** SPDX-License-Identifier: Apache-2.0 +**/ + +package io.openlineage.flink.visitor.lifecycle; + +import io.openlineage.client.OpenLineage; +import io.openlineage.client.OpenLineage.RunEvent; +import io.openlineage.client.OpenLineage.RunEvent.EventType; +import io.openlineage.client.OpenLineage.RunEventBuilder; +import io.openlineage.flink.SinkLineage; +import io.openlineage.flink.TransformationUtils; +import io.openlineage.flink.api.OpenLineageContext; +import io.openlineage.flink.client.CheckpointFacet; +import io.openlineage.flink.client.EventEmitter; +import io.openlineage.flink.visitor.Visitor; +import io.openlineage.flink.visitor.VisitorFactory; +import io.openlineage.flink.visitor.VisitorFactoryImpl; +import java.time.ZonedDateTime; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.HashSet; +import java.util.List; +import java.util.Set; +import java.util.UUID; +import java.util.stream.Collectors; +import lombok.Builder; +import lombok.Getter; +import lombok.extern.slf4j.Slf4j; +import org.apache.commons.lang3.exception.ExceptionUtils; +import org.apache.flink.api.common.JobExecutionResult; +import org.apache.flink.api.common.JobID; +import org.apache.flink.api.dag.Transformation; + +@Slf4j +@Builder +public class FlinkExecutionContext implements ExecutionContext { + + @Getter private final JobID jobId; + protected final UUID runId; + protected final EventEmitter eventEmitter; + protected final OpenLineageContext openLineageContext; + private final String jobName; + private final String jobNamespace; + + @Getter private final List&lt;Transformation&lt;?&gt;&gt; transformations; + + @Override + public void onJobSubmitted() { + log.debug("JobClient - jobId: {}", jobId); + RunEvent runEvent = + buildEventForEventType(EventType.START) + .run(new OpenLineage.RunBuilder().runId(runId).build()) + .build(); + log.debug("Posting event for onJobSubmitted {}: {}", jobId, runEvent); + eventEmitter.emit(runEvent); + } + + @Override + public void onJobCheckpoint(CheckpointFacet facet) { + log.debug("JobClient - jobId: {}", jobId); + RunEvent runEvent = + buildEventForEventType(EventType.RUNNING) + .run( + new OpenLineage.RunBuilder() + .runId(runId) + .facets(new OpenLineage.RunFacetsBuilder().put("checkpoints", facet).build()) + .build()) + .build(); + log.debug("Posting event for onJobCheckpoint {}: {}", jobId, runEvent); + eventEmitter.emit(runEvent); + } + + public OpenLineage.RunEventBuilder buildEventForEventType(EventType eventType) { + TransformationUtils converter = new TransformationUtils(); + + List&lt;SinkLineage&gt; sinkLineages = converter.convertToVisitable(transformations); + + VisitorFactory visitorFactory = new VisitorFactoryImpl(); + List&lt;OpenLineage.InputDataset&gt; inputDatasets = new ArrayList&lt;&gt;(); + List&lt;OpenLineage.OutputDataset&gt; outputDatasets = new ArrayList&lt;&gt;(); + + Set&lt;Object&gt; sources = new HashSet&lt;&gt;(); + for (var lineage : sinkLineages) { + sources.addAll(lineage.getSources()); + outputDatasets.addAll(getOutputDatasets(visitorFactory, lineage.getSink())); + } + inputDatasets.addAll(getInputDatasets(visitorFactory, Arrays.asList(sources.toArray()))); + + return commonEventBuilder().inputs(inputDatasets).outputs(outputDatasets).eventType(eventType); + } + + @Override + public void onJobCompleted(JobExecutionResult jobExecutionResult) { + OpenLineage openLineage = openLineageContext.getOpenLineage(); + eventEmitter.emit( + commonEventBuilder() + .run(openLineage.newRun(runId, null)) + .eventType(EventType.COMPLETE) + .build()); + } + + @Override + public void onJobFailed(Throwable failed) { + OpenLineage openLineage = openLineageContext.getOpenLineage(); + eventEmitter.emit( + commonEventBuilder() + .run( + openLineage.newRun( + runId, + openLineage + .newRunFacetsBuilder() + .errorMessage( + openLineage.newErrorMessageRunFacet( + failed.getMessage(), "JAVA", ExceptionUtils.getStackTrace(failed))) + .build())) + .eventType(EventType.FAIL) + .eventTime(ZonedDateTime.now()) + .build()); + } + + /**** + ** Builds common elements of Openlineage events constructed. + ** + ** @return + **/ + private RunEventBuilder commonEventBuilder() { + OpenLineage openLineage = openLineageContext.getOpenLineage(); + return openLineage + .newRunEventBuilder() + .job(openLineage.newJob(jobNamespace, jobName, null)) + .eventTime(ZonedDateTime.now()); + } + + private List&lt;OpenLineage.InputDataset&gt; getInputDatasets( + VisitorFactory visitorFactory, List&lt;Object&gt; sources) { + List&lt;OpenLineage.InputDataset&gt; inputDatasets = new ArrayList&lt;&gt;(); + List&lt;Visitor&lt;OpenLineage.InputDataset&gt;&gt; inputVisitors = + visitorFactory.getInputVisitors(openLineageContext); + + for (var transformation : sources) { + inputDatasets.addAll( + inputVisitors.stream() + .filter(inputVisitor -&gt; inputVisitor.isDefinedAt(transformation)) + .map(inputVisitor -&gt; inputVisitor.apply(transformation)) + .flatMap(List::stream) + .collect(Collectors.toList())); + } + return inputDatasets; + } + + private List&lt;OpenLineage.OutputDataset&gt; getOutputDatasets( + VisitorFactory visitorFactory, Object sink) { + List&lt;Visitor&lt;OpenLineage.OutputDataset&gt;&gt; outputVisitors = + visitorFactory.getOutputVisitors(openLineageContext); + + return outputVisitors.stream() + .filter(inputVisitor -&gt; inputVisitor.isDefinedAt(sink)) + .map(inputVisitor -&gt; inputVisitor.apply(sink)) + .flatMap(List::stream) + .collect(Collectors.toList()); + } +} + +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+ 👍 Luigi Scorzato +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Luigi Scorzato + (luigi.scorzato@gmail.com) +
+
2023-08-08 17:46:17
 +
+

*Thread Reply:* thank you! So, if I understand correctly, the key is that even eventType=START, admits an output datasets. Correct? What determines how often are the eventType=RUNNING emitted?

+ + + +
+ 👍 Harel Shein +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Luigi Scorzato + (luigi.scorzato@gmail.com) +
+
2023-08-09 03:25:16
 +
+

*Thread Reply:* now I see, RUNNING events are emitted on onJobCheckpoint

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Luigi Scorzato + (luigi.scorzato@gmail.com) +
+
2023-08-08 08:59:40
 +
+

3rd Question: I am looking for information about the time when the next run should start, in case of scheduled jobs. I see that the Run Facet has a Nominal Time Facet, but -- if I understand correctly -- it refers to the current run, so it is always emitted after the fact. Is the Nominal Start Time of the next run available somewhere? If not, where do you recommend to add it as a custom field? In principle, it belongs to the Job object, but would that maybe cause an undesirable fast change in the Job object?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harel Shein + (harel.shein@gmail.com) +
+
2023-08-08 11:10:47
 +
+

*Thread Reply:* For Airflow, this is part of the AirflowRunFacet, here: https://github.com/OpenLineage/OpenLineage/blob/81372ca2bc2afecab369eab4a54cc6380dda49d0/integration/airflow/facets/AirflowRunFacet.json#L100

+ +

For other orchestrators / schedulers, that would depend..

+ + + +
+ 👍 Luigi Scorzato +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kiran Hiremath + (kiran_hiremath@intuit.com) +
+
2023-08-08 10:30:56
 +
+

Hi Team, Question regarding Databricks OpenLineage init script, is the path /mnt/driver-daemon/jars common to all the clusters? or its unique to each cluster? https://github.com/OpenLineage/OpenLineage/blob/81372ca2bc2afecab369eab4a54cc6380d[…]da49d0/integration/spark/databricks/open-lineage-init-script.sh

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-08 12:15:40
 +
+

*Thread Reply:* I might be wrong, but I believe it's unique for each cluster - the common part is dbfs\.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-08-09 02:38:54
 +
+

*Thread Reply:* dbfs is mounted to a databricks workspace which can run multiple clusters. so i think, it's common.

+ +

Worth mentioning: init-scripts located in dbfs are becoming deprecated next month and we plan moving them into workspaces.

+ + + +
+ 👍 Kiran Hiremath +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kiran Hiremath + (kiran_hiremath@intuit.com) +
+
2023-08-11 01:33:24
 +
+

*Thread Reply:* yes, the init scripts are moved at workspace level.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
GitHubOpenLineageIssues + (githubopenlineageissues@gmail.com) +
+
2023-08-08 14:19:40
 +
+

Hi @Paweł Leszczyński Will really aprecaite if you please let me know once this PR is good to go. Will love to test in our environment : https://github.com/OpenLineage/OpenLineage/pull/2036. Thank you for all your help.

+
+ + + + +
#2036 [SPARK] append output dataset name to job name
+
+ <p><strong>Problem</strong></p> + +<p>👋 Thanks for opening a pull request! Please include a brief summary of the problem your change is trying to solve, or bug fix. If your change fixes a bug or you'd like to provide context on why you're making the change, please <a href="https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue">link the issue</a> as follows:</p> + +<p>Closes: #ISSUE-NUMBER</p> + +<p><strong>Solution</strong></p> + +<p>Please describe your change as it relates to the problem, or bug fix, as well as any dependencies. If your change requires a schema change, please describe the schema modifcation(s) and whether it's a <em>backwards-incompatible</em> or <em>backwards-compatible</em> change, then select one of the following:</p> + +<blockquote> + <p><strong>Note:</strong> All schema changes require discussion. Please <a href="https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue">link the issue</a> for context.</p> +</blockquote> + +<p>☐ Your change modifies the <a href="https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.json">core</a> OpenLineage model +☐ Your change modifies one or more OpenLineage <a href="https://github.com/OpenLineage/OpenLineage/tree/main/spec/facets">facets</a></p> + +<p>If you're contributing a new integration, please specify the scope of the integration and how/where it has been tested (e.g., Apache Spark integration supports <code>S3</code> and <code>GCS</code> filesystem operations, tested with AWS EMR).</p> + +<p><strong>One-line summary:</strong> +<strong>Checklist</strong></p> + +<p>☐ You've <a href="https://github.com/OpenLineage/OpenLineage/blob/main/why-the-dco.md">signed-off</a> your work +☐ Your pull request title follows our <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CONTRIBUTING.md#creating-pull-requests">guidelines</a> +☐ Your changes are accompanied by tests (<em>if relevant</em>) +☐ Your change contains a <a href="https://kurtisnusbaum.medium.com/stacked-diffs-keeping-phabricator-diffs-small-d9964f4dcfa6">small diff</a> and is self-contained +☐ You've updated any relevant documentation (<em>if relevant</em>) +☐ Your comment includes a one-liner for the changelog about the specific purpose of the change (<em>if necessary</em>) +☐ You've versioned the core OpenLineage model or facets according to <a href="https://docs.snowplowanalytics.com/docs/pipeline-components-and-applications/iglu/common-architecture/schemaver">SchemaVer</a> (<em>if relevant</em>) +☐ You've added a <a href="https://github.com/OpenLineage/OpenLineage/tree/main/.github/header_templates.md">header</a> to source files (<em>if relevant</em>)</p> + +<hr /> + +<p>SPDX-License-Identifier: Apache-2.0 <br /> +Copyright 2018-2023 contributors to the OpenLineage project</p> +
+ +
+
Labels
+ integration/spark +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-08-09 02:35:28
 +
+

*Thread Reply:* great to hear. I still need some time as there are few corner cases. For example: what should be the behaviour when alter table rename is called 😉 But sure, you can test it if you like. ci is failing on integration tests but ./gradlew clean build with unit tests are fine.

+ + + +
+ :gratitude_thank_you: GitHubOpenLineageIssues +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-08-10 03:33:50
 +
+

*Thread Reply:* @GitHubOpenLineageIssues Feel invited to join todays community and advocate for the importance of this issue. Such discussions are extremely helpful in prioritising backlog the right way.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Gaurav Singh + (gaurav.singh@razorpay.com) +
+
2023-08-09 07:54:33
 +
+

Hi Team, +I'm doing a POC with open lineage to extract column lineage of Spark. I'm using it on databricks notebook. I'm facing a issue where I,m trying to get the column lineage in a join involving external tables on s3. The lineage that is being extracted is returning on base path of the table ie on the s3 file path and not on the corresponding tables. Is there a way to extract/map columns of output to the columns of base tables instead of storage location.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Gaurav Singh + (gaurav.singh@razorpay.com) +
+
2023-08-09 07:55:28
 +
+

*Thread Reply:* Query: +INSERT INTO test.merchant_md +(Select + m.`id`, + m.name, + m.activated, + m.parent_id, + md.contact_name, + md.contact_email +FROM + test.merchants_0 m + LEFT JOIN merchant_details md ON m.id = md.merchant_id +WHERE + m.created_date &gt; '2023-08-01')

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Gaurav Singh + (gaurav.singh@razorpay.com) +
+
2023-08-09 08:01:56
 +
+

*Thread Reply:* "columnLineage":{ + "_producer":"<https://github.com/OpenLineage/OpenLineage/tree/0.30.1/integration/spark>", + "_schemaURL":"<https://openlineage.io/spec/facets/1-0-1/ColumnLineageDatasetFacet.json#/$defs/ColumnLineageDatasetFacet>", + "fields":{ + "merchant_id":{ + "inputFields":[ + { + "namespace":"<s3a://datalake>", + "name":"/test/merchants", + "field":"id" + } + ] + }, + "merchant_name":{ + "inputFields":[ + { + "namespace":"<s3a://datalake>", + "name":"/test/merchants", + "field":"name" + } + ] + }, + "activated":{ + "inputFields":[ + { + "namespace":"<s3a://datalake>", + "name":"/test/merchants", + "field":"activated" + } + ] + }, + "parent_id":{ + "inputFields":[ + { + "namespace":"<s3a://datalake>", + "name":"/test/merchants", + "field":"parent_id" + } + ] + }, + "contact_name":{ + "inputFields":[ + { + "namespace":"<s3a://datalake>", + "name":"/test/merchant_details", + "field":"contact_name" + } + ] + }, + "contact_email":{ + "inputFields":[ + { + "namespace":"<s3a://datalake>", + "name":"/test/merchant_details", + "field":"contact_email" + } + ] + } + } + }, + "symlinks":{ + "_producer":"<https://github.com/OpenLineage/OpenLineage/tree/0.30.1/integration/spark>", + "_schemaURL":"<https://openlineage.io/spec/facets/1-0-0/SymlinksDatasetFacet.json#/$defs/SymlinksDatasetFacet>", + "identifiers":[ + { + "namespace":"/warehouse/test.db", + "name":"test.merchant_md", + "type":"TABLE" + }

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Gaurav Singh + (gaurav.singh@razorpay.com) +
+
2023-08-09 08:23:57
 +
+

*Thread Reply:* "contact_name":{ + "inputFields":[ + { + "namespace":"<s3a://datalake>", + "name":"/test/merchant_details", + "field":"contact_name" + } + ] + } +This is returning mapping from the s3 location on which the table is created.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Zahi Fail + (zahi.fail@gmail.com) +
+
2023-08-09 10:56:27
 +
+

Hey, +I’m running Spark application (spark version 3.4) with OL integration. +I changed spark to use “debug” level, and I see the OL events with the below message: +“Emitting lineage completed successfully:”

+ +

With all the above, I can’t see the event in Marquez.

+ +

Attaching the OL configurations. +When changing the OL-spark version to 0.6.+, I do see event created in Marquez with only “Start” status (attached below).

+ +

The OL-spark version is matching the Spark version? Is there a known issues with the Spark / OL versions ?

+ +
+
+ + + + + + +
+
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-09 11:23:42
 +
+

*Thread Reply:* > OL-spark version to 0.6.+ +This OL version is ancient. You can try with 1.0.0

+ +

I think you're hitting this issue which duplicates jobs: https://github.com/OpenLineage/OpenLineage/issues/1943

+
+ + + + +
#1943 spark: two jobs operating on different datasets in same spark file can be named the same
+
+ <p>Currently, a Spark job name is created from action name which is misleading as different actions run with the same name can be writing to different output datasets. So, although the jobs are different, they end up within same graph nodes blurring the lineage.</p> + +<p>The solution for this would be to: append dataset name to the end of the job, the same way in which it is actually done for databricks platform.</p> +
+ +
+
Assignees
+ <a href="https://github.com/pawel-big-lebowski">@pawel-big-lebowski</a> +
+ +
+
Labels
+ integration/spark +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Zahi Fail + (zahi.fail@gmail.com) +
+
2023-08-10 01:46:08
 +
+

*Thread Reply:* I haven’t mentioned that I tried multiple OL versions - 1.0.0 / 0.30.1 / 0.6.+ … +None of them worked for me. +@Maciej Obuchowski

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-10 05:25:49
 +
+

*Thread Reply:* @Zahi Fail understood. Can you provide sample job that reproduces this behavior, and possibly some logs?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-10 05:26:11
 +
+

*Thread Reply:* If you can, it might be better to create issue at github and communicate there.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Zahi Fail + (zahi.fail@gmail.com) +
+
2023-08-10 08:34:01
 +
+

*Thread Reply:* Before creating an issue in GIT, I wanted to check if my issue only related to versions compatibility..

+ +

This is the sample of my test: +```from pyspark.sql import SparkSession +from pyspark.sql.functions import col

+ +

spark = SparkSession.builder\ + .config('spark.jars.packages', 'io.openlineage:openlineage_spark:1.0.0') \ + .config('spark.extraListeners', 'io.openlineage.spark.agent.OpenLineageSparkListener') \ + .config('spark.openlineage.host', 'http://localhost:9000') \ +.config('spark.openlineage.namespace', 'default') \ + .getOrCreate()

+ +

spark.sparkContext.setLogLevel("DEBUG")

+ +

csv_file = location.csv

+ +

df = spark.read.format("csv").option("header","true").option("sep","^").load(csv_file)

+ +

df = df.select("campaignid","revenue").groupby("campaignid").sum("revenue").show()``` +Part of the logs with the OL configurations and the processed event

+ +
+
+ + + + + + +
+
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-10 08:40:13
 +
+

*Thread Reply:* try spark.openlineage.transport.url instead of spark.openlineage.host

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-10 08:40:27
 +
+

*Thread Reply:* and possibly link the doc where you've seen spark.openlineage.host 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Zahi Fail + (zahi.fail@gmail.com) +
+
2023-08-10 08:59:27
 +
+

*Thread Reply:* https://openlineage.io/blog/openlineage-spark/

+
+
openlineage.io
+ + + +
Tracing Data Lineage with OpenLineage and Apache Spark | OpenLineage
+
+ Spark ushered in a brand new age of data democratization... and left us with a mess of hidden dependencies, stale datasets, and failed jobs. +
+ + + + + + +
Original URL: https://openlineage.io/blog/openlineage-spark/
+ + + +
+ +
+
+ + + + + + +
+ + +
+ 👍 Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Zahi Fail + (zahi.fail@gmail.com) +
+
2023-08-10 09:04:56
 +
+

*Thread Reply:* changing to “spark.openlineage.transport.url” didn’t make any change

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-10 09:09:42
 +
+

*Thread Reply:* do you see the ConsoleTransport log? it suggests Spark integration did not register that you want to send events to Marquez

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-10 09:10:09
 +
+

*Thread Reply:* let's try adding spark.openlineage.transport.type to http

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Zahi Fail + (zahi.fail@gmail.com) +
+
2023-08-10 09:14:50
 +
+

*Thread Reply:* Now it works !

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Zahi Fail + (zahi.fail@gmail.com) +
+
2023-08-10 09:14:58
 +
+

*Thread Reply:* thanks @Maciej Obuchowski

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-10 09:23:04
 +
+

*Thread Reply:* Cool 🙂 however it should not require it if you provide spark.openlineage.transport.url - I'll create issue for debugging that.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-08-09 14:37:24
 +
+

@channel +This month’s TSC meeting is tomorrow! All are welcome. https://openlineage.slack.com/archives/C01CK9T7HKR/p1691422200847979

+
+ + +
+ + + } + + Michael Robinson + (https://openlineage.slack.com/team/U02LXF3HUN7) +
+ + + +
+
+ <p>@channel +This month’s TSC meeting is this Thursday, August 10th at 10:00 a.m. PT. On the tentative agenda: +• announcements +• recent releases +• Airflow provider progress update +• OpenLineage 1.0 overview +• open discussion +• more (TBA) +More info and the meeting link can be found on the <a href="https://openlineage.io/meetings/">website</a>. All are welcome! Also, feel free to reply or DM me with discussion topics, agenda items, etc.</p> +
+ + + + + + +
Original URL: https://openlineage.slack.com/archives/C01CK9T7HKR/p1691422200847979
+ + +
+ + Slack Conversation +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Athitya Kumar + (athityakumar@gmail.com) +
+
2023-08-10 02:11:07
 +
+

While using the spark integration, we're unable to see the query in the job facet for any spark-submit - is this a known issue/limitation, and can someone point to the code where this is currently extracted / can be enhanced?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-08-10 02:55:46
 +
+

*Thread Reply:* Let me first rephrase my understanding of the question assume a user runs spark.sql('INSERT INTO ...'). Are we able to include sql queryINSERT INTO ...within SQL facet?

+ +

We once had a look at it and found it difficult. Given an SQL, spark immediately translates it to a logical plan (which our integration is based on) and we didn't find any place where we could inject our code and get access to sql being run.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Athitya Kumar + (athityakumar@gmail.com) +
+
2023-08-10 04:27:51
 +
+

*Thread Reply:* Got it. So for spark.sql() - there's no interaction with sqlparser-rs and we directly try stitching the input/output & column lineage from the spark logical plan. Would something like this fall under the spark.jdbc() route or the spark.sql() route (say, if the df is collected / written somewhere)?

+ +

val df = spark.read.format("jdbc") + .option("url", url) + .option("user", user) + .option("password", password) + .option("fetchsize", fetchsize) + .option("driver", driver)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-10 05:15:17
 +
+

*Thread Reply:* @Athitya Kumar I understand your issue. From my side, there's one problem with this - potentially there can be multiple queries for one spark job. You can imagine something like joining results of two queries - possible to separate systems - and then one SqlJobFacet would be misleading. This needs more thorough spec discussion

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Luigi Scorzato + (luigi.scorzato@gmail.com) +
+
2023-08-10 05:33:47
 +
+

Hi Team, has anyone experience with integrating OpenLineage with the SAP ecosystem? And with Salesforce/MuleSoft?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Steven + (xli@zjuici.com) +
+
2023-08-10 05:40:47
 +
+

Hi, +Are there any ways to save list of string directly in the dataset facets? Such as the myfacets field in this dict +"facets": { + "metadata_facet": { + "_producer": "<https://github.com/OpenLineage/OpenLineage/tree/0.29.2/client/python>", + "_schemaURL": "<https://sth/schemas/facets.json#/definitions/SomeFacet>", + "myfacets": ["a", "b", "c"] + } + }

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Steven + (xli@zjuici.com) +
+
2023-08-10 05:42:20
 +
+

*Thread Reply:* I'm using python OpenLineage package and extend the BaseFacet class

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-10 05:53:57
 +
+

*Thread Reply:* for custom facets, as long as it's valid json - go for it

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Steven + (xli@zjuici.com) +
+
2023-08-10 05:55:03
 +
+

*Thread Reply:* However I tried to insert a list of string. And I tried to get the dataset, the returned valued of that list field is empty.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Steven + (xli@zjuici.com) +
+
2023-08-10 05:55:57
 +
+

*Thread Reply:* @attr.s +class MyFacet(BaseFacet): + columns: list[str] = attr.ib() +Here's my python code.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-10 05:59:02
 +
+

*Thread Reply:* How did you emit, serialized the event, and where did you look when you said you tried to get the dataset?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-10 06:00:27
 +
+

*Thread Reply:* I assume the problem is somewhere there, not on the level of facet definition, since SchemaDatasetFacet looks pretty much the same and it works

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Steven + (xli@zjuici.com) +
+
2023-08-10 06:00:54
 +
+

*Thread Reply:* I use the python openlineage client to emit the RunEvent. +openlineage_client.emit( + RunEvent( + eventType=RunState.COMPLETE, + eventTime=datetime.now().isoformat(), + run=run, + job=job, + producer=PRODUCER, + outputs=outputs, + ) + ) +And use marquez to visualize the get data result

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Steven + (xli@zjuici.com) +
+
2023-08-10 06:02:12
 +
+

*Thread Reply:* Yah, list of objects is working, but list of string is not.😩

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Steven + (xli@zjuici.com) +
+
2023-08-10 06:03:23
 +
+

*Thread Reply:* I think the problem is related to the openlineage package openlineage.client.serde.py. The function Serde.to_json()

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Steven + (xli@zjuici.com) +
+
2023-08-10 06:05:56
 +
+

*Thread Reply:*

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Steven + (xli@zjuici.com) +
+
2023-08-10 06:19:34
 +
+

*Thread Reply:* I think the code here filters out those string values in the list

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-10 06:21:39
 +
+

*Thread Reply:* 👀

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Steven + (xli@zjuici.com) +
+
2023-08-10 06:24:48
 +
+

*Thread Reply:* Yah, the value in list will end up False in this code and be filtered out +isinstance(_x_, dict)

+ +

😳

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-10 06:26:33
 +
+

*Thread Reply:* wow, that's right 😬

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-10 06:26:47
 +
+

*Thread Reply:* want to create PR fixing that?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Steven + (xli@zjuici.com) +
+
2023-08-10 06:27:20
 +
+

*Thread Reply:* Sure! May do this later tomorrow.

+ + + +
+ 👍 Maciej Obuchowski, Paweł Leszczyński +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Steven + (xli@zjuici.com) +
+
2023-08-10 23:59:28
 +
+

*Thread Reply:* I created the pr at https://github.com/OpenLineage/OpenLineage/pull/2044 +But the ci on integration-test-integration-spark FAILED

+
+ + + + +
#2044 fix: serde filtering
+
+ <p><strong>Problem</strong></p> + +<p>Values in list objects will be accidently filtered out so that they will not be included in the output.</p> + +<p><strong>Solution</strong></p> + +<p>Remove unnecessary instance type check (which filters out single value by mistake) in Serde.</p> + +<blockquote> + <p><strong>Note:</strong> All schema changes require discussion. Please <a href="https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue">link the issue</a> for context.</p> +</blockquote> + +<p>☐ Your change modifies the <a href="https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.json">core</a> OpenLineage model +☐ Your change modifies one or more OpenLineage <a href="https://github.com/OpenLineage/OpenLineage/tree/main/spec/facets">facets</a></p> + +<p><strong>One-line summary:</strong></p> + +<p>Fix the bug that causes values in list objects being accidently filtered.</p> + +<p><strong>Checklist</strong></p> + +<p>☐ You've <a href="https://github.com/OpenLineage/OpenLineage/blob/main/why-the-dco.md">signed-off</a> your work +☐ Your pull request title follows our <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CONTRIBUTING.md#creating-pull-requests">guidelines</a></p> + +<hr /> + +<p>SPDX-License-Identifier: Apache-2.0 <br /> +Copyright 2018-2023 contributors to the OpenLineage project</p> +
+ +
+
Labels
+ client/python +
+ +
+
Comments
+ 1 +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-11 04:17:01
 +
+

*Thread Reply:* @Steven sorry for that - some tests require credentials that are not present on the forked versions of CI. It will work once I push it to origin. Anyway Spark tests failing aren't blocker for this Python PR

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-11 04:17:45
 +
+

*Thread Reply:* I would only ask to add some tests for that case with facets containing list of string

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Steven + (xli@zjuici.com) +
+
2023-08-11 04:18:21
 +
+

*Thread Reply:* Yeah sure, I will add them now

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-11 04:25:19
 +
+

*Thread Reply:* ah we had other CI problem, go version was too old in one of the jobs - neverthless I won't judge your PR on stuff failing outside your PR anyway 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Steven + (xli@zjuici.com) +
+
2023-08-11 04:36:57
 +
+

*Thread Reply:* LOL🤣 I've added some tests and made a force push

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
savan + (SavanSharan_Navalgi@intuit.com) +
+
2023-10-20 08:31:45
 +
+

*Thread Reply:* @GitHubOpenLineageIssues +I am trying to contribute to Integration tests which is listed here as good first issue +the CONTRIBUTING.md mentions that i can trigger CI for integration tests from forked branch. +using this tool. +but i am unable to do so, is there a way to trigger CI from forked brach or do i have to get permission from someone to run the CI?

+ +

i am getting this error when i run this command sudo git-push-fork-to-upstream-branch upstream savannavalgi:hacktober +> Username for '<https://github.com>': savannavalgi +&gt; Password for '<https://savannavalgi@github.com>': +&gt; remote: Permission to OpenLineage/OpenLineage.git denied to savannavalgi. +&gt; fatal: unable to access '<https://github.com/OpenLineage/OpenLineage.git/>': The requested URL returned error: 403 +i have tried to configure ssh key +also tried to trigger CI from another brach, +and tried all of this after fetching the latest upstream

+ +

cc: @Athitya Kumar @Maciej Obuchowski @Steven

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-10-23 04:57:44
 +
+

*Thread Reply:* what PR is the probem related to? I can run git-push-fork-to-upstream-branch for you

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
savan + (SavanSharan_Navalgi@intuit.com) +
+
2023-10-25 01:08:41
 +
+

*Thread Reply:* @Paweł Leszczyński thanks for approving my PR - ( link )

+ +

I will make the changes needed for the new integration test case for drop table (good first issue) , in another PR, +I would need your help to run the integration tests again, thank you

+
+ + + + +
#2143 Integration test for drop table does not verify anything
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
savan + (SavanSharan_Navalgi@intuit.com) +
+
2023-10-26 07:48:52
 +
+

*Thread Reply:* @Paweł Leszczyński +opened a PR ( link ) for integration test for drop table +can you please help run the integration test

+
+ + + + +
#2214 spark: integration test case for drop table
+
+ <p><strong>Problem</strong></p> + +<p>drop table integration test did not verify the drop table events generated by openlineage.</p> + +<p>Closes: <a href="https://github.com/OpenLineage/OpenLineage/issues/2143">#2143</a></p> + +<p><strong>Solution</strong></p> + +<p>for drop table the COMPLETE event generated has empty inputs and outputs facet.</p> + +<p>but the START event has the table to be dropped/deleted in the outputs facet and a empty inputs facet.</p> + +<p><strong>One-line summary:</strong></p> + +<p>added START event for the drop table integration test</p> + +<p><strong>Checklist</strong></p> + +<p>☑︎ You've <a href="https://github.com/OpenLineage/OpenLineage/blob/main/why-the-dco.md">signed-off</a> your work +☑︎ Your pull request title follows our <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CONTRIBUTING.md#creating-pull-requests">guidelines</a> +☑︎ Your changes are accompanied by tests (<em>if relevant</em>) +☑︎ Your change contains a <a href="https://kurtisnusbaum.medium.com/stacked-diffs-keeping-phabricator-diffs-small-d9964f4dcfa6">small diff</a> and is self-contained +☑︎ You've updated any relevant documentation (<em>if relevant</em>) +☑︎ Your comment includes a one-liner for the changelog about the specific purpose of the change (<em>if necessary</em>) +☐ You've versioned the core OpenLineage model or facets according to <a href="https://docs.snowplowanalytics.com/docs/pipeline-components-and-applications/iglu/common-architecture/schemaver">SchemaVer</a> (<em>if relevant</em>) +☐ You've added a <a href="https://github.com/OpenLineage/OpenLineage/tree/main/.github/header_templates.md">header</a> to source files (<em>if relevant</em>)</p> + +<hr /> + +<p>SPDX-License-Identifier: Apache-2.0 <br /> +Copyright 2018-2023 contributors to the OpenLineage project</p> +
+ +
+
Labels
+ documentation, integration/spark +
+ +
+
Comments
+ 1 +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-10-26 07:50:29
 +
+

*Thread Reply:* sure, some of our tests require access to S3/BigQuery secret keys, so will not work automatically from the fork, and require action on our side. working on that

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
savan + (SavanSharan_Navalgi@intuit.com) +
+
2023-10-29 09:31:22
 +
+

*Thread Reply:* thanks @Paweł Leszczyński +let me know if i can help in any way

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
savan + (SavanSharan_Navalgi@intuit.com) +
+
2023-11-15 02:31:50
 +
+

*Thread Reply:* @Paweł Leszczyński any action item on my side?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Athitya Kumar + (athityakumar@gmail.com) +
+
2023-08-11 07:36:57
 +
+

Hey folks! 👋

+ +

Had a query/observation regarding columnLineage inferred in spark integration - opened this issue for the same. Basically, when we do something like this in our spark-sql: +SELECT t1.c1, t1.c2, t1.c3, t2.c4 FROM t1 LEFT JOIN t2 ON t1.c1 = t2.c1 AND t1.c2 = t2.c2 +The expected column lineage for output table t3 is: +t3.c1 -&gt; Comes from both t1.c1 &amp; t2.c1 (SELECT + JOIN clause) +t3.c2 -&gt; Comes from both t1.c2 &amp; t2.c2 (SELECT + JOIN clause) +t3.c3 -&gt; Comes from t1.c3 +t3.c4 -&gt; Comes from t2.c4 +However, actual column lineage for output table t3 is: +t3.c1 -&gt; Comes from t1.c1 (Only based on SELECT clause) +t3.c2 -&gt; Comes from t1.c1 (Only based on SELECT clause) +t3.c3 -&gt; Comes from t1.c3 +t3.c4 -&gt; Comes from t2.c4 +Is this a known issue/behaviour?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-11 09:18:44
 +
+

*Thread Reply:* Hmm... this is kinda "logical" difference - is column level lineage taken from actual "physical" operations - like in this case, we always take from t1 - or from "logical" where t2 is used only for predicate, yet we still want to indicate it as a source?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-11 09:18:58
 +
+

*Thread Reply:* I think your interpretation is more useful

+ + + +
+ 🙏 Athitya Kumar +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Athitya Kumar + (athityakumar@gmail.com) +
+
2023-08-11 09:25:03
 +
+

*Thread Reply:* @Maciej Obuchowski - Yup, especially for use-cases where we wanna depend on column lineage for impact analysis, I think we should be considering even predicates. For example, if t2.c1 / t2.c2 gets corrupted or dropped, the query would be impacted - which means that we should be including even predicates (t2.c1 / t2.c2) in the column lineage imo

+ +

But is there any technical limitation if we wanna implement this / make an OSS contribution for this (like logical predicate columns not being part of the spark logical plan object that we get in the PlanVisitor or something like that)?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-11 11:14:58
 +
+

*Thread Reply:* It's probably a bit of work, but can't think it's impossible on parser side - @Paweł Leszczyński will know better about spark collection

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ernie Ostic + (ernie.ostic@getmanta.com) +
+
2023-08-11 12:45:34
 +
+

*Thread Reply:* This is a case where it would be nice to have an alternate indication (perhaps in the Column lineage facet?) for this type of "suggested" lineage. As noted, this is especially important for impact analysis purposes. We (and I believe others do the same or similar) call that "indirect" lineage at Manta.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-11 12:49:10
 +
+

*Thread Reply:* Something like additional flag in inputFields, right?

+ + + +
+ 👍 Athitya Kumar, Ernie Ostic, Paweł Leszczyński +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-08-14 02:36:34
 +
+

*Thread Reply:* Yes, this would require some extension to the spec. What do you mean spark-sql : spark.sql() with some spark query or SQL in spark JDBC?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Athitya Kumar + (athityakumar@gmail.com) +
+
2023-08-15 15:16:49
 +
+

*Thread Reply:* Sorry, missed your question @Paweł Leszczyński. By spark-sql, I'm referring to the former: spark.sql() with some spark query

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-08-16 03:10:57
 +
+

*Thread Reply:* cc @Jens Pfau - you may be also interested in extending column level lineage facet.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-08-22 02:23:08
 +
+

*Thread Reply:* Hi, is there a github issue for this feature? Seems like a really cool and exciting functionality to have!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Athitya Kumar + (athityakumar@gmail.com) +
+
2023-08-22 08:03:49
 +
+

*Thread Reply:* @Anirudh Shrinivason - Are you referring to this issue: https://github.com/OpenLineage/OpenLineage/issues/2048?

+
+ + + + +
#2048 Spark: Column lineage is not taking join fields into consideration for `inputFields` of an output field
+
+ <p>Hey folks! 👋</p> + +<p>Had a query/observation regarding columnLineage inferred in spark integration. When we do something like this in our spark-sql:</p> + +<pre><code>SELECT t1.c1, t1.c2, t1.c3, t2.c4 FROM t1 LEFT JOIN t2 ON t1.c1 = t2.c1 AND t1.c2 = t2.c2 +</code></pre> + +<p>Expected column lineage for output table t3 is:</p> + +<pre><code>t3.c1 -&gt; Comes from both t1.c1 &amp; t2.c1 (SELECT + JOIN clause) +t3.c2 -&gt; Comes from both t1.c2 &amp; t2.c2 (SELECT + JOIN clause) +t3.c3 -&gt; Comes from t1.c3 +t3.c4 -&gt; Comes from t2.c4 +</code></pre> + +<p>However, the actual column lineage for output table t3 is:</p> + +<pre><code>t3.c1 -&gt; Comes from t1.c1 (Only based on SELECT clause) +t3.c2 -&gt; Comes from t1.c1 (Only based on SELECT clause) +t3.c3 -&gt; Comes from t1.c3 +t3.c4 -&gt; Comes from t2.c4 +</code></pre> + +<p>Is this a known issue/behaviour?</p> +
+ +
+
Comments
+ 2 +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+ :gratitude_thank_you: Anirudh Shrinivason +
+ +
+ ✅ Anirudh Shrinivason +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Athitya Kumar + (athityakumar@gmail.com) +
+
2023-08-14 05:13:48
 +
+

Hey team 👋

+ +

Is there a way we can feed the logical plan directly to check the open-lineage events being built, without actually running a spark-job with open-lineage configs? Basically interested to see if we can mock a dry-run of a spark job w/ open-lineage by mimicking the logical plan 😄

+ +

cc @Shubh

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-14 06:00:21
 +
+

*Thread Reply:* Not really I think - the integration does not rely purely on the logical plan

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-14 06:00:44
 +
+

*Thread Reply:* At least, not in all cases. For some maybe

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-08-14 07:34:39
 +
+

*Thread Reply:* We're using pretty similar approach in our column level lineage tests where we run some spark commands, register custom listener https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/app/src/tes[…]eage/spark/agent/util/LastQueryExecutionSparkEventListener.java which catches the logical plan. Further we run our tests on the captured logical plan.

+ +

The difference here, between what you're asking about, is that we still have an access to the same spark session.

+ +

In many cases, our integration uses active Spark session to fetch some dataset details. This happens pretty often (like fetch dataset location) and cannot be taken just from a Logical Plan.

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/app/src/test/java/io/openlineage/spark/agent/util/LastQueryExecutionSparkEventListener.java | LastQueryExecutionSparkEventListener.java>
+
+ <pre><code>/** +/** Copyright 2018-2022 contributors to the OpenLineage project +/** SPDX-License-Identifier: Apache-2.0 +**/ + +package io.openlineage.spark.agent.util; + +import java.util.ArrayList; +import java.util.List; +import java.util.Optional; +import org.apache.spark.scheduler.SparkListener; +import org.apache.spark.scheduler.SparkListenerEvent; +import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan; +import org.apache.spark.sql.execution.QueryExecution; +import org.apache.spark.sql.execution.SQLExecution; +import org.apache.spark.sql.execution.ui.SparkListenerSQLExecutionStart; + +public class LastQueryExecutionSparkEventListener extends SparkListener { + + private static List&lt;LogicalPlan&gt; queryExecutions = new ArrayList&lt;&gt;(); + + @Override + public void onOtherEvent(SparkListenerEvent event) { + if (event instanceof SparkListenerSQLExecutionStart) { + QueryExecution queryExecution = + SQLExecution.getQueryExecution(((SparkListenerSQLExecutionStart) event).executionId()); + + if (queryExecution != null) { + queryExecutions.add(queryExecution.optimizedPlan()); + } + } + } + + public static Optional&lt;LogicalPlan&gt; getLastExecutedLogicalPlan() { + if (queryExecutions.isEmpty()) { + return Optional.empty(); + } else { + return Optional.of(queryExecutions.get(queryExecutions.size() - 1)); + } + } + + public static List&lt;LogicalPlan&gt; getExecutedLogicalPlans() { + return queryExecutions; + } +} + +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Athitya Kumar + (athityakumar@gmail.com) +
+
2023-08-14 11:03:28
 +
+

*Thread Reply:* @Paweł Leszczyński - We're mainly interested to see the inputs/outputs (mainly column schema and column lineage) for different logical plans. Is that something that could be done in a static manner without running spark jobs in your opinion?

+ +

For example, I know that we can statically create logical plans

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-08-16 03:05:44
 +
+

*Thread Reply:* The more we talk the more I am wondering what is the purpose of doing so? Do you want to test openlineage coverage or is there any production scenario where you would like to apply this?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Athitya Kumar + (athityakumar@gmail.com) +
+
2023-08-16 04:01:39
 +
+

*Thread Reply:* @Paweł Leszczyński - This is for testing openlineage coverage so that we can be more confident on what're the happy path scenarios and what're the scenarios where it may not work / work partially etc

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-08-16 04:22:01
 +
+

*Thread Reply:* If this is for testing, then you're also capable of mocking some SparkSession/catalog methods when Openlineage integration tries to access them. If you want to reuse LogicalPlans from your prod environment, you will encounter logicalplan serialization issues. On the other hand, if you generate logical plans from some example Spark jobs, then the same can be easier achieved in a way the integration tests are run with mockserver.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-08-14 09:45:31
 +
+

Hi Team,

+ +

Spark & Databricks related question: Starting 1st September Databricks is going to block running init_scripts located in dbfs and this is the way our integration works (https://www.databricks.com/blog/securing-databricks-cluster-init-scripts).

+ +

We have two ways of mitigating this in our docs and quickstart: + (1) move initscripts to workspace + (2) move initscripts to S3

+ +

None of them is perfect. (1) requires creating init_script file manually through databricks UI and copy/paste its content. I couldn't find the way to load it programatically. (2) requires quickstart user to have s3 bucket access.

+ +

Would love to hear your opinion on this. Perhaps there's some better way to do that. Thanks. `

+
+
Databricks
+ + + +
Securing Databricks cluster init scripts
+
+ Protecting the Databricks platform and continuously raising the bar with security improvements is the mission of our Security team and the main reason why we invest in our bug bounty program. +
+ + + + + + + + +
Original URL: https://www.databricks.com/blog/securing-databricks-cluster-init-scripts
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-08-15 01:13:49
 +
+

*Thread Reply:* We're uploading the init scripts to s3 via tf. But yeah ig there are some access permissions that the user needs to have

+ + + +
+ :gratitude_thank_you: Paweł Leszczyński +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Abdallah + (abdallah@terrab.me) +
+
2023-08-16 07:32:00
 +
+

*Thread Reply:* Hello +I am new here and I am asking why do you need an init script ? +If it's a spark integration we can just specify --package=io.openlineage...

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-08-16 07:41:25
 +
+

*Thread Reply:* https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/databricks/open-lineage-init-script.sh -> I think the issue was in having openlineage-jar installed immediately on the classpath bcz it's required when OpenLineageSparkListener is instantiated. It didn't work without it.

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/databricks/open-lineage-init-script.sh | open-lineage-init-script.sh>
+
+ <pre><code>#!/bin/bash +# +# Copyright 2018-2023 contributors to the OpenLineage project +# SPDX-License-Identifier: Apache-2.0 + +STAGE_DIR="/dbfs/databricks/openlineage" + +echo "BEGIN: Upload Spark Listener JARs" +cp -f $STAGE_DIR/openlineage-spark-**.jar /mnt/driver-daemon/jars || { echo "Error copying Spark Listener library file"; exit 1;} +echo "END: Upload Spark Listener JARs" + +echo "BEGIN: Modify Spark config settings" +cat &lt;&lt; 'EOF' &gt; /databricks/driver/conf/openlineage-spark-driver-defaults.conf +[driver] { + "spark.extraListeners" = "io.openlineage.spark.agent.OpenLineageSparkListener" +} +EOF +echo "END: Modify Spark config settings" + +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Abdallah + (abdallah@terrab.me) +
+
2023-08-16 07:43:55
 +
+

*Thread Reply:* Yes it happens if you use --jars s3://.../...openlineage-spark-VERSION.jar parameter. (I made a ticket for this issue in Databricks support) +But if you use --package io.openlineage... (the package will be downloaded from maven) it works fine.

+ + + +
+ 👀 Paweł Leszczyński +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Abdallah + (abdallah@terrab.me) +
+
2023-08-16 07:47:50
 +
+

*Thread Reply:* I think they don't use the right class loader.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-08-16 08:36:14
 +
+

*Thread Reply:* To make sure: are you able to run Openlineage & Spark on Databricks Runtime without init_scripts?

+ +

I was doing this a second ago and this ended up with Caused by: java.lang.ClassNotFoundException: io.openlineage.spark.agent.OpenLineageSparkListener not found in com.databricks.backend.daemon.driver.ClassLoaders$LibraryClassLoader@1609ed55

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Alexandre Campelo + (aleqi200@gmail.com) +
+
2023-08-14 19:49:00
 +
+

Hello, I just downloaded Marquez and I'm trying to send a sample request but I'm getting a 403 (forbidden). Any idea how to find the authentication details?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Alexandre Campelo + (aleqi200@gmail.com) +
+
2023-08-15 12:19:34
 +
+

*Thread Reply:* Ok, nevermind. I figured it out. The port 5000 is reserved in MACOS so I had to start on port 9000 instead.

+ + + +
+ 👍 Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-08-15 01:25:48
 +
+

Hi, I noticed that while capturing lineage for merge into commands, some of the tables/columns are unaccounted for the lineage. Example: +```fdummyfunnelstg = spark.sql("""WITH dummyfunnel AS ( + SELECT ** + FROM fdummyfunnelone + WHERE dateid BETWEEN {startdateid} AND {enddateid}

+ +
        UNION ALL
+
+        SELECT **
+        FROM f_dummy_funnel_two
+        WHERE date_id BETWEEN {start_date_id} AND {end_date_id}
+
+        UNION ALL
+
+        SELECT **
+        FROM f_dummy_funnel_three
+        WHERE date_id BETWEEN {start_date_id} AND {end_date_id}
+
+        UNION ALL
+
+        SELECT **
+        FROM f_dummy_funnel_four
+        WHERE date_id BETWEEN {start_date_id} AND {end_date_id}
+
+        UNION ALL
+
+        SELECT **
+        FROM f_dummy_funnel_five
+        WHERE date_id BETWEEN {start_date_id} AND {end_date_id}
+
+    )
+    SELECT DISTINCT
+        dummy_funnel.customer_id,
+        dummy_funnel.product,
+        dummy_funnel.date_id,
+        dummy_funnel.country_id,
+        dummy_funnel.city_id,
+        dummy_funnel.dummy_type_id,
+        dummy_funnel.num_attempts,
+        dummy_funnel.num_transactions,
+        dummy_funnel.gross_merchandise_value,
+        dummy_funnel.sub_category_id,
+        dummy_funnel.is_dummy_flag
+    FROM dummy_funnel
+    INNER JOIN d_dummy_identity as dummy_identity
+        ON dummy_identity.id = dummy_funnel.customer_id
+    WHERE
+        date_id BETWEEN {start_date_id} AND {end_date_id}""")
+
+ +

spark.sql(f""" + MERGE INTO {tablename} + USING fdummyfunnelstg + ON + fdummyfunnelstg.customerid = {tablename}.customerid + AND fdummyfunnelstg.product = {tablename}.product + AND fdummyfunnelstg.dateid = {tablename}.dateid + AND fdummyfunnelstg.countryid = {tablename}.countryid + AND fdummyfunnelstg.cityid = {tablename}.cityid + AND fdummyfunnelstg.dummytypeid = {tablename}.dummytypeid + AND fdummyfunnelstg.subcategoryid = {tablename}.subcategoryid + AND fdummyfunnelstg.isdummyflag = {tablename}.isdummyflag + WHEN MATCHED THEN + UPDATE SET + {tablename}.numattempts = fdummyfunnelstg.numattempts + , {tablename}.numtransactions = fdummyfunnelstg.numtransactions + , {tablename}.grossmerchandisevalue = fdummyfunnelstg.grossmerchandisevalue + WHEN NOT MATCHED + THEN INSERT ( + customerid, + product, + dateid, + countryid, + cityid, + dummytypeid, + numattempts, + numtransactions, + grossmerchandisevalue, + subcategoryid, + isdummyflag + ) + VALUES ( + fdummyfunnelstg.customerid, + fdummyfunnelstg.product, + fdummyfunnelstg.dateid, + fdummyfunnelstg.countryid, + fdummyfunnelstg.cityid, + fdummyfunnelstg.dummytypeid, + fdummyfunnelstg.numattempts, + fdummyfunnelstg.numtransactions, + fdummyfunnelstg.grossmerchandisevalue, + fdummyfunnelstg.subcategoryid, + fdummyfunnelstg.isdummyflag + ) + """)`` +In cases like this, I notice that the full lineage is not actually captured... I'd expect to see this having 5 upstreams:dummyfunnelone, dummyfunneltwo, dummyfunnelthree, dummyfunnelfour, dummyfunnel_five` , but I notice only 1-2 upstreams for this case... +Would like to learn more about why this might happen, and whether this is expected behaviour or not. Thanks!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-15 06:48:43
 +
+

*Thread Reply:* Would be useful to see generated event or any logs

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-08-16 03:09:05
 +
+

*Thread Reply:* @Anirudh Shrinivason what if there is just one union instead of four? What if there are just two columns selected instead of 10? What if inner join is skipped? Does merge into matter?

+ +

The smaller SQL to reproduce the problem, the easier it is to find the root cause. Most of the issues are reproducible with just few lines of code.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-08-16 03:34:30
 +
+

*Thread Reply:* Yup let me try to identify the cause from my end. Give me some time haha. I'll reach out again once there is more clarity on the occurence

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Abdallah + (abdallah@terrab.me) +
+
2023-08-16 07:33:21
 +
+

Hello,

+ +

The OpenLineage Databricks integration is not working properly in our side which due to filtering adaptive_spark_plan

+ +

Please find the issue link.

+ +

https://github.com/OpenLineage/OpenLineage/issues/2058

+
+ + + + +
#2058 Issue with Event Filtering in Databricks Integration for `adaptive_spark_plan`
+
+ <p>Hello OpenLineage team,</p> + +<p>While implementing the current OpenLineage Spark integration in Databricks, we've made an interesting observation regarding event filtering and its behavior with narrow and wide transformations:</p> + +<ol><li>For a <strong>narrow transformation</strong> (without shuffle), the event is <code>execute_insert_into_hadoop_fs_relation_command</code> and it doesn't include <code>adaptive_spark_plan</code>.</li><li>For a <strong>wide transformation</strong> (with shuffle), the event is <code>adaptive_spark_plan</code> and doesn't have <code>execute_insert_into_hadoop_fs_relation_command</code>.</li> +</ol> + +<p>We've identified that the <code>adaptive_spark_plan</code> is actively filtered out in the <code>DatabricksEventFilter</code> class. This results in the event being completely ignored due to this line:</p> + +<p><a href="https://github.com/OpenLineage/OpenLineage/blob/41dbbf1799595bd9cd1567df0a7027de08619741/integration/spark/shared/src/main/java/io/openlineage/spark/agent/filters/DatabricksEventFilter.java#L30">OpenLineage/integration/spark/shared/src/main/java/io/openlineage/spark/agent/filters/DatabricksEventFilter.java</a></p> + +<p>Line 30 in </OpenLineage/OpenLineage/commit/41dbbf1799595bd9cd1567df0a7027de08619741|41dbbf1></p> + +<p>To validate, we compiled a JAR without this filtering line, and the integration worked perfectly from Databricks.</p> + +<p><strong>Example Spark Jobs:</strong></p> + +<ol><li><strong>Narrow Transformation</strong> +<pre><code>val data = Seq((1,"a"),(2,"b"),(3,"c")) +val rdd = spark.sparkContext.parallelize(data) +val df = rdd.toDF("id", "value") +df.write.parquet("/path/to/output/narrow_transformation") +</code></pre></li><li><strong>Wide Transformation</strong> +<pre><code>val data = Seq((1,"a"),(2,"b"),(3,"c"),(1,"d")) +val rdd = spark.sparkContext.parallelize(data) +val df = rdd.toDF("id", "value") +df.groupBy("id").count().write.parquet("/path/to/output/wide_transformation") +</code></pre></li> +</ol> + +<p><strong>Proposed Solutions</strong>:</p> + +<p>• Short-term fix: Remove adaptive<em>spark</em>plan from the excludedNodes in DatabricksEventFilter. +• Long-term resolution: Consider integrating the proposal from <a href="https://github.com/OpenLineage/OpenLineage/issues/2056">#2056</a> for a more universal filtering mechanism.</p> + +<p>Environment: <br /> +Databricks version: 12.2 LTS (includes Apache Spark 3.3.2, Scala 2.12)</p> + +<p>I hope this provides a clear picture of the issue at hand. We're eager to collaborate on resolving this.</p> + +<p>Thank you!</p> + +<p>Best regards,</p> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+ ⬆️ Mouad MOUSSABBIH, Abdallah +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harel Shein + (harel.shein@gmail.com) +
+
2023-08-16 09:24:09
 +
+

*Thread Reply:* thanks @Abdallah for the thoughtful issue that you submitted! +was wondering if you’d consider opening up a PR? would love to help you as a contributor is that’s something you are interested in.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Abdallah + (abdallah@terrab.me) +
+
2023-08-17 11:59:51
 +
+

*Thread Reply:* Hello

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Abdallah + (abdallah@terrab.me) +
+
2023-08-17 11:59:58
 +
+

*Thread Reply:* Yes I am working on it

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Abdallah + (abdallah@terrab.me) +
+
2023-08-17 12:00:14
 +
+

*Thread Reply:* I deleted the line that has that filter.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Abdallah + (abdallah@terrab.me) +
+
2023-08-17 12:00:24
 +
+

*Thread Reply:* I am adding some tests now

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Abdallah + (abdallah@terrab.me) +
+
2023-08-17 12:00:45
 +
+

*Thread Reply:* But running +./gradlew --no-daemon databricksIntegrationTest -x test -Pspark.version=3.4.0 -PdatabricksHost=$DATABRICKS_HOST -PdatabricksToken=$DATABRICKS_TOKEN

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Abdallah + (abdallah@terrab.me) +
+
2023-08-17 12:01:11
 +
+

*Thread Reply:* gives me +A problem occurred evaluating project ':app'. +&gt; Could not resolve all files for configuration ':app:spark33'. + &gt; Could not resolve io.openlineage:openlineage_java:1.1.0-SNAPSHOT. + Required by: + project :app &gt; project :shared + &gt; Could not resolve io.openlineage:openlineage_java:1.1.0-SNAPSHOT. + &gt; Unable to load Maven meta-data from <https://astronomer.jfrog.io/artifactory/maven-public-libs-snapshot/io/openlineage/openlineage-java/1.1.0-SNAPSHOT/maven-metadata.xml>. + &gt; org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 326; The reference to entity "display" must end with the ';' delimiter. + &gt; Could not resolve io.openlineage:openlineage_sql_java:1.1.0-SNAPSHOT. + Required by: + project :app &gt; project :shared + &gt; Could not resolve io.openlineage:openlineage_sql_java:1.1.0-SNAPSHOT. + &gt; Unable to load Maven meta-data from <https://astronomer.jfrog.io/artifactory/maven-public-libs-snapshot/io/openlineage/openlineage-sql-java/1.1.0-SNAPSHOT/maven-metadata.xml>. + &gt; org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 326; The reference to entity "display" must end with the ';' delimiter.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Abdallah + (abdallah@terrab.me) +
+
2023-08-17 12:01:25
 +
+

*Thread Reply:* And I am trying to understand what should I do.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Abdallah + (abdallah@terrab.me) +
+
2023-08-17 12:13:37
 +
+

*Thread Reply:* I am compiling sql integration

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Abdallah + (abdallah@terrab.me) +
+
2023-08-17 13:04:15
 +
+

*Thread Reply:* I built the java client

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Abdallah + (abdallah@terrab.me) +
+
2023-08-17 13:04:29
 +
+

*Thread Reply:* but having +A problem occurred evaluating project ':app'. +&gt; Could not resolve all files for configuration ':app:spark33'. + &gt; Could not resolve io.openlineage:openlineage_java:1.1.0-SNAPSHOT. + Required by: + project :app &gt; project :shared + &gt; Could not resolve io.openlineage:openlineage_java:1.1.0-SNAPSHOT. + &gt; Unable to load Maven meta-data from <https://astronomer.jfrog.io/artifactory/maven-public-libs-snapshot/io/openlineage/openlineage-java/1.1.0-SNAPSHOT/maven-metadata.xml>. + &gt; org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 326; The reference to entity "display" must end with the ';' delimiter. + &gt; Could not resolve io.openlineage:openlineage_sql_java:1.1.0-SNAPSHOT. + Required by: + project :app &gt; project :shared + &gt; Could not resolve io.openlineage:openlineage_sql_java:1.1.0-SNAPSHOT. + &gt; Unable to load Maven meta-data from <https://astronomer.jfrog.io/artifactory/maven-public-libs-snapshot/io/openlineage/openlineage-sql-java/1.1.0-SNAPSHOT/maven-metadata.xml>. + &gt; org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 326; The reference to entity "display" must end with the ';' delimiter.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-17 14:47:41
 +
+

*Thread Reply:* Please do ./gradlew publishToMavenLocal in client/java directory

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Abdallah + (abdallah@terrab.me) +
+
2023-08-17 14:47:59
 +
+

*Thread Reply:* Okay thanks

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Abdallah + (abdallah@terrab.me) +
+
2023-08-17 14:48:01
 +
+

*Thread Reply:* will do

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Abdallah + (abdallah@terrab.me) +
+
2023-08-22 10:33:02
 +
+

*Thread Reply:* Hello back

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Abdallah + (abdallah@terrab.me) +
+
2023-08-22 10:33:12
 +
+

*Thread Reply:* I created a databricks cluster.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Abdallah + (abdallah@terrab.me) +
+
2023-08-22 10:35:00
 +
+

*Thread Reply:* And I had somme issues that -PdatabricksHost doesn't work with System.getProperty("databricksHost") So I changed to -DdatabricksHost with System.getenv("databricksHost")

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Abdallah + (abdallah@terrab.me) +
+
2023-08-22 10:36:19
 +
+

*Thread Reply:* Then I had some issue that the path dbfs:/databricks/openlineage/ doesn't exist, I, then, created the folder /dbfs/databricks/openlineage/

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Abdallah + (abdallah@terrab.me) +
+
2023-08-22 10:38:03
 +
+

*Thread Reply:* And now I am investigating this issue : +java.lang.NullPointerException + at io.openlineage.spark.agent.DatabricksUtils.uploadOpenlineageJar(DatabricksUtils.java:226) + at io.openlineage.spark.agent.DatabricksUtils.init(DatabricksUtils.java:66) + at io.openlineage.spark.agent.DatabricksIntegrationTest.setup(DatabricksIntegrationTest.java:54) + at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) + at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) + at ... +worker.org.gradle.process.internal.worker.GradleWorkerMain.main(GradleWorkerMain.java:74) + Suppressed: com.databricks.sdk.core.DatabricksError: Missing required field: cluster_id + at app//com.databricks.sdk.core.error.ApiErrors.readErrorFromResponse(ApiErrors.java:48) + at app//com.databricks.sdk.core.error.ApiErrors.checkForRetry(ApiErrors.java:22) + at app//com.databricks.sdk.core.ApiClient.executeInner(ApiClient.java:236) + at app//com.databricks.sdk.core.ApiClient.getResponse(ApiClient.java:197) + at app//com.databricks.sdk.core.ApiClient.execute(ApiClient.java:187) + at app//com.databricks.sdk.core.ApiClient.POST(ApiClient.java:149) + at app//com.databricks.sdk.service.compute.ClustersImpl.delete(ClustersImpl.java:31) + at app//com.databricks.sdk.service.compute.ClustersAPI.delete(ClustersAPI.java:191) + at app//com.databricks.sdk.service.compute.ClustersAPI.delete(ClustersAPI.java:180) + at app//io.openlineage.spark.agent.DatabricksUtils.shutdown(DatabricksUtils.java:96) + at app//io.openlineage.spark.agent.DatabricksIntegrationTest.shutdown(DatabricksIntegrationTest.java:65) + at +...

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Abdallah + (abdallah@terrab.me) +
+
2023-08-22 10:39:22
 +
+

*Thread Reply:* Suppressed: com.databricks.sdk.core.DatabricksError: Missing required field: cluster_id

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Abdallah + (abdallah@terrab.me) +
+
2023-08-22 10:40:18
 +
+

*Thread Reply:* at io.openlineage.spark.agent.DatabricksUtils.uploadOpenlineageJar(DatabricksUtils.java:226)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Abdallah + (abdallah@terrab.me) +
+
2023-08-22 10:54:51
 +
+

*Thread Reply:* I did this !echo "xxx" &gt; /dbfs/databricks/openlineage/openlineage-spark-V.jar

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Abdallah + (abdallah@terrab.me) +
+
2023-08-22 10:55:29
 +
+

*Thread Reply:* To create some fake file that can be deleted in uploadOpenlineageJar function.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Abdallah + (abdallah@terrab.me) +
+
2023-08-22 10:56:09
 +
+

*Thread Reply:* Because if there is no file, this part fails +StreamSupport.stream( + workspace.dbfs().list("dbfs:/databricks/openlineage/").spliterator(), false) + .filter(f -&gt; f.getPath().contains("openlineage-spark")) + .filter(f -&gt; f.getPath().endsWith(".jar")) + .forEach(f -&gt; workspace.dbfs().delete(f.getPath()));

+ + + +
+ 😬 Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-22 11:47:17
 +
+

*Thread Reply:* does this work after +!echo "xxx" &gt; /dbfs/databricks/openlineage/openlineage-spark-V.jar +?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Abdallah + (abdallah@terrab.me) +
+
2023-08-22 11:47:36
 +
+

*Thread Reply:* Yes

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Abdallah + (abdallah@terrab.me) +
+
2023-08-22 19:02:05
 +
+

*Thread Reply:* I am now having another error in the driver

+ +

23/08/22 22:56:26 ERROR SparkContext: Error initializing SparkContext. +org.apache.spark.SparkException: Exception when registering SparkListener + at org.apache.spark.SparkContext.setupAndStartListenerBus(SparkContext.scala:3121) + at org.apache.spark.SparkContext.&lt;init&gt;(SparkContext.scala:835) + at com.databricks.backend.daemon.driver.DatabricksILoop$.$anonfun$initializeSharedDriverContext$1(DatabricksILoop.scala:362) +... + at com.databricks.DatabricksMain.main(DatabricksMain.scala:146) + at com.databricks.backend.daemon.driver.DriverDaemon.main(DriverDaemon.scala) +Caused by: java.lang.ClassNotFoundException: io.openlineage.spark.agent.OpenLineageSparkListener not found in com.databricks.backend.daemon.driver.ClassLoaders$LibraryClassLoader@298cfe89 + at com.databricks.backend.daemon.driver.ClassLoaders$MultiReplClassLoader.loadClass(ClassLoaders.scala:115) + at java.lang.ClassLoader.loadClass(ClassLoader.java:352) + at java.lang.Class.forName0(Native Method) + at java.lang.Class.forName(Class.java:348) + at org.apache.spark.util.Utils$.classForName(Utils.scala:263)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Abdallah + (abdallah@terrab.me) +
+
2023-08-22 19:19:29
 +
+

*Thread Reply:* Can you please share with me your json conf for the cluster ?

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Abdallah + (abdallah@terrab.me) +
+
2023-08-22 19:55:57
 +
+

*Thread Reply:* It's because in mu build file I have

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Abdallah + (abdallah@terrab.me) +
+
2023-08-22 19:56:27
 +
+

*Thread Reply:* and the one that was copied is

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Abdallah + (abdallah@terrab.me) +
+
2023-08-22 20:01:12
 +
+

*Thread Reply:* due to the findAny 😕 +private static void uploadOpenlineageJar(WorkspaceClient workspace) { + Path jarFile = + Files.list(Paths.get("../build/libs/")) + .filter(p -&gt; p.getFileName().toString().startsWith("openlineage-spark-")) + .filter(p -&gt; p.getFileName().toString().endsWith("jar")) + .findAny() + .orElseThrow(() -&gt; new RuntimeException("openlineage-spark jar not found"));

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Abdallah + (abdallah@terrab.me) +
+
2023-08-22 20:35:10
 +
+

*Thread Reply:* It works finally 😄

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Abdallah + (abdallah@terrab.me) +
+
2023-08-23 05:16:19
 +
+

*Thread Reply:* The PR 😄 +https://github.com/OpenLineage/OpenLineage/pull/2061

+
+ + + + +
#2061 Bug/fix ignored event adaptive spark plan databricks
+
+ <p><strong>Problem</strong></p> + +<p>While implementing the current OpenLineage Spark integration in Databricks, we've made an interesting observation regarding event filtering and its behavior with narrow and wide transformations:</p> + +<p>For a narrow transformation (without shuffle), the event is execute<em>insert</em>into<em>hadoop</em>fs<em>relation</em>command and it doesn't include adaptive<em>spark</em>plan. <br /> +For a wide transformation (with shuffle), the event is adaptive<em>spark</em>plan and doesn't have execute<em>insert</em>into<em>hadoop</em>fs<em>relation</em>command. <br /> +We've identified that the adaptive<em>spark</em>plan is actively filtered out in the DatabricksEventFilter class. This results in the event being completely ignored due to this line:</p> + +<p>Closes: <a href="https://github.com/OpenLineage/OpenLineage/issues/2058">#2058</a></p> + +<p><strong>Solution</strong></p> + +<p>Short-term fix: Remove adaptive<em>spark</em>plan from the excludedNodes in DatabricksEventFilter.</p> + +<p><strong>One-line summary:</strong></p> + +<p>Remove adaptive<em>spark</em>plan from the excludedNodes in DatabricksEventFilter.</p> + +<p><strong>Checklist</strong></p> + +<p>• [x ] You've <a href="https://github.com/OpenLineage/OpenLineage/blob/main/why-the-dco.md">signed-off</a> your work +☑︎ Your pull request title follows our <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CONTRIBUTING.md#creating-pull-requests">guidelines</a> +• [x ] Your changes are accompanied by tests (<em>if relevant</em>) +• [ x] Your change contains a <a href="https://kurtisnusbaum.medium.com/stacked-diffs-keeping-phabricator-diffs-small-d9964f4dcfa6">small diff</a> and is self-contained +☐ You've updated any relevant documentation (<em>if relevant</em>) +• [x ] Your comment includes a one-liner for the changelog about the specific purpose of the change (<em>if necessary</em>) +☐ You've versioned the core OpenLineage model or facets according to <a href="https://docs.snowplowanalytics.com/docs/pipeline-components-and-applications/iglu/common-architecture/schemaver">SchemaVer</a> (<em>if relevant</em>) +• [ x] You've added a <a href="https://github.com/OpenLineage/OpenLineage/tree/main/.github/header_templates.md">header</a> to source files (<em>if relevant</em>)</p> + +<hr /> + +<p>SPDX-License-Identifier: Apache-2.0 <br /> +Copyright 2018-2023 contributors to the OpenLineage project</p> +
+ +
+
Labels
+ integration/spark +
+ +
+
Comments
+ 1 +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-23 08:23:49
 +
+

*Thread Reply:* thanks for the pr 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-23 08:24:02
 +
+

*Thread Reply:* code formatting checks complain now

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-23 08:25:09
 +
+

*Thread Reply:* for the JAR issues, do you also want to create PR as you've fixed the issue on your end?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-23 09:06:26
 +
+

*Thread Reply:* @Abdallah you're using newer version of Java than 8, right?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-23 09:07:07
 +
+

*Thread Reply:* AFAIK googleJavaFormat behaves differently between Java versions

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Abdallah + (abdallah@terrab.me) +
+
2023-08-23 09:15:41
 +
+

*Thread Reply:* Okay I will switch back to another java version

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Abdallah + (abdallah@terrab.me) +
+
2023-08-23 09:25:06
 +
+

*Thread Reply:* terra@MacBook-Pro-M3 spark % java -version +java version "1.8.0_381" +Java(TM) SE Runtime Environment (build 1.8.0_381-b09) +Java HotSpot(TM) 64-Bit Server VM (build 25.381-b09, mixed mode)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Abdallah + (abdallah@terrab.me) +
+
2023-08-23 09:28:28
 +
+

*Thread Reply:* Can you tell me which java version should I use ?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Abdallah + (abdallah@terrab.me) +
+
2023-08-23 09:49:42
 +
+

*Thread Reply:* Hello, I have +@mobuchowski ERROR: Missing environment variable {i} +Can you please check what does it come from ?

+
+ + + + +
Maciej Obuchowski
+
+ OpenLineage committer. +Software Engineer @getindata +
+ +
+
Company
+ @getindata +
+ +
+
Location
+ Warsaw +
+ +
+
Repositories
+ 16 +
+ +
+
Followers
+ 25 +
+ + + + + + + + + + +
+ + + +
+
+
+
+ + + + + +
+ +
+ + + + + +
+
+ + + + +
+ +
Abdallah + (abdallah@terrab.me) +
+
2023-08-23 09:50:24
 +
+

*Thread Reply:* Can you help please ?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-23 10:08:43
 +
+

*Thread Reply:* Java 8

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-23 10:10:14
 +
+

*Thread Reply:* ```Hello, I have

+ +

@mobuchowski ERROR: Missing environment variable {i} +Can you please check what does it come from ? (edited) ``` +Yup, for now I have to manually make our CI account pick your changes up if you make PR from fork. Just did that

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-23 10:11:10
 +
+

*Thread Reply:* running here now: https://app.circleci.com/pipelines/github/OpenLineage/OpenLineage/7881/workflows/90793f46-796a-4f59-9de3-5d58cbcbf162

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-23 10:53:34
 +
+

*Thread Reply:* @Abdallah merged 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Abdallah + (abdallah@terrab.me) +
+
2023-08-23 10:59:22
 +
+

*Thread Reply:* Thank you !

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-08-16 14:21:26
 +
+

@channel +Meetup notice: on Monday, 9/18, at 5:00 pm ET OpenLineage will be gathering in Toronto at Airflow Summit. Coming to the summit? Based in or near Toronto? Please join us to discuss topics such as: +• recent developments in the project including the addition of static lineage support and the OpenLineage Airflow Provider, +• the project’s history and architecture, +• opportunities to contribute, +• resources for getting started, +• + more. +Please visit medium=referral&utmcampaign=share-btnsavedeventssharemodal&utmsource=link|the meetup page> for the specific location (which is not the conference hotel) and to sign up. Hope to see some of you there! (Please note that the start time is 5:00 pm ET.)

+
+
Meetup
+ + + +
Toronto OpenLineage Meetup at Airflow Summit, Mon, Sep 18, 2023, 2:00 PM | Meetup
+
+ Data engineers and pipeline managers know that producing data lineage – end-to-end pipeline metadata instrumented at runtime or parsed at design time – is a heavy lift with +
+ + + + + + + + +
Original URL: https://www.meetup.com/openlineage/events/295488014/?utm_medium=referral&amp;utm_campaign=share-btn_savedevents_share_modal&amp;utm_source=link
+ + + +
+ + + +
+ ❤️ Julien Le Dem, Maciej Obuchowski, Harel Shein, Paweł Leszczyński, Athitya Kumar, tati +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ldacey + (lance.dacey2@sutherlandglobal.com) +
+
2023-08-20 17:45:41
 +
+

i saw OpenLineage was built into Airflow recently as a provider but the documentation seems really light (https://airflow.apache.org/docs/apache-airflow-providers-openlineage/stable/guides/user.html), is the documentation from openlineage the correct way I should proceed?

+ +

https://openlineage.io/docs/integrations/airflow/usage

+
+
openlineage.io
+ + + +
Using the Airflow integration | OpenLineage
+
+ PREREQUISITES +
+ + + + + + +
Original URL: https://openlineage.io/docs/integrations/airflow/usage
+ + + +
+ + + +
+ 👍 Sheeri Cabral (Collibra) +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2023-08-21 20:26:56
 +
+

*Thread Reply:* openlineage-airflow is the package maintained in the OpenLineage project and to be used for versions of Airflow before 2.7. You could use it with 2.7 as well but you’d be staying on the “old” integration. +apache-airflow-providers-openlineage is the new package, maintained in the Airflow project that can be used starting Airflow 2.7 and is the recommended package moving forward. It is compatible with the configuration of the old package described in that usage page. CC: @Maciej Obuchowski @Jakub Dardziński It looks like this page needs improvement.

+
+
PyPI
+ + + +
openlineage-airflow
+
+ OpenLineage integration with Airflow +
+ + + + + + + + +
Original URL: https://pypi.org/project/openlineage-airflow/
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-22 05:03:28
 +
+

*Thread Reply:* Yeah, I'll fix that

+ + + +
+ :gratitude_thank_you: Julien Le Dem +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-08-22 17:55:08
 +
+

*Thread Reply:* https://github.com/apache/airflow/pull/33610

+ +

fyi

+
+ + + + +
#33610 openlineage: finish user guide
+
+ <p>Explains what is required to get OpenLineage provider running and explains the configuration options.</p> +
+ +
+
Labels
+ area:providers, kind:documentation, provider:openlineage +
+ +
+
Comments
+ 1 +
+ + + + + + + +
+ + <a href="https://github.com/apache/airflow">apache/airflow</a> +
+ + +
+ + + +
+ 🙌 ldacey, Julien Le Dem +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ldacey + (lance.dacey2@sutherlandglobal.com) +
+
2023-08-22 17:54:20
 +
+

do I label certain raw data sources as a dataset, for example SFTP/FTP sites, 0365 emails, etc? I extract that data into a bucket for the client in a "folder" called "raw" which I know will be an OL Dataset. Would this GCS folder (after extracting the data with Airflow) be the first Dataset OL is aware of?

+ +

<gcs://client-bucket/source-system-lob/raw>

+ +

I then process that data into partitioned parquet datasets which would also be OL Datasets: +<gcs://client-bucket/source-system-lob/staging> +<gcs://client-bucket/source-system-lob/analytics>

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-08-22 18:02:46
 +
+

*Thread Reply:* that really depends on the use case IMHO +if you consider a whole directory/folder as a dataset (meaning that each file inside folds into a larger whole) you should label dataset as directory

+ +

you might as well have directory with each file being something different - in this case it would be best to set each file separately as dataset

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-08-22 18:04:32
 +
+

*Thread Reply:* there was also SymlinksDatasetFacet introduced to store alternative dataset names, might be useful: https://github.com/OpenLineage/OpenLineage/pull/936

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ldacey + (lance.dacey2@sutherlandglobal.com) +
+
2023-08-22 18:07:26
 +
+

*Thread Reply:* cool, yeah in general each file is just a snapshot of data from a client (for example, daily dump). the parquet datasets are normally partitioned and might have small fragments and I definitely picture it as more of a table than individual files

+ + + +
+ 👍 Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-23 08:22:09
 +
+

*Thread Reply:* Agree with Jakub here - with object storage, people use different patterns, but usually some directory layer vs file is the valid abstraction level, especially if your pattern is adding files with new data inside

+ + + +
+ 👍 Jakub Dardziński +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ldacey + (lance.dacey2@sutherlandglobal.com) +
+
2023-08-25 10:26:52
 +
+

*Thread Reply:* I tested a dataset for each raw file versus the folder and the folder looks much cleaner (not sure if I can collapse individual datasets/files into a group?)

+ +

from 2022, this particular source had 6 raw schema changes (client controlled, no warning). what should I do to make that as obvious as possible if I track the dataset at a folder level?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ldacey + (lance.dacey2@sutherlandglobal.com) +
+
2023-08-25 10:32:19
 +
+

*Thread Reply:* I was thinking that I could name the dataset based on the schema_version (identified by the raw column names), so in this example I would have 6 OL datasets feeding into one "staging" dataset

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ldacey + (lance.dacey2@sutherlandglobal.com) +
+
2023-08-25 10:32:57
 +
+

*Thread Reply:* not sure what the best practice would be in this scenario though

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ldacey + (lance.dacey2@sutherlandglobal.com) +
+
2023-08-22 17:55:38
 +
+

• also saw the docs reference URI = gs://{bucket name}{path} and I wondered if the path would include the filename, or if it was just the base path like I showed above

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mars Lan + (mars@metaphor.io) +
+
2023-08-22 18:35:45
 +
+

Has anyone managed to get the OL Airflow integration to work on AWS MWAA? We've tried pretty much every trick but still ended up with the following error: +Broken plugin: [openlineage.airflow.plugin] No module named 'openlineage.airflow'; 'openlineage' is not a package

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-23 05:22:18
 +
+

*Thread Reply:* Which version are you trying to use?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-23 05:22:45
 +
+

*Thread Reply:* Both OL and MWAA/Airflow 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-23 05:23:52
 +
+

*Thread Reply:* 'openlineage' is not a package +suggests that something went wrong with import process, for example cycle in import path

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mars Lan + (mars@metaphor.io) +
+
2023-08-23 16:50:34
 +
+

*Thread Reply:* MWAA: 2.6.3 +OL: 1.0.0

+ +

I can see from the log that OL has been successfully installed to the webserver: +Successfully installed openlineage-airflow-1.0.0 openlineage-integration-common-1.0.0 openlineage-python-1.0.0 openlineage-sql-1.0.0 +This is the full stacktrace: +```Traceback (most recent call last):

+ +

File "/usr/local/airflow/.local/lib/python3.10/site-packages/airflow/pluginsmanager.py", line 229, in loadentrypointplugins +pluginclass = entrypoint.load() +File "/usr/local/airflow/.local/lib/python3.10/site-packages/importlibmetadata/init.py", line 209, in load +module = importmodule(match.group('module')) +File "/usr/lib/python3.10/importlib/init.py", line 126, in importmodule +return bootstrap.gcdimport(name[level:], package, level) +File "<frozen importlib.bootstrap>", line 1050, in gcdimport +File "<frozen importlib.bootstrap>", line 1027, in _findandload +File "<frozen importlib.bootstrap>", line 992, in findandloadunlocked +File "<frozen importlib.bootstrap>", line 241, in _callwithframesremoved +File "<frozen importlib.bootstrap>", line 1050, in _gcdimport +File "<frozen importlib.bootstrap>", line 1027, in _findandload +File "<frozen importlib.bootstrap>", line 1001, in findandloadunlocked +ModuleNotFoundError: No module named 'openlineage.airflow'; 'openlineage' is not a package```

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-08-24 08:18:36
 +
+

*Thread Reply:* It’s taking long to update MWAA environment but I tested 2.6.3 version with the followingrequirements.txt: +openlineage-airflow +and +openlineage-airflow==1.0.0 +is there any step that might lead to some unexpected results?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mars Lan + (mars@metaphor.io) +
+
2023-08-24 08:29:30
 +
+

*Thread Reply:* Yeah, it takes forever to update MWAA even for a simple change. If you open either the webserver log (in CloudWatch) or the AirFlow UI, you should see the above error message.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-08-24 08:33:53
 +
+

*Thread Reply:* The thing is that I don’t see any error messages. +I wrote simple DAG to test too: +```from future import annotations

+ +

from datetime import datetime

+ +

from airflow.models import DAG

+ +

try: + from airflow.operators.empty import EmptyOperator +except ModuleNotFoundError: + from airflow.operators.dummy import DummyOperator as EmptyOperator # type: ignore

+ +

from openlineage.airflow.adapter import OpenLineageAdapter +from openlineage.client.client import OpenLineageClient

+ +

from airflow.operators.python import PythonOperator

+ +

DAGID = "exampleol"

+ +

def callable(): + client = OpenLineageClient() + adapter = OpenLineageAdapter() + print(client, adapter)

+ +

with DAG( + dagid=DAGID, + startdate=datetime(2021, 1, 1), + schedule="@once", + catchup=False, +) as dag: + begin = EmptyOperator(taskid="begin")

+ +
test = PythonOperator(task_id='print_client', python_callable=callable)```
+
+ +

and it gives expected results as well

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mars Lan + (mars@metaphor.io) +
+
2023-08-24 08:48:11
 +
+

*Thread Reply:* Oh how interesting. I did have a plugin that sets the endpoint & key via env var. Let me try to disable that to see if it fixes the issue. Will report back after 30 mins, or however long it takes to update MWAA 😉

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-08-24 08:50:05
 +
+

*Thread Reply:* ohh, I see +you probably followed this guide: https://aws.amazon.com/blogs/big-data/automate-data-lineage-on-amazon-mwaa-with-openlineage/?

+
+
Amazon Web Services
+ + + +
Automate data lineage on Amazon MWAA with OpenLineage | Amazon Web Services
+
+ In modern data architectures, datasets are combined across an organization using a variety of purpose-built services to unlock insights. As a result, data governance becomes a key component for data consumers and producers to know that their data-driven decisions are based on trusted and accurate datasets. One aspect of data governance is data lineage, which […] +
+ + + + + + + + +
Original URL: https://aws.amazon.com/blogs/big-data/automate-data-lineage-on-amazon-mwaa-with-openlineage/
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mars Lan + (mars@metaphor.io) +
+
2023-08-24 09:04:27
 +
+

*Thread Reply:* Actually no. I'm not aware of this guide. I assume it's outdated already?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-08-24 09:04:54
 +
+

*Thread Reply:* tbh I don’t know

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mars Lan + (mars@metaphor.io) +
+
2023-08-24 09:04:55
 +
+

*Thread Reply:* Actually while we're on that topic, what's the recommended way to pass the URL & API Key in MWAA?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-08-24 09:28:00
 +
+

*Thread Reply:* I think it's still a plugin that sets env vars

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mars Lan + (mars@metaphor.io) +
+
2023-08-24 09:32:18
 +
+

*Thread Reply:* Yeah based on the page you shared, secret manager + plugin seems like the way to go.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mars Lan + (mars@metaphor.io) +
+
2023-08-24 10:31:50
 +
+

*Thread Reply:* Alas after disabling the plugin and restarting the cluster, I'm still getting the same error. Do you mind to share a screenshot of your cluster's settings so I can compare?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-24 11:57:04
 +
+

*Thread Reply:* Are you maybe importing some top level OpenLineage code anywhere? This error is most likely circular import

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mars Lan + (mars@metaphor.io) +
+
2023-08-24 12:01:12
 +
+

*Thread Reply:* Let me try removing all the dags to see if it helps.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mars Lan + (mars@metaphor.io) +
+
2023-08-24 18:42:49
 +
+

*Thread Reply:* @Maciej Obuchowski you were correct! It was indeed the DAGs. The errors are gone after removing all the dags. Now just need to figure what caused the circular import since I didn't import OL directly in DAG.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mars Lan + (mars@metaphor.io) +
+
2023-08-24 18:44:33
 +
+

*Thread Reply:* Could this be the issue? +from airflow.lineage.entities import File, Table +How could I declare lineage manually if I can't import these classes?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-25 06:52:47
 +
+

*Thread Reply:* @Mars Lan I'll look in more details next week, as I'm in transit now

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-25 06:53:18
 +
+

*Thread Reply:* but if you could narrow down a problem to single dag that I or @Jakub Dardziński could reproduce, ideally locally, it would help a lot

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mars Lan + (mars@metaphor.io) +
+
2023-08-25 07:07:11
 +
+

*Thread Reply:* Thanks. I think I understand how this works much better now. Found a few useful BQ example dags. Will give them a try and report back.

+ + + +
+ 🔥 Jakub Dardziński, Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Nitin + (nitinkhannain@yahoo.com) +
+
2023-08-23 07:14:44
 +
+

Hi All, +I want to capture, source and target table details as lineage information with openlineage for Amazon Redshift. Please let me know, if anyone has done it

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-08-23 07:32:19
 +
+

*Thread Reply:* are you using Airflow to connect to Redshift?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Nitin + (nitinkhannain@yahoo.com) +
+
2023-08-24 06:50:05
 +
+

*Thread Reply:* Hi @Jakub Dardziński, +Thank you for your reply. +No, we are not using Airflow. +We are using load/Unload cmd with Pyspark and also Pandas with JDBC connection

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-08-25 13:28:37
 +
+

*Thread Reply:* @Paweł Leszczyński might know answer if Spark<->OL integration works with Redshift. Eventually JDBC is supported with sqlparser

+ +

for Pandas I think there wasn’t too much work done

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-08-28 02:18:49
 +
+

*Thread Reply:* @Nitin If you're using jdbc within Spark, the lineage should be obtained via sqlparser-rs library https://github.com/sqlparser-rs/sqlparser-rs. In case it's not, please try to provide some minimal SQL code (or pyspark) which leads to uncaught lineage.

+
+ + + + +
sqlparser-rs/sqlparser-rs
+
+ Extensible SQL Lexer and Parser for Rust +
+ +
+
Stars
+ 1980 +
+ +
+
Language
+ Rust +
+ + + + + + + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Nitin + (nitinkhannain@yahoo.com) +
+
2023-08-28 04:53:03
 +
+

*Thread Reply:* Hi @Jakub Dardziński / @Paweł Leszczyński, thank you for taking out time to reply on my query. We need to capture only load and unload query lineage which we are running using Spark.

+ +

If you have any sample implementation for reference, it will be indeed helpful

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-08-28 06:12:46
 +
+

*Thread Reply:* I think we don't support load yet on our side: https://github.com/OpenLineage/OpenLineage/blob/main/integration/sql/impl/src/visitor.rs#L8

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/integration/sql/impl/src/visitor.rs | visitor.rs>
+
+ <pre><code>use sqlparser::ast::{ +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Nitin + (nitinkhannain@yahoo.com) +
+
2023-08-28 08:18:14
 +
+

*Thread Reply:* Yeah! any way you can think of, we can accommodate it specially load and unload statement. +Also, we would like to capture, lineage information where our endpoints are Sagemaker and Redis

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Nitin + (nitinkhannain@yahoo.com) +
+
2023-08-28 13:20:37
 +
+

*Thread Reply:* @Paweł Leszczyński can we use this code base integration/common/openlineage/common/provider/redshift_data.py for redshift lineage capture

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-08-28 14:26:40
 +
+

*Thread Reply:* it still expects input and output tables that are usually retrieved from sqlparser

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-08-28 14:31:00
 +
+

*Thread Reply:* for Sagemaker there is an Airflow integration written, might be an example possibly +https://github.com/OpenLineage/OpenLineage/blob/main/integration/airflow/openlineage/airflow/extractors/sagemaker_extractors.py

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Abdallah + (abdallah@terrab.me) +
+
2023-08-23 10:55:10
 +
+

Approve a new release please 🙂 +• Fix spark integration filtering Databricks events.

+ + + +
+ ➕ Abdallah, Tristan GUEZENNEC -CROIX-, Mouad MOUSSABBIH, Ayoub Oudmane, Asmae Tounsi, Jakub Dardziński, Michael Robinson, Harel Shein, Willy Lulciuc, Maciej Obuchowski, Julien Le Dem +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-08-23 12:27:15
 +
+

*Thread Reply:* Thank you for requesting a release @Abdallah. Three +1s from committers will authorize.

+ + + +
+ 🙌 Abdallah +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-08-23 13:13:18
 +
+

*Thread Reply:* Thanks, all. The release is authorized and will be initiated within 2 business days.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Athitya Kumar + (athityakumar@gmail.com) +
+
2023-08-23 13:08:48
 +
+

Hey folks! Do we have clear step-by-step documentation on how we can leverage the ServiceLoader based approach for injecting specific OpenLineage customisations for tweaking the transport type with defaults / tweaking column level lineage etc?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-23 13:24:32
 +
+

*Thread Reply:* This proposal - https://github.com/OpenLineage/OpenLineage/blob/41dbbf1799595bd9cd1567df0a7027de08619741/proposals/168/making_spark_visitors_extensible.md

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-23 13:29:05
 +
+

*Thread Reply:* For custom transport, you have to provide implementation of interface https://github.com/OpenLineage/OpenLineage/blob/4a1a5c3bf9767467b71ca0e1b6d820ba9e[…]ain/java/io/openlineage/client/transports/TransportBuilder.java and point to it in META_INF file

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/4a1a5c3bf9767467b71ca0e1b6d820ba9e0a1a1d/client/java/src/main/java/io/openlineage/client/transports/TransportBuilder.java | TransportBuilder.java>
+
+ <pre><code>public interface TransportBuilder { +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-23 13:29:52
 +
+

*Thread Reply:* But if I understand correctly, if you want to change behavior rather than extend, the correct way may be to either contribute it to repo - if that behavior is useful to anyone, or fork the repo

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Athitya Kumar + (athityakumar@gmail.com) +
+
2023-08-23 15:14:43
 +
+

*Thread Reply:* @Maciej Obuchowski - Can you elaborate more on the "point to it in META_INF file"? Let's say we have the custom transport type built in a standalone jar by extending transport builder - what're the exact next steps to use this custom transport in the standalone jar when doing spark-submit?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-23 15:23:13
 +
+

*Thread Reply:* @Athitya Kumar your jar needs to have META-INF/services/io.openlineage.client.transports.TransportBuilder with fully qualified class names of your custom TransportBuilders there - like openlineage-spark has +io.openlineage.client.transports.HttpTransportBuilder +io.openlineage.client.transports.KafkaTransportBuilder +io.openlineage.client.transports.ConsoleTransportBuilder +io.openlineage.client.transports.FileTransportBuilder +io.openlineage.client.transports.KinesisTransportBuilder

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Athitya Kumar + (athityakumar@gmail.com) +
+
2023-08-25 01:49:29
 +
+

*Thread Reply:* @Maciej Obuchowski - I think this change may be required for consumers to leverage custom transports, can you check & verify this GH comment? +https://github.com/OpenLineage/OpenLineage/issues/2007#issuecomment-1690350630

+
+ + + + +
Comment on #2007 [PROPOSAL] Ability to support custom injectable dynamic header generation class/logic for HTTP transports
+
+ <p><a href="https://github.com/mobuchowski">@mobuchowski</a> <a href="https://github.com/pawel-big-lebowski">@pawel-big-lebowski</a> - I think the <code>Type</code> enum in the Transport.java class has default visibility, which makes it invisible for OpenLineage consumers who are trying to define a custom transport logic</p> + +<p><a href="https://github.com/OpenLineage/OpenLineage/blob/main/client/java/src/main/java/io/openlineage/client/transports/Transport.java#L14">https://github.com/OpenLineage/OpenLineage/blob/main/client/java/src/main/java/io/openlineage/client/transports/Transport.java#L14</a></p> + +<p>I think it'll be great if we can make this <code>Type</code> enum have public visibility. Also, can we add a <code>CUSTOM</code> Type here for folks who are trying to define a custom transport of their own?</p> + +<p>Please lmk If the changes proposed seem to be fine, I can draft a PR for the same</p> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-25 06:52:30
 +
+

*Thread Reply:* Probably, I will look at more details next week @Athitya Kumar as I'm in transit

+ + + +
+ 👍 Athitya Kumar +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-08-23 15:04:10
 +
+

@channel +We released OpenLineage 1.1.0, including: +Additions: +• Flink: create Openlineage configuration based on Flink configuration #2033 @pawel-big-lebowski +• Java: add Javadocs to the Java client #2004 @julienledem +• Spark: append output dataset name to a job name #2036 @pawel-big-lebowski +• Spark: support Spark 3.4.1 #2057 @pawel-big-lebowski +Fixes: +• Flink: fix a bug when getting schema for KafkaSink #2042 @pentium3 +• Spark: fix ignored event adaptive_spark_plan in Databricks #2061 @algorithmy1 +Plus additional bug fixes, doc changes and more. +Thanks to all the contributors, especially new contributors @pentium3 and @Abdallah! +Release: https://github.com/OpenLineage/OpenLineage/releases/tag/1.1.0 +Changelog: https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md +Commit history: https://github.com/OpenLineage/OpenLineage/compare/1.0.0...1.1.0 +Maven: https://oss.sonatype.org/#nexus-search;quick~openlineage +PyPI: https://pypi.org/project/openlineage-python/

+ + + +
+ 👏 Ayoub Oudmane, Abdallah, Yuanli Wang, Athitya Kumar, Mars Lan, Maciej Obuchowski, Harel Shein, Kiran Hiremath, Thomas Abraham +
+ +
+ :gratitude_thank_you: GitHubOpenLineageIssues +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-08-25 10:29:23
 +
+

@channel +Friendly reminder: our next in-person meetup is next Wednesday, August 30th in San Francisco at Astronomer’s offices in the Financial District. You can sign up and find the details on the medium=referral&utmcampaign=share-btnsavedeventssharemodal&utmsource=link|meetup event page>.

+
+
Meetup
+ + + +
OpenLineage Meetup @ Astronomer, Wed, Aug 30, 2023, 5:30 PM | Meetup
+
+ Data engineers and pipeline managers know that producing data lineage – end-to-end pipeline metadata instrumented at runtime or parsed at design time – is a heavy lift with +
+ + + + + + + + +
Original URL: https://www.meetup.com/meetup-group-bnfqymxe/events/295195280/?utm_medium=referral&amp;utm_campaign=share-btn_savedevents_share_modal&amp;utm_source=link
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
George Polychronopoulos + (george.polychronopoulos@6point6.co.uk) +
+
2023-08-25 10:57:30
 +
+

hi Openlineage team , we would like to join one of your meetups(me and @Madhav Kakumani nad @Phil Rolph and we're wondering if you are hosting any meetups after the 18/9 ? We are trying to join this but air - tickets are quite expensive

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harel Shein + (harel.shein@gmail.com) +
+
2023-08-25 11:32:12
 +
+

*Thread Reply:* there will certainly be more meetups, don’t worry about that!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harel Shein + (harel.shein@gmail.com) +
+
2023-08-25 11:32:30
 +
+

*Thread Reply:* where are you located? perhaps we can try to organize a meetup closer to where you are.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
George Polychronopoulos + (george.polychronopoulos@6point6.co.uk) +
+
2023-08-25 11:49:37
 +
+

*Thread Reply:* Thanks a lot for the response, we are in London. We'd be glad to help you organise a meetup and also meet in person!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-08-25 11:51:39
 +
+

*Thread Reply:* This is awesome, thanks @George Polychronopoulos. I’ll start a channel and invite you

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Juan Luis Cano Rodríguez + (juan_luis_cano@mckinsey.com) +
+
2023-08-28 04:47:53
 +
+

hi folks, I'm looking into exporting static metadata, and found that DatasetEvent requires a eventTime, which in my mind doesn't make sense for static events. I'm setting it to None and the Python client seems to work, but wanted to ask if I'm missing something.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-08-28 05:59:10
 +
+

*Thread Reply:* Although you emit DatasetEvent, you still emit an event and eventTime is a valid marker.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Juan Luis Cano Rodríguez + (juan_luis_cano@mckinsey.com) +
+
2023-08-28 06:01:40
 +
+

*Thread Reply:* so, should I use the current time at the moment of emitting it and that's it?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-08-28 06:01:53
 +
+

*Thread Reply:* yes, that should be it

+ + + +
+ :gratitude_thank_you: Juan Luis Cano Rodríguez +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Juan Luis Cano Rodríguez + (juan_luis_cano@mckinsey.com) +
+
2023-08-28 04:49:21
 +
+

and something else: I understand that Marquez does not yet support the 2.0 spec, hence it's incompatible with static metadata right? I tried to emit a list of DatasetEvent s and got HTTPError: 422 Client Error: Unprocessable Entity for url: <http://localhost:3000/api/v1/lineage> (I'm using a FileTransport for now)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-08-28 06:02:49
 +
+

*Thread Reply:* marquez is not capable of reflecting DatasetEvents in DB but it should respond with Unsupported event type

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-08-28 06:03:15
 +
+

*Thread Reply:* and return 200 instead of 201 created

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Juan Luis Cano Rodríguez + (juan_luis_cano@mckinsey.com) +
+
2023-08-28 06:05:41
 +
+

*Thread Reply:* I'll have a deeper look then, probably I'm doing something wrong. thanks @Paweł Leszczyński

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Joshua Dotson + (josdotso@cisco.com) +
+
2023-08-28 13:25:58
 +
+

Hi folks. I have some pure golang jobs from which I need to emit OL events to Marquez. Is the right way to go about this to generate a Golang client from the Marquez OpenAPI spec and use that client from my go jobs?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-08-28 14:23:24
 +
+

*Thread Reply:* I'd rather generate them from OL spec (compliant with JSON Schema)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Joshua Dotson + (josdotso@cisco.com) +
+
2023-08-28 15:12:21
 +
+

*Thread Reply:* I'll look into this. I take you to mean that I would use the OL spec which is available as a set of JSON schemas to create the data object and then HTTP POST it using vanilla Golang. Is that correct? Thank you for your help!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-08-28 15:30:05
 +
+

*Thread Reply:* Correct! You’re also very welcome to contribute Golang client (currently we have Python & Java clients) if you manage to send events using golang 🙂

+ + + +
+ 👏 Joshua Dotson +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-08-28 17:28:31
 +
+

@channel +The agenda for the medium=referral&utmcampaign=share-btnsavedeventssharemodal&utmsource=link|Toronto Meetup at Airflow Summit> on 9/18 has been updated. This promises to be an exciting, richly productive discussion. Don’t miss it if you’ll be in the area!

+ +
  1. Intros
  2. Evolution of spec presentation/discussion (project background/history)
  3. State of the community
  4. Spark/Column lineage update
  5. Airflow Provider update
  6. Roadmap Discussion
  7. Action items review/next steps
    8. +
+
+
Meetup
+ + + +
Toronto OpenLineage Meetup at Airflow Summit, Mon, Sep 18, 2023, 2:00 PM | Meetup
+
+ Data engineers and pipeline managers know that producing data lineage – end-to-end pipeline metadata instrumented at runtime or parsed at design time – is a heavy lift with +
+ + + + + + + + +
Original URL: https://www.meetup.com/openlineage/events/295488014/?utm_medium=referral&amp;utm_campaign=share-btn_savedevents_share_modal&amp;utm_source=link
+ + + +
+ + + +
+ ❤️ Jarek Potiuk, Paweł Leszczyński, tati +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-08-28 20:05:37
 +
+

New on the OpenLineage blog: a close look at the new OpenLineage Airflow Provider, including: +• the critical improvements it brings to the integration +• the high-level design +• implementation details +• an example operator +• planned enhancements +• a list of supported operators +• more. +The post, by @Maciej Obuchowski, @Julien Le Dem and myself is live now on the OpenLineage blog.

+
+
openlineage.io
+ + + +
The OpenLineage Airflow Provider is Here | OpenLineage
+
+ Built-in OpenLineage support in Airflow means big improvements in reliability, lineage output, and custom operator implementation. +
+ + + + + + +
Original URL: https://openlineage.io/blog/airflow-provider
+ + + +
+ + + +
+ 🎉 Drew Meyers, Harel Shein, Maciej Obuchowski, Julian LaNeve, Mars Lan +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sarwat Fatima + (sarwatfatimam@gmail.com) +
+
2023-08-29 03:18:04
 +
+

Hello, I'm currently in the process of following the instructions outlined in the provided getting started guide at https://openlineage.io/getting-started/. However, I've encountered a problem while attempting to complete *Step 1* of the guide. Unfortunately, I'm encountering an internal server error at this stage. I did manage to successfully run Marquez, but it appears that there might be an issue that needs to be addressed. I have attached screen shots.

+ +
+
image.png
+ + + + + + + + +
+
+
image.png
+ + + + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-08-29 03:20:18
 +
+

*Thread Reply:* is 5000 port taken by any other application? or ./docker/up.sh has some errors in logs?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sarwat Fatima + (sarwatfatimam@gmail.com) +
+
2023-08-29 05:23:01
 +
+

*Thread Reply:* @Jakub Dardziński 5000 port is not taken by any other application. The logs show some errors but I am not sure what is the issue here.

+ +
+
image.png
+ + + + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-29 10:02:38
 +
+

*Thread Reply:* I think Marquez is running on WSL while you're trying to connect from host computer?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Juan Luis Cano Rodríguez + (juan_luis_cano@mckinsey.com) +
+
2023-08-29 05:20:39
 +
+

hi folks, for now I'm producing .jsonl (or .ndjson ) files with one event per line, do you know if there's any way to validate those? would standard JSON Schema tools work?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Juan Luis Cano Rodríguez + (juan_luis_cano@mckinsey.com) +
+
2023-08-29 10:58:29
 +
+

*Thread Reply:* reply by @Julian LaNeve: yes 🙂💯

+ + + +
+ 👍 Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ldacey + (lance.dacey2@sutherlandglobal.com) +
+
2023-08-29 13:12:32
 +
+

for namespaces, if my data is moving between sources (SFTP -> GCS -> Azure Blob (synapse connects to parquet datasets) then should my namespace be based on the client I am working with? my current namespace has been to refer to the bucket, but that falls apart when considering the data sources and some destinations. perhaps I should just add a field for client-name instead to have a consolidated view?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-30 10:53:08
 +
+

*Thread Reply:* > then should my namespace be based on the client I am working with? +I think each of those sources should be a different namespace?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ldacey + (lance.dacey2@sutherlandglobal.com) +
+
2023-08-30 12:59:53
 +
+

*Thread Reply:* got it, yeah I was kind of picturing as one namespace for the client (we handle many clients but they are completely distinct entities). I was able to get it to work with multiple namespaces like you suggested and Marquez was able to plot everything correctly in the visualization

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ldacey + (lance.dacey2@sutherlandglobal.com) +
+
2023-08-30 13:01:18
 +
+

*Thread Reply:* I noticed some of my Dataset facets make more sense as Run facets, for example, the name of the specific file I processed and how many rows of data / size of the data for that schedule. that won't impact the Run facets Airflow provides right? I can still have the schedule information + my custom run facets?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-30 13:06:38
 +
+

*Thread Reply:* Yes, unless you name it the same as one of the Airflow facets 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
GitHubOpenLineageIssues + (githubopenlineageissues@gmail.com) +
+
2023-08-30 08:15:29
 +
+

Hi, Will really appreciate if someone can guide me or provide me any pointer - if they have been able to implement authentication/authorization for access to Marquez. Have not seen much info around it. Any pointers greatly appreciated. Thanks in advance.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2023-08-30 12:23:18
 +
+

*Thread Reply:* I’ve seen people do this through the ingress controller in Kubernetes. Unfortunately I don’t have documentation besides k8s specific ones you would find for the ingress controller you’re using. You’d redirect any unauthenticated request to your identity provider

+ + + +
+ :gratitude_thank_you: GitHubOpenLineageIssues +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-08-30 11:50:05
 +
+

@channel +Friendly reminder: there’s a meetup tonight at Astronomer’s offices in SF!

+
+ + +
+ + + } + + Michael Robinson + (https://openlineage.slack.com/team/U02LXF3HUN7) +
+ + + +
+
+ <p>@channel +Friendly reminder: our next in-person meetup is next Wednesday, August 30th in San Francisco at Astronomer’s offices in the Financial District. You can sign up and find the details on the <https://www.meetup.com/meetup-group-bnfqymxe/events/295195280/?utm<em>medium=referral&amp;utm</em>campaign=share-btn<em>savedevents</em>share<em>modal&amp;utm</em>source=link|meetup event page>.</p> +
+ + + + + + +
Original URL: https://openlineage.slack.com/archives/C01CK9T7HKR/p1692973763570629
+ + +
+ + Slack Conversation +
+ + +
+ + + +
+ ✅ Sheeri Cabral (Collibra) +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2023-08-30 12:15:31
 +
+

*Thread Reply:* I’ll be there and looking forward to see @John Lukenoff ‘s presentation

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Barrientos + (mbarrien@gmail.com) +
+
2023-08-30 21:38:31
 +
+

Can anyone let 3 people stuck downstairs into the 7th floor?

+ + + +
+ 👍 Willy Lulciuc +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2023-08-30 23:25:21
 +
+

*Thread Reply:* Sorry about that!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Yunhe + (yunhe52203334@outlook.com) +
+
2023-08-31 02:31:48
 +
+

hello,everyone,i can run openLineage spark code in my notebook with python,but when use my idea to execute scala code like this: +import org.apache.spark.internal.Logging +import org.apache.spark.sql.SparkSession +import io.openlineage.client.OpenLineageClientUtils.loadOpenLineageYaml +import org.apache.spark.scheduler.{SparkListener, SparkListenerApplicationEnd, SparkListenerApplicationStart} +import sun.java2d.marlin.MarlinUtils.logInfo +object Test { + def main(args: Array[String]): Unit = {

+ +
val spark = SparkSession
+  .builder()
+  .master("local")
+  .appName("test")
+  .config("spark.jars.packages","io.openlineage:openlineage_spark:0.12.0")
+  .config("spark.extraListeners","io.openlineage.spark.agent.OpenLineageSparkListener")
+  .config("spark.openlineage.transport.type","console")
+  .getOrCreate()
+
+spark.sparkContext.setLogLevel("INFO")
+
+//spark.sparkContext.addSparkListener(new MySparkAppListener)
+import spark.implicits._
+val input = Seq((1, "zs", 2020), (2, "ls", 2023)).toDF("id", "name", "year")
+
+input.select("id", "name").orderBy("id").show()
+
+ +

}

+ +

}

+ +

there is something wrong: +Exception in thread "spark-listener-group-shared" java.lang.NoSuchMethodError: io.openlineage.client.OpenLineageClientUtils.loadOpenLineageYaml(Ljava/io/InputStream;)Lio/openlineage/client/OpenLineageYaml; + at io.openlineage.spark.agent.ArgumentParser.extractOpenlineageConfFromSparkConf(ArgumentParser.java:114) + at io.openlineage.spark.agent.ArgumentParser.parse(ArgumentParser.java:78) + at io.openlineage.spark.agent.OpenLineageSparkListener.initializeContextFactoryIfNotInitialized(OpenLineageSparkListener.java:277) + at io.openlineage.spark.agent.OpenLineageSparkListener.onApplicationStart(OpenLineageSparkListener.java:267) + at org.apache.spark.scheduler.SparkListenerBus.doPostEvent(SparkListenerBus.scala:55) + at org.apache.spark.scheduler.SparkListenerBus.doPostEvent$(SparkListenerBus.scala:28) + at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37) + at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37) + at org.apache.spark.util.ListenerBus.postToAll(ListenerBus.scala:117) + at org.apache.spark.util.ListenerBus.postToAll$(ListenerBus.scala:101) + at org.apache.spark.scheduler.AsyncEventQueue.super$postToAll(AsyncEventQueue.scala:105) + at org.apache.spark.scheduler.AsyncEventQueue.$anonfun$dispatch$1(AsyncEventQueue.scala:105) + at scala.runtime.java8.JFunction0$mcJ$sp.apply(JFunction0$mcJ$sp.java:23) + at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62) + at org.apache.spark.scheduler.AsyncEventQueue.org$apache$spark$scheduler$AsyncEventQueue$$dispatch(AsyncEventQueue.scala:100) + at org.apache.spark.scheduler.AsyncEventQueue$$anon$2.$anonfun$run$1(AsyncEventQueue.scala:96) + at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1446) + at org.apache.spark.scheduler.AsyncEventQueue$$anon$2.run(AsyncEventQueue.scala:96)

+ +

i want to know how can i set idea scala environment correctly

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-08-31 02:58:41
 +
+

*Thread Reply:* io.openlineage:openlineage_spark:0.12.0 -> could you repeat the steps with newer version?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Yunhe + (yunhe52203334@outlook.com) +
+
2023-08-31 03:51:52
 +
+

ok,it`s my first use thie lineage tool. first,I added dependences in my pom.xml like this: +<dependency> + <groupId>io.openlineage</groupId> + <artifactId>openlineage-java</artifactId> + <version>0.12.0</version> + </dependency> + <dependency> + <groupId>org.apache.logging.log4j</groupId> + <artifactId>log4j-api</artifactId> + <version>2.7</version> + </dependency> + <dependency> + <groupId>org.apache.logging.log4j</groupId> + <artifactId>log4j-core</artifactId> + <version>2.7</version> + </dependency> + <dependency> + <groupId>org.apache.logging.log4j</groupId> + <artifactId>log4j-slf4j-impl</artifactId> + <version>2.7</version> + </dependency> + <dependency> + <groupId>io.openlineage</groupId> + <artifactId>openlineage-spark</artifactId> + <version>0.30.1</version> + </dependency>

+ +

my spark version is 3.3.1 and the version can not change

+ +

second, in file Openlineage/intergration/spark I enter command : docker-compose up and follow the steps in this doc: +https://openlineage.io/docs/integrations/spark/quickstart_local +there is no erro when i use notebook to execute pyspark for openlineage and I could get json message. +but after I enter "docker-compose up" ,I want to use my Idea tool to execute scala code like above,the erro happend like above. It seems that I does not configure the environment correctly. so how can i fix the problem .

+
+
openlineage.io
+ + + +
Quickstart with Jupyter | OpenLineage
+
+ Trying out the Spark integration is super easy if you already have Docker Desktop and git installed. +
+ + + + + + +
Original URL: https://openlineage.io/docs/integrations/spark/quickstart_local
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-09-01 05:15:28
 +
+

*Thread Reply:* please use latest io.openlineage:openlineage_spark:1.1.0 instead. openlineage-java is already contained in the jar, no need to add it on your own.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sheeri Cabral (Collibra) + (sheeri.cabral@collibra.com) +
+
2023-08-31 15:33:19
 +
+

Will the August meeting be put up at https://wiki.lfaidata.foundation/display/OpenLineage/Monthly+TSC+meeting soon? (usually it’s up in a few days 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-09-01 06:00:53
 +
+

*Thread Reply:* @Michael Robinson

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-09-01 17:13:32
 +
+

*Thread Reply:* The recording is on the youtube channel here. I’ll update the wiki ASAP

+
+
YouTube
+ +
+ + + } + + OpenLineage Project + (https://www.youtube.com/@openlineageproject6897) +
+ + + +
OpenLineage Community Meeting | August 10, 2023
+
+ +
+ + + + + + + + +
Original URL: https://youtu.be/0Q5dWHvIDLo
+ + + +
+ + + +
+ ✅ Sheeri Cabral (Collibra) +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2023-08-31 18:10:20
 +
+

It sounds like there have been a few announcements at Google Next: +https://cloud.google.com/data-catalog/docs/how-to/open-lineage +https://cloud.google.com/dataproc/docs/guides/lineage

+
+
Google Cloud
+ + + +
Integrate with OpenLineage  |  Data Catalog Documentation  |  Google Cloud
+
+ +
+ + + + + + + + +
Original URL: https://cloud.google.com/data-catalog/docs/how-to/open-lineage
+ + + +
+
+
Google Cloud
+ + + +
Use data lineage in Dataproc  |  Dataproc Documentation  |  Google Cloud
+
+ +
+ + + + + + + + +
Original URL: https://cloud.google.com/dataproc/docs/guides/lineage
+ + + +
+ + + +
+ 🎉 Harel Shein, Willy Lulciuc, Kevin Languasco, Peter Hicks, Maciej Obuchowski, Paweł Leszczyński, Sheeri Cabral (Collibra), Ross Turk, Michael Robinson, Jakub Dardziński, Kiran Hiremath, Laurent Paris, Anastasia Khomyakova +
+ +
+ 🙌 Harel Shein, Willy Lulciuc, Mars Lan, Peter Hicks, Maciej Obuchowski, Paweł Leszczyński, Eric Veleker, Sheeri Cabral (Collibra), Ross Turk, Michael Robinson +
+ +
+ ❤️ Willy Lulciuc, Maciej Obuchowski, ldacey, Ross Turk, Michael Robinson +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2023-09-01 23:09:55
 +
+

*Thread Reply:* https://www.youtube.com/watch?v=zvCdrNJsxBo&t=2260s

+
+
YouTube
+ +
+ + + } + + Google Cloud + (https://www.youtube.com/@googlecloud) +
+ + + +
What’s new in data governance
+
+ +
+ + + + + + + + +
Original URL: https://www.youtube.com/watch?v=zvCdrNJsxBo&amp;t=2260s
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-09-01 17:16:21
 +
+

@channel +The latest issue of OpenLineage News is out now! Please subscribe to get it directly in your inbox each month.

+
+
apache.us14.list-manage.com
+ + + +
OpenLineage Project
+
+ OpenLineage Project Email Forms +
+ + + + + + +
Original URL: http://bit.ly/OL_news
+ + + +
+ + + +
+ 🙌 Jakub Dardziński, Maciej Obuchowski +
+ +
+ 🙌:skin_tone_3: Juan Luis Cano Rodríguez +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-09-04 03:38:28
 +
+

Hi guys, I'd like to capture the spark.databricks.clusterUsageTags.clusterAllTags property from databricks. However, the value of this is a list of keys, and therefore cannot be supported by custom environment facet builder. +I was thinking that capturing this property might be useful for most databricks workloads, and whether it might make sense to auto-capture it along with other databricks variables, similar to how we capture mount points for the databricks jobs. +Does this sound okay? If so, then I can help to contribute this functionality

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-09-04 06:43:47
 +
+

*Thread Reply:* Sounds good to me

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-09-11 05:15:03
 +
+

*Thread Reply:* Added this here: https://github.com/OpenLineage/OpenLineage/pull/2099

+
+ + + + +
#2099 Capture clusterAllTags variable from databricks
+
+ <p><strong>Problem</strong></p> + +<p>Auto-collect spark.databricks.clusterUsageTags.clusterAllTags environment variable from databricks</p> + +<p>Closes: <a href="https://github.com/OpenLineage/OpenLineage/issues/2098">#2098</a></p> + +<p><strong>Solution</strong></p> + +<p>Please describe your change as it relates to the problem, or bug fix, as well as any dependencies. If your change requires a schema change, please describe the schema modification(s) and whether it's a <em>backwards-incompatible</em> or <em>backwards-compatible</em> change, then select one of the following:</p> + +<blockquote> + <p><strong>Note:</strong> All schema changes require discussion. Please <a href="https://github.com/OpenLineage/OpenLineage/issues/2098">link the issue</a> for context.</p> +</blockquote> + +<p>☐ Your change modifies the <a href="https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.json">core</a> OpenLineage model +☐ Your change modifies one or more OpenLineage <a href="https://github.com/OpenLineage/OpenLineage/tree/main/spec/facets">facets</a></p> + +<p>If you're contributing a new integration, please specify the scope of the integration and how/where it has been tested (e.g., Apache Spark integration supports <code>S3</code> and <code>GCS</code> filesystem operations, tested with AWS EMR).</p> + +<p><strong>One-line summary:</strong> +<strong>Checklist</strong></p> + +<p>☑︎ You've <a href="https://github.com/OpenLineage/OpenLineage/blob/main/why-the-dco.md">signed-off</a> your work +☐ Your pull request title follows our <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CONTRIBUTING.md#creating-pull-requests">guidelines</a> +☐ Your changes are accompanied by tests (<em>if relevant</em>) +☐ Your change contains a <a href="https://kurtisnusbaum.medium.com/stacked-diffs-keeping-phabricator-diffs-small-d9964f4dcfa6">small diff</a> and is self-contained +☐ You've updated any relevant documentation (<em>if relevant</em>) +☐ Your comment includes a one-liner for the changelog about the specific purpose of the change (<em>if necessary</em>) +☐ You've versioned the core OpenLineage model or facets according to <a href="https://docs.snowplowanalytics.com/docs/pipeline-components-and-applications/iglu/common-architecture/schemaver">SchemaVer</a> (<em>if relevant</em>) +☐ You've added a <a href="https://github.com/OpenLineage/OpenLineage/tree/main/.github/header_templates.md">header</a> to source files (<em>if relevant</em>)</p> + +<hr /> + +<p>SPDX-License-Identifier: Apache-2.0 <br /> +Copyright 2018-2023 contributors to the OpenLineage project</p> +
+ +
+
Labels
+ integration/spark +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-09-04 06:39:05
 +
+

Also, another small clarification is that when using MergeIntoCommand, I'm receiving the lineage events on the backend, but I cannot seem to find any logging of the payload when I enable debug mode in openlineage. I remember there was a similar issue reported by another user in the past. May I check if it might be possible to help with this? It's making debugging quite hard for these cases. Thanks!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-09-04 06:54:12
 +
+

*Thread Reply:* I think it only depends on log4j configuration

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-09-04 06:57:15
 +
+

*Thread Reply:* ```# Set everything to be logged to the console +log4j.rootCategory=INFO, console +log4j.appender.console=org.apache.log4j.ConsoleAppender +log4j.appender.console.target=System.err +log4j.appender.console.layout=org.apache.log4j.PatternLayout +log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n

+ +

set the log level for the openlineage spark library

+ +

log4j.logger.io.openlineage.spark=DEBUG`` +this is what we have inlog4j.properties` in test environment and it works

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-09-04 11:28:11
 +
+

*Thread Reply:* Hmm... I can see the logs for the other commands, like createViewCommand etc. I just cannot see it for any of the delta runs

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-09-05 03:33:03
 +
+

*Thread Reply:* that's interesting. So, logging is done here: https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/app/src/main/java/io/openlineage/spark/agent/EventEmitter.java#L63 and this code is unaware of delta.

+ +

The possible problem could be filtering delta events (which we do bcz of delta being noisy)

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/app/src/main/java/io/openlineage/spark/agent/EventEmitter.java | EventEmitter.java>
+
+ <pre><code> log.debug( +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-09-05 03:33:36
 +
+

*Thread Reply:* Recently, we've closed that https://github.com/OpenLineage/OpenLineage/issues/1982 which prevents generating events for ` +createOrReplaceTempView

+
+ + + + +
#1982 [Spark] CreateViewCommand
+
+ <p><code>CreateViewCommand</code> does not contain is a spark action triggered with:</p> + +<pre><code>words.createOrReplaceTempView('words') +</code></pre> + +<p>It should be filtered in order not to generate Openlineag events, as they do have no inputs or outputs.</p> +
+ +
+
Assignees
+ <a href="https://github.com/pawel-big-lebowski">@pawel-big-lebowski</a> +
+ +
+
Labels
+ integration/spark +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-09-05 03:35:12
 +
+

*Thread Reply:* and this is the code change: https://github.com/OpenLineage/OpenLineage/pull/1987/files

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-09-05 05:19:22
 +
+

*Thread Reply:* Hmm I'm a little confused here. I thought we are only filtering out events for certain specific commands, like show table etc. because its noisy right? Some important commands like MergeInto or SaveIntoDataSource used to be logged before, but I notice now that its not being logged anymore... +I'm using 0.23.0 openlineage version.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-09-05 05:47:51
 +
+

*Thread Reply:* yes, we do. it's just sometimes when doing a filter, we can remove too much. but SaveIntoDataSource and MergeInto should be fine, as we do check them within the tests

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ldacey + (lance.dacey2@sutherlandglobal.com) +
+
2023-09-04 21:35:05
 +
+

it looks like my dynamic task mapping in Airflow has the same run ID in marquez, so even if I am processing 100 files, there is only one version of the data. is there a way to have a separate version of each dynamic task so I can track the filename etc?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-09-05 08:54:57
 +
+

*Thread Reply:* map_index should be indeed included when calculating run ID (it’s deterministic in Airflow integration) +what version of Airflow are you using btw?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ldacey + (lance.dacey2@sutherlandglobal.com) +
+
2023-09-05 09:04:14
 +
+

*Thread Reply:* 2.7.0

+ +

I do see this error log in all of my dynamic tasks which might explain it:

+ +

[2023-09-05, 00:31:57 UTC] {manager.py:200} ERROR - Extractor returns non-valid metadata: None +[2023-09-05, 00:31:57 UTC] {utils.py:401} ERROR - cannot import name 'get_operator_class' from 'airflow.providers.openlineage.utils' (/home/airflow/.local/lib/python3.11/site-packages/airflow/providers/openlineage/utils/__init__.py) +Traceback (most recent call last): + File "/home/airflow/.local/lib/python3.11/site-packages/airflow/providers/openlineage/utils/utils.py", line 399, in wrapper + return f(**args, ****kwargs) + ^^^^^^^^^^^^^^^^^^ + File "/home/airflow/.local/lib/python3.11/site-packages/airflow/providers/openlineage/plugins/listener.py", line 93, in on_running + ****get_custom_facets(task_instance), + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + File "/home/airflow/.local/lib/python3.11/site-packages/airflow/providers/openlineage/utils/utils.py", line 148, in get_custom_facets + custom_facets["airflow_mappedTask"] = AirflowMappedTaskRunFacet.from_task_instance(task_instance) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + File "/home/airflow/.local/lib/python3.11/site-packages/airflow/providers/openlineage/plugins/facets.py", line 36, in from_task_instance + from airflow.providers.openlineage.utils import get_operator_class +ImportError: cannot import name 'get_operator_class' from 'airflow.providers.openlineage.utils' (/home/airflow/.local/lib/python3.11/site-packages/airflow/providers/openlineage/utils/__init__.py)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ldacey + (lance.dacey2@sutherlandglobal.com) +
+
2023-09-05 09:05:34
 +
+

*Thread Reply:* I only have a few custom operators with the on_complete facet so I think this is a built in one - it runs before my task custom logs for example

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ldacey + (lance.dacey2@sutherlandglobal.com) +
+
2023-09-05 09:06:05
 +
+

*Thread Reply:* and any time I messed up my custom facet, the error would be at the bottom of the logs. this is on top, probably an on_start facet?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-09-05 09:16:32
 +
+

*Thread Reply:* seems like some circular import

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-09-05 09:19:47
 +
+

*Thread Reply:* I just tested it manually, it’s a bug in OL provider. let me fix that

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ldacey + (lance.dacey2@sutherlandglobal.com) +
+
2023-09-05 10:53:28
 +
+

*Thread Reply:* cool, thanks. I am glad it is just a bug, I was afraid dynamic tasks were not supported for a minute there

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ldacey + (lance.dacey2@sutherlandglobal.com) +
+
2023-09-07 11:46:20
 +
+

*Thread Reply:* how do the provider updates work? they can be released in between Airflow releases and issues for them are raised on the main Airflow repo?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-09-07 11:50:07
 +
+

*Thread Reply:* generally speaking anything related to OL-Airflow should be placed to Airflow repo, important changes/bug fixes would be implemented in OL repo as well

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ldacey + (lance.dacey2@sutherlandglobal.com) +
+
2023-09-07 15:40:31
 +
+

*Thread Reply:* got it, thanks

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ldacey + (lance.dacey2@sutherlandglobal.com) +
+
2023-09-07 19:43:46
 +
+

*Thread Reply:* is there a way for me to install the openlineage provider based on the commit you made to fix the circular imports?

+ +

i was going to try to install from Airflow main branch but didnt want to mess anything up

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ldacey + (lance.dacey2@sutherlandglobal.com) +
+
2023-09-07 19:44:39
 +
+

*Thread Reply:* I saw it was merged to airflow main but it is not in 2.7.1 and there is no 1.0.3 provider version yet, so I wondered if I could manually install it for the time being

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-09-08 05:45:48
 +
+

*Thread Reply:* https://github.com/apache/airflow/blob/main/BREEZE.rst#preparing-provider-packages +building the provider package on your own could be best idea probably? that depends on how you manage your Airflow instance

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-09-08 12:01:53
 +
+

*Thread Reply:* there's 1.1.0rc1 btw

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ldacey + (lance.dacey2@sutherlandglobal.com) +
+
2023-09-08 13:44:44
 +
+

*Thread Reply:* perfect, thanks. I got started with breeze but then stopped haha

+ + + +
+ 👍 Jakub Dardziński +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ldacey + (lance.dacey2@sutherlandglobal.com) +
+
2023-09-10 20:29:00
 +
+

*Thread Reply:* The dynamic task mapping error is gone, I did run into this:

+ +

File "/home/airflow/.local/lib/python3.11/site-packages/airflow/providers/openlineage/extractors/base.py", line 70, in disabledoperators + operator.strip() for operator in conf.get("openlineage", "disabledfor_operators").split(";") + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + File "/home/airflow/.local/lib/python3.11/site-packages/airflow/configuration.py", line 1065, in get + raise AirflowConfigException(f"section/key [{section}/{key}] not found in config")

+ +

I am redeploying now with that option added to my config. I guess it did not use the default which should be ""

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ldacey + (lance.dacey2@sutherlandglobal.com) +
+
2023-09-10 20:49:17
 +
+

*Thread Reply:* added "disabledforoperators" to my openlineage config and it worked (using Airflow helm chart - not sure if that means there is an error because the value I provided should just be the default value, not sure why I needed to explicitly specify it)

+ +

openlineage: + disabledforoperators: "" + ...

+ +

this is so much better and makes a lot more sense. most of my tasks are dynamic so I was missing a lot of metadata before the fix, thanks!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Abdallah + (abdallah@terrab.me) +
+
2023-09-06 16:43:07
 +
+

Hello Everyone,

+ +

I've been diving into the Marquez codebase and found a performance bottleneck in JobDao.java for the query related to namespaceName=MyNameSpace limit=10 and 12s with limit=25. I managed to optimize it using CTEs, and the execution times dropped dramatically to 300ms (for limit=100) and under 100ms (for limit=25 ) on the same cluster. +Issue link : https://github.com/MarquezProject/marquez/issues/2608

+ +

I believe there's even more room for optimization, especially if we adjust the job_facets_view to include the namespace_name column.

+ +

Would the team be open to a PR where I share the optimized query and discuss potential further refinements? I believe these changes could significantly enhance the Marquez web UI experience.

+ +

PR link : https://github.com/MarquezProject/marquez/pull/2609

+ +

Looking forward to your feedback.

+
+ + + + +
#2608 Performance Issue with Query on Large Data Sets in JobDao.java
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/MarquezProject/marquez">MarquezProject/marquez</a> +
+ + +
+
+ + + + +
#2609 Perf/improve jobdao query
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/MarquezProject/marquez">MarquezProject/marquez</a> +
+ + +
+ + + +
+ 🔥 Jakub Dardziński, Harel Shein, Paweł Leszczyński, Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-09-06 18:03:01
 +
+

*Thread Reply:* @Willy Lulciuc wdyt?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Bernat Gabor + (gaborjbernat@gmail.com) +
+
2023-09-06 17:44:12
 +
+

Has there been any conversation on the extensibility of facets/concepts? E.g.: +• how does one extends the list of run states https://openlineage.io/docs/spec/run-cycle to add a paused/resumed state? +• how does one extend https://openlineage.io/docs/spec/facets/run-facets/nominal_time to add a created at field?

+
+
openlineage.io
+ + + +
The Run Cycle | OpenLineage
+
+ The OpenLineage object model is event-based and updates provide an OpenLineage backend with details about the activities of a Job. +
+ + + + + + +
Original URL: https://openlineage.io/docs/spec/run-cycle
+ + + +
+
+
openlineage.io
+ + + +
Nominal Time Facet | OpenLineage
+
+ The facet to describe the nominal start and end time of the run. The nominal usually means the time the job run was expected to run (like a scheduled time), and the actual time can be different. +
+ + + + + + +
Original URL: https://openlineage.io/docs/spec/facets/run-facets/nominal_time
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2023-09-06 18:28:17
 +
+

*Thread Reply:* Hello Bernat,

+ +

The primary mechanism to extend the model is through facets. You can either: +• create new standard facets in the spec: https://github.com/OpenLineage/OpenLineage/tree/main/spec/facets +• create custom facets defined somewhere else with a prefix in their name: https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.md#custom-facet-naming +• Update existing facets with a backward compatible change (example: adding an optional field). +The core spec can also be modified. Here is an example of adding a state +That being said I think more granular states like pause/resume are probably better suited in a run facet. There was an issue opened for that particular one a while ago: https://github.com/OpenLineage/OpenLineage/issues/9 maybe that particular discussion can continue there.

+ +

For the nominal time facet, You could open an issue describing the use case and on community agreement follow up with a PR on the facet itself: https://github.com/OpenLineage/OpenLineage/blob/main/spec/facets/NominalTimeRunFacet.json +(adding an optional field is backwards compatible)

+ + + +
+ 👀 Juan Luis Cano Rodríguez +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Bernat Gabor + (gaborjbernat@gmail.com) +
+
2023-09-06 18:31:12
 +
+

*Thread Reply:* I see, so in general one is best copying a standard facet and maintain it under a different name. That way can be made mandatory 🙂 and one does not need to be blocked for a long time until there's a community agreement 🤔

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2023-09-06 18:35:43
 +
+

*Thread Reply:* Yes, The goal of custom facets is to allow you to experiment and extend the spec however you want without having to wait for approval. +If the custom facet is very specific to a third party project/product then it makes sense for it to stay a custom facet. +If it is more generic then it makes sense to add it to the core facets as part of the spec. +Hopefully community agreement can be achieved relatively quickly. Unless someone is strongly against something, it can be added without too much red tape. Typically with support in at least one of the integrations to validate the model.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-09-07 15:12:20
 +
+

@channel +This month’s TSC meeting is next Thursday the 14th at 10am PT. On the tentative agenda: +• announcements +• recent releases +• demo: Spark integration tests in Databricks runtime +• open discussion +• more (TBA) +More info and the meeting link can be found on the website. All are welcome! Also, feel free to reply or DM me with discussion topics, agenda items, etc.

+
+
openlineage.io
+ + + +
TSC Meetings | OpenLineage
+
+ The OpenLineage Technical Steering Committee meets monthly, and is open to all. +
+ + + + + + +
Original URL: https://openlineage.io/meetings/
+ + + +
+ + + +
+ 👍 Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-09-11 10:07:41
 +
+

@channel +The first Toronto OpenLineage Meetup, featuring a presentation by recent adopter Metaphor, is just one week away. On the agenda:

+ +
  1. Evolution of spec presentation/discussion (project background/history)
  2. State of the community
  3. Integrating OpenLineage with Metaphor (by special guests Ye & Ivan)
  4. Spark/Column lineage update
  5. Airflow Provider update
  6. Roadmap Discussion +Find more details and RSVP https://www.meetup.com/openlineage/events/295488014/?utmmedium=referral&utmcampaign=share-btnsavedeventssharemodal&utmsource=link|here.
    7. +
+
+
metaphor.io
+ + + +
Metaphor - The Social Platform for Data
+
+ Making Data Actionable, At Scale - Designed for data teams building cloud-native, self-service data platforms for their business users. Explore our Data Governance, Data Lineage, Data Discovery, and Data Trust capabilities today. +
+ + + + + + + + +
Original URL: https://metaphor.io/
+ + + +
+
+
Meetup
+ + + +
Toronto OpenLineage Meetup at Airflow Summit, Mon, Sep 18, 2023, 5:00 PM | Meetup
+
+ Data engineers and pipeline managers know that producing data lineage – end-to-end pipeline metadata instrumented at runtime or parsed at design time – is a heavy lift with +
+ + + + + + + + +
Original URL: https://www.meetup.com/openlineage/events/295488014/?utm_medium=referral&amp;utm_campaign=share-btn_savedevents_share_modal&amp;utm_source=link
+ + + +
+ + + +
+ 🙌 Mars Lan, Jarek Potiuk, Harel Shein, Maciej Obuchowski, Peter Hicks, Paweł Leszczyński, Dongjin Seo +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Lukenoff + (john@jlukenoff.com) +
+
2023-09-11 17:07:26
 +
+

I’m seeing some odd behavior with my http transport when upgrading airflow/openlineage-airflow from 2.3.2 -> 2.6.3 and 0.24.0 -> 0.28.0. Previously I had a config like this that let me provide my own auth tokens. However, after upgrading I’m getting a 401 from the endpoint and further debugging seems to reveal that we’re not using the token provided in my TokenProvider. Does anyone know if something changed between these versions that could be causing this? (more details in 🧵 ) +transport: + type: http + url: <https://my.fake-marquez-endpoint.com> + auth: + type: some.fully.qualified.classpath

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Lukenoff + (john@jlukenoff.com) +
+
2023-09-11 17:09:40
 +
+

*Thread Reply:* If I log this line I can tell the TokenProvider is the class instance I would expect: https://github.com/OpenLineage/OpenLineage/blob/45d94fb73b5488d34b8ca544b58317382ceb3980/client/python/openlineage/client/transport/http.py#L55

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/45d94fb73b5488d34b8ca544b58317382ceb3980/client/python/openlineage/client/transport/http.py | http.py>
+
+ <pre><code> subclass = try_import_from_string(of_type) +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Lukenoff + (john@jlukenoff.com) +
+
2023-09-11 17:11:14
 +
+

*Thread Reply:* However, if I log the token_provider here I get the origin TokenProvider: https://github.com/OpenLineage/OpenLineage/blob/45d94fb73b5488d34b8ca544b58317382ceb3980/client/python/openlineage/client/transport/http.py#L154

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/45d94fb73b5488d34b8ca544b58317382ceb3980/client/python/openlineage/client/transport/http.py | http.py>
+
+ <pre><code> bearer = token_provider.get_bearer() +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Lukenoff + (john@jlukenoff.com) +
+
2023-09-11 17:18:56
 +
+

*Thread Reply:* Ah I think I see the issue. Looks like this was introduced here, we are instantiating with the base token provider here when we should be using the subclass: https://github.com/OpenLineage/OpenLineage/pull/1869/files#diff-2f8ea6f9a22b5567de8ab56c6a63da8e7adf40cb436ee5e7e6b16e70a82afe05R57

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Lukenoff + (john@jlukenoff.com) +
+
2023-09-11 17:37:42
 +
+

*Thread Reply:* Opened a PR for this here: https://github.com/OpenLineage/OpenLineage/pull/2100

+
+ + + + +
#2100 python: fix custom http transport TokenProvider
+
+ <p><strong>Problem</strong></p> + +<p>Looks like in <a href="https://github.com/OpenLineage/OpenLineage/pull/1869">#1869</a> we introduced a bug where we would always use the base <code>TokenProvider</code> class instead of a given custom token provider, even if the validation conditions passed.</p> + +<p><strong>Solution</strong></p> + +<p>Revert this to instantiate the subclass</p> + +<p>☐ Your change modifies the <a href="https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.json">core</a> OpenLineage model +☐ Your change modifies one or more OpenLineage <a href="https://github.com/OpenLineage/OpenLineage/tree/main/spec/facets">facets</a></p> + +<p>If you're contributing a new integration, please specify the scope of the integration and how/where it has been tested (e.g., Apache Spark integration supports <code>S3</code> and <code>GCS</code> filesystem operations, tested with AWS EMR).</p> + +<p><strong>One-line summary:</strong> +<strong>Checklist</strong></p> + +<p>☑︎ You've <a href="https://github.com/OpenLineage/OpenLineage/blob/main/why-the-dco.md">signed-off</a> your work +☑︎ Your pull request title follows our <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CONTRIBUTING.md#creating-pull-requests">guidelines</a> +☐ Your changes are accompanied by tests (<em>if relevant</em>) +☑︎ Your change contains a <a href="https://kurtisnusbaum.medium.com/stacked-diffs-keeping-phabricator-diffs-small-d9964f4dcfa6">small diff</a> and is self-contained +☐ You've updated any relevant documentation (<em>if relevant</em>) +☐ Your comment includes a one-liner for the changelog about the specific purpose of the change (<em>if necessary</em>) +☐ You've versioned the core OpenLineage model or facets according to <a href="https://docs.snowplowanalytics.com/docs/pipeline-components-and-applications/iglu/common-architecture/schemaver">SchemaVer</a> (<em>if relevant</em>) +☐ You've added a <a href="https://github.com/OpenLineage/OpenLineage/tree/main/.github/header_templates.md">header</a> to source files (<em>if relevant</em>)</p> + +<hr /> + +<p>SPDX-License-Identifier: Apache-2.0 <br /> +Copyright 2018-2023 contributors to the OpenLineage project</p> +
+ +
+
Labels
+ client/python +
+ +
+
Comments
+ 1 +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+ ❤️ Julien Le Dem +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sarwat Fatima + (sarwatfatimam@gmail.com) +
+
2023-09-12 08:14:06
 +
+

This particular code in docker-compose exits with code 1 because it is unable to find wait-for-it.sh, file in the container. I have checked the mounting path from the local machine, It is correct and the path on the container for Marquez is also correct i.e. /usr/src/app but it is unable to mount the wait-for-it.sh. Does anyone know why is this? This code exists in the open lineage repository as well https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/docker-compose.yml +# Marquez as an OpenLineage Client + api: + image: marquezproject/marquez + container_name: marquez-api + ports: + - "5000:5000" + - "5001:5001" + volumes: + - ./docker/wait-for-it.sh:/usr/src/app/wait-for-it.sh + links: + - "db:postgres" + depends_on: + - db + entrypoint: [ "./wait-for-it.sh", "db:5432", "--", "./entrypoint.sh" ]

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/docker-compose.yml | docker-compose.yml>
+
+ <pre><code>version: "3.7" +services: + notebook: + image: jupyter/pyspark-notebook:spark-3.1.2 + ports: + - "8888:8888" + volumes: + - ./docker/notebooks:/home/jovyan/notebooks + - ./build:/home/jovyan/openlineage + links: + - "api:marquez" + depends_on: + - api + +# Marquez as an OpenLineage Client + api: + image: marquezproject/marquez + container_name: marquez-api + ports: + - "5000:5000" + - "5001:5001" + volumes: + - ./docker/wait-for-it.sh:/usr/src/app/wait-for-it.sh + links: + - "db:postgres" + depends_on: + - db + entrypoint: [ "./wait-for-it.sh", "db:5432", "--", "./entrypoint.sh" ] + + db: + image: postgres:12.1 + container_name: marquez-db + ports: + - "5432:5432" + environment: + - POSTGRES_USER=postgres + - POSTGRES_PASSWORD=password + - MARQUEZ_DB=marquez + - MARQUEZ_USER=marquez + - MARQUEZ_PASSWORD=marquez + volumes: + - ./docker/init-db.sh:/docker-entrypoint-initdb.d/init-db.sh + # Enables SQL statement logging (see: &lt;https://www.postgresql.org/docs/12/runtime-config-logging.html#GUC-LOG-STATEMENT&gt;) + # command: ["postgres", "-c", "log_statement=all"] +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sarwat Fatima + (sarwatfatimam@gmail.com) +
+
2023-09-12 08:15:19
 +
+

*Thread Reply:* This is the error message:

+ +
+
image.png
+ + + + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-09-12 10:38:41
 +
+

*Thread Reply:* no permissions?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Guntaka Jeevan Paul + (jeevan@acceldata.io) +
+
2023-09-12 15:11:45
 +
+

I am trying to run Google Cloud Composer where i have added the openlineage-airflow pypi packagae as a dependency and have added the env OPENLINEAGEEXTRACTORS to point to my custom extractor. I have added a folder by name dependencies and inside that i have placed my extractor file, and the path given to OPENLINEAGEEXTRACTORS is dependencies.<filename>.<extractorclass_name>…still it fails with the exception saying No module named ‘dependencies’. Can anyone kindly help me out on correcting my mistake

+ +
+
Screenshot 2023-09-13 at 12.38.55 AM.png
+ + + + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harel Shein + (harel.shein@gmail.com) +
+
2023-09-12 17:15:36
 +
+

*Thread Reply:* Hey @Guntaka Jeevan Paul, can you share some details on which versions of airflow and openlineage you’re using?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Guntaka Jeevan Paul + (jeevan@acceldata.io) +
+
2023-09-12 17:16:26
 +
+

*Thread Reply:* airflow ---> 2.5.3, openlinegae-airflow ---> 1.1.0

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Guntaka Jeevan Paul + (jeevan@acceldata.io) +
+
2023-09-12 17:45:08
 +
+

*Thread Reply:* ```import traceback +import uuid +from typing import List, Optional

+ +

from openlineage.airflow.extractors.base import BaseExtractor, TaskMetadata +from openlineage.airflow.utils import getjobname

+ +

class BigQueryInsertJobExtractor(BaseExtractor): + def init(self, operator): + super().init(operator)

+ +
@classmethod
+def get_operator_classnames(cls) -&gt; List[str]:
+    return ['BigQueryInsertJobOperator']
+
+def extract(self) -&gt; Optional[TaskMetadata]:
+    return None
+
+def extract_on_complete(self, task_instance) -&gt; Optional[TaskMetadata]:
+    self.log.debug(f"JEEVAN ---&gt; extract_on_complete({task_instance})")
+    random_uuid = str(uuid.uuid4())
+    self.log.debug(f"JEEVAN ---&gt; Randomly Generated UUID --&gt; {random_uuid}")
+
+    self.operator.job_id = random_uuid
+
+    return TaskMetadata(
+        name=get_job_name(task=self.operator)
+    )```
+
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Guntaka Jeevan Paul + (jeevan@acceldata.io) +
+
2023-09-12 17:45:24
 +
+

*Thread Reply:* this is the custom extractor code that im trying with

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harel Shein + (harel.shein@gmail.com) +
+
2023-09-12 21:10:02
 +
+

*Thread Reply:* thanks @Guntaka Jeevan Paul, will try to take a deeper look tomorrow

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-09-13 07:54:26
 +
+

*Thread Reply:* No module named 'dependencies'. +This sounds like general Python problem

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-09-13 07:55:12
 +
+

*Thread Reply:* https://stackoverflow.com/questions/69991553/how-to-import-custom-modules-in-cloud-composer

+
+
Stack Overflow
+ + + +
how to import custom modules in Cloud Composer
+
+ I created a local project with apache Airflow and i want to run it in cloud composer. My project contains custom modules and a main file that calls them. +Example : from src.kuzzle import KuzzleQuery +
+ + + + + + + + +
Original URL: https://stackoverflow.com/questions/69991553/how-to-import-custom-modules-in-cloud-composer
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-09-13 07:56:28
 +
+

*Thread Reply:* basically, if you're able to import the file from your dag code, OL should be able too

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Guntaka Jeevan Paul + (jeevan@acceldata.io) +
+
2023-09-13 08:01:12
 +
+

*Thread Reply:* The Problem is in the GCS Composer there is a component called Triggerer, which they say is used for deferrable operators…i have logged into that pod and i could see that the GCS Bucket is not mounted on this, but i am unable to understand why is the initialisation happening inside the triggerer pod

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Guntaka Jeevan Paul + (jeevan@acceldata.io) +
+
2023-09-13 08:01:32
 +
+

*Thread Reply:*

+ +
+
Screenshot 2023-09-13 at 5.31.22 PM.png
+ + + + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-09-13 08:01:47
 +
+

*Thread Reply:* > The Problem is in the GCS Composer there is a component called Triggerer, which they say is used for deferrable operators…i have logged into that pod and i could see that the GCS Bucket is not mounted on this, but i am unable to understand why is the initialisation happening inside the triggerer pod +OL integration is not running on triggerer, only on worker and scheduler pods

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Guntaka Jeevan Paul + (jeevan@acceldata.io) +
+
2023-09-13 08:01:53
 +
+

*Thread Reply:*

+ +
+
Screenshot 2023-09-13 at 5.31.44 PM.png
+ + + + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Guntaka Jeevan Paul + (jeevan@acceldata.io) +
+
2023-09-13 08:03:26
 +
+

*Thread Reply:* As you can see in this screenshot i am seeing the logs of the triggerer and it says clearly unable to import plugin openlineage

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Guntaka Jeevan Paul + (jeevan@acceldata.io) +
+
2023-09-13 08:03:29
 +
+

*Thread Reply:* https://openlineage.slack.com/files/U05QL7LN2GH/F05SUDUQEDN/screenshot_2023-09-13_at_5.31.22_pm.png

+ +
+
Screenshot 2023-09-13 at 5.31.22 PM.png
+ + + + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-09-13 08:10:32
 +
+

*Thread Reply:* I see. There are few possible things to do here - composer could mount the user files, Airflow could not start plugins on triggerer, or we could detect we're on triggerer and not import anything there. However, does it impact OL or Airflow operation in other way than this log?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-09-13 08:12:06
 +
+

*Thread Reply:* Probably we'd have to do something if that really bothers you as there won't be further changes to Airflow 2.5

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Guntaka Jeevan Paul + (jeevan@acceldata.io) +
+
2023-09-13 08:18:14
 +
+

*Thread Reply:* The Problem is it is actually not registering this custom extractor written by me, henceforth i am just receiving the DefaultExtractor things and my piece of extractor code is not even getting triggered

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Guntaka Jeevan Paul + (jeevan@acceldata.io) +
+
2023-09-13 08:22:49
 +
+

*Thread Reply:* any suggestions to try @Maciej Obuchowski

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-09-13 08:27:48
 +
+

*Thread Reply:* Could you share worker logs?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-09-13 08:27:56
 +
+

*Thread Reply:* and check if module is importable from your dag code?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Guntaka Jeevan Paul + (jeevan@acceldata.io) +
+
2023-09-13 08:31:25
 +
+

*Thread Reply:* these are the worker pod logs…where there is no log of openlineageplugin

+ +
+
downloaded-logs-20230913-180017.json
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Guntaka Jeevan Paul + (jeevan@acceldata.io) +
+
2023-09-13 08:31:52
 +
+

*Thread Reply:* https://openlineage.slack.com/archives/C01CK9T7HKR/p1694608076879469?thread_ts=1694545905.974339&cid=C01CK9T7HKR --> sure will check now on this one

+
+ + +
+ + + } + + Maciej Obuchowski + (https://openlineage.slack.com/team/U01RA9B5GG2) +
+ + + +
+
+ <p>and check if module is importable from your dag code?</p> +
+ + + + + + +
Original URL: https://openlineage.slack.com/archives/C01CK9T7HKR/p1694608076879469?thread_ts=1694545905.974339&amp;cid=C01CK9T7HKR
+ + +
+ + Thread in Slack Conversation +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-09-13 08:38:32
 +
+

*Thread Reply:* { + "textPayload": "Traceback (most recent call last): File \"/opt/python3.8/lib/python3.8/site-packages/openlineage/airflow/utils.py\", line 427, in import_from_string module = importlib.import_module(module_path) File \"/opt/python3.8/lib/python3.8/importlib/__init__.py\", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File \"&lt;frozen importlib._bootstrap&gt;\", line 1014, in _gcd_import File \"&lt;frozen importlib._bootstrap&gt;\", line 991, in _find_and_load File \"&lt;frozen importlib._bootstrap&gt;\", line 961, in _find_and_load_unlocked File \"&lt;frozen importlib._bootstrap&gt;\", line 219, in _call_with_frames_removed File \"&lt;frozen importlib._bootstrap&gt;\", line 1014, in _gcd_import File \"&lt;frozen importlib._bootstrap&gt;\", line 991, in _find_and_load File \"&lt;frozen importlib._bootstrap&gt;\", line 961, in _find_and_load_unlocked File \"&lt;frozen importlib._bootstrap&gt;\", line 219, in _call_with_frames_removed File \"&lt;frozen importlib._bootstrap&gt;\", line 1014, in _gcd_import File \"&lt;frozen importlib._bootstrap&gt;\", line 991, in _find_and_load File \"&lt;frozen importlib._bootstrap&gt;\", line 973, in _find_and_load_unlockedModuleNotFoundError: No module named 'airflow.gcs'", + "insertId": "pt2eu6fl9z5vw", + "resource": { + "type": "cloud_composer_environment", + "labels": { + "environment_name": "openlineage", + "location": "us-west1", + "project_id": "acceldata-acm" + } + }, + "timestamp": "2023-09-13T06:20:44.131577764Z", + "severity": "ERROR", + "labels": { + "worker_id": "airflow-worker-xttt8" + }, + "logName": "projects/acceldata-acm/logs/airflow-worker", + "receiveTimestamp": "2023-09-13T06:20:48.847319607Z" + }, +it doesn't see No module named 'airflow.gcs' that is part of your extractor path airflow.gcs.dags.big_query_insert_job_extractor.BigQueryInsertJobExtractor +however, is it necessary? I generally see people using imports directly from dags folder

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Guntaka Jeevan Paul + (jeevan@acceldata.io) +
+
2023-09-13 08:44:11
 +
+

*Thread Reply:* this is one of the experimentation that i have did, but then i reverted it back to keeping it to dependencies.bigqueryinsertjobextractor.BigQueryInsertJobExtractor…where dependencies is a module i have created inside my dags folder

+ +
+
Screenshot 2023-09-13 at 6.13.36 PM.png
+ + + + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Guntaka Jeevan Paul + (jeevan@acceldata.io) +
+
2023-09-13 08:44:33
 +
+

*Thread Reply:* https://openlineage.slack.com/files/U05QL7LN2GH/F05RM6EV6DV/screenshot_2023-09-13_at_12.38.55_am.png

+ +
+
Screenshot 2023-09-13 at 12.38.55 AM.png
+ + + + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Guntaka Jeevan Paul + (jeevan@acceldata.io) +
+
2023-09-13 08:45:46
 +
+

*Thread Reply:* these are the logs of the triggerer pod specifically

+ +
+
Screenshot 2023-09-13 at 6.15.30 PM.png
+ + + + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-09-13 08:46:31
 +
+

*Thread Reply:* yeah it would be expected to have this in triggerer where it's not mounted, but will it behave the same for worker where it's mounted?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-09-13 08:47:09
 +
+

*Thread Reply:* maybe ___init___.py is missing for top-level dag path?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Guntaka Jeevan Paul + (jeevan@acceldata.io) +
+
2023-09-13 08:49:01
 +
+

*Thread Reply:* these are the logs of the worker pod at startup, where it does not complain of the plugin like in triggerer, but when tasks are run on this worker…somehow it is not picking up the extractor for the operator that i have written it for

+ +
+
Screenshot 2023-09-13 at 6.17.46 PM.png
+ + + + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Guntaka Jeevan Paul + (jeevan@acceldata.io) +
+
2023-09-13 08:49:54
 +
+

*Thread Reply:* https://openlineage.slack.com/archives/C01CK9T7HKR/p1694609229577469?thread_ts=1694545905.974339&cid=C01CK9T7HKR --> you mean to make the dags folder as well like a module by adding the init.py?

+
+ + +
+ + + } + + Maciej Obuchowski + (https://openlineage.slack.com/team/U01RA9B5GG2) +
+ + + +
+
+ <p>maybe <code>___init___.py</code> is missing for top-level dag path?</p> +
+ + + + + + +
Original URL: https://openlineage.slack.com/archives/C01CK9T7HKR/p1694609229577469?thread_ts=1694545905.974339&amp;cid=C01CK9T7HKR
+ + +
+ + Thread in Slack Conversation +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-09-13 08:55:24
 +
+

*Thread Reply:* yes, I would put whole custom code directly in dags folder, to make sure import paths are the problem

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-09-13 08:55:48
 +
+

*Thread Reply:* and would be nice if you could set +AIRFLOW__LOGGING__LOGGING_LEVEL="DEBUG"

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Guntaka Jeevan Paul + (jeevan@acceldata.io) +
+
2023-09-13 09:14:58
 +
+

*Thread Reply:* ```Starting the process, got command: triggerer +Initializing airflow.cfg. +airflow.cfg initialization is done. +[2023-09-13T13:11:46.620+0000] {settings.py:267} DEBUG - Setting up DB connection pool (PID 8) +[2023-09-13T13:11:46.622+0000] {settings.py:372} DEBUG - settings.prepareengineargs(): Using pool settings. poolsize=5, maxoverflow=10, poolrecycle=570, pid=8 +[2023-09-13T13:11:46.742+0000] {cliactionloggers.py:39} DEBUG - Adding <function defaultactionlog at 0x7ff39ca1d3a0> to pre execution callback +[2023-09-13T13:11:47.638+0000] {cliactionloggers.py:65} DEBUG - Calling callbacks: [<function defaultactionlog at 0x7ff39ca1d3a0>] + __ ___ + _ |( )__ / /_ _ +_ /| |_ / / /_ _ / _ _ | /| / / +_ | / _ / _ _/ _ / / // /_ |/ |/ / + // |// // // // _/_/|/ +[2023-09-13T13:11:50.527+0000] {pluginsmanager.py:300} DEBUG - Loading plugins +[2023-09-13T13:11:50.580+0000] {pluginsmanager.py:244} DEBUG - Loading plugins from directory: /home/airflow/gcs/plugins +[2023-09-13T13:11:50.581+0000] {pluginsmanager.py:224} DEBUG - Loading plugins from entrypoints +[2023-09-13T13:11:50.587+0000] {pluginsmanager.py:227} DEBUG - Importing entrypoint plugin OpenLineagePlugin +[2023-09-13T13:11:50.740+0000] {utils.py:430} WARNING - No module named 'boto3' +[2023-09-13T13:11:50.743+0000] {utils.py:430} WARNING - No module named 'botocore' +[2023-09-13T13:11:50.833+0000] {utils.py:430} WARNING - No module named 'airflow.providers.sftp' +[2023-09-13T13:11:51.144+0000] {utils.py:430} WARNING - No module named 'bigqueryinsertjobextractor' +[2023-09-13T13:11:51.145+0000] {pluginsmanager.py:237} ERROR - Failed to import plugin OpenLineagePlugin +Traceback (most recent call last): + File "/opt/python3.8/lib/python3.8/site-packages/openlineage/airflow/utils.py", line 427, in importfromstring + module = importlib.importmodule(modulepath) + File "/opt/python3.8/lib/python3.8/importlib/init.py", line 127, in importmodule + return bootstrap.gcdimport(name[level:], package, level) + File "<frozen importlib.bootstrap>", line 1014, in gcdimport + File "<frozen importlib.bootstrap>", line 991, in _findandload + File "<frozen importlib.bootstrap>", line 973, in findandloadunlocked +ModuleNotFoundError: No module named 'bigqueryinsertjobextractor'

+ +

The above exception was the direct cause of the following exception:

+ +

Traceback (most recent call last): + File "/opt/python3.8/lib/python3.8/site-packages/airflow/pluginsmanager.py", line 229, in loadentrypointplugins + pluginclass = entrypoint.load() + File "/opt/python3.8/lib/python3.8/site-packages/setuptools/vendor/importlibmetadata/init.py", line 194, in load + module = importmodule(match.group('module')) + File "/opt/python3.8/lib/python3.8/importlib/init.py", line 127, in importmodule + return _bootstrap.gcdimport(name[level:], package, level) + File "<frozen importlib.bootstrap>", line 1014, in gcdimport + File "<frozen importlib.bootstrap>", line 991, in _findandload + File "<frozen importlib.bootstrap>", line 975, in findandloadunlocked + File "<frozen importlib.bootstrap>", line 671, in _loadunlocked + File "<frozen importlib.bootstrapexternal>", line 843, in execmodule + File "<frozen importlib.bootstrap>", line 219, in callwithframesremoved + File "/opt/python3.8/lib/python3.8/site-packages/openlineage/airflow/plugin.py", line 32, in <module> + from openlineage.airflow import listener + File "/opt/python3.8/lib/python3.8/site-packages/openlineage/airflow/listener.py", line 75, in <module> + extractormanager = ExtractorManager() + File "/opt/python3.8/lib/python3.8/site-packages/openlineage/airflow/extractors/manager.py", line 16, in init + self.tasktoextractor = Extractors() + File "/opt/python3.8/lib/python3.8/site-packages/openlineage/airflow/extractors/extractors.py", line 122, in init + extractor = importfromstring(extractor.strip()) + File "/opt/python3.8/lib/python3.8/site-packages/openlineage/airflow/utils.py", line 431, in importfromstring + raise ImportError(f"Failed to import {path}") from e +ImportError: Failed to import bigqueryinsertjobextractor.BigQueryInsertJobExtractor +[2023-09-13T13:11:51.235+0000] {pluginsmanager.py:227} DEBUG - Importing entrypoint plugin composermenuplugin +[2023-09-13T13:11:51.719+0000] {pluginsmanager.py:316} DEBUG - Loading 1 plugin(s) took 1.14 seconds +[2023-09-13T13:11:51.733+0000] {triggererjob.py:101} INFO - Starting the triggerer +[2023-09-13T13:11:51.734+0000] {selectorevents.py:59} DEBUG - Using selector: EpollSelector +[2023-09-13T13:11:56.118+0000] {basejob.py:240} DEBUG - [heartbeat] +[2023-09-13T13:12:01.359+0000] {basejob.py:240} DEBUG - [heartbeat] +[2023-09-13T13:12:06.665+0000] {basejob.py:240} DEBUG - [heartbeat] +[2023-09-13T13:12:11.880+0000] {basejob.py:240} DEBUG - [heartbeat] +[2023-09-13T13:12:17.098+0000] {basejob.py:240} DEBUG - [heartbeat] +[2023-09-13T13:12:22.323+0000] {basejob.py:240} DEBUG - [heartbeat] +[2023-09-13T13:12:27.597+0000] {basejob.py:240} DEBUG - [heartbeat] +[2023-09-13T13:12:32.826+0000] {basejob.py:240} DEBUG - [heartbeat] +[2023-09-13T13:12:38.049+0000] {basejob.py:240} DEBUG - [heartbeat] +[2023-09-13T13:12:43.275+0000] {basejob.py:240} DEBUG - [heartbeat] +[2023-09-13T13:12:48.509+0000] {basejob.py:240} DEBUG - [heartbeat] +[2023-09-13T13:12:53.867+0000] {basejob.py:240} DEBUG - [heartbeat] +[2023-09-13T13:12:59.087+0000] {basejob.py:240} DEBUG - [heartbeat] +[2023-09-13T13:13:04.300+0000] {basejob.py:240} DEBUG - [heartbeat] +[2023-09-13T13:13:09.539+0000] {basejob.py:240} DEBUG - [heartbeat] +[2023-09-13T13:13:14.785+0000] {basejob.py:240} DEBUG - [heartbeat] +[2023-09-13T13:13:20.007+0000] {basejob.py:240} DEBUG - [heartbeat] +[2023-09-13T13:13:25.274+0000] {basejob.py:240} DEBUG - [heartbeat] +[2023-09-13T13:13:30.510+0000] {basejob.py:240} DEBUG - [heartbeat] +[2023-09-13T13:13:35.729+0000] {basejob.py:240} DEBUG - [heartbeat] +[2023-09-13T13:13:40.960+0000] {basejob.py:240} DEBUG - [heartbeat] +[2023-09-13T13:13:46.444+0000] {basejob.py:240} DEBUG - [heartbeat] +[2023-09-13T13:13:51.751+0000] {basejob.py:240} DEBUG - [heartbeat] +[2023-09-13T13:13:57.084+0000] {basejob.py:240} DEBUG - [heartbeat] +[2023-09-13T13:14:02.310+0000] {basejob.py:240} DEBUG - [heartbeat] +[2023-09-13T13:14:07.535+0000] {basejob.py:240} DEBUG - [heartbeat] +[2023-09-13T13:14:12.754+0000] {basejob.py:240} DEBUG - [heartbeat] +[2023-09-13T13:14:17.967+0000] {basejob.py:240} DEBUG - [heartbeat] +[2023-09-13T13:14:23.185+0000] {basejob.py:240} DEBUG - [heartbeat] +[2023-09-13T13:14:28.406+0000] {basejob.py:240} DEBUG - [heartbeat] +[2023-09-13T13:14:33.661+0000] {basejob.py:240} DEBUG - [heartbeat] +[2023-09-13T13:14:38.883+0000] {basejob.py:240} DEBUG - [heartbeat] +[2023-09-13T13:14:44.247+0000] {base_job.py:240} DEBUG - [heartbeat]```

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Guntaka Jeevan Paul + (jeevan@acceldata.io) +
+
2023-09-13 09:15:10
 +
+

*Thread Reply:* still the same error in the triggerer pod

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Guntaka Jeevan Paul + (jeevan@acceldata.io) +
+
2023-09-13 09:16:23
 +
+

*Thread Reply:* have changed the dags folder where i have added the init file as you suggested and then have updated the OPENLINEAGEEXTRACTORS to bigqueryinsertjob_extractor.BigQueryInsertJobExtractor…still the same thing

+ +
+
Screenshot 2023-09-13 at 6.45.26 PM.png
+ + + + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-09-13 09:36:27
 +
+

*Thread Reply:* > still the same error in the triggerer pod +it won't change, we're not trying to fix the triggerer import but worker, and should look only at worker pod at this point

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Guntaka Jeevan Paul + (jeevan@acceldata.io) +
+
2023-09-13 09:43:34
 +
+

*Thread Reply:* ```extractor for <class 'airflow.providers.google.cloud.operators.bigquery.BigQueryInsertJobOperator'> is <class 'bigqueryinsertjobextractor.BigQueryInsertJobExtractor'

+ +

Using extractor BigQueryInsertJobExtractor tasktype=BigQueryInsertJobOperator airflowdagid=dataanalyticsdag taskid=joinbqdatasets.bqjoinholidaysweatherdata2021 airflowrunid=manual_2023-09-13T13:24:08.946947+00:00

+ +

fatal: not a git repository (or any parent up to mount point /home/airflow) +Stopping at filesystem boundary (GITDISCOVERYACROSSFILESYSTEM not set). +fatal: not a git repository (or any parent up to mount point /home/airflow) +Stopping at filesystem boundary (GITDISCOVERYACROSSFILESYSTEM not set).```

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Guntaka Jeevan Paul + (jeevan@acceldata.io) +
+
2023-09-13 09:44:44
 +
+

*Thread Reply:* able to see these logs in the worker pod…so what you said is right that it is able to get the extractor but i get the below error immediately where it says not a git repository

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Guntaka Jeevan Paul + (jeevan@acceldata.io) +
+
2023-09-13 09:45:24
 +
+

*Thread Reply:* seems like we are almost there nearby…am i missing something obvious

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-09-13 10:06:35
 +
+

*Thread Reply:* > fatal: not a git repository (or any parent up to mount point /home/airflow) +&gt; Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set). +&gt; fatal: not a git repository (or any parent up to mount point /home/airflow) +&gt; Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set). +hm, this could be the actual bug?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-09-13 10:06:51
 +
+

*Thread Reply:* that’s casual log in composer

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-09-13 10:12:16
 +
+

*Thread Reply:* extractor for &lt;class 'airflow.providers.google.cloud.operators.bigquery.BigQueryInsertJobOperator'&gt; is &lt;class 'big_query_insert_job_extractor.BigQueryInsertJobExtractor' +that’s actually class from your custom module, right?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-09-13 10:14:03
 +
+

*Thread Reply:* I’ve done experiment, that’s how gcs looks like

+ +
+
image.png
+ + + + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-09-13 10:14:09
 +
+

*Thread Reply:* and env vars

+ +
+
image.png
+ + + + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-09-13 10:14:19
 +
+

*Thread Reply:* I have this extractor detected as expected

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-09-13 10:15:06
 +
+

*Thread Reply:* seens as &lt;class 'dependencies.bq.BigQueryInsertJobExtractor'&gt;

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-09-13 10:16:02
 +
+

*Thread Reply:* no __init__.py in base dags folder

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-09-13 10:17:02
 +
+

*Thread Reply:* I also checked that triggerer pod indeed has no gcsfuse set up, tbh no idea why, maybe some kind of optimization +the only effect is that when loading plugins in triggerer it throws some errors in logs, we don’t do anything at the moment there

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Guntaka Jeevan Paul + (jeevan@acceldata.io) +
+
2023-09-13 10:19:26
 +
+

*Thread Reply:* okk…got it @Jakub Dardziński…so the init at the top level of dags is as well not reqd, got it. Just one more doubt, there is a requirement where i want to change the operators property in the extractor inside the extract function, will that be taken into account and the operator’s execute be called with the property that i have populated in my extractor?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Guntaka Jeevan Paul + (jeevan@acceldata.io) +
+
2023-09-13 10:21:28
 +
+

*Thread Reply:* for example i want to add a custom jobid to the BigQueryInsertJobOperator, so wheneerv someone uses the BigQueryInsertJobOperator operator i want to intercept that and add this jobid property to the operator…will that work?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-09-13 10:24:46
 +
+

*Thread Reply:* I’m not sure if using OL for such thing is best choice. Wouldn’t it be better to subclass the operator?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-09-13 10:25:37
 +
+

*Thread Reply:* but the answer is: it dependes on the airflow version, in 2.3+ I’m pretty sure the changed property stays in execute method

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Guntaka Jeevan Paul + (jeevan@acceldata.io) +
+
2023-09-13 10:27:49
 +
+

*Thread Reply:* yeah ideally that is how we should have done this but the problem is our client is having around 1000+ Dag’s in different google cloud projects, which are owned by multiple teams…so they are not willing to change anything in their dag. Thankfully they are using airflow 2.4.3

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-09-13 10:31:15
 +
+

*Thread Reply:* task_policy might be better tool for that: https://airflow.apache.org/docs/apache-airflow/2.6.0/administration-and-deployment/cluster-policies.html

+ + + +
+ ➕ Jakub Dardziński +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-09-13 10:35:30
 +
+

*Thread Reply:* btw I double-checked - execute method is in different process so this would not change task’s attribute there

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Guntaka Jeevan Paul + (jeevan@acceldata.io) +
+
2023-09-16 03:32:49
 +
+

*Thread Reply:* @Jakub Dardziński any idea how can we achieve this one. ---> https://openlineage.slack.com/archives/C01CK9T7HKR/p1694849427228709

+
+ + +
+ + + } + + Guntaka Jeevan Paul + (https://openlineage.slack.com/team/U05QL7LN2GH) +
+ + + +
+
+ <p>@here we have dataproc operator getting called from a dag which submits a spark job, we wanted to maintain that continuity of parent job in the spark job so according to the documentation we can acheive that by using a macro called lineage<em>run</em>id that requires task and task<em>instance as the parameters. The problem we are facing is that our client’s have 1000's of dags, so asking them to change this everywhere it is used is not feasible, so we thought of using the task</em>policy feature in the airflow…but the problem is that task<em>policy gives you access to only the task/operator, but we don’t have the access to the task instance..that is required as a parameter to the lineage</em>run_id function. Can anyone kindly help us on how should we go about this one +<code>t1 = DataProcPySparkOperator( + task_id=job_name, + &lt;b&gt;#required&lt;/b&gt; pyspark configuration, + job_name=job_name, + dataproc_pyspark_properties={ + 'spark.driver.extraJavaOptions': + f"-javaagent:{jar}={os.environ.get('OPENLINEAGE_URL')}/api/v1/namespaces/{os.getenv('OPENLINEAGE_NAMESPACE', 'default')}/jobs/{job_name}/runs/{{{{macros.OpenLineagePlugin.lineage_run_id(task, task_instance)}}}}?api_key={os.environ.get('OPENLINEAGE_API_KEY')}" + dag=dag)</code></p> +
+ + + + + + +
Original URL: https://openlineage.slack.com/archives/C01CK9T7HKR/p1694849427228709
+ + +
+ + Slack Conversation +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Guntaka Jeevan Paul + (jeevan@acceldata.io) +
+
2023-09-12 17:26:01
 +
+

@here has anyone succeded in getting a custom extractor to work in GCP Cloud Composer or AWS MWAA, seems like there is no way

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mars Lan + (mars@metaphor.io) +
+
2023-09-12 17:34:29
 +
+

*Thread Reply:* I'm getting quite close with MWAA. See https://openlineage.slack.com/archives/C01CK9T7HKR/p1692743745585879.

+
+ + +
+ + + } + + Mars Lan + (https://openlineage.slack.com/team/U01HVNU6A4C) +
+ + + +
+
+ <p>Has anyone managed to get the OL Airflow integration to work on AWS MWAA? We've tried pretty much every trick but still ended up with the following error: +<code>Broken plugin: [openlineage.airflow.plugin] No module named 'openlineage.airflow'; 'openlineage' is not a package</code></p> +
+ + + + + + +
Original URL: https://openlineage.slack.com/archives/C01CK9T7HKR/p1692743745585879
+ + +
+ + Thread in Slack Conversation +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Suraj Gupta + (suraj.gupta@atlan.com) +
+
2023-09-13 01:44:27
 +
+

I am exploring Spark - OpenLineage integration (using the latest PySpark and OL versions). I tested a simple pipeline which: +• Reads JSON data into PySpark DataFrame +• Apply data transformations +• Write transformed data to MySQL database +Observed that we receive 4 events (2 START and 2 COMPLETE) for the same job name. The events are almost identical with a small diff in the facets. All the events share the same runId, and we don't get any parentRunId. +Team, can you please confirm if this behaviour is expected? Seems to be different from the Airflow integration where we relate jobs to Parent Jobs.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Damien Hawes + (damien.hawes@booking.com) +
+
2023-09-13 02:54:37
 +
+

*Thread Reply:* The Spark integration requires that two parameters are passed to it, namely:

+ +

spark.openlineage.parentJobName +spark.openlineage.parentRunId +You can find the list of parameters here:

+ +

https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/README.md

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Suraj Gupta + (suraj.gupta@atlan.com) +
+
2023-09-13 02:55:51
 +
+

*Thread Reply:* Thanks, will check this out

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Damien Hawes + (damien.hawes@booking.com) +
+
2023-09-13 02:57:43
 +
+

*Thread Reply:* As for double accounting of events - that's a bit harder to diagnose.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-09-13 04:33:03
 +
+

*Thread Reply:* Can you share the the job and events? +Also @Paweł Leszczyński

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Suraj Gupta + (suraj.gupta@atlan.com) +
+
2023-09-13 06:03:49
 +
+

*Thread Reply:* Sure, sharing Job and events.

+ +
+
+ + + + + + +
+
+
spark_events.txt
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Suraj Gupta + (suraj.gupta@atlan.com) +
+
2023-09-13 06:06:21
 +
+

*Thread Reply:*

+ +
+
etl-mysql.py
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-09-13 06:39:02
 +
+

*Thread Reply:* Hi @Suraj Gupta,

+ +

Thanks for providing such a detailed description of the problem.

+ +

It is not expected behaviour, it's an issue. The events correspond to the same logical plan which for some reason lead to sending two OL events. Is it reproducible aka. does it occur each time? If yes, we please feel free to raise an issue for that.

+ +

We have added in recent months several tests to verify amount of OL events being generated but we haven't tested it that way with JDBC. BTW. will the same happen if you write your data df_transformed to a file (like parquet file) ?

+ + + +
+ :gratitude_thank_you: Suraj Gupta +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Suraj Gupta + (suraj.gupta@atlan.com) +
+
2023-09-13 07:28:03
 +
+

*Thread Reply:* Thanks @Paweł Leszczyński, will confirm about writing to file and get back.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Suraj Gupta + (suraj.gupta@atlan.com) +
+
2023-09-13 07:33:35
 +
+

*Thread Reply:* And yes, the issue is reproducible. Will raise an issue for this.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-09-13 07:33:54
 +
+

*Thread Reply:* even if you write onto a file?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Suraj Gupta + (suraj.gupta@atlan.com) +
+
2023-09-13 07:37:21
 +
+

*Thread Reply:* Yes, even when I write to a parquet file.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-09-13 07:49:28
 +
+

*Thread Reply:* ok. i think i was able to reproduce it locally with https://github.com/OpenLineage/OpenLineage/pull/2103/files

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Suraj Gupta + (suraj.gupta@atlan.com) +
+
2023-09-13 07:56:11
 +
+

*Thread Reply:* Opened an issue: https://github.com/OpenLineage/OpenLineage/issues/2104

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Suraj Gupta + (suraj.gupta@atlan.com) +
+
2023-09-25 16:32:09
 +
+

*Thread Reply:* @Paweł Leszczyński I see that the PR is work in progress. Any rough estimate on when we can expect this fix to be released?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-09-26 03:32:03
 +
+

*Thread Reply:* @Suraj Gupta put a comment within your issue. it's a bug we need to solve but I cannot bring any estimates today.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Suraj Gupta + (suraj.gupta@atlan.com) +
+
2023-09-26 04:33:03
 +
+

*Thread Reply:* Thanks for update @Paweł Leszczyński, also please look into this comment. It might related and I'm not sure if expected behaviour.

+
+ + + + +
Comment on #2104 [Spark Integration] Receiving duplicate OpenLineage events for a Spark Job
+
+ <p>Hi <a href="https://github.com/pawel-big-lebowski">@pawel-big-lebowski</a> I understand your point. I've also come across cases where the duplicate events have the exact same payload with no difference in facets. I'm assuming there should be some diff in the payload for the backend to merge them. I've also seen a single Job emit 7 events and there is no fixed pattern for this. <br /> +I also agree that there should be only a single START and a single COMPLETE event for any Job since this what the OL spec says.</p> + +<p>This Job emits 7 events (5 Start and 2 Complete):</p> + +<pre><code>from pyspark.sql import SparkSession +from pyspark.sql.functions import col +from pyspark.sql.functions import lit +import random + +spark = (SparkSession.builder.master('local') + .appName('spark-pipeline-v1') + .config('spark.jars.packages', "io.openlineage:openlineage_spark:1.1.0," + "mysql:mysql_connector_java:8.0.33," + "net.snowflake:snowflake_jdbc:3.13.14," + "net.snowflake:spark_snowflake_2.12:2.10.0-spark_3.2") + .config('spark.extraListeners', 'io.openlineage.spark.agent.OpenLineageSparkListener') + .config('spark.openlineage.transport.type', 'http') + .config('spark.openlineage.transport.url', '&lt;http://host.docker.internal:5009&gt;') + .config('spark.openlineage.transport.endpoint', '/events/openlineage/spark/api/v1/lineage') + .config('spark.openlineage.namespace', 'staging') + .config('spark.openlineage.transport.auth.type', 'api_key') + .config('spark.openlineage.transport.auth.apiKey', 'test-key') + .config('spark.openlineage.parentJobName', 'suraj-test-job') + .config('spark.openlineage.parentRunId', 'acd-eheh-ththth-wnjwnj') + .getOrCreate()) + + +mysql_connection_properties = { + "user": "&lt;user&gt;", + "password": "&lt;pwd&gt;", + "driver": "com.mysql.cj.jdbc.Driver", +} + +snowflake_options = { + "sfURL": "&lt;account&gt;.&lt;a href="http://snowflakecomputing.com"&gt;snowflakecomputing.com&lt;/a&gt;", + "sfUser": "&lt;usr&gt;", + "sfPassword": "&lt;pwd&gt;", + "sfDatabase": "ANALYTICS", + "sfWarehouse": "COMPUTE_WH", + "sfSchema": "PUBLIC", + "sfRole": "ACCOUNTADMIN", +} + +mysql_url = "&lt;mysql-jdbc-url&gt;" + +cats_df = spark.read.jdbc(url=mysql_url, table="cats", properties=mysql_connection_properties) +dogs_df = spark.read.jdbc(url=mysql_url, table="dogs", properties=mysql_connection_properties) + +cats_df = cats_df.withColumnRenamed("Country", "cat_name") + +joined_df = cats_df.join(dogs_df, on="owner", how="inner") + +filtered_df = joined_df.filter(col("cat_name") == 'Cookie') + +agg_df = filtered_df.groupBy("cat_name").count() + +agg_df = agg_df.withColumn("id", lit(random.randint(0, 1000))) + +agg_df.write \ + .format("snowflake") \ + .options(****snowflake_options) \ + .option("dbtable", "COUNT_TABLE") \ + .mode("overwrite") \ + .save() + +spark.stop() +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-09-13 14:20:32
 +
+

@channel +This month’s TSC meeting, open to all, is tomorrow: https://openlineage.slack.com/archives/C01CK9T7HKR/p1694113940400549

+
+ + +
+ + + } + + Michael Robinson + (https://openlineage.slack.com/team/U02LXF3HUN7) +
+ + + +
+
+ <p>@channel +This month’s TSC meeting is next Thursday the 14th at 10am PT. On the tentative agenda: +• announcements +• recent releases +• demo: Spark integration tests in Databricks runtime +• open discussion +• more (TBA) +More info and the meeting link can be found on the <a href="https://openlineage.io/meetings/">website</a>. All are welcome! Also, feel free to reply or DM me with discussion topics, agenda items, etc.</p> +
+ + + + + + +
Original URL: https://openlineage.slack.com/archives/C01CK9T7HKR/p1694113940400549
+ + +
+ + Slack Conversation +
+ + +
+ + + +
+ ✅ Sheeri Cabral (Collibra) +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Damien Hawes + (damien.hawes@booking.com) +
+
2023-09-14 06:20:15
 +
+

Context:

+ +

We use Spark with YARN, running on Hadoop 2.x (I can't remember the exact minor version) with Hive support.

+ +

Problem:

+ +

I'm noticed that CreateDataSourceAsSelectCommand objects are always transformed to an OutputDataset with a namespace value set to file - which is curious, because the inputs always have a (correct) namespace of hdfs://&lt;name-node&gt; - is this a known issue? A flaw with Apache Spark? A bug in the resolution logic?

+ +

For reference:

+ +

```public class CreateDataSourceTableCommandVisitor + extends QueryPlanVisitor<CreateDataSourceTableCommand, OpenLineage.OutputDataset> {

+ +

public CreateDataSourceTableCommandVisitor(OpenLineageContext context) { + super(context); + }

+ +

@Override + public List<OpenLineage.OutputDataset> apply(LogicalPlan x) { + CreateDataSourceTableCommand command = (CreateDataSourceTableCommand) x; + CatalogTable catalogTable = command.table();

+ +
return Collections.singletonList(
+    outputDataset()
+        .getDataset(
+            PathUtils.fromCatalogTable(catalogTable),
+            catalogTable.schema(),
+            OpenLineage.LifecycleStateChangeDatasetFacet.LifecycleStateChange.CREATE));
+
+ +

} +}`` +Running this:cat events.log | jq '{eventTime: .eventTime, eventType: .eventType, runId: .run.runId, jobNamespace: .job.namespace, jobName: .job.name, outputs: .outputs[] | {namespace: .namespace, name: .name}, inputs: .inputs[] | {namespace: .namespace, name: .name}}'`

+ +

This is an output: +{ + "eventTime": "2023-09-13T16:01:27.059Z", + "eventType": "START", + "runId": "bbbb5763-3615-46c0-95ca-1fc398c91d5d", + "jobNamespace": "spark.cluster-1", + "jobName": "ol_hadoop_test.execute_create_data_source_table_as_select_command.dhawes_db_ol_test_hadoop_tgt", + "outputs": { + "namespace": "file", + "name": "/user/hive/warehouse/dhawes.db/ol_test_hadoop_tgt" + }, + "inputs": { + "namespace": "<hdfs://nn1>", + "name": "/user/hive/warehouse/dhawes.db/ol_test_hadoop_src" + } +}

+ + + +
+ 👀 Paweł Leszczyński +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-09-14 07:32:25
 +
+

*Thread Reply:* Seems like an issue on our side. Do you know how the source is read? What LogicalPlan leaf is used to read src? Would love to find how is this done differently

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Damien Hawes + (damien.hawes@booking.com) +
+
2023-09-14 09:16:58
 +
+

*Thread Reply:* Hmm, I'll have to do explain plan to see what exactly it is.

+ +

However my sample job uses spark.sql("SELECT ** FROM dhawes.ol_test_hadoop_src")

+ +

which itself is created using

+ +

spark.sql("SELECT 1 AS id").write.format("orc").mode("overwrite").saveAsTable("dhawes.ol_test_hadoop_src")

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Damien Hawes + (damien.hawes@booking.com) +
+
2023-09-14 09:23:59
 +
+

*Thread Reply:* ``&gt;&gt;&gt; spark.sql("SELECT ** FROM dhawes.ol_test_hadoop_src").explain(True) +== Parsed Logical Plan == +'Project [**] ++- 'UnresolvedRelationdhawes.oltesthadoop_src`

+ +

== Analyzed Logical Plan == +id: int +Project [id#3] ++- SubqueryAlias dhawes.ol_test_hadoop_src + +- Relation[id#3] orc

+ +

== Optimized Logical Plan == +Relation[id#3] orc

+ +

== Physical Plan == +**(1) FileScan orc dhawes.oltesthadoop_src[id#3] Batched: true, Format: ORC, Location: InMemoryFileIndex[], PartitionFilters: [], PushedFilters: [], ReadSchema: struct<id:int>```

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
tati + (tatiana.alchueyr@astronomer.io) +
+
2023-09-14 10:03:41
 +
+

Hey everyone, +Any chance we could have a openlineage-integration-common 1.1.1 release with the following changes..? +• https://github.com/OpenLineage/OpenLineage/pull/2106 +• https://github.com/OpenLineage/OpenLineage/pull/2108

+ + + +
+ ➕ Michael Robinson, Harel Shein, Maciej Obuchowski, Jakub Dardziński, Paweł Leszczyński, Julien Le Dem +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
tati + (tatiana.alchueyr@astronomer.io) +
+
2023-09-14 10:05:19
 +
+

*Thread Reply:* Specially the first PR is affecting users of the astronomer-cosmos library: https://github.com/astronomer/astronomer-cosmos/issues/533

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-09-14 10:05:24
 +
+

*Thread Reply:* Thanks @tati for requesting your first OpenLineage release! Three +1s from committers will authorize

+ + + +
+ :gratitude_thank_you: tati +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-09-14 11:59:55
 +
+

*Thread Reply:* The release is authorized and will be initiated within two business days.

+ + + +
+ 🎉 tati +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
tati + (tatiana.alchueyr@astronomer.io) +
+
2023-09-15 04:40:12
 +
+

*Thread Reply:* Thanks a lot, @Michael Robinson!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2023-09-14 20:23:01
 +
+

Per discussion in the OpenLineage sync today here is a very early strawman proposal for an OpenLineage registry that producers and consumers could be registered in. +Feedback or alternate proposals welcome +https://docs.google.com/document/d/1zIxKST59q3I6ws896M4GkUn7IsueLw8ejct5E-TR0vY/edit +Once this is sufficiently fleshed out, I’ll create an actual proposal on github

+ + + +
+ 👍 Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2023-10-03 20:33:35
 +
+

*Thread Reply:* I have cleaned up the registry proposal. +https://docs.google.com/document/d/1zIxKST59q3I6ws896M4GkUn7IsueLw8ejct5E-TR0vY/edit +In particular: +• I clarified that option 2 is preferred at this point. +• I moved discussion notes to the bottom. they will go away at some point +• Once it is stable, I’ll create a proposal with the preferred option. +• we need a good proposal for the core facets prefix. My suggestion is to move core facets to core in the registry. The drawback is prefix would be inconsistent.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2023-10-05 17:34:12
 +
+

*Thread Reply:* I have created a ticket to make this easier to find. Once I get more feedback I’ll turn it into a md file in the repo: https://docs.google.com/document/d/1zIxKST59q3I6ws896M4GkUn7IsueLw8ejct5E-TR0vY/edit#heading=h.enpbmvu7n8gu +https://github.com/OpenLineage/OpenLineage/issues/2161

+
+ + + + +
#2161 [PROPOSAL] Add a Registry of Producers and Consumers in OpenLineage
+
+ <p><strong>Purpose</strong></p> + +<p>This is the early stage of an idea to get community feedback on what an OpenLineage registry for producers, custom facets and consumers could be. Once this document is stable enough, I’ll create an official proposal on the OpenLineage repo.</p> + +<p><strong>Goal</strong></p> + +<p>Allow third parties to register their implementations or custom extensions to make them easy to discover. <br /> +Shorten “Producer” and “schema url” values</p> + +<p><strong>Proposed implementation</strong></p> + +<p>Current draft for discussion:</p> + +<p><a href="https://docs.google.com/document/d/1zIxKST59q3I6ws896M4GkUn7IsueLw8ejct5E-TR0vY/edit#heading=h.br8d2vy9wme9">https://docs.google.com/document/d/1zIxKST59q3I6ws896M4GkUn7IsueLw8ejct5E-TR0vY/edit#heading=h.br8d2vy9wme9</a></p> +
+ +
+
Labels
+ proposal +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-09-15 12:03:27
 +
+

@channel +Friendly reminder: the next OpenLineage meetup, our first in Toronto, is happening this coming Monday at 5 PM ET https://openlineage.slack.com/archives/C01CK9T7HKR/p1694441261486759

+
+ + +
+ + + } + + Michael Robinson + (https://openlineage.slack.com/team/U02LXF3HUN7) +
+ + + +
+
+ <p>@channel +The first Toronto OpenLineage Meetup, featuring a presentation by recent adopter <a href="https://metaphor.io/">Metaphor</a>, is just one week away. On the agenda:</p> + +<ol><li><strong>Evolution of spec presentation/discussion (project background/history)</strong></li><li><strong>State of the community</strong></li><li><strong>Integrating OpenLineage with <a href="https://metaphor.io/">Metaphor</a> (by special guests <a href="https://www.linkedin.com/in/yeliu84/">Ye</a> &amp; <a href="https://www.linkedin.com/in/ivanperepelitca/">Ivan</a>)</strong></li><li><strong>Spark/Column lineage update</strong></li><li><strong>Airflow Provider update</strong></li><li><strong>Roadmap Discussion</strong> +<strong>Find more details and RSVP <a href="https://www.meetup.com/openlineage/events/295488014/?utm_medium=referral&amp;utm_campaign=share-btn_savedevents_share_modal&amp;utm_source=link|here">https://www.meetup.com/openlineage/events/295488014/?utm<em>medium=referral&amp;utm</em>campaign=share-btn<em>savedevents</em>share<em>modal&amp;utm</em>source=link|here</a></strong>.</li> +</ol> +
+ + + + + + +
Original URL: https://openlineage.slack.com/archives/C01CK9T7HKR/p1694441261486759
+ + +
+ + Slack Conversation +
+ + +
+ + + +
+ 👍 Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Guntaka Jeevan Paul + (jeevan@acceldata.io) +
+
2023-09-16 03:30:27
 +
+

@here we have dataproc operator getting called from a dag which submits a spark job, we wanted to maintain that continuity of parent job in the spark job so according to the documentation we can acheive that by using a macro called lineagerunid that requires task and taskinstance as the parameters. The problem we are facing is that our client’s have 1000's of dags, so asking them to change this everywhere it is used is not feasible, so we thought of using the taskpolicy feature in the airflow…but the problem is that taskpolicy gives you access to only the task/operator, but we don’t have the access to the task instance..that is required as a parameter to the lineagerun_id function. Can anyone kindly help us on how should we go about this one +t1 = DataProcPySparkOperator( + task_id=job_name, + <b>#required</b> pyspark configuration, + job_name=job_name, + dataproc_pyspark_properties={ + 'spark.driver.extraJavaOptions': + f"-javaagent:{jar}={os.environ.get('OPENLINEAGE_URL')}/api/v1/namespaces/{os.getenv('OPENLINEAGE_NAMESPACE', 'default')}/jobs/{job_name}/runs/{{{{macros.OpenLineagePlugin.lineage_run_id(task, task_instance)}}}}?api_key={os.environ.get('OPENLINEAGE_API_KEY')}" + dag=dag)

+ + + +
+ ➕ Abdallah +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-09-16 04:22:47
 +
+

*Thread Reply:* you don't need actual task instance to do that. you only should set additional argument as jinja template, same as above

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-09-16 04:25:28
 +
+

*Thread Reply:* task_instance in this case is just part of string which is evaluated when jinja render happens

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Guntaka Jeevan Paul + (jeevan@acceldata.io) +
+
2023-09-16 04:27:10
 +
+

*Thread Reply:* ohh…then we could use the same example as above inside the task_policy to intercept the Operator and add the openlineage specific additions properties?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-09-16 04:30:59
 +
+

*Thread Reply:* correct, just remember not to override all properties, just add ol specific

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Guntaka Jeevan Paul + (jeevan@acceldata.io) +
+
2023-09-16 04:32:02
 +
+

*Thread Reply:* yeah sure…thank you so much @Jakub Dardziński, will try this out and keep you posted

+ + + +
+ 👍 Jakub Dardziński +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-09-16 05:00:24
 +
+

*Thread Reply:* We want to automate setting those options at some point inside the operator itself

+ + + +
+ ➕ Guntaka Jeevan Paul +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Guntaka Jeevan Paul + (jeevan@acceldata.io) +
+
2023-09-16 19:40:27
 +
+

@here is there a way by which we could add custom headers to openlineage client in airflow, i see that provision is there for spark integration via these properties spark.openlineage.transport.headers.xyz --> abcdef

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-09-19 16:40:55
 +
+

*Thread Reply:* there’s no out-of-the-box possibility to do that yet, you’re very welcome to create an issue in GitHub and maybe contribute as well! 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mars Lan + (mars@metaphor.io) +
+
2023-09-17 09:07:41
 +
+

It doesn't seem like there's a way to override the OL endpoint from the default (/api/v1/lineage) in Airflow? I tried setting the OPENLINEAGE_ENDPOINT environment to no avail. Based on this statement, it seems that only OPENLINEAGE_URL was used to construct HttpConfig ?

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/client/python/openlineage/client/transport/factory.py | factory.py>
+
+ <pre><code> config = HttpConfig( + url=os.environ["OPENLINEAGE_URL"], + auth=create_token_provider( + { + "type": "api_key", + "apiKey": os.environ.get("OPENLINEAGE_API_KEY", ""), + }, + ), +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-09-18 16:25:11
 +
+

*Thread Reply:* That’s correct. For now there’s no way to configure the endpoint via env var. You can do that by using config file

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mars Lan + (mars@metaphor.io) +
+
2023-09-18 16:30:39
 +
+

*Thread Reply:* How do you do that in Airflow? Any particular reason for excluding endpoint override via env var? Happy to create a PR to fix that.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-09-18 16:52:48
 +
+

*Thread Reply:* historical I guess? go for the PR, of course 🚀

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mars Lan + (mars@metaphor.io) +
+
2023-10-03 08:52:16
 +
+

*Thread Reply:* https://github.com/OpenLineage/OpenLineage/pull/2151

+
+ + + + +
#2151 Allow setting client's endpoint via environment variable
+
+ <p><strong>Problem</strong></p> + +<p>Currently, it's not possible to set the OpenLineage endpoint (hard-coded to <code>/api/v1/lineage</code>) using an environment variable when running the Airflow integration.</p> + +<p><strong>Solution</strong></p> + +<p>Given that it's not possible to create the client manually in Airflow, especially now that OpenLineage has become an official Airflow provider, this change seems like the only feasible solution.</p> + +<p>☐ Your change modifies the <a href="https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.json">core</a> OpenLineage model +☐ Your change modifies one or more OpenLineage <a href="https://github.com/OpenLineage/OpenLineage/tree/main/spec/facets">facets</a></p> + +<p><strong>One-line summary:</strong></p> + +<p>Allow setting client's endpoint via environment variable.</p> + +<p><strong>Checklist</strong></p> + +<p>☑︎ You've <a href="https://github.com/OpenLineage/OpenLineage/blob/main/why-the-dco.md">signed-off</a> your work +☑︎ Your pull request title follows our <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CONTRIBUTING.md#creating-pull-requests">guidelines</a> +☑︎ Your changes are accompanied by tests (<em>if relevant</em>) +☑︎ Your change contains a <a href="https://kurtisnusbaum.medium.com/stacked-diffs-keeping-phabricator-diffs-small-d9964f4dcfa6">small diff</a> and is self-contained +☑︎ You've updated any relevant documentation (<em>if relevant</em>) +☑︎ Your comment includes a one-liner for the changelog about the specific purpose of the change (<em>if necessary</em>) +☐ You've versioned the core OpenLineage model or facets according to <a href="https://docs.snowplowanalytics.com/docs/pipeline-components-and-applications/iglu/common-architecture/schemaver">SchemaVer</a> (<em>if relevant</em>) +☐ You've added a <a href="https://github.com/OpenLineage/OpenLineage/tree/main/.github/header_templates.md">header</a> to source files (<em>if relevant</em>)</p> + +<hr /> + +<p>SPDX-License-Identifier: Apache-2.0 <br /> +Copyright 2018-2023 contributors to the OpenLineage project</p> +
+ +
+
Labels
+ documentation, client/python +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Terese Larsson + (terese@jclab.se) +
+
2023-09-18 08:22:34
 +
+

Hi! I'm in need of help with wrapping my head around OpenLineage. My team have the goal of collecting metadata from the Airflow operators GreatExpectationsOperator, PythonOperator, MsSqlOperator and BashOperator (for dbt). Where can I see the sourcecode for what is collected for each operator, and is there support for these in the new provider apache-airflow-providers-openlineage? I am super confused and feel lost in the docs. 🤯 We are using MSSQL/ODBC to connect to our db, and this data does not seem to appear as datasets in Marquez, do I need to configure this? If so, HOW and WHERE? 🥲

+ +

Happy for any help, big or small! 🙏

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-09-18 16:26:07
 +
+

*Thread Reply:* there’s no actual single source of what integrations are currently implemented in openlineage Airflow provider. That’s something we should work on so it’s more visible

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-09-18 16:26:46
 +
+

*Thread Reply:* answering this quickly - GE & MS SQL are not currently implemented yet in the provider

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-09-18 16:26:58
 +
+

*Thread Reply:* but I also invite you to contribute if you’re interested! 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
sarathch + (sarathch@hpe.com) +
+
2023-09-19 02:47:47
 +
+

Hi I need help in extracting OpenLineage for PostgresOperator in json format. +any suggestions or comments would be greatly appreciated

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-09-19 16:40:06
 +
+

*Thread Reply:* If you're using Airflow 2.7, take a look at https://airflow.apache.org/docs/apache-airflow-providers-openlineage/stable/guides/user.html

+ + + +
+ ❤️ sarathch +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-09-19 16:40:54
 +
+

*Thread Reply:* If you use one of the lower versions, take a look here https://openlineage.io/docs/integrations/airflow/usage

+
+
openlineage.io
+ + + +
Using the Airflow integration | OpenLineage
+
+ PREREQUISITES +
+ + + + + + +
Original URL: https://openlineage.io/docs/integrations/airflow/usage
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
sarathch + (sarathch@hpe.com) +
+
2023-09-20 06:26:56
 +
+

*Thread Reply:* Maciej, +Thanks for sharing the link https://airflow.apache.org/docs/apache-airflow-providers-openlineage/stable/guides/user.html +this should address the issue

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Juan Luis Cano Rodríguez + (juan_luis_cano@mckinsey.com) +
+
2023-09-20 09:36:54
 +
+

congrats folks 🥳 https://lfaidata.foundation/blog/2023/09/20/lf-ai-data-foundation-announces-graduation-of-openlineage-project

+ + + +
+ 🎉 Jakub Dardziński, Mars Lan, Ross Turk, Guntaka Jeevan Paul, Peter Hicks, Maciej Obuchowski, Athitya Kumar, John Lukenoff, Harel Shein, Francis McGregor-Macdonald, Laurent Paris +
+ +
+ 👍 Athitya Kumar +
+ +
+ ❤️ Harel Shein +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-09-20 17:08:58
 +
+

@channel +We released OpenLineage 1.2.2! +Added +• Spark: publish the ProcessingEngineRunFacet as part of the normal operation of the OpenLineageSparkEventListener #2089 @d-m-h +• Spark: capture and emit spark.databricks.clusterUsageTags.clusterAllTags variable from databricks environment #2099 @Anirudh181001 +Fixed +• Common: support parsing dbt_project.yml without target-path #2106 @tatiana +• Proxy: fix Proxy chart #2091 @harels +• Python: fix serde filtering #2044 @xli-1026 +• Python: use non-deprecated apiKey if loading it from env variables @2029 @mobuchowski +• Spark: Improve RDDs on S3 integration. #2039 @pawel-big-lebowski +• Flink: prevent sending running events after job completes #2075 @pawel-big-lebowski +• Spark & Flink: Unify dataset naming from URI objects #2083 @pawel-big-lebowski +• Spark: Databricks improvements #2076 @pawel-big-lebowski +Removed +• SQL: remove sqlparser dependency from iface-java and iface-py #2090 @JDarDagran +Thanks to all the contributors, including new contributors @tati, @xli-1026, and @d-m-h! +Release: https://github.com/OpenLineage/OpenLineage/releases/tag/1.2.2 +Changelog: https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md +Commit history: https://github.com/OpenLineage/OpenLineage/compare/1.1.0...1.2.2 +Maven: https://oss.sonatype.org/#nexus-search;quick~openlineage +PyPI: https://pypi.org/project/openlineage-python/

+ + + +
+ 🔥 Maciej Obuchowski, Harel Shein, Anirudh Shrinivason +
+ +
+ 👍 Guntaka Jeevan Paul, John Rosenbaum, Sangeeta Mishra +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Yevhenii Soboliev + (esoboliev@griddynamics.com) +
+
2023-09-22 21:05:20
 +
+

*Thread Reply:* Hi @Michael Robinson Thank you! I love the job that you’ve done. If you have a few seconds, please hint at how I can push lineage gathered from Airflow and Spark jobs into DataHub for visualization? I didn’t find any solutions or official support neither at Openlineage nor at DataHub, but I still want to continue using Openlineage

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-09-22 21:30:22
 +
+

*Thread Reply:* Hi Yevhenii, thank you for using OpenLineage. The DataHub integration is new to us, but perhaps the experts on Spark and Airflow know more. @Paweł Leszczyński @Maciej Obuchowski @Jakub Dardziński

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-09-23 08:11:17
 +
+

*Thread Reply:* @Yevhenii Soboliev at Airflow Summit, Shirshanka Das from DataHub mentioned this as upcoming feature.

+ + + +
+ 👍 Yevhenii Soboliev +
+ +
+ 🎯 Yevhenii Soboliev +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Suraj Gupta + (suraj.gupta@atlan.com) +
+
2023-09-21 02:11:10
 +
+

Hi, we're using Custom Operators in airflow(2.5) and are planning to expose lineage via default extractors: https://openlineage.io/docs/integrations/airflow/default-extractors/ +Question: Now if we upgrade our Airflow version to 2.7 in the future, would our code be backward compatible? +Since OpenLineage has now moved inside airflow and I think there is no concept of extractors in the latest version.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Suraj Gupta + (suraj.gupta@atlan.com) +
+
2023-09-21 02:15:00
 +
+

*Thread Reply:* Also, do we have any docs on how OL works with the latest airflow version? Few questions: +• How is it replacing the concept of custom extractors and Manually Annotated Lineage in the latest version? +• Do we have any examples of setting up the integration to emit input/output datasets for non supported Operators like PythonOperator?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-09-27 10:04:09
 +
+

*Thread Reply:* > Question: Now if we upgrade our Airflow version to 2.7 in the future, would our code be backward compatible? +It will be compatible, “default extractors” is generally the same concept as we’re using in the 2.7 integration. +One thing that might be good to update is import paths, from openlineage.airflow to airflow.providers.openlineage but should work both ways

+ +

> • Do we have any code samples/docs of setting up the integration to emit input/output datasets for non supported Operators like PythonOperator? +Our experience with that is currently lacking - this means, it works like in bare airflow, if you annotate your PythonOperator tasks with old Airflow lineage like in this doc.

+ +

We want to make this experience better - by doing few things +• instrumenting hooks, then collecting lineage from them +• integration with AIP-48 datasets +• allowing to emit lineage collected inside Airflow task by other means, by providing core Airflow API for that +All those things require changing core Airflow in a couple of ways: +• tracking which hooks were used during PythonOperator execution +• just being able to emit datasets (airflow inlets/outlets) from inside of a task - they are now a static thing, so if you try that it does not work +• providing better API for emitting that lineage, preferably based on OpenLineage itself rather than us having to convert that later. +As this requires core Airflow changes, it won’t be live until Airflow 2.8 at the earliest.

+ +

thanks to @Maciej Obuchowski for this response

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason Yip + (jasonyip@gmail.com) +
+
2023-09-21 18:36:17
 +
+

I am using this accelerator that leverages OpenLineage on Databricks to publish lineage info to Purview, but it's using a rather old version of OpenLineage aka 0.18, anybody has tried it on a newer version of OpenLineage? I am facing some issues with the inputs and outputs for the same object is having different json +https://github.com/microsoft/Purview-ADB-Lineage-Solution-Accelerator/

+
+ + + + +
microsoft/Purview-ADB-Lineage-Solution-Accelerator
+
+ A connector to ingest Azure Databricks lineage into Microsoft Purview +
+ +
+
Stars
+ 77 +
+ +
+
Language
+ C# +
+ + + + + + + + +
+ + + +
+ ✅ Harel Shein +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason Yip + (jasonyip@gmail.com) +
+
2023-09-21 21:51:41
 +
+

I installed 1.2.2 on Databricks, followed the below init script: https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/databricks/open-lineage-init-script.sh

+ +

my cluster config looks like this:

+ +

spark.openlineage.version v1 +spark.openlineage.namespace adb-5445974573286168.8#default +spark.openlineage.endpoint v1/lineage +spark.openlineage.url.param.code 8kZl0bo2TJfnbpFxBv-R2v7xBDj-PgWMol3yUm5iP1vaAzFu9kIZGg== +spark.openlineage.url https://f77b-50-35-69-138.ngrok-free.app

+ +

But it is not calling the API, it works fine with 0.18 version

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/databricks/open-lineage-init-script.sh | open-lineage-init-script.sh>
+
+ <pre><code>#!/bin/bash +# +# Copyright 2018-2023 contributors to the OpenLineage project +# SPDX-License-Identifier: Apache-2.0 + +STAGE_DIR="/dbfs/databricks/openlineage" + +echo "BEGIN: Upload Spark Listener JARs" +cp -f $STAGE_DIR/openlineage-spark-**.jar /mnt/driver-daemon/jars || { echo "Error copying Spark Listener library file"; exit 1;} +echo "END: Upload Spark Listener JARs" + +echo "BEGIN: Modify Spark config settings" +cat &lt;&lt; 'EOF' &gt; /databricks/driver/conf/openlineage-spark-driver-defaults.conf +[driver] { + "spark.extraListeners" = "io.openlineage.spark.agent.OpenLineageSparkListener" +} +EOF +echo "END: Modify Spark config settings" + +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+ ✅ Harel Shein +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason Yip + (jasonyip@gmail.com) +
+
2023-09-21 23:16:10
 +
+

I am attaching the log4j, there is no openlineagecontext

+ +
+
log4j-active (3).txt
+ + + + + + +
+ + +
+ ✅ Harel Shein +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason Yip + (jasonyip@gmail.com) +
+
2023-09-21 23:47:22
 +
+

*Thread Reply:* this issue is resolved, solution can be found here: https://openlineage.slack.com/archives/C01CK9T7HKR/p1691592987038929

+
+ + +
+ + + } + + Zahi Fail + (https://openlineage.slack.com/team/U05KNSP01TR) +
+ + + +
+
+ <p>Hey, +I’m running Spark application (spark version 3.4) with OL integration. +I changed spark to use “debug” level, and I see the OL events with the below message: +“Emitting lineage completed successfully:”</p> + +<p>With all the above, I can’t see the event in Marquez.</p> + +<p>Attaching the OL configurations. +When changing the OL-spark version to 0.6.+, I do see event created in Marquez with only “Start” status (attached below).</p> + +<p>The OL-spark version is matching the Spark version? Is there a known issues with the Spark / OL versions ?</p> +
+ + + + + + +
Original URL: https://openlineage.slack.com/archives/C01CK9T7HKR/p1691592987038929
+ + +
+ + Thread in Slack Conversation +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harel Shein + (harel.shein@gmail.com) +
+
2023-09-25 08:59:10
 +
+

*Thread Reply:* We were all out at Airflow Summit last week, so apologies for the delayed response. Glad you were able to resolve the issue!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sangeeta Mishra + (sangeeta@acceldata.io) +
+
2023-09-25 05:11:50
 +
+

@here I'm presently addressing a particular scenario that pertains to Openlineage authentication, specifically involving the use of an access key and secret.

+ +

I've implemented a custom token provider called AccessKeySecretKeyTokenProvider, which extends the TokenProvider class. This token provider communicates with another service, obtaining a token and an expiration time based on the provided access key, secret, and client ID.

+ +

My goal is to retain this token in a cache prior to its expiration, thereby eliminating the need for network calls to the third-party service. Is it possible without relying on an external caching system.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harel Shein + (harel.shein@gmail.com) +
+
2023-09-25 08:56:53
 +
+

*Thread Reply:* Hey @Sangeeta Mishra, I’m not sure that I fully understand your question here. What do you mean by OpenLineage authentication? +What are you using to generate OL events? What’s your OL receiving backend?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sangeeta Mishra + (sangeeta@acceldata.io) +
+
2023-09-25 09:04:33
 +
+

*Thread Reply:* Hey @Harel Shein, +I wanted to clarify the previous message. I apologize for any confusion. When I mentioned "OpenLineage authentication," I was actually referring to the authentication process for the OpenLineage backend, specifically using HTTP transport. This involves using my custom token provider, which utilizes access keys and secrets for authentication. The OL backend is http based backend . I hope this clears things up!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harel Shein + (harel.shein@gmail.com) +
+
2023-09-25 09:05:12
 +
+

*Thread Reply:* Are you using Marquez?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sangeeta Mishra + (sangeeta@acceldata.io) +
+
2023-09-25 09:05:55
 +
+

*Thread Reply:* We are trying to leverage our own backend here.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harel Shein + (harel.shein@gmail.com) +
+
2023-09-25 09:07:03
 +
+

*Thread Reply:* I see.. I’m not sure the OpenLineage community could help here. Which webserver framework are you using?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sangeeta Mishra + (sangeeta@acceldata.io) +
+
2023-09-25 09:08:56
 +
+

*Thread Reply:* KTOR framework

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sangeeta Mishra + (sangeeta@acceldata.io) +
+
2023-09-25 09:15:33
 +
+

*Thread Reply:* Our backend authentication operates based on either a pair of keys or a single bearer token, with a limited time of expiry. Hence, wanted to cache this information inside the token provider.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harel Shein + (harel.shein@gmail.com) +
+
2023-09-25 09:26:57
 +
+

*Thread Reply:* I see, I would ask this question here https://ktor.io/support/

+
+
Ktor Framework
+ + + +
Support
+
+ Kotlin Server and Client Framework for microservices, HTTP APIs, and RESTful services +
+ + + + + + + + +
Original URL: https://ktor.io/support/
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sangeeta Mishra + (sangeeta@acceldata.io) +
+
2023-09-25 10:12:52
 +
+

*Thread Reply:* Thank you

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-09-26 04:13:20
 +
+

*Thread Reply:* @Sangeeta Mishra which openlineage client are you using: java or python?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sangeeta Mishra + (sangeeta@acceldata.io) +
+
2023-09-26 04:19:53
 +
+

*Thread Reply:* @Paweł Leszczyński I am using python client

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Suraj Gupta + (suraj.gupta@atlan.com) +
+
2023-09-25 13:36:25
 +
+

I'm using the Spark OpenLineage integration. In the outputStatistics output dataset facet we receive rowCount and size. +The Job performs a SQL insert into a MySQL table and I'm receiving the size as 0. +{ + "outputStatistics": + { + "_producer": "<https://github.com/OpenLineage/OpenLineage/tree/1.1.0/integration/spark>", + "_schemaURL": "<https://openlineage.io/spec/facets/1-0-0/OutputStatisticsOutputDatasetFacet.json#/$defs/OutputStatisticsOutputDatasetFacet>", + "rowCount": 1, + "size": 0 + } +} +I'm not sure what the size means here. Does this mean number of bytes inserted/updated? +Also, do we have any documentation for Spark specific Job and Run facets?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-09-27 09:56:00
 +
+

*Thread Reply:* I am not sure it's stated in the doc. Here's the list of spark facets schemas: https://github.com/OpenLineage/OpenLineage/tree/main/integration/spark/shared/facets/spark/v1

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Guntaka Jeevan Paul + (jeevan@acceldata.io) +
+
2023-09-26 00:51:30
 +
+

@here In Airflow Integration we send across a lineage Event for Dag start and complete, but that is not the case with spark integration…we don’t receive any event for the application start and complete in spark…is this expected behaviour or am i missing something?

+ + + +
+ ➕ Suraj Gupta +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-09-27 09:47:39
 +
+

*Thread Reply:* For spark we do send start and complete for each spark action being run (single operation that causes spark processing being run). However, it is difficult for us to know if we're dealing with the last action within spark job or a spark script.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-09-27 09:49:35
 +
+

*Thread Reply:* I think we need to look deeper into that as there is reoccuring need to capture such information

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-09-27 09:49:57
 +
+

*Thread Reply:* and spark listener event has methods like onApplicationStart and onApplicationEnd

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Guntaka Jeevan Paul + (jeevan@acceldata.io) +
+
2023-09-27 09:50:13
 +
+

*Thread Reply:* We are using the SparkListener, which has a function called OnApplicationStart which gets called whenever a spark application starts, so i was thinking why cant we send one at start and simlarly at end as well

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-09-27 09:50:33
 +
+

*Thread Reply:* additionally, we would like to have a concept of a parent run for a spark job which aggregates all actions run within a single spark job context

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Guntaka Jeevan Paul + (jeevan@acceldata.io) +
+
2023-09-27 09:51:11
 +
+

*Thread Reply:* yeah exactly. the way that it works with airflow integration

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-09-27 09:51:26
 +
+

*Thread Reply:* we do have an issue for that https://github.com/OpenLineage/OpenLineage/issues/2105

+
+ + + + +
#2105 [PROPOSAL] Spark jobs should have a parent Job describing the Run State of Spark pipeline.
+
+ <p><strong>Purpose:</strong> <br /> +Currently there is no way to group spark jobs running within the same SparkContext since there is no concept of parent Jobs in Spark. In airflow we support Parent Runs and this way we can group Tasks together within a single DAG.</p> + +<p><strong>Use cases:</strong></p> + +<p>• For data catalog based OL backends, a user might want to search for Spark ETL pipelines first and then dive deeper into a specific pipeline to view the Spark Jobs. This is similar to how a user can search for airflow DAG assets and then look into the Tasks for that DAG. +• Also this will help display operational metadata for Spark pipelines. For Airflow we have DAG run status (parent job status), but for Spark we don't have any such concept to display the completion state of a pipeline run. We can only track it at SparkJob level.</p> + +<p><strong>Proposed implementation</strong> <br /> +This should be similar to the airflow implementation.</p> +
+ +
+
Labels
+ proposal +
+ +
+
Comments
+ 2 +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-09-27 09:52:08
 +
+

*Thread Reply:* what you can is: come to our monthly Openlineage open meetings and raise that issue and convince the community about its importance

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Guntaka Jeevan Paul + (jeevan@acceldata.io) +
+
2023-09-27 09:53:32
 +
+

*Thread Reply:* yeah sure would love to do that…how can i join them, will that be posted here in this slack channel?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-09-27 09:54:08
 +
+

*Thread Reply:* Hi, you can see the schedule and RSVP here: https://openlineage.io/community

+
+
openlineage.io
+ + + +
Community | OpenLineage
+
+ Data lineage is the foundation for a new generation of powerful, context-aware data tools and best practices. OpenLineage enables consistent collection of lineage metadata, creating a deeper understanding of how data is produced and used. +
+ + + + + + +
Original URL: https://openlineage.io/community
+ + + +
+ + + +
+ 🙌 Paweł Leszczyński +
+ +
+ :gratitude_thank_you: Guntaka Jeevan Paul +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-09-27 11:19:16
 +
+

Meetup recap: Toronto Meetup @ Airflow Summit, September 18, 2023 +It was great to see so many members of our community at this event! I counted 32 total attendees, with all but a handful being first-timers. +Topics included: +• Presentation on the history, architecture and roadmap of the project by @Julien Le Dem and @Harel Shein +• Discussion of OpenLineage support in Marquez by @Willy Lulciuc +• Presentation by Ye Liu and Ivan Perepelitca from Metaphor, the social platform for data, about their integration +• Presentation by @Paweł Leszczyński about the Spark integration +• Presentation by @Maciej Obuchowski about the Apache Airflow Provider +Thanks to all the presenters and attendees with a shout out to @Harel Shein for the help with organizing and day-of logistics, @Jakub Dardziński for the help with set up/clean up, and @Sheeri Cabral (Collibra) for the crucial assist with the signup sheet. +This was our first meetup in Toronto, and we learned some valuable lessons about planning events in new cities — the first and foremost being to ask for a pic of the building! 🙂 But it seemed like folks were undeterred, and the space itself lived up to expectations. +For a recording and clips from the meetup, head over to our YouTube channel. +Upcoming events: +• October 5th in San Francisco: Marquez Meetup @ Astronomer (sign up https://www.meetup.com/meetup-group-bnfqymxe/events/295444209/?utmmedium=referral&utmcampaign=share-btnsavedeventssharemodal&utmsource=link|here) +• November: Warsaw meetup (details, date TBA) +• January: London meetup (details, date TBA) +Are you interested in hosting or co-hosting an OpenLineage or Marquez meetup? DM me!

+
+
metaphor.io
+ + + +
Metaphor - The Social Platform for Data
+
+ Making Data Actionable, At Scale - Designed for data teams building cloud-native, self-service data platforms for their business users. Explore our Data Governance, Data Lineage, Data Discovery, and Data Trust capabilities today. +
+ + + + + + + + +
Original URL: https://metaphor.io/
+ + + +
+
+
YouTube
+ + + +
OpenLineage Project
+
+ Meetings, talks and tutorials by the OpenLineage Project, an Open Standard for lineage metadata collection +
+ + + + + + + + +
Original URL: https://www.youtube.com/channel/UCRMLy4AaSw_ka-gNV9nl7VQ/
+ + + +
+
+
Meetup
+ + + +
Marquez Meetup @ Astronomer, Thu, Oct 5, 2023, 5:30 PM | Meetup
+
+ Join us on Thursday, October 5th, from 5:30_8:30 pm to learn about the Marquez project. Meet other members of the community, get tips on making the most of the latest impro +
+ + + + + + + + +
Original URL: https://www.meetup.com/meetup-group-bnfqymxe/events/295444209/?utm_medium=referral&amp;utm_campaign=share-btn_savedevents_share_modal&amp;utm_source=link
+ + + +
+ +
+
IMG_5456.jpg
+ + + + + + + + +
+
+
IMG_5452.jpg
+ + + + + + + + +
+
+
20230918_171849.jpg
+ + + + + + + + +
+
+
20230918_192522.jpg
+ + + + + + + + +
+
+
20230918_185612.jpg
+ + + + + + + + +
+ + +
+ 🙌 Mars Lan, Harel Shein, Paweł Leszczyński +
+ +
+ ❤️ Jakub Dardziński, Harel Shein, Rodrigo Maia, Paweł Leszczyński, Julien Le Dem, Willy Lulciuc +
+ +
+ 🚀 Jakub Dardziński, Kevin Languasco +
+ +
+ 😅 Harel Shein +
+ +
+ ✅ Sheeri Cabral (Collibra) +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-09-27 11:55:47
 +
+

*Thread Reply:* A few more pics:

+ +
+
IMG_5462.jpg
+ + + + + + + + +
+
+
20230918_172756.jpg
+ + + + + + + + +
+
+
20230918_172935.jpg
+ + + + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Damien Hawes + (damien.hawes@booking.com) +
+
2023-09-27 12:23:05
 +
+

Hi folks, am I correct in my observations that the Spark integration does not generate inputs and outputs for Kafka-to-Kafka pipelines?

+ +

EDIT: Removed the crazy wall of text. Relevant GitHub issue is here.

+
+ + + + +
#2137 [SPARK] The integration fails to produce inputs and outputs in a Kafka-to-Kafka scenario
+
+ <p><strong>Problem Statement</strong></p> + +<p>As the title suggests, when a Spark job is running in a streaming configuration that performs a Kafka-to-Kafka data flow, the integration fails to emit any events.</p> + +<p>Here's an example of an event that the connector emits:</p> + +<p>The connector emits the following output:</p> + +<pre><code>23/09/27 18:17:22 DEBUG OpenLineageRunEventBuilder: Visiting query plan Optional[== Parsed Logical Plan == +WriteToMicroBatchDataSource org.apache.spark.sql.kafka010.KafkaSourceProvider$KafkaTable@2e679177, c3df858a-d481-456f-9665-3ff0c2d6d19a, [kafka.bootstrap.servers=localhost:9092,localhost:9093,localhost:9094, topic=target, checkpointLocation=/Users/dhawes/Projects/spark-streaming-openlineage], Append, 0 ++- StreamingDataSourceV2Relation [key#7, value#8, topic#9, partition#10, offset#11L, timestamp#12, timestampType#13], org.apache.spark.sql.kafka010.KafkaSourceProvider$KafkaScan@2095a606, KafkaV2[Subscribe[source]], {"source":{"0":11}}, {"source":{"0":11}} + +== Analyzed Logical Plan == +WriteToMicroBatchDataSource org.apache.spark.sql.kafka010.KafkaSourceProvider$KafkaTable@2e679177, c3df858a-d481-456f-9665-3ff0c2d6d19a, [kafka.bootstrap.servers=localhost:9092,localhost:9093,localhost:9094, topic=target, checkpointLocation=/Users/dhawes/Projects/spark-streaming-openlineage], Append, 0 ++- StreamingDataSourceV2Relation [key#7, value#8, topic#9, partition#10, offset#11L, timestamp#12, timestampType#13], org.apache.spark.sql.kafka010.KafkaSourceProvider$KafkaScan@2095a606, KafkaV2[Subscribe[source]], {"source":{"0":11}}, {"source":{"0":11}} + +== Optimized Logical Plan == +WriteToDataSourceV2 org.apache.spark.sql.execution.streaming.sources.MicroBatchWrite@6b69d31b ++- StreamingDataSourceV2Relation [key#7, value#8, topic#9, partition#10, offset#11L, timestamp#12, timestampType#13], org.apache.spark.sql.kafka010.KafkaSourceProvider$KafkaScan@2095a606, KafkaV2[Subscribe[source]], {"source":{"0":11}}, {"source":{"0":11}} + +== Physical Plan == +WriteToDataSourceV2 org.apache.spark.sql.execution.streaming.sources.MicroBatchWrite@6b69d31b, org.apache.spark.sql.execution.datasources.v2.DataSourceV2Strategy$$Lambda$1945/0x000000c001c246c8@28b23a0a ++- **(1) Project [key#7, value#8, topic#9, partition#10, offset#11L, timestamp#12, timestampType#13] + +- MicroBatchScan[key#7, value#8, topic#9, partition#10, offset#11L, timestamp#12, timestampType#13] class org.apache.spark.sql.kafka010.KafkaSourceProvider$KafkaScan +] with output dataset builders [&lt;function1&gt;, &lt;function1&gt;, &lt;function1&gt;, &lt;function1&gt;, &lt;function1&gt;, &lt;function1&gt;, &lt;function1&gt;, &lt;function1&gt;] + +23/09/27 18:17:22 INFO ConsoleTransport: {"eventTime":"2023_09_27T16:17:22.263Z","producer":"&lt;https://github.com/OpenLineage/OpenLineage/tree/1.2.2/integration/spark&gt;","schemaURL":"&lt;https://openlineage.io/spec/2-0-2/OpenLineage.json#/$defs/RunEvent&gt;","eventType":"COMPLETE","run":{"runId":"3468112f_96c3_44b5_bd84_847c2d99db8f","facets":{"spark_version":{"_producer":"&lt;https://github.com/OpenLineage/OpenLineage/tree/1.2.2/integration/spark&gt;","_schemaURL":"&lt;https://openlineage.io/spec/2-0-2/OpenLineage.json#/$defs/RunFacet&gt;","spark-version":"3.3.0","openlineage_spark_version":"1.2.2"},"processing_engine":{"_producer":"&lt;https://github.com/OpenLineage/OpenLineage/tree/1.2.2/integration/spark&gt;","_schemaURL":"&lt;https://openlineage.io/spec/facets/1-1-0/ProcessingEngineRunFacet.json#/$defs/ProcessingEngineRunFacet&gt;","version":"3.3.0","name":"spark","openlineageAdapterVersion":"1.2.2"},"environment_properties":{"_producer":"&lt;https://github.com/OpenLineage/OpenLineage/tree/1.2.2/integration/spark&gt;","_schemaURL":"&lt;https://openlineage.io/spec/2-0-2/OpenLineage.json#/$defs/RunFacet&gt;","environment-properties":{}}}},"job":{"namespace":"default","name":"spark_streaming_example.write_to_data_source_v2","facets":{}},"inputs":[],"outputs":[]} +</code></pre> + +<p>As you can see, there are <em>zero</em> inputs and outputs.</p> + +<p><strong>Stuff to reproduce</strong> +<strong>Code</strong></p> + +<pre><code>package streaming; + +import org.apache.spark.sql.Dataset; +import org.apache.spark.sql.Row; +import org.apache.spark.sql.SparkSession; +import org.apache.spark.sql.streaming.StreamingQuery; +import org.apache.spark.sql.streaming.Trigger; + +import java.util.concurrent.TimeUnit; + +public class SparkStreamingExampleApplication { + public static void main(String[] args) throws Exception { + SparkSession spark = SparkSession + .builder() + .appName("spark-streaming-example") + .master("local") + .config("spark.ui.enabled", false) + .config("spark.jars.packages", "io.openlineage:openlineage_spark:1.2.2") + .config("spark.extraListeners", "io.openlineage.spark.agent.OpenLineageSparkListener") + .config("spark.openlineage.transport.type", "console") + .config("spark.openlineage.facets.disabled", "[spark_unknown;spark.logicalPlan]") + .getOrCreate(); + + Dataset&lt;Row&gt; df = spark.readStream() + .format("kafka") + .option("kafka.bootstrap.servers", "localhost:9092,localhost:9093,localhost:9094") + .option("subscribe", "source") + .load(); + + StreamingQuery kafkaWriteQuery = df.writeStream() + .outputMode("append") + .format("kafka") + .option("kafka.bootstrap.servers", "localhost:9092,localhost:9093,localhost:9094") + .option("topic", "target") + .option("checkpointLocation", "/Users/dhawes/Projects/spark-streaming-openlineage") + .trigger(Trigger.ProcessingTime(10, TimeUnit.SECONDS)) + .start(); + + kafkaWriteQuery.awaitTermination(); + } +} +</code></pre> + +<p><strong>build.gradle.kts</strong></p> + +<pre><code>plugins { + java + application +} + +java { + sourceCompatibility = JavaVersion.VERSION_1_8 + targetCompatibility = JavaVersion.VERSION_1_8 + + toolchain { + languageVersion.set(JavaLanguageVersion.of(8)) + } +} + +repositories { + mavenLocal() + mavenCentral() +} + +dependencies { + implementation("org.apache.spark:spark_core_2.12:3.3.0") + implementation("org.apache.spark:spark_sql_2.12:3.3.0") + implementation("org.apache.spark:spark_sql_kafka_0_10_2.12:3.3.0") + implementation("org.apache.spark:spark_streaming_2.12:3.3.0") + implementation("org.apache.spark:spark_streaming_kafka_0_10_2.12:3.3.0") + implementation("io.openlineage:openlineage_spark:1.2.2") +} + +application { + mainClass = "streaming.SparkStreamingExampleApplication" +} +</code></pre> + +<p><strong>docker-compose.yaml</strong></p> + +<p>``` +version: '3'</p> + +<p>services: + zookeeper: + image: confluentinc/cp-zookeeper:latest + container<em>name: 'spark-streaming-zookeeper' + environment: + ZOOKEEPER</em>CLIENT_PORT: 2181</p> + +<p>kafka1: + image: 'confluentinc/cp-kafka:latest' + container<em>name: 'spark-streaming-kafka-1' + depends</em>on: + - 'zookeeper' + environment: + KAFKA<em>BROKER</em>ID: 1 + KAFKA<em>ZOOKEEPER</em>CONNECT: zookeeper:2181 + KAFKA<em>ADVERTISED</em>LISTENERS: <INSIDE://kafka1:9092,OUTSIDE://localhost:9092> + KAFKA<em>LISTENER</em>SECURITY<em>PROTOCOL</em>MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT + KAFKA<em>INTER</em>BROKER<em>LISTENER</em>NAME: INSIDE + KAFKA_LISTENERS: <INSIDE://0.0.0.0:9092,OUTSIDE://0.0.0.0:9094> + ports: + - "9092:9094"</p> + +<p>kafka2: + image: 'confluentinc/cp-kafka:latest' + container<em>name: 'spark-streaming-kafka-2' + depends</em>on: + - 'zookeeper' + environment: + KAFKA<em>BROKER</em>ID: 2 + KAFKA<em>ZOOKEEPER</em>CONNECT: zookeeper:2181 + KAFKA<em>ADVERTISED</em>LISTENERS: <INSIDE://kafka2:9092,OUTSIDE://localhost:9093> + KAFKA<em>LISTENER</em>SECURITY<em>PROTOCOL</em>MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT + KAFKA<em>INTER</em>BROKER<em>LISTENER</em>NAME: INSIDE + KAFKA_LISTENERS: <INSIDE://0.0.0.0:9092,OUTSIDE://0.0.0.0:9095> + ports: + - "9093:9095"</p> + +<p>kafka3: + image: 'confluentinc/cp-kafka:latest' + container<em>name: 'spark-streaming-kafka-3' + depends</em>on: + - 'zookeeper' + environment: + KAFKA<em>BROKER</em>ID: 3 + KAFKA<em>ZOOKEEPER</em>CONNECT: zookeeper:2181 + KAFKA<em>ADVERTISED</em>LISTENERS: <INSIDE://kafka3:9092…</p> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+ 👀 Paweł Leszczyński +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-09-28 02:42:18
 +
+

*Thread Reply:* responded within the issue

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Erik Alfthan + (slack@alfthan.eu) +
+
2023-09-28 02:40:40
 +
+

Hello community +First time poster - bear with me :)

+ +

I am looking to make a minor PR on the airflow integration (fixing github #2130), and the code change is easy enough, but I fail to install the python environment. I have tried the simple ones +OpenLineage/integration/airflow &gt; pip install -e . + or +OpenLineage/integration/airflow &gt; pip install -r dev-requirements.txt +but they both fail on +ERROR: No matching distribution found for openlineage-sql==1.3.0

+ +

(which I think is an unreleased version in the git project)

+ +

How would I go about to install the requirements?

+ +

//Erik

+ +

PS. Sorry for posting this in general if there is a specific integration or contribution channel - I didnt find a better channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-09-28 03:04:48
 +
+

*Thread Reply:* Hi @Erik Alfthan, the channel is totally OK. I am not airflow integration expert, but what it looks to me, you're missing openlineage-sql library, which is a rust library used to extract lineage from sql queries. This is how we do that in circle ci: +https://app.circleci.com/pipelines/github/OpenLineage/OpenLineage/8080/workflows/aba53369-836c-48f5-a2dd-51bc0740a31c/jobs/140113

+ +

and subproject page with build instructions: https://github.com/OpenLineage/OpenLineage/tree/main/integration/sql

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Erik Alfthan + (slack@alfthan.eu) +
+
2023-09-28 03:07:23
 +
+

*Thread Reply:* Ok, so I go and "manually" build the internal dependency so that it becomes available in the pip cache?

+ +

I was hoping for something more automagical, but that should work

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-09-28 03:08:06
 +
+

*Thread Reply:* I think so. @Jakub Dardziński am I right?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-09-28 03:18:27
 +
+

*Thread Reply:* https://openlineage.io/docs/development/developing/python/setup +there’s a guide how to setup the dev environment

+ +

> Typically, you first need to build openlineage-sql locally (see README). After each release you have to repeat this step in order to bump local version of the package. +This might be somewhat exposed more in GitHub repository README as well

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Erik Alfthan + (slack@alfthan.eu) +
+
2023-09-28 03:27:20
 +
+

*Thread Reply:* It didnt find the wheel in the cache, but if I used the line in the sql/README.md +pip install openlineage-sql --no-index --find-links ../target/wheels --force-reinstall +It is installed and thus skipped/passed when pip later checks if it needs to be installed.

+ +

Now I have a second issue because it is expecting me to have mysqlclient-2.2.0 which seems to need a binary +Command 'pkg-config --exists mysqlclient' returned non-zero exit status 127 +and +Command 'pkg-config --exists mariadb' returned non-zero exit status 127 +I am on Ubuntu 22.04 in WSL2. Should I go to apt and grab me a mysql client?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-09-28 03:31:52
 +
+

*Thread Reply:* > It didnt find the wheel in the cache, but if I used the line in the sql/README.md +> pip install openlineage-sql --no-index --find-links ../target/wheels --force-reinstall +> It is installed and thus skipped/passed when pip later checks if it needs to be installed. +That’s actually expected. You should build new wheel locally and then install it.

+ +

> Now I have a second issue because it is expecting me to have mysqlclient-2.2.0 which seems to need a binary +> Command 'pkg-config --exists mysqlclient' returned non-zero exit status 127 +> and +> Command 'pkg-config --exists mariadb' returned non-zero exit status 127 +> I am on Ubuntu 22.04 in WSL2. Should I go to apt and grab me a mysql client? +We’ve left some system specific configuration, e.g. mysqlclient, to users as it’s a bit aside from OpenLineage and more of general development task.

+ +

probably +sudo apt-get install python3-dev default-libmysqlclient-dev build-essential +should work

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Erik Alfthan + (slack@alfthan.eu) +
+
2023-09-28 03:32:04
 +
+

*Thread Reply:* I just realized that I should probably skip setting up my wsl and just run the tests in the docker setup you prepared

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-09-28 03:35:46
 +
+

*Thread Reply:* You could do that as well but if you want to test your changes vs many Airflow versions that wouldn’t be possible I think (run them with tox btw)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Erik Alfthan + (slack@alfthan.eu) +
+
2023-09-28 04:54:39
 +
+

*Thread Reply:* This is starting to feel like a rabbit hole 😞

+ +

When I run tox, I get a lot of build errors +• client needs to be built +• sql needs to be built to a different target than its readme says +• a lot of builds fail on cython_sources

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-09-28 05:19:34
 +
+

*Thread Reply:* would you like to share some exact log lines? I’ve never seen such errors, they probably are system specific

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Erik Alfthan + (slack@alfthan.eu) +
+
2023-09-28 06:45:48
 +
+

*Thread Reply:* Getting requirements to build wheel did not run successfully. +│ exit code: 1 +╰─&gt; [62 lines of output] + /tmp/pip-build-env-q1pay0xo/overlay/lib/python3.10/site-packages/setuptools/config/setupcfg.py:293: _DeprecatedConfig: Deprecated config insetup.cfg` +!!`

+ +
        `****************************************************************************************************************************************************************`
+        `The license_file parameter is deprecated, use license_files instead.`
+
+        `By 2023-Oct-30, you need to update your project and remove deprecated calls`
+        `or your builds will no longer be supported.`
+
+        `See <https://setuptools.pypa.io/en/latest/userguide/declarative_config.html> for details.`
+        `****************************************************************************************************************************************************************`
+
+`!!`
+  `parsed = self.parsers.get(option_name, lambda x: x)(value)`
+`running egg_info`
+`writing lib3/PyYAML.egg-info/PKG-INFO`
+`writing dependency_links to lib3/PyYAML.egg-info/dependency_links.txt`
+`writing top-level names to lib3/PyYAML.egg-info/top_level.txt`
+`Traceback (most recent call last):`
+  `File "/home/obr_erikal/projects/OpenLineage/integration/airflow/.tox/py3-airflow-2.1.4/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in &lt;module&gt;`
+    `main()`
+  `File "/home/obr_erikal/projects/OpenLineage/integration/airflow/.tox/py3-airflow-2.1.4/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main`
+    `json_out['return_val'] = hook(****hook_input['kwargs'])`
+  `File "/home/obr_erikal/projects/OpenLineage/integration/airflow/.tox/py3-airflow-2.1.4/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 118, in get_requires_for_build_wheel`
+    `return hook(config_settings)`
+  `File "/tmp/pip-build-env-q1pay0xo/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 355, in get_requires_for_build_wheel`
+    `return self._get_build_requires(config_settings, requirements=['wheel'])`
+  `File "/tmp/pip-build-env-q1pay0xo/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 325, in _get_build_requires`
+    `self.run_setup()`
+  `File "/tmp/pip-build-env-q1pay0xo/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 341, in run_setup`
+    `exec(code, locals())`
+  `File "&lt;string&gt;", line 271, in &lt;module&gt;`
+  `File "/tmp/pip-build-env-q1pay0xo/overlay/lib/python3.10/site-packages/setuptools/__init__.py", line 103, in setup`
+    `return distutils.core.setup(****attrs)`
+  `File "/tmp/pip-build-env-q1pay0xo/overlay/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 185, in setup`
+    `return run_commands(dist)`
+  `File "/tmp/pip-build-env-q1pay0xo/overlay/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 201, in run_commands`
+    `dist.run_commands()`
+  `File "/tmp/pip-build-env-q1pay0xo/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands`
+    `self.run_command(cmd)`
+  `File "/tmp/pip-build-env-q1pay0xo/overlay/lib/python3.10/site-packages/setuptools/dist.py", line 989, in run_command`
+    `super().run_command(command)`
+  `File "/tmp/pip-build-env-q1pay0xo/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 988, in run_command`
+    `cmd_obj.run()`
+  `File "/tmp/pip-build-env-q1pay0xo/overlay/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 318, in run`
+    `self.find_sources()`
+  `File "/tmp/pip-build-env-q1pay0xo/overlay/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 326, in find_sources`
+    `mm.run()`
+  `File "/tmp/pip-build-env-q1pay0xo/overlay/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 548, in run`
+    `self.add_defaults()`
+  `File "/tmp/pip-build-env-q1pay0xo/overlay/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 586, in add_defaults`
+    `sdist.add_defaults(self)`
+  `File "/tmp/pip-build-env-q1pay0xo/overlay/lib/python3.10/site-packages/setuptools/command/sdist.py", line 113, in add_defaults`
+    `super().add_defaults()`
+  `File "/tmp/pip-build-env-q1pay0xo/overlay/lib/python3.10/site-packages/setuptools/_distutils/command/sdist.py", line 251, in add_defaults`
+    `self._add_defaults_ext()`
+  `File "/tmp/pip-build-env-q1pay0xo/overlay/lib/python3.10/site-packages/setuptools/_distutils/command/sdist.py", line 336, in _add_defaults_ext`
+    `self.filelist.extend(build_ext.get_source_files())`
+  `File "&lt;string&gt;", line 201, in get_source_files`
+  `File "/tmp/pip-build-env-q1pay0xo/overlay/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 107, in __getattr__`
+    `raise AttributeError(attr)`
+`AttributeError: cython_sources`
+`[end of output]`
+
+ +

note: This error originates from a subprocess, and is likely not a problem with pip. +py3-airflow-2.1.4: exit 1 (7.85 seconds) /home/obr_erikal/projects/OpenLineage/integration/airflow&gt; python -m pip install --find-links target/wheels/ --find-links ../sql/iface-py/target/wheels --use-deprecated=legacy-resolver --constraint=<https://raw.githubusercontent.com/apache/airflow/constraints-2.1.4/constraints-3.8.txt> apache-airflow==2.1.4 'mypy&gt;=0.9.6' pytest pytest-mock -r dev-requirements.txt pid=368621 +py3-airflow-2.1.4: FAIL ✖ in 7.92 seconds

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Erik Alfthan + (slack@alfthan.eu) +
+
2023-09-28 06:53:54
 +
+

*Thread Reply:* Then, for the actual error in my PR: Evidently you are not using isort, so what linter/fixer should I use for imports?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-09-28 06:58:15
 +
+

*Thread Reply:* for the error - I think there’s a mistake in the docs. Could you please run maturin build --out target/wheels as a temp solution?

+ + + +
+ 👀 Erik Alfthan +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-09-28 06:58:57
 +
+

*Thread Reply:* we’re using ruff , tox runs it as one of commands

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Erik Alfthan + (slack@alfthan.eu) +
+
2023-09-28 07:00:37
 +
+

*Thread Reply:* Not in the airflow folder? +OpenLineage/integration/airflow$ maturin build --out target/wheels +💥 maturin failed + Caused by: pyproject.toml at /home/obr_erikal/projects/OpenLineage/integration/airflow/pyproject.toml is invalid + Caused by: TOML parse error at line 1, column 1 + | +1 | [tool.ruff] + | ^ +missing fieldbuild-system``

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-09-28 07:02:32
 +
+

*Thread Reply:* I meant change here https://github.com/OpenLineage/OpenLineage/blob/main/integration/sql/README.md

+ +

so +cd iface-py +python -m pip install maturin +maturin build --out ../target/wheels +becomes +cd iface-py +python -m pip install maturin +maturin build --out target/wheels +tox runs +install_command = python -m pip install {opts} --find-links target/wheels/ \ + --find-links ../sql/iface-py/target/wheels +but it should be +install_command = python -m pip install {opts} --find-links target/wheels/ \ + --find-links ../sql/target/wheels +actually and I’m posting PR to fix that

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Erik Alfthan + (slack@alfthan.eu) +
+
2023-09-28 07:05:12
 +
+

*Thread Reply:* yes, that part I actually worked out myself, but the cython_sources error I fail to understand cause. I have python3-dev installed on WSL Ubuntu with python version 3.10.12 in a virtualenv. Anything in that that could cause issues?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-09-28 07:12:20
 +
+

*Thread Reply:* looks like it has something to do with latest release of Cython? +pip install "Cython&lt;3" maybe solves the issue?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Erik Alfthan + (slack@alfthan.eu) +
+
2023-09-28 07:15:06
 +
+

*Thread Reply:* I didnt have any cython before the install. Also no change. Could it be some update to setuptools itself? seems like the depreciation notice and the error is coming from inside setuptools

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Erik Alfthan + (slack@alfthan.eu) +
+
2023-09-28 07:16:59
 +
+

*Thread Reply:* (I.e. I tried the pip install "Cython&lt;3" command without any change in the output )

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Erik Alfthan + (slack@alfthan.eu) +
+
2023-09-28 07:20:30
 +
+

*Thread Reply:* Applying ruff lint on the converter.py file fixed the issue on the PR though so unless you have any feedback on the change itself, I will set it up on my own computer later instead (right now doing changes on behalf of a client on the clients computer)

+ +

If the issue persists on my own computer, I'll dig a bit further

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-09-28 07:21:03
 +
+

*Thread Reply:* It’s a bit hard for me to find the root cause as I cannot reproduce this locally and CI works fine as well

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Erik Alfthan + (slack@alfthan.eu) +
+
2023-09-28 07:22:41
 +
+

*Thread Reply:* Yeah, I am thinking that if I run into the same problem "at home", I might find it worthwhile to understand the issue. Right now, the client only wants the fix.

+ + + +
+ 👍 Jakub Dardziński +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Erik Alfthan + (slack@alfthan.eu) +
+
2023-09-28 07:25:10
 +
+

*Thread Reply:* Is there an official release cycle?

+ +

or more specific, given that the PRs are approved, how soon can they reach openlineage-dbt and apache-airflow-providers-openlineage ?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-09-28 07:28:58
 +
+

*Thread Reply:* we need to differentiate some things:

+ +
  1. OpenLineage repository: +a. dbt integration - this is the only place where it is maintained +b. Airflow integration - here we only keep backwards compatibility but generally speaking starting from Airflow 2.7+ we would like to do all the job in Airflow repo as OL Airflow provider
  2. Airflow repository - there’s only Airflow Openlineage provider compatible (and works best) with Airflow 2.7+
    3. +
+ +

we have control over releases (obviously) in OL repo - it’s monthly cycle so beginning next week that should happen. There’s also a possibility to ask for ad-hoc release in #general slack channel and with approvals of committers the new version is also released

+ +

For Airflow providers - the cycle is monthly as well

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-09-28 07:31:30
 +
+

*Thread Reply:* it’s a bit complex for this split but needed temporarily

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Erik Alfthan + (slack@alfthan.eu) +
+
2023-09-28 07:31:47
 +
+

*Thread Reply:* oh, I did the fix in the wrong place! The client is on airflow 2.7 and is using the provider. Is it syncing?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-09-28 07:32:28
 +
+

*Thread Reply:* it’s not, two separate places a~nd we haven’t even added the whole thing with converting old lineage objects to OL specific~

+ +

editing, that’s not true

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-09-28 07:34:40
 +
+

*Thread Reply:* the code’s here: +https://github.com/apache/airflow/blob/main/airflow/providers/openlineage/extractors/manager.py#L154

+
+ + + + +
<https://github.com/apache/airflow/blob/main/airflow/providers/openlineage/extractors/manager.py | manager.py>
+
+ <pre><code> def extract_inlets_and_outlets( +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/apache/airflow">apache/airflow</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-09-28 07:35:17
 +
+

*Thread Reply:* sorry I did not mention this earlier. we definitely need to add some guidance how to proceed with contributions to OL and Airflow OL provider

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Erik Alfthan + (slack@alfthan.eu) +
+
2023-09-28 07:36:10
 +
+

*Thread Reply:* anyway, the dbt fix is the blocking issue, so if that parts comes next week, there is no real urgency in getting the columns. It is a nice to have for our ingest parquet files.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-09-28 07:37:12
 +
+

*Thread Reply:* may I ask if you use some custom operator / python operator there?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Erik Alfthan + (slack@alfthan.eu) +
+
2023-09-28 07:37:33
 +
+

*Thread Reply:* yeah, taskflow with inlets/outlets

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Erik Alfthan + (slack@alfthan.eu) +
+
2023-09-28 07:38:38
 +
+

*Thread Reply:* so we extract from sources and use pyarrow to create parquet files in storage that an mssql-server can use as external tables

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-09-28 07:39:54
 +
+

*Thread Reply:* awesome 👍 +we have plans to integrate more with Python operator as well but not earlier than in Airflow 2.8

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Erik Alfthan + (slack@alfthan.eu) +
+
2023-09-28 07:43:41
 +
+

*Thread Reply:* I guess writing a generic extractor for the python operator is quite hard, but if you could support some inlet/outlet type for tabular fileformat / their python libraries like pyarrow or maybe even pandas and document it, I think a lot of people would understand how to use them

+ + + +
+ ➕ Harel Shein +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-09-28 16:16:24
 +
+

Are you located in the Brussels area or within commutable distance? Interested in attending a meetup between October 16-20? If so, please DM @Sheeri Cabral (Collibra) or myself. TIA

+ + + +
+ ❤️ Sheeri Cabral (Collibra) +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-10-02 11:58:32
 +
+

@channel +Hello all, I’d like to open a vote to release OpenLineage 1.3.0, including: +• support for Spark 3.5 in the Spark integration +• scheme preservation bug fix in the Spark integration +• find-links path in tox bug in the Airflow integration fix +• more graceful logging when no OL provider is installed in the Airflow integration +• columns as schema facet for airflow.lineage.Table addition +• SQLSERVER to supported dbt profile types addition +Three +1s from committers will authorize. Thanks in advance.

+ + + +
+ 🙌 Harel Shein, Paweł Leszczyński, Rodrigo Maia +
+ +
+ 👍 Jason Yip, Paweł Leszczyński +
+ +
+ ➕ Willy Lulciuc, Jakub Dardziński, Erik Alfthan, Julien Le Dem +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-10-02 17:00:08
 +
+

*Thread Reply:* Thanks all. The release is authorized and will be initiated within 2 business days.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason Yip + (jasonyip@gmail.com) +
+
2023-10-02 17:11:46
 +
+

*Thread Reply:* looking forward to that, I am seeing inconsistent results in Databricks for Spark 3.4+, sometimes there's no inputs / outputs, hope that is fixed?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harel Shein + (harel.shein@gmail.com) +
+
2023-10-03 09:59:24
 +
+

*Thread Reply:* @Jason Yip if it isn’t fixed for you, would love it if you could open up an issue that will allow us to reproduce and fix

+ + + +
+ 👍 Jason Yip +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason Yip + (jasonyip@gmail.com) +
+
2023-10-03 20:23:40
 +
+

*Thread Reply:* @Harel Shein the issue still exists -> Spark 3.4 and above, including 3.5, saveAsTable and create table won't have inputs and outputs in Databricks

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason Yip + (jasonyip@gmail.com) +
+
2023-10-03 20:30:15
 +
+

*Thread Reply:* https://github.com/OpenLineage/OpenLineage/issues/2124

+
+ + + + +
#2124 Same Delta Table not catching the location on write
+
+ <p><strong>What is the target system?</strong></p> + +<p>Spark / Databricks</p> + +<p><strong>What kind of integration is this?</strong></p> + +<p>☐ Produces OpenLineage metadata +☐ Consumes OpenLineage metadata +☐ Something else</p> + +<p><strong>How should this integration be implemented?</strong></p> + +<p>I am using OL 1.2.2, Azure Databricks Runtime 11.3 LTS. When creating a table writing into a ADLS location, OL won't be able to catch the location of the output. But when I read the same object it will be able to read the location as INPUT.</p> + +<p>Please note I have also tested Databricks Runtime 13.3 LTS, Spark 3.4.1 - it will give correct ADLS location in INPUT but the input will only show up once in a blue moon. Most of the time the inputs and outputs are blank.</p> + +<pre><code> "inputs": [], + "outputs": [] +</code></pre> + +<pre><code>CREATE OR REPLACE TABLE transactions_adj +USING DELTA LOCATION '&lt;wasbs://studio@clororetaildevadls.blob.core.windows.net/examples/data/csv/completejourney/silver/transactions_adj&gt;' +AS + SELECT + household_id, + basket_id, + week_no, + day, + transaction_time, + store_id, + product_id, + amount_list, + campaign_coupon_discount, + manuf_coupon_discount, + manuf_coupon_match_discount, + total_coupon_discount, + instore_discount, + amount_paid, + units + FROM ( + SELECT + household_id, + basket_id, + week_no, + day, + transaction_time, + store_id, + product_id, + COALESCE(sales_amount - discount_amount - coupon_discount - coupon_discount_match,0.0) as amount_list, + CASE + WHEN COALESCE(coupon_discount_match,0.0) = 0.0 THEN -1 ** COALESCE(coupon_discount,0.0) + ELSE 0.0 + END as campaign_coupon_discount, + CASE + WHEN COALESCE(coupon_discount_match,0.0) != 0.0 THEN -1 ** COALESCE(coupon_discount,0.0) + ELSE 0.0 + END as manuf_coupon_discount, + -1 ** COALESCE(coupon_discount_match,0.0) as manuf_coupon_match_discount, + -1 ** COALESCE(coupon_discount - coupon_discount_match,0.0) as total_coupon_discount, + COALESCE(-1 ** discount_amount,0.0) as instore_discount, + COALESCE(sales_amount,0.0) as `amount_paid,` + quantity as units + FROM transactions + ); +</code></pre> + +<p>Here's the COMPLETE event:</p> + +<pre><code> + "outputs":[ + { + "namespace":"dbfs", + "name":"/user/hive/warehouse/journey.db/transactions_adj", + "facets":{ + "dataSource":{ + "_producer":"&lt;https://github.com/OpenLineage/OpenLineage/tree/1.2.2/integration/spark&gt;", + "_schemaURL":"&lt;https://openlineage.io/spec/facets/1-0-0/DatasourceDatasetFacet.json#/$defs/DatasourceDatasetFacet&gt;", + "name":"dbfs", + "uri":"dbfs" + }, + +</code></pre> + +<p>Below logical plan shows the path:</p> + +<pre><code>== Analyzed Logical Plan == +num_affected_rows: bigint, num_inserted_rows: bigint +ReplaceTableAsSelect TableSpec(Map(),Some(DELTA),Map(),Some(&lt;wasbs://studio@clororetaildevadls.blob.core.windows.net/examples/data/csv/completejourney/silver/transactions_adj&gt;),None,None,false,Set()), true +:- ResolvedIdentifier com.databricks.sql.managedcatalog.UnityCatalogV2Proxy@6251a8df, default.transactions_adj ++- Project [household_id#184, basket_id#185L, week_no#193, day#186, transaction_time#192, store_id#190, product_id#187, amount_list#147, campaign_coupon_discount#148, manuf_coupon_discount#149, manuf_coupon_match_discount#150, total_coupon_discount#151, instore_discount#152, amount_paid#153, units#154] + +- SubqueryAlias __auto_generated_subquery_name + +- Project [household_id#184, basket_id#185L, week_no#193, day#186, transaction_time#192, store_id#190, product_id#187, coalesce(cast((((sales_amount#189 - discount_amount#191) - coupon_discount#194) - coupon_discount_match#195) as double), cast(0.0 as double)) AS amount_list#147, CASE WHEN (coalesce(cast(coupon_discount_match#195 as double), cast(0.0 as double)) = cast(0.0 as double)) THEN (cast(-1 as double) ** coalesce(cast(coupon_discount#194 as double), cast(0.0 as double))) ELSE cast(0.0 as double) END AS campaign_coupon_discount#148, CASE WHEN NOT (coalesce(cast(coupon_discount_match#195 as double), cast(0.0 as double)) = cast(0.0 as double)) THEN (cast(-1 as double) ** coalesce(cast(coupon_discount#194 as double), cast(0.0 as double))) ELSE cast(0.0 as double) END AS manuf_coupon_discount#149, (cast(-1 as double) ** coalesce(cast(coupon_discount_match#195 as double), cast(0.0 as double))) AS manuf_coupon_match_discount#150, (cast(-1 as double) ** coalesce(cast((coupon_discount#194 - coupon_discount_match#195) as double), cast(0.0 as double))) AS total_coupon_discount#151, coalesce(cast((cast(-1 as float) ** discount_amount#191) as double), cast(0.0 as double)) AS instore_discount#152, coalesce(cast(sales_amount#189 as double), cast(0.0 as double)) AS amount_paid#153, quantity#188 AS units#154] + +- SubqueryAlias spark_catalog.default.transactions + +- Relation spark_catalog.default.transactions[household_id#184,basket_id#185L,day#186,product_id#187,quantity#188,sales_amount#189,store_id#190,discount_amount#191,transaction_time#192,week_no#193,coupon_discount#194,coupon_discount_match#195] parquet +</code></pre> + +<p><strong>Where should this integration be implemented?</strong></p> + +<p>☐ In the target system +☐ In the OpenLineage repo +☐ Somewhere else</p> + +<p><strong>Do you plan to make this contribution yourself?</strong></p> + +<p>☐ I am interested in doing this work</p> +
+ +
+
Labels
+ integration/spark, integration/databricks +
+ +
+
Comments
+ 1 +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason Yip + (jasonyip@gmail.com) +
+
2023-10-03 20:30:21
 +
+

*Thread Reply:* and of course this issue still exists

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harel Shein + (harel.shein@gmail.com) +
+
2023-10-03 21:45:09
 +
+

*Thread Reply:* thanks for posting, we’ll continue looking into this.. if you find any clues that might help, please let us know.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason Yip + (jasonyip@gmail.com) +
+
2023-10-03 21:46:27
 +
+

*Thread Reply:* is there any instructions on how to hook up a debugger to OL?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harel Shein + (harel.shein@gmail.com) +
+
2023-10-04 09:04:16
 +
+

*Thread Reply:* @Paweł Leszczyński has been working on adding a debug facet, but more suggestions are more than welcome!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harel Shein + (harel.shein@gmail.com) +
+
2023-10-04 09:05:58
 +
+

*Thread Reply:* https://github.com/OpenLineage/OpenLineage/pull/2147

+
+ + + + +
#2147 [SPARK] add debug facet to help resolving Spark integration issues
+
+ <p><strong>Problem</strong></p> + +<p>Debugging openlineage-spark problems is tedious job. We would like to have debug facet that will collect automatically meaningful information when enabled.</p> + +<p>Closes: <a href="https://github.com/OpenLineage/OpenLineage/issues/2135">#2135</a></p> + +<p><strong>Solution</strong></p> + +<p>• Create debug facet (more details in <a href="https://github.com/OpenLineage/OpenLineage/issues/2135">#2135</a> ), +• Facet is disabled by default, +• Allow enabling it thourgh SparkConf.</p> + +<blockquote> + <p><strong>Note:</strong> All schema changes require discussion. Please <a href="https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue">link the issue</a> for context.</p> +</blockquote> + +<p>☐ Your change modifies the <a href="https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.json">core</a> OpenLineage model +☐ Your change modifies one or more OpenLineage <a href="https://github.com/OpenLineage/OpenLineage/tree/main/spec/facets">facets</a></p> + +<p>If you're contributing a new integration, please specify the scope of the integration and how/where it has been tested (e.g., Apache Spark integration supports <code>S3</code> and <code>GCS</code> filesystem operations, tested with AWS EMR).</p> + +<p><strong>One-line summary:</strong> +<strong>Checklist</strong></p> + +<p>☑︎ You've <a href="https://github.com/OpenLineage/OpenLineage/blob/main/why-the-dco.md">signed-off</a> your work +☑︎ Your pull request title follows our <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CONTRIBUTING.md#creating-pull-requests">guidelines</a> +☑︎ Your changes are accompanied by tests (<em>if relevant</em>) +☑︎ Your change contains a <a href="https://kurtisnusbaum.medium.com/stacked-diffs-keeping-phabricator-diffs-small-d9964f4dcfa6">small diff</a> and is self-contained +☐ You've updated any relevant documentation (<em>if relevant</em>) +☐ Your comment includes a one-liner for the changelog about the specific purpose of the change (<em>if necessary</em>) +☐ You've versioned the core OpenLineage model or facets according to <a href="https://docs.snowplowanalytics.com/docs/pipeline-components-and-applications/iglu/common-architecture/schemaver">SchemaVer</a> (<em>if relevant</em>) +☐ You've added a <a href="https://github.com/OpenLineage/OpenLineage/tree/main/.github/header_templates.md">header</a> to source files (<em>if relevant</em>)</p> + +<hr /> + +<p>SPDX-License-Identifier: Apache-2.0 <br /> +Copyright 2018-2023 contributors to the OpenLineage project</p> +
+ +
+
Labels
+ documentation, integration/spark +
+ +
+
Assignees
+ <a href="https://github.com/pawel-big-lebowski">@pawel-big-lebowski</a> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+ 👀 Paweł Leszczyński +
+ +
+ 👍 Jason Yip +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason Yip + (jasonyip@gmail.com) +
+
2023-10-05 03:20:11
 +
+

*Thread Reply:* @Paweł Leszczyński do you have a build for the PR? Appreciated!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harel Shein + (harel.shein@gmail.com) +
+
2023-10-05 15:05:08
 +
+

*Thread Reply:* we’ll ask for a release once it’s reviewed and merged

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-10-02 12:28:28
 +
+

@channel +The September issue of OpenLineage News is here! This issue covers the big news about OpenLineage coming out of Airflow Summit, progress on the Airflow Provider, highlights from our meetup in Toronto, and much more. +To get the newsletter directly in your inbox each month, sign up here.

+
+
apache.us14.list-manage.com
+ + + +
OpenLineage Project
+
+ OpenLineage Project Email Forms +
+ + + + + + +
Original URL: http://bit.ly/OL_news
+ + + +
+ + + +
+ 🦆 Harel Shein, Paweł Leszczyński +
+ +
+ 🔥 Willy Lulciuc, Jakub Dardziński, Paweł Leszczyński +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Damien Hawes + (damien.hawes@booking.com) +
+
2023-10-03 03:44:36
 +
+

Hi folks - I'm wondering if its just me, but does io.openlineage:openlineage_sql_java:1.2.2 ship with the arm64.dylib binary? When i try and run code that uses the Java package on an Apple M1, the binary isn't found, The workaround is to checkout 1.2.2 and then build and publish it locally.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-10-03 09:01:38
 +
+

*Thread Reply:* Not sure if I follow your question. Whenever OL is released, there is a script new-version.sh - https://github.com/OpenLineage/OpenLineage/blob/main/new-version.sh being run and modify the codebase.

+ +

So, If you pull the code, it contains OL version that has not been released yet and in case of dependencies, one need to build them on their own.

+ +

For example, here https://github.com/OpenLineage/OpenLineage/tree/main/integration/spark#preparation Preparation section describes how to build openlineage-java and openlineage-sql in order to build openlineage-spark.

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/new-version.sh | new-version.sh>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Damien Hawes + (damien.hawes@booking.com) +
+
2023-10-04 05:27:26
 +
+

*Thread Reply:* Hmm. Let's elaborate my use case a bit.

+ +

We run Apache Hive on-premise. Hive provides query execution hooks for pre-query, post-query, and I think failed query.

+ +

Any way, as part of the hook, you're given the query string.

+ +

So I, naturally, tried to pass the query string into OpenLineageSql.parse(Collections.singletonList(hookContext.getQueryPlan().getQueryStr()), "hive") in order to test this out.

+ +

I was using openlineage-sql-java:1.2.2 at that time, and no matter what query string I gave it, nothing was returned.

+ +

I then stepped through the code and noticed that it was looking for the arm64 lib, and I noticed that that package (downloaded from maven central) lacked that particular native binary.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Damien Hawes + (damien.hawes@booking.com) +
+
2023-10-04 05:27:36
 +
+

*Thread Reply:* I hope that helps.

+ + + +
+ 👍 Paweł Leszczyński +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-10-04 09:03:02
 +
+

*Thread Reply:* I get in now. In Circle CI we do have 3 build steps: +- build-integration-sql-x86 + - build-integration-sql-arm + - build-integration-sql-macos +but no mac m1. I think at that time circle CI did not have a proper resource class in free plan. Additionally, @Maciej Obuchowski would prefer to migrate this to github actions as he claims this can be achieved there in a cleaner way (https://github.com/OpenLineage/OpenLineage/issues/1624).

+ +

Feel free to create an issue for this. Others would be able to upvote it in case they have similar experience.

+
+ + + + +
#1624 CI: PoC building openlineage-sql on GitHub Actions with actions-rs
+
+ <p>Building sql parser currently is the most convoluted CI process due to need to construct different binaries in multiple dimensions; both for Java and Python, and for multiple architectures; like Linux x86, Linux ARM, MacOS x86 etc. The jobs also differ in different context: release workflow has different jobs than build one, which in turn does not build all of the architectures.</p> + +<p>To simplify that, we should try using GitHub Actions with <a href="https://actions-rs.github.io/">https://actions-rs.github.io/</a> that should solve the problems we've currently had to replicate manually.</p> + +<p>End result of that task should be having various SQL artifacts produced by GitHub actions and available by GH Actions artifacts API:</p> +
+ +
+
Assignees
+ <a href="https://github.com/mobuchowski">@mobuchowski</a> +
+ +
+
Labels
+ ci, integration/sql +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-10-23 11:56:12
 +
+

*Thread Reply:* It doesn't have the free resource class still 😞 +We're blocked on that unfortunately. Other solution would be to migrate to GH actions, where most of our solution could be replaced by something like that https://github.com/PyO3/maturin-action

+
+ + + + +
PyO3/maturin-action
+
+ GitHub Action to install and run a custom maturin command with built-in support for cross compilation +
+ +
+
Stars
+ 98 +
+ +
+
Language
+ TypeScript +
+ + + + + + + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-10-03 10:56:03
 +
+

@channel +We released OpenLineage 1.3.1! +Added: +• Airflow: add some basic stats to the Airflow integration #1845 @harels +• Airflow: add columns as schema facet for airflow.lineage.Table (if defined) #2138 @erikalfthan +• DBT: add SQLSERVER to supported dbt profile types #2136 @erikalfthan +• Spark: support for latest 3.5 #2118 @pawel-big-lebowski +Fixed: +• Airflow: fix find-links path in tox #2139 @JDarDagran +• Airflow: add more graceful logging when no OpenLineage provider installed #2141 @JDarDagran +• Spark: fix bug in PathUtils’ prepareDatasetIdentifierFromDefaultTablePath (CatalogTable) to correctly preserve scheme from CatalogTable’s location #2142 @d-m-h +Thanks to all the contributors, including new contributor @Erik Alfthan! +Release: https://github.com/OpenLineage/OpenLineage/releases/tag/1.3.1 +Changelog: https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md +Commit history: https://github.com/OpenLineage/OpenLineage/compare/1.2.2...1.3.1 +Maven: https://oss.sonatype.org/#nexus-search;quick~openlineage +PyPI: https://pypi.org/project/openlineage-python/

+ + + +
+ 👍 Jason Yip, Peter Hicks, Peter Huang, Mars Lan +
+ +
+ 🎉 Sheeri Cabral (Collibra) +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mars Lan + (mars@metaphor.io) +
+
2023-10-04 07:42:59
 +
+

*Thread Reply:* Any chance we can do a 1.3.2 soonish to include https://github.com/OpenLineage/OpenLineage/pull/2151 instead of waiting for the next monthly release?

+
+ + + + +
#2151 Allow setting client's endpoint via environment variable
+
+ <p><strong>Problem</strong></p> + +<p>Currently, it's not possible to set the OpenLineage endpoint (hard-coded to <code>/api/v1/lineage</code>) using an environment variable when running the Airflow integration.</p> + +<p><strong>Solution</strong></p> + +<p>Given that it's not possible to create the client manually in Airflow, especially now that OpenLineage has become an official Airflow provider, this change seems like the only feasible solution.</p> + +<p>☐ Your change modifies the <a href="https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.json">core</a> OpenLineage model +☐ Your change modifies one or more OpenLineage <a href="https://github.com/OpenLineage/OpenLineage/tree/main/spec/facets">facets</a></p> + +<p><strong>One-line summary:</strong></p> + +<p>Allow setting client's endpoint via environment variable.</p> + +<p><strong>Checklist</strong></p> + +<p>☑︎ You've <a href="https://github.com/OpenLineage/OpenLineage/blob/main/why-the-dco.md">signed-off</a> your work +☑︎ Your pull request title follows our <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CONTRIBUTING.md#creating-pull-requests">guidelines</a> +☑︎ Your changes are accompanied by tests (<em>if relevant</em>) +☑︎ Your change contains a <a href="https://kurtisnusbaum.medium.com/stacked-diffs-keeping-phabricator-diffs-small-d9964f4dcfa6">small diff</a> and is self-contained +☑︎ You've updated any relevant documentation (<em>if relevant</em>) +☑︎ Your comment includes a one-liner for the changelog about the specific purpose of the change (<em>if necessary</em>) +☐ You've versioned the core OpenLineage model or facets according to <a href="https://docs.snowplowanalytics.com/docs/pipeline-components-and-applications/iglu/common-architecture/schemaver">SchemaVer</a> (<em>if relevant</em>) +☐ You've added a <a href="https://github.com/OpenLineage/OpenLineage/tree/main/.github/header_templates.md">header</a> to source files (<em>if relevant</em>)</p> + +<hr /> + +<p>SPDX-License-Identifier: Apache-2.0 <br /> +Copyright 2018-2023 contributors to the OpenLineage project</p> +
+ +
+
Labels
+ documentation, client/python +
+ +
+
Comments
+ 4 +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Matthew Paras + (matthewparas2020@u.northwestern.edu) +
+
2023-10-03 12:34:57
 +
+

Hey everyone - does anyone have a good mechanism for alerting on issues with open lineage? For example, maybe alerting when an event times out - perhaps to prometheus or some other kind of generic endpoint? Not sure the best approach here (if the meta inf extension would be able to achieve it)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-10-04 03:01:02
 +
+

*Thread Reply:* That's a great usecase for OpenLineage. Unfortunately, we don't have any doc or recomendation on that.

+ +

I would try using FluentD proxy we have (https://github.com/OpenLineage/OpenLineage/tree/main/proxy/fluentd) to copy event stream (alerting is just one of usecases for lineage events) and write fluentd plugin to send it asynchronously further to alerting service like PagerDuty.

+ +

It looks cool to me but I never had enough time to test this approach.

+ + + +
+ 👍 Matthew Paras +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-10-05 14:44:14
 +
+

@channel +This month’s TSC meeting is next Thursday the 12th at 10am PT. On the tentative agenda: +• announcements +• recent releases +• Airflow Summit recap +• tutorial: migrating to the Airflow Provider +• discussion topic: observability for OpenLineage/Marquez +• open discussion +• more (TBA) +More info and the meeting link can be found on the website. All are welcome! Do you have a discussion topic, use case or integration you’d like to demo? DM me to be added to the agenda.

+
+
openlineage.io
+ + + +
TSC Meetings | OpenLineage
+
+ The OpenLineage Technical Steering Committee meets monthly, and is open to all. +
+ + + + + + +
Original URL: https://openlineage.io/meetings/
+ + + +
+ + + +
+ 👀 Sheeri Cabral (Collibra), Julian LaNeve, Peter Hicks +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2023-10-05 20:40:40
 +
+

The Marquez meetup in San Francisco is happening right now! +https://www.meetup.com/meetup-group-bnfqymxe/events/295444209/?utmmedium=referral&utmcampaign=share-btnsavedeventssharemodal&utmsource=link|https://www.meetup.com/meetup-group-bnfqymxe/events/295444209/?utmmedium=referral&utmcampaign=share-btnsavedeventssharemodal&utmsource=link

+
+
Meetup
+ + + +
Marquez Meetup @ Astronomer, Thu, Oct 5, 2023, 5:30 PM | Meetup
+
+ Join us on Thursday, October 5th, from 5:30_8:30 pm to learn about the Marquez project. Meet other members of the community, get tips on making the most of the latest impro +
+ + + + + + + + +
Original URL: https://www.meetup.com/meetup-group-bnfqymxe/events/295444209/?utm_medium=referral&amp;utm_campaign=share-btn_savedevents_share_modal&amp;utm_source=link
+ + + +
+ + + +
+ 🎉 Paweł Leszczyński, Rodrigo Maia +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mars Lan + (mars@metaphor.io) +
+
2023-10-06 07:19:01
 +
+

@Michael Robinson can we cut a new release to include this change? +• https://github.com/OpenLineage/OpenLineage/pull/2151

+
+ + + + +
#2151 Allow setting client's endpoint via environment variable
+
+ <p><strong>Problem</strong></p> + +<p>Currently, it's not possible to set the OpenLineage endpoint (hard-coded to <code>/api/v1/lineage</code>) using an environment variable when running the Airflow integration.</p> + +<p><strong>Solution</strong></p> + +<p>Given that it's not possible to create the client manually in Airflow, especially now that OpenLineage has become an official Airflow provider, this change seems like the only feasible solution.</p> + +<p>☐ Your change modifies the <a href="https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.json">core</a> OpenLineage model +☐ Your change modifies one or more OpenLineage <a href="https://github.com/OpenLineage/OpenLineage/tree/main/spec/facets">facets</a></p> + +<p><strong>One-line summary:</strong></p> + +<p>Allow setting client's endpoint via environment variable.</p> + +<p><strong>Checklist</strong></p> + +<p>☑︎ You've <a href="https://github.com/OpenLineage/OpenLineage/blob/main/why-the-dco.md">signed-off</a> your work +☑︎ Your pull request title follows our <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CONTRIBUTING.md#creating-pull-requests">guidelines</a> +☑︎ Your changes are accompanied by tests (<em>if relevant</em>) +☑︎ Your change contains a <a href="https://kurtisnusbaum.medium.com/stacked-diffs-keeping-phabricator-diffs-small-d9964f4dcfa6">small diff</a> and is self-contained +☑︎ You've updated any relevant documentation (<em>if relevant</em>) +☑︎ Your comment includes a one-liner for the changelog about the specific purpose of the change (<em>if necessary</em>) +☐ You've versioned the core OpenLineage model or facets according to <a href="https://docs.snowplowanalytics.com/docs/pipeline-components-and-applications/iglu/common-architecture/schemaver">SchemaVer</a> (<em>if relevant</em>) +☐ You've added a <a href="https://github.com/OpenLineage/OpenLineage/tree/main/.github/header_templates.md">header</a> to source files (<em>if relevant</em>)</p> + +<hr /> + +<p>SPDX-License-Identifier: Apache-2.0 <br /> +Copyright 2018-2023 contributors to the OpenLineage project</p> +
+ +
+
Labels
+ documentation, client/python +
+ +
+
Comments
+ 6 +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+ ➕ Harel Shein, Jakub Dardziński, Julien Le Dem, Michael Robinson, Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-10-06 19:16:02
 +
+

*Thread Reply:* Thanks for requesting a release, @Mars Lan. It has been approved and will be initiated within 2 business days of next Monday.

+ + + +
+ 🙏 Mars Lan +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Guntaka Jeevan Paul + (jeevan@acceldata.io) +
+
2023-10-08 23:59:36
 +
+

@here I am trying out the openlineage integration of spark on databricks. There is no event getting emitted from Openlineage, I see logs saying OpenLineage Event Skipped. I am attaching the Notebook that i am trying to run and the cluster logs. Kindly can someone help me on this

+ +
+
Screenshot 2023-10-09 at 9.28.18 AM.png
+ + + + + + + + +
+
+
cluster-logs
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason Yip + (jasonyip@gmail.com) +
+
2023-10-09 00:02:10
 +
+

*Thread Reply:* from my experience, it will only work on Spark 3.3.x or below, aka Runtime 12.2 or below. Anything above the events will show up once in a blue moon

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Guntaka Jeevan Paul + (jeevan@acceldata.io) +
+
2023-10-09 00:04:38
 +
+

*Thread Reply:* ohh, thanks for the information @Jason Yip, I am trying out with 13.3 Databricks Version and Spark 3.4.1, will try using a below version as you suggested. Any issue tracking this bug @Jason Yip

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason Yip + (jasonyip@gmail.com) +
+
2023-10-09 00:06:06
 +
+

*Thread Reply:* https://github.com/OpenLineage/OpenLineage/issues/2124

+
+ + + + +
#2124 Same Delta Table not catching the location on write
+
+ <p><strong>What is the target system?</strong></p> + +<p>Spark / Databricks</p> + +<p><strong>What kind of integration is this?</strong></p> + +<p>☐ Produces OpenLineage metadata +☐ Consumes OpenLineage metadata +☐ Something else</p> + +<p><strong>How should this integration be implemented?</strong></p> + +<p>I am using OL 1.2.2, Azure Databricks Runtime 11.3 LTS. When creating a table writing into a ADLS location, OL won't be able to catch the location of the output. But when I read the same object it will be able to read the location as INPUT.</p> + +<p>Please note I have also tested Databricks Runtime 13.3 LTS, Spark 3.4.1 - it will give correct ADLS location in INPUT but the input will only show up once in a blue moon. Most of the time the inputs and outputs are blank.</p> + +<pre><code> "inputs": [], + "outputs": [] +</code></pre> + +<pre><code>CREATE OR REPLACE TABLE transactions_adj +USING DELTA LOCATION '&lt;wasbs://studio@clororetaildevadls.blob.core.windows.net/examples/data/csv/completejourney/silver/transactions_adj&gt;' +AS + SELECT + household_id, + basket_id, + week_no, + day, + transaction_time, + store_id, + product_id, + amount_list, + campaign_coupon_discount, + manuf_coupon_discount, + manuf_coupon_match_discount, + total_coupon_discount, + instore_discount, + amount_paid, + units + FROM ( + SELECT + household_id, + basket_id, + week_no, + day, + transaction_time, + store_id, + product_id, + COALESCE(sales_amount - discount_amount - coupon_discount - coupon_discount_match,0.0) as amount_list, + CASE + WHEN COALESCE(coupon_discount_match,0.0) = 0.0 THEN -1 ** COALESCE(coupon_discount,0.0) + ELSE 0.0 + END as campaign_coupon_discount, + CASE + WHEN COALESCE(coupon_discount_match,0.0) != 0.0 THEN -1 ** COALESCE(coupon_discount,0.0) + ELSE 0.0 + END as manuf_coupon_discount, + -1 ** COALESCE(coupon_discount_match,0.0) as manuf_coupon_match_discount, + -1 ** COALESCE(coupon_discount - coupon_discount_match,0.0) as total_coupon_discount, + COALESCE(-1 ** discount_amount,0.0) as instore_discount, + COALESCE(sales_amount,0.0) as `amount_paid,` + quantity as units + FROM transactions + ); +</code></pre> + +<p>Here's the COMPLETE event:</p> + +<pre><code> + "outputs":[ + { + "namespace":"dbfs", + "name":"/user/hive/warehouse/journey.db/transactions_adj", + "facets":{ + "dataSource":{ + "_producer":"&lt;https://github.com/OpenLineage/OpenLineage/tree/1.2.2/integration/spark&gt;", + "_schemaURL":"&lt;https://openlineage.io/spec/facets/1-0-0/DatasourceDatasetFacet.json#/$defs/DatasourceDatasetFacet&gt;", + "name":"dbfs", + "uri":"dbfs" + }, + +</code></pre> + +<p>Below logical plan shows the path:</p> + +<pre><code>== Analyzed Logical Plan == +num_affected_rows: bigint, num_inserted_rows: bigint +ReplaceTableAsSelect TableSpec(Map(),Some(DELTA),Map(),Some(&lt;wasbs://studio@clororetaildevadls.blob.core.windows.net/examples/data/csv/completejourney/silver/transactions_adj&gt;),None,None,false,Set()), true +:- ResolvedIdentifier com.databricks.sql.managedcatalog.UnityCatalogV2Proxy@6251a8df, default.transactions_adj ++- Project [household_id#184, basket_id#185L, week_no#193, day#186, transaction_time#192, store_id#190, product_id#187, amount_list#147, campaign_coupon_discount#148, manuf_coupon_discount#149, manuf_coupon_match_discount#150, total_coupon_discount#151, instore_discount#152, amount_paid#153, units#154] + +- SubqueryAlias __auto_generated_subquery_name + +- Project [household_id#184, basket_id#185L, week_no#193, day#186, transaction_time#192, store_id#190, product_id#187, coalesce(cast((((sales_amount#189 - discount_amount#191) - coupon_discount#194) - coupon_discount_match#195) as double), cast(0.0 as double)) AS amount_list#147, CASE WHEN (coalesce(cast(coupon_discount_match#195 as double), cast(0.0 as double)) = cast(0.0 as double)) THEN (cast(-1 as double) ** coalesce(cast(coupon_discount#194 as double), cast(0.0 as double))) ELSE cast(0.0 as double) END AS campaign_coupon_discount#148, CASE WHEN NOT (coalesce(cast(coupon_discount_match#195 as double), cast(0.0 as double)) = cast(0.0 as double)) THEN (cast(-1 as double) ** coalesce(cast(coupon_discount#194 as double), cast(0.0 as double))) ELSE cast(0.0 as double) END AS manuf_coupon_discount#149, (cast(-1 as double) ** coalesce(cast(coupon_discount_match#195 as double), cast(0.0 as double))) AS manuf_coupon_match_discount#150, (cast(-1 as double) ** coalesce(cast((coupon_discount#194 - coupon_discount_match#195) as double), cast(0.0 as double))) AS total_coupon_discount#151, coalesce(cast((cast(-1 as float) ** discount_amount#191) as double), cast(0.0 as double)) AS instore_discount#152, coalesce(cast(sales_amount#189 as double), cast(0.0 as double)) AS amount_paid#153, quantity#188 AS units#154] + +- SubqueryAlias spark_catalog.default.transactions + +- Relation spark_catalog.default.transactions[household_id#184,basket_id#185L,day#186,product_id#187,quantity#188,sales_amount#189,store_id#190,discount_amount#191,transaction_time#192,week_no#193,coupon_discount#194,coupon_discount_match#195] parquet +</code></pre> + +<p><strong>Where should this integration be implemented?</strong></p> + +<p>☐ In the target system +☐ In the OpenLineage repo +☐ Somewhere else</p> + +<p><strong>Do you plan to make this contribution yourself?</strong></p> + +<p>☐ I am interested in doing this work</p> +
+ +
+
Labels
+ integration/spark, integration/databricks +
+ +
+
Comments
+ 2 +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Guntaka Jeevan Paul + (jeevan@acceldata.io) +
+
2023-10-09 00:11:54
 +
+

*Thread Reply:* tried with databricks 12.2 --> spark 3.3.2, still the same behaviour no event getting emitted

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason Yip + (jasonyip@gmail.com) +
+
2023-10-09 00:12:35
 +
+

*Thread Reply:* you can do 11.3, its the most stable one I know

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Guntaka Jeevan Paul + (jeevan@acceldata.io) +
+
2023-10-09 00:12:46
 +
+

*Thread Reply:* sure, let me try that out

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Guntaka Jeevan Paul + (jeevan@acceldata.io) +
+
2023-10-09 00:31:51
 +
+

*Thread Reply:* still the same problem…the jar that i am using is the latest openlineage-spark-1.3.1.jar, do you think that can be the problem

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Guntaka Jeevan Paul + (jeevan@acceldata.io) +
+
2023-10-09 00:43:59
 +
+

*Thread Reply:* tried with openlineage-spark-1.2.2.jar, still the same issue, seems like they are skipping some events

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason Yip + (jasonyip@gmail.com) +
+
2023-10-09 01:47:20
 +
+

*Thread Reply:* Probably not all events will be captured, I have only tested create tables and jobs

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-10-09 04:31:12
 +
+

*Thread Reply:* Hi @Guntaka Jeevan Paul, how did you configure openlineage and what is your job doing?

+ +

We do have a bunch of integration tests on Databricks platform available here and they're passing on databricks runtime 13.0.x-scala2.12.

+ +

Could you also try running code same as our test does (this one)? If you run it and see OL events, this will make us sure your config is OK and we can continue further debug.

+ +

Looking at your spark script: could you save your dataset and see if you still don't see any events?

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/1.3.1/integration/spark/app/src/test/java/io/openlineage/spark/agent/DatabricksIntegrationTest.java | DatabricksIntegrationTest.java>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/1.3.1/integration/spark/app/src/test/resources/databricks_notebooks/ctas.py | ctas.py>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Guntaka Jeevan Paul + (jeevan@acceldata.io) +
+
2023-10-09 05:06:41
 +
+

*Thread Reply:* babynames = spark.read.format("csv").option("header", "true").option("inferSchema", "true").load("dbfs:/FileStore/babynames.csv") +babynames.createOrReplaceTempView("babynames_table") +years = spark.sql("select distinct(Year) from babynames_table").rdd.map(lambda row : row[0]).collect() +years.sort() +dbutils.widgets.dropdown("year", "2014", [str(x) for x in years]) +display(babynames.filter(babynames.Year == dbutils.widgets.get("year")))

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Guntaka Jeevan Paul + (jeevan@acceldata.io) +
+
2023-10-09 05:08:09
 +
+

*Thread Reply:* this is the script that i am running @Paweł Leszczyński…kindly let me know if i’m doing any mistake. I have added the init script at the cluster level and from the logs i could see that openlineage is configured as i see a log statement

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-10-09 05:10:30
 +
+

*Thread Reply:* there's nothing wrong in that script. It's just we decided to limit amount of OL events for jobs that don't write their data anywhere and just do collect operation

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-10-09 05:11:02
 +
+

*Thread Reply:* this is also a potential reason why can't you see any events

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Guntaka Jeevan Paul + (jeevan@acceldata.io) +
+
2023-10-09 05:14:33
 +
+

*Thread Reply:* ohh…okk, will try out the test script that you have mentioned above. Kindly correct me if my understanding is correct, so if there are a few transformatiosna nd finally writing somewhere that is where the OL events are expected to be emitted?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-10-09 05:16:54
 +
+

*Thread Reply:* yes. main purpose of the lineage is to track dependencies between the datasets, when a job reads from dataset A and writes to dataset B. In case of databricks notebook, that do show or collect and print some query result on the screen, there may be no reason to track it in the sense of lineage.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-10-09 15:25:14
 +
+

@channel +We released OpenLineage 1.4.1! +Additions: +• Client: allow setting client’s endpoint via environment variable 2151 @Mars Lan +• Flink: expand Iceberg source types 2149 @Peter Huang +• Spark: add debug facet 2147 @Paweł Leszczyński +• Spark: enable Nessie REST catalog 2165 @julwin +Thanks to all the contributors, especially new contributors @Peter Huang and @julwin! +Release: https://github.com/OpenLineage/OpenLineage/releases/tag/1.4.1 +Changelog: https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md +Commit history: https://github.com/OpenLineage/OpenLineage/compare/1.3.1...1.4.1 +Maven: https://oss.sonatype.org/#nexus-search;quick~openlineage +PyPI: https://pypi.org/project/openlineage-python/

+ + + +
+ 👍 Jason Yip, Ross Turk, Mars Lan, Harel Shein, Rodrigo Maia +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Drew Bittenbender + (drew@salt.io) +
+
2023-10-09 16:55:35
 +
+

Hello. I am getting started with OL and Marquez with dbt. I am using dbt-ol. The namespace of the dataset showing up in Marquez is not the namespace I provide using OPENLINEAGENAMESPACE. It happens to be the same as the source in Marquez which is the snowflake account uri. It's obviously picking up the other env variable OPENLINEAGEURL so i am pretty sure its not the environment. Is this expected?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-10-09 18:56:13
 +
+

*Thread Reply:* Hi Drew, thank you for using OpenLineage! I don’t know the details of your use case, but I believe this is expected, yes. In general, the dataset namespace is different. Jobs are namespaced separately from datasets, which are namespaced by their containing datasources. This is the case so datasets have the same name regardless of the job writing to them, as datasets are sometimes shared by jobs in different namespaces.

+ + + +
+ 👍 Drew Bittenbender +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason Yip + (jasonyip@gmail.com) +
+
2023-10-10 01:05:11
 +
+

Any idea why "environment-properties" is gone in Spark 3.4+ in StartEvent?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason Yip + (jasonyip@gmail.com) +
+
2023-10-10 20:53:59
 +
+

example:

+ +

{"environment_properties":{"spark.databricks.clusterUsageTags.clusterName":"<a href="mailto:jason.yip@tredence.com">jason.yip@tredence.com</a>'s Cluster","spark.databricks.job.runId":"","spark.databricks.job.type":"","spark.databricks.clusterUsageTags.azureSubscriptionId":"a4f54399_8db8_4849_adcc_a42aed1fb97f","spark.databricks.notebook.path":"/Repos/jason.yip@tredence.com/segmentation/01_Data Prep","spark.databricks.clusterUsageTags.clusterOwnerOrgId":"4679476628690204","MountPoints":[{"MountPoint":"/databricks-datasets","Source":"databricks_datasets"},{"MountPoint":"/Volumes","Source":"UnityCatalogVolumes"},{"MountPoint":"/databricks/mlflow-tracking","Source":"databricks/mlflow-tracking"},{"MountPoint":"/databricks-results","Source":"databricks_results"},{"MountPoint":"/databricks/mlflow-registry","Source":"databricks/mlflow-registry"},{"MountPoint":"/Volume","Source":"DbfsReserved"},{"MountPoint":"/volumes","Source":"DbfsReserved"},{"MountPoint":"/","Source":"DatabricksRoot"},{"MountPoint":"/volume","Source":"DbfsReserved"}],"User":"<a href="mailto:jason.yip@tredence.com">jason.yip@tredence.com</a>","UserId":"4768657035718622","OrgId":"4679476628690204"}}

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-10-11 03:46:13
 +
+

*Thread Reply:* Is this related to any OL version? In OL 1.2.2. we've added extra variable spark.databricks.clusterUsageTags.clusterAllTags to be captured, but this should not break things.

+ +

I think we're facing some issues on recent databricks runtime versions. Here is an issue for this: https://github.com/OpenLineage/OpenLineage/issues/2131

+ +

Is the problem you describe specific to some databricks runtime versions?

+
+ + + + +
#2131 [Spark Databricks] NoSuchMethodError on ReplaceTableAsSelect
+
+ <p>More details here: <a href="https://github.com/OpenLineage/OpenLineage/issues/2121">#2121</a> <br /> +Looks like different class implementation on databricks platform.</p> +
+ +
+
Labels
+ integration/spark, integration/databricks +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason Yip + (jasonyip@gmail.com) +
+
2023-10-11 11:17:06
 +
+

*Thread Reply:* yes, exactly Spark 3.4+

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason Yip + (jasonyip@gmail.com) +
+
2023-10-11 21:12:27
 +
+

*Thread Reply:* Btw I don't understand the code flow entirely, if we are talking about a different classpath only, I see there's Unity Catalog handler in the code and it says it works the same as Delta, but I am not seeing it subclassing Delta. I suppose it will work the same.

+ +

I am happy to jump on a call to show you if needed

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason Yip + (jasonyip@gmail.com) +
+
2023-10-16 02:58:56
 +
+

*Thread Reply:* @Paweł Leszczyński do you think in Spark 3.4+ only one event would happen?

+ +

/** + * We get exact copies of OL events for org.apache.spark.scheduler.SparkListenerJobStart and + * org.apache.spark.sql.execution.ui.SparkListenerSQLExecutionStart. The same happens for end + * events. + * + * @return + */ + private boolean isOnJobStartOrEnd(SparkListenerEvent event) { + return event instanceof SparkListenerJobStart || event instanceof SparkListenerJobEnd; + }

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Guntaka Jeevan Paul + (jeevan@acceldata.io) +
+
2023-10-10 23:43:39
 +
+

@here i am trying out the databricks spark integration and in one of the events i am getting a openlineage event where the output dataset is having a facet called symlinks , the statement that generated this event is this sql +CREATE TABLE IF NOT EXISTS covid_research.covid_data +USING CSV +LOCATION '<abfss://oltptestdata@jeevanacceldata.dfs.core.windows.net/testdata/johns-hopkins-covid-19-daily-dashboard-cases-by-states.csv>' +OPTIONS (header "true", inferSchema "true"); +Can someone kindly let me know what this symlinks facet is. i tried seeing the spec but did not get it completely

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason Yip + (jasonyip@gmail.com) +
+
2023-10-10 23:44:53
 +
+

*Thread Reply:* I use it to get the table with database name

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Guntaka Jeevan Paul + (jeevan@acceldata.io) +
+
2023-10-10 23:47:15
 +
+

*Thread Reply:* so can i think it like if there is a synlink, then that table is kind of a reference to the original dataset

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason Yip + (jasonyip@gmail.com) +
+
2023-10-11 01:25:44
 +
+

*Thread Reply:* yes

+ + + +
+ 🙌 Paweł Leszczyński +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Guntaka Jeevan Paul + (jeevan@acceldata.io) +
+
2023-10-11 06:55:58
 +
+

@here When i am running this sql as part of a databricks notebook, i am recieving an OL event where i see only an output dataset and there is no input dataset or a symlink facet inside the dataset to map it to the underlying azure storage object. Can anyone kindly help on this +spark.sql(f"CREATE TABLE IF NOT EXISTS covid_research.uscoviddata USING delta LOCATION '<abfss://oltptestdata@jeevanacceldata.dfs.core.windows.net/testdata/modified-delta>'") +{ + "eventTime": "2023-10-11T10:47:36.296Z", + "producer": "<https://github.com/OpenLineage/OpenLineage/tree/1.2.2/integration/spark>", + "schemaURL": "<https://openlineage.io/spec/2-0-2/OpenLineage.json#/$defs/RunEvent>", + "eventType": "COMPLETE", + "run": { + "runId": "d0f40be9-b921-4c84-ac9f-f14a86c29ff7", + "facets": { + "spark.logicalPlan": { + "_producer": "<https://github.com/OpenLineage/OpenLineage/tree/1.2.2/integration/spark>", + "_schemaURL": "<https://openlineage.io/spec/2-0-2/OpenLineage.json#/$defs/RunFacet>", + "plan": [ + { + "class": "org.apache.spark.sql.catalyst.plans.logical.CreateTable", + "num-children": 1, + "name": 0, + "tableSchema": [], + "partitioning": [], + "tableSpec": null, + "ignoreIfExists": true + }, + { + "class": "org.apache.spark.sql.catalyst.analysis.ResolvedIdentifier", + "num-children": 0, + "catalog": null, + "identifier": null + } + ] + }, + "spark_version": { + "_producer": "<https://github.com/OpenLineage/OpenLineage/tree/1.2.2/integration/spark>", + "_schemaURL": "<https://openlineage.io/spec/2-0-2/OpenLineage.json#/$defs/RunFacet>", + "spark-version": "3.3.0", + "openlineage-spark-version": "1.2.2" + }, + "processing_engine": { + "_producer": "<https://github.com/OpenLineage/OpenLineage/tree/1.2.2/integration/spark>", + "_schemaURL": "<https://openlineage.io/spec/facets/1-1-0/ProcessingEngineRunFacet.json#/$defs/ProcessingEngineRunFacet>", + "version": "3.3.0", + "name": "spark", + "openlineageAdapterVersion": "1.2.2" + } + } + }, + "job": { + "namespace": "default", + "name": "adb-3942203504488904.4.azuredatabricks.net.create_table.covid_research_db_uscoviddata", + "facets": {} + }, + "inputs": [], + "outputs": [ + { + "namespace": "dbfs", + "name": "/user/hive/warehouse/covid_research.db/uscoviddata", + "facets": { + "dataSource": { + "_producer": "<https://github.com/OpenLineage/OpenLineage/tree/1.2.2/integration/spark>", + "_schemaURL": "<https://openlineage.io/spec/facets/1-0-0/DatasourceDatasetFacet.json#/$defs/DatasourceDatasetFacet>", + "name": "dbfs", + "uri": "dbfs" + }, + "schema": { + "_producer": "<https://github.com/OpenLineage/OpenLineage/tree/1.2.2/integration/spark>", + "_schemaURL": "<https://openlineage.io/spec/facets/1-0-0/SchemaDatasetFacet.json#/$defs/SchemaDatasetFacet>", + "fields": [] + }, + "storage": { + "_producer": "<https://github.com/OpenLineage/OpenLineage/tree/1.2.2/integration/spark>", + "_schemaURL": "<https://openlineage.io/spec/facets/1-0-0/StorageDatasetFacet.json#/$defs/StorageDatasetFacet>", + "storageLayer": "unity", + "fileFormat": "parquet" + }, + "symlinks": { + "_producer": "<https://github.com/OpenLineage/OpenLineage/tree/1.2.2/integration/spark>", + "_schemaURL": "<https://openlineage.io/spec/facets/1-0-0/SymlinksDatasetFacet.json#/$defs/SymlinksDatasetFacet>", + "identifiers": [ + { + "namespace": "/user/hive/warehouse/covid_research.db", + "name": "covid_research.uscoviddata", + "type": "TABLE" + } + ] + }, + "lifecycleStateChange": { + "_producer": "<https://github.com/OpenLineage/OpenLineage/tree/1.2.2/integration/spark>", + "_schemaURL": "<https://openlineage.io/spec/facets/1-0-0/LifecycleStateChangeDatasetFacet.json#/$defs/LifecycleStateChangeDatasetFacet>", + "lifecycleStateChange": "CREATE" + } + }, + "outputFacets": {} + } + ] +}

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Damien Hawes + (damien.hawes@booking.com) +
+
2023-10-11 06:57:46
 +
+

*Thread Reply:* Hey Guntaka - can I ask you a favour? Can you please stop using @here or @channel - please keep in mind, you're pinging over 1000 people when you use that mention. Its incredibly distracting to have Slack notify me of a message that isn't pertinent to me.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Guntaka Jeevan Paul + (jeevan@acceldata.io) +
+
2023-10-11 06:58:50
 +
+

*Thread Reply:* sure noted @Damien Hawes

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Damien Hawes + (damien.hawes@booking.com) +
+
2023-10-11 06:59:34
 +
+

*Thread Reply:* Thank you!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Madhav Kakumani + (madhav.kakumani@6point6.co.uk) +
+
2023-10-11 12:04:24
 +
+

Hi @there, I am trying to make API call to get column-lineage information could you please let me know the url construct to retrieve the same? As per the API documentation I am passing the following url to GET column-lineage: http://localhost:5000/api/v1/column-lineage but getting error code:400. Thanks

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2023-10-12 13:55:26
 +
+

*Thread Reply:* Make sure to provide a dataset field nodeId as a query param in your request. If you’ve seeded Marquez with test metadata, you can use: +curl -XGET "<http://localhost:5002/api/v1/column-lineage?nodeId=datasetField%3Afood_delivery%3Apublic.delivery_7_days%3Acustomer_email>" +You can view the API docs for column lineage here!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Madhav Kakumani + (madhav.kakumani@6point6.co.uk) +
+
2023-10-17 05:57:36
 +
+

*Thread Reply:* Thanks Willy. The documentation says 'name space' so i constructed API Like this: +'http://marquez-web:3000/api/v1/column-lineage/nodeId=datasetField:file:/home/jovyan/Downloads/event_attribute.csv:eventType' +but it is still not working 😞

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Madhav Kakumani + (madhav.kakumani@6point6.co.uk) +
+
2023-10-17 06:07:06
 +
+

*Thread Reply:* nodeId is constructed like this: datasetField:<namespace>:<dataset>:<field name>

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-10-11 13:00:01
 +
+

@channel +Friendly reminder: this month’s TSC meeting, open to all, is tomorrow at 10 am PT: https://openlineage.slack.com/archives/C01CK9T7HKR/p1696531454431629

+
+ + +
+ + + } + + Michael Robinson + (https://openlineage.slack.com/team/U02LXF3HUN7) +
+ + + +
+
+ <p><strong>@channel</strong> +This month’s TSC meeting is next Thursday the 12th at 10am PT. On the tentative agenda: +• announcements +• recent releases +• Airflow Summit recap +• tutorial: migrating to the Airflow Provider +• discussion topic: observability for OpenLineage/Marquez +• open discussion +• more (TBA) +More info and the meeting link can be found on the <a href="https://openlineage.io/meetings/">website</a>. All are welcome! Do you have a discussion topic, use case or integration you’d like to demo? DM me to be added to the agenda.</p> +
+ + + + + + +
Original URL: https://openlineage.slack.com/archives/C01CK9T7HKR/p1696531454431629
+ + +
+ + Slack Conversation +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-10-11 14:26:45
 +
+

*Thread Reply:* Newly added discussion topics: +• a proposal to add a Registry of Consumers and Producers +• a dbt issue to add OpenLineage Dataset names to the Manifest +• a proposal to add Dataset support in Spark LogicalPlan Nodes +• a proposal to institute a certification process for new integrations

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason Yip + (jasonyip@gmail.com) +
+
2023-10-12 15:08:34
 +
+

This might be a dumb question, I guess I need to setup local Spark in order for the Spark tests to run successfully?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-10-13 01:56:19
 +
+

*Thread Reply:* just follow these instructions: https://github.com/OpenLineage/OpenLineage/tree/main/integration/spark#build

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Guntaka Jeevan Paul + (jeevan@acceldata.io) +
+
2023-10-13 06:41:56
 +
+

*Thread Reply:* when trying to install openlineage-java in local via this command --> cd ../../client/java/ && ./gradlew publishToMavenLocal, i am receiving this error +```> Task :signMavenJavaPublication FAILED

+ +

FAILURE: Build failed with an exception.

+ +

** What went wrong: +Execution failed for task ':signMavenJavaPublication'. +> Cannot perform signing task ':signMavenJavaPublication' because it has no configured signatory```

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason Yip + (jasonyip@gmail.com) +
+
2023-10-13 13:35:06
 +
+

*Thread Reply:* @Paweł Leszczyński this is what I am getting

+ +
+
image.png
+ + + + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason Yip + (jasonyip@gmail.com) +
+
2023-10-13 13:36:00
 +
+

*Thread Reply:* attaching the html

+ +
+
io.openlineage.spark.agent.lifecycle.plan.AlterTableAddPartitionCommandVisitorTest.html
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-10-16 03:02:13
 +
+

*Thread Reply:* which java are you using? what is your operation system (is it windows?)?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason Yip + (jasonyip@gmail.com) +
+
2023-10-16 03:35:18
 +
+

*Thread Reply:* yes it is Windows, i downloaded java 8 but I can try to build it with Linux subsystem or Mac

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Guntaka Jeevan Paul + (jeevan@acceldata.io) +
+
2023-10-16 03:35:51
 +
+

*Thread Reply:* In my case it is Mac

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason Yip + (jasonyip@gmail.com) +
+
2023-10-16 03:56:09
 +
+

*Thread Reply: * Where: +Build file '/mnt/c/Users/jason/Downloads/github/OpenLineage/integration/spark/build.gradle' line: 9

+ +

** What went wrong: +An exception occurred applying plugin request [id: 'com.adarshr.test-logger', version: '3.2.0'] +> Failed to apply plugin [id 'com.adarshr.test-logger'] + > Could not generate a proxy class for class com.adarshr.gradle.testlogger.TestLoggerExtension.

+ +

** Try:

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason Yip + (jasonyip@gmail.com) +
+
2023-10-16 03:56:23
 +
+

*Thread Reply:* tried with Linux subsystem

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-10-16 04:04:29
 +
+

*Thread Reply:* we don't have any restrictions for windows builds, however it is something we don't test regularly. 2h ago we did have a successful build on circle CI https://app.circleci.com/pipelines/github/OpenLineage/OpenLineage/8271/workflows/0ec521ae-cd21-444a-bfec-554d101770ea

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason Yip + (jasonyip@gmail.com) +
+
2023-10-16 04:13:04
 +
+

*Thread Reply:* ... 111 more +Caused by: java.lang.ClassNotFoundException: org.gradle.api.provider.HasMultipleValues + ... 117 more

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason Yip + (jasonyip@gmail.com) +
+
2023-10-17 00:26:07
 +
+

*Thread Reply:* @Paweł Leszczyński now I am doing gradlew instead of gradle on windows coz Linux one doesn't work. The doc didn't mention about setting up Spark / Hadoop and that's my original question -- do I need to setup local Spark? Now it's throwing an error on Hadoop: java.io.FileNotFoundException: java.io.FileNotFoundException: HADOOP_HOME and hadoop.home.dir are unset.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason Yip + (jasonyip@gmail.com) +
+
2023-10-21 23:33:48
 +
+

*Thread Reply:* Got it working with Mac, couldn't get it working with Windows / Linux subsystem

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason Yip + (jasonyip@gmail.com) +
+
2023-10-22 13:08:40
 +
+

*Thread Reply:* Now getting class not found despite build and test succeeded

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason Yip + (jasonyip@gmail.com) +
+
2023-10-22 21:46:23
 +
+

*Thread Reply:* I uploaded the wrong jar.. there are so many jars, only the jar in the spark folder works, not subfolder

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-10-13 02:48:40
 +
+

Hi team, I am running the following pyspark code in a cell: +```print("SELECTING 100 RECORDS FROM METADATA TABLE") +df = spark.sql("""select ** from limit 100""")

+ +

print("WRITING (1) 100 RECORDS FROM METADATA TABLE") +df.write.mode("overwrite").format('delta').save("") +df.createOrReplaceTempView("temp_metadata")

+ +

print("WRITING (2) 100 RECORDS FROM METADATA TABLE") +df.write.mode("overwrite").format("delta").save("")

+ +

print("READING (1) 100 RECORDS FROM METADATA TABLE") +dfread = spark.read.format('delta').load("") +dfread.createOrReplaceTempView("metadata_1")

+ +

print("DOING THE MERGE INTO SQL STEP!") +dfnew = spark.sql(""" + MERGE INTO metadata1 + USING

+ ON metadata1.id = tempmetadata.id + WHEN MATCHED THEN UPDATE SET + metadata1.id = tempmetadata.id, + metadata1.aspect = tempmetadata.aspect + WHEN NOT MATCHED THEN INSERT (id, aspect) + VALUES (tempmetadata.id, tempmetadata.aspect) +""")`` +I am running with debug log levels. I actually don't see any of the events being logged forSaveIntoDataSourceCommandor theMergeIntoCommand`, but OL is in fact emitting events to the backend. It seems like the events are just not being logged... I actually observe this for all delta table related spark sql queries...

+ + + + + + + + + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-10-16 00:01:42
 +
+

*Thread Reply:* Hi @Paweł Leszczyński is this expected? CMIIW but we should expect to see the events being logged when running with debug log level right?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Damien Hawes + (damien.hawes@booking.com) +
+
2023-10-16 04:17:30
 +
+

*Thread Reply:* It's impossible to know without seeing how you've configured the listener.

+ +

Can you show this configuration?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-10-17 03:15:20
 +
+

*Thread Reply:* spark.openlineage.transport.url &lt;url&gt; +spark.openlineage.transport.endpoint /&lt;endpoint&gt; +spark.openlineage.transport.type http +spark.extraListeners io.openlineage.spark.agent.OpenLineageSparkListener +spark.openlineage.facets.custom_environment_variables [BUNCH_OF_VARIABLES;] +spark.openlineage.facets.disabled [spark_unknown\;spark.logicalPlan] +These are my spark configs... I'm setting log level to debug with sc.setLogLevel("DEBUG")

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Damien Hawes + (damien.hawes@booking.com) +
+
2023-10-17 04:40:03
 +
+

*Thread Reply:* Two things:

+ +
  1. If you want debug logs, you're going to have to provide a log4j.properties file or log4j2.properties file depending on the version of spark you're running. In that file, you will need to configure the logging levels. If I am not mistaken, the sc.setLogLevel controls ONLY the log levels of Spark namespaced components (i.e., org.apache.spark)
  2. You're telling the listener to emit to a URL. If you want to see the events emitted to the console, then set spark.openlineage.transport.type=console, and remove the other spark.openlineage.transport.** configurations. +Do either (1) or (2).
    3. +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-10-20 00:49:45
 +
+

*Thread Reply:* @Damien Hawes Hi, sflr.

+ +
  1. So enabling sc.setLogLevel does actually enable debug logs from Openlineage. I can see the events and everyting being logged if I save it as a parquet format instead of delta.
  2. I do want to emit events to the url. But, I would like to just see what exactly are the events being emitted for some specific jobs, since I see that the lineage is incorrect for some MergeInto cases
    3. +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-10-26 04:56:50
 +
+

*Thread Reply:* Hi @Damien Hawes would like to check again on whether you'd have any thoughts about this... Thanks! 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Rodrigo Maia + (rodrigo.maia@manta.io) +
+
2023-10-17 03:17:57
 +
+

Hello All 👋! +We are currently trying to work the the spark integration for OpenLineage in our Databricks instance. The general setup is done and working with a few hicups here and there. +But one thing we are still struggling is how to link all spark jobs events with a Databricks job or a notebook run. +We´ve recently noticed that some of the events produced by OL have the "environment-properties" attribute with information (for our context) regarding notebook path (if it is a notebook run), or the the job run ID (if its a databricks job run). But the thing is that these attributes are not always present. +I ran some samples yesterday for a job with 4 notebook tasks. From all 20 json payload sent by the OL listener, only 3 presented the "environment-properties" attribute. Its not only happening with Databricks jobs. When i run single notebooks and each cell has its onw set of spark jobs, not all json events presented that property either.

+ +

So my question is what is the criteria to have this attributes present or not in the event json file? Or maybe this in an issue? @Jason Yip did you find out anything about this?

+ +

⚙️ Spark 3.4 / OL-Spark 1.4.1

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-10-17 06:55:47
 +
+

*Thread Reply:* In general, we assume that OL events per run are cumulative. So, if you have 20 events with the same runId , then even if a single event contains some facet, we consider this is OK and let the backend combine it together. That's what we do in Marquez project (a reference backend architecture for OL) and that's why it is worth to use in Marquez as a rest API.

+ +

Are you able to use job namespace to aggregate all the Spark actions run within the databricks notebook? This is something that should serve this purpose.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason Yip + (jasonyip@gmail.com) +
+
2023-10-17 12:48:33
 +
+

*Thread Reply:* @Rodrigo Maia for Spark 3.4 I don't see the environment-properties showing up at all, but if you run the code as it is, register a listener on SparkListenerJobStart and get the properties, all of those properties will show up. There's an event filter that filters out the SparkListenerJobStart, I suspect that filtered out the "unneccessary" events.. was trying to do a custom build to do that, but still trying to setup Hadoop and Spark on my local

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Rodrigo Maia + (rodrigo.maia@manta.io) +
+
2023-10-18 05:23:16
 +
+

*Thread Reply:* @Paweł Leszczyński you are right. This is what we are doing as well, combining events with the same runId to process the information on our backend. But even so, there are several runIds without this information. I went through these events to have a better view of what was happening. As you can see from 7 runIds, only 3 were showing the "environment-properties" attribute. Some condition is not being met here, or maybe it is what @Jason Yip suspects and there's some sort of filtering of unnecessary events

+ +
+
image.png
+ + + + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-10-19 02:28:03
 +
+

*Thread Reply:* @Rodrigo Maia, If you are able to provide a small Spark script such that none of the OL events contain the environment-properties, but at least one should, please raise an issue for this.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-10-19 02:29:11
 +
+

*Thread Reply:* It's extremely helpful when community open issues that are not only described well, but also contain small piece of code needed to reproduce this.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Rodrigo Maia + (rodrigo.maia@manta.io) +
+
2023-10-19 02:59:39
 +
+

*Thread Reply:* I know. that's the goal. that is why I wanted to understand in the first place if there was any condition preventing this from happening, but now i get that this is not expected behaviour.

+ + + +
+ 👍 Paweł Leszczyński +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason Yip + (jasonyip@gmail.com) +
+
2023-10-19 13:44:00
 +
+

*Thread Reply:* @Paweł Leszczyński @Rodrigo Maia I am referring to this: https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/shared/src/main/java/io/openlineage/spark/agent/filters/DeltaEventFilter.java#L51

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/shared/src/main/java/io/openlineage/spark/agent/filters/DeltaEventFilter.java | DeltaEventFilter.java>
+
+ <pre><code> || isOnJobStartOrEnd(event); +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason Yip + (jasonyip@gmail.com) +
+
2023-10-19 14:49:03
 +
+

*Thread Reply:* Please note that I am getting the same behavior, no code is needed, Spark 3.4+ won't be generating no matter what. I have been testing the same code for 2 months from this issue: https://github.com/OpenLineage/OpenLineage/issues/2124

+ +

I tried the code without OL and it worked perfectly, so it is OL filtering out the event for sure. I will try posting the code I use to collect the properties.

+
+ + + + +
#2124 Same Delta Table not catching the location on write
+
+ <p><strong>What is the target system?</strong></p> + +<p>Spark / Databricks</p> + +<p><strong>What kind of integration is this?</strong></p> + +<p>☐ Produces OpenLineage metadata +☐ Consumes OpenLineage metadata +☐ Something else</p> + +<p><strong>How should this integration be implemented?</strong></p> + +<p>I am using OL 1.2.2, Azure Databricks Runtime 11.3 LTS. When creating a table writing into a ADLS location, OL won't be able to catch the location of the output. But when I read the same object it will be able to read the location as INPUT.</p> + +<p>Please note I have also tested Databricks Runtime 13.3 LTS, Spark 3.4.1 - it will give correct ADLS location in INPUT but the input will only show up once in a blue moon. Most of the time the inputs and outputs are blank.</p> + +<pre><code> "inputs": [], + "outputs": [] +</code></pre> + +<pre><code>CREATE OR REPLACE TABLE transactions_adj +USING DELTA LOCATION '&lt;wasbs://studio@clororetaildevadls.blob.core.windows.net/examples/data/csv/completejourney/silver/transactions_adj&gt;' +AS + SELECT + household_id, + basket_id, + week_no, + day, + transaction_time, + store_id, + product_id, + amount_list, + campaign_coupon_discount, + manuf_coupon_discount, + manuf_coupon_match_discount, + total_coupon_discount, + instore_discount, + amount_paid, + units + FROM ( + SELECT + household_id, + basket_id, + week_no, + day, + transaction_time, + store_id, + product_id, + COALESCE(sales_amount - discount_amount - coupon_discount - coupon_discount_match,0.0) as amount_list, + CASE + WHEN COALESCE(coupon_discount_match,0.0) = 0.0 THEN -1 ** COALESCE(coupon_discount,0.0) + ELSE 0.0 + END as campaign_coupon_discount, + CASE + WHEN COALESCE(coupon_discount_match,0.0) != 0.0 THEN -1 ** COALESCE(coupon_discount,0.0) + ELSE 0.0 + END as manuf_coupon_discount, + -1 ** COALESCE(coupon_discount_match,0.0) as manuf_coupon_match_discount, + -1 ** COALESCE(coupon_discount - coupon_discount_match,0.0) as total_coupon_discount, + COALESCE(-1 ** discount_amount,0.0) as instore_discount, + COALESCE(sales_amount,0.0) as `amount_paid,` + quantity as units + FROM transactions + ); +</code></pre> + +<p>Here's the COMPLETE event:</p> + +<pre><code> + "outputs":[ + { + "namespace":"dbfs", + "name":"/user/hive/warehouse/journey.db/transactions_adj", + "facets":{ + "dataSource":{ + "_producer":"&lt;https://github.com/OpenLineage/OpenLineage/tree/1.2.2/integration/spark&gt;", + "_schemaURL":"&lt;https://openlineage.io/spec/facets/1-0-0/DatasourceDatasetFacet.json#/$defs/DatasourceDatasetFacet&gt;", + "name":"dbfs", + "uri":"dbfs" + }, + +</code></pre> + +<p>Below logical plan shows the path:</p> + +<pre><code>== Analyzed Logical Plan == +num_affected_rows: bigint, num_inserted_rows: bigint +ReplaceTableAsSelect TableSpec(Map(),Some(DELTA),Map(),Some(&lt;wasbs://studio@clororetaildevadls.blob.core.windows.net/examples/data/csv/completejourney/silver/transactions_adj&gt;),None,None,false,Set()), true +:- ResolvedIdentifier com.databricks.sql.managedcatalog.UnityCatalogV2Proxy@6251a8df, default.transactions_adj ++- Project [household_id#184, basket_id#185L, week_no#193, day#186, transaction_time#192, store_id#190, product_id#187, amount_list#147, campaign_coupon_discount#148, manuf_coupon_discount#149, manuf_coupon_match_discount#150, total_coupon_discount#151, instore_discount#152, amount_paid#153, units#154] + +- SubqueryAlias __auto_generated_subquery_name + +- Project [household_id#184, basket_id#185L, week_no#193, day#186, transaction_time#192, store_id#190, product_id#187, coalesce(cast((((sales_amount#189 - discount_amount#191) - coupon_discount#194) - coupon_discount_match#195) as double), cast(0.0 as double)) AS amount_list#147, CASE WHEN (coalesce(cast(coupon_discount_match#195 as double), cast(0.0 as double)) = cast(0.0 as double)) THEN (cast(-1 as double) ** coalesce(cast(coupon_discount#194 as double), cast(0.0 as double))) ELSE cast(0.0 as double) END AS campaign_coupon_discount#148, CASE WHEN NOT (coalesce(cast(coupon_discount_match#195 as double), cast(0.0 as double)) = cast(0.0 as double)) THEN (cast(-1 as double) ** coalesce(cast(coupon_discount#194 as double), cast(0.0 as double))) ELSE cast(0.0 as double) END AS manuf_coupon_discount#149, (cast(-1 as double) ** coalesce(cast(coupon_discount_match#195 as double), cast(0.0 as double))) AS manuf_coupon_match_discount#150, (cast(-1 as double) ** coalesce(cast((coupon_discount#194 - coupon_discount_match#195) as double), cast(0.0 as double))) AS total_coupon_discount#151, coalesce(cast((cast(-1 as float) ** discount_amount#191) as double), cast(0.0 as double)) AS instore_discount#152, coalesce(cast(sales_amount#189 as double), cast(0.0 as double)) AS amount_paid#153, quantity#188 AS units#154] + +- SubqueryAlias spark_catalog.default.transactions + +- Relation spark_catalog.default.transactions[household_id#184,basket_id#185L,day#186,product_id#187,quantity#188,sales_amount#189,store_id#190,discount_amount#191,transaction_time#192,week_no#193,coupon_discount#194,coupon_discount_match#195] parquet +</code></pre> + +<p><strong>Where should this integration be implemented?</strong></p> + +<p>☐ In the target system +☐ In the OpenLineage repo +☐ Somewhere else</p> + +<p><strong>Do you plan to make this contribution yourself?</strong></p> + +<p>☐ I am interested in doing this work</p> +
+ +
+
Labels
+ integration/spark, integration/databricks +
+ +
+
Comments
+ 3 +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason Yip + (jasonyip@gmail.com) +
+
2023-10-19 23:46:17
 +
+

*Thread Reply:* this code proves that the prosperities are still there, somehow got filtered out by OL:

+ +

```%scala +import org.apache.spark.scheduler._

+ +

class JobStartListener extends SparkListener { + override def onJobStart(jobStart: SparkListenerJobStart): Unit = { + // Extract properties here + val jobId = jobStart.jobId + val stageInfos = jobStart.stageInfos + val properties = jobStart.properties

+ +
// You can print properties or save them somewhere
+println(s"JobId: $jobId, Stages: ${stageInfos.size}, Properties: $properties")
+
+ +

} +}

+ +

val listener = new JobStartListener() +spark.sparkContext.addSparkListener(listener)

+ +

val df = spark.range(1000).repartition(10) +df.count()```

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason Yip + (jasonyip@gmail.com) +
+
2023-10-19 23:55:05
 +
+

*Thread Reply:* of course feel free to test this logic as well, it still works -- if not the filtering:

+ +

https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/shared/src/[…]ark/agent/facets/builder/DatabricksEnvironmentFacetBuilder.java

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/shared/src/main/java/io/openlineage/spark/agent/facets/builder/DatabricksEnvironmentFacetBuilder.java | DatabricksEnvironmentFacetBuilder.java>
+
+ <pre><code>/** +/** Copyright 2018-2023 contributors to the OpenLineage project +/** SPDX-License-Identifier: Apache-2.0 +**/ + +package io.openlineage.spark.agent.facets.builder; + +import com.databricks.backend.daemon.dbutils.MountInfo; +import com.databricks.dbutils_v1.DbfsUtils; +import io.openlineage.spark.agent.facets.EnvironmentFacet; +import io.openlineage.spark.agent.models.DatabricksMountpoint; +import io.openlineage.spark.api.CustomFacetBuilder; +import io.openlineage.spark.api.OpenLineageContext; +import java.lang.reflect.Constructor; +import java.lang.reflect.InvocationTargetException; +import java.lang.reflect.Parameter; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.Optional; +import java.util.function.BiConsumer; +import lombok.extern.slf4j.Slf4j; +import org.apache.spark.scheduler.SparkListenerJobStart; +import scala.collection.JavaConversions; + +/**** + ** {@link CustomFacetBuilder} that generates a {@link EnvironmentFacet} when using OpenLineage on + ** Databricks. + **/ +@Slf4j +public class DatabricksEnvironmentFacetBuilder + extends CustomFacetBuilder&lt;SparkListenerJobStart, EnvironmentFacet&gt; { + private Map&lt;String, Object&gt; dbProperties; + private Class dbutilsClass; + private DbfsUtils dbutils; + + public static boolean isDatabricksRuntime() { + return System.getenv().containsKey("DATABRICKS_RUNTIME_VERSION"); + } + + public DatabricksEnvironmentFacetBuilder() {} + + public DatabricksEnvironmentFacetBuilder(OpenLineageContext openLineageContext) { + dbProperties = new HashMap&lt;&gt;(); + // extract some custom environment variables if needed + openLineageContext + .getCustomEnvironmentVariables() + .ifPresent( + envVars -&gt; + envVars.forEach(envVar -&gt; dbProperties.put(envVar, System.getenv().get(envVar)))); + } + + @Override + protected void build( + SparkListenerJobStart event, BiConsumer&lt;String, ? super EnvironmentFacet&gt; consumer) { + consumer.accept( + "environment-properties", + new EnvironmentFacet(getDatabricksEnvironmentalAttributes(event))); + } + + private Map&lt;String, Object&gt; getDatabricksEnvironmentalAttributes(SparkListenerJobStart jobStart) { + if (dbProperties == null) { + dbProperties = new HashMap&lt;&gt;(); + } + + // These are useful properties to extract if they are available + List&lt;String&gt; dbPropertiesKeys = + Arrays.asList( + "orgId", + "spark.databricks.clusterUsageTags.clusterOwnerOrgId", + "spark.databricks.notebook.path", + "spark.databricks.job.type", + "spark.databricks.job.id", + "spark.databricks.job.runId", + "user", + "userId", + "spark.databricks.clusterUsageTags.clusterName", + "spark.databricks.clusterUsageTags.clusterAllTags", + "spark.databricks.clusterUsageTags.azureSubscriptionId"); + dbPropertiesKeys.stream() + .forEach( + (p) -&gt; { + dbProperties.put(p, jobStart.properties().getProperty(p)); + }); + + /**** + ** Azure Databricks makes available a dbutils mount point to list aliased paths to cloud + ** storage. However, that dbutils object is not available inside a spark listener. We must + ** access it via reflection. + **/ + try { + Optional&lt;DbfsUtils&gt; dbfsUtils = getDbfsUtils(); + if (!dbfsUtils.isPresent()) { + dbProperties.put("mountPoints", new ArrayList&lt;DatabricksMountpoint&gt;()); + } else { + dbProperties.put("mountPoints", getDatabricksMountpoints(dbfsUtils.get())); + } + + } catch (Exception e) { + log.warn("Failed to load dbutils in OpenLineageListener:", e); + dbProperties.put("mountPoints", new ArrayList&lt;DatabricksMountpoint&gt;()); + } + return dbProperties; + } + + // Starting in Databricks Runtime 11, there is a new constructor for DbFsUtils + // If running on an older version, the constructor has no parameters. + // If running on DBR 11 or above, you need to specify whether you allow mount operations (true or + // false) + private static Optional&lt;DbfsUtils&gt; getDbfsUtils() + throws ClassNotFoundException, InstantiationException, IllegalAccessException, + IllegalArgumentException, InvocationTargetException { + Class dbutilsClass = Class.forName("com.databricks.dbutils_v1.impl.DbfsUtilsImpl"); + Constructor[] dbutilsConstructors = dbutilsClass.getDeclaredConstructors(); + if (dbutilsConstructors.length == 0) { + log.warn( + "Failed to load dbutils in OpenLineageListener as there were no declared constructors"); + return Optional.empty(); + } + Constructor firstConstructor = dbutilsConstructors[0]; + Parameter[] constructorParams = firstConstructor.getParameters(); + if (constructorParams.length == 0) { + log.debug("DbUtils constructor had no parameters"); + return Optional.of((DbfsUtils) firstConstructor.newInstance()); + } else if (constructorParams.length == 1 + &amp;&amp; constructorParams[0].getName().equals("allowMountOperations")) { + log.debug("DbUtils constructor had one parameter named allowMountOperations"); + return Optional.of((DbfsUtils) firstConstructor.newInstance(true)); + } else { + log.warn( + "dbutils had {} constructors and the first constructor had {} params", + dbutilsConstructors.length, + constructorParams.length); + return Optional.empty(); + } + } + + private static List&lt;DatabricksMountpoint&gt; getDatabricksMountpoints(DbfsUtils dbutils) { + List&lt;DatabricksMountpoint&gt; mountpoints = new ArrayList&lt;&gt;(); + List&lt;MountInfo&gt; mountsList = JavaConversions.seqAsJavaList(dbutils.mounts()); + for (MountInfo mount : mountsList) { + mountpoints.add(new DatabricksMountpoint(mount.mountPoint(), mount.source())); + } + return mountpoints; + } +} + +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Rodrigo Maia + (rodrigo.maia@manta.io) +
+
2023-10-30 04:46:16
 +
+

*Thread Reply:* Any ideas on how could i test it?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ankit jain + (ankit.goods10@gmail.com) +
+
2023-10-17 22:57:03
 +
+

Hello All, I am completely new for Openlineage, I have to setup the lab to conduct POC on various aspects like Lineage, metadata management , etc. As per openlineage site, i tried downloading Ubuntu, docker and binary files for Marquez. But I am lost somewhere and unable to configure whole setup. Can someone please assist in steps to start from scratch so that i can delve into the Openlineage capabilities. Many thanks

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-10-18 01:32:01
 +
+

*Thread Reply:* hey, did you try to follow one of these guides? +https://openlineage.io/docs/guides/about

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-10-18 09:14:08
 +
+

*Thread Reply:* Which guide were you using, and what errors/issues are you encountering?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ankit jain + (ankit.goods10@gmail.com) +
+
2023-10-21 15:43:14
 +
+

*Thread Reply:* Thanks Jakub for the response.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ankit jain + (ankit.goods10@gmail.com) +
+
2023-10-21 15:45:42
 +
+

*Thread Reply:* In docker, marquez-api image is not running and exiting with the exit code 127.

+ +
+
image.png
+ + + + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-10-22 09:34:53
 +
+

*Thread Reply:* @ankit jain thanks. I don't recognize 127, but 9 times out of 10 if the API or DB container fails the reason is a port conflict. Have you checked if port 5000 is available?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-10-22 09:54:10
 +
+

*Thread Reply:* could you please check what’s the output of +git config --get core.autocrlf +or +git config --global --get core.autocrlf +?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ankit jain + (ankit.goods10@gmail.com) +
+
2023-10-24 08:09:14
 +
+

*Thread Reply:* @Michael Robinson thanks , I checked the port 5000 is not available. +I tried deleting docker images and recreating them, but still the same issue persist stating +/Usr/bin/env bash/r not found. +Gradle build is successful.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ankit jain + (ankit.goods10@gmail.com) +
+
2023-10-24 08:09:54
 +
+

*Thread Reply:* @Jakub Dardziński thanks, first command resulted as true and second command has no response

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-10-24 08:15:57
 +
+

*Thread Reply:* are you running docker and git in Windows or Mac OS before 10.0?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Matthew Paras + (matthewparas2020@u.northwestern.edu) +
+
2023-10-19 15:00:42
 +
+

Hey all - we've been noticing that some events go unreported by openlineage (spark) when the AsyncEventQueue fills up and starts dropping events. Wondering if anyone has experienced this before, and knows why it is happening? We've expanded the event queue capacity and thrown more hardware at the problem but no dice

+ +

Also as a note, the query plans from this job are pretty big - could the listener just be choking up? Happy to open a github issue as well if we suspect that it could be the listener itself having issues

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-10-20 02:57:50
 +
+

*Thread Reply:* Hi, just checking, are you excluding the sparkPlan from the events? Or is it sending the spark plan too

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-10-23 11:59:40
 +
+

*Thread Reply:* yeah - setting spark.openlineage.facets.disabled to [spark_unknown;spark.logicalPlan] should help

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Matthew Paras + (matthewparas2020@u.northwestern.edu) +
+
2023-10-24 17:50:26
 +
+

*Thread Reply:* sorry for the late reply - turns out this job is just whack 😄 we were going in circles trying to figure it out, we end up dropping events without open lineage enabled at all. But good to know that disabling the logical plan should speed us up if we run into this again

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
praveen kanamarlapudi + (kpraveen420@gmail.com) +
+
2023-10-20 18:18:37
 +
+

Hi,

+ +

We are using openlineage spark connector. We have used spark 3.2 and scala 2.12 so far. We have triggered a new job with Spark 3.4 and scala 2.13 and faced below exception.

+ +

java.lang.NoSuchMethodError: 'scala.collection.Seq org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.map(scala.Function1)' + at io.openlineage.spark.agent.lifecycle.OpenLineageRunEventBuilder.lambda$buildInputDatasets$6(OpenLineageRunEventBuilder.java:341) + at java.base/java.util.Optional.map(Optional.java:265) + at io.openlineage.spark.agent.lifecycle.OpenLineageRunEventBuilder.buildInputDatasets(OpenLineageRunEventBuilder.java:339) + at io.openlineage.spark.agent.lifecycle.OpenLineageRunEventBuilder.populateRun(OpenLineageRunEventBuilder.java:295) + at io.openlineage.spark.agent.lifecycle.OpenLineageRunEventBuilder.buildRun(OpenLineageRunEventBuilder.java:279) + at io.openlineage.spark.agent.lifecycle.OpenLineageRunEventBuilder.buildRun(OpenLineageRunEventBuilder.java:222) + at io.openlineage.spark.agent.lifecycle.SparkSQLExecutionContext.start(SparkSQLExecutionContext.java:72) + at io.openlineage.spark.agent.OpenLineageSparkListener.lambda$sparkSQLExecStart$0(OpenLineageSparkListener.java:91)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-10-23 04:56:25
 +
+

*Thread Reply:* Hmy, that is interesting. Did it occur on databricks runtime? Could you give it a try with Scala 2.12? I think we don't test scala 2.13.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
praveen kanamarlapudi + (kpraveen420@gmail.com) +
+
2023-10-23 12:02:13
 +
+

*Thread Reply:* I believe our Scala 2.12 jobs are working fine. It's not databricks runtime. We run Spark on Kube.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-10-24 06:47:14
 +
+

*Thread Reply:* Ok. I think You can raise an issue to support Scala 2.13 for latest Spark versions.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
priya narayana + (n.priya88@gmail.com) +
+
2023-10-26 06:13:40
 +
+

Hi I want to customise the events which comes from Openlineage spark . Can some one give some information

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-10-26 07:45:41
 +
+

*Thread Reply:* Hi @priya narayana, please get familiar with Extending section on our docs: https://github.com/OpenLineage/OpenLineage/tree/main/integration/spark#extending

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
priya narayana + (n.priya88@gmail.com) +
+
2023-10-26 09:53:07
 +
+

*Thread Reply:* Okay thank you. Just checking any other docs or git code which also can help me

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
harsh loomba + (hloomba@upgrade.com) +
+
2023-10-26 13:11:17
 +
+

Hello Team

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
harsh loomba + (hloomba@upgrade.com) +
+
2023-10-26 13:12:38
 +
+

Im upgrading the version from openlineage-airflow==0.24.0 to openlineage-airflow 1.4.1 but im seeing the following error, any help is appreciated

+ +
+
Screenshot 2023-10-26 at 10.11.34 AM.png
+ + + + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
harsh loomba + (hloomba@upgrade.com) +
+
2023-10-26 13:14:02
 +
+

*Thread Reply:* @Jakub Dardziński any thoughts?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-10-26 13:14:24
 +
+

*Thread Reply:* what version of Airflow are you using?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
harsh loomba + (hloomba@upgrade.com) +
+
2023-10-26 13:14:52
 +
+

*Thread Reply:* 2.6.3 that satisfies the requirement

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-10-26 13:16:38
 +
+

*Thread Reply:* is it possible you have some custom operator?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
harsh loomba + (hloomba@upgrade.com) +
+
2023-10-26 13:17:15
 +
+

*Thread Reply:* i think its the base operator causing the issue

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
harsh loomba + (hloomba@upgrade.com) +
+
2023-10-26 13:17:36
 +
+

*Thread Reply:* so no i believe

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-10-26 13:18:43
 +
+

*Thread Reply:* BaseOperator is parent class for any other operators, it defines how to do deepcopy

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
harsh loomba + (hloomba@upgrade.com) +
+
2023-10-26 13:19:11
 +
+

*Thread Reply:* yeah so its controlled by Airflow itself, I didnt customize it

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-10-26 13:19:49
 +
+

*Thread Reply:* uhm, maybe it's possible you could share dag code? you may hide sensitive data

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
harsh loomba + (hloomba@upgrade.com) +
+
2023-10-26 13:21:23
 +
+

*Thread Reply:* let me try with lower versions of openlineage, what's say

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
harsh loomba + (hloomba@upgrade.com) +
+
2023-10-26 13:21:39
 +
+

*Thread Reply:* its a big jump from 0.24.0 to 1.4.1

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
harsh loomba + (hloomba@upgrade.com) +
+
2023-10-26 13:22:25
 +
+

*Thread Reply:* but i will help here to investigate this issue

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-10-26 13:24:03
 +
+

*Thread Reply:* for me it seems that within dag or task you're defining some object that is not easy to copy

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
harsh loomba + (hloomba@upgrade.com) +
+
2023-10-26 13:26:05
 +
+

*Thread Reply:* possible, but with 0.24.0 that issue is not occurring, so worry is that the version upgrade could potentially break things

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-10-26 13:39:34
 +
+

*Thread Reply:* 0.24.0 is not that old 🤔

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
harsh loomba + (hloomba@upgrade.com) +
+
2023-10-26 13:45:07
 +
+

*Thread Reply:* i see the issue with 0.24.0 I see it as warning +[airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File "/usr/lib64/python3.8/threading.py", line 932, in _bootstrap_inner +[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - self.run() +[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File "/usr/lib64/python3.8/threading.py", line 870, in run +[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - self._target(**self._args, ****self._kwargs) +[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File "/home/upgrade/.local/lib/python3.8/site-packages/openlineage/airflow/listener.py", line 89, in on_running +[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - task_instance_copy = copy.deepcopy(task_instance) +[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File "/usr/lib64/python3.8/copy.py", line 172, in deepcopy +[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - y = _reconstruct(x, memo, **rv) +[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File "/usr/lib64/python3.8/copy.py", line 270, in _reconstruct +[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - state = deepcopy(state, memo) +[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File "/usr/lib64/python3.8/copy.py", line 146, in deepcopy +[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - y = copier(x, memo) +[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File "/usr/lib64/python3.8/copy.py", line 230, in _deepcopy_dict +[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - y[deepcopy(key, memo)] = deepcopy(value, memo) +[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File "/usr/lib64/python3.8/copy.py", line 172, in deepcopy +[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - y = _reconstruct(x, memo, **rv) +[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File "/usr/lib64/python3.8/copy.py", line 270, in _reconstruct +[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - state = deepcopy(state, memo) +[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File "/usr/lib64/python3.8/copy.py", line 146, in deepcopy +[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - y = copier(x, memo) +[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File "/usr/lib64/python3.8/copy.py", line 230, in _deepcopy_dict +[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - y[deepcopy(key, memo)] = deepcopy(value, memo) +[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File "/usr/lib64/python3.8/copy.py", line 153, in deepcopy +[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - y = copier(memo) +[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File "/home/upgrade/.local/lib/python3.8/site-packages/airflow/models/dag.py", line 2162, in __deepcopy__ +[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - setattr(result, k, copy.deepcopy(v, memo)) +[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File "/usr/lib64/python3.8/copy.py", line 146, in deepcopy +[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - y = copier(x, memo) +[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File "/usr/lib64/python3.8/copy.py", line 230, in _deepcopy_dict +[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - y[deepcopy(key, memo)] = deepcopy(value, memo) +[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File "/usr/lib64/python3.8/copy.py", line 153, in deepcopy +[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - y = copier(memo) +[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File "/home/upgrade/.local/lib/python3.8/site-packages/airflow/models/baseoperator.py", line 1224, in __deepcopy__ +[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - setattr(result, k, copy.deepcopy(v, memo)) +[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File "/usr/lib64/python3.8/copy.py", line 172, in deepcopy +[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - y = _reconstruct(x, memo, **rv) +[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File "/usr/lib64/python3.8/copy.py", line 270, in _reconstruct +[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - state = deepcopy(state, memo) +[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File "/usr/lib64/python3.8/copy.py", line 146, in deepcopy +[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - y = copier(x, memo) +[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File "/usr/lib64/python3.8/copy.py", line 230, in _deepcopy_dict +[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - y[deepcopy(key, memo)] = deepcopy(value, memo) +[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File "/usr/lib64/python3.8/copy.py", line 146, in deepcopy +[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - y = copier(x, memo) +[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File "/usr/lib64/python3.8/copy.py", line 230, in _deepcopy_dict +[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - y[deepcopy(key, memo)] = deepcopy(value, memo) +[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File "/usr/lib64/python3.8/copy.py", line 153, in deepcopy +[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - y = copier(memo) +[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File "/home/upgrade/.local/lib/python3.8/site-packages/airflow/models/baseoperator.py", line 1224, in __deepcopy__ +[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - setattr(result, k, copy.deepcopy(v, memo)) +[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File "/usr/lib64/python3.8/copy.py", line 146, in deepcopy +[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - y = copier(x, memo) +[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File "/usr/lib64/python3.8/copy.py", line 230, in _deepcopy_dict +[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - y[deepcopy(key, memo)] = deepcopy(value, memo) +[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File "/usr/lib64/python3.8/copy.py", line 161, in deepcopy +[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - rv = reductor(4) +[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - TypeError: cannot pickle 'module' object +but with 1.4.1 its stopped processing any further and threw error

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
harsh loomba + (hloomba@upgrade.com) +
+
2023-10-26 14:18:08
 +
+

*Thread Reply:* I see the difference of calling in these 2 versions, current versions checks if Airflow is >2.6 then directly runs on_running but earlier version was running on separate thread. IS this what's raising this exception?

+ +
+
Screenshot 2023-10-26 at 11.16.47 AM.png
+ + + + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
harsh loomba + (hloomba@upgrade.com) +
+
2023-10-26 14:24:49
 +
+

*Thread Reply:* this is the issue - https://github.com/OpenLineage/OpenLineage/blob/c343835c1664eda94d5c315897ae6702854c81bd/integration/airflow/openlineage/airflow/listener.py#L89 while copying the task

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/c343835c1664eda94d5c315897ae6702854c81bd/integration/airflow/openlineage/airflow/listener.py | listener.py>
+
+ <pre><code> task_instance_copy = copy.deepcopy(task_instance) +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
harsh loomba + (hloomba@upgrade.com) +
+
2023-10-26 14:25:21
 +
+

*Thread Reply:* since we are directly running if version>2.6.0 therefore its throwing error in main processing

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
harsh loomba + (hloomba@upgrade.com) +
+
2023-10-26 14:28:02
 +
+

*Thread Reply:* may i know which Airflow version we tested this process?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
harsh loomba + (hloomba@upgrade.com) +
+
2023-10-26 14:28:39
 +
+

*Thread Reply:* im on 2.6.3

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-10-26 14:30:53
 +
+

*Thread Reply:* 2.1.4, 2.2.4, 2.3.4, 2.4.3, 2.5.2, 2.6.1 +usually there are not too many changes between minor versions

+ +

I still believe it might be some code you might improve and probably is also an antipattern in airflow

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
harsh loomba + (hloomba@upgrade.com) +
+
2023-10-26 14:34:26
 +
+

*Thread Reply:* hummm...that's a valid observation but I dont write DAGS, other teams do, so imagine if many people wrote such DAGS I can't ask everyone to change their patterns right? If something is running on current openlineage version with warning that should still be running on upgraded version isn't it?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
harsh loomba + (hloomba@upgrade.com) +
+
2023-10-26 14:38:04
 +
+

*Thread Reply:* however I see ur point

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
harsh loomba + (hloomba@upgrade.com) +
+
2023-10-26 14:49:52
 +
+

*Thread Reply:* So that specific task has 570 line of query and pretty bulky query, let me split into smaller units

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
harsh loomba + (hloomba@upgrade.com) +
+
2023-10-26 14:50:15
 +
+

*Thread Reply:* that should help right? @Jakub Dardziński

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-10-26 14:51:27
 +
+

*Thread Reply:* query length shouldn’t be the issue, rather any python code

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-10-26 14:51:50
 +
+

*Thread Reply:* I get your point too, we might figure out some mechanism to skip irrelevant parts of task instance so that it doesn’t fail then

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
harsh loomba + (hloomba@upgrade.com) +
+
2023-10-26 14:52:12
 +
+

*Thread Reply:* actually its failing on that task itself

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
harsh loomba + (hloomba@upgrade.com) +
+
2023-10-26 14:52:33
 +
+

*Thread Reply:* let me try it will be pretty quick

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
harsh loomba + (hloomba@upgrade.com) +
+
2023-10-26 14:58:58
 +
+

*Thread Reply:* @Jakub Dardziński but ur right we have to fix this at Openlineage side as well. Because ideally Openlineage shouldn't be causing any issue to the main DAG processing

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-10-26 17:51:05
 +
+

*Thread Reply:* it doesn’t break any airflow functionality, execution is wrapped into try/except block, only exception traceback is logged as you can see

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-10-27 05:25:54
 +
+

*Thread Reply:* Can you migrate to Airflow 2.7 and use apache-airflow-providers-openlineage? Ideally we wouldn't make meaningful changes to openlineage-airflow

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
harsh loomba + (hloomba@upgrade.com) +
+
2023-10-27 11:35:44
 +
+

*Thread Reply:* yup thats what im planning to do

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
harsh loomba + (hloomba@upgrade.com) +
+
2023-10-27 13:59:03
 +
+

*Thread Reply:* referencing to https://openlineage.slack.com/archives/C01CK9T7HKR/p1698398754823079?threadts=1698340358.557159&cid=C01CK9T7HKR|this conversation - what it takes to move to openlineage provider package from openlineage-airflow. Im updating Airflow to 2.7.2 but moving off of openlineage-airflow to provider package Im trying to estimate the amount of work it takes, any thoughts? reading changelogs I dont think its too much of a change but please share your thoughts and if somewhere its drafted please do share that as well

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-10-30 08:21:10
 +
+

*Thread Reply:* Generally not much - I would maybe think of a operator coverage. For example, for BigQuery old openlineage-airflow supports BigQueryExecuteQueryOperator. However, new apache-airflow-providers-openlineage supports BigQueryInsertJobOperator - because it's intended replacement for BigQueryExecuteQueryOperator and Airflow community does not want to accept contributions to deprecated operators.

+ + + +
+ 🙏 harsh loomba +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
harsh loomba + (hloomba@upgrade.com) +
+
2023-10-31 15:00:38
 +
+

*Thread Reply:* one question if someone is around - when im keeping both openlineage-airflow and apache-airflow-providers-openlineage in my requirement file, i see the following error - +from openlineage.airflow.extractors import Extractors +ModuleNotFoundError: No module named 'openlineage.airflow' +any thoughts?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Lukenoff + (john@jlukenoff.com) +
+
2023-10-31 15:37:07
 +
+

*Thread Reply:* I would usually do a pip freeze | grep openlineage as a sanity check to validate that the module is actually installed. Not sure how the provider and the module play together though

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
harsh loomba + (hloomba@upgrade.com) +
+
2023-10-31 17:07:41
 +
+

*Thread Reply:* yeah so @John Lukenoff im not getting how i can use the specific extractor when i run my operator. Say for example, I have custom datawarehouseOperator and i want to override getopenlineagefacetsonstart and getopenlineagefacetsoncomplete using the redshift extractor then how would i do that?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Rodrigo Maia + (rodrigo.maia@manta.io) +
+
2023-10-27 05:49:25
 +
+

Spark Integration Logs +Hey There +Are these events skipped because it's not supported or it's configured somewhere? +23/10/27 08:25:58 INFO SparkSQLExecutionContext: OpenLineage received Spark event that is configured to be skipped: SparkListenerSQLExecutionStart +23/10/27 08:25:58 INFO SparkSQLExecutionContext: OpenLineage received Spark event that is configured to be skipped: SparkListenerSQLExecutionEnd

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Hitesh + (splicer9904@gmail.com) +
+
2023-10-27 08:12:32
 +
+

Hi People, actually I want to intercept the OpenLineage spark events right after the job ends and before they are emitted, so that I can add some extra information to the events or remove some information that I don't want. +Is there any way of doing this? Can someone please help me

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-10-30 09:03:57
 +
+

*Thread Reply:* It general, I think this kind of use case is probably best served by facets, but what do you think @Paweł Leszczyński?

+
+
openlineage.io
+ + + +
Understanding and Using Facets | OpenLineage
+
+ Adapted from the OpenLineage spec. +
+ + + + + + +
Original URL: https://openlineage.io/docs/guides/facets
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kavitha + (kkandaswamy@cardinalcommerce.com) +
+
2023-10-27 17:01:12
 +
+

Hello, has anyone run into similar error as posted in this github open issues[https://github.com/MarquezProject/marquez/issues/2468] while setting up marquez on an EC2 Instance, would appreciate any help to get past the errors

+
+ + + + +
#2468 Connection error PostgreSQL and Marquez
+
+ <p>Hi all!</p> + +<p>I really appreciate this repository and would love to use it for its data lineage aspect! <br /> +Last few days, I've been trying to get the <code>marquez api</code> Docker image up and running but whatever I try, it does not seem to work.</p> + +<p>It returns the following errors in Docker Desktop:</p> + +<p><code>2023-04-03 18:07:04 marquez-db | 2023-04-03 23:07:04.726 GMT [35] FATAL: password authentication failed for user "marquez" 2023-04-03 18:07:04 marquez-db | 2023-04-03 23:07:04.726 GMT [35] DETAIL: Role "marquez" does not exist.</code></p> + +<p>and</p> + +<p><code>ERROR [2023-04-03 22:44:31,217] org.apache.tomcat.jdbc.pool.ConnectionPool: Unable to create initial connections of pool. 2023-04-03 17:44:31 ! org.postgresql.util.PSQLException: FATAL: password authentication failed for user "marquez"</code></p> + +<p>All the steps in the README.md file were used to configure the PostgreSQL database for Marquez and to set the correct environment variables. <br /> +During my trouble shooting practices, I found a relatively similar issue on StackOverflow which refers to the danger in using the same ports for Postgres and Marquez but this did not help me yet in solving the issue (<a href="https://stackoverflow.com/questions/62115827/org-postgresql-util-psqlexception-fatal-password-authentication-failed-for-use">https://stackoverflow.com/questions/62115827/org-postgresql-util-psqlexception-fatal-password-authentication-failed-for-use</a>).</p> + +<p>Could you please help me out?</p> + +<p>Kind regards,</p> + +<p>Tom</p> +
+ +
+
Comments
+ 5 +
+ + + + + + + +
+ + <a href="https://github.com/MarquezProject/marquez">MarquezProject/marquez</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2023-10-27 17:04:30
 +
+

*Thread Reply:* Hmm, have you looked over our Running on AWS docs?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2023-10-27 17:06:08
 +
+

*Thread Reply:* More specifically, the AWS RDS section. How are you deploying Marquez on Ec2?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kavitha + (kkandaswamy@cardinalcommerce.com) +
+
2023-10-27 17:08:05
 +
+

*Thread Reply:* we were primarily referencing this document on git - https://github.com/MarquezProject/marquez

+
+ + + + +
MarquezProject/marquez
+
+ Collect, aggregate, and visualize a data ecosystem's metadata +
+ +
+
Website
+ <https://marquezproject.ai> +
+ +
+
Stars
+ 1450 +
+ + + + + + + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kavitha + (kkandaswamy@cardinalcommerce.com) +
+
2023-10-27 17:09:05
 +
+

*Thread Reply:* leveraged docker and docker-compose

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2023-10-27 17:13:10
 +
+

*Thread Reply:* hmm so you’re running docker-compose up on an Ec2 instance you’ve ssh’d into? (just trying to understand your setup better)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kavitha + (kkandaswamy@cardinalcommerce.com) +
+
2023-10-27 17:13:26
 +
+

*Thread Reply:* yes, thats correct

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2023-10-27 17:16:39
 +
+

*Thread Reply:* I’ve only used docker compose for local dev or integration tests. but, ok you’re probably in the PoC phase. Can you run the docker cmd on you local machine successfully? What OS is stalled on the Ec2 instance?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kavitha + (kkandaswamy@cardinalcommerce.com) +
+
2023-10-27 17:18:00
 +
+

*Thread Reply:* yes, i can run and the OS is Ubuntu 20.04.6 LTS

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kavitha + (kkandaswamy@cardinalcommerce.com) +
+
2023-10-27 17:19:27
 +
+

*Thread Reply:* we initiallly ran into a permission denied error related to postgressql.conf file and we had to update file permissions to 777 and after which we started to see below errors

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kavitha + (kkandaswamy@cardinalcommerce.com) +
+
2023-10-27 17:19:36
 +
+

*Thread Reply:* marquez-db | 2023-10-27 20:35:52.512 GMT [35] FATAL: no pghba.conf entry for host "172.18.0.5", user "marquez", database "marquez", no encryption + marquez-db | 2023-10-27 20:35:52.529 GMT [36] FATAL: no pghba.conf entry for host "172.18.0.5", user "marquez", database "marquez", no encryption

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kavitha + (kkandaswamy@cardinalcommerce.com) +
+
2023-10-27 17:20:12
 +
+

*Thread Reply:* we then manually updated pg_hba.conf file to include host user and db details

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2023-10-27 17:20:42
 +
+

*Thread Reply:* Did you also update the marquez.yml with the db user / password?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kavitha + (kkandaswamy@cardinalcommerce.com) +
+
2023-10-27 17:20:48
 +
+

*Thread Reply:* after which we started to see the errors posted in the github open issues page

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2023-10-27 17:21:33
 +
+

*Thread Reply:* hmm are you using an external database or are you spinning up the entire Marquez stack with docker compose?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kavitha + (kkandaswamy@cardinalcommerce.com) +
+
2023-10-27 17:21:56
 +
+

*Thread Reply:* we are spinning up the entire Marquez stack with docker compose

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kavitha + (kkandaswamy@cardinalcommerce.com) +
+
2023-10-27 17:23:24
 +
+

*Thread Reply:* we did not change anything in the marquez.yml, i think we did not find that file in the github repo that we cloned into our local instance

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2023-10-27 17:26:31
 +
+

*Thread Reply:* It’s important that the init-db.sh script runs, but I don’t think it is

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2023-10-27 17:26:56
 +
+

*Thread Reply:* can you grab all the docker compose logs and share them? it’s hard to debug otherwise

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kavitha + (kkandaswamy@cardinalcommerce.com) +
+
2023-10-27 17:29:59
 +
+

*Thread Reply:*

+ +
+
logs.txt
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2023-10-27 17:33:15
 +
+

*Thread Reply:* I would first suggest to remove the --build flag since you are specifying a version of Marquez to use via --tag

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2023-10-27 17:33:49
 +
+

*Thread Reply:* no the issue per se, but will help clear up some of the logs

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kavitha + (kkandaswamy@cardinalcommerce.com) +
+
2023-10-27 17:35:06
 +
+

*Thread Reply:* for sure thanks. we could get the logs without the --build portion, we tried with that option just once

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kavitha + (kkandaswamy@cardinalcommerce.com) +
+
2023-10-27 17:35:40
 +
+

*Thread Reply:* the errors were the same with/without --build option

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kavitha + (kkandaswamy@cardinalcommerce.com) +
+
2023-10-27 17:36:02
 +
+

*Thread Reply:* marquez-api | ERROR [2023-10-27 21:34:58,019] org.apache.tomcat.jdbc.pool.ConnectionPool: Unable to create initial connections of pool. + marquez-api | ! org.postgresql.util.PSQLException: FATAL: password authentication failed for user "marquez" + marquez-api | ! at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:693) + marquez-api | ! at org.postgresql.core.v3.ConnectionFactoryImpl.tryConnect(ConnectionFactoryImpl.java:203) + marquez-api | ! at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:258) + marquez-api | ! at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:54) + marquez-api | ! at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:253) + marquez-api | ! at org.postgresql.Driver.makeConnection(Driver.java:434) + marquez-api | ! at org.postgresql.Driver.connect(Driver.java:291) + marquez-api | ! at org.apache.tomcat.jdbc.pool.PooledConnection.connectUsingDriver(PooledConnection.java:346) + marquez-api | ! at org.apache.tomcat.jdbc.pool.PooledConnection.connect(PooledConnection.java:227) + marquez-api | ! at org.apache.tomcat.jdbc.pool.ConnectionPool.createConnection(ConnectionPool.java:768) + marquez-api | ! at org.apache.tomcat.jdbc.pool.ConnectionPool.borrowConnection(ConnectionPool.java:696) + marquez-api | ! at org.apache.tomcat.jdbc.pool.ConnectionPool.init(ConnectionPool.java:495) + marquez-api | ! at org.apache.tomcat.jdbc.pool.ConnectionPool.<init>(ConnectionPool.java:153) + marquez-api | ! at org.apache.tomcat.jdbc.pool.DataSourceProxy.pCreatePool(DataSourceProxy.java:118) + marquez-api | ! at org.apache.tomcat.jdbc.pool.DataSourceProxy.createPool(DataSourceProxy.java:107) + marquez-api | ! at org.apache.tomcat.jdbc.pool.DataSourceProxy.getConnection(DataSourceProxy.java:131) + marquez-api | ! at org.flywaydb.core.internal.jdbc.JdbcUtils.openConnection(JdbcUtils.java:48) + marquez-api | ! at org.flywaydb.core.internal.jdbc.JdbcConnectionFactory.<init>(JdbcConnectionFactory.java:75) + marquez-api | ! at org.flywaydb.core.FlywayExecutor.execute(FlywayExecutor.java:147) + marquez-api | ! at org.flywaydb.core.Flyway.info(Flyway.java:190) + marquez-api | ! at marquez.db.DbMigration.hasPendingDbMigrations(DbMigration.java:73) + marquez-api | ! at marquez.db.DbMigration.migrateDbOrError(DbMigration.java:27) + marquez-api | ! at marquez.MarquezApp.run(MarquezApp.java:105) + marquez-api | ! at marquez.MarquezApp.run(MarquezApp.java:48) + marquez-api | ! at io.dropwizard.cli.EnvironmentCommand.run(EnvironmentCommand.java:67) + marquez-api | ! at io.dropwizard.cli.ConfiguredCommand.run(ConfiguredCommand.java:98) + marquez-api | ! at io.dropwizard.cli.Cli.run(Cli.java:78) + marquez-api | ! at io.dropwizard.Application.run(Application.java:94) + marquez-api | ! at marquez.MarquezApp.main(MarquezApp.java:60) + marquez-api | INFO [2023-10-27 21:34:58,024] marquez.MarquezApp: Stopping app...

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2023-10-27 17:38:52
 +
+

*Thread Reply:* debugging docker issues like this is so difficult

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2023-10-27 17:40:44
 +
+

*Thread Reply:* it could be a number of things, but you are connected to the database it’s just that the marquez user hasn’t been created

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2023-10-27 17:41:59
 +
+

*Thread Reply:* the /init-db.sh is what manages user creation

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2023-10-27 17:42:17
 +
+

*Thread Reply:* so it’s possible that the script isn’t running for whatever reason on your Ec2 instance

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2023-10-27 17:44:20
 +
+

*Thread Reply:* do you have other services running on that Ec2 instance? Like, other than Marquez

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2023-10-27 17:44:52
 +
+

*Thread Reply:* is there a postgres process running outside of docker?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kavitha + (kkandaswamy@cardinalcommerce.com) +
+
2023-10-27 20:34:50
 +
+

*Thread Reply:* no other services except marquez on this EC2 instance

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kavitha + (kkandaswamy@cardinalcommerce.com) +
+
2023-10-27 20:35:49
 +
+

*Thread Reply:* this was a new Ec2 instance that was spun up to install and use marquez

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kavitha + (kkandaswamy@cardinalcommerce.com) +
+
2023-10-27 20:36:09
 +
+

*Thread Reply:* n we can confirm that no postgres process runs outside of docker

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason Yip + (jasonyip@gmail.com) +
+
2023-10-29 03:06:28
 +
+

I realize in Spark 3.4+, some job ids don't have a start event. What part of the code is responsible for triggering the START and COMPLETE event

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-10-30 09:59:53
 +
+

*Thread Reply:* hi @Jason Yip could you provide an example of such a job?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason Yip + (jasonyip@gmail.com) +
+
2023-10-30 16:51:55
 +
+

*Thread Reply:* @Paweł Leszczyński same old:

+ +

delete the old table if needed

+ +

_ = spark.sql('DROP TABLE IF EXISTS transactions')

+ +

expected structure of the file

+ +

transactionsschema = StructType([ + StructField('householdid', IntegerType()), + StructField('basketid', LongType()), + StructField('day', IntegerType()), + StructField('productid', IntegerType()), + StructField('quantity', IntegerType()), + StructField('salesamount', FloatType()), + StructField('storeid', IntegerType()), + StructField('discountamount', FloatType()), + StructField('transactiontime', IntegerType()), + StructField('weekno', IntegerType()), + StructField('coupondiscount', FloatType()), + StructField('coupondiscountmatch', FloatType()) + ])

+ +

read data to dataframe

+ +

df = (spark + .read + .csv( + adlsRootPath + '/examples/data/csv/completejourney/transactiondata.csv', + header=True, + schema=transactionsschema))

+ +

df.write\ + .format('delta')\ + .mode('overwrite')\ + .option('overwriteSchema', 'true')\ + .option('path', adlsRootPath + '/examples/data/csv/completejourney/silver/transactions')\ + .saveAsTable('transactions')

+ +

df.count()

+ +

# create table object to make delta lake queryable

+ +

_ = spark.sql(f'''

+ +

CREATE TABLE transactions

+ +

USING DELTA

+ +

LOCATION '{adlsRootPath}/examples/data/csv/completejourney/silver/transactions'

+ +

''')

+ +

show data

+ +

display( + spark.table('transactions') + )

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Lukenoff + (john@jlukenoff.com) +
+
2023-10-30 18:51:43
 +
+

👋 Hi team, cross-posting from the Marquez Channel in case anyone here has a better idea of the spec

+ +

> For most of our lineage extractors in airflow, we are using the rust sql parser from openlineage-sql to extract table lineage via sql statements. When errors occur we are adding an extractionError run facet similar to what is being done here. I’m finding in the case that multiple statements were extracted but one failed to parse while many others were successful, the lineage for these runs doesn’t appear as expected in Marquez. Is there any logic around the extractionError run facet that could be causing this? It seems reasonable to assume that we might take this to mean the entire run event is invalid if we have any extraction errors. +> +> I would still expect to see the other lineage we sent for the run but am instead just seeing the extractionError in the marquez UI, in the database, runs with an extractionError facet don’t seem to make it to the job_versions_io_mapping table

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/integration/airflow/openlineage/airflow/extractors/sql_extractor.py | sql_extractor.py>
+
+ <pre><code> if sql_meta.errors: + run_facets['extractionError'] = ExtractionErrorRunFacet( + totalTasks=len(self.operator.sql) if isinstance(self.operator.sql, list) else 1, + failedTasks=len(sql_meta.errors), + errors=[ + ExtractionError( + errorMessage=error.message, + stackTrace=None, + task=error.origin_statement, + taskNumber=error.index + ) + for error + in sql_meta.errors + ] + ) +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-10-31 06:34:05
 +
+

*Thread Reply:* Can you show the actual event? Should be in the events tab in Marquez

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kavitha + (kkandaswamy@cardinalcommerce.com) +
+
2023-10-31 11:59:07
 +
+

*Thread Reply:* @John Lukenoff, would you mind posting the link to Marquez teams slack channel?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Lukenoff + (john@jlukenoff.com) +
+
2023-10-31 12:15:37
 +
+

*Thread Reply:* yep here is the link: https://marquezproject.slack.com/archives/C01E8MQGJP7/p1698702140709439

+ +

This is the full event, sanitized of internal info: +{ + "job": { + "name": "some_dag.some_task", + "facets": {}, + "namespace": "default" + }, + "run": { + "runId": "a9565df2-f1a1-3ee3-b202-7626f8c4b92d", + "facets": { + "extractionError": { + "errors": [ + { + "task": "ALTER SESSION UNSET QUERY_TAG;", + "_producer": "<https://github.com/OpenLineage/OpenLineage/tree/0.24.0/client/python>", + "_schemaURL": "<https://raw.githubusercontent.com/OpenLineage/OpenLineage/main/spec/OpenLineage.json#/definitions/BaseFacet>", + "taskNumber": 0, + "errorMessage": "Expected one of TABLE or INDEX, found: SESSION" + } + ], + "_producer": "<https://github.com/OpenLineage/OpenLineage/tree/0.24.0/client/python>", + "_schemaURL": "<https://raw.githubusercontent.com/OpenLineage/OpenLineage/main/spec/OpenLineage.json#/definitions/ExtractionErrorRunFacet>", + "totalTasks": 1, + "failedTasks": 1 + } + } + }, + "inputs": [ + { + "name": "foo.bar", + "facets": {}, + "namespace": "snowflake" + }, + { + "name": "fizz.buzz", + "facets": {}, + "namespace": "snowflake" + } + ], + "outputs": [ + { "name": "foo1.bar2", "facets": {}, "namespace": "snowflake" }, + { + "name": "fizz1.buzz2", + "facets": {}, + "namespace": "snowflake" + } + ], + "producer": "<https://github.com/MyCompany/repo/blob/next-master/company/data/pipelines/airflow_utils/openlineage_utils/client.py>", + "eventTime": "2023-10-30T02:46:13.367274Z", + "eventType": "COMPLETE" +}

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kavitha + (kkandaswamy@cardinalcommerce.com) +
+
2023-10-31 12:43:07
 +
+

*Thread Reply:* thank you!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kavitha + (kkandaswamy@cardinalcommerce.com) +
+
2023-10-31 13:14:29
 +
+

*Thread Reply:* @John Lukenoff, sorry to trouble again, is the slack channel still active? for whatever reason i cant get to this workspace

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Lukenoff + (john@jlukenoff.com) +
+
2023-10-31 13:15:26
 +
+

*Thread Reply:* yep it’s still active, maybe you need to join the workspace first? https://join.slack.com/t/marquezproject/shared_invite/zt-266fdhg9g-TE7e0p~EHK50GJMMqNH4tg

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kavitha + (kkandaswamy@cardinalcommerce.com) +
+
2023-10-31 13:25:51
 +
+

*Thread Reply:* that was a good call. the link you just shared worked! thank you!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-10-31 13:27:55
 +
+

*Thread Reply:* yeah from OL perspective this looks good - the inputs and outputs are there, the extraction error facet looks like it should

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-10-31 13:28:05
 +
+

*Thread Reply:* must be some Marquez hiccup 🙂

+ + + +
+ 👍 John Lukenoff +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Lukenoff + (john@jlukenoff.com) +
+
2023-10-31 13:28:45
 +
+

*Thread Reply:* Makes sense, I’ll tail my marquez logs today to see if I can find anything

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Lukenoff + (john@jlukenoff.com) +
+
2023-11-01 19:37:06
 +
+

*Thread Reply:* Somehow this started working after we switched from our beta to prod infrastructure. I suspect something was failing due to constraints on the size of our db and the load of poor quality data it was under after months of testing against it

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-11-01 11:34:43
 +
+

@channel +I’m opening a vote to release OpenLineage 1.5.0, including: +• support for Cassandra Connectors lineage in the Flink integration +• support for Databricks Runtime 13.3 in the Spark integration +• support for rdd and toDF operations from the Spark Scala API in Spark +• lowered requirements for attrs and requests packages in the Airflow integration +• lazy rendering of yaml configs in the dbt integration +• bug fixes, tests, infra fixes, doc changes, and more. +Three +1s from committers will authorize an immediate release.

+ + + +
+ ➕ Jakub Dardziński, William Angel, Abdallah, Willy Lulciuc, Paweł Leszczyński, Julien Le Dem +
+ +
+ 👍 Jason Yip +
+ +
+ 🚀 Luca Soato, tati +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-11-02 05:11:58
 +
+

*Thread Reply:* Thanks, all. The release is authorized and will be initiated within 2 business days.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-11-01 13:29:09
 +
+

@channel +The October 2023 issue of OpenLineage News is available now! to get in directly in your inbox each month.

+
+
apache.us14.list-manage.com
+ + + +
OpenLineage Project
+
+ OpenLineage Project Email Forms +
+ + + + + + +
Original URL: https://openlineage.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&amp;id=123767f606&amp;e=ef0563a7f8
+ + + +
+ + + +
+ 👍 Mars Lan, harsh loomba +
+ +
+ 🎉 tati +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Lukenoff + (john@jlukenoff.com) +
+
2023-11-01 19:40:39
 +
+

Hi team 👋 , we’re finding that for our Spark jobs we are almost always getting some junk characters in our dataset names. We’ve pushed the regex filter to its limits and would like to extend the logic of deriving the dataset name in openlineage-spark (currently on 1.4.1). I seem to recall hearing we could do this by implementing our own LogicalPlanVisitor or something along those lines? Is that still the recommended approach and if so would this be possible to implement in Scala vs. Java (scala noob here 🙂)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-11-02 03:34:15
 +
+

*Thread Reply:* Hi John, we're always happy to help with the contribution.

+ +

One of the possible solutions to this would be to do that just in openlineage-java client: +• introduce config entry like normalizeDatasetNameToAscii : enabled/disabled +• modify DatasetIdentifier class to contain static member boolean normalizeDatasetNameToAscii and normalize dataset name according to this setting +• additionally, you would need to add config entry in io.openlineage.client.OpenLineageYaml and make sure both loadOpenLineageYaml methods set DatasetIdentifier.normalizeDatasetNameToAscii based on the config +• document this in the doc +So, no Scala nor custom logical plan visitors required.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-11-02 03:34:47
 +
+

*Thread Reply:* https://github.com/OpenLineage/OpenLineage/blob/main/client/java/src/main/java/io/openlineage/client/utils/DatasetIdentifier.java

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/client/java/src/main/java/io/openlineage/client/utils/DatasetIdentifier.java | DatasetIdentifier.java>
+
+ <pre><code>/** +/** Copyright 2018-2023 contributors to the OpenLineage project +/** SPDX-License-Identifier: Apache-2.0 +**/ +package io.openlineage.client.utils; + +import java.util.LinkedList; +import java.util.List; +import lombok.Value; + +@Value +public class DatasetIdentifier { + String name; + String namespace; + List&lt;Symlink&gt; symlinks; + + public enum SymlinkType { + TABLE + }; + + public DatasetIdentifier(String name, String namespace) { + this.name = name; + this.namespace = namespace; + this.symlinks = new LinkedList&lt;&gt;(); + } + + public DatasetIdentifier withSymlink(String name, String namespace, SymlinkType type) { + symlinks.add(new Symlink(name, namespace, type)); + return this; + } + + public DatasetIdentifier withSymlink(Symlink symlink) { + symlinks.add(symlink); + return this; + } + + @Value + public static class Symlink { + String name; + String namespace; + SymlinkType type; + } +} + +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+ 🙌 John Lukenoff +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mike Fang + (fangmik@amazon.com) +
+
2023-11-01 20:30:38
 +
+

I am looking to send OpenLineage events to an AWS API Gateway endpoint from an AWS MWAA instance. The problem is that all requests to AWS services need to be signed with SigV4, and using API Gateway with IAM authentication would require requests to API Gateway be signed with SigV4. Would the best way to do so be to just modify the python client HTTP transport to include a new config option for signing emitted OpenLineage events with SigV4? Are there any alternatives?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-11-02 02:41:50
 +
+

*Thread Reply:* there’s actually an issue for that: +https://github.com/OpenLineage/OpenLineage/issues/2189

+ +

but the way to do this is imho to create new custom transport (it might inherit from HTTP transport) and register it in transport factory

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mike Fang + (fangmik@amazon.com) +
+
2023-11-02 13:05:05
 +
+

*Thread Reply:* I am thinking of just modifying the HTTP transport and using requests.auth.AuthBase to create different auth methods instead of a TokenProvider class

+ +

Classes which subclass requests.auth.AuthBase can also just directly be given to the requests call in the auth parameter

+ + + +
+ 👍 Jakub Dardziński +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-11-02 14:40:24
 +
+

*Thread Reply:* would you like to contribute? 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mike Fang + (fangmik@amazon.com) +
+
2023-11-02 14:43:05
 +
+

*Thread Reply:* I was about to contribute, but I actually just realized that there is an existing way to provide a custom transport that would solve form y use case. My only question is how do I register this custom transport in my MWAA environment? Can I provide the custom transport as an Airflow plugin and then specify the class in the Openlineage.yml config? Will it automatically pick it up?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-11-02 15:45:56
 +
+

*Thread Reply:* although I did not test this in MWAA but locally only: I’ve created Airflow plugin that in __init__.py has defined (or imported) following code: +```from openlineage.client.transport import register_transport, Transport, Config

+ +

@register_transport +class FakeTransport(Transport): + kind = "fake" + config = Config

+ +
def __init__(self, config: Config) -> None:
+    print(config)
+
+def emit(self, event) -> None:
+    print(event)```
+
+ +

setting AIRFLOW__OPENLINEAGE__TRANSPORT='{"type": "fake"}' does take effect and I can see output in Airflow logs

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-11-02 15:47:45
 +
+

*Thread Reply:* in setup.py it’s: +..., + entry_points={ + 'airflow.plugins': [ + 'custom_transport = custom_transport:CustomTransportPlugin', + ], + }, + install_requires=["openlineage-python"] +)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mike Fang + (fangmik@amazon.com) +
+
2023-11-03 12:52:55
 +
+

*Thread Reply:* ok great thanks for following up on this, super helpful

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-11-02 12:00:00
 +
+

@channel +We released OpenLineage 1.5.0, including: +• support for Cassandra Connectors lineage in the Flink integration by @Peter Huang +• support for Databricks Runtime 13.3 in the Spark integration by @Paweł Leszczyński +• support for rdd and toDF operations from the Spark Scala API in Spark by @Paweł Leszczyński +• lowered requirements for attrs and requests packages in the Airflow integration by @Jakub Dardziński +• lazy rendering of yaml configs in the dbt integration by @Jakub Dardziński +• bug fixes, tests, infra fixes, doc changes, and more. +Thanks to all the contributors, including new contributor @Sophie LY! +Release: https://github.com/OpenLineage/OpenLineage/releases/tag/1.5.0 +Changelog: https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md +Commit history: https://github.com/OpenLineage/OpenLineage/compare/1.4.1...1.5.0 +Maven: https://oss.sonatype.org/#nexus-search;quick~openlineage +PyPI: https://pypi.org/project/openlineage-python/

+ + + +
+ 👍 Jason Yip, Sophie LY, Tristan GUEZENNEC -CROIX-, Mars Lan, Sangeeta Mishra +
+ +
+ 🚀 tati +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason Yip + (jasonyip@gmail.com) +
+
2023-11-02 14:49:18
 +
+

@Paweł Leszczyński I tested 1.5.0, it works great now, but the environment facets is gone in START... which I very much want it.. any thoughts?

+ +
+
data (10).json
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason Yip + (jasonyip@gmail.com) +
+
2023-11-03 04:18:11
 +
+

actually, it shows up in one of the RUNNING now... behavior is consistent between 11.3 and 13.3, thanks for fixing this issue

+ + + +
+ 👍 Paweł Leszczyński +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason Yip + (jasonyip@gmail.com) +
+
2023-11-04 15:44:22
 +
+

*Thread Reply:* @Paweł Leszczyński looks like I need to bring bad news.. 13.3 is fixed for specific scenarios, but 11.3 is still reading output as dbfs.. there are scenarios that it's not producing input and output like:

+ +

create table table using delta as +location 'abfss://....' +Select ** from parquet.`abfss://....'

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason Yip + (jasonyip@gmail.com) +
+
2023-11-04 15:44:31
 +
+

*Thread Reply:* Will test more and ope issues

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Rodrigo Maia + (rodrigo.maia@manta.io) +
+
2023-11-06 05:34:33
 +
+

*Thread Reply:* @Jason Yiphow did you manage the get the environment attribute. it's not showing up to me at all. I've tried databricks abut also tried a local instance of spark.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason Yip + (jasonyip@gmail.com) +
+
2023-11-07 18:32:02
 +
+

*Thread Reply:* @Rodrigo Maia its showing up in one of the RUNNING events, not in the START event anymore

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Rodrigo Maia + (rodrigo.maia@manta.io) +
+
2023-11-08 03:04:32
 +
+

*Thread Reply:* I never had a running event 🫠 Am I filtering something?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason Yip + (jasonyip@gmail.com) +
+
2023-11-08 13:03:26
 +
+

*Thread Reply:* Umm.. ok show me your code, will try on my end

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason Yip + (jasonyip@gmail.com) +
+
2023-11-08 14:26:06
 +
+

*Thread Reply:* @Paweł Leszczyński @Rodrigo Maia actually if you are using UC-enabled cluster, you won't get any RUNNING events

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-11-03 12:00:07
 +
+

@channel +This month’s TSC meeting (open to all) is next Thursday the 9th at 10am PT. On the agenda: +• announcements +• recent releases +• recent additions to the Flink integration by @Peter Huang +• recent additions to the Spark integration by @Paweł Leszczyński +• updates on proposals by @Julien Le Dem +• discussion topics +• open discussion +More info and the meeting link can be found on the website. All are welcome! Do you have a discussion topic, use case or integration you’d like to demo? DM me to be added to the agenda.

+
+
openlineage.io
+ + + +
TSC Meetings | OpenLineage
+
+ The OpenLineage Technical Steering Committee meets monthly, and is open to all. +
+ + + + + + +
Original URL: https://openlineage.io/meetings/
+ + + +
+ + + +
+ 👍 harsh loomba +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
priya narayana + (n.priya88@gmail.com) +
+
2023-11-04 07:08:10
 +
+

Hi Team , we are trying to customize the events by writing custom lineage listener extending OpenLineageSparkListener, but would need some direction how to capture the events

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-11-04 07:11:46
 +
+

*Thread Reply:* https://openlineage.slack.com/archives/C01CK9T7HKR/p1698315220142929 +Do you need some more guidance than that?

+
+ + +
+ + + } + + priya narayana + (https://openlineage.slack.com/team/U062Q95A1FG) +
+ + + +
+
+ <p>Hi I want to customise the events which comes from Openlineage spark . Can some one give some information</p> +
+ + + + + + +
Original URL: https://openlineage.slack.com/archives/C01CK9T7HKR/p1698315220142929
+ + +
+ + Thread in Slack Conversation +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
priya narayana + (n.priya88@gmail.com) +
+
2023-11-04 07:13:47
 +
+

*Thread Reply:* yes

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-11-04 07:15:21
 +
+

*Thread Reply:* It seems pretty extensively described, what kind of help do you need?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
priya narayana + (n.priya88@gmail.com) +
+
2023-11-04 07:16:13
 +
+

*Thread Reply:* io.openlineage.spark.api.OpenLineageEventHandlerFactory if i use this how will i pass custom listener to my spark submit

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
priya narayana + (n.priya88@gmail.com) +
+
2023-11-04 07:17:25
 +
+

*Thread Reply:* I would like to know how will i customize my events using this . For example: - In "input" Facet i want only symlinks name i am not intereseted in anything else

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
priya narayana + (n.priya88@gmail.com) +
+
2023-11-04 07:17:32
 +
+

*Thread Reply:* can you please provide some guidance

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
priya narayana + (n.priya88@gmail.com) +
+
2023-11-04 07:18:36
 +
+

*Thread Reply:* @Jakub Dardziński this is the doubt i have

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
priya narayana + (n.priya88@gmail.com) +
+
2023-11-04 08:17:25
 +
+

*Thread Reply:* Some one who did spark integration throw some light

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-11-04 08:21:22
 +
+

*Thread Reply:* it's weekend for most of us so you probably need to wait until Monday for precise answers

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
David Goss + (david.goss@matillion.com) +
+
2023-11-06 04:03:42
 +
+

👋 I raised a PR https://github.com/OpenLineage/OpenLineage/pull/2223 off the back of some Marquez conversations a while back to try and clarify how names of Snowflake objects should be expressed in OL events. I used Snowflake’s OL view as a guide, but also I appreciate there are other OL producers that involve Snowflake too (Airflow? dbt?). Any feedback on this would be appreciated!

+
+ + + + +
#2223 spec: add clarity to snowflake naming docs
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+
+ + + + +
Snowflake-Labs/OpenLineage-AccessHistory-Setup
+
+ +
+ +
+
Stars
+ 11 +
+ +
+
Last updated
+ 3 months ago +
+ + + + + + + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
David Goss + (david.goss@matillion.com) +
+
2023-11-08 10:42:35
 +
+

*Thread Reply:* Thanks for merging this @Maciej Obuchowski!

+ + + +
+ 👍 Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Athitya Kumar + (athityakumar@gmail.com) +
+
2023-11-06 05:22:03
 +
+

Hey team! 👋

+ +

We're trying to use openlineage-flink, and would like provide the openlineage.transport.type=http and configure other transport configs, but we're not able to find sufficient docs (tried this doc) on where/how these configs can be provided.

+ +

For example, in spark, the changes mostly were delegated to the spark-submit command like +spark-submit --conf "spark.extraListeners=io.openlineage.spark.agent.OpenLineageSparkListener" \ + --packages "io.openlineage:openlineage_spark:&lt;spark-openlineage-version&gt;" \ + --conf "spark.openlineage.transport.url=http://{openlineage.client.host}/api/v1/namespaces/spark_integration/" \ + --class com.mycompany.MySparkApp my_application.jar +And the OpenLineageSparkListener has a method to retrieve the provided spark confs as an object in the ArgumentParser. Similarly, looking for some pointers on how the openlineage.transport configs can be provided to OpenLineageFlinkJobListener & how the flink listener parses/uses these configs

+ +

TIA! 😄

+
+
openlineage.io
+ + + +
Apache Flink | OpenLineage
+
+ This integration is considered experimental: only specific workflows and use cases are supported. +
+ + + + + + +
Original URL: https://openlineage.io/docs/integrations/flink
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-11-07 05:56:09
 +
+

*Thread Reply:* similarly to spark config, you can use flink config

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Athitya Kumar + (athityakumar@gmail.com) +
+
2023-11-07 22:36:53
 +
+

*Thread Reply:* @Maciej Obuchowski - Got it. Our use-case is that we're trying to build a wrapper on top of openlineage-flink for productionising for our flink jobs.

+ +

We're trying to have a wrapper class that extends OpenLineageFlinkJobListener class, and overwrites the HTTP transport endpoint/url to a constant value (say, example.com and /api/v1/flink). But we see that the OpenLineageFlinkJobListener constructor is defined as a private constructor - just wanted to check with the team whether it was just a default scope, or intended to be private. If it was just a default scope, can we contribute a PR to make it public, to make it friendly for teams trying to adopt & extend openlineage?

+ +

And also, we wanted to understand better on where we're reading the HTTP transport endpoint/url configs in OpenLineageFlinkJobListener and what'd be the best place to override it to the constant endpoint/url for our use-case

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-11-08 05:55:43
 +
+

*Thread Reply:* We parse flink conf to get that information: https://github.com/OpenLineage/OpenLineage/blob/26494b596e9669d2ada164066a73c44e04[…]ink/src/main/java/io/openlineage/flink/client/EventEmitter.java

+ +

> But we see that the OpenLineageFlinkJobListener constructor is defined as a private constructor - just wanted to check with the team whether it was just a default scope, or intended to be private. +The way to construct is is a public builder in the same class

+ +

I think easier way than wrapper class would be use existing flink configuration, or to set up OPENLINEAGE_URL env variable, or have openlineage.yml config file - not sure why this is the way you've chosen?

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/26494b596e9669d2ada164066a73c44e04e884ba/integration/flink/src/main/java/io/openlineage/flink/client/EventEmitter.java | EventEmitter.java>
+
+ <pre><code> public EventEmitter(Configuration configuration) { +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Athitya Kumar + (athityakumar@gmail.com) +
+
2023-11-09 12:41:02
 +
+

*Thread Reply:* > I think easier way than wrapper class would be use existing flink configuration, or to set up OPENLINEAGE_URL env variable, or have openlineage.yml config file - not sure why this is the way you've chosen? +@Maciej Obuchowski - The reasoning behind going with a wrapper class is that we can abstract out the nitty-gritty like how/where we're publishing openlineage events etc - especially for companies that have a lot of teams that may be adopting openlineage.

+ +

For example, if we wanna move away from http transport to kafka transport - we'd be changing only this wrapper class and ask folks to update their wrapper class dependency version. If we went without the wrapper class, then the exact config changes would need to be synced and done by many different teams, who may not have enough context.

+ +

Similarly, if we wanna enable some other default best-practise configs, or inject any company-specific configs etc, the wrapper would be useful in abstracting out the details and be the 1 place that handles all openlineage related integrations for any future changes.

+ +

That's why we wanna extend openlineage's listener class & leverage most of the OSS code as-is; and at the same time, have the ability to extend & inject customisations. I think that's where some things like having getters for the class object attributes, or having public constructors would be really helpful 😄

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-11-09 13:03:56
 +
+

*Thread Reply:* @Athitya Kumar that makes sense. Feel free to provide PR adding getters and stuff.

+ + + +
+ 🎉 Athitya Kumar +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Yannick Libert + (yannick.libert.partner@decathlon.com) +
+
2023-11-07 06:03:49
 +
+

Hi all, we (I work with @Sophie LY and @Abdallah) have a quick question regarding the spark integration: +if a spark app contains several jobs, they will be named "mysparkappname.job1" and "mysparkappname.job2" +eg: +sparkjob.collectlimit +sparkjob.mappartitionsparallelcollection

+ +

If I understood correctly, the spark integration maps one Spark job to a single OpenLineage Job, and the application itself should be assigned a Run id at startup and each job that executes will report the application's Run id as its parent job run (taken from: https://openlineage.io/docs/integrations/spark/).

+ +

In our case, the app Run Id is never created, and the jobs runs don't contain any parent facets. We tested it with a recent integration version in 1.4.1 and also an older one (0.26.0). +Did we miss something in the OL spark integration config?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-11-07 06:07:51
 +
+

*Thread Reply:* hey, a name of the output dataset should be put at the end of the job name. This was introduced to help with jobs that call multiple spark actions

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Yannick Libert + (yannick.libert.partner@decathlon.com) +
+
2023-11-07 07:05:52
 +
+

*Thread Reply:* Hi Paweł, +Thanks for your answer, yes indeed with the newer version of OL, we automatically have the name of the output dataset at the end of the job name, but no App run id, nor any parent run facet.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-11-07 08:16:44
 +
+

*Thread Reply:* yes, you're right. I mean you can set in config spark.openlineage.parentJobName which will be shared through whole app run, but this needs to be set manually

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Yannick Libert + (yannick.libert.partner@decathlon.com) +
+
2023-11-07 08:36:58
 +
+

*Thread Reply:* I see, thanks a lot for your reply we'll try that

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ldacey + (lance.dacey2@sutherlandglobal.com) +
+
2023-11-07 10:49:25
 +
+

if I have a dataset on adls gen2 which synapse connects to as an external delta table, is that the use case of a symlink dataset? the delta table is connected to by PBI and by Synapse, but the underlying data is exactly the same

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-11-08 10:49:04
 +
+

*Thread Reply:* Sounds like it, yes - if the logical dataset names are different but physical one is the same

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Rodrigo Maia + (rodrigo.maia@manta.io) +
+
2023-11-08 12:38:52
 +
+

Has anyone here tried OpenLineage with Spark on Amazon EMR?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason Yip + (jasonyip@gmail.com) +
+
2023-11-08 13:01:16
 +
+

*Thread Reply:* No but it should work the same I tried on AWS and Google Colab and Azure

+ + + +
+ 👍 Jakub Dardziński +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tristan GUEZENNEC -CROIX- + (tristan.guezennec@decathlon.com) +
+
2023-11-09 03:10:54
 +
+

*Thread Reply:* Yes. @Abdallah could provide some details if needed.

+ + + +
+ 👍 Abdallah +
+ +
+ 🔥 Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Rodrigo Maia + (rodrigo.maia@manta.io) +
+
2023-11-20 11:29:26
 +
+

*Thread Reply:* Thanks @Tristan GUEZENNEC -CROIX- +HI @Abdallah i was able to set up a spark cluster on AWS EMR but im struggling to configure the OL Listener. Ive tried with steps and bootstrap actions for the jar and it didn't work out. How did you manage to include the jar? Besides, what about the spark configuration? Could you send me a sample of these configs?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-11-08 12:44:54
 +
+

@channel +Friendly reminder: this month’s TSC meeting, open to all, is tomorrow at 10 am PT: https://openlineage.slack.com/archives/C01CK9T7HKR/p1699027207361229

+
+ + +
+ + + } + + Michael Robinson + (https://openlineage.slack.com/team/U02LXF3HUN7) +
+ + + +
+
+ <p>@channel +This month’s TSC meeting (open to all) is next Thursday the 9th at 10am PT. On the agenda: +• announcements +• recent releases +• recent additions to the Flink integration by @Peter Huang +• recent additions to the Spark integration by @Paweł Leszczyński +• updates on proposals by @Julien Le Dem +• discussion topics +• open discussion +More info and the meeting link can be found on the <a href="https://openlineage.io/meetings/">website</a>. All are welcome! Do you have a discussion topic, use case or integration you’d like to demo? DM me to be added to the agenda.</p> +
+ + + + + + +
Original URL: https://openlineage.slack.com/archives/C01CK9T7HKR/p1699027207361229
+ + +
+ + Slack Conversation +
+ + +
+ + + +
+ 👍 Jakub Dardziński +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason Yip + (jasonyip@gmail.com) +
+
2023-11-10 15:25:45
 +
+

@Paweł Leszczyński regarding to https://github.com/OpenLineage/OpenLineage/issues/2124, OL is parsing out the table location in Hive metastore, it is the location of the table in the catalog and not the physical location of the data. It is both right and wrong because it is a table, just it is an external table.

+ +

https://docs.databricks.com/en/sql/language-manual/sql-ref-external-tables.html

+
+
docs.databricks.com
+ + + +
External tables
+
+ Learn about Unity Catalog external tables in Databricks SQL and Databricks Runtime. +
+ + + + + + + + +
Original URL: https://docs.databricks.com/en/sql/language-manual/sql-ref-external-tables.html
+ + + +
+
+ + + + +
#2124 Same Delta Table not catching the location on write
+
+ <p><strong>What is the target system?</strong></p> + +<p>Spark / Databricks</p> + +<p><strong>What kind of integration is this?</strong></p> + +<p>☐ Produces OpenLineage metadata +☐ Consumes OpenLineage metadata +☐ Something else</p> + +<p><strong>How should this integration be implemented?</strong></p> + +<p>I am using OL 1.2.2, Azure Databricks Runtime 11.3 LTS. When creating a table writing into a ADLS location, OL won't be able to catch the location of the output. But when I read the same object it will be able to read the location as INPUT.</p> + +<p>Please note I have also tested Databricks Runtime 13.3 LTS, Spark 3.4.1 - it will give correct ADLS location in INPUT but the input will only show up once in a blue moon. Most of the time the inputs and outputs are blank.</p> + +<pre><code> "inputs": [], + "outputs": [] +</code></pre> + +<pre><code>CREATE OR REPLACE TABLE transactions_adj +USING DELTA LOCATION '&lt;wasbs://studio@clororetaildevadls.blob.core.windows.net/examples/data/csv/completejourney/silver/transactions_adj&gt;' +AS + SELECT + household_id, + basket_id, + week_no, + day, + transaction_time, + store_id, + product_id, + amount_list, + campaign_coupon_discount, + manuf_coupon_discount, + manuf_coupon_match_discount, + total_coupon_discount, + instore_discount, + amount_paid, + units + FROM ( + SELECT + household_id, + basket_id, + week_no, + day, + transaction_time, + store_id, + product_id, + COALESCE(sales_amount - discount_amount - coupon_discount - coupon_discount_match,0.0) as amount_list, + CASE + WHEN COALESCE(coupon_discount_match,0.0) = 0.0 THEN -1 ** COALESCE(coupon_discount,0.0) + ELSE 0.0 + END as campaign_coupon_discount, + CASE + WHEN COALESCE(coupon_discount_match,0.0) != 0.0 THEN -1 ** COALESCE(coupon_discount,0.0) + ELSE 0.0 + END as manuf_coupon_discount, + -1 ** COALESCE(coupon_discount_match,0.0) as manuf_coupon_match_discount, + -1 ** COALESCE(coupon_discount - coupon_discount_match,0.0) as total_coupon_discount, + COALESCE(-1 ** discount_amount,0.0) as instore_discount, + COALESCE(sales_amount,0.0) as `amount_paid,` + quantity as units + FROM transactions + ); +</code></pre> + +<p>Here's the COMPLETE event:</p> + +<pre><code> + "outputs":[ + { + "namespace":"dbfs", + "name":"/user/hive/warehouse/journey.db/transactions_adj", + "facets":{ + "dataSource":{ + "_producer":"&lt;https://github.com/OpenLineage/OpenLineage/tree/1.2.2/integration/spark&gt;", + "_schemaURL":"&lt;https://openlineage.io/spec/facets/1-0-0/DatasourceDatasetFacet.json#/$defs/DatasourceDatasetFacet&gt;", + "name":"dbfs", + "uri":"dbfs" + }, + +</code></pre> + +<p>Below logical plan shows the path:</p> + +<pre><code>== Analyzed Logical Plan == +num_affected_rows: bigint, num_inserted_rows: bigint +ReplaceTableAsSelect TableSpec(Map(),Some(DELTA),Map(),Some(&lt;wasbs://studio@clororetaildevadls.blob.core.windows.net/examples/data/csv/completejourney/silver/transactions_adj&gt;),None,None,false,Set()), true +:- ResolvedIdentifier com.databricks.sql.managedcatalog.UnityCatalogV2Proxy@6251a8df, default.transactions_adj ++- Project [household_id#184, basket_id#185L, week_no#193, day#186, transaction_time#192, store_id#190, product_id#187, amount_list#147, campaign_coupon_discount#148, manuf_coupon_discount#149, manuf_coupon_match_discount#150, total_coupon_discount#151, instore_discount#152, amount_paid#153, units#154] + +- SubqueryAlias __auto_generated_subquery_name + +- Project [household_id#184, basket_id#185L, week_no#193, day#186, transaction_time#192, store_id#190, product_id#187, coalesce(cast((((sales_amount#189 - discount_amount#191) - coupon_discount#194) - coupon_discount_match#195) as double), cast(0.0 as double)) AS amount_list#147, CASE WHEN (coalesce(cast(coupon_discount_match#195 as double), cast(0.0 as double)) = cast(0.0 as double)) THEN (cast(-1 as double) ** coalesce(cast(coupon_discount#194 as double), cast(0.0 as double))) ELSE cast(0.0 as double) END AS campaign_coupon_discount#148, CASE WHEN NOT (coalesce(cast(coupon_discount_match#195 as double), cast(0.0 as double)) = cast(0.0 as double)) THEN (cast(-1 as double) ** coalesce(cast(coupon_discount#194 as double), cast(0.0 as double))) ELSE cast(0.0 as double) END AS manuf_coupon_discount#149, (cast(-1 as double) ** coalesce(cast(coupon_discount_match#195 as double), cast(0.0 as double))) AS manuf_coupon_match_discount#150, (cast(-1 as double) ** coalesce(cast((coupon_discount#194 - coupon_discount_match#195) as double), cast(0.0 as double))) AS total_coupon_discount#151, coalesce(cast((cast(-1 as float) ** discount_amount#191) as double), cast(0.0 as double)) AS instore_discount#152, coalesce(cast(sales_amount#189 as double), cast(0.0 as double)) AS amount_paid#153, quantity#188 AS units#154] + +- SubqueryAlias spark_catalog.default.transactions + +- Relation spark_catalog.default.transactions[household_id#184,basket_id#185L,day#186,product_id#187,quantity#188,sales_amount#189,store_id#190,discount_amount#191,transaction_time#192,week_no#193,coupon_discount#194,coupon_discount_match#195] parquet +</code></pre> + +<p><strong>Where should this integration be implemented?</strong></p> + +<p>☐ In the target system +☐ In the OpenLineage repo +☐ Somewhere else</p> + +<p><strong>Do you plan to make this contribution yourself?</strong></p> + +<p>☐ I am interested in doing this work</p> +
+ +
+
Labels
+ integration/spark, integration/databricks +
+ +
+
Comments
+ 5 +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason Yip + (jasonyip@gmail.com) +
+
2023-11-10 15:32:28
 +
+

*Thread Reply:* Here's for more reference: https://dilorom.medium.com/finding-the-path-to-a-table-in-databricks-2c74c6009dbb

+
+
Medium
+ + + +
Finding the Path to a Managed Table in Databricks
+
+ This article shows how to find a path for a managed Databricks table. +
+ +
+
Reading time
+ 2 min read +
+ + + + + + + + +
Original URL: https://dilorom.medium.com/finding-the-path-to-a-table-in-databricks-2c74c6009dbb
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason Yip + (jasonyip@gmail.com) +
+
2023-11-11 03:29:33
 +
+

@Paweł Leszczyński this is why if create a table with adls location it won't show input and output:

+ +

https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/spark35/src[…]k35/agent/lifecycle/plan/CreateReplaceOutputDatasetBuilder.java

+ +

Because the catalog object is not there.

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/spark35/src/main/java/io/openlineage/spark35/agent/lifecycle/plan/CreateReplaceOutputDatasetBuilder.java | CreateReplaceOutputDatasetBuilder.java>
+
+ <pre><code> if (!di.isPresent()) { + return Collections.emptyList(); + } +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason Yip + (jasonyip@gmail.com) +
+
2023-11-11 03:30:44
 +
+

Databricks needs to be re-written in a way that supports Databricks it seems like

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason Yip + (jasonyip@gmail.com) +
+
2023-11-13 03:00:42
 +
+

@Paweł Leszczyński I went back to 1.4.1, output does show adls location. But environment facet is gone in 1.4.1. It shows up in 1.5.0 but namespace is back to dbfs....

+ +
+
data (11).json
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason Yip + (jasonyip@gmail.com) +
+
2023-11-13 03:18:37
 +
+

@Paweł Leszczyński I diff CreateReplaceDatasetBuilder.java and CreateReplaceOutputDatasetBuilder.java and they are the same except for the class name, so I am not sure what is causing the change. I also realize you don't have a test case for ADLS

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-11-13 04:52:07
 +
+

*Thread Reply:* Thanks @Jason Yip for your engagement in finding the cause and solution to this issue.

+ +

Among the technical problems, another problem here is that our databricks integration tests are run on AWS and the issue you describe occurs in Azure. I would consider this a primary issue as it is difficult for me to verify the behaviour you describe and fix it with a failing integration test at the start.

+ +

Are you able to reproduce the issue on AWS Databricks environment so that we could include it in our integration tests and make sure the behvaiour will not change later on in future?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason Yip + (jasonyip@gmail.com) +
+
2023-11-13 18:06:44
 +
+

*Thread Reply:* I didn't know Azure and AWS Databricks are different. Let me try it on AWS as well

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Naresh reddy + (naresh.naresh36@gmail.com) +
+
2023-11-15 07:17:24
 +
+

Hi +Can anyone point me to the deck on how Airflow can be integrated using Openlineage?

+ + + +
+
+
+
+ + + + + +
+ +
+ + + + + +
+
+ + + + +
+ +
Naresh reddy + (naresh.naresh36@gmail.com) +
+
2023-11-15 07:27:55
 +
+

*Thread Reply:* thank you @Maciej Obuchowski

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Naresh reddy + (naresh.naresh36@gmail.com) +
+
2023-11-15 11:09:24
 +
+

Can anyone tell me why OL is better than other competitors if you can provide an analysis that would be great

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harel Shein + (harel.shein@gmail.com) +
+
2023-11-16 11:46:16
 +
+

*Thread Reply:* Hey @Naresh reddy can you help me understand what you mean by competitors? +OL is a specification that can be used to solve various problems, so if you have a clear problem statement, maybe I can help with pros/cons for that problem

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Naresh reddy + (naresh.naresh36@gmail.com) +
+
2023-11-15 11:10:58
 +
+

what are the pros and cons of OL. we often talk about positives to market it but what are the pain points using OL,how it's addressing user issues?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-11-16 13:38:42
 +
+

*Thread Reply:* Hi @Naresh reddy, thanks for your question. We’ve heard that OpenLineage is attractive because of its desirable integrations, including a best-in-class Spark integration, its extensibility, the fact that it’s not destructive, and the fact that it’s open source. I’m not aware of pain points per se, but there are certainly features and integrations that we wish we could focus on but can’t at the moment — like the Dagster integration, which needs a new maintainer. OpenLineage is like any other open standard in that ecosystem coverage is a constant process rather than a journey, and it requires contributions in order to get close to 100%. Thankfully, we are gaining users and contributors all the time, and integrations are being added or improved upon daily. See the Ecosystem page on the website for a list of consumers and producers and links to more resources, and check out the GitHub repo for the codebase, commit history, contributors, governance procedures, and more. We’re quick to respond to messages here and issues on GitHub — usually within one day.

+
+ + + + +
OpenLineage/OpenLineage
+
+ An Open Standard for lineage metadata collection +
+ +
+
Website
+ <http://openlineage.io> +
+ +
+
Stars
+ 1449 +
+ + + + + + + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
karthik nandagiri + (karthik.nandagiri@gmail.com) +
+
2023-11-19 23:57:38
 +
+

Hi So we can use openlineage to identify column level lineage with Airflow , Spark? will it also allow to connect to Power BI and derive the downstream column lineage ?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-11-20 06:07:36
 +
+

*Thread Reply:* Yes, it works with Airflow and Spark - there is caveat that amount of operators that support it on Airflow side is fairly small and limited generally to most popular SQL operators. +> will it also allow to connect to Power BI and derive the downstream column lineage ? +No, there is no such feature yet 🙂 +However, there's nothing preventing this - if you wish to work on such implementation, we'd be happy to help.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
karthik nandagiri + (karthik.nandagiri@gmail.com) +
+
2023-11-21 00:20:11
 +
+

*Thread Reply:* Thank you Maciej Obuchowski for the update. Currently we are looking out for a tool which can support connecting to Power Bi and pull column level lineage information for reports and dashboards. How this can be achieved with OL ? Can you give some idea?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-11-21 07:59:10
 +
+

*Thread Reply:* I don't think I can help you with that now, unless you want to work on your own integration with PowerBI 🙁

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Rafał Wójcik + (rwojcik@griddynamics.com) +
+
2023-11-21 07:02:08
 +
+

Hi Everyone, first of all - big shout to all contributors - You do amazing job here. +I want to use OpenLineage in our project - to do so I want to setup some POC and experiment with possibilities library provides - I start working on sample from the conference talk: https://github.com/getindata/openlineage-bbuzz2023-column-lineage but when I go into spark transformation after staring context with openlineage I have issues with SessionHiveMetaStoreClient on section 3- does anyone has other plain sample to play with, to not setup everything from scratch?

+
+ + + + +
getindata/openlineage-bbuzz2023-column-lineage
+
+ +
+ +
+
Language
+ Jupyter Notebook +
+ +
+
Last updated
+ 5 months ago +
+ + + + + + + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-11-21 07:37:00
 +
+

*Thread Reply:* Can you provide details about those issues? Like exceptions, logs, details of the jobs and how do you run them?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Rafał Wójcik + (rwojcik@griddynamics.com) +
+
2023-11-21 07:45:37
 +
+

*Thread Reply:* Hi @Maciej Obuchowski - I rerun docker container after deleting metadata_db folder possibly created by other local test, and fix this one but got problem with OpenLineageListener - during initialization of spark: +while I execute: +spark = (SparkSession.builder.master('local') + .appName('Food Delivery') + .config('spark.extraListeners', 'io.openlineage.spark.agent.OpenLineageSparkListener') + .config('spark.jars', '&lt;local-path&gt;/openlineage-spark-0.27.2.jar,&lt;local-path&gt;/postgresql-42.6.0.jar') + .config('spark.openlineage.transport.type', 'http') + .config('spark.openlineage.transport.url', '<http://api:5000>') + .config('spark.openlineage.facets.disabled', '[spark_unknown;spark.logicalPlan]') + .config('spark.openlineage.namespace', 'food-delivery') + .config('spark.sql.warehouse.dir', '/tmp/spark-warehouse/') + .config("spark.sql.repl.eagerEval.enabled", True) + .enableHiveSupport() + .getOrCreate()) +I got +Py4JJavaError: An error occurred while calling None.org.apache.spark.api.java.JavaSparkContext. +: org.apache.spark.SparkException: Exception when registering SparkListener + at org.apache.spark.SparkContext.setupAndStartListenerBus(SparkContext.scala:2563) + at org.apache.spark.SparkContext.&lt;init&gt;(SparkContext.scala:643) + at org.apache.spark.api.java.JavaSparkContext.&lt;init&gt;(JavaSparkContext.scala:58) + at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) + at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77) + at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) + at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499) + at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:480) + at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:247) + at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357) + at py4j.Gateway.invoke(Gateway.java:238) + at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80) + at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69) + at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182) + at py4j.ClientServerConnection.run(ClientServerConnection.java:106) + at java.base/java.lang.Thread.run(Thread.java:833) +Caused by: java.lang.ClassNotFoundException: io.openlineage.spark.agent.OpenLineageSparkListener + at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:445) + at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:587) + at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:520) + at java.base/java.lang.Class.forName0(Native Method) + at java.base/java.lang.Class.forName(Class.java:467) + at org.apache.spark.util.Utils$.classForName(Utils.scala:218) + at org.apache.spark.util.Utils$.$anonfun$loadExtensions$1(Utils.scala:2921) + at scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:293) + at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62) + at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55) + at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49) + at scala.collection.TraversableLike.flatMap(TraversableLike.scala:293) + at scala.collection.TraversableLike.flatMap$(TraversableLike.scala:290) + at scala.collection.AbstractTraversable.flatMap(Traversable.scala:108) + at org.apache.spark.util.Utils$.loadExtensions(Utils.scala:2919) + at org.apache.spark.SparkContext.$anonfun$setupAndStartListenerBus$1(SparkContext.scala:2552) + at org.apache.spark.SparkContext.$anonfun$setupAndStartListenerBus$1$adapted(SparkContext.scala:2551) + at scala.Option.foreach(Option.scala:407) + at org.apache.spark.SparkContext.setupAndStartListenerBus(SparkContext.scala:2551) + ... 15 more +looks like by some reasons jars are not loaded - need to look into it

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-11-21 07:58:09
 +
+

*Thread Reply:* 🤔 Jars are added during image building: https://github.com/getindata/openlineage-bbuzz2023-column-lineage/blob/main/Dockerfile#L12C1-L12C29

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-11-21 07:58:28
 +
+

*Thread Reply:* are you sure &lt;local-path&gt; is right?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Rafał Wójcik + (rwojcik@griddynamics.com) +
+
2023-11-21 08:00:49
 +
+

*Thread Reply:* yes, it's same as in sample - wondering why it's not get added: +```from pyspark.sql import SparkSession

+ +

spark = (SparkSession.builder.master('local') + .appName('Food Delivery') + .config('spark.jars', '/home/jovyan/jars/openlineage-spark-0.27.2.jar,/home/jovyan/jars/postgresql-42.6.0.jar') + .config('spark.sql.warehouse.dir', '/tmp/spark-warehouse/') + .config("spark.sql.repl.eagerEval.enabled", True) + .enableHiveSupport() + .getOrCreate())

+ +

print(spark.sparkContext._jsc.sc().listJars())

+ +

Vector()```

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-11-21 08:04:31
 +
+

*Thread Reply:* can you make sure jars are in this directory? just by docker run --entrypoint /usr/local/bin/bash IMAGE_NAME "ls /home/jovyan/jars"

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-11-21 08:06:27
 +
+

*Thread Reply:* another option to try is to replace spark.jars with spark.jars.packages io.openlineage:openlineage_spark:1.5.0,org.postgresql:postgresql:42.7.0

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-11-21 08:16:54
 +
+

*Thread Reply:* I think this was done for the purpose of presentation to make sure the demo will work without internet access. This can be the reason to add jar manually to a docker. openlineage-spark can be added to Spark via spark.jar.packages , like we do here https://openlineage.io/docs/integrations/spark/quickstart_local

+
+
openlineage.io
+ + + +
Quickstart with Jupyter | OpenLineage
+
+ Trying out the Spark integration is super easy if you already have Docker Desktop and git installed. +
+ + + + + + +
Original URL: https://openlineage.io/docs/integrations/spark/quickstart_local
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Rafał Wójcik + (rwojcik@griddynamics.com) +
+
2023-11-21 09:21:59
 +
+

*Thread Reply:* got it guys - thanks a lot for help - it turns out that spark context from notebook 2 and 3 has come kind of metadata conflict - when I combine those 2 and recreate image to clean up old metadata it works. +One more note is that sometimes kernels return weird results but it may be caused by some local nuances - anyways thx !

+ + + +
+
+
+
+ + + + + + + + + + + \ No newline at end of file diff --git a/channel/gx-integration/index.html b/channel/gx-integration/index.html new file mode 100644 index 0000000..2b0e130 --- /dev/null +++ b/channel/gx-integration/index.html @@ -0,0 +1,598 @@ + + + + + + Slack Export - #gx-integration + + + + + +
+ + + +
+ + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-09-27 13:38:03
 +
+

@Michael Robinson has joined the channel

+ + + +
+ 🎉 Harel Shein +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Don Heppner + (don@greatexpectations.io) +
+
2023-09-27 13:38:23
 +
+

@Don Heppner has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harel Shein + (harel.shein@gmail.com) +
+
2023-09-27 13:38:23
 +
+

@Harel Shein has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-09-27 13:38:23
 +
+

@Jakub Dardziński has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-09-27 13:41:17
 +
+

@Maciej Obuchowski has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harel Shein + (harel.shein@gmail.com) +
+
2023-09-27 14:44:18
 +
+

@Don Heppner it was great meeting earlier, looking forward to collaborating on this!

+ + + +
+ ➕ Don Heppner, Jakub Dardziński +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Bill Dirks + (bill@greatexpectations.io) +
+
2023-09-28 11:59:31
 +
+

@Bill Dirks has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-10-09 07:46:52
 +
+

Hello guys! I’ve been looking recently into changes in GX. +https://greatexpectations.io/blog/the-fluent-way-to-connect-to-data-sources-in-gx/ +is this the major change you’d like to introduce in OL<-> GX?

+
+
greatexpectations.io
+ + + +
The fluent way to connect to data sources in GX
+
+ Creating a Datasource just got (much) easier +
+ + + + + + + + +
Original URL: https://greatexpectations.io/blog/the-fluent-way-to-connect-to-data-sources-in-gx/
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harel Shein + (harel.shein@gmail.com) +
+
2023-10-09 10:14:39
 +
+

@Don Heppner @Bill Dirks ^^

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Bill Dirks + (bill@greatexpectations.io) +
+
2023-10-10 19:12:02
 +
+

Just seeing this, we had a company holiday yesterday. Yes, fluent data sources are our new way of connecting to data and the older "block-style" is deprecated and will be removed when we cut 0.18.0. I'm not sure of the timing of that but likely in the next couple months.

+ + + +
+ 👍 Jakub Dardziński +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harel Shein + (harel.shein@gmail.com) +
+
2023-10-13 08:47:55
 +
+

@Bill Dirks would be great if we could get your eyes on this PR: https://github.com/OpenLineage/OpenLineage/pull/2134

+
+ + + + +
#2134 Support more recent versions of GX
+
+ <p>Limit upper GX version to &lt;0.16.0.</p> + +<p>Reopening <a href="https://github.com/OpenLineage/OpenLineage/pull/2024">#2024</a> accidentally closed by me.</p> + +<p>Kudos to <a href="https://github.com/ivanstillfront">@ivanstillfront</a>.</p> + +<p><strong>Checklist</strong></p> + +<p>☑︎ You've <a href="https://github.com/OpenLineage/OpenLineage/blob/main/why-the-dco.md">signed-off</a> your work +☑︎ Your pull request title follows our <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CONTRIBUTING.md#creating-pull-requests">guidelines</a> +☑︎ Your changes are accompanied by tests (<em>if relevant</em>) +☑︎ Your change contains a <a href="https://kurtisnusbaum.medium.com/stacked-diffs-keeping-phabricator-diffs-small-d9964f4dcfa6">small diff</a> and is self-contained +☐ You've updated any relevant documentation (<em>if relevant</em>) +☐ Your comment includes a one-liner for the changelog about the specific purpose of the change (<em>if necessary</em>) +☐ You've versioned the core OpenLineage model or facets according to <a href="https://docs.snowplowanalytics.com/docs/pipeline-components-and-applications/iglu/common-architecture/schemaver">SchemaVer</a> (<em>if relevant</em>) +☐ You've added a <a href="https://github.com/OpenLineage/OpenLineage/tree/main/.github/header_templates.md">header</a> to source files (<em>if relevant</em>)</p> + +<hr /> + +<p>SPDX-License-Identifier: Apache-2.0 <br /> +Copyright 2018-2023 contributors to the OpenLineage project</p> +
+ +
+
Labels
+ integration/great-expectations, common +
+ +
+
Comments
+ 6 +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Bill Dirks + (bill@greatexpectations.io) +
+
2023-10-13 15:00:37
 +
+

*Thread Reply:* I'm a bit slammed today but can look on Tuesday.

+ + + +
+ ✅ Harel Shein +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason + (sicotte.jason@gmail.com) +
+
2023-10-19 13:02:24
 +
+

@Jason has joined the channel

+ + + +
+
+
+
+ + + +
+
+ + + + \ No newline at end of file diff --git a/channel/london-meetup/index.html b/channel/london-meetup/index.html new file mode 100644 index 0000000..455ea1e --- /dev/null +++ b/channel/london-meetup/index.html @@ -0,0 +1,661 @@ + + + + + + Slack Export - #london-meetup + + + + + +
+ + + +
+ + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-08-25 11:52:05
 +
+

@Michael Robinson has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
George Polychronopoulos + (george.polychronopoulos@6point6.co.uk) +
+
2023-08-25 11:52:15
 +
+

@George Polychronopoulos has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harel Shein + (harel.shein@gmail.com) +
+
2023-08-25 11:52:15
 +
+

@Harel Shein has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Madhav Kakumani + (madhav.kakumani@6point6.co.uk) +
+
2023-08-25 11:52:40
 +
+

@Madhav Kakumani has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
George Polychronopoulos + (george.polychronopoulos@6point6.co.uk) +
+
2023-08-25 11:52:49
 +
+

Hi Michael

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
George Polychronopoulos + (george.polychronopoulos@6point6.co.uk) +
+
2023-08-25 11:53:36
 +
+

thanks so much !

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-08-25 11:53:54
 +
+

Hi George, nice to meet you. Thanks for asking about future meetups. Would November be too soon, or what’s a good timeframe for you all?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
George Polychronopoulos + (george.polychronopoulos@6point6.co.uk) +
+
2023-08-25 11:54:12
 +
+

thats perfect !

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-08-25 11:54:31
 +
+

Great! Do you happen to have space we could use?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
George Polychronopoulos + (george.polychronopoulos@6point6.co.uk) +
+
2023-08-25 11:55:58
 +
+

I will have to confirm but 99% yes

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
George Polychronopoulos + (george.polychronopoulos@6point6.co.uk) +
+
2023-08-25 11:57:47
 +
+

I am pretty sure you can use our 6point6 offices or at least part of it

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
George Polychronopoulos + (george.polychronopoulos@6point6.co.uk) +
+
2023-08-25 11:58:00
 +
+

and if that not the case i can provide personal space

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-08-25 11:58:46
 +
+

OK! Would you please let me know when you know, and we’ll go from there?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
George Polychronopoulos + (george.polychronopoulos@6point6.co.uk) +
+
2023-08-25 11:59:29
 +
+

yes absolutely will give you an answer by Monday

+ + + +
+ 👍 Michael Robinson +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Madhav Kakumani + (madhav.kakumani@6point6.co.uk) +
+
2023-08-25 12:51:08
 +
+

Thanks Michael for starting this channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Madhav Kakumani + (madhav.kakumani@6point6.co.uk) +
+
2023-08-25 12:51:21
 +
+

hopefully meet you soon in London

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-08-25 13:19:58
 +
+

Yes, hope so! Thank you for your interest in joining a meetup!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-25 13:34:21
 +
+

@Maciej Obuchowski has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mike O'Connor + (mike@entos.ai) +
+
2023-08-31 16:04:17
 +
+

@Mike O'Connor has joined the channel

+ + + +
+
+
+
+ + + +
+
+ + + + \ No newline at end of file diff --git a/channel/mark-grover/index.html b/channel/mark-grover/index.html new file mode 100644 index 0000000..2cc8c15 --- /dev/null +++ b/channel/mark-grover/index.html @@ -0,0 +1,944 @@ + + + + + + Slack Export - #mark-grover + + + + + +
+ + + +
+ + + +
+
+ +
+ + +
+ +
+ +
+
2021-01-26 13:47:55
 +
+

has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
slackbot + +
+
2021-01-26 23:53:17
 +
+

Stemma has joined this channel by invitation from Subsurface.

+ + + +
+
+
+
+ + + + + +
+
+ +
+ + +
+ +
+ +
+
2021-01-27 00:18:31
 +
+

has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ +
+ + +
+ +
+ +
+
2021-01-27 01:13:58
 +
+

has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ +
+ + +
+ +
+ +
+
2021-01-27 13:48:27
 +
+

Can you share a screenshot of what you see?

+ + + +
+
+
+
+ + + + + +
+
+ +
+ + +
+ +
+ +
+
2021-01-27 13:53:05
 +
+

@U01K2E1DX8T, @U016EJ380H0, @U015ZRHGVAB can you help? On the phone with Mark

+ + + +
+
+
+
+ + + + + +
+
+ +
+ + +
+ +
+ +
+
2021-01-27 13:53:09
 +
+

has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ +
+ + +
+ +
+ +
+
2021-01-27 13:54:26
 +
+

@U016BBEQRRV are you able to click on the link in the bottom right?

+ + + +
+
+
+
+ + + + + +
+
+ +
+ + +
+ +
+ +
+
2021-01-27 13:55:26
 +
+

@U016BBEQRRV Everything okay? Did it boot you out when you shared your screen?

+ + + +
+
+
+
+ + + + + +
+
+ +
+ + +
+ +
+ +
+
2021-01-27 13:55:29
 +
+

Did you try the link in your calendar invite first?

+ + + +
+
+
+
+ + + + + +
+
+ +
+ + +
+ +
+ +
+
2021-01-27 13:56:35
 +
+

Hi @UGHF1SHT9 Melissa is sending you a link right now

+ + + +
+
+
+
+ + + + + +
+
+ +
+ + +
+ +
+ +
+
2021-01-27 13:56:41
 +
+

@here everything is running

+ + + +
+
+
+
+ + + + + +
+
+ +
+ + +
+ +
+ +
+
2021-01-27 13:56:50
 +
+

Thank you for your patience Mark

+ + + +
+
+
+
+ + + + + +
+
+ +
+ + +
+ +
+ +
+
2021-01-27 13:57:52
 +
+

Hi @U016BBEQRRV Here is the link - you do need to register https://hopin.com/events/subsurfacelivewinter2021?code=oRF56uHFWjmZORii8ddI1HqMi

+
+
hopin.com
+ + + +
Subsurface LIVE Winter 2021 | Hopin
+
+ Welcome to Subsurface! The Premier Cloud Data Lake Conference Over the next two days, you will hear from you peers and other industry leaders at Subsurface™ LIVE Winter 2021 as we explore the latest open source innovations and motivate you with... +
+ + + + + + + + +
Original URL: https://hopin.com/events/subsurfacelivewinter2021?code=oRF56uHFWjmZORii8ddI1HqMi
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ +
+ + +
+ +
+ +
+
2021-01-27 14:00:03
 +
+

@U016BBEQRRV did that link work for you?

+ + + +
+
+
+
+ + + + + +
+
+ +
+ + +
+ +
+ +
+
2021-01-27 14:01:22
 +
+

*Thread Reply:* He is presenting right now

+ + + +
+
+
+
+ + + + + +
+
+ +
+ + +
+ +
+ +
+
2021-01-27 14:08:23
 +
+

has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ +
+ + +
+ +
+ +
+
2021-01-27 14:08:34
 +
+

has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ +
+ + +
+ +
+ +
+
2021-01-27 14:11:05
 +
+

has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ +
+ + +
+ +
+ +
+
2021-01-27 14:18:26
 +
+

has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
slackbot + +
+
2021-01-27 14:28:48
 +
+

This message was deleted.

+ + + +
+
+
+
+ + + + + +
+
+ +
+ + +
+ +
+ +
+
2021-01-27 14:35:49
 +
+

*Thread Reply:* Anytime!

+ + + +
+
+
+
+ + + + + + + +
+
+ + + + +
+ +
slackbot + +
+
2021-01-27 14:30:11
 +
+

OpenLineage has joined this channel by invitation from Subsurface.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paul De Schacht + (pauldeschacht@gmail.com) +
+
2021-01-27 14:30:11
 +
+

@Paul De Schacht has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ +
+ + +
+ +
+ +
+
2021-01-27 14:30:46
 +
+

has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ +
+ + +
+ +
+ +
+
2021-01-27 15:27:46
 +
+

has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ +
+ + +
+ +
+ +
+
2021-01-27 18:21:11
 +
+

has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
slackbot + +
+
2021-02-15 07:21:26
 +
+

Subsurface has removed your organization from this channel. You’ll continue to have access to this archived copy.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
slackbot + +
+
2023-08-03 00:00:00
 +
+

Some older messages are unavailable. Due to the retention policies of an organization in this channel, all their messages and files from before this date have been deleted.

+ + + +
+
+
+
+ + + +
+
+ + + + \ No newline at end of file diff --git a/channel/nyc-meetup/index.html b/channel/nyc-meetup/index.html new file mode 100644 index 0000000..5ac62bd --- /dev/null +++ b/channel/nyc-meetup/index.html @@ -0,0 +1,477 @@ + + + + + + Slack Export - #nyc-meetup + + + + + +
+ + + +
+ + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-04-04 14:10:04
 +
+

@Michael Robinson has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harel Shein + (harel.shein@gmail.com) +
+
2023-04-04 14:10:23
 +
+

@Harel Shein has joined the channel

+ + + +
+ 🙌 Harel Shein +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2023-04-04 14:10:24
 +
+

@Ross Turk has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Benji Lampel + (benjamin@astronomer.io) +
+
2023-04-04 14:11:38
 +
+

@Benji Lampel has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Peter Hicks + (peter@datakin.com) +
+
2023-04-19 15:20:00
 +
+

@Peter Hicks has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Viraj Parekh + (vmpvmp94@gmail.com) +
+
2023-04-19 15:20:45
 +
+

@Viraj Parekh has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-05-04 10:37:42
 +
+

Please join the meetup group: https://www.meetup.com/data-lineage-meetup/

+
+
Meetup
+ + + +
New York OpenLineage Meetup (Providence, RI)
+
+ Imagine a data landscape where data and jobs are described predictably and consistently. What would you be able to accomplish if operational lineage information traveled easily among all your tools?Op +
+ + + + + + + + +
Original URL: https://www.meetup.com/data-lineage-meetup/
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Yuanli Wang + (yuanliw@bu.edu) +
+
2023-06-01 18:34:18
 +
+

@Yuanli Wang has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-06-12 14:24:40
 +
+

Hi folks, we’re hosting another NYC meetup on 6/22 at Collibra (thanks, @Sheeri Cabral (Collibra)!). Please RSVP by 6/20. Hope to see you there.

+
+
Meetup
+ + + +
Data Lineage Meetup with Collibra, Thu, Jun 22, 2023, 6:00 PM | Meetup
+
+ Join us on Thursday, June 22nd, 2023, from 6:00_8:00 pm ET at Collibra's HQ in New York to discuss the present and future of OpenLineage. Meet other members of the ecosyste +
+ + + + + + + + +
Original URL: https://www.meetup.com/data-lineage-meetup/events/294065396/
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Nam Nguyen + (nam@astrafy.io) +
+
2023-07-14 05:37:44
 +
+

@Nam Nguyen has joined the channel

+ + + +
+
+
+
+ + + +
+
+ + + + \ No newline at end of file diff --git a/channel/open-lineage-plus-bacalhau/index.html b/channel/open-lineage-plus-bacalhau/index.html new file mode 100644 index 0000000..693df2e --- /dev/null +++ b/channel/open-lineage-plus-bacalhau/index.html @@ -0,0 +1,1077 @@ + + + + + + Slack Export - #open-lineage-plus-bacalhau + + + + + +
+ + + +
+ + + +
+
+ + + + +
+ +
David Aronchick + (aronchick@gmail.com) +
+
2022-12-07 04:14:09
 +
+

@David Aronchick has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2022-12-07 04:14:20
 +
+

@Julien Le Dem has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Enrico Rotundo + (enrico.rotundo@gmail.com) +
+
2022-12-07 04:14:20
 +
+

@Enrico Rotundo has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
David Aronchick + (aronchick@gmail.com) +
+
2022-12-07 04:14:50
 +
+

Hi all! I’ve created this channel to chat about bringing Open Lineage to Bacalhau (https://www.bacalhau.org/)

+
+
bacalhau.org
+ + + +
Bacalhau
+
+ Reinventing Science through reproducible data pipelines +
+ + + + + + + + +
Original URL: https://www.bacalhau.org/
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
David Aronchick + (aronchick@gmail.com) +
+
2022-12-07 04:15:23
 +
+

the idea would be at every step of a DAG execution, we automatically read the inputs (if any) and create a metadata file with open lineage information in it

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
David Aronchick + (aronchick@gmail.com) +
+
2022-12-07 04:18:30
 +
+

Enrico has done some early thinking on this - https://www.notion.so/pl-strflt/Initial-design-doc-Oct-22-d2b032bd16e340d3ada39171c9ad524d

+
+
PL-STRFLT on Notion
+ + + +
Initial design doc (Oct.22)
+
+ This doc was originally posted on HackMD at this url. +
+ + + + + + +
Original URL: https://www.notion.so/pl-strflt/Initial-design-doc-Oct-22-d2b032bd16e340d3ada39171c9ad524d
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
David Aronchick + (aronchick@gmail.com) +
+
2022-12-07 04:19:06
 +
+

and in parallel created an Airflow operator -> https://github.com/enricorotundo/bacalhau-airflow-provider

+
+ + + + +
enricorotundo/bacalhau-airflow-provider
+
+ An Apache Airflow Provider for Bacalhau (a Compute over Data framework) +
+ +
+
Website
+ <https://github.com/filecoin-project/bacalhau> +
+ +
+
Stars
+ 1 +
+ + + + + + + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
David Aronchick + (aronchick@gmail.com) +
+
2022-12-07 04:19:12
 +
+

lemme know how i can help!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
David Aronchick + (aronchick@gmail.com) +
+
2022-12-07 04:19:30
 +
+

cc @Enrico Rotundo @Julien Le Dem

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Enrico Rotundo + (enrico.rotundo@gmail.com) +
+
2022-12-07 07:45:04
 +
+

hey @Julien Le Dem glad to meet you! The TLDR; is we’re going to use Airflow to orchestrate Bacalhau pipelines, and would love to add open-linage to pipelines too. I see Marquez integrates already with Airflow so that may be a good fit!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Philippe + (philippe@polyphene.io) +
+
2022-12-07 08:40:28
 +
+

@Philippe has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2022-12-07 22:03:29
 +
+

Hello @Enrico Rotundo nice to meet you as well. FYI we have the monthly meeting on zoom tomorrow if you guys want to join.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2022-12-07 23:35:21
 +
+

https://openlineage.slack.com/archives/C01CK9T7HKR/p1670432586277209

+
+ + +
+ + + } + + Michael Robinson + (https://openlineage.slack.com/team/U02LXF3HUN7) +
+ + + +
+
+ <p>@channel +This month’s TSC meeting is tomorrow at 10 am PT. All are welcome! <a href="https://openlineage.slack.com/archives/C01CK9T7HKR/p1669925470878699">https://openlineage.slack.com/archives/C01CK9T7HKR/p1669925470878699</a></p> +
+ + + + + + +
Original URL: https://openlineage.slack.com/archives/C01CK9T7HKR/p1670432586277209
+ + +
+ + Thread in Slack Conversation +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Enrico Rotundo + (enrico.rotundo@gmail.com) +
+
2022-12-08 05:22:55
 +
+

thanks for sharing that! I’ll join you!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Enrico Rotundo + (enrico.rotundo@gmail.com) +
+
2023-02-08 04:54:05
 +
+

Hi @Julien Le Dem As I’m starting to build an airflow operator for Bacalhau (which will include support for OpenLineage 🙂), I was wondering if you could share your knowledge about building operators. +Why did you place the current operator in the OpenLineage repo rather than raising a PR to the “official” community-built Airflow repo? ~Is there any specific reason (community guidelines, technical, etc.)?~

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Enrico Rotundo + (enrico.rotundo@gmail.com) +
+
2023-02-08 04:59:43
 +
+

*Thread Reply:* Oh now that I’m reading your new operator draft AIP it all makes sense

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Enrico Rotundo + (enrico.rotundo@gmail.com) +
+
2023-02-08 05:21:02
 +
+

*Thread Reply:* ok so at this point the question is… for newborn operators, do you suggest to start with their own package or try to merge into airflow.providers directly?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2023-02-08 20:42:34
 +
+

*Thread Reply:* I think you can create your own Provider package with your operator. This is more a question for the airflow mailing list.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2023-02-08 20:43:25
 +
+

*Thread Reply:* I would recommend this to add lineage support for your operator: +https://openlineage.io/docs/integrations/airflow/operator/ +https://openlineage.io/docs/integrations/airflow/extractors/default-extractors

+
+
openlineage.io
+ + + +
Exposing lineage in Airflow operators | OpenLineage Docs
+
+ Since: 0.18.0 +
+ + + + + + +
Original URL: https://openlineage.io/docs/integrations/airflow/operator/
+ + + +
+
+
openlineage.io
+ + + +
Default extractors | OpenLineage Docs
+
+ Default extractors are a new way in Airflow 2.3+ and OpenLineage 0.17.0+ to easily add lineage to your data pipelines by modifying your Airflow operators directly. This means custom operators—built in house or forked from another project—can provide you and your team with lineage data without having to modify the OpenLineage project directly, with data sent to your lineage backend of choice, most commonly using the OPENLINEAGE_URL environment variable. +
+ + + + + + +
Original URL: https://openlineage.io/docs/integrations/airflow/extractors/default-extractors
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2023-02-08 20:43:41
 +
+

*Thread Reply:* And nice to heare from you @Enrico Rotundo!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2023-02-08 20:44:11
 +
+

*Thread Reply:* The next monthly meeting is tomorrow if you want to join

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2023-02-08 20:45:05
 +
+

*Thread Reply:* I added you to the invite just in case

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2023-02-08 20:46:17
 +
+

*Thread Reply:* To answer your original question, we started outside the Airflow community, now that the project is more established it is easier to get this approved. Hopping to get this to a vote soon

+ + + +
+ 🙌 Enrico Rotundo +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mike Dillion + (mike.dillion@gmail.com) +
+
2023-02-11 18:51:40
 +
+

@Mike Dillion has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
jrich + (jasonrich85@icloud.com) +
+
2023-03-10 14:52:21
 +
+

@jrich has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Nam Nguyen + (nam@astrafy.io) +
+
2023-07-14 05:37:45
 +
+

@Nam Nguyen has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Silvia Pina + (silviampina@gmail.com) +
+
2023-07-15 12:10:17
 +
+

@Silvia Pina has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
YYYY XXXX + (mail4registering@gmail.com) +
+
2023-07-17 21:21:36
 +
+

@YYYY XXXX has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
patrice quillevere + (patrice.quillevere.csgroup.eu@gmail.com) +
+
2023-07-18 11:11:13
 +
+

@patrice quillevere has joined the channel

+ + + +
+
+
+
+ + + +
+
+ + + + \ No newline at end of file diff --git a/channel/providence-meetup/index.html b/channel/providence-meetup/index.html new file mode 100644 index 0000000..ba8dae9 --- /dev/null +++ b/channel/providence-meetup/index.html @@ -0,0 +1,577 @@ + + + + + + Slack Export - #providence-meetup + + + + + +
+ + + +
+ + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-02-24 14:37:52
 +
+

@Michael Robinson has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harel Shein + (harel.shein@gmail.com) +
+
2023-02-24 14:39:45
 +
+

@Harel Shein has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Benji Lampel + (benjamin@astronomer.io) +
+
2023-02-24 14:39:45
 +
+

@Benji Lampel has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Viraj Parekh + (vmpvmp94@gmail.com) +
+
2023-02-24 14:39:46
 +
+

@Viraj Parekh has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sheeri Cabral (Collibra) + (sheeri.cabral@collibra.com) +
+
2023-02-24 14:39:46
 +
+

@Sheeri Cabral (Collibra) has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2023-02-24 14:39:46
 +
+

@Ross Turk has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Eric Veleker + (eric@atlan.com) +
+
2023-02-24 14:39:46
 +
+

@Eric Veleker has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-02-24 14:42:28
 +
+

Hi everyone! I created this for coordinating travel, etc. Please add anyone I forgot. Thanks and looking forward to meeting up IRL.

+ + + +
+ ✅ Sheeri Cabral (Collibra) +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harel Shein + (harel.shein@gmail.com) +
+
2023-02-24 14:57:53
 +
+

Thanks, Michael! Looking forward to it!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harel Shein + (harel.shein@gmail.com) +
+
2023-02-24 14:58:29
 +
+

New Yorkers, assuming y’all don’t own a car, I’m happy to rent one and drive us up there.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harel Shein + (harel.shein@gmail.com) +
+
2023-02-24 16:22:03
 +
+

*Thread Reply:* @Viraj Parekh / @Benji Lampel anyone else?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-03-01 16:57:30
 +
+

Some hotel ideas if you’re traveling and thinking of spending the night: +• The Graduate (formerly the Biltmore — kinda fancy but not too pricey) +• The Dean +• Aloft (a more expensive option but right next to the meetup location) +• Hampton Inn

+ + + +
+ 🙌 Ross Turk, Sheeri Cabral (Collibra) +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
karen + (karenjn@outlook.fr) +
+
2023-03-06 11:19:07
 +
+

@karen has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-03-09 10:17:29
 +
+

Here’s the event space info for your convenience: +CIC Providence & District Hall +225 Dyer St. +Providence, RI 02903 + +Hope Island Room, 3rd Floor +Tell the concierge you’re looking for the Data Lineage Meetup

+ + + +
+ ✅ Sheeri Cabral (Collibra), Ross Turk +
+ +
+
+
+
+ + + + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-04-04 16:40:01
 +
+

archived the channel

+ + + +
+
+
+
+ + + +
+
+ + + + \ No newline at end of file diff --git a/channel/sf-meetup/index.html b/channel/sf-meetup/index.html new file mode 100644 index 0000000..73f22ed --- /dev/null +++ b/channel/sf-meetup/index.html @@ -0,0 +1,657 @@ + + + + + + Slack Export - #sf-meetup + + + + + +
+ + + +
+ + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-05-04 10:35:50
 +
+

@Michael Robinson has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2023-05-04 10:36:16
 +
+

@Julien Le Dem has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Peter Hicks + (peter@datakin.com) +
+
2023-05-04 10:36:16
 +
+

@Peter Hicks has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2023-05-04 10:36:16
 +
+

@Willy Lulciuc has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Atif Tahir + (atif.tahir13@gmail.com) +
+
2023-05-23 14:38:04
 +
+

@Atif Tahir has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Bernat Gabor + (gaborjbernat@gmail.com) +
+
2023-05-26 12:42:29
 +
+

@Bernat Gabor has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Yuanli Wang + (yuanliw@bu.edu) +
+
2023-06-01 18:33:53
 +
+

@Yuanli Wang has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-06-21 17:40:49
 +
+

Our first San Francisco meetup is next Tuesday at Astronomer’s HQ in the financial district https://www.meetup.com/meetup-group-bnfqymxe/events/293448130/

+
+
Meetup
+ + + +
OpenLineage Meetup @ Astronomer, Tue, Jun 27, 2023, 5:30 PM | Meetup
+
+ Data engineers and pipeline managers know that producing data lineage – end-to-end pipeline metadata instrumented at runtime or parsed at design time – is a heavy lift with +
+ + + + + + + + +
Original URL: https://www.meetup.com/meetup-group-bnfqymxe/events/293448130/
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jeff L + (jeffreyl@alumni.cmu.edu) +
+
2023-06-27 21:02:24
 +
+

@Jeff L has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Nam Nguyen + (nam@astrafy.io) +
+
2023-07-14 05:37:49
 +
+

@Nam Nguyen has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-08-02 12:59:23
 +
+

Our second SF meetup will be happening on August 30th at 5:30 PM PT, at Astronomer. Please medium=referral&utmcampaign=share-btnsavedeventssharemodal&utmsource=link|sign up> to let us know you’re coming!

+
+
Meetup
+ + + +
OpenLineage Meetup @ Astronomer, Wed, Aug 30, 2023, 5:30 PM | Meetup
+
+ Data engineers and pipeline managers know that producing data lineage – end-to-end pipeline metadata instrumented at runtime or parsed at design time – is a heavy lift with +
+ + + + + + + + +
Original URL: https://www.meetup.com/meetup-group-bnfqymxe/events/295195280/?utm_medium=referral&amp;utm_campaign=share-btn_savedevents_share_modal&amp;utm_source=link
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-08-30 15:11:18
 +
+

Adding the venue info in case it’s more convenient than the meetup page:

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-08-30 15:12:55
 +
+

Time: 5:30_8:30 pm +Address: 8 California St., San Francisco, CA, seventh floor +Getting in: someone from Astronomer will be in the lobby to direct you

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kevin Languasco + (kevin@haystack.tv) +
+
2023-08-31 18:29:01
 +
+

@Kevin Languasco has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2023-08-31 23:18:10
 +
+

Some pictures from last night

+ +
+
IMG_2417
+ + + + + + + + +
+
+
IMG_2416
+ + + + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Aaruna Godthi + (aaruna6@gmail.com) +
+
2023-09-23 16:47:37
 +
+

@Aaruna Godthi has joined the channel

+ + + +
+
+
+
+ + + +
+
+ + + + \ No newline at end of file diff --git a/channel/spec-compliance/index.html b/channel/spec-compliance/index.html new file mode 100644 index 0000000..213f96a --- /dev/null +++ b/channel/spec-compliance/index.html @@ -0,0 +1,1096 @@ + + + + + + Slack Export - #spec-compliance + + + + + +
+ + + +
+ + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2023-01-12 08:55:50
 +
+

@Julien Le Dem has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2023-01-12 09:00:07
 +
+

This channel is to support discussion following up on @Sheeri Cabral (Collibra)’s presentation at the monthly meeting. Notes and recording here: https://wiki.lfaidata.foundation/display/OpenLineage/Monthly+TSC+meeting#MonthlyTSCmeeting-December8,2022(10amPT) +Sheeri will start a google doc to document our conclusions and proposal to evolve the spec.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sheeri Cabral (Collibra) + (sheeri.cabral@collibra.com) +
+
2023-01-12 09:00:32
 +
+

@Sheeri Cabral (Collibra) has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mandy Chessell + (mandy.e.chessell@gmail.com) +
+
2023-01-12 09:00:33
 +
+

@Mandy Chessell has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ernie Ostic + (ernie.ostic@getmanta.com) +
+
2023-01-12 09:00:33
 +
+

@Ernie Ostic has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-01-12 09:13:48
 +
+

@Maciej Obuchowski has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-01-12 11:40:03
 +
+

@Michael Robinson has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sheeri Cabral (Collibra) + (sheeri.cabral@collibra.com) +
+
2023-01-12 12:47:20
 +
+

Document is here - https://docs.google.com/document/d/1ysZR13QwDvAiY_rQJedLHpnNn3deQeow7BmCNckd2uM/edit

+ +

At this point I only have the notes + recording link 😄

+ + + +
+ :gratitude_thank_you: Michael Collado, Will Johnson +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sheeri Cabral (Collibra) + (sheeri.cabral@collibra.com) +
+
2023-01-12 12:47:39
 +
+

(right now everyone can edit it, if that becomes a problem, we can restrict it).

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harel Shein + (harel.shein@gmail.com) +
+
2023-01-12 13:40:17
 +
+

@Harel Shein has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Laurent Paris + (laurent@datakin.com) +
+
2023-01-12 14:00:16
 +
+

@Laurent Paris has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2023-01-12 14:03:24
 +
+

@Michael Collado has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Daniel Henneberger + (me@danielhenneberger.com) +
+
2023-01-12 14:05:11
 +
+

@Daniel Henneberger has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sheeri Cabral (Collibra) + (sheeri.cabral@collibra.com) +
+
2023-01-12 14:05:59
 +
+

I will synthesize what I think are the main points, but please change what I got wrong 😄

+ + + +
+ 🙌 Willy Lulciuc +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2023-01-13 18:37:43
 +
+

*Thread Reply:* thanks for kick-starting / driving this @Sheeri Cabral (Collibra) (long overdue)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2023-01-12 14:06:10
 +
+

@Willy Lulciuc has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kengo Seki + (sekikn@gmail.com) +
+
2023-01-14 19:29:10
 +
+

@Kengo Seki has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2023-01-15 10:49:44
 +
+

@Ross Turk has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Orbit + +
+
2023-01-24 14:34:11
 +
+

@Orbit has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sheeri Cabral (Collibra) + (sheeri.cabral@collibra.com) +
+
2023-02-09 14:01:39
 +
+

What are some custom facets people have created, or want to see?

+ +

I’ll start! Custom facet of “rows affected” for DML (e.g. rows inserted, rows deleted, rows updated.

+ +

(and a grand vision would be to set data quality thresholds against this - an application could warn if a run deviates more than 10% from the mean/median rows affected from previous run jobs)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sheeri Cabral (Collibra) + (sheeri.cabral@collibra.com) +
+
2023-02-09 14:02:44
 +
+

*Thread Reply:* Note that it’s possible this has already been implemented - that’s fine! We’re just gathering ideas, nothing is being set in stone here.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2023-02-09 14:03:04
 +
+

*Thread Reply:* you mean output statistics?

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/spec/facets/OutputStatisticsOutputDatasetFacet.json | OutputStatisticsOutputDatasetFacet.json>
+
+ <pre><code>{ + "$schema" : "&lt;https://json-schema.org/draft/2020-12/schema&gt;", + "$id" : "&lt;https://openlineage.io/spec/facets/1-0-0/OutputStatisticsOutputDatasetFacet.json&gt;", + "$defs" : { + "OutputStatisticsOutputDatasetFacet" : { + "allOf" : [ { + "$ref" : "&lt;https://openlineage.io/spec/1-0-2/OpenLineage.json#/$defs/OutputDatasetFacet&gt;" + }, { + "type" : "object", + "properties" : { + "rowCount" : { + "description" : "The number of rows written to the dataset", + "type" : "integer" + }, + "size" : { + "description" : "The size in bytes written to the dataset", + "type" : "integer" + } + }, + "required" : [ "rowCount" ] + } ] + } + }, + "type" : "object", + "properties" : { + "outputStatistics" : { + "$ref" : "#/$defs/OutputStatisticsOutputDatasetFacet" + } + } +} +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sheeri Cabral (Collibra) + (sheeri.cabral@collibra.com) +
+
2023-02-09 14:04:07
 +
+

*Thread Reply:* Yes! it’s already part of output statistics. I was just giving one example of the kind of ideas we are looking for people to throw out there. Whether or not the facet exists already. 😄 it’s brainstorming, no wrong answers. (in this case the answer is “congrats! it’s already there!“)

+ + + +
+ 👏 Michael Collado +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2023-02-09 14:14:42
 +
+

*Thread Reply:* I’ve always been interested in finding a way for devs to publish business metrics from their apps - i.e., metrics specific to the job or dataset like rowsFiltered or averageValues or something. I think enabling comparison of these metrics with lineage would provide tremendous value to data pipeline engineers

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sheeri Cabral (Collibra) + (sheeri.cabral@collibra.com) +
+
2023-02-09 14:16:53
 +
+

*Thread Reply:* oh yeah. even rowsExamined - e.g. rowsExamined - rowsFiltered = rowsAffected +some databases give those metrics from queries. And this is why they’re optional, BUT if you make a new company that has software to analyze pipeline metrics, that might be required for your software, even though it’s optional in the openlineage spec)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harel Shein + (harel.shein@gmail.com) +
+
2023-02-10 10:07:13
 +
+

*Thread Reply:* I’m thinking about some custom facets that would allow intractability between different frameworks (possibly vendors). for example, I can throw in a link to the Airflow UI for a specific task that we captured metadata for, and another tool (say, a data catalog) can also use my custom facet to help guide users directly to where the process is running. there’s a commercial aspect to this as well, but I think the community aspect is interesting. +I don’t think it should be required to comply with the spec, but I’d like to be able to share this custom facet for others to see if it exists and decorate accordingly. does that make sense?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mike Dillion + (mike.dillion@gmail.com) +
+
2023-02-11 18:51:45
 +
+

@Mike Dillion has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mahesh Gawde + (mahesh.gawde@originml.ai) +
+
2023-02-28 05:53:27
 +
+

@Mahesh Gawde has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Thijs Koot + (thijs.koot@gmail.com) +
+
2023-02-28 08:36:33
 +
+

@Thijs Koot has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-03-09 14:12:58
 +
+

@Anirudh Shrinivason has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
jrich + (jasonrich85@icloud.com) +
+
2023-03-10 14:52:40
 +
+

@jrich has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Suraj Gupta + (suraj.gupta@atlan.com) +
+
2023-06-01 13:51:29
 +
+

@Suraj Gupta has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Yuanli Wang + (yuanliw@bu.edu) +
+
2023-07-12 17:18:25
 +
+

@Yuanli Wang has joined the channel

+ + + +
+
+
+
+ + + +
+
+ + + + \ No newline at end of file diff --git a/channel/toronto-meetup/index.html b/channel/toronto-meetup/index.html new file mode 100644 index 0000000..6b43ed5 --- /dev/null +++ b/channel/toronto-meetup/index.html @@ -0,0 +1,693 @@ + + + + + + Slack Export - #toronto-meetup + + + + + +
+ + + +
+ + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-08-16 14:21:43
 +
+

@Michael Robinson has joined the channel

+ + + +
+ 🙌 Willy Lulciuc +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harel Shein + (harel.shein@gmail.com) +
+
2023-08-16 14:22:08
 +
+

@Harel Shein has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2023-08-16 14:22:08
 +
+

@Julien Le Dem has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-16 14:22:08
 +
+

@Maciej Obuchowski has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-08-16 14:22:08
 +
+

@Paweł Leszczyński has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-08-16 14:22:08
 +
+

@Jakub Dardziński has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2023-08-16 14:32:41
 +
+

@Willy Lulciuc has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Peter Hicks + (peter.hicks@astronomer.io) +
+
2023-08-16 14:33:10
 +
+

@Peter Hicks has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-08-25 13:30:07
 +
+

Some belated updates on this in case you’re not aware: +• Date: 9/18 +• Time: 5-8:00 PM ET +• Place: Canarts, 600 Bay St., #410 (around the corner from the Airflow Summit venue) +• Venue phone: +• Meetup for more info and to sign up: https://www.meetup.com/openlineage/events/295488014/?utm_medium=referral&utm_campaign=share-btn_savedevents_share_modal&utm_source=link

+
+
Meetup
+ + + +
Toronto OpenLineage Meetup at Airflow Summit, Mon, Sep 18, 2023, 2:00 PM | Meetup
+
+ Data engineers and pipeline managers know that producing data lineage – end-to-end pipeline metadata instrumented at runtime or parsed at design time – is a heavy lift with +
+ + + + + + + + +
Original URL: https://www.meetup.com/openlineage/events/295488014/?utm_medium=referral&amp;utm_campaign=share-btn_savedevents_share_modal&amp;utm_source=link
+ + + +
+ + + +
+ 🎉 Harel Shein, Maciej Obuchowski, Paweł Leszczyński, Willy Lulciuc +
+ +
+ 🙌 Harel Shein, Maciej Obuchowski, Paweł Leszczyński, Willy Lulciuc +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harel Shein + (harel.shein@gmail.com) +
+
2023-08-25 13:33:42
 +
+

really looking forward to meeting all of you in Toronto!!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2023-09-01 23:10:51
 +
+

Most OpenLineage regular contributors will be there. It will be fun to be all in person. Everyone is encouraged to join

+ + + +
+ 🙌 Harel Shein, Maciej Obuchowski, Paweł Leszczyński, Michael Robinson, Willy Lulciuc, Peter Hicks +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-09-11 10:13:57
 +
+

@channel +It’s hard to believe this is happening in just one week! Here’s the updated agenda:

+ +
  1. Intros
  2. Evolution of spec presentation/discussion (project background/history)
  3. State of the community
  4. Integrating OpenLineage with Metaphor (by special guests Ye & Ivan)
  5. Spark/Column lineage update
  6. Airflow Provider update
  7. Roadmap Discussion
  8. Action items review/next steps +Find the details and RSVP https://www.meetup.com/openlineage/events/295488014/?utmmedium=referral&utmcampaign=share-btnsavedeventssharemodal&utmsource=link|here.
    9. +
+
+
metaphor.io
+ + + +
Metaphor - The Social Platform for Data
+
+ Making Data Actionable, At Scale - Designed for data teams building cloud-native, self-service data platforms for their business users. Explore our Data Governance, Data Lineage, Data Discovery, and Data Trust capabilities today. +
+ + + + + + + + +
Original URL: https://metaphor.io/
+ + + +
+
+
Meetup
+ + + +
Toronto OpenLineage Meetup at Airflow Summit, Mon, Sep 18, 2023, 5:00 PM | Meetup
+
+ Data engineers and pipeline managers know that producing data lineage – end-to-end pipeline metadata instrumented at runtime or parsed at design time – is a heavy lift with +
+ + + + + + + + +
Original URL: https://www.meetup.com/openlineage/events/295488014/?utm_medium=referral&amp;utm_campaign=share-btn_savedevents_share_modal&amp;utm_source=link
+ + + +
+ + + +
+ 🙌 Willy Lulciuc +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Greg Kim + (kgkwiz@gmail.com) +
+
2023-09-15 10:11:11
 +
+

@Greg Kim has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-09-15 12:17:29
 +
+

Looking forward to seeing you on Monday! Here’s the time/place info again for your convenience: +• Date: 9/18 +• Time: 5-8:00 PM ET +• Place: Canarts, 600 Bay St., #410 (around the corner from the Airflow Summit venue) +• Venue phone: +• Meetup page with more info and signup: https://www.meetup.com/openlineage/events/295488014/?utm_medium=referral&utm_campaign=share-btn_savedevents_share_modal&utm_source=link +Please send a message if you find yourself stuck in the lobby, etc.

+
+
Meetup
+ + + +
Toronto OpenLineage Meetup at Airflow Summit, Mon, Sep 18, 2023, 5:00 PM | Meetup
+
+ Data engineers and pipeline managers know that producing data lineage – end-to-end pipeline metadata instrumented at runtime or parsed at design time – is a heavy lift with +
+ + + + + + + + +
Original URL: https://www.meetup.com/openlineage/events/295488014/?utm_medium=referral&amp;utm_campaign=share-btn_savedevents_share_modal&amp;utm_source=link
+ + + +
+ + + +
+ 🙌 Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-09-18 16:20:33
 +
+

Hi, if you’re wondering if you’re in the right place: look for Uncle Tetsu’s Cheesecake nextdoor and for the address (600 Bay St) above the door. The building is an older one (unlike the meeting space itself, which is modern and well-appointed)

+ + + +
+
+
+
+ + + +
+
+ + + + \ No newline at end of file diff --git a/index.html b/index.html new file mode 100644 index 0000000..b6a82de --- /dev/null +++ b/index.html @@ -0,0 +1,151812 @@ + + + + + + Slack Export - #general + + + + + +
+ + + +
+ + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2020-10-20 21:01:02
 +
+

@Julien Le Dem has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mars Lan + (mars.th.lan@gmail.com) +
+
2020-10-21 08:23:39
 +
+

@Mars Lan has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Wes McKinney + (wesmckinn@gmail.com) +
+
2020-10-21 11:39:13
 +
+

@Wes McKinney has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ryan Blue + (rblue@netflix.com) +
+
2020-10-21 12:46:39
 +
+

@Ryan Blue has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Drew Banin + (drew@fishtownanalytics.com) +
+
2020-10-21 12:53:42
 +
+

@Drew Banin has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2020-10-21 13:29:49
 +
+

@Willy Lulciuc has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Lewis Hemens + (lewis@dataform.co) +
+
2020-10-21 13:52:50
 +
+

@Lewis Hemens has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2020-10-21 14:15:41
 +
+

This is the official start of the OpenLineage initiative. Thank you all for joining. First item is to provide feedback on the doc: https://docs.google.com/document/d/1qL_mkd9lFfe_FMoLTyPIn80-fpvZUAdEIfrabn8bfLE/edit

+ + + +
+ 🎉 Willy Lulciuc, Abe Gong +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Abe Gong + (abe@superconductive.com) +
+
2020-10-21 23:22:03
 +
+

@Abe Gong has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Shirshanka Das + (sdas@linkedin.com) +
+
2020-10-22 13:50:35
 +
+

@Shirshanka Das has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
deleted_profile + (fengtao04@gmail.com) +
+
2020-10-23 15:03:44
 +
+

@deleted_profile has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Chris White + (chris@prefect.io) +
+
2020-10-23 19:30:36
 +
+

@Chris White has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2020-10-24 19:29:04
 +
+

Thanks all for joining. In addition to the google doc, I have opened a pull request with an initial openapi spec: https://github.com/OpenLineage/OpenLineage/pull/1 +The goal is to specify the initial model (just plain lineage) that will be extended with various facets. +It does not intend to restrict to HTTP. Those same PUT calls without output can be translated to any async protocol

+
+
GitHub
+ + + +
initial spec proposal by julienledem · Pull Request #1 · OpenLineage/OpenLineage
+
+ Signed-off-by: Julien Le Dem <a href="mailto:julien@apache.org">julien@apache.org</a> +
+ + + + + + + + +
Original URL: https://github.com/OpenLineage/OpenLineage/pull/1
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2020-10-24 19:31:09
 +
+

For reference, the slides of the kickoff meeting: https://docs.google.com/presentation/d/1bOnm4J7y1JRJBJtSImm-3vvXzvqqkL-UsCShAuub5oU/edit?usp=sharing

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Wes McKinney + (wesmckinn@gmail.com) +
+
2020-10-25 12:13:26
 +
+

Am I the only weirdo that would prefer a Google Group mailing list to Slack for communicating?

+ + + +
+ 👍 Ryan Blue +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2020-10-25 17:22:09
 +
+

*Thread Reply:* slack is the new email?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Wes McKinney + (wesmckinn@gmail.com) +
+
2020-10-25 17:40:19
 +
+

*Thread Reply:* :(

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ryan Blue + (rblue@netflix.com) +
+
2020-10-27 12:27:04
 +
+

*Thread Reply:* I'd prefer a google group as well

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ryan Blue + (rblue@netflix.com) +
+
2020-10-27 12:27:25
 +
+

*Thread Reply:* I think that is better for keeping people engaged, since it isn't just a ton of history to go through

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ryan Blue + (rblue@netflix.com) +
+
2020-10-27 12:27:38
 +
+

*Thread Reply:* And I think it is also better for having thoughtful design discussions

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2020-10-29 15:40:14
 +
+

*Thread Reply:* I’m happy to create a google group if that would help.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2020-10-29 15:45:23
 +
+

*Thread Reply:* Here it is: https://groups.google.com/g/openlineage

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2020-10-29 15:46:34
 +
+

*Thread Reply:* Slack is more of a way to nudge discussions along, we can use github issues or the mailing list to discuss specific points

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2020-11-03 17:34:53
 +
+

*Thread Reply:* @Ryan Blue and @Wes McKinney any recommendations on automating sending github issues update to that list?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ryan Blue + (rblue@netflix.com) +
+
2020-11-03 17:35:34
 +
+

*Thread Reply:* I don't really know how to do that

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ravi Suhag + (suhag.ravi@gmail.com) +
+
2021-04-02 07:18:25
 +
+

*Thread Reply:* @Julien Le Dem How about using Github discussions. They are specifically meant to solve this problem. Feature is still in beta, but it be enabled from repository settings. One positive side i see is that it will really easy to follow through and one separate place to go and look for discussions and ideas which are being discussed.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-04-02 19:51:55
 +
+

*Thread Reply:* I just enabled it: https://github.com/OpenLineage/OpenLineage/discussions

+ + + +
+ 🙌 Ravi Suhag +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Wes McKinney + (wesmckinn@gmail.com) +
+
2020-10-25 12:14:06
 +
+

Or GitHub Issues

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2020-10-25 17:21:44
 +
+

*Thread Reply:* the plan is to use github issues for discussions on the spec. This is to supplement

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Laurent Paris + (laurent@datakin.com) +
+
2020-10-26 19:28:17
 +
+

@Laurent Paris has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Josh Benamram + (josh@databand.ai) +
+
2020-10-27 21:17:30
 +
+

@Josh Benamram has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Victor Shafran + (victor.shafran@databand.ai) +
+
2020-10-28 04:07:27
 +
+

@Victor Shafran has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Victor Shafran + (victor.shafran@databand.ai) +
+
2020-10-28 04:09:00
 +
+

👋 Hi everyone!

+ + + +
+ 👋 Willy Lulciuc, Abe Gong, Drew Banin, Julien Le Dem +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Zhamak Dehghani + (zdehghan@thoughtworks.com) +
+
2020-10-29 17:59:31
 +
+

@Zhamak Dehghani has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2020-11-02 18:30:51
 +
+

I’ve opened a github issue to propose OpenAPI as the way to define the lineage metadata: https://github.com/OpenLineage/OpenLineage/issues/2 +I have also started a thread on the OpenLineage group: https://groups.google.com/g/openlineage/c/2i7ogPl1IP4 +Discussion should happen there: ^

+
+
GitHub
+ + + +
Proposal to use an openapi spec to define the metadata. · Issue #2 · OpenLineage/OpenLineage
+
+ I'm proposing to use openapi to formalize the core lineage model. This doesn't preclude having the ability to serialize in some other binary format (avro, protobuf,...) +
+ + + + + + + + +
Original URL: https://github.com/OpenLineage/OpenLineage/issues/2
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Evgeny Shulman + (evgeny.shulman@databand.ai) +
+
2020-11-04 10:56:00
 +
+

@Evgeny Shulman has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2020-11-05 20:51:22
 +
+

FYI I have updated the PR with a simple genrator: https://github.com/OpenLineage/OpenLineage/pull/1

+
+ + +
+ + + } + + julienledem + (https://github.com/julienledem) +
+ + + +
#1 initial spec proposal
+
+ <p>The goal is to specify the initial model (just plain lineage) that will be extended with various facets. <br /> +It does not intend to restrict to HTTP. Those same PUT calls without output can be translated to any async protocol</p> + +<p>Signed-off-by: Julien Le Dem <a href="mailto:julien@apache.org">julien@apache.org</a></p> +
+ +
+
Comments
+ 6 +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Daniel Henneberger + (danny@datakin.com) +
+
2020-11-11 15:05:46
 +
+

@Daniel Henneberger has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2020-12-08 17:27:57
 +
+

Please send me your github ids if you wish to be added to the github repo

+ + + +
+ 👍 Willy Lulciuc +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Fabrice Etanchaud + (fabrice.etanchaud@netc.fr) +
+
2020-12-10 02:10:35
 +
+

@Fabrice Etanchaud has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2020-12-10 17:04:29
 +
+

As mentioned on the mailing List, the initial spec is ready for a final review. Thanks for all who gave feedback so far.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2020-12-10 17:04:39
 +
+

*Thread Reply:* https://github.com/OpenLineage/OpenLineage/pull/1

+
+ + +
+ + + } + + julienledem + (https://github.com/julienledem) +
+ + + +
#1 initial spec proposal
+
+ <p>The goal is to specify the initial model (just plain lineage) that will be extended with various facets. <br /> +It does not intend to restrict to HTTP. Those same PUT calls without output can be translated to any async protocol</p> + +<p>Signed-off-by: Julien Le Dem <a href="mailto:julien@apache.org">julien@apache.org</a></p> +
+ +
+
Comments
+ 6 +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2020-12-10 17:04:51
 +
+

The next step will be to define individual facets

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2020-12-13 00:28:11
 +
+

I have opened a PR to update the ReadMe: https://openlineage.slack.com/archives/C01EB6DCLHX/p1607835827000100

+
+ + +
+ + + } + + julienledem + (https://github.com/julienledem) +
+ + +
Pull request opened by julienledem
+
#4 Adding README
+
+ <p>Signed-off-by: Julien Le Dem <a href="mailto:julien@apache.org">julien@apache.org</a></p> +
+ + + + + + +
Original URL: https://openlineage.slack.com/archives/C01EB6DCLHX/p1607835827000100
+ + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+ 👍 Willy Lulciuc +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2020-12-14 17:55:46
 +
+

*Thread Reply:* Looks great!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maxime Beauchemin + (max@preset.io) +
+
2020-12-13 17:45:49
 +
+

👋

+ + + +
+ 👋 Shirshanka Das, Julien Le Dem, Willy Lulciuc, Arthur Wiedmer, Mario Measic +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2020-12-14 20:19:57
 +
+

I’m planning to merge https://github.com/OpenLineage/OpenLineage/pull/1 soon. That will be the base that we can iterate on and will enable starting the discussion on individual facets

+
+ + +
+ + + } + + julienledem + (https://github.com/julienledem) +
+ + + +
#1 initial spec proposal
+
+ <p>The goal is to specify the initial model (just plain lineage) that will be extended with various facets. <br /> +It does not intend to restrict to HTTP. Those same PUT calls without output can be translated to any async protocol</p> + +<p>Signed-off-by: Julien Le Dem <a href="mailto:julien@apache.org">julien@apache.org</a></p> +
+ +
+
Comments
+ 6 +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2020-12-16 21:40:52
 +
+

Thank you all for the feedback. I have made an update to the initial spec adressing the final comments

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2020-12-16 21:41:16
 +
+

*Thread Reply:* https://github.com/OpenLineage/OpenLineage/pull/1

+
+ + +
+ + + } + + julienledem + (https://github.com/julienledem) +
+ + + +
#1 initial spec proposal
+
+ <p>The goal is to specify the initial model (just plain lineage) that will be extended with various facets. <br /> +It does not intend to restrict to HTTP. Those same PUT calls without output can be translated to any async protocol</p> + +<p>Signed-off-by: Julien Le Dem <a href="mailto:julien@apache.org">julien@apache.org</a></p> +
+ +
+
Comments
+ 7 +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2020-12-19 11:21:27
 +
+

The contributing guide is available here: https://github.com/OpenLineage/OpenLineage/blob/main/CONTRIBUTING.md +Here is an example proposal for adding a new facet: https://github.com/OpenLineage/OpenLineage/issues/9

+
+ + + + +
#9 [PROPOSAL] track custom state and transitions
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+ 👍 Josh Benamram, Victor Shafran +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2020-12-19 18:27:36
 +
+

Welcome to the newly joined members 🙂 👋

+ + + +
+ 👋 Chris Lambert, Ananth Packkildurai, Arthur Wiedmer, Abe Gong, ale, James Le, Ha Pham, David Krevitt, Harel Shein +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ash Berlin-Taylor + (ash@apache.org) +
+
2020-12-21 05:23:21
 +
+

Hello! Airflow PMC member here. Super interested in this effort

+ + + +
+ 👋 Willy Lulciuc +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2020-12-21 12:15:42
 +
+

*Thread Reply:* Welcome!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ash Berlin-Taylor + (ash@apache.org) +
+
2020-12-21 05:25:07
 +
+

I'm joining this slack now, but I'm basically done for the year, so will investigate proposals etc next year

+ + + +
+ 🙌 Willy Lulciuc +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Zachary Friedman + (zafriedman@gmail.com) +
+
2020-12-21 10:02:37
 +
+

Hey all 👋 Super curious what people's thoughts are on the best way for data quality tools i.e. Great Expectations to integrate with OpenLineage. Probably a Dataset level facet of some sort (from the 25 minutes of deep spec knowledge I have 😆), but curious if that's something being worked on? @Abe Gong

+ + + +
+ 👋 Abe Gong, Willy Lulciuc +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Abe Gong + (abe@superconductive.com) +
+
2020-12-21 10:30:51
 +
+

*Thread Reply:* Yes, that’s about right.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Abe Gong + (abe@superconductive.com) +
+
2020-12-21 10:31:45
 +
+

*Thread Reply:* There’s some subtlety here.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Abe Gong + (abe@superconductive.com) +
+
2020-12-21 10:32:02
 +
+

*Thread Reply:* The initial OpenLineage spec is pretty explicit about linking metadata primarily to execution of specific tasks, which is appropriate for ValidationResults in Great Expectations

+ + + +
+ ✅ Zachary Friedman +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Abe Gong + (abe@superconductive.com) +
+
2020-12-21 10:32:57
 +
+

*Thread Reply:* There isn’t as strong a concept of persistent data objects (e.g. a specific table, or batches of data from a specific table)

+ + + +
+ ✅ Zachary Friedman +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Abe Gong + (abe@superconductive.com) +
+
2020-12-21 10:33:20
 +
+

*Thread Reply:* (In the GE ecosystem, we call these DataAssets and Batches)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Abe Gong + (abe@superconductive.com) +
+
2020-12-21 10:33:56
 +
+

*Thread Reply:* This is also an important conceptual unit, since it’s the level of analysis where Expectations and data docs would typically attach.

+ + + +
+ ✅ Zachary Friedman +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Abe Gong + (abe@superconductive.com) +
+
2020-12-21 10:34:47
 +
+

*Thread Reply:* @James Campbell and I have had some productive conversations with @Julien Le Dem and others about this topic

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2020-12-21 12:20:53
 +
+

*Thread Reply:* Yep! The next step will be to open a few github issues with proposals to add to or amend the spec. We would probably start with a Descriptive Dataset facet of a dataset profile (or dataset update profile). There are other aspects to clarify as well as @Abe Gong is explaining above.

+ + + +
+ ✅ James Campbell +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Zachary Friedman + (zafriedman@gmail.com) +
+
2020-12-21 10:08:24
 +
+

Also interesting to see where this would hook into Dagster. Because one of the many great features of Dagster IMO is it let you do stuff like this (without a formal spec albeit). An OpenLineageMaterialization could be interesting

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2020-12-21 12:23:41
 +
+

*Thread Reply:* Totally! We had a quick discussion with Dagster. Looking forward to proposals along those lines.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harikiran Nayak + (hari@streamsets.com) +
+
2020-12-21 14:35:11
 +
+

Congrats @Julien Le Dem @Willy Lulciuc and team on launching OpenLineage!

+ + + +
+ 🙌 Willy Lulciuc +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2020-12-21 14:48:11
 +
+

*Thread Reply:* Thanks, @Harikiran Nayak! It’s amazing to see such interest in the community on defining a standard for lineage metadata collection.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harikiran Nayak + (hari@streamsets.com) +
+
2020-12-21 15:03:29
 +
+

*Thread Reply:* Yep! Its a validation that the problem is real!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kriti + (kathuriakritihp@gmail.com) +
+
2020-12-22 02:05:45
 +
+

Hey folks! +Worked on a variety of lineage problems across domains. Super excited about this initiative!

+ + + +
+ 👋 Willy Lulciuc +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2020-12-22 13:23:43
 +
+

*Thread Reply:* Welcome!

+ + + +
+ 👋 Kriti +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2020-12-30 22:30:23
 +
+

*Thread Reply:* What are you current use cases for lineage?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2020-12-22 19:54:33
 +
+

(for review) Proposal issue template: https://github.com/OpenLineage/OpenLineage/pull/11

+
+
GitHub
+ + + +
Add Proposal issue template by julienledem · Pull Request #11 · OpenLineage/OpenLineage
+
+ To formalize a bit more proposal submission +
+ + + + + + + + +
Original URL: https://github.com/OpenLineage/OpenLineage/pull/11
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2020-12-22 19:55:16
 +
+

for people interested, <#C01EB6DCLHX|github-notifications> has the github integration that will notify of new PRs …

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Martin Charrel + (martin.charrel@datadoghq.com) +
+
2020-12-29 09:39:46
 +
+

👋 Hello! I'm currently working on lineage systems @ Datadog. Super excited to learn more about this effort

+ + + +
+ 👋 Willy Lulciuc +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2020-12-30 22:28:54
 +
+

*Thread Reply:* Welcome!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2020-12-30 22:29:43
 +
+

*Thread Reply:* Would you mind sharing your main use cases for collecting lineage?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marko Jamedzija + (marko@popcore.com) +
+
2021-01-03 05:54:34
 +
+

Hi! I’m also working on a similar topic for some time. Really looking forward to having these ideas standardized 🙂

+ + + +
+ 👋 Willy Lulciuc +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Alexander Gilfillan + (agilfillan@dealerinspire.com) +
+
2021-01-05 11:29:31
 +
+

I would be interested to see how to extend this to dashboards/visualizations. If that still falls with the scope of this project.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-01-05 12:55:01
 +
+

*Thread Reply:* Definitely, each dashboard should become a node in the lineage graph. That way you can understand all the dependencies of a given dashboard. SOme example of interesting metadata around this: is the dashboard updated in a timely fashion (data freshness); is the data correct (data quality)? Observing changes upstream of the dashboard will provide insights to what’s hapening when freshness or quality suffer

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Alexander Gilfillan + (agilfillan@dealerinspire.com) +
+
2021-01-05 13:20:41
 +
+

*Thread Reply:* 100%. On a granular scale, the difference between a visualization and dashboard can be interesting. One visualization can be connected to multiple dashboards. But of course this depends on the BI tool, Redash would be an example in this case.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-01-05 15:15:23
 +
+

*Thread Reply:* We would need to decide how to model those things. Possibly as a Job type for dashboard and visualization.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Alexander Gilfillan + (agilfillan@dealerinspire.com) +
+
2021-01-06 18:20:06
 +
+

*Thread Reply:* It could be. Its interesting in Redash for example you create custom queries that run at certain intervals to produce the data you need to visualize. Pretty much equivalent to job. But you then build certain visualizations off of that “job”. Then you build dashboards off of visualizations. So you could model it as an job or it could make sense for it to be more modeled like an dataset.

+ +

Thats the hard part of this. How to you model a visualization/dashboard to all the possible ways they can be created since it differs depending on how the tool you use abstracts away creating an visualization.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason Reid + (reid.david.jason@gmail.com) +
+
2021-01-05 17:06:02
 +
+

👋 Hi everyone!

+ + + +
+ 🙌 Willy Lulciuc, Arthur Wiedmer +
+ +
+ 👋 Abe Gong +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason Reid + (reid.david.jason@gmail.com) +
+
2021-01-05 17:10:22
 +
+

*Thread Reply:* Part of my role at Netflix is to oversee our data lineage story so very interested in this effort and hope to be able to participate in its success

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-01-05 18:12:48
 +
+

*Thread Reply:* Hi Jason and welcome

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-01-05 18:15:12
 +
+

A reference implementation of the OpenLineage initial spec is in progress in Marquez: https://github.com/MarquezProject/marquez/pull/880

+
+ + +
+ + + } + + henneberger + (https://github.com/henneberger) +
+ + + +
#880 Add open lineage endpoint
+
+ <p>Signed-off-by: henneberger <a href="mailto:git@danielhenneberger.com">git@danielhenneberger.com</a></p> +
+ +
+
Comments
+ 1 +
+ + + + + + + +
+ + <a href="https://github.com/MarquezProject/marquez">MarquezProject/marquez</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-01-07 12:46:19
 +
+

*Thread Reply:* The OpenLineage reference implementation in Marquez will be presented this morning Thursday (01/07) at 10AM PST, at the Marquez Community meeting.

+ +

When: Thursday, January 7th at 10AM PST +Where: https://us02web.zoom.us/j/89344845719?pwd=Y09RZkxMZHc2U3pOTGZ6SnVMUUVoQT09

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-01-07 12:46:36
 +
+

*Thread Reply:* https://wiki.lfaidata.foundation/pages/viewpage.action?pageId=18481442

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-01-07 12:46:44
 +
+

*Thread Reply:* that’s in 15 min

+ + + +
+
+
+
+ + + + + +
+ +
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-01-12 17:10:23
 +
+

*Thread Reply:* And it’s merged!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-01-12 17:10:53
 +
+

*Thread Reply:* Marquez now has a reference implementation of the initial OpenLineage spec

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jon Loyens + (jon@data.world) +
+
2021-01-06 17:43:02
 +
+

👋 Hi everyone! I'm one of the co-founder at data.world and looking forward to hanging out here

+ + + +
+ 👋 Julien Le Dem, Willy Lulciuc +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Elena Goydina + (egoydina@provectus.com) +
+
2021-01-11 11:39:20
 +
+

👋 Hi everyone! I was looking for the roadmap and don't see any. Does it exist?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-01-13 19:06:34
 +
+

*Thread Reply:* There’s no explicit roadmap so far. With the initial spec defined and the reference implementation implemented, next steps are to define more facets (for example, data shape, dataset size, etc), provide clients to facilitate integrations (java, python, …), implement more integrations (Spark in the works). Members of the community are welcome to drive their own initiatives around the core spec. One of the design goals of the facet is to enable numerous and independant parallel efforts

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-01-13 19:06:48
 +
+

*Thread Reply:* Is there something you are interested about in particular?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-01-13 19:09:42
 +
+

I have opened a proposal to move the spec to JSONSchema, this will make it more focused and decouple from http: https://github.com/OpenLineage/OpenLineage/issues/15

+
+ + +
+ + + } + + julienledem + (https://github.com/julienledem) +
+ + + +
#15 [PROPOSAL] Move the OpenLineage spec to JonSchema (from OpenAPI)
+
+ <p><strong>Purpose:</strong> <br /> +OpenAPI and JsonSchema are mostly the same model with a goal of OpenAPI to be a strict superset of JSONSchema in the near future. <br /> +The goal is to make OpenLineage independent from the underlying protocol. OpenAPI assumes a REST endpoint which is not our intent here. This would still make possible to define an OpenLineage HTTP endpoint while decoupling the model and make it more obvious that it can be used with Kafka for example.</p> + +<p><strong>Proposed implementation</strong> <br /> +Convert the OpenAPI spec to a json schema.</p> +
+ +
+
Assignees
+ julienledem +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+ 👍 Willy Lulciuc +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-01-19 12:26:39
 +
+

Here is a PR with the corresponding change: https://github.com/OpenLineage/OpenLineage/pull/17

+
+ + +
+ + + } + + julienledem + (https://github.com/julienledem) +
+ + + +
#17 adding Json-Schema for OpenLineage
+
+ <p>resolves <a href="https://github.com/OpenLineage/OpenLineage/issues/15">#15</a> <br /> +Signed-off-by: Julien Le Dem <a href="mailto:julien@apache.org">julien@apache.org</a></p> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Xinbin Huang + (bin.huangxb@gmail.com) +
+
2021-02-01 17:07:50
 +
+

Really excited to see this project! I am curious what's the current state and the roadmap of it?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-02-01 17:55:59
 +
+

*Thread Reply:* You can find the initial spec here: https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.md +The process to contribute to the model is described here: https://github.com/OpenLineage/OpenLineage/blob/main/CONTRIBUTING.md +In particular, now we’d want to contribute more facets and integrations. +Marquez has a reference implementation: https://github.com/MarquezProject/marquez/pull/880 +On the roadmap: +• define more facets: data profile, etc +• more integrations +• java/python client +You can see current discussions here: https://github.com/OpenLineage/OpenLineage/issues

+ + + +
+ ✅ Xinbin Huang +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-02-01 17:56:43
 +
+

For people curious about following github activity you can subscribe to: <#C01EB6DCLHX|github-notifications>

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-02-01 17:57:05
 +
+

*Thread Reply:* It is not on general, as it can be a bit noisy

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Zachary Friedman + (zafriedman@gmail.com) +
+
2021-02-09 13:50:17
 +
+

Random-ish question: why is producer and schemaURL nested under nominalTime facet in the spec for postRunStateUpdate? It seems like the producer of its metadata isn’t related to the time of the lineage event?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-02-09 20:02:48
 +
+

*Thread Reply:* Hi @Zachary Friedman! I replied bellow. https://openlineage.slack.com/archives/C01CK9T7HKR/p1612918909009900

+
+ + +
+ + + } + + Julien Le Dem + (https://openlineage.slack.com/team/U01DCLP0GU9) +
+ + + +
+
+ <p><code>producer</code> and <code>schemaURL</code> are defined in the <code>BaseFacet</code> type and therefore all facets (including <code>nominalTime</code>) have it. +• The <code>producer</code> is an identifier for the code that produced the metadata. The idea is that different facets in the same event can be produced by different libraries. For example In a Spark integration, Iceberg could emit it’s own facet in addition to other facets. The producer identifies what produced what. +• The <code>_schemaURL</code> is the identifier of the version of the schema for a given facet. Similarly an event could contain a mixture of Core facets from the spec as well as custom facets. This makes explicit what the definition for this facet is.</p> +
+ + + + + + +
Original URL: https://openlineage.slack.com/archives/C01CK9T7HKR/p1612918909009900
+ + +
+ + Slack Conversation +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-02-09 20:01:49
 +
+

producer and schemaURL are defined in the BaseFacet type and therefore all facets (including nominalTime) have it. +• The producer is an identifier for the code that produced the metadata. The idea is that different facets in the same event can be produced by different libraries. For example In a Spark integration, Iceberg could emit it’s own facet in addition to other facets. The producer identifies what produced what. +• The _schemaURL is the identifier of the version of the schema for a given facet. Similarly an event could contain a mixture of Core facets from the spec as well as custom facets. This makes explicit what the definition for this facet is.

+ + + +
+ 👍 Zachary Friedman +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-02-09 21:27:05
 +
+

As discussed previously, I have separated a Json Schema spec for the OpenLineage events from the OpenAPI spec defining a HTTP endpoint: https://github.com/OpenLineage/OpenLineage/pull/17

+
+ + +
+ + + } + + julienledem + (https://github.com/julienledem) +
+ + + +
#17 adding Json-Schema for OpenLineage
+
+ <p>resolves <a href="https://github.com/OpenLineage/OpenLineage/issues/15">#15</a> <br /> +Signed-off-by: Julien Le Dem <a href="mailto:julien@apache.org">julien@apache.org</a></p> +
+ +
+
Reviewers
+ @wslulciuc, @henneberger +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-02-09 21:27:26
 +
+

*Thread Reply:* Feel free to comment, this is ready to merge

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2021-02-11 20:12:18
 +
+

*Thread Reply:* Thanks, Julien. The new spec format looks great 👍

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-02-09 21:34:31
 +
+

And the corresponding code generator to start the java (and other languages) client: https://github.com/OpenLineage/OpenLineage/pull/18

+
+ + +
+ + + } + + julienledem + (https://github.com/julienledem) +
+ + + +
#18 OpenLineage java model Generator
+
+ <p>This a generator of the OpenLineage model as java classes with Jackson serialization</p> +
+ +
+
Reviewers
+ @wslulciuc +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+ 👍 Willy Lulciuc +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-02-11 22:25:24
 +
+

those are merged, we now have a jsonschema, an openapi spec that extends it and a generated java model

+ + + +
+ 🎉 Willy Lulciuc +
+ +
+ 🙌 Willy Lulciuc +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-02-17 19:39:55
 +
+

Following up on a previous discussion: +This proposal and the accompanying PR add the notion of InputFacets and OutputFacets: https://github.com/OpenLineage/OpenLineage/issues/20 +In summary, we are collecting metadata about jobs and datasets. +At the Job level, when it’s fairly static metadata (not changing every run, like the current code version of the job) it goes in a JobFacet. When it is dynamic and changes every run (like the schedule time of the run), it goes in a RunFacet. +This proposal is adding the same notion at the Dataset level: when it is static and doesn’t change every run (like the dataset schema) it goes in a Dataset facet. When it is dynamic and changes every run (like the input time interval of the dataset being read, or the statistics of the dataset being written) it goes in an inputFacet or an outputFacet. +This enables Job and Dataset versioning logic, to keep track of what changes in the definition of something vs runtime changes

+
+ + +
+ + + } + + julienledem + (https://github.com/julienledem) +
+ + + +
#20 [PROPOSAL] Add InputFacets and OutputFacets
+
+ <p><strong>Purpose:</strong> <br /> +We currently have JobFacets for everything that does not change every run (example: the current version in source control like the git sha). We have RunFacets for everything that changes every Run (example: the schedule time). <br /> +We are missing the same notion for input and output datasets. <br /> +Currently we only have Dataset facets for things that don't change every run (example: Schema). I propose to add Input and Output facets to capture things that change in datasets every run (ex: number of row produced, etc)</p> + +<p><strong>Proposed implementation</strong> <br /> +Add InputFacets and OutputFacets in the core model. Directly in the Input or Output field.</p> + +<pre><code>"inputs": { + "description": "The set of ****input**** datasets.", + "type": "array", + "items": { + "allOf": [ + { "$ref": "#/definitions/Dataset" }, + { + "type": "object", + "properties": { + "inputFacets": { + ... + } + } + } + ] + } + } + +</code></pre> +
+ +
+
Labels
+ proposal +
+ +
+
Comments
+ 1 +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+ 👍 Kevin Mellott, Petr Šimeček +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-02-19 14:27:23
 +
+

*Thread Reply:* @Kevin Mellott and @Petr Šimeček Thanks for the confirmation on this slack message. To make your comment visible to the wider community, please chime in on the github issue as well: https://github.com/OpenLineage/OpenLineage/issues/20 +Thank you.

+
+ + +
+ + + } + + julienledem + (https://github.com/julienledem) +
+ + + +
#20 [PROPOSAL] Add InputFacets and OutputFacets
+
+ <p><strong>Purpose:</strong> <br /> +We currently have JobFacets for everything that does not change every run (example: the current version in source control like the git sha). We have RunFacets for everything that changes every Run (example: the schedule time). <br /> +We are missing the same notion for input and output datasets. <br /> +Currently we only have Dataset facets for things that don't change every run (example: Schema). I propose to add Input and Output facets to capture things that change in datasets every run (ex: number of row produced, etc)</p> + +<p><strong>Proposed implementation</strong> <br /> +Add InputFacets and OutputFacets in the core model. Directly in the Input or Output field.</p> + +<pre><code>"inputs": { + "description": "The set of ****input**** datasets.", + "type": "array", + "items": { + "allOf": [ + { "$ref": "#/definitions/Dataset" }, + { + "type": "object", + "properties": { + "inputFacets": { + ... + } + } + } + ] + } + } + +</code></pre> +
+ +
+
Labels
+ proposal +
+ +
+
Comments
+ 1 +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-02-19 14:27:46
 +
+

*Thread Reply:* The PR is out for this: https://github.com/OpenLineage/OpenLineage/pull/23

+
+ + +
+ + + } + + julienledem + (https://github.com/julienledem) +
+ + + +
#23 add input and output facets
+
+ <p>This PR resolves <a href="https://github.com/OpenLineage/OpenLineage/issues/20">#20</a> <br /> +Signed-off-by: Julien Le Dem <a href="mailto:julien@apache.org">julien@apache.org</a></p> +
+ +
+
Reviewers
+ @jcampbell, @abegong, @henneberger +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Weixi Li + (ashlee.happy@gmail.com) +
+
2021-02-19 04:14:59
 +
+

Hi, I am really interested in this project and Marquez. I am a bit not clear about the differences and relationship between those two projects. As my understanding, OpenLineage provides an api specification for other tools running jobs (e.g. Spark, Airflow) to send out an event to update the run state of the job, then for example Marquez can be the destination for those events and show the data lineage from those run state updates. When you are saying there is an reference implementation of the OpenLineage spec in Marquez, do you mean there is an /lineage endpoint implemented in the Marquez api https://github.com/MarquezProject/marquez/blob/main/api/src/main/java/marquez/api/OpenLineageResource.java? Then my question is what is next step after Marquez has this api? How does Marquez use that endpoint to integrate with airflow for example? I did not find the usage of that endpoint in Marquez project. The library marquez-airflow which integrates Airflow with Marquez seems like only use the other marquez apis to build the data lineage. Or did I misunderstand something? Thank you very much!

+
+ + + + +
api/src/main/java/marquez/api/OpenLineageResource.java
+
+ <pre><code>/** + ** Licensed under the Apache License, Version 2.0 (the "License"); + ** you may not use this file except in compliance with the License. + ** You may obtain a copy of the License at + ** + ** &lt;http://www.apache.org/licenses/LICENSE-2.0&gt; + ** + ** Unless required by applicable law or agreed to in writing, software + ** distributed under the License is distributed on an "AS IS" BASIS, + ** WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + ** See the License for the specific language governing permissions and + ** limitations under the License. + **/ + +package marquez.api; + +import static javax.ws.rs.core.MediaType.APPLICATION_JSON; +import static javax.ws.rs.core.Response.Status.INTERNAL_SERVER_ERROR; + +import com.codahale.metrics.annotation.ExceptionMetered; +import com.codahale.metrics.annotation.ResponseMetered; +import com.codahale.metrics.annotation.Timed; +import com.fasterxml.jackson.core.JsonProcessingException; +import java.sql.SQLException; +import javax.validation.Valid; +import javax.validation.constraints.NotNull; +import javax.ws.rs.Consumes; +import &lt;a href="http://javax.ws.rs.POST"&gt;javax.ws.rs.POST&lt;/a&gt;; +import javax.ws.rs.Path; +import javax.ws.rs.Produces; +import javax.ws.rs.container.AsyncResponse; +import javax.ws.rs.container.Suspended; +import javax.ws.rs.core.Response; +import lombok.extern.slf4j.Slf4j; +import marquez.service.OpenLineageService; +import marquez.service.models.LineageEvent; + +@Slf4j +@Path("/api/v1/lineage") +public class OpenLineageResource { + private final OpenLineageService openLineageService; + + public OpenLineageResource(OpenLineageService openLineageService) { + this.openLineageService = openLineageService; + } + + @Timed + @ResponseMetered + @ExceptionMetered + @POST + @Consumes(APPLICATION_JSON) + @Produces(APPLICATION_JSON) + public void create( + @Valid @NotNull LineageEvent event, @Suspended final AsyncResponse asyncResponse) + throws JsonProcessingException, SQLException { + openLineageService + .createAsync(event) + .whenComplete( + (result, err) -&amp;gt; { + if (err != null) { + log.error("Unexpected error while processing request", err); + asyncResponse.resume(Response.status(INTERNAL_SERVER_ERROR).build()); + } else { + asyncResponse.resume(Response.status(201).build()); + } + }); + } +} + +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/MarquezProject/marquez">MarquezProject/marquez</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Weixi Li + (ashlee.happy@gmail.com) +
+
2021-02-19 05:03:21
 +
+

*Thread Reply:* Okay, I found the spark integration in Marquez calls the /lineage endpoint. But I am still curious about the future plan to integrate with other tools, like airflow?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-02-19 12:41:23
 +
+

*Thread Reply:* Just restating some of my answers from teh marquez slack for the benefits of folks here.

+ +

• OpenLineage defines the schema to collect metadata +• Marquez has a /lineage endpoint implementing the OpenLineage spec to receive this metadata, implemented by the OpenLineageResource you pointed out +• In the future other projects will also have OpenLineage endpoints to receive this metadata +•  The Marquez Spark integration produces OpenLineage events: https://github.com/MarquezProject/marquez/tree/main/integrations/spark +• The Marquez airflow integration still uses the original marquez api but will be migrated to open lineage. +• All new integrations will use OpenLineage metadata

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Weixi Li + (ashlee.happy@gmail.com) +
+
2021-02-22 03:55:18
 +
+

*Thread Reply:* thank you! very clear answer🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ernie Ostic + (ernie.ostic@getmanta.com) +
+
2021-03-02 13:49:04
 +
+

Hi Everyone. Just got started with the Marquez REST API and a little bit into the Open Lineage aspects. Very easy to use. Great work on the curl examples for getting started. I'm working with Postman and am happy to share a collection I have once I finish testing. A question about tags --- are there plans for a "post new tag" call in the API? ...or maybe I missed it. Thx. --ernie

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-03-02 17:51:29
 +
+

*Thread Reply:* I forgot to reply in thread 🙂 https://openlineage.slack.com/archives/C01CK9T7HKR/p1614725462008300

+
+ + +
+ + + } + + Julien Le Dem + (https://openlineage.slack.com/team/U01DCLP0GU9) +
+ + + +
+
+ <p>OpenLineage doesn’t have a Tag facet yet (but tags are defined in the Marquez api). Feel free to open a proposal on the github repo. <a href="https://github.com/OpenLineage/OpenLineage/issues/new/choose">https://github.com/OpenLineage/OpenLineage/issues/new/choose</a></p> +
+ + + + + + +
Original URL: https://openlineage.slack.com/archives/C01CK9T7HKR/p1614725462008300
+ + +
+ + Slack Conversation +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-03-02 17:51:02
 +
+

OpenLineage doesn’t have a Tag facet yet (but tags are defined in the Marquez api). Feel free to open a proposal on the github repo. https://github.com/OpenLineage/OpenLineage/issues/new/choose

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-03-16 11:21:37
 +
+

Hey everyone. What's the story for stream processing (like Flink jobs) for OpenLineage? +It does not fit cleanly with runEvent model, which +It is required to issue 1 START event and 1 of [ COMPLETE, ABORT, FAIL ] event per run. +as unbounded stream jobs usually do not complete.

+ +

I'd imagine few "workarounds" that work for some cases - for example, imagine a job calculating hourly aggregations of transactions and dumpling them into parquet files for further analysis. The job could issue OTHER event type adding additional output dataset every hour. Another option would be to create new "run" every hour, just indicating the added data.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Adam Bellemare + (adam.bellemare@shopify.com) +
+
2021-03-16 15:07:04
 +
+

*Thread Reply:* Ha, I signed up just to ask this precise question!

+ + + +
+ 😀 Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Adam Bellemare + (adam.bellemare@shopify.com) +
+
2021-03-16 15:07:44
 +
+

*Thread Reply:* I’m still looking into the spec myself. Are we required to have 1 or more runs per Job? Or can a Job exist without a run event?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ravi Suhag + (suhag.ravi@gmail.com) +
+
2021-04-02 07:24:39
 +
+

*Thread Reply:* Run event can be emitted when it starts. and it can stay in RUNNING state unless something happens to the job. Additionally, you could send event periodically as state RUNNING to inform the system that job is healthy.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Adam Bellemare + (adam.bellemare@shopify.com) +
+
2021-03-16 15:09:31
 +
+

Similar to @Maciej Obuchowski question about Flink / Streaming jobs - what about Streaming sources (eg: a Kafka topic)? It does fit into the dataset model, more or less. But, has anyone used this yet for a set of streaming sources? Particularly with schema changes over time?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-03-16 18:30:46
 +
+

Hi @Maciej Obuchowski and @Adam Bellemare, streaming jobs are meant to be covered by the spec but I agree there are a few details to iron out.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-03-16 18:31:55
 +
+

In particular, streaming job still have runs. If they run continuously they do not run forever and you want to track that a job has been started at a point in time with a given version of the code, then stopped and started again after being upgraded for example.

+ + + +
+ 👍 Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-03-16 18:32:23
 +
+

I agree with @Maciej Obuchowski that we would also send OTHER events to keep track of progress.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-03-16 18:32:46
 +
+

For example one could track checkpointing this way.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-03-16 18:35:35
 +
+

For a Kafka topic you could have streaming dataset specific facets or even Kafka specific facets (ex: list of offsets we stopped reading at, schema id, etc )

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-03-17 10:05:53
 +
+

*Thread Reply:* That's good idea.

+ +

Now I'm wondering - let's say we want to track on which offset checkpoint ended processing. That would mean we want to expose checkpoint id, time, and offset. +I suppose we don't want to overwrite previous checkpoint info, so we want to have some collection of data in this facet.

+ +

Something like appendable facets would be nice, to just add new checkpoint info to the collection, instead of having to push all the checkpoint infos all the time we just want to add new data point.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-03-16 18:45:23
 +
+

Let me know if you have more thoughts

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Adam Bellemare + (adam.bellemare@shopify.com) +
+
2021-03-17 09:18:49
 +
+

*Thread Reply:* Thanks Julien! I will try to wrap my head around some use-cases and see how it maps to the current spec. From there, I can see if I can figure out any proposals

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-03-17 13:43:29
 +
+

*Thread Reply:* You can use the proposal issue template to propose a new facet for example: https://github.com/OpenLineage/OpenLineage/issues/new/choose

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Carlos Zubieta + (carlos.zubieta@wizeline.com) +
+
2021-03-16 18:49:00
 +
+

Hi everyone, I just hear about OpenLineage and would like to learn more about it. The talks in the repo explain nicely the purpose and general ideas but I have a couple of questions. Are there any working implementations to produce/consume the spec? Also, are there any discussions/guides standard information, naming conventions, etc. in the facets?

+ + + +
+
+
+
+ + + + + +
+ +
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-03-16 20:05:06
 +
+

Hi @Carlos Zubieta here are some pointers ^

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-03-16 20:06:51
 +
+

Marquez has a reference implementation of an OpenLineage endpoint. The Spark integration emits OpenLineage events.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Carlos Zubieta + (carlos.zubieta@wizeline.com) +
+
2021-03-16 20:56:37
 +
+

Thank you @Julien Le Dem!!! Will take a close look

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Adam Bellemare + (adam.bellemare@shopify.com) +
+
2021-03-17 15:41:50
 +
+

Q related to People/Teams/Stakeholders/Owners with regards to Jobs and Datasets (didn’t find anything in search): +Let’s say I have a dataset , and there are a number of other downstream jobs that ingest from it. In the case that the dataset is mutated in some way (or deleted, archived, etc), how would I go about notifying the stakeholders of that set about the changes?

+ +

Just to be clear, I’m not concerned about the mechanics of doing this, just that there is someone that needs to be notified, who has self-registered on this set. +Similarly, I want to manage the datasets I am concerned about , where I can grab a list of all the datasets I tagged myself on.

+ +

This seems to suggest that we could do with additional entities outside of Dataset, Run, Job. However, at the same time, I can see how this can lead to an explosion of other entities. Any thoughts on this particular domain? I think I could achieve something similar with aspects, but this would require that I update the aspect on each entity if I want to wholesale update the user contact, say their email address.

+ +

Has anyone else run into something like this? Have you any advice? Or is this something that may be upcoming in the spec?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Adam Bellemare + (adam.bellemare@shopify.com) +
+
2021-03-17 16:42:24
 +
+

*Thread Reply:* One thing we were considering is just adding these in as Facets ( Tags as per Marquez), and then plugging into some external people managing system. However, I think the question can be generalized to “should there be some sort of generic entity that can enable relationships between itself and Datasets, Jobs, Runs) as part of an integration element?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-03-18 16:03:55
 +
+

*Thread Reply:* That’s a great topic of discussion. I would definitely use the OpenLineage facets to capture what you describe as aspect above. The current Marquez model has a simple notion of ownership at the namespace model but this need to be extended to enable use cases you are describing (owning a dataset or a job) . Right now the owner is just a generic identifier as a string (a user id or a group id for example). Once things are tagged (in some way), you can use the lineage API to find all the downstream or upstream jobs and datasets. In OpenLineage I would start by being able to capture the owner identifier in a facet with contact info optional if it’s available at runtime. It will have the advantage of keeping track of how that changed over time. This definitely deserves its own discussion.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-03-18 17:52:13
 +
+

*Thread Reply:* And also to make sure I understand your use case, you want to be able to notify the consumers of a dataset that it is being discontinued/replaced/… ? What else are you thinking about?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Adam Bellemare + (adam.bellemare@shopify.com) +
+
2021-03-22 09:15:19
 +
+

*Thread Reply:* Let me pull in my colleagues

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Adam Bellemare + (adam.bellemare@shopify.com) +
+
2021-03-22 09:15:24
 +
+

*Thread Reply:* Standby

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Olessia D'Souza + (olessia.dsouza@shopify.com) +
+
2021-03-22 10:59:57
 +
+

*Thread Reply:* 👋 Hi Julien. I’m Olessia, I’m working on the metadata collection implementation with Adam. Some thought on this:

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Olessia D'Souza + (olessia.dsouza@shopify.com) +
+
2021-03-22 11:00:45
 +
+

*Thread Reply:* To start off, we’re thinking that there often isn’t a single owner, but rather a set of Stakeholders that evolve over time. So we’d like to be able to attach multiple entries, possibly of different types, to a Dataset. We’re also thinking that a dataset should have at least one owner. So a few things I’d like to confirm/discuss options:

  • If I were to stay true to the spec as it’s defined atm I wouldn’t be able to add a required facet. True/false?
  • According to the readme, “...emiting a new facet with the same name for the same entity replaces the previous facet instance for that entity entirely”. If we were to store multiple stakeholders, we’d have a field “stakeholders” and its value would be a list? This would make queries involving stakeholders not very straightforward. If the facet is overwritten every time, how do I a) add individuals to the list b) track changes to the list over time. Let me know what I’m missing, because based on what you said above tracking facet changes over time is possible.
  • Run events are issued by a scheduler. Why should it be in the domain of the scheduler to know the entire list of Stakeholders?
  • I noticed that Marquez has separate endpoints to capture information about Datasets, and some additional information beyond what’s described in the spec is required. In this context, we could add a required Stakeholder facets on a Dataset, and potentially even additional end points to add and remove Stakeholders. Is that a valid way to go about this, in your opinion?
    • +
+ +

Curious to hear your thoughts on all of this!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-03-24 17:06:50
 +
+

*Thread Reply:* > To start off, we’re thinking that there often isn’t a single owner, but rather a set of Stakeholders that evolve over time. So we’d like to be able to attach multiple entries, possibly of different types, to a Dataset. We’re also thinking > that a dataset should have at least one owner. So a few things I’d like to confirm/discuss options: +> -> If I were to stay true to the spec as it’s defined atm I wouldn’t be able to add a required facet. True/false? +Correct, The spec defines what facets looks like (and how you can make your own custom facets) but it does not make statements about whether facets are required. However, you can have your own validation and make certain things required if you wish to on the client side? +  +> - According to the readme, “...emiting a new facet with the same name for the same entity replaces the previous facet instance for that entity entirely”. If we were to store multiple stakeholders, we’d have a field “stakeholders” and its value would be a list?  +Yes, I would indeed consider such a facet on the dataset with the stakeholder.

+ +

> This would make queries involving stakeholders not very straightforward. If the facet is overwritten every time, how do I  +> a) add individuals to the list +You would provide the new list of stake holders. OpenLineage standardizes lineage collection and defines a format for expressing metadata. Marquez will keep track of how metadata has evolved over time.

+ +

> b) track changes to the list over time. Let me know what I’m missing, because based on what you said above tracking facet changes over time is possible. +Each event is an observation at a point in time. In a sense they are each immutable. There’s a “current” version but also all the previous ones stored in Marquez. +Marquez stores each version of a dataset it received through OpenLineage and exposes an API to see how that evolved over time.

+ +

> - Run events are issued by a scheduler. Why should it be in the domain of the scheduler to know the entire list of Stakeholders? +The scheduler emits the information that it knows about. For example: “I started this job and it’s reading from this dataset and is writing to this other dataset.” +It may or may not be in the domain of the scheduler to know the list of stakeholders. If not then you could emit different types of events to add a stakeholder facet to a dataset. We may want to refine the spec for that. Actually I would be curious to hear what you think should be the source of truth for stakeholders. It is not the intent to force everything coming from the scheduler.

  • example 1: stakeholders are people on call for the job, they are defined as part of the job and that also enables alerting
  • example 2: stakeholders are consumers of the jobs: they may be defined somewhere else
    • +
+ +

> - I noticed that Marquez has separate endpoints to capture information about Datasets, and some additional information beyond what’s described in the spec is required. In this context, we could add a required Stakeholder facets on a Dataset, and potentially even additional end points to add and remove Stakeholders. Is that a valid way to go about this, in your opinion?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-03-24 17:06:50
 +
+

*Thread Reply:* +Marquez existed before OpenLineage. In particular the /run end-point to create and update runs will be deprecated as the OpenLineage /lineage endpoint replaces it. At the moment we are mapping OpenLineage metadata to Marquez. Soon Marquez will have all the facets exposed in the Marquez API. (See: https://github.com/MarquezProject/marquez/pull/894/files) +We could make Marquez Configurable or Pluggable for validation purposes. There is already a notion of LineageListener for example. +Although Marquez collects the metadata. I feel like this validation would be better upstream or with some some other mechanism. The question is when do you create a dataset vs when do you become a stakeholder? What are the various stakeholder and what is the responsibility of the minimum one stakeholder? I would probably make it required to deploy the job that the stakeholder is defined. This would apply to the output dataset and would be collected in Marquez.

+ +

In general, you are very welcome to make suggestion on additional endpoints for Marquez and I’m happy to discuss this further as those ideas are progressing.

+ +

> Curious to hear your thoughts on all of this! +Thanks for taking the time!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-05-24 16:27:03
 +
+

*Thread Reply:* https://openlineage.slack.com/archives/C01CK9T7HKR/p1621887895004200

+
+ + +
+ + + } + + Julien Le Dem + (https://openlineage.slack.com/team/U01DCLP0GU9) +
+ + + +
+
+ <p>I have created a channel to discuss &lt;#C022MMLU31B|user-generated-metadata> since this came up in a few discussions.</p> +
+ + + + + + +
Original URL: https://openlineage.slack.com/archives/C01CK9T7HKR/p1621887895004200
+ + +
+ + Slack Conversation +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-03-24 18:58:00
 +
+

Thanks for the Python client submission @Maciej Obuchowski +https://github.com/OpenLineage/OpenLineage/pull/34

+
+ + +
+ + + } + + mobuchowski + (https://github.com/mobuchowski) +
+ + + +
#34 OpenLineage python client implementation
+
+ <p>Signed-off-by: Maciej Obuchowski <a href="mailto:maciej.obuchowski@getindata.com">maciej.obuchowski@getindata.com</a></p> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+ 🙌 Willy Lulciuc +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-03-24 18:59:50
 +
+

I also have added a spec to define a standard naming policy. Please review: https://github.com/OpenLineage/OpenLineage/pull/31/files

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-03-31 23:45:35
 +
+

We now have a python client! Thanks @Maciej Obuchowski

+ + + +
+ 👍 Maciej Obuchowski, Kevin Mellott, Ravi Suhag, Ross Turk, Willy Lulciuc, Mirko Raca +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Zachary Friedman + (zafriedman@gmail.com) +
+
2021-04-02 19:37:36
 +
+

Question, what do you folks see as the canonical mechanism for receiving OpenLineage events? Do you see an agent like statsd? Or do you see this as purely an API spec that services could implement? Do you see producers of lineage data writing code to send formatted OpenLineage payloads to arbitrary servers that implement receipt of these events? Curious what the long-term vision is here related to how an ecosystem of producers and consumers of payloads would interact?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-04-02 19:54:52
 +
+

*Thread Reply:* Marquez is the reference implementation for receiving events and tracking changes. But the definition of the API let’s other receive them (and also enables using openlineage events to sync between systems)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-04-02 19:55:32
 +
+

*Thread Reply:* In particular, Egeria is involved in enabling receiving and emitting openlineage

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Zachary Friedman + (zafriedman@gmail.com) +
+
2021-04-03 18:03:01
 +
+

*Thread Reply:* Thanks @Julien Le Dem. So to get specific, if dbt were to emit OpenLineage events, how would this work? Would dbt Cloud hypothetically allow users to configure an endpoint to send OpenLineage events to, similar in UI implementation to configuring a Stripe webhook perhaps? And then whatever server the user would input here would point to somewhere that implements receipt of OpenLineage payloads? This is all a very hypothetical example, but trying to ground it in something I have a solid mental model for.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2021-04-05 17:51:57
 +
+

*Thread Reply:* hypothetically speaking, that all sounds right. so a user, who, e.g., has a dbt pipeline and an AWS glue pipeline could configure both of those projects to point to the same open lineage service and get their entire lineage graph even if the two pipelines aren't connected.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2021-04-06 20:33:51
 +
+

*Thread Reply:* Yeah, OpenLineage events need to be published to a backend (can be Kafka, can be a graphDB, etc). Your Stripe webhook analogy is aligned with how events can be received. For example, in Marquez, we expose a /lineage endpoint that consumes OpenLineage events. We then map an OpenLineage event to the Marquez model (sources, datasets, jobs, runs) that’s persisted in postgres.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Zachary Friedman + (zafriedman@gmail.com) +
+
2021-04-07 10:47:06
 +
+

*Thread Reply:* Thanks both!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-04-13 20:52:53
 +
+

*Thread Reply:* sorry, I was away last week. Yes that sounds right.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Moravec + (jkb.moravec@gmail.com) +
+
2021-04-07 09:41:09
 +
+

Hi everyone, I just started discovering OpenLineage and Marquez, it looks great and the quick-start tutorial is very helpful! One question though, I pushed some metadata to Marquez using the Lineage POST endpoint, and when I try to confirm that everything was created using Marquez REST API, everything is there ... but I don't see these new objects in the Marquez UI... what is the best way how to investigate where the issue is?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2021-04-14 13:12:31
 +
+

*Thread Reply:* Welcome, @Jakub Moravec 👋 . Given that you're able to retrieve metadata using the marquezAPI, you should be able to also view dataset and job metadata in the UI. Mind using the search bar in the top right-hand corner in the UI to see if your metadata is searchable? The UI only renders jobs and datasets that are connected in the lineage graph. We're working towards a more general metadata exploration experience, but currently the lineage graph is the main experience.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakob Külzer + (jakob.kulzer@shopify.com) +
+
2021-04-08 11:23:18
 +
+

Hi friends, we're exploring OpenLineage and while building out integration for existing systems we realized there is no obvious way for an input to specify what "version" of that dataset is being consumed. For example, we have a job that rolls up a variable number of what OpenLineage calls dataset versions. By specifying only that dataset, we can't represent the specific instances of it that are actually rolled up. We think that would be a very important part of the lineage graph.

+ +

Are there any thoughts on how to address specific dataset versions? Is this where custom input facets would come to play?

+ +

Furthermore, based on the spec, it appears that events can provide dataset facets for both inputs and outputs and this seems to open the door to race conditions in which two runs concurrently create dataset versions of a dataset. Is this where the eventTime field is supposed to be used?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-04-13 20:56:42
 +
+

*Thread Reply:* Your intuition is right here. I think we should define an input facet that specifies which dataset version is being read. Similarly you would have an output facet that specifies what version is being produced. This would apply to storage layers like Deltalake and Iceberg as well.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-04-13 20:57:58
 +
+

*Thread Reply:* Regarding the race condition, input and output facets are attached to the run. The version of the dataset that was read is an attribute of a run and should not modify the dataset itself.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-04-13 21:01:34
 +
+

*Thread Reply:* See the Dataset description here: https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.md#core-lineage-model

+
+ + + + +
spec/OpenLineage.md
+
+ <pre><code># OpenLineage Spec + +## Specification + +The specification for OpenLineage is formalized as a JsonSchema [OpenLineage.json](OpenLineage.json). +An OpenAPI spec is also provided for HTTP based implementations: [OpenLineage.yml](OpenLineage.yml) +The documentation is published at: &lt;https://openlineage.github.io/&gt; +It allows extensions to the spec using `Custom Facets` as described in this document. + +## Core concepts + +### Core Lineage Model + +![Open Lineage model](OpenLineageModel.svg) + +- ****Run Event****: and event describing an observed state of a job run. It is required to at least send one event for a START transition and a COMPLETE/FAIL/ABORT transition. Aditional events are optional. + +- ****Job****: a process definition that consumes and produces datasets (defined as its inputs and outputs). It is identified by a unique name within a namespace (which is typicaly assigned to the scheduler starting the jobs). The **Job** evolves over time and this change is captured when the job runs. + +- ****Dataset****: an abstract representation of data. It has a unique name within a namespace derived from its physical location (for example db.host.database.schema.table). Typicaly, a **Dataset** changes when a job writing to it completes. Similarly to the **Job** and **Run** distinction, metadata that is more static from run to run is captured in a DatasetFacet (for example, the schema that does not change every run), what changes every **Run** is captured as an **InputFacet** or an **OutputFacet** (for example, what subset of the data set was read or written, like a time partition). + +- ****Run****: An instance of a running job with a start and completion (or failure) time. It is uniquely identified by an id relative to its job definition. + +- ****Facet****: A piece of metadata attached to one of the entities defined above. + +example: +Here is an example of a simple start run event not adding any facet information: +</code></pre> + +<p>{ + "transition": "START", + "eventTime": "2020-12-09T23:37:31.081Z", + "run": { + "runId": "345", + }, + "job": { + "namespace": "my-scheduler-namespace", + "name": "myjob.mytask", + }, + "inputs": [ + { + "namespace": "my-datasource-namespace", + "name": "instance.schema.table", + } + ], + "outputs": [ + { + "namespace": "my-datasource-namespace", + "name": "instance.schema.output_table", + } + ], + "producer": "<a href="https://github.com/OpenLineage/OpenLineage/blob/v1-0-0/client">https://github.com/OpenLineage/OpenLineage/blob/v1-0-0/client</a>" +}</p> + +<pre><code> +### Lifecycle + +The OpenLineage API defines events to capture the lifecycle of a **Run** for a given **Job**. +When a **job** is being **run**, we capture metadata by sending run events when the state of the job transitions to a different state. +We might observe different aspects of the job run at different stages. This means that different metadata might be collected in each event during the lyfecycle of a run. +All metadata is additive. for example, if more inputs or outputs are detected as the job is running we might send additional events specifically for those datasets without re-emiting previously observed inputs or outputs. +Example: + - When the run starts, we collect the following Metadata: + - Run Id + - Job id + - transition: START + - event time + - source location and version (ex: git sha) + - If known: Job inputs and outputs. (input schema, ...) + - When the run completes: + - Run Id + - Job id + - transition: COMPLETE + - event time + - Output datasets schema (and other metadata). + +### Facets + +Facets are pieces of metadata that can be attached to the core entities: +- Run +- Job +- Dataset (Inputs or Outputs) + +A facet is an atomic piece of metadata identified by its name. This means that emiting a new facet whith the same name for the same entity replaces the previous facet instance for that entity entirely). It is defined as a JSON object that can be either part of the spec or custom facets defined in a different project. + +Custom facets must use a distinct prefix named after the project defining them to avoid colision with standard facets defined in the [OpenLineage.json](OpenLineage.json) spec. +They have a \_schemaURL field pointing to the corresponding version of the facet schema (as a JSONPointer: [$ref URL location](&lt;https://swagger.io/docs/specification/using-ref/&gt;) ). + +Example: &lt;https://raw.githubusercontent.com/OpenLineage/OpenLineage/main/spec/OpenLineage.json#/definitions/MyCustomJobFacet&gt; + +The versioned URL must be an immutable pointer to the version of the facet schema. For example, it should include a tag of a git sha and not a branch name. This should also be a canonical URL. There should be only one URL used for a given version of a schema. + +Custom facets can be promoted to the standard by including them in the spec. + +### Standard Facets + +#### Run Facets + +- ****nominalTime****: Captures the time this run is scheduled for. This is a typical usage for time based scheduled job. The job has a nominal schedule time that will be different from the actual time it is running at. + +- ****parent****: Captures the parent job and Run when the run was spawn from a parent run. For example in the case of Airflow, there's a run for the DAG that then spawns runs for individual tasks that would refer to the parent run as the DAG run. Similarly when a SparkOperator starts a Spark job, this creates a separate run that refers to the task run as its parent. + +#### Job Facets + +- ****sourceCodeLocation****: Captures the source code location and version (example: git sha) of the job. + +- ****sql****: Capture the SQL query if this job is a SQL query. + +#### Dataset Facets + +- ****schema****: Captures the schema of the dataset + +- ****dataSource****: Captures the Database instance containing this datasets (ex: Database schema. Object store bucket, ...) + +#### Input Dataset Facets + +**Note: for now there no input facets defined** + +#### Output Dataset Facets + +**Note: for now there no output facets defined** + +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Stephen Pimentel + (stephenpiment@gmail.com) +
+
2021-04-14 18:20:42
 +
+

Hi everyone! I’m exploring what existing, open-source integrations are available, specifically for Spark, Airflow, and Trino (PrestoSQL). My team is looking both to use and contribute to these integrations. I’m aware of the integration in the Marquez repo: +• Spark: https://github.com/MarquezProject/marquez/tree/main/integrations/spark +• Airflow: https://github.com/MarquezProject/marquez/tree/main/integrations/airflow +Are there other efforts I should be aware of, whether for these two or for Trino? Thanks for any information!

+ + + +
+ 👋 Arthur Wiedmer, Maciej Obuchowski, Peter Hicks +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Zachary Friedman + (zafriedman@gmail.com) +
+
2021-04-19 16:17:06
 +
+

*Thread Reply:* I think for Trino integration you'd be looking at writing a Trino extractor if I'm not mistaken, yes?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Zachary Friedman + (zafriedman@gmail.com) +
+
2021-04-19 16:17:23
 +
+

*Thread Reply:* But extractor would obviously be at the Marquez layer not OpenLineage

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Zachary Friedman + (zafriedman@gmail.com) +
+
2021-04-19 16:19:00
 +
+

*Thread Reply:* And hopefully the metadata you'd be looking to extract from Trino wouldn't have any connector-specific syntax restrictions.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Antonio Moctezuma + (antoniomoctezuma@northwesternmutual.com) +
+
2021-04-16 15:37:24
 +
+

Hey all! Right now I am working on getting OpenLineage integrated with some microservices here at Northwestern Mutual and was looking for some advice. The current service I am trying to integrate it with moves files from one AWS S3 bucket to another so i was hoping to track that movement with OpenLineage. However by my understanding the inputs that would be passed along in a runEvent are meant to be datasets that have schema and other properties. But I wanted to have that input represent the file being moved. Is this a proper usage of Open Lineage? Or is this a use case that is still being developed? Any and all help is appreciated!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-04-19 21:42:14
 +
+

*Thread Reply:* This is a proper usage. That schema is optional if it’s not available.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-04-19 21:43:27
 +
+

*Thread Reply:* You would model it as a job reading from a folder (the input dataset) in the input bucket and writing to a folder (the output dataset) in the output bucket

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-04-19 21:43:58
 +
+

*Thread Reply:* This is similar to how this is modeled in the spark integration (spark job reading and writing to s3 buckets)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-04-19 21:47:06
 +
+

*Thread Reply:* for reference: getting the urls for the inputs: https://github.com/MarquezProject/marquez/blob/c5e5d7b8345e347164aa5aa173e8cf35062[…]marquez/spark/agent/lifecycle/plan/HadoopFsRelationVisitor.java

+
+ + + + +
<https://github.com/MarquezProject/marquez/blob/c5e5d7b8345e347164aa5aa173e8cf35062641e8/integrations/spark/src/main/java/marquez/spark/agent/lifecycle/plan/HadoopFsRelationVisitor.java | HadoopFsRelationVisitor.java>
+
+ <pre><code> public List&amp;lt;Dataset&amp;gt; apply(LogicalPlan x) { +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/MarquezProject/marquez">MarquezProject/marquez</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-04-19 21:47:54
 +
+

*Thread Reply:* getting the output URL: https://github.com/MarquezProject/marquez/blob/c5e5d7b8345e347164aa5aa173e8cf35062[…]ark/agent/lifecycle/plan/InsertIntoHadoopFsRelationVisitor.java

+
+ + + + +
<https://github.com/MarquezProject/marquez/blob/c5e5d7b8345e347164aa5aa173e8cf35062641e8/integrations/spark/src/main/java/marquez/spark/agent/lifecycle/plan/InsertIntoHadoopFsRelationVisitor.java | InsertIntoHadoopFsRelationVisitor.java>
+
+ <pre><code> return Collections.singletonList( +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/MarquezProject/marquez">MarquezProject/marquez</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-04-19 21:48:48
 +
+

*Thread Reply:* See the spec (comments welcome) for the naming of S3 datasets: https://github.com/OpenLineage/OpenLineage/pull/31/files#diff-e3a8184544e9bc70d8a12e76b58b109051c182a914f0b28529680e6ced0e2a1cR87

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Antonio Moctezuma + (antoniomoctezuma@northwesternmutual.com) +
+
2021-04-20 11:11:38
 +
+

*Thread Reply:* Hey Julien, thank you so much for getting back to me. I'll take a look at the documentation/implementations you've sent me and will reach out if I have anymore questions. Thanks again!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Antonio Moctezuma + (antoniomoctezuma@northwesternmutual.com) +
+
2021-04-20 17:39:24
 +
+

*Thread Reply:* @Julien Le Dem I left a quick comment on that spec PR you mentioned. Just wanted to let you know.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-04-20 17:49:15
 +
+

*Thread Reply:* thanks

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Josh Quintus + (josh.quintus@gmail.com) +
+
2021-04-28 09:41:45
 +
+

Hello all. I was reading through the OpenLineage documentation on GitHub and noticed a very minor typo (an instance where and should have been an). I was just about to create a PR for it but wanted to check with someone to see if that would be something that the team is interested in.

+ +

Thanks for the tool, I'm looking forward to learning more about it.

+ + + +
+ 👍 Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-04-28 20:56:53
 +
+

*Thread Reply:* Thank you! Please do fix typos, I’ll approve your PR.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Josh Quintus + (josh.quintus@gmail.com) +
+
2021-04-28 23:21:44
 +
+

*Thread Reply:* No problem. Here's the PR. https://github.com/OpenLineage/OpenLineage/pull/47

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Josh Quintus + (josh.quintus@gmail.com) +
+
2021-04-28 23:22:41
 +
+

*Thread Reply:* Once I fixed the ones I saw I figured "Why not just run it through a spell checker just in case... " and found a few additional ones.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2021-05-20 16:30:05
 +
+

For your enjoyment, @Julien Le Dem was on the Data Engineering Podcast talking about OpenLineage!

+ +

https://www.dataengineeringpodcast.com/openlineage-data-lineage-specification-episode-187/

+
+
Data Engineering Podcast
+ + + +
Unlocking The Power of Data Lineage In Your Platform with OpenLineage - Episode 187
+
+ Data lineage is the common thread that ties together all of your data pipelines, workflows, and systems. In order to get a holistic understanding of your data +
+ + + + + + + + +
Original URL: https://www.dataengineeringpodcast.com/openlineage-data-lineage-specification-episode-187/
+ + + +
+ + + +
+ 🙌 Willy Lulciuc, Maciej Obuchowski, Peter Hicks, Mario Measic +
+ +
+ ❤️ Willy Lulciuc, Maciej Obuchowski, Peter Hicks, Rogier Werschkull, A Pospiech, Kedar Rajwade, James Le +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2021-05-20 16:30:09
 +
+

share and enjoy 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-05-21 18:21:23
 +
+

Also happened yesterday: OpenLineage being accepted by the LFAI&Data.

+ + + +
+ 🎉 Abe Gong, Willy Lulciuc, Peter Hicks, Maciej Obuchowski, Daniel Henneberger, Harel Shein, Antonio Moctezuma, Josh Quintus, Mariusz Górski, James Le +
+ +
+ 👏 Matt Turck +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2021-05-21 19:20:55
 +
+

*Thread Reply:* Huge milestone! 🙌💯🎊

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-05-24 16:24:55
 +
+

I have created a channel to discuss <#C022MMLU31B|user-generated-metadata> since this came up in a few discussions.

+ + + +
+ 🙌 Willy Lulciuc +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jonathon Mitchal + (bigmit83@gmail.com) +
+
2021-05-31 01:28:35
 +
+

hey guys, does anyone have any sample openlineage schemas for S3 please? potentially including facets for attributes in a parquet file? that would help heaps thanks. i am trying to slowly bring in a common metadata interface and this will help shape some of the conversations 🙂 with a move to marquez/datahub et al over time

+ + + +
+ 🙌 Willy Lulciuc +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2021-06-01 17:56:16
 +
+

*Thread Reply:* We currently don’t have S3 (or distributed filesystem specific facets) at the moment, but such support would be a great addition! @Julien Le Dem would be best to answer if any work has been done in this area 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2021-06-01 17:57:19
 +
+

*Thread Reply:* Also, happy to answer any Marquez specific questions, @Jonathon Mitchal when you’re thinking of making the move. Marquez supports OpenLineage out of the box 🙌

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-06-01 19:58:21
 +
+

*Thread Reply:* @Jonathon Mitchal You can follow the naming strategy here for referring to a S3 dataset: https://github.com/OpenLineage/OpenLineage/blob/main/spec/Naming.md#s3

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/spec/Naming.md | Naming.md>
+
+ <p><strong>Naming</strong></p> + +<p>We define the unique name strategy per datasource to ensure it is followed uniformly independently from the type of job consuming and producing data. <br /> +The namespace for a dataset is the unique name for its datasource. <br /> +For jobs it is the scheduler.</p> + +<p><strong>Datasets:</strong></p> + +<p>The namespace and name of a datasource can be combined to form a URI (scheme:[//authority]path)</p> + +<p>• Namespace = scheme:[//authority] (the datasource) +• Name = path (the datasets)</p> + +<p><strong>Data warehouses/data bases.</strong></p> + +<p>Datasets are called tables. Tables are organised in databases and schemas.</p> + +<p><strong>Postgres:</strong></p> + +<p>Datasource hierarchy:</p> + +<p>• Host +• Port</p> + +<p>Naming hierarchy:</p> + +<p>• Database +• Schema +• Table</p> + +<p>Identifier:</p> + +<p>• Namespace: postgres://{host}:{port} of the service instance. + • Scheme = postgres + • Authority = {host}:{port} +• Unique name: {database}.{schema}.{table} + • URI = postgres://{host}:{port}/{database}.{schema}.{table}</p> + +<p><strong>Snowflake</strong></p> + +<p>See: <a href="https://docs.snowflake.com/en/sql-reference/identifiers.html">Object Identifiers — Snowflake Documentation</a></p> + +<p>Datasource hierarchy:</p> + +<p>• account name</p> + +<p>Naming hierarchy:</p> + +<p>• Database: {database name} =&gt; unique across the account +• Schema: {schema name} =&gt; unique within the database +• Table: {table name} =&gt; unique within the schema</p> + +<p>Identifier:</p> + +<p>• Namespace: snowflake://{account name} + • Scheme = snowflake + • Authority = {account name} +• Name: {database}.{schema}.{table} + • URI = snowflake://{account name}/{database}.{schema}.{table}</p> + +<p><strong>BigQuery</strong></p> + +<p>See: <br /> +<a href="https://cloud.google.com/resource-manager/docs/creating-managing-projects|Creating and managing projects | Resource Manager Documentation">https://cloud.google.com/resource-manager/docs/creating-managing-projects|Creating and managing projects | Resource Manager Documentation</a> <br /> +<a href="https://cloud.google.com/bigquery/docs/datasets-intro|Introduction to datasets | BigQuery">https://cloud.google.com/bigquery/docs/datasets-intro|Introduction to datasets | BigQuery</a> <br /> +<a href="https://cloud.google.com/bigquery/docs/tables-intro|Introduction to tables | BigQuery">https://cloud.google.com/bigquery/docs/tables-intro|Introduction to tables | BigQuery</a></p> + +<p>Datasource hierarchy:</p> + +<p>• bigquery</p> + +<p>Naming hierarchy:</p> + +<p>• Project Name: {project name} =&gt; is not unique +• Project number: {project number} =&gt; numeric: is unique across google cloud +• Project ID: {project id} =&gt; readable: is unique across google cloud +• dataset: {dataset name} =&gt; is unique within a project +• table: {table name} =&gt; is unique within a dataset</p> + +<p>Identifier :</p> + +<p>• Namespace: bigquery + • Scheme = bigquery + • Authority = +• Unique name: {project id}.{dataset name}.{table name} + • URI = bigquery:{project id}.{schema}.{table}</p> + +<p><strong>Distributed file systems/blob stores</strong> +<strong>GCS</strong></p> + +<p>Datasource hierarchy: none, naming is global</p> + +<p>Naming hierarchy:</p> + +<p>• bucket name =&gt; globally unique +• Path</p> + +<p>Identifier :</p> + +<p>• Namespace: gs://{bucket name} + • Scheme = gs + • Authority = {bucket name} +• Unique name: {path} + • URI = gs://{bucket name}{path}</p> + +<p><strong>S3</strong></p> + +<p>Naming hierarchy:</p> + +<p>• bucket name =&gt; globally unique +• Path</p> + +<p>Identifier :</p> + +<p>• Namespace: s3://{bucket name} + • Scheme = s3 + • Authority = {bucket name} +• Unique name: {path} + • URI = s3://{bucket name}{path}</p> + +<p><strong>HDFS</strong></p> + +<p>Naming hierarchy:</p> + +<p>• Namenode: host + port +• Path</p> + +<p>Identifier :</p> + +<p>• Namespace: hdfs://{namenode host}:{namenode port} + • Scheme = hdfs + • Authority = {namenode host}:{namenode port} +• Unique name: {path} + • URI = hdfs://{namenode host}:{namenode port}{path}</p> + +<p><strong>Schedulers</strong> +<strong>Airflow</strong></p> + +<p>Naming hierarchy:</p> + +<p>• job =&gt; workspace/DAG +• Task =&gt; unique within the job</p> + +<p>Identifier :</p> + +<p>• Namespace: {scheduler namespace} +• Unique name: {job}.{task}</p> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-06-01 19:59:30
 +
+

*Thread Reply:* There is no facet yet for the attributes of a Parquet file. I can give you feedback if you want to start defining one. https://github.com/OpenLineage/OpenLineage/blob/main/CONTRIBUTING.md#proposing-changes

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/CONTRIBUTING.md | CONTRIBUTING.md>
+
+ <p><strong>Contributing to OpenLineage</strong></p> + +<p>This project welcomes contributors from any organization or background, provided they are <br /> +willing to follow the simple processes outlined below, as well as adhere to the <br /> +&lt;CODE<em>OF</em>CONDUCT.md|Code of Conduct&gt;.</p> + +<p><strong>Joining the community</strong></p> + +<p>The community collaborates primarily through <code>GitHub</code> and the instance messaging tool, <code>Slack</code>. <br /> +There is also a mailing list. <br /> +See how to join <a href="https://github.com/OpenLineage/OpenLineage#community">here</a></p> + +<p><strong>Reporting an Issue</strong></p> + +<p>Please use the <a href="https://github.com/OpenLineage/OpenLineage/issues">issues</a> section of the OpenLineage repository and search for a similar problem. If you don't find it, submit your bug, question, proposal or feature request.</p> + +<p>In the case of bugs, please provide steps to reproduce it and tag your issue with "bug"</p> + +<p><strong>Contributing to the project</strong> +<strong>Creating Pull Requests</strong></p> + +<p>Before sending a Pull Request with significant changes, please use the <a href="https://github.com/OpenLineage/OpenLineage/issues">issue tracker</a> to discuss the potential improvements you want to make.</p> + +<p>OpenLineage uses <a href="https://help.github.com/articles/about-collaborative-development-models/">GitHub's fork and pull model</a> <br /> +to create a contribution.</p> + +<p>Every contribution is signed to say that the contributor has the rights to make the contribution and <br /> +agrees with the &lt;why-the-dco.md|Developer Certificate of Origin (DCO)&gt;</p> + +<p><strong>Proposing changes</strong></p> + +<p>Create an issue and prefix the description with "[PROPOSAL]"</p> + +<p>In the description provide the following sections:</p> + +<p>• Purpose (Why?): What is the use case this is for. +• Proposed implementation (How?): Quick description of how do you propose to implement it. Are you proposing a new facet?</p> + +<p>This can be just a couple paragraphs to start with.</p> + +<p><strong>First-Time Contributors</strong></p> + +<p>If this is your first contribution to open source, you can <a href="https://github.com/firstcontributions/first-contributions#first-contributions">follow this tutorial</a> or check <a href="https://egghead.io/courses/how-to-contribute-to-an-open-source-project-on-github">this video series</a> to learn about the contribution workflow with GitHub.</p> + +<p>Look tickets labeled <https://github.com/OpenLineage/OpenLineage/labels/good%20first%20issue|'good first issue'> and <https://github.com/OpenLineage/OpenLineage/labels/help%20wanted|'help wanted'>. These are a great starting point if you want to contribute. Don't hesitate to ask questions about the issue if you are not sure about the strategy to follow.</p> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-06-01 20:00:50
 +
+

*Thread Reply:* Adding Parquet metadata as a facet would make a lot of sense. It is mainly a matter of specifying what the json would look like

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-06-01 20:01:54
 +
+

*Thread Reply:* for reference the parquet metadata is defined here: https://github.com/apache/parquet-format/blob/master/src/main/thrift/parquet.thrift

+
+ + + + +
<https://github.com/apache/parquet-format/blob/master/src/main/thrift/parquet.thrift | parquet.thrift>
+
+ <p>``` +/<em>*</em>* + <em>* Licensed to the Apache Software Foundation (ASF) under one + *</em> or more contributor license agreements. See the NOTICE file + <em>* distributed with this work for additional information + *</em> regarding copyright ownership. The ASF licenses this file + <em>* to you under the Apache License, Version 2.0 (the + *</em> "License"); you may not use this file except in compliance + <em>* with the License. You may obtain a copy of the License at + *</em> + <em>* <a href="http://www.apache.org/licenses/LICENSE-2.0">http://www.apache.org/licenses/LICENSE-2.0</a> + *</em> + <em>* Unless required by applicable law or agreed to in writing, + *</em> software distributed under the License is distributed on an + <em>* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + *</em> KIND, either express or implied. See the License for the + <em>* specific language governing permissions and limitations + *</em> under the License. + **/</p> + +<p>/<em>*</em>* + <em>* File format description for the parquet file format + *</em>/ +namespace cpp parquet +namespace java org.apache.parquet.format</p> + +<p>/<em>*</em>* + <em>* Types supported by Parquet. These types are intended to be used in combination + *</em> with the encodings to control the on disk storage format. + <em>* For example INT16 is not included as a type since a good encoding of INT32 + *</em> would handle this. + **/ +enum Type { + BOOLEAN = 0; + INT32 = 1; + INT64 = 2; + INT96 = 3; // deprecated, only used by legacy implementations. + FLOAT = 4; + DOUBLE = 5; + BYTE<em>ARRAY = 6; + FIXED</em>LEN<em>BYTE</em>ARRAY = 7; +}</p> + +<p>/<strong><em>* + *</em> DEPRECATED: Common types used by frameworks(e.g. hive, pig) using parquet. + <em>* ConvertedType is superseded by LogicalType. This enum should not be extended. + *</em> + <em>* See LogicalTypes.md for conversion between ConvertedType and LogicalType. + *</em>/ +enum ConvertedType { + /<em>*</strong> a BYTE_ARRAY actually contains UTF8 encoded chars *</em>/ + UTF8 = 0;</p> + +<p>/<em>*</em>* a map is converted as an optional field containing a repeated key/value pair **/ + MAP = 1;</p> + +<p>/<em>*</em>* a key/value pair is converted into a group of two fields **/ + MAP<em>KEY</em>VALUE = 2;</p> + +<p>/<em>*</em>* a list is converted into an optional field containing a repeated field for its + <em>* values *</em>/ + LIST = 3;</p> + +<p>/<em>*</em>* an enum is converted into a binary field **/ + ENUM = 4;</p> + +<p>/<em>*</em>* + <em>* A decimal value. + *</em> + <em>* This may be used to annotate binary or fixed primitive types. The + *</em> underlying byte array stores the unscaled value encoded as two's + <em>* complement using big-endian byte order (the most significant byte is the + *</em> zeroth element). The value of the decimal is the value <em>* 10^{-scale}. + *</em> + <em>* This must be accompanied by a (maximum) precision and a scale in the + *</em> SchemaElement. The precision specifies the number of digits in the decimal + <em>* and the scale stores the location of the decimal point. For example 1.23 + *</em> would have precision 3 (3 total digits) and scale 2 (the decimal point is + <em>* 2 digits over). + *</em>/ + DECIMAL = 5;</p> + +<p>/<em>*</em>* + <em>* A Date + *</em> + <em>* Stored as days since Unix epoch, encoded as the INT32 physical type. + *</em> + **/ + DATE = 6;</p> + +<p>/<em>*</em>* + <em>* A time + *</em> + <em>* The total number of milliseconds since midnight. The value is stored + *</em> as an INT32 physical type. + **/ + TIME_MILLIS = 7;</p> + +<p>/<em>*</em>* + <em>* A time. + *</em> + <em>* The total number of microseconds since midnight. The value is stored as + *</em> an INT64 physical type. + **/ + TIME_MICROS = 8;</p> + +<p>/<em>*</em>* + <em>* A date/time combination + *</em> + <em>* Date and time recorded as milliseconds since the Unix epoch. Recorded as + *</em> a physical type of INT64. + **/ + TIMESTAMP_MILLIS = 9;</p> + +<p>/<em>*</em>* + <em>* A date/time combination + *</em> + <em>* Date and time recorded as microseconds since the Unix epoch. The value is + *</em> stored as an INT64 physical type. + **/ + TIMESTAMP_MICROS = 10;</p> + +<p>/<em>*</em>* + <em>* An unsigned integer value. + *</em> + <em>* The number describes the maximum number of meaningful data bits in + *</em> the stored value. 8, 16 and 32 bit values are stored using the + <em>* INT32 physical type. 64 bit values are stored using the INT64 + *</em> physical type. + <em>* + *</em>/ + UINT<em>8 = 11; + UINT</em>16 = 12; + UINT<em>32 = 13; + UINT</em>64 = 14;</p> + +<p>/<em>*</em>* + <em>* A signed integer value. + *</em> + <em>* The number describes the maximum number of meaningful data bits in + *</em> the stored value. 8, 16 and 32 bit values are stored using the + <em>* INT32 physical type. 64 bit values are stored using the INT64 + *</em> physical type. + <em>* + *</em>/ + INT<em>8 = 15; + INT</em>16 = 16; + INT<em>32 = 17; + INT</em>64 = 18;</p> + +<p>/<em>*</em>* + <em>* An embedded JSON document + *</em> + <em>* A JSON document embedded within a single UTF8 column. + *</em>/ + JSON = 19;</p> + +<p>/<em>*</em>* + <em>* An embedded BSON document + *</em> + <em>* A BSON document embedded within a single BINARY column. + *</em>/ + BSON = 20;</p> + +<p>/<em>*</em>* + <em>* An interval of time + *</em> + <em>* This type annotates data stored as a FIXED_LEN_BYTE_ARRAY of length 12 + *</em> This data is composed of three separate little endian unsigned + <em>* integers. Each stores a component of a duration of time. The first + *</em> integer identifies the number of months associated with the duration, + <em>* the second identifies the number of days associated with the duration + *</em> and the third identifies the number of milliseconds associated with + <em>* the provided duration. This duration of time is independent of any + *</em> particular timezone or date. + **/ + INTERVAL = 21; +}</p> + +<p>/<strong><em>* + *</em> Representation of Schemas + <em>*/ +enum FieldRepetitionType { + /</em><em></strong> This field is required (can not be null) and each record has exactly 1 value. *</em>/ + REQUIRED = 0;</p> + +<p>/<em>*</em>* The field is optional (can be null) and each record has 0 or 1 values. **/ + OPTIONAL = 1;</p> + +<p>/<em>*</em>* The field is repeated and can contain 0 or more values **/ + REPEATED = 2; +}</p> + +<p>/<strong><em>* + *</em> Statistics per row group and per page + <em>* All fields are optional. + *</em>/ +struct Statistics { + /<em>*</strong> + *</em> DEPRECATED: min and max value of the column. Use min<em>value and max</em>value. + <em>* + *</em> Values are encoded using PLAIN encoding, except that variable-length byte + <em>* arrays do not include a length prefix. + *</em> + <em>* These fields encode min and max values determined by signed comparison + *</em> only. New files should use the correct order for a column's logical type + <em>* and store the values in the min_value and max_value fields. + *</em> + <em>* To support older readers, these may be set when the column order is + *</em> signed. + <strong>/ + 1: optional binary max; + 2: optional binary min; + /<em>*</strong> count of null value in the column <strong>/ + 3: optional i64 null_count; + /</em><em></strong> count of distinct values occurring <strong>/ + 4: optional i64 distinct_count; + /</em><em></strong> + *</em> Min and max values for the column, determined by its ColumnOrder. + <em>* + *</em> Values are encoded using PLAIN encoding, except that variable-length byte + <em>* arrays do not include a length prefix. + *</em>/ + 5: optional binary max<em>value; + 6: optional binary min</em>value; +}</p> + +<p>/<em>*</em>* Empty structs to use as logical type annotations **/ +struct StringType {} // allowed for BINARY, must be encoded with UTF-8 +struct UUIDType {} // allowed for FIXED[16], must encoded raw UUID bytes +struct MapType {} // see LogicalTypes.md +struct ListType {} // see LogicalTypes.md +struct EnumType {} // allowed for BINARY, must be encoded with UTF-8 +struct DateType {} // allowed for INT32</p> + +<p>/<em>*</em>* + <em>* Logical type to annotate a column that is always null. + *</em> + <em>* Sometimes when discovering the schema of existing data, values are always + *</em> null and the physical type can't be determined. This annotation signals + <em>* the case where the physical type was guessed from all null values. + *</em>/ +struct NullType {} // allowed for any physical type, only null values stored</p> + +<p>/<em>*</em>* + <em>* Decimal logical type annotation + *</em> + <em>* To maintain forward-compatibility in v1, implementations using this logical + *</em> type must also set scale and precision on the annotated SchemaElement. + <em>* + *</em> Allowed for physical types: INT32, INT64, FIXED, and BINARY + **/ +struct DecimalType { + 1: required i32 scale + 2: required i32 precision +}</p> + +<p>/<em>*</em>* Time units for logical types **/ +struct MilliSeconds {} +struct MicroSeconds {} +struct NanoSeconds {} +union TimeUnit { + 1: MilliSeconds MILLIS + 2: MicroSeconds MICROS + 3: NanoSeconds NANOS +}</p> + +<p>/<em>*</em>* + <em>* Timestamp logical type annotation + *</em> + ** Allowed for physica…</p> +
+ + + + + + + +
+ + <a href="https://github.com/apache/parquet-format">apache/parquet-format</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jonathon Mitchal + (bigmit83@gmail.com) +
+
2021-06-01 23:20:50
 +
+

*Thread Reply:* Thats awesome, thanks for the guidance Willy and Julien ... will report back on how we get on

+ + + +
+ 🙏 Willy Lulciuc +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Pedram + (pedram@hightouch.io) +
+
2021-06-01 17:52:08
 +
+

hi all! just wanted to introduce myself, I'm the Head of Data at Hightouch.io, we build reverse etl pipelines from the warehouse into various destinations. I've been following OpenLineage for a while now and thought it would be nice to build and expose our runs via the standard and potentially save that back to the warehouse for analysis/alerting. Really interesting concept, looking forward to playing around with it

+ + + +
+ 👋 Willy Lulciuc, Ross Turk +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-06-01 20:02:34
 +
+

*Thread Reply:* Welcome! Let use know if you have any questions

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Leo + (leorobinovitch@gmail.com) +
+
2021-06-03 19:22:10
 +
+

Hi all! I have a noob question. As I understand it, one of the main purposes of OpenLineage is to avoid runaway proliferation of bespoke connectors for each data lineage/cataloging/provenance tool to each data source/job scheduler/query engine etc. as illustrated in the problem diagram from the main repo below.

+ +

My understanding is that instead, things push to OpenLineage which provides pollable endpoints for metadata tools.

+ +

I’m looking at Amundsen, and it seems to have bespoke connectors, but these are pull-based - I don’t need to instrument my data resources to push to Amundsen, I just need to configure Amundsen to poll my data resources (e.g. the Postgres metadata extractor here).

+ +

Can OpenLineage do something similar where I can just point it at something to extract metadata from it, rather than instrumenting that thing to push metadata to OpenLineage? If not, I’m wondering why?

+ +

Is it the case that Open Lineage defines the general framework but doesn’t actually enforce push or pull-based implementations, it just so happens that the reference implementation (Marquez) uses push?

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-06-04 04:45:15
 +
+

*Thread Reply:* > Is it the case that Open Lineage defines the general framework but doesn’t actually enforce push or pull-based implementations, it just so happens that the reference implementation (Marquez) uses push? +Yes, at core OpenLineage just enforces format of the event. We also aim to provide clients - REST, later Kafka, etc. and some reference implementations - which are now in Marquez repo. https://raw.githubusercontent.com/OpenLineage/OpenLineage/main/doc/Scope.png

+ +

There are several differences between push and poll models. Most important one is that with push model, latency between your job and emitting OpenLineage events is very low. With some systems, with internal, push based model you have more runtime metadata available than when looking from outside. Another one would be that naive poll implementation would need to "rebuild the world" on each change. There are also disadvantages, such as that usually, it's easier to write plugin that extracts data from outside the system than hooking up to the internals.

+ +

Integration with Amundsen specifically is planned. Although, right now it seems to me that way to do it is to bypass the databuilder framework and push directly to underlying database, such as Neo4j, or make Marquez backend for Metadata Service: https://raw.githubusercontent.com/amundsen-io/amundsen/master/docs/img/Amundsen_Architecture.png

+
+ + + + +
+
+ +
+ + + + + + + + +
Original URL: https://raw.githubusercontent.com/OpenLineage/OpenLineage/main/doc/Scope.png
+ + + +
+
+ + + + +
+
+ +
+ + + + + + + + +
Original URL: https://raw.githubusercontent.com/amundsen-io/amundsen/master/docs/img/Amundsen_Architecture.png
+ + + +
+ + + +
+ ❤️ Julien Le Dem +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Leo + (leorobinovitch@gmail.com) +
+
2021-06-04 10:39:51
 +
+

*Thread Reply:* This is really helpful, thank you @Maciej Obuchowski!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Leo + (leorobinovitch@gmail.com) +
+
2021-06-04 10:40:59
 +
+

*Thread Reply:* Similar to what you say about push vs pull, I found DataHub’s comment to be interesting yesterday: +> Push is better than pull: While pulling metadata directly from the source seems like the most straightforward way to gather metadata, developing and maintaining a centralized fleet of domain-specific crawlers quickly becomes a nightmare. It is more scalable to have individual metadata providers push the information to the central repository via APIs or messages. This push-based approach also ensures a more timely reflection of new and updated metadata.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-06-04 21:59:59
 +
+

*Thread Reply:* yes. You can also “pull-to-push” for things that don’t push.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mariusz Górski + (gorskimariusz13@gmail.com) +
+
2021-06-17 10:01:37
 +
+

*Thread Reply:* @Maciej Obuchowski any particular reason for bypassing databuilder and go directly to neo4j? By design databuilder is supposed to be very abstract so any kind of backend can be used with Amundsen. Currently there are at least 4 and neo4j is just one of them.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-06-17 10:28:52
 +
+

*Thread Reply:* Databuilder's pull model is very different than OpenLineage's push model, where the events are generated while the dataset itself is generated.

+ +

So, how would you see using it? Just to proxy the events to concrete search and metadata backend?

+ +

I'm definitely not an Amundsen expert, so feel free to correct me if I'm getting it wrong.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-07-07 19:59:28
 +
+

*Thread Reply:* @Mariusz Górski my slide that Maciej is referring to might be a bit misleading. The Amundsen integration does not exist yet. Please add your input in the ticket: https://github.com/OpenLineage/OpenLineage/issues/86

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/issues/86|#86 Add backend for sending lineage to Amundsen>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mariusz Górski + (gorskimariusz13@gmail.com) +
+
2021-07-09 02:22:06
 +
+

*Thread Reply:* thanks Julien! will take a look

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kedar Rajwade + (kedar@cloudzealous.com) +
+
2021-06-08 10:00:47
 +
+

@here Hello, My name is Kedar Rajwade. I happened to come across the OpenLineage project and it looks quite interesting. Is there some kind of getting start guide that I can follow. Also are there any weekly/bi-weekly calls that I can attend to know the current/future plans ?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-06-08 14:16:42
 +
+

*Thread Reply:* Welcome! You can look here: https://github.com/OpenLineage/OpenLineage/blob/main/CONTRIBUTING.md

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/CONTRIBUTING.md | CONTRIBUTING.md>
+
+ <p><strong>Contributing to OpenLineage</strong></p> + +<p>This project welcomes contributors from any organization or background, provided they are <br /> +willing to follow the simple processes outlined below, as well as adhere to the <br /> +&lt;CODE<em>OF</em>CONDUCT.md|Code of Conduct&gt;.</p> + +<p><strong>Joining the community</strong></p> + +<p>The community collaborates primarily through <code>GitHub</code> and the instance messaging tool, <code>Slack</code>. <br /> +There is also a mailing list. <br /> +See how to join <a href="https://github.com/OpenLineage/OpenLineage#community">here</a></p> + +<p><strong>Reporting an Issue</strong></p> + +<p>Please use the <a href="https://github.com/OpenLineage/OpenLineage/issues">issues</a> section of the OpenLineage repository and search for a similar problem. If you don't find it, submit your bug, question, proposal or feature request.</p> + +<p>In the case of bugs, please provide steps to reproduce it and tag your issue with "bug"</p> + +<p><strong>Contributing to the project</strong> +<strong>Creating Pull Requests</strong></p> + +<p>Before sending a Pull Request with significant changes, please use the <a href="https://github.com/OpenLineage/OpenLineage/issues">issue tracker</a> to discuss the potential improvements you want to make.</p> + +<p>OpenLineage uses <a href="https://help.github.com/articles/about-collaborative-development-models/">GitHub's fork and pull model</a> <br /> +to create a contribution.</p> + +<p>Every contribution is signed to say that the contributor has the rights to make the contribution and <br /> +agrees with the &lt;why-the-dco.md|Developer Certificate of Origin (DCO)&gt;</p> + +<p><strong>Proposing changes</strong></p> + +<p>Create an issue and prefix the description with "[PROPOSAL]"</p> + +<p>In the description provide the following sections:</p> + +<p>• Purpose (Why?): What is the use case this is for. +• Proposed implementation (How?): Quick description of how do you propose to implement it. Are you proposing a new facet?</p> + +<p>This can be just a couple paragraphs to start with.</p> + +<p><strong>First-Time Contributors</strong></p> + +<p>If this is your first contribution to open source, you can <a href="https://github.com/firstcontributions/first-contributions#first-contributions">follow this tutorial</a> or check <a href="https://egghead.io/courses/how-to-contribute-to-an-open-source-project-on-github">this video series</a> to learn about the contribution workflow with GitHub.</p> + +<p>Look tickets labeled <https://github.com/OpenLineage/OpenLineage/labels/good%20first%20issue|'good first issue'> and <https://github.com/OpenLineage/OpenLineage/labels/help%20wanted|'help wanted'>. These are a great starting point if you want to contribute. Don't hesitate to ask questions about the issue if you are not sure about the strategy to follow.</p> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-06-08 14:17:19
 +
+

*Thread Reply:* We’re starting a monthly call, I will publish more details here

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-06-08 14:17:48
 +
+

*Thread Reply:* Do you have a specific use case in mind?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kedar Rajwade + (kedar@cloudzealous.com) +
+
2021-06-08 21:32:02
 +
+

*Thread Reply:* Nothing specific yet

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-06-09 00:49:09
 +
+

The first instance of the OpenLineage Monthly meeting is tomorrow June 9 at 9am PT: https://calendar.google.com/event?action=TEMPLATE&tmeid=MDRubzk0cXAwZzA4bXRmY24yZjBkdTZzbDNfMjAyMTA2MDlUMTYwMDAwWiBqdWxpZW5AZGF0YWtpbi5jb20&tmsrc=julien%40datakin.com&scp=ALL|https://calendar.google.com/event?action=TEMPLATE&tmeid=MDRubzk0cXAwZzA4bXRmY24yZjBkdT[…]qdWxpZW5AZGF0YWtpbi5jb20&tmsrc=julien%40datakin.com&scp=ALL

+
+
accounts.google.com
+ + + +
Google Calendar
+
+ With Google's free online calendar, it’s easy to keep track of life’s important events all in one place. +
+ + + + + + +
Original URL: https://calendar.google.com/event?action=TEMPLATE&amp;tmeid=MDRubzk0cXAwZzA4bXRmY24yZjBkdTZzbDNfMjAyMTA2MDlUMTYwMDAwWiBqdWxpZW5AZGF0YWtpbi5jb20&amp;tmsrc=julien%40datakin.com&amp;scp=ALL
+ + + +
+ + + +
+ 🎉 Willy Lulciuc, Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Victor Shafran + (victor.shafran@databand.ai) +
+
2021-06-09 08:33:45
 +
+

*Thread Reply:* Hey @Julien Le Dem, I can’t add a link to my calendar… Can you send an invite?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Leo + (leorobinovitch@gmail.com) +
+
2021-06-09 11:00:05
 +
+

*Thread Reply:* Same!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-06-09 11:01:45
 +
+

*Thread Reply:* Will do. Also if you send your email in dm you can get added to the invite

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-06-09 11:59:22
 +
+

*Thread Reply:* You can find the invitation on the tsc mailing list: https://lists.lfaidata.foundation/g/openlineage-tsc/topic/invitation_openlineage/83423919?p=,,,20,0,0,0::recentpostdate%2Fsticky,,,20,2,0,83423919

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kedar Rajwade + (kedar@cloudzealous.com) +
+
2021-06-09 12:00:30
 +
+

*Thread Reply:* @Julien Le Dem Can't access the calendar.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kedar Rajwade + (kedar@cloudzealous.com) +
+
2021-06-09 12:00:43
 +
+

*Thread Reply:* Can you please share the meeting details

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-06-09 12:01:12
 +
+

*Thread Reply:*

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-06-09 12:01:24
 +
+

*Thread Reply:*

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2021-06-09 12:01:55
 +
+

*Thread Reply:* The calendar invite says 9am PDT, not 10am. Which is right?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kedar Rajwade + (kedar@cloudzealous.com) +
+
2021-06-09 12:01:58
 +
+

*Thread Reply:* Thanks

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-06-09 13:25:13
 +
+

*Thread Reply:* it is 9am,thanks

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-06-09 18:37:02
 +
+

*Thread Reply:* I have posted the notes on the wiki (includes link to recording) https://wiki.lfaidata.foundation/display/OpenLineage/Monthly+meeting+archive

+ + + +
+ 🙌 Willy Lulciuc, Victor Shafran +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Pedram + (pedram@hightouch.io) +
+
2021-06-10 13:53:18
 +
+

Hi! Are there some 'close-to-real' sample events available to build off and compare to? I'd like to make sure what I'm outputting makes sense but it's hard when only comparing to very synthetic data.

+ + + +
+ 👋 Willy Lulciuc +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2021-06-10 13:55:51
 +
+

*Thread Reply:* We’ve recently worked on a getting started guide for OpenLineage that we’d like to publish on the OpenLineage website. That should help with making things a bit more clear on usage. @Ross Turk / @Julien Le Dem might know of when that might become available. Otherwise, happy to answer any immediate questions you might have about posting/collecting OpenLineage events

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Pedram + (pedram@hightouch.io) +
+
2021-06-10 13:58:58
 +
+

*Thread Reply:* Here's a sample of what I'm producing, would appreciate any feedback if it's on the right track. One of our challenges is that 'dataset' is a little loosely defined for us as outputs since we take data from a warehouse/database and output to things like Salesforce, Airtable, Hubspot and even Slack.

+ +

{ + eventType: 'START', + eventTime: '2021-06-09T08:45:00.395+00:00', + run: { runId: '2821819' }, + job: { + namespace: '<hightouch://my-workspace>', + name: '<hightouch://my-workspace/sync/123>' + }, + inputs: [ + { + namespace: '<snowflake://abc1234>', + name: '<snowflake://abc1234/my_source_table>' + } + ], + outputs: [ + { + namespace: '<salesforce://mysf_instance.salesforce.com>', + name: 'accounts' + } + ], + producer: 'hightouch-event-producer-v.0.0.1' +} +{ + eventType: 'COMPLETE', + eventTime: '2021-06-09T08:45:30.519+00:00', + run: { runId: '2821819' }, + job: { + namespace: '<hightouch://my-workspace>', + name: '<hightouch://my-workspace/sync/123>' + }, + inputs: [ + { + namespace: '<snowflake://abc1234>', + name: '<snowflake://abc1234/my_source_table>' + } + ], + outputs: [ + { + namespace: '<salesforce://mysf_instance.salesforce.com>', + name: 'accounts' + } + ], + producer: 'hightouch-event-producer-v.0.0.1' +}

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Pedram + (pedram@hightouch.io) +
+
2021-06-10 14:02:59
 +
+

*Thread Reply:* One other question I have is really around how customers might take the metadata we emit at Hightouch and integrate that with OpenLineage metadata emitted from other tools like dbt, Airflow, and other integrations to create a true lineage of their data.

+ +

For example, if the data goes from S3 -&gt; Snowflake via Airflow and then from Snowflake -&gt; Salesforce via Hightouch, this would mean both Airflow/Hightouch would need to define the Snowflake dataset in exactly the same way to get the benefits of lineage?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2021-06-17 19:13:14
 +
+

*Thread Reply:* Hey, @Dejan Peretin! Sorry for the late replay here! Your OL events look solid and only have a few of suggestions:

+ +
  1. I would use a valid UUID for the run ID as the spec will standardize on that type, see https://github.com/OpenLineage/OpenLineage/pull/65
  2. You don’t need to provide the input dataset again on the COMPLETE event as the input datasets have already been associated with the run ID
  3. For the producer, I’d recommend using a link to the producer source code version to link the producer version with the OL event that was emitted.
    4. +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2021-06-17 19:13:59
 +
+

*Thread Reply:* You can now reference our OL getting started guide for a close-to-real example 🙂 , see http://openlineage.io/getting-started

+
+
openlineage.io
+ + + +
Getting Started
+
+ Data lineage is the foundation for a new generation of powerful, context-aware data tools and best practices. OpenLineage enables consistent collection of lineage metadata, creating a deeper understanding of how data is produced and used. +
+ + + + + + +
Original URL: http://openlineage.io/getting-started
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2021-06-17 19:18:19
 +
+

*Thread Reply:* > … this would mean both Airflow/Hightouch would need to define the Snowflake dataset in exactly the same way to get the benefits of lineage? +Yes, the dataset and the namespace that it was registered under would have to be the same to properly build the lineage graph. We’re working on defining unique dataset names and have made some good progress in this area. I’d suggest reviewing the OL naming conventions if you haven’t already: https://github.com/OpenLineage/OpenLineage/blob/main/spec/Naming.md

+ + + +
+ 🙌 Pedram +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Pedram + (pedram@hightouch.io) +
+
2021-06-19 01:09:27
 +
+

*Thread Reply:* Thanks! I'm really excited to see what the future holds, I think there are so many great possibilities here. Will be keeping a watchful eye. 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2021-06-22 15:14:39
 +
+

*Thread Reply:* 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Antonio Moctezuma + (antoniomoctezuma@northwesternmutual.com) +
+
2021-06-11 09:53:39
 +
+

Hey everyone! I've been running into a minor OpenLineage issue and I was curious if anyone had any advice. So according to OpenLineage specs its suggested that for a dataset coming from S3 that its namespace be in the form of s3://<bucket>. We have implemented our code to do so and RunEvents are published without issue but when trying to retrieve the information of this RunEvent (like the job) I am unable to retrieve it based on namespace from both /api/v1/namespaces/s3%3A%2F%2F<bucket name> (encoding since : and / are special characters in URL) and the beta endpoint of /api/v1-beta/lineage?nodeId=<dataset>:<namespace>:<name> and instead get a 400 error with a "Ambiguous Segment in URI" message.

+ +

Any and all advice would be super helpful! Thank you so much!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-06-11 10:16:41
 +
+

*Thread Reply:* Sounds like problem is with Marquez - might be worth to open issue here: https://github.com/MarquezProject/marquez/issues

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Antonio Moctezuma + (antoniomoctezuma@northwesternmutual.com) +
+
2021-06-11 10:25:58
 +
+

*Thread Reply:* Thank you! Will do.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-06-11 15:31:41
 +
+

*Thread Reply:* Thanks for reporting Antonio

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-06-16 19:01:52
 +
+

I have opened a proposal for versioning and publishing the spec: https://github.com/OpenLineage/OpenLineage/issues/63

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/issues/63|#63 [PROPOSAL] Define the OpenLineage spec versioning mechanism>
+
+ <p><strong>Purpose:</strong> <br /> +We need to define how the OpenLineage spec gets versioned and published. <br /> +Some requirements:</p> + +<p>• The OpenLineage spec and related libraries are in the OpenLineage repo. +• The OpenLineage spec version should change only when the spec itself changes. +• The libraries in the repo change more frequently than the spec (including when the spec changes). +• We want to version the OpenLineage spec independently of the api spec. +• The mechanism to version and publish the OpenLineage core spec, should apply to publishing custom facets.</p> + +<p><strong>Proposed implementation</strong> <br /> +I have started a google doc to make it easier to collaborate: <br /> +<a href="https://docs.google.com/document/d/1inhmb90SB62VyYf8nkkpjBDPpyr2wxkIXQ_AFIQiF9I/edit">https://docs.google.com/document/d/1inhmb90SB62VyYf8nkkpjBDPpyr2wxkIXQ_AFIQiF9I/edit</a> <br /> +please comment in the doc.</p> +
+ +
+
Labels
+ proposal +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-06-18 15:00:20
 +
+

We have a nice OpenLineage website now. https://openlineage.io/ +Thank you to contributors: @Ross Turk @Willy Lulciuc @Michael Collado!

+
+
openlineage.io
+ + + +
Home
+
+ Data lineage is the foundation for a new generation of powerful, context-aware data tools and best practices. OpenLineage enables consistent collection of lineage metadata, creating a deeper understanding of how data is produced and used. +
+ + + + + + +
Original URL: https://openlineage.io/
+ + + +
+ + + +
+ ❤️ Ross Turk, Kevin Mellott, Leo, Peter Hicks, Willy Lulciuc, Edgar Ramírez Mondragón, Maciej Obuchowski, Supratim Mukherjee +
+ +
+ 👍 Kedar Rajwade, Mukund +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Leo + (leorobinovitch@gmail.com) +
+
2021-06-18 15:09:18
 +
+

*Thread Reply:* Very nice!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Bruno Canal + (bcanal@gmail.com) +
+
2021-06-20 10:08:43
 +
+

Hi everyone! Im trying to run a spark job with openlineage and marquez...But Im getting some errors

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Bruno Canal + (bcanal@gmail.com) +
+
2021-06-20 10:09:28
 +
+

*Thread Reply:* Here is the error...

+ +

21/06/20 11:02:56 WARN ArgumentParser: missing jobs in [, api, v1, namespaces, spark_integration] at 5 +21/06/20 11:02:56 WARN ArgumentParser: missing runs in [, api, v1, namespaces, spark_integration] at 7 +21/06/20 11:03:01 ERROR AsyncEventQueue: Listener SparkListener threw an exception +java.lang.NullPointerException + at marquez.spark.agent.SparkListener.onJobEnd(SparkListener.java:165) + at org.apache.spark.scheduler.SparkListenerBus$class.doPostEvent(SparkListenerBus.scala:39) + at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37) + at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37) + at org.apache.spark.util.ListenerBus$class.postToAll(ListenerBus.scala:91) + at <a href="http://org.apache.spark.scheduler.AsyncEventQueue.org">org.apache.spark.scheduler.AsyncEventQueue.org</a>$apache$spark$scheduler$AsyncEventQueue$$super$postToAll(AsyncEventQueue.scala:92) + at org.apache.spark.scheduler.AsyncEventQueue$$anonfun$org$apache$spark$scheduler$AsyncEventQueue$$dispatch$1.apply$mcJ$sp(AsyncEventQueue.scala:92) + at org.apache.spark.scheduler.AsyncEventQueue$$anonfun$org$apache$spark$scheduler$AsyncEventQueue$$dispatch$1.apply(AsyncEventQueue.scala:87) + at org.apache.spark.scheduler.AsyncEventQueue$$anonfun$org$apache$spark$scheduler$AsyncEventQueue$$dispatch$1.apply(AsyncEventQueue.scala:87) + at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58) + at <a href="http://org.apache.spark.scheduler.AsyncEventQueue.org">org.apache.spark.scheduler.AsyncEventQueue.org</a>$apache$spark$scheduler$AsyncEventQueue$$dispatch(AsyncEventQueue.scala:87) + at org.apache.spark.scheduler.AsyncEventQueue$$anon$1$$anonfun$run$1.apply$mcV$sp(AsyncEventQueue.scala:83) + at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1302) + at org.apache.spark.scheduler.AsyncEventQueue$$anon$1.run(AsyncEventQueue.scala:82)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Bruno Canal + (bcanal@gmail.com) +
+
2021-06-20 10:10:41
 +
+

*Thread Reply:* Here is my code ...

+ +

```from pyspark.sql import SparkSession +from pyspark.sql.functions import lit

+ +

spark = SparkSession.builder \ + .master('local[1]') \ + .config('spark.jars.packages', 'io.github.marquezproject:marquezspark:0.15.2') \ + .config('spark.extraListeners', 'marquez.spark.agent.SparkListener') \ + .config('openlineage.url', 'http://localhost:5000/api/v1/namespaces/spark_integration/') \ + .config('openlineage.namespace', 'sparkintegration') \ + .getOrCreate()

+ +

Supress success

+ +

spark.sparkContext.jsc.hadoopConfiguration().set('mapreduce.fileoutputcommitter.marksuccessfuljobs', 'false') +spark.sparkContext.jsc.hadoopConfiguration().set('parquet.summary.metadata.level', 'NONE')

+ +

dfsourcetrip = spark.read \ + .option('inferSchema', True) \ + .option('header', True) \ + .option('delimiter', '|') \ + .csv('/Users/bcanal/Workspace/poc-marquez/pocspark/resources/data/source/trip.csv') \ + .createOrReplaceTempView('sourcetrip')

+ +

dfdrivers = spark.table('sourcetrip') \ + .select('driver') \ + .distinct() \ + .withColumn('drivername', lit('Bruno')) \ + .withColumnRenamed('driver', 'driverid') \ + .createOrReplaceTempView('source_driver')

+ +

df = spark.sql( + """ + SELECT d., t. + FROM sourcetrip t, sourcedriver d + WHERE t.driver = d.driver_id + """ +)

+ +

df.coalesce(1) \ + .drop('driverid') \ + .write.mode('overwrite') \ + .option('path', '/Users/bcanal/Workspace/poc-marquez/pocspark/resources/data/target') \ + .saveAsTable('trip')```

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Bruno Canal + (bcanal@gmail.com) +
+
2021-06-20 10:12:27
 +
+

*Thread Reply:* After this execution, I can see just the source from first dataframe called dfsourcetrip...

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Bruno Canal + (bcanal@gmail.com) +
+
2021-06-20 10:13:04
 +
+

*Thread Reply:*

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Bruno Canal + (bcanal@gmail.com) +
+
2021-06-20 10:13:45
 +
+

*Thread Reply:* I was expecting to see all source dataframes, target dataframes and the job

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Bruno Canal + (bcanal@gmail.com) +
+
2021-06-20 10:14:35
 +
+

*Thread Reply:* I`m running spark local on my laptop and I followed marquez getting start to up it

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Bruno Canal + (bcanal@gmail.com) +
+
2021-06-20 10:14:44
 +
+

*Thread Reply:* Can anyone help me?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2021-06-22 14:42:03
 +
+

*Thread Reply:* I think there's a race condition that causes the context to be missing when the job finishes too quickly. If I just add +spark.sparkContext.setLogLevel('info') +to the setup code, everything works reliably. Also works if you remove the master('local[1]') - at least when running in a notebook

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
anup agrawal + (anup.agrawal500@gmail.com) +
+
2021-06-22 13:48:34
 +
+

@here Hi everyone,

+ + + +
+ 👋 Willy Lulciuc +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
anup agrawal + (anup.agrawal500@gmail.com) +
+
2021-06-22 13:49:10
 +
+

i need to implement export functionality for my data lineage project.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
anup agrawal + (anup.agrawal500@gmail.com) +
+
2021-06-22 13:50:26
 +
+

as part of this i need to convert the information fetched from graph db (neo4j) to CSV format and send in response.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
anup agrawal + (anup.agrawal500@gmail.com) +
+
2021-06-22 13:51:21
 +
+

can someone please direct me to the CSV format of open lineage data

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2021-06-22 15:26:55
 +
+

*Thread Reply:* Hey, @anup agrawal. This is a great question! The OpenLineage spec is defined using the Json Schema format, and it’s mainly for the transport layer of OL events. In terms of how OL events are eventually stored, that’s determined by the backend consumer of the events. For example, Marquez stores the raw event in a lineage_events table, but that’s mainly for convenience and replayability of events . As for importing / exporting OL events from storage, as long as you can translate the CSV to an OL event, then HTTP backends like Marquez that support OL can consume them

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2021-06-22 15:27:29
 +
+

*Thread Reply:* > as part of this i need to convert the information fetched from graph db (neo4j) to CSV format and send in response. +Depending on the exported CSV, I would translate the CSV to an OL event, see https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.json

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2021-06-22 15:29:58
 +
+

*Thread Reply:* When you say “send in response”, who would be the consumer of the lineage metadata exported for the graph db?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
anup agrawal + (anup.agrawal500@gmail.com) +
+
2021-06-22 23:33:05
 +
+

*Thread Reply:* so far what i understood about my requirement is that. 1. my service will receive OL events

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
anup agrawal + (anup.agrawal500@gmail.com) +
+
2021-06-22 23:33:24
 +
+

*Thread Reply:* 2. store it in graph db (neo4j)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
anup agrawal + (anup.agrawal500@gmail.com) +
+
2021-06-22 23:38:28
 +
+

*Thread Reply:* 3. this lineage information will be displayed on ui, based on the request.

+ +
  1. now my part in that is to implement an Export functionality, so that someone can download it from UI. in UI there will be option to download the report.
  2. so i need to fetch data from storage and convert it into CSV format, send to UI
  3. they can download the report from UI.
    4. +
+ +

SO my question here is that i have never seen how that CSV report look like and how do i achieve that ? +when i had asked my team how should CSV look like they directed me to your website.

+ + + +
+ 👍 Willy Lulciuc +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2021-07-01 19:18:35
 +
+

*Thread Reply:* I see. @Julien Le Dem might have some thoughts on how an OL event would be represented in different formats like CSV (but, of course, there’s also avro, parquet, etc). The Json Schema is the recommended format for importing / exporting lineage metadata. And, for a file, each line would be an OL event. But, given that CSV is a requirement, I’m not sure how that would be structured. Or at least, it’s something we haven’t previously discussed

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
anup agrawal + (anup.agrawal500@gmail.com) +
+
2021-06-22 13:51:51
 +
+

i am very new to this .. sorry for any silly questions

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2021-06-22 20:29:22
 +
+

*Thread Reply:* There are no silly questions! 😉

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Abdulmalik AN + (lord.of.d1@gmail.com) +
+
2021-06-29 11:46:33
 +
+

Hello, I have read every topic and listened to 4 talks and the podcast episode about OpenLineage and Marquez due to my basic understanding for the data engineering field, I have a couple of questions which I did not understand: +1- What are events and facets and what are their purpose? +2- Can I implement the OpenLineage API to any software? or does the software needs to be integrated with the OpenLineage API? +3- Can I say that OpenLineage is about observability and Marquez is about collecting and storing the metadata? +Thank you all for being cooperative.

+ + + +
+ 👍 Stephen Pimentel, Kedar Rajwade +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2021-07-01 19:07:27
 +
+

*Thread Reply:* Welcome, @Abdulmalik AN 👋 Hopefully the talks / podcasts have been informative! And, sure, happy to clarify a few things:

+ +

> What are events and facets and what are their purpose? +An OpenLineage event is used to capture the lineage metadata at a point in time for a given run in execution. That is, the runs state transition, the inputs and outputs consumed/produced and the job associated with the run are part of the event. The metadata defined in the event can then be consumed by an HTTP backend (as well as other transport layers). Marquez is an HTTP backend implementation that consumes OL events via a REST API call. The OL core model only defines the metadata that should be captured in the context of a run, while the processing of the event is up to the backend implementation consuming the event (think consumer / producer model here). For Marquez, the end-to-end lineage metadata is stored for pipelines (composed of multiple jobs) with built-in metadata versioning support. Now, for the second part of your question: the OL core model is highly extensible via facets. A facet is user-defined metadata and enables entity enrichment. I’d recommend checking out the getting started guide for OL 🙂

+ +

> Can I implement the OpenLineage API to any software? or does the software needs to be integrated with the OpenLineage API? +Do you mean HTTP vs other protocols? Currently, OL defines an API spec for HTTP backends, that Marquez has adopted to ingest OL events. But there are also plans to support Kafka and many others.

+ +

> Can I say that OpenLineage is about observability and Marquez is about collecting and storing the metadata? +> Thank you all for being cooperative. +Yep! OL defines the metadata to collect for running jobs / pipelines that can later be used for root cause analysis / troubleshooting failing jobs, while Marquez is a metadata service that implements the OL standard to both consume and store lineage metadata while also exposing a REST API to query dataset, job and run metadata.

+
+ + + + +
<https://github.com/MarquezProject/marquez/blob/main/api/src/main/java/marquez/api/OpenLineageResource.java | OpenLineageResource.java>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/MarquezProject/marquez">MarquezProject/marquez</a> +
+ + +
+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.yml | OpenLineage.yml>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+
+
openlineage.io
+ + + +
Getting Started
+
+ Data lineage is the foundation for a new generation of powerful, context-aware data tools and best practices. OpenLineage enables consistent collection of lineage metadata, creating a deeper understanding of how data is produced and used. +
+ + + + + + +
Original URL: https://openlineage.io/getting-started/
+ + + +
+ + + +
+ 👍 Kedar Rajwade +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Nic Colley + (nic.colley@alation.com) +
+
2021-06-30 17:46:52
 +
+

Hi OpenLineage team! Has anyone got this working on databricks yet? I’ve been working on this for a few days and can’t get it to register lineage. I’ve attached my notebook in this thread.

+ +

silly question - does the jar file need be on the cluster? +Which versions of spark does OpenLineage support?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Nic Colley + (nic.colley@alation.com) +
+
2021-06-30 18:16:58
 +
+

*Thread Reply:* I based my code on this previous post https://openlineage.slack.com/archives/C01CK9T7HKR/p1624198123045800

+
+ + +
+ + + } + + Bruno Canal + (https://openlineage.slack.com/team/U025LV2BJUB) +
+ + + +
+
+ <p>Hi everyone! I<code>m trying to run a spark job with openlineage and marquez...But I</code>m getting some errors</p> +
+ + + + + + +
Original URL: https://openlineage.slack.com/archives/C01CK9T7HKR/p1624198123045800
+ + +
+ + Thread in Slack Conversation +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Nic Colley + (nic.colley@alation.com) +
+
2021-06-30 18:36:59
 +
+

*Thread Reply:*

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2021-07-01 13:45:42
 +
+

*Thread Reply:* In your first cell, you have +from pyspark.sql import SparkSession +from pyspark.sql.functions import lit +spark.sparkContext.setLogLevel('info') +unfortunately, the reference to sparkContext in the third line forces the initialization of the SparkContext so that in the next cell, your new configuration is ignored. In pyspark, you must initialize your SparkSession before any references to the SparkContext. It works if you remove the setLogInfo call from the first cell and make your 2nd cell +spark = SparkSession.builder \ + .config('spark.jars.packages', 'io.github.marquezproject:marquez_spark:0.15.2') \ + .config('spark.extraListeners', 'marquez.spark.agent.SparkListener') \ + .config('openlineage.url', '<https://domain.com>') \ + .config('openlineage.namespace', 'my-namespace') \ + .getOrCreate() +spark.sparkContext.setLogLevel('info')

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Samia Rahman + (srahman@thoughtworks.com) +
+
2021-06-30 19:26:42
 +
+

How would one capture lineage for job that's processing streaming data? Is that in scope for OpenLineage?

+ + + +
+ ➕ Josh Quintus, Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2021-07-01 16:32:18
 +
+

*Thread Reply:* It’s absolutely in scope! We’ve primarily focused on the batch use case (ETL jobs, etc), but the OpenLineage standard supports both batch and streaming jobs. You can check out our roadmap here, where you’ll find Flink and Beam on our list of future integrations.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2021-07-01 16:32:57
 +
+

*Thread Reply:* Is there a streaming framework you’d like to see added to our roadmap?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
mohamed chorfa + (chorfa672@gmail.com) +
+
2021-06-30 20:33:25
 +
+

👋 Hello everyone!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2021-07-01 16:24:16
 +
+

*Thread Reply:* Welcome, @mohamed chorfa 👋 . Let’s us know if you have any questions!

+ + + +
+ 👍 mohamed chorfa +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
mohamed chorfa + (chorfa672@gmail.com) +
+
2021-07-03 19:37:58
 +
+

*Thread Reply:* Really looking follow the evolution of the specification from RawData to the ML-Model

+ + + +
+ ❤️ Julien Le Dem, Willy Lulciuc +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-07-02 16:53:01
 +
+

Hello OpenLineage community, +We have been working on fleshing out the OpenLineage roadmap. +See on github on the currently prioritized effort: https://github.com/OpenLineage/OpenLineage/projects +Please add your feedback to the roadmap by either commenting on the github issues or opening new issues.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-07-02 17:04:13
 +
+

In particular, I have opened an issue to finalize our mission statement: https://github.com/OpenLineage/OpenLineage/issues/84

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/issues/84|#84 Finalize the OpenLineage mission statement>
+
+ <p>It is part of our technical charter. <br /> +Currently we have: "to enable the industry at-large to collect lineage metadata consistently and comprehensively across complex pipelines, creating a deeper understanding of data."</p> + +<p>We'd want to capture that this is specifically operational lineage, or lineage and metadata collected as the job is running.</p> + +<p>It is also something more widely applicable than just data pipelines, and should reflect that,</p> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+ ❤️ Ross Turk, Maciej Obuchowski, Peter Hicks +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-07-07 19:53:17
 +
+

*Thread Reply:* Based on community feedback, +The new proposed mission statement: “to enable the industry at-large to collect real-time lineage metadata consistently across complex ecosystems, creating a deeper understanding of how data is produced and used”

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-07-07 20:23:24
 +
+

I have updated the proposal for the spec versioning: https://github.com/OpenLineage/OpenLineage/issues/63

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/issues/63|#63 [PROPOSAL] Define the OpenLineage spec versioning mechanism>
+
+ <p><strong>Purpose:</strong> <br /> +We need to define how the OpenLineage spec gets versioned and published. <br /> +Some requirements:</p> + +<p>• The OpenLineage spec and related libraries are in the OpenLineage repo. +• The OpenLineage spec version should change only when the spec itself changes. +• The libraries in the repo change more frequently than the spec (including when the spec changes). +• We want to version the OpenLineage spec independently of the api spec. +• The mechanism to version and publish the OpenLineage core spec, should apply to publishing custom facets.</p> + +<p><strong>Proposed implementation</strong></p> + +<p>• The spec defines it’s current version using the “$id” field: <br /> + See: <br /> + <a href="https://json-schema.org/draft/2020-12/json-schema-core.html#rfc.section.8.2.1">json schema core doc</a> <br /> + <a href="https://json-schema.org/draft/2019-09/schema">Json schema spec $id</a> <a href="https://github.com/json-schema-org/json-schema-spec/blob/draft-next/meta/core.json">also on github</a> <br /> + Example: <br /> + "$id": "<a href="https://openlineage.io/spec/0.1.0/OpenLineage.json">https://openlineage.io/spec/0.1.0/OpenLineage.json</a>" <br /> + The URL in $id is resolvable and returns that version of the spec. <br /> + We use github pages to publish the spec to <a href="http://openlineage.io">openlineage.io</a> +• The $id urls uses a SEMVER compliant version, following the <a href="https://docs.snowplowanalytics.com/docs/pipeline-components-and-applications/iglu/common-architecture/schemaver/">SCHEMAVER semantics</a> <br /> + MODEL-REVISION-ADDITION + • MODEL when you make a breaking schema change which will prevent interaction with any historical data + • REVISION when you introduce a schema change which may prevent interaction with some historical data + • ADDITION when you make a schema change that is compatible with all historical data +• CI verification that: + • the version changes when the spec changed: <br /> + When resolving “$id”, the build fails if the spec is not exactly the same. + • The version does not change when the spec does not change. <br /> + TODO: define a mechanism to verify that the current version of the spec is not already published with a different. Possibly hash based. + • Libraries are generating event with current version + • Make sure the spec is backward compatible (only add optional fields) and consistent with the versioning semantics +• git pre commit: <br /> + Increments the versions automatically when the spec changes. +• CI publish (on tag): <br /> + The spec gets published automatically to the $id URL on <a href="http://OpenLineage.io">OpenLineage.io</a> <br /> + Check the $id field has the right domain prefix <br /> + Publish current OpenLineage.json to that URL</p> + +<p>The discussion is available on a google doc: <br /> +<a href="https://docs.google.com/document/d/1inhmb90SB62VyYf8nkkpjBDPpyr2wxkIXQ_AFIQiF9I/edit">https://docs.google.com/document/d/1inhmb90SB62VyYf8nkkpjBDPpyr2wxkIXQ_AFIQiF9I/edit</a></p> +
+ +
+
Assignees
+ julienledem +
+ +
+
Labels
+ proposal +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+ 🙌 Willy Lulciuc +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jorik + (jorik.blaas-sigmond@nn.nl) +
+
2021-07-08 07:06:53
 +
+

Hi all. I'm trying to get my bearings on openlineage. Love the concept. In our data transformation pipelines, output datasets are explicitly versioned (we have an incrementing snapshot id). Our storage layer (deltalake) allows us to also ingest 'older' versions of the same dataset, etc. If I understand it correctly I would have to add some inputFacets and outputFacets to run to store the actual version being referenced. Is that something that is currently available, or on the roadmap, or is it something I could extend myself?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-07-08 18:57:44
 +
+

*Thread Reply:* It is on the roadmap and there’s a ticket open but nobody is working on it at the moment. You are very welcome to contribute a spec and implementation

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-07-08 18:59:00
 +
+

*Thread Reply:* Please comment here and feel free to make a proposal: https://github.com/OpenLineage/OpenLineage/issues/35

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/issues/35|#35 [PROPOSAL] Dataset version facet>
+
+ <p><strong>Purpose:</strong> <br /> +Many databases have explicit versioning in place for datasets. It is highly desirable to track the specific version of the dataset if it is known at consumption/generation time. Rather than relying on the service to correctly tie job inputs to specific versions of a dataset, the client should be allowed to report that information if it is available.</p> + +<p><strong>Proposed implementation</strong> <br /> +A dataset version facet may be sufficient to report dataset version information, if it exists. Custom facets can be used to include any database specific version information that may exist, while the minimum required fields should include a versionName string field.</p> +
+ +
+
Labels
+ proposal +
+ +
+
Comments
+ 2 +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jorik + (jorik.blaas-sigmond@nn.nl) +
+
2021-07-08 07:07:29
 +
+

TL;DR: our database supports time-travel, and runs can be set up to use a specific point-in-time of an input. How do we make sure to keep that information within openlineage

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mariusz Górski + (gorskimariusz13@gmail.com) +
+
2021-07-09 02:23:29
 +
+

Hi, on a subject of spark integrations - I know that there is spark-marquez but was curious did you also consider https://github.com/AbsaOSS/spline-spark-agent ? It seems like this and spark-marquez are doing similar thing and maybe it would make sense to add openlineage support to spline spark agent?

+
+ + + + +
AbsaOSS/spline-spark-agent
+
+ Spline agent for Apache Spark +
+ +
+
Website
+ <https://absaoss.github.io/spline/> +
+ +
+
Stars
+ 36 +
+ + + + + + + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mariusz Górski + (gorskimariusz13@gmail.com) +
+
2021-07-09 02:23:42
 +
+

*Thread Reply:* cc @Julien Le Dem @Maciej Obuchowski

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-07-09 04:28:38
 +
+

*Thread Reply:* @Michael Collado

+ + + +
+ 👀 Michael Collado +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-07-12 21:17:12
 +
+

The OpenLineage Technical Steering Committee meetings are Monthly on the Second Wednesday 9:00am to 10:00am US Pacific and the link to join the meeting is https://us02web.zoom.us/j/81831865546?pwd=RTladlNpc0FTTDlFcWRkM2JyazM4Zz09 +The next meeting is this Wednesday +All are welcome. +•  Agenda: + ◦ Finalize the OpenLineage Mission Statement + ◦ Review OpenLineage 0.1 scope + ◦ Roadmap + ◦ Open discussion  + ◦ Slides: https://docs.google.com/presentation/d/1fD_TBUykuAbOqm51Idn7GeGqDnuhSd7f/edit#slide=id.ge4b57c6942_0_46 +notes are posted here: https://wiki.lfaidata.foundation/display/OpenLineage/Monthly+TSC+meeting.,.,_

+ + + +
+ 🙌 Willy Lulciuc, Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-07-12 21:18:04
 +
+

*Thread Reply:* Feel free to share your email with me if you want to be added to the gcal invite

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-07-14 12:03:31
 +
+

*Thread Reply:* It is starting now

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jiří Sedláček + (yirie.sedlahczech@gmail.com) +
+
2021-07-13 08:22:40
 +
+

Hello, is it possible to track lineage on column level? For example for SQL like this: +CREATE TABLE T2 AS SELECT c1,c2 FROM T1; +I would like to record this lineage: +T1.C1 -- job1 --&gt; T2.C1 +T1.C2 -- job1 --&gt; T2.C2 +Would that be possible to record in OL format?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jiří Sedláček + (yirie.sedlahczech@gmail.com) +
+
2021-07-13 08:29:52
 +
+

(the important thing for me is to be able to tell that T1.C1 has no effect on T2.C2)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-07-14 17:00:12
 +
+

I have updated the notes and added the link to the recording of the meeting this morning: https://wiki.lfaidata.foundation/display/OpenLineage/Monthly+TSC+meeting

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-07-14 17:04:18
 +
+

*Thread Reply:* In particular, please review the versioning proposal: https://github.com/OpenLineage/OpenLineage/issues/63

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/issues/63|#63 [PROPOSAL] Define the OpenLineage spec versioning mechanism>
+
+ <p><strong>Purpose:</strong> <br /> +We need to define how the OpenLineage spec gets versioned and published. <br /> +Some requirements:</p> + +<p>• The OpenLineage spec and related libraries are in the OpenLineage repo. +• The OpenLineage spec version should change only when the spec itself changes. +• The libraries in the repo change more frequently than the spec (including when the spec changes). +• We want to version the OpenLineage spec independently of the api spec. +• The mechanism to version and publish the OpenLineage core spec, should apply to publishing custom facets.</p> + +<p><strong>Proposed implementation</strong></p> + +<p>• The spec defines it’s current version using the <code>“$id”</code> field: + • See: + • <a href="https://json-schema.org/draft/2020-12/json-schema-core.html#rfc.section.8.2.1">json schema core doc</a> + • <a href="https://json-schema.org/draft/2019-09/schema">Json schema spec $id</a> + • <a href="https://github.com/json-schema-org/json-schema-spec/blob/draft-next/meta/core.json">also on github</a> + • Example: <br /> + <code>"$id": "&lt;https://openlineage.io/spec/0.1.0/OpenLineage.json&gt;"</code> <br /> + The URL in $id is resolvable and returns that version of the spec. <br /> + We use github pages to publish the spec to <a href="http://openlineage.io">openlineage.io</a> +• The $id urls uses a SEMVER compliant version, following the <a href="https://docs.snowplowanalytics.com/docs/pipeline-components-and-applications/iglu/common-architecture/schemaver/">SCHEMAVER semantics</a> <br /> + MODEL-REVISION-ADDITION + • MODEL when you make a breaking schema change which will prevent interaction with any historical data + • REVISION when you introduce a schema change which may prevent interaction with some historical data + • ADDITION when you make a schema change that is compatible with all historical data +• The spec in /spec/OpenLineage.json might be using a prerelease version (ex: 0.1.0-rc0) +• CI verifies that: + • the version changes when the spec changed: <br /> + When resolving “$id”, the build fails if the spec is not exactly the same. + • The version does not change when the spec does not change. <br /> + We can verify that the current version of the spec is not already published with a different version. + • Libraries are generating event with current version + • Make sure the spec is backward compatible (only add optional fields) and consistent with the versioning semantics +• git pre commit: <br /> + Increments the versions automatically when the spec changes. +• spec publication: + • update the $id to the correct value + • tag with spec-{version} + • The spec is published automatically to the $id URL on <a href="http://OpenLineage.io">OpenLineage.io</a> + • Validation that the $id field has the right domain prefix</p> + +<p>The discussion is available on a google doc: <br /> +<a href="https://docs.google.com/document/d/1inhmb90SB62VyYf8nkkpjBDPpyr2wxkIXQ_AFIQiF9I/edit">https://docs.google.com/document/d/1inhmb90SB62VyYf8nkkpjBDPpyr2wxkIXQ_AFIQiF9I/edit</a></p> +
+ +
+
Assignees
+ julienledem +
+ +
+
Labels
+ proposal +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-07-14 17:04:33
 +
+

*Thread Reply:* and the mission statement: https://github.com/OpenLineage/OpenLineage/issues/84

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/issues/84|#84 Finalize the OpenLineage mission statement>
+
+ <p>The mission statement is part of our technical charter. <br /> +We started from:</p> + +<p>&gt; "The mission of the Project is to enable the industry at-large to collect lineage metadata consistently and comprehensively across complex pipelines, creating a deeper understanding of data."</p> + +<p>We'd want to capture that this is specifically operational lineage, or lineage and metadata collected as the job is running. It is also something more widely applicable than just data pipelines, and should reflect that.</p> + +<p>The current proposal is:</p> + +<p>&gt; "The mission of the Project is to enable the industry at-large to collect real-time lineage metadata consistently across complex ecosystems, creating a deeper understanding of how data is produced and used"</p> +
+ +
+
Comments
+ 2 +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-07-14 17:05:02
 +
+

*Thread Reply:* for this one, please give explicit approval in the ticket

+ + + +
+ 👍 Willy Lulciuc +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-07-14 21:10:42
 +
+

*Thread Reply:* @Zhamak Dehghani @Daniel Henneberger @Drew Banin @James Campbell @Ryan Blue @Maciej Obuchowski @Willy Lulciuc ^

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-07-27 18:58:35
 +
+

*Thread Reply:* Per the votes in the github ticket, I have finalized the charter here: https://docs.google.com/document/d/11xo2cPtuYHmqRLnR-vt9ln4GToe0y60H/edit

+ + + +
+ 🙌 Willy Lulciuc +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jarek Potiuk + (jarek@potiuk.com) +
+
2021-07-16 01:25:56
 +
+

Hi Everyone. I am PMC member and committer of Apache Airflow. Watched the talk at the summit https://airflowsummit.org/sessions/2021/data-lineage-with-apache-airflow-using-openlineage/ and thought I might help (after the Summit is gone 🙂 with making OpenLineage/Marquez more seemlesly integrated in Airflow

+
+
airflowsummit.org
+ + + +
Data Lineage with Apache Airflow using OpenLineage
+
+ If you manage a lot of data, and you’re attending this summit, you likely rely on Apache Airflow to do a lot of the heavy lifting. Like any powerful tool, Apache Airflow allows you to accomplish what you couldn’t before… but also creates new challenges. As DAGs pile up, complexity layers on top of complexity and it becomes hard to grasp how a failed or delayed DAG will affect everything downstream. +
+ + + + + + + + +
Original URL: https://airflowsummit.org/sessions/2021/data-lineage-with-apache-airflow-using-openlineage/
+ + + +
+ + + +
+ ❤️ Abe Gong, WingCode, Maciej Obuchowski, Ross Turk, Julien Le Dem, Michael Collado, Samia Rahman, mohamed chorfa +
+ +
+ 🙌 Maciej Obuchowski +
+ +
+ 👍 Jorik +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Samia Rahman + (srahman@thoughtworks.com) +
+
2021-07-20 16:38:38
 +
+

*Thread Reply:* The demo in this does not really use the openlineage spec does it?

+ +

Did I miss something - the API that was should for lineage was that of Marquez, how does Marquest use the open lineage spec?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Samia Rahman + (srahman@thoughtworks.com) +
+
2021-07-20 18:09:01
 +
+

*Thread Reply:* I have a question about the SQLJobFacet in the job schema - isn't it better to call it the TransformationJob Facet or the ProjecessJobFacet such that any logic in the appropriate language and be described? Am I misinterpreting the intention of SQLJobFacet is to capture the logic that runs for a job?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2021-07-26 19:06:43
 +
+

*Thread Reply:* > The demo in this does not really use the openlineage spec does it? +@Samia Rahman In our Airflow talk, the demo used the marquez-airflow lib that sends OpenLineage events to Marquez’s . You can check out the how does Airflow works with OpenLineage + Marquez here https://openlineage.io/integration/apache-airflow/

+
+
openlineage.io
+ + + +
Apache Airflow
+
+ Airflow, a common tool used for workflow management, is the foundation of many data pipelines. Enabling OpenLineage in Apache Airflow automatically tracks metadata about jobs and datasets as DAGs execute. +
+ + + + + + +
Original URL: https://openlineage.io/integration/apache-airflow/
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2021-07-26 19:07:51
 +
+

*Thread Reply:* > Did I miss something - the API that was should for lineage was that of Marquez, how does Marquest use the open lineage spec? +Yes, Marquez ingests OpenLineage events that confirm to the spec via the . Hope this helps!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kenton (swiple.io) + (kknoxparton@gmail.com) +
+
2021-07-21 07:52:32
 +
+

Hi all, does OpenLineage intend on creating lineage off of query logs?

+ +

From what I have read, there are a number of supported integrations but none that cater to regular SQL based ETL. Is this on the OpenLineage roadmap?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2021-07-26 18:54:46
 +
+

*Thread Reply:* I would say this is more of an ingestion pattern, then something the OpenLineage spec would support directly. Though I completely agree, query logs are a great source of lineage metadata with minimal effort. On our roadmap, we have Kafka as a supported backend which would enable streaming lineage metadata from query logs into a topic. That said, confluent has some great blog posts on Change Data Capture: +• https://www.confluent.io/blog/no-more-silos-how-to-integrate-your-databases-with-apache-kafka-and-cdc/ +• https://www.confluent.io/blog/simplest-useful-kafka-connect-data-pipeline-world-thereabouts-part-1/

+
+
Confluent
+ + + +
No More Silos: How to Integrate your Databases with Apache Kafka and CDC | Confluent
+
+ This article shows you how Apache Kafka integrates with existing RDBMS seamlessly - and discusses Change Data Capture (CDC) options. +
+ + + + + + + + +
Original URL: https://www.confluent.io/blog/no-more-silos-how-to-integrate-your-databases-with-apache-kafka-and-cdc/
+ + + +
+
+
Confluent
+ + + +
The Simplest Useful Kafka Connect Data Pipeline in the World – Part 1
+
+ Learn how to stream data from a database (MySQL) into Apache Kafka and from Kafka into a text file with the Kafka Connect API. +
+ + + + + + + + +
Original URL: https://www.confluent.io/blog/simplest-useful-kafka-connect-data-pipeline-world-thereabouts-part-1/
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2021-07-26 18:57:59
 +
+

*Thread Reply:* Q: @Kenton (swiple.io) Are you planning on using Kafka connect? If so, I see 2 reasonable options:

+ +
  1. Stream query logs to a topic using the JDBC source connector, then have a consumer read the query logs off the topic, parse the logs, then stream the result of the query parsing to another topic as an OpenLineage event
  2. Add direct support for OpenLineage to the JDBC connector or any other application you planned to use to read the query logs.
    3. +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2021-07-26 19:01:31
 +
+

*Thread Reply:* Either way, I think this is a great question and a common ingestion pattern we should document or have best practices for. Also, more details on how you plan to ingestion the query logs would be help drive the discussion.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kenton (swiple.io) + (kknoxparton@gmail.com) +
+
2021-08-05 12:01:55
 +
+

*Thread Reply:* Using something like sqlflow could be a good starting point? Demo https://sqlflow.gudusoft.com/?utm_source=gspsite&utm_medium=blog&utm_campaign=support_article#/

+
+
sqlflow.gudusoft.com
+ + + +
SQLFlow: Visualize column impact and data lineage to track columns across transformations by analyzing SQL query.
+
+ SQLFlow: Visualize column impact and data lineage to track columns across transformations by analyzing SQL query. supported databases: bigquery, couchbase, dax, db2, greenplum, hana, hive, impala, informix, mdx, mysql, netezza, openedge, oracle, postgresql, redshift, snowflake, sqlserver, sybase, teradata, vertica +
+ + + + + + +
Original URL: https://sqlflow.gudusoft.com/?utm_source=gspsite&amp;utm_medium=blog&amp;utm_campaign=support_article#/
+ + + +
+
+ + + + +
sqlparser/sqlflow_public
+
+ Document, sample code and other materials for SQLFlow +
+ +
+
Stars
+ 79 +
+ +
+
Language
+ Python +
+ + + + + + + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2021-09-21 20:22:26
 +
+

*Thread Reply:* @Kenton (swiple.io) I haven’t heard of sqlflow but it does look promising. It’s not on our current roadmap, but I think there is a need to have support for parsing query logs as OpenLineage events. Do you mind opening an issue and outlining you thoughts? It’d be great to start the discussion if you’d like to drive this feature and help prioritize this 💯

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Samia Rahman + (srahman@thoughtworks.com) +
+
2021-07-21 08:49:23
 +
+

The openlineage implementation for airflow and spark code integration currently lives in Marquez repo, my understanding from the open lineage scope is that the the integration implementation is the scope of open lineage, are the spark code migrations going to be moved to open lineage?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2021-07-21 11:35:12
 +
+

@Samia Rahman Yes, that is the plan. For details you can see https://github.com/OpenLineage/OpenLineage/issues/73

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/issues/73|#73 Move Marquez integrations to OpenLineage>
+
+ <p>The <a href="https://github.com/MarquezProject/marquez">Marquez</a> community is excited to announce that the following integrations will be moved to the wider OpenLineage ecosystem:</p> + +<p>• <a href="https://github.com/MarquezProject/marquez/tree/main/integrations/airflow"><code>marquez-airflow</code></a>: A library that integrates Airflow DAGs to emit OpenLineage metadata. +• <a href="https://github.com/MarquezProject/marquez/tree/main/integrations/spark"><code>marquez-spark</code></a>: A spark agent using jvm instrumentation to emit OpenLineage metadata. +• <a href="https://github.com/MarquezProject/marquez/tree/main/integrations/common"><code>marquez-integration-common</code></a>: Shared code across integrations. +• <code>marquez-dbt-snowflake</code>: A library that integrates with DBT to emit OpenLineage metadata for Snowflake. +• <code>marquez-dbt-bigquery</code>: A library that integrates with DBT to emit OpenLineage metadata for BigQuery.</p> + +<p>&gt; Note: You can view the integrations in the Marquez repo under <a href="https://github.com/MarquezProject/marquez/tree/main/integrations"><code>marquez/integrations/</code></a>. Marquez is an official reference implementation of the OpenLineage API and all the integrations adopt the OpenLineage standard.</p> + +<p><strong>Why are the integrations being moved to OpenLineage?</strong></p> + +<p>The OpenLineage standard was influenced by Marquez's underlying data model (<em>datasets</em>, <em>jobs</em>, <em>runs</em>) while also drawing inspiration from the projects lineage ingestion API. With OpenLineage now a standalone project under <a href="https://lfaidata.foundation">LFAI &amp; Data Foundation</a>, we've decided that now would be the right time to move the integrations over as more and more integrations are being added to the <a href="https://github.com/OpenLineage/OpenLineage/projects/1">projects roadmap</a> (<em>Looker</em>, <em>Flink</em>, <em>Beam</em>, etc). This would also allow for the OpenLineage community to grow and learn from existing integration implementations, prioritize bug fixes, propose new integrations and changes to the spec, etc.</p> + +<p><strong>What is the timeline for moving the integrations to OpenLineage?</strong></p> + +<p>Below we outline a high-level timeline of moving the integrations to OpenLineage. We encourage everyone to provide feedback and express any concerns you may have by commenting in this issue:</p> + +<p>☑︎ Release Marquez <a href="https://github.com/MarquezProject/marquez/projects/12"><code>0.16.0</code></a> +☐ Move the following integrations from Marquez to OpenLineage: + ☐ <code>marquez-airflow</code> #93 + ☐ <code>marquez-spark</code> + ☑︎ <code>marquez-integration-common</code> #74 + ☐ <code>marquez-dbt-snowflake</code> + ☐ <code>marquez-dbt-bigquery</code> +☐ Deprecate all integrations in Marquez, then link to the new location in the OpenLineage repo</p> + +<p>Then, going forward, new integrations and improvements will be made by open pull requests against the OpenLineage repo. <strong>Note that during this migration phase, existing Marquez specific functionality and environment variables will be maintained to ensure backwards compatibility. The main difference is that integration packages will get a new name: <code>openlineage-**</code> instead of <code>marquez-**</code></strong>.</p> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+ 🙌 Samia Rahman, Willy Lulciuc +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Samia Rahman + (srahman@thoughtworks.com) +
+
2021-07-21 18:13:11
 +
+

I have a question about the SQLJobFacet in the job schema - isn't it better to call it the TransformationJob Facet or the ProjecessJobFacet such that any logic in the appropriate language and be described, can be scala or python code that runs in the job facet and processing streaming or batch data? Am I misinterpreting the intention of SQLJobFacet is to capture the logic that runs for a job?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2021-07-21 18:22:01
 +
+

*Thread Reply:* Hey, @Samia Rahman 👋. Yeah, great question! The SQLJobFacet is used only for SQL-based jobs. That is, it’s not intended to capture the code being executed, but rather the just the SQL if it’s present. The SQL fact can be used later for display purposes. For example, in Marquez, we use the SQLJobFacet to display the SQL executed by a given job to the user via the UI.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2021-07-21 18:23:03
 +
+

*Thread Reply:* To capture the logic of the job (meaning, the code being executed), the OpenLineage spec defines the SourceCodeLocationJobFacet that builds the link to source in version control

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.json | OpenLineage.json>
+
+ <pre><code> "SourceCodeLocationJobFacet": { +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-07-22 17:56:41
 +
+

The process started a few months back when the LF AI & Data voted to accept OpenLineage as part of the foundation. It is now official, OpenLineage joined the LFAI & data Foundation. + https://lfaidata.foundation/blog/2021/07/22/openlineage-joins-lf-ai-data-as-new-sandbox-project/

+
+
LF AI
+ + + +
OpenLineage Joins LF AI &amp; Data as New Sandbox Project - LF AI
+
+ LF AI &amp; Data Foundation—the organization building an ecosystem to sustain open source innovation in artificial intelligence (AI) and data open source projects, today is announcing OpenLineage as its latest... +
+ +
+
Written by
+ Jacqueline Z Cardoso +
+ +
+
Est. reading time
+ 3 minutes +
+ + + + + + + + +
Original URL: https://lfaidata.foundation/blog/2021/07/22/openlineage-joins-lf-ai-data-as-new-sandbox-project/
+ + + +
+ + + +
+ 🙌 Ross Turk, Luke Smith, Maciej Obuchowski, Gyan Kapur, Dr Daniel Smith, Jarek Potiuk, Peter Hicks, Kedar Rajwade, Abe Gong, Damian Warszawski, Willy Lulciuc +
+ +
+ ❤️ Ross Turk, Jarek Potiuk, Peter Hicks, Abe Gong, Willy Lulciuc +
+ +
+ 🎉 Laurent Paris, Rifa Achrinza, Minkyu Park, Peter Hicks, mohamed chorfa, Jarek Potiuk, Abe Gong, Damian Warszawski, Willy Lulciuc, James Le +
+ +
+ 👏 Matt Turck +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Namron + (ian.norman@avanade.com) +
+
2021-07-29 11:20:17
 +
+

Hi, I am trying to create lineage between two datasets. Following the Spec, I can see the syntax for declaring the input and output datasets, and for all creating the associated Job (which I take to be the process in the middle joining the two datasets together). What I can't see is where in the specification to relate the job to the inputs and outputs. Do you have an example of this?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2021-07-30 17:24:44
 +
+

*Thread Reply:* The run event is always tied to exactly one job. It's up to the backend to store the relationship between the job and its inputs/outputs. E.g., in marquez, this is where we associate the input datasets with the job- https://github.com/MarquezProject/marquez/blob/main/api/src/main/java/marquez/db/OpenLineageDao.java#L132-L143

+
+ + + + +
<https://github.com/MarquezProject/marquez/blob/main/api/src/main/java/marquez/db/OpenLineageDao.java | OpenLineageDao.java>
+
+ <pre><code> JobRow job = + jobDao.upsertJob( + UUID.randomUUID(), + getJobType(event.getJob()), + now, + namespace.getUuid(), + namespace.getName(), + event.getJob().getName(), + description, + jobContext.getUuid(), + location, + jobDao.toJson(toDatasetId(event.getInputs()), mapper)); +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/MarquezProject/marquez">MarquezProject/marquez</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-08-03 15:06:58
 +
+

the OuputStatistics facet PR is updated based on your comments @Michael Collado https://github.com/OpenLineage/OpenLineage/pull/114

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/pull/114|#114 add OutputStatistics facet>
+
+ <p>Signed-off-by: Julien Le Dem <a href="mailto:julien@apache.org">julien@apache.org</a></p> +
+ +
+
Comments
+ 1 +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+ 🙌 Michael Collado +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2021-08-03 15:11:56
 +
+

*Thread Reply:* /|~~~ + ///| + /////| + ///////| + /////////| + \==========|===/ +~~~~~~~~~~~~~~~~~~~~~

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-08-03 19:59:03
 +
+

*Thread Reply:* ⛵

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-08-03 19:59:38
 +
+

I have updated the DataQuality metrics proposal and the corresponding PR: https://github.com/OpenLineage/OpenLineage/issues/101 +https://github.com/OpenLineage/OpenLineage/pull/115

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/issues/101|#101 Add the DataQualityMetrics Dataset facet to the core spec>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/pull/115|#115 Data quality facet>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+ 🙌 Willy Lulciuc, Bruno González +
+ +
+ 💯 Willy Lulciuc, Dominique Tipton +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Oleksandr Dvornik + (oleksandr.dvornik@getindata.com) +
+
2021-08-04 10:42:48
 +
+

Guys, I've merged circleCI publish snapshot PR

+ +

Snapshots can be found bellow: +https://datakin.jfrog.io/artifactory/maven-public-libs-snapshot-local/io/openlineage/openlineage-java/0.0.1-SNAPSHOT/ +openlineage-java-0.0.1-20210804.142910-6.jar +https://datakin.jfrog.io/artifactory/maven-public-libs-snapshot-local/io/openlineage/openlineage-spark/0.1.0-SNAPSHOT/ +openlineage-spark-0.1.0-20210804.143452-5.jar

+ +

Build on main passed (edited)

+ +
+
+ + + + + + +
+ + +
+ 🎉 Julien Le Dem +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-08-04 23:08:08
 +
+

I added a mechanism to enforce spec versioning per: https://github.com/OpenLineage/OpenLineage/issues/63 +https://github.com/OpenLineage/OpenLineage/pull/140

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/issues/63|#63 [PROPOSAL] Define the OpenLineage spec versioning mechanism>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/pull/140|#140 Add mechanism to enforce spec versioning>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ben Teeuwen-Schuiringa + (ben.teeuwen@booking.com) +
+
2021-08-05 10:02:49
 +
+

Hi all, at Booking.com we’re using Spline to extract granular lineage information from spark jobs to be able to trace lineage on column-level and the operations in between. We wrote a custom python parser to create graph-like structure that is sent into arangodb. But tbh, the process is far from stable and is not able to quickly answer questions like ‘which root input columns are used to construct column x’.

+ +

My impression with openlineage thus far is it’s focusing on less granular, table input-output information. Is anyone here trying to accomplish something similar on a column-level?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Luke Smith + (luke.smith@kinandcarta.com) +
+
2021-08-05 12:56:48
 +
+

*Thread Reply:* Also interested in use case / implementation differences between Spline and OL. Watching this thread.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-08-05 14:46:44
 +
+

*Thread Reply:* It would be great to have the option to produce the spline lineage info as OpenLineage. +To capture the column level lineage, you would want to add a ColumnLineage facet to the Output dataset facets. +Which is something that is needed in the spec. +Here is a proposal, please chime in: https://github.com/OpenLineage/OpenLineage/issues/148 +Is this something you would be interested to do?

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/issues/148|#148 [PROPOSAL] column level lineage facet>
+
+ <p><strong>Purpose:</strong> <br /> +For transformations like SQL queries (but also in other cases), we can extract column level lineage. This allows answering questions like ‘which root input columns are used to construct column x?’. We need to be able to capture this information in the OpenLineage model</p> + +<p><strong>Proposed implementation</strong> <br /> +We propose to introduce a new dataset facet: <br /> +example:</p> + +<pre><code>{ + "eventType": "COMPLETE", + "eventTime": "2020-12-28T20:52:00.001+10:00", + "run" : { + "runId": "uuid" + }, + "job": { + "namespace": "scheduler", + "name": "myjob", + "facets": { + "sql": { + "query": "Insert into outputTable from select ** from inputTable" + } + } + }, + "inputs": [ + { + "namespace": "N1", + "name": "inputTable", + "facets": { + "schema": { + "fields": [ {"name": "col_a", "type": "VARCHAR"}, {"name": "col_b", "type": "int"}] + } + } + } + ], + "outputs": [ + { + "namespace": "N2", + "name": "outputTable", + "facets": { + "schema": { + "fields": [ {"name": "col_a", "type": "VARCHAR"}, {"name": "col_b", "type": "int"}] + }, + "columnLineage": { + "fields": { + "col_a": [ { "namespace": "N1", "name": "col_a" } ], + "col_b": [ { "namespace": "N1", "name": "col_b" } ] + } + } + } + } + ] +} +</code></pre> + +<p>Schema:</p> + +<pre><code> "columnLineage": { + "type": "object", + "properties": { + "fields": { + "type": "array", + "items": { + "type": "object", + "additionalProperties": { + "type": "object", + "properties": { + "namespace": { + "type": "string" + }, + "name": { + "type": "string" + }, + }, + "required": [ + "namespace", "name" + ] + } + } + } + } +</code></pre> +
+ +
+
Labels
+ proposal +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-08-09 19:49:51
 +
+

*Thread Reply:* regarding the difference of implementation, the OpenLineage spark integration focuses on extracting metadata and exposing it as a standard representation. (The OpenLineage LineageEvents described in the JSON-Schema spec). The goal is really to have a common language to express lineage and related metadata across everything. We’d be happy if Spline can produce or consume OpenLineage as well and be part of that ecosystem.

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.json | OpenLineage.json>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ben Teeuwen-Schuiringa + (ben.teeuwen@booking.com) +
+
2021-08-18 08:09:38
 +
+

*Thread Reply:* Does anyone know if the Spline developers are in this slack group?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ben Teeuwen-Schuiringa + (ben.teeuwen@booking.com) +
+
2022-08-03 03:07:56
 +
+

*Thread Reply:* @Luke Smith how have things progressed on your side the past year?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-08-09 19:39:28
 +
+

I have opened an issue to track the facet versioning discussion: +https://github.com/OpenLineage/OpenLineage/issues/153

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/issues/153|#153 [PROPOSAL] Version facets independently of the core spec>
+
+ <p><strong>Purpose:</strong> <br /> +The OpenLineage core spec is versioned using a version URL in the <code>$id</code> field: <br /> +See:</p> + +<p>• <a href="https://github.com/OpenLineage/OpenLineage/blob/main/spec/Versioning.md">https://github.com/OpenLineage/OpenLineage/blob/main/spec/Versioning.md</a> +• <a href="https://openlineage.io/spec/1-0-0/OpenLineage.json">https://openlineage.io/spec/1-0-0/OpenLineage.json</a></p> + +<pre><code>{ + "$schema": "&lt;https://json-schema.org/draft/2020-12/schema&gt;", + "$id": "&lt;https://openlineage.io/spec/1-0-0/OpenLineage.json&gt;", +</code></pre> + +<p>Currently all the core facets are also defined in that same core spec.</p> + +<p>This means that the core spec gets a new version whenever a facet is changed along with every other facet. <br /> +We'd rather have the core spec and each facet versioned independently so that their version changes only when they change. <br /> +This matches much better the use cases for using the version number.</p> + +<p>For example:</p> + +<p>• a consumer of OpenLineage would care only about the compatibility of the core spec, accept any version of any facet and handles specific facets independently. +• a producer of a facet cares what version of the core model and that specific facet (core or custom) it is generating. +• a consumer of a specific facet cares about the version of that specific facet and ignore changes to facets it does not consume.</p> + +<p><strong>Proposed implementation</strong></p> + +<p>This proposal is to move each core facet definition into its own file and use the same versioning mechanism (per file) as the core spec.</p> + +<p>This leaves the core spec to only contain the <code>LineageEvent</code> and related core entities (LineageEvent, Run, Job, Dataset) but none of the core facets.</p> + +<p>See for example: <br /> +<a href="https://github.com/OpenLineage/OpenLineage/blob/789b55fc6e6e07d0aedbe130f775721829e83618/spec/OpenLineage.json">https://github.com/OpenLineage/OpenLineage/blob/789b55fc6e6e07d0aedbe130f775721829e83618/spec/OpenLineage.json</a> <br /> +This would be the spec that a consumer would implement to accept any facet version. <br /> +This would ensure that the core spec version changes rarely, only when we add fields to the core model. <br /> +Each facet would be versioned independently following the same versioning mechanism. Each facet version will change if and only if this particular facet schema changes. It also makes core facets and custom facets work exactly the same, they are just maintained in different repositories.</p> + +<p>Example: <br /> +<a href="https://openlineage.io/spec/1-0-0/DatasetSchemaFacet.json">https://openlineage.io/spec/1-0-0/DatasetSchemaFacet.json</a></p> + +<pre><code>{ + "$schema": "&lt;https://json-schema.org/draft/2020-12/schema&gt;", + "$id": "&lt;https://openlineage.io/spec/1-0-0/DatasetSchemaFacet.json&gt;", + "type": "object", + "properties": { + "schema": { + "allOf": [ + { + "$ref": "&lt;https://openlineage.io/spec/1-0-0/OpenLineage.json#/definitions/DatasetFacet&gt;" + }, + { + "type": "object", + "properties": { + "fields": { + "description": "The fields of the table.", + "type": "array", + "items": { + "type": "object", + "properties": { + "name": { + "description": "The name of the field.", + "type": "string", + "example": "column1" + }, + "type": { + "description": "The type of the field.", + "type": "string", + "example": "VARCHAR|INT|..." + }, + "description": { + "description": "The description of the field.", + "type": "string" + } + }, + "required": [ + "name", + "type" + ] + } + } + } + } + ], + "type": "object" + }, + } + } +} +</code></pre> +
+ +
+
Labels
+ proposal +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-08-09 20:16:18
 +
+

I have updated the agenda to the OpenLineage monthly TSC meeting: +https://wiki.lfaidata.foundation/display/OpenLineage/Monthly+TSC+meeting +(meeting information bellow for reference, you can also DM me your email to get added to a google calendar invite)

+ +

The OpenLineage Technical Steering Committee meetings are Monthly on the Second Wednesday 9:00am to 10:00am US Pacific and the link to join the meeting is https://us02web.zoom.us/j/81831865546?pwd=RTladlNpc0FTTDlFcWRkM2JyazM4Zz09 +All are welcome.

+ +

Aug 11th 2021 +• Agenda: + ◦ Coming in OpenLineage 0.1 + ▪︎ OpenLineage spec versioning + ▪︎ Clients + ◦ Marquez integrations imported in OpenLineage + ▪︎ Apache Airflow: + • BigQuery  + • Postgres + • Snowflake + • Redshift + • Great Expectations + ▪︎ Apache Spark + ▪︎ dbt + ◦ OpenLineage 0.2 scope discussion + ▪︎ Facet versioning mechanism + ▪︎ OpenLineage Proxy Backend () + ▪︎ Kafka client + ◦ Roadmap + ◦ Open discussion +• Slides: https://docs.google.com/presentation/d/1Lxp2NB9xk8sTXOnT0_gTXicKX5FsktWa/edit#slide=id.ge80fbcb367_0_14

+ + + +
+ 🙌 Willy Lulciuc, Maciej Obuchowski, Dr Daniel Smith +
+ +
+ 💯 Willy Lulciuc, Dr Daniel Smith +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-08-11 10:05:27
 +
+

*Thread Reply:* Just a reminder that this is in 2 hours

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-08-11 18:50:32
 +
+

*Thread Reply:* I have added the notes to the meeting page: https://wiki.lfaidata.foundation/display/OpenLineage/Monthly+TSC+meeting

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-08-11 18:51:19
 +
+

*Thread Reply:* The recording of the meeting is linked there: +https://us02web.zoom.us/rec/share/2k4O-Rjmmd5TYXzT-pEQsbYXt6o4V6SnS6Vi7a27BPve9aoMmjm-bP8UzBBzsFzg.uY1je-PyT4qTgYLZ?startTime=1628697944000 +• Passcode: =RBUj01C

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Daniel Avancini + (dpavancini@gmail.com) +
+
2021-08-11 13:30:52
 +
+

Hi guys, great discussion today. Something we are particularly interested on is the integration with Airflow 2. I've been searching into Marquez and Openlineage repos and I couldn't find a clear answer on the status of that. I did some work locally to update the marquez-airflow package but I would like to know if someone else is working on this and maybe we could give it some help too.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-08-11 13:36:43
 +
+

*Thread Reply:* @Daniel Avancini I'm working on it. Some changes in airflow made current approach unfeasible, so slight change in a way how we capture events is needed. You can take a look at progress here: https://github.com/OpenLineage/OpenLineage/tree/airflow/2

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Daniel Avancini + (dpavancini@gmail.com) +
+
2021-08-11 13:48:36
 +
+

*Thread Reply:* Thank you Maciej. I'll take a look

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-08-11 20:37:09
 +
+

I have migrated the Marquez issues related to OpenLineage integrations to the OpenLineage repo

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-08-13 19:02:54
 +
+

And OpenLineage 0.1.0 is out ! https://github.com/OpenLineage/OpenLineage/releases/tag/0.1.0

+ + + +
+ 🙌 Peter Hicks, Maciej Obuchowski, Willy Lulciuc, Oleksandr Dvornik, Luke Smith, Daniel Avancini, Matt Gee +
+ +
+ ❤️ Willy Lulciuc, Matt Gee +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Oleksandr Dvornik + (oleksandr.dvornik@getindata.com) +
+
2021-08-16 11:42:24
 +
+

PR ready for review

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/pull/204|#204 Introduce UnknownFacet for spark>
+
+ <p>PR for <a href="https://github.com/OpenLineage/OpenLineage/issues/175">Issue:</a> <br /> +Adding new custom facet for spark with name"spark_unknown". <br /> +Holds information about unvisited nodes from spark plan.</p> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+ 👍 Willy Lulciuc +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Luke Smith + (luke.smith@kinandcarta.com) +
+
2021-08-20 13:54:08
 +
+

Anyone have experience parsing spark's logical plan to generate column-level lineage and DAGs with more human readable operations? I assume I could recreate a graph like the one below using the spark.logicalPlan facet. The analysts writing the SQL / spark queries aren't familiar with ShuffledRowRDD , MapPartitionsRDD, etc... It'd be better if I could convert this plan into spark SQL (or capture spark SQL as a facet at runtime).

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2021-08-26 16:46:53
 +
+

*Thread Reply:* The logicalPlan facet currently returns the Logical Plan, not the physical plan. This means you end up with expressions like Aggregate and Join rather than WholeStageCodegen and Exchange. I don't know if it's possible to reverse engineer the SQL- it's worth looking into the API and trying to find a way to generate that

+ + + +
+
+
+
+ + + + + + + +
+
+ + + + +
+ +
Erick Navarro + (Erick.Navarro@gt.ey.com) +
+
2021-08-31 14:26:35
 +
+

👋 Hi everyone!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Erick Navarro + (Erick.Navarro@gt.ey.com) +
+
2021-08-31 14:27:00
 +
+

Nice to e-meet you 🙂 +I want to use OpenLineage integration for spark in my Azure Databricks clusters, but I am having problems with the configuration of the listener in the cluster, I was wondering if you could help me, if you know any tutorial for the integration of spark with Azure Databricks that can help me, or some more specific guide for this scenario, I would really appreciate it.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Erick Navarro + (Erick.Navarro@gt.ey.com) +
+
2021-08-31 14:27:33
 +
+

I added this configuration to my cluster :

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Erick Navarro + (Erick.Navarro@gt.ey.com) +
+
2021-08-31 14:28:37
 +
+

I receive this error message:

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2021-08-31 14:30:00
 +
+

*Thread Reply:* Hey, @Erick Navarro 👋 . Are you using the openlineage-spark lib? (Note, the marquez-spark lib has been deprecated)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Luke Smith + (luke.smith@kinandcarta.com) +
+
2021-08-31 14:43:20
 +
+

*Thread Reply:* My team had this issue as well. Our read of the error is that Databricks attempts to register the listener before installing packages defined with either spark.jars or spark.jars.packages. Since the listener lib is not yet installed, the listener cannot be found. To solve the issue, we

+ +
  1. copy the OL JAR to a staging directory on DBFS (we use /dbfs/databricks/init/lineage)
  2. using an init script, copy the JAR from the staging directory to the default JAR location for the Databricks driver -- /mnt/driver-daemon/jars
  3. Within the same init script, write the spark config parameters to a .conf file in /databricks/driver/conf (we use open-lineage.conf) +The .conf file will be read by the driver on initialization. It should follow this format (lineagehosturl should point to your API): +[driver] { +"spark.jars" = "/mnt/driver-daemon/jars/openlineage-spark-0.1-SNAPSHOT.jar" +"spark.extraListeners" = "com.databricks.backend.daemon.driver.DBCEventLoggingListener,openlineage.spark.agent.OpenLineageSparkListener" +"spark.openlineage.url" = "$lineage_host_url" +} +Your cluster must be configured to call the init script (enabling lineage for entire cluster). OL is not friendly to notebook-level init as far as we can tell.
    4. +
+ +

@Willy Lulciuc -- I have some utils and init script templates that simplify this process. May be worth adding them to the OL repo along with a readme.

+ + + +
+ 🙏 Erick Navarro +
+ +
+ ❤️ Erick Navarro +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2021-08-31 14:51:46
 +
+

*Thread Reply:* Absolutely, thanks for elaborating on your spark + OL deployment process and I think that’d be great to document. @Michael Collado what are your thoughts?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2021-08-31 14:57:02
 +
+

*Thread Reply:* I haven't tried with Databricks specifically, but there should be no issue registering the OL listener in the Spark config as long as it's done before the Spark session is created- e.g., this example from the README works fine in a vanilla Jupyter notebook- https://github.com/OpenLineage/OpenLineage/tree/main/integration/spark#openlineagesparklistener-as-a-plain-spark-listener

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2021-08-31 15:11:37
 +
+

*Thread Reply:* Looks like Databricks' notebooks come with a Spark instance pre-configured- configuring lineage within the SparkSession configuration doesn't seem possible- https://docs.databricks.com/notebooks/notebooks-manage.html#attach-a-notebook-to-a-cluster 😞

+
+
docs.databricks.com
+ + + +
Manage notebooks | Databricks on AWS
+
+ Learn how to manage notebooks using the UI, the CLI, and by invoking the Workspace API. +
+ + + + + + +
Original URL: https://docs.databricks.com/notebooks/notebooks-manage.html#attach-a-notebook-to-a-cluster
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2021-08-31 15:11:53
 +
+

*Thread Reply:*

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Luke Smith + (luke.smith@kinandcarta.com) +
+
2021-08-31 15:59:38
 +
+

*Thread Reply:* Right, Databricks provides preconfigured spark context / session objects. With Spline, you can set some cluster level config (e.g. spark.spline.lineageDispatcher.http.producer.url ) and install the library on the cluster, but then enable tracking at a notebook level with:

+ +

%scala +import za.co.absa.spline.harvester.SparkLineageInitializer._ +sparkSession.enableLineageTracking() +In OL, it would be nice to install and config OL at a cluster level, but to enable it at a notebook level. This way, users could control whether all notebooks run on a cluster emit lineage or just those with lineage explicitly enabled.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2021-08-31 16:01:00
 +
+

*Thread Reply:* Seems, at the very least, we need to provide a way to specify the job name at the notebook level

+ + + +
+ 👍 Luke Smith +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Luke Smith + (luke.smith@kinandcarta.com) +
+
2021-08-31 16:03:50
 +
+

*Thread Reply:* Agreed. I'd like a default that uses the notebook name that can also be overridden in the notebook.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2021-08-31 16:10:42
 +
+

*Thread Reply:* if you have some insight into the available options, it would be great if you can open an issue on the OL project. I'll have to carve out some time to play with a databricks cluster and learn what options we have

+ + + +
+ 👍 Luke Smith +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Erick Navarro + (Erick.Navarro@gt.ey.com) +
+
2021-08-31 18:26:11
 +
+

*Thread Reply:* Thank you @Luke Smith, the method you recommend works for me, the cluster is running and apparently it fetch the configuration it was my first progress in over a week testing openlineage in azure databricks. Thank you!

+ +

Now I have this:

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Luke Smith + (luke.smith@kinandcarta.com) +
+
2021-08-31 18:52:15
 +
+

*Thread Reply:* Is this error thrown during init or job execution?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2021-08-31 18:55:30
 +
+

*Thread Reply:* this is likely a race condition- I've seen it happen for jobs that start and complete very quickly- things like defining temp views or similar

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Erick Navarro + (Erick.Navarro@gt.ey.com) +
+
2021-08-31 19:59:15
 +
+

*Thread Reply:* During the execution of the job @Luke Smith, thank you @Michael Collado, that was exactly the scenario, the job that I executed was empty, now the cluster is running ok, I don't have errors, I have run some jobs successfully, but I don't see any information in my datakin explorer

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2021-08-31 20:00:46
 +
+

*Thread Reply:* Awesome! Great to hear you’re up and running. For datakin specific questions, mind if we move the discussion to the datakin user slack channel?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Erick Navarro + (Erick.Navarro@gt.ey.com) +
+
2021-08-31 20:01:17
 +
+

*Thread Reply:* Yes Willy, thank you!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Erick Navarro + (Erick.Navarro@gt.ey.com) +
+
2021-09-02 10:06:00
 +
+

*Thread Reply:* Hi , @Luke Smith, thank you for your help, are you familiar with this error in azure databricks when you use OL?

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Erick Navarro + (Erick.Navarro@gt.ey.com) +
+
2021-09-02 10:07:07
 +
+

*Thread Reply:*

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Erick Navarro + (Erick.Navarro@gt.ey.com) +
+
2021-09-02 10:17:17
 +
+

*Thread Reply:* I found the solution here: +https://docs.microsoft.com/en-us/answers/questions/170730/handshake-fails-trying-to-connect-from-azure-datab.html

+
+
docs.microsoft.com
+ + + +
Handshake fails trying to connect from Azure Databricks to Azure PostgreSQL with SSL - Microsoft Q&amp;A
+
+ Microsoft Q&amp;A is the best place to get answers to all your technical questions on Microsoft products and services. Community. Forum. +
+ + + + + + +
Original URL: https://docs.microsoft.com/en-us/answers/questions/170730/handshake-fails-trying-to-connect-from-azure-datab.html
+ + + +
+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Erick Navarro + (Erick.Navarro@gt.ey.com) +
+
2021-09-02 10:17:28
 +
+

*Thread Reply:* It works now! 😄

+ + + +
+ 👍 Luke Smith, Maciej Obuchowski, Minkyu Park, Willy Lulciuc +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2021-09-02 16:33:01
 +
+

*Thread Reply:* @Erick Navarro This might be a helpful to add to our openlineage spark docs for others trying out openlineage-spark with Databricks. Let me know if that’s something you’d like to contribute 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Erick Navarro + (Erick.Navarro@gt.ey.com) +
+
2021-09-02 19:59:10
 +
+

*Thread Reply:* Yes of course @Willy Lulciuc, I will prepare a small tutorial for my colleagues and I will share it with you 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2021-09-02 20:44:36
 +
+

*Thread Reply:* Awesome. Thanks!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Thomas Fredriksen + (thomafred90@gmail.com) +
+
2021-09-02 03:47:35
 +
+

Hello everyone! I am currently evaluating OpenLineage and am finding it very interesting as Prefect is in the list of integrations. However, I am not seeing any documentation or code for this. How far are you from supporting Prefect?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-09-02 04:57:55
 +
+

*Thread Reply:* Hey! If you mean this picture, it provides concept of how OpenLineage works, not current state of integration. We don't have Prefect support yet; hovewer, it's on our roadmap.

+
+ + + + +
+
+ +
+ + + + + + + + +
Original URL: https://raw.githubusercontent.com/OpenLineage/OpenLineage/main/doc/problem.png
+ + + +
+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/issues/81|#81 [INTEGRATION] Add Prefect support>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Thomas Fredriksen + (thomafred90@gmail.com) +
+
2021-09-02 05:22:15
 +
+

*Thread Reply:* great, thanks 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-09-02 11:49:48
 +
+

*Thread Reply:* @Thomas Fredriksen Feel free to chime in the github issue Maciej linked if you want.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Luke Smith + (luke.smith@kinandcarta.com) +
+
2021-09-02 13:13:05
 +
+

What's the timeline to support spark 3.0 within OL? One breaking change we've found is within DatasetSourceVisitor.java -- the DataSourceV2 is deprecated in spark 3.0. There may be other issues we haven't found yet. Is there a good feel for the scope of work required to make OL spark 3.0 compatible?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-09-02 14:28:11
 +
+

*Thread Reply:* It is being worked on right now. @Oleksandr Dvornik is adding an integration test in the build so that we run test for both spark 2.4 and spark 3. Please open an issue with the stack trace if you can. From our perspective, it should be mostly compatible with a few exceptions like this one that we’d want to add test cases for.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-09-02 14:36:19
 +
+

*Thread Reply:* The goal is to be able to make a release in the next few weeks. The integration is being used with Spark 3 already.

+ + + +
+ 🙌 Luke Smith +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Luke Smith + (luke.smith@kinandcarta.com) +
+
2021-09-02 15:50:14
 +
+

*Thread Reply:* Great, I'll take some time to open an issue for this particular issue and a few others.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2021-09-02 17:33:08
 +
+

*Thread Reply:* are you actually using the DatasetSource interface in any capacity? Or are you just scanning the source code to find incompatibilities?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Luke Smith + (luke.smith@kinandcarta.com) +
+
2021-09-03 12:36:20
 +
+

*Thread Reply:* Turns out this has more to do with a how Databricks handles the delta format. It's related to https://github.com/AbsaOSS/spline-spark-agent/issues/96.

+
+ + + + +
<https://github.com/AbsaOSS/spline-spark-agent/issues/96|#96 Throwing runtime exception for delta commands>
+
+ <p>Current spline is not supported for delta commands such as MergeInto, Update, Delete</p> + +<p>I have added <code>io.delta</code> package with scope provided as databricks cluster by default has <code>io.delta</code> imported, and made changes to <code>WriteCommandExtractor.scala</code> file.</p> + +<pre><code>import org.apache.spark.sql.delta.commands._ +</code></pre> + +<p>and added below code to commandsToBeImplemented just to see if it throws custom exception when run in Required mode.</p> + +<pre><code>classOf[MergeIntoCommand] +</code></pre> + +<p>But it throws following exception when I run an application</p> + +<pre><code>Caused by: java.lang.ClassNotFoundException: org.apache.spark.sql.delta.commands.MergeIntoCommand + +</code></pre> + +<p>I am trying to debug the issue but your support on this is highly appreciated.</p> +
+ +
+
Labels
+ question +
+ +
+
Comments
+ 5 +
+ + + + + + + +
+ + <a href="https://github.com/AbsaOSS/spline-spark-agent">AbsaOSS/spline-spark-agent</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Luke Smith + (luke.smith@kinandcarta.com) +
+
2021-09-03 13:42:43
 +
+

*Thread Reply:* I haven't been chasing this issue down on my team -- turns out some things were lost in communication. There are really two problems here:

+ +
  1. When attempting to do delta I/O with Spark 3 on Databricks, e.g. +insert into . . . values . . . +We get an error related to DataSourceV2: +java.lang.NoSuchMethodError: org.apache.spark.sql.execution.datasources.v2.DataSourceV2Relation.source()Lorg/apache/spark/sql/sources/v2/DataSourceV2;
  2. Using Spline, which is Spark 3 compatible, we have issues with the way Databricks handles delta table io. This is related: https://github.com/AbsaOSS/spline-spark-agent/issues/96
    3. +
+ +

So there are two stacked issues related to spark 3 on Databricks with delta IO, not just one. Hope this clears things up.

+
+ + + + +
<https://github.com/AbsaOSS/spline-spark-agent/issues/96|#96 Throwing runtime exception for delta commands>
+
+ <p>Current spline is not supported for delta commands such as MergeInto, Update, Delete</p> + +<p>I have added <code>io.delta</code> package with scope provided as databricks cluster by default has <code>io.delta</code> imported, and made changes to <code>WriteCommandExtractor.scala</code> file.</p> + +<pre><code>import org.apache.spark.sql.delta.commands._ +</code></pre> + +<p>and added below code to commandsToBeImplemented just to see if it throws custom exception when run in Required mode.</p> + +<pre><code>classOf[MergeIntoCommand] +</code></pre> + +<p>But it throws following exception when I run an application</p> + +<pre><code>Caused by: java.lang.ClassNotFoundException: org.apache.spark.sql.delta.commands.MergeIntoCommand + +</code></pre> + +<p>I am trying to debug the issue but your support on this is highly appreciated.</p> +
+ +
+
Labels
+ question +
+ +
+
Comments
+ 5 +
+ + + + + + + +
+ + <a href="https://github.com/AbsaOSS/spline-spark-agent">AbsaOSS/spline-spark-agent</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2021-09-03 13:44:54
 +
+

*Thread Reply:* So, the first issue is OpenLineage related directly, and the second issue applies to both OpenLineage and Spline?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Luke Smith + (luke.smith@kinandcarta.com) +
+
2021-09-03 13:45:49
 +
+

*Thread Reply:* Yes, that's my read of what I'm getting from others on the team.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2021-09-03 13:46:56
 +
+

*Thread Reply:* For the first issue- can you give some details about the target of the INSERT INTO... ? Is it a data source defined in Databricks? a Hive table? a view on GCS?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2021-09-03 13:47:40
 +
+

*Thread Reply:* oh, it's a Delta table?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Luke Smith + (luke.smith@kinandcarta.com) +
+
2021-09-03 14:48:15
 +
+

*Thread Reply:* Yes, it's created via

+ +

CREATE TABLE . . . using DELTA location "/dbfs/mnt/ . . . "

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-09-02 14:28:53
 +
+

I have opened a PR to fix some outdated language in the spec: https://github.com/OpenLineage/OpenLineage/pull/241 Thank you @Mandy Chessell for the feedback

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/pull/241|#241 correct remaining use of transition>
+
+ <p>Signed-off-by: Julien Le Dem <a href="mailto:julien@apache.org">julien@apache.org</a></p> +
+ +
+
Comments
+ 1 +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-09-02 14:37:27
 +
+

The next OpenLineage monthly meeting is next week. Please chime in this thread if you’d like something added to the agenda

+ + + +
+ 🙌 Willy Lulciuc +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
marko + (marko.kristian.helin@gmail.com) +
+
2021-09-04 12:53:54
 +
+

*Thread Reply:* Apache Beam integration? I have a very crude integration at the moment. Maybe it’s better to integrate on the orchestration level (airflow, luigi). Thoughts?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-09-05 13:06:19
 +
+

*Thread Reply:* I think it makes a lot of sense to have a Beam level integration similar to the spark one. Feel free to post a draft PR if you want to share.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-09-07 21:04:09
 +
+

*Thread Reply:* I have added Beam as a topic for the roadmap discussion slide: https://docs.google.com/presentation/d/1fI0u8aE0iX9vG4GGrnQYAEcsJM9z7Rlv/edit#slide=id.ge7d4b64ef4_0_0

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-09-07 21:03:08
 +
+

I have prepared slides for the OpenLineage meeting tomorrow morning: https://docs.google.com/presentation/d/1fI0u8aE0iX9vG4GGrnQYAEcsJM9z7Rlv/edit#slide=id.ge7d4b64ef4_0_0

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-09-07 21:03:32
 +
+

*Thread Reply:* There will be a quick demo of the dbt integration (thanks @Willy Lulciuc!)

+ + + +
+ 🙌 Willy Lulciuc +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-09-07 21:05:13
 +
+

*Thread Reply:* Information to join and archive of previous meetings: https://wiki.lfaidata.foundation/display/OpenLineage/Monthly+TSC+meeting

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-09-08 14:49:52
 +
+

*Thread Reply:* The recording and notes are now available: https://wiki.lfaidata.foundation/display/OpenLineage/Monthly+TSC+meeting

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Venkatesh Tadinada + (venkat@mlacademy.io) +
+
2021-09-08 21:58:09
 +
+

*Thread Reply:* Good meeting today. @Julien Le Dem. Thanks

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Shreyas Kaushik + (shreyask@gmail.com) +
+
2021-09-08 04:03:29
 +
+

Hello, was looking to get some lineage out for BQ in my Airflow DAGs and saw that the BQ extractor here - https://github.com/OpenLineage/OpenLineage/blob/main/integration/airflow/openlineage/airflow/extractors/bigquery_extractor.py#L47 is using an operator that has been deprecated by Airflow - https://github.com/apache/airflow/blob/main/airflow/contrib/operators/bigquery_operator.py#L44 and most of my DAGs are using the operator BigQueryExecuteQueryOperator mentioned there. I presume with this lineage extraction wouldn’t work and some work is needed to support both these operators with the same ( or differnt) extractor. Is that correct or am I missing something ?

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/integration/airflow/openlineage/airflow/extractors/bigquery_extractor.py | bigquery_extractor.py>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+
+ + + + +
<https://github.com/apache/airflow/blob/main/airflow/contrib/operators/bigquery_operator.py | bigquery_operator.py>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/apache/airflow">apache/airflow</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-09-08 04:27:04
 +
+

*Thread Reply:* We're working on updating our integration to airflow 2. Some changes in airflow made current approach unfeasible, so slight change in a way how we capture events is needed. You can take a look at progress here: https://github.com/OpenLineage/OpenLineage/tree/airflow/2

+
+ + + + +
<https://github.com/apache/airflow/pull/10956|#10956 Fully support running more than one scheduler concurrently>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/apache/airflow">apache/airflow</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Shreyas Kaushik + (shreyask@gmail.com) +
+
2021-09-08 04:27:38
 +
+

*Thread Reply:* Thanks @Maciej Obuchowski When is this expected to land in a release ?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Daniel Zagales + (dzagales@gmail.com) +
+
2021-11-11 06:35:24
 +
+

*Thread Reply:* hi @Maciej Obuchowski I wanted to follow up on this to understand when the more recent BQ Operators will be supported, specifically BigQueryInsertJobOperator

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-09-11 22:30:31
 +
+

The PR to separate facets in their own file (and allowing versioning them independently) is now available: https://github.com/OpenLineage/OpenLineage/pull/118

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/pull/118|#118 separate facets to allow versioning them independently>
+
+ <p>This PR includes:</p> + +<p>• Splitting facet outside of the core spec into their own file. + • I wrote a separate automated program to generate each facet file from the original so you don't need to check that they have all the fields. + • You might to review what has been removed from OpenLineage.json and the general structure of each facet +• Updating code gen to deal with that + • Code generation merges all the definitions to be compatible with the existing logic. + • Adding more precise list of BaseFacets (for each entity type). Which in code gen allows for type checking adding the right type of facet in the right place + • It also adds a default implementation to the BaseFacets to allow deserializing custom facets.</p> + +<p>Signed-off-by: Julien Le Dem <a href="mailto:julien@apache.org">julien@apache.org</a></p> +
+ +
+
Comments
+ 1 +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jose Badeau + (jose.badeau@gmail.com) +
+
2021-09-13 03:46:20
 +
+

Hi, new to the channel but I think OL is a great initiative. Currently we are focused on beam/spark/delta but are moving to beam/flink/iceberg and I’m happy to help where I can.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2021-09-13 15:40:01
 +
+

*Thread Reply:* Welcome, @Jose Badeau 👋. That’s exciting to hear as we have Beam, Flink and Iceberg on our roadmap! Your welcome to join the discussion :)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-09-13 20:56:11
 +
+

Per the discussion last week, Ryan updated the metadata that would be available in Iceberg: https://github.com/OpenLineage/OpenLineage/issues/167#issuecomment-917237320

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/issues/167#issuecomment-917237320|Comment on #167 [INTEGRATION] Iceberg should be able to send OpenLineage events independently from where the library is used (including in a plain java process)>
+
+ <p>Here's an outline of what I talked about at the last open lineage meeting:</p> + +<p>OpenLineage and Iceberg</p> + +<p>What is Iceberg?</p> + +<p>• A table format that tracks the files in a table +• A library that implements the table format +• In the format, at "snapshot" is a version of a table +• Provides atomic operations to update the contents of a table +• Provides helper utilities for scanning data</p> + +<p>Iceberg provides notifications at certain points</p> + +<p>• For example, it plans a table scan, Iceberg summarizes that scan and sends a notification with + • The table that was scanned + • The snapshot that was used + • The filters that were pushed down + • The set of columns that were projected</p> + +<p>We want to do the same thing for metrics</p> + +<p>• Scan</p> + +<pre><code>• How many manifest files were read? Data and delete manifests? +• How many data files were read? How many total delete files? How many were reused? +• How long did each manifest read take? +• How long did the total scan planning take? +• How many files per second were located? What was the distribution of files/second over time? +</code></pre> + +<p>• Commit</p> + +<pre><code>• What snapshot was committed? +• What columns were written? +• What partitions were modified or appended to? +</code></pre> + +<p>• Commit metrics</p> + +<pre><code>• How many manifests were rewritten? +• How many manifests were created? +• How many data files were created? +• How many data files were removed? +• How many delete files were created? +• How much data (bytes) was written? +• How much data (bytes) was removed? +• How many retries were needed to complete the commit? +• How long did the first attempt take? +• What was the average length of a retry? +• For each retry, how many manifests were rewritten/created/removed? +• For each retry, how many data/delete files were created/removed? +</code></pre> + +<p>• Commits create a dataset facet</p> + +<pre><code>• Snapshot/version of the table +• Total number of rows +• Total number of data/delete files +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-09-13 21:00:54
 +
+

I have also created tickets for follow up discussions: (#269 and #270): https://wiki.lfaidata.foundation/display/OpenLineage/Monthly+TSC+meeting

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tomas Satka + (satka.tomas@gmail.com) +
+
2021-09-14 04:50:22
 +
+

Hello. I find OpenLineage an interesting tool however can someone help me with integration?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tomas Satka + (satka.tomas@gmail.com) +
+
2021-09-14 04:52:50
 +
+

I am trying to capture lineage from spark 3.1.1 but when executing i constantly get: java.lang.NoSuchMethodError: org.apache.spark.sql.execution.datasources.v2.WriteToDataSourceV2.writer()Lorg/apache/spark/sql/sources/v2/writer/DataSourceWriter; + at openlineage.spark.agent.lifecycle.plan.DatasetSourceVisitor.findDatasetSource(DatasetSourceVisitor.java:57) as if i would be using openlineage on wrong spark version (2.4) I have tried also spark jar from branch feature/itspark3. Is there any branch or release that works or can be tried with spark 3+?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Oleksandr Dvornik + (oleksandr.dvornik@getindata.com) +
+
2021-09-14 05:03:45
 +
+

*Thread Reply:* Hello Tomas. We are currently working on support for spark v3. Can you please raise an issue with stack trace, that would help us to track and solve it. We are currently adding integration tests. Next step would be fix changes in method signatures for v3 (that's what you have)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tomas Satka + (satka.tomas@gmail.com) +
+
2021-09-14 05:12:45
 +
+

*Thread Reply:* Hi @Oleksandr Dvornik i raised https://github.com/OpenLineage/OpenLineage/issues/272

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/issues/272|#272 spark 3 lineage>
+
+ <p>Hi Team</p> + +<p>I am trying to run openlineage with spark 3.1.1. I have tried the version that is released 0.2.2 but also build from main from branch feature/it<em>spark</em>3 but i always get the same exception.</p> + +<pre><code>java.lang.NoSuchMethodError: org.apache.spark.sql.execution.datasources.v2.WriteToDataSourceV2.writer()Lorg/apache/spark/sql/sources/v2/writer/DataSourceWriter; + at openlineage.spark.agent.lifecycle.plan.DatasetSourceVisitor.findDatasetSource(DatasetSourceVisitor.java:57) + at openlineage.spark.agent.lifecycle.plan.DatasetSourceVisitor.isDefinedAt(DatasetSourceVisitor.java:33) + at openlineage.spark.agent.lifecycle.plan.DatasetSourceVisitor.isDefinedAt(DatasetSourceVisitor.java:28) + at openlineage.spark.agent.lifecycle.plan.wrapper.OutputDatasetVisitor.isDefinedAt(OutputDatasetVisitor.java:27) + at openlineage.spark.agent.lifecycle.plan.wrapper.OutputDatasetVisitor.isDefinedAt(OutputDatasetVisitor.java:16) + at scala.PartialFunction$OrElse.isDefinedAt(PartialFunction.scala:170) + at scala.PartialFunction$OrElse.isDefinedAt(PartialFunction.scala:170) + at scala.PartialFunction$OrElse.isDefinedAt(PartialFunction.scala:170) + at scala.PartialFunction$OrElse.isDefinedAt(PartialFunction.scala:170) + at scala.PartialFunction$OrElse.isDefinedAt(PartialFunction.scala:170) + at openlineage.spark.agent.lifecycle.plan.PlanTraversal.isDefinedAt(PlanTraversal.java:32) + at openlineage.spark.agent.lifecycle.SparkSQLExecutionContext.start(SparkSQLExecutionContext.java:87) + at openlineage.spark.agent.OpenLineageSparkListener.lambda$onJobStart$3(OpenLineageSparkListener.java:154) + at java.util.Optional.ifPresent(Optional.java:159) + at openlineage.spark.agent.OpenLineageSparkListener.onJobStart(OpenLineageSparkListener.java:143) + at org.apache.spark.scheduler.SparkListenerBus.doPostEvent(SparkListenerBus.scala:37) + at org.apache.spark.scheduler.SparkListenerBus.doPostEvent$(SparkListenerBus.scala:28) + at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37) + at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37) + at org.apache.spark.util.ListenerBus.postToAll(ListenerBus.scala:117) + at org.apache.spark.util.ListenerBus.postToAll$(ListenerBus.scala:101) + at org.apache.spark.scheduler.AsyncEventQueue.super$postToAll(AsyncEventQueue.scala:105) + at org.apache.spark.scheduler.AsyncEventQueue.$anonfun$dispatch$1(AsyncEventQueue.scala:105) + at scala.runtime.java8.JFunction0$mcJ$sp.apply(JFunction0$mcJ$sp.java:23) + at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62) + at &lt;a href="http://org.apache.spark.scheduler.AsyncEventQueue.org"&gt;org.apache.spark.scheduler.AsyncEventQueue.org&lt;/a&gt;$apache$spark$scheduler$AsyncEventQueue$$dispatch(AsyncEventQueue.scala:100) + at org.apache.spark.scheduler.AsyncEventQueue$$anon$2.$anonfun$run$1(AsyncEventQueue.scala:96) + at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1381) + at org.apache.spark.scheduler.AsyncEventQueue$$anon$2.run(AsyncEventQueue.scala:96) + + +</code></pre> + +<p>Do i do something wrong or is spark 3 simply not yet supported? <br /> +Thanks</p> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+ 👍 Oleksandr Dvornik +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tomas Satka + (satka.tomas@gmail.com) +
+
2021-09-14 08:47:39
 +
+

I also tried to downgrade spark to 2.4.0 and retry with 0.2.2 but i also faced issue.. so my preferred way would be to push for spark 3.1.1 but depends a bit on when you plan to release version supporting it. As backup plan i would try spark 2.4.0 but this is blocking me also: https://github.com/OpenLineage/OpenLineage/issues/274

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/issues/274|#274 spark 2.4 ArrayIndexOutOfBoundsException>
+
+ <p>Hi Team</p> + +<p>I have tried to collect lineage on spark 2.4 with openlineage 0.2.2. <br /> +Here is my pom.xml;</p> + +<pre><code>&amp;lt;project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" + xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"&amp;gt; + &amp;lt;modelVersion&amp;gt;4.0.0&amp;lt;/modelVersion&amp;gt; + + &amp;lt;groupId&amp;gt;foo&amp;lt;/groupId&amp;gt; + &amp;lt;artifactId&amp;gt;bar&amp;lt;/artifactId&amp;gt; + &amp;lt;version&amp;gt;0.1.0-SNAPSHOT&amp;lt;/version&amp;gt; + + &amp;lt;properties&amp;gt; + &amp;lt;java.version&amp;gt;1.8&amp;lt;/java.version&amp;gt; + &amp;lt;maven.compiler.source&amp;gt;${java.version}&amp;lt;/maven.compiler.source&amp;gt; + &amp;lt;maven.compiler.target&amp;gt;${java.version}&amp;lt;/maven.compiler.target&amp;gt; + &amp;lt;project.build.sourceEncoding&amp;gt;UTF-8&amp;lt;/project.build.sourceEncoding&amp;gt; + &amp;lt;project.reporting.outputEncoding&amp;gt;UTF-8&amp;lt;/project.reporting.outputEncoding&amp;gt; + &amp;lt;spark2.version&amp;gt;2.4.0&amp;lt;/spark2.version&amp;gt; + &amp;lt;/properties&amp;gt; + + &amp;lt;build&amp;gt; + &amp;lt;plugins&amp;gt; + &amp;lt;plugin&amp;gt; + &amp;lt;groupId&amp;gt;org.apache.maven.plugins&amp;lt;/groupId&amp;gt; + &amp;lt;artifactId&amp;gt;maven-compiler-plugin&amp;lt;/artifactId&amp;gt; + &amp;lt;version&amp;gt;3.8.1&amp;lt;/version&amp;gt; + &amp;lt;/plugin&amp;gt; + &amp;lt;plugin&amp;gt; + &amp;lt;groupId&amp;gt;org.apache.maven.plugins&amp;lt;/groupId&amp;gt; + &amp;lt;artifactId&amp;gt;maven-jar-plugin&amp;lt;/artifactId&amp;gt; + &amp;lt;version&amp;gt;2.3.2&amp;lt;/version&amp;gt; + &amp;lt;configuration&amp;gt; + &amp;lt;finalName&amp;gt;${project.artifactId}&amp;lt;/finalName&amp;gt; + &amp;lt;/configuration&amp;gt; + &amp;lt;/plugin&amp;gt; + &amp;lt;plugin&amp;gt; + &amp;lt;groupId&amp;gt;org.apache.maven.plugins&amp;lt;/groupId&amp;gt; + &amp;lt;artifactId&amp;gt;maven-shade-plugin&amp;lt;/artifactId&amp;gt; + &amp;lt;version&amp;gt;3.2.4&amp;lt;/version&amp;gt; + &amp;lt;executions&amp;gt; + &amp;lt;execution&amp;gt; + &amp;lt;phase&amp;gt;package&amp;lt;/phase&amp;gt; + &amp;lt;goals&amp;gt; + &amp;lt;goal&amp;gt;shade&amp;lt;/goal&amp;gt; + &amp;lt;/goals&amp;gt; + &amp;lt;configuration&amp;gt; + &amp;lt;filters&amp;gt; + &amp;lt;filter&amp;gt; + &amp;lt;artifact&amp;gt;**:**&amp;lt;/artifact&amp;gt; + &amp;lt;excludes&amp;gt; + &amp;lt;exclude&amp;gt;META-INF/LICENSE&amp;lt;/exclude&amp;gt; + &amp;lt;exclude&amp;gt;META-INF/**.SF&amp;lt;/exclude&amp;gt; + &amp;lt;exclude&amp;gt;META-INF/**.DSA&amp;lt;/exclude&amp;gt; + &amp;lt;exclude&amp;gt;META-INF/**.RSA&amp;lt;/exclude&amp;gt; + &amp;lt;/excludes&amp;gt; + &amp;lt;/filter&amp;gt; + &amp;lt;/filters&amp;gt; + &amp;lt;transformers&amp;gt; + &amp;lt;transformer + implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"/&amp;gt; + &amp;lt;/transformers&amp;gt; + &amp;lt;/configuration&amp;gt; + &amp;lt;/execution&amp;gt; + &amp;lt;/executions&amp;gt; + &amp;lt;/plugin&amp;gt; + &amp;lt;/plugins&amp;gt; + &amp;lt;/build&amp;gt; + + &amp;lt;dependencies&amp;gt; + &amp;lt;dependency&amp;gt; + &amp;lt;groupId&amp;gt;org.apache.spark&amp;lt;/groupId&amp;gt; + &amp;lt;artifactId&amp;gt;spark-sql-kafka-0-10_2.12&amp;lt;/artifactId&amp;gt; + &amp;lt;version&amp;gt;${spark2.version}&amp;lt;/version&amp;gt; + &amp;lt;/dependency&amp;gt; + &amp;lt;dependency&amp;gt; + &amp;lt;groupId&amp;gt;org.apache.spark&amp;lt;/groupId&amp;gt; + &amp;lt;artifactId&amp;gt;spark-core_2.12&amp;lt;/artifactId&amp;gt; + &amp;lt;version&amp;gt;${spark2.version}&amp;lt;/version&amp;gt; + &amp;lt;scope&amp;gt;provided&amp;lt;/scope&amp;gt; + &amp;lt;/dependency&amp;gt; + &amp;lt;dependency&amp;gt; + &amp;lt;groupId&amp;gt;org.apache.spark&amp;lt;/groupId&amp;gt; + &amp;lt;artifactId&amp;gt;spark-sql_2.12&amp;lt;/artifactId&amp;gt; + &amp;lt;version&amp;gt;${spark2.version}&amp;lt;/version&amp;gt; + &amp;lt;scope&amp;gt;provided&amp;lt;/scope&amp;gt; + &amp;lt;/dependency&amp;gt; + &amp;lt;dependency&amp;gt; + &amp;lt;groupId&amp;gt;org.apache.spark&amp;lt;/groupId&amp;gt; + &amp;lt;artifactId&amp;gt;spark-streaming_2.12&amp;lt;/artifactId&amp;gt; + &amp;lt;version&amp;gt;${spark2.version}&amp;lt;/version&amp;gt; + &amp;lt;scope&amp;gt;provided&amp;lt;/scope&amp;gt; + &amp;lt;/dependency&amp;gt; + &amp;lt;dependency&amp;gt; + &amp;lt;groupId&amp;gt;io.openlineage&amp;lt;/groupId&amp;gt; + &amp;lt;artifactId&amp;gt;openlineage-spark&amp;lt;/artifactId&amp;gt; + &amp;lt;version&amp;gt;0.2.2&amp;lt;/version&amp;gt; + &amp;lt;/dependency&amp;gt; + + &amp;lt;/dependencies&amp;gt; + +&amp;lt;/project&amp;gt; +</code></pre> + +<p>This is my simple test that reads csv and write to csv:</p> + +<pre><code>import org.apache.spark.sql.SaveMode; +import org.apache.spark.sql.SparkSession; + +import java.time.Duration; + +public class Example { + public static void main(String[] args) throws InterruptedException { + SparkSession.Builder builder = SparkSession.builder(); + SparkSession session = builder + .appName("java example app") + .master("local[**]") + .config("spark.jars.packages", "io.openlineage:openlineage_spark:0.2.2") + .config("spark.extraListeners", "io.openlineage.spark.agent.OpenLineageSparkListener") + .config("spark.openlineage.url", "&lt;http://localhost:5000/api/v1/namespaces/spark_integration/&gt;") + .getOrCreate(); + + session.read() + .option("header", "true") + .option("inferSchema", "true") + .csv("src/main/resources/wikidata.csv") + .as("source") + .write() + .mode(SaveMode.Overwrite) + .csv("src/main/resources/java-sample.csv"); + Thread.sleep(Duration.ofMinutes(3).toMillis()); + } +} +</code></pre> + +<p>but when i run it i am getting following exception:</p> + +<p>``` +Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: Index 10582 out of bounds for length 249 + at com.thoughtworks.paranamer.BytecodeReadingParanamer$ClassReader.accept(BytecodeReadingParanamer.java:563) + at com.thoughtworks.paranamer.BytecodeReadingParanamer$ClassReader.access$200(BytecodeReadingParanamer.java:338) + at com.thoughtworks.paranamer.BytecodeReadingParanamer.lookupParameterNames(BytecodeReadingParanamer.java:103) + at com.thoughtworks.paranamer.CachingParanamer.lookupParameterNames(CachingParanamer.java:90) + at com.fasterxml.jackson.module.scala.introspect.BeanIntrospector$.getCtorParams(BeanIntrospector.scala:44) + at com.fasterxml.jackson.module.scala.introspect.BeanIntrospector$.$anonfun$apply$1(BeanIntrospector.scala:58) + at com.fasterxml.jackson.module.scala.introspect.BeanIntrospector$.$anonfun$apply$1$adapted(BeanIntrospector.scala:58) + at scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:240) + at scala.collection.Iterator.foreach(Iterator.scala:937) + at scala.collection.Iterator.foreach$(Iterator.scala:937) + at scala.collection.AbstractIterator.foreach(Iterator.scala:1425) + at scala.collection.IterableLike.foreach(IterableLike.scala:70) + at scala.collection.IterableLike.foreach$(IterableLike.scala:69) + at scala.collection.AbstractIterable.foreach(Iterable.scala:54) + at scala.collection.TraversableLike.flatMap(TraversableLike.scala:240) + at scala.collection.TraversableLike.flatMap$(TraversableLike.scala:237) + at scala.collection.AbstractTraversable.flatMap(Traversable.scala:104) + at com.fasterxml.jackson.module.scala.introspect.BeanIntrospector$.findConstructorParam$1(BeanIntrospector.scala:58) + at com.fasterxml.jackson.module.scala.introspect.BeanIntrospector$.$anonfun$apply$19(BeanIntrospector.scala:176) + at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:233) + at scala.collection.In…</p> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-09-14 08:55:44
 +
+

*Thread Reply:* I think this might be actually spark issue: https://stackoverflow.com/questions/53787624/spark-throwing-arrayindexoutofboundsexception-when-parallelizing-list/53787847

+
+
Stack Overflow
+ + + +
Spark throwing ArrayIndexOutOfBoundsException when parallelizing list
+
+ I've tried to create a Spark implementation of QuickSort to test against a serial implementation. I've got the serial implementation working, but the parallel implementation throws an +
+ + + + + + + + +
Original URL: https://stackoverflow.com/questions/53787624/spark-throwing-arrayindexoutofboundsexception-when-parallelizing-list/53787847
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-09-14 08:56:10
 +
+

*Thread Reply: Can you try newer version in 2.4.* line, like 2.4.7?

+ + + +
+ 👀 Tomas Satka +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-09-14 08:57:30
 +
+

*Thread Reply:* This might be also spark 2.4 with scala 2.12 issue - I'd recomment 2.11 versions.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tomas Satka + (satka.tomas@gmail.com) +
+
2021-09-14 09:04:26
 +
+

*Thread Reply:* @Maciej Obuchowski with 2.4.7 i get following exc:

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tomas Satka + (satka.tomas@gmail.com) +
+
2021-09-14 09:04:27
 +
+

*Thread Reply:* 21/09/14 15:03:25 WARN RddExecutionContext: Unable to access job conf from RDD +java.lang.NoSuchFieldException: config$1 + at java.base/java.lang.Class.getDeclaredField(Class.java:2411)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tomas Satka + (satka.tomas@gmail.com) +
+
2021-09-14 09:04:48
 +
+

*Thread Reply:* i can also try to switch to 2.11 scala

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tomas Satka + (satka.tomas@gmail.com) +
+
2021-09-14 09:05:37
 +
+

*Thread Reply:* or do you have some recommended setup that works for sure?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-09-14 09:09:58
 +
+

*Thread Reply:* One more check - you're using Java 8 with this, right?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-09-14 09:10:17
 +
+

*Thread Reply:* This is what works for me: +-&gt; % cat tools/spark-2.4/RELEASE +Spark 2.4.8 (git revision 4be4064) built for Hadoop 2.7.3 +Build flags: -B -Pmesos -Pyarn -Pkubernetes -Pflume -Psparkr -Pkafka-0-8 -Phadoop-2.7 -Phive -Phive-thriftserver -DzincPort=3036

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-09-14 09:11:23
 +
+

*Thread Reply:* spark-shell: +Using Scala version 2.11.12 (OpenJDK 64-Bit Server VM, Java 1.8.0_292)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tomas Satka + (satka.tomas@gmail.com) +
+
2021-09-14 09:12:05
 +
+

*Thread Reply:* awesome let me try 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tomas Satka + (satka.tomas@gmail.com) +
+
2021-09-14 09:26:00
 +
+

*Thread Reply:* data has been sent to marquez. coolio. however i noticed nullpointer being thrown: 21/09/14 15:23:53 ERROR AsyncEventQueue: Listener OpenLineageSparkListener threw an exception +java.lang.NullPointerException + at io.openlineage.spark.agent.OpenLineageSparkListener.onJobEnd(OpenLineageSparkListener.java:164) + at org.apache.spark.scheduler.SparkListenerBus$class.doPostEvent(SparkListenerBus.scala:39) + at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37) + at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37) + at org.apache.spark.util.ListenerBus$class.postToAll(ListenerBus.scala:91) + at <a href="http://org.apache.spark.scheduler.AsyncEventQueue.org">org.apache.spark.scheduler.AsyncEventQueue.org</a>$apache$spark$scheduler$AsyncEventQueue$$super$postToAll(AsyncEventQueue.scala:92) + at org.apache.spark.scheduler.AsyncEventQueue$$anonfun$org$apache$spark$scheduler$AsyncEventQueue$$dispatch$1.apply$mcJ$sp(AsyncEventQueue.scala:92) + at org.apache.spark.scheduler.AsyncEventQueue$$anonfun$org$apache$spark$scheduler$AsyncEventQueue$$dispatch$1.apply(AsyncEventQueue.scala:87) + at org.apache.spark.scheduler.AsyncEventQueue$$anonfun$org$apache$spark$scheduler$AsyncEventQueue$$dispatch$1.apply(AsyncEventQueue.scala:87) + at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58) + at <a href="http://org.apache.spark.scheduler.AsyncEventQueue.org">org.apache.spark.scheduler.AsyncEventQueue.org</a>$apache$spark$scheduler$AsyncEventQueue$$dispatch(AsyncEventQueue.scala:87) + at org.apache.spark.scheduler.AsyncEventQueue$$anon$1$$anonfun$run$1.apply$mcV$sp(AsyncEventQueue.scala:83) + at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1302) + at org.apache.spark.scheduler.AsyncEventQueue$$anon$1.run(AsyncEventQueue.scala:82)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tomas Satka + (satka.tomas@gmail.com) +
+
2021-09-14 10:59:45
 +
+

*Thread Reply:* closed related issue #274

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tomas Satka + (satka.tomas@gmail.com) +
+
2021-09-14 11:02:42
 +
+

does openlineage capture streaming in spark? as this example is not showing me anything unless i replace readStream() with batch read() and writeStream() with write() +```SparkSession.Builder builder = SparkSession.builder(); + SparkSession session = builder + .appName("quantweave") + .master("local[**]") + .config("spark.jars.packages", "io.openlineage:openlineage_spark:0.2.2") + .config("spark.extraListeners", "io.openlineage.spark.agent.OpenLineageSparkListener") + .config("spark.openlineage.url", "http://localhost:5000/api/v1/namespaces/spark_integration/") + .getOrCreate();

+ +
    Dataset&lt;Row&gt; df = session
+            .readStream()
+            .format("kafka")
+            .option("kafka.bootstrap.servers", "localhost:9092")
+            .option("subscribe", "topic1")
+            .option("startingOffsets", "earliest")
+            .load();
+
+    Dataset&lt;Row&gt; dff = df
+            .selectExpr("CAST(key AS STRING)", "CAST(value AS STRING)").as("data");
+
+    dff
+            .writeStream()
+            .format("kafka")
+            .option("kafka.bootstrap.servers", "localhost:9092")
+            .option("topic", "topic2")
+            .option("checkpointLocation", "/tmp/checkpoint")
+            .start();```
+
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-09-14 13:38:09
 +
+

*Thread Reply:* Not at the moment, but it is in scope. You are welcome to open an issue with your example to track this or even propose an implementation if you have the time.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Oleksandr Dvornik + (oleksandr.dvornik@getindata.com) +
+
2021-09-14 15:12:01
 +
+

*Thread Reply:* @Tomas Satka it would be great, if you can add an containerized integration test for kafka with your test case. You can take this as an example here

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/src/test/java/io/openlineage/spark/agent/SparkContainerIntegrationTest.java | SparkContainerIntegrationTest.java>
+
+ <p>``` +package io.openlineage.spark.agent;</p> + +<p>import static java.nio.file.Files.readAllBytes; +import static org.mockserver.model.HttpRequest.request; +import static org.mockserver.model.JsonBody.json;</p> + +<p>import io.openlineage.spark.agent.client.OpenLineageClient; +import java.io.IOException; +import java.nio.file.Path; +import java.nio.file.Paths; +import java.time.Duration; +import java.time.temporal.ChronoUnit; +import org.junit.jupiter.api.AfterAll; +import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.BeforeAll; +import org.junit.jupiter.api.Tag; +import org.junit.jupiter.api.Test; +import org.mockserver.client.MockServerClient; +import org.mockserver.matchers.MatchType; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import org.testcontainers.containers.GenericContainer; +import org.testcontainers.containers.MockServerContainer; +import org.testcontainers.containers.Network; +import org.testcontainers.containers.wait.strategy.Wait; +import org.testcontainers.junit.jupiter.Container; +import org.testcontainers.junit.jupiter.Testcontainers; +import org.testcontainers.utility.DockerImageName;</p> + +<p>@Tag("integration-test") +@Testcontainers +public class SparkContainerIntegrationTest {</p> + +<p>private static final Network network = Network.newNetwork();</p> + +<p>@Container + private static final MockServerContainer openLineageClientMockContainer = + makeMockServerContainer();</p> + +<p>private static GenericContainer&lt;?&gt; pyspark; + private static MockServerClient mockServerClient;</p> + +<p>@BeforeAll + public static void setup() { + mockServerClient = + new MockServerClient( + openLineageClientMockContainer.getHost(), + openLineageClientMockContainer.getServerPort()); + mockServerClient + .when(request("/api/v1/lineage")) + .respond(org.mockserver.model.HttpResponse.response().withStatusCode(201)); + }</p> + +<p>@AfterEach + public void cleanupSpark() { + pyspark.stop(); + }</p> + +<p>@AfterAll + public static void tearDown() { + Logger logger = LoggerFactory.getLogger(SparkContainerIntegrationTest.class); + try { + openLineageClientMockContainer.stop(); + } catch (Exception e2) { + logger.error("Unable to shut down openlineage client container", e2); + } + try { + pyspark.stop(); + } catch (Exception e2) { + logger.error("Unable to shut down pyspark container", e2); + } + network.close(); + }</p> + +<p>private static MockServerContainer makeMockServerContainer() { + return new MockServerContainer( + DockerImageName.parse("jamesdbloom/mockserver:mockserver-5.11.2")) + .withNetwork(network) + .withNetworkAliases("openlineageclient"); + }</p> + +<p>private static GenericContainer&lt;?&gt; makePysparkContainer(String... command) { + return new GenericContainer&lt;&gt;( + DockerImageName.parse("godatadriven/pyspark:" + System.getProperty("spark.version"))) + .withNetwork(network) + .withNetworkAliases("spark") + .withFileSystemBind("src/test/resources/test<em>data", "/test</em>data") + .withFileSystemBind("src/test/resources/spark<em>scripts", "/opt/spark</em>scripts") + .withFileSystemBind("build/libs", "/opt/libs") + .withLogConsumer(SparkContainerIntegrationTest::consumeOutput) + .withStartupTimeout(Duration.of(2, ChronoUnit.MINUTES)) + .dependsOn(openLineageClientMockContainer) + .withReuse(true) + .withCommand(command); + }</p> + +<p>private static void consumeOutput(org.testcontainers.containers.output.OutputFrame of) { + try { + switch (of.getType()) { + case STDOUT: + System.out.write(of.getBytes()); + break; + case STDERR: + System.err.write(of.getBytes()); + break; + case END: + System.out.println(of.getUtf8String()); + break; + } + } catch (IOException ioe) { + throw new RuntimeException(ioe); + } + }</p> + +<p>@Test + public void testPysparkWordCountWithCliArgs() throws IOException, InterruptedException { + pyspark = + makePysparkContainer( + "--master", + "local", + "--conf", + "spark.openlineage.url=" + + "<a href="http://openlineageclient:1080/api/v1/namespaces/testPysparkWordCountWithCliArgs">http://openlineageclient:1080/api/v1/namespaces/testPysparkWordCountWithCliArgs</a>", + "--conf", + "spark.extraListeners=" + OpenLineageSparkListener.class.getName(), + "--jars", + "/opt/libs/" + System.getProperty("openlineage.spark.jar"), + "/opt/spark<em>scripts/spark</em>word_count.py"); + pyspark.setWaitStrategy(Wait.forLogMessage(".<strong>ShutdownHookManager: Shutdown hook called.</strong>", 1)); + pyspark.start();</p> + +<pre><code>Path eventFolder = Paths.get("integrations/container/"); +String startEvent = + new String(readAllBytes(eventFolder.resolve("pysparkWordCountWithCliArgsStartEvent.json"))) + .replaceAll( + "&lt;https://github.com/OpenLineage/OpenLineage/tree/\\$VERSION/integration/spark&gt;", + OpenLineageClient.OPEN_LINEAGE_CLIENT_URI.toString()); + +String completeEvent = + new String( + readAllBytes(eventFolder.resolve("pysparkWordCountWithCliArgsCompleteEvent.json"))) + .replaceAll( + "&lt;https://github.com/OpenLineage/OpenLineage/tree/\\$VERSION/integration/spark&gt;", + OpenLineageClient.OPEN_LINEAGE_CLIENT_URI.toString()); +mockServerClient.verify( + request() + .withPath("/api/v1/lineage") + .withBody(json(startEvent, MatchType.ONLY_MATCHING_FIELDS)), + request() + .withPath("/api/v1/lineage") + .withBody(json(completeEvent, MatchType.ONLY_MATCHING_FIELDS))); +</code></pre> + +<p>}</p> + +<p>@Test + public void testPysparkRddToTable() throws IOException, InterruptedException { + pyspark = + makePysparkContainer( + "--master", + "local", + "--conf", + "spark.openlineage.host=" + "<a href="http://openlineageclient:1080">http://openlineageclient:1080</a>", + "--conf", + "spark.openlineage.namespace=testPysparkRddToTable", + "--conf", + "spark.extraListeners=" + OpenLineageSparkListener.class.getName(), + "--jars", + "/opt/libs/" + System.getProperty("openlineage.spark.jar"), + "/opt/spark<em>scripts/spark</em>rdd<em>to</em>table.py"); + pyspark.setWaitStrategy(Wait.forLogMessage(".<strong>ShutdownHookManager: Shutdown hook called.</strong>", 1)); + pyspark.start();</p> + +<pre><code>Path eventFolder = Paths.get("integrations/container/"); +String startCsvEvent = + new String(readAllBytes(eventFolder.resolve("pysparkRddToCsvStartEvent.json"))) + .replaceAll( + "&lt;https://github.com/OpenLineage/OpenLineage/tree/\\$VERSION/integration/spark&gt;", + OpenLineageClient.OPEN_LINEAGE_CLIENT_URI.toString()); +String completeCsvEvent = + new String(readAllBytes(eventFolder.resolve("pysparkRddToCsvCompleteEvent.json"))) + .replaceAll( + "&lt;https://github.com/OpenLineage/OpenLineage/tree/\\$VERSION/integration/spark&gt;", + OpenLineageClient.OPEN_LINEAGE_CLIENT_URI.toString()); + +String startTableEvent = + new String(readAllBytes(eventFolder.resolve("pysparkRddToTableStartEvent.json"))) + .replaceAll( + "&lt;https://github.com/OpenLineage/OpenLineage/tree/\\$VERSION/integration/spark&gt;", + OpenLineageClient.OPEN_LINEAGE_CLIENT_URI.toString()); + +String completeTableEvent = + new String(readAllBytes(eventFolder.resolve("pysparkRddToTableCompleteEvent.json"))) + .replaceAll( + "&lt;https://github.com/OpenLineage/OpenLineage/tree/\\$VERSION/integration/spark&gt;", + OpenLineageClient.OPEN_LINEAGE_CLIENT_URI.toString()); + +mockServerClient.verify( + request() + .withPath("/api/v1/lineage") + .withBody(json(startCsvEvent, MatchType.ONLY_MATCHING_FIELDS)), + request() + .withPath("/api/v1/lineage") + .withBody(json(completeCsvEvent, MatchType.ONLY_MATCHING_FIELDS)), + request() + … +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tomas Satka + (satka.tomas@gmail.com) +
+
2021-09-14 18:02:05
 +
+

*Thread Reply:* Hi @Oleksandr Dvornik i wrote a test for simple read/write from kafka topic using kafka testcontainer. However i discovered a bug. When writing to kafka topic getting java.lang.IllegalArgumentException: One of the following options must be specified for Kafka source: subscribe, subscribepattern, assign. See the docs for more details.

+ +

• How would you like me to add the test? Fork openlineage and create PR

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tomas Satka + (satka.tomas@gmail.com) +
+
2021-09-14 18:02:50
 +
+

*Thread Reply:* • Shall i raise bug for writing to kafka that should have only "topic" instead of "subscribe"

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tomas Satka + (satka.tomas@gmail.com) +
+
2021-09-14 18:03:42
 +
+

*Thread Reply:* • Since i dont know expected payload to openlineage mock server can somebody help me to create it?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Oleksandr Dvornik + (oleksandr.dvornik@getindata.com) +
+
2021-09-14 19:06:41
 +
+

*Thread Reply:* Hi @Tomas Satka, yes you should create a fork and raise a PR from that. For more details, please take a look at. Not sure about kafka, cause we don't have that integration yet. About expected payload, as a first step, I would suggest to leave that test without assertion for now. Second step would be investigation (what we can get from that plan node). Third step - implementation and asserting a payload. Basically we parse spark optimized plan, and get as much information as we can for specific implementation. You can take a look at recent PR for HIVE. We visit root node and leaves to get output datasets and input datasets accordingly.

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/pull/264|#264 [INTEGRATION[spark] add visitors for Hive>
+
+ <p>Implement visitors for <code>InsertIntoHiveTable</code> and <code>InsertIntoHiveDirCommand</code>.</p> + +<p>Solves <a href="https://github.com/OpenLineage/OpenLineage/issues/187">OpenLineage/OpenLineage#187</a>.</p> + +<p>Signed-off-by: Maciej Obuchowski <a href="mailto:maciej.obuchowski@getindata.com">maciej.obuchowski@getindata.com</a></p> +
+ +
+
Comments
+ 1 +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tomas Satka + (satka.tomas@gmail.com) +
+
2021-09-15 04:37:59
 +
+

*Thread Reply:* Hi @Oleksandr Dvornik PR for step one : https://github.com/OpenLineage/OpenLineage/pull/279

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/pull/279|#279 adding initial test for spark kafka integration>
+
+ <p>Added integration test using kafka testcontainer that is setup on same network. Intention is only to spin up kafka in independent container, run spark job to read from topic1 and write to topic2.</p> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+ 👍 Oleksandr Dvornik +
+ +
+ 🙌 Julien Le Dem +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Luke Smith + (luke.smith@kinandcarta.com) +
+
2021-09-14 15:52:41
 +
+

There may not be an answer to these questions yet, but I'm curious about the plan for Tableau lineage.

+ +

• How will this integration be packaged and attached to Tableau instances? + ◦ via Extensions API, REST API? +• What is the architecture? +https://github.com/OpenLineage/OpenLineage/issues/78

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/issues/78|#78 [INTEGRATION] Add support for Tableau lineage>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Thomas Fredriksen + (thomafred90@gmail.com) +
+
2021-09-15 01:58:37
 +
+

Hi everyone - Following up on my previous post on prefect. The technical integration does not seem very difficult, but I am wondering about how to structure the lineage logic. +Is it the case that each prefect task should be mapped to a lineage job? If so, how do we connect the jobs together? Does there have to be a dataset between each job? +I am OpenLineage with Marquez by the way

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-09-15 09:19:23
 +
+

*Thread Reply:* Hey Thomas!

+ +

Following what we do with Airflow, yes, I think that each task should be mapped to job.

+ +

You don't need datasets between each tasks. It's necessary only where you consume and produce datasets - and it does not matter where in uour job graph you've produced them.

+ +

To map tasks togther In Airflow, we use ParentRunFacet , and the same approach could be used here. In Prefect, I think using flow_run_id would work.

+ + + +
+ 👍 Julien Le Dem +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Thomas Fredriksen + (thomafred90@gmail.com) +
+
2021-09-15 09:26:21
 +
+

*Thread Reply:* this is very helpful, thank you

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Thomas Fredriksen + (thomafred90@gmail.com) +
+
2021-09-15 09:26:43
 +
+

*Thread Reply:* what would be the namespace used in the Job -definition of each task?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-09-15 09:31:34
 +
+

*Thread Reply:* In contrast to dataset namespaces - which we try to standardize, job namespaces should be provided by user, or operator of particular scheduler.

+ +

For example, it would be good if it helped you identify Prefect instance where the job was run.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-09-15 09:32:23
 +
+

*Thread Reply:* If you use openlineage-python client, you can provide namespace either in client constuctor, or via OPENLINEAGE_NAMESPACE env variable.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Thomas Fredriksen + (thomafred90@gmail.com) +
+
2021-09-15 09:32:55
 +
+

*Thread Reply:* awesome, thank you 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Brad + (bradley.mcelroy@live.com) +
+
2021-09-15 17:03:07
 +
+

*Thread Reply:* Hey @Thomas Fredriksen - just chiming in, I’m also keen for a prefect integration. Let me know if I can help out at all

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-09-15 17:27:20
 +
+

*Thread Reply:* Please chime in on https://github.com/OpenLineage/OpenLineage/issues/81

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/issues/81|#81 [INTEGRATION] Add Prefect support>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Brad + (bradley.mcelroy@live.com) +
+
2021-09-15 18:29:20
 +
+

*Thread Reply:* Done!

+ + + +
+ ❤️ Julien Le Dem +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Brad + (bradley.mcelroy@live.com) +
+
2021-09-16 00:06:41
 +
+

*Thread Reply:* For now I'm prototyping in a separate repo https://github.com/limx0/caching_flow_runner/tree/open_lineage

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Thomas Fredriksen + (thomafred90@gmail.com) +
+
2021-09-17 01:55:08
 +
+

*Thread Reply:* I really like your PR, @Brad. I think that using FlowRunner and TaskRunner may be a more "proper" way of doing this, as opposed as adding a state-handler to each task the way I do it.

+ +

How are you dealing with Prefect-library tasks such as the included BigQuery-tasks and such? Is it necessary to create DatasetTask for them to show up in the lineage graph?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Brad + (bradley.mcelroy@live.com) +
+
2021-09-17 02:04:19
 +
+

*Thread Reply:* Hey @Thomas Fredriksen! At the moment I'm not dealing with any task-specific things. The plan (in my head, and after speaking with another prefect user @davzucky) would be that we add a LineageTask subclass where you could define custom facets on a per task basis

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Brad + (bradley.mcelroy@live.com) +
+
2021-09-17 02:05:21
 +
+

*Thread Reply:* or some sort of other hook where basically you would define some lineage attribute or put something in the prefect.context that the TaskRunner would find and attach

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Brad + (bradley.mcelroy@live.com) +
+
2021-09-17 02:06:23
 +
+

*Thread Reply:* Sorry I misread your question - any tasks should be automatically tracked (I believe but have not tested yet!)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Thomas Fredriksen + (thomafred90@gmail.com) +
+
2021-09-17 02:16:02
 +
+

*Thread Reply:* @Brad Could you elaborate a bit on your ideas around adding custom context attributes?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Brad + (bradley.mcelroy@live.com) +
+
2021-09-17 02:21:57
 +
+

*Thread Reply:* yeah so basically we just need some hooks that you can easily access from the task decorator or somewhere else that we can pass through to the open lineage adapter to do things like custom facets

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Brad + (bradley.mcelroy@live.com) +
+
2021-09-17 02:24:31
 +
+

*Thread Reply:* like for your bigquery example - you might want to record some facets like in https://github.com/OpenLineage/OpenLineage/blob/main/integration/common/openlineage/common/provider/bigquery.py and we need a way to do that with the Prefect bigquery task

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Brad + (bradley.mcelroy@live.com) +
+
2021-09-17 02:28:28
 +
+

*Thread Reply:* @davzucky

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Thomas Fredriksen + (thomafred90@gmail.com) +
+
2021-09-17 02:29:12
 +
+

*Thread Reply:* I see. Is this supported by the airflow-integration?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Brad + (bradley.mcelroy@live.com) +
+
2021-09-17 02:29:32
 +
+

*Thread Reply:* I think so, yes

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Brad + (bradley.mcelroy@live.com) +
+
2021-09-17 02:30:51
 +
+

*Thread Reply:* The airflow code is here https://github.com/OpenLineage/OpenLineage/blob/main/integration/airflow/openlineage/airflow/extractors/bigquery_extractor.py

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Brad + (bradley.mcelroy@live.com) +
+
2021-09-17 02:31:54
 +
+

*Thread Reply:* (I don't actually use airflow or bigquery - but for my own use case I can see wanting to do thing like this)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Thomas Fredriksen + (thomafred90@gmail.com) +
+
2021-09-17 03:18:27
 +
+

*Thread Reply:* Interesting, I like how dynamic this is

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Chris Baynes + (chris@contiamo.com) +
+
2021-09-15 09:09:21
 +
+

HI all, I have a clarification question about dataset namespaces. What's the difference between a dataset namespace (in the input/output) and a dataSource name (in the dataSource facet)? +The dbt integration appears to set those to the same value (e.g. <snowflake://myprofile>), however it seems that Marquez assumes the dataset namespace to be a more generic concept (similar to a nice user provided name like the job namespace).

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-09-15 09:29:25
 +
+

*Thread Reply:* Hey. +Generally, dataSource name should be namespace of particular dataset.

+ +

In some cases, like Postgres, dataSource facet is used to provide additionally connection strings, with info like particular host and port that we're connected to.

+ +

In case of Snowflake - or Bigquery, or S3, or multiple systems where we have only "global" instance, so the dataSource facet does not carry any other additional information.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Chris Baynes + (chris@contiamo.com) +
+
2021-09-15 10:11:19
 +
+

*Thread Reply:* Thanks. So then perhaps marquez could differentiate a bit more between job & dataset namespaces. Right now it doesn't quite feel right to have a single global list of namespaces for jobs & datasets, especially as they also have a separate concept of sources (which are not in a namespace).

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-09-15 10:18:59
 +
+

*Thread Reply:* @Willy Lulciuc what do you think?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Chris Baynes + (chris@contiamo.com) +
+
2021-09-15 10:41:20
 +
+

*Thread Reply:* As an example, in marquez I have this list of namespaces (from some sample data): dbt-sales, default, <snowflake://my-account1>, <snowflake://my-account2>. +I think the new marquez UI with the nice namespace dropdown and job/dataset search is awesome, and I'd expect to be able to filter by job namespace everywhere, but how about being able to filter datasets by source (which would be populated by the OL dataset namespace) and not persist dataset namespaces in the global namespace table?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-09-15 18:38:03
 +
+

The dbt integration (https://github.com/OpenLineage/OpenLineage/tree/main/integration/dbt) is pretty awesome but there are still a few improvements we could make. +Here are a few thoughts. +• In dbt-ol if the configuration is wrong or missing we will fail silently. This one seems like a good first thing to fix by logging the error to stdout +• We need to wait until the end to know if it worked at all. It would be nice if we checked the config at the beginning and display an error right away. Possibly by adding a parent job/run with a start event at the beginning and an end event at the end when all is done. +• While we are sending events at the end the console will hang until it’s done. It’s not clear that progress is made. We could have a simple progress bar by printing a dot for every event sent. (ex: sending 10 OpenLineage events: .........) +• We could also write at the beginning that the OL events will be sent at the end so that the user knows what to expect. +What do you think? (@Maciej Obuchowski in particular, but anyone using dbt in general)

+ + + +
+ 👀 Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-09-15 18:43:18
 +
+

*Thread Reply:* Last point is that we should persist the configuration and not just have it in environment variables. What is the best way to do this in dbt?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-09-15 18:49:21
 +
+

*Thread Reply:* We could have something similar to https://docs.getdbt.com/dbt-cli/configure-your-profile - or even put our config in there

+ + + +
+ ❤️ Julien Le Dem +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-09-15 18:51:42
 +
+

*Thread Reply:* I think we should assume that variables/config should be set and valid - and fail the run if they aren't. After all, if someone wouldn't need lineage events, they wouldn't use our wrapper.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-09-15 18:56:36
 +
+

*Thread Reply:* 3rd point would be easy to address if we could send events async/in parallel. But there could be dataset version dependencies, and we don't want to get into needless complexity of recognizing that, building a dag etc.

+ +

We could batch events if the network roundtrips are responsible for majority of the slowdown. However, we can't assume any particular environment.

+ +

Maybe just notifying about the progress is the best thing we can do right now.

+ + + +
+ 👀 Mario Measic +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-09-15 18:58:22
 +
+

*Thread Reply:* About second point, I want to add recognizing if we already have a parent run - for example, if running via airflow. If not, creating run for this purpose is a good idea.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-09-15 21:31:35
 +
+

*Thread Reply:* @Maciej Obuchowski can you open github issues to propose those changes?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-09-16 09:11:31
 +
+

*Thread Reply:* Done

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2021-09-16 12:05:10
 +
+

*Thread Reply:* FWIW, I have been putting my config in ~/.openlineage/config so it can be mapped into a container

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-09-16 17:56:23
 +
+

*Thread Reply:* Makes sense, also, all clients could use that config

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mario Measic + (mario.measic.gavran@gmail.com) +
+
2021-10-18 04:47:08
 +
+

*Thread Reply:* if dbt could actually stream the events, that would be great.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-10-18 09:59:12
 +
+

*Thread Reply:* Unfortunately, this seems very unlikely for now, due to the fact that we rely on metadata files that dbt only produces after end of execution.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-09-15 22:52:09
 +
+

The split of facets in their own schemas is ready to be merged: https://github.com/OpenLineage/OpenLineage/pull/118

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/pull/118|#118 separate facets to allow versioning them independently>
+
+ <p>This PR includes:</p> + +<p>• Splitting facet outside of the core spec into their own file. + • I wrote a separate automated program to generate each facet file from the original so you don't need to check that they have all the fields. + • You might to review what has been removed from OpenLineage.json and the general structure of each facet +• Updating code gen to deal with that + • Code generation merges all the definitions to be compatible with the existing logic. + • Adding more precise list of BaseFacets (for each entity type). Which in code gen allows for type checking adding the right type of facet in the right place + • It also adds a default implementation to the BaseFacets to allow deserializing custom facets.</p> + +<p>Signed-off-by: Julien Le Dem <a href="mailto:julien@apache.org">julien@apache.org</a></p> +
+ +
+
Comments
+ 1 +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Brad + (bradley.mcelroy@live.com) +
+
2021-09-16 00:12:02
 +
+

Hey @Julien Le Dem I'm going to start a thread here for any issues I run into trying to build a prefect integration

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Brad + (bradley.mcelroy@live.com) +
+
2021-09-16 00:16:44
 +
+

*Thread Reply:* This might be useful to others https://github.com/OpenLineage/OpenLineage/pull/284

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Brad + (bradley.mcelroy@live.com) +
+
2021-09-16 00:18:44
 +
+

*Thread Reply:* So I'm trying to push a simple event to marquez, but getting the following response: +'{"code":400,"message":"Unable to process JSON"}' +The JSON I'm pushing:

+ +

{ + "eventTime": "2021-09-16T04:00:28.343702", + "eventType": "START", + "inputs": {}, + "job": { + "facets": {}, + "name": "prefect.core.parameter.p", + "namespace": "default" + }, + "producer": "<https://github.com/OpenLineage/OpenLineage/tree/0.0.0/integration/prefect>", + "run": { + "facets": {}, + "runId": "3bce33cb-9495-4c58-b326-6aac71634ace" + } +} +Does anything look obviously wrong here?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
marko + (marko.kristian.helin@gmail.com) +
+
2021-09-16 02:41:11
 +
+

*Thread Reply:* What I did previously when debugging something like this was to remove half of the payload until I found the culprit. Binary search essentially. I was running Marquez locally, so probably could’ve enabled better logging as well. +Aren’t inputs and facets arrays?

+ + + +
+ 👍 Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Brad + (bradley.mcelroy@live.com) +
+
2021-09-16 03:14:54
 +
+

*Thread Reply:* Thanks for the response @marko - this is a greatly reduced payload already (but I'll keep going). Yep they are supposed to be arrays (I've since fixed that)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Brad + (bradley.mcelroy@live.com) +
+
2021-09-16 03:46:01
 +
+

*Thread Reply:* okay it was my timestamp 🥲

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Brad + (bradley.mcelroy@live.com) +
+
2021-09-16 19:07:16
 +
+

*Thread Reply:* Okay - I've got a simply working example now https://github.com/limx0/caching_flow_runner/blob/open_lineage/caching_flow_runner/task_runner.py

+
+ + + + +
<https://github.com/limx0/caching_flow_runner/blob/open_lineage/caching_flow_runner/task_runner.py | task_runner.py>
+
+ <pre><code>import datetime +from typing import Dict, List + +import prefect +from openlineage.client.facet import DataSourceDatasetFacet +from openlineage.client.run import InputDataset +from openlineage.client.run import OutputDataset +from prefect import Parameter +from prefect.core import Edge +from prefect.engine import TaskRunner +from prefect.engine.result import Result +from prefect.engine.state import Failed +from prefect.engine.state import Pending +from prefect.engine.state import Running +from prefect.engine.state import State +from prefect.engine.state import Success + +from caching_flow_runner.adapter import OpenLineageAdapter +from caching_flow_runner.util import task_qualified_name + + +def utc_now() -&amp;gt; str: + """Create an openlineage compatible timestamp string""" + return datetime.datetime.now(datetime.timezone.utc).isoformat()[:-6] + "Z" + + +def flow_namespace(): + """Create a namespace from a flow (and optionally project) name""" + project = prefect.context.get("project_name") + prefix = f"{project}." if project else "" + return f"{prefix}{prefect.context.flow_name}" + + +def result_location(result: Result, ****raw_inputs) -&amp;gt; str: + """Determine results location using the same formatting kwargs prefect does in `task_runner.get_task_run_state`""" + formatting_kwargs = { + ****prefect.context.get("parameters", {}).copy(), + ****prefect.context, + ****raw_inputs, + } + clone = result.copy().format(****formatting_kwargs) + return clone.location + + +class OpenLineageTaskRunner(TaskRunner): + def __init__(self, **args, client: OpenLineageAdapter, ****kwargs): + super().__init__(**args, ****kwargs) + self._client = client + self.task_full_name = task_qualified_name(task=self.task) + self.state_handlers.append(self.on_state_changed) + self.task_inputs = None + self.inputs_to_tasks = {} + + def on_state_changed(self, _, old_state: State, new_state: State): + if isinstance(old_state, Running) and isinstance(new_state, Success): + self._on_success(new_state) + elif isinstance(old_state, (Pending, Running)) and isinstance(new_state, Failed): + self._on_failure(state=new_state) + + def _prefect_inputs_to_input_dataset(self, inputs: Dict[str, Result]) -&amp;gt; List[InputDataset]: + """Convert prefect inputs to input Datasets for OpenLineage""" + if not inputs: + return [] + return [ + InputDataset( + namespace=flow_namespace(), + name=self.inputs_to_tasks[k], + facets={}, + inputFacets={}, + ) + for k, v in inputs.items() + ] + + def prefect_result_to_output_dataset(self, result: Result) -&amp;gt; OutputDataset: + output_facets = {} + if not isinstance(self.task, Parameter): + output_facets["output-dataset"] = DataSourceDatasetFacet( + name=f"{self.task_full_name}-output", + uri=result_location(result, ****self.task_inputs), + ) + return OutputDataset( + namespace=flow_namespace(), + name=f"{self.task_full_name}", + facets={}, + outputFacets=output_facets, + ) + + def _task_description(self): + if isinstance(self.task, Parameter): + # Parameters don't have any doc / description at this stage, simply return the name + return self.task.name + else: + return self.task.__doc__ + + def _on_start(self, inputs: Dict[str, Result]): + context = prefect.context + run_id = self._client.start_task( + run_id=context.task_run_id, + job_name=self.task_full_name, + job_description=self._task_description(), + event_time=utc_now(), + parent_run_id=context.flow_run_id, + code_location=None, + inputs=self._prefect_inputs_to_input_dataset(inputs), + outputs=None, + ) + &lt;a href="http://self.logger.info"&gt;self.logger.info&lt;/a&gt;(f"Marquez run CREATED run_id: {run_id}") + + def _on_success(self, state: Success): + context = prefect.context + run_id = self._client.complete_task( + run_id=context.task_run_id, + job_name=self.task_full_name, + inputs=None, + end_time=utc_now(), + outputs=[self.prefect_result_to_output_dataset(result=state._result)], + ) + &lt;a href="http://self.logger.info"&gt;self.logger.info&lt;/a&gt;(f"Marquez run COMPLETE run_id: {run_id}") + + def _on_failure(self, state: Failed): + context = prefect.context + run_id = self._client.fail_task( + run_id=context.task_run_id, + job_name=self.task_full_name, + inputs=None, + outputs=None, + end_time=utc_now(), + ) + &lt;a href="http://self.logger.info"&gt;self.logger.info&lt;/a&gt;(f"Marquez run FAILED run_id: {run_id}") + + def get_task_inputs( + self, state: State, upstream_states: Dict[Edge, State] + ) -&amp;gt; Dict[str, Result]: + for upstream in upstream_states: + self.inputs_to_tasks[upstream.key] = task_qualified_name(task=upstream.upstream_task) + task_inputs = super().get_task_inputs(state=state, upstream_states=upstream_states) + return task_inputs + + def set_task_to_running(self, state: State, inputs: Dict[str, Result]) -&amp;gt; State: + self.task_inputs = inputs + state = super().set_task_to_running(state=state, inputs=inputs) + self._on_start(inputs=inputs) + return state + +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/limx0/caching_flow_runner">limx0/caching_flow_runner</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Brad + (bradley.mcelroy@live.com) +
+
2021-09-16 19:07:37
 +
+

*Thread Reply:* I might move this into a proper PR @Julien Le Dem

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Brad + (bradley.mcelroy@live.com) +
+
2021-09-16 19:08:12
 +
+

*Thread Reply:* Successfully got a basic prefect flow working

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Brad + (bradley.mcelroy@live.com) +
+
2021-09-16 02:11:53
 +
+

A question about DatasetType - is there a representation for a file-like type? For files stored in S3/FTP/NFS etc (assuming a fully resolvable url)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-09-16 09:53:24
 +
+

*Thread Reply:* I think there was some talk somewhere to actually drop the DatasetType concept; can't find where though.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-09-16 10:04:09
 +
+

*Thread Reply:* I've taken a look at your repo. Looks great so far!

+ +

One thing I've noticed I don't think you need to use any stuff from Marquez to emit events. It's lineage ingestion API is deprecated - you can just use openlineage-python client. If there's something you think it's missing from it, feel free to write that here or open issue.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Brad + (bradley.mcelroy@live.com) +
+
2021-09-16 17:12:31
 +
+

*Thread Reply:* And would that be replaced by just some Input/Output notion @Maciej Obuchowski?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Brad + (bradley.mcelroy@live.com) +
+
2021-09-16 17:13:26
 +
+

*Thread Reply:* Oh yeah I got a little confused by the single lineage endpoint - but I’ve realised how it all works now. I’m still using the marquez backend to view things but I’ll use the openlineage-client to talk to it

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-09-16 17:34:46
 +
+

*Thread Reply:* Yes 🙌

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tomas Satka + (satka.tomas@gmail.com) +
+
2021-09-16 06:04:30
 +
+

When trying to fix failing checks, i see integration-test-integration-airflow to fail +```#!/bin/bash -eo pipefail +if [[ GCLOUDSERVICEKEY,GOOGLEPROJECTID == "" ]]; then + echo "No required environment variables to check; moving on" +else + IFS="," read -ra PARAMS <<< "GCLOUDSERVICEKEY,GOOGLEPROJECTID"

+ +

for i in "${PARAMS[@]}"; do + if [[ -z "${!i}" ]]; then + echo "ERROR: Missing environment variable {i}" >&2

+ +
  if [[ -n "" ]]; then
+    echo "" &gt;&amp;2
+  fi
+
+  exit 1
+else
+  echo "Yes, ${i} is defined!"
+fi
+
+ +

done +fi

+ +

ERROR: Missing environment variable {i}

+ +

Exited with code exit status 1 +CircleCI received exit code 1``` +However i havent touch airflow at all.. can somebody help please?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-09-16 06:59:34
 +
+

*Thread Reply:* Hey, Airflow integration tests do not pass env variables to PRs from forks due to security reasons - everyone could create malicious PR and dump secrets

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-09-16 07:00:29
 +
+

*Thread Reply:* So, they will fail and there's nothing to do from your side 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-09-16 07:00:55
 +
+

*Thread Reply:* We probably should split those into ones that don't touch external systems, and run those for all PRs

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tomas Satka + (satka.tomas@gmail.com) +
+
2021-09-16 07:08:03
 +
+

*Thread Reply:* ah okie. good to know. +and in build-integration-spark Could not resolve all artifacts. Is that also known issue? Or something from my side that i could fix?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-09-16 07:11:12
 +
+

*Thread Reply:* Looks like gradle server problem? +&gt; Could not get resource '<https://plugins.gradle.org/m2/com/diffplug/spotless/spotless-lib/2.13.2/spotless-lib-2.13.2.module>'. + &gt; Could not GET '<https://plugins.gradle.org/m2/com/diffplug/spotless/spotless-lib/2.13.2/spotless-lib-2.13.2.module>'. Received status code 500 from server: Internal Server Error

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-09-16 07:34:44
 +
+

*Thread Reply:* After retry, there's spotless error:

+ +

+········.orElse(Collections.emptyList()).stream()

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-09-16 07:35:15
 +
+

*Thread Reply:* I think this is due to mismatch between behavior of spotless in Java 8 and Java 11+ - which you probably used 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tomas Satka + (satka.tomas@gmail.com) +
+
2021-09-16 07:40:01
 +
+

*Thread Reply:* ah.. i used java11. so shall i rerun something with java8 setup as sdk?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-09-16 07:44:31
 +
+

*Thread Reply:* For spotless, you can just fix this one line 🙂 +Though I don't guarantee that tests that run later will pass, so you might need Java 8 for later testing

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tomas Satka + (satka.tomas@gmail.com) +
+
2021-09-16 08:04:36
 +
+

*Thread Reply:* yup looks better now

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tomas Satka + (satka.tomas@gmail.com) +
+
2021-09-16 08:04:41
 +
+

*Thread Reply:* thanks

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tomas Satka + (satka.tomas@gmail.com) +
+
2021-09-16 14:27:02
 +
+

*Thread Reply:* will somebody please review my PR? had to already adjust due to updates on same test class 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Brad + (bradley.mcelroy@live.com) +
+
2021-09-16 20:36:28
 +
+

Hey team - I've opened https://github.com/OpenLineage/OpenLineage/pull/293 for a very WIP prefect integration

+ + + +
+ 🙌 Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Brad + (bradley.mcelroy@live.com) +
+
2021-09-16 20:37:27
 +
+

*Thread Reply:* @Thomas Fredriksen would love any feedback

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Thomas Fredriksen + (thomafred90@gmail.com) +
+
2021-09-17 04:21:13
 +
+

*Thread Reply:* nicely done! As we discussed in another thread - the way you have implemented lineage using FlowRunner and TaskRunner is likely the best way to do this. Let me know if you need any help, I would love to see this PR get merged!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-09-17 07:28:33
 +
+

*Thread Reply:* Hey @Brad, it looks great!

+ +

I've seen you're using task_qualified_name to name datasets and I don't think it's the right way. +I'd take a look at naming conventions here: https://github.com/OpenLineage/OpenLineage/blob/main/spec/Naming.md

+ +

Getting that right is key to making sure that lineage is properly tracked between systems - for example, if you use Prefect to schedule dbt runs or pyspark jobs, the unified naming makes sure that all those integrations properly refer to the same dataset.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Brad + (bradley.mcelroy@live.com) +
+
2021-09-17 08:12:50
 +
+

*Thread Reply:* Hey @Maciej Obuchowski thanks for the feedback. Yep the naming was a bit of a placeholder. Open to any recommendations.. I think things like dbt or pyspark are straight forward (we could add special handling for tasks like that) but what about regular transformation type tasks that run in a scheduler? Do you have any naming preference? Say I just had some pandas transform task in prefect for example

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-09-17 08:28:04
 +
+

*Thread Reply:* First of all, not all tasks are producing and consuming datasets. For example, I wouldn't expect any of the Github tasks to have any datasets.

+ +

Second, in Airflow we have a concept of Extractor where you can write specialized code to expose datasets. For example, for BigQuery we extract datasets from query plan. Now, I'm not sure if this concept would translate well to Prefect - but if yes, then we have some helpers inside openlineage common library that could be reused. Also, this way allows to emit additional facets, some of which are really useful - like query statistics for BigQuery, and data quality tests for dbt.

+ +

Third, if we're talking about generalized tasks like FunctionTask or ShellTask, then I think the right way is to expose functionality to user to expose lineage themselves. I'm not sure how exactly that would look in Prefect.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Brad + (bradley.mcelroy@live.com) +
+
2021-09-19 23:03:14
 +
+

*Thread Reply:* You've raised some good points @Maciej Obuchowski - I might have been thinking about this integration in slightly the wrong way. I think based on your comments I'll refactor some of the code to hook into the Results object in prefect (The Result object is the way in which data is serialized and persisted).

+ +

> Now, I'm not sure if this concept would translate well to Prefect - but if yes, then we have some helpers inside openlineage common library that could be reused +This definitely applies to prefect and the similar tasks exist in prefect and we should definitely leverage the common library in this case.

+ +

> Third, if we're talking about generalized tasks like FunctionTask or ShellTask, then I think the right way is to expose functionality to user to expose lineage themselves. I'm not sure how exactly that would look in Prefect. +Yeah I agree with this. I'd like to make it as easy a possible to opt-in, but I think you're right that there needs to be some hooks for user defined lineage. I'll think about this a little more.

+ +

> First of all, not all tasks are producing and consuming datasets. For example, I wouldn't expect any of the Github tasks to have any datasets. +My initial thoughts here were that it would still be good to have lineage as these tasks do have side effects, and downstream consumers of the lineage data might want to know about these tasks. However I don't have a good feeling yet how best to do this, so I'm going to park those thoughts for now.

+ + + +
+ 🙌 Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-09-20 06:30:51
 +
+

*Thread Reply:* > Yeah I agree with this. I'd like to make it as easy a possible to opt-in, but I think you're right that there needs to be some hooks for user defined lineage. I'll think about this a little more. +First version of an integration doesn't have to be perfect. in particular, not handling this use case would be okay, since it does not lock us into some particular way of doing it later.

+ +

> My initial thoughts here were that it would still be good to have lineage as these tasks do have side effects, and downstream consumers of the lineage data might want to know about these tasks. However I don't have a good feeling yet how best to do this, so I'm going to park those thoughts for now. +I'd think of two options first, before modeling it as a dataset: +Won't existence of a event be enough? After all, we'll still have it despite it not having any input and output datasets. +If not, then wouldn't custom run or job facet be a better fit?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Brad + (bradley.mcelroy@live.com) +
+
2021-09-23 17:27:49
 +
+

*Thread Reply:* > Won’t existence of a event be enough? After all, we’ll still have it despite it not having any input and output datasets. +Duh, yep you’re right @Maciej Obuchowski, I’m over thinking this. I’m going to clean this up based on your comments

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Thomas Fredriksen + (thomafred90@gmail.com) +
+
2021-10-06 03:39:28
 +
+

*Thread Reply:* Hi @Brad. How will this integration work for Prefect flows running in Prefect Cloud or on Prefect Server?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Brad + (bradley.mcelroy@live.com) +
+
2021-10-06 03:40:44
 +
+

*Thread Reply:* Hi @Thomas Fredriksen - it'll relate to the agent actually - you'll need to pass the flow runner class to the agent when running

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Thomas Fredriksen + (thomafred90@gmail.com) +
+
2021-10-06 03:48:14
 +
+

*Thread Reply:* nice!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Brad + (bradley.mcelroy@live.com) +
+
2021-10-06 03:48:54
 +
+

*Thread Reply:* Unfortunately I've been a little busy the past week, and I will be for the rest of this week

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Brad + (bradley.mcelroy@live.com) +
+
2021-10-06 03:49:09
 +
+

*Thread Reply:* but I do plan to pick this up next week

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Brad + (bradley.mcelroy@live.com) +
+
2021-10-06 03:49:23
 +
+

*Thread Reply:* (the additional changes I mention above)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Thomas Fredriksen + (thomafred90@gmail.com) +
+
2021-10-06 03:50:08
 +
+

*Thread Reply:* looking forward to it 🙂 let me know if you need any help!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Brad + (bradley.mcelroy@live.com) +
+
2021-10-06 03:50:34
 +
+

*Thread Reply:* yeah when I get this next lot of stuff in - I'd love for people to test it out

+ + + +
+ 🙌 Thomas Fredriksen, Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Adam Pocock + (adam.pocock@oracle.com) +
+
2021-09-20 17:38:51
 +
+

Is there a preferred academic citation for OpenLineage? I’m writing a paper on the provenance system in our machine learning library, and I’d like to cite OpenLineage as an example of future work on data lineage to integrate with.

+ + + +
+ 🙌 Willy Lulciuc +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-09-20 19:18:53
 +
+

*Thread Reply:* I think you can reffer to https://openlineage.io/

+
+
openlineage.io
+ + + +
Home
+
+ Data lineage is the foundation for a new generation of powerful, context-aware data tools and best practices. OpenLineage enables consistent collection of lineage metadata, creating a deeper understanding of how data is produced and used. +
+ + + + + + +
Original URL: https://openlineage.io/
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-09-20 19:31:30
 +
+

We’re starting to see the beginning of larger contributions (Spark streaming, prefect, …) and I think we need to define a way to accept those contributions incrementally. +If we take the example of Streaming (Spark streaming, Flink or Beam) support (but really this applies in general, sorry to pick on you Tomas, this is great!): +The first Spark streaming PR ( https://github.com/OpenLineage/OpenLineage/pull/279 ) lays the ground work for testing spark streaming but there’s more work to have a full feature. +I’m in favor of merging Spark streaming support into main once it’s working end to end (possibly with partial input/output coverage). +So I see 2 options:

+ +
  1. start a branch for spark streaming support. Have PRs like this one go into it until it’s completed (smaller reviews). Then merge the whole thing as a PR in main when it’s finished
  2. Keep working on that PR until it’s fully implemented, but it will get big, and make reviews difficult. +I have seen the model 1) work well. It’s easier to do multiple smaller reviews for larger projects.
    3. +
+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/pull/279|#279 adding initial test for spark kafka integration>
+
+ <p>Added integration test using kafka testcontainer that is setup on same network. Intention is only to spin up kafka in independent container, run spark job to read from topic1 and write to topic2.</p> +
+ +
+
Comments
+ 2 +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+ 👍 Ross Turk, Maciej Obuchowski, Faouzi +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Yannick Endrion + (yannick.endrion@gmail.com) +
+
2021-09-24 05:10:04
 +
+

Thank you @Ross Turk for this really useful article: https://openlineage.io/blog/dbt-with-marquez/?s=03 +Is anyone aware of additional environment being supported by the dbt<->OpenLineage<->Marquez integration ? I think only Snowflake and BigQuery are supported now. +I am really interested by SQLServer or even Dremio (which could be great because capable of read from multiples DB).

+ +

Thank you

+
+
openlineage.io
+ + + +
Using Marquez to Visualize dbt Models
+
+ Each time dbt runs it generates a trove of metadata about datasets and the work it performs with them. In this post, I’d like to show you how to harvest this metadata and put it to good use. +
+ + + + + + +
Original URL: https://openlineage.io/blog/dbt-with-marquez/?s=03
+ + + +
+ + + +
+ 🎉 Minkyu Park, Ross Turk +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-09-24 05:15:31
 +
+

*Thread Reply:* It should be really easy to add additional databases. Basically, we'd need to know how to get namespace for that database: https://github.com/OpenLineage/OpenLineage/blob/main/integration/common/openlineage/common/provider/dbt.py#L467

+ +

The first step should be to add SQLServer or Dremio to the dataset naming schema here https://github.com/OpenLineage/OpenLineage/blob/main/spec/Naming.md

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/integration/common/openlineage/common/provider/dbt.py | dbt.py>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/spec/Naming.md | Naming.md>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Yannick Endrion + (yannick.endrion@gmail.com) +
+
2021-10-04 16:22:59
 +
+

*Thread Reply:* Thank you @Maciej Obuchowski, +I tried to give it a try but without success yet. Not sure where I am suppose to add the sqlserver naming schema... +If you have any documentation that I could read I would be glad =) +Many thanks

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-10-07 15:13:43
 +
+

*Thread Reply:* This would be adding a paragraph similar to this one: https://github.com/OpenLineage/OpenLineage/blob/main/spec/Naming.md#snowflake

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/spec/Naming.md | Naming.md>
+
+ <p><strong>Naming</strong></p> + +<p>We define the unique name strategy per resource to ensure it is followed uniformly independently from who is producing metadata and we can connect lineage from various sources.</p> + +<p>Both Jobs and Datasets are in their own namespaces. Job namespaces are related to their scheduler. The namespace for a dataset is the unique name for its datasource.</p> + +<p><strong>Datasets</strong></p> + +<p>The namespace and name of a datasource can be combined to form a URI (scheme:[//authority]path)</p> + +<p>• Namespace = scheme:[//authority] (the datasource) +• Name = path (the datasets)</p> + +<p><strong>Data warehouses/data bases</strong></p> + +<p>Datasets are called tables. Tables are organised in databases and schemas.</p> + +<p><strong>Postgres:</strong></p> + +<p>Datasource hierarchy:</p> + +<p>• Host +• Port</p> + +<p>Naming hierarchy:</p> + +<p>• Database +• Schema +• Table</p> + +<p>Identifier:</p> + +<p>• Namespace: postgres://{host}:{port} of the service instance. + • Scheme = postgres + • Authority = {host}:{port} +• Unique name: {database}.{schema}.{table} + • URI = postgres://{host}:{port}/{database}.{schema}.{table}</p> + +<p><strong>Snowflake</strong></p> + +<p>See: <a href="https://docs.snowflake.com/en/sql-reference/identifiers.html">Object Identifiers — Snowflake Documentation</a></p> + +<p>Datasource hierarchy:</p> + +<p>• account name</p> + +<p>Naming hierarchy:</p> + +<p>• Database: {database name} =&gt; unique across the account +• Schema: {schema name} =&gt; unique within the database +• Table: {table name} =&gt; unique within the schema</p> + +<p>Identifier:</p> + +<p>• Namespace: snowflake://{account name} + • Scheme = snowflake + • Authority = {account name} +• Name: {database}.{schema}.{table} + • URI = snowflake://{account name}/{database}.{schema}.{table}</p> + +<p><strong>BigQuery</strong></p> + +<p>See: <br /> +<a href="https://cloud.google.com/resource-manager/docs/creating-managing-projects|Creating and managing projects | Resource Manager Documentation">https://cloud.google.com/resource-manager/docs/creating-managing-projects|Creating and managing projects | Resource Manager Documentation</a> <br /> +<a href="https://cloud.google.com/bigquery/docs/datasets-intro|Introduction to datasets | BigQuery">https://cloud.google.com/bigquery/docs/datasets-intro|Introduction to datasets | BigQuery</a> <br /> +<a href="https://cloud.google.com/bigquery/docs/tables-intro|Introduction to tables | BigQuery">https://cloud.google.com/bigquery/docs/tables-intro|Introduction to tables | BigQuery</a></p> + +<p>Datasource hierarchy:</p> + +<p>• bigquery</p> + +<p>Naming hierarchy:</p> + +<p>• Project Name: {project name} =&gt; is not unique +• Project number: {project number} =&gt; numeric: is unique across google cloud +• Project ID: {project id} =&gt; readable: is unique across google cloud +• dataset: {dataset name} =&gt; is unique within a project +• table: {table name} =&gt; is unique within a dataset</p> + +<p>Identifier :</p> + +<p>• Namespace: bigquery + • Scheme = bigquery + • Authority = +• Unique name: {project id}.{dataset name}.{table name} + • URI = bigquery:{project id}.{schema}.{table}</p> + +<p><strong>Distributed file systems/blob stores</strong> +<strong>GCS</strong></p> + +<p>Datasource hierarchy: none, naming is global</p> + +<p>Naming hierarchy:</p> + +<p>• bucket name =&gt; globally unique +• Path</p> + +<p>Identifier :</p> + +<p>• Namespace: gs://{bucket name} + • Scheme = gs + • Authority = {bucket name} +• Unique name: {path} + • URI = gs://{bucket name}{path}</p> + +<p><strong>S3</strong></p> + +<p>Naming hierarchy:</p> + +<p>• bucket name =&gt; globally unique +• Path</p> + +<p>Identifier :</p> + +<p>• Namespace: s3://{bucket name} + • Scheme = s3 + • Authority = {bucket name} +• Unique name: {path} + • URI = s3://{bucket name}{path}</p> + +<p><strong>HDFS</strong></p> + +<p>Naming hierarchy:</p> + +<p>• Namenode: host + port +• Path</p> + +<p>Identifier :</p> + +<p>• Namespace: hdfs://{namenode host}:{namenode port} + • Scheme = hdfs + • Authority = {namenode host}:{namenode port} +• Unique name: {path} + • URI = hdfs://{namenode host}:{namenode port}{path}</p> + +<p><strong>Jobs</strong> +<strong>Context</strong></p> + +<p>A <code>Job</code> is a recurring data transformation with Inputs and outputs. Each execution is captured as a <code>Run</code> with corresponding metadata. <br /> +A <code>Run</code> event identifies the <code>Job</code> it is an instance of by providing the job’s unique identifier. <br /> +The <code>Job</code> identifier is composed of a <code>Namespace</code> and a <code>Name</code>. The job name is unique within that namespace.</p> + +<p>The core property we want to identify about a <code>Job</code> is how it changes over time. Different schedules of the same logic applied to different datasets (possibly with different parameters) are different jobs. The notion of a <code>job</code> is tied to a recurring schedule with specific inputs and outputs. It could be an incremental update or a full reprocess or even a streaming job.</p> + +<p>If the same code artifact (for example a spark jar or a templated SQL query) is used in the context of different schedules with different input or outputs then they are different jobs. <br /> +We are interested first in how they affect the datasets they produce.</p> + +<p><strong>Job Namespace and constructing job names</strong></p> + +<p>Jobs have a <code>name</code> that is unique to them in their <code>namespace</code> by construction.</p> + +<p>The Namespace is the root of the naming hierarchy. The job name is constructed to identify the job within that namespace.</p> + +<p>Example:</p> + +<p>• Airflow: + • Namespace: the namespace is assigned to the airflow instance. Ex: airflow-staging, airflow-prod + • Job: each task in a DAG is a job. name: {dag name}.{task name} +• Spark: + • Namespace: the namespace is provided as a configuration parameter as in airflow. If there's a parent job, we use the same namespace, otherwise it is provided by configuration. + • Spark app job name: the spark.app.name + • Spark action job name: {spark.app.name}.{node.name}</p> + +<p><strong>Parent job run: a nested hierarchy of Jobs</strong></p> + +<p>It is often the case that jobs are part of a nested hierarchy. <br /> +For example an Airflow DAG contains tasks. An instance of the DAG is finished when all of the tasks are finished. Similarly a Spark job can spawn multiple actions each of them running independently. Additionally, a Spark job can be launched by an Airflow task within a DAG.</p> + +<p>Since what we care about is identifying the job as rooted in a recurring schedule, we want to capture that connection and make sure that we treat the same application logic triggered at different schedules as different jobs. For example: if an Airflow DAG runs individual tasks per partition (for example market segments) using the same underlying job logic, they will be tracked as separate jobs.</p> + +<p>To capture this, a run event provides <a href="https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.json#L282-L331">a <code>ParentRun</code> facet</a>, referring to the parent <code>Job</code> and <code>Run</code>. This allows tracking a recurring job from the root of the schedule it is running for. <br /> +If there's a parent job, we use the same namespace, otherwise it is provided by configuration.</p> + +<p>Example:</p> + +<pre><code>{ + "run": { + "runId": "run_uuid" + }, + "job": { + "namespace": "job_namespace", + "name": "job_name" + } +} + +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-10-07 15:14:30
 +
+

*Thread Reply:* Snowflake +See: Object Identifiers — Snowflake Documentation +Datasource hierarchy: +• account name +Naming hierarchy: +• Database: {database name} => unique across the account +• Schema: {schema name} => unique within the database +• Table: {table name} => unique within the schema +Identifier: +• Namespace: snowflake://{account name} + ◦ Scheme = snowflake + ◦ Authority = {account name} +• Name: {database}.{schema}.{table} + ◦ URI = snowflake://{account name}/{database}.{schema}.{table}

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marty Pitt + (martypitt@vyne.co) +
+
2021-09-24 06:53:05
 +
+

Hi all. I'm the Founder / CTO of a data discovery & transformation platform that captures very rich lineage information. We're interested in exposing / making our lineage data consumable via open standards, which is what lead me to this project. A couple of questions:

+ +

A) Am I right in considering that's the goal of this project? +B) Are you also considering provedance as well as lineage? +C) What's a good starting point to understand the models we should be exposing our data in, to make it consumable?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marty Pitt + (martypitt@vyne.co) +
+
2021-09-24 07:06:20
 +
+

*Thread Reply:* For clarity on the provedance vs lineage point (in case I'm using those terms incorrectly...)

+ +

Our platform performs automated enrichment and processing of data. In doing so, we often make calls to functions or out to other data services (such as APIs, or SELECTs against databases). We capture the inputs that pass to these, along with the outputs. (And, if the input is derived from other outputs, we capture the full chain, right back to the root).

+ +

That's the kinda stuff our customers are really interested in, and we feel like there's value in making is consumable.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-09-24 08:47:35
 +
+

*Thread Reply:* Not sure I understand you right, but are you interested in tracking individual API calls, and for example, values of some parameters passed for one call?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-09-24 08:51:16
 +
+

*Thread Reply:* I guess that's not in OpenLineage scope, as we're interested more in tracking metadata for whole datasets. But I might be wrong, some other people might chime in.

+ +

We could of course model this situation, but that would capture for example schema of those parameters. Not their values.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-09-24 08:52:16
 +
+

*Thread Reply:* I think this might be better suited for https://opentelemetry.io/

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marty Pitt + (martypitt@vyne.co) +
+
2021-09-24 10:55:54
 +
+

*Thread Reply:* Kinda, but not really. Telemetery data is metadata about the API calls. We have that, but it's not interesting to our customers. It's the metadata about the data that Vyne provides that we want to expose.

+ +

Our customers use Vyne to fetch data from lots of different sources. Eg:

+ +

> "Whenever a trade is booked, calculate it's compliance against these regulations, to report to the regulators". +or

+ +

> "Whenever a customer buys a $thing, capture the transaction data, client data, and account data, and store it in this table." +Providing answers to those questions involves fetching and transforming data, before storing it, or outputting it. We capture all that data, on a per-attribute basis, so we can answer the question "how did we get this value?" That's the lineage information we want to publish.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2021-09-30 15:10:51
 +
+

*Thread Reply:* The core OpenLineage model is documented at https://github.com/OpenLineage/OpenLineage/#core-model . The model is really focused on Jobs and Datasets. Jobs have Runs which have start and end times (typically scheduled start/end times as well) and read from and/or write to the target datasets. If your transformation chain fits within that model, then I think you can definitely record and share the lineage information with your customers. The existing implementations are all focused on batch data access, though streaming should be possible to capture as well

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Drew Bittenbender + (drew@salt.io) +
+
2021-09-29 11:10:29
 +
+

Hello. I am trying the openlineage-airflow integration with Marquez as the backend and have 3 questions.

+ +
  1. Does it only work for PostgresOperators?
  2. Which is the recommended integration: marquez-airflow or openlineage-airflow
  3. How do you enable more detailed logging? I tried OPENLINEAGELOGLEVEL and MARQUEZLOGLEVEL and neither seemed to affect logging. I assume this is logged to the airflow worker
    4. +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Faouzi + (faouzi@dataroots.io) +
+
2021-09-29 13:46:59
 +
+

*Thread Reply:* Hello @Drew Bittenbender!

+ +

For your two first questions:

+ +

• Yes right now only the PostgresOperator is integrated. I learnt it the hard way ^_^. Spent hours trying with MySQL. There were attempts to integrate with MySQL actually. If engineers do not integrate it I will allocate myself some time to try to implement other airflow db operators. +• Use the openlineage one. It is the recommended approach now.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Drew Bittenbender + (drew@salt.io) +
+
2021-09-29 13:49:41
 +
+

*Thread Reply:* Thank you @Faouzi. Is there any documentation/best practices to write your own extractor, or is it "read the code"? We use the Python, Docker and SSH operators a lot. Maybe those don't fit into the lineage paradigm well, but want to give it a shot

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Faouzi + (faouzi@dataroots.io) +
+
2021-09-29 13:52:16
 +
+

*Thread Reply:* To the best of my knowledge there is no documentation to guide through the design of your own extractor. So yes we need to read the code. Here a link where you can see how they did for postgre extractor and others. https://github.com/OpenLineage/OpenLineage/tree/main/integration/airflow/openlineage/airflow/extractors

+ + + +
+ 👍 Drew Bittenbender +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-09-30 05:08:53
 +
+

*Thread Reply:* I think in case of "bring your own code" operators like Python or Docker ones, it might be better to use lineage_run_id macro and use openlineage-python library inside, instead of implementing extractor.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2021-09-30 15:14:47
 +
+

*Thread Reply:* I think @Maciej Obuchowski is right here. The airflow integration will create the parent jobs, but to get the dataset input/output links, it's best to do that directly from the python/docker scripts. If you report the parent run id, Marquez will link the jobs together correctly

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-10-07 15:09:55
 +
+

*Thread Reply:* To clarify on what airflow operators are supported out of the box: +• postgres +• bigquery +• snowflake +• Great expectations (with extra config) +See: https://github.com/OpenLineage/OpenLineage/blob/3a1ccbd854bbf202bbe6437bf81786cb01[…]ntegration/airflow/openlineage/airflow/extractors/extractors.py +Mysql is not at the moment. We should track it as an issue

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/3a1ccbd854bbf202bbe6437bf81786cb01ed4d01/integration/airflow/openlineage/airflow/extractors/extractors.py | extractors.py>
+
+ <pre><code>_extractors = [ + PostgresExtractor, + BigQueryExtractor, + GreatExpectationsExtractor, + SnowflakeExtractor +] +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Yuki Tannai + (tannai-yuki@dmm.com) +
+
2021-09-30 09:21:35
 +
+

Hi there! +I’m trying to enhance the lineage functionality of a data infrastructure I’m working on. +All of the tools I found only visualize the relationships between tables before and after the transformation, but the DataHub RFC discusses Field Level Lineage, which I thought was close to the functionality I was looking for. +Does OpenLineage support the same functionality? +https://datahubproject.io/docs/rfc/active/1841-lineage/field_level_lineage/

+
+
datahubproject.io
+ + + +
RFC - Field Level Lineage | DataHub
+
+ - Start Date: 2020-08-28 +
+ + + + + + +
Original URL: https://datahubproject.io/docs/rfc/active/1841-lineage/field_level_lineage/
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-10-07 15:03:40
 +
+

*Thread Reply:* OpenLineage doesn’t have field level lineage yet. Here is the proposal for adding it: https://github.com/OpenLineage/OpenLineage/issues/148

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/issues/148|#148 [PROPOSAL] column level lineage facet>
+
+ <p><strong>Purpose:</strong> <br /> +For transformations like SQL queries (but also in other cases), we can extract column level lineage. This allows answering questions like ‘which root input columns are used to construct column x?’. We need to be able to capture this information in the OpenLineage model</p> + +<p><strong>Proposed implementation</strong> <br /> +We propose to introduce a new dataset facet: <br /> +example:</p> + +<pre><code>{ + "eventType": "COMPLETE", + "eventTime": "2020-12-28T20:52:00.001+10:00", + "run" : { + "runId": "uuid" + }, + "job": { + "namespace": "scheduler", + "name": "myjob", + "facets": { + "sql": { + "query": "Insert into outputTable from select ** from inputTable" + } + } + }, + "inputs": [ + { + "namespace": "N1", + "name": "inputTable", + "facets": { + "schema": { + "fields": [ {"name": "col_a", "type": "VARCHAR"}, {"name": "col_b", "type": "int"}] + } + } + } + ], + "outputs": [ + { + "namespace": "N2", + "name": "outputTable", + "facets": { + "schema": { + "fields": [ {"name": "col_a", "type": "VARCHAR"}, {"name": "col_b", "type": "int"}] + }, + "columnLineage": { + "fields": { + "col_a": [ { "namespace": "N1", "name": "inputTable", "field": "col_a"} ], + "col_b": [ { "namespace": "N1", "name": "inputTable", "field": "col_b" } ] + } + } + } + } + ] +} +</code></pre> + +<p>Schema:</p> + +<pre><code> "columnLineage": { + "type": "object", + "properties": { + "fields": { + "type": "array", + "items": { + "type": "object", + "additionalProperties": { + "type": "object", + "properties": { + "namespace": { + "type": "string", + "description": "the input dataset namespace" + }, + "name": { + "type": "string", + "description": "the input dataset name" + }, + "field": { + "type": "string", + "description": "the input field" + } + }, + "required": [ + "namespace", "name" + ] + } + } + } + } +</code></pre> +
+ +
+
Labels
+ proposal +
+ +
+
Comments
+ 2 +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+ 👀 Yuki Tannai, Ricardo Gaspar +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-10-07 15:04:36
 +
+

*Thread Reply:* Those two specs look compatible, so Datahub should be able to consume this lineage metadata in the future

+ + + +
+ 👍 Yuki Tannai +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
павел клопотюк + (klopotuk@gmail.com) +
+
2021-10-04 14:27:24
 +
+

Hello, everyone. I'm trying to work with OL and Airflow 2.1.4 and it doesn't work. I found that OL is supported for Airflow 1.10.12++. Does it support Airflow 2.X.Y?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2021-10-04 15:38:47
 +
+

*Thread Reply:* Hi! Airflow 2.x is currently in development - you can follow along with the progress here: +https://github.com/OpenLineage/OpenLineage/issues/205

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/issues/205|#205 Airflow 2.0 support>
+
+ <p>☑︎ Test suite for 2.0 +☑︎ Working LineageBackend PoC that passes existing tests +☑︎ <a href="https://github.com/apache/airflow/issues/17984">Issue in Airflow repo to create mechanism for reporting failures</a> +☑︎ <a href="https://github.com/OpenLineage/OpenLineage/issues/242">Way to add custom Extractors</a> +☑︎ <a href="https://github.com/apache/airflow/pull/18470">Airflow PR</a> +☑︎ Check if it's feasible to have both Airflows supported in one package +☐ Implement support in existing package</p> +
+ +
+
Assignees
+ mobuchowski +
+ +
+
Comments
+ 5 +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
павел клопотюк + (klopotuk@gmail.com) +
+
2021-10-05 03:01:54
 +
+

*Thread Reply:* Thank you for your reply!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-10-07 15:02:23
 +
+

*Thread Reply:* There should be a first version of Airflow 2.X support soon: https://github.com/OpenLineage/OpenLineage/pull/305 +We’re labelling it experimental because the config step might change as discussion in the airflow github evolve. It will track succesful jobs in its current state.

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/pull/305|#305 [INTEGRATION][airflow] integration working on both Airflow 2 and 1.10>
+
+ <p>Implement support for both Airflow versions on one python package.</p> + +<p>Airflow 2's support is done via Airflow's <code>LineageBackend</code> and is currently experimental. It has limitations - can't track job fails, and job starts are only tracked on job end.</p> + +<p>The end-user has only to specify <code>openlineage.airflow.backend.OpenLineageBackend</code> as their <code>LineageBackend</code> in the Airflow config or via env variable.</p> + +<p>To make sure this integration works on both versions, some helpers had to be implemented, including ones that correctly import based on version, like <code>safe_import_airflow</code>. Also, unit and integration tests are running on both versions of Airflow now.</p> + +<p>Closes <a href="https://github.com/OpenLineage/OpenLineage/issues/205">#205</a> <a href="https://github.com/OpenLineage/OpenLineage/issues/240">#240</a> <a href="https://github.com/OpenLineage/OpenLineage/issues/126">#126</a></p> + +<p>Signed-off-by: Maciej Obuchowski <a href="mailto:maciej.obuchowski@getindata.com">maciej.obuchowski@getindata.com</a></p> +
+ +
+
Comments
+ 2 +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
SAM + (skhettri@gmail.com) +
+
2021-10-04 23:14:26
 +
+

Hi All, I’m working on openlineage-dbt integration with Marquez as backend. I want to integrate OL with DBT cloud, would you please help to provide steps that I need to follow?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-10-05 04:18:42
 +
+

*Thread Reply:* Take a look at this: https://docs.getdbt.com/docs/dbt-cloud/dbt-cloud-api/metadata/metadata-overview

+
+
docs.getdbt.com
+ + + +
Metadata API Overview | dbt Docs
+
+ Every time that dbt Cloud runs a dbt project, it generates metadata which pertains to the accuracy, recency, configuration, and structure of the views and tables in the warehouse. dbt Cloud serves a GraphQL API which supports arbitrary queries over this metadata; the endpoint for this API is <https://metadata.cloud.getdbt.com/graphql>. This API is an incredibly rich resource for evaluating data health longitudinally or at a point in time. +
+ + + + + + + + +
Original URL: https://docs.getdbt.com/docs/dbt-cloud/dbt-cloud-api/metadata/metadata-overview
+ + + +
+ + + +
+ ✅ SAM +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-10-07 14:58:24
 +
+

*Thread Reply:* @SAM Let us know of your progress.

+ + + +
+ 👍 SAM +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ale + (alessandro.lollo@gmail.com) +
+
2021-10-05 16:23:41
 +
+

Hey folks 😊 +I’m trying to run dbt-ol with Redshift target, but I get the following error message +Traceback (most recent call last): + File "/usr/local/bin/dbt-ol", line 61, in &lt;module&gt; + main() + File "/usr/local/bin/dbt-ol", line 54, in main + events = processor.parse().events() + File "/usr/local/lib/python3.8/site-packages/openlineage/common/provider/dbt.py", line 97, in parse + self.extract_dataset_namespace(profile) + File "/usr/local/lib/python3.8/site-packages/openlineage/common/provider/dbt.py", line 368, in extract_dataset_namespace + self.dataset_namespace = self.extract_namespace(profile) + File "/usr/local/lib/python3.8/site-packages/openlineage/common/provider/dbt.py", line 382, in extract_namespace + raise NotImplementedError( +NotImplementedError: Only 'snowflake' and 'bigquery' adapters are supported right now. Passed redshift +I know that Redshift is not the best cloud DWH we can use… 😅 +But, still….do you have any plan to support it? +Thanks!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-10-05 16:41:30
 +
+

*Thread Reply:* Hey, can you create ticket in OpenLineage repository? FWIW Redshift is very similar to postgres, so supporting it won't be hard.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ale + (alessandro.lollo@gmail.com) +
+
2021-10-05 16:43:39
 +
+

*Thread Reply:* Hey @Maciej Obuchowski 😊 +Yep, will do now! Thanks!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ale + (alessandro.lollo@gmail.com) +
+
2021-10-05 16:46:26
 +
+

*Thread Reply:* Well...will do tomorrow morning 😅

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ale + (alessandro.lollo@gmail.com) +
+
2021-10-06 03:03:16
 +
+

*Thread Reply:* Here’s the issue: https://github.com/OpenLineage/OpenLineage/issues/318

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/issues/318|#318 Add Redshift support to dbt integration>
+
+ <p>As of <code>dbt-ol==0.2.2</code>, it is not possible to generate lineage if the target DWH is Redshift. <br /> +Try running <code>dbt-ol</code> and you get the following error:</p> + +<pre><code>Traceback (most recent call last): + File "/usr/local/bin/dbt-ol", line 61, in &amp;lt;module&amp;gt; + main() + File "/usr/local/bin/dbt-ol", line 54, in main + events = processor.parse().events() + File "/usr/local/lib/python3.8/site-packages/openlineage/common/provider/dbt.py", line 97, in parse + self.extract_dataset_namespace(profile) + File "/usr/local/lib/python3.8/site-packages/openlineage/common/provider/dbt.py", line 368, in extract_dataset_namespace + self.dataset_namespace = self.extract_namespace(profile) + File "/usr/local/lib/python3.8/site-packages/openlineage/common/provider/dbt.py", line 382, in extract_namespace + raise NotImplementedError( +NotImplementedError: Only 'snowflake' and 'bigquery' adapters are supported right now. Passed redshift + +</code></pre> + +<p>As <a href="https://github.com/mobuchowski">@mobuchowski</a> suggested <a href="https://openlineage.slack.com/archives/C01CK9T7HKR/p1633465421127100">here</a>, I'm opening this issue as it should be not hard to support Redshift since it is very similar to Postgres.</p> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-10-07 14:51:08
 +
+

*Thread Reply:* Thanks a lot. I pulled it in the current project.

+ + + +
+ 👍 ale +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ale + (alessandro.lollo@gmail.com) +
+
2021-10-08 05:48:28
 +
+

*Thread Reply:* @Julien Le Dem @Maciej Obuchowski I’m not familiar with dbt-ol codebase, but I’m willing to help on this if you guys can give me a bit of guidance 😅

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-10-08 05:53:05
 +
+

*Thread Reply:* @ale can you help us define naming schema for redshift, as we have for other databases? https://github.com/OpenLineage/OpenLineage/blob/main/spec/Naming.md

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ale + (alessandro.lollo@gmail.com) +
+
2021-10-08 05:53:21
 +
+

*Thread Reply:* Sure!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ale + (alessandro.lollo@gmail.com) +
+
2021-10-08 05:54:21
 +
+

*Thread Reply:* will work on this today and I’ll try to submit a PR by EOD

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ale + (alessandro.lollo@gmail.com) +
+
2021-10-08 06:36:12
 +
+

*Thread Reply:* There you go https://github.com/OpenLineage/OpenLineage/pull/324

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/pull/324|#324 Add redshift naming spec>
+
+ <p>This PR adds Redshift naming specs.</p> + +<p>Given that Redshift is based on Postgres, the proposed naming specs are almost identical to Postgres naming specs.</p> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-10-08 06:39:35
 +
+

*Thread Reply:* Host would be something like +examplecluster.&lt;XXXXXXXXXXXX&gt;.<a href="http://us-west-2.redshift.amazonaws.com">us-west-2.redshift.amazonaws.com</a> +right?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ale + (alessandro.lollo@gmail.com) +
+
2021-10-08 07:13:51
 +
+

*Thread Reply:* Yep, let me update the PR

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ale + (alessandro.lollo@gmail.com) +
+
2021-10-08 07:27:42
 +
+

*Thread Reply:* Done

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-10-08 07:31:40
 +
+

*Thread Reply:* 🙌

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-10-08 07:35:30
 +
+

*Thread Reply:* If you want to look at dbt integration itself, there are two things:

+ +

We need to determine how Redshift adapter reports metrics https://github.com/OpenLineage/OpenLineage/blob/610a687bf69df2b52ec4ac4da80b4a05580e8d32/integration/common/openlineage/common/provider/dbt.py#L412

+ +

And how we can create namespace and job name based on the job naming schema that you created: +https://github.com/OpenLineage/OpenLineage/blob/610a687bf69df2b52ec4ac4da80b4a05580e8d32/integration/common/openlineage/common/provider/dbt.py#L512

+ +

One thing how to get this info is to run the dbt yourself and look at resulting metadata files - in target dir of the dbt directory

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/610a687bf69df2b52ec4ac4da80b4a05580e8d32/integration/common/openlineage/common/provider/dbt.py | dbt.py>
+
+ <pre><code> def extract_namespace(self, profile: Dict) -&amp;gt; str: +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ale + (alessandro.lollo@gmail.com) +
+
2021-10-08 08:33:31
 +
+

*Thread Reply:* I figured out how to generate the namespace. +But I can’t understand which of the JSON files is inspected for metrics. Is it run_results.json ?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-10-08 09:48:50
 +
+

*Thread Reply:* yes, run_results.json - it's different in bigquery and snowflake, so I presume it's different in redshift too

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ale + (alessandro.lollo@gmail.com) +
+
2021-10-08 11:02:32
 +
+

*Thread Reply:* Ok thanks!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ale + (alessandro.lollo@gmail.com) +
+
2021-10-08 11:11:57
 +
+

*Thread Reply:* Should be stats:rows:value

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ale + (alessandro.lollo@gmail.com) +
+
2021-10-08 11:19:59
 +
+

*Thread Reply:* Regarding namespace: if env_var is used in profiles.yml , how is this handled now?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-10-08 11:44:50
 +
+

*Thread Reply:* Well, it isn't. This is relevant only if you passed cluster hostname this way, right?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ale + (alessandro.lollo@gmail.com) +
+
2021-10-08 11:53:52
 +
+

*Thread Reply:* Exactly

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ale + (alessandro.lollo@gmail.com) +
+
2021-10-11 07:10:38
 +
+

*Thread Reply:* If you think it make sense, I can submit a PR to handle dbt profile with env_var

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-10-11 07:18:01
 +
+

*Thread Reply:* Do you want to run jinja on the dbt profile?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-10-11 07:20:18
 +
+

*Thread Reply:* Theoretically, we'd need to run it also on dbt_project.yml , but we only take target path and profile name from it.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ale + (alessandro.lollo@gmail.com) +
+
2021-10-11 07:20:32
 +
+

*Thread Reply:* The env_var syntax in the profile is quite simple, I was thinking of extracting the env var name using re and then retrieving the value from os

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-10-11 07:23:59
 +
+

*Thread Reply:* It would work, but we can actually use jinja - if you're using dbt, it's already included. +The method is pretty simple: +``` @contextmember + @staticmethod + def envvar(var: str, default: Optional[str] = None) -> str: + """The envvar() function. Return the environment variable named 'var'. + If there is no such environment variable set, return the default.

+ +
    If the default is None, raise an exception for an undefined variable.
+    """
+    if var in os.environ:
+        return os.environ[var]
+    elif default is not None:
+        return default
+    else:
+        msg = f"Env var required but not provided: '{var}'"
+        undefined_error(msg)```
+
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ale + (alessandro.lollo@gmail.com) +
+
2021-10-11 07:25:07
 +
+

*Thread Reply:* Oh cool! +I will definitely use this one!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-10-11 07:25:09
 +
+

*Thread Reply:* We'd be sure that our implementation matches dbt's one, right? Also, you'd support default method for free

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ale + (alessandro.lollo@gmail.com) +
+
2021-10-11 07:26:34
 +
+

*Thread Reply:* So this env_varmethod is defined in dbt and not in OpenLineage codebase, right?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-10-11 07:27:01
 +
+

*Thread Reply:* yes

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-10-11 07:27:14
 +
+

*Thread Reply:* dbt is on Apache license 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ale + (alessandro.lollo@gmail.com) +
+
2021-10-11 07:28:06
 +
+

*Thread Reply:* Should we import dbt package and use the method or should we just copy/paste the method inside OpenLineage codebase?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ale + (alessandro.lollo@gmail.com) +
+
2021-10-11 07:28:28
 +
+

*Thread Reply:* I’m asking for guidance here 😊

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-10-11 07:34:44
 +
+

*Thread Reply:* I think we should just do basic jinja template rendering in our code like in the quick example: https://realpython.com/primer-on-jinja-templating/#quick-examples

+ +

just with the env_var method passed to the render method 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-10-11 07:37:05
 +
+

*Thread Reply:* basically, here in the code we should read the file, do the jinja render, and load yaml from string instead of straight from file +https://github.com/OpenLineage/OpenLineage/blob/610a687bf69df2b52ec4ac4da80b4a05580e8d32/integration/common/openlineage/common/provider/dbt.py#L176

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/610a687bf69df2b52ec4ac4da80b4a05580e8d32/integration/common/openlineage/common/provider/dbt.py | dbt.py>
+
+ <pre><code> def load_yaml(path: str) -&amp;gt; Dict: +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ale + (alessandro.lollo@gmail.com) +
+
2021-10-11 07:38:53
 +
+

*Thread Reply:* ok, got it. +Will try to implement following your suggestions. +Thanks @Maciej Obuchowski 🙌

+ + + +
+ 🙌 Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ale + (alessandro.lollo@gmail.com) +
+
2021-10-11 08:36:13
 +
+

*Thread Reply:* We need to:

+ +
  1. load the template profile from the profile.yml
  2. replace any env vars we found +For the first step, we can use jinja2.Template +However, to replace the env vars we find, we have to actually search for those env vars… 🤔
    3. +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-10-11 08:43:06
 +
+

*Thread Reply:* The dbt method implements that: +``` @contextmember + @staticmethod + def envvar(var: str, default: Optional[str] = None) -> str: + """The envvar() function. Return the environment variable named 'var'. + If there is no such environment variable set, return the default.

+ +
    If the default is None, raise an exception for an undefined variable.
+    """
+    if var in os.environ:
+        return os.environ[var]
+    elif default is not None:
+        return default
+    else:
+        msg = f"Env var required but not provided: '{var}'"
+        undefined_error(msg)```
+
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ale + (alessandro.lollo@gmail.com) +
+
2021-10-11 08:45:54
 +
+

*Thread Reply:* Ok, but I need to pass var to the env_var method. +And to pass the var value, I need to look into the loaded Template and search for env var names…

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-10-11 08:46:54
 +
+

*Thread Reply:* that's what jinja does - you're passing function to jinja render, and it's calling it itself

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-10-11 08:47:45
 +
+

*Thread Reply:* you can try the quick example from here, but just pass the env_var method (slightly adjusted - as a standalone function and without undefined error) and call it inside the template: https://realpython.com/primer-on-jinja-templating/#quick-examples

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ale + (alessandro.lollo@gmail.com) +
+
2021-10-11 08:51:19
 +
+

*Thread Reply:* Ok, will try

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ale + (alessandro.lollo@gmail.com) +
+
2021-10-11 09:37:49
 +
+

*Thread Reply:* I’m trying to run +pip install -e ".[dev]" +so that I can test my changes, but I get +ERROR: Could not find a version that satisfies the requirement openlineage-integration-common[dbt]==0.2.3 (from openlineage-dbt[dev]) (from versions: 0.0.1rc7, 0.0.1rc8, 0.0.1, 0.1.0rc5, 0.1.0, 0.2.0, 0.2.1, 0.2.2) +ERROR: No matching distribution found for openlineage-integration-common[dbt]==0.2.3 +I don’t understand what I’m doing wrong…

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-10-11 09:41:47
 +
+

*Thread Reply:* can you try installing it manually?

+ +

pip install openlineage-integration-common[dbt]==0.2.3

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-10-11 09:42:13
 +
+

*Thread Reply:* I mean, it exists in pypi: https://pypi.org/project/openlineage-integration-common/#files

+
+
PyPI
+ + + +
openlineage-integration-common
+
+ OpenLineage common python library for integrations +
+ + + + + + + + +
Original URL: https://pypi.org/project/openlineage-integration-common/#files
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ale + (alessandro.lollo@gmail.com) +
+
2021-10-11 09:44:57
 +
+

*Thread Reply:* Yep, maybe it’s our internal Pypi repo which is not synced. +Installing from the public pypi resolved the issue

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ale + (alessandro.lollo@gmail.com) +
+
2021-10-11 12:04:55
 +
+

*Thread Reply:* Can;’t seem to make env_var working as the render method of a Template 😅

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-10-11 12:57:07
 +
+

*Thread Reply:* try this:

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-10-11 12:57:09
 +
+

*Thread Reply:* ```import os +from typing import Optional +from jinja2 import Template

+ +

def envvar(var: str, default: Optional[str] = None) -> str: + """The envvar() function. Return the environment variable named 'var'. + If there is no such environment variable set, return the default.

+ +
If the default is None, raise an exception for an undefined variable.
+"""
+if var in os.environ:
+    return os.environ[var]
+elif default is not None:
+    return default
+else:
+    msg = f"Env var required but not provided: '{var}'"
+    raise Exception("")
+
+ +

if name == 'main': + t = Template("Hello {{ envvar('ENVVAR') }}!") + print(t.render(envvar=envvar))```

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-10-11 12:57:42
 +
+

*Thread Reply:* works for me: +mobuchowski@thinkpad [18:57:14] [~] +-&gt; % ENV_VAR=world python jinja_example.py +Hello world!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ale + (alessandro.lollo@gmail.com) +
+
2021-10-11 16:59:13
 +
+

*Thread Reply:* Finally 😅 +https://github.com/OpenLineage/OpenLineage/pull/328

+ +

There are minimal tests for Redshift and env vars. +Feedbacks and suggestions are welcome!

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/pull/328|#328 - Added support for Redshift profile>
+
+ <p>• Added support for Redshift profile +• Added support for handling env vars in profiles.yml</p> + +<p>Please note that I had to enclose <code>{{ config(...) }}</code> between <code>raw</code> Jinja tags, otherwise it complains about an unexpected '.'</p> + +<p>Signed-off-by: Alessandro Lollo <a href="mailto:alessandro.lollo@cloudacademy.com">alessandro.lollo@cloudacademy.com</a></p> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ale + (alessandro.lollo@gmail.com) +
+
2021-10-12 03:10:45
 +
+

*Thread Reply:* Hi @Maciej Obuchowski 😊 +Regarding this comment https://github.com/OpenLineage/OpenLineage/pull/328#discussion_r726586564

+ +

How can we distinguish between snowflake, bigquery and redshift in this method?

+ +

A simple, but not very clean solution, would be to split this +bytes = get_from_multiple_chains( + node.catalog_node, + [ + ['stats', 'num_bytes', 'value'], # bigquery + ['stats', 'bytes', 'value'], # snowflake + ['stats', 'size', 'value'] # redshift (Note: size = count of 1MB blocks) + ] + ) +into two pieces, one checking for snowflake and bigquery and the other checking for redshift.

+ +

A better solution would be to have the profile type inside method node_to_output_dataset , but I’m struggling understanding how to do that

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-10-12 05:35:00
 +
+

*Thread Reply:* Well, why not do something like

+ +

```bytes = getfrommultiple_chains(... rest of stuff)

+ +

if adapter == 'redshift': + bytes = 10241024```

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-10-12 05:36:49
 +
+

*Thread Reply:* we can store adapter type in the class

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-10-12 05:38:47
 +
+

*Thread Reply:* well, I've looked at last commit and that's exactly what you did 👍

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-10-12 05:40:35
 +
+

*Thread Reply:* Now, have you tested your branch on real redshift cluster? I don't think we 100% need automated tests for that now, but would be nice to have confirmation that it works.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ale + (alessandro.lollo@gmail.com) +
+
2021-10-12 06:35:04
 +
+

*Thread Reply:* Not yet, but I'll try to do that this afternoon. +Need to figure out how to build the lib locally, then I can use it to test with Redshift

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-10-12 06:40:58
 +
+

*Thread Reply:* I think pip install -e .[dbt] in common directory should be enough

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ale + (alessandro.lollo@gmail.com) +
+
2021-10-12 09:29:13
 +
+

*Thread Reply:* I was able to run my local branch with my Redshift cluster and metadata is pushed to Marquez. +However, I’m not sure about the namespace . +I also see exceptions in Marquez logs

+ +
+
+ + + + + + +
+
+
+ + + + + + +
+
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-10-12 09:33:26
 +
+

*Thread Reply:* namespace: well, if it matches what you put into your profile, there's not much we can do. I don't understand why you connect to redshift via host, maybe this is related to IAM?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-10-12 09:44:17
 +
+

*Thread Reply:* I think the marquez error is because we don't send SourceCodeLocationJobFacet

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ale + (alessandro.lollo@gmail.com) +
+
2021-10-12 09:46:17
 +
+

*Thread Reply:* Regarding the namespace, I will check it and figure it out 😊 +Regarding the error: in the context of this PR, is it something I should worry about or not?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-10-12 09:54:17
 +
+

*Thread Reply:* I think not in the context of the PR. It certainly deserves separate issue in Marquez repository.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ale + (alessandro.lollo@gmail.com) +
+
2021-10-12 10:24:38
 +
+

*Thread Reply:* 👍

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ale + (alessandro.lollo@gmail.com) +
+
2021-10-12 10:24:51
 +
+

*Thread Reply:* Is there anything else I can do to improve the PR?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-10-12 10:27:44
 +
+

*Thread Reply:* did you figure out the namespace stuff? +I think it's ready to be merged outside of that

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ale + (alessandro.lollo@gmail.com) +
+
2021-10-12 10:49:06
 +
+

*Thread Reply:* Not yet

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ale + (alessandro.lollo@gmail.com) +
+
2021-10-12 10:58:07
 +
+

*Thread Reply:* Ok i figured it out. +When running dbt locally, we connect to Redshift using an SSH tunnel. +dbt runs on Docker, hence it can access the tunnel using host.docker.internal

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ale + (alessandro.lollo@gmail.com) +
+
2021-10-12 10:58:16
 +
+

*Thread Reply:* So the namespace is correct

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-10-12 11:04:12
 +
+

*Thread Reply:* Makes sense. So, let's merge it, after DCO bot gets up again.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ale + (alessandro.lollo@gmail.com) +
+
2021-10-12 11:04:37
 +
+

*Thread Reply:* 👍

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-10-13 05:29:48
 +
+

*Thread Reply:* merged your PR 🙌

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ale + (alessandro.lollo@gmail.com) +
+
2021-10-13 10:54:09
 +
+

*Thread Reply:* 🎉

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-10-13 12:01:20
 +
+

*Thread Reply:* I think I'm going to change it up a bit. +The problem is that we can try to render jinja everywhere, including comments. +I tried to make it skip unknown methods and values here, but I think the right solution is to load the yaml, and then try to render jinja for values.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ale + (alessandro.lollo@gmail.com) +
+
2021-10-13 14:27:37
 +
+

*Thread Reply:* Ok sounds good to me!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
SAM + (skhettri@gmail.com) +
+
2021-10-06 10:50:43
 +
+

Hey there, I’m not sure why I’m getting below error, after I ran OPENLINEAGE_URL=<http://localhost:5000> dbt-ol run , although running this command dbt debug doesn’t show any error. Pls help.

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-10-06 10:54:32
 +
+

*Thread Reply:* Does it work with simply dbt run?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-10-06 10:55:51
 +
+

*Thread Reply:* also, do you have dbt-snowflake installed?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
SAM + (skhettri@gmail.com) +
+
2021-10-06 11:00:42
 +
+

*Thread Reply:* it works with dbt run

+ + + +
+ 👀 Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
SAM + (skhettri@gmail.com) +
+
2021-10-06 11:01:22
 +
+

*Thread Reply:* no i haven’t installed dbt-snowflake

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-10-06 12:04:19
 +
+

*Thread Reply:* what the dbt says - the snowflake profile with dev target - is that what you ment to run or was it something else?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-10-06 12:04:46
 +
+

*Thread Reply:* it feels very weird to me, since the dbt-ol script just runs dbt run underneath

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
SAM + (skhettri@gmail.com) +
+
2021-10-06 12:19:27
 +
+

*Thread Reply:* this is my profiles.yml file: +```snowflake: + target: dev + outputs: + dev: + type: snowflake + account: xxxxxxx

+ +
  # User/password auth
+  user: xxxxxx
+  password: xxxxx
+
+  role: poc_db_temp_fullaccess
+  database: POC_DB
+  warehouse: poc_wh
+  schema: temp
+  threads: 2
+  client_session_keep_alive: False
+  query_tag: dbt_ol```
+
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-10-06 12:26:39
 +
+

*Thread Reply:* Yes, it looks that everything is okay on your side...

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
SAM + (skhettri@gmail.com) +
+
2021-10-06 12:28:19
 +
+

*Thread Reply:* may be I’ll restart my machine and try again

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-10-06 12:30:25
 +
+

*Thread Reply:* can you try +OPENLINEAGE_URL=<http://localhost:5000> dbt-ol debug

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
SAM + (skhettri@gmail.com) +
+
2021-10-07 05:59:03
 +
+

*Thread Reply:* Actually i had to use venv that fixed above issue. However, i ran into another problem which is no jobs / datasets found in marquez:

+ +
+
+ + + + + + +
+
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-10-07 06:00:28
 +
+

*Thread Reply:* Good that you fixed that one 🙂 Regarding last one, I've found it independently yesterday and PR fixing it is already waiting for review: https://github.com/OpenLineage/OpenLineage/pull/322

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/pull/322|#322 dbt: support manifest and run-results v3>
+
+ <p>There are no major differences between v2 and v3 metadata, at least those impacting information that we're collecting. <br /> +This PR makes <code>dbt-ol</code> treat v3 metadata the same way we treated v2.</p> + +<p>Signed-off-by: Maciej Obuchowski <a href="mailto:maciej.obuchowski@getindata.com">maciej.obuchowski@getindata.com</a></p> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
SAM + (skhettri@gmail.com) +
+
2021-10-07 06:00:46
 +
+

*Thread Reply:* oh, thanks a lot

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-10-07 14:50:01
 +
+

*Thread Reply:* There will be a release soon: https://openlineage.slack.com/archives/C01CK9T7HKR/p1633631825147900

+
+ + +
+ + + } + + Willy Lulciuc + (https://openlineage.slack.com/team/U01DCMDFHBK) +
+ + + +
+
+ <p>@channel: We’ve recently become aware that our integration with dbt no longer works with the <em>latest</em> dbt manifest version (<code>v3</code>), see <a href="https://openlineage.slack.com/archives/C01CK9T7HKR/p1633531843132400">original</a> discussion. The manifest version change was introduced in dbt <code>0.21</code> , see <a href="https://github.com/dbt-labs/dbt/commit/ea07729bbf0a3481e220615644b7955659001529#diff-33c94e40283afdfeba411a98c0628fc610c287e41c3477d88559f1e0244259c3">diff</a>. That said, we do have a fix: <a href="https://github.com/OpenLineage/OpenLineage/pull/322">PR #322</a> contributed by @Maciej Obuchowski! Here’s our plan to rollout the <code>openlineage-dbt</code> hotfix for those using the <em>latest</em> version of dbt (<strong>NOTE: for those using an older dbt version, you will NOT not be affected by this bug</strong>):</p> + +<p><strong>Releasing OpenLineage <code>0.2.3</code></strong> <strong>with dbt <code>v3</code> manifest support:</strong></p> + +<ol><li>Branch off <code>0.2.2</code> tagged commit, and create a <code>openlineage-0.2.x</code> branch</li><li>Cherry pick the commit with the dbt manifest <code>v3</code> fix</li><li>Release <code>0.2.3</code> batch release +We will be releasing <code>0.2.3</code> <em>today</em>. Please reach out to us with any questions!</li> +</ol> +
+ + + + + + +
Original URL: https://openlineage.slack.com/archives/C01CK9T7HKR/p1633631825147900
+ + +
+ + Slack Conversation +
+ + +
+ + + +
+ 👍 SAM +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
SAM + (skhettri@gmail.com) +
+
2021-10-07 23:23:26
 +
+

*Thread Reply:* Hi, +openlineage-dbt==0.2.3 worked, thanks a lot for the quick fix.

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Alex P + (alexander.pelivan@scout24.com) +
+
2021-10-07 07:46:16
 +
+

Hi, I just started playing around with Marquez. When submitting some lineage data, after some experimenting, the visualisation becomes a bit cluttered with all the naive attempts of building a meaningful graph. Can I clear this up somehow? Or is there a tip, how to hide certain information?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Alex P + (alexander.pelivan@scout24.com) +
+
2021-10-07 07:46:59
 +
+

*Thread Reply:*

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Alex P + (alexander.pelivan@scout24.com) +
+
2021-10-07 09:51:40
 +
+

*Thread Reply:* So, as a quick fix, shutting down and re-starting the docker container resets everything. +./docker/up.sh

+ + + +
+ 👍 Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-10-07 12:28:25
 +
+

*Thread Reply:* I guess that it's the easiest way now. There should be API for that.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2021-10-07 14:09:50
 +
+

*Thread Reply:* @Alex P Yeah, we're realizing that being able to delete metadata is becoming very important. And, as @Maciej Obuchowski mentioned, dropping your entire database is the only way currently (not ideal!). We do have an issue in the Marquez backlog to expose delete APIs: https://github.com/MarquezProject/marquez/issues/754

+
+ + + + +
<https://github.com/MarquezProject/marquez/issues/754|#754 Delete endpoint for routes>
+
+ <p>It would be super convenient to have DELETE calls for endpoints.</p> + +<p>I.e</p> + +<p>As a user conducting testing of a data set I wish to DELETE it and then start a new <br /> +As a user creating a job I linked the wrong datasets, I wish to the DELETE the job</p> + +<p>Ideally for:</p> + +<p>• Sources +• Datasets +• Jobs +• Tags</p> +
+ +
+
Labels
+ feature, api +
+ + + + + + + +
+ + <a href="https://github.com/MarquezProject/marquez">MarquezProject/marquez</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2021-10-07 14:10:36
 +
+

*Thread Reply:* A bit more discussion is needed though. Like what if a dataset is deleted, but you still want to keep track that it existed at some point? (i.e. soft vs hard deletes). But, for the case that you just want to clear metadata because you were testing things out, then yeah, that's more obvious and requires little discussion of the API upfront.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2021-10-07 14:12:52
 +
+

*Thread Reply:* @Alex P I moved the delete APIs to the Marquez 0.20.0 release

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-10-07 14:39:03
 +
+

*Thread Reply:* Thanks Willy.

+ + + +
+ 🙌 Willy Lulciuc +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-10-07 14:48:37
 +
+

*Thread Reply:* I have also updated a corresponding issue to track this in OpenLineage: https://github.com/OpenLineage/OpenLineage/issues/323

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/issues/323|#323 [PROPOSAL] Ability to annotate datasets with metadata without a job run>
+
+ <p><strong>Purpose:</strong> <br /> +We need a way to make observations about a Dataset current state outside of a run reading or writing the data. <br /> +For example, this could be:</p> + +<p>• collecting schema information about a table from a warehouse +• informing that a table has been deleted +• etc</p> + +<p><strong>Proposed implementation</strong> <br /> +Adding Dataset event types that are not lineage events. This would be the Dataset entity with its facets: </p> + +<p><a href="https://github.com/OpenLineage/OpenLineage/blob/3a1ccbd854bbf202bbe6437bf81786cb01ed4d01/spec/OpenLineage.json#L178-L203">OpenLineage/spec/OpenLineage.json</a></p> + +<p>Lines 178 to 203 in &lt;/OpenLineage/OpenLineage/commit/3a1ccbd854bbf202bbe6437bf81786cb01ed4d01|3a1ccbd&gt;</p> + +<pre><code>"Dataset": { + "type": "object", + "properties": { + "namespace": { + "description": "The namespace containing that dataset", + "type": "string", + "example": "my-datasource-namespace" + }, + "name": { + "description": "The unique name for that dataset within that namespace", + "type": "string", + "example": "instance.schema.table" + }, + "facets": { + "description": "The facets for this dataset", + "type": "object", + "additionalProperties": { + "$ref": "#/$defs/DatasetFacet" + } + } + }, + "required": [ + "namespace", + "name" + ] + } + +</code></pre> +
+ +
+
Labels
+ proposal +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-10-07 13:36:48
 +
+

The next OpenLineage monthly meeting is on the 13th. https://wiki.lfaidata.foundation/display/OpenLineage/Monthly+TSC+meeting +please chime in here if you’d like a topic to be added to the agenda

+ + + +
+ 🙌 Willy Lulciuc, Maciej Obuchowski, Peter Hicks +
+ +
+ ❤️ Willy Lulciuc, Maciej Obuchowski, Peter Hicks +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-10-13 10:47:49
 +
+

*Thread Reply:* Reminder that the meeting is today. See you soon

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-10-13 19:49:21
 +
+

*Thread Reply:* The recording and notes of the meeting are now available: +https://wiki.lfaidata.foundation/display/OpenLineage/Monthly+TSC+meeting#MonthlyTSCmeeting-Oct13th2021

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2021-10-07 14:37:05
 +
+

@channel: We’ve recently become aware that our integration with dbt no longer works with the latest dbt manifest version (v3), see original discussion. The manifest version change was introduced in dbt 0.21 , see diff. That said, we do have a fix: PR #322 contributed by @Maciej Obuchowski! Here’s our plan to rollout the openlineage-dbt hotfix for those using the latest version of dbt (NOTE: for those using an older dbt version, you will NOT not be affected by this bug):

+ +

Releasing OpenLineage 0.2.3 with dbt v3 manifest support:

+ +
  1. Branch off 0.2.2 tagged commit, and create a openlineage-0.2.x branch
  2. Cherry pick the commit with the dbt manifest v3 fix
  3. Release 0.2.3 batch release +We will be releasing 0.2.3 today. Please reach out to us with any questions!
    4. +
+
+ + +
+ + + } + + Samjhana Khettri + (https://openlineage.slack.com/team/U02EYPQNU58) +
+ + + +
+
+ <p>Hey there, I’m not sure why I’m getting below error, after I ran <code>OPENLINEAGE_URL=&lt;http://localhost:5000&gt; dbt-ol run</code> , although running this command <code>dbt debug</code> doesn’t show any error. Pls help.</p> +
+ + + + + + +
Original URL: https://openlineage.slack.com/archives/C01CK9T7HKR/p1633531843132400
+ + +
+ + Thread in Slack Conversation +
+ + +
+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/pull/322|#322 dbt: support manifest and run-results v3>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+ 🙌 Mario Measic, Minkyu Park, Peter Hicks +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-10-07 14:55:35
 +
+

*Thread Reply:* For people following along, dbt changed the schema of its metadata which broke the openlineage integration. However we were a bit too stringent on validating the schema version (they increment it every time event if it’s backwards compatible, which it is in this case). We will fix that so that future compatible changes don’t prevent the ol integration to work.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mario Measic + (mario.measic.gavran@gmail.com) +
+
2021-10-07 16:44:28
 +
+

*Thread Reply:* As one of the main integrations, would be good to connect more within the dbt community for the next releases, by testing the release candidates 👍

+ +

Thanks for the PR

+ + + +
+ 💯 Willy Lulciuc +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2021-10-07 16:46:40
 +
+

*Thread Reply:* Yeah, I totally agree with you. We also should be more proactive and also be more aware in what’s coming in future dbt releases. Sorry if you were effected by this bug :ladybug:

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2021-10-07 18:12:22
 +
+

*Thread Reply:* We’ve release OpenLineage 0.2.3 with the hotfix for adding dbt v3 manifest support, see https://github.com/OpenLineage/OpenLineage/releases/tag/0.2.3

+ +

You can download and install openlineage-dbt 0.2.3 with the fix using:

+ +

$ pip3 install openlineage-dbt==0.2.3

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Drew Bittenbender + (drew@salt.io) +
+
2021-10-07 19:02:37
 +
+

Hello. I have a question about dbt-ol. I run dbt in a docker container and alias the dbt command to execute in that docker container. dbt-ol doesn't seem to use that alias. Do you know of a way to force it to use the alias?...or is there an alternative to getting the linage into Marquez?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-10-07 21:10:36
 +
+

*Thread Reply:* @Maciej Obuchowski might know

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-10-08 04:23:17
 +
+

*Thread Reply:* @Drew Bittenbender dbt-ol always calls dbt command now, without spawning shell - so it does not have access to bash aliases.

+ +

Can you elaborate about your use case? Do you mean that dbt in your path does docker run or something like this? It still might be a problem if we won't have access to artifacts generated by dbt in target directory.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Drew Bittenbender + (drew@salt.io) +
+
2021-10-08 10:59:32
 +
+

*Thread Reply:* I am running on a mac and I have aliased (.zshrc) dbt to execute docker run against the fishtownanalytics docker image rather than installing dbt natively (homebrew, etc). I am doing this so that the dbt configuration is portable and reusable by others.

+ +

It seems that by installing openlineage-dbt in a virtual environment, it pulls down it's own version of dbt which it calls inline rather than shelling out and executing the dbt setup resident in the host system. I understand that opening a shell is a security risk so that is understandable.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-10-08 11:05:00
 +
+

*Thread Reply:* It does not pull down, it just assumes that it's in the system. It would fail if it isn't.

+ +

For now I think you could build your own image based on official one, and install openlineage-dbt inside, something like:

+ +

FROM fishtownanalytics/dbt:0.21.0 +RUN pip install openlineage-dbt +ENTRYPOINT ["dbt-ol"]

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-10-08 11:05:15
 +
+

*Thread Reply:* and then pass OPENLINEAGE_URL in env while doing docker run

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-10-08 11:06:55
 +
+

*Thread Reply:* Also, to make sure that using shell would help in your case: do you bind mount your dbt directory to home? dbt-ol can't run without access to dbt's target directory, so if it's not visible in host, the only option is to have dbt-ol in container.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
SAM + (skhettri@gmail.com) +
+
2021-10-08 07:00:43
 +
+

Hi, I found below issues, not sure what is the root-cause:

+ +
  1. Marquez UI does not show any jobs/datasets, but if I search my table name then only it shows in search result section.
  2. After running dbt docs generate there is not schema information available in marquez?
    3. +
+ +
+
+ + + + + + +
+
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-10-08 08:16:37
 +
+

*Thread Reply:* Regarding 2), the data is only visible after next dbt-ol run - dbt docs generate does not emit events itself, but generates data that run take into account.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
SAM + (skhettri@gmail.com) +
+
2021-10-08 08:24:57
 +
+

*Thread Reply:* oh got it, since its in default, i need to click on it and choose my dbt profile’s account name. thnx

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
SAM + (skhettri@gmail.com) +
+
2021-10-08 11:25:22
 +
+

*Thread Reply:* May I know, why these highlighted ones dont have schema? FYI, I used sources in dbt.

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-10-08 11:26:18
 +
+

*Thread Reply:* Do they have it in dbt docs?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
SAM + (skhettri@gmail.com) +
+
2021-10-08 11:33:59
 +
+

*Thread Reply:* I prepared this yaml file, not sure this is what u asked

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ale + (alessandro.lollo@gmail.com) +
+
2021-10-12 04:14:08
 +
+

Hey folks 😊 +DCO checks on this PR https://github.com/OpenLineage/OpenLineage/pull/328 seem to be stuck. +Any suggestions on how to unblock it?

+ +

Thanks!

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/pull/328|#328 - Added support for Redshift profile>
+
+ <p>• Added support for Redshift profile +• Added support for handling env vars in profiles.yml</p> + +<p>Please note that I had to enclose <code>{{ config(...) }}</code> between <code>raw</code> Jinja tags, otherwise it complains about an unexpected '.'</p> + +<p>Signed-off-by: Alessandro Lollo <a href="mailto:alessandro.lollo@cloudacademy.com">alessandro.lollo@cloudacademy.com</a></p> +
+ +
+
Comments
+ 1 +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-10-12 07:21:33
 +
+

*Thread Reply:* I don't think anything is wrong with your branch. It's also not working on my one. Maybe it's globally stuck?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mark Taylor + (marktayl@microsoft.com) +
+
2021-10-12 15:17:02
 +
+

We are working on the hackathon and have a couple of questions about generating lineage information. @Willy Lulciuc would you have time to help answer a couple of questions?

+ +

• Is there a way to generate OpenLineage output that contains a mapping between input and output fields? +• In Azure Databricks sources often map to ADB mount points. We are looking for a way to translate this into source metadata in the OL output. Is there some configuration that would make this possible, or any other suggestions?

+ + + +
+ 👋 Willy Lulciuc +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2021-10-12 15:50:20
 +
+

*Thread Reply:* > Is there a way to generate OpenLineage output that contains a mapping between input and output fields? +OpenLineage defines discrete classes for both OpenLineage.InputDataset and OpenLineage.OutputDataset datasets. But, for clarification, are you asking:

+ +
  1. If a job reads / writes to the same dataset, how can OpenLineage track which fields were used in job’s logic as input and which fields were used to write back to the resulting output?
  2. Or, if a job reads / writes from two different dataset, how can OpenLineage track which input fields were used in the job’s logic for the resulting output dataset? (i.e. column-level lineage)
    3. +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2021-10-12 15:56:18
 +
+

*Thread Reply:* > In Azure Databricks sources often map to ADB mount points.  We are looking for a way to translate this into source metadata in the OL output.  Is there some configuration that would make this possible, or any other suggestions? +I would look into our OutputDatasetVisitors class (as a starting point) that extracts metadata from the spark logical plan to construct a mapping between a logic plan to one or more OpenLineage.Dataset for the spark job. But, I think @Michael Collado will have a more detailed suggestion / approach to what you’re asking

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2021-10-12 15:59:41
 +
+

*Thread Reply:* are the sources mounted like local filesystem mounts? are you ending up with datasources that point to the local filesystem rather than some dbfs url? (sorry, I'm not familiar with databricks or azure at this point)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mark Taylor + (marktayl@microsoft.com) +
+
2021-10-12 16:59:38
 +
+

*Thread Reply:* I think under the covers they are an os level fs mount, but it is using an ADB specific api, dbutils.fs.mount. It is using the ADB filesystem.

+
+
docs.microsoft.com
+ + + +
Databricks File System (DBFS) - Azure Databricks
+
+ Learn about Databricks File System (DBFS). +
+ + + + + + + + +
Original URL: https://docs.microsoft.com/en-us/azure/databricks/data/databricks-file-system
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2021-10-12 17:01:23
 +
+

*Thread Reply:* Do you use the dbfs scheme to access the files from Spark as in the example on that page? +df = spark.read.text("dbfs:/mymount/my_file.txt")

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mark Taylor + (marktayl@microsoft.com) +
+
2021-10-12 17:04:52
 +
+

*Thread Reply:* @Willy Lulciuc In our project, @Will Johnson had generated some sample OL output from just reading in and writing out a dataset to blob storage. In the resulting output, I see the columns represented as fields under the schema element with a set represented for output and another for input. I would need the mapping of in and out columns to generate column level lineage so wondering if it is possible to get or am I just missing it somewhere? Thanks for your help!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2021-10-12 17:26:35
 +
+

*Thread Reply:* Ahh, well currently, no, but it has been discussed and on the OpenLineage roadmap. Here’s a proposal opened by @Julien Le Dem, column level lineage facet, that starts the discussion to add the columnLineage face to the datasets model in order to support column-level lineage. Would be great to get your thoughts!

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/issues/148|#148 [PROPOSAL] column level lineage facet>
+
+ <p><strong>Purpose:</strong> <br /> +For transformations like SQL queries (but also in other cases), we can extract column level lineage. This allows answering questions like ‘which root input columns are used to construct column x?’. We need to be able to capture this information in the OpenLineage model</p> + +<p><strong>Proposed implementation</strong> <br /> +We propose to introduce a new dataset facet: <br /> +example:</p> + +<pre><code>{ + "eventType": "COMPLETE", + "eventTime": "2020-12-28T20:52:00.001+10:00", + "run" : { + "runId": "uuid" + }, + "job": { + "namespace": "scheduler", + "name": "myjob", + "facets": { + "sql": { + "query": "Insert into outputTable from select ** from inputTable" + } + } + }, + "inputs": [ + { + "namespace": "N1", + "name": "inputTable", + "facets": { + "schema": { + "fields": [ {"name": "col_a", "type": "VARCHAR"}, {"name": "col_b", "type": "int"}] + } + } + } + ], + "outputs": [ + { + "namespace": "N2", + "name": "outputTable", + "facets": { + "schema": { + "fields": [ {"name": "col_a", "type": "VARCHAR"}, {"name": "col_b", "type": "int"}] + }, + "columnLineage": { + "fields": { + "col_a": [ { "namespace": "N1", "name": "inputTable", "field": "col_a"} ], + "col_b": [ { "namespace": "N1", "name": "inputTable", "field": "col_b" } ] + } + } + } + } + ] +} +</code></pre> + +<p>Schema:</p> + +<pre><code> "columnLineage": { + "type": "object", + "properties": { + "fields": { + "type": "array", + "items": { + "type": "object", + "additionalProperties": { + "type": "object", + "properties": { + "namespace": { + "type": "string", + "description": "the input dataset namespace" + }, + "name": { + "type": "string", + "description": "the input dataset name" + }, + "field": { + "type": "string", + "description": "the input field" + } + }, + "required": [ + "namespace", "name" + ] + } + } + } + } +</code></pre> + +<p>References:</p> + +<p>• Datahub spec: <a href="https://datahubproject.io/docs/rfc/active/1841-lineage/field_level_lineage/">https://datahubproject.io/docs/rfc/active/1841-lineage/field<em>level</em>lineage/</a></p> +
+ +
+
Labels
+ proposal +
+ +
+
Comments
+ 3 +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2021-10-12 17:41:41
 +
+

*Thread Reply:* @Michael Collado - Databricks allows you to reference a file called /mnt/someMount/some/file/path The way you have referenced it would let you hit the file with local file system stuff like pandas / local python.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-10-12 17:49:37
 +
+

*Thread Reply:* For column level lineage, you can add your own custom facets: Here’s an example in the Spark integration: (LogicalPlanFacet) https://github.com/OpenLineage/OpenLineage/blob/5f189a94990dad715745506c0282e16fd8[…]openlineage/spark/agent/lifecycle/SparkSQLExecutionContext.java +Here is the paragraph about this in the spec: https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.md#custom-facet-naming

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/5f189a94990dad715745506c0282e16fd857b43a/integration/spark/src/main/java/io/openlineage/spark/agent/lifecycle/SparkSQLExecutionContext.java | SparkSQLExecutionContext.java>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.md | OpenLineage.md>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-10-12 17:51:24
 +
+

*Thread Reply:* This example adds facets to the run, but you can also add them to the job

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2021-10-12 17:52:46
 +
+

*Thread Reply:* unfortunately, there's not yet a way to add your own custom facets to the spark integration- there's some work on extensibility to be done

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2021-10-12 17:54:07
 +
+

*Thread Reply:* for the hackathon's sake, you can check out the package and just add in whatever you want

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2021-10-12 18:26:44
 +
+

*Thread Reply:* Thank you guys!!

+ + + +
+ 🙌 Willy Lulciuc +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2021-10-12 20:42:20
 +
+

Question on the Spark Integration and its SPARKCONFURL_KEY configuration variable.

+ +

https://github.com/OpenLineage/OpenLineage/blob/8afc4ff88b8dd8090cd9c45061a9f669fe[…]rk/src/main/java/io/openlineage/spark/agent/ArgumentParser.java

+ +

It looks like I can pass in any url but I'm not sure if I can pass in query parameters along with that URL. For example, if I had https://localhost/myendpoint?secret_code=123 I THINK that is used for the endpoint and it does not append /lineage to the end of the url. Is that a fair assessment of what happens when the url is provided?

+ +

Thank you for any guidance!

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/8afc4ff88b8dd8090cd9c45061a9f669fea2151e/integration/spark/src/main/java/io/openlineage/spark/agent/ArgumentParser.java | ArgumentParser.java>
+
+ <pre><code> public static ArgumentParser parse(String clientUrl) { +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-10-12 21:46:12
 +
+

*Thread Reply:* You can also pass the settings independently if you want something more flexible: https://github.com/OpenLineage/OpenLineage/blob/8afc4ff88b8dd8090cd9c45061a9f669fe[…]n/java/io/openlineage/spark/agent/OpenLineageSparkListener.java

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/8afc4ff88b8dd8090cd9c45061a9f669fea2151e/integration/spark/src/main/java/io/openlineage/spark/agent/OpenLineageSparkListener.java | OpenLineageSparkListener.java>
+
+ <pre><code> private ArgumentParser parseConf(SparkConf conf) { +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-10-12 21:47:36
 +
+

*Thread Reply:* SparkSession.builder() + .config("spark.jars.packages", "io.openlineage:openlineage_spark:0.2.+") + .config("spark.extraListeners", "io.openlineage.spark.agent.OpenLineageSparkListener") + .config("spark.openlineage.host", "<https://localhost>") + .config("spark.openlineage.apiKey", "your api key") + .config("spark.openlineage.namespace", "&lt;NAMESPACE_NAME&gt;") // Replace with the name of your Spark cluster. + .getOrCreate()

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-10-12 21:48:57
 +
+

*Thread Reply:* It is going to add /lineage in the end: https://github.com/OpenLineage/OpenLineage/blob/8afc4ff88b8dd8090cd9c45061a9f669fe[…]rc/main/java/io/openlineage/spark/agent/OpenLineageContext.java

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/8afc4ff88b8dd8090cd9c45061a9f669fea2151e/integration/spark/src/main/java/io/openlineage/spark/agent/OpenLineageContext.java | OpenLineageContext.java>
+
+ <pre><code> new URI(String.format("%s/api/%s/lineage", argument.getHost(), argument.getVersion())); +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-10-12 21:49:37
 +
+

*Thread Reply:* the apiKey setting is sent in an “Authorization” header

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-10-12 21:49:55
 +
+

*Thread Reply:* “Bearer $KEY”

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-10-12 21:51:09
 +
+

*Thread Reply:* https://github.com/OpenLineage/OpenLineage/blob/a6eea7a55fef444b6561005164869a9082[…]n/java/io/openlineage/spark/agent/client/OpenLineageClient.java

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/a6eea7a55fef444b6561005164869a90827cef41/integration/spark/src/main/java/io/openlineage/spark/agent/client/OpenLineageClient.java | OpenLineageClient.java>
+
+ <pre><code> request.addHeader(AUTHORIZATION, "Bearer " + apiKey.get()); +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2021-10-12 22:54:22
 +
+

*Thread Reply:* Thank you @Julien Le Dem it seems in both cases (defining the url endpoint with spark.openlineage.url and with the components: spark.openlineage.host / openlineage.version / openlineage.namespace / etc.) OpenLineage will strip out url parameters and rebuild the url endpoint with /lineage.

+ +

I think we might need to add in a url parameter configuration for our hackathon. We're using a bit of serverless code to shuttle open lineage events to a queue so that another job and/or serverless application can read that queue at its leisure.

+ +

Using the apiKey that feeds into the Authorization header as a Bearer token is great and would suffice but for our services we use OAuth tokens that would expire after two hours AND most of our customers wouldn't want to generate an access token themselves and feed it to Spark. ☹️

+ +

Would you guys entertain a proposal to support a spark.openlineage.urlParams configuration variable that lets you add url parameters to the derived lineage url?

+ +

Thank you for the detailed replies and deep links!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-10-13 10:46:22
 +
+

*Thread Reply:* Yes, please open an issue detailing the use case.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2021-10-13 13:02:06
 +
+

Quick question, is it expected, when using Spark SQL and the Spark Integration for Spark3 that we receive and INPUT but no OUTPUTS when doing a CREATE TABLE ... AS SELECT ... .

+ +

I'm reading from a Spark SQL table (underlying CSV) and then writing it to a DELTA lake table.

+ +

I get a COMPLETE event type with an INPUT but no OUTPUT and then I get an exception for the AsyncEvent Queue but I'm guessing it's unrelated 😅

+ +

21/10/13 15:38:15 INFO OpenLineageContext: Lineage completed successfully: ResponseMessage(responseCode=200, body=null, error=null) {"eventType":"COMPLETE","eventTime":"2021-10-13T15:38:15.878Z","run":{"runId":"2cfe52b3-e08f-4888-8813-ffcdd2b27c89","facets":{"spark_unknown":{"_producer":"<https://github.com/OpenLineage/OpenLineage/tree/0.2.3-SNAPSHOT/integration/spark>","_schemaURL":"<https://openlineage.io/spec/1-0-2/OpenLineage.json#/$defs/RunFacet>","output":{"description":{"@class":"org.apache.spark.sql.catalyst.plans.logical.Project","traceEnabled":false,"streaming":false,"cacheId":null,"canonicalizedPlan":false},"inputAttributes":[{"name":"id","type":"long","metadata":{}}],"outputAttributes":[{"name":"id","type":"long","metadata":{}},{"name":"action_date","type":"date","metadata":{}}]},"inputs":[{"description":{"@class":"org.apache.spark.sql.catalyst.plans.logical.Range","streaming":false,"traceEnabled":false,"cacheId":null,"canonicalizedPlan":false},"inputAttributes":[],"outputAttributes":[{"name":"id","type":"long","metadata":{}}]}]},"spark.logicalPlan":{"_producer":"<https://github.com/OpenLineage/OpenLineage/tree/0.2.3-SNAPSHOT/integration/spark>","_schemaURL":"<https://openlineage.io/spec/1-0-2/OpenLineage.json#/$defs/RunFacet>","plan":[{"class":"org.apache.spark.sql.catalyst.plans.logical.Project","num-children":1,"projectList":[[{"class":"org.apache.spark.sql.catalyst.expressions.AttributeReference","num_children":0,"name":"id","dataType":"long","nullable":false,"metadata":{},"exprId":{"product_class":"org.apache.spark.sql.catalyst.expressions.ExprId","id":111,"jvmId":"4bdfd808-97d5-455f-ad6a-a3b29855e85b"},"qualifier":[]}],[{"class":"org.apache.spark.sql.catalyst.expressions.Alias","num-children":1,"child":0,"name":"action_date","exprId":{"product-class":"org.apache.spark.sql.catalyst.expressions.ExprId","id":113,"jvmId":"4bdfd808_97d5_455f_ad6a_a3b29855e85b"},"qualifier":[],"explicitMetadata":{},"nonInheritableMetadataKeys":"[__dataset_id, __col_position]"},{"class":"org.apache.spark.sql.catalyst.expressions.CurrentDate","num_children":0,"timeZoneId":"Etc/UTC"}]],"child":0},{"class":"org.apache.spark.sql.catalyst.plans.logical.Range","num-children":0,"start":0,"end":5,"step":1,"numSlices":8,"output":[[{"class":"org.apache.spark.sql.catalyst.expressions.AttributeReference","num_children":0,"name":"id","dataType":"long","nullable":false,"metadata":{},"exprId":{"product_class":"org.apache.spark.sql.catalyst.expressions.ExprId","id":111,"jvmId":"4bdfd808-97d5-455f-ad6a-a3b29855e85b"},"qualifier":[]}]],"isStreaming":false}]}}},"job":{"namespace":"sparknamespace","name":"databricks_shell.project"},"inputs":[],"outputs":[],"producer":"<https://github.com/OpenLineage/OpenLineage/tree/0.2.3-SNAPSHOT/integration/spark>","schemaURL":"<https://openlineage.io/spec/1-0-2/OpenLineage.json#/$defs/RunEvent>"} +21/10/13 15:38:16 INFO FileSizeAutoTuner: File size tuning result: {"tuningType":"autoTuned","tunedConfs":{"spark.databricks.delta.optimize.minFileSize":"268435456","spark.databricks.delta.optimize.maxFileSize":"268435456"}} +21/10/13 15:38:16 INFO FileFormatWriter: Write Job e062f36c-8b9d-4252-8db9-73b58bd67b15 committed. +21/10/13 15:38:16 INFO FileFormatWriter: Finished processing stats for write job e062f36c-8b9d-4252-8db9-73b58bd67b15. +21/10/13 15:38:18 INFO CodeGenerator: Code generated in 253.294028 ms +21/10/13 15:38:18 INFO SparkContext: Starting job: collect at DataSkippingReader.scala:430 +21/10/13 15:38:18 INFO DAGScheduler: Job 1 finished: collect at DataSkippingReader.scala:430, took 0.000333 s +21/10/13 15:38:18 ERROR AsyncEventQueue: Listener OpenLineageSparkListener threw an exception +java.lang.NullPointerException + at io.openlineage.spark.agent.OpenLineageSparkListener.onJobEnd(OpenLineageSparkListener.java:167) + at org.apache.spark.scheduler.SparkListenerBus.doPostEvent(SparkListenerBus.scala:39) + at org.apache.spark.scheduler.SparkListenerBus.doPostEvent$(SparkListenerBus.scala:28) + at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37) + at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37) + at org.apache.spark.util.ListenerBus.postToAll(ListenerBus.scala:119) + at org.apache.spark.util.ListenerBus.postToAll$(ListenerBus.scala:103) + at org.apache.spark.scheduler.AsyncEventQueue.super$postToAll(AsyncEventQueue.scala:105) + at org.apache.spark.scheduler.AsyncEventQueue.$anonfun$dispatch$1(AsyncEventQueue.scala:105) + at scala.runtime.java8.JFunction0$mcJ$sp.apply(JFunction0$mcJ$sp.java:23) + at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62) + at <a href="http://org.apache.spark.scheduler.AsyncEventQueue.org">org.apache.spark.scheduler.AsyncEventQueue.org</a>$apache$spark$scheduler$AsyncEventQueue$$dispatch(AsyncEventQueue.scala:100) + at org.apache.spark.scheduler.AsyncEventQueue$$anon$2.$anonfun$run$1(AsyncEventQueue.scala:96) + at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1547) + at org.apache.spark.scheduler.AsyncEventQueue$$anon$2.run(AsyncEventQueue.scala:96)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-10-13 17:54:22
 +
+

*Thread Reply:* This is because this specific action is not covered yet. You can see the “spark_unknown” facet is describing things that are not understood yet +run": { +... + "facets": { + "spark_unknown": { +... + "output": { + "description": { + "@class": "org.apache.spark.sql.catalyst.plans.logical.Project", + "traceEnabled": false, + "streaming": false, + "cacheId": null, + "canonicalizedPlan": false + },

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-10-13 17:54:43
 +
+

*Thread Reply:* I think this is part of the Spark 3 gap

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-10-13 17:55:46
 +
+

*Thread Reply:* an unknown output will cause missing output lineage

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-10-13 18:05:57
 +
+

*Thread Reply:* Output handling is here: https://github.com/OpenLineage/OpenLineage/blob/e0f1852422f325dc019b0eab0e466dc905[…]io/openlineage/spark/agent/lifecycle/OutputDatasetVisitors.java

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/e0f1852422f325dc019b0eab0e466dc905876a48/integration/spark/src/main/java/io/openlineage/spark/agent/lifecycle/OutputDatasetVisitors.java | OutputDatasetVisitors.java>
+
+ <pre><code> public List&amp;lt;PartialFunction&amp;lt;LogicalPlan, List&amp;lt;OpenLineage.OutputDataset&amp;gt;&amp;gt;&amp;gt; get() { +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+ 🙌 Will Johnson +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2021-10-13 22:49:08
 +
+

*Thread Reply:* Ah! Thank you so much, Julien! This is very helpful to understand where that is set. This is a big gap that we want to help address after our hackathon. Thank you!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-10-13 20:09:17
 +
+

Following up on the meeting this morning, I have created an issue to formalize a design doc review process: https://github.com/OpenLineage/OpenLineage/issues/336 +If that sounds good I’ll create the first doc to describe this as a PR. (how meta!)

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/issues/336|#336 [PROPOSAL] Add a design doc process for proposals>
+
+ <p><strong>Purpose:</strong> <br /> +Add a design doc section for proposals. The process would be the following:</p> + +<p>• open an issue with the "PROPOSAL" template +• if the scope warrants it create a PR to add a proposal doc. +• have the PR approved +• Once the proposal is accepted you can open another issue for the implementation.</p> + +<p><strong>Proposed implementation</strong></p> + +<p>• Proposals live under OpenLineage/OpenLineage/proposals +• one folder per design (includes images and the proposal document) +• the proposal is keyed by the github issue id: "/proposals/{github issue id}/{short description}.md" +• each doc has a header with: + • Status: APPROVED | IN PROGRESS | RELEASED + • Author: + • Created: + • issue tracker: #xxx +• assignees for reviews will be in the issue itself +• community can vote on the issue</p> + +<pre><code>--- +Status: Proposal +Author: +Created: +Issue: +--- + +</code></pre> +
+ +
+
Labels
+ proposal +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-10-13 20:13:02
 +
+

*Thread Reply:* the github wiki is backed by a git repo but it does not allow PRs. (people do hacks but I’d rather avoid those)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-10-18 10:24:25
 +
+

We're discussing creating Transport abstraction for OpenLineage clients, that would allow us creating better experience for people that expect to be able to emit their events using something else than http interface. Please tell us what you think of proposed mechanism - encouraging emojis are helpful too 😉 +https://github.com/OpenLineage/OpenLineage/pull/344

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-10-18 20:57:04
 +
+

OpenLineage release 0.3 is coming. Please chiming if there’s anything blocker that should go in the release: https://github.com/OpenLineage/OpenLineage/projects/4

+ + + +
+ ❤️ Willy Lulciuc +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Carlos Quintas + (cdquintas@gmail.com) +
+
2021-10-19 06:36:05
 +
+

👋 Hi everyone!

+ + + +
+ 👋 Ross Turk, Willy Lulciuc, Michael Collado +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Carlos Quintas + (cdquintas@gmail.com) +
+
2021-10-22 05:38:14
 +
+

openlineage with DBT and Trino, is there any forecast?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-10-22 05:44:17
 +
+

*Thread Reply:* Maybe you want to contribute it? +It's not that hard, mostly testing, and figuring out what would be the naming of openlineage namespace for Trino, and how some additional statistics work.

+ +

For example, recently we had added support for Redshift by community member @ale

+ +

https://github.com/OpenLineage/OpenLineage/pull/328

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/pull/328|#328 - Added support for Redshift profile>
+
+ <p>• Added support for Redshift profile +• Added support for handling env vars in profiles.yml</p> + +<p>Please note that I had to enclose <code>{{ config(...) }}</code> between <code>raw</code> Jinja tags, otherwise it complains about an unexpected '.'</p> + +<p>Signed-off-by: Alessandro Lollo <a href="mailto:alessandro.lollo@cloudacademy.com">alessandro.lollo@cloudacademy.com</a></p> +
+ +
+
Comments
+ 1 +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Carlos Quintas + (cdquintas@gmail.com) +
+
2021-10-22 05:42:52
 +
+

Done. PASS=5 WARN=0 ERROR=0 SKIP=0 TOTAL=5 +Traceback (most recent call last): + File "/home/labuser/.local/bin/dbt-ol", line 61, in <module> + main() + File "/home/labuser/.local/bin/dbt-ol", line 54, in main + events = processor.parse().events() + File "/home/labuser/.local/lib/python3.8/site-packages/openlineage/common/provider/dbt.py", line 98, in parse + self.extractdatasetnamespace(profile) + File "/home/labuser/.local/lib/python3.8/site-packages/openlineage/common/provider/dbt.py", line 377, in extractdatasetnamespace + self.datasetnamespace = self.extractnamespace(profile) + File "/home/labuser/.local/lib/python3.8/site-packages/openlineage/common/provider/dbt.py", line 391, in extract_namespace + raise NotImplementedError( +NotImplementedError: Only 'snowflake' and 'bigquery' adapters are supported right now. Passed trino

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2021-10-22 12:41:08
 +
+

Hey folks, we've released OpenLineage 0.3.1. There are quite a few changes, including doc improvements, Redshift support in dbt, bugfixes, a new server-side client code base, but the real highlights are

+ +
  1. Official Spark 3 support- this is still a work in progress (the whole Spark integration is), but the big deal is we've split the source tree to support both Spark 2 and Spark 3 specific plan visitors. This will enable us to work with the Spark 3 API explicitly and to add support for those interfaces and classes that didn't exist in Spark 2. We're also running all integration tests against both Spark 2.4.7 and Spark 3.1.0
  2. Airflow 2 support- also a work in progress, but we have a new LineageBackend implementation that allows us to begin tracking lineage for successful Airflow 2 DAGs. We're working to support failure notifications so we can also trace failed jobs. The LineageBackend can also be enabled in Airflow 1.10.X to improve the reporting of task completion times. +Check the READMEs for more details and to get started with the new features. Thanks to @Maciej Obuchowski , @Oleksandr Dvornik, @ale, and @Willy Lulciuc for their contributions. See the full changelog
    3. +
+ + + +
+ 🎉 Willy Lulciuc, Maciej Obuchowski, Minkyu Park, Ross Turk, Peter Hicks, RamanD, Ry Walker +
+ +
+ 🙌 Willy Lulciuc, Maciej Obuchowski, Minkyu Park, Will Johnson, Ross Turk, Peter Hicks, Ry Walker +
+ +
+ 🔥 Ry Walker +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
David Virgil + (david.virgil.naranjo@googlemail.com) +
+
2021-10-28 07:27:12
 +
+

Hello community. I am starting using marquez. I try to connect dbt with Marquez, but the spark adapter is not yet available.

+ +

Are you planning to implement this spark dbt adapter in next openlineage versions?

+ +

NotImplementedError: Only 'snowflake', 'bigquery', and 'redshift' adapters are supported right now. Passed spark +In my company we are starting to use as well the athena dbt adapter. Are you planning to implement this integration? Thanks a lot community

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-10-28 12:20:27
 +
+

*Thread Reply:* That would make sense. I think you are the first person to request this. Is this something you would want to contribute to the project?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
David Virgil + (david.virgil.naranjo@googlemail.com) +
+
2021-10-28 17:37:53
 +
+

*Thread Reply:* I would like to Julien, but not sure how can I do it. Could you guide me how can i start? or show me other integration.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Matthew Mullins + (mmullins@aginity.com) +
+
2021-10-31 07:57:55
 +
+

*Thread Reply:* @David Virgil look at the pull request for the addition of Redshift as a starting guide. https://github.com/OpenLineage/OpenLineage/pull/328

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/pull/328|#328 - Added support for Redshift profile>
+
+ <p>• Added support for Redshift profile +• Added support for handling env vars in profiles.yml</p> + +<p>Please note that I had to enclose <code>{{ config(...) }}</code> between <code>raw</code> Jinja tags, otherwise it complains about an unexpected '.'</p> + +<p>Signed-off-by: Alessandro Lollo <a href="mailto:alessandro.lollo@cloudacademy.com">alessandro.lollo@cloudacademy.com</a></p> +
+ +
+
Comments
+ 1 +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
David Virgil + (david.virgil.naranjo@googlemail.com) +
+
2021-11-01 12:01:41
 +
+

*Thread Reply:* Thanks @Matthew Mullins I ll try to add dbt spark integration

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mario Measic + (mario.measic.gavran@gmail.com) +
+
2021-10-28 09:31:01
 +
+

Hey folks, quick question, are we able to run dbt-ol without providing OPENLINEAGE_URL? I find it quite limiting that I need to have a service set up in order to emit/generate OL events/messages. Is there a way to just output them to the console?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mario Measic + (mario.measic.gavran@gmail.com) +
+
2021-10-28 10:05:09
 +
+

*Thread Reply:* OK, was changed here: https://github.com/OpenLineage/OpenLineage/pull/286

+ +

Did you think about this?

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/pull/286|#286 [INTEGRATION][dbt] fail execution if OPENLINEAGE_URL is unset>
+
+ <p>Currently, <code>dbt-ol</code> will run even if <code>OPENLINEAGE_URL</code> is unset. This causes nomal dbt execution, after which user sees that events weren't emitted.</p> + +<p>We should assume that variables/config should be set and valid - and fail the run if they aren't. After all, if someone wouldn't need lineage events, they wouldn't use our wrapper.</p> + +<p>When <a href="https://github.com/OpenLineage/OpenLineage/issues/287">OpenLineage/OpenLineage#287</a> is done, we should use emitting dbt parent run as way to check if URL is valid and reachable.</p> + +<p>This will require change with introduction of client <code>Backend</code> or <code>Transport</code> - we should check if any transport is configured correctly, instead of just simple http one.</p> + +<p>Signed-off-by: Maciej Obuchowski <a href="mailto:maciej.obuchowski@getindata.com">maciej.obuchowski@getindata.com</a></p> +
+ +
+
Comments
+ 1 +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-10-28 12:19:27
 +
+

*Thread Reply:* In Marquez there was a mechanism to do that. Something like OPENLINEAGE_BACKEND=HTTP|LOG

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-10-28 13:56:42
 +
+

*Thread Reply:* @Mario Measic We're going to add Transport mechanism, that will address use cases like yours. Please comment on this PR what would you expect: https://github.com/OpenLineage/OpenLineage/pull/344

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/pull/344|#344 [PROPOSAL] Transport proposal document>
+
+ <p>This document summarizes existing discussion on <code>Transport</code> abstraction from issue <a href="https://github.com/OpenLineage/OpenLineage/issues/269">OpenLineage/OpenLineage#269</a>, following method proposed here: <a href="https://github.com/OpenLineage/OpenLineage/issues/336">OpenLineage/OpenLineage#336</a></p> + +<p>Draft implementation of <code>Transport</code> abstraction for Python is here: <a href="https://github.com/OpenLineage/OpenLineage/pull/329">OpenLineage/OpenLineage#329</a></p> + +<p>Signed-off-by: Maciej Obuchowski <a href="mailto:maciej.obuchowski@getindata.com">maciej.obuchowski@getindata.com</a></p> +
+ +
+
Comments
+ 1 +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+ 👀 Mario Measic +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mario Measic + (mario.measic.gavran@gmail.com) +
+
2021-10-28 15:29:50
 +
+

*Thread Reply:* Nice, thanks @Julien Le Dem and @Maciej Obuchowski.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mario Measic + (mario.measic.gavran@gmail.com) +
+
2021-10-28 15:46:45
 +
+

*Thread Reply:* Also, dbt build is not working which is kind of the biggest feature of the version 0.21.0, I will try testing the code with modifications to the https://github.com/OpenLineage/OpenLineage/blob/c3aa70e161244091969951d0da4f37619bcbe36f/integration/dbt/scripts/dbt-ol#L141

+ +

I guess there's a reason for it that I didn't see since you support v3 of the manifest.

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/c3aa70e161244091969951d0da4f37619bcbe36f/integration/dbt/scripts/dbt-ol | dbt-ol>
+
+ <pre><code> if len(sys.argv) &amp;lt; 2 or sys.argv[1] not in ['run', 'test']: +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mario Measic + (mario.measic.gavran@gmail.com) +
+
2021-10-29 03:45:27
 +
+

*Thread Reply:* Also, is it normal not to see the column descriptions for the model/table even though these are provided in the YAML file, persisted in Redshift and also dbt docs generate has been run before dbt-ol run?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mario Measic + (mario.measic.gavran@gmail.com) +
+
2021-10-29 04:26:22
 +
+

*Thread Reply:* Tried with dbt versions 0.20.2 and 0.21.0, openlineage-dbt==0.3.1

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-10-29 10:39:10
 +
+

*Thread Reply:* I'll take a look at that. Supporting descriptions might be simple, but dbt build might be a little larger task.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-11-01 19:12:01
 +
+

*Thread Reply:* I opened a ticket to track this: https://github.com/OpenLineage/OpenLineage/issues/376

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/issues/376|#376 [INTEGRATION][dbt] support dbt built>
+
+ <p>dbt build is a superset of the dbt run and dbt test commands. <br /> +We should apply the same logic there. <br /> +<a href="https://docs.getdbt.com/reference/commands/build">https://docs.getdbt.com/reference/commands/build</a></p> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+ 👀 Mario Measic +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-11-02 05:48:06
 +
+

*Thread Reply:* The column description issue should be fixed here: https://github.com/OpenLineage/OpenLineage/pull/383

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/pull/383|#383 dbt: use column description from metadata.json even if we have catalog>
+
+ <p>Column descriptions are properly set in <code>metadata.json</code> file. Fetch it from there, even if we have <code>catalog.json</code>. <br /> +Also, remove empty column descriptions from being checked in most tests.</p> + +<p>Signed-off-by: Maciej Obuchowski <a href="mailto:maciej.obuchowski@getindata.com">maciej.obuchowski@getindata.com</a></p> + +<p>Closes: <a href="https://github.com/OpenLineage/OpenLineage/issues/382">OpenLineage/OpenLineage#382</a></p> + +<p><strong>Checklist</strong></p> + +<p>☑︎ You've <a href="https://github.com/OpenLineage/OpenLineage/blob/main/why-the-dco.md">signed-off</a> your work +☑︎ Your pull request title follows our <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CONTRIBUTING.md#creating-pull-requests">guidelines</a> +☑︎ Your changes are accompanied by tests (<em>if relevant</em>) +☑︎ Your change contains a <a href="https://kurtisnusbaum.medium.com/stacked-diffs-keeping-phabricator-diffs-small-d9964f4dcfa6">small diff</a> and is self-contained +☐ You've updated any relevant documentation (<em>if relevant</em>) +☐ You've updated the <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md"><code>CHANGELOG.md</code></a> with details about your change under the "Unreleased" section (<em>if relevant, depending on the change, this may not be necessary</em>) +☐ You've versioned the core OpenLineage model or facets according to <a href="https://docs.snowplowanalytics.com/docs/pipeline-components-and-applications/iglu/common-architecture/schemaver">SchemaVer</a> (<em>if relevant</em>)</p> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-10-28 12:27:17
 +
+

I’m looking for feedback on my proposal to improve the proposal process ! https://github.com/OpenLineage/OpenLineage/issues/336

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/issues/336|#336 [PROPOSAL] Add a design doc process for proposals>
+
+ <p><strong>Purpose:</strong> <br /> +Add a design doc section for proposals. The process would be the following:</p> + +<p>• open an issue with the "PROPOSAL" template +• if the scope warrants it create a PR to add a proposal doc. +• have the PR approved +• Once the proposal is accepted you can open another issue for the implementation.</p> + +<p><strong>Proposed implementation</strong></p> + +<p>• Proposals live under OpenLineage/OpenLineage/proposals +• one folder per design (includes images and the proposal document) +• the proposal is keyed by the github issue id: "/proposals/{github issue id}/{short description}.md" +• each doc has a header with: + • Status: APPROVED | IN PROGRESS | RELEASED + • Author: + • Created: + • issue tracker: #xxx +• assignees for reviews will be in the issue itself +• community can vote on the issue</p> + +<pre><code>--- +Status: Proposal +Author: +Created: +Issue: +--- + +</code></pre> +
+ +
+
Assignees
+ wslulciuc, mobuchowski, mandy-chessell, collado-mike +
+ +
+
Labels
+ proposal +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Brad + (bradley.mcelroy@live.com) +
+
2021-10-28 18:49:12
 +
+

Hey guys - just an update on my prefect PR (https://github.com/OpenLineage/OpenLineage/pull/293) - there a little spiel on the ticket but I've closed that PR in favour of opening a new one. Prefect have just release a 2.0a technical preview, which they would like to make stable near the start of next year. I think it makes sense to target this release, and I've had one of the prefect team reach out and is keen to get some sort of lineage implemented in prefect.

+ + + +
+ 👍 Kevin Kho, Maciej Obuchowski, Willy Lulciuc, Michael Collado, Julien Le Dem, Thomas Fredriksen +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Brad + (bradley.mcelroy@live.com) +
+
2021-10-28 18:51:10
 +
+

*Thread Reply:* If anyone has any questions or comments - happy to discuss here

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Brad + (bradley.mcelroy@live.com) +
+
2021-10-28 18:51:15
 +
+

*Thread Reply:* @davzucky

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2021-10-28 23:01:29
 +
+

*Thread Reply:* Thanks for updating the community, Brad!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
davzucky + (davzucky@hotmail.com) +
+
2021-10-28 23:47:02
 +
+

*Thread Reply:* Than you Brad. Looking forward to see how to integrated that with v2

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kevin Kho + (kdykho@gmail.com) +
+
2021-10-28 18:53:23
 +
+

Hello, joining here from Prefect. Because of community requests from users like Brad above, we are looking to implement lineage for Prefect this quarter. Good to meet you all!

+ + + +
+ ❤️ Minkyu Park, Faouzi, John Thomas, Maciej Obuchowski, Kevin Mellott, Thomas Fredriksen +
+ +
+ 👍 Minkyu Park, Faouzi, John Thomas +
+ +
+ 🙌 Michael Collado, Faouzi, John Thomas +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2021-10-28 18:54:56
 +
+

*Thread Reply:* Welcome, @Kevin Kho 👋. Really excited to see this integration kick off! 💯🚀

+ + + +
+ 👍 Kevin Kho, Maciej Obuchowski, Peter Hicks, Faouzi +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
David Virgil + (david.virgil.naranjo@googlemail.com) +
+
2021-11-01 12:03:14
 +
+

Hello,

+ +

i am integratng openLineage with Airflow 2.2.0

+ +

Do you consider in the future airflow manual inlets and outlets?

+ +

Seeing the documentation I can see that is not possible.

+ +

OpenLineageBackend does not take into account manually configured inlets and outlets. +Thanks

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2021-11-01 12:23:11
 +
+

*Thread Reply:* While it’s not something we’re supporting at the moment, it’s definitely something that we’re considering!

+ +

If you can give me a little more detail on what your system infrastructure is like, it’ll help us set priority and design

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
David Virgil + (david.virgil.naranjo@googlemail.com) +
+
2021-11-01 13:57:34
 +
+

*Thread Reply:* So basic architecture of a datalake. We are using airflow to trigger jobs. Every job is a pipeline that runs a spark job (in our case it spin up an EMR). So the idea of lineage would be defining in the dags inlets and outlets based on the airflow lineage:

+ +

https://airflow.apache.org/docs/apache-airflow/stable/lineage.html

+ +

I think you need to be able to include these inlets and outlets in the picture of openlineage

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-11-01 14:01:24
 +
+

*Thread Reply:* Why not use spark integration? https://github.com/OpenLineage/OpenLineage/tree/main/integration/spark

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
David Virgil + (david.virgil.naranjo@googlemail.com) +
+
2021-11-01 14:05:02
 +
+

*Thread Reply:* because there are some other jobs that are not spark, some jobs they run in dbt, other jobs they run in redshift @Maciej Obuchowski

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-11-01 14:08:58
 +
+

*Thread Reply:* So, combo of https://github.com/OpenLineage/OpenLineage/tree/main/integration/dbt and PostgresExtractor from airflow integration should cover Redshift if you're using it from PostgresOperator 🙂

+ +

It's definitely interesting use case - you'd be using most of the existing integrations we have.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
David Virgil + (david.virgil.naranjo@googlemail.com) +
+
2021-11-01 15:04:44
 +
+

*Thread Reply:* @Maciej Obuchowski Do i need to define any extractor in the airflow startup?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Francis McGregor-Macdonald + (francis@mc-mac.com) +
+
2021-11-05 23:48:21
 +
+

*Thread Reply:* I am using Redshift with PostgresOperator and it is returning…

+ +

[2021-11-06 03:43:06,541] {{__init__.py:92}} ERROR - Failed to extract metadata 'NoneType' object has no attribute 'host' task_type=PostgresOperator airflow_dag_id=counter task_id=inc airflow_run_id=scheduled__2021-11-06T03:42:00+00:00 +Traceback (most recent call last): + File "/usr/local/airflow/.local/lib/python3.7/site-packages/openlineage/lineage_backend/__init__.py", line 83, in _extract_metadata + task_metadata = self._extract(extractor, task_instance) + File "/usr/local/airflow/.local/lib/python3.7/site-packages/openlineage/lineage_backend/__init__.py", line 104, in _extract + task_metadata = extractor.extract_on_complete(task_instance) + File "/usr/local/airflow/.local/lib/python3.7/site-packages/openlineage/airflow/extractors/base.py", line 61, in extract_on_complete + return self.extract() + File "/usr/local/airflow/.local/lib/python3.7/site-packages/openlineage/airflow/extractors/postgres_extractor.py", line 65, in extract + authority=self._get_authority(), + File "/usr/local/airflow/.local/lib/python3.7/site-packages/openlineage/airflow/extractors/postgres_extractor.py", line 120, in _get_authority + if self.conn.host and self.conn.port: +AttributeError: 'NoneType' object has no attribute 'host'

+ +

I can’t see this raised as an issue.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
David Virgil + (david.virgil.naranjo@googlemail.com) +
+
2021-11-01 13:57:54
 +
+

Hello, I am trying to integrate Airflow with openlineage.

+ +

It is not working for me.

+ +

What I tried:

+ +
  1. Adding openlineage-airflow to requirements.txt
  2. Adding +```- AIRFLOWLINEAGEBACKEND=openlineage.airflow.backend.OpenLineageBackend
    3. +
+ +

During handling of the above exception, another exception occurred:

+ +

Traceback (most recent call last): + File "/home/airflow/.local/bin/airflow", line 8, in <module> + sys.exit(main()) + File "/home/airflow/.local/lib/python3.8/site-packages/airflow/main.py", line 40, in main + args.func(args) + File "/home/airflow/.local/lib/python3.8/site-packages/airflow/cli/cliparser.py", line 47, in command + func = importstring(importpath) + File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/moduleloading.py", line 32, in importstring + module = importmodule(modulepath) + File "/usr/local/lib/python3.8/importlib/init.py", line 127, in importmodule + return bootstrap.gcdimport(name[level:], package, level) + File "<frozen importlib.bootstrap>", line 1014, in gcdimport + File "<frozen importlib.bootstrap>", line 991, in _findandload + File "<frozen importlib.bootstrap>", line 975, in findandloadunlocked + File "<frozen importlib.bootstrap>", line 671, in _loadunlocked + File "<frozen importlib.bootstrapexternal>", line 843, in execmodule + File "<frozen importlib.bootstrap>", line 219, in callwithframesremoved + File "/home/airflow/.local/lib/python3.8/site-packages/airflow/cli/commands/dbcommand.py", line 24, in <module> + from airflow.utils import cli as cliutils, db + File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/db.py", line 26, in <module> + from airflow.jobs.basejob import BaseJob # noqa: F401 + File "/home/airflow/.local/lib/python3.8/site-packages/airflow/jobs/init.py", line 19, in <module> + import airflow.jobs.backfilljob + File "/home/airflow/.local/lib/python3.8/site-packages/airflow/jobs/backfilljob.py", line 29, in <module> + from airflow import models + File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/init.py", line 20, in <module> + from airflow.models.baseoperator import BaseOperator, BaseOperatorLink + File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/baseoperator.py", line 196, in <module> + class BaseOperator(Operator, LoggingMixin, TaskMixin, metaclass=BaseOperatorMeta): + File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/baseoperator.py", line 941, in BaseOperator + def postexecute(self, context: Any, result: Any = None): + File "/home/airflow/.local/lib/python3.8/site-packages/airflow/lineage/init.py", line 103, in applylineage + _backend = getbackend() + File "/home/airflow/.local/lib/python3.8/site-packages/airflow/lineage/init.py", line 52, in get_backend + clazz = conf.getimport("lineage", "backend", fallback=None) + File "/home/airflow/.local/lib/python3.8/site-packages/airflow/configuration.py", line 469, in getimport + raise AirflowConfigException( +airflow.exceptions.AirflowConfigException: The object could not be loaded. Please check "backend" key in "lineage" section. Current value: "openlineage.airflow.backend.OpenLineageBackend".```

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-11-01 14:06:12
 +
+

*Thread Reply:* 1. Please use openlineage.lineage_backend.OpenLineageBackend as AIRFLOW__LINEAGE__BACKEND

+ +
  1. Please tell us where you've seen openlineage.airflow.backend.OpenLineageBackend, so we can fix the documentation 🙂
    2. +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-11-01 19:07:21
 +
+

*Thread Reply:* https://pypi.org/project/openlineage-airflow/

+
+
PyPI
+ + + +
openlineage-airflow
+
+ OpenLineage integration with Airflow +
+ + + + + + + + +
Original URL: https://pypi.org/project/openlineage-airflow/
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-11-01 19:08:03
 +
+

*Thread Reply:* (I googled it and found that page that seems to have an outdated doc)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
David Virgil + (david.virgil.naranjo@googlemail.com) +
+
2021-11-02 02:38:59
 +
+

*Thread Reply:* @Maciej Obuchowski +@Julien Le Dem that's the page i followed. Please guys revise the documentation, as it is very important

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-11-02 04:34:14
 +
+

*Thread Reply:* It should just copy actual readme

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/integration/airflow/setup.py | setup.py>
+
+ <pre><code> long_description=readme, +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2021-11-03 16:30:00
 +
+

*Thread Reply:* PyPi is using the README at the time of the release 0.3.1, rather than the current README, which is 0.4.0. If we send the new release to PyPi it should also update the README

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
David Virgil + (david.virgil.naranjo@googlemail.com) +
+
2021-11-01 15:09:54
 +
+

Related the Airflow integration. Is it required to install openlineage-airflow and setup the environment variables in both scheduler and webserver, or just in the scheduler?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
David Virgil + (david.virgil.naranjo@googlemail.com) +
+
2021-11-01 15:19:18
 +
+

*Thread Reply:* I set i up in the scheduler and it starts to log data to marquez. But it fails with this error:

+ +

Traceback (most recent call last): + File "/home/airflow/.local/lib/python3.8/site-packages/openlineage/client/client.py", line 49, in __init__ + raise ValueError(f"Need valid url for OpenLineageClient, passed {url}") +ValueError: Need valid url for OpenLineageClient, passed "<http://marquez-internal-eks.eu-west-1.dev.hbi.systems>"

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
David Virgil + (david.virgil.naranjo@googlemail.com) +
+
2021-11-01 15:19:26
 +
+

*Thread Reply:* why is it not a valid URL?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2021-11-01 18:39:58
 +
+

*Thread Reply:* Which version of the OpenLineage client are you using? On first check it should be fine

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
David Virgil + (david.virgil.naranjo@googlemail.com) +
+
2021-11-02 05:14:30
 +
+

*Thread Reply:* @John Thomas I was appending double quotes as part of the url. Forget about this error

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2021-11-02 10:35:28
 +
+

*Thread Reply:* aaaah, gotcha, good catch!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
David Virgil + (david.virgil.naranjo@googlemail.com) +
+
2021-11-02 05:15:52
 +
+

Hello, I am receiving this error today when I deployed openlineage in development environment (not using docker-compose locally).

+ +

I am running with KubernetesExecutor

+ +

airflow.exceptions.AirflowConfigException: The object could not be loaded. Please check "backend" key in "lineage" section. Current value: "openlineage.lineage_backend.OpenLineageBackend".

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-11-02 05:18:18
 +
+

*Thread Reply:* Are you sure that openlineage-airflow is present in the container?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
David Virgil + (david.virgil.naranjo@googlemail.com) +
+
2021-11-02 05:23:09
 +
+

So in this case in my template I am adding:

+ +

```env:
+ ADDITIONALPYTHONDEPS: "openpyxl==3.0.3 smartopen==2.0.0 apache-airflow-providers-http apache-airflow-providers-cncf-kubernetes apache-airflow-providers-amazon openlineage-airflow" + OPENLINEAGEURL: https://marquez-internal-eks.eu-west-1.dev.hbi.systems + OPENLINEAGENAMESPACE: dnsairflow + AIRFLOWKUBERNETESENVIRONMENTVARIABLESOPENLINEAGEURL: https://marquez-internal-eks.eu-west-1.dev.hbi.systems + AIRFLOWKUBERNETESENVIRONMENTVARIABLESOPENLINEAGENAMESPACE: dns_airflow

+ +

configmap: + mountPath: /var/airflow/config # mount path of the configmap + data: + airflow.cfg: | + [lineage] + backend = openlineage.lineage_backend.OpenLineageBackend

+ +
pod_template_file.yaml: |
+
+    containers:
+      - args: []
+        command: []
+        env:
+          - name: AIRFLOW__KUBERNETES_ENVIRONMENT_VARIABLES__OPENLINEAGE_URL
+            value: <https://marquez-internal-eks.eu-west-1.dev.hbi.systems>
+          - name: AIRFLOW__KUBERNETES_ENVIRONMENT_VARIABLES__OPENLINEAGE_NAMESPACE
+            value: dns_airflow
+          - name: AIRFLOW__LINEAGE__BACKEND
+            value: openlineage.lineage_backend.OpenLineageBackend```
+
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
David Virgil + (david.virgil.naranjo@googlemail.com) +
+
2021-11-02 05:23:31
 +
+

I am installing openlineage in the ADDITIONAL_PYTHON_DEPS

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-11-02 05:25:43
 +
+

*Thread Reply:* Maybe ADDITIONAL_PYTHON_DEPS are dependencies needed by the tasks, and are installed after Airflow tries to initialize LineageBackend?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
David Virgil + (david.virgil.naranjo@googlemail.com) +
+
2021-11-02 06:34:11
 +
+

*Thread Reply:* I am checking this accessing the Kubernetes pod

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
David Virgil + (david.virgil.naranjo@googlemail.com) +
+
2021-11-02 06:34:54
 +
+

I have a question related airflow and open lineage:

+ +

I have a dag that contains 2 tasks:

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
David Virgil + (david.virgil.naranjo@googlemail.com) +
+
2021-11-02 06:35:34
 +
+

I see that every task is displayed as a different job. I was expecting to see one job per dag.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
David Virgil + (david.virgil.naranjo@googlemail.com) +
+
2021-11-02 07:29:43
 +
+

Is this the expected behaviour??

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-11-02 07:34:47
 +
+

*Thread Reply:* Yes

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-11-02 07:35:53
 +
+

*Thread Reply:* Probably what you want is job hierarchy: https://github.com/MarquezProject/marquez/issues/1737

+
+ + + + +
<https://github.com/MarquezProject/marquez/issues/1737|#1737 Implement job hierarchy based on parent job>
+
+ <p>jobs should have a name that is relative to the parent job facet when created through OpenLineage events.</p> +
+ +
+
Assignees
+ collado-mike +
+ + + + + + + +
+ + <a href="https://github.com/MarquezProject/marquez">MarquezProject/marquez</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
David Virgil + (david.virgil.naranjo@googlemail.com) +
+
2021-11-02 07:46:02
 +
+

*Thread Reply:* I do not see any benefit of just having some airflow task metadata. I do not see relationship between tasks. Every task is a job. When I was thinking about lineage when i started working on my company integration with openlineage i though that openlineage would give me relationship between task or datasets and the only thing i see is some metadata of the history of airflow runs that is already provided by airflow

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
David Virgil + (david.virgil.naranjo@googlemail.com) +
+
2021-11-02 07:46:20
 +
+

*Thread Reply:* i was expecting to see a nice graph. I think it is missing some features

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
David Virgil + (david.virgil.naranjo@googlemail.com) +
+
2021-11-02 07:46:25
 +
+

*Thread Reply:* at this early stage

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-11-02 07:50:10
 +
+

*Thread Reply:* It probably depends on whether those tasks are covered by the extractors: https://github.com/OpenLineage/OpenLineage/tree/main/integration/airflow/openlineage/airflow/extractors

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
David Virgil + (david.virgil.naranjo@googlemail.com) +
+
2021-11-02 07:55:50
 +
+

*Thread Reply:* We are not using any of those operators: bigquery, postsgress or snowflake.

+ +

And what is it doing GreatExpectactions extractor?

+ +

It would be good if there is one extractor that relies in the inlets and outlets that you can define in any Airflow task, and that that can be the general way to make relationships between datasets

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
David Virgil + (david.virgil.naranjo@googlemail.com) +
+
2021-11-02 07:56:30
 +
+

*Thread Reply:* And that the same dag graph can be seen in marquez, and not one job per task.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-11-02 08:07:06
 +
+

*Thread Reply:* > It would be good if there is one extractor that relies in the inlets and outlets that you can define in any Airflow task +I think this is good idea. Overall, OpenLineage strongly focuses on automatic metadata collection. However, using them would be a nice fallback for not-covered-yet cases.

+ +

> And that the same dag graph can be seen in marquez, and not one job per task. +This currently depends on dataset hierarchy. If you're not using any of the covered extractors, then Marquez can't build dataset graph like in the demo: https://raw.githubusercontent.com/MarquezProject/marquez/main/web/docs/demo.gif

+ +

With the job hierarchy ticket, probably some graph could be generated using just the job data though.

+
+ + + + +
+
+ +
+ + + + + + + + +
Original URL: https://raw.githubusercontent.com/MarquezProject/marquez/main/web/docs/demo.gif
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-11-02 08:09:55
 +
+

*Thread Reply:* Created issue for the manual fallback: https://github.com/OpenLineage/OpenLineage/issues/384

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/issues/384|#384 [INTEGRATION][Airflow] Consider using manual inlets/outlets as a fallback for automatic extractor mechanism>
+
+ <p>There are way more operators than we can reasonably cover. Consider adding manual fallback mechanism that would cover cases where we don't have any extractors.</p> + +<p>Advantage of this approach is that it's generally easier for users to annotate dag than to modify it to use OpenLineage client manually.</p> +
+ +
+
Assignees
+ mobuchowski +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
David Virgil + (david.virgil.naranjo@googlemail.com) +
+
2021-11-02 08:28:29
 +
+

*Thread Reply:* @Maciej Obuchowski how many people are working full time in this library? I really would like to adopt it in my company, as we use airflow and spark, but i see that yet it does not have the features we would like to.

+ +

At the moment the same info we have in marquez related the tasks, is available in airflow UI or using airflow API.

+ +

The game changer for us would be that it could give us features/metadata that we cannot query directly from airflow. That's why if the airflow inlets/outlets could be used, then it really would make much more sense for us to adopt it.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-11-02 09:33:31
 +
+

*Thread Reply:* > how many people are working full time in this library? +On Airflow integration or on OpenLineage overall? 🙂

+ +

> The game changer for us would be that it could give us features/metadata that we cannot query directly from airflow. +I think there are three options there:

+ +
  1. Contribute relevant extractors for Airflow operators that you use
  2. Use those extractors as custom extractors: https://github.com/OpenLineage/OpenLineage/tree/main/integration/airflow#custom-extractors
  3. Create that manual fallback mechanism with Airflow inlets/outlets: https://github.com/OpenLineage/OpenLineage/issues/384
    4. +
+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/issues/384|#384 [INTEGRATION][Airflow] Consider using manual inlets/outlets as a fallback for automatic extractor mechanism>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-11-02 09:35:10
 +
+

*Thread Reply:* But first, before implementing last option, I'd like to get consensus about it - so feel free to comment there about your use case

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
David Virgil + (david.virgil.naranjo@googlemail.com) +
+
2021-11-02 09:19:14
 +
+

@Maciej Obuchowski even i can contribute or help with my ideas (from what i consider that should be lineage from a client side)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
David Virgil + (david.virgil.naranjo@googlemail.com) +
+
2021-11-03 07:58:56
 +
+

@Maciej Obuchowski I was able to put to work Airflow in Kubernetes pointing to Marquez using the openlineage library. I have a few problems I found that would be good to comment.

+ +

I see a warning +[2021-11-03 11:47:04,309] {great_expectations_extractor.py:27} WARNING - Did not find great_expectations_provider library or failed to import it +I couldnt find any information about GreatExpectationsExtractor. Could you tell me what is this extractor about?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-11-03 08:00:34
 +
+

*Thread Reply:* It should only affect you if you're using https://greatexpectations.io/

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Francis McGregor-Macdonald + (francis@mc-mac.com) +
+
2021-11-03 15:57:02
 +
+

*Thread Reply:* I have a similar message after installing openlineage into Amazon MWAA from the scheduler logs:

+ +

WARNING:/usr/local/airflow/.local/lib/python3.7/site-packages/openlineage/airflow/extractors/great_expectations_extractor.py:Did not find great_expectations_provider library or failed to import it

+ +

I am not using great expectations in the DAG.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
David Virgil + (david.virgil.naranjo@googlemail.com) +
+
2021-11-03 08:00:52
 +
+

I see a few priorities for Airflow integration:

+ +
  1. Direct relationship 1-1 between Dag && Job. At the moment every task is a different job in marquez. What i consider wrong.
  2. Airflow Inlets/outlets integration with marquez +When do you think you guys can have this? If you need any help I can happily contribute, but I would need some help
    3. +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-11-03 08:08:21
 +
+

*Thread Reply:* I don't think 1) is a good idea. You can have multiple tasks in one dag, processing different datasets and producing different datasets. If you want visual linking of jobs that produce disjoint datasets, then I think you want this: https://github.com/MarquezProject/marquez/issues/1737 +which wuill affect visual layer.

+ +

Regarding 2), I think we need to get along with Airflow maintainers regarding long term mechanism on which OL will work: https://github.com/apache/airflow/issues/17984

+ +

I think using inlets/outlets as a fallback mechanism when we're not doing automatic metadata extraction is a good idea, but we don't know if hypothetical future mechanism will have access to these. It's hard to commit to mechanism which might disappear soon.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
David Virgil + (david.virgil.naranjo@googlemail.com) +
+
2021-11-03 08:13:28
 +
+

Another option is that I build my own extractor, do you have any example of how to create a custom extractor? How I can apply that customExtractor to specific operators? Is there a way to link an extractor with an operator, so at runtime airflow knows which extractor to run?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-11-03 08:19:00
 +
+

*Thread Reply:* https://github.com/OpenLineage/OpenLineage/tree/main/integration/airflow#custom-extractors

+ +

I think you can base your code on any existing extractor, like PostgresExtractor: https://github.com/OpenLineage/OpenLineage/blob/main/integration/airflow/openlineage/airflow/extractors/postgres_extractor.py#L53

+ +

Custom extractors work just like buildin ones, just that you need to add bit of mapping between operator and extractor, like OPENLINEAGE_EXTRACTOR_PostgresOperator=openlineage.airflow.extractors.postgres_extractor.PostgresExtractor

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/integration/airflow/openlineage/airflow/extractors/postgres_extractor.py | postgres_extractor.py>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+ 👍 Francis McGregor-Macdonald +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
David Virgil + (david.virgil.naranjo@googlemail.com) +
+
2021-11-03 08:35:59
 +
+

*Thread Reply:* Thank you very much @Maciej Obuchowski

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
David Virgil + (david.virgil.naranjo@googlemail.com) +
+
2021-11-03 08:36:52
 +
+

Last question of the morning. Running one task that failed i could see that no information appeared in Marquez. Is this something that is expected to happen? I would like to see in Marquez all the history of runs, successful and unsucessful them.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-11-03 08:41:14
 +
+

*Thread Reply:* It worked like that in Airflow 1.10.

+ +

This is an unfortunate limitation of LineageBackend API that we're using for Airflow 2. We're trying to work out solution for this with Airflow maintainers: https://github.com/apache/airflow/issues/17984

+
+ + + + +
<https://github.com/apache/airflow/issues/17984|#17984 Add possibility to LineageBackend to be notified of task instance execution start and failure>
+
+ <p><strong>Description</strong></p> + +<p>Provide way for <code>LineageBackend</code> to be notified of task instance start and failed execution.</p> + +<p>Proposed implementation:</p> + +<ol><li>add start and fail methods to LineageBackend</li><li>prepare<em>lineage decorator should call LineageBackend's start method the same way that apply</em>lineage calls send_lineage.</li><li>call fail method on where on<em>failure</em>callback is called.</li> +</ol> + +<p><strong>Use case/motivation</strong></p> + +<p>I'm working on <a href="https://openlineage.io/">OpenLineage</a> integration for Airflow 2. <a href="https://github.com/OpenLineage/OpenLineage/blob/airflow/2/integration/airflow/openlineage/airflow/backend.py">I've created MVP of this.</a></p> + +<p>In contrast to our integration for Airflow 1.10 done via subclassing DAG, <code>LineageBackend</code> provides easy way to integrate OpenLineage just via configuration.</p> + +<p>However, <code>LineageBackend.send_lineage</code> is called just on successful task execution, while we also want to track newly started and failed tasks.</p> + +<p><strong>Related issues</strong></p> + +<p><em>No response</em></p> + +<p><strong>Are you willing to submit a PR?</strong></p> + +<p>☑︎ Yes I am willing to submit a PR!</p> + +<p><strong>Code of Conduct</strong></p> + +<p>☑︎ I agree to follow this project's <a href="https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md">Code of Conduct</a></p> +
+ +
+
Labels
+ kind:feature, area:lineage +
+ +
+
Comments
+ 23 +
+ + + + + + + +
+ + <a href="https://github.com/apache/airflow">apache/airflow</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
David Virgil + (david.virgil.naranjo@googlemail.com) +
+
2021-11-04 03:41:38
 +
+

Hello openlineage community.

+ +

Yesterday I tried the integration with spark.

+ +

The result was not satisfactory. This is what I did:

+ +
  1. Add openlineage-spark dependency
  2. Add these lines: +.config("spark.jars.packages", "io.openlineage:openlineage_spark:0.3.1") +.config("spark.extraListeners", "io.openlineage.spark.agent.OpenLineageSparkListener") +.config("spark.openlineage.url", "<https://marquez-internal-eks.eu-west-1.dev.hbi.systems/api/v1/namespaces/spark_integration/>" +This job was doing spark.read from 2 different json location. +It is doing spark write to 5 different parquet location in s3. +The job finished succesfully and the result in marquez is:
    3. +
+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
David Virgil + (david.virgil.naranjo@googlemail.com) +
+
2021-11-04 03:43:40
 +
+

It created 3 namespaces. One was the one that I point in the spark config property. The other 2 are the bucket that we are writing to () and the bucket where we are reading from ()

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
David Virgil + (david.virgil.naranjo@googlemail.com) +
+
2021-11-04 03:44:00
 +
+

If I enter in the bucket namespaces I see nowthing inside

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
David Virgil + (david.virgil.naranjo@googlemail.com) +
+
2021-11-04 03:48:35
 +
+

I can see if i enter in one of the weird jobs generated this:

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-11-04 18:47:41
 +
+

*Thread Reply:* This job with no output is a symptom of the output not being understood. you should be able to see the facets for that job. There will be a spark_unknown facet with more information about the problem. If you put that into an issue with some more details about this job we should be able to help.

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
David Virgil + (david.virgil.naranjo@googlemail.com) +
+
2021-11-05 04:36:30
 +
+

*Thread Reply:* I ll try to put all the info in a ticket, as it is not working as i would expect

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
David Virgil + (david.virgil.naranjo@googlemail.com) +
+
2021-11-04 03:52:24
 +
+

And i am seeing this as well

+ +

If I check the logs of marquez-web and marquez I can't see any error there

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
David Virgil + (david.virgil.naranjo@googlemail.com) +
+
2021-11-04 03:54:38
 +
+

When I try to open the job fulfilments.execute_insert_into_hadoop_fs_relation_command I see this window:

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
David Virgil + (david.virgil.naranjo@googlemail.com) +
+
2021-11-04 04:06:29
 +
+

The page froze and no link from the menu works. Apart from that I see that there are no messages in the logs

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-11-04 18:49:31
 +
+

*Thread Reply:* Is there an error in the browser javascript console? (example on chrome: View -> Developer -> Javascript console)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Alessandro Rizzo + (l.alessandrorizzo@gmail.com) +
+
2021-11-04 17:22:29
 +
+

Hi #general, I'm a data engineer for a UK-based insuretech (part of one of the biggest UK retail insurers). We run a series of tech meetups and we'd love to have someone from the OpenLineage project to give us a demo of the tool. Would anyone be interested (DM if so 🙂 ) ?

+ + + +
+ 👍 Ross Turk +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Taleb Zeghmi + (talebz@zillowgroup.com) +
+
2021-11-04 21:30:24
 +
+

Hi! Is there an example of tracking lineage when using Pandas to read/write and transform data?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2021-11-04 21:35:16
 +
+

*Thread Reply:* Hi Taleb - I don’t know of a generalized example of lineage tracking with Pandas, but you should be able to accomplish this by sending the runEvents manually to the OpenLineage API in your code: +https://openlineage.io/docs/openapi/

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Taleb Zeghmi + (talebz@zillowgroup.com) +
+
2021-11-04 21:38:25
 +
+

*Thread Reply:* Is this a work in progress, that we can investigate? Because I see it in this image https://github.com/OpenLineage/OpenLineage/blob/main/doc/Scope.png

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/doc/Scope.png | Scope.png>
+
+ +
+ + + + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2021-11-04 21:54:51
 +
+

*Thread Reply:* To my knowledge, while there are a few proposals around adding a wrapper on some Pandas methods to output runEvents, it’s not something that’s had work started on it yet

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2021-11-04 21:56:26
 +
+

*Thread Reply:* I sent some feelers out to get a little more context from folks who are more informed about this than I am, so I’ll get you more info about potential future plans and the considerations around them when I know more

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2021-11-04 23:04:47
 +
+

*Thread Reply:* So, Pandas is tricky because unlike Airflow, DBT, or Spark, Pandas doesn’t own the whole flow, and you might dip in and out of it to use other Python Packages (at least I did when I was doing more Data Science).

+ +

We have this issue open in OpenLineage that you should go +1 to help with our planning 🙂

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/issues/108|#108 [INTEGRATION] Add support for `pandas.read_sql`>
+
+ <p>Let's add support for the Panda wrapper method <a href="https://pandas.pydata.org/docs/reference/api/pandas.read_sql.html"><code>read_sql</code></a>:</p> + +<pre><code>pandas.read_sql(sql, con, index_col=None, coerce_float=True, + params=None, parse_dates=None, columns=None, chunksize=None) + +</code></pre> + +<p>Since we have access to the <code>sql</code> at time of method execution, we can easily parse the SQL to capture the in / out tables before sending an OpenLineage event. We also have access to <code>columns</code> (if present) on table reads and may also want to support <code>read_sql_table</code> and <code>read_sql_query</code> (for backward compatibility).</p> +
+ +
+
Labels
+ proposal +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Taleb Zeghmi + (talebz@zillowgroup.com) +
+
2021-11-05 15:08:09
 +
+

*Thread Reply:* interesting... what if it were instead on all the read_** to_** functions?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Lyndon Armitage + (lyndon.armitage@gmail.com) +
+
2021-11-05 12:00:57
 +
+

Hi! I am working alongside David at integrating OpenLineage into our Data Pipelines. I have a questions around Marquez and OpenLineage's divergent APIs: +That is to say, these 2 APIs differ: +https://openlineage.io/docs/openapi/ +https://marquezproject.github.io/marquez/openapi.html +This makes sense since they are at different layers of abstraction, but Marquez requires a few things that are absent from OpenLineage's API, for example the type in a data source, the distinctions between physicalName and sourceName in Datasets. Is that intentional? And can these be set using the OpenLineage API as some additional facets or keys? I noticed that the DatasourceDatasetFacet has a map of additionalProperties .

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2021-11-05 12:59:49
 +
+

*Thread Reply:* The Marquez write APIs are artifacts from before OpenLineage existed, and they’re already slated for deprecation soon.

+ +

If you POST an OpenLineage runEvent to the /lineage endpoint in Marquez, it’ll create any missing jobs or datasets that are relevant.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Lyndon Armitage + (lyndon.armitage@gmail.com) +
+
2021-11-05 13:06:06
 +
+

*Thread Reply:* Thanks for the response. That sounds good. Does this include the query interface e.g. +http://localhost:5000/api/v1/namespaces/testing_java/datasets/incremental_data +as that currently returns the Marquez version of a dataset including default set fields for type and the above mentioned properties.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2021-11-05 17:01:55
 +
+

*Thread Reply:* I believe the intention for type is to support a new facet- TBH, it hasn't been the most pressing concern for most users, as most people are only recording tables, not streams. However, there's been some recent work to support Kafka in Spark- maybe it's time to address that deficiency.

+ +

I don't actually know what happened to the datasource type field- maybe @Julien Le Dem can comment on whether that field was dropped intentionally or whether it was an oversight.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-11-05 18:18:06
 +
+

*Thread Reply:* It looks like an oversight, currently Marquez hard codes it to POSGRESQL: https://github.com/MarquezProject/marquez/blob/734bfd691636cb00212d7d22b1a489bd4870fb04/api/src/main/java/marquez/db/OpenLineageDao.java#L438

+
+ + + + +
<https://github.com/MarquezProject/marquez/blob/734bfd691636cb00212d7d22b1a489bd4870fb04/api/src/main/java/marquez/db/OpenLineageDao.java | OpenLineageDao.java>
+
+ <pre><code> default String getSourceType(Dataset ds) { +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/MarquezProject/marquez">MarquezProject/marquez</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-11-05 18:18:25
 +
+

*Thread Reply:* https://github.com/MarquezProject/marquez/blob/734bfd691636cb00212d7d22b1a489bd4870fb04/api/src/main/java/marquez/db/OpenLineageDao.java#L438-L440

+
+ + + + +
<https://github.com/MarquezProject/marquez/blob/734bfd691636cb00212d7d22b1a489bd4870fb04/api/src/main/java/marquez/db/OpenLineageDao.java | OpenLineageDao.java>
+
+ <pre><code> default String getSourceType(Dataset ds) { + return SourceType.of("POSTGRESQL").getValue(); + } +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/MarquezProject/marquez">MarquezProject/marquez</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-11-05 18:20:25
 +
+

*Thread Reply:* The source has a name though: https://github.com/OpenLineage/OpenLineage/blob/8afc4ff88b8dd8090cd9c45061a9f669fea2151e/spec/facets/DatasourceDatasetFacet.json#L12

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/8afc4ff88b8dd8090cd9c45061a9f669fea2151e/spec/facets/DatasourceDatasetFacet.json | DatasourceDatasetFacet.json>
+
+ <pre><code> "type" : "string" +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-11-05 18:07:16
 +
+

The next OpenLineage monthly meeting is this coming Wednesday at 9am PT +The tentative agenda is: +• OL Client use cases for Apache Iceberg [Ryan] +• OpenLineage and Azure Purview [Shrikanth] +• Proxy Backend and Egeria integration progress update (Issue #152) [Mandy] +• OpenLineage last release overview (0.3.1) + ◦ Facet versioning + ◦ Airflow 2 / Spark 3 support, dbt improvements +• OpenLineage 0.4 scope review + ◦ Proxy Backend (Issue #152) + ◦ Spark, Airflow, dbt improvements (documentation, coverage, ...) + ◦ improvements to the OpenLineage model +• Open discussion 

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/issues/152|#152 Proposal for a ProxyBackend>
+
+ <p>This issue is to track the discussion and development of a proxy backend. The diagram below shows the basic idea.</p> + +<p><a href="https://user-images.githubusercontent.com/12354195/128734055-2a08372e-8d1b-426e-9c25-c06aeb551e36.png">image</a></p> +
+ +
+
Assignees
+ mandy-chessell +
+ +
+
Comments
+ 3 +
+ + + + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+ 🙌 Maciej Obuchowski, Peter Hicks +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-11-05 18:07:57
 +
+

*Thread Reply:* If you want to add something please chime in this thread

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-11-05 19:27:44
 +
+

*Thread Reply:* https://wiki.lfaidata.foundation/display/OpenLineage/Monthly+TSC+meeting

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-11-09 19:47:26
 +
+

*Thread Reply:* The monthly meeting is happening tomorrow. +The purview team will present at the December meeting instead +See full agenda here: https://wiki.lfaidata.foundation/display/OpenLineage/Monthly+TSC+meeting +You are welcome to contribute

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-11-10 11:10:17
 +
+

*Thread Reply:* The slides for the meeting later today: https://docs.google.com/presentation/d/1z2NTkkL8hg_2typHRYhcFPyD5az-5-tl/edit#slide=id.ge7d4b64ef4_0_0

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-11-10 12:02:23
 +
+

*Thread Reply:* It’s happening now ^

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-11-16 19:57:23
 +
+

*Thread Reply:* I have posted the notes and the recording from the last instance of our monthly meeting: https://wiki.lfaidata.foundation/display/OpenLineage/Monthly+TSC+meeting#MonthlyTSCmeeting-Nov10th2021(9amPT) +I have a few TODOs to follow up on tickets

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-11-05 18:09:10
 +
+

The next release of OpenLineage is being scoped: https://github.com/OpenLineage/OpenLineage/projects/6 +Please chime in if you want to raise the priority of something or are planning to contribute

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anthony Ivanov + (anthvt@gmail.com) +
+
2021-11-09 08:18:11
 +
+

Hi, I have been looking at open lineage for some time. And I really like it. It is very simple specification that covers a lot of use-cases. You can create any provider or consumer in a very simple way. So that’s pretty powerful. +I have some questions about things that are not clear to me. I am not sure if this is the best place to ask. Please refer me to other place if this is not appropriate.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anthony Ivanov + (anthvt@gmail.com) +
+
2021-11-09 08:18:58
 +
+

*Thread Reply:* How do you model continuous process (not batch processes). For example a flume or spark job that does some real time processing on data.

+ +

Maybe it’s simply a “Job” But than what is run ?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anthony Ivanov + (anthvt@gmail.com) +
+
2021-11-09 08:19:44
 +
+

*Thread Reply:* How do you model consumers at the end - they can be reports? Data applications, ML model deployments, APIs, GUI consumed by end users ?

+ +

Have you considered having some examples of different use cases like those?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anthony Ivanov + (anthvt@gmail.com) +
+
2021-11-09 08:21:43
 +
+

*Thread Reply: By definition, Job is a process definition that consumes and produces datasets. It is many to many relations? I’ve been wondering about that.Shouldn’t be more restrictive? +For example important use-case for lineage is troubleshooting or error notifications (e.g mark report or job as temporarily in bad state if upstream data integration is broken). +In order to be able to that you need to be able to traverse the graph to find the original error. So having multiple inputs produce single output make sense (e.g insert into output_1 select * from x,y group by a,b) . +But what are the cases where you’d want to see multiple outputs ? You can have single process produce multiple tables (in above example) but they’d alway be separate queries. The actual inputs for each output would be different.

+ +

But having multiple outputs create ambiguity as now If x or y is broken but have multiple outputs I do not know which is really impacted?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-11-09 08:34:01
 +
+

*Thread Reply:* > How do you model continuous process (not batch processes). For example a flume or spark job that does some real time processing on data. +> +> Maybe it’s simply a “Job” But than what is run ? +Every continuous process eventually has end - for example, you can deploy new version of your Flink pipeline. The new version would be the next Run for the same Job.

+ +

Moreover, OTHER event type is useful to update metadata like amount of processed records. In this Flink example, it could be emitted per checkpoint.

+ +

I think more attention for streaming use cases will be given soon.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-11-09 08:43:09
 +
+

*Thread Reply:* > How do you model consumers at the end - they can be reports? Data applications, ML model deployments, APIs, GUI consumed by end users ? +Our reference implementation is an web application https://marquezproject.github.io/marquez/

+ +

We definitely do not exclude any of the things you're talking about - and it would make a lot of sense to talk more about potential usages.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-11-09 08:45:47
 +
+

*Thread Reply:* > By definition, Job is a process definition that consumes and produces datasets. It is many to many relations? I’ve been wondering about that.Shouldn’t be more restrictive? +I think this is too SQL-centric view 🙂

+ +

Not everything is a query. For example, those Flink streaming jobs can produce side outputs, or even push data to multiple sinks. We need to model those types of jobs too.

+ +

If your application does not do multiple outputs, then I don't see how specification allowing those would impact you.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anthony Ivanov + (anthvt@gmail.com) +
+
2021-11-17 12:11:37
 +
+

*Thread Reply:* > We definitely do not exclude any of the things you’re talking about - and it would make a lot of sense to talk more about potential usages. +Yes I think that would be great if we expand on potential usages. if Open Lineage documentation (perhaps) has all kind of examples for different use-cases or case studies. Financal or healthcase industry case study and how would someone doing integration with OpenLineage. It would be easier to understand the concepts and make sure things are modeled consistently.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anthony Ivanov + (anthvt@gmail.com) +
+
2021-11-17 14:19:19
 +
+

*Thread Reply:* > I think this is too SQL-centric view 🙂 +> +> Not everything is a query. For example, those Flink streaming jobs can produce side outputs, or even push data to multiple sinks. We need to model those types of jobs too. +Thanks for answering @Maciej Obuchowski

+ +

Even in SQL you can have multiple outputs if you look thing at transaction level. I was simply using it as an example.

+ +

Maybe it would be clear what I mean in another example . Let’s say we have those phases

+ +
  1. Ingest from sources
  2. Process/transform
  3. export to somewhere +(image/diagram) +https://mermaid.ink/img/eyJjb2RlIjoiXG5ncmFwaCBMUlxuICAgIHN1YmdyYXBoIFNvdXJjZXNcbi[…]yIjpmYWxzZSwiYXV0b1N5bmMiOnRydWUsInVwZGF0ZURpYWdyYW0iOmZhbHNlfQ
    4. +
+ +

Let’s look at those two cases:

+ +
  1. Within a single flink job and even task: Inventory & UI are both written to both S3, DB
  2. Within a single flink job and even task: Inventory is written only to S3, UI is written only to DB
    3. +
+ +

In 1. open lineage run event could look like {inputs: [ui, inventory], outputs: [s3, db] }

+ +

In 2. user can either do same as 1. (because data changes or copy-paste) which would be an error since both do not go to both +Likely accurate one would be +{inputs: [ui], outputs: [s3] } {inputs: [ui], outputs: [db] }

+ +

If the specification standard required single output then

+ +
  1. would be modelled like run event {inputs: [ui, inventory], outputs: [s3] } ; {inputs: [ui, inventory], outputs: [db] } which is still correct if more verbose.
  2. could only be modelled this way: +{inputs: [ui], outputs: [s3] }; {inputs: [ui], outputs: [db] }
    3. +
+ +

The more restrictive specification seems to lower the chance for an error doesn’t it?

+ +

Also if tools know spec guarantees single output , they’d be able to write tracing capabilities which are more precise because the structure would allow for less ambiguity. +Storage backends that implement the spec could be also written in more optimal ways perhaps I have not looked into those accuracy of those hypothesis though.

+ +

Those were the thoughts I was thinking when asking about that. I’d be curious if there’s document on the research of pros/cons and alternatives for the design of the current specifications

+
+ + + + +
+
+ +
+ + + + + + + + +
Original URL: https://mermaid.ink/img/eyJjb2RlIjoiXG5ncmFwaCBMUlxuICAgIHN1YmdyYXBoIFNvdXJjZXNcbiAgICAgICAgSW52ZW50b3J5XG4gICAgICAgIFVJIFxuICAgIGVuZFxuICAgIHN1YmdyYXBoIFByb2Nlc3NcbiAgICAgICAgRmxpbmtKb2IxXG4gICAgICAgIEludmVudG9yeSAtLT4gfCBldmVudHMgIHwgRmxpbmtKb2IxXG4gICAgICAgIFVJIC0tPiB8IGV2ZW50cyB8IEZsaW5rSm9iMVxuICAgIGVuZFxuICAgIHN1YmdyYXBoIERlc3RpbmF0aW9uIFxuICAgICAgICBTM1xuICAgICAgICBEQlxuICAgICAgICBGbGlua0pvYjEgLS0-IFMzXG4gICAgICAgIEZsaW5rSm9iMSAtLT4gREJcbiAgICBlbmRcbiIsIm1lcm1haWQiOnsidGhlbWUiOiJmb3Jlc3QifSwidXBkYXRlRWRpdG9yIjpmYWxzZSwiYXV0b1N5bmMiOnRydWUsInVwZGF0ZURpYWdyYW0iOmZhbHNlfQ
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-11-23 05:38:11
 +
+

*Thread Reply:* @Anthony Ivanov I see what you're trying to model. I think this could be solved by column level lineage though - when we'll have it. OL consumer could look at particular columns and derive which table contained particular error.

+ +

> 2. Within a single flink job and even task: Inventory is written only to S3, UI is written only to DB +Does that actually happen? I understand this in case of job, but having single operator write to two different systems seems like bad design. Wouldn't that leave the possibility of breaking exactly-once unless you're going full into two phase commit?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anthony Ivanov + (anthvt@gmail.com) +
+
2021-11-23 17:02:36
 +
+

*Thread Reply:* > Does that actually happen? I understand this in case of job, but having single operator write to two different systems seems like bad design +In a Spark or flink job it is less likely now that you mention it. But in a batch job (airflow python or kubernetes operator for example) users could do anything and then they’d need lineage to figure out what is wrong if even if what they did is suboptimal 🙂

+ +

> I see what you’re trying to model. +I am not trying to model something specific. I am trying to understand how would openlineage be used in different organisations/companies and use-cases.

+ +

> I think this could be solved by column level lineage though +There’s something specific planned ? I could not find a ticket in github. I thought you can use Dataset Facets - Schema for example could be subset of columns for a table …

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-11-24 04:55:41
 +
+

*Thread Reply:* @Anthony Ivanov take a look at this: https://github.com/OpenLineage/OpenLineage/issues/148

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Lyndon Armitage + (lyndon.armitage@gmail.com) +
+
2021-11-10 13:21:23
 +
+

How do you deleting jobs/runs from Marquez/OpenLineage?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2021-11-10 16:17:10
 +
+

*Thread Reply:* We’re adding APIs to delete metadata in Marquez 0.20.0. Here’s the related issue, https://github.com/MarquezProject/marquez/issues/1736

+
+ + + + +
<https://github.com/MarquezProject/marquez/issues/1736|#1736 Ability to DELETE jobs and datasets>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/MarquezProject/marquez">MarquezProject/marquez</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2021-11-10 16:17:37
 +
+

*Thread Reply:* Until then, you can connected to the DB directly and drop the rows from both the datasets and jobs tables (I know, not dieal)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Lyndon Armitage + (lyndon.armitage@gmail.com) +
+
2021-11-11 05:03:50
 +
+

*Thread Reply:* Thanks! I assume deleting information will remain a Marquez only feature rather than becoming part of OpenLineage itself?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2021-12-10 14:07:57
 +
+

*Thread Reply:* Yes! Delete operations will be an action supported by consumers of OpenLineage events

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Lyndon Armitage + (lyndon.armitage@gmail.com) +
+
2021-11-11 05:13:31
 +
+

Am I understanding namespaces correctly? A job namespace is different to a Dataset namespace. +And that job namespaces define a job environment, like Airflow, Spark or some other system that executes jobs. But Dataset namespace define data locations, like an S3 bucket, local file system or schema in a Database?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Lyndon Armitage + (lyndon.armitage@gmail.com) +
+
2021-11-11 05:14:39
 +
+

*Thread Reply:* I've been skimming this page: https://github.com/OpenLineage/OpenLineage/blob/main/spec/Naming.md

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/spec/Naming.md | Naming.md>
+
+ <p><strong>Naming</strong></p> + +<p>We define the unique name strategy per resource to ensure it is followed uniformly independently from who is producing metadata and we can connect lineage from various sources.</p> + +<p>Both Jobs and Datasets are in their own namespaces. Job namespaces are related to their scheduler. The namespace for a dataset is the unique name for its datasource.</p> + +<p><strong>Datasets</strong></p> + +<p>The namespace and name of a datasource can be combined to form a URI (scheme:[//authority]path)</p> + +<p>• Namespace = scheme:[//authority] (the datasource) +• Name = path (the datasets)</p> + +<p><strong>Data warehouses/data bases</strong></p> + +<p>Datasets are called tables. Tables are organised in databases and schemas.</p> + +<p><strong>Postgres:</strong></p> + +<p>Datasource hierarchy:</p> + +<p>• Host +• Port</p> + +<p>Naming hierarchy:</p> + +<p>• Database +• Schema +• Table</p> + +<p>Identifier:</p> + +<p>• Namespace: postgres://{host}:{port} of the service instance. + • Scheme = postgres + • Authority = {host}:{port} +• Unique name: {database}.{schema}.{table} + • URI = postgres://{host}:{port}/{database}.{schema}.{table}</p> + +<p><strong>Redshift:</strong></p> + +<p>Datasource hierarchy:</p> + +<p>• Host: examplecluster.&lt;XXXXXXXXXXXX&gt;.<a href="http://us-west-2.redshift.amazonaws.com">us-west-2.redshift.amazonaws.com</a> +• Port: 5439</p> + +<p>Naming hierarchy:</p> + +<p>• Database +• Schema +• Table</p> + +<p>Identifier:</p> + +<p>• Namespace: redshift://{host}:{port} of the cluster instance. + • Scheme = redshift + • Authority = {host}:{port} +• Unique name: {database}.{schema}.{table} + • URI = redshift://{host}:{port}/{database}.{schema}.{table}</p> + +<p><strong>Snowflake</strong></p> + +<p>See: <a href="https://docs.snowflake.com/en/sql-reference/identifiers.html">Object Identifiers — Snowflake Documentation</a></p> + +<p>Datasource hierarchy:</p> + +<p>• account name</p> + +<p>Naming hierarchy:</p> + +<p>• Database: {database name} =&gt; unique across the account +• Schema: {schema name} =&gt; unique within the database +• Table: {table name} =&gt; unique within the schema</p> + +<p>Identifier:</p> + +<p>• Namespace: snowflake://{account name} + • Scheme = snowflake + • Authority = {account name} +• Name: {database}.{schema}.{table} + • URI = snowflake://{account name}/{database}.{schema}.{table}</p> + +<p><strong>BigQuery</strong></p> + +<p>See: <br /> +<a href="https://cloud.google.com/resource-manager/docs/creating-managing-projects|Creating and managing projects | Resource Manager Documentation">https://cloud.google.com/resource-manager/docs/creating-managing-projects|Creating and managing projects | Resource Manager Documentation</a> <br /> +<a href="https://cloud.google.com/bigquery/docs/datasets-intro|Introduction to datasets | BigQuery">https://cloud.google.com/bigquery/docs/datasets-intro|Introduction to datasets | BigQuery</a> <br /> +<a href="https://cloud.google.com/bigquery/docs/tables-intro|Introduction to tables | BigQuery">https://cloud.google.com/bigquery/docs/tables-intro|Introduction to tables | BigQuery</a></p> + +<p>Datasource hierarchy:</p> + +<p>• bigquery</p> + +<p>Naming hierarchy:</p> + +<p>• Project Name: {project name} =&gt; is not unique +• Project number: {project number} =&gt; numeric: is unique across google cloud +• Project ID: {project id} =&gt; readable: is unique across google cloud +• dataset: {dataset name} =&gt; is unique within a project +• table: {table name} =&gt; is unique within a dataset</p> + +<p>Identifier :</p> + +<p>• Namespace: bigquery + • Scheme = bigquery + • Authority = +• Unique name: {project id}.{dataset name}.{table name} + • URI = bigquery:{project id}.{schema}.{table}</p> + +<p><strong>Distributed file systems/blob stores</strong> +<strong>GCS</strong></p> + +<p>Datasource hierarchy: none, naming is global</p> + +<p>Naming hierarchy:</p> + +<p>• bucket name =&gt; globally unique +• Path</p> + +<p>Identifier :</p> + +<p>• Namespace: gs://{bucket name} + • Scheme = gs + • Authority = {bucket name} +• Unique name: {path} + • URI = gs://{bucket name}{path}</p> + +<p><strong>S3</strong></p> + +<p>Naming hierarchy:</p> + +<p>• bucket name =&gt; globally unique +• Path</p> + +<p>Identifier :</p> + +<p>• Namespace: s3://{bucket name} + • Scheme = s3 + • Authority = {bucket name} +• Unique name: {path} + • URI = s3://{bucket name}{path}</p> + +<p><strong>HDFS</strong></p> + +<p>Naming hierarchy:</p> + +<p>• Namenode: host + port +• Path</p> + +<p>Identifier :</p> + +<p>• Namespace: hdfs://{namenode host}:{namenode port} + • Scheme = hdfs + • Authority = {namenode host}:{namenode port} +• Unique name: {path} + • URI = hdfs://{namenode host}:{namenode port}{path}</p> + +<p><strong>DBFS (Databricks File System)</strong></p> + +<p>Naming hierarchy:</p> + +<p>• workspace name: globally unique +• Path</p> + +<p>Identifier :</p> + +<p>• Namespace: hdfs://{workspace name} + • Scheme = hdfs + • Authority = workspace name +• Unique name: {path} + • URI = hdfs://{workspace name}{path}</p> + +<p><strong>Jobs</strong> +<strong>Context</strong></p> + +<p>A <code>Job</code> is a recurring data transformation with Inputs and outputs. Each execution is captured as a <code>Run</code> with corresponding metadata. <br /> +A <code>Run</code> event identifies the <code>Job</code> it is an instance of by providing the job’s unique identifier. <br /> +The <code>Job</code> identifier is composed of a <code>Namespace</code> and a <code>Name</code>. The job name is unique within that namespace.</p> + +<p>The core property we want to identify about a <code>Job</code> is how it changes over time. Different schedules of the same logic applied to different datasets (possibly with different parameters) are different jobs. The notion of a <code>job</code> is tied to a recurring schedule with specific inputs and outputs. It could be an incremental update or a full reprocess or even a streaming job.</p> + +<p>If the same code artifact (for example a spark jar or a templated SQL query) is used in the context of different schedules with different input or outputs then they are different jobs. <br /> +We are interested first in how they affect the datasets they produce.</p> + +<p><strong>Job Namespace and constructing job names</strong></p> + +<p>Jobs have a <code>name</code> that is unique to them in their <code>namespace</code> by construction.</p> + +<p>The Namespace is the root of the naming hierarchy. The job name is constructed to identify the job within that namespace.</p> + +<p>Example:</p> + +<p>• Airflow: + • Namespace: the namespace is assigned to the airflow instance. Ex: airflow-staging, airflow-prod + • Job: each task in a DAG is a job. name: {dag name}.{task name} +• Spark: + • Namespace: the namespace is provided as a configuration parameter as in airflow. If there's a parent job, we use the same namespace, otherwise it is provided by configuration. + • Spark app job name: the spark.app.name + • Spark action job name: {spark.app.name}.{node.name}</p> + +<p><strong>Parent job run: a nested hierarchy of Jobs</strong></p> + +<p>It is often the case that jobs are part of a nested hierarchy. <br /> +For example an Airflow DAG contains tasks. An instance of the DAG is finished when all of the tasks are finished. Similarly a Spark job can spawn multiple actions each of them running independently. Additionally, a Spark job can be launched by an Airflow task within a DAG.</p> + +<p>Since what we care about is identifying the job as rooted in a recurring schedule, we want to capture that connection and make sure that we treat the same application logic triggered at different schedules as different jobs. For example: if an Airflow DAG runs individual tasks per partition (for example market segments) using the same underlying job logic, they will be tracked as separate jobs.</p> + +<p>To capture this, a run event provides <a href="https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.json#L282-L331">a <code>ParentRun</code> facet</a>, referring to the parent <code>Job</code> and <code>Run</code>. This allows tracking a recurring job from the root of the schedule it is running for. <br /> +If there's a parent job, we use the same namespace, otherwise it is provided by configuration.</p> + +<p>Example:</p> + +<pre><code>{ + "run": { + "runId": "run_uuid" + }, + "job": { + "namespace": "job_namespace", + "name": "job_name" + } +} + +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-11-11 05:46:06
 +
+

*Thread Reply:* Yes!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Lyndon Armitage + (lyndon.armitage@gmail.com) +
+
2021-11-11 06:17:01
 +
+

*Thread Reply:* Excellent, I think I had mistakenly conflated the two originally. This document makes it a little clearer. +As an additional question: +When viewing a Dataset in Marquez will it cross the job namespace bounds? As in, will I see jobs from different job namespaces?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Lyndon Armitage + (lyndon.armitage@gmail.com) +
+
2021-11-11 09:20:14
 +
+

*Thread Reply:* In this example I have 1 job namespace and 2 dataset namespaces: +sql-runner-dev is the job namespace. +I cannot see a graph of my job now. Is this something to do with the namespace names?

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Lyndon Armitage + (lyndon.armitage@gmail.com) +
+
2021-11-11 09:21:46
 +
+

*Thread Reply:* The above document seems to have implied a namespace could be like a connection string for a database

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Lyndon Armitage + (lyndon.armitage@gmail.com) +
+
2021-11-11 09:22:25
 +
+

*Thread Reply:* Wait, it does work? Marquez was being temperamental

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Lyndon Armitage + (lyndon.armitage@gmail.com) +
+
2021-11-11 09:24:01
 +
+

*Thread Reply:* Yes, marquez is unable to fetch lineage for either dataset

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Lyndon Armitage + (lyndon.armitage@gmail.com) +
+
2021-11-11 09:32:19
 +
+

*Thread Reply:* Here's what I mean:

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-11-11 09:59:24
 +
+

*Thread Reply:* I think you might have hit this issue: https://github.com/MarquezProject/marquez/issues/1744

+
+ + + + +
<https://github.com/MarquezProject/marquez/issues/1744|#1744 UI doesn't urlencode dataset names>
+
+ <p>The UI doesn't urlencode dataset names, so any dataset that has a <code>/</code> or <code>:</code> in the name causes a 404. This is notable with any S3 or GCS dataset, as their structure is always</p> + +<pre><code>{"namespace": "&lt;gs://bucket&gt;", +"datasetName": "/path/to/data"} + +</code></pre> +
+ +
+
Labels
+ bug +
+ + + + + + + +
+ + <a href="https://github.com/MarquezProject/marquez">MarquezProject/marquez</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-11-11 10:00:29
 +
+

*Thread Reply:* or, maybe not? It was released already.

+ +

Can you create issue on github with those helpful gifs? @Lyndon Armitage

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Lyndon Armitage + (lyndon.armitage@gmail.com) +
+
2021-11-11 10:58:25
 +
+

*Thread Reply:* I think you are right Maciej

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Lyndon Armitage + (lyndon.armitage@gmail.com) +
+
2021-11-11 10:58:52
 +
+

*Thread Reply:* Was that patched in 0,19.1?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-11-11 11:06:06
 +
+

*Thread Reply:* As far as I see yes: https://github.com/MarquezProject/marquez/releases/tag/0.19.1

+ +

Haven't tested this myself unfortunately.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Lyndon Armitage + (lyndon.armitage@gmail.com) +
+
2021-11-11 11:07:07
 +
+

*Thread Reply:* Perhaps not. It is urlencoding them: +<http://localhost:3000/lineage/dataset/jdbc%3Ah2%3Amem%3Asql_tests_like/HBMOFA.ORDDETP> +But the error seems to be in marquez getting them.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Lyndon Armitage + (lyndon.armitage@gmail.com) +
+
2021-11-11 11:09:23
 +
+

*Thread Reply:* This is an example Lineage event JSON I am sending.

+ +
+
Example lineage event
+ + + + + + +
+ + +
+ 👀 Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Lyndon Armitage + (lyndon.armitage@gmail.com) +
+
2021-11-11 11:11:29
 +
+

*Thread Reply:* I did run into another issue with really long names not being supported due to Marquez's DB using a fixed size string for a column, but that is understandable and probably a non-issue (my test code was generating temporary folders with long names).

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Lyndon Armitage + (lyndon.armitage@gmail.com) +
+
2021-11-11 11:22:00
 +
+

*Thread Reply:* A 404 is returned for: http://localhost:3000/api/v1/lineage/?nodeId=dataset:jdbc%3Ah2%3Amem%3Asql_tests_like:HBMOFA.ORDDETP

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-11-11 11:36:01
 +
+

*Thread Reply:* @Lyndon Armitage can you create issue on the Marquez repo? https://github.com/MarquezProject/marquez/issues

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Lyndon Armitage + (lyndon.armitage@gmail.com) +
+
2021-11-11 11:52:36
 +
+

*Thread Reply:* https://github.com/MarquezProject/marquez/issues/1761 Is this sufficient?

+
+ + + + +
<https://github.com/MarquezProject/marquez/issues/1761|#1761 404 returned when querying nodes that contain URL Encoded characters>
+
+ <p>With an event similar to the attached scheme 3 namespaces are created. 2 of these are Dataset namespaces and 1 is a job namespace.</p> + +<p>When trying to access one of these datasets a 404 is encountered</p> + +<p><strong>URL:</strong> <a href="http://localhost:3000/api/v1/lineage/?nodeId=dataset:jdbc%3Ah2%3Amem%3Asql_tests_like:HBMOFA.ORDDETP|http://localhost:3000/api/v1/lineage/?nodeId=dataset:jdbc%3Ah2%3Amem%3Asql_tests_like:HBMOFA.ORDDETP">http://localhost:3000/api/v1/lineage/?nodeId=dataset:jdbc%3Ah2%3Amem%3Asql<em>tests</em>like:HBMOFA.ORDDETP|http://localhost:3000/api/v1/lineage/?nodeId=dataset:jdbc%3Ah2%3Amem%3Asql<em>tests</em>like:HBMOFA.ORDDETP</a></p> + +<pre><code>{ + "eventType": "COMPLETE", + "eventTime": 1636646662.687894, + "run": { + "runId": "d3968e5f-84ac-48c1-954c-f999ff27ef3a", + "facets": null + }, + "job": { + "namespace": "sql-runner-dev", + "name": "ORDDETP - ORDDETP.avro", + "facets": { + "sourceCodeLocation": null, + "sql": { + "_producer": "lyndon-thinkpad/127.0.1.1", + "_schemaURL": "&lt;https://openlineage.io/spec/facets/1-0-0/SQLJobFacet.json#/$defs/SQLJobFacet&gt;", + "query": "SELECT t.**, CURRENT_DATE AS ingest_date FROM HBMOFA.ORDDETP t WHERE (BAYY &amp;gt;= 93 AND BAMMDD &amp;gt;= 801 AND BAMMDD &amp;lt; 1111) OR (BAYY = 92 AND BAMMDD &amp;gt;= 1301)" + }, + "documentation": { + "_producer": "lyndon-thinkpad/127.0.1.1", + "_schemaURL": "&lt;https://openlineage.io/spec/facets/1-0-0/DocumentationJobFacet.json#/$defs/DocumentationJobFacet&gt;", + "description": "SQL Runner Job for /tmp/sql_runner_tests4560779590026189736/config.conf" + } + } + }, + "inputs": [ + { + "namespace": "jdbc:h2:mem:sql_tests_like", + "name": "HBMOFA.ORDDETP", + "facets": { + "documentation": null, + "dataSource": { + "_producer": "lyndon-thinkpad/127.0.1.1", + "_schemaURL": "&lt;https://openlineage.io/spec/facets/1-0-0/DatasourceDatasetFacet.json#/$defs/DatasourceDatasetFacet&gt;", + "name": "jdbc:h2:mem:sql_tests_like", + "uri": "jdbc:h2:mem:sql_tests_like" + }, + "schema": null + }, + "inputFacets": null + } + ], + "outputs": [ + { + "namespace": "&lt;s3://sql-runner&gt;", + "name": "2021-11-11/incremental/ORDDETP.avro", + "facets": { + "documentation": null, + "dataSource": { + "_producer": "lyndon-thinkpad/127.0.1.1", + "_schemaURL": "&lt;https://openlineage.io/spec/facets/1-0-0/DatasourceDatasetFacet.json#/$defs/DatasourceDatasetFacet&gt;", + "name": "&lt;s3://sql-runner&gt;", + "uri": "&lt;s3://sql-runner&gt;" + }, + "schema": null + }, + "outputFacets": null + } + ], + "producer": "lyndon-thinkpad/127.0.1.1", + "schemaURL": "&lt;https://openlineage.io/spec/1-0-2/OpenLineage.json#/$defs/RunEvent&gt;" +} +</code></pre> + +<p>A partial stack trace from the docker process looks as follows:</p> + +<pre><code>marquez-api | 172.21.0.4 - - [11/Nov/2021:16:49:56 +0000] "GET /api/v1/namespaces/jdbc%3Ah2%3Amem%3Asql_tests_like/datasets/HBMOFA.ORDDETP/versions?limit=100&amp;amp;offset=0 HTTP/1.1" 200 648 "&lt;http://localhost:3000/lineage/dataset/jdbc%3Ah2%3Amem%3Asql_tests_like/HBMOFA.ORDDETP&gt;" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.81 Safari/537.36" 27 +marquez-api | ERROR [2021-11-11 16:49:59,168] io.dropwizard.jersey.errors.IllegalStateExceptionMapper: Error handling a request: 7b955941cb928966 +marquez-api | ! java.lang.IllegalStateException: No match available +marquez-api | ! at java.base/java.util.regex.Matcher.start(Unknown Source) +marquez-api | ! at marquez.service.models.NodeId.parts(NodeId.java:190) +marquez-api | ! at marquez.service.models.NodeId.asDatasetId(NodeId.java:214) +marquez-api | ! at marquez.service.LineageService.getJobUuid(LineageService.java:191) +marquez-api | ! at marquez.service.LineageService.lineage(LineageService.java:40) +marquez-api | ! at marquez.api.OpenLineageResource.getLineage(OpenLineageResource.java:96) + +</code></pre> + +<p>Note that the original OpenLineage POST request succeeds.</p> +
+ + + + + + + +
+ + <a href="https://github.com/MarquezProject/marquez">MarquezProject/marquez</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-11-11 11:54:41
 +
+

*Thread Reply:* Yup, thanks!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Francis McGregor-Macdonald + (francis@mc-mac.com) +
+
2021-11-15 13:00:39
 +
+

I am looking at an AWS Glue Crawler lineage event. The glue crawler creates or updates a table schema, and I have a few questions on aligning to best practice.

+ +
  1. Is this a dataset create/update or…
  2. … a job with no dataset inputs and only dataset outputs or
  3. … is the path in S3 the input and the Glue table the output?
  4. Is there an example of the lineage even here I can clone or work from? +Thanks.
    5. +
+ + + +
+ 🚀 Willy Lulciuc +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2021-11-15 13:04:19
 +
+

*Thread Reply:* Hi Francis, for the event is it creating a new table with new data in glue / adding new data to an existing one or is it simply reformatting an existing table or making an empty one?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Francis McGregor-Macdonald + (francis@mc-mac.com) +
+
2021-11-15 13:35:00
 +
+

*Thread Reply:* The table does not exist in the Glue catalog until …

+ +

A Glue crawler connects to one or more data stores (in this case S3), determines the data structures, and writes tables into the Data Catalog.

+ +

The data/objects are in S3, the Glue catalog is a metadata representation (HIVE) as as table.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2021-11-15 13:41:14
 +
+

*Thread Reply:* Hmm, interesting, so the lineage of interest here would be of the metadata flow not of the data itself?

+ +

In that case I’d say that the glue Crawler is a job that outputs a dataset.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2021-11-15 15:03:36
 +
+

*Thread Reply:* The crawler is a job that discovers a dataset. It doesn't create it. If you're posting lineage yourself, I'd post it as an input event, not an output. The thing that actually wrote the data - generated the records and stored them in S3 - is the thing that would be outputting the dataset

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Francis McGregor-Macdonald + (francis@mc-mac.com) +
+
2021-11-15 15:23:23
 +
+

*Thread Reply:* @Michael Collado I agree the crawler discovers the S3 dataset. It also creates an event which creates/updates the HIVE/Glue table.

+ +

If the Glue table isn’t a distinct dataset from the S3 data, how does this compare to a view in a database on top of a table. Are they 2 datasets or just one?

+ +

Glue can discover data in remote databases too, in those cases does it make sense to have only the source dataset?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Francis McGregor-Macdonald + (francis@mc-mac.com) +
+
2021-11-15 15:24:39
 +
+

*Thread Reply:* @John Thomas yes, its the metadata flow.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2021-11-15 15:24:52
 +
+

*Thread Reply:* that's how the Spark integration currently treats Hive datasets- I'd like to add a facet to attach that indicates that it is being read as a Hive table, and include all the appropriate metadata, but it uses the dataset's location in S3 as the canonical dataset identifier

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2021-11-15 15:29:22
 +
+

*Thread Reply:* @Francis McGregor-Macdonald I think the way to represent this is predicated on what you’re looking to accomplish by sending a runEvent for the Glue crawler. What are your broader objectives in adding this?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Francis McGregor-Macdonald + (francis@mc-mac.com) +
+
2021-11-15 15:50:37
 +
+

*Thread Reply:* I am working through AWS native services seeing how they could, can, or do best integrate with openlineage (I’m an AWS SA). Hence the questions on best practice.

+ +

Aligning with the Spark integration sounds like it might make sense then. Is there an example I could build from?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2021-11-15 17:56:17
 +
+

*Thread Reply:* an example of reporting lineage? you can look at the Spark integration here - https://github.com/OpenLineage/OpenLineage/tree/main/integration/spark/

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2021-11-15 17:59:14
 +
+

*Thread Reply:* Ahh, in that case I would have to agree with Michael’s approach to things!

+ + + +
+ ✅ Diogo +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Francis McGregor-Macdonald + (francis@mc-mac.com) +
+
2021-11-19 03:30:03
 +
+

*Thread Reply:* @Michael Collado I am following the Spark integration you recommended (for a Glue job) and while everything appears to be set up correct, I am getting no lineage appear in marquez (a request.get from the pyspark script can reach the endpoint). Is there a way to enable a debug log so I can look to identify where the issue is? +Is there a specific place to look in the regular logs?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2021-11-19 13:39:01
 +
+

*Thread Reply:* listener output should be present in the driver logs. you can turn on debug logging in your log4j config (or whatever logging tool you use) for the package io.openlineage.spark.agent

+ + + +
+ ✅ Francis McGregor-Macdonald +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2021-11-19 19:44:06
 +
+

Woo hoo! Initial Spark <-> Kafka support has been merged 🙂 https://github.com/OpenLineage/OpenLineage/pull/387

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/pull/387|#387 [spark] support read/write to kafka>
+
+ <p><strong>Problem</strong></p> + +<p>On write to kafka in spark exception occurs.</p> + +<p>Closes: #280</p> + +<p><strong>Solution</strong></p> + +<p>Adds kafka read/write support</p> + +<p>☐ Your change modifies the <a href="https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.json">core</a> OpenLineage model +☐ Your change modifies one or more OpenLineage <a href="https://github.com/OpenLineage/OpenLineage/tree/main/spec/facets">facets</a></p> + +<p>If you're contributing a new integration, please specify the scope of the integration and how/where it has been tested (e.g., Apache Spark integration supports <code>S3</code> and <code>GCS</code> filesystem operations, tested with AWS EMR).</p> + +<p><strong>Checklist</strong></p> + +<p>☑︎ You've <a href="https://github.com/OpenLineage/OpenLineage/blob/main/why-the-dco.md">signed-off</a> your work +☑︎ Your pull request title follows our <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CONTRIBUTING.md#creating-pull-requests">guidelines</a> +☑︎ Your changes are accompanied by tests (<em>if relevant</em>) +☐ Your change contains a <a href="https://kurtisnusbaum.medium.com/stacked-diffs-keeping-phabricator-diffs-small-d9964f4dcfa6">small diff</a> and is self-contained +☐ You've updated any relevant documentation (<em>if relevant</em>) +☐ You've updated the <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md"><code>CHANGELOG.md</code></a> with details about your change under the "Unreleased" section (<em>if relevant, depending on the change, this may not be necessary</em>) +☐ You've versioned the core OpenLineage model or facets according to <a href="https://docs.snowplowanalytics.com/docs/pipeline-components-and-applications/iglu/common-architecture/schemaver">SchemaVer</a> (<em>if relevant</em>)</p> +
+ +
+
Comments
+ 1 +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+ 🎉 Willy Lulciuc, John Thomas, Peter Hicks, Maciej Obuchowski +
+ +
+ 🙌 Willy Lulciuc, John Thomas, Francis McGregor-Macdonald, Peter Hicks, Maciej Obuchowski +
+ +
+ 🚀 Willy Lulciuc, John Thomas, Peter Hicks, Francis McGregor-Macdonald, Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Francis McGregor-Macdonald + (francis@mc-mac.com) +
+
2021-11-22 13:32:57
 +
+

I am “successfully” exporting lineage to openlineage from AWS Glue using the listener. Only the source load is showing, not the transforms, or the sink

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Francis McGregor-Macdonald + (francis@mc-mac.com) +
+
2021-11-22 13:34:15
 +
+

*Thread Reply:* Output event:

+ +

2021-11-22 08:12:15,513 INFO [spark-listener-group-shared] agent.OpenLineageContext (OpenLineageContext.java:emit(50)): Lineage completed successfully: ResponseMessage(responseCode=201, body=, error=null) { + “eventType”: “COMPLETE”, + “eventTime”: “2021-11-22T08:12:15.478Z”, + “run”: { + “runId”: “03bfc770-2151-499e-9265-8457a38ceec3”, + “facets”: { + “sparkversion”: { + “producer”: “https://github.com/OpenLineage/OpenLineage/tree/0.3.1/integration/spark”, + “schemaURL”: “https://openlineage.io/spec/1-0-2/OpenLineage.json#/$defs/RunFacet”, + “spark-version”: “3.1.1-amzn-0”, + “openlineage-spark-version”: “0.3.1” + } + } + }, + “job”: { + “namespace”: “sparkintegration”, + “name”: “nyctaxirawstage.mappartitionsunionmappartitionsnew_hadoop” + }, + “inputs”: [ + { + “namespace”: “s3.cdkdl-dev-foundationstoragef3787fa8-raw1d6fb60a-171gwxf2sixt9”, + “name”: “” + } + ], + “outputs”: [], + “producer”: “https://github.com/OpenLineage/OpenLineage/tree/0.3.1/integration/spark”, + “schemaURL”: “https://openlineage.io/spec/1-0-2/OpenLineage.json#/$defs/RunEvent” +}

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Francis McGregor-Macdonald + (francis@mc-mac.com) +
+
2021-11-22 13:34:59
 +
+

*Thread Reply:* This sink record is missing details …

+ +

2021-11-22 08:12:15,481 INFO [Thread-7] sinks.HadoopDataSink (HadoopDataSink.scala:$anonfun$writeDynamicFrame$1(275)): nameSpace: , table:

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Francis McGregor-Macdonald + (francis@mc-mac.com) +
+
2021-11-22 13:40:30
 +
+

*Thread Reply:* I can also see multiple history events (presumably for each transform, each as above) emitted for the same Glue Job, with different RunId, with the same inputs and the same (null) output.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2021-11-22 14:31:06
 +
+

*Thread Reply:* Are you using the existing spark integration for the spark lineage?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Francis McGregor-Macdonald + (francis@mc-mac.com) +
+
2021-11-22 14:46:47
 +
+

*Thread Reply:* I followed: https://github.com/OpenLineage/OpenLineage/tree/main/integration/spark +In the Glue context I was not clear on the correct settings for “spark.openlineage.parentJobName” and “spark.openlineage.parentRunId”, I put in static values (which may be incorrect)? +I injected these via: "--conf": "spark.openlineage.parentJobName=nyc-taxi-raw-stage",

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Francis McGregor-Macdonald + (francis@mc-mac.com) +
+
2021-11-22 14:47:54
 +
+

*Thread Reply:* Happy to share what is working when I am done, I can’t seem to find an AWS Glue specific example to walk me through.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2021-11-22 15:03:31
 +
+

*Thread Reply:* yeah, We haven’t spent any significant time with AWS Glue, but we just released the Databricks integration, which might help guide the way you’re working a little bit more

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Francis McGregor-Macdonald + (francis@mc-mac.com) +
+
2021-11-22 15:12:15
 +
+

*Thread Reply:* from what I can see in the DBX integration (https://github.com/OpenLineage/OpenLineage/tree/main/integration/spark/databricks) all of what is being done here I am doing in Glue (upload the jar, embed the settings into the Glue spark job). +It is emitting the above for each transform in the Glue job, but does not seem to capture the output …

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Francis McGregor-Macdonald + (francis@mc-mac.com) +
+
2021-11-22 15:13:54
 +
+

*Thread Reply:* Is there a standard Spark test script in use with openlineage I could put into Glue to test without using any Glue specific functionality (without for example the GlueContext, or Glue dynamic frames)?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Francis McGregor-Macdonald + (francis@mc-mac.com) +
+
2021-11-22 15:25:30
 +
+

*Thread Reply:* The initialisation does appear to be working if I compare it to the DBX README +Mine from AWS Glue… +21/11/22 18:48:48 INFO SparkContext: Registered listener io.openlineage.spark.agent.OpenLineageSparkListener +21/11/22 18:48:49 INFO OpenLineageContext: Init OpenLineageContext: Args: ArgumentParser(host=<http://ec2>-….<a href="http://compute-1.amazonaws.com:5000">compute-1.amazonaws.com:5000</a>, version=v1, namespace=spark_integration, jobName=default, parentRunId=null, apiKey=Optional.empty) URI: <http://ec2>-….<a href="http://compute-1.amazonaws.com:5000/api/v1/lineage">compute-1.amazonaws.com:5000/api/v1/lineage</a> +21/11/22 18:48:49 INFO AsyncEventQueue: Process of event SparkListenerApplicationStart(nyc-taxi-raw-stage,Some(spark-application-1637606927106),1637606926281,spark,None,None,None) by listener OpenLineageSparkListener took 1.092252643s.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2021-11-22 16:12:40
 +
+

*Thread Reply:* We don’t have a test run, unfortunately, but you could follow this blog post’s processes in each and see what the differences are? https://openlineage.io/blog/openlineage-spark/

+
+
openlineage.io
+ + + +
Tracing Data Lineage with OpenLineage and Apache Spark
+
+ Spark ushered in a brand new age of data democratization... and left us with a mess of hidden dependencies, stale datasets, and failed jobs. +
+ + + + + + +
Original URL: https://openlineage.io/blog/openlineage-spark/
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Francis McGregor-Macdonald + (francis@mc-mac.com) +
+
2021-11-22 16:43:23
 +
+

*Thread Reply:* Thanks, I have been looking at that. I will create a Glue job aligned with that. What is the best way to pass feedback? Keep it here?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2021-11-22 16:49:50
 +
+

*Thread Reply:* yeah, this thread will work great 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ilya Davidov + (idavidov@marpaihealth.com) +
+
2022-07-18 11:37:02
 +
+

*Thread Reply:* @Francis McGregor-Macdonald are you managed to enable it?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Francis McGregor-Macdonald + (francis@mc-mac.com) +
+
2022-07-18 15:14:47
 +
+

*Thread Reply:* Just DM you the code I used a while back (app.py + CDK code). I haven’t used it in a while, and there is some duplication in it. I had openlineage enabled, but dynamic frames not working yet with lineage. Let me know how you go. +I haven’t had the space to look at it in a while, but happy to support if you are looking at it.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Dinakar Sundar + (dinakar_sundar@condenast.com) +
+
2021-11-23 08:48:51
 +
+

how to use the Open lineage with amundsen ?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-11-23 09:01:11
 +
+

*Thread Reply:* You can use this: https://github.com/amundsen-io/amundsen/pull/1444

+
+ + + + +
<https://github.com/amundsen-io/amundsen/pull/1444|#1444 feat: OpenLineage extractor for databuilder>
+
+ <p>Signed-off-by: dechoma <a href="mailto:dominik.choma@gmail.com">dominik.choma@gmail.com</a></p> + +<p><strong>Summary of Changes</strong></p> + +<p>Databulder extractor to extract lineage information from OpenLineage format <br /> +<a href="https://github.com/OpenLineage/OpenLineage">https://github.com/OpenLineage/OpenLineage</a></p> + +<p><strong>Tests</strong> +<strong>Documentation</strong> +<strong>CheckList</strong></p> + +<p>Make sure you have checked <strong>all</strong> steps below to ensure a timely review.</p> + +<p>☐ PR title addresses the issue accurately and concisely. Example: "Updates the version of Flask to v1.0.2" + • In case you are adding a dependency, check if the license complies with the <a href="https://www.apache.org/legal/resolved.html#category-x">ASF 3rd Party License Policy</a>. +☐ PR includes a summary of changes. +☐ PR adds unit tests, updates existing unit tests, <strong>OR</strong> documents why no test additions or modifications are needed. +☐ In case of new functionality, my PR adds documentation that describes how to use it. + • All the public functions and the classes in the PR contain docstrings that explain what it does</p> +
+ +
+
Labels
+ area:databuilder, area:dev-tools, area:docs +
+ +
+
Comments
+ 2 +
+ + + + + + + +
+ + <a href="https://github.com/amundsen-io/amundsen">amundsen-io/amundsen</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2021-11-23 09:38:44
 +
+

*Thread Reply:* you can also check out this section from the Amundsen Community Meeting in october: https://www.youtube.com/watch?v=7WgECcmLSRk

+
+
YouTube
+ +
+ + + } + + Amundsen + (https://www.youtube.com/channel/UCgOyzG0sEoolxuC9YXDYPeg) +
+ + + +
Pandas profiling with Amundsen &amp; OpenLineage integration with Amundsen - Oct 2021 community meeting
+
+ +
+ + + + + + + + +
Original URL: https://www.youtube.com/watch?v=7WgECcmLSRk
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Dinakar Sundar + (dinakar_sundar@condenast.com) +
+
2021-11-23 08:49:16
 +
+

do we need to use the Marquez ?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2021-11-23 12:45:34
 +
+

*Thread Reply:* No, I believe the databuilder OpenLineage extractor for Amundsen will continue to store lineage metadata in Atlas

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2021-11-23 12:47:01
 +
+

*Thread Reply:* We've spoken to the Amundsen team, and though using Marquez to store lineage metadata isn't an option, it's an integration that makes sense but hasn't yet been prioritized

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Dinakar Sundar + (dinakar_sundar@condenast.com) +
+
2021-11-23 13:51:00
 +
+

*Thread Reply:* Thanks , Right now amundsen has no support for lineage extraction from spark or airflow , if this case do we need to use marquez for open lineage implementation to capture the lineage from airflow & spark

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2021-11-23 13:57:13
 +
+

*Thread Reply:* Maybe, that would mean running the full Amundsen stack as well as the Marquez stack along side each other (not ideal). The OpenLineage integration for Amundsen is very recent, so haven't had a chance to look deeply into the implementation. But, briefly looking over the config for Openlineagetablelineageextractor, you can only send metadata to Atlas

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Dinakar Sundar + (dinakar_sundar@condenast.com) +
+
2021-11-24 00:36:56
 +
+

*Thread Reply:* @Willy Lulciuc thats our real concern , running the two stacks will make a mess environment , let me explain our amundsen setup , we are having neo4j as backend , (front end , search service , metadata service,elastic search & neo4j) . our requirement to capture lineage from spark and airflow , imported into amundsen

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Vinith Krishnan US + (vinithk@nvidia.com) +
+
2022-03-11 22:33:39
 +
+

*Thread Reply:* We are running into a similar issue. @Dinakar Sundar were you able to get the Amundsen OpenLineage integration to work with a neo4j backend?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
bitsofinfo + (bitsofinfo.g@gmail.com) +
+
2021-11-24 11:41:31
 +
+

Hi all - i just watched the presentation on this and Marquez from the Airflow 21 summit. I was pretty impressed with this. My question is what other open source players are in this space or are pretty much people consolidating around this? (which would be great). Was looking at the available datasource extractors for the airflow side and would hope to see more here, looking at the code doesn't seem like too huge of a deal. Is there a roadmap available?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-11-24 11:49:14
 +
+

*Thread Reply:* You can take a look at https://github.com/OpenLineage/OpenLineage/projects

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Martin Fiser + (fisa@keboola.com) +
+
2021-11-24 19:24:48
 +
+

Hi all, I was wondering what is the status of native support of openlineage for DataHub or Amundzen. re https://openlineage.slack.com/archives/C01CK9T7HKR/p1633633476151000?thread_ts=1633008095.115900&cid=C01CK9T7HKR +Many thanks!

+
+ + +
+ + + } + + Julien Le Dem + (https://openlineage.slack.com/team/U01DCLP0GU9) +
+ + + +
+
+ <p>Those two specs look compatible, so Datahub should be able to consume this lineage metadata in the future</p> +
+ + + + + + +
Original URL: https://openlineage.slack.com/archives/C01CK9T7HKR/p1633633476151000?thread_ts=1633008095.115900&amp;cid=C01CK9T7HKR
+ + +
+ + Thread in Slack Conversation +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Martin Fiser + (fisa@keboola.com) +
+
2021-12-01 16:35:17
 +
+

*Thread Reply:* Anyone? Thanks!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Dinakar Sundar + (dinakar_sundar@condenast.com) +
+
2021-11-25 01:42:26
 +
+

our amundsen setup , we are having neo4j as backend , (front end , search service , metadata service,elastic search & neo4j) . our requirement to capture lineage from spark and airflow , imported into amundsen ?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2021-11-29 23:30:12
 +
+

Hello, OpenLineage folks - I'm curious if anyone here has ran into an issue like we're running into as we look to extend OpenLineage's Spark integration into Databricks.

+ +

Has anyone ran into an issue where a scala class should exist (based on a decompiled jar, I see that it's a public class) but you keep getting an error like object SqlDWRelation in package sqldw cannot be accessed in package com.databricks.spark.sqldw?

+ +

Databricks has a Synapse SQL DW connector: https://docs.databricks.com/data/data-sources/azure/synapse-analytics.html

+ +

I want to extract the database URL, table, and schema from the logical plan but

+ +

I execute something like the below command that runs a SELECT ** on the given tableName ("borrower" in this case) in the Azure Synapse database.

+ +

val df = spark.read.format("com.databricks.spark.sqldw") +.option("url", sqlDwUrl) +.option("tempDir", tempDir) +.option("forwardSparkAzureStorageCredentials", "true") +.option("dbTable", tableName) +.load() +val logicalPlan = df.queryExecution.logical +val logicalRelation = logicalPlan.asInstanceOf[LogicalRelation] +val sqlBaseRelation = logicalRelation.relation +I end up with something like this, all good so far: +```logicalPlan: org.apache.spark.sql.catalyst.plans.logical.LogicalPlan = +Relation[memberId#97,residentialState#98,yearsEmployment#99,homeOwnership#100,annualIncome#101,incomeVerified#102,dtiRatio#103,lengthCreditHistory#104,numTotalCreditLines#105,numOpenCreditLines#106,numOpenCreditLines1Year#107,revolvingBalance#108,revolvingUtilizationRate#109,numDerogatoryRec#110,numDelinquency2Years#111,numChargeoff1year#112,numInquiries6Mon#113] SqlDWRelation("borrower")

+ +

logicalRelation: org.apache.spark.sql.execution.datasources.LogicalRelation = +Relation[memberId#97,residentialState#98,yearsEmployment#99,homeOwnership#100,annualIncome#101,incomeVerified#102,dtiRatio#103,lengthCreditHistory#104,numTotalCreditLines#105,numOpenCreditLines#106,numOpenCreditLines1Year#107,revolvingBalance#108,revolvingUtilizationRate#109,numDerogatoryRec#110,numDelinquency2Years#111,numChargeoff1year#112,numInquiries6Mon#113] SqlDWRelation("borrower")

+ +

sqlBaseRelation: org.apache.spark.sql.sources.BaseRelation = SqlDWRelation("borrower")`` +Schema, I can easily get withsqlBaseRelation.schema` but I cannot figure out:

+ +
  1. How I can get the database name from the logical relation
  2. How I can get the table name from the logical relation ("borrower" is the table name so I can always parse the string if necessary" +I know that Databricks has the SqlDWRelation class which I think I need to cast the BaseRelation to BUT it appears to be in a jar / package that is inaccessible during the execution of a notebook. Specifically import com.databricks.spark.sqldw.SqlDWRelation is the relation and it appears to have a few accessors that would help me answer some of these questions: params and JDBCWrapper
    3. +
+ +

Of course this is undocumented on the Databricks side 😰

+ +

If I could cast the BaseRelation into this SqlDWRelation, I'd be able to get this info. However, whenever I attempt to use the imported SqlDWRelation, I get an error object SqlDWRelation in package sqldw cannot be accessed in package com.databricks.spark.sqldw I'm hoping someone has run into something similar in the past on the Spark / Databricks / Scala side and might share some advice. Thank you for any guidance!

+
+
docs.databricks.com
+ + + +
Azure Synapse Analytics | Databricks on AWS
+
+ Learn how to read and write data to Azure Synapse Analytics (formerly SQL Data Warehouse) using Databricks. +
+ + + + + + +
Original URL: https://docs.databricks.com/data/data-sources/azure/synapse-analytics.html
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-11-30 07:03:30
 +
+

*Thread Reply:* Have you tried reflection? https://commons.apache.org/proper/commons-lang/apidocs/org/apache/commons/lang3/reflect/FieldUtils.html#getDeclaredField-jav[…].lang.String-boolean-

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2021-11-30 11:21:34
 +
+

*Thread Reply:* I have not! Will give it a try, Maciej! Thank you for the reply!

+ + + +
+ 🙌 Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2021-11-30 15:20:18
 +
+

*Thread Reply:* 🙏 @Maciej Obuchowski we're not worthy! That was the magic we needed. Seems like a hack since we're snooping in on private classes but if it works...

+ +

Thank you so much for pointing to those utilities!

+ + + +
+ ❤️ Julien Le Dem +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-11-30 15:48:25
 +
+

*Thread Reply:* Glad I could help!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Francis McGregor-Macdonald + (francis@mc-mac.com) +
+
2021-11-30 19:43:03
 +
+

A colleague pointed me at https://open-metadata.org/, is there anywhere a view or comparison of this and openlineage?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mario Measic + (mario.measic.gavran@gmail.com) +
+
2021-12-01 08:51:28
 +
+

*Thread Reply:* Different concepts. OL is focused on describing the lineage and metadata of the running jobs. So it keeps track of all the metadata (schema, ...) of inputs and outputs at the time transformation occurs + transformation metadata (code version, cost, etc.)

+ +

OM I am not an expert but it's a metadata model with clients and API around it.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
RamanD + (romantanzar@gmail.com) +
+
2021-12-01 12:33:51
 +
+

Hey! OpenLineage is a beautiful initiative, to be honest! We also try to accommodate it. One question, maybe it's already described somewhere then many apologies :) if we need to propagate run id from Airflow to a child task (AWS Batch job, for instance) what will be the best way to do it in the current realization (as we get run id only at post execute phase)?.. We use Airflow 2+ integration.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-12-01 12:40:53
 +
+

*Thread Reply:* Hey. For technical reasons, we can't automatically register macro that does this job, as we could in Airflow 1 integration. You could put it yourself:

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-12-01 12:41:02
 +
+

*Thread Reply:* ```def lineageparentid(run_id, task): + """ + Macro function which returns the generated job and run id for a given task. This + can be used to forward the ids from a task to a child run so the job + hierarchy is preserved. Child run can create ParentRunFacet from those ids. + Invoke as a jinja template, e.g.

+ +
PythonOperator(
+    task_id='render_template',
+    python_callable=my_task_function,
+    op_args=['{{ lineage_parent_id(run_id, task) }}'], # lineage_run_id macro invoked
+    provide_context=False,
+    dag=dag
+)
+
+:param run_id:
+:param task:
+:return:
+"""
+with create_session() as session:
+    job_name = openlineage_job_name(task.dag_id, task.task_id)
+    ids = JobIdMapping.get(job_name, run_id, session)
+    if ids is None:
+        return ""
+    elif isinstance(ids, list):
+        run_id = "" if len(ids) == 0 else ids[0]
+    else:
+        run_id = str(ids)
+    return f"{_DAG_NAMESPACE}/{job_name}/{run_id}"
+
+ +

def openlineagejobname(dagid: str, taskid: str) -> str: + return f'{dagid}.{taskid}'```

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-12-01 12:41:13
 +
+

*Thread Reply:* from here: https://github.com/OpenLineage/OpenLineage/blob/main/integration/airflow/openlineage/airflow/dag.py#L77

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/integration/airflow/openlineage/airflow/dag.py | dag.py>
+
+ <pre><code>def lineage_parent_id(run_id, task): +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
RamanD + (romantanzar@gmail.com) +
+
2021-12-01 12:53:27
 +
+

*Thread Reply:* the quickest response ever! And that works like a charm 🙌

+ + + +
+ 👍 Michael Collado +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-12-01 13:21:16
 +
+

*Thread Reply:* Glad I could help!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2021-12-01 14:14:23
 +
+

@Maciej Obuchowski and @Michael Collado given your work on the Spark Integration, what's the right way to explore the Write operations' logical plans? When doing a read, it's easy! In scala df.queryExecution.logical gives you exactly what you need but how do you guys interactively explore what sort of commands are being used during a write? We are exploring some of the DataSourceV2 data sources and are hoping to learn from you guys a bit more, please 😃

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-12-01 14:18:00
 +
+

*Thread Reply:* For SQL, EXPLAIN EXTENDED and show() in scala-shell is helpful:

+ +

spark.sql("EXPLAIN EXTENDED CREATE TABLE tbl USING delta LOCATION '/tmp/delta' AS SELECT ** FROM tmp").show(false) +```|== Parsed Logical Plan == +'CreateTableAsSelectStatement [tbl], delta, /tmp/delta, false, false ++- 'Project [**] + +- 'UnresolvedRelation [tmp], [], false

+ +

== Analyzed Logical Plan ==

+ +

CreateTableAsSelect org.apache.spark.sql.delta.catalog.DeltaCatalog@63c5b63a, default.tbl, [provider=delta, location=/tmp/delta], false ++- Project [x#12, y#13] + +- SubqueryAlias tmp + +- LocalRelation [x#12, y#13]

+ +

== Optimized Logical Plan == +CreateTableAsSelect org.apache.spark.sql.delta.catalog.DeltaCatalog@63c5b63a, default.tbl, [provider=delta, location=/tmp/delta], false ++- LocalRelation [x#12, y#13]

+ +

== Physical Plan == +AtomicCreateTableAsSelect org.apache.spark.sql.delta.catalog.DeltaCatalog@63c5b63a, default.tbl, LocalRelation [x#12, y#13], [provider=delta, location=/tmp/delta, owner=mobuchowski], [], false ++- LocalTableScan [x#12, y#13] +|```

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-12-01 14:27:25
 +
+

*Thread Reply:* For dataframe api, I'm usually just either logging plan to console from OpenLineage listener, or looking at sparklogicalPlan or sparkunknown facets send by listener - even when the particular write operation isn't supported by integration, those facets should have some relevant info.

+ + + +
+ 🙌 Will Johnson +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-12-01 14:27:40
 +
+

*Thread Reply:* For example, for the query I've send at comment above, the spark_logicalPlan facet looks like this:

+ +

"spark.logicalPlan": { + "_producer": "<https://github.com/OpenLineage/OpenLineage/tree/0.4.0-SNAPSHOT/integration/spark>", + "_schemaURL": "<https://openlineage.io/spec/1-0-2/OpenLineage.json#/$defs/RunFacet>", + "plan": [ + { + "allowExisting": false, + "child": [ + { + "class": "org.apache.spark.sql.catalyst.plans.logical.LocalRelation", + "data": null, + "isStreaming": false, + "num-children": 0, + "output": [ + [ + { + "class": "org.apache.spark.sql.catalyst.expressions.AttributeReference", + "dataType": "integer", + "exprId": { + "id": 2, + "jvmId": "e03e2860-a24b-41f5-addb-c35226173f7c", + "product-class": "org.apache.spark.sql.catalyst.expressions.ExprId" + }, + "metadata": {}, + "name": "x", + "nullable": false, + "num-children": 0, + "qualifier": [] + } + ], + [ + { + "class": "org.apache.spark.sql.catalyst.expressions.AttributeReference", + "dataType": "integer", + "exprId": { + "id": 3, + "jvmId": "e03e2860-a24b-41f5-addb-c35226173f7c", + "product-class": "org.apache.spark.sql.catalyst.expressions.ExprId" + }, + "metadata": {}, + "name": "y", + "nullable": false, + "num-children": 0, + "qualifier": [] + } + ] + ] + } + ], + "class": "org.apache.spark.sql.execution.command.CreateViewCommand", + "name": { + "product-class": "org.apache.spark.sql.catalyst.TableIdentifier", + "table": "tmp" + }, + "num-children": 0, + "properties": null, + "replace": true, + "userSpecifiedColumns": [], + "viewType": { + "object": "org.apache.spark.sql.catalyst.analysis.LocalTempView$" + } + } + ] + },

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2021-12-01 14:38:55
 +
+

*Thread Reply:* Okay! That is very helpful! I wasn't sure if there was a fancier trick but I can definitely do logging 🙂 Our challenge was that our proprietary packages were resulting in Null Pointer Exceptions when it tried to push to OpenLineage 😞

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2021-12-01 14:39:02
 +
+

*Thread Reply:* Thank you as usual!!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2021-12-01 14:40:25
 +
+

*Thread Reply:* You can always add test cases and add breakpoints to debug in your IDE. That doesn't work for the container tests, but it does work for the other ones

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2021-12-01 14:47:20
 +
+

*Thread Reply:* Ah! That's a great point! I definitely would appreciate being able to poke at the objects interactively in a debug mode. Thank you for the guidance as well!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ricardo Gaspar + (ricardogaspar2@gmail.com) +
+
2021-12-03 11:49:10
 +
+

hi everyone! 👋 +Very noob question here: I’ve been wanting to play with Marquez and open lineage for my company’s projects. I use mostly scala & spark, but also Airflow. +I’ve been reading and watching talks about OpenLineage and Marquez. +So far i didn’t quite discover if Marquez or OpenLineage does field-level lineage (with Spark), like spline tries to.

+ +

Any idea?

+ +

Other sources about this topic +• https://medium.com/cdapio/data-integration-with-field-level-lineage-5d9986524316 +• https://medium.com/cdapio/field-level-lineage-part-1-3cc5c9e1d8c6 +• https://medium.com/cdapio/designing-field-level-lineage-part-2-b6c7e6af5bf4 +• https://www.youtube.com/playlist?list=PL897MHVe_nHeEQC8UnCfXecmZdF0vka_T +• https://www.youtube.com/watch?v=gKYGKXIBcZ0 +• https://www.youtube.com/watch?v=eBep6rRh7ic

+
+
Medium
+ + + +
Data Integration with Field Level Lineage
+
+ Arguably, data is the most important asset that an organization owns, and in today’s climate of data security and stewardship it’s more… +
+ +
+
Reading time
+ 6 min read +
+ + + + + + + + +
Original URL: https://medium.com/cdapio/data-integration-with-field-level-lineage-5d9986524316
+ + + +
+
+
Medium
+ + + +
Field Level Lineage — part 1
+
+ How the CDAP team designed a tool to track the journey of data in enterprise production environments. +
+ +
+
Reading time
+ 6 min read +
+ + + + + + + + +
Original URL: https://medium.com/cdapio/field-level-lineage-part-1-3cc5c9e1d8c6
+ + + +
+
+
Medium
+ + + +
Designing field Level Lineage — part 2
+
+ How the CDAP team designed a tool to track the journey of data in enterprise production environments. +
+ +
+
Reading time
+ 10 min read +
+ + + + + + + + +
Original URL: https://medium.com/cdapio/designing-field-level-lineage-part-2-b6c7e6af5bf4
+ + + +
+
+
YouTube
+ + + +
CDAP How To’s
+
+ CDAP How To Videos +
+ + + + + + + + +
Original URL: https://www.youtube.com/playlist?list=PL897MHVe_nHeEQC8UnCfXecmZdF0vka_T
+ + + +
+
+
YouTube
+ +
+ + + } + + CDAP + (https://www.youtube.com/c/CDAPio) +
+ + + +
CDAP Community Meetup Maintaining full data lineage and governance across billions of data partition
+
+ +
+ + + + + + + + +
Original URL: https://www.youtube.com/watch?v=gKYGKXIBcZ0
+ + + +
+ + + +
+ 🙌 Francis McGregor-Macdonald +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2021-12-03 11:55:17
 +
+

*Thread Reply:* Hi Ricardo - OpenLineage doesn’t currently have support for field-level lineage, but it’s definitely something we’ve been looking into. This is a great collection of resources 🙂

+ +

We’ve to-date been working on our integrations library, making it as easy to set up as possible.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ricardo Gaspar + (ricardogaspar2@gmail.com) +
+
2021-12-03 12:01:25
 +
+

*Thread Reply:* Thanks John! I was checking the issues on github and other posts here. Just wanted to clarify that. +I’ll keep an eye on it

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-12-06 20:25:19
 +
+

The next OpenLineage monthly meeting is this Wednesday at 9am PT. (everybody is welcome to join) +The slides are here: https://docs.google.com/presentation/d/1q2Be7WTKlIhjLPgvH-eXAnf5p4w7To9v/edit#slide=id.ge4b57c6942_0_75 +tentative agenda: +• SPDX headers [Mandy Chessel] +• Azure Purview + OpenLineage [Will Johnson, Mark Taylor] +• Logging backend (OpenTelemetry, ...) [Julien Le Dem] +• Open discussion +Please chime in in this thread if you’d want to add something

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-12-06 20:28:09
 +
+

*Thread Reply:* The link to join the meeting is on the wiki: https://wiki.lfaidata.foundation/display/OpenLineage/Monthly+TSC+meeting

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2021-12-06 20:28:25
 +
+

*Thread Reply:* Please reach out to me if you’d like to be added to a gcal invite

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Dinakar Sundar + (dinakar_sundar@condenast.com) +
+
2021-12-06 22:37:29
 +
+

@John Thomas we in Condenast currently exploring the features of open lineage to integrate to databricks , https://github.com/OpenLineage/OpenLineage/tree/main/integration/spark/databricks , spark configuration not working ,

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2021-12-08 02:03:37
 +
+

*Thread Reply:* Hi Dinakar. Can you give some specifics regarding what kind of problem you're running into?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Dinakar Sundar + (dinakar_sundar@condenast.com) +
+
2021-12-09 10:15:50
 +
+

*Thread Reply:* Hi @Michael Collado, were able to set the spark configuration for spark extra listener & placed jars as well , wen i ran the sapark job , Lineage is not get tracked into the marquez

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Dinakar Sundar + (dinakar_sundar@condenast.com) +
+
2021-12-09 10:34:39
 +
+

*Thread Reply:* {"producer":"https://github.com/OpenLineage/OpenLineage/tree/0.3.1/integration/spark","schemaURL":"https://github.com/OpenLineage/OpenLineage/tree/0.3.1/integration/spark/facets/spark/v1/output-statistics-facet.json","rowCount":0,"size":-1,"status":"DEPRECATED"}},"outputFacets":{"outputStatistics":{"producer":"https://github.com/OpenLineage/OpenLineage/tree/0.3.1/integration/spark","schemaURL":"https://openlineage.io/spec/facets/1-0-0/OutputStatisticsOutputDatasetFacet.json#/$defs/OutputStatisticsOutputDatasetFacet","rowCount":0,"size":-1}}}],"producer":"https://github.com/OpenLineage/OpenLineage/tree/0.3.1/integration/spark","schemaURL":"https://openlineage.io/spec/1-0-2/OpenLineage.json#/$defs/RunEvent"} +OpenLineageHttpException(code=0, message=java.lang.IllegalArgumentException: Cannot construct instance of io.openlineage.spark.agent.client.HttpError (although at least one Creator exists): no String-argument constructor/factory method to deserialize from String value ('{"code":404,"message":"HTTP 404 Not Found"}') + at [Source: UNKNOWN; line: -1, column: -1], details=java.util.concurrent.CompletionException: java.lang.IllegalArgumentException: Cannot construct instance of io.openlineage.spark.agent.client.HttpError (although at least one Creator exists): no String-argument constructor/factory method to deserialize from String value ('{"code":404,"message":"HTTP 404 Not Found"}') + at [Source: UNKNOWN; line: -1, column: -1]) + at io.openlineage.spark.agent.OpenLineageContext.emit(OpenLineageContext.java:48) + at io.openlineage.spark.agent.lifecycle.SparkSQLExecutionContext.start(SparkSQLExecutionContext.java:122) + at io.openlineage.spark.agent.OpenLineageSparkListener.lambda$onJobStart$3(OpenLineageSparkListener.java:159) + at java.util.Optional.ifPresent(Optional.java:159) + at io.openlineage.spark.agent.OpenLineageSparkListener.onJobStart(OpenLineageSparkListener.java:148) + at org.apache.spark.scheduler.SparkListenerBus.doPostEvent(SparkListenerBus.scala:37) + at org.apache.spark.scheduler.SparkListenerBus.doPostEvent$(SparkListenerBus.scala:28) + at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37) + at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37) + at org.apache.spark.util.ListenerBus.postToAll(ListenerBus.scala:119) + at org.apache.spark.util.ListenerBus.postToAll$(ListenerBus.scala:103) + at org.apache.spark.scheduler.AsyncEventQueue.super$postToAll(AsyncEventQueue.scala:105) + at org.apache.spark.scheduler.AsyncEventQueue.$anonfun$dispatch$1(AsyncEventQueue.scala:105) + at scala.runtime.java8.JFunction0$mcJ$sp.apply(JFunction0$mcJ$sp.java:23) + at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62) + at org.apache.spark.scheduler.AsyncEventQueue.org$apache$spark$scheduler$AsyncEventQueue$$dispatch(AsyncEventQueue.scala:100) + at org.apache.spark.scheduler.AsyncEventQueue$$anon$2.$anonfun$run$1(AsyncEventQueue.scala:96) + at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1585) + at org.apache.spark.scheduler.AsyncEventQueue$$anon$2.run(AsyncEventQueue.scala:96)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Dinakar Sundar + (dinakar_sundar@condenast.com) +
+
2021-12-09 13:29:42
 +
+

*Thread Reply:* Issue solved , mentioned the version wrongly as 1 instead v1

+ + + +
+ 🙌 Michael Collado +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jitendra Sharma + (jitendra_sharma@condenast.com) +
+
2021-12-07 02:07:06
 +
+

👋 Hi everyone!

+ + + +
+ 👋 Willy Lulciuc, Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
kavuri raghavendra + (kavuri.raghavendra@gmail.com) +
+
2021-12-08 05:37:44
 +
+

Hello Everyone.. we are exploring Openlineage for capturing Spark lineage.. but form the GitHub(https://github.com/OpenLineage/OpenLineage/tree/main/integration/spark) ..I see that the output send to API (Marquez).. how can I send it to Kafka topic.. can some body please guide me on this.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Minkyu Park + (minkyu@datakin.com) +
+
2021-12-08 12:15:38
 +
+

*Thread Reply:* https://github.com/OpenLineage/OpenLineage/pull/400/files

+ +

there’s ongoing PR for proxy backend, which opens http API and redirects events to Kafka.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2021-12-08 12:17:38
 +
+

*Thread Reply:* Hi Kavuri, as minkyu said, there's currently work going on to simplify this process.

+ +

For now, you'll need to make something to capture the HTTP api events and send them to the Kafka topic. Changing the spark.openlineage.url parameter will send the runEvents wherever you like, but obviously you can't directly produce HTTP events to a topic

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
kavuri raghavendra + (kavuri.raghavendra@gmail.com) +
+
2021-12-08 22:13:09
 +
+

*Thread Reply:* Many Thanks for the Reply.. As I understand, currently pushing lineage to kafka topic is not yet there. it is under implementation. If you can help me out in understanding in which version it is going to be present, that will help me a lot. Thanks in advance.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Minkyu Park + (minkyu@datakin.com) +
+
2021-12-09 12:57:10
 +
+

*Thread Reply:* Not sure about the release plan, but the http endpoint is just regular RESTful API, and you will be able to write a super simple proxy for your own use case if you want.

+ + + +
+ 🙌 Will Johnson +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2021-12-12 00:13:54
 +
+

Hi, Open Lineage team - For the Spark Integration, I'm looking to extract information from a DataSourceV2 data source.

+ +

I'm working on the WRITE side of the data source and right now I'm touching the AppendData logical plan (I can't find the Java Doc): https://github.com/rdblue/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicLogicalOperators.scala#L446

+ +

I was able to extract out the table name (from the named relation) but I'm struggling getting out the schema next.

+ +

I noticed that the AppendData offers inputSet, schema, and outputSet. +• inputSet gives me an AttributeSet which does contain the names of my columns (https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/AttributeSet.scala#L69) +• schema returns an empty StructType +• outputSet is an empty AttributeSet +I thought I read in the Spark Internals book that outputSet would only be populated if there was some sort of change to the DataFrame columns but I cannot find that page and searching for spark outputSet turns up few relevant results.

+ +

Has anyone else worked with the AppendData plan and gotten the schema out of it? Am I going down the wrong path with this snippet of code below? Thank you for any guidance!

+ +

if (logical instanceof AppendData) { + AppendData appendOp = (AppendData) logical; + NamedRelation namedRel = appendOp.table(); + <a href="http://log.info">log.info</a>(namedRel.name()); // Works great! + <a href="http://log.info">log.info</a>(appendOp.inputSet().toString());// This will get you a rough schema + StructType schema = appendOp.schema(); // This is an empty StructType + <a href="http://log.info">log.info</a>(schema.json()); // Nothing useful here + }

+
+ + + + +
<https://github.com/rdblue/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicLogicalOperators.scala | basicLogicalOperators.scala>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/rdblue/spark">rdblue/spark</a> +
+ + +
+
+ + + + +
<https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/AttributeSet.scala | AttributeSet.scala>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/apache/spark">apache/spark</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-12-12 07:34:13
 +
+

*Thread Reply:* One thing, you're looking at Ryan's fork of Spark, which is few thousand commits behind head 🙂

+ +

This one should be good: +https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/v2Commands.scala#L72

+ +

About schema: looking at AppendData's query schema should work, if there's no change to columns, because to pass analysis, data being inserted have to match table's schema. I would test that though 🙂

+ +

On the other hand, current AppendDataVisitor just looks at AppendData's table and tries to extract dataset from it using list of common output visitors:

+ +

https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/src/main/co[…]o/openlineage/spark/agent/lifecycle/plan/AppendDataVisitor.java

+ +

In this case, the DataSourceV2RelationVisitor would look at it, provided we're using Spark 3:

+ +

https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/src/main/sp[…]ge/spark3/agent/lifecycle/plan/DataSourceV2RelationVisitor.java

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-12-12 07:37:04
 +
+

*Thread Reply:* In this case, we basically need more info about nature of this DataSourceV2Relation, because this is provider-dependent. We have Iceberg in main branch and Delta here: https://github.com/OpenLineage/OpenLineage/pull/393/files#diff-7b66a9bd5905f4ba42914b73a87d834c1321ebcf75137c1e2a2413c0d85d9db6

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2021-12-13 14:54:13
 +
+

*Thread Reply:* Ah! Maciej! As always, thank you! Looking through the DataSourceV2RelationVisitor you provided, it looks like the connector (Azure Cosmos Db) doesn't provide that Provider property 😞 😞 😞

+ +

Is there any other method for determining the type of DataSourceV2Relation?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2021-12-13 14:57:06
 +
+

*Thread Reply:* And, to make sure I close out on my original question, it was as simple as the code that Maciej was using:

+ +

I merely needed to use DataSourceV2Realtion rather than NamedRelation!

+ +

DataSourceV2Relation relation = (DataSourceV2Relation)appendOp.table(); + <a href="http://log.info">log.info</a>(relation.schema().toString()); + <a href="http://log.info">log.info</a>(relation.name());

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-12-15 06:20:31
 +
+

*Thread Reply:* Are we talking about this connector? https://github.com/Azure/azure-sdk-for-java/blob/934200f63dc5bc7d5502a95f8daeb8142[…]/src/main/scala/com/azure/cosmos/spark/ItemsReadOnlyTable.scala

+
+ + + + +
<https://github.com/Azure/azure-sdk-for-java/blob/934200f63dc5bc7d5502a95f8daeb81426bb600e/sdk/cosmos/azure-cosmos-spark_3-1_2-12/src/main/scala/com/azure/cosmos/spark/ItemsReadOnlyTable.scala | ItemsReadOnlyTable.scala>
+
+ <pre><code>private[spark] class ItemsReadOnlyTable(val sparkSession: SparkSession, +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/Azure/azure-sdk-for-java">Azure/azure-sdk-for-java</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-12-15 06:22:05
 +
+

*Thread Reply:* I guess you can use object.getClass.getCanonicalName() to find if the passed class matches the one that Cosmos provider uses.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2021-12-15 09:53:24
 +
+

*Thread Reply:* Yes! That's the one, Maciej! I will give getCanonicalName a try but also make a PR into that repo to get the provider property set up correctly 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2021-12-15 09:53:28
 +
+

*Thread Reply:* Thank you so much!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-12-15 10:09:39
 +
+

*Thread Reply:* Glad to help 😄

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-12-15 10:22:58
 +
+

*Thread Reply:* @Will Johnson could you tell on which commands from https://github.com/OpenLineage/OpenLineage/issues/368#issue-1038510649 you'll be working?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-12-15 10:24:14
 +
+

*Thread Reply:* If any, of course 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2021-12-15 10:49:31
 +
+

*Thread Reply:* From all of our tests on that Cosmos connector, it looks like it strictly uses athe AppendData operation. However @Harish Sune is looking at more of these commands from a Delta data source.

+ + + +
+ 👍 Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2021-12-22 22:43:34
 +
+

*Thread Reply:* Just to close the loop on this one - I submitted a PR for the work we've been doing. Looking forward to any feedback! https://github.com/OpenLineage/OpenLineage/pull/450

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/pull/450|#450 [INTEGRATION][SPARK] Support Azure Cosmos DB Spark Connector in Open Lineage>
+
+ <p><strong>Problem</strong></p> + +<p>We would like to extend OpenLineage's Spark integration to work with the <a href="https://github.com/Azure/azure-sdk-for-java/tree/main/sdk/cosmos/azure-cosmos-spark_3-1_2-12">Azure Cosmos DB Spark Connector for Spark3</a>.</p> + +<p>This would enable people using Spark, OpenLineage, and Cosmos DB to observe the input and output to their spark jobs.</p> + +<p>Closes: <a href="https://github.com/OpenLineage/OpenLineage/issues/449">#449</a></p> + +<p><strong>Solution</strong></p> + +<p>The solution requires extending the DataSourceV2RelationVisitor to support the Azure Cosmos DB as a provider.</p> + +<p>• Azure Cosmos DB Spark Connector does not use the provider property on the table but can be determined based on name. + • The Cosmos DB connector defines a Cosmos "table" as <code>com.azure.cosmos.spark.items.&amp;lt;serviceName&amp;gt;.&amp;lt;databaseName&amp;gt;.&amp;lt;collectionName&amp;gt;</code> +• The Azure Cosmos DB write operations are all <code>AppendData</code>. As such, the <code>Spark3VisitorFactoryImpl.java</code> must include <code>DataSourceV2RelationVisitor</code> as one of its common visitors. + • This gets fed into <code>AppendData</code> so the underlying relation can be tested against a <code>DataSourceV2RelationVisitor</code>. +• When reading Cosmos DB, Spark uses a <code>DataSourceV2ScanRelation</code>. + • This class has a <code>relation</code> property which is actually <code>DataSourceV2Relation</code>. + • Since it was that simple, rather than making an entirely new SCAN relation class, I hitched it onto the existing DataSourceV2Relation class. +• I followed the same process as Iceberg which does not check if you are using any Iceberg Jars (like the BigQuery integration).</p> + +<p>To support this addition, I added two tests similar to the Delta and Iceberge DataSourceV2Relation tests. I also modified <code>testIsDefinedAtForNonDefinedProvider</code> so that a mock table name was provided since the implementation now checks if the table name matches the pattern for Cosmos DB.</p> + +<p><strong>Scope</strong>: This solution has been tested locally on a Spark-Shell with the <code>--packages "com.azure.cosmos.spark:azure_cosmos_spark_3_1_2_12:4.5.0"</code> parameter along with in Azure Databricks on a Spark 3.1.2 cluster.</p> + +<p><strong>Checklist</strong></p> + +<p>☑︎ You've <a href="https://github.com/OpenLineage/OpenLineage/blob/main/why-the-dco.md">signed-off</a> your work +☑︎ Your pull request title follows our <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CONTRIBUTING.md#creating-pull-requests">guidelines</a> +☑︎ Your changes are accompanied by tests (<em>if relevant</em>) +☑︎ Your change contains a <a href="https://kurtisnusbaum.medium.com/stacked-diffs-keeping-phabricator-diffs-small-d9964f4dcfa6">small diff</a> and is self-contained +☐ You've updated any relevant documentation (<em>if relevant</em>) +☐ You've updated the <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md"><code>CHANGELOG.md</code></a> with details about your change under the "Unreleased" section (<em>if relevant, depending on the change, this may not be necessary</em>) +☐ You've versioned the core OpenLineage model or facets according to <a href="https://docs.snowplowanalytics.com/docs/pipeline-components-and-applications/iglu/common-architecture/schemaver">SchemaVer</a> (<em>if relevant</em>)</p> +
+ +
+
Comments
+ 1 +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-12-23 05:04:36
 +
+

*Thread Reply:* Thanks @Will Johnson! I added one question about dataset naming.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2021-12-14 19:45:59
 +
+

Finally got this doc posted - https://github.com/OpenLineage/OpenLineage/pull/437 (see the readable version here ) +Looking for feedback, @Willy Lulciuc @Maciej Obuchowski @Will Johnson

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/pull/437|#437 Add proposal doc for making Spark event handling code more extensible>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/90321b8b7e44c070efd1f6ba6b4bcafb3b78f5e9/proposals/168/making_spark_visitors_extensible.md | making_spark_visitors_extensible.md>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2021-12-15 10:54:41
 +
+

*Thread Reply:* Yes! This is awesome!! How might this work for an existing command like the DataSourceV2Visitor.

+ +

Right now, OpenLineage checks based on the provider property if it's an Iceberg or Delta provider.

+ +

Ideally, we'd be able to extend the list of providers or have a custom "CosmosDbDataSourceV2Visitor" that knew how to work with a custom DataSourceV2.

+ +

Would that cause any conflicts if the base class is already accounted for in OpenLineage?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-12-15 11:13:20
 +
+

*Thread Reply:* Resolving this would be nice addition to the doc (and, to the implementation) - currently, we're just returning result of first function for which isDefinedAt is satisfied.

+ +

This means, that we can depend on the order of the visitors...

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2021-12-15 13:59:12
 +
+

*Thread Reply:* great question. For posterity, I'd like to move this to the PR discussion. I'll address the question there.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2021-12-14 19:50:57
 +
+

Oh, and I forgot to post yesterday +OpenLineage 0.4.0 was released 🥳

+ +

This was a big one. +• Split tests for Spark 2 and Spark 3 +• Spark output metrics +• Databricks support with init scripts +• Initial Iceberg support for Spark +• Initial Kafka support for Spark +• dbt build support +• forward compatibility for dbt versions +• lots of bug fixes 🙂 +Check the full changelog for details

+ + + +
+ 🙌 Maciej Obuchowski, Will Johnson, Peter Hicks, Manuel, Peter Hanssens +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Dinakar Sundar + (dinakar_sundar@condenast.com) +
+
2021-12-14 21:42:40
 +
+

Hi @Michael Collado is there any documentation on using great expectations with open lineage

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2021-12-15 11:50:47
 +
+

*Thread Reply:* hmm, actually the only documentation we have right now is on the demo.datakin.com site https://demo.datakin.com/onboarding . The great expectations tab should be enough to get you started

+
+
demo.datakin.com
+ + + +
datakin
+
+ Observability, Lineage and Visualization for Data Pipelines +
+ + + + + + +
Original URL: https://demo.datakin.com/onboarding
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2021-12-15 11:51:04
 +
+

*Thread Reply:* I'll open a ticket to copy that documentation to the OpenLineage site repo

+ + + +
+ 👍 Madhu Maddikera, Dinakar Sundar +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Carlos Meza + (omar.m.8x@gmail.com) +
+
2021-12-15 09:52:51
 +
+

Hello ! I am new on OpenLineage , awesome project !! ; anybody knows about integration with Deequ ? Or a way to capture dataset stats with openlineage ? Thanks ! Appreciate the help !

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2021-12-15 19:01:50
 +
+

*Thread Reply:* Hi! We don't have any integration with deequ yet. We have a structure for recording data quality assertions and statistics, though - see https://github.com/OpenLineage/OpenLineage/blob/main/spec/facets/DataQualityAssertionsDatasetFacet.json and https://github.com/OpenLineage/OpenLineage/blob/main/spec/facets/DataQualityMetricsInputDatasetFacet.json for the specs.

+ +

Check the great expectations integration to see how those facets are being used

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Bruno González + (brugms2@gmail.com) +
+
2022-05-24 06:20:50
 +
+

*Thread Reply:* This is great. Thanks @Michael Collado!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anatoliy Zhyzhkevych + (Anatoliy.Zhyzhkevych@franklintempleton.com) +
+
2021-12-19 22:40:33
 +
+

Hi,

+ +

I am testing Open Lineage/Marquez 0.4.0 with dbt 1.0.0 using dbt-ol build +It seems 12 events were generated but UI shows only history of runs with "Nothing to show here" in detail section about datasets/tests failures in dbt namespace. +The warehouse namespace shows lineage but no details about dataset/test failures .

+ +

Please advice.

+ +

02:57:54 Done. PASS=4 WARN=0 ERROR=3 SKIP=2 TOTAL=9 +02:57:54 Error sending message, disabling tracking +Emitting OpenLineage events: 100%|██████████████████████████████████████████████████████| 12/12 [00:00<00:00, 12.50it/s]

+ +
+
+ + + + + + +
+
+
+ + + + + + +
+
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-12-20 04:15:51
 +
+

*Thread Reply:* This is nothing to show here when you click on test node, right? What about run node?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anatoliy Zhyzhkevych + (Anatoliy.Zhyzhkevych@franklintempleton.com) +
+
2021-12-20 12:28:21
 +
+

*Thread Reply:* There is no details about failure.

+ +

```dbt-ol build -t DEV --profile cdp --profiles-dir /c/Work/dbt/cdp100/profiles --project-dir /c/Work/dbt/cdp100 --select +riskrawmastersharedshareclass +Running OpenLineage dbt wrapper version 0.4.0 +This wrapper will send OpenLineage events at the end of dbt execution. +02:57:21 Running with dbt=1.0.0 +02:57:23 [WARNING]: Configuration paths exist in your dbtproject.yml file which do not apply to any resources. +There are 1 unused configuration paths:

  • models.cdp.risk.raw.liquidity.shared
    • +
+ +

02:57:23 Found 158 models, 181 tests, 0 snapshots, 0 analyses, 574 macros, 0 operations, 2 seed files, 56 sources, 1 exposure, 0 metrics +02:57:23 +02:57:35 Concurrency: 10 threads (target='DEV') +02:57:35 +02:57:35 1 of 9 START test dbtexpectationssourceexpectcompoundcolumnstobeuniquebsesharedpbshareclassEDMPORTFOLIOIDSHARECLASSCODEanyvalueismissingDELETEDFLAGFalse [RUN] +02:57:37 1 of 9 PASS dbtexpectationssourceexpectcompoundcolumnstobeuniquebsesharedpbshareclassEDMPORTFOLIOIDSHARECLASSCODEanyvalueismissingDELETEDFLAGFalse [PASS in 2.67s] +02:57:37 2 of 9 START view model REPL.SHARECLASSDIM.................................... [RUN] +02:57:39 2 of 9 OK created view model REPL.SHARECLASSDIM............................... [SUCCESS 1 in 2.12s] +02:57:39 3 of 9 START test dbtexpectationsexpectcompoundcolumnstobeuniquerawreplpbsharedshareclassRISKPORTFOLIOIDSHARECLASSCODEanyvalueismissingDELETEDFLAGFalse [RUN] +02:57:43 3 of 9 PASS dbtexpectationsexpectcompoundcolumnstobeuniquerawreplpbsharedshareclassRISKPORTFOLIOIDSHARECLASSCODEanyvalueismissingDELETEDFLAGFalse [PASS in 3.42s] +02:57:43 4 of 9 START view model RAWRISKDEV.STG.SHARECLASSDIM........................ [RUN] +02:57:46 4 of 9 OK created view model RAWRISKDEV.STG.SHARECLASSDIM................... [SUCCESS 1 in 3.44s] +02:57:46 5 of 9 START view model RAWRISKDEV.MASTER.SHARECLASSDIM..................... [RUN] +02:57:46 6 of 9 START test relationshipsriskrawstgsharedshareclassRISKINSTRUMENTIDRISKINSTRUMENTIDrefriskrawstgsharedsecurity_ [RUN] +02:57:46 7 of 9 START test relationshipsriskrawstgsharedshareclassRISKPORTFOLIOIDRISKPORTFOLIOIDrefriskrawstgsharedportfolio_ [RUN] +02:57:51 5 of 9 ERROR creating view model RAWRISKDEV.MASTER.SHARECLASSDIM............ [ERROR in 4.31s] +02:57:51 8 of 9 SKIP test relationshipsriskrawmastersharedshareclassRISKINSTRUMENTIDRISKINSTRUMENTIDrefriskrawmastersharedsecurity_ [SKIP] +02:57:51 9 of 9 SKIP test relationshipsriskrawmastersharedshareclassRISKPORTFOLIOIDRISKPORTFOLIOIDrefriskrawmastersharedportfolio_ [SKIP] +02:57:52 7 of 9 FAIL 7282 relationshipsriskrawstgsharedshareclassRISKPORTFOLIOIDRISKPORTFOLIOIDrefriskrawstgsharedportfolio_ [FAIL 7282 in 5.41s] +02:57:54 6 of 9 FAIL 6520 relationshipsriskrawstgsharedshareclassRISKINSTRUMENTIDRISKINSTRUMENTIDrefriskrawstgsharedsecurity_ [FAIL 6520 in 7.23s] +02:57:54 +02:57:54 Finished running 6 tests, 3 view models in 30.71s. +02:57:54 +02:57:54 Completed with 3 errors and 0 warnings: +02:57:54 +02:57:54 Database Error in model riskrawmastersharedshareclass (models/risk/raw/master/shared/riskrawmastersharedshareclass.sql) +02:57:54 002003 (42S02): SQL compilation error: +02:57:54 Object 'RAWRISKDEV.AUDIT.STGSHARECLASSDIMRELATIONSHIPRISKINSTRUMENTID' does not exist or not authorized. +02:57:54 compiled SQL at target/run/cdp/models/risk/raw/master/shared/riskrawmastersharedshareclass.sql +02:57:54 +02:57:54 Failure in test relationshipsriskrawstgsharedshareclassRISKPORTFOLIOIDRISKPORTFOLIOIDrefriskrawstgsharedportfolio (models/risk/raw/stg/shared/riskrawstgsharedschema.yml) +02:57:54 Got 7282 results, configured to fail if != 0 +02:57:54 +02:57:54 compiled SQL at target/compiled/cdp/models/risk/raw/stg/shared/riskrawstgsharedschema.yml/relationshipsriskrawstgsha19e10fb324f7d0cccf2aab512683f693.sql +02:57:54 +02:57:54 Failure in test relationshipsriskrawstgsharedshareclassRISKINSTRUMENTIDRISKINSTRUMENTID_refriskrawstgsharedsecurity_ (models/risk/raw/stg/shared/riskrawstgsharedschema.yml) +02:57:54 Got 6520 results, configured to fail if != 0 +02:57:54 +02:57:54 compiled SQL at target/compiled/cdp/models/risk/raw/stg/shared/riskrawstgsharedschema.yml/relationshipsriskrawstgsha_e3148a1627817f17f7f5a9eb841ef16f.sql +02:57:54 +02:57:54 See test failures:

+ +
+ +

select ** from RAWRISKDEV.AUDIT.STGSHARECLASSDIMrelationship_RISKINSTRUMENT_ID

+ +
+ +

02:57:54 +02:57:54 Done. PASS=4 WARN=0 ERROR=3 SKIP=2 TOTAL=9 +02:57:54 Error sending message, disabling tracking +Emitting OpenLineage events: 100%|██████████████████████████████████████████████████████| 12/12 [00:00<00:00, 12.50it/s]Emitted 14 openlineage events +(dbt) linux@dblnbk152371:/c/Work/dbt/cdp$```

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-12-20 12:30:20
 +
+

*Thread Reply:* I'm talking on clicking on non-test node in Marquez UI - the screenshots shared show you clicked on the one ending in test

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anatoliy Zhyzhkevych + (Anatoliy.Zhyzhkevych@franklintempleton.com) +
+
2021-12-20 16:46:11
 +
+

*Thread Reply:* There are two types of failures: tests failed on stage model (relationships) and physical error in master model (no table with such name). The stage test node in Marquez does not show any indication of failures and dataset node indicates failure but without number of failed records or table name for persistent test storage. The failed master model shows in red but no details of failure. Master model tests were skipped because of model failure but UI reports "Complete".

+ +
+
+ + + + + + +
+
+
+ + + + + + +
+
+
+ + + + + + +
+
+
+ + + + + + +
+
+
+ + + + + + +
+
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-12-20 18:11:50
 +
+

*Thread Reply:* If I understood correctly, for model you would like OpenLineage to capture message error, like this one +22:52:07 Database Error in model customers (models/customers.sql) +22:52:07 Syntax error: Expected "(" or keyword SELECT or keyword WITH but got identifier "PLEASE_REMOVE" at [56:12] +22:52:07 compiled SQL at target/run/jaffle_shop/models/customers.sql +And for dbt test failures, to visualize better that error is happening, for example like that:

+ +
+
+ + + + + + +
+
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-12-20 18:23:12
 +
+

*Thread Reply:* We actually do the first one for Airflow and Spark, I've missed it for dbt 😞

+ +

Created issue to add it to spec in a generic way: +https://github.com/OpenLineage/OpenLineage/issues/446

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/issues/446|#446 [PROPOSAL] Add generic ErrorRunFacet>
+
+ <p>On execution failure, multiple systems provide text-based error messages, like dbt:</p> + +<pre><code>22:52:07 Database Error in model customers (models/customers.sql) +22:52:07 Syntax error: Expected "(" or keyword SELECT or keyword WITH but got identifier "PLEASE_REMOVE" at [56:12] +22:52:07 compiled SQL at target/run/jaffle_shop/models/customers.sql + +</code></pre> + +<p>Airflow:</p> + +<pre><code>airflow.exceptions.AirflowException: Bash command failed. The command returned a non-zero exit code. + +</code></pre> + +<p>Spark:</p> + +<pre><code>org.apache.spark.sql.AnalysisException: Table or view not found: wrong_table_name; line 1 pos 14; + +</code></pre> + +<p>We should add simple <code>ErrorRunFacet</code> to capture human-readable messages. Currently, we have some custom facets for those for <code>BigQuery</code> and <code>Spark</code></p> + +<p>Proposed schema would look like that:</p> + +<pre><code>{ + "$schema" : "&lt;https://json-schema.org/draft/2020-12/schema&gt;", + "$id" : "&lt;https://openlineage.io/spec/facets/1-0-0/ErrorRunFacet.json&gt;", + "$defs" : { + "ErrorRunFacet" : { + "allOf" : [ { + "$ref" : "&lt;https://openlineage.io/spec/1-0-2/OpenLineage.json#/$defs/RunFacet&gt;" + }, { + "type" : "object", + "properties" : { + "message" : { + "description" : "A human-readable string representing error message generated by observed system", + "type" : "string", + "example" : "org.apache.spark.sql.AnalysisException: Table or view not found: wrong_table_name; line 1 pos 14" + }, + "stackTrace" : { + "description" : "A language-specific stack trace generated by observed system", + "type" : "string", + "example" : "Exception in thread \"main\" java.lang.RuntimeException: A test exception\nat io.openlineage.SomeClass.method(SomeClass.java:13)\nat io.openlineage.SomeClass.anotherMethod(SomeClass.java:9)" + } + }, + "required" : [ "message" ] + } ], + "type" : "object" + } + }, + "type" : "object", + "properties" : { + "error" : { + "$ref" : "#/$defs/ErrorRunFacet" + } + } +} +</code></pre> +
+ +
+
Labels
+ proposal +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anatoliy Zhyzhkevych + (Anatoliy.Zhyzhkevych@franklintempleton.com) +
+
2021-12-20 22:49:54
 +
+

*Thread Reply:* Sounds great. Failed/Skipped Tests/Models could be color-coded as well. Thanks.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jorge Reyes (Zenta Group) + (jorge.reyes@zentagroup.com) +
+
2021-12-22 12:37:00
 +
+

hello everyone , i'm learning Openlineage, I am trying to connect with airflow 2, is it possible? or that version is not yet released. this is currently throwing me airflow

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-12-22 12:38:26
 +
+

*Thread Reply:* Hey. If you're using Airflow 2, you should use LineageBackend method described here: https://github.com/OpenLineage/OpenLineage/tree/main/integration/airflow#airflow-21-experimental

+ + + +
+ 🙌 Jorge Reyes (Zenta Group) +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2021-12-22 12:39:06
 +
+

*Thread Reply:* You don't need to do anything with DAG import then.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jorge Reyes (Zenta Group) + (jorge.reyes@zentagroup.com) +
+
2021-12-22 12:40:30
 +
+

*Thread Reply:* Thanks!!!!! i'll try

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2021-12-27 16:49:20
 +
+

The PR at https://github.com/OpenLineage/OpenLineage/pull/451 should be everything needed to complete the implementation for https://github.com/OpenLineage/OpenLineage/pull/437 . The PR is in draft mode, as I still need ~1 day to update the integration test expectations to match the refactoring (there are some new events, but from my cursory look, the old events still match expected contents). But I think it's in a state that can be reviewed before the tests are updated.

+ +

There are two other PRs that this one is based on - broken up for easier reviewing +• https://github.com/OpenLineage/OpenLineage/pull/447 +• https://github.com/OpenLineage/OpenLineage/pull/448

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2021-12-27 16:49:56
 +
+

*Thread Reply:* @Will Johnson @Maciej Obuchowski FYI 👆

+ + + +
+ 🙌 Will Johnson, Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-01-07 15:25:11
 +
+

The next OpenLineage Technical Steering Committee meeting is Wednesday, January 12! Meetings are on the second Wednesday of each month from 9:00 to 10:00am PT.  +Join us on Zoom: https://us02web.zoom.us/j/81831865546?pwd=RTladlNpc0FTTDlFcWRkM2JyazM4Zz09 +All are welcome. +Agenda: +• OpenLineage 0.4 and 0.5 releases +• Egeria version 3.4 support for OpenLineage +• Airflow TaskListener to simplify OpenLineage integration [Maciej] +• Open Discussion +Notes: https://tinyurl.com/openlineagetsc

+ + + +
+ 🙌 Maciej Obuchowski, Ross Turk, John Thomas, Minkyu Park, Joshua Wankowski, Dalin Kim +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
David Virgil + (david.virgil.naranjo@googlemail.com) +
+
2022-01-11 12:16:09
 +
+

Hello community,

+ +

We are able to post this datasource in marquez. But then the information about the facet with the datasource is not displayed in the UI.

+ +

We want to display the S3 location (URI) where this datasource is pointing to. +{ + id: { + namespace: "<s3://hbi-dns-staging>", + name: "PCHG" + }, + type: "DB_TABLE", + name: "PCHG", + physicalName: "PCHG", + createdAt: "2022-01-11T16:15:54.887Z", + updatedAt: "2022-01-11T16:56:04.093153Z", + namespace: "<s3://hbi-dns-staging>", + sourceName: "<s3://hbi-dns-staging>", + fields: [], + tags: [], + lastModifiedAt: null, + description: null, + currentVersion: "c565864d-1a66-4cff-a5d9-2e43175cbf88", + facets: { + dataSource: { + uri: "<s3://hbi-dns-staging/sql-runner/2022-01-11/PCHG.avro>", + name: "<s3://hbi-dns-staging>", + _producer: "<a href="http://ip-172-25-23-163.dir.prod.aws.hollandandbarrett.comeu-west-1.com/172.25.23.163">ip-172-25-23-163.dir.prod.aws.hollandandbarrett.comeu-west-1.com/172.25.23.163</a>", + _schemaURL: "<https://openlineage.io/spec/facets/1-0-0/DatasourceDatasetFacet.json#/$defs/DatasourceDatasetFacet>" + } + } +}

+ + + +
+
+
+
+ + + + + + + +
+
+ + + + +
+ +
David Virgil + (david.virgil.naranjo@googlemail.com) +
+
2022-01-11 12:24:00
 +
+

As you see there is no much info in openlineage UI

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-01-11 13:02:16
 +
+

The OpenLineage TSC meeting is tomorrow! https://openlineage.slack.com/archives/C01CK9T7HKR/p1641587111000700

+
+ + +
+ + + } + + Michael Robinson + (https://openlineage.slack.com/team/U02LXF3HUN7) +
+ + + +
+
+ <p>The next OpenLineage Technical Steering Committee meeting is Wednesday, January 12! Meetings are on the second Wednesday of each month from 9:00 to 10:00am PT.  +Join us on Zoom: <a href="https://us02web.zoom.us/j/81831865546?pwd=RTladlNpc0FTTDlFcWRkM2JyazM4Zz09">https://us02web.zoom.us/j/81831865546?pwd=RTladlNpc0FTTDlFcWRkM2JyazM4Zz09</a> +All are welcome. +Agenda: +• OpenLineage 0.4 and 0.5 releases +• Egeria version 3.4 support for OpenLineage +• Airflow TaskListener to simplify OpenLineage integration [Maciej] +• Open Discussion +Notes: <a href="https://tinyurl.com/openlineagetsc">https://tinyurl.com/openlineagetsc</a></p> +
+ + + + + + +
Original URL: https://openlineage.slack.com/archives/C01CK9T7HKR/p1641587111000700
+ + +
+ + Slack Conversation +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2022-01-12 11:59:44
 +
+

*Thread Reply:* ^ It’s happening now!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
David Virgil + (david.virgil.naranjo@googlemail.com) +
+
2022-01-14 06:46:44
 +
+

any idea guys about the previous question?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Minkyu Park + (minkyu@datakin.com) +
+
2022-01-18 14:19:39
 +
+

*Thread Reply:* Just to be clear, were you able to get a datasource information from API but just now showing up in the UI? Or you weren’t able to get it from API too?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
SAM + (skhettri@gmail.com) +
+
2022-01-17 03:41:56
 +
+

Hi everyone !! I am doing POC of OpenLineage with Airflow version 2.1, before that would like to know, if this version is supported by OpenLineage?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Conor Beverland + (conorbev@gmail.com) +
+
2022-01-18 11:40:00
 +
+

*Thread Reply:* It does generally work, but, there's a known limitation in that only successful task runs are reported to the lineage backend. This is planned to be fixed in Airflow 2.3.

+ + + +
+ ✅ SAM +
+ +
+ ❤️ Julien Le Dem +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
SAM + (skhettri@gmail.com) +
+
2022-01-18 20:35:52
 +
+

*Thread Reply:* thank you. 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
SAM + (skhettri@gmail.com) +
+
2022-01-17 06:47:54
 +
+

Hello there, I’m using docker Airflow version 2.1.0 , below were the steps I performed but I encountered error, pls help:

+ +
  1. Inside requirements.txt file i added openlineage-airflow . Then ran pip install -r requirements.txt .
  2. Added environmental variable using this command +export AIRFLOW__LINEAGE__BACKEND = openlineage.lineage_backend.OpenLineageBackend
  3. Then configured HTTP Backend environment variables inside “airflow” folder: +export OPENLINEAGE_URL=<http://marquez:5000>
  4. Ran Marquez using ./docker/up.sh & open web frontend UI and saw below error msg:
    5. +
+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Conor Beverland + (conorbev@gmail.com) +
+
2022-01-18 11:30:38
 +
+

*Thread Reply:* hey, I'm aware of one small bug ( which will be fixed in the upcoming OpenLineage 0.5.0 ) which means you would also have to include google-cloud-bigquery in your requirements.txt. This is the bug: https://github.com/OpenLineage/OpenLineage/issues/438

+ + + +
+ ✅ SAM +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Conor Beverland + (conorbev@gmail.com) +
+
2022-01-18 11:31:51
 +
+

*Thread Reply:* The other thing I think you should check is, did you def define the AIRFLOW__LINEAGE__BACKEND variable correctly? What you pasted above looks a little odd with the 2 = signs

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Conor Beverland + (conorbev@gmail.com) +
+
2022-01-18 11:34:25
 +
+

*Thread Reply:* I'm looking a task log inside my own Airflow and I see msgs like: +INFO - Constructing openlineage client to send events to

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Conor Beverland + (conorbev@gmail.com) +
+
2022-01-18 11:34:47
 +
+

*Thread Reply:* ^ i.e. I think checking the task logs you can see if it's at least attempting to send data

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Conor Beverland + (conorbev@gmail.com) +
+
2022-01-18 11:34:52
 +
+

*Thread Reply:* hope this helps!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
SAM + (skhettri@gmail.com) +
+
2022-01-18 20:40:37
 +
+

*Thread Reply:* Thank you, will try again.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2022-01-18 20:10:25
 +
+

Just published OpenLineage 0.5.0 . Big items here are +• dbt-spark support +• New proxy message broker for forwarding OpenLineage messages to Kafka +• New extensibility API for Spark integration +Accompanying tweet thread on the latter two items here: https://twitter.com/PeladoCollado/status/1483607050953232385

+ + + +
+ 🙌 Maciej Obuchowski, Kevin Mellott +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2022-01-19 12:39:30
 +
+

*Thread Reply:* BTW, this was actually the 0.5.1 release. Because, pypi... 🤷‍♂️:skintone4:

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mario Measic + (mario.measic.gavran@gmail.com) +
+
2022-01-27 06:45:08
 +
+

*Thread Reply:* nice on the dbt-spark support 👍

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mohamed El IBRAHIMI + (mohamedelibrahimi700@gmail.com) +
+
2022-01-19 11:12:14
 +
+

HELLO everyone . I’ve been reading and watching talks about OpenLineage and Marquez . this solution is exactly what we been looking to lineage our etls . GREAT WORK . our etls based on postgres redshift and airflow. SO

+ +

I tried to implement the example respecting all the steps required. everything runs successfully (the two dags on airflow ) on host http://localhost:3000/ but nothing appeared on marquez ui . am i missing something ? .

+ +

I’am thinking about create a simple etl pandas to a pandas with some transformation . Like to have a poc to show it to my team . I REALLY NEED SOME HELP

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-01-19 11:13:35
 +
+

*Thread Reply:* Are you using docker on mac with "Use Docker Compose V2" enabled?

+ +

We've just found yesterday that it somehow breaks our example...

+ + + +
+ ✅ Mohamed El IBRAHIMI +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mohamed El IBRAHIMI + (mohamedelibrahimi700@gmail.com) +
+
2022-01-19 11:14:51
 +
+

*Thread Reply:* yes i just installed docker on mac

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mohamed El IBRAHIMI + (mohamedelibrahimi700@gmail.com) +
+
2022-01-19 11:15:02
 +
+

*Thread Reply:* and docker compose version 1.29.2

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-01-19 11:20:24
 +
+

*Thread Reply:* What you can do is to uncheck this, do docker system prune -a and try again.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mohamed El IBRAHIMI + (mohamedelibrahimi700@gmail.com) +
+
2022-01-19 11:21:56
 +
+

*Thread Reply:* done but i get this : Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-01-19 11:22:15
 +
+

*Thread Reply:* Try to restart docker for mac

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-01-19 11:23:00
 +
+

*Thread Reply:* It needs to show Docker Desktop is running :

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mohamed El IBRAHIMI + (mohamedelibrahimi700@gmail.com) +
+
2022-01-19 11:24:01
 +
+

*Thread Reply:* yeah done . I will try to implement the example again and see thank you very much

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mohamed El IBRAHIMI + (mohamedelibrahimi700@gmail.com) +
+
2022-01-19 11:32:55
 +
+

*Thread Reply:* i dont why i getting this when i $ docker-compose up :

+ +

WARNING: The TAG variable is not set. Defaulting to a blank string. +WARNING: The APIPORT variable is not set. Defaulting to a blank string. +WARNING: The APIADMINPORT variable is not set. Defaulting to a blank string. +WARNING: The WEBPORT variable is not set. Defaulting to a blank string. +ERROR: The Compose file ‘./../docker-compose.yml’ is invalid because: +services.api.ports contains an invalid type, it should be a number, or an object +services.api.ports contains an invalid type, it should be a number, or an object +services.web.ports contains an invalid type, it should be a number, or an object +services.api.ports value [‘:’, ‘:’] has non-unique elements

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-01-19 11:46:12
 +
+

*Thread Reply:* are you running it exactly like here, with respect to directories, etc?

+ +

https://github.com/MarquezProject/marquez/tree/main/examples/airflow

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mohamed El IBRAHIMI + (mohamedelibrahimi700@gmail.com) +
+
2022-01-19 11:59:36
 +
+

*Thread Reply:* yeah yeah my bad . every things work fine know . I see the graph in the ui

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mohamed El IBRAHIMI + (mohamedelibrahimi700@gmail.com) +
+
2022-01-19 12:04:01
 +
+

*Thread Reply:* one more question plz . As i said our etls based on postgres redshift and airflow . any advice you have for us to integrate OL to our pipeline ?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mohamed El IBRAHIMI + (mohamedelibrahimi700@gmail.com) +
+
2022-01-19 11:12:17
 +
+

thank you very much

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kevin Mellott + (kevin.r.mellott@gmail.com) +
+
2022-01-19 17:29:51
 +
+

I’m upgrading our OL Java client from an older version (0.2.3) and noticed that the ol.newCustomFacetBuilder() method to create custom facets no longer exists. I can see in this code diff that it might be replaced by simply adding to the additional properties of the standard element you are extending.

+ +

Can you please let me know if I’m understanding this change correctly? In other words, is the code in the diff functionally equivalent or is there a large change I should be understanding better?

+ +

https://github.com/OpenLineage/OpenLineage/compare/0.2.3...0.4.0#diff-f0381d7e68797d9ec60551c96897809072582350e1657d23425747358ec6e471L196

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2022-01-19 17:50:39
 +
+

*Thread Reply:* Hi Kevin - to my understanding that's correct. Do you guys have a custom extractor using this?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kevin Mellott + (kevin.r.mellott@gmail.com) +
+
2022-01-19 20:49:49
 +
+

*Thread Reply:* Thanks John! We have custom code emitting OL events within our ingestion pipeline and it includes a custom facet. I’ll refactor the code to the new format and should be good to go.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kevin Mellott + (kevin.r.mellott@gmail.com) +
+
2022-01-21 00:34:37
 +
+

*Thread Reply:* Just to follow up, this code update worked as expected and we are all good on the upgrade.

+ + + +
+ 👍 Minkyu Park, John Thomas, Julien Le Dem +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
SAM + (skhettri@gmail.com) +
+
2022-01-21 02:13:51
 +
+

I’m not sure what went wrong, with Airflow docker, version 2.1.0 , below were the steps I performed but Marquez UI is showing no jobs, pls help:

+ +
  1. requirements.txt i added openlineage-airflow==0.5.1 . Then ran pip install -r requirements.txt .
  2. Added environmental variable inside my airflow docker folder using this command: +export AIRFLOW__LINEAGE__BACKEND = openlineage.lineage_backend.OpenLineageBackend
  3. Then configured HTTP Backend environment variables inside same airflow docker folder: +export OPENLINEAGE_URL=<http://localhost:5000>
  4. Ran Marquez using ./docker/up.sh  which is in another folder, Front end UI is not showing any job, its empty:
  5. Attached in the airflow DAG log.
    6. +
+ +
+
+ + + + + + +
+
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-01-25 14:46:58
 +
+

*Thread Reply:* Hm, that is odd. Usually there are a few lines in the DAG log from the OpenLineage bits. I’d expect to see something about not having an extractor for the operator you are using.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-01-25 14:47:53
 +
+

*Thread Reply:* If you open a shell in your Airflow Scheduler container and check for the presence of AIRFLOW__LINEAGE__BACKEND is it properly set? Possible the env isn’t making it all the way there.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Lena Kullab + (Lena.Kullab@storable.com) +
+
2022-01-21 13:38:37
 +
+

Hi All,

+ +

I am working on a POC of OpenLineage-Airflow integration and was attempting to get it configured with Amundsen (also working on a POC). Reading through the tutorial here https://openlineage.io/integration/apache-airflow/, under the Prerequisites section it says: +To use the OpenLineage Airflow integration, you'll need a running Airflow instance. You'll also need an OpenLineage compatible HTTP backend. +The example uses Marquez, but I was trying to figure out how to get it to send metadata to the Amundsen graph db backend. Does the Airflow integration only support configuration with an HTTP compatible backend?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2022-01-21 14:03:29
 +
+

*Thread Reply:* Hi Lena! That’s correct, Openlineage is designed to send events to an HTTP backend. There’s a ticket on the future section of the roadmap to support pushing to Amundsen, but it’s not yet been worked on (Ref: Roadmap Issue #86)

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/issues/86|#86 Add backend for sending lineage to Amundsen>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Lena Kullab + (Lena.Kullab@storable.com) +
+
2022-01-21 14:08:35
 +
+

*Thread Reply:* Thank you for the info!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
naman shaundik + (namanshaundik@gmail.com) +
+
2022-01-30 11:01:42
 +
+

hi , i am completely new to openlineage and marquez, i have to integrate openlineage to my existing java project but i am completely confused on where to start, i have gone through documentation and all but i am not able to understand how to integrate openlineage using marquez http backend in my existing project. please someone help me. I may sound naive here but i am in dire need of help.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2022-01-30 12:37:39
 +
+

*Thread Reply:* what do you mean by “Integrate Openlineage”?

+ +

Can you give a little more information on what you’re trying to accomplish and what the existing project is?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
naman shaundik + (namanshaundik@gmail.com) +
+
2022-01-31 03:49:22
 +
+

*Thread Reply:* I work in a datalake team and we are trying to implement data lineage property in our project using openlineage. our project basically keeps track of datasets coming from different sources(hive, redshift, elasticsearch etc.) and jobs.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2022-01-31 15:01:31
 +
+

*Thread Reply:* Gotcha!

+ +

Broadly speaking, all an integration needs to do is to send runEvents to Marquez.

+ +

I'd start by understanding the OpenLineage data model, and then looking at your system to identify when / where runEvents should be sent from, and what information needs to be included.

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.md | OpenLineage.md>
+
+ <p><strong>OpenLineage Spec</strong> +<strong>Specification</strong></p> + +<p>The specification for OpenLineage is formalized as a JsonSchema &lt;OpenLineage.json|OpenLineage.json&gt;. <br /> +An OpenAPI spec is also provided for HTTP based implementations: &lt;OpenLineage.yml|OpenLineage.yml&gt; <br /> +The documentation is published at: <a href="https://openlineage.github.io/">https://openlineage.github.io/</a> <br /> +It allows extensions to the spec using <code>Custom Facets</code> as described in this document.</p> + +<p><strong>Core concepts</strong> +<strong>Core Lineage Model</strong></p> + +<p>&lt;OpenLineageModel.svg|Open Lineage model&gt;</p> + +<p>• <strong>Run Event</strong>: an event describing an observed state of a job run. It is required to at least send a START event and a COMPLETE/FAIL/ABORT event. Additional events are optional. +• <strong>Job</strong>: a process definition that consumes and produces datasets (defined as its inputs and outputs). It is &lt;Naming.md#Jobs|identified by a unique name within a namespace&gt; (which is assigned to the scheduler starting the jobs). The <em>Job</em> evolves over time and this change is captured when the job runs. +• <strong>Dataset</strong>: an abstract representation of data. It has a &lt;Naming.md#Datasets|unique name within the datasource namespace&gt; derived from its physical location (for example db.host.database.schema.table). Typically, a <em>Dataset</em> changes when a job writing to it completes. Similarly to the <em>Job</em> and <em>Run</em> distinction, metadata that is more static from run to run is captured in a DatasetFacet (for example, the schema that does not change every run), what changes every <em>Run</em> is captured as an <em>InputFacet</em> or an <em>OutputFacet</em> (for example, what subset of the data set was read or written, like a time partition). +• <strong>Run</strong>: An instance of a running job with a start and completion (or failure) time. A run is identified by a globally unique ID relative to its job definition. A run ID <strong>must</strong> be an <a href="https://datatracker.ietf.org/doc/html/rfc4122">UUID</a>. +• <strong>Facet</strong>: A piece of metadata attached to one of the entities defined above.</p> + +<p>example: <br /> +Here is an example of a simple start run event not adding any facet information:</p> + +<pre><code>{ + "eventType": "START", + "eventTime": "2020-12-09T23:37:31.081Z", + "run": { + "runId": "3b452093-782c-4ef2-9c0c-aafe2aa6f34d", + }, + "job": { + "namespace": "my-scheduler-namespace", + "name": "myjob.mytask", + }, + "inputs": [ + { + "namespace": "my-datasource-namespace", + "name": "instance.schema.table", + } + ], + "outputs": [ + { + "namespace": "my-datasource-namespace", + "name": "instance.schema.output_table", + } + ], + "producer": "&lt;https://github.com/OpenLineage/OpenLineage/blob/v1-0-0/client&gt;", + "schemaURL": "&lt;https://openlineage.io/spec/1-0-0/OpenLineage.json#/definitions/RunEvent&gt;" +} + +</code></pre> + +<p><strong>Lifecycle</strong></p> + +<p>The OpenLineage API defines events to capture the lifecycle of a <em>Run</em> for a given <em>Job</em>. <br /> +When a <em>job</em> is being <em>run</em>, we capture metadata by sending run events when the state of the job transitions to a different state. <br /> +We might observe different aspects of the job run at different stages. This means that different metadata might be collected in each event during the lifecycle of a run. <br /> +All metadata is additive. for example, if more inputs or outputs are detected as the job is running we might send additional events specifically for those datasets without re-emitting previously observed inputs or outputs. <br /> +Example:</p> + +<p>• When the run starts, we collect the following Metadata: + • Run Id + • Job id + • eventType: START + • event time + • source location and version (ex: git sha) + • If known: Job inputs and outputs. (input schema, ...) +• When the run completes: + • Run Id + • Job id + • eventType: COMPLETE + • event time + • Output datasets schema (and other metadata).</p> + +<p><strong>Facets</strong></p> + +<p>Facets are pieces of metadata that can be attached to the core entities:</p> + +<p>• Run +• Job +• Dataset (Inputs or Outputs)</p> + +<p>A facet is an atomic piece of metadata identified by its name. This means that emitting a new facet with the same name for the same entity replaces the previous facet instance for that entity entirely). It is defined as a JSON object that can be either part of the spec or custom facets defined in a different project.</p> + +<p>Custom facets must use a distinct prefix named after the project defining them to avoid collision with standard facets defined in the &lt;OpenLineage.json|OpenLineage.json&gt; spec. <br /> +They have a _schemaURL field pointing to the corresponding version of the facet schema (as a JSONPointer: <a href="https://swagger.io/docs/specification/using-ref/">$ref URL location</a> ).</p> + +<p>Example: <a href="https://raw.githubusercontent.com/OpenLineage/OpenLineage/main/spec/OpenLineage.json#/definitions/MyCustomJobFacet">https://raw.githubusercontent.com/OpenLineage/OpenLineage/main/spec/OpenLineage.json#/definitions/MyCustomJobFacet</a></p> + +<p>The versioned URL must be an immutable pointer to the version of the facet schema. For example, it should include a tag of a git sha and not a branch name. This should also be a canonical URL. There should be only one URL used for a given version of a schema.</p> + +<p>Custom facets can be promoted to the standard by including them in the spec.</p> + +<p><strong>Custom Facet Naming</strong></p> + +<p>Naming of custom facets should follow pattern <code>{prefix}{name}{entity}Facet</code> PascalCased. <br /> +Prefix must be distinct identifier named after the project defining them to avoid colision with standard facets defined in the &lt;OpenLineage.json|OpenLineage.json&gt; spec. <br /> +Entity is the core entity for which the facet is attached.</p> + +<p>When attached to core entity, the key should follow pattern <code>{prefix}_{name}</code>, where both prefix and name follow snakeCase pattern.</p> + +<p>Example of valid name is <code>BigQueryStatisticsJobFacet</code> and it's key <code>bigQuery_statistics</code>.</p> + +<p><strong>Standard Facets</strong> +<strong>Run Facets</strong></p> + +<p>• <strong>nominalTime</strong>: Captures the time this run is scheduled for. This is a typical usage for time based scheduled job. The job has a nominal schedule time that will be different from the actual time it is running at. +• <strong>parent</strong>: Captures the parent job and Run when the run was spawn from a parent run. For example in the case of Airflow, there's a run for the DAG that then spawns runs for individual tasks that would refer to the parent run as the DAG run. Similarly when a SparkOperator starts a Spark job, this creates a separate run that refers to the task run as its parent.</p> + +<p><strong>Job Facets</strong></p> + +<p>• <strong>sourceCodeLocation</strong>: Captures the source code location and version (example: git sha) of the job. +• <strong>sql</strong>: Capture the SQL query if this job is a SQL query.</p> + +<p><strong>Dataset Facets</strong></p> + +<p>• <strong>schema</strong>: Captures the schema of the dataset +• <strong>dataSource</strong>: Captures the Database instance containing this datasets (ex: Database schema. Object store bucket, ...)</p> + +<p><strong>Input Dataset Facets</strong></p> + +<p>• <strong>dataQualityMetrics</strong>: Captures dataset level and column level data quality metrics when scanning a dataset whith a DataQuality library (row count, byte size, null count, distinct count, average, min, max, quantiles). +• <strong>dataQualityAssertions</strong>: Captures the result of running data tests on dataset or its columns.</p> + +<p><strong>Output Dataset Facets</strong></p> + +<p>• <strong>outputStatistics</strong>: Captures the size of the output written to a dataset (row count and byte size).</p> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
TJ Tang + (tj@tapdata.io) +
+
2022-02-15 15:28:03
 +
+

*Thread Reply:* I suppose OpenLineage itself only defines the standard/protocol to design your data model. To be able to visualize/trace the lineage, you either have to implement your self with the standard data models or including Marquez in your project. You would need to use HTTP API to send lineage events from your Java project to Marquez in this case.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2022-02-16 11:17:13
 +
+

*Thread Reply:* Exactly! This project also includes connectors for more common data tools (Airflow, dbt, spark, etc), but at it's core OpenLineage is a standard and protocol

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-02-02 19:55:13
 +
+

The next OpenLineage Technical Steering Committee meeting is Wednesday, February 9. Meetings are on the second Wednesday of each month from 9:00 to 10:00am PT. +Join us on Zoom: https://us02web.zoom.us/j/81831865546?pwd=RTladlNpc0FTTDlFcWRkM2JyazM4Zz09 +All are welcome. Agenda items are always welcome, as well. Reply in thread with yours. +Current agenda: +• OpenLineage 0.5.1 release +• Apache Flink effort +• Dagster integration +• Open Discussion +Notes: https://tinyurl.com/openlineagetsc

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jensen Yap + (jensen@contxts.io) +
+
2022-02-03 00:33:45
 +
+

Hi everybody!

+ + + +
+ 👋 Maciej Obuchowski, John Thomas +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2022-02-03 12:39:57
 +
+

*Thread Reply:* Hello!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Albert Bikeev + (albert.bikeev@gmail.com) +
+
2022-02-04 09:36:46
 +
+

Hi everybody! +Very cool initiative, thank you! Is there any traction on Apache Atlas integration? Is there some way to help you there?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2022-02-04 15:07:07
 +
+

*Thread Reply:* Hey Albert! There aren't yet any issues or proposals around Apache Atlas yet, but that's definitely something you can help with!

+ +

I'm not super familiar with Atlas, were you thinking in terms of enabling Atlas to receive runEvents from OpenLineage connectors?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Albert Bikeev + (albert.bikeev@gmail.com) +
+
2022-02-07 05:49:16
 +
+

*Thread Reply:* Hi John! +Yes, exactly, it’d be nice to see Atlas as a receiver side of the OpenLineage events. Is there some guidelines on how to implement it? I guess we need OpenLineage-compatible server implementation so we could receive events and send them to Atlas, right?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2022-02-07 11:30:14
 +
+

*Thread Reply:* exactly - This would be a change on the Atlas side. I’d start by opening an issue in the atlas repo about making an API endpoint that can receive OpenLineage events. +Marquez is our reference implementation of OpenLineage, so I’d look around in that repo to see how it’s been implemented :)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Albert Bikeev + (albert.bikeev@gmail.com) +
+
2022-02-07 11:50:27
 +
+

*Thread Reply:* Got it, thanks! Did that: https://issues.apache.org/jira/browse/ATLAS-4550 +If it’d not get any traction we at New Work might contribute as well

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2022-02-07 11:56:09
 +
+

*Thread Reply:* awesome! if you guys have any questions, reach out and I can get you in touch with some of the engineers on our end

+ + + +
+ 👍 Albert Bikeev +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-02-08 11:20:47
 +
+

*Thread Reply:* @Albert Bikeev one minor thing that could be helpful: java OpenLineage library contains server model classes: https://github.com/OpenLineage/OpenLineage/pull/300#issuecomment-923489097

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/pull/300#issuecomment-923489097|Comment on #300 server model>
+
+ <p>the new classes look like this:</p> + +<p>``` +package io.openlineage.server;</p> + +<p>import com.fasterxml.jackson.annotation.JsonAnyGetter; +import com.fasterxml.jackson.annotation.JsonAnySetter; +import com.fasterxml.jackson.annotation.JsonCreator; +import com.fasterxml.jackson.annotation.JsonProperty; +import com.fasterxml.jackson.annotation.JsonPropertyOrder; +import java.lang.Object; +import java.lang.String; +import java.net.URI; +import java.time.ZonedDateTime; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; +import java.util.UUID;</p> + +<p>public final class OpenLineage { + @JsonPropertyOrder + public static final class RunFacets { + @JsonAnySetter + private final Map&lt;String, RunFacet&gt; additionalProperties;</p> + +<pre><code>@JsonCreator +public RunFacets() { + this.additionalProperties = new LinkedHashMap&amp;lt;&amp;gt;(); +} + +/**** + ** @return additional properties + **/ +@JsonAnyGetter +public Map&amp;lt;String, RunFacet&amp;gt; getAdditionalProperties() { + return additionalProperties; +} +</code></pre> + +<p>}</p> + +<p>@JsonPropertyOrder({ + "eventType", + "eventTime", + "run", + "job", + "inputs", + "outputs", + "producer", + "schemaURL" + }) + public static final class RunEvent { + private final String eventType;</p> + +<pre><code>private final ZonedDateTime eventTime; + +private final Run run; + +private final Job job; + +private final List&amp;lt;InputDataset&amp;gt; inputs; + +private final List&amp;lt;OutputDataset&amp;gt; outputs; + +private final URI producer; + +private final URI schemaURL; + +@JsonAnySetter +private final Map&amp;lt;String, Object&amp;gt; additionalProperties; + +/**** + ** @param eventType the current transition of the run state. It is required to issue 1 START event and 1 of [ COMPLETE, ABORT, FAIL ] event per run. Additional events with OTHER eventType can be added to the same run. For example to send additional metadata after the run is complete + ** @param eventTime the time the event occurred at + ** @param run the run + ** @param job the job + ** @param inputs The set of ****input**** datasets. + ** @param outputs The set of ****output**** datasets. + ** @param producer URI identifying the producer of this metadata. For example this could be a git url with a given tag or sha + ** @param schemaURL The JSON Pointer (&lt;https://tools.ietf.org/html/rfc6901&gt;) URL to the corresponding version of the schema definition for this RunEvent + **/ +@JsonCreator +public RunEvent(@JsonProperty("eventType") String eventType, + @JsonProperty("eventTime") ZonedDateTime eventTime, @JsonProperty("run") Run run, + @JsonProperty("job") Job job, @JsonProperty("inputs") List&amp;lt;InputDataset&amp;gt; inputs, + @JsonProperty("outputs") List&amp;lt;OutputDataset&amp;gt; outputs, + @JsonProperty("producer") URI producer, @JsonProperty("schemaURL") URI schemaURL) { + this.eventType = eventType; + this.eventTime = eventTime; + this.run = run; + this.job = job; + this.inputs = inputs; + this.outputs = outputs; + this.producer = producer; + this.schemaURL = schemaURL; + this.additionalProperties = new LinkedHashMap&amp;lt;&amp;gt;(); +} + +/**** + ** @return the current transition of the run state. It is required to issue 1 START event and 1 of [ COMPLETE, ABORT, FAIL ] event per run. Additional events with OTHER eventType can be added to the same run. For example to send additional metadata after the run is complete + **/ +public String getEventType() { + return eventType; +} + +/**** + ** @return the time the event occurred at + **/ +public ZonedDateTime getEventTime() { + return eventTime; +} + +public Run getRun() { + return run; +} + +public Job getJob() { + return job; +} + +/**** + ** @return The set of ****input**** datasets. + **/ +public List&amp;lt;InputDataset&amp;gt; getInputs() { + return inputs; +} + +/**** + ** @return The set of ****output**** datasets. + **/ +public List&amp;lt;OutputDataset&amp;gt; getOutputs() { + return outputs; +} + +/**** + ** @return URI identifying the producer of this metadata. For example this could be a git url with a given tag or sha + **/ +public URI getProducer() { + return producer; +} + +/**** + ** @return The JSON Pointer (&lt;https://tools.ietf.org/html/rfc6901&gt;) URL to the corresponding version of the schema definition for this RunEvent + **/ +public URI getSchemaURL() { + return schemaURL; +} + +/**** + ** @return additional properties + **/ +@JsonAnyGetter +public Map&amp;lt;String, Object&amp;gt; getAdditionalProperties() { + return additionalProperties; +} +</code></pre> + +<p>}</p> + +<p>@JsonPropertyOrder + public static final class JobFacets { + @JsonAnySetter + private final Map&lt;String, JobFacet&gt; additionalProperties;</p> + +<pre><code>@JsonCreator +public JobFacets() { + this.additionalProperties = new LinkedHashMap&amp;lt;&amp;gt;(); +} + +/**** + ** @return additional properties + **/ +@JsonAnyGetter +public Map&amp;lt;String, JobFacet&amp;gt; getAdditionalProperties() { + return additionalProperties; +} +</code></pre> + +<p>}</p> + +<p>@JsonPropertyOrder + public static final class InputDatasetInputFacets { + @JsonAnySetter + private final Map&lt;String, InputDatasetFacet&gt; additionalProperties;</p> + +<pre><code>@JsonCreator +public InputDatasetInputFacets() { + this.additionalProperties = new LinkedHashMap&amp;lt;&amp;gt;(); +} + +/**** + ** @return additional properties + **/ +@JsonAnyGetter +public Map&amp;lt;String, InputDatasetFacet&amp;gt; getAdditionalProperties() { + return additionalProperties; +} +</code></pre> + +<p>}</p> + +<p>@JsonPropertyOrder({ + "<em>producer", + "</em>schemaURL" + }) + public static final class DatasetFacet implements BaseFacet { + private final URI _producer;</p> + +<pre><code>private final URI _schemaURL; + +@JsonAnySetter +private final Map&amp;lt;String, Object&amp;gt; additionalProperties; + +/**** + ** @param _producer URI identifying the producer of this metadata. For example this could be a git url with a given tag or sha + ** @param _schemaURL The JSON Pointer (&lt;https://tools.ietf.org/html/rfc6901&gt;) URL to the corresponding version of the schema definition for this facet + **/ +@JsonCreator +public DatasetFacet(@JsonProperty("_producer") URI _producer, + @JsonProperty("_schemaURL") URI _schemaURL) { + this._producer = _producer; + this._schemaURL = _schemaURL; + this.additionalProperties = new LinkedHashMap&amp;lt;&amp;gt;(); +} + +/**** + ** @return URI identifying the producer of this metadata. For example this could be a git url with a given tag or sha + **/ +public URI get_producer() { + return _producer; +} + +/**** + ** @return The JSON Pointer (&lt;https://tools.ietf.org/html/rfc6901&gt;) URL to the corresponding version of the schema definition for this facet + **/ +public URI get_schemaURL() { + return _schemaURL; +} + +/**** + ** @return additional properties + **/ +@JsonAnyGetter +public Map&amp;lt;String, Object&amp;gt; getAdditionalProperties() { + return additionalProperties; +} +</code></pre> + +<p>}</p> + +<p>@JsonPropertyOrder({ + "<em>producer", + "</em>schemaURL" + }) + public static final class OutputDatasetFacet implements BaseFacet { + private final URI _producer;</p> + +<pre><code>private final URI _schemaURL; + +@JsonAnySetter +private final Map&amp;lt;String, Object&amp;gt; additionalProperties; + +/**** + ** @param _producer URI identifying the producer of this metadata. For example this could be a git url with a given tag or sha + ** @param _schemaURL The JSON Pointer (&lt;https://tools.ietf.org/html/rfc6901&gt;) URL to the corresponding version of the schema definition for this facet + **/ +@JsonCreator +public OutputDatasetFacet(@JsonProperty("_producer") URI _producer, + @JsonProperty("_schemaURL") URI _schemaURL) { + this._producer = _producer; + this._schemaURL = _schemaURL; + this.additionalProperties = new LinkedHashMap&amp;lt;&amp;gt;(); +} + +/**** + ** @return URI identifying the producer of this … +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Albert Bikeev + (albert.bikeev@gmail.com) +
+
2022-02-08 11:32:12
 +
+

*Thread Reply:* Got it, thank you!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Juan Carlos Fernández Rodríguez + (jcfernandez@keedio.com) +
+
2022-05-04 11:12:23
 +
+

*Thread Reply:* This is a quite old discussion, but isn't possible to use openlineage proxy to send json to kafka topic and let Atlas read that json without any modification? +It would be needed to create a new model for spark, other than https://github.com/apache/atlas/blob/release-2.1.0-rc3/addons/models/1000-Hadoop/1100-spark_model.json and upload it to atlas (what could be done with a call to the atlas Api) +Does it makes sense?

+
+ + + + +
<https://github.com/apache/atlas/blob/release-2.1.0-rc3/addons/models/1000-Hadoop/1100-spark_model.json | 1100-spark_model.json>
+
+ <p>``` +{ + "enumDefs": [], + "structDefs": [], + "classificationDefs": [], + "entityDefs": [ + { + "name": "spark<em>db", + "superTypes": [ + "DataSet" + ], + "serviceType": "spark", + "typeVersion": "1.0", + "attributeDefs": [ + { + "name": "location", + "typeName": "string", + "isOptional": true, + "cardinality": "SINGLE", + "isUnique": false, + "isIndexable": false, + "searchWeight": 5 + }, + { + "name": "clusterName", + "typeName": "string", + "isOptional": true, + "cardinality": "SINGLE", + "isUnique": false, + "isIndexable": false, + "searchWeight": 8 + }, + { + "name": "parameters", + "typeName": "map&lt;string,string&gt;", + "isOptional": true, + "cardinality": "SINGLE", + "isUnique": false, + "isIndexable": false + }, + { + "name": "ownerType", + "typeName": "string", + "isOptional": true, + "cardinality": "SINGLE", + "isUnique": false, + "isIndexable": false + } + ] + }, + { + "name": "spark</em>table", + "superTypes": [ + "DataSet" + ], + "serviceType": "spark", + "typeVersion": "1.0", + "attributeDefs": [ + { + "name": "tableType", + "typeName": "string", + "isOptional": true, + "cardinality": "SINGLE", + "isUnique": false, + "isIndexable": false + }, + { + "name": "provider", + "typeName": "string", + "isOptional": true, + "cardinality": "SINGLE", + "isUnique": false, + "isIndexable": false, + "searchWeight": 5 + }, + { + "name": "partitionColumnNames", + "typeName": "array&lt;string&gt;", + "isOptional": true, + "cardinality": "SINGLE", + "isUnique": false, + "isIndexable": false + }, + { + "name": "bucketSpec", + "typeName": "map&lt;string,string&gt;", + "isOptional": true, + "cardinality": "SINGLE", + "isUnique": false, + "isIndexable": false + }, + { + "name": "ownerType", + "typeName": "string", + "isOptional": true, + "cardinality": "SINGLE", + "isUnique": false, + "isIndexable": false + }, + { + "name": "createTime", + "typeName": "date", + "isOptional": true, + "cardinality": "SINGLE", + "isUnique": false, + "isIndexable": false + }, + { + "name": "parameters", + "typeName": "map&lt;string,string&gt;", + "isOptional": true, + "cardinality": "SINGLE", + "isUnique": false, + "isIndexable": false + }, + { + "name": "comment", + "typeName": "string", + "isOptional": true, + "cardinality": "SINGLE", + "isUnique": false, + "isIndexable": false, + "searchWeight": 9 + }, + { + "name": "unsupportedFeatures", + "typeName": "array&lt;string&gt;", + "isOptional": true, + "cardinality": "SINGLE", + "isUnique": false, + "isIndexable": false + }, + { + "name": "viewOriginalText", + "typeName": "string", + "isOptional": true, + "cardinality": "SINGLE", + "isUnique": false, + "isIndexable": false, + "searchWeight": 9 + }, + { + "name": "schemaDesc", + "typeName": "string", + "isOptional": true, + "cardinality": "SINGLE", + "isUnique": false, + "isIndexable": false, + "searchWeight": 5 + }, + { + "name": "partitionProvider", + "typeName": "string", + "isOptional": true, + "cardinality": "SINGLE", + "isUnique": false, + "isIndexable": false + } + ] + }, + { + "name": "spark<em>column", + "superTypes": [ + "DataSet" + ], + "serviceType": "spark", + "typeVersion": "1.0", + "attributeDefs": [ + { + "name": "type", + "typeName": "string", + "isOptional": false, + "cardinality": "SINGLE", + "isUnique": false, + "isIndexable": true + }, + { + "name": "nullable", + "typeName": "boolean", + "isOptional": true, + "cardinality": "SINGLE", + "isUnique": false, + "isIndexable": false + }, + { + "name": "metadata", + "typeName": "string", + "isOptional": true, + "cardinality": "SINGLE", + "isUnique": false, + "isIndexable": false + }, + { + "name": "comment", + "typeName": "string", + "isOptional": true, + "cardinality": "SINGLE", + "isUnique": false, + "isIndexable": false, + "searchWeight": 9 + } + ] + }, + { + "name": "spark</em>ml<em>directory", + "superTypes": [ + "DataSet" + ], + "serviceType": "spark", + "typeVersion": "1.0", + "attributeDefs": [ + { + "name": "uri", + "typeName": "string", + "isOptional": false, + "cardinality": "SINGLE", + "isUnique": false, + "isIndexable": true, + "searchWeight": 10 + }, + { + "name": "directory", + "typeName": "string", + "isOptional": false, + "cardinality": "SINGLE", + "isUnique": false, + "isIndexable": true, + "searchWeight": 10 + } + ] + }, + { + "name": "spark</em>storagedesc", + "superTypes": [ + "Referenceable" + ], + "serviceType": "spark", + "typeVersion": "1.0", + "attributeDefs": [ + { + "name": "location", + "typeName": "string", + "isOptional": true, + "cardinality": "SINGLE", + "isUnique": false, + "isIndexable": false, + "searchWeight": 10 + }, + { + "name": "inputFormat", + "typeName": "string", + "isOptional": true, + "cardinality": "SINGLE", + "isUnique": false, + "isIndexable": false + }, + { + "name": "outputFormat", + "typeName": "string", + "isOptional": true, + "cardinality": "SINGLE", + "isUnique": false, + "isIndexable": false + }, + { + "name": "serde", + "typeName": "string", + "isOptional": true, + "cardinality": "SINGLE", + "isUnique": false, + "isIndexable": false + }, + { + "name": "compressed", + "typeName": "boolean", + "isOptional": false, + "cardinality": "SINGLE", + "isUnique": false, + "isIndexable": true + }, + { + "name": "parameters", + "typeName": "map&lt;string,string&gt;", + "isOptional": true, + "cardinality": "SINGLE", + "isUnique": false, + "isIndexable": false + } + ] + }, + { + "name": "spark<em>ml</em>model", + "superTypes": [ + "DataSet" + ], + "serviceType": "spark", + "typeVersion": "1.0", + "attributeDefs": [ + { + "name": "extra", + "typeName": "string", + "isOptional": true, + "cardinality": "SINGLE", + "isUnique": false, + "isIndexable": false + } + ] + }, + { + "name": "spark_application", + "superTypes": [ + "Process" + ], + "serviceType": "spark", + "typeVersion": "1.0", + "attributeDefs": [ + { + "name": "currentUser", + "typeName": "string", + "isOptional": true, + "cardinality": "SINGLE", + "isUniq…</p> +
+ + + + + + + +
+ + <a href="https://github.com/apache/atlas">apache/atlas</a> +
+ + +
+ + + +
+ 👍 Albert Bikeev +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-05-04 11:24:02
 +
+

*Thread Reply:* @Juan Carlos Fernández Rodríguez - You still need to build a bridge between the OpenLineage Spec and the Apache Atlas entity JSON. So far, no one has contributed something like that to the open source community... yet!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Juan Carlos Fernández Rodríguez + (jcfernandez@keedio.com) +
+
2022-05-04 14:24:28
 +
+

*Thread Reply:* sorry for the ignorance, +But what is the purpose of the bridge?the comunicación with atlas should be done throw kafka, and that messages can be sent by the proxy. What are I missing?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2022-05-04 16:37:33
 +
+

*Thread Reply:* "bridge" in this case refers to a service of some sort that converts from OpenLineage run event to Atlas entity JSON, since there's currently nothing that will do that

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
xiang chen + (cdmikechen@hotmail.com) +
+
2022-05-19 09:08:23
 +
+

*Thread Reply:* If OpenLineage send an event to kafka, I think we can use kafka stream or kafka connect to rebuild message to atlas event.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
xiang chen + (cdmikechen@hotmail.com) +
+
2022-05-19 09:11:37
 +
+

*Thread Reply:* @John Thomas Our company used to use atlas as a metadata service. I just came into know this project. After I learned how openlineage works, I think I can create an issue to describe my design first.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
xiang chen + (cdmikechen@hotmail.com) +
+
2022-05-19 09:13:36
 +
+

*Thread Reply:* @Juan Carlos Fernández Rodríguez If you already have some experience and design, can you directly create an issue so that we can discuss it in more detail ?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Juan Carlos Fernández Rodríguez + (jcfernandez@keedio.com) +
+
2022-05-19 12:42:31
 +
+

*Thread Reply:* Hi @xiang chen we are discussing internally in my company if rewrite to atlas or another alternative. If we do this, we will share and could involve you in some way.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-02-04 15:02:29
 +
+

Who here is working with OpenLineage at Dagster or Flink? We would love to hear about your work at the next on February 9 at 9 a.m. PT. Please reply here or message me to coordinate. @Ziyoiddin Yusupov

+ + + +
+ 👍 Ziyoiddin Yusupov +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Luca Soato + (lucasoato@gmail.com) +
+
2022-02-04 19:18:24
 +
+

Hi everyone, +OpenLineage is wonderful, we really needed something like this! +Has anyone else used it with Databricks, Delta tables or Spark? If someone is interested into these technologies we can work together to get a POC and share some thoughts. +Thanks and have a nice weekend! :)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julius Rentergent + (julius.rentergent@thetradedesk.com) +
+
2022-02-25 13:06:16
 +
+

*Thread Reply:* Hi Luca, I agree this looks really promising. I’m working on getting it to run on Databricks, but I’m only just starting out 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-02-08 12:00:02
 +
+

Friendly reminder: this month’s OpenLineage TSC meeting is tomorrow! https://openlineage.slack.com/archives/C01CK9T7HKR/p1643849713216459

+
+ + +
+ + + } + + Michael Robinson + (https://openlineage.slack.com/team/U02LXF3HUN7) +
+ + + +
+
+ <p>The next OpenLineage Technical Steering Committee meeting is Wednesday, February 9. Meetings are on the second Wednesday of each month from 9:00 to 10:00am PT. +Join us on Zoom: <a href="https://us02web.zoom.us/j/81831865546?pwd=RTladlNpc0FTTDlFcWRkM2JyazM4Zz09">https://us02web.zoom.us/j/81831865546?pwd=RTladlNpc0FTTDlFcWRkM2JyazM4Zz09</a> +All are welcome. Agenda items are always welcome, as well. Reply in thread with yours. +Current agenda: +• OpenLineage 0.5.1 release +• Apache Flink effort +• Dagster integration +• Open Discussion +Notes: <a href="https://tinyurl.com/openlineagetsc">https://tinyurl.com/openlineagetsc</a></p> +
+ + + + + + +
Original URL: https://openlineage.slack.com/archives/C01CK9T7HKR/p1643849713216459
+ + +
+ + Slack Conversation +
+ + +
+ + + +
+ ❤️ Kevin Mellott, John Thomas +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Albert Bikeev + (albert.bikeev@gmail.com) +
+
2022-02-10 08:22:28
 +
+

Hi people, +One question regarding error reporting - what is the mechanism for that? E.g. if I send duplicated job to Openlineage, is there a way to notify me about that?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-02-10 09:05:39
 +
+

*Thread Reply:* By duplicated, you mean with the same runId?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Albert Bikeev + (albert.bikeev@gmail.com) +
+
2022-02-10 11:40:55
 +
+

*Thread Reply:* It’s only one example, could be also duplicated job name or anything else. The question is if there is mechanism to report that

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-02-14 17:21:20
 +
+

Reducing the Logging of Spark Integration

+ +

Hey, OpenLineage community! I'm curious if there are any quick tricks / fixes to reduce the amount of logging happening in the OpenLineage Spark Integration. Each job seems to print out the Logical Plan with INFO level logging. The default behavior of Databricks is to print out INFO level logs and so it gets pretty cluttered and noisy.

+ +

I'm hoping there's a feature flag that would help me shut off those kind of logs in OpenLineage's Spark integration 🤞

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-02-15 05:15:12
 +
+

*Thread Reply:* I think this log should be dropped to debug: https://github.com/OpenLineage/OpenLineage/blob/d66c41872f3cc7f7cd5c99664d401e070e[…]c/main/common/java/io/openlineage/spark/agent/EventEmitter.java

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/d66c41872f3cc7f7cd5c99664d401e070e60ff48/integration/spark/src/main/common/java/io/openlineage/spark/agent/EventEmitter.java | EventEmitter.java>
+
+ <pre><code> &lt;a href="http://log.info"&gt;log.info&lt;/a&gt;("Lineage completed successfully: {} {}", resp, mapper.writeValueAsString(event)); +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-02-15 23:27:07
 +
+

*Thread Reply:* @Maciej Obuchowski that is a good one! It would be nice to still have SOME logging in info to know that the event complete successfully but that response and event is very verbose.

+ +

I was also thinking about here: +https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/src/main/common/java/io/openlineage/spark/agent/lifecycle/OpenLineageRunEventBuilder.java#L337-L340

+ +

and here: +https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/src/main/common/java/io/openlineage/spark/agent/lifecycle/OpenLineageRunEventBuilder.java#L405-L408

+ +

These spots are where it's printing out the full logical plan for some reason.

+ +

Can I just open up a PR and switch these to log.debug instead?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-02-16 04:59:17
 +
+

*Thread Reply:* Yes, that would be good solution for now. Later would be nice to have some option to raise the log level - OL logs are absolutely drowning in logs from rest of Spark cluster when set to debug.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-02-16 13:35:15
 +
+

[SPARK][INTEGRATION] Need Brainstorming Ideas - How to Persist / Access Spark Configs in JobEnd

+ +

Hey, OL community! I'm working on PR#490 and I finally have all tests passing but now my desired behavior - display environment properties during COMPLETE / JobEnd events - is not happening 😭

+ +

The previous approach stored the spark properties in the OpenLineageContext with a properties attribute but that was part of all of the test failures I believe.

+ +

What are some other ways to store the jobStart's properties and make them accessible to the corresponding jobEnd? Hopefully it's okay to tag @Maciej Obuchowski, @Michael Collado, and @Paweł Leszczyński who have been extremely helpful in the past and brought great ideas to the table.

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/pull/490|#490 Added generic facet to collect environmental properties (EnvironmentFacet)>
+
+ <p><strong>Problem</strong></p> + +<p>👋 Introduce the ability to produce environmental properties using EnvironmentFacet, this facet can be used to work with any environment, based on the conditional check to verify environment EnvironmentFacet can be added.</p> + +<p>Closes: <a href="https://github.com/OpenLineage/OpenLineage/issues/464">#464</a></p> + +<p><strong>Solution</strong></p> + +<p>Generic implementation of EnvironmentFacet gives the ability to add environmental details. Since EnvironmentFacet produces HashMap it can handle properties dynamically.</p> + +<p>In the current implementation added ability to collect properties for Databricks environment including mount point details. <br /> +Below are some examples:</p> + +<p>• "orgId" +• "spark.databricks.clusterUsageTags.clusterOwnerOrgId" +• "spark.databricks.notebook.path" +• "spark.databricks.job.type" +• "spark.databricks.job.id" +• "spark.databricks.job.runId" +• "user" +• "userId" +• "spark.databricks.clusterUsageTags.clusterName" +• "spark.databricks.clusterUsageTags.azureSubscriptionId"</p> + +<p><strong>Checklist</strong></p> + +<p>☑︎ You've <a href="https://github.com/OpenLineage/OpenLineage/blob/main/why-the-dco.md">signed-off</a> your work +☑︎ Your pull request title follows our <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CONTRIBUTING.md#creating-pull-requests">guidelines</a> +☐ Your changes are accompanied by tests (<em>if relevant</em>) +☑︎ Your change contains a <a href="https://kurtisnusbaum.medium.com/stacked-diffs-keeping-phabricator-diffs-small-d9964f4dcfa6">small diff</a> and is self-contained +☐ You've updated any relevant documentation (<em>if relevant</em>) +☐ You've updated the <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md"><code>CHANGELOG.md</code></a> with details about your change under the "Unreleased" section (<em>if relevant, depending on the change, this may not be necessary</em>) +☐ You've versioned the core OpenLineage model or facets according to <a href="https://docs.snowplowanalytics.com/docs/pipeline-components-and-applications/iglu/common-architecture/schemaver">SchemaVer</a> (<em>if relevant</em>)</p> +
+ +
+
Comments
+ 6 +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2022-02-16 13:44:30
 +
+

*Thread Reply:* Hey, I responded on the issue, but just to make it clear for everyone, the OL events for a run are not expected to be an accumulation of all past events. Events should be treated as additive by the backend - each event can post what information it has about the run and the backend is responsible for constructing a holistic picture of the run

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/pull/490#issuecomment-1042011803|Comment on #490 Added generic facet to collect environmental properties (EnvironmentFacet)>
+
+ <p>&gt; how can I persist this information over to the COMPLETE events coming from OpenLineage?</p> + +<p>The different events are intended to be additive - it is not expected that each event is a cumulative representation of the previous events. If you send the environment facet on the <code>START</code> event, it will still be present for the run even if it's missing from the <code>COMPLETE</code>.</p> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2022-02-16 13:47:18
 +
+

*Thread Reply:* e.g., here is the marquez code that fetches the facets for a run. Note that all of the facets are included from all events with the requested run_uuid. If the env facet is present on any event, it will be returned by the API

+
+ + + + +
<https://github.com/MarquezProject/marquez/blob/main/api/src/main/java/marquez/db/RunDao.java | RunDao.java>
+
+ <pre><code> + "(\n" + + " SELECT le.run_uuid, JSON_AGG(event-&amp;gt;'run'-&amp;gt;'facets') AS facets\n" + + " FROM lineage_events le\n" + + " GROUP BY le.run_uuid\n" + + ") AS f ON r.uuid=f.run_uuid\n" +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/MarquezProject/marquez">MarquezProject/marquez</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-02-16 13:51:30
 +
+

*Thread Reply:* Ah! Thanks for that @Michael Collado it's good to understand the OpenLineage perspective.

+ +

So, we do need to maintain some state. That makes total sense, Mike.

+ +

How does Marquez handle failed jobs currently? Based on this issue (https://github.com/OpenLineage/OpenLineage/issues/436) I think Marquez would show a START but no COMPLETE event, right?

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/issues/436|#436 [INTEGRATION][Spark] use executionFailure on SparkListenerSQLExecutionEnd to indicate failed execution >
+
+ <p>In Spark 3, <code>SparkListenerSQLExecutionEnd</code> has private property that allows us to find out if job failed. We should use it if we're using Spark 3.</p> + +<p><a href="https://github.com/OpenLineage/OpenLineage/pull/393#discussion_r768049758">#393 (comment)</a></p> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2022-02-16 14:00:03
 +
+

*Thread Reply:* If I were building the backend, I would store events, then calculate the end state later, rather than trying to "maintain some state" (maybe we mean the same thing, but using different words here 😀). +Re: the failure events, I think job failures will currently result in one FAIL event and one COMPLETE event. The SparkListenerJobEnd event will trigger a FAIL event but the SparkListenerSQLExecutionEnd event will trigger the COMPLETE event.

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/src/main/common/java/io/openlineage/spark/agent/lifecycle/OpenLineageRunEventBuilder.java | OpenLineageRunEventBuilder.java>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/src/main/common/java/io/openlineage/spark/agent/lifecycle/OpenLineageRunEventBuilder.java | OpenLineageRunEventBuilder.java>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-02-16 15:16:27
 +
+

*Thread Reply:* Oooh! I did not know we already could get a FAIL event! That is super helpful to know, Mike! Thank you so much!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-02-21 10:04:18
 +
+

[SPARK] Connecting SparkListenerSQLExecutionStart to the various SparkListenerJobStarts

+ +

TL;DR: How can I connect the SparkListenerSQLExecutionStart to the SparkListenerJobStart events coming out of OpenLineage? The events appear to have two separate run ids and no link to indicate that the ExecutionStart event owns the subsequent JobStart events.

+ +

More Context:

+ +

Recently, I implemented a connector for Azure Synapse (data warehouse on the Microsoft cloud) for the Spark integration and now with https://github.com/OpenLineage/OpenLineage/pull/490, I realize now that the SparkListenerSQLExecutionStart events carries with it the necessary inputs and outputs to tell the "real" lineage. The way the Synapse in Databricks works is:

+ +

• SparkListenerSQLExecutionStart fires off an event with the end to end input and output (e.g. S3 as input and SQL table as output) +• SparkListenerJobStart events fire off that move content from one S3 location to a "staging" location controlled by Azure Synapse. OpenLineage records this event with INPUT S3 and output is a WASB "tempfolder" (which is a temporary locatio and not really useful for lineage since it will be destroyed at the end of the job) +• The final operation actually happens ALL in Synapse and OpenLineage does not fire off an event it seems. The Synapse database has a "COPY" command which moves the data from "tempfolder" in to the database. +• Finally a SparkListenerSQLExecutionEnd event happens and the query is complete. +Ideally, I could connect the SQLExecutionStart of SQLExecutionEnd with the SparkListenerJobStart so that I can get the JobStart properties. I see that ExecutionStart has an execution id and JobStart should have the same Execution Id BUT I think by the time I reach the ExecutionEND, all the JobStart events would have been removed from the HashMap that contains all of the events in OpenLineage.

+ +

Any guidance on how to reach a JobStart properties from an ExecutionStart or ExecutionEnd would be greatly appreciated!

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/pull/490|#490 Added generic facet to collect environmental properties (EnvironmentFacet)>
+
+ <p><strong>Problem</strong></p> + +<p>👋 Introduce the ability to produce environmental properties using EnvironmentFacet, this facet can be used to work with any environment, based on the conditional check to verify environment EnvironmentFacet can be added.</p> + +<p>Closes: <a href="https://github.com/OpenLineage/OpenLineage/issues/464">#464</a></p> + +<p><strong>Solution</strong></p> + +<p>Generic implementation of EnvironmentFacet gives the ability to add environmental details. Since EnvironmentFacet produces HashMap it can handle properties dynamically.</p> + +<p>In the current implementation added ability to collect properties for Databricks environment including mount point details. <br /> +Below are some examples:</p> + +<p>• "orgId" +• "spark.databricks.clusterUsageTags.clusterOwnerOrgId" +• "spark.databricks.notebook.path" +• "spark.databricks.job.type" +• "spark.databricks.job.id" +• "spark.databricks.job.runId" +• "user" +• "userId" +• "spark.databricks.clusterUsageTags.clusterName" +• "spark.databricks.clusterUsageTags.azureSubscriptionId"</p> + +<p><strong>Checklist</strong></p> + +<p>☑︎ You've <a href="https://github.com/OpenLineage/OpenLineage/blob/main/why-the-dco.md">signed-off</a> your work +☑︎ Your pull request title follows our <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CONTRIBUTING.md#creating-pull-requests">guidelines</a> +☐ Your changes are accompanied by tests (<em>if relevant</em>) +☑︎ Your change contains a <a href="https://kurtisnusbaum.medium.com/stacked-diffs-keeping-phabricator-diffs-small-d9964f4dcfa6">small diff</a> and is self-contained +☐ You've updated any relevant documentation (<em>if relevant</em>) +☐ You've updated the <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md"><code>CHANGELOG.md</code></a> with details about your change under the "Unreleased" section (<em>if relevant, depending on the change, this may not be necessary</em>) +☐ You've versioned the core OpenLineage model or facets according to <a href="https://docs.snowplowanalytics.com/docs/pipeline-components-and-applications/iglu/common-architecture/schemaver">SchemaVer</a> (<em>if relevant</em>)</p> +
+ +
+
Comments
+ 7 +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+ 🤔 Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-02-22 09:02:48
 +
+

*Thread Reply:* I think this scenario only happens when spark job spawns another "sub-job", right?

+ +

I think that maybe you can check sparkContext.getLocalProperty("spark.sql.execution.id")

+ +

> I see that ExecutionStart has an execution id and JobStart should have the same Execution Id BUT I think by the time I reach the ExecutionEND, all the JobStart events would have been removed from the HashMap that contains all of the events in OpenLineage. +But pairwise, those starts and ends should at least have the same runId as they were created with same OpenLineageContext, right?

+ +

Anyway, what @Michael Collado wrote on the issue is true: https://github.com/OpenLineage/OpenLineage/pull/490#issuecomment-1042011803 - you should not assume that we hold all the metadata somewhere in memory during whole execution of the run. The backend should be able to take care of it.

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/pull/490#issuecomment-1042011803|Comment on #490 Added generic facet to collect environmental properties (EnvironmentFacet)>
+
+ <p>&gt; how can I persist this information over to the COMPLETE events coming from OpenLineage?</p> + +<p>The different events are intended to be additive - it is not expected that each event is a cumulative representation of the previous events. If you send the environment facet on the <code>START</code> event, it will still be present for the run even if it's missing from the <code>COMPLETE</code>.</p> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-02-22 10:53:09
 +
+

*Thread Reply:* @Maciej Obuchowski - I was hoping they'd have the same run id as well but they do not 😞

+ +

But that is the expectation? A SparkSQLExecutionStart and JobStart SHOULD have the same execution ID, right?

+ +

I will take a look at sparkContext.getLocalProperty. Thank you so much for the reply Maciej!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-02-22 10:57:24
 +
+

*Thread Reply:* SparkSQLExecutionStart and SparkSQLExecutionEnd should have the same runId, as well as JobStart and JobEnd events. Beyond those it can get wild. For example, some jobs don't emit JobStart/JobEnd events. Some jobs, like Delta emit multiple, that aren't easily tied to SQL event.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-02-23 03:48:38
 +
+

*Thread Reply:* Okay, I dug into the Databricks Synapse Connector and it does the following:

+ +
  1. SparkSQLExecutionStart with execution id of 8 happens (so gets runid of abc123). It contains the real inputs and outputs that we want.
  2. The Synapse connector starts executing JDBC commands. These commands prepare the synapse database to connect with data that Spark will land in a staging area in the cloud. (I don't know how it' executing arbitrary commands before the official job start begins 😞 )
  3. SparkJobStart beings with execution id of 9 happens (so it gets runid of jkl456). This contains the inputs and an output to a temp folder (NOT the real output we want but a staging location) +a. There are four JobIds 0 - 3, all of which point back to execution id 9 with the same physical plan. +b. After job1, it runs more JDBC commands. +c. I think at Job2, it runs the actual Spark code to query and join my raw input data and land it in a cloud storage account "tempfolder"/ +d. After job3, it runs the final JDBC commands to actually move the data from "tempfolder/" to Synapse Db.
  4. Finally, the SparkSQLListenerEnd event occurs. +I can see this in the Spark UI as well.
    5. +
+ +

Because the Databricks Synapse connector somehow adds these additional JobStarts WITHOUT referencing the original SparkSQLExeuctionStart execution ID, we have to rely on heuristics to connect the /tempfolder to the real downstream table that was already provided in the ExecutionStart event 😞

+ +

I've attached the logs and a screenshot of what I'm seeing the Spark UI. If you had a chance to take a look, it's a bit verbose but I'd appreciate a second pair of eyes on my analysis. Hopefully I got something wrong 😅

+ +
+
log4j-SYNAPSE2-SUBSET.txt
+ + + + + + +
+
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-02-23 07:19:01
 +
+

*Thread Reply:* I think we've encountered the same stuff in Delta before 🙂

+ +

https://github.com/OpenLineage/OpenLineage/issues/388#issuecomment-964401860

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2022-02-23 14:13:18
 +
+

*Thread Reply:* @Will Johnson , am I reading your report correctly that the SparkListenerJobStart event is reported with a spark.sql.execution.id that differs from the execution id of the SparkSQLExecutionStart?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2022-02-23 14:18:04
 +
+

*Thread Reply:* WILLJ: We're deep inside this thing and have an executionid |9| +😂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-02-23 21:56:48
 +
+

*Thread Reply:* Hah @Michael Collado I see you found my method of debugging in Databricks 😅

+ +

But you're exactly right, there's a SparkSQLExecutionStart event with execution id 8 and then a set of JobStart events all with execution id 9!

+ +

I don't know enough about Spark internals on how you can just run arbitrary Scala code while making it look like a Spark Job but that's what it looks like. As if the SqlDwWriter somehow submits a new job without a ExecutionStart... maybe it's an RDD operation instead? This has given me another idea to add some more log.info statements to my jar 😅😬

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-02-28 14:00:23
 +
+

One of our own will be talking OpenLineage, Airflow and Spark at the Subsurface Conference this week. Register to attend @Michael Collado’s session on March 3rd at 11:45. You can register and learn more here: https://www.dremio.com/subsurface/live/winter2022/

+
+
Dremio
+ + + +
Subsurface Live Winter 2022, The Cloud Data Lake Conference
+
+ Subsurface LIVE is tailored for data architects and data engineers to discuss open source projects driving innovation in cloud data lakes +
+ + + + + + + + +
Original URL: https://www.dremio.com/subsurface/live/winter2022/
+ + + +
+ +
+
+ + + + + + +
+ + +
+ 🎉 Willy Lulciuc, Maciej Obuchowski +
+ +
+ 🙌 Will Johnson, Ziyoiddin Yusupov, Julien Le Dem +
+ +
+ 👍 Ziyoiddin Yusupov +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2022-02-28 14:00:56
 +
+

*Thread Reply:* You won’t want to miss this talk!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Martin Fiser + (fisa@keboola.com) +
+
2022-02-28 15:06:43
 +
+

I have a question about DataHub integration through OpenLineage standard. Is anyone working on it, or was it rather just an icon used in previous materials? We have build a openlineage API endpoint in our product and we were hoping OL will gain enough traction so it will be a native way to connect to variaty of data discovery/observability tools, such as datahub, amundzen, etc.

+ +

Many thanks!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2022-02-28 15:29:58
 +
+

*Thread Reply:* hi Martin - when you talk about a DataHub integration, did you mean a method to collect information from DataHub? I don't see a current issue open for that, but I recommend you make one and to kick off the discussion around it.

+ +

If you mean sending information to DataHub, that should already be possible if users pass a datahub api endpoint to the OPENLINEAGE_ENDPOINT variable

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Martin Fiser + (fisa@keboola.com) +
+
2022-02-28 16:29:54
 +
+

*Thread Reply:* Hi, thanks for a reply! I meant to emit Openlineage JSON structure to datahub.

+ +

Could you be please more specific, possibly link an article how to find the endpoint on the datahub side? Many thanks!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2022-02-28 17:15:31
 +
+

*Thread Reply:* ooooh, sorry I misread - I thought you meant that datahub had built an endpoint. Your integration should emit openlineage events to an endpoint, but datahub would have to build that support into their product likely? I'm not sure how to go about it

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2022-02-28 17:16:27
 +
+

*Thread Reply:* I'd reach out to datahub, potentially?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Martin Fiser + (fisa@keboola.com) +
+
2022-02-28 17:21:51
 +
+

*Thread Reply:* i see. ok, will do!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2022-03-02 18:15:21
 +
+

*Thread Reply:* It has been discussed in the past but I don’t think there is something yet. The Kafka transport PR that is in flight should facilitate this

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Martin Fiser + (fisa@keboola.com) +
+
2022-03-02 18:33:45
 +
+

*Thread Reply:* Thanks for the response! though dragging Kafka in just for data delivery bit is too much. I think the clearest way would be to push Datahub to make an API endpoint and parser for OL /lineage data structure.

+ +

I see this is more political think that would require join effort of DataHub team and OpenLineage with a common goal.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-02-28 17:22:47
 +
+

Is there a topic you think the community should discuss at the next OpenLineage TSC meeting? Reply or DM with your item, and we’ll add it to the agenda. Mark your calendars: the next TSC meeting is Wednesday, March 9 at 9 am PT on zoom.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-03-02 10:24:58
 +
+

The next OpenLineage Technical Steering Committee meeting is Wednesday, March 9! Meetings are on the second Wednesday of each month from 9:00 to 10:00am PT. +Join us on Zoom: https://us02web.zoom.us/j/81831865546?pwd=RTladlNpc0FTTDlFcWRkM2JyazM4Zz09 +All are welcome. +Agenda: +• New committers +• Release overview (0.6.0) +• New process for blog posts +• Retrospective: Spark integration +Notes: https://tinyurl.com/openlineagetsc

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2022-03-02 14:29:33
 +
+

FYI, there's a talk on OpenLineage at Subsurface live tomorrow - https://www.dremio.com/subsurface/live/winter2022/session/cross-platform-data-lineage-with-openlineage/

+
+
Dremio
+ + + +
Cross-Platform Data Lineage with OpenLineage | Subsurface Live Winter 2022
+
+ Data today is distributed and heterogeneous. Data lineage helps by tracing the relationship between datasets and placing them in context. +
+ +
+
Est. reading time
+ 1 minute +
+ + + + + + + + +
Original URL: https://www.dremio.com/subsurface/live/winter2022/session/cross-platform-data-lineage-with-openlineage/
+ + + +
+ + + +
+ 🙌 Maciej Obuchowski, John Thomas, Paweł Leszczyński, Francis McGregor-Macdonald +
+ +
+ 👍 Ziyoiddin Yusupov, Michael Robinson, Jac. +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-03-04 15:25:20
 +
+

@channel The latest release (0.6.0) of OpenLineage is now available, featuring a new Dagster integration, updates to the Airflow and Java integrations, a generic facet for env properties, bug fixes, and more. For more info, visit https://github.com/OpenLineage/OpenLineage/releases/tag/0.6.0

+ + + +
+ 🙌 Conor Beverland, Dalin Kim, Ziyoiddin Yusupov, Luca Soato +
+ +
+ 👍 Julien Le Dem +
+ +
+ 👀 William Angel, Francis McGregor-Macdonald +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-07 14:06:19
 +
+

Hello Guys,

+ +

Where do I find an example of building a custom extractor? We have several custom airflow operators that I need to integrate

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2022-03-07 14:56:58
 +
+

*Thread Reply:* Hi marco - we don't have documentation on that yet, but the Postgres extractor is a pretty good example of how they're implemented.

+ +

all the included extractors are here: https://github.com/OpenLineage/OpenLineage/tree/main/integration/airflow/openlineage/airflow/extractors

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-07 15:07:41
 +
+

*Thread Reply:* Thanks. I can follow that to build my own. Also I am installing this environment right now in Airflow 2. It seems I need Marquez and openlinegae-aiflow library. It seems that by this example I can put my extractors in any path as long as it is referenced in the environment variable. Is that correct? +OPENLINEAGE_EXTRACTOR_&lt;operator&gt;=full.path.to.ExtractorClass +Also do I need anything else other than Marquez and openlineage_airflow?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-03-07 15:30:45
 +
+

*Thread Reply:* Yes, as long as the extractors are in the python path.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-03-07 15:31:59
 +
+

*Thread Reply:* I built one a little while ago for a custom operator, I'd be happy to share what I did. I put it in the same file as the operator class for convenience.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-07 15:32:51
 +
+

*Thread Reply:* That will be great help. Thanks

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-03-08 20:38:27
 +
+

*Thread Reply:* This is the one I wrote:

+ +
+
http_to_bigquery.py
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-03-08 20:39:30
 +
+

*Thread Reply:* to make it work, I set this environment variable:

+ +

OPENLINEAGE_EXTRACTOR_HttpToBigQueryOperator=http_to_bigquery.HttpToBigQueryExtractor

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-03-08 20:40:57
 +
+

*Thread Reply:* the extractor starts at line 183, and the really important bits start at line 218

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-03-07 15:16:37
 +
+

@channel At the next OpenLineage TSC meeting, we’ll be reminiscing about the Spark integration. If you’ve had a hand in OL support for Spark, please join and share! The meeting will start at 9 am PT on Wednesday this week. @Maciej Obuchowski @Oleksandr Dvornik @Willy Lulciuc @Michael Collado https://wiki.lfaidata.foundation/display/OpenLineage/Monthly+TSC+meeting

+ + + +
+ 👍 Ross Turk, Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-07 18:44:26
 +
+

Would Marquez create some lineage for operators that don't have a custom extractor built yet?

+ + + +
+ ✅ Fuming Shih +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-03-08 12:05:25
 +
+

*Thread Reply:* You would see that job was run - but we couldn't extract dataset lineage from it.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-03-08 12:05:49
 +
+

*Thread Reply:* The good news is that we're working to solve this problem in general.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-08 12:15:52
 +
+

*Thread Reply:* I see, so i definitively will need the custom extractor built. I just need to understand where to set the path to the extractor. I can build one by following the postgress extractor you have built.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-03-08 12:50:00
 +
+

*Thread Reply:* That depends how you deploy Airflow. Our tests use environment in docker-compose: https://github.com/OpenLineage/OpenLineage/blob/main/integration/airflow/tests/integration/tests/docker-compose-2.yml#L34

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/integration/airflow/tests/integration/tests/docker-compose-2.yml | docker-compose-2.yml>
+
+ <pre><code> OPENLINEAGE_EXTRACTOR_CustomOperator: custom_extractor.CustomExtractor +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-08 13:19:37
 +
+

*Thread Reply:* Thanks for the example. I can show this to my infra support person for his reference.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-03-08 11:47:11
 +
+

This month’s OpenLineage TSC community meeting is tomorrow at 9am PT! It’s not too late to add an item to the agenda. Reply here or msg me with yours. https://openlineage.slack.com/archives/C01CK9T7HKR/p1646234698326859

+
+ + +
+ + + } + + Michael Robinson + (https://openlineage.slack.com/team/U02LXF3HUN7) +
+ + + +
+
+ <p>The next OpenLineage Technical Steering Committee meeting is Wednesday, March 9! Meetings are on the second Wednesday of each month from 9:00 to 10:00am PT. +Join us on Zoom: <a href="https://us02web.zoom.us/j/81831865546?pwd=RTladlNpc0FTTDlFcWRkM2JyazM4Zz09">https://us02web.zoom.us/j/81831865546?pwd=RTladlNpc0FTTDlFcWRkM2JyazM4Zz09</a> +All are welcome. +Agenda: +• New committers +• Release overview (0.6.0) +• New process for blog posts +• Retrospective: Spark integration +Notes: <a href="https://tinyurl.com/openlineagetsc">https://tinyurl.com/openlineagetsc</a></p> +
+ + + + + + +
Original URL: https://openlineage.slack.com/archives/C01CK9T7HKR/p1646234698326859
+ + +
+ + Slack Conversation +
+ + +
+ + + +
+ 👍 Ross Turk +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-09 19:31:23
 +
+

I am running the last command to install marquez in AWS +helm upgrade --install marquez . + --set marquez.db.host &lt;AWS-RDS-HOST&gt; + --set marquez.db.user &lt;AWS-RDS-USERNAME&gt; + --set marquez.db.password &lt;AWS-RDS-PASSWORD&gt; + --namespace marquez + --atomic + --wait +And I am receiving this error +Error: query: failed to query with labels: secrets is forbidden: User "xxx@xxx.xx" cannot list resource "secrets" in API group "" in the namespace "default"

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2022-03-10 12:46:18
 +
+

*Thread Reply:* Do you need to specify a namespace that is not « default »?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-09 19:31:48
 +
+

Can anyone let me know what is happening? My DI guy said it is a chart issue

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-03-10 07:40:13
 +
+

*Thread Reply:* @Kevin Mellott aren't you the chart wizard? Maybe you could help 🙂

+ + + +
+ 👀 Kevin Mellott +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-10 14:09:26
 +
+

*Thread Reply:* Ok so I had to update a chart dependency

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-10 14:10:39
 +
+

*Thread Reply:* Now I installed the service in amazon using this +helm install marquez . --dependency-update --set marquez.db.host=myhost --set marquez.db.user=myuser --set marquez.db.password=mypassword --namespace marquez --atomic --wait

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-10 14:11:31
 +
+

*Thread Reply:* i can see marquez-web running and marquez as well as the database i set up manually

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-10 14:12:27
 +
+

*Thread Reply:* however I can not fetch initial data when login into the endpoint

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kevin Mellott + (kevin.r.mellott@gmail.com) +
+
2022-03-10 14:52:06
 +
+

*Thread Reply:* 👋 @Marco Diaz happy to hear that the Helm install is completing without error! To help troubleshoot the error above, can you please let me know if this endpoint is available and working?

+ +

http://localhost:5000/api/v1/namespaces

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-10 15:13:16
 +
+

*Thread Reply:* i got this +{"namespaces":[{"name":"default","createdAt":"2022_03_10T18:05:55.780593Z","updatedAt":"2022-03-10T19:03:31.309713Z","ownerName":"anonymous","description":"The default global namespace for dataset, job, and run metadata not belonging to a user-specified namespace."}]}

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-10 15:13:34
 +
+

*Thread Reply:* i have to use the namespace marquez to redirect there +kubectl port-forward svc/marquez 5000:80 -n marquez

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-10 15:13:48
 +
+

*Thread Reply:* is there something i need to change in a config file?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-10 15:14:39
 +
+

*Thread Reply:* also how would i change the "localhost" address to something that is accessible in amazon without the need to redirect?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-10 15:14:59
 +
+

*Thread Reply:* Sorry for all the questions. I am not an infra guy and have had to do all this by myself

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kevin Mellott + (kevin.r.mellott@gmail.com) +
+
2022-03-10 15:39:23
 +
+

*Thread Reply:* No problem at all, I think there are a couple of things at play here. With the local setup, it appears that the web is attempting to access the API on the wrong port number (3000 instead of 5000). I’ll create an issue for that one so that we can fix it.

+ +

As to the EKS installation (or any non-local install), this is where you would need to use what’s called an ingress controller to expose the services outside of the Kubernetes cluster. There are different flavors of these (NGINX is popular), and I believe that AWS EKS has some built-in capabilities that might help as well.

+ +

https://www.eksworkshop.com/beginner/130_exposing-service/ingress/

+
+
Amazon EKS Workshop
+ + + +
EKSworkshop.com
+
+ Amazon EKS Workshop +
+ + + + + + + + +
Original URL: https://www.eksworkshop.com/beginner/130_exposing-service/ingress/
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-10 15:40:50
 +
+

*Thread Reply:* So how do i fix this issue?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kevin Mellott + (kevin.r.mellott@gmail.com) +
+
2022-03-10 15:46:56
 +
+

*Thread Reply:* If your goal is to deploy to AWS, then you would need to get the EKS ingress configured. It’s not a trivial task, but they do have a bit of a walkthrough at https://www.eksworkshop.com/beginner/130_exposing-service/.

+ +

However, if you are just seeking to explore Marquez and try things out, then I would highly recommend the “Open in Gitpod” functionality at https://github.com/MarquezProject/marquez#try-it. That will perform a full deployment for you in a temporary environment very quickly.

+
+
Amazon EKS Workshop
+ + + +
EKSworkshop.com
+
+ Amazon EKS Workshop +
+ + + + + + + + +
Original URL: https://www.eksworkshop.com/beginner/130_exposing-service/
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-10 16:02:05
 +
+

*Thread Reply:* i need to use it in aws for a POC

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-10 19:15:08
 +
+

*Thread Reply:* Is there a better guide on how to install and setup Marquez in AWS? +This guide is omitting many steps +https://marquezproject.github.io/marquez/running-on-aws.html

+
+
Marquez
+ + + +
Running Marquez on AWS
+
+ Collect, aggregate, and visualize a data ecosystem’s metadata +
+ + + + + + +
Original URL: https://marquezproject.github.io/marquez/running-on-aws.html
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-03-10 12:35:37
 +
+

We're trying to find best way to track upstream releases of projects we have integrations for, to support newer versions faster and with less bugs. If you have any opinions on this topic, please chime in here

+ +

https://github.com/OpenLineage/OpenLineage/issues/602

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-11 13:34:30
 +
+

@Kevin Mellott Hello Kevin I followed the tutorial you sent me and I have exposed my services. However I am still seeing the same errors (this comes from the api/namescape call) +{"namespaces":[{"name":"default","createdAt":"2022_03_10T18:05:55.780593Z","updatedAt":"2022-03-10T19:03:31.309713Z","ownerName":"anonymous","description":"The default global namespace for dataset, job, and run metadata not belonging to a user-specified namespace."}]}

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-11 13:35:08
 +
+

Is there something i need to change in the chart? I do not have access to the default namespace in kubernetes only marquez namescpace

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kevin Mellott + (kevin.r.mellott@gmail.com) +
+
2022-03-11 13:56:27
 +
+

@Marco Diaz that is actually a good response! This is the JSON returned back by the API to show some of the default Marquez data created by the install. Is there another error you are experiencing?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-11 13:59:28
 +
+

*Thread Reply:* I still see this +https://files.slack.com/files-pri/T01CWUYP5AR-F036JKN77EW/image.png

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-11 14:00:09
 +
+

*Thread Reply:* I created my own database and changed the values for host, user and password inside the chart.yml

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kevin Mellott + (kevin.r.mellott@gmail.com) +
+
2022-03-11 14:00:23
 +
+

*Thread Reply:* Does it show that within the AWS deployment? It looks to show localhost in your screenshot.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kevin Mellott + (kevin.r.mellott@gmail.com) +
+
2022-03-11 14:00:52
 +
+

*Thread Reply:* Or are you working through the local deploy right now?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-11 14:01:57
 +
+

*Thread Reply:* It shows the same using the exposed service

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-11 14:02:09
 +
+

*Thread Reply:* i just didnt do another screenshot

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-11 14:02:27
 +
+

*Thread Reply:* Could it be communication with the DB?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kevin Mellott + (kevin.r.mellott@gmail.com) +
+
2022-03-11 14:04:37
 +
+

*Thread Reply:* What do you see if you view the network traffic within your web browser (right click -> Inspect -> Network). Specifically, wondering what the response code from the Marquez API URL looks like.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-11 14:14:48
 +
+

*Thread Reply:* i see this error +Error occured while trying to proxy to: <a href="http://xxxxxxxxxxxxxxxxxxxxxxxxx.us-east-1.elb.amazonaws.com/api/v1/namespaces">xxxxxxxxxxxxxxxxxxxxxxxxx.us-east-1.elb.amazonaws.com/api/v1/namespaces</a>

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-11 14:16:00
 +
+

*Thread Reply:* it seems to be trying to use the same address to access the api endpoint

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-11 14:16:26
 +
+

*Thread Reply:* however the api service is in a different endpoint

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-11 14:18:24
 +
+

*Thread Reply:* The API resides here +<a href="http://Xxxxxxxxxxxxxxxxxxxxxx-2064419849.us-east-1.elb.amazonaws.com">Xxxxxxxxxxxxxxxxxxxxxx-2064419849.us-east-1.elb.amazonaws.com</a>

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-11 14:19:13
 +
+

*Thread Reply:* The web service resides here +<a href="http://xxxxxxxxxxxxxxxxxxxxxxxxxxx-335729662.us-east-1.elb.amazonaws.com">xxxxxxxxxxxxxxxxxxxxxxxxxxx-335729662.us-east-1.elb.amazonaws.com</a>

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-11 14:19:25
 +
+

*Thread Reply:* do they both need to be under the same LB?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-11 14:19:56
 +
+

*Thread Reply:* How would i do that is they install as separate services?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kevin Mellott + (kevin.r.mellott@gmail.com) +
+
2022-03-11 14:27:15
 +
+

*Thread Reply:* You are correct, both the website and API are expecting to be exposed on the same ALB. This will give you a single URL that can reach your Kubernetes cluster, and then the ALB will allow you to configure Ingress rules to route the traffic based on the request.

+ +

Here is an example from one of the AWS repos - in the ingress resource you can see the single rule setup to point traffic to a given service.

+ +

https://github.com/kubernetes-sigs/aws-load-balancer-controller/blob/main/docs/examples/2048/2048_full.yaml

+
+ + + + +
<https://github.com/kubernetes-sigs/aws-load-balancer-controller/blob/main/docs/examples/2048/2048_full.yaml | 2048_full.yaml>
+
+ <pre><code>--- +apiVersion: v1 +kind: Namespace +metadata: + name: game-2048 +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + namespace: game-2048 + name: deployment-2048 +spec: + selector: + matchLabels: + &lt;a href="http://app.kubernetes.io/name"&gt;app.kubernetes.io/name&lt;/a&gt;: app-2048 + replicas: 5 + template: + metadata: + labels: + &lt;a href="http://app.kubernetes.io/name"&gt;app.kubernetes.io/name&lt;/a&gt;: app-2048 + spec: + containers: + - image: public.ecr.aws/l6m2t8p7/docker-2048:latest + imagePullPolicy: Always + name: app-2048 + ports: + - containerPort: 80 +--- +apiVersion: v1 +kind: Service +metadata: + namespace: game-2048 + name: service-2048 +spec: + ports: + - port: 80 + targetPort: 80 + protocol: TCP + type: NodePort + selector: + &lt;a href="http://app.kubernetes.io/name"&gt;app.kubernetes.io/name&lt;/a&gt;: app-2048 +--- +apiVersion: &lt;a href="http://networking.k8s.io/v1"&gt;networking.k8s.io/v1&lt;/a&gt; +kind: Ingress +metadata: + namespace: game-2048 + name: ingress-2048 + annotations: + &lt;a href="http://alb.ingress.kubernetes.io/scheme"&gt;alb.ingress.kubernetes.io/scheme&lt;/a&gt;: internet-facing + &lt;a href="http://alb.ingress.kubernetes.io/target-type"&gt;alb.ingress.kubernetes.io/target-type&lt;/a&gt;: ip +spec: + ingressClassName: alb + rules: + - http: + paths: + - path: / + pathType: Prefix + backend: + service: + name: service-2048 + port: + number: 80 + +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/kubernetes-sigs/aws-load-balancer-controller">kubernetes-sigs/aws-load-balancer-controller</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-11 14:36:40
 +
+

*Thread Reply:* Thanks for the help. Now I know what the issue is

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kevin Mellott + (kevin.r.mellott@gmail.com) +
+
2022-03-11 14:51:34
 +
+

*Thread Reply:* Great to hear!!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sandeep Bhat + (bhatsandeep424@gmail.com) +
+
2022-03-16 00:55:36
 +
+

👋 Hi everyone! Our company is looking to adopt data lineage tool, so i have few queries on open lineage, so 1. Is this completey free.

+ +
  1. What are tha database it supports?
    2. +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-03-16 10:29:06
 +
+

*Thread Reply:* Hi! Yes, OpenLineage is free. It is an open source standard for collection, and it provides the agents that integrate with pipeline tools to capture lineage metadata. You also need a metadata server, and there is an open source one called Marquez that you can use.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-03-16 10:29:15
 +
+

*Thread Reply:* It supports the databases listed here: https://openlineage.io/integration

+
+
openlineage.io
+ + + +
Integrations
+
+ Data lineage is the foundation for a new generation of powerful, context-aware data tools and best practices. OpenLineage enables consistent collection of lineage metadata, creating a deeper understanding of how data is produced and used. +
+ + + + + + +
Original URL: https://openlineage.io/integration
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sandeep Bhat + (bhatsandeep424@gmail.com) +
+
2022-03-16 08:27:20
 +
+

and when i run the ./docker/up.sh --seed i got the result from java code(sample example) But how to get the same thing in python example?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-03-16 10:29:53
 +
+

*Thread Reply:* Not sure I understand - are you looking for example code in Python that shows how to make OpenLineage calls?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sandeep Bhat + (bhatsandeep424@gmail.com) +
+
2022-03-16 12:45:14
 +
+

*Thread Reply:* yup

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sandeep Bhat + (bhatsandeep424@gmail.com) +
+
2022-03-16 13:10:04
 +
+

*Thread Reply:* how to run

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-03-16 23:08:31
 +
+

*Thread Reply:* this is a good post for getting started with Marquez: https://openlineage.io/blog/explore-lineage-api/

+
+
openlineage.io
+ + + +
Exploring Lineage History via the Marquez API
+
+ Taking advantage of recent changes to the Marquez API, this post shows how to diagnose job failures and explore the impact of code changes on downstream dependents. +
+ + + + + + +
Original URL: https://openlineage.io/blog/explore-lineage-api/
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-03-16 23:08:51
 +
+

*Thread Reply:* once you have run ./docker/up.sh, you should be able to run through that and see how the system runs

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-03-16 23:09:45
 +
+

*Thread Reply:* There is a python client you can find here: https://github.com/OpenLineage/OpenLineage/tree/main/client/python

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sandeep Bhat + (bhatsandeep424@gmail.com) +
+
2022-03-17 00:05:58
 +
+

*Thread Reply:* Thank you

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-03-19 00:00:32
 +
+

*Thread Reply:* You are welcome 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mirko Raca + (racamirko@gmail.com) +
+
2022-04-19 09:28:50
 +
+

*Thread Reply:* Hey @Ross Turk, (and potentially @Maciej Obuchowski) - what are the plans for OL Python client? I'd like to use it, but without a pip package it's not really project-friendly.

+ +

Is there any work in that direction, is the current client code considered mature and just needs re-packaging, or is it just a thought sketch and some serious work is needed?

+ +

I'm trying to avoid re-inventing the wheel, so if there's already something in motion, I'd rather support than start (badly) from scratch?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-04-19 09:32:17
 +
+

*Thread Reply:* What do you mean without pip-package?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-04-19 09:32:18
 +
+

*Thread Reply:* https://pypi.org/project/openlineage-python/

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-04-19 09:35:08
 +
+

*Thread Reply:* It's still developed, for example next release will have pluggable backends - like Kafka +https://github.com/OpenLineage/OpenLineage/pull/530

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mirko Raca + (racamirko@gmail.com) +
+
2022-04-19 09:40:11
 +
+

*Thread Reply:* My apologies Maciej! +In my defense - looking for "open lineage" on pypi doesn't show this in the first 20 results. Still, should have checked setup.py. My bad, and thank you for the pointer!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-04-19 10:00:49
 +
+

*Thread Reply:* We might need to add some keywords to setup.py - right now we have only "openlineage" there 😉

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mirko Raca + (racamirko@gmail.com) +
+
2022-04-20 08:12:29
 +
+

*Thread Reply:* My mistake was that I was expecting a separate repo for the clients. But now I'm playing around with the package and trying to figure out the OL concepts. Thank you for your contribution, it's much nicer to experiment from ipynb than curl 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-03-16 12:00:01
 +
+

@Julien Le Dem and @Willy Lulciuc will be at Data Council Austin next week talking OpenLineage and Airflow https://www.datacouncil.ai/talks/data-lineage-with-apache-airflow-using-openlineage?hsLang=en

+
+
datacouncil.ai
+ + + +
Data Lineage with Apache Airflow using OpenLineage
+
+ Previous Data Council talks from past conferences +
+ + + + + + +
Original URL: https://www.datacouncil.ai/talks/data-lineage-with-apache-airflow-using-openlineage?hsLang=en
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sandeep Bhat + (bhatsandeep424@gmail.com) +
+
2022-03-16 12:50:20
 +
+

I couldn't figure out for the sample lineage flow (etldelivery7_days) when we ran the seed command after from which file its fetching data

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2022-03-16 14:35:14
 +
+

*Thread Reply:* the seed data is being inserted by this command here: https://github.com/MarquezProject/marquez/blob/main/api/src/main/java/marquez/cli/SeedCommand.java

+
+ + + + +
<https://github.com/MarquezProject/marquez/blob/main/api/src/main/java/marquez/cli/SeedCommand.java | SeedCommand.java>
+
+ <p>``` +/<em>* SPDX-License-Identifier: Apache-2.0 *</em>/</p> + +<p>package marquez.cli;</p> + +<p>import static com.google.common.base.Preconditions.checkArgument; +import static marquez.client.models.JobType.BATCH; +import static marquez.common.base.MorePreconditions.checkNotBlank;</p> + +<p>import com.google.common.collect.ImmutableList; +import com.google.common.collect.ImmutableMap; +import com.google.common.collect.ImmutableSet; +import com.google.common.collect.Iterables; +import com.google.common.collect.Lists; +import com.google.common.collect.Maps; +import com.google.common.collect.Sets; +import io.dropwizard.cli.ConfiguredCommand; +import io.dropwizard.setup.Bootstrap; +import java.net.URL; +import java.time.Duration; +import java.time.Instant; +import java.time.temporal.ChronoUnit; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; +import java.util.Optional; +import java.util.Random; +import javax.annotation.Nullable; +import lombok.EqualsAndHashCode; +import lombok.Getter; +import lombok.NonNull; +import lombok.ToString; +import lombok.extern.slf4j.Slf4j; +import marquez.MarquezConfig; +import marquez.client.MarquezClient; +import marquez.client.Utils; +import marquez.client.models.Dataset; +import marquez.client.models.DatasetId; +import marquez.client.models.DatasetMeta; +import marquez.client.models.DbTableMeta; +import marquez.client.models.Field; +import marquez.client.models.Job; +import marquez.client.models.JobMeta; +import marquez.client.models.Namespace; +import marquez.client.models.NamespaceMeta; +import marquez.client.models.Run; +import marquez.client.models.RunMeta; +import marquez.client.models.Source; +import marquez.client.models.SourceMeta;</p> + +<p>/<em>*</em>* + <em>* A command to seed the HTTP API with source, dataset, and job metadata. You can override the + *</em> default {@code host} and {@code port} using the command-line arguments {@code --host} and {@code + <em>* --port}. This command is meant to be used to explore the features of Marquez. For example, + *</em> lineage graph, dataset schemas, job run history, etc. + <em>* + *</em> &lt;h3&gt;Usage&lt;/h3&gt; + <em>* + *</em> For example, to override the {@code port}: + <em>* + *</em> &lt;pre&gt;{@code + <em>* java -jar marquez-api.jar seed --port 5001 marquez.yml + *</em> }&lt;/pre&gt; + <em>* + *</em> Note that all metadata is defined within this class and requires a running instance of Marquez. + **/ +@Slf4j +public final class SeedCommand extends ConfiguredCommand&lt;MarquezConfig&gt; { + static final String DEFAULT<em>MARQUEZ</em>HOST = "localhost"; + static final int DEFAULT<em>MARQUEZ</em>PORT = 8080;</p> + +<p>public static final String NAMESPACE<em>NAME = "food</em>delivery"; + static final String SOURCE<em>NAME = "analytics</em>db";</p> + +<p>static final int LINEAGE<em>GRAPH</em>24<em>HOUR</em>WINDOW = 24; + static final int RUN<em>TIME</em>IN<em>SEC</em>MIN = 120; + static final int RUN<em>TIME</em>IN<em>SEC</em>MAX = 240;</p> + +<p>public SeedCommand() { + super("seed", "seeds the HTTP API with metadata"); + }</p> + +<p>@Override + public void configure(@NonNull final net.sourceforge.argparse4j.inf.Subparser subparser) { + super.configure(subparser); + subparser + .addArgument("--host") + .dest("host") + .type(String.class) + .required(false) + .setDefault(DEFAULT<em>MARQUEZ</em>HOST) + .help("the HTTP API server host"); + subparser + .addArgument("--port") + .dest("port") + .type(Integer.class) + .required(false) + .setDefault(DEFAULT<em>MARQUEZ</em>PORT) + .help("the HTTP API server port"); + }</p> + +<p>@Override + protected void run( + @NonNull final Bootstrap&lt;MarquezConfig&gt; bootstrap, + @NonNull final net.sourceforge.argparse4j.inf.Namespace namespace, + @NonNull final MarquezConfig config) { + final String host = namespace.getString("host"); + final int port = namespace.getInt("port");</p> + +<pre><code>final URL baseUrl = Utils.toUrl(String.format("&lt;http://%s:%d&gt;", host, port)); +final MarquezClient client = MarquezClient.builder().baseUrl(baseUrl).build(); +seedApiWithMeta(client, LINEAGE_GRAPH_24_HOUR_WINDOW); +</code></pre> + +<p>}</p> + +<p>public void seedApiWithMeta(@NonNull MarquezClient client, int additionalIterations) { + // (1) Create namespace + final NamespaceMeta namespaceMeta = + NamespaceMeta.builder() + .ownerName("<a href="mailto:owner@food.com">owner@food.com</a>") + .description("Food delivery example!") + .build(); + final Namespace newNamespace = client.createNamespace(NAMESPACE_NAME, namespaceMeta); + <a href="http://log.info">log.info</a>("Created namespace: {}", newNamespace);</p> + +<pre><code>// (2) Create source +final SourceMeta sourceMeta = + SourceMeta.builder() + .type("POSTGRESQL") + .connectionUrl("jdbc:&lt;postgres://localhost:3306/deliveries%22|postgres://localhost:3306/deliveries"&gt;) + .description("Contains all food delivery orders.") + .build(); +final Source newSource = client.createSource(SOURCE_NAME, sourceMeta); +&lt;a href="http://log.info"&gt;log.info&lt;/a&gt;("Created source: {}", newSource); + +// (3) Seed dataset meta +DATASET_META.forEach( + (datasetName, datasetMeta) -&amp;gt; { + final Dataset newDataset = client.createDataset(NAMESPACE_NAME, datasetName, datasetMeta); + &lt;a href="http://log.info"&gt;log.info&lt;/a&gt;("Created dataset: {}", newDataset); + }); + +// (4) Seed job meta +JOB_META.forEach( + (jobName, jobMeta) -&amp;gt; { + final Job newJob = client.createJob(NAMESPACE_NAME, jobName, jobMeta); + &lt;a href="http://log.info"&gt;log.info&lt;/a&gt;("Created job: {}", newJob); + }); + +// (5) Define run start times for each graph level +final Instant startTimesGraphLevel0 = Instant.now(); +final Instant startTimesGraphLevel1 = startTimesGraphLevel0.plusSeconds(RUN_TIME_IN_SEC_MAX); +final Instant startTimesGraphLevel2 = startTimesGraphLevel1.plusSeconds(RUN_TIME_IN_SEC_MAX); +final Instant startTimesGraphLevel3 = startTimesGraphLevel2.plusSeconds(RUN_TIME_IN_SEC_MAX); +final Instant startTimesGraphLevel4 = startTimesGraphLevel3.plusSeconds(RUN_TIME_IN_SEC_MAX); + +final Instant[] startTimesByGraphLevel = { + startTimesGraphLevel0, + startTimesGraphLevel1, + startTimesGraphLevel2, + startTimesGraphLevel3, + startTimesGraphLevel4 +}; + +// (6) Seed run meta for jobs using graph level start times +for (int hourOfDay = additionalIterations; hourOfDay &amp;gt;= 0; hourOfDay--) { + for (final Map.Entry&amp;lt;String, JobMeta&amp;gt; entry : JOB_META.entrySet()) { + final String jobName = entry.getKey(); + final JobMeta jobMeta = entry.getValue(); + + if (hourOfDay &amp;gt;= ACTIVE_RUN_META.get(jobName).size()) { + continue; + } + + // On code change, create a new job version + final ActiveRunMeta activeRunMeta = ACTIVE_RUN_META.get(jobName).get(hourOfDay); + activeRunMeta + .getCodeChange() + .ifPresent( + codeChange -&amp;gt; { + client.createJob( + NAMESPACE_NAME, + jobName, + JobMeta.builder() + .type(jobMeta.getType()) + .inputs(jobMeta.getInputs()) + .outputs(jobMeta.getOutputs()) + .location(codeChange.getToUrl()) + .context(jobMeta.getContext()) + .description(jobMeta.getDescription().orElse(null)) + .build()); + }); + + // Set run start and end times + final Instant runStartedAt = + startTimesByGraphLevel[activeRunMeta.getLevelInGraph()].minus( + Duration.ofHours(hourOfDay)); + final Instant runEndedAt = runStartedAt.plusSeconds(secondsToAdd()); + + // Create run + final RunMeta runMeta = + RunMeta.builder() + .nominalStartTime(runStartedAt.truncatedTo(ChronoUnit.MINUTES)) + .nominalEndTime(runEndedAt.truncatedTo(ChronoUnit.MINUTES)) + .build(); + final Run run = client.createRun(NAMESPACE_NAME, jobName, runMeta); + &lt;a href="http://log.info"&gt;log.info&lt;/a&gt;("Create… +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/MarquezProject/marquez">MarquezProject/marquez</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sandeep Bhat + (bhatsandeep424@gmail.com) +
+
2022-03-17 00:06:53
 +
+

*Thread Reply:* Got it, but if i changed the code in this java file lets say i added another job here satisfying the syntax its not appearing in the lineage flow

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-22 18:18:22
 +
+

@Kevin Mellott Hello Kevin, sorry to bother you again. I was finally able to configure Marquez in AWS using an ALB. Now I am receiving this error when calling the API

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-22 18:18:32
 +
+

Is this an issue accessing the database?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-22 18:19:15
 +
+

I created the database and host manually and passed the parameters using helm --set

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-22 18:19:33
 +
+

Do the database services need to be exposed too through the ALB?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kevin Mellott + (kevin.r.mellott@gmail.com) +
+
2022-03-23 10:20:47
 +
+

*Thread Reply:* I’m not too familiar with the 504 error in ALB, but found a guide with troubleshooting steps. If this is an issue with connectivity to the Postgres database, then you should be able to see errors within the marquez pod in EKS (kubectl logs <marquez pod name>) to confirm.

+ +

I know that EKS needs to have connectivity established to the Postgres database, even in the case of RDS, so that could be the culprit.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-23 16:09:09
 +
+

*Thread Reply:* @Kevin Mellott This is the error I am seeing in the logs +[HPM] Proxy created: /api/v1 -&gt; <http://localhost:5000/> +App listening on port 3000! +[HPM] Error occurred while trying to proxy request /api/v1/namespaces from <a href="http://marquez-interface-test.di.rbx.com">marquez-interface-test.di.rbx.com</a> to <http://localhost:5000/> (ECONNREFUSED) (<https://nodejs.org/api/errors.html#errors_common_system_errors>)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kevin Mellott + (kevin.r.mellott@gmail.com) +
+
2022-03-23 16:22:13
 +
+

*Thread Reply:* It looks like the website is attempting to find the API on localhost. I believe this can be resolved by setting the following Helm chart value within your deployment.

+ +

marquez.hostname=marquez-interface-test.di.rbx.com

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kevin Mellott + (kevin.r.mellott@gmail.com) +
+
2022-03-23 16:22:54
 +
+

*Thread Reply:* assuming that is the DNS used by the website

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-23 16:48:53
 +
+

*Thread Reply:* thanks, that did it. I have a question regarding the database

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-23 16:50:01
 +
+

*Thread Reply:* I made my own database manually. Do the marquez tables should be created automatically when install marquez?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-23 16:56:10
 +
+

*Thread Reply:* Also could you put both the API and interface on the same port (3000)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-23 17:21:58
 +
+

*Thread Reply:* Seems I am still having the forwarding issue +[HPM] Proxy created: /api/v1 -&gt; <http://marquez-interface-test.di.rbx.com:5000/> +App listening on port 3000! +[HPM] Error occurred while trying to proxy request /api/v1/namespaces from <a href="http://marquez-interface-test.di.rbx.com">marquez-interface-test.di.rbx.com</a> to <http://marquez-interface-test.di.rbx.com:5000/> (ECONNRESET) (<https://nodejs.org/api/errors.html#errors_common_system_errors>)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-03-23 09:08:14
 +
+

Guidance on How / When a Spark SQL Execution event Controls JobStart Events?

+ +

@Maciej Obuchowski and @Paweł Leszczyński and @Michael Collado I'd really appreciate your thoughts on how / when JobStart events are triggered for a given execution. I've ran into two situations now where a SQLExecutionStart event fires with execution id X and then JobStart events fire with execution id Y.

+ +

• Spark 2 Delta SaveIntoDataSourceCommand on Databricks - I see it has a SparkSQLExecutionStart event but only on Spark 3 does it have JobStart events with the SaveIntoDataSourceCommand and the same execution id. +• Databricks Synapse Connector - A SparkSQLExecutionStart event occurs but then the job starts are different execution ids. +Is there any guidance / books / videos that dive deeper into how these events are triggered?

+ +

We need the JobStart event with the same execution id so that we can get some environment properties stored in the job start event.

+ +

Thanks you so much for any guidance!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-03-23 09:25:18
 +
+

*Thread Reply:* It's always Delta, isn't it?

+ +

When I originally worked on Delta support I tried to find answer on Delta slack and got an answer:

+ +

Hi Maciej, the main reason is that Delta will run queries on metadata to figure out what files should be read for a particular version of a Delta table and that's why you might see multiple jobs. In general Delta treats metadata as data and leverages Spark to handle them to make it scalable.

+ + + +
+ 🤣 Will Johnson +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-03-23 09:25:48
 +
+

*Thread Reply:* I haven't touched how it works in Spark 2 - wanted to make it work with Spark 3's new catalogs, so can't help you there.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-03-23 09:46:14
 +
+

*Thread Reply:* Argh!! It's always Databricks doing something 🙄

+ +

Thanks, Maciej!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-03-23 09:51:59
 +
+

*Thread Reply:* One last question for you, @Maciej Obuchowski, any thoughts on how I could identify WHY a particular JobStart event fired? Is it just stepping through every event? Was that your approach to getting Spark3 Delta working? Thank you so much for the insights!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-03-23 09:58:08
 +
+

*Thread Reply:* Before that, we were using just JobStart/JobEnd events and I couldn't find events that correspond to logical plan that has anything to do with what job was actually doing. I just found out that SQLExecution events have what I want, so I just started using them and stopped worrying about Projection or Aggregate, or other events that don't really matter here - and that's how filtering idea was born: https://github.com/OpenLineage/OpenLineage/issues/423

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-03-23 09:59:37
 +
+

*Thread Reply:* Are you trying to get environment info from those events, or do you actually get Job event with proper logical plans like SaveIntoDataSourceCommand?

+ +

Might be worth to just post here all the events + logical plans that are generated for particular job, as I've done in that issue

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-03-23 09:59:40
 +
+

*Thread Reply:* scala&gt; spark.sql("CREATE TABLE tbl USING delta AS SELECT ** FROM tmp") +21/11/09 19:01:46 WARN SparkSQLExecutionContext: SparkListenerSQLExecutionStart - executionId: 3 +21/11/09 19:01:46 WARN SparkSQLExecutionContext: org.apache.spark.sql.catalyst.plans.logical.CreateTableAsSelect +21/11/09 19:01:46 WARN SparkSQLExecutionContext: SparkListenerSQLExecutionStart - executionId: 4 +21/11/09 19:01:46 WARN SparkSQLExecutionContext: org.apache.spark.sql.catalyst.plans.logical.LocalRelation +21/11/09 19:01:46 WARN SparkSQLExecutionContext: SparkListenerJobStart - executionId: 4 +21/11/09 19:01:46 WARN SparkSQLExecutionContext: org.apache.spark.sql.catalyst.plans.logical.LocalRelation +21/11/09 19:01:47 WARN SparkSQLExecutionContext: SparkListenerJobEnd - executionId: 4 +21/11/09 19:01:47 WARN SparkSQLExecutionContext: org.apache.spark.sql.catalyst.plans.logical.LocalRelation +21/11/09 19:01:47 WARN SparkSQLExecutionContext: SparkListenerSQLExecutionEnd - executionId: 4 +21/11/09 19:01:47 WARN SparkSQLExecutionContext: org.apache.spark.sql.catalyst.plans.logical.LocalRelation +21/11/09 19:01:48 WARN SparkSQLExecutionContext: SparkListenerSQLExecutionStart - executionId: 5 +21/11/09 19:01:48 WARN SparkSQLExecutionContext: org.apache.spark.sql.catalyst.plans.logical.Aggregate +21/11/09 19:01:48 WARN SparkSQLExecutionContext: SparkListenerJobStart - executionId: 5 +21/11/09 19:01:48 WARN SparkSQLExecutionContext: org.apache.spark.sql.catalyst.plans.logical.Aggregate +21/11/09 19:01:49 WARN SparkSQLExecutionContext: SparkListenerJobEnd - executionId: 5 +21/11/09 19:01:49 WARN SparkSQLExecutionContext: org.apache.spark.sql.catalyst.plans.logical.Aggregate +21/11/09 19:01:49 WARN SparkSQLExecutionContext: SparkListenerSQLExecutionEnd - executionId: 5 +21/11/09 19:01:49 WARN SparkSQLExecutionContext: org.apache.spark.sql.catalyst.plans.logical.Aggregate +21/11/09 19:01:49 WARN SparkSQLExecutionContext: SparkListenerSQLExecutionEnd - executionId: 3 +21/11/09 19:01:49 WARN SparkSQLExecutionContext: org.apache.spark.sql.catalyst.plans.logical.CreateTableAsSelect

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-03-23 11:41:37
 +
+

*Thread Reply:* The JobStart event contains a Properties field and that contains a bunch of fields we want to extract to get more precise lineage information within Databricks.

+ +

As far as we know, the SQLExecutionStart event does not have any way to get these properties :(

+ +

https://github.com/OpenLineage/OpenLineage/blob/21b039b78bdcb5fb2e6c2489c4de840ebb[…]ark/agent/facets/builder/DatabricksEnvironmentFacetBuilder.java

+ +

As a result, I do have to care about the subsequent JobStart events coming from a given ExecutionStart 😢

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/21b039b78bdcb5fb2e6c2489c4de840ebb138759/integration/spark/src/main/common/java/io/openlineage/spark/agent/facets/builder/DatabricksEnvironmentFacetBuilder.java | DatabricksEnvironmentFacetBuilder.java>
+
+ <pre><code> private HashMap&amp;lt;String, Object&amp;gt; getDatabricksEnvironmentalAttributes( + SparkListenerJobStart jobStart) { + dbProperties = new HashMap&amp;lt;&amp;gt;(); + // These are useful properties to extract if they are available + + List&amp;lt;String&amp;gt; dbPropertiesKeys = + Arrays.asList( + "orgId", + "spark.databricks.clusterUsageTags.clusterOwnerOrgId", + "spark.databricks.notebook.path", + "spark.databricks.job.type", + "spark.databricks.job.id", + "spark.databricks.job.runId", + "user", + "userId", + "spark.databricks.clusterUsageTags.clusterName", + "spark.databricks.clusterUsageTags.azureSubscriptionId"); + dbPropertiesKeys.stream() + .forEach( + (p) -&amp;gt; { + dbProperties.put(p, jobStart.properties().getProperty(p)); + }); + + /**** + ** Azure Databricks makes available a dbutils mount point to list aliased paths to cloud + ** storage. However, that dbutils object is not available inside a spark listener. We must + ** access it via reflection. + **/ + try { + dbutilsClass = Class.forName("com.databricks.dbutils_v1.impl.DbfsUtilsImpl"); + dbutils = (DbfsUtils) dbutilsClass.getDeclaredConstructor().newInstance(); + dbProperties.put("mountPoints", getDatabricksMountpoints(dbutils)); + } catch (Exception e) { + log.warn("Failed to load dbutils in OpenLineageListener"); + dbProperties.put("mountPoints", new ArrayList&amp;lt;DatabricksMountpoint&amp;gt;()); + } + + return dbProperties; + } +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-03-23 11:42:33
 +
+

*Thread Reply:* I started down this path with the Project statement but I agree with @Michael Collado that a ProjectVisitor isn't a great idea.

+ +

https://github.com/OpenLineage/OpenLineage/issues/617

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/issues/617|#617 [INTEGRATION][SPARK] Support org.apache.spark.sql.catalyst.plans.logical.Project for Delta Tables>
+
+ <p>It seems that a <code>Project</code> logical plan is the terminal node for a Delta event. Currently OpenLineage does not support this as a Visitor and as a result, there are no output datasets. I'd like to propose adding a ProjectVisitor that extracts the appropriate output dataset contents.</p> + +<p>This example demonstrates the issue on Azure Databricks Runtime 6.4 (Spark 2.4.5)</p> + +<pre><code>val storageServiceName = "inputservice" +val storageContainerName = "inputcontainer" +val abfssRootPath = "abfss://"+storageContainerName+"@"+storageServiceName+".&lt;a href="http://dfs.core.windows.net"&gt;dfs.core.windows.net&lt;/a&gt;" +val outRootPath = "abfss://"+"outputdata"+"@"+storageServiceName+".&lt;a href="http://dfs.core.windows.net"&gt;dfs.core.windows.net&lt;/a&gt;" + +val exampleA = ( + spark.read.format("delta") + .load(abfssRootPath+"/examples/data/delta/exampleInputA") +) + +val exampleB = ( + spark.read.format("delta") + .load(abfssRootPath+"/examples/data/delta/exampleInputB") +) +val outputDf = exampleA.join(exampleB, exampleA("id") === exampleB("id"), "inner").drop(exampleB("id")) + +outputDf.write.format("delta").mode("append").save(outRootPath + "/outputdata/delta/delta-in-delta-out") +</code></pre> + +<p>Here's the truncated open lineage event that is produced:</p> + +<pre><code>{ + "eventType": "START", + "eventTime": "2022-03-18T12:58:53.112Z", + "run": { + "runId": "6068fde5-0e3e-4bd6-8bd9-fc3ebdb283ce", + "facets": { + "spark_unknown": { + "_producer": "&lt;https://github.com/OpenLineage/OpenLineage/tree/0.6.0-SNAPSHOT/integration/spark&gt;", + "_schemaURL": "&lt;https://openlineage.io/spec/1-0-2/OpenLineage.json#/$defs/RunFacet&gt;", + "output": { + "description": { + "@class": "org.apache.spark.sql.catalyst.plans.logical.Project", + "id": 1, + "projectList": ["some columns"], + "origin": 5, + "tags": {}, + "schema": { + "type": "struct", + "fields": ["the fields"] + } + }, + "spark_version": { + "_producer": "&lt;https://github.com/OpenLineage/OpenLineage/tree/0.6.0-SNAPSHOT/integration/spark&gt;", + "_schemaURL": "&lt;https://openlineage.io/spec/1-0-2/OpenLineage.json#/$defs/RunFacet&gt;", + "spark-version": "2.4.5", + "openlineage-spark-version": "0.6.0-SNAPSHOT" + } + } + }, + "job": { + "namespace": "adbpurviewol1", + "name": "databricks_shell.project", + "facets": {} + }, + "inputs": [ + { + "namespace": "&lt;abfss://CONTAINER@SERVICE.dfs.core.windows.net&gt;", + "name": "/examples/data/delta/exampleInputA", + "facets": { + "dataSource": { + "_producer": "&lt;https://github.com/OpenLineage/OpenLineage/tree/0.6.0-SNAPSHOT/integration/spark&gt;", + "_schemaURL": "&lt;https://openlineage.io/spec/facets/1-0-0/DatasourceDatasetFacet.json#/$defs/DatasourceDatasetFacet&gt;", + "name": "&lt;abfss://CONTAINER@SERVICE.dfs.core.windows.net&gt;", + "uri": "&lt;abfss://CONTAINER@SERVICE.dfs.core.windows.net&gt;" + }, + "schema": { + "_producer": "&lt;https://github.com/OpenLineage/OpenLineage/tree/0.6.0-SNAPSHOT/integration/spark&gt;", + "_schemaURL": "&lt;https://openlineage.io/spec/facets/1-0-0/SchemaDatasetFacet.json#/$defs/SchemaDatasetFacet&gt;", + "fields": [ + { + "name": "id", + "type": "integer" + }, + { + "name": "postalCode", + "type": "string" + }, + { + "name": "streetAddress", + "type": "string" + } + ] + } + }, + "inputFacets": {} + }, + { + "namespace": "&lt;abfss://CONTAINER@SERVICE.dfs.core.windows.net&gt;", + "name": "/examples/data/delta/exampleInputB", + "facets": { + "dataSource": { + "_producer": "&lt;https://github.com/OpenLineage/OpenLineage/tree/0.6.0-SNAPSHOT/integration/spark&gt;", + "_schemaURL": "&lt;https://openlineage.io/spec/facets/1-0-0/DatasourceDatasetFacet.json#/$defs/DatasourceDatasetFacet&gt;", + "name": "&lt;abfss://CONTAINER@SERVICE.dfs.core.windows.net&gt;", + "uri": "&lt;abfss://CONTAINER@SERVICE.dfs.core.windows.net&gt;" + }, + "schema": { + "_producer": "&lt;https://github.com/OpenLineage/OpenLineage/tree/0.6.0-SNAPSHOT/integration/spark&gt;", + "_schemaURL": "&lt;https://openlineage.io/spec/facets/1-0-0/SchemaDatasetFacet.json#/$defs/SchemaDatasetFacet&gt;", + "fields": [ + { + "name": "id", + "type": "integer" + }, + { + "name": "city", + "type": "string" + }, + { + "name": "stateAbbreviation", + "type": "string" + } + ] + } + }, + "inputFacets": {} + } + ], + "outputs": [], + "producer": "&lt;https://github.com/OpenLineage/OpenLineage/tree/0.6.0-SNAPSHOT/integration/spark&gt;", + "schemaURL": "&lt;https://openlineage.io/spec/1-0-2/OpenLineage.json#/$defs/RunEvent&gt;" +} +</code></pre> +
+ +
+
Comments
+ 2 +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-03-24 09:43:38
 +
+

Hey. I'm working on replacing current SQL parser - on which we rely for Postgres, Snowflake, Great Expectations - and I'd appreciate your opinion.

+ +

https://github.com/OpenLineage/OpenLineage/pull/627/files

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-25 19:30:29
 +
+

Am i supposed to see this when I open marquez fro the first time on an empty database?

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2022-03-25 20:33:02
 +
+

*Thread Reply:* Marquez and OpenLineage are job-focused lineage tools, so once you run a job in an OL-integrated instance of Airflow (or any other supported integration), you should see the jobs and DBs appear in the marquez ui

+ + + +
+ 👍 Marco Diaz +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-03-25 21:44:54
 +
+

*Thread Reply:* If you want to seed it with some data, just to try it out, you can run docker/up.sh -s and it will run a seeding job as it starts.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-25 19:31:09
 +
+

Would datasets be created when I send data from airflow?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2022-03-31 18:34:40
 +
+

*Thread Reply:* Yep! Marquez will register all in/out datasets present in the OL event as well as link them to the run

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2022-03-31 18:35:47
 +
+

*Thread Reply:* FYI, @Peter Hicks is working on displaying the dataset version to run relationship in the web UI, see https://github.com/MarquezProject/marquez/pull/1929

+
+ + + + +
<https://github.com/MarquezProject/marquez/pull/1929|#1929 Web: Created by view for dataset versions>
+
+ <p>Signed-off-by: Peter Hicks <a href="mailto:peter@datakin.com">peter@datakin.com</a></p> + +<p><strong>Problem</strong></p> + +<p>Some Marquez users were eager to see the run that created a dataset version.</p> + +<p><strong>Solution</strong></p> + +<p>We've added the <code>Created by Run</code> section on the dataset page as well as the dataset version list view. Also updated is syntax highlighting support for <code>sql</code> contexts and the empty states for context that is missing for us. <br /> +<a href="https://user-images.githubusercontent.com/7514204/160910381-644773d7-6ed6-499d-ad96-2ebdde14495b.png">image</a> <br /> +<a href="https://user-images.githubusercontent.com/7514204/160910466-5c62380f-0ad1-4e14-8292-c5c0b98b5ccb.png">image</a> <br /> +<a href="https://user-images.githubusercontent.com/7514204/160910773-4fbc6329-5ec3-483e-bf82-9269c0223906.png">image</a> <br /> +<a href="https://user-images.githubusercontent.com/7514204/160917522-b6a6c52f-e2bb-4590-85c1-6f6ed46b4eb7.png">image</a></p> + +<p>&gt; <strong>Note:</strong> All database schema changes require discussion. Please <a href="https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue">link the issue</a> for context.</p> + +<p><strong>Checklist</strong></p> + +<p>☑︎ You've <a href="https://github.com/MarquezProject/marquez/blob/main/CONTRIBUTING.md#sign-your-work">signed-off</a> your work +☐ Your changes are accompanied by tests (<em>if relevant</em>) +☑︎ Your change contains a <a href="https://kurtisnusbaum.medium.com/stacked-diffs-keeping-phabricator-diffs-small-d9964f4dcfa6">small diff</a> and is self-contained +☐ You've updated any relevant documentation (<em>if relevant</em>) +☐ You've updated the <a href="https://github.com/MarquezProject/marquez/blob/main/CHANGELOG.md#unreleased"><code>CHANGELOG.md</code></a> with details about your change under the "Unreleased" section (<em>if relevant, depending on the change, this may not be necessary</em>) +☐ You've versioned your <code>.sql</code> database schema migration according to <a href="https://flywaydb.org/documentation/concepts/migrations#naming">Flyway's naming convention</a> (<em>if relevant</em>)</p> +
+ +
+
Labels
+ feature, review, web, javascript +
+ +
+
Comments
+ 1 +
+ + + + + + + + + +
+ + <a href="https://github.com/MarquezProject/marquez">MarquezProject/marquez</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-28 14:31:32
 +
+

How is Datakin used in conjunction with Openlineage and Marquez?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2022-03-28 15:43:46
 +
+

*Thread Reply:* Hi Marco,

+ +

Datakin is a reporting tool built on the Marquez API, and therefore designed to take in Lineage using the OpenLineage specification.

+ +

Did you have a more specific question?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-28 15:47:53
 +
+

*Thread Reply:* No, that is it. Got it. So, i can install Datakin and still use openlineage and marquez?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2022-03-28 15:55:07
 +
+

*Thread Reply:* if you set up a datakin account, you'll have to change the environment variables used by your OpenLineage integrations, and the runEvents will be sent to Datakin rather than Marquez. You shouldn't have any loss of functionality, and you also won't have to keep manually hosting Marquez

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-28 16:10:25
 +
+

*Thread Reply:* Will I still be able to use facets for backfills?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2022-03-28 17:04:03
 +
+

*Thread Reply:* yeah it works in the same way - Datakin actually submodules the Marquez API

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-28 16:52:41
 +
+

Another question. I installed the open-lineage library and now I am trying to configure Airflow 2 to use it +Do I follow these steps?

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-28 16:53:20
 +
+

If I have marquez access via alb ingress what would i use the marquezurl variable or openlineageurl?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-28 16:54:53
 +
+

So, i don't need to modify my dags in Airflow 2 to use the library? Would this just allow me to start collecting data? +openlineage.lineage_backend.OpenLineageBackend

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-03-29 06:24:21
 +
+

*Thread Reply:* Yes, you don't need to modify dags in Airflow 2.1+

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-29 17:47:39
 +
+

*Thread Reply:* ok, I added that environment variable. Now my question is how do i configure my other variables. +I have marquez running in AWS with an ingress. +Do i use OpenLineageURL or Marquez_URL?

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-29 17:48:09
 +
+

*Thread Reply:* Also would a new namespace be created if i add the variable?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
data_fool + (data.fool.me@gmail.com) +
+
2022-03-29 02:12:30
 +
+

Hello! Are there any plans for openlineage to support dbt on trino?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2022-03-30 14:59:13
 +
+

*Thread Reply:* Hi Datafool - I'm not familiar with how trino works, but the DBT-OL integration works by wrapping the dbt run command with dtb-ol run , and capturing lineage data from the runresult file

+ +

These things don't necessarily preclude you from using OpenLineage on trino, so it may work already.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
data_fool + (data.fool.me@gmail.com) +
+
2022-03-30 18:34:38
 +
+

*Thread Reply:* hey @John Thomas yep, tried to use dbt-ol run command but it seems trino is not supported, only bigquery, redshift and few others.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2022-03-30 18:36:41
 +
+

*Thread Reply:* aaah I misunderstood what Trino is - yeah we don't currently support jobs that are running outside of those environments.

+ +

We don't currently have plans for this, but a great first step would be opening an issue in the OpenLineage repo.

+ +

If you're interested in implementing the support yourself I'm also happy to connect you to people that can help you get started.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
data_fool + (data.fool.me@gmail.com) +
+
2022-03-30 20:23:46
 +
+

*Thread Reply:* oh okay, got it, yes I can contribute, I'll see if I can get some time in the next few weeks. Thanks @John Thomas

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Francis McGregor-Macdonald + (francis@mc-mac.com) +
+
2022-03-30 16:08:39
 +
+

I can see 2 articles using Spline with BMW and Capital One. Could OpenLineage be doing the same job as Spline here? What would the differences be? +Are there any similar references for OpenLineage? I can see Northwestern Mutual but that article does not contain a lot of detail.

+
+
SpringerLink
+ + + +
Collecting and visualizing data lineage of Spark jobs
+
+ Datenbank-Spektrum - Metadata management constitutes a key prerequisite for enterprises as they engage in data analytics and governance. Today, however, the context of data is often only manually... +
+ + + + + + + + +
Original URL: https://link.springer.com/article/10.1007/s13222-021-00387-7
+ + + +
+
+
Capital One
+ + + +
Capturing &amp; Displaying Data Transformations with Spline | Capital One
+
+ Capturing data lineage manually--especially as an application’s logic changes over time--can be time consuming and prone to human error. As a data engineer, it’s advantageous to capture data lineage as a normal part of application flow, and to be able to display that data lineage in a format that is easily understood for documentation purposes. +
+ + + + + + + + +
Original URL: https://www.capitalone.com/tech/software-engineering/spline-spark-data-lineage/
+ + + +
+
+
openlineage.io
+ + + +
How Northwestern Mutual Simplified Data Observability with OpenLineage &amp; Marquez
+
+ Northwestern Mutual is building an Enterprise Data Platform. In this guest blog, learn about the experiences and decisions that led them to embrace the OpenLineage and Marquez communities. +
+ + + + + + +
Original URL: https://openlineage.io/blog/openlineage-at-northwestern-mutual/
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-31 12:47:59
 +
+

Could anyone help me wit this custom extractor. I am not sure what I am doing wrong. I added the variable to airflow2, but I still see this in the logs +[2022-03-31, 16:43:39 UTC] {__init__.py:97} WARNING - Unable to find an extractor. task_type=QueryOperator +Here is the code

+ +

```import logging +from typing import Optional, List +from openlineage.airflow.extractors.base import BaseExtractor,TaskMetadata +from openlineage.client.facet import SqlJobFacet, ExternalQueryRunFacet +from openlineage.common.sql import SqlMeta, SqlParser

+ +

logger = logging.getLogger(name)

+ +

class QueryOperatorExtractor(BaseExtractor):

+ +
def __init__(self, operator):
+    super().__init__(operator)
+
+@classmethod
+def get_operator_classnames(cls) -&gt; List[str]:
+    return ['QueryOperator']
+
+def extract(self) -&gt; Optional[TaskMetadata]:
+    # (1) Parse sql statement to obtain input / output tables.
+    sql_meta: SqlMeta = SqlParser.parse(self.operator.hql)
+    inputs = sql_meta.in_tables
+    outputs = sql_meta.out_tables
+    task_name = f"{self.operator.dag_id}.{self.operator.task_id}"
+    run_facets = {}
+    job_facets = {
+        'hql': SqlJobFacet(self.operator.hql)
+    }
+
+    return TaskMetadata(
+        name=task_name,
+        inputs=[inputs.to_openlineage_dataset()],
+        outputs=[outputs.to_openlineage_dataset()],
+        run_facets=run_facets,
+        job_facets=job_facets
+    )```
+
+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Orbit + +
+
2022-03-31 13:20:55
 +
+

@Orbit has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Orbit + +
+
2022-03-31 13:21:23
 +
+

@Orbit has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-31 14:07:24
 +
+

@Ross Turk Could you please take a look if you have a minute☝️? I know you have built one extractor before

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-03-31 14:11:35
 +
+

*Thread Reply:* Hmmmm. Are you running in Docker? Is it possible for you to shell into your scheduler container and make sure the ENV is properly set?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-03-31 14:11:57
 +
+

*Thread Reply:* looks to me like the value you posted is correct, and return ['QueryOperator'] seems right to me

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-31 14:33:00
 +
+

*Thread Reply:* It is in an EKS cluster +I checked and the variable is there +OPENLINEAGE_EXTRACTOR_QUERYOPERATOR=shared.plugins.ol_custom_extractors.QueryOperatorExtractor

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-31 14:33:56
 +
+

*Thread Reply:* I am wondering if it is an issue with my extractor code. Something not rendering well

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-03-31 14:40:17
 +
+

*Thread Reply:* I don’t think it’s even executing your extractor code. The error message traces back to here: +https://github.com/OpenLineage/OpenLineage/blob/249868fa9b97d218ee35c4a198bcdf231a9b874b/integration/airflow/openlineage/lineage_backend/__init__.py#L77

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/249868fa9b97d218ee35c4a198bcdf231a9b874b/integration/airflow/openlineage/lineage_backend/__init__.py | __init__.py>
+
+ <pre><code> extractor = self._get_extractor(task) +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-03-31 14:40:45
 +
+

*Thread Reply:* I am currently digging into _get_extractor to see where it might be missing yours 🤔

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-31 14:46:36
 +
+

*Thread Reply:* Thanks

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-03-31 14:47:19
 +
+

*Thread Reply:* silly idea, but you could add a log message to __init__ in your extractor.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-03-31 14:47:25
 +
+

*Thread Reply:* https://github.com/OpenLineage/OpenLineage/blob/249868fa9b97d218ee35c4a198bcdf231a[…]ntegration/airflow/openlineage/airflow/extractors/extractors.py

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/249868fa9b97d218ee35c4a198bcdf231a9b874b/integration/airflow/openlineage/airflow/extractors/extractors.py | extractors.py>
+
+ <pre><code> self.extractors[operator] = import_from_string(value) +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-03-31 14:48:20
 +
+

*Thread Reply:* the openlineage client actually tries to import the value of that env variable from pos 22. if that happens, but for some reason it fails to register the extractor, we can at least know that it’s importing

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-03-31 14:48:54
 +
+

*Thread Reply:* if you add a log line, you can verify that your PYTHONPATH and env are correct

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-31 14:49:23
 +
+

*Thread Reply:* will try that

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-31 14:49:29
 +
+

*Thread Reply:* and let you know

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-03-31 14:49:39
 +
+

*Thread Reply:* ok!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-03-31 15:04:05
 +
+

*Thread Reply:* @Marco Diaz can you try env variable OPENLINEAGE_EXTRACTOR_QueryOperator instead of full caps?

+ + + +
+ 👍 Ross Turk +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-31 15:13:37
 +
+

*Thread Reply:* Will try that too

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-31 15:13:44
 +
+

*Thread Reply:* Thanks for helping

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-31 16:58:24
 +
+

*Thread Reply:* @Maciej Obuchowski My setup does not allow me to submit environment variables with lowercases. Is the name of the variable used to register the extractor?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-03-31 17:15:57
 +
+

*Thread Reply:* yes, it's case sensitive...

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-31 17:18:42
 +
+

*Thread Reply:* i see

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-31 17:39:16
 +
+

*Thread Reply:* So it is definitively the name of the variable. I changed the name of the operator to capitals and now is being registered

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-03-31 17:39:44
 +
+

*Thread Reply:* Could there be a way not to make this case sensitive?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-03-31 18:31:27
 +
+

*Thread Reply:* yes - could you create issue on OpenLineage repository?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-04-01 10:46:59
 +
+

*Thread Reply:* sure

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-04-01 10:48:28
 +
+

I have another question. I have this query +INSERT OVERWRITE TABLE schema.daily_play_sessions_v2 + PARTITION (ds = '2022-03-30') + SELECT + platform_id, + universe_id, + pii_userid, + NULL as session_id, + NULL as session_start_ts, + COUNT(1) AS session_cnt, + SUM( + UNIX_TIMESTAMP(stopped) - UNIX_TIMESTAMP(joined) + ) AS time_spent_sec + FROM schema.fct_play_sessions_merged + WHERE ds = '2022-03-30' + AND UNIX_TIMESTAMP(stopped) - UNIX_TIMESTAMP(joined) BETWEEN 0 AND 28800 + GROUP BY + platform_id, + universe_id, + pii_userid +And I am seeing the following inputs +[DbTableName(None,'schema','fct_play_sessions_merged','schema.fct_play_sessions_merged')] +But the outputs are empty +Shouldn't this be an output table +schema.daily_play_sessions_v2

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-04-01 13:25:52
 +
+

*Thread Reply:* Yes, it should. This line is the likely culprit: +https://github.com/OpenLineage/OpenLineage/blob/431251d25f03302991905df2dc24357823d9c9c3/integration/common/openlineage/common/sql/parser.py#L30

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/431251d25f03302991905df2dc24357823d9c9c3/integration/common/openlineage/common/sql/parser.py | parser.py>
+
+ <pre><code> return _match_on(token, ['INTO']) +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-04-01 13:26:25
 +
+

*Thread Reply:* I bet if that said ['INTO','OVERWRITE'] it would work

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-04-01 13:27:23
 +
+

*Thread Reply:* @Maciej Obuchowski do you agree? should OVERWRITE be a token we look for? if so, I can submit a short PR.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-04-01 13:30:36
 +
+

*Thread Reply:* we have a better solution

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-04-01 13:30:37
 +
+

*Thread Reply:* https://github.com/OpenLineage/OpenLineage/pull/644

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/pull/644|#644 sql: implementation of parser API in Rust>
+
+ <p>Description in proposal doc: <a href="https://github.com/OpenLineage/OpenLineage/pull/627">#627</a></p> + +<p>Signed-off-by: Maciej Obuchowski <a href="mailto:obuchowski.maciej@gmail.com">obuchowski.maciej@gmail.com</a></p> + +<p>Closes: <a href="https://github.com/OpenLineage/OpenLineage/issues/582">#582</a></p> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-04-01 13:31:27
 +
+

*Thread Reply:* ah! I heard there was a new SQL parser, but did not know it was imminent!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-04-01 13:31:30
 +
+

*Thread Reply:* I've added this case as a test and it works: https://github.com/OpenLineage/OpenLineage/blob/764dfdb885112cd0840ebc7384ff958bf20d4a70/integration/sql/tests/tests_insert.rs

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/764dfdb885112cd0840ebc7384ff958bf20d4a70/integration/sql/tests/tests_insert.rs | tests_insert.rs>
+
+ <pre><code>use openlineage_sql::{parse_sql, SqlMeta}; +use sqlparser::dialect::PostgreSqlDialect; + +#[macro_use] +mod test_utils; +use test_utils::**; + +#[test] +fn insert_values() { + assert_eq!( + test_sql("INSERT INTO TEST VALUES(1)",), + SqlMeta { + in_tables: vec![], + out_tables: table("TEST") + } + ); +} + +#[test] +fn insert_cols_values() { + assert_eq!( + test_sql("INSERT INTO tbl(col1, col2) VALUES (1, 2), (2, 3)",), + SqlMeta { + in_tables: vec![], + out_tables: table("tbl") + } + ); +} + +#[test] +fn insert_select_table() { + assert_eq!( + test_sql("INSERT INTO TEST SELECT ** FROM TEMP",), + SqlMeta { + in_tables: table("TEMP"), + out_tables: table("TEST") + } + ); +} + +#[test] +fn insert_nested_select() { + assert_eq!(test_sql(" + INSERT INTO popular_orders_day_of_week (order_day_of_week, order_placed_on,orders_placed) + SELECT EXTRACT(ISODOW FROM order_placed_on) AS order_day_of_week, + order_placed_on, + COUNT(**) AS orders_placed + FROM top_delivery_times + GROUP BY order_placed_on; + ", + ), SqlMeta { + in_tables: table("top_delivery_times"), + out_tables: table("popular_orders_day_of_week") + }) +} + +#[test] +fn insert_cmpl() { + assert_eq!( + test_sql( + "\ + INSERT OVERWRITE TABLE schema.daily_play_sessions_v2 + PARTITION (ds = '2022-03-30') + SELECT + platform_id, + universe_id, + pii_userid, + NULL as session_id, + NULL as session_start_ts, + COUNT(1) AS session_cnt, + SUM( + UNIX_TIMESTAMP(stopped) - UNIX_TIMESTAMP(joined) + ) AS time_spent_sec + FROM schema.fct_play_sessions_merged + WHERE ds = '2022-03-30' + AND UNIX_TIMESTAMP(stopped) - UNIX_TIMESTAMP(joined) BETWEEN 0 AND 28800 + GROUP BY + platform_id, + universe_id, + pii_userid + " + ), + SqlMeta { + in_tables: table("schema.fct_play_sessions_merged"), + out_tables: table("schema.daily_play_sessions_v2") + } + ) +} + +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+ 👍 Ross Turk, Paweł Leszczyński +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-04-01 13:31:33
 +
+

*Thread Reply:* let me review this PR

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-04-01 13:36:32
 +
+

*Thread Reply:* Do i have to download a new version of the opelineage-airflow python library

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-04-01 13:36:41
 +
+

*Thread Reply:* If so which version?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-04-01 13:37:22
 +
+

*Thread Reply:* this PR isn’t merged yet 😞 so if you wanted to try this you’d have to build the python client from the sql/rust-parser-impl branch

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-04-01 13:38:17
 +
+

*Thread Reply:* ok, np. I am not in a hurry yet. Do you have an ETA for the merge?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-04-01 13:39:50
 +
+

*Thread Reply:* Hard to say, it’s currently in-review. Let me pull some strings, see if I can get eyes on it.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-04-01 13:40:34
 +
+

*Thread Reply:* I will check again next week don't worry. I still need to make some things in my extractor work

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-04-01 13:40:36
 +
+

*Thread Reply:* after it’s merged, we’ll have to do an OpenLineage release as well - perhaps next week?

+ + + +
+ 👍 Michael Robinson +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-04-01 13:40:41
 +
+

*Thread Reply:* 👍

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tien Nguyen + (tiennguyenhotel97@gmail.com) +
+
2022-04-01 12:25:48
 +
+

Hi everyone, I just started using openlineage to connect with DBT for my company. I work as data engineering. After the connection and run test on dbt-ol run, it gives me this error. I have looked up online to find the answer but couldn't see the answer anywhere. Can somebody please help me with? The error tells me that the correct version is DBT Schemajson version 2 instead of 3. I don't know where to change the schemajson version. Thank you everyone @channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-04-01 13:34:10
 +
+

*Thread Reply:* Hm - what version of dbt are you using?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-04-01 13:47:50
 +
+

*Thread Reply:* @Tien Nguyen The dbt schema version changes with different versions of dbt. If you have recently updated, you may have to make some changes: https://docs.getdbt.com/docs/guides/migration-guide/upgrading-to-v1.0

+
+
docs.getdbt.com
+ + + +
Upgrading to v1.0 | dbt Docs
+
+ Resources +
+ + + + + + + + +
Original URL: https://docs.getdbt.com/docs/guides/migration-guide/upgrading-to-v1.0
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-04-01 13:48:27
 +
+

*Thread Reply:* also make sure you are on the latest version of openlineage-dbt - I believe we have made it a bit more tolerant of dbt schema changes.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tien Nguyen + (tiennguyenhotel97@gmail.com) +
+
2022-04-01 13:52:46
 +
+

*Thread Reply:* @Ross Turk Thank you very much for your answer. I will update those and see if I can resolve the issues.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tien Nguyen + (tiennguyenhotel97@gmail.com) +
+
2022-04-01 14:20:00
 +
+

*Thread Reply:* @Ross Turk Thank you very much for your help. The latest version of dbt couldn't work. But version 0.20.0 works for this problem.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-04-01 14:22:42
 +
+

*Thread Reply:* Hmm. Interesting, I remember when dbt 1.0 came out we fixed a very similar issue: https://github.com/OpenLineage/OpenLineage/pull/397

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/pull/397|#397 [INTEGRATION][dbt] allow to parse unspecified, future version of artifacts>
+
+ <p>Currently, if <code>DbtMetadataArtifact</code> encounters artifact with schema version not exactly matching one that is supported by <code>dbt-ol</code>, it throws exception. This PR changes this behavior, when the metadata version is higher than currently supported (and tested with). If we encounter artifact with higher version, we warn the user and continue processing.</p> + +<p>Also, change raw <code>print()</code> calls to logger calls - this allows more granular control over output, and allows to test it easily.</p> + +<p>Signed-off-by: Maciej Obuchowski <a href="mailto:maciej.obuchowski@getindata.com">maciej.obuchowski@getindata.com</a></p> + +<p>Closes: <a href="https://github.com/OpenLineage/OpenLineage/issues/389">#389</a></p> + +<p><strong>Checklist</strong></p> + +<p>☑︎ You've <a href="https://github.com/OpenLineage/OpenLineage/blob/main/why-the-dco.md">signed-off</a> your work +☑︎ Your pull request title follows our <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CONTRIBUTING.md#creating-pull-requests">guidelines</a> +☑︎ Your changes are accompanied by tests (<em>if relevant</em>) +☑︎ Your change contains a <a href="https://kurtisnusbaum.medium.com/stacked-diffs-keeping-phabricator-diffs-small-d9964f4dcfa6">small diff</a> and is self-contained +☐ You've updated any relevant documentation (<em>if relevant</em>) +☐ You've updated the <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md"><code>CHANGELOG.md</code></a> with details about your change under the "Unreleased" section (<em>if relevant, depending on the change, this may not be necessary</em>) +☐ You've versioned the core OpenLineage model or facets according to <a href="https://docs.snowplowanalytics.com/docs/pipeline-components-and-applications/iglu/common-architecture/schemaver">SchemaVer</a> (<em>if relevant</em>)</p> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-04-01 14:25:17
 +
+

*Thread Reply:* if you run pip3 list | grep openlineage-dbt, what version does it show?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-04-01 14:26:26
 +
+

*Thread Reply:* I wonder if you have somehow ended up with an older version of the integration

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tien Nguyen + (tiennguyenhotel97@gmail.com) +
+
2022-04-01 14:33:43
 +
+

*Thread Reply:* it is 0.1.0

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tien Nguyen + (tiennguyenhotel97@gmail.com) +
+
2022-04-01 14:34:23
 +
+

*Thread Reply:* is it 0.1.0 the older version of openlineage ?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-04-01 14:43:14
 +
+

*Thread Reply:* ❯ pip3 list | grep openlineage-dbt +openlineage-dbt 0.6.2

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-04-01 14:43:26
 +
+

*Thread Reply:* the latest is 0.6.2 - that might be your issue

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-04-01 14:43:59
 +
+

*Thread Reply:* How are you going about installing it?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tien Nguyen + (tiennguyenhotel97@gmail.com) +
+
2022-04-01 18:35:26
 +
+

*Thread Reply:* @Ross Turk. I follow instruction from open lineage "pip3 install openlineage-dbt"

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-04-01 18:36:00
 +
+

*Thread Reply:* Hm! Interesting. I did the same thing to get 0.6.2.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tien Nguyen + (tiennguyenhotel97@gmail.com) +
+
2022-04-01 18:51:36
 +
+

*Thread Reply:* @Ross Turk Yes. I have tried to reinstall and clear cache but it still install 0.1.0

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tien Nguyen + (tiennguyenhotel97@gmail.com) +
+
2022-04-01 18:53:07
 +
+

*Thread Reply:* But thanks for the version. I reinstall 0.6.2 version by specify the version

+ + + +
+ 👍 Ross Turk +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-04-02 17:37:59
 +
+

@Ross Turk @Maciej Obuchowski FYI the sql parser also seems not to return any inputs or outpus for queries that have subqueries +Example +INSERT OVERWRITE TABLE mytable + PARTITION (ds = '2022-03-31') + SELECT + ** + FROM + (SELECT ** FROM table2) a +INSERT OVERWRITE TABLE mytable + PARTITION (ds = '2022-03-31') + SELECT + ** + FROM + (SELECT ** FROM table2 + UNION + SELECT ** FROM table3 + UNION ALL + SELECT ** FROM table4) a

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-04-03 15:07:09
 +
+

*Thread Reply:* they'll work with new parser - added test for those

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/2e8e51981483e17534445a751723dda787794739/integration/sql/tests/tests_insert.rs | tests_insert.rs>
+
+ <pre><code>fn insert_overwrite_multiple_subqueries() { +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-04-03 15:07:39
 +
+

*Thread Reply:* btw, thank you very much for notifying us about multiple bugs @Marco Diaz!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-04-03 15:20:55
 +
+

*Thread Reply:* @Maciej Obuchowski thank you for making sure these cases are taken into account. I am getting more familiar with the Open lineage code as i build my extractors. If I see anything else I will let you know. Any ETA on the new parser release date?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-04-03 15:55:28
 +
+

*Thread Reply:* it should be week-two, unless anything comes up

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-04-03 17:10:02
 +
+

*Thread Reply:* I see. Keeping my fingers crossed this is the only thing delaying me right now.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-04-02 20:27:37
 +
+

Also what would happen if someone uses a CTE in the SQL? Is the parser taken those cases in consideration?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-04-03 15:02:13
 +
+

*Thread Reply:* current one handles cases where you have one CTE (like this test) but not multiple - next one will handle arbitrary number of CTEs (like this test)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-04-04 10:54:47
 +
+

Agenda items are requested for the next OpenLineage Technical Steering Committee meeting on Wednesday, April 13. Please reply here or ping me with your items!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-04-04 11:11:53
 +
+

*Thread Reply:* I've mentioned it before but I want to talk a bit about new SQL parser

+ + + +
+ 🙌 Will Johnson, Ross Turk +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-04-04 13:25:17
 +
+

*Thread Reply:* Will the parser be released after the 13?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-04-08 11:47:05
 +
+

*Thread Reply:* @Michael Robinson added additional item to Agenda - client transports feature that we'll have in next release

+ + + +
+ 🙌 Michael Robinson +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-04-08 12:56:44
 +
+

*Thread Reply:* Thanks, Maciej

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sukanya Patra + (Sukanya_Patra@mckinsey.com) +
+
2022-04-05 02:39:59
 +
+

Hi Everyone,

+ +

I have come across OpenLineage at Data Council Austin, 2022 and am curious to try it out. I have reviewed the Getting Started section (https://openlineage.io/getting-started/) of OpenLineage docs but couldn't find clear reference documentation for using the API +• Are there any swagger API docs or equivalent dedicated for OpenLineage API? There is some reference docs of Marquez API: https://marquezproject.github.io/marquez/openapi.html#tag/Lineage +Secondly are there any means to use Open Lineage independent of Marquez? Any pointers would be appreciated.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Patrick Mol + (patrick.mol@prolin.com) +
+
2022-04-05 10:28:08
 +
+

*Thread Reply:* I had kind of the same question. +I found https://marquezproject.github.io/marquez/openapi.html#tag/Lineage +With some of the entries marked Deprecated, I am not sure how to proceed.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2022-04-05 11:55:35
 +
+

*Thread Reply:* Hey folks, are you looking for the OpenAPI specification found here?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2022-04-05 15:33:23
 +
+

*Thread Reply:* @Patrick Mol, Marquez's deprecated endpoints were the old methods for creating lineage (making jobs, dataset, and runs independently), they were deprecated because we moved over to using the OpenLineage spec for all lineage collection purposes.

+ +

The GET methods for jobs/datasets/etc are still functional

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sarat Chandra + (saratchandra9494@gmail.com) +
+
2022-04-05 21:10:39
 +
+

*Thread Reply:* Hey John,

+ +

Thanks for sharing the OpenAPI docs. Was wondering if there are any means to setup OpenLineage API that will receive events without a consumer like Marquez or is it essential to always pair with a consumer to receive the events?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2022-04-05 21:47:13
 +
+

*Thread Reply:* the OpenLineage integrations don’t have any way to recieve events, since they’re designed to send events to other apps - what were you expecting OpenLinege to do?

+ +

Marquez is our reference implementation of an OpenLineage consumer, but egeria also has a functional endpoint

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Patrick Mol + (patrick.mol@prolin.com) +
+
2022-04-06 09:53:31
 +
+

*Thread Reply:* Hi @John Thomas, +Would creation of Sources and Datasets have an equivalent in the OpenLineage specification ? +Sofar I only see the Inputs and Outputs in the Run Event spec.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2022-04-06 11:31:10
 +
+

*Thread Reply:* Inputs and outputs in the OL spec are Datasets in the old MZ spec, so they're equivalent

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-04-05 14:24:50
 +
+

Hey Guys,

+ +

The BaseExtractor is working fine with operators that are derived from Airflow BaseOperator. However for operators derived from LivyOperator the BaseExtractor does not seem to work. Is there a fix for this? We use livyoperator to run sparkjobs

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2022-04-05 15:16:34
 +
+

*Thread Reply:* Hi Marco - it looks like LivyOperator itself does derive from BaseOperator, have you seen any other errors around this problem?

+ +

@Maciej Obuchowski might be more help here

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-04-05 15:21:03
 +
+

*Thread Reply:* It is the operators that inherit from LivyOperator. It doesn't find the parameters like sql, connection etc

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-04-05 15:25:42
 +
+

*Thread Reply:* My guess is that operators that inherit from other operators (not baseoperator) will have the same problem

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2022-04-05 15:32:13
 +
+

*Thread Reply:* interesting! I'm not sure about that. I can look into it if I have time, but Maciej is definitely the person who would know the most.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-04-06 15:49:48
 +
+

*Thread Reply:* @Marco Diaz I wonder - perhaps it would be better to instrument spark with OpenLineage. It doesn’t seem that Airflow will know much about what’s happening underneath here. Have you looked into openlineage-spark?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-04-06 15:51:57
 +
+

*Thread Reply:* I have not tried that library yet. I need to see how it implement because we have several spark custom operators that use livy

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-04-06 15:52:59
 +
+

*Thread Reply:* Do you have any examples?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-04-06 15:54:01
 +
+

*Thread Reply:* there is a good blog post from @Michael Collado: https://openlineage.io/blog/openlineage-spark/

+
+
openlineage.io
+ + + +
Tracing Data Lineage with OpenLineage and Apache Spark
+
+ Spark ushered in a brand new age of data democratization... and left us with a mess of hidden dependencies, stale datasets, and failed jobs. +
+ + + + + + +
Original URL: https://openlineage.io/blog/openlineage-spark/
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-04-06 15:54:37
 +
+

*Thread Reply:* and the doc page here has a good overview: +https://openlineage.io/integration/apache-spark/

+
+
openlineage.io
+ + + +
Apache Spark
+
+ OpenLineage can automatically track lineage of jobs and datasets across Spark jobs. +
+ + + + + + +
Original URL: https://openlineage.io/integration/apache-spark/
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-04-06 16:38:15
 +
+

*Thread Reply:* is this all we need to pass? +spark-submit --conf "spark.extraListeners=io.openlineage.spark.agent.OpenLineageSparkListener" \ + --packages "io.openlineage:openlineage_spark:0.2.+" \ + --conf "spark.openlineage.host=http://&lt;your_ol_endpoint&gt;" \ + --conf "spark.openlineage.namespace=my_job_namespace" \ + --class com.mycompany.MySparkApp my_application.jar

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-04-06 16:38:49
 +
+

*Thread Reply:* If so, yes our operators have a way to pass configurations to spark and we may be able to implement it.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2022-04-06 16:41:27
 +
+

*Thread Reply:* Looks right to me

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-04-06 16:42:03
 +
+

*Thread Reply:* Will give it a try

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-04-06 16:42:50
 +
+

*Thread Reply:* Do we have to install the library on the spark side or the airflow side?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-04-06 16:42:58
 +
+

*Thread Reply:* I assume is the spark side

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2022-04-06 16:44:25
 +
+

*Thread Reply:* The —packages argument tells spark where to get the jar (you'll want to upgrade to 0.6.1)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-04-06 16:44:54
 +
+

*Thread Reply:* sounds good

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Varun Singh + (varuntestaz@outlook.com) +
+
2022-04-06 00:04:14
 +
+

Hi, I saw there was some work done for integrating OpenLineage with Azure Purview + + + +

+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-04-06 04:54:27
 +
+

*Thread Reply:* @Will Johnson

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-04-07 12:43:27
 +
+

*Thread Reply:* Hey @Varun Singh! We are building a github repository that deploys a few resources that will support a limited number of Azure data sources being pushed into Azure Purview. You can expect a public release near the end of the month! Feel free to direct message me if you'd like more details!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-04-06 15:05:39
 +
+

The next OpenLineage Technical Steering Committee meeting is Wednesday, April 13! Meetings are on the second Wednesday of each month from 9:00 to 10:00am PT. +Join us on Zoom: https://astronomer.zoom.us/j/87156607114?pwd=a3B0K210dnRaQmdkaFdGMytBREZEQT09 +All are welcome. +Agenda: +• OpenLineage 0.6.2 release overview +• Airflow integration update +• Dagster integration retrospective +• Open discussion +Notes: https://tinyurl.com/openlineagetsc

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
slackbot + +
+
2022-04-06 21:40:16
 +
+

This message was deleted.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-04-07 01:00:43
 +
+

*Thread Reply:* Are both airflow2 and Marquez installed locally on your computer?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jorge Reyes (Zenta Group) + (jorge.reyes@zentagroup.com) +
+
2022-04-07 09:04:19
 +
+

*Thread Reply:* yes Marco

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-04-07 15:00:18
 +
+

*Thread Reply:* can you open marquez on +<http://localhost:3000>

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-04-07 15:00:40
 +
+

*Thread Reply:* and get a response from +<http://localhost:5000/api/v1/namespaces>

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jorge Reyes (Zenta Group) + (jorge.reyes@zentagroup.com) +
+
2022-04-07 15:26:41
 +
+

*Thread Reply:* yes , i used this guide https://openlineage.io/getting-started and execute un post to marquez correctly

+
+
openlineage.io
+ + + +
Getting Started
+
+ Data lineage is the foundation for a new generation of powerful, context-aware data tools and best practices. OpenLineage enables consistent collection of lineage metadata, creating a deeper understanding of how data is produced and used. +
+ + + + + + +
Original URL: https://openlineage.io/getting-started
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-04-07 22:17:34
 +
+

*Thread Reply:* In theory you should receive events in jobs under airflow namespace

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tien Nguyen + (tiennguyenhotel97@gmail.com) +
+
2022-04-07 14:18:05
 +
+

Hi Everyone, Can someone please help me to debug this error ? Thank you very much all

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2022-04-07 14:59:06
 +
+

*Thread Reply:* It looks like you need to add a payment method to your DBT account

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tyler Farris + (tyler@kickstand.work) +
+
2022-04-11 12:46:41
 +
+

Hello. Does Airflow's TaskFlow API work with OpenLineage?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-04-11 12:50:48
 +
+

*Thread Reply:* It does, but admittedly not very well. It can't recognize what you're doing inside your tasks. The good news is that we're working on it and long term everything should work well.

+ + + +
+ 👍 Howard Yoo +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tyler Farris + (tyler@kickstand.work) +
+
2022-04-11 12:58:28
 +
+

*Thread Reply:* Thanks for the quick reply Maciej.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
sandeep + (sandeepgame07@gmail.com) +
+
2022-04-12 09:56:44
 +
+

Hi all, watched few of your demos with airflow(astronomer) recently, really liked them. +Thanks for doing those

+ +

Questions:

+ +
  1. Are there plans to have a hive listener similar to the open-lineage spark integration ?
  2. If not will the sql parser work with the HiveQL ?
  3. Maybe one for presto too ?
  4. Will the run version and dataset version come out of the box or do we need to define some facets ?
  5. I read the blog on facets, is there a tutorial on how to create a sample facet ? +Background: +We have hive, spark jobs and big query tasks running from airflow in GCP Dataproc
    6. +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2022-04-12 13:56:53
 +
+

*Thread Reply:* Hi Sandeep,

+ +

1&3: We don't currently have Hive or Presto on the roadmap! The best way to start the conversation around them would be to create a proposal in the OpenLineage repo, outlining your thoughts on implementation and benefits.

+ +

2: I'm not familiar enough with HiveQL, but you can read about the new SQL parser we're implementing here

+ +

  1. you can see the Standard Facets here - Dataset Version is included out of the box, but Run Version would have to be defined.

  2. the best place to start looking into making facets is the Spec doc here. We don't have a dedicated tutorial, but if you have more specific questions please feel free to reach out again on slack

    3. +
+ + + +
+ 👍 sandeep +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
sandeep + (sandeepgame07@gmail.com) +
+
2022-04-12 15:39:23
 +
+

*Thread Reply:* Thank you John +The standard facets links to the github issues currently

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2022-04-12 15:40:33
 +
+

*Thread Reply:* ah here -https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.md#standard-facets

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
sandeep + (sandeepgame07@gmail.com) +
+
2022-04-12 15:41:01
 +
+

*Thread Reply:* Will check it out thank you

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-04-12 10:37:58
 +
+

Reminder: this month’s OpenLineage TSC meeting is tomorrow, 4/13, at 9 am PT! https://openlineage.slack.com/archives/C01CK9T7HKR/p1649271939878419

+
+ + +
+ + + } + + Michael Robinson + (https://openlineage.slack.com/team/U02LXF3HUN7) +
+ + + +
+
+ <p>The next OpenLineage Technical Steering Committee meeting is Wednesday, April 13! Meetings are on the second Wednesday of each month from 9:00 to 10:00am PT. +Join us on Zoom: <a href="https://astronomer.zoom.us/j/87156607114?pwd=a3B0K210dnRaQmdkaFdGMytBREZEQT09">https://astronomer.zoom.us/j/87156607114?pwd=a3B0K210dnRaQmdkaFdGMytBREZEQT09</a> +All are welcome. +Agenda: +• OpenLineage 0.6.2 release overview +• Airflow integration update +• Dagster integration retrospective +• Open discussion +Notes: <a href="https://tinyurl.com/openlineagetsc">https://tinyurl.com/openlineagetsc</a></p> +
+ + + + + + +
Original URL: https://openlineage.slack.com/archives/C01CK9T7HKR/p1649271939878419
+ + +
+ + Slack Conversation +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
sandeep + (sandeepgame07@gmail.com) +
+
2022-04-12 15:43:29
 +
+

I setup the open-lineage spark integration for spark(dataproc) tasks from airflow. It’s able to post data to the marquez end point and I see the job information in Marquez UI.

+ +

I don’t see any dataset information in it, I see just the jobs ? Is there some setup I need to do or something else I need to configure ?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2022-04-12 16:08:30
 +
+

*Thread Reply:* is there anything in your marquez-api logs that might indicate issues?

+ +

What guide did you follow to setup the spark integration?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
sandeep + (sandeepgame07@gmail.com) +
+
2022-04-12 16:10:07
 +
+

*Thread Reply:* Followed this guide https://openlineage.io/integration/apache-spark/ and used the spark-defaults.conf approach

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
sandeep + (sandeepgame07@gmail.com) +
+
2022-04-12 16:11:04
 +
+

*Thread Reply:* The logs from dataproc side show no errors, let me check from the marquez api side +To confirm, we should be able to see the datasets from the marquez UI with the spark integration right ?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2022-04-12 16:11:50
 +
+

*Thread Reply:* I'm not super familiar with the spark integration, since I work more with airflow - I'd start with looking through the readme for the spark integration here

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
sandeep + (sandeepgame07@gmail.com) +
+
2022-04-12 16:14:44
 +
+

*Thread Reply:* Hmm, the readme says it aims to generate the input and output datasets

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-04-12 16:40:38
 +
+

*Thread Reply:* Are you looking at the same namespace?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
sandeep + (sandeepgame07@gmail.com) +
+
2022-04-12 16:40:51
 +
+

*Thread Reply:* Yes, the same one where I can see the job

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
sandeep + (sandeepgame07@gmail.com) +
+
2022-04-12 16:54:49
 +
+

*Thread Reply:* Tailing the API logs and rerunning the spark job now to hopefully catch errors if any, will ping back here

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
sandeep + (sandeepgame07@gmail.com) +
+
2022-04-12 17:01:10
 +
+

*Thread Reply:* Don’t see any failures in the logs, any suggestions on how to debug this ?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2022-04-12 17:08:24
 +
+

*Thread Reply:* I'd next set up a basic spark notebook and see if you can't get it to send dataset information on something simple in order to check if it's a setup issue or a problem with your spark job specifically

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
sandeep + (sandeepgame07@gmail.com) +
+
2022-04-12 17:14:43
 +
+

*Thread Reply:* ok, that sounds good, will try that

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
sandeep + (sandeepgame07@gmail.com) +
+
2022-04-12 17:16:06
 +
+

*Thread Reply:* before that, I see that spark-lineage integration posts lineage to the api +https://marquezproject.github.io/marquez/openapi.html#tag/Lineage/paths/~1lineage/post +We don’t seem to add a DataSet in this, does marquez internally create this “dataset” based on Output and fields ?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2022-04-12 17:16:34
 +
+

*Thread Reply:* yeah, you should be seeing "input" and "output" in the runEvents - that's where datasets come from

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2022-04-12 17:17:00
 +
+

*Thread Reply:* I'm not sure if it's a problem with your specific spark job or with the integration itself, however

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
sandeep + (sandeepgame07@gmail.com) +
+
2022-04-12 17:19:16
 +
+

*Thread Reply:* By runEvents, do you mean a job Object or lineage Object ? +The integration seems to be only POSTing lineage objects

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2022-04-12 17:20:34
 +
+

*Thread Reply:* yep, a runEvent is body that gets POSTed to the /lineage endpoint:

+ +

https://openlineage.io/docs/openapi/

+ + + +
+ 👍 sandeep +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-04-12 17:41:01
 +
+

*Thread Reply:* > Yes, the same one where I can see the job +I think you should look at other namespace, which name depends on what systems you're actually using

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
sandeep + (sandeepgame07@gmail.com) +
+
2022-04-12 17:48:24
 +
+

*Thread Reply:* Shouldn’t the dataset would be created in the same namespace we define in the spark properties?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
sandeep + (sandeepgame07@gmail.com) +
+
2022-04-15 10:19:06
 +
+

*Thread Reply:* I found few datasets in the table location, I ran it in a similar (hive metastore, gcs, sparksql and scala spark jobs) setup to the one mentioned in this post https://openlineage.slack.com/archives/C01CK9T7HKR/p1649967405659519

+
+ + +
+ + + } + + Will Johnson + (https://openlineage.slack.com/team/U02H4FF5M36) +
+ + + +
+
+ <p>Before I create an issue around it, maybe I'm just not seeing it in Databricks. In the Spark Integration, does OpenLineage report Hive Metastore tables or it ONLY reports the file path?</p> + +<p>For example, if I have a Hive table called default.myTable stored at LOCATION /usr/hive/warehouse/default/mytable.</p> + +<p>For a query that reads a CSV file and inserts into default.myTable, would I see an output of default.myTable or /usr/hive/warehoues/default/mytable?</p> + +<p>We want to include a link between the physical path and the hive metastore table but it seems that OpenLineage (at least on Databricks) only reports the physical path with the table name showing up in the catalog but not as a facet.</p> +
+ + + + + + +
Original URL: https://openlineage.slack.com/archives/C01CK9T7HKR/p1649967405659519
+ + +
+ + Thread in Slack Conversation +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
sandeep + (sandeepgame07@gmail.com) +
+
2022-04-12 15:49:46
 +
+

Is this the correct place for this Q or should I reach out to Marquez slack ? +I followed this post https://openlineage.io/integration/apache-spark/

+
+
openlineage.io
+ + + +
Apache Spark
+
+ OpenLineage can automatically track lineage of jobs and datasets across Spark jobs. +
+ + + + + + +
Original URL: https://openlineage.io/integration/apache-spark/
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-04-14 16:16:45
 +
+

Before I create an issue around it, maybe I'm just not seeing it in Databricks. In the Spark Integration, does OpenLineage report Hive Metastore tables or it ONLY reports the file path?

+ +

For example, if I have a Hive table called default.myTable stored at LOCATION /usr/hive/warehouse/default/mytable.

+ +

For a query that reads a CSV file and inserts into default.myTable, would I see an output of default.myTable or /usr/hive/warehoues/default/mytable?

+ +

We want to include a link between the physical path and the hive metastore table but it seems that OpenLineage (at least on Databricks) only reports the physical path with the table name showing up in the catalog but not as a facet.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
sandeep + (sandeepgame07@gmail.com) +
+
2022-04-15 10:17:55
 +
+

*Thread Reply:* This was my experience as well, I was under the impression we would see the table as a dataset. +Looking forward to understanding the expected behavior

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-04-15 10:39:34
 +
+

*Thread Reply:* relevant: https://github.com/OpenLineage/OpenLineage/issues/435

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/issues/435|#435 [INTEGRATION][Spark] provide physical location of a table facet in case we're using logical table name>
+
+ <p>When dealing with a logical Hive table, we've decided to use logical Hive table name as a dataset name. In some other cases, some other job - not necessarily Spark one - might deal with this dataset by directly knowing it's physical location. In this case we'll have the same dataset known by two different names in different namespaces.</p> + +<p>To make sure those datasets can be "linked" by OpenLineage consumers, when available, we should provide additional facet (<code>PhysicalLocationFacet</code>?) that indicates physical location at a storage system (like <code>hdfs</code>, <code>s3</code> etc) or other underlying storage mechanism.</p> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+ 👍 Howard Yoo +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-04-15 12:36:08
 +
+

*Thread Reply:* Ah! Thank you both for confirming this! And it's great to see the proposal, Maciej!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sharanya Santhanam + (santhanamsharanya@gmail.com) +
+
2022-06-10 12:37:41
 +
+

*Thread Reply:* Is there a timeline around when we can expect this fix ?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-06-10 12:46:47
 +
+

*Thread Reply:* Not a simple fix, but I guess we'll start working on this relatively soon.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sharanya Santhanam + (santhanamsharanya@gmail.com) +
+
2022-06-10 13:10:31
 +
+

*Thread Reply:* I see, thanks for the update ! We are very much interested in this feature.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-04-15 15:42:22
 +
+

@channel A significant number of us have a conflict with the current TSC meeting day/time, so, unfortunately, we need to reschedule the meeting. When you have a moment, please share your availability here: https://doodle.com/meeting/participate/id/ejRnMlPe. Thanks in advance for your input!

+
+
doodle.com
+ + + +
Doodle
+
+ Doodle is the simplest way to schedule meetings with clients, colleagues, or friends. Find the best time for one-to-ones and team meetings with our user-friendly calendar tool. Get started today! +
+ + + + + + +
Original URL: https://doodle.com/meeting/participate/id/ejRnMlPe
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Arturo + (ggrmos@gmail.com) +
+
2022-04-19 13:35:23
 +
+

Hello everyone, I'm learning Openlineage, I finally achieved the connection between Airflow 2+ and Openlineage+Marquez. The issue is that I don't see nothing on Marquez. Do I need to modify current airflow operators?

+ +
+
+ + + + + + +
+
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-04-19 13:40:54
 +
+

*Thread Reply:* You probably need to change dataset from default

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Arturo + (ggrmos@gmail.com) +
+
2022-04-19 13:47:04
 +
+

*Thread Reply:* I click it on everything 😕 I manually (joining to the pod and send curl to the marquez local endpoint) created a namespaces to check if there is a network issue I was ok, I created a namespaces called: data-dev . The airflow is mounted over k8s using helm chart. +``` config: + AIRFLOWWEBSERVERBASEURL: "http://airflow.dev.test.io" + PYTHONPATH: "/opt/airflow/dags/repo/config" + AIRFLOWAPIAUTHBACKEND: "airflow.api.auth.backend.basicauth" + AIRFLOWCOREPLUGINSFOLDER: "/opt/airflow/dags/repo/plugins" + AIRFLOWLINEAGEBACKEND: "openlineage.lineage_backend.OpenLineageBackend"

+ +

. +. +. +.

+ +

extraEnv: + - name: OPENLINEAGEURL + value: http://marquez-dev.data-dev.svc.cluster.local + - name: OPENLINEAGENAMESPACE + value: data-dev```

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-04-19 15:16:47
 +
+

*Thread Reply:* I think answer is somewhere in airflow logs 🙂 +For some reason, OpenLineage events aren't send to Marquez.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Arturo + (ggrmos@gmail.com) +
+
2022-04-20 11:08:09
 +
+

*Thread Reply:* Thanks, finally was my error .. I created a dummy dag to see if maybe it's an issue over the dag and now I can see something over Marquez

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mirko Raca + (racamirko@gmail.com) +
+
2022-04-20 08:15:32
 +
+

One really novice question - there doesn't seem to be a way of deleting lineage elements (any of them)? While I can imagine that in production system we want to keep history, it's not practical while testing/developing. I'm using throw-away namespaces to step around the issue. Is there a better way, or alternatively - did I miss an API somewhere?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-04-20 08:20:35
 +
+

*Thread Reply:* That's more of a Marquez question 🙂 +We have a long-standing issue to add that API https://github.com/MarquezProject/marquez/issues/1736

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mirko Raca + (racamirko@gmail.com) +
+
2022-04-20 09:32:19
 +
+

*Thread Reply:* I see it already got skipped for 2 releases, and my only conclusion is that people using Marquez don't make mistakes - ergo, API not needed 🙂 Lets see if I can stick around the project long enough to offer a bit of help, now I just need to showcase it and get interest in my org.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Dan Mahoney + (dan.mahoney@sphericalanalytics.io) +
+
2022-04-20 10:08:33
 +
+

Good day all. I’m trying out the openlineage-dagster plugin +• I’ve got dagit, dagster-daemon and marquez running locally +• The openlineagesensor is recognized in dagit and the daemon. +But, when I run a job, I see the following message in the daemon’s shell: +Sensor openlineage_sensor skipped: Last cursor: {"last_storage_id": 9, "running_pipelines": {"97e2efdf-9499-4ffd-8528-d7fea5b9362c": {"running_steps": {}, "repository_name": "hello_cereal_repository"}}} +I’ve attached my repos.py and serialjob.py. +Any thoughts?

+ +
+
serial_job.py
+ + + + + + +
+
+
repos.py
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
David + (drobin1437@gmail.com) +
+
2022-04-20 10:40:03
 +
+

Hi All, +I am walking through the curl examples on this page and have a question on the first curl example: +https://openlineage.io/getting-started/ +The curl command completes, and I can see the input file and job in the namespace, but the lineage graph does not show the input file connected as an input to the job. This only seems to happen after the job is marked complete.

+ +

Is there a way to have a running job show connections to its input files in the lineage? +Thanks!

+
+
openlineage.io
+ + + +
Getting Started
+
+ Data lineage is the foundation for a new generation of powerful, context-aware data tools and best practices. OpenLineage enables consistent collection of lineage metadata, creating a deeper understanding of how data is produced and used. +
+ + + + + + +
Original URL: https://openlineage.io/getting-started/
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
raghanag + (raghanag@gmail.com) +
+
2022-04-20 18:06:29
 +
+

Hi Team, we are using spark as a service, and we are planning to integrate open lineage spark listener and looking at the below params that we need to pass, we don't know the name of the spark cluster, is the spark.openlineage.namespace conf param mandatory? +spark-submit --conf "spark.extraListeners=io.openlineage.spark.agent.OpenLineageSparkListener" \ + --packages "io.openlineage:openlineage_spark:0.2.+" \ + --conf "spark.openlineage.host=http://&lt;your_ol_endpoint&gt;" \ + --conf "spark.openlineage.namespace=my_job_namespace" \ + --class com.mycompany.MySparkApp my_application.jar

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-04-20 18:11:19
 +
+

*Thread Reply:* Namespace is defined by you, it does not have to be name of the spark cluster.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-04-20 18:11:42
 +
+

*Thread Reply:* And I definitely recommend to use newer version than 0.2.+ 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
raghanag + (raghanag@gmail.com) +
+
2022-04-20 18:13:32
 +
+

*Thread Reply:* oh i see that someone mentioned that it has to be replaced with name of the spark clsuter

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
raghanag + (raghanag@gmail.com) +
+
2022-04-20 18:13:57
 +
+

*Thread Reply:* https://openlineage.slack.com/archives/C01CK9T7HKR/p1634089656188400?thread_ts=1634085740.187700&cid=C01CK9T7HKR

+
+ + +
+ + + } + + Julien Le Dem + (https://openlineage.slack.com/team/U01DCLP0GU9) +
+ + + +
+
+ <p><code>SparkSession.builder() + .config("spark.jars.packages", "io.openlineage:openlineage_spark:0.2.+") + .config("spark.extraListeners", "io.openlineage.spark.agent.OpenLineageSparkListener") + .config("spark.openlineage.host", "&lt;https://localhost&gt;") + .config("spark.openlineage.apiKey", "your api key") + .config("spark.openlineage.namespace", "&amp;lt;NAMESPACE_NAME&amp;gt;") // Replace with the name of your Spark cluster. + .getOrCreate()</code></p> +
+ + + + + + +
Original URL: https://openlineage.slack.com/archives/C01CK9T7HKR/p1634089656188400?thread_ts=1634085740.187700&amp;cid=C01CK9T7HKR
+ + +
+ + Thread in Slack Conversation +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
raghanag + (raghanag@gmail.com) +
+
2022-04-20 18:19:19
 +
+

*Thread Reply:* @Maciej Obuchowski may i know if i can add the --packages "io.openlineage:openlineage_spark:0.2.+" as part of the spark jar file, that meant as part of the pom.xml

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-04-21 03:54:25
 +
+

*Thread Reply:* I think it needs to run on the driver

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mirko Raca + (racamirko@gmail.com) +
+
2022-04-21 05:53:34
 +
+

Hello, +when looking through Marquez API it seems that most individual-element creation APIs are marked as deprecated and are going to be removed by 0.25, with a point of switching to open lineage. That makes POST to /api/v1/lineage the only creation point of elements, but OpenLineage API is very limited in attributes that can be passed.

+ +

Is that intended to stay that way? One practical question/example: how do we create a job of type STREAMING, when OL API only allows to pass name, namespace and facets. Do we now move all properties into facets?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-04-21 07:16:44
 +
+

*Thread Reply:* > OpenLineage API is very limited in attributes that can be passed. +Can you specify where do you think it's limited? The way to solve that problems would be to evolve OpenLineage.

+ +

> One practical question/example: how do we create a job of type STREAMING, +So, here I think the question is more how streaming jobs differ from batch jobs. One obvious difference is that output of the job is continuous (in practice, probably "microbatched" or commited on checkpoint). However, deprecated Marquez API didn't give us tools to properly indicate that. On the contrary, OpenLineage with different event types allows us to properly do that. +> Do we now move all properties into facets? +Basically, yes. Marquez should handle specific facets. For example, https://github.com/MarquezProject/marquez/pull/1847

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mirko Raca + (racamirko@gmail.com) +
+
2022-04-21 07:23:11
 +
+

*Thread Reply:* Hey Maciej

+ +

first off - thanks for being active on the channel!

+ +

> So, here I think the question is more how streaming jobs differ from batch jobs +Not really. I just gave an example of how would you express a specific job type creation which can be done with https://marquezproject.github.io/marquez/openapi.html#tag/Jobs/paths/~1namespaces~1{namespace}~1jobs~1{job}/put|/api/v1/namespaces/.../jobs/... , by passing the type field which is required. In the call to /api/v1/lineage the job field offers just to specify (namespace, name), but no other attributes.

+ +

> However, deprecated Marquez API didn't give us tools to properly indicate that. On the contrary, OpenLineage with different event types allows us to properly do that. +I have the feeling I'm still missing some key concepts on how OpenLineage is designed. I think I went over the API and documentation, but trying to use just OpenLineage failed to reproduce mildly complex chain-of-job scenarios, and when I took a look how Marquez seed demo is doing it - it was heavily based on deprecated API. So, I'm kinda lost on how to use OpenLineage.

+ +

I'm looking forward to some open-public meeting, as I don't think asking these long questions on chat really works. 😞 +Any pointers are welcome!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-04-21 07:53:59
 +
+

*Thread Reply:* > I just gave an example of how would you express a specific job type creation +Yes, but you're trying to achieve something by passing this parameter or creating a job in a certain way. We're trying to cover everything in OpenLineage API. Even if we don't have everything, the spec from the beginning is focused to allow emitting custom data by custom facet mechanism.

+ +

> I have the feeling I'm still missing some key concepts on how OpenLineage is designed. +This talk by @Julien Le Dem is a great place to start: https://www.youtube.com/watch?v=HEJFCQLwdtk

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-04-21 11:29:20
 +
+

*Thread Reply:* > Any pointers are welcome! +BTW: OpenLineage is an open standard. Everyone is welcome to contribute and discuss. Every feedback ultimately helps us build better systems.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mirko Raca + (racamirko@gmail.com) +
+
2022-04-22 03:32:48
 +
+

*Thread Reply:* I agree, but for now I'm more likely to be in the I didn't get it category, and not in the brilliant new idea category 🙂

+ +

My temporary goal is to go over the documentation and to write the gaps that confused me (and the solutions) and maybe publish that as an article for wider audience. So far I realized that: +• I don't get the naming convention - it became clearer that it's important with the Naming examples, but more info is needed +• I mis-interpret the namespaces. I was placing datasources and jobs in the same namespace which caused a lot of issues until I started using different ones. Not sure why... So now I'm interpreting namespaces=source as suggested by the naming convention +• JSON schema actually clarified things a lot, but that's not the most reader-friendly of resources, so surely there should be a better one +• I was questioning whether to move away from Marquez completely and go with DataHub, but for my scenario Marquez (with limitations outstanding) is still most suitable +• Marquez for some reason does not tolerate the datetimes if they're missing the 'T' delimiter in the ISO, which caused a lot of trial-and-error because the message is just "JSON parsing failed" +• Marquez doesn't give you (at least by default) meaningful OpenLineage parsing errors, so running examples against it is a very slow learning process

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Karatuğ Ozan BİRCAN + (karatugo@gmail.com) +
+
2022-04-21 10:20:55
 +
+

Hi everyone,

+ +

I'm running the Spark Listener on Databricks. It works fine for the event emit part for a basic Databricks SQL Create Table query. Nevertheless, it throws a NullPointerException exception after sending lineage successfully.

+ +

I tried to debug a bit. Looks like it's thrown at the line: +QueryExecution queryExecution = SQLExecution.getQueryExecution(executionId); +So, does this mean that the listener can't get the query exec from Spark SQL execution?

+ +

Please see the logs in the thread. Thanks.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Karatuğ Ozan BİRCAN + (karatugo@gmail.com) +
+
2022-04-21 10:21:33
 +
+

*Thread Reply:* Driver logs from Databricks:

+ +

```22/04/21 14:05:07 INFO EventEmitter: Lineage completed successfully: ResponseMessage(responseCode=200, body={}, error=null) {"eventType":"COMPLETE",[...], "schemaURL":"https://openlineage.io/spec/1-0-2/OpenLineage.json#/$defs/RunEvent"}

+ +

22/04/21 14:05:07 ERROR AsyncEventQueue: Listener OpenLineageSparkListener threw an exception +java.lang.NullPointerException + at io.openlineage.spark.agent.lifecycle.ContextFactory.createSparkSQLExecutionContext(ContextFactory.java:43) + at io.openlineage.spark.agent.OpenLineageSparkListener.lambda$getSparkSQLExecutionContext$8(OpenLineageSparkListener.java:221) + at java.util.HashMap.computeIfAbsent(HashMap.java:1127) + at java.util.Collections$SynchronizedMap.computeIfAbsent(Collections.java:2674) + at io.openlineage.spark.agent.OpenLineageSparkListener.getSparkSQLExecutionContext(OpenLineageSparkListener.java:220) + at io.openlineage.spark.agent.OpenLineageSparkListener.sparkSQLExecStart(OpenLineageSparkListener.java:143) + at io.openlineage.spark.agent.OpenLineageSparkListener.onOtherEvent(OpenLineageSparkListener.java:135) + at org.apache.spark.scheduler.SparkListenerBus.doPostEvent(SparkListenerBus.scala:102) + at org.apache.spark.scheduler.SparkListenerBus.doPostEvent$(SparkListenerBus.scala:28) + at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37) + at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37) + at org.apache.spark.util.ListenerBus.postToAll(ListenerBus.scala:119) + at org.apache.spark.util.ListenerBus.postToAll$(ListenerBus.scala:103) + at org.apache.spark.scheduler.AsyncEventQueue.super$postToAll(AsyncEventQueue.scala:105) + at org.apache.spark.scheduler.AsyncEventQueue.$anonfun$dispatch$1(AsyncEventQueue.scala:105) + at scala.runtime.java8.JFunction0$mcJ$sp.apply(JFunction0$mcJ$sp.java:23) + at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62) + at org.apache.spark.scheduler.AsyncEventQueue.org$apache$spark$scheduler$AsyncEventQueue$$dispatch(AsyncEventQueue.scala:100) + at org.apache.spark.scheduler.AsyncEventQueue$$anon$2.$anonfun$run$1(AsyncEventQueue.scala:96) + at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1588) + at org.apache.spark.scheduler.AsyncEventQueue$$anon$2.run(AsyncEventQueue.scala:96)```

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-04-21 11:32:37
 +
+

*Thread Reply:* @Karatuğ Ozan BİRCAN are you running on Spark 3.2? If yes, then new release should have fixed your problem: https://github.com/OpenLineage/OpenLineage/issues/609

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Karatuğ Ozan BİRCAN + (karatugo@gmail.com) +
+
2022-04-21 11:33:15
 +
+

*Thread Reply:* Spark 3.1.2 with Scala 2.12

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Karatuğ Ozan BİRCAN + (karatugo@gmail.com) +
+
2022-04-21 11:33:50
 +
+

*Thread Reply:* In fact, I couldn't make it work in Spark 3.2. But I'll test it again. Thanks for the info.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Vinith Krishnan US + (vinithk@nvidia.com) +
+
2022-05-20 16:15:47
 +
+

*Thread Reply:* Has this been resolved? +I am facing the same issue with spark 3.2.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ben + (ben@meridian.sh) +
+
2022-04-21 11:51:33
 +
+

Does anyone have thoughts on the difference between the sourceCode and sql job facets - and whether we’d expect to ever see both on a particular job?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2022-04-21 15:34:24
 +
+

*Thread Reply:* I don't think that the facets are particularly strongly defined, but I would expect that it could be possible to see both on a pythonOperator that's executing SQL queries, depending on how the extractor was written

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ben + (ben@meridian.sh) +
+
2022-04-21 15:34:45
 +
+

*Thread Reply:* ah sure, that makes sense

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Xiaoyong Zhu + (xiaoyzhu@outlook.com) +
+
2022-04-21 15:14:03
 +
+

Just get to know open lineage and it's really a great project! One question for the granularity on Spark + Openlineage - is it possible to track column level lineage (rather than the table lineage that's currently there)? Thanks!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-04-21 16:17:59
 +
+

*Thread Reply:* We're actively working on it - expect it in next OpenLineage release. https://github.com/OpenLineage/OpenLineage/pull/645

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/pull/645|#645 [Spark][Integration] column level lineage>
+
+ +
+ +
+
Labels
+ enhancement, integration/spark +
+ +
+
Milestone
+ <a href="https://github.com/OpenLineage/OpenLineage/milestone/4">0.8.0</a> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Xiaoyong Zhu + (xiaoyzhu@outlook.com) +
+
2022-04-21 16:24:16
 +
+

*Thread Reply:* nice -thanks!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Xiaoyong Zhu + (xiaoyzhu@outlook.com) +
+
2022-04-21 16:25:19
 +
+

*Thread Reply:* Assuming we don't need to do anything except using the next update? Or do you expect that we need to change quite a lot of configs?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-04-21 17:44:46
 +
+

*Thread Reply:* No, it should be automatic.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-04-24 14:37:33
 +
+

Hey, Team - We are starting to get requests for other, non Microsoft data sources (e.g. Teradata) for the Spark Integration. We (I) don't have a lot of bandwidth to fill every request but I DO want to help these people new to OpenLineage get started.

+ +

Has anyone on the team written up a blog post about extending open lineage or is this an area that we could collaborate on for the OpenLineage blog? Alternatively, is it a bad idea to write this down since the internals have changed a few times over the past six months?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mirko Raca + (racamirko@gmail.com) +
+
2022-04-25 03:52:20
 +
+

*Thread Reply:* Hey Will,

+ +

while I would not consider myself in the team, I'm dabbling in OL, hitting walls and learning as I go. If I don't have enough experience to contribute, I'd be happy to at least proof-read and point out things which are not clear from a novice perspective. Let me know!

+ + + +
+ 👍 Will Johnson +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-04-25 13:49:48
 +
+

*Thread Reply:* I'll hold you to that @Mirko Raca 😉

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-04-25 17:18:02
 +
+

*Thread Reply:* I will support! I’ve done a few recent presentations on the internals of OpenLineage that might also be useful - maybe some diagrams can be reused.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-04-25 17:56:44
 +
+

*Thread Reply:* Any chance you have links to those old presentations? Would be great to build off of an existing one and then update for some of the new naming conventions.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-04-25 18:00:26
 +
+

*Thread Reply:* the most recent one was an astronomer webinar

+ +

happy to share the slides with you if you want 👍 here’s a PDF:

+ +
+
+ + + + + + +
+ + +
+ 🙌 Will Johnson +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-04-25 18:00:44
 +
+

*Thread Reply:* the other ones have not been public, unfortunately 😕

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-04-25 18:02:24
 +
+

*Thread Reply:* architecture, object model, run lifecycle, naming conventions == the basics IMO

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-04-26 09:14:42
 +
+

*Thread Reply:* Thank you so much, Ross! This is a great base to work from.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-04-26 14:49:04
 +
+

Your periodical reminder that Github stars are one of those trivial things that make a significant difference for an OS project like ours. Have you starred us yet?

+ +
+
+ + + + + + +
+ + +
+ 👍 Ross Turk +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
raghanag + (raghanag@gmail.com) +
+
2022-04-26 15:02:10
 +
+

Hi All, I have a simple spark job from converting csv to parquet and I am using https://openlineage.io/integration/apache-spark/ to generate lineage events and posting to maquez but I see that both events (START & COMPLETE) have the same event except eventType, i thought we should see outputsarray in the complete event right?

+
+
openlineage.io
+ + + +
Apache Spark
+
+ OpenLineage can automatically track lineage of jobs and datasets across Spark jobs. +
+ + + + + + +
Original URL: https://openlineage.io/integration/apache-spark/
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-04-27 00:36:05
 +
+

*Thread Reply:* For a spark job like that, you'd have at least four events:

+ +
  1. START event - This represents the SparkSQLExecutionStart
  2. START event #2 - This represents a JobStart event
  3. COMPLET event - This represents a JobEnd event
  4. COMPLETE event #2 - This represents a SparkSQLExectionEnd event +For CSV to Parquet, you should be seeing inputs and outputs that match across each event. OpenLineage scans the logical plan and reports back the inputs / outputs / metadata across the different facets for each event BECAUSE each event might give you some different information.
    5. +
+ +

For example, the JobStart event might give you access to properties that weren't there before. The JobEnd event might give you information about how many rows were written.

+ +

Marquez / OpenLineage expects that you collect all of the resulting events and then aggregate the results.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
raghanag + (raghanag@gmail.com) +
+
2022-04-27 21:51:07
 +
+

*Thread Reply:* Hi @Will Johnson good evening. We are seeing an issue while using spark integaration and found that when we provide openlinegae.host property a value like <http://lineage.com/common/marquez> where my marquez api is running I see that the below line is modifying the host to become <http://lineage.com/api/v1/lineage> instead of <http://lineage.com/common/marquez/api/v1/lineage> which is causing the problem +https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/src/main/common/java/io/openlineage/spark/agent/EventEmitter.java#L49 +I see that it has been added 5 months ago and released it as part of 0.4.0, is there anyway that we can fix the line to be like below +this.lineageURI = + new URI( + hostURI.getScheme(), + hostURI.getAuthority(), + hostURI.getPath() + uriPath, + queryParams, + null);

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/src/main/common/java/io/openlineage/spark/agent/EventEmitter.java | EventEmitter.java>
+
+ <pre><code> new URI(hostURI.getScheme(), hostURI.getAuthority(), uriPath, queryParams, null); +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-04-28 14:31:42
 +
+

*Thread Reply:* Can you open up a Github issue for this? I had this same issue and so our implementation always has to feature the /api/v1/lineage. The host config is literally the host. You're specifying a host and path. I'd be happy to see greater flexibility with the api endpoint but the /v1/ is important to know which version of OpenLineage's specification you're communicating with.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Arturo + (ggrmos@gmail.com) +
+
2022-04-27 14:12:38
 +
+

Hi all, guys ... anyone have an example of a custom extractor with different source-destination, I'm trying to build an extractor from a custom operator like mysql_to_s3

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-04-27 15:10:24
 +
+

*Thread Reply:* @Michael Collado made one for a recent webinar:

+ +

https://gist.github.com/collado-mike/d1854958b7b1672f5a494933f80b8b58

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-04-27 15:11:38
 +
+

*Thread Reply:* it's not exactly for an operator that has source-destination, but it shows how to format lineage events for a few different kinds of datasets

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Arturo + (ggrmos@gmail.com) +
+
2022-04-27 15:51:32
 +
+

*Thread Reply:* Thanks! I'm going to take a look

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-04-27 23:04:18
 +
+

A release has been requested by @Howard Yoo and @Ross Turk pending the merging of PR 644. Are there any +1s?

+ + + +
+ 👍 Julien Le Dem, Maciej Obuchowski, Ross Turk, Conor Beverland +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-04-28 17:44:00
 +
+

*Thread Reply:* Thanks for your input. The release is authorized. Look for it tomorrow!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
raghanag + (raghanag@gmail.com) +
+
2022-04-28 14:29:13
 +
+

Hi All, We are seeing the below exception when we integrate the openlineage-spark into our spark job, can anyone share pointers +Exception uncaught: java.lang.NoSuchMethodError: com.fasterxml.jackson.databind.SerializationConfig.hasExplicitTimeZone()Z at openlineage.jackson.datatype.jsr310.ser.InstantSerializerBase.formatValue(InstantSerializerBase.java:144) at openlineage.jackson.datatype.jsr310.ser.InstantSerializerBase.serialize(InstantSerializerBase.java:103) at openlineage.jackson.datatype.jsr310.ser.ZonedDateTimeSerializer.serialize(ZonedDateTimeSerializer.java:79) at openlineage.jackson.datatype.jsr310.ser.ZonedDateTimeSerializer.serialize(ZonedDateTimeSerializer.java:13) at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:727) at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:719) at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:155) at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider._serialize(DefaultSerializerProvider.java:480) at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:319) at com.fasterxml.jackson.databind.ObjectMapper._configAndWriteValue(ObjectMapper.java:3906) at com.fasterxml.jackson.databind.ObjectMapper.writeValueAsString(ObjectMapper.java:3220) at io.openlineage.spark.agent.client.OpenLineageClient.executeAsync(OpenLineageClient.java:123) at io.openlineage.spark.agent.client.OpenLineageClient.executeSync(OpenLineageClient.java:85) at <a href="http://io.openlineage.spark.agent.client.OpenLineageClient.post">io.openlineage.spark.agent.client.OpenLineageClient.post</a>(OpenLineageClient.java:80) at <a href="http://io.openlineage.spark.agent.client.OpenLineageClient.post">io.openlineage.spark.agent.client.OpenLineageClient.post</a>(OpenLineageClient.java:75) at <a href="http://io.openlineage.spark.agent.client.OpenLineageClient.post">io.openlineage.spark.agent.client.OpenLineageClient.post</a>(OpenLineageClient.java:70) at io.openlineage.spark.agent.EventEmitter.emit(EventEmitter.java:67) at io.openlineage.spark.agent.lifecycle.SparkSQLExecutionContext.start(SparkSQLExecutionContext.java:69) at io.openlineage.spark.agent.OpenLineageSparkListener.lambda$sparkSQLExecStart$0(OpenLineageSparkListener.java:90) at java.util.Optional.ifPresent(Optional.java:159) at io.openlineage.spark.agent.OpenLineageSparkListener.sparkSQLExecStart(OpenLineageSparkListener.java:90) at io.openlineage.spark.agent.OpenLineageSparkListener.onOtherEvent(OpenLineageSparkListener.java:81) at org.apache.spark.scheduler.SparkListenerBus$class.doPostEvent(SparkListenerBus.scala:80) at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37) at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37) at org.apache.spark.util.ListenerBus$class.postToAll(ListenerBus.scala:91) at <a href="http://org.apache.spark.scheduler.AsyncEventQueue.org">org.apache.spark.scheduler.AsyncEventQueue.org</a>$apache$spark$scheduler$AsyncEventQueue$$super$postToAll(AsyncEventQueue.scala:92) at org.apache.spark.scheduler.AsyncEventQueue$$anonfun$org$apache$spark$scheduler$AsyncEventQueue$$dispatch$1.apply$mcJ$sp(AsyncEventQueue.scala:92) at org.apache.spark.scheduler.AsyncEventQueue$$anonfun$org$apache$spark$scheduler$AsyncEventQueue$$dispatch$1.apply(AsyncEventQueue.scala:87) at org.apache.spark.scheduler.AsyncEventQueue$$anonfun$org$apache$spark$scheduler$AsyncEventQueue$$dispatch$1.apply(AsyncEventQueue.scala:87) at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58) at <a href="http://org.apache.spark.scheduler.AsyncEventQueue.org">org.apache.spark.scheduler.AsyncEventQueue.org</a>$apache$spark$scheduler$AsyncEventQueue$$dispatch(AsyncEventQueue.scala:87) at org.apache.spark.scheduler.AsyncEventQueue$$anon$1$$anonfun$run$1.apply$mcV$sp(AsyncEventQueue.scala:83) at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1302) at org.apache.spark.scheduler.AsyncEventQueue$$anon$1.run(AsyncEventQueue.scala:82)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2022-04-28 14:41:10
 +
+

*Thread Reply:* What's the spark job that's running - this looks similar to an error that can happen when jobs have a very short lifecycle

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
raghanag + (raghanag@gmail.com) +
+
2022-04-28 14:47:27
 +
+

*Thread Reply:* nothing in spark job, its just a simple csv to parquet conversion file

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2022-04-28 14:48:50
 +
+

*Thread Reply:* ah yeah that's probably it - when the job is finished before the Openlineage integration can poll it for information this error is thrown. Since the job is very quick it creates a race condition

+ + + +
+ :gratitude_thank_you: raghanag +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
raghanag + (raghanag@gmail.com) +
+
2022-05-03 17:16:39
 +
+

*Thread Reply:* @John Thomas may i know how to solve this kind of issue?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2022-05-03 17:20:11
 +
+

*Thread Reply:* This is probably an issue with the integration - for now you can either open an issue, or see if you're still getting a subset of events and take it as is. I'm not sure what you could do on your end aside from adding a sleep call or similar

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
raghanag + (raghanag@gmail.com) +
+
2022-05-03 17:21:17
 +
+

*Thread Reply:* https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/src/main/common/java/io/openlineage/spark/agent/OpenLineageSparkListener.java#L151 you meant if we add a sleep in this method this will solve this

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/src/main/common/java/io/openlineage/spark/agent/OpenLineageSparkListener.java | OpenLineageSparkListener.java>
+
+ <pre><code> public void onJobEnd(SparkListenerJobEnd jobEnd) { +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2022-05-03 18:44:43
 +
+

*Thread Reply:* oh no I meant making sure your jobs don't close too quickly

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
raghanag + (raghanag@gmail.com) +
+
2022-05-06 00:14:15
 +
+

*Thread Reply:* Hi @John Thomas we figured out the error that it is indeed causing with conflicted versions and with shadowJar and shading, we are not seeing it anymore.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-04-29 18:40:41
 +
+

@channel The latest release (0.8.1) of OpenLineage is now available, featuring a new TaskInstance listener API for Airflow 2.3+, an HTTP client in the openlineage-java library for emitting run events, support for HiveTableRelation as an input source in the Spark integration, a new SQL parser used by multiple integrations, and bug fixes. For more info, visit https://github.com/OpenLineage/OpenLineage/releases/tag/0.8.1

+ + + +
+ 🚀 Willy Lulciuc, John Thomas, Minkyu Park, Ross Turk, Marco Diaz, Conor Beverland, Kevin Mellott, Howard Yoo, Peter Hicks, Maciej Obuchowski, Mario Measic +
+ +
+ 🙌 Francis McGregor-Macdonald, Ross Turk, Marco Diaz, Peter Hicks +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2022-04-29 18:41:37
 +
+

*Thread Reply:* Amazing work on the new sql parser @Maciej Obuchowski 💯 :firstplacemedal:

+ + + +
+ 👍 Ross Turk, Howard Yoo, Peter Hicks +
+ +
+ 🙌 Ross Turk, Howard Yoo, Peter Hicks, Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-04-30 07:54:48
 +
+

The May meeting of the TSC will be postponed because most of the TSC will be attending the Astronomer Spring Summit the week of May 9th. Details to follow along with a new meeting day/time for the meeting going forward (thanks to all who responded to the poll!).

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Hubert Dulay + (hubert.dulay@gmail.com) +
+
2022-05-01 09:25:23
 +
+

Are there examples of using openlineage with streaming data pipelines? Thanks

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mirko Raca + (racamirko@gmail.com) +
+
2022-05-03 04:12:09
 +
+

*Thread Reply:* Hi @Hubert Dulay,

+ +

while I'm not an expert, I can offer the following: +• Marquez has had the but what I got here - that API is not encouraged +• I personally don't find the run->job metaphor to work nicely with streaming transformation, but I'm using that in my current setup (until someone points me in a better direction 😉 ) +• I register each change of the stream processing as a new "run", which ends immediately - so duration information is lost, but current set of parameters is recorded. It's not pretty, I know. +Maybe stream processing is a scenario to be re-evaluated in OL meetings, or at least clarified?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Hubert Dulay + (hubert.dulay@gmail.com) +
+
2022-05-03 21:19:06
 +
+

*Thread Reply:* Thanks for the details

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kostikey Mustakas + (kostikey.mustakas@gmail.com) +
+
2022-05-02 09:32:23
 +
+

Hey OL! My company is in the process of migrating off of Palantir and into Databricks/Azure. There are a couple of business units not wanting to budge due to the built-in data lineage and code reference features Palantir has. I am tasked with researching an alternative data lineage solution and I quickly came across OL. I love what I have read and seen demos of so far and want to do a POC for my org of its capabilities. I was able to set up the Marquez server on a VM and get it talking to Databricks. I also have the iniit script installed on the cluster and I can see from the log4j logs it’s communicating fine (I think). However, I am embarrassed to admit I can’t figure out how the instrumentation works for the databricks notebooks. I ran a simple notebook that loads data, runs a simple transform, and saves the output somewhere but I don’t see any entries in my namespace I configured. I am sure I missed something very obvious somewhere, but are there examples of how to get a simple example into Marquez from databricks? Thanks so much for any guidance you can give!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2022-05-02 13:26:52
 +
+

*Thread Reply:* Hi Kostikey - this blog has an example with Spark and jupyter, which might be a good place to start!

+
+
openlineage.io
+ + + +
Tracing Data Lineage with OpenLineage and Apache Spark
+
+ Spark ushered in a brand new age of data democratization... and left us with a mess of hidden dependencies, stale datasets, and failed jobs. +
+ + + + + + +
Original URL: https://openlineage.io/blog/openlineage-spark/
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kostikey Mustakas + (kostikey.mustakas@gmail.com) +
+
2022-05-02 14:58:29
 +
+

*Thread Reply:* Hi @John Thomas, thanks for the reply. I think I am close but my cluster is unable to talk to the marquez server. After looking at log4j I see the following rows:

+ +

22/05/02 18:43:39 INFO SparkContext: Registered listener io.openlineage.spark.agent.OpenLineageSparkListener +22/05/02 18:43:40 INFO EventEmitter: Init OpenLineageContext: Args: ArgumentParser(host=<http://135.170.226.91:8400>, version=v1, namespace=gus-namespace, jobName=default, parentRunId=null, apiKey=Optional.empty, urlParams=Optional[{}]) URI: <http://135.170.226.91:8400/api/v1/lineage>? +22/05/02 18:46:21 ERROR EventEmitter: Could not emit lineage [responseCode=0]: {"eventType":"START","eventTime":"2022-05-02T18:44:08.36Z","run":{"runId":"91fd4e13-52ac-4175-8956-c06d7dee97fc","facets":{"spark_version":{"_producer":"<https://github.com/OpenLineage/OpenLineage/tree/0.8.1/integration/spark>","_schemaURL":"<https://openlineage.io/spec/1-0-2/OpenLineage.json#/$defs/RunFacet>","spark-version":"3.2.1","openlineage_spark_version":"0.8.1"},"spark.logicalPlan":{"_producer":"<https://github.com/OpenLineage/OpenLineage/tree/0.8.1/integration/spark>","_schemaURL":"<https://openlineage.io/spec/1-0-2/OpenLineage.json#/$defs/RunFacet>","plan":[{"class":"org.apache.spark.sql.catalyst.plans.logical.ShowNamespaces","num-children":1,"namespace":0,"output":[[{"class":"org.apache.spark.sql.catalyst.expressions.AttributeReference","num-children":0,"name":"databaseName","dataType":"string","nullable":false,"metadata":{},"exprId":{"product-class":"org.apache.spark.sql.catalyst.expressions.ExprId","id":4,"jvmId":"eaa0543b_5e04_4f5b_844b_0e4598f019a7"},"qualifier":[]}]]},{"class":"org.apache.spark.sql.catalyst.analysis.ResolvedNamespace","num_children":0,"catalog":null,"namespace":[]}]},"spark_unknown":{"_producer":"<https://github.com/OpenLineage/OpenLineage/tree/0.8.1/integration/spark>","_schemaURL":"<https://openlineage.io/spec/1-0-2/OpenLineage.json#/$defs/RunFacet>","output":{"description":"Unable to serialize logical plan due to: Infinite recursion (StackOverflowError) ... + OpenLineageHttpException(code=0, message=java.lang.RuntimeException: java.util.concurrent.ExecutionException: openlineage.hc.client5.http.ConnectTimeoutException: Connect to <http://135.170.226.91:8400> [/135.170.226.91] failed: Connection timed out, details=java.util.concurrent.CompletionException: java.lang.RuntimeException: java.util.concurrent.ExecutionException: openlineage.hc.client5.http.ConnectTimeoutException: Connect to <http://135.170.226.91:8400> [/135.170.226.91] failed: Connection timed out) + at io.openlineage.spark.agent.EventEmitter.emit(EventEmitter.java:68) + at io.openlineage.spark.agent.lifecycle.SparkSQLExecutionContext.start(SparkSQLExecutionContext.java:69) + at io.openlineage.spark.agent.OpenLineageSparkListener.lambda$sparkSQLExecStart$0(OpenLineageSparkListener.java:90) + at java.util.Optional.ifPresent(Optional.java:159) + at io.openlineage.spark.agent.OpenLineageSparkListener.sparkSQLExecStart(OpenLineageSparkListener.java:90) + at io.openlineage.spark.agent.OpenLineageSparkListener.onOtherEvent(OpenLineageSparkListener.java:81) + at org.apache.spark.scheduler.SparkListenerBus.doPostEvent(SparkListenerBus.scala:102) + at org.apache.spark.scheduler.SparkListenerBus.doPostEvent$(SparkListenerBus.scala:28) + at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37) + at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37) + at org.apache.spark.util.ListenerBus.postToAll(ListenerBus.scala:119) + at org.apache.spark.util.ListenerBus.postToAll$(ListenerBus.scala:103) + at org.apache.spark.scheduler.AsyncEventQueue.super$postToAll(AsyncEventQueue.scala:105) + at org.apache.spark.scheduler.AsyncEventQueue.$anonfun$dispatch$1(AsyncEventQueue.scala:105) + at scala.runtime.java8.JFunction0$mcJ$sp.apply(JFunction0$mcJ$sp.java:23) + at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62) + at <a href="http://org.apache.spark.scheduler.AsyncEventQueue.org">org.apache.spark.scheduler.AsyncEventQueue.org</a>$apache$spark$scheduler$AsyncEventQueue$$dispatch(AsyncEventQueue.scala:100) + at org.apache.spark.scheduler.AsyncEventQueue$$anon$2.$anonfun$run$1(AsyncEventQueue.scala:96) + at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1612) + at org.apache.spark.scheduler.AsyncEventQueue$$anon$2.run(AsyncEventQueue.scala:96) +the connection timeout is surprising because I can connect just fine using the example curl code from the same cluster:

+ +

%sh +curl -X POST <http://135.170.226.91:8400/api/v1/lineage> \ + -H 'Content-Type: application/json' \ + -d '{ + "eventType": "START", + "eventTime": "2020-12-28T19:52:00.001+10:00", + "run": { + "runId": "d46e465b-d358-4d32-83d4-df660ff614dd" + }, + "job": { + "namespace": "gus2~-namespace", + "name": "my-job" + }, + "inputs": [{ + "namespace": "gus2-namespace", + "name": "gus-input" + }], + "producer": "<https://github.com/OpenLineage/OpenLineage/blob/v1-0-0/client>" + }' +Spark config: +spark.openlineage.host <http://135.170.226.91:8400> +spark.openlineage.version v1 +spark.openlineage.namespace gus-namespace +Not sure what is going on, the EventEmitter init log looks like it's right but clearly something is off. Thanks so much for the help

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2022-05-02 15:03:40
 +
+

*Thread Reply:* hmmm, interesting - if it's easy could you spin both up locally and check that it's just a communication issue? It helps with diagnosis

+ +

It might also be a firewall issue, but your cURL should preclude that

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kostikey Mustakas + (kostikey.mustakas@gmail.com) +
+
2022-05-02 15:05:38
 +
+

*Thread Reply:* Since it's Databricks I was having a hard time figuring out how to try locally. Other than just using plain 'ol spark on my laptop and a localhost Marquez...

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2022-05-02 15:07:13
 +
+

*Thread Reply:* hmm, that could be an interesting test to see if it's a databricks issue - the databricks integration is pretty much the same as the spark integration, just with a little bit of a wrapper and the init script

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kostikey Mustakas + (kostikey.mustakas@gmail.com) +
+
2022-05-02 15:08:44
 +
+

*Thread Reply:* yeah, i was going to try that but it just didnt seem like helpful troubleshooting for exactly that reason... but i may just do that anyways just so i can see something working 🙂 (morale booster)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2022-05-02 15:09:22
 +
+

*Thread Reply:* oh totally! Network issues are a huge pain in the ass, and if you're still seeing issues locally with spark/mz then we'll know a lot more than we do now 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kostikey Mustakas + (kostikey.mustakas@gmail.com) +
+
2022-05-02 15:11:19
 +
+

*Thread Reply:* sounds good, i will give it a go!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-05-02 15:16:16
 +
+

*Thread Reply:* @Kostikey Mustakas - I think spark.openlineage.version should be equal to 1 not v1.

+ +

In addition, is http://135.170.226.91:8400 accessible to Databricks? Could you try doing a %sh command inside of a databricks notebook and see if you can ping that IP address (https://linux.die.net/man/8/ping)?

+ +

For your Databricks cluster did you VNET inject it into an existing VNET? If it's in an existing VNET, you should confirm that the VM running marquez can access it. If it's in a non-VNET injected VNET, you probably need to redeploy to a VNET that has that VM or has connectivity to that VM.

+
+
linux.die.net
+ + + +
ping(8) - Linux man page
+
+ ping uses the ICMP protocol's mandatory ECHO_REQUEST datagram to elicit an ICMP ECHO_RESPONSE from a host or gateway. ECHO_REQUEST datagrams (''pings'') ... +
+ + + + + + +
Original URL: https://linux.die.net/man/8/ping
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kostikey Mustakas + (kostikey.mustakas@gmail.com) +
+
2022-05-02 15:19:22
 +
+

*Thread Reply:* Ya, know i meant to ask about that. Docs say 1 like you mention: https://github.com/OpenLineage/OpenLineage/tree/main/integration/spark/databricks. I second guessed from this thread https://openlineage.slack.com/archives/C01CK9T7HKR/p1638848249159700.

+
+ + +
+ + + } + + Dinakar Sundar + (https://openlineage.slack.com/team/U02MQ8E22HF) +
+ + + +
+
+ <p>@John Thomas we in Condenast currently exploring the features of open lineage to integrate to databricks , <a href="https://github.com/OpenLineage/OpenLineage/tree/main/integration/spark/databricks">https://github.com/OpenLineage/OpenLineage/tree/main/integration/spark/databricks</a> , spark configuration not working ,</p> +
+ + + + + + +
Original URL: https://openlineage.slack.com/archives/C01CK9T7HKR/p1638848249159700
+ + +
+ + Thread in Slack Conversation +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kostikey Mustakas + (kostikey.mustakas@gmail.com) +
+
2022-05-02 15:23:42
 +
+

*Thread Reply:* @Will Johnson, ping fails... this is surprising as the curl command mentioned above works fine.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julius Rentergent + (julius.rentergent@thetradedesk.com) +
+
2022-05-02 15:37:00
 +
+

*Thread Reply:* I’m also trying to set up Databricks according to Running Marquez on AWS. Right now I’m stuck on the database part rather than the Marquez part — I can’t connect my EKS cluster to the RDS database which I described in more detail on the Marquez slack.

+ +

@Kostikey Mustakas Sorry for the distraction, but I’m curious how you have set up your networking to make the API requests work with Databricks. +Good luck with your issue!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kostikey Mustakas + (kostikey.mustakas@gmail.com) +
+
2022-05-02 15:47:17
 +
+

*Thread Reply:* @Julius Rentergent We are using Azure and leverage Private Endpoints to connect resources in separate subscriptions. There is a Bastion proxy in place that we can map http traffic through and I have a Load Balancer Inbound NAT rule I setup that maps one our whitelisted port ranges (8400) to 5000.

+ + + +
+ :gratitude_thank_you: Julius Rentergent +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kostikey Mustakas + (kostikey.mustakas@gmail.com) +
+
2022-05-02 20:15:01
 +
+

*Thread Reply:* @Will Johnson a little progress maybe... I created a private endpoint and updated dns to point to it. Now I get a 404 Not Found error instead of a timeout

+ + + +
+ 🙌 Will Johnson +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kostikey Mustakas + (kostikey.mustakas@gmail.com) +
+
2022-05-02 20:16:41
 +
+

*Thread Reply:* 22/05/03 00:09:24 ERROR EventEmitter: Could not emit lineage [responseCode=404]: {"eventType":"START","eventTime":"2022-05-03T00:09:22.498Z","run":{"runId":"f41575a0-e59d-4cbc-a401-9b52d2b020e0","facets":{"spark_version":{"_producer":"<https://github.com/OpenLineage/OpenLineage/tree/0.8.1/integration/spark>","_schemaURL":"<https://openlineage.io/spec/1-0-2/OpenLineage.json#/$defs/RunFacet>","spark-version":"3.2.1","openlineage_spark_version":"0.8.1"},"spark.logicalPlan":{"_producer":"<https://github.com/OpenLineage/OpenLineage/tree/0.8.1/integration/spark>","_schemaURL":"<https://openlineage.io/spec/1-0-2/OpenLineage.json#/$defs/RunFacet>","plan":[{"class":"org.apache.spark.sql.catalyst.plans.logical.ShowNamespaces","num-children":1,"namespace":0,"output":[[{"class":"org.apache.spark.sql.catalyst.expressions.AttributeReference","num-children":0,"name":"databaseName","dataType":"string","nullable":false,"metadata":{},"exprId":{"product-class":"org.apache.spark.sql.catalyst.expressions.ExprId","id":4,"jvmId":"aad3656d_8903_4db3_84f0_fe6d773d71c3"},"qualifier":[]}]]},{"class":"org.apache.spark.sql.catalyst.analysis.ResolvedNamespace","num_children":0,"catalog":null,"namespace":[]}]},"spark_unknown":{"_producer":"<https://github.com/OpenLineage/OpenLineage/tree/0.8.1/integration/spark>","_schemaURL":"<https://openlineage.io/spec/1-0-2/OpenLineage.json#/$defs/RunFacet>","output":{"description":"Unable to serialize logical plan due to: Infinite recursion (StackOverflowError) (through reference chain: org.apache.spark.sql.catalyst.expressions.AttributeReference[\"preCanonicalized\"] .... +OpenLineageHttpException(code=null, message={"code":404,"message":"HTTP 404 Not Found"}, details=null) + at io.openlineage.spark.agent.EventEmitter.emit(EventEmitter.java:68)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julius Rentergent + (julius.rentergent@thetradedesk.com) +
+
2022-05-27 00:03:30
 +
+

*Thread Reply:* Following up on this as I encounter the same issue with the Openlineage Databricks integration. This issue seems quite malicious as it crashes the Spark Context and requires a restart.

+ +

I have marquez running on AWS EKS; I’m using Openlineage 0.8.2 on Databricks 10.4 (Spark 3.2.1) and my Spark config looks like this: +spark.openlineage.host <https://internal-xxxxxxxxxxxxxxxxxxxxxxx-xxxxxxxxxxx.us-east-1.elb.amazonaws.com> +spark.openlineage.namespace default +spark.openlineage.version v1 &lt;- also tried "1" +I can run some simple read and write commands and successfully find the log4j events highlighted in the docs: +INFO SparkContext; +INFO OpenLineageContext; +INFO AsyncEventQueue for each time I run the cell +After doing this a few times I get The spark context has stopped and the driver is restarting. Your notebook will be automatically reattached. +stderr shows a bunch of things. log4j shows the same as for Kostikey: ERROR EventEmitter: [...] Unable to serialize logical plan due to: Infinite recursion (StackOverflowError)

+ +

I have one more piece of information which I can’t make much sense of, but hopefully someone else can; if I include the port in the host, I can very reliably crash the Spark Context on the first attempt. So: +<https://internal-xxxxxxxxxxxxxxxxxxxxxxx-xxxxxxxxxxx.us-east-1.elb.amazonaws.com> &lt;- crashes after a couple of attempts, sometimes it takes me a while to reproduce it while repeatedly reading/writing the same datasets +<https://internal-xxxxxxxxxxxxxxxxxxxxxxx-xxxxxxxxxxx.us-east-1.elb.amazonaws.com:80> &lt;- crashes on first try +Any insights would be greatly appreciated! 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julius Rentergent + (julius.rentergent@thetradedesk.com) +
+
2022-05-27 00:22:27
 +
+

*Thread Reply:* I tried two more things: +• curl works, ping fails, just like in the previous report +• Databricks allows providing spark configs without quotes, whereas quotes are generally required for Spark. So I added the quotes to the host name, but now I’m getting: ERROR OpenLineageSparkListener: Unable to parse open lineage endpoint. Lineage events will not be collected

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Martin Fiser + (fisa@keboola.com) +
+
2022-05-27 14:00:38
 +
+

*Thread Reply:* @Kostikey Mustakas May I ask what is the reason for migration from Palantir? Sorry for this off-topic question!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-05-30 05:46:27
 +
+

*Thread Reply:* @Julius Rentergent created issue on project github: https://github.com/OpenLineage/OpenLineage/issues/795

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/issues/795|#795 spark: port included in host causes crash of listener>
+
+ <p>Issue reported on Slack: <a href="https://openlineage.slack.com/archives/C01CK9T7HKR/p1653624210694359?thread_ts=1651498343.959749&amp;cid=C01CK9T7HKR|https://openlineage.slack.com/archives/C01CK9T7HKR/p1653624210694359?thread_ts=1651498343.959749&amp;cid=C01CK9T7HKR">https://openlineage.slack.com/archives/C01CK9T7HKR/p1653624210694359?thread<em>ts=1651498343.959749&amp;cid=C01CK9T7HKR|https://openlineage.slack.com/archives/C01CK9T7HKR/p1653624210694359?thread</em>ts=1651498343.959749&amp;cid=C01CK9T7HKR</a></p> + +<p>Following up on this as I encounter the same issue with the Openlineage Databricks integration. This issue seems quite malicious as it crashes the Spark Context and requires a restart. <br /> +I have marquez running on AWS EKS; I’m using Openlineage 0.8.2 on Databricks 10.4 (Spark 3.2.1) and my Spark config looks like this:</p> + +<pre><code>spark.openlineage.host &lt;https://internal-xxxxxxxxxxxxxxxxxxxxxxx-xxxxxxxxxxx.us-east-1.elb.amazonaws.com/&gt; +spark.openlineage.namespace default +spark.openlineage.version v1 &amp;lt;- also tried "1" + +</code></pre> + +<p>I can run some simple read and write commands and successfully find the log4j events <a href="https://github.com/OpenLineage/OpenLineage/tree/main/integration/spark/databricks">highlighted in the docs</a>:</p> + +<pre><code>INFO SparkContext; +INFO OpenLineageContext; +INFO AsyncEventQueue for each time I run the cell + +</code></pre> + +<p>After doing this a few times I get The spark context has stopped and the driver is restarting. Your notebook will be automatically reattached. <br /> +<a href="https://pastebin.com/9NbEjnGL">stderr shows a bunch of things</a>. log4j shows the same as for Kostikey: ERROR EventEmitter: [...] Unable to serialize logical plan due to: Infinite recursion (StackOverflowError) <br /> +I have one more piece of information which I can’t make much sense of, but hopefully someone else can; if I include the port in the host, I can very reliably crash the Spark Context on the first attempt. So:</p> + +<pre><code>&lt;https://internal-xxxxxxxxxxxxxxxxxxxxxxx-xxxxxxxxxxx.us-east-1.elb.amazonaws.com/&gt; &amp;lt;- crashes after a couple of attempts, sometimes it takes me a while to reproduce it while repeatedly reading/writing the same datasets +&lt;https://internal-xxxxxxxxxxxxxxxxxxxxxxx-xxxxxxxxxxx.us-east-1.elb.amazonaws.com:80/&gt; &amp;lt;- crashes on first try + +</code></pre> + +<p>Any insights would be greatly appreciated!</p> +
+ +
+
Labels
+ bug, integration/spark +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julius Rentergent + (julius.rentergent@thetradedesk.com) +
+
2022-06-01 11:15:26
 +
+

*Thread Reply:* Thank you @Maciej Obuchowski. +Just to clarify, the Spark Context crashes with and without port; it’s just that adding the port causes it to crash more quickly (on the 1st attempt).

+ +

I will run some more experiments when I have time, and add the results to the ticket.

+ +

Edit - added to issue:

+ +

I ran some more experiments, this time with a fake host and on OpenLineage 0.9.0, and was not able to reproduce the issue with regards to the port; instead, the new experiments show that Spark 3.2 looks to be involved.

+ +

On Spark 3.2.1 / Databricks 10.4 LTS: Using (fake) host http://ac7aca38330144df9.amazonaws.com:5000 crashes when the first notebook cell is evaluated with The spark context has stopped and the driver is restarting. + The same occurs when the port is removed.

+ +

On Spark 3.1.2 / Databricks 9.1 LTS: Using (fake) host http://ac7aca38330144df9.amazonaws.com:5000 does not impede the cluster but, reasonably, produces for each lineage event ERROR EventEmitter: Could not emit lineage w/ exception io.openlineage.client.OpenLineageClientException: java.net.UnknownHostException + The same occurs when the port is removed.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-05-02 14:52:09
 +
+

@channel The poll results are in, and the new day/time for the monthly TSC meeting is each second Thursday at 10 am PT. The next meeting will take place on Thursday, 5/19, at 10 am PT, due to a conflict with the Astronomer Spring Summit. Future meetings will take place on the second Thursday of each month. Calendar updates will be forthcoming. Thanks!

+ + + +
+ 🙌 Willy Lulciuc, Mynor Choc +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-05-02 15:09:42
 +
+

*Thread Reply:* @Michael Robinson - just to be sure, is the 5/19 meeting at 10 AM PT as well?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-05-02 15:14:11
 +
+

*Thread Reply:* Yes, and I’ll update the msg for others. Thank you

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-05-02 15:16:25
 +
+

*Thread Reply:* Thank you!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sandeep Bhat + (bhatsandeep424@gmail.com) +
+
2022-05-02 21:45:39
 +
+

Hii Team, as i saw marquez is building lineage by java code, from seed command, what should i do to connect with mysql (our database) with credentials and building a lineage for our data?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-05-03 12:40:55
 +
+

@here How do we clear old jobs, datasets and namespaces from Marquez?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mirko Raca + (racamirko@gmail.com) +
+
2022-05-04 07:04:48
 +
+

*Thread Reply:* It seems we can't for now. This was the same question I had last week:

+ +

https://github.com/MarquezProject/marquez/issues/1736

+
+ + + + +
<https://github.com/MarquezProject/marquez/issues/1736|#1736 Ability to DELETE jobs and datasets>
+
+ <p>When jobs or datasets are deleted (or renamed) we need a way to tell Marquez that they don't exist anymore so that they don't clutter the lineage view. <br /> +It would make sense to have <code>DELETE</code> support on <code>/api/v1/namespaces/{ns}/datasets/{name}</code> and <code>/api/v1/namespaces/{ns}/jobs/{name}</code> resources. <br /> +This is a soft delete in the internal marquez model. It creates a new <code>deleted</code> version. The resources will not show up in the "current" lineage view in <code>GET /api/v1/lineage</code></p> +
+ +
+
Comments
+ 1 +
+ + + + + + + +
+ + <a href="https://github.com/MarquezProject/marquez">MarquezProject/marquez</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-05-04 10:56:35
 +
+

*Thread Reply:* Seems that it's really popular request 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tyler Farris + (tyler@kickstand.work) +
+
2022-05-03 13:43:56
 +
+

Hello, +I'm sending lineage events to astrocloud.datakin DB with the Marquez API. The event is sent- but the metadata for inputs and outputs isn't coming through. Below is an example of the event I'm sending. Not sure if this is the place for this question. Cross-posting to Marquez Slack. +{ + "eventTime": "2022-05-03T17:20:04.151087+00:00", + "run": { + "runId": "2dfc6dcd4011d2a1c3dc1e5861127e5b" + }, + "job": { + "namespace": "from-airflow", + "name": "Postgres_1_to_Snowflake_2.extract" + }, + "producer": "<https://github.com/OpenLineage/OpenLineage/blob/v1-0-0/client>", + "inputs": [ + { + "name": "Postgres_1_to_Snowflake_2.extract", + "namespace": "from-airflow" + } + ] +} +Thanks.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tyler Farris + (tyler@kickstand.work) +
+
2022-05-04 11:28:48
 +
+

*Thread Reply:* @Mirko Raca pointed out that I was missing eventType.

+ +

Mirko Raca : +"From a quick glance - you're missing "eventType": "START", attribute. It's also worth noting that metadata typically shows up after the second event (type COMPLETE)"

+ +

thanks again.

+ + + +
+ 👍 Mirko Raca +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sandeep Bhat + (bhatsandeep424@gmail.com) +
+
2022-05-06 05:01:34
 +
+

Hii Team, could anyone tell me, to view lineage in marquez do we have to write metadata as a code, or does marquez has a feature to scan the sql code and build a lineage automatically?please clarify my doubt regarding this.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Juan Carlos Fernández Rodríguez + (jcfernandez@keedio.com) +
+
2022-05-06 05:26:16
 +
+

*Thread Reply:* As far as I understand, OpenLineage has tools to extract metadata from sources. Depend on your source, you could find an integration, if it doesn't exists you should write your own integration (and collaborate with the project)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-05-06 12:59:06
 +
+

*Thread Reply:* @Sandeep Bhat take a look at https://openlineage.io/integration - there is some info there on the different integrations that can be used to automatically pull metadata.

+
+
openlineage.io
+ + + +
Integrations
+
+ Data lineage is the foundation for a new generation of powerful, context-aware data tools and best practices. OpenLineage enables consistent collection of lineage metadata, creating a deeper understanding of how data is produced and used. +
+ + + + + + +
Original URL: https://openlineage.io/integration
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-05-06 13:00:39
 +
+

*Thread Reply:* The Airflow integration, in particular, uses a SQL parser to determine input/output tables (in cases where the data store can't be queried for that info)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jorik + (jorik@scivis.net) +
+
2022-05-12 05:13:01
 +
+

Hi all. We are looking at using OpenLineage for capturing some lineage in our custom processing system. I think we got the lineage events understood, but we have often datasets that get appended, or get overwritten by an operation. Is there anything in openlineage that would facilitate making this distinction? (ie. if a set gets overwritten we would be interested in the lineage events from the last overwrite, if it gets appended we would like to have all of these in the display)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mirko Raca + (racamirko@gmail.com) +
+
2022-05-12 05:48:43
 +
+

*Thread Reply:* To my understanding - datasets model the structure, not the content. So, as long as your table doesn't change number of columns, it's the same thing.

+ +

The catch-all would be to create a Dataset facet which would record the distinction between append/overwrite per run. But, while this is supported by the standard, Marquez does not handle custom facets at the moment (I'll happily be corrected).

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jorik + (jorik@scivis.net) +
+
2022-05-12 06:05:36
 +
+

*Thread Reply:* Thanks, that makes sense. We're looking for a way to get the lineage of table contents. We may have to opt for new names on overwrite, or indeed extend a facet to flag these.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jorik + (jorik@scivis.net) +
+
2022-05-12 06:06:44
 +
+

*Thread Reply:* Use case is compliancy, where we need to show how a certain delivered data product (at a given point in time) was constructed. We have all our transforms/transfers as code, but there are a few parts where datasets get recreated in the process after fixes have been made, and I wouldn't want to bother the auditors with those stray paths

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-05-12 06:12:09
 +
+

*Thread Reply:* We have LifecycleStateChangeDataset facet that captures this information. It's currently emitted when using Spark integration

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-05-12 06:13:25
 +
+

*Thread Reply:* > But, while this is supported by the standard, Marquez does not handle custom facets at the moment (I'll happily be corrected). +It displays this information when it exists

+ + + +
+ 🙌 Mirko Raca +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jorik + (jorik@scivis.net) +
+
2022-05-12 06:13:29
 +
+

*Thread Reply:* Oh that looks perfect! I completely missed that, thanks!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Marco Diaz + (mdiaz@roblox.com) +
+
2022-05-12 15:46:04
 +
+

Are there any examples on how to use this facet ColumnLineageDatasetFacet.json?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-05-13 05:19:47
 +
+

*Thread Reply:* Work with Spark is not yet fully merged

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
raghanag + (raghanag@gmail.com) +
+
2022-05-12 17:49:23
 +
+

Hi All, I am trying to see where we can provide owner details when using openlineage-spark configuration, i see only namespace and other config parameters but not the owner. Can we add owner configuration also as part of openlineage-spark like spark.openlineage.owner? Owner will be used to even filter namespaces when showing the jobs or namespaces in Marquez UI.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-05-13 19:07:04
 +
+

@channel The next OpenLineage Technical Steering Committee meeting is next Thursday, 5/19, at 10 am PT! Going forward, meetings will take place on the second Thursday of each month at 10 am PT. +Join us on Zoom: +https://astronomer.zoom.us/j/87156607114?pwd=a3B0K210dnRaQmdkaFdGMytBREZEQT09 +All are welcome! +Agenda: +• releases 0.7.1 & 0.8.1 +• column-level lineage +• open lineage +For notes and the agenda visit the wiki: https://tinyurl.com/openlineagetsc

+ + + +
+ 🙌 Maciej Obuchowski, Ross Turk +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Yannick Libert + (yan@ioly.fr) +
+
2022-05-16 11:02:23
 +
+

Hi all, we are considering using OL to send lineage events from various jobs and places in our company. Since there will be multiple producers, we would like to use Kafka as our main hub for communication. One of our sources will be Airflow (more particularly MWAA, ie airflow in its 2.2.2 version). Is there a way to configure the Airflow lineage backend to send event to kafka instead of Marquez directly? So far, from what I've seen in the docs and in here, the only way would be to create a simple proxy to stream the http events to Kafka. Is it still the case?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-05-16 11:31:17
 +
+

*Thread Reply:* I think you can either use proxy backend: https://github.com/OpenLineage/OpenLineage/tree/main/proxy

+ +

or configure OL client to send data to kafka: +https://github.com/OpenLineage/OpenLineage/tree/main/client/python#kafka

+ + + +
+ 👍 Yannick Libert +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Yannick Libert + (yan@ioly.fr) +
+
2022-05-16 12:15:59
 +
+

*Thread Reply:* Thank you very much for the useful pointers. The proxy solutions could indeed work in our case but it implies creating another service in front of Kafka, and thus and another layer of complexity to the architecture. If there is another more "native" way of streaming event directly from the Airflow backend that'll be great to know

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-05-16 12:37:10
 +
+

*Thread Reply:* The second link 😉

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Yannick Libert + (yan@ioly.fr) +
+
2022-05-17 03:46:03
 +
+

*Thread Reply:* Sure, we already implemented the python client for jobs outside airflow and it works great 🙂 +You are saying that there is a way to use this python client in conjonction with the MWAA lineage backend to relay the job events that come with the airflow integration (without including it in the DAGs)? +Our strategy is to use both the airflow backend to collect automatic lineage events without modifying any existing DAGs, and the in-code implementation to allow our data engineers to send their own events if they want to. +The second option works perfectly but the first one is where we struggle a bit, especially with MWAA.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-05-17 05:24:30
 +
+

*Thread Reply:* If you can mount file to MWAA, then yes - it should work with config file option: https://github.com/OpenLineage/OpenLineage/tree/main/client/python#config-file

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Yannick Libert + (yan@ioly.fr) +
+
2022-05-17 05:40:45
 +
+

*Thread Reply:* Brilliant! I'm going to test that. Thank you Maciej!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-05-17 15:20:58
 +
+

A release has been requested. Are there any +1s? Three from committers will authorize. Thanks.

+ + + +
+ ➕ Maciej Obuchowski, Ross Turk, Willy Lulciuc, Michael Collado +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-05-18 10:33:03
 +
+

The OpenLineage TSC meeting is tomorrow at 10am PT! https://openlineage.slack.com/archives/C01CK9T7HKR/p1652483224119229

+
+ + +
+ + + } + + Michael Robinson + (https://openlineage.slack.com/team/U02LXF3HUN7) +
+ + + +
+
+ <p>@channel The next OpenLineage Technical Steering Committee meeting is next <strong>Thursday, 5/19, at 10 am PT</strong>! Going forward, meetings will take place on the second Thursday of each month at 10 am PT. +Join us on Zoom: +<a href="https://astronomer.zoom.us/j/87156607114?pwd=a3B0K210dnRaQmdkaFdGMytBREZEQT09">https://astronomer.zoom.us/j/87156607114?pwd=a3B0K210dnRaQmdkaFdGMytBREZEQT09</a> +All are welcome! +Agenda: +• releases 0.7.1 &amp; 0.8.1 +• column-level lineage +• open lineage +For notes and the agenda visit the wiki: <a href="https://tinyurl.com/openlineagetsc">https://tinyurl.com/openlineagetsc</a></p> +
+ + + + + + +
Original URL: https://openlineage.slack.com/archives/C01CK9T7HKR/p1652483224119229
+ + +
+ + Slack Conversation +
+ + +
+ + + +
+ 🙌 Willy Lulciuc +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tyler Farris + (tyler@kickstand.work) +
+
2022-05-18 16:23:56
 +
+

Hey all, +Do custom extractors work with the taskflow api?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2022-05-18 16:34:25
 +
+

*Thread Reply:* Hey Tyler - A custom extractor just needs to be able to assemble the runEvents and send the information out to the lineage backends.

+ +

If the things you're sending/receiving with TaskFlow are accessible in terms of metadata in the environment the DAG is running in, then you should be able to make one that would work!

+ +

This Webinar goes over creating custom extractors for reference.

+ +

Does that answer your question?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-05-18 16:41:16
 +
+

*Thread Reply:* Taskflow internally is just PythonOperator. If you'd write extractor that assumes something more than just it being PythonOperator then you'd probably make it work 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tyler Farris + (tyler@kickstand.work) +
+
2022-05-18 17:15:52
 +
+

*Thread Reply:* Thanks @John Thomas @Maciej Obuchowski, Your answers both make sense. I just keep running into this error in my logs: +[2022-05-18, 20:52:34 UTC] {__init__.py:97} WARNING - Unable to find an extractor. task_type=_PythonDecoratedOperator airflow_dag_id=Postgres_1_to_Snowflake_1_v3 task_id=Postgres_1 airflow_run_id=scheduled__2022-05-18T20:51:34.334045+00:00 +The picture is my custom extractor, it's not doing anything currently as this is just a test.

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tyler Farris + (tyler@kickstand.work) +
+
2022-05-18 17:16:05
 +
+

*Thread Reply:* thanks again for the help yall

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2022-05-18 17:16:34
 +
+

*Thread Reply:* did you set the environment variable with the path to your extractor?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tyler Farris + (tyler@kickstand.work) +
+
2022-05-18 17:16:46
 +
+

*Thread Reply:*

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tyler Farris + (tyler@kickstand.work) +
+
2022-05-18 17:17:13
 +
+

*Thread Reply:* i believe thats correct @John Thomas

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tyler Farris + (tyler@kickstand.work) +
+
2022-05-18 17:18:35
 +
+

*Thread Reply:* and the versions im using: +Astronomer Runtime 5.0.0 based on Airflow 2.3.0+astro.1

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2022-05-18 17:25:58
 +
+

*Thread Reply:* this might not be the problem, but you should have only one of extract and extract_on_complete - which one are you meaning to use?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tyler Farris + (tyler@kickstand.work) +
+
2022-05-18 17:32:26
 +
+

*Thread Reply:* ahh thanks John, as of right now extract_on_complete.

+ +

This is a similar setup as Michael had in the video.

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john@datakin.com) +
+
2022-05-18 17:33:31
 +
+

*Thread Reply:* if it's still not working I'm not really sure at this point - that's about what I had when I spun up my own custom extractor

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-05-18 17:39:44
 +
+

*Thread Reply:* is there anything in logs regarding extractors?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tyler Farris + (tyler@kickstand.work) +
+
2022-05-18 17:40:36
 +
+

*Thread Reply:* just this: +[2022-05-18, 21:36:59 UTC] {__init__.py:97} WARNING - Unable to find an extractor. task_type=_PythonDecoratedOperator airflow_dag_id=competitive_oss_projects_git_to_snowflake task_id=Transform_git_logs_to_S3 airflow_run_id=scheduled__2022-05-18T21:35:57.694690+00:00

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tyler Farris + (tyler@kickstand.work) +
+
2022-05-18 17:41:11
 +
+

*Thread Reply:* @John Thomas Thanks, I appreciate your help.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-05-19 06:01:52
 +
+

*Thread Reply:* No Failed to import messages?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tyler Farris + (tyler@kickstand.work) +
+
2022-05-19 11:26:34
 +
+

*Thread Reply: @Maciej Obuchowski None that I can see. Here is the full log: +``` Failed to verify remote log exists s3:///dag_id=Postgres_1_to_Snowflake_1_v3/run_id=scheduled2022-05-19T15:23:49.248097+00:00/task_id=Postgres_1/attempt=1.log. +Please provide a bucket_name instead of "s3:///dag_id=Postgres_1_to_Snowflake_1_v3/run_id=scheduled2022-05-19T15:23:49.248097+00:00/task_id=Postgres_1/attempt=1.log" + Falling back to local log +* Reading local file: /usr/local/airflow/logs/dagid=Postgres1toSnowflake1v3/runid=scheduled2022-05-19T15:23:49.248097+00:00/taskid=Postgres1/attempt=1.log +[2022-05-19, 15:24:50 UTC] {taskinstance.py:1158} INFO - Dependencies all met for <TaskInstance: Postgres1toSnowflake1v3.Postgres1 scheduled2022-05-19T15:23:49.248097+00:00 [queued]> +[2022-05-19, 15:24:50 UTC] {taskinstance.py:1158} INFO - Dependencies all met for <TaskInstance: Postgres1toSnowflake1v3.Postgres1 scheduled_2022-05-19T15:23:49.248097+00:00 [queued]>

+ +

[2022-05-19, 15:24:50 UTC] {taskinstance.py:1355} INFO -

+ +

[2022-05-19, 15:24:50 UTC] {taskinstance.py:1356} INFO - Starting attempt 1 of 1

+ +

[2022-05-19, 15:24:50 UTC] {taskinstance.py:1357} INFO -

+ +

[2022-05-19, 15:24:50 UTC] {taskinstance.py:1376} INFO - Executing <Task(PythonDecoratedOperator): Postgres1> on 2022-05-19 15:23:49.248097+00:00 +[2022-05-19, 15:24:50 UTC] {standardtaskrunner.py:52} INFO - Started process 3957 to run task +[2022-05-19, 15:24:50 UTC] {standardtaskrunner.py:79} INFO - Running: ['airflow', 'tasks', 'run', 'Postgres1toSnowflake1v3', 'Postgres1', 'scheduled2022-05-19T15:23:49.248097+00:00', '--job-id', '96473', '--raw', '--subdir', 'DAGSFOLDER/pgtosnow.py', '--cfg-path', '/tmp/tmp9n7u3i4t', '--error-file', '/tmp/tmp9a55v9b'] +[2022-05-19, 15:24:50 UTC] {standardtaskrunner.py:80} INFO - Job 96473: Subtask Postgres1 +[2022-05-19, 15:24:50 UTC] {loggingmixin.py:115} WARNING - /usr/local/lib/python3.9/site-packages/airflow/configuration.py:470 DeprecationWarning: The sqlalchemyconn option in [core] has been moved to the sqlalchemyconn option in [database] - the old setting has been used, but please update your config. +[2022-05-19, 15:24:50 UTC] {taskcommand.py:369} INFO - Running <TaskInstance: Postgres1toSnowflake1v3.Postgres1 scheduled2022-05-19T15:23:49.248097+00:00 [running]> on host 056ca0b6c7f5 +[2022-05-19, 15:24:50 UTC] {taskinstance.py:1568} INFO - Exporting the following env vars: +AIRFLOWCTXDAGOWNER=airflow +AIRFLOWCTXDAGID=Postgres1toSnowflake1v3 +AIRFLOWCTXTASKID=Postgres1 +AIRFLOWCTXEXECUTIONDATE=20220519T15:23:49.248097+00:00 +AIRFLOWCTXTRYNUMBER=1 +AIRFLOWCTXDAGRUNID=scheduled2022-05-19T15:23:49.248097+00:00 +[2022-05-19, 15:24:50 UTC] {loggingmixin.py:115} WARNING - /usr/local/lib/python3.9/site-packages/airflow/utils/context.py:202 AirflowContextDeprecationWarning: Accessing 'executiondate' from the template is deprecated and will be removed in a future version. Please use 'dataintervalstart' or 'logicaldate' instead. +[2022-05-19, 15:24:50 UTC] {loggingmixin.py:115} WARNING - /usr/local/lib/python3.9/site-packages/airflow/utils/context.py:202 AirflowContextDeprecationWarning: Accessing 'nextds' from the template is deprecated and will be removed in a future version. Please use '{{ dataintervalend | ds }}' instead. +[2022-05-19, 15:24:50 UTC] {loggingmixin.py:115} WARNING - /usr/local/lib/python3.9/site-packages/airflow/utils/context.py:202 AirflowContextDeprecationWarning: Accessing 'nextdsnodash' from the template is deprecated and will be removed in a future version. Please use '{{ dataintervalend | dsnodash }}' instead. +[2022-05-19, 15:24:50 UTC] {loggingmixin.py:115} WARNING - /usr/local/lib/python3.9/site-packages/airflow/utils/context.py:202 AirflowContextDeprecationWarning: Accessing 'nextexecutiondate' from the template is deprecated and will be removed in a future version. Please use 'dataintervalend' instead. +[2022-05-19, 15:24:50 UTC] {loggingmixin.py:115} WARNING - /usr/local/lib/python3.9/site-packages/airflow/utils/context.py:202 AirflowContextDeprecationWarning: Accessing 'prevds' from the template is deprecated and will be removed in a future version. +[2022-05-19, 15:24:50 UTC] {loggingmixin.py:115} WARNING - /usr/local/lib/python3.9/site-packages/airflow/utils/context.py:202 AirflowContextDeprecationWarning: Accessing 'prevdsnodash' from the template is deprecated and will be removed in a future version. +[2022-05-19, 15:24:50 UTC] {loggingmixin.py:115} WARNING - /usr/local/lib/python3.9/site-packages/airflow/utils/context.py:202 AirflowContextDeprecationWarning: Accessing 'prevexecutiondate' from the template is deprecated and will be removed in a future version. +[2022-05-19, 15:24:50 UTC] {loggingmixin.py:115} WARNING - /usr/local/lib/python3.9/site-packages/airflow/utils/context.py:202 AirflowContextDeprecationWarning: Accessing 'prevexecutiondatesuccess' from the template is deprecated and will be removed in a future version. Please use 'prevdataintervalstartsuccess' instead. +[2022-05-19, 15:24:50 UTC] {loggingmixin.py:115} WARNING - /usr/local/lib/python3.9/site-packages/airflow/utils/context.py:202 AirflowContextDeprecationWarning: Accessing 'tomorrowds' from the template is deprecated and will be removed in a future version. +[2022-05-19, 15:24:50 UTC] {loggingmixin.py:115} WARNING - /usr/local/lib/python3.9/site-packages/airflow/utils/context.py:202 AirflowContextDeprecationWarning: Accessing 'tomorrowdsnodash' from the template is deprecated and will be removed in a future version. +[2022-05-19, 15:24:50 UTC] {loggingmixin.py:115} WARNING - /usr/local/lib/python3.9/site-packages/airflow/utils/context.py:202 AirflowContextDeprecationWarning: Accessing 'yesterdayds' from the template is deprecated and will be removed in a future version. +[2022-05-19, 15:24:50 UTC] {loggingmixin.py:115} WARNING - /usr/local/lib/python3.9/site-packages/airflow/utils/context.py:202 AirflowContextDeprecationWarning: Accessing 'yesterdaydsnodash' from the template is deprecated and will be removed in a future version. +[2022-05-19, 15:24:50 UTC] {python.py:173} INFO - Done. Returned value was: extract +[2022-05-19, 15:24:50 UTC] {loggingmixin.py:115} WARNING - /usr/local/lib/python3.9/site-packages/airflow/models/baseoperator.py:1369 DeprecationWarning: Passing 'executiondate' to 'TaskInstance.xcompush()' is deprecated. +[2022-05-19, 15:24:50 UTC] {init.py:97} WARNING - Unable to find an extractor. tasktype=PythonDecoratedOperator airflowdagid=Postgres1toSnowflake1v3 taskid=Postgres1 airflowrunid=scheduled2022-05-19T15:23:49.248097+00:00 +[2022-05-19, 15:24:50 UTC] {client.py:74} INFO - Constructing openlineage client to send events to https://api.astro-livemaps.datakin.com/ +[2022-05-19, 15:24:50 UTC] {taskinstance.py:1394} INFO - Marking task as SUCCESS. dagid=Postgres1toSnowflake1v3, taskid=Postgres1, executiondate=20220519T152349, startdate=20220519T152450, enddate=20220519T152450 +[2022-05-19, 15:24:50 UTC] {localtaskjob.py:156} INFO - Task exited with return code 0 +[2022-05-19, 15:24:50 UTC] {localtask_job.py:273} INFO - 1 downstream tasks scheduled from follow-on schedule check```

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Josh Owens + (Josh@kickstand.work) +
+
2022-05-19 16:57:38
 +
+

*Thread Reply:* @Maciej Obuchowski is our ENV var wrong maybe? Do we need to mention the file to import somewhere else that we may have missed?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-05-20 10:26:01
 +
+

*Thread Reply:* @Josh Owens one thing I can think of is that you might have older openlineage integration version, as OPENLINEAGE_EXTRACTORS variable was added very recently: https://github.com/OpenLineage/OpenLineage/pull/694

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tyler Farris + (tyler@kickstand.work) +
+
2022-05-20 11:58:28
 +
+

*Thread Reply:* @Maciej Obuchowski, that was it! For some reason, my requirements.txt wasn't pulling the latest version of openlineage-airflow. Working now with 0.8.2

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-05-20 11:59:01
 +
+

*Thread Reply:* 🙌

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Raymond + (michael.raymond@cervest.earth) +
+
2022-05-19 05:32:06
 +
+

Hi 👋, I'm looking at OpenLineage as a solution for fine-grained data lineage tracking. Could I clarify a couple of points?

+ +

Where does one specify the version of an input dataset in the RunEvent? In the Marquez seed data I can see that it's recorded, but I'm not sure where it goes from looking at the OpenLineage schema. Or does it just assume the last version?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-05-19 05:59:59
 +
+

*Thread Reply:* Currently, it assumes latest version. +There's an effort with DatasetVersionDatasetFacet to be able to specify it manually - or extract this information from cases like Iceberg or Delta Lake tables.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Raymond + (michael.raymond@cervest.earth) +
+
2022-05-19 06:14:59
 +
+

*Thread Reply:* Ah ok. Is it Marquez assuming the latest version when it records the OpenLineage event?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-05-19 06:18:20
 +
+

*Thread Reply:* yes

+ + + +
+ ✅ Michael Raymond +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Raymond + (michael.raymond@cervest.earth) +
+
2022-05-19 06:54:40
 +
+

*Thread Reply:* Thanks, that's very helpful 👍

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Howard Yoo + (howardyoo@gmail.com) +
+
2022-05-19 15:23:33
 +
+

Hi all, +I was testing https://github.com/MarquezProject/marquez/tree/main/examples/airflow#step-21-create-dag-counter, and the following error was observed in my airflow env:

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Howard Yoo + (howardyoo@gmail.com) +
+
2022-05-19 15:23:52
 +
+

Anybody know why this is happening? Any comments would be welcomed.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tyler Farris + (tyler@kickstand.work) +
+
2022-05-19 15:27:35
 +
+

*Thread Reply:* @Howard Yoo What version of airflow?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Howard Yoo + (howardyoo@gmail.com) +
+
2022-05-19 15:27:51
 +
+

*Thread Reply:* it's 2.3

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Howard Yoo + (howardyoo@gmail.com) +
+
2022-05-19 15:28:42
 +
+

*Thread Reply:* (sorry, it's 2.4)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tyler Farris + (tyler@kickstand.work) +
+
2022-05-19 15:29:28
 +
+

*Thread Reply:* https://github.com/OpenLineage/OpenLineage/tree/main/integration/airflow Id refer to the docs again.

+ +

"Airflow 2.3+ +Integration automatically registers itself for Airflow 2.3 if it's installed on Airflow worker's python. This means you don't have to do anything besides configuring it, which is described in Configuration section."

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Howard Yoo + (howardyoo@gmail.com) +
+
2022-05-19 15:29:53
 +
+

*Thread Reply:* Right, configuring I don't see any issues

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tyler Farris + (tyler@kickstand.work) +
+
2022-05-19 15:30:56
 +
+

*Thread Reply:* so you dont need:

+ +

from openlineage.airflow import DAG

+ +

in your dag files

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Howard Yoo + (howardyoo@gmail.com) +
+
2022-05-19 15:31:41
 +
+

*Thread Reply:* Okay... that makes sense then

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tyler Farris + (tyler@kickstand.work) +
+
2022-05-19 15:32:47
 +
+

*Thread Reply:* so if you need to import DAG it would just be: +from airflow import DAG

+ + + +
+ 👍 Howard Yoo +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Howard Yoo + (howardyoo@gmail.com) +
+
2022-05-19 15:56:19
 +
+

*Thread Reply:* Thanks!

+ + + +
+ 👍 Tyler Farris +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-05-19 17:13:02
 +
+

@channel OpenLineage 0.8.2 is now available! The project now supports credentialing from the Airflow Secrets Backend and for the Azure Databricks Credential Passthrough, detection of datasets wrapped by ExternalRDDs, bug fixes, and more. For the details, see: https://github.com/OpenLineage/OpenLineage/releases/tag/0.8.2

+ + + +
+ 🎉 Marco Diaz, Howard Yoo, Willy Lulciuc, Michael Collado, Ross Turk, Francis McGregor-Macdonald, Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
xiang chen + (cdmikechen@hotmail.com) +
+
2022-05-19 22:18:42
 +
+

Hi~ everyone Is there possible to let openlineage to support camel pipeline?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-05-20 10:23:55
 +
+

*Thread Reply:* What changes do you mean by letting openlineage support? +Or, do you mean, to write Apache Camel integration?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
xiang chen + (cdmikechen@hotmail.com) +
+
2022-05-22 19:54:17
 +
+

*Thread Reply:* @Maciej Obuchowski Yes, let openlineage work as same as airflow

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
xiang chen + (cdmikechen@hotmail.com) +
+
2022-05-22 19:56:47
 +
+

*Thread Reply:* I think this is a very valuable thing. I wish openlineage can support some commonly used pipeline tools, and try to abstract out some general interfaces so that users can expand by themselves

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-05-23 05:20:30
 +
+

*Thread Reply:* For Python, we have OL client, common libraries (well, at least beginning of them) and SQL parser

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-05-23 05:20:44
 +
+

*Thread Reply:* As we support more systems, the general libraries will grow as well.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Conor Beverland + (conorbev@gmail.com) +
+
2022-05-20 13:50:53
 +
+

I see a change in the metadata collected from Airflow jobs which I think was introduced with the combination of Airflow 2.3/OpenLineage 0.8.1. There's an airflow_version facet that contains an operator attribute.

+ +

Previously that attribute had values such as: airflow.providers.postgres.operators.postgres.PostgresOperator but I now see that for the very same task the operator is now tracked as: airflow.models.taskinstance.TaskInstance

+ +

( fwiw there's also a taskInfo attribute in there containing a json string which itself has a operator that is still set to PostgresOperator )

+ +

Is this an already known issue?

+ + + +
+ 👀 Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2022-05-20 20:23:15
 +
+

*Thread Reply:* This looks like a bug. we are probably not looking at the right instance in the TaskInstanceListener

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Conor Beverland + (conorbev@gmail.com) +
+
2022-05-21 14:17:19
 +
+

*Thread Reply:* @Howard Yoo I filed: https://github.com/OpenLineage/OpenLineage/issues/767 for this

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/issues/767|#767 [BUG][INTEGRATION][Airflow] Incorrect Operator tracked in Airflow 2.3>
+
+ <p>I see a change in the metadata collected from Airflow jobs which I think was introduced with the combination of Airflow 2.3/OpenLineage 0.8.1. There's an <code>airflow_version</code> facet that contains an <code>operator</code> attribute that is added by the integration.</p> + +<p>In Airflow 2.2 for a <code>PostgresOperator</code> that attribute would have had a value: <code>airflow.providers.postgres.operators.postgres.PostgresOperator</code> but now in Airflow 2.3 that operator is tracked as: <code>airflow.models.taskinstance.TaskInstance</code> <br /> +( fwiw there's also a <code>taskInfo</code> attribute in there containing a json string which itself has a <code>operator</code> that is still set to <code>PostgresOperator</code> )</p> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-05-20 21:42:46
 +
+

Would anyone happen to have a link to the Technical Steering Committee meeting recordings?

+ +

I have quite a few people interested in seeing the overview of column lineage that Pawel provided during the Technical Steering Committee meeting on Thursday May 19th.

+ +

The wiki does not include a link to the recordings: https://wiki.lfaidata.foundation/display/OpenLineage/Monthly+TSC+meeting

+ +

Are the recordings made public? Thank you for any links and guidance!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2022-05-20 21:55:09
 +
+

That would be @Michael Robinson Yes the recordings are made public.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-05-20 22:05:27
 +
+

@Will Johnson I’ll put this on the https://wiki.lfaidata.foundation/display/OpenLineage/Monthly+TSC+meeting|wiki soon, but here is the link to the recording: https://astronomer.zoom.us/rec/share/xUBW-n6G4u1WS89tCSXStx8BMl99rCfCC6jGdXLnkN6gMGn5G-_BC7pxHKKeELhG.0JFl88isqb64xX-3 +PW: 1VJ=K5&X

+ + + +
+ 🙌 Will Johnson +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-05-21 09:42:21
 +
+

*Thread Reply:* Thank you so much, Michael!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tyler Farris + (tyler@kickstand.work) +
+
2022-05-23 15:00:10
 +
+

Is there documentation/examples around creating custom facets?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-05-24 06:41:11
 +
+

*Thread Reply:* In Python or Java?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-05-24 06:44:32
 +
+

*Thread Reply:* In python just inherit BaseFacet and add _get_schema static method that would point to some place where you have your json schema of a facet. For example our DbtVersionRunFacet

+ +

In Java you can take a look at Spark's custom facets.

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/integration/common/openlineage/common/provider/dbt.py | dbt.py>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/269107690da704ee19e3b4a0362c3fe8da155eaf/integration/spark/src/main/common/java/io/openlineage/spark/agent/facets/TableProviderFacet.java | TableProviderFacet.java>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tyler Farris + (tyler@kickstand.work) +
+
2022-05-24 16:40:00
 +
+

*Thread Reply:* Thanks, @Maciej Obuchowski, I was asking in regards to Python, sorry I should have clarified.

+ +

I'm not sure what the disconnect is, but the facets aren't showing up in the inputs and outputs. The Lineage event is sent successfully to my astrocloud.

+ +

below is the facet and extractor, any help is appreciated. Thanks!

+ +

```import logging +from openlineage.airflow.extractors.base import BaseExtractor, TaskMetadata +from openlineage.client.run import InputDataset, OutputDataset +from typing import List, Optional +from openlineage.client.facet import BaseFacet +import attr

+ +

log = logging.getLogger(name)

+ +

@attr.s +class ManualLineageFacet(BaseFacet): + database: Optional[str] = attr.ib(default=None) + cluster: Optional[str] = attr.ib(default=None) + connectionUrl: Optional[str] = attr.ib(default=None) + target: Optional[str] = attr.ib(default=None) + source: Optional[str] = attr.ib(default=None) + _producer: str = attr.ib(init=False) + _schemaURL: str = attr.ib(init=False)

+ +
@staticmethod
+def _get_schema() -&gt; str:
+    return {
+        "$schema": "<http://json-schema.org/schema#>",
+        "$defs": {
+            "ManualLineageFacet": {
+                "allOf": [
+                    {
+                        "type": "object",
+                        "properties": {
+                            "database": {
+                                "type": "string",
+                                "example": "Snowflake",
+                            },
+                            "cluster": {
+                                "type": "string",
+                                "example": "us-west-2",
+                            },
+                            "connectionUrl": {
+                                "type": "string",
+                                "example": "<http://snowflake>",
+                            },
+                            "target": {
+                                "type": "string",
+                                "example": "Postgres",
+                            },
+                            "source": {
+                                "type": "string",
+                                "example": "Stripe",
+                            },
+                            "description": {
+                                "type": "string",
+                                "example": "Description of inlet/outlet",
+                            },
+                            "_producer": {
+                                "type": "string",
+                            },
+                            "_schemaURL": {
+                                "type": "string",
+                            },
+                        },
+                    },
+                ],
+                "type": "object",
+            }
+        },
+    }
+
+ +

class ManualLineageExtractor(BaseExtractor): + @classmethod + def getoperatorclassnames(cls) -> List[str]: + return ["PythonOperator", "_PythonDecoratedOperator"]

+ +
def extract_on_complete(self, task_instance) -&gt; Optional[TaskMetadata]:
+
+    return TaskMetadata(
+        f"{task_instance.dag_run.dag_id}.{task_instance.task_id}",
+        inputs=[
+            InputDataset(
+                namespace="default",
+                name=self.operator.get_inlet_defs()[0]["name"],
+                inputFacets=ManualLineageFacet(
+                    database=self.operator.get_inlet_defs()[0]["database"],
+                    cluster=self.operator.get_inlet_defs()[0]["cluster"],
+                    connectionUrl=self.operator.get_inlet_defs()[0][
+                        "connectionUrl"
+                    ],
+                    target=self.operator.get_inlet_defs()[0]["target"],
+                    source=self.operator.get_inlet_defs()[0]["source"],
+                ),
+            )
+            if self.operator.get_inlet_defs()
+            else {},
+        ],
+        outputs=[
+            OutputDataset(
+                namespace="default",
+                name=self.operator.get_outlet_defs()[0]["name"],
+                outputFacets=ManualLineageFacet(
+                    database=self.operator.get_outlet_defs()[0]["database"],
+                    cluster=self.operator.get_outlet_defs()[0]["cluster"],
+                    connectionUrl=self.operator.get_outlet_defs()[0][
+                        "connectionUrl"
+                    ],
+                    target=self.operator.get_outlet_defs()[0]["target"],
+                    source=self.operator.get_outlet_defs()[0]["source"],
+                ),
+            )
+            if self.operator.get_outlet_defs()
+            else {},
+        ],
+        job_facets={},
+        run_facets={},
+    )
+
+def extract(self) -&gt; Optional[TaskMetadata]:
+    pass```
+
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-05-25 09:21:02
 +
+

*Thread Reply:* _get_schema should return address to the schema hosted somewhere else - afaik sending object field where server expects string field might cause some problems

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-05-25 09:21:59
 +
+

*Thread Reply:* can you register ManualLineageFacet as facets not as inputFacets or outputFacets?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tyler Farris + (tyler@kickstand.work) +
+
2022-05-25 13:15:30
 +
+

*Thread Reply:* Thanks for the advice @Maciej Obuchowski, I was able to get it working! +Also great talk today at the airflow summit.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-05-25 13:25:17
 +
+

*Thread Reply:* Thanks 🙇

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Bruno González + (brugms2@gmail.com) +
+
2022-05-24 06:26:25
 +
+

Hey guys! I'm pretty new with OL but would like to start using it for a combination of data lineage in Airflow + data quality metrics collection. I was wondering if that was possible, but Ross clarified that in the deeper dive webinar from some weeks ago (great one by the way!).

+ +

I'm referencing this comment from Julien to see if you have any updates or more examples apart from the one from great expectations. We have some custom operators and would like to push lineage and data quality metrics to Marquez using custom extractors. Any reference will be highly appreciated. Thanks in advance!

+
+ + +
+ + + } + + Julien Le Dem + (https://openlineage.slack.com/team/U01DCLP0GU9) +
+ + + +
+
+ <p>I have updated the DataQuality metrics proposal and the corresponding PR: <a href="https://github.com/OpenLineage/OpenLineage/issues/101">https://github.com/OpenLineage/OpenLineage/issues/101</a> +<a href="https://github.com/OpenLineage/OpenLineage/pull/115">https://github.com/OpenLineage/OpenLineage/pull/115</a></p> +
+ + + + + + + +
+ + Slack Conversation +
+ + +
+
+
YouTube
+ +
+ + + } + + Astronomer + (https://www.youtube.com/c/Astronomer) +
+ + + +
OpenLineage and Airflow: A Deeper Dive
+
+ +
+ + + + + + + + +
Original URL: https://www.youtube.com/watch?v=LRr-ja8_Wjs
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-05-24 06:35:05
 +
+

*Thread Reply:* We're also getting data quality from dbt if you're running dbt test or dbt build +https://github.com/OpenLineage/OpenLineage/blob/main/integration/common/openlineage/common/provider/dbt.py#L399

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/integration/common/openlineage/common/provider/dbt.py | dbt.py>
+
+ <pre><code> def parse_test( +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-05-24 06:37:15
 +
+

*Thread Reply:* Generally, you'd need to construct DataQualityAssertionsDatasetFacet and/or DataQualityMetricsInputDatasetFacet and attach it to tested dataset

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/2604990e5387d1b8fb73a61664e35e702f25eed5/client/python/openlineage/client/facet.py | facet.py>
+
+ <pre><code>class DataQualityAssertionsDatasetFacet(BaseFacet): +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Bruno González + (brugms2@gmail.com) +
+
2022-05-24 13:23:34
 +
+

*Thread Reply:* Thanks @Maciej Obuchowski!!!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Howard Yoo + (howardyoo@gmail.com) +
+
2022-05-24 16:55:08
 +
+

Hi all, https://github.com/OpenLineage/OpenLineage/tree/main/integration/airflow#development <-- does this still work? I did follow the instructions, but running pytest failed with error messages like +________________________________________________ ERROR collecting tests/extractors/test_bigquery_extractor.py ________________________________________________ +ImportError while importing test module '/Users/howardyoo/git/OpenLineage/integration/airflow/tests/extractors/test_bigquery_extractor.py'. +Hint: make sure your test modules/packages have valid Python names. +Traceback: +openlineage/airflow/utils.py:251: in import_from_string + module = importlib.import_module(module_path) +/opt/homebrew/Caskroom/miniconda/base/envs/airflow/lib/python3.9/importlib/__init__.py:127: in import_module + return _bootstrap._gcd_import(name[level:], package, level) +&lt;frozen importlib._bootstrap&gt;:1030: in _gcd_import + ??? +&lt;frozen importlib._bootstrap&gt;:1007: in _find_and_load + ??? +&lt;frozen importlib._bootstrap&gt;:986: in _find_and_load_unlocked + ??? +&lt;frozen importlib._bootstrap&gt;:680: in _load_unlocked + ??? +&lt;frozen importlib._bootstrap_external&gt;:850: in exec_module + ??? +&lt;frozen importlib._bootstrap&gt;:228: in _call_with_frames_removed + ??? +../../../airflow.master/airflow/providers/google/cloud/operators/bigquery.py:39: in &lt;module&gt; + from airflow.providers.google.cloud.hooks.bigquery import BigQueryHook, BigQueryJob +../../../airflow.master/airflow/providers/google/cloud/hooks/bigquery.py:46: in &lt;module&gt; + from googleapiclient.discovery import Resource, build +E ModuleNotFoundError: No module named 'googleapiclient'

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Howard Yoo + (howardyoo@gmail.com) +
+
2022-05-24 16:55:09
 +
+

...

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Howard Yoo + (howardyoo@gmail.com) +
+
2022-05-24 16:55:54
 +
+

looks like just running the pytest wouldn't be able to run all the tests - as some of these dag tests seems to be requiring connectivities to google's big query, databases, etc..

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mardaunt + (miostat@yandex.ru) +
+
2022-05-25 16:32:08
 +
+

👋 Hi everyone! +I didn't find this in the documentation. +Can open lineage show me which source columns the final DataFrame column came from? (Spark)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-05-25 16:59:47
 +
+

*Thread Reply:* We're working on this feature - should be in the next release from OpenLineage side

+ + + +
+ 🙌 Mardaunt +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mardaunt + (miostat@yandex.ru) +
+
2022-05-25 17:06:12
 +
+

*Thread Reply:* Thanks! I will keep an eye on updates.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Martin Fiser + (fisa@keboola.com) +
+
2022-05-25 21:08:39
 +
+

Hi all, showcase time:

+ +

We have implemented a native OpenLineage endpoint and metadata writer in our Keboola all-in-one data platform. +The reason was that for more complex data pipeline scenarios it is beneficial to display the lineage in more detail. Additionally, we hope that OpenLineage as a standard will catch up and open up the ability to push lineage data into other data governance tools than Marquez. +The implementation started as an internal POC of tweaking our metadata into OpenLineage /lineage format and resulted into a native API endpoint and later on an app within Keboola platform ecosystem - feeding platform job metadata in a regular cadence. +We furthermore use a namespace for each keboola project so users can observe the data through their whole data mesh setup (multi-project architecture). +Please reach me out if you have any questions!

+ +
+
+ + + + + + +
+
+
+ + + + + + +
+
+
+ + + + + + +
+
+
+ + + + + + +
+ + +
+ 🙌 Maciej Obuchowski, Michael Robinson +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-05-26 06:05:33
 +
+

*Thread Reply:* Looks great! Thanks for sharing!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Gopi Krishnan Rajbahadur + (gopikrishnanrajbahadur@gmail.com) +
+
2022-05-26 10:13:26
 +
+

Hi OpenLineage team,

+ +

I am Gopi Krishnan Rajbahadur, one of the core members of OpenDatalogy project (a project that we are currently trying to sandbox as a part of LF-AI). Our OpenDatalogy project focuses on providing a process that allows users of publicly available datasets (e.g., CIFAR-10) to ensure license compliance. In addition, we also aim to provide a public repo that documents the final rights and obligations associated with common publicly available datasets, so that users of these datasets can use them compliantly in their AI models and software.

+ +

One of the key aspects of conducting dataset license compliance analysis involves tracking the lineage and provenance of the dataset (as we highlight in this paper here: https://arxiv.org/abs/2111.02374). We think that in this regard, our projects (i.e., OpenLineage and OpenDatalogy) could work together to use the existing OpenLineage standard and also collaborate to adopt/modify/enhance and use OpenLineage to track and document the lineage of a publicly available dataset. On that note, we are also working with the SPDX community to make the lineage and provenance of a dataset be tracked as a part of the SPDX BOM that is in the works for representing AI software (AI SBOM).

+ +

We think our projects could mutually benefit from collaborating with each other. Our project's Github could be found here: https://github.com/OpenDataology/OpenDataology. Any feedback that you have about our project would be greatly appreciated. Also, as we are trying to sandbox our project, if you could also show us your support we would greatly appreciate it!

+ +

Look forward to hearing back from you

+ +

Sincerely, +Gopi

+
+
arXiv.org
+ + + +
Can I use this publicly available dataset to build commercial AI...
+
+ Publicly available datasets are one of the key drivers for commercial AI software. The use of publicly available datasets is governed by dataset licenses. These dataset licenses outline the rights... +
+ + + + + + + + +
Original URL: https://arxiv.org/abs/2111.02374
+ + + +
+
+ + + + +
OpenDataology/OpenDataology
+
+ Practice of AI dataset metadata and license compliance +
+ +
+
Stars
+ 3 +
+ +
+
Last updated
+ 3 days ago +
+ + + + + + + + +
+ + + +
+ 👀 Howard Yoo, Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ilqar Memmedov + (ccmilgar@gmail.com) +
+
2022-05-30 04:25:10
 +
+

Hi guys, sorry for basics. +I did some PoC for OpenLineage usage for gathering metrics on Spark job, especially for table creation, alter and drop +I detect that Drop/Alter table statements is not trigger listener to post lineage data, Is it normal behaviour?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-05-30 05:38:41
 +
+

*Thread Reply:* Might be that case if you're using Spark 3.2

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-05-30 05:38:54
 +
+

*Thread Reply:* There were some changes to those operators

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-05-30 05:39:09
 +
+

*Thread Reply:* If you're not using 3.2, please share more details 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ilqar Memmedov + (ccmilgar@gmail.com) +
+
2022-05-30 07:58:58
 +
+

*Thread Reply:* Yeap, im using spark version 3.2.1

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ilqar Memmedov + (ccmilgar@gmail.com) +
+
2022-05-30 07:59:35
 +
+

*Thread Reply:* is it open issue, or i have some option to force them to be sent?)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ilqar Memmedov + (ccmilgar@gmail.com) +
+
2022-05-30 07:59:58
 +
+

*Thread Reply:* btw thank you for quick response @Maciej Obuchowski

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-05-30 08:00:34
 +
+

*Thread Reply:* Yes, we have issue for AlterTable at least

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2022-06-01 02:52:14
 +
+

*Thread Reply:* https://github.com/OpenLineage/OpenLineage/issues/616 -> that’s the issue for altering tables in Spark 3.2. +@Ilqar Memmedov Did you mean drop table or drop columns? I am not aware of any drop table issue.

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/issues/616|#616 [INTEGRATION][SPARK] Implement dataset builders for alter table commands>
+
+ <p>Spark 3.2.1 introduced new alter table commands. In order to collect this information, new dataset builders need to be created.</p> + +<p><a href="https://github.com/apache/spark/blob/v3.2.1/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/v2AlterTableCommands.scala">https://github.com/apache/spark/blob/v3.2.1/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/v2AlterTableCommands.scala</a></p> +
+ +
+
Assignees
+ <a href="https://github.com/tnazarew">@tnazarew</a> +
+ +
+
Labels
+ enhancement, integration/spark +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ilqar Memmedov + (ccmilgar@gmail.com) +
+
2022-06-01 06:03:38
 +
+

*Thread Reply:* @Paweł Leszczyński drop table statement.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ilqar Memmedov + (ccmilgar@gmail.com) +
+
2022-06-01 06:05:58
 +
+

*Thread Reply:* For reproduce it, i just create simple spark job. +Create table as select from other, +Select data from table, and then drop entire table.

+ +

Lineage data was posted only for "Create table as select" part

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
xiang chen + (cdmikechen@hotmail.com) +
+
2022-06-01 05:16:01
 +
+

Hi~all, I have a question about lineage. I am now running airflow 2.3.1 and have started a latest marquez service by docker-compose. I found that using the example DAG of airflow can only see the job information, but not the lineage of the job. How can I configure it to see the lineage ?

+ +
+
+ + + + + + +
+
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-06-03 14:20:16
 +
+

*Thread Reply:* hi xiang 👋 lineage in airflow depends on the operator. some operators have extractors as part of the integration, but when they are missing you only see job information in Marquez.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-06-03 14:20:51
 +
+

*Thread Reply:* take a look at https://github.com/OpenLineage/OpenLineage/tree/main/integration/airflow#extractors--sending-the-correct-data-from-your-dags for a bit more detail

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
xiang chen + (cdmikechen@hotmail.com) +
+
2022-06-01 05:23:54
 +
+

Another problem is that if I declare a skip task(e.g. DummyOperator) in the DAG, it will never appear in the job list. I think this is a problem, because even if it can not run, it should be able to see it as a metadata object.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-06-01 10:19:33
 +
+

@channel The next OpenLineage Technical Steering Committee meeting is on Thursday, June 9 at 10 am PT. Join us on Zoom: https://us02web.zoom.us/j/81831865546?pwd=RTladlNpc0FTTDlFcWRkM2JyazM4Zz09 +All are welcome! +Agenda:

+ +
  1. a recent blog post about Snowflake
  2. the Great Expectations integration
  3. the dbt integration
  4. Open discussion +Notes: https://tinyurl.com/openlineagetsc +Is there a topic you think the community should discuss at this or a future meeting? DM me to add items to the agenda.
    5. +
+ + + +
+ 👀 Howard Yoo, Francis McGregor-Macdonald +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-06-04 09:45:41
 +
+

@channel OpenLineage 0.9.0 is now available, featuring column-level lineage in the Spark integration, bug fixes and more! For the details, see: https://github.com/OpenLineage/OpenLineage/releases/tag/0.9.0 and https://github.com/OpenLineage/OpenLineage/compare/0.8.2...0.9.0. Thanks to all the contributors who made this release possible, including @Paweł Leszczyński for authoring the column-level lineage PRs and new contributor @JDarDagran!

+ + + +
+ 👍 Howard Yoo, Jarek Potiuk, Maciej Obuchowski, Ross Turk, Minkyu Park, pankaj koti, Jorik, Li Ding, Faouzi, Howard Yoo, Mardaunt +
+ +
+ 🎉 pankaj koti, Faouzi, Howard Yoo, Sheeri Cabral (Collibra), Mardaunt +
+ +
+ ❤️ Faouzi, Howard Yoo, Mardaunt +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tyler Farris + (tyler@kickstand.work) +
+
2022-06-06 16:14:52
 +
+

Hey, all. Working on a PR to OpenLineage. I'm curious about file naming conventions for facets. Im noticing that there are two conventions being used:

+ +

• In OpenLineage.spec.facets; ex. ExampleFacet.json +• In OpenLineage.integration.common.openlineage.common.schema; ex. example-facet.json. +Thanks

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-06-08 08:02:58
 +
+

*Thread Reply:* I think internal naming is more important 🙂

+ +

I guess, for now, try to match what the local directory has.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tyler Farris + (tyler@kickstand.work) +
+
2022-06-08 10:59:39
 +
+

*Thread Reply:* Thanks @Maciej Obuchowski

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
raghanag + (raghanag@gmail.com) +
+
2022-06-07 03:24:03
 +
+

Hi Team, we are seeing DatasetName as the Custom query when we run a spark job which queries Oracle DB using JDBC with a Custom Query and the custom query is having newline syntax in it which is causing the NodeId ID_PATTERN match to fail. How to give custom dataset name when we use custom queries?

+ +

Marquez API regex ref: https://github.com/MarquezProject/marquez/blob/main/api/src/main/java/marquez/service/models/NodeId.java#L44 +ERROR [2022-06-07 06:11:49,592] io.dropwizard.jersey.errors.LoggingExceptionMapper: Error handling a request: 3648e87216d7815b +! java.lang.IllegalArgumentException: node ID (dataset:oracle:thin:_//&lt;host-name&gt;:1521:( +! SELECT +! RULE.RULE_ID, +! ASSG.ASSIGNED_OBJECT_ID, ASSG.ORG_ID, ASSG.SPLIT_PCT, +! PRTCP.PARTICIPANT_NAME, PRTCP.START_DATE, PRTCP.END_DATE +! FROM RULE RULE, +! ASSG ASSG, +! PRTCP PRTCP +! WHERE +! RULE.RULE_ID = ASSG.RULE_ID(+) +! --AND RULE.RULE_ID = 300100207891651 +! AND PRTCP.PARTICIPANT_ID = ASSG.ASSIGNED_OBJECT_ID +! -- and RULE.created_by = ' 1=1 ' +! and 1=1 +! )) must start with 'dataset', 'job', or 'run'

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
George Zachariah V + (manish.zack@gmail.com) +
+
2022-06-08 07:48:16
 +
+

Hi Team, +We have a spark job xyz that uses OpenLineageListener which posts Lineage events to Marquez server. But we are seeing some unknown jobs in the Marquez UI : +• xyz.collect_limit +• xyz.execute_insert_into_hadoop_fs_relation_command +What jobs are these (collect_limit, execute_insert_into_hadoop_fs_relation_command ) ? +How do we get the lineage listener to post only our job (xyz) ?

+ + + +
+ 👍 Pradeep S +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-06-08 11:00:41
 +
+

*Thread Reply:* Those jobs are actually what Spark does underneath 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-06-08 11:00:57
 +
+

*Thread Reply:* Are you using Delta Lake btw?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Moiz + (moiz.groups@gmail.com) +
+
2022-06-08 12:02:39
 +
+

*Thread Reply:* No, this is not Delta Lake. It is a normal Spark app .

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
raghanag + (raghanag@gmail.com) +
+
2022-06-08 13:58:05
 +
+

*Thread Reply:* @Maciej Obuchowski i think David posted about this before. https://openlineage.slack.com/archives/C01CK9T7HKR/p1636011698055200

+
+ + +
+ + + } + + David Virgil + (https://openlineage.slack.com/team/U02K9U58X7F) +
+ + + +
+
+ <p>Hello openlineage community.</p> + +<p>Yesterday I tried the integration with spark.</p> + +<p>The result was not satisfactory. This is what I did:</p> + +<ol><li>Add openlineage-spark dependency</li><li>Add these lines: +<code>.config("spark.jars.packages", "io.openlineage:openlineage_spark:0.3.1") +.config("spark.extraListeners", "io.openlineage.spark.agent.OpenLineageSparkListener") +.config("spark.openlineage.url", "&lt;https://marquez-internal-eks.eu-west-1.dev.hbi.systems/api/v1/namespaces/spark_integration/&gt;"</code> +This job was doing spark.read from 2 different json location. +It is doing spark write to 5 different parquet location in s3. +The job finished succesfully and the result in marquez is:</li> +</ol> +
+ + + + + + +
Original URL: https://openlineage.slack.com/archives/C01CK9T7HKR/p1636011698055200
+ + +
+ + Slack Conversation +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-06-08 14:27:46
 +
+

*Thread Reply:* I agree that it looks bad on UI, but I also think integration is going good job here. The eventual "aggregation" should be done by event consumer.

+ +

If anything, we should filter some 'useless' nodes like collect_limit since they add nothing.

+ +

We have an issue for doing this to specifically delta lake operations, as they are the biggest offenders: https://github.com/OpenLineage/OpenLineage/issues/628

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/issues/628|#628 [INTEGRATION][SPARK] Filter events for Delta operations>
+
+ <p>Delta operations in Spark SQL generate a ton of excess run events. There are two general causes for this.</p> + +<ol><li>Delta reads and writes Snapshot and delta files during execution - it access these files as DataFrames, which causes any access to be treated as a new SparkSQL execution - see the <a href="https://github.com/delta-io/delta/blob/master/core/src/main/scala/org/apache/spark/sql/delta/Snapshot.scala#L429-L457">source code</a>.</li><li>Delta writes start a new Spark SQLExecution separate from the original plan - this query references only the original query, not the write command. It converts the query to an RDD, then executes a new RDD job that writes to the target path.</li> +</ol> + +<p>Examples of the above two behaviors:</p> + +<p>The following events are generated by running the following Spark code, reading som JSON from a local file and writing it to a delta table</p> + +<pre><code> Dataset&amp;lt;Row&amp;gt; df = spark.read().json("file://" + testFile.toAbsolutePath()); + + df.filter("age &amp;gt; 100").write().format("delta").mode("append").save(deltaDir.toString()); +</code></pre> + +<hr /> + +<p><strong>Snapshot Operations</strong></p> + +<p>Abbreviated event for Snapshot query - note the input path includes <code>deltaTable/_delta_log</code>. All delta log file operations seem to generate a query similar to this.</p> + +<p>``` +{ + "eventType": "START", + "eventTime": "2021-01-01T00:00:00Z", + "run": { + "runId": "453ddea0-0f36-4eae-b420-d1b3eb3d232f", + "facets": { + "parent": { + "<em>producer": "<a href="https://github.com/OpenLineage/OpenLineage/tree/0.7.0-SNAPSHOT/integration/spark">https://github.com/OpenLineage/OpenLineage/tree/0.7.0-SNAPSHOT/integration/spark</a>", + "</em>schemaURL": "<a href="https://openlineage.io/spec/facets/1-0-0/ParentRunFacet.json#/$defs/ParentRunFacet">https://openlineage.io/spec/facets/1-0-0/ParentRunFacet.json#/$defs/ParentRunFacet</a>", + "run": { + "runId": "e6018ec7-a314-439c-888b-ddd63e927c05" + }, + "job": { + "namespace": "Namespace", + "name": "ParentJob" + } + }, + "spark<em>version": { + "</em>producer": "<a href="https://github.com/OpenLineage/OpenLineage/tree/0.7.0-SNAPSHOT/integration/spark">https://github.com/OpenLineage/OpenLineage/tree/0.7.0-SNAPSHOT/integration/spark</a>", + "<em>schemaURL": "<a href="https://openlineage.io/spec/1-0-2/OpenLineage.json#/$defs/RunFacet">https://openlineage.io/spec/1-0-2/OpenLineage.json#/$defs/RunFacet</a>", + "spark-version": "3.1.2" + }, + "spark.logicalPlan": { + "</em>producer": "<a href="https://github.com/OpenLineage/OpenLineage/tree/0.7.0-SNAPSHOT/integration/spark">https://github.com/OpenLineage/OpenLineage/tree/0.7.0-SNAPSHOT/integration/spark</a>", + "<em>schemaURL": "<a href="https://openlineage.io/spec/1-0-2/OpenLineage.json#/$defs/RunFacet">https://openlineage.io/spec/1-0-2/OpenLineage.json#/$defs/RunFacet</a>", + "plan": [ + { + "class": "org.apache.spark.sql.catalyst.plans.logical.Aggregate", + "num-children": 1, + "groupingExpressions": [], + "aggregateExpressions": [...], + "child": 0 + }, + { + "class": "org.apache.spark.sql.catalyst.plans.logical.Project", + "num-children": 1, + "projectList": [...], + "child": 0 + }, + { + "class": "org.apache.spark.sql.execution.LogicalRDD", + "num-children": 0, + "output": [...], + "rdd": null, + "outputPartitioning": { + "product-class": "org.apache.spark.sql.catalyst.plans.physical.UnknownPartitioning", + "numPartitions": 0 + }, + "outputOrdering": [], + "isStreaming": false, + "session": null + } + ] + }, + "spark</em>unknown": { + "<em>producer": "<a href="https://github.com/OpenLineage/OpenLineage/tree/0.7.0-SNAPSHOT/integration/spark">https://github.com/OpenLineage/OpenLineage/tree/0.7.0-SNAPSHOT/integration/spark</a>", + "</em>schemaURL": "<a href="https://openlineage.io/spec/1-0-2/OpenLineage.json#/$defs/RunFacet">https://openlineage.io/spec/1-0-2/OpenLineage.json#/$defs/RunFacet</a>", + "output": { + "description": { + "@class": "org.apache.spark.sql.catalyst.plans.logical.Aggregate", + "serializedId": 1, + "groupingExpressions": [], + "aggregateExpressions": [...], + "child": { + "@class": "org.apache.spark.sql.catalyst.plans.logical.Project", + "serializedId": 173, + "projectList": [ + 59, + 73, + 142, + 35, + 5 + ], + "child": { + "@class": "org.apache.spark.sql.execution.LogicalRDD", + "serializedId": 174, + "output": [...], + "rdd": { + "@class": "org.apache.spark.rdd.MapPartitionsRDD", + "serializedId": 179, + "prev": { + "@class": "org.apache.spark.sql.execution.SQLExecutionRDD", + "serializedId": 180, + "sqlRDD": { + "@class": "org.apache.spark.rdd.MapPartitionsRDD", + "serializedId": 181, + "prev": { + "@class": "org.apache.spark.rdd.MapPartitionsRDD", + "serializedId": 182, + "prev": { + "@class": "org.apache.spark.rdd.MapPartitionsRDD", + "serializedId": 183, + "prev": { + "@class": "org.apache.spark.rdd.MapPartitionsRDD", + "serializedId": 184, + "prev": { + "@class": "org.apache.spark.sql.execution.ShuffledRowRDD", + "serializedId": 185, + "dependency": { + ..., + "rdd": { + "@class": "org.apache.spark.rdd.MapPartitionsRDD", + "serializedId": 191, + "prev": { + "@class": "org.apache.spark.rdd.MapPartitionsRDD", + "serializedId": 192, + "prev": { + "@class": "org.apache.spark.rdd.MapPartitionsRDD", + "serializedId": 193, + "prev": { + "@class": "org.apache.spark.rdd.MapPartitionsRDD", + "serializedId": 194, + "prev": { + "@class": "org.apache.spark.rdd.MapPartitionsRDD", + "serializedId": 195, + "prev": { + "@class": "org.apache.spark.sql.execution.datasources.FileScanRDD", + "serializedId": 196, + "readFunction": { + "serializedId": 197 + }, + ... + }, + ... + }, + ... + }, + ... + }, + ... + }, + ... + } + }, + ... + }, + ... + }, + ... + }, + ... + }, + ... + }, + ... + }, + ... + }, + ... + }, + ... + }, + ... + }, + "inputAtt…</p> +
+ +
+
Milestone
+ <a href="https://github.com/OpenLineage/OpenLineage/milestone/6">0.10.0</a> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+ 👍 George Zachariah V +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
raghanag + (raghanag@gmail.com) +
+
2022-06-08 14:33:09
 +
+

*Thread Reply:* @Maciej Obuchowski but we only see these 2 jobs in the namespace, no other jobs were part of the lineage metadata, are we doing something wrong?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
raghanag + (raghanag@gmail.com) +
+
2022-06-08 16:09:15
 +
+

*Thread Reply:* @Michael Robinson On this note, may we know how to form a lineage if we have different set of API's before calling the spark job (already integrated with OpenLineageSparkListener), we want to see how the different set of params pass thru these components before landing into the spark job. If we use openlineage client to post the lineage events into the Marquez, do we need to mention the same Run UUID across the lineage events for the run or is there any other way to do this? Can you pls advise?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-06-08 22:51:38
 +
+

*Thread Reply:* I think I understand what you are asking -

+ +

The runID is used to correlate different state updates (i.e., start, fail, complete, abort) across the lifespan of a run. So if you are trying to add additional metadata to the same job run, you’d use the same runID.

+ +

So you’d generate a runID and send a START event, then in the various components you could send OTHER events containing the same runID + params you want to study in facets, then at the end you would send a COMPLETE.

+ +

(I think there should be an UPDATE event type in the spec for this sort of thing.)

+ + + +
+ 👍 George Zachariah V, raghanag +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
raghanag + (raghanag@gmail.com) +
+
2022-06-08 22:59:39
 +
+

*Thread Reply:* thanks @Ross Turk but what i am looking for is lets say for example, if we have 4 components in the system then we want to show the 4 components as job icons in the graph and the datasets between them would show the input/output parameters that these components use. +A(job) --> DS1(dataset) --> B(job) --> DS2(dataset) --> C(job) --> DS3(dataset) --> D(job)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-06-08 23:04:37
 +
+

*Thread Reply:* then you would need to have separate Jobs for each, with inputs and outputs defined

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-06-08 23:06:03
 +
+

*Thread Reply:* so there would be a Run of job B that shows DS1 as an input and DS2 as an output

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
raghanag + (raghanag@gmail.com) +
+
2022-06-08 23:06:18
 +
+

*Thread Reply:* got it

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-06-08 23:06:34
 +
+

*Thread Reply:* (fyi: I know openlineage but my understanding stops at spark 😄)

+ + + +
+ 👍 raghanag +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sharanya Santhanam + (santhanamsharanya@gmail.com) +
+
2022-06-10 12:27:58
 +
+

*Thread Reply:* > The eventual “aggregation” should be done by event consumer. +@Maciej Obuchowski Are there any known client side libraries that support this aggregation already ? In case of spark applications running as part of ETL pipelines, most of the times our end user is interested in seeing only the aggregated view where all jobs spawned as part of a single application are rolled up into 1 job.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-06-10 12:32:14
 +
+

*Thread Reply:* I believe Microsoft @Will Johnson has something similar to that, but it's probably proprietary.

+ +

We'd love to have something like it, but AFAIK it affects only some percentage of Spark jobs and we can only do so much.

+ +

With exception of Delta Lake/Databricks, where it affects every job, and we know some nodes that could be safely filtered client side.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-06-11 23:38:27
 +
+

*Thread Reply:* @Maciej Obuchowski Microsoft ❤️ OSS!

+ +

Apache Atlas doesn't have the same model as Marquez. It only knows of effectively one entity that represents the complete asset.

+ +

@Mark Taylor designed this solution available now on Github to consolidate OpenLineage messages

+ +

https://github.com/microsoft/Purview-ADB-Lineage-Solution-Accelerator/blob/d6514f2[…]/Function.Domain/Helpers/OlProcessing/OlMessageConsolodation.cs

+ +

In addition, we do some filtering only based on inputs and outputs to limit the messages AFTER it has been emitted.

+
+ + + + +
<https://github.com/microsoft/Purview-ADB-Lineage-Solution-Accelerator/blob/d6514f25c7f96151a197429efb9b7d5d425d0776/function-app/adb-to-purview/src/Function.Domain/Helpers/OlProcessing/OlMessageConsolodation.cs | OlMessageConsolodation.cs>
+
+ <pre><code> public class OlMessageConsolodation : IOlMessageConsolodation +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/microsoft/Purview-ADB-Lineage-Solution-Accelerator">microsoft/Purview-ADB-Lineage-Solution-Accelerator</a> +
+ + +
+ + + +
+ 🙌 Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sharanya Santhanam + (santhanamsharanya@gmail.com) +
+
2022-06-19 09:37:06
 +
+

*Thread Reply:* thank you !

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-06-08 10:54:32
 +
+

@channel The next OpenLineage TSC meeting is tomorrow! https://openlineage.slack.com/archives/C01CK9T7HKR/p1654093173961669

+
+ + +
+ + + } + + Michael Robinson + (https://openlineage.slack.com/team/U02LXF3HUN7) +
+ + + +
+
+ <p>@channel The next OpenLineage Technical Steering Committee meeting is on <strong>Thursday, June 9 at 10 am PT</strong>. Join us on Zoom: <a href="https://us02web.zoom.us/j/81831865546?pwd=RTladlNpc0FTTDlFcWRkM2JyazM4Zz09">https://us02web.zoom.us/j/81831865546?pwd=RTladlNpc0FTTDlFcWRkM2JyazM4Zz09</a> +All are welcome! +Agenda:</p> + +<ol><li>a recent blog post about Snowflake</li><li>the Great Expectations integration</li><li>the dbt integration</li><li>Open discussion +Notes: <a href="https://tinyurl.com/openlineagetsc">https://tinyurl.com/openlineagetsc</a> +Is there a topic you think the community should discuss at this or a future meeting? DM me to add items to the agenda.</li> +</ol> +
+ + + + + + +
Original URL: https://openlineage.slack.com/archives/C01CK9T7HKR/p1654093173961669
+ + +
+ + Slack Conversation +
+ + +
+ + + +
+ 👍 Maciej Obuchowski, Sheeri Cabral (Collibra), Willy Lulciuc, raghanag, Mardaunt +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Moravec + (jkb.moravec@gmail.com) +
+
2022-06-09 13:04:00
 +
+

*Thread Reply:* Hi, is the link correct? The meeting room is empty

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-06-09 16:04:23
 +
+

*Thread Reply:* sorry about that, thanks for letting us know

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mark Beebe + (mark_j_beebe@progressive.com) +
+
2022-06-13 15:13:59
 +
+

Hello all, after sending dbt openlineage events to Marquez, I am now looking to use the Marquez API to extract the lineage information. I am able to use python requests to call the Marquez API to get other information such as namespaces, datasets, etc., but I am a little bit confused about what I need to enter to get the lineage. I included screenshots for what the API reference shows regarding retrieving the lineage where it shows that a nodeId is required. However, this is where I seem to be having problems. It is not exactly clear where the nodeId needs to be set or what the nodeId needs to include. I would really appreciate any insights. Thank you!

+ +
+
+ + + + + + +
+
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-06-13 18:49:37
 +
+

*Thread Reply:* Hey @Mark Beebe!

+ +

In this case, nodeId is going to be either a dataset or a job. You need to tell Marquez where to start since there is likely to be more than one graph. So you need to get your hands on an identifier for that starting node.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-06-13 18:50:07
 +
+

*Thread Reply:* You can do this in a few ways (that I can think of). First, by looking for a namespace, then querying for the datasets in that namespace:

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-06-13 18:53:43
 +
+

*Thread Reply:* Or you can search, if you know the name of the dataset:

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-06-13 18:53:54
 +
+

*Thread Reply:* aaaaannnnd that’s actually all the ways I can think of.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mark Beebe + (mark_j_beebe@progressive.com) +
+
2022-06-14 08:11:30
 +
+

*Thread Reply:* That worked, thank you so much!

+ + + +
+ 👍 Ross Turk +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Varun Singh + (varuntestaz@outlook.com) +
+
2022-06-14 05:52:39
 +
+

Hi all, I need to send the lineage information from spark integration directly to a kafka topic. Java client seems to have a KafkaTransport, is it planned to have this support from inside the spark integration as well?

+ + + +
+ 👀 Francis McGregor-Macdonald +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-06-14 10:35:48
 +
+

Hi all, I’m working on a blog post about the Spark integration and would like to credit @tnazarew and @Sbargaoui for their contributions. Anyone know these contributors’ names? Are you on here? Thanks for any leads.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-06-14 10:37:01
 +
+

*Thread Reply:* tnazarew - Tomasz Nazarewicz

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-06-14 10:37:14
 +
+

*Thread Reply:* 🙌

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-06-15 12:46:45
 +
+

*Thread Reply:* 👍

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Conor Beverland + (conorbev@gmail.com) +
+
2022-06-14 13:58:07
 +
+

Has anyone tried getting the OpenLineage Spark integration working with GCP Dataproc ?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Peter Hanssens + (peter@cloudshuttle.com.au) +
+
2022-06-15 15:49:17
 +
+

Hi Folks, +DataEngBytes is a community data engineering conference here in Australia and will be hosted on the 27th and 29th of September. Our CFP is open for just under a month and tickets are on sale now: +Call for paper: https://sessionize.com/dataengbytes-2022/ +Tickets: https://www.tickettailor.com/events/dataengbytes/713307 +Promo video +https://youtu.be/1HE_XNLvHss

+
+
sessionize.com
+ + + +
DataEngBytes 2022: Call for Speakers/Papers
+
+ DataEngBytes! is a community conference centred on real world data engineering problems and solutions.We will have two days of in-person conferences i... +
+ + + + + + +
Original URL: https://sessionize.com/dataengbytes-2022/
+ + + +
+
+
tickettailor.com
+ + + +
Buy tickets – DataEngBytes 2022, Sydney! – Sydney, Thu 29 Sep 2022 9:00 AM - 5:00 PM
+
+ DataEngBytes 2022, Sydney! – Sydney, Thu 29 Sep 2022 - DataEngBytes! is a community conference centred on real world data engineering problems and solutions. We will have two days of in-person conferences in Melbourne (on the 27th) and Sydney (on the 29th) of September with both days broadcast online on our YouTube ... +
+ + + + + + + + +
Original URL: https://www.tickettailor.com/events/dataengbytes/713307
+ + + +
+
+
YouTube
+ +
+ + + } + + DataEngAU + (https://www.youtube.com/c/DataEngAU) +
+ + + +
DataEngBytes 2022 Announcement
+
+ +
+ + + + + + + + +
Original URL: https://youtu.be/1HE_XNLvHss
+ + + +
+ + + +
+ 👀 Ross Turk, Michael Collado +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-06-17 16:23:32
 +
+

A release of OpenLineage has been requested pending the merging of #856. Three +1s will authorize a release today. +@Willy Lulciuc @Michael Collado @Ross Turk @Maciej Obuchowski @Paweł Leszczyński @Mandy Chessell @Daniel Henneberger @Drew Banin @Julien Le Dem @Ryan Blue @Will Johnson @Zhamak Dehghani

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/pull/856|#856 Fix spark casting error and session catalog support for `iceberg`>
+
+ <p><strong>Problem</strong></p> + +<p>Class cast exception for <code>DataSourceV2ScanRelation</code>:</p> + +<pre><code>java.lang.ClassCastException: class org.apache.spark.sql.execution.datasources.v2.DataSourceV2ScanRelation cannot be cast to class org.apache.spark.sql.execution.datasources.v2.DataSourceV2Relation (org.apache.spark.sql.execution.datasources.v2.DataSourceV2ScanRelation and org.apache.spark.sql.execution.datasources.v2.DataSourceV2Relation are in unnamed module of loader 'app') + at io.openlineage.spark3.agent.lifecycle.plan.TableContentChangeDatasetBuilder.apply(TableContentChangeDatasetBuilder.java:83) ~[openlineage-spark-0.9.0.jar:0.9.0] + at io.openlineage.spark.api.AbstractQueryPlanDatasetBuilder.apply(AbstractQueryPlanDatasetBuilder.java:92) ~[openlineage-spark-0.9.0.jar:0.9.0] + at io.openlineage.spark.api.AbstractQueryPlanDatasetBuilder$1.apply(AbstractQueryPlanDatasetBuilder.java:86) ~[openlineage-spark-0.9.0.jar:0.9.0] + at io.openlineage.spark.api.AbstractQueryPlanDatasetBuilder$1.apply(AbstractQueryPlanDatasetBuilder.java:77) ~[openlineage-spark-0.9.0.jar:0.9.0] + at io.openlineage.spark.api.AbstractQueryPlanDatasetBuilder.lambda$apply$0(AbstractQueryPlanDatasetBuilder.java:67) ~[openlineage-spark-0.9.0.jar:0.9.0] + at java.util.Optional.map(Optional.java:265) ~[?:?] + . + . + +</code></pre> + +<p>Unsupported spark catalog session:</p> + +<pre><code>ERROR PlanUtils3: Catalog org.apache.iceberg.spark.SparkSessionCatalog is unsupported +io.openlineage.spark3.agent.lifecycle.plan.catalog.UnsupportedCatalogException: org.apache.iceberg.spark.SparkSessionCatalog + at io.openlineage.spark3.agent.lifecycle.plan.catalog.CatalogUtils3.lambda$getDatasetIdentifier$2(CatalogUtils3.java:50) ~[openlineage-spark-0.9.0.jar:0.9.0] + at java.util.Optional.orElseThrow(Optional.java:408) ~[?:?] + at io.openlineage.spark3.agent.lifecycle.plan.catalog.CatalogUtils3.getDatasetIdentifier(CatalogUtils3.java:50) ~[openlineage-spark-0.9.0.jar:0.9.0] + at io.openlineage.spark3.agent.lifecycle.plan.catalog.CatalogUtils3.getDatasetIdentifier(CatalogUtils3.java:36) ~[openlineage-spark-0.9.0.jar:0.9.0] + at io.openlineage.spark3.agent.utils.PlanUtils3.getDatasetIdentifier(PlanUtils3.java:67) ~[openlineage-spark-0.9.0.jar:0.9.0] + at io.openlineage.spark3.agent.utils.PlanUtils3.fromDataSourceV2Relation(PlanUtils3.java:110) ~[openlineage-spark-0.9.0.jar:0.9.0] + . + . + +</code></pre> + +<p><strong>Solution</strong></p> + +<p>Fix class cast exception for <code>DataSourceV2ScanRelation</code> and add support for <code>org.apache.iceberg.spark.SparkSessionCatalog</code>.</p> + +<p><strong>Checklist</strong></p> + +<p>☑︎ You've <a href="https://github.com/OpenLineage/OpenLineage/blob/main/why-the-dco.md">signed-off</a> your work +☑︎ Your pull request title follows our <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CONTRIBUTING.md#creating-pull-requests">guidelines</a> +☐ Your changes are accompanied by tests (<em>if relevant</em>) +☑︎ Your change contains a <a href="https://kurtisnusbaum.medium.com/stacked-diffs-keeping-phabricator-diffs-small-d9964f4dcfa6">small diff</a> and is self-contained +☐ You've updated any relevant documentation (<em>if relevant</em>) +☐ You've updated the <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md"><code>CHANGELOG.md</code></a> with details about your change under the "Unreleased" section (<em>if relevant, depending on the change, this may not be necessary</em>) +☐ You've versioned the core OpenLineage model or facets according to <a href="https://docs.snowplowanalytics.com/docs/pipeline-components-and-applications/iglu/common-architecture/schemaver">SchemaVer</a> (<em>if relevant</em>)</p> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+ ➕ Willy Lulciuc, Maciej Obuchowski, Michael Collado +
+ +
+ ✅ Michael Collado +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Chase Christensen + (christensenc3526@gmail.com) +
+
2022-06-22 17:09:18
 +
+

👋 Hi everyone!

+ + + +
+ 👋 Conor Beverland, Ross Turk, Maciej Obuchowski, Michael Robinson, George Zachariah V, Willy Lulciuc, Dinakar Sundar +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Lee + (chenzuoli709@gmail.com) +
+
2022-06-23 21:54:05
 +
+

hi

+ + + +
+ 👋 Maciej Obuchowski, Sheeri Cabral (Collibra), Willy Lulciuc, Michael Robinson, Dinakar Sundar +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-06-25 07:34:32
 +
+

@channel OpenLineage 0.10.0 is now available! We added SnowflakeOperatorAsync extractor support to the Airflow integration, an InMemoryRelationInputDatasetBuilder for InMemory datasets to the Spark integration, a static code analysis tool to run in CircleCI on Python modules, a copyright to all source files, and a debugger called PMD to the build process. +Changes we made include skipping FunctionRegistry.class serialization in the Spark integration, installing the new rust-based SQL parser by default in the Airflow integration, improving the integration tests for the Airflow integration, reducing event payload size by excluding local data and including an output node in start events, and splitting the Spark integration into submodules. +Thanks to all the contributors who made this release possible! +Release: https://github.com/OpenLineage/OpenLineage/releases/tag/0.10.0 +Changelog: https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md +Commit history: https://github.com/OpenLineage/OpenLineage/compare/0.9.0...0.10.0 +Maven: https://oss.sonatype.org/#nexus-search;quick~openlineage +PyPI: https://pypi.org/project/openlineage-python/

+ + + +
+ 🙌 Maciej Obuchowski, Filipe Comparini Vieira, Manuel, Dinakar Sundar, Ross Turk, Paweł Leszczyński, Willy Lulciuc, Adisesha Reddy G, Conor Beverland, Francis McGregor-Macdonald, Jam Car +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mike brenes + (brenesmi@gmail.com) +
+
2022-06-28 18:29:29
 +
+

Why has put dataset been deprecated? How do I add an initial data set via api?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2022-06-28 18:39:16
 +
+

*Thread Reply:* I think you’re reference the deprecation of the DatasetAPI in Marquez? A milestone for the Marquez is to only collect metadata via OpenLineage events. This includes metadata for datasets , jobs , and runs . The DatasetAPI won’t be removed until support for collecting dataset metadata via OpenLineage has been added, see https://github.com/OpenLineage/OpenLineage/issues/323

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/issues/323|#323 [PROPOSAL] Ability to annotate datasets with metadata without a job run>
+
+ <p><strong>Purpose:</strong> <br /> +We need a way to make observations about a Dataset current state outside of a run reading or writing the data. <br /> +For example, this could be:</p> + +<p>• collecting schema information about a table from a warehouse +• informing that a table has been deleted +• etc</p> + +<p><strong>Proposed implementation</strong> <br /> +Adding Dataset event types that are not lineage events. This would be the Dataset entity with its facets: </p> + +<p><a href="https://github.com/OpenLineage/OpenLineage/blob/3a1ccbd854bbf202bbe6437bf81786cb01ed4d01/spec/OpenLineage.json#L178-L203">OpenLineage/spec/OpenLineage.json</a></p> + +<p>Lines 178 to 203 in &lt;/OpenLineage/OpenLineage/commit/3a1ccbd854bbf202bbe6437bf81786cb01ed4d01|3a1ccbd&gt;</p> + +<pre><code>"Dataset": { + "type": "object", + "properties": { + "namespace": { + "description": "The namespace containing that dataset", + "type": "string", + "example": "my-datasource-namespace" + }, + "name": { + "description": "The unique name for that dataset within that namespace", + "type": "string", + "example": "instance.schema.table" + }, + "facets": { + "description": "The facets for this dataset", + "type": "object", + "additionalProperties": { + "$ref": "#/$defs/DatasetFacet" + } + } + }, + "required": [ + "namespace", + "name" + ] + } + +</code></pre> +
+ +
+
Assignees
+ <a href="https://github.com/mobuchowski">@mobuchowski</a> +
+ +
+
Labels
+ proposal +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2022-06-28 18:40:28
 +
+

*Thread Reply:* Once the spec supports dataset metadata, we’ll outline steps in the Marquez project to switch to using the new dataset event type

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2022-06-28 18:43:20
 +
+

*Thread Reply:* The DatasetAPI was also deprecated to avoid confusion around which API to use

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mike brenes + (brenesmi@gmail.com) +
+
2022-06-28 18:41:38
 +
+

🥺

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mike brenes + (brenesmi@gmail.com) +
+
2022-06-28 18:42:21
 +
+

So how would you propose I create the initial node if I am trying to do a POC?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2022-06-28 18:44:49
 +
+

*Thread Reply:* Do you want to register just datasets? Or are you extracting metadata for a job that would include input / output datasets? (outside of Airflow of course)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mike brenes + (brenesmi@gmail.com) +
+
2022-06-28 18:45:09
 +
+

*Thread Reply:* Sorry didn't notice you over here ! lol

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mike brenes + (brenesmi@gmail.com) +
+
2022-06-28 18:45:53
 +
+

*Thread Reply:* So ideally I would like to map out our current data flow from on prem to aws

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2022-06-28 18:47:39
 +
+

*Thread Reply:* What do you mean by mapping to AWS? Like send OL events to a service on AWS that would process the lineage metadata?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mike brenes + (brenesmi@gmail.com) +
+
2022-06-28 18:48:14
 +
+

*Thread Reply:* no, just visualize the current migration flow.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2022-06-28 18:48:53
 +
+

*Thread Reply:* Ah I see, youre doing a infra migration from on prem to AWS 👌

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mike brenes + (brenesmi@gmail.com) +
+
2022-06-28 18:49:08
 +
+

*Thread Reply:* really AWS is irrelevant. Source sink -> migration scriipts -> s3 -> additional processing -> final sink

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mike brenes + (brenesmi@gmail.com) +
+
2022-06-28 18:49:19
 +
+

*Thread Reply:* correct

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2022-06-28 18:49:45
 +
+

*Thread Reply:* right right. so you want to map out that flow and visualize it in Marquez? (or some other meta service)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mike brenes + (brenesmi@gmail.com) +
+
2022-06-28 18:50:05
 +
+

*Thread Reply:* yes

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mike brenes + (brenesmi@gmail.com) +
+
2022-06-28 18:50:26
 +
+

*Thread Reply:* which I think I can do once the first nodes exist

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mike brenes + (brenesmi@gmail.com) +
+
2022-06-28 18:51:18
 +
+

*Thread Reply:* But I don't know how to get that initial node. I tried using the input facet at job start , that didn't do it. I also can't get the sql context that is in these examples.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mike brenes + (brenesmi@gmail.com) +
+
2022-06-28 18:51:54
 +
+

*Thread Reply:* really just want to re-create food_devlivery using my own biz context

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2022-06-28 18:52:14
 +
+

*Thread Reply:* Have you looked over our workshops and this example? (assuming you’re using python?)

+
+ + + + +
<https://github.com/OpenLineage/workshops/blob/main/airflow/e2-lineage-api/python/generate-events.py | generate-events.py>
+
+ <pre><code>#!/usr/bin/env python3 + +from openlineage.client.run import RunEvent, RunState, Run, Job, Dataset +from openlineage.client.client import OpenLineageClient +from datetime import datetime +from uuid import uuid4 + +# Initialize the OpenLineage client +client = OpenLineageClient.from_environment() + +# Specify the producer of this lineage metadata +producer = "&lt;https://github.com/OpenLineage/workshops&gt;" + +# Create some basic Dataset objects for our fictional pipeline +monthly_summary = Dataset(namespace="&lt;postgres://workshop-db:None&gt;", name="workshop.public.monthly_summary") +commissions = Dataset(namespace="&lt;postgres://workshop-db:None&gt;", name="workshop.public.commissions") +taxes = Dataset(namespace="&lt;postgres://workshop-db:None&gt;", name="workshop.public.taxes") + +# Create a Job object +job = Job(namespace="workshop", name="monthly_accounting") + +# Create a Run object with a unique ID +run = Run(str(uuid4())) + +# Emit a START run event +client.emit( + RunEvent( + RunState.START, + datetime.now().isoformat(), + run, job, producer + ) +) + +# +# This is where our application would do the actual work :) +# + +# Emit a COMPLETE run event +client.emit( + RunEvent( + RunState.COMPLETE, + datetime.now().isoformat(), + run, job, producer, + inputs=[monthly_summary], + outputs=[commissions, taxes], + ) +) + +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/workshops">OpenLineage/workshops</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2022-06-28 18:53:49
 +
+

*Thread Reply:* that goes over the py client with some OL examples, but really calling openlineage.emit(...) method with RunEvents and specifying Marquez as the backend will get you up and running!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2022-06-28 18:54:32
 +
+

*Thread Reply:* Don’t forget to configure the transport for the client

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mike brenes + (brenesmi@gmail.com) +
+
2022-06-28 18:54:45
 +
+

*Thread Reply:* sweet. Thank you! I'll take a look. Also.. Just came across datakin for the first time. very nice 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2022-06-28 18:55:25
 +
+

*Thread Reply:* thanks! …. but we’re now part of astronomer.io 😉

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2022-06-28 18:55:48
 +
+

*Thread Reply:* making airflow oh-so-easy-to-use one DAG at a time

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mike brenes + (brenesmi@gmail.com) +
+
2022-06-28 18:55:52
 +
+

*Thread Reply:* saw that too !

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2022-06-28 18:56:03
 +
+

*Thread Reply:* you’re on top of it!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mike brenes + (brenesmi@gmail.com) +
+
2022-06-28 18:56:28
 +
+

*Thread Reply:* ha. Thanks again!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mike brenes + (brenesmi@gmail.com) +
+
2022-06-28 18:42:40
 +
+

This would be outside of Airflow

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Fenil Doshi + (fdoshi@salesforce.com) +
+
2022-06-28 18:43:22
 +
+

Hello, +Is OpenLineage planning to add support for inlets and outlets for Airflow integration? I am working on a project that relies on it and was hoping to contribute to this feature if its something that is in the talks. +I saw an open issue here

+ +

I am willing to work on it. My plan was to just support Files and Tables entities (for inlets and outlets). +Pass the inlets and outlets info into extract_metadata function here and then convert Airflow entities into TaskMetaData entities here.

+ +

Does this sound reasonable?

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/issues/384|#384 [INTEGRATION][Airflow] Consider using manual inlets/outlets as a fallback for automatic extractor mechanism>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/integration/airflow/openlineage/lineage_backend/__init__.py | __init__.py>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2022-06-28 18:59:38
 +
+

*Thread Reply:* Honestly, I’ve been a huge fan of using / falling back on inlets and outlets since day 1. AND if you’re willing to contribute this support, you get a +1 from me (I’ll add some minor comments to the issue) /cc @Julien Le Dem

+ + + +
+ 🙌 Fenil Doshi +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2022-06-28 18:59:59
 +
+

*Thread Reply:* would be great to get @Maciej Obuchowski thoughts on this as well

+ + + +
+ 👍 Fenil Doshi +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Fenil Doshi + (fdoshi@salesforce.com) +
+
2022-07-08 12:40:39
 +
+

*Thread Reply:* I have created a draft PR for this here. +Please let me know if the changes make sense.

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/pull/914|#914 [INTEGRATION][AIRFLOW] Allow lineage metadata to flow through inlets and outlets>
+
+ <p>Signed-off-by: Fenil Doshi <a href="mailto:fdoshi@slack-corp.com">fdoshi@slack-corp.com</a></p> + +<p><strong>Problem</strong></p> + +<p>Currently, in Airflow integration, lineage is only captured via a few extractors which greatly limits its use case. Airflow allows users to manually annotate lineage for the tasks using inlets and outlets. These can be a great fallback mechanism when extractors cannot automatically extract the input and output Datasets. <br /> +The PR closes <a href="https://github.com/OpenLineage/OpenLineage/issues/384">#384</a></p> + +<p>Addresses: <a href="https://github.com/OpenLineage/OpenLineage/issues/384">#384</a></p> + +<p><strong>Solution</strong></p> + +<p>Airflow sends inlets and outlets information to <code>LineageBackend</code> class. Inlets and Outlets can be one of <a href="https://github.com/apache/airflow/blob/main/airflow/lineage/entities.py">these</a> entities. The code only currently supports <code>Table</code> entity because OpenLineage does not track column-level lineage yet.</p> + +<p>• Check the TaskMetadata object that is created after corresponding extractors extract the lineage data and see whether inputs and outputs exist +• Check whether inlets and outlets are provided for the task +• If TaskMetadata object does not have inputs/outputs, use Airflow's inlets and outlets. +• Get all the Table entities and convert them to Dataset entities that are suitable for TaskMetaData object +Your change modifies the <a href="https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.json">core</a> OpenLineage model +Your change modifies one or more OpenLineage <a href="https://github.com/OpenLineage/OpenLineage/tree/main/spec/facets">facets</a></p> + +<p><strong>Checklist</strong></p> + +<p>☑︎ You've <a href="https://github.com/OpenLineage/OpenLineage/blob/main/why-the-dco.md">signed-off</a> your work +☑︎ Your pull request title follows our <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CONTRIBUTING.md#creating-pull-requests">guidelines</a> +☐ Your changes are accompanied by tests (<em>if relevant</em>) +☑︎ Your change contains a <a href="https://kurtisnusbaum.medium.com/stacked-diffs-keeping-phabricator-diffs-small-d9964f4dcfa6">small diff</a> and is self-contained +☐ You've updated any relevant documentation (<em>if relevant</em>) +☐ You've updated the <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md"><code>CHANGELOG.md</code></a> with details about your change under the "Unreleased" section (<em>if relevant, depending on the change, this may not be necessary</em>) +☐ You've versioned the core OpenLineage model or facets according to <a href="https://docs.snowplowanalytics.com/docs/pipeline-components-and-applications/iglu/common-architecture/schemaver">SchemaVer</a> (<em>if relevant</em>)</p> +
+ +
+
Comments
+ 1 +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-07-08 12:42:30
 +
+

*Thread Reply:* I think this effort: https://github.com/OpenLineage/OpenLineage/pull/904 ultimately makes more sense, since it will allow getting lineage on Airflow 2.3+ too

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/pull/904|#904 [INTEGRATION][AIRFLOW] Manual inlets/outlets issue#384>
+
+ <p><strong>Problem</strong></p> + +<p>Addressing issue <a href="https://github.com/OpenLineage/OpenLineage/issues/384">#384</a>.</p> + +<p>• We want the ability to set inlets and outlets manually</p> + +<p><strong>Solution</strong></p> + +<p>• expand the <code>postgres_extractor</code> to except <code>_PythonDecoratedOperator</code>. +• add env check to attach <code>inputs</code> and <code>outputs</code> to <code>TaskMetadata</code> when <code>OPENLINEAGE_COLLECT_MANUALLY</code> is set to <code>true</code>. +• added <code>extract_inlets_and_outlets_to_dataset</code> function. +Your change modifies the <a href="https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.json">core</a> OpenLineage model +Your change modifies one or more OpenLineage <a href="https://github.com/OpenLineage/OpenLineage/tree/main/spec/facets">facets</a></p> + +<p>If you're contributing a new integration, please specify the scope of the integration and how/where it has been tested (e.g., Apache Spark integration supports <code>S3</code> and <code>GCS</code> filesystem operations, tested with AWS EMR).</p> + +<p><strong>Checklist</strong></p> + +<p>• [ x] You've <a href="https://github.com/OpenLineage/OpenLineage/blob/main/why-the-dco.md">signed-off</a> your work +• [x ] Your pull request title follows our <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CONTRIBUTING.md#creating-pull-requests">guidelines</a> +☐ Your changes are accompanied by tests (<em>if relevant</em>) +☐ Your change contains a <a href="https://kurtisnusbaum.medium.com/stacked-diffs-keeping-phabricator-diffs-small-d9964f4dcfa6">small diff</a> and is self-contained +☐ You've updated any relevant documentation (<em>if relevant</em>) +☐ You've updated the <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md"><code>CHANGELOG.md</code></a> with details about your change under the "Unreleased" section (<em>if relevant, depending on the change, this may not be necessary</em>) +☐ You've versioned the core OpenLineage model or facets according to <a href="https://docs.snowplowanalytics.com/docs/pipeline-components-and-applications/iglu/common-architecture/schemaver">SchemaVer</a> (<em>if relevant</em>)</p> +
+ +
+
Comments
+ 2 +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+ ✅ Fenil Doshi +
+ +
+ 👀 Fenil Doshi +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Fenil Doshi + (fdoshi@salesforce.com) +
+
2022-07-08 18:12:47
 +
+

*Thread Reply:* I have made the changes in-line to the mentioned comments here. +Does this look good?

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/pull/914|#914 [INTEGRATION][AIRFLOW] Allow lineage metadata to flow through inlets and outlets>
+
+ <p>Signed-off-by: Fenil Doshi <a href="mailto:fdoshi@slack-corp.com">fdoshi@slack-corp.com</a></p> + +<p><strong>Problem</strong></p> + +<p>Currently, in Airflow integration, lineage is only captured via a few extractors which greatly limits its use case. Airflow allows users to manually annotate lineage for the tasks using inlets and outlets. These can be a great fallback mechanism when extractors cannot automatically extract the input and output Datasets. <br /> +The PR closes <a href="https://github.com/OpenLineage/OpenLineage/issues/384">#384</a></p> + +<p>Addresses: <a href="https://github.com/OpenLineage/OpenLineage/issues/384">#384</a></p> + +<p><strong>Solution</strong></p> + +<p>Airflow sends inlets and outlets information to <code>LineageBackend</code> class. Inlets and Outlets can be one of <a href="https://github.com/apache/airflow/blob/main/airflow/lineage/entities.py">these</a> entities. The code only currently supports <code>Table</code> entity because OpenLineage does not track column-level lineage yet.</p> + +<p>• Check the TaskMetadata object that is created after corresponding extractors extract the lineage data and see whether inputs and outputs exist +• Check whether inlets and outlets are provided for the task +• If TaskMetadata object does not have inputs/outputs, use Airflow's inlets and outlets. +• Get all the Table entities and convert them to Dataset entities that are suitable for TaskMetaData object +Your change modifies the <a href="https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.json">core</a> OpenLineage model +Your change modifies one or more OpenLineage <a href="https://github.com/OpenLineage/OpenLineage/tree/main/spec/facets">facets</a></p> + +<p><strong>Checklist</strong></p> + +<p>☑︎ You've <a href="https://github.com/OpenLineage/OpenLineage/blob/main/why-the-dco.md">signed-off</a> your work +☑︎ Your pull request title follows our <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CONTRIBUTING.md#creating-pull-requests">guidelines</a> +☐ Your changes are accompanied by tests (<em>if relevant</em>) +☑︎ Your change contains a <a href="https://kurtisnusbaum.medium.com/stacked-diffs-keeping-phabricator-diffs-small-d9964f4dcfa6">small diff</a> and is self-contained +☐ You've updated any relevant documentation (<em>if relevant</em>) +☐ You've updated the <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md"><code>CHANGELOG.md</code></a> with details about your change under the "Unreleased" section (<em>if relevant, depending on the change, this may not be necessary</em>) +☐ You've versioned the core OpenLineage model or facets according to <a href="https://docs.snowplowanalytics.com/docs/pipeline-components-and-applications/iglu/common-architecture/schemaver">SchemaVer</a> (<em>if relevant</em>)</p> +
+ +
+
Comments
+ 1 +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-07-12 09:35:22
 +
+

*Thread Reply:* I think it looks good! Would be great to have tests for this feature though.

+ + + +
+ 👍 Fenil Doshi, Julien Le Dem +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Fenil Doshi + (fdoshi@salesforce.com) +
+
2022-07-15 21:56:50
 +
+

*Thread Reply:* I have added the tests! Would really appreciate it if someone can take a look and let me know if anything else needs to be done. +Thank you for the support! 😄

+ + + +
+ 👀 Willy Lulciuc, Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-07-18 06:48:03
 +
+

*Thread Reply:* One change and I think it will be good for now.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-07-18 06:48:07
 +
+

*Thread Reply:* Have you tested it manually?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Fenil Doshi + (fdoshi@salesforce.com) +
+
2022-07-20 13:22:04
 +
+

*Thread Reply:* Thanks a lot for the review! Appreciate it 🙌 +Yes, I tested it manually (for Airflow versions 2.1.4 and 2.3.3) and it works 🎉

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Conor Beverland + (conorbev@gmail.com) +
+
2022-07-20 13:24:55
 +
+

*Thread Reply:* I think this is such a useful feature to have, thank you! Would you mind adding a little example to the PR of how to use it? Like a little example DAG or something? ( either in a comment or edit the PR description )

+ + + +
+ 👍 Fenil Doshi +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Fenil Doshi + (fdoshi@salesforce.com) +
+
2022-07-20 15:20:32
 +
+

*Thread Reply:* Yes, Sure! I will add it in the PR description

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-07-21 05:30:56
 +
+

*Thread Reply:* I think it would be easy to convert to integration test then if you provided example dag

+ + + +
+ 👍 Fenil Doshi +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Conor Beverland + (conorbev@gmail.com) +
+
2022-07-27 12:20:43
 +
+

*Thread Reply:* ping @Fenil Doshi if possible I would really love to see the example DAG on there 🙂 🙏

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Fenil Doshi + (fdoshi@salesforce.com) +
+
2022-07-27 12:26:22
 +
+

*Thread Reply:* Yes, I was going to but the PR got merged so did not update the description. Should I just update the description of merged PR? Or should I add it somewhere in the docs?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Conor Beverland + (conorbev@gmail.com) +
+
2022-07-27 12:42:29
 +
+

*Thread Reply:* ^ @Ross Turk is it easy for @Fenil Doshi to contribute doc for manual inlet definition on the new doc site?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-07-27 12:48:32
 +
+

*Thread Reply:* It is easy 🙂 it's just markdown: https://github.com/openlineage/docs/

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-07-27 12:49:23
 +
+

*Thread Reply:* @Fenil Doshi feel free to create new page here and don't sweat where to put it, we'll still figuring the structure of it out and will move it then

+ + + +
+ 👍 Ross Turk, Fenil Doshi +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-07-27 13:12:31
 +
+

*Thread Reply:* exactly, yes - don’t be worried about the doc quality right now, the doc site is still in a pre-release state. so whatever you write will be likely edited or moved before it becomes official 👍

+ + + +
+ 👍 Fenil Doshi +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Fenil Doshi + (fdoshi@salesforce.com) +
+
2022-07-27 20:37:34
 +
+

*Thread Reply:* I added documentations here - https://github.com/OpenLineage/docs/pull/16

+ +

Also, have added an example for it. 🙂 +Let me know if something is unclear and needs to be updated.

+
+ + + + +
<https://github.com/OpenLineage/docs/pull/16|#16 Add documentation for openlineage-airflow integration via inlets-outlets>
+
+ <p>Add documentation and example for making changes to the Airflow task via inlets and outlets in order to facilitate openlineage to collect lineage metadata.</p> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/docs">OpenLineage/docs</a> +
+ + +
+ + + +
+ ✅ Conor Beverland +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Conor Beverland + (conorbev@gmail.com) +
+
2022-07-28 12:50:54
 +
+

*Thread Reply:* Thanks! very cool.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Conor Beverland + (conorbev@gmail.com) +
+
2022-07-28 12:52:22
 +
+

*Thread Reply:* Does Airflow check the types of the inlets/outlets btw?

+ +

Like I wonder if a user could directly define an OpenLineage DataSet ( which might even have various other facets included on it ) and specify it in the inlets/outlets ?

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/client/python/openlineage/client/run.py | run.py>
+
+ <pre><code>class Dataset(RedactMixin): +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-07-28 12:54:56
 +
+

*Thread Reply:* Yeah, I was also curious about using the models from airflow.lineage.entities as opposed to openlineage.client.run.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-07-28 12:55:42
 +
+

*Thread Reply:* I am accustomed to creating OpenLineage entities like this:

+ +

taxes = Dataset(namespace="<postgres://foobar>", name="schema.table")

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-07-28 12:56:45
 +
+

*Thread Reply:* I don’t dislike the airflow.lineage.entities models especially, but if we only support one of them…

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Conor Beverland + (conorbev@gmail.com) +
+
2022-07-28 12:58:18
 +
+

*Thread Reply:* yeah, if Airflow allows that class within inlets/outlets it'd be nice to support both imo.

+ +

Like we would suggest users to use openlineage.client.run.Dataset but if a user already has DAGs that use Table then they'd still work in a best efforts way.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-07-28 13:03:07
 +
+

*Thread Reply:* either Airflow depends on OpenLineage or we can probably change those entities as part of AIP-48 overhaul to more openlineage-like ones

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-07-28 17:18:35
 +
+

*Thread Reply:* hm, not sure I understand the dependency issue. isn’t this extractor living in openlineage-airflow?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Conor Beverland + (conorbev@gmail.com) +
+
2022-08-15 09:49:02
 +
+

*Thread Reply:* I gave manual lineage a try with native OL Datasets specified in the Airflow inlets/outlets and it seems to work! Had to make some small tweaks which I have attempted here: https://github.com/OpenLineage/OpenLineage/pull/1015

+ +

( I left the support for converting the Airflow Table to Dataset because I think that's nice to have also )

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/pull/1015|#1015 [Integration][Airflow] Support OL Datasets in manual lineage inputs/outputs>
+
+ <p>Signed-off-by: Conor Beverland <a href="mailto:conorbev@gmail.com">conorbev@gmail.com</a></p> + +<p><strong>Problem</strong></p> + +<p>The current support for manual lineage definition requires a user to create an Airflow <code>airflow.lineage.entities.Table</code> ( which is then converted to an OpenLineage Dataset ).</p> + +<p>It would be good if users could create OpenLineage Dataset classes directly in their DAGs with no special conversion necessary.</p> + +<p><strong>Solution</strong></p> + +<p>This extends the current implementation to simply pass through Datasets which are specified in inlets or outlets without modification.</p> + +<p>In addition it makes the BashOperatorExtractor behave more similarly to the PythonOperator when source code collection is disabled which allows it to work with the manual lineage collection feature.</p> + +<p>☐ Your change modifies the <a href="https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.json">core</a> OpenLineage model +☐ Your change modifies one or more OpenLineage <a href="https://github.com/OpenLineage/OpenLineage/tree/main/spec/facets">facets</a></p> + +<p><strong>Checklist</strong></p> + +<p>• [ x] You've <a href="https://github.com/OpenLineage/OpenLineage/blob/main/why-the-dco.md">signed-off</a> your work +• [ x] Your pull request title follows our <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CONTRIBUTING.md#creating-pull-requests">guidelines</a> +• [x ] Your changes are accompanied by tests (<em>if relevant</em>) +• [x ] Your change contains a <a href="https://kurtisnusbaum.medium.com/stacked-diffs-keeping-phabricator-diffs-small-d9964f4dcfa6">small diff</a> and is self-contained</p> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mike brenes + (brenesmi@gmail.com) +
+
2022-06-28 18:44:24
 +
+

food_delivery example example.etl_categories node

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mike brenes + (brenesmi@gmail.com) +
+
2022-06-28 18:44:40
 +
+

how do I recreate that using Openlineage?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2022-06-28 18:45:52
 +
+

*Thread Reply:* Ahh great question! I actually just updated the seeding cmd for Marquez to do just this (but in java of course)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2022-06-28 18:46:15
 +
+

*Thread Reply:* Give me a sec to send you over the diff…

+ + + +
+ ❤️ Mike brenes +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2022-06-28 18:56:35
 +
+

*Thread Reply:* … continued here https://openlineage.slack.com/archives/C01CK9T7HKR/p1656456734272809?thread_ts=1656456141.097229&cid=C01CK9T7HKR

+
+ + +
+ + + } + + Willy Lulciuc + (https://openlineage.slack.com/team/U01DCMDFHBK) +
+ + + +
+
+ <p>Have you looked over our <a href="https://github.com/OpenLineage/workshops">workshops</a> and <a href="https://github.com/OpenLineage/workshops/blob/main/airflow/e2-lineage-api/python/generate-events.py">this</a> example? (assuming you’re using python?)</p> +
+ + + + + + +
Original URL: https://openlineage.slack.com/archives/C01CK9T7HKR/p1656456734272809?thread_ts=1656456141.097229&amp;cid=C01CK9T7HKR
+ + +
+ + Thread in Slack Conversation +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Conor Beverland + (conorbev@gmail.com) +
+
2022-06-28 20:05:33
 +
+

I'm very new to DBT but wanted to give it a try with OL. I had a couple of questions when going through the DBT tutorial here: https://docs.getdbt.com/guides/getting-started/learning-more/getting-started-dbt-core

+ +
  1. An earlier part of the tutorial has you build a model in a single sql file: https://docs.getdbt.com/guides/getting-started/learning-more/getting-started-dbt-core#build-your-first-model When I did this and ran dbt-ol I got a lineage graph like this:
    2. +
+ + + +
+ 👀 Maciej Obuchowski +
+ +
+
+
+
+ + + + + + + +
+
+ + + + +
+ +
Conor Beverland + (conorbev@gmail.com) +
+
2022-06-28 20:07:11
 +
+

then a later part of the tutorial has you split that same example into multiple models and when I run it again I get the graph like:

+ + + +
+
+
+
+ + + + + + + +
+
+ + + + +
+ +
Conor Beverland + (conorbev@gmail.com) +
+
2022-06-28 20:08:54
 +
+

^ I'm just kind of curious if it's working as expected? And/or could it be possible to parse the DBT .sql so that the lineage in the first case would still show those staging tables?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-06-29 10:04:14
 +
+

*Thread Reply:* I think you should declare those as sources? Or do you need something different?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Conor Beverland + (conorbev@gmail.com) +
+
2022-06-29 21:15:33
 +
+

*Thread Reply:* I'll try to experiment with this.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Conor Beverland + (conorbev@gmail.com) +
+
2022-06-28 20:09:19
 +
+
  1. I see that DBT has a concept of adding tests to your models. Could those add data quality facets in OL ?
    2. +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-06-29 10:02:17
 +
+

*Thread Reply:* this should already be working if you run dbt-ol test or dbt-ol build

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Conor Beverland + (conorbev@gmail.com) +
+
2022-06-29 21:15:25
 +
+

*Thread Reply:* oh, nice!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
shweta p + (shweta.pbs@gmail.com) +
+
2022-07-04 02:48:35
 +
+

Hi everyone, i am trying openlineage-dbt. It works perfectly on locally when i try to publish the events to Marquez...but when i run the same commands from mwaa...i dont see those events triggered..i amnt able to view any logs if there is any error. How do i debug the issue

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2022-07-06 14:26:59
 +
+

*Thread Reply:* Maybe @Maciej Obuchowski knows? You need to check, it's using the dbt-ol command and that the configuration is available. (environment variables or conf file)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-07-06 15:31:20
 +
+

*Thread Reply:* Maybe some aws networking stuff? I'm not really sure how mwaa works internally (or, at all - never used it)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-07-06 15:35:06
 +
+

*Thread Reply:* anyway, any logs/errors should be in the same space where your task logs are

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-07-06 05:32:28
 +
+

Agenda items are requested for the next OpenLineage Technical Steering Committee meeting on July 14. Reply in thread or ping me with your item(s)!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-07-06 10:21:50
 +
+

*Thread Reply:* What is the status on the Flink / Streaming decisions being made for OpenLineage / Marquez?

+ +

A few months ago, Flink was being introduced and it was said that more thought was needed around supporting streaming services in OpenLineage.

+ +

It would be very helpful to know where the community stands on how streaming data sources should work in OpenLineage.

+ + + +
+ 👍 Michael Robinson +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-07-06 11:08:01
 +
+

*Thread Reply:* @Will Johnson added your item

+ + + +
+ 👍 Will Johnson +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-07-06 10:19:44
 +
+

Request for Creating a New OpenLineage Release

+ +

Hello #general, as per the Governance guide (https://github.com/OpenLineage/OpenLineage/blob/main/GOVERNANCE.md#openlineage-project-releases), I am asking that we generate a new release based on the latest commit by @Maciej Obuchowski (c92a93cdf3df636a02984188563d019474904b2b) which fixes a critical issue running OpenLineage on Azure Databricks.

+ +

Having this release made available to the general public on Maven would allow us to enable the hundred+ users of the solution to run OpenLineage on the latest LTS versions of Databricks. In addition, it would enable the Microsoft team to integrate the amazing column level lineage feature contributed by @Paweł Leszczyński with our solution for Microsoft Purview.

+ + + +
+ 👍 Maciej Obuchowski, Jakub Dardziński, Ross Turk, Willy Lulciuc, Will Johnson, Julien Le Dem +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-07-07 10:33:41
 +
+

@channel The next OpenLineage Technical Steering Committee meeting is on Thursday, July 14 at 10 am PT. Join us on Zoom: https://bit.ly/OLzoom +All are welcome! +Agenda:

+ +
  1. Announcements/recent talks
  2. Release 0.10.0 overview
  3. Flink integration retrospective
  4. Discuss: streaming services in Flink integration
  5. Open discussion +Notes: https://bit.ly/OLwiki +Is there a topic you think the community should discuss at this or a future meeting? Reply or DM me to add items to the agenda.
    6. +
+
+
Zoom Video
+ + + +
Join our Cloud HD Video Meeting
+
+ Zoom is the leader in modern enterprise video communications, with an easy, reliable cloud platform for video and audio conferencing, chat, and webinars across mobile, desktop, and room systems. Zoom Rooms is the original software-based conference room solution used around the world in board, conference, huddle, and training rooms, as well as executive offices and classrooms. Founded in 2011, Zoom helps businesses and organizations bring their teams together in a frictionless environment to get more done. Zoom is a publicly traded company headquartered in San Jose, CA. +
+ + + + + + +
Original URL: https://bit.ly/OLzoom
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
David Cecchi + (david_cecchi@cargill.com) +
+
2022-07-11 10:30:34
 +
+

*Thread Reply:* would appreciate a TSC discussion on OL philosophy for Streaming in general and where/if it fits in the vision and strategy for OL. fully appreciate current maturity, moreso just validating how OL is being positioned from a vision perspective. as we consider aligning enterprise lineage solution around OL want to make sure we're not making bad assumptions. neat discussion might be "imagine that Confluent decided to make Stream Lineage OL compliant/capable - are we cool with that and what are the implications?".

+ + + +
+ 👍 Michael Robinson +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-07-12 12:36:17
 +
+

*Thread Reply:* @Michael Robinson could I also have a quick 5m to talk about plans for a documentation site?

+ + + +
+ 👍 Michael Robinson, Sheeri Cabral (Collibra) +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-07-12 12:46:29
 +
+

*Thread Reply:* @David Cecchi @Ross Turk Added your items to the agenda. Thanks and looking forward to the discussion!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
David Cecchi + (david_cecchi@cargill.com) +
+
2022-07-12 15:08:48
 +
+

*Thread Reply:* this is great - will keep an eye out for recording. if it got tabled due to lack of attendance will pick it up next TSC.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sheeri Cabral (Collibra) + (sheeri.cabral@collibra.com) +
+
2022-07-12 16:12:43
 +
+

*Thread Reply:* I think OpenLineage should have some representation at https://impactdatasummit.com/2022

+ +

I’m happy to help craft the abstract, look over slides, etc. (I could help present, but all I’ve done with OpenLineage is one tutorial, so I’m hardly an expert).

+ +

CfP closes 31 Aug so there’s plenty of time, but if you want a 2nd set of eyes on things, we can’t just wait until the last minute to submit 😄

+
+
impactdatasummit.com
+ + + +
IMPACT 2022 - The Data Observability Summit
+
+ Announcing Monte Carlo’s annual Data Observability event, IMPACT 2022: The Data Observability Summit. This year we will be spotlighting some of the industry’s most prominent voices, as well as the broader community of data leaders paving the way forward for reliable data. +
+ + + + + + +
Original URL: https://impactdatasummit.com/2022
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-07-07 12:04:09
 +
+

How to create custom facets without recompiling OpenLineage?

+ +

I have a customer who is interested in using OpenLineage but wants to extend the facets WITHOUT recompiling OL / maintaining a clone of OL with their changes.

+ +

Do we have any examples of how someone might create their own jar but using the OpenLineage CustomFacetBuilder and then have that jar's classes be injected into OpenLineage?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-07-07 12:04:55
 +
+

*Thread Reply:* @Michael Collado would you have any thoughts on how to extend the Facets without having to alter OpenLineage itself?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2022-07-07 15:16:45
 +
+

*Thread Reply:* This is described here. Notably: +> Custom implementations are registered by following Java's ServiceLoader conventions. A file called io.openlineage.spark.api.OpenLineageEventHandlerFactory must exist in the application or jar's META-INF/service directory. Each line of that file must be the fully qualified class name of a concrete implementation of OpenLineageEventHandlerFactory. More than one implementation can be present in a single file. This might be useful to separate extensions that are targeted toward different environments - e.g., one factory may contain Azure-specific extensions, while another factory may contain GCP extensions.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2022-07-07 15:17:55
 +
+

*Thread Reply:* This example is present in the test package - https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/app/src/tes[…]ervices/io.openlineage.spark.api.OpenLineageEventHandlerFactory

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/app/src/test/resources/META-INF/services/io.openlineage.spark.api.OpenLineageEventHandlerFactory | io.openlineage.spark.api.OpenLineageEventHandlerFactory>
+
+ <pre><code>io.openlineage.spark.agent.util.TestOpenLineageEventHandlerFactory +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-07-07 20:19:01
 +
+

*Thread Reply:* @Michael Collado you are amazing! Thank you so much for pointing me to the docs and example!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-07-07 19:27:47
 +
+

@channel @Will Johnson +OpenLineage 0.11.0 is now available! +We added: +• an HTTP option to override timeout and properly close connections in openlineage-java lib, +• dynamic mapped tasks support to the Airflow integration, +• a SqlExtractor to the Airflow integration, +• PMD to Java and Spark builds in CI. +We changed: +• when testing extractors in the Airflow integration, the extractor list length assertion is now dynamic, +• templates are rendered at the start of integration tests for the TaskListener in the Airflow integration. +Thanks to all the contributors who made this release possible! +For the bug fixes and more details, see: +Release: https://github.com/OpenLineage/OpenLineage/releases/tag/0.11.0 +Changelog: https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md +Commit history: https://github.com/OpenLineage/OpenLineage/compare/0.10.0...0.11.0 +Maven: https://oss.sonatype.org/#nexus-search;quick~openlineage +PyPI: https://pypi.org/project/openlineage-python/

+ + + +
+ 👍 Chandru TMBA, John Thomas, Maciej Obuchowski, Fenil Doshi +
+ +
+ 👏 John Thomas, Willy Lulciuc, Ricardo Gaspar +
+ +
+ 🙌 Will Johnson, Maciej Obuchowski, Sergio Sicre +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Varun Singh + (varuntestaz@outlook.com) +
+
2022-07-11 07:06:36
 +
+

Hi all, I am using openlineage-spark in my project where I lock the dependency versions in gradle.lockfile. After release 0.10.0, this is not working. Is this a known limitation of switching to splitting the integration into submodules?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-07-14 06:18:29
 +
+

*Thread Reply:* Can you expand on what's not working exactly?

+ +

This is not something we're aware of.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Varun Singh + (varuntestaz@outlook.com) +
+
2022-07-19 04:09:39
 +
+

*Thread Reply:* @Maciej Obuchowski Sure, I have my own library where I am creating a shadowJar. This includes the open lineage library into the new uber jar. This worked fine till 0.9.0 but now building the shadowJar gives this error +Could not determine the dependencies of task ':shadowJar'. +&gt; Could not resolve all dependencies for configuration ':runtimeClasspath'. + &gt; Could not find spark:app:0.10.0. + Searched in the following locations: + - <https://repo.maven.apache.org/maven2/spark/app/0.10.0/app-0.10.0.pom> + If the artifact you are trying to retrieve can be found in the repository but without metadata in 'Maven POM' format, you need to adjust the 'metadataSources { ... }' of the repository declaration. + Required by: + project : &gt; io.openlineage:openlineage_spark:0.10.0 + &gt; Could not find spark:shared:0.10.0. + Searched in the following locations: + - <https://repo.maven.apache.org/maven2/spark/shared/0.10.0/shared-0.10.0.pom> + If the artifact you are trying to retrieve can be found in the repository but without metadata in 'Maven POM' format, you need to adjust the 'metadataSources { ... }' of the repository declaration. + Required by: + project : &gt; io.openlineage:openlineage_spark:0.10.0 + &gt; Could not find spark:spark2:0.10.0. + Searched in the following locations: + - <https://repo.maven.apache.org/maven2/spark/spark2/0.10.0/spark2-0.10.0.pom> + If the artifact you are trying to retrieve can be found in the repository but without metadata in 'Maven POM' format, you need to adjust the 'metadataSources { ... }' of the repository declaration. + Required by: + project : &gt; io.openlineage:openlineage_spark:0.10.0 + &gt; Could not find spark:spark3:0.10.0. + Searched in the following locations: + - <https://repo.maven.apache.org/maven2/spark/spark3/0.10.0/spark3-0.10.0.pom> + If the artifact you are trying to retrieve can be found in the repository but without metadata in 'Maven POM' format, you need to adjust the 'metadataSources { ... }' of the repository declaration. + Required by: + project : &gt; io.openlineage:openlineage_spark:0.10.0

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-07-19 05:00:02
 +
+

*Thread Reply:* Can you try 0.11? I think we might already fixed that.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Varun Singh + (varuntestaz@outlook.com) +
+
2022-07-19 05:50:03
 +
+

*Thread Reply:* Tried with that as well. Doesn't work

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Varun Singh + (varuntestaz@outlook.com) +
+
2022-07-19 05:56:50
 +
+

*Thread Reply:* Same error with 0.11.0 as well

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-07-19 08:11:13
 +
+

*Thread Reply:* I think I see - we removed internal dependencies from maven's pom.xml but we also publish gradle metadata: https://repo1.maven.org/maven2/io/openlineage/openlineage-spark/0.11.0/openlineage-spark-0.11.0.module

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-07-19 08:11:34
 +
+

*Thread Reply:* we should remove the dependencies or disable the gradle metadata altogether, it's not required

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-07-19 08:16:18
 +
+

*Thread Reply:* @Varun Singh For now I think you can try ignoring gradle metadata: https://docs.gradle.org/current/userguide/declaring_repositories.html#sec:supported_metadata_sources

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Hanbing Wang + (doris.wang200902@gmail.com) +
+
2022-07-19 14:18:45
 +
+

*Thread Reply:* @Varun Singh did you find out how to build shadowJar successful with release 0.10.0. I can build shadowJar with 0.9.0, but not higher version. If your problem already resolved, could you share some suggestion. thanks ^^

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Varun Singh + (varuntestaz@outlook.com) +
+
2022-07-20 03:44:40
 +
+

*Thread Reply:* @Hanbing Wang I followed @Maciej Obuchowski's instructions (Thank you!) and added this to my build.gradle file: +repositories { + mavenCentral() { + metadataSources { + mavenPom() + ignoreGradleMetadataRedirection() + } + } +} +I am able to build the jar now. I am not proficient in gradle so don't know if this is the right way to do this. Please correct me if I am wrong.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Varun Singh + (varuntestaz@outlook.com) +
+
2022-07-20 05:26:04
 +
+

*Thread Reply:* Also, I am not able to see the 3rd party dependencies in the dependency lock file, but they are present in some folder inside the jar (relocated in subproject's build file). But this is a different problem ig

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Hanbing Wang + (doris.wang200902@gmail.com) +
+
2022-07-20 18:45:50
 +
+

*Thread Reply:* Thanks @Varun Singh for the very helpful info. I will also try update build.gradle and rebuild shadowJar again.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-07-13 01:10:01
 +
+

Java Question: Why Can't I Find a Class on the Class Path? / How the heck does the ClassLoader know where to find a class?

+ +

Are there any java pros that would be willing to share alternatives to searching if a given class exists or help explain what should change in the Kusto package to make it work for the behaviors as seen in Kafka and SQL DW relation visitors? +--- Details --- +@Hanna Moazam and I are trying to introduce two new Azure data sources into OpenLineage's Spark integration. The https://github.com/Azure/azure-kusto-spark package is nearly done but we're getting tripped up on some Java concepts. In order to know if we should add the KustoRelationVisitor to the input dataset visitors, we need to see if the Kusto jar is installed on the spark / databricks cluster. In this case, the com.microsoft.kusto.spark.datasource.DefaultSource is a public class but it cannot be found using the KustRelationVisitor.class.getClassLoader().loadClass("class name") methods as seen in:

+ +

• https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/shared/src/[…]nlineage/spark/agent/lifecycle/plan/SqlDWDatabricksVisitor.java +• https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/shared/src/[…]penlineage/spark/agent/lifecycle/plan/KafkaRelationVisitor.java +At first I thought it was the Azure packages but then I tried to do the same approach with a simple java library

+ +

I instantiate a spark-shell like this +spark-shell --master local[4] \ +--conf spark.driver.extraClassPath=/mnt/repos/SparkListener-Basic/lib/build/libs/custom-listener.jar \ +--conf spark.extraListeners=listener.MyListener +--jars /mnt/repos/wjtestlib/lib/build/libs/lib.jar +With lib.jar containing a class that looks like this: +```package wjtestlib;

+ +

public class WillLibrary { + public boolean someLibraryMethod() { + return true; + } +} +And the custom listener is very simple. +public class MyListener extends org.apache.spark.scheduler.SparkListener {

+ +

private static final Logger log = LoggerFactory.getLogger("MyLogger");

+ +

public MyListener() { + log.info("INITIALIZING"); + }

+ +

@Override + public void onJobStart(SparkListenerJobStart jobStart) { + log.info("MYLISTENER: ON JOB START"); + try{ + log.info("Trying wjtestlib.WillLibrary"); + MyListener.class.getClassLoader().loadClass("wjtestlib.WillLibrary"); + log.info("Got wjtestlib.WillLibrary"); + } catch(ClassNotFoundException e){ + log.info("Could not get wjtestlib.WillLibrary"); + }

+ +
try{
+  <a href="http://log.info">log.info</a>("Trying wjtestlib.WillLibrary using Class.forName");
+  Class.forName("wjtestlib.WillLibrary", false, this.getClass().getClassLoader());
+  <a href="http://log.info">log.info</a>("Got wjtestlib.WillLibrary using Class.forName");
+} catch(ClassNotFoundException e){
+  <a href="http://log.info">log.info</a>("Could not get wjtestlib.WillLibrary using Class.forName");
+}
+
+ +

} +} +And I still a result indicating it cannot find the class. +2022-07-12 23:58:22,048 INFO MyLogger: MYLISTENER: ON JOB START +2022-07-12 23:58:22,048 INFO MyLogger: Trying wjtestlib.WillLibrary +2022-07-12 23:58:22,057 INFO MyLogger: Could not get wjtestlib.WillLibrary +2022-07-12 23:58:22,058 INFO MyLogger: Trying wjtestlib.WillLibrary using Class.forName +2022-07-12 23:58:22,065 INFO MyLogger: Could not get wjtestlib.WillLibrary using Class.forName``` +Are there any java pros that would be willing to share alternatives to searching if a given class exists or help explain what should change in the Kusto package to make it work for the behaviors as seen in Kafka and SQL DW relation visitors?

+ +

Thank you for any guidance.!

+
+ + + + +
Azure/azure-kusto-spark
+
+ +
+ +
+
Stars
+ 58 +
+ +
+
Language
+ Scala +
+ + + + + + + + +
+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/shared/src/main/java/io/openlineage/spark/agent/lifecycle/plan/SqlDWDatabricksVisitor.java | SqlDWDatabricksVisitor.java>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2022-07-13 08:50:15
 +
+

*Thread Reply:* Could you unzip the created jar and verify that classes you’re trying to use are present? Perhaps there’s some relocate in shadowJar plugin, which renames the classes. Making sure the classes are present in jar good point to start.

+ +

Then you can try doing classForName just from the spark-shell without any listeners added. The classes should be available there.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-07-13 11:42:25
 +
+

*Thread Reply:* Thank you for the reply Pawel! Hanna and I just wrapped up some testing.

+ +

It looks like Databricks AND open source spark does some magic when you install a library OR use --jars on the spark-shell. In both Databricks and Apache Spark, the thread running the SparkListener cannot see the additional libraries installed unless they're on the original / main class path.

+ +

• Confirmed the uploaded jars are NOT shaded / renamed. +• The databricks class path ($CLASSPATH) is focused on /databricks/jars +• The added libraries are in /local_disk0/tmp and are not found in $CLASSPATH. +• The sparklistener only recognizes $CLASSPATH. +• Using a classloader with an object like spark does not find our installed class: spark.getClass().getClassLoader().getResource("com/microsoft/kusto/spark/datasource/KustoSourceOptions.class") +• When we use a classloader on a class we installed and imported, it DOES find the class. myImportedClass.getClass().getClassLoader().getResource("com/microsoft/kusto/spark/datasource/KustoSourceOptions.class") +@Michael Collado and @Maciej Obuchowski have you seen any challenges with using --jars on the spark-shell and detecting if the class is installed?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-07-13 12:02:05
 +
+

*Thread Reply:* We run tests using --packages for external stuff like Delta - which is the same as --jars , but getting them from maven central, not local disk, and it works, like in KafkaRelationVisitor.

+ +

What if you did it like it? By that I mean adding it to your code with compileOnly in gradle or provided in maven, compiling with it, then using static method to check if it loads?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-07-13 12:02:36
 +
+

*Thread Reply:* > • When we use a classloader on a class we installed and imported, it DOES find the class. myImportedClass.getClass().getClassLoader().getResource("com/microsoft/kusto/spark/datasource/KustoSourceOptions.class") +Isn't that this actual scenario?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-07-13 12:36:47
 +
+

*Thread Reply:* Thank you for the reply, Maciej!

+ +

I will try the compileOnly route tonight!

+ +

Re: myImportedClass.getClass().getClassLoader().getResource("com/microsoft/kusto/spark/datasource/KustoSourceOptions.class")

+ +

I failed to mention that this was only achieved in the interactive shell / Databricks notebook. It never worked inside the SparkListener UNLESS we installed the Kusto jar on the databricks class path.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2022-07-14 06:43:47
 +
+

*Thread Reply:* The difference between --jars and --packages is that for packages all transitive dependencies will be handled. But this does not seem to be the case here.

+ +

More doc can be found here: (https://spark.apache.org/docs/latest/submitting-applications.html#advanced-dependency-management)

+ +

When starting a SparkContext, all the jars available on the classpath should be listed and put into Spark logs. So that’s the place one can check if the jar is loaded or not.

+ +

If --conf spark.driver.extraClassPath is working, you can add multiple jar files there (they must be separated by commas).

+ +

Other examples of adding multiple jars to spark classpath can be found here -> https://sparkbyexamples.com/spark/add-multiple-jars-to-spark-submit-classpath/

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-07-14 11:20:02
 +
+

*Thread Reply:* @Paweł Leszczyński thank you for the reply! Hanna and I experimented with jars vs extraClassPath.

+ +

When using jars, the spark listener does NOT find the class using a classloader.

+ +

When using extraClassPath, the spark listener DOES find the class using a classloader.

+ +

When using --jars, we can see in the spark logs that after spark starts (and after the spark listener is already established?) there are Spark.AddJar commands being executed.

+ +

@Maciej Obuchowski we also experimented with doing a compileOnly on OpenLineage's spark listener, it did not change the behavior. OpenLineage still failed to identify that I had the kusto-spark-connector.

+ +

I'm going to reach out to Databricks to see if there is any guidance on letting the SparkListener be aware of classes added via their libraries / --jar method on the spark-shell.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-07-14 11:22:01
 +
+

*Thread Reply:* So, this is only relevant to Databricks now? Because I don't understand what do you do different than us with Kafka/Iceberg/Delta

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-07-14 11:22:48
 +
+

*Thread Reply:* I'm not the spark/classpath expert though - maybe @Michael Collado have something to add?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-07-14 11:24:12
 +
+

*Thread Reply:* @Maciej Obuchowski that's a super good question on Iceberg. How do you instantiate a spark job with Iceberg installed?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-07-14 11:26:04
 +
+

*Thread Reply:* It is still relevant to apache spark because I can't get OpenLineage to find the installed package UNLESS I use extraClassPath.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-07-14 11:29:13
 +
+

*Thread Reply:* Basically, by adding --packages org.apache.iceberg:iceberg_spark_runtime_3.1_2.12:0.13.0

+ +

https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/app/src/tes[…]a/io/openlineage/spark/agent/SparkContainerIntegrationTest.java

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/app/src/test/java/io/openlineage/spark/agent/SparkContainerIntegrationTest.java | SparkContainerIntegrationTest.java>
+
+ <pre><code> void testWriteIcebergTableVersion() { +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-07-14 11:29:51
 +
+

*Thread Reply:* Trying with --pacakges right now.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-07-14 11:54:37
 +
+

*Thread Reply:* Using --packages wouldn't let me find the Spark relation's default source:

+ +

Spark Shell command +spark-shell --master local[4] \ +--conf spark.driver.extraClassPath=/customListener-1.0-SNAPSHOT.jar \ +--conf spark.extraListeners=listener.MyListener \ +--jars /WillLibrary.jar \ +--packages com.microsoft.azure.kusto:kusto_spark_3.0_2.12:3.0.0 +Code inside customListener:

+ +

try{ + <a href="http://log.info">log.info</a>("Trying Kusto DefaultSource"); + MyListener.class.getClassLoader().loadClass("com.microsoft.kusto.spark.datasource.DefaultSource"); + <a href="http://log.info">log.info</a>("Got Kusto DefaultSource!!!!"); + } catch(ClassNotFoundException e){ + <a href="http://log.info">log.info</a>("Could not get Kusto DefaultSource"); + } +Logs indicating it still can't find the class when using --packages. +2022-07-14 10:47:35,997 INFO MyLogger: MYLISTENER: ON JOB START +2022-07-14 10:47:35,997 INFO MyLogger: Trying wjtestlib.WillLibrary +2022-07-14 10:47:36,000 INFO 2022-07-14 10:47:36,052 INFO MyLogger: Trying LogicalRelation +2022-07-14 10:47:36,053 INFO MyLogger: Got logical relation +2022-07-14 10:47:36,053 INFO MyLogger: Trying Kusto DefaultSource +2022-07-14 10:47:36,064 INFO MyLogger: Could not get Kusto DefaultSource +😢

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-07-14 11:59:07
 +
+

*Thread Reply:* what if you load your listener using also packages?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-07-14 12:00:38
 +
+

*Thread Reply:* That's how I'm doing it locally using spark.conf: +spark.jars.packages com.google.cloud.bigdataoss:gcs_connector:hadoop3-2.2.2,io.delta:delta_core_2.12:1.0.0,org.apache.iceberg:iceberg_spark3_runtime:0.12.1,io.openlineage:openlineage_spark:0.9.0

+ + + +
+ 👀 Will Johnson +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-07-14 12:20:47
 +
+

*Thread Reply:* @Maciej Obuchowski - You beautiful bearded man! +🙏 +2022-07-14 11:14:21,266 INFO MyLogger: Trying LogicalRelation +2022-07-14 11:14:21,266 INFO MyLogger: Got logical relation +2022-07-14 11:14:21,266 INFO MyLogger: Trying org.apache.iceberg.catalog.Catalog +2022-07-14 11:14:21,295 INFO MyLogger: Got org.apache.iceberg.catalog.Catalog!!!! +2022-07-14 11:14:21,295 INFO MyLogger: Trying Kusto DefaultSource +2022-07-14 11:14:21,361 INFO MyLogger: Got Kusto DefaultSource!!!! +I ended up setting my spark-shell like this (and used --jars for my custom spark listener since it's not on Maven).

+ +

spark-shell --master local[4] \ +--conf spark.extraListeners=listener.MyListener \ +--packages org.apache.iceberg:iceberg_spark_runtime_3.1_2.12:0.13.0,com.microsoft.azure.kusto:kusto_spark_3.0_2.12:3.0.0 \ +--jars customListener-1.0-SNAPSHOT.jar +So, now I just need to figure out how Databricks differs from this approach 😢

+ + + +
+ 😂 Maciej Obuchowski, Jakub Dardziński, Hanna Moazam +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2022-07-14 12:21:35
 +
+

*Thread Reply:* This is an annoying detail about Java ClassLoaders and the way Spark loads extra jars/packages

+ +

Remember Java's ClassLoaders are hierarchical - there are parent ClassLoaders and child ClassLoaders. Parents can't see their children's classes, but children can see their parent's classes.

+ +

When you use --spark.driver.extraClassPath , you're adding a jar to the main application ClassLoader. But when you use --jars or --packages, you're instructing the Spark application itself to load the extra jars into its own ClassLoader - a child of the main application ClassLoader that the Spark code creates and manages separately. Since your listener class is loaded by the main application ClassLoader, it can't see any classes that are loaded by the Spark child ClassLoader. Either both jars need to be on the driver classpath or both jars need to be loaded by the --jar or --packages configuration parameter

+ + + +
+ 🙌 Will Johnson, Paweł Leszczyński +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2022-07-14 12:26:15
 +
+

*Thread Reply:* In Databricks, we were not able to simply use the --packages argument to load the listener, which is why we have that init script that copies the jar into the classpath that Databricks uses for application startup (the main ClassLoader). You need to copy your visitor jar into the same location so that both jars are loaded by the same ClassLoader and can see each other

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/databricks/open-lineage-init-script.sh | open-lineage-init-script.sh>
+
+ <pre><code>cp -f $STAGE_DIR/openlineage-spark-**.jar /mnt/driver-daemon/jars || { echo "Error copying Spark Listener library file"; exit 1;} +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2022-07-14 12:29:09
 +
+

*Thread Reply:* (as an aside, this is one of the major drawbacks of the java agent approach and one reason why all the documentation recommends using the spark.jars.packages configuration parameter for loading the OL library - it guarantees that any DataSource nodes loaded by the Spark ClassLoader can be seen by the OL library and we don't have to use reflection for everything)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-07-14 12:30:25
 +
+

*Thread Reply:* @Michael Collado Thank you so much for the reply. The challenge is that Databricks has their own mechanism for installing libraries / packages.

+ +

https://docs.microsoft.com/en-us/azure/databricks/libraries/

+ +

These packages are installed on databricks AFTER spark is started and the physical files are located in a folder that is different than the main classpath.

+ +

I'm going to reach out to Databricks and see if we can get any guidance on this 😢

+
+
docs.microsoft.com
+ + + +
Libraries - Azure Databricks
+
+ Learn how to use and manage libraries in Azure Databricks. +
+ + + + + + + + +
Original URL: https://docs.microsoft.com/en-us/azure/databricks/libraries/
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-07-14 12:31:32
 +
+

*Thread Reply:* Unfortunately, I can't ask users to install their packages on Databricks in a non-standard way (e.g. via an init script) because no one will follow that recommendation.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2022-07-14 12:32:46
 +
+

*Thread Reply:* yeah, I'd prefer if we didn't need an init script to get OL on Databricks either 🤷‍♂️:skintone4:

+ + + +
+ 🤣 Will Johnson +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-07-17 01:03:02
 +
+

*Thread Reply:* Quick update: +• Turns out using a class loader from a Scala spark listener does not have this problem. +• https://stackoverflow.com/questions/7671888/scala-classloaders-confusion +• I'm trying to use URLClassLoader as recommended by a few MSFT folks and point it at the /local_disk0/tmp folder. +• https://stackoverflow.com/questions/17724481/set-classloader-different-directory +• I'm not having luck so far but hoping I can reason about it tomorrow and Monday. This is blocking us from adding additional data sources that are not pre-installed on databricks 😢

+
+
Stack Overflow
+ + + +
scala classloaders confusion
+
+ Please consider the following test program (using scala 2.9.0.1) object test { def main(args:Array[String]) = { println(ClassLoader.getSystemClassLoader.getResource("toto")) println(this. +
+ + + + + + + + +
Original URL: https://stackoverflow.com/questions/7671888/scala-classloaders-confusion
+ + + +
+
+
Stack Overflow
+ + + +
Set classloader different directory
+
+ I've searched some topics, but couldn't find the answer. What i need is just to set up an additional path for default ClassLoader. Right now i have such class: public class Loader extends ClassLo... +
+ + + + + + + + +
Original URL: https://stackoverflow.com/questions/17724481/set-classloader-different-directory
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-07-18 05:45:59
 +
+

*Thread Reply:* Can't help you now, but I'd love if you dumped the knowledge you've gained through this process into some doc on new OpenLineage doc site 🙏

+ + + +
+ 👍 Hanna Moazam +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Hanna Moazam + (hannamoazam@microsoft.com) +
+
2022-07-18 05:48:15
 +
+

*Thread Reply:* We'll definitely put all of it together as a reference for others, and hopefully have a solution by the end of it too

+ + + +
+ 🙌 Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-07-13 12:06:24
 +
+

@channel The next OpenLineage TSC meeting is tomorrow at 10 am PT! https://openlineage.slack.com/archives/C01CK9T7HKR/p1657204421157959

+
+ + +
+ + + } + + Michael Robinson + (https://openlineage.slack.com/team/U02LXF3HUN7) +
+ + + +
+
+ <p><strong>@channel</strong> The next OpenLineage Technical Steering Committee meeting is on <strong>Thursday, July 14 at 10 am PT</strong>. Join us on Zoom: <a href="https://bit.ly/OLzoom">https://bit.ly/OLzoom</a> +All are welcome! +Agenda:</p> + +<ol><li>Announcements/recent talks</li><li>Release 0.10.0 overview</li><li>Flink integration retrospective</li><li>Discuss: streaming services in Flink integration</li><li>Open discussion +Notes: <a href="https://bit.ly/OLwiki">https://bit.ly/OLwiki</a> +Is there a topic you think the community should discuss at this or a future meeting? Reply or DM me to add items to the agenda.</li> +</ol> +
+ + + + + + +
Original URL: https://openlineage.slack.com/archives/C01CK9T7HKR/p1657204421157959
+ + +
+ + Thread in Slack Conversation +
+ + +
+ + + +
+ 🙌 Willy Lulciuc, Maciej Obuchowski +
+ +
+ 💯 Willy Lulciuc, Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
David Cecchi + (david_cecchi@cargill.com) +
+
2022-07-13 16:32:12
 +
+

check this out folks - marklogic datahub flow lineage into OL/marquez with jobs and runs and more. i would guess this is a pretty narrow use case but it went together really smoothly and thought i'd share sometimes it's just cool to see what people are working on

+ +
+
+ + + + + + +
+ + +
+ 🍺 Willy Lulciuc, Conor Beverland, Maciej Obuchowski, Paweł Leszczyński +
+ +
+ ❤️ Willy Lulciuc, Conor Beverland, Julien Le Dem, Michael Robinson, Maciej Obuchowski, Minkyu Park +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2022-07-13 16:40:48
 +
+

*Thread Reply:* Soo cool, @David Cecchi 💯💯💯. I’m not familiar with marklogic, but pretty awesome ETL platform and the lineage graph looks 👌! Did you have to write any custom integration code? Or where you able to use our off the self integrations to get things working? (Also, thanks for sharing!)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
David Cecchi + (david_cecchi@cargill.com) +
+
2022-07-13 16:57:29
 +
+

*Thread Reply:* team had to write some custom stuff but it's all framework so it can be repurposed not rewritten over and over. i would see this as another "Platform" in the context of the integrations semantic OL uses, so no, we didn't start w/ an existing solution. just used internal hooks and then called lineage APIs.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2022-07-13 17:02:53
 +
+

*Thread Reply:* Ah totally make sense. Would you be open to a brief presentation and/or demo in a future OL community meeting? The community is always looking to hear how OL is used in the wild, and this seems aligned with that (assuming you can talk about the implementation at a high-level)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2022-07-13 17:05:35
 +
+

*Thread Reply:* No pressure, of course 😉

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
David Cecchi + (david_cecchi@cargill.com) +
+
2022-07-13 17:08:50
 +
+

*Thread Reply:* ha not feeling any pressure. familiar with the intentions and dynamic. let's keep that on radar - i don't keep tabs on community meetings but mid/late august would be workable. and to be clear, this is being used in the wild in a sandbox 🙂.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2022-07-13 17:12:55
 +
+

*Thread Reply:* Sounds great, and a reasonable timeline! (cc @Michael Robinson can follow up). Even if it’s in a sandbox, talking about the level of effort helps with improving our APIs or sharing with others how smooth it can be!

+ + + +
+ 👍 David Cecchi +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-07-13 17:18:27
 +
+

*Thread Reply:* chiming in as well to say this is really cool 👍

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2022-07-13 18:26:28
 +
+

*Thread Reply:* Nice! Would this become a product feature in Marklogic Data Hub?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mark Chiarelli + (mark.chiarelli@marklogic.com) +
+
2022-07-14 11:07:42
 +
+

*Thread Reply:* MarkLogic is a multi-model database and search engine. This implementation triggers off the MarkLogic Datahub Github batch records created when running the datahub flows. Just a toe in the water so far.

+
+ + + + +
MarkLogic
+
+ +
+ +
+
Location
+ San Carlos, CA USA +
+ +
+
URL
+ <http://developer.marklogic.com> +
+ +
+
Repositories
+ 23 +
+ + + + + + + + + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2022-07-14 20:31:18
 +
+

@Ross Turk, in the OL community meeting today, you presented the new doc site (awesome!) that isn’t up (yet!), but I’ve been talk with @Julien Le Dem about the usage of _producer and would like to add a section on the use / function of _producer in OL events. Feel like the new doc site would be a great place to add this! Let me know when’s a good time to start crowd sourcing content for the site

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-07-14 20:37:25
 +
+

*Thread Reply:* That sounds like a good idea to me. Be good to have some guidance on that.

+ +

The repo is open for business! Feel free to add the page where you think it fits.

+
+ + + + +
OpenLineage/docs
+
+ Documentation for OpenLineage +
+ +
+
Website
+ <https://docs.openlineage.io> +
+ +
+
Stars
+ 1 +
+ + + + + + + + +
+ + + +
+ ❤️ Willy Lulciuc +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2022-07-14 20:42:09
 +
+

*Thread Reply:* OK! Let’s do this!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2022-07-14 20:59:36
 +
+

*Thread Reply:* @Ross Turk, feel free to assign to me https://github.com/OpenLineage/docs/issues/1!

+
+ + + + +
<https://github.com/OpenLineage/docs/issues/1|#1 Doc usage of `_producer` and `_schemaURL`>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/docs">OpenLineage/docs</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-07-14 20:39:26
 +
+

Hey everyone! As Willy says, there is a new documentation site for OpenLineage in the works.

+ +

It’s not quite ready to be, uh, a proper reference yet. But it’s not too far away. Help us get there by submitting issues, making page stubs, and adding sections via PR.

+ +

https://github.com/openlineage/docs/

+
+ + + + +
OpenLineage/docs
+
+ Documentation for OpenLineage +
+ +
+
Website
+ <https://docs.openlineage.io> +
+ +
+
Stars
+ 1 +
+ + + + + + + + +
+ + + +
+ 🙌 Maciej Obuchowski, Michael Robinson +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2022-07-14 20:43:09
 +
+

*Thread Reply:* Thanks, @Ross Turk for finding a home for more technical / how-to docs… long overdue 💯

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-07-14 21:22:09
 +
+

*Thread Reply:* BTW you can see the current site at http://openlineage.io/docs/ - merges to main will ship a new site.

+
+
openlineage.io
+ + + +
About OpenLineage | OpenLineage Docs
+
+ This page has not yet been written! You're welcome to contribute using the Edit link at the bottom. +
+ + + + + + +
Original URL: http://openlineage.io/docs/
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2022-07-14 21:23:32
 +
+

*Thread Reply:* great, was using <a href="http://docs.openlineage.io">docs.openlineage.io</a> … we’ll eventually want the docs to live under the docs subdomain though?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-07-14 21:25:32
 +
+

*Thread Reply:* TBH I activated GitHub Pages on the repo expecting it to live at openlineage.github.io/docs, thinking we could look at it there before it's ready to be published and linked in to the website

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-07-14 21:25:39
 +
+

*Thread Reply:* and it came live at openlineage.io/docs 😄

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2022-07-14 21:26:06
 +
+

*Thread Reply:* nice and sounds good 👍

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-07-14 21:26:31
 +
+

*Thread Reply:* still do not understand why, but I'll take it as a happy accident. we can move to docs.openlineage.io easily - just need to add the A record in the LF infra + the CNAME file in the static dir of this repo

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
shweta p + (shweta.pbs@gmail.com) +
+
2022-07-15 09:10:46
 +
+

Hi #general, how do i link the tasks of airflow which may not have any input or output datasets as they are running some conditions. the dataset is generated only on the last task

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
shweta p + (shweta.pbs@gmail.com) +
+
2022-07-15 09:11:25
 +
+

In the lineage, though there is option to link the parent , it doesnt show up the lineage of job -> job

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
shweta p + (shweta.pbs@gmail.com) +
+
2022-07-15 09:11:43
 +
+

does it need to be job -> dataset -> job only ?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-07-15 14:41:30
 +
+

*Thread Reply:* yes - openlineage is job -> dataset -> job. particularly, the model is designed to observe the movement of data

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-07-15 14:43:41
 +
+

*Thread Reply:* the spec is based around run events, which are observed states of job runs. jobs are observed to see how they affect datasets, and that relationship is what OpenLineage traces

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ilya Davidov + (idavidov@marpaihealth.com) +
+
2022-07-18 11:32:06
 +
+

👋 Hi everyone!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ilya Davidov + (idavidov@marpaihealth.com) +
+
2022-07-18 11:32:51
 +
+

i am looking for some information regarding openlineage integration with AWS Glue jobs/workflows

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ilya Davidov + (idavidov@marpaihealth.com) +
+
2022-07-18 11:33:32
 +
+

i am wondering if it possible and someone already give a try and maybe documented it?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john.thomas@astronomer.io) +
+
2022-07-18 15:16:54
 +
+

*Thread Reply:* This thread covers glue in some detail: https://openlineage.slack.com/archives/C01CK9T7HKR/p1637605977118000?threadts=1637605977.118000&cid=C01CK9T7HKR|https://openlineage.slack.com/archives/C01CK9T7HKR/p1637605977118000?threadts=1637605977.118000&cid=C01CK9T7HKR

+
+ + +
+ + + } + + Francis McGregor-Macdonald + (https://openlineage.slack.com/team/U02K353H2KF) +
+ + + +
+
+ <p>I am “successfully” exporting lineage to openlineage from AWS Glue using the listener. Only the source load is showing, not the transforms, or the sink</p> +
+ + + + + + +
Original URL: https://openlineage.slack.com/archives/C01CK9T7HKR/p1637605977118000?thread_ts=1637605977.118000&amp;cid=C01CK9T7HKR
+ + +
+ + Thread in Slack Conversation +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john.thomas@astronomer.io) +
+
2022-07-18 15:17:49
 +
+

*Thread Reply:* TL;Dr: you can use the spark integration to capture some lineage, but it's not comprehensive

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
David Cecchi + (david_cecchi@cargill.com) +
+
2022-07-18 16:29:02
 +
+

*Thread Reply:* i suspect there will be opportunities to influence AWS to be a "fast follower" if OL adoption and buy-in starts to feel authentically real in non-aws portions of the stack. i discussed OL casually with AWS analytics leadership (Rahul Pathak) last winter and he seemed curious and open to this type of idea. to be clear, ~95% chance he's forgotten that conversation now but hey it's still something.

+ + + +
+ 👍 Ross Turk +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Francis McGregor-Macdonald + (francis@mc-mac.com) +
+
2022-07-18 19:34:32
 +
+

*Thread Reply:* There are a couple of aws people here (including me) following.

+ + + +
+ 👍 David Cecchi, Ross Turk +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mikkel Kringelbach + (mikkel@theoremlp.com) +
+
2022-07-19 18:01:46
 +
+

Hi all, I have been playing around with Marquez for a hackday. I have been able to get some lineage information loaded in (using the local docker version for now). I have been trying set the location (for the link) and description information for a job (the text saying "Nothing to show here") but I haven't been able to figure out how to do this using the /lineage api. Any help would be appreciated.

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-07-19 20:11:38
 +
+

*Thread Reply:* I believe what you want is the DocumentationJobFacet. It adds a description property to a job.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-07-19 20:13:03
 +
+

*Thread Reply:* You can see a Python example here, in the Airflow integration: https://github.com/OpenLineage/OpenLineage/blob/65a5f021a1ba3035d5198e759587737a05b242e1/integration/airflow/openlineage/airflow/adapter.py#L217

+ + + +
+ :gratitude_thank_you: Mikkel Kringelbach +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-07-19 20:13:18
 +
+

*Thread Reply:* (looking for a curl example…)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mikkel Kringelbach + (mikkel@theoremlp.com) +
+
2022-07-19 20:25:49
 +
+

*Thread Reply:* I see, so there are special facet keys which will get translated into something special in the ui, is that correct?

+ +

Are these documented anywhere?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-07-19 20:27:55
 +
+

*Thread Reply:* Correct - info from the various OpenLineage facets are used in the Marquez UI.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-07-19 20:28:28
 +
+

*Thread Reply:* I couldn’t find a curl example with a description field, but I did generate this one with a sql field:

+ +

{ + "job": { + "name": "order_analysis.find_popular_products", + "facets": { + "sql": { + "query": "DROP TABLE IF EXISTS top_products;\n\nCREATE TABLE top_products AS\nSELECT\n product,\n COUNT(order_id) AS num_orders,\n SUM(quantity) AS total_quantity,\n SUM(price ** quantity) AS total_value\nFROM\n orders\nGROUP BY\n product\nORDER BY\n total_value desc,\n num_orders desc;", + "_producer": "https: //github.com/OpenLineage/OpenLineage/tree/0.11.0/integration/airflow", + "_schemaURL": "<https://raw.githubusercontent.com/OpenLineage/OpenLineage/main/spec/OpenLineage.json#/definitions/SqlJobFacet>" + } + }, + "namespace": "workshop" + }, + "run": { + "runId": "13460e52-a829-4244-8c45-587192cfa009", + "facets": {} + }, + "inputs": [ + ... + ], + "outputs": [ + ... + ], + "producer": "<https://github.com/OpenLineage/OpenLineage/tree/0.11.0/integration/airflow>", + "eventTime": "2022-07-20T00: 23: 06.986998Z", + "eventType": "COMPLETE" +}

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-07-19 20:28:58
 +
+

*Thread Reply:* The facets (at least, those in the core spec) are here: https://github.com/OpenLineage/OpenLineage/tree/65a5f021a1ba3035d5198e759587737a05b242e1/spec/facets

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-07-19 20:29:19
 +
+

*Thread Reply:* it’s designed so that facets can exist outside the core, in other repos, as well

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mikkel Kringelbach + (mikkel@theoremlp.com) +
+
2022-07-19 22:25:39
 +
+

*Thread Reply:* Thank you for sharing these, I was able to get the sql query highlighting to work. But I failed to get the location link or the documentation to work. My facet attempt looked like: +{ + "facets": { + "description": "test-description-job", + "sql": { + "query": "SELECT QUERY", + "_schema": "<https://raw.githubusercontent.com/OpenLineage/OpenLineage/main/spec/OpenLineage.json#/definitions/SqlJobFacet>" + }, + "documentation": { + "documentation": "Test docs?", + "_schema": "<https://raw.githubusercontent.com/OpenLineage/OpenLineage/main/spec/OpenLineage.json#/definitions/DocumentationJobFacet>" + }, + "link": { + "type": "", + "url": "<a href="http://www.google.com/test_url">www.google.com/test_url</a>", + "_schema": "<https://raw.githubusercontent.com/OpenLineage/OpenLineage/main/spec/OpenLineage.json#/definitions/SourceCodeLocationJobFacet>" + } + } +}

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mikkel Kringelbach + (mikkel@theoremlp.com) +
+
2022-07-19 22:36:55
 +
+

*Thread Reply:* I got the documentation link to work by renaming the property from documentation -> description . I still haven't been able to get the external link to work

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-07-20 10:33:36
 +
+

Hey all. I've been doing a cleanup of issues on GitHub. If I've closed your issue that you think is still relevant, please reopen it and let us know.

+ + + +
+ 🙌 Jakub Dardziński, Michael Collado, Will Johnson, Ross Turk +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sheeri Cabral (Collibra) + (sheeri.cabral@collibra.com) +
+
2022-07-21 16:09:08
 +
+

Is https://databricks.com/blog/2022/06/08/announcing-the-availability-of-data-lineage-with-unity-catalog.html - are they using OpenLineage? I know there’s been a lot of work to make sure OpenLineage integrates with Databricks, even earlier this year.

+
+
Databricks
+ + + +
Automated and Real-time Data Lineage With Unity Catalog
+
+ Learn more about data lineage, its organizational importance and how Unity Catalog provides automated and real-time data lineage at a granular level for all workloads (SQL, R, Python, Scala) and across all asset types (notebooks, workflows, dashboards) +
+ + + + + + + + +
Original URL: https://databricks.com/blog/2022/06/08/announcing-the-availability-of-data-lineage-with-unity-catalog.html
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-07-21 16:25:47
 +
+

*Thread Reply:* There’s a good integration between OL and Databricks for pulling metadata out of running Spark clusters. But there’s not currently a connection between OL and the Unity Catalog.

+ +

I think it would be cool to see some discussions start to develop around it 👍

+ + + +
+ 👍 Sheeri Cabral (Collibra), Julius Rentergent +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sheeri Cabral (Collibra) + (sheeri.cabral@collibra.com) +
+
2022-07-21 16:26:44
 +
+

*Thread Reply:* Absolutely. I saw some mention of APIs and access, and was wondering if maybe they used OpenLineage as a framework, which would be awesome.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sheeri Cabral (Collibra) + (sheeri.cabral@collibra.com) +
+
2022-07-21 16:30:55
 +
+

*Thread Reply:* (and since Azure Databricks uses it - https://openlineage.io/blog/openlineage-microsoft-purview/ I wasn’t sure about Unity Catalog)

+
+
openlineage.io
+ + + +
Microsoft Purview Accelerates Lineage Extraction from Azure Databricks
+
+ A new collaboration between Microsoft and OpenLineage is making lineage extraction possible for Azure Databricks and Microsoft Purview users. +
+ + + + + + +
Original URL: https://openlineage.io/blog/openlineage-microsoft-purview/
+ + + +
+ + + +
+ 👍 Will Johnson +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2022-07-21 16:56:24
 +
+

*Thread Reply:* We're in the early stages of discussion regarding an OpenLineage integration for Unity. You showing interest would help increase the priority of that on the DB side.

+ + + +
+ 👍 Sheeri Cabral (Collibra), Will Johnson, Thijs Koot +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Thijs Koot + (thijs.koot@gmail.com) +
+
2022-07-27 11:41:48
 +
+

*Thread Reply:* I'm interested in Databricks enabling an openlineage endpoint, serving as a catalogue. Similar to how they provide hosted MLFlow. I can mention this to our Databricks reps as well

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Joao Vicente + (joao.diogo.vicente@gmail.com) +
+
2022-07-23 04:09:55
 +
+

Hi all +I am trying to find the state of columnLineage in OL +I see a proposal and some examples in https://github.com/OpenLineage/OpenLineage/search?q=columnLineage&type=|https://github.com/OpenLineage/OpenLineage/search?q=columnLineage&type= but I can't find it in the spec. +Can anyone shed any light why this would be the case?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Joao Vicente + (joao.diogo.vicente@gmail.com) +
+
2022-07-23 04:12:26
 +
+

*Thread Reply:* Link to spec where I looked https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.json

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.json | OpenLineage.json>
+
+ <pre><code>{ + "$schema": "&lt;https://json-schema.org/draft/2020-12/schema&gt;", + "$id": "&lt;https://openlineage.io/spec/1-0-2/OpenLineage.json&gt;", + "$defs": { + "RunEvent": { + "type": "object", + "properties": { + "eventType": { + "description": "the current transition of the run state. It is required to issue 1 START event and 1 of [ COMPLETE, ABORT, FAIL ] event per run. Additional events with OTHER eventType can be added to the same run. For example to send additional metadata after the run is complete", + "type": "string", + "enum": [ + "START", + "COMPLETE", + "ABORT", + "FAIL", + "OTHER" + ], + "example": "START|COMPLETE|ABORT|FAIL|OTHER" + }, + "eventTime": { + "description": "the time the event occurred at", + "type": "string", + "format": "date-time" + }, + "run": { + "$ref": "#/$defs/Run" + }, + "job": { + "$ref": "#/$defs/Job" + }, + "inputs": { + "description": "The set of ****input**** datasets.", + "type": "array", + "items": { + "$ref": "#/$defs/InputDataset" + } + }, + "outputs": { + "description": "The set of ****output**** datasets.", + "type": "array", + "items": { + "$ref": "#/$defs/OutputDataset" + } + }, + "producer": { + "description": "URI identifying the producer of this metadata. For example this could be a git url with a given tag or sha", + "type": "string", + "format": "uri", + "example": "&lt;https://github.com/OpenLineage/OpenLineage/blob/v1-0-0/client&gt;" + }, + "schemaURL": { + "description": "The JSON Pointer (&lt;https://tools.ietf.org/html/rfc6901&gt;) URL to the corresponding version of the schema definition for this RunEvent", + "type": "string", + "format": "uri", + "example": "&lt;https://openlineage.io/spec/0-0-1/OpenLineage.json&gt;" + } + }, + "required": [ + "run", + "job", + "eventTime", + "producer", + "schemaURL" + ] + }, + "Run": { + "type": "object", + "properties": { + "runId": { + "description": "The globally unique ID of the run associated with the job.", + "type": "string", + "format": "uuid" + }, + "facets": { + "description": "The run facets.", + "type": "object", + "additionalProperties": { + "$ref": "#/$defs/RunFacet" + } + } + }, + "required": [ + "runId" + ] + }, + "RunFacet": { + "description": "A Run Facet", + "type": "object", + "allOf": [ + { "$ref": "#/$defs/BaseFacet" } + ] + }, + "Job": { + "type": "object", + "properties": { + "namespace": { + "description": "The namespace containing that job", + "type": "string", + "example": "my-scheduler-namespace" + }, + "name": { + "description": "The unique name for that job within that namespace", + "type": "string", + "example": "myjob.mytask" + }, + "facets": { + "description": "The job facets.", + "type": "object", + "additionalProperties": { + "$ref": "#/$defs/JobFacet" + } + } + }, + "required": [ + "namespace", + "name" + ] + }, + "JobFacet": { + "description": "A Job Facet", + "type": "object", + "allOf": [ + { "$ref": "#/$defs/BaseFacet" } + ] + }, + "InputDataset": { + "description": "An input dataset", + "type": "object", + "allOf": [ + { "$ref": "#/$defs/Dataset" }, + { + "type": "object", + "properties": { + "inputFacets": { + "description": "The input facets for this dataset.", + "type": "object", + "additionalProperties": { + "$ref": "#/$defs/InputDatasetFacet" + } + } + } + } + ] + }, + "InputDatasetFacet": { + "description": "An Input Dataset Facet", + "type": "object", + "allOf": [ + { "$ref": "#/$defs/BaseFacet" } + ] + }, + "OutputDataset": { + "description": "An output dataset", + "type": "object", + "allOf": [ + { "$ref": "#/$defs/Dataset" }, + { + "type": "object", + "properties": { + "outputFacets": { + "description": "The output facets for this dataset", + "type": "object", + "additionalProperties": { + "$ref": "#/$defs/OutputDatasetFacet" + } + } + } + } + ] + }, + "OutputDatasetFacet": { + "description": "An Output Dataset Facet", + "type": "object", + "allOf": [ + { "$ref": "#/$defs/BaseFacet" } + ] + }, + "Dataset": { + "type": "object", + "properties": { + "namespace": { + "description": "The namespace containing that dataset", + "type": "string", + "example": "my-datasource-namespace" + }, + "name": { + "description": "The unique name for that dataset within that namespace", + "type": "string", + "example": "instance.schema.table" + }, + "facets": { + "description": "The facets for this dataset", + "type": "object", + "additionalProperties": { + "$ref": "#/$defs/DatasetFacet" + } + } + }, + "required": [ + "namespace", + "name" + ] + }, + "DatasetFacet": { + "description": "A Dataset Facet", + "type": "object", + "allOf": [ + { "$ref": "#/$defs/BaseFacet" } + ] + }, + "BaseFacet": { + "description": "all fields of the base facet are prefixed with _ to avoid name conflicts in facets", + "type": "object", + "properties": { + "_producer": { + "description": "URI identifying the producer of this metadata. For example this could be a git url with a given tag or sha", + "type": "string", + "format": "uri", + "example": "&lt;https://github.com/OpenLineage/OpenLineage/blob/v1-0-0/client&gt;" + }, + "_schemaURL": { + "description": "The JSON Pointer (&lt;https://tools.ietf.org/html/rfc6901&gt;) URL to the corresponding version of the schema definition for this facet", + "type": "string", + "format": "uri", + "example": "&lt;https://openlineage.io/spec/1-0-2/OpenLineage.json#/$defs/BaseFacet&gt;" + } + }, + "additionalProperties": true, + "required": [ + "_producer", + "_schemaURL" + ] + } + }, + "$ref": "#/$defs/RunEvent" +} + +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Joao Vicente + (joao.diogo.vicente@gmail.com) +
+
2022-07-23 04:37:11
 +
+

*Thread Reply:* My bad. I realize now that column lineage has been implemented as a facet, hence not visible in the main spec https://github.com/OpenLineage/OpenLineage/search?q=ColumnLineageDatasetFacet&type=|https://github.com/OpenLineage/OpenLineage/search?q=ColumnLineageDatasetFacet&type=

+ + + +
+ 👍 Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2022-07-26 19:37:54
 +
+

*Thread Reply:* It is supported in the Spark integration

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2022-07-26 19:39:13
 +
+

*Thread Reply:* @Paweł Leszczyński could you add the Column Lineage facet here in the spec? https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.md#standard-facets

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-07-24 16:24:15
 +
+

SundayFunday

+ +

Putting together some internal training for OpenLineage and highlighting some of the areas that have been useful to me on my journey with OpenLineage. Many thanks to @Michael Collado, @Maciej Obuchowski, and @Paweł Leszczyński for the continued technical support and guidance.

+ +
+
+ + + + + + +
+ + +
+ ❤️ Hanna Moazam, Ross Turk, Minkyu Park, Atif Tahir, Paweł Leszczyński +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-07-24 16:26:59
 +
+

*Thread Reply:* @Ross Turk I still want to contribute something like this to the OpenLineage docs / new site but the bar for an internal doc is lower in my mind 😅

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-07-25 11:49:54
 +
+

*Thread Reply:* 😄

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-07-25 11:50:54
 +
+

*Thread Reply:* @Will Johnson happy to help you with docs, when the time comes! sketching outline --> editing, whatever you need

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2022-07-26 19:39:56
 +
+

*Thread Reply:* This looks nice by the way.

+ + + +
+ ❤️ Will Johnson +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sylvia Seow + (sylviaseow@gmail.com) +
+
2022-07-26 09:06:28
 +
+

hi all, really appreciate if anyone could help. I have been trying to create a poc project with openlineage with dbt. attached will be the pip list of the openlineage packages that i have. However, when i run "dbt-ol"command, it prompted as öpen as file, instead of running as a command. the regular dbt run can be executed without issue. i would want i had done wrong or if any configuration that i have missed. Thanks a lot

+ +
+
+ + + + + + +
+
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-07-26 10:39:57
 +
+

*Thread Reply:* do you have proper execute permissions?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-07-26 10:41:09
 +
+

*Thread Reply:* not sure how that works on windows, but it just looks like it does not recognize dbt-ol as executable

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sylvia Seow + (sylviaseow@gmail.com) +
+
2022-07-26 10:43:00
 +
+

*Thread Reply:* yes i have admin rights. how to make this as executable?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sylvia Seow + (sylviaseow@gmail.com) +
+
2022-07-26 10:43:25
 +
+

*Thread Reply:* btw do we have a sample docker image where dbt-ol can run?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-07-26 17:33:08
 +
+

*Thread Reply:* I have also never tried on Windows 😕 but you might try python3 dbt-ol run?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sylvia Seow + (sylviaseow@gmail.com) +
+
2022-07-26 21:03:43
 +
+

*Thread Reply:* will try that

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-07-26 16:41:04
 +
+

Running a single unit test on the Spark Integration - How it works with the different modules?

+ +

Prior to splitting up the OpenLineage spark integration, I could run a command like the one below to test a single test or even a single test method. Now I get a failure and it's pointing to the app: module. Can anyone share the right syntax for running a unit test with the current package structure? Thank you!!

+ +

```wj@DESKTOP-ECF9QME:~/repos/OpenLineageWill/integration/spark$ ./gradlew test --tests io.openlineage.spark.agent.OpenLineageSparkListenerTest

+ +

> Task :app:test FAILED

+ +

SUCCESS: Executed 0 tests in 872ms

+ +

FAILURE: Build failed with an exception.

+ +

** What went wrong: +Execution failed for task ':app:test'. +> No tests found for given includes: io.openlineage.spark.agent.OpenLineageSparkListenerTest

+ +

** Try: +> Run with --stacktrace option to get the stack trace. +> Run with --info or --debug option to get more log output. +> Run with --scan to get full insights.

+ +

** Get more help at https://help.gradle.org

+ +

Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.

+ +

You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.

+ +

See https://docs.gradle.org/7.4/userguide/command_line_interface.html#sec:command_line_warnings

+ +

BUILD FAILED in 2s +18 actionable tasks: 4 executed, 14 up-to-date```

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2022-07-27 01:54:31
 +
+

*Thread Reply:* This may be a result of splitting Spark integration into multiple submodules: app, shared, spark2, spark3, spark32, etc. If the test case is from shared submodule (this one looks like that), you could try running: +./gradlew :shared:test --tests io.openlineage.spark.agent.OpenLineageSparkListenerTest

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Hanna Moazam + (hannamoazam@microsoft.com) +
+
2022-07-27 03:18:42
 +
+

*Thread Reply:* @Paweł Leszczyński, I tried running that command, and I get the following error:

+ +

```> Task :shared:test FAILED

+ +

FAILURE: Build failed with an exception.

+ +

** What went wrong: +Execution failed for task ':shared:test'. +> No tests found for given includes: io.openlineage.spark.agent.OpenLineageSparkListenerTest

+ +

** Try: +> Run with --stacktrace option to get the stack trace. +> Run with --info or --debug option to get more log output. +> Run with --scan to get full insights.

+ +

** Get more help at https://help.gradle.org

+ +

Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.

+ +

You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.

+ +

See https://docs.gradle.org/7.4/userguide/command_line_interface.html#sec:command_line_warnings

+ +

BUILD FAILED in 971ms +6 actionable tasks: 2 executed, 4 up-to-date```

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Hanna Moazam + (hannamoazam@microsoft.com) +
+
2022-07-27 03:24:41
 +
+

*Thread Reply:* When running build and test for all the submodules, I can see outputs for tests in different submodules (spark3, spark2 etc), but for some reason, I cannot find any indication that the tests in +OpenLineage/integration/spark/app/src/test/java/io/openlineage/spark/agent/lifecycle/plan +are being run at all.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2022-07-27 03:42:43
 +
+

*Thread Reply:* That’s interesting. Let’s ask @Tomasz Nazarewicz about that.

+ + + +
+ 👍 Hanna Moazam +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Hanna Moazam + (hannamoazam@microsoft.com) +
+
2022-07-27 03:57:08
 +
+

*Thread Reply:* For reference, I attached the stdout and stderr messages from running the following: +./gradlew :shared:spotlessApply &amp;&amp; ./gradlew :app:spotlessApply &amp;&amp; ./gradlew clean build test

+ +
+
testOutput.txt
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tomasz Nazarewicz + (tomasz.nazarewicz@getindata.com) +
+
2022-07-27 04:27:23
 +
+

*Thread Reply:* I'll look into it

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tomasz Nazarewicz + (tomasz.nazarewicz@getindata.com) +
+
2022-07-28 05:17:36
 +
+

*Thread Reply:* Update: some test appeared to not be visible after split, that's fixed but now I have to solevr some dependency issues

+ + + +
+ 🙌 Hanna Moazam, Will Johnson +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Hanna Moazam + (hannamoazam@microsoft.com) +
+
2022-07-28 05:19:16
 +
+

*Thread Reply:* That's great, thank you!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Hanna Moazam + (hannamoazam@microsoft.com) +
+
2022-07-29 06:05:55
 +
+

*Thread Reply:* Hi Tomasz, thanks so much for looking into this. Is this your PR (https://github.com/OpenLineage/OpenLineage/pull/953) that fixes the whole issue, or is there still some work to do to solve the dependency issues you mentioned?

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/pull/953|#953 making OpenLineageSparkListenerTest.java visible for tests and fixing them>
+
+ <p>… them</p> + +<p>Signed-off-by: tomasznazarewicz <a href="mailto:t.nazarewicz94@gmail.com">t.nazarewicz94@gmail.com</a></p> + +<p><strong>Problem</strong></p> + +<p><a href="https://openlineage.slack.com/archives/C01CK9T7HKR/p1658868064009839">https://openlineage.slack.com/archives/C01CK9T7HKR/p1658868064009839</a></p> + +<p>io.openlineage.spark.agent.OpenLineageSparkListenerTest not visible for test</p> + +<p>also one test is failing when its run</p> + +<p>Closes:</p> + +<p><strong>Solution</strong></p> + +<p>build script for <code>app</code> module changed so the tests are visible, failing test fixed</p> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tomasz Nazarewicz + (tomasz.nazarewicz@getindata.com) +
+
2022-07-29 06:07:58
 +
+

*Thread Reply:* I'm still testing it, should've changed it to draft, sorry

+ + + +
+ 👍 Hanna Moazam, Will Johnson +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Hanna Moazam + (hannamoazam@microsoft.com) +
+
2022-07-29 06:08:59
 +
+

*Thread Reply:* No worries! If I can help with testing or anything please let me know!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tomasz Nazarewicz + (tomasz.nazarewicz@getindata.com) +
+
2022-07-29 06:09:29
 +
+

*Thread Reply:* Will do! Thanks :)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Hanna Moazam + (hannamoazam@microsoft.com) +
+
2022-08-02 11:06:31
 +
+

*Thread Reply:* Hi @Tomasz Nazarewicz, if possible, could you please share an estimated timeline for resolving the issue? We have 3 PRs which we are either waiting to open or to update which are dependent on the tests.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tomasz Nazarewicz + (tomasz.nazarewicz@getindata.com) +
+
2022-08-02 13:45:34
 +
+

*Thread Reply:* @Hanna Moazam hi, it's quite difficult to do that because the issue is that all the tests are passing when I execute ./gradlew app:test +but one is failing with ./gradlew app:build

+ +

but if it fixes your problem I can disable this test for now and make a PR without it, then you can maybe unblock your stuff and I will have more time to investigate the issue.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Hanna Moazam + (hannamoazam@microsoft.com) +
+
2022-08-02 14:54:45
 +
+

*Thread Reply:* Oh that's a strange issue. Yes that would be really helpful if you can, because we have some tests we implemented which we need to make sure pass as expected.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Hanna Moazam + (hannamoazam@microsoft.com) +
+
2022-08-02 14:54:52
 +
+

*Thread Reply:* Thank you for your help Tomasz!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tomasz Nazarewicz + (tomasz.nazarewicz@getindata.com) +
+
2022-08-03 06:12:07
 +
+

*Thread Reply:* @Hanna Moazam https://github.com/OpenLineage/OpenLineage/pull/980 here is the pull request with the changes

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/pull/980|#980 enable tests for app, fix bugs, disable LibraryTest.testRdd>
+
+ <p>Signed-off-by: tomasznazarewicz <a href="mailto:t.nazarewicz94@gmail.com">t.nazarewicz94@gmail.com</a></p> + +<p><strong>Problem</strong></p> + +<p><code>app</code> module was only running integration tests</p> + +<p>Original solution assumed that the test will be enabled and all potential bugs fixed. That worked for all tests except <code>LibraryTest.testRdd</code>, which has two problems:</p> + +<ol><li><code>java.lang.NoSuchMethodError: io.openlineage.client.OpenLineageClientUtils.newObjectMapper()Lcom/fasterxml/jackson/databind/ObjectMapper;</code></li><li>when ObjectMapper was changed there is issue with serialization, if test is run with <code>./gradlew app:test</code> the tests are passing but if it's run with <code>./gradlew app:build</code> the <code>OpenLineage.RunFacets.additionalProperties</code> is serialized as additional field and test is failing</li> +</ol> + +<p>EDIT: <br /> +It turns out there are also issues with Delta.</p> + +<p>Closes: potentially -&gt; <a href="https://openlineage.slack.com/archives/C01CK9T7HKR/p1658868064009839">https://openlineage.slack.com/archives/C01CK9T7HKR/p1658868064009839</a></p> + +<p><strong>Solution</strong></p> + +<p>It seems to be a blocking issue so the proposed solution for now is to disable <code>LibraryTest.testRdd</code> and <code>DeltaDataSourceTest.testInsertIntoDeltaSource</code> so everything is passing and in the meantime investigate both issues</p> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+ 🙌 Hanna Moazam +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tomasz Nazarewicz + (tomasz.nazarewicz@getindata.com) +
+
2022-08-03 06:12:26
 +
+

*Thread Reply:* its waiting for review currently

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Hanna Moazam + (hannamoazam@microsoft.com) +
+
2022-08-03 06:20:41
 +
+

*Thread Reply:* Thank you!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Conor Beverland + (conorbev@gmail.com) +
+
2022-07-26 18:44:47
 +
+

Is there any doc yet about column level lineage? I see a spec for the facet here: https://github.com/openlineage/openlineage/issues/148

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/issues/148|#148 [PROPOSAL] column level lineage facet>
+
+ <p><strong>Purpose:</strong> <br /> +For transformations like SQL queries (but also in other cases), we can extract column level lineage. This allows answering questions like ‘which root input columns are used to construct column x?’. We need to be able to capture this information in the OpenLineage model</p> + +<p><strong>Proposed implementation</strong> <br /> +We propose to introduce a new dataset facet: <br /> +example:</p> + +<pre><code>{ + "eventType": "COMPLETE", + "eventTime": "2020-12-28T20:52:00.001+10:00", + "run" : { + "runId": "uuid" + }, + "job": { + "namespace": "scheduler", + "name": "myjob", + "facets": { + "sql": { + "query": "Insert into outputTable from select ** from inputTable" + } + } + }, + "inputs": [ + { + "namespace": "N1", + "name": "inputTable", + "facets": { + "schema": { + "fields": [ {"name": "col_a", "type": "VARCHAR"}, {"name": "col_b", "type": "int"}] + } + } + } + ], + "outputs": [ + { + "namespace": "N2", + "name": "outputTable", + "facets": { + "schema": { + "fields": [ {"name": "col_a", "type": "VARCHAR"}, {"name": "col_b", "type": "int"}] + }, + "columnLineage": { + "fields": { + "col_a": [ { "namespace": "N1", "name": "inputTable", "field": "col_a"} ], + "col_b": [ { "namespace": "N1", "name": "inputTable", "field": "col_b" } ] + } + } + } + } + ] +} +</code></pre> + +<p>Schema:</p> + +<pre><code> "columnLineage": { + "type": "object", + "properties": { + "fields": { + "type": "array", + "items": { + "type": "object", + "additionalProperties": { + "type": "object", + "properties": { + "namespace": { + "type": "string", + "description": "the input dataset namespace" + }, + "name": { + "type": "string", + "description": "the input dataset name" + }, + "field": { + "type": "string", + "description": "the input field" + } + }, + "required": [ + "namespace", "name" + ] + } + } + } + } +</code></pre> + +<p>References:</p> + +<p>• Datahub spec: <a href="https://datahubproject.io/docs/rfc/active/1841-lineage/field_level_lineage/">https://datahubproject.io/docs/rfc/active/1841-lineage/field<em>level</em>lineage/</a></p> +
+ +
+
Assignees
+ <a href="https://github.com/pawel-big-lebowski">@pawel-big-lebowski</a> +
+ +
+
Labels
+ proposal +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2022-07-26 19:41:13
 +
+

*Thread Reply:* The doc site would benefit from a page about it. Maybe @Paweł Leszczyński?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2022-07-27 01:59:27
 +
+

*Thread Reply:* Sure, it’s already on my list, will do

+ + + +
+ :gratitude_thank_you: Julien Le Dem +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2022-07-29 07:55:40
 +
+

*Thread Reply:* https://openlineage.io/docs/integrations/spark/spark_column_lineage

+
+
openlineage.io
+ + + +
Column Level Lineage | OpenLineage Docs
+
+ Column level lineage provides fine grained information on datasets' dependencies. Not only we know the dependency exist, but we are also able to understand which input columns are used to produce output columns. This allows answering questions like Which root input columns are used to construct column x? +
+ + + + + + +
Original URL: https://openlineage.io/docs/integrations/spark/spark_column_lineage
+ + + +
+ + + +
+ ✅ Conor Beverland +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Conor Beverland + (conorbev@gmail.com) +
+
2022-07-26 20:03:55
 +
+

maybe another question for @Paweł Leszczyński: I was watching the Airflow summit talk that you and @Maciej Obuchowski did ( very nice! ). How is this exposed? I'm wondering if it shows up as an edge on the graph in Marquez? ( I guess it may be tracked as a parent run and if so probably does not show on the graph directly at this time? )

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2022-07-27 04:08:18
 +
+

*Thread Reply:* To be honest, I have never seen that in action and would love to have that in our documentation.

+ +

@Michael Collado or @Maciej Obuchowski: are you able to create some doc? I think one of you was working on that.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-07-27 04:24:19
 +
+

*Thread Reply:* Yes, parent run

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
shweta p + (shweta.pbs@gmail.com) +
+
2022-07-27 01:29:05
 +
+

Hi #general, there has been a issue with airflow+dbt+openlineage. This was working fine with openlineage-dbt v0.11.0 but there has been some change to the typeextensions due to which i had to upgrade to latest dbt (from 1.0.0 to 1.1.0) and now the dbt-ol is failing with schema version support (the version generated is v5 vs dbt-ol supports only v4). Has anyone else been able to fix this

+ + + +
+ 👀 Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-07-27 04:47:18
 +
+

*Thread Reply:* Will take a look

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-07-27 04:47:40
 +
+

*Thread Reply:* But generally this support message is just a warning

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-07-27 10:04:20
 +
+

*Thread Reply:* @shweta p any actual error you've found? +I've tested it with dbt-bigquery on 1.1.0 and it works despite warning:

+ +

➜ small OPENLINEAGE_URL=<http://localhost:5050> dbt-ol build +Running OpenLineage dbt wrapper version 0.11.0 +This wrapper will send OpenLineage events at the end of dbt execution. +14:03:16 Running with dbt=1.1.0 +14:03:17 Found 2 models, 3 tests, 0 snapshots, 0 analyses, 191 macros, 0 operations, 0 seed files, 0 sources, 0 exposures, 0 metrics +14:03:17 +14:03:17 Concurrency: 2 threads (target='dev') +14:03:17 +14:03:17 1 of 5 START table model dbt_test1.my_first_dbt_model .......................... [RUN] +14:03:21 1 of 5 OK created table model dbt_test1.my_first_dbt_model ..................... [CREATE TABLE (2.0 rows, 0 processed) in 3.31s] +14:03:21 2 of 5 START test unique_my_first_dbt_model_id ................................. [RUN] +14:03:22 2 of 5 PASS unique_my_first_dbt_model_id ....................................... [PASS in 1.55s] +14:03:22 3 of 5 START view model dbt_test1.my_second_dbt_model .......................... [RUN] +14:03:24 3 of 5 OK created view model dbt_test1.my_second_dbt_model ..................... [OK in 1.38s] +14:03:24 4 of 5 START test not_null_my_second_dbt_model_id .............................. [RUN] +14:03:24 5 of 5 START test unique_my_second_dbt_model_id ................................ [RUN] +14:03:25 5 of 5 PASS unique_my_second_dbt_model_id ...................................... [PASS in 1.38s] +14:03:25 4 of 5 PASS not_null_my_second_dbt_model_id .................................... [PASS in 1.42s] +14:03:25 +14:03:25 Finished running 1 table model, 3 tests, 1 view model in 8.44s. +14:03:25 +14:03:25 Completed successfully +14:03:25 +14:03:25 Done. PASS=5 WARN=0 ERROR=0 SKIP=0 TOTAL=5 +Artifact schema version: <https://schemas.getdbt.com/dbt/manifest/v5.json> is above dbt-ol supported version 4. This might cause errors. +Emitting OpenLineage events: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:00&lt;00:00, 274.42it/s] +Emitted 10 openlineage events

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Fenil Doshi + (fdoshi@salesforce.com) +
+
2022-07-27 20:39:21
 +
+

When will the next version of OpenLineage be available tentatively?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-07-27 20:41:44
 +
+

*Thread Reply:* I think it's safe to say we'll see a release by the end of next week

+ + + +
+ :gratitude_thank_you: Fenil Doshi +
+ +
+ 👍 Fenil Doshi +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Yehuda Korotkin + (yehudak@elementor.com) +
+
2022-07-28 04:02:06
 +
+

👋 Hi everyone! +Yesterday was a great presentation by @Julien Le Dem that talked about OpenLineage and did grate comparison between OL and Open-Telemetry, (i wrote a small summary here: https://bit.ly/3z5caOI )

+ +

Julian’s charm sparked inside me curiosity especially regarding OL in streaming. +I saw the design/architecture of OL I got some questions/discussions that I would like to understand better.

+ +

In the context of streaming jobs reporting “start job” - “end job” might be more relevant in the context of a batch mode. +or do you mean reporting start job/end job should be processed each event?

  • and this will be equivalent to starting job each row in a table via UDF, for example.
    • +
+ +

Thank you in advance

+
+
linkedin.com
+ + + +
Yehuda Korotkin on LinkedIn: #new_tech #Kafka #Kafka
+
+ Summey of the meetup: / Kafka alternative / Open-Telemetry / Open-Lineage. was worth going. See details: <b>#new_tech</b> Memphis.dev Idan Asulin Talked about... +
+ + + + + + + + +
Original URL: https://bit.ly/3z5caOI
+ + + +
+ + + +
+ 🙌 Maciej Obuchowski, Michael Robinson, Paweł Leszczyński +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-07-28 08:50:44
 +
+

*Thread Reply:* Welcome to the community!

+ +

We talked about this exact topic in the most recent community call. +https://wiki.lfaidata.foundation/display/OpenLineage/Monthly+TSC+meeting#MonthlyTSCmeeting-Nextmeeting:Nov10th2021(9amPT)

+ +

Discussion: streaming in Flink integration +• Has there been any evolution in the thinking on support for streaming? + ◦ Julien: start event, complete event, snapshots in between limited to certain number per time interval + ◦ Paweł: we can make the snapshot volume configurable +• Does Flink support sending data to multiple tables like Spark? + ◦ Yes, multiple outputs supported by OpenLineage model + ◦ Marquez, the reference implementation of OL, combines the outputs

+ + + +
+ 🙏 Yehuda Korotkin +
+ +
+ ❤️ Julien Le Dem +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-07-28 09:56:05
 +
+

*Thread Reply:* > or do you mean reporting start job/end job should be processed each event? +We definitely want to avoid tracking every single event 🙂

+ +

One thing worth mentioning is that OpenLineage events are meant to be cumulative - the streaming jobs start, run, and eventually finish or restart. In the meantime, we capture additional events "in the middle" - for example, on Apache Flink checkpoint, or every few minutes - where we can emit additional information connected to the state of the job.

+ + + +
+ 🙏 Yehuda Korotkin +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Yehuda Korotkin + (yehudak@elementor.com) +
+
2022-07-28 11:11:17
 +
+

*Thread Reply:* @Will Johnson and @Maciej Obuchowski Thank you for your answer

+ +

jobs start, run, and eventually finish or restart

+ +

This is the perspective that I have a hard time understanding in the context of streaming.

+ +

The classic streaming job should always be on it should not be “finish” event (Except failure). +usually, streaming data is “dripping”.

+ +

It is possible to understand if the job starts/ends in the resolution of the running application and represents when the application begin and when it failed.

+ +

if you do start/stop events from the checkpoints on Flink it might be the wrong representation instead use the concept of event-driven for example reporting state.

+ +

What do you think?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Yehuda Korotkin + (yehudak@elementor.com) +
+
2022-07-28 11:11:36
 +
+

*Thread Reply:*

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-07-28 12:00:34
 +
+

*Thread Reply:* The idea is that jobs usually get upgraded - for example, you change Apache Flink version, increase resources, or change the structure of a job - that's the difference for us. The stop events make sense, because if you for example changed SQL of your Flink SQL job, you probably would want this to be captured - from X to Y job was running with older SQL version well, but after change, the second run started and throughput dropped to 10% of the previous one.

+ +

> if you do start/stop events from the checkpoints on Flink it might be the wrong representation instead use the concept of event-driven for example reporting state. +But this is an misunderstanding 🙂 +The information exposed from a checkpoints are in addition to start and stop events.

+ +

We want to get information from running job - I just argue that sometimes end of a streaming job is also relevant.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-07-28 12:01:16
 +
+

*Thread Reply:* The checkpoint would be captured as a new eventType: RUNNING - do I miss something why you want to add StateFacet?

+ + + +
+ 👍 Yehuda Korotkin +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Yehuda Korotkin + (yehudak@elementor.com) +
+
2022-07-28 14:24:03
 +
+

*Thread Reply:* About argue - it’s depends on what the definition of job in streaming mode, i agree that if you already have ‘job’ you want to know about the job more information.

+ +

each event that entering the sub process (job) should do REST call “Start job” and “End job” ?

+ +

Nope, I just represented two possible ways that i thought, + or StateFacet + or add new Event type eg. RUNNING 😉

+ + + +
+ 👍 Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-07-28 09:14:28
 +
+

Hi everyone, I’d like to request a release to publish the new Flink integration (thanks, @Maciej Obuchowski) and an important fix to the Spark integration (thanks, @Paweł Leszczyński). As per our policy here, 3 +1s from committers will authorize an immediate release. Thanks!

+ + + +
+ ➕ Maciej Obuchowski, Paweł Leszczyński, Willy Lulciuc, Will Johnson, Julien Le Dem +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-07-28 17:30:33
 +
+

*Thread Reply:* Thanks for the +1s. We will initiate the release by Tuesday.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Barak F + (fargoun@gmail.com) +
+
2022-07-28 10:30:15
 +
+

Static code annotations for OpenLineage: hi everyone, i heard yesterday a great lecture by @Julien Le Dem on OpenLineage, and as i'm very interested in this area, i wanted to raise a question: are there any plans to have OpenLineage-like annotations on actual code (e.g. Spark, AirFlow, arbitrary code) to allow deducing some of the lineage informtion from static code analysis?

+ +

The reason i'm asking this is because while OpenLineage does a great job of integrating with multiple platforms (AirFlow, Dbt, Spark), some companies still have a lot of legacy-related data processing stack that will probably not get full OpenLineage (as it's a one-off, and the companies themselves will probably won't implement OpenLineage support for their custom frameworks). +Having some standard way to annotate code with information like: "reads from X; writes to Y; Job name regexp: Z", may allow writing a "generic" OpenLineage colelctor that can go over the source code, collect this configuration information and then use it when constructing the lineage graph (even though it won't be as complete and full as the full OpenLineage info).

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-08-03 08:30:15
 +
+

*Thread Reply:* I think this is an interesting idea, however, just the static analysis does not convey any runtime information.

+ +

We're doing something similar within Airflow now, but as a fallback mechanism: https://github.com/OpenLineage/OpenLineage/pull/914

+ +

You can manually annotate DAG with information instead of writing extractor for your operator. This still gives you runtime information. Similar features might get added to other integrations, especially with such a vast scope as Airflow has - but I think it's unlikely we'd work on a feature for just statically traversing code without runtime context.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Barak F + (fargoun@gmail.com) +
+
2022-08-03 14:25:31
 +
+

*Thread Reply:* Thanks for the detailed response @Maciej Obuchowski! It seems like this solution is specific only to AirFlow, and i wonder why wouldn't we generalize this outside of just AirFlow? My thinking is that there are other areas where there is vast scope (e.g. arbitrary code that does data manipulations), and without such an option, the only path is to provide full runtime information via building your own extractor, which might be a bit hard/expensive to do. +If i understand your response correctly, then you assume that OpenLineage can get wide enough "native" support across the stack without resorting to a fallback like 'static code analysis'. Is that your base assumption?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Petr Hajek + (petr.hajek@profinit.eu) +
+
2022-07-29 04:36:03
 +
+

Hi all, does anybody have an experience extracting Airflow lineage using Marquez as documented here https://www.astronomer.io/guides/airflow-openlineage/#generating-and-viewing-lineage-data ? +We tested it on our Airflow instance with Marquez hoping to get the standard .json files describing lineage in accord with open-lineage model as described in https://json-schema.org/draft/2020-12/schema. +But there seems to be only one GET method related to lineage export in Marquez API library called "Get a lineage graph". This produces quite different .json structure than what we know from open-lineage. Could anybody help if there is a chance to get open-lineage .json structure from Marquez?

+
+
astronomer.io
+ + + +
OpenLineage and Airflow - Airflow Guides
+
+ Using OpenLineage and Marquez to get lineage data from your Airflow DAGs. +
+ + + + + + + + +
Original URL: https://www.astronomer.io/guides/airflow-openlineage/#generating-and-viewing-lineage-data
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-07-29 12:58:38
 +
+

*Thread Reply:* The query API has a different spec than the reporting API, so what you’d get from Marquez would look different from what Marquez receives.

+ +

Few ideas:

+ +
  1. you could send the lineage to a pipedream endpoint to inspect, if you’re just trying to experiment
  2. you could grab them from the lineage table in Marquez’s postgres
    3. +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Petr Hajek + (petr.hajek@profinit.eu) +
+
2022-07-30 16:29:24
 +
+

*Thread Reply:* ok, now I understand, thank you

+ + + +
+ 👍 Jan Kopic +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-08-03 08:25:57
 +
+

*Thread Reply:* FYI we want to have something like that too: https://github.com/MarquezProject/marquez/issues/1927

+ +

But if you need just the raw events endpoint, without UI, then Marquez might be overkill for your needs

+
+ + + + +
<https://github.com/MarquezProject/marquez/issues/1927|#1927 Option to dump OpenLineage events that correspond to dataset/namespace from web>
+
+ <p>Having this feature would make debugging, and replicating errors much faster.</p> +
+ +
+
Comments
+ 2 +
+ +
+
Milestone
+ <a href="https://github.com/MarquezProject/marquez/milestone/4">Roadmap</a> +
+ + + + + + + +
+ + <a href="https://github.com/MarquezProject/marquez">MarquezProject/marquez</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Dinakar Sundar + (dinakar_sundar@condenast.com) +
+
2022-07-30 13:44:13
 +
+

Hi @everyone , we are trying to extract lineage information and import into amundsen .please point us right direction to move - based on the documentation -> Databricks + marquez + amundsen is this the only way to move on ?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Thomas + (john.thomas@astronomer.io) +
+
2022-07-30 13:49:25
 +
+

*Thread Reply:* Short of implementing an open lineage endpoint in Amundsen, yes that's the right approach.

+ +

The Lineage endpoint in Marquez can output the whole graph centered on a node ID, and you can use the jobs/datasets apis to grab lists of each for reference

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Barak F + (fargoun@gmail.com) +
+
2022-07-31 00:35:06
 +
+

*Thread Reply:* Is your lineage information coming via OpenLineage? if so - you can quickly use the Amundsen scripts in order to load data into Amundsen, for example, see this script here: https://github.com/amundsen-io/amundsendatabuilder/blob/master/example/scripts/sample_data_loader.py

+ +

Where is your lineage coming from?

+
+ + + + +
<https://github.com/amundsen-io/amundsendatabuilder/blob/master/example/scripts/sample_data_loader.py | sample_data_loader.py>
+
+ <p>```</p> + +<h1>Copyright Contributors to the Amundsen project.</h1> + +<h1>SPDX-License-Identifier: Apache-2.0</h1> + +<p>""" +This is a example script demonstrating how to load data into Neo4j and +Elasticsearch without using an Airflow DAG.</p> + +<p>It contains several jobs:</p><ul><li><code>run_csv_job</code>: runs a job that extracts table data from a CSV, loads (writes) +this into a different local directory as a csv, then publishes this data to +neo4j.</li><li><code>run_table_column_job</code>: does the same thing as <code>run_csv_job</code>, but with a csv +containing column data.</li><li><code>create_last_updated_job</code>: creates a job that gets the current time, dumps it +into a predefined model schema, and publishes this to neo4j.</li><li><code>create_es_publisher_sample_job</code>: creates a job that extracts data from neo4j +and pubishes it into elasticsearch.</li> +</ul> + +<p>For other available extractors, please take a look at +<a href="https://github.com/amundsen-io/amundsendatabuilder#list-of-extractors">https://github.com/amundsen-io/amundsendatabuilder#list-of-extractors</a> +"""</p> + +<p>import logging +import os +import sys +import uuid</p> + +<p>from elasticsearch import Elasticsearch +from pyhocon import ConfigFactory +from sqlalchemy.ext.declarative import declarative_base</p> + +<p>from databuilder.extractor.csv<em>extractor import ( + CsvColumnLineageExtractor, CsvExtractor, CsvTableBadgeExtractor, CsvTableColumnExtractor, CsvTableLineageExtractor, +) +from databuilder.extractor.es</em>last<em>updated</em>extractor import EsLastUpdatedExtractor +from databuilder.extractor.neo4j<em>search</em>data<em>extractor import Neo4jSearchDataExtractor +from databuilder.job.job import DefaultJob +from databuilder.loader.file</em>system<em>elasticsearch</em>json<em>loader import FSElasticsearchJSONLoader +from databuilder.loader.file</em>system<em>neo4j</em>csv<em>loader import FsNeo4jCSVLoader +from databuilder.publisher.elasticsearch</em>constants import ( + DASHBOARD<em>ELASTICSEARCH</em>INDEX<em>MAPPING, USER</em>ELASTICSEARCH<em>INDEX</em>MAPPING, +) +from databuilder.publisher.elasticsearch<em>publisher import ElasticsearchPublisher +from databuilder.publisher.neo4j</em>csv<em>publisher import Neo4jCsvPublisher +from databuilder.task.task import DefaultTask +from databuilder.transformer.base</em>transformer import ChainedTransformer, NoopTransformer +from databuilder.transformer.dict<em>to</em>model import MODEL<em>CLASS, DictToModel +from databuilder.transformer.generic</em>transformer import ( + CALLBACK<em>FUNCTION, FIELD</em>NAME, GenericTransformer, +)</p> + +<p>es<em>host = os.getenv('CREDENTIALS</em>ELASTICSEARCH<em>PROXY</em>HOST', 'localhost') +neo<em>host = os.getenv('CREDENTIALS</em>NEO4J<em>PROXY</em>HOST', 'localhost')</p> + +<p>es<em>port = os.getenv('CREDENTIALS</em>ELASTICSEARCH<em>PROXY</em>PORT', 9200) +neo<em>port = os.getenv('CREDENTIALS</em>NEO4J<em>PROXY</em>PORT', 7687) +if len(sys.argv) &gt; 1: + es<em>host = sys.argv[1] +if len(sys.argv) &gt; 2: + neo</em>host = sys.argv[2]</p> + +<p>es = Elasticsearch([ + {'host': es<em>host, 'port': es</em>port}, +])</p> + +<p>Base = declarative_base()</p> + +<p>NEO4J<em>ENDPOINT = f'bolt://{neo</em>host}:{neo_port}'</p> + +<p>neo4j<em>endpoint = NEO4J</em>ENDPOINT</p> + +<p>neo4j<em>user = 'neo4j' +neo4j</em>password = 'test'</p> + +<p>LOGGER = logging.getLogger(<strong>name</strong>)</p> + +<p>def run<em>csv</em>job(file<em>loc, job</em>name, model): + tmp<em>folder = f'/var/tmp/amundsen/{job</em>name}' + node<em>files</em>folder = f'{tmp<em>folder}/nodes' + relationship</em>files<em>folder = f'{tmp</em>folder}/relationships'</p> + +<pre><code>csv_extractor = CsvExtractor() +csv_loader = FsNeo4jCSVLoader() + +task = DefaultTask(extractor=csv_extractor, + loader=csv_loader, + transformer=NoopTransformer()) + +job_config = ConfigFactory.from_dict({ + 'extractor.csv.file_location': file_loc, + 'extractor.csv.model_class': model, + 'loader.filesystem_csv_neo4j.node_dir_path': node_files_folder, + 'loader.filesystem_csv_neo4j.relationship_dir_path': relationship_files_folder, + 'loader.filesystem_csv_neo4j.delete_created_directories': True, + 'publisher.neo4j.node_files_directory': node_files_folder, + 'publisher.neo4j.relation_files_directory': relationship_files_folder, + 'publisher.neo4j.neo4j_endpoint': neo4j_endpoint, + 'publisher.neo4j.neo4j_user': neo4j_user, + 'publisher.neo4j.neo4j_password': neo4j_password, + 'publisher.neo4j.neo4j_encrypted': False, + 'publisher.neo4j.job_publish_tag': 'unique_tag', # should use unique tag here like {ds} +}) + +DefaultJob(conf=job_config, + task=task, + publisher=Neo4jCsvPublisher()).launch() +</code></pre> + +<p>def run<em>table</em>badge<em>job(table</em>path, badge<em>path): + tmp</em>folder = '/var/tmp/amundsen/table<em>badge' + node</em>files<em>folder = f'{tmp</em>folder}/nodes' + relationship<em>files</em>folder = f'{tmp<em>folder}/relationships' + extractor = CsvTableBadgeExtractor() + csv</em>loader = FsNeo4jCSVLoader() + task = DefaultTask(extractor=extractor, + loader=csv<em>loader, + transformer=NoopTransformer()) + job</em>config = ConfigFactory.from<em>dict({ + 'extractor.csvtablebadge.table</em>file<em>location': table</em>path, + 'extractor.csvtablebadge.badge<em>file</em>location': badge<em>path, + 'loader.filesystem</em>csv<em>neo4j.node</em>dir<em>path': node</em>files<em>folder, + 'loader.filesystem</em>csv<em>neo4j.relationship</em>dir<em>path': relationship</em>files<em>folder, + 'loader.filesystem</em>csv<em>neo4j.delete</em>created<em>directories': True, + 'publisher.neo4j.node</em>files<em>directory': node</em>files<em>folder, + 'publisher.neo4j.relation</em>files<em>directory': relationship</em>files<em>folder, + 'publisher.neo4j.neo4j</em>endpoint': neo4j<em>endpoint, + 'publisher.neo4j.neo4j</em>user': neo4j<em>user, + 'publisher.neo4j.neo4j</em>password': neo4j<em>password, + 'publisher.neo4j.neo4j</em>encrypted': False, + 'publisher.neo4j.job<em>publish</em>tag': 'unique<em>tag</em>b', # should use unique tag here like {ds} + }) + job = DefaultJob(conf=job_config, + task=task, + publisher=Neo4jCsvPublisher()) + job.launch()</p> + +<p>def run<em>table</em>column<em>job(table</em>path, column<em>path): + tmp</em>folder = '/var/tmp/amundsen/table<em>column' + node</em>files<em>folder = f'{tmp</em>folder}/nodes' + relationship<em>files</em>folder = f'{tmp<em>folder}/relationships' + extractor = CsvTableColumnExtractor() + csv</em>loader = FsNeo4jCSVLoader() + task = DefaultTask(extractor, + loader=csv<em>loader, + transformer=NoopTransformer()) + job</em>config = ConfigFactory.from<em>dict({ + 'extractor.csvtablecolumn.table</em>file<em>location': table</em>path, + 'extractor.csvtablecolumn.column<em>file</em>location': column<em>path, + 'loader.filesystem</em>csv<em>neo4j.node</em>dir<em>path': node</em>files<em>folder, + 'loader.filesystem</em>csv<em>neo4j.relationship</em>dir<em>path': relationship</em>files<em>folder, + 'loader.filesystem</em>csv<em>neo4j.delete</em>created<em>directories': True, + 'publisher.neo4j.node</em>files<em>directory': node</em>files<em>folder, + 'publisher.neo4j.relation</em>files<em>directory': relationship</em>files<em>folder, + 'publisher.neo4j.neo4j</em>endpoint': neo4j<em>endpoint, + 'publisher.neo4j.neo4j</em>user': neo4j<em>user, + 'publisher.neo4j.neo4j</em>password': neo4j<em>password, + 'publisher.neo4j.neo4j</em>encrypted': False, + 'publisher.neo4j.job<em>publish</em>tag': 'unique<em>tag', # should use unique tag here like {ds} + }) + job = DefaultJob(conf=job</em>config, + task=task, + publisher=Neo4jCsvPublisher()) + job.launch()</p> + +<p>def run<em>table</em>lineage<em>job(table</em>lineage<em>path): + tmp</em>folder = '/var/tmp/amundsen/table<em>column' + node</em>files<em>folder = f'{tmp</em>folder}/nodes' + relationship<em>files</em>folder = f'{tmp<em>folder}/relationships' + extractor = CsvTableLineageExtractor() + csv</em>loader = FsNeo4jCSVLoader() + task = DefaultTask(extractor, + loader=csv<em>loader, + transformer=NoopTransformer()) + job</em>config = ConfigFactory.from<em>dict({ + 'extractor.csvtablelineage.table</em>lineage<em>file</em>location': table<em>lineage</em>path, + 'loader.filesystem<em>csv</em>neo4j.node<em>dir</em>path': node<em>files</em>folder, + 'loader.filesystem<em>csv</em>neo4j.relationship<em>dir</em>path': relationship<em>files</em>folder, + 'loader.filesystem<em>csv</em>neo4j.delete<em>created</em>directories': True, + 'publisher.neo4j.node<em>files</em>directory': node<em>files</em>folder, + 'publisher.neo4j…</p> +
+ + + + + + + +
+ + <a href="https://github.com/amundsen-io/amundsendatabuilder">amundsen-io/amundsendatabuilder</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Dinakar Sundar + (dinakar_sundar@condenast.com) +
+
2022-08-01 20:17:22
 +
+

*Thread Reply:* yes @Barak F we are using open lineage

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Barak F + (fargoun@gmail.com) +
+
2022-08-02 01:26:18
 +
+

*Thread Reply:* So, have you tried using Amundsen data builder scripts to load the lineage information into Amundsen? (maybe you'll have to "play" with those a bit)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-08-03 08:24:58
 +
+

*Thread Reply:* AFAIK there is OpenLineage extractor: https://www.amundsen.io/amundsen/databuilder/#openlineagetablelineageextractor

+ +

Not sure it solves your issue though 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Dinakar Sundar + (dinakar_sundar@condenast.com) +
+
2022-08-05 04:46:45
 +
+

*Thread Reply:* thanks

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-08-01 17:08:46
 +
+

@channel +OpenLineage 0.12.0 is now available! +We added: +• an Apache Flink integration, +• support for Spark 3.3.0, +• the ability to extend column level lineage mechanism, +• an ErrorMessageRunFacet to the OpenLineage spec, +• SQLCheckExtractors, a RedshiftSQLExtractor & RedshiftDataExtractor to the Airflow integration, +• a dataset builder to the AlterTableCommand class in the Spark integration. +We changed: +• the filtering of Delta events to reduce noise, +• the flow of metadata in the Airflow integration to allow metadata from Airflow through inlets and outlets. +Thanks to all the contributors who made this release possible! +For the bug fixes and more details, see: +Release: https://github.com/OpenLineage/OpenLineage/releases/tag/0.12.0 +Changelog: https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md +Commit history: https://github.com/OpenLineage/OpenLineage/compare/0.11.0...0.12.0 +Maven: https://oss.sonatype.org/#nexus-search;quick~openlineage +PyPI: https://pypi.org/project/openlineage-python/ (edited)

+ + + +
+ ❤️ Minkyu Park, Harel Shein, Willy Lulciuc, Peter Hicks, Fenil Doshi, Maciej Obuchowski, Howard Yoo, Paul Wilson Villena, Jarek Potiuk, Dinakar Sundar, Shubham Mehta, Sharanya Santhanam, Sheeri Cabral (Collibra) +
+ +
+ 🎉 Minkyu Park, Peter Hicks, Fenil Doshi, Howard Yoo, Jarek Potiuk, Paweł Leszczyński, Ryan Peterson +
+ +
+ 🚀 Minkyu Park, Howard Yoo, Jarek Potiuk +
+ +
+ 🙌 Minkyu Park, Willy Lulciuc, Maciej Obuchowski, Howard Yoo, Jarek Potiuk +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sharanya Santhanam + (santhanamsharanya@gmail.com) +
+
2022-08-02 10:12:01
 +
+

What is the right way of handling/parsing facets on the server side?

+ +

I see the generated server side stubs are generic : https://github.com/OpenLineage/OpenLineage/blob/main/client/java/generator/src/main/java/io/openlineage/client/Generator.java#L131 and dont have any resolved facet information. +Marquez seems to have duplicated the OL model with https://github.com/MarquezProject/marquez/blob/main/api/src/main/java/marquez/service/models/LineageEvent.java#L71 and converts the incoming OL events to a “LineageEvent” for appropriate handling. Is there a cleaner approach where in the known facets can be generated in io.openlineage.server?

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/client/java/generator/src/main/java/io/openlineage/client/Generator.java | Generator.java>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+
+ + + + +
<https://github.com/MarquezProject/marquez/blob/main/api/src/main/java/marquez/service/models/LineageEvent.java | LineageEvent.java>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/MarquezProject/marquez">MarquezProject/marquez</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-08-02 12:28:11
 +
+

*Thread Reply:* I think the reason for server model being very generic is because new facets can be added later (also as custom facets) - and generally server wants to accept all valid events and get the facet information that it can actually use, rather than reject event because it has unknown field.

+ +

Server model was added here after some discussion in Marquez which is relevant - I think @Michael Collado @Willy Lulciuc can add to that

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/pull/300|#300 server model>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+
+ + + + +
<https://github.com/MarquezProject/marquez/issues/1650|#1650 Binary incompatibility after deleting LineageEvent classes>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/MarquezProject/marquez">MarquezProject/marquez</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sharanya Santhanam + (santhanamsharanya@gmail.com) +
+
2022-08-02 15:54:24
 +
+

*Thread Reply:* Thanks for the response. I realize the server stubs were created to support flexibility , but it also makes the parsing logic on server side a bit more complex as we need to maintain code on the server side to look for specific facets & their properties from maps or like maquez duplicate the OL model on our end with the facets we care about. Wanted to know whats the guidance around managing this server side. @Willy Lulciuc @Michael Collado Any suggestions ?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-08-02 18:27:27
 +
+

Agenda items are requested for the next OpenLineage Technical Steering Committee meeting on August 11 at 10am PT. Reply in thread or ping me with your item(s)!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Varun Singh + (varuntestaz@outlook.com) +
+
2022-08-03 04:16:22
 +
+

Hi all, +I am trying out the openlineage spark integration and can't find any column lineage information included with the events. I tried it out with an input dataset where I renamed one of the columns but the columnLineage facet was not present. Can anyone suggest some other examples where it might show up?

+ +

Thanks!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-08-03 04:45:36
 +
+

*Thread Reply:* @Paweł Leszczyński do we collect column level lineage on renames?

+ + + +
+
+
+
+ + + + + +
+ +
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2022-08-05 05:55:12
 +
+

*Thread Reply:* I’ve created an issue for column lineage in case of renaming: +https://github.com/OpenLineage/OpenLineage/issues/993

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/issues/993|#993 collect column level lineage for renames>
+
+ <p>In case of <code>alter table rename column</code> or <code>alter table rename</code> we do not collect column level lineage at the moment, which may be useful in some scenarios.</p> +
+ +
+
Labels
+ integration/spark +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Varun Singh + (varuntestaz@outlook.com) +
+
2022-08-08 09:37:43
 +
+

*Thread Reply:* Thanks @Paweł Leszczyński!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-08-03 12:58:44
 +
+

Hey everyone! I am looking into Fivetran a bit, and it occurs to me that the NAMING.md document does not have an opinion about how to deal with entire systems as datasets. More in 🧵.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-08-03 13:00:22
 +
+

*Thread Reply:* Fivetran is a tool that copies data from source systems to target databases. One of these source systems might be SalesForce, for example.

+ +

This copying results in thousands of SQL queries run against the target database for each sync. I don’t think each of these queries should map to an OpenLineage job, I think the entire synchronization should. Maybe I’m wrong here.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-08-03 13:01:00
 +
+

*Thread Reply:* But if I’m right, that means that there needs to be a way to specify “SalesForce Account #45123452233” as a dataset.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-08-03 13:01:44
 +
+

*Thread Reply:* or it ends up just being a job with outputs and no inputs…but that’s not very illuminating

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-08-03 13:02:27
 +
+

*Thread Reply:* or is that good enough?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-08-04 10:31:11
 +
+

*Thread Reply:* You are looking at a pretty big topic here 🙂

+ +

Basically you're asking what is a job in OpenLineage - and it's not fully answered yet.

+ +

I think the discussion is kinda relevant to this proposed facet and I kinda replied there: https://github.com/OpenLineage/OpenLineage/issues/812#issuecomment-1205337556

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harel Shein + (harel.shein@gmail.com) +
+
2022-08-04 15:50:22
 +
+

*Thread Reply:* my 2 cents on this is that in the Salesforce example, the system is to complex to capture as a single dataset. and so maybe different objects within a salesforce account (org/account/opportunity/etc…) could be treated as individual datasets. But as @Maciej Obuchowski pointed out, this is quite a large topic 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-08-08 13:46:31
 +
+

*Thread Reply:* I guess it depends on whether you actually care about the table/column level lineage for an operation like “copy salesforce to snowflake”.

+ +

I can see it being a nuisance having all of that on a lineage graph. OTOH, I can see it being useful to know that a datum can be traced back to a specific endpoint at SFDC.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-08-08 13:46:55
 +
+

*Thread Reply:* this is a design decision, IMO.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-08-04 11:30:00
 +
+

@channel The next OpenLineage Technical Steering Committee meeting is on Thursday, August 11 at 10 am PT. Join us on Zoom: https://bit.ly/OLzoom +All are welcome! +Agenda:

+ +
  1. Announcements
  2. Docs site update
  3. Release 0.11.0 and 0.12.0 overview
  4. Extractors: examples and how to write them
  5. Open discussion +Notes: https://bit.ly/OLwiki +Is there a topic you think the community should discuss at this or a future meeting? Reply or DM me to add items to the agenda. (edited)
    6. +
+
+
Zoom Video
+ + + +
Join our Cloud HD Video Meeting
+
+ Zoom is the leader in modern enterprise video communications, with an easy, reliable cloud platform for video and audio conferencing, chat, and webinars across mobile, desktop, and room systems. Zoom Rooms is the original software-based conference room solution used around the world in board, conference, huddle, and training rooms, as well as executive offices and classrooms. Founded in 2011, Zoom helps businesses and organizations bring their teams together in a frictionless environment to get more done. Zoom is a publicly traded company headquartered in San Jose, CA. +
+ + + + + + +
Original URL: https://bit.ly/OLzoom
+ + + +
+ + + +
+ 🙌 Maciej Obuchowski, Harel Shein, Paul Wilson Villena +
+ +
+ 👀 Francis McGregor-Macdonald +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Chris Coulthrust + (coulthrust@gmail.com) +
+
2022-08-06 12:06:47
 +
+

👋 Hi everyone!

+ + + +
+ 👋 Jakub Dardziński, Michael Robinson, Ross Turk, Harel Shein, Willy Lulciuc, Howard Yoo +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-08-10 11:00:01
 +
+

@channel The next OpenLineage TSC meeting is tomorrow! https://openlineage.slack.com/archives/C01CK9T7HKR/p1659627000308969

+
+ + +
+ + + } + + Michael Robinson + (https://openlineage.slack.com/team/U02LXF3HUN7) +
+ + + +
+
+ <p><strong>@channel</strong> The next OpenLineage Technical Steering Committee meeting is on <strong>Thursday, August 11 at 10 am PT</strong>. Join us on Zoom: <a href="https://bit.ly/OLzoom">https://bit.ly/OLzoom</a> +All are welcome! +Agenda:</p> + +<ol><li>Announcements</li><li>Docs site update</li><li>Release 0.11.0 and 0.12.0 overview</li><li>Extractors: examples and how to write them</li><li>Open discussion +Notes: <a href="https://bit.ly/OLwiki">https://bit.ly/OLwiki</a> +Is there a topic you think the community should discuss at this or a future meeting? Reply or DM me to add items to the agenda. (edited)</li> +</ol> +
+ + + + + + +
Original URL: https://openlineage.slack.com/archives/C01CK9T7HKR/p1659627000308969
+ + +
+ + Slack Conversation +
+ + +
+ + + +
+ 👀 Howard Yoo +
+ +
+ ❤️ Minkyu Park +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-08-10 22:34:29
 +
+

*Thread Reply:* I am so sad I'm going to miss this month's meeting 😰 Looking forward to the recording!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-08-12 06:19:58
 +
+

*Thread Reply:* We missed you too @Will Johnson 😉

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Raj Mishra + (hax0755@gmail.com) +
+
2022-08-11 18:50:18
 +
+

Hi everyone! I have a REST endpoint that I use for other pipelines that can POST their RunEvent and I forward that to marquez. I'm expecting a JSON which has the RunEvent details, which also has the input or output dataset depending upon the EventType. I can see the Run details always shows up on the marquez UI, but the dataset has issues. I can see the dataset listed but when I can click on it, just shows "something went wrong." I don't see any details of that dataset. +{ + "eventType": "START", + "eventTime": "2022-08-09T19:49:24.201361Z", + "run": { + "runId": "d46e465b-d358-4d32-83d4-df660ff614dd" + }, + "job": { + "namespace": "TEST-NAMESPACE", + "name": "test-job" + }, + "inputs": [ + { + "namespace": "TEST-NAMESPACE", + "name": "my-test-input", + "facets": { + "schema": { + "_producer": "<https://github.com/OpenLineage/OpenLineage/blob/v1-0-0/client>", + "_schemaURL": "<https://github.com/OpenLineage/OpenLineage/blob/v1-0-0/spec/OpenLineage.json#/definitions/SchemaDatasetFacet>", + "fields": [ + { + "name": "a", + "type": "INTEGER" + }, + { + "name": "b", + "type": "TIMESTAMP" + }, + { + "name": "c", + "type": "INTEGER" + }, + { + "name": "d", + "type": "INTEGER" + } + ] + } + } + } + ], + "producer": "<https://github.com/OpenLineage/OpenLineage/blob/v1-0-0/client>" +} +In above payload, the input data set is never created on marquez. I can only see the Run details, but input data set is just empty. Does the input data set needs to created first and then only the RunEvent can be created?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-08-12 06:09:57
 +
+

*Thread Reply:* From the first look, you're missing outputsfield in your event - this might break something

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-08-12 06:10:20
 +
+

*Thread Reply:* If not, then Marquez logs might help to see something

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Raj Mishra + (hax0755@gmail.com) +
+
2022-08-12 13:12:56
 +
+

*Thread Reply:* Does the START event needs to have an output?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-08-12 13:19:24
 +
+

*Thread Reply:* It can have empty output 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-08-12 13:32:43
 +
+

*Thread Reply:* well, in your case you need to send COMPLETE event

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-08-12 13:33:44
 +
+

*Thread Reply:* Internally, Marquez does not create dataset version until you complete event. It makes sense when your semantics are transactional - you can still read from previous dataset version until it's finished writing.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-08-12 13:34:06
 +
+

*Thread Reply:* After I send COMPLETE event with the same information I can see the dataset.

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Raj Mishra + (hax0755@gmail.com) +
+
2022-08-12 13:56:37
 +
+

*Thread Reply:* Thanks for the explanation @Maciej Obuchowski So, if I understand this correct. I won't see the my-test-input dataset till I have the COMPLETE event with input and output?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-08-12 14:34:51
 +
+

*Thread Reply:* @Raj Mishra Yes and no 🙂

+ +

Basically your COMPLETE event does not need to contain any input and output datasets at all - OpenLineage model is cumulative, so it's enough to have datasets on either start or complete. +That also means you can add different datasets in different moment of a run lifecycle - for example, you know inputs, but not outputs, so you emit inputs on START , but not COMPLETE.

+ +

Or, the job is modifying the same dataset it reads from (which happens surprisingly often), Then, you want to collect various input metadata from the dataset before modifying it - most likely you won't have them on COMPLETE 🙂

+ +

In this example I've added my-test-input on START and my-test-input2 on COMPLETE :

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Raj Mishra + (hax0755@gmail.com) +
+
2022-08-12 14:47:56
 +
+

*Thread Reply:* @Maciej Obuchowski Thank you so much! This is great explanation.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sharanya Santhanam + (santhanamsharanya@gmail.com) +
+
2022-08-11 20:28:40
 +
+

Effectively handling file datasets on server side. We have a common usecase where dataset of type is produced/consumed per day. On the Lineage UI/server side it would be ideal to treat all files of this pattern as 1 dataset Vs 1 dataset per daily file. Any suggestions ?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sharanya Santhanam + (santhanamsharanya@gmail.com) +
+
2022-08-11 20:35:33
 +
+

*Thread Reply:* Would adding support for alias/grouping as a config on OL client side be valuable to other users ? i.e OL client could pass down an Alias/grouping facet Or should this be treated purely a server side feature

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-08-12 06:11:21
 +
+

*Thread Reply:* Agreed 🙂

+ +

How do you produce this dataset? Spark integration? Are you using any system like Apache Iceberg/Delta Lake or just writing raw files?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sharanya Santhanam + (santhanamsharanya@gmail.com) +
+
2022-08-12 12:59:48
 +
+

*Thread Reply:* these are raw files written from Spark or map reduce jobs. And downstream Spark jobs read these raw files to produce tables

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-08-12 13:27:34
 +
+

*Thread Reply:* written using Spark dataframe API, like +df.write.format("parquet").save("/tmp/spark_output/parquet") + or RDD?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-08-12 13:27:59
 +
+

*Thread Reply:* the actual API used matters, because we're handling different cases separately

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sharanya Santhanam + (santhanamsharanya@gmail.com) +
+
2022-08-12 13:29:48
 +
+

*Thread Reply:* I see. Let me look that up to be absolutely sure

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sharanya Santhanam + (santhanamsharanya@gmail.com) +
+
2022-08-12 19:21:41
 +
+

*Thread Reply:* It is like. this : df.write.format("parquet").save("/tmp/spark_output/parquet")

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sharanya Santhanam + (santhanamsharanya@gmail.com) +
+
2022-08-15 12:43:45
 +
+

*Thread Reply:* @Maciej Obuchowski curious what you had in mind with respect to RDDs & Dataframes. Also what if we cannot integrate OL with the frameworks that produce this dataset , but only those that consume from the already produced datasets. Is there a way we could still capture the dataset appropriately ?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-08-16 05:30:57
 +
+

*Thread Reply:* @Sharanya Santhanam the naming should be consistent between reading and writing, so it wouldn't change much of you can't integrate OL into writers. For the rest, can you create an issue on OL GitHub so someone can pick it up? I'm at vacation now.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sharanya Santhanam + (santhanamsharanya@gmail.com) +
+
2022-08-16 15:08:41
 +
+

*Thread Reply:* Sounds good , Ty !

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Varun Singh + (varuntestaz@outlook.com) +
+
2022-08-12 06:02:00
 +
+

Hi, Minor Suggestion: +This line https://github.com/OpenLineage/OpenLineage/blob/46efab1e7c2a0aa5ebe8d11185fe8d5225[…]/app/src/main/java/io/openlineage/spark/agent/EventEmitter.java is printing variables like api key and other parameters in the logs. Wouldn't it be more appropriate to use log.debug instead? +I'll create an issue if others agree

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/46efab1e7c2a0aa5ebe8d11185fe8d5225c0f78b/integration/spark/app/src/main/java/io/openlineage/spark/agent/EventEmitter.java | EventEmitter.java>
+
+ <pre><code> "Init OpenLineageContext: Args: %s URI: %s", argument, lineageURI.toString())); +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-08-12 06:09:11
 +
+

*Thread Reply:* yes

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-08-12 06:09:32
 +
+

*Thread Reply:* please do create 🙂

+ + + +
+ ✅ Varun Singh +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Conor Beverland + (conorbev@gmail.com) +
+
2022-08-15 09:01:47
 +
+

dumb question but, is it easy to run all the OpenLineage tests locally? ( and if so how? 🙂 )

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2022-08-17 13:54:19
 +
+

*Thread Reply:* it's per project. +java based: ./gradlew test +python based: https://github.com/OpenLineage/OpenLineage/tree/main/integration/airflow#development

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-08-18 23:45:30
 +
+

Spark Integration: The Order of Processing Events in the Async Event Queue

+ +

Hey, OpenLineage team, I'm working on a PR (https://github.com/OpenLineage/OpenLineage/pull/849/) that is going to store information given in different spark events (e.g. SparkListenerSQLExecutionStart, SparkListenerJobStart).

+ +

However, I want to avoid holding all this data once the execution of the job is complete. As a result, I want to remove the data once I receive a SparkListenerSQLExecutionEnd.

+ +

However, can I be guaranteed that the ExecutionEnd event will be processed AFTER the JobStart event? Is it possible that I can take too long to process the the JobStart event that the ExecutionEnd executes prior to the JobStart finishing?

+ +

I know we do something similar to this with sparkSqlExecutionRegistry (https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/app/src/mai[…]n/java/io/openlineage/spark/agent/OpenLineageSparkListener.java) but do we have any docs to help explain how the AsyncEventQueue orders and consumes events for a listener?

+ +

Thank you so much for any insights

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/pull/849|#849 Feature/spark parent run facet execution>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/app/src/main/java/io/openlineage/spark/agent/OpenLineageSparkListener.java | OpenLineageSparkListener.java>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2022-08-19 18:38:10
 +
+

*Thread Reply:* Hey Will! A bunch of folks are on vacation or out this week. Sorry for the delay, I am personally not sure but if it's not too urgent you can have an answer when knowledgable folks are back.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-08-19 20:21:18
 +
+

*Thread Reply:* Hah! No worries, @Julien Le Dem! I can definitely wait for the lucky people who are enjoying the last few weeks of summer unlike the rest of us 😋

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-08-29 05:31:32
 +
+

*Thread Reply:* @Paweł Leszczyński might want to look at that

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Hanbing Wang + (doris.wang200902@gmail.com) +
+
2022-08-19 01:53:56
 +
+

Hi, +I try to find out if openLineage spark support pyspark (Non-sql) use cases? +Is there any doc I could get more details about non-sql openLineage support? +Thanks a lot

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2022-08-19 12:30:08
 +
+

*Thread Reply:* Hello Hanbing, the spark integration works for PySpark since pyspark is wrapped into regular spark operators.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Hanbing Wang + (doris.wang200902@gmail.com) +
+
2022-08-19 13:49:35
 +
+

*Thread Reply:* @Julien Le Dem Thanks a lot for your help. I searched around, but I couldn't find any doc introduce how pyspark supported in openLineage. +My company want to integrate with openLineage-spark, I am working on figure out what info does OpenLineage make available for non-sql and does it at least have support for logging the logical plan?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2022-08-19 18:26:48
 +
+

*Thread Reply:* Yes, it does send the logical plan as part of the event

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2022-08-19 18:27:32
 +
+

*Thread Reply:* This configuration here should work as well for pyspark https://openlineage.io/docs/integrations/spark/

+
+
openlineage.io
+ + + +
Apache Spark | OpenLineage Docs
+
+ Spark jobs typically run on clusters of machines. A single machine hosts the "driver" application, +
+ + + + + + +
Original URL: https://openlineage.io/docs/integrations/spark/
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2022-08-19 18:28:11
 +
+

*Thread Reply:* --conf "spark.extraListeners=io.openlineage.spark.agent.OpenLineageSparkListener"

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2022-08-19 18:28:26
 +
+

*Thread Reply:* you need to add the jar, set the listener and pass your OL config

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2022-08-19 18:31:11
 +
+

*Thread Reply:* Actually I'm demoing this at 27:10 right here 🙂 https://pretalx.com/bbuzz22/talk/FHEHAL/

+
+
pretalx
+ + + +
Cross-Platform Data Lineage with OpenLineage Berlin Buzzwords 2022
+
+ There are more data tools available than ever before, and it's easier to build a pipeline than it's ever been. This has resulted in an explosion of innovation, but it also means that data within today's organizations has become increasingly distributed. It can't be contained within a single brain, a single team, or a single platform. Data lineage can help by tracing the relationships between datasets and providing a map of your entire data universe. OpenLineage provides a standard for lineage collection that spans multiple platforms, including Apache Airflow, Apache Spark, Flink, and dbt. This empowers teams to diagnose and address widespread data quality and efficiency issues in real time. In this session, Julien Le Dem from Datakin will show how to trace data lineage across Apache Spark and Apache Airflow. He will walk through the OpenLineage architecture and provide a live demo of a running pipeline with real-time data lineage. +
+ + + + + + + + +
Original URL: https://pretalx.com/bbuzz22/talk/FHEHAL/
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2022-08-19 18:32:11
 +
+

*Thread Reply:* you can see the parameters I'm passing to the pyspark command line in the video

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Hanbing Wang + (doris.wang200902@gmail.com) +
+
2022-08-19 18:35:50
 +
+

*Thread Reply:* @Julien Le Dem Thanks for the info, Let me take a look at the video now.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2022-08-19 18:40:10
 +
+

*Thread Reply:* The full demo starts at 24:40. It shows lineage connected together in Marquez coming from 3 different sources: Airflow, Spark and a custom integration

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-08-22 14:32:53
 +
+

Hi everyone, a release has been requested by @Harel Shein. As per our policy here, 3 +1s from committers will authorize an immediate release. Thanks! +Unreleased commits: https://github.com/OpenLineage/OpenLineage/compare/0.12.0...HEAD

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/GOVERNANCE.md | GOVERNANCE.md>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+ ➕ Willy Lulciuc, Michael Robinson, Minkyu Park, Jakub Dardziński, Julien Le Dem +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2022-08-22 14:38:58
 +
+

*Thread Reply:* @Michael Robinson can we start posting the “Unreleased” section in the changelog along with the release request? That way, we / the community will know what will be in the upcoming release

+ + + +
+ 👍 Michael Robinson +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-08-22 15:00:37
 +
+

*Thread Reply:* The release is approved. Thanks @Willy Lulciuc, @Minkyu Park, @Harel Shein

+ + + +
+ 🙌 Willy Lulciuc, Harel Shein +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-08-22 16:18:30
 +
+

@channel +OpenLineage 0.13.0 is now available! +We added: +• BigQuery check support +• RUNNING EventType in the spec and Python client +• databases and schemas to SQL extractors +• an event forwarding feature via HTTP +• Azure Cosmos Handler to the Spark integration +• support for OL datasets in manual lineage inputs/outputs +• ownership facets. +We changed: +• use RUNNING EventType in Flink integration for currently running jobs +• convert task object into JSON encodable when creating Airflow version facet. +Thanks to all the contributors who made this release possible! +For the bug fixes and more details, see: +Release: https://github.com/OpenLineage/OpenLineage/releases/tag/0.13.0 +Changelog: https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md +Commit history: https://github.com/OpenLineage/OpenLineage/compare/0.12.0...0.13.0 +Maven: https://oss.sonatype.org/#nexus-search;quick~openlineage +PyPI: https://pypi.org/project/openlineage-python/ (edited)

+ + + +
+ 🎉 Harel Shein, Ross Turk, Jarek Potiuk, Sheeri Cabral (Collibra), Willy Lulciuc, Howard Yoo, Howard Yoo, Ernie Ostic, Francis McGregor-Macdonald +
+ +
+ ✅ Sheeri Cabral (Collibra), Howard Yoo +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Conor Beverland + (conorbev@gmail.com) +
+
2022-08-23 03:55:24
 +
+

*Thread Reply:* Cool! Are the new ownership facets populated by the Airflow integration ?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
AMRIT SARKAR + (sarkaramrit2@gmail.com) +
+
2022-08-24 08:23:35
 +
+

Hi everyone, excited to work with OpenLineage. I am new to both OpenLineage and Data Lineage in general. Are there working examples/blog posts around actually integrating OpenLineage with existing graph DBs like Neo4J, Neptune etc? (I understand the service layer in between) I understand we have Amundsen with sample open lineage sample data - databuilder/example/sample_data/openlineage/sample_openlineage_events.ndjson. Thanks in advance.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2022-08-25 18:15:59
 +
+

*Thread Reply:* There is not that I know of besides the Amundsen integration example you pointed at. +A basic idea to do such a thing would be to implement an OpenLineage endpoint (receive the lineage events through http posts) and convert them to a format the graph db understand. If others in the community have ideas, please chime in

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
AMRIT SARKAR + (sarkaramrit2@gmail.com) +
+
2022-09-01 13:48:09
 +
+

*Thread Reply:* Understood, thanks a lot Julien. Make sense.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harel Shein + (harel.shein@gmail.com) +
+
2022-08-25 17:30:46
 +
+

Hey all, can I ask for a release for OpenLineage?

+ + + +
+ 👍 Harel Shein, Minkyu Park, Michael Robinson, Michael Collado, Ross Turk, Julien Le Dem, Willy Lulciuc, Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2022-08-25 17:32:44
 +
+

*Thread Reply:* @Michael Robinson ^

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-08-25 17:34:04
 +
+

*Thread Reply:* Thanks, Harel. 3 +1s from committers is all we need to make this happen today.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Minkyu Park + (minkyu@datakin.com) +
+
2022-08-25 17:52:40
 +
+

*Thread Reply:* 🙏

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-08-25 18:09:51
 +
+

*Thread Reply:* Thanks, all. The release is authorized

+ + + +
+ 🎉 Willy Lulciuc +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2022-08-25 18:16:44
 +
+

*Thread Reply:* can you also state the main purpose for this release?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-08-25 18:25:49
 +
+

*Thread Reply:* I believe (correct me if wrong, @Harel Shein) that this is to make available a fix of a bug in the compare functionality

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Minkyu Park + (minkyu@datakin.com) +
+
2022-08-25 18:27:53
 +
+

*Thread Reply:* ParentRunFacet from the airflow integration is not compliant to OpenLineage spec and this release includes the fix of that so that the marquez can handle parent run/job information.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-08-25 18:49:30
 +
+

@channel +OpenLineage 0.13.1 is now available! +We fixed: +• Rename all parentRun occurrences to parent from Airflow integration #1037 @fm100 +• Do not change task instance during on_running event #1028 @JDarDagran +Release: https://github.com/OpenLineage/OpenLineage/releases/tag/0.13.1 +Changelog: https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md +Commit history: https://github.com/OpenLineage/OpenLineage/compare/0.13.0...0.13.1 +Maven: https://oss.sonatype.org/#nexus-search;quick~openlineage +PyPI: https://pypi.org/project/openlineage-python/

+ + + +
+ 🎉 Harel Shein, Minkyu Park, Ross Turk, Michael Collado, Howard Yoo +
+ +
+ ❤️ Minkyu Park, Ross Turk, Howard Yoo +
+ +
+ 🥳 Minkyu Park, Ross Turk, Howard Yoo +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason + (shzhan@coupang.com) +
+
2022-08-26 18:58:17
 +
+

Hi, I am new to openlineage. Any one know how to enable spark column level lineage? I saw the code comment, it said default is disabled, thanks

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harel Shein + (harel.shein@gmail.com) +
+
2022-08-26 19:26:22
 +
+

*Thread Reply:* What version of Spark are you using? it should be enabled by default for Spark 3 +https://openlineage.io/docs/integrations/spark/spark_column_lineage

+
+
openlineage.io
+ + + +
Column Level Lineage | OpenLineage Docs
+
+ Column level lineage for Spark is turned on by default and requires no additional work to be done. The following documentation describes its internals. +
+ + + + + + +
Original URL: https://openlineage.io/docs/integrations/spark/spark_column_lineage
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason + (shzhan@coupang.com) +
+
2022-08-26 20:21:12
 +
+

*Thread Reply:* Thanks. Good to here that. I am use 0.9.+ . I will try again

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason + (shzhan@coupang.com) +
+
2022-08-29 13:14:01
 +
+

*Thread Reply:* I tested 0.9.+ 0.12.+ with spark 3.0 and 3.2 version. There still do not have dataset facet columnlineage. This is strange. I saw the column lineage design proposals 148. It should support from 0.9.+ Do I miss something?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason + (shzhan@coupang.com) +
+
2022-08-29 13:14:41
 +
+

*Thread Reply:* @Harel Shein

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-08-30 00:56:18
 +
+

*Thread Reply:* @Jason it depends on the data source. What sort of data are you trying to read? Is it in a hive metastore? Is it on an S3 bucket? Is it a delta file format?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason + (shzhan@coupang.com) +
+
2022-08-30 13:51:03
 +
+

*Thread Reply:* I tried read hive megastore on s3 and cave file on local. All are miss the columnlineage

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-08-31 00:33:17
 +
+

*Thread Reply:* @Jason - Sorry, you'll have to translate a bit for me. Can you share a snippet of code you're using to do the read and write? Is it a special package you need to install or is it just using the hadoop standard for S3? https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/index.html

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason + (shzhan@coupang.com) +
+
2022-08-31 20:00:47
 +
+

*Thread Reply:* spark.read \ + .option("header", "true") \ + .option("inferschema", "true") \ + .csv("data/input/batch/wikidata.csv") \ + .write \ + .mode('overwrite') \ + .csv("data/output/batch/python-sample.csv")

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason + (shzhan@coupang.com) +
+
2022-08-31 20:01:21
 +
+

*Thread Reply:* This is simple code run on my local for testing

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-08-31 21:41:31
 +
+

*Thread Reply:* Which version of OpenLineage are you running? You might look at the code on the main branch. This looks like a HadoopFSRelation which I implemented for column lineage but the latest release (0.13.1) does not include it yet.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-08-31 21:42:05
 +
+

*Thread Reply:* Specifically this commit is what implemented it. +https://github.com/OpenLineage/OpenLineage/commit/ce30178cc81b63b9930be11ac7500ed34808edd3

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason + (shzhan@coupang.com) +
+
2022-08-31 22:02:16
 +
+

*Thread Reply:* I see. I use 0.13.0

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harel Shein + (harel.shein@gmail.com) +
+
2022-09-01 12:04:41
 +
+

*Thread Reply:* @Jason we have our monthly release coming up now, so it should be included in 0.14.0 when released today/tomorrow

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason + (shzhan@coupang.com) +
+
2022-09-01 12:52:52
 +
+

*Thread Reply:* Great. Thanks Harel.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Raj Mishra + (hax0755@gmail.com) +
+
2022-08-28 17:46:38
 +
+

Hi! I have ran into some issues and wanted to clarify my doubts. +• Why are input schema changes(column delete, new columns) doesn't show up on the UI. I have changed the input schema for the same job, but I'm not seeing getting updated on the UI. +• Why is there only ever 1 input schema version. Every change I make in input schema, I only see output schema has multiple versions but only 1 version for input schema. +• Is there a reason why can't we see the input schema till the COMPLETE event is posted? +I have used the examples from here. https://openlineage.io/getting-started/ +curl -X POST <http://localhost:5000/api/v1/lineage> \ + -H 'Content-Type: application/json' \ + -d '{ + "eventType": "START", + "eventTime": "2020-12-28T19:52:00.001+10:00", + "run": { + "runId": "d46e465b-d358-4d32-83d4-df660ff614dd" + }, + "job": { + "namespace": "my-namespace", + "name": "my-job" + }, + "inputs": [{ + "namespace": "my-namespace", + "name": "my-input" + }], + "producer": "<https://github.com/OpenLineage/OpenLineage/blob/v1-0-0/client>" + }' +curl -X POST <http://localhost:5000/api/v1/lineage> \ + -H 'Content-Type: application/json' \ + -d '{ + "eventType": "COMPLETE", + "eventTime": "2020-12-28T20:52:00.001+10:00", + "run": { + "runId": "d46e465b-d358-4d32-83d4-df660ff614dd" + }, + "job": { + "namespace": "my-namespace", + "name": "my-job" + }, + "outputs": [{ + "namespace": "my-namespace", + "name": "my-output", + "facets": { + "schema": { + "_producer": "<https://github.com/OpenLineage/OpenLineage/blob/v1-0-0/client>", + "_schemaURL": "<https://github.com/OpenLineage/OpenLineage/blob/v1-0-0/spec/OpenLineage.json#/definitions/SchemaDatasetFacet>", + "fields": [ + { "name": "a", "type": "VARCHAR"}, + { "name": "b", "type": "VARCHAR"} + ] + } + } + }], + "producer": "<https://github.com/OpenLineage/OpenLineage/blob/v1-0-0/client>" + }' +Changing the inputs schema for START doesn't change the schema input version and doesn't update the UI. +Thanks!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-08-29 05:29:52
 +
+

*Thread Reply:* Reading dataset - which input dataset implies - does not mutate the dataset 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-08-29 05:30:14
 +
+

*Thread Reply:* If you change the dataset, it would be represented as some other job with this datasets in the outputs list

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Raj Mishra + (hax0755@gmail.com) +
+
2022-08-29 12:42:55
 +
+

*Thread Reply:* So, changing the input dataset will always create new output data versions? Sorry I have trouble understanding this, but if the input is changing, shouldn't the input data set will have different versions?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-09-01 08:35:42
 +
+

*Thread Reply:* @Raj Mishra if input is changing, there should be something else in your data infrastructure that changes this dataset - and it should emit this dataset as output

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Iftach Schonbaum + (iftach.schonbaum@hunters.ai) +
+
2022-08-29 12:21:52
 +
+

Hi Everyone, new here. i went thourhg the docs and examples. cant seem to understand how can i model views on top of base tables if not from a data processing job but rather via modeling something static that is coming from some software internals. i.e. i want to issue the lineage my self rather it will learn it dynamically from some Airflow DAG or spark DAG

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-08-29 12:35:32
 +
+

*Thread Reply:* I think you want to emit raw events using python or java client: https://openlineage.io/docs/client/python

+
+
openlineage.io
+ + + +
Python | OpenLineage Docs
+
+ Overview +
+ + + + + + +
Original URL: https://openlineage.io/docs/client/python
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-08-29 12:35:46
 +
+

*Thread Reply:* (docs in progress 😉)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Iftach Schonbaum + (iftach.schonbaum@hunters.ai) +
+
2022-08-30 02:07:02
 +
+

*Thread Reply:* can you give a hind what should i look for for modeling a dataset on top of other dataset? potentially also map columns?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Iftach Schonbaum + (iftach.schonbaum@hunters.ai) +
+
2022-08-30 02:12:50
 +
+

*Thread Reply:* i can only see that i can have a dataset as input to a job run and not for another dataset

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-09-01 08:34:35
 +
+

*Thread Reply:* Not sure I understand - jobs process input datasets into output datasets. There is always something that can be modeled into a job that consumes input and produces output.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Iftach Schonbaum + (iftach.schonbaum@hunters.ai) +
+
2022-09-01 10:30:51
 +
+

*Thread Reply:* so openlineage force me to put a job between datasets? does not fit our use case

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Iftach Schonbaum + (iftach.schonbaum@hunters.ai) +
+
2022-09-01 10:31:09
 +
+

*Thread Reply:* unless we can some how easily hide the process that does that on the graph.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason + (shzhan@coupang.com) +
+
2022-08-29 20:41:19
 +
+

QQ, I saw that spark Column level lineage start with open lineage 0.9.+ version with spark 3.+, Does it mean it needs to run lower than open lineage 0.9 if our spark is 2.3 or 2.4?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-08-30 04:44:06
 +
+

*Thread Reply:* I don't think it will work for Spark 2.X.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason + (shzhan@coupang.com) +
+
2022-08-30 13:42:20
 +
+

*Thread Reply:* Is there have plan to support spark 2.x?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-08-30 14:00:38
 +
+

*Thread Reply:* Nope - on the other hand we plan to drop any support for it, as it's unmaintained for quite a bit and vendors are dropping support for it too - afaik Databricks in April 2023.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason + (shzhan@coupang.com) +
+
2022-08-30 17:19:43
 +
+

*Thread Reply:* I see. Thanks. Amazon Emr still support spark 2.x

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-08-30 01:15:10
 +
+

Spark Integration: Handling Data Source V2 API datasets

+ +

Is it expected that a DataSourceV2 relation has a start event with inputs and outputs but a complete event with only outputs? Based on @Michael Collado’s previous comments, I think it's fair to say YES this is expected and we just need to handle it. https://openlineage.slack.com/archives/C01CK9T7HKR/p1645037070719159?thread_ts=1645036515.163189&cid=C01CK9T7HKR

+ +

@Hanna Moazam and I noticed this behavior when we looked at the Cosmos Db visitor and then reproduced it for the Iceberg visitor. We traced it down to the fact that the AbstractQueryPlanInputDatasetBuilder (which is the parent of DataSourceV2RelationInputDatasetBuilder) has an isDefinedAt that only includes SparkListenerJobStart and SparkListenerSQLExecutionStart

+ +

This means an Iceberg COMPLETE event will NEVER contain inputs because the isDefinedAt will always be false (since COMPLETE only fires for JobEnd and ExecutionEnd events). Does that sound correct (@Paweł Leszczyński)?

+ +

It seems that Delta tables (or at least Delta on Databricks) does not follow this same code path and as a result our complete events includes outputs AND inputs.

+
+ + +
+ + + } + + Michael Collado + (https://openlineage.slack.com/team/U01NNCBCP6K) +
+ + + +
+
+ <p>Hey, I <a href="https://github.com/OpenLineage/OpenLineage/pull/490#issuecomment-1042011803">responded on the issue</a>, but just to make it clear for everyone, the OL events for a run are not expected to be an accumulation of all past events. Events should be treated as additive by the backend - each event can post what information it has about the run and the backend is responsible for constructing a holistic picture of the run</p> +
+ + + + + + +
Original URL: https://openlineage.slack.com/archives/C01CK9T7HKR/p1645037070719159?thread_ts=1645036515.163189&amp;cid=C01CK9T7HKR
+ + +
+ + Thread in Slack Conversation +
+ + +
+ + + +
+ 👀 Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-09-01 05:56:13
 +
+

*Thread Reply:* At least for Iceberg I've done it, since I want to emit DatasetVersionDatasetFacet for input dataset only at START - and after I finish writing the dataset might have different version than before writing.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-09-01 05:58:59
 +
+

*Thread Reply:* Same should be for output AFAIK - output version should be emitted only on COMPLETE, since the version changes after I finish writing.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-09-01 09:52:30
 +
+

*Thread Reply:* Ah! Okay, so this still requires us to truly combine START and COMPLETE to get a TOTAL picture of the entire run. Is that fair?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-09-01 10:30:41
 +
+

*Thread Reply:* Yes

+ + + +
+ 👍 Will Johnson +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-09-01 10:31:21
 +
+

*Thread Reply:* As usual, thank you Maciej for the responses and insights!

+ + + +
+ 🙌 Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason + (shzhan@coupang.com) +
+
2022-08-31 22:19:44
 +
+

QQ team, I use spark sql with openlineage namespace weblog: spark.sql(“select ** from weblog where dt=‘1’”).write.orc(“…”) there have two issues 1, there have no upstream dataset weblog on Marquez UI. 2, there have new namespace s3-cdp-prod-hive created. It should the bucket of s3. Am I missing something? Thanks

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason + (shzhan@coupang.com) +
+
2022-09-07 14:13:34
 +
+

*Thread Reply:* Anyone can help for it? Does I miss something

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason + (shzhan@coupang.com) +
+
2022-08-31 22:21:57
 +
+

Here is the Marquez UI

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-09-01 07:34:24
 +
+

Hi everyone, I’m opening up a vote on this month’s OpenLineage release. 3 +1s from committers will authorize. Additions include support for KustoRelationHandler in Kusto (Azure Data Explorer) and for ABFSS and Hadoop Logical Relation, both in the Spark integration. All commits can be found here: https://github.com/OpenLineage/OpenLineage/compare/0.13.1...HEAD. Thanks in advance!

+ + + +
+ ➕ Maciej Obuchowski, Ross Turk, Paweł Leszczyński, Will Johnson, Hanna Moazam +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-09-01 13:18:59
 +
+

*Thread Reply:* Thanks. The release is authorized. It will be initiated within 2 business days.

+ + + +
+ 🙌 Will Johnson, Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
srutikanta hota + (srutikanta.hota@gmail.com) +
+
2022-09-05 07:57:02
 +
+

Is there a reference on how to deploy openlineage on a Non AWS infrastructure ?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-09-08 10:31:44
 +
+

*Thread Reply:* Which integration are you looking to implement?

+ +

And what environment are you looking to deploy it on? The Cloud? On-Prem?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
srutikanta hota + (srutikanta.hota@gmail.com) +
+
2022-09-08 10:40:11
 +
+

*Thread Reply:* We are planning to deploy on premise with Kerberos as authentication for postgres

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-09-08 11:27:06
 +
+

*Thread Reply:* Ah! Are you planning on running Marquez as well and that is your main concern or are you planning on building your own store of OpenLineage Events and using the SQL integration to generate those events?

+ +

https://github.com/OpenLineage/OpenLineage/tree/main/integration

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
srutikanta hota + (srutikanta.hota@gmail.com) +
+
2022-09-08 11:33:44
 +
+

*Thread Reply:* I am looking to deploy Marquez on-prem with onprem postgres as back-end with Kerberos authentication.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
srutikanta hota + (srutikanta.hota@gmail.com) +
+
2022-09-08 11:34:32
 +
+

*Thread Reply:* Is the the right forum for Marquez as well or there is different slack channel for Marquez available

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-09-08 11:46:35
 +
+

*Thread Reply:* https://bit.ly/MarquezSlack

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-09-08 11:47:14
 +
+

*Thread Reply:* There is another slack channel just for Marquez! That might be a better spot with more dedicated Marquez developers.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-09-06 15:52:32
 +
+

@channel +OpenLineage 0.14.0 is now available! +We added: +• Support ABFSS and Hadoop Logical Relation in Column-level lineage #1008 @wjohnson +• Add Kusto relation visitor #939 @hmoazam +• Add ColumnLevelLineage facet doc #1020 @julienledem +• Include symlinks dataset facet #935 @pawel-big-lebowski +• Add support for dbt 1.3 beta’s metadata changes #1051 @mobuchowski +• Support Flink 1.15 #1009 @mzareba382 +• Add Redshift dialect to the SQL integration #1066 @mobuchowski +We changed: +• Make the timeout configurable in the Spark integration #1050 @tnazarew +We fixed: +• Add a dialect parameter to Great Expectations SQL parser calls #1049 @collado-mike +• Fix Delta 2.1.0 with Spark 3.3.0 #1065 @pawel-big-lebowski +Release: https://github.com/OpenLineage/OpenLineage/releases/tag/0.14.0 +Changelog: https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md +Commit history: https://github.com/OpenLineage/OpenLineage/compare/0.13.1...0.14.0 +Maven: https://oss.sonatype.org/#nexus-search;quick~openlineage +PyPI: https://pypi.org/project/openlineage-python/

+ + + +
+ ❤️ Willy Lulciuc, Howard Yoo, Alexander Wagner, Hanna Moazam, Minkyu Park, Grayson Stream, Paweł Leszczyński, Maciej Obuchowski, Conor Beverland, Jason +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2022-09-06 15:54:30
 +
+

*Thread Reply:* Thanks for breaking up the changes in the release! Love the new format 💯

+ + + +
+ 🙌 Michael Robinson +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-09-07 09:05:35
 +
+

Hello all, I’m requesting a patch release to fix a bug in the Spark integration. Currently, OpenlineageSparkListener fails when no openlineage.timeout is provided. PR #1069 by @Paweł Leszczyński, merged today, will fix it. As per our policy here, 3 +1s from committers will authorize an immediate release.

+ + + +
+ ➕ Paweł Leszczyński, Maciej Obuchowski, Howard Yoo, Willy Lulciuc, Ross Turk, Julien Le Dem +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2022-09-07 10:00:11
 +
+

*Thread Reply:* Is PR #1069 all that’s going in 0.14.1 ?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-09-07 10:27:39
 +
+

*Thread Reply:* There’s also 1058. 1069 is urgently needed. We can technically wait…

+ + + +
+ 🙌 Willy Lulciuc +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-09-07 10:30:31
 +
+

*Thread Reply:* (edited prior message because I’m not sure how accurately I was describing the issue)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2022-09-07 10:39:32
 +
+

*Thread Reply:* Thanks for clarifying!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-09-07 10:50:29
 +
+

*Thread Reply:* Thanks, all. The release is authorized.

+ + + +
+ ❤️ Willy Lulciuc +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-09-07 11:04:39
 +
+

*Thread Reply:* 1058 also fixes some bugs

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Iftach Schonbaum + (iftach.schonbaum@hunters.ai) +
+
2022-09-08 01:55:41
 +
+

Hello all, question: Views on top of base table is also a use case for lineage and there is no job in between. i dont seem to find a way to have a dataset on top of others to represent a view on top of tables. is there a way to do that without a job in between?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-09-08 04:41:07
 +
+

*Thread Reply:* Usually there is something creating the view, for example dbt materialization: https://docs.getdbt.com/docs/building-a-dbt-project/building-models/materializations

+ +

Besides that, there is this proposal that did not get enough love yet https://github.com/OpenLineage/OpenLineage/issues/323

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Iftach Schonbaum + (iftach.schonbaum@hunters.ai) +
+
2022-09-08 04:53:23
 +
+

*Thread Reply:* but we are not working iwth dbt. we try to model lineage of our internal view/tables hirarchy which is related to a propriety application of ours. so we like OpenLineage that lets me explicily model stuff and not only via scanning some DW. but in that case we dont want a job in between.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Iftach Schonbaum + (iftach.schonbaum@hunters.ai) +
+
2022-09-08 04:58:47
 +
+

*Thread Reply:* this PR does not seem to support lineage between datasets

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-09-08 12:49:48
 +
+

*Thread Reply:* This is something core to the OpenLineage design - the lineage relationships are defined as dataset-job-dataset, not dataset-dataset.

+ +

In OpenLineage, something observes the lineage relationship being created.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-09-08 12:50:13
 +
+

*Thread Reply:*

+ +
+
+ + + + + + +
+ + +
+ 🙌 Will Johnson, Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-09-08 12:51:15
 +
+

*Thread Reply:* It’s a bit different from some other lineage approaches, but OL is intended to be a push model. A job is observed as it runs, metadata is pushed to the backend.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-09-08 12:54:27
 +
+

*Thread Reply:* so in this case, according to openlineage 🙂, the job would be whatever runs within the pipeline that creates the view. very operational point of view.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Iftach Schonbaum + (iftach.schonbaum@hunters.ai) +
+
2022-09-11 12:27:42
 +
+

*Thread Reply:* but what about the view definition use case? u have lineage of columns in view/base table relation ships

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Iftach Schonbaum + (iftach.schonbaum@hunters.ai) +
+
2022-09-11 12:28:05
 +
+

*Thread Reply:* how would you model that in OpenLineage? would you create a dummy job ?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Iftach Schonbaum + (iftach.schonbaum@hunters.ai) +
+
2022-09-11 12:31:57
 +
+

*Thread Reply:* would you say that because this is my use case i might better choose some other lineage tool?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Iftach Schonbaum + (iftach.schonbaum@hunters.ai) +
+
2022-09-11 12:33:04
 +
+

*Thread Reply:* for the context: i am not talking about some view and table definitions in some warehouse e.g. SF but its internal data processing mechanism with propriety view/tables definition (in Flink SQL) and we want to push this metadata for visibility

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-09-12 17:20:13
 +
+

*Thread Reply:* Ah, gotcha. Yeah, I would say it’s probably best to create a job in this case. You can send the view definition using a sourcecodefacet, so it will be collected as well. You’d want to send START and STOP events for it.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-09-12 17:22:03
 +
+

*Thread Reply:* regarding the PR linked before, you are right - I wonder if someday the spec should have a way to express “the system was made aware that these datasets are related, but did not observe the relationship being created so it can’t tell you i.e. how long it took or whether it changed over time”

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-09-09 10:25:21
 +
+

@channel +OpenLineage 0.14.1 is now available! +We fixed: +• Fix Spark integration issues including error when no openlineage.timeout #1069 @pawel-big-lebowski +Bug fixes were also included in this release. +Release: https://github.com/OpenLineage/OpenLineage/releases/tag/0.14.1 +Changelog: https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md +Commit history: https://github.com/OpenLineage/OpenLineage/compare/0.14.0...0.14.1 +Maven: https://oss.sonatype.org/#nexus-search;quick~openlineage +PyPI: https://pypi.org/project/openlineage-python/

+ + + +
+ 🙌 Maciej Obuchowski, Willy Lulciuc, Howard Yoo, Francis McGregor-Macdonald, AMRIT SARKAR +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
data_fool + (data.fool.me@gmail.com) +
+
2022-09-09 13:52:39
 +
+

Hello, any future plans for integrating Airbyte with openlineage?

+ + + +
+ 👋 Willy Lulciuc, Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2022-09-09 14:01:13
 +
+

*Thread Reply:* Hey, @data_fool! Not in the near term. but of course we’d love to see this happen. We’re open to having an Airbyte integration driven by the community. Want to open an issue to start the discussion?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
data_fool + (data.fool.me@gmail.com) +
+
2022-09-09 15:36:20
 +
+

*Thread Reply:* hey @Willy Lulciuc, Yep, will open an issue. Thanks!

+ + + +
+ 🙌 Willy Lulciuc +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Hubert Dulay + (hubert.dulay@gmail.com) +
+
2022-09-10 22:00:10
 +
+

Hi can you create lineage across namespaces? Thanks

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2022-09-12 19:26:25
 +
+

*Thread Reply:* yes!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
srutikanta hota + (srutikanta.hota@gmail.com) +
+
2022-09-26 10:31:56
 +
+

*Thread Reply:* Any example or ticket on how to lineage across namespace

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Iftach Schonbaum + (iftach.schonbaum@hunters.ai) +
+
2022-09-12 02:27:49
 +
+

Hello, Does OpenLineage support column level lineage?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-09-12 04:56:13
 +
+

*Thread Reply:* Yes https://openlineage.io/blog/column-lineage/

+
+
openlineage.io
+ + + +
The Current State of Column-level Lineage
+
+ Column-level lineage helps organizations navigate a complex regulatory landscape. +
+ + + + + + +
Original URL: https://openlineage.io/blog/column-lineage/
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2022-09-22 02:18:45
 +
+

*Thread Reply:* • More details on Spark & Column level lineage integration: https://openlineage.io/docs/integrations/spark/spark_column_lineage +• Proposal on how to implement column level lineage in Marquez (implementation is currently work in progress): https://github.com/MarquezProject/marquez/blob/main/proposals/2045-column-lineage-endpoint.md +@Iftach Schonbaum let us know if you find the information useful.

+
+
openlineage.io
+ + + +
Column Level Lineage | OpenLineage Docs
+
+ Column level lineage for Spark is turned on by default and requires no additional work to be done. The following documentation describes its internals. +
+ + + + + + +
Original URL: https://openlineage.io/docs/integrations/spark/spark_column_lineage
+ + + +
+
+ + + + +
<https://github.com/MarquezProject/marquez/blob/main/proposals/2045-column-lineage-endpoint.md | 2045-column-lineage-endpoint.md>
+
+ <p><strong>Proposal: Column lineage endpoint proposal</strong></p> + +<p>Author(s): <a href="https://github.com/julienledem">@julienledem</a>, <a href="https://github.com/pawel-big-lebowski">@pawel-big-lebowski</a></p> + +<p>Created: 20022-08-18</p> + +<p>Dicussion: <a href="https://github.com/MarquezProject/marquez/issues/2045">column lineage endpoint issue #2045</a></p> + +<p><strong>Overview</strong> +<strong>Use cases</strong></p> + +<p>• Find the current upstream dependencies of a column. A column in a dataset is derived from columns in upstream datasets. +• See column-level lineage in the dataset level lineage when available. +• Retrieve point-in-time upstream lineage for a dataset or a column. What did the lineage look like yesterday compared to today?</p> + +<p><strong>Existing elements</strong></p> + +<p>• OpenLineage defines a [column-level lineage facet]- (<a href="https://github.com/OpenLineage/OpenLineage/blob/main/spec/facets/ColumnLineageDatasetFacet.json">https://github.com/OpenLineage/OpenLineage/blob/main/spec/facets/ColumnLineageDatasetFacet.json</a>). +• Marquez has a lineage endpoint <code>GET /api/v1/lineage</code> that returns the current lineage graph connected to a job or a dataset</p> + +<p><strong>Column lineage characteristics and general assumptions</strong></p> + +<p>Column level lineage is a different lineage graph due to a different node granularity - kind of zoomed-in view of existing lineage. Instead of datasets and jobs being lineage graph nodes, each dataset field becomes a node. Additionally, there are edges between dataset fields, instead of datasets itself. Thus, enriching existing lineage with column lineage information would not be sufficient. That’s why we propose another API endpoint with column lineage graph.</p> + +<p>Upstream and downstream edges do have different characteristics. An output dataset is always produced by a single version of input dataset (one upstream), while a single input datset version can have multiple output dataset versions. Lineage graph can be then easily flooded by downstream subgraph which blurs the overall view. That's why we consider an upstream column lineage as a default one. Downstream lineage will be returned only when requested explicitly.</p> + +<p><strong>New Elements</strong></p> + +<p>We propose the following changes:</p> + +<p>• Add column lineage to the dataset resource endpoint. Column lineage will NOT be added to existing <code>/lineage</code> endpoint as it may be a heavy database operation run on each lineage graph's node which we want to avoid. Based on that, column level lineage get be requested per dataset in separate requests when required. +• A new column-lineage endpoint leveraging the column lineage facet to retrieve lineage for a given column. +• Point-in-time upstream (dataset or column level) lineage given a version of a dataset.</p> + +<p><strong>Proposal</strong> +<strong>Add column lineage to existing datasets endpoint</strong></p> + +<p>In the <code>GET /api/v1/namespaces/{namespace}/datasets</code> api, add column lineage facet to returned dataset resource.</p> + +<p><strong>Add a column-level-lineage endpoint:</strong></p> + +<p>New endpoints to retrieve a column lineage of a single field or a whole dataset will be added:</p> + +<pre><code>GET /column-lineage?nodeId=dataset:{namespace}:{dataset} +GET /column-lineage?nodeId=datasetField:{namespace}:{dataset}:{field} +</code></pre> + +<p>For example:</p> + +<pre><code>GET /column-lineage?nodeId=dataset:food_delivery:public.delivery_7_days +GET /column-lineage?nodeId=datasetField:food_delivery:public.delivery_7_days:a +</code></pre> + +<p>Although creating a new endpoint, we would like to reuse existing data structures with a new <code>NodeType.FIELD</code> introduced.</p> + +<p>The logic returns dataset field node:</p> + +<pre><code>GET /column-lineage?nodeId=datasetField:db1:table1:a +... +{ + graph: [ + { + "id": "datasetField:db1:table1:a", + "type": "DATASET_FIELD", + "data": { + "namespace": "DB1", + "name": "table2", + "field": "a", + "type": "integer", + "transformationDescription": "identical", + "transformationType": "IDENTITY", + "inputFields": [ + { "namespace": "DBA", "name": "tableA", "field": "columnA"}, + { "namespace": "DBB", "name": "tableB", "field": "columnB"}, + { "namespace": "DBC", "name": "tableC", "field": "columnC"} + ] + "inEdges": [ + { + "origin": "datasetField:db1:table1:a", + "destination": "datasetField:DBA:tableA:columnA" + }, + { + "origin": "datasetField:db1:table1:a", + "destination": "datasetField:DBB:tableB:columnB" + }, + { + "origin": "datasetField:db1:table1:a", + "destination": "datasetField:DBB:tableB:columnC" + } + ], + }, + ... + # Input fields, present within "inEdges", can be also returned within a graph due to a `depth` parameter greate than 0. + } + ] +} +</code></pre> + +<p>The <code>depth</code> parameter controls how many edges, from a given dataset field, shall be returned. The default is set to <code>0</code>. In case of default equal <code>1</code>, each <code>inputField</code> will be returned as a separate node within a response graph with <code>inputFields</code> used to produce it. Please note that extending depth may increase the graph size and affect request performance.</p> + +<p>The endpoints above fetches upstream column-lineage for given dataset field or all fields within a dataset. Downstream column lineage is turned off by default. However, this can be turned on with an extra <code>withDownstream</code> parameter like:</p> + +<pre><code>GET /column-lineage?nodeId=datasetField:food_delivery:public.delivery_7_days:a&amp;amp;withDownstream=true + +</code></pre> + +<p>This will include <code>outEdges</code> within the returned node of the graph.</p> + +<p><strong>Point in time upstream lineage</strong></p> + +<p>Point in time lineage for newly proposed <code>/api/v1/column-lineage</code> endpoint:</p> + +<pre><code>GET /column-lineage?nodeId=dataset_field:food_delivery:public.delivery_7_days:a&amp;amp;datasetVersion=123e4567-e89b-12d3-a456-426614174000 +GET /column-lineage?nodeId=dataset_field:food_delivery:public.delivery_7_days:a&amp;amp;lineageAt=1661846242 +</code></pre> + +<p>Point in time can be controlled by:</p> + +<p>• <strong>datasetVersion</strong> - uuid of a specific dataset version, +• <strong>lineageAt</strong> - which contains a unix timestamp.</p> + +<p>When <strong>lineageAt</strong> specified, the latest dataset version before timestamp will be found. Regardles <strong>datasetVersion</strong> or <strong>lineageAt</strong> parameters applied, responses will be the same as below:</p> + +<pre><code>{ + graph: [ + { +&amp;lt; "id": "datasetField:db1:table1:a", +&amp;gt; "id": "datasetField:db1:table1:a#{VERSION UUID}", + "type": "DATASET_FIELD", + "data": { + .... +} +</code></pre> + +<p><strong>Implementation</strong> +<strong>columne lineage facet in dataset resource endpoint</strong></p> + +<p>Adding the columnLineage facet requires a formatting of existing facet data (work in progress).</p> + +<p><strong>column lineage endpoint</strong></p> + +<p>The <code>/column-lineage</code> endpoint leverages the <code>/lineage</code> endpoint and then filters down the payload to return the expected result.</p> + +<p><strong>point-in-time upstream lineage</strong></p> + +<p>The point-in-time upstream lineage leverages the run to dataset version relation to track back the lineage of a given dataset of job version. <br /> +Dataset version -&gt; run that produced it -&gt; consumed Dataset Versions.</p> + +<p><strong>Next Steps</strong></p> + +<p>Review of this proposal and production of detailed design for the implementation.:</p> +
+ + + + + + + +
+ + <a href="https://github.com/MarquezProject/marquez">MarquezProject/marquez</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paul Lee + (paullee@lyft.com) +
+
2022-09-12 15:29:12
 +
+

where can i find docs on just simply using extractors? without marquez. for example, a basic BashOperator on Airflow 1.10.15

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paul Lee + (paullee@lyft.com) +
+
2022-09-12 15:30:08
 +
+

*Thread Reply:* or is it automatic for anything that exists in extractors/?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Howard Yoo + (howard.yoo@astronomer.io) +
+
2022-09-12 15:30:16
 +
+

*Thread Reply:* Yes

+ + + +
+ 👍 Paul Lee +
+ +
+ :gratitude_thank_you: Paul Lee +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paul Lee + (paullee@lyft.com) +
+
2022-09-12 15:31:12
 +
+

*Thread Reply:* so anything i add to extractors directory with the same name as the operator will automatically extract the metadata from the operator is that correct?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Howard Yoo + (howard.yoo@astronomer.io) +
+
2022-09-12 15:31:31
 +
+

*Thread Reply:* Well, not entirely

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Howard Yoo + (howard.yoo@astronomer.io) +
+
2022-09-12 15:31:47
 +
+

*Thread Reply:* please take a look at the source code of one of the extractors

+ + + +
+ 👍 Paul Lee +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Howard Yoo + (howard.yoo@astronomer.io) +
+
2022-09-12 15:32:13
 +
+

*Thread Reply:* also, there are docs available at openlineage.io/docs

+ + + +
+ 🙏 Paul Lee +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paul Lee + (paullee@lyft.com) +
+
2022-09-12 15:33:45
 +
+

*Thread Reply:* ok, i'll take a look. i think one thing that would be helpful is having a custom setup without marquez. a lot of the docs or videos i found were integrated with marquez

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Howard Yoo + (howard.yoo@astronomer.io) +
+
2022-09-12 15:34:29
 +
+

*Thread Reply:* I see. Marquez is a openlineage backend that stores the lineage data, so many examples do need them.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Howard Yoo + (howard.yoo@astronomer.io) +
+
2022-09-12 15:34:47
 +
+

*Thread Reply:* If you do not want to run marquez but just test out the openlineage, you can also take a look at OpenLineage Proxy.

+ + + +
+ 👍 Paul Lee +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paul Lee + (paullee@lyft.com) +
+
2022-09-12 15:35:14
 +
+

*Thread Reply:* awesome thanks Howard! i'll take a look at these resources and come back around if i need to

+ + + +
+ 👍 Howard Yoo +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-09-12 16:01:45
 +
+

*Thread Reply:* http://openlineage.io/docs/integrations/airflow/extractor - this is the doc you might want to read

+
+
openlineage.io
+ + + +
Custom extractors | OpenLineage Docs
+
+ OpenLineage integration works by detecting which Airflow operators your dag is using, and extracting lineage +
+ + + + + + +
Original URL: http://openlineage.io/docs/integrations/airflow/extractor
+ + + +
+ + + +
+ 🎉 Paul Lee +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paul Lee + (paullee@lyft.com) +
+
2022-09-12 17:08:49
 +
+

*Thread Reply:* yeah, saw that doc earlier. thanks @Maciej Obuchowski appreciate it 🙏

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jay + (sanjay.sudhakaran@trovemoney.co.nz) +
+
2022-09-21 20:55:24
 +
+

Hey team! I’m pretty new to the field in general

+ +

In the real world, I would be running pyspark scripts on AWS EMR. Could you explain to me how the metadata is sent to Marquez from my pyspark script, and where it’s persisted?

+ +

Would I need to set up an S3 bucket to store the lineage data?

+ +

I’m also unsure about how I would run the Marquez UI on AWS - Would I need to have an EC2 instance running permanently in order to access that UI?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jay + (sanjay.sudhakaran@trovemoney.co.nz) +
+
2022-09-21 20:57:39
 +
+

*Thread Reply:* In my head, I have:

+ +

Pyspark script -> Store metadata in S3 -> Marquez UI gets data from S3 and displays it

+ +

I suspect this is incorrect?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2022-09-22 02:14:50
 +
+

*Thread Reply: It’s more like: you add openlineage jar to Spark job, configure it what to do with the events. Popular options are: + * sent to rest endpoint (like Marquez), + * send as an event onto Kafka, + * print it onto console +There is no S3 in between Spark & Marquez by default. +Marquez serves both as an API where events are sent and UI to investigate them.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jay + (sanjay.sudhakaran@trovemoney.co.nz) +
+
2022-09-22 17:36:10
 +
+

*Thread Reply:* Yeah S3 was just an example for a storage option.

+ +

I actually found the answer I was looking for, turns out I had to look at Marquez documentation: +https://marquezproject.ai/resources/deployment/

+ +

The answer is that Marquez uses a postgres instance to persist the metadata it is given. Thanks for your time though! I appreciate the effort 🙂

+ + + +
+ 👍 Kevin Adams +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Hanbing Wang + (doris.wang200902@gmail.com) +
+
2022-09-25 17:06:41
 +
+

Hello team, +For the OpenLineage Spark, even when I processed one Spark sql query (CTAS Create Table As Select), I will received multiple events back (2+ Start events, 2 Complete events). +I try to understand why OpenLineage need to send back that much events, and what is the primary difference between Start VS Start events, Start VS Complete events? +Do we have any doc can help me understand more on it? +Thanks

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-09-26 00:27:05
 +
+

*Thread Reply:* The Spark execution model follows:

+ +
  1. Spark SQL Execution Start event
  2. Spark Job Start event
  3. Spark Job End event
  4. Spark SQL Execution End event +As a result, OpenLineage tracks all of those execution and jobs. There is a proposed plan to distinguish between those events (e.g. you wouldn't get two starts but one Start and one Job Start or something like that).
    5. +
+ +

You should collect all of these events in order to be sure you are receiving all the data since each event may contain a subset of the complete facets that represent what occurred in the job.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Hanbing Wang + (doris.wang200902@gmail.com) +
+
2022-09-26 15:16:26
 +
+

*Thread Reply:* Thanks @Will Johnson +Can I get an example of how the proposed plan can be used to distinguish between start and job start events? +Because I compare the 2 starts events I got, only the event_time is different, all other information are the same.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Hanbing Wang + (doris.wang200902@gmail.com) +
+
2022-09-26 15:30:34
 +
+

*Thread Reply:* One followup question, if I process multiple queries in one command, for example (Drop + Create Table + Insert Overwrite), should I expected for +(1). 1 Spark SQL execution start event +(2). 3 Spark job start event (Each query has a job start event ) +(3). 3 Spark job end event (Each query has a job end event ) +(4). 1 Spark SQL execution end event

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-09-27 10:25:47
 +
+

*Thread Reply:* Re: Distinguish between start and job start events. There was a proposal to differentiate the two (https://github.com/OpenLineage/OpenLineage/issues/636) but the current discussion is here: https://github.com/OpenLineage/OpenLineage/issues/599 As it currently stands, there is not a way to tell which one is which (I believe). The design of OpenLineage is such that you should consume ALL events under the same run id and job name / namespace.

+ +

Re: Multiple Queries in One Command: This is where Spark's execution model comes into play. I believe each one of those commands are executed sequentially and as a result, you'd actually get three execution start and three execution end. If you chose DROP + Create Table As Select, that would be only two commands and thus only two execution start events.

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/issues/636|#636 [INTEGRATION][SPARK] Change event types for JobStart and JobEnd events>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/issues/599|#599 Spec: clarify event order lifecycle>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Hanbing Wang + (doris.wang200902@gmail.com) +
+
2022-09-27 16:49:37
 +
+

*Thread Reply:* Thanks a lot for your help 🙏 @Will Johnson, +For multiple queries in one command, I still have a confused place why Drop + CreateTable and Drop + CreateTableAsSelect act different.

+ +

When I test Drop + Create Table +Query: +DROP TABLE IF EXISTS shadow_test.test_sparklineage_4; CREATE TABLE IF NOT EXISTS shadow_test.test_sparklineage_4 (val INT, region STRING) PARTITIONED BY ( ds STRING ) STORED AS PARQUET; +I only received 1 start + 1 complete event +And the events only contains DropTableCommandVisitor/DropTableCommand. +I expected we should also received start and complete events for CreateTable query with CreateTableCommanVisitor/CreateTableComman .

+ +

But when I test Drop + Create Table As Select +Query: +DROP TABLE IF EXISTS shadow_test.test_sparklineage_5; CREATE TABLE IF NOT EXISTS shadow_test.test_sparklineage_5 AS SELECT ** from shadow_test.test_sparklineage where ds &gt; '2022-08-24'" +I received 1 start + 1 complete event with DropTableCommandVisitor/DropTableCommand +And 2 start + 2 complete events with CreateHiveTableAsSelectCommandVisitor/CreateHiveTableAsSelectCommand

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-09-27 22:03:38
 +
+

*Thread Reply:* @Hanbing Wang are you running this on Databricks with a hive metastore that is defaulting to Delta by any chance?

+ +

I THINK there are some gaps in OpenLineage because of the way Databricks Delta handles things and now there is Unity catalog that is causing some hiccups as well.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-09-28 09:18:48
 +
+

*Thread Reply:* > For multiple queries in one command, I still have a confused place why Drop + CreateTable and Drop + CreateTableAsSelect act different. +@Hanbing Wang That's basically why we capture all the events (SQL Execution, Job) instead of one of them. We're just inconsistently notified of them by Spark.

+ +

Some computations emit SQL Execution events, some emit Job events, I think majority emits both. This also differs by spark version.

+ +

The solution OpenLineage assumes is having cumulative model of job execution, where your backend deals with possible duplication of information.

+ +

> I THINK there are some gaps in OpenLineage because of the way Databricks Delta handles things and now there is Unity catalog that is causing some hiccups as well. +@Will Johnson would be great if you created issue with some complete examples

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Hanbing Wang + (doris.wang200902@gmail.com) +
+
2022-09-28 15:44:45
 +
+

*Thread Reply:* @Will Johnson and @Maciej Obuchowski Thanks a lot for your help +We are not running on Databricks. +We implemented the OpenLineage Spark listener, and custom the Event Transport which emitting the events to our own events pipeline with a hive metastore. +We are using Spark version 3.2.1 +OpenLineage version 0.14.1

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-09-29 15:16:28
 +
+

*Thread Reply:* Ooof! @Hanbing Wang then I'm not certain why you're not receiving the extra event 😞 You may need to run your spark cluster in debug mode to step through the Spark Listener.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-09-29 15:17:08
 +
+

*Thread Reply:* @Maciej Obuchowski - I'll add it to my list!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Hanbing Wang + (doris.wang200902@gmail.com) +
+
2022-09-30 15:34:01
 +
+

*Thread Reply:* @Will Johnson Thanks a lot for your help. Let us debug and continue investigating on this issue.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Yujia Yang + (yujia@tubi.tv) +
+
2022-09-26 03:46:19
 +
+

Hi team, I find Openlineage posts a lot for run events to the backend.

+ +

eg. I submit jar to Spark cluster with computations like

+ +
  1. count from table1. --> this will have more than one run events inputs:[table1], outputs:[]
  2. count from table2 --> this will have more than one run events inputs:[table2], outputs:[]
  3. write Seq[(t1, count1), (t2, count2)) to table3. --> this may give inputs:[] outputs [table3] +can I just get one post with a summary telling me, inputs:[table1, table2], outputs:[table3] alongside with a merged columnareLineage?
    4. +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2022-09-28 08:34:20
 +
+

*Thread Reply:* One of assumptions was to create a stateless integration model where multiple events can be sent for a single job run. This has several advantages like sending events for jobs which suddenly fail, sending events immediately, etc.

+ +

The events can be merged then at the backend side. The behavior, you describe, can be then achieved by using backends like Marquez and Marquez API to obtain combined data.

+ +

Currently, we’re developing column-lineage dedicated endpoint in Marquez according to the proposal: https://github.com/MarquezProject/marquez/blob/main/proposals/2045-column-lineage-endpoint.md +This will allow you to request whole column lineage graph based on multiple jobs.

+
+ + + + +
<https://github.com/MarquezProject/marquez/blob/main/proposals/2045-column-lineage-endpoint.md | 2045-column-lineage-endpoint.md>
+
+ <p><strong>Proposal: Column lineage endpoint proposal</strong></p> + +<p>Author(s): <a href="https://github.com/julienledem">@julienledem</a>, <a href="https://github.com/pawel-big-lebowski">@pawel-big-lebowski</a></p> + +<p>Created: 20022-08-18</p> + +<p>Dicussion: <a href="https://github.com/MarquezProject/marquez/issues/2045">column lineage endpoint issue #2045</a></p> + +<p><strong>Overview</strong> +<strong>Use cases</strong></p> + +<p>• Find the current upstream dependencies of a column. A column in a dataset is derived from columns in upstream datasets. +• See column-level lineage in the dataset level lineage when available. +• Retrieve point-in-time upstream lineage for a dataset or a column. What did the lineage look like yesterday compared to today?</p> + +<p><strong>Existing elements</strong></p> + +<p>• OpenLineage defines a [column-level lineage facet]- (<a href="https://github.com/OpenLineage/OpenLineage/blob/main/spec/facets/ColumnLineageDatasetFacet.json">https://github.com/OpenLineage/OpenLineage/blob/main/spec/facets/ColumnLineageDatasetFacet.json</a>). +• Marquez has a lineage endpoint <code>GET /api/v1/lineage</code> that returns the current lineage graph connected to a job or a dataset</p> + +<p><strong>Column lineage characteristics and general assumptions</strong></p> + +<p>Column level lineage is a different lineage graph due to a different node granularity - kind of zoomed-in view of existing lineage. Instead of datasets and jobs being lineage graph nodes, each dataset field becomes a node. Additionally, there are edges between dataset fields, instead of datasets itself. Thus, enriching existing lineage with column lineage information would not be sufficient. That’s why we propose another API endpoint with column lineage graph.</p> + +<p>Upstream and downstream edges do have different characteristics. An output dataset is always produced by a single version of input dataset (one upstream), while a single input datset version can have multiple output dataset versions. Lineage graph can be then easily flooded by downstream subgraph which blurs the overall view. That's why we consider an upstream column lineage as a default one. Downstream lineage will be returned only when requested explicitly.</p> + +<p><strong>New Elements</strong></p> + +<p>We propose the following changes:</p> + +<p>• Add column lineage to the dataset resource endpoint. Column lineage will NOT be added to existing <code>/lineage</code> endpoint as it may be a heavy database operation run on each lineage graph's node which we want to avoid. Based on that, column level lineage get be requested per dataset in separate requests when required. +• A new column-lineage endpoint leveraging the column lineage facet to retrieve lineage for a given column. +• Point-in-time upstream (dataset or column level) lineage given a version of a dataset.</p> + +<p><strong>Proposal</strong> +<strong>Add column lineage to existing datasets endpoint</strong></p> + +<p>In the <code>GET /api/v1/namespaces/{namespace}/datasets</code> api, add column lineage facet to returned dataset resource.</p> + +<p><strong>Add a column-level-lineage endpoint:</strong></p> + +<p>New endpoints to retrieve a column lineage of a single field or a whole dataset will be added:</p> + +<pre><code>GET /column-lineage?nodeId=dataset:{namespace}:{dataset} +GET /column-lineage?nodeId=datasetField:{namespace}:{dataset}:{field} +</code></pre> + +<p>For example:</p> + +<pre><code>GET /column-lineage?nodeId=dataset:food_delivery:public.delivery_7_days +GET /column-lineage?nodeId=datasetField:food_delivery:public.delivery_7_days:a +</code></pre> + +<p>Although creating a new endpoint, we would like to reuse existing data structures with a new <code>NodeType.FIELD</code> introduced.</p> + +<p>The logic returns dataset field node:</p> + +<pre><code>GET /column-lineage?nodeId=datasetField:db1:table1:a +... +{ + graph: [ + { + "id": "datasetField:db1:table1:a", + "type": "DATASET_FIELD", + "data": { + "namespace": "DB1", + "name": "table2", + "field": "a", + "type": "integer", + "transformationDescription": "identical", + "transformationType": "IDENTITY", + "inputFields": [ + { "namespace": "DBA", "name": "tableA", "field": "columnA"}, + { "namespace": "DBB", "name": "tableB", "field": "columnB"}, + { "namespace": "DBC", "name": "tableC", "field": "columnC"} + ] + "inEdges": [ + { + "origin": "datasetField:db1:table1:a", + "destination": "datasetField:DBA:tableA:columnA" + }, + { + "origin": "datasetField:db1:table1:a", + "destination": "datasetField:DBB:tableB:columnB" + }, + { + "origin": "datasetField:db1:table1:a", + "destination": "datasetField:DBB:tableB:columnC" + } + ], + }, + ... + # Input fields, present within "inEdges", can be also returned within a graph due to a `depth` parameter greate than 0. + } + ] +} +</code></pre> + +<p>The <code>depth</code> parameter controls how many edges, from a given dataset field, shall be returned. The default is set to <code>0</code>. In case of default equal <code>1</code>, each <code>inputField</code> will be returned as a separate node within a response graph with <code>inputFields</code> used to produce it. Please note that extending depth may increase the graph size and affect request performance.</p> + +<p>The endpoints above fetches upstream column-lineage for given dataset field or all fields within a dataset. Downstream column lineage is turned off by default. However, this can be turned on with an extra <code>withDownstream</code> parameter like:</p> + +<pre><code>GET /column-lineage?nodeId=datasetField:food_delivery:public.delivery_7_days:a&amp;amp;withDownstream=true + +</code></pre> + +<p>This will include <code>outEdges</code> within the returned node of the graph.</p> + +<p><strong>Point in time upstream lineage</strong></p> + +<p>Point in time lineage for newly proposed <code>/api/v1/column-lineage</code> endpoint:</p> + +<pre><code>GET /column-lineage?nodeId=dataset_field:food_delivery:public.delivery_7_days:a&amp;amp;datasetVersion=123e4567-e89b-12d3-a456-426614174000 +GET /column-lineage?nodeId=dataset_field:food_delivery:public.delivery_7_days:a&amp;amp;lineageAt=1661846242 +</code></pre> + +<p>Point in time can be controlled by:</p> + +<p>• <strong>datasetVersion</strong> - uuid of a specific dataset version, +• <strong>lineageAt</strong> - which contains a unix timestamp.</p> + +<p>When <strong>lineageAt</strong> specified, the latest dataset version before timestamp will be found. Regardles <strong>datasetVersion</strong> or <strong>lineageAt</strong> parameters applied, responses will be the same as below:</p> + +<pre><code>{ + graph: [ + { +&amp;lt; "id": "datasetField:db1:table1:a", +&amp;gt; "id": "datasetField:db1:table1:a#{VERSION UUID}", + "type": "DATASET_FIELD", + "data": { + .... +} +</code></pre> + +<p><strong>Implementation</strong> +<strong>columne lineage facet in dataset resource endpoint</strong></p> + +<p>Adding the columnLineage facet requires a formatting of existing facet data (work in progress).</p> + +<p><strong>column lineage endpoint</strong></p> + +<p>The <code>/column-lineage</code> endpoint leverages the <code>/lineage</code> endpoint and then filters down the payload to return the expected result.</p> + +<p><strong>point-in-time upstream lineage</strong></p> + +<p>The point-in-time upstream lineage leverages the run to dataset version relation to track back the lineage of a given dataset of job version. <br /> +Dataset version -&gt; run that produced it -&gt; consumed Dataset Versions.</p> + +<p><strong>Next Steps</strong></p> + +<p>Review of this proposal and production of detailed design for the implementation.:</p> +
+ + + + + + + +
+ + <a href="https://github.com/MarquezProject/marquez">MarquezProject/marquez</a> +
+ + +
+ + + +
+ :gratitude_thank_you: Yujia Yang +
+ +
+ 👀 Yujia Yang +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
srutikanta hota + (srutikanta.hota@gmail.com) +
+
2022-09-28 09:47:55
 +
+

Is there a provision to include additional MDC properties as part of openlineage ? +Or something like sparkSession.sparkContext().setLocalProperties("key","value")

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2022-09-29 14:30:37
 +
+

*Thread Reply:* Hello @srutikanta hota, could you elaborate a bit on your use case? I'm not sure what you are trying to achieve. Possibly @Paweł Leszczyński will know.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-09-29 15:24:26
 +
+

*Thread Reply:* @srutikanta hota - Not sure what MDC properties stands for but you might take inspiration from the DatabricksEnvironmentHandler Facet Builder: https://github.com/OpenLineage/OpenLineage/blob/65a5f021a1ba3035d5198e759587737a05[…]ark/agent/facets/builder/DatabricksEnvironmentFacetBuilder.java

+ +

You can create a facet that could extract out the properties that you might set from within the spark session.

+ +

I don't think OpenLineage / a Spark Listener can affect the SparkSession itself so you wouldn't be able to SET the properties in the listener.

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/65a5f021a1ba3035d5198e759587737a05b242e1/integration/spark/shared/src/main/java/io/openlineage/spark/agent/facets/builder/DatabricksEnvironmentFacetBuilder.java | DatabricksEnvironmentFacetBuilder.java>
+
+ <pre><code> private Map&amp;lt;String, Object&amp;gt; getDatabricksEnvironmentalAttributes(SparkListenerJobStart jobStart) { + dbProperties = new HashMap&amp;lt;&amp;gt;(); + // These are useful properties to extract if they are available + + List&amp;lt;String&amp;gt; dbPropertiesKeys = + Arrays.asList( + "orgId", + "spark.databricks.clusterUsageTags.clusterOwnerOrgId", + "spark.databricks.notebook.path", + "spark.databricks.job.type", + "spark.databricks.job.id", + "spark.databricks.job.runId", + "user", + "userId", + "spark.databricks.clusterUsageTags.clusterName", + "spark.databricks.clusterUsageTags.azureSubscriptionId"); + dbPropertiesKeys.stream() + .forEach( + (p) -&amp;gt; { + dbProperties.put(p, jobStart.properties().getProperty(p)); + }); + + /**** + ** Azure Databricks makes available a dbutils mount point to list aliased paths to cloud + ** storage. However, that dbutils object is not available inside a spark listener. We must + ** access it via reflection. + **/ + try { + dbutilsClass = Class.forName("com.databricks.dbutils_v1.impl.DbfsUtilsImpl"); + dbutils = (DbfsUtils) dbutilsClass.getDeclaredConstructor().newInstance(); + dbProperties.put("mountPoints", getDatabricksMountpoints(dbutils)); + } catch (Exception e) { + log.warn("Failed to load dbutils in OpenLineageListener"); + dbProperties.put("mountPoints", new ArrayList&amp;lt;DatabricksMountpoint&amp;gt;()); + } + + return dbProperties; + } +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
srutikanta hota + (srutikanta.hota@gmail.com) +
+
2022-09-30 04:56:25
 +
+

*Thread Reply:* Many thanks for the details. My usecase is simple, I like to default the sparkgroupjob Id as openlineage parent runid if there is no parent run Id set. +sc.setJobGroup("myjobgroupid", "job description goes here") +This set the value in spark as +setLocalProperty(SparkContext.SPARKJOBGROUPID, group_id)

+ +

I like to use myjobgroup_id as openlineage parent run id

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2022-09-30 05:01:08
 +
+

*Thread Reply:* MDC is an ability to add extra key -> value pairs to a log entry, while not doing this within message body. So the question here is (I believe): how to add custom entries / custom facets to OpenLineage events?

+ +

@srutikanta hota What information would you like to include? There is great chance we already have some fields for that. If not it’s still worth putting in in write place like: is this info job specific, run specific or relates to some of input / output datasets?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-09-30 05:04:34
 +
+

*Thread Reply:* @srutikanta hota sounds like you want to set up +spark.openlineage.parentJobName +spark.openlineage.parentRunId +https://openlineage.io/docs/integrations/spark/

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
srutikanta hota + (srutikanta.hota@gmail.com) +
+
2022-09-30 05:15:18
 +
+

*Thread Reply:* @… we are having a long-running spark context(the context may run for a week) where we submit jobs. Settings the parentrunid at beginning won't help. We are submitting the job with sparkgroupid. I like to use the group Id as parentRunId

+ +

https://spark.apache.org/docs/1.6.1/api/R/setJobGroup.html

+ + + +
+ 🤔 Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Trevor Swan + (trevor.swan@matillion.com) +
+
2022-09-29 13:59:20
 +
+

Hi team - I am from Matillion and we would like to build support for openlineage. Who would be best placed to move the conversation with my product team?

+ + + +
+ 🙌 Will Johnson, Maciej Obuchowski, Francis McGregor-Macdonald +
+ +
+ 🎉 Michael Robinson +
+ +
+ 👍 Ernie Ostic +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2022-09-29 14:22:06
 +
+

*Thread Reply:* Hi Trevor, thank you for reaching out. I’d be happy to discuss with you how we can help you support OpenLineage. Let me send you an email.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jarek Potiuk + (jarek@potiuk.com) +
+
2022-09-29 15:58:35
 +
+

cccccbctlvggfhvrcdlbbvtgeuredtbdjrdfttbnldcb

+ + + +
+ 🐈 Julien Le Dem, Jakub Dardziński, Maciej Obuchowski, Paweł Leszczyński +
+ +
+ 🐈‍⬛ Julien Le Dem, Maciej Obuchowski, Paweł Leszczyński +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Petr Hajek + (petr.hajek@profinit.eu) +
+
2022-09-30 02:52:51
 +
+

Hi Everyone! Would anybody be interested in participation in MANTA Open Lineage connector testing? We are specially looking for an environment with rich Airflow implementation but we will be happy to test on any other OL Producer technology. Send me a direct message for more information. Thanks, Petr

+ + + +
+ 🙌 Michael Robinson, Ross Turk +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sheeri Cabral (Collibra) + (sheeri.cabral@collibra.com) +
+
2022-09-30 14:34:45
 +
+

Question about Apache Airflow that I think folks here would know, because doing a web search has failed me:

+ +

Is there a way to interact with Apache Airflow to retrieve the contents of the files in the sql directory, but NOT to run them?

+ +

(the APIs all seem to run sql, and when I search I just get “how to use the airflow API to run queries”)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-09-30 14:38:34
 +
+

*Thread Reply:* Is this in the context of an OpenLineage extractor?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sheeri Cabral (Collibra) + (sheeri.cabral@collibra.com) +
+
2022-09-30 14:40:47
 +
+

*Thread Reply:* Yes! I was specifically looking at the PostgresOperator

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sheeri Cabral (Collibra) + (sheeri.cabral@collibra.com) +
+
2022-09-30 14:41:54
 +
+

*Thread Reply:* (as Snowflake lineage can be retrieved from their internal ACCESS_HISTORY tables, we wouldn’t need to use Airflow’s SnowflakeOperator to get lineage, we’d use the method on the openlineage blog)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-09-30 14:43:08
 +
+

*Thread Reply:* The extractor for the SQL operators gets the query like this: +https://github.com/OpenLineage/OpenLineage/blob/45fda47d8ef29dd6d25103bb491fb8c443[…]gration/airflow/openlineage/airflow/extractors/sql_extractor.py

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/45fda47d8ef29dd6d25103bb491fb8c44355eb3c/integration/airflow/openlineage/airflow/extractors/sql_extractor.py | sql_extractor.py>
+
+ <pre><code> job_facets = {"sql": SqlJobFacet(query=self.operator.sql)} +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+ 👍 Sheeri Cabral (Collibra) +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-09-30 14:43:48
 +
+

*Thread Reply:* let me see if I can find the corresponding part of the Airflow API docs...

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sheeri Cabral (Collibra) + (sheeri.cabral@collibra.com) +
+
2022-09-30 14:45:00
 +
+

*Thread Reply:* aha! I’m not so far behind the times, it was only put in during July https://github.com/OpenLineage/OpenLineage/pull/907

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/pull/907|#907 [INTEGRATION][Airflow][SQL] Add SqlExtractor class.>
+
+ <p>Signed-off-by: Jakub Dardzinski <a href="mailto:kuba0221@gmail.com">kuba0221@gmail.com</a></p> + +<p><strong>Problem</strong></p> + +<p>Current SQL based extractors have code duplicated.</p> + +<p><strong>Solution</strong></p> + +<p>Added common SqlExtractor class. Updated unit tests to match changes.</p> + +<p><strong>Checklist</strong></p> + +<p>☑︎ You've <a href="https://github.com/OpenLineage/OpenLineage/blob/main/why-the-dco.md">signed-off</a> your work +☑︎ Your pull request title follows our <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CONTRIBUTING.md#creating-pull-requests">guidelines</a> +☑︎ Your changes are accompanied by tests (<em>if relevant</em>) +☑︎ Your change contains a <a href="https://kurtisnusbaum.medium.com/stacked-diffs-keeping-phabricator-diffs-small-d9964f4dcfa6">small diff</a> and is self-contained</p> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-09-30 14:47:28
 +
+

*Thread Reply:* Hm. The PostgresOperator seems to extend BaseOperator directly: +https://github.com/apache/airflow/blob/029ebacd9cbbb5e307a03530bdaf111c2c3d4f51/airflow/providers/postgres/operators/postgres.py#L58

+
+ + + + +
<https://github.com/apache/airflow/blob/029ebacd9cbbb5e307a03530bdaf111c2c3d4f51/airflow/providers/postgres/operators/postgres.py | postgres.py>
+
+ <pre><code> sql: str | Iterable[str], +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/apache/airflow">apache/airflow</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sheeri Cabral (Collibra) + (sheeri.cabral@collibra.com) +
+
2022-09-30 14:48:01
 +
+

*Thread Reply:* yeah 😞 I couldn’t find a way to make that work as an end-user.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-09-30 14:48:08
 +
+

*Thread Reply:* perhaps that can't be assumed for all operators that deal with SQL. I know that @Maciej Obuchowski has spent a lot of time on this.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-09-30 14:49:14
 +
+

*Thread Reply:* I don't know enough about the airflow internals 😞

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sheeri Cabral (Collibra) + (sheeri.cabral@collibra.com) +
+
2022-09-30 14:50:00
 +
+

*Thread Reply:* No worries. In case it saves you work, I also had a look at https://github.com/apache/airflow/blob/029ebacd9cbbb5e307a03530bdaf111c2c3d4f51/airflow/providers/common/sql/operators/sql.py - which also extends BaseOperator but not with a way to just get the SQL.

+
+ + + + +
<https://github.com/apache/airflow/blob/029ebacd9cbbb5e307a03530bdaf111c2c3d4f51/airflow/providers/common/sql/operators/sql.py | sql.py>
+
+ <p>``` +#</p> + +<h1>Licensed to the Apache Software Foundation (ASF) under one</h1> + +<h1>or more contributor license agreements. See the NOTICE file</h1> + +<h1>distributed with this work for additional information</h1> + +<h1>regarding copyright ownership. The ASF licenses this file</h1> + +<h1>to you under the Apache License, Version 2.0 (the</h1> + +<h1>"License"); you may not use this file except in compliance</h1> + +<h1>with the License. You may obtain a copy of the License at</h1> + +<p>#</p> + +<h1><a href="http://www.apache.org/licenses/LICENSE-2.0">http://www.apache.org/licenses/LICENSE-2.0</a></h1> + +<p>#</p> + +<h1>Unless required by applicable law or agreed to in writing,</h1> + +<h1>software distributed under the License is distributed on an</h1> + +<h1>"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY</h1> + +<h1>KIND, either express or implied. See the License for the</h1> + +<h1>specific language governing permissions and limitations</h1> + +<h1>under the License.</h1> + +<p>from <strong>future</strong> import annotations</p> + +<p>import re +from typing import TYPE_CHECKING, Any, Iterable, Mapping, Sequence, SupportsAbs</p> + +<p>from packaging.version import Version</p> + +<p>from airflow.compat.functools import cached<em>property +from airflow.exceptions import AirflowException +from airflow.hooks.base import BaseHook +from airflow.models import BaseOperator, SkipMixin +from airflow.providers.common.sql.hooks.sql import DbApiHook, _backported</em>get_hook +from airflow.version import version</p> + +<p>if TYPE_CHECKING: + from airflow.utils.context import Context</p> + +<p>def parse_boolean(val: str) -&gt; str | bool: + """Try to parse a string into boolean.</p> + +<pre><code>Raises ValueError if the input is not a valid true- or false-like string value. +""" +val = val.lower() +if val in ('y', 'yes', 't', 'true', 'on', '1'): + return True +if val in ('n', 'no', 'f', 'false', 'off', '0'): + return False +raise ValueError(f"{val!r} is not a boolean-like string value") +</code></pre> + +<p>def <em>get</em>failed<em>checks(checks, col=None): + if col: + return [ + f"Column: {col}\nCheck: {check},\nCheck Values: {check</em>values}\n" + for check, check<em>values in checks.items() + if not check</em>values["success"] + ] + return [ + f"\tCheck: {check},\n\tCheck Values: {check<em>values}\n" + for check, check</em>values in checks.items() + if not check_values["success"] + ]</p> + +<p><em>PROVIDERS</em>MATCHER = re.compile(r'airflow.providers.(.<strong>).hooks.</strong>')</p> + +<p><em>MIN</em>SUPPORTED<em>PROVIDERS</em>VERSION = { + "amazon": "4.1.0", + "apache.drill": "2.1.0", + "apache.druid": "3.1.0", + "apache.hive": "3.1.0", + "apache.pinot": "3.1.0", + "databricks": "3.1.0", + "elasticsearch": "4.1.0", + "exasol": "3.1.0", + "google": "8.2.0", + "jdbc": "3.1.0", + "mssql": "3.1.0", + "mysql": "3.1.0", + "odbc": "3.1.0", + "oracle": "3.1.0", + "postgres": "5.1.0", + "presto": "3.1.0", + "qubole": "3.1.0", + "slack": "5.1.0", + "snowflake": "3.1.0", + "sqlite": "3.1.0", + "trino": "3.1.0", + "vertica": "3.1.0", +}</p> + +<p>class BaseSQLOperator(BaseOperator): + """ + This is a base class for generic SQL Operator to get a DB Hook</p> + +<pre><code>The provided method is .get_db_hook(). The default behavior will try to +retrieve the DB hook based on connection type. +You can custom the behavior by overriding the .get_db_hook() method. +""" + +def __init__( + self, + **, + conn_id: str | None = None, + database: str | None = None, + hook_params: dict | None = None, + ****kwargs, +): + super().__init__(****kwargs) + self.conn_id = conn_id + self.database = database + self.hook_params = {} if hook_params is None else hook_params + +@cached_property +def _hook(self): + """Get DB Hook based on connection type""" + self.log.debug("Get connection for %s", self.conn_id) + conn = BaseHook.get_connection(self.conn_id) + if Version(version) &amp;gt;= Version('2.3'): + # "hook_params" were introduced to into "get_hook()" only in Airflow 2.3. + hook = conn.get_hook(hook_params=self.hook_params) # ignore airflow compat check + else: + # For supporting Airflow versions &amp;lt; 2.3, we backport "get_hook()" method. This should be removed + # when "apache-airflow-providers-common-sql" will depend on Airflow &amp;gt;= 2.3. + hook = _backported_get_hook(conn, hook_params=self.hook_params) + if not isinstance(hook, DbApiHook): + from airflow.hooks.dbapi_hook import DbApiHook as _DbApiHook + + if isinstance(hook, _DbApiHook): + # This case might happen if user installed common.sql provider but did not upgrade the + # Other provider's versions to a version that supports common.sql provider + class_module = hook.__class__.__module__ + match = _PROVIDERS_MATCHER.match(class_module) + if match: + provider = match.group(1) + min_version = _MIN_SUPPORTED_PROVIDERS_VERSION.get(provider) + if min_version: + raise AirflowException( + f'You are trying to use common-sql with {hook.__class__.__name__},' + f' but the Hook class comes from provider {provider} that does not support it.' + f' Please upgrade provider {provider} to at least {min_version}.' + ) + raise AirflowException( + f'You are trying to use `common-sql` with {hook.__class__.__name__},' + ' but its provider does not support it. Please upgrade the provider to a version that' + ' supports `common-sql`. The hook class should be a subclass of' + ' `airflow.providers.common.sql.hooks.sql.DbApiHook`.' + f' Got {hook.__class__.__name__} Hook with class hierarchy: {hook.__class__.mro()}' + ) + + if self.database: + hook.schema = self.database + + return hook + +def get_db_hook(self) -&amp;gt; DbApiHook: + """ + Get the database hook for the connection. + + :return: the database hook object. + :rtype: DbApiHook + """ + return self._hook +</code></pre> + +<p>class SQLColumnCheckOperator(BaseSQLOperator): + """ + Performs one or more of the templated checks in the column<em>checks dictionary. + Checks are performed on a per-column basis specified by the column</em>mapping. + Each check can take one or more of the following options: + - equal<em>to: an exact value to equal, cannot be used with other comparison options + - greater</em>than: value that result should be strictly greater than + - less<em>than: value that results should be strictly less than + - geq</em>to: value that results should be greater than or equal to + - leq_to: value that results should be less than or equal to + - tolerance: the percentage that the result may be off from the expected value</p> + +<pre><code>:param table: the table to run checks on +:param column_mapping: the dictionary of columns and their associated checks, e.g. + +.. code-block:: python + + { + "col_name": { + "null_check": { + "equal_to": 0, + }, + "min": { + "greater_than": 5, + "leq_to": 10, + "tolerance": 0.2, + }, + "max": {"less_than": 1000, "geq_to": 10, "tolerance": 0.01}, + } + } + +:param partition_clause: a partial SQL statement that is added to a WHERE clause in the query built by + the operator that creates partition_clauses for the checks to run on, e.g. + +.. code-block:: python + + "date = '1970-01-01'" + +:param conn_id: the connection ID used to connect to the database +:param database: name of database which overwrite the defined one in connection + +.. seealso:: + For more information on how to use this operator, take a look at the guide: + :ref:`howto/operator:SQLColumnCheckOperator` +""" + +template_fields = ("partition_clause",) + +column_checks = { + "null_check": "SUM(CASE WH… +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/apache/airflow">apache/airflow</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2022-09-30 15:22:24
 +
+

*Thread Reply:* that's more of an Airflow question indeed. As far as I understand you need to read file with SQL statement within Airflow Operator and do something but run the query (like pass as an XCom)? SQLExtractors we have get same SQL that operators render and uses it to extract additional information like table schema straight from database

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sheeri Cabral (Collibra) + (sheeri.cabral@collibra.com) +
+
2022-09-30 14:36:18
 +
+

(I’m also ok with a way to get the SQL that has been run - but from Airflow, not the data source - I’m looking for a db-neutral way to do this, otherwise I can just parse query logs on any specific db system)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paul Lee + (paullee@lyft.com) +
+
2022-09-30 18:45:09
 +
+

👋 are there any docs on how the listener hooks in and gets run with openlineage-airflow? trying to write some unit tests but no docs seem to exist on the flow.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2022-09-30 19:06:47
 +
+

*Thread Reply:* There's a design doc linked from the PR: https://github.com/apache/airflow/pull/20443 +https://docs.google.com/document/d/1L3xfdlWVUrdnFXng1Di4nMQYQtzMfhvvWDR9K4wXnDU/edit

+
+ + + + +
<https://github.com/apache/airflow/pull/20443|#20443 Add Listener Plugin API that tracks TaskInstance state changes>
+
+ <p>This PR adds new Plugin API - listeners. It enables plugin authors to write <a href="https://pluggy.readthedocs.io/en/stable/"><code>pluggy</code> hook implementation</a> that will be called on certain formalized extension points. To differentiate between current Airflow extension points, like plugins, and current Airflow hooks, implementations of those hooks are called <code>listeners</code>.</p> + +<p>The API is ment to be called across all dags, and all operators - in contrast to current <code>on_success_callback</code>, <code>pre_execute</code> and related family which are meant to provide callbacks for particular dag authors, or operator creators.</p> + +<p><code>pluggy</code> mechanism enables us to execute multiple, or none, listeners that implement particular extension point, so that users can use multiple listeners seamlessly.</p> + +<p>In this PR, three such extension points are added. <br /> +When TaskInstance's state is changed to <code>RUNNING</code>, <code>on_task_instance_running</code> hook is called. On change to<code>SUCCESS</code> <code>on_task_instance_success</code> is called, similarly on <code>FAILED</code> <code>on_task_instance_failed</code> is called.</p> + +<p>Actual notification mechanism is be implemented using <a href="https://docs.sqlalchemy.org/en/13/orm/session_events.html#after-flush">SQLAlchemy’s events mechanism</a>. This ensures that plugins will get every change of state, regardless of where in the codebase it happened, and not require manual annotation of TI state changes across the codebase.</p> + +<p>To make sure that this change is not affecting performance, running this mechanism on scheduler is disabled by default. The SQLAlchemy event mechanism is also not affected by default - the event listener is only added if we have any plugin which actually provides any listener.</p> + +<p>Design doc: <br /> +<a href="https://docs.google.com/document/d/1L3xfdlWVUrdnFXng1Di4nMQYQtzMfhvvWDR9K4wXnDU/edit?usp=sharing">https://docs.google.com/document/d/1L3xfdlWVUrdnFXng1Di4nMQYQtzMfhvvWDR9K4wXnDU/edit?usp=sharing</a></p> + +<p>Background to this change: <br /> +When discussing changing LineageBackend api for OpenLineage plugin, we came to consensus that the changes proposed there would be better suited to the new API <a href="https://github.com/apache/airflow/issues/17984">#17984</a></p> + +<p>related: <a href="https://github.com/apache/airflow/issues/17984">#17984</a></p> +
+ +
+
Labels
+ area:scheduler/executor, area:dev-tools, area:plugins, type:new-feature, full tests needed +
+ +
+
Comments
+ 3 +
+ + + + + + + +
+ + <a href="https://github.com/apache/airflow">apache/airflow</a> +
+ + +
+ + + +
+ 👀 Paul Lee +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paul Lee + (paullee@lyft.com) +
+
2022-09-30 19:18:47
 +
+

*Thread Reply:* amazing thank you I will take a look

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-10-03 11:32:52
 +
+

@channel +Hello everyone, I’m opening up a vote on releasing OpenLineage 0.15.0, including +• an improved development experience in the Airflow integration +• updated proposal and integration templates +• a change to the BigQuery client in the Airflow integration +• plus bug fixes across the project. +3 +1s from committers will authorize an immediate release. For all the commits, see: https://github.com/OpenLineage/OpenLineage/compare/0.14.0...HEAD. Note: this will be the last release to support Airflow 1.x! +Thanks!

+ + + +
+ 🎉 Paul Lee, Howard Yoo, Minkyu Park, Michael Collado, Paweł Leszczyński, Maciej Obuchowski, Harel Shein +
+ +
+ 👍 Michael Collado, Julien Le Dem, Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-10-03 11:33:30
 +
+

*Thread Reply:* Hey @Michael Robinson. Removal of Airflow 1.x support is planned for next release after 0.15.0

+ + + +
+ 👍 Jakub Dardziński, Paul Lee +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-10-03 11:37:03
 +
+

*Thread Reply:* 0.15.0 would be the last release supporting Airflow 1.x

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-10-03 11:37:07
 +
+

*Thread Reply:* just caught this myself. I’ll make the change

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paul Lee + (paullee@lyft.com) +
+
2022-10-03 11:40:33
 +
+

*Thread Reply:* we’re still on 1.10.15 at the moment so i guess our team would have to rely on <=0.15.0?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-10-03 11:49:47
 +
+

*Thread Reply:* Is this something you want to continue doing or do you want to migrate relatively soon?

+ +

We want to remove 1.10 integration because for multiple PRs, maintaining compatibility with it takes a lot of time; the code is littered with checks like this. +if parse_version(AIRFLOW_VERSION) &gt;= parse_version("2.0.0"):

+ + + +
+ 👍 Paul Lee +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paul Lee + (paullee@lyft.com) +
+
2022-10-03 12:03:40
 +
+

*Thread Reply:* hey Maciej, we do have plans to migrate in the coming months but for right now we need to stay on 1.10.15.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-10-04 09:39:11
 +
+

*Thread Reply:* Thanks, all. The release is authorized, and you can expect it by Thursday.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paul Lee + (paullee@lyft.com) +
+
2022-10-03 17:56:08
 +
+

👋 what would be a possible reason for the built in airflow backend being utilized instead of a custom wrapper over airflow.lineage.Backend ? double checked the [lineage] key in our airflow.cfg

+ +

there doesn't seem to be any errors being thrown and the object loads 🤔

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paul Lee + (paullee@lyft.com) +
+
2022-10-03 17:56:36
 +
+

*Thread Reply:* running airflow 2.3.4 with openlineage-airflow 0.14.1

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-10-03 18:03:03
 +
+

*Thread Reply:* if you're talking about LineageBackend, it is used in Airflow 2.1-2.2. It did not have functionality where you can be notified on task start or failure, so we wanted to expand the functionality: https://github.com/apache/airflow/issues/17984

+ +

Consensus of Airflow maintainers wasn't positive about changing this interface, so we went with another direction: https://github.com/apache/airflow/pull/20443

+
+ + + + +
<https://github.com/apache/airflow/issues/17984|#17984 Add possibility to LineageBackend to be notified of task instance execution start and failure>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/apache/airflow">apache/airflow</a> +
+ + +
+
+ + + + +
<https://github.com/apache/airflow/pull/20443|#20443 Add Listener Plugin API that tracks TaskInstance state changes>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/apache/airflow">apache/airflow</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-10-03 18:06:58
 +
+

*Thread Reply:* Why nothing happens? https://github.com/OpenLineage/OpenLineage/blob/895160423643398348154a87e0682c3ab5c8704b/integration/airflow/openlineage/lineage_backend/__init__.py#L91

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/895160423643398348154a87e0682c3ab5c8704b/integration/airflow/openlineage/lineage_backend/__init__.py | __init__.py>
+
+ <pre><code> if parse_version(AIRFLOW_VERSION) &amp;gt;= parse_version("2.3.0.dev0"): +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paul Lee + (paullee@lyft.com) +
+
2022-10-03 18:30:32
 +
+

*Thread Reply:* ah hmm ok, i will double check. i commented that part out so technically it should run but maybe i missed something

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paul Lee + (paullee@lyft.com) +
+
2022-10-03 18:30:42
 +
+

*Thread Reply:* thank you for your fast response @Maciej Obuchowski ! i appreciate it

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paul Lee + (paullee@lyft.com) +
+
2022-10-03 18:31:13
 +
+

*Thread Reply:* it seems like it doesn't use my custom wrapper but instead uses the openlineage implementation.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paul Lee + (paullee@lyft.com) +
+
2022-10-03 20:11:15
 +
+

*Thread Reply:* @Maciej Obuchowski ok, after checking we are emitting events with our custom backend but an odd thing is an attempt is always made with the openlineage backend. is there something obvious i am perhaps missing 🤔

+ +

ends up with requests.exceptions.HTTPError: 401 Client Error: Unauthorized for url immediately after task start. but by the end on task success/failure it emits the event with our custom backend both RunState.COMPLETE and RunState.START into our own pipeline.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-10-04 06:19:06
 +
+

*Thread Reply:* If you're on 2.3 and trying to use some wrapped LineageBackend, what I think is happening is OpenLineagePlugin that automatically registers via setup.py entrypoint https://github.com/OpenLineage/OpenLineage/blob/65a5f021a1ba3035d5198e759587737a05b242e1/integration/airflow/openlineage/airflow/plugin.py#L30

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/65a5f021a1ba3035d5198e759587737a05b242e1/integration/airflow/openlineage/airflow/plugin.py | plugin.py>
+
+ <pre><code> listeners = [listener] +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-10-04 06:23:48
 +
+

*Thread Reply:* I think if you want to extend it with proprietary code there are two good options.

+ +

First, if your code only needs to touch HTTP client side - which I guess is the case due to 401 error - then you can create custom Transport.

+ +

Second, is that you fork OL code and create your own package, without entrypoint script or with adding your own if you decide to extend OpenLineagePlugin instead of LineageBackend

+ + + +
+ 👍 Paul Lee +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paul Lee + (paullee@lyft.com) +
+
2022-10-04 14:23:33
 +
+

*Thread Reply:* amazing thank you for your help. i will take a look

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paul Lee + (paullee@lyft.com) +
+
2022-10-04 14:49:47
 +
+

*Thread Reply:* @Maciej Obuchowski is there a way to extend the plugin like how we can wrap the custom backend with 2.2? or would it be necessary to fork it.

+ +

we're trying to not fork and instead opt with extending.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-10-05 04:55:05
 +
+

*Thread Reply:* I think it's best to fork, since it's getting loaded by Airflow as an entrypoint: https://github.com/OpenLineage/OpenLineage/blob/133110300e8ea4e42e3640608cfed459683d5a8d/integration/airflow/setup.py#L70

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/133110300e8ea4e42e3640608cfed459683d5a8d/integration/airflow/setup.py | setup.py>
+
+ <pre><code> "airflow.plugins": ["OpenLineagePlugin = openlineage.airflow.plugin:OpenLineagePlugin"] +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+ 🙏 Paul Lee +
+ +
+ :gratitude_thank_you: Paul Lee +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paul Lee + (paullee@lyft.com) +
+
2022-10-05 13:29:24
 +
+

*Thread Reply:* got it. and in terms of the openlineage.yml and defining a custom transport is there a way i can define where openlineage-python should look for the custom transport? e.g. different path

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paul Lee + (paullee@lyft.com) +
+
2022-10-05 13:30:04
 +
+

*Thread Reply:* because from the docs i. can't tell except for the file i'm supposed to copy and implement.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-10-05 14:18:19
 +
+

*Thread Reply:* @Paul Lee you should derive from Transport base class and register type as full python import path to your custom transport, for example https://github.com/OpenLineage/OpenLineage/blob/f8533266491acea2159f602f782a99a4f8a82cca/client/python/tests/openlineage.yml#L2

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/f8533266491acea2159f602f782a99a4f8a82cca/client/python/tests/openlineage.yml | openlineage.yml>
+
+ <pre><code> type: "tests.transport.AccumulatingTransport" +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-10-05 14:20:48
 +
+

*Thread Reply:* your custom transport should have also define custom class Config , and this class should implement from_dict method

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-10-05 14:20:56
 +
+

*Thread Reply:* the whole process is here: https://github.com/OpenLineage/OpenLineage/blob/a62484ec14359a985d283c639ac7e8b9cfc54c2e/client/python/openlineage/client/transport/factory.py#L47

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/a62484ec14359a985d283c639ac7e8b9cfc54c2e/client/python/openlineage/client/transport/factory.py | factory.py>
+
+ <pre><code> def _create_transport(self, config: dict): +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-10-05 14:21:09
 +
+

*Thread Reply:* and I know we need to document this better 🙂

+ + + +
+ 🙏 Paul Lee +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paul Lee + (paullee@lyft.com) +
+
2022-10-05 15:35:31
 +
+

*Thread Reply:* amazing, thanks for all your help 🙂 +1 to the docs, if i have some time when done i will push up some docs to document what i've done

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-10-05 15:50:29
 +
+

*Thread Reply:* https://github.com/openlineage/docs/ - let me know and I'll review 🙂

+
+ + + + +
OpenLineage/docs
+
+ Documentation for OpenLineage +
+ +
+
Website
+ <https://openlineage.io/docs> +
+ +
+
Stars
+ 4 +
+ + + + + + + + +
+ + + +
+ 🎉 Paul Lee +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-10-04 12:39:59
 +
+

@channel +Hi everyone, opening a vote on a release (0.15.1) to add #1131 to fix the release process on CI. 3 +1s from committers will authorize an immediate release. Thanks. More details are here: +https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/pull/1131|#1131 CI: build macos release package on medium resouce class>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md | CHANGELOG.md>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+ ✅ Michael Collado, Maciej Obuchowski, Julien Le Dem +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-10-04 14:25:49
 +
+

*Thread Reply:* Thanks, all. The release is authorized.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-10-05 10:46:46
 +
+

@channel +OpenLineage 0.15.1 is now available! +We added: +• Airflow: improve development experience #1101 @JDarDagran +• Documentation: update issue templates for proposal & add new integration template #1116 @rossturk +• Spark: add description for URL parameters in readme, change overwriteName to appName #1130 @tnazarew +We changed: +• Airflow: lazy load BigQuery client #1119 @mobuchowski +Many bug fixes were also included in this release. +Release: https://github.com/OpenLineage/OpenLineage/releases/tag/0.15.1 +Changelog: https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md +Commit history: https://github.com/OpenLineage/OpenLineage/compare/0.14.1...0.15.1 +Maven: https://oss.sonatype.org/#nexus-search;quick~openlineage +PyPI: https://pypi.org/project/openlineage-python/

+ + + +
+ 🙌 Maciej Obuchowski, Jakub Dardziński, Howard Yoo, Harel Shein, Paul Lee, Paweł Leszczyński +
+ +
+ 🎉 Howard Yoo, Harel Shein, Paul Lee +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-10-06 07:35:00
 +
+

Is there a topic you think the community should discuss at the next OpenLineage TSC meeting? Reply or DM with your item, and we’ll add it to the agenda.

+ + + +
+ 🌟 Paul Lee +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paul Lee + (paullee@lyft.com) +
+
2022-10-06 13:29:30
 +
+

*Thread Reply:* would love to add improvement in docs :) for newcomers

+ + + +
+ 👏 Jakub Dardziński +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paul Lee + (paullee@lyft.com) +
+
2022-10-06 13:31:07
 +
+

*Thread Reply:* also, what’s TSC?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-10-06 15:20:23
 +
+

*Thread Reply:* Technical Steering Committee, but it’s open to everyone

+ + + +
+ 👍 Paul Lee +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-10-06 15:20:45
 +
+

*Thread Reply:* and we encourage newcomers to attend

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paul Lee + (paullee@lyft.com) +
+
2022-10-06 13:49:00
 +
+

has anyone seen their COMPLETE/FAILED listeners not firing on Airflow 2.3.4 but START events do emit? using openlineage-airflow 0.14.1

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2022-10-06 14:39:27
 +
+

*Thread Reply:* is there any error/warn message logged maybe?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paul Lee + (paullee@lyft.com) +
+
2022-10-06 14:40:53
 +
+

*Thread Reply:* none that i'm seeing on our workers. i do see that our custom http transport is being utilized on START.

+ +

but on SUCCESS nothing fires.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paul Lee + (paullee@lyft.com) +
+
2022-10-06 14:41:21
 +
+

*Thread Reply:* which makes me believe the listeners themselves aren't being utilized? 🤔

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2022-10-06 16:37:54
 +
+

*Thread Reply:* uhm, any chance you're experiencing this with custom extractors?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2022-10-06 16:38:13
 +
+

*Thread Reply:* I'd be happy to jump on a quick call if you wish

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2022-10-06 16:38:40
 +
+

*Thread Reply:* but in more EU friendly hours 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paul Lee + (paullee@lyft.com) +
+
2022-10-07 16:19:47
 +
+

*Thread Reply:* no custom extractors, its usingt he base extractor. a call would be 👍. let me look at my calendar and EU hours.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-10-06 15:23:27
 +
+

@channel The next OpenLineage Technical Steering Committee meeting is on Thursday, October 13 at 10 am PT. Join us on Zoom: https://bit.ly/OLzoom +All are welcome! +Agenda:

+ +
  1. Announcements
  2. Recent Release 0.15.1
  3. Project roadmap review
  4. Open discussion +Notes: https://bit.ly/OLwiki +Is there a topic you think the community should discuss at this or a future meeting? Reply or DM me to add items to the agenda.
    5. +
+ + + +
+ 🙌 Paul Lee, Harel Shein +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Srinivasa Raghavan + (gsrinir@gmail.com) +
+
2022-10-07 06:52:42
 +
+

hello all. I am trying to run the airflow example from here +I changed the Marquez web port from 5000 to 15000 but when I start the docker images, it seems to always default to port 5000 and therefore when I go to localhost:3000, the jobs don't load up as they are not able to connect to the marquez app running in 15000. I've overriden the values in docker-compose.yml and in openLineage.env but it seems to be picking up the 5000 value from some other location. +This is what I see in the logs. Any pointers on this or please redirect me to the appropriate channel. Thanks! +INFO [2022-10-07 10:48:58,022] org.eclipse.jetty.server.AbstractConnector: Started application@782fd504{HTTP/1.1, (http/1.1)}{0.0.0.0:5000} +INFO [2022-10-07 10:48:58,034] org.eclipse.jetty.server.AbstractConnector: Started admin@1537c744{HTTP/1.1, (http/1.1)}{0.0.0.0:5001}

+ + + +
+ 👀 Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Srinivasa Raghavan + (gsrinir@gmail.com) +
+
2022-10-20 05:11:09
 +
+

*Thread Reply:* Apparently the value is hard coded in the code somewhere that I couldn't figure out but at-least learnt that in my Mac where this port 5000 is being held up can be freed by following the below simple step.

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Hanna Moazam + (hannamoazam@microsoft.com) +
+
2022-10-10 18:00:17
 +
+

Hi #general - @Will Johnson and I are working on adding support for Snowflake to OL, and as we were going to specify the package under the compileOnly dependencies in gradle, we had some doubts looking at the existing dependencies. Taking bigQuery as an example - we see it's included as a dependency in both the shared build.gradle file, and in the app build.gradle file. We're a bit confused about the following:

+ +
  1. Why do we need to have the bigQuery package in shared's dependencies? App of course contains the bigQueryNodeVisitor but we couldn't spot where it's being used within shared.
  2. For all the dependencies in the shared gradle file, the versions for Scala and Spark are fixed (Scala 2.11, Spark 2.4.8), whereas for app, the versionsMap allows for different combinations of spark and scala versions. Why is this so?
  3. How do the dependencies between app and shared interact? Does one or the other take precedence for which version of the bigQuery connector is compiled? +We'd appreciate any guidance!
    4. +
+ +

Thank you in advance!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2022-10-11 03:47:31
 +
+

*Thread Reply:* Hi @Hanna Moazam,

+ +

Within recent PR https://github.com/OpenLineage/OpenLineage/pull/1111, I removed BigQuery dependencies from spark2, spark32 and spark3 subprojects. It has to stay in sharedbecause of BigQueryNodeVisitor. The usage of BigQueryNodeVisitor is tricky as we never know if bigquery classes are available on runtime or not. The check is done in io.openlineage.spark.agent.lifecycle.BaseVisitorFactory +if (BigQueryNodeVisitor.hasBigQueryClasses()) { + list.add(new BigQueryNodeVisitor(context, factory)); + } +Regarding point 2, there were some Spark versions which allowed two Scala versions (2.11 and 2.12). Then it makes sense to make it configurable. On the other hand, for Spark 3.2 we only support 2.12 which is hardcoded in build.gradle.

+ +

The idea of app project is let's create a separate project to aggregate all the dependecies and run integration tests on it . Subprojects spark2, spark3, etc. do depend on shared . Putting integration tests in shared would create additional opposite-way dependency, which we wanted to avoid.

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/pull/1111|#1111 fix bigquery connector>
+
+ <p>Signed-off-by: Pawel Leszczynski <a href="mailto:leszczynski.pawel@gmail.com">leszczynski.pawel@gmail.com</a></p> + +<p><strong>Problem</strong></p> + +<p>Spark integration fails with spark-bigquery-connector &gt;=0.25.0</p> + +<p>Closes: <a href="https://github.com/OpenLineage/OpenLineage/issues/1105">#1105</a></p> + +<p><strong>Solution</strong></p> + +<p>• make Spark integration work with the latest connector +• provide a change in a way that the old version would work either</p> + +<p>&gt; <strong>Note:</strong> All schema changes require discussion. Please <a href="https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue">link the issue</a> for context.</p> + +<p>☐ Your change modifies the <a href="https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.json">core</a> OpenLineage model +☐ Your change modifies one or more OpenLineage <a href="https://github.com/OpenLineage/OpenLineage/tree/main/spec/facets">facets</a></p> + +<p>If you're contributing a new integration, please specify the scope of the integration and how/where it has been tested (e.g., Apache Spark integration supports <code>S3</code> and <code>GCS</code> filesystem operations, tested with AWS EMR).</p> + +<p><strong>Checklist</strong></p> + +<p>☑︎ You've <a href="https://github.com/OpenLineage/OpenLineage/blob/main/why-the-dco.md">signed-off</a> your work +☑︎ Your pull request title follows our <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CONTRIBUTING.md#creating-pull-requests">guidelines</a> +☐ Your changes are accompanied by tests (<em>if relevant</em>) +☑︎ Your change contains a <a href="https://kurtisnusbaum.medium.com/stacked-diffs-keeping-phabricator-diffs-small-d9964f4dcfa6">small diff</a> and is self-contained +☐ You've updated any relevant documentation (<em>if relevant</em>) +☐ You've updated the <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md"><code>CHANGELOG.md</code></a> with details about your change under the "Unreleased" section (<em>if relevant, depending on the change, this may not be necessary</em>) +☐ You've versioned the core OpenLineage model or facets according to <a href="https://docs.snowplowanalytics.com/docs/pipeline-components-and-applications/iglu/common-architecture/schemaver">SchemaVer</a> (<em>if relevant</em>) +☐ You've added a <a href="https://github.com/OpenLineage/OpenLineage/tree/main/.github/header_templates.md">header</a> to source files (<em>if relevant</em>)</p> +
+ +
+
Labels
+ bug, documentation, integration/spark, integration/bigquery +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-10-11 09:20:44
 +
+

*Thread Reply:* So, if we wanted to add Snowflake, we would need to:

+ +
  1. Pick a version of snowflake's spark library
  2. Pick a version of scala that we target (i.e. we are only going to support Snowflake in Spark 3.2 so scala 2.12 will be hard coded)
  3. Add the visitor code to Shared
  4. Add the dependencies to app (ONLY if there is an integration test in app?? This is the confusing part still)
    5. +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2022-10-12 03:51:54
 +
+

*Thread Reply:* Yes. Please note that snowflake library will not be included in target OpenLineage jar. So you may test it manually against multiple Snowflake library versions or even adjust code in case of minor differences.

+ + + +
+ 👍 Hanna Moazam, Will Johnson +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Hanna Moazam + (hannamoazam@microsoft.com) +
+
2022-10-12 05:20:17
 +
+

*Thread Reply:* Thank you Pawel!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-10-12 12:18:16
 +
+

*Thread Reply:* Basically the same pattern you've already done with Kusto 😉 +https://github.com/OpenLineage/OpenLineage/blob/a96ecdabe66567151e7739e25cd9dd03d6[…]va/io/openlineage/spark/agent/lifecycle/BaseVisitorFactory.java

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/a96ecdabe66567151e7739e25cd9dd03d67e03e8/integration/spark/shared/src/main/java/io/openlineage/spark/agent/lifecycle/BaseVisitorFactory.java | BaseVisitorFactory.java>
+
+ <pre><code> if (KustoRelationVisitor.hasKustoClasses()) { +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Hanna Moazam + (hannamoazam@microsoft.com) +
+
2022-10-12 12:26:35
 +
+

*Thread Reply:* We actually used only reflection for Kusto and were hoping to do it the 'better' way with the package itself for snowflake - if it's possible :)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Akash r + (akashrn25@gmail.com) +
+
2022-10-11 02:04:28
 +
+

Hi Community,

+ +

I was going through the code of dbt integration with Open lineage, Once the events has been emitted from client code , I wanted to check the server code where the events are read and the lineage is formed. Where can I find that code ?

+ +

Thanks

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-10-11 05:03:26
 +
+

*Thread Reply:* Reference implementation of OpenLineage consumer is Marquez: https://github.com/MarquezProject/marquez

+
+ + + + +
MarquezProject/marquez
+
+ Collect, aggregate, and visualize a data ecosystem's metadata +
+ +
+
Website
+ <https://marquezproject.ai> +
+ +
+
Stars
+ 1187 +
+ + + + + + + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-10-12 11:59:55
 +
+

This month’s OpenLineage TSC meeting is tomorrow at 10 am PT! https://openlineage.slack.com/archives/C01CK9T7HKR/p1665084207602369

+
+ + +
+ + + } + + Michael Robinson + (https://openlineage.slack.com/team/U02LXF3HUN7) +
+ + + +
+
+ <p><strong>@channel</strong> The next OpenLineage Technical Steering Committee meeting is on <strong>Thursday, October 13 at 10 am PT</strong>. Join us on Zoom: <a href="https://bit.ly/OLzoom">https://bit.ly/OLzoom</a> +All are welcome! +Agenda:</p> + +<ol><li>Announcements</li><li>Recent Release 0.15.1</li><li>Project roadmap review</li><li>Open discussion +Notes: <a href="https://bit.ly/OLwiki">https://bit.ly/OLwiki</a> +Is there a topic you think the community should discuss at this or a future meeting? Reply or DM me to add items to the agenda.</li> +</ol> +
+ + + + + + +
Original URL: https://openlineage.slack.com/archives/C01CK9T7HKR/p1665084207602369
+ + +
+ + Slack Conversation +
+ + +
+ + + +
+ 🙌 Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sheeri Cabral (Collibra) + (sheeri.cabral@collibra.com) +
+
2022-10-13 12:05:17
 +
+

Is there anyone in the Open Lineage community in San Diego? I’ll be there Nov 1-3 and would love to meet some of y’all in person

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paul Lee + (paullee@lyft.com) +
+
2022-10-20 13:49:39
 +
+

👋 is there a way to define a base extractor to be defaulted to? for example, i'd like to have all our operators (50+) default to my custom base extractor instead of having a list of 50+ operators in get_operator_classnames

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Howard Yoo + (howard.yoo@astronomer.io) +
+
2022-10-20 13:53:55
 +
+

I don't think that's possible yet, as the extractor checks are based on the class name... and it wouldn't check which parent operator has it inherited from.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paul Lee + (paullee@lyft.com) +
+
2022-10-20 14:05:38
 +
+

😢 ok, i would contribute upstream but unfortunately we're still on 1.10.15. looking like we might have to hardcode for a bit.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paul Lee + (paullee@lyft.com) +
+
2022-10-20 14:06:01
 +
+

is this the correct assumption? we're still on 0.14.1 ^

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-10-20 14:33:49
 +
+

If you'll move to 2.x series and OpenLineage 0.16, you could use this feature: https://github.com/OpenLineage/OpenLineage/pull/1162

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/pull/1162|#1162 airflow: add default extractor>
+
+ <p>As a way of extending OpenLineage airflow integration, we are providing a way to add custom extractors. This is great for people who want to add integration with external operators - for example, regular Airflow user who wants to support <code>GCSToS3Operator</code>.</p> + +<p>However, it's too cumbersome for people who own operators, and want to add default implementation of OpenLineage for their operators for external users. This PR adds support for <code>DefaultExtractor</code> which looks up particular method - <code>get_openlineage_facets</code> on Operator, and when it's present, uses it to get lineage data.</p> + +<p>Signed-off-by: Maciej Obuchowski <a href="mailto:obuchowski.maciej@gmail.com">obuchowski.maciej@gmail.com</a></p> +
+ +
+
Labels
+ integration/airflow, extractor +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+ 👍 Paul Lee +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paul Lee + (paullee@lyft.com) +
+
2022-10-20 14:46:36
 +
+

thanks @Maciej Obuchowski we're working on it. hoping we'll land on 2.3.4 in the coming month.

+ + + +
+ 🔥 Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Austin Poulton + (austin.poulton@equalexperts.com) +
+
2022-10-26 05:31:07
 +
+

👋 Hi everyone!

+ + + +
+ 👋 Jakub Dardziński, Maciej Obuchowski, Michael Robinson, Ross Turk, Willy Lulciuc, Paweł Leszczyński, Harel Shein +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harel Shein + (harel.shein@gmail.com) +
+
2022-10-26 15:22:22
 +
+

*Thread Reply:* Hey @Austin Poulton, welcome! 👋

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Austin Poulton + (austin.poulton@equalexperts.com) +
+
2022-10-31 06:09:41
 +
+

*Thread Reply:* thanks Harel 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-11-01 09:44:18
 +
+

@channel +Hi everyone, I’m opening a vote to release OpenLineage 0.16.0, featuring: +• support for boolean arguments in the DefaultExtractor +• a more efficient get_connection_uri method in the Airflow integration +• a reorganized, Rust-based SQL integration (easing the addition of language interfaces in the future) +• bug fixes and more. +3 +1s from committers will authorize an immediate release. Thanks. More details are here: +https://github.com/OpenLineage/OpenLineage/compare/0.15.1...HEAD

+ + + +
+ 🙌 Howard Yoo, Paweł Leszczyński, Maciej Obuchowski +
+ +
+ 👍 Ross Turk, Paweł Leszczyński, Maciej Obuchowski +
+ +
+ ➕ Willy Lulciuc, Mandy Chessell, Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-11-01 13:37:54
 +
+

*Thread Reply:* Thanks, all! The release is authorized. We will initiate it within 48 hours.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Iftach Schonbaum + (iftach.schonbaum@hunters.ai) +
+
2022-11-02 08:45:20
 +
+

Anybody with a success use-case of ingesting column-level lineage into amundsen?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-11-02 09:19:43
 +
+

*Thread Reply:* I think amundsen-openlineage dataloader precedes column-level lineage in OL by a bit, so I doubt this works

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harel Shein + (harel.shein@gmail.com) +
+
2022-11-02 15:54:31
 +
+

*Thread Reply:* do you want to open up an issue for it @Iftach Schonbaum?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-11-02 12:36:22
 +
+

Hi everyone, you might notice Dependabot opening PRs to update dependencies now that it’s been configured and turned on (https://github.com/OpenLineage/OpenLineage/pull/1182). There will probably be a large number of PRs to start with, but this shouldn’t always be the case and we can change the tool’s behavior, as well. (Some background: this will help us earn the OSSF Silver badge for the project, which will help us advance in the LFAI.)

+ + + +
+ 👍 Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-11-03 07:53:31
 +
+

@channel +I’m opening a vote to release OpenLineage 0.16.1 to fix an issue in the SQL integration. This release will also include all the commits announced for 0.16.0. +3 +1s from committers will authorize an immediate release. Thanks.

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/pull/1228|#1228 fix: sql macos build directory>
+
+ <p>Since package renaming, mac os CI release build was run in the wrong directory.</p> + +<p>Signed-off-by: Maciej Obuchowski <a href="mailto:obuchowski.maciej@gmail.com">obuchowski.maciej@gmail.com</a></p> +
+ +
+
Labels
+ integration/sql +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+ ➕ Maciej Obuchowski, Hanna Moazam, Jakub Dardziński, Ross Turk, Paweł Leszczyński, Jarek Potiuk, Willy Lulciuc +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-11-03 12:25:29
 +
+

*Thread Reply:* Thanks, all. The release is authorized and will be initiated shortly.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-11-03 13:46:58
 +
+

@channel +OpenLineage 0.16.1 is now available, featuring: +Additions: +• Airflow: add dag_run information to Airflow version run facet #1133 @fm100 +• Airflow: add LoggingMixin to extractors #1149 @JDarDagran +• Airflow: add default extractor #1162 @mobuchowski +• Airflow: add on_complete argument in DefaultExtractor #1188 @JDarDagran +• SQL: reorganize the library into multiple packages #1167 @StarostaGit @mobuchowski +Changes: +• Airflow: move get_connection_uri as extractor’s classmethod #1169 @JDarDagran +• Airflow: change get_openlineage_facets_on_start/complete behavior #1201 @JDarDagran +Bug fixes and more! +Release: https://github.com/OpenLineage/OpenLineage/releases/tag/0.16.1 +Changelog: https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md +Commit history: https://github.com/OpenLineage/OpenLineage/compare/0.15.1...0.16.1 +Maven: https://oss.sonatype.org/#nexus-search;quick~openlineage +PyPI: https://pypi.org/project/openlineage-python/

+ + + +
+ 🙌 Maciej Obuchowski, Francis McGregor-Macdonald, Eric Veleker +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Phil Chen + (phil@gpr.com) +
+
2022-11-03 13:59:29
 +
+

Are there any tutorial and documentation how to create an Openlinage connector. For example, what if we Argo workflow instead of Apache airflow for orchestrating ETL jobs? How are we going to create Openlinage Argo workflow connector? How much efforts, roughly? And can people contribute such connectors to the community if they create one?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-11-04 06:34:27
 +
+

*Thread Reply:* > Are there any tutorial and documentation how to create an Openlinage connector. +We have somewhat of a start of a doc: +https://openlineage.io/docs/development/developing/

+ +

Here we have an example of using Python OL client to emit OL events: https://openlineage.io/docs/client/python#start-docker-and-marquez

+ +

> How much efforts, roughly? +I'm not familiar with Argo workflows, but usually the effort needed depends on extensibility of the underlying system. From the first look, Argo looks like it has sufficient mechanisms for that: https://argoproj.github.io/argo-workflows/executor_plugins/#examples-and-community-contributed-plugins

+ +

Then, it depends if you can get the information that you need in that plugin. Basic need is to have information from which datasets the workflow/job is reading and to which datasets it's writing.

+ +

> And can people contribute such connectors to the community if they create one? +Definitely! And if you need help with anything OpenLineage feel free to write here on Slack

+
+
openlineage.io
+ + + +
Python | OpenLineage Docs
+
+ Overview +
+ + + + + + +
Original URL: https://openlineage.io/docs/client/python#start-docker-and-marquez
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-11-03 17:57:37
 +
+

Is there a topic you think the community should discuss at the next OpenLineage TSC meeting? Reply or DM with your item, and we’ll add it to the agenda.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-11-03 18:03:18
 +
+

@channel +This month’s OpenLineage TSC meeting is next Thursday, November 10th, at 10 am PT. Join us on Zoom: https://bit.ly/OLzoom. All are welcome! +On the tentative agenda:

+ +
  1. Recent release overview [Michael R.]
  2. Update on LFAI & Data Foundation progress [Michael R.]
  3. Proposal: Defining “implementing OpenLineage” [Julien]
  4. Update from MANTA on their OpenLineage integration [Eric and/or Petr from MANTA]
  5. Linking CMF (a common ML metadata framework) and OpenLineage [Suparna and AnnMary from HP Enterprise]
  6. Open discussion
    7. +
+ + + +
+ 👍 Luca Soato, Maciej Obuchowski, Paul Lee, Willy Lulciuc +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kenton (swiple.io) + (kknoxparton@gmail.com) +
+
2022-11-08 04:47:41
 +
+

Hi all 👋 I’m Kenton — a Software Engineer and founder of Swiple. I’m looking forward to working with OpenLineage and its community to integrate data lineage and data observability. +https://swiple.io

+
+
swiple.io
+ + + +
Build trust in every dataset | Build trust in every dataset
+
+ Swiple is an automated data monitoring platform that helps analytics and data engineering teams seamlessly monitor the quality of their data. With automated data analysis and profiling, scheduling and alerting, teams can resolve data quality issues before they impact mission critical resources. +
+ + + + + + +
Original URL: https://swiple.io
+ + + +
+ + + +
+ 🙌 Maciej Obuchowski, Jakub Dardziński, Michael Robinson, Ross Turk, John Thomas, Julien Le Dem, Willy Lulciuc, Varun Singh +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-11-08 10:22:15
 +
+

*Thread Reply:* Welcome Kenton! Happy to help 👍

+ + + +
+ 👍 Kenton (swiple.io) +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Deepika Prabha + (deepikaprabha@gmail.com) +
+
2022-11-08 05:35:03
 +
+

Hi everyone, +We wanted to pass some dynamic metadata from spark job that we can catch up in OpenLineage event and use it for processing. Presently I have seen that we have few conf parameters like openlineage params that we can send only with Spark conf. Is there any other option we have where we can send some information dynamically from the spark jobs.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-11-08 10:06:10
 +
+

*Thread Reply:* What kind of data? My first feeling is that you need to extend the Spark integration

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Deepika Prabha + (deepikaprabha@gmail.com) +
+
2022-11-09 00:35:29
 +
+

*Thread Reply:* Yes, we wanted to add information like user/job description that we can use later with rest of openlineage event fields in our system

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Deepika Prabha + (deepikaprabha@gmail.com) +
+
2022-11-09 00:41:35
 +
+

*Thread Reply:* I can see in this PR https://github.com/OpenLineage/OpenLineage/pull/490 that env values can be captured which we can use to add some custom metadata but it seems it is specific to Databricks only.

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/pull/490|#490 Added generic facet to collect environmental properties (EnvironmentFacet)>
+
+ <p><strong>Problem</strong></p> + +<p>👋 Introduce the ability to produce environmental properties using EnvironmentFacet, this facet can be used to work with any environment, based on the conditional check to verify environment EnvironmentFacet can be added.</p> + +<p>Closes: <a href="https://github.com/OpenLineage/OpenLineage/issues/464">#464</a></p> + +<p><strong>Solution</strong></p> + +<p>Generic implementation of EnvironmentFacet gives the ability to add environmental details. Since EnvironmentFacet produces HashMap it can handle properties dynamically.</p> + +<p>In the current implementation added ability to collect properties for Databricks environment including mount point details. <br /> +Below are some examples:</p> + +<p>• "orgId" +• "spark.databricks.clusterUsageTags.clusterOwnerOrgId" +• "spark.databricks.notebook.path" +• "spark.databricks.job.type" +• "spark.databricks.job.id" +• "spark.databricks.job.runId" +• "user" +• "userId" +• "spark.databricks.clusterUsageTags.clusterName" +• "spark.databricks.clusterUsageTags.azureSubscriptionId"</p> + +<p><strong>Checklist</strong></p> + +<p>☑︎ You've <a href="https://github.com/OpenLineage/OpenLineage/blob/main/why-the-dco.md">signed-off</a> your work +☑︎ Your pull request title follows our <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CONTRIBUTING.md#creating-pull-requests">guidelines</a> +☐ Your changes are accompanied by tests (<em>if relevant</em>) +☑︎ Your change contains a <a href="https://kurtisnusbaum.medium.com/stacked-diffs-keeping-phabricator-diffs-small-d9964f4dcfa6">small diff</a> and is self-contained +☐ You've updated any relevant documentation (<em>if relevant</em>) +☐ You've updated the <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md"><code>CHANGELOG.md</code></a> with details about your change under the "Unreleased" section (<em>if relevant, depending on the change, this may not be necessary</em>) +☐ You've versioned the core OpenLineage model or facets according to <a href="https://docs.snowplowanalytics.com/docs/pipeline-components-and-applications/iglu/common-architecture/schemaver">SchemaVer</a> (<em>if relevant</em>)</p> +
+ +
+
Comments
+ 8 +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-11-09 05:14:50
 +
+

*Thread Reply:* I think it makes sense to have something like that, but generic, if you want to contribute it

+ + + +
+ 👍 Will Johnson, Deepika Prabha +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Varun Singh + (varuntestaz@outlook.com) +
+
2022-11-14 03:28:35
 +
+

*Thread Reply:* @Maciej Obuchowski Do you mean adding something like +spark.openlineage.jobFacet.FacetName.Key=Value to the spark conf should add a new job facet like +"FacetName": { + "Key": "Value" +}

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-11-14 05:56:02
 +
+

*Thread Reply:* We can argue about name of that key, but yes, something like that. Just notice that while it's possible to attach something to run and job facets directly, it would be much harder to do this with datasets

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
slackbot + +
+
2022-11-09 11:15:49
 +
+

This message was deleted.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2022-11-10 02:22:18
 +
+

*Thread Reply:* Hi @Varun Singh, what version of openlineage-spark where you using? Are you able to copy lineage event here?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-11-09 12:31:10
 +
+

@channel +This month’s TSC meeting is tomorrow at 10 am PT! https://openlineage.slack.com/archives/C01CK9T7HKR/p1667512998061829

+
+ + +
+ + + } + + Michael Robinson + (https://openlineage.slack.com/team/U02LXF3HUN7) +
+ + + +
+
+ <p>@channel +This month’s OpenLineage TSC meeting is <strong>next Thursday, November 10th, at 10 am PT</strong>. Join us on Zoom: <a href="https://bit.ly/OLzoom">https://bit.ly/OLzoom</a>. All are welcome! +On the tentative agenda:</p> + +<ol><li>Recent release overview [Michael R.]</li><li>Update on LFAI &amp; Data Foundation progress [Michael R.]</li><li>Proposal: Defining “implementing OpenLineage” [Julien]</li><li>Update from MANTA on their OpenLineage integration [Eric and/or Petr from MANTA]</li><li>Linking CMF (a common ML metadata framework) and OpenLineage [Suparna and AnnMary from HP Enterprise]</li><li>Open discussion</li> +</ol> +
+ + + + + + +
Original URL: https://openlineage.slack.com/archives/C01CK9T7HKR/p1667512998061829
+ + +
+ + Slack Conversation +
+ + +
+ + + +
+ 💥 Willy Lulciuc, Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Hanna Moazam + (hannamoazam@microsoft.com) +
+
2022-11-11 11:32:54
 +
+

Hi #general, quick question: do we plan to disable spark 2 support in the near future?

+ +

Longer question: +I've recently made a PR (https://github.com/OpenLineage/OpenLineage/pull/1231) to support capturing lineage from Snowflake, but it fails at a specific integration test due to what we think is a dependency mismatch for guava. I've tried to exclude any transient dependencies which may cause the problem but no luck with that so far.

+ +

Just wondering if:

+ +
  1. It makes sense to spend more time trying to ensure that test passes? Especially if we plan to remove spark 2 support soon.
  2. Assuming we do want to make sure to pass the test, does anyone have any other ideas for where to look/modify to prevent the error? +Here's the test failure message: +```io.openlineage.spark.agent.lifecycle.LibraryTest testRdd(SparkSession) FAILED (16s)
    3. +
+ +

java.lang.IllegalAccessError: tried to access method com.google.common.base.Stopwatch.<init>()V from class org.apache.hadoop.mapred.FileInputFormat + at io.openlineage.spark.agent.lifecycle.LibraryTest.testRdd(LibraryTest.java:113) ``` +Thanks in advance!

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/pull/1231|#1231 spark: add Snowflake Relation Visitor>
+
+ <p><strong>Problem</strong></p> + +<p>OpenLineage's Spark integration does not currently support Snowflake.</p> + +<p><strong>Solution</strong></p> + +<p>Implemented the SnowflakeRelationVisitor to support lineage for the <a href="https://github.com/snowflakedb/spark-snowflake">Snowflake Spark connector</a>.</p> + +<p>There are two paths in OpenLineage to the SnowflakeRelationVisitor:</p> + +<p>• Write events reach the SaveIntoDataSourceCommandVisitor, which routes the command to the <code>createSnowflakeDatasets</code> method. +• Read events trigger SnowflakeRelationVisitor's apply method.</p> + +<p>The implementation has been tested locally and on Azure Databricks.</p> + +<p>☐ Your change modifies the <a href="https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.json">core</a> OpenLineage model +☐ Your change modifies one or more OpenLineage <a href="https://github.com/OpenLineage/OpenLineage/tree/main/spec/facets">facets</a></p> + +<p><strong>Checklist</strong></p> + +<p>☑︎ You've <a href="https://github.com/OpenLineage/OpenLineage/blob/main/why-the-dco.md">signed-off</a> your work +☑︎ Your pull request title follows our <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CONTRIBUTING.md#creating-pull-requests">guidelines</a> +☑︎ Your changes are accompanied by tests (<em>if relevant</em>) +☑︎ Your change contains a <a href="https://kurtisnusbaum.medium.com/stacked-diffs-keeping-phabricator-diffs-small-d9964f4dcfa6">small diff</a> and is self-contained +☐ You've updated any relevant documentation (<em>if relevant</em>) +☐ You've updated the <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md"><code>CHANGELOG.md</code></a> with details about your change under the "Unreleased" section (<em>if relevant, depending on the change, this may not be necessary</em>) +☐ You've versioned the core OpenLineage model or facets according to <a href="https://docs.snowplowanalytics.com/docs/pipeline-components-and-applications/iglu/common-architecture/schemaver">SchemaVer</a> (<em>if relevant</em>) +☑︎ You've added a <a href="https://github.com/OpenLineage/OpenLineage/tree/main/.github/header_templates.md">header</a> to source files (<em>if relevant</em>)</p> +
+ +
+
Labels
+ documentation, integration/spark, spec +
+ +
+
Comments
+ 4 +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-11-11 16:28:07
 +
+

*Thread Reply:* What if we just not include it in the BaseVisitorFactory but only in the Spark3 visitor factories?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paul Lee + (paullee@lyft.com) +
+
2022-11-11 14:52:19
 +
+

quick question: how do i get the &lt;&lt;non-serializable Time...to show in the extraction? or really any object that gets passed in.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-11-11 16:24:30
 +
+

*Thread Reply:* You might look here: https://github.com/OpenLineage/OpenLineage/blob/f7049c599a0b1416408860427f0759624326677d/client/python/openlineage/client/serde.py#L51

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/f7049c599a0b1416408860427f0759624326677d/client/python/openlineage/client/serde.py | serde.py>
+
+ <pre><code> default=lambda o: f"&lt;&lt;non-serializable: {type(o).__qualname__}&gt;&gt;" +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+ :gratitude_thank_you: Paul Lee +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
srutikanta hota + (srutikanta.hota@gmail.com) +
+
2022-11-14 01:12:45
 +
+

Is there a way I can update the detaset description and the column description. While generating the open lineage spark events and columns

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2022-11-15 02:09:25
 +
+

*Thread Reply:* I don’t think this is possible at the moment.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2022-11-15 15:47:49
 +
+

Hey all, I'd like to ask for a release for OpenLineage. #1256 fixes bug in DefaultExtractor. This blocks people from migrating code from custom extractors to get_openlineage_facets methods.

+ + + +
+ ➕ Michael Robinson, Howard Yoo, Maciej Obuchowski, Willy Lulciuc +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-11-16 09:13:17
 +
+

*Thread Reply:* Thanks, all. The release is authorized.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-11-16 10:41:07
 +
+

*Thread Reply:* The PR for the changelog updates: https://github.com/OpenLineage/OpenLineage/pull/1306

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Varun Singh + (varuntestaz@outlook.com) +
+
2022-11-16 03:34:01
 +
+

Hi, small question: Is it possible to disable the /api/{version}/lineage suffix that gets added to every url automatically? Thanks!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-11-16 12:27:12
 +
+

*Thread Reply:* I think we had similar request before, but nothing was implemented.

+ + + +
+ 👍 Varun Singh +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-11-16 12:23:54
 +
+

@channel +OpenLineage 0.17.0 is now available, featuring: +Additions: +• Spark: support latest Spark 3.3.1 #1183 @pawel-big-lebowski +• Spark: add Kinesis Transport and support config Kinesis in Spark integration #1200 @yogyang +• Spark: disable specified facets #1271 @pawel-big-lebowski +• Python: add facets implementation to Python client #1233 @pawel-big-lebowski +• SQL: add Rust parser interface #1172 @StarostaGit @mobuchowski +• Proxy: add helm chart for the proxy backend #1068 @wslulciuc +• Spec: include possible facets usage in spec #1249 @pawel-big-lebowski +• Website: publish YML version of spec to website #1300 @rossturk +• Docs: update language on nominating new committers #1270 @rossturk +Changes: +• Website: publish spec into new website repo location #1295 @rossturk +• Airflow: change how pip installs packages in tox environments #1302 @JDarDagran +Removals: +• Deprecate HttpTransport.Builder in favor of HttpConfig #1287 @collado-mike +Bug fixes and more! +Release: https://github.com/OpenLineage/OpenLineage/releases/tag/0.17.0 +Changelog: https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md +Commit history: https://github.com/OpenLineage/OpenLineage/compare/0.16.1...0.17.0 +Maven: https://oss.sonatype.org/#nexus-search;quick~openlineage +PyPI: https://pypi.org/project/openlineage-python/

+ + + +
+ 🙌 Howard Yoo, Maciej Obuchowski, Ross Turk, Aphra Bloomfield, Harel Shein, Kengo Seki, Paweł Leszczyński, pankaj koti, Varun Singh +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Diego Cesar + (dcesar@krakenrobotik.de) +
+
2022-11-18 05:40:53
 +
+

Hi everyone,

+ +

I'm trying to get the lineage of a dataset per version. I initially had something like

+ +

Dataset A -&gt; Dataset B -&gt; DataSet C (version 1)

+ +

then:

+ +

Dataset D -&gt; Dataset E -&gt; DataSet C (version 2)

+ +

I can get the graph for version 2 without problems, but I'm wondering if there's any way to retrieve the entire graph for DataSet C version 1.

+ +

Thanks

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-11-22 13:40:44
 +
+

*Thread Reply:* It's kind of a hard problem UI side. Backend can express that relationship

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Diego Cesar + (dcesar@krakenrobotik.de) +
+
2022-11-22 13:48:58
 +
+

*Thread Reply:* Thanks for replying. Could you please point me to the API that allows me to do that? I've been calling GET /lineage with dataset in the node ID, e g., nodeId=dataset:my_dataset . Where could I specify the version of my dataset?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paul Lee + (paullee@lyft.com) +
+
2022-11-18 17:55:24
 +
+

👋 how do we get the actual values from macros? e.g. a schema name is passed in with {{params.table_name}} and thats what shows in lineage instead of the actual table name

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2022-11-19 04:54:13
 +
+

*Thread Reply:* Templated fields are rendered before generating lineage data. Do you have some sample code or logs preferrably?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-11-22 13:40:11
 +
+

*Thread Reply:* If you're on 1.10 then I think it won't work

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paul Lee + (paullee@lyft.com) +
+
2022-11-28 12:50:39
 +
+

*Thread Reply:* @Maciej Obuchowski we are still on airflow 1.10.15 unfortunately.

+ +

cc. @Eli Schachar @Allison Suarez

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paul Lee + (paullee@lyft.com) +
+
2022-11-28 12:50:49
 +
+

*Thread Reply:* is there no workaround we can make work?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paul Lee + (paullee@lyft.com) +
+
2022-11-28 12:51:01
 +
+

*Thread Reply:* @Jakub Dardziński is this for airflow versions 2.0+?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Varun Singh + (varuntestaz@outlook.com) +
+
2022-11-21 07:07:10
 +
+

Hey, quick question: I see there is Kafka transport in the java client, but it's not supported in the spark integration, right?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-11-21 07:28:04
 +
+

*Thread Reply:* Yeah. However, to add it, just tiny bit of code would be required.

+ +

Either in the URL version https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/app/src/main/java/io/openlineage/spark/agent/EventEmitter.java#L48

+ +

Or as separate Spark config entry: https://github.com/OpenLineage/OpenLineage/blob/182d2e5a907e6602f7fe132e07ea569c7e[…]n/java/io/openlineage/spark/agent/OpenLineageSparkListener.java

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/app/src/main/java/io/openlineage/spark/agent/EventEmitter.java | EventEmitter.java>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/182d2e5a907e6602f7fe132e07ea569c7e923dff/integration/spark/app/src/main/java/io/openlineage/spark/agent/OpenLineageSparkListener.java | OpenLineageSparkListener.java>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
srutikanta hota + (srutikanta.hota@gmail.com) +
+
2022-11-22 13:03:41
 +
+

How can we auto instrument a dataset owner at Java agent level? Is there any spark property available?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
srutikanta hota + (srutikanta.hota@gmail.com) +
+
2022-11-22 16:47:37
 +
+

Is there a way if we are running a job with business day as yesterday to capture the information. Just think if I am running yesterday missing job today. Or Friday's file on Monday as we received file late from vendor etc..

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-11-22 18:45:48
 +
+

*Thread Reply:* I think that's what NominalTimeFacet covers

+
+
openlineage.io
+ + + +
Nominal Time Facet | OpenLineage Docs
+
+ The facet to describe the nominal start and end time of the run. The nominal usually means the time the job run was expected to run (like a scheduled time), and the actual time can be different. +
+ + + + + + +
Original URL: https://openlineage.io/docs/spec/facets/run-facets/nominal_time
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Rahul Sharma + (panditrahul151197@gmail.com) +
+
2022-11-24 09:15:45
 +
+

hello Team, i wanna to use data lineage using airflow but not getting understand from docs please let me know if someone have pretty docs

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harel Shein + (harel.shein@gmail.com) +
+
2022-11-28 10:29:58
 +
+

*Thread Reply:* Hey @Rahul Sharma, what version of Airflow are you running?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Rahul Sharma + (panditrahul151197@gmail.com) +
+
2022-11-28 10:30:14
 +
+

*Thread Reply:* i am using airflow 2.x

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Rahul Sharma + (panditrahul151197@gmail.com) +
+
2022-11-28 10:30:27
 +
+

*Thread Reply:* can we connect if you have time ?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harel Shein + (harel.shein@gmail.com) +
+
2022-11-28 11:11:58
 +
+

*Thread Reply:* did you see these docs before? https://openlineage.io/integration/apache-airflow/#airflow-20

+
+
openlineage.io
+ + + +
Apache Airflow
+
+ Enabling OpenLineage in Apache Airflow automatically tracks metadata about jobs and datasets as DAGs execute. +
+ + + + + + +
Original URL: https://openlineage.io/integration/apache-airflow/#airflow-20
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Rahul Sharma + (panditrahul151197@gmail.com) +
+
2022-11-28 11:12:22
 +
+

*Thread Reply:* yes

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Rahul Sharma + (panditrahul151197@gmail.com) +
+
2022-11-28 11:12:36
 +
+

*Thread Reply:* i already set configuration in airflow.cfg file

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harel Shein + (harel.shein@gmail.com) +
+
2022-11-28 11:12:57
 +
+

*Thread Reply:* where are you sending the events to?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Rahul Sharma + (panditrahul151197@gmail.com) +
+
2022-11-28 11:13:24
 +
+

*Thread Reply:* i have a docker machine on which marquez is working

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harel Shein + (harel.shein@gmail.com) +
+
2022-11-28 11:13:47
 +
+

*Thread Reply:* so, what is the issue you are seeing?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Rahul Sharma + (panditrahul151197@gmail.com) +
+
2022-11-28 11:15:37
 +
+

*Thread Reply:* there is no error

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Rahul Sharma + (panditrahul151197@gmail.com) +
+
2022-11-28 11:16:01
 +
+

*Thread Reply:* ```[lineage]

+ +

what lineage backend to use

+ +

backend =openlineage.lineage_backend.OpenLineageBackend

+ +

MARQUEZ_URL=http://10.36.37.178:3000

+ +

MARQUEZ_NAMESPACE=airflow

+ +

MARQUEZBACKEND=HTTP +MARQUEZURL=http://10.36.37.178:5000

+ +

MARQUEZAPIKEY=[YOURAPIKEY]

+ +

MARQUEZ_NAMESPACE=airflow```

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Rahul Sharma + (panditrahul151197@gmail.com) +
+
2022-11-28 11:16:09
 +
+

*Thread Reply:* above config i have set

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Rahul Sharma + (panditrahul151197@gmail.com) +
+
2022-11-28 11:16:22
 +
+

*Thread Reply:* please let me know any other thing need to do

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mohamed Nabil H + (m.nabil.hafez@gmail.com) +
+
2022-11-24 14:02:27
 +
+

hey i wonder if somebody can link me to the lineage ( table lineage ) event schema ?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2022-11-25 02:20:40
 +
+

*Thread Reply:* please have a look at openapi definition of the event: https://openlineage.io/apidocs/openapi/

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Murali Krishna + (vmurali.krishnaraju@genpact.com) +
+
2022-11-30 02:34:51
 +
+

Hello Team, I am from Genpact Data Analytics team, we are looking for demo of your product

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Conor Beverland + (conorbev@gmail.com) +
+
2022-11-30 14:10:10
 +
+

*Thread Reply:* hey, I'll DM you.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-12-01 15:00:28
 +
+

Hello all, I’m calling for a vote on releasing OpenLineage 0.18.0, including: +• improvements to the Spark integration, +• extractors for Sagemaker operators and SFTPOperator in the Airflow integration, +• a change to the Databricks integration to support Databricks Runtime 11.3, +• new governance docs, +• bug fixes, +• and more. +Three +1s from committers will authorize an immediate release.

+ + + +
+ ➕ Maciej Obuchowski, Will Johnson, Bramha Aelem +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-12-06 13:56:17
 +
+

*Thread Reply:* Thanks, all. The release is authorized will be initiated within two business days.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-12-01 15:11:10
 +
+

@channel +This month’s OpenLineage TSC meeting is next Thursday, December 8th, at 10 am PT. Join us on Zoom: https://bit.ly/OLzoom. All are welcome! +On the tentative agenda:

+ +
  1. an overview of the new Rust implementation of the SQL integration
  2. a pesentation/discussion of what it actually means to “implement” OpenLineage
  3. open discussion.
    4. +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Scott Anderson + (scott.anderson@alteryx.com) +
+
2022-12-02 13:57:07
 +
+

Hello everyone! General question here, aside from ‘consumer’ orgs/integrations (dbt/dagster/manta), is anyone aware of any enterprise organizations that are leveraging OpenLineage today? Example lighthouse brands?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-12-02 15:21:20
 +
+

*Thread Reply:* Microsoft https://openlineage.io/blog/openlineage-microsoft-purview/

+
+
openlineage.io
+ + + +
Microsoft Purview Accelerates Lineage Extraction from Azure Databricks
+
+ A new collaboration between Microsoft and OpenLineage is making lineage extraction possible for Azure Databricks and Microsoft Purview users. +
+ + + + + + +
Original URL: https://openlineage.io/blog/openlineage-microsoft-purview/
+ + + +
+ + + +
+ 🙌 Will Johnson +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-12-05 13:54:06
 +
+

*Thread Reply:* I think we can share that we have over 2,000 installs of that Microsoft solution accelerator using OpenLineage.

+ +

That means we have thousands of companies having experimented with OpenLineage and Microsoft Purview.

+ +

We can't name any customers at this point unfortunately.

+ + + +
+ 🎉 Conor Beverland, Kengo Seki +
+ +
+ 👍 Scott Anderson +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-12-07 12:03:06
 +
+

@channel +This month’s TSC meeting is tomorrow at 10 am PT. All are welcome! https://openlineage.slack.com/archives/C01CK9T7HKR/p1669925470878699

+
+ + +
+ + + } + + Michael Robinson + (https://openlineage.slack.com/team/U02LXF3HUN7) +
+ + + +
+
+ <p><strong>@channel</strong> +This month’s OpenLineage TSC meeting is <strong>next Thursday, December 8th, at 10 am PT</strong>. Join us on Zoom: <a href="https://bit.ly/OLzoom">https://bit.ly/OLzoom</a>. All are welcome! +On the tentative agenda:</p> + +<ol><li>an overview of the new Rust implementation of the SQL integration</li><li>a pesentation/discussion of what it actually means to “implement” OpenLineage</li><li>open discussion.</li> +</ol> +
+ + + + + + +
Original URL: https://openlineage.slack.com/archives/C01CK9T7HKR/p1669925470878699
+ + +
+ + Slack Conversation +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2022-12-07 14:22:58
 +
+

*Thread Reply:* For open discussion, I'd like to ask the team for an overview of how the different gradle files are working together for the Spark implementation. I'm terribly confused on where dependencies need to be added (whether it's in shared, app, or a spark version specific folder). Maybe @Maciej Obuchowski...?

+ + + +
+ 👍 Michael Robinson +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2022-12-07 14:25:12
 +
+

*Thread Reply:* Unfortunately I'll be unable to attend the meeting @Will Johnson 😞

+ + + +
+ 😭 Will Johnson +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2022-12-08 13:03:08
 +
+

*Thread Reply:* This is starting now. CC @Will Johnson

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2022-12-09 19:24:15
 +
+

*Thread Reply:* @Will Johnson Check the notes and the recording. @Michael Collado did a pass at explaining the relationship between shared, app and the versions

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2022-12-09 19:24:30
 +
+

*Thread Reply:* feel free to follow up here as well

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2022-12-09 19:39:37
 +
+

*Thread Reply:* ascii art to the rescue! (top “depends on” bottom)

+ +
              /   \
+             / / \ \
+            / /   \ \
+           / /     \ \
+          / /       \ \
+         / |         | \
+        /  |         |  \
+       /   |         |   \
+      /    |         |    \
+     /     |         |     \
+    /      |         |      \
+   /       |         |       \
+spark2   spark3   spark32   spark33
+   \        |        |       /
+    \       |        |      /
+     \      |        |     /
+      \     |        |    /
+       \    |        |   /
+        \   |        |  /
+         \  |        | /
+          \ |       / /
+           \ \     / /
+            \ \   / /
+             \ \ / /
+              \   /
+               \ /
+             share
+
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2022-12-09 19:40:05
 +
+

*Thread Reply:* 😍

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2022-12-09 19:41:13
 +
+

*Thread Reply:* (btw, we should have written datakin to output ascii art; it’s obviously the superior way to generate graphs 😜)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2022-12-14 05:18:53
 +
+

*Thread Reply:* Hi, is there a recording for this meeting?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Christian Lundgren + (christian@lunit.io) +
+
2022-12-07 20:33:19
 +
+

Hi! I have a basic question about the naming conventions for blob storage. The spec is not totally clear to me. Is the convention to use (1) namespace=bucket name=bucket+path or (2) namespace=bucket name=path?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2022-12-07 22:05:25
 +
+

*Thread Reply:* The namespace is the bucket and the dataset name is the path. Is there a blob storage provider in particular you are thinking of?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Christian Lundgren + (christian@lunit.io) +
+
2022-12-07 23:13:41
 +
+

*Thread Reply:* Thanks, that makes sense. We use GCS, so it is already covered by the naming conventions documented. I was just not sure if I was understanding the document correctly or not.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2022-12-07 23:34:33
 +
+

*Thread Reply:* No problem. Let us know if you have suggestions on the wording to make the doc clearer

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2022-12-08 11:44:49
 +
+

@channel +OpenLineage 0.18.0 is available now, featuring: +• Airflow: support SQLExecuteQueryOperator #1379 @JDarDagran +• Airflow: introduce a new extractor for SFTPOperator #1263 @sekikn +• Airflow: add Sagemaker extractors #1136 @fhoda +• Airflow: add S3 extractor for Airflow operators #1166 @fhoda +• Spec: add spec file for ExternalQueryRunFacet #1262 @howardyoo +• Docs: add a TSC doc #1303 @merobi-hub +• Plus bug fixes. +Thanks to all our contributors, including new contributor @Faisal Hoda! +Release: https://github.com/OpenLineage/OpenLineage/releases/tag/0.18.0 +Changelog: https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md +Commit history: https://github.com/OpenLineage/OpenLineage/compare/0.17.0...0.18.0 +Maven: https://oss.sonatype.org/#nexus-search;quick~openlineage +PyPI: https://pypi.org/project/openlineage-python/

+ + + +
+ 🚀 Willy Lulciuc, Minkyu Park, Kengo Seki, Enrico Rotundo, Faisal Hoda +
+ +
+ 🙌 Howard Yoo, Minkyu Park, Kengo Seki, Enrico Rotundo, Faisal Hoda +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
srutikanta hota + (srutikanta.hota@gmail.com) +
+
2022-12-09 01:42:59
 +
+

1) Is there a specifications to capture dataset dependency. ds1 is dependent on ds2

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-12-09 11:51:16
 +
+

*Thread Reply:* Dataset dependencies are represented through common relationship with a Job - e.g., the task that performed the transformation.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
srutikanta hota + (srutikanta.hota@gmail.com) +
+
2022-12-11 09:01:19
 +
+

*Thread Reply:* Is it possible to populate table level dependency without any transformation using open lineage specifications? Like to define dataset 1 is dependent of table 1 and table 2 which can be represented as separate datasets

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-12-13 15:24:20
 +
+

*Thread Reply:* Not explicitly, in today's spec. The guiding principle is that something created that dependency, and the dependency changes over time in a way that is important to study.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-12-13 15:25:12
 +
+

*Thread Reply:* I say this to explain why it is the way it is - but the spec can change over time to serve new uses cases, certainly!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2022-12-14 05:18:10
 +
+

Hi everyone, I'd like to use openlineage to capture column level lineage for spark. I would also like to capture a few custom environment variables along with the column lineage. May I know how this can be done? Thanks!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2022-12-14 09:56:22
 +
+

*Thread Reply:* Hi @Anirudh Shrinivason, you could start with column-lineage & spark workshop available here -> https://github.com/OpenLineage/workshops/tree/main/spark

+ + + +
+ ❤️ Ricardo Gaspar +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2022-12-14 10:05:54
 +
+

*Thread Reply:* Hi @Paweł Leszczyński Thanks for the link! But this does not really answer the concern.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2022-12-14 10:06:08
 +
+

*Thread Reply:* I am already able to capture column lineage

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2022-12-14 10:06:33
 +
+

*Thread Reply:* What I would like is to capture some extra environment variables, and send it to the server along with the lineage

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2022-12-14 11:22:59
 +
+

*Thread Reply:* i remember we already have a facet for that: https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/shared/src/main/java/io/openlineage/spark/agent/facets/EnvironmentFacet.java

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/shared/src/main/java/io/openlineage/spark/agent/facets/EnvironmentFacet.java | EnvironmentFacet.java>
+
+ <pre><code>/** +/** Copyright 2018-2022 contributors to the OpenLineage project +/** SPDX-License-Identifier: Apache-2.0 +**/ + +package io.openlineage.spark.agent.facets; + +import com.fasterxml.jackson.annotation.JsonProperty; +import io.openlineage.client.OpenLineage; +import io.openlineage.spark.agent.Versions; +import java.util.Map; + +/**** + ** Facet used to report environment specific properties. For example, reporting the name of the + ** cluster used, reporting certain environment variables, or resolving mount points. + **/ +public class EnvironmentFacet extends OpenLineage.DefaultRunFacet { + @JsonProperty("environment-properties") + @SuppressWarnings("PMD") + private Map&lt;String, Object&gt; properties; + + public EnvironmentFacet(Map&lt;String, Object&gt; environmentDetails) { + super(Versions.OPEN_LINEAGE_PRODUCER_URI); + this.properties = environmentDetails; + } +} + +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2022-12-14 11:24:07
 +
+

*Thread Reply:* but it is only used at the moment to capture some databricks environment attributes

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2022-12-14 11:28:29
 +
+

*Thread Reply:* so you can contribute to project and add a feature which adds specified/al environment variables to lineage event.

+ +

you can also have a look at extending section of spark integration docs (https://github.com/OpenLineage/OpenLineage/tree/main/integration/spark#extending) and create a class thats add run facet builder according to your needs.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2022-12-14 11:29:28
 +
+

*Thread Reply:* the third way is to create an issue related to this bcz being able to send selected/all environment variables in OL event seems to be really cool feature.

+ + + +
+ 👍 Anirudh Shrinivason +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2022-12-14 21:49:19
 +
+

*Thread Reply:* That is great! Thank you so much! This really helps!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2022-12-15 01:44:42
 +
+

*Thread Reply:* List&lt;String&gt; dbPropertiesKeys = + Arrays.asList( + "orgId", + "spark.databricks.clusterUsageTags.clusterOwnerOrgId", + "spark.databricks.notebook.path", + "spark.databricks.job.type", + "spark.databricks.job.id", + "spark.databricks.job.runId", + "user", + "userId", + "spark.databricks.clusterUsageTags.clusterName", + "spark.databricks.clusterUsageTags.azureSubscriptionId"); + dbPropertiesKeys.stream() + .forEach( + (p) -&gt; { + dbProperties.put(p, jobStart.properties().getProperty(p)); + }); +It seems like it is obtaining these env variable information from the jobStart obj, but not capturing from the env directly?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2022-12-15 01:57:05
 +
+

*Thread Reply:* I have opened an issue in the community here: https://github.com/OpenLineage/OpenLineage/issues/1419

+
+ + + + +
#1419 [PROPOSAL] Add custom environment variables to the RunEvent
+
+ <p>Hi, <br /> +I would like to capture some custom environment variables and send it along with the column lineage to the server. This feature will be of great use as there is a potential use case to capture and persist some environment-specific metadata from spark clusters. <br /> +Wondering if this feature can be supported, or is on the roadmap potentially? Thanks a lot! <br /> +cc: <a href="https://github.com/pawel-big-lebowski">@pawel-big-lebowski</a></p> +
+ +
+
Labels
+ proposal +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-02-01 02:24:39
 +
+

*Thread Reply:* Hi @Paweł Leszczyński I have opened a PR for helping to add this use case. Please do help to see if we can merge it in. Thanks! +https://github.com/OpenLineage/OpenLineage/pull/1545

+
+ + + + +
#1545 Add CustomEnvironmentFacetBuilder class
+
+ <p><strong>Problem</strong></p> + +<p>The main idea behind this pull request is to enable a way to capture custom environment variables from a spark integration. The flow defined here, as applicable to our use case is to have a CUSTOM_VARS="VAR1,VAR2,VAR3..." environment variable that indicates which environment variables should be captured. Then, we would want to capture those variables, ie VAR1, VAR2, VAR3, ... from the spark clusters.</p> + +<p>Closes: <a href="https://github.com/OpenLineage/OpenLineage/issues/1419">#1419</a></p> + +<p><strong>Solution</strong></p> + +<p>The solution proposed extends the EnvironmentFacets class, to build a CustomEnvironmentFacetBuilder class. This class then is used to get the values of the environment variables and pass on to the startEvent json.</p> + +<p>☐ Your change modifies the <a href="https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.json">core</a> OpenLineage model +☑︎ Your change modifies one or more OpenLineage <a href="https://github.com/OpenLineage/OpenLineage/tree/main/spec/facets">facets</a></p> + +<p>If you're contributing a new integration, please specify the scope of the integration and how/where it has been tested (e.g., Apache Spark integration supports <code>S3</code> and <code>GCS</code> filesystem operations, tested with AWS EMR).</p> + +<p><strong>Checklist</strong></p> + +<p>☑︎ You've <a href="https://github.com/OpenLineage/OpenLineage/blob/main/why-the-dco.md">signed-off</a> your work +☐ Your pull request title follows our <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CONTRIBUTING.md#creating-pull-requests">guidelines</a> +☐ Your changes are accompanied by tests (<em>if relevant</em>) +☐ Your change contains a <a href="https://kurtisnusbaum.medium.com/stacked-diffs-keeping-phabricator-diffs-small-d9964f4dcfa6">small diff</a> and is self-contained +☐ You've updated any relevant documentation (<em>if relevant</em>) +☐ You've updated the <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md"><code>CHANGELOG.md</code></a> with details about your change under the "Unreleased" section (<em>if relevant, depending on the change, this may not be necessary</em>) +☐ You've versioned the core OpenLineage model or facets according to <a href="https://docs.snowplowanalytics.com/docs/pipeline-components-and-applications/iglu/common-architecture/schemaver">SchemaVer</a> (<em>if relevant</em>) +☐ You've added a <a href="https://github.com/OpenLineage/OpenLineage/tree/main/.github/header_templates.md">header</a> to source files (<em>if relevant</em>)</p> + +<hr /> + +<p>SPDX-License-Identifier: Apache-2.0 <br /> +Copyright 2018-2023 contributors to the OpenLineage project</p> +
+ +
+
Labels
+ integration/spark +
+ +
+
Comments
+ 1 +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+ 👀 Maciej Obuchowski, Ross Turk +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-02-02 11:45:52
 +
+

*Thread Reply:* Hey @Anirudh Shrinivason, sorry for late reply, but I reviewed the PR.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-02-06 03:06:42
 +
+

*Thread Reply:* Hey thanks a lot! I have made the requested changes! Thanks!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-02-06 03:06:49
 +
+

*Thread Reply:* @Maciej Obuchowski ^ 🙂

+ + + +
+ 👀 Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-02-06 09:09:34
 +
+

*Thread Reply:* Hey @Anirudh Shrinivason, took a look at it but it unfortunately fails integration tests (throws NPE), can you take a look again?

+ +

23/02/06 12:18:39 ERROR AsyncEventQueue: Listener OpenLineageSparkListener threw an exception + java.lang.NullPointerException + at io.openlineage.spark.agent.EventEmitter.&lt;init&gt;(EventEmitter.java:39) + at io.openlineage.spark.agent.OpenLineageSparkListener.initializeContextFactoryIfNotInitialized(OpenLineageSparkListener.java:276) + at io.openlineage.spark.agent.OpenLineageSparkListener.onOtherEvent(OpenLineageSparkListener.java:80) + at org.apache.spark.scheduler.SparkListenerBus.doPostEvent(SparkListenerBus.scala:100) + at org.apache.spark.scheduler.SparkListenerBus.doPostEvent$(SparkListenerBus.scala:28) + at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37) + at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37) + at org.apache.spark.util.ListenerBus.postToAll(ListenerBus.scala:117) + at org.apache.spark.util.ListenerBus.postToAll$(ListenerBus.scala:101) + at org.apache.spark.scheduler.AsyncEventQueue.super$postToAll(AsyncEventQueue.scala:105) + at org.apache.spark.scheduler.AsyncEventQueue.$anonfun$dispatch$1(AsyncEventQueue.scala:105) + at scala.runtime.java8.JFunction0$mcJ$sp.apply(JFunction0$mcJ$sp.java:23) + at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62) + at <a href="http://org.apache.spark.scheduler.AsyncEventQueue.org">org.apache.spark.scheduler.AsyncEventQueue.org</a>$apache$spark$scheduler$AsyncEventQueue$$dispatch(AsyncEventQueue.scala:100) + at org.apache.spark.scheduler.AsyncEventQueue$$anon$2.$anonfun$run$1(AsyncEventQueue.scala:96) + at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1433) + at org.apache.spark.scheduler.AsyncEventQueue$$anon$2.run(AsyncEventQueue.scala:96)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-02-07 04:17:02
 +
+

*Thread Reply:* Hi yeah my bad. It should be fixed in the latest push. But I think the tests are not running in the CI because of some GCP environment issue? I am not really sure how to fix it...

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-02-07 04:18:46
 +
+

*Thread Reply:* I can make them run, it's just that running them on forks is disabled. We need to make it more clear I suppose

+ + + +
+ 👍 Anirudh Shrinivason +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-02-07 04:24:38
 +
+

*Thread Reply:* Ahh I see thanks! Also, some of the tests are failing on my local, such as https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/app/src/test/java/io/openlineage/spark/agent/lifecycle/DeltaDataSourceTest.java. Is this expected behaviour?

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/app/src/test/java/io/openlineage/spark/agent/lifecycle/DeltaDataSourceTest.java | DeltaDataSourceTest.java>
+
+ <pre><code>/** +/** Copyright 2018-2023 contributors to the OpenLineage project +/** SPDX-License-Identifier: Apache-2.0 +**/ + +package io.openlineage.spark.agent.lifecycle; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertTrue; + +import io.openlineage.client.OpenLineage; +import io.openlineage.client.OpenLineage.RunEvent; +import io.openlineage.client.OpenLineage.RunEvent.EventType; +import io.openlineage.spark.agent.SparkAgentTestExtension; +import java.io.IOException; +import java.nio.file.Path; +import java.util.Arrays; +import java.util.List; +import java.util.Optional; +import java.util.concurrent.TimeoutException; +import org.apache.spark.sql.Dataset; +import org.apache.spark.sql.Row; +import org.apache.spark.sql.SaveMode; +import org.apache.spark.sql.SparkSession; +import org.apache.spark.sql.catalyst.expressions.GenericRowWithSchema; +import org.apache.spark.sql.types.LongType$; +import org.apache.spark.sql.types.Metadata; +import org.apache.spark.sql.types.StringType$; +import org.apache.spark.sql.types.StructField; +import org.apache.spark.sql.types.StructType; +import org.junit.jupiter.api.Tag; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.extension.ExtendWith; +import org.junit.jupiter.api.io.TempDir; +import org.mockito.ArgumentCaptor; +import org.mockito.Mockito; + +@ExtendWith(SparkAgentTestExtension.class) +@Tag("delta") +class DeltaDataSourceTest { + + @Test + void testInsertIntoDeltaSource(@TempDir Path tempDir, SparkSession spark) + throws IOException, InterruptedException, TimeoutException { + StructType tableSchema = + new StructType( + new StructField[] { + new StructField("name", StringType$.MODULE$, false, Metadata.empty()), + new StructField("age", LongType$.MODULE$, false, Metadata.empty()) + }); + Dataset&lt;Row&gt; df = + spark.createDataFrame( + Arrays.asList( + new GenericRowWithSchema(new Object[] {"john", 25L}, tableSchema), + new GenericRowWithSchema(new Object[] {"sam", 22L}, tableSchema), + new GenericRowWithSchema(new Object[] {"alicia", 35L}, tableSchema), + new GenericRowWithSchema(new Object[] {"bob", 47L}, tableSchema), + new GenericRowWithSchema(new Object[] {"jordan", 52L}, tableSchema), + new GenericRowWithSchema(new Object[] {"liz", 19L}, tableSchema), + new GenericRowWithSchema(new Object[] {"marcia", 83L}, tableSchema), + new GenericRowWithSchema(new Object[] {"maria", 40L}, tableSchema), + new GenericRowWithSchema(new Object[] {"luis", 8L}, tableSchema), + new GenericRowWithSchema(new Object[] {"gabriel", 30L}, tableSchema)), + tableSchema); + + String deltaDir = tempDir.resolve("deltaData").toAbsolutePath().toString(); + df.write().format("delta").option("path", deltaDir).mode(SaveMode.Overwrite).save(); + // wait for event processing to complete + StaticExecutionContextFactory.waitForExecutionEnd(); + + ArgumentCaptor&lt;RunEvent&gt; lineageEvent = ArgumentCaptor.forClass(RunEvent.class); + + Mockito.verify(SparkAgentTestExtension.OPEN_LINEAGE_SPARK_CONTEXT, Mockito.atLeast(2)) + .emit(lineageEvent.capture()); + List&lt;RunEvent&gt; events = lineageEvent.getAllValues(); + Optional&lt;RunEvent&gt; completionEvent = + events.stream() + .filter(e -&gt; e.getEventType().equals(EventType.COMPLETE) &amp;&amp; !e.getOutputs().isEmpty()) + .findFirst(); + assertTrue(completionEvent.isPresent()); + RunEvent event = completionEvent.get(); + List&lt;OpenLineage.OutputDataset&gt; outputs = event.getOutputs(); + assertEquals(1, outputs.size()); + assertEquals("file", outputs.get(0).getNamespace()); + assertEquals(deltaDir, outputs.get(0).getName()); + } +} + +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-02-07 07:20:11
 +
+

*Thread Reply:* tests failing isn't expected behaviour 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-02-08 03:37:23
 +
+

*Thread Reply:* Ahh yeap it was a local ide issue on my side. I added some tests to verify the presence of env variables too.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-02-08 03:47:22
 +
+

*Thread Reply:* @Anirudh Shrinivason let me know then when you'll push fixed version, I can run full tests then

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-02-08 03:49:35
 +
+

*Thread Reply:* I have pushed just now

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-02-08 03:49:39
 +
+

*Thread Reply:* You can run the tests

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-02-08 04:13:07
 +
+

*Thread Reply:* @Maciej Obuchowski mb I pushed again rn. Missed out a closing bracket.

+ + + +
+ 👍 Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-02-10 00:47:04
 +
+

*Thread Reply:* @Maciej Obuchowski Hi, could we merge this PR in? I'd like to see if we can have these changes in the new release...

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Bramha Aelem + (bramhaaelem@gmail.com) +
+
2022-12-15 17:14:02
 +
+

Hi All- I am sending lineage from ADF for each activity which i am performing. But the individual activities are representing correctly. How can I represent task1 as a parent to task2. can someone please share the sample json request for it.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-12-16 13:29:44
 +
+

*Thread Reply:* Hi 👋 this would require a series of JSON calls:

+ +
  1. start the first task
  2. end the first task, specify output dataset
  3. start the second task, specify input dataset
  4. end the second task
    5. +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-12-16 13:32:08
 +
+

*Thread Reply:* in OpenLineage relationships are typically Job -> Dataset -> Job, so +• you create a relationship between datasets by referring to them in the same job - i.e., this task ran that read from these datasets and wrote to those datasets +• you create a relationship between tasks by referring to the same datasets across both of them - i.e., this task wrote that dataset and this other task read from it

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-12-16 13:35:06
 +
+

*Thread Reply:* @Bramha Aelem if you look in this directory, you can find example start/complete JSON calls that show how to specify input/output datasets.

+ +

(it’s an airflow workshop, but those examples are for a part of the workshop that doesn’t involve airflow)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2022-12-16 13:35:46
 +
+

*Thread Reply:* (these can also be found in the docs)

+
+
openlineage.io
+ + + +
Example Lineage Events | OpenLineage Docs
+
+ Simple Examples +
+ + + + + + +
Original URL: https://openlineage.io/docs/development/examples
+ + + +
+ + + +
+ 👍 Ross Turk +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Bramha Aelem + (bramhaaelem@gmail.com) +
+
2022-12-16 14:49:30
 +
+

*Thread Reply:* @Ross Turk - Thanks for the details. will try and get back to you on it

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Bramha Aelem + (bramhaaelem@gmail.com) +
+
2022-12-17 19:53:21
 +
+

*Thread Reply:* @Ross Turk - Good Evening, It worked as expected. I am able to replicate the scenarios which I am looking for.

+ + + +
+ 👍 Ross Turk +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Bramha Aelem + (bramhaaelem@gmail.com) +
+
2022-12-17 19:53:48
 +
+

*Thread Reply:* @Ross Turk - Thanks for your response.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Bramha Aelem + (bramhaaelem@gmail.com) +
+
2023-01-12 13:23:56
 +
+

*Thread Reply:* @Ross Turk - First activity : I am making HTTP Call to pull the lookup data and store it in ADLS. +Second Activity : After the completion of first activity I am making Azure databricks call to use the lookup file and generate the output tables. How I can refer the databricks generated tables facets as an input to the subsequent activities in the pipeline. +When I refer it's as an input the spark tables metadata is not showing up. How can this be achievable. +After the execution of each activity in ADF Pipeline I am sending start and complete/fail event lineage to Marquez.

+ +

Can someone please guide me on this.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Bramha Aelem + (bramhaaelem@gmail.com) +
+
2022-12-15 17:19:34
 +
+

I am not using airflow in my Process. pls suggest

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Bramha Aelem + (bramhaaelem@gmail.com) +
+
2022-12-19 12:40:26
 +
+

Hi All - Good Morning, how the column lineage of data source when it ran by different teams and jobs in openlineage.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Al (Koii) + (al@koii.network) +
+
2022-12-20 14:26:57
 +
+

Hey folks! I'm al from Koii.network, very happy to have heard about this project :)

+ + + +
+ 👋 Willy Lulciuc, Maciej Obuchowski, Julien Le Dem +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2022-12-20 14:27:59
 +
+

*Thread Reply:* welcome! let’s us know if you have any questions

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Matt Menzenski + (matt@payitgov.com) +
+
2022-12-29 08:22:26
 +
+

Hello! I found the OpenLineage project today after searching for “OpenTelemetry” in the dbt Slack.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harel Shein + (harel.shein@gmail.com) +
+
2022-12-29 10:47:00
 +
+

*Thread Reply:* Hey Matt! Happy to have you here! Feel free to reach out if you have any questions

+ + + +
+ :gratitude_thank_you: Matt Menzenski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Max + (maxime.broussard@gmail.com) +
+
2022-12-30 05:33:40
 +
+

Hi guys - I am really excited to test open lineage. +I had a quick question, sorry if this is not the right place for it. +We are testing dbt-ol with airflow and I was hoping this would by default push the number of rows updated/created in that dbt transformation to marquez. +It runs fine on airflow, but when I check in marquez there doesn't seem to be a 'dataset' created, only 'jobs' with job level metadata. +When i check here I see that the dataset facets should have it though https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.md +Does anyone know if creating a dataset & sending row counts to OL is out of the box on dbt-ol or if I need to build another script to get that number from my snowflake instance and push it to OL as another step in my process? +Thanks a lot!

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.md | OpenLineage.md>
+
+ <p><strong>OpenLineage Spec</strong> +<strong>Specification</strong></p> + +<p>The specification for OpenLineage is formalized as a JsonSchema <OpenLineage.json|OpenLineage.json>. <br /> +An OpenAPI spec is also provided for HTTP based implementations: <OpenLineage.yml|OpenLineage.yml> <br /> +The documentation is published at: <a href="https://openlineage.github.io/">https://openlineage.github.io/</a> <br /> +It allows extensions to the spec using <code>Custom Facets</code> as described in this document.</p> + +<p><strong>Core concepts</strong> +<strong>Core Lineage Model</strong></p> + +<p><OpenLineageModel.svg|Open Lineage model></p> + +<p>• <strong>Run Event</strong>: an event describing an observed state of a job run. It is required to at least send a START event and a COMPLETE/FAIL/ABORT event. Additional events are optional. +• <strong>Job</strong>: a process definition that consumes and produces datasets (defined as its inputs and outputs). It is <Naming.md#Jobs|identified by a unique name within a namespace> (which is assigned to the scheduler starting the jobs). The <em>Job</em> evolves over time and this change is captured when the job runs. +• <strong>Dataset</strong>: an abstract representation of data. It has a <Naming.md#Datasets|unique name within the datasource namespace> derived from its physical location (for example db.host.database.schema.table). Typically, a <em>Dataset</em> changes when a job writing to it completes. Similarly to the <em>Job</em> and <em>Run</em> distinction, metadata that is more static from run to run is captured in a DatasetFacet (for example, the schema that does not change every run), what changes every <em>Run</em> is captured as an <em>InputFacet</em> or an <em>OutputFacet</em> (for example, what subset of the data set was read or written, like a time partition). +• <strong>Run</strong>: An instance of a running job with a start and completion (or failure) time. A run is identified by a globally unique ID relative to its job definition. A run ID <strong>must</strong> be an <a href="https://datatracker.ietf.org/doc/html/rfc4122">UUID</a>. +• <strong>Facet</strong>: A piece of metadata attached to one of the entities defined above.</p> + +<p>example: <br /> +Here is an example of a simple start run event not adding any facet information:</p> + +<pre><code>{ + "eventType": "START", + "eventTime": "2020-12-09T23:37:31.081Z", + "run": { + "runId": "3b452093-782c-4ef2-9c0c-aafe2aa6f34d", + }, + "job": { + "namespace": "my-scheduler-namespace", + "name": "myjob.mytask", + }, + "inputs": [ + { + "namespace": "my-datasource-namespace", + "name": "instance.schema.table", + } + ], + "outputs": [ + { + "namespace": "my-datasource-namespace", + "name": "instance.schema.output_table", + } + ], + "producer": "&lt;https://github.com/OpenLineage/OpenLineage/blob/v1-0-0/client&gt;", + "schemaURL": "&lt;https://openlineage.io/spec/1-0-0/OpenLineage.json#/definitions/RunEvent&gt;" +} +</code></pre> + +<p><strong>Lifecycle</strong></p> + +<p>The OpenLineage API defines events to capture the lifecycle of a <em>Run</em> for a given <em>Job</em>. <br /> +When a <em>job</em> is being <em>run</em>, we capture metadata by sending run events when the state of the job transitions to a different state. <br /> +We might observe different aspects of the job run at different stages. This means that different metadata might be collected in each event during the lifecycle of a run. <br /> +All metadata is additive. for example, if more inputs or outputs are detected as the job is running we might send additional events specifically for those datasets without re-emitting previously observed inputs or outputs. <br /> +Example:</p> + +<p>• When the run starts, we collect the following Metadata: + • Run Id + • Job id + • eventType: START + • event time + • source location and version (ex: git sha) + • If known: Job inputs and outputs. (input schema, ...) +• When the run completes: + • Run Id + • Job id + • eventType: COMPLETE + • event time + • Output datasets schema (and other metadata).</p> + +<p><strong>Facets</strong></p> + +<p>Facets are pieces of metadata that can be attached to the core entities:</p> + +<p>• Run +• Job +• Dataset (Inputs or Outputs)</p> + +<p>A facet is an atomic piece of metadata identified by its name. This means that emitting a new facet with the same name for the same entity replaces the previous facet instance for that entity entirely). It is defined as a JSON object that can be either part of the spec or custom facets defined in a different project.</p> + +<p>Custom facets must use a distinct prefix named after the project defining them to avoid collision with standard facets defined in the <OpenLineage.json|OpenLineage.json> spec. <br /> +They have a _schemaURL field pointing to the corresponding version of the facet schema (as a JSONPointer: <a href="https://swagger.io/docs/specification/using-ref/">$ref URL location</a> ).</p> + +<p>Example: <a href="https://raw.githubusercontent.com/OpenLineage/OpenLineage/main/spec/OpenLineage.json#/definitions/MyCustomJobFacet">https://raw.githubusercontent.com/OpenLineage/OpenLineage/main/spec/OpenLineage.json#/definitions/MyCustomJobFacet</a></p> + +<p>The versioned URL must be an immutable pointer to the version of the facet schema. For example, it should include a tag of a git sha and not a branch name. This should also be a canonical URL. There should be only one URL used for a given version of a schema.</p> + +<p>Custom facets can be promoted to the standard by including them in the spec.</p> + +<p><strong>Custom Facet Naming</strong></p> + +<p>Naming of custom facets should follow pattern <code>{prefix}{name}{entity}Facet</code> PascalCased. <br /> +Prefix must be distinct identifier named after the project defining them to avoid colision with standard facets defined in the <OpenLineage.json|OpenLineage.json> spec. <br /> +Entity is the core entity for which the facet is attached.</p> + +<p>When attached to core entity, the key should follow pattern <code>{prefix}_{name}</code>, where both prefix and name follow snakeCase pattern.</p> + +<p>Example of valid name is <code>BigQueryStatisticsJobFacet</code> and it's key <code>bigQuery_statistics</code>.</p> + +<p><strong>Standard Facets</strong> +<strong>Run Facets</strong></p> + +<p>• <strong>nominalTime</strong>: Captures the time this run is scheduled for. This is a typical usage for time based scheduled job. The job has a nominal schedule time that will be different from the actual time it is running at. +• <strong>parent</strong>: Captures the parent job and Run when the run was spawn from a parent run. For example in the case of Airflow, there's a run for the DAG that then spawns runs for individual tasks that would refer to the parent run as the DAG run. Similarly when a SparkOperator starts a Spark job, this creates a separate run that refers to the task run as its parent. +• <strong>errorMessage</strong>: Captures potential error message, programming language - and optionally stack trace - with which the run failed.</p> + +<p><strong>Job Facets</strong></p> + +<p>• <strong>sourceCodeLocation</strong>: Captures the source code location and version (example: git sha) of the job. +• <strong>sourceCode</strong>: Captures language (ex. python) and actual source code of the job. +• <strong>sql</strong>: Capture the SQL query if this job is a SQL query. +• <strong>ownership</strong>: Captures the owners of the job</p> + +<p><strong>Dataset Facets</strong></p> + +<p>• <strong>schema</strong>: Captures the schema of the dataset +• <strong>dataSource</strong>: Captures the Database instance containing this datasets (ex: Database schema. Object store bucket, ...) +• <strong>lifecycleStateChange</strong>: Captures the lifecycle states of the dataset like: alter, create, drop, overwrite, rename, truncate. +• <strong>version</strong>: Captures the dataset version when versioning is defined by database (ex. Iceberg snapshot ID) +• <facets/ColumnLineageDatasetFacet.md|<strong>columnLineage</strong>&gt;: Captures the column-level lineage +• <strong>ownership</strong>: Captures the owners of the dataset</p> + +<p><strong>Input Dataset Facets</strong></p> + +<p>• <strong>dataQualityMetrics</strong>: Captures dataset level and column level data quality metrics when scanning a dataset whith a DataQuality library (row count, byte size, null count, distinct count, average, min, max, quantiles). +• <strong>dataQualityAssertions</strong>: Captures the result of running data tests on dataset or its columns.</p> + +<p><strong>Output Dataset Facets</strong></p> + +<p>• <strong>outputStatistics</strong>: Captures the size of the output written to a dataset (row count and byte size).</p> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Viraj Parekh + (vmpvmp94@gmail.com) +
+
2023-01-03 13:20:14
 +
+

*Thread Reply:* @Ross Turk maybe you can help with this?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2023-01-03 13:34:23
 +
+

*Thread Reply:* hmm, I believe the dbt-ol integration does capture bytes/rows, but only for some data sources: https://github.com/OpenLineage/OpenLineage/blob/6ae1fd5665d5fd539b05d044f9b6fb831ce9d475/integration/common/openlineage/common/provider/dbt.py#L567

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/6ae1fd5665d5fd539b05d044f9b6fb831ce9d475/integration/common/openlineage/common/provider/dbt.py | dbt.py>
+
+ <pre><code> def node_to_output_dataset( +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2023-01-03 13:34:58
 +
+

*Thread Reply:* I haven't personally tried it with Snowflake in a few versions, but the code suggests that it's one of them.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2023-01-03 13:35:42
 +
+

*Thread Reply:* @Max you say your dbt-ol run is resulting in only jobs and no datasets emitted, is that correct?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2023-01-03 13:38:06
 +
+

*Thread Reply:* if so, I'd say something rather strange is going on because in my experience each model should result in a Job and a Dataset.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kuldeep + (kuldeep.marathe@affirm.com) +
+
2023-01-03 00:41:09
 +
+

Hi All, Curious to see if there is an openlineage integration with luigi or any open source projects working on it.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kuldeep + (kuldeep.marathe@affirm.com) +
+
2023-01-03 01:53:10
 +
+

*Thread Reply:* I was looking for something similar to the airflow integration

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Viraj Parekh + (vmpvmp94@gmail.com) +
+
2023-01-03 13:21:18
 +
+

*Thread Reply:* hey @Kuldeep - i don't think there's something for Luigi right now - is that something you'd potentially be interested in?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kuldeep + (kuldeep.marathe@affirm.com) +
+
2023-01-03 13:23:53
 +
+

*Thread Reply:* @Viraj Parekh Yes this is something we are interested in! There are a lot of projects out there that use luigi

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-01-03 11:05:48
 +
+

Hello all, I’m opening a vote to release OpenLineage 0.19.0, including: +• new extractors for Trino and S3FileTransformOperator in the Airflow integration +• a new, standardized run facet in the Airflow integration +• a new NominalTimeRunFacet and OwnershipJobFacet in the Airflow integration +• Postgres support in the dbt integration +• a new client-side proxy (skeletal version) +• a new, improved mechanism for passing conf parameters to the OpenLineage client in the Spark integration +• a new ExtractionErrorRunFacet to reflect internal processing errors for the SQL parser +• testing improvements, bug fixes and more. +As always, three +1s from committers will authorize an immediate release. Thanks in advance!

+ + + +
+ ➕ Willy Lulciuc, Maciej Obuchowski, Paweł Leszczyński, Jakub Dardziński, Julien Le Dem +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-01-03 23:07:59
 +
+

*Thread Reply:* Hi @Michael Robinson a new, improved mechanism for passing conf parameters to the OpenLineage client in the Spark integration +Would it be possible to have more details on what this entails please? Thanks!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-01-04 09:21:46
 +
+

*Thread Reply:* @Tomasz Nazarewicz might explain this better

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tomasz Nazarewicz + (tomasz.nazarewicz@getindata.com) +
+
2023-01-04 10:04:22
 +
+

*Thread Reply:* @Anirudh Shrinivason until now If you wanted to add new property to OL client, you had to also implement it in the integration because it had to parse all properties, create appropriate objects etc. New implementation makes client properties transparent to integration, they are only passed through and parsing happens inside the client.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-01-04 13:02:39
 +
+

*Thread Reply:* Thanks, all. The release is authorized and will commence shortly 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-01-04 22:00:55
 +
+

*Thread Reply:* @Tomasz Nazarewicz Ahh I see. Okay thanks!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-01-05 10:37:09
 +
+

@channel +This month’s OpenLineage TSC meeting is next Thursday, January 12th, at 10 am PT. Join us on Zoom: https://bit.ly/OLzoom. All are welcome! +On the tentative agenda:

+ +
  1. Recent release overview @Michael Robinson
  2. Column lineage update @Maciej Obuchowski
  3. Airflow integration improvements @Jakub Dardziński
  4. Discussions: +• Real-world implementation of OpenLineage (What does it really mean?) @Sheeri Cabral (Collibra) +• Using namespaces @Michael Robinson
  5. Open discussion +Notes: https://bit.ly/OLwiki +Is there a topic you think the community should discuss at this or a future meeting? Reply or DM me to add items to the agenda.
    6. +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-01-05 23:45:38
 +
+

*Thread Reply:* @Michael Robinson Will there be a recording?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-01-06 09:10:50
 +
+

*Thread Reply:* @Anirudh Shrinivason Yes, and the recording will be here: https://wiki.lfaidata.foundation/display/OpenLineage/Monthly+TSC+meeting

+ + + +
+ :gratitude_thank_you: Anirudh Shrinivason +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-01-05 13:00:01
 +
+

OpenLineage 0.19.2 is available now, including: +• Airflow: add Trino extractor #1288 @sekikn +• Airflow: add S3FileTransformOperator extractor #1450 @sekikn +• Airflow: add standardized run facet #1413 @JDarDagran +• Airflow: add NominalTimeRunFacet and OwnershipJobFacet #1410 @JDarDagran +• dbt: add support for postgres datasources #1417 @julienledem +• Proxy: add client-side proxy (skeletal version) #1439 #1420 @fm100 +• Proxy: add CI job to publish Docker image #1086 @wslulciuc +• SQL: add ExtractionErrorRunFacet #1442 @mobuchowski +• SQL: add column-level lineage to SQL parser #1432 #1461 @mobuchowski @StarostaGit +• Spark: pass config parameters to the OL client #1383 @tnazarew +• Plus bug fixes and testing and CI improvements. +Thanks to all the contributors, including new contributor Saurabh (@versaurabh) +Release: https://github.com/OpenLineage/OpenLineage/releases/tag/0.19.2 +Changelog: https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md +Commit history: https://github.com/OpenLineage/OpenLineage/compare/0.18.0...0.19.2 +Maven: https://oss.sonatype.org/#nexus-search;quick~openlineage +PyPI: https://pypi.org/project/openlineage-python/

+ + + +
+ ❤️ Julien Le Dem, Howard Yoo, Willy Lulciuc, Maciej Obuchowski, Kengo Seki, Harel Shein, Jarek Potiuk, Varun Singh +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2023-01-06 01:07:18
 +
+

Question on Spark Integration and External Hive Metastores

+ +

@Hanna Moazam and I are working with a team using OpenLineage and wants to extract out the server name of the hive metastore they're using when writing to a Hive table through Spark.

+ +

For example, the hive metastore is an Azure SQL database and the table name is sales.transactions.

+ +

OpenLineage will give something like /usr/hive/warehouse/sales.db/transactions for the name.

+ +

However, this is not a complete picture since sales.db/transactions is defined like this for a given hive metastore. In Hive, you'd define the fully qualified name as sales.transactions@sqlservername.database.windows.net .

+ +

Has anyone else come across this before? If not, we plan on raising an issue and suggesting we extract out the spark.hadoop.javax.jdo.option.ConnectionURL in the DatabricksEnvironmentFacetBuilder but ideally there would be a better way of extracting this.

+ +

https://learn.microsoft.com/en-us/azure/databricks/data/metastores/external-hive-metastore#set-up-an-external-metastore-using-the-ui

+ +

There was an issue by @Maciej Obuchowski or @Paweł Leszczyński that talked about providing a facet of the alias of a path but I can't find it at this point :(

+
+
learn.microsoft.com
+ + + +
External Apache Hive metastore - Azure Databricks
+
+ Learn how to connect to external Apache Hive metastores in Azure Databricks. +
+ + + + + + + + +
Original URL: https://learn.microsoft.com/en-us/azure/databricks/data/metastores/external-hive-metastore#set-up-an-external-metastore-using-the-ui
+ + + +
+ + + +
+ 👀 Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-01-09 02:28:43
 +
+

*Thread Reply:* Hi @Hanna Moazam, we've written Jupyter notebook to demo dataset symlinks feature: +https://github.com/OpenLineage/workshops/blob/main/spark/dataset_symlinks.ipynb

+ +

For scenario you describe, there should be symlink facet sent similar to: +{ + "_producer": "<https://github.com/OpenLineage/OpenLineage/tree/0.15.1/integration/spark>", + "_schemaURL": "<https://openlineage.io/spec/facets/1-0-0/SymlinksDatasetFacet.json#/$defs/SymlinksDatasetFacet>", + "identifiers": [ + { + "namespace": "<hive://metastore>", + "name": "default.some_table", + "type": "TABLE" + } + ] +} +Within Openlineage Spark integration code, symlinks are included here: +https://github.com/OpenLineage/OpenLineage/blob/0.19.2/integration/spark/shared/src/main/java/io/openlineage/spark/agent/util/PathUtils.java#L75

+ +

and they are added only when spark catalog is hive and metastore URI in spark conf is present.

+
+ + + + +
<https://github.com/OpenLineage/workshops/blob/main/spark/dataset_symlinks.ipynb | dataset_symlinks.ipynb>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/workshops">OpenLineage/workshops</a> +
+ + +
+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/0.19.2/integration/spark/shared/src/main/java/io/openlineage/spark/agent/util/PathUtils.java | PathUtils.java>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+ ➕ Maciej Obuchowski +
+ +
+ 🤯 Will Johnson +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2023-01-09 14:21:10
 +
+

*Thread Reply:* This is so awesome, @Paweł Leszczyński - Thank you so much for sharing this! I'm wondering if we could extend this to capture the hive JDBC Connection URL. I will explore this and put in an issue and PR to try and extend it. Thank you for the insights!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-01-11 12:00:02
 +
+

@channel +Friendly reminder: this month’s OpenLineage TSC meeting is tomorrow at 10am, and all are welcome. https://openlineage.slack.com/archives/C01CK9T7HKR/p1672933029317449

+
+ + +
+ + + } + + Michael Robinson + (https://openlineage.slack.com/team/U02LXF3HUN7) +
+ + + +
+
+ <p><strong>@channel</strong> +This month’s OpenLineage TSC meeting is <strong>next Thursday, January 12th, at 10 am PT</strong>. Join us on Zoom: <a href="https://bit.ly/OLzoom">https://bit.ly/OLzoom</a>. All are welcome! +On the tentative agenda:</p> + +<ol><li>Recent release overview @Michael Robinson </li><li>Column lineage update @Maciej Obuchowski </li><li>Airflow integration improvements @Jakub Dardziński </li><li>Discussions: +• Real-world implementation of OpenLineage (What does it really mean?) @Sheeri Cabral (Collibra) +• Using namespaces @Michael Robinson </li><li>Open discussion +Notes: <a href="https://bit.ly/OLwiki">https://bit.ly/OLwiki</a> +Is there a topic you think the community should discuss at this or a future meeting? Reply or DM me to add items to the agenda.</li> +</ol> +
+ + + + + + +
Original URL: https://openlineage.slack.com/archives/C01CK9T7HKR/p1672933029317449
+ + +
+ + Thread in Slack Conversation +
+ + +
+ + + +
+ 🙌 Maciej Obuchowski, Will Johnson, John Bagnall, AnnMary Justine, Willy Lulciuc, Minkyu Park, Paweł Leszczyński, Varun Singh +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Varun Singh + (varuntestaz@outlook.com) +
+
2023-01-12 06:37:56
 +
+

Hi, are there any plans to add an Azure EventHub transport similar to the Kinesis one?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2023-01-12 17:31:12
 +
+

*Thread Reply:* @Varun Singh why not just use the KafkaTransport and the Event Hub's Kafka endpoint?

+ +

https://github.com/yogyang/OpenLineage/blob/2b7fa8bbd19a2207d54756e79aea7a542bf7bb[…]/main/java/io/openlineage/client/transports/KafkaTransport.java

+ +

https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-kafka-stream-analytics

+
+
learn.microsoft.com
+ + + +
Azure Event Hubs - Process Apache Kafka events - Azure Event Hubs
+
+ Tutorial: This article shows how to process Kafka events that are ingested through event hubs by using Azure Stream Analytics +
+ + + + + + + + +
Original URL: https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-kafka-stream-analytics
+ + + +
+
+ + + + +
<https://github.com/yogyang/OpenLineage/blob/2b7fa8bbd19a2207d54756e79aea7a542bf7bb4c/client/java/src/main/java/io/openlineage/client/transports/KafkaTransport.java | KafkaTransport.java>
+
+ <pre><code>/** +/** Copyright 2018-2022 contributors to the OpenLineage project +/** SPDX-License-Identifier: Apache-2.0 +**/ + +package io.openlineage.client.transports; + +import io.openlineage.client.OpenLineage; +import io.openlineage.client.OpenLineageClientUtils; +import lombok.NonNull; +import lombok.extern.slf4j.Slf4j; +import org.apache.kafka.clients.producer.KafkaProducer; +import org.apache.kafka.clients.producer.ProducerRecord; + +@Slf4j +public final class KafkaTransport extends Transport { + private final String topicName; + private final String localServerId; + private final KafkaProducer&lt;String, String&gt; producer; + + public KafkaTransport(@NonNull final KafkaConfig kafkaConfig) { + this(new KafkaProducer&lt;&gt;(kafkaConfig.getProperties()), kafkaConfig); + } + + public KafkaTransport( + @NonNull final KafkaProducer&lt;String, String&gt; kafkaProducer, + @NonNull final KafkaConfig kafkaConfig) { + super(Type.KAFKA); + this.topicName = kafkaConfig.getTopicName(); + this.localServerId = kafkaConfig.getLocalServerId(); + this.producer = kafkaProducer; + } + + @Override + public void emit(@NonNull OpenLineage.RunEvent runEvent) { + final String eventAsJson = OpenLineageClientUtils.toJson(runEvent); + log.debug("Received lineage event: {}", eventAsJson); + final ProducerRecord&lt;String, String&gt; record = + new ProducerRecord&lt;&gt;(topicName, localServerId, eventAsJson); + try { + producer.send(record); + } catch (Exception e) { + log.error("Failed to collect lineage event: {}", eventAsJson, e); + } + } +} + +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/yogyang/OpenLineage">yogyang/OpenLineage</a> +
+ + +
+ + + +
+ 👍 Varun Singh +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2023-01-12 09:01:24
 +
+

Following up on last month’s discussion (), I created the <#C04JPTTC876|spec-compliance> channel for further discussion

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2023-01-12 17:43:55
 +
+

*Thread Reply:* @Julien Le Dem is there a channel to discuss the community call / ask follow-up questions on the communiyt call topics? For example, I wanted to ask more about the AirflowFacet and if we expected to introduce more tool specific facets into the spec. Where's the right place to ask that question? On the PR?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2023-01-17 15:11:05
 +
+

*Thread Reply:* I think asking in #general is the right place. If there’s a specific github issue/PR, his is a good place as well. You can tag the relevant folks as well to get their attention

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Allison Suarez + (asuarezmiranda@lyft.com) +
+
2023-01-12 18:37:24
 +
+

@here I am using the Spark listener and whenever a query like INSERT OVERWRITE TABLE gets executed it looks like I can see some outputs, but there are no symlinks for the output table. The operation type being executed is InsertIntoHadoopFsRelationCommand . I am not sure why I cna see symlinks for all the input tables but not the output tables. Anyone know the reason behind this?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-01-13 02:30:37
 +
+

*Thread Reply:* Hello @Allison Suarez, in case of InsertIntoHadoopFsRelationCommand, Spark Openlineage implementation uses method: +DatasetIdentifier di = PathUtils.fromURI(command.outputPath().toUri(), "file"); +(https://github.com/OpenLineage/OpenLineage/blob/0.19.2/integration/spark/shared/sr[…]ark/agent/lifecycle/plan/InsertIntoHadoopFsRelationVisitor.java)

+ +

If the dataset identifier is constructed from a path, then no symlinks are added. That's the current behaviour.

+ +

Calling io.openlineage.spark.agent.util.DatasetIdentifier#withSymlink(io.openlineage.spark.agent.util.DatasetIdentifier.Symlink) on DatasretIdentifier in InsertIntoHadoopFsRelationVisitor +could be a remedy to that.

+ +

Do you have some Spark code snippet to reproduce this issue?

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/0.19.2/integration/spark/shared/src/main/java/io/openlineage/spark/agent/lifecycle/plan/InsertIntoHadoopFsRelationVisitor.java | InsertIntoHadoopFsRelationVisitor.java>
+
+ <pre><code> DatasetIdentifier di = PathUtils.fromURI(command.outputPath().toUri(), "file"); +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2023-01-22 10:04:56
 +
+

*Thread Reply:* @Allison Suarez it would also be good to know what compute engine you're using to run your code on? On-Prem Apache Spark? Azure/AWS/GCP Databricks?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Allison Suarez + (asuarezmiranda@lyft.com) +
+
2023-02-13 18:18:52
 +
+

*Thread Reply:* I created a custom visitor and fixed the issue that way, thank you!

+ + + +
+ 🙌 Will Johnson +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Varun Singh + (varuntestaz@outlook.com) +
+
2023-01-13 11:44:19
 +
+

Hi, I am trying to use kafka transport in spark for sending events to an EventHub but it requires me to set a property sasl.jaas.config which needs to have semicolons (;) in its value. But this gives an error about being unable to convert Array to a String. I think this is due to this line which splits property values into an array if they have a semicolon https://github.com/OpenLineage/OpenLineage/blob/92adbc877f0f4008928a420a1b8a93f394[…]pp/src/main/java/io/openlineage/spark/agent/ArgumentParser.java +Does this seem like a bug or is it intentional?

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/92adbc877f0f4008928a420a1b8a93f394530d04/integration/spark/app/src/main/java/io/openlineage/spark/agent/ArgumentParser.java | ArgumentParser.java>
+
+ <pre><code> Arrays.stream(value.split(DISABLED_FACETS_SEPARATOR)) +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harel Shein + (harel.shein@gmail.com) +
+
2023-01-13 14:39:51
 +
+

*Thread Reply:* seems like a bug to me, but tagging @Tomasz Nazarewicz / @Paweł Leszczyński

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tomasz Nazarewicz + (tomasz.nazarewicz@getindata.com) +
+
2023-01-13 15:22:19
 +
+

*Thread Reply:* So we needed a generic way of passing parameters to client and made an assumption that every field with ; will be treated as an array

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Varun Singh + (varuntestaz@outlook.com) +
+
2023-01-14 02:00:04
 +
+

*Thread Reply:* Thanks for the confirmation, should I add a condition to split only if it's a key that can have array values? We can have a list of such keys like facets.disabled

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tomasz Nazarewicz + (tomasz.nazarewicz@getindata.com) +
+
2023-01-14 02:28:41
 +
+

*Thread Reply:* We thought about this solution but it forces us to know the structure of each config and we wanted to avoid that as much as possible

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tomasz Nazarewicz + (tomasz.nazarewicz@getindata.com) +
+
2023-01-14 02:34:06
 +
+

*Thread Reply:* Maybe the condition could be having ; and [] in the value

+ + + +
+ 👍 Varun Singh +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Varun Singh + (varuntestaz@outlook.com) +
+
2023-01-15 08:14:14
 +
+

*Thread Reply:* Makes sense, I can add this check. Thanks @Tomasz Nazarewicz!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Varun Singh + (varuntestaz@outlook.com) +
+
2023-01-16 01:15:19
 +
+

*Thread Reply:* Created issue https://github.com/OpenLineage/OpenLineage/issues/1506 for this

+
+ + + + +
#1506 Spark configs with semicolons (;) in their value get split into an array
+
+ <p>Semicolon (;) is being used as the delimiter for array configs like <code>disabled.facets</code> but this also converts string configs which require a semicolon into arrays. For example, EventHub authentication requires a SAS key which has a semicolon in it but it doesn't need to be converted into an array.</p> + +<p>This conversion is being done in the following lines </p> + +<p><a href="https://github.com/OpenLineage/OpenLineage/blob/92adbc877f0f4008928a420a1b8a93f394530d04/integration/spark/app/src/main/java/io/openlineage/spark/agent/ArgumentParser.java#L150-L154">OpenLineage/integration/spark/app/src/main/java/io/openlineage/spark/agent/ArgumentParser.java</a></p> + +<p>Lines 150 to 154 in </OpenLineage/OpenLineage/commit/92adbc877f0f4008928a420a1b8a93f394530d04|92adbc8></p> +
+ +
+
Comments
+ 2 +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-01-17 12:00:02
 +
+

Hi everyone, I’m excited to share some good news about our progress in the LFAI & Data Foundation: we’ve achieved Incubation status! This required us to earn a Silver Badge from the OpenSSF, get 300+ stars on GitHub (which was NBD as we have over 1100 already), and win the approval of the LFAI & Data’s TAC. Now that we’ve cleared this hurdle, we have access to additional services from the foundation, including assistance with creative work, marketing and communication support, and event-planning assistance. Graduation from the program, which will earn us a voting seat on the TAC, is on the horizon. Stay tuned for updates on our progress with the foundation.

+ +

LF AI & Data is an umbrella foundation of the Linux Foundation that supports open source innovation in artificial intelligence (AI) and data. LF AI & Data was created to support open source AI and data, and to create a sustainable open source AI and data ecosystem that makes it easy to create AI and data products and services using open source technologies. They foster collaboration under a neutral environment with an open governance in support of the harmonization and acceleration of open source technical projects.

+ +

For more info about the foundation and other LFAI & Data projects, visit their website.

+ + + +
+ ❤️ Julien Le Dem, Paweł Leszczyński, Maciej Obuchowski, Ross Turk, Jakub Dardziński, Minkyu Park, Howard Yoo, Jarek Potiuk, Danilo Mota, Willy Lulciuc, Kengo Seki, Harel Shein +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2023-01-17 15:53:12
 +
+

if you want to share this news (and I hope you do!) there is a blog post here: https://openlineage.io/blog/incubation-stage-lfai/

+
+
openlineage.io
+ + + +
OpenLineage Advances to Incubation Stage with the LFAI &amp; Data
+
+ OpenLineage has achieved Incubation status with the LFAI &amp; Data. +
+ + + + + + +
Original URL: https://openlineage.io/blog/incubation-stage-lfai/
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2023-01-17 15:54:07
 +
+

and I'll add a quick shoutout of @Michael Robinson, who has done a whole lot of work to make this happen 🎉 thanks, man, you're awesome!

+ + + +
+ 🙌 Howard Yoo, Maciej Obuchowski, Jarek Potiuk, Minkyu Park, Willy Lulciuc, Kengo Seki, Paweł Leszczyński, Varun Singh +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-01-17 15:56:38
 +
+

*Thread Reply:* Thank you, Ross!! I appreciate it. I might have coordinated it, but it’s been a team effort. Lots of folks shared knowledge and time to help us check all the boxes, literally and figuratively (lots of boxes). ;)

+ + + +
+ ☑️ Willy Lulciuc, Paweł Leszczyński, Viraj Parekh +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jarek Potiuk + (jarek@potiuk.com) +
+
2023-01-17 16:03:36
 +
+

Congrats @Michael Robinson and @Ross Turk - > major step for Open Lineage!

+ + + +
+ 🙌 Michael Robinson, Maciej Obuchowski, Jakub Dardziński, Julien Le Dem, Ross Turk, Willy Lulciuc, Kengo Seki, Viraj Parekh, Paweł Leszczyński, Anirudh Shrinivason, Robert +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sudhir Nune + (sudhir.nune@kraftheinz.com) +
+
2023-01-18 11:15:02
 +
+

Hi all, I am new to the https://openlineage.io/integration/dbt/, I followed the steps on Windows Laptop. But the dbt-ol does not get executed.

+ +

'dbt-ol' is not recognized as an internal or external command, +operable program or batch file.

+ +

I see the following Packages installed too +openlineage-dbt==0.19.2 +openlineage-integration-common==0.19.2 +openlineage-python==0.19.2

+
+
openlineage.io
+ + + +
dbt
+
+ Enabling OpenLineage in dbt can capture lineage metadata for transformations running within your data warehouse. +
+ + + + + + +
Original URL: https://openlineage.io/integration/dbt/
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-01-18 11:17:14
 +
+

*Thread Reply:* What are the errors?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sudhir Nune + (sudhir.nune@kraftheinz.com) +
+
2023-01-18 11:18:09
 +
+

*Thread Reply:* 'dbt-ol' is not recognized as an internal or external command, +operable program or batch file.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-01-19 11:11:09
 +
+

*Thread Reply:* Hm, I think this is due to different windows conventions around scripts.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2023-01-19 14:26:35
 +
+

*Thread Reply:* I have not tried it on Windows before myself, but on mac/linux if you make a Python virtual environment in venv/ and run pip install openlineage-dbt, the script winds up in ./venv/bin/dbt-ol.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2023-01-19 14:27:04
 +
+

*Thread Reply:* (maybe that helps!)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-01-19 14:38:23
 +
+

*Thread Reply:* This might not work, but I think I have an idea that would allow it to run as python -m dbt-ol run ...

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-01-19 14:38:27
 +
+

*Thread Reply:* That needs one fix though

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sudhir Nune + (sudhir.nune@kraftheinz.com) +
+
2023-01-19 14:40:52
 +
+

*Thread Reply:* Hi @Maciej Obuchowski, thanks for the input, when I try to use python -m dbt-ol run, I see the below error :( +\python.exe: No module named dbt-ol

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-01-24 13:23:56
 +
+

*Thread Reply:* We’re seeing a similar issue with the Great Expectations integration at the moment. This is purely a guess, but what happens when you try with openlineage-dbt 0.18.0?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-01-24 13:24:36
 +
+

*Thread Reply:* @Michael Robinson GE issue is on Windows?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-01-24 13:24:49
 +
+

*Thread Reply:* No, not Windows

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-01-24 13:24:55
 +
+

*Thread Reply:* (that I know of)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sudhir Nune + (sudhir.nune@kraftheinz.com) +
+
2023-01-24 13:46:39
 +
+

*Thread Reply:* @Michael Robinson - I see the same error. I used 2 Combinations

+ +
  1. Python 3.8.10 with openlineage-dbt 0.18.0 & Latest
  2. Python 3.9.7 with openlineage-dbt 0.18.0 & Latest
    3. +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2023-01-24 13:49:19
 +
+

*Thread Reply:* Hm. You should be able to find the dbt-ol command wherever pip is installing the packages. In my case, that's usually in a virtual environment.

+ +

But if I am not in a virtual environment, it installs the packages in my PYTHONPATH. You might try this to see if the dbt-ol script can be found in one of the directories in sys.path.

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2023-01-24 13:58:38
 +
+

*Thread Reply:* this can help you verify that your PYTHONPATH and PATH are correct - installing an unrelated python command-line tool and seeing if you can execute it:

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-01-24 13:59:42
 +
+

*Thread Reply:* Again, I think this is windows issue

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2023-01-24 14:00:54
 +
+

*Thread Reply:* @Maciej Obuchowski you think even if dbt-ol could be found in the path, that might not be the issue?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sudhir Nune + (sudhir.nune@kraftheinz.com) +
+
2023-01-24 14:15:13
 +
+

*Thread Reply:* Hi @Ross Turk - I could not find the dbt-ol in the site-packages.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2023-01-24 14:16:48
 +
+

*Thread Reply:* Hm 😕 then perhaps @Maciej Obuchowski is right and there is a bigger issue here

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sudhir Nune + (sudhir.nune@kraftheinz.com) +
+
2023-01-24 14:31:15
 +
+

*Thread Reply:* @Ross Turk & @Maciej Obuchowski I see the issue event when I do the install using the https://pypi.org/project/openlineage-dbt/#files - openlineage-dbt-0.19.2.tar.gz.

+ +

For some reason, I see only the following folder created

+ +
  1. openlineage
  2. openlineage_dbt-0.19.2.dist-info
  3. openlineageintegrationcommon-0.19.2.dist-info
  4. openlineage_python-0.19.2.dist-info +and not brining in the openlineage-dbt-0.19.2, which has the scripts/dbt-ol
    5. +
+ +

If it helps I am using pip 21.2.4

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Francis McGregor-Macdonald + (francis@mc-mac.com) +
+
2023-01-18 18:40:32
 +
+

@Paul Villena @Stephen Said and Vishwanatha Nayak published an AWS blog Automate data lineage on Amazon MWAA with OpenLineage

+
+
Amazon Web Services
+ + + +
Automate data lineage on Amazon MWAA with OpenLineage | Amazon Web Services
+
+ In modern data architectures, datasets are combined across an organization using a variety of purpose-built services to unlock insights. As a result, data governance becomes a key component for data consumers and producers to know that their data-driven decisions are based on trusted and accurate datasets. One aspect of data governance is data lineage, which […] +
+ + + + + + + + +
Original URL: https://aws.amazon.com/blogs/big-data/automate-data-lineage-on-amazon-mwaa-with-openlineage/
+ + + +
+ + + +
+ 👀 Ross Turk, Peter Hicks, Willy Lulciuc +
+ +
+ 🔥 Ross Turk, Willy Lulciuc, Michael Collado, Peter Hicks, Minkyu Park, Julien Le Dem, Kengo Seki, Anirudh Shrinivason, Paweł Leszczyński, Maciej Obuchowski, Harel Shein, Paul Wilson Villena +
+ +
+ ❤️ Willy Lulciuc, Minkyu Park, Julien Le Dem, Kengo Seki, Paweł Leszczyński, Viraj Parekh +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2023-01-18 18:54:57
 +
+

*Thread Reply:* This is excellent! May we promote it on openlineage and marquez social channels?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2023-01-18 18:55:30
 +
+

*Thread Reply:* This is an amazing write up! 🔥 💯 🚀

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Francis McGregor-Macdonald + (francis@mc-mac.com) +
+
2023-01-18 19:49:46
 +
+

*Thread Reply:* Happy to have it promoted. 😄 +Vish posted on LinkedIn: https://www.linkedin.com/posts/vishwanatha-nayak-b8462054automate-data-lineage-on-amazon-mwaa-with-activity-7021589819763945473-yMHF?utmsource=share&utmmedium=memberios|https://www.linkedin.com/posts/vishwanatha-nayak-b8462054automate-data-lineage-on-amazon-mwaa-with-activity-7021589819763945473-yMHF?utmsource=share&utmmedium=memberios if you want something to repost there.

+
+
linkedin.com
+ + + +
Vishwanatha Nayak on LinkedIn: Automate data lineage on Amazon MWAA with OpenLineage | Amazon Web Services
+
+ We (Stephen Said, Paul Villena and I) are happy to start the year with a new blog on automating data lineage using Amazon Managed Workflows for Apache Airflow… +
+ + + + + + + + +
Original URL: https://www.linkedin.com/posts/vishwanatha-nayak-b8462054_automate-data-lineage-on-amazon-mwaa-with-activity-7021589819763945473-yMHF?utm_source=share&amp;utm_medium=member_ios
+ + + +
+ + + +
+ ❤️ Willy Lulciuc, Ross Turk +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-01-19 00:13:26
 +
+

Hi guys, I am trying to build the openlineage jar locally for spark. I ran ./gradlew shadowJar in the /integration/spark directory. However, I am getting this issue: +** What went wrong: +A problem occurred evaluating project ':app'. +&gt; Could not resolve all files for configuration ':app:spark33'. + &gt; Could not resolve io.openlineage:openlineage_java:0.20.0-SNAPSHOT. + Required by: + project :app &gt; project :shared + &gt; Could not resolve io.openlineage:openlineage_java:0.20.0-SNAPSHOT. + &gt; Unable to load Maven meta-data from <https://astronomer.jfrog.io/artifactory/maven-public-libs-snapshot/io/openlineage/openlineage-java/0.20.0-SNAPSHOT/maven-metadata.xml>. + &gt; Could not GET '<https://astronomer.jfrog.io/artifactory/maven-public-libs-snapshot/io/openlineage/openlineage-java/0.20.0-SNAPSHOT/maven-metadata.xml>'. Received status code 401 from server: Unauthorized +It used to work a few weeks ago...May I ask if anyone would know what the reason might be? Thanks! 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-01-19 03:58:42
 +
+

*Thread Reply:* Hello @Anirudh Shrinivason, you need to build your openlineage-java package first. Possibly you built in some time ao in different version

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-01-19 03:59:28
 +
+

*Thread Reply:* ./gradlew clean build publishToMavenLocal +in /client/java should help.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-01-19 04:34:33
 +
+

*Thread Reply:* Ahh yeap this works thanks! 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sheeri Cabral (Collibra) + (sheeri.cabral@collibra.com) +
+
2023-01-19 09:17:01
 +
+

Are there any resources to explain the differences between lineage with Apache Atlas vs. lineage using OpenLineage? we have discussions with customers and partners, and some of them are looking into which is more “ready for industry”.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-01-19 11:03:39
 +
+

*Thread Reply:* It's been a while since I looked at Atlas, but does it even now supports something else than very Java Apache-adjacent projects like Hive and HBase?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2023-01-19 13:10:11
 +
+

*Thread Reply:* To directly answer your question @Sheeri Cabral (Collibra): I am not aware of any resources currently that explain this 😞 but I would welcome the creation of one & pitch in where possible!

+ + + +
+ ✅ Sheeri Cabral (Collibra) +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sheeri Cabral (Collibra) + (sheeri.cabral@collibra.com) +
+
2023-01-20 17:00:25
 +
+

*Thread Reply:* I don’t know enough about Atlas to make that doc.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Justine Boulant + (justine.boulant@seenovate.com) +
+
2023-01-19 10:43:18
 +
+

Hi everyone, I am currently working on a project and we have some questions to use OpenLineage with Apache Airflow : +• How does it work : ux vs code/script? How can we implement it? a schema of its architecture for example +• What are the visual outputs available? +• Is the lineage done from A to Z? if there are multiple intermediary transformations for example? +• Is the lineage done horizontally across the organization or vertically on different system levels? or both? +• Can we upgrade it to industry-level? +• Does it work with Python and/or R? +• Does it read metadata or scripts? +Thanks a lot if you can help 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-01-19 11:00:54
 +
+

*Thread Reply:* I think most of your questions will be answered by this video: https://www.youtube.com/watch?v=LRr-ja8_Wjs

+
+
YouTube
+ +
+ + + } + + Astronomer + (https://www.youtube.com/@Astronomer) +
+ + + +
OpenLineage and Airflow: A Deeper Dive
+
+ +
+ + + + + + + + +
Original URL: https://www.youtube.com/watch?v=LRr-ja8_Wjs
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2023-01-19 13:10:58
 +
+

*Thread Reply:* I agree - a lot of the answers are in that overview video. You might also take a look at the docs, they do a pretty good job of explaining how it works.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2023-01-19 13:19:34
 +
+

*Thread Reply:* More explicitly: +• Airflow is an interesting platform to observe because it runs a large variety of workloads and lineage can only be automatically extracted for some of them +• In general, OpenLineage is essentially a standard and data model for lineage. There are integrations for various systems, including Airflow, that cause them to emit lineage events to an OpenLineage compatible backend. It's a push model. +• Marquez is one such backend, and the one I recommend for testing & development +• There are a few approaches for lineage in Airflow: + ◦ Extractors, which pair with Operators to extract and emit lineage + ◦ Manual inlets/outlets on a task, defined by a developer - useful for PythonOperator and other cases where an extractor can't do it auto + ◦ Orchestration of an underlying OpenLineage integration, like openlineage-dbt +• IDK about "A to Z", that depends on your environment. The goal is to capture every transformation. Depending on your pipeline, there may be a set of integrations that give you the coverage you need. We often find that there are gaps. +• It works with Python. You can use the openlineage-python client to emit lineage events to a backend. This is useful if there isn't an integration for something your pipeline does. +• It describes the pipeline by observing running jobs and the way they affect datasets, not the organization. I don't know what you mean by "industry-level". +• I am not aware of an integration that parses source code to determine lineage at this time. +• The openlineage-dbt integration consumes the various metadata that dbt leaves behind to construct lineage. Dunno if that's what you mean by "read metadata".

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2023-01-19 13:23:33
 +
+

*Thread Reply:* FWIW I did a workshop on openlineage and airflow a while back, and it's all in this repo. You can find slides + a quick Python example + a simple Airflow example in there.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Justine Boulant + (justine.boulant@seenovate.com) +
+
2023-01-20 03:44:22
 +
+

*Thread Reply:* Thanks a lot!! Very helpful!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2023-01-20 11:42:43
 +
+

*Thread Reply:* 👍

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Brad Paskewitz + (bradford.paskewitz@fivetran.com) +
+
2023-01-20 15:28:06
 +
+

Hey folks, my team is working on a solution that would support the OL standard with column level lineage. I'm working through the architecture now and I'm wondering if everyone uses the standard rest api backed by a db or if other teams found success using other technologies such as webhooks, streams, etc in order to capture and process lineage events. I'd be very curious to connect on the topic

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2023-01-20 19:45:55
 +
+

*Thread Reply:* Hello Brad, on top of my head:

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2023-01-20 19:47:15
 +
+

*Thread Reply:* • Marquez uses the API HTTP Post. so does Astro +• Egeria and Purview prefer consuming through a Kafka topic. There is a ProxyBackend that takes HTTP Posts and writes to Kafka. The client can also be configured to write to Kafka

+ + + +
+ 👍 Jakub Dardziński +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2023-01-20 19:48:09
 +
+

*Thread Reply:* @Will Johnson @Mandy Chessell might have opinions

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2023-01-20 19:49:10
 +
+

*Thread Reply:* The Microsoft Purview approach is documented here: https://learn.microsoft.com/en-us/samples/microsoft/purview-adb-lineage-solution-accelerator/azure-databricks-to-purview-lineage-connector/

+
+
learn.microsoft.com
+ + + +
Azure Databricks to Purview Lineage Connector - Code Samples
+
+ This solution accelerator, together with the [OpenLineage](<http://openlineage.io>) project, provides a connector that will transfer lineage metadata fr... +
+ + + + + + + + +
Original URL: https://learn.microsoft.com/en-us/samples/microsoft/purview-adb-lineage-solution-accelerator/azure-databricks-to-purview-lineage-connector/
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2023-01-20 19:49:47
 +
+

*Thread Reply:* There’s a blog post about Egeria here: https://openlineage.io/blog/openlineage-egeria/

+
+
openlineage.io
+ + + +
OpenLineage Support in Egeria
+
+ The Egeria project uses OpenLineage to enhance its production of holistic metadata about an organization's operations. +
+ + + + + + +
Original URL: https://openlineage.io/blog/openlineage-egeria/
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2023-01-22 10:00:56
 +
+

*Thread Reply:* @Brad Paskewitz at Microsoft, the solution that Julien linked above, we are using the HTTP Transport (REST API) as we are consuming the OpenLineage Events and transforming them to Apache Atlas / Microsoft Purview.

+ +

However, there is a good deal of interest in using the kafka transport instead and that's our future roadmap.

+ + + +
+ 👍 Ross Turk, Brad Paskewitz +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Quentin Nambot + (qnambot@gmail.com) +
+
2023-01-25 09:59:13
 +
+

❓ Hi everyone, I am trying to use openlineage with Databricks (using 11.3 LTS runtime, and openlineage 0.19.2) +Using this documentation I managed to install openlineage and send events to marquez +However marquez did not received all COMPLETE events, it seems like databricks cluster is shutdown immediatly at the end of the job. It is not the first time that i see this with databricks, last year I tried to use spline and we noticed that Databricks seems to not wait that spark session is nicely closed before shutting down instances (see this issue) +My question is: has anyone faced the same issue? Does somebody know a workaround? 🙏

+
+
spline
+ + + +
0.7
+
+ Data Lineage Tracking And Visualization Solution +
+ + + + + + +
Original URL: https://absaoss.github.io/spline/
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2023-01-25 12:04:48
 +
+

*Thread Reply:* Hmm, if Databricks is shutting the process down without waiting for the ListenerBus to clear, I don’t know that there’s a lot we can do. The best thing is to somehow delay the main application thread from exiting. One thing you could try is to subclass the OpenLineageSparkListener and generate a lock for each SparkListenerSQLExecutionStart and release it when the accompanying SparkListenerSQLExecutionEnd event is processed. Then, in the main application, block until all such locks are released. If you try it and it works, let us know!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Quentin Nambot + (qnambot@gmail.com) +
+
2023-01-26 05:46:35
 +
+

*Thread Reply:* Ok thanks for the idea! I'll tell you if I try this and if it works 🤞

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Petr Hajek + (petr.hajek@profinit.eu) +
+
2023-01-25 10:12:42
 +
+

Hi, would anybody be able and willing to help us configure S3 and Snowflake extractors within Airflow integration for one of our clients? Our trouble is that Airflow integration returns valid OpenLineage .json files but it lacks any information about input and output DataSets. Thanks in advance 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-01-25 10:38:03
 +
+

*Thread Reply:* Hey Petr. Please DM me or describe the issue here 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Susmitha Anandarao + (susmitha.anandarao@gmail.com) +
+
2023-01-27 15:24:47
 +
+

Hello.. I am trying to play with openlineage spark integration with Kafka and currently trying to just use the config as part of the spark submit command but I run into errors. Details in the 🧵

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Susmitha Anandarao + (susmitha.anandarao@gmail.com) +
+
2023-01-27 15:25:04
 +
+

*Thread Reply:* Command +spark-submit --packages "io.openlineage:openlineage_spark:0.19.+" \ + --conf "spark.extraListeners=io.openlineage.spark.agent.OpenLineageSparkListener" \ + --conf "spark.openlineage.transport.type=kafka" \ + --conf "spark.openlineage.transport.topicName=topicname" \ + --conf "spark.openlineage.transport.localServerId=Kafka_server" \ + file.py

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Susmitha Anandarao + (susmitha.anandarao@gmail.com) +
+
2023-01-27 15:25:14
 +
+

*Thread Reply:* 23/01/27 17:29:06 ERROR AsyncEventQueue: Listener OpenLineageSparkListener threw an exception +java.lang.NullPointerException + at io.openlineage.client.transports.TransportFactory.build(TransportFactory.java:44) + at io.openlineage.spark.agent.EventEmitter.&lt;init&gt;(EventEmitter.java:40) + at io.openlineage.spark.agent.OpenLineageSparkListener.initializeContextFactoryIfNotInitialized(OpenLineageSparkListener.java:278) + at io.openlineage.spark.agent.OpenLineageSparkListener.onApplicationStart(OpenLineageSparkListener.java:267) + at org.apache.spark.scheduler.SparkListenerBus.doPostEvent(SparkListenerBus.scala:55) + at org.apache.spark.scheduler.SparkListenerBus.doPostEvent$(SparkListenerBus.scala:28) + at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37) + at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37) + at org.apache.spark.util.ListenerBus.postToAll(ListenerBus.scala:117) + at org.apache.spark.util.ListenerBus.postToAll$(ListenerBus.scala:101) + at org.apache.spark.scheduler.AsyncEventQueue.super$postToAll(AsyncEventQueue.scala:105) + at org.apache.spark.scheduler.AsyncEventQueue.$anonfun$dispatch$1(AsyncEventQueue.scala:105) + at scala.runtime.java8.JFunction0$mcJ$sp.apply(JFunction0$mcJ$sp.java:23) + at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62) + at <a href="http://org.apache.spark.scheduler.AsyncEventQueue.org">org.apache.spark.scheduler.AsyncEventQueue.org</a>$apache$spark$scheduler$AsyncEventQueue$$dispatch(AsyncEventQueue.scala:100) + at org.apache.spark.scheduler.AsyncEventQueue$$anon$2.$anonfun$run$1(AsyncEventQueue.scala:96) + at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1446) + at org.apache.spark.scheduler.AsyncEventQueue$$anon$2.run(AsyncEventQueue.scala:96)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Susmitha Anandarao + (susmitha.anandarao@gmail.com) +
+
2023-01-27 15:25:31
 +
+

*Thread Reply:* I would appreciate any pointers on getting started with using openlineage-spark with Kafka.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Susmitha Anandarao + (susmitha.anandarao@gmail.com) +
+
2023-01-27 16:15:00
 +
+

*Thread Reply:* Also this might seem a little elementary but the kafka topic itself, should it be hosted on the spark cluster or could it be any kafka topic?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Susmitha Anandarao + (susmitha.anandarao@gmail.com) +
+
2023-01-30 08:37:07
 +
+

*Thread Reply:* 👀 Could I get some help on this, please?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-01-30 09:07:08
 +
+

*Thread Reply:* I think any NullPointerException is clearly our bug, can you open issue on OL GitHub?

+ + + +
+ 👍 Susmitha Anandarao +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Susmitha Anandarao + (susmitha.anandarao@gmail.com) +
+
2023-01-30 09:30:51
 +
+

*Thread Reply:* @Maciej Obuchowski Another interesting thing is if I use 0.19.2 version specifically, I get +23/01/30 14:28:33 INFO RddExecutionContext: RDDs are empty: skipping sending OpenLineage event

+ +

I am trying to print to console at the moment. I haven't been able to get Kafka transport type working though.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-01-30 09:41:12
 +
+

*Thread Reply:* Are you getting events printed on the console though? This log should not affect you if you're running, for example Spark SQL jobs

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Susmitha Anandarao + (susmitha.anandarao@gmail.com) +
+
2023-01-30 09:42:28
 +
+

*Thread Reply:* I am trying to run a python file using pyspark. 23/01/30 14:40:49 INFO RddExecutionContext: RDDs are empty: skipping sending OpenLineage event +I see this and don't see any events on the console.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-01-30 09:55:41
 +
+

*Thread Reply:* Any logs filling pattern +log.warn("Unable to access job conf from RDD", nfe); +or +<a href="http://log.info">log.info</a>("Found job conf from RDD {}", jc); +before?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Susmitha Anandarao + (susmitha.anandarao@gmail.com) +
+
2023-01-30 09:57:20
 +
+

*Thread Reply:* ```23/01/30 14:40:48 INFO DAGScheduler: Submitting ShuffleMapStage 0 (PairwiseRDD[2] at reduceByKey at /tmp/spark-20487725-f49b-4587-986d-e63a61890673/statusapidemo.py:47), which has no missing parents +23/01/30 14:40:49 WARN RddExecutionContext: Unable to access job conf from RDD +java.lang.NoSuchFieldException: Field is not instance of HadoopMapRedWriteConfigUtil + at io.openlineage.spark.agent.lifecycle.RddExecutionContext.lambda$setActiveJob$0(RddExecutionContext.java:117) + at java.util.Optional.orElseThrow(Optional.java:290) + at io.openlineage.spark.agent.lifecycle.RddExecutionContext.setActiveJob(RddExecutionContext.java:115) + at java.util.Optional.ifPresent(Optional.java:159) + at io.openlineage.spark.agent.OpenLineageSparkListener.lambda$onJobStart$9(OpenLineageSparkListener.java:148) + at java.util.Optional.ifPresent(Optional.java:159) + at io.openlineage.spark.agent.OpenLineageSparkListener.onJobStart(OpenLineageSparkListener.java:145) + at org.apache.spark.scheduler.SparkListenerBus.doPostEvent(SparkListenerBus.scala:37) + at org.apache.spark.scheduler.SparkListenerBus.doPostEvent$(SparkListenerBus.scala:28) + at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37) + at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37) + at org.apache.spark.util.ListenerBus.postToAll(ListenerBus.scala:117) + at org.apache.spark.util.ListenerBus.postToAll$(ListenerBus.scala:101) + at org.apache.spark.scheduler.AsyncEventQueue.super$postToAll(AsyncEventQueue.scala:105) + at org.apache.spark.scheduler.AsyncEventQueue.$anonfun$dispatch$1(AsyncEventQueue.scala:105) + at scala.runtime.java8.JFunction0$mcJ$sp.apply(JFunction0$mcJ$sp.java:23) + at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62) + at org.apache.spark.scheduler.AsyncEventQueue.org$apache$spark$scheduler$AsyncEventQueue$$dispatch(AsyncEventQueue.scala:100) + at org.apache.spark.scheduler.AsyncEventQueue$$anon$2.$anonfun$run$1(AsyncEventQueue.scala:96) + at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1446) + at org.apache.spark.scheduler.AsyncEventQueue$$anon$2.run(AsyncEventQueue.scala:96)

+ +

23/01/30 14:40:49 INFO RddExecutionContext: Found job conf from RDD Configuration: core-default.xml, core-site.xml, mapred-default.xml, mapred-site.xml, yarn-default.xml, yarn-site.xml, hdfs-default.xml, hdfs-rbf-default.xml, hdfs-site.xml, hdfs-rbf-site.xml, resource-types.xml

+ +

23/01/30 14:40:49 INFO RddExecutionContext: Found output path null from RDD PythonRDD[5] at collect at /tmp/spark-20487725-f49b-4587-986d-e63a61890673/statusapidemo.py:48 +23/01/30 14:40:49 INFO RddExecutionContext: RDDs are empty: skipping sending OpenLineage event``` +I see both actually.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-01-30 10:03:35
 +
+

*Thread Reply:* I think this is same problem as this: https://github.com/OpenLineage/OpenLineage/issues/1521

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-01-30 10:04:14
 +
+

*Thread Reply:* and I think I might have solution on a branch for it, just need to polish it up to release

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Susmitha Anandarao + (susmitha.anandarao@gmail.com) +
+
2023-01-30 10:13:37
 +
+

*Thread Reply:* Aah got it. I will give it a try with SQL and a jar.

+ +

Do you have a ETA on when the python issue would be fixed?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Susmitha Anandarao + (susmitha.anandarao@gmail.com) +
+
2023-01-30 10:37:51
 +
+

*Thread Reply:* @Maciej Obuchowski Well I run into the same errors if I run spark-submit on a jar.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-01-30 10:38:44
 +
+

*Thread Reply:* I think that has nothing to do with python

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-01-30 10:39:16
 +
+

*Thread Reply:* BTW, which Spark version are you using?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Susmitha Anandarao + (susmitha.anandarao@gmail.com) +
+
2023-01-30 10:41:22
 +
+

*Thread Reply:* We are on 3.3.1

+ + + +
+ 👍 Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Susmitha Anandarao + (susmitha.anandarao@gmail.com) +
+
2023-01-30 11:38:24
 +
+

*Thread Reply:* @Maciej Obuchowski Do you have a estimated release date for the fix. Our team is specifically interested in using the Emitter to write out to Kafka.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-01-30 11:46:30
 +
+

*Thread Reply:* I think we plan to release somewhere in the next week

+ + + +
+ :gratitude_thank_you: Susmitha Anandarao +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-02-06 09:21:25
 +
+

*Thread Reply:* @Susmitha Anandarao PR fixing this has been merged, release should be today

+ + + +
+ :gratitude_thank_you: Susmitha Anandarao +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paul Lee + (paullee@lyft.com) +
+
2023-01-27 16:31:45
 +
+

👋 +what would be the reason conn_id on something like SQLCheckOperator ends up being None when OpenLineage attempts to extract metadata but is fine on task execution?

+ +

i'm using OpenLineage for Airflow 0.14.1 on 2.3.4 and i'm getting an error about connid not being found. it's a SQLCheckOperator where the check runs fine but the task fails because when OpenLineage goes to extract task metadata it attempts to grab the connid but at that moment it finds it to be None.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2023-01-27 18:38:40
 +
+

*Thread Reply:* hmmm, I am not sure. perhaps @Benji Lampel can help, he’s very familiar with those operators.

+ + + +
+ 👍 Paul Lee +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paul Lee + (paullee@lyft.com) +
+
2023-01-27 18:46:15
 +
+

*Thread Reply:* @Benji Lampel any help would be appreciated!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Benji Lampel + (benjamin@astronomer.io) +
+
2023-01-30 09:01:34
 +
+

*Thread Reply:* Hey Paul, the SQLCheckExtractors were written with the intent that they would be used by a provider that inherits for them - they are all treated as a sort of base class. What is the exact error message you're getting? And what is the operator code? +Could you try this with a PostgresCheckOperator ? +(Also, only the SqlColumnCheckOperator and SqlTableCheckOperator will provide data quality facets in their output, those functions are not implementable in the other operators at this time)

+ + + +
+ 👀 Paul Lee +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paul Lee + (paullee@lyft.com) +
+
2023-01-31 14:36:07
 +
+

*Thread Reply:* @Benji Lampel here is the error message. i am not sure what the operator code is.

+ +

3-01-31, 00:32:38 UTC] {logging_mixin.py:115} WARNING - Traceback (most recent call last): +[2023-01-31, 00:32:38 UTC] {logging_mixin.py:115} WARNING - File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner +[2023-01-31, 00:32:38 UTC] {logging_mixin.py:115} WARNING - self.run() +[2023-01-31, 00:32:38 UTC] {logging_mixin.py:115} WARNING - File "/usr/lib/python3.8/threading.py", line 870, in run +[2023-01-31, 00:32:38 UTC] {logging_mixin.py:115} WARNING - self._target(**self._args, ****self._kwargs) +[2023-01-31, 00:32:38 UTC] {logging_mixin.py:115} WARNING - File "/code/venvs/venv/lib/python3.8/site-packages/openlineage/airflow/listener.py", line 99, in on_running +[2023-01-31, 00:32:38 UTC] {logging_mixin.py:115} WARNING - task_metadata = extractor_manager.extract_metadata(dagrun, task) +[2023-01-31, 00:32:38 UTC] {logging_mixin.py:115} WARNING - File "/code/venvs/venv/lib/python3.8/site-packages/openlineage/airflow/extractors/manager.py", line 28, in extract_metadata +[2023-01-31, 00:32:38 UTC] {logging_mixin.py:115} WARNING - extractor = self._get_extractor(task) +[2023-01-31, 00:32:38 UTC] {logging_mixin.py:115} WARNING - File "/code/venvs/venv/lib/python3.8/site-packages/openlineage/airflow/extractors/manager.py", line 96, in _get_extractor +[2023-01-31, 00:32:38 UTC] {logging_mixin.py:115} WARNING - self.task_to_extractor.instantiate_abstract_extractors(task) +[2023-01-31, 00:32:38 UTC] {logging_mixin.py:115} WARNING - File "/code/venvs/venv/lib/python3.8/site-packages/openlineage/airflow/extractors/extractors.py", line 118, in instantiate_abstract_extractors +[2023-01-31, 00:32:38 UTC] {logging_mixin.py:115} WARNING - task_conn_type = BaseHook.get_connection(task.conn_id).conn_type +[2023-01-31, 00:32:38 UTC] {logging_mixin.py:115} WARNING - File "/code/venvs/venv/lib/python3.8/site-packages/airflow/hooks/base.py", line 67, in get_connection +[2023-01-31, 00:32:38 UTC] {logging_mixin.py:115} WARNING - conn = Connection.get_connection_from_secrets(conn_id) +[2023-01-31, 00:32:38 UTC] {logging_mixin.py:115} WARNING - File "/code/venvs/venv/lib/python3.8/site-packages/airflow/models/connection.py", line 430, in get_connection_from_secrets +[2023-01-31, 00:32:38 UTC] {logging_mixin.py:115} WARNING - raise AirflowNotFoundException(f"The conn_id `{conn_id}` isn't defined") +[2023-01-31, 00:32:38 UTC] {logging_mixin.py:115} WARNING - airflow.exceptions.AirflowNotFoundException: The conn_id `None` isn't defined

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paul Lee + (paullee@lyft.com) +
+
2023-01-31 14:37:06
 +
+

*Thread Reply:* and above that

+ +

[2023-01-31, 00:32:38 UTC] {connection.py:424} ERROR - Unable to retrieve connection from secrets backend (EnvironmentVariablesBackend). Checking subsequent secrets backend. +Traceback (most recent call last): + File "/code/venvs/venv/lib/python3.8/site-packages/airflow/models/connection.py", line 420, in get_connection_from_secrets + conn = secrets_backend.get_connection(conn_id=conn_id) + File "/code/venvs/venv/lib/python3.8/site-packages/airflow/secrets/base_secrets.py", line 91, in get_connection + value = self.get_conn_value(conn_id=conn_id) + File "/code/venvs/venv/lib/python3.8/site-packages/airflow/secrets/environment_variables.py", line 48, in get_conn_value + return os.environ.get(CONN_ENV_PREFIX + conn_id.upper())

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paul Lee + (paullee@lyft.com) +
+
2023-01-31 14:39:31
 +
+

*Thread Reply:* sorry, i should mention we're wrapping over the CheckOperator as we're still migrating from 1.10.15 @Benji Lampel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Benji Lampel + (benjamin@astronomer.io) +
+
2023-01-31 15:09:51
 +
+

*Thread Reply:* What do you mean by wrapping the CheckOperator? Like how so, exactly? Can you show me the operator code you're using in the DAG?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paul Lee + (paullee@lyft.com) +
+
2023-01-31 17:38:45
 +
+

*Thread Reply:* like so

+ +

class CustomSQLCheckOperator(CheckOperator): +....

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paul Lee + (paullee@lyft.com) +
+
2023-01-31 17:39:30
 +
+

*Thread Reply:* i think i found the issue though, we have our own get_hook function and so we don't follow the traditional Airflow way of setting CONN_ID which is why CONN_ID is always None and that path only gets called through OpenLineage which doesn't ever get called with our custom wrapper

+ + + +
+ ✅ Benji Lampel +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-01-30 03:50:39
 +
+

Hi everyone, I am using openlineage to capture column level lineage from spark databricks. I noticed that the environment variables captured are only present in the start event, but are not present in the complete event. Is there a reason why it is implemented like this? It seems more intuitive that whatever variables are present in the start event should also be present in the complete event...

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Susmitha Anandarao + (susmitha.anandarao@gmail.com) +
+
2023-01-31 08:30:37
 +
+

Hi everyone.. Does the DBT integration provide an option to emit events to a Kafka topic similar to the Spark integration? I could not find anything regarding this in the documentation and I wanted to make sure if only http transport type is supported. Thank you!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2023-01-31 12:57:47
 +
+

*Thread Reply:* The dbt integration uses the python client, you should be able to do something similar than with the java client. See here: https://github.com/OpenLineage/OpenLineage/tree/main/client/python#kafka

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Susmitha Anandarao + (susmitha.anandarao@gmail.com) +
+
2023-01-31 13:26:33
 +
+

*Thread Reply:* Thank you for this!

+ +

I created a openlineage.yml file with the following data to test out the integration locally. +transport: + type: "kafka" + config: { 'bootstrap.servers': 'localhost:9092', } + topic: "ol_dbt_events" +However, I run into a no module named 'confluent_kafka' error from this code. +Running OpenLineage dbt wrapper version 0.19.2 +This wrapper will send OpenLineage events at the end of dbt execution. +Traceback (most recent call last): + File "/Users/susmithaanandarao/.pyenv/virtualenvs/dbt-examples-domain-repo/3.9.8/bin/dbt-ol", line 168, in &lt;module&gt; + main() + File "/Users/susmithaanandarao/.pyenv/virtualenvs/dbt-examples-domain-repo/3.9.8/bin/dbt-ol", line 94, in main + client = OpenLineageClient.from_environment() + File "/Users/susmithaanandarao/.pyenv/virtualenvs/dbt-examples-domain-repo/3.9.8/lib/python3.9/site-packages/openlineage/client/client.py", line 73, in from_environment + return cls(transport=get_default_factory().create()) + File "/Users/susmithaanandarao/.pyenv/virtualenvs/dbt-examples-domain-repo/3.9.8/lib/python3.9/site-packages/openlineage/client/transport/factory.py", line 37, in create + return self._create_transport(yml_config) + File "/Users/susmithaanandarao/.pyenv/virtualenvs/dbt-examples-domain-repo/3.9.8/lib/python3.9/site-packages/openlineage/client/transport/factory.py", line 69, in _create_transport + return transport_class(config_class.from_dict(config)) + File "/Users/susmithaanandarao/.pyenv/virtualenvs/dbt-examples-domain-repo/3.9.8/lib/python3.9/site-packages/openlineage/client/transport/kafka.py", line 43, in __init__ + import confluent_kafka as kafka +ModuleNotFoundError: No module named 'confluent_kafka' +Manually installing confluent-kafka worked. But I am curious why it was not automatically installed and if I am missing any config.

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/3d0fee9002a8d7a85f47c8a6e57446174f433e34/client/python/setup.py | setup.py>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-02-02 14:39:29
 +
+

*Thread Reply:* @Susmitha Anandarao It's not installed because it's large binary package. We don't want to install for every user something giant majority won't use, and it's 100x bigger than rest of the client.

+ +

We need to indicate this way better, and do not throw this error directly at user thought, both in docs and code.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Benji Lampel + (benjamin@astronomer.io) +
+
2023-01-31 11:28:53
 +
+

~Hey, would love to see a release of OpenLineage~

+ + + +
+ ➕ Michael Robinson, Jakub Dardziński, Ross Turk, Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2023-01-31 12:51:44
 +
+

Hello, I have been working on a proposal to bring an OpenLineage provider to +Airflow. I am currently looking for feedback on a draft AIP. See the thread here: https://lists.apache.org/thread/2brvl4ynkxcff86zlokkb47wb5gx8hw7

+ + + +
+ 🔥 Maciej Obuchowski, Viraj Parekh, Jakub Dardziński, Enrico Rotundo, Harel Shein, Paweł Leszczyński +
+ +
+ 👀 Enrico Rotundo +
+ +
+ 🙌 Will Johnson +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Bramha Aelem + (bramhaaelem@gmail.com) +
+
2023-01-31 14:02:21
 +
+

@Willy Lulciuc, - Any updates on - https://github.com/OpenLineage/OpenLineage/discussions/1494

+
+ + + + +
#1494 Usage Of databricks generated tables as an in/out in the Datapipeline Process
+
+ <p>Hi,</p> + +<p>First activity : Making HTTP Call to pull the lookup data and store it in datalake. <br /> +Second Activity : After the completion of first activity, invoking Azure databricks to use the lookup file and generate the output tables. What are all the steps should be follow to refer databricks generated tables facets as an output in the current activity &amp; input to the subsequent activities in the pipeline. <br /> +When I configure spark generated tables as output to the current activity the existing spark metadata is not showing up. <br /> +How can this be achievable. <br /> +Can someone please guide me on this. <br /> +Note: After the execution of each activity in ADF Pipeline I am sending start and complete/fail event lineage to Marquez.</p> + +<p>Thanks, <br /> +Brahma</p> +
+ +
+
Category
+ Ideas +
+ +
+
Comments
+ 3 +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Natalie Zeller + (natalie.zeller@naturalint.com) +
+
2023-02-02 08:26:38
 +
+

Hello, +While trying to use OpenLineage with spark, I've noticed that sometimes the query execution is missing or already got closed (here is the relevant code). As a result, some of the events are skipped. Is this a known issue? Is there a way to overcome it?

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/02a01a1ed60916424bff7755942e0f2e10ae8c67/integration/spark/app/src/main/java/io/openlineage/spark/agent/lifecycle/ContextFactory.java | ContextFactory.java>
+
+ <pre><code> return Optional.ofNullable(SQLExecution.getQueryExecution(executionId)) +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-02-02 08:39:34
 +
+

*Thread Reply:* https://github.com/OpenLineage/OpenLineage/issues/999#issuecomment-1209048556

+ +

Does this fit your experience?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-02-02 08:39:59
 +
+

*Thread Reply:* We sometimes experience this in context of very small, quick jobs

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Natalie Zeller + (natalie.zeller@naturalint.com) +
+
2023-02-02 08:43:24
 +
+

*Thread Reply:* Yes, my scenarios are dealing with quick jobs. +Good to know that we will be able to solve it with future spark versions. Thanks!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-02-02 11:09:13
 +
+

@channel +This month’s OpenLineage TSC meeting is next Thursday, February 9th, at 10 am PT. Join us on Zoom: https://bit.ly/OLzoom. All are welcome! +On the tentative agenda:

+ +
  1. Recent release overview @Michael Robinson
  2. AIP: OpenLineage in Airflow
  3. Discussions: +• Real-world implementation of OpenLineage (What does it really mean?) @Sheeri Cabral (Collibra) (continued) +• Using namespaces @Michael Robinson
  4. Open discussion +Notes: https://bit.ly/OLwiki +Is there a topic you think the community should discuss at this or a future meeting? Reply or DM me to add items to the agenda.
    5. +
+ + + +
+ 🔥 Maciej Obuchowski, Bramha Aelem, Viraj Parekh, Brad Paskewitz, Harel Shein +
+ +
+ 👍 Bramha Aelem, Viraj Parekh, Enrico Rotundo, Daniel Henneberger +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-02-03 13:22:51
 +
+

Hi folks, I’m opening a vote to release OpenLineage 0.20.0, featuring: +• Airflow: add new extractor for GCSToGCSOperator + Adds a new extractor for this operator. +• Proxy: implement lineage event validator for client proxy
+ Implements logic in the proxy (which is still in development) for validating and handling lineage events. +• A fix of a breaking change in the common integration and other bug fixes in the DBT, Airflow, Spark, and SQL integrations and in the Java and Python clients. +As per the policy here, three +1s from committers will authorize. Thanks in advance.

+ + + +
+ ➕ Willy Lulciuc, Maciej Obuchowski, Julien Le Dem, Jakub Dardziński, Howard Yoo +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2023-02-03 13:24:03
 +
+

*Thread Reply:* exciting to see the client proxy work being released by @Minkyu Park 💯

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-02-03 13:35:38
 +
+

*Thread Reply:* This was without a doubt among the fastest release votes we’ve ever had 😉 . Thank you! You can expect the release to happen on Monday.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Minkyu Park + (minkyu@datakin.com) +
+
2023-02-03 14:02:52
 +
+

*Thread Reply:* Lol the proxy is still in development and not ready for use

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2023-02-03 14:03:26
 +
+

*Thread Reply:* Good point! Let’s make that clear in the release / docs?

+ + + +
+ 👍 Michael Robinson, Minkyu Park +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Minkyu Park + (minkyu@datakin.com) +
+
2023-02-03 14:03:33
 +
+

*Thread Reply:* But it doesn’t block anything anyway, so happy to see the release

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Minkyu Park + (minkyu@datakin.com) +
+
2023-02-03 14:04:38
 +
+

*Thread Reply:* We can celebrate that the proposal for the proxy is merged. I’m happy with that 🥳

+ + + +
+ 🎊 Willy Lulciuc +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Daniel Joanes + (djoanes@gmail.com) +
+
2023-02-06 00:01:49
 +
+

Hey 👋 From what I gather, there's no solution to getting column level lineage from spark streaming jobs. Is there a issue I can follow to keep track?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2023-02-06 14:47:15
 +
+

*Thread Reply:* Hey @Daniel Joanes! thanks for the question.

+ +

I am not aware of an issue that captures this. Column-level lineage is a somewhat new facet in the spec, and implementations across the various integrations are in varying states of readiness.

+ +

I invite you to create the issue - that way it's attributed to you, which makes sense because you're the one who first raised it. But I'm happy to create it for you & give you the PR# if you'd rather, just let me know 👍

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Daniel Joanes + (djoanes@gmail.com) +
+
2023-02-06 14:50:59
 +
+

*Thread Reply:* Go for it, once it's created i'll add a watch

+ + + +
+ 👍 Ross Turk +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Daniel Joanes + (djoanes@gmail.com) +
+
2023-02-06 14:51:13
 +
+

*Thread Reply:* Thanks Ross!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2023-02-06 23:10:30
 +
+

*Thread Reply:* https://github.com/OpenLineage/OpenLineage/issues/1581

+
+ + + + +
#1581 Spark integration does not support column-lineage for streaming jobs
+
+ <p>There does not seem to be support for column-level lineage in the Spark integration for streaming jobs. Or, alternatively, it is insufficiently documented.</p> +
+ +
+
Labels
+ integration/spark, column-level-lineage +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-02-07 18:46:50
 +
+

@channel +OpenLineage 0.20.4 is now available, including: +Additions: +• Airflow: add new extractor for GCSToGCSOperator #1495 @sekikn +• Flink: resolve topic names from regex, support 1.16.0 #1522 @pawel-big-lebowski +• Proxy: implement lineage event validator for client proxy #1469 @fm100 +Changes: +• CI: use ruff instead of flake8, isort, etc., for linting and formatting #1526 @mobuchowski +Plus many bug fixes & doc changes. +Thank you to all our contributors! +Release: https://github.com/OpenLineage/OpenLineage/releases/tag/0.20.4 +Changelog: https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md +Commit history: https://github.com/OpenLineage/OpenLineage/compare/0.19.2...0.20.4 +Maven: https://oss.sonatype.org/#nexus-search;quick~openlineage +PyPI: https://pypi.org/project/openlineage-python/

+ + + +
+ 🎉 Kengo Seki, Harel Shein, Willy Lulciuc, Nadav Geva +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-02-08 15:31:32
 +
+

@channel +Friendly reminder: this month’s OpenLineage TSC meeting is tomorrow at 10am, and all are welcome. https://openlineage.slack.com/archives/C01CK9T7HKR/p1675354153489629

+ + + +
+ ❤️ Minkyu Park, Kengo Seki, Paweł Leszczyński, Harel Shein, Sheeri Cabral (Collibra), Enrico Rotundo +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harel Shein + (harel.shein@gmail.com) +
+
2023-02-09 10:50:07
 +
+

Hey, can we please schedule a release of OpenLineage? I would like to have a release that includes the latest fixes for Async Operator on Airflow and some dbt bug fixes.

+ + + +
+ ➕ Michael Robinson, Maciej Obuchowski, Benji Lampel, Willy Lulciuc +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-02-09 10:50:49
 +
+

*Thread Reply:* Thanks for requesting a release. 3 +1s from committers will authorize an immediate release.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-02-09 11:15:35
 +
+

*Thread Reply:* 0.20.5 ?

+ + + +
+ ➕ Harel Shein +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Benji Lampel + (benjamin@astronomer.io) +
+
2023-02-09 11:28:20
 +
+

*Thread Reply:* @Michael Robinson auth'd

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-02-09 11:32:06
 +
+

*Thread Reply:* 👍 the release is authorized

+ + + +
+ ❤️ Sheeri Cabral (Collibra), Willy Lulciuc, Paweł Leszczyński +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Avinash Pancham + (avinashpancham@outlook.com) +
+
2023-02-09 15:57:58
 +
+

Hi all, I have been experimenting with OpenLineage for a few days and it's great! I successfully setup the openlineage-spark listener on my Databricks cluster and that pushes openlineage data to our Marquez backend. That was all pretty easy to do 🙂

+ +

Now for my challenge: I would like to actually extend the metadata that my cluster pushes with custom values (you can think of spark config settings, commit hash of the executed code, or maybe even runtime defined values). I browsed through some documentation and found custom facets one can define. The link below describes how to use Python to push custom metadata to a backend, but I was actually hoping that there was a way to do this automatically in Spark. So ideally I would like to write my own OpenLineage.json (that has my custom facet) and tell Spark to use that Openlineage spec instead of the default one. In that way I hope my custom metadata will be forwarded automatically.

+ +

I just do not know how to do that (and whether that is even possible), since I could not find any tutorials on that topic. Any help on this would be greatly appreciated!

+ +

https://openlineage.io/docs/spec/facets/custom-facets

+
+
openlineage.io
+ + + +
Custom Facets | OpenLineage Docs
+
+ In addition to the existing facets mentioned in this documentation, users can extend the base facets and provide their own facet definition as part of the payload in OpenLineage event. For example, when OpenLineage event is emitted from the Apache Airflow using OpenLineage's Airflow integration, the following facets can be observed: +
+ + + + + + +
Original URL: https://openlineage.io/docs/spec/facets/custom-facets
+ + + +
+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.json | OpenLineage.json>
+
+ <p>``` +{ + "$schema": "<a href="https://json-schema.org/draft/2020-12/schema">https://json-schema.org/draft/2020-12/schema</a>", + "$id": "<a href="https://openlineage.io/spec/1-0-5/OpenLineage.json">https://openlineage.io/spec/1-0-5/OpenLineage.json</a>", + "$defs": { + "RunEvent": { + "type": "object", + "properties": { + "eventType": { + "description": "the current transition of the run state. It is required to issue 1 START event and 1 of [ COMPLETE, ABORT, FAIL ] event per run. Additional events with OTHER eventType can be added to the same run. For example to send additional metadata after the run is complete", + "type": "string", + "enum": [ + "START", + "RUNNING", + "COMPLETE", + "ABORT", + "FAIL", + "OTHER" + ], + "example": "START|RUNNING|COMPLETE|ABORT|FAIL|OTHER" + }, + "eventTime": { + "description": "the time the event occurred at", + "type": "string", + "format": "date-time" + }, + "run": { + "$ref": "#/$defs/Run" + }, + "job": { + "$ref": "#/$defs/Job" + }, + "inputs": { + "description": "The set of <strong><em>*input</em><em></strong> datasets.", + "type": "array", + "items": { + "$ref": "#/$defs/InputDataset" + } + }, + "outputs": { + "description": "The set of <strong></em><em>output</em>*</strong> datasets.", + "type": "array", + "items": { + "$ref": "#/$defs/OutputDataset" + } + }, + "producer": { + "description": "URI identifying the producer of this metadata. For example this could be a git url with a given tag or sha", + "type": "string", + "format": "uri", + "example": "<a href="https://github.com/OpenLineage/OpenLineage/blob/v1-0-0/client">https://github.com/OpenLineage/OpenLineage/blob/v1-0-0/client</a>" + }, + "schemaURL": { + "description": "The JSON Pointer (<a href="https://tools.ietf.org/html/rfc6901">https://tools.ietf.org/html/rfc6901</a>) URL to the corresponding version of the schema definition for this RunEvent", + "type": "string", + "format": "uri", + "example": "<a href="https://openlineage.io/spec/0-0-1/OpenLineage.json">https://openlineage.io/spec/0-0-1/OpenLineage.json</a>" + } + }, + "required": [ + "run", + "job", + "eventTime", + "producer", + "schemaURL" + ] + }, + "Run": { + "type": "object", + "properties": { + "runId": { + "description": "The globally unique ID of the run associated with the job.", + "type": "string", + "format": "uuid" + }, + "facets": { + "description": "The run facets.", + "type": "object", + "anyOf": [ + { + "type": "object", + "additionalProperties": { "$ref": "#/$defs/RunFacet" } + }, + { "$ref": "facets/ErrorMessageRunFacet.json" }, + { "$ref": "facets/ExternalQueryRunFacet.json" }, + { "$ref": "facets/NominalTimeRunFacet.json" }, + { "$ref": "facets/ParentRunFacet.json" } + ] + } + }, + "required": [ + "runId" + ] + }, + "RunFacet": { + "description": "A Run Facet", + "type": "object", + "allOf": [ + { "$ref": "#/$defs/BaseFacet" } + ] + }, + "Job": { + "type": "object", + "properties": { + "namespace": { + "description": "The namespace containing that job", + "type": "string", + "example": "my-scheduler-namespace" + }, + "name": { + "description": "The unique name for that job within that namespace", + "type": "string", + "example": "myjob.mytask" + }, + "facets": { + "description": "The job facets.", + "type": "object", + "anyOf": [ + { + "type": "object", + "additionalProperties": { "$ref": "#/$defs/JobFacet" } + }, + { "$ref": "facets/DocumentationJobFacet.json" }, + { "$ref": "facets/OwnershipJobFacet.json" }, + { "$ref": "facets/SourceCodeJobFacet.json" }, + { "$ref": "facets/SourceCodeLocationJobFacet.json" }, + { "$ref": "facets/SQLJobFacet.json" } + ] + } + }, + "required": [ + "namespace", + "name" + ] + }, + "JobFacet": { + "description": "A Job Facet", + "type": "object", + "allOf": [ + { "$ref": "#/$defs/BaseFacet" } + ] + }, + "InputDataset": { + "description": "An input dataset", + "type": "object", + "allOf": [ + { "$ref": "#/$defs/Dataset" }, + { + "type": "object", + "properties": { + "inputFacets": { + "description": "The input facets for this dataset.", + "type": "object", + "anyOf": [ + { + "type": "object", + "additionalProperties": { "$ref": "#/$defs/InputDatasetFacet" } + }, + { "$ref": "facets/DataQualityMetricsInputDatasetFacet.json"} + ] + } + } + } + ] + }, + "InputDatasetFacet": { + "description": "An Input Dataset Facet", + "type": "object", + "allOf": [ + { "$ref": "#/$defs/BaseFacet" } + ] + }, + "OutputDataset": { + "description": "An output dataset", + "type": "object", + "allOf": [ + { "$ref": "#/$defs/Dataset" }, + { + "type": "object", + "properties": { + "outputFacets": { + "description": "The output facets for this dataset", + "type": "object", + "anyOf": [ + { + "type": "object", + "additionalProperties": { "$ref": "#/$defs/OutputDatasetFacet" } + }, + { "$ref": "facets/OutputStatisticsOutputDatasetFacet.json"} + ] + } + } + } + ] + }, + "OutputDatasetFacet": { + "description": "An Output Dataset Facet", + "type": "object", + "allOf": [ + { "$ref": "#/$defs/BaseFacet" } + ] + }, + "Dataset": { + "type": "object", + "properties": { + "namespace": { + "description": "The namespace containing that dataset", + "type": "string", + "example": "my-datasource-namespace" + }, + "name": { + "description": "The unique name for that dataset within that namespace", + "type": "string", + "example": "instance.schema.table" + }, + "facets": { + "description": "The facets for this dataset", + "type": "object", + "anyOf": [ + { + "type": "object", + "additionalProperties": { "$ref": "#/$defs/DatasetFacet" } + }, + { "$ref": "facets/ColumnLineageDatasetFacet.json" }, + { "$ref": "facets/DatasourceDatasetFacet.json" }, + { "$ref": "facets/DataQualityAssertionsDatasetFacet.json" }, + { "$ref": "facets/LifecycleStateChangeDatasetFacet.json" }, + { "$ref": "facets/OwnershipDatasetFacet.json" }, + { "$ref": "facets/SchemaDatasetFacet.json" }, + { "$ref": "facets/StorageDatasetFacet.json" }, + { "$ref": "facets/SymlinksDatasetFacet.json" }, + { "$ref": "facets/DatasetVersionDatasetFacet.json" } + ] + } + }, + "required": [ + "namespace", + "name" + ] + }, + "DatasetFacet": { + "description": "A Dataset Facet", + "type": "object", + "allOf": [ + { "$ref": "#/$defs/BaseFacet" } + ] + }, + "BaseFacet": { + "description": "all fields of the base facet are prefixed with _ to avoid name conflicts in facets", + "type": "object", + "properties": { + "_producer": { + "description": "URI identifying the producer of this metadata. For example this could be a git url with a given tag or sha", + "type": "string", + "format": "uri", + "example": "<https://github.com/OpenLineage/OpenLineage/blob/v1-0-0/clie…</p> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Susmitha Anandarao + (susmitha.anandarao@gmail.com) +
+
2023-02-09 16:23:36
 +
+

*Thread Reply:* I am also exploring something similar, but writing to kafka, and would want to know more on how we could add custom metadata from spark.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-02-10 02:23:40
 +
+

*Thread Reply:* Hi @Avinash Pancham @Susmitha Anandarao, it's great to hear about successful experimenting on your side.

+ +

Although Openlineage spec provides some built-in facets definition, a facet object can be anything you want (https://openlineage.io/apidocs/openapi/#tag/OpenLineage/operation/postRunEvent). The example metadata provided in this chat could be put into job or run facets I believe.

+ +

There is also a way to extend Spark integration to collect custom metadata described here (https://github.com/OpenLineage/OpenLineage/tree/main/integration/spark#extending). One needs to create own JAR with DatasetFacetBuilders, RunFacetsBuilder (whatever is needed). openlineage-spark integration will make use of those bulders.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sheeri Cabral (Collibra) + (sheeri.cabral@collibra.com) +
+
2023-02-10 09:09:10
 +
+

*Thread Reply:* (I would love to see what your specs are! I’m not with Astronomer, just a community member, but I am finding that many of the customizations people are making to the spec are valuable ones that we should consider adding to core)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Susmitha Anandarao + (susmitha.anandarao@gmail.com) +
+
2023-02-14 16:51:28
 +
+

*Thread Reply:* Are there any examples out there of customizations already done in Spark? An example would definitely help!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-02-15 08:43:08
 +
+

*Thread Reply:* I think @Will Johnson might have something to add about customization

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2023-02-15 23:58:36
 +
+

*Thread Reply:* Oh man... Mike Collado did a nice write up on Slack of how many different ways there are to customize / extend OpenLineage. I know we all talked about doing a blog post at one point!

+ +

@Susmitha Anandarao - You might take a look at https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/shared/src/[…]ark/agent/facets/builder/DatabricksEnvironmentFacetBuilder.java which has a hard coded set of properties we are extracting.

+ +

It looks like Avinash's changes were accepted as well: https://github.com/OpenLineage/OpenLineage/pull/1545

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/shared/src/main/java/io/openlineage/spark/agent/facets/builder/DatabricksEnvironmentFacetBuilder.java | DatabricksEnvironmentFacetBuilder.java>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+
+ + + + +
#1545 Add CustomEnvironmentFacetBuilder class
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-02-10 12:42:24
 +
+

@channel +OpenLineage 0.20.6 is now available, including: +Additions +• Airflow: add new extractor for FTPFileTransmitOperator #1603 @sekikn +Changes +• Airflow: make extractors for async operators work #1601 @JDarDagran +Thanks to all our contributors! +For the bug fixes and details, see: +Release: https://github.com/OpenLineage/OpenLineage/releases/tag/0.20.6 +Changelog: https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md +Commit history: https://github.com/OpenLineage/OpenLineage/compare/0.20.4...0.20.6 +Maven: https://oss.sonatype.org/#nexus-search;quick~openlineage +PyPI: https://pypi.org/project/openlineage-python/

+ + + +
+ 🥳 Minkyu Park, Willy Lulciuc, Kengo Seki, Paweł Leszczyński, Anirudh Shrinivason, pankaj koti, Maciej Obuchowski +
+ +
+ ❤️ Minkyu Park, Ross Turk, Willy Lulciuc, Kengo Seki, Paweł Leszczyński, Anirudh Shrinivason, pankaj koti +
+ +
+ 🎉 Minkyu Park, Willy Lulciuc, Kengo Seki, Anirudh Shrinivason, pankaj koti +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-02-13 14:20:26
 +
+

Hi everyone, in case you missed the announcement at the most recent community meeting, our first-ever meetup will be held on March 9th in Providence, RI. Join us there to learn more about the present and future of OpenLineage, meet other members of the ecosystem, learn about the project’s goals and fundamental design, and participate in a discussion about the future of the project. +Food will be provided, and the meetup is open to all. Don’t miss this opportunity to influence the direction of this important new standard! We hope to see you there. +More information: https://openlineage.io/blog/data-lineage-meetup/

+
+
openlineage.io
+ + + +
Happening Soon - Our First Meetup!
+
+ The inaugural Data Lineage Meetup will take place on March 9th in Providence, RI. +
+ + + + + + +
Original URL: https://openlineage.io/blog/data-lineage-meetup/
+ + + +
+ + + +
+ 🎉 Harel Shein, Ross Turk, Maciej Obuchowski, Kengo Seki, Paweł Leszczyński, Willy Lulciuc, Sheeri Cabral (Collibra) +
+ +
+ 🔥 Harel Shein, Ross Turk, Maciej Obuchowski, Anirudh Shrinivason, Kengo Seki, Paweł Leszczyński, Willy Lulciuc, Sheeri Cabral (Collibra) +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Quentin Nambot + (qnambot@gmail.com) +
+
2023-02-15 04:52:27
 +
+

Hi, I opened a PR to fix the way that Athena extractor get the database, but spark integration tests failed. However I don't think that it is related to my PR, since I only updated the Airflow integration +Can anybody help me with that please? 🙏

+
+ + + + +
#1631 airflow: get table database in athena extractor
+
+ <p><strong>Problem</strong></p> + +<p>Athena extractor get database from operator. However it is possible to interact with tables from other databases.</p> + +<p>Closes: <a href="https://github.com/OpenLineage/OpenLineage/issues/1627">#ISSUE-NUMBER</a></p> + +<p><strong>Solution</strong></p> + +<p>If a table is used with a database (ex: <code>catalog.my_db.my_table</code> or <code>my_db.my_table</code>), the field <code>table.schema</code> is the database. However it is <code>None</code>. <br /> +So we can use this field to get database, or use operator default database if <code>table.schema is None</code></p> + +<p>☐ Your change modifies the <a href="https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.json">core</a> OpenLineage model +☐ Your change modifies one or more OpenLineage <a href="https://github.com/OpenLineage/OpenLineage/tree/main/spec/facets">facets</a></p> + +<p><strong>Checklist</strong></p> + +<p>☑︎ You've <a href="https://github.com/OpenLineage/OpenLineage/blob/main/why-the-dco.md">signed-off</a> your work +☑︎ Your pull request title follows our <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CONTRIBUTING.md#creating-pull-requests">guidelines</a> +☑︎ Your changes are accompanied by tests (<em>if relevant</em>) +☑︎ Your change contains a <a href="https://kurtisnusbaum.medium.com/stacked-diffs-keeping-phabricator-diffs-small-d9964f4dcfa6">small diff</a> and is self-contained +☐ You've updated any relevant documentation (<em>if relevant</em>) +☐ Your comment includes a one-liner for the changelog about the specific purpose of the change (<em>if necessary</em>) +☐ You've versioned the core OpenLineage model or facets according to <a href="https://docs.snowplowanalytics.com/docs/pipeline-components-and-applications/iglu/common-architecture/schemaver">SchemaVer</a> (<em>if relevant</em>) +☐ You've added a <a href="https://github.com/OpenLineage/OpenLineage/tree/main/.github/header_templates.md">header</a> to source files (<em>if relevant</em>)</p> + +<hr /> + +<p>SPDX-License-Identifier: Apache-2.0 <br /> +Copyright 2018-2023 contributors to the OpenLineage project</p> +
+ +
+
Labels
+ integration/airflow, extractor +
+ +
+
Comments
+ 2 +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Quentin Nambot + (qnambot@gmail.com) +
+
2023-02-15 04:52:59
 +
+

*Thread Reply:* https://app.circleci.com/pipelines/github/OpenLineage/OpenLineage/6398/workflows/9d2d19c8-f2d9-4148-a4f3-5dad3ba99eb1/jobs/97759 +ERROR: Missing environment variable {i}

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-02-15 07:19:39
 +
+

*Thread Reply:* @Quentin Nambot this happens because we run additional integration tests against real databases (like BigQuery) which aren't ever configured on forks, since we don't want to expose our secrets. We need to figure out how to make this experience better, but in the meantime we've pushed your code using git-push-fork-to-upstream-branch and it passes all the tests.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-02-15 07:21:49
 +
+

*Thread Reply:* Feel free to un-draft your PR if you think it's ready for review

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Quentin Nambot + (qnambot@gmail.com) +
+
2023-02-15 08:03:56
 +
+

*Thread Reply:* Ok nice thanks 👍

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Quentin Nambot + (qnambot@gmail.com) +
+
2023-02-15 08:04:49
 +
+

*Thread Reply:* I think it's ready, however should I update the version somewhere?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-02-15 08:42:39
 +
+

*Thread Reply:* @Quentin Nambot I don't think so - it's just that you opened PR as Draft , so I'm not sure if you want to add something else to it.

+ + + +
+ 👍 Quentin Nambot +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Quentin Nambot + (qnambot@gmail.com) +
+
2023-02-15 08:43:36
 +
+

*Thread Reply:* No I don't want to add anything so I opened it 👍

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Allison Suarez + (asuarezmiranda@lyft.com) +
+
2023-02-15 21:26:37
 +
+

@here I have a question about extending the spark integration. Is there a way to use a custom visitor factory? I am trying to see if I can add a visitor for a command that is not currently covered in this integration (AlterTableAddPartitionCommand). It seems that because its not in the base visitor factory I am unable to use the visitor I created.

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/shared/src/main/java/io/openlineage/spark/agent/lifecycle/BaseVisitorFactory.java | BaseVisitorFactory.java>
+
+ <pre><code> list.add(new AlterTableAddColumnsCommandVisitor(context)); +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2023-02-15 21:30:47
 +
+

*Thread Reply:* You can add your own EventHandlerFactory https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/shared/src/main/java/io/openlineage/spark/api/OpenLineageEventHandlerFactory.java . See the docs about extending here - https://github.com/OpenLineage/OpenLineage/tree/main/integration/spark#extending

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/shared/src/main/java/io/openlineage/spark/api/OpenLineageEventHandlerFactory.java | OpenLineageEventHandlerFactory.java>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Allison Suarez + (asuarezmiranda@lyft.com) +
+
2023-02-15 21:32:19
 +
+

*Thread Reply:* I have that set up already like this: +public class LyftOpenLineageEventHandlerFactory implements OpenLineageEventHandlerFactory { + @Override + public Collection&lt;PartialFunction&lt;LogicalPlan, List&lt;OutputDataset&gt;&gt;&gt; + createOutputDatasetQueryPlanVisitors(OpenLineageContext context) { + Collection&lt;PartialFunction&lt;LogicalPlan, List&lt;OutputDataset&gt;&gt;&gt; visitors = new ArrayList&lt;PartialFunction&lt;LogicalPlan, List&lt;OutputDataset&gt;&gt;&gt;(); + visitors.add(new LyftInsertIntoHadoopFsRelationVisitor(context)); + visitors.add(new AlterTableAddPartitionVisitor(context)); + visitors.add(new AlterTableDropPartitionVisitor(context)); + return visitors; + } +}

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Allison Suarez + (asuarezmiranda@lyft.com) +
+
2023-02-15 21:33:35
 +
+

*Thread Reply:* do I just add a constructor? the visitorFactory is private so I wasn't sure if that's something that was intended to change

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Allison Suarez + (asuarezmiranda@lyft.com) +
+
2023-02-15 21:34:30
 +
+

*Thread Reply:* .

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Allison Suarez + (asuarezmiranda@lyft.com) +
+
2023-02-15 21:34:49
 +
+

*Thread Reply:* @Michael Collado

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2023-02-15 21:35:14
 +
+

*Thread Reply:* The VisitorFactory is only used by the internal EventHandlerFactory. It shouldn’t be needed for your custom one

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2023-02-15 21:35:48
 +
+

*Thread Reply:* Have you added the file to the META-INF folder of your jar?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Allison Suarez + (asuarezmiranda@lyft.com) +
+
2023-02-16 11:01:56
 +
+

*Thread Reply:* yes, I am able to use my custom event handler factory with a list of visitors but for some reason I cant access the visitors for some commands (AlterTableAddPartitionCommand) is one

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Allison Suarez + (asuarezmiranda@lyft.com) +
+
2023-02-16 11:02:29
 +
+

*Thread Reply:* so even if I set up everything correctly I am unable to reach the code for that specific visitor

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Allison Suarez + (asuarezmiranda@lyft.com) +
+
2023-02-16 11:05:22
 +
+

*Thread Reply:* and my assumption is I can reach other commands but not this one because the command is not defined in the BaseVisitorFactory but maybe im wrong @Michael Collado

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2023-02-16 15:05:19
 +
+

*Thread Reply:* the VisitorFactory is loaded by the InternalEventHandlerFactory here. However, the createOutputDatasetQueryPlanVisitors should contain a union of everything defined by the VisitorFactory as well as your custom visitors: see this code.

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/app/src/main/java/io/openlineage/spark/agent/lifecycle/InternalEventHandlerFactory.java | InternalEventHandlerFactory.java>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/app/src/main/java/io/openlineage/spark/agent/lifecycle/InternalEventHandlerFactory.java | InternalEventHandlerFactory.java>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Collado + (collado.mike@gmail.com) +
+
2023-02-16 15:09:21
 +
+

*Thread Reply:* there might be a conflict with another visitor that’s being matched against that command. Can you turn on debug logging and look for this line to see what visitor is being applied to that command?

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/shared/src/main/java/io/openlineage/spark/agent/util/PlanUtils.java | PlanUtils.java>
+
+ <pre><code> if (log.isDebugEnabled()) { + log.debug( + "Visitor {} visited {}, returned {}", + pfn.getClass().getCanonicalName(), + x.getClass().getCanonicalName(), + collection); + } +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Allison Suarez + (asuarezmiranda@lyft.com) +
+
2023-02-16 16:54:46
 +
+

*Thread Reply:* This was helpful, it works now, thank you so much Michael!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
slackbot + +
+
2023-02-16 19:08:26
 +
+

This message was deleted.

+ + + +
+ 👋 Willy Lulciuc +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2023-02-16 19:09:49
 +
+

*Thread Reply:* what is the curl cmd you are running? and what endpoint are you hitting? (assuming Marquez?)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
thebruuu + (bruno.c@inwind.it) +
+
2023-02-16 19:18:28
 +
+

*Thread Reply:* yep +I am running curl - X curl -X POST http://localhost:5000/api/v1/namespaces/test ^ + -H 'Content-Type: application/json' ^ + -d '{ownerName:"me", description:"no description"^ + }'

+ +

the weird thing is the log where I don't have a 0.0.0.0 IP (the log correspond to the equivament postman command)

+ +

marquez-api | WARN [2023-02-17 00:14:32,695] marquez.logging.LoggingMdcFilter: status: 405 +marquez-api | XXX.23.0.1 - - [17/Feb/2023:00:14:32 +0000] "POST /api/v1/namespaces/test HTTP/1.1" 405 52 "-" "PostmanRuntime/7.30.0" 2

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2023-02-16 19:23:08
 +
+

*Thread Reply:* Marquez logs all supported endpoints (and methods) on start up. For example, here are all the supported methods on /api/v1/namespaces/{namespace} : +marquez-api | DELETE /api/v1/namespaces/{namespace} (marquez.api.NamespaceResource) +marquez-api | GET /api/v1/namespaces/{namespace} (marquez.api.NamespaceResource) +marquez-api | PUT /api/v1/namespaces/{namespace} (marquez.api.NamespaceResource) +To ADD a namespace, you’ll want to use PUT (see API docs)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
thebruuu + (bruno.c@inwind.it) +
+
2023-02-16 19:26:23
 +
+

*Thread Reply:* 3rd stupid question of the night +Sorry kept on trying POST who knows why

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2023-02-16 19:26:56
 +
+

*Thread Reply:* no worries! keep the questions coming!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2023-02-16 19:29:46
 +
+

*Thread Reply:* well, maybe because it’s so late on your end! get some rest!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
thebruuu + (bruno.c@inwind.it) +
+
2023-02-16 19:36:25
 +
+

*Thread Reply:* Yeah but I want to see how it works +Right now I have a response 200 for the creation of the names ... but it seems that nothing occurred +nor on marquez front end (localhost:3000) +nor on the database

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2023-02-16 19:37:13
 +
+

*Thread Reply:* can you curl the list namespaces endpoint?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
thebruuu + (bruno.c@inwind.it) +
+
2023-02-16 19:38:14
 +
+

*Thread Reply:* yep : nothing changed +only default and food_delivery

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2023-02-16 19:38:47
 +
+

*Thread Reply:* can you post your server logs? you should see the request

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
thebruuu + (bruno.c@inwind.it) +
+
2023-02-16 19:40:41
 +
+

*Thread Reply:* marquez-api | XXX.23.0.4 - - [17/Feb/2023:00:30:38 +0000] "PUT /api/v1/namespaces/ciro HTTP/1.1" 500 110 "-" "-" 7 +marquez-api | INFO [2023-02-17 00:32:07,072] marquez.logging.LoggingMdcFilter: status: 200

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2023-02-16 19:41:12
 +
+

*Thread Reply:* the server is returning a 500 ?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2023-02-16 19:41:57
 +
+

*Thread Reply:* odd that LoggingMdcFilter is logging 200

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
thebruuu + (bruno.c@inwind.it) +
+
2023-02-16 19:43:24
 +
+

*Thread Reply:* Bit confused because now I realize that postman is returning bad request

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
thebruuu + (bruno.c@inwind.it) +
+
2023-02-16 19:43:51
 +
+

*Thread Reply:*

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
thebruuu + (bruno.c@inwind.it) +
+
2023-02-16 19:44:30
 +
+

*Thread Reply:* You'll notice that I go to use 3000 in the url +If I use 5000 I get No host

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2023-02-17 01:14:50
 +
+

*Thread Reply:* odd, the API should be using port 5000, have you followed our quickstart for Marquez?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
thebruuu + (bruno.c@inwind.it) +
+
2023-02-17 03:43:29
 +
+

*Thread Reply:* Hello Willy +I am starting from scratch followin instruction from https://openlineage.io/docs/getting-started/ +I am on Windows +Instead of +git clone git@github.com:MarquezProject/marquez.git && cd marquez +I run the +git clone

+ +

git clone <https://github.com/MarquezProject/marquez.git> +But before I had to clear the auto carriage return in git +git config --global core.autocrlf false +This avoid an error message on marquez-api when running wait-for-it.sh àt line 1 where +#!/usr/bin/env bash +is otherwise read as +#!/usr/bin/env bash\r'

+ +

It turns out that when switching off the autocr, this impacts some file containing marquez password ... and I get a fail on accessing the db +to overcome this I run notepad++ and replaced ALL the \r\n with \n +And in this way I managed to run +docker\up.sh and docker\down.sh +correctly (with or without seed ... with access to the db, via pgadmin)

+ + + +
+ 👍 Ernie Ostic +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
thebruuu + (bruno.c@inwind.it) +
+
2023-02-20 03:40:48
 +
+

*Thread Reply:* The issue is related to PostMan

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-02-17 03:39:07
 +
+

Hi, I'd like to capture column lineage from spark, but also capture how the columns are transformed, and any column operations that are done too. May I ask if this feature is supported currently, or will be supported in future based on current timeline? Thanks!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-02-17 03:54:47
 +
+

*Thread Reply:* Hi @Anirudh Shrinivason, this is a great question. We included extra fields in OpenLineage spec to contain that information: +"transformationDescription": { + "type": "string", + "description": "a string representation of the transformation applied" +}, +"transformationType": { + "type": "string", + "description": "IDENTITY|MASKED reflects a clearly defined behavior. IDENTITY: exact same as input; MASKED: no original data available (like a hash of PII for example)" +} +so the standard is ready to support it. We included two fields, so that one can contain human readable description of what is happening. However, we don't have this implemented in Spark integration.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-02-17 04:02:30
 +
+

*Thread Reply:* Thanks a lot! That is great. Is there a potential plan in the roadmap to support this for spark?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-02-17 04:08:16
 +
+

*Thread Reply:* I think there will be a growing interest in that. In general a dependency may really difficult to express if many Spark operators are used on input columns to produce output one. The simple version would be just to detect indetity operation or some kind of hashing.

+ +

To sum up, we don't have yet a proposal on that but this seems to be a natural next step in enriching column lineage features.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-02-17 04:40:04
 +
+

*Thread Reply:* Got it. Thanks! If this item potentially comes on the roadmap, then I'd be happy to work with other interested developers to help contribute! 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-02-17 04:43:00
 +
+

*Thread Reply:* Great to hear that. What you could perhaps start with, is come to our monthly OpenLineage meetings and ask @Michael Robinson to put this item on discussions' list. There are many strategies to address this issue and hearing your story, usage scenario and would are you trying to achieve, would be super helpful in design and implementation phase.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-02-17 04:44:18
 +
+

*Thread Reply:* Got it! The monthly meeting might be a bit hard for me to attend live, because of the time zone. But I'll try my best to make it to the next one! thanks!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-02-17 09:46:22
 +
+

*Thread Reply:* Thank you for bringing this up, @Anirudh Shrinivason. I’ll add it to the agenda of our next meeting because there might be interest from others in adding this to the roadmap.

+ + + +
+ 👍 Anirudh Shrinivason +
+ +
+ :gratitude_thank_you: Anirudh Shrinivason +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
thebruuu + (bruno.c@inwind.it) +
+
2023-02-17 15:12:57
 +
+

Hello +how can I improve the verbosity of the marquez-api? +Regards

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-02-20 02:10:13
 +
+

*Thread Reply:* Hi @thebruuu, pls take a look at logging documentation of Dropwizard (https://www.dropwizard.io/en/latest/manual/core.html#logging) - the framework Marquez is implemented in. The logging configuration section is present in marquez.yml .

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
thebruuu + (bruno.c@inwind.it) +
+
2023-02-20 03:29:07
 +
+

*Thread Reply:* Thank You Pavel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-02-21 02:23:40
 +
+

Hey, can we please schedule a release of OpenLineage? I would like to have the release that includes the feature to capture custom env variables from spark clusters... Thanks!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-02-21 09:12:17
 +
+

*Thread Reply:* We generally schedule a release every month, next one will be in the next week - is that okay @Anirudh Shrinivason?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-02-21 11:38:50
 +
+

*Thread Reply:* Yes, there’s one scheduled for next Wednesday, if that suits.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-02-21 21:45:58
 +
+

*Thread Reply:* Okay yeah sure that works. Thanks

+ + + +
+ 🙌 Michael Robinson +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-03-01 10:12:45
 +
+

*Thread Reply:* @Anirudh Shrinivason we’re expecting the release to happen today or tomorrow, FYI

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-03-01 21:22:40
 +
+

*Thread Reply:* Awesome thanks

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jingyi Chen + (jingyi@cloudshuttle.com.au) +
+
2023-02-23 23:43:23
 +
+

Hello team, we used OpenLineage and Great Expectations integrated. I want to use GE to verify the table in Snowflake. I found that the configuration I added OpenLineage into GE produced this error after running. Could someone please give me some answers? 👀 +File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/great_expectations/validation_operators/validation_operators.py", line 469, in _run_actions + action_result = self.actions[action["name"]].run( + File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/great_expectations/checkpoint/actions.py", line 106, in run + return self._run( + File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/openlineage/common/provider/great_expectations/action.py", line 156, in _run + datasets = self._fetch_datasets_from_sql_source( + File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/openlineage/common/provider/great_expectations/action.py", line 362, in _fetch_datasets_from_sql_source + self._get_sql_table( + File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/openlineage/common/provider/great_expectations/action.py", line 395, in _get_sql_table + if engine.connection_string: +AttributeError: 'Engine' object has no attribute 'connection_string' +'Engine' object has no attribute 'connection_string'

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jingyi Chen + (jingyi@cloudshuttle.com.au) +
+
2023-02-23 23:44:03
 +
+

*Thread Reply:* This is my checkponit configuration in GE. +```name: 'openlineagecheckpoint' +configversion: 1.0 +templatename: +modulename: greatexpectations.checkpoint +classname: Checkpoint +runnametemplate: '%Y%m%d-%H%M%S-mycheckpoint' +expectationsuitename: EMAILVALIDATION +batchrequest: +actionlist:

  • name: storevalidationresult +action: + class_name: StoreValidationResultAction
  • name: storeevaluationparams +action: + class_name: StoreEvaluationParametersAction
  • name: updatedatadocs +action: + classname: UpdateDataDocsAction + sitenames: []
  • name: openlineage +action: + classname: OpenLineageValidationAction + modulename: openlineage.common.provider.greatexpectations + openlineagehost: http://localhost:5000 + # openlineageapiKey: 12345 + openlineagenamespace: geexpectations # Replace with your job namespace; we recommend a meaningful namespace like dev or prod, etc. + jobname: gevalidation +evaluationparameters: {} +runtime_configuration: {} +validations:
  • batchrequest: + datasourcename: LANDINGDEV + dataconnectorname: defaultinferreddataconnectorname + dataassetname: 'snowpipe.pii' + dataconnectorquery: + index: -1 +expectationsuitename: EMAILVALIDATION
    • +
+ +

profilers: [] +gecloudid: +expectationsuitegecloudid:```

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Benji Lampel + (benjamin@astronomer.io) +
+
2023-02-24 11:31:05
 +
+

*Thread Reply:* What version of GX are you running? And is this being run directly through GX or through Airflow with the operator?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jingyi Chen + (jingyi@cloudshuttle.com.au) +
+
2023-02-26 20:05:12
 +
+

*Thread Reply:* I use the latest version of Great Expectations. This error occurs either directly through Great Expectations or airflow

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Benji Lampel + (benjamin@astronomer.io) +
+
2023-02-27 09:10:00
 +
+

*Thread Reply:* I noticed another issue in the latest version as well. Try dropping to GE version great-expectations==0.15.44 for now. That is the latest one that works for me.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Benji Lampel + (benjamin@astronomer.io) +
+
2023-02-27 09:11:34
 +
+

*Thread Reply:* You should definitely open an issue here, and you can tag me @denimalpaca in the comment

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jingyi Chen + (jingyi@cloudshuttle.com.au) +
+
2023-02-27 18:07:29
 +
+

*Thread Reply:* Thanks Benji, but I still have the same problem after I drop to great-expectations==0.15.44 , this is my requirement file

+ +
great_expectations==0.15.44
+sqlalchemy
+psycopg2-binary
+numpy
+pandas
+snowflake-connector-python
+snowflake-sqlalchem
+
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Benji Lampel + (benjamin@astronomer.io) +
+
2023-02-28 13:34:03
 +
+

*Thread Reply:* interesting... I do think this may be a GX issue so let's see if they say anything. I can also cross post this thread to their slack

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Saravanan + (saravanan@athivatech.com) +
+
2023-03-01 00:27:30
 +
+

Hello Team, I’m trying to use Open Lineage with AWS Glue and Marquez. Has anyone successfully integrated AWS Workflows/ Glue ETL jobs with Open Lineage?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sheeri Cabral (Collibra) + (sheeri.cabral@collibra.com) +
+
2023-05-01 11:47:40
 +
+

*Thread Reply:* I know I’m responding to an older post - I’m not sure if this would work in your environment? https://aws.amazon.com/blogs/big-data/build-data-lineage-for-data-lakes-using-aws-glue-amazon-neptune-and-spline/ +Are you using AWS Glue with Spark jobs?

+
+
Amazon Web Services
+ + + +
Build data lineage for data lakes using AWS Glue, Amazon Neptune, and Spline | Amazon Web Services
+
+ Data lineage is one of the most critical components of a data governance strategy for data lakes. Data lineage helps ensure that accurate, complete and trustworthy data is being used to drive business decisions. While a data catalog provides metadata management features and search capabilities, data lineage shows the full context of your data by […] +
+ + + + + + + + +
Original URL: https://aws.amazon.com/blogs/big-data/build-data-lineage-for-data-lakes-using-aws-glue-amazon-neptune-and-spline/
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Saravanan + (saravanan@athivatech.com) +
+
2023-05-02 15:16:14
 +
+

*Thread Reply:* This was proposed by our AWS Solution architect but we are not seeing much improvement compared to open lineage. Have you deployed the above solution to prod?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sheeri Cabral (Collibra) + (sheeri.cabral@collibra.com) +
+
2023-05-25 11:30:44
 +
+

*Thread Reply:* We are currently in the research phase, so we have not deployed to prod. We have customers with thousands of existing scripts that they don’t want to rewrite to add openlineage libraries - i would imagine that if you are already integrating OpenLineage in your code, the spark listener isn’t an improvement. Our research is on magically getting lineage from existing scripts 😄

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-03-01 09:42:23
 +
+

Hello everyone, I’m opening a vote to release OpenLineage 0.21.0, featuring: +• a new CustomEnvironmentFacetBuilder class and new output visitors AlterTableAddPartitionCommandVisitor and AlterTableSetLocationCommandVisitor in the Spark integration +• a Linux-ARM version of the SQL parser’s native library +• DEBUG logging of events in transports +• bug fixes and more. +Three +1s from committers will authorize an immediate release.

+ + + +
+ ➕ Maciej Obuchowski, Jakub Dardziński, Benji Lampel, Natalie Zeller, Paweł Leszczyński +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-03-01 10:26:22
 +
+

*Thread Reply:* Thanks, all. The release is authorized and will be initiated as soon as possible.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Nigel Jones + (nigel.l.jones@gmail.com) +
+
2023-03-02 03:52:03
 +
+

I’ve got some security related questions/observations. The main site suggests opening an issue to report vulnerabilities etc. I wanted to check if there is a private mailing list/DM channel to just check a few things first? I’m happy to use github issues otherwise. Thanks!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Moritz E. Beber + (midnighter@posteo.net) +
+
2023-03-02 05:15:55
 +
+

*Thread Reply:* GitHub has a new issue template for reporting vulnerabilities, actually. If you use a config that enables this issue template.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-03-02 10:21:16
 +
+

Reminder: our first meetup is one week from today in Providence, RI! You can find the details in the meetup blog post. And if you’re coming, it would be great if you could RSVP. Looking forward to seeing some of you there!

+
+
openlineage.io
+ + + +
Happening Soon - Our First Meetup!
+
+ The inaugural Data Lineage Meetup will take place on March 9th in Providence, RI. +
+ + + + + + +
Original URL: https://openlineage.io/blog/data-lineage-meetup/
+ + + +
+
+
Meetup
+ + + +
Data Lineage Meetup, Thu, Mar 9, 2023, 6:00 PM | Meetup
+
+ Join us on Thursday, March 9, 2023 from 6-8 pm in Providence, Rhode Island to learn more about the present and future of OpenLineage. Meet other members of the ecosystem, l +
+ + + + + + + + +
Original URL: https://www.meetup.com/providence-data-lineage-meetup/events/291847062/
+ + + +
+ + + +
+ 🎉 Kengo Seki +
+ +
+ 🚀 Kengo Seki +
+ +
+ ✅ Sheeri Cabral (Collibra) +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-03-02 16:52:50
 +
+

@channel +We released OpenLineage 0.21.1, including: +Additions +• Clients: add DEBUG logging of events to transports #1633 by @mobuchowski +• Spark: add CustomEnvironmentFacetBuilder class #1545 by New contributor @Anirudh181001 +• Spark: introduce the new output visitors AlterTableAddPartitionCommandVisitor and AlterTableSetLocationCommandVisitor #1629 by New contributor @nataliezeller1 +• Spark: add column lineage for JDBC relations #1636 by @tnazarew +• SQL: add linux-aarch64 native library to Java SQL parser #1664 by @mobuchowski +Changes +• Airflow: get table database in Athena extractor #1631 by New contributor @rinzool +Removals +• Airflow: remove JobIdMapping and update macros to better support Airflow version 2+ #1645 by @JDarDagran +Thanks to all our contributors! +For the bug fixes and details, see: +Release: https://github.com/OpenLineage/OpenLineage/releases/tag/0.21.1 +Changelog: https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md +Commit history: https://github.com/OpenLineage/OpenLineage/compare/0.20.6...0.21.1 +Maven: https://oss.sonatype.org/#nexus-search;quick~openlineage +PyPI: https://pypi.org/project/openlineage-python/

+ + + +
+ 🎉 Kengo Seki, Harel Shein, Maciej Obuchowski +
+ +
+ 🚀 Kengo Seki, Harel Shein, Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paul Lee + (paullee@lyft.com) +
+
2023-03-02 19:01:23
 +
+

how do you turn off the openlineage listener in airflow 2? for some reason we're seeing a Thread-2 and seeing it fire twice in tasks

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harel Shein + (harel.shein@gmail.com) +
+
2023-03-02 20:04:19
 +
+

*Thread Reply:* Hey @Paul Lee, are you seeing this happen for Async operators?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harel Shein + (harel.shein@gmail.com) +
+
2023-03-02 20:06:00
 +
+

*Thread Reply:* might be related to this issue https://github.com/OpenLineage/OpenLineage/pull/1601 +that was fixed in 0.20.6

+
+ + + + +
#1601 Airflow: make extractor for async operators work
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paul Lee + (paullee@lyft.com) +
+
2023-03-03 16:15:44
 +
+

*Thread Reply:* hmm perhaps.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paul Lee + (paullee@lyft.com) +
+
2023-03-03 16:15:55
 +
+

*Thread Reply:* @Harel Shein if i want to turn off openlineage listener how do i do that? do i just remove the package?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harel Shein + (harel.shein@gmail.com) +
+
2023-03-03 16:24:07
 +
+

*Thread Reply:* meaning, you don’t want openlineage to collect any information from your Airflow deployment?

+ + + +
+ 👍 Paul Lee +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harel Shein + (harel.shein@gmail.com) +
+
2023-03-03 16:24:50
 +
+

*Thread Reply:* in that case, you could either remove it from your requirements file, or set OPENLINEAGE_DISABLED=True in your Airflow env vars

+ + + +
+ 👍 Paul Lee +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paul Lee + (paullee@lyft.com) +
+
2023-03-06 14:43:56
 +
+

*Thread Reply:* removed it from requirements and also the backend key in airflow config. needed both

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-03-02 20:29:42
 +
+

@channel +This month’s OpenLineage TSC meeting is next Thursday, March 9th, at 10 am PT. Join us on Zoom: https://bit.ly/OLzoom. All are welcome! +On the tentative agenda:

+ +
  1. Recent release overview
  2. A new consumer
  3. Custom env variable support in Spark
  4. Async operator support in Airflow
  5. JDBC relations support in Spark
  6. Discussion topics: +• New feature idea: column transformations/operations in the Spark integration +• Using namespaces
  7. Open discussion +Notes: https://bit.ly/OLwiki +Is there a topic you think the community should discuss at this or a future meeting? Reply or DM me to add items to the agenda.
    8. +
+ + + +
+ 🙌 Willy Lulciuc, Paweł Leszczyński, Maciej Obuchowski, alexandre bergere +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-03-02 21:48:29
 +
+

Hi everyone, I noticed that Openlineage is sending each of the events twice for spark. Is this expected? Is there some way to disable this behaviour?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Will Johnson + (will@willj.co) +
+
2023-03-02 23:46:08
 +
+

*Thread Reply:* Are you seeing duplicate START events or do you see two events one that is a START and one that is COMPLETE?

+ +

OpenLineage's events may send partial information. You should expect to collect all events for a given RunId and merge them together to get the complete events.

+ +

In addition, some data sources are really chatty like Delta tables. That may cause you to see many events that look very similar.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-03-03 00:45:19
 +
+

*Thread Reply:* Hmm...I'm seeing 2 start events for the same runnable command

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-03-03 00:45:27
 +
+

*Thread Reply:* And 2 complete

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-03-03 00:46:08
 +
+

*Thread Reply:* I am currently only testing on parquet tables...

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-03-03 02:31:28
 +
+

*Thread Reply:* One of openlineage assumptions is the ability to merge lineage events in the backend to make client integrations stateless. So, it is possible that Spark can emit multiple events for the same job. However, sometimes it does not make any sense to send or collect some events, which happened to us some time ago with delta. In that case we decided to filter them and created filtering mechanism (https://github.com/OpenLineage/OpenLineage/tree/main/integration/spark/shared/src/main/java/io/openlineage/spark/agent/filters) than can be extended in case of other unwanted events being generated and sent.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-03-05 22:59:06
 +
+

*Thread Reply:* Ahh I see...okay thanks!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Daniel Joanes + (djoanes@gmail.com) +
+
2023-03-07 00:05:48
 +
+

*Thread Reply:* in general , you should build any event consumer system with at least once semantics. Even if this issue is fixed, there is a possibility of duplicates for other valid scenarios

+ + + +
+ ➕ Maciej Obuchowski, Anirudh Shrinivason +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-03-09 14:10:47
 +
+

*Thread Reply:* Hi..I compared some duplicate 'START' events just now, and noticed that they are exactly the same, with the only exception of one of them having an 'environment-properties' field... Could I just quickly check if this is a bug or a feature haha?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-03-10 01:18:18
 +
+

*Thread Reply:* CC: @Paweł Leszczyński ^

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-03-08 11:15:48
 +
+

@channel +Reminder: this month’s OpenLineage TSC meeting is tomorrow at 10am PT. All are welcome. https://openlineage.slack.com/archives/C01CK9T7HKR/p1677806982084969

+
+ + +
+ + + } + + Michael Robinson + (https://openlineage.slack.com/team/U02LXF3HUN7) +
+ + + +
+
+ <p><strong>@channel</strong> +This month’s OpenLineage TSC meeting is <strong>next Thursday, March 9th, at 10 am PT</strong>. Join us on Zoom: <a href="https://bit.ly/OLzoom">https://bit.ly/OLzoom</a>. All are welcome! +On the tentative agenda:</p> + +<ol><li>Recent release overview </li><li>A new consumer</li><li>Discussion topics: +• New feature idea: column transformations/operations in the Spark integration +• Using namespaces</li><li>Open discussion +Notes: <a href="https://bit.ly/OLwiki">https://bit.ly/OLwiki</a> +Is there a topic you think the community should discuss at this or a future meeting? Reply or DM me to add items to the agenda.</li> +</ol> +
+ + + + + + +
Original URL: https://openlineage.slack.com/archives/C01CK9T7HKR/p1677806982084969
+ + +
+ + Slack Conversation +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Susmitha Anandarao + (susmitha.anandarao@gmail.com) +
+
2023-03-08 15:51:07
 +
+

Hi if we have OpenLineage listener configured as a default spark conf, is there an easy way to disable ol for a specific notebook?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-03-08 17:30:44
 +
+

*Thread Reply:* if you can set up env variables for particular notebooks, you can set OPENLINEAGE_DISABLED=true

+ + + +
+ :gratitude_thank_you: Susmitha Anandarao +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Benji Lampel + (benjamin@astronomer.io) +
+
2023-03-10 13:15:41
 +
+

Hey all,

+ +

I opened a PR (and corresponding issue) to change how naming works in OpenLineage. The idea generally is to move from Naming.md as the end-all-be-all of names for integrations, and towards JSON schemas per integration, with each schema defining very precisely what fields a name and namespace should contain, how they're connected, and how they're validated. Would really appreciate some feedback as this is a pretty big change!

+
+ + + + +
#1702 Add proposal for new naming schemas.
+
+ <p>Closes: <a href="https://github.com/OpenLineage/OpenLineage/issues/1681">#1681</a></p> + +<p><strong>Checklist</strong></p> + +<p>☑︎ You've <a href="https://github.com/OpenLineage/OpenLineage/blob/main/why-the-dco.md">signed-off</a> your work +☑︎ Your pull request title follows our <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CONTRIBUTING.md#creating-pull-requests">guidelines</a> +☐ Your changes are accompanied by tests (<em>if relevant</em>) +☐ Your change contains a <a href="https://kurtisnusbaum.medium.com/stacked-diffs-keeping-phabricator-diffs-small-d9964f4dcfa6">small diff</a> and is self-contained +☐ You've updated any relevant documentation (<em>if relevant</em>) +☐ Your comment includes a one-liner for the changelog about the specific purpose of the change (<em>if necessary</em>) +☐ You've versioned the core OpenLineage model or facets according to <a href="https://docs.snowplowanalytics.com/docs/pipeline-components-and-applications/iglu/common-architecture/schemaver">SchemaVer</a> (<em>if relevant</em>) +☐ You've added a <a href="https://github.com/OpenLineage/OpenLineage/tree/main/.github/header_templates.md">header</a> to source files (<em>if relevant</em>)</p> + +<hr /> + +<p>SPDX-License-Identifier: Apache-2.0 <br /> +Copyright 2018-2023 contributors to the OpenLineage project</p> +
+ +
+
Labels
+ documentation, proposal +
+ +
+
Comments
+ 1 +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sunil Patil + (spatil@twilio.com) +
+
2023-03-13 17:05:56
 +
+

What do i need to do to enable dag level metric capturing for airflow. I followed the instruction to install openlineage 0.21.1 on airflow 2.3.3. When i run a DAG i see metrics related to Task start, success/failure. But i dont see any metrics for Dag success/failure. Do i have to do something to enable DAG execution capturing ?

+
+
openlineage.io
+ + + +
Using OpenLineage with Airflow | OpenLineage Docs
+
+ This tutorial introduces you to using OpenLineage with Airflow. OpenLineage has various integrations that will enable airflow to emit OpenLineage events when using Airflow Integrations. In this tutorial, you will be running a local instance of Airflow using docker compose, and learn how to enable and setup OpenLineage to emit data lineage events. The tutorial will use two backends to check the data lineage, 1) using OpenLineage Proxy, and 2) using Marquez. +
+ + + + + + +
Original URL: https://openlineage.io/docs/guides/airflow
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sunil Patil + (spatil@twilio.com) +
+
2023-03-13 17:08:53
 +
+

*Thread Reply:* is DAG run capturing enabled starting airflow 2.5.1 ? https://github.com/apache/airflow/pull/27113

+
+ + + + +
#27113 notification: add dag run state notification system
+
+ <p>This PR expands <code>listeners</code> API to be notified about DagRun state changes.</p> + +<p>PR <a href="https://github.com/apache/airflow/pull/20443">#20443</a> introduced <code>Listener API</code> to Airflow, enabling users and developers to write <code>Pluggy</code> plugins. Here the same mechanism is used to provide notifications about <code>DagRun</code> states - whether <code>DagRun</code> started, succeeded or failed.</p> + +<p>Additionally, this PR adds lifecycle methods for plugins - when Airflow <code>Job</code> - whether <code>SchedulerJob</code>, <code>BackfillJob</code> or <code>LocalTaskJob</code> runs, it notifies plugin that it starts or finishes.</p> + +<p>Signed-off-by: Maciej Obuchowski <a href="mailto:obuchowski.maciej@gmail.com">obuchowski.maciej@gmail.com</a></p> +
+ +
+
Labels
+ area:scheduler/executor, type:new-feature +
+ +
+
Comments
+ 6 +
+ + + + + + + +
+ + <a href="https://github.com/apache/airflow">apache/airflow</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-03-13 17:11:47
 +
+

*Thread Reply:* you're right, only the change was included in 2.5.0

+ + + +
+ 🙏 Sunil Patil +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sunil Patil + (spatil@twilio.com) +
+
2023-03-13 17:43:15
 +
+

*Thread Reply:* Thanks Jakub

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-03-14 15:37:34
 +
+

Fresh on the heels of our first-ever in-person event, we’re meeting up again soon at Data Council Austin! Join us on March 30th (the same day as @Julien Le Dem’s talk) at 12:15 pm to discuss the project’s goals and design, meet other members of the data ecosystem, and help shape the future of the spec. For more info, check out the OpenLineage blog. If you haven’t registered for the conference yet, click and use promo code OpenLineage20 for a special rate. Hope to see you there!

+
+
openlineage.io
+ + + +
Meet Us at Data Council Austin
+
+ The OpenLineage community will be gathering on March 30th at Data Council Austin -- join us! +
+ + + + + + +
Original URL: https://openlineage.io/blog/data-council-meetup/
+ + + +
+
+
tickettailor.com
+ + + +
Buy tickets – Data Council Austin 2023 – AT&amp;T Hotel and Conference Center, Tue Mar 28, 2023 8:00 AM - Thu Mar 30, 2023 5:00 PM
+
+ Data Council Austin 2023 – AT&amp;T Hotel and Conference Center, Tue Mar 28, 2023 - Thu Mar 30, 2023 - Data Council US 2023 in Austin, TX is where data's most influential makers, doers, and thinkers gather to shape the future of data innovation at every layer of the stack. If you’re a technical professional who wants to learn about new tools... +
+ + + + + + + + +
Original URL: https://www.tickettailor.com/events/datacouncil/747883?hsCtaTracking=dc0af239-1ec5-4c9f-a175-f487bb858074%7Ca9aa7625-43fb-436d-b05d-9666bab6e414&amp;_ga=2.239870401.316757827.1677608801-1529956930.1676647160
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sheeri Cabral (Collibra) + (sheeri.cabral@collibra.com) +
+
2023-03-15 15:11:18
 +
+

If someone is using airflow and DAG-docs for lineage, can they export the lineage in, say, OL format?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harel Shein + (harel.shein@gmail.com) +
+
2023-03-15 15:18:22
 +
+

*Thread Reply:* I don’t see it currently on the AirflowRunFacet, but probably not a big deal to add it? @Benji Lampel wdyt?

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/integration/airflow/facets/AirflowRunFacet.json | AirflowRunFacet.json>
+
+ <pre><code>// TODO: that is Airflow specific facet that should be placed in new system-specific directory +{ + "$schema": "&lt;https://json-schema.org/draft/2020-12/schema&gt;", + "$id": "&lt;https://openlineage.io/spec/facets/1-0-0/AirflowRunFacet.json&gt;", + "$defs": { + "AirflowRunFacet": { + "allOf": [ + { + "$ref": "&lt;https://openlineage.io/spec/1-0-3/OpenLineage.json#/$defs/RunFacet&gt;" + }, + { + "type": "object", + "properties": { + "task": { + "$ref": "#/$defs/Task" + }, + "dag": { + "$ref": "#/$defs/DAG" + }, + "taskInstance": { + "$ref": "#/$defs/TaskInstance" + }, + "dagRun": { + "$ref": "#/$defs/DagRun" + }, + "taskUuid": { + "type": "string" + } + }, + "required": ["task", "dag", "taskInstance", "dagRun", "taskUuid"] + } + ] + }, + "Task": { + "type": "object", + "description": "An object containing task-related attributes", + "properties": { + "operator_class": { + "description": "Module + class name of the operator", + "type": "string" + }, + "email": { + "type": "string" + }, + "args": { + "type": "object", + "additionalProperties": true + }, + "downstream_group_ids": { + "type": "string" + }, + "downstream_task_ids": { + "type": "string" + }, + "upstream_group_ids": { + "type": "string" + }, + "upstream_task_ids": { + "type": "string" + }, + "task_group": { + "description": "Task group related information", + "type": "object", + "properties": { + "children": { + "type": "object", + "additionalProperties": true + }, + "downstream_group_ids": { + "type": "string" + }, + "downstream_task_ids": { + "type": "string" + }, + "prefix_group_id": { + "type": "boolean" + }, + "tooltip": { + "type": "string" + }, + "upstream_group_ids": { + "type": "string" + }, + "upstream_task_ids": { + "type": "string" + } + }, + "additionalProperties": true + } + }, + "required": ["operator_class"] + }, + "DAG": { + "type": "object", + "description": "A DAG-attributes representing object", + "properties": { + "dag_id": { + "type": "string" + }, + "schedule_interval": { + "type": "string" + }, + "tags": { + "type": "string" + }, + "start_date": { + "type": "string", + "format": "date-time" + }, + "timetable": { + "description": "Describes timetable (successor of schedule_interval)", + "type": "object", + "additionalProperties": true + } + }, + "additionalProperties": true, + "required": ["dag_id", "schedule_interval", "tags", "start_date"] + }, + "TaskInstance": { + "type": "object", + "properties": { + "duration": { + "type": "number" + }, + "try_number": { + "type": "integer" + }, + "map_index": { + "type": "integer" + } + }, + "additionalProperties": true, + "required": ["duration", "try_number"] + }, + "DagRun": { + "type": "object", + "properties": { + "conf": { + "type": "object", + "additionalProperties": true + }, + "dag_id": { + "type": "string" + }, + "data_interval_start": { + "type": "string", + "format": "date-time" + }, + "data_interval_end": { + "type": "string", + "format": "date-time" + }, + "run_id": { + "type": "string" + }, + "run_type": { + "type": "string" + }, + "external_trigger": { + "type": "boolean" + } + }, + "additionalProperties": true, + "required": ["dag_id", "external_trigger"] + } + }, + "type": "object", + "properties": { + "airflow": { + "$ref": "#/$defs/AirflowRunFacet" + } + } +} + +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+ ❤️ Sheeri Cabral (Collibra) +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Benji Lampel + (benjamin@astronomer.io) +
+
2023-03-15 15:22:00
 +
+

*Thread Reply:* Definitely could be a good thing to have--is there not some info facet that could hold this data already? I don't see an issue with adding to the AirflowRunFacet tho (full disclosure, I'm not super familiar with this facet)

+ + + +
+ ❤️ Sheeri Cabral (Collibra) +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2023-03-15 15:58:40
 +
+

*Thread Reply:* Perhaps DocumentationJobFacet or DocumentationDatasetFacet?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sheeri Cabral (Collibra) + (sheeri.cabral@collibra.com) +
+
2023-03-15 15:13:55
 +
+

(is it https://docs.astronomer.io/learn/airflow-openlineage ? )

+
+
docs.astronomer.io
+ + + +
Integrate OpenLineage and Airflow | Astronomer Documentation
+
+ Learn about OpenLineage concepts and benefits of integrating with Airflow. +
+ + + + + + + + +
Original URL: https://docs.astronomer.io/learn/airflow-openlineage
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Susmitha Anandarao + (susmitha.anandarao@gmail.com) +
+
2023-03-17 12:31:02
 +
+

Happy Friday 👋 I am looking for some help setting the parent information for a dbt run. I have set the namespace variable in the openlineage.yml but doesn't seem to take effect and ends up using the default value of dbt. Also using openlineage.yml to set the transport properties for emitting to kafka. Is there a way to set parent namespace, name and run id in the yml file? Thank you!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-03-18 12:09:23
 +
+

*Thread Reply:* dbt-ol does not read from openlineage.yml so you need to pass this information in OPENLINEAGE_NAMESPACE environment variable

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2023-03-20 15:17:03
 +
+

*Thread Reply:* Hmmm. Interesting! I thought that it used client = OpenLineageClient.from_environment(), I’ll do some testing with Kafka backends.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Susmitha Anandarao + (susmitha.anandarao@gmail.com) +
+
2023-03-20 15:22:07
 +
+

*Thread Reply:* Thank you for the hint. I was able to make it work with specifying the env OPENLINEAGE_CONFIGto specify the yml file holding transport info and OPENLINEAGE_NAMESPACE

+ + + +
+ 👍 Ross Turk +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2023-03-20 15:24:05
 +
+

*Thread Reply:* Awesome! That’s exactly what I was going to test.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2023-03-20 15:25:04
 +
+

*Thread Reply:* I think it also works if you put it in $HOME/.openlineage/openlineage.yml.

+ + + +
+ :gratitude_thank_you: Susmitha Anandarao +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-03-21 08:32:17
 +
+

*Thread Reply:* @Susmitha Anandarao I might have provided misleading information. I meant that dbt-ol does not read OL namespace from openlineage.yml but from OPENLINEAGE_NAMESPACE env var instead

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-03-21 13:48:28
 +
+

Data Council Austin, the host of our next meetup, is one week away: https://openlineage.slack.com/archives/C01CK9T7HKR/p1678822654288379

+
+ + +
+ + + } + + Michael Robinson + (https://openlineage.slack.com/team/U02LXF3HUN7) +
+ + + +
+
+ <p>Fresh on the heels of our first-ever in-person event, we’re meeting up again soon at <a href="https://www.datacouncil.ai/austin">Data Council Austin</a>! Join us on March 30th (the same day as @Julien Le Dem’s <a href="https://www.datacouncil.ai/talks/ten-years-of-building-open-source-standards?hsLang=en">talk</a>) at 12:15 pm to discuss the project’s goals and design, meet other members of the data ecosystem, and help shape the future of the spec. For more info, check out the OpenLineage <a href="https://openlineage.io/blog/data-council-meetup/">blog</a>. If you haven’t registered for the conference yet, click <https://www.tickettailor.com/events/datacouncil/747883?hsCtaTracking=dc0af239-1ec5-4c9f-a175-f487bb858074%7Ca9aa7625-43fb-436d-b05d-9666bab6e414&amp;_ga=2.239870401.316757827.1677608801-1529956930.1676647160|this link> and use promo code <code>OpenLineage20</code> for a special rate. Hope to see you there!</p> +
+ + + + + + +
Original URL: https://openlineage.slack.com/archives/C01CK9T7HKR/p1678822654288379
+ + +
+ + Slack Conversation +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-03-21 13:52:52
 +
+

In addition to Data Council Austin next week, the hybrid Big Data Technology Warsaw Summit will be taking place on March 28th-30th, featuring three of our committers: @Maciej Obuchowski, @Paweł Leszczyński and @Ross Turk ! There’s more info here: https://bigdatatechwarsaw.eu/

+
+
Big Data Technology Warsaw Summit
+ + + +
BDTWS2023 - Big Data Technology Warsaw Summit
+
+ Big Data Tech Warsaw 2023 is one of the largest Big Data conferences in Poland and Eastern Europe +
+ +
+
Estimated reading time
+ 6 minutes +
+ + + + + + + + +
Original URL: https://bigdatatechwarsaw.eu/
+ + + +
+ + + +
+ 🙌 Howard Yoo, Maciej Obuchowski, Jakub Dardziński, Ross Turk, Perttu Salonen +
+ +
+ 👍 thebruuu +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Brad Paskewitz + (bradford.paskewitz@fivetran.com) +
+
2023-03-22 22:38:26
 +
+

hey folks, is anyone capturing dataset metadata for multi-table schemas? I'm looking at the schema dataset facet: https://openlineage.io/docs/spec/facets/dataset-facets/schema but it looks like this only represents a single table so im wondering if I'll need to write a custom facet

+
+
openlineage.io
+ + + +
Schema Dataset Facet | OpenLineage Docs
+
+ The schema dataset facet contains the schema of a particular dataset. +
+ + + + + + +
Original URL: https://openlineage.io/docs/spec/facets/dataset-facets/schema
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-03-23 04:25:19
 +
+

*Thread Reply:* It should be represented by multiple datasets, unless I misunderstood what you mean by multi-table

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Brad Paskewitz + (bradford.paskewitz@fivetran.com) +
+
2023-03-23 10:55:58
 +
+

*Thread Reply:* here at Fivetran when we sync data it is generally 1 schema with multiple tables (sometimes many) so we would want to represent all of that

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-03-23 11:11:25
 +
+

*Thread Reply:* So what I understand:

+ +
  1. your single job represents synchronization of multiple tables
  2. you want to have precise input-output dataset lineage? +am I right?
    3. +
+ +

I would model that as multiple OL jobs that describe each dataset mappings. Additionally, I'd have one "wrapping" job that represents your definition of a job. Rest of those jobs would refer to it in ParentRunFacet.

+ +

This is a pattern we use for Airflow and dbt dags.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Brad Paskewitz + (bradford.paskewitz@fivetran.com) +
+
2023-03-23 12:57:15
 +
+

*Thread Reply:* Yes your statements are correct. Thanks for sharing that model, that makes sense to me

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Brad Paskewitz + (bradford.paskewitz@fivetran.com) +
+
2023-03-24 15:56:27
 +
+

has anyone had success creating custom facets using java? I'm following this guide: https://openlineage.io/docs/spec/facets/custom-facets and im wondering if it makes sense to manually create POJOs or if others are creating the json schema for the object and then automatically generating the java code?

+
+
openlineage.io
+ + + +
Custom Facets | OpenLineage Docs
+
+ In addition to the existing facets mentioned in this documentation, users can extend the base facets and provide their own facet definition as part of the payload in OpenLineage event. For example, when OpenLineage event is emitted from the Apache Airflow using OpenLineage's Airflow integration, the following facets can be observed: +
+ + + + + + +
Original URL: https://openlineage.io/docs/spec/facets/custom-facets
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-03-27 05:26:06
 +
+

*Thread Reply:* I think it's better to just create POJO. This is what we do in Spark integration, for example.

+ +

For now, JSON Schema generator isn't flexible enough to generate custom facets from whatever schema we give it, so it would be unnecessary complexity

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/ce5874ca86a5101720d4cb3b9bb195463b13438a/integration/spark/shared/src/main/java/io/openlineage/spark/agent/facets/EnvironmentFacet.java | EnvironmentFacet.java>
+
+ <pre><code>public class EnvironmentFacet extends OpenLineage.DefaultRunFacet { +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+ :gratitude_thank_you: Brad Paskewitz +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2023-03-27 12:29:57
 +
+

*Thread Reply:* Agreed, just a POJO would work. This is using Jackson, so you would use annotations as needed. You can also use a Jackson JSONNode or even Map.

+ + + +
+ :gratitude_thank_you: Brad Paskewitz +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Brad Paskewitz + (bradford.paskewitz@fivetran.com) +
+
2023-03-27 14:01:07
 +
+

One other question: I'm in the process of adding different types of facets to our base payloads and I'm wondering if we have any related guidelines / best practices / standards / conventions. For example if I add a full source schema as a schema dataset facet to every start event it seems like that could be inefficient compared to a 1-time full-source-schema followed by incremental diffs for each following sync. Curious how others are thinking about + solving these types of problems in practice

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-03-27 17:59:28
 +
+

*Thread Reply:* That depends on the OL consumer, but for something like SchemaDatasetFacet it seems to be okay to assume schema stays the same if not send.

+ +

For others, like OutputStatisticsOutputDatasetFacet you definitely can't assume that, as the data is unique to each run.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Brad Paskewitz + (bradford.paskewitz@fivetran.com) +
+
2023-03-27 19:05:14
 +
+

*Thread Reply:* ok great thanks, that makes sense to me

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Saravanan + (saravanan@athivatech.com) +
+
2023-03-27 21:42:20
 +
+

Hi Team, I’m seeing creating data source, dataset API’s marked as deprecated . Can anyone point me how to create datasets via API calls?

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-03-28 04:47:31
 +
+

*Thread Reply:* OpenLineage API: https://openlineage.io/docs/getting-started/

+
+
openlineage.io
+ + + +
Getting Started | OpenLineage Docs
+
+ This guide covers how you can quickly get started collecting dataset, job, and run metadata using OpenLineage. We'll show how to collect run-level metadata as OpenLineage events using Marquez as the HTTP backend, then explore lineage metadata via the Marquez UI. +
+ + + + + + +
Original URL: https://openlineage.io/docs/getting-started/
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-03-28 06:08:18
 +
+

Hi everyone, I recently encountered this error saying V2SessionCatalog is not supported by openlineage. May I ask if support for this will be added in near future? Thanks!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-03-28 08:05:30
 +
+

*Thread Reply:* I think it would be great to support V2SessionCatalog, and it would very much help if you created GitHub issue with more explanation and examples of it's use.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-03-29 02:53:37
 +
+

*Thread Reply:* Sure thanks!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-03-29 05:34:37
 +
+

*Thread Reply:* https://github.com/OpenLineage/OpenLineage/issues/1747 +I have opened an issue here. Thanks! 🙂

+
+ + + + +
#1747 [PROPOSAL] Support for V2SessionCatalog
+
+ <p>Delta catalog uses V2SessionCatalog for some operations. All catalog calls except the delta lake ones are delegated to V2SessionCatalog. Currently, openlineage does not support this spark catalog. <br /> +Adding support for the V2SessionCatalog will be of great use as we are currently unable to emit metadata from some important pipelines which use this spark catalog, to our backend. <br /> +Wondering if this feature can be supported, or is on the roadmap potentially? Thanks a lot!</p> +
+ +
+
Labels
+ proposal +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-04-17 11:53:52
 +
+

*Thread Reply:* Hi @Maciej Obuchowski Just curious, is this issue on the potential roadmap for the next Openlineage release?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tom van Eijk + (t.m.h.vaneijk@tilburguniversity.edu) +
+
2023-04-02 19:37:27
 +
+

Hi all! Can anyone provide me some advice on how to solve this error: +ValueError: `emit` only accepts RunEvent class +[2023-04-02, 23:22:00 UTC] {taskinstance.py:1326} INFO - Marking task as FAILED. dag_id=etl_openlineage, task_id=send_ol_events, execution_date=20230402T232112, start_date=20230402T232114, end_date=20230402T232200 +[2023-04-02, 23:22:00 UTC] {standard_task_runner.py:105} ERROR - Failed to execute job 400 for task send_ol_events (`emit` only accepts RunEvent class; 28020) +[2023-04-02, 23:22:00 UTC] {local_task_job.py:212} INFO - Task exited with return code 1 +[2023-04-02, 23:22:00 UTC] {taskinstance.py:2585} INFO - 0 downstream tasks scheduled from follow-on schedule check +I'm trying to follow this tutorial (https://openlineage.io/blog/openlineage-snowflake/) on connecting Snowflake to OpenLineage through Apache Airflow, however, the last step (sending the OpenLineage events) returns an error.

+
+
openlineage.io
+ + + +
Data Lineage with Snowflake | OpenLineage Docs
+
+ The OpenLineage Adapter offers Snowflake's enterprise users a powerful tool for analyzing their pipelines. +
+ + + + + + +
Original URL: https://openlineage.io/blog/openlineage-snowflake/
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-04-03 09:32:46
 +
+

*Thread Reply:* The blog post is a bit old and in the meantime there were changes in OpenLineage Python Client introduced. +May I ask if you want just to test the flow or looking for any viable Snowflake data lineage solution?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2023-04-03 10:47:57
 +
+

*Thread Reply:* I believe that this will work if you change the line to client.transport.emit()

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2023-04-03 10:49:05
 +
+

*Thread Reply:* (this would be in the dags/lineage folder, if memory serves)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-04-03 10:57:23
 +
+

*Thread Reply:* Ross is right, that should work

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tom van Eijk + (t.m.h.vaneijk@tilburguniversity.edu) +
+
2023-04-04 12:23:13
 +
+

*Thread Reply:* This works! Thank you so much!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tom van Eijk + (t.m.h.vaneijk@tilburguniversity.edu) +
+
2023-04-04 12:24:40
 +
+

*Thread Reply:* @Jakub Dardziński I want to use a viable Snowflake data lineage solution alongside a Amazon DataZone Catalog 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2023-04-04 13:03:58
 +
+

*Thread Reply:* I have been meaning to revisit that tutorial 👍

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-04-03 10:52:42
 +
+

Hello all, +I’d like to open a vote to release OpenLineage 0.22.0, including: +• a new properties facet in the Spark integration +• a new field in HttpConfig for passing custom headers in the Spark integration +• improved namespace generation for JDBC connections in the Spark integration +• removal of unnecessary warnings about column lineage in the Spark integration +• support for alter, truncate, and drop statements in the SQL parser +• typing hints in the SQL integration +• a new from_dict class method in the Python client to support creating it from a dictionary +• a case-insensitive env variable for disabling OpenLineage in the Python client and Airflow integration +• bug fixes, docs changes, and more. +Three +1s from committers will authorize an immediate release. For more details about the release process, see GOVERNANCE.md.

+ + + +
+ ➕ Maciej Obuchowski, Perttu Salonen, Jakub Dardziński, Ross Turk +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-04-03 15:39:46
 +
+

*Thread Reply:* Thanks, all. The release is authorized and will be initiated within 48 hours.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-04-03 16:55:44
 +
+

@channel +We released OpenLineage 0.22.0, including: +Additions: +• Spark: add properties facet #1717 by @tnazarew +• SQL: SQLParser supports alter, truncate and drop statements #1695 by @pawel-big-lebowski +• Common/SQL: provide public interface for openlineage_sql package #1727 by @JDarDagran +• Java client: add configurable headers to HTTP transport #1718 by @tnazarew +• Python client: create client from dictionary #1745 by @JDarDagran +Changes: +• Spark: remove URL parameters for JDBC namespaces #1708 by @tnazarew +• Make OPENLINEAGE_DISABLED case-insensitive #1705 by @jedcunningham +Removals: +• Spark: remove unnecessary warnings for column lineage #1700 by @pawel-big-lebowski +• Spark: remove deprecated configs #1711 by @tnazarew +Thanks to all the contributors! +For the bug fixes and details, see: +Release: https://github.com/OpenLineage/OpenLineage/releases/tag/0.22.0 +Changelog: https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md +Commit history: https://github.com/OpenLineage/OpenLineage/compare/0.21.1...0.22.0 +Maven: https://oss.sonatype.org/#nexus-search;quick~openlineage +PyPI: https://pypi.org/project/openlineage-python/

+ + + +
+ 🙌 Jakub Dardziński, Francis McGregor-Macdonald, Howard Yoo, 김형은, Kengo Seki, Anirudh Shrinivason, Perttu Salonen, Paweł Leszczyński, Maciej Obuchowski, Harel Shein +
+ +
+ 🎉 Ross Turk, 김형은, Kengo Seki, Anirudh Shrinivason, Perttu Salonen +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-04-04 01:49:37
 +
+

Hi everyone, if I set executors to 0, and bind address to localhost, and then if I want to use openlineage to capture metadata, I seem to run into an error where the executor tries to fetch the spark jar from the driver, even though there is no executor set. Then, it fails because a connection cannot be established. This is some of the error stack trace: +INFO Executor: Fetching spark://&lt;DRIVER_IP&gt;:44541/jars/io.openlineage_openlineage-spark-0.21.1.jar with timestamp 1680506544239 +ERROR Utils: Aborting task +java.io.IOException: Failed to connect to /&lt;DRIVER_IP&gt;:44541 + at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:287) + at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:218) + at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:230) + at org.apache.spark.rpc.netty.NettyRpcEnv.downloadClient(NettyRpcEnv.scala:399) + at org.apache.spark.rpc.netty.NettyRpcEnv.$anonfun$openChannel$4(NettyRpcEnv.scala:367) + at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) + at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1473) + at org.apache.spark.rpc.netty.NettyRpcEnv.openChannel(NettyRpcEnv.scala:366) + at org.apache.spark.util.Utils$.doFetchFile(Utils.scala:755) + at org.apache.spark.util.Utils$.fetchFile(Utils.scala:541) + at org.apache.spark.executor.Executor.$anonfun$updateDependencies$13(Executor.scala:953) + at org.apache.spark.executor.Executor.$anonfun$updateDependencies$13$adapted(Executor.scala:945) + at scala.collection.TraversableLike$WithFilter.$anonfun$foreach$1(TraversableLike.scala:877) + at scala.collection.mutable.HashMap.$anonfun$foreach$1(HashMap.scala:149) + at scala.collection.mutable.HashTable.foreachEntry(HashTable.scala:237) + at scala.collection.mutable.HashTable.foreachEntry$(HashTable.scala:230) + at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:44) + at scala.collection.mutable.HashMap.foreach(HashMap.scala:149) + at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:876) + at <a href="http://org.apache.spark.executor.Executor.org">org.apache.spark.executor.Executor.org</a>$apache$spark$executor$Executor$$updateDependencies(Executor.scala:945) + at org.apache.spark.executor.Executor.&lt;init&gt;(Executor.scala:247) + at org.apache.spark.scheduler.local.LocalEndpoint.&lt;init&gt;(LocalSchedulerBackend.scala:64) + at org.apache.spark.scheduler.local.LocalSchedulerBackend.start(LocalSchedulerBackend.scala:132) + at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:220) + at org.apache.spark.SparkContext.&lt;init&gt;(SparkContext.scala:579) + at org.apache.spark.api.java.JavaSparkContext.&lt;init&gt;(JavaSparkContext.scala:58) + at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) + at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source) + at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source) + at java.base/java.lang.reflect.Constructor.newInstance(Unknown Source) + at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:247) + at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357) + at py4j.Gateway.invoke(Gateway.java:238) + at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80) + at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69) + at py4j.GatewayConnection.run(GatewayConnection.java:238) + at java.base/java.lang.Thread.run(Unknown Source) +Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /&lt;DRIVER_IP&gt;:44541 +Caused by: java.net.ConnectException: Connection refused + at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) + at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source) + at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:330) + at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) + at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:702) + at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650) + at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576) + at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) + at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) + at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) + at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) + at java.base/java.lang.Thread.run(Unknown Source) +Just curious if anyone here has run into a similar problem before, and what the recommended way to resolve this would be...

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-04-04 13:39:19
 +
+

*Thread Reply:* Do you have small configuration and job to replicate this?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-04-04 22:21:35
 +
+

*Thread Reply:* Yeah. For configs: +spark.driver.bindAddress: "localhost" +spark.master: "local[**]" +spark.sql.catalogImplementation: "hive" + spark.openlineage.transport.endpoint: "&lt;endpoint&gt;" + spark.openlineage.transport.type: "http" + spark.sql.catalog.spark_catalog: "org.apache.spark.sql.delta.catalog.DeltaCatalog" + spark.openlineage.transport.url: "&lt;url&gt;" + spark.extraListeners: "io.openlineage.spark.agent.OpenLineageSparkListener" + and job is submitted via spark submit in client mode with number of executors set to 0. +The spark job by itself could be anything...I think the job fails before initializing the spark session itself.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-04-04 22:23:19
 +
+

*Thread Reply:* The issue is because of the spark.jars.packages config... spark.jars config also runs into the same issue. Because the executor tries to fetch the jar from driver for some reason even though there is no executors set...

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-04-05 05:38:55
 +
+

*Thread Reply:* TBH I'm not sure if we can do anything about it. Seems like just having any SparkListener which is not in Spark jars would fall under the same problems, right?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-04-10 06:07:11
 +
+

*Thread Reply:* Yeah... Actually, this was because of binding the driver ip to localhost. In that case, the executor was not able to get the jar from the driver. But yeah I don't think we could have done anything from openlienage end anyway for this. Was just an interesting error to encounter lol

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Lq Dodo + (tryopenmetadata@gmail.com) +
+
2023-04-04 12:07:21
 +
+

Hi, I am new to open lineage. I was able to follow https://openlineage.io/getting-started/ to create a lineage "my-input-->my-job-->my-output". I want to use "my-output" as an input dataset, and connect to the next job, thing like this "my-input-->my-job-->my-output-->my-job2-->my-final-output". How to do it? I have trouble to set eventType and runId, etc. Once the new lineages get massed up, the Marquez UI becomes blank (which is a separated issue).

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2023-04-04 13:02:21
 +
+

*Thread Reply:* In this case you would have four runevents:

+ +
  1. a START event on my-job where my-input is the input and my-output is the output, with a runId you generate on the client
  2. a COMPLETE event on my-job with the same runId from #1
  3. a START event on my-job2 where the input is my-output and the output is my-final-output, with a separate runId you generate
  4. a COMPLETE event on my-job2 with the same runId from #3
    5. +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Lq Dodo + (tryopenmetadata@gmail.com) +
+
2023-04-04 14:53:14
 +
+

*Thread Reply:* thanks for the response. I tried it but now the UI only shows like one second and then turn to blank. I has similar issue before. It seems to me every time when I added a bad lineage, the UI stops working. I have to delete the docker image:-( Not sure whether it is MacOS M1 related issue.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2023-04-04 16:07:06
 +
+

*Thread Reply:* Hmmm, that's interesting. Not sure I've seen that before. If you happen to catch it in that state again, perhaps capture the contents of the lineage_events table so it can be replicated.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Lq Dodo + (tryopenmetadata@gmail.com) +
+
2023-04-04 16:24:28
 +
+

*Thread Reply:* I can fairly easy to reproduce this blank UI issue. Apparently I used the same runId for two different jobs. If I use different unId (which I should), the lineage displays correctly. Thanks again!

+ + + +
+ 👍 Ross Turk +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Lq Dodo + (tryopenmetadata@gmail.com) +
+
2023-04-04 16:41:54
 +
+

Is it possible to add column level lineage via api? Let's say I have fields A,B,C from my-input, and A,B from my-output, and B,C from my-output-s3. I want to see, filter, or query by the column name.

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-04-05 05:35:02
 +
+

*Thread Reply:* You can add https://openlineage.io/docs/spec/facets/dataset-facets/column_lineage_facet/ to your datasets.

+ +

However, I don't think you can currently do any filtering over it

+
+
openlineage.io
+ + + +
Column Level Lineage Dataset Facet | OpenLineage Docs
+
+ Column level lineage provides fine grained information on datasets' dependencies. Not only we know the dependency exist, but we are also able to understand which input columns are used to produce output columns. This allows answering questions like Which root input columns are used to construct column x? +
+ + + + + + +
Original URL: https://openlineage.io/docs/spec/facets/dataset-facets/column_lineage_facet/
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2023-04-05 13:20:20
 +
+

*Thread Reply:* you can see a good example here, @Lq Dodo: https://github.com/MarquezProject/marquez/blob/289fa3eef967c8f7915b074325bb6f8f55480030/docker/metadata.json#L430

+
+ + + + +
<https://github.com/MarquezProject/marquez/blob/289fa3eef967c8f7915b074325bb6f8f55480030/docker/metadata.json | metadata.json>
+
+ <pre><code> "columnLineage": { +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/MarquezProject/marquez">MarquezProject/marquez</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Lq Dodo + (tryopenmetadata@gmail.com) +
+
2023-04-06 11:48:48
 +
+

*Thread Reply:* those examples really help. I can at least build the lineage with column level info using the apis. thanks a lot! Ideally I'd like select one column from the UI and then show me the column level graph. Seems not possible.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@datakin.com) +
+
2023-04-06 12:46:54
 +
+

*Thread Reply:* correct, right now there isn't column-level metadata on the lineage graph 😞

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Pavani + (ylpavani@gmail.com) +
+
2023-04-05 22:01:33
 +
+

Is airflow mandatory, while integrating snowflake with openlineage?

+ +

I am currently looking for a solution which can capture lineage details from snowflake execution

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harel Shein + (harel.shein@gmail.com) +
+
2023-04-06 10:22:17
 +
+

*Thread Reply:* something needs to trigger lineage collection, are you using some sort of scheduler / execution engine?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Pavani + (ylpavani@gmail.com) +
+
2023-04-06 11:26:13
 +
+

*Thread Reply:* Nope... We currently don't have scheduling tool. Isn't it possible to use open lineage api and collect the details?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-04-06 13:12:44
 +
+

@channel +This month’s OpenLineage TSC meeting is on Thursday, April 20th, at 10 am PT. Meeting info: https://openlineage.io/meetings/. All are welcome! +On the tentative agenda:

+ +
  1. Announcements
  2. Updates (new!) +a. OpenLineage in Airflow AIP +b. Static lineage support +c. Reworking namespaces
  3. Recent release overview
  4. A new consumer
  5. Caching support for column lineage
  6. Discussion items +a. Snowflake tagging
  7. Open discussion +Notes: https://bit.ly/OLwiki +Is there a topic you think the community should discuss at this or a future meeting? Reply or DM me to add items to the agenda.
    8. +
+
+
openlineage.io
+ + + +
TSC Meetings | OpenLineage Docs
+
+ The OpenLineage Technical Steering Committee meets monthly, and is open to all. +
+ + + + + + +
Original URL: https://openlineage.io/meetings/
+ + + +
+ + + +
+ 🚀 alexandre bergere, Paweł Leszczyński +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tom van Eijk + (t.m.h.vaneijk@tilburguniversity.edu) +
+
2023-04-06 15:27:41
 +
+

Hi!

+ +

I have a specific question about how OpenLineage fits in between Amazon MWAA and Marquez on AWS EKS. I guess I need to change for example the etl_openlineage DAG in this Snowflake integration tutorial and the OPENLINEAGE_URL here. However, I'm wondering how to reproduce the Docker containers airflow, airflow_scheduler, and airflow_worker here.

+ +

I heard from @Ross Turk that @Willy Lulciuc and @Michael Collado are experts on the K8s integration for OpenLineage and Marquez. Could you provide me some recommendations on how to approach this integration? Or can anyone else help me?

+ +

Kind regards,

+ +

Tom

+
+
openlineage.io
+ + + +
Data Lineage with Snowflake | OpenLineage Docs
+
+ The OpenLineage Adapter offers Snowflake's enterprise users a powerful tool for analyzing their pipelines. +
+ + + + + + +
Original URL: https://openlineage.io/blog/openlineage-snowflake/
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Lukenoff + (john@jlukenoff.com) +
+
2023-04-07 12:47:18
 +
+

[RESOLVED]👋 Hi there, I’m doing a POC of OpenLineage for our airflow deployment. We have a ton of custom operators and I’m trying to test out extracting lineage using the get_openlineage_facets_on_start method. Currently when I’m testing I can see that the OpenLineage plugin is running via airflow plugins but am not able to see that the method is ever getting called. Do I need to do anything else to tell the default extractor to use get_openlineage_facets_on_start? This is the documentation I’m referencing: https://openlineage.io/docs/integrations/airflow/extractors/default-extractors

+
+
openlineage.io
+ + + +
Default extractors | OpenLineage Docs
+
+ Default extractors are a new way in OpenLineage 0.17.0+ to easily add lineage to your data pipelines by modifying your Airflow operators directly. This means custom operators—built in house or forked from another project—can provide you and your team with lineage data without having to modify the OpenLineage project directly, with data sent to your lineage backend of choice, most commonly using the OPENLINEAGE_URL environment variable. +
+ + + + + + +
Original URL: https://openlineage.io/docs/integrations/airflow/extractors/default-extractors
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Lukenoff + (john@jlukenoff.com) +
+
2023-04-07 12:50:14
 +
+

*Thread Reply:* E.g. do I need to update my custom operators to inherit from DefaultExtractor?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Lukenoff + (john@jlukenoff.com) +
+
2023-04-07 13:18:05
 +
+

*Thread Reply:* FWIW, I can tell some level of connectivity to my Marquez deployment is working since I can see it created the default namespace I defined in my OPENLINEAGE_NAMESPACE env var.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-04-07 18:37:44
 +
+

*Thread Reply:* hey John, it is enough to add the method to your custom operator. Perhaps something breaks inside the method. Did anything show up in the logs?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Lukenoff + (john@jlukenoff.com) +
+
2023-04-07 19:03:01
 +
+

*Thread Reply:* That’s the strange part. I’m not seeing anything to suggest that the method is ever getting called. I’m also expecting that the listener created by the plugin should at least be calling this log line when the task runs. However, I’m not seeing that either. I’m able to verify the plugin is registered using airflow plugins and have debug level logging enabled via AIRFLOW__LOGGING__LOGGING_LEVEL='DEBUG'. This is the output of airflow plugins

+ +

name | macros | listeners | source +==================+================================================+==============================+================================================= +OpenLineagePlugin | openlineage.airflow.macros.lineage_run_id,open | openlineage.airflow.listener | openlineage-airflow==0.22.0: + | lineage.airflow.macros.lineage_parent_id | | EntryPoint(name='OpenLineagePlugin', + | | | value='openlineage.airflow.plugin:OpenLineagePlu + | | | gin', group='airflow.plugins') +Appreciate any ideas you might have!

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/d2fa073347ece847c4b6e0ee75a533a95055eb18/integration/airflow/openlineage/airflow/listener.py | listener.py>
+
+ <pre><code> log.debug("OpenLineage listener got notification about task instance start") +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Lukenoff + (john@jlukenoff.com) +
+
2023-04-11 13:09:05
 +
+

*Thread Reply:* Figured this out. Just needed to run the airflow scheduler and trigger tasks through the DAGs vs. airflow tasks test …

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sheeri Cabral (Collibra) + (sheeri.cabral@collibra.com) +
+
2023-04-07 16:29:03
 +
+

I have a question that I believe will be very easy to answer, and I think I know the answer already, but I want to confirm my understanding of extracting OpenLineage with airflow python scripts.

+ +

Extractors extract lineage from operators, so they have to be using operators, right? If someone asks if I can get lineage from their Airflow-orchestrated python scripts, and they show me their scripts but they’re not importing anything starting with airflow.operators, then I can’t use extractors and therefore can’t get lineage. Is that accurate?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sheeri Cabral (Collibra) + (sheeri.cabral@collibra.com) +
+
2023-04-07 16:30:00
 +
+

*Thread Reply:* (they are importing dagkit sdk stuff like Job, JobContext, ExecutionContext, and NodeContext.)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-04-07 18:40:39
 +
+

*Thread Reply:* Do they run those scripts in PythonOperator? If so, they should receive some events but with no datasets extracted

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sheeri Cabral (Collibra) + (sheeri.cabral@collibra.com) +
+
2023-04-07 21:28:25
 +
+

*Thread Reply:* How can I know that? Would it be in the scripts or the airflow configuration or...

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sheeri Cabral (Collibra) + (sheeri.cabral@collibra.com) +
+
2023-04-08 07:13:56
 +
+

*Thread Reply:* And "with no datasets extracted" that means I wouldn't have the schema of the input and output datasets? (I need the db/schema/table/column names for my purposes)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-04-11 02:49:07
 +
+

*Thread Reply:* That really depends what is the current code but in general any custom code in Airflow does not extract any extra information, especially datasets. One can write their own extractors (more in the docs)

+
+
openlineage.io
+ + + +
Default extractors | OpenLineage Docs
+
+ Default extractors are a new way in OpenLineage 0.17.0+ to easily add lineage to your data pipelines by modifying your Airflow operators directly. This means custom operators—built in house or forked from another project—can provide you and your team with lineage data without having to modify the OpenLineage project directly, with data sent to your lineage backend of choice, most commonly using the OPENLINEAGE_URL environment variable. +
+ + + + + + +
Original URL: https://openlineage.io/docs/integrations/airflow/extractors/default-extractors
+ + + +
+ + + +
+ ✅ Sheeri Cabral (Collibra) +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sheeri Cabral (Collibra) + (sheeri.cabral@collibra.com) +
+
2023-04-12 16:52:04
 +
+

*Thread Reply:* Thanks! This is very helpful. Exactly what I needed.

+ + + +
+ 👍 Jakub Dardziński +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tushar Jain + (tujain@ivp.in) +
+
2023-04-09 12:48:04
 +
+

Hi. I was exploring OpenLineage and I want to know does OpenLineage integrate with MS-SQL (Microsoft SQL Server) ? If yes, how to generate OpenLineage events for MS-SQL Views/Tables/Queries?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-04-12 02:30:19
 +
+

*Thread Reply:* Currently there's no extractor implemented for MS-SQL. We try to update list of supported databases here: https://openlineage.io/docs/integrations/about/

+
+
openlineage.io
+ + + +
OpenLineage Integrations | OpenLineage Docs
+
+ Capability Matrix +
+ + + + + + +
Original URL: https://openlineage.io/docs/integrations/about/
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-04-10 12:00:03
 +
+

@channel +Save the date: the next OpenLineage meetup will be in New York on April 26th! More info is coming soon…

+ + + +
+ ✅ Sheeri Cabral (Collibra), Ross Turk, Minkyu Park +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-04-10 19:00:38
 +
+

@channel +Due to many TSC members being on vacation this week, this month’s TSC meeting will be moved to next Thursday, April 20th. All are welcome! https://openlineage.slack.com/archives/C01CK9T7HKR/p1680801164289949

+
+ + +
+ + + } + + Michael Robinson + (https://openlineage.slack.com/team/U02LXF3HUN7) +
+ + + +
+
+ <p>@channel +This month’s OpenLineage TSC meeting is <strong>on Thursday, April 20th, at 10 am PT</strong>. Meeting info: <a href="https://openlineage.io/meetings/">https://openlineage.io/meetings/</a>. All are welcome! +On the tentative agenda:</p> + +<ol><li>Announcements</li><li>Updates (new!) +a. OpenLineage in Airflow AIP +b. Static lineage support +c. Reworking namespaces</li><li>Recent release overview</li><li>A new consumer</li><li>Caching support for column lineage</li><li>Discussion items +a. Snowflake tagging</li><li>Open discussion +Notes: <a href="https://bit.ly/OLwiki">https://bit.ly/OLwiki</a> +Is there a topic you think the community should discuss at this or a future meeting? Reply or DM me to add items to the agenda.</li> +</ol> +
+ + + + + + +
Original URL: https://openlineage.slack.com/archives/C01CK9T7HKR/p1680801164289949
+ + +
+ + Slack Conversation +
+ + +
+ + + +
+ ✅ Sheeri Cabral (Collibra) +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tom van Eijk + (t.m.h.vaneijk@tilburguniversity.edu) +
+
2023-04-11 13:42:03
 +
+

Hi everyone!

+ +

I'm so sorry for all the messages but I'm trying to get Snowflake, OpenLineage and Marquez working for days now. Hopefully, this is my last question. +The snowflake.connector import connect package seems to be outdated here in extract_openlineage.py and is not working for airflow. Does anyone know how to rewrite this code (e.g., with SnowflakeOperator ) and extract the openlineage access history? You'd be my absolute hero!!!

+
+ + + + +
<https://github.com/Snowflake-Labs/OpenLineage-AccessHistory-Setup/blob/main/examples/airflow/dags/lineage/extract_openlineage.py | extract_openlineage.py>
+
+ <pre><code>import json +import os +from pendulum import datetime + +from airflow import DAG +from airflow.decorators import task +from openlineage.client import OpenLineageClient +from snowflake.connector import connect + +SNOWFLAKE_USER = os.getenv('SNOWFLAKE_USER') +SNOWFLAKE_PASSWORD = os.getenv('SNOWFLAKE_PASSWORD') +SNOWFLAKE_ACCOUNT = os.getenv('SNOWFLAKE_ACCOUNT') + + +@task +def send_ol_events(): + client = OpenLineageClient.from_environment() + + with connect( + user=SNOWFLAKE_USER, + password=SNOWFLAKE_PASSWORD, + account=SNOWFLAKE_ACCOUNT, + database='OPENLINEAGE', + schema='PUBLIC', + ) as conn: + with conn.cursor() as cursor: + ol_view = 'OPENLINEAGE_ACCESS_HISTORY' + ol_event_time_tag = 'OL_LATEST_EVENT_TIME' + + var_query = f''' + use warehouse {SNOWFLAKE_WAREHOUSE}; + ''' + + cursor.execute(var_query) + + var_query = f''' + set current_organization='{SNOWFLAKE_ACCOUNT}'; + ''' + + cursor.execute(var_query) + + ol_query = f''' + SELECT ** FROM {ol_view} + WHERE EVENT:eventTime &gt; system$get_tag('{ol_event_time_tag}', '{ol_view}', 'table') + ORDER BY EVENT:eventTime ASC; + ''' + + cursor.execute(ol_query) + ol_events = [json.loads(ol_event[0]) for ol_event in cursor.fetchall()] + + for ol_event in ol_events: + client.emit(ol_event) + + if len(ol_events) &gt; 0: + latest_event_time = ol_events[-1]['eventTime'] + cursor.execute(f''' + ALTER VIEW {ol_view} SET TAG {ol_event_time_tag} = '{latest_event_time}'; + ''') + + +with DAG( + 'etl_openlineage', + start_date=datetime(2022, 4, 12), + schedule_interval='@hourly', + catchup=False, + default_args={ + 'owner': 'openlineage', + 'depends_on_past': False, + 'email_on_failure': False, + 'email_on_retry': False, + 'email': ['&lt;a href="mailto:demo@openlineage.io"&gt;demo@openlineage.io&lt;/a&gt;'], + 'snowflake_conn_id': 'openlineage_snowflake' + }, + description='Send OL events every minutes.', + tags=["extract"], +) as dag: + send_ol_events() + +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/Snowflake-Labs/OpenLineage-AccessHistory-Setup">Snowflake-Labs/OpenLineage-AccessHistory-Setup</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-04-11 17:05:35
 +
+

*Thread Reply:* > The snowflake.connector import connect package seems to be outdated here in extract_openlineage.py and is not working for airflow. +What's the error?

+ +

> Does anyone know how to rewrite this code (e.g., with SnowflakeOperator ) +Current extractor for SnowflakeOperator extracts lineage for SQL executed in the task, in contrast to the method above with OPENLINEAGE_ACCESS_HISTORY view

+
+ + + + +
<https://github.com/Snowflake-Labs/OpenLineage-AccessHistory-Setup/blob/main/examples/airflow/dags/lineage/extract_openlineage.py | extract_openlineage.py>
+
+ <pre><code>import json +import os +from pendulum import datetime + +from airflow import DAG +from airflow.decorators import task +from openlineage.client import OpenLineageClient +from snowflake.connector import connect + +SNOWFLAKE_USER = os.getenv('SNOWFLAKE_USER') +SNOWFLAKE_PASSWORD = os.getenv('SNOWFLAKE_PASSWORD') +SNOWFLAKE_ACCOUNT = os.getenv('SNOWFLAKE_ACCOUNT') + + +@task +def send_ol_events(): + client = OpenLineageClient.from_environment() + + with connect( + user=SNOWFLAKE_USER, + password=SNOWFLAKE_PASSWORD, + account=SNOWFLAKE_ACCOUNT, + database='OPENLINEAGE', + schema='PUBLIC', + ) as conn: + with conn.cursor() as cursor: + ol_view = 'OPENLINEAGE_ACCESS_HISTORY' + ol_event_time_tag = 'OL_LATEST_EVENT_TIME' + + var_query = f''' + use warehouse {SNOWFLAKE_WAREHOUSE}; + ''' + + cursor.execute(var_query) + + var_query = f''' + set current_organization='{SNOWFLAKE_ACCOUNT}'; + ''' + + cursor.execute(var_query) + + ol_query = f''' + SELECT ** FROM {ol_view} + WHERE EVENT:eventTime &gt; system$get_tag('{ol_event_time_tag}', '{ol_view}', 'table') + ORDER BY EVENT:eventTime ASC; + ''' + + cursor.execute(ol_query) + ol_events = [json.loads(ol_event[0]) for ol_event in cursor.fetchall()] + + for ol_event in ol_events: + client.emit(ol_event) + + if len(ol_events) &gt; 0: + latest_event_time = ol_events[-1]['eventTime'] + cursor.execute(f''' + ALTER VIEW {ol_view} SET TAG {ol_event_time_tag} = '{latest_event_time}'; + ''') + + +with DAG( + 'etl_openlineage', + start_date=datetime(2022, 4, 12), + schedule_interval='@hourly', + catchup=False, + default_args={ + 'owner': 'openlineage', + 'depends_on_past': False, + 'email_on_failure': False, + 'email_on_retry': False, + 'email': ['&lt;a href="mailto:demo@openlineage.io"&gt;demo@openlineage.io&lt;/a&gt;'], + 'snowflake_conn_id': 'openlineage_snowflake' + }, + description='Send OL events every minutes.', + tags=["extract"], +) as dag: + send_ol_events() + +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/Snowflake-Labs/OpenLineage-AccessHistory-Setup">Snowflake-Labs/OpenLineage-AccessHistory-Setup</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tom van Eijk + (t.m.h.vaneijk@tilburguniversity.edu) +
+
2023-04-11 18:13:49
 +
+

*Thread Reply:* Hi Maciej!Thank you so much for the reply! I managed to generate a working combination on Windows between the airflow example in the marquez git and the snowflake openlineage git. The only error I still get is: +****** Log file does not exist: /opt/bitnami/airflow/logs/dag_id=etl_openlineage/run_id=manual__2023-04-10T14:12:53.764783+00:00/task_id=send_ol_events/attempt=1.log +****** Fetching from: <http://1c8bb4a78f14:8793/log/dag_id=etl_openlineage/run_id=manual__2023-04-10T14:12:53.764783+00:00/task_id=send_ol_events/attempt=1.log> +****** !!!! Please make sure that all your Airflow components (e.g. schedulers, webservers and workers) have the same 'secret_key' configured in 'webserver' section and time is synchronized on all your machines (for example with ntpd) !!!!! +************ See more at <https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#secret-key> +************ Failed to fetch log file from worker. Client error '403 FORBIDDEN' for url '<http://1c8bb4a78f14:8793/log/dag_id=etl_openlineage/run_id=manual__2023-04-10T14:12:53.764783+00:00/task_id=send_ol_events/attempt=1.log>' +For more information check: <https://httpstatuses.com/403> +This one doesn't make sense to me. I found a workaround for the ETL examples in the OpenLineage git by manually creating a Snowflake connector in Airflow, however, the error is still present for the extract_openlineage.py file. I noticed this file is the only one that uses snowflake.connector import connect and not airflow.providers.snowflake.operators.snowflake import SnowflakeOperator like the other ETL Dags.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-04-12 05:35:41
 +
+

*Thread Reply:* I think it's Airflow error related to getting logs from worker

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-04-12 05:36:07
 +
+

*Thread Reply:* snowflake.connector is a Snowflake connector library that SnowflakeOperator uses underneath to connect to Snowflake

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tom van Eijk + (t.m.h.vaneijk@tilburguniversity.edu) +
+
2023-04-12 10:15:21
 +
+

*Thread Reply:* Ah alright! Thanks for pointing that out! 🙂 Do you know how to solve it? Or do you have any recommendations on how to look for the solution?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-04-12 10:19:53
 +
+

*Thread Reply:* I have no experience with Windows, and I think it's the issue: https://github.com/apache/airflow/issues/10388

+ +

I would try running it in Docker TBH

+
+ + + + +
#10388 Windows support for Airflow
+
+ <p><strong>Description</strong></p> + +<p>Currently, the airflow project uses PEP-3143 style daemons to launch tasks (as implemented in <a href="https://pypi.org/project/python-daemon/">https://pypi.org/project/python-daemon/</a>), however this is targeted towards unix daemons. As a result, running airflow on windows requires multiple levels of abstraction each with their own problems. Would it be possible to use something like daemoniker (<a href="https://daemoniker.readthedocs.io/en/latest/">https://daemoniker.readthedocs.io/en/latest/</a>) to launch tasks? What are the challenges and issues?</p> + +<p>In machine learning workflows, with large datasets, it is a huge time-saver if the pipeline tasks can be run on the GPU. WSL 1 does not support GPU passthrough, docker through WSL 2 supports GPU passthrough only with the Insiders build, additionally it has issues with networking when connected to VPN (<a href="https://github.com/microsoft/WSL/issues/5068">microsoft/WSL#5068</a>).</p> + +<p><strong>Use case / motivation</strong></p> + +<p>Natively running airflow without WSL 1/2 or docker on Windows. This is helpful in cases where the company ecosystem is windows-based.</p> + +<p><strong>Possible implementation</strong></p> + +<p>The daemon module is only used to daemonize the scheduler and webserver. Here's a sample code that runs the scheduler (airflow origin/v1-10-stable) using daemoniker, comments are welcome:</p> + +<pre><code># airflow/bin/cli.py +from daemoniker import Daemonizer + +... + +if args.daemon: + with Daemonizer() as (is_setup, daemonizer): + if is_setup: + pid, stdout, stderr, log_file = setup_locations("scheduler", + args.pid, + args.stdout, + args.stderr, + args.log_file) + + _is_parent = daemonizer( + pid, + stdout_goto=stdout, + stderr_goto=stderr + ) + + job.run() +</code></pre> +
+ +
+
Labels
+ kind:feature +
+ +
+
Comments
+ 22 +
+ + + + + + + +
+ + <a href="https://github.com/apache/airflow">apache/airflow</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tom van Eijk + (t.m.h.vaneijk@tilburguniversity.edu) +
+
2023-04-12 11:47:41
 +
+

*Thread Reply:* Yeah I was running Airflow in Docker but this didn't work. I'll try to use my Macbook for now because I don't think there is a solution for this in the short time. Thank you so much for the support though!!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Peter Hanssens + (peter@cloudshuttle.com.au) +
+
2023-04-13 04:55:41
 +
+

Hi All, +My team and I have been building a status page based on open lineage and I did a talk about it… keen for feedback and thoughts: +https://youtu.be/nGh5_j3hXrE

+
+
YouTube
+ +
+ + + } + + DataEngAU + (https://www.youtube.com/@DataEngAU) +
+ + + +
Peter Hanssens - Sydney Data Engineering meetup, March Edition
+
+ +
+ + + + + + + + +
Original URL: https://youtu.be/nGh5_j3hXrE
+ + + +
+ + + +
+ 👀 Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-04-13 11:19:57
 +
+

*Thread Reply:* Very interesting!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2023-04-13 13:28:53
 +
+

*Thread Reply:* that’s awesome 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ernie Ostic + (ernie.ostic@getmanta.com) +
+
2023-04-13 08:22:50
 +
+

Hi Peter. Looks good. I like the way you introduced the premise of, and benefits of, using OpenLineage for your project. Have you also explored other integrations in addition to dbt?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Peter Hanssens + (peter@cloudshuttle.com.au) +
+
2023-04-13 08:36:01
 +
+

*Thread Reply:* Thanks Ernie, I’m looking at Airflow as well as GE and would like to contribute back to the project as well… we’re close to getting a public preview release of our product done and then we want to help build out open lineage

+ + + +
+ ❤️ Julien Le Dem, Harel Shein +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Lukenoff + (john@jlukenoff.com) +
+
2023-04-13 14:08:38
 +
+

[Resolved] Has anyone seen this error before where the openlineage-airflow plugin / listener fails to deepcopy the task instance? I’m using the native airflow DAG / BashOperator objects to do a basic test of static lineage tagging. More details in 🧵

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Lukenoff + (john@jlukenoff.com) +
+
2023-04-13 14:10:08
 +
+

*Thread Reply:* The dag is basically just: +```dag = DAG( + dagid="asanaexampledag", + defaultargs=defaultargs, + scheduleinterval=None, +)

+ +

samplelineagetask = BashOperator( + taskid="samplelineagetask", + bashcommand='echo $OPENLINEAGEURL', + dag=dag, + inlets=[Table(database="redshift", cluster="someschema", name="someinputtable")], + outlets=[Table(database="redshift", cluster="someotherschema", name="someoutputtable")] +)```

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Lukenoff + (john@jlukenoff.com) +
+
2023-04-13 14:11:02
 +
+

*Thread Reply:* This is the error I’m getting, seems to be coming from this line: +[2023-04-13, 17:45:33 UTC] {logging_mixin.py:115} WARNING - Exception in thread Thread-1: +Traceback (most recent call last): + File "/opt/conda/lib/python3.7/threading.py", line 926, in _bootstrap_inner + self.run() + File "/opt/conda/lib/python3.7/threading.py", line 870, in run + self._target(**self._args, ****self._kwargs) + File "/opt/conda/lib/python3.7/site-packages/openlineage/airflow/listener.py", line 89, in on_running + task_instance_copy = copy.deepcopy(task_instance) + File "/opt/conda/lib/python3.7/copy.py", line 180, in deepcopy + y = _reconstruct(x, memo, **rv) + File "/opt/conda/lib/python3.7/copy.py", line 281, in _reconstruct + state = deepcopy(state, memo) + File "/opt/conda/lib/python3.7/copy.py", line 150, in deepcopy + y = copier(x, memo) + File "/opt/conda/lib/python3.7/copy.py", line 241, in _deepcopy_dict + y[deepcopy(key, memo)] = deepcopy(value, memo) + File "/opt/conda/lib/python3.7/copy.py", line 161, in deepcopy + y = copier(memo) + File "/opt/conda/lib/python3.7/site-packages/airflow/models/baseoperator.py", line 1156, in __deepcopy__ + setattr(result, k, copy.deepcopy(v, memo)) + File "/opt/conda/lib/python3.7/copy.py", line 150, in deepcopy + y = copier(x, memo) + File "/opt/conda/lib/python3.7/copy.py", line 241, in _deepcopy_dict + y[deepcopy(key, memo)] = deepcopy(value, memo) + File "/opt/conda/lib/python3.7/copy.py", line 161, in deepcopy + y = copier(memo) + File "/opt/conda/lib/python3.7/site-packages/airflow/models/dag.py", line 1941, in __deepcopy__ + setattr(result, k, copy.deepcopy(v, memo)) + File "/opt/conda/lib/python3.7/copy.py", line 150, in deepcopy + y = copier(x, memo) + File "/opt/conda/lib/python3.7/copy.py", line 241, in _deepcopy_dict + y[deepcopy(key, memo)] = deepcopy(value, memo) + File "/opt/conda/lib/python3.7/copy.py", line 161, in deepcopy + y = copier(memo) + File "/opt/conda/lib/python3.7/site-packages/airflow/models/baseoperator.py", line 1156, in __deepcopy__ + setattr(result, k, copy.deepcopy(v, memo)) + File "/opt/conda/lib/python3.7/copy.py", line 180, in deepcopy + y = _reconstruct(x, memo, **rv) + File "/opt/conda/lib/python3.7/copy.py", line 281, in _reconstruct + state = deepcopy(state, memo) + File "/opt/conda/lib/python3.7/copy.py", line 150, in deepcopy + y = copier(x, memo) + File "/opt/conda/lib/python3.7/copy.py", line 241, in _deepcopy_dict + y[deepcopy(key, memo)] = deepcopy(value, memo) + File "/opt/conda/lib/python3.7/copy.py", line 150, in deepcopy + y = copier(x, memo) + File "/opt/conda/lib/python3.7/copy.py", line 241, in _deepcopy_dict + y[deepcopy(key, memo)] = deepcopy(value, memo) + File "/opt/conda/lib/python3.7/copy.py", line 161, in deepcopy + y = copier(memo) + File "/opt/conda/lib/python3.7/site-packages/airflow/models/baseoperator.py", line 1156, in __deepcopy__ + setattr(result, k, copy.deepcopy(v, memo)) + File "/opt/conda/lib/python3.7/site-packages/airflow/models/baseoperator.py", line 1000, in __setattr__ + self.set_xcomargs_dependencies() + File "/opt/conda/lib/python3.7/site-packages/airflow/models/baseoperator.py", line 1107, in set_xcomargs_dependencies + XComArg.apply_upstream_relationship(self, arg) + File "/opt/conda/lib/python3.7/site-packages/airflow/models/xcom_arg.py", line 186, in apply_upstream_relationship + op.set_upstream(ref.operator) + File "/opt/conda/lib/python3.7/site-packages/airflow/models/taskmixin.py", line 241, in set_upstream + self._set_relatives(task_or_task_list, upstream=True, edge_modifier=edge_modifier) + File "/opt/conda/lib/python3.7/site-packages/airflow/models/taskmixin.py", line 185, in _set_relatives + dags: Set["DAG"] = {task.dag for task in [**self.roots, **task_list] if task.has_dag() and task.dag} + File "/opt/conda/lib/python3.7/site-packages/airflow/models/taskmixin.py", line 185, in &lt;setcomp&gt; + dags: Set["DAG"] = {task.dag for task in [**self.roots, **task_list] if task.has_dag() and task.dag} + File "/opt/conda/lib/python3.7/site-packages/airflow/models/dag.py", line 508, in __hash__ + val = tuple(self.task_dict.keys()) +AttributeError: 'DAG' object has no attribute 'task_dict'

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/d2fa073347ece847c4b6e0ee75a533a95055eb18/integration/airflow/openlineage/airflow/listener.py | listener.py>
+
+ <pre><code> task_instance_copy = copy.deepcopy(task_instance) +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+ 👀 Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Lukenoff + (john@jlukenoff.com) +
+
2023-04-13 14:12:11
 +
+

*Thread Reply:* This is with Airflow 2.3.2 and openlineage-airflow 0.22.0

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Lukenoff + (john@jlukenoff.com) +
+
2023-04-13 14:13:34
 +
+

*Thread Reply:* Seems like it might be some issue like this with a circular structure? https://stackoverflow.com/questions/46283738/attributeerror-when-using-python-deepcopy

+
+
Stack Overflow
+ + + +
AttributeError when using python deepcopy
+
+ I have a class that has __eq__ and __hash__ overridden, to make its objects act as dictionary keys. Each object also carries a dictionary, keyed by other objects of the same class. I get a weird +
+ + + + + + + + +
Original URL: https://stackoverflow.com/questions/46283738/attributeerror-when-using-python-deepcopy
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-04-14 08:44:36
 +
+

*Thread Reply:* Just by quick look at it, it will definitely be fixed with Airflow 2.6, as it won't need to deepcopy anything.

+ + + +
+ 👍 John Lukenoff +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-04-14 08:47:16
 +
+

*Thread Reply:* I can't seem to reproduce the issue. I ran following example DAG with same Airflow and OL versions as yours: +```import datetime

+ +

from airflow.lineage.entities import Table +from airflow.models import DAG +from airflow.operators.bash import BashOperator

+ +

defaultargs = { + "startdate": datetime.datetime.now() +}

+ +

dag = DAG( + dagid="asanaexampledag", + defaultargs=defaultargs, + scheduleinterval=None, +)

+ +

samplelineagetask = BashOperator( + taskid="samplelineagetask", + bashcommand='echo $OPENLINEAGEURL', + dag=dag, + inlets=[Table(database="redshift", cluster="someschema", name="someinputtable")], + outlets=[Table(database="redshift", cluster="someotherschema", name="someoutputtable")] +)```

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-04-14 08:53:48
 +
+

*Thread Reply:* is there any extra configuration you made possibly?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-04-14 13:02:40
 +
+

*Thread Reply:* @John Lukenoff, I was finally able to reproduce this when passing xcom as task.output +looks like this was reported here and solved by this PR (not sure if this was released in 2.3.3 or later)

+
+ + + + +
#24525 mini-scheduler raises AttributeError: 'NoneType' object has no attribute 'keys'
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/apache/airflow">apache/airflow</a> +
+ + +
+
+ + + + +
#19965 Map and Partial DAG authoring interface for Dynamic Task Mapping
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/apache/airflow">apache/airflow</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Lukenoff + (john@jlukenoff.com) +
+
2023-04-14 13:06:59
 +
+

*Thread Reply:* Ah interesting. Let me see if bumping my Airflow version resolves this. Haven’t had a chance to tinker with it much since yesterday.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-04-14 13:13:21
 +
+

*Thread Reply:* I ran it against 2.4 and same dag works

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Lukenoff + (john@jlukenoff.com) +
+
2023-04-14 13:15:35
 +
+

*Thread Reply:* 👍 Looks like a fix for that issue was rolled out in 2.3.3. I’m gonna try that for now (my company has a notoriously difficult time with airflow major version updates 😅)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-04-14 13:17:06
 +
+

*Thread Reply:* 👍

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Lukenoff + (john@jlukenoff.com) +
+
2023-04-17 12:29:09
 +
+

*Thread Reply:* Got this working! We just monkey patched the __deepcopy__ method of the BaseOperator for now until we can get bandwidth for an airflow upgrade. Thanks for the help here!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-04-17 03:45:47
 +
+

Hi everyone, I am facing this null pointer error: +ERROR AsyncEventQueue: Listener OpenLineageSparkListener threw an exception +java.lang.NullPointerException +java.base/java.util.concurrent.ConcurrentHashMap.putVal(Unknown Source) +java.base/java.util.concurrent.ConcurrentHashMap.put(Unknown Source) +io.openlineage.spark.agent.JobMetricsHolder.addMetrics(JobMetricsHolder.java:40) +io.openlineage.spark.agent.OpenLineageSparkListener.onTaskEnd(OpenLineageSparkListener.java:179) +org.apache.spark.scheduler.SparkListenerBus.doPostEvent(SparkListenerBus.scala:45) +org.apache.spark.scheduler.SparkListenerBus.doPostEvent$(SparkListenerBus.scala:28) +org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37) +org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37) +org.apache.spark.util.ListenerBus.postToAll(ListenerBus.scala:117) +org.apache.spark.util.ListenerBus.postToAll$(ListenerBus.scala:101) +org.apache.spark.scheduler.AsyncEventQueue.super$postToAll(AsyncEventQueue.scala:105) +org.apache.spark.scheduler.AsyncEventQueue.$anonfun$dispatch$1(AsyncEventQueue.scala:105) +scala.runtime.java8.JFunction0$mcJ$sp.apply(JFunction0$mcJ$sp.java:23) +scala.util.DynamicVariable.withValue(DynamicVariable.scala:62) +<a href="http://org.apache.spark.scheduler.AsyncEventQueue.org">org.apache.spark.scheduler.AsyncEventQueue.org</a>$apache$spark$scheduler$AsyncEventQueue$$dispatch(AsyncEventQueue.scala:100) +org.apache.spark.scheduler.AsyncEventQueue$$anon$2.$anonfun$run$1(AsyncEventQueue.scala:96) +org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1381) +org.apache.spark.scheduler.AsyncEventQueue$$anon$2.run(AsyncEventQueue.scala:96) +Could I get some help on this pls 🙇

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-04-17 03:56:30
 +
+

*Thread Reply:* This is the spark submit command: +spark-submit --py-files /usr/local/lib/common_utils.zip,/usr/local/lib/team_utils.zip,/usr/local/lib/project_utils.zip + --conf spark.executor.cores=16 + --conf spark.hadoop.fs.s3a.connection.maximum=100 --conf spark.sql.shuffle.partitions=1000 + --conf spark.speculation=true --conf spark.sql.adaptive.advisoryPartitionSizeInBytes=256MB + --conf spark.hadoop.fs.s3a.multiobjectdelete.enable=false --conf spark.memory.fraction=0.7 --conf spark.kubernetes.executor.label.experiment=some_label --conf spark.kubernetes.executor.label.team=team_name --conf spark.driver.memory=26112m --conf <a href="http://spark.kubernetes.executor.label.app.kubernetes.io/managed-by=pipeline_name">spark.kubernetes.executor.label.app.kubernetes.io/managed-by=pipeline_name</a> --conf spark.kubernetes.executor.label.instance-type=4xlarge --conf spark.executor.instances=10 --conf spark.kubernetes.executor.label.env=prd --conf spark.kubernetes.executor.label.job-name=job_name --conf spark.kubernetes.executor.label.owner=owner --conf spark.kubernetes.executor.label.pipeline=pipeline --conf spark.kubernetes.executor.label.platform-name=platform_name --conf spark.speculation.multiplier=10 --conf spark.memory.storageFraction=0.4 --conf spark.driver.maxResultSize=26112m --conf spark.kubernetes.executor.request.cores=15000m --conf spark.speculation.interval=1s --conf spark.executor.memory=104g --conf spark.sql.catalogImplementation=hive --conf spark.eventLog.dir=file:///logs/spark-events --conf spark.hadoop.fs.s3a.threads.max=100 --conf spark.speculation.quantile=0.75 job.py

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-04-17 04:09:57
 +
+

*Thread Reply:* @Anirudh Shrinivason pls create an issue for this and I will look at it. Although it may be difficult to find the root cause, null pointer exception should be always avoided and this seems to be a bug.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-04-17 04:14:41
 +
+

*Thread Reply:* Hmm yeah sure. I'll create an issue on github for this issue. Thanks!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-04-17 05:13:54
 +
+

*Thread Reply:* https://github.com/OpenLineage/OpenLineage/issues/1784 +Opened an issue here

+
+ + + + +
#1784 Null Pointer Exception with OpenlineageSparkListener
+
+ <p>Hi everyone, <br /> +I am facing this null pointer exception in OpenlineageSparkListener for my job saving into delta table. <br /> +<code>ERROR AsyncEventQueue: Listener OpenLineageSparkListener threw an exception java.lang.NullPointerException java.base/java.util.concurrent.ConcurrentHashMap.putVal(Unknown Source) java.base/java.util.concurrent.ConcurrentHashMap.put(Unknown Source) io.openlineage.spark.agent.JobMetricsHolder.addMetrics(JobMetricsHolder.java:40) io.openlineage.spark.agent.OpenLineageSparkListener.onTaskEnd(OpenLineageSparkListener.java:179) org.apache.spark.scheduler.SparkListenerBus.doPostEvent(SparkListenerBus.scala:45) org.apache.spark.scheduler.SparkListenerBus.doPostEvent$(SparkListenerBus.scala:28) org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37) org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37) org.apache.spark.util.ListenerBus.postToAll(ListenerBus.scala:117) org.apache.spark.util.ListenerBus.postToAll$(ListenerBus.scala:101) org.apache.spark.scheduler.AsyncEventQueue.super$postToAll(AsyncEventQueue.scala:105) org.apache.spark.scheduler.AsyncEventQueue.$anonfun$dispatch$1(AsyncEventQueue.scala:105) scala.runtime.java8.JFunction0$mcJ$sp.apply(JFunction0$mcJ$sp.java:23) scala.util.DynamicVariable.withValue(DynamicVariable.scala:62) &lt;a href="http://org.apache.spark.scheduler.AsyncEventQueue.org"&gt;org.apache.spark.scheduler.AsyncEventQueue.org&lt;/a&gt;$apache$spark$scheduler$AsyncEventQueue$$dispatch(AsyncEventQueue.scala:100) org.apache.spark.scheduler.AsyncEventQueue$$anon$2.$anonfun$run$1(AsyncEventQueue.scala:96) org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1381) org.apache.spark.scheduler.AsyncEventQueue$$anon$2.run(AsyncEventQueue.scala:96)</code></p> + +<p>The spark submit command used is: <br /> +<code>spark-submit --py-files /usr/local/lib/common_utils.zip,/usr/local/lib/team_utils.zip,/usr/local/lib/project_utils.zip --conf spark.executor.cores=16 --conf spark.hadoop.fs.s3a.connection.maximum=100 --conf spark.sql.shuffle.partitions=1000 --conf spark.speculation=true --conf spark.sql.adaptive.advisoryPartitionSizeInBytes=256MB --conf spark.hadoop.fs.s3a.multiobjectdelete.enable=false --conf spark.memory.fraction=0.7 --conf spark.kubernetes.executor.label.experiment=some_label --conf spark.kubernetes.executor.label.team=team_name --conf spark.driver.memory=26112m --conf &lt;a href="http://spark.kubernetes.executor.label.app.kubernetes.io/managed-by=pipeline_name"&gt;spark.kubernetes.executor.label.app.kubernetes.io/managed-by=pipeline_name&lt;/a&gt; --conf spark.kubernetes.executor.label.instance-type=4xlarge --conf spark.executor.instances=10 --conf spark.kubernetes.executor.label.env=prd --conf spark.kubernetes.executor.label.job-name=job_name --conf spark.kubernetes.executor.label.owner=owner --conf spark.kubernetes.executor.label.pipeline=pipeline --conf spark.kubernetes.executor.label.platform-name=platform_name --conf spark.speculation.multiplier=10 --conf spark.memory.storageFraction=0.4 --conf spark.driver.maxResultSize=26112m --conf spark.kubernetes.executor.request.cores=15000m --conf spark.speculation.interval=1s --conf spark.executor.memory=104g --conf spark.sql.catalogImplementation=hive --conf spark.eventLog.dir=file:///logs/spark-events --conf spark.hadoop.fs.s3a.threads.max=100 --conf spark.speculation.quantile=0.75 job.py</code></p> + +<p>Would greatly appreciate any help on this! Thanks!</p> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Allison Suarez + (asuarezmiranda@lyft.com) +
+
2023-04-17 19:32:23
 +
+

Hey! Question about spark column lineage. What is the intended way to write custom code for getting column lineage? i am trying to implement CustomColumnLineageVisitor but when I try to do so I get: +io.openlineage.spark3.agent.lifecycle.plan.column.CustomColumnLineageVisitor is not public in io.openlineage.spark3.agent.lifecycle.plan.column; cannot be accessed from outside package

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-04-18 02:25:04
 +
+

*Thread Reply:* Hi @Allison Suarez, CustomColumnLineageVisitor should be definitely public. I'll prepare a fix PR for that. We do have a test for custom column lineage visitors (CustomColumnLineageVisitorTestImpl), but they're in the same package. Thanks for bringing this.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-04-18 03:07:11
 +
+

*Thread Reply:* This PR should resolve problem: +https://github.com/OpenLineage/OpenLineage/pull/1788

+
+ + + + +
#1788 [SPARK] fix custom column lineage visitors
+
+ <p><strong>Problem</strong></p> + +<p><code>CustomColumnLineageVisitor</code> interface is not public.</p> + +<p><strong>Solution</strong></p> + +<p>Make it public and rearrange test using it, so that visitor implementation is in a different package.</p> + +<blockquote> + <p><strong>Note:</strong> All schema changes require discussion. Please <a href="https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue">link the issue</a> for context.</p> +</blockquote> + +<p>☐ Your change modifies the <a href="https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.json">core</a> OpenLineage model +☐ Your change modifies one or more OpenLineage <a href="https://github.com/OpenLineage/OpenLineage/tree/main/spec/facets">facets</a></p> + +<p>If you're contributing a new integration, please specify the scope of the integration and how/where it has been tested (e.g., Apache Spark integration supports <code>S3</code> and <code>GCS</code> filesystem operations, tested with AWS EMR).</p> + +<p><strong>One-line summary:</strong> +<strong>Checklist</strong></p> + +<p>☑︎ You've <a href="https://github.com/OpenLineage/OpenLineage/blob/main/why-the-dco.md">signed-off</a> your work +☑︎ Your pull request title follows our <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CONTRIBUTING.md#creating-pull-requests">guidelines</a> +☑︎ Your changes are accompanied by tests (<em>if relevant</em>) +☐ Your change contains a <a href="https://kurtisnusbaum.medium.com/stacked-diffs-keeping-phabricator-diffs-small-d9964f4dcfa6">small diff</a> and is self-contained +☐ You've updated any relevant documentation (<em>if relevant</em>) +☐ Your comment includes a one-liner for the changelog about the specific purpose of the change (<em>if necessary</em>) +☐ You've versioned the core OpenLineage model or facets according to <a href="https://docs.snowplowanalytics.com/docs/pipeline-components-and-applications/iglu/common-architecture/schemaver">SchemaVer</a> (<em>if relevant</em>) +☐ You've added a <a href="https://github.com/OpenLineage/OpenLineage/tree/main/.github/header_templates.md">header</a> to source files (<em>if relevant</em>)</p> + +<hr /> + +<p>SPDX-License-Identifier: Apache-2.0 <br /> +Copyright 2018-2023 contributors to the OpenLineage project</p> +
+ +
+
Labels
+ integration/spark +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Allison Suarez + (asuarezmiranda@lyft.com) +
+
2023-04-18 13:34:43
 +
+

*Thread Reply:* Thank you so much @Paweł Leszczyński 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Allison Suarez + (asuarezmiranda@lyft.com) +
+
2023-04-18 13:35:46
 +
+

*Thread Reply:* How does the release process work for OL? Do we have to wait a certain amount of time to get this change in a new release?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Allison Suarez + (asuarezmiranda@lyft.com) +
+
2023-04-18 17:34:29
 +
+

*Thread Reply:* @Maciej Obuchowski ^

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-04-19 01:49:33
 +
+

*Thread Reply:* 0.22.0 was released two weeks ago, so the next schedule should be in next two weeks. We can ask @Michael Robinson his opinion on releasing 0.22.1 before that.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-04-19 09:08:58
 +
+

*Thread Reply:* Hi Allison 👋, +Anyone can request a release in the #general channel. I encourage you to go this route. You’ll need three +1s (there’s more info about the process here: https://github.com/OpenLineage/OpenLineage/blob/main/GOVERNANCE.md), but I don’t know of any reasons why we can’t do a mid-cycle release. 🙂

+ + + +
+ 🙏 Allison Suarez +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Allison Suarez + (asuarezmiranda@lyft.com) +
+
2023-04-19 16:23:20
 +
+

*Thread Reply:* seems like we got enough +1s

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-04-19 16:24:33
 +
+

*Thread Reply:* We need three committers to give a +1. I’ll reach out again to see if I can recruit a third

+ + + +
+ 🙌 Allison Suarez +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Allison Suarez + (asuarezmiranda@lyft.com) +
+
2023-04-19 16:24:55
 +
+

*Thread Reply:* oooh

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-04-19 16:32:47
 +
+

*Thread Reply:* Yeah, sorry I forgot to mention that!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-04-20 05:02:46
 +
+

*Thread Reply:* we have it now

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-04-19 09:52:02
 +
+

@channel +This month’s TSC meeting is tomorrow, 4/20, at 10 am PT: https://openlineage.slack.com/archives/C01CK9T7HKR/p1681167638153879

+
+ + +
+ + + } + + Michael Robinson + (https://openlineage.slack.com/team/U02LXF3HUN7) +
+ + + +
+
+ <p>@channel +Due to many TSC members being on vacation this week, this month’s TSC meeting will be moved to next <strong>Thursday, April 20th</strong>. All are welcome! <a href="https://openlineage.slack.com/archives/C01CK9T7HKR/p1680801164289949">https://openlineage.slack.com/archives/C01CK9T7HKR/p1680801164289949</a></p> +
+ + + + + + +
Original URL: https://openlineage.slack.com/archives/C01CK9T7HKR/p1681167638153879
+ + +
+ + Slack Conversation +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Allison Suarez + (asuarezmiranda@lyft.com) +
+
2023-04-19 13:40:31
 +
+

I would like to get a 0.22.1 patch release to get the issue described in this thread before the next scheduled release.

+
+ + +
+ + + } + + Allison Suarez + (https://openlineage.slack.com/team/U04BNREL8PM) +
+ + + +
+
+ <p>Hey! Question about spark column lineage. What is the intended way to write custom code for getting column lineage? i am trying to implement <code>CustomColumnLineageVisitor</code> but when I try to do so I get: +<code>io.openlineage.spark3.agent.lifecycle.plan.column.CustomColumnLineageVisitor is not public in io.openlineage.spark3.agent.lifecycle.plan.column; cannot be accessed from outside package</code></p> +
+ + + + + + +
Original URL: https://openlineage.slack.com/archives/C01CK9T7HKR/p1681774343143249
+ + +
+ + Thread in Slack Conversation +
+ + +
+ + + +
+ ➕ Michael Robinson, Paweł Leszczyński, Rohit Menon, Maciej Obuchowski, Julien Le Dem, Jakub Dardziński +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-04-20 09:46:06
 +
+

*Thread Reply:* The release is authorized and will be initiated within 2 business days (not including tomorrow).

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-04-19 15:19:38
 +
+

Here are the details about next week’s OpenLineage Meetup at Astronomer’s NY offices: https://openlineage.io/blog/nyc-meetup. Hope to see you there if you can make it!

+
+
openlineage.io
+ + + +
Meet Us in NYC Later This Month! | OpenLineage Docs
+
+ The next OpenLineage Meetup will take place on April 26th in NYC. +
+ + + + + + +
Original URL: https://openlineage.io/blog/nyc-meetup
+ + + +
+ + + +
+ 👍 Ernie Ostic +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sai + (saivenkatesh161@gmail.com) +
+
2023-04-20 07:38:55
 +
+

Hi Team, I tried integrating openLineage with spark databricks and followed the steps as per the documentation. Installation and all looks good as the listener is enabled, but no event is getting passed to Marquez. I can see below message in log4j logs. Am I missing any configuration to be set?

+ +

Running few spark commands in databricks notebook to create events.

+ +

23/04/20 11:10:34 INFO SparkSQLExecutionContext: OpenLineage received Spark event that is configured to be skipped: SparkListenerSQLExecutionStart +23/04/20 11:10:34 INFO SparkSQLExecutionContext: OpenLineage received Spark event that is configured to be skipped: SparkListenerSQLExecutionEnd

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-04-20 08:57:45
 +
+

*Thread Reply:* Hi Sai,

+ +

Perhaps you could try within printing OpenLineage events into logs. This can be achieved with Spark config parameter: +spark.openlineage.transport.type +equal to console .

+ +

This can help you determine if a problem is generating Openlineage events itself or emitting them into Marquez.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sai + (saivenkatesh161@gmail.com) +
+
2023-04-20 09:18:53
 +
+

*Thread Reply:* Hi @Paweł Leszczyński I passed this config as below, but could not see any changes in the logs. The events are getting generated sometimes like below:

+ +

23/04/20 10:00:15 INFO ConsoleTransport: {"eventType":"START","eventTime":"2023-04-20T10:00:15.085Z","run":{"runId":"ef4f46d1-d13a-420a-87c3-19fbf6ffa231","facets":{"spark.logicalPlan":{"producer":"https://github.com/OpenLineage/OpenLineage/tree/0.22.0/integration/spark","schemaURL":"https://openlineage.io/spec/1-0-5/OpenLineage.json#/$defs/RunFacet","plan":[{"class":"org.apache.spark.sql.catalyst.plans.logical.CreateTableAsSelect","num-children":2,"name":0,"partitioning":[],"query":1,"tableSpec":null,"writeOptions":null,"ignoreIfExists":false},{"class":"org.apache.spark.sql.catalyst.analysis.ResolvedTableName","num-children":0,"catalog":null,"ident":null},{"class":"org.apache.spark.sql.catalyst.plans.logical.Project","num-children":1,"projectList":[[{"class":"org.apache.spark.sql.catalyst.expressions.AttributeReference","num_children":0,"name":"workorderid","dataType":"integer","nullable":true,"metadata":{},"exprId":{"product-cl

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-04-20 09:19:37
 +
+

*Thread Reply:* Ok, great. This means the issue is related to Spark <-> Marquez connection

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-04-20 09:20:33
 +
+

*Thread Reply:* Some time ago Spark config has changed and here is the up-to-date-documentation: https://github.com/OpenLineage/OpenLineage/tree/main/integration/spark

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-04-20 09:21:10
 +
+

*Thread Reply:* please note that spark.openlineage.transport.url has to be used which is different from what you have on screenshot attached

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sai + (saivenkatesh161@gmail.com) +
+
2023-04-20 09:22:40
 +
+

*Thread Reply:* You mean instead of "spark.openlineage.host" I need to use "spark.openlineage.transport.url"?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-04-20 09:23:04
 +
+

*Thread Reply:* yes, please give it a try

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sai + (saivenkatesh161@gmail.com) +
+
2023-04-20 09:23:40
 +
+

*Thread Reply:* sure will give a try and let you know the outcome

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-04-20 09:23:48
 +
+

*Thread Reply:* and set spark.openlineage.transport.type to http

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sai + (saivenkatesh161@gmail.com) +
+
2023-04-20 09:24:04
 +
+

*Thread Reply:* okay

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sai + (saivenkatesh161@gmail.com) +
+
2023-04-20 09:26:42
 +
+

*Thread Reply:* does these configs suffice or I need to add anything else

+ +

spark.extraListeners io.openlineage.spark.agent.OpenLineageSparkListener +spark.openlineage.consoleTransport true +spark.openlineage.version v1 +spark.openlineage.transport.type http +spark.openlineage.transport.url http://<host>:5000/api/v1/namespaces/sparkintegrationpoc/

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-04-20 09:27:07
 +
+

*Thread Reply:* spark.openlineage.consoleTransport true this one can be removed

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-04-20 09:27:33
 +
+

*Thread Reply:* otherwise shall be OK

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sai + (saivenkatesh161@gmail.com) +
+
2023-04-20 10:01:30
 +
+

*Thread Reply:* I added these configs and run, but still same issue. Now I am not able to see the events in log file as well.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sai + (saivenkatesh161@gmail.com) +
+
2023-04-20 10:04:27
 +
+

*Thread Reply:* 23/04/20 13:51:22 INFO SparkSQLExecutionContext: OpenLineage received Spark event that is configured to be skipped: SparkListenerSQLExecutionStart +23/04/20 13:51:22 INFO SparkSQLExecutionContext: OpenLineage received Spark event that is configured to be skipped: SparkListenerSQLExecutionEnd

+ +

Does this need any changes in the config side?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sheeri Cabral (Collibra) + (sheeri.cabral@collibra.com) +
+
2023-04-20 13:02:23
 +
+

If you are trying to get into the OpenLineage Technical Steering Committee meeting, you have to RSVP to the specific event at https://www.addevent.com/calendar/pP575215 to get the password (in the invitation to add to your calendar)

+
+
addevent.com
+ + + +
OpenLineage TSC Meetings
+
+ This calendar contains upcoming meetings for the OpenLineage Technical Steering Committee. +
+ + + + + + + + +
Original URL: https://www.addevent.com/calendar/pP575215
+ + + +
+ + + +
+ 🙌 Michael Robinson +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-04-20 13:53:31
 +
+

Here is a nice article I found online that briefly explains about the spark catalogs just for some context: https://www.waitingforcode.com/apache-spark-sql/pluggable-catalog-api/read +In reference to the V2SessionCatalog use case brought up in the meeting just now

+
+
waitingforcode.com
+ + + +
Pluggable Catalog API
+
+ Despite working with Apache Spark for a while, I still have some undiscovered components. One of them crossed my path while I was writing the first blog post from the ACID file formats series. The lucky one is the Catalog API. +
+ + + + + + + + +
Original URL: https://www.waitingforcode.com/apache-spark-sql/pluggable-catalog-api/read
+ + + +
+ + + +
+ 🙌 Michael Robinson, Maciej Obuchowski, Paweł Leszczyński +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-04-24 06:49:43
 +
+

*Thread Reply:* @Anirudh Shrinivason Thanks for linking this as it contains a clear explanation on Spark catalogs. However, I am still unable to write a failing integration test that reproduces the scenario. Could you provide an example of Spark which is failing on V2SessionCatalog and provide more details how are you trying to read/write data?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-04-24 07:14:04
 +
+

*Thread Reply:* Hi @Paweł Leszczyński I noticed this issue on one of our pipelines before actually. I didn't note down which pipeline the issue was occuring in unfortunately. I'll keep checking from my end to identify the spark job that ran into this error. In the meantime, I'll also try to see for which cases deltaCatalog makes use of the V2SessionCatalog to understand this better. Thanks!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-04-26 03:44:15
 +
+

*Thread Reply:* Hi @Paweł Leszczyński +''' + CREATE TABLE IF NOT EXISTS TABLE_NAME ( + SOME COLUMNS + ) USING delta + PARTITIONED BY (col) + location 's3 location' + ''' +A spark sql like this actually triggers the V2SessionCatalog

+ + + +
+ ❤️ Paweł Leszczyński +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-04-26 03:44:48
 +
+

*Thread Reply:* Thanks @Anirudh Shrinivason, will look into that.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-04-26 05:06:05
 +
+

*Thread Reply:* which spark & delta versions are you using?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-04-27 02:35:50
 +
+

*Thread Reply:* I am not 100% sure if this is something you described, but this was an error I was able to replicate and fix. Please look at the exception stacktrace and let me know if it is same on your side. +https://github.com/OpenLineage/OpenLineage/pull/1798

+
+ + + + +
#1798 Spark/fix input dataset
+
+ <p><strong>Problem</strong></p> + +<p>Spark integration is trying to include output dataset's details on START event. This may lead to errors in case of creating tables as the table does not exist on <code>START</code>. This did not break lineage mechanics, although caused the following entry in logs:</p> + +<pre><code>org.apache.spark.sql.catalyst.analysis.NoSuchTableException: Table or view 'tbl' not found in database 'default' + at org.apache.spark.sql.catalyst.catalog.SessionCatalog.requireTableExists(SessionCatalog.scala:226) + at org.apache.spark.sql.catalyst.catalog.SessionCatalog.getTableRawMetadata(SessionCatalog.scala:514) + at org.apache.spark.sql.catalyst.catalog.SessionCatalog.getTableMetadata(SessionCatalog.scala:500) + at org.apache.spark.sql.execution.datasources.v2.V2SessionCatalog.loadTable(V2SessionCatalog.scala:66) + at org.apache.spark.sql.connector.catalog.DelegatingCatalogExtension.loadTable(DelegatingCatalogExtension.java:66) + at org.apache.spark.sql.delta.catalog.DeltaCatalog.loadTable(DeltaCatalog.scala:169) + at io.openlineage.spark3.agent.lifecycle.plan.catalog.DeltaHandler.getDatasetIdentifier(DeltaHandler.java:73) + at io.openlineage.spark3.agent.lifecycle.plan.catalog.CatalogUtils3.lambda$getDatasetIdentifier$2(CatalogUtils3.java:65) + at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) + at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175) + at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175) + at java.util.ArrayList$ArrayListSpliterator.tryAdvance(ArrayList.java:1361) + at java.util.stream.ReferencePipeline.forEachWithCancel(ReferencePipeline.java:126) + at java.util.stream.AbstractPipeline.copyIntoWithCancel(AbstractPipeline.java:499) + at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:486) + at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472) + at java.util.stream.FindOps$FindOp.evaluateSequential(FindOps.java:152) + at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) + at java.util.stream.ReferencePipeline.findAny(ReferencePipeline.java:536) + at io.openlineage.spark3.agent.lifecycle.plan.catalog.CatalogUtils3.getDatasetIdentifier(CatalogUtils3.java:67) + at io.openlineage.spark3.agent.lifecycle.plan.catalog.CatalogUtils3.getDatasetIdentifier(CatalogUtils3.java:48) + at io.openlineage.spark3.agent.utils.PlanUtils3.getDatasetIdentifier(PlanUtils3.java:78) + at io.openlineage.spark33.agent.lifecycle.plan.CreateReplaceDatasetBuilder.apply(CreateReplaceDatasetBuilder.java:143) + at io.openlineage.spark33.agent.lifecycle.plan.CreateReplaceDatasetBuilder.lambda$apply$1(CreateReplaceDatasetBuilder.java:86) + at java.util.Optional.map(Optional.java:215) +</code></pre> + +<p>Potentially, this was related to <a href="https://github.com/OpenLineage/OpenLineage/issues/1747">#1747</a></p> + +<p><strong>Solution</strong></p> + +<p>Catch <code>NoSuchTableException</code> and prevent an error with a stack trace being printed.</p> + +<blockquote> + <p><strong>Note:</strong> All schema changes require discussion. Please <a href="https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue">link the issue</a> for context.</p> +</blockquote> + +<p>• [] Your change modifies the <a href="https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.json">core</a> OpenLineage model +☐ Your change modifies one or more OpenLineage <a href="https://github.com/OpenLineage/OpenLineage/tree/main/spec/facets">facets</a></p> + +<p>If you're contributing a new integration, please specify the scope of the integration and how/where it has been tested (e.g., Apache Spark integration supports <code>S3</code> and <code>GCS</code> filesystem operations, tested with AWS EMR).</p> + +<p><strong>One-line summary:</strong> +<strong>Checklist</strong></p> + +<p>☑︎ You've <a href="https://github.com/OpenLineage/OpenLineage/blob/main/why-the-dco.md">signed-off</a> your work +☑︎ Your pull request title follows our <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CONTRIBUTING.md#creating-pull-requests">guidelines</a> +☑︎ Your changes are accompanied by tests (<em>if relevant</em>) +☑︎ Your change contains a <a href="https://kurtisnusbaum.medium.com/stacked-diffs-keeping-phabricator-diffs-small-d9964f4dcfa6">small diff</a> and is self-contained +☐ You've updated any relevant documentation (<em>if relevant</em>) +☐ Your comment includes a one-liner for the changelog about the specific purpose of the change (<em>if necessary</em>) +☐ You've versioned the core OpenLineage model or facets according to <a href="https://docs.snowplowanalytics.com/docs/pipeline-components-and-applications/iglu/common-architecture/schemaver">SchemaVer</a> (<em>if relevant</em>) +☐ You've added a <a href="https://github.com/OpenLineage/OpenLineage/tree/main/.github/header_templates.md">header</a> to source files (<em>if relevant</em>)</p> + +<hr /> + +<p>SPDX-License-Identifier: Apache-2.0 <br /> +Copyright 2018-2023 contributors to the OpenLineage project</p> +
+ +
+
Labels
+ documentation, integration/spark +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+ :gratitude_thank_you: Anirudh Shrinivason +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-04-27 02:36:20
 +
+

*Thread Reply:* Hi

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-04-27 02:36:45
 +
+

*Thread Reply:* Hmm actually I am noticing this error on my local

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-04-27 02:37:01
 +
+

*Thread Reply:* But on the prod job, I am seeing no such error in the logs...

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-04-27 02:37:28
 +
+

*Thread Reply:* Also, I was using spark 3.1.2

+ + + +
+ 👀 Paweł Leszczyński +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-04-27 02:37:39
 +
+

*Thread Reply:* then perhaps it's sth different :face_palm: will try to replicate on spark 3.1.2

+ + + +
+ :gratitude_thank_you: Anirudh Shrinivason +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-04-27 02:37:42
 +
+

*Thread Reply:* Not too sure which delta version the prod job was using...

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-04-28 03:30:49
 +
+

*Thread Reply:* I was running on Spark 3.1.2 the following command: +spark.sql( + "CREATE TABLE t_partitioned (a int, b int) USING delta " + + "PARTITIONED BY (a) LOCATION '/tmp/delta/tbl'" + ); +and I got Openlineage event emitted with t_partitioned output dataset.

+ + + +
+ :gratitude_thank_you: Anirudh Shrinivason +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-04-28 03:31:47
 +
+

*Thread Reply:* Oh... hmm... that is strange. Let me check more from my end too

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-04-28 03:33:01
 +
+

*Thread Reply:* for spark 3.1, we're using delta 1.0.0

+ + + +
+ 👀 Anirudh Shrinivason +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Cory Visi + (cvisi@amazon.com) +
+
2023-04-20 14:41:23
 +
+

Hi team! I have two Spark jobs chained together to process incoming data files, and I'm using openlineage-spark-0.22.0 with Marquez to visualize. +I'm struggling to figure out the best way to use spark.openlineage.parentRunId and spark.openlineage.parentJobName. Should these values be unique for each Spark job? Should they be unique for each execution of the chain of both spark jobs? Or should they be the same for all runs? +I'm setting them to be unique to the execution of the chain and I'm getting strange results (jobs are not showing completed, and not showing at all)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-04-24 05:38:09
 +
+

*Thread Reply:* Hi Cory, I think the definition of ParentRunFacet (https://openlineage.io/docs/spec/facets/run-facets/parent_run) contains answer to that: +Commonly, scheduler systems like Apache Airflow will trigger processes on remote systems, such as on Apache Spark or Apache Beam jobs. Those systems might have their own OpenLineage integration and report their own job runs and dataset inputs/outputs. The ParentRunFacet allows those downstream jobs to report which jobs spawned them to preserve job hierarchy. To do that, the scheduler system should have a way to pass its own job and run id to the child job. +For example, when airflow is used to run Spark job, we want Spark events to contain some information on what triggered the spark job and parameters, you ask about, are used to pass that information from airflow operator to spark job.

+
+
openlineage.io
+ + + +
Parent Run Facet | OpenLineage Docs
+
+ Commonly, scheduler systems like Apache Airflow will trigger processes on remote systems, such as on Apache Spark or Apache Beam jobs. +
+ + + + + + +
Original URL: https://openlineage.io/docs/spec/facets/run-facets/parent_run
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Cory Visi + (cvisi@amazon.com) +
+
2023-04-26 17:28:39
 +
+

*Thread Reply:* Thank you for pointing me at this documentation; I did not see it previously. In my setup, the calling system is AWS Step Functions, which have no integration with OpenLineage.

+ +

So I've been essentially passing non-existing parent job information to OpenLineage. It has been useful as a data point for searches and reporting though.

+ +

Is there any harm in doing what I am doing? Is it causing the jobs that I see never completing?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-04-27 04:59:39
 +
+

*Thread Reply:* I think parentRunId should be the same for Openlineage START and COMPLETE event. Is it like this in your case?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Cory Visi + (cvisi@amazon.com) +
+
2023-05-03 11:13:58
 +
+

*Thread Reply:* that makes sense, and based on my configuration, i would think that it would be. however, given that i am seeing incomplete jobs in Marquez, i'm wondering if somehow the parentrunID is changing. I need to investigate

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-04-20 15:44:39
 +
+

@channel +We released OpenLineage 0.23.0, including: +Additions: +• SQL: parser improvements to support: copy into, create stage, pivot #1742 @pawel-big-lebowski +• dbt: add support for snapshots #1787 @JDarDagran +Changes: +• Spark: change custom column lineage visitors #1788 @pawel-big-lebowski +Plus bug fixes, doc changes and more. +Thanks to all the contributors! +For the bug fixes and details, see: +Release: https://github.com/OpenLineage/OpenLineage/releases/tag/0.23.0 +Changelog: https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md +Commit history: https://github.com/OpenLineage/OpenLineage/compare/0.22.0...0.23.0 +Maven: https://oss.sonatype.org/#nexus-search;quick~openlineage +PyPI: https://pypi.org/project/openlineage-python/

+ + + +
+ 🎉 Harel Shein, Maciej Obuchowski, Anirudh Shrinivason, Kengo Seki, Paweł Leszczyński, Perttu Salonen +
+ +
+ 👍 Cory Visi, Maciej Obuchowski, Anirudh Shrinivason, Kengo Seki +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-04-21 05:07:30
 +
+

Just curious, how long before we can see 0.23.0 over here: https://mvnrepository.com/artifact/io.openlineage/openlineage-spark

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-04-21 09:06:06
 +
+

*Thread Reply:* I think @Michael Robinson has to manually promote artifacts

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-04-21 09:08:06
 +
+

*Thread Reply:* I promoted the artifacts, but there is a delay before they appear in Maven. A couple releases ago, the delay was about 24 hours long

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-04-21 09:26:09
 +
+

*Thread Reply:* Ahh I see... Thanks!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-04-21 10:10:38
 +
+

*Thread Reply:* @Anirudh Shrinivason are you using search.maven.org by chance? Version 0.23.0 is not appearing there yet, but I do see it on central.sonatype.com.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-04-21 10:15:00
 +
+

*Thread Reply:* Hmm I can see it now on search.maven.org actually. But I still cannot see it on https://mvnrepository.com/artifact/io.openlineage/openlineage-spark ...

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-04-21 10:19:38
 +
+

*Thread Reply:* Understood. I believe you can download the 0.23.0 jars from central.sonatype.com. For Spark, try going here: https://central.sonatype.com/artifact/io.openlineage/openlineage-spark/0.23.0/versions

+
+
Maven Central
+ + + +
Maven Central: io.openlineage:openlineage-spark:0.23.0
+
+ Maven Central: io.openlineage:openlineage_spark:0.23.0 +
+ + + + + + +
Original URL: https://central.sonatype.com/artifact/io.openlineage/openlineage-spark/0.23.0/versions
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-04-22 06:11:10
 +
+

*Thread Reply:* Yup. I can see it on all maven repos now haha. I think its just the delay.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-04-22 06:11:18
 +
+

*Thread Reply:* ~24 hours ig

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-04-24 16:49:15
 +
+

*Thread Reply:* 👍

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Doe + (adarsh.pansari@tigeranalytics.com) +
+
2023-04-21 08:49:54
 +
+

Hello Everyone, I am facing an issue while trying to integrate openlineage with Jupyter notebook. I am following the Docs. My containers are running and I am getting the URL for Jupyter notebook but when I try with the token in the terminal, I get invalid credentials error. Can someone please help resolve this ? Am I doing something wrong..

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Doe + (adarsh.pansari@tigeranalytics.com) +
+
2023-04-21 09:28:18
 +
+

*Thread Reply:* Good news, everyone! The login worked on the second attempt after starting the Docker containers. Although it's unclear why it failed the first time.

+ + + +
+ 👍 Maciej Obuchowski, Anirudh Shrinivason, Michael Robinson, Paweł Leszczyński +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Natalie Zeller + (natalie.zeller@naturalint.com) +
+
2023-04-23 23:52:34
 +
+

Hi team, +I have a question regarding the customization of transport types in OpenLineage. +At my company, we are using OpenLineage to report lineage from our Spark jobs to OpenMetadata. We have created a custom OpenMetadataTransport to send lineage to the OpenMetadata APIs, conforming to the OpenMetadata format. +Currently, we are using a fork of OpenLineage, as we needed to make some changes in the core to identify the new TransportConfig. +We believe it would be more optimal for OpenLineage to support custom transport types, which would allow us to use OpenLineage JAR alongside our own JAR containing the custom transport. +I noticed some comments in the code suggesting that customizations are possible. However, I couldn't make it work without modifying the TransportFactory and the TransportConfig interface, as the transport types are hardcoded. Am I missing something? 🤔 +If custom transport types are not currently supported, we would be more than happy to contribute a PR that enables custom transports. +What are your thoughts on this?

+ + + +
+ ❤️ Paweł Leszczyński +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-04-24 02:32:51
 +
+

*Thread Reply:* Hi Natalie, it's wonderful to hear you're planning to contribute. Yes, you're right about TransportFactory . What other transport type was in your mind? If it is something generic, then it is surely OK to include it within TransportFactory. If it is a custom feature, we could follow ServiceLoader pattern that we're using to allow including custom plan visitors and dataset builders.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Natalie Zeller + (natalie.zeller@naturalint.com) +
+
2023-04-24 02:54:40
 +
+

*Thread Reply:* Hi @Paweł Leszczyński +Yes, I was planning to change TransportFactory to support custom/generic transport types using ServiceLoader pattern. After this change is done, I will be able to use our custom OpenMetadataTransport without changing anything in OpenLineage core. For now I don't have other types in mind, but after we'll add the customization support anyone will be able to create their own transport type and report the lineage to different backends

+ + + +
+ 👍 Paweł Leszczyński +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-04-24 03:28:30
 +
+

*Thread Reply:* Perhaps it's not strictly related to this particular usecase, but you may also find interesting our recent PoC about Fluentd & Openlineage integration. This will bring some cool backend features like: copy event and send it to multiple backends, send it to backends supported by fluentd output plugins etc. https://github.com/OpenLineage/OpenLineage/pull/1757/files?short_path=4fc5534#diff-4fc55343748f353fa1def0e00c553caa735f9adcb0da18baad50a989c0f2e935

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Natalie Zeller + (natalie.zeller@naturalint.com) +
+
2023-04-24 05:36:24
 +
+

*Thread Reply:* Sounds interesting. Thanks, I will look into it

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-04-24 16:37:33
 +
+

Are you planning to come to the first New York OpenLineage Meetup this Wednesday at Astronomer’s offices in the Flatiron District? Don’t forget to RSVP so we know much food and drink to order!

+
+
Meetup
+ + + +
New York OpenLineage Meetup, Wed, Apr 26, 2023, 5:30 PM | Meetup
+
+ Join us on Wednesday, April 26th, 2023 from 5:30_8:30 pm at the Astronomer offices in New York, NY, to learn more about the present and future of OpenLineage. Meet other me +
+ + + + + + + + +
Original URL: https://www.meetup.com/data-lineage-meetup/events/292897496
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sudhar Balaji + (sudharshan.dataaces@gmail.com) +
+
2023-04-25 03:20:57
 +
+

Hi, I'm new to Open data lineage and I'm trying to connect snowflake database with marquez using airflow and getting the error in etl_openlineage while running the airflow dag on local ubuntu environment and unable to see the marquez UI once it etl_openlineage has ran completed as success.

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-04-25 08:07:36
 +
+

*Thread Reply:* What's the extract_openlineage.py file? Looks like your code?

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sudhar Balaji + (sudharshan.dataaces@gmail.com) +
+
2023-04-25 08:43:04
 +
+

*Thread Reply:* import json +import os +from pendulum import datetime

+ +

from airflow import DAG +from airflow.decorators import task +from openlineage.client import OpenLineageClient +from snowflake.connector import connect

+ +

SNOWFLAKEUSER = os.getenv('SNOWFLAKEUSER') +SNOWFLAKEPASSWORD = os.getenv('SNOWFLAKEPASSWORD') +SNOWFLAKEACCOUNT = os.getenv('SNOWFLAKEACCOUNT')

+ +

@task +def sendolevents(): + client = OpenLineageClient.from_environment()

+ +
with connect(
+    user=SNOWFLAKE_USER,
+    password=SNOWFLAKE_PASSWORD,
+    account=SNOWFLAKE_ACCOUNT,
+    database='OPENLINEAGE',
+    schema='PUBLIC',
+) as conn:
+    with conn.cursor() as cursor:
+        ol_view = 'OPENLINEAGE_ACCESS_HISTORY'
+        ol_event_time_tag = 'OL_LATEST_EVENT_TIME'
+
+        var_query = f'''
+            use warehouse {SNOWFLAKE_WAREHOUSE};
+        '''
+
+        cursor.execute(var_query)
+
+        var_query = f'''
+            set current_organization='{SNOWFLAKE_ACCOUNT}';
+        '''
+
+        cursor.execute(var_query)
+
+        ol_query = f'''
+            SELECT ** FROM {ol_view}
+            WHERE EVENT:eventTime &gt; system$get_tag('{ol_event_time_tag}', '{ol_view}', 'table')
+            ORDER BY EVENT:eventTime ASC;
+        '''
+
+        cursor.execute(ol_query)
+        ol_events = [json.loads(ol_event[0]) for ol_event in cursor.fetchall()]
+
+        for ol_event in ol_events:
+            client.emit(ol_event)
+
+        if len(ol_events) &gt; 0:
+            latest_event_time = ol_events[-1]['eventTime']
+            cursor.execute(f'''
+                ALTER VIEW {ol_view} SET TAG {ol_event_time_tag} = '{latest_event_time}';
+            ''')
+
+ +

with DAG( + 'etlopenlineage', + startdate=datetime(2022, 4, 12), + scheduleinterval='@hourly', + catchup=False, + defaultargs={ + 'owner': 'openlineage', + 'dependsonpast': False, + 'emailonfailure': False, + 'emailonretry': False, + 'email': ['demo@openlineage.io'], + 'snowflakeconnid': 'openlineagesnowflake' + }, + description='Send OL events every minutes.', + tags=["extract"], +) as dag: + sendol_events()

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-04-25 09:52:33
 +
+

*Thread Reply:* OpenLineageClient expects RunEvent classes and you're sending it raw json. I think at this point your options are either sending them by constructing your own HTTP client, using something like requests, or using something like https://github.com/python-attrs/cattrs to structure json to RunEvent

+
+ + + + +
python-attrs/cattrs
+
+ Complex custom class converters for attrs. +
+ +
+
Website
+ <https://catt.rs> +
+ +
+
Stars
+ 625 +
+ + + + + + + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-04-25 10:05:57
 +
+

*Thread Reply:* @Jakub Dardziński suggested that you can +change client.emit(ol_event) to client.transport.emit(ol_event) and it should work

+ + + +
+ 👍 Ross Turk, Sudhar Balaji +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@rossturk.com) +
+
2023-04-25 12:24:08
 +
+

*Thread Reply:* @Maciej Obuchowski I believe this is from https://github.com/Snowflake-Labs/OpenLineage-AccessHistory-Setup/blob/main/examples/airflow/dags/lineage/extract_openlineage.py

+
+ + + + +
<https://github.com/Snowflake-Labs/OpenLineage-AccessHistory-Setup/blob/main/examples/airflow/dags/lineage/extract_openlineage.py | extract_openlineage.py>
+
+ <pre><code>import json +import os +from pendulum import datetime + +from airflow import DAG +from airflow.decorators import task +from openlineage.client import OpenLineageClient +from snowflake.connector import connect + +SNOWFLAKE_USER = os.getenv('SNOWFLAKE_USER') +SNOWFLAKE_PASSWORD = os.getenv('SNOWFLAKE_PASSWORD') +SNOWFLAKE_ACCOUNT = os.getenv('SNOWFLAKE_ACCOUNT') + + +@task +def send_ol_events(): + client = OpenLineageClient.from_environment() + + with connect( + user=SNOWFLAKE_USER, + password=SNOWFLAKE_PASSWORD, + account=SNOWFLAKE_ACCOUNT, + database='OPENLINEAGE', + schema='PUBLIC', + ) as conn: + with conn.cursor() as cursor: + ol_view = 'OPENLINEAGE_ACCESS_HISTORY' + ol_event_time_tag = 'OL_LATEST_EVENT_TIME' + + var_query = f''' + use warehouse {SNOWFLAKE_WAREHOUSE}; + ''' + + cursor.execute(var_query) + + var_query = f''' + set current_organization='{SNOWFLAKE_ACCOUNT}'; + ''' + + cursor.execute(var_query) + + ol_query = f''' + SELECT ** FROM {ol_view} + WHERE EVENT:eventTime &gt; system$get_tag('{ol_event_time_tag}', '{ol_view}', 'table') + ORDER BY EVENT:eventTime ASC; + ''' + + cursor.execute(ol_query) + ol_events = [json.loads(ol_event[0]) for ol_event in cursor.fetchall()] + + for ol_event in ol_events: + client.emit(ol_event) + + if len(ol_events) &gt; 0: + latest_event_time = ol_events[-1]['eventTime'] + cursor.execute(f''' + ALTER VIEW {ol_view} SET TAG {ol_event_time_tag} = '{latest_event_time}'; + ''') + + +with DAG( + 'etl_openlineage', + start_date=datetime(2022, 4, 12), + schedule_interval='@hourly', + catchup=False, + default_args={ + 'owner': 'openlineage', + 'depends_on_past': False, + 'email_on_failure': False, + 'email_on_retry': False, + 'email': ['&lt;a href="mailto:demo@openlineage.io"&gt;demo@openlineage.io&lt;/a&gt;'], + 'snowflake_conn_id': 'openlineage_snowflake' + }, + description='Send OL events every minutes.', + tags=["extract"], +) as dag: + send_ol_events() + +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/Snowflake-Labs/OpenLineage-AccessHistory-Setup">Snowflake-Labs/OpenLineage-AccessHistory-Setup</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@rossturk.com) +
+
2023-04-25 12:25:26
 +
+

*Thread Reply:* I believe this example no longer works - perhaps a new access history pull/push example could be created that is simpler and doesn’t use airflow.

+ + + +
+ 👀 Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-04-26 08:34:02
 +
+

*Thread Reply:* I think separating the actual getting data from the view and Airflow DAG would make sense

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@rossturk.com) +
+
2023-04-26 13:57:34
 +
+

*Thread Reply:* Yeah - I also think that Airflow confuses the issue. You don’t need Airflow to get lineage from Snowflake Access History, the only reason Airflow is in the example is a) to simulate a pipeline that can be viewed in Marquez; b) to establish a mechanism that regularly pulls and emits lineage…

+ +

but most people will already have A, and the simplest example doesn’t need to accomplish B.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@rossturk.com) +
+
2023-04-26 13:58:59
 +
+

*Thread Reply:* just a few weeks ago 🙂 I was working on a script that you could run like SNOWFLAKE_USER=foo ./process_snowflake_lineage.py --from-date=xxxx-xx-xx --to-date=xxxx-xx-xx

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tom van Eijk + (t.m.h.vaneijk@tilburguniversity.edu) +
+
2023-04-27 11:13:58
 +
+

*Thread Reply:* Hi @Ross Turk! Do you have a link to this script? Perhaps this script can fix the connection issue 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ross Turk + (ross@rossturk.com) +
+
2023-04-27 11:47:20
 +
+

*Thread Reply:* No, it never became functional before I stopped to take on another task 😕

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sudhar Balaji + (sudharshan.dataaces@gmail.com) +
+
2023-04-25 07:47:57
 +
+

Hi, +Currently, In the .env file, we have using the OPENLINEAGE_URL as <http://marquez-api:5000> and got the error +requests.exceptions.HTTPError: 422 Client Error: for url: <http://marquez-api:5000/api/v1/lineage> +we have tried using OPENLINEAGE_URL as <http://localhost:5000> and getting the error as +requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=5000): Max retries exceeded with url: /api/v1/lineage (Caused by NewConnectionError('&lt;urllib3.connection.HTTPConnection object at 0x7fc71edb9590&gt;: Failed to establish a new connection: [Errno 111] Connection refused')) +I'm not sure which variable value to use for OPENLINEAGE_URL, so please offer the correct variable value.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-04-25 09:54:07
 +
+

*Thread Reply:* Looks like the first URL is proper, but there's something wrong with entity - Marquez logs would help here.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sudhar Balaji + (sudharshan.dataaces@gmail.com) +
+
2023-04-25 09:57:36
 +
+

*Thread Reply:* This is my log in airflow, can you please prvide more info over it.

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-04-25 10:13:37
 +
+

*Thread Reply:* Airflow log does not tell us why Marquez rejected the event. Marquez logs would be more helpful

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sudhar Balaji + (sudharshan.dataaces@gmail.com) +
+
2023-04-26 05:48:08
 +
+

*Thread Reply:* We investigated the marquez container logs and were unable to locate the error. Could you please specify the log file that belongs to marquez while connecting the airflow or snowflake?

+ +

Is it correct that the marquez-web log points to <http://api:5000/>? +[HPM] Proxy created: /api/v1 -&gt; <http://api:5000/> +App listening on port 3000!

+ +
+
+ + + + + + +
+
+
+ + + + + + +
+ + +
+ 👀 Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tom van Eijk + (t.m.h.vaneijk@tilburguniversity.edu) +
+
2023-04-26 11:26:36
 +
+

*Thread Reply:* I've the same error at the moment but can provide some additional screenshots. The Event data in Snowflake seems fine and the data is being retrieved correctly by the Airflow DAG. However, there seems to be a warning in the Marquez API logs. Hopefully we can troubleshoot this together!

+ +
+
+ + + + + + +
+
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tom van Eijk + (t.m.h.vaneijk@tilburguniversity.edu) +
+
2023-04-26 11:33:35
 +
+

*Thread Reply:*

+ +
+
+ + + + + + +
+
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-04-26 13:06:30
 +
+

*Thread Reply:* Possibly the Python part between does some weird things, like double-jsonning the data? I can imagine it being wrapped in second, unnecessary JSON object

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-04-26 13:08:18
 +
+

*Thread Reply:* I guess only way to check is print one of those events - in the form they are send in Python part, not Snowflake - and see how they are like. For example using ConsoleTransport or setting DEBUG log level in Airflow

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tom van Eijk + (t.m.h.vaneijk@tilburguniversity.edu) +
+
2023-04-26 14:37:32
 +
+

*Thread Reply:* Here is a code snippet by using logging in DEBUG on the snowflake python connector:

+ +

[20230426T17:16:55.166+0000] {cursor.py:593} DEBUG - binding: [set currentorganization='[PRIVATE]';] with input=[None], processed=[{}] +[2023-04-26T17:16:55.166+0000] {cursor.py:800} INFO - query: [set currentorganization='[PRIVATE]';] +[2023-04-26T17:16:55.166+0000] {connection.py:1363} DEBUG - sequence counter: 2 +[2023-04-26T17:16:55.167+0000] {cursor.py:467} DEBUG - Request id: f7bca188-dda0-4fe6-8d5c-a92dc5f9c7ac +[2023-04-26T17:16:55.167+0000] {cursor.py:469} DEBUG - running query [set currentorganization='[PRIVATE]';] +[2023-04-26T17:16:55.168+0000] {cursor.py:476} DEBUG - isfiletransfer: True +[2023-04-26T17:16:55.168+0000] {connection.py:1035} DEBUG - _cmdquery +[2023-04-26T17:16:55.168+0000] {connection.py:1062} DEBUG - sql=[set currentorganization='[PRIVATE]';], sequenceid=[2], isfiletransfer=[False] +[2023-04-26T17:16:55.168+0000] {network.py:1162} DEBUG - Session status for SessionPool [PRIVATE]', SessionPool 1/1 active sessions +[2023-04-26T17:16:55.169+0000] {network.py:850} DEBUG - remaining request timeout: None, retry cnt: 1 +[2023-04-26T17:16:55.169+0000] {network.py:828} DEBUG - Request guid: 4acea1c3-6a68-4691-9af4-22f184e0f660 +[2023-04-26T17:16:55.169+0000] {network.py:1021} DEBUG - socket timeout: 60 +[2023-04-26T17:16:55.259+0000] {connectionpool.py:465} DEBUG - [PRIVATE]"POST /queries/v1/query-request?requestId=f7bca188-dda0-4fe6-8d5c-a92dc5f9c7ac&requestguid=4acea1c3-6a68-4691-9af4-22f184e0f660 HTTP/1.1" 200 1118 +[2023-04-26T17:16:55.261+0000] {network.py:1047} DEBUG - SUCCESS +[2023-04-26T17:16:55.261+0000] {network.py:1168} DEBUG - Session status for SessionPool [PRIVATE], SessionPool 0/1 active sessions +[2023-04-26T17:16:55.261+0000] {network.py:729} DEBUG - ret[code] = None, after post request +[2023-04-26T17:16:55.261+0000] {network.py:751} DEBUG - Query id: 01abe3ac-0603-4df4-0042-c78307975eb2 +[2023-04-26T17:16:55.262+0000] {cursor.py:807} DEBUG - sfqid: 01abe3ac-0603-4df4-0042-c78307975eb2 +[2023-04-26T17:16:55.262+0000] {cursor.py:813} INFO - query execution done +[2023-04-26T17:16:55.262+0000] {cursor.py:827} DEBUG - SUCCESS +[2023-04-26T17:16:55.262+0000] {cursor.py:846} DEBUG - PUT OR GET: False +[2023-04-26T17:16:55.263+0000] {cursor.py:941} DEBUG - Query result format: json +[2023-04-26T17:16:55.263+0000] {resultbatch.py:433} DEBUG - parsing for result batch id: 1 +[2023-04-26T17:16:55.263+0000] {cursor.py:956} INFO - Number of results in first chunk: 1 +[2023-04-26T17:16:55.263+0000] {cursor.py:735} DEBUG - executing SQL/command +[2023-04-26T17:16:55.263+0000] {cursor.py:593} DEBUG - binding: [SELECT * FROM OPENLINEAGE_ACCESS_HISTORY WHERE EVENT:eventTime > system$get_tag(...] with input=[None], processed=[{}] +[2023-04-26T17:16:55.264+0000] {cursor.py:800} INFO - query: [SELECT * FROM OPENLINEAGEACCESSHISTORY WHERE EVENT:eventTime > system$gettag(...] +[2023-04-26T17:16:55.264+0000] {connection.py:1363} DEBUG - sequence counter: 3 +[2023-04-26T17:16:55.264+0000] {cursor.py:467} DEBUG - Request id: 21e2ab85-4995-4010-865d-df06cf5ee5b5 +[2023-04-26T17:16:55.265+0000] {cursor.py:469} DEBUG - running query [SELECT ** FROM OPENLINEAGEACCESSHISTORY WHERE EVENT:eventTime > system$gettag(...] +[2023-04-26T17:16:55.265+0000] {cursor.py:476} DEBUG - isfiletransfer: True +[2023-04-26T17:16:55.265+0000] {connection.py:1035} DEBUG - cmdquery +[2023-04-26T17:16:55.265+0000] {connection.py:1062} DEBUG - sql=[SELECT ** FROM OPENLINEAGEACCESSHISTORY WHERE EVENT:eventTime > system$gettag(...], sequenceid=[3], isfiletransfer=[False] +[2023-04-26T17:16:55.266+0000] {network.py:1162} DEBUG - Session status for SessionPool '[PRIVATE}', SessionPool 1/1 active sessions +[2023-04-26T17:16:55.267+0000] {network.py:850} DEBUG - remaining request timeout: None, retry cnt: 1 +[2023-04-26T17:16:55.268+0000] {network.py:828} DEBUG - Request guid: aba82952-a5c2-4c6b-9c70-a10545b8772c +[2023-04-26T17:16:55.268+0000] {network.py:1021} DEBUG - socket timeout: 60 +[2023-04-26T17:17:21.844+0000] {connectionpool.py:465} DEBUG - [PRIVATE] "POST /queries/v1/query-request?requestId=21e2ab85-4995-4010-865d-df06cf5ee5b5&requestguid=aba82952-a5c2-4c6b-9c70-a10545b8772c HTTP/1.1" 200 None +[2023-04-26T17:17:21.879+0000] {network.py:1047} DEBUG - SUCCESS +[2023-04-26T17:17:21.881+0000] {network.py:1168} DEBUG - Session status for SessionPool '[PRIVATE}', SessionPool 0/1 active sessions +[2023-04-26T17:17:21.882+0000] {network.py:729} DEBUG - ret[code] = None, after post request +[2023-04-26T17:17:21.882+0000] {network.py:751} DEBUG - Query id: 01abe3ac-0603-4df4-0042-c78307975eb6 +[2023-04-26T17:17:21.882+0000] {cursor.py:807} DEBUG - sfqid: 01abe3ac-0603-4df4-0042-c78307975eb6 +[2023-04-26T17:17:21.882+0000] {cursor.py:813} INFO - query execution done +[2023-04-26T17:17:21.883+0000] {cursor.py:827} DEBUG - SUCCESS +[2023-04-26T17:17:21.883+0000] {cursor.py:846} DEBUG - PUT OR GET: False +[2023-04-26T17:17:21.883+0000] {cursor.py:941} DEBUG - Query result format: arrow +[2023-04-26T17:17:21.903+0000] {resultbatch.py:102} DEBUG - chunk size=256 +[2023-04-26T17:17:21.920+0000] {cursor.py:956} INFO - Number of results in first chunk: 112 +[2023-04-26T17:17:21.949+0000] {arrowiterator.cpython-37m-x8664-linux-gnu.so:0} DEBUG - Batches read: 1 +[2023-04-26T17:17:21.950+0000] {CArrowIterator.cpp:16} DEBUG - Arrow BatchSize: 1 +[2023-04-26T17:17:21.950+0000] {CArrowChunkIterator.cpp:50} DEBUG - Arrow chunk info: batchCount 1, columnCount 1, usenumpy: 0 +[2023-04-26T17:17:21.950+0000] {resultset.py:232} DEBUG - result batch 1 has id: data001 +[2023-04-26T17:17:21.951+0000] {resultset.py:232} DEBUG - result batch 2 has id: data002 +[2023-04-26T17:17:21.951+0000] {resultset.py:232} DEBUG - result batch 3 has id: data003 +[2023-04-26T17:17:21.951+0000] {resultset.py:232} DEBUG - result batch 4 has id: data010 +[2023-04-26T17:17:21.951+0000] {resultset.py:232} DEBUG - result batch 5 has id: data011 +[2023-04-26T17:17:21.951+0000] {resultset.py:232} DEBUG - result batch 6 has id: data012 +[2023-04-26T17:17:21.952+0000] {resultset.py:232} DEBUG - result batch 7 has id: data013 +[2023-04-26T17:17:21.952+0000] {resultset.py:232} DEBUG - result batch 8 has id: data020 +[2023-04-26T17:17:21.952+0000] {resultset.py:232} DEBUG - result batch 9 has id: data02_1

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-04-26 14:45:26
 +
+

*Thread Reply:* I don't see any Airflow standard logs here, but anyway I looked at it and debugging it would not work if you're bypassing OpenLineageClient.emit and going directly to transport - the logging is done on Client level https://github.com/OpenLineage/OpenLineage/blob/acc207d63e976db7c48384f04bc578409f08cc8a/client/python/openlineage/client/client.py#L73

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/acc207d63e976db7c48384f04bc578409f08cc8a/client/python/openlineage/client/client.py | client.py>
+
+ <pre><code> f"OpenLineageClient will emit event {Serde.to_json(event).encode('utf-8')}" +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tom van Eijk + (t.m.h.vaneijk@tilburguniversity.edu) +
+
2023-04-27 11:16:20
 +
+

*Thread Reply:* I'm sorry, do you have a code snippet on how to get these logs from https://github.com/Snowflake-Labs/OpenLineage-AccessHistory-Setup/blob/main/examples/airflow/dags/lineage/extract_openlineage.py? I still get the ValueError for OpenLineageClient.emit

+
+ + + + +
<https://github.com/Snowflake-Labs/OpenLineage-AccessHistory-Setup/blob/main/examples/airflow/dags/lineage/extract_openlineage.py | extract_openlineage.py>
+
+ <pre><code>import json +import os +from pendulum import datetime + +from airflow import DAG +from airflow.decorators import task +from openlineage.client import OpenLineageClient +from snowflake.connector import connect + +SNOWFLAKE_USER = os.getenv('SNOWFLAKE_USER') +SNOWFLAKE_PASSWORD = os.getenv('SNOWFLAKE_PASSWORD') +SNOWFLAKE_ACCOUNT = os.getenv('SNOWFLAKE_ACCOUNT') + + +@task +def send_ol_events(): + client = OpenLineageClient.from_environment() + + with connect( + user=SNOWFLAKE_USER, + password=SNOWFLAKE_PASSWORD, + account=SNOWFLAKE_ACCOUNT, + database='OPENLINEAGE', + schema='PUBLIC', + ) as conn: + with conn.cursor() as cursor: + ol_view = 'OPENLINEAGE_ACCESS_HISTORY' + ol_event_time_tag = 'OL_LATEST_EVENT_TIME' + + var_query = f''' + use warehouse {SNOWFLAKE_WAREHOUSE}; + ''' + + cursor.execute(var_query) + + var_query = f''' + set current_organization='{SNOWFLAKE_ACCOUNT}'; + ''' + + cursor.execute(var_query) + + ol_query = f''' + SELECT ** FROM {ol_view} + WHERE EVENT:eventTime &gt; system$get_tag('{ol_event_time_tag}', '{ol_view}', 'table') + ORDER BY EVENT:eventTime ASC; + ''' + + cursor.execute(ol_query) + ol_events = [json.loads(ol_event[0]) for ol_event in cursor.fetchall()] + + for ol_event in ol_events: + client.emit(ol_event) + + if len(ol_events) &gt; 0: + latest_event_time = ol_events[-1]['eventTime'] + cursor.execute(f''' + ALTER VIEW {ol_view} SET TAG {ol_event_time_tag} = '{latest_event_time}'; + ''') + + +with DAG( + 'etl_openlineage', + start_date=datetime(2022, 4, 12), + schedule_interval='@hourly', + catchup=False, + default_args={ + 'owner': 'openlineage', + 'depends_on_past': False, + 'email_on_failure': False, + 'email_on_retry': False, + 'email': ['&lt;a href="mailto:demo@openlineage.io"&gt;demo@openlineage.io&lt;/a&gt;'], + 'snowflake_conn_id': 'openlineage_snowflake' + }, + description='Send OL events every minutes.', + tags=["extract"], +) as dag: + send_ol_events() + +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/Snowflake-Labs/OpenLineage-AccessHistory-Setup">Snowflake-Labs/OpenLineage-AccessHistory-Setup</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tom van Eijk + (t.m.h.vaneijk@tilburguniversity.edu) +
+
2023-05-04 10:56:34
 +
+

*Thread Reply:* Hey does anyone have an idea on this? I'm still stuck on this issue 😞

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-05-05 08:58:49
 +
+

*Thread Reply:* I've found the root cause. It's because facets don't have _producer and _schemaURL set. I'll provide a fix soon

+ + + +
+ ♥️ Tom van Eijk, Michael Robinson +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-04-26 11:36:23
 +
+

The first New York OpenLineage Meetup is happening today at 5:30 pm ET at Astronomer’s offices in the Flatiron District! https://openlineage.slack.com/archives/C01CK9T7HKR/p1681931978353159

+
+ + +
+ + + } + + Michael Robinson + (https://openlineage.slack.com/team/U02LXF3HUN7) +
+ + + +
+
+ <p>Here are the details about next week’s OpenLineage Meetup at Astronomer’s NY offices: <a href="https://openlineage.io/blog/nyc-meetup">https://openlineage.io/blog/nyc-meetup</a>. Hope to see you there if you can make it!</p> +
+ + + + + + +
Original URL: https://openlineage.slack.com/archives/C01CK9T7HKR/p1681931978353159
+ + +
+ + Slack Conversation +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2023-04-26 11:36:57
 +
+

*Thread Reply:* I’ll be there! I’m looking forward to see you all.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2023-04-26 11:37:23
 +
+

*Thread Reply:* We’ll talk about the evolution of the spec.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-04-27 02:55:00
 +
+

delta_table = DeltaTable.forPath(spark, path) +delta_table.alias("source").merge(df.alias("update"),lookup_statement).whenMatchedUpdateAll().whenNotMatchedInsertAll().execute() +If I write based on df operations like this, I notice that OL does not emit any event. May I know whether these or similar cases can be supported too? 🙇

+ + + +
+ 👀 Paweł Leszczyński +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-04-28 04:23:24
 +
+

*Thread Reply:* I've created an integration test based on your example. The Openlineage event gets sent, however it does not contain output dataset. I will look deeper into that.

+ + + +
+ :gratitude_thank_you: Anirudh Shrinivason +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-04-28 08:55:43
 +
+

*Thread Reply:* Hey, sorry do you mean input dataset is empty? Or output dataset?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-04-28 08:55:51
 +
+

*Thread Reply:* I am seeing that input dataset is empty

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-04-28 08:56:05
 +
+

*Thread Reply:* ooh, I see input datasets

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-04-28 08:56:11
 +
+

*Thread Reply:* Hmm

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-04-28 08:56:12
 +
+

*Thread Reply:* I see

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-04-28 08:57:07
 +
+

*Thread Reply:* I create a test in SparkDeltaIntegrationTest class a test method: +```@Test + void testDeltaMergeInto() { + Dataset<Row> dataset = + spark + .createDataFrame( + ImmutableList.of( + RowFactory.create(1L, "bat"), + RowFactory.create(2L, "mouse"), + RowFactory.create(3L, "horse") + ), + new StructType( + new StructField[] { + new StructField("a", LongType$.MODULE$, false, Metadata.empty()), + new StructField("b", StringType$.MODULE$, false, Metadata.empty()) + })) + .repartition(1); + dataset.createOrReplaceTempView("temp");

+ +
spark.sql("CREATE TABLE t1 USING delta LOCATION '/tmp/delta/t1' AS SELECT ** FROM temp");
+spark.sql("CREATE TABLE t2 USING delta LOCATION '/tmp/delta/t2' AS SELECT ** FROM temp");
+
+DeltaTable.forName("t1")
+    .merge(spark.read().table("t2"),"t1.a = t2.a")
+    .whenMatched().updateAll()
+    .whenNotMatched().insertAll()
+    .execute();
+
+verifyEvents(mockServer, "pysparkDeltaMergeIntoCompleteEvent.json");
+
+ +

}```

+ + + +
+ 👍 Anirudh Shrinivason +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-04-28 08:59:14
 +
+

*Thread Reply:* Oh yeah my bad. I am seeing output dataset is empty.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-04-28 08:59:21
 +
+

*Thread Reply:* Checks out with your observation

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-05-03 23:23:36
 +
+

*Thread Reply:* Hi @Paweł Leszczyński just curious, has a fix for this been implemented alr?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-05-04 02:40:11
 +
+

*Thread Reply:* Hi @Anirudh Shrinivason, I had some days ooo. I will look into this soon.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-05-04 07:37:52
 +
+

*Thread Reply:* Ahh okie! Thanks so much! Hope you had a good rest!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-05-04 07:38:38
 +
+

*Thread Reply:* yeah. this was an amazing extended weekend 😉

+ + + +
+ 🎉 Anirudh Shrinivason +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-05-05 02:09:10
 +
+

*Thread Reply:* This should be it: https://github.com/OpenLineage/OpenLineage/pull/1823

+
+ + + + +
#1823 Spark/delta merge into
+
+ <p><strong>Problem</strong></p> + +<p><code>merge into</code> queries are not supported for delta tables.</p> + +<p><strong>Solution</strong></p> + +<p>MergeIntoCommand is transformed into following plan:</p> + +<pre><code> == Optimized Logical Plan == +SerializeFromObject ... ++- MapPartitions org.apache.spark.sql.delta.commands.MergeIntoCommand$$Lambda$ ... + +- Join FullOuter, (a#815L = a#819L) + :- Project [a#819L, b#820, UDF() AS _source_row_present_#1052] + : +- Relation[a#819L,b#820] parquet + +- Project [a#815L, b#816, true AS _target_row_present_#1058] + +- Relation[a#815L,b#816] parquet +</code></pre> + +<p>Within the implementation, we check if there is a full outer join with a joined projection containing field <code>_target_row_present_</code>. If so, it extracts output dataset from that projection.</p> + +<blockquote> + <p><strong>Note:</strong> All schema changes require discussion. Please <a href="https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue">link the issue</a> for context.</p> +</blockquote> + +<p>• [] Your change modifies the <a href="https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.json">core</a> OpenLineage model +☐ Your change modifies one or more OpenLineage <a href="https://github.com/OpenLineage/OpenLineage/tree/main/spec/facets">facets</a></p> + +<p>If you're contributing a new integration, please specify the scope of the integration and how/where it has been tested (e.g., Apache Spark integration supports <code>S3</code> and <code>GCS</code> filesystem operations, tested with AWS EMR).</p> + +<p><strong>One-line summary:</strong> +<strong>Checklist</strong></p> + +<p>☑︎ You've <a href="https://github.com/OpenLineage/OpenLineage/blob/main/why-the-dco.md">signed-off</a> your work +☑︎ Your pull request title follows our <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CONTRIBUTING.md#creating-pull-requests">guidelines</a> +☑︎ Your changes are accompanied by tests (<em>if relevant</em>) +☑︎ Your change contains a <a href="https://kurtisnusbaum.medium.com/stacked-diffs-keeping-phabricator-diffs-small-d9964f4dcfa6">small diff</a> and is self-contained +☐ You've updated any relevant documentation (<em>if relevant</em>) +☐ Your comment includes a one-liner for the changelog about the specific purpose of the change (<em>if necessary</em>) +☐ You've versioned the core OpenLineage model or facets according to <a href="https://docs.snowplowanalytics.com/docs/pipeline-components-and-applications/iglu/common-architecture/schemaver">SchemaVer</a> (<em>if relevant</em>) +☐ You've added a <a href="https://github.com/OpenLineage/OpenLineage/tree/main/.github/header_templates.md">header</a> to source files (<em>if relevant</em>)</p> + +<hr /> + +<p>SPDX-License-Identifier: Apache-2.0 <br /> +Copyright 2018-2023 contributors to the OpenLineage project</p> +
+ +
+
Labels
+ integration/spark +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-05-05 02:43:24
 +
+

*Thread Reply:* Hi @Anirudh Shrinivason, please let me know if there is still something to be done within #1747 PROPOSAL] Support for V2SessionCatalog. I could not reproduce exactly what you described but fixed some issue nearby.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-05-05 02:49:38
 +
+

*Thread Reply:* Hmm yeah sure let me find out the exact cause of the issue. The pipeline that was causing the issue is now inactive haha. So I'm trying to backtrace from the limited logs I captured last time. Let me get back by next week thanks! 🙇

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-05-05 09:35:00
 +
+

*Thread Reply:* Hi @Paweł Leszczyński I was trying to replicate the issue from my end, but couldn't do so. I think we can close the issue for now, and revisit later on if the issue resurfaces. Does that sound okay?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-05-05 09:40:33
 +
+

*Thread Reply:* sounds cool. we can surely create a new issue later on.

+ + + +
+ 👍 Anirudh Shrinivason +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harshini Devathi + (harshini.devathi@tigeranalytics.com) +
+
2023-05-09 23:34:04
 +
+

*Thread Reply:* @Paweł Leszczyński - I was trying to implement these new changes in databricks. I was wondering which java file should I use for building the jar file? Could you plese help me?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-06-09 00:46:34
 +
+

*Thread Reply:* .

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-06-09 02:37:49
 +
+

*Thread Reply:* Hi I found that these merge operations have no input datasets/col lineage: +```df.write.format(fileformat).mode(mode).option("mergeSchema", mergeschema).option("overwriteSchema", overwriteSchema).save(path)

+ +

df.write.format(fileformat).mode(mode).option("mergeSchema", mergeschema).option("overwriteSchema", overwriteSchema)\ + .partitionBy(**partitions).save(path)

+ +

df.write.format(fileformat).mode(mode).option("mergeSchema", mergeschema).option("overwriteSchema", overwriteSchema)\ + .partitionBy(**partitions).option("replaceWhere", where_clause).save(path)`` +I also noticed the same issue when using theMERGE INTO` command from spark sql. +Would it be possible to extend the support to these df operations. too please? Thanks! +CC: @Paweł Leszczyński

+ + + +
+ 👀 Paweł Leszczyński +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-06-09 02:41:24
 +
+

*Thread Reply:* Hi @Anirudh Shrinivason, great to hear from you. Could you create an issue out of this? I am working at the moment on Spark 3.4. Once this is ready, I will look at the spark issues. And this one seems to be nicely reproducible. Thanks for that.

+ + + +
+ 👍 Anirudh Shrinivason +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-06-09 02:49:56
 +
+

*Thread Reply:* Sure let me create an issue! Thanks!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-06-09 02:55:21
 +
+

*Thread Reply:* Created an issue here! https://github.com/OpenLineage/OpenLineage/issues/1919 +Thanks! 🙇

+
+ + + + +
#1919 Lineage not captured for MergeIntoCommands
+
+ <p>Hi, I found that these merge operations have no input datasets/col lineage:</p> + +<pre><code>df.write.format(file_format).mode(mode).option("mergeSchema", merge_schema).option("overwriteSchema", overwriteSchema).save(path) + +df.write.format(file_format).mode(mode).option("mergeSchema", merge_schema).option("overwriteSchema", overwriteSchema)\ + .partitionBy(**partitions).save(path) + +df.write.format(file_format).mode(mode).option("mergeSchema", merge_schema).option("overwriteSchema", overwriteSchema)\ + .partitionBy(**partitions).option("replaceWhere", where_clause).save(path) +</code></pre> + +<p>I also noticed the same issue when using the <code>MERGE INTO</code> command from spark sql. <br /> +Would it be possible to extend the support to these df operations. too please? Thanks!</p> +
+ +
+
Labels
+ proposal +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-06-15 10:39:50
 +
+

*Thread Reply:* Hi @Paweł Leszczyński I just realised, https://github.com/OpenLineage/OpenLineage/pull/1823/files +This PR doesn't actually capture column lineage for the MergeIntoCommand? It looks like there is no column lineage field in the events json.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-06-17 04:21:24
 +
+

*Thread Reply:* Hi @Paweł Leszczyński Is there a potential timeline in mind to support column lineage for the MergeIntoCommand? We're really excited for this feature and would be a huge help to overcome a current blocker. Thanks!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-04-28 14:11:34
 +
+

Thanks to everyone who came out to Wednesday night’s meetup in New York! In addition to great pizza from Grimaldi’s (thanks for the tip, @Harel Shein), we enjoyed a spirited discussion of: +• the state of observability tooling in the data space today +• the history and high-level architecture of the project courtesy of @Julien Le Dem +• exciting news of an OpenLineage Scanner being planned at MANTA courtesy of @Ernie Ostic +• updates on the project roadmap and some exciting proposals from @Julien Le Dem, @Harel Shein and @Willy Lulciuc +• an introduction to and demo of Marquez from project lead @Willy Lulciuc +• and more. +Be on the lookout for an announcement about the next meetup!

+ +
+
+ + + + + + +
+ + +
+ ❤️ Harel Shein, Maciej Obuchowski, Peter Hicks, Jakub Dardziński, Atif Tahir +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-04-28 16:02:22
 +
+

As discussed during the April TSC meeting, comments are sought from the community on a proposal to support RunEvent-less (AKA static) lineage metadata emission. This is currently a WIP. For details and to comment, please see: +• https://docs.google.com/document/d/1366bAPkk0OqKkNA4mFFt-41X0cFUQ6sOvhSWmh4Iydo/edit?usp=sharing +• https://docs.google.com/document/d/1gKJw3ITJHArTlE-Iinb4PLkm88moORR0xW7I7hKZIQA/edit?usp=sharing

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ernie Ostic + (ernie.ostic@getmanta.com) +
+
2023-04-30 21:35:47
 +
+

Hi all. Probably I just need to study the spec further, but what is the significance of _producer vs producer in the context of where they are used? (same question also for _schemaURL vs schemaURL)? Thx!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sheeri Cabral (Collibra) + (sheeri.cabral@collibra.com) +
+
2023-05-01 12:02:13
 +
+

*Thread Reply:* “producer” is an element of the event run itself - e.g. what produced the JSON packet you’re studying. There is only one of these per event run. You can think of it as a top-level property.

+ +

“producer” (and “schemaURL”) are elements of a facet. They are the 2 required elements for any customized facet (though I don’t agree they should be required, or at least I believe they should be able to be compatible with a blank value and a null value).

+ +

A packet sent to an API should only have one “producer” element, but can have many _producer elements in sub-objects (though, only one _producer per facet).

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ernie Ostic + (ernie.ostic@getmanta.com) +
+
2023-05-01 12:06:52
 +
+

*Thread Reply:* just curious --- is/was there any specific reason for the underscore prefix? If they are in a facet, they would already be qualified.......

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sheeri Cabral (Collibra) + (sheeri.cabral@collibra.com) +
+
2023-05-01 13:13:28
 +
+

*Thread Reply:* The facet “BaseFacet” that’s used for customization, has 2 required elements - _producer and _schemaURL. so I don’t believe it’s related to qualification.

+ + + +
+ 👍 Ernie Ostic +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-05-01 11:33:02
 +
+

I’m opening a vote to release OpenLineage 0.24.0, including: +• a new OpenLineage extractor for dbt Cloud +• a new interface - TransportBuilder - for creating custom transport types without modifying core components of OpenLineage +• a fix to the LogicalPlanSerializer in the Spark integration to make it operational again +• a new configuration parameter in the Spark integration for making dataset paths less verbose +• a fix to the Flink integration CI +• and more. + Three +1s from committers will authorize an immediate release.

+ + + +
+ ➕ Jakub Dardziński, Willy Lulciuc, Julien Le Dem +
+ +
+ ✅ Sheeri Cabral (Collibra) +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-05-02 19:43:12
 +
+

*Thread Reply:* Thanks for voting. The release will commence within 2 days.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sheeri Cabral (Collibra) + (sheeri.cabral@collibra.com) +
+
2023-05-01 12:03:19
 +
+

Does the Spark integration for OpenLineage also support ETL that uses the Apache Spark Structured Streaming framework?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-05-04 02:33:32
 +
+

*Thread Reply:* Although it is not documented, we do have an integration test for that: https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/app/src/test/resources/spark_scripts/spark_kafka.py

+ +

The test reads and writes data to Kafka and verifies if input/output datasets are collected.

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/app/src/test/resources/spark_scripts/spark_kafka.py | spark_kafka.py>
+
+ <pre><code># Copyright 2018-2023 contributors to the OpenLineage project +# SPDX-License-Identifier: Apache-2.0 + +from pyspark.sql import SparkSession + +appName = "Kafka Examples" +master = "local" + +spark = SparkSession.builder \ + .master(master) \ + .appName(appName) \ + .getOrCreate() + +kafka_servers = "kafka:9092" +checkpointLocation = '/test_data/checkpoint' + + +people = spark.createDataFrame([("Bilbo Baggins", 50), ("Gandalf", 1000), ("Thorin", 195), ("Balin", 178), ("Kili", 77), + ("Dwalin", 169), ("Oin", 167), ("Gloin", 158), ("Fili", 82), ("Bombur", None)], ["name", "age"]) + +ds = people \ + .selectExpr("CAST(name AS STRING) as key", "CAST(age AS STRING) as value") \ + .write \ + .format("kafka") \ + .option("kafka.bootstrap.servers", kafka_servers) \ + .option("topic", "topic1") \ + .option("checkpointLocation", checkpointLocation) \ + .save() + +df = spark \ + .read \ + .format("kafka") \ + .option("kafka.bootstrap.servers", kafka_servers) \ + .option("subscribe", "topic1") \ + .load() + +df.show() + +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+ ✅ Sheeri Cabral (Collibra) +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sheeri Cabral (Collibra) + (sheeri.cabral@collibra.com) +
+
2023-05-01 13:14:14
 +
+

Also, does it work for pyspark jobs? (Forgive me if Spark job = pyspark, I don’t have a lot of depth on how Spark works.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-05-01 22:37:25
 +
+

*Thread Reply:* From my experience, yeah it works for pyspark

+ + + +
+ 🙌 Paweł Leszczyński, Sheeri Cabral (Collibra) +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sheeri Cabral (Collibra) + (sheeri.cabral@collibra.com) +
+
2023-05-01 13:35:41
 +
+

(and in a less generic question, would it work on top of this Spline agent/lineage harvester, or is it a replacement for it?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-05-01 22:39:18
 +
+

*Thread Reply:* Also from my experience, I think we can only use one of them as we can only configure one spark listener... correct me if I'm wrong. But it seems like the latest releases of spline are already using openlineage to some capacity?

+ + + +
+ ✅ Sheeri Cabral (Collibra) +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-05-08 09:46:15
 +
+

*Thread Reply:* In spark.extraListeners you can configure multiple listeners by comma separating them - I think you can use multiple ones with OpenLineage without obvious problems. I think we do pretty similar things to Spline though

+ + + +
+ 👍 Anirudh Shrinivason +
+ +
+ ✅ Sheeri Cabral (Collibra) +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sheeri Cabral (Collibra) + (sheeri.cabral@collibra.com) +
+
2023-05-25 11:28:41
 +
+

*Thread Reply:* (I never said thank you for this, so, thank you!)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sai + (saivenkatesh161@gmail.com) +
+
2023-05-02 04:03:40
 +
+

Hi Team,

+ +

I have configured Open lineage with databricks and it is sending events to Marquez as expected. I have a notebook which joins 3 tables and write the result data frame to an azure adls location. Each time I run the notebook manually, it creates two start events and two complete events for one run as shown in the screenshot. Is this something expected or I am missing something?

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-05-02 10:45:37
 +
+

*Thread Reply:* Hello Sai, thanks for your question! A number of folks who could help with this are OOO, but someone will reply as soon as possible.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-05-04 02:44:46
 +
+

*Thread Reply:* That is interesting @Sai. Are you able to reproduce this with a simple code snippet? Which Openlineage version are you using?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sai + (saivenkatesh161@gmail.com) +
+
2023-05-05 01:16:20
 +
+

*Thread Reply:* Yes @Paweł Leszczyński. Each join query I run on top of delta tables have two start and two complete events. We are using below jar for openlineage.

+ +

openlineage-spark-0.22.0.jar

+ + + +
+ 👀 Paweł Leszczyński +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-05-05 02:41:26
 +
+

*Thread Reply:* https://github.com/OpenLineage/OpenLineage/issues/1828

+
+ + + + +
#1828 [Spark] duplicate events when for delta
+
+ <p>Each join query run on top of delta tables have two start and two complete events. Openlineage 0.22..00 <br /> +<a href="https://openlineage.slack.com/archives/C01CK9T7HKR/p1683014620909079">https://openlineage.slack.com/archives/C01CK9T7HKR/p1683014620909079</a></p> +
+ +
+
Assignees
+ <a href="https://github.com/pawel-big-lebowski">@pawel-big-lebowski</a> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sai + (saivenkatesh161@gmail.com) +
+
2023-05-08 04:05:26
 +
+

*Thread Reply:* Hi @Paweł Leszczyński any updates on this issue?

+ +

Also, OL is not giving column level lineage for group by operations on tables. Is this expected?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-05-08 04:07:04
 +
+

*Thread Reply:* Hi @Sai, https://github.com/OpenLineage/OpenLineage/pull/1830 should fix duplication issue

+
+ + + + +
#1830 [SPARK] filter delta adaptive plan events
+
+ <p><strong>Problem</strong></p> + +<p>Spark connector sends duplicate events when processing delta tables.</p> + +<p>Closes: <a href="https://github.com/OpenLineage/OpenLineage/issues/1828">#1828</a></p> + +<p><strong>Solution</strong></p> + +<p>The problem occurs for adaptive Spark plans when OL events are generated extra for adaptive plans when query is rewritten. As a solution, <code>DeltaEventFilter</code> class is extended to filter the events in such case.</p> + +<blockquote> + <p><strong>Note:</strong> All schema changes require discussion. Please <a href="https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue">link the issue</a> for context.</p> +</blockquote> + +<p>☐ Your change modifies the <a href="https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.json">core</a> OpenLineage model +☐ Your change modifies one or more OpenLineage <a href="https://github.com/OpenLineage/OpenLineage/tree/main/spec/facets">facets</a></p> + +<p>If you're contributing a new integration, please specify the scope of the integration and how/where it has been tested (e.g., Apache Spark integration supports <code>S3</code> and <code>GCS</code> filesystem operations, tested with AWS EMR).</p> + +<p><strong>One-line summary:</strong> +<strong>Checklist</strong></p> + +<p>☑︎ You've <a href="https://github.com/OpenLineage/OpenLineage/blob/main/why-the-dco.md">signed-off</a> your work +☑︎ Your pull request title follows our <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CONTRIBUTING.md#creating-pull-requests">guidelines</a> +☑︎ Your changes are accompanied by tests (<em>if relevant</em>) +☑︎ Your change contains a <a href="https://kurtisnusbaum.medium.com/stacked-diffs-keeping-phabricator-diffs-small-d9964f4dcfa6">small diff</a> and is self-contained +☐ You've updated any relevant documentation (<em>if relevant</em>) +☐ Your comment includes a one-liner for the changelog about the specific purpose of the change (<em>if necessary</em>) +☐ You've versioned the core OpenLineage model or facets according to <a href="https://docs.snowplowanalytics.com/docs/pipeline-components-and-applications/iglu/common-architecture/schemaver">SchemaVer</a> (<em>if relevant</em>) +☐ You've added a <a href="https://github.com/OpenLineage/OpenLineage/tree/main/.github/header_templates.md">header</a> to source files (<em>if relevant</em>)</p> + +<hr /> + +<p>SPDX-License-Identifier: Apache-2.0 <br /> +Copyright 2018-2023 contributors to the OpenLineage project</p> +
+ +
+
Labels
+ documentation, integration/spark +
+ +
+
Assignees
+ <a href="https://github.com/pawel-big-lebowski">@pawel-big-lebowski</a> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sai + (saivenkatesh161@gmail.com) +
+
2023-05-08 04:08:06
 +
+

*Thread Reply:* this would be part of next release?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-05-08 04:08:30
 +
+

*Thread Reply:* Regarding column lineage & group by issue, I think it's something on databricks side -> we do have an open issue for that #1821

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-05-08 04:09:24
 +
+

*Thread Reply:* once #1830 is reviewed and merged, it will be the part of the next relase

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sai + (saivenkatesh161@gmail.com) +
+
2023-05-08 04:11:01
 +
+

*Thread Reply:* sure.. thanks @Paweł Leszczyński

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sai + (saivenkatesh161@gmail.com) +
+
2023-05-16 03:27:01
 +
+

*Thread Reply:* @Paweł Leszczyński I have used the latest jar (0.25.0) and still this issue persists. I see two events for same input/output lineage.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Thomas + (xsist10@gmail.com) +
+
2023-05-03 03:55:44
 +
+

Has anyone used Open Lineage for application lineage? I'm particularly interested in how if/how you handled service boundaries like APIs and Kafka topics and what Dataset Naming (URI) you used.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Thomas + (xsist10@gmail.com) +
+
2023-05-03 04:06:37
 +
+

*Thread Reply:* For example, MySQL is stored as producer + host + port + database + table as something like <mysql://db.foo.com:6543/metrics.orders> +For an API (especially one following REST conditions), I was thinking something like method + host + port + path or GET <https://api.service.com:433/v1/users>

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-05-03 10:13:25
 +
+

*Thread Reply:* Hi Thomas, thanks for asking about this — it sounds cool! I don’t know of others working on this kind of thing, but I’ve been developing a SQLAlchemy integration and have been experimenting with job naming — which I realize isn’t exactly what you’re working on. Hopefully others will chime in here, but in the meantime, would you be willing to create an issue about this? It seems worth discussing how we could expand the spec for this kind of use case.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Thomas + (xsist10@gmail.com) +
+
2023-05-03 10:58:32
 +
+

*Thread Reply:* I suspect this will definitely be a bigger discussion. Let me ponder on the problem a bit more and come back with something a bit more concrete.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-05-03 10:59:21
 +
+

*Thread Reply:* Looking forward to hearing more!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Thomas + (xsist10@gmail.com) +
+
2023-05-03 11:05:47
 +
+

*Thread Reply:* On a tangential note, does OpenLineage's column level lineage have support for (I see it can be extended but want to know if someone had to map this before): +• Properties as a path in a structure (like a JSON structure, Avro schema, protobuf, etc) maybe using something like JSON Path or XPath notation. +• Fragments (when a column is a JSON blob, there is an entire sub-structure that needs to be described) +• Transformation description (how an input affects an output. Is it a direct copy of the value or is it part of a formula)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-05-03 11:22:21
 +
+

*Thread Reply:* I don’t know, but I’ll ping some folks who might.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-05-04 03:24:01
 +
+

*Thread Reply:* Hi @Thomas. Column-lineage support currently does not include json fields. We have included in specification fields like transformationDescription and transformationType to store a string representation of the transformation applied and its type like IDENTITY|MASKED. However, those fields aren't filled within Spark integration at the moment.

+ + + +
+ 🙌 Thomas, Michael Robinson +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-05-03 09:54:57
 +
+

@channel +We released OpenLineage 0.24.0, including: +Additions: +• Support custom transport types #1795 @nataliezeller1 +• Airflow: dbt Cloud integration #1418 @howardyoo +• Spark: support dataset name modification using regex #1796 @pawel-big-lebowski +Plus bug fixes and more. +Thanks to all the contributors! +For the bug fixes and details, see: +Release: https://github.com/OpenLineage/OpenLineage/releases/tag/0.24.0 +Changelog: https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md +Commit history: https://github.com/OpenLineage/OpenLineage/compare/0.23.0...0.24.0 +Maven: https://oss.sonatype.org/#nexus-search;quick~openlineage +PyPI: https://pypi.org/project/openlineage-python/

+ + + +
+ 🎉 Harel Shein, tati +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
GreetBot + +
+
2023-05-03 10:45:32
 +
+

@GreetBot has joined the channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-05-04 11:25:23
 +
+

@channel +This month’s TSC meeting is next Thursday, May 11th, at 10:00 am PT. The tentative agenda will be on the wiki. More info and the meeting link can be found on the website. All are welcome! Also, feel free to reply or DM me with discussion topics, agenda items, etc.

+
+
openlineage.io
+ + + +
TSC Meetings | OpenLineage Docs
+
+ The OpenLineage Technical Steering Committee meets monthly, and is open to all. +
+ + + + + + +
Original URL: https://openlineage.io/meetings/
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harshini Devathi + (harshini.devathi@tigeranalytics.com) +
+
2023-05-05 12:11:37
 +
+

Hello all, noticed that openlineage is not able to give column level lineage if there is a groupby operation on a spark dataframe. Has anyone else faced this issue and have any fixes or workarounds? Apache Spark 3.0.1 and Openlineage version 1 are being used. Also tried on Spark version 3.3.0

+ +

Log4j error details follow:

+ +

23/05/05 18:09:11 ERROR ColumnLevelLineageUtils: Error when invoking static method 'buildColumnLineageDatasetFacet' for Spark3 +java.lang.reflect.InvocationTargetException + at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) + at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) + at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) + at java.lang.reflect.Method.invoke(Method.java:498) + at io.openlineage.spark.agent.lifecycle.plan.column.ColumnLevelLineageUtils.buildColumnLineageDatasetFacet(ColumnLevelLineageUtils.java:35) + at io.openlineage.spark.agent.lifecycle.OpenLineageRunEventBuilder.lambda$buildOutputDatasets$21(OpenLineageRunEventBuilder.java:424) + at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) + at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) + at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) + at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472) + at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) + at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) + at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:566) + at io.openlineage.spark.agent.lifecycle.OpenLineageRunEventBuilder.buildOutputDatasets(OpenLineageRunEventBuilder.java:437) + at io.openlineage.spark.agent.lifecycle.OpenLineageRunEventBuilder.populateRun(OpenLineageRunEventBuilder.java:296) + at io.openlineage.spark.agent.lifecycle.OpenLineageRunEventBuilder.buildRun(OpenLineageRunEventBuilder.java:279) + at io.openlineage.spark.agent.lifecycle.OpenLineageRunEventBuilder.buildRun(OpenLineageRunEventBuilder.java:222) + at io.openlineage.spark.agent.lifecycle.SparkSQLExecutionContext.start(SparkSQLExecutionContext.java:70) + at io.openlineage.spark.agent.OpenLineageSparkListener.lambda$sparkSQLExecStart$0(OpenLineageSparkListener.java:91) + at java.util.Optional.ifPresent(Optional.java:159) + at io.openlineage.spark.agent.OpenLineageSparkListener.sparkSQLExecStart(OpenLineageSparkListener.java:91) + at io.openlineage.spark.agent.OpenLineageSparkListener.onOtherEvent(OpenLineageSparkListener.java:82) + at org.apache.spark.scheduler.SparkListenerBus.doPostEvent(SparkListenerBus.scala:102) + at org.apache.spark.scheduler.SparkListenerBus.doPostEvent$(SparkListenerBus.scala:28) + at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:39) + at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:39) + at org.apache.spark.util.ListenerBus.postToAll(ListenerBus.scala:118) + at org.apache.spark.util.ListenerBus.postToAll$(ListenerBus.scala:102) + at org.apache.spark.scheduler.AsyncEventQueue.super$postToAll(AsyncEventQueue.scala:107) + at org.apache.spark.scheduler.AsyncEventQueue.$anonfun$dispatch$1(AsyncEventQueue.scala:107) + at scala.runtime.java8.JFunction0$mcJ$sp.apply(JFunction0$mcJ$sp.java:23) + at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62) + at org.apache.spark.scheduler.AsyncEventQueue.org$apache$spark$scheduler$AsyncEventQueue$$dispatch(AsyncEventQueue.scala:102) + at org.apache.spark.scheduler.AsyncEventQueue$$anon$2.$anonfun$run$1(AsyncEventQueue.scala:98) + at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1639) + at org.apache.spark.scheduler.AsyncEventQueue$$anon$2.run(AsyncEventQueue.scala:98) +Caused by: java.lang.NoSuchMethodError: org.apache.spark.sql.catalyst.expressions.aggregate.AggregateExpression.resultId()Lorg/apache/spark/sql/catalyst/expressions/ExprId; + at io.openlineage.spark3.agent.lifecycle.plan.column.ExpressionDependencyCollector.traverseExpression(ExpressionDependencyCollector.java:79) + at io.openlineage.spark3.agent.lifecycle.plan.column.ExpressionDependencyCollector.lambda$traverseExpression$4(ExpressionDependencyCollector.java:74) + at java.util.Iterator.forEachRemaining(Iterator.java:116) + at scala.collection.convert.Wrappers$IteratorWrapper.forEachRemaining(Wrappers.scala:31) + at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801) + at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:647) + at io.openlineage.spark3.agent.lifecycle.plan.column.ExpressionDependencyCollector.traverseExpression(ExpressionDependencyCollector.java:74) + at io.openlineage.spark3.agent.lifecycle.plan.column.ExpressionDependencyCollector.lambda$null$2(ExpressionDependencyCollector.java:60) + at java.util.LinkedList$LLSpliterator.forEachRemaining(LinkedList.java:1235) + at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:647) + at io.openlineage.spark3.agent.lifecycle.plan.column.ExpressionDependencyCollector.lambda$collect$3(ExpressionDependencyCollector.java:60) + at org.apache.spark.sql.catalyst.trees.TreeNode.foreach(TreeNode.scala:285) + at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$foreach$1(TreeNode.scala:286) + at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$foreach$1$adapted(TreeNode.scala:286) + at scala.collection.Iterator.foreach(Iterator.scala:943) + at scala.collection.Iterator.foreach$(Iterator.scala:943) + at scala.collection.AbstractIterator.foreach(Iterator.scala:1431) + at scala.collection.IterableLike.foreach(IterableLike.scala:74) + at scala.collection.IterableLike.foreach$(IterableLike.scala:73) + at scala.collection.AbstractIterable.foreach(Iterable.scala:56) + at org.apache.spark.sql.catalyst.trees.TreeNode.foreach(TreeNode.scala:286) + at io.openlineage.spark3.agent.lifecycle.plan.column.ExpressionDependencyCollector.collect(ExpressionDependencyCollector.java:38) + at io.openlineage.spark3.agent.lifecycle.plan.column.ColumnLevelLineageUtils.collectInputsAndExpressionDependencies(ColumnLevelLineageUtils.java:70) + at io.openlineage.spark3.agent.lifecycle.plan.column.ColumnLevelLineageUtils.buildColumnLineageDatasetFacet(ColumnLevelLineageUtils.java:40) + ... 36 more

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-05-08 07:38:19
 +
+

*Thread Reply:* Hi @Harshini Devathi, I think this the same as issue: https://github.com/OpenLineage/OpenLineage/issues/1821

+
+ + + + +
#1821 Azure databricks use aggregate funcation can not buildColumnLineageDatasetFacet
+
+ <p>Enviroment: <br /> +databricks 12.1 (includes Apache Spark 3.3.1, Scala 2.12) <br /> +Openlineage 0.21.1 <br /> +sql: <br /> +INSERT INTO TABLE openliange<em>test <br /> +SELECT <br /> +union</em>id, <br /> +first(name) AS name <br /> +FROM tmp <br /> +GROUP BY union_id; <br /> +when we use 'GROUP BY' function, the program report err: <br /> +23/03/10 07:49:15 ERROR ColumnLevelLineageUtils: Error when invoking static method 'buildColumnLineageDatasetFacet' for Spark3 <br /> +java.lang.reflect.InvocationTargetException <br /> +at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) <br /> +at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) <br /> +at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) <br /> +at java.lang.reflect.Method.invoke(Method.java:498) <br /> +at io.openlineage.spark.agent.lifecycle.plan.column.ColumnLevelLineageUtils.buildColumnLineageDatasetFacet(ColumnLevelLineageUtils.java:35) <br /> +at io.openlineage.spark.agent.lifecycle.OpenLineageRunEventBuilder.lambda$buildOutputDatasets$21(OpenLineageRunEventBuilder.java:424) <br /> +at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) <br /> +at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) <br /> +at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) <br /> +at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472) <br /> +at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) <br /> +at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) <br /> +at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:566) <br /> +at io.openlineage.spark.agent.lifecycle.OpenLineageRunEventBuilder.buildOutputDatasets(OpenLineageRunEventBuilder.java:437) <br /> +at io.openlineage.spark.agent.lifecycle.OpenLineageRunEventBuilder.populateRun(OpenLineageRunEventBuilder.java:296) <br /> +at io.openlineage.spark.agent.lifecycle.OpenLineageRunEventBuilder.buildRun(OpenLineageRunEventBuilder.java:279) <br /> +at io.openlineage.spark.agent.lifecycle.OpenLineageRunEventBuilder.buildRun(OpenLineageRunEventBuilder.java:231) <br /> +at io.openlineage.spark.agent.lifecycle.SparkSQLExecutionContext.end(SparkSQLExecutionContext.java:95) <br /> +at io.openlineage.spark.agent.OpenLineageSparkListener.sparkSQLExecEnd(OpenLineageSparkListener.java:98) <br /> +at io.openlineage.spark.agent.OpenLineageSparkListener.onOtherEvent(OpenLineageSparkListener.java:84) <br /> +at org.apache.spark.scheduler.SparkListenerBus.doPostEvent(SparkListenerBus.scala:102) <br /> +at org.apache.spark.scheduler.SparkListenerBus.doPostEvent$(SparkListenerBus.scala:28) <br /> +at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:42) <br /> +at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:42) <br /> +at org.apache.spark.util.ListenerBus.postToAll(ListenerBus.scala:118) <br /> +at org.apache.spark.util.ListenerBus.postToAll$(ListenerBus.scala:102) <br /> +at org.apache.spark.scheduler.AsyncEventQueue.super$postToAll(AsyncEventQueue.scala:114) <br /> +at org.apache.spark.scheduler.AsyncEventQueue.$anonfun$dispatch$1(AsyncEventQueue.scala:114) <br /> +at scala.runtime.java8.JFunction0$mcJ$sp.apply(JFunction0$mcJ$sp.java:23) <br /> +at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62) <br /> +at <a href="http://org.apache.spark.scheduler.AsyncEventQueue.org">org.apache.spark.scheduler.AsyncEventQueue.org</a>$apache$spark$scheduler$AsyncEventQueue$$dispatch(AsyncEventQueue.scala:109) <br /> +at org.apache.spark.scheduler.AsyncEventQueue$$anon$2.$anonfun$run$1(AsyncEventQueue.scala:105) <br /> +at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1655) <br /> +at org.apache.spark.scheduler.AsyncEventQueue$$anon$2.run(AsyncEventQueue.scala:105) <br /> +Caused by: java.lang.NoSuchMethodError: org.apache.spark.sql.catalyst.expressions.aggregate.AggregateExpression.resultId()Lorg/apache/spark/sql/catalyst/expressions/ExprId; <br /> +at io.openlineage.spark3.agent.lifecycle.plan.column.ExpressionDependencyCollector.traverseExpression(ExpressionDependencyCollector.java:71) <br /> +at io.openlineage.spark3.agent.lifecycle.plan.column.ExpressionDependencyCollector.lambda$traverseExpression$4(ExpressionDependencyCollector.java:66) <br /> +at java.util.Iterator.forEachRemaining(Iterator.java:116) <br /> +at scala.collection.convert.Wrappers$IteratorWrapper.forEachRemaining(Wrappers.scala:31) <br /> +at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801) <br /> +at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:647) <br /> +at io.openlineage.spark3.agent.lifecycle.plan.column.ExpressionDependencyCollector.traverseExpression(ExpressionDependencyCollector.java:66) <br /> +at io.openlineage.spark3.agent.lifecycle.plan.column.ExpressionDependencyCollector.lambda$null$2(ExpressionDependencyCollector.java:52) <br /> +at java.util.LinkedList$LLSpliterator.forEachRemaining(LinkedList.java:1235) <br /> +at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:647) <br /> +at io.openlineage.spark3.agent.lifecycle.plan.column.ExpressionDependencyCollector.lambda$collect$3(ExpressionDependencyCollector.java:52) <br /> +at org.apache.spark.sql.catalyst.trees.TreeNode.foreach(TreeNode.scala:292) <br /> +at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$foreach$1(TreeNode.scala:293) <br /> +at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$foreach$1$adapted(TreeNode.scala:293) <br /> +at scala.collection.Iterator.foreach(Iterator.scala:943) <br /> +at scala.collection.Iterator.foreach$(Iterator.scala:943) <br /> +at scala.collection.AbstractIterator.foreach(Iterator.scala:1431) <br /> +at scala.collection.IterableLike.foreach(IterableLike.scala:74) <br /> +at scala.collection.IterableLike.foreach$(IterableLike.scala:73) <br /> +at scala.collection.AbstractIterable.foreach(Iterable.scala:56) <br /> +at org.apache.spark.sql.catalyst.trees.TreeNode.foreach(TreeNode.scala:293) <br /> +at io.openlineage.spark3.agent.lifecycle.plan.column.ExpressionDependencyCollector.collect(ExpressionDependencyCollector.java:35) <br /> +at io.openlineage.spark3.agent.lifecycle.plan.column.ColumnLevelLineageUtils.buildColumnLineageDatasetFacet(ColumnLevelLineageUtils.java:32) <br /> +... 34 more</p> +
+ +
+
Assignees
+ <a href="https://github.com/pawel-big-lebowski">@pawel-big-lebowski</a> +
+ +
+
Labels
+ integration/spark, integration/databricks +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harshini Devathi + (harshini.devathi@tigeranalytics.com) +
+
2023-05-08 19:44:26
 +
+

*Thread Reply:* Thank you @Paweł Leszczyński. So, is this an issue with databricks. The issue thread says that it was able to work on AWS Glue. If so, is there some kind of solution to make it work on Databricks?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harshini Devathi + (harshini.devathi@tigeranalytics.com) +
+
2023-05-05 12:22:06
 +
+

Hello all, is there a way to get lineage in azure synapse analytics with openlineage

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2023-05-09 20:17:38
 +
+

*Thread Reply:* maybe @Will Johnson knows?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sai + (saivenkatesh161@gmail.com) +
+
2023-05-08 07:06:37
 +
+

Hi Team,

+ +

I have a usecase where we are connecting to Azure sql database from databricks to extract, transform and load data to delta tables. I could see the lineage is getting build, but there is no column level lineage through its 1:1 mapping from source. Could you please check and update on this.

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-05-09 10:06:02
 +
+

*Thread Reply:* There are few possible issues:

+ +
  1. The column-level lineage is not implemented for particular part of Spark LogicalPlan
  2. Azure SQL or Databricks have their own implementations of some Spark class, which does not exactly match our extractor. We've seen that happen
  3. You're using SQL JDBC connection with SELECT ** - in which case we can't do anything for now, since we don't know the input columns.
  4. Possibly something else 🙂 @Paweł Leszczyński might have an idea +To fully understand the issue, we'd have to see logs, LogicalPlan of the Spark job, or the job code itself
    5. +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-05-10 02:35:32
 +
+

*Thread Reply:* @Sai, providing a short code snippet that is able to reproduce this would be super helpful in examining that.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sai + (saivenkatesh161@gmail.com) +
+
2023-05-10 02:59:24
 +
+

*Thread Reply:* sure Pawel +Will share the code I used in sometime

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sai + (saivenkatesh161@gmail.com) +
+
2023-05-10 03:37:54
 +
+

*Thread Reply:* Here is the code we use.

+ +
+
Azure SQL_DeltaLake.py
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sai + (saivenkatesh161@gmail.com) +
+
2023-05-16 03:23:13
 +
+

*Thread Reply:* Hi Team, Any updates on this?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sai + (saivenkatesh161@gmail.com) +
+
2023-05-16 03:23:37
 +
+

*Thread Reply:* I tried with putting a sql query having column names in it, still the lineage didn't show up..

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-05-09 10:00:39
 +
+

2023-05-09T13:37:48.526698281Z java.lang.ClassCastException: class org.apache.spark.scheduler.ShuffleMapStage cannot be cast to class java.lang.Boolean (org.apache.spark.scheduler.ShuffleMapStage is in unnamed module of loader 'app'; java.lang.Boolean is in module java.base of loader 'bootstrap') +2023-05-09T13:37:48.526703550Z at scala.runtime.BoxesRunTime.unboxToBoolean(BoxesRunTime.java:87) +2023_05_09T13:37:48.526707874Z at scala.collection.LinearSeqOptimized.forall(LinearSeqOptimized.scala:85) +2023_05_09T13:37:48.526712381Z at scala.collection.LinearSeqOptimized.forall$(LinearSeqOptimized.scala:82) +2023_05_09T13:37:48.526716848Z at scala.collection.immutable.List.forall(List.scala:91) +2023_05_09T13:37:48.526723183Z at io.openlineage.spark.agent.lifecycle.OpenLineageRunEventBuilder.registerJob(OpenLineageRunEventBuilder.java:181) +2023_05_09T13:37:48.526727604Z at io.openlineage.spark.agent.lifecycle.SparkSQLExecutionContext.setActiveJob(SparkSQLExecutionContext.java:152) +2023_05_09T13:37:48.526732292Z at java.base/java.util.Optional.ifPresent(Unknown Source) +2023-05-09T13:37:48.526736352Z at io.openlineage.spark.agent.OpenLineageSparkListener.lambda$onJobStart$10(OpenLineageSparkListener.java:150) +2023_05_09T13:37:48.526740471Z at java.base/java.util.Optional.ifPresent(Unknown Source) +2023-05-09T13:37:48.526744887Z at io.openlineage.spark.agent.OpenLineageSparkListener.onJobStart(OpenLineageSparkListener.java:147) +2023_05_09T13:37:48.526750258Z at org.apache.spark.scheduler.SparkListenerBus.doPostEvent(SparkListenerBus.scala:37) +2023_05_09T13:37:48.526753454Z at org.apache.spark.scheduler.SparkListenerBus.doPostEvent$(SparkListenerBus.scala:28) +2023_05_09T13:37:48.526756235Z at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37) +2023_05_09T13:37:48.526759315Z at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37) +2023_05_09T13:37:48.526762133Z at org.apache.spark.util.ListenerBus.postToAll(ListenerBus.scala:117) +2023_05_09T13:37:48.526764941Z at org.apache.spark.util.ListenerBus.postToAll$(ListenerBus.scala:101) +2023_05_09T13:37:48.526767739Z at org.apache.spark.scheduler.AsyncEventQueue.super$postToAll(AsyncEventQueue.scala:105) +2023_05_09T13:37:48.526776059Z at org.apache.spark.scheduler.AsyncEventQueue.$anonfun$dispatch$1(AsyncEventQueue.scala:105) +2023_05_09T13:37:48.526778937Z at scala.runtime.java8.JFunction0$mcJ$sp.apply(JFunction0$mcJ$sp.java:23) +2023_05_09T13:37:48.526781728Z at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62) +2023_05_09T13:37:48.526786986Z at <a href="http://org.apache.spark.scheduler.AsyncEventQueue.org">org.apache.spark.scheduler.AsyncEventQueue.org</a>$apache$spark$scheduler$AsyncEventQueue$$dispatch(AsyncEventQueue.scala:100) +2023_05_09T13:37:48.526789893Z at org.apache.spark.scheduler.AsyncEventQueue$$anon$2.$anonfun$run$1(AsyncEventQueue.scala:96) +2023_05_09T13:37:48.526792722Z at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1446) +2023_05_09T13:37:48.526795463Z at org.apache.spark.scheduler.AsyncEventQueue$$anon$2.run(AsyncEventQueue.scala:96) +Hi, noticing this error message from OL... anyone know why its happening?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-05-09 10:02:25
 +
+

*Thread Reply:* @Anirudh Shrinivason what's your OL and Spark version?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-05-09 10:03:29
 +
+

*Thread Reply:* Some example job would also help, or logs/LogicalPlan 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-05-09 10:05:54
 +
+

*Thread Reply:* OL version is 0.23.0 and spark version is 3.3.1

+ + + +
+ 👍 Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-05-09 11:00:22
 +
+

*Thread Reply:* Hmm actually, it seems like the error is intermittent actually. I ran the same job again, but did not notice any errors this time...

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-05-10 02:27:19
 +
+

*Thread Reply:* This is interesting and it happens within a line: +job.finalStage().parents().forall(toScalaFn(stage -&gt; stageMap.put(stage.id(), stage))); +The result of stageMap.put is Stage and for some reason which I don't undestand it tries doing unboxToBoolean . We could rewrite that to: +job.finalStage().parents().forall(toScalaFn(stage -&gt; { +stageMap.put(stage.id(), stage) +return true; +})); +but this is so weird that it is intermittent and I don't get why is it happening.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-05-11 02:22:25
 +
+

*Thread Reply:* @Anirudh Shrinivason, please let us know if it is still a valid issue. If so, we can create an issue for that.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-05-11 03:11:13
 +
+

*Thread Reply:* Hi @Paweł Leszczyński Sflr. Yeah, I think if we are able to fix this, it'll be better. If this is the dedicated fix, then I can create an issue and raise an MR.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-05-11 04:12:46
 +
+

*Thread Reply:* Opened an issue and PR. Do help check if its okay thanks!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-05-11 04:29:33
 +
+

*Thread Reply:* please run ./gradlew spotlessApply with Java 8

+ + + +
+ ✅ Anirudh Shrinivason +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Pietro Brunetti + (pietrobrunetti89@gmail.com) +
+
2023-05-10 05:49:00
 +
+

Hi all, +I’m new to openlineage (and marquez) so I’m trying to figure out if it could be the right option form a client usecase in which: +• a legacy custom data catalog (mongo backend + Java API backend for fronted in angular) +• AS-IS component lineage realations are retrieve in a custom way from the each component’s APIs +• the customer would like to bring in a basic data lineage feature based on already published metadata that represent custom workloads type (batch,streaming,interactive ones) + data access pattern (no direct relation with the datasources right now but only a abstraction layer upon them) +I’d like to exploit directly Marquez as the metastore to publish metadata about datasource, workload (the workload is the declaration + business logic code deployed into the customer platform) once the component is deployed (e.g. the service that exposes the specific access pattern, or the workload custom declaration), but I saw the openlinage spec is based on strictly coupling between run,job and datasource; I mean I want to be able to publish one item at a time and then (maybe in a future release of the customer product) be able to exploit runtime lineage also

+ +

Am I in the right place? +Thanks anyway :)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-05-10 07:36:33
 +
+

*Thread Reply:* > I mean I want to be able to publish one item at a time and then (maybe in a future release of the customer product) be able to exploit runtime lineage also +This is not something that we support yet - there are definitely a lot of plans and preliminary work for that.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Pietro Brunetti + (pietrobrunetti89@gmail.com) +
+
2023-05-10 07:57:44
 +
+

*Thread Reply:* Thanks for the response, btw I already took a look at the current capabilities provided by openlineage, so my “hidden” question is how do achieve what the customer want to in order to be integrated in some way with openalineage+marquez? +should I choose between make or buy (between already supported platforms) and then try to align “static” (aka declarative) lineage metadata within the openlinage conceptual model?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-05-10 11:04:20
 +
+

@channel +This month’s TSC meeting is tomorrow at 10am PT. All are welcome! https://openlineage.slack.com/archives/C01CK9T7HKR/p1683213923529529

+
+ + +
+ + + } + + Michael Robinson + (https://openlineage.slack.com/team/U02LXF3HUN7) +
+ + + +
+
+ <p>@channel +This month’s TSC meeting is next Thursday, May 11th, at 10:00 am PT. The tentative agenda will be on the <a href="https://bit.ly/OLwiki">wiki</a>. More info and the meeting link can be found on the <a href="https://openlineage.io/meetings/">website</a>. All are welcome! Also, feel free to reply or DM me with discussion topics, agenda items, etc.</p> +
+ + + + + + +
Original URL: https://openlineage.slack.com/archives/C01CK9T7HKR/p1683213923529529
+ + +
+ + Slack Conversation +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Lukenoff + (john@jlukenoff.com) +
+
2023-05-11 12:59:42
 +
+

Does anyone here have experience with vendors in this space like Atlan or Manta? I’m advocating pretty heavily for OpenLineage at my company and have a strong suspicion that the LoE of enabling an equivalent solution from a vendor is equal or greater than that of OL/Marquez. Curious if anyone has first-hand experience with these tools they might be willing to share?

+ + + +
+ 👋 Eric Veleker +
+ +
+ 👀 Pietro Brunetti +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ernie Ostic + (ernie.ostic@getmanta.com) +
+
2023-05-11 13:58:28
 +
+

*Thread Reply:* Hi John. Great question! [full disclosure, I am with Manta 🙂 ]. I'll let others answer as to their experience with ourselves or many other vendors that provide lineage, but want to mention that a variety of our customers are finding it beneficial to bring code based static lineage together with the event-based runtime lineage that OpenLineage provides. This gives them the best of both worlds, for analyzing the lineage of their existing systems, where rich parsers already exist (for everything from legacy ETL tools, reporting tools, rdbms, etc.), to newer or home-grown technologies where applying OpenLineage is a viable alternative.

+ + + +
+ 👍 John Lukenoff +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Brad Paskewitz + (bradford.paskewitz@fivetran.com) +
+
2023-05-11 14:12:04
 +
+

*Thread Reply:* @Ernie Ostic do you see a single front-runner in the static lineage space? The static/event-based situation you describe is exactly the product roadmap I'm seeing here at Fivetran and I'm wondering if there's an opportunity to drive consensus towards a best-practice solution. If I'm not mistaken weren't there plans to start supporting non-run-based events in OL as well?

+ + + +
+ 👋 Eric Veleker +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Lukenoff + (john@jlukenoff.com) +
+
2023-05-11 14:16:34
 +
+

*Thread Reply:* I definitely like the idea of a 3rd party solution being complementary to OSS tools we can maintain ourselves while allowing us to offload maintenance effort where possible. Currently I have strong opinions on both sides of the build vs. buy aisle and this seems like the best of both worlds.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harel Shein + (harel.shein@gmail.com) +
+
2023-05-11 14:52:40
 +
+

*Thread Reply:* @Brad Paskewitz that’s 100% our plan to extend the OL spec to support “run-less” events. We want to collect that static metadata for Datasets and Jobs outside of the context of a run through OpenLineage. +happy to get your feedback here as well: https://github.com/OpenLineage/OpenLineage/pull/1839

+ + + +
+ :gratitude_thank_you: Brad Paskewitz +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Eric Veleker + (eric@atlan.com) +
+
2023-05-11 14:57:46
 +
+

*Thread Reply:* Hi @John Lukenoff. Here at Atlan we've been working with the OpenLineage community for quite some time to unlock the use case you describe. These efforts are adjacent to our ongoing integration with Fivetran. Happy to connect and give you a demo of what we've built and dig into your use case specifics.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Lukenoff + (john@jlukenoff.com) +
+
2023-05-12 11:26:32
 +
+

*Thread Reply:* Thanks all! These comments are really informative, it’s exciting to hear about vendors leaning into the project to let us continue to benefit from the tremendous progress being made by the community. Had a great discussion with Atlan yesterday and plan to connect with Manta next week to discuss our use-cases.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ernie Ostic + (ernie.ostic@getmanta.com) +
+
2023-05-12 12:34:32
 +
+

*Thread Reply:* Reach out anytime, John. @John Lukenoff Looking forward to engaging further with you on these topics!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harshini Devathi + (harshini.devathi@tigeranalytics.com) +
+
2023-05-12 11:15:10
 +
+

Hello all, I would like to have a new release of Openlineage as the new code base seems to have some issues fixed. I need these fixes for my project.

+ + + +
+ ➕ Michael Robinson, Maciej Obuchowski, Julien Le Dem, Jakub Dardziński, Anirudh Shrinivason, Harshini Devathi, Paweł Leszczyński, pankaj koti +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-05-12 11:19:02
 +
+

*Thread Reply:* Thank you for requesting an OpenLineage release. As stated here, three +1s from committers will authorize an immediate release. Our policy is not to release on Fridays, so the earliest we could initiate would be Monday.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harshini Devathi + (harshini.devathi@tigeranalytics.com) +
+
2023-05-12 13:12:43
 +
+

*Thread Reply:* A release on Monday is totally fine @Michael Robinson.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-05-15 08:37:39
 +
+

*Thread Reply:* The release will be initiated today. Thanks @Harshini Devathi

+ + + +
+ 👍 Anirudh Shrinivason, Harshini Devathi +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harshini Devathi + (harshini.devathi@tigeranalytics.com) +
+
2023-05-16 20:16:07
 +
+

*Thread Reply:* Appreciate it @Michael Robinson and thanks to all the committers for the prompt response

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-05-15 12:09:24
 +
+

@channel +We released OpenLineage 0.25.0, including: +Additions: +• Spark: merge into query support #1823 @pawel-big-lebowski +Fixes: +• Spark: fix JDBC query handling #1808 @nataliezeller1 +• Spark: filter Delta adaptive plan events #1830 @pawel-big-lebowski +• Spark: fix Java class cast exception #1844 @Anirudh181001 +• Flink: include missing fields of Openlineage events #1840 @pawel-big-lebowski +Plus doc changes and more. +Thanks to all the contributors! +For the details, see: +Release: https://github.com/OpenLineage/OpenLineage/releases/tag/0.25.0 +Changelog: https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md +Commit history: https://github.com/OpenLineage/OpenLineage/compare/0.24.0...0.25.0 +Maven: https://oss.sonatype.org/#nexus-search;quick~openlineage +PyPI: https://pypi.org/project/openlineage-python/

+ + + +
+ 🙌 Jakub Dardziński, Sai, pankaj koti, Paweł Leszczyński, Perttu Salonen, Maciej Obuchowski, Fraser Marlow, Ross Turk, Harshini Devathi, tati +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-05-16 14:03:01
 +
+

@channel +If you’re planning on being in San Francisco at the end of June — perhaps for this year’s Data+AI Summit — please stop by Astronomer’s offices on California Street on 6/27 for the first SF OpenLineage Meetup. We’ll be discussing spec changes planned for OpenLineage v1.0.0, progress on Airflow AIP 53, and more. Plus, dinner will be provided! For more info and to sign up, check out the OL blog. Join us!

+
+
openlineage.io
+ + + +
Meet Us in San Francisco on June 27th! | OpenLineage Docs
+
+ The first SF OpenLineage Meetup will take place on June 27th. +
+ + + + + + +
Original URL: https://openlineage.io/blog/sf-meetup
+ + + +
+ + + +
+ 🙌 alexandre bergere, Anirudh Shrinivason, Harel Shein, Willy Lulciuc, Jarek Potiuk, Ross Turk, John Lukenoff, tati +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2023-05-16 14:13:16
 +
+

*Thread Reply:* Can’t wait! 💯

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-05-17 00:09:23
 +
+

Hi, I've been noticing this error that is intermittently popping up in some of the spark jobs: +AsyncEventQueue: Dropping event from queue appStatus. This likely means one of the listeners is too slow and cannot keep up with the rate at which tasks are being started by the scheduler. +spark.scheduler.listenerbus.eventqueue.size Increasing this spark config did not help either. +Any ideas on how to mitigate this issue? Seeing this in spark 3.1.2 btw

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-05-17 01:58:28
 +
+

*Thread Reply:* Hi @Anirudh Shrinivason, are you able to send the OL events to console. This would let us confirm if the issue is related with event generation or emitting it and waiting for the backend to repond.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-05-17 01:59:03
 +
+

*Thread Reply:* Ahh okay sure. Let me see if I can do that

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sai + (saivenkatesh161@gmail.com) +
+
2023-05-17 01:52:15
 +
+

Hi Team

+ +

We are seeing an issue with OL configured cluster where delta table merge is failing with below error. It is running fine when we run with other clusters where OL is not configured. I ran it multiple times assuming its intermittent issue with memory, but it keeps on failing with same error. Attached the code for reference. We are using the latest release (0.25.0)

+ +

org.apache.spark.SparkException: Job aborted due to stage failure: Task serialization failed: java.lang.StackOverflowError

+ +

@Paweł Leszczyński @Michael Robinson

+ +
+
merge-test.py
+ + + + + + +
+ + +
+ 👀 Paweł Leszczyński +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sai + (saivenkatesh161@gmail.com) +
+
2023-05-19 03:55:51
 +
+

*Thread Reply:* Hi @Paweł Leszczyński

+ +

Thanks for fixing the issue and with new release merge is working. But I could not see any input and output datasets for this. Let me know if you need any further details to look into this.

+ +
},
+"job": {
+    "namespace": "openlineage_poc",
+    "name": "spark_ol_integration_execute_merge_into_command_edge",
+    "facets": {}
+},
+"inputs": [],
+"outputs": [],
+
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-05-19 04:00:01
 +
+

*Thread Reply:* Oh man, it's just that vanilla spark differs from the one available in databricks platform. our integration tests do verify behaviour on vanilla spark which still leaves a possibility for inconsistency. will need to get back to it then at some time.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sai + (saivenkatesh161@gmail.com) +
+
2023-06-02 02:11:28
 +
+

*Thread Reply:* Hi @Paweł Leszczyński

+ +

Did you get chance to look into this issue?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-06-02 02:13:18
 +
+

*Thread Reply:* Hi Sai, I am going back to spark. I am working on support for Spark 3.4, which is going to add some event filtering on internal delta operations that trigger unncecessarly the events

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-06-02 02:13:28
 +
+

*Thread Reply:* this may be releated to issue you created

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-06-02 02:14:13
 +
+

*Thread Reply:* I do have planned creating integration test for databricks which will be helpful to tackle the issues you raised

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-06-02 02:14:27
 +
+

*Thread Reply:* so yes, I am looking at the Spark

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sai + (saivenkatesh161@gmail.com) +
+
2023-06-02 02:20:06
 +
+

*Thread Reply:* thanks much Pawel.. I am looking more into the merge part as first priority as we use is frequently.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-06-02 02:21:01
 +
+

*Thread Reply:* I know, this is important.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-06-02 02:21:14
 +
+

*Thread Reply:* It just need still some time.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-06-02 02:21:46
 +
+

*Thread Reply:* thank you for your patience and being so proactive on those issues.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sai + (saivenkatesh161@gmail.com) +
+
2023-06-02 02:22:12
 +
+

*Thread Reply:* no problem.. Please do keep us posted with updates..

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-05-17 10:47:27
 +
+

Our recent Openlineage release (0.25.0) proved there are many users that use Openlineage on databricks, which is incredible. I am super happy to know that, although we realised that as a side effect of a bug. Sorry for that.

+ +

I would like to opt for a new release which contains PR #1858 and should unblock databricks users.

+ + + +
+ ➕ Paweł Leszczyński, Maciej Obuchowski, Harshini Devathi, Jakub Dardziński, Sai, Anirudh Shrinivason, Anbarasi +
+ +
+ 👍 Michael Robinson +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-05-18 10:26:48
 +
+

*Thread Reply:* The release request has been approved and will be initiated shortly.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-05-17 22:49:41
 +
+

Actually, I noticed a few other stack overflow errors on 0.25.0. Let me raise an issue. Could we cut a release once this bug are fixed too please?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-05-18 02:29:55
 +
+

*Thread Reply:* Hi Anirudh, I saw your issue and I think it is the same one as solved within #1858. Are you able to reproduce it on a version built on the top of main?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-05-18 06:21:05
 +
+

*Thread Reply:* Hi I haven't managed to try with the main branch. But if its the same error then all's good! If the error resurfaces then we can look into it.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Lovenish Goyal + (lovenishgoyal@gmail.com) +
+
2023-05-18 02:21:13
 +
+

Hi All,

+ +

We are in POC phase OpenLineage integration with our core DBT, can anyone help me with document to start with.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-05-18 02:28:31
 +
+

*Thread Reply:* I know this one: https://openlineage.io/docs/integrations/dbt

+
+
openlineage.io
+ + + +
dbt | OpenLineage Docs
+
+ dbt (data build tool) is a powerful transformation engine. It operates on data already within a warehouse, making it easy for data engineers to build complex pipelines from the comfort of their laptops. While it doesn’t perform extraction and loading of data, it’s extremely powerful at transformations. +
+ + + + + + +
Original URL: https://openlineage.io/docs/integrations/dbt
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Lovenish Goyal + (lovenishgoyal@gmail.com) +
+
2023-05-18 02:41:39
 +
+

*Thread Reply:* Hi @Paweł Leszczyński Thanks for the revert, I tried same but facing below issue

+ +

requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=5000): Max retries exceeded with url:

+ +

Looks like I need to start the service

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-05-18 02:44:09
 +
+

*Thread Reply:* @Lovenish Goyal, exactly. You need to start Marquez. +More about it: https://marquezproject.ai/quickstart

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harel Shein + (harel.shein@gmail.com) +
+
2023-05-18 10:27:52
 +
+

*Thread Reply:* @Lovenish Goyal how are you running dbt core currently?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Lovenish Goyal + (lovenishgoyal@gmail.com) +
+
2023-05-19 01:55:20
 +
+

*Thread Reply:* Trying but facing issue while running marquezproject @Jakub Dardziński

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Lovenish Goyal + (lovenishgoyal@gmail.com) +
+
2023-05-19 01:56:03
 +
+

*Thread Reply:* @Harel Shein we have created custom docker image of DBT + Airflow and running it on an EC2

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harel Shein + (harel.shein@gmail.com) +
+
2023-05-19 09:05:31
 +
+

*Thread Reply:* for running dbt core on Airflow, we have a utility that helps develop dbt natively on Airflow. There’s also built in support for collecting lineage if you have the airflow-openlineage provider installed. +https://astronomer.github.io/astronomer-cosmos/#quickstart

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harel Shein + (harel.shein@gmail.com) +
+
2023-05-19 09:06:30
 +
+

*Thread Reply:* RE issues running Marquez, can you share what those are? I’m guessing that since you are running both of them in individual docker images, the airflow deployment might not be able to communicate with the Marquez endpoints?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-05-19 09:06:53
 +
+

*Thread Reply:* @Harel Shein I've already helped with running Marquez 🙂

+ + + +
+ :first_place_medal: Harel Shein, Paweł Leszczyński, Michael Robinson +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anbarasi + (anbujothi@gmail.com) +
+
2023-05-18 02:29:53
 +
+

@Paweł Leszczyński We are facing the following issue with Azure databricks. When we use aggregate functions in databricks notebooks, Open lineage is not able to provide column level lineage. I understand its an existing issue. Can you please let me know in which release this issue will be fixed ? It is one of the most needed feature for us to implement openlineage in our current project. Kindly let me know

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-05-18 02:34:35
 +
+

*Thread Reply:* I am not sure if this is the same. If you see OL events collected with column-lineage missing, then it's a different one.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-05-18 02:41:11
 +
+

*Thread Reply:* Please also be aware, that it is extremely helpful to investigate the issues on your own before creating them.

+ +

Our integration traverses spark's logical plans and extracts lineage events from plan nodes that it understands. Some plan nodes are not supported yet and, from my experience, when working on an issue, 80% of time is spent on reproducing the scenario.

+ +

So, if you are able to produce a minimal amount of spark code that reproduces an issue, this can be extremely helpful and significantly speed up resolution time.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anbarasi + (anbujothi@gmail.com) +
+
2023-05-18 03:52:30
 +
+

*Thread Reply:* @Paweł Leszczyński Thanks for the prompt response.

+ +

Provided sample codes with and without using aggregate functions and its respective lineage events for reference.

+ +

  1. Please find the code without using aggregate function: + finaldf=spark.sql(""" + select productid + ,OrderQty as TotalOrderQty + ,ReceivedQty as TotalReceivedQty + ,StockedQty as TotalStockedQty + ,RejectedQty as TotalRejectedQty + from openlineagepoc.purchaseorder + --group by productid + order by productid""")

    + +
       final_df.write.mode("overwrite").saveAsTable("openlineage_poc.productordertest1")
+
    2. +
+ +

Please find the Openlineage Events for the Input, Ouput datasets. We could find the column lineage in this.

+ +

"inputs": [ + { + "namespace": "dbfs", + "name": "/mnt/dlzones/warehouse/openlineagepoc/gold/purchaseorder", + "facets": { + "dataSource": { + "producer": "https://github.com/OpenLineage/OpenLineage/tree/0.25.0/integration/spark", + "schemaURL": "https://openlineage.io/spec/facets/1-0-0/DatasourceDatasetFacet.json#/$defs/DatasourceDatasetFacet", + "name": "dbfs", + "uri": "dbfs" + }, + "schema": { + "producer": "https://github.com/OpenLineage/OpenLineage/tree/0.25.0/integration/spark", + "schemaURL": "https://openlineage.io/spec/facets/1-0-0/SchemaDatasetFacet.json#/$defs/SchemaDatasetFacet", + "fields": [ + { + "name": "PurchaseOrderID", + "type": "integer" + }, + { + "name": "PurchaseOrderDetailID", + "type": "integer" + }, + { + "name": "DueDate", + "type": "timestamp" + }, + { + "name": "OrderQty", + "type": "short" + }, + { + "name": "ProductID", + "type": "integer" + }, + { + "name": "UnitPrice", + "type": "decimal(19,4)" + }, + { + "name": "LineTotal", + "type": "decimal(19,4)" + }, + { + "name": "ReceivedQty", + "type": "decimal(8,2)" + }, + { + "name": "RejectedQty", + "type": "decimal(8,2)" + }, + { + "name": "StockedQty", + "type": "decimal(9,2)" + }, + { + "name": "RevisionNumber", + "type": "integer" + }, + { + "name": "Status", + "type": "integer" + }, + { + "name": "EmployeeID", + "type": "integer" + }, + { + "name": "NationalIDNumber", + "type": "string" + }, + { + "name": "JobTitle", + "type": "string" + }, + { + "name": "Gender", + "type": "string" + }, + { + "name": "MaritalStatus", + "type": "string" + }, + { + "name": "VendorID", + "type": "integer" + }, + { + "name": "ShipMethodID", + "type": "integer" + }, + { + "name": "ShipMethodName", + "type": "string" + }, + { + "name": "ShipMethodrowguid", + "type": "string" + }, + { + "name": "OrderDate", + "type": "timestamp" + }, + { + "name": "ShipDate", + "type": "timestamp" + }, + { + "name": "SubTotal", + "type": "decimal(19,4)" + }, + { + "name": "TaxAmt", + "type": "decimal(19,4)" + }, + { + "name": "Freight", + "type": "decimal(19,4)" + }, + { + "name": "TotalDue", + "type": "decimal(19,4)" + } + ] + }, + "symlinks": { + "producer": "https://github.com/OpenLineage/OpenLineage/tree/0.25.0/integration/spark", + "schemaURL": "https://openlineage.io/spec/facets/1-0-0/SymlinksDatasetFacet.json#/$defs/SymlinksDatasetFacet", + "identifiers": [ + { + "namespace": "/mnt/dlzones/warehouse/openlineagepoc/gold", + "name": "openlineagepoc.purchaseorder", + "type": "TABLE" + } + ] + } + }, + "inputFacets": {} + } + ], + "outputs": [ + { + "namespace": "dbfs", + "name": "/mnt/dlzones/warehouse/openlineagepoc/productordertest1", + "facets": { + "dataSource": { + "producer": "https://github.com/OpenLineage/OpenLineage/tree/0.25.0/integration/spark", + "schemaURL": "https://openlineage.io/spec/facets/1-0-0/DatasourceDatasetFacet.json#/$defs/DatasourceDatasetFacet", + "name": "dbfs", + "uri": "dbfs" + }, + "schema": { + "producer": "https://github.com/OpenLineage/OpenLineage/tree/0.25.0/integration/spark", + "schemaURL": "https://openlineage.io/spec/facets/1-0-0/SchemaDatasetFacet.json#/$defs/SchemaDatasetFacet", + "fields": [ + { + "name": "productid", + "type": "integer" + }, + { + "name": "TotalOrderQty", + "type": "short" + }, + { + "name": "TotalReceivedQty", + "type": "decimal(8,2)" + }, + { + "name": "TotalStockedQty", + "type": "decimal(9,2)" + }, + { + "name": "TotalRejectedQty", + "type": "decimal(8,2)" + } + ] + }, + "storage": { + "producer": "https://github.com/OpenLineage/OpenLineage/tree/0.25.0/integration/spark", + "schemaURL": "https://openlineage.io/spec/facets/1-0-0/StorageDatasetFacet.json#/$defs/StorageDatasetFacet", + "storageLayer": "unity", + "fileFormat": "parquet" + }, + "columnLineage": { + "producer": "https://github.com/OpenLineage/OpenLineage/tree/0.25.0/integration/spark", + "schemaURL": "https://openlineage.io/spec/facets/1-0-1/ColumnLineageDatasetFacet.json#/$defs/ColumnLineageDatasetFacet", + "fields": { + "productid": { + "inputFields": [ + { + "namespace": "dbfs", + "name": "/mnt/dlzones/warehouse/openlineagepoc/gold/purchaseorder", + "field": "ProductID" + } + ] + }, + "TotalOrderQty": { + "inputFields": [ + { + "namespace": "dbfs", + "name": "/mnt/dlzones/warehouse/openlineagepoc/gold/purchaseorder", + "field": "OrderQty" + } + ] + }, + "TotalReceivedQty": { + "inputFields": [ + { + "namespace": "dbfs", + "name": "/mnt/dlzones/warehouse/openlineagepoc/gold/purchaseorder", + "field": "ReceivedQty" + } + ] + }, + "TotalStockedQty": { + "inputFields": [ + { + "namespace": "dbfs", + "name": "/mnt/dlzones/warehouse/openlineagepoc/gold/purchaseorder", + "field": "StockedQty" + } + ] + }, + "TotalRejectedQty": { + "inputFields": [ + { + "namespace": "dbfs", + "name": "/mnt/dlzones/warehouse/openlineagepoc/gold/purchaseorder", + "field": "RejectedQty" + } + ] + } + } + }, + "symlinks": { + "producer": "https://github.com/OpenLineage/OpenLineage/tree/0.25.0/integration/spark", + "schemaURL": "https://openlineage.io/spec/facets/1-0-0/SymlinksDatasetFacet.json#/$defs/SymlinksDatasetFacet", + "identifiers": [ + { + "namespace": "/mnt/dlzones/warehouse/openlineagepoc", + "name": "openlineagepoc.productordertest1", + "type": "TABLE" + } + ] + }, + "lifecycleStateChange": { + "producer": "https://github.com/OpenLineage/OpenLineage/tree/0.25.0/integration/spark", + "_schemaURL": "https://openlineage.io/spec/facets/1-0-0/LifecycleStateChangeDatasetFacet.json#/$defs/LifecycleStateChangeDatasetFacet", + "lifecycleStateChange": "OVERWRITE" + } + }, + "outputFacets": {} + } + ]

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anbarasi + (anbujothi@gmail.com) +
+
2023-05-18 03:55:04
 +
+

*Thread Reply:* 2. Please find the code using aggregate function:

+ +
    final_df=spark.sql("""
+    select productid
+    ,sum(OrderQty) as TotalOrderQty
+    ,sum(ReceivedQty) as TotalReceivedQty
+    ,sum(StockedQty) as TotalStockedQty
+    ,sum(RejectedQty) as TotalRejectedQty
+    from openlineage_poc.purchaseorder
+    group by productid
+    order by productid""")
+
+    final_df.write.mode("overwrite").saveAsTable("openlineage_poc.productordertest2")
+
+ +

Please find the Openlineage Events for the Input, Ouput datasets. We couldnt find the column lineage in output section. Please find the sample

+ +

"inputs": [ + { + "namespace": "dbfs", + "name": "/mnt/dlzones/warehouse/openlineagepoc/gold/purchaseorder", + "facets": { + "dataSource": { + "producer": "https://github.com/OpenLineage/OpenLineage/tree/0.25.0/integration/spark", + "schemaURL": "https://openlineage.io/spec/facets/1-0-0/DatasourceDatasetFacet.json#/$defs/DatasourceDatasetFacet", + "name": "dbfs", + "uri": "dbfs" + }, + "schema": { + "producer": "https://github.com/OpenLineage/OpenLineage/tree/0.25.0/integration/spark", + "schemaURL": "https://openlineage.io/spec/facets/1-0-0/SchemaDatasetFacet.json#/$defs/SchemaDatasetFacet", + "fields": [ + { + "name": "PurchaseOrderID", + "type": "integer" + }, + { + "name": "PurchaseOrderDetailID", + "type": "integer" + }, + { + "name": "DueDate", + "type": "timestamp" + }, + { + "name": "OrderQty", + "type": "short" + }, + { + "name": "ProductID", + "type": "integer" + }, + { + "name": "UnitPrice", + "type": "decimal(19,4)" + }, + { + "name": "LineTotal", + "type": "decimal(19,4)" + }, + { + "name": "ReceivedQty", + "type": "decimal(8,2)" + }, + { + "name": "RejectedQty", + "type": "decimal(8,2)" + }, + { + "name": "StockedQty", + "type": "decimal(9,2)" + }, + { + "name": "RevisionNumber", + "type": "integer" + }, + { + "name": "Status", + "type": "integer" + }, + { + "name": "EmployeeID", + "type": "integer" + }, + { + "name": "NationalIDNumber", + "type": "string" + }, + { + "name": "JobTitle", + "type": "string" + }, + { + "name": "Gender", + "type": "string" + }, + { + "name": "MaritalStatus", + "type": "string" + }, + { + "name": "VendorID", + "type": "integer" + }, + { + "name": "ShipMethodID", + "type": "integer" + }, + { + "name": "ShipMethodName", + "type": "string" + }, + { + "name": "ShipMethodrowguid", + "type": "string" + }, + { + "name": "OrderDate", + "type": "timestamp" + }, + { + "name": "ShipDate", + "type": "timestamp" + }, + { + "name": "SubTotal", + "type": "decimal(19,4)" + }, + { + "name": "TaxAmt", + "type": "decimal(19,4)" + }, + { + "name": "Freight", + "type": "decimal(19,4)" + }, + { + "name": "TotalDue", + "type": "decimal(19,4)" + } + ] + }, + "symlinks": { + "producer": "https://github.com/OpenLineage/OpenLineage/tree/0.25.0/integration/spark", + "schemaURL": "https://openlineage.io/spec/facets/1-0-0/SymlinksDatasetFacet.json#/$defs/SymlinksDatasetFacet", + "identifiers": [ + { + "namespace": "/mnt/dlzones/warehouse/openlineagepoc/gold", + "name": "openlineagepoc.purchaseorder", + "type": "TABLE" + } + ] + } + }, + "inputFacets": {} + } + ], + "outputs": [ + { + "namespace": "dbfs", + "name": "/mnt/dlzones/warehouse/openlineagepoc/productordertest2", + "facets": { + "dataSource": { + "producer": "https://github.com/OpenLineage/OpenLineage/tree/0.25.0/integration/spark", + "schemaURL": "https://openlineage.io/spec/facets/1-0-0/DatasourceDatasetFacet.json#/$defs/DatasourceDatasetFacet", + "name": "dbfs", + "uri": "dbfs" + }, + "schema": { + "producer": "https://github.com/OpenLineage/OpenLineage/tree/0.25.0/integration/spark", + "schemaURL": "https://openlineage.io/spec/facets/1-0-0/SchemaDatasetFacet.json#/$defs/SchemaDatasetFacet", + "fields": [ + { + "name": "productid", + "type": "integer" + }, + { + "name": "TotalOrderQty", + "type": "long" + }, + { + "name": "TotalReceivedQty", + "type": "decimal(18,2)" + }, + { + "name": "TotalStockedQty", + "type": "decimal(19,2)" + }, + { + "name": "TotalRejectedQty", + "type": "decimal(18,2)" + } + ] + }, + "storage": { + "producer": "https://github.com/OpenLineage/OpenLineage/tree/0.25.0/integration/spark", + "schemaURL": "https://openlineage.io/spec/facets/1-0-0/StorageDatasetFacet.json#/$defs/StorageDatasetFacet", + "storageLayer": "unity", + "fileFormat": "parquet" + }, + "symlinks": { + "producer": "https://github.com/OpenLineage/OpenLineage/tree/0.25.0/integration/spark", + "schemaURL": "https://openlineage.io/spec/facets/1-0-0/SymlinksDatasetFacet.json#/$defs/SymlinksDatasetFacet", + "identifiers": [ + { + "namespace": "/mnt/dlzones/warehouse/openlineagepoc", + "name": "openlineagepoc.productordertest2", + "type": "TABLE" + } + ] + }, + "lifecycleStateChange": { + "producer": "https://github.com/OpenLineage/OpenLineage/tree/0.25.0/integration/spark", + "schemaURL": "https://openlineage.io/spec/facets/1-0-0/LifecycleStateChangeDatasetFacet.json#/$defs/LifecycleStateChangeDatasetFacet", + "lifecycleStateChange": "OVERWRITE" + } + }, + "outputFacets": {} + } + ]

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-05-18 04:09:17
 +
+

*Thread Reply:* amazing. https://github.com/OpenLineage/OpenLineage/issues/1861

+
+ + + + +
#1861 Column lineage not working for aggregates on Databricks
+
+ <p>The code</p> + +<pre><code>final_df=spark.sql(""" + select productid + ,OrderQty as TotalOrderQty + ,ReceivedQty as TotalReceivedQty + ,StockedQty as TotalStockedQty + ,RejectedQty as TotalRejectedQty + from openlineage_poc.purchaseorder + --group by productid + order by productid""") + final_df.write.mode("overwrite").saveAsTable("openlineage_poc.productordertest1") +</code></pre> + +<p>generates OL event with column lineage. However, in case of group by enabled</p> + +<pre><code> final_df=spark.sql(""" + select productid + ,sum(OrderQty) as TotalOrderQty + ,sum(ReceivedQty) as TotalReceivedQty + ,sum(StockedQty) as TotalStockedQty + ,sum(RejectedQty) as TotalRejectedQty + from openlineage_poc.purchaseorder + group by productid + order by productid""") + final_df.write.mode("overwrite").saveAsTable("openlineage_poc.productordertest2") +</code></pre> + +<p>it's not present anymore. <br /> +<a href="https://github.com/OpenLineage/OpenLineage/files/11505766/lineage.txt">lineage.txt</a></p> + +<p>Requested on slack: <a href="https://openlineage.slack.com/archives/C01CK9T7HKR/p1684391393701839">https://openlineage.slack.com/archives/C01CK9T7HKR/p1684391393701839</a></p> +
+ +
+
Assignees
+ <a href="https://github.com/pawel-big-lebowski">@pawel-big-lebowski</a> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anbarasi + (anbujothi@gmail.com) +
+
2023-05-18 04:11:56
 +
+

*Thread Reply:* Thanks for considering the request and looking into it

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-05-18 13:12:35
 +
+

@channel +We released OpenLineage 0.26.0, including: +Additions: +• Proxy: Fluentd proxy support (experimental) #1757 @pawel-big-lebowski +Changes: +• Python client: use Hatchling over setuptools to orchestrate Python env setup #1856 @gaborbernat +Fixes: +• Spark: fix logicalPlan serialization issue on Databricks #1858 @pawel-big-lebowski +Plus an additional fix, doc changes and more. +Thanks to all the contributors, including new contributor @gaborbernat! +For the details, see: +Release: https://github.com/OpenLineage/OpenLineage/releases/tag/0.26.0 +Changelog: https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md +Commit history: https://github.com/OpenLineage/OpenLineage/compare/0.25.0...0.26.0 +Maven: https://oss.sonatype.org/#nexus-search;quick~openlineage +PyPI: https://pypi.org/project/openlineage-python/

+ + + +
+ ❤️ Paweł Leszczyński, Maciej Obuchowski, Anirudh Shrinivason, Peter Hicks, pankaj koti +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Bramha Aelem + (bramhaaelem@gmail.com) +
+
2023-05-18 14:42:49
 +
+

Hi Team , can someone please address https://github.com/OpenLineage/OpenLineage/issues/1866.

+
+ + + + +
#1866 Reg: Lineage Representation With Available Datasets Generated by the Jobs
+
+ <p><strong>Purpose:</strong> <br /> +As part of the data ingestion pipeline process. process#1 generating the tables using spark. How can refer process#1 as an input for my process#2. When we specify them in lineage process by refering spark namespace where the datasets resides it's not showling accurately on Marquez UI.</p> + +<p><strong>Proposed implementation</strong> <br /> +If N number of jobs generates the datasets under respective namespaces. We should refer the same in the lineage representation process as is insated of them craeting again and again under the job execution namespace.</p> +
+ +
+
Labels
+ proposal +
+ +
+
Comments
+ 1 +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2023-05-18 20:13:09
 +
+

*Thread Reply:* Hi @Bramha Aelem I replied in the ticket. Thank you for opening it.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Bramha Aelem + (bramhaaelem@gmail.com) +
+
2023-05-18 21:15:30
 +
+

*Thread Reply:* Hi @Julien Le Dem - Thanks for quick response. I replied in the ticket. Please let me know if you need any more details.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-05-19 02:13:57
 +
+

*Thread Reply:* Hi @Bramha Aelem - asked for more details in the ticket.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Bramha Aelem + (bramhaaelem@gmail.com) +
+
2023-05-22 11:08:58
 +
+

*Thread Reply:* Hi @Paweł Leszczyński - I replied with necessary details in the ticket. Please let me know if you need any more details.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Bramha Aelem + (bramhaaelem@gmail.com) +
+
2023-05-25 15:22:42
 +
+

*Thread Reply:* Hi @Paweł Leszczyński - any further updates on issue?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-05-26 01:56:47
 +
+

*Thread Reply:* hi @Bramha Aelem, i was out of office for a few days. will get back into this soon. thanks for update.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Bramha Aelem + (bramhaaelem@gmail.com) +
+
2023-05-27 18:46:27
 +
+

*Thread Reply:* Hi @Paweł Leszczyński -Thanks for your reply. will wait for your response to proceed further on the issue.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Bramha Aelem + (bramhaaelem@gmail.com) +
+
2023-06-02 19:29:08
 +
+

*Thread Reply:* Hi @Paweł Leszczyński -Hope you are doing well. Did you get a chance to look into the samples which are provided in the ticket. Kindly let me know your observations/recommendations.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Bramha Aelem + (bramhaaelem@gmail.com) +
+
2023-06-09 12:43:54
 +
+

*Thread Reply:* Hi @Paweł Leszczyński - Hope you are doing well. Did you get a chance to look into the samples which are provided in the ticket. Kindly let me know your observations/recommendations.

+ + + +
+ 👀 Paweł Leszczyński +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Bramha Aelem + (bramhaaelem@gmail.com) +
+
2023-07-06 10:29:01
 +
+

*Thread Reply:* Hi @Paweł Leszczyński - Good day. Did you get a chance to look into query which I have posted. can you please provide any thoughts on my observation/query.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Doe + (adarsh.pansari@tigeranalytics.com) +
+
2023-05-19 03:42:21
 +
+

Hello Everyone, I was trying to integrate openlineage with Jupyter Notebooks, I followed the docs but when I run the sample notebook I am getting an error +23/05/19 07:39:08 ERROR EventEmitter: Could not emit lineage w/ exception +Can someone Please help understand why am I getting this error and the resolution.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-05-19 03:49:27
 +
+

*Thread Reply:* Hello @John Doe, this mostly means there's somehting wrong with your transport config for emitting Openlineage events.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-05-19 03:49:41
 +
+

*Thread Reply:* what do you want to do with the events?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Doe + (adarsh.pansari@tigeranalytics.com) +
+
2023-05-19 04:10:24
 +
+

*Thread Reply:* Hi @Paweł Leszczyński, I am working on a PoC to understand the use cases of OL and how it build Lineages.

+ +

As for the transport config I am using the codes from the documentation to setup OL. +https://openlineage.io/docs/integrations/spark/quickstart_local

+ +

Apart from these I dont have anything else in my nb

+
+
openlineage.io
+ + + +
Quickstart with Jupyter | OpenLineage Docs
+
+ Trying out the Spark integration is super easy if you already have Docker Desktop and git installed. +
+ + + + + + +
Original URL: https://openlineage.io/docs/integrations/spark/quickstart_local
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-05-19 04:38:58
 +
+

*Thread Reply:* ok, I am wondering if what you experience isn't similar to issue #1860. Could you try openlineage 0.23.0 to see if get the same error?

+ +

<https://github.com/OpenLineage/OpenLineage/issues/1860>

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Doe + (adarsh.pansari@tigeranalytics.com) +
+
2023-05-19 10:05:59
 +
+

*Thread Reply:* I tried with 0.23.0 still getting the same error

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Doe + (adarsh.pansari@tigeranalytics.com) +
+
2023-05-23 02:34:52
 +
+

*Thread Reply:* @Paweł Leszczyński any other way I can try to setup. The issue still persists

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-05-29 03:53:04
 +
+

*Thread Reply:* hmyy, I've just redone steps from https://openlineage.io/docs/integrations/spark/quickstart_local with 0.26.0 and could not reproduce behaviour you encountered.

+
+
openlineage.io
+ + + +
Quickstart with Jupyter | OpenLineage Docs
+
+ Trying out the Spark integration is super easy if you already have Docker Desktop and git installed. +
+ + + + + + +
Original URL: https://openlineage.io/docs/integrations/spark/quickstart_local
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tom van Eijk + (t.m.h.vaneijk@tilburguniversity.edu) +
+
2023-05-22 09:41:55
 +
+

Hello Team!!! A part of my master thesis's case study was about data lineage in data mesh and how open-source initiatives such as OpenLineage and Marquez can realize this. Can you recommend me some material that can support the writing part of my thesis (more context: I tried to extract lineage events from Snowflake through Airlfow and used Docker Compose on EC2 to connect Airflow and the Marquez webserver)? We will divide the thesis into a few academic papers to make the content more digestible and publish one of them soon hopefully!

+ + + +
+ 👍 Ernie Ostic, Maciej Obuchowski, Ross Turk, Michael Robinson +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-05-22 16:34:00
 +
+

*Thread Reply:* Tom, thanks for your question. This is really exciting! I assume you’ve already started checking out the docs, but there are many other resources on the website, as well (on the blog and resources pages in particular). And don’t skip the YouTube channel, where we’ve recently started to upload short, more digestible excerpts from the community meetings. Please keep us updated as you make progress!

+ + + +
+ 👀 Tom van Eijk +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tom van Eijk + (t.m.h.vaneijk@tilburguniversity.edu) +
+
2023-05-22 16:48:06
 +
+

*Thread Reply:* Hi Michael! Thank you so much for sending these resources! I've been working on this thesis for quite some time already and it's almost finished. I just needed some additional information to help in accurately describing some of the processes in OpenLineage and Marquez. Will send you the case study chapter later this week to get some feedback if possible. Keep you posted on things such as publication! Perhaps it can make OpenLineage even more popular than it already is 😉

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-05-22 16:52:18
 +
+

*Thread Reply:* Yes, please share it! Looking forward to checking it out. Super cool!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ernie Ostic + (ernie.ostic@getmanta.com) +
+
2023-05-22 09:57:50
 +
+

Hi Tom. Good luck. Sounds like a great case study. You might want to compare and contrast various kinds of lineage solutions....all of which complement each other, as well as having their own pros and cons. (...code based lineage via parsing, data similarity lineage, run-time lineage reporting, etc.) ...and then focus on open source and OpenLineage with Marquez in particular.

+ + + +
+ 🙏 Tom van Eijk +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tom van Eijk + (t.m.h.vaneijk@tilburguniversity.edu) +
+
2023-05-22 10:04:44
 +
+

*Thread Reply:* Thank you so much Ernie! That sounds like a very interesting direction to keep in mind during research!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-05-22 16:37:44
 +
+

@channel +For an easily digestible recap of recent events, communications and releases in the community, please sign up for our new monthly newsletter! Look for it in your inbox soon.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Bernat Gabor + (gaborjbernat@gmail.com) +
+
2023-05-22 23:32:16
 +
+

looking here https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.json#L64 it show that the schemaURL must be set, but then the examples in https://openlineage.io/getting-started#step-1-start-a-run do not contain it, is this a bug, expected? 😄

+
+
openlineage.io
+ + + +
Getting Started | OpenLineage Docs
+
+ Data lineage is the foundation for a new generation of powerful, context-aware data tools and best practices. OpenLineage enables consistent collection of lineage metadata, creating a deeper understanding of how data is produced and used. +
+ + + + + + +
Original URL: https://openlineage.io/getting-started#step-1-start-a-run
+ + + +
+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.json | OpenLineage.json>
+
+ <pre><code> "schemaURL" +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-05-23 07:24:09
 +
+

*Thread Reply:* yeah, it's a bug

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Bernat Gabor + (gaborjbernat@gmail.com) +
+
2023-05-23 12:00:48
 +
+

*Thread Reply:* so it's optional then? 😄 or bug in the example?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Bernat Gabor + (gaborjbernat@gmail.com) +
+
2023-05-23 12:02:09
 +
+

I noticed that DataQualityAssertionsDatasetFacet inherits from InputDatasetFacet, https://github.com/OpenLineage/OpenLineage/blob/main/spec/facets/DataQualityAssertionsDatasetFacet.json though I think should do from DatasetFacet like all else 🤔

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-05-23 14:20:09
 +
+

@channel +Two years ago last Saturday, we released the first version of OpenLineage, a test release of the Python client. So it seemed like an appropriate time to share our first annual ecosystem survey, which is both a milestone in the project’s growth and an important effort to set our course. This survey has been designed to help us learn more about who is using OpenLineage, what your lineage needs are, and what new tools you hope the project will support. Thank you in advance for taking the time to share your opinions and vision for the project! (Please note: the survey might seem longer than it actually is due to the large number of optional questions. Not all questions apply to all use cases.)

+
+
Google Docs
+ + + +
2023 OpenLineage Ecosystem Survey
+
+ Our annual survey is designed to help us learn more about who is using OpenLineage, what your lineage needs are, and what new tools you hope the project will support. Your responses will help shape the future of the project. Thank you for participating! +
+ + + + + + + + +
Original URL: https://bit.ly/ecosystem_survey
+ + + +
+ + + +
+ 🙌 Harel Shein, Maciej Obuchowski, Atif Tahir, Peter Hicks, Tamara Fingerlin, Paweł Leszczyński +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sharanya Santhanam + (santhanamsharanya@gmail.com) +
+
2023-05-23 18:59:46
 +
+

Open Lineage Spark Integration our spark workloads on Spark 2.4 are correctly setting .config("spark.sql.catalogImplementation", "hive") however sql queries for CREATE/INSERT INTO dont recoognize the datasets as “Hive”. As per https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/supported-commands.md USING HIVE is needed for appropriate parsing. Why is that the case ? Why cant HQL format for CREATE/INSERT be supported?

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/supported-commands.md | supported-commands.md>
+
+ <p><strong>Supported Commands</strong></p> + +<p>The page lists Spark commands which are supported by OpenLineage.</p> + +<p><strong>V2 commands</strong></p> + +<p>Spark commands to integrate with external catalogs like <code>iceberg</code>, <code>delta</code>, etc.</p> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sharanya Santhanam + (santhanamsharanya@gmail.com) +
+
2023-05-23 19:01:43
 +
+

*Thread Reply:* @Michael Collado wondering if you could shed some light here

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-05-24 05:39:01
 +
+

*Thread Reply:* can you show logical plan of your Spark job? I think using hive is not the most important part, but whether job's LogicalPlan parses to CreateHiveTableAsSelectCommand or InsertIntoHiveTable

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sharanya Santhanam + (santhanamsharanya@gmail.com) +
+
2023-05-24 19:37:02
 +
+

*Thread Reply:* It parses into InsertIntoHadoopFsRelationCommand. example +== Optimized Logical Plan == +InsertIntoHadoopFsRelationCommand <s3a://uchmsdev03/default/sharanyaOutputTable>, false, [id#89], Parquet, [serialization.format=1, mergeSchema=false, partitionOverwriteMode=dynamic], Append, CatalogTable( +Database: default +Table: sharanyaoutputtable +Owner: 2700940971 +Created Time: Thu Jun 09 11:13:35 PDT 2022 +Last Access: UNKNOWN +Created By: Spark 3.2.0 +Type: EXTERNAL +Provider: hive +Table Properties: [transient_lastDdlTime=1654798415] +Location: <s3a://uchmsdev03/default/sharanyaOutputTable> +Serde Library: org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe +InputFormat: org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat +OutputFormat: org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat +Storage Properties: [serialization.format=1] +Partition Provider: Catalog +Partition Columns: [`id`] +Schema: root + |-- displayName: string (nullable = true) + |-- serialnum: string (nullable = true) + |-- osversion: string (nullable = true) + |-- productfamily: string (nullable = true) + |-- productmodel: string (nullable = true) + |-- id: string (nullable = true) +), org.apache.spark.sql.execution.datasources.CatalogFileIndex@5fe23214, [displayName, serialnum, osversion, productfamily, productmodel, id] ++- Union false, false + :- Relation default.tablea[displayName#84,serialnum#85,osversion#86,productfamily#87,productmodel#88,id#89] parquet + +- Relation default.tableb[displayName#90,serialnum#91,osversion#92,productfamily#93,productmodel#94,id#95] parquet

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sharanya Santhanam + (santhanamsharanya@gmail.com) +
+
2023-05-24 19:39:54
 +
+

*Thread Reply:* using spark 3.2 & this is the query +spark.sql(s"INSERT INTO default.sharanyaOutput select ** from (SELECT ** from default.tableA union all " + + s"select ** from default.tableB)")

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Rakesh Jain + (rakeshj@us.ibm.com) +
+
2023-05-24 01:09:58
 +
+

Is there any example of how sourceCodeLocation / git info can be used from a spark job? What do we need to set to be able to see that as part of metadata?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-05-24 05:37:06
 +
+

*Thread Reply:* I think we can't really get it from Spark context, as Spark jobs are submitted in compiled, jar form, instead of plain text like for example Airflow dags.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Rakesh Jain + (rakeshj@us.ibm.com) +
+
2023-05-25 02:15:35
 +
+

*Thread Reply:* How about Jupyter Notebook based spark job?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-05-25 08:44:18
 +
+

*Thread Reply:* I don't think it changes much - but maybe @Paweł Leszczyński knows more

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-05-25 11:24:21
 +
+

@channel +Deprecation notice: support for Airflow 2.1 will end in about two weeks, when it will be removed from testing. The exact date will be announced as we get closer to it — this is just a heads up. After that date, use 2.1 at your own risk! (Note: the next release, 0.27.0, will still support 2.1.)

+ + + +
+ 👍 Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sheeri Cabral (Collibra) + (sheeri.cabral@collibra.com) +
+
2023-05-25 11:27:39
 +
+

For the OpenLineageSparkListener, is there a way to configure it to send packets locally, e.g. save to a file? (instead of pushing to a URL destination)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
alexandre bergere + (alexandre.pro.bergere@gmail.com) +
+
2023-05-25 12:00:04
 +
+

*Thread Reply:* We developed a FileTransport class in order to save locally in json file our metrics if you interested in

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sheeri Cabral (Collibra) + (sheeri.cabral@collibra.com) +
+
2023-05-25 12:00:37
 +
+

*Thread Reply:* Does it also save the openlineage information, e.g. inputs/outputs?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
alexandre bergere + (alexandre.pro.bergere@gmail.com) +
+
2023-05-25 12:02:07
 +
+

*Thread Reply:* yes it save all json information, inputs / ouputs included

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sheeri Cabral (Collibra) + (sheeri.cabral@collibra.com) +
+
2023-05-25 12:03:03
 +
+

*Thread Reply:* Yes! then I am very interested. Is there guidance on how to use the FileTransport class?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-05-25 13:06:22
 +
+

*Thread Reply:* @alexandre bergere it would be pretty useful contribution if you can submit it 🙂

+ + + +
+ 🙌 alexandre bergere +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
alexandre bergere + (alexandre.pro.bergere@gmail.com) +
+
2023-05-25 13:08:28
 +
+

*Thread Reply:* We are using it on a transformed OpenLineage library we developed ! I'm going to make a PR in order to share it with you :)

+ + + +
+ 👍 Julien Le Dem, Anirudh Shrinivason +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-05-25 13:56:48
 +
+

*Thread Reply:* would be great to have. I had it in mind to implement as an enabler for databricks integration tests. great to hear that!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
alexandre bergere + (alexandre.pro.bergere@gmail.com) +
+
2023-05-29 08:19:46
 +
+

*Thread Reply:* PR sent: https://github.com/OpenLineage/OpenLineage/pull/1891 🙂 +@Maciej Obuchowski could you tell me how to update the documentation once approved please?

+
+ + + + +
#1891 Adding a file transport
+
+ <p><strong>Problem</strong></p> + +<p>We were missing a File transport for OpenLineage as we have a lot of log ingestion software depending on object storage/ (often fuse-mounted in our runtime environments). It was also asked on the Projects's Slack, so here's our take</p> + +<p><strong>Solution</strong></p> + +<p>A FileTransport and its configuration classes were created. This transport supports append mode or write-new-file mode (especially useful on object storage not supporting append mode e.g: Databricks DBFS fuse)</p> + +<p>Code written with <a href="https://github.com/TomRibuot">@TomRibuot</a> (<a href="https://github.com/TomRibuot">https://github.com/TomRibuot</a>).</p> + +<p><strong>One-line summary:</strong></p> + +<p>Adding a file transport</p> + +<p><strong>Checklist</strong></p> + +<p>☐ You've <a href="https://github.com/OpenLineage/OpenLineage/blob/main/why-the-dco.md">signed-off</a> your work +☐ Your pull request title follows our <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CONTRIBUTING.md#creating-pull-requests">guidelines</a> +☐ Your changes are accompanied by tests (<em>if relevant</em>) +☐ Your change contains a <a href="https://kurtisnusbaum.medium.com/stacked-diffs-keeping-phabricator-diffs-small-d9964f4dcfa6">small diff</a> and is self-contained +☐ You've updated any relevant documentation (<em>if relevant</em>) +☐ Your comment includes a one-liner for the changelog about the specific purpose of the change (<em>if necessary</em>) +☐ You've versioned the core OpenLineage model or facets according to <a href="https://docs.snowplowanalytics.com/docs/pipeline-components-and-applications/iglu/common-architecture/schemaver">SchemaVer</a> (<em>if relevant</em>) +☐ You've added a <a href="https://github.com/OpenLineage/OpenLineage/tree/main/.github/header_templates.md">header</a> to source files (<em>if relevant</em>)</p> + +<hr /> + +<p>SPDX-License-Identifier: Apache-2.0 <br /> +Copyright 2018-2023 contributors to the OpenLineage project</p> +
+ +
+
Labels
+ client/python +
+ +
+
Comments
+ 1 +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-05-29 08:36:21
 +
+

*Thread Reply:* @alexandre bergere we have separate repo for website + docs: https://github.com/OpenLineage/docs

+
+ + + + +
OpenLineage/docs
+
+ Documentation for OpenLineage +
+ +
+
Website
+ <https://openlineage.io/docs> +
+ +
+
Stars
+ 5 +
+ + + + + + + + +
+ + + +
+ 🙏 alexandre bergere +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Bramha Aelem + (bramhaaelem@gmail.com) +
+
2023-05-25 16:40:26
 +
+

Hi Team- When we run databricks job, lot of dbfs path namespaces are getting created. Can someone please let us know how to overwrite the symlink namespaces and link with the spark app name or openlineage namespace marquez UI.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harshini Devathi + (harshini.devathi@tigeranalytics.com) +
+
2023-05-26 09:09:09
 +
+

Hello,

+ +

I am looking to connect the common data model in postgres marquez database and Azure Purview (which uses Apache Atlas API's) lineage endpoint. Does anyone have any how-to on this or can point me to some useful links for this?

+ +

Thanks in advance.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-05-26 13:08:56
 +
+

*Thread Reply:* I wonder if this blog post might help? https://openlineage.io/blog/openlineage-microsoft-purview

+
+
openlineage.io
+ + + +
Microsoft Purview Accelerates Lineage Extraction from Azure Databricks | OpenLineage Docs
+
+ A new collaboration between Microsoft and OpenLineage is making lineage extraction possible for Azure Databricks and Microsoft Purview users. +
+ + + + + + +
Original URL: https://openlineage.io/blog/openlineage-microsoft-purview
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-05-26 16:13:38
 +
+

*Thread Reply:* This might not fully match your use case, either, but might help: https://learn.microsoft.com/en-us/samples/microsoft/purview-adb-lineage-solution-accelerator/azure-databricks-to-purview-lineage-connector/

+
+
learn.microsoft.com
+ + + +
Azure Databricks to Purview Lineage Connector - Code Samples
+
+ This solution accelerator, together with the [OpenLineage](<http://openlineage.io>) project, provides a connector that will transfer lineage metadata fr... +
+ + + + + + + + +
Original URL: https://learn.microsoft.com/en-us/samples/microsoft/purview-adb-lineage-solution-accelerator/azure-databricks-to-purview-lineage-connector/
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harshini Devathi + (harshini.devathi@tigeranalytics.com) +
+
2023-06-01 23:23:49
 +
+

*Thread Reply:* Thanks @Michael Robinson

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Bernat Gabor + (gaborjbernat@gmail.com) +
+
2023-05-26 12:44:09
 +
+

Are there any constraints on facets? Such as is reasonable to expect that a single job will have a single parent? The schema hints to this by making the parent a single entry; but then one can send different parents for the START and COMPLETE event? 🤔

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-05-29 05:04:32
 +
+

*Thread Reply:* I think, for now such thing is not defined other than by implementation of consumers.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Bernat Gabor + (gaborjbernat@gmail.com) +
+
2023-05-30 10:32:09
 +
+

*Thread Reply:* Any reason for that?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-06-01 10:25:33
 +
+

*Thread Reply:* The idea is that for particular run, facets can be attached to any event type.

+ +

This has advantages, for example, job that modifies dataset that it's also reading from, can get particular version of dataset it's reading from and attach it on start; it would not work if you tried to do it on complete as the dataset would change by then.

+ +

Similarly, if the job is creating dataset, we could not get additional metadata on it, so we can attach those information only on complete.

+ +

There are also cases where we want facets to be cumulative. The reason for this are streaming jobs. For example, with Apache Flink, we could emit metadata on each checkpoint (or every N checkpoints) that contain metadata that show us how the job is progressing.

+ +

Generally consumers should be agnostic for that, but we don't want to overspecify what consumers should do - as people might want to use OL data in different ways, or even ignore some data we're sending.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Bernat Gabor + (gaborjbernat@gmail.com) +
+
2023-05-30 17:49:54
 +
+

Any reason why the lifecycle state change facet is not just on the output? But is also allowed on the inputs? 🤔 https://openlineage.io/docs/spec/facets/dataset-facets/lifecycle_state_change I can't see how would it be interpreted for an input 🤔

+
+
openlineage.io
+ + + +
Lifecycle State Change Facet | OpenLineage Docs
+
+ Example: +
+ + + + + + +
Original URL: https://openlineage.io/docs/spec/facets/dataset-facets/lifecycle_state_change
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-06-01 10:18:48
 +
+

*Thread Reply:* I think it should be output-only, yes.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-06-01 10:19:14
 +
+

*Thread Reply:* @Paweł Leszczyński what do you think?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-06-02 08:35:13
 +
+

*Thread Reply:* yes, should be output only I think

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Bernat Gabor + (gaborjbernat@gmail.com) +
+
2023-06-05 13:39:07
 +
+

*Thread Reply:* should we move it over then? 😄

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Bernat Gabor + (gaborjbernat@gmail.com) +
+
2023-06-05 13:39:31
 +
+

*Thread Reply:* under Output Dataset Facets that is

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-06-01 12:30:00
 +
+

@channel +The first issue of OpenLineage News is now available. To get it directly in your inbox when it’s published, become a subscriber.

+ + + +
+ 🚀 Willy Lulciuc, Jakub Dardziński, Maciej Obuchowski, Bernat Gabor, Harel Shein, Laurent Paris, Tamara Fingerlin, Perttu Salonen +
+ +
+ 🔥 Willy Lulciuc, Natalie Zeller, Ernie Ostic, Laurent Paris +
+ +
+ 💯 Willy Lulciuc +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-06-01 14:23:17
 +
+

*Thread Reply:* Correction: Julien and Willy’s talk at Data+AI Summit will take place on June 28

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-06-01 13:50:23
 +
+

Hello all, I’m opening a vote to release 0.27.0, featuring: +• Spark: fixed column lineage from databricks in the case of aggregate queries +• Python client: configurable job-name filtering +• Airflow: fixed urllib.parse.urlparse in case of [] values +Three +1s from committers will authorize an immediate release.

+ + + +
+ ➕ Jakub Dardziński, Maciej Obuchowski, Willy Lulciuc, Paweł Leszczyński +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-06-02 10:30:39
 +
+

*Thread Reply:* Thanks, all. The release is authorized and will be initiated on Monday in accordance with our policy here.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-06-02 13:13:18
 +
+

@channel +This month’s TSC meeting is next Thursday, June 8th, at 10:00 am PT. On the tentative agenda: announcements, meetup updates, recent releases, static lineage progress, and open discussion. More info and the meeting link can be found on the website. All are welcome! Also, feel free to reply or DM me with discussion topics, agenda items, etc.

+
+
openlineage.io
+ + + +
TSC Meetings | OpenLineage Docs
+
+ The OpenLineage Technical Steering Committee meets monthly, and is open to all. +
+ + + + + + +
Original URL: https://openlineage.io/meetings/
+ + + +
+ + + +
+ 🙌 Sheeri Cabral (Collibra), Maciej Obuchowski, Harel Shein, alexandre bergere, Paweł Leszczyński, Willy Lulciuc +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-06-05 12:34:29
 +
+

@channel +We released OpenLineage 0.27.1, including: +Additions: +• Python client: add emission filtering mechanism and exact, regex filters #1878 @mobuchowski +Fixes: +• Spark: fix column lineage for aggregate queries on databricks #1867 @pawel-big-lebowski +• Airflow: fix unquoted [ and ] in Snowflake URIs #1883 @JDarDagran +Plus a CI fix and a proposal. +For the details, see: +Release: https://github.com/OpenLineage/OpenLineage/releases/tag/0.27.1 +Changelog: https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md +Commit history: https://github.com/OpenLineage/OpenLineage/compare/0.26.0...0.27.1 +Maven: https://oss.sonatype.org/#nexus-search;quick~openlineage +PyPI: https://pypi.org/project/openlineage-python/

+ + + +
+ 🙌 Sheeri Cabral (Collibra) +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Bernat Gabor + (gaborjbernat@gmail.com) +
+
2023-06-05 13:01:06
 +
+

Looking for a reviewer under: https://github.com/OpenLineage/OpenLineage/pull/1892 🙂

+
+ + + + +
#1892 Use prettier to format the spec
+
+ <p>Signed-off-by: Bernát Gábor <a href="mailto:bgabor8@bloomberg.net">bgabor8@bloomberg.net</a></p> +
+ +
+
Labels
+ documentation, spec +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+ 🙌 Sheeri Cabral (Collibra), Paweł Leszczyński, Michael Robinson +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-06-05 15:47:08
 +
+

*Thread Reply:* @Bernat Gabor thanks for the PR!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-06-06 08:17:47
 +
+

Hey, I request release 0.27.2 to fix potential breaking change in Python client in 0.27.1: https://github.com/OpenLineage/OpenLineage/pull/1908

+ + + +
+ ➕ Jakub Dardziński, Paweł Leszczyński, Michael Robinson, Willy Lulciuc +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-06-06 10:58:23
 +
+

*Thread Reply:* Thanks @Maciej Obuchowski. The release is authorized and will be initiated as soon as possible.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-06-06 12:33:55
 +
+

@channel +We released OpenLineage 0.27.2, including: +Fixes: +• Python client: deprecate client.from_environment, do not skip loading config #1908 @Maciej Obuchowski
+For the details, see: +Release: https://github.com/OpenLineage/OpenLineage/releases/tag/0.27.2 +Changelog: https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md +Commit history: https://github.com/OpenLineage/OpenLineage/compare/0.27.1...0.27.2 +Maven: https://oss.sonatype.org/#nexus-search;quick~openlineage +PyPI: https://pypi.org/project/openlineage-python/

+ + + +
+ 👍 Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Bernat Gabor + (gaborjbernat@gmail.com) +
+
2023-06-06 14:22:18
 +
+

Found a major bug in the python client - https://github.com/OpenLineage/OpenLineage/pull/1917, if someone can review

+
+ + + + +
#1917 Add schemaURL to run event
+
+ <p>Resolves <a href="https://github.com/OpenLineage/OpenLineage/issues/1909">#1909</a></p> +
+ +
+
Labels
+ client/python, common +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Bernat Gabor + (gaborjbernat@gmail.com) +
+
2023-06-06 14:54:47
 +
+

And also https://github.com/OpenLineage/OpenLineage/pull/1913 🙂 that fixes the type information not being packaged

+
+ + + + +
#1913 Fix type checked marker and packaging
+
+ <p>Resolves <a href="https://github.com/OpenLineage/OpenLineage/issues/1911">#1911</a></p> +
+ +
+
Labels
+ integration/airflow, integration/great-expectations, client/python, common, integration/dagster, extractor +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-06-07 09:48:58
 +
+

@channel +This month’s TSC meeting is tomorrow, and all are welcome! https://openlineage.slack.com/archives/C01CK9T7HKR/p1685725998982879

+
+ + +
+ + + } + + Michael Robinson + (https://openlineage.slack.com/team/U02LXF3HUN7) +
+ + + +
+
+ <p>@channel +This month’s TSC meeting is next Thursday, June 8th, at 10:00 am PT. On the tentative agenda: announcements, meetup updates, recent releases, static lineage progress, and open discussion. More info and the meeting link can be found on the <a href="https://openlineage.io/meetings/">website</a>. All are welcome! Also, feel free to reply or DM me with discussion topics, agenda items, etc.</p> +
+ + + + + + +
Original URL: https://openlineage.slack.com/archives/C01CK9T7HKR/p1685725998982879
+ + +
+ + Slack Conversation +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Rachana Gandhi + (rachana.gandhi410@gmail.com) +
+
2023-06-08 11:11:31
 +
+

Hi team,

+ +

I wanted a lineage of my data for my tables and column level. +I am using jupyter notebook and spark code.

+ +

spark = (SparkSession.builder.master('local') + .appName('samplespark') + .config('spark.extraListeners', 'io.openlineage.spark.agent.OpenLineageSparkListener') + .config('spark.jars.packages', 'io.openlineage:openlineagespark:0.12.0') + .config('spark.openlineage.host', 'http://marquez-api:5000') + .config('spark.openlineage.namespace', 'spark_integration') + .getOrCreate())

+ +

I used this and then opened the localhost:3000 for marquez

+ +

I can see my job there but when i click on the job when its supposed to show lineage, its just an empty screen

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Lukenoff + (john@jlukenoff.com) +
+
2023-06-08 12:39:20
 +
+

*Thread Reply:* Do you get any output in your devtools? I just ran into this yesterday and it looks like it’s related to this issue: https://github.com/MarquezProject/marquez/issues/2410

+
+ + + + +
#2410 UI error: max call stack size exceeded
+
+ <p><a href="https://user-images.githubusercontent.com/1553185/217618061-2a4c3f55-2b9d-4c41-9799-dfd035da5477.png">Screen Shot 2023-02-03 at 4 16 40 PM (1)</a></p> +
+ +
+
Labels
+ bug +
+ +
+
Comments
+ 2 +
+ + + + + + + + + +
+ + <a href="https://github.com/MarquezProject/marquez">MarquezProject/marquez</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Lukenoff + (john@jlukenoff.com) +
+
2023-06-08 12:40:01
 +
+

*Thread Reply:* Seems like more of a Marquez client-side issue than something with OL

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Rachana Gandhi + (rachana.gandhi410@gmail.com) +
+
2023-06-08 12:43:02
 +
+

*Thread Reply:* ohh but if i try using the console output, it throws ClientProtocolError

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Lukenoff + (john@jlukenoff.com) +
+
2023-06-08 12:43:41
 +
+

*Thread Reply:* Sorry I mean in the dev console of your web browser

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Rachana Gandhi + (rachana.gandhi410@gmail.com) +
+
2023-06-08 12:44:43
 +
+

*Thread Reply:* this is the dev console in browser

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Lukenoff + (john@jlukenoff.com) +
+
2023-06-08 12:47:59
 +
+

*Thread Reply:* Seems like it’s coming from this line. Are there any job facets defined when you fetch from the API directly? That seems like kind of an old version of OL so maybe the schema is incompatible with the version Marquez is expecting

+
+ + + + +
<https://github.com/MarquezProject/marquez/blob/9fece97cf4481bdffd3b71c7e46de2dcac74265f/web/src/components/jobs/RunInfo.tsx | RunInfo.tsx>
+
+ <pre><code> {&lt;MqCode code={(jobFacets?.sql as SqlFacet)?.query} language={'sql'}/&gt;} +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/MarquezProject/marquez">MarquezProject/marquez</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Rachana Gandhi + (rachana.gandhi410@gmail.com) +
+
2023-06-08 12:51:21
 +
+

*Thread Reply:* from pyspark.sql import SparkSession

+ +

spark = (SparkSession.builder.master('local') + .appName('sample_spark') + .config('spark.extraListeners', 'io.openlineage.spark.agent.OpenLineageSparkListener') + .config('spark.jars.packages', 'io.openlineage:openlineage_spark:0.12.0') + .config('spark.openlineage.host', '<http://marquez-api:5000>') + .config('spark.openlineage.namespace', 'spark_integration')
+ .getOrCreate())

+ +

spark.sparkContext.setLogLevel("INFO")

+ +

spark.createDataFrame([ + {'a': 1, 'b': 2}, + {'a': 3, 'b': 4} +]).write.mode("overwrite").saveAsTable("temp_table8")

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Rachana Gandhi + (rachana.gandhi410@gmail.com) +
+
2023-06-08 12:51:49
 +
+

*Thread Reply:* This is my only code, I havent done anything apart from this

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Lukenoff + (john@jlukenoff.com) +
+
2023-06-08 12:52:30
 +
+

*Thread Reply:* I would try a more recent version of OL. Looks like you’re using 0.12.0 and I think the project is on 0.27.x currently

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Rachana Gandhi + (rachana.gandhi410@gmail.com) +
+
2023-06-08 12:55:07
 +
+

*Thread Reply:* so i should change io.openlineage:openlineage_spark:0.12.0 to io.openlineage:openlineage_spark:0.27.1?

+ + + +
+ 👍 John Lukenoff, Julien Le Dem +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Rachana Gandhi + (rachana.gandhi410@gmail.com) +
+
2023-06-08 13:10:03
 +
+

*Thread Reply:* it executed well, unable to see it in marquez

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Rachana Gandhi + (rachana.gandhi410@gmail.com) +
+
2023-06-08 13:18:16
 +
+

*Thread Reply:* marquez didnt get updated

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Rachana Gandhi + (rachana.gandhi410@gmail.com) +
+
2023-06-08 13:20:44
 +
+

*Thread Reply:* I am actually doing a POC on OpenLineage to find table and column level lineage for my team at Amazon. +If this goes through, the team could use openlineage to track data lineage on a larger scale..

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Lukenoff + (john@jlukenoff.com) +
+
2023-06-08 13:24:49
 +
+

*Thread Reply:* Maybe marquez is still pulling the data from the previous run using the old OL version. Do you still get the same error in the browser console? Do you get the same result if you rebuild and start with a clean marquez db?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Rachana Gandhi + (rachana.gandhi410@gmail.com) +
+
2023-06-08 13:25:10
 +
+

*Thread Reply:* yes i did that as well

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Rachana Gandhi + (rachana.gandhi410@gmail.com) +
+
2023-06-08 13:25:49
 +
+

*Thread Reply:* the error was present only once you clicked on any of the jobs in marquez, +since my job isnt showing up i cant check for the error itself

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Rachana Gandhi + (rachana.gandhi410@gmail.com) +
+
2023-06-08 13:26:29
 +
+

*Thread Reply:* docker run --network sparkdefault -p 3000:3000 -e MARQUEZHOST=marquez-api -e MARQUEZ_PORT=5000 --link marquez-api:marquez-api marquezproject/marquez-web:0.19.1

+ +

used this to rebuild marquez

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Lukenoff + (john@jlukenoff.com) +
+
2023-06-08 13:26:54
 +
+

*Thread Reply:* That’s odd, sorry, that’s probably the most I can help, I’m kinda new to OL/Marquez as well 😅

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Rachana Gandhi + (rachana.gandhi410@gmail.com) +
+
2023-06-08 13:27:41
 +
+

*Thread Reply:* no problem, can you refer me to someone who would know, so that i can ask them?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Lukenoff + (john@jlukenoff.com) +
+
2023-06-08 13:29:25
 +
+

*Thread Reply:* Actually looking at in now I think you’re using a slightly outdated version of marquez-web too. I would update that tag to at least 0.33.0. that’s what I’m using

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Lukenoff + (john@jlukenoff.com) +
+
2023-06-08 13:30:10
 +
+

*Thread Reply:* Other than that I would ask in the marquez slack channel or raise an issue in github on that project. Seems like more of an issue with Marquez since some at least some data is rendering in the UI initially

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Rachana Gandhi + (rachana.gandhi410@gmail.com) +
+
2023-06-08 13:32:58
 +
+

*Thread Reply:* nope that version also didnt help

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Rachana Gandhi + (rachana.gandhi410@gmail.com) +
+
2023-06-08 13:33:19
 +
+

*Thread Reply:* can you share their slack link?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Lukenoff + (john@jlukenoff.com) +
+
2023-06-08 13:34:52
 +
+

*Thread Reply:* http://bit.ly/MarquezSlack

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Rachana Gandhi + (rachana.gandhi410@gmail.com) +
+
2023-06-08 13:35:08
 +
+

*Thread Reply:* that link is no longer active

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2023-06-09 18:44:25
 +
+

*Thread Reply:* Hello @Rachana Gandhi could you point to the doc where you found the example .config(‘spark.jars.packages’, ‘io.openlineage:openlineage_spark:0.12.0’) ? We should update it to have the latest version instead.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Rachana Gandhi + (rachana.gandhi410@gmail.com) +
+
2023-06-09 18:54:49
 +
+

*Thread Reply:* https://openlineage.io/docs/integrations/spark/quickstart_local/

+
+
openlineage.io
+ + + +
Quickstart with Jupyter | OpenLineage Docs
+
+ Trying out the Spark integration is super easy if you already have Docker Desktop and git installed. +
+ + + + + + +
Original URL: https://openlineage.io/docs/integrations/spark/quickstart_local/
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Rachana Gandhi + (rachana.gandhi410@gmail.com) +
+
2023-06-09 18:59:17
 +
+

*Thread Reply:* https://openlineage.io/docs/guides/spark

+ +

also the docker compose here has an earlier version of marquez

+
+
openlineage.io
+ + + +
Using OpenLineage with Spark | OpenLineage Docs
+
+ Adapted from a blog post by Michael Collado +
+ + + + + + +
Original URL: https://openlineage.io/docs/guides/spark
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harshit Soni + (harshit.soni@angelbroking.com) +
+
2023-07-13 17:00:54
 +
+

*Thread Reply:* Facing same issue with my initial POC. Did we get any solution for this?

+ + + +
+
+
+
+ + + + + + + +
+
+ + + + +
+ +
Bernat Gabor + (gaborjbernat@gmail.com) +
+
2023-06-08 14:36:38
 +
+

Approve a new release 🙂

+ + + +
+ ➕ Michael Robinson, Willy Lulciuc, Maciej Obuchowski, Jakub Dardziński +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-06-08 14:43:55
 +
+

*Thread Reply:* Requesting a release? 3 +1s from committers will authorize. More info here: https://github.com/OpenLineage/OpenLineage/blob/main/GOVERNANCE.md

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Bernat Gabor + (gaborjbernat@gmail.com) +
+
2023-06-08 14:44:14
 +
+

*Thread Reply:* Yeah, that one 😊

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Bernat Gabor + (gaborjbernat@gmail.com) +
+
2023-06-08 14:44:44
 +
+

*Thread Reply:* Because the python client is broken as is today without a new release

+ + + +
+ 👍 Michael Robinson +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-06-08 18:45:04
 +
+

*Thread Reply:* Thanks, all. The release is authorized and will be initiated by EOB next Tuesday, but in all likelihood well before then.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Bernat Gabor + (gaborjbernat@gmail.com) +
+
2023-06-08 19:06:34
 +
+

*Thread Reply:* cool

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-06-12 13:15:26
 +
+

@channel +We released OpenLineage 0.28.0, including: +Added +• dbt: add Databricks compatibility #1829 @Ines70 +Fixed +• Fix type-checked marker and packaging #1913 @gaborbernat +• Python client: add schemaURL to run event #1917 @gaborbernat +For the details, see: +Release: https://github.com/OpenLineage/OpenLineage/releases/tag/0.28.0 +Changelog: https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md +Commit history: https://github.com/OpenLineage/OpenLineage/compare/0.27.2...0.28.0 +Maven: https://oss.sonatype.org/#nexus-search;quick~openlineage +PyPI: https://pypi.org/project/openlineage-python/

+ + + +
+ 🚀 Maciej Obuchowski, Willy Lulciuc, Francis McGregor-Macdonald +
+ +
+ 👍 Ines DAHOUMANE -COWORKING PARIS- +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-06-12 14:35:56
 +
+

@channel +Meetup announcement: there’s another meetup happening soon! This one will be an evening event on 6/22 in New York at Collibra’s HQ. For details and to sign up, please join the meetup group: https://www.meetup.com/data-lineage-meetup/events/294065396/. Thanks to @Sheeri Cabral (Collibra) for cohosting and providing a space.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-06-12 23:27:16
 +
+

Hi, just curious, does openlineage have a log4j integration?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-06-13 04:44:28
 +
+

*Thread Reply:* Do you mean to just log events to logging backend?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-06-13 04:54:30
 +
+

*Thread Reply:* Hmm more like have a separate logging config for sending all the logs to a backend

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-06-13 04:54:38
 +
+

*Thread Reply:* Not the events itself

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-06-13 05:01:10
 +
+

*Thread Reply:* @Anirudh Shrinivason with Spark integration?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-06-13 05:01:59
 +
+

*Thread Reply:* It uses slf4j so you should be able to set up your log4j logger

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-06-13 05:10:55
 +
+

*Thread Reply:* Yeah with the spark integration. Ahh I see. Okay sure thanks!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-06-21 23:21:14
 +
+

*Thread Reply:* ~Hi @Maciej Obuchowski May I know what the class path I should be using for setting up the log4j if I want to set it up for OL related logs? Is there some guide or runbook to setting up the log4j with OL? Thanks!~ +Nvm lol found it! 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Vamshi krishna + (vnallamothu@cardinalcommerce.com) +
+
2023-06-13 12:19:01
 +
+

Hello all, we are just starting to use Marquez as part of our POC. We are following the getting started guide at https://openlineage.io/getting-started/ to set the environment on an AWS Ec2 instance. When we are running ./docker/up.sh, it is not bringing up marquez-web container. Also, we are not able to access Admin UI at 5000 and 5001 ports.

+ +

Docker version: 24.0.2 +Docker compose version: 2.18.1 +OS: Ubuntu_20.04

+ +

Can someone please let me know what I am missing? +Note: I had to modify docker-compose command in up.sh as per docker compose V2.

+ +

Also, we are seeing following log when our loadbalancer is checking for health:

+ +

WARN [2023-06-13 15:35:31,040] marquez.logging.LoggingMdcFilter: status: 404 +172.30.1.206 - - [13/Jun/2023:15:35:42 +0000] "GET / HTTP/1.1" 200 535 "-" "ELB-HealthChecker/2.0" 1 +172.30.1.206 - - [13/Jun/2023:15:35:42 +0000] "GET / HTTP/1.1" 404 43 "-" "ELB-HealthChecker/2.0" 2 +WARN [2023-06-13 15:35:42,866] marquez.logging.LoggingMdcFilter: status: 404

+
+
openlineage.io
+ + + +
Getting Started | OpenLineage Docs
+
+ Data lineage is the foundation for a new generation of powerful, context-aware data tools and best practices. OpenLineage enables consistent collection of lineage metadata, creating a deeper understanding of how data is produced and used. +
+ + + + + + +
Original URL: https://openlineage.io/getting-started/
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kavitha + (kkandaswamy@cardinalcommerce.com) +
+
2023-06-14 10:42:41
 +
+

*Thread Reply:* Hello, is anyone eho has recently installed latest version of marquez/open-lineage-spark using docker image available to help Vamshi and I or provide any pointers? Thank you

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-06-15 03:38:38
 +
+

*Thread Reply:* if you're working on mac, you can have an issue related to port 5000. Instructions here https://github.com/MarquezProject/marquez#quickstart provides a workaround for that ./docker/up.sh --api-port 9000

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kavitha + (kkandaswamy@cardinalcommerce.com) +
+
2023-06-15 08:43:33
 +
+

*Thread Reply:* @Paweł Leszczyński, thank you, we were using ubuntu on an EC2 instance and each time we are running into different errors and are never able to access the application page, web server, the admin interface, we have run out of ideas of what else to try differently to get this setup up and running

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Vamshi krishna + (vnallamothu@cardinalcommerce.com) +
+
2023-06-22 14:47:00
 +
+

*Thread Reply:* @Michael Robinson Can you please help us here?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-06-22 14:58:57
 +
+

*Thread Reply:* @Vamshi krishna I’m sorry you’re still blocked. Thanks for the information about your system. Would you please share some of the errors you are getting? More details would help us reproduce and diagnose.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kavitha + (kkandaswamy@cardinalcommerce.com) +
+
2023-06-22 16:35:00
 +
+

*Thread Reply:* @Michael Robinson, thank you, vamshi and i will share the errors that we are running into shortly

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Vamshi krishna + (vnallamothu@cardinalcommerce.com) +
+
2023-06-23 09:48:16
 +
+

*Thread Reply:* We are following https://openlineage.io/getting-started/ guide and trying to set up Marquez on a ubuntu ec2 instance. Following are versions of docker, docker compose and ubuntu

+
+
openlineage.io
+ + + +
Getting Started | OpenLineage Docs
+
+ Data lineage is the foundation for a new generation of powerful, context-aware data tools and best practices. OpenLineage enables consistent collection of lineage metadata, creating a deeper understanding of how data is produced and used. +
+ + + + + + +
Original URL: https://openlineage.io/getting-started/
+ + + +
+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Vamshi krishna + (vnallamothu@cardinalcommerce.com) +
+
2023-06-23 09:49:51
 +
+

*Thread Reply:* @Michael Robinson When we follow the documentation without changing anything and run sudo ./docker/up.sh we are seeing following errors:

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Vamshi krishna + (vnallamothu@cardinalcommerce.com) +
+
2023-06-23 10:00:38
 +
+

*Thread Reply:* So, I edited up.sh file and modified docker compose command by removing --log-level flag and ran sudo ./docker/up.sh and found following errors:

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Vamshi krishna + (vnallamothu@cardinalcommerce.com) +
+
2023-06-23 10:02:29
 +
+

*Thread Reply:* Then I copied .env.example to .env since compose needs .env file

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Vamshi krishna + (vnallamothu@cardinalcommerce.com) +
+
2023-06-23 10:05:04
 +
+

*Thread Reply:* I got this error:

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Vamshi krishna + (vnallamothu@cardinalcommerce.com) +
+
2023-06-23 10:09:24
 +
+

*Thread Reply:* since I am getting timeouts, I thought it might be an issue with proxy. So, I followed this doc: https://stackoverflow.com/questions/58841014/set-proxy-on-docker and added my outbound proxy and tried

+
+
Stack Overflow
+ + + +
Set proxy on docker
+
+ Im using centos 7.7 and docker 19.03. + +I cannot pull images and getting : + +Error response from daemon: Get <https://registry-1.docker.io.v2>./: dial tcp lookup : server misbehaving +I did what all gu... +
+ + + + + + + + +
Original URL: https://stackoverflow.com/questions/58841014/set-proxy-on-docker
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Vamshi krishna + (vnallamothu@cardinalcommerce.com) +
+
2023-06-23 10:23:46
 +
+

*Thread Reply:* @Michael Robinson Then it kind of worked but seeing following errors:

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Vamshi krishna + (vnallamothu@cardinalcommerce.com) +
+
2023-06-23 10:24:31
 +
+

*Thread Reply:*

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Vamshi krishna + (vnallamothu@cardinalcommerce.com) +
+
2023-06-23 10:25:29
 +
+

*Thread Reply:* @Michael Robinson @Paweł Leszczyński Can you please see above steps and let us know what are we missing/doing wrong? I appreciate your help and time.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-06-23 10:45:39
 +
+

*Thread Reply:* The latest errors look to me like they’re being caused by postgres and might reflect a port conflict. Are you using the default port for the API (5000)? You might try using a different port. More info about this in the Marquez readme: https://github.com/MarquezProject/marquez/blob/0.35.0/README.md.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Vamshi krishna + (vnallamothu@cardinalcommerce.com) +
+
2023-06-23 10:46:55
 +
+

*Thread Reply:* Yes we are using default ports: +APIPORT=5000 +APIADMINPORT=5001 +WEBPORT=3000 +TAG=0.35.0

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Vamshi krishna + (vnallamothu@cardinalcommerce.com) +
+
2023-06-23 10:47:40
 +
+

*Thread Reply:* We see these postgres permission issues only occasionally. Other times we only see db and api containers up but not the web

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-06-23 10:52:38
 +
+

*Thread Reply:* I would try running ./docker/up.sh --api-port 9000 (see Pawel’s message above for more context.)

+ + + +
+ 👍 Vamshi krishna +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Vamshi krishna + (vnallamothu@cardinalcommerce.com) +
+
2023-06-23 10:54:18
 +
+

*Thread Reply:* Still no luck. Seeing same errors.

+ +

2023-06-23 14:53:23.971 GMT [1] LOG: could not open configuration file "/etc/postgresql/postgresql.conf": Permission denied +marquez-db | 2023-06-23 14:53:23.971 GMT [1] FATAL: configuration file "/etc/postgresql/postgresql.conf" contains errors

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Vamshi krishna + (vnallamothu@cardinalcommerce.com) +
+
2023-06-23 10:54:43
 +
+

*Thread Reply:* ERROR [2023-06-23 14:53:42,269] org.apache.tomcat.jdbc.pool.ConnectionPool: Unable to create initial connections of pool. +marquez-api | ! java.net.UnknownHostException: postgres +marquez-api | ! at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:567) +marquez-api | ! at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327) +marquez-api | ! at java.base/java.net.Socket.connect(Socket.java:633) +marquez-api | ! at org.postgresql.core.PGStream.createSocket(PGStream.java:243) +marquez-api | ! at org.postgresql.core.PGStream.&lt;init&gt;(PGStream.java:98) +marquez-api | ! at org.postgresql.core.v3.ConnectionFactoryImpl.tryConnect(ConnectionFactoryImpl.java:132) +marquez-api | ! at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:258) +marquez-api | ! ... 26 common frames omitted +marquez-api | ! Causing: org.postgresql.util.PSQLException: The connection attempt failed. +marquez-api | ! at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:354) +marquez-api | ! at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:54) +marquez-api | ! at org.postgresql.jdbc.PgConnection.&lt;init&gt;(PgConnection.java:253) +marquez-api | ! at org.postgresql.Driver.makeConnection(Driver.java:434) +marquez-api | ! at org.postgresql.Driver.connect(Driver.java:291) +marquez-api | ! at org.apache.tomcat.jdbc.pool.PooledConnection.connectUsingDriver(PooledConnection.java:346) +marquez-api | ! at org.apache.tomcat.jdbc.pool.PooledConnection.connect(PooledConnection.java:227) +marquez-api | ! at org.apache.tomcat.jdbc.pool.ConnectionPool.createConnection(ConnectionPool.java:768) +marquez-api | ! at org.apache.tomcat.jdbc.pool.ConnectionPool.borrowConnection(ConnectionPool.java:696) +marquez-api | ! at org.apache.tomcat.jdbc.pool.ConnectionPool.init(ConnectionPool.java:495) +marquez-api | ! at org.apache.tomcat.jdbc.pool.ConnectionPool.&lt;init&gt;(ConnectionPool.java:153) +marquez-api | ! at org.apache.tomcat.jdbc.pool.DataSourceProxy.pCreatePool(DataSourceProxy.java:118) +marquez-api | ! at org.apache.tomcat.jdbc.pool.DataSourceProxy.createPool(DataSourceProxy.java:107) +marquez-api | ! at org.apache.tomcat.jdbc.pool.DataSourceProxy.getConnection(DataSourceProxy.java:131) +marquez-api | ! at org.flywaydb.core.internal.jdbc.JdbcUtils.openConnection(JdbcUtils.java:48) +marquez-api | ! at org.flywaydb.core.internal.jdbc.JdbcConnectionFactory.&lt;init&gt;(JdbcConnectionFactory.java:75) +marquez-api | ! at org.flywaydb.core.FlywayExecutor.execute(FlywayExecutor.java:147) +marquez-api | ! at <a href="http://org.flywaydb.core.Flyway.info">org.flywaydb.core.Flyway.info</a>(Flyway.java:190) +marquez-api | ! at marquez.db.DbMigration.hasPendingDbMigrations(DbMigration.java:73) +marquez-api | ! at marquez.db.DbMigration.migrateDbOrError(DbMigration.java:27) +marquez-api | ! at marquez.MarquezApp.run(MarquezApp.java:105) +marquez-api | ! at marquez.MarquezApp.run(MarquezApp.java:48) +marquez-api | ! at io.dropwizard.cli.EnvironmentCommand.run(EnvironmentCommand.java:67) +marquez-api | ! at io.dropwizard.cli.ConfiguredCommand.run(ConfiguredCommand.java:98) +marquez-api | ! at io.dropwizard.cli.Cli.run(Cli.java:78) +marquez-api | ! at io.dropwizard.Application.run(Application.java:94) +marquez-api | ! at marquez.MarquezApp.main(MarquezApp.java:60) +marquez-api | INFO [2023-06-23 14:53:42,274] marquez.MarquezApp: Stopping app...

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-06-23 11:06:32
 +
+

*Thread Reply:* Why do you run docker up with sudo? some of your screenshots suggest docker is not able to access docker registry. The last error java.net.UnknownHostException: postgres may be just a result of container being down. Could you verify if all the containers are up and running and if not what's the error? Are you able to test this docker.up in your laptop or other environment?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Vamshi krishna + (vnallamothu@cardinalcommerce.com) +
+
2023-06-23 11:08:34
 +
+

*Thread Reply:* Docker commands require sudo and cannot run with other user. +Postgres container is not coming up. It is failing with following errors:

+ +

2023-06-23 14:53:23.971 GMT [1] LOG: could not open configuration file "/etc/postgresql/postgresql.conf": Permission denied +marquez-db | 2023-06-23 14:53:23.971 GMT [1] FATAL: configuration file "/etc/postgresql/postgresql.conf" contains errors

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-06-23 11:10:19
 +
+

*Thread Reply:* and what does docker ps -a say about postgres container? why did it fail?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Vamshi krishna + (vnallamothu@cardinalcommerce.com) +
+
2023-06-23 11:11:36
 +
+

*Thread Reply:*

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-06-23 11:25:17
 +
+

*Thread Reply:* hmyy, no changes on our side have been done in postgresql.conf since August 2022. Did you apply any changes or have a clean clone of a repo?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Vamshi krishna + (vnallamothu@cardinalcommerce.com) +
+
2023-06-23 11:29:46
 +
+

*Thread Reply:* No we didn't make any changes

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-06-23 11:32:21
 +
+

*Thread Reply:* you did write earlier Note: I had to modify docker-compose command in up.sh as per docker compose V2.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Vamshi krishna + (vnallamothu@cardinalcommerce.com) +
+
2023-06-23 11:34:54
 +
+

*Thread Reply:* Yes all I did was modified this line: docker-compose --log-level ERROR $compose_files up $ARGS to +docker compose $compose_files up $ARGS since docker compose v2 doesn't support --log-level flag

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Vamshi krishna + (vnallamothu@cardinalcommerce.com) +
+
2023-06-23 11:37:03
 +
+

*Thread Reply:* Let me pull an older version and try

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Vamshi krishna + (vnallamothu@cardinalcommerce.com) +
+
2023-06-23 12:09:43
 +
+

*Thread Reply:* Still no luck same exact errors. Tried on a different ubuntu instance. Still seeing same errors with postgres

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Vamshi krishna + (vnallamothu@cardinalcommerce.com) +
+
2023-06-23 15:06:32
 +
+

*Thread Reply:* @Jeremy W

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-06-15 10:40:47
 +
+

Hi all, a general doubt. Would the column lineage associated with a job be present in both the start events and the complete events? Or could there be cases where the column lineage, and any output information is only present in one of the events, but not the other?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-06-15 10:49:42
 +
+

*Thread Reply:* > Or could there be cases where the column lineage, and any output information is only present in one of the events, but not the other? +Yes. Generally events regarding single run are cumulative

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-06-15 11:07:03
 +
+

*Thread Reply:* Ahh I see... Is it fair to assume that if I see column lineage in a start event, it's the full column lineage? Or could it be possible that half the lineage is in the start event, and half the lineage is in the complete event?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-06-15 22:50:51
 +
+

*Thread Reply:* Hi @Maciej Obuchowski just pinging in case you'd missed the above message. 🙇

+ + + +
+ 👀 Paweł Leszczyński +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-06-16 04:48:57
 +
+

*Thread Reply:* Actually, in this case this definitely should not happen. @Paweł Leszczyński am I right?

+ + + +
+ :gratitude_thank_you: Anirudh Shrinivason +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-06-16 04:50:16
 +
+

*Thread Reply:* @Maciej Obuchowski yes, you're

+ + + +
+ :gratitude_thank_you: Anirudh Shrinivason +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
nivethika R + (nivethikar8@gmail.com) +
+
2023-06-15 11:14:33
 +
+

Hi All.. Is JDBC supported for openLineage and marquez for columnlineage? I did some POC using tables in postgresdb and I am able to see all events but for columnLineage Iam getting it as NULL. Not sure where I am missing.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-06-16 02:14:19
 +
+

*Thread Reply:* ~No, we do have an open issue for that: https://github.com/OpenLineage/OpenLineage/issues/1758~

+
+ + + + +
#1758 Spark JDBC column lineage support
+
+ <p>Please add column lineage support also for Spark JDBC sources as discussed also in <a href="https://github.com/OpenLineage/OpenLineage/issues/1584">#1584</a></p> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-06-16 05:02:26
 +
+

*Thread Reply:* @nivethika R, I am sorry for misleading response, we've merged PR for that https://github.com/OpenLineage/OpenLineage/pull/1636. It does not support select ** but besides that, it should be operational.

+ +

Could you please try a query from our integration tests to verify if this is working for you or not: https://github.com/OpenLineage/OpenLineage/pull/1636/files#diff-137aa17091138b69681510e13e3b7d66aa9c9c7c81fe8fe13f09f0de76448dd5R46 ?

+
+ + + + +
#1636 Spark/jdbc relations
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Nagendra Kolisetty + (nkolisetty@geico.com) +
+
2023-06-16 12:12:00
 +
+

Hi There,

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Nagendra Kolisetty + (nkolisetty@geico.com) +
+
2023-06-16 12:12:43
 +
+

We are trying to install the image on the private AKS cluster and we ended up in below error

+ +

kubectl : pod marquez/pgsql-postgresql-client terminated (StartError) +At line:1 char:1

  • kubectl run pgsql-postgresql-client --rm --tty -i --restart='Never' `
  • ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +
    • CategoryInfo : NotSpecified: (pod marquez/pgs...ed (StartError):String) [], RemoteException
    • FullyQualifiedErrorId : NativeCommandError
      • +
    • +
+ +

failed to create containerd task: failed to create shim task: OCI runtime create failed: container_linux.go:380: starting container process caused: exec: "PGPASSWORD=macondo": executable file not found in $PATH: +unknown

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Nagendra Kolisetty + (nkolisetty@geico.com) +
+
2023-06-16 12:13:13
 +
+

We followed the below article to install Marquez in AKS (Azure). +By the way, we pulled the images from docker pushed it to our acr. +tried installing the postgresql via ACR and it failed with the error

+ +

https://github.com/MarquezProject/marquez/blob/main/docs/running-on-aws.md

+
+ + + + +
<https://github.com/MarquezProject/marquez/blob/main/docs/running-on-aws.md | running-on-aws.md>
+
+ <hr /> + +<p><strong>layout: running-on-aws</strong> +<strong>Running Marquez on AWS</strong></p> + +<p>This guide helps you deploy and manage Marquez on AWS <a href="https://aws.amazon.com/eks">EKS</a>.</p> + +<p><strong>PREREQUISITES</strong></p> + +<p>• <a href="https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html">AWS CLI</a> +• <a href="https://helm.sh/docs/helm/helm_install/">helm</a> +• <a href="https://kubernetes.io/docs/tasks/tools/">kubectl</a></p> + +<p><strong>AWS EKS Cluster</strong></p> + +<p>To create an AWS EKS cluster, please follow the steps outlined in the AWS EKS <a href="https://docs.aws.amazon.com/eks/latest/userguide/create-cluster.html">documentation</a>.</p> + +<p><strong>CONNECT TO AWS EKS CLUSTER</strong></p> + +<ol><li>Make sure you have configured your <a href="https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html">AWS CLI</a>, then create or update the <strong>kubeconfig</strong> file for your cluster: +<pre><code>$ aws eks --region &lt;AWS-REGION&gt; update-kubeconfig --name &lt;AWS-EKS-CLUSTER&gt; +</code></pre></li><li>Verify that the context has been switched: +<pre><code>$ kubectl config current-context +arn:aws:eks:&lt;AWS-REGION&gt;:&lt;AWS-ACCOUNT-ID&gt;:cluster/&lt;AWS-EKS-CLUSTER&gt; +</code></pre></li><li><p>Using <code>kubectl</code>, verify that you can connect to your cluster:</p> + +<pre><code>$ kubectl get svc +NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE +svc/kubernetes ClusterIP 10.100.0.1 &lt;none&gt; 443/TCP 1m +</code></pre> + +<blockquote> + <p>Note: If you're having issues connecting to your cluster, please see <a href="https://aws.amazon.com/premiumsupport/knowledge-center/eks-cluster-connection">Why can't I connect to my AWS EKS cluster?</a></p> +</blockquote></li> +</ol> + +<p><strong>AWS RDS</strong></p> + +<p>Next, create an AWS RDS instance as outlined in the AWS RDS <a href="https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_CreateDBInstance.html">documentation</a>. This database will be used to store dataset, job, and run metadata collected as <a href="https://openlineage.io">OpenLineage</a> events via the Marquez <a href="https://marquezproject.github.io/marquez/openapi.html">HTTP API</a>.</p> + +<p><strong>CREATE AWS RDS DATABASE</strong></p> + +<ol><li>Navigate to the AWS <a href="https://console.aws.amazon.com/rds/home">RDS</a> page and create a PostgreSQL database, leaving the database template as <strong>Production</strong>.</li><li>Use <code>marquez</code> as the database identifier and set the master username to <code>marquez</code>.</li><li>Choose a master password to use later in your Helm deployment (see <a href="https://github.com/MarquezProject/marquez/blob/main/chart/values.yaml#L32">password</a> in <code>values.yaml</code>).</li><li>Leave public access to the database <strong>off</strong>.</li><li>Choose the same VPC where your AWS EKS cluster resides.</li><li>In a separate tab, navigate to the AWS EKS cluster page and make note of the security group attached to your cluster.</li><li>Navigate back to the AWS RDS page and, in the security group section, add the AWS EKS cluster’s security group from <strong>step 6</strong>.</li><li>Next, under the <strong>Additional Configuration</strong> tab, enter <code>marquez</code> as the initial database name.</li><li>Finally, select <strong>Create Database</strong>.</li> +</ol> + +<p><strong>CONNECT TO AWS RDS DATABASE</strong></p> + +<ol><li>Create a <code>marquez</code> namespace: +<pre><code>$ kubectl create namespace marquez +</code></pre></li><li>Next, run the following command with your AWS RDS <code>host</code>, <code>username</code>, and <code>password</code>: +<pre><code>kubectl run pgsql-postgresql-client --rm --tty -i --restart='Never' \ + --namespace marquez \ + --image &lt;a href="http://docker.io/bitnami/postgresql:12-debian-10"&gt;docker.io/bitnami/postgresql:12-debian-10&lt;/a&gt; \ + --env="PGPASSWORD=&lt;AWS-RDS-PASSWORD&gt;" \ + --command -- psql marquez --host &lt;AWS-RDS-HOST&gt; -U &lt;AWS-RDS-USERNAME&gt; -d marquez -p 5432 +</code></pre></li> +</ol> + +<p><strong>Deploy Marquez on AWS EKS</strong> +<strong>INSTALLING MARQUEZ</strong></p> + +<ol><li>Get Marquez: +<pre><code>$ git clone git@github.com:MarquezProject/marquez.git &amp;&amp; cd chart +</code></pre></li><li><p>Install Marquez:</p> + +<pre><code>helm upgrade --install marquez . + --set marquez.db.host=&lt;AWS-RDS-HOST&gt; + --set marquez.db.user=&lt;AWS-RDS-USERNAME&gt; + --set marquez.db.password=&lt;AWS-RDS-PASSWORD&gt; + --namespace marquez + --atomic + --wait +</code></pre> + +<blockquote> + <p><strong>Note:</strong> To avoid overriding deployment settings via the command line, update the <a href="https://github.com/MarquezProject/marquez/blob/main/chart/values.yaml#L27">marquez.db</a> section of the Marquez Helm chart's <code>values.yaml</code> to include the AWS RDS <code>host</code>, <code>username</code>, and <code>password</code> in your deployment.</p> +</blockquote></li><li>Verify all the pods have come up correctly: +<pre><code>$ kubectl get pods --namespace marquez +</code></pre></li> +</ol> + +<p><strong>UNINSTALLING MARQUEZ</strong></p> + +<pre><code>helm uninstall marquez --namespace marquez +</code></pre> + +<hr /> + +<p>SPDX-License-Identifier: Apache-2.0 <br /> +Copyright 2018-2023 contributors to the Marquez project.</p> +
+ + + + + + + +
+ + <a href="https://github.com/MarquezProject/marquez">MarquezProject/marquez</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-06-21 11:07:04
 +
+

*Thread Reply:* Hi Nagendra, sorry you’re running into this error. We’re looking into it!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-06-18 09:53:19
 +
+

Hi, found this error in couple of the spark jobs: https://github.com/OpenLineage/OpenLineage/issues/1930 +Would request your help to kindly help patch thanks!

+
+ + + + +
#1930 Error in ColumnLevelLineageUtils
+
+ <p>`ERROR ColumnLevelLineageUtils: Error when invoking static method 'buildColumnLineageDatasetFacet' for Spark3 java.lang.reflect.InvocationTargetException at jdk.internal.reflect.GeneratedMethodAccessor476.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at io.openlineage.spark.agent.lifecycle.plan.column.ColumnLevelLineageUtils.buildColumnLineageDatasetFacet(ColumnLevelLineageUtils.java:35) at io.openlineage.spark.agent.lifecycle.OpenLineageRunEventBuilder.lambda$buildOutputDatasets$21(OpenLineageRunEventBuilder.java:431) at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195) at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1655) at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913) at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578) at io.openlineage.spark.agent.lifecycle.OpenLineageRunEventBuilder.buildOutputDatasets(OpenLineageRunEventBuilder.java:444) at io.openlineage.spark.agent.lifecycle.OpenLineageRunEventBuilder.populateRun(OpenLineageRunEventBuilder.java:303) at io.openlineage.spark.agent.lifecycle.OpenLineageRunEventBuilder.buildRun(OpenLineageRunEventBuilder.java:286) at io.openlineage.spark.agent.lifecycle.OpenLineageRunEventBuilder.buildRun(OpenLineageRunEventBuilder.java:238) at io.openlineage.spark.agent.lifecycle.SparkSQLExecutionContext.end(SparkSQLExecutionContext.java:95) at io.openlineage.spark.agent.OpenLineageSparkListener.sparkSQLExecEnd(OpenLineageSparkListener.java:98) at io.openlineage.spark.agent.OpenLineageSparkListener.onOtherEvent(OpenLineageSparkListener.java:84) at org.apache.spark.scheduler.SparkListenerBus.doPostEvent(SparkListenerBus.scala:100) at org.apache.spark.scheduler.SparkListenerBus.doPostEvent$(SparkListenerBus.scala:28) at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37) at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37) at org.apache.spark.util.ListenerBus.postToAll(ListenerBus.scala:117) at org.apache.spark.util.ListenerBus.postToAll$(ListenerBus.scala:101) at org.apache.spark.scheduler.AsyncEventQueue.super$postToAll(AsyncEventQueue.scala:105) at org.apache.spark.scheduler.AsyncEventQueue.$anonfun$dispatch$1(AsyncEventQueue.scala:105) at scala.runtime.java8.JFunction0$mcJ$sp.apply(JFunction0$mcJ$sp.java:23) at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62) at <a href="http://org.apache.spark.scheduler.AsyncEventQueue.org">org.apache.spark.scheduler.AsyncEventQueue.org</a>$apache$spark$scheduler$AsyncEventQueue$$dispatch(AsyncEventQueue.scala:100) at org.apache.spark.scheduler.AsyncEventQueue$$anon$2.$anonfun$run$1(AsyncEventQueue.scala:96) at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1381) at org.apache.spark.scheduler.AsyncEventQueue$$anon$2.run(AsyncEventQueue.scala:96) Caused by: java.lang.NullPointerException at io.openlineage.spark.agent.util.ScalaConversionUtils.fromSeq(ScalaConversionUtils.java:61) at io.openlineage.spark.agent.lifecycle.Rdds.findFileLikeRdds(Rdds.java:69) at io.openlineage.spark3.agent.lifecycle.plan.column.InputFieldsCollector.extractDatasetIdentifier(InputFieldsCollector.java:149) at io.openlineage.spark3.agent.lifecycle.plan.column.InputFieldsCollector.extractDatasetIdentifier(InputFieldsCollector.java:124) at io.openlineage.spark3.agent.lifecycle.plan.column.InputFieldsCollector.discoverInputsFromNode(InputFieldsCollector.java:70) at io.openlineage.spark3.agent.lifecycle.plan.column.InputFieldsCollector.collect(InputFieldsCollector.java:54) at io.openlineage.spark3.agent.lifecycle.plan.column.InputFieldsCollector.lambda$collect$0(InputFieldsCollector.java:64) at java.base/java.util.Iterator.forEachRemaining(Iterator.java:133) at scala.collection.convert.Wrappers$IteratorWrapper.forEachRemaining(Wrappers.scala:30) at java.base/java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801) at java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:658) at io.openlineage.spark3.agent.lifecycle.plan.column.InputFieldsCollector.collect(InputFieldsCollector.java:64) at io.openlineage.spark3.agent.lifecycle.plan.column.InputFieldsCollector.lambda$collect$0(InputFieldsCollector.java:64) at java.base/java.util.Iterator.forEachRemaining(Iterator.java:133) at scala.collection.convert.Wrappers$IteratorWrapper.forEachRemaining(Wrappers.scala:30) at java.base/java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801) at java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:658) at io.openlineage.spark3.agent.lifecycle.plan.column.InputFieldsCollector.collect(InputFieldsCollector.java:64) at io.openlineage.spark3.agent.lifecycle.plan.column.InputFieldsCollector.lambda$collect$0(InputFieldsCollector.java:64) at java.base/java.util.Iterator.forEachRemaining(Iterator.java:133) at scala.collection.convert.Wrappers$IteratorWrapper.forEachRemaining(Wrappers.scala:30) at java.base/java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801) at java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:658) at io.openlineage.spark3.agent.lifecycle.plan.column.InputFieldsCollector.collect(InputFieldsCollector.java:64) at io.openlineage.spark3.agent.lifecycle.plan.column.InputFieldsCollector.lambda$collect$0(InputFieldsCollector.java:64) at java.base/java.util.Iterator.forEachRemaining(Iterator.java:133) at scala.collection.convert.Wrappers$IteratorWrapper.forEachRemaining(Wrappers.scala:30) at java.base/java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801) at java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:658) at io.openlineage.spark3.agent.lifecycle.plan.column.InputFieldsCollector.collect(InputFieldsCollector.java:64) at io.openlineage.spark3.agent.lifecycle.plan.column.InputFieldsCollector.lambda$collect$0(InputFieldsCollector.java:64) at java.base/java.util.Iterator.forEachRemaining(Iterator.java:133) at scala.collection.convert.Wrappers$IteratorWrapper.forEachRemaining(Wrappers.scala:30) at java.base/java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801) at java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:658) at io.openlineage.spark3.agent.lifecycle.plan.column.InputFieldsCollector.collect(InputFieldsCollector.java:64) at io.openlineage.spark3.agent.lifecycle.plan.column.InputFieldsCollector.lambda$collect$0(InputFieldsCollector.java:64) at java.base/java.util.Iterator.forEachRemaining(Iterator.java:133) at scala.collection.convert.Wrappers$IteratorWrapper.forEachRemaining(Wrappers.scala:30) at java.base/java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801) at java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:658) at io.openlineage.spark3.agent.lifecycle.plan.column.InputFieldsCollector.collect(InputFieldsCollector.java:64) at io.openlineage.spark3.agent.lifecycle.plan.column.InputFieldsCollector.lambda$collect$0(InputFieldsCollector.java:64) at java.base/java.util.Iterator.forEachRemaining(Iterator.java:133) at scala.collection.convert.Wrappers$IteratorWrapper.forEachRemaining(Wrappers.scala:30) at java.base/java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801) at java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:658) at io.openlineage.spark3.agent.lifecycle.plan.column.InputFieldsCollector.co…</p> +
+ +
+
Labels
+ proposal +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-06-19 09:37:20
 +
+

*Thread Reply:* Hey @Anirudh Shrinivason, me and Paweł are at Berlin Buzzwords right now. Will definitely look at it later

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-06-19 10:47:06
 +
+

*Thread Reply:* Oh nice! Thanks!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ayush mittal + (ayushmittal057@gmail.com) +
+
2023-06-20 03:14:02
 +
+

Hi Team, we are not able to generate lineage for aggregate functions while joining two tables. below is the query +df2 = spark.sql("select th.ProductID as Pid, pd.Name as N, sum(th.quantity) as TotalQuantity, sum(th.ActualCost) as TotalCost from silveradventureworks.transactionhistory as th join productdescription_dim as pd on th.ProductID = pd.ProductID group by th.ProductID, pd.Name ")

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Rahul + (rahul812ry@gmail.com) +
+
2023-06-20 03:47:50
 +
+

*Thread Reply:* This is the event generated for above query.

+ +
+
log4j-active (11).txt
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ayush mittal + (ayushmittal057@gmail.com) +
+
2023-06-20 03:18:22
 +
+

and one more issue, we are not able to generate the open lineage events on top of view being created by joining multiple tables. +i have attached log events for your reference.

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ayush mittal + (ayushmittal057@gmail.com) +
+
2023-06-20 03:31:11
 +
+

this is event for view for which no lineage is being generated

+ +
+
view.txt
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Lukenoff + (john@jlukenoff.com) +
+
2023-06-20 13:59:00
 +
+

Has anyone here successfully implemented the Amundsen OpenLineage extractor? I’m a little confused on the best way to output my lineage events to ndjson files in a scalable way as the docs seem to suggest. Currently I’m pushing all my lineage events to Marquez via REST API. I suppose I could change my transports to Kinesis and write the events to s3 but that comes with the cost of having to build some new way of getting the events to Marquez.

+ +

In any case, this seems like a problem someone must have solved before?

+ +

Edit: looking at the source code for this Amundsen extractor, it seems like it should be pretty straightforward to just implement our own extractor that can pull these records from the Marquez backend. Will give that a shot and see about getting that merged into Amundsen later.

+ + + +
+ 👍 Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-06-20 17:34:08
 +
+

*Thread Reply:* Hi John, glad to hear you figured out a path forward on this! Please let us know what you learn 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-06-20 14:21:03
 +
+

Our New York meetup with Collibra is happening in just two days! https://openlineage.slack.com/archives/C01CK9T7HKR/p1686594956030059

+
+ + +
+ + + } + + Michael Robinson + (https://openlineage.slack.com/team/U02LXF3HUN7) +
+ + + +
+
+ <p>@channel +Meetup announcement: there’s another meetup happening soon! This one will be an evening event on 6/22 in New York at Collibra’s HQ. For details and to sign up, please join the meetup group: <a href="https://www.meetup.com/data-lineage-meetup/events/294065396/">https://www.meetup.com/data-lineage-meetup/events/294065396/</a>. Thanks to @Sheeri Cabral (Collibra) for cohosting and providing a space.</p> +
+ + + + + + +
Original URL: https://openlineage.slack.com/archives/C01CK9T7HKR/p1686594956030059
+ + +
+ + Slack Conversation +
+ + +
+ + + +
+ 👍 Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harshini Devathi + (harshini.devathi@tigeranalytics.com) +
+
2023-06-20 14:31:56
 +
+

Hello all, Do you know if we have th possibility of persisting column orders while creating lineage as it may be available in the table or data set from which it originates. Or, is there some way in which we can get the column order (id or something).

+ +

For example, if a dataset has columns xyz, abc, fgh, dec, I would like to know which column shows first in the dataset in the common data model. Please let me know. m

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-06-20 17:33:36
 +
+

*Thread Reply:* Hi Harshini, I’ve alerted our resident Spark and column-lineage expert about this. Hope to have an answer for you soon.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harshini Devathi + (harshini.devathi@tigeranalytics.com) +
+
2023-06-20 19:39:46
 +
+

*Thread Reply:* Thank you Michael, looking forward to it

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-06-21 02:58:41
 +
+

*Thread Reply:* Hello @Harshini Devathi. An interesting topic which I have never thought about. The ordering of the fields, we get for Spark Apps, comes from Spark logical plans we extract information from and we do not apply any sorting on them. So, if Spark plan contains columns a , b, c we trust it's the order of columns for a dataset and don't want to check it on our own.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-06-21 02:59:45
 +
+

*Thread Reply:* btw. please let us know how do you obtain your lineage: within a Spark app or from some SQL's scheduled by Airflow?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harshini Devathi + (harshini.devathi@tigeranalytics.com) +
+
2023-06-23 14:40:31
 +
+

*Thread Reply:* Hello @Paweł Leszczyński, thank you for the response. We do not need you to check the ordering specifically but I assume that the spark logical plan maintains the column order based on the input datasets. Can we retain that order by adding column id or some sequence number which helps to represent the lineage in the same order.

+ +

The lineage we are capturing using Spark openlineage connector, by posting custom lineage to Marquez through API calls, and also in process of leveraging SQL connector feature using Airflow.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-06-26 04:35:43
 +
+

*Thread Reply:* Hi @Harshini Devathi, are you asking about schema facet within a dataset? This should have an order from spark logical plans. Or, are you asking about columnLineage facet? Or Marquez API responses? It's not clear to me why do you need it. Each column, is identified by a dataset (dataset namespace + dataset name) and field name. You can, on your side, generate and column id based on that and order columns based on the id, but still I think I am missing some arguments behind doing so.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-06-21 17:41:48
 +
+

Attention all Bay-area data friends and Data+AI Summit attendees: our first San Francisco meetup is next Tuesday! https://www.meetup.com/meetup-group-bnfqymxe/events/293448130/

+
+
Meetup
+ + + +
OpenLineage Meetup @ Astronomer, Tue, Jun 27, 2023, 5:30 PM | Meetup
+
+ Data engineers and pipeline managers know that producing data lineage – end-to-end pipeline metadata instrumented at runtime or parsed at design time – is a heavy lift with +
+ + + + + + + + +
Original URL: https://www.meetup.com/meetup-group-bnfqymxe/events/293448130/
+ + + +
+ + + +
+ 🙌 alexandre bergere +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-06-23 16:41:29
 +
+

Last night in New York we held a meetup with Collibra at their lovely HQ in the Financial District! Many thanks to @Sheeri Cabral (Collibra) for inviting us. +Over a bunch of tasty snacks (thanks for the idea @Harel Shein), we discussed: +• the history and evolution of the spec, and trends in adoption +• progress on the OpenLineage Provider in Airflow (AIP 53) +• progress on “static” AKA design lineage support (expected soon in OpenLineage 1.0.0) +• progress in the LFAI program +• a proposal to add “jobless run” support for auditing use cases and similar edge cases +• an idea to throw a hackathon for creating validation tests and example payloads (would you be interested in participating? let us know!) +• and more. +Many thanks to: +• @Julien Le Dem for making the trip +• Sheeri & Collibra for hosting +• everyone for coming, including second-timer @Ernie Ostic and new member @Shirley Lu +It was great meeting/catching up with everyone. Hope to see you and more new faces at the next one!

+ +
+
+ + + + + + +
+ + +
+ 🎉 Harel Shein, Peter Hanssens, Ernie Ostic, Paweł Leszczyński, Maciej Obuchowski, Shirley Lu +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-06-26 10:59:08
 +
+

Our first San Francisco meetup is tomorrow at 5:30 PM at Astronomer’s offices in the Financial District. https://openlineage.slack.com/archives/C01CK9T7HKR/p1687383708927189

+
+ + +
+ + + } + + Michael Robinson + (https://openlineage.slack.com/team/U02LXF3HUN7) +
+ + + +
+
+ <p>Attention all Bay-area data friends and Data+AI Summit attendees: our first San Francisco meetup is next Tuesday! <a href="https://www.meetup.com/meetup-group-bnfqymxe/events/293448130/">https://www.meetup.com/meetup-group-bnfqymxe/events/293448130/</a></p> +
+ + + + + + +
Original URL: https://openlineage.slack.com/archives/C01CK9T7HKR/p1687383708927189
+ + +
+ + Slack Conversation +
+ + +
+ + + +
+ 🚀 alexandre bergere +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Rakesh Jain + (rakeshj@us.ibm.com) +
+
2023-06-27 03:43:10
 +
+

I can’t seem to get OL logging working with Spark. Any guidance please?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-06-27 03:45:31
 +
+

*Thread Reply:* Is it because the logLevel is set to WARN or ERROR?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Rakesh Jain + (rakeshj@us.ibm.com) +
+
2023-06-27 12:07:12
 +
+

*Thread Reply:* No, I set it to INFO, may be I need to add some jars?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-06-27 12:30:02
 +
+

*Thread Reply:* Hmm have you set the relevant spark configs?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Rakesh Jain + (rakeshj@us.ibm.com) +
+
2023-06-27 12:32:50
 +
+

*Thread Reply:* yep, I have http working. But not the console +spark.extraListeners=io.openlineage.spark.agent.OpenLineageSparkListener +spark.openlineage.transport.type=console

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-06-27 12:35:27
 +
+

*Thread Reply:* Oh wait http works but not console...

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-06-27 12:37:02
 +
+

*Thread Reply:* If you want to see the console events which are emitted, then need to set logLevel to DEBUG

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Rakesh Jain + (rakeshj@us.ibm.com) +
+
2023-06-27 12:37:44
 +
+

*Thread Reply:* tried that too, still nothing

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-06-27 12:38:54
 +
+

*Thread Reply:* Is the openlienage jar installed and added to config?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Rakesh Jain + (rakeshj@us.ibm.com) +
+
2023-06-27 12:39:09
 +
+

*Thread Reply:* yep, that’s why http works

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Rakesh Jain + (rakeshj@us.ibm.com) +
+
2023-06-27 12:39:26
 +
+

*Thread Reply:* the only thing I see in the logs is this: +23/06/27 07:39:11 INFO SparkSQLExecutionContext: OpenLineage received Spark event that is configured to be skipped: SparkListenerJobEnd

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-06-27 12:40:59
 +
+

*Thread Reply:* Hmm if an event is still emitted for this case, but logs not showing up then I'm not sure... Maybe someone with more knowledge on this can help

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Rakesh Jain + (rakeshj@us.ibm.com) +
+
2023-06-27 12:42:37
 +
+

*Thread Reply:* sure, thanks for trying @Anirudh Shrinivason

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-06-28 05:23:36
 +
+

*Thread Reply:* What job are you trying this on? If there's this message, then logging is working afaik

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-06-28 12:16:52
 +
+

*Thread Reply:* Hi @Maciej Obuchowski Actually I also noticed a similar issue... For some spark pipelines, the log level is set to debug, but I'm not seeing any events being logged. I am however receiving these events in the backend. Have any of the logging been removed from some places?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Rakesh Jain + (rakeshj@us.ibm.com) +
+
2023-06-28 20:57:45
 +
+

*Thread Reply:* yep, exactly same thing here also @Maciej Obuchowski, I can get the events on http, but changing to console gets me nothing from ConsoleTransport.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Lukenoff + (john@jlukenoff.com) +
+
2023-06-27 20:45:15
 +
+

@here A bunch of us are downstairs in the lobby at 8 California but no one is down here to let us up. Anyone here to help?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-06-29 03:36:36
 +
+

Hi guys, I noticed a few of the jobs getting OOMed while running with openlineage. Even increasing the number of executors and doubling the memory does not seem to fix it actually. This is observed especially when using the graphx libs. Is this a known issue? Just curious as to what the cause might be... The same jobs run fine once openlineage is disabled. Are there some rogue threads from the listener or any connections we aren't closing properly?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-06-29 05:57:59
 +
+

*Thread Reply:* Hi @Anirudh Shrinivason, could you disable serializing spark.logicalPlan to see if the behaviour is the same?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-06-29 05:58:28
 +
+

*Thread Reply:* https://github.com/OpenLineage/OpenLineage/tree/main/integration/spark -> spark.openlineage.facets.disabled -> [spark_unknown;spark.logicalPlan]

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-06-29 05:59:55
 +
+

*Thread Reply:* We do serialize logicalPlan because this is useful in many cases, but sometimes can lead to serializing things that shouldn't be serialized

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-06-29 15:49:35
 +
+

*Thread Reply:* Ahh I see. Yeah okay let me try that

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-06-30 08:01:34
 +
+

Hello all, I’m opening a vote to release OpenLineage 0.29.0, including: +• support for Spark 3.4 +• support for Flink 1.17.1 +• a fix in the Flink integration to enable dataset schema extraction for a KafkaSource when GenericRecord is used +• removal of the unused Golang proxy client (made redundant by the fluentd proxy) +• security vulnerability fixes, doc changes, test improvements, and more. +Three +1s from committers will authorize an immediate release.

+ + + +
+ ➕ Jakub Dardziński, Paweł Leszczyński, Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-06-30 08:05:53
 +
+

*Thread Reply:* Thanks, all. The release is authorized.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-06-30 13:27:35
 +
+

@channel +We released OpenLineage 0.29.2, including: +Added +• Flink: support Flink version 1.17.1 #1947 @pawel-big-lebowski +• Spark: support Spark version 3.4 #1790 @pawel-big-lebowski +Removed +• Proxy: remove unused Golang client approach #1926 @mobuchowski +• Req: bump minimum supported Python version to 3.8 #1950 @mobuchowski + ◦ Note: this removes support for Python 3.7, which is at EOL. +Plus test improvements, docs changes, bug fixes and more. +Thanks to all the contributors! +Release: https://github.com/OpenLineage/OpenLineage/releases/tag/0.29.2 +Changelog: https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md +Commit history: https://github.com/OpenLineage/OpenLineage/compare/0.28.0...0.29.2 +Maven: https://oss.sonatype.org/#nexus-search;quick~openlineage +PyPI: https://pypi.org/project/openlineage-python/

+ + + +
+ 👍 Shirley Lu, Maciej Obuchowski, Paweł Leszczyński, Tamara Fingerlin +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-06-30 17:23:04
 +
+

@channel +The latest issue of OpenLineage News is now available, featuring a recap of recent events, releases, and more. To get it directly in your inbox each month, sign up https://openlineage.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&id=e598962936&e=ef0563a7f8|here.

+
+
apache.us14.list-manage.com
+ + + +
OpenLineage Project
+
+ OpenLineage Project Email Forms +
+ + + + + + +
Original URL: https://openlineage.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&amp;id=e598962936&amp;e=ef0563a7f8
+ + + +
+ +
+
+ + + + + + +
+ + +
+ 👍 Maciej Obuchowski, Paweł Leszczyński, Tristan GUEZENNEC -CROIX-, Tamara Fingerlin, Jeremy W, Anirudh Shrinivason, Julien Le Dem, Sheeri Cabral (Collibra), alexandre bergere +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-07-06 13:36:44
 +
+

@channel +This month’s TSC meeting is next Thursday, 7/13, at a special time: 8 am PT. +All are welcome! +On the tentative agenda: +• announcements +• updates +• recent releases +• a new DataGalaxy integration +• open discussion

+ + + +
+ ✅ Sheeri Cabral (Collibra), Maciej Obuchowski, alexandre bergere, Paweł Leszczyński, Willy Lulciuc, Anirudh Shrinivason, Shirley Lu +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sheeri Cabral (Collibra) + (sheeri.cabral@collibra.com) +
+
2023-07-07 10:35:08
 +
+

Wow, I just got finished watching @Julien Le Dem and @Willy Lulciuc’s presentation of OpenLineage at databricks and it’s really fantastic! There isn’t a better 30 minutes of content on theory + practice than this, IMO. https://www.databricks.com/dataaisummit/session/cross-platform-data-lineage-openlineage/ (you can watch for free by making an account. I’m not affiliated with databricks…)

+
+
databricks.com
+ + + +
Cross-Platform Data Lineage with OpenLineage - Data + AI Summit 2023 | Databricks
+
+ +
+ + + + + + + + +
Original URL: https://www.databricks.com/dataaisummit/session/cross-platform-data-lineage-openlineage/
+ + + +
+ + + +
+ ❤️ Willy Lulciuc, Harel Shein, Yuanli Wang, Ross Turk, Michael Robinson, Jakub Dardziński, Conor Beverland, Maciej Obuchowski, Jarek Potiuk, Julien Le Dem, Chris Folkes, Anirudh Shrinivason, Shirley Lu +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2023-07-07 10:37:49
 +
+

*Thread Reply:* thanks for watching and sharing! the recording is also on youtube 😉 https://www.youtube.com/watch?v=rO3BPqUtWrI

+
+
YouTube
+ +
+ + + } + + Databricks + (https://www.youtube.com/@Databricks) +
+ + + +
Cross-Platform Data Lineage with OpenLineage
+
+ +
+ + + + + + + + +
Original URL: https://www.youtube.com/watch?v=rO3BPqUtWrI
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sheeri Cabral (Collibra) + (sheeri.cabral@collibra.com) +
+
2023-07-07 10:38:01
 +
+

*Thread Reply:* ❤️

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jarek Potiuk + (jarek@potiuk.com) +
+
2023-07-08 13:35:10
 +
+

*Thread Reply:* Very much agree. I’ve even forwarded to a few people here and there, those who I think should learn about it.

+ + + +
+ ❤️ Sheeri Cabral (Collibra) +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2023-07-08 13:47:17
 +
+

*Thread Reply:* You’re both too kind :) +Thank you for your support and being part of the community.

+ + + +
+ ❤️ Sheeri Cabral (Collibra), Jarek Potiuk +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-07-07 15:44:33
 +
+

@channel +If you registered for TSC meetings through AddEvent, first of all, thank you! Second of all, I’ve had to create a new event series there to enable the editing of individual events. When you have a moment, would you please register for next week’s meeting? Apologies for the inconvenience.

+
+
addevent.com
+ + + +
OpenLineage Monthly TSC Meeting
+
+ This is the monthly meeting for the OpenLineage maintainers.Agenda:The agenda is updated the week prior to the meetingArchive:   <https://wiki.lfaidata.foundation/display/OpenLineage/Monthly+TSC+meeting> - How to be part of the discussion:Slack: <https://bit.ly/lineageslackMailing> list: <https://lists.lfaidata.foundation/g/openlineage-tscConfluence>: <https://wiki.lfaidata.foundation/display/OpenLineage/OpenLineage+Home> +
+ + + + + + + + +
Original URL: https://evt.to/ahdseuidw
+ + + +
+ + + +
+ 👍 Kiran Hiremath, Willy Lulciuc, Shirley Lu +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Juan Manuel Cappi + (juancappi@gmail.com) +
+
2023-07-10 12:29:31
 +
+

Hi community, we are interested in capturing time-travel usage for Iceberg Spark sql in column lineage. For instance, INSERT INTO schema.table select ** from schema.another_table version as of 'some_version' . Column lineage is currently missing the version, if used, which it’s actually quite relevant. I’ve gone through the open issues and didn’t see anything similar. Does it look like a valid use case scenario? We started going through the OL, iceberg and Spark code in trying to capture/expose it, but so far we haven’t been able to. If anyone can give a hint/idea/pointer, we are willing to give it try a contribute back with the code

+ + + +
+ 👀 Rakesh Jain, Nitin Ramchandani +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2023-07-11 05:46:36
 +
+

*Thread Reply:* I think yes this is a great use case. @Paweł Leszczyński is more familiar with the spark integration code than I. +I think in this case, we would add the datasetVersion facet with the underlying Iceberg version: https://github.com/OpenLineage/OpenLineage/blob/main/spec/facets/DatasetVersionDatasetFacet.json +We extract this information in a few places: +https://github.com/search?q=repo%3AOpenLineage%2FOpenLineage%20%20DatasetVersionDatasetFacet&type=code

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/spec/facets/DatasetVersionDatasetFacet.json | DatasetVersionDatasetFacet.json>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-07-11 05:57:17
 +
+

*Thread Reply:* Yes, we do have datasetVersion which captures for output and input datasets their iceberg or delta version. Input versions are collected on START while output are collected on COMPLETE in case a job reads and writes to the same dataset. So, even though column-lineage facet is missing the version, it should be available within events related to a particular run.

+ +

If it is not, then perhaps the case here is the lack of support of as of syntax. As far as I remeber, we always get a current version of a dataset and this may be a missing part here.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-07-11 05:58:49
 +
+

*Thread Reply:* link to a method that gets dataset version for iceberg: https://github.com/OpenLineage/OpenLineage/blob/0.29.2/integration/spark/spark3/sr[…]lineage/spark3/agent/lifecycle/plan/catalog/IcebergHandler.java

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/0.29.2/integration/spark/spark3/src/main/java/io/openlineage/spark3/agent/lifecycle/plan/catalog/IcebergHandler.java | IcebergHandler.java>
+
+ <pre><code> public Optional&lt;String&gt; getDatasetVersion( +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Juan Manuel Cappi + (juancappi@gmail.com) +
+
2023-07-11 10:57:26
 +
+

*Thread Reply:* Thank you @Julien Le Dem and @Paweł Leszczyński +Based on what I’ve seen so far, indeed it seems that only the current snapshot is tracked. When IcebergHandler.getDatasetVersion() +Initially I was expecting to be able to obtain the snapshotId from the SparkTable which comes within getDatasetVersion() but now I realize that OL is using an older version of Iceberg runtime, (0.12.1) which does not support time travel (introduced in 0.14.1). +The evidence is: +• Iceberg documentation for release 0.14.1: https://iceberg.apache.org/docs/0.14.0/spark-queries/#sql +• Iceberg release notes https://iceberg.apache.org/releases/#0140-release +• Comparing the source code, I see the SparkTable from 0.14.1 onward does have a snapshotId instance variable, while previous versions don’t +https://github.com/apache/iceberg/blob/0.14.x/spark/v3.0/spark/src/main/java/org/apache/iceberg/spark/source/SparkTable.java#L82 +https://github.com/apache/iceberg/blob/0.12.x/spark3/src/main/java/org/apache/iceberg/spark/source/SparkTable.java#L78

+ +

I don’t see anyone complaining about the old version of Iceberg runtime being used and there is no open issue to upgrade so I’ll open the issue and please let me know if that seems reasonable as the immediate next step to take

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Juan Manuel Cappi + (juancappi@gmail.com) +
+
2023-07-11 15:48:53
 +
+

*Thread Reply:* Created issues: #1969 and #1970

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-07-12 07:15:14
 +
+

*Thread Reply:* Thanks @Juan Manuel Cappi. openlineage-spark jar contains modules like spark3 , spark32 , spark33 and spark34 that is going to be merged soon (we do have a ready PR for that). spark34 will be compiled against latest iceberg version. Once this is done #1969 can be closed. For 1970, one would need to implement datasetBuilder within spark34 module and visits node within spark's logical plan that is responsible for as of and creates dataset for OpenLineage event other way than getting latest snapshot version.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Juan Manuel Cappi + (juancappi@gmail.com) +
+
2023-07-13 12:51:19
 +
+

*Thread Reply:* @Paweł Leszczyński I’ve see PR #1971 and I see a new spark34 project with the latest iceberg-spark dependency version, but other versions (spark33, spark32, etc) have not being upgraded in that PR. Since the change is small and does not break any tests, I’ve created PR #1976 for to fix #1969. That alone is unlocking some time travel lineage (i.e. dataset identifier now becomes schema.table.version or schema.table.snapshot_id). Hope it makes sense

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-07-14 04:37:55
 +
+

*Thread Reply:* Hi @Juan Manuel Cappi, You're right and after discussion with you I realized we support some version of iceberg (for spark 3.3 it's still 0.14.0) but this is not the latest iceberg version matching spark version.

+ +

There's some tricky part here. Although we wan't our code to succeed with latest spark, we don't want it to fail in a nasty way (class not found exception) when a user is working with an old iceberg version. There are places in our code where we do check are iceberg classes on the classpath? We need to extend this to are iceberg classes on classpath is iceberg version above 0.14 or not For sure this is the case for merge into commands I am working on at the moment. Let's see if the other integration tests are affected in your PR

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Amod Bhalerao + (amod.bhalerao@gmail.com) +
+
2023-07-11 08:09:57
 +
+

HI Team, I Seen that Kafka lineage is not coming properly in for Spark streaming, Are we working on this?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-07-11 08:28:59
 +
+

*Thread Reply:* what do you mean by that? there is a pyspark & kafka integration test that verifies event being sent when reading or writing to kafka topic: https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/app/src/tes[…]a/io/openlineage/spark/agent/SparkContainerIntegrationTest.java

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/app/src/test/java/io/openlineage/spark/agent/SparkContainerIntegrationTest.java | SparkContainerIntegrationTest.java>
+
+ <pre><code> void testPysparkKafkaReadWrite() { +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-07-11 09:28:56
 +
+

*Thread Reply:* We do have an old issue https://github.com/OpenLineage/OpenLineage/issues/372 to support more spark plans that are stream related. But, if you had an example of streaming that is not working for you, this would have been really helpful.

+
+ + + + +
#372 spark: Cover streaming LogicalPlans
+
+ <p>Cover implementations of LogicalPlan in <br /> +org.apache.spark.sql.execution.streaming.continuous <br /> +WriteToContinuousDataSource <br /> +org.apache.spark.sql.execution.streaming.sources <br /> +MemoryPlan <br /> +WriteToMicroBatchDataSource</p> +
+ +
+
Labels
+ integration/spark, streaming +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Amod Bhalerao + (amod.bhalerao@gmail.com) +
+
2023-07-26 08:03:30
 +
+

*Thread Reply:* I have a pipeline Which reads from topic and send data to 3 HIVE tables and one postgres , Its not emitting any lineage for this pipeline

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Amod Bhalerao + (amod.bhalerao@gmail.com) +
+
2023-07-26 08:06:51
 +
+

*Thread Reply:* just one task is getting created

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-07-12 05:55:19
 +
+

Hi guys, I notice that with the below spark configs: +```from pyspark.sql import SparkSession +import os

+ +

os.environ["TEST_VAR"] = "1"

+ +

spark = (SparkSession.builder.master('local') + .appName('samplespark') + .config('spark.jars.packages', 'io.openlineage:openlineagespark:0.29.2,io.delta:deltacore2.12:1.0.1') + .config('spark.extraListeners', 'io.openlineage.spark.agent.OpenLineageSparkListener') + .config('spark.openlineage.transport.type', 'console') + .config('spark.sql.catalog.sparkcatalog', "org.apache.spark.sql.delta.catalog.DeltaCatalog") + .config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension") + .config("hive.metastore.schema.verification", False) + .config("spark.sql.warehouse.dir","/tmp/") + .config("hive.metastore.warehouse.dir","/tmp/") + .config("javax.jdo.option.ConnectionURL","jdbc:derby:;databaseName=/tmp/metastoredb;create=true") + .config("spark.openlineage.facets.customenvironmentvariables","[TESTVAR;]") + .config("spark.openlineage.facets.disabled","[sparkunknown;spark.logicalPlan]") + .config("spark.hadoop.fs.permissions.unmask-mode","000") + .enableHiveSupport() + .getOrCreate())``` +The custom environment variables facet is not kicking in. However, when all the delta related spark configs are removed, it is working fine. Is this a known issue? Are there any workarounds for it? Thanks!

+ + + +
+ 👀 Paweł Leszczyński +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Juan Manuel Cappi + (juancappi@gmail.com) +
+
2023-07-12 06:14:41
 +
+

*Thread Reply:* Hi @Anirudh Shrinivason, I’m not familiar with Delta, but enabling debugging helped me a lot to understand what’s going when things fail silently. Just add at the end: +spark.sparkContext.setLogLevel("DEBUG")

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-07-12 06:20:47
 +
+

*Thread Reply:* Yeah I checked on debug

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-07-12 06:20:50
 +
+

*Thread Reply:* There are no errors

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-07-12 06:21:10
 +
+

*Thread Reply:* Just that there is no environment-properties in the event that is being emitted

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-07-12 07:31:01
 +
+

*Thread Reply:* Hi @Anirudh Shrinivason, what spark version is that? i see you delta version is pretty old. Anyway, the observation is weird and don't know how come delta interferes with environment facet builder. These are so disjoint features. Are you sure you create a new session (there is getOrCreate) ?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Glen M + (glen_m@apple.com) +
+
2023-07-12 19:29:06
 +
+

*Thread Reply:* @Paweł Leszczyński its because of this line : https://github.com/OpenLineage/OpenLineage/blob/0.29.2/integration/spark/app/src/m[…]nlineage/spark/agent/lifecycle/InternalEventHandlerFactory.java

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/0.29.2/integration/spark/app/src/main/java/io/openlineage/spark/agent/lifecycle/InternalEventHandlerFactory.java | InternalEventHandlerFactory.java>
+
+ <pre><code> } else if (context.getCustomEnvironmentVariables() != null) { +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Glen M + (glen_m@apple.com) +
+
2023-07-12 19:32:44
 +
+

*Thread Reply:* Assuming this is https://learn.microsoft.com/en-us/azure/databricks/delta/ ... delta .. which is azure datbricks. @Anirudh Shrinivason

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-07-12 22:58:13
 +
+

*Thread Reply:* Hmm I wasn't using databricks

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-07-12 22:59:12
 +
+

*Thread Reply:* @Paweł Leszczyński I'm using spark 3.1 btw

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-07-13 08:05:49
 +
+

*Thread Reply:* @Anirudh Shrinivason This should resolve the issue https://github.com/OpenLineage/OpenLineage/pull/1973

+
+ + + + +
#1973 [SPARK] fix custom environment variables facet
+
+ <p><strong>Problem</strong></p> + +<p>Custom environment facet behaviour is non-deterministic: it's present in some case and it's not available when working on delta tables in Spark (<a href="https://openlineage.slack.com/archives/C01CK9T7HKR/p1689155719105569">https://openlineage.slack.com/archives/C01CK9T7HKR/p1689155719105569</a>). This is weird as the facet and delta format have nothin in common. However they do, as <code>CustomEnvironmentFacetBuilder</code> has been implemented to be triggered <code>SparkListenerJobStart</code>. We do have two types of start events: <code>SparkListenerJobStart</code> and <code>SparkListenerSQLExecutionStart</code> and Openlineage event is only sent for one of them. It seems as other event type got triggered with delta tables.</p> + +<p><strong>Solution</strong></p> + +<p>• Prepare an integration test to verify facet is there. +• Fix <code>CustomEnvironmentFacetBuilder</code> class.</p> + +<blockquote> + <p><strong>Note:</strong> All schema changes require discussion. Please <a href="https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue">link the issue</a> for context.</p> +</blockquote> + +<p>☐ Your change modifies the <a href="https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.json">core</a> OpenLineage model +☐ Your change modifies one or more OpenLineage <a href="https://github.com/OpenLineage/OpenLineage/tree/main/spec/facets">facets</a></p> + +<p>If you're contributing a new integration, please specify the scope of the integration and how/where it has been tested (e.g., Apache Spark integration supports <code>S3</code> and <code>GCS</code> filesystem operations, tested with AWS EMR).</p> + +<p><strong>One-line summary:</strong> +<strong>Checklist</strong></p> + +<p>☐ You've <a href="https://github.com/OpenLineage/OpenLineage/blob/main/why-the-dco.md">signed-off</a> your work +☐ Your pull request title follows our <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CONTRIBUTING.md#creating-pull-requests">guidelines</a> +☐ Your changes are accompanied by tests (<em>if relevant</em>) +☐ Your change contains a <a href="https://kurtisnusbaum.medium.com/stacked-diffs-keeping-phabricator-diffs-small-d9964f4dcfa6">small diff</a> and is self-contained +☐ You've updated any relevant documentation (<em>if relevant</em>) +☐ Your comment includes a one-liner for the changelog about the specific purpose of the change (<em>if necessary</em>) +☐ You've versioned the core OpenLineage model or facets according to <a href="https://docs.snowplowanalytics.com/docs/pipeline-components-and-applications/iglu/common-architecture/schemaver">SchemaVer</a> (<em>if relevant</em>) +☐ You've added a <a href="https://github.com/OpenLineage/OpenLineage/tree/main/.github/header_templates.md">header</a> to source files (<em>if relevant</em>)</p> + +<hr /> + +<p>SPDX-License-Identifier: Apache-2.0 <br /> +Copyright 2018-2023 contributors to the OpenLineage project</p> +
+ +
+
Labels
+ documentation, integration/spark +
+ +
+
Comments
+ 1 +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-07-13 08:06:11
 +
+

*Thread Reply:* PR description contains info on how come the observed behaviour was possible

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-07-13 08:07:47
 +
+

*Thread Reply:* As always, thank you @Anirudh Shrinivason for providing clear information on how to reproduce the issue 🚀 :medal: 👍

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-07-13 09:52:29
 +
+

*Thread Reply:* Ohh that is really great! Thankss so much for the help! 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-07-12 13:50:51
 +
+

@channel +A friendly reminder: this month’s TSC meeting — open to all — is tomorrow at 8 am PT. https://openlineage.slack.com/archives/C01CK9T7HKR/p1688665004736219

+
+ + +
+ + + } + + Michael Robinson + (https://openlineage.slack.com/team/U02LXF3HUN7) +
+ + + +
+
+ <p>@channel +This month’s TSC meeting is next Thursday, 7/13, at a special time: <a href="https://astronomer.zoom.us/j/87156607114?pwd=a3B0K210dnRaQmdkaFdGMytBREZEQT09">8 am PT</a>. +All are welcome! +On the tentative agenda: +• announcements +• updates +• recent releases +• a new DataGalaxy integration +• open discussion</p> +
+ + + + + + +
Original URL: https://openlineage.slack.com/archives/C01CK9T7HKR/p1688665004736219
+ + +
+ + Slack Conversation +
+ + +
+ + + +
+ 👍 Dongjin Seo +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
thebruuu + (bruno.c@inwind.it) +
+
2023-07-12 14:54:29
 +
+

Hi Team +How are you ? +Is there any chance to use airflow to run queries against Access file? +Sorry to bother with a question that is not directly related to openlineage ... but I am kind of stuck

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harel Shein + (harel.shein@gmail.com) +
+
2023-07-12 15:22:52
 +
+

*Thread Reply:* what do you mean by Access file?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
thebruuu + (bruno.c@inwind.it) +
+
2023-07-12 16:09:03
 +
+

*Thread Reply:* ... accdb file, Microsoft Access File: I am in a reverse engineering projet facing a spaghetti style development and would have loved to use, airflow and openlineage as a magic wand, to help me in this damn work

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harel Shein + (harel.shein@gmail.com) +
+
2023-07-12 21:44:21
 +
+

*Thread Reply:* oof.. I’d look into https://airflow.apache.org/docs/apache-airflow-providers-odbc/4.0.0/ +but I really have no clue..

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
thebruuu + (bruno.c@inwind.it) +
+
2023-07-13 09:47:02
 +
+

*Thread Reply:* Thank you Harel +I started from that too ... but it became foggy after the initial step

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Aaman Lamba + (aamanlamba@gmail.com) +
+
2023-07-12 16:30:41
 +
+

Hi folks, having an issue ingesting the seed metadata when starting the docker container. The output shows "seed-marquez-with-metadata exited with code 0" but no information is visible in marquez What can be the issue?

+ + + +
+ ✅ Aaman Lamba +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-07-12 16:55:00
 +
+

*Thread Reply:* Did you check the namespace menu in the top right for a food_delivery namespace?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-07-12 16:55:12
 +
+

*Thread Reply:* (Hi Aaman!)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Aaman Lamba + (aamanlamba@gmail.com) +
+
2023-07-12 16:55:45
 +
+

*Thread Reply:* Hi! Thank you that helped!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Aaman Lamba + (aamanlamba@gmail.com) +
+
2023-07-12 16:55:55
 +
+

*Thread Reply:* I think that should be added to the quickstart guide

+ + + +
+ 🙌 Michael Robinson +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-07-12 16:56:23
 +
+

*Thread Reply:* Great idea, thank you

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2023-07-13 12:09:29
 +
+

As discussed in the Monthly meeting, I have opened a PR to propose adding deletion to facets for static lineage metadata: https://github.com/OpenLineage/OpenLineage/pull/1975

+
+ + + + +
#1975 Add facet deletion
+
+ <p><strong>Problem</strong></p> + +<p>The <a href="https://github.com/OpenLineage/OpenLineage/blob/main/proposals/1837/static_lineage.md#semantics-of-facets-across-jobs-and-dataset-version">last section of the static lineage proposal</a> defines semantics of static facets attached to a job or dataset.</p> + +<p>“job and dataset facets apply to that entity from that moment on until they are replaced by a new version of the same facet”</p> + +<p>However this doesn’t allow deleting the facet altogether if the facet is no longer applicable. <br /> +For example: the dataset no longer has an owner.</p> + +<p>This proposal adds a mechanism for deleting job and dataset facets.</p> + +<p><strong>Solution</strong></p> + +<p>We propose to add a { _deleted: true } object that can take the place of any job or dataset facet (but not run or input/output facets which are valid only for a specific run)</p> + +<p>☑︎ Your change modifies the <a href="https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.json">core</a> OpenLineage model</p> + +<p><strong>One-line summary: add a mechanism for deleting job and dataset facets.</strong> +<strong>Checklist</strong></p> + +<p>☐ You've <a href="https://github.com/OpenLineage/OpenLineage/blob/main/why-the-dco.md">signed-off</a> your work +☐ Your pull request title follows our <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CONTRIBUTING.md#creating-pull-requests">guidelines</a> +☐ Your changes are accompanied by tests (<em>if relevant</em>) +☐ Your change contains a <a href="https://kurtisnusbaum.medium.com/stacked-diffs-keeping-phabricator-diffs-small-d9964f4dcfa6">small diff</a> and is self-contained +☐ You've updated any relevant documentation (<em>if relevant</em>) +☐ Your comment includes a one-liner for the changelog about the specific purpose of the change (<em>if necessary</em>) +☐ You've versioned the core OpenLineage model or facets according to <a href="https://docs.snowplowanalytics.com/docs/pipeline-components-and-applications/iglu/common-architecture/schemaver">SchemaVer</a> (<em>if relevant</em>) +☐ You've added a <a href="https://github.com/OpenLineage/OpenLineage/tree/main/.github/header_templates.md">header</a> to source files (<em>if relevant</em>)</p> + +<hr /> + +<p>SPDX-License-Identifier: Apache-2.0 <br /> +Copyright 2018-2023 contributors to the OpenLineage project</p> +
+ +
+
Labels
+ documentation, proposal +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Steven + (xli@zjuici.com) +
+
2023-07-13 23:21:29
 +
+

Hi, I'm using OL python client. +client.emit( + DatasetEvent( + _eventTime_=datetime.now().isoformat(), + _producer_=producer, + _schemaURL_="<https://openlineage.io/spec/1-0-5/OpenLineage.json#/definitions/DatasetEvent>", + _dataset_=Dataset(_namespace_=namespace, _name_=f"input-file"), + ) + ) +I want to send a dataset event once files been uploaded. But I received 422 from api/v1/lineage, saying that run and job must not be null. I don't have a job or run yet. How can I solve this?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-07-14 04:09:15
 +
+

*Thread Reply:* Hi @Steven, I assume you send your Openlineage events to Marquez. 422 http code is a response from backend and Marquez is still waiting for the PR https://github.com/MarquezProject/marquez/pull/2495 to be merged and released. This PR makes Marquez understand DatasetEvents. They won't be saved in Marquez database (this is to be implemented in future), but at least one will not experience error response code.

+ +

To sum up: what you do is correct. You are using a feature that is allowed on a client side but still not implemented on a backend.

+
+ + + + +
#2495 Ability to decode static metadata events
+
+ <p><strong>Problem</strong></p> + +<p>Static metadata events (known also as runless) where presented within in the proposal: <br /> +<a href="https://github.com/OpenLineage/OpenLineage/blob/main/proposals/1837/static_lineage.md">https://github.com/OpenLineage/OpenLineage/blob/main/proposals/1837/static_lineage.md</a></p> + +<p>and is being introduced to Openlineage specification within the PR: <a href="https://github.com/OpenLineage/OpenLineage/pull/1880">OpenLineage/OpenLineage#1880</a></p> + +<p>Of course, we would like to implement it within Marquez. We would like to achieve this ultimate goal gradually and this PR contains the first step.</p> + +<p>The changes introduced within the PR:</p> + +<p>• Don't affect the way <code>RunEvent</code> events are being collected. Events are saved into database and http status code <code>201 (created)</code> is returned. +• <code>DatasetEvent</code> and <code>JobEvent</code> are distinguished by <code>schemaURL</code> property. If distinction cannot be made, events are treated the default way, as <code>RunEvent</code>. +• <code>DatasetEvent</code> and <code>JobEvent</code>events are deserialised by standard Jackso library <code>DatasetEvent</code>/<code>JobEvent</code> classes. The response returns in such cases <code>200 OK</code> http status code but <strong>DOES NOT SAVE</strong> data to database (this will be introduced later). The great benefit of that is that it is a good integration test and a demo of of how to do events' distinction on the server side.</p> + +<p>Relates to: <a href="https://github.com/MarquezProject/marquez/issues/1868">#1868</a></p> + +<p><strong>Solution</strong></p> + +<p>• EventTypeResolver is introduced which decodes incoming <code>/create</code> requests to <code>LineageEvent</code>, <code>DatasetEvent</code> or <code>JobEvent</code> based on the <code>schemaURL</code> field. +• Nothing changes for <code>RunEvents</code>. +• In case of <code>DatasetEvent</code> or <code>JobEvent</code>, they get deserialised and distinguished, however no operation on database is performed.</p> + +<p>One-line summary:</p> + +<p><strong>Checklist</strong></p> + +<p>☑︎ You've <a href="https://github.com/MarquezProject/marquez/blob/main/CONTRIBUTING.md#sign-your-work">signed-off</a> your work +☑︎ Your changes are accompanied by tests (<em>if relevant</em>) +☐ Your change contains a <a href="https://kurtisnusbaum.medium.com/stacked-diffs-keeping-phabricator-diffs-small-d9964f4dcfa6">small diff</a> and is self-contained +☐ You've updated any relevant documentation (<em>if relevant</em>) +☐ You've included a one-line summary of your change for the <a href="https://github.com/MarquezProject/marquez/blob/main/CHANGELOG.md#unreleased"><code>CHANGELOG.md</code></a> (<em>Depending on the change, this may not be necessary</em>). +☐ You've versioned your <code>.sql</code> database schema migration according to <a href="https://flywaydb.org/documentation/concepts/migrations#naming">Flyway's naming convention</a> (<em>if relevant</em>) +☐ You've included a <a href="https://github.com/MarquezProject/marquez/blob/main/CONTRIBUTING.md#copyright--license">header</a> in any source code files (<em>if relevant</em>)</p> +
+ +
+
Labels
+ docs, api, client/java +
+ +
+
Comments
+ 1 +
+ + + + + + + +
+ + <a href="https://github.com/MarquezProject/marquez">MarquezProject/marquez</a> +
+ + +
+ + + +
+ ✅ Steven +
+ +
+ 🥳 Steven +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Steven + (xli@zjuici.com) +
+
2023-07-14 04:10:30
 +
+

*Thread Reply:* Thanks!!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harshit Soni + (harshit.soni@angelbroking.com) +
+
2023-07-14 08:36:23
 +
+

@here Hi Team, I am trying to run a spark application with openLineage +Spark :- 3.3.3 +Openlineage :- 0.29.2 +I am getting below error can you please me, what I could be doing wrong.

+ +

``` spark = (SparkSession + .builder + .config('spark.port.maxRetries', 100) + .appName(app_name) + .config("spark.openlineage.url","http://localhost/api/v1/namespaces/spark_integration/") + .config("spark.extraListeners","io.openlineage.spark.agent.OpenLineageSparkListener") + .getOrCreate())

+ +

23/07/14 18:04:01 ERROR Utils: uncaught error in thread spark-listener-group-shared, stopping SparkContext +java.lang.UnsatisfiedLinkError: /private/var/folders/z6/pl8p30z11v50zf6pv51p259m0000gp/T/native-lib4983292552717270883/libopenlineagesqljava.dylib: dlopen(/private/var/folders/z6/pl8p30z11v50zf6pv51p259m0000gp/T/native-lib4983292552717270883/libopenlineagesqljava.dylib, 0x0001): tried: '/private/var/folders/z6/pl8p30z11v50zf6pv51p259m0000gp/T/native-lib4983292552717270883/libopenlineagesqljava.dylib' (mach-o file, but is an incompatible architecture (have 'x8664', need 'arm64')), '/System/Volumes/Preboot/Cryptexes/OS/private/var/folders/z6/pl8p30z11v50zf6pv51p259m0000gp/T/native-lib4983292552717270883/libopenlineagesqljava.dylib' (no such file), '/private/var/folders/z6/pl8p30z11v50zf6pv51p259m0000gp/T/native-lib4983292552717270883/libopenlineagesqljava.dylib' (mach-o file, but is an incompatible architecture (have 'x8664', need 'arm64')) + at java.lang.ClassLoader$NativeLibrary.load(Native Method)```

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-07-18 02:35:18
 +
+

*Thread Reply:* Hi @Harshit Soni, where are you deploying your spark? locally or not? is it on mac? Calling @Maciej Obuchowski to help with ibopenlineage_sql_java architecture compilation issue

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harshit Soni + (harshit.soni@angelbroking.com) +
+
2023-07-18 02:38:03
 +
+

*Thread Reply:* Currently, was testing on local.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harshit Soni + (harshit.soni@angelbroking.com) +
+
2023-07-18 02:39:43
 +
+

*Thread Reply:* We have created a centralised utility for all data ingestion needs and want to see how lineage is created for same using Openlineage.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-07-18 05:16:55
 +
+

*Thread Reply:* 👀

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-07-14 13:00:29
 +
+

@channel +If you missed this month’s TSC meeting, the recording is now available on our YouTube channel: https://youtu.be/2vD6-Uwr7ZE. +A clip of Alexandre Bergere’s DataGalaxy integration demo is also available: https://youtu.be/l_HbEtpXphY.

+
+
YouTube
+ +
+ + + } + + OpenLineage Project + (https://www.youtube.com/@openlineageproject6897) +
+ + + +
OpenLineage Community Meeting | July 14, 2023
+
+ +
+ + + + + + + + +
Original URL: https://youtu.be/2vD6-Uwr7ZE
+ + + +
+
+
YouTube
+ +
+ + + } + + OpenLineage Project + (https://www.youtube.com/@openlineageproject6897) +
+ + + +
DataGalaxy Integration Demo | July 13, 2023
+
+ +
+ + + + + + + + +
Original URL: https://youtu.be/l_HbEtpXphY
+ + + +
+ + + +
+ 👍 Kiran Hiremath, alexandre bergere, Harel Shein, Paweł Leszczyński +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Robin Fehr + (robin.fehr@acosom.com) +
+
2023-07-16 17:39:26
 +
+

Hey guys - trying to get a grip on the ecosystem regarding flink lineage 🙂 as far as my research has revealed, the openlineage project is the only one that supports flink lineage with an out of the box library that can be integrated in jobs. at least as far as i've seen the for other toolings such as datahub we'd have to write our custom hooks that implement their api. as for my question - is my current assumption correct that an integration into the openlineage project of for example datahub/openmetadata would also require support from datahub/openmetadata itself so that they can work with the openlineage spec? or would it somewhat work to write a mapper in between to support their spec? (more of an architectural decision i assume but would be interested in knowing what the openlinage's approach is regarding that)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-07-17 08:13:49
 +
+

*Thread Reply:* > or would it somewhat work to write a mapper in between to support their spec? +I think yeah - maybe https://github.com/Natural-Intelligence/openLineage-openMetadata-transporter would work out of the box if I understand correctly?

+
+ + + + +
Natural-Intelligence/openLineage-openMetadata-transporter
+
+ Transporter for integrating OpenLineage with OpenMetadata +
+ +
+
Website
+ <https://www.top10.com> +
+ +
+
Language
+ Java +
+ + + + + + + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harel Shein + (harel.shein@gmail.com) +
+
2023-07-17 08:38:59
 +
+

*Thread Reply:* Tagging @Natalie Zeller in case you want to collaborate

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Natalie Zeller + (natalie.zeller@naturalint.com) +
+
2023-07-17 08:47:34
 +
+

*Thread Reply:* Hi, +We've implemented a transporter that transmits lineage from OpenLineage to OpenMetadata, you can find the github project here. +I've also published a blog post that explains this integration and how to use it. +I'll be happy to help if you have any question

+
+ + + + +
Natural-Intelligence/openLineage-openMetadata-transporter
+
+ Transporter for integrating OpenLineage with OpenMetadata +
+ +
+
Website
+ <https://www.top10.com> +
+ +
+
Language
+ Java +
+ + + + + + + + +
+ + + +
+ 🙌 Robin Fehr +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Robin Fehr + (robin.fehr@acosom.com) +
+
2023-07-17 09:49:30
 +
+

*Thread Reply:* very cool! thanks a lot for responding so quickly

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-07-17 18:23:53
 +
+

🚀 We recently hit the 1000-member mark on here! Thank you for joining the movement to establish an open standard for data lineage across the data ecosystem! Tell your friends 🙂! +💯💯💯💯💯💯💯💯💯💯 +https://bit.ly/lineageslack

+ + + +
+ 🎉 Juan Manuel Cappi, Harel Shein, Paweł Leszczyński, Maciej Obuchowski, Willy Lulciuc, Viraj Parekh +
+ +
+ 💯 Harel Shein, Anirudh Shrinivason, Paweł Leszczyński, Maciej Obuchowski, Willy Lulciuc, Robin Fehr, Viraj Parekh, Ernie Ostic +
+ +
+ 👏 thebruuu +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-07-18 04:58:14
 +
+

Btw, just curious what exactly does the runId correspond to in the OL spark integration? Is it possible to obtain the spark application id from the event too?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-07-18 05:10:31
 +
+

*Thread Reply:* runId is an UUID assigned per spark action (compute trigger within a spark job). A single spark script can result in multiple runs then

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-07-18 05:13:17
 +
+

*Thread Reply:* adding an extra facet with applicationId looks like a good idea to me: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/SparkContext.html#applicationId:String

+
+
spark.apache.org
+ + + +
Spark 3.4.1 ScalaDoc - org.apache.spark.SparkContext
+
+ Spark 3.4.1 ScalaDoc - org.apache.spark.SparkContext +
+ + + + + + +
Original URL: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/SparkContext.html#applicationId:String
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-07-18 23:06:01
 +
+

*Thread Reply:* Got it thanks!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Madhav Kakumani + (madhav.kakumani@6point6.co.uk) +
+
2023-07-18 09:47:47
 +
+

Hi, I have an usecase to integrate queries run in Jupyter notebook using pandas integrate with OpenLineage to get the Lineage in Marquez. Did anyone implemented this before? please let me know. Thanks

+ + + +
+ 🤩 thebruuu +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-07-20 06:48:54
 +
+

*Thread Reply:* I think we don't have pandas support so far. So, if one uses pandas to read local files on disk, then perhaps Openlineage (OL) has little sense to do. There is an old pandas issues in our backlog (over 2 years old) -> https://github.com/OpenLineage/OpenLineage/issues/108

+ +

Surely one can use use python OL client to create manully events and send them to MQZ, which may be less convenient (https://github.com/OpenLineage/OpenLineage/tree/main/client/python)

+ +

Anyway, we would like to know what's you usecase? this would be super helpful in understanding why OL & pandas integration may be useful.

+
+ + + + +
#108 [INTEGRATION] Add support for `pandas.read_sql`
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Madhav Kakumani + (madhav.kakumani@6point6.co.uk) +
+
2023-07-20 06:52:32
 +
+

*Thread Reply:* Thanks Pawel for responding

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-07-19 02:57:57
 +
+

Hi guys, when can we expect the next Openlineage release? Excited for MergeIntoCommand column lineage feature!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-07-19 03:40:20
 +
+

*Thread Reply:* Hi @Anirudh Shrinivason, I am still working on that. It's kind of complex because I want to refactor column level lineage so that it can work with multiple Spark versions and multiple delta jars as merge into implementation for delta differs for different delta releases. I thought it's ready, but this needs some extra work to be done in next days. I am excited about that too!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-07-19 03:54:37
 +
+

*Thread Reply:* Ahh I see... Got it! Is there a tentative timeline for when we can expect this? So sorry haha don't mean to rush you. Just curious to know thats all! 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-07-19 22:06:10
 +
+

*Thread Reply:* Can we author a release sometime soon? Would like to use the CustomEnvironmentFacetBuilder for delta catalog!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-07-20 05:28:43
 +
+

*Thread Reply:* we're pretty close i think with merge into delta which is under review. waiting for it would be nice. anyway, we're 3 weeks after the last release.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-07-20 06:50:56
 +
+

*Thread Reply:* @Anirudh Shrinivason releases are available basically on-demand using our process in GOVERNANCE.md. I recommend watching 1958 and then making a request in #general once it’s been merged. But, as Paweł suggested, we have a scheduled release coming soon, anyway. Thanks for your interest in the fix!

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/GOVERNANCE.md | GOVERNANCE.md>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+
+ + + + +
#1958 [SPARK] fix merge into for delta
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-07-20 11:01:14
 +
+

*Thread Reply:* Ahh I see. Got it. Thanks! @Michael Robinson @Paweł Leszczyński

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-07-21 03:12:22
 +
+

*Thread Reply:* @Anirudh Shrinivason it's merged -> https://github.com/OpenLineage/OpenLineage/pull/1958

+
+ + + + +
#1958 [SPARK] fix merge into for delta
+
+ <p><strong>Problem</strong></p> + +<p>Symptoms: <code>MERGE INTO</code> is not generating column level lineage for delta queries.</p> + +<p>This is connected with some other issues that should be resolved within this PR:</p> + +<p>• column level lineage is not tested for each supported spark version (its tests should be moved to <code>app</code> subproject) +• column level lineage does not support including visitors per Spark version</p> + +<p>Closes: <a href="https://github.com/OpenLineage/OpenLineage/issues/1919">#1919</a></p> + +<p><strong>Solution</strong></p> + +<p>• run column lineage tests on all supported versions, +• implement merge into per spark (per delta version)</p> + +<blockquote> + <p><strong>Note:</strong> All schema changes require discussion. Please <a href="https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue">link the issue</a> for context.</p> +</blockquote> + +<p>☐ Your change modifies the <a href="https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.json">core</a> OpenLineage model +☐ Your change modifies one or more OpenLineage <a href="https://github.com/OpenLineage/OpenLineage/tree/main/spec/facets">facets</a></p> + +<p>If you're contributing a new integration, please specify the scope of the integration and how/where it has been tested (e.g., Apache Spark integration supports <code>S3</code> and <code>GCS</code> filesystem operations, tested with AWS EMR).</p> + +<p><strong>One-line summary:</strong> +<strong>Checklist</strong></p> + +<p>☐ You've <a href="https://github.com/OpenLineage/OpenLineage/blob/main/why-the-dco.md">signed-off</a> your work +☐ Your pull request title follows our <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CONTRIBUTING.md#creating-pull-requests">guidelines</a> +☐ Your changes are accompanied by tests (<em>if relevant</em>) +☐ Your change contains a <a href="https://kurtisnusbaum.medium.com/stacked-diffs-keeping-phabricator-diffs-small-d9964f4dcfa6">small diff</a> and is self-contained +☐ You've updated any relevant documentation (<em>if relevant</em>) +☐ Your comment includes a one-liner for the changelog about the specific purpose of the change (<em>if necessary</em>) +☐ You've versioned the core OpenLineage model or facets according to <a href="https://docs.snowplowanalytics.com/docs/pipeline-components-and-applications/iglu/common-architecture/schemaver">SchemaVer</a> (<em>if relevant</em>) +☐ You've added a <a href="https://github.com/OpenLineage/OpenLineage/tree/main/.github/header_templates.md">header</a> to source files (<em>if relevant</em>)</p> + +<hr /> + +<p>SPDX-License-Identifier: Apache-2.0 <br /> +Copyright 2018-2023 contributors to the OpenLineage project</p> +
+ +
+
Labels
+ documentation, integration/spark +
+ +
+
Comments
+ 2 +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-07-21 04:19:15
 +
+

*Thread Reply:* Awesome thanks so much! @Paweł Leszczyński

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Juan Manuel Cappi + (juancappi@gmail.com) +
+
2023-07-19 06:59:31
 +
+

Hi there, related to my question a few days ago about usage of time travel in iceberg, currently only the alias used (i.e. tag, branch) is captured as part of the dataset identifier for lineage. If the tag is removed, or even worse, if it’s removed and re-created with the same name pointing to a difference snapshotid, the lineage will be capturing an inaccurate history. So, ideally, we’d like to capture the actual snapshotid behind the named reference, as part of the lineage. Anyone else thinking this is a reasonable scenario? => more in 🧵

+ + + +
+ 👀 Paweł Leszczyński, Dongjin Seo +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Juan Manuel Cappi + (juancappi@gmail.com) +
+
2023-07-19 07:14:54
 +
+

*Thread Reply:* One hacky approach would be to update the current dataset identifier to include the snapshot_id, so, for schema.table.tag we would have something like schema.table.tag-snapshot_id. The benefit is that it’s explicit and it doesn’t require a change in the OL schemas. The obvious downside (though not that serious in my opinion) is that impacts readability. Not sure though if there are other non-obvious side-effects.

+ +

Another alternative would be to add a dedicated property. For instance, the job > latestRun schema, the input/output dataset version objects could look like this: +"inputDatasetVersions": [ + { + "datasetVersionId": { + "namespace": "<s3a://warehouse>", + "name": "schema.table.tag", + "snapshot_id": "7056736771450556218", + "version": "1c634e18-e357-347b-b758-4337ac352d6d" + }, + "facets": {} + } +] +And column lineage could look like: +```"columnLineage": [ + { + "name": "somefield", + "inputFields": [ + { + "namespace": "s3a:warehouse", + "dataset": "schema.table.tag", + "snapshotid": "7056736771450556218", + "field": "some_field", + ... + }, + ...

+ +
  ],
+
+ +

...```

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-07-19 08:33:43
 +
+

*Thread Reply:* @Paweł Leszczyński what do you think?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-07-19 08:38:16
 +
+

*Thread Reply:* 1. How does snapshotId differ from version? Could one make OL version property to be a string concat of iceberg-snapshot-id.iceberg-version

+ +
  1. I don't think it's necessary (or don't understand why) to add snapshot-id within column-linegea. Each entry within inputFields of columnLineage is already available within inputs of the OL event related to this run.
    2. +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Juan Manuel Cappi + (juancappi@gmail.com) +
+
2023-07-19 18:43:31
 +
+

*Thread Reply:* Yes, I think follow the idea. The problem with that is the version is tied to the dataset name, i.e. my_namespace.table_A.tag_v1 which stays the same for the source dataset, which is the one being used with time travel. +Suppose the following sequence: +step 1 => +tableA.tagv1 has snapshot id 123-abc +run job: table_A.tag_v1 -> job x -> table_B +the inputDatasetVersions > datasetVersionId > version for table_B points to an object which represents table_A.tag_v1 with snapshot id 123-abc correctly captured within facets > version > datasetVersion

+ +

step 2 => +delete tag_v1, insert some data, create tag_v1 again +now table_A.tag_v1 has snapshot id 456-def +run job again: table_A.tag_v1 -> job x -> table_B +the inputDatasetVersions > datasetVersionId > version for table_B points to the same object which represents table_A.tag_v1 only now snapshot id has been replaced by 456-def within facets > version > datasetVersion which means I don’t have way to know which was the snapshot id used in the step 1

+ +

The “hack” I mentioned above though seems to solve the issue, since a new dataset is captured for each combination, so no information is overwritten/lost, i.e., the datasets referenced in inputDatasetVersions are now named: +table_A.tag_v1-123-abc +table_A.tag_v1-456-def

+ +

As a side effect, the column lineage also gets “fixed”, since the lineage for the step 1 and step 2 job runs, without the “hack” both referenced table_A.tag_v1 as the source of input field, though in each run the snapshot id was different. With the hack, one run references table_A.tag_v1-123-abc and the other one table_A.tag_v1-456-def

+ +

Hope it makes sense. If it helps, I can put together a few json files with the examples I’ve been using to experiment

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-07-20 06:35:22
 +
+

*Thread Reply:* So, my understanding of the problem is that icberg version is not unique. So, if you have version 3, revert to version 2, and then write something again, one ends up again with version 3.

+ +

I would not like to mess with dataset names because on the backend sides like Marquez, dataset names being the same in different jobs and runs allow creating lineage graph. If dataset names are different, then there is no way to build lineage graph across multiple jobs.

+ +

Adding snapshot_id to datasetVersion is one option to go. My concern here is that this is so iceberg specific while we're aiming to have a general solution to dataset versioning.

+ +

Some other options are: send concat of version+snapshotId as a version or send only snapshot_id as a version. The second ain't that bad as actually snapashotId is something we're aiming to get as a version, isn't it?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-07-21 04:21:26
 +
+

Hi guys, I’d like to open a vote to release the next OpenLineage version! We'd really like to use the fixed CustomEnvironmentFacetBuilder for delta catalogs, and column lineage for Merge Into command in the spark integration! Thanks! 🙂

+ + + +
+ ➕ Jakub Dardziński, Willy Lulciuc, Michael Robinson, Maciej Obuchowski, Anirudh Shrinivason +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-07-21 13:09:39
 +
+

*Thread Reply:* Thanks, all. The release is authorized and will be initiated within two business days per our policy here.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-07-25 13:44:47
 +
+

*Thread Reply:* @Anirudh Shrinivason and others waiting on this release: the release process isn’t working as expected due to security improvements recently made to the website, ironically enough, which is the source for the spec. But we’re working on a fix and hope to complete the release soon.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-07-25 15:19:49
 +
+

*Thread Reply:* @Anirudh Shrinivason the release (0.30.1) is out now. Thanks for your patience 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-07-25 23:21:14
 +
+

*Thread Reply:* Hi @Michael Robinson Thanks a lot!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-07-26 08:52:24
 +
+

*Thread Reply:* 👍

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Madhav Kakumani + (madhav.kakumani@6point6.co.uk) +
+
2023-07-21 06:38:16
 +
+

Hi, I am running a job in Marquez with 180 rows of metadata but it is running for more than an hour. Is there a way to check the log on Marquez? Below is the screenshot of the job:

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2023-07-21 08:10:58
 +
+

*Thread Reply:* > I am running a job in Marquez with 180 rows of metadata +Do you mean that you have +100 rows of metadata in the jobs table for Marquez? Or that the job never finishes?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2023-07-21 08:11:47
 +
+

*Thread Reply:* Also, yes, we have an even viewer that allows you to query the raw OL events

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2023-07-21 08:12:19
 +
+

*Thread Reply:* If you post a sample of your events, it’d be helpful to troubleshoot your issue

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Madhav Kakumani + (madhav.kakumani@6point6.co.uk) +
+
2023-07-21 08:53:25
 +
+

*Thread Reply:*

+ +
+
Untitled
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Madhav Kakumani + (madhav.kakumani@6point6.co.uk) +
+
2023-07-21 08:53:31
 +
+

*Thread Reply:* Sure Willy thanks for your response. The job is still running. This is the code I am running from jupyter notebook using Python client:

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Madhav Kakumani + (madhav.kakumani@6point6.co.uk) +
+
2023-07-21 08:54:33
 +
+

*Thread Reply:* as you can see my input and output datasets are just 1 row

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Madhav Kakumani + (madhav.kakumani@6point6.co.uk) +
+
2023-07-21 08:55:02
 +
+

*Thread Reply:* included column lineage but job keeps running so I don't know if it is working

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Madhav Kakumani + (madhav.kakumani@6point6.co.uk) +
+
2023-07-21 06:38:49
 +
+

Please ignore 'UPDATED AT' timestamp

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Madhav Kakumani + (madhav.kakumani@6point6.co.uk) +
+
2023-07-21 07:56:48
 +
+

@Paweł Leszczyński there is lot of interest in our organisation to implement Openlineage in several project and we might take the spark route so on that note a small question: Does open lineage works from extracting data from the Catalyst optimiser's Physical/Logical plans etc?

+ + + +
+ 👍 Paweł Leszczyński +
+ +
+ ❤️ Willy Lulciuc, Paweł Leszczyński, Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-07-21 08:20:33
 +
+

*Thread Reply:* spark integration is based on extracting lineage from optimized plans

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-07-21 08:25:35
 +
+

*Thread Reply:* https://youtu.be/rO3BPqUtWrI?t=1326 i recommend whole presentation but in case you're just interested in Spark integration, there few mins that explain how this is achieved (link points to 22:06 min of video)

+
+
YouTube
+ +
+ + + } + + Databricks + (https://www.youtube.com/@Databricks) +
+ + + +
Cross-Platform Data Lineage with OpenLineage
+
+ +
+ + + + + + + + +
Original URL: https://youtu.be/rO3BPqUtWrI?t=1326
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Madhav Kakumani + (madhav.kakumani@6point6.co.uk) +
+
2023-07-21 08:43:47
 +
+

*Thread Reply:* Thanks Pawel for sharing. I will take a look. Have a nice weekend.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jens Pfau + (jenspfau@google.com) +
+
2023-07-21 08:22:51
 +
+

Hello everyone!

+ + + +
+ 👋 Jakub Dardziński, Maciej Obuchowski, Willy Lulciuc, Michael Robinson, Harel Shein, Ross Turk, Robin Fehr, Julien Le Dem +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-07-21 09:57:51
 +
+

*Thread Reply:* Welcome, @Jens Pfau!

+ + + +
+ 😀 Jens Pfau +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
George Polychronopoulos + (george.polychronopoulos@6point6.co.uk) +
+
2023-07-23 08:36:38
 +
+

hello everyone! I am trying to follow your guide +https://openlineage.io/docs/integrations/spark/quickstart_local +and when i execute +spark.createDataFrame([ + {'a': 1, 'b': 2}, + {'a': 3, 'b': 4} +]).write.mode("overwrite").saveAsTable("temp1")

+ +

i not getting the expected result

+
+
openlineage.io
+ + + +
Quickstart with Jupyter | OpenLineage Docs
+
+ Trying out the Spark integration is super easy if you already have Docker Desktop and git installed. +
+ + + + + + +
Original URL: https://openlineage.io/docs/integrations/spark/quickstart_local
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
George Polychronopoulos + (george.polychronopoulos@6point6.co.uk) +
+
2023-07-23 08:37:55
 +
+

``23/07/23 12:35:20 INFO OpenLineageRunEventBuilder: Visiting query plan Optional[== Parsed Logical Plan == +'CreateTabletemp1`, Overwrite ++- LogicalRDD [a#6L, b#7L], false

+ +

== Analyzed Logical Plan ==

+ +

CreateDataSourceTableAsSelectCommand temp1, Overwrite, [a, b] ++- LogicalRDD [a#6L, b#7L], false

+ +

== Optimized Logical Plan == +CreateDataSourceTableAsSelectCommand temp1, Overwrite, [a, b] ++- LogicalRDD [a#6L, b#7L], false

+ +

== Physical Plan == +Execute CreateDataSourceTableAsSelectCommand temp1, Overwrite, [a, b] ++- **(1) Scan ExistingRDD[a#6L,b#7L] +] with input dataset builders [<function1>, <function1>, <function1>, <function1>, <function1>] +23/07/23 12:35:20 INFO OpenLineageRunEventBuilder: Visiting query plan Optional[== Parsed Logical Plan == +'CreateTable temp1, Overwrite ++- LogicalRDD [a#6L, b#7L], false

+ +

== Analyzed Logical Plan ==

+ +

CreateDataSourceTableAsSelectCommand temp1, Overwrite, [a, b] ++- LogicalRDD [a#6L, b#7L], false

+ +

== Optimized Logical Plan == +CreateDataSourceTableAsSelectCommand temp1, Overwrite, [a, b] ++- LogicalRDD [a#6L, b#7L], false

+ +

== Physical Plan == +Execute CreateDataSourceTableAsSelectCommand temp1, Overwrite, [a, b] ++- **(1) Scan ExistingRDD[a#6L,b#7L] +] with output dataset builders [<function1>, <function1>, <function1>, <function1>, <function1>, <function1>, <function1>] +23/07/23 12:35:20 INFO CreateDataSourceTableAsSelectCommandVisitor: Matched io.openlineage.spark.agent.lifecycle.plan.CreateDataSourceTableAsSelectCommandVisitor<org.apache.spark.sql.execution.command.CreateDataSourceTableAsSelectCommand,io.openlineage.client.OpenLineage$OutputDataset> to logical plan CreateDataSourceTableAsSelectCommand temp1, Overwrite, [a, b] ++- LogicalRDD [a#6L, b#7L], false

+ +

23/07/23 12:35:20 INFO CreateDataSourceTableAsSelectCommandVisitor: Matched io.openlineage.spark.agent.lifecycle.plan.CreateDataSourceTableAsSelectCommandVisitor<org.apache.spark.sql.execution.command.CreateDataSourceTableAsSelectCommand,io.openlineage.client.OpenLineage$OutputDataset> to logical plan CreateDataSourceTableAsSelectCommand temp1, Overwrite, [a, b] ++- LogicalRDD [a#6L, b#7L], false

+ +

23/07/23 12:35:20 ERROR EventEmitter: Could not emit lineage w/ exception +io.openlineage.client.OpenLineageClientException: io.openlineage.spark.shaded.org.apache.http.client.ClientProtocolException + at io.openlineage.client.transports.HttpTransport.emit(HttpTransport.java:105) + at io.openlineage.client.OpenLineageClient.emit(OpenLineageClient.java:34) + at io.openlineage.spark.agent.EventEmitter.emit(EventEmitter.java:71) + at io.openlineage.spark.agent.lifecycle.SparkSQLExecutionContext.start(SparkSQLExecutionContext.java:77) + at io.openlineage.spark.agent.OpenLineageSparkListener.lambda$sparkSQLExecStart$0(OpenLineageSparkListener.java:99) + at java.base/java.util.Optional.ifPresent(Optional.java:183) + at io.openlineage.spark.agent.OpenLineageSparkListener.sparkSQLExecStart(OpenLineageSparkListener.java:99) + at io.openlineage.spark.agent.OpenLineageSparkListener.onOtherEvent(OpenLineageSparkListener.java:90) + at org.apache.spark.scheduler.SparkListenerBus.doPostEvent(SparkListenerBus.scala:100) + at org.apache.spark.scheduler.SparkListenerBus.doPostEvent$(SparkListenerBus.scala:28) + at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37) + at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37) + at org.apache.spark.util.ListenerBus.postToAll(ListenerBus.scala:117) + at org.apache.spark.util.ListenerBus.postToAll$(ListenerBus.scala:101) + at org.apache.spark.scheduler.AsyncEventQueue.super$postToAll(AsyncEventQueue.scala:105) + at org.apache.spark.scheduler.AsyncEventQueue.$anonfun$dispatch$1(AsyncEventQueue.scala:105) + at scala.runtime.java8.JFunction0$mcJ$sp.apply(JFunction0$mcJ$sp.java:23) + at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62) + at org.apache.spark.scheduler.AsyncEventQueue.org$apache$spark$scheduler$AsyncEventQueue$$dispatch(AsyncEventQueue.scala:100) + at org.apache.spark.scheduler.AsyncEventQueue$$anon$2.$anonfun$run$1(AsyncEventQueue.scala:96) + at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1381) + at org.apache.spark.scheduler.AsyncEventQueue$$anon$2.run(AsyncEventQueue.scala:96) +Caused by: io.openlineage.spark.shaded.org.apache.http.client.ClientProtocolException + at io.openlineage.spark.shaded.org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:187) + at io.openlineage.spark.shaded.org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) + at io.openlineage.spark.shaded.org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108) + at io.openlineage.client.transports.HttpTransport.emit(HttpTransport.java:100) + ... 21 more +Caused by: io.openlineage.spark.shaded.org.apache.http.ProtocolException: Target host is not specified + at io.openlineage.spark.shaded.org.apache.http.impl.conn.DefaultRoutePlanner.determineRoute(DefaultRoutePlanner.java:71) + at io.openlineage.spark.shaded.org.apache.http.impl.client.InternalHttpClient.determineRoute(InternalHttpClient.java:125) + at io.openlineage.spark.shaded.org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184) + ... 24 more +23/07/23 12:35:20 INFO ParquetFileFormat: Using default output committer for Parquet: org.apache.parquet.hadoop.ParquetOutputCommitter +23/07/23 12:35:20 INFO FileOutputCommitter: File Output Committer Algorithm version is 1 +23/07/23 12:35:20 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false +23/07/23 12:35:20 INFO SQLHadoopMapReduceCommitProtocol: Using user defined output committer class org.apache.parquet.hadoop.ParquetOutputCommitter +23/07/23 12:35:20 INFO FileOutputCommitter: File Output Committer Algorithm version is 1 +23/07/23 12:35:20 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false +23/07/23 12:35:20 INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.parquet.hadoop.ParquetOutputCommitter +23/07/23 12:35:20 INFO CodeGenerator: Code generated in 120.989125 ms +23/07/23 12:35:21 INFO SparkContext: Starting job: saveAsTable at NativeMethodAccessorImpl.java:0 +23/07/23 12:35:21 INFO DAGScheduler: Got job 0 (saveAsTable at NativeMethodAccessorImpl.java:0) with 1 output partitions +23/07/23 12:35:21 INFO DAGScheduler: Final stage: ResultStage 0 (saveAsTable at NativeMethodAccessorImpl.java:0) +23/07/23 12:35:21 INFO DAGScheduler: Parents of final stage: List() +23/07/23 12:35:21 INFO DAGScheduler: Missing parents: List() +23/07/23 12:35:21 INFO OpenLineageRunEventBuilder: Visiting query plan Optional[== Parsed Logical Plan == +'CreateTable temp1, Overwrite ++- LogicalRDD [a#6L, b#7L], false

+ +

== Analyzed Logical Plan ==

+ +

CreateDataSourceTableAsSelectCommand temp1, Overwrite, [a, b] ++- LogicalRDD [a#6L, b#7L], false

+ +

== Optimized Logical Plan == +CreateDataSourceTableAsSelectCommand temp1, Overwrite, [a, b] ++- LogicalRDD [a#6L, b#7L], false

+ +

== Physical Plan == +Execute CreateDataSourceTableAsSelectCommand temp1, Overwrite, [a, b] ++- **(1) Scan ExistingRDD[a#6L,b#7L] +] with input dataset builders [<function1>, <function1>, <function1>, <function1>, <function1>] +23/07/23 12:35:21 INFO OpenLineageRunEventBuilder: Visiting query plan Optional[== Parsed Logical Plan == +'CreateTable temp1, Overwrite ++- LogicalRDD [a#6L, b#7L], false

+ +

== Analyzed Logical Plan ==

+ +

CreateDataSourceTableAsSelectCommand temp1, Overwrite, [a, b] ++- LogicalRDD [a#6L, b#7L], false

+ +

== Optimized Logical Plan == +CreateDataSourceTableAsSelectCommand temp1, Overwrite, [a, b] ++- LogicalRDD [a#6L, b#7L], false

+ +

== Physical Plan == +Execute CreateDataSourceTableAsSelectCommand temp1, Overwrite, [a, b] ++- **(1) Scan ExistingRDD[a#6L,b#7L] +] with output dataset builders [<function1>, <function1>, <function1>, <function1>, <function1>, <function1>, <function1>] +23/07/23 12:35:21 INFO CreateDataSourceTableAsSelectCommandVisitor: Matched io.openlineage.spark.agent.lifecycle.plan.CreateDataSourceTableAsSelectCommandVisitor<org.apache.spark.sql.execution.command.CreateDataSourceTableAsSelectCommand,io.openlineage.client.OpenLineage$OutputDataset> to logical plan CreateDataSourceTableAsSelectCommand temp1, Overwrite, [a, b] ++- LogicalRDD [a#6L, b#7L], false

+ +

23/07/23 12:35:21 INFO CreateDataSourceTableAsSelectCommandVisitor: Matched io.openlineage.spark.agent.lifecycle.plan.CreateDataSourceTableAsSelectCommandVisitor<org.apache.spark.sql.execution.command.CreateDataSourceTableAsSelectCommand,io.openlineage.client.OpenLineage$OutputDataset> to logical plan CreateDataSourceTableAsSelectCommand temp1, Overwrite, [a, b] ++- LogicalRDD [a#6L, b#7L], false

+ +

23/07/23 12:35:21 INFO DAGScheduler: Submitting ResultStage 0 (MapPartitionsRDD[10] at saveAsTable at NativeMethodAccessorImpl.java:0), which has no missing parents +23/07/23 12:35:21 ERROR EventEmitter: Could not emit lineage w/ exception +io.openlineage.client.OpenLineageClientException: io.openlineage.spark.shaded.org.apache.http.client.ClientProtocolException + at io.openlineage.client.transports.HttpTransport.emit(HttpTransport.java:105) + at io.openlineage.client.OpenLineageClient.emit(OpenLineageClient.java:34) + at io.openlineage.spark.agent.EventEmitter.emit(EventEmitter.java:71) + at io.openlineage.spark.agent.lifecycle.SparkSQLExecutionContext.start(SparkSQLExecutionContext.java:174) + at io.openlineage.spark.agent.OpenLineageSparkListener.lambda$onJobStart$9(OpenLineageSparkListener.java:153) + at java.base/java.util.Optional.ifPresent(Optional.java:183) + at io.openlineage.spark.agent.OpenLineageSparkListener.onJobStart(OpenLineageSparkListener.java:149) + at org.apache.spark.scheduler.SparkListenerBus.doPostEvent(SparkListenerBus.scala:37) + at org.apache.spark.scheduler.SparkListenerBus.doPostEvent$(SparkListenerBus.scala:28) + at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37) + at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37) + at org.apache.spark.util.ListenerBus.postToAll(ListenerBus.scala:117) + at org.apache.spark.util.ListenerBus.postToAll$(ListenerBus.scala:101) + at org.apache.spark.scheduler.AsyncEventQueue.super$postToAll(AsyncEventQueue.scala:105) + at org.apache.spark.scheduler.AsyncEventQueue.$anonfun$dispatch$1(AsyncEventQueue.scala:105) + at scala.runtime.java8.JFunction0$mcJ$sp.apply(JFunction0$mcJ$sp.java:23) + at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62) + at org.apache.spark.scheduler.AsyncEventQueue.org$apache$spark$scheduler$AsyncEventQueue$$dispatch(AsyncEventQueue.scala:100) + at org.apache.spark.scheduler.AsyncEventQueue$$anon$2.$anonfun$run$1(AsyncEventQueue.scala:96) + at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1381) + at org.apache.spark.scheduler.AsyncEventQueue$$anon$2.run(AsyncEventQueue.scala:96) +Caused by: io.openlineage.spark.shaded.org.apache.http.client.ClientProtocolException + at io.openlineage.spark.shaded.org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:187) + at io.openlineage.spark.shaded.org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) + at io.openlineage.spark.shaded.org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108) + at io.openlineage.client.transports.HttpTransport.emit(HttpTransport.java:100) + ... 20 more +Caused by: io.openlineage.spark.shaded.org.apache.http.ProtocolException: Target host is not specified + at io.openlineage.spark.shaded.org.apache.http.impl.conn.DefaultRoutePlanner.determineRoute(```

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
George Polychronopoulos + (george.polychronopoulos@6point6.co.uk) +
+
2023-07-23 08:38:46
 +
+

23/07/23 12:35:20 ERROR EventEmitter: Could not emit lineage w/ exception +io.openlineage.client.OpenLineageClientException: io.openlineage.spark.shaded.org.apache.http.client.ClientProtocolException + at io.openlineage.client.transports.HttpTransport.emit(HttpTransport.java:105) + at io.openlineage.client.OpenLineageClient.emit(OpenLineageClient.java:34) + at io.openlineage.spark.agent.EventEmitter.emit(EventEmitter.java:71) + at io.openlineage.spark.agent.lifecycle.SparkSQLExecutionContext.start(SparkSQLExecutionContext.java:77) + at io.openlineage.spark.agent.OpenLineageSparkListener.lambda$sparkSQLExecStart$0(OpenLineageSparkListener.java:99)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-07-23 13:31:53
 +
+

*Thread Reply:* That looks like your URL provided to OpenLineage is missing http:// or https:// in the front

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
George Polychronopoulos + (george.polychronopoulos@6point6.co.uk) +
+
2023-07-23 14:54:55
 +
+

*Thread Reply:* sorry how can i resolve this ? do i need to add this ? i just follow the guide step by step . You dont mention anywhere to add anything. You provide smth that

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
George Polychronopoulos + (george.polychronopoulos@6point6.co.uk) +
+
2023-07-23 14:55:05
 +
+

*Thread Reply:* really does not work out of the box

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
George Polychronopoulos + (george.polychronopoulos@6point6.co.uk) +
+
2023-07-23 14:55:13
 +
+

*Thread Reply:* anbd this is supposed to be a demo

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-07-23 17:07:49
 +
+

*Thread Reply:* bumping e.g. to io.openlineage:openlineage_spark:0.29.2 seems to be fixing the issue

+ +

not sure why it stopped working for 0.12.0 but we’ll take a look and fix accordingly

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-07-24 04:51:34
 +
+

*Thread Reply:* ...probably by bumping the version on this page 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
George Polychronopoulos + (george.polychronopoulos@6point6.co.uk) +
+
2023-07-24 05:00:28
 +
+

*Thread Reply:* thank you both for coming back to me , I bumped to 0.29 and i think that it now runs.Is this the expected output ? +23/07/24 08:43:55 INFO ConsoleTransport: {"eventTime":"2023_07_24T08:43:55.941Z","producer":"<https://github.com/OpenLineage/OpenLineage/tree/0.29.2/integration/spark>","schemaURL":"<https://openlineage.io/spec/2-0-0/OpenLineage.json#/$defs/RunEvent>","eventType":"COMPLETE","run":{"runId":"186c06c0_e79c_43cf_8bb7_08e1ab4c86a5","facets":{"spark.logicalPlan":{"_producer":"<https://github.com/OpenLineage/OpenLineage/tree/0.29.2/integration/spark>","_schemaURL":"<https://openlineage.io/spec/2-0-0/OpenLineage.json#/$defs/RunFacet>","plan":[{"class":"org.apache.spark.sql.execution.command.CreateDataSourceTableAsSelectCommand","num-children":1,"table":{"product-class":"org.apache.spark.sql.catalyst.catalog.CatalogTable","identifier":{"product-class":"org.apache.spark.sql.catalyst.TableIdentifier","table":"temp2","database":"default"},"tableType":{"product-class":"org.apache.spark.sql.catalyst.catalog.CatalogTableType","name":"MANAGED"},"storage":{"product_class":"org.apache.spark.sql.catalyst.catalog.CatalogStorageFormat","compressed":false,"properties":null},"schema":{"type":"struct","fields":[]},"provider":"parquet","partitionColumnNames":[],"owner":"","createTime":1690188235517,"lastAccessTime":-1,"createVersion":"","properties":null,"unsupportedFeatures":[],"tracksPartitionsInCatalog":false,"schemaPreservesCase":true,"ignoredProperties":null},"mode":null,"query":0,"outputColumnNames":"[a, b]"},{"class":"org.apache.spark.sql.execution.LogicalRDD","num_children":0,"output":[[{"class":"org.apache.spark.sql.catalyst.expressions.AttributeReference","num-children":0,"name":"a","dataType":"long","nullable":true,"metadata":{},"exprId":{"product-class":"org.apache.spark.sql.catalyst.expressions.ExprId","id":12,"jvmId":"173725f4_02c4_4174_9d18_3a61aa311d62"},"qualifier":[]}],[{"class":"org.apache.spark.sql.catalyst.expressions.AttributeReference","num_children":0,"name":"b","dataType":"long","nullable":true,"metadata":{},"exprId":{"product_class":"org.apache.spark.sql.catalyst.expressions.ExprId","id":13,"jvmId":"173725f4-02c4-4174-9d18-3a61aa311d62"},"qualifier":[]}]],"rdd":null,"outputPartitioning":{"product_class":"org.apache.spark.sql.catalyst.plans.physical.UnknownPartitioning","numPartitions":0},"outputOrdering":[],"isStreaming":false,"session":null}]},"spark_version":{"_producer":"<https://github.com/OpenLineage/OpenLineage/tree/0.29.2/integration/spark>","_schemaURL":"<https://openlineage.io/spec/2-0-0/OpenLineage.json#/$defs/RunFacet>","spark-version":"3.1.2","openlineage_spark_version":"0.29.2"}}},"job":{"namespace":"default","name":"sample_spark.execute_create_data_source_table_as_select_command","facets":{}},"inputs":[],"outputs":[{"namespace":"file","name":"/home/jovyan/spark-warehouse/temp2","facets":{"dataSource":{"_producer":"<https://github.com/OpenLineage/OpenLineage/tree/0.29.2/integration/spark>","_schemaURL":"<https://openlineage.io/spec/facets/1-0-0/DatasourceDatasetFacet.json#/$defs/DatasourceDatasetFacet>","name":"file","uri":"file"},"schema":{"_producer":"<https://github.com/OpenLineage/OpenLineage/tree/0.29.2/integration/spark>","_schemaURL":"<https://openlineage.io/spec/facets/1-0-0/SchemaDatasetFacet.json#/$defs/SchemaDatasetFacet>","fields":[{"name":"a","type":"long"},{"name":"b","type":"long"}]},"symlinks":{"_producer":"<https://github.com/OpenLineage/OpenLineage/tree/0.29.2/integration/spark>","_schemaURL":"<https://openlineage.io/spec/facets/1-0-0/SymlinksDatasetFacet.json#/$defs/SymlinksDatasetFacet>","identifiers":[{"namespace":"/home/jovyan/spark-warehouse","name":"default.temp2","type":"TABLE"}]},"lifecycleStateChange":{"_producer":"<https://github.com/OpenLineage/OpenLineage/tree/0.29.2/integration/spark>","_schemaURL":"<https://openlineage.io/spec/facets/1-0-0/LifecycleStateChangeDatasetFacet.json#/$defs/LifecycleStateChangeDatasetFacet>","lifecycleStateChange":"CREATE"}},"outputFacets":{}}]} +? Also i then proceeded to run +docker run --network spark_default -p 3000:3000 -e MARQUEZ_HOST=marquez-api -e MARQUEZ_PORT=5000 --link marquez-api:marquez-api marquezproject/marquez-web:0.19.1 +but the page is empty

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-07-24 11:11:08
 +
+

*Thread Reply:* You'd need to set up spark.openlineage.transport.url to send OpenLineage events to Marquez

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
George Polychronopoulos + (george.polychronopoulos@6point6.co.uk) +
+
2023-07-24 11:12:28
 +
+

*Thread Reply:* where n how can i do this ?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
George Polychronopoulos + (george.polychronopoulos@6point6.co.uk) +
+
2023-07-24 11:13:04
 +
+

*Thread Reply:* do i need to edit the conf ?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-07-24 11:37:09
 +
+

*Thread Reply:* yes, in the spark conf

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
George Polychronopoulos + (george.polychronopoulos@6point6.co.uk) +
+
2023-07-24 11:37:48
 +
+

*Thread Reply:* what this url should be ?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
George Polychronopoulos + (george.polychronopoulos@6point6.co.uk) +
+
2023-07-24 11:37:51
 +
+

*Thread Reply:* http://localhost:3000/ ?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-07-24 11:43:30
 +
+

*Thread Reply:* That depends how you ran Marquez, but looking at your screenshot UI is at 3000, I guess API would be at 5000

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-07-24 11:43:46
 +
+

*Thread Reply:* as that's default in Marquez docker-compose

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
George Polychronopoulos + (george.polychronopoulos@6point6.co.uk) +
+
2023-07-24 11:44:14
 +
+

*Thread Reply:* i cannot see spark conf

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
George Polychronopoulos + (george.polychronopoulos@6point6.co.uk) +
+
2023-07-24 11:44:23
 +
+

*Thread Reply:* is it in there or do i need to create it ?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-07-24 16:42:53
 +
+

*Thread Reply:* Is something like +```from pyspark.sql import SparkSession

+ +

spark = (SparkSession.builder.master('local') + .appName('samplespark') + .config('spark.extraListeners', 'io.openlineage.spark.agent.OpenLineageSparkListener') + .config('spark.jars.packages', 'io.openlineage:openlineagespark:0.29.2') + .config('spark.openlineage.transport.url', 'http://marquez:5000') + .config('spark.openlineage.transport.type', 'http') + .getOrCreate())``` +not working?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
George Polychronopoulos + (george.polychronopoulos@6point6.co.uk) +
+
2023-07-25 05:08:08
 +
+

*Thread Reply:* OK when i use the snippet you provided and then execute +docker run --network sparkdefault -p 3000:3000 -e MARQUEZHOST=marquez-api -e MARQUEZ_PORT=5000 --link marquez-api:marquez-api marquezproject/marquez-web:0.19.1

+ +

I can now see this

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
George Polychronopoulos + (george.polychronopoulos@6point6.co.uk) +
+
2023-07-25 05:08:52
 +
+

*Thread Reply:* but when i click on the job i then get this

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
George Polychronopoulos + (george.polychronopoulos@6point6.co.uk) +
+
2023-07-25 05:09:05
 +
+

*Thread Reply:* so i cannot see any details of the job

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sarwat Fatima + (sarwatfatimam@gmail.com) +
+
2023-09-05 05:54:50
 +
+

*Thread Reply:* @George Polychronopoulos Hi, I am facing the same issue. After adding spark conf and using the docker run command, marquez is still showing empty. Do I need to change something in the run command?

+ +
+
image.png
+ + + + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
George Polychronopoulos + (george.polychronopoulos@6point6.co.uk) +
+
2023-09-05 05:55:15
 +
+

*Thread Reply:* yes i will tell you

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sarwat Fatima + (sarwatfatimam@gmail.com) +
+
2023-09-05 07:36:41
 +
+

*Thread Reply:* For the docker command that I used, I updated the marquez-web version to 0.40.0 and I also updated the Marquez_host which I am not sure if I have to or not. The UI is running but not showing anything docker run --network spark_default -p 3000:3000 -e MARQUEZ_HOST=localhost -e MARQUEZ_PORT=5000 --link marquez-api:marquez-api marquez/marquez-web:0.40.0

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
George Polychronopoulos + (george.polychronopoulos@6point6.co.uk) +
+
2023-09-05 07:36:52
 +
+

*Thread Reply:* is because you are running this command right

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
George Polychronopoulos + (george.polychronopoulos@6point6.co.uk) +
+
2023-09-05 07:36:55
 +
+

*Thread Reply:* yes thats it

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
George Polychronopoulos + (george.polychronopoulos@6point6.co.uk) +
+
2023-09-05 07:36:58
 +
+

*Thread Reply:* you need 0.40

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
George Polychronopoulos + (george.polychronopoulos@6point6.co.uk) +
+
2023-09-05 07:37:03
 +
+

*Thread Reply:* and there is a lot of stuff

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
George Polychronopoulos + (george.polychronopoulos@6point6.co.uk) +
+
2023-09-05 07:37:07
 +
+

*Thread Reply:* you need rto chwange

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
George Polychronopoulos + (george.polychronopoulos@6point6.co.uk) +
+
2023-09-05 07:37:10
 +
+

*Thread Reply:* in the Docker

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
George Polychronopoulos + (george.polychronopoulos@6point6.co.uk) +
+
2023-09-05 07:37:24
 +
+

*Thread Reply:* so the spark

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
George Polychronopoulos + (george.polychronopoulos@6point6.co.uk) +
+
2023-09-05 07:37:25
 +
+

*Thread Reply:* version

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
George Polychronopoulos + (george.polychronopoulos@6point6.co.uk) +
+
2023-09-05 07:37:27
 +
+

*Thread Reply:* the python

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
George Polychronopoulos + (george.polychronopoulos@6point6.co.uk) +
+
2023-09-05 07:38:05
 +
+

*Thread Reply:* version: "3.10" +services: + notebook: + image: jupyter/pyspark-notebook:spark-3.4.1 + ports: + - "8888:8888" + volumes: + - ./docker/notebooks:/home/jovyan/notebooks + - ./build:/home/jovyan/openlineage + links: + - "api:marquez" + depends_on: + - api

+ +

Marquez as an OpenLineage Client

+ +

api: + image: marquezproject/marquez + containername: marquez-api + ports: + - "5000:5000" + - "5001:5001" + volumes: + - ./docker/wait-for-it.sh:/usr/src/app/wait-for-it.sh + links: + - "db:postgres" + dependson: + - db + entrypoint: [ "./wait-for-it.sh", "db:5432", "--", "./entrypoint.sh" ]

+ +

db: + image: postgres:12.1 + containername: marquez-db + ports: + - "5432:5432" + environment: + - POSTGRESUSER=postgres + - POSTGRESPASSWORD=password + - MARQUEZDB=marquez + - MARQUEZUSER=marquez + - MARQUEZPASSWORD=marquez + volumes: + - ./docker/init-db.sh:/docker-entrypoint-initdb.d/init-db.sh + # Enables SQL statement logging (see: https://www.postgresql.org/docs/12/runtime-config-logging.html#GUC-LOG-STATEMENT) + # command: ["postgres", "-c", "log_statement=all"]

+
+
PostgreSQL Documentation
+ + + +
19.8. Error Reporting and Logging
+
+ 19.8.&amp;nbsp;Error Reporting and Logging 19.8.1. Where to Log 19.8.2. When to Log 19.8.3. What to Log 19.8.4. Using CSV-Format Log … +
+ + + + + + + + +
Original URL: https://www.postgresql.org/docs/12/runtime-config-logging.html#GUC-LOG-STATEMENT
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
George Polychronopoulos + (george.polychronopoulos@6point6.co.uk) +
+
2023-09-05 07:38:10
 +
+

*Thread Reply:* this is hopw mine looks

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
George Polychronopoulos + (george.polychronopoulos@6point6.co.uk) +
+
2023-09-05 07:38:20
 +
+

*Thread Reply:* it is all tested and letest version

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
George Polychronopoulos + (george.polychronopoulos@6point6.co.uk) +
+
2023-09-05 07:38:31
 +
+

*Thread Reply:* postgres does not work beyond 12

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
George Polychronopoulos + (george.polychronopoulos@6point6.co.uk) +
+
2023-09-05 07:38:56
 +
+

*Thread Reply:* if you run this docker-compose up

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
George Polychronopoulos + (george.polychronopoulos@6point6.co.uk) +
+
2023-09-05 07:38:58
 +
+

*Thread Reply:* the notebooks

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
George Polychronopoulos + (george.polychronopoulos@6point6.co.uk) +
+
2023-09-05 07:39:02
 +
+

*Thread Reply:* are 10 faster

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
George Polychronopoulos + (george.polychronopoulos@6point6.co.uk) +
+
2023-09-05 07:39:06
 +
+

*Thread Reply:* and give no errors

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
George Polychronopoulos + (george.polychronopoulos@6point6.co.uk) +
+
2023-09-05 07:39:14
 +
+

*Thread Reply:* also you need to update other stuff

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
George Polychronopoulos + (george.polychronopoulos@6point6.co.uk) +
+
2023-09-05 07:39:18
 +
+

*Thread Reply:* such as

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
George Polychronopoulos + (george.polychronopoulos@6point6.co.uk) +
+
2023-09-05 07:39:26
 +
+

*Thread Reply:* dont run what is in the docs

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
George Polychronopoulos + (george.polychronopoulos@6point6.co.uk) +
+
2023-09-05 07:39:34
 +
+

*Thread Reply:* but run what is in github

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
George Polychronopoulos + (george.polychronopoulos@6point6.co.uk) +
+
2023-09-05 07:40:13
 +
+

*Thread Reply:* https://github.com/OpenLineage/OpenLineage/tree/main/integration/spark

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
George Polychronopoulos + (george.polychronopoulos@6point6.co.uk) +
+
2023-09-05 07:40:22
 +
+

*Thread Reply:* run in your notebooks what is in here

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
George Polychronopoulos + (george.polychronopoulos@6point6.co.uk) +
+
2023-09-05 07:40:32
 +
+

*Thread Reply:* ```from pyspark.sql import SparkSession

+ +

spark = (SparkSession.builder.master('local') + .appName('samplespark') + .config('spark.jars.packages', 'io.openlineage:openlineagespark:1.1.0') + .config('spark.extraListeners', 'io.openlineage.spark.agent.OpenLineageSparkListener') + .config('spark.openlineage.transport.url', 'http://{openlineage.client.host}/api/v1/namespaces/spark_integration/') + .getOrCreate())```

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
George Polychronopoulos + (george.polychronopoulos@6point6.co.uk) +
+
2023-09-05 07:40:38
 +
+

*Thread Reply:* the dont update documentation

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
George Polychronopoulos + (george.polychronopoulos@6point6.co.uk) +
+
2023-09-05 07:40:44
 +
+

*Thread Reply:* it took me 4 weeks to get here

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
George Polychronopoulos + (george.polychronopoulos@6point6.co.uk) +
+
2023-07-23 08:39:13
 +
+

is this a known error ? does anyone know how to debug this ?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Steven + (xli@zjuici.com) +
+
2023-07-23 23:57:43
 +
+

Hi, +Using Marquez. I tried to get the dataset version through two apis. +First: +http://host/api/v1/namespaces/{namespace}/datasets/{dataset} +It will include a currentVersion in the response. +Then: +http://host/api/v1/namespaces/{namespace}/datasets/{dataset}/versions/{currentVersion} +But the version used here refers to the "version" column in table dataset_versions. Not the primary key "uuid". Which leads to 404 not found. +I checked other apis but seemed that there are no other way to get the version through "currentVersion". +Any help?

+ + + +
+ 👀 Maciej Obuchowski, Willy Lulciuc +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Steven + (xli@zjuici.com) +
+
2023-07-24 00:14:43
 +
+

*Thread Reply:* Like I want to change the facets of a specific dataset.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-07-24 16:45:18
 +
+

*Thread Reply:* @Willy Lulciuc do you have any idea? 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Steven + (xli@zjuici.com) +
+
2023-07-25 05:02:47
 +
+

*Thread Reply:* I solved this by adding a new job which outputs to the same dataset. This ended up in a newer dataset version.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2023-07-25 06:20:58
 +
+

*Thread Reply:* @Steven great to hear that you solved the issue! but there are some minor logical inconsistencies that we’d like to address with versioning (for both datasets and jobs) in Marquez. The tl;dr is the version column wasn’t meant to be used externally, but internally within Marquez. The issue is “minor” as it’s more of a pointer thing. We’ll be addressing soon. For some background, you can look at: +• https://github.com/MarquezProject/marquez/issues/2071 +• https://github.com/MarquezProject/marquez/pull/2153

+
+ + + + +
#2071 Proposal: Update DatasetVersion versioning
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/MarquezProject/marquez">MarquezProject/marquez</a> +
+ + +
+
+ + + + +
#2153 Proposal/2071 update version versioning
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/MarquezProject/marquez">MarquezProject/marquez</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Steven + (xli@zjuici.com) +
+
2023-07-25 05:06:48
 +
+

Hi, +Are there any keys to set in marquez.yaml to skip db initialization and use existing db? I am deploying the marquez client on k8s client, which uses a cloud postgres. Every time I restart the marquez deployment I have to drop all those tables otherwise it will raise table already exists ERROR

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2023-07-25 06:43:32
 +
+

*Thread Reply:* @Steven ahh very good point, it’s technically not “error” in the true sense, but annoying nonetheless. I think you’re referencing the init container in the Marquez helm chart? https://github.com/MarquezProject/marquez/blob/main/chart/templates/marquez/deployment.yaml#L37

+
+ + + + +
<https://github.com/MarquezProject/marquez/blob/main/chart/templates/marquez/deployment.yaml | deployment.yaml>
+
+ <pre><code> - until pg_isready -h ${POSTGRES_HOST} -p ${POSTGRES_PORT} -U ${POSTGRES_USER}; do echo waiting for database; sleep 2; done; +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/MarquezProject/marquez">MarquezProject/marquez</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2023-07-25 06:45:24
 +
+

*Thread Reply:* hmm, actually what raises the error you’re referencing? the Maruez http server?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2023-07-25 06:49:08
 +
+

*Thread Reply:* > Every time I restart the marquez deployment I have to drop all those tables otherwise it will raise table already exists ERROR +This shouldn’t be an error. I’m trying to understand the scenario in which this error is thrown (any info is helpful). We use flyway to manage our db schema, but you may have gotten in an odd state somehow

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sheeri Cabral (Collibra) + (sheeri.cabral@collibra.com) +
+
2023-07-25 12:52:51
 +
+

For Databricks notebooks, does the Spark listener work without any notebook changes? (I see that Azure Databricks -> purview needs no changes, but I’m not sure if that applies to anywhere….e.g. if I have an existing databricks notebook, and I add a spark listener, can I get column-level lineage? or do I need to change my notebook to use openlineage libraries, like I do with an arbitrary Python script?)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-07-31 03:35:58
 +
+

*Thread Reply:* Nope, one should modify the cluster as per doc <https://openlineage.io/docs/integrations/spark/quickstart_databricks> but no changes in notebook are required.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sheeri Cabral (Collibra) + (sheeri.cabral@collibra.com) +
+
2023-08-02 10:59:00
 +
+

*Thread Reply:* Right, great, that’s exactly what I was hoping 😄

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-07-25 15:24:17
 +
+

@channel +We released OpenLineage 0.30.1, including: +Added +• Flink: support Iceberg sinks #1960 @pawel-big-lebowski +• Spark: column-level lineage for merge into on delta tables #1958 @pawel-big-lebowski +• Spark: column-level lineage for merge into on Iceberg tables #1971 @pawel-big-lebowski +• Spark: add supprt for Iceberg REST catalog #1963 @juancappi +• Airflow: add possibility to force direct-execution based on environment variable #1934 @mobuchowski +• SQL: add support for Apple Silicon to openlineage-sql-java #1981 @davidjgoss +• Spec: add facet deletion #1975 @julienledem +• Client: add a file transport #1891 @alexandre bergere +Changed +• Airflow: do not run plugin if OpenLineage provider is installed #1999 @JDarDagran +• Python: rename config to config_class #1998 @mobuchowski +Plus test improvements, docs changes, bug fixes and more. +Thanks to all the contributors, including new contributors @davidjgoss, @alexandre bergere and @Juan Manuel Cappi! +Release: https://github.com/OpenLineage/OpenLineage/releases/tag/0.30.1 +Changelog: https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md +Commit history: https://github.com/OpenLineage/OpenLineage/compare/0.29.2...0.30.1 +Maven: https://oss.sonatype.org/#nexus-search;quick~openlineage +PyPI: https://pypi.org/project/openlineage-python/

+ + + +
+ 👏 Julian Rossi, Bernat Gabor, Anirudh Shrinivason, Maciej Obuchowski, Jens Pfau, Sheeri Cabral (Collibra) +
+ +
+ 👍 Athitya Kumar, Sheeri Cabral (Collibra) +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Codrut Stoicescu + (codrut.stoicescu@gmail.com) +
+
2023-07-27 11:53:09
 +
+

Hello everyone! I’m part of a team trying to integrate OpenLineage and Marquez with multiple tools in our ecosystem. Integration with Spark and Iceberg was fairly easy with the listener you guys developed. We are now trying to integrate with Ray and we are having some trouble there. I was wondering if anybody has tried any work in that direction, so we can chat and exchange ideas. Thank you!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-07-27 14:47:18
 +
+

*Thread Reply:* This is the first I’ve heard of someone trying to do this, but others have tried getting lineage from pandas. There isn’t support for this currently, but this thread contains a link to an issue that might be helpful: https://openlineage.slack.com/archives/C01CK9T7HKR/p1689850134978429?thread_ts=1689688067.729469&cid=C01CK9T7HKR.

+
+ + +
+ + + } + + Paweł Leszczyński + (https://openlineage.slack.com/team/U02MK6YNAQ5) +
+ + + +
+
+ <p>I think we don't have pandas support so far. So, if one uses pandas to read local files on disk, then perhaps Openlineage (OL) has little sense to do. There is an old pandas issues in our backlog (over 2 years old) -&gt; <a href="https://github.com/OpenLineage/OpenLineage/issues/108">https://github.com/OpenLineage/OpenLineage/issues/108</a></p> + +<p>Surely one can use use python OL client to create manully events and send them to MQZ, which may be less convenient (<a href="https://github.com/OpenLineage/OpenLineage/tree/main/client/python">https://github.com/OpenLineage/OpenLineage/tree/main/client/python</a>)</p> + +<p>Anyway, we would like to know what's you usecase? this would be super helpful in understanding why OL &amp; pandas integration may be useful.</p> +
+ + + + + + +
Original URL: https://openlineage.slack.com/archives/C01CK9T7HKR/p1689850134978429?thread_ts=1689688067.729469&amp;cid=C01CK9T7HKR
+ + +
+ + Thread in Slack Conversation +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Codrut Stoicescu + (codrut.stoicescu@gmail.com) +
+
2023-07-28 02:10:14
 +
+

*Thread Reply:* Thank you for your response. We have implemented the “manual way” of emitting events with python OL client. We are now looking for a more automated way, so that updates to the scripts that run in Ray are minimal to none

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-07-28 13:03:43
 +
+

*Thread Reply:* If you're actively using Ray, then you know way more about it than me, or probably any other OL contributor 🙂 +I don't know how it works or is deployed, but I would recommend checking if there's robust way of being notified in the runtime about processing occuring there.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Codrut Stoicescu + (codrut.stoicescu@gmail.com) +
+
2023-07-31 12:17:07
 +
+

*Thread Reply:* Thank you for the tip. That’s the kind of details I’m looking for, but couldn’t find yet

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tereza Trojanová + (tereza.trojanova@revolt.bi) +
+
2023-07-28 09:20:34
 +
+

Hi, does anyone have experience integrating OpenLineage and Marquez with Keboola? I am new to OpenLineage and struggling with the KBC component configuration.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-07-28 10:53:35
 +
+

*Thread Reply:* @Martin Fiser can you share any resources or pointers that might be helpful?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Martin Fiser + (fisa@keboola.com) +
+
2023-08-21 19:17:17
 +
+

*Thread Reply:* Hi, apologies - vacation period has hit m. However here are the resources:

+ +

API endpoint: +https://app.swaggerhub.com/apis-docs/keboola/job-queue-api/1.3.4#/Jobs/getJobOpenApiLineage|job-queue-api | 1.3.4 | keboola | SwaggerHub +Dedicated component to push data into openlineage(Marquez instance): +https://components.keboola.com/components/keboola.wr-openlineage|OpenLineage data destination | Keboola Developer Portal

+ + + +
+ 🙌 Michael Robinson +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Damien Hawes + (damien.hawes@booking.com) +
+
2023-07-31 12:32:22
 +
+

Hi folks. I'm looking to find the complete spec in openapi format. For example, if I want to find the complete spec of 1.0.5 , where would I find that? I've looked here: https://openlineage.io/apidocs/openapi/ however when I download the spec, things are missing, specifically the facets. This makes it difficult to generate clients / backend interfaces from the (limited) openapi spec.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Silvia Pina + (silviampina@gmail.com) +
+
2023-08-01 05:14:58
 +
+

*Thread Reply:* +1, I could also really use this!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Silvia Pina + (silviampina@gmail.com) +
+
2023-08-01 05:27:34
 +
+

*Thread Reply:* Found a way: you download it as json in the above link (“Download OpenAPI specification”), then if you copy paste it to editor.swagger.io it asks f you want to convert to yaml :)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Damien Hawes + (damien.hawes@booking.com) +
+
2023-08-01 10:25:49
 +
+

*Thread Reply:* Whilst that works, it isn't complete. The issue is that the "facets" are not resolved. Exploring the website repository (https://github.com/OpenLineage/website/tree/main/static/spec) shows that facets aren't published alongside the spec, beyond 1.0.1 - which means its hard to know which revisions of the facets belong to which version of the spec.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Silvia Pina + (silviampina@gmail.com) +
+
2023-08-01 10:26:54
 +
+

*Thread Reply:* Good point! Would be good if we could clarify how to get the full spec, in that case

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Damien Hawes + (damien.hawes@booking.com) +
+
2023-08-01 10:30:57
 +
+

*Thread Reply:* Granted. If the spec follows backwards compatible evolution rules, then this shouldn't be a problem, i.e., new fields must be optional, you can not remove existing fields, you can not modify existing fields, etc.

+ + + +
+ 🙌 Silvia Pina +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-01 12:15:22
 +
+

*Thread Reply:* We don't have facets with newer version than 1.1.0

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-01 12:15:56
 +
+

*Thread Reply:* @Damien Hawes we've moved to merge docs and website repos here: https://github.com/OpenLineage/docs

+
+ + + + +
OpenLineage/docs
+
+ Documentation for OpenLineage +
+ +
+
Website
+ <https://openlineage.io/docs> +
+ +
+
Stars
+ 5 +
+ + + + + + + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-01 12:18:23
 +
+

*Thread Reply:* > Would be good if we could clarify how to get the full spec, in that case +Is using https://github.com/OpenLineage/OpenLineage/tree/main/spec not enough? We have separate files with facets definition to be able to evolve them separetely from main spec

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Damien Hawes + (damien.hawes@booking.com) +
+
2023-08-02 04:53:03
 +
+

*Thread Reply:* @Maciej Obuchowski - thanks for your input. I understand the desire to want to evolve the facets independently from the main spec, yet I keep running into a mental wall.

+ +

If I say, 'My application is compatible with OpenLineage 1.0.5' - what does that mean exactly? Does it mean that I am at least compatible with the base definition of RunEvent and its nested components, but not facets?

+ +

That's what I'm finding difficult to wrap my head around. Right now, I can not define (for my own sake and the sake of my org) what 'OpenLineage 1.0.5' means.

+ +

When I read the Marquez source code, I see that they state they implement 1.0.5, but again, it isn't clear what that completely entails.

+ +

I hope I am making sense.

+ + + +
+ 👍 Silvia Pina +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Damien Hawes + (damien.hawes@booking.com) +
+
2023-08-02 04:56:36
 +
+

*Thread Reply:* If I approach this from a conventional software engineering standpoint, where I provide a library to my consumers. The library has a version associated with it, and that version encompasses all the objects located within that particular library. If I release a new version of my library, it implies that some form of evolution has happened. Whether it is a bug fix, a documentation change, or evolving the API of my objects it means something has changed and the new version is there to indicate that.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-02 04:56:53
 +
+

*Thread Reply:* Yes - it means you can read and understand base spec. Facets are completely optional - reading them might provide you additional information, but you as a event consumer need to define what you do with them. Basically, the needs can be very different between consumers, spec should not define behavior of a consumer.

+ + + +
+ 🙌 Silvia Pina +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Damien Hawes + (damien.hawes@booking.com) +
+
2023-08-02 05:01:26
 +
+

*Thread Reply:* OK. Thanks for the clarification. That clears things up for me.

+ + + +
+ 👍 Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-07-31 16:42:48
 +
+

This month’s issue of OpenLineage News was just sent out. Please to get it directly in your inbox each month!

+
+
apache.us14.list-manage.com
+ + + +
OpenLineage Project
+
+ OpenLineage Project Email Forms +
+ + + + + + +
Original URL: https://openlineage.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&amp;id=785a308136&amp;e=ce16eef4ef
+ + + +
+ +
+
+ + + + + + +
+ + +
+ 👍 Ross Turk, Maciej Obuchowski, Shirley Lu +
+ +
+ 🎉 Harel Shein +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-01 12:35:22
 +
+

Hello, I request OpenLineage release, especially for two things: +• Snowflake/HTTP/Airflow bugfix: https://github.com/OpenLineage/OpenLineage/pull/2025 +• Spec: removing refs from core: https://github.com/OpenLineage/OpenLineage/pull/1997 +Three approvals from committers will authorize release. @Michael Robinson

+ + + +
+ ➕ Jakub Dardziński, Harel Shein, Michael Robinson, George Polychronopoulos, Willy Lulciuc, Shirley Lu +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-08-01 13:26:30
 +
+

*Thread Reply:* Thanks, @Maciej Obuchowski

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-08-01 15:43:00
 +
+

*Thread Reply:* Thanks, all. The release is authorized and will be initiated within two business days.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-08-01 16:42:32
 +
+

@channel +We released OpenLineage 1.0.0, featuring static lineage capability! +Added: +• Airflow: convert lineage from legacy File definition #2006 @Maciej Obuchowski +Removed: +• Spec: remove facet ref from core #1997 @JDarDagran +Changed +• Airflow: change log level to DEBUG when extractor isn’t found #2012 @kaxil +• Airflow: make sure we cannot fail in thread despite direct execution #2010 @Maciej Obuchowski +Plus test improvements, docs changes, bug fixes and more. +*See prior releases for additional changes related to static lineage. +Thanks to all the contributors, including new contributors @kaxil and @Mars Lan! +*Release: https://github.com/OpenLineage/OpenLineage/releases/tag/1.0.0 +Changelog: https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md +Commit history: https://github.com/OpenLineage/OpenLineage/compare/0.30.1...1.0.0 +Maven: https://oss.sonatype.org/#nexus-search;quick~openlineage +PyPI: https://pypi.org/project/openlineage-python/

+ + + +
+ 🙌 Julian LaNeve, Bernat Gabor, Maciej Obuchowski, Peter Hicks, Ross Turk, Harel Shein, Willy Lulciuc, Paweł Leszczyński, Peter Hicks +
+ +
+ 🥳 Julian LaNeve, alexandre bergere, Maciej Obuchowski, Peter Hicks, Juan Manuel Cappi, Ross Turk, Harel Shein, Paweł Leszczyński, Peter Hicks +
+ +
+ 🚀 alexandre bergere, Peter Hicks, Ross Turk, Harel Shein, Paweł Leszczyński, Peter Hicks +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Juan Luis Cano Rodríguez + (juan_luis_cano@mckinsey.com) +
+
2023-08-02 08:51:57
 +
+

hi folks! so happy to see that static lineage is making its way through OL. one question: is the OpenAPI spec up to date? https://openlineage.io/apidocs/openapi/ IIUC, proposal 1837 says that JobEvent and DatasetEvent can be emitted independently from RunEvents now, but it's not clear how this affected the spec.

+ +

I see the Python client https://pypi.org/project/openlineage-python/1.0.0/ includes these changes already, so I assume I can go ahead and use it already? (I'm also keeping tabs on https://github.com/MarquezProject/marquez/issues/2544)

+
+ + + + +
#2544 Proposal for static lineage support
+
+ <p>As of PR <a href="https://github.com/MarquezProject/marquez/pull/2495">#2495</a>, static lineage events are collected, but not stored. Let's write a proposal outlining on how static lineage events will:</p> + +<p>• be validated and stored in the <code>lineage_events</code> table +• change the core model logic; more specifically, the relationship between jobs, job versions, and runs +• how static lineage will be represented and queried via REST API</p> +
+ +
+
Assignees
+ <a href="https://github.com/wslulciuc">@wslulciuc</a> +
+ + + + + + + +
+ + <a href="https://github.com/MarquezProject/marquez">MarquezProject/marquez</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-02 10:09:33
 +
+

*Thread Reply:* I think the apidocs are not up to date 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-02 10:09:43
 +
+

*Thread Reply:* https://openlineage.io/spec/2-0-2/OpenLineage.json has the newest spec

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Juan Luis Cano Rodríguez + (juan_luis_cano@mckinsey.com) +
+
2023-08-02 10:44:23
 +
+

*Thread Reply:* thanks for the pointer @Maciej Obuchowski

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-08-02 10:49:17
 +
+

*Thread Reply:* Also working on updating the apidocs

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-08-02 11:21:14
 +
+

*Thread Reply:* The API docs are now up to date @Juan Luis Cano Rodríguez! Thank you for raising this issue.

+ + + +
+ 🙌:skin_tone_3: Juan Luis Cano Rodríguez +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-08-02 12:58:15
 +
+

@channel +If you can, please join us in San Francisco for a meetup at Astronomer on August 30th at 5:30 PM PT. +On the agenda: a presentation by special guest @John Lukenoff plus updates on the Airflow Provider, static lineage, and more. +Food will be provided, and all are welcome. +Please https://www.meetup.com/meetup-group-bnfqymxe/events/295195280/?utmmedium=referral&utmcampaign=share-btnsavedeventssharemodal&utmsource=link|RSVP to let us know you’re coming.

+
+
Meetup
+ + + +
OpenLineage Meetup @ Astronomer, Wed, Aug 30, 2023, 5:30 PM | Meetup
+
+ Data engineers and pipeline managers know that producing data lineage – end-to-end pipeline metadata instrumented at runtime or parsed at design time – is a heavy lift with +
+ + + + + + + + +
Original URL: https://www.meetup.com/meetup-group-bnfqymxe/events/295195280/?utm_medium=referral&amp;utm_campaign=share-btn_savedevents_share_modal&amp;utm_source=link
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Zahi Fail + (zahi.fail@gmail.com) +
+
2023-08-03 03:18:08
 +
+

Hey, I hope this is the right channel for this kind of question - I’m running a tests to integrate Airflow (2.4.3) with Marquez (Openlineage 0.30.1). Currently, I’m testing the postgres operator and for some reason queries like “Copy” and “Unload” are being sent as events, but doesn’t appear in the graph. Any idea how to solve it?

+ +

You can see attached

+ +
  1. The graph of an airflow DAG with all the tasks beside the copy and unload.
  2. The graph with the unload task that isn’t connected to the other flow.
    3. +
+ +
+
+ + + + + + +
+
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-03 05:36:04
 +
+

*Thread Reply:* I think our underlying SQL parser does not hancle the Postgres versions of those queries

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-03 05:36:14
 +
+

*Thread Reply:* Can you post the (anonymized?) queries?

+ + + +
+ 👍 Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Zahi Fail + (zahi.fail@gmail.com) +
+
2023-08-03 07:03:09
 +
+

*Thread Reply:* for example

+ +
copy bi.marquez_test_2 from '******' iam_role '**********' delimiter as '^' gzi
+
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-07 13:35:30
 +
+

*Thread Reply:* @Zahi Fail iam_role suggests you want redshift version of this supported, not Postgres one right?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Zahi Fail + (zahi.fail@gmail.com) +
+
2023-08-08 04:04:35
 +
+

*Thread Reply:* @Maciej Obuchowski hey, actually I tried both Postgres and Redshift to S3 operators. +Both of them sent a new event through OL to Marquez, and still wasn’t part of the entire flow.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Athitya Kumar + (athityakumar@gmail.com) +
+
2023-08-04 01:40:15
 +
+

Hey team! 👋

+ +

We were exploring open-lineage and had a couple of questions:

+ +
  1. Does open-lineage support presto-sql?
  2. Do we have any docs/benchmarks on query coverage (inner joins, subqueries, etc) & source/sink coverage (spark.read from JDBC, Files etc) for spark-sql?
  3. Can someone point to the code where we currently parse the input/output facets from the spark integration (like sql queries / transformations) and if it's extendable?
    4. +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-08-04 02:17:19
 +
+

*Thread Reply:* Hey @Athitya Kumar,

+ +
  1. For parsing SQL queries, we're using sqlparser-rs (https://github.com/sqlparser-rs/sqlparser-rs) which already has great coverage of sql syntax and supports different dialects. it's open source project and we already did contribute to it for snowflake dialect.
  2. We don't have such a benchmark, but if you like, you could contribute and help us providing such. We do support joins, subqueries, iceberg and delta tables, jdbc for Spark and much more. Everything we do support, is covered in our tests.
  3. Not sure if got it properly. Marquez is our reference backend implementation which parses all the facets and stores them in relational db in a relational manner (facets, jobs, datasets and runs in separate tables).
    4. +
+
+ + + + +
sqlparser-rs/sqlparser-rs
+
+ Extensible SQL Lexer and Parser for Rust +
+ +
+
Stars
+ 1956 +
+ +
+
Language
+ Rust +
+ + + + + + + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Athitya Kumar + (athityakumar@gmail.com) +
+
2023-08-04 02:29:53
 +
+

*Thread Reply:* For (3), I was referring to where we call the sqlparser-rs in our spark-openlineage event listener / integration; and how customising/improving them would look like

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-08-04 02:37:20
 +
+

*Thread Reply:* sqlparser-rs is a rust libary and we bundle it within iface-java (https://github.com/OpenLineage/OpenLineage/blob/main/integration/sql/iface-java/src/main/java/io/openlineage/sql/SqlMeta.java). It's capable of extracting input/output datasets, column lineage information from SQL

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/integration/sql/iface-java/src/main/java/io/openlineage/sql/SqlMeta.java | SqlMeta.java>
+
+ <pre><code>/** +/** Copyright 2018-2023 contributors to the OpenLineage project +/** SPDX-License-Identifier: Apache-2.0 +**/ + +package io.openlineage.sql; + +import java.util.Arrays; +import java.util.List; +import org.apache.commons.lang3.builder.HashCodeBuilder; + +public class SqlMeta { + private final List&lt;DbTableMeta&gt; inTables; + private final List&lt;DbTableMeta&gt; outTables; + private final List&lt;ColumnLineage&gt; columnLineage; + private final List&lt;ExtractionError&gt; errors; + + public SqlMeta( + List&lt;DbTableMeta&gt; in, + List&lt;DbTableMeta&gt; out, + List&lt;ColumnLineage&gt; columnLineage, + List&lt;ExtractionError&gt; errors) { + this.inTables = in; + this.outTables = out; + this.columnLineage = columnLineage; + this.errors = errors; + } + + public List&lt;DbTableMeta&gt; inTables() { + return inTables; + } + + public List&lt;DbTableMeta&gt; outTables() { + return outTables; + } + + public List&lt;ColumnLineage&gt; columnLineage() { + return columnLineage; + } + + public List&lt;ExtractionError&gt; errors() { + return errors; + } + + @Override + public String toString() { + return String.format( + "{\"inTables\": %s, \"outTables\": %s, \"columnLineage\": %s, \"errors\": %s}", + Arrays.toString(inTables.toArray()), + Arrays.toString(outTables.toArray()), + Arrays.toString(columnLineage.toArray()), + Arrays.toString(errors.toArray())); + } + + @Override + public boolean equals(Object o) { + if (o == this) { + return true; + } + + if (!(o instanceof SqlMeta)) { + return false; + } + + SqlMeta other = (SqlMeta) o; + return other.inTables.equals(inTables) + &amp;&amp; other.outTables.equals(outTables) + &amp;&amp; other.columnLineage.equals(columnLineage) + &amp;&amp; other.errors.equals(errors); + } + + @Override + public int hashCode() { + return new HashCodeBuilder() + .append(inTables) + .append(outTables) + .append(columnLineage) + .append(errors) + .toHashCode(); + } +} + +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-08-04 02:40:02
 +
+

*Thread Reply:* and this is Spark code that extracts it from JdbcRelation -> https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/shared/src/[…]ge/spark/agent/lifecycle/plan/handlers/JdbcRelationHandler.java

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/shared/src/main/java/io/openlineage/spark/agent/lifecycle/plan/handlers/JdbcRelationHandler.java | JdbcRelationHandler.java>
+
+ <pre><code>/** +/** Copyright 2018-2023 contributors to the OpenLineage project +/** SPDX-License-Identifier: Apache-2.0 +**/ + +package io.openlineage.spark.agent.lifecycle.plan.handlers; + +import io.openlineage.client.OpenLineage; +import io.openlineage.spark.agent.util.JdbcUtils; +import io.openlineage.spark.api.DatasetFactory; +import io.openlineage.sql.ColumnMeta; +import io.openlineage.sql.DbTableMeta; +import io.openlineage.sql.SqlMeta; +import java.util.Collections; +import java.util.List; +import java.util.Optional; +import java.util.stream.Collectors; +import org.apache.spark.sql.execution.datasources.LogicalRelation; +import org.apache.spark.sql.execution.datasources.jdbc.JDBCRelation; +import org.apache.spark.sql.types.StructField; +import org.apache.spark.sql.types.StructType; + +public class JdbcRelationHandler&lt;D extends OpenLineage.Dataset&gt; { + + private final DatasetFactory&lt;D&gt; datasetFactory; + + public JdbcRelationHandler(DatasetFactory&lt;D&gt; datasetFactory) { + this.datasetFactory = datasetFactory; + } + + public List&lt;D&gt; handleRelation(LogicalRelation x) { + // strip the jdbc: prefix from the url. this leaves us with a url like + // postgresql://&lt;hostname&gt;:&lt;port&gt;/&lt;database_name&gt;?params + // we don't parse the URI here because different drivers use different + // connection + // formats that aren't always amenable to how Java parses URIs. E.g., the oracle + // driver format looks like oracle:&lt;drivertype&gt;:&lt;user&gt;/&lt;password&gt;@&lt;database&gt; + // whereas postgres, mysql, and sqlserver use the &lt;scheme://hostname:port/db&gt; + // format. + JDBCRelation relation = (JDBCRelation) x.relation(); + String url = JdbcUtils.sanitizeJdbcUrl(relation.jdbcOptions().url()); + return getDatasets(relation, url); + } + + public List&lt;D&gt; getDatasets(JDBCRelation relation, String url) { + Optional&lt;SqlMeta&gt; sqlMeta = JdbcUtils.extractQueryFromSpark(relation); + if (!sqlMeta.isPresent()) { + return Collections.emptyList(); + } + if (sqlMeta.get().columnLineage().isEmpty()) { + return Collections.singletonList( + datasetFactory.getDataset( + sqlMeta.get().inTables().get(0).qualifiedName(), url, relation.schema())); + } + return sqlMeta.get().inTables().stream() + .map( + dbtm -&gt; + datasetFactory.getDataset( + dbtm.qualifiedName(), + url, + generateJDBCSchema(dbtm, relation.schema(), sqlMeta.get()))) + .collect(Collectors.toList()); + } + + private static StructType generateJDBCSchema( + DbTableMeta origin, StructType schema, SqlMeta sqlMeta) { + StructType originSchema = new StructType(); + for (StructField f : schema.fields()) { + List&lt;ColumnMeta&gt; fields = + sqlMeta.columnLineage().stream() + .filter(cl -&gt; cl.descendant().name().equals(f.name())) + .flatMap( + cl -&gt; + cl.lineage().stream() + .filter( + cm -&gt; cm.origin().isPresent() &amp;&amp; cm.origin().get().equals(origin))) + .collect(Collectors.toList()); + for (ColumnMeta cm : fields) { + originSchema = originSchema.add(cm.name(), f.dataType()); + } + } + return originSchema; + } +} + +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-04 04:08:53
 +
+

*Thread Reply:* I think 3 question relates generally to Spark SQL handling, rather than handling JDBC connections inside Spark, right?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Athitya Kumar + (athityakumar@gmail.com) +
+
2023-08-04 04:24:57
 +
+

*Thread Reply:* Yup, both actually. Related to getting the JDBC connection info in the input/output facet, as well as spark-sql queries we do on that JDBC connection

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-04 06:00:17
 +
+

*Thread Reply:* For Spark SQL - it's translated to Spark's internal query LogicalPlan. We take that plan, and process it's nodes. From root node we can take output dataset, from leaf nodes we can take input datasets, and inside internal nodes we track columns to extract column-level lineage. We express those (table-level) operations by implementing classes like QueryPlanVisitor

+ +

You can extend that, for example for additional types of nodes that we don't support by implementing your own QueryPlanVisitor, and then implementing OpenLineageEventHandlerFactory and packaging this into a .jar deployed alongside OpenLineage jar - this would be loaded by us using Java's ServiceLoader .

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/a9809c3e3ccb959c93af8edd6e3358514a8edd2e/integration/spark/shared/src/main/java/io/openlineage/spark/api/QueryPlanVisitor.java | QueryPlanVisitor.java>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/3e5ffee320fc52a8b287d99c678a89335c9c468c/integration/spark/shared/src/main/java/io/openlineage/spark/api/OpenLineageEventHandlerFactory.java | OpenLineageEventHandlerFactory.java>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+ 👍 Kiran Hiremath +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Athitya Kumar + (athityakumar@gmail.com) +
+
2023-08-08 05:06:07
 +
+

*Thread Reply:* @Maciej Obuchowski @Paweł Leszczyński - Thanks for your responses! I had a follow-up query regarding the sqlparser-rs that's used internally by open-lineage: we see that these are the SQL dialects supported by sqlparser-rs here doesn't include spark-sql / presto-sql dialects which means they'd fallback to generic dialect:

+ +

"--ansi" =&gt; Box::new(AnsiDialect {}), +"--bigquery" =&gt; Box::new(BigQueryDialect {}), +"--postgres" =&gt; Box::new(PostgreSqlDialect {}), +"--ms" =&gt; Box::new(MsSqlDialect {}), +"--mysql" =&gt; Box::new(MySqlDialect {}), +"--snowflake" =&gt; Box::new(SnowflakeDialect {}), +"--hive" =&gt; Box::new(HiveDialect {}), +"--redshift" =&gt; Box::new(RedshiftSqlDialect {}), +"--clickhouse" =&gt; Box::new(ClickHouseDialect {}), +"--duckdb" =&gt; Box::new(DuckDbDialect {}), +"--generic" | "" =&gt; Box::new(GenericDialect {}), +Any idea on how much coverage generic dialect provides for spark-sql / how different they are etc?

+
+ + + + +
<https://github.com/sqlparser-rs/sqlparser-rs/blob/173a6db8189a0f1bb9896117c86374c4e691f0f3/examples/cli.rs | cli.rs>
+
+ <pre><code> let dialect: Box&lt;dyn Dialect&gt; = match std::env::args().nth(2).unwrap_or_default().as_ref() { +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/sqlparser-rs/sqlparser-rs">sqlparser-rs/sqlparser-rs</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-08-08 05:21:32
 +
+

*Thread Reply:* spark-sql integration is based on spark LogicalPlan's tree. Extracting input/output datasets from tree nodes which is more detailed than sql parsing

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-08 07:04:52
 +
+

*Thread Reply:* I think presto/trino dialect is very standard - there shouldn't be any problems with regular queries

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Athitya Kumar + (athityakumar@gmail.com) +
+
2023-08-08 11:19:53
 +
+

*Thread Reply:* @Paweł Leszczyński - Got it, and would you be able to point me to where within the openlineage-spark integration do we:

+ +
  1. provide the Spark Logical Plan / query to sqlparser-rs
  2. get the output of sqlparser-rs (parsed query AST) & stitch back the inputs/outputs in the open-lineage events?
    3. +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Athitya Kumar + (athityakumar@gmail.com) +
+
2023-08-08 12:09:06
 +
+

*Thread Reply:* For example, we'd like to understand which dialectname of sqlparser-rs would be used in which scenario by open-lineage and what're the interactions b/w open-lineage & sqlparser-rs

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Athitya Kumar + (athityakumar@gmail.com) +
+
2023-08-09 12:18:47
 +
+

*Thread Reply:* @Paweł Leszczyński - Incase you missed the above messages ^

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-08-10 03:31:32
 +
+

*Thread Reply:* Sqlparser-rs is used within Spark integration only for spark jdbc queries (queries to external databases). That's the only scenario. For spark.sql(...) , instead of SQL parsing, we rely on a logical plan of a job and extract information from it. For jdbc queries, that user sqlparser-rs, dialect is extracted from url: +https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/shared/src/main/java/io/openlineage/spark/agent/util/JdbcUtils.java#L69

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/shared/src/main/java/io/openlineage/spark/agent/util/JdbcUtils.java | JdbcUtils.java>
+
+ <pre><code> private static String extractDialectFromJdbcUrl(String jdbcUrl) { +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+ 👍 Athitya Kumar +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
nivethika R + (nivethikar8@gmail.com) +
+
2023-08-06 07:16:53
 +
+

Hi.. Is column lineage available for spark version 2.4.0?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-06 17:25:31
 +
+

*Thread Reply:* No, it's not.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
nivethika R + (nivethikar8@gmail.com) +
+
2023-08-06 23:53:17
 +
+

*Thread Reply:* Is it only available for spark version 3+?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-07 04:53:41
 +
+

*Thread Reply:* Yes

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
GitHubOpenLineageIssues + (githubopenlineageissues@gmail.com) +
+
2023-08-07 11:18:25
 +
+

Hi, Will really appreciate if I can learn how community have been able to harness spark integration. In our testing where a spark application writes to S3 multiple times (different location), OL generates the same job name for all writes (namepsacename.executeinsertintohadoopfsrelation_command ) rendering the OL graph final output less helpful. Say for example if I have series of transformation/writes 5 times , in Lineage graph we are just seeing last 1. There is an open bug and hopefully will be resolved soon.

+ +

Curious how much is adoption of OL spark integration in presence of that bug, as generating same name for a job makes it less usable for anything other than trivial one output application.

+ +

Example from 2 write application +EXPECTED : first produce weather dataset and the subsequent produce weather40. (generated/mocked using 2 spark app). (1st image) +ACTUAL OL : weather40. see only last one. (2nd image)

+ +

Will really appreciate community guidance as in how successful they have been in utilizing spark integration (vanilla not Databricks) . Thank you

+ +

Expected. vs Actual.

+ +
+
+ + + + + + +
+
+
+ + + + + + +
+ + +
+
+
+
+ + + + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-08-07 11:30:00
 +
+

@channel +This month’s TSC meeting is this Thursday, August 10th at 10:00 a.m. PT. On the tentative agenda: +• announcements +• recent releases +• Airflow provider progress update +• OpenLineage 1.0 overview +• open discussion +• more (TBA) +More info and the meeting link can be found on the website. All are welcome! Also, feel free to reply or DM me with discussion topics, agenda items, etc.

+
+
openlineage.io
+ + + +
TSC Meetings | OpenLineage
+
+ The OpenLineage Technical Steering Committee meets monthly, and is open to all. +
+ + + + + + +
Original URL: https://openlineage.io/meetings/
+ + + +
+ + + +
+ 👍 Maciej Obuchowski, Athitya Kumar, Anirudh Shrinivason, Paweł Leszczyński +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
추호관 + (hogan.chu@toss.im) +
+
2023-08-08 04:39:45
 +
+

I can’t see output when saveAsTable 100+ columns in spark. Any help or ideas for issue? Really thanks.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-08 04:59:23
 +
+

*Thread Reply:* Does this work with similar jobs, but with small amount of columns?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
추호관 + (hogan.chu@toss.im) +
+
2023-08-08 05:12:52
 +
+

*Thread Reply:* thanks for reply @Maciej Obuchowski yes it works for small amount of columns +but not work in big amount of columns

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-08 05:14:04
 +
+

*Thread Reply:* one more question: how much data the jobs approximately process and how long does the execution take?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
추호관 + (hogan.chu@toss.im) +
+
2023-08-08 05:14:54
 +
+

*Thread Reply:* ah… it’s like 20 min ~ 30 min various +data size is like 2000,0000 rows with columns 100 ~ 1000

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-08-08 05:15:17
 +
+

*Thread Reply:* that's interesting. we could prepare integration test for that. 100 cols shouldn't make a difference

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
추호관 + (hogan.chu@toss.im) +
+
2023-08-08 05:15:37
 +
+

*Thread Reply:* honestly sorry for typo its 1000 columns

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
추호관 + (hogan.chu@toss.im) +
+
2023-08-08 05:15:44
 +
+

*Thread Reply:* pivoting features

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
추호관 + (hogan.chu@toss.im) +
+
2023-08-08 05:16:09
 +
+

*Thread Reply:* i check it works good for small numbers of columns

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-08 05:16:39
 +
+

*Thread Reply:* if it's 1000, then maybe we're over event size - event is too large and backend can't accept that

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-08 05:17:06
 +
+

*Thread Reply:* maybe debug logs could tell us something

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
추호관 + (hogan.chu@toss.im) +
+
2023-08-08 05:19:27
 +
+

*Thread Reply:* i’ll do spark.sparkContext.setLogLevel("DEBUG") ing

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-08-08 05:19:30
 +
+

*Thread Reply:* are there any errors in the logs? perhaps pivoting uses contains nodes in SparkPlan that we don't support yet

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-08-08 05:19:52
 +
+

*Thread Reply:* did you check pivoting that results in less columns?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-08 05:20:33
 +
+

*Thread Reply:* @추호관 would also be good to disable logicalPlan facet: +spark.openlineage.facets.disabled: [spark_unknown;spark.logicalPlan] +in spark conf

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
추호관 + (hogan.chu@toss.im) +
+
2023-08-08 05:23:40
 +
+

*Thread Reply:* got it can’t we do in python config +.config("spark.dynamicAllocation.enabled", "true") \ +.config("spark.dynamicAllocation.initialExecutors", "5") \ +.config("spark.openlineage.facets.disabled", [spark_unknown;spark.logicalPlan]

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-08-08 05:24:31
 +
+

*Thread Reply:* .config("spark.dynamicAllocation.enabled", "true") \ +.config("spark.dynamicAllocation.initialExecutors", "5") \ +.config("spark.openlineage.facets.disabled", "[spark_unknown;spark.logicalPlan]"

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
추호관 + (hogan.chu@toss.im) +
+
2023-08-08 05:24:42
 +
+

*Thread Reply:* ah.. string got it

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
추호관 + (hogan.chu@toss.im) +
+
2023-08-08 05:36:03
 +
+

*Thread Reply:* ah… there are no errors nor debug level issue successfully Registered listener ìo.openlineage.spark.agent.OpenLineageSparkListener

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
추호관 + (hogan.chu@toss.im) +
+
2023-08-08 05:39:40
 +
+

*Thread Reply: maybe df.groupBy(some column).pivot(some_column).agg(*agg_cols) is not supported

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
추호관 + (hogan.chu@toss.im) +
+
2023-08-08 05:43:44
 +
+

*Thread Reply:* oh.. interesting spark.openlineage.facets.disabled this option gives me output when eventType is START +“eventType”: “START” +“outputs”: [ +… +columns +…. +]

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
추호관 + (hogan.chu@toss.im) +
+
2023-08-08 05:54:13
 +
+

*Thread Reply:* Yes +"spark.openlineage.facets.disabled", "[spark_unknown;spark.logicalPlan]" <- this option gives output when eventType is START but not give output bunches of columns when that config is not set

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-08-08 05:55:18
 +
+

*Thread Reply:* this option prevents logicalPlan being serialized and sent as a part of Openlineage event which included in one of the facets

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-08-08 05:56:12
 +
+

*Thread Reply:* possibly, serializing logicalPlans, in case of pivots, leads to size of the events that are not acceptable

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
추호관 + (hogan.chu@toss.im) +
+
2023-08-08 05:57:56
 +
+

*Thread Reply:* Ah… so you mean pivot makes serializing logical plan not availble for generating event because of size. +and disable logical plan with not serializing make availabe to generate event cuz not serialiize logical plan made by pivot

+ +

Can we overcome this

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-08-08 05:58:48
 +
+

*Thread Reply:* we've seen such issues for some plans some time ago

+ + + +
+ 🙌 추호관 +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
추호관 + (hogan.chu@toss.im) +
+
2023-08-08 05:59:29
 +
+

*Thread Reply:* oh…. how did you solve it?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-08-08 05:59:32
 +
+

*Thread Reply:* https://github.com/OpenLineage/OpenLineage/blob/c3a5211f919c01870a7f79f48588177a9b[…]io/openlineage/spark/agent/lifecycle/LogicalPlanSerializer.java

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/c3a5211f919c01870a7f79f48588177a9b687fdf/integration/spark/shared/src/main/java/io/openlineage/spark/agent/lifecycle/LogicalPlanSerializer.java | LogicalPlanSerializer.java>
+
+ <pre><code> @JsonIgnoreProperties({ +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+ 🙌 추호관 +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-08-08 05:59:51
 +
+

*Thread Reply:* by excluding some properties from plan to be serialized

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-08-08 06:01:14
 +
+

*Thread Reply:* here https://github.com/OpenLineage/OpenLineage/blob/c3a5211f919c01870a7f79f48588177a9b[…]io/openlineage/spark/agent/lifecycle/LogicalPlanSerializer.java we exclude certain classes

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/c3a5211f919c01870a7f79f48588177a9b687fdf/integration/spark/shared/src/main/java/io/openlineage/spark/agent/lifecycle/LogicalPlanSerializer.java | LogicalPlanSerializer.java>
+
+ <pre><code> .put(PythonRDD.class, PythonRDDMixin.class) +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+ 🙌 추호관 +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
추호관 + (hogan.chu@toss.im) +
+
2023-08-08 06:02:00
 +
+

*Thread Reply:* AH…. excluded properties cause ignore logical plan’s of pivointing

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-08-08 06:08:25
 +
+

*Thread Reply:* you can start with writing a failing test here -> https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/app/src/tes[…]/openlineage/spark/agent/lifecycle/SparkReadWriteIntegTest.java

+ +

then you can try to debug logical plan trying to find out what should be excluded from it when it's being serialized. Even, if you find this difficult, a failing integration test is super helpful to let others help you in that.

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/app/src/test/java/io/openlineage/spark/agent/lifecycle/SparkReadWriteIntegTest.java | SparkReadWriteIntegTest.java>
+
+ <p>``` +/<em>* +/</em>* Copyright 2018-2023 contributors to the OpenLineage project +/<em>* SPDX-License-Identifier: Apache-2.0 +*</em>/</p> + +<p>package io.openlineage.spark.agent.lifecycle;</p> + +<p>import static io.openlineage.client.OpenLineage.RunEvent; +import static org.assertj.core.api.Assertions.assertThat; +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertNotNull; +import static org.junit.jupiter.api.Assertions.assertTrue; +import static org.mockito.ArgumentMatchers.eq; +import static org.mockito.Mockito.atLeast; +import static org.mockito.Mockito.reset; +import static org.mockito.Mockito.when; +import static org.mockito.internal.verification.VerificationModeFactory.times;</p> + +<p>import com.fasterxml.jackson.core.JsonGenerator; +import com.fasterxml.jackson.databind.ObjectMapper; +import com.google.cloud.bigquery.MockBigQueryRelationProvider; +import com.google.cloud.bigquery.connector.common.BigQueryUtil; +import com.google.cloud.spark.bigquery.repackaged.com.google.cloud.bigquery.BigQuery; +import com.google.cloud.spark.bigquery.repackaged.com.google.cloud.bigquery.Field; +import com.google.cloud.spark.bigquery.repackaged.com.google.cloud.bigquery.Schema; +import com.google.cloud.spark.bigquery.repackaged.com.google.cloud.bigquery.StandardSQLTypeName; +import com.google.cloud.spark.bigquery.repackaged.com.google.cloud.bigquery.StandardTableDefinition; +import com.google.cloud.spark.bigquery.repackaged.com.google.cloud.bigquery.TableId; +import com.google.cloud.spark.bigquery.repackaged.com.google.inject.Binder; +import com.google.cloud.spark.bigquery.repackaged.com.google.inject.Module; +import com.google.cloud.spark.bigquery.repackaged.com.google.inject.Provides; +import com.google.common.collect.ImmutableMap; +import io.openlineage.client.OpenLineage; +import io.openlineage.client.OpenLineage.DatasetFacets; +import io.openlineage.client.OpenLineage.InputDataset; +import io.openlineage.client.OpenLineage.OutputDataset; +import io.openlineage.client.OpenLineage.SchemaDatasetFacet; +import io.openlineage.client.OpenLineage.SchemaDatasetFacetFields; +import io.openlineage.spark.agent.SparkAgentTestExtension; +import io.openlineage.spark.agent.Versions; +import io.openlineage.spark.agent.util.PlanUtils; +import io.openlineage.spark.agent.util.SparkVersionUtils; +import io.openlineage.spark.agent.util.TestOpenLineageEventHandlerFactory; +import java.io.FileOutputStream; +import java.io.IOException; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.nio.file.Path; +import java.nio.file.Paths; +import java.util.Arrays; +import java.util.List; +import java.util.Optional; +import java.util.Properties; +import java.util.Random; +import java.util.UUID; +import java.util.concurrent.CompletableFuture; +import java.util.concurrent.ExecutionException; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.TimeoutException; +import lombok.extern.slf4j.Slf4j; +import org.apache.commons.lang3.tuple.Pair; +import org.apache.hadoop.io.LongWritable; +import org.apache.hadoop.io.Text; +import org.apache.hadoop.mapred.FileInputFormat; +import org.apache.hadoop.mapred.JobConf; +import org.apache.hadoop.mapred.TextInputFormat; +import org.apache.kafka.clients.producer.KafkaProducer; +import org.apache.kafka.clients.producer.ProducerRecord; +import org.apache.kafka.common.serialization.StringSerializer; +import org.apache.spark.api.java.JavaRDD; +import org.apache.spark.api.java.JavaSparkContext; +import org.apache.spark.sql.AnalysisException; +import org.apache.spark.sql.Dataset; +import org.apache.spark.sql.Row; +import org.apache.spark.sql.SaveMode; +import org.apache.spark.sql.SparkSession; +import org.apache.spark.sql.catalyst.expressions.GenericRow; +import org.apache.spark.sql.catalyst.expressions.GenericRowWithSchema; +import org.apache.spark.sql.types.BinaryType$; +import org.apache.spark.sql.types.IntegerType$; +import org.apache.spark.sql.types.LongType$; +import org.apache.spark.sql.types.Metadata; +import org.apache.spark.sql.types.StringType$; +import org.apache.spark.sql.types.StructField; +import org.apache.spark.sql.types.StructType; +import org.assertj.core.api.InstanceOfAssertFactories; +import org.assertj.core.api.ObjectAssert; +import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Tag; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.condition.EnabledIfSystemProperty; +import org.junit.jupiter.api.extension.ExtendWith; +import org.junit.jupiter.api.io.TempDir; +import org.mockito.ArgumentCaptor; +import org.mockito.Mockito; +import org.testcontainers.containers.KafkaContainer; +import org.testcontainers.utility.DockerImageName; +import scala.Tuple2; +import scala.collection.immutable.HashMap;</p> + +<p>@ExtendWith(SparkAgentTestExtension.class) +@Tag("integration-test") +@Slf4j +class SparkReadWriteIntegTest {</p> + +<p>private static final String EVENT<em>TYPE = "eventType"; + private static final String NAMESPACE = "namespace"; + private static final String FILE = "file"; + private static final String NAME = "name"; + private static final String AGE = "age"; + private static final String FILE</em>URI_PREFIX = "file://"; + private final KafkaContainer kafkaContainer = + new KafkaContainer(DockerImageName.parse("confluentinc/cp-kafka:7.0.0"));</p> + +<p>@BeforeEach + public void setUp() { + reset(MockBigQueryRelationProvider.BIG<em>QUERY); + when(SparkAgentTestExtension.OPEN</em>LINEAGE<em>SPARK</em>CONTEXT.getParentRunId()) + .thenReturn(Optional.of(UUID.randomUUID())); + when(SparkAgentTestExtension.OPEN<em>LINEAGE</em>SPARK<em>CONTEXT.getParentJobName()) + .thenReturn("ParentJob"); + when(SparkAgentTestExtension.OPEN</em>LINEAGE<em>SPARK</em>CONTEXT.getJobNamespace()) + .thenReturn("Namespace"); + }</p> + +<p>@AfterEach + public void tearDown() { + if (kafkaContainer.isCreated()) { + kafkaContainer.stop(); + } + }</p> + +<p>@Test + void testBigQueryReadWriteToFile(@TempDir Path writeDir, SparkSession spark) + throws InterruptedException, TimeoutException { + TableId tableId = TableId.of("testproject", "dataset", "MyTable"); + BigQuery bq = MockBigQueryRelationProvider.BIG_QUERY; + StructType tableSchema = + new StructType( + new StructField[] { + new StructField(NAME, StringType$.MODULE$, false, Metadata.empty()), + new StructField(AGE, LongType$.MODULE$, false, Metadata.empty()) + });</p> + +<pre><code>MockBigQueryRelationProvider.INJECTOR.setTestModule( + new Module() { + @Override + public void configure(Binder binder) {} + + @Provides + public Dataset&lt;Row&gt; testData() { + return spark.createDataFrame( + Arrays.asList( + new GenericRowWithSchema(new Object[] {"john", 25L}, tableSchema), + new GenericRowWithSchema(new Object[] {"sam", 22L}, tableSchema), + new GenericRowWithSchema(new Object[] {"alicia", 35L}, tableSchema), + new GenericRowWithSchema(new Object[] {"bob", 47L}, tableSchema), + new GenericRowWithSchema(new Object[] {"jordan", 52L}, tableSchema), + new GenericRowWithSchema(new Object[] {"liz", 19L}, tableSchema), + new GenericRowWithSchema(new Object[] {"marcia", 83L}, tableSchema), + new GenericRowWithSchema(new Object[] {"maria", 40L}, tableSchema), + new GenericRowWithSchema(new Object[] {"luis", 8L}, tableSchema), + new GenericRowWithSchema(new Object[] {"gabriel", 30L}, tableSchema)), + tableSchema); + } + }); +when(bq.getTable(eq(tableId))) + .thenAnswer( + invocation -&gt; + MockBigQueryRelationProvider.makeTable( + tableId, + StandardTableDefinition.newBuilder() + .setSchema( + Schema.of( + Field.of(NAME, StandardSQLTypeName.STRING), + … +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
추호관 + (hogan.chu@toss.im) +
+
2023-08-08 06:24:54
 +
+

*Thread Reply:* okay i would look into and maybe pr thanks

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
추호관 + (hogan.chu@toss.im) +
+
2023-08-08 06:38:45
 +
+

*Thread Reply:* Can I ask if there are any suspicious properties?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-08-08 06:39:25
 +
+

*Thread Reply:* sure

+ + + +
+ 👍 추호관 +
+ +
+ 🙂 추호관 +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
추호관 + (hogan.chu@toss.im) +
+
2023-08-08 07:10:40
 +
+

*Thread Reply:* Thanks I would also try to find the property too

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-08-08 05:34:46
 +
+

Hi guys, I've a generic sql-parsing doubt... what would be the recommended way (if any) to check for sql similarity? I understand that most sql parsers parse the query into an AST, but are there any well known ways to measure semantic similarities between 2 or more ASTs? Just curious lol... Any ideas appreciated! Thanks!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Guy Biecher + (guy.biecher21@gmail.com) +
+
2023-08-08 07:49:55
 +
+

*Thread Reply:* Hi @Anirudh Shrinivason, +I think I would take a look on this +https://sqlglot.com/sqlglot/diff.html

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-08-09 23:12:37
 +
+

*Thread Reply:* Hey @Guy Biecher Yeah I was looking at this... but it seems to calculate similarity from a more textual context, as opposed to a more semantic one... +eg: SELECT ** FROM TABLE_1 and SELECT col1,col2,col3 FROM TABLE_1 could be the same semantic query, but sqlglot would give diffs in the ast because its textual...

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Guy Biecher + (guy.biecher21@gmail.com) +
+
2023-08-10 02:26:51
 +
+

*Thread Reply:* I totally get you. In such cases without the metadata of the TABLE_1, it's impossible what I would do I would replace all ** before you use the diff function.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-08-10 07:04:37
 +
+

*Thread Reply:* Yeah I was thinking about the same... But the more nested and complex your queries get, the harder it'll become to accurately pre-process before running the ast diff too... +But yeah that's probably the approach I'd be taking haha... Happy to discuss and learn if there are better ways to doing this

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Luigi Scorzato + (luigi.scorzato@gmail.com) +
+
2023-08-08 08:36:46
 +
+

dear all, I have some novice questions. I put them in separate messages for clarity. 1st Question: I understand from the examples in the documentation that the main lineage events are RunEvent's, which can contain link to Run ID, Job ID, Dataset ID (I see they are RunEvent because they have EventType, correct?). However, the main openlineage json object contains also JobEvent and DatasetEvent. When are JobEvent and DatasetEvent supposed to be used in the workflow? Do you have relevant examples? thanks!

+
+
openlineage.io
+ + + +
Example Lineage Events | OpenLineage
+
+ Simple Examples +
+ + + + + + +
Original URL: https://openlineage.io/docs/development/examples
+ + + +
+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.json | OpenLineage.json>
+
+ <p>``` +{ + "$schema": "<a href="https://json-schema.org/draft/2020-12/schema">https://json-schema.org/draft/2020-12/schema</a>", + "$id": "<a href="https://openlineage.io/spec/2-0-2/OpenLineage.json">https://openlineage.io/spec/2-0-2/OpenLineage.json</a>", + "$defs": { + "BaseEvent": { + "type": "object", + "properties": { + "eventTime": { + "description": "the time the event occurred at", + "type": "string", + "format": "date-time" + }, + "producer": { + "description": "URI identifying the producer of this metadata. For example this could be a git url with a given tag or sha", + "type": "string", + "format": "uri", + "example": "<a href="https://github.com/OpenLineage/OpenLineage/blob/v1-0-0/client">https://github.com/OpenLineage/OpenLineage/blob/v1-0-0/client</a>" + }, + "schemaURL": { + "description": "The JSON Pointer (<a href="https://tools.ietf.org/html/rfc6901">https://tools.ietf.org/html/rfc6901</a>) URL to the corresponding version of the schema definition for this RunEvent", + "type": "string", + "format": "uri", + "example": "<a href="https://openlineage.io/spec/0-0-1/OpenLineage.json">https://openlineage.io/spec/0-0-1/OpenLineage.json</a>" + } + }, + "required": ["eventTime", "producer", "schemaURL"] + }, + "RunEvent": { + "allOf": [ + { "$ref": "#/$defs/BaseEvent" }, + { + "type": "object", + "properties": { + "eventType": { + "description": "the current transition of the run state. It is required to issue 1 START event and 1 of [ COMPLETE, ABORT, FAIL ] event per run. Additional events with OTHER eventType can be added to the same run. For example to send additional metadata after the run is complete", + "type": "string", + "enum": ["START", "RUNNING", "COMPLETE", "ABORT", "FAIL", "OTHER"], + "example": "START|RUNNING|COMPLETE|ABORT|FAIL|OTHER" + }, + "run": { + "$ref": "#/$defs/Run" + }, + "job": { + "$ref": "#/$defs/Job" + }, + "inputs": { + "description": "The set of <strong><em>*input</em><em></strong> datasets.", + "type": "array", + "items": { + "$ref": "#/$defs/InputDataset" + } + }, + "outputs": { + "description": "The set of <strong></em><em>output</em><em></strong> datasets.", + "type": "array", + "items": { + "$ref": "#/$defs/OutputDataset" + } + } + }, + "required": ["run", "job"] + } + ] + }, + "DatasetEvent": { + "allOf": [ + { "$ref": "#/$defs/BaseEvent" }, + { + "type": "object", + "properties": { + "dataset": { + "$ref": "#/$defs/StaticDataset" + } + }, + "required": ["dataset"], + "not": { "required": ["job", "run"] } + } + ] + }, + "JobEvent": { + "allOf": [ + { "$ref": "#/$defs/BaseEvent" }, + { + "type": "object", + "properties": { + "job": { + "$ref": "#/$defs/Job" + }, + "inputs": { + "description": "The set of <strong></em><em>input</em><em></strong> datasets.", + "type": "array", + "items": { + "$ref": "#/$defs/InputDataset" + } + }, + "outputs": { + "description": "The set of <strong></em><em>output</em>*</strong> datasets.", + "type": "array", + "items": { + "$ref": "#/$defs/OutputDataset" + } + } + }, + "required": ["job"], + "not": { "required": ["run"] } + } + ] + }, + "Run": { + "type": "object", + "properties": { + "runId": { + "description": "The globally unique ID of the run associated with the job.", + "type": "string", + "format": "uuid" + }, + "facets": { + "description": "The run facets.", + "type": "object", + "anyOf": [ + { + "type": "object", + "additionalProperties": { "$ref": "#/$defs/RunFacet" } + } + ] + } + }, + "required": ["runId"] + }, + "RunFacet": { + "description": "A Run Facet", + "type": "object", + "allOf": [{ "$ref": "#/$defs/BaseFacet" }] + }, + "Job": { + "type": "object", + "properties": { + "namespace": { + "description": "The namespace containing that job", + "type": "string", + "example": "my-scheduler-namespace" + }, + "name": { + "description": "The unique name for that job within that namespace", + "type": "string", + "example": "myjob.mytask" + }, + "facets": { + "description": "The job facets.", + "type": "object", + "anyOf": [ + { + "type": "object", + "additionalProperties": { "$ref": "#/$defs/JobFacet" } + } + ] + } + }, + "required": ["namespace", "name"] + }, + "JobFacet": { + "description": "A Job Facet", + "type": "object", + "allOf": [ + { "$ref": "#/$defs/BaseFacet" }, + { + "type": "object", + "properties": { + "<em>deleted": { + "description": "set to true to delete a facet", + "type": "boolean" + } + } + } + ] + }, + "InputDataset": { + "description": "An input dataset", + "type": "object", + "allOf": [ + { "$ref": "#/$defs/Dataset" }, + { + "type": "object", + "properties": { + "inputFacets": { + "description": "The input facets for this dataset.", + "type": "object", + "anyOf": [ + { + "type": "object", + "additionalProperties": { + "$ref": "#/$defs/InputDatasetFacet" + } + } + ] + } + } + } + ] + }, + "InputDatasetFacet": { + "description": "An Input Dataset Facet", + "type": "object", + "allOf": [{ "$ref": "#/$defs/BaseFacet" }] + }, + "OutputDataset": { + "description": "An output dataset", + "type": "object", + "allOf": [ + { "$ref": "#/$defs/Dataset" }, + { + "type": "object", + "properties": { + "outputFacets": { + "description": "The output facets for this dataset", + "type": "object", + "anyOf": [ + { + "type": "object", + "additionalProperties": { + "$ref": "#/$defs/OutputDatasetFacet" + } + } + ] + } + } + } + ] + }, + "OutputDatasetFacet": { + "description": "An Output Dataset Facet", + "type": "object", + "allOf": [{ "$ref": "#/$defs/BaseFacet" }] + }, + "Dataset": { + "type": "object", + "properties": { + "namespace": { + "description": "The namespace containing that dataset", + "type": "string", + "example": "my-datasource-namespace" + }, + "name": { + "description": "The unique name for that dataset within that namespace", + "type": "string", + "example": "instance.schema.table" + }, + "facets": { + "description": "The facets for this dataset", + "type": "object", + "anyOf": [ + { + "type": "object", + "additionalProperties": { "$ref": "#/$defs/DatasetFacet" } + } + ] + } + }, + "required": ["namespace", "name"] + }, + "StaticDataset": { + "description": "A Dataset sent within static metadata events", + "type": "object", + "allOf": [{ "$ref": "#/$defs/Dataset" }] + }, + "DatasetFacet": { + "description": "A Dataset Facet", + "type": "object", + "allOf": [ + { "$ref": "#/$defs/BaseFacet" }, + { + "type": "object", + "properties": { + "</em>deleted": { + …</p> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harel Shein + (harel.shein@gmail.com) +
+
2023-08-08 09:53:05
 +
+

*Thread Reply:* Hey @Luigi Scorzato! +You can read about these 2 event types in this blog post: https://openlineage.io/blog/static-lineage

+
+
openlineage.io
+ + + +
OpenLineage 1.0, Featuring Static Lineage, is Coming Soon | OpenLineage
+
+ The release of OpenLineage 1.0 will add static lineage support. +
+ + + + + + +
Original URL: https://openlineage.io/blog/static-lineage
+ + + +
+ + + +
+ 👍 Luigi Scorzato +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harel Shein + (harel.shein@gmail.com) +
+
2023-08-08 09:53:38
 +
+

*Thread Reply:* we’ll work on getting the documentation improved to clarify the expected use cases for each event type. this is a relatively new addition to the spec.

+ + + +
+ 👍 Luigi Scorzato +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Luigi Scorzato + (luigi.scorzato@gmail.com) +
+
2023-08-08 10:08:28
 +
+

*Thread Reply:* this sounds relevant for my 3rd question, doesn't it? But I do not see scheduling information among the use cases, am I wrong?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harel Shein + (harel.shein@gmail.com) +
+
2023-08-08 11:16:39
 +
+

*Thread Reply:* you’re not wrong, these 2 events were not designed for runtime lineage, but rather “static” lineage that gets emitted after the fact

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Luigi Scorzato + (luigi.scorzato@gmail.com) +
+
2023-08-08 08:46:39
 +
+

2nd Question. I see that the input dataset appears in the RunEvent with EventType=START, the output dataset appears in the RunEvent with EventType=COMPLETE only, the RunEvent with EventType=RUNNING has no dataset attached. This makes sense for ETL jobs, but for streaming (e.g. Flink), the run could run very long and never terminate with a COMPLETE. On the other hand, emitting all the info about the output dataset in every RUNNING event would be far too verbose. What is the recommended set up in this case? TLDR: what is the recommended configuration of the frequency and data model of the lineage events for streaming systems like Flink?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harel Shein + (harel.shein@gmail.com) +
+
2023-08-08 09:54:40
 +
+

*Thread Reply:* great question! did you get a chance to look at the current Flink integration?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Luigi Scorzato + (luigi.scorzato@gmail.com) +
+
2023-08-08 10:07:06
 +
+

*Thread Reply:* to be honest, I only quickly went through this and I did not identfy what I needed. Can you please point me to the relevant section?

+
+
openlineage.io
+ + + +
Apache Flink | OpenLineage
+
+ This integration is considered experimental: only specific workflows and use cases are supported. +
+ + + + + + +
Original URL: https://openlineage.io/docs/integrations/flink
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harel Shein + (harel.shein@gmail.com) +
+
2023-08-08 11:13:17
 +
+

*Thread Reply:* here’s an example START event for Flink: https://github.com/OpenLineage/OpenLineage/blob/main/integration/flink/src/test/resources/events/expected_kafka.json

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/integration/flink/src/test/resources/events/expected_kafka.json | expected_kafka.json>
+
+ <pre><code>{ + "eventType" : "START", + "eventTime": "${json-unit.any-string}", + "job" : { + "namespace" : "flink_job_namespace", + "name": "flink_examples_stateful" + }, + "run": { + "runId": "${json-unit.any-string}" + }, + "inputs": [ + { + "namespace": "kafka:9092", + "name": "io.openlineage.flink.kafka.input1", + "facets": { + "schema": { + "fields": [ + { + "name": "id", + "type": "string" + }, + { + "name": "version", + "type": "long" + } + ] + } + } + }, + { + "namespace": "kafka:9092", + "name": "io.openlineage.flink.kafka.input2", + "facets": { + "schema": { + "fields": [ + { + "name": "id", + "type": "string" + }, + { + "name": "version", + "type": "long" + } + ] + } + } + } + ], + "outputs": [ + { + "namespace": "kafka:9092", + "name": "io.openlineage.flink.kafka.output", + "facets": { + "schema": { + "fields": [ + { + "name": "id", + "type": "string" + }, + { + "name": "version", + "type": "long" + }, + { + "name": "counter", + "type": "long" + } + ] + } + } + } + ] +}, + +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harel Shein + (harel.shein@gmail.com) +
+
2023-08-08 11:13:26
 +
+

*Thread Reply:* or a checkpoint (RUNNING) event: https://github.com/OpenLineage/OpenLineage/blob/main/integration/flink/src/test/resources/events/expected_kafka_checkpoints.json

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/integration/flink/src/test/resources/events/expected_kafka_checkpoints.json | expected_kafka_checkpoints.json>
+
+ <pre><code>{ + "eventType" : "RUNNING", + "eventTime": "${json-unit.any-string}", + "job" : { + "namespace" : "flink_job_namespace", + "name": "flink_examples_stateful" + }, + "run": { + "runId": "${json-unit.any-string}", + "facets" : { + "checkpoints": { + "completed": "${json-unit.any-number}", + "failed": "${json-unit.any-number}", + "in-progress": "${json-unit.any-number}", + "restored": "${json-unit.any-number}", + "total": "${json-unit.any-number}" + } + } + }, + "inputs": [ + { + "namespace": "kafka:9092", + "name": "io.openlineage.flink.kafka.input1", + "facets": { + "schema": { + "fields": [ + { + "name": "id", + "type": "string" + }, + { + "name": "version", + "type": "long" + } + ] + } + } + }, + { + "namespace": "kafka:9092", + "name": "io.openlineage.flink.kafka.input2", + "facets": { + "schema": { + "fields": [ + { + "name": "id", + "type": "string" + }, + { + "name": "version", + "type": "long" + } + ] + } + } + } + ], + "outputs": [ + { + "namespace": "kafka:9092", + "name": "io.openlineage.flink.kafka.output", + "facets": { + "schema": { + "fields": [ + { + "name": "id", + "type": "string" + }, + { + "name": "version", + "type": "long" + }, + { + "name": "counter", + "type": "long" + } + ] + } + } + } + ] +} + +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harel Shein + (harel.shein@gmail.com) +
+
2023-08-08 11:15:55
 +
+

*Thread Reply:* generally speaking, you can see the execution contexts that invoke generation of OL events here: https://github.com/OpenLineage/OpenLineage/blob/main/integration/flink/src/main/ja[…]/openlineage/flink/visitor/lifecycle/FlinkExecutionContext.java

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/integration/flink/src/main/java/io/openlineage/flink/visitor/lifecycle/FlinkExecutionContext.java | FlinkExecutionContext.java>
+
+ <pre><code>/** +/** Copyright 2018-2023 contributors to the OpenLineage project +/** SPDX-License-Identifier: Apache-2.0 +**/ + +package io.openlineage.flink.visitor.lifecycle; + +import io.openlineage.client.OpenLineage; +import io.openlineage.client.OpenLineage.RunEvent; +import io.openlineage.client.OpenLineage.RunEvent.EventType; +import io.openlineage.client.OpenLineage.RunEventBuilder; +import io.openlineage.flink.SinkLineage; +import io.openlineage.flink.TransformationUtils; +import io.openlineage.flink.api.OpenLineageContext; +import io.openlineage.flink.client.CheckpointFacet; +import io.openlineage.flink.client.EventEmitter; +import io.openlineage.flink.visitor.Visitor; +import io.openlineage.flink.visitor.VisitorFactory; +import io.openlineage.flink.visitor.VisitorFactoryImpl; +import java.time.ZonedDateTime; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.HashSet; +import java.util.List; +import java.util.Set; +import java.util.UUID; +import java.util.stream.Collectors; +import lombok.Builder; +import lombok.Getter; +import lombok.extern.slf4j.Slf4j; +import org.apache.commons.lang3.exception.ExceptionUtils; +import org.apache.flink.api.common.JobExecutionResult; +import org.apache.flink.api.common.JobID; +import org.apache.flink.api.dag.Transformation; + +@Slf4j +@Builder +public class FlinkExecutionContext implements ExecutionContext { + + @Getter private final JobID jobId; + protected final UUID runId; + protected final EventEmitter eventEmitter; + protected final OpenLineageContext openLineageContext; + private final String jobName; + private final String jobNamespace; + + @Getter private final List&lt;Transformation&lt;?&gt;&gt; transformations; + + @Override + public void onJobSubmitted() { + log.debug("JobClient - jobId: {}", jobId); + RunEvent runEvent = + buildEventForEventType(EventType.START) + .run(new OpenLineage.RunBuilder().runId(runId).build()) + .build(); + log.debug("Posting event for onJobSubmitted {}: {}", jobId, runEvent); + eventEmitter.emit(runEvent); + } + + @Override + public void onJobCheckpoint(CheckpointFacet facet) { + log.debug("JobClient - jobId: {}", jobId); + RunEvent runEvent = + buildEventForEventType(EventType.RUNNING) + .run( + new OpenLineage.RunBuilder() + .runId(runId) + .facets(new OpenLineage.RunFacetsBuilder().put("checkpoints", facet).build()) + .build()) + .build(); + log.debug("Posting event for onJobCheckpoint {}: {}", jobId, runEvent); + eventEmitter.emit(runEvent); + } + + public OpenLineage.RunEventBuilder buildEventForEventType(EventType eventType) { + TransformationUtils converter = new TransformationUtils(); + + List&lt;SinkLineage&gt; sinkLineages = converter.convertToVisitable(transformations); + + VisitorFactory visitorFactory = new VisitorFactoryImpl(); + List&lt;OpenLineage.InputDataset&gt; inputDatasets = new ArrayList&lt;&gt;(); + List&lt;OpenLineage.OutputDataset&gt; outputDatasets = new ArrayList&lt;&gt;(); + + Set&lt;Object&gt; sources = new HashSet&lt;&gt;(); + for (var lineage : sinkLineages) { + sources.addAll(lineage.getSources()); + outputDatasets.addAll(getOutputDatasets(visitorFactory, lineage.getSink())); + } + inputDatasets.addAll(getInputDatasets(visitorFactory, Arrays.asList(sources.toArray()))); + + return commonEventBuilder().inputs(inputDatasets).outputs(outputDatasets).eventType(eventType); + } + + @Override + public void onJobCompleted(JobExecutionResult jobExecutionResult) { + OpenLineage openLineage = openLineageContext.getOpenLineage(); + eventEmitter.emit( + commonEventBuilder() + .run(openLineage.newRun(runId, null)) + .eventType(EventType.COMPLETE) + .build()); + } + + @Override + public void onJobFailed(Throwable failed) { + OpenLineage openLineage = openLineageContext.getOpenLineage(); + eventEmitter.emit( + commonEventBuilder() + .run( + openLineage.newRun( + runId, + openLineage + .newRunFacetsBuilder() + .errorMessage( + openLineage.newErrorMessageRunFacet( + failed.getMessage(), "JAVA", ExceptionUtils.getStackTrace(failed))) + .build())) + .eventType(EventType.FAIL) + .eventTime(ZonedDateTime.now()) + .build()); + } + + /**** + ** Builds common elements of Openlineage events constructed. + ** + ** @return + **/ + private RunEventBuilder commonEventBuilder() { + OpenLineage openLineage = openLineageContext.getOpenLineage(); + return openLineage + .newRunEventBuilder() + .job(openLineage.newJob(jobNamespace, jobName, null)) + .eventTime(ZonedDateTime.now()); + } + + private List&lt;OpenLineage.InputDataset&gt; getInputDatasets( + VisitorFactory visitorFactory, List&lt;Object&gt; sources) { + List&lt;OpenLineage.InputDataset&gt; inputDatasets = new ArrayList&lt;&gt;(); + List&lt;Visitor&lt;OpenLineage.InputDataset&gt;&gt; inputVisitors = + visitorFactory.getInputVisitors(openLineageContext); + + for (var transformation : sources) { + inputDatasets.addAll( + inputVisitors.stream() + .filter(inputVisitor -&gt; inputVisitor.isDefinedAt(transformation)) + .map(inputVisitor -&gt; inputVisitor.apply(transformation)) + .flatMap(List::stream) + .collect(Collectors.toList())); + } + return inputDatasets; + } + + private List&lt;OpenLineage.OutputDataset&gt; getOutputDatasets( + VisitorFactory visitorFactory, Object sink) { + List&lt;Visitor&lt;OpenLineage.OutputDataset&gt;&gt; outputVisitors = + visitorFactory.getOutputVisitors(openLineageContext); + + return outputVisitors.stream() + .filter(inputVisitor -&gt; inputVisitor.isDefinedAt(sink)) + .map(inputVisitor -&gt; inputVisitor.apply(sink)) + .flatMap(List::stream) + .collect(Collectors.toList()); + } +} + +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+ 👍 Luigi Scorzato +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Luigi Scorzato + (luigi.scorzato@gmail.com) +
+
2023-08-08 17:46:17
 +
+

*Thread Reply:* thank you! So, if I understand correctly, the key is that even eventType=START, admits an output datasets. Correct? What determines how often are the eventType=RUNNING emitted?

+ + + +
+ 👍 Harel Shein +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Luigi Scorzato + (luigi.scorzato@gmail.com) +
+
2023-08-09 03:25:16
 +
+

*Thread Reply:* now I see, RUNNING events are emitted on onJobCheckpoint

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Luigi Scorzato + (luigi.scorzato@gmail.com) +
+
2023-08-08 08:59:40
 +
+

3rd Question: I am looking for information about the time when the next run should start, in case of scheduled jobs. I see that the Run Facet has a Nominal Time Facet, but -- if I understand correctly -- it refers to the current run, so it is always emitted after the fact. Is the Nominal Start Time of the next run available somewhere? If not, where do you recommend to add it as a custom field? In principle, it belongs to the Job object, but would that maybe cause an undesirable fast change in the Job object?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harel Shein + (harel.shein@gmail.com) +
+
2023-08-08 11:10:47
 +
+

*Thread Reply:* For Airflow, this is part of the AirflowRunFacet, here: https://github.com/OpenLineage/OpenLineage/blob/81372ca2bc2afecab369eab4a54cc6380dda49d0/integration/airflow/facets/AirflowRunFacet.json#L100

+ +

For other orchestrators / schedulers, that would depend..

+ + + +
+ 👍 Luigi Scorzato +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kiran Hiremath + (kiran_hiremath@intuit.com) +
+
2023-08-08 10:30:56
 +
+

Hi Team, Question regarding Databricks OpenLineage init script, is the path /mnt/driver-daemon/jars common to all the clusters? or its unique to each cluster? https://github.com/OpenLineage/OpenLineage/blob/81372ca2bc2afecab369eab4a54cc6380d[…]da49d0/integration/spark/databricks/open-lineage-init-script.sh

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-08 12:15:40
 +
+

*Thread Reply:* I might be wrong, but I believe it's unique for each cluster - the common part is dbfs\.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-08-09 02:38:54
 +
+

*Thread Reply:* dbfs is mounted to a databricks workspace which can run multiple clusters. so i think, it's common.

+ +

Worth mentioning: init-scripts located in dbfs are becoming deprecated next month and we plan moving them into workspaces.

+ + + +
+ 👍 Kiran Hiremath +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kiran Hiremath + (kiran_hiremath@intuit.com) +
+
2023-08-11 01:33:24
 +
+

*Thread Reply:* yes, the init scripts are moved at workspace level.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
GitHubOpenLineageIssues + (githubopenlineageissues@gmail.com) +
+
2023-08-08 14:19:40
 +
+

Hi @Paweł Leszczyński Will really aprecaite if you please let me know once this PR is good to go. Will love to test in our environment : https://github.com/OpenLineage/OpenLineage/pull/2036. Thank you for all your help.

+
+ + + + +
#2036 [SPARK] append output dataset name to job name
+
+ <p><strong>Problem</strong></p> + +<p>👋 Thanks for opening a pull request! Please include a brief summary of the problem your change is trying to solve, or bug fix. If your change fixes a bug or you'd like to provide context on why you're making the change, please <a href="https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue">link the issue</a> as follows:</p> + +<p>Closes: #ISSUE-NUMBER</p> + +<p><strong>Solution</strong></p> + +<p>Please describe your change as it relates to the problem, or bug fix, as well as any dependencies. If your change requires a schema change, please describe the schema modifcation(s) and whether it's a <em>backwards-incompatible</em> or <em>backwards-compatible</em> change, then select one of the following:</p> + +<blockquote> + <p><strong>Note:</strong> All schema changes require discussion. Please <a href="https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue">link the issue</a> for context.</p> +</blockquote> + +<p>☐ Your change modifies the <a href="https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.json">core</a> OpenLineage model +☐ Your change modifies one or more OpenLineage <a href="https://github.com/OpenLineage/OpenLineage/tree/main/spec/facets">facets</a></p> + +<p>If you're contributing a new integration, please specify the scope of the integration and how/where it has been tested (e.g., Apache Spark integration supports <code>S3</code> and <code>GCS</code> filesystem operations, tested with AWS EMR).</p> + +<p><strong>One-line summary:</strong> +<strong>Checklist</strong></p> + +<p>☐ You've <a href="https://github.com/OpenLineage/OpenLineage/blob/main/why-the-dco.md">signed-off</a> your work +☐ Your pull request title follows our <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CONTRIBUTING.md#creating-pull-requests">guidelines</a> +☐ Your changes are accompanied by tests (<em>if relevant</em>) +☐ Your change contains a <a href="https://kurtisnusbaum.medium.com/stacked-diffs-keeping-phabricator-diffs-small-d9964f4dcfa6">small diff</a> and is self-contained +☐ You've updated any relevant documentation (<em>if relevant</em>) +☐ Your comment includes a one-liner for the changelog about the specific purpose of the change (<em>if necessary</em>) +☐ You've versioned the core OpenLineage model or facets according to <a href="https://docs.snowplowanalytics.com/docs/pipeline-components-and-applications/iglu/common-architecture/schemaver">SchemaVer</a> (<em>if relevant</em>) +☐ You've added a <a href="https://github.com/OpenLineage/OpenLineage/tree/main/.github/header_templates.md">header</a> to source files (<em>if relevant</em>)</p> + +<hr /> + +<p>SPDX-License-Identifier: Apache-2.0 <br /> +Copyright 2018-2023 contributors to the OpenLineage project</p> +
+ +
+
Labels
+ integration/spark +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-08-09 02:35:28
 +
+

*Thread Reply:* great to hear. I still need some time as there are few corner cases. For example: what should be the behaviour when alter table rename is called 😉 But sure, you can test it if you like. ci is failing on integration tests but ./gradlew clean build with unit tests are fine.

+ + + +
+ :gratitude_thank_you: GitHubOpenLineageIssues +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-08-10 03:33:50
 +
+

*Thread Reply:* @GitHubOpenLineageIssues Feel invited to join todays community and advocate for the importance of this issue. Such discussions are extremely helpful in prioritising backlog the right way.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Gaurav Singh + (gaurav.singh@razorpay.com) +
+
2023-08-09 07:54:33
 +
+

Hi Team, +I'm doing a POC with open lineage to extract column lineage of Spark. I'm using it on databricks notebook. I'm facing a issue where I,m trying to get the column lineage in a join involving external tables on s3. The lineage that is being extracted is returning on base path of the table ie on the s3 file path and not on the corresponding tables. Is there a way to extract/map columns of output to the columns of base tables instead of storage location.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Gaurav Singh + (gaurav.singh@razorpay.com) +
+
2023-08-09 07:55:28
 +
+

*Thread Reply:* Query: +INSERT INTO test.merchant_md +(Select + m.`id`, + m.name, + m.activated, + m.parent_id, + md.contact_name, + md.contact_email +FROM + test.merchants_0 m + LEFT JOIN merchant_details md ON m.id = md.merchant_id +WHERE + m.created_date &gt; '2023-08-01')

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Gaurav Singh + (gaurav.singh@razorpay.com) +
+
2023-08-09 08:01:56
 +
+

*Thread Reply:* "columnLineage":{ + "_producer":"<https://github.com/OpenLineage/OpenLineage/tree/0.30.1/integration/spark>", + "_schemaURL":"<https://openlineage.io/spec/facets/1-0-1/ColumnLineageDatasetFacet.json#/$defs/ColumnLineageDatasetFacet>", + "fields":{ + "merchant_id":{ + "inputFields":[ + { + "namespace":"<s3a://datalake>", + "name":"/test/merchants", + "field":"id" + } + ] + }, + "merchant_name":{ + "inputFields":[ + { + "namespace":"<s3a://datalake>", + "name":"/test/merchants", + "field":"name" + } + ] + }, + "activated":{ + "inputFields":[ + { + "namespace":"<s3a://datalake>", + "name":"/test/merchants", + "field":"activated" + } + ] + }, + "parent_id":{ + "inputFields":[ + { + "namespace":"<s3a://datalake>", + "name":"/test/merchants", + "field":"parent_id" + } + ] + }, + "contact_name":{ + "inputFields":[ + { + "namespace":"<s3a://datalake>", + "name":"/test/merchant_details", + "field":"contact_name" + } + ] + }, + "contact_email":{ + "inputFields":[ + { + "namespace":"<s3a://datalake>", + "name":"/test/merchant_details", + "field":"contact_email" + } + ] + } + } + }, + "symlinks":{ + "_producer":"<https://github.com/OpenLineage/OpenLineage/tree/0.30.1/integration/spark>", + "_schemaURL":"<https://openlineage.io/spec/facets/1-0-0/SymlinksDatasetFacet.json#/$defs/SymlinksDatasetFacet>", + "identifiers":[ + { + "namespace":"/warehouse/test.db", + "name":"test.merchant_md", + "type":"TABLE" + }

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Gaurav Singh + (gaurav.singh@razorpay.com) +
+
2023-08-09 08:23:57
 +
+

*Thread Reply:* "contact_name":{ + "inputFields":[ + { + "namespace":"<s3a://datalake>", + "name":"/test/merchant_details", + "field":"contact_name" + } + ] + } +This is returning mapping from the s3 location on which the table is created.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Zahi Fail + (zahi.fail@gmail.com) +
+
2023-08-09 10:56:27
 +
+

Hey, +I’m running Spark application (spark version 3.4) with OL integration. +I changed spark to use “debug” level, and I see the OL events with the below message: +“Emitting lineage completed successfully:”

+ +

With all the above, I can’t see the event in Marquez.

+ +

Attaching the OL configurations. +When changing the OL-spark version to 0.6.+, I do see event created in Marquez with only “Start” status (attached below).

+ +

The OL-spark version is matching the Spark version? Is there a known issues with the Spark / OL versions ?

+ +
+
+ + + + + + +
+
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-09 11:23:42
 +
+

*Thread Reply:* > OL-spark version to 0.6.+ +This OL version is ancient. You can try with 1.0.0

+ +

I think you're hitting this issue which duplicates jobs: https://github.com/OpenLineage/OpenLineage/issues/1943

+
+ + + + +
#1943 spark: two jobs operating on different datasets in same spark file can be named the same
+
+ <p>Currently, a Spark job name is created from action name which is misleading as different actions run with the same name can be writing to different output datasets. So, although the jobs are different, they end up within same graph nodes blurring the lineage.</p> + +<p>The solution for this would be to: append dataset name to the end of the job, the same way in which it is actually done for databricks platform.</p> +
+ +
+
Assignees
+ <a href="https://github.com/pawel-big-lebowski">@pawel-big-lebowski</a> +
+ +
+
Labels
+ integration/spark +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Zahi Fail + (zahi.fail@gmail.com) +
+
2023-08-10 01:46:08
 +
+

*Thread Reply:* I haven’t mentioned that I tried multiple OL versions - 1.0.0 / 0.30.1 / 0.6.+ … +None of them worked for me. +@Maciej Obuchowski

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-10 05:25:49
 +
+

*Thread Reply:* @Zahi Fail understood. Can you provide sample job that reproduces this behavior, and possibly some logs?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-10 05:26:11
 +
+

*Thread Reply:* If you can, it might be better to create issue at github and communicate there.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Zahi Fail + (zahi.fail@gmail.com) +
+
2023-08-10 08:34:01
 +
+

*Thread Reply:* Before creating an issue in GIT, I wanted to check if my issue only related to versions compatibility..

+ +

This is the sample of my test: +```from pyspark.sql import SparkSession +from pyspark.sql.functions import col

+ +

spark = SparkSession.builder\ + .config('spark.jars.packages', 'io.openlineage:openlineage_spark:1.0.0') \ + .config('spark.extraListeners', 'io.openlineage.spark.agent.OpenLineageSparkListener') \ + .config('spark.openlineage.host', 'http://localhost:9000') \ +.config('spark.openlineage.namespace', 'default') \ + .getOrCreate()

+ +

spark.sparkContext.setLogLevel("DEBUG")

+ +

csv_file = location.csv

+ +

df = spark.read.format("csv").option("header","true").option("sep","^").load(csv_file)

+ +

df = df.select("campaignid","revenue").groupby("campaignid").sum("revenue").show()``` +Part of the logs with the OL configurations and the processed event

+ +
+
+ + + + + + +
+
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-10 08:40:13
 +
+

*Thread Reply:* try spark.openlineage.transport.url instead of spark.openlineage.host

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-10 08:40:27
 +
+

*Thread Reply:* and possibly link the doc where you've seen spark.openlineage.host 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Zahi Fail + (zahi.fail@gmail.com) +
+
2023-08-10 08:59:27
 +
+

*Thread Reply:* https://openlineage.io/blog/openlineage-spark/

+
+
openlineage.io
+ + + +
Tracing Data Lineage with OpenLineage and Apache Spark | OpenLineage
+
+ Spark ushered in a brand new age of data democratization... and left us with a mess of hidden dependencies, stale datasets, and failed jobs. +
+ + + + + + +
Original URL: https://openlineage.io/blog/openlineage-spark/
+ + + +
+ +
+
+ + + + + + +
+ + +
+ 👍 Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Zahi Fail + (zahi.fail@gmail.com) +
+
2023-08-10 09:04:56
 +
+

*Thread Reply:* changing to “spark.openlineage.transport.url” didn’t make any change

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-10 09:09:42
 +
+

*Thread Reply:* do you see the ConsoleTransport log? it suggests Spark integration did not register that you want to send events to Marquez

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-10 09:10:09
 +
+

*Thread Reply:* let's try adding spark.openlineage.transport.type to http

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Zahi Fail + (zahi.fail@gmail.com) +
+
2023-08-10 09:14:50
 +
+

*Thread Reply:* Now it works !

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Zahi Fail + (zahi.fail@gmail.com) +
+
2023-08-10 09:14:58
 +
+

*Thread Reply:* thanks @Maciej Obuchowski

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-10 09:23:04
 +
+

*Thread Reply:* Cool 🙂 however it should not require it if you provide spark.openlineage.transport.url - I'll create issue for debugging that.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-08-09 14:37:24
 +
+

@channel +This month’s TSC meeting is tomorrow! All are welcome. https://openlineage.slack.com/archives/C01CK9T7HKR/p1691422200847979

+
+ + +
+ + + } + + Michael Robinson + (https://openlineage.slack.com/team/U02LXF3HUN7) +
+ + + +
+
+ <p>@channel +This month’s TSC meeting is this Thursday, August 10th at 10:00 a.m. PT. On the tentative agenda: +• announcements +• recent releases +• Airflow provider progress update +• OpenLineage 1.0 overview +• open discussion +• more (TBA) +More info and the meeting link can be found on the <a href="https://openlineage.io/meetings/">website</a>. All are welcome! Also, feel free to reply or DM me with discussion topics, agenda items, etc.</p> +
+ + + + + + +
Original URL: https://openlineage.slack.com/archives/C01CK9T7HKR/p1691422200847979
+ + +
+ + Slack Conversation +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Athitya Kumar + (athityakumar@gmail.com) +
+
2023-08-10 02:11:07
 +
+

While using the spark integration, we're unable to see the query in the job facet for any spark-submit - is this a known issue/limitation, and can someone point to the code where this is currently extracted / can be enhanced?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-08-10 02:55:46
 +
+

*Thread Reply:* Let me first rephrase my understanding of the question assume a user runs spark.sql('INSERT INTO ...'). Are we able to include sql queryINSERT INTO ...within SQL facet?

+ +

We once had a look at it and found it difficult. Given an SQL, spark immediately translates it to a logical plan (which our integration is based on) and we didn't find any place where we could inject our code and get access to sql being run.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Athitya Kumar + (athityakumar@gmail.com) +
+
2023-08-10 04:27:51
 +
+

*Thread Reply:* Got it. So for spark.sql() - there's no interaction with sqlparser-rs and we directly try stitching the input/output & column lineage from the spark logical plan. Would something like this fall under the spark.jdbc() route or the spark.sql() route (say, if the df is collected / written somewhere)?

+ +

val df = spark.read.format("jdbc") + .option("url", url) + .option("user", user) + .option("password", password) + .option("fetchsize", fetchsize) + .option("driver", driver)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-10 05:15:17
 +
+

*Thread Reply:* @Athitya Kumar I understand your issue. From my side, there's one problem with this - potentially there can be multiple queries for one spark job. You can imagine something like joining results of two queries - possible to separate systems - and then one SqlJobFacet would be misleading. This needs more thorough spec discussion

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Luigi Scorzato + (luigi.scorzato@gmail.com) +
+
2023-08-10 05:33:47
 +
+

Hi Team, has anyone experience with integrating OpenLineage with the SAP ecosystem? And with Salesforce/MuleSoft?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Steven + (xli@zjuici.com) +
+
2023-08-10 05:40:47
 +
+

Hi, +Are there any ways to save list of string directly in the dataset facets? Such as the myfacets field in this dict +"facets": { + "metadata_facet": { + "_producer": "<https://github.com/OpenLineage/OpenLineage/tree/0.29.2/client/python>", + "_schemaURL": "<https://sth/schemas/facets.json#/definitions/SomeFacet>", + "myfacets": ["a", "b", "c"] + } + }

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Steven + (xli@zjuici.com) +
+
2023-08-10 05:42:20
 +
+

*Thread Reply:* I'm using python OpenLineage package and extend the BaseFacet class

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-10 05:53:57
 +
+

*Thread Reply:* for custom facets, as long as it's valid json - go for it

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Steven + (xli@zjuici.com) +
+
2023-08-10 05:55:03
 +
+

*Thread Reply:* However I tried to insert a list of string. And I tried to get the dataset, the returned valued of that list field is empty.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Steven + (xli@zjuici.com) +
+
2023-08-10 05:55:57
 +
+

*Thread Reply:* @attr.s +class MyFacet(BaseFacet): + columns: list[str] = attr.ib() +Here's my python code.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-10 05:59:02
 +
+

*Thread Reply:* How did you emit, serialized the event, and where did you look when you said you tried to get the dataset?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-10 06:00:27
 +
+

*Thread Reply:* I assume the problem is somewhere there, not on the level of facet definition, since SchemaDatasetFacet looks pretty much the same and it works

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Steven + (xli@zjuici.com) +
+
2023-08-10 06:00:54
 +
+

*Thread Reply:* I use the python openlineage client to emit the RunEvent. +openlineage_client.emit( + RunEvent( + eventType=RunState.COMPLETE, + eventTime=datetime.now().isoformat(), + run=run, + job=job, + producer=PRODUCER, + outputs=outputs, + ) + ) +And use marquez to visualize the get data result

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Steven + (xli@zjuici.com) +
+
2023-08-10 06:02:12
 +
+

*Thread Reply:* Yah, list of objects is working, but list of string is not.😩

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Steven + (xli@zjuici.com) +
+
2023-08-10 06:03:23
 +
+

*Thread Reply:* I think the problem is related to the openlineage package openlineage.client.serde.py. The function Serde.to_json()

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Steven + (xli@zjuici.com) +
+
2023-08-10 06:05:56
 +
+

*Thread Reply:*

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Steven + (xli@zjuici.com) +
+
2023-08-10 06:19:34
 +
+

*Thread Reply:* I think the code here filters out those string values in the list

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-10 06:21:39
 +
+

*Thread Reply:* 👀

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Steven + (xli@zjuici.com) +
+
2023-08-10 06:24:48
 +
+

*Thread Reply:* Yah, the value in list will end up False in this code and be filtered out +isinstance(_x_, dict)

+ +

😳

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-10 06:26:33
 +
+

*Thread Reply:* wow, that's right 😬

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-10 06:26:47
 +
+

*Thread Reply:* want to create PR fixing that?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Steven + (xli@zjuici.com) +
+
2023-08-10 06:27:20
 +
+

*Thread Reply:* Sure! May do this later tomorrow.

+ + + +
+ 👍 Maciej Obuchowski, Paweł Leszczyński +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Steven + (xli@zjuici.com) +
+
2023-08-10 23:59:28
 +
+

*Thread Reply:* I created the pr at https://github.com/OpenLineage/OpenLineage/pull/2044 +But the ci on integration-test-integration-spark FAILED

+
+ + + + +
#2044 fix: serde filtering
+
+ <p><strong>Problem</strong></p> + +<p>Values in list objects will be accidently filtered out so that they will not be included in the output.</p> + +<p><strong>Solution</strong></p> + +<p>Remove unnecessary instance type check (which filters out single value by mistake) in Serde.</p> + +<blockquote> + <p><strong>Note:</strong> All schema changes require discussion. Please <a href="https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue">link the issue</a> for context.</p> +</blockquote> + +<p>☐ Your change modifies the <a href="https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.json">core</a> OpenLineage model +☐ Your change modifies one or more OpenLineage <a href="https://github.com/OpenLineage/OpenLineage/tree/main/spec/facets">facets</a></p> + +<p><strong>One-line summary:</strong></p> + +<p>Fix the bug that causes values in list objects being accidently filtered.</p> + +<p><strong>Checklist</strong></p> + +<p>☐ You've <a href="https://github.com/OpenLineage/OpenLineage/blob/main/why-the-dco.md">signed-off</a> your work +☐ Your pull request title follows our <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CONTRIBUTING.md#creating-pull-requests">guidelines</a></p> + +<hr /> + +<p>SPDX-License-Identifier: Apache-2.0 <br /> +Copyright 2018-2023 contributors to the OpenLineage project</p> +
+ +
+
Labels
+ client/python +
+ +
+
Comments
+ 1 +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-11 04:17:01
 +
+

*Thread Reply:* @Steven sorry for that - some tests require credentials that are not present on the forked versions of CI. It will work once I push it to origin. Anyway Spark tests failing aren't blocker for this Python PR

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-11 04:17:45
 +
+

*Thread Reply:* I would only ask to add some tests for that case with facets containing list of string

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Steven + (xli@zjuici.com) +
+
2023-08-11 04:18:21
 +
+

*Thread Reply:* Yeah sure, I will add them now

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-11 04:25:19
 +
+

*Thread Reply:* ah we had other CI problem, go version was too old in one of the jobs - neverthless I won't judge your PR on stuff failing outside your PR anyway 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Steven + (xli@zjuici.com) +
+
2023-08-11 04:36:57
 +
+

*Thread Reply:* LOL🤣 I've added some tests and made a force push

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
savan + (SavanSharan_Navalgi@intuit.com) +
+
2023-10-20 08:31:45
 +
+

*Thread Reply:* @GitHubOpenLineageIssues +I am trying to contribute to Integration tests which is listed here as good first issue +the CONTRIBUTING.md mentions that i can trigger CI for integration tests from forked branch. +using this tool. +but i am unable to do so, is there a way to trigger CI from forked brach or do i have to get permission from someone to run the CI?

+ +

i am getting this error when i run this command sudo git-push-fork-to-upstream-branch upstream savannavalgi:hacktober +> Username for '<https://github.com>': savannavalgi +&gt; Password for '<https://savannavalgi@github.com>': +&gt; remote: Permission to OpenLineage/OpenLineage.git denied to savannavalgi. +&gt; fatal: unable to access '<https://github.com/OpenLineage/OpenLineage.git/>': The requested URL returned error: 403 +i have tried to configure ssh key +also tried to trigger CI from another brach, +and tried all of this after fetching the latest upstream

+ +

cc: @Athitya Kumar @Maciej Obuchowski @Steven

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-10-23 04:57:44
 +
+

*Thread Reply:* what PR is the probem related to? I can run git-push-fork-to-upstream-branch for you

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
savan + (SavanSharan_Navalgi@intuit.com) +
+
2023-10-25 01:08:41
 +
+

*Thread Reply:* @Paweł Leszczyński thanks for approving my PR - ( link )

+ +

I will make the changes needed for the new integration test case for drop table (good first issue) , in another PR, +I would need your help to run the integration tests again, thank you

+
+ + + + +
#2143 Integration test for drop table does not verify anything
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
savan + (SavanSharan_Navalgi@intuit.com) +
+
2023-10-26 07:48:52
 +
+

*Thread Reply:* @Paweł Leszczyński +opened a PR ( link ) for integration test for drop table +can you please help run the integration test

+
+ + + + +
#2214 spark: integration test case for drop table
+
+ <p><strong>Problem</strong></p> + +<p>drop table integration test did not verify the drop table events generated by openlineage.</p> + +<p>Closes: <a href="https://github.com/OpenLineage/OpenLineage/issues/2143">#2143</a></p> + +<p><strong>Solution</strong></p> + +<p>for drop table the COMPLETE event generated has empty inputs and outputs facet.</p> + +<p>but the START event has the table to be dropped/deleted in the outputs facet and a empty inputs facet.</p> + +<p><strong>One-line summary:</strong></p> + +<p>added START event for the drop table integration test</p> + +<p><strong>Checklist</strong></p> + +<p>☑︎ You've <a href="https://github.com/OpenLineage/OpenLineage/blob/main/why-the-dco.md">signed-off</a> your work +☑︎ Your pull request title follows our <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CONTRIBUTING.md#creating-pull-requests">guidelines</a> +☑︎ Your changes are accompanied by tests (<em>if relevant</em>) +☑︎ Your change contains a <a href="https://kurtisnusbaum.medium.com/stacked-diffs-keeping-phabricator-diffs-small-d9964f4dcfa6">small diff</a> and is self-contained +☑︎ You've updated any relevant documentation (<em>if relevant</em>) +☑︎ Your comment includes a one-liner for the changelog about the specific purpose of the change (<em>if necessary</em>) +☐ You've versioned the core OpenLineage model or facets according to <a href="https://docs.snowplowanalytics.com/docs/pipeline-components-and-applications/iglu/common-architecture/schemaver">SchemaVer</a> (<em>if relevant</em>) +☐ You've added a <a href="https://github.com/OpenLineage/OpenLineage/tree/main/.github/header_templates.md">header</a> to source files (<em>if relevant</em>)</p> + +<hr /> + +<p>SPDX-License-Identifier: Apache-2.0 <br /> +Copyright 2018-2023 contributors to the OpenLineage project</p> +
+ +
+
Labels
+ documentation, integration/spark +
+ +
+
Comments
+ 1 +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-10-26 07:50:29
 +
+

*Thread Reply:* sure, some of our tests require access to S3/BigQuery secret keys, so will not work automatically from the fork, and require action on our side. working on that

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
savan + (SavanSharan_Navalgi@intuit.com) +
+
2023-10-29 09:31:22
 +
+

*Thread Reply:* thanks @Paweł Leszczyński +let me know if i can help in any way

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
savan + (SavanSharan_Navalgi@intuit.com) +
+
2023-11-15 02:31:50
 +
+

*Thread Reply:* @Paweł Leszczyński any action item on my side?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Athitya Kumar + (athityakumar@gmail.com) +
+
2023-08-11 07:36:57
 +
+

Hey folks! 👋

+ +

Had a query/observation regarding columnLineage inferred in spark integration - opened this issue for the same. Basically, when we do something like this in our spark-sql: +SELECT t1.c1, t1.c2, t1.c3, t2.c4 FROM t1 LEFT JOIN t2 ON t1.c1 = t2.c1 AND t1.c2 = t2.c2 +The expected column lineage for output table t3 is: +t3.c1 -&gt; Comes from both t1.c1 &amp; t2.c1 (SELECT + JOIN clause) +t3.c2 -&gt; Comes from both t1.c2 &amp; t2.c2 (SELECT + JOIN clause) +t3.c3 -&gt; Comes from t1.c3 +t3.c4 -&gt; Comes from t2.c4 +However, actual column lineage for output table t3 is: +t3.c1 -&gt; Comes from t1.c1 (Only based on SELECT clause) +t3.c2 -&gt; Comes from t1.c1 (Only based on SELECT clause) +t3.c3 -&gt; Comes from t1.c3 +t3.c4 -&gt; Comes from t2.c4 +Is this a known issue/behaviour?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-11 09:18:44
 +
+

*Thread Reply:* Hmm... this is kinda "logical" difference - is column level lineage taken from actual "physical" operations - like in this case, we always take from t1 - or from "logical" where t2 is used only for predicate, yet we still want to indicate it as a source?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-11 09:18:58
 +
+

*Thread Reply:* I think your interpretation is more useful

+ + + +
+ 🙏 Athitya Kumar +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Athitya Kumar + (athityakumar@gmail.com) +
+
2023-08-11 09:25:03
 +
+

*Thread Reply:* @Maciej Obuchowski - Yup, especially for use-cases where we wanna depend on column lineage for impact analysis, I think we should be considering even predicates. For example, if t2.c1 / t2.c2 gets corrupted or dropped, the query would be impacted - which means that we should be including even predicates (t2.c1 / t2.c2) in the column lineage imo

+ +

But is there any technical limitation if we wanna implement this / make an OSS contribution for this (like logical predicate columns not being part of the spark logical plan object that we get in the PlanVisitor or something like that)?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-11 11:14:58
 +
+

*Thread Reply:* It's probably a bit of work, but can't think it's impossible on parser side - @Paweł Leszczyński will know better about spark collection

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Ernie Ostic + (ernie.ostic@getmanta.com) +
+
2023-08-11 12:45:34
 +
+

*Thread Reply:* This is a case where it would be nice to have an alternate indication (perhaps in the Column lineage facet?) for this type of "suggested" lineage. As noted, this is especially important for impact analysis purposes. We (and I believe others do the same or similar) call that "indirect" lineage at Manta.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-11 12:49:10
 +
+

*Thread Reply:* Something like additional flag in inputFields, right?

+ + + +
+ 👍 Athitya Kumar, Ernie Ostic, Paweł Leszczyński +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-08-14 02:36:34
 +
+

*Thread Reply:* Yes, this would require some extension to the spec. What do you mean spark-sql : spark.sql() with some spark query or SQL in spark JDBC?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Athitya Kumar + (athityakumar@gmail.com) +
+
2023-08-15 15:16:49
 +
+

*Thread Reply:* Sorry, missed your question @Paweł Leszczyński. By spark-sql, I'm referring to the former: spark.sql() with some spark query

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-08-16 03:10:57
 +
+

*Thread Reply:* cc @Jens Pfau - you may be also interested in extending column level lineage facet.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-08-22 02:23:08
 +
+

*Thread Reply:* Hi, is there a github issue for this feature? Seems like a really cool and exciting functionality to have!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Athitya Kumar + (athityakumar@gmail.com) +
+
2023-08-22 08:03:49
 +
+

*Thread Reply:* @Anirudh Shrinivason - Are you referring to this issue: https://github.com/OpenLineage/OpenLineage/issues/2048?

+
+ + + + +
#2048 Spark: Column lineage is not taking join fields into consideration for `inputFields` of an output field
+
+ <p>Hey folks! 👋</p> + +<p>Had a query/observation regarding columnLineage inferred in spark integration. When we do something like this in our spark-sql:</p> + +<pre><code>SELECT t1.c1, t1.c2, t1.c3, t2.c4 FROM t1 LEFT JOIN t2 ON t1.c1 = t2.c1 AND t1.c2 = t2.c2 +</code></pre> + +<p>Expected column lineage for output table t3 is:</p> + +<pre><code>t3.c1 -&gt; Comes from both t1.c1 &amp; t2.c1 (SELECT + JOIN clause) +t3.c2 -&gt; Comes from both t1.c2 &amp; t2.c2 (SELECT + JOIN clause) +t3.c3 -&gt; Comes from t1.c3 +t3.c4 -&gt; Comes from t2.c4 +</code></pre> + +<p>However, the actual column lineage for output table t3 is:</p> + +<pre><code>t3.c1 -&gt; Comes from t1.c1 (Only based on SELECT clause) +t3.c2 -&gt; Comes from t1.c1 (Only based on SELECT clause) +t3.c3 -&gt; Comes from t1.c3 +t3.c4 -&gt; Comes from t2.c4 +</code></pre> + +<p>Is this a known issue/behaviour?</p> +
+ +
+
Comments
+ 2 +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+ :gratitude_thank_you: Anirudh Shrinivason +
+ +
+ ✅ Anirudh Shrinivason +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Athitya Kumar + (athityakumar@gmail.com) +
+
2023-08-14 05:13:48
 +
+

Hey team 👋

+ +

Is there a way we can feed the logical plan directly to check the open-lineage events being built, without actually running a spark-job with open-lineage configs? Basically interested to see if we can mock a dry-run of a spark job w/ open-lineage by mimicking the logical plan 😄

+ +

cc @Shubh

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-14 06:00:21
 +
+

*Thread Reply:* Not really I think - the integration does not rely purely on the logical plan

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-14 06:00:44
 +
+

*Thread Reply:* At least, not in all cases. For some maybe

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-08-14 07:34:39
 +
+

*Thread Reply:* We're using pretty similar approach in our column level lineage tests where we run some spark commands, register custom listener https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/app/src/tes[…]eage/spark/agent/util/LastQueryExecutionSparkEventListener.java which catches the logical plan. Further we run our tests on the captured logical plan.

+ +

The difference here, between what you're asking about, is that we still have an access to the same spark session.

+ +

In many cases, our integration uses active Spark session to fetch some dataset details. This happens pretty often (like fetch dataset location) and cannot be taken just from a Logical Plan.

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/app/src/test/java/io/openlineage/spark/agent/util/LastQueryExecutionSparkEventListener.java | LastQueryExecutionSparkEventListener.java>
+
+ <pre><code>/** +/** Copyright 2018-2022 contributors to the OpenLineage project +/** SPDX-License-Identifier: Apache-2.0 +**/ + +package io.openlineage.spark.agent.util; + +import java.util.ArrayList; +import java.util.List; +import java.util.Optional; +import org.apache.spark.scheduler.SparkListener; +import org.apache.spark.scheduler.SparkListenerEvent; +import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan; +import org.apache.spark.sql.execution.QueryExecution; +import org.apache.spark.sql.execution.SQLExecution; +import org.apache.spark.sql.execution.ui.SparkListenerSQLExecutionStart; + +public class LastQueryExecutionSparkEventListener extends SparkListener { + + private static List&lt;LogicalPlan&gt; queryExecutions = new ArrayList&lt;&gt;(); + + @Override + public void onOtherEvent(SparkListenerEvent event) { + if (event instanceof SparkListenerSQLExecutionStart) { + QueryExecution queryExecution = + SQLExecution.getQueryExecution(((SparkListenerSQLExecutionStart) event).executionId()); + + if (queryExecution != null) { + queryExecutions.add(queryExecution.optimizedPlan()); + } + } + } + + public static Optional&lt;LogicalPlan&gt; getLastExecutedLogicalPlan() { + if (queryExecutions.isEmpty()) { + return Optional.empty(); + } else { + return Optional.of(queryExecutions.get(queryExecutions.size() - 1)); + } + } + + public static List&lt;LogicalPlan&gt; getExecutedLogicalPlans() { + return queryExecutions; + } +} + +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Athitya Kumar + (athityakumar@gmail.com) +
+
2023-08-14 11:03:28
 +
+

*Thread Reply:* @Paweł Leszczyński - We're mainly interested to see the inputs/outputs (mainly column schema and column lineage) for different logical plans. Is that something that could be done in a static manner without running spark jobs in your opinion?

+ +

For example, I know that we can statically create logical plans

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-08-16 03:05:44
 +
+

*Thread Reply:* The more we talk the more I am wondering what is the purpose of doing so? Do you want to test openlineage coverage or is there any production scenario where you would like to apply this?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Athitya Kumar + (athityakumar@gmail.com) +
+
2023-08-16 04:01:39
 +
+

*Thread Reply:* @Paweł Leszczyński - This is for testing openlineage coverage so that we can be more confident on what're the happy path scenarios and what're the scenarios where it may not work / work partially etc

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-08-16 04:22:01
 +
+

*Thread Reply:* If this is for testing, then you're also capable of mocking some SparkSession/catalog methods when Openlineage integration tries to access them. If you want to reuse LogicalPlans from your prod environment, you will encounter logicalplan serialization issues. On the other hand, if you generate logical plans from some example Spark jobs, then the same can be easier achieved in a way the integration tests are run with mockserver.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-08-14 09:45:31
 +
+

Hi Team,

+ +

Spark & Databricks related question: Starting 1st September Databricks is going to block running init_scripts located in dbfs and this is the way our integration works (https://www.databricks.com/blog/securing-databricks-cluster-init-scripts).

+ +

We have two ways of mitigating this in our docs and quickstart: + (1) move initscripts to workspace + (2) move initscripts to S3

+ +

None of them is perfect. (1) requires creating init_script file manually through databricks UI and copy/paste its content. I couldn't find the way to load it programatically. (2) requires quickstart user to have s3 bucket access.

+ +

Would love to hear your opinion on this. Perhaps there's some better way to do that. Thanks. `

+
+
Databricks
+ + + +
Securing Databricks cluster init scripts
+
+ Protecting the Databricks platform and continuously raising the bar with security improvements is the mission of our Security team and the main reason why we invest in our bug bounty program. +
+ + + + + + + + +
Original URL: https://www.databricks.com/blog/securing-databricks-cluster-init-scripts
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-08-15 01:13:49
 +
+

*Thread Reply:* We're uploading the init scripts to s3 via tf. But yeah ig there are some access permissions that the user needs to have

+ + + +
+ :gratitude_thank_you: Paweł Leszczyński +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Abdallah + (abdallah@terrab.me) +
+
2023-08-16 07:32:00
 +
+

*Thread Reply:* Hello +I am new here and I am asking why do you need an init script ? +If it's a spark integration we can just specify --package=io.openlineage...

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-08-16 07:41:25
 +
+

*Thread Reply:* https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/databricks/open-lineage-init-script.sh -> I think the issue was in having openlineage-jar installed immediately on the classpath bcz it's required when OpenLineageSparkListener is instantiated. It didn't work without it.

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/databricks/open-lineage-init-script.sh | open-lineage-init-script.sh>
+
+ <pre><code>#!/bin/bash +# +# Copyright 2018-2023 contributors to the OpenLineage project +# SPDX-License-Identifier: Apache-2.0 + +STAGE_DIR="/dbfs/databricks/openlineage" + +echo "BEGIN: Upload Spark Listener JARs" +cp -f $STAGE_DIR/openlineage-spark-**.jar /mnt/driver-daemon/jars || { echo "Error copying Spark Listener library file"; exit 1;} +echo "END: Upload Spark Listener JARs" + +echo "BEGIN: Modify Spark config settings" +cat &lt;&lt; 'EOF' &gt; /databricks/driver/conf/openlineage-spark-driver-defaults.conf +[driver] { + "spark.extraListeners" = "io.openlineage.spark.agent.OpenLineageSparkListener" +} +EOF +echo "END: Modify Spark config settings" + +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Abdallah + (abdallah@terrab.me) +
+
2023-08-16 07:43:55
 +
+

*Thread Reply:* Yes it happens if you use --jars s3://.../...openlineage-spark-VERSION.jar parameter. (I made a ticket for this issue in Databricks support) +But if you use --package io.openlineage... (the package will be downloaded from maven) it works fine.

+ + + +
+ 👀 Paweł Leszczyński +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Abdallah + (abdallah@terrab.me) +
+
2023-08-16 07:47:50
 +
+

*Thread Reply:* I think they don't use the right class loader.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-08-16 08:36:14
 +
+

*Thread Reply:* To make sure: are you able to run Openlineage & Spark on Databricks Runtime without init_scripts?

+ +

I was doing this a second ago and this ended up with Caused by: java.lang.ClassNotFoundException: io.openlineage.spark.agent.OpenLineageSparkListener not found in com.databricks.backend.daemon.driver.ClassLoaders$LibraryClassLoader@1609ed55

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Alexandre Campelo + (aleqi200@gmail.com) +
+
2023-08-14 19:49:00
 +
+

Hello, I just downloaded Marquez and I'm trying to send a sample request but I'm getting a 403 (forbidden). Any idea how to find the authentication details?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Alexandre Campelo + (aleqi200@gmail.com) +
+
2023-08-15 12:19:34
 +
+

*Thread Reply:* Ok, nevermind. I figured it out. The port 5000 is reserved in MACOS so I had to start on port 9000 instead.

+ + + +
+ 👍 Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-08-15 01:25:48
 +
+

Hi, I noticed that while capturing lineage for merge into commands, some of the tables/columns are unaccounted for the lineage. Example: +```fdummyfunnelstg = spark.sql("""WITH dummyfunnel AS ( + SELECT ** + FROM fdummyfunnelone + WHERE dateid BETWEEN {startdateid} AND {enddateid}

+ +
        UNION ALL
+
+        SELECT **
+        FROM f_dummy_funnel_two
+        WHERE date_id BETWEEN {start_date_id} AND {end_date_id}
+
+        UNION ALL
+
+        SELECT **
+        FROM f_dummy_funnel_three
+        WHERE date_id BETWEEN {start_date_id} AND {end_date_id}
+
+        UNION ALL
+
+        SELECT **
+        FROM f_dummy_funnel_four
+        WHERE date_id BETWEEN {start_date_id} AND {end_date_id}
+
+        UNION ALL
+
+        SELECT **
+        FROM f_dummy_funnel_five
+        WHERE date_id BETWEEN {start_date_id} AND {end_date_id}
+
+    )
+    SELECT DISTINCT
+        dummy_funnel.customer_id,
+        dummy_funnel.product,
+        dummy_funnel.date_id,
+        dummy_funnel.country_id,
+        dummy_funnel.city_id,
+        dummy_funnel.dummy_type_id,
+        dummy_funnel.num_attempts,
+        dummy_funnel.num_transactions,
+        dummy_funnel.gross_merchandise_value,
+        dummy_funnel.sub_category_id,
+        dummy_funnel.is_dummy_flag
+    FROM dummy_funnel
+    INNER JOIN d_dummy_identity as dummy_identity
+        ON dummy_identity.id = dummy_funnel.customer_id
+    WHERE
+        date_id BETWEEN {start_date_id} AND {end_date_id}""")
+
+ +

spark.sql(f""" + MERGE INTO {tablename} + USING fdummyfunnelstg + ON + fdummyfunnelstg.customerid = {tablename}.customerid + AND fdummyfunnelstg.product = {tablename}.product + AND fdummyfunnelstg.dateid = {tablename}.dateid + AND fdummyfunnelstg.countryid = {tablename}.countryid + AND fdummyfunnelstg.cityid = {tablename}.cityid + AND fdummyfunnelstg.dummytypeid = {tablename}.dummytypeid + AND fdummyfunnelstg.subcategoryid = {tablename}.subcategoryid + AND fdummyfunnelstg.isdummyflag = {tablename}.isdummyflag + WHEN MATCHED THEN + UPDATE SET + {tablename}.numattempts = fdummyfunnelstg.numattempts + , {tablename}.numtransactions = fdummyfunnelstg.numtransactions + , {tablename}.grossmerchandisevalue = fdummyfunnelstg.grossmerchandisevalue + WHEN NOT MATCHED + THEN INSERT ( + customerid, + product, + dateid, + countryid, + cityid, + dummytypeid, + numattempts, + numtransactions, + grossmerchandisevalue, + subcategoryid, + isdummyflag + ) + VALUES ( + fdummyfunnelstg.customerid, + fdummyfunnelstg.product, + fdummyfunnelstg.dateid, + fdummyfunnelstg.countryid, + fdummyfunnelstg.cityid, + fdummyfunnelstg.dummytypeid, + fdummyfunnelstg.numattempts, + fdummyfunnelstg.numtransactions, + fdummyfunnelstg.grossmerchandisevalue, + fdummyfunnelstg.subcategoryid, + fdummyfunnelstg.isdummyflag + ) + """)`` +In cases like this, I notice that the full lineage is not actually captured... I'd expect to see this having 5 upstreams:dummyfunnelone, dummyfunneltwo, dummyfunnelthree, dummyfunnelfour, dummyfunnel_five` , but I notice only 1-2 upstreams for this case... +Would like to learn more about why this might happen, and whether this is expected behaviour or not. Thanks!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-15 06:48:43
 +
+

*Thread Reply:* Would be useful to see generated event or any logs

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-08-16 03:09:05
 +
+

*Thread Reply:* @Anirudh Shrinivason what if there is just one union instead of four? What if there are just two columns selected instead of 10? What if inner join is skipped? Does merge into matter?

+ +

The smaller SQL to reproduce the problem, the easier it is to find the root cause. Most of the issues are reproducible with just few lines of code.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-08-16 03:34:30
 +
+

*Thread Reply:* Yup let me try to identify the cause from my end. Give me some time haha. I'll reach out again once there is more clarity on the occurence

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Abdallah + (abdallah@terrab.me) +
+
2023-08-16 07:33:21
 +
+

Hello,

+ +

The OpenLineage Databricks integration is not working properly in our side which due to filtering adaptive_spark_plan

+ +

Please find the issue link.

+ +

https://github.com/OpenLineage/OpenLineage/issues/2058

+
+ + + + +
#2058 Issue with Event Filtering in Databricks Integration for `adaptive_spark_plan`
+
+ <p>Hello OpenLineage team,</p> + +<p>While implementing the current OpenLineage Spark integration in Databricks, we've made an interesting observation regarding event filtering and its behavior with narrow and wide transformations:</p> + +<ol><li>For a <strong>narrow transformation</strong> (without shuffle), the event is <code>execute_insert_into_hadoop_fs_relation_command</code> and it doesn't include <code>adaptive_spark_plan</code>.</li><li>For a <strong>wide transformation</strong> (with shuffle), the event is <code>adaptive_spark_plan</code> and doesn't have <code>execute_insert_into_hadoop_fs_relation_command</code>.</li> +</ol> + +<p>We've identified that the <code>adaptive_spark_plan</code> is actively filtered out in the <code>DatabricksEventFilter</code> class. This results in the event being completely ignored due to this line:</p> + +<p><a href="https://github.com/OpenLineage/OpenLineage/blob/41dbbf1799595bd9cd1567df0a7027de08619741/integration/spark/shared/src/main/java/io/openlineage/spark/agent/filters/DatabricksEventFilter.java#L30">OpenLineage/integration/spark/shared/src/main/java/io/openlineage/spark/agent/filters/DatabricksEventFilter.java</a></p> + +<p>Line 30 in </OpenLineage/OpenLineage/commit/41dbbf1799595bd9cd1567df0a7027de08619741|41dbbf1></p> + +<p>To validate, we compiled a JAR without this filtering line, and the integration worked perfectly from Databricks.</p> + +<p><strong>Example Spark Jobs:</strong></p> + +<ol><li><strong>Narrow Transformation</strong> +<pre><code>val data = Seq((1,"a"),(2,"b"),(3,"c")) +val rdd = spark.sparkContext.parallelize(data) +val df = rdd.toDF("id", "value") +df.write.parquet("/path/to/output/narrow_transformation") +</code></pre></li><li><strong>Wide Transformation</strong> +<pre><code>val data = Seq((1,"a"),(2,"b"),(3,"c"),(1,"d")) +val rdd = spark.sparkContext.parallelize(data) +val df = rdd.toDF("id", "value") +df.groupBy("id").count().write.parquet("/path/to/output/wide_transformation") +</code></pre></li> +</ol> + +<p><strong>Proposed Solutions</strong>:</p> + +<p>• Short-term fix: Remove adaptive<em>spark</em>plan from the excludedNodes in DatabricksEventFilter. +• Long-term resolution: Consider integrating the proposal from <a href="https://github.com/OpenLineage/OpenLineage/issues/2056">#2056</a> for a more universal filtering mechanism.</p> + +<p>Environment: <br /> +Databricks version: 12.2 LTS (includes Apache Spark 3.3.2, Scala 2.12)</p> + +<p>I hope this provides a clear picture of the issue at hand. We're eager to collaborate on resolving this.</p> + +<p>Thank you!</p> + +<p>Best regards,</p> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+ ⬆️ Mouad MOUSSABBIH, Abdallah +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harel Shein + (harel.shein@gmail.com) +
+
2023-08-16 09:24:09
 +
+

*Thread Reply:* thanks @Abdallah for the thoughtful issue that you submitted! +was wondering if you’d consider opening up a PR? would love to help you as a contributor is that’s something you are interested in.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Abdallah + (abdallah@terrab.me) +
+
2023-08-17 11:59:51
 +
+

*Thread Reply:* Hello

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Abdallah + (abdallah@terrab.me) +
+
2023-08-17 11:59:58
 +
+

*Thread Reply:* Yes I am working on it

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Abdallah + (abdallah@terrab.me) +
+
2023-08-17 12:00:14
 +
+

*Thread Reply:* I deleted the line that has that filter.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Abdallah + (abdallah@terrab.me) +
+
2023-08-17 12:00:24
 +
+

*Thread Reply:* I am adding some tests now

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Abdallah + (abdallah@terrab.me) +
+
2023-08-17 12:00:45
 +
+

*Thread Reply:* But running +./gradlew --no-daemon databricksIntegrationTest -x test -Pspark.version=3.4.0 -PdatabricksHost=$DATABRICKS_HOST -PdatabricksToken=$DATABRICKS_TOKEN

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Abdallah + (abdallah@terrab.me) +
+
2023-08-17 12:01:11
 +
+

*Thread Reply:* gives me +A problem occurred evaluating project ':app'. +&gt; Could not resolve all files for configuration ':app:spark33'. + &gt; Could not resolve io.openlineage:openlineage_java:1.1.0-SNAPSHOT. + Required by: + project :app &gt; project :shared + &gt; Could not resolve io.openlineage:openlineage_java:1.1.0-SNAPSHOT. + &gt; Unable to load Maven meta-data from <https://astronomer.jfrog.io/artifactory/maven-public-libs-snapshot/io/openlineage/openlineage-java/1.1.0-SNAPSHOT/maven-metadata.xml>. + &gt; org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 326; The reference to entity "display" must end with the ';' delimiter. + &gt; Could not resolve io.openlineage:openlineage_sql_java:1.1.0-SNAPSHOT. + Required by: + project :app &gt; project :shared + &gt; Could not resolve io.openlineage:openlineage_sql_java:1.1.0-SNAPSHOT. + &gt; Unable to load Maven meta-data from <https://astronomer.jfrog.io/artifactory/maven-public-libs-snapshot/io/openlineage/openlineage-sql-java/1.1.0-SNAPSHOT/maven-metadata.xml>. + &gt; org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 326; The reference to entity "display" must end with the ';' delimiter.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Abdallah + (abdallah@terrab.me) +
+
2023-08-17 12:01:25
 +
+

*Thread Reply:* And I am trying to understand what should I do.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Abdallah + (abdallah@terrab.me) +
+
2023-08-17 12:13:37
 +
+

*Thread Reply:* I am compiling sql integration

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Abdallah + (abdallah@terrab.me) +
+
2023-08-17 13:04:15
 +
+

*Thread Reply:* I built the java client

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Abdallah + (abdallah@terrab.me) +
+
2023-08-17 13:04:29
 +
+

*Thread Reply:* but having +A problem occurred evaluating project ':app'. +&gt; Could not resolve all files for configuration ':app:spark33'. + &gt; Could not resolve io.openlineage:openlineage_java:1.1.0-SNAPSHOT. + Required by: + project :app &gt; project :shared + &gt; Could not resolve io.openlineage:openlineage_java:1.1.0-SNAPSHOT. + &gt; Unable to load Maven meta-data from <https://astronomer.jfrog.io/artifactory/maven-public-libs-snapshot/io/openlineage/openlineage-java/1.1.0-SNAPSHOT/maven-metadata.xml>. + &gt; org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 326; The reference to entity "display" must end with the ';' delimiter. + &gt; Could not resolve io.openlineage:openlineage_sql_java:1.1.0-SNAPSHOT. + Required by: + project :app &gt; project :shared + &gt; Could not resolve io.openlineage:openlineage_sql_java:1.1.0-SNAPSHOT. + &gt; Unable to load Maven meta-data from <https://astronomer.jfrog.io/artifactory/maven-public-libs-snapshot/io/openlineage/openlineage-sql-java/1.1.0-SNAPSHOT/maven-metadata.xml>. + &gt; org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 326; The reference to entity "display" must end with the ';' delimiter.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-17 14:47:41
 +
+

*Thread Reply:* Please do ./gradlew publishToMavenLocal in client/java directory

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Abdallah + (abdallah@terrab.me) +
+
2023-08-17 14:47:59
 +
+

*Thread Reply:* Okay thanks

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Abdallah + (abdallah@terrab.me) +
+
2023-08-17 14:48:01
 +
+

*Thread Reply:* will do

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Abdallah + (abdallah@terrab.me) +
+
2023-08-22 10:33:02
 +
+

*Thread Reply:* Hello back

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Abdallah + (abdallah@terrab.me) +
+
2023-08-22 10:33:12
 +
+

*Thread Reply:* I created a databricks cluster.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Abdallah + (abdallah@terrab.me) +
+
2023-08-22 10:35:00
 +
+

*Thread Reply:* And I had somme issues that -PdatabricksHost doesn't work with System.getProperty("databricksHost") So I changed to -DdatabricksHost with System.getenv("databricksHost")

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Abdallah + (abdallah@terrab.me) +
+
2023-08-22 10:36:19
 +
+

*Thread Reply:* Then I had some issue that the path dbfs:/databricks/openlineage/ doesn't exist, I, then, created the folder /dbfs/databricks/openlineage/

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Abdallah + (abdallah@terrab.me) +
+
2023-08-22 10:38:03
 +
+

*Thread Reply:* And now I am investigating this issue : +java.lang.NullPointerException + at io.openlineage.spark.agent.DatabricksUtils.uploadOpenlineageJar(DatabricksUtils.java:226) + at io.openlineage.spark.agent.DatabricksUtils.init(DatabricksUtils.java:66) + at io.openlineage.spark.agent.DatabricksIntegrationTest.setup(DatabricksIntegrationTest.java:54) + at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) + at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) + at ... +worker.org.gradle.process.internal.worker.GradleWorkerMain.main(GradleWorkerMain.java:74) + Suppressed: com.databricks.sdk.core.DatabricksError: Missing required field: cluster_id + at app//com.databricks.sdk.core.error.ApiErrors.readErrorFromResponse(ApiErrors.java:48) + at app//com.databricks.sdk.core.error.ApiErrors.checkForRetry(ApiErrors.java:22) + at app//com.databricks.sdk.core.ApiClient.executeInner(ApiClient.java:236) + at app//com.databricks.sdk.core.ApiClient.getResponse(ApiClient.java:197) + at app//com.databricks.sdk.core.ApiClient.execute(ApiClient.java:187) + at app//com.databricks.sdk.core.ApiClient.POST(ApiClient.java:149) + at app//com.databricks.sdk.service.compute.ClustersImpl.delete(ClustersImpl.java:31) + at app//com.databricks.sdk.service.compute.ClustersAPI.delete(ClustersAPI.java:191) + at app//com.databricks.sdk.service.compute.ClustersAPI.delete(ClustersAPI.java:180) + at app//io.openlineage.spark.agent.DatabricksUtils.shutdown(DatabricksUtils.java:96) + at app//io.openlineage.spark.agent.DatabricksIntegrationTest.shutdown(DatabricksIntegrationTest.java:65) + at +...

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Abdallah + (abdallah@terrab.me) +
+
2023-08-22 10:39:22
 +
+

*Thread Reply:* Suppressed: com.databricks.sdk.core.DatabricksError: Missing required field: cluster_id

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Abdallah + (abdallah@terrab.me) +
+
2023-08-22 10:40:18
 +
+

*Thread Reply:* at io.openlineage.spark.agent.DatabricksUtils.uploadOpenlineageJar(DatabricksUtils.java:226)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Abdallah + (abdallah@terrab.me) +
+
2023-08-22 10:54:51
 +
+

*Thread Reply:* I did this !echo "xxx" &gt; /dbfs/databricks/openlineage/openlineage-spark-V.jar

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Abdallah + (abdallah@terrab.me) +
+
2023-08-22 10:55:29
 +
+

*Thread Reply:* To create some fake file that can be deleted in uploadOpenlineageJar function.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Abdallah + (abdallah@terrab.me) +
+
2023-08-22 10:56:09
 +
+

*Thread Reply:* Because if there is no file, this part fails +StreamSupport.stream( + workspace.dbfs().list("dbfs:/databricks/openlineage/").spliterator(), false) + .filter(f -&gt; f.getPath().contains("openlineage-spark")) + .filter(f -&gt; f.getPath().endsWith(".jar")) + .forEach(f -&gt; workspace.dbfs().delete(f.getPath()));

+ + + +
+ 😬 Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-22 11:47:17
 +
+

*Thread Reply:* does this work after +!echo "xxx" &gt; /dbfs/databricks/openlineage/openlineage-spark-V.jar +?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Abdallah + (abdallah@terrab.me) +
+
2023-08-22 11:47:36
 +
+

*Thread Reply:* Yes

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Abdallah + (abdallah@terrab.me) +
+
2023-08-22 19:02:05
 +
+

*Thread Reply:* I am now having another error in the driver

+ +

23/08/22 22:56:26 ERROR SparkContext: Error initializing SparkContext. +org.apache.spark.SparkException: Exception when registering SparkListener + at org.apache.spark.SparkContext.setupAndStartListenerBus(SparkContext.scala:3121) + at org.apache.spark.SparkContext.&lt;init&gt;(SparkContext.scala:835) + at com.databricks.backend.daemon.driver.DatabricksILoop$.$anonfun$initializeSharedDriverContext$1(DatabricksILoop.scala:362) +... + at com.databricks.DatabricksMain.main(DatabricksMain.scala:146) + at com.databricks.backend.daemon.driver.DriverDaemon.main(DriverDaemon.scala) +Caused by: java.lang.ClassNotFoundException: io.openlineage.spark.agent.OpenLineageSparkListener not found in com.databricks.backend.daemon.driver.ClassLoaders$LibraryClassLoader@298cfe89 + at com.databricks.backend.daemon.driver.ClassLoaders$MultiReplClassLoader.loadClass(ClassLoaders.scala:115) + at java.lang.ClassLoader.loadClass(ClassLoader.java:352) + at java.lang.Class.forName0(Native Method) + at java.lang.Class.forName(Class.java:348) + at org.apache.spark.util.Utils$.classForName(Utils.scala:263)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Abdallah + (abdallah@terrab.me) +
+
2023-08-22 19:19:29
 +
+

*Thread Reply:* Can you please share with me your json conf for the cluster ?

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Abdallah + (abdallah@terrab.me) +
+
2023-08-22 19:55:57
 +
+

*Thread Reply:* It's because in mu build file I have

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Abdallah + (abdallah@terrab.me) +
+
2023-08-22 19:56:27
 +
+

*Thread Reply:* and the one that was copied is

+ +
+
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Abdallah + (abdallah@terrab.me) +
+
2023-08-22 20:01:12
 +
+

*Thread Reply:* due to the findAny 😕 +private static void uploadOpenlineageJar(WorkspaceClient workspace) { + Path jarFile = + Files.list(Paths.get("../build/libs/")) + .filter(p -&gt; p.getFileName().toString().startsWith("openlineage-spark-")) + .filter(p -&gt; p.getFileName().toString().endsWith("jar")) + .findAny() + .orElseThrow(() -&gt; new RuntimeException("openlineage-spark jar not found"));

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Abdallah + (abdallah@terrab.me) +
+
2023-08-22 20:35:10
 +
+

*Thread Reply:* It works finally 😄

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Abdallah + (abdallah@terrab.me) +
+
2023-08-23 05:16:19
 +
+

*Thread Reply:* The PR 😄 +https://github.com/OpenLineage/OpenLineage/pull/2061

+
+ + + + +
#2061 Bug/fix ignored event adaptive spark plan databricks
+
+ <p><strong>Problem</strong></p> + +<p>While implementing the current OpenLineage Spark integration in Databricks, we've made an interesting observation regarding event filtering and its behavior with narrow and wide transformations:</p> + +<p>For a narrow transformation (without shuffle), the event is execute<em>insert</em>into<em>hadoop</em>fs<em>relation</em>command and it doesn't include adaptive<em>spark</em>plan. <br /> +For a wide transformation (with shuffle), the event is adaptive<em>spark</em>plan and doesn't have execute<em>insert</em>into<em>hadoop</em>fs<em>relation</em>command. <br /> +We've identified that the adaptive<em>spark</em>plan is actively filtered out in the DatabricksEventFilter class. This results in the event being completely ignored due to this line:</p> + +<p>Closes: <a href="https://github.com/OpenLineage/OpenLineage/issues/2058">#2058</a></p> + +<p><strong>Solution</strong></p> + +<p>Short-term fix: Remove adaptive<em>spark</em>plan from the excludedNodes in DatabricksEventFilter.</p> + +<p><strong>One-line summary:</strong></p> + +<p>Remove adaptive<em>spark</em>plan from the excludedNodes in DatabricksEventFilter.</p> + +<p><strong>Checklist</strong></p> + +<p>• [x ] You've <a href="https://github.com/OpenLineage/OpenLineage/blob/main/why-the-dco.md">signed-off</a> your work +☑︎ Your pull request title follows our <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CONTRIBUTING.md#creating-pull-requests">guidelines</a> +• [x ] Your changes are accompanied by tests (<em>if relevant</em>) +• [ x] Your change contains a <a href="https://kurtisnusbaum.medium.com/stacked-diffs-keeping-phabricator-diffs-small-d9964f4dcfa6">small diff</a> and is self-contained +☐ You've updated any relevant documentation (<em>if relevant</em>) +• [x ] Your comment includes a one-liner for the changelog about the specific purpose of the change (<em>if necessary</em>) +☐ You've versioned the core OpenLineage model or facets according to <a href="https://docs.snowplowanalytics.com/docs/pipeline-components-and-applications/iglu/common-architecture/schemaver">SchemaVer</a> (<em>if relevant</em>) +• [ x] You've added a <a href="https://github.com/OpenLineage/OpenLineage/tree/main/.github/header_templates.md">header</a> to source files (<em>if relevant</em>)</p> + +<hr /> + +<p>SPDX-License-Identifier: Apache-2.0 <br /> +Copyright 2018-2023 contributors to the OpenLineage project</p> +
+ +
+
Labels
+ integration/spark +
+ +
+
Comments
+ 1 +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-23 08:23:49
 +
+

*Thread Reply:* thanks for the pr 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-23 08:24:02
 +
+

*Thread Reply:* code formatting checks complain now

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-23 08:25:09
 +
+

*Thread Reply:* for the JAR issues, do you also want to create PR as you've fixed the issue on your end?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-23 09:06:26
 +
+

*Thread Reply:* @Abdallah you're using newer version of Java than 8, right?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-23 09:07:07
 +
+

*Thread Reply:* AFAIK googleJavaFormat behaves differently between Java versions

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Abdallah + (abdallah@terrab.me) +
+
2023-08-23 09:15:41
 +
+

*Thread Reply:* Okay I will switch back to another java version

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Abdallah + (abdallah@terrab.me) +
+
2023-08-23 09:25:06
 +
+

*Thread Reply:* terra@MacBook-Pro-M3 spark % java -version +java version "1.8.0_381" +Java(TM) SE Runtime Environment (build 1.8.0_381-b09) +Java HotSpot(TM) 64-Bit Server VM (build 25.381-b09, mixed mode)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Abdallah + (abdallah@terrab.me) +
+
2023-08-23 09:28:28
 +
+

*Thread Reply:* Can you tell me which java version should I use ?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Abdallah + (abdallah@terrab.me) +
+
2023-08-23 09:49:42
 +
+

*Thread Reply:* Hello, I have +@mobuchowski ERROR: Missing environment variable {i} +Can you please check what does it come from ?

+
+ + + + +
Maciej Obuchowski
+
+ OpenLineage committer. +Software Engineer @getindata +
+ +
+
Company
+ @getindata +
+ +
+
Location
+ Warsaw +
+ +
+
Repositories
+ 16 +
+ +
+
Followers
+ 25 +
+ + + + + + + + + + +
+ + + +
+
+
+
+ + + + + +
+ +
+ + + + + +
+
+ + + + +
+ +
Abdallah + (abdallah@terrab.me) +
+
2023-08-23 09:50:24
 +
+

*Thread Reply:* Can you help please ?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-23 10:08:43
 +
+

*Thread Reply:* Java 8

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-23 10:10:14
 +
+

*Thread Reply:* ```Hello, I have

+ +

@mobuchowski ERROR: Missing environment variable {i} +Can you please check what does it come from ? (edited) ``` +Yup, for now I have to manually make our CI account pick your changes up if you make PR from fork. Just did that

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-23 10:11:10
 +
+

*Thread Reply:* running here now: https://app.circleci.com/pipelines/github/OpenLineage/OpenLineage/7881/workflows/90793f46-796a-4f59-9de3-5d58cbcbf162

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-23 10:53:34
 +
+

*Thread Reply:* @Abdallah merged 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Abdallah + (abdallah@terrab.me) +
+
2023-08-23 10:59:22
 +
+

*Thread Reply:* Thank you !

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-08-16 14:21:26
 +
+

@channel +Meetup notice: on Monday, 9/18, at 5:00 pm ET OpenLineage will be gathering in Toronto at Airflow Summit. Coming to the summit? Based in or near Toronto? Please join us to discuss topics such as: +• recent developments in the project including the addition of static lineage support and the OpenLineage Airflow Provider, +• the project’s history and architecture, +• opportunities to contribute, +• resources for getting started, +• + more. +Please visit medium=referral&utmcampaign=share-btnsavedeventssharemodal&utmsource=link|the meetup page> for the specific location (which is not the conference hotel) and to sign up. Hope to see some of you there! (Please note that the start time is 5:00 pm ET.)

+
+
Meetup
+ + + +
Toronto OpenLineage Meetup at Airflow Summit, Mon, Sep 18, 2023, 2:00 PM | Meetup
+
+ Data engineers and pipeline managers know that producing data lineage – end-to-end pipeline metadata instrumented at runtime or parsed at design time – is a heavy lift with +
+ + + + + + + + +
Original URL: https://www.meetup.com/openlineage/events/295488014/?utm_medium=referral&amp;utm_campaign=share-btn_savedevents_share_modal&amp;utm_source=link
+ + + +
+ + + +
+ ❤️ Julien Le Dem, Maciej Obuchowski, Harel Shein, Paweł Leszczyński, Athitya Kumar, tati +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ldacey + (lance.dacey2@sutherlandglobal.com) +
+
2023-08-20 17:45:41
 +
+

i saw OpenLineage was built into Airflow recently as a provider but the documentation seems really light (https://airflow.apache.org/docs/apache-airflow-providers-openlineage/stable/guides/user.html), is the documentation from openlineage the correct way I should proceed?

+ +

https://openlineage.io/docs/integrations/airflow/usage

+
+
openlineage.io
+ + + +
Using the Airflow integration | OpenLineage
+
+ PREREQUISITES +
+ + + + + + +
Original URL: https://openlineage.io/docs/integrations/airflow/usage
+ + + +
+ + + +
+ 👍 Sheeri Cabral (Collibra) +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2023-08-21 20:26:56
 +
+

*Thread Reply:* openlineage-airflow is the package maintained in the OpenLineage project and to be used for versions of Airflow before 2.7. You could use it with 2.7 as well but you’d be staying on the “old” integration. +apache-airflow-providers-openlineage is the new package, maintained in the Airflow project that can be used starting Airflow 2.7 and is the recommended package moving forward. It is compatible with the configuration of the old package described in that usage page. CC: @Maciej Obuchowski @Jakub Dardziński It looks like this page needs improvement.

+
+
PyPI
+ + + +
openlineage-airflow
+
+ OpenLineage integration with Airflow +
+ + + + + + + + +
Original URL: https://pypi.org/project/openlineage-airflow/
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-22 05:03:28
 +
+

*Thread Reply:* Yeah, I'll fix that

+ + + +
+ :gratitude_thank_you: Julien Le Dem +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-08-22 17:55:08
 +
+

*Thread Reply:* https://github.com/apache/airflow/pull/33610

+ +

fyi

+
+ + + + +
#33610 openlineage: finish user guide
+
+ <p>Explains what is required to get OpenLineage provider running and explains the configuration options.</p> +
+ +
+
Labels
+ area:providers, kind:documentation, provider:openlineage +
+ +
+
Comments
+ 1 +
+ + + + + + + +
+ + <a href="https://github.com/apache/airflow">apache/airflow</a> +
+ + +
+ + + +
+ 🙌 ldacey, Julien Le Dem +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ldacey + (lance.dacey2@sutherlandglobal.com) +
+
2023-08-22 17:54:20
 +
+

do I label certain raw data sources as a dataset, for example SFTP/FTP sites, 0365 emails, etc? I extract that data into a bucket for the client in a "folder" called "raw" which I know will be an OL Dataset. Would this GCS folder (after extracting the data with Airflow) be the first Dataset OL is aware of?

+ +

<gcs://client-bucket/source-system-lob/raw>

+ +

I then process that data into partitioned parquet datasets which would also be OL Datasets: +<gcs://client-bucket/source-system-lob/staging> +<gcs://client-bucket/source-system-lob/analytics>

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-08-22 18:02:46
 +
+

*Thread Reply:* that really depends on the use case IMHO +if you consider a whole directory/folder as a dataset (meaning that each file inside folds into a larger whole) you should label dataset as directory

+ +

you might as well have directory with each file being something different - in this case it would be best to set each file separately as dataset

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-08-22 18:04:32
 +
+

*Thread Reply:* there was also SymlinksDatasetFacet introduced to store alternative dataset names, might be useful: https://github.com/OpenLineage/OpenLineage/pull/936

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ldacey + (lance.dacey2@sutherlandglobal.com) +
+
2023-08-22 18:07:26
 +
+

*Thread Reply:* cool, yeah in general each file is just a snapshot of data from a client (for example, daily dump). the parquet datasets are normally partitioned and might have small fragments and I definitely picture it as more of a table than individual files

+ + + +
+ 👍 Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-23 08:22:09
 +
+

*Thread Reply:* Agree with Jakub here - with object storage, people use different patterns, but usually some directory layer vs file is the valid abstraction level, especially if your pattern is adding files with new data inside

+ + + +
+ 👍 Jakub Dardziński +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ldacey + (lance.dacey2@sutherlandglobal.com) +
+
2023-08-25 10:26:52
 +
+

*Thread Reply:* I tested a dataset for each raw file versus the folder and the folder looks much cleaner (not sure if I can collapse individual datasets/files into a group?)

+ +

from 2022, this particular source had 6 raw schema changes (client controlled, no warning). what should I do to make that as obvious as possible if I track the dataset at a folder level?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ldacey + (lance.dacey2@sutherlandglobal.com) +
+
2023-08-25 10:32:19
 +
+

*Thread Reply:* I was thinking that I could name the dataset based on the schema_version (identified by the raw column names), so in this example I would have 6 OL datasets feeding into one "staging" dataset

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ldacey + (lance.dacey2@sutherlandglobal.com) +
+
2023-08-25 10:32:57
 +
+

*Thread Reply:* not sure what the best practice would be in this scenario though

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ldacey + (lance.dacey2@sutherlandglobal.com) +
+
2023-08-22 17:55:38
 +
+

• also saw the docs reference URI = gs://{bucket name}{path} and I wondered if the path would include the filename, or if it was just the base path like I showed above

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mars Lan + (mars@metaphor.io) +
+
2023-08-22 18:35:45
 +
+

Has anyone managed to get the OL Airflow integration to work on AWS MWAA? We've tried pretty much every trick but still ended up with the following error: +Broken plugin: [openlineage.airflow.plugin] No module named 'openlineage.airflow'; 'openlineage' is not a package

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-23 05:22:18
 +
+

*Thread Reply:* Which version are you trying to use?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-23 05:22:45
 +
+

*Thread Reply:* Both OL and MWAA/Airflow 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-23 05:23:52
 +
+

*Thread Reply:* 'openlineage' is not a package +suggests that something went wrong with import process, for example cycle in import path

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mars Lan + (mars@metaphor.io) +
+
2023-08-23 16:50:34
 +
+

*Thread Reply:* MWAA: 2.6.3 +OL: 1.0.0

+ +

I can see from the log that OL has been successfully installed to the webserver: +Successfully installed openlineage-airflow-1.0.0 openlineage-integration-common-1.0.0 openlineage-python-1.0.0 openlineage-sql-1.0.0 +This is the full stacktrace: +```Traceback (most recent call last):

+ +

File "/usr/local/airflow/.local/lib/python3.10/site-packages/airflow/pluginsmanager.py", line 229, in loadentrypointplugins +pluginclass = entrypoint.load() +File "/usr/local/airflow/.local/lib/python3.10/site-packages/importlibmetadata/init.py", line 209, in load +module = importmodule(match.group('module')) +File "/usr/lib/python3.10/importlib/init.py", line 126, in importmodule +return bootstrap.gcdimport(name[level:], package, level) +File "<frozen importlib.bootstrap>", line 1050, in gcdimport +File "<frozen importlib.bootstrap>", line 1027, in _findandload +File "<frozen importlib.bootstrap>", line 992, in findandloadunlocked +File "<frozen importlib.bootstrap>", line 241, in _callwithframesremoved +File "<frozen importlib.bootstrap>", line 1050, in _gcdimport +File "<frozen importlib.bootstrap>", line 1027, in _findandload +File "<frozen importlib.bootstrap>", line 1001, in findandloadunlocked +ModuleNotFoundError: No module named 'openlineage.airflow'; 'openlineage' is not a package```

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-08-24 08:18:36
 +
+

*Thread Reply:* It’s taking long to update MWAA environment but I tested 2.6.3 version with the followingrequirements.txt: +openlineage-airflow +and +openlineage-airflow==1.0.0 +is there any step that might lead to some unexpected results?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mars Lan + (mars@metaphor.io) +
+
2023-08-24 08:29:30
 +
+

*Thread Reply:* Yeah, it takes forever to update MWAA even for a simple change. If you open either the webserver log (in CloudWatch) or the AirFlow UI, you should see the above error message.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-08-24 08:33:53
 +
+

*Thread Reply:* The thing is that I don’t see any error messages. +I wrote simple DAG to test too: +```from future import annotations

+ +

from datetime import datetime

+ +

from airflow.models import DAG

+ +

try: + from airflow.operators.empty import EmptyOperator +except ModuleNotFoundError: + from airflow.operators.dummy import DummyOperator as EmptyOperator # type: ignore

+ +

from openlineage.airflow.adapter import OpenLineageAdapter +from openlineage.client.client import OpenLineageClient

+ +

from airflow.operators.python import PythonOperator

+ +

DAGID = "exampleol"

+ +

def callable(): + client = OpenLineageClient() + adapter = OpenLineageAdapter() + print(client, adapter)

+ +

with DAG( + dagid=DAGID, + startdate=datetime(2021, 1, 1), + schedule="@once", + catchup=False, +) as dag: + begin = EmptyOperator(taskid="begin")

+ +
test = PythonOperator(task_id='print_client', python_callable=callable)```
+
+ +

and it gives expected results as well

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mars Lan + (mars@metaphor.io) +
+
2023-08-24 08:48:11
 +
+

*Thread Reply:* Oh how interesting. I did have a plugin that sets the endpoint & key via env var. Let me try to disable that to see if it fixes the issue. Will report back after 30 mins, or however long it takes to update MWAA 😉

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-08-24 08:50:05
 +
+

*Thread Reply:* ohh, I see +you probably followed this guide: https://aws.amazon.com/blogs/big-data/automate-data-lineage-on-amazon-mwaa-with-openlineage/?

+
+
Amazon Web Services
+ + + +
Automate data lineage on Amazon MWAA with OpenLineage | Amazon Web Services
+
+ In modern data architectures, datasets are combined across an organization using a variety of purpose-built services to unlock insights. As a result, data governance becomes a key component for data consumers and producers to know that their data-driven decisions are based on trusted and accurate datasets. One aspect of data governance is data lineage, which […] +
+ + + + + + + + +
Original URL: https://aws.amazon.com/blogs/big-data/automate-data-lineage-on-amazon-mwaa-with-openlineage/
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mars Lan + (mars@metaphor.io) +
+
2023-08-24 09:04:27
 +
+

*Thread Reply:* Actually no. I'm not aware of this guide. I assume it's outdated already?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-08-24 09:04:54
 +
+

*Thread Reply:* tbh I don’t know

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mars Lan + (mars@metaphor.io) +
+
2023-08-24 09:04:55
 +
+

*Thread Reply:* Actually while we're on that topic, what's the recommended way to pass the URL & API Key in MWAA?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-08-24 09:28:00
 +
+

*Thread Reply:* I think it's still a plugin that sets env vars

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mars Lan + (mars@metaphor.io) +
+
2023-08-24 09:32:18
 +
+

*Thread Reply:* Yeah based on the page you shared, secret manager + plugin seems like the way to go.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mars Lan + (mars@metaphor.io) +
+
2023-08-24 10:31:50
 +
+

*Thread Reply:* Alas after disabling the plugin and restarting the cluster, I'm still getting the same error. Do you mind to share a screenshot of your cluster's settings so I can compare?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-24 11:57:04
 +
+

*Thread Reply:* Are you maybe importing some top level OpenLineage code anywhere? This error is most likely circular import

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mars Lan + (mars@metaphor.io) +
+
2023-08-24 12:01:12
 +
+

*Thread Reply:* Let me try removing all the dags to see if it helps.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mars Lan + (mars@metaphor.io) +
+
2023-08-24 18:42:49
 +
+

*Thread Reply:* @Maciej Obuchowski you were correct! It was indeed the DAGs. The errors are gone after removing all the dags. Now just need to figure what caused the circular import since I didn't import OL directly in DAG.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mars Lan + (mars@metaphor.io) +
+
2023-08-24 18:44:33
 +
+

*Thread Reply:* Could this be the issue? +from airflow.lineage.entities import File, Table +How could I declare lineage manually if I can't import these classes?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-25 06:52:47
 +
+

*Thread Reply:* @Mars Lan I'll look in more details next week, as I'm in transit now

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-25 06:53:18
 +
+

*Thread Reply:* but if you could narrow down a problem to single dag that I or @Jakub Dardziński could reproduce, ideally locally, it would help a lot

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mars Lan + (mars@metaphor.io) +
+
2023-08-25 07:07:11
 +
+

*Thread Reply:* Thanks. I think I understand how this works much better now. Found a few useful BQ example dags. Will give them a try and report back.

+ + + +
+ 🔥 Jakub Dardziński, Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Nitin + (nitinkhannain@yahoo.com) +
+
2023-08-23 07:14:44
 +
+

Hi All, +I want to capture, source and target table details as lineage information with openlineage for Amazon Redshift. Please let me know, if anyone has done it

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-08-23 07:32:19
 +
+

*Thread Reply:* are you using Airflow to connect to Redshift?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Nitin + (nitinkhannain@yahoo.com) +
+
2023-08-24 06:50:05
 +
+

*Thread Reply:* Hi @Jakub Dardziński, +Thank you for your reply. +No, we are not using Airflow. +We are using load/Unload cmd with Pyspark and also Pandas with JDBC connection

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-08-25 13:28:37
 +
+

*Thread Reply:* @Paweł Leszczyński might know answer if Spark<->OL integration works with Redshift. Eventually JDBC is supported with sqlparser

+ +

for Pandas I think there wasn’t too much work done

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-08-28 02:18:49
 +
+

*Thread Reply:* @Nitin If you're using jdbc within Spark, the lineage should be obtained via sqlparser-rs library https://github.com/sqlparser-rs/sqlparser-rs. In case it's not, please try to provide some minimal SQL code (or pyspark) which leads to uncaught lineage.

+
+ + + + +
sqlparser-rs/sqlparser-rs
+
+ Extensible SQL Lexer and Parser for Rust +
+ +
+
Stars
+ 1980 +
+ +
+
Language
+ Rust +
+ + + + + + + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Nitin + (nitinkhannain@yahoo.com) +
+
2023-08-28 04:53:03
 +
+

*Thread Reply:* Hi @Jakub Dardziński / @Paweł Leszczyński, thank you for taking out time to reply on my query. We need to capture only load and unload query lineage which we are running using Spark.

+ +

If you have any sample implementation for reference, it will be indeed helpful

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-08-28 06:12:46
 +
+

*Thread Reply:* I think we don't support load yet on our side: https://github.com/OpenLineage/OpenLineage/blob/main/integration/sql/impl/src/visitor.rs#L8

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/integration/sql/impl/src/visitor.rs | visitor.rs>
+
+ <pre><code>use sqlparser::ast::{ +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Nitin + (nitinkhannain@yahoo.com) +
+
2023-08-28 08:18:14
 +
+

*Thread Reply:* Yeah! any way you can think of, we can accommodate it specially load and unload statement. +Also, we would like to capture, lineage information where our endpoints are Sagemaker and Redis

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Nitin + (nitinkhannain@yahoo.com) +
+
2023-08-28 13:20:37
 +
+

*Thread Reply:* @Paweł Leszczyński can we use this code base integration/common/openlineage/common/provider/redshift_data.py for redshift lineage capture

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-08-28 14:26:40
 +
+

*Thread Reply:* it still expects input and output tables that are usually retrieved from sqlparser

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-08-28 14:31:00
 +
+

*Thread Reply:* for Sagemaker there is an Airflow integration written, might be an example possibly +https://github.com/OpenLineage/OpenLineage/blob/main/integration/airflow/openlineage/airflow/extractors/sagemaker_extractors.py

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Abdallah + (abdallah@terrab.me) +
+
2023-08-23 10:55:10
 +
+

Approve a new release please 🙂 +• Fix spark integration filtering Databricks events.

+ + + +
+ ➕ Abdallah, Tristan GUEZENNEC -CROIX-, Mouad MOUSSABBIH, Ayoub Oudmane, Asmae Tounsi, Jakub Dardziński, Michael Robinson, Harel Shein, Willy Lulciuc, Maciej Obuchowski, Julien Le Dem +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-08-23 12:27:15
 +
+

*Thread Reply:* Thank you for requesting a release @Abdallah. Three +1s from committers will authorize.

+ + + +
+ 🙌 Abdallah +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-08-23 13:13:18
 +
+

*Thread Reply:* Thanks, all. The release is authorized and will be initiated within 2 business days.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Athitya Kumar + (athityakumar@gmail.com) +
+
2023-08-23 13:08:48
 +
+

Hey folks! Do we have clear step-by-step documentation on how we can leverage the ServiceLoader based approach for injecting specific OpenLineage customisations for tweaking the transport type with defaults / tweaking column level lineage etc?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-23 13:24:32
 +
+

*Thread Reply:* This proposal - https://github.com/OpenLineage/OpenLineage/blob/41dbbf1799595bd9cd1567df0a7027de08619741/proposals/168/making_spark_visitors_extensible.md

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-23 13:29:05
 +
+

*Thread Reply:* For custom transport, you have to provide implementation of interface https://github.com/OpenLineage/OpenLineage/blob/4a1a5c3bf9767467b71ca0e1b6d820ba9e[…]ain/java/io/openlineage/client/transports/TransportBuilder.java and point to it in META_INF file

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/4a1a5c3bf9767467b71ca0e1b6d820ba9e0a1a1d/client/java/src/main/java/io/openlineage/client/transports/TransportBuilder.java | TransportBuilder.java>
+
+ <pre><code>public interface TransportBuilder { +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-23 13:29:52
 +
+

*Thread Reply:* But if I understand correctly, if you want to change behavior rather than extend, the correct way may be to either contribute it to repo - if that behavior is useful to anyone, or fork the repo

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Athitya Kumar + (athityakumar@gmail.com) +
+
2023-08-23 15:14:43
 +
+

*Thread Reply:* @Maciej Obuchowski - Can you elaborate more on the "point to it in META_INF file"? Let's say we have the custom transport type built in a standalone jar by extending transport builder - what're the exact next steps to use this custom transport in the standalone jar when doing spark-submit?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-23 15:23:13
 +
+

*Thread Reply:* @Athitya Kumar your jar needs to have META-INF/services/io.openlineage.client.transports.TransportBuilder with fully qualified class names of your custom TransportBuilders there - like openlineage-spark has +io.openlineage.client.transports.HttpTransportBuilder +io.openlineage.client.transports.KafkaTransportBuilder +io.openlineage.client.transports.ConsoleTransportBuilder +io.openlineage.client.transports.FileTransportBuilder +io.openlineage.client.transports.KinesisTransportBuilder

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Athitya Kumar + (athityakumar@gmail.com) +
+
2023-08-25 01:49:29
 +
+

*Thread Reply:* @Maciej Obuchowski - I think this change may be required for consumers to leverage custom transports, can you check & verify this GH comment? +https://github.com/OpenLineage/OpenLineage/issues/2007#issuecomment-1690350630

+
+ + + + +
Comment on #2007 [PROPOSAL] Ability to support custom injectable dynamic header generation class/logic for HTTP transports
+
+ <p><a href="https://github.com/mobuchowski">@mobuchowski</a> <a href="https://github.com/pawel-big-lebowski">@pawel-big-lebowski</a> - I think the <code>Type</code> enum in the Transport.java class has default visibility, which makes it invisible for OpenLineage consumers who are trying to define a custom transport logic</p> + +<p><a href="https://github.com/OpenLineage/OpenLineage/blob/main/client/java/src/main/java/io/openlineage/client/transports/Transport.java#L14">https://github.com/OpenLineage/OpenLineage/blob/main/client/java/src/main/java/io/openlineage/client/transports/Transport.java#L14</a></p> + +<p>I think it'll be great if we can make this <code>Type</code> enum have public visibility. Also, can we add a <code>CUSTOM</code> Type here for folks who are trying to define a custom transport of their own?</p> + +<p>Please lmk If the changes proposed seem to be fine, I can draft a PR for the same</p> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-25 06:52:30
 +
+

*Thread Reply:* Probably, I will look at more details next week @Athitya Kumar as I'm in transit

+ + + +
+ 👍 Athitya Kumar +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-08-23 15:04:10
 +
+

@channel +We released OpenLineage 1.1.0, including: +Additions: +• Flink: create Openlineage configuration based on Flink configuration #2033 @pawel-big-lebowski +• Java: add Javadocs to the Java client #2004 @julienledem +• Spark: append output dataset name to a job name #2036 @pawel-big-lebowski +• Spark: support Spark 3.4.1 #2057 @pawel-big-lebowski +Fixes: +• Flink: fix a bug when getting schema for KafkaSink #2042 @pentium3 +• Spark: fix ignored event adaptive_spark_plan in Databricks #2061 @algorithmy1 +Plus additional bug fixes, doc changes and more. +Thanks to all the contributors, especially new contributors @pentium3 and @Abdallah! +Release: https://github.com/OpenLineage/OpenLineage/releases/tag/1.1.0 +Changelog: https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md +Commit history: https://github.com/OpenLineage/OpenLineage/compare/1.0.0...1.1.0 +Maven: https://oss.sonatype.org/#nexus-search;quick~openlineage +PyPI: https://pypi.org/project/openlineage-python/

+ + + +
+ 👏 Ayoub Oudmane, Abdallah, Yuanli Wang, Athitya Kumar, Mars Lan, Maciej Obuchowski, Harel Shein, Kiran Hiremath, Thomas Abraham +
+ +
+ :gratitude_thank_you: GitHubOpenLineageIssues +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-08-25 10:29:23
 +
+

@channel +Friendly reminder: our next in-person meetup is next Wednesday, August 30th in San Francisco at Astronomer’s offices in the Financial District. You can sign up and find the details on the medium=referral&utmcampaign=share-btnsavedeventssharemodal&utmsource=link|meetup event page>.

+
+
Meetup
+ + + +
OpenLineage Meetup @ Astronomer, Wed, Aug 30, 2023, 5:30 PM | Meetup
+
+ Data engineers and pipeline managers know that producing data lineage – end-to-end pipeline metadata instrumented at runtime or parsed at design time – is a heavy lift with +
+ + + + + + + + +
Original URL: https://www.meetup.com/meetup-group-bnfqymxe/events/295195280/?utm_medium=referral&amp;utm_campaign=share-btn_savedevents_share_modal&amp;utm_source=link
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
George Polychronopoulos + (george.polychronopoulos@6point6.co.uk) +
+
2023-08-25 10:57:30
 +
+

hi Openlineage team , we would like to join one of your meetups(me and @Madhav Kakumani nad @Phil Rolph and we're wondering if you are hosting any meetups after the 18/9 ? We are trying to join this but air - tickets are quite expensive

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harel Shein + (harel.shein@gmail.com) +
+
2023-08-25 11:32:12
 +
+

*Thread Reply:* there will certainly be more meetups, don’t worry about that!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harel Shein + (harel.shein@gmail.com) +
+
2023-08-25 11:32:30
 +
+

*Thread Reply:* where are you located? perhaps we can try to organize a meetup closer to where you are.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
George Polychronopoulos + (george.polychronopoulos@6point6.co.uk) +
+
2023-08-25 11:49:37
 +
+

*Thread Reply:* Thanks a lot for the response, we are in London. We'd be glad to help you organise a meetup and also meet in person!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-08-25 11:51:39
 +
+

*Thread Reply:* This is awesome, thanks @George Polychronopoulos. I’ll start a channel and invite you

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Juan Luis Cano Rodríguez + (juan_luis_cano@mckinsey.com) +
+
2023-08-28 04:47:53
 +
+

hi folks, I'm looking into exporting static metadata, and found that DatasetEvent requires a eventTime, which in my mind doesn't make sense for static events. I'm setting it to None and the Python client seems to work, but wanted to ask if I'm missing something.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-08-28 05:59:10
 +
+

*Thread Reply:* Although you emit DatasetEvent, you still emit an event and eventTime is a valid marker.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Juan Luis Cano Rodríguez + (juan_luis_cano@mckinsey.com) +
+
2023-08-28 06:01:40
 +
+

*Thread Reply:* so, should I use the current time at the moment of emitting it and that's it?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-08-28 06:01:53
 +
+

*Thread Reply:* yes, that should be it

+ + + +
+ :gratitude_thank_you: Juan Luis Cano Rodríguez +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Juan Luis Cano Rodríguez + (juan_luis_cano@mckinsey.com) +
+
2023-08-28 04:49:21
 +
+

and something else: I understand that Marquez does not yet support the 2.0 spec, hence it's incompatible with static metadata right? I tried to emit a list of DatasetEvent s and got HTTPError: 422 Client Error: Unprocessable Entity for url: <http://localhost:3000/api/v1/lineage> (I'm using a FileTransport for now)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-08-28 06:02:49
 +
+

*Thread Reply:* marquez is not capable of reflecting DatasetEvents in DB but it should respond with Unsupported event type

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-08-28 06:03:15
 +
+

*Thread Reply:* and return 200 instead of 201 created

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Juan Luis Cano Rodríguez + (juan_luis_cano@mckinsey.com) +
+
2023-08-28 06:05:41
 +
+

*Thread Reply:* I'll have a deeper look then, probably I'm doing something wrong. thanks @Paweł Leszczyński

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Joshua Dotson + (josdotso@cisco.com) +
+
2023-08-28 13:25:58
 +
+

Hi folks. I have some pure golang jobs from which I need to emit OL events to Marquez. Is the right way to go about this to generate a Golang client from the Marquez OpenAPI spec and use that client from my go jobs?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-08-28 14:23:24
 +
+

*Thread Reply:* I'd rather generate them from OL spec (compliant with JSON Schema)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Joshua Dotson + (josdotso@cisco.com) +
+
2023-08-28 15:12:21
 +
+

*Thread Reply:* I'll look into this. I take you to mean that I would use the OL spec which is available as a set of JSON schemas to create the data object and then HTTP POST it using vanilla Golang. Is that correct? Thank you for your help!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-08-28 15:30:05
 +
+

*Thread Reply:* Correct! You’re also very welcome to contribute Golang client (currently we have Python & Java clients) if you manage to send events using golang 🙂

+ + + +
+ 👏 Joshua Dotson +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-08-28 17:28:31
 +
+

@channel +The agenda for the medium=referral&utmcampaign=share-btnsavedeventssharemodal&utmsource=link|Toronto Meetup at Airflow Summit> on 9/18 has been updated. This promises to be an exciting, richly productive discussion. Don’t miss it if you’ll be in the area!

+ +
  1. Intros
  2. Evolution of spec presentation/discussion (project background/history)
  3. State of the community
  4. Spark/Column lineage update
  5. Airflow Provider update
  6. Roadmap Discussion
  7. Action items review/next steps
    8. +
+
+
Meetup
+ + + +
Toronto OpenLineage Meetup at Airflow Summit, Mon, Sep 18, 2023, 2:00 PM | Meetup
+
+ Data engineers and pipeline managers know that producing data lineage – end-to-end pipeline metadata instrumented at runtime or parsed at design time – is a heavy lift with +
+ + + + + + + + +
Original URL: https://www.meetup.com/openlineage/events/295488014/?utm_medium=referral&amp;utm_campaign=share-btn_savedevents_share_modal&amp;utm_source=link
+ + + +
+ + + +
+ ❤️ Jarek Potiuk, Paweł Leszczyński, tati +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-08-28 20:05:37
 +
+

New on the OpenLineage blog: a close look at the new OpenLineage Airflow Provider, including: +• the critical improvements it brings to the integration +• the high-level design +• implementation details +• an example operator +• planned enhancements +• a list of supported operators +• more. +The post, by @Maciej Obuchowski, @Julien Le Dem and myself is live now on the OpenLineage blog.

+
+
openlineage.io
+ + + +
The OpenLineage Airflow Provider is Here | OpenLineage
+
+ Built-in OpenLineage support in Airflow means big improvements in reliability, lineage output, and custom operator implementation. +
+ + + + + + +
Original URL: https://openlineage.io/blog/airflow-provider
+ + + +
+ + + +
+ 🎉 Drew Meyers, Harel Shein, Maciej Obuchowski, Julian LaNeve, Mars Lan +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sarwat Fatima + (sarwatfatimam@gmail.com) +
+
2023-08-29 03:18:04
 +
+

Hello, I'm currently in the process of following the instructions outlined in the provided getting started guide at https://openlineage.io/getting-started/. However, I've encountered a problem while attempting to complete *Step 1* of the guide. Unfortunately, I'm encountering an internal server error at this stage. I did manage to successfully run Marquez, but it appears that there might be an issue that needs to be addressed. I have attached screen shots.

+ +
+
image.png
+ + + + + + + + +
+
+
image.png
+ + + + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-08-29 03:20:18
 +
+

*Thread Reply:* is 5000 port taken by any other application? or ./docker/up.sh has some errors in logs?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sarwat Fatima + (sarwatfatimam@gmail.com) +
+
2023-08-29 05:23:01
 +
+

*Thread Reply:* @Jakub Dardziński 5000 port is not taken by any other application. The logs show some errors but I am not sure what is the issue here.

+ +
+
image.png
+ + + + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-29 10:02:38
 +
+

*Thread Reply:* I think Marquez is running on WSL while you're trying to connect from host computer?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Juan Luis Cano Rodríguez + (juan_luis_cano@mckinsey.com) +
+
2023-08-29 05:20:39
 +
+

hi folks, for now I'm producing .jsonl (or .ndjson ) files with one event per line, do you know if there's any way to validate those? would standard JSON Schema tools work?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Juan Luis Cano Rodríguez + (juan_luis_cano@mckinsey.com) +
+
2023-08-29 10:58:29
 +
+

*Thread Reply:* reply by @Julian LaNeve: yes 🙂💯

+ + + +
+ 👍 Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ldacey + (lance.dacey2@sutherlandglobal.com) +
+
2023-08-29 13:12:32
 +
+

for namespaces, if my data is moving between sources (SFTP -> GCS -> Azure Blob (synapse connects to parquet datasets) then should my namespace be based on the client I am working with? my current namespace has been to refer to the bucket, but that falls apart when considering the data sources and some destinations. perhaps I should just add a field for client-name instead to have a consolidated view?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-30 10:53:08
 +
+

*Thread Reply:* > then should my namespace be based on the client I am working with? +I think each of those sources should be a different namespace?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ldacey + (lance.dacey2@sutherlandglobal.com) +
+
2023-08-30 12:59:53
 +
+

*Thread Reply:* got it, yeah I was kind of picturing as one namespace for the client (we handle many clients but they are completely distinct entities). I was able to get it to work with multiple namespaces like you suggested and Marquez was able to plot everything correctly in the visualization

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ldacey + (lance.dacey2@sutherlandglobal.com) +
+
2023-08-30 13:01:18
 +
+

*Thread Reply:* I noticed some of my Dataset facets make more sense as Run facets, for example, the name of the specific file I processed and how many rows of data / size of the data for that schedule. that won't impact the Run facets Airflow provides right? I can still have the schedule information + my custom run facets?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-08-30 13:06:38
 +
+

*Thread Reply:* Yes, unless you name it the same as one of the Airflow facets 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
GitHubOpenLineageIssues + (githubopenlineageissues@gmail.com) +
+
2023-08-30 08:15:29
 +
+

Hi, Will really appreciate if someone can guide me or provide me any pointer - if they have been able to implement authentication/authorization for access to Marquez. Have not seen much info around it. Any pointers greatly appreciated. Thanks in advance.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2023-08-30 12:23:18
 +
+

*Thread Reply:* I’ve seen people do this through the ingress controller in Kubernetes. Unfortunately I don’t have documentation besides k8s specific ones you would find for the ingress controller you’re using. You’d redirect any unauthenticated request to your identity provider

+ + + +
+ :gratitude_thank_you: GitHubOpenLineageIssues +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-08-30 11:50:05
 +
+

@channel +Friendly reminder: there’s a meetup tonight at Astronomer’s offices in SF!

+
+ + +
+ + + } + + Michael Robinson + (https://openlineage.slack.com/team/U02LXF3HUN7) +
+ + + +
+
+ <p>@channel +Friendly reminder: our next in-person meetup is next Wednesday, August 30th in San Francisco at Astronomer’s offices in the Financial District. You can sign up and find the details on the <https://www.meetup.com/meetup-group-bnfqymxe/events/295195280/?utm<em>medium=referral&amp;utm</em>campaign=share-btn<em>savedevents</em>share<em>modal&amp;utm</em>source=link|meetup event page>.</p> +
+ + + + + + +
Original URL: https://openlineage.slack.com/archives/C01CK9T7HKR/p1692973763570629
+ + +
+ + Slack Conversation +
+ + +
+ + + +
+ ✅ Sheeri Cabral (Collibra) +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2023-08-30 12:15:31
 +
+

*Thread Reply:* I’ll be there and looking forward to see @John Lukenoff ‘s presentation

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Barrientos + (mbarrien@gmail.com) +
+
2023-08-30 21:38:31
 +
+

Can anyone let 3 people stuck downstairs into the 7th floor?

+ + + +
+ 👍 Willy Lulciuc +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2023-08-30 23:25:21
 +
+

*Thread Reply:* Sorry about that!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Yunhe + (yunhe52203334@outlook.com) +
+
2023-08-31 02:31:48
 +
+

hello,everyone,i can run openLineage spark code in my notebook with python,but when use my idea to execute scala code like this: +import org.apache.spark.internal.Logging +import org.apache.spark.sql.SparkSession +import io.openlineage.client.OpenLineageClientUtils.loadOpenLineageYaml +import org.apache.spark.scheduler.{SparkListener, SparkListenerApplicationEnd, SparkListenerApplicationStart} +import sun.java2d.marlin.MarlinUtils.logInfo +object Test { + def main(args: Array[String]): Unit = {

+ +
val spark = SparkSession
+  .builder()
+  .master("local")
+  .appName("test")
+  .config("spark.jars.packages","io.openlineage:openlineage_spark:0.12.0")
+  .config("spark.extraListeners","io.openlineage.spark.agent.OpenLineageSparkListener")
+  .config("spark.openlineage.transport.type","console")
+  .getOrCreate()
+
+spark.sparkContext.setLogLevel("INFO")
+
+//spark.sparkContext.addSparkListener(new MySparkAppListener)
+import spark.implicits._
+val input = Seq((1, "zs", 2020), (2, "ls", 2023)).toDF("id", "name", "year")
+
+input.select("id", "name").orderBy("id").show()
+
+ +

}

+ +

}

+ +

there is something wrong: +Exception in thread "spark-listener-group-shared" java.lang.NoSuchMethodError: io.openlineage.client.OpenLineageClientUtils.loadOpenLineageYaml(Ljava/io/InputStream;)Lio/openlineage/client/OpenLineageYaml; + at io.openlineage.spark.agent.ArgumentParser.extractOpenlineageConfFromSparkConf(ArgumentParser.java:114) + at io.openlineage.spark.agent.ArgumentParser.parse(ArgumentParser.java:78) + at io.openlineage.spark.agent.OpenLineageSparkListener.initializeContextFactoryIfNotInitialized(OpenLineageSparkListener.java:277) + at io.openlineage.spark.agent.OpenLineageSparkListener.onApplicationStart(OpenLineageSparkListener.java:267) + at org.apache.spark.scheduler.SparkListenerBus.doPostEvent(SparkListenerBus.scala:55) + at org.apache.spark.scheduler.SparkListenerBus.doPostEvent$(SparkListenerBus.scala:28) + at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37) + at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37) + at org.apache.spark.util.ListenerBus.postToAll(ListenerBus.scala:117) + at org.apache.spark.util.ListenerBus.postToAll$(ListenerBus.scala:101) + at org.apache.spark.scheduler.AsyncEventQueue.super$postToAll(AsyncEventQueue.scala:105) + at org.apache.spark.scheduler.AsyncEventQueue.$anonfun$dispatch$1(AsyncEventQueue.scala:105) + at scala.runtime.java8.JFunction0$mcJ$sp.apply(JFunction0$mcJ$sp.java:23) + at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62) + at org.apache.spark.scheduler.AsyncEventQueue.org$apache$spark$scheduler$AsyncEventQueue$$dispatch(AsyncEventQueue.scala:100) + at org.apache.spark.scheduler.AsyncEventQueue$$anon$2.$anonfun$run$1(AsyncEventQueue.scala:96) + at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1446) + at org.apache.spark.scheduler.AsyncEventQueue$$anon$2.run(AsyncEventQueue.scala:96)

+ +

i want to know how can i set idea scala environment correctly

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-08-31 02:58:41
 +
+

*Thread Reply:* io.openlineage:openlineage_spark:0.12.0 -> could you repeat the steps with newer version?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Yunhe + (yunhe52203334@outlook.com) +
+
2023-08-31 03:51:52
 +
+

ok,it`s my first use thie lineage tool. first,I added dependences in my pom.xml like this: +<dependency> + <groupId>io.openlineage</groupId> + <artifactId>openlineage-java</artifactId> + <version>0.12.0</version> + </dependency> + <dependency> + <groupId>org.apache.logging.log4j</groupId> + <artifactId>log4j-api</artifactId> + <version>2.7</version> + </dependency> + <dependency> + <groupId>org.apache.logging.log4j</groupId> + <artifactId>log4j-core</artifactId> + <version>2.7</version> + </dependency> + <dependency> + <groupId>org.apache.logging.log4j</groupId> + <artifactId>log4j-slf4j-impl</artifactId> + <version>2.7</version> + </dependency> + <dependency> + <groupId>io.openlineage</groupId> + <artifactId>openlineage-spark</artifactId> + <version>0.30.1</version> + </dependency>

+ +

my spark version is 3.3.1 and the version can not change

+ +

second, in file Openlineage/intergration/spark I enter command : docker-compose up and follow the steps in this doc: +https://openlineage.io/docs/integrations/spark/quickstart_local +there is no erro when i use notebook to execute pyspark for openlineage and I could get json message. +but after I enter "docker-compose up" ,I want to use my Idea tool to execute scala code like above,the erro happend like above. It seems that I does not configure the environment correctly. so how can i fix the problem .

+
+
openlineage.io
+ + + +
Quickstart with Jupyter | OpenLineage
+
+ Trying out the Spark integration is super easy if you already have Docker Desktop and git installed. +
+ + + + + + +
Original URL: https://openlineage.io/docs/integrations/spark/quickstart_local
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-09-01 05:15:28
 +
+

*Thread Reply:* please use latest io.openlineage:openlineage_spark:1.1.0 instead. openlineage-java is already contained in the jar, no need to add it on your own.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sheeri Cabral (Collibra) + (sheeri.cabral@collibra.com) +
+
2023-08-31 15:33:19
 +
+

Will the August meeting be put up at https://wiki.lfaidata.foundation/display/OpenLineage/Monthly+TSC+meeting soon? (usually it’s up in a few days 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-09-01 06:00:53
 +
+

*Thread Reply:* @Michael Robinson

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-09-01 17:13:32
 +
+

*Thread Reply:* The recording is on the youtube channel here. I’ll update the wiki ASAP

+
+
YouTube
+ +
+ + + } + + OpenLineage Project + (https://www.youtube.com/@openlineageproject6897) +
+ + + +
OpenLineage Community Meeting | August 10, 2023
+
+ +
+ + + + + + + + +
Original URL: https://youtu.be/0Q5dWHvIDLo
+ + + +
+ + + +
+ ✅ Sheeri Cabral (Collibra) +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2023-08-31 18:10:20
 +
+

It sounds like there have been a few announcements at Google Next: +https://cloud.google.com/data-catalog/docs/how-to/open-lineage +https://cloud.google.com/dataproc/docs/guides/lineage

+
+
Google Cloud
+ + + +
Integrate with OpenLineage  |  Data Catalog Documentation  |  Google Cloud
+
+ +
+ + + + + + + + +
Original URL: https://cloud.google.com/data-catalog/docs/how-to/open-lineage
+ + + +
+
+
Google Cloud
+ + + +
Use data lineage in Dataproc  |  Dataproc Documentation  |  Google Cloud
+
+ +
+ + + + + + + + +
Original URL: https://cloud.google.com/dataproc/docs/guides/lineage
+ + + +
+ + + +
+ 🎉 Harel Shein, Willy Lulciuc, Kevin Languasco, Peter Hicks, Maciej Obuchowski, Paweł Leszczyński, Sheeri Cabral (Collibra), Ross Turk, Michael Robinson, Jakub Dardziński, Kiran Hiremath, Laurent Paris, Anastasia Khomyakova +
+ +
+ 🙌 Harel Shein, Willy Lulciuc, Mars Lan, Peter Hicks, Maciej Obuchowski, Paweł Leszczyński, Eric Veleker, Sheeri Cabral (Collibra), Ross Turk, Michael Robinson +
+ +
+ ❤️ Willy Lulciuc, Maciej Obuchowski, ldacey, Ross Turk, Michael Robinson +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2023-09-01 23:09:55
 +
+

*Thread Reply:* https://www.youtube.com/watch?v=zvCdrNJsxBo&t=2260s

+
+
YouTube
+ +
+ + + } + + Google Cloud + (https://www.youtube.com/@googlecloud) +
+ + + +
What’s new in data governance
+
+ +
+ + + + + + + + +
Original URL: https://www.youtube.com/watch?v=zvCdrNJsxBo&amp;t=2260s
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-09-01 17:16:21
 +
+

@channel +The latest issue of OpenLineage News is out now! Please subscribe to get it directly in your inbox each month.

+
+
apache.us14.list-manage.com
+ + + +
OpenLineage Project
+
+ OpenLineage Project Email Forms +
+ + + + + + +
Original URL: http://bit.ly/OL_news
+ + + +
+ + + +
+ 🙌 Jakub Dardziński, Maciej Obuchowski +
+ +
+ 🙌:skin_tone_3: Juan Luis Cano Rodríguez +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-09-04 03:38:28
 +
+

Hi guys, I'd like to capture the spark.databricks.clusterUsageTags.clusterAllTags property from databricks. However, the value of this is a list of keys, and therefore cannot be supported by custom environment facet builder. +I was thinking that capturing this property might be useful for most databricks workloads, and whether it might make sense to auto-capture it along with other databricks variables, similar to how we capture mount points for the databricks jobs. +Does this sound okay? If so, then I can help to contribute this functionality

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-09-04 06:43:47
 +
+

*Thread Reply:* Sounds good to me

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-09-11 05:15:03
 +
+

*Thread Reply:* Added this here: https://github.com/OpenLineage/OpenLineage/pull/2099

+
+ + + + +
#2099 Capture clusterAllTags variable from databricks
+
+ <p><strong>Problem</strong></p> + +<p>Auto-collect spark.databricks.clusterUsageTags.clusterAllTags environment variable from databricks</p> + +<p>Closes: <a href="https://github.com/OpenLineage/OpenLineage/issues/2098">#2098</a></p> + +<p><strong>Solution</strong></p> + +<p>Please describe your change as it relates to the problem, or bug fix, as well as any dependencies. If your change requires a schema change, please describe the schema modification(s) and whether it's a <em>backwards-incompatible</em> or <em>backwards-compatible</em> change, then select one of the following:</p> + +<blockquote> + <p><strong>Note:</strong> All schema changes require discussion. Please <a href="https://github.com/OpenLineage/OpenLineage/issues/2098">link the issue</a> for context.</p> +</blockquote> + +<p>☐ Your change modifies the <a href="https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.json">core</a> OpenLineage model +☐ Your change modifies one or more OpenLineage <a href="https://github.com/OpenLineage/OpenLineage/tree/main/spec/facets">facets</a></p> + +<p>If you're contributing a new integration, please specify the scope of the integration and how/where it has been tested (e.g., Apache Spark integration supports <code>S3</code> and <code>GCS</code> filesystem operations, tested with AWS EMR).</p> + +<p><strong>One-line summary:</strong> +<strong>Checklist</strong></p> + +<p>☑︎ You've <a href="https://github.com/OpenLineage/OpenLineage/blob/main/why-the-dco.md">signed-off</a> your work +☐ Your pull request title follows our <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CONTRIBUTING.md#creating-pull-requests">guidelines</a> +☐ Your changes are accompanied by tests (<em>if relevant</em>) +☐ Your change contains a <a href="https://kurtisnusbaum.medium.com/stacked-diffs-keeping-phabricator-diffs-small-d9964f4dcfa6">small diff</a> and is self-contained +☐ You've updated any relevant documentation (<em>if relevant</em>) +☐ Your comment includes a one-liner for the changelog about the specific purpose of the change (<em>if necessary</em>) +☐ You've versioned the core OpenLineage model or facets according to <a href="https://docs.snowplowanalytics.com/docs/pipeline-components-and-applications/iglu/common-architecture/schemaver">SchemaVer</a> (<em>if relevant</em>) +☐ You've added a <a href="https://github.com/OpenLineage/OpenLineage/tree/main/.github/header_templates.md">header</a> to source files (<em>if relevant</em>)</p> + +<hr /> + +<p>SPDX-License-Identifier: Apache-2.0 <br /> +Copyright 2018-2023 contributors to the OpenLineage project</p> +
+ +
+
Labels
+ integration/spark +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-09-04 06:39:05
 +
+

Also, another small clarification is that when using MergeIntoCommand, I'm receiving the lineage events on the backend, but I cannot seem to find any logging of the payload when I enable debug mode in openlineage. I remember there was a similar issue reported by another user in the past. May I check if it might be possible to help with this? It's making debugging quite hard for these cases. Thanks!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-09-04 06:54:12
 +
+

*Thread Reply:* I think it only depends on log4j configuration

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-09-04 06:57:15
 +
+

*Thread Reply:* ```# Set everything to be logged to the console +log4j.rootCategory=INFO, console +log4j.appender.console=org.apache.log4j.ConsoleAppender +log4j.appender.console.target=System.err +log4j.appender.console.layout=org.apache.log4j.PatternLayout +log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n

+ +

set the log level for the openlineage spark library

+ +

log4j.logger.io.openlineage.spark=DEBUG`` +this is what we have inlog4j.properties` in test environment and it works

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-09-04 11:28:11
 +
+

*Thread Reply:* Hmm... I can see the logs for the other commands, like createViewCommand etc. I just cannot see it for any of the delta runs

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-09-05 03:33:03
 +
+

*Thread Reply:* that's interesting. So, logging is done here: https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/app/src/main/java/io/openlineage/spark/agent/EventEmitter.java#L63 and this code is unaware of delta.

+ +

The possible problem could be filtering delta events (which we do bcz of delta being noisy)

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/app/src/main/java/io/openlineage/spark/agent/EventEmitter.java | EventEmitter.java>
+
+ <pre><code> log.debug( +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-09-05 03:33:36
 +
+

*Thread Reply:* Recently, we've closed that https://github.com/OpenLineage/OpenLineage/issues/1982 which prevents generating events for ` +createOrReplaceTempView

+
+ + + + +
#1982 [Spark] CreateViewCommand
+
+ <p><code>CreateViewCommand</code> does not contain is a spark action triggered with:</p> + +<pre><code>words.createOrReplaceTempView('words') +</code></pre> + +<p>It should be filtered in order not to generate Openlineag events, as they do have no inputs or outputs.</p> +
+ +
+
Assignees
+ <a href="https://github.com/pawel-big-lebowski">@pawel-big-lebowski</a> +
+ +
+
Labels
+ integration/spark +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-09-05 03:35:12
 +
+

*Thread Reply:* and this is the code change: https://github.com/OpenLineage/OpenLineage/pull/1987/files

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-09-05 05:19:22
 +
+

*Thread Reply:* Hmm I'm a little confused here. I thought we are only filtering out events for certain specific commands, like show table etc. because its noisy right? Some important commands like MergeInto or SaveIntoDataSource used to be logged before, but I notice now that its not being logged anymore... +I'm using 0.23.0 openlineage version.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-09-05 05:47:51
 +
+

*Thread Reply:* yes, we do. it's just sometimes when doing a filter, we can remove too much. but SaveIntoDataSource and MergeInto should be fine, as we do check them within the tests

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ldacey + (lance.dacey2@sutherlandglobal.com) +
+
2023-09-04 21:35:05
 +
+

it looks like my dynamic task mapping in Airflow has the same run ID in marquez, so even if I am processing 100 files, there is only one version of the data. is there a way to have a separate version of each dynamic task so I can track the filename etc?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-09-05 08:54:57
 +
+

*Thread Reply:* map_index should be indeed included when calculating run ID (it’s deterministic in Airflow integration) +what version of Airflow are you using btw?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ldacey + (lance.dacey2@sutherlandglobal.com) +
+
2023-09-05 09:04:14
 +
+

*Thread Reply:* 2.7.0

+ +

I do see this error log in all of my dynamic tasks which might explain it:

+ +

[2023-09-05, 00:31:57 UTC] {manager.py:200} ERROR - Extractor returns non-valid metadata: None +[2023-09-05, 00:31:57 UTC] {utils.py:401} ERROR - cannot import name 'get_operator_class' from 'airflow.providers.openlineage.utils' (/home/airflow/.local/lib/python3.11/site-packages/airflow/providers/openlineage/utils/__init__.py) +Traceback (most recent call last): + File "/home/airflow/.local/lib/python3.11/site-packages/airflow/providers/openlineage/utils/utils.py", line 399, in wrapper + return f(**args, ****kwargs) + ^^^^^^^^^^^^^^^^^^ + File "/home/airflow/.local/lib/python3.11/site-packages/airflow/providers/openlineage/plugins/listener.py", line 93, in on_running + ****get_custom_facets(task_instance), + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + File "/home/airflow/.local/lib/python3.11/site-packages/airflow/providers/openlineage/utils/utils.py", line 148, in get_custom_facets + custom_facets["airflow_mappedTask"] = AirflowMappedTaskRunFacet.from_task_instance(task_instance) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + File "/home/airflow/.local/lib/python3.11/site-packages/airflow/providers/openlineage/plugins/facets.py", line 36, in from_task_instance + from airflow.providers.openlineage.utils import get_operator_class +ImportError: cannot import name 'get_operator_class' from 'airflow.providers.openlineage.utils' (/home/airflow/.local/lib/python3.11/site-packages/airflow/providers/openlineage/utils/__init__.py)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ldacey + (lance.dacey2@sutherlandglobal.com) +
+
2023-09-05 09:05:34
 +
+

*Thread Reply:* I only have a few custom operators with the on_complete facet so I think this is a built in one - it runs before my task custom logs for example

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ldacey + (lance.dacey2@sutherlandglobal.com) +
+
2023-09-05 09:06:05
 +
+

*Thread Reply:* and any time I messed up my custom facet, the error would be at the bottom of the logs. this is on top, probably an on_start facet?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-09-05 09:16:32
 +
+

*Thread Reply:* seems like some circular import

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-09-05 09:19:47
 +
+

*Thread Reply:* I just tested it manually, it’s a bug in OL provider. let me fix that

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ldacey + (lance.dacey2@sutherlandglobal.com) +
+
2023-09-05 10:53:28
 +
+

*Thread Reply:* cool, thanks. I am glad it is just a bug, I was afraid dynamic tasks were not supported for a minute there

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ldacey + (lance.dacey2@sutherlandglobal.com) +
+
2023-09-07 11:46:20
 +
+

*Thread Reply:* how do the provider updates work? they can be released in between Airflow releases and issues for them are raised on the main Airflow repo?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-09-07 11:50:07
 +
+

*Thread Reply:* generally speaking anything related to OL-Airflow should be placed to Airflow repo, important changes/bug fixes would be implemented in OL repo as well

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ldacey + (lance.dacey2@sutherlandglobal.com) +
+
2023-09-07 15:40:31
 +
+

*Thread Reply:* got it, thanks

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ldacey + (lance.dacey2@sutherlandglobal.com) +
+
2023-09-07 19:43:46
 +
+

*Thread Reply:* is there a way for me to install the openlineage provider based on the commit you made to fix the circular imports?

+ +

i was going to try to install from Airflow main branch but didnt want to mess anything up

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ldacey + (lance.dacey2@sutherlandglobal.com) +
+
2023-09-07 19:44:39
 +
+

*Thread Reply:* I saw it was merged to airflow main but it is not in 2.7.1 and there is no 1.0.3 provider version yet, so I wondered if I could manually install it for the time being

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-09-08 05:45:48
 +
+

*Thread Reply:* https://github.com/apache/airflow/blob/main/BREEZE.rst#preparing-provider-packages +building the provider package on your own could be best idea probably? that depends on how you manage your Airflow instance

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-09-08 12:01:53
 +
+

*Thread Reply:* there's 1.1.0rc1 btw

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ldacey + (lance.dacey2@sutherlandglobal.com) +
+
2023-09-08 13:44:44
 +
+

*Thread Reply:* perfect, thanks. I got started with breeze but then stopped haha

+ + + +
+ 👍 Jakub Dardziński +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ldacey + (lance.dacey2@sutherlandglobal.com) +
+
2023-09-10 20:29:00
 +
+

*Thread Reply:* The dynamic task mapping error is gone, I did run into this:

+ +

File "/home/airflow/.local/lib/python3.11/site-packages/airflow/providers/openlineage/extractors/base.py", line 70, in disabledoperators + operator.strip() for operator in conf.get("openlineage", "disabledfor_operators").split(";") + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + File "/home/airflow/.local/lib/python3.11/site-packages/airflow/configuration.py", line 1065, in get + raise AirflowConfigException(f"section/key [{section}/{key}] not found in config")

+ +

I am redeploying now with that option added to my config. I guess it did not use the default which should be ""

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ldacey + (lance.dacey2@sutherlandglobal.com) +
+
2023-09-10 20:49:17
 +
+

*Thread Reply:* added "disabledforoperators" to my openlineage config and it worked (using Airflow helm chart - not sure if that means there is an error because the value I provided should just be the default value, not sure why I needed to explicitly specify it)

+ +

openlineage: + disabledforoperators: "" + ...

+ +

this is so much better and makes a lot more sense. most of my tasks are dynamic so I was missing a lot of metadata before the fix, thanks!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Abdallah + (abdallah@terrab.me) +
+
2023-09-06 16:43:07
 +
+

Hello Everyone,

+ +

I've been diving into the Marquez codebase and found a performance bottleneck in JobDao.java for the query related to namespaceName=MyNameSpace limit=10 and 12s with limit=25. I managed to optimize it using CTEs, and the execution times dropped dramatically to 300ms (for limit=100) and under 100ms (for limit=25 ) on the same cluster. +Issue link : https://github.com/MarquezProject/marquez/issues/2608

+ +

I believe there's even more room for optimization, especially if we adjust the job_facets_view to include the namespace_name column.

+ +

Would the team be open to a PR where I share the optimized query and discuss potential further refinements? I believe these changes could significantly enhance the Marquez web UI experience.

+ +

PR link : https://github.com/MarquezProject/marquez/pull/2609

+ +

Looking forward to your feedback.

+
+ + + + +
#2608 Performance Issue with Query on Large Data Sets in JobDao.java
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/MarquezProject/marquez">MarquezProject/marquez</a> +
+ + +
+
+ + + + +
#2609 Perf/improve jobdao query
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/MarquezProject/marquez">MarquezProject/marquez</a> +
+ + +
+ + + +
+ 🔥 Jakub Dardziński, Harel Shein, Paweł Leszczyński, Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-09-06 18:03:01
 +
+

*Thread Reply:* @Willy Lulciuc wdyt?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Bernat Gabor + (gaborjbernat@gmail.com) +
+
2023-09-06 17:44:12
 +
+

Has there been any conversation on the extensibility of facets/concepts? E.g.: +• how does one extends the list of run states https://openlineage.io/docs/spec/run-cycle to add a paused/resumed state? +• how does one extend https://openlineage.io/docs/spec/facets/run-facets/nominal_time to add a created at field?

+
+
openlineage.io
+ + + +
The Run Cycle | OpenLineage
+
+ The OpenLineage object model is event-based and updates provide an OpenLineage backend with details about the activities of a Job. +
+ + + + + + +
Original URL: https://openlineage.io/docs/spec/run-cycle
+ + + +
+
+
openlineage.io
+ + + +
Nominal Time Facet | OpenLineage
+
+ The facet to describe the nominal start and end time of the run. The nominal usually means the time the job run was expected to run (like a scheduled time), and the actual time can be different. +
+ + + + + + +
Original URL: https://openlineage.io/docs/spec/facets/run-facets/nominal_time
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2023-09-06 18:28:17
 +
+

*Thread Reply:* Hello Bernat,

+ +

The primary mechanism to extend the model is through facets. You can either: +• create new standard facets in the spec: https://github.com/OpenLineage/OpenLineage/tree/main/spec/facets +• create custom facets defined somewhere else with a prefix in their name: https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.md#custom-facet-naming +• Update existing facets with a backward compatible change (example: adding an optional field). +The core spec can also be modified. Here is an example of adding a state +That being said I think more granular states like pause/resume are probably better suited in a run facet. There was an issue opened for that particular one a while ago: https://github.com/OpenLineage/OpenLineage/issues/9 maybe that particular discussion can continue there.

+ +

For the nominal time facet, You could open an issue describing the use case and on community agreement follow up with a PR on the facet itself: https://github.com/OpenLineage/OpenLineage/blob/main/spec/facets/NominalTimeRunFacet.json +(adding an optional field is backwards compatible)

+ + + +
+ 👀 Juan Luis Cano Rodríguez +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Bernat Gabor + (gaborjbernat@gmail.com) +
+
2023-09-06 18:31:12
 +
+

*Thread Reply:* I see, so in general one is best copying a standard facet and maintain it under a different name. That way can be made mandatory 🙂 and one does not need to be blocked for a long time until there's a community agreement 🤔

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2023-09-06 18:35:43
 +
+

*Thread Reply:* Yes, The goal of custom facets is to allow you to experiment and extend the spec however you want without having to wait for approval. +If the custom facet is very specific to a third party project/product then it makes sense for it to stay a custom facet. +If it is more generic then it makes sense to add it to the core facets as part of the spec. +Hopefully community agreement can be achieved relatively quickly. Unless someone is strongly against something, it can be added without too much red tape. Typically with support in at least one of the integrations to validate the model.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-09-07 15:12:20
 +
+

@channel +This month’s TSC meeting is next Thursday the 14th at 10am PT. On the tentative agenda: +• announcements +• recent releases +• demo: Spark integration tests in Databricks runtime +• open discussion +• more (TBA) +More info and the meeting link can be found on the website. All are welcome! Also, feel free to reply or DM me with discussion topics, agenda items, etc.

+
+
openlineage.io
+ + + +
TSC Meetings | OpenLineage
+
+ The OpenLineage Technical Steering Committee meets monthly, and is open to all. +
+ + + + + + +
Original URL: https://openlineage.io/meetings/
+ + + +
+ + + +
+ 👍 Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-09-11 10:07:41
 +
+

@channel +The first Toronto OpenLineage Meetup, featuring a presentation by recent adopter Metaphor, is just one week away. On the agenda:

+ +
  1. Evolution of spec presentation/discussion (project background/history)
  2. State of the community
  3. Integrating OpenLineage with Metaphor (by special guests Ye & Ivan)
  4. Spark/Column lineage update
  5. Airflow Provider update
  6. Roadmap Discussion +Find more details and RSVP https://www.meetup.com/openlineage/events/295488014/?utmmedium=referral&utmcampaign=share-btnsavedeventssharemodal&utmsource=link|here.
    7. +
+
+
metaphor.io
+ + + +
Metaphor - The Social Platform for Data
+
+ Making Data Actionable, At Scale - Designed for data teams building cloud-native, self-service data platforms for their business users. Explore our Data Governance, Data Lineage, Data Discovery, and Data Trust capabilities today. +
+ + + + + + + + +
Original URL: https://metaphor.io/
+ + + +
+
+
Meetup
+ + + +
Toronto OpenLineage Meetup at Airflow Summit, Mon, Sep 18, 2023, 5:00 PM | Meetup
+
+ Data engineers and pipeline managers know that producing data lineage – end-to-end pipeline metadata instrumented at runtime or parsed at design time – is a heavy lift with +
+ + + + + + + + +
Original URL: https://www.meetup.com/openlineage/events/295488014/?utm_medium=referral&amp;utm_campaign=share-btn_savedevents_share_modal&amp;utm_source=link
+ + + +
+ + + +
+ 🙌 Mars Lan, Jarek Potiuk, Harel Shein, Maciej Obuchowski, Peter Hicks, Paweł Leszczyński, Dongjin Seo +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Lukenoff + (john@jlukenoff.com) +
+
2023-09-11 17:07:26
 +
+

I’m seeing some odd behavior with my http transport when upgrading airflow/openlineage-airflow from 2.3.2 -> 2.6.3 and 0.24.0 -> 0.28.0. Previously I had a config like this that let me provide my own auth tokens. However, after upgrading I’m getting a 401 from the endpoint and further debugging seems to reveal that we’re not using the token provided in my TokenProvider. Does anyone know if something changed between these versions that could be causing this? (more details in 🧵 ) +transport: + type: http + url: <https://my.fake-marquez-endpoint.com> + auth: + type: some.fully.qualified.classpath

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Lukenoff + (john@jlukenoff.com) +
+
2023-09-11 17:09:40
 +
+

*Thread Reply:* If I log this line I can tell the TokenProvider is the class instance I would expect: https://github.com/OpenLineage/OpenLineage/blob/45d94fb73b5488d34b8ca544b58317382ceb3980/client/python/openlineage/client/transport/http.py#L55

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/45d94fb73b5488d34b8ca544b58317382ceb3980/client/python/openlineage/client/transport/http.py | http.py>
+
+ <pre><code> subclass = try_import_from_string(of_type) +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Lukenoff + (john@jlukenoff.com) +
+
2023-09-11 17:11:14
 +
+

*Thread Reply:* However, if I log the token_provider here I get the origin TokenProvider: https://github.com/OpenLineage/OpenLineage/blob/45d94fb73b5488d34b8ca544b58317382ceb3980/client/python/openlineage/client/transport/http.py#L154

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/45d94fb73b5488d34b8ca544b58317382ceb3980/client/python/openlineage/client/transport/http.py | http.py>
+
+ <pre><code> bearer = token_provider.get_bearer() +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Lukenoff + (john@jlukenoff.com) +
+
2023-09-11 17:18:56
 +
+

*Thread Reply:* Ah I think I see the issue. Looks like this was introduced here, we are instantiating with the base token provider here when we should be using the subclass: https://github.com/OpenLineage/OpenLineage/pull/1869/files#diff-2f8ea6f9a22b5567de8ab56c6a63da8e7adf40cb436ee5e7e6b16e70a82afe05R57

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Lukenoff + (john@jlukenoff.com) +
+
2023-09-11 17:37:42
 +
+

*Thread Reply:* Opened a PR for this here: https://github.com/OpenLineage/OpenLineage/pull/2100

+
+ + + + +
#2100 python: fix custom http transport TokenProvider
+
+ <p><strong>Problem</strong></p> + +<p>Looks like in <a href="https://github.com/OpenLineage/OpenLineage/pull/1869">#1869</a> we introduced a bug where we would always use the base <code>TokenProvider</code> class instead of a given custom token provider, even if the validation conditions passed.</p> + +<p><strong>Solution</strong></p> + +<p>Revert this to instantiate the subclass</p> + +<p>☐ Your change modifies the <a href="https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.json">core</a> OpenLineage model +☐ Your change modifies one or more OpenLineage <a href="https://github.com/OpenLineage/OpenLineage/tree/main/spec/facets">facets</a></p> + +<p>If you're contributing a new integration, please specify the scope of the integration and how/where it has been tested (e.g., Apache Spark integration supports <code>S3</code> and <code>GCS</code> filesystem operations, tested with AWS EMR).</p> + +<p><strong>One-line summary:</strong> +<strong>Checklist</strong></p> + +<p>☑︎ You've <a href="https://github.com/OpenLineage/OpenLineage/blob/main/why-the-dco.md">signed-off</a> your work +☑︎ Your pull request title follows our <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CONTRIBUTING.md#creating-pull-requests">guidelines</a> +☐ Your changes are accompanied by tests (<em>if relevant</em>) +☑︎ Your change contains a <a href="https://kurtisnusbaum.medium.com/stacked-diffs-keeping-phabricator-diffs-small-d9964f4dcfa6">small diff</a> and is self-contained +☐ You've updated any relevant documentation (<em>if relevant</em>) +☐ Your comment includes a one-liner for the changelog about the specific purpose of the change (<em>if necessary</em>) +☐ You've versioned the core OpenLineage model or facets according to <a href="https://docs.snowplowanalytics.com/docs/pipeline-components-and-applications/iglu/common-architecture/schemaver">SchemaVer</a> (<em>if relevant</em>) +☐ You've added a <a href="https://github.com/OpenLineage/OpenLineage/tree/main/.github/header_templates.md">header</a> to source files (<em>if relevant</em>)</p> + +<hr /> + +<p>SPDX-License-Identifier: Apache-2.0 <br /> +Copyright 2018-2023 contributors to the OpenLineage project</p> +
+ +
+
Labels
+ client/python +
+ +
+
Comments
+ 1 +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+ ❤️ Julien Le Dem +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sarwat Fatima + (sarwatfatimam@gmail.com) +
+
2023-09-12 08:14:06
 +
+

This particular code in docker-compose exits with code 1 because it is unable to find wait-for-it.sh, file in the container. I have checked the mounting path from the local machine, It is correct and the path on the container for Marquez is also correct i.e. /usr/src/app but it is unable to mount the wait-for-it.sh. Does anyone know why is this? This code exists in the open lineage repository as well https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/docker-compose.yml +# Marquez as an OpenLineage Client + api: + image: marquezproject/marquez + container_name: marquez-api + ports: + - "5000:5000" + - "5001:5001" + volumes: + - ./docker/wait-for-it.sh:/usr/src/app/wait-for-it.sh + links: + - "db:postgres" + depends_on: + - db + entrypoint: [ "./wait-for-it.sh", "db:5432", "--", "./entrypoint.sh" ]

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/docker-compose.yml | docker-compose.yml>
+
+ <pre><code>version: "3.7" +services: + notebook: + image: jupyter/pyspark-notebook:spark-3.1.2 + ports: + - "8888:8888" + volumes: + - ./docker/notebooks:/home/jovyan/notebooks + - ./build:/home/jovyan/openlineage + links: + - "api:marquez" + depends_on: + - api + +# Marquez as an OpenLineage Client + api: + image: marquezproject/marquez + container_name: marquez-api + ports: + - "5000:5000" + - "5001:5001" + volumes: + - ./docker/wait-for-it.sh:/usr/src/app/wait-for-it.sh + links: + - "db:postgres" + depends_on: + - db + entrypoint: [ "./wait-for-it.sh", "db:5432", "--", "./entrypoint.sh" ] + + db: + image: postgres:12.1 + container_name: marquez-db + ports: + - "5432:5432" + environment: + - POSTGRES_USER=postgres + - POSTGRES_PASSWORD=password + - MARQUEZ_DB=marquez + - MARQUEZ_USER=marquez + - MARQUEZ_PASSWORD=marquez + volumes: + - ./docker/init-db.sh:/docker-entrypoint-initdb.d/init-db.sh + # Enables SQL statement logging (see: &lt;https://www.postgresql.org/docs/12/runtime-config-logging.html#GUC-LOG-STATEMENT&gt;) + # command: ["postgres", "-c", "log_statement=all"] +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sarwat Fatima + (sarwatfatimam@gmail.com) +
+
2023-09-12 08:15:19
 +
+

*Thread Reply:* This is the error message:

+ +
+
image.png
+ + + + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-09-12 10:38:41
 +
+

*Thread Reply:* no permissions?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Guntaka Jeevan Paul + (jeevan@acceldata.io) +
+
2023-09-12 15:11:45
 +
+

I am trying to run Google Cloud Composer where i have added the openlineage-airflow pypi packagae as a dependency and have added the env OPENLINEAGEEXTRACTORS to point to my custom extractor. I have added a folder by name dependencies and inside that i have placed my extractor file, and the path given to OPENLINEAGEEXTRACTORS is dependencies.<filename>.<extractorclass_name>…still it fails with the exception saying No module named ‘dependencies’. Can anyone kindly help me out on correcting my mistake

+ +
+
Screenshot 2023-09-13 at 12.38.55 AM.png
+ + + + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harel Shein + (harel.shein@gmail.com) +
+
2023-09-12 17:15:36
 +
+

*Thread Reply:* Hey @Guntaka Jeevan Paul, can you share some details on which versions of airflow and openlineage you’re using?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Guntaka Jeevan Paul + (jeevan@acceldata.io) +
+
2023-09-12 17:16:26
 +
+

*Thread Reply:* airflow ---> 2.5.3, openlinegae-airflow ---> 1.1.0

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Guntaka Jeevan Paul + (jeevan@acceldata.io) +
+
2023-09-12 17:45:08
 +
+

*Thread Reply:* ```import traceback +import uuid +from typing import List, Optional

+ +

from openlineage.airflow.extractors.base import BaseExtractor, TaskMetadata +from openlineage.airflow.utils import getjobname

+ +

class BigQueryInsertJobExtractor(BaseExtractor): + def init(self, operator): + super().init(operator)

+ +
@classmethod
+def get_operator_classnames(cls) -&gt; List[str]:
+    return ['BigQueryInsertJobOperator']
+
+def extract(self) -&gt; Optional[TaskMetadata]:
+    return None
+
+def extract_on_complete(self, task_instance) -&gt; Optional[TaskMetadata]:
+    self.log.debug(f"JEEVAN ---&gt; extract_on_complete({task_instance})")
+    random_uuid = str(uuid.uuid4())
+    self.log.debug(f"JEEVAN ---&gt; Randomly Generated UUID --&gt; {random_uuid}")
+
+    self.operator.job_id = random_uuid
+
+    return TaskMetadata(
+        name=get_job_name(task=self.operator)
+    )```
+
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Guntaka Jeevan Paul + (jeevan@acceldata.io) +
+
2023-09-12 17:45:24
 +
+

*Thread Reply:* this is the custom extractor code that im trying with

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harel Shein + (harel.shein@gmail.com) +
+
2023-09-12 21:10:02
 +
+

*Thread Reply:* thanks @Guntaka Jeevan Paul, will try to take a deeper look tomorrow

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-09-13 07:54:26
 +
+

*Thread Reply:* No module named 'dependencies'. +This sounds like general Python problem

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-09-13 07:55:12
 +
+

*Thread Reply:* https://stackoverflow.com/questions/69991553/how-to-import-custom-modules-in-cloud-composer

+
+
Stack Overflow
+ + + +
how to import custom modules in Cloud Composer
+
+ I created a local project with apache Airflow and i want to run it in cloud composer. My project contains custom modules and a main file that calls them. +Example : from src.kuzzle import KuzzleQuery +
+ + + + + + + + +
Original URL: https://stackoverflow.com/questions/69991553/how-to-import-custom-modules-in-cloud-composer
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-09-13 07:56:28
 +
+

*Thread Reply:* basically, if you're able to import the file from your dag code, OL should be able too

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Guntaka Jeevan Paul + (jeevan@acceldata.io) +
+
2023-09-13 08:01:12
 +
+

*Thread Reply:* The Problem is in the GCS Composer there is a component called Triggerer, which they say is used for deferrable operators…i have logged into that pod and i could see that the GCS Bucket is not mounted on this, but i am unable to understand why is the initialisation happening inside the triggerer pod

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Guntaka Jeevan Paul + (jeevan@acceldata.io) +
+
2023-09-13 08:01:32
 +
+

*Thread Reply:*

+ +
+
Screenshot 2023-09-13 at 5.31.22 PM.png
+ + + + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-09-13 08:01:47
 +
+

*Thread Reply:* > The Problem is in the GCS Composer there is a component called Triggerer, which they say is used for deferrable operators…i have logged into that pod and i could see that the GCS Bucket is not mounted on this, but i am unable to understand why is the initialisation happening inside the triggerer pod +OL integration is not running on triggerer, only on worker and scheduler pods

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Guntaka Jeevan Paul + (jeevan@acceldata.io) +
+
2023-09-13 08:01:53
 +
+

*Thread Reply:*

+ +
+
Screenshot 2023-09-13 at 5.31.44 PM.png
+ + + + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Guntaka Jeevan Paul + (jeevan@acceldata.io) +
+
2023-09-13 08:03:26
 +
+

*Thread Reply:* As you can see in this screenshot i am seeing the logs of the triggerer and it says clearly unable to import plugin openlineage

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Guntaka Jeevan Paul + (jeevan@acceldata.io) +
+
2023-09-13 08:03:29
 +
+

*Thread Reply:* https://openlineage.slack.com/files/U05QL7LN2GH/F05SUDUQEDN/screenshot_2023-09-13_at_5.31.22_pm.png

+ +
+
Screenshot 2023-09-13 at 5.31.22 PM.png
+ + + + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-09-13 08:10:32
 +
+

*Thread Reply:* I see. There are few possible things to do here - composer could mount the user files, Airflow could not start plugins on triggerer, or we could detect we're on triggerer and not import anything there. However, does it impact OL or Airflow operation in other way than this log?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-09-13 08:12:06
 +
+

*Thread Reply:* Probably we'd have to do something if that really bothers you as there won't be further changes to Airflow 2.5

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Guntaka Jeevan Paul + (jeevan@acceldata.io) +
+
2023-09-13 08:18:14
 +
+

*Thread Reply:* The Problem is it is actually not registering this custom extractor written by me, henceforth i am just receiving the DefaultExtractor things and my piece of extractor code is not even getting triggered

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Guntaka Jeevan Paul + (jeevan@acceldata.io) +
+
2023-09-13 08:22:49
 +
+

*Thread Reply:* any suggestions to try @Maciej Obuchowski

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-09-13 08:27:48
 +
+

*Thread Reply:* Could you share worker logs?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-09-13 08:27:56
 +
+

*Thread Reply:* and check if module is importable from your dag code?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Guntaka Jeevan Paul + (jeevan@acceldata.io) +
+
2023-09-13 08:31:25
 +
+

*Thread Reply:* these are the worker pod logs…where there is no log of openlineageplugin

+ +
+
downloaded-logs-20230913-180017.json
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Guntaka Jeevan Paul + (jeevan@acceldata.io) +
+
2023-09-13 08:31:52
 +
+

*Thread Reply:* https://openlineage.slack.com/archives/C01CK9T7HKR/p1694608076879469?thread_ts=1694545905.974339&cid=C01CK9T7HKR --> sure will check now on this one

+
+ + +
+ + + } + + Maciej Obuchowski + (https://openlineage.slack.com/team/U01RA9B5GG2) +
+ + + +
+
+ <p>and check if module is importable from your dag code?</p> +
+ + + + + + +
Original URL: https://openlineage.slack.com/archives/C01CK9T7HKR/p1694608076879469?thread_ts=1694545905.974339&amp;cid=C01CK9T7HKR
+ + +
+ + Thread in Slack Conversation +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-09-13 08:38:32
 +
+

*Thread Reply:* { + "textPayload": "Traceback (most recent call last): File \"/opt/python3.8/lib/python3.8/site-packages/openlineage/airflow/utils.py\", line 427, in import_from_string module = importlib.import_module(module_path) File \"/opt/python3.8/lib/python3.8/importlib/__init__.py\", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File \"&lt;frozen importlib._bootstrap&gt;\", line 1014, in _gcd_import File \"&lt;frozen importlib._bootstrap&gt;\", line 991, in _find_and_load File \"&lt;frozen importlib._bootstrap&gt;\", line 961, in _find_and_load_unlocked File \"&lt;frozen importlib._bootstrap&gt;\", line 219, in _call_with_frames_removed File \"&lt;frozen importlib._bootstrap&gt;\", line 1014, in _gcd_import File \"&lt;frozen importlib._bootstrap&gt;\", line 991, in _find_and_load File \"&lt;frozen importlib._bootstrap&gt;\", line 961, in _find_and_load_unlocked File \"&lt;frozen importlib._bootstrap&gt;\", line 219, in _call_with_frames_removed File \"&lt;frozen importlib._bootstrap&gt;\", line 1014, in _gcd_import File \"&lt;frozen importlib._bootstrap&gt;\", line 991, in _find_and_load File \"&lt;frozen importlib._bootstrap&gt;\", line 973, in _find_and_load_unlockedModuleNotFoundError: No module named 'airflow.gcs'", + "insertId": "pt2eu6fl9z5vw", + "resource": { + "type": "cloud_composer_environment", + "labels": { + "environment_name": "openlineage", + "location": "us-west1", + "project_id": "acceldata-acm" + } + }, + "timestamp": "2023-09-13T06:20:44.131577764Z", + "severity": "ERROR", + "labels": { + "worker_id": "airflow-worker-xttt8" + }, + "logName": "projects/acceldata-acm/logs/airflow-worker", + "receiveTimestamp": "2023-09-13T06:20:48.847319607Z" + }, +it doesn't see No module named 'airflow.gcs' that is part of your extractor path airflow.gcs.dags.big_query_insert_job_extractor.BigQueryInsertJobExtractor +however, is it necessary? I generally see people using imports directly from dags folder

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Guntaka Jeevan Paul + (jeevan@acceldata.io) +
+
2023-09-13 08:44:11
 +
+

*Thread Reply:* this is one of the experimentation that i have did, but then i reverted it back to keeping it to dependencies.bigqueryinsertjobextractor.BigQueryInsertJobExtractor…where dependencies is a module i have created inside my dags folder

+ +
+
Screenshot 2023-09-13 at 6.13.36 PM.png
+ + + + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Guntaka Jeevan Paul + (jeevan@acceldata.io) +
+
2023-09-13 08:44:33
 +
+

*Thread Reply:* https://openlineage.slack.com/files/U05QL7LN2GH/F05RM6EV6DV/screenshot_2023-09-13_at_12.38.55_am.png

+ +
+
Screenshot 2023-09-13 at 12.38.55 AM.png
+ + + + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Guntaka Jeevan Paul + (jeevan@acceldata.io) +
+
2023-09-13 08:45:46
 +
+

*Thread Reply:* these are the logs of the triggerer pod specifically

+ +
+
Screenshot 2023-09-13 at 6.15.30 PM.png
+ + + + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-09-13 08:46:31
 +
+

*Thread Reply:* yeah it would be expected to have this in triggerer where it's not mounted, but will it behave the same for worker where it's mounted?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-09-13 08:47:09
 +
+

*Thread Reply:* maybe ___init___.py is missing for top-level dag path?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Guntaka Jeevan Paul + (jeevan@acceldata.io) +
+
2023-09-13 08:49:01
 +
+

*Thread Reply:* these are the logs of the worker pod at startup, where it does not complain of the plugin like in triggerer, but when tasks are run on this worker…somehow it is not picking up the extractor for the operator that i have written it for

+ +
+
Screenshot 2023-09-13 at 6.17.46 PM.png
+ + + + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Guntaka Jeevan Paul + (jeevan@acceldata.io) +
+
2023-09-13 08:49:54
 +
+

*Thread Reply:* https://openlineage.slack.com/archives/C01CK9T7HKR/p1694609229577469?thread_ts=1694545905.974339&cid=C01CK9T7HKR --> you mean to make the dags folder as well like a module by adding the init.py?

+
+ + +
+ + + } + + Maciej Obuchowski + (https://openlineage.slack.com/team/U01RA9B5GG2) +
+ + + +
+
+ <p>maybe <code>___init___.py</code> is missing for top-level dag path?</p> +
+ + + + + + +
Original URL: https://openlineage.slack.com/archives/C01CK9T7HKR/p1694609229577469?thread_ts=1694545905.974339&amp;cid=C01CK9T7HKR
+ + +
+ + Thread in Slack Conversation +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-09-13 08:55:24
 +
+

*Thread Reply:* yes, I would put whole custom code directly in dags folder, to make sure import paths are the problem

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-09-13 08:55:48
 +
+

*Thread Reply:* and would be nice if you could set +AIRFLOW__LOGGING__LOGGING_LEVEL="DEBUG"

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Guntaka Jeevan Paul + (jeevan@acceldata.io) +
+
2023-09-13 09:14:58
 +
+

*Thread Reply:* ```Starting the process, got command: triggerer +Initializing airflow.cfg. +airflow.cfg initialization is done. +[2023-09-13T13:11:46.620+0000] {settings.py:267} DEBUG - Setting up DB connection pool (PID 8) +[2023-09-13T13:11:46.622+0000] {settings.py:372} DEBUG - settings.prepareengineargs(): Using pool settings. poolsize=5, maxoverflow=10, poolrecycle=570, pid=8 +[2023-09-13T13:11:46.742+0000] {cliactionloggers.py:39} DEBUG - Adding <function defaultactionlog at 0x7ff39ca1d3a0> to pre execution callback +[2023-09-13T13:11:47.638+0000] {cliactionloggers.py:65} DEBUG - Calling callbacks: [<function defaultactionlog at 0x7ff39ca1d3a0>] + __ ___ + _ |( )__ / /_ _ +_ /| |_ / / /_ _ / _ _ | /| / / +_ | / _ / _ _/ _ / / // /_ |/ |/ / + // |// // // // _/_/|/ +[2023-09-13T13:11:50.527+0000] {pluginsmanager.py:300} DEBUG - Loading plugins +[2023-09-13T13:11:50.580+0000] {pluginsmanager.py:244} DEBUG - Loading plugins from directory: /home/airflow/gcs/plugins +[2023-09-13T13:11:50.581+0000] {pluginsmanager.py:224} DEBUG - Loading plugins from entrypoints +[2023-09-13T13:11:50.587+0000] {pluginsmanager.py:227} DEBUG - Importing entrypoint plugin OpenLineagePlugin +[2023-09-13T13:11:50.740+0000] {utils.py:430} WARNING - No module named 'boto3' +[2023-09-13T13:11:50.743+0000] {utils.py:430} WARNING - No module named 'botocore' +[2023-09-13T13:11:50.833+0000] {utils.py:430} WARNING - No module named 'airflow.providers.sftp' +[2023-09-13T13:11:51.144+0000] {utils.py:430} WARNING - No module named 'bigqueryinsertjobextractor' +[2023-09-13T13:11:51.145+0000] {pluginsmanager.py:237} ERROR - Failed to import plugin OpenLineagePlugin +Traceback (most recent call last): + File "/opt/python3.8/lib/python3.8/site-packages/openlineage/airflow/utils.py", line 427, in importfromstring + module = importlib.importmodule(modulepath) + File "/opt/python3.8/lib/python3.8/importlib/init.py", line 127, in importmodule + return bootstrap.gcdimport(name[level:], package, level) + File "<frozen importlib.bootstrap>", line 1014, in gcdimport + File "<frozen importlib.bootstrap>", line 991, in _findandload + File "<frozen importlib.bootstrap>", line 973, in findandloadunlocked +ModuleNotFoundError: No module named 'bigqueryinsertjobextractor'

+ +

The above exception was the direct cause of the following exception:

+ +

Traceback (most recent call last): + File "/opt/python3.8/lib/python3.8/site-packages/airflow/pluginsmanager.py", line 229, in loadentrypointplugins + pluginclass = entrypoint.load() + File "/opt/python3.8/lib/python3.8/site-packages/setuptools/vendor/importlibmetadata/init.py", line 194, in load + module = importmodule(match.group('module')) + File "/opt/python3.8/lib/python3.8/importlib/init.py", line 127, in importmodule + return _bootstrap.gcdimport(name[level:], package, level) + File "<frozen importlib.bootstrap>", line 1014, in gcdimport + File "<frozen importlib.bootstrap>", line 991, in _findandload + File "<frozen importlib.bootstrap>", line 975, in findandloadunlocked + File "<frozen importlib.bootstrap>", line 671, in _loadunlocked + File "<frozen importlib.bootstrapexternal>", line 843, in execmodule + File "<frozen importlib.bootstrap>", line 219, in callwithframesremoved + File "/opt/python3.8/lib/python3.8/site-packages/openlineage/airflow/plugin.py", line 32, in <module> + from openlineage.airflow import listener + File "/opt/python3.8/lib/python3.8/site-packages/openlineage/airflow/listener.py", line 75, in <module> + extractormanager = ExtractorManager() + File "/opt/python3.8/lib/python3.8/site-packages/openlineage/airflow/extractors/manager.py", line 16, in init + self.tasktoextractor = Extractors() + File "/opt/python3.8/lib/python3.8/site-packages/openlineage/airflow/extractors/extractors.py", line 122, in init + extractor = importfromstring(extractor.strip()) + File "/opt/python3.8/lib/python3.8/site-packages/openlineage/airflow/utils.py", line 431, in importfromstring + raise ImportError(f"Failed to import {path}") from e +ImportError: Failed to import bigqueryinsertjobextractor.BigQueryInsertJobExtractor +[2023-09-13T13:11:51.235+0000] {pluginsmanager.py:227} DEBUG - Importing entrypoint plugin composermenuplugin +[2023-09-13T13:11:51.719+0000] {pluginsmanager.py:316} DEBUG - Loading 1 plugin(s) took 1.14 seconds +[2023-09-13T13:11:51.733+0000] {triggererjob.py:101} INFO - Starting the triggerer +[2023-09-13T13:11:51.734+0000] {selectorevents.py:59} DEBUG - Using selector: EpollSelector +[2023-09-13T13:11:56.118+0000] {basejob.py:240} DEBUG - [heartbeat] +[2023-09-13T13:12:01.359+0000] {basejob.py:240} DEBUG - [heartbeat] +[2023-09-13T13:12:06.665+0000] {basejob.py:240} DEBUG - [heartbeat] +[2023-09-13T13:12:11.880+0000] {basejob.py:240} DEBUG - [heartbeat] +[2023-09-13T13:12:17.098+0000] {basejob.py:240} DEBUG - [heartbeat] +[2023-09-13T13:12:22.323+0000] {basejob.py:240} DEBUG - [heartbeat] +[2023-09-13T13:12:27.597+0000] {basejob.py:240} DEBUG - [heartbeat] +[2023-09-13T13:12:32.826+0000] {basejob.py:240} DEBUG - [heartbeat] +[2023-09-13T13:12:38.049+0000] {basejob.py:240} DEBUG - [heartbeat] +[2023-09-13T13:12:43.275+0000] {basejob.py:240} DEBUG - [heartbeat] +[2023-09-13T13:12:48.509+0000] {basejob.py:240} DEBUG - [heartbeat] +[2023-09-13T13:12:53.867+0000] {basejob.py:240} DEBUG - [heartbeat] +[2023-09-13T13:12:59.087+0000] {basejob.py:240} DEBUG - [heartbeat] +[2023-09-13T13:13:04.300+0000] {basejob.py:240} DEBUG - [heartbeat] +[2023-09-13T13:13:09.539+0000] {basejob.py:240} DEBUG - [heartbeat] +[2023-09-13T13:13:14.785+0000] {basejob.py:240} DEBUG - [heartbeat] +[2023-09-13T13:13:20.007+0000] {basejob.py:240} DEBUG - [heartbeat] +[2023-09-13T13:13:25.274+0000] {basejob.py:240} DEBUG - [heartbeat] +[2023-09-13T13:13:30.510+0000] {basejob.py:240} DEBUG - [heartbeat] +[2023-09-13T13:13:35.729+0000] {basejob.py:240} DEBUG - [heartbeat] +[2023-09-13T13:13:40.960+0000] {basejob.py:240} DEBUG - [heartbeat] +[2023-09-13T13:13:46.444+0000] {basejob.py:240} DEBUG - [heartbeat] +[2023-09-13T13:13:51.751+0000] {basejob.py:240} DEBUG - [heartbeat] +[2023-09-13T13:13:57.084+0000] {basejob.py:240} DEBUG - [heartbeat] +[2023-09-13T13:14:02.310+0000] {basejob.py:240} DEBUG - [heartbeat] +[2023-09-13T13:14:07.535+0000] {basejob.py:240} DEBUG - [heartbeat] +[2023-09-13T13:14:12.754+0000] {basejob.py:240} DEBUG - [heartbeat] +[2023-09-13T13:14:17.967+0000] {basejob.py:240} DEBUG - [heartbeat] +[2023-09-13T13:14:23.185+0000] {basejob.py:240} DEBUG - [heartbeat] +[2023-09-13T13:14:28.406+0000] {basejob.py:240} DEBUG - [heartbeat] +[2023-09-13T13:14:33.661+0000] {basejob.py:240} DEBUG - [heartbeat] +[2023-09-13T13:14:38.883+0000] {basejob.py:240} DEBUG - [heartbeat] +[2023-09-13T13:14:44.247+0000] {base_job.py:240} DEBUG - [heartbeat]```

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Guntaka Jeevan Paul + (jeevan@acceldata.io) +
+
2023-09-13 09:15:10
 +
+

*Thread Reply:* still the same error in the triggerer pod

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Guntaka Jeevan Paul + (jeevan@acceldata.io) +
+
2023-09-13 09:16:23
 +
+

*Thread Reply:* have changed the dags folder where i have added the init file as you suggested and then have updated the OPENLINEAGEEXTRACTORS to bigqueryinsertjob_extractor.BigQueryInsertJobExtractor…still the same thing

+ +
+
Screenshot 2023-09-13 at 6.45.26 PM.png
+ + + + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-09-13 09:36:27
 +
+

*Thread Reply:* > still the same error in the triggerer pod +it won't change, we're not trying to fix the triggerer import but worker, and should look only at worker pod at this point

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Guntaka Jeevan Paul + (jeevan@acceldata.io) +
+
2023-09-13 09:43:34
 +
+

*Thread Reply:* ```extractor for <class 'airflow.providers.google.cloud.operators.bigquery.BigQueryInsertJobOperator'> is <class 'bigqueryinsertjobextractor.BigQueryInsertJobExtractor'

+ +

Using extractor BigQueryInsertJobExtractor tasktype=BigQueryInsertJobOperator airflowdagid=dataanalyticsdag taskid=joinbqdatasets.bqjoinholidaysweatherdata2021 airflowrunid=manual_2023-09-13T13:24:08.946947+00:00

+ +

fatal: not a git repository (or any parent up to mount point /home/airflow) +Stopping at filesystem boundary (GITDISCOVERYACROSSFILESYSTEM not set). +fatal: not a git repository (or any parent up to mount point /home/airflow) +Stopping at filesystem boundary (GITDISCOVERYACROSSFILESYSTEM not set).```

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Guntaka Jeevan Paul + (jeevan@acceldata.io) +
+
2023-09-13 09:44:44
 +
+

*Thread Reply:* able to see these logs in the worker pod…so what you said is right that it is able to get the extractor but i get the below error immediately where it says not a git repository

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Guntaka Jeevan Paul + (jeevan@acceldata.io) +
+
2023-09-13 09:45:24
 +
+

*Thread Reply:* seems like we are almost there nearby…am i missing something obvious

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-09-13 10:06:35
 +
+

*Thread Reply:* > fatal: not a git repository (or any parent up to mount point /home/airflow) +&gt; Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set). +&gt; fatal: not a git repository (or any parent up to mount point /home/airflow) +&gt; Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set). +hm, this could be the actual bug?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-09-13 10:06:51
 +
+

*Thread Reply:* that’s casual log in composer

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-09-13 10:12:16
 +
+

*Thread Reply:* extractor for &lt;class 'airflow.providers.google.cloud.operators.bigquery.BigQueryInsertJobOperator'&gt; is &lt;class 'big_query_insert_job_extractor.BigQueryInsertJobExtractor' +that’s actually class from your custom module, right?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-09-13 10:14:03
 +
+

*Thread Reply:* I’ve done experiment, that’s how gcs looks like

+ +
+
image.png
+ + + + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-09-13 10:14:09
 +
+

*Thread Reply:* and env vars

+ +
+
image.png
+ + + + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-09-13 10:14:19
 +
+

*Thread Reply:* I have this extractor detected as expected

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-09-13 10:15:06
 +
+

*Thread Reply:* seens as &lt;class 'dependencies.bq.BigQueryInsertJobExtractor'&gt;

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-09-13 10:16:02
 +
+

*Thread Reply:* no __init__.py in base dags folder

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-09-13 10:17:02
 +
+

*Thread Reply:* I also checked that triggerer pod indeed has no gcsfuse set up, tbh no idea why, maybe some kind of optimization +the only effect is that when loading plugins in triggerer it throws some errors in logs, we don’t do anything at the moment there

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Guntaka Jeevan Paul + (jeevan@acceldata.io) +
+
2023-09-13 10:19:26
 +
+

*Thread Reply:* okk…got it @Jakub Dardziński…so the init at the top level of dags is as well not reqd, got it. Just one more doubt, there is a requirement where i want to change the operators property in the extractor inside the extract function, will that be taken into account and the operator’s execute be called with the property that i have populated in my extractor?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Guntaka Jeevan Paul + (jeevan@acceldata.io) +
+
2023-09-13 10:21:28
 +
+

*Thread Reply:* for example i want to add a custom jobid to the BigQueryInsertJobOperator, so wheneerv someone uses the BigQueryInsertJobOperator operator i want to intercept that and add this jobid property to the operator…will that work?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-09-13 10:24:46
 +
+

*Thread Reply:* I’m not sure if using OL for such thing is best choice. Wouldn’t it be better to subclass the operator?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-09-13 10:25:37
 +
+

*Thread Reply:* but the answer is: it dependes on the airflow version, in 2.3+ I’m pretty sure the changed property stays in execute method

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Guntaka Jeevan Paul + (jeevan@acceldata.io) +
+
2023-09-13 10:27:49
 +
+

*Thread Reply:* yeah ideally that is how we should have done this but the problem is our client is having around 1000+ Dag’s in different google cloud projects, which are owned by multiple teams…so they are not willing to change anything in their dag. Thankfully they are using airflow 2.4.3

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-09-13 10:31:15
 +
+

*Thread Reply:* task_policy might be better tool for that: https://airflow.apache.org/docs/apache-airflow/2.6.0/administration-and-deployment/cluster-policies.html

+ + + +
+ ➕ Jakub Dardziński +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-09-13 10:35:30
 +
+

*Thread Reply:* btw I double-checked - execute method is in different process so this would not change task’s attribute there

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Guntaka Jeevan Paul + (jeevan@acceldata.io) +
+
2023-09-16 03:32:49
 +
+

*Thread Reply:* @Jakub Dardziński any idea how can we achieve this one. ---> https://openlineage.slack.com/archives/C01CK9T7HKR/p1694849427228709

+
+ + +
+ + + } + + Guntaka Jeevan Paul + (https://openlineage.slack.com/team/U05QL7LN2GH) +
+ + + +
+
+ <p>@here we have dataproc operator getting called from a dag which submits a spark job, we wanted to maintain that continuity of parent job in the spark job so according to the documentation we can acheive that by using a macro called lineage<em>run</em>id that requires task and task<em>instance as the parameters. The problem we are facing is that our client’s have 1000's of dags, so asking them to change this everywhere it is used is not feasible, so we thought of using the task</em>policy feature in the airflow…but the problem is that task<em>policy gives you access to only the task/operator, but we don’t have the access to the task instance..that is required as a parameter to the lineage</em>run_id function. Can anyone kindly help us on how should we go about this one +<code>t1 = DataProcPySparkOperator( + task_id=job_name, + &lt;b&gt;#required&lt;/b&gt; pyspark configuration, + job_name=job_name, + dataproc_pyspark_properties={ + 'spark.driver.extraJavaOptions': + f"-javaagent:{jar}={os.environ.get('OPENLINEAGE_URL')}/api/v1/namespaces/{os.getenv('OPENLINEAGE_NAMESPACE', 'default')}/jobs/{job_name}/runs/{{{{macros.OpenLineagePlugin.lineage_run_id(task, task_instance)}}}}?api_key={os.environ.get('OPENLINEAGE_API_KEY')}" + dag=dag)</code></p> +
+ + + + + + +
Original URL: https://openlineage.slack.com/archives/C01CK9T7HKR/p1694849427228709
+ + +
+ + Slack Conversation +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Guntaka Jeevan Paul + (jeevan@acceldata.io) +
+
2023-09-12 17:26:01
 +
+

@here has anyone succeded in getting a custom extractor to work in GCP Cloud Composer or AWS MWAA, seems like there is no way

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mars Lan + (mars@metaphor.io) +
+
2023-09-12 17:34:29
 +
+

*Thread Reply:* I'm getting quite close with MWAA. See https://openlineage.slack.com/archives/C01CK9T7HKR/p1692743745585879.

+
+ + +
+ + + } + + Mars Lan + (https://openlineage.slack.com/team/U01HVNU6A4C) +
+ + + +
+
+ <p>Has anyone managed to get the OL Airflow integration to work on AWS MWAA? We've tried pretty much every trick but still ended up with the following error: +<code>Broken plugin: [openlineage.airflow.plugin] No module named 'openlineage.airflow'; 'openlineage' is not a package</code></p> +
+ + + + + + +
Original URL: https://openlineage.slack.com/archives/C01CK9T7HKR/p1692743745585879
+ + +
+ + Thread in Slack Conversation +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Suraj Gupta + (suraj.gupta@atlan.com) +
+
2023-09-13 01:44:27
 +
+

I am exploring Spark - OpenLineage integration (using the latest PySpark and OL versions). I tested a simple pipeline which: +• Reads JSON data into PySpark DataFrame +• Apply data transformations +• Write transformed data to MySQL database +Observed that we receive 4 events (2 START and 2 COMPLETE) for the same job name. The events are almost identical with a small diff in the facets. All the events share the same runId, and we don't get any parentRunId. +Team, can you please confirm if this behaviour is expected? Seems to be different from the Airflow integration where we relate jobs to Parent Jobs.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Damien Hawes + (damien.hawes@booking.com) +
+
2023-09-13 02:54:37
 +
+

*Thread Reply:* The Spark integration requires that two parameters are passed to it, namely:

+ +

spark.openlineage.parentJobName +spark.openlineage.parentRunId +You can find the list of parameters here:

+ +

https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/README.md

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Suraj Gupta + (suraj.gupta@atlan.com) +
+
2023-09-13 02:55:51
 +
+

*Thread Reply:* Thanks, will check this out

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Damien Hawes + (damien.hawes@booking.com) +
+
2023-09-13 02:57:43
 +
+

*Thread Reply:* As for double accounting of events - that's a bit harder to diagnose.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-09-13 04:33:03
 +
+

*Thread Reply:* Can you share the the job and events? +Also @Paweł Leszczyński

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Suraj Gupta + (suraj.gupta@atlan.com) +
+
2023-09-13 06:03:49
 +
+

*Thread Reply:* Sure, sharing Job and events.

+ +
+
+ + + + + + +
+
+
spark_events.txt
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Suraj Gupta + (suraj.gupta@atlan.com) +
+
2023-09-13 06:06:21
 +
+

*Thread Reply:*

+ +
+
etl-mysql.py
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-09-13 06:39:02
 +
+

*Thread Reply:* Hi @Suraj Gupta,

+ +

Thanks for providing such a detailed description of the problem.

+ +

It is not expected behaviour, it's an issue. The events correspond to the same logical plan which for some reason lead to sending two OL events. Is it reproducible aka. does it occur each time? If yes, we please feel free to raise an issue for that.

+ +

We have added in recent months several tests to verify amount of OL events being generated but we haven't tested it that way with JDBC. BTW. will the same happen if you write your data df_transformed to a file (like parquet file) ?

+ + + +
+ :gratitude_thank_you: Suraj Gupta +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Suraj Gupta + (suraj.gupta@atlan.com) +
+
2023-09-13 07:28:03
 +
+

*Thread Reply:* Thanks @Paweł Leszczyński, will confirm about writing to file and get back.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Suraj Gupta + (suraj.gupta@atlan.com) +
+
2023-09-13 07:33:35
 +
+

*Thread Reply:* And yes, the issue is reproducible. Will raise an issue for this.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-09-13 07:33:54
 +
+

*Thread Reply:* even if you write onto a file?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Suraj Gupta + (suraj.gupta@atlan.com) +
+
2023-09-13 07:37:21
 +
+

*Thread Reply:* Yes, even when I write to a parquet file.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-09-13 07:49:28
 +
+

*Thread Reply:* ok. i think i was able to reproduce it locally with https://github.com/OpenLineage/OpenLineage/pull/2103/files

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Suraj Gupta + (suraj.gupta@atlan.com) +
+
2023-09-13 07:56:11
 +
+

*Thread Reply:* Opened an issue: https://github.com/OpenLineage/OpenLineage/issues/2104

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Suraj Gupta + (suraj.gupta@atlan.com) +
+
2023-09-25 16:32:09
 +
+

*Thread Reply:* @Paweł Leszczyński I see that the PR is work in progress. Any rough estimate on when we can expect this fix to be released?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-09-26 03:32:03
 +
+

*Thread Reply:* @Suraj Gupta put a comment within your issue. it's a bug we need to solve but I cannot bring any estimates today.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Suraj Gupta + (suraj.gupta@atlan.com) +
+
2023-09-26 04:33:03
 +
+

*Thread Reply:* Thanks for update @Paweł Leszczyński, also please look into this comment. It might related and I'm not sure if expected behaviour.

+
+ + + + +
Comment on #2104 [Spark Integration] Receiving duplicate OpenLineage events for a Spark Job
+
+ <p>Hi <a href="https://github.com/pawel-big-lebowski">@pawel-big-lebowski</a> I understand your point. I've also come across cases where the duplicate events have the exact same payload with no difference in facets. I'm assuming there should be some diff in the payload for the backend to merge them. I've also seen a single Job emit 7 events and there is no fixed pattern for this. <br /> +I also agree that there should be only a single START and a single COMPLETE event for any Job since this what the OL spec says.</p> + +<p>This Job emits 7 events (5 Start and 2 Complete):</p> + +<pre><code>from pyspark.sql import SparkSession +from pyspark.sql.functions import col +from pyspark.sql.functions import lit +import random + +spark = (SparkSession.builder.master('local') + .appName('spark-pipeline-v1') + .config('spark.jars.packages', "io.openlineage:openlineage_spark:1.1.0," + "mysql:mysql_connector_java:8.0.33," + "net.snowflake:snowflake_jdbc:3.13.14," + "net.snowflake:spark_snowflake_2.12:2.10.0-spark_3.2") + .config('spark.extraListeners', 'io.openlineage.spark.agent.OpenLineageSparkListener') + .config('spark.openlineage.transport.type', 'http') + .config('spark.openlineage.transport.url', '&lt;http://host.docker.internal:5009&gt;') + .config('spark.openlineage.transport.endpoint', '/events/openlineage/spark/api/v1/lineage') + .config('spark.openlineage.namespace', 'staging') + .config('spark.openlineage.transport.auth.type', 'api_key') + .config('spark.openlineage.transport.auth.apiKey', 'test-key') + .config('spark.openlineage.parentJobName', 'suraj-test-job') + .config('spark.openlineage.parentRunId', 'acd-eheh-ththth-wnjwnj') + .getOrCreate()) + + +mysql_connection_properties = { + "user": "&lt;user&gt;", + "password": "&lt;pwd&gt;", + "driver": "com.mysql.cj.jdbc.Driver", +} + +snowflake_options = { + "sfURL": "&lt;account&gt;.&lt;a href="http://snowflakecomputing.com"&gt;snowflakecomputing.com&lt;/a&gt;", + "sfUser": "&lt;usr&gt;", + "sfPassword": "&lt;pwd&gt;", + "sfDatabase": "ANALYTICS", + "sfWarehouse": "COMPUTE_WH", + "sfSchema": "PUBLIC", + "sfRole": "ACCOUNTADMIN", +} + +mysql_url = "&lt;mysql-jdbc-url&gt;" + +cats_df = spark.read.jdbc(url=mysql_url, table="cats", properties=mysql_connection_properties) +dogs_df = spark.read.jdbc(url=mysql_url, table="dogs", properties=mysql_connection_properties) + +cats_df = cats_df.withColumnRenamed("Country", "cat_name") + +joined_df = cats_df.join(dogs_df, on="owner", how="inner") + +filtered_df = joined_df.filter(col("cat_name") == 'Cookie') + +agg_df = filtered_df.groupBy("cat_name").count() + +agg_df = agg_df.withColumn("id", lit(random.randint(0, 1000))) + +agg_df.write \ + .format("snowflake") \ + .options(****snowflake_options) \ + .option("dbtable", "COUNT_TABLE") \ + .mode("overwrite") \ + .save() + +spark.stop() +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-09-13 14:20:32
 +
+

@channel +This month’s TSC meeting, open to all, is tomorrow: https://openlineage.slack.com/archives/C01CK9T7HKR/p1694113940400549

+
+ + +
+ + + } + + Michael Robinson + (https://openlineage.slack.com/team/U02LXF3HUN7) +
+ + + +
+
+ <p>@channel +This month’s TSC meeting is next Thursday the 14th at 10am PT. On the tentative agenda: +• announcements +• recent releases +• demo: Spark integration tests in Databricks runtime +• open discussion +• more (TBA) +More info and the meeting link can be found on the <a href="https://openlineage.io/meetings/">website</a>. All are welcome! Also, feel free to reply or DM me with discussion topics, agenda items, etc.</p> +
+ + + + + + +
Original URL: https://openlineage.slack.com/archives/C01CK9T7HKR/p1694113940400549
+ + +
+ + Slack Conversation +
+ + +
+ + + +
+ ✅ Sheeri Cabral (Collibra) +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Damien Hawes + (damien.hawes@booking.com) +
+
2023-09-14 06:20:15
 +
+

Context:

+ +

We use Spark with YARN, running on Hadoop 2.x (I can't remember the exact minor version) with Hive support.

+ +

Problem:

+ +

I'm noticed that CreateDataSourceAsSelectCommand objects are always transformed to an OutputDataset with a namespace value set to file - which is curious, because the inputs always have a (correct) namespace of hdfs://&lt;name-node&gt; - is this a known issue? A flaw with Apache Spark? A bug in the resolution logic?

+ +

For reference:

+ +

```public class CreateDataSourceTableCommandVisitor + extends QueryPlanVisitor<CreateDataSourceTableCommand, OpenLineage.OutputDataset> {

+ +

public CreateDataSourceTableCommandVisitor(OpenLineageContext context) { + super(context); + }

+ +

@Override + public List<OpenLineage.OutputDataset> apply(LogicalPlan x) { + CreateDataSourceTableCommand command = (CreateDataSourceTableCommand) x; + CatalogTable catalogTable = command.table();

+ +
return Collections.singletonList(
+    outputDataset()
+        .getDataset(
+            PathUtils.fromCatalogTable(catalogTable),
+            catalogTable.schema(),
+            OpenLineage.LifecycleStateChangeDatasetFacet.LifecycleStateChange.CREATE));
+
+ +

} +}`` +Running this:cat events.log | jq '{eventTime: .eventTime, eventType: .eventType, runId: .run.runId, jobNamespace: .job.namespace, jobName: .job.name, outputs: .outputs[] | {namespace: .namespace, name: .name}, inputs: .inputs[] | {namespace: .namespace, name: .name}}'`

+ +

This is an output: +{ + "eventTime": "2023-09-13T16:01:27.059Z", + "eventType": "START", + "runId": "bbbb5763-3615-46c0-95ca-1fc398c91d5d", + "jobNamespace": "spark.cluster-1", + "jobName": "ol_hadoop_test.execute_create_data_source_table_as_select_command.dhawes_db_ol_test_hadoop_tgt", + "outputs": { + "namespace": "file", + "name": "/user/hive/warehouse/dhawes.db/ol_test_hadoop_tgt" + }, + "inputs": { + "namespace": "<hdfs://nn1>", + "name": "/user/hive/warehouse/dhawes.db/ol_test_hadoop_src" + } +}

+ + + +
+ 👀 Paweł Leszczyński +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-09-14 07:32:25
 +
+

*Thread Reply:* Seems like an issue on our side. Do you know how the source is read? What LogicalPlan leaf is used to read src? Would love to find how is this done differently

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Damien Hawes + (damien.hawes@booking.com) +
+
2023-09-14 09:16:58
 +
+

*Thread Reply:* Hmm, I'll have to do explain plan to see what exactly it is.

+ +

However my sample job uses spark.sql("SELECT ** FROM dhawes.ol_test_hadoop_src")

+ +

which itself is created using

+ +

spark.sql("SELECT 1 AS id").write.format("orc").mode("overwrite").saveAsTable("dhawes.ol_test_hadoop_src")

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Damien Hawes + (damien.hawes@booking.com) +
+
2023-09-14 09:23:59
 +
+

*Thread Reply:* ``&gt;&gt;&gt; spark.sql("SELECT ** FROM dhawes.ol_test_hadoop_src").explain(True) +== Parsed Logical Plan == +'Project [**] ++- 'UnresolvedRelationdhawes.oltesthadoop_src`

+ +

== Analyzed Logical Plan == +id: int +Project [id#3] ++- SubqueryAlias dhawes.ol_test_hadoop_src + +- Relation[id#3] orc

+ +

== Optimized Logical Plan == +Relation[id#3] orc

+ +

== Physical Plan == +**(1) FileScan orc dhawes.oltesthadoop_src[id#3] Batched: true, Format: ORC, Location: InMemoryFileIndex[], PartitionFilters: [], PushedFilters: [], ReadSchema: struct<id:int>```

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
tati + (tatiana.alchueyr@astronomer.io) +
+
2023-09-14 10:03:41
 +
+

Hey everyone, +Any chance we could have a openlineage-integration-common 1.1.1 release with the following changes..? +• https://github.com/OpenLineage/OpenLineage/pull/2106 +• https://github.com/OpenLineage/OpenLineage/pull/2108

+ + + +
+ ➕ Michael Robinson, Harel Shein, Maciej Obuchowski, Jakub Dardziński, Paweł Leszczyński, Julien Le Dem +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
tati + (tatiana.alchueyr@astronomer.io) +
+
2023-09-14 10:05:19
 +
+

*Thread Reply:* Specially the first PR is affecting users of the astronomer-cosmos library: https://github.com/astronomer/astronomer-cosmos/issues/533

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-09-14 10:05:24
 +
+

*Thread Reply:* Thanks @tati for requesting your first OpenLineage release! Three +1s from committers will authorize

+ + + +
+ :gratitude_thank_you: tati +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-09-14 11:59:55
 +
+

*Thread Reply:* The release is authorized and will be initiated within two business days.

+ + + +
+ 🎉 tati +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
tati + (tatiana.alchueyr@astronomer.io) +
+
2023-09-15 04:40:12
 +
+

*Thread Reply:* Thanks a lot, @Michael Robinson!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2023-09-14 20:23:01
 +
+

Per discussion in the OpenLineage sync today here is a very early strawman proposal for an OpenLineage registry that producers and consumers could be registered in. +Feedback or alternate proposals welcome +https://docs.google.com/document/d/1zIxKST59q3I6ws896M4GkUn7IsueLw8ejct5E-TR0vY/edit +Once this is sufficiently fleshed out, I’ll create an actual proposal on github

+ + + +
+ 👍 Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2023-10-03 20:33:35
 +
+

*Thread Reply:* I have cleaned up the registry proposal. +https://docs.google.com/document/d/1zIxKST59q3I6ws896M4GkUn7IsueLw8ejct5E-TR0vY/edit +In particular: +• I clarified that option 2 is preferred at this point. +• I moved discussion notes to the bottom. they will go away at some point +• Once it is stable, I’ll create a proposal with the preferred option. +• we need a good proposal for the core facets prefix. My suggestion is to move core facets to core in the registry. The drawback is prefix would be inconsistent.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2023-10-05 17:34:12
 +
+

*Thread Reply:* I have created a ticket to make this easier to find. Once I get more feedback I’ll turn it into a md file in the repo: https://docs.google.com/document/d/1zIxKST59q3I6ws896M4GkUn7IsueLw8ejct5E-TR0vY/edit#heading=h.enpbmvu7n8gu +https://github.com/OpenLineage/OpenLineage/issues/2161

+
+ + + + +
#2161 [PROPOSAL] Add a Registry of Producers and Consumers in OpenLineage
+
+ <p><strong>Purpose</strong></p> + +<p>This is the early stage of an idea to get community feedback on what an OpenLineage registry for producers, custom facets and consumers could be. Once this document is stable enough, I’ll create an official proposal on the OpenLineage repo.</p> + +<p><strong>Goal</strong></p> + +<p>Allow third parties to register their implementations or custom extensions to make them easy to discover. <br /> +Shorten “Producer” and “schema url” values</p> + +<p><strong>Proposed implementation</strong></p> + +<p>Current draft for discussion:</p> + +<p><a href="https://docs.google.com/document/d/1zIxKST59q3I6ws896M4GkUn7IsueLw8ejct5E-TR0vY/edit#heading=h.br8d2vy9wme9">https://docs.google.com/document/d/1zIxKST59q3I6ws896M4GkUn7IsueLw8ejct5E-TR0vY/edit#heading=h.br8d2vy9wme9</a></p> +
+ +
+
Labels
+ proposal +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-09-15 12:03:27
 +
+

@channel +Friendly reminder: the next OpenLineage meetup, our first in Toronto, is happening this coming Monday at 5 PM ET https://openlineage.slack.com/archives/C01CK9T7HKR/p1694441261486759

+
+ + +
+ + + } + + Michael Robinson + (https://openlineage.slack.com/team/U02LXF3HUN7) +
+ + + +
+
+ <p>@channel +The first Toronto OpenLineage Meetup, featuring a presentation by recent adopter <a href="https://metaphor.io/">Metaphor</a>, is just one week away. On the agenda:</p> + +<ol><li><strong>Evolution of spec presentation/discussion (project background/history)</strong></li><li><strong>State of the community</strong></li><li><strong>Integrating OpenLineage with <a href="https://metaphor.io/">Metaphor</a> (by special guests <a href="https://www.linkedin.com/in/yeliu84/">Ye</a> &amp; <a href="https://www.linkedin.com/in/ivanperepelitca/">Ivan</a>)</strong></li><li><strong>Spark/Column lineage update</strong></li><li><strong>Airflow Provider update</strong></li><li><strong>Roadmap Discussion</strong> +<strong>Find more details and RSVP <a href="https://www.meetup.com/openlineage/events/295488014/?utm_medium=referral&amp;utm_campaign=share-btn_savedevents_share_modal&amp;utm_source=link|here">https://www.meetup.com/openlineage/events/295488014/?utm<em>medium=referral&amp;utm</em>campaign=share-btn<em>savedevents</em>share<em>modal&amp;utm</em>source=link|here</a></strong>.</li> +</ol> +
+ + + + + + +
Original URL: https://openlineage.slack.com/archives/C01CK9T7HKR/p1694441261486759
+ + +
+ + Slack Conversation +
+ + +
+ + + +
+ 👍 Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Guntaka Jeevan Paul + (jeevan@acceldata.io) +
+
2023-09-16 03:30:27
 +
+

@here we have dataproc operator getting called from a dag which submits a spark job, we wanted to maintain that continuity of parent job in the spark job so according to the documentation we can acheive that by using a macro called lineagerunid that requires task and taskinstance as the parameters. The problem we are facing is that our client’s have 1000's of dags, so asking them to change this everywhere it is used is not feasible, so we thought of using the taskpolicy feature in the airflow…but the problem is that taskpolicy gives you access to only the task/operator, but we don’t have the access to the task instance..that is required as a parameter to the lineagerun_id function. Can anyone kindly help us on how should we go about this one +t1 = DataProcPySparkOperator( + task_id=job_name, + <b>#required</b> pyspark configuration, + job_name=job_name, + dataproc_pyspark_properties={ + 'spark.driver.extraJavaOptions': + f"-javaagent:{jar}={os.environ.get('OPENLINEAGE_URL')}/api/v1/namespaces/{os.getenv('OPENLINEAGE_NAMESPACE', 'default')}/jobs/{job_name}/runs/{{{{macros.OpenLineagePlugin.lineage_run_id(task, task_instance)}}}}?api_key={os.environ.get('OPENLINEAGE_API_KEY')}" + dag=dag)

+ + + +
+ ➕ Abdallah +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-09-16 04:22:47
 +
+

*Thread Reply:* you don't need actual task instance to do that. you only should set additional argument as jinja template, same as above

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-09-16 04:25:28
 +
+

*Thread Reply:* task_instance in this case is just part of string which is evaluated when jinja render happens

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Guntaka Jeevan Paul + (jeevan@acceldata.io) +
+
2023-09-16 04:27:10
 +
+

*Thread Reply:* ohh…then we could use the same example as above inside the task_policy to intercept the Operator and add the openlineage specific additions properties?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-09-16 04:30:59
 +
+

*Thread Reply:* correct, just remember not to override all properties, just add ol specific

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Guntaka Jeevan Paul + (jeevan@acceldata.io) +
+
2023-09-16 04:32:02
 +
+

*Thread Reply:* yeah sure…thank you so much @Jakub Dardziński, will try this out and keep you posted

+ + + +
+ 👍 Jakub Dardziński +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-09-16 05:00:24
 +
+

*Thread Reply:* We want to automate setting those options at some point inside the operator itself

+ + + +
+ ➕ Guntaka Jeevan Paul +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Guntaka Jeevan Paul + (jeevan@acceldata.io) +
+
2023-09-16 19:40:27
 +
+

@here is there a way by which we could add custom headers to openlineage client in airflow, i see that provision is there for spark integration via these properties spark.openlineage.transport.headers.xyz --> abcdef

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-09-19 16:40:55
 +
+

*Thread Reply:* there’s no out-of-the-box possibility to do that yet, you’re very welcome to create an issue in GitHub and maybe contribute as well! 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mars Lan + (mars@metaphor.io) +
+
2023-09-17 09:07:41
 +
+

It doesn't seem like there's a way to override the OL endpoint from the default (/api/v1/lineage) in Airflow? I tried setting the OPENLINEAGE_ENDPOINT environment to no avail. Based on this statement, it seems that only OPENLINEAGE_URL was used to construct HttpConfig ?

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/client/python/openlineage/client/transport/factory.py | factory.py>
+
+ <pre><code> config = HttpConfig( + url=os.environ["OPENLINEAGE_URL"], + auth=create_token_provider( + { + "type": "api_key", + "apiKey": os.environ.get("OPENLINEAGE_API_KEY", ""), + }, + ), +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-09-18 16:25:11
 +
+

*Thread Reply:* That’s correct. For now there’s no way to configure the endpoint via env var. You can do that by using config file

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mars Lan + (mars@metaphor.io) +
+
2023-09-18 16:30:39
 +
+

*Thread Reply:* How do you do that in Airflow? Any particular reason for excluding endpoint override via env var? Happy to create a PR to fix that.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-09-18 16:52:48
 +
+

*Thread Reply:* historical I guess? go for the PR, of course 🚀

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mars Lan + (mars@metaphor.io) +
+
2023-10-03 08:52:16
 +
+

*Thread Reply:* https://github.com/OpenLineage/OpenLineage/pull/2151

+
+ + + + +
#2151 Allow setting client's endpoint via environment variable
+
+ <p><strong>Problem</strong></p> + +<p>Currently, it's not possible to set the OpenLineage endpoint (hard-coded to <code>/api/v1/lineage</code>) using an environment variable when running the Airflow integration.</p> + +<p><strong>Solution</strong></p> + +<p>Given that it's not possible to create the client manually in Airflow, especially now that OpenLineage has become an official Airflow provider, this change seems like the only feasible solution.</p> + +<p>☐ Your change modifies the <a href="https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.json">core</a> OpenLineage model +☐ Your change modifies one or more OpenLineage <a href="https://github.com/OpenLineage/OpenLineage/tree/main/spec/facets">facets</a></p> + +<p><strong>One-line summary:</strong></p> + +<p>Allow setting client's endpoint via environment variable.</p> + +<p><strong>Checklist</strong></p> + +<p>☑︎ You've <a href="https://github.com/OpenLineage/OpenLineage/blob/main/why-the-dco.md">signed-off</a> your work +☑︎ Your pull request title follows our <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CONTRIBUTING.md#creating-pull-requests">guidelines</a> +☑︎ Your changes are accompanied by tests (<em>if relevant</em>) +☑︎ Your change contains a <a href="https://kurtisnusbaum.medium.com/stacked-diffs-keeping-phabricator-diffs-small-d9964f4dcfa6">small diff</a> and is self-contained +☑︎ You've updated any relevant documentation (<em>if relevant</em>) +☑︎ Your comment includes a one-liner for the changelog about the specific purpose of the change (<em>if necessary</em>) +☐ You've versioned the core OpenLineage model or facets according to <a href="https://docs.snowplowanalytics.com/docs/pipeline-components-and-applications/iglu/common-architecture/schemaver">SchemaVer</a> (<em>if relevant</em>) +☐ You've added a <a href="https://github.com/OpenLineage/OpenLineage/tree/main/.github/header_templates.md">header</a> to source files (<em>if relevant</em>)</p> + +<hr /> + +<p>SPDX-License-Identifier: Apache-2.0 <br /> +Copyright 2018-2023 contributors to the OpenLineage project</p> +
+ +
+
Labels
+ documentation, client/python +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Terese Larsson + (terese@jclab.se) +
+
2023-09-18 08:22:34
 +
+

Hi! I'm in need of help with wrapping my head around OpenLineage. My team have the goal of collecting metadata from the Airflow operators GreatExpectationsOperator, PythonOperator, MsSqlOperator and BashOperator (for dbt). Where can I see the sourcecode for what is collected for each operator, and is there support for these in the new provider apache-airflow-providers-openlineage? I am super confused and feel lost in the docs. 🤯 We are using MSSQL/ODBC to connect to our db, and this data does not seem to appear as datasets in Marquez, do I need to configure this? If so, HOW and WHERE? 🥲

+ +

Happy for any help, big or small! 🙏

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-09-18 16:26:07
 +
+

*Thread Reply:* there’s no actual single source of what integrations are currently implemented in openlineage Airflow provider. That’s something we should work on so it’s more visible

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-09-18 16:26:46
 +
+

*Thread Reply:* answering this quickly - GE & MS SQL are not currently implemented yet in the provider

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-09-18 16:26:58
 +
+

*Thread Reply:* but I also invite you to contribute if you’re interested! 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
sarathch + (sarathch@hpe.com) +
+
2023-09-19 02:47:47
 +
+

Hi I need help in extracting OpenLineage for PostgresOperator in json format. +any suggestions or comments would be greatly appreciated

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-09-19 16:40:06
 +
+

*Thread Reply:* If you're using Airflow 2.7, take a look at https://airflow.apache.org/docs/apache-airflow-providers-openlineage/stable/guides/user.html

+ + + +
+ ❤️ sarathch +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-09-19 16:40:54
 +
+

*Thread Reply:* If you use one of the lower versions, take a look here https://openlineage.io/docs/integrations/airflow/usage

+
+
openlineage.io
+ + + +
Using the Airflow integration | OpenLineage
+
+ PREREQUISITES +
+ + + + + + +
Original URL: https://openlineage.io/docs/integrations/airflow/usage
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
sarathch + (sarathch@hpe.com) +
+
2023-09-20 06:26:56
 +
+

*Thread Reply:* Maciej, +Thanks for sharing the link https://airflow.apache.org/docs/apache-airflow-providers-openlineage/stable/guides/user.html +this should address the issue

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Juan Luis Cano Rodríguez + (juan_luis_cano@mckinsey.com) +
+
2023-09-20 09:36:54
 +
+

congrats folks 🥳 https://lfaidata.foundation/blog/2023/09/20/lf-ai-data-foundation-announces-graduation-of-openlineage-project

+ + + +
+ 🎉 Jakub Dardziński, Mars Lan, Ross Turk, Guntaka Jeevan Paul, Peter Hicks, Maciej Obuchowski, Athitya Kumar, John Lukenoff, Harel Shein, Francis McGregor-Macdonald, Laurent Paris +
+ +
+ 👍 Athitya Kumar +
+ +
+ ❤️ Harel Shein +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-09-20 17:08:58
 +
+

@channel +We released OpenLineage 1.2.2! +Added +• Spark: publish the ProcessingEngineRunFacet as part of the normal operation of the OpenLineageSparkEventListener #2089 @d-m-h +• Spark: capture and emit spark.databricks.clusterUsageTags.clusterAllTags variable from databricks environment #2099 @Anirudh181001 +Fixed +• Common: support parsing dbt_project.yml without target-path #2106 @tatiana +• Proxy: fix Proxy chart #2091 @harels +• Python: fix serde filtering #2044 @xli-1026 +• Python: use non-deprecated apiKey if loading it from env variables @2029 @mobuchowski +• Spark: Improve RDDs on S3 integration. #2039 @pawel-big-lebowski +• Flink: prevent sending running events after job completes #2075 @pawel-big-lebowski +• Spark & Flink: Unify dataset naming from URI objects #2083 @pawel-big-lebowski +• Spark: Databricks improvements #2076 @pawel-big-lebowski +Removed +• SQL: remove sqlparser dependency from iface-java and iface-py #2090 @JDarDagran +Thanks to all the contributors, including new contributors @tati, @xli-1026, and @d-m-h! +Release: https://github.com/OpenLineage/OpenLineage/releases/tag/1.2.2 +Changelog: https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md +Commit history: https://github.com/OpenLineage/OpenLineage/compare/1.1.0...1.2.2 +Maven: https://oss.sonatype.org/#nexus-search;quick~openlineage +PyPI: https://pypi.org/project/openlineage-python/

+ + + +
+ 🔥 Maciej Obuchowski, Harel Shein, Anirudh Shrinivason +
+ +
+ 👍 Guntaka Jeevan Paul, John Rosenbaum, Sangeeta Mishra +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Yevhenii Soboliev + (esoboliev@griddynamics.com) +
+
2023-09-22 21:05:20
 +
+

*Thread Reply:* Hi @Michael Robinson Thank you! I love the job that you’ve done. If you have a few seconds, please hint at how I can push lineage gathered from Airflow and Spark jobs into DataHub for visualization? I didn’t find any solutions or official support neither at Openlineage nor at DataHub, but I still want to continue using Openlineage

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-09-22 21:30:22
 +
+

*Thread Reply:* Hi Yevhenii, thank you for using OpenLineage. The DataHub integration is new to us, but perhaps the experts on Spark and Airflow know more. @Paweł Leszczyński @Maciej Obuchowski @Jakub Dardziński

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-09-23 08:11:17
 +
+

*Thread Reply:* @Yevhenii Soboliev at Airflow Summit, Shirshanka Das from DataHub mentioned this as upcoming feature.

+ + + +
+ 👍 Yevhenii Soboliev +
+ +
+ 🎯 Yevhenii Soboliev +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Suraj Gupta + (suraj.gupta@atlan.com) +
+
2023-09-21 02:11:10
 +
+

Hi, we're using Custom Operators in airflow(2.5) and are planning to expose lineage via default extractors: https://openlineage.io/docs/integrations/airflow/default-extractors/ +Question: Now if we upgrade our Airflow version to 2.7 in the future, would our code be backward compatible? +Since OpenLineage has now moved inside airflow and I think there is no concept of extractors in the latest version.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Suraj Gupta + (suraj.gupta@atlan.com) +
+
2023-09-21 02:15:00
 +
+

*Thread Reply:* Also, do we have any docs on how OL works with the latest airflow version? Few questions: +• How is it replacing the concept of custom extractors and Manually Annotated Lineage in the latest version? +• Do we have any examples of setting up the integration to emit input/output datasets for non supported Operators like PythonOperator?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-09-27 10:04:09
 +
+

*Thread Reply:* > Question: Now if we upgrade our Airflow version to 2.7 in the future, would our code be backward compatible? +It will be compatible, “default extractors” is generally the same concept as we’re using in the 2.7 integration. +One thing that might be good to update is import paths, from openlineage.airflow to airflow.providers.openlineage but should work both ways

+ +

> • Do we have any code samples/docs of setting up the integration to emit input/output datasets for non supported Operators like PythonOperator? +Our experience with that is currently lacking - this means, it works like in bare airflow, if you annotate your PythonOperator tasks with old Airflow lineage like in this doc.

+ +

We want to make this experience better - by doing few things +• instrumenting hooks, then collecting lineage from them +• integration with AIP-48 datasets +• allowing to emit lineage collected inside Airflow task by other means, by providing core Airflow API for that +All those things require changing core Airflow in a couple of ways: +• tracking which hooks were used during PythonOperator execution +• just being able to emit datasets (airflow inlets/outlets) from inside of a task - they are now a static thing, so if you try that it does not work +• providing better API for emitting that lineage, preferably based on OpenLineage itself rather than us having to convert that later. +As this requires core Airflow changes, it won’t be live until Airflow 2.8 at the earliest.

+ +

thanks to @Maciej Obuchowski for this response

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason Yip + (jasonyip@gmail.com) +
+
2023-09-21 18:36:17
 +
+

I am using this accelerator that leverages OpenLineage on Databricks to publish lineage info to Purview, but it's using a rather old version of OpenLineage aka 0.18, anybody has tried it on a newer version of OpenLineage? I am facing some issues with the inputs and outputs for the same object is having different json +https://github.com/microsoft/Purview-ADB-Lineage-Solution-Accelerator/

+
+ + + + +
microsoft/Purview-ADB-Lineage-Solution-Accelerator
+
+ A connector to ingest Azure Databricks lineage into Microsoft Purview +
+ +
+
Stars
+ 77 +
+ +
+
Language
+ C# +
+ + + + + + + + +
+ + + +
+ ✅ Harel Shein +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason Yip + (jasonyip@gmail.com) +
+
2023-09-21 21:51:41
 +
+

I installed 1.2.2 on Databricks, followed the below init script: https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/databricks/open-lineage-init-script.sh

+ +

my cluster config looks like this:

+ +

spark.openlineage.version v1 +spark.openlineage.namespace adb-5445974573286168.8#default +spark.openlineage.endpoint v1/lineage +spark.openlineage.url.param.code 8kZl0bo2TJfnbpFxBv-R2v7xBDj-PgWMol3yUm5iP1vaAzFu9kIZGg== +spark.openlineage.url https://f77b-50-35-69-138.ngrok-free.app

+ +

But it is not calling the API, it works fine with 0.18 version

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/databricks/open-lineage-init-script.sh | open-lineage-init-script.sh>
+
+ <pre><code>#!/bin/bash +# +# Copyright 2018-2023 contributors to the OpenLineage project +# SPDX-License-Identifier: Apache-2.0 + +STAGE_DIR="/dbfs/databricks/openlineage" + +echo "BEGIN: Upload Spark Listener JARs" +cp -f $STAGE_DIR/openlineage-spark-**.jar /mnt/driver-daemon/jars || { echo "Error copying Spark Listener library file"; exit 1;} +echo "END: Upload Spark Listener JARs" + +echo "BEGIN: Modify Spark config settings" +cat &lt;&lt; 'EOF' &gt; /databricks/driver/conf/openlineage-spark-driver-defaults.conf +[driver] { + "spark.extraListeners" = "io.openlineage.spark.agent.OpenLineageSparkListener" +} +EOF +echo "END: Modify Spark config settings" + +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+ ✅ Harel Shein +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason Yip + (jasonyip@gmail.com) +
+
2023-09-21 23:16:10
 +
+

I am attaching the log4j, there is no openlineagecontext

+ +
+
log4j-active (3).txt
+ + + + + + +
+ + +
+ ✅ Harel Shein +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason Yip + (jasonyip@gmail.com) +
+
2023-09-21 23:47:22
 +
+

*Thread Reply:* this issue is resolved, solution can be found here: https://openlineage.slack.com/archives/C01CK9T7HKR/p1691592987038929

+
+ + +
+ + + } + + Zahi Fail + (https://openlineage.slack.com/team/U05KNSP01TR) +
+ + + +
+
+ <p>Hey, +I’m running Spark application (spark version 3.4) with OL integration. +I changed spark to use “debug” level, and I see the OL events with the below message: +“Emitting lineage completed successfully:”</p> + +<p>With all the above, I can’t see the event in Marquez.</p> + +<p>Attaching the OL configurations. +When changing the OL-spark version to 0.6.+, I do see event created in Marquez with only “Start” status (attached below).</p> + +<p>The OL-spark version is matching the Spark version? Is there a known issues with the Spark / OL versions ?</p> +
+ + + + + + +
Original URL: https://openlineage.slack.com/archives/C01CK9T7HKR/p1691592987038929
+ + +
+ + Thread in Slack Conversation +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harel Shein + (harel.shein@gmail.com) +
+
2023-09-25 08:59:10
 +
+

*Thread Reply:* We were all out at Airflow Summit last week, so apologies for the delayed response. Glad you were able to resolve the issue!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sangeeta Mishra + (sangeeta@acceldata.io) +
+
2023-09-25 05:11:50
 +
+

@here I'm presently addressing a particular scenario that pertains to Openlineage authentication, specifically involving the use of an access key and secret.

+ +

I've implemented a custom token provider called AccessKeySecretKeyTokenProvider, which extends the TokenProvider class. This token provider communicates with another service, obtaining a token and an expiration time based on the provided access key, secret, and client ID.

+ +

My goal is to retain this token in a cache prior to its expiration, thereby eliminating the need for network calls to the third-party service. Is it possible without relying on an external caching system.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harel Shein + (harel.shein@gmail.com) +
+
2023-09-25 08:56:53
 +
+

*Thread Reply:* Hey @Sangeeta Mishra, I’m not sure that I fully understand your question here. What do you mean by OpenLineage authentication? +What are you using to generate OL events? What’s your OL receiving backend?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sangeeta Mishra + (sangeeta@acceldata.io) +
+
2023-09-25 09:04:33
 +
+

*Thread Reply:* Hey @Harel Shein, +I wanted to clarify the previous message. I apologize for any confusion. When I mentioned "OpenLineage authentication," I was actually referring to the authentication process for the OpenLineage backend, specifically using HTTP transport. This involves using my custom token provider, which utilizes access keys and secrets for authentication. The OL backend is http based backend . I hope this clears things up!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harel Shein + (harel.shein@gmail.com) +
+
2023-09-25 09:05:12
 +
+

*Thread Reply:* Are you using Marquez?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sangeeta Mishra + (sangeeta@acceldata.io) +
+
2023-09-25 09:05:55
 +
+

*Thread Reply:* We are trying to leverage our own backend here.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harel Shein + (harel.shein@gmail.com) +
+
2023-09-25 09:07:03
 +
+

*Thread Reply:* I see.. I’m not sure the OpenLineage community could help here. Which webserver framework are you using?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sangeeta Mishra + (sangeeta@acceldata.io) +
+
2023-09-25 09:08:56
 +
+

*Thread Reply:* KTOR framework

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sangeeta Mishra + (sangeeta@acceldata.io) +
+
2023-09-25 09:15:33
 +
+

*Thread Reply:* Our backend authentication operates based on either a pair of keys or a single bearer token, with a limited time of expiry. Hence, wanted to cache this information inside the token provider.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harel Shein + (harel.shein@gmail.com) +
+
2023-09-25 09:26:57
 +
+

*Thread Reply:* I see, I would ask this question here https://ktor.io/support/

+
+
Ktor Framework
+ + + +
Support
+
+ Kotlin Server and Client Framework for microservices, HTTP APIs, and RESTful services +
+ + + + + + + + +
Original URL: https://ktor.io/support/
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sangeeta Mishra + (sangeeta@acceldata.io) +
+
2023-09-25 10:12:52
 +
+

*Thread Reply:* Thank you

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-09-26 04:13:20
 +
+

*Thread Reply:* @Sangeeta Mishra which openlineage client are you using: java or python?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Sangeeta Mishra + (sangeeta@acceldata.io) +
+
2023-09-26 04:19:53
 +
+

*Thread Reply:* @Paweł Leszczyński I am using python client

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Suraj Gupta + (suraj.gupta@atlan.com) +
+
2023-09-25 13:36:25
 +
+

I'm using the Spark OpenLineage integration. In the outputStatistics output dataset facet we receive rowCount and size. +The Job performs a SQL insert into a MySQL table and I'm receiving the size as 0. +{ + "outputStatistics": + { + "_producer": "<https://github.com/OpenLineage/OpenLineage/tree/1.1.0/integration/spark>", + "_schemaURL": "<https://openlineage.io/spec/facets/1-0-0/OutputStatisticsOutputDatasetFacet.json#/$defs/OutputStatisticsOutputDatasetFacet>", + "rowCount": 1, + "size": 0 + } +} +I'm not sure what the size means here. Does this mean number of bytes inserted/updated? +Also, do we have any documentation for Spark specific Job and Run facets?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-09-27 09:56:00
 +
+

*Thread Reply:* I am not sure it's stated in the doc. Here's the list of spark facets schemas: https://github.com/OpenLineage/OpenLineage/tree/main/integration/spark/shared/facets/spark/v1

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Guntaka Jeevan Paul + (jeevan@acceldata.io) +
+
2023-09-26 00:51:30
 +
+

@here In Airflow Integration we send across a lineage Event for Dag start and complete, but that is not the case with spark integration…we don’t receive any event for the application start and complete in spark…is this expected behaviour or am i missing something?

+ + + +
+ ➕ Suraj Gupta +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-09-27 09:47:39
 +
+

*Thread Reply:* For spark we do send start and complete for each spark action being run (single operation that causes spark processing being run). However, it is difficult for us to know if we're dealing with the last action within spark job or a spark script.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-09-27 09:49:35
 +
+

*Thread Reply:* I think we need to look deeper into that as there is reoccuring need to capture such information

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-09-27 09:49:57
 +
+

*Thread Reply:* and spark listener event has methods like onApplicationStart and onApplicationEnd

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Guntaka Jeevan Paul + (jeevan@acceldata.io) +
+
2023-09-27 09:50:13
 +
+

*Thread Reply:* We are using the SparkListener, which has a function called OnApplicationStart which gets called whenever a spark application starts, so i was thinking why cant we send one at start and simlarly at end as well

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-09-27 09:50:33
 +
+

*Thread Reply:* additionally, we would like to have a concept of a parent run for a spark job which aggregates all actions run within a single spark job context

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Guntaka Jeevan Paul + (jeevan@acceldata.io) +
+
2023-09-27 09:51:11
 +
+

*Thread Reply:* yeah exactly. the way that it works with airflow integration

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-09-27 09:51:26
 +
+

*Thread Reply:* we do have an issue for that https://github.com/OpenLineage/OpenLineage/issues/2105

+
+ + + + +
#2105 [PROPOSAL] Spark jobs should have a parent Job describing the Run State of Spark pipeline.
+
+ <p><strong>Purpose:</strong> <br /> +Currently there is no way to group spark jobs running within the same SparkContext since there is no concept of parent Jobs in Spark. In airflow we support Parent Runs and this way we can group Tasks together within a single DAG.</p> + +<p><strong>Use cases:</strong></p> + +<p>• For data catalog based OL backends, a user might want to search for Spark ETL pipelines first and then dive deeper into a specific pipeline to view the Spark Jobs. This is similar to how a user can search for airflow DAG assets and then look into the Tasks for that DAG. +• Also this will help display operational metadata for Spark pipelines. For Airflow we have DAG run status (parent job status), but for Spark we don't have any such concept to display the completion state of a pipeline run. We can only track it at SparkJob level.</p> + +<p><strong>Proposed implementation</strong> <br /> +This should be similar to the airflow implementation.</p> +
+ +
+
Labels
+ proposal +
+ +
+
Comments
+ 2 +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-09-27 09:52:08
 +
+

*Thread Reply:* what you can is: come to our monthly Openlineage open meetings and raise that issue and convince the community about its importance

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Guntaka Jeevan Paul + (jeevan@acceldata.io) +
+
2023-09-27 09:53:32
 +
+

*Thread Reply:* yeah sure would love to do that…how can i join them, will that be posted here in this slack channel?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-09-27 09:54:08
 +
+

*Thread Reply:* Hi, you can see the schedule and RSVP here: https://openlineage.io/community

+
+
openlineage.io
+ + + +
Community | OpenLineage
+
+ Data lineage is the foundation for a new generation of powerful, context-aware data tools and best practices. OpenLineage enables consistent collection of lineage metadata, creating a deeper understanding of how data is produced and used. +
+ + + + + + +
Original URL: https://openlineage.io/community
+ + + +
+ + + +
+ 🙌 Paweł Leszczyński +
+ +
+ :gratitude_thank_you: Guntaka Jeevan Paul +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-09-27 11:19:16
 +
+

Meetup recap: Toronto Meetup @ Airflow Summit, September 18, 2023 +It was great to see so many members of our community at this event! I counted 32 total attendees, with all but a handful being first-timers. +Topics included: +• Presentation on the history, architecture and roadmap of the project by @Julien Le Dem and @Harel Shein +• Discussion of OpenLineage support in Marquez by @Willy Lulciuc +• Presentation by Ye Liu and Ivan Perepelitca from Metaphor, the social platform for data, about their integration +• Presentation by @Paweł Leszczyński about the Spark integration +• Presentation by @Maciej Obuchowski about the Apache Airflow Provider +Thanks to all the presenters and attendees with a shout out to @Harel Shein for the help with organizing and day-of logistics, @Jakub Dardziński for the help with set up/clean up, and @Sheeri Cabral (Collibra) for the crucial assist with the signup sheet. +This was our first meetup in Toronto, and we learned some valuable lessons about planning events in new cities — the first and foremost being to ask for a pic of the building! 🙂 But it seemed like folks were undeterred, and the space itself lived up to expectations. +For a recording and clips from the meetup, head over to our YouTube channel. +Upcoming events: +• October 5th in San Francisco: Marquez Meetup @ Astronomer (sign up https://www.meetup.com/meetup-group-bnfqymxe/events/295444209/?utmmedium=referral&utmcampaign=share-btnsavedeventssharemodal&utmsource=link|here) +• November: Warsaw meetup (details, date TBA) +• January: London meetup (details, date TBA) +Are you interested in hosting or co-hosting an OpenLineage or Marquez meetup? DM me!

+
+
metaphor.io
+ + + +
Metaphor - The Social Platform for Data
+
+ Making Data Actionable, At Scale - Designed for data teams building cloud-native, self-service data platforms for their business users. Explore our Data Governance, Data Lineage, Data Discovery, and Data Trust capabilities today. +
+ + + + + + + + +
Original URL: https://metaphor.io/
+ + + +
+
+
YouTube
+ + + +
OpenLineage Project
+
+ Meetings, talks and tutorials by the OpenLineage Project, an Open Standard for lineage metadata collection +
+ + + + + + + + +
Original URL: https://www.youtube.com/channel/UCRMLy4AaSw_ka-gNV9nl7VQ/
+ + + +
+
+
Meetup
+ + + +
Marquez Meetup @ Astronomer, Thu, Oct 5, 2023, 5:30 PM | Meetup
+
+ Join us on Thursday, October 5th, from 5:30_8:30 pm to learn about the Marquez project. Meet other members of the community, get tips on making the most of the latest impro +
+ + + + + + + + +
Original URL: https://www.meetup.com/meetup-group-bnfqymxe/events/295444209/?utm_medium=referral&amp;utm_campaign=share-btn_savedevents_share_modal&amp;utm_source=link
+ + + +
+ +
+
IMG_5456.jpg
+ + + + + + + + +
+
+
IMG_5452.jpg
+ + + + + + + + +
+
+
20230918_171849.jpg
+ + + + + + + + +
+
+
20230918_192522.jpg
+ + + + + + + + +
+
+
20230918_185612.jpg
+ + + + + + + + +
+ + +
+ 🙌 Mars Lan, Harel Shein, Paweł Leszczyński +
+ +
+ ❤️ Jakub Dardziński, Harel Shein, Rodrigo Maia, Paweł Leszczyński, Julien Le Dem, Willy Lulciuc +
+ +
+ 🚀 Jakub Dardziński, Kevin Languasco +
+ +
+ 😅 Harel Shein +
+ +
+ ✅ Sheeri Cabral (Collibra) +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-09-27 11:55:47
 +
+

*Thread Reply:* A few more pics:

+ +
+
IMG_5462.jpg
+ + + + + + + + +
+
+
20230918_172756.jpg
+ + + + + + + + +
+
+
20230918_172935.jpg
+ + + + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Damien Hawes + (damien.hawes@booking.com) +
+
2023-09-27 12:23:05
 +
+

Hi folks, am I correct in my observations that the Spark integration does not generate inputs and outputs for Kafka-to-Kafka pipelines?

+ +

EDIT: Removed the crazy wall of text. Relevant GitHub issue is here.

+
+ + + + +
#2137 [SPARK] The integration fails to produce inputs and outputs in a Kafka-to-Kafka scenario
+
+ <p><strong>Problem Statement</strong></p> + +<p>As the title suggests, when a Spark job is running in a streaming configuration that performs a Kafka-to-Kafka data flow, the integration fails to emit any events.</p> + +<p>Here's an example of an event that the connector emits:</p> + +<p>The connector emits the following output:</p> + +<pre><code>23/09/27 18:17:22 DEBUG OpenLineageRunEventBuilder: Visiting query plan Optional[== Parsed Logical Plan == +WriteToMicroBatchDataSource org.apache.spark.sql.kafka010.KafkaSourceProvider$KafkaTable@2e679177, c3df858a-d481-456f-9665-3ff0c2d6d19a, [kafka.bootstrap.servers=localhost:9092,localhost:9093,localhost:9094, topic=target, checkpointLocation=/Users/dhawes/Projects/spark-streaming-openlineage], Append, 0 ++- StreamingDataSourceV2Relation [key#7, value#8, topic#9, partition#10, offset#11L, timestamp#12, timestampType#13], org.apache.spark.sql.kafka010.KafkaSourceProvider$KafkaScan@2095a606, KafkaV2[Subscribe[source]], {"source":{"0":11}}, {"source":{"0":11}} + +== Analyzed Logical Plan == +WriteToMicroBatchDataSource org.apache.spark.sql.kafka010.KafkaSourceProvider$KafkaTable@2e679177, c3df858a-d481-456f-9665-3ff0c2d6d19a, [kafka.bootstrap.servers=localhost:9092,localhost:9093,localhost:9094, topic=target, checkpointLocation=/Users/dhawes/Projects/spark-streaming-openlineage], Append, 0 ++- StreamingDataSourceV2Relation [key#7, value#8, topic#9, partition#10, offset#11L, timestamp#12, timestampType#13], org.apache.spark.sql.kafka010.KafkaSourceProvider$KafkaScan@2095a606, KafkaV2[Subscribe[source]], {"source":{"0":11}}, {"source":{"0":11}} + +== Optimized Logical Plan == +WriteToDataSourceV2 org.apache.spark.sql.execution.streaming.sources.MicroBatchWrite@6b69d31b ++- StreamingDataSourceV2Relation [key#7, value#8, topic#9, partition#10, offset#11L, timestamp#12, timestampType#13], org.apache.spark.sql.kafka010.KafkaSourceProvider$KafkaScan@2095a606, KafkaV2[Subscribe[source]], {"source":{"0":11}}, {"source":{"0":11}} + +== Physical Plan == +WriteToDataSourceV2 org.apache.spark.sql.execution.streaming.sources.MicroBatchWrite@6b69d31b, org.apache.spark.sql.execution.datasources.v2.DataSourceV2Strategy$$Lambda$1945/0x000000c001c246c8@28b23a0a ++- **(1) Project [key#7, value#8, topic#9, partition#10, offset#11L, timestamp#12, timestampType#13] + +- MicroBatchScan[key#7, value#8, topic#9, partition#10, offset#11L, timestamp#12, timestampType#13] class org.apache.spark.sql.kafka010.KafkaSourceProvider$KafkaScan +] with output dataset builders [&lt;function1&gt;, &lt;function1&gt;, &lt;function1&gt;, &lt;function1&gt;, &lt;function1&gt;, &lt;function1&gt;, &lt;function1&gt;, &lt;function1&gt;] + +23/09/27 18:17:22 INFO ConsoleTransport: {"eventTime":"2023_09_27T16:17:22.263Z","producer":"&lt;https://github.com/OpenLineage/OpenLineage/tree/1.2.2/integration/spark&gt;","schemaURL":"&lt;https://openlineage.io/spec/2-0-2/OpenLineage.json#/$defs/RunEvent&gt;","eventType":"COMPLETE","run":{"runId":"3468112f_96c3_44b5_bd84_847c2d99db8f","facets":{"spark_version":{"_producer":"&lt;https://github.com/OpenLineage/OpenLineage/tree/1.2.2/integration/spark&gt;","_schemaURL":"&lt;https://openlineage.io/spec/2-0-2/OpenLineage.json#/$defs/RunFacet&gt;","spark-version":"3.3.0","openlineage_spark_version":"1.2.2"},"processing_engine":{"_producer":"&lt;https://github.com/OpenLineage/OpenLineage/tree/1.2.2/integration/spark&gt;","_schemaURL":"&lt;https://openlineage.io/spec/facets/1-1-0/ProcessingEngineRunFacet.json#/$defs/ProcessingEngineRunFacet&gt;","version":"3.3.0","name":"spark","openlineageAdapterVersion":"1.2.2"},"environment_properties":{"_producer":"&lt;https://github.com/OpenLineage/OpenLineage/tree/1.2.2/integration/spark&gt;","_schemaURL":"&lt;https://openlineage.io/spec/2-0-2/OpenLineage.json#/$defs/RunFacet&gt;","environment-properties":{}}}},"job":{"namespace":"default","name":"spark_streaming_example.write_to_data_source_v2","facets":{}},"inputs":[],"outputs":[]} +</code></pre> + +<p>As you can see, there are <em>zero</em> inputs and outputs.</p> + +<p><strong>Stuff to reproduce</strong> +<strong>Code</strong></p> + +<pre><code>package streaming; + +import org.apache.spark.sql.Dataset; +import org.apache.spark.sql.Row; +import org.apache.spark.sql.SparkSession; +import org.apache.spark.sql.streaming.StreamingQuery; +import org.apache.spark.sql.streaming.Trigger; + +import java.util.concurrent.TimeUnit; + +public class SparkStreamingExampleApplication { + public static void main(String[] args) throws Exception { + SparkSession spark = SparkSession + .builder() + .appName("spark-streaming-example") + .master("local") + .config("spark.ui.enabled", false) + .config("spark.jars.packages", "io.openlineage:openlineage_spark:1.2.2") + .config("spark.extraListeners", "io.openlineage.spark.agent.OpenLineageSparkListener") + .config("spark.openlineage.transport.type", "console") + .config("spark.openlineage.facets.disabled", "[spark_unknown;spark.logicalPlan]") + .getOrCreate(); + + Dataset&lt;Row&gt; df = spark.readStream() + .format("kafka") + .option("kafka.bootstrap.servers", "localhost:9092,localhost:9093,localhost:9094") + .option("subscribe", "source") + .load(); + + StreamingQuery kafkaWriteQuery = df.writeStream() + .outputMode("append") + .format("kafka") + .option("kafka.bootstrap.servers", "localhost:9092,localhost:9093,localhost:9094") + .option("topic", "target") + .option("checkpointLocation", "/Users/dhawes/Projects/spark-streaming-openlineage") + .trigger(Trigger.ProcessingTime(10, TimeUnit.SECONDS)) + .start(); + + kafkaWriteQuery.awaitTermination(); + } +} +</code></pre> + +<p><strong>build.gradle.kts</strong></p> + +<pre><code>plugins { + java + application +} + +java { + sourceCompatibility = JavaVersion.VERSION_1_8 + targetCompatibility = JavaVersion.VERSION_1_8 + + toolchain { + languageVersion.set(JavaLanguageVersion.of(8)) + } +} + +repositories { + mavenLocal() + mavenCentral() +} + +dependencies { + implementation("org.apache.spark:spark_core_2.12:3.3.0") + implementation("org.apache.spark:spark_sql_2.12:3.3.0") + implementation("org.apache.spark:spark_sql_kafka_0_10_2.12:3.3.0") + implementation("org.apache.spark:spark_streaming_2.12:3.3.0") + implementation("org.apache.spark:spark_streaming_kafka_0_10_2.12:3.3.0") + implementation("io.openlineage:openlineage_spark:1.2.2") +} + +application { + mainClass = "streaming.SparkStreamingExampleApplication" +} +</code></pre> + +<p><strong>docker-compose.yaml</strong></p> + +<p>``` +version: '3'</p> + +<p>services: + zookeeper: + image: confluentinc/cp-zookeeper:latest + container<em>name: 'spark-streaming-zookeeper' + environment: + ZOOKEEPER</em>CLIENT_PORT: 2181</p> + +<p>kafka1: + image: 'confluentinc/cp-kafka:latest' + container<em>name: 'spark-streaming-kafka-1' + depends</em>on: + - 'zookeeper' + environment: + KAFKA<em>BROKER</em>ID: 1 + KAFKA<em>ZOOKEEPER</em>CONNECT: zookeeper:2181 + KAFKA<em>ADVERTISED</em>LISTENERS: <INSIDE://kafka1:9092,OUTSIDE://localhost:9092> + KAFKA<em>LISTENER</em>SECURITY<em>PROTOCOL</em>MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT + KAFKA<em>INTER</em>BROKER<em>LISTENER</em>NAME: INSIDE + KAFKA_LISTENERS: <INSIDE://0.0.0.0:9092,OUTSIDE://0.0.0.0:9094> + ports: + - "9092:9094"</p> + +<p>kafka2: + image: 'confluentinc/cp-kafka:latest' + container<em>name: 'spark-streaming-kafka-2' + depends</em>on: + - 'zookeeper' + environment: + KAFKA<em>BROKER</em>ID: 2 + KAFKA<em>ZOOKEEPER</em>CONNECT: zookeeper:2181 + KAFKA<em>ADVERTISED</em>LISTENERS: <INSIDE://kafka2:9092,OUTSIDE://localhost:9093> + KAFKA<em>LISTENER</em>SECURITY<em>PROTOCOL</em>MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT + KAFKA<em>INTER</em>BROKER<em>LISTENER</em>NAME: INSIDE + KAFKA_LISTENERS: <INSIDE://0.0.0.0:9092,OUTSIDE://0.0.0.0:9095> + ports: + - "9093:9095"</p> + +<p>kafka3: + image: 'confluentinc/cp-kafka:latest' + container<em>name: 'spark-streaming-kafka-3' + depends</em>on: + - 'zookeeper' + environment: + KAFKA<em>BROKER</em>ID: 3 + KAFKA<em>ZOOKEEPER</em>CONNECT: zookeeper:2181 + KAFKA<em>ADVERTISED</em>LISTENERS: <INSIDE://kafka3:9092…</p> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+ 👀 Paweł Leszczyński +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-09-28 02:42:18
 +
+

*Thread Reply:* responded within the issue

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Erik Alfthan + (slack@alfthan.eu) +
+
2023-09-28 02:40:40
 +
+

Hello community +First time poster - bear with me :)

+ +

I am looking to make a minor PR on the airflow integration (fixing github #2130), and the code change is easy enough, but I fail to install the python environment. I have tried the simple ones +OpenLineage/integration/airflow &gt; pip install -e . + or +OpenLineage/integration/airflow &gt; pip install -r dev-requirements.txt +but they both fail on +ERROR: No matching distribution found for openlineage-sql==1.3.0

+ +

(which I think is an unreleased version in the git project)

+ +

How would I go about to install the requirements?

+ +

//Erik

+ +

PS. Sorry for posting this in general if there is a specific integration or contribution channel - I didnt find a better channel

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-09-28 03:04:48
 +
+

*Thread Reply:* Hi @Erik Alfthan, the channel is totally OK. I am not airflow integration expert, but what it looks to me, you're missing openlineage-sql library, which is a rust library used to extract lineage from sql queries. This is how we do that in circle ci: +https://app.circleci.com/pipelines/github/OpenLineage/OpenLineage/8080/workflows/aba53369-836c-48f5-a2dd-51bc0740a31c/jobs/140113

+ +

and subproject page with build instructions: https://github.com/OpenLineage/OpenLineage/tree/main/integration/sql

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Erik Alfthan + (slack@alfthan.eu) +
+
2023-09-28 03:07:23
 +
+

*Thread Reply:* Ok, so I go and "manually" build the internal dependency so that it becomes available in the pip cache?

+ +

I was hoping for something more automagical, but that should work

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-09-28 03:08:06
 +
+

*Thread Reply:* I think so. @Jakub Dardziński am I right?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-09-28 03:18:27
 +
+

*Thread Reply:* https://openlineage.io/docs/development/developing/python/setup +there’s a guide how to setup the dev environment

+ +

> Typically, you first need to build openlineage-sql locally (see README). After each release you have to repeat this step in order to bump local version of the package. +This might be somewhat exposed more in GitHub repository README as well

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Erik Alfthan + (slack@alfthan.eu) +
+
2023-09-28 03:27:20
 +
+

*Thread Reply:* It didnt find the wheel in the cache, but if I used the line in the sql/README.md +pip install openlineage-sql --no-index --find-links ../target/wheels --force-reinstall +It is installed and thus skipped/passed when pip later checks if it needs to be installed.

+ +

Now I have a second issue because it is expecting me to have mysqlclient-2.2.0 which seems to need a binary +Command 'pkg-config --exists mysqlclient' returned non-zero exit status 127 +and +Command 'pkg-config --exists mariadb' returned non-zero exit status 127 +I am on Ubuntu 22.04 in WSL2. Should I go to apt and grab me a mysql client?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-09-28 03:31:52
 +
+

*Thread Reply:* > It didnt find the wheel in the cache, but if I used the line in the sql/README.md +> pip install openlineage-sql --no-index --find-links ../target/wheels --force-reinstall +> It is installed and thus skipped/passed when pip later checks if it needs to be installed. +That’s actually expected. You should build new wheel locally and then install it.

+ +

> Now I have a second issue because it is expecting me to have mysqlclient-2.2.0 which seems to need a binary +> Command 'pkg-config --exists mysqlclient' returned non-zero exit status 127 +> and +> Command 'pkg-config --exists mariadb' returned non-zero exit status 127 +> I am on Ubuntu 22.04 in WSL2. Should I go to apt and grab me a mysql client? +We’ve left some system specific configuration, e.g. mysqlclient, to users as it’s a bit aside from OpenLineage and more of general development task.

+ +

probably +sudo apt-get install python3-dev default-libmysqlclient-dev build-essential +should work

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Erik Alfthan + (slack@alfthan.eu) +
+
2023-09-28 03:32:04
 +
+

*Thread Reply:* I just realized that I should probably skip setting up my wsl and just run the tests in the docker setup you prepared

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-09-28 03:35:46
 +
+

*Thread Reply:* You could do that as well but if you want to test your changes vs many Airflow versions that wouldn’t be possible I think (run them with tox btw)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Erik Alfthan + (slack@alfthan.eu) +
+
2023-09-28 04:54:39
 +
+

*Thread Reply:* This is starting to feel like a rabbit hole 😞

+ +

When I run tox, I get a lot of build errors +• client needs to be built +• sql needs to be built to a different target than its readme says +• a lot of builds fail on cython_sources

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-09-28 05:19:34
 +
+

*Thread Reply:* would you like to share some exact log lines? I’ve never seen such errors, they probably are system specific

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Erik Alfthan + (slack@alfthan.eu) +
+
2023-09-28 06:45:48
 +
+

*Thread Reply:* Getting requirements to build wheel did not run successfully. +│ exit code: 1 +╰─&gt; [62 lines of output] + /tmp/pip-build-env-q1pay0xo/overlay/lib/python3.10/site-packages/setuptools/config/setupcfg.py:293: _DeprecatedConfig: Deprecated config insetup.cfg` +!!`

+ +
        `****************************************************************************************************************************************************************`
+        `The license_file parameter is deprecated, use license_files instead.`
+
+        `By 2023-Oct-30, you need to update your project and remove deprecated calls`
+        `or your builds will no longer be supported.`
+
+        `See <https://setuptools.pypa.io/en/latest/userguide/declarative_config.html> for details.`
+        `****************************************************************************************************************************************************************`
+
+`!!`
+  `parsed = self.parsers.get(option_name, lambda x: x)(value)`
+`running egg_info`
+`writing lib3/PyYAML.egg-info/PKG-INFO`
+`writing dependency_links to lib3/PyYAML.egg-info/dependency_links.txt`
+`writing top-level names to lib3/PyYAML.egg-info/top_level.txt`
+`Traceback (most recent call last):`
+  `File "/home/obr_erikal/projects/OpenLineage/integration/airflow/.tox/py3-airflow-2.1.4/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in &lt;module&gt;`
+    `main()`
+  `File "/home/obr_erikal/projects/OpenLineage/integration/airflow/.tox/py3-airflow-2.1.4/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main`
+    `json_out['return_val'] = hook(****hook_input['kwargs'])`
+  `File "/home/obr_erikal/projects/OpenLineage/integration/airflow/.tox/py3-airflow-2.1.4/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 118, in get_requires_for_build_wheel`
+    `return hook(config_settings)`
+  `File "/tmp/pip-build-env-q1pay0xo/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 355, in get_requires_for_build_wheel`
+    `return self._get_build_requires(config_settings, requirements=['wheel'])`
+  `File "/tmp/pip-build-env-q1pay0xo/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 325, in _get_build_requires`
+    `self.run_setup()`
+  `File "/tmp/pip-build-env-q1pay0xo/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 341, in run_setup`
+    `exec(code, locals())`
+  `File "&lt;string&gt;", line 271, in &lt;module&gt;`
+  `File "/tmp/pip-build-env-q1pay0xo/overlay/lib/python3.10/site-packages/setuptools/__init__.py", line 103, in setup`
+    `return distutils.core.setup(****attrs)`
+  `File "/tmp/pip-build-env-q1pay0xo/overlay/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 185, in setup`
+    `return run_commands(dist)`
+  `File "/tmp/pip-build-env-q1pay0xo/overlay/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 201, in run_commands`
+    `dist.run_commands()`
+  `File "/tmp/pip-build-env-q1pay0xo/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands`
+    `self.run_command(cmd)`
+  `File "/tmp/pip-build-env-q1pay0xo/overlay/lib/python3.10/site-packages/setuptools/dist.py", line 989, in run_command`
+    `super().run_command(command)`
+  `File "/tmp/pip-build-env-q1pay0xo/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 988, in run_command`
+    `cmd_obj.run()`
+  `File "/tmp/pip-build-env-q1pay0xo/overlay/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 318, in run`
+    `self.find_sources()`
+  `File "/tmp/pip-build-env-q1pay0xo/overlay/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 326, in find_sources`
+    `mm.run()`
+  `File "/tmp/pip-build-env-q1pay0xo/overlay/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 548, in run`
+    `self.add_defaults()`
+  `File "/tmp/pip-build-env-q1pay0xo/overlay/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 586, in add_defaults`
+    `sdist.add_defaults(self)`
+  `File "/tmp/pip-build-env-q1pay0xo/overlay/lib/python3.10/site-packages/setuptools/command/sdist.py", line 113, in add_defaults`
+    `super().add_defaults()`
+  `File "/tmp/pip-build-env-q1pay0xo/overlay/lib/python3.10/site-packages/setuptools/_distutils/command/sdist.py", line 251, in add_defaults`
+    `self._add_defaults_ext()`
+  `File "/tmp/pip-build-env-q1pay0xo/overlay/lib/python3.10/site-packages/setuptools/_distutils/command/sdist.py", line 336, in _add_defaults_ext`
+    `self.filelist.extend(build_ext.get_source_files())`
+  `File "&lt;string&gt;", line 201, in get_source_files`
+  `File "/tmp/pip-build-env-q1pay0xo/overlay/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 107, in __getattr__`
+    `raise AttributeError(attr)`
+`AttributeError: cython_sources`
+`[end of output]`
+
+ +

note: This error originates from a subprocess, and is likely not a problem with pip. +py3-airflow-2.1.4: exit 1 (7.85 seconds) /home/obr_erikal/projects/OpenLineage/integration/airflow&gt; python -m pip install --find-links target/wheels/ --find-links ../sql/iface-py/target/wheels --use-deprecated=legacy-resolver --constraint=<https://raw.githubusercontent.com/apache/airflow/constraints-2.1.4/constraints-3.8.txt> apache-airflow==2.1.4 'mypy&gt;=0.9.6' pytest pytest-mock -r dev-requirements.txt pid=368621 +py3-airflow-2.1.4: FAIL ✖ in 7.92 seconds

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Erik Alfthan + (slack@alfthan.eu) +
+
2023-09-28 06:53:54
 +
+

*Thread Reply:* Then, for the actual error in my PR: Evidently you are not using isort, so what linter/fixer should I use for imports?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-09-28 06:58:15
 +
+

*Thread Reply:* for the error - I think there’s a mistake in the docs. Could you please run maturin build --out target/wheels as a temp solution?

+ + + +
+ 👀 Erik Alfthan +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-09-28 06:58:57
 +
+

*Thread Reply:* we’re using ruff , tox runs it as one of commands

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Erik Alfthan + (slack@alfthan.eu) +
+
2023-09-28 07:00:37
 +
+

*Thread Reply:* Not in the airflow folder? +OpenLineage/integration/airflow$ maturin build --out target/wheels +💥 maturin failed + Caused by: pyproject.toml at /home/obr_erikal/projects/OpenLineage/integration/airflow/pyproject.toml is invalid + Caused by: TOML parse error at line 1, column 1 + | +1 | [tool.ruff] + | ^ +missing fieldbuild-system``

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-09-28 07:02:32
 +
+

*Thread Reply:* I meant change here https://github.com/OpenLineage/OpenLineage/blob/main/integration/sql/README.md

+ +

so +cd iface-py +python -m pip install maturin +maturin build --out ../target/wheels +becomes +cd iface-py +python -m pip install maturin +maturin build --out target/wheels +tox runs +install_command = python -m pip install {opts} --find-links target/wheels/ \ + --find-links ../sql/iface-py/target/wheels +but it should be +install_command = python -m pip install {opts} --find-links target/wheels/ \ + --find-links ../sql/target/wheels +actually and I’m posting PR to fix that

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Erik Alfthan + (slack@alfthan.eu) +
+
2023-09-28 07:05:12
 +
+

*Thread Reply:* yes, that part I actually worked out myself, but the cython_sources error I fail to understand cause. I have python3-dev installed on WSL Ubuntu with python version 3.10.12 in a virtualenv. Anything in that that could cause issues?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-09-28 07:12:20
 +
+

*Thread Reply:* looks like it has something to do with latest release of Cython? +pip install "Cython&lt;3" maybe solves the issue?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Erik Alfthan + (slack@alfthan.eu) +
+
2023-09-28 07:15:06
 +
+

*Thread Reply:* I didnt have any cython before the install. Also no change. Could it be some update to setuptools itself? seems like the depreciation notice and the error is coming from inside setuptools

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Erik Alfthan + (slack@alfthan.eu) +
+
2023-09-28 07:16:59
 +
+

*Thread Reply:* (I.e. I tried the pip install "Cython&lt;3" command without any change in the output )

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Erik Alfthan + (slack@alfthan.eu) +
+
2023-09-28 07:20:30
 +
+

*Thread Reply:* Applying ruff lint on the converter.py file fixed the issue on the PR though so unless you have any feedback on the change itself, I will set it up on my own computer later instead (right now doing changes on behalf of a client on the clients computer)

+ +

If the issue persists on my own computer, I'll dig a bit further

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-09-28 07:21:03
 +
+

*Thread Reply:* It’s a bit hard for me to find the root cause as I cannot reproduce this locally and CI works fine as well

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Erik Alfthan + (slack@alfthan.eu) +
+
2023-09-28 07:22:41
 +
+

*Thread Reply:* Yeah, I am thinking that if I run into the same problem "at home", I might find it worthwhile to understand the issue. Right now, the client only wants the fix.

+ + + +
+ 👍 Jakub Dardziński +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Erik Alfthan + (slack@alfthan.eu) +
+
2023-09-28 07:25:10
 +
+

*Thread Reply:* Is there an official release cycle?

+ +

or more specific, given that the PRs are approved, how soon can they reach openlineage-dbt and apache-airflow-providers-openlineage ?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-09-28 07:28:58
 +
+

*Thread Reply:* we need to differentiate some things:

+ +
  1. OpenLineage repository: +a. dbt integration - this is the only place where it is maintained +b. Airflow integration - here we only keep backwards compatibility but generally speaking starting from Airflow 2.7+ we would like to do all the job in Airflow repo as OL Airflow provider
  2. Airflow repository - there’s only Airflow Openlineage provider compatible (and works best) with Airflow 2.7+
    3. +
+ +

we have control over releases (obviously) in OL repo - it’s monthly cycle so beginning next week that should happen. There’s also a possibility to ask for ad-hoc release in #general slack channel and with approvals of committers the new version is also released

+ +

For Airflow providers - the cycle is monthly as well

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-09-28 07:31:30
 +
+

*Thread Reply:* it’s a bit complex for this split but needed temporarily

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Erik Alfthan + (slack@alfthan.eu) +
+
2023-09-28 07:31:47
 +
+

*Thread Reply:* oh, I did the fix in the wrong place! The client is on airflow 2.7 and is using the provider. Is it syncing?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-09-28 07:32:28
 +
+

*Thread Reply:* it’s not, two separate places a~nd we haven’t even added the whole thing with converting old lineage objects to OL specific~

+ +

editing, that’s not true

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-09-28 07:34:40
 +
+

*Thread Reply:* the code’s here: +https://github.com/apache/airflow/blob/main/airflow/providers/openlineage/extractors/manager.py#L154

+
+ + + + +
<https://github.com/apache/airflow/blob/main/airflow/providers/openlineage/extractors/manager.py | manager.py>
+
+ <pre><code> def extract_inlets_and_outlets( +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/apache/airflow">apache/airflow</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-09-28 07:35:17
 +
+

*Thread Reply:* sorry I did not mention this earlier. we definitely need to add some guidance how to proceed with contributions to OL and Airflow OL provider

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Erik Alfthan + (slack@alfthan.eu) +
+
2023-09-28 07:36:10
 +
+

*Thread Reply:* anyway, the dbt fix is the blocking issue, so if that parts comes next week, there is no real urgency in getting the columns. It is a nice to have for our ingest parquet files.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-09-28 07:37:12
 +
+

*Thread Reply:* may I ask if you use some custom operator / python operator there?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Erik Alfthan + (slack@alfthan.eu) +
+
2023-09-28 07:37:33
 +
+

*Thread Reply:* yeah, taskflow with inlets/outlets

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Erik Alfthan + (slack@alfthan.eu) +
+
2023-09-28 07:38:38
 +
+

*Thread Reply:* so we extract from sources and use pyarrow to create parquet files in storage that an mssql-server can use as external tables

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-09-28 07:39:54
 +
+

*Thread Reply:* awesome 👍 +we have plans to integrate more with Python operator as well but not earlier than in Airflow 2.8

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Erik Alfthan + (slack@alfthan.eu) +
+
2023-09-28 07:43:41
 +
+

*Thread Reply:* I guess writing a generic extractor for the python operator is quite hard, but if you could support some inlet/outlet type for tabular fileformat / their python libraries like pyarrow or maybe even pandas and document it, I think a lot of people would understand how to use them

+ + + +
+ ➕ Harel Shein +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-09-28 16:16:24
 +
+

Are you located in the Brussels area or within commutable distance? Interested in attending a meetup between October 16-20? If so, please DM @Sheeri Cabral (Collibra) or myself. TIA

+ + + +
+ ❤️ Sheeri Cabral (Collibra) +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-10-02 11:58:32
 +
+

@channel +Hello all, I’d like to open a vote to release OpenLineage 1.3.0, including: +• support for Spark 3.5 in the Spark integration +• scheme preservation bug fix in the Spark integration +• find-links path in tox bug in the Airflow integration fix +• more graceful logging when no OL provider is installed in the Airflow integration +• columns as schema facet for airflow.lineage.Table addition +• SQLSERVER to supported dbt profile types addition +Three +1s from committers will authorize. Thanks in advance.

+ + + +
+ 🙌 Harel Shein, Paweł Leszczyński, Rodrigo Maia +
+ +
+ 👍 Jason Yip, Paweł Leszczyński +
+ +
+ ➕ Willy Lulciuc, Jakub Dardziński, Erik Alfthan, Julien Le Dem +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-10-02 17:00:08
 +
+

*Thread Reply:* Thanks all. The release is authorized and will be initiated within 2 business days.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason Yip + (jasonyip@gmail.com) +
+
2023-10-02 17:11:46
 +
+

*Thread Reply:* looking forward to that, I am seeing inconsistent results in Databricks for Spark 3.4+, sometimes there's no inputs / outputs, hope that is fixed?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harel Shein + (harel.shein@gmail.com) +
+
2023-10-03 09:59:24
 +
+

*Thread Reply:* @Jason Yip if it isn’t fixed for you, would love it if you could open up an issue that will allow us to reproduce and fix

+ + + +
+ 👍 Jason Yip +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason Yip + (jasonyip@gmail.com) +
+
2023-10-03 20:23:40
 +
+

*Thread Reply:* @Harel Shein the issue still exists -> Spark 3.4 and above, including 3.5, saveAsTable and create table won't have inputs and outputs in Databricks

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason Yip + (jasonyip@gmail.com) +
+
2023-10-03 20:30:15
 +
+

*Thread Reply:* https://github.com/OpenLineage/OpenLineage/issues/2124

+
+ + + + +
#2124 Same Delta Table not catching the location on write
+
+ <p><strong>What is the target system?</strong></p> + +<p>Spark / Databricks</p> + +<p><strong>What kind of integration is this?</strong></p> + +<p>☐ Produces OpenLineage metadata +☐ Consumes OpenLineage metadata +☐ Something else</p> + +<p><strong>How should this integration be implemented?</strong></p> + +<p>I am using OL 1.2.2, Azure Databricks Runtime 11.3 LTS. When creating a table writing into a ADLS location, OL won't be able to catch the location of the output. But when I read the same object it will be able to read the location as INPUT.</p> + +<p>Please note I have also tested Databricks Runtime 13.3 LTS, Spark 3.4.1 - it will give correct ADLS location in INPUT but the input will only show up once in a blue moon. Most of the time the inputs and outputs are blank.</p> + +<pre><code> "inputs": [], + "outputs": [] +</code></pre> + +<pre><code>CREATE OR REPLACE TABLE transactions_adj +USING DELTA LOCATION '&lt;wasbs://studio@clororetaildevadls.blob.core.windows.net/examples/data/csv/completejourney/silver/transactions_adj&gt;' +AS + SELECT + household_id, + basket_id, + week_no, + day, + transaction_time, + store_id, + product_id, + amount_list, + campaign_coupon_discount, + manuf_coupon_discount, + manuf_coupon_match_discount, + total_coupon_discount, + instore_discount, + amount_paid, + units + FROM ( + SELECT + household_id, + basket_id, + week_no, + day, + transaction_time, + store_id, + product_id, + COALESCE(sales_amount - discount_amount - coupon_discount - coupon_discount_match,0.0) as amount_list, + CASE + WHEN COALESCE(coupon_discount_match,0.0) = 0.0 THEN -1 ** COALESCE(coupon_discount,0.0) + ELSE 0.0 + END as campaign_coupon_discount, + CASE + WHEN COALESCE(coupon_discount_match,0.0) != 0.0 THEN -1 ** COALESCE(coupon_discount,0.0) + ELSE 0.0 + END as manuf_coupon_discount, + -1 ** COALESCE(coupon_discount_match,0.0) as manuf_coupon_match_discount, + -1 ** COALESCE(coupon_discount - coupon_discount_match,0.0) as total_coupon_discount, + COALESCE(-1 ** discount_amount,0.0) as instore_discount, + COALESCE(sales_amount,0.0) as `amount_paid,` + quantity as units + FROM transactions + ); +</code></pre> + +<p>Here's the COMPLETE event:</p> + +<pre><code> + "outputs":[ + { + "namespace":"dbfs", + "name":"/user/hive/warehouse/journey.db/transactions_adj", + "facets":{ + "dataSource":{ + "_producer":"&lt;https://github.com/OpenLineage/OpenLineage/tree/1.2.2/integration/spark&gt;", + "_schemaURL":"&lt;https://openlineage.io/spec/facets/1-0-0/DatasourceDatasetFacet.json#/$defs/DatasourceDatasetFacet&gt;", + "name":"dbfs", + "uri":"dbfs" + }, + +</code></pre> + +<p>Below logical plan shows the path:</p> + +<pre><code>== Analyzed Logical Plan == +num_affected_rows: bigint, num_inserted_rows: bigint +ReplaceTableAsSelect TableSpec(Map(),Some(DELTA),Map(),Some(&lt;wasbs://studio@clororetaildevadls.blob.core.windows.net/examples/data/csv/completejourney/silver/transactions_adj&gt;),None,None,false,Set()), true +:- ResolvedIdentifier com.databricks.sql.managedcatalog.UnityCatalogV2Proxy@6251a8df, default.transactions_adj ++- Project [household_id#184, basket_id#185L, week_no#193, day#186, transaction_time#192, store_id#190, product_id#187, amount_list#147, campaign_coupon_discount#148, manuf_coupon_discount#149, manuf_coupon_match_discount#150, total_coupon_discount#151, instore_discount#152, amount_paid#153, units#154] + +- SubqueryAlias __auto_generated_subquery_name + +- Project [household_id#184, basket_id#185L, week_no#193, day#186, transaction_time#192, store_id#190, product_id#187, coalesce(cast((((sales_amount#189 - discount_amount#191) - coupon_discount#194) - coupon_discount_match#195) as double), cast(0.0 as double)) AS amount_list#147, CASE WHEN (coalesce(cast(coupon_discount_match#195 as double), cast(0.0 as double)) = cast(0.0 as double)) THEN (cast(-1 as double) ** coalesce(cast(coupon_discount#194 as double), cast(0.0 as double))) ELSE cast(0.0 as double) END AS campaign_coupon_discount#148, CASE WHEN NOT (coalesce(cast(coupon_discount_match#195 as double), cast(0.0 as double)) = cast(0.0 as double)) THEN (cast(-1 as double) ** coalesce(cast(coupon_discount#194 as double), cast(0.0 as double))) ELSE cast(0.0 as double) END AS manuf_coupon_discount#149, (cast(-1 as double) ** coalesce(cast(coupon_discount_match#195 as double), cast(0.0 as double))) AS manuf_coupon_match_discount#150, (cast(-1 as double) ** coalesce(cast((coupon_discount#194 - coupon_discount_match#195) as double), cast(0.0 as double))) AS total_coupon_discount#151, coalesce(cast((cast(-1 as float) ** discount_amount#191) as double), cast(0.0 as double)) AS instore_discount#152, coalesce(cast(sales_amount#189 as double), cast(0.0 as double)) AS amount_paid#153, quantity#188 AS units#154] + +- SubqueryAlias spark_catalog.default.transactions + +- Relation spark_catalog.default.transactions[household_id#184,basket_id#185L,day#186,product_id#187,quantity#188,sales_amount#189,store_id#190,discount_amount#191,transaction_time#192,week_no#193,coupon_discount#194,coupon_discount_match#195] parquet +</code></pre> + +<p><strong>Where should this integration be implemented?</strong></p> + +<p>☐ In the target system +☐ In the OpenLineage repo +☐ Somewhere else</p> + +<p><strong>Do you plan to make this contribution yourself?</strong></p> + +<p>☐ I am interested in doing this work</p> +
+ +
+
Labels
+ integration/spark, integration/databricks +
+ +
+
Comments
+ 1 +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason Yip + (jasonyip@gmail.com) +
+
2023-10-03 20:30:21
 +
+

*Thread Reply:* and of course this issue still exists

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harel Shein + (harel.shein@gmail.com) +
+
2023-10-03 21:45:09
 +
+

*Thread Reply:* thanks for posting, we’ll continue looking into this.. if you find any clues that might help, please let us know.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason Yip + (jasonyip@gmail.com) +
+
2023-10-03 21:46:27
 +
+

*Thread Reply:* is there any instructions on how to hook up a debugger to OL?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harel Shein + (harel.shein@gmail.com) +
+
2023-10-04 09:04:16
 +
+

*Thread Reply:* @Paweł Leszczyński has been working on adding a debug facet, but more suggestions are more than welcome!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harel Shein + (harel.shein@gmail.com) +
+
2023-10-04 09:05:58
 +
+

*Thread Reply:* https://github.com/OpenLineage/OpenLineage/pull/2147

+
+ + + + +
#2147 [SPARK] add debug facet to help resolving Spark integration issues
+
+ <p><strong>Problem</strong></p> + +<p>Debugging openlineage-spark problems is tedious job. We would like to have debug facet that will collect automatically meaningful information when enabled.</p> + +<p>Closes: <a href="https://github.com/OpenLineage/OpenLineage/issues/2135">#2135</a></p> + +<p><strong>Solution</strong></p> + +<p>• Create debug facet (more details in <a href="https://github.com/OpenLineage/OpenLineage/issues/2135">#2135</a> ), +• Facet is disabled by default, +• Allow enabling it thourgh SparkConf.</p> + +<blockquote> + <p><strong>Note:</strong> All schema changes require discussion. Please <a href="https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue">link the issue</a> for context.</p> +</blockquote> + +<p>☐ Your change modifies the <a href="https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.json">core</a> OpenLineage model +☐ Your change modifies one or more OpenLineage <a href="https://github.com/OpenLineage/OpenLineage/tree/main/spec/facets">facets</a></p> + +<p>If you're contributing a new integration, please specify the scope of the integration and how/where it has been tested (e.g., Apache Spark integration supports <code>S3</code> and <code>GCS</code> filesystem operations, tested with AWS EMR).</p> + +<p><strong>One-line summary:</strong> +<strong>Checklist</strong></p> + +<p>☑︎ You've <a href="https://github.com/OpenLineage/OpenLineage/blob/main/why-the-dco.md">signed-off</a> your work +☑︎ Your pull request title follows our <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CONTRIBUTING.md#creating-pull-requests">guidelines</a> +☑︎ Your changes are accompanied by tests (<em>if relevant</em>) +☑︎ Your change contains a <a href="https://kurtisnusbaum.medium.com/stacked-diffs-keeping-phabricator-diffs-small-d9964f4dcfa6">small diff</a> and is self-contained +☐ You've updated any relevant documentation (<em>if relevant</em>) +☐ Your comment includes a one-liner for the changelog about the specific purpose of the change (<em>if necessary</em>) +☐ You've versioned the core OpenLineage model or facets according to <a href="https://docs.snowplowanalytics.com/docs/pipeline-components-and-applications/iglu/common-architecture/schemaver">SchemaVer</a> (<em>if relevant</em>) +☐ You've added a <a href="https://github.com/OpenLineage/OpenLineage/tree/main/.github/header_templates.md">header</a> to source files (<em>if relevant</em>)</p> + +<hr /> + +<p>SPDX-License-Identifier: Apache-2.0 <br /> +Copyright 2018-2023 contributors to the OpenLineage project</p> +
+ +
+
Labels
+ documentation, integration/spark +
+ +
+
Assignees
+ <a href="https://github.com/pawel-big-lebowski">@pawel-big-lebowski</a> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+ 👀 Paweł Leszczyński +
+ +
+ 👍 Jason Yip +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason Yip + (jasonyip@gmail.com) +
+
2023-10-05 03:20:11
 +
+

*Thread Reply:* @Paweł Leszczyński do you have a build for the PR? Appreciated!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harel Shein + (harel.shein@gmail.com) +
+
2023-10-05 15:05:08
 +
+

*Thread Reply:* we’ll ask for a release once it’s reviewed and merged

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-10-02 12:28:28
 +
+

@channel +The September issue of OpenLineage News is here! This issue covers the big news about OpenLineage coming out of Airflow Summit, progress on the Airflow Provider, highlights from our meetup in Toronto, and much more. +To get the newsletter directly in your inbox each month, sign up here.

+
+
apache.us14.list-manage.com
+ + + +
OpenLineage Project
+
+ OpenLineage Project Email Forms +
+ + + + + + +
Original URL: http://bit.ly/OL_news
+ + + +
+ + + +
+ 🦆 Harel Shein, Paweł Leszczyński +
+ +
+ 🔥 Willy Lulciuc, Jakub Dardziński, Paweł Leszczyński +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Damien Hawes + (damien.hawes@booking.com) +
+
2023-10-03 03:44:36
 +
+

Hi folks - I'm wondering if its just me, but does io.openlineage:openlineage_sql_java:1.2.2 ship with the arm64.dylib binary? When i try and run code that uses the Java package on an Apple M1, the binary isn't found, The workaround is to checkout 1.2.2 and then build and publish it locally.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-10-03 09:01:38
 +
+

*Thread Reply:* Not sure if I follow your question. Whenever OL is released, there is a script new-version.sh - https://github.com/OpenLineage/OpenLineage/blob/main/new-version.sh being run and modify the codebase.

+ +

So, If you pull the code, it contains OL version that has not been released yet and in case of dependencies, one need to build them on their own.

+ +

For example, here https://github.com/OpenLineage/OpenLineage/tree/main/integration/spark#preparation Preparation section describes how to build openlineage-java and openlineage-sql in order to build openlineage-spark.

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/new-version.sh | new-version.sh>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Damien Hawes + (damien.hawes@booking.com) +
+
2023-10-04 05:27:26
 +
+

*Thread Reply:* Hmm. Let's elaborate my use case a bit.

+ +

We run Apache Hive on-premise. Hive provides query execution hooks for pre-query, post-query, and I think failed query.

+ +

Any way, as part of the hook, you're given the query string.

+ +

So I, naturally, tried to pass the query string into OpenLineageSql.parse(Collections.singletonList(hookContext.getQueryPlan().getQueryStr()), "hive") in order to test this out.

+ +

I was using openlineage-sql-java:1.2.2 at that time, and no matter what query string I gave it, nothing was returned.

+ +

I then stepped through the code and noticed that it was looking for the arm64 lib, and I noticed that that package (downloaded from maven central) lacked that particular native binary.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Damien Hawes + (damien.hawes@booking.com) +
+
2023-10-04 05:27:36
 +
+

*Thread Reply:* I hope that helps.

+ + + +
+ 👍 Paweł Leszczyński +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-10-04 09:03:02
 +
+

*Thread Reply:* I get in now. In Circle CI we do have 3 build steps: +- build-integration-sql-x86 + - build-integration-sql-arm + - build-integration-sql-macos +but no mac m1. I think at that time circle CI did not have a proper resource class in free plan. Additionally, @Maciej Obuchowski would prefer to migrate this to github actions as he claims this can be achieved there in a cleaner way (https://github.com/OpenLineage/OpenLineage/issues/1624).

+ +

Feel free to create an issue for this. Others would be able to upvote it in case they have similar experience.

+
+ + + + +
#1624 CI: PoC building openlineage-sql on GitHub Actions with actions-rs
+
+ <p>Building sql parser currently is the most convoluted CI process due to need to construct different binaries in multiple dimensions; both for Java and Python, and for multiple architectures; like Linux x86, Linux ARM, MacOS x86 etc. The jobs also differ in different context: release workflow has different jobs than build one, which in turn does not build all of the architectures.</p> + +<p>To simplify that, we should try using GitHub Actions with <a href="https://actions-rs.github.io/">https://actions-rs.github.io/</a> that should solve the problems we've currently had to replicate manually.</p> + +<p>End result of that task should be having various SQL artifacts produced by GitHub actions and available by GH Actions artifacts API:</p> +
+ +
+
Assignees
+ <a href="https://github.com/mobuchowski">@mobuchowski</a> +
+ +
+
Labels
+ ci, integration/sql +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-10-23 11:56:12
 +
+

*Thread Reply:* It doesn't have the free resource class still 😞 +We're blocked on that unfortunately. Other solution would be to migrate to GH actions, where most of our solution could be replaced by something like that https://github.com/PyO3/maturin-action

+
+ + + + +
PyO3/maturin-action
+
+ GitHub Action to install and run a custom maturin command with built-in support for cross compilation +
+ +
+
Stars
+ 98 +
+ +
+
Language
+ TypeScript +
+ + + + + + + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-10-03 10:56:03
 +
+

@channel +We released OpenLineage 1.3.1! +Added: +• Airflow: add some basic stats to the Airflow integration #1845 @harels +• Airflow: add columns as schema facet for airflow.lineage.Table (if defined) #2138 @erikalfthan +• DBT: add SQLSERVER to supported dbt profile types #2136 @erikalfthan +• Spark: support for latest 3.5 #2118 @pawel-big-lebowski +Fixed: +• Airflow: fix find-links path in tox #2139 @JDarDagran +• Airflow: add more graceful logging when no OpenLineage provider installed #2141 @JDarDagran +• Spark: fix bug in PathUtils’ prepareDatasetIdentifierFromDefaultTablePath (CatalogTable) to correctly preserve scheme from CatalogTable’s location #2142 @d-m-h +Thanks to all the contributors, including new contributor @Erik Alfthan! +Release: https://github.com/OpenLineage/OpenLineage/releases/tag/1.3.1 +Changelog: https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md +Commit history: https://github.com/OpenLineage/OpenLineage/compare/1.2.2...1.3.1 +Maven: https://oss.sonatype.org/#nexus-search;quick~openlineage +PyPI: https://pypi.org/project/openlineage-python/

+ + + +
+ 👍 Jason Yip, Peter Hicks, Peter Huang, Mars Lan +
+ +
+ 🎉 Sheeri Cabral (Collibra) +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mars Lan + (mars@metaphor.io) +
+
2023-10-04 07:42:59
 +
+

*Thread Reply:* Any chance we can do a 1.3.2 soonish to include https://github.com/OpenLineage/OpenLineage/pull/2151 instead of waiting for the next monthly release?

+
+ + + + +
#2151 Allow setting client's endpoint via environment variable
+
+ <p><strong>Problem</strong></p> + +<p>Currently, it's not possible to set the OpenLineage endpoint (hard-coded to <code>/api/v1/lineage</code>) using an environment variable when running the Airflow integration.</p> + +<p><strong>Solution</strong></p> + +<p>Given that it's not possible to create the client manually in Airflow, especially now that OpenLineage has become an official Airflow provider, this change seems like the only feasible solution.</p> + +<p>☐ Your change modifies the <a href="https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.json">core</a> OpenLineage model +☐ Your change modifies one or more OpenLineage <a href="https://github.com/OpenLineage/OpenLineage/tree/main/spec/facets">facets</a></p> + +<p><strong>One-line summary:</strong></p> + +<p>Allow setting client's endpoint via environment variable.</p> + +<p><strong>Checklist</strong></p> + +<p>☑︎ You've <a href="https://github.com/OpenLineage/OpenLineage/blob/main/why-the-dco.md">signed-off</a> your work +☑︎ Your pull request title follows our <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CONTRIBUTING.md#creating-pull-requests">guidelines</a> +☑︎ Your changes are accompanied by tests (<em>if relevant</em>) +☑︎ Your change contains a <a href="https://kurtisnusbaum.medium.com/stacked-diffs-keeping-phabricator-diffs-small-d9964f4dcfa6">small diff</a> and is self-contained +☑︎ You've updated any relevant documentation (<em>if relevant</em>) +☑︎ Your comment includes a one-liner for the changelog about the specific purpose of the change (<em>if necessary</em>) +☐ You've versioned the core OpenLineage model or facets according to <a href="https://docs.snowplowanalytics.com/docs/pipeline-components-and-applications/iglu/common-architecture/schemaver">SchemaVer</a> (<em>if relevant</em>) +☐ You've added a <a href="https://github.com/OpenLineage/OpenLineage/tree/main/.github/header_templates.md">header</a> to source files (<em>if relevant</em>)</p> + +<hr /> + +<p>SPDX-License-Identifier: Apache-2.0 <br /> +Copyright 2018-2023 contributors to the OpenLineage project</p> +
+ +
+
Labels
+ documentation, client/python +
+ +
+
Comments
+ 4 +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Matthew Paras + (matthewparas2020@u.northwestern.edu) +
+
2023-10-03 12:34:57
 +
+

Hey everyone - does anyone have a good mechanism for alerting on issues with open lineage? For example, maybe alerting when an event times out - perhaps to prometheus or some other kind of generic endpoint? Not sure the best approach here (if the meta inf extension would be able to achieve it)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-10-04 03:01:02
 +
+

*Thread Reply:* That's a great usecase for OpenLineage. Unfortunately, we don't have any doc or recomendation on that.

+ +

I would try using FluentD proxy we have (https://github.com/OpenLineage/OpenLineage/tree/main/proxy/fluentd) to copy event stream (alerting is just one of usecases for lineage events) and write fluentd plugin to send it asynchronously further to alerting service like PagerDuty.

+ +

It looks cool to me but I never had enough time to test this approach.

+ + + +
+ 👍 Matthew Paras +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-10-05 14:44:14
 +
+

@channel +This month’s TSC meeting is next Thursday the 12th at 10am PT. On the tentative agenda: +• announcements +• recent releases +• Airflow Summit recap +• tutorial: migrating to the Airflow Provider +• discussion topic: observability for OpenLineage/Marquez +• open discussion +• more (TBA) +More info and the meeting link can be found on the website. All are welcome! Do you have a discussion topic, use case or integration you’d like to demo? DM me to be added to the agenda.

+
+
openlineage.io
+ + + +
TSC Meetings | OpenLineage
+
+ The OpenLineage Technical Steering Committee meets monthly, and is open to all. +
+ + + + + + +
Original URL: https://openlineage.io/meetings/
+ + + +
+ + + +
+ 👀 Sheeri Cabral (Collibra), Julian LaNeve, Peter Hicks +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Julien Le Dem + (julien@apache.org) +
+
2023-10-05 20:40:40
 +
+

The Marquez meetup in San Francisco is happening right now! +https://www.meetup.com/meetup-group-bnfqymxe/events/295444209/?utmmedium=referral&utmcampaign=share-btnsavedeventssharemodal&utmsource=link|https://www.meetup.com/meetup-group-bnfqymxe/events/295444209/?utmmedium=referral&utmcampaign=share-btnsavedeventssharemodal&utmsource=link

+
+
Meetup
+ + + +
Marquez Meetup @ Astronomer, Thu, Oct 5, 2023, 5:30 PM | Meetup
+
+ Join us on Thursday, October 5th, from 5:30_8:30 pm to learn about the Marquez project. Meet other members of the community, get tips on making the most of the latest impro +
+ + + + + + + + +
Original URL: https://www.meetup.com/meetup-group-bnfqymxe/events/295444209/?utm_medium=referral&amp;utm_campaign=share-btn_savedevents_share_modal&amp;utm_source=link
+ + + +
+ + + +
+ 🎉 Paweł Leszczyński, Rodrigo Maia +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mars Lan + (mars@metaphor.io) +
+
2023-10-06 07:19:01
 +
+

@Michael Robinson can we cut a new release to include this change? +• https://github.com/OpenLineage/OpenLineage/pull/2151

+
+ + + + +
#2151 Allow setting client's endpoint via environment variable
+
+ <p><strong>Problem</strong></p> + +<p>Currently, it's not possible to set the OpenLineage endpoint (hard-coded to <code>/api/v1/lineage</code>) using an environment variable when running the Airflow integration.</p> + +<p><strong>Solution</strong></p> + +<p>Given that it's not possible to create the client manually in Airflow, especially now that OpenLineage has become an official Airflow provider, this change seems like the only feasible solution.</p> + +<p>☐ Your change modifies the <a href="https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.json">core</a> OpenLineage model +☐ Your change modifies one or more OpenLineage <a href="https://github.com/OpenLineage/OpenLineage/tree/main/spec/facets">facets</a></p> + +<p><strong>One-line summary:</strong></p> + +<p>Allow setting client's endpoint via environment variable.</p> + +<p><strong>Checklist</strong></p> + +<p>☑︎ You've <a href="https://github.com/OpenLineage/OpenLineage/blob/main/why-the-dco.md">signed-off</a> your work +☑︎ Your pull request title follows our <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CONTRIBUTING.md#creating-pull-requests">guidelines</a> +☑︎ Your changes are accompanied by tests (<em>if relevant</em>) +☑︎ Your change contains a <a href="https://kurtisnusbaum.medium.com/stacked-diffs-keeping-phabricator-diffs-small-d9964f4dcfa6">small diff</a> and is self-contained +☑︎ You've updated any relevant documentation (<em>if relevant</em>) +☑︎ Your comment includes a one-liner for the changelog about the specific purpose of the change (<em>if necessary</em>) +☐ You've versioned the core OpenLineage model or facets according to <a href="https://docs.snowplowanalytics.com/docs/pipeline-components-and-applications/iglu/common-architecture/schemaver">SchemaVer</a> (<em>if relevant</em>) +☐ You've added a <a href="https://github.com/OpenLineage/OpenLineage/tree/main/.github/header_templates.md">header</a> to source files (<em>if relevant</em>)</p> + +<hr /> + +<p>SPDX-License-Identifier: Apache-2.0 <br /> +Copyright 2018-2023 contributors to the OpenLineage project</p> +
+ +
+
Labels
+ documentation, client/python +
+ +
+
Comments
+ 6 +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+ ➕ Harel Shein, Jakub Dardziński, Julien Le Dem, Michael Robinson, Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-10-06 19:16:02
 +
+

*Thread Reply:* Thanks for requesting a release, @Mars Lan. It has been approved and will be initiated within 2 business days of next Monday.

+ + + +
+ 🙏 Mars Lan +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Guntaka Jeevan Paul + (jeevan@acceldata.io) +
+
2023-10-08 23:59:36
 +
+

@here I am trying out the openlineage integration of spark on databricks. There is no event getting emitted from Openlineage, I see logs saying OpenLineage Event Skipped. I am attaching the Notebook that i am trying to run and the cluster logs. Kindly can someone help me on this

+ +
+
Screenshot 2023-10-09 at 9.28.18 AM.png
+ + + + + + + + +
+
+
cluster-logs
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason Yip + (jasonyip@gmail.com) +
+
2023-10-09 00:02:10
 +
+

*Thread Reply:* from my experience, it will only work on Spark 3.3.x or below, aka Runtime 12.2 or below. Anything above the events will show up once in a blue moon

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Guntaka Jeevan Paul + (jeevan@acceldata.io) +
+
2023-10-09 00:04:38
 +
+

*Thread Reply:* ohh, thanks for the information @Jason Yip, I am trying out with 13.3 Databricks Version and Spark 3.4.1, will try using a below version as you suggested. Any issue tracking this bug @Jason Yip

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason Yip + (jasonyip@gmail.com) +
+
2023-10-09 00:06:06
 +
+

*Thread Reply:* https://github.com/OpenLineage/OpenLineage/issues/2124

+
+ + + + +
#2124 Same Delta Table not catching the location on write
+
+ <p><strong>What is the target system?</strong></p> + +<p>Spark / Databricks</p> + +<p><strong>What kind of integration is this?</strong></p> + +<p>☐ Produces OpenLineage metadata +☐ Consumes OpenLineage metadata +☐ Something else</p> + +<p><strong>How should this integration be implemented?</strong></p> + +<p>I am using OL 1.2.2, Azure Databricks Runtime 11.3 LTS. When creating a table writing into a ADLS location, OL won't be able to catch the location of the output. But when I read the same object it will be able to read the location as INPUT.</p> + +<p>Please note I have also tested Databricks Runtime 13.3 LTS, Spark 3.4.1 - it will give correct ADLS location in INPUT but the input will only show up once in a blue moon. Most of the time the inputs and outputs are blank.</p> + +<pre><code> "inputs": [], + "outputs": [] +</code></pre> + +<pre><code>CREATE OR REPLACE TABLE transactions_adj +USING DELTA LOCATION '&lt;wasbs://studio@clororetaildevadls.blob.core.windows.net/examples/data/csv/completejourney/silver/transactions_adj&gt;' +AS + SELECT + household_id, + basket_id, + week_no, + day, + transaction_time, + store_id, + product_id, + amount_list, + campaign_coupon_discount, + manuf_coupon_discount, + manuf_coupon_match_discount, + total_coupon_discount, + instore_discount, + amount_paid, + units + FROM ( + SELECT + household_id, + basket_id, + week_no, + day, + transaction_time, + store_id, + product_id, + COALESCE(sales_amount - discount_amount - coupon_discount - coupon_discount_match,0.0) as amount_list, + CASE + WHEN COALESCE(coupon_discount_match,0.0) = 0.0 THEN -1 ** COALESCE(coupon_discount,0.0) + ELSE 0.0 + END as campaign_coupon_discount, + CASE + WHEN COALESCE(coupon_discount_match,0.0) != 0.0 THEN -1 ** COALESCE(coupon_discount,0.0) + ELSE 0.0 + END as manuf_coupon_discount, + -1 ** COALESCE(coupon_discount_match,0.0) as manuf_coupon_match_discount, + -1 ** COALESCE(coupon_discount - coupon_discount_match,0.0) as total_coupon_discount, + COALESCE(-1 ** discount_amount,0.0) as instore_discount, + COALESCE(sales_amount,0.0) as `amount_paid,` + quantity as units + FROM transactions + ); +</code></pre> + +<p>Here's the COMPLETE event:</p> + +<pre><code> + "outputs":[ + { + "namespace":"dbfs", + "name":"/user/hive/warehouse/journey.db/transactions_adj", + "facets":{ + "dataSource":{ + "_producer":"&lt;https://github.com/OpenLineage/OpenLineage/tree/1.2.2/integration/spark&gt;", + "_schemaURL":"&lt;https://openlineage.io/spec/facets/1-0-0/DatasourceDatasetFacet.json#/$defs/DatasourceDatasetFacet&gt;", + "name":"dbfs", + "uri":"dbfs" + }, + +</code></pre> + +<p>Below logical plan shows the path:</p> + +<pre><code>== Analyzed Logical Plan == +num_affected_rows: bigint, num_inserted_rows: bigint +ReplaceTableAsSelect TableSpec(Map(),Some(DELTA),Map(),Some(&lt;wasbs://studio@clororetaildevadls.blob.core.windows.net/examples/data/csv/completejourney/silver/transactions_adj&gt;),None,None,false,Set()), true +:- ResolvedIdentifier com.databricks.sql.managedcatalog.UnityCatalogV2Proxy@6251a8df, default.transactions_adj ++- Project [household_id#184, basket_id#185L, week_no#193, day#186, transaction_time#192, store_id#190, product_id#187, amount_list#147, campaign_coupon_discount#148, manuf_coupon_discount#149, manuf_coupon_match_discount#150, total_coupon_discount#151, instore_discount#152, amount_paid#153, units#154] + +- SubqueryAlias __auto_generated_subquery_name + +- Project [household_id#184, basket_id#185L, week_no#193, day#186, transaction_time#192, store_id#190, product_id#187, coalesce(cast((((sales_amount#189 - discount_amount#191) - coupon_discount#194) - coupon_discount_match#195) as double), cast(0.0 as double)) AS amount_list#147, CASE WHEN (coalesce(cast(coupon_discount_match#195 as double), cast(0.0 as double)) = cast(0.0 as double)) THEN (cast(-1 as double) ** coalesce(cast(coupon_discount#194 as double), cast(0.0 as double))) ELSE cast(0.0 as double) END AS campaign_coupon_discount#148, CASE WHEN NOT (coalesce(cast(coupon_discount_match#195 as double), cast(0.0 as double)) = cast(0.0 as double)) THEN (cast(-1 as double) ** coalesce(cast(coupon_discount#194 as double), cast(0.0 as double))) ELSE cast(0.0 as double) END AS manuf_coupon_discount#149, (cast(-1 as double) ** coalesce(cast(coupon_discount_match#195 as double), cast(0.0 as double))) AS manuf_coupon_match_discount#150, (cast(-1 as double) ** coalesce(cast((coupon_discount#194 - coupon_discount_match#195) as double), cast(0.0 as double))) AS total_coupon_discount#151, coalesce(cast((cast(-1 as float) ** discount_amount#191) as double), cast(0.0 as double)) AS instore_discount#152, coalesce(cast(sales_amount#189 as double), cast(0.0 as double)) AS amount_paid#153, quantity#188 AS units#154] + +- SubqueryAlias spark_catalog.default.transactions + +- Relation spark_catalog.default.transactions[household_id#184,basket_id#185L,day#186,product_id#187,quantity#188,sales_amount#189,store_id#190,discount_amount#191,transaction_time#192,week_no#193,coupon_discount#194,coupon_discount_match#195] parquet +</code></pre> + +<p><strong>Where should this integration be implemented?</strong></p> + +<p>☐ In the target system +☐ In the OpenLineage repo +☐ Somewhere else</p> + +<p><strong>Do you plan to make this contribution yourself?</strong></p> + +<p>☐ I am interested in doing this work</p> +
+ +
+
Labels
+ integration/spark, integration/databricks +
+ +
+
Comments
+ 2 +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Guntaka Jeevan Paul + (jeevan@acceldata.io) +
+
2023-10-09 00:11:54
 +
+

*Thread Reply:* tried with databricks 12.2 --> spark 3.3.2, still the same behaviour no event getting emitted

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason Yip + (jasonyip@gmail.com) +
+
2023-10-09 00:12:35
 +
+

*Thread Reply:* you can do 11.3, its the most stable one I know

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Guntaka Jeevan Paul + (jeevan@acceldata.io) +
+
2023-10-09 00:12:46
 +
+

*Thread Reply:* sure, let me try that out

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Guntaka Jeevan Paul + (jeevan@acceldata.io) +
+
2023-10-09 00:31:51
 +
+

*Thread Reply:* still the same problem…the jar that i am using is the latest openlineage-spark-1.3.1.jar, do you think that can be the problem

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Guntaka Jeevan Paul + (jeevan@acceldata.io) +
+
2023-10-09 00:43:59
 +
+

*Thread Reply:* tried with openlineage-spark-1.2.2.jar, still the same issue, seems like they are skipping some events

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason Yip + (jasonyip@gmail.com) +
+
2023-10-09 01:47:20
 +
+

*Thread Reply:* Probably not all events will be captured, I have only tested create tables and jobs

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-10-09 04:31:12
 +
+

*Thread Reply:* Hi @Guntaka Jeevan Paul, how did you configure openlineage and what is your job doing?

+ +

We do have a bunch of integration tests on Databricks platform available here and they're passing on databricks runtime 13.0.x-scala2.12.

+ +

Could you also try running code same as our test does (this one)? If you run it and see OL events, this will make us sure your config is OK and we can continue further debug.

+ +

Looking at your spark script: could you save your dataset and see if you still don't see any events?

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/1.3.1/integration/spark/app/src/test/java/io/openlineage/spark/agent/DatabricksIntegrationTest.java | DatabricksIntegrationTest.java>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/1.3.1/integration/spark/app/src/test/resources/databricks_notebooks/ctas.py | ctas.py>
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Guntaka Jeevan Paul + (jeevan@acceldata.io) +
+
2023-10-09 05:06:41
 +
+

*Thread Reply:* babynames = spark.read.format("csv").option("header", "true").option("inferSchema", "true").load("dbfs:/FileStore/babynames.csv") +babynames.createOrReplaceTempView("babynames_table") +years = spark.sql("select distinct(Year) from babynames_table").rdd.map(lambda row : row[0]).collect() +years.sort() +dbutils.widgets.dropdown("year", "2014", [str(x) for x in years]) +display(babynames.filter(babynames.Year == dbutils.widgets.get("year")))

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Guntaka Jeevan Paul + (jeevan@acceldata.io) +
+
2023-10-09 05:08:09
 +
+

*Thread Reply:* this is the script that i am running @Paweł Leszczyński…kindly let me know if i’m doing any mistake. I have added the init script at the cluster level and from the logs i could see that openlineage is configured as i see a log statement

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-10-09 05:10:30
 +
+

*Thread Reply:* there's nothing wrong in that script. It's just we decided to limit amount of OL events for jobs that don't write their data anywhere and just do collect operation

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-10-09 05:11:02
 +
+

*Thread Reply:* this is also a potential reason why can't you see any events

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Guntaka Jeevan Paul + (jeevan@acceldata.io) +
+
2023-10-09 05:14:33
 +
+

*Thread Reply:* ohh…okk, will try out the test script that you have mentioned above. Kindly correct me if my understanding is correct, so if there are a few transformatiosna nd finally writing somewhere that is where the OL events are expected to be emitted?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-10-09 05:16:54
 +
+

*Thread Reply:* yes. main purpose of the lineage is to track dependencies between the datasets, when a job reads from dataset A and writes to dataset B. In case of databricks notebook, that do show or collect and print some query result on the screen, there may be no reason to track it in the sense of lineage.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-10-09 15:25:14
 +
+

@channel +We released OpenLineage 1.4.1! +Additions: +• Client: allow setting client’s endpoint via environment variable 2151 @Mars Lan +• Flink: expand Iceberg source types 2149 @Peter Huang +• Spark: add debug facet 2147 @Paweł Leszczyński +• Spark: enable Nessie REST catalog 2165 @julwin +Thanks to all the contributors, especially new contributors @Peter Huang and @julwin! +Release: https://github.com/OpenLineage/OpenLineage/releases/tag/1.4.1 +Changelog: https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md +Commit history: https://github.com/OpenLineage/OpenLineage/compare/1.3.1...1.4.1 +Maven: https://oss.sonatype.org/#nexus-search;quick~openlineage +PyPI: https://pypi.org/project/openlineage-python/

+ + + +
+ 👍 Jason Yip, Ross Turk, Mars Lan, Harel Shein, Rodrigo Maia +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Drew Bittenbender + (drew@salt.io) +
+
2023-10-09 16:55:35
 +
+

Hello. I am getting started with OL and Marquez with dbt. I am using dbt-ol. The namespace of the dataset showing up in Marquez is not the namespace I provide using OPENLINEAGENAMESPACE. It happens to be the same as the source in Marquez which is the snowflake account uri. It's obviously picking up the other env variable OPENLINEAGEURL so i am pretty sure its not the environment. Is this expected?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-10-09 18:56:13
 +
+

*Thread Reply:* Hi Drew, thank you for using OpenLineage! I don’t know the details of your use case, but I believe this is expected, yes. In general, the dataset namespace is different. Jobs are namespaced separately from datasets, which are namespaced by their containing datasources. This is the case so datasets have the same name regardless of the job writing to them, as datasets are sometimes shared by jobs in different namespaces.

+ + + +
+ 👍 Drew Bittenbender +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason Yip + (jasonyip@gmail.com) +
+
2023-10-10 01:05:11
 +
+

Any idea why "environment-properties" is gone in Spark 3.4+ in StartEvent?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason Yip + (jasonyip@gmail.com) +
+
2023-10-10 20:53:59
 +
+

example:

+ +

{"environment_properties":{"spark.databricks.clusterUsageTags.clusterName":"<a href="mailto:jason.yip@tredence.com">jason.yip@tredence.com</a>'s Cluster","spark.databricks.job.runId":"","spark.databricks.job.type":"","spark.databricks.clusterUsageTags.azureSubscriptionId":"a4f54399_8db8_4849_adcc_a42aed1fb97f","spark.databricks.notebook.path":"/Repos/jason.yip@tredence.com/segmentation/01_Data Prep","spark.databricks.clusterUsageTags.clusterOwnerOrgId":"4679476628690204","MountPoints":[{"MountPoint":"/databricks-datasets","Source":"databricks_datasets"},{"MountPoint":"/Volumes","Source":"UnityCatalogVolumes"},{"MountPoint":"/databricks/mlflow-tracking","Source":"databricks/mlflow-tracking"},{"MountPoint":"/databricks-results","Source":"databricks_results"},{"MountPoint":"/databricks/mlflow-registry","Source":"databricks/mlflow-registry"},{"MountPoint":"/Volume","Source":"DbfsReserved"},{"MountPoint":"/volumes","Source":"DbfsReserved"},{"MountPoint":"/","Source":"DatabricksRoot"},{"MountPoint":"/volume","Source":"DbfsReserved"}],"User":"<a href="mailto:jason.yip@tredence.com">jason.yip@tredence.com</a>","UserId":"4768657035718622","OrgId":"4679476628690204"}}

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-10-11 03:46:13
 +
+

*Thread Reply:* Is this related to any OL version? In OL 1.2.2. we've added extra variable spark.databricks.clusterUsageTags.clusterAllTags to be captured, but this should not break things.

+ +

I think we're facing some issues on recent databricks runtime versions. Here is an issue for this: https://github.com/OpenLineage/OpenLineage/issues/2131

+ +

Is the problem you describe specific to some databricks runtime versions?

+
+ + + + +
#2131 [Spark Databricks] NoSuchMethodError on ReplaceTableAsSelect
+
+ <p>More details here: <a href="https://github.com/OpenLineage/OpenLineage/issues/2121">#2121</a> <br /> +Looks like different class implementation on databricks platform.</p> +
+ +
+
Labels
+ integration/spark, integration/databricks +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason Yip + (jasonyip@gmail.com) +
+
2023-10-11 11:17:06
 +
+

*Thread Reply:* yes, exactly Spark 3.4+

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason Yip + (jasonyip@gmail.com) +
+
2023-10-11 21:12:27
 +
+

*Thread Reply:* Btw I don't understand the code flow entirely, if we are talking about a different classpath only, I see there's Unity Catalog handler in the code and it says it works the same as Delta, but I am not seeing it subclassing Delta. I suppose it will work the same.

+ +

I am happy to jump on a call to show you if needed

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason Yip + (jasonyip@gmail.com) +
+
2023-10-16 02:58:56
 +
+

*Thread Reply:* @Paweł Leszczyński do you think in Spark 3.4+ only one event would happen?

+ +

/** + * We get exact copies of OL events for org.apache.spark.scheduler.SparkListenerJobStart and + * org.apache.spark.sql.execution.ui.SparkListenerSQLExecutionStart. The same happens for end + * events. + * + * @return + */ + private boolean isOnJobStartOrEnd(SparkListenerEvent event) { + return event instanceof SparkListenerJobStart || event instanceof SparkListenerJobEnd; + }

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Guntaka Jeevan Paul + (jeevan@acceldata.io) +
+
2023-10-10 23:43:39
 +
+

@here i am trying out the databricks spark integration and in one of the events i am getting a openlineage event where the output dataset is having a facet called symlinks , the statement that generated this event is this sql +CREATE TABLE IF NOT EXISTS covid_research.covid_data +USING CSV +LOCATION '<abfss://oltptestdata@jeevanacceldata.dfs.core.windows.net/testdata/johns-hopkins-covid-19-daily-dashboard-cases-by-states.csv>' +OPTIONS (header "true", inferSchema "true"); +Can someone kindly let me know what this symlinks facet is. i tried seeing the spec but did not get it completely

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason Yip + (jasonyip@gmail.com) +
+
2023-10-10 23:44:53
 +
+

*Thread Reply:* I use it to get the table with database name

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Guntaka Jeevan Paul + (jeevan@acceldata.io) +
+
2023-10-10 23:47:15
 +
+

*Thread Reply:* so can i think it like if there is a synlink, then that table is kind of a reference to the original dataset

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason Yip + (jasonyip@gmail.com) +
+
2023-10-11 01:25:44
 +
+

*Thread Reply:* yes

+ + + +
+ 🙌 Paweł Leszczyński +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Guntaka Jeevan Paul + (jeevan@acceldata.io) +
+
2023-10-11 06:55:58
 +
+

@here When i am running this sql as part of a databricks notebook, i am recieving an OL event where i see only an output dataset and there is no input dataset or a symlink facet inside the dataset to map it to the underlying azure storage object. Can anyone kindly help on this +spark.sql(f"CREATE TABLE IF NOT EXISTS covid_research.uscoviddata USING delta LOCATION '<abfss://oltptestdata@jeevanacceldata.dfs.core.windows.net/testdata/modified-delta>'") +{ + "eventTime": "2023-10-11T10:47:36.296Z", + "producer": "<https://github.com/OpenLineage/OpenLineage/tree/1.2.2/integration/spark>", + "schemaURL": "<https://openlineage.io/spec/2-0-2/OpenLineage.json#/$defs/RunEvent>", + "eventType": "COMPLETE", + "run": { + "runId": "d0f40be9-b921-4c84-ac9f-f14a86c29ff7", + "facets": { + "spark.logicalPlan": { + "_producer": "<https://github.com/OpenLineage/OpenLineage/tree/1.2.2/integration/spark>", + "_schemaURL": "<https://openlineage.io/spec/2-0-2/OpenLineage.json#/$defs/RunFacet>", + "plan": [ + { + "class": "org.apache.spark.sql.catalyst.plans.logical.CreateTable", + "num-children": 1, + "name": 0, + "tableSchema": [], + "partitioning": [], + "tableSpec": null, + "ignoreIfExists": true + }, + { + "class": "org.apache.spark.sql.catalyst.analysis.ResolvedIdentifier", + "num-children": 0, + "catalog": null, + "identifier": null + } + ] + }, + "spark_version": { + "_producer": "<https://github.com/OpenLineage/OpenLineage/tree/1.2.2/integration/spark>", + "_schemaURL": "<https://openlineage.io/spec/2-0-2/OpenLineage.json#/$defs/RunFacet>", + "spark-version": "3.3.0", + "openlineage-spark-version": "1.2.2" + }, + "processing_engine": { + "_producer": "<https://github.com/OpenLineage/OpenLineage/tree/1.2.2/integration/spark>", + "_schemaURL": "<https://openlineage.io/spec/facets/1-1-0/ProcessingEngineRunFacet.json#/$defs/ProcessingEngineRunFacet>", + "version": "3.3.0", + "name": "spark", + "openlineageAdapterVersion": "1.2.2" + } + } + }, + "job": { + "namespace": "default", + "name": "adb-3942203504488904.4.azuredatabricks.net.create_table.covid_research_db_uscoviddata", + "facets": {} + }, + "inputs": [], + "outputs": [ + { + "namespace": "dbfs", + "name": "/user/hive/warehouse/covid_research.db/uscoviddata", + "facets": { + "dataSource": { + "_producer": "<https://github.com/OpenLineage/OpenLineage/tree/1.2.2/integration/spark>", + "_schemaURL": "<https://openlineage.io/spec/facets/1-0-0/DatasourceDatasetFacet.json#/$defs/DatasourceDatasetFacet>", + "name": "dbfs", + "uri": "dbfs" + }, + "schema": { + "_producer": "<https://github.com/OpenLineage/OpenLineage/tree/1.2.2/integration/spark>", + "_schemaURL": "<https://openlineage.io/spec/facets/1-0-0/SchemaDatasetFacet.json#/$defs/SchemaDatasetFacet>", + "fields": [] + }, + "storage": { + "_producer": "<https://github.com/OpenLineage/OpenLineage/tree/1.2.2/integration/spark>", + "_schemaURL": "<https://openlineage.io/spec/facets/1-0-0/StorageDatasetFacet.json#/$defs/StorageDatasetFacet>", + "storageLayer": "unity", + "fileFormat": "parquet" + }, + "symlinks": { + "_producer": "<https://github.com/OpenLineage/OpenLineage/tree/1.2.2/integration/spark>", + "_schemaURL": "<https://openlineage.io/spec/facets/1-0-0/SymlinksDatasetFacet.json#/$defs/SymlinksDatasetFacet>", + "identifiers": [ + { + "namespace": "/user/hive/warehouse/covid_research.db", + "name": "covid_research.uscoviddata", + "type": "TABLE" + } + ] + }, + "lifecycleStateChange": { + "_producer": "<https://github.com/OpenLineage/OpenLineage/tree/1.2.2/integration/spark>", + "_schemaURL": "<https://openlineage.io/spec/facets/1-0-0/LifecycleStateChangeDatasetFacet.json#/$defs/LifecycleStateChangeDatasetFacet>", + "lifecycleStateChange": "CREATE" + } + }, + "outputFacets": {} + } + ] +}

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Damien Hawes + (damien.hawes@booking.com) +
+
2023-10-11 06:57:46
 +
+

*Thread Reply:* Hey Guntaka - can I ask you a favour? Can you please stop using @here or @channel - please keep in mind, you're pinging over 1000 people when you use that mention. Its incredibly distracting to have Slack notify me of a message that isn't pertinent to me.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Guntaka Jeevan Paul + (jeevan@acceldata.io) +
+
2023-10-11 06:58:50
 +
+

*Thread Reply:* sure noted @Damien Hawes

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Damien Hawes + (damien.hawes@booking.com) +
+
2023-10-11 06:59:34
 +
+

*Thread Reply:* Thank you!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Madhav Kakumani + (madhav.kakumani@6point6.co.uk) +
+
2023-10-11 12:04:24
 +
+

Hi @there, I am trying to make API call to get column-lineage information could you please let me know the url construct to retrieve the same? As per the API documentation I am passing the following url to GET column-lineage: http://localhost:5000/api/v1/column-lineage but getting error code:400. Thanks

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2023-10-12 13:55:26
 +
+

*Thread Reply:* Make sure to provide a dataset field nodeId as a query param in your request. If you’ve seeded Marquez with test metadata, you can use: +curl -XGET "<http://localhost:5002/api/v1/column-lineage?nodeId=datasetField%3Afood_delivery%3Apublic.delivery_7_days%3Acustomer_email>" +You can view the API docs for column lineage here!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Madhav Kakumani + (madhav.kakumani@6point6.co.uk) +
+
2023-10-17 05:57:36
 +
+

*Thread Reply:* Thanks Willy. The documentation says 'name space' so i constructed API Like this: +'http://marquez-web:3000/api/v1/column-lineage/nodeId=datasetField:file:/home/jovyan/Downloads/event_attribute.csv:eventType' +but it is still not working 😞

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Madhav Kakumani + (madhav.kakumani@6point6.co.uk) +
+
2023-10-17 06:07:06
 +
+

*Thread Reply:* nodeId is constructed like this: datasetField:<namespace>:<dataset>:<field name>

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-10-11 13:00:01
 +
+

@channel +Friendly reminder: this month’s TSC meeting, open to all, is tomorrow at 10 am PT: https://openlineage.slack.com/archives/C01CK9T7HKR/p1696531454431629

+
+ + +
+ + + } + + Michael Robinson + (https://openlineage.slack.com/team/U02LXF3HUN7) +
+ + + +
+
+ <p><strong>@channel</strong> +This month’s TSC meeting is next Thursday the 12th at 10am PT. On the tentative agenda: +• announcements +• recent releases +• Airflow Summit recap +• tutorial: migrating to the Airflow Provider +• discussion topic: observability for OpenLineage/Marquez +• open discussion +• more (TBA) +More info and the meeting link can be found on the <a href="https://openlineage.io/meetings/">website</a>. All are welcome! Do you have a discussion topic, use case or integration you’d like to demo? DM me to be added to the agenda.</p> +
+ + + + + + +
Original URL: https://openlineage.slack.com/archives/C01CK9T7HKR/p1696531454431629
+ + +
+ + Slack Conversation +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-10-11 14:26:45
 +
+

*Thread Reply:* Newly added discussion topics: +• a proposal to add a Registry of Consumers and Producers +• a dbt issue to add OpenLineage Dataset names to the Manifest +• a proposal to add Dataset support in Spark LogicalPlan Nodes +• a proposal to institute a certification process for new integrations

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason Yip + (jasonyip@gmail.com) +
+
2023-10-12 15:08:34
 +
+

This might be a dumb question, I guess I need to setup local Spark in order for the Spark tests to run successfully?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-10-13 01:56:19
 +
+

*Thread Reply:* just follow these instructions: https://github.com/OpenLineage/OpenLineage/tree/main/integration/spark#build

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Guntaka Jeevan Paul + (jeevan@acceldata.io) +
+
2023-10-13 06:41:56
 +
+

*Thread Reply:* when trying to install openlineage-java in local via this command --> cd ../../client/java/ && ./gradlew publishToMavenLocal, i am receiving this error +```> Task :signMavenJavaPublication FAILED

+ +

FAILURE: Build failed with an exception.

+ +

** What went wrong: +Execution failed for task ':signMavenJavaPublication'. +> Cannot perform signing task ':signMavenJavaPublication' because it has no configured signatory```

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason Yip + (jasonyip@gmail.com) +
+
2023-10-13 13:35:06
 +
+

*Thread Reply:* @Paweł Leszczyński this is what I am getting

+ +
+
image.png
+ + + + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason Yip + (jasonyip@gmail.com) +
+
2023-10-13 13:36:00
 +
+

*Thread Reply:* attaching the html

+ +
+
io.openlineage.spark.agent.lifecycle.plan.AlterTableAddPartitionCommandVisitorTest.html
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-10-16 03:02:13
 +
+

*Thread Reply:* which java are you using? what is your operation system (is it windows?)?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason Yip + (jasonyip@gmail.com) +
+
2023-10-16 03:35:18
 +
+

*Thread Reply:* yes it is Windows, i downloaded java 8 but I can try to build it with Linux subsystem or Mac

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Guntaka Jeevan Paul + (jeevan@acceldata.io) +
+
2023-10-16 03:35:51
 +
+

*Thread Reply:* In my case it is Mac

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason Yip + (jasonyip@gmail.com) +
+
2023-10-16 03:56:09
 +
+

*Thread Reply: * Where: +Build file '/mnt/c/Users/jason/Downloads/github/OpenLineage/integration/spark/build.gradle' line: 9

+ +

** What went wrong: +An exception occurred applying plugin request [id: 'com.adarshr.test-logger', version: '3.2.0'] +> Failed to apply plugin [id 'com.adarshr.test-logger'] + > Could not generate a proxy class for class com.adarshr.gradle.testlogger.TestLoggerExtension.

+ +

** Try:

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason Yip + (jasonyip@gmail.com) +
+
2023-10-16 03:56:23
 +
+

*Thread Reply:* tried with Linux subsystem

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-10-16 04:04:29
 +
+

*Thread Reply:* we don't have any restrictions for windows builds, however it is something we don't test regularly. 2h ago we did have a successful build on circle CI https://app.circleci.com/pipelines/github/OpenLineage/OpenLineage/8271/workflows/0ec521ae-cd21-444a-bfec-554d101770ea

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason Yip + (jasonyip@gmail.com) +
+
2023-10-16 04:13:04
 +
+

*Thread Reply:* ... 111 more +Caused by: java.lang.ClassNotFoundException: org.gradle.api.provider.HasMultipleValues + ... 117 more

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason Yip + (jasonyip@gmail.com) +
+
2023-10-17 00:26:07
 +
+

*Thread Reply:* @Paweł Leszczyński now I am doing gradlew instead of gradle on windows coz Linux one doesn't work. The doc didn't mention about setting up Spark / Hadoop and that's my original question -- do I need to setup local Spark? Now it's throwing an error on Hadoop: java.io.FileNotFoundException: java.io.FileNotFoundException: HADOOP_HOME and hadoop.home.dir are unset.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason Yip + (jasonyip@gmail.com) +
+
2023-10-21 23:33:48
 +
+

*Thread Reply:* Got it working with Mac, couldn't get it working with Windows / Linux subsystem

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason Yip + (jasonyip@gmail.com) +
+
2023-10-22 13:08:40
 +
+

*Thread Reply:* Now getting class not found despite build and test succeeded

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason Yip + (jasonyip@gmail.com) +
+
2023-10-22 21:46:23
 +
+

*Thread Reply:* I uploaded the wrong jar.. there are so many jars, only the jar in the spark folder works, not subfolder

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-10-13 02:48:40
 +
+

Hi team, I am running the following pyspark code in a cell: +```print("SELECTING 100 RECORDS FROM METADATA TABLE") +df = spark.sql("""select ** from

limit 100""")

+ +

print("WRITING (1) 100 RECORDS FROM METADATA TABLE") +df.write.mode("overwrite").format('delta').save("") +df.createOrReplaceTempView("temp_metadata")

+ +

print("WRITING (2) 100 RECORDS FROM METADATA TABLE") +df.write.mode("overwrite").format("delta").save("")

+ +

print("READING (1) 100 RECORDS FROM METADATA TABLE") +dfread = spark.read.format('delta').load("") +dfread.createOrReplaceTempView("metadata_1")

+ +

print("DOING THE MERGE INTO SQL STEP!") +dfnew = spark.sql(""" + MERGE INTO metadata1 + USING

+ ON metadata1.id = tempmetadata.id + WHEN MATCHED THEN UPDATE SET + metadata1.id = tempmetadata.id, + metadata1.aspect = tempmetadata.aspect + WHEN NOT MATCHED THEN INSERT (id, aspect) + VALUES (tempmetadata.id, tempmetadata.aspect) +""")`` +I am running with debug log levels. I actually don't see any of the events being logged forSaveIntoDataSourceCommandor theMergeIntoCommand`, but OL is in fact emitting events to the backend. It seems like the events are just not being logged... I actually observe this for all delta table related spark sql queries...

+ + + + + + + + + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-10-16 00:01:42
 +
+

*Thread Reply:* Hi @Paweł Leszczyński is this expected? CMIIW but we should expect to see the events being logged when running with debug log level right?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Damien Hawes + (damien.hawes@booking.com) +
+
2023-10-16 04:17:30
 +
+

*Thread Reply:* It's impossible to know without seeing how you've configured the listener.

+ +

Can you show this configuration?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-10-17 03:15:20
 +
+

*Thread Reply:* spark.openlineage.transport.url &lt;url&gt; +spark.openlineage.transport.endpoint /&lt;endpoint&gt; +spark.openlineage.transport.type http +spark.extraListeners io.openlineage.spark.agent.OpenLineageSparkListener +spark.openlineage.facets.custom_environment_variables [BUNCH_OF_VARIABLES;] +spark.openlineage.facets.disabled [spark_unknown\;spark.logicalPlan] +These are my spark configs... I'm setting log level to debug with sc.setLogLevel("DEBUG")

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Damien Hawes + (damien.hawes@booking.com) +
+
2023-10-17 04:40:03
 +
+

*Thread Reply:* Two things:

+ +
  1. If you want debug logs, you're going to have to provide a log4j.properties file or log4j2.properties file depending on the version of spark you're running. In that file, you will need to configure the logging levels. If I am not mistaken, the sc.setLogLevel controls ONLY the log levels of Spark namespaced components (i.e., org.apache.spark)
  2. You're telling the listener to emit to a URL. If you want to see the events emitted to the console, then set spark.openlineage.transport.type=console, and remove the other spark.openlineage.transport.** configurations. +Do either (1) or (2).
    3. +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-10-20 00:49:45
 +
+

*Thread Reply:* @Damien Hawes Hi, sflr.

+ +
  1. So enabling sc.setLogLevel does actually enable debug logs from Openlineage. I can see the events and everyting being logged if I save it as a parquet format instead of delta.
  2. I do want to emit events to the url. But, I would like to just see what exactly are the events being emitted for some specific jobs, since I see that the lineage is incorrect for some MergeInto cases
    3. +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-10-26 04:56:50
 +
+

*Thread Reply:* Hi @Damien Hawes would like to check again on whether you'd have any thoughts about this... Thanks! 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Rodrigo Maia + (rodrigo.maia@manta.io) +
+
2023-10-17 03:17:57
 +
+

Hello All 👋! +We are currently trying to work the the spark integration for OpenLineage in our Databricks instance. The general setup is done and working with a few hicups here and there. +But one thing we are still struggling is how to link all spark jobs events with a Databricks job or a notebook run. +We´ve recently noticed that some of the events produced by OL have the "environment-properties" attribute with information (for our context) regarding notebook path (if it is a notebook run), or the the job run ID (if its a databricks job run). But the thing is that these attributes are not always present. +I ran some samples yesterday for a job with 4 notebook tasks. From all 20 json payload sent by the OL listener, only 3 presented the "environment-properties" attribute. Its not only happening with Databricks jobs. When i run single notebooks and each cell has its onw set of spark jobs, not all json events presented that property either.

+ +

So my question is what is the criteria to have this attributes present or not in the event json file? Or maybe this in an issue? @Jason Yip did you find out anything about this?

+ +

⚙️ Spark 3.4 / OL-Spark 1.4.1

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-10-17 06:55:47
 +
+

*Thread Reply:* In general, we assume that OL events per run are cumulative. So, if you have 20 events with the same runId , then even if a single event contains some facet, we consider this is OK and let the backend combine it together. That's what we do in Marquez project (a reference backend architecture for OL) and that's why it is worth to use in Marquez as a rest API.

+ +

Are you able to use job namespace to aggregate all the Spark actions run within the databricks notebook? This is something that should serve this purpose.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason Yip + (jasonyip@gmail.com) +
+
2023-10-17 12:48:33
 +
+

*Thread Reply:* @Rodrigo Maia for Spark 3.4 I don't see the environment-properties showing up at all, but if you run the code as it is, register a listener on SparkListenerJobStart and get the properties, all of those properties will show up. There's an event filter that filters out the SparkListenerJobStart, I suspect that filtered out the "unneccessary" events.. was trying to do a custom build to do that, but still trying to setup Hadoop and Spark on my local

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Rodrigo Maia + (rodrigo.maia@manta.io) +
+
2023-10-18 05:23:16
 +
+

*Thread Reply:* @Paweł Leszczyński you are right. This is what we are doing as well, combining events with the same runId to process the information on our backend. But even so, there are several runIds without this information. I went through these events to have a better view of what was happening. As you can see from 7 runIds, only 3 were showing the "environment-properties" attribute. Some condition is not being met here, or maybe it is what @Jason Yip suspects and there's some sort of filtering of unnecessary events

+ +
+
image.png
+ + + + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-10-19 02:28:03
 +
+

*Thread Reply:* @Rodrigo Maia, If you are able to provide a small Spark script such that none of the OL events contain the environment-properties, but at least one should, please raise an issue for this.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-10-19 02:29:11
 +
+

*Thread Reply:* It's extremely helpful when community open issues that are not only described well, but also contain small piece of code needed to reproduce this.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Rodrigo Maia + (rodrigo.maia@manta.io) +
+
2023-10-19 02:59:39
 +
+

*Thread Reply:* I know. that's the goal. that is why I wanted to understand in the first place if there was any condition preventing this from happening, but now i get that this is not expected behaviour.

+ + + +
+ 👍 Paweł Leszczyński +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason Yip + (jasonyip@gmail.com) +
+
2023-10-19 13:44:00
 +
+

*Thread Reply:* @Paweł Leszczyński @Rodrigo Maia I am referring to this: https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/shared/src/main/java/io/openlineage/spark/agent/filters/DeltaEventFilter.java#L51

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/shared/src/main/java/io/openlineage/spark/agent/filters/DeltaEventFilter.java | DeltaEventFilter.java>
+
+ <pre><code> || isOnJobStartOrEnd(event); +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason Yip + (jasonyip@gmail.com) +
+
2023-10-19 14:49:03
 +
+

*Thread Reply:* Please note that I am getting the same behavior, no code is needed, Spark 3.4+ won't be generating no matter what. I have been testing the same code for 2 months from this issue: https://github.com/OpenLineage/OpenLineage/issues/2124

+ +

I tried the code without OL and it worked perfectly, so it is OL filtering out the event for sure. I will try posting the code I use to collect the properties.

+
+ + + + +
#2124 Same Delta Table not catching the location on write
+
+ <p><strong>What is the target system?</strong></p> + +<p>Spark / Databricks</p> + +<p><strong>What kind of integration is this?</strong></p> + +<p>☐ Produces OpenLineage metadata +☐ Consumes OpenLineage metadata +☐ Something else</p> + +<p><strong>How should this integration be implemented?</strong></p> + +<p>I am using OL 1.2.2, Azure Databricks Runtime 11.3 LTS. When creating a table writing into a ADLS location, OL won't be able to catch the location of the output. But when I read the same object it will be able to read the location as INPUT.</p> + +<p>Please note I have also tested Databricks Runtime 13.3 LTS, Spark 3.4.1 - it will give correct ADLS location in INPUT but the input will only show up once in a blue moon. Most of the time the inputs and outputs are blank.</p> + +<pre><code> "inputs": [], + "outputs": [] +</code></pre> + +<pre><code>CREATE OR REPLACE TABLE transactions_adj +USING DELTA LOCATION '&lt;wasbs://studio@clororetaildevadls.blob.core.windows.net/examples/data/csv/completejourney/silver/transactions_adj&gt;' +AS + SELECT + household_id, + basket_id, + week_no, + day, + transaction_time, + store_id, + product_id, + amount_list, + campaign_coupon_discount, + manuf_coupon_discount, + manuf_coupon_match_discount, + total_coupon_discount, + instore_discount, + amount_paid, + units + FROM ( + SELECT + household_id, + basket_id, + week_no, + day, + transaction_time, + store_id, + product_id, + COALESCE(sales_amount - discount_amount - coupon_discount - coupon_discount_match,0.0) as amount_list, + CASE + WHEN COALESCE(coupon_discount_match,0.0) = 0.0 THEN -1 ** COALESCE(coupon_discount,0.0) + ELSE 0.0 + END as campaign_coupon_discount, + CASE + WHEN COALESCE(coupon_discount_match,0.0) != 0.0 THEN -1 ** COALESCE(coupon_discount,0.0) + ELSE 0.0 + END as manuf_coupon_discount, + -1 ** COALESCE(coupon_discount_match,0.0) as manuf_coupon_match_discount, + -1 ** COALESCE(coupon_discount - coupon_discount_match,0.0) as total_coupon_discount, + COALESCE(-1 ** discount_amount,0.0) as instore_discount, + COALESCE(sales_amount,0.0) as `amount_paid,` + quantity as units + FROM transactions + ); +</code></pre> + +<p>Here's the COMPLETE event:</p> + +<pre><code> + "outputs":[ + { + "namespace":"dbfs", + "name":"/user/hive/warehouse/journey.db/transactions_adj", + "facets":{ + "dataSource":{ + "_producer":"&lt;https://github.com/OpenLineage/OpenLineage/tree/1.2.2/integration/spark&gt;", + "_schemaURL":"&lt;https://openlineage.io/spec/facets/1-0-0/DatasourceDatasetFacet.json#/$defs/DatasourceDatasetFacet&gt;", + "name":"dbfs", + "uri":"dbfs" + }, + +</code></pre> + +<p>Below logical plan shows the path:</p> + +<pre><code>== Analyzed Logical Plan == +num_affected_rows: bigint, num_inserted_rows: bigint +ReplaceTableAsSelect TableSpec(Map(),Some(DELTA),Map(),Some(&lt;wasbs://studio@clororetaildevadls.blob.core.windows.net/examples/data/csv/completejourney/silver/transactions_adj&gt;),None,None,false,Set()), true +:- ResolvedIdentifier com.databricks.sql.managedcatalog.UnityCatalogV2Proxy@6251a8df, default.transactions_adj ++- Project [household_id#184, basket_id#185L, week_no#193, day#186, transaction_time#192, store_id#190, product_id#187, amount_list#147, campaign_coupon_discount#148, manuf_coupon_discount#149, manuf_coupon_match_discount#150, total_coupon_discount#151, instore_discount#152, amount_paid#153, units#154] + +- SubqueryAlias __auto_generated_subquery_name + +- Project [household_id#184, basket_id#185L, week_no#193, day#186, transaction_time#192, store_id#190, product_id#187, coalesce(cast((((sales_amount#189 - discount_amount#191) - coupon_discount#194) - coupon_discount_match#195) as double), cast(0.0 as double)) AS amount_list#147, CASE WHEN (coalesce(cast(coupon_discount_match#195 as double), cast(0.0 as double)) = cast(0.0 as double)) THEN (cast(-1 as double) ** coalesce(cast(coupon_discount#194 as double), cast(0.0 as double))) ELSE cast(0.0 as double) END AS campaign_coupon_discount#148, CASE WHEN NOT (coalesce(cast(coupon_discount_match#195 as double), cast(0.0 as double)) = cast(0.0 as double)) THEN (cast(-1 as double) ** coalesce(cast(coupon_discount#194 as double), cast(0.0 as double))) ELSE cast(0.0 as double) END AS manuf_coupon_discount#149, (cast(-1 as double) ** coalesce(cast(coupon_discount_match#195 as double), cast(0.0 as double))) AS manuf_coupon_match_discount#150, (cast(-1 as double) ** coalesce(cast((coupon_discount#194 - coupon_discount_match#195) as double), cast(0.0 as double))) AS total_coupon_discount#151, coalesce(cast((cast(-1 as float) ** discount_amount#191) as double), cast(0.0 as double)) AS instore_discount#152, coalesce(cast(sales_amount#189 as double), cast(0.0 as double)) AS amount_paid#153, quantity#188 AS units#154] + +- SubqueryAlias spark_catalog.default.transactions + +- Relation spark_catalog.default.transactions[household_id#184,basket_id#185L,day#186,product_id#187,quantity#188,sales_amount#189,store_id#190,discount_amount#191,transaction_time#192,week_no#193,coupon_discount#194,coupon_discount_match#195] parquet +</code></pre> + +<p><strong>Where should this integration be implemented?</strong></p> + +<p>☐ In the target system +☐ In the OpenLineage repo +☐ Somewhere else</p> + +<p><strong>Do you plan to make this contribution yourself?</strong></p> + +<p>☐ I am interested in doing this work</p> +
+ +
+
Labels
+ integration/spark, integration/databricks +
+ +
+
Comments
+ 3 +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason Yip + (jasonyip@gmail.com) +
+
2023-10-19 23:46:17
 +
+

*Thread Reply:* this code proves that the prosperities are still there, somehow got filtered out by OL:

+ +

```%scala +import org.apache.spark.scheduler._

+ +

class JobStartListener extends SparkListener { + override def onJobStart(jobStart: SparkListenerJobStart): Unit = { + // Extract properties here + val jobId = jobStart.jobId + val stageInfos = jobStart.stageInfos + val properties = jobStart.properties

+ +
// You can print properties or save them somewhere
+println(s"JobId: $jobId, Stages: ${stageInfos.size}, Properties: $properties")
+
+ +

} +}

+ +

val listener = new JobStartListener() +spark.sparkContext.addSparkListener(listener)

+ +

val df = spark.range(1000).repartition(10) +df.count()```

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason Yip + (jasonyip@gmail.com) +
+
2023-10-19 23:55:05
 +
+

*Thread Reply:* of course feel free to test this logic as well, it still works -- if not the filtering:

+ +

https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/shared/src/[…]ark/agent/facets/builder/DatabricksEnvironmentFacetBuilder.java

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/shared/src/main/java/io/openlineage/spark/agent/facets/builder/DatabricksEnvironmentFacetBuilder.java | DatabricksEnvironmentFacetBuilder.java>
+
+ <pre><code>/** +/** Copyright 2018-2023 contributors to the OpenLineage project +/** SPDX-License-Identifier: Apache-2.0 +**/ + +package io.openlineage.spark.agent.facets.builder; + +import com.databricks.backend.daemon.dbutils.MountInfo; +import com.databricks.dbutils_v1.DbfsUtils; +import io.openlineage.spark.agent.facets.EnvironmentFacet; +import io.openlineage.spark.agent.models.DatabricksMountpoint; +import io.openlineage.spark.api.CustomFacetBuilder; +import io.openlineage.spark.api.OpenLineageContext; +import java.lang.reflect.Constructor; +import java.lang.reflect.InvocationTargetException; +import java.lang.reflect.Parameter; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.Optional; +import java.util.function.BiConsumer; +import lombok.extern.slf4j.Slf4j; +import org.apache.spark.scheduler.SparkListenerJobStart; +import scala.collection.JavaConversions; + +/**** + ** {@link CustomFacetBuilder} that generates a {@link EnvironmentFacet} when using OpenLineage on + ** Databricks. + **/ +@Slf4j +public class DatabricksEnvironmentFacetBuilder + extends CustomFacetBuilder&lt;SparkListenerJobStart, EnvironmentFacet&gt; { + private Map&lt;String, Object&gt; dbProperties; + private Class dbutilsClass; + private DbfsUtils dbutils; + + public static boolean isDatabricksRuntime() { + return System.getenv().containsKey("DATABRICKS_RUNTIME_VERSION"); + } + + public DatabricksEnvironmentFacetBuilder() {} + + public DatabricksEnvironmentFacetBuilder(OpenLineageContext openLineageContext) { + dbProperties = new HashMap&lt;&gt;(); + // extract some custom environment variables if needed + openLineageContext + .getCustomEnvironmentVariables() + .ifPresent( + envVars -&gt; + envVars.forEach(envVar -&gt; dbProperties.put(envVar, System.getenv().get(envVar)))); + } + + @Override + protected void build( + SparkListenerJobStart event, BiConsumer&lt;String, ? super EnvironmentFacet&gt; consumer) { + consumer.accept( + "environment-properties", + new EnvironmentFacet(getDatabricksEnvironmentalAttributes(event))); + } + + private Map&lt;String, Object&gt; getDatabricksEnvironmentalAttributes(SparkListenerJobStart jobStart) { + if (dbProperties == null) { + dbProperties = new HashMap&lt;&gt;(); + } + + // These are useful properties to extract if they are available + List&lt;String&gt; dbPropertiesKeys = + Arrays.asList( + "orgId", + "spark.databricks.clusterUsageTags.clusterOwnerOrgId", + "spark.databricks.notebook.path", + "spark.databricks.job.type", + "spark.databricks.job.id", + "spark.databricks.job.runId", + "user", + "userId", + "spark.databricks.clusterUsageTags.clusterName", + "spark.databricks.clusterUsageTags.clusterAllTags", + "spark.databricks.clusterUsageTags.azureSubscriptionId"); + dbPropertiesKeys.stream() + .forEach( + (p) -&gt; { + dbProperties.put(p, jobStart.properties().getProperty(p)); + }); + + /**** + ** Azure Databricks makes available a dbutils mount point to list aliased paths to cloud + ** storage. However, that dbutils object is not available inside a spark listener. We must + ** access it via reflection. + **/ + try { + Optional&lt;DbfsUtils&gt; dbfsUtils = getDbfsUtils(); + if (!dbfsUtils.isPresent()) { + dbProperties.put("mountPoints", new ArrayList&lt;DatabricksMountpoint&gt;()); + } else { + dbProperties.put("mountPoints", getDatabricksMountpoints(dbfsUtils.get())); + } + + } catch (Exception e) { + log.warn("Failed to load dbutils in OpenLineageListener:", e); + dbProperties.put("mountPoints", new ArrayList&lt;DatabricksMountpoint&gt;()); + } + return dbProperties; + } + + // Starting in Databricks Runtime 11, there is a new constructor for DbFsUtils + // If running on an older version, the constructor has no parameters. + // If running on DBR 11 or above, you need to specify whether you allow mount operations (true or + // false) + private static Optional&lt;DbfsUtils&gt; getDbfsUtils() + throws ClassNotFoundException, InstantiationException, IllegalAccessException, + IllegalArgumentException, InvocationTargetException { + Class dbutilsClass = Class.forName("com.databricks.dbutils_v1.impl.DbfsUtilsImpl"); + Constructor[] dbutilsConstructors = dbutilsClass.getDeclaredConstructors(); + if (dbutilsConstructors.length == 0) { + log.warn( + "Failed to load dbutils in OpenLineageListener as there were no declared constructors"); + return Optional.empty(); + } + Constructor firstConstructor = dbutilsConstructors[0]; + Parameter[] constructorParams = firstConstructor.getParameters(); + if (constructorParams.length == 0) { + log.debug("DbUtils constructor had no parameters"); + return Optional.of((DbfsUtils) firstConstructor.newInstance()); + } else if (constructorParams.length == 1 + &amp;&amp; constructorParams[0].getName().equals("allowMountOperations")) { + log.debug("DbUtils constructor had one parameter named allowMountOperations"); + return Optional.of((DbfsUtils) firstConstructor.newInstance(true)); + } else { + log.warn( + "dbutils had {} constructors and the first constructor had {} params", + dbutilsConstructors.length, + constructorParams.length); + return Optional.empty(); + } + } + + private static List&lt;DatabricksMountpoint&gt; getDatabricksMountpoints(DbfsUtils dbutils) { + List&lt;DatabricksMountpoint&gt; mountpoints = new ArrayList&lt;&gt;(); + List&lt;MountInfo&gt; mountsList = JavaConversions.seqAsJavaList(dbutils.mounts()); + for (MountInfo mount : mountsList) { + mountpoints.add(new DatabricksMountpoint(mount.mountPoint(), mount.source())); + } + return mountpoints; + } +} + +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Rodrigo Maia + (rodrigo.maia@manta.io) +
+
2023-10-30 04:46:16
 +
+

*Thread Reply:* Any ideas on how could i test it?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ankit jain + (ankit.goods10@gmail.com) +
+
2023-10-17 22:57:03
 +
+

Hello All, I am completely new for Openlineage, I have to setup the lab to conduct POC on various aspects like Lineage, metadata management , etc. As per openlineage site, i tried downloading Ubuntu, docker and binary files for Marquez. But I am lost somewhere and unable to configure whole setup. Can someone please assist in steps to start from scratch so that i can delve into the Openlineage capabilities. Many thanks

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-10-18 01:32:01
 +
+

*Thread Reply:* hey, did you try to follow one of these guides? +https://openlineage.io/docs/guides/about

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-10-18 09:14:08
 +
+

*Thread Reply:* Which guide were you using, and what errors/issues are you encountering?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ankit jain + (ankit.goods10@gmail.com) +
+
2023-10-21 15:43:14
 +
+

*Thread Reply:* Thanks Jakub for the response.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ankit jain + (ankit.goods10@gmail.com) +
+
2023-10-21 15:45:42
 +
+

*Thread Reply:* In docker, marquez-api image is not running and exiting with the exit code 127.

+ +
+
image.png
+ + + + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-10-22 09:34:53
 +
+

*Thread Reply:* @ankit jain thanks. I don't recognize 127, but 9 times out of 10 if the API or DB container fails the reason is a port conflict. Have you checked if port 5000 is available?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-10-22 09:54:10
 +
+

*Thread Reply:* could you please check what’s the output of +git config --get core.autocrlf +or +git config --global --get core.autocrlf +?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ankit jain + (ankit.goods10@gmail.com) +
+
2023-10-24 08:09:14
 +
+

*Thread Reply:* @Michael Robinson thanks , I checked the port 5000 is not available. +I tried deleting docker images and recreating them, but still the same issue persist stating +/Usr/bin/env bash/r not found. +Gradle build is successful.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ankit jain + (ankit.goods10@gmail.com) +
+
2023-10-24 08:09:54
 +
+

*Thread Reply:* @Jakub Dardziński thanks, first command resulted as true and second command has no response

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-10-24 08:15:57
 +
+

*Thread Reply:* are you running docker and git in Windows or Mac OS before 10.0?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Matthew Paras + (matthewparas2020@u.northwestern.edu) +
+
2023-10-19 15:00:42
 +
+

Hey all - we've been noticing that some events go unreported by openlineage (spark) when the AsyncEventQueue fills up and starts dropping events. Wondering if anyone has experienced this before, and knows why it is happening? We've expanded the event queue capacity and thrown more hardware at the problem but no dice

+ +

Also as a note, the query plans from this job are pretty big - could the listener just be choking up? Happy to open a github issue as well if we suspect that it could be the listener itself having issues

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Anirudh Shrinivason + (anirudh.shrinivason@grabtaxi.com) +
+
2023-10-20 02:57:50
 +
+

*Thread Reply:* Hi, just checking, are you excluding the sparkPlan from the events? Or is it sending the spark plan too

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-10-23 11:59:40
 +
+

*Thread Reply:* yeah - setting spark.openlineage.facets.disabled to [spark_unknown;spark.logicalPlan] should help

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Matthew Paras + (matthewparas2020@u.northwestern.edu) +
+
2023-10-24 17:50:26
 +
+

*Thread Reply:* sorry for the late reply - turns out this job is just whack 😄 we were going in circles trying to figure it out, we end up dropping events without open lineage enabled at all. But good to know that disabling the logical plan should speed us up if we run into this again

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
praveen kanamarlapudi + (kpraveen420@gmail.com) +
+
2023-10-20 18:18:37
 +
+

Hi,

+ +

We are using openlineage spark connector. We have used spark 3.2 and scala 2.12 so far. We have triggered a new job with Spark 3.4 and scala 2.13 and faced below exception.

+ +

java.lang.NoSuchMethodError: 'scala.collection.Seq org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.map(scala.Function1)' + at io.openlineage.spark.agent.lifecycle.OpenLineageRunEventBuilder.lambda$buildInputDatasets$6(OpenLineageRunEventBuilder.java:341) + at java.base/java.util.Optional.map(Optional.java:265) + at io.openlineage.spark.agent.lifecycle.OpenLineageRunEventBuilder.buildInputDatasets(OpenLineageRunEventBuilder.java:339) + at io.openlineage.spark.agent.lifecycle.OpenLineageRunEventBuilder.populateRun(OpenLineageRunEventBuilder.java:295) + at io.openlineage.spark.agent.lifecycle.OpenLineageRunEventBuilder.buildRun(OpenLineageRunEventBuilder.java:279) + at io.openlineage.spark.agent.lifecycle.OpenLineageRunEventBuilder.buildRun(OpenLineageRunEventBuilder.java:222) + at io.openlineage.spark.agent.lifecycle.SparkSQLExecutionContext.start(SparkSQLExecutionContext.java:72) + at io.openlineage.spark.agent.OpenLineageSparkListener.lambda$sparkSQLExecStart$0(OpenLineageSparkListener.java:91)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-10-23 04:56:25
 +
+

*Thread Reply:* Hmy, that is interesting. Did it occur on databricks runtime? Could you give it a try with Scala 2.12? I think we don't test scala 2.13.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
praveen kanamarlapudi + (kpraveen420@gmail.com) +
+
2023-10-23 12:02:13
 +
+

*Thread Reply:* I believe our Scala 2.12 jobs are working fine. It's not databricks runtime. We run Spark on Kube.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-10-24 06:47:14
 +
+

*Thread Reply:* Ok. I think You can raise an issue to support Scala 2.13 for latest Spark versions.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
priya narayana + (n.priya88@gmail.com) +
+
2023-10-26 06:13:40
 +
+

Hi I want to customise the events which comes from Openlineage spark . Can some one give some information

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-10-26 07:45:41
 +
+

*Thread Reply:* Hi @priya narayana, please get familiar with Extending section on our docs: https://github.com/OpenLineage/OpenLineage/tree/main/integration/spark#extending

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
priya narayana + (n.priya88@gmail.com) +
+
2023-10-26 09:53:07
 +
+

*Thread Reply:* Okay thank you. Just checking any other docs or git code which also can help me

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
harsh loomba + (hloomba@upgrade.com) +
+
2023-10-26 13:11:17
 +
+

Hello Team

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
harsh loomba + (hloomba@upgrade.com) +
+
2023-10-26 13:12:38
 +
+

Im upgrading the version from openlineage-airflow==0.24.0 to openlineage-airflow 1.4.1 but im seeing the following error, any help is appreciated

+ +
+
Screenshot 2023-10-26 at 10.11.34 AM.png
+ + + + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
harsh loomba + (hloomba@upgrade.com) +
+
2023-10-26 13:14:02
 +
+

*Thread Reply:* @Jakub Dardziński any thoughts?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-10-26 13:14:24
 +
+

*Thread Reply:* what version of Airflow are you using?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
harsh loomba + (hloomba@upgrade.com) +
+
2023-10-26 13:14:52
 +
+

*Thread Reply:* 2.6.3 that satisfies the requirement

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-10-26 13:16:38
 +
+

*Thread Reply:* is it possible you have some custom operator?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
harsh loomba + (hloomba@upgrade.com) +
+
2023-10-26 13:17:15
 +
+

*Thread Reply:* i think its the base operator causing the issue

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
harsh loomba + (hloomba@upgrade.com) +
+
2023-10-26 13:17:36
 +
+

*Thread Reply:* so no i believe

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-10-26 13:18:43
 +
+

*Thread Reply:* BaseOperator is parent class for any other operators, it defines how to do deepcopy

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
harsh loomba + (hloomba@upgrade.com) +
+
2023-10-26 13:19:11
 +
+

*Thread Reply:* yeah so its controlled by Airflow itself, I didnt customize it

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-10-26 13:19:49
 +
+

*Thread Reply:* uhm, maybe it's possible you could share dag code? you may hide sensitive data

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
harsh loomba + (hloomba@upgrade.com) +
+
2023-10-26 13:21:23
 +
+

*Thread Reply:* let me try with lower versions of openlineage, what's say

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
harsh loomba + (hloomba@upgrade.com) +
+
2023-10-26 13:21:39
 +
+

*Thread Reply:* its a big jump from 0.24.0 to 1.4.1

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
harsh loomba + (hloomba@upgrade.com) +
+
2023-10-26 13:22:25
 +
+

*Thread Reply:* but i will help here to investigate this issue

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-10-26 13:24:03
 +
+

*Thread Reply:* for me it seems that within dag or task you're defining some object that is not easy to copy

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
harsh loomba + (hloomba@upgrade.com) +
+
2023-10-26 13:26:05
 +
+

*Thread Reply:* possible, but with 0.24.0 that issue is not occurring, so worry is that the version upgrade could potentially break things

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-10-26 13:39:34
 +
+

*Thread Reply:* 0.24.0 is not that old 🤔

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
harsh loomba + (hloomba@upgrade.com) +
+
2023-10-26 13:45:07
 +
+

*Thread Reply:* i see the issue with 0.24.0 I see it as warning +[airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File "/usr/lib64/python3.8/threading.py", line 932, in _bootstrap_inner +[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - self.run() +[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File "/usr/lib64/python3.8/threading.py", line 870, in run +[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - self._target(**self._args, ****self._kwargs) +[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File "/home/upgrade/.local/lib/python3.8/site-packages/openlineage/airflow/listener.py", line 89, in on_running +[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - task_instance_copy = copy.deepcopy(task_instance) +[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File "/usr/lib64/python3.8/copy.py", line 172, in deepcopy +[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - y = _reconstruct(x, memo, **rv) +[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File "/usr/lib64/python3.8/copy.py", line 270, in _reconstruct +[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - state = deepcopy(state, memo) +[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File "/usr/lib64/python3.8/copy.py", line 146, in deepcopy +[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - y = copier(x, memo) +[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File "/usr/lib64/python3.8/copy.py", line 230, in _deepcopy_dict +[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - y[deepcopy(key, memo)] = deepcopy(value, memo) +[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File "/usr/lib64/python3.8/copy.py", line 172, in deepcopy +[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - y = _reconstruct(x, memo, **rv) +[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File "/usr/lib64/python3.8/copy.py", line 270, in _reconstruct +[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - state = deepcopy(state, memo) +[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File "/usr/lib64/python3.8/copy.py", line 146, in deepcopy +[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - y = copier(x, memo) +[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File "/usr/lib64/python3.8/copy.py", line 230, in _deepcopy_dict +[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - y[deepcopy(key, memo)] = deepcopy(value, memo) +[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File "/usr/lib64/python3.8/copy.py", line 153, in deepcopy +[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - y = copier(memo) +[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File "/home/upgrade/.local/lib/python3.8/site-packages/airflow/models/dag.py", line 2162, in __deepcopy__ +[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - setattr(result, k, copy.deepcopy(v, memo)) +[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File "/usr/lib64/python3.8/copy.py", line 146, in deepcopy +[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - y = copier(x, memo) +[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File "/usr/lib64/python3.8/copy.py", line 230, in _deepcopy_dict +[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - y[deepcopy(key, memo)] = deepcopy(value, memo) +[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File "/usr/lib64/python3.8/copy.py", line 153, in deepcopy +[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - y = copier(memo) +[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File "/home/upgrade/.local/lib/python3.8/site-packages/airflow/models/baseoperator.py", line 1224, in __deepcopy__ +[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - setattr(result, k, copy.deepcopy(v, memo)) +[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File "/usr/lib64/python3.8/copy.py", line 172, in deepcopy +[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - y = _reconstruct(x, memo, **rv) +[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File "/usr/lib64/python3.8/copy.py", line 270, in _reconstruct +[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - state = deepcopy(state, memo) +[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File "/usr/lib64/python3.8/copy.py", line 146, in deepcopy +[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - y = copier(x, memo) +[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File "/usr/lib64/python3.8/copy.py", line 230, in _deepcopy_dict +[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - y[deepcopy(key, memo)] = deepcopy(value, memo) +[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File "/usr/lib64/python3.8/copy.py", line 146, in deepcopy +[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - y = copier(x, memo) +[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File "/usr/lib64/python3.8/copy.py", line 230, in _deepcopy_dict +[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - y[deepcopy(key, memo)] = deepcopy(value, memo) +[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File "/usr/lib64/python3.8/copy.py", line 153, in deepcopy +[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - y = copier(memo) +[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File "/home/upgrade/.local/lib/python3.8/site-packages/airflow/models/baseoperator.py", line 1224, in __deepcopy__ +[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - setattr(result, k, copy.deepcopy(v, memo)) +[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File "/usr/lib64/python3.8/copy.py", line 146, in deepcopy +[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - y = copier(x, memo) +[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File "/usr/lib64/python3.8/copy.py", line 230, in _deepcopy_dict +[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - y[deepcopy(key, memo)] = deepcopy(value, memo) +[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File "/usr/lib64/python3.8/copy.py", line 161, in deepcopy +[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - rv = reductor(4) +[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - TypeError: cannot pickle 'module' object +but with 1.4.1 its stopped processing any further and threw error

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
harsh loomba + (hloomba@upgrade.com) +
+
2023-10-26 14:18:08
 +
+

*Thread Reply:* I see the difference of calling in these 2 versions, current versions checks if Airflow is >2.6 then directly runs on_running but earlier version was running on separate thread. IS this what's raising this exception?

+ +
+
Screenshot 2023-10-26 at 11.16.47 AM.png
+ + + + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
harsh loomba + (hloomba@upgrade.com) +
+
2023-10-26 14:24:49
 +
+

*Thread Reply:* this is the issue - https://github.com/OpenLineage/OpenLineage/blob/c343835c1664eda94d5c315897ae6702854c81bd/integration/airflow/openlineage/airflow/listener.py#L89 while copying the task

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/c343835c1664eda94d5c315897ae6702854c81bd/integration/airflow/openlineage/airflow/listener.py | listener.py>
+
+ <pre><code> task_instance_copy = copy.deepcopy(task_instance) +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
harsh loomba + (hloomba@upgrade.com) +
+
2023-10-26 14:25:21
 +
+

*Thread Reply:* since we are directly running if version>2.6.0 therefore its throwing error in main processing

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
harsh loomba + (hloomba@upgrade.com) +
+
2023-10-26 14:28:02
 +
+

*Thread Reply:* may i know which Airflow version we tested this process?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
harsh loomba + (hloomba@upgrade.com) +
+
2023-10-26 14:28:39
 +
+

*Thread Reply:* im on 2.6.3

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-10-26 14:30:53
 +
+

*Thread Reply:* 2.1.4, 2.2.4, 2.3.4, 2.4.3, 2.5.2, 2.6.1 +usually there are not too many changes between minor versions

+ +

I still believe it might be some code you might improve and probably is also an antipattern in airflow

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
harsh loomba + (hloomba@upgrade.com) +
+
2023-10-26 14:34:26
 +
+

*Thread Reply:* hummm...that's a valid observation but I dont write DAGS, other teams do, so imagine if many people wrote such DAGS I can't ask everyone to change their patterns right? If something is running on current openlineage version with warning that should still be running on upgraded version isn't it?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
harsh loomba + (hloomba@upgrade.com) +
+
2023-10-26 14:38:04
 +
+

*Thread Reply:* however I see ur point

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
harsh loomba + (hloomba@upgrade.com) +
+
2023-10-26 14:49:52
 +
+

*Thread Reply:* So that specific task has 570 line of query and pretty bulky query, let me split into smaller units

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
harsh loomba + (hloomba@upgrade.com) +
+
2023-10-26 14:50:15
 +
+

*Thread Reply:* that should help right? @Jakub Dardziński

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-10-26 14:51:27
 +
+

*Thread Reply:* query length shouldn’t be the issue, rather any python code

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-10-26 14:51:50
 +
+

*Thread Reply:* I get your point too, we might figure out some mechanism to skip irrelevant parts of task instance so that it doesn’t fail then

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
harsh loomba + (hloomba@upgrade.com) +
+
2023-10-26 14:52:12
 +
+

*Thread Reply:* actually its failing on that task itself

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
harsh loomba + (hloomba@upgrade.com) +
+
2023-10-26 14:52:33
 +
+

*Thread Reply:* let me try it will be pretty quick

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
harsh loomba + (hloomba@upgrade.com) +
+
2023-10-26 14:58:58
 +
+

*Thread Reply:* @Jakub Dardziński but ur right we have to fix this at Openlineage side as well. Because ideally Openlineage shouldn't be causing any issue to the main DAG processing

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-10-26 17:51:05
 +
+

*Thread Reply:* it doesn’t break any airflow functionality, execution is wrapped into try/except block, only exception traceback is logged as you can see

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-10-27 05:25:54
 +
+

*Thread Reply:* Can you migrate to Airflow 2.7 and use apache-airflow-providers-openlineage? Ideally we wouldn't make meaningful changes to openlineage-airflow

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
harsh loomba + (hloomba@upgrade.com) +
+
2023-10-27 11:35:44
 +
+

*Thread Reply:* yup thats what im planning to do

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
harsh loomba + (hloomba@upgrade.com) +
+
2023-10-27 13:59:03
 +
+

*Thread Reply:* referencing to https://openlineage.slack.com/archives/C01CK9T7HKR/p1698398754823079?threadts=1698340358.557159&cid=C01CK9T7HKR|this conversation - what it takes to move to openlineage provider package from openlineage-airflow. Im updating Airflow to 2.7.2 but moving off of openlineage-airflow to provider package Im trying to estimate the amount of work it takes, any thoughts? reading changelogs I dont think its too much of a change but please share your thoughts and if somewhere its drafted please do share that as well

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-10-30 08:21:10
 +
+

*Thread Reply:* Generally not much - I would maybe think of a operator coverage. For example, for BigQuery old openlineage-airflow supports BigQueryExecuteQueryOperator. However, new apache-airflow-providers-openlineage supports BigQueryInsertJobOperator - because it's intended replacement for BigQueryExecuteQueryOperator and Airflow community does not want to accept contributions to deprecated operators.

+ + + +
+ 🙏 harsh loomba +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
harsh loomba + (hloomba@upgrade.com) +
+
2023-10-31 15:00:38
 +
+

*Thread Reply:* one question if someone is around - when im keeping both openlineage-airflow and apache-airflow-providers-openlineage in my requirement file, i see the following error - +from openlineage.airflow.extractors import Extractors +ModuleNotFoundError: No module named 'openlineage.airflow' +any thoughts?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Lukenoff + (john@jlukenoff.com) +
+
2023-10-31 15:37:07
 +
+

*Thread Reply:* I would usually do a pip freeze | grep openlineage as a sanity check to validate that the module is actually installed. Not sure how the provider and the module play together though

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
harsh loomba + (hloomba@upgrade.com) +
+
2023-10-31 17:07:41
 +
+

*Thread Reply:* yeah so @John Lukenoff im not getting how i can use the specific extractor when i run my operator. Say for example, I have custom datawarehouseOperator and i want to override getopenlineagefacetsonstart and getopenlineagefacetsoncomplete using the redshift extractor then how would i do that?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Rodrigo Maia + (rodrigo.maia@manta.io) +
+
2023-10-27 05:49:25
 +
+

Spark Integration Logs +Hey There +Are these events skipped because it's not supported or it's configured somewhere? +23/10/27 08:25:58 INFO SparkSQLExecutionContext: OpenLineage received Spark event that is configured to be skipped: SparkListenerSQLExecutionStart +23/10/27 08:25:58 INFO SparkSQLExecutionContext: OpenLineage received Spark event that is configured to be skipped: SparkListenerSQLExecutionEnd

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Hitesh + (splicer9904@gmail.com) +
+
2023-10-27 08:12:32
 +
+

Hi People, actually I want to intercept the OpenLineage spark events right after the job ends and before they are emitted, so that I can add some extra information to the events or remove some information that I don't want. +Is there any way of doing this? Can someone please help me

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-10-30 09:03:57
 +
+

*Thread Reply:* It general, I think this kind of use case is probably best served by facets, but what do you think @Paweł Leszczyński?

+
+
openlineage.io
+ + + +
Understanding and Using Facets | OpenLineage
+
+ Adapted from the OpenLineage spec. +
+ + + + + + +
Original URL: https://openlineage.io/docs/guides/facets
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kavitha + (kkandaswamy@cardinalcommerce.com) +
+
2023-10-27 17:01:12
 +
+

Hello, has anyone run into similar error as posted in this github open issues[https://github.com/MarquezProject/marquez/issues/2468] while setting up marquez on an EC2 Instance, would appreciate any help to get past the errors

+
+ + + + +
#2468 Connection error PostgreSQL and Marquez
+
+ <p>Hi all!</p> + +<p>I really appreciate this repository and would love to use it for its data lineage aspect! <br /> +Last few days, I've been trying to get the <code>marquez api</code> Docker image up and running but whatever I try, it does not seem to work.</p> + +<p>It returns the following errors in Docker Desktop:</p> + +<p><code>2023-04-03 18:07:04 marquez-db | 2023-04-03 23:07:04.726 GMT [35] FATAL: password authentication failed for user "marquez" 2023-04-03 18:07:04 marquez-db | 2023-04-03 23:07:04.726 GMT [35] DETAIL: Role "marquez" does not exist.</code></p> + +<p>and</p> + +<p><code>ERROR [2023-04-03 22:44:31,217] org.apache.tomcat.jdbc.pool.ConnectionPool: Unable to create initial connections of pool. 2023-04-03 17:44:31 ! org.postgresql.util.PSQLException: FATAL: password authentication failed for user "marquez"</code></p> + +<p>All the steps in the README.md file were used to configure the PostgreSQL database for Marquez and to set the correct environment variables. <br /> +During my trouble shooting practices, I found a relatively similar issue on StackOverflow which refers to the danger in using the same ports for Postgres and Marquez but this did not help me yet in solving the issue (<a href="https://stackoverflow.com/questions/62115827/org-postgresql-util-psqlexception-fatal-password-authentication-failed-for-use">https://stackoverflow.com/questions/62115827/org-postgresql-util-psqlexception-fatal-password-authentication-failed-for-use</a>).</p> + +<p>Could you please help me out?</p> + +<p>Kind regards,</p> + +<p>Tom</p> +
+ +
+
Comments
+ 5 +
+ + + + + + + +
+ + <a href="https://github.com/MarquezProject/marquez">MarquezProject/marquez</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2023-10-27 17:04:30
 +
+

*Thread Reply:* Hmm, have you looked over our Running on AWS docs?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2023-10-27 17:06:08
 +
+

*Thread Reply:* More specifically, the AWS RDS section. How are you deploying Marquez on Ec2?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kavitha + (kkandaswamy@cardinalcommerce.com) +
+
2023-10-27 17:08:05
 +
+

*Thread Reply:* we were primarily referencing this document on git - https://github.com/MarquezProject/marquez

+
+ + + + +
MarquezProject/marquez
+
+ Collect, aggregate, and visualize a data ecosystem's metadata +
+ +
+
Website
+ <https://marquezproject.ai> +
+ +
+
Stars
+ 1450 +
+ + + + + + + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kavitha + (kkandaswamy@cardinalcommerce.com) +
+
2023-10-27 17:09:05
 +
+

*Thread Reply:* leveraged docker and docker-compose

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2023-10-27 17:13:10
 +
+

*Thread Reply:* hmm so you’re running docker-compose up on an Ec2 instance you’ve ssh’d into? (just trying to understand your setup better)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kavitha + (kkandaswamy@cardinalcommerce.com) +
+
2023-10-27 17:13:26
 +
+

*Thread Reply:* yes, thats correct

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2023-10-27 17:16:39
 +
+

*Thread Reply:* I’ve only used docker compose for local dev or integration tests. but, ok you’re probably in the PoC phase. Can you run the docker cmd on you local machine successfully? What OS is stalled on the Ec2 instance?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kavitha + (kkandaswamy@cardinalcommerce.com) +
+
2023-10-27 17:18:00
 +
+

*Thread Reply:* yes, i can run and the OS is Ubuntu 20.04.6 LTS

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kavitha + (kkandaswamy@cardinalcommerce.com) +
+
2023-10-27 17:19:27
 +
+

*Thread Reply:* we initiallly ran into a permission denied error related to postgressql.conf file and we had to update file permissions to 777 and after which we started to see below errors

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kavitha + (kkandaswamy@cardinalcommerce.com) +
+
2023-10-27 17:19:36
 +
+

*Thread Reply:* marquez-db | 2023-10-27 20:35:52.512 GMT [35] FATAL: no pghba.conf entry for host "172.18.0.5", user "marquez", database "marquez", no encryption + marquez-db | 2023-10-27 20:35:52.529 GMT [36] FATAL: no pghba.conf entry for host "172.18.0.5", user "marquez", database "marquez", no encryption

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kavitha + (kkandaswamy@cardinalcommerce.com) +
+
2023-10-27 17:20:12
 +
+

*Thread Reply:* we then manually updated pg_hba.conf file to include host user and db details

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2023-10-27 17:20:42
 +
+

*Thread Reply:* Did you also update the marquez.yml with the db user / password?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kavitha + (kkandaswamy@cardinalcommerce.com) +
+
2023-10-27 17:20:48
 +
+

*Thread Reply:* after which we started to see the errors posted in the github open issues page

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2023-10-27 17:21:33
 +
+

*Thread Reply:* hmm are you using an external database or are you spinning up the entire Marquez stack with docker compose?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kavitha + (kkandaswamy@cardinalcommerce.com) +
+
2023-10-27 17:21:56
 +
+

*Thread Reply:* we are spinning up the entire Marquez stack with docker compose

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kavitha + (kkandaswamy@cardinalcommerce.com) +
+
2023-10-27 17:23:24
 +
+

*Thread Reply:* we did not change anything in the marquez.yml, i think we did not find that file in the github repo that we cloned into our local instance

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2023-10-27 17:26:31
 +
+

*Thread Reply:* It’s important that the init-db.sh script runs, but I don’t think it is

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2023-10-27 17:26:56
 +
+

*Thread Reply:* can you grab all the docker compose logs and share them? it’s hard to debug otherwise

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kavitha + (kkandaswamy@cardinalcommerce.com) +
+
2023-10-27 17:29:59
 +
+

*Thread Reply:*

+ +
+
logs.txt
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2023-10-27 17:33:15
 +
+

*Thread Reply:* I would first suggest to remove the --build flag since you are specifying a version of Marquez to use via --tag

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2023-10-27 17:33:49
 +
+

*Thread Reply:* no the issue per se, but will help clear up some of the logs

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kavitha + (kkandaswamy@cardinalcommerce.com) +
+
2023-10-27 17:35:06
 +
+

*Thread Reply:* for sure thanks. we could get the logs without the --build portion, we tried with that option just once

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kavitha + (kkandaswamy@cardinalcommerce.com) +
+
2023-10-27 17:35:40
 +
+

*Thread Reply:* the errors were the same with/without --build option

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kavitha + (kkandaswamy@cardinalcommerce.com) +
+
2023-10-27 17:36:02
 +
+

*Thread Reply:* marquez-api | ERROR [2023-10-27 21:34:58,019] org.apache.tomcat.jdbc.pool.ConnectionPool: Unable to create initial connections of pool. + marquez-api | ! org.postgresql.util.PSQLException: FATAL: password authentication failed for user "marquez" + marquez-api | ! at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:693) + marquez-api | ! at org.postgresql.core.v3.ConnectionFactoryImpl.tryConnect(ConnectionFactoryImpl.java:203) + marquez-api | ! at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:258) + marquez-api | ! at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:54) + marquez-api | ! at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:253) + marquez-api | ! at org.postgresql.Driver.makeConnection(Driver.java:434) + marquez-api | ! at org.postgresql.Driver.connect(Driver.java:291) + marquez-api | ! at org.apache.tomcat.jdbc.pool.PooledConnection.connectUsingDriver(PooledConnection.java:346) + marquez-api | ! at org.apache.tomcat.jdbc.pool.PooledConnection.connect(PooledConnection.java:227) + marquez-api | ! at org.apache.tomcat.jdbc.pool.ConnectionPool.createConnection(ConnectionPool.java:768) + marquez-api | ! at org.apache.tomcat.jdbc.pool.ConnectionPool.borrowConnection(ConnectionPool.java:696) + marquez-api | ! at org.apache.tomcat.jdbc.pool.ConnectionPool.init(ConnectionPool.java:495) + marquez-api | ! at org.apache.tomcat.jdbc.pool.ConnectionPool.<init>(ConnectionPool.java:153) + marquez-api | ! at org.apache.tomcat.jdbc.pool.DataSourceProxy.pCreatePool(DataSourceProxy.java:118) + marquez-api | ! at org.apache.tomcat.jdbc.pool.DataSourceProxy.createPool(DataSourceProxy.java:107) + marquez-api | ! at org.apache.tomcat.jdbc.pool.DataSourceProxy.getConnection(DataSourceProxy.java:131) + marquez-api | ! at org.flywaydb.core.internal.jdbc.JdbcUtils.openConnection(JdbcUtils.java:48) + marquez-api | ! at org.flywaydb.core.internal.jdbc.JdbcConnectionFactory.<init>(JdbcConnectionFactory.java:75) + marquez-api | ! at org.flywaydb.core.FlywayExecutor.execute(FlywayExecutor.java:147) + marquez-api | ! at org.flywaydb.core.Flyway.info(Flyway.java:190) + marquez-api | ! at marquez.db.DbMigration.hasPendingDbMigrations(DbMigration.java:73) + marquez-api | ! at marquez.db.DbMigration.migrateDbOrError(DbMigration.java:27) + marquez-api | ! at marquez.MarquezApp.run(MarquezApp.java:105) + marquez-api | ! at marquez.MarquezApp.run(MarquezApp.java:48) + marquez-api | ! at io.dropwizard.cli.EnvironmentCommand.run(EnvironmentCommand.java:67) + marquez-api | ! at io.dropwizard.cli.ConfiguredCommand.run(ConfiguredCommand.java:98) + marquez-api | ! at io.dropwizard.cli.Cli.run(Cli.java:78) + marquez-api | ! at io.dropwizard.Application.run(Application.java:94) + marquez-api | ! at marquez.MarquezApp.main(MarquezApp.java:60) + marquez-api | INFO [2023-10-27 21:34:58,024] marquez.MarquezApp: Stopping app...

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2023-10-27 17:38:52
 +
+

*Thread Reply:* debugging docker issues like this is so difficult

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2023-10-27 17:40:44
 +
+

*Thread Reply:* it could be a number of things, but you are connected to the database it’s just that the marquez user hasn’t been created

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2023-10-27 17:41:59
 +
+

*Thread Reply:* the /init-db.sh is what manages user creation

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2023-10-27 17:42:17
 +
+

*Thread Reply:* so it’s possible that the script isn’t running for whatever reason on your Ec2 instance

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2023-10-27 17:44:20
 +
+

*Thread Reply:* do you have other services running on that Ec2 instance? Like, other than Marquez

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Willy Lulciuc + (willy@datakin.com) +
+
2023-10-27 17:44:52
 +
+

*Thread Reply:* is there a postgres process running outside of docker?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kavitha + (kkandaswamy@cardinalcommerce.com) +
+
2023-10-27 20:34:50
 +
+

*Thread Reply:* no other services except marquez on this EC2 instance

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kavitha + (kkandaswamy@cardinalcommerce.com) +
+
2023-10-27 20:35:49
 +
+

*Thread Reply:* this was a new Ec2 instance that was spun up to install and use marquez

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kavitha + (kkandaswamy@cardinalcommerce.com) +
+
2023-10-27 20:36:09
 +
+

*Thread Reply:* n we can confirm that no postgres process runs outside of docker

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason Yip + (jasonyip@gmail.com) +
+
2023-10-29 03:06:28
 +
+

I realize in Spark 3.4+, some job ids don't have a start event. What part of the code is responsible for triggering the START and COMPLETE event

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-10-30 09:59:53
 +
+

*Thread Reply:* hi @Jason Yip could you provide an example of such a job?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason Yip + (jasonyip@gmail.com) +
+
2023-10-30 16:51:55
 +
+

*Thread Reply:* @Paweł Leszczyński same old:

+ +

delete the old table if needed

+ +

_ = spark.sql('DROP TABLE IF EXISTS transactions')

+ +

expected structure of the file

+ +

transactionsschema = StructType([ + StructField('householdid', IntegerType()), + StructField('basketid', LongType()), + StructField('day', IntegerType()), + StructField('productid', IntegerType()), + StructField('quantity', IntegerType()), + StructField('salesamount', FloatType()), + StructField('storeid', IntegerType()), + StructField('discountamount', FloatType()), + StructField('transactiontime', IntegerType()), + StructField('weekno', IntegerType()), + StructField('coupondiscount', FloatType()), + StructField('coupondiscountmatch', FloatType()) + ])

+ +

read data to dataframe

+ +

df = (spark + .read + .csv( + adlsRootPath + '/examples/data/csv/completejourney/transactiondata.csv', + header=True, + schema=transactionsschema))

+ +

df.write\ + .format('delta')\ + .mode('overwrite')\ + .option('overwriteSchema', 'true')\ + .option('path', adlsRootPath + '/examples/data/csv/completejourney/silver/transactions')\ + .saveAsTable('transactions')

+ +

df.count()

+ +

# create table object to make delta lake queryable

+ +

_ = spark.sql(f'''

+ +

CREATE TABLE transactions

+ +

USING DELTA

+ +

LOCATION '{adlsRootPath}/examples/data/csv/completejourney/silver/transactions'

+ +

''')

+ +

show data

+ +

display( + spark.table('transactions') + )

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Lukenoff + (john@jlukenoff.com) +
+
2023-10-30 18:51:43
 +
+

👋 Hi team, cross-posting from the Marquez Channel in case anyone here has a better idea of the spec

+ +

> For most of our lineage extractors in airflow, we are using the rust sql parser from openlineage-sql to extract table lineage via sql statements. When errors occur we are adding an extractionError run facet similar to what is being done here. I’m finding in the case that multiple statements were extracted but one failed to parse while many others were successful, the lineage for these runs doesn’t appear as expected in Marquez. Is there any logic around the extractionError run facet that could be causing this? It seems reasonable to assume that we might take this to mean the entire run event is invalid if we have any extraction errors. +> +> I would still expect to see the other lineage we sent for the run but am instead just seeing the extractionError in the marquez UI, in the database, runs with an extractionError facet don’t seem to make it to the job_versions_io_mapping table

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/integration/airflow/openlineage/airflow/extractors/sql_extractor.py | sql_extractor.py>
+
+ <pre><code> if sql_meta.errors: + run_facets['extractionError'] = ExtractionErrorRunFacet( + totalTasks=len(self.operator.sql) if isinstance(self.operator.sql, list) else 1, + failedTasks=len(sql_meta.errors), + errors=[ + ExtractionError( + errorMessage=error.message, + stackTrace=None, + task=error.origin_statement, + taskNumber=error.index + ) + for error + in sql_meta.errors + ] + ) +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-10-31 06:34:05
 +
+

*Thread Reply:* Can you show the actual event? Should be in the events tab in Marquez

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kavitha + (kkandaswamy@cardinalcommerce.com) +
+
2023-10-31 11:59:07
 +
+

*Thread Reply:* @John Lukenoff, would you mind posting the link to Marquez teams slack channel?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Lukenoff + (john@jlukenoff.com) +
+
2023-10-31 12:15:37
 +
+

*Thread Reply:* yep here is the link: https://marquezproject.slack.com/archives/C01E8MQGJP7/p1698702140709439

+ +

This is the full event, sanitized of internal info: +{ + "job": { + "name": "some_dag.some_task", + "facets": {}, + "namespace": "default" + }, + "run": { + "runId": "a9565df2-f1a1-3ee3-b202-7626f8c4b92d", + "facets": { + "extractionError": { + "errors": [ + { + "task": "ALTER SESSION UNSET QUERY_TAG;", + "_producer": "<https://github.com/OpenLineage/OpenLineage/tree/0.24.0/client/python>", + "_schemaURL": "<https://raw.githubusercontent.com/OpenLineage/OpenLineage/main/spec/OpenLineage.json#/definitions/BaseFacet>", + "taskNumber": 0, + "errorMessage": "Expected one of TABLE or INDEX, found: SESSION" + } + ], + "_producer": "<https://github.com/OpenLineage/OpenLineage/tree/0.24.0/client/python>", + "_schemaURL": "<https://raw.githubusercontent.com/OpenLineage/OpenLineage/main/spec/OpenLineage.json#/definitions/ExtractionErrorRunFacet>", + "totalTasks": 1, + "failedTasks": 1 + } + } + }, + "inputs": [ + { + "name": "foo.bar", + "facets": {}, + "namespace": "snowflake" + }, + { + "name": "fizz.buzz", + "facets": {}, + "namespace": "snowflake" + } + ], + "outputs": [ + { "name": "foo1.bar2", "facets": {}, "namespace": "snowflake" }, + { + "name": "fizz1.buzz2", + "facets": {}, + "namespace": "snowflake" + } + ], + "producer": "<https://github.com/MyCompany/repo/blob/next-master/company/data/pipelines/airflow_utils/openlineage_utils/client.py>", + "eventTime": "2023-10-30T02:46:13.367274Z", + "eventType": "COMPLETE" +}

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kavitha + (kkandaswamy@cardinalcommerce.com) +
+
2023-10-31 12:43:07
 +
+

*Thread Reply:* thank you!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kavitha + (kkandaswamy@cardinalcommerce.com) +
+
2023-10-31 13:14:29
 +
+

*Thread Reply:* @John Lukenoff, sorry to trouble again, is the slack channel still active? for whatever reason i cant get to this workspace

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Lukenoff + (john@jlukenoff.com) +
+
2023-10-31 13:15:26
 +
+

*Thread Reply:* yep it’s still active, maybe you need to join the workspace first? https://join.slack.com/t/marquezproject/shared_invite/zt-266fdhg9g-TE7e0p~EHK50GJMMqNH4tg

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Kavitha + (kkandaswamy@cardinalcommerce.com) +
+
2023-10-31 13:25:51
 +
+

*Thread Reply:* that was a good call. the link you just shared worked! thank you!

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-10-31 13:27:55
 +
+

*Thread Reply:* yeah from OL perspective this looks good - the inputs and outputs are there, the extraction error facet looks like it should

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-10-31 13:28:05
 +
+

*Thread Reply:* must be some Marquez hiccup 🙂

+ + + +
+ 👍 John Lukenoff +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Lukenoff + (john@jlukenoff.com) +
+
2023-10-31 13:28:45
 +
+

*Thread Reply:* Makes sense, I’ll tail my marquez logs today to see if I can find anything

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Lukenoff + (john@jlukenoff.com) +
+
2023-11-01 19:37:06
 +
+

*Thread Reply:* Somehow this started working after we switched from our beta to prod infrastructure. I suspect something was failing due to constraints on the size of our db and the load of poor quality data it was under after months of testing against it

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-11-01 11:34:43
 +
+

@channel +I’m opening a vote to release OpenLineage 1.5.0, including: +• support for Cassandra Connectors lineage in the Flink integration +• support for Databricks Runtime 13.3 in the Spark integration +• support for rdd and toDF operations from the Spark Scala API in Spark +• lowered requirements for attrs and requests packages in the Airflow integration +• lazy rendering of yaml configs in the dbt integration +• bug fixes, tests, infra fixes, doc changes, and more. +Three +1s from committers will authorize an immediate release.

+ + + +
+ ➕ Jakub Dardziński, William Angel, Abdallah, Willy Lulciuc, Paweł Leszczyński, Julien Le Dem +
+ +
+ 👍 Jason Yip +
+ +
+ 🚀 Luca Soato, tati +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-11-02 05:11:58
 +
+

*Thread Reply:* Thanks, all. The release is authorized and will be initiated within 2 business days.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-11-01 13:29:09
 +
+

@channel +The October 2023 issue of OpenLineage News is available now! to get in directly in your inbox each month.

+
+
apache.us14.list-manage.com
+ + + +
OpenLineage Project
+
+ OpenLineage Project Email Forms +
+ + + + + + +
Original URL: https://openlineage.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&amp;id=123767f606&amp;e=ef0563a7f8
+ + + +
+ + + +
+ 👍 Mars Lan, harsh loomba +
+ +
+ 🎉 tati +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
John Lukenoff + (john@jlukenoff.com) +
+
2023-11-01 19:40:39
 +
+

Hi team 👋 , we’re finding that for our Spark jobs we are almost always getting some junk characters in our dataset names. We’ve pushed the regex filter to its limits and would like to extend the logic of deriving the dataset name in openlineage-spark (currently on 1.4.1). I seem to recall hearing we could do this by implementing our own LogicalPlanVisitor or something along those lines? Is that still the recommended approach and if so would this be possible to implement in Scala vs. Java (scala noob here 🙂)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-11-02 03:34:15
 +
+

*Thread Reply:* Hi John, we're always happy to help with the contribution.

+ +

One of the possible solutions to this would be to do that just in openlineage-java client: +• introduce config entry like normalizeDatasetNameToAscii : enabled/disabled +• modify DatasetIdentifier class to contain static member boolean normalizeDatasetNameToAscii and normalize dataset name according to this setting +• additionally, you would need to add config entry in io.openlineage.client.OpenLineageYaml and make sure both loadOpenLineageYaml methods set DatasetIdentifier.normalizeDatasetNameToAscii based on the config +• document this in the doc +So, no Scala nor custom logical plan visitors required.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-11-02 03:34:47
 +
+

*Thread Reply:* https://github.com/OpenLineage/OpenLineage/blob/main/client/java/src/main/java/io/openlineage/client/utils/DatasetIdentifier.java

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/client/java/src/main/java/io/openlineage/client/utils/DatasetIdentifier.java | DatasetIdentifier.java>
+
+ <pre><code>/** +/** Copyright 2018-2023 contributors to the OpenLineage project +/** SPDX-License-Identifier: Apache-2.0 +**/ +package io.openlineage.client.utils; + +import java.util.LinkedList; +import java.util.List; +import lombok.Value; + +@Value +public class DatasetIdentifier { + String name; + String namespace; + List&lt;Symlink&gt; symlinks; + + public enum SymlinkType { + TABLE + }; + + public DatasetIdentifier(String name, String namespace) { + this.name = name; + this.namespace = namespace; + this.symlinks = new LinkedList&lt;&gt;(); + } + + public DatasetIdentifier withSymlink(String name, String namespace, SymlinkType type) { + symlinks.add(new Symlink(name, namespace, type)); + return this; + } + + public DatasetIdentifier withSymlink(Symlink symlink) { + symlinks.add(symlink); + return this; + } + + @Value + public static class Symlink { + String name; + String namespace; + SymlinkType type; + } +} + +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+ 🙌 John Lukenoff +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mike Fang + (fangmik@amazon.com) +
+
2023-11-01 20:30:38
 +
+

I am looking to send OpenLineage events to an AWS API Gateway endpoint from an AWS MWAA instance. The problem is that all requests to AWS services need to be signed with SigV4, and using API Gateway with IAM authentication would require requests to API Gateway be signed with SigV4. Would the best way to do so be to just modify the python client HTTP transport to include a new config option for signing emitted OpenLineage events with SigV4? Are there any alternatives?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-11-02 02:41:50
 +
+

*Thread Reply:* there’s actually an issue for that: +https://github.com/OpenLineage/OpenLineage/issues/2189

+ +

but the way to do this is imho to create new custom transport (it might inherit from HTTP transport) and register it in transport factory

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mike Fang + (fangmik@amazon.com) +
+
2023-11-02 13:05:05
 +
+

*Thread Reply:* I am thinking of just modifying the HTTP transport and using requests.auth.AuthBase to create different auth methods instead of a TokenProvider class

+ +

Classes which subclass requests.auth.AuthBase can also just directly be given to the requests call in the auth parameter

+ + + +
+ 👍 Jakub Dardziński +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-11-02 14:40:24
 +
+

*Thread Reply:* would you like to contribute? 🙂

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mike Fang + (fangmik@amazon.com) +
+
2023-11-02 14:43:05
 +
+

*Thread Reply:* I was about to contribute, but I actually just realized that there is an existing way to provide a custom transport that would solve form y use case. My only question is how do I register this custom transport in my MWAA environment? Can I provide the custom transport as an Airflow plugin and then specify the class in the Openlineage.yml config? Will it automatically pick it up?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-11-02 15:45:56
 +
+

*Thread Reply:* although I did not test this in MWAA but locally only: I’ve created Airflow plugin that in __init__.py has defined (or imported) following code: +```from openlineage.client.transport import register_transport, Transport, Config

+ +

@register_transport +class FakeTransport(Transport): + kind = "fake" + config = Config

+ +
def __init__(self, config: Config) -> None:
+    print(config)
+
+def emit(self, event) -> None:
+    print(event)```
+
+ +

setting AIRFLOW__OPENLINEAGE__TRANSPORT='{"type": "fake"}' does take effect and I can see output in Airflow logs

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-11-02 15:47:45
 +
+

*Thread Reply:* in setup.py it’s: +..., + entry_points={ + 'airflow.plugins': [ + 'custom_transport = custom_transport:CustomTransportPlugin', + ], + }, + install_requires=["openlineage-python"] +)

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Mike Fang + (fangmik@amazon.com) +
+
2023-11-03 12:52:55
 +
+

*Thread Reply:* ok great thanks for following up on this, super helpful

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-11-02 12:00:00
 +
+

@channel +We released OpenLineage 1.5.0, including: +• support for Cassandra Connectors lineage in the Flink integration by @Peter Huang +• support for Databricks Runtime 13.3 in the Spark integration by @Paweł Leszczyński +• support for rdd and toDF operations from the Spark Scala API in Spark by @Paweł Leszczyński +• lowered requirements for attrs and requests packages in the Airflow integration by @Jakub Dardziński +• lazy rendering of yaml configs in the dbt integration by @Jakub Dardziński +• bug fixes, tests, infra fixes, doc changes, and more. +Thanks to all the contributors, including new contributor @Sophie LY! +Release: https://github.com/OpenLineage/OpenLineage/releases/tag/1.5.0 +Changelog: https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md +Commit history: https://github.com/OpenLineage/OpenLineage/compare/1.4.1...1.5.0 +Maven: https://oss.sonatype.org/#nexus-search;quick~openlineage +PyPI: https://pypi.org/project/openlineage-python/

+ + + +
+ 👍 Jason Yip, Sophie LY, Tristan GUEZENNEC -CROIX-, Mars Lan, Sangeeta Mishra +
+ +
+ 🚀 tati +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason Yip + (jasonyip@gmail.com) +
+
2023-11-02 14:49:18
 +
+

@Paweł Leszczyński I tested 1.5.0, it works great now, but the environment facets is gone in START... which I very much want it.. any thoughts?

+ +
+
data (10).json
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason Yip + (jasonyip@gmail.com) +
+
2023-11-03 04:18:11
 +
+

actually, it shows up in one of the RUNNING now... behavior is consistent between 11.3 and 13.3, thanks for fixing this issue

+ + + +
+ 👍 Paweł Leszczyński +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason Yip + (jasonyip@gmail.com) +
+
2023-11-04 15:44:22
 +
+

*Thread Reply:* @Paweł Leszczyński looks like I need to bring bad news.. 13.3 is fixed for specific scenarios, but 11.3 is still reading output as dbfs.. there are scenarios that it's not producing input and output like:

+ +

create table table using delta as +location 'abfss://....' +Select ** from parquet.`abfss://....'

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason Yip + (jasonyip@gmail.com) +
+
2023-11-04 15:44:31
 +
+

*Thread Reply:* Will test more and ope issues

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Rodrigo Maia + (rodrigo.maia@manta.io) +
+
2023-11-06 05:34:33
 +
+

*Thread Reply:* @Jason Yiphow did you manage the get the environment attribute. it's not showing up to me at all. I've tried databricks abut also tried a local instance of spark.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason Yip + (jasonyip@gmail.com) +
+
2023-11-07 18:32:02
 +
+

*Thread Reply:* @Rodrigo Maia its showing up in one of the RUNNING events, not in the START event anymore

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Rodrigo Maia + (rodrigo.maia@manta.io) +
+
2023-11-08 03:04:32
 +
+

*Thread Reply:* I never had a running event 🫠 Am I filtering something?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason Yip + (jasonyip@gmail.com) +
+
2023-11-08 13:03:26
 +
+

*Thread Reply:* Umm.. ok show me your code, will try on my end

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason Yip + (jasonyip@gmail.com) +
+
2023-11-08 14:26:06
 +
+

*Thread Reply:* @Paweł Leszczyński @Rodrigo Maia actually if you are using UC-enabled cluster, you won't get any RUNNING events

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-11-03 12:00:07
 +
+

@channel +This month’s TSC meeting (open to all) is next Thursday the 9th at 10am PT. On the agenda: +• announcements +• recent releases +• recent additions to the Flink integration by @Peter Huang +• recent additions to the Spark integration by @Paweł Leszczyński +• updates on proposals by @Julien Le Dem +• discussion topics +• open discussion +More info and the meeting link can be found on the website. All are welcome! Do you have a discussion topic, use case or integration you’d like to demo? DM me to be added to the agenda.

+
+
openlineage.io
+ + + +
TSC Meetings | OpenLineage
+
+ The OpenLineage Technical Steering Committee meets monthly, and is open to all. +
+ + + + + + +
Original URL: https://openlineage.io/meetings/
+ + + +
+ + + +
+ 👍 harsh loomba +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
priya narayana + (n.priya88@gmail.com) +
+
2023-11-04 07:08:10
 +
+

Hi Team , we are trying to customize the events by writing custom lineage listener extending OpenLineageSparkListener, but would need some direction how to capture the events

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-11-04 07:11:46
 +
+

*Thread Reply:* https://openlineage.slack.com/archives/C01CK9T7HKR/p1698315220142929 +Do you need some more guidance than that?

+
+ + +
+ + + } + + priya narayana + (https://openlineage.slack.com/team/U062Q95A1FG) +
+ + + +
+
+ <p>Hi I want to customise the events which comes from Openlineage spark . Can some one give some information</p> +
+ + + + + + +
Original URL: https://openlineage.slack.com/archives/C01CK9T7HKR/p1698315220142929
+ + +
+ + Thread in Slack Conversation +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
priya narayana + (n.priya88@gmail.com) +
+
2023-11-04 07:13:47
 +
+

*Thread Reply:* yes

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-11-04 07:15:21
 +
+

*Thread Reply:* It seems pretty extensively described, what kind of help do you need?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
priya narayana + (n.priya88@gmail.com) +
+
2023-11-04 07:16:13
 +
+

*Thread Reply:* io.openlineage.spark.api.OpenLineageEventHandlerFactory if i use this how will i pass custom listener to my spark submit

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
priya narayana + (n.priya88@gmail.com) +
+
2023-11-04 07:17:25
 +
+

*Thread Reply:* I would like to know how will i customize my events using this . For example: - In "input" Facet i want only symlinks name i am not intereseted in anything else

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
priya narayana + (n.priya88@gmail.com) +
+
2023-11-04 07:17:32
 +
+

*Thread Reply:* can you please provide some guidance

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
priya narayana + (n.priya88@gmail.com) +
+
2023-11-04 07:18:36
 +
+

*Thread Reply:* @Jakub Dardziński this is the doubt i have

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
priya narayana + (n.priya88@gmail.com) +
+
2023-11-04 08:17:25
 +
+

*Thread Reply:* Some one who did spark integration throw some light

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jakub Dardziński + (jakub.dardzinski@getindata.com) +
+
2023-11-04 08:21:22
 +
+

*Thread Reply:* it's weekend for most of us so you probably need to wait until Monday for precise answers

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
David Goss + (david.goss@matillion.com) +
+
2023-11-06 04:03:42
 +
+

👋 I raised a PR https://github.com/OpenLineage/OpenLineage/pull/2223 off the back of some Marquez conversations a while back to try and clarify how names of Snowflake objects should be expressed in OL events. I used Snowflake’s OL view as a guide, but also I appreciate there are other OL producers that involve Snowflake too (Airflow? dbt?). Any feedback on this would be appreciated!

+
+ + + + +
#2223 spec: add clarity to snowflake naming docs
+
+ +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+
+ + + + +
Snowflake-Labs/OpenLineage-AccessHistory-Setup
+
+ +
+ +
+
Stars
+ 11 +
+ +
+
Last updated
+ 3 months ago +
+ + + + + + + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
David Goss + (david.goss@matillion.com) +
+
2023-11-08 10:42:35
 +
+

*Thread Reply:* Thanks for merging this @Maciej Obuchowski!

+ + + +
+ 👍 Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Athitya Kumar + (athityakumar@gmail.com) +
+
2023-11-06 05:22:03
 +
+

Hey team! 👋

+ +

We're trying to use openlineage-flink, and would like provide the openlineage.transport.type=http and configure other transport configs, but we're not able to find sufficient docs (tried this doc) on where/how these configs can be provided.

+ +

For example, in spark, the changes mostly were delegated to the spark-submit command like +spark-submit --conf "spark.extraListeners=io.openlineage.spark.agent.OpenLineageSparkListener" \ + --packages "io.openlineage:openlineage_spark:&lt;spark-openlineage-version&gt;" \ + --conf "spark.openlineage.transport.url=http://{openlineage.client.host}/api/v1/namespaces/spark_integration/" \ + --class com.mycompany.MySparkApp my_application.jar +And the OpenLineageSparkListener has a method to retrieve the provided spark confs as an object in the ArgumentParser. Similarly, looking for some pointers on how the openlineage.transport configs can be provided to OpenLineageFlinkJobListener & how the flink listener parses/uses these configs

+ +

TIA! 😄

+
+
openlineage.io
+ + + +
Apache Flink | OpenLineage
+
+ This integration is considered experimental: only specific workflows and use cases are supported. +
+ + + + + + +
Original URL: https://openlineage.io/docs/integrations/flink
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-11-07 05:56:09
 +
+

*Thread Reply:* similarly to spark config, you can use flink config

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Athitya Kumar + (athityakumar@gmail.com) +
+
2023-11-07 22:36:53
 +
+

*Thread Reply:* @Maciej Obuchowski - Got it. Our use-case is that we're trying to build a wrapper on top of openlineage-flink for productionising for our flink jobs.

+ +

We're trying to have a wrapper class that extends OpenLineageFlinkJobListener class, and overwrites the HTTP transport endpoint/url to a constant value (say, example.com and /api/v1/flink). But we see that the OpenLineageFlinkJobListener constructor is defined as a private constructor - just wanted to check with the team whether it was just a default scope, or intended to be private. If it was just a default scope, can we contribute a PR to make it public, to make it friendly for teams trying to adopt & extend openlineage?

+ +

And also, we wanted to understand better on where we're reading the HTTP transport endpoint/url configs in OpenLineageFlinkJobListener and what'd be the best place to override it to the constant endpoint/url for our use-case

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-11-08 05:55:43
 +
+

*Thread Reply:* We parse flink conf to get that information: https://github.com/OpenLineage/OpenLineage/blob/26494b596e9669d2ada164066a73c44e04[…]ink/src/main/java/io/openlineage/flink/client/EventEmitter.java

+ +

> But we see that the OpenLineageFlinkJobListener constructor is defined as a private constructor - just wanted to check with the team whether it was just a default scope, or intended to be private. +The way to construct is is a public builder in the same class

+ +

I think easier way than wrapper class would be use existing flink configuration, or to set up OPENLINEAGE_URL env variable, or have openlineage.yml config file - not sure why this is the way you've chosen?

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/26494b596e9669d2ada164066a73c44e04e884ba/integration/flink/src/main/java/io/openlineage/flink/client/EventEmitter.java | EventEmitter.java>
+
+ <pre><code> public EventEmitter(Configuration configuration) { +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Athitya Kumar + (athityakumar@gmail.com) +
+
2023-11-09 12:41:02
 +
+

*Thread Reply:* > I think easier way than wrapper class would be use existing flink configuration, or to set up OPENLINEAGE_URL env variable, or have openlineage.yml config file - not sure why this is the way you've chosen? +@Maciej Obuchowski - The reasoning behind going with a wrapper class is that we can abstract out the nitty-gritty like how/where we're publishing openlineage events etc - especially for companies that have a lot of teams that may be adopting openlineage.

+ +

For example, if we wanna move away from http transport to kafka transport - we'd be changing only this wrapper class and ask folks to update their wrapper class dependency version. If we went without the wrapper class, then the exact config changes would need to be synced and done by many different teams, who may not have enough context.

+ +

Similarly, if we wanna enable some other default best-practise configs, or inject any company-specific configs etc, the wrapper would be useful in abstracting out the details and be the 1 place that handles all openlineage related integrations for any future changes.

+ +

That's why we wanna extend openlineage's listener class & leverage most of the OSS code as-is; and at the same time, have the ability to extend & inject customisations. I think that's where some things like having getters for the class object attributes, or having public constructors would be really helpful 😄

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-11-09 13:03:56
 +
+

*Thread Reply:* @Athitya Kumar that makes sense. Feel free to provide PR adding getters and stuff.

+ + + +
+ 🎉 Athitya Kumar +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Yannick Libert + (yannick.libert.partner@decathlon.com) +
+
2023-11-07 06:03:49
 +
+

Hi all, we (I work with @Sophie LY and @Abdallah) have a quick question regarding the spark integration: +if a spark app contains several jobs, they will be named "mysparkappname.job1" and "mysparkappname.job2" +eg: +sparkjob.collectlimit +sparkjob.mappartitionsparallelcollection

+ +

If I understood correctly, the spark integration maps one Spark job to a single OpenLineage Job, and the application itself should be assigned a Run id at startup and each job that executes will report the application's Run id as its parent job run (taken from: https://openlineage.io/docs/integrations/spark/).

+ +

In our case, the app Run Id is never created, and the jobs runs don't contain any parent facets. We tested it with a recent integration version in 1.4.1 and also an older one (0.26.0). +Did we miss something in the OL spark integration config?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-11-07 06:07:51
 +
+

*Thread Reply:* hey, a name of the output dataset should be put at the end of the job name. This was introduced to help with jobs that call multiple spark actions

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Yannick Libert + (yannick.libert.partner@decathlon.com) +
+
2023-11-07 07:05:52
 +
+

*Thread Reply:* Hi Paweł, +Thanks for your answer, yes indeed with the newer version of OL, we automatically have the name of the output dataset at the end of the job name, but no App run id, nor any parent run facet.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-11-07 08:16:44
 +
+

*Thread Reply:* yes, you're right. I mean you can set in config spark.openlineage.parentJobName which will be shared through whole app run, but this needs to be set manually

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Yannick Libert + (yannick.libert.partner@decathlon.com) +
+
2023-11-07 08:36:58
 +
+

*Thread Reply:* I see, thanks a lot for your reply we'll try that

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
ldacey + (lance.dacey2@sutherlandglobal.com) +
+
2023-11-07 10:49:25
 +
+

if I have a dataset on adls gen2 which synapse connects to as an external delta table, is that the use case of a symlink dataset? the delta table is connected to by PBI and by Synapse, but the underlying data is exactly the same

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-11-08 10:49:04
 +
+

*Thread Reply:* Sounds like it, yes - if the logical dataset names are different but physical one is the same

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Rodrigo Maia + (rodrigo.maia@manta.io) +
+
2023-11-08 12:38:52
 +
+

Has anyone here tried OpenLineage with Spark on Amazon EMR?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason Yip + (jasonyip@gmail.com) +
+
2023-11-08 13:01:16
 +
+

*Thread Reply:* No but it should work the same I tried on AWS and Google Colab and Azure

+ + + +
+ 👍 Jakub Dardziński +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Tristan GUEZENNEC -CROIX- + (tristan.guezennec@decathlon.com) +
+
2023-11-09 03:10:54
 +
+

*Thread Reply:* Yes. @Abdallah could provide some details if needed.

+ + + +
+ 👍 Abdallah +
+ +
+ 🔥 Maciej Obuchowski +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Rodrigo Maia + (rodrigo.maia@manta.io) +
+
2023-11-20 11:29:26
 +
+

*Thread Reply:* Thanks @Tristan GUEZENNEC -CROIX- +HI @Abdallah i was able to set up a spark cluster on AWS EMR but im struggling to configure the OL Listener. Ive tried with steps and bootstrap actions for the jar and it didn't work out. How did you manage to include the jar? Besides, what about the spark configuration? Could you send me a sample of these configs?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-11-08 12:44:54
 +
+

@channel +Friendly reminder: this month’s TSC meeting, open to all, is tomorrow at 10 am PT: https://openlineage.slack.com/archives/C01CK9T7HKR/p1699027207361229

+
+ + +
+ + + } + + Michael Robinson + (https://openlineage.slack.com/team/U02LXF3HUN7) +
+ + + +
+
+ <p>@channel +This month’s TSC meeting (open to all) is next Thursday the 9th at 10am PT. On the agenda: +• announcements +• recent releases +• recent additions to the Flink integration by @Peter Huang +• recent additions to the Spark integration by @Paweł Leszczyński +• updates on proposals by @Julien Le Dem +• discussion topics +• open discussion +More info and the meeting link can be found on the <a href="https://openlineage.io/meetings/">website</a>. All are welcome! Do you have a discussion topic, use case or integration you’d like to demo? DM me to be added to the agenda.</p> +
+ + + + + + +
Original URL: https://openlineage.slack.com/archives/C01CK9T7HKR/p1699027207361229
+ + +
+ + Slack Conversation +
+ + +
+ + + +
+ 👍 Jakub Dardziński +
+ +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason Yip + (jasonyip@gmail.com) +
+
2023-11-10 15:25:45
 +
+

@Paweł Leszczyński regarding to https://github.com/OpenLineage/OpenLineage/issues/2124, OL is parsing out the table location in Hive metastore, it is the location of the table in the catalog and not the physical location of the data. It is both right and wrong because it is a table, just it is an external table.

+ +

https://docs.databricks.com/en/sql/language-manual/sql-ref-external-tables.html

+
+
docs.databricks.com
+ + + +
External tables
+
+ Learn about Unity Catalog external tables in Databricks SQL and Databricks Runtime. +
+ + + + + + + + +
Original URL: https://docs.databricks.com/en/sql/language-manual/sql-ref-external-tables.html
+ + + +
+
+ + + + +
#2124 Same Delta Table not catching the location on write
+
+ <p><strong>What is the target system?</strong></p> + +<p>Spark / Databricks</p> + +<p><strong>What kind of integration is this?</strong></p> + +<p>☐ Produces OpenLineage metadata +☐ Consumes OpenLineage metadata +☐ Something else</p> + +<p><strong>How should this integration be implemented?</strong></p> + +<p>I am using OL 1.2.2, Azure Databricks Runtime 11.3 LTS. When creating a table writing into a ADLS location, OL won't be able to catch the location of the output. But when I read the same object it will be able to read the location as INPUT.</p> + +<p>Please note I have also tested Databricks Runtime 13.3 LTS, Spark 3.4.1 - it will give correct ADLS location in INPUT but the input will only show up once in a blue moon. Most of the time the inputs and outputs are blank.</p> + +<pre><code> "inputs": [], + "outputs": [] +</code></pre> + +<pre><code>CREATE OR REPLACE TABLE transactions_adj +USING DELTA LOCATION '&lt;wasbs://studio@clororetaildevadls.blob.core.windows.net/examples/data/csv/completejourney/silver/transactions_adj&gt;' +AS + SELECT + household_id, + basket_id, + week_no, + day, + transaction_time, + store_id, + product_id, + amount_list, + campaign_coupon_discount, + manuf_coupon_discount, + manuf_coupon_match_discount, + total_coupon_discount, + instore_discount, + amount_paid, + units + FROM ( + SELECT + household_id, + basket_id, + week_no, + day, + transaction_time, + store_id, + product_id, + COALESCE(sales_amount - discount_amount - coupon_discount - coupon_discount_match,0.0) as amount_list, + CASE + WHEN COALESCE(coupon_discount_match,0.0) = 0.0 THEN -1 ** COALESCE(coupon_discount,0.0) + ELSE 0.0 + END as campaign_coupon_discount, + CASE + WHEN COALESCE(coupon_discount_match,0.0) != 0.0 THEN -1 ** COALESCE(coupon_discount,0.0) + ELSE 0.0 + END as manuf_coupon_discount, + -1 ** COALESCE(coupon_discount_match,0.0) as manuf_coupon_match_discount, + -1 ** COALESCE(coupon_discount - coupon_discount_match,0.0) as total_coupon_discount, + COALESCE(-1 ** discount_amount,0.0) as instore_discount, + COALESCE(sales_amount,0.0) as `amount_paid,` + quantity as units + FROM transactions + ); +</code></pre> + +<p>Here's the COMPLETE event:</p> + +<pre><code> + "outputs":[ + { + "namespace":"dbfs", + "name":"/user/hive/warehouse/journey.db/transactions_adj", + "facets":{ + "dataSource":{ + "_producer":"&lt;https://github.com/OpenLineage/OpenLineage/tree/1.2.2/integration/spark&gt;", + "_schemaURL":"&lt;https://openlineage.io/spec/facets/1-0-0/DatasourceDatasetFacet.json#/$defs/DatasourceDatasetFacet&gt;", + "name":"dbfs", + "uri":"dbfs" + }, + +</code></pre> + +<p>Below logical plan shows the path:</p> + +<pre><code>== Analyzed Logical Plan == +num_affected_rows: bigint, num_inserted_rows: bigint +ReplaceTableAsSelect TableSpec(Map(),Some(DELTA),Map(),Some(&lt;wasbs://studio@clororetaildevadls.blob.core.windows.net/examples/data/csv/completejourney/silver/transactions_adj&gt;),None,None,false,Set()), true +:- ResolvedIdentifier com.databricks.sql.managedcatalog.UnityCatalogV2Proxy@6251a8df, default.transactions_adj ++- Project [household_id#184, basket_id#185L, week_no#193, day#186, transaction_time#192, store_id#190, product_id#187, amount_list#147, campaign_coupon_discount#148, manuf_coupon_discount#149, manuf_coupon_match_discount#150, total_coupon_discount#151, instore_discount#152, amount_paid#153, units#154] + +- SubqueryAlias __auto_generated_subquery_name + +- Project [household_id#184, basket_id#185L, week_no#193, day#186, transaction_time#192, store_id#190, product_id#187, coalesce(cast((((sales_amount#189 - discount_amount#191) - coupon_discount#194) - coupon_discount_match#195) as double), cast(0.0 as double)) AS amount_list#147, CASE WHEN (coalesce(cast(coupon_discount_match#195 as double), cast(0.0 as double)) = cast(0.0 as double)) THEN (cast(-1 as double) ** coalesce(cast(coupon_discount#194 as double), cast(0.0 as double))) ELSE cast(0.0 as double) END AS campaign_coupon_discount#148, CASE WHEN NOT (coalesce(cast(coupon_discount_match#195 as double), cast(0.0 as double)) = cast(0.0 as double)) THEN (cast(-1 as double) ** coalesce(cast(coupon_discount#194 as double), cast(0.0 as double))) ELSE cast(0.0 as double) END AS manuf_coupon_discount#149, (cast(-1 as double) ** coalesce(cast(coupon_discount_match#195 as double), cast(0.0 as double))) AS manuf_coupon_match_discount#150, (cast(-1 as double) ** coalesce(cast((coupon_discount#194 - coupon_discount_match#195) as double), cast(0.0 as double))) AS total_coupon_discount#151, coalesce(cast((cast(-1 as float) ** discount_amount#191) as double), cast(0.0 as double)) AS instore_discount#152, coalesce(cast(sales_amount#189 as double), cast(0.0 as double)) AS amount_paid#153, quantity#188 AS units#154] + +- SubqueryAlias spark_catalog.default.transactions + +- Relation spark_catalog.default.transactions[household_id#184,basket_id#185L,day#186,product_id#187,quantity#188,sales_amount#189,store_id#190,discount_amount#191,transaction_time#192,week_no#193,coupon_discount#194,coupon_discount_match#195] parquet +</code></pre> + +<p><strong>Where should this integration be implemented?</strong></p> + +<p>☐ In the target system +☐ In the OpenLineage repo +☐ Somewhere else</p> + +<p><strong>Do you plan to make this contribution yourself?</strong></p> + +<p>☐ I am interested in doing this work</p> +
+ +
+
Labels
+ integration/spark, integration/databricks +
+ +
+
Comments
+ 5 +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason Yip + (jasonyip@gmail.com) +
+
2023-11-10 15:32:28
 +
+

*Thread Reply:* Here's for more reference: https://dilorom.medium.com/finding-the-path-to-a-table-in-databricks-2c74c6009dbb

+
+
Medium
+ + + +
Finding the Path to a Managed Table in Databricks
+
+ This article shows how to find a path for a managed Databricks table. +
+ +
+
Reading time
+ 2 min read +
+ + + + + + + + +
Original URL: https://dilorom.medium.com/finding-the-path-to-a-table-in-databricks-2c74c6009dbb
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason Yip + (jasonyip@gmail.com) +
+
2023-11-11 03:29:33
 +
+

@Paweł Leszczyński this is why if create a table with adls location it won't show input and output:

+ +

https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/spark35/src[…]k35/agent/lifecycle/plan/CreateReplaceOutputDatasetBuilder.java

+ +

Because the catalog object is not there.

+
+ + + + +
<https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/spark35/src/main/java/io/openlineage/spark35/agent/lifecycle/plan/CreateReplaceOutputDatasetBuilder.java | CreateReplaceOutputDatasetBuilder.java>
+
+ <pre><code> if (!di.isPresent()) { + return Collections.emptyList(); + } +</code></pre> +
+ + + + + + + +
+ + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
+ + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason Yip + (jasonyip@gmail.com) +
+
2023-11-11 03:30:44
 +
+

Databricks needs to be re-written in a way that supports Databricks it seems like

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason Yip + (jasonyip@gmail.com) +
+
2023-11-13 03:00:42
 +
+

@Paweł Leszczyński I went back to 1.4.1, output does show adls location. But environment facet is gone in 1.4.1. It shows up in 1.5.0 but namespace is back to dbfs....

+ +
+
data (11).json
+ + + + + + +
+ + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason Yip + (jasonyip@gmail.com) +
+
2023-11-13 03:18:37
 +
+

@Paweł Leszczyński I diff CreateReplaceDatasetBuilder.java and CreateReplaceOutputDatasetBuilder.java and they are the same except for the class name, so I am not sure what is causing the change. I also realize you don't have a test case for ADLS

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-11-13 04:52:07
 +
+

*Thread Reply:* Thanks @Jason Yip for your engagement in finding the cause and solution to this issue.

+ +

Among the technical problems, another problem here is that our databricks integration tests are run on AWS and the issue you describe occurs in Azure. I would consider this a primary issue as it is difficult for me to verify the behaviour you describe and fix it with a failing integration test at the start.

+ +

Are you able to reproduce the issue on AWS Databricks environment so that we could include it in our integration tests and make sure the behvaiour will not change later on in future?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Jason Yip + (jasonyip@gmail.com) +
+
2023-11-13 18:06:44
 +
+

*Thread Reply:* I didn't know Azure and AWS Databricks are different. Let me try it on AWS as well

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Naresh reddy + (naresh.naresh36@gmail.com) +
+
2023-11-15 07:17:24
 +
+

Hi +Can anyone point me to the deck on how Airflow can be integrated using Openlineage?

+ + + +
+
+
+
+ + + + + +
+ +
+ + + + + +
+
+ + + + +
+ +
Naresh reddy + (naresh.naresh36@gmail.com) +
+
2023-11-15 07:27:55
 +
+

*Thread Reply:* thank you @Maciej Obuchowski

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Naresh reddy + (naresh.naresh36@gmail.com) +
+
2023-11-15 11:09:24
 +
+

Can anyone tell me why OL is better than other competitors if you can provide an analysis that would be great

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Harel Shein + (harel.shein@gmail.com) +
+
2023-11-16 11:46:16
 +
+

*Thread Reply:* Hey @Naresh reddy can you help me understand what you mean by competitors? +OL is a specification that can be used to solve various problems, so if you have a clear problem statement, maybe I can help with pros/cons for that problem

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Naresh reddy + (naresh.naresh36@gmail.com) +
+
2023-11-15 11:10:58
 +
+

what are the pros and cons of OL. we often talk about positives to market it but what are the pain points using OL,how it's addressing user issues?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Michael Robinson + (michael.robinson@astronomer.io) +
+
2023-11-16 13:38:42
 +
+

*Thread Reply:* Hi @Naresh reddy, thanks for your question. We’ve heard that OpenLineage is attractive because of its desirable integrations, including a best-in-class Spark integration, its extensibility, the fact that it’s not destructive, and the fact that it’s open source. I’m not aware of pain points per se, but there are certainly features and integrations that we wish we could focus on but can’t at the moment — like the Dagster integration, which needs a new maintainer. OpenLineage is like any other open standard in that ecosystem coverage is a constant process rather than a journey, and it requires contributions in order to get close to 100%. Thankfully, we are gaining users and contributors all the time, and integrations are being added or improved upon daily. See the Ecosystem page on the website for a list of consumers and producers and links to more resources, and check out the GitHub repo for the codebase, commit history, contributors, governance procedures, and more. We’re quick to respond to messages here and issues on GitHub — usually within one day.

+
+ + + + +
OpenLineage/OpenLineage
+
+ An Open Standard for lineage metadata collection +
+ +
+
Website
+ <http://openlineage.io> +
+ +
+
Stars
+ 1449 +
+ + + + + + + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
karthik nandagiri + (karthik.nandagiri@gmail.com) +
+
2023-11-19 23:57:38
 +
+

Hi So we can use openlineage to identify column level lineage with Airflow , Spark? will it also allow to connect to Power BI and derive the downstream column lineage ?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-11-20 06:07:36
 +
+

*Thread Reply:* Yes, it works with Airflow and Spark - there is caveat that amount of operators that support it on Airflow side is fairly small and limited generally to most popular SQL operators. +> will it also allow to connect to Power BI and derive the downstream column lineage ? +No, there is no such feature yet 🙂 +However, there's nothing preventing this - if you wish to work on such implementation, we'd be happy to help.

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
karthik nandagiri + (karthik.nandagiri@gmail.com) +
+
2023-11-21 00:20:11
 +
+

*Thread Reply:* Thank you Maciej Obuchowski for the update. Currently we are looking out for a tool which can support connecting to Power Bi and pull column level lineage information for reports and dashboards. How this can be achieved with OL ? Can you give some idea?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-11-21 07:59:10
 +
+

*Thread Reply:* I don't think I can help you with that now, unless you want to work on your own integration with PowerBI 🙁

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Rafał Wójcik + (rwojcik@griddynamics.com) +
+
2023-11-21 07:02:08
 +
+

Hi Everyone, first of all - big shout to all contributors - You do amazing job here. +I want to use OpenLineage in our project - to do so I want to setup some POC and experiment with possibilities library provides - I start working on sample from the conference talk: https://github.com/getindata/openlineage-bbuzz2023-column-lineage but when I go into spark transformation after staring context with openlineage I have issues with SessionHiveMetaStoreClient on section 3- does anyone has other plain sample to play with, to not setup everything from scratch?

+
+ + + + +
getindata/openlineage-bbuzz2023-column-lineage
+
+ +
+ +
+
Language
+ Jupyter Notebook +
+ +
+
Last updated
+ 5 months ago +
+ + + + + + + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-11-21 07:37:00
 +
+

*Thread Reply:* Can you provide details about those issues? Like exceptions, logs, details of the jobs and how do you run them?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Rafał Wójcik + (rwojcik@griddynamics.com) +
+
2023-11-21 07:45:37
 +
+

*Thread Reply:* Hi @Maciej Obuchowski - I rerun docker container after deleting metadata_db folder possibly created by other local test, and fix this one but got problem with OpenLineageListener - during initialization of spark: +while I execute: +spark = (SparkSession.builder.master('local') + .appName('Food Delivery') + .config('spark.extraListeners', 'io.openlineage.spark.agent.OpenLineageSparkListener') + .config('spark.jars', '&lt;local-path&gt;/openlineage-spark-0.27.2.jar,&lt;local-path&gt;/postgresql-42.6.0.jar') + .config('spark.openlineage.transport.type', 'http') + .config('spark.openlineage.transport.url', '<http://api:5000>') + .config('spark.openlineage.facets.disabled', '[spark_unknown;spark.logicalPlan]') + .config('spark.openlineage.namespace', 'food-delivery') + .config('spark.sql.warehouse.dir', '/tmp/spark-warehouse/') + .config("spark.sql.repl.eagerEval.enabled", True) + .enableHiveSupport() + .getOrCreate()) +I got +Py4JJavaError: An error occurred while calling None.org.apache.spark.api.java.JavaSparkContext. +: org.apache.spark.SparkException: Exception when registering SparkListener + at org.apache.spark.SparkContext.setupAndStartListenerBus(SparkContext.scala:2563) + at org.apache.spark.SparkContext.&lt;init&gt;(SparkContext.scala:643) + at org.apache.spark.api.java.JavaSparkContext.&lt;init&gt;(JavaSparkContext.scala:58) + at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) + at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77) + at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) + at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499) + at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:480) + at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:247) + at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357) + at py4j.Gateway.invoke(Gateway.java:238) + at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80) + at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69) + at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182) + at py4j.ClientServerConnection.run(ClientServerConnection.java:106) + at java.base/java.lang.Thread.run(Thread.java:833) +Caused by: java.lang.ClassNotFoundException: io.openlineage.spark.agent.OpenLineageSparkListener + at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:445) + at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:587) + at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:520) + at java.base/java.lang.Class.forName0(Native Method) + at java.base/java.lang.Class.forName(Class.java:467) + at org.apache.spark.util.Utils$.classForName(Utils.scala:218) + at org.apache.spark.util.Utils$.$anonfun$loadExtensions$1(Utils.scala:2921) + at scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:293) + at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62) + at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55) + at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49) + at scala.collection.TraversableLike.flatMap(TraversableLike.scala:293) + at scala.collection.TraversableLike.flatMap$(TraversableLike.scala:290) + at scala.collection.AbstractTraversable.flatMap(Traversable.scala:108) + at org.apache.spark.util.Utils$.loadExtensions(Utils.scala:2919) + at org.apache.spark.SparkContext.$anonfun$setupAndStartListenerBus$1(SparkContext.scala:2552) + at org.apache.spark.SparkContext.$anonfun$setupAndStartListenerBus$1$adapted(SparkContext.scala:2551) + at scala.Option.foreach(Option.scala:407) + at org.apache.spark.SparkContext.setupAndStartListenerBus(SparkContext.scala:2551) + ... 15 more +looks like by some reasons jars are not loaded - need to look into it

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-11-21 07:58:09
 +
+

*Thread Reply:* 🤔 Jars are added during image building: https://github.com/getindata/openlineage-bbuzz2023-column-lineage/blob/main/Dockerfile#L12C1-L12C29

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-11-21 07:58:28
 +
+

*Thread Reply:* are you sure &lt;local-path&gt; is right?

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Rafał Wójcik + (rwojcik@griddynamics.com) +
+
2023-11-21 08:00:49
 +
+

*Thread Reply:* yes, it's same as in sample - wondering why it's not get added: +```from pyspark.sql import SparkSession

+ +

spark = (SparkSession.builder.master('local') + .appName('Food Delivery') + .config('spark.jars', '/home/jovyan/jars/openlineage-spark-0.27.2.jar,/home/jovyan/jars/postgresql-42.6.0.jar') + .config('spark.sql.warehouse.dir', '/tmp/spark-warehouse/') + .config("spark.sql.repl.eagerEval.enabled", True) + .enableHiveSupport() + .getOrCreate())

+ +

print(spark.sparkContext._jsc.sc().listJars())

+ +

Vector()```

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-11-21 08:04:31
 +
+

*Thread Reply:* can you make sure jars are in this directory? just by docker run --entrypoint /usr/local/bin/bash IMAGE_NAME "ls /home/jovyan/jars"

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
+
2023-11-21 08:06:27
 +
+

*Thread Reply:* another option to try is to replace spark.jars with spark.jars.packages io.openlineage:openlineage_spark:1.5.0,org.postgresql:postgresql:42.7.0

+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
+
2023-11-21 08:16:54
 +
+

*Thread Reply:* I think this was done for the purpose of presentation to make sure the demo will work without internet access. This can be the reason to add jar manually to a docker. openlineage-spark can be added to Spark via spark.jar.packages , like we do here https://openlineage.io/docs/integrations/spark/quickstart_local

+
+
openlineage.io
+ + + +
Quickstart with Jupyter | OpenLineage
+
+ Trying out the Spark integration is super easy if you already have Docker Desktop and git installed. +
+ + + + + + +
Original URL: https://openlineage.io/docs/integrations/spark/quickstart_local
+ + + +
+ + + +
+
+
+
+ + + + + +
+
+ + + + +
+ +
Rafał Wójcik + (rwojcik@griddynamics.com) +
+
2023-11-21 09:21:59
 +
+

*Thread Reply:* got it guys - thanks a lot for help - it turns out that spark context from notebook 2 and 3 has come kind of metadata conflict - when I combine those 2 and recreate image to clean up old metadata it works. +One more note is that sometimes kernels return weird results but it may be caused by some local nuances - anyways thx !

+ + + +
+
+
+
+ + + + + + + + + + + \ No newline at end of file diff --git a/static/viewer.css b/static/viewer.css new file mode 100644 index 0000000..96be753 --- /dev/null +++ b/static/viewer.css @@ -0,0 +1,241 @@ +@import url('https://fonts.googleapis.com/css?family=Lato:400,900'); + +html { + font-family: 'Lato', sans-serif; +} + +body { + padding: 0; + margin: 0; +} + +#slack-archive-viewer { + padding: 0; + margin: 0; + height: 100vh; + overflow: hidden; +} + +#sidebar { + display: inline-block; + width: 280px; + color: white; + text-align: left; + background-color: #4D394B; + z-index: 10; + overflow-y: scroll; + overflow-x: auto; + height: 100vh; + user-select: none; +} + +#sidebar a { + color: white; + font-size: 14px; +} + +#sidebar h3 { + margin: 20px 20px; + color: white; + font-weight: 900; +} + +#sidebar h3:hover { + cursor: pointer; +} + +#sidebar h3::after { + content: '❯ '; + display: inline-block; + -webkit-transform: rotate(90deg); + transform: rotate(90deg); + margin-left: 15px; +} + +#sidebar h3.arrow::after { + margin-left: 10px; + -webkit-transform: none; + transform: none; +} + +.messages { + width: calc(100vw - 325px); + height: 100vh; + text-align: left; + display: inline-block; + padding-left: 20px; + padding-right: 20px; + overflow-y: scroll; +} + +.message-container { + clear: left; + min-height: 56px; +} + +.message-container:first-child { + margin-top: 20px; +} + +.message-container:last-child { + margin-bottom: 20px; +} + +.message-container .user_icon { + background-color: rgb(248, 244, 240); + width: 36px; + height: 36px; + border-radius: 0.2em; + display: inline-block; + vertical-align: top; + margin-right: 0.65em; + float: left; +} + +.message-container .user_icon_reply { + background-color: rgb(248, 244, 240); + width: 36px; + height: 36px; + border-radius: 0.2em; + display: inline-block; + vertical-align: top; + margin-right: 0.65em; + margin-left: 40px; + float: left; +} + +.message-container .time { + display: inline-block; + color: rgb(200, 200, 200); + margin-left: 0.5em; +} + +.message-container .username { + display: inline-block; + font-weight: 600; + line-height: 1; +} + +.message-container .user-email { + font-weight: normal; + font-style: italic; +} + +.message-container .message { + display: inline-block; + vertical-align: top; + line-height: 1; + width: calc(100% - 3em); +} + +.message-container .reply { + vertical-align: top; + line-height: 1; + width: calc(100% - 3em); + margin-left: 80px; +} + +.message-container .msg p { + white-space: pre-wrap; +} + +.message-container .msg pre { + background-color: #E6E5DF; + white-space: pre-wrap; +} + +.message-container .message .msg { + line-height: 1.5; +} + +.message-container .reply .msg { + line-height: 1.5; +} + +.message-container .message .msg a { + overflow-wrap: anywhere; +} + +.message-container .reply .msg a { + overflow-wrap: anywhere; +} + +.message-container .message-attachment { + padding-left: 5px; + border-left: 2px gray solid; + overflow-wrap: anywhere; +} + +.message-container .message-attachment .service-name { + color: #999999; +} + +.message-container .icon { + max-width: 10px; +} + +.channel_join .msg, .channel_topic .msg, +.bot_add .msg, .app_conversation_join .msg { + font-style: italic; +} + +.attachment-footer { + font-size: small; +} + +.list { + margin: 0; + padding: 0; + list-style-type: none; +} + +.list li { + padding: 4px 20px; +} + +.list li a { + width: 100%; + padding: 10px 20px; +} + +.list li.active { + background-color: #4C9689; +} + +.list li.active:hover { + background-color: #4C9689; +} + +.list li:hover { + text-decoration: none; + background: #3E313C; +} + +.list li a:hover { + text-decoration: none; +} + +a:link, +a:visited, +a:active { + color: #2a80b9; + text-decoration: none; +} + +a:hover { + color: #439fe0; + text-decoration: underline; +} + +.close { + display: none; +} + +@media screen { + .print-only { display: none } +} + +img.preview { + max-width: 100%; + height: auto; +} \ No newline at end of file diff --git a/venv/bin/Activate.ps1 b/venv/bin/Activate.ps1 deleted file mode 100644 index b49d77b..0000000 --- a/venv/bin/Activate.ps1 +++ /dev/null @@ -1,247 +0,0 @@ -<# -.Synopsis -Activate a Python virtual environment for the current PowerShell session. - -.Description -Pushes the python executable for a virtual environment to the front of the -$Env:PATH environment variable and sets the prompt to signify that you are -in a Python virtual environment. Makes use of the command line switches as -well as the `pyvenv.cfg` file values present in the virtual environment. - -.Parameter VenvDir -Path to the directory that contains the virtual environment to activate. The -default value for this is the parent of the directory that the Activate.ps1 -script is located within. - -.Parameter Prompt -The prompt prefix to display when this virtual environment is activated. By -default, this prompt is the name of the virtual environment folder (VenvDir) -surrounded by parentheses and followed by a single space (ie. '(.venv) '). - -.Example -Activate.ps1 -Activates the Python virtual environment that contains the Activate.ps1 script. - -.Example -Activate.ps1 -Verbose -Activates the Python virtual environment that contains the Activate.ps1 script, -and shows extra information about the activation as it executes. - -.Example -Activate.ps1 -VenvDir C:\Users\MyUser\Common\.venv -Activates the Python virtual environment located in the specified location. - -.Example -Activate.ps1 -Prompt "MyPython" -Activates the Python virtual environment that contains the Activate.ps1 script, -and prefixes the current prompt with the specified string (surrounded in -parentheses) while the virtual environment is active. - -.Notes -On Windows, it may be required to enable this Activate.ps1 script by setting the -execution policy for the user. You can do this by issuing the following PowerShell -command: - -PS C:\> Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope CurrentUser - -For more information on Execution Policies: -https://go.microsoft.com/fwlink/?LinkID=135170 - -#> -Param( - [Parameter(Mandatory = $false)] - [String] - $VenvDir, - [Parameter(Mandatory = $false)] - [String] - $Prompt -) - -<# Function declarations --------------------------------------------------- #> - -<# -.Synopsis -Remove all shell session elements added by the Activate script, including the -addition of the virtual environment's Python executable from the beginning of -the PATH variable. - -.Parameter NonDestructive -If present, do not remove this function from the global namespace for the -session. - -#> -function global:deactivate ([switch]$NonDestructive) { - # Revert to original values - - # The prior prompt: - if (Test-Path -Path Function:_OLD_VIRTUAL_PROMPT) { - Copy-Item -Path Function:_OLD_VIRTUAL_PROMPT -Destination Function:prompt - Remove-Item -Path Function:_OLD_VIRTUAL_PROMPT - } - - # The prior PYTHONHOME: - if (Test-Path -Path Env:_OLD_VIRTUAL_PYTHONHOME) { - Copy-Item -Path Env:_OLD_VIRTUAL_PYTHONHOME -Destination Env:PYTHONHOME - Remove-Item -Path Env:_OLD_VIRTUAL_PYTHONHOME - } - - # The prior PATH: - if (Test-Path -Path Env:_OLD_VIRTUAL_PATH) { - Copy-Item -Path Env:_OLD_VIRTUAL_PATH -Destination Env:PATH - Remove-Item -Path Env:_OLD_VIRTUAL_PATH - } - - # Just remove the VIRTUAL_ENV altogether: - if (Test-Path -Path Env:VIRTUAL_ENV) { - Remove-Item -Path env:VIRTUAL_ENV - } - - # Just remove VIRTUAL_ENV_PROMPT altogether. - if (Test-Path -Path Env:VIRTUAL_ENV_PROMPT) { - Remove-Item -Path env:VIRTUAL_ENV_PROMPT - } - - # Just remove the _PYTHON_VENV_PROMPT_PREFIX altogether: - if (Get-Variable -Name "_PYTHON_VENV_PROMPT_PREFIX" -ErrorAction SilentlyContinue) { - Remove-Variable -Name _PYTHON_VENV_PROMPT_PREFIX -Scope Global -Force - } - - # Leave deactivate function in the global namespace if requested: - if (-not $NonDestructive) { - Remove-Item -Path function:deactivate - } -} - -<# -.Description -Get-PyVenvConfig parses the values from the pyvenv.cfg file located in the -given folder, and returns them in a map. - -For each line in the pyvenv.cfg file, if that line can be parsed into exactly -two strings separated by `=` (with any amount of whitespace surrounding the =) -then it is considered a `key = value` line. The left hand string is the key, -the right hand is the value. - -If the value starts with a `'` or a `"` then the first and last character is -stripped from the value before being captured. - -.Parameter ConfigDir -Path to the directory that contains the `pyvenv.cfg` file. -#> -function Get-PyVenvConfig( - [String] - $ConfigDir -) { - Write-Verbose "Given ConfigDir=$ConfigDir, obtain values in pyvenv.cfg" - - # Ensure the file exists, and issue a warning if it doesn't (but still allow the function to continue). - $pyvenvConfigPath = Join-Path -Resolve -Path $ConfigDir -ChildPath 'pyvenv.cfg' -ErrorAction Continue - - # An empty map will be returned if no config file is found. - $pyvenvConfig = @{ } - - if ($pyvenvConfigPath) { - - Write-Verbose "File exists, parse `key = value` lines" - $pyvenvConfigContent = Get-Content -Path $pyvenvConfigPath - - $pyvenvConfigContent | ForEach-Object { - $keyval = $PSItem -split "\s*=\s*", 2 - if ($keyval[0] -and $keyval[1]) { - $val = $keyval[1] - - # Remove extraneous quotations around a string value. - if ("'""".Contains($val.Substring(0, 1))) { - $val = $val.Substring(1, $val.Length - 2) - } - - $pyvenvConfig[$keyval[0]] = $val - Write-Verbose "Adding Key: '$($keyval[0])'='$val'" - } - } - } - return $pyvenvConfig -} - - -<# Begin Activate script --------------------------------------------------- #> - -# Determine the containing directory of this script -$VenvExecPath = Split-Path -Parent $MyInvocation.MyCommand.Definition -$VenvExecDir = Get-Item -Path $VenvExecPath - -Write-Verbose "Activation script is located in path: '$VenvExecPath'" -Write-Verbose "VenvExecDir Fullname: '$($VenvExecDir.FullName)" -Write-Verbose "VenvExecDir Name: '$($VenvExecDir.Name)" - -# Set values required in priority: CmdLine, ConfigFile, Default -# First, get the location of the virtual environment, it might not be -# VenvExecDir if specified on the command line. -if ($VenvDir) { - Write-Verbose "VenvDir given as parameter, using '$VenvDir' to determine values" -} -else { - Write-Verbose "VenvDir not given as a parameter, using parent directory name as VenvDir." - $VenvDir = $VenvExecDir.Parent.FullName.TrimEnd("\\/") - Write-Verbose "VenvDir=$VenvDir" -} - -# Next, read the `pyvenv.cfg` file to determine any required value such -# as `prompt`. -$pyvenvCfg = Get-PyVenvConfig -ConfigDir $VenvDir - -# Next, set the prompt from the command line, or the config file, or -# just use the name of the virtual environment folder. -if ($Prompt) { - Write-Verbose "Prompt specified as argument, using '$Prompt'" -} -else { - Write-Verbose "Prompt not specified as argument to script, checking pyvenv.cfg value" - if ($pyvenvCfg -and $pyvenvCfg['prompt']) { - Write-Verbose " Setting based on value in pyvenv.cfg='$($pyvenvCfg['prompt'])'" - $Prompt = $pyvenvCfg['prompt']; - } - else { - Write-Verbose " Setting prompt based on parent's directory's name. (Is the directory name passed to venv module when creating the virtual environment)" - Write-Verbose " Got leaf-name of $VenvDir='$(Split-Path -Path $venvDir -Leaf)'" - $Prompt = Split-Path -Path $venvDir -Leaf - } -} - -Write-Verbose "Prompt = '$Prompt'" -Write-Verbose "VenvDir='$VenvDir'" - -# Deactivate any currently active virtual environment, but leave the -# deactivate function in place. -deactivate -nondestructive - -# Now set the environment variable VIRTUAL_ENV, used by many tools to determine -# that there is an activated venv. -$env:VIRTUAL_ENV = $VenvDir - -if (-not $Env:VIRTUAL_ENV_DISABLE_PROMPT) { - - Write-Verbose "Setting prompt to '$Prompt'" - - # Set the prompt to include the env name - # Make sure _OLD_VIRTUAL_PROMPT is global - function global:_OLD_VIRTUAL_PROMPT { "" } - Copy-Item -Path function:prompt -Destination function:_OLD_VIRTUAL_PROMPT - New-Variable -Name _PYTHON_VENV_PROMPT_PREFIX -Description "Python virtual environment prompt prefix" -Scope Global -Option ReadOnly -Visibility Public -Value $Prompt - - function global:prompt { - Write-Host -NoNewline -ForegroundColor Green "($_PYTHON_VENV_PROMPT_PREFIX) " - _OLD_VIRTUAL_PROMPT - } - $env:VIRTUAL_ENV_PROMPT = $Prompt -} - -# Clear PYTHONHOME -if (Test-Path -Path Env:PYTHONHOME) { - Copy-Item -Path Env:PYTHONHOME -Destination Env:_OLD_VIRTUAL_PYTHONHOME - Remove-Item -Path Env:PYTHONHOME -} - -# Add the venv to the PATH -Copy-Item -Path Env:PATH -Destination Env:_OLD_VIRTUAL_PATH -$Env:PATH = "$VenvExecDir$([System.IO.Path]::PathSeparator)$Env:PATH" diff --git a/venv/bin/activate b/venv/bin/activate deleted file mode 100644 index 78ead9a..0000000 --- a/venv/bin/activate +++ /dev/null @@ -1,69 +0,0 @@ -# This file must be used with "source bin/activate" *from bash* -# you cannot run it directly - -deactivate () { - # reset old environment variables - if [ -n "${_OLD_VIRTUAL_PATH:-}" ] ; then - PATH="${_OLD_VIRTUAL_PATH:-}" - export PATH - unset _OLD_VIRTUAL_PATH - fi - if [ -n "${_OLD_VIRTUAL_PYTHONHOME:-}" ] ; then - PYTHONHOME="${_OLD_VIRTUAL_PYTHONHOME:-}" - export PYTHONHOME - unset _OLD_VIRTUAL_PYTHONHOME - fi - - # This should detect bash and zsh, which have a hash command that must - # be called to get it to forget past commands. Without forgetting - # past commands the $PATH changes we made may not be respected - if [ -n "${BASH:-}" -o -n "${ZSH_VERSION:-}" ] ; then - hash -r 2> /dev/null - fi - - if [ -n "${_OLD_VIRTUAL_PS1:-}" ] ; then - PS1="${_OLD_VIRTUAL_PS1:-}" - export PS1 - unset _OLD_VIRTUAL_PS1 - fi - - unset VIRTUAL_ENV - unset VIRTUAL_ENV_PROMPT - if [ ! "${1:-}" = "nondestructive" ] ; then - # Self destruct! - unset -f deactivate - fi -} - -# unset irrelevant variables -deactivate nondestructive - -VIRTUAL_ENV="/Users/michael/Documents/astronomer_projects/openlineage_slack_archives/venv" -export VIRTUAL_ENV - -_OLD_VIRTUAL_PATH="$PATH" -PATH="$VIRTUAL_ENV/bin:$PATH" -export PATH - -# unset PYTHONHOME if set -# this will fail if PYTHONHOME is set to the empty string (which is bad anyway) -# could use `if (set -u; : $PYTHONHOME) ;` in bash -if [ -n "${PYTHONHOME:-}" ] ; then - _OLD_VIRTUAL_PYTHONHOME="${PYTHONHOME:-}" - unset PYTHONHOME -fi - -if [ -z "${VIRTUAL_ENV_DISABLE_PROMPT:-}" ] ; then - _OLD_VIRTUAL_PS1="${PS1:-}" - PS1="(venv) ${PS1:-}" - export PS1 - VIRTUAL_ENV_PROMPT="(venv) " - export VIRTUAL_ENV_PROMPT -fi - -# This should detect bash and zsh, which have a hash command that must -# be called to get it to forget past commands. Without forgetting -# past commands the $PATH changes we made may not be respected -if [ -n "${BASH:-}" -o -n "${ZSH_VERSION:-}" ] ; then - hash -r 2> /dev/null -fi diff --git a/venv/bin/activate.csh b/venv/bin/activate.csh deleted file mode 100644 index 68089b0..0000000 --- a/venv/bin/activate.csh +++ /dev/null @@ -1,26 +0,0 @@ -# This file must be used with "source bin/activate.csh" *from csh*. -# You cannot run it directly. -# Created by Davide Di Blasi . -# Ported to Python 3.3 venv by Andrew Svetlov - -alias deactivate 'test $?_OLD_VIRTUAL_PATH != 0 && setenv PATH "$_OLD_VIRTUAL_PATH" && unset _OLD_VIRTUAL_PATH; rehash; test $?_OLD_VIRTUAL_PROMPT != 0 && set prompt="$_OLD_VIRTUAL_PROMPT" && unset _OLD_VIRTUAL_PROMPT; unsetenv VIRTUAL_ENV; unsetenv VIRTUAL_ENV_PROMPT; test "\!:*" != "nondestructive" && unalias deactivate' - -# Unset irrelevant variables. -deactivate nondestructive - -setenv VIRTUAL_ENV "/Users/michael/Documents/astronomer_projects/openlineage_slack_archives/venv" - -set _OLD_VIRTUAL_PATH="$PATH" -setenv PATH "$VIRTUAL_ENV/bin:$PATH" - - -set _OLD_VIRTUAL_PROMPT="$prompt" - -if (! "$?VIRTUAL_ENV_DISABLE_PROMPT") then - set prompt = "(venv) $prompt" - setenv VIRTUAL_ENV_PROMPT "(venv) " -endif - -alias pydoc python -m pydoc - -rehash diff --git a/venv/bin/activate.fish b/venv/bin/activate.fish deleted file mode 100644 index 0600a95..0000000 --- a/venv/bin/activate.fish +++ /dev/null @@ -1,69 +0,0 @@ -# This file must be used with "source /bin/activate.fish" *from fish* -# (https://fishshell.com/); you cannot run it directly. - -function deactivate -d "Exit virtual environment and return to normal shell environment" - # reset old environment variables - if test -n "$_OLD_VIRTUAL_PATH" - set -gx PATH $_OLD_VIRTUAL_PATH - set -e _OLD_VIRTUAL_PATH - end - if test -n "$_OLD_VIRTUAL_PYTHONHOME" - set -gx PYTHONHOME $_OLD_VIRTUAL_PYTHONHOME - set -e _OLD_VIRTUAL_PYTHONHOME - end - - if test -n "$_OLD_FISH_PROMPT_OVERRIDE" - set -e _OLD_FISH_PROMPT_OVERRIDE - # prevents error when using nested fish instances (Issue #93858) - if functions -q _old_fish_prompt - functions -e fish_prompt - functions -c _old_fish_prompt fish_prompt - functions -e _old_fish_prompt - end - end - - set -e VIRTUAL_ENV - set -e VIRTUAL_ENV_PROMPT - if test "$argv[1]" != "nondestructive" - # Self-destruct! - functions -e deactivate - end -end - -# Unset irrelevant variables. -deactivate nondestructive - -set -gx VIRTUAL_ENV "/Users/michael/Documents/astronomer_projects/openlineage_slack_archives/venv" - -set -gx _OLD_VIRTUAL_PATH $PATH -set -gx PATH "$VIRTUAL_ENV/bin" $PATH - -# Unset PYTHONHOME if set. -if set -q PYTHONHOME - set -gx _OLD_VIRTUAL_PYTHONHOME $PYTHONHOME - set -e PYTHONHOME -end - -if test -z "$VIRTUAL_ENV_DISABLE_PROMPT" - # fish uses a function instead of an env var to generate the prompt. - - # Save the current fish_prompt function as the function _old_fish_prompt. - functions -c fish_prompt _old_fish_prompt - - # With the original prompt function renamed, we can override with our own. - function fish_prompt - # Save the return status of the last command. - set -l old_status $status - - # Output the venv prompt; color taken from the blue of the Python logo. - printf "%s%s%s" (set_color 4B8BBE) "(venv) " (set_color normal) - - # Restore the return status of the previous command. - echo "exit $old_status" | . - # Output the original/"old" prompt. - _old_fish_prompt - end - - set -gx _OLD_FISH_PROMPT_OVERRIDE "$VIRTUAL_ENV" - set -gx VIRTUAL_ENV_PROMPT "(venv) " -end diff --git a/venv/bin/pip b/venv/bin/pip deleted file mode 100755 index aa64131..0000000 --- a/venv/bin/pip +++ /dev/null @@ -1,8 +0,0 @@ -#!/Users/michael/Documents/astronomer_projects/openlineage_slack_archives/venv/bin/python3.11 -# -*- coding: utf-8 -*- -import re -import sys -from pip._internal.cli.main import main -if __name__ == '__main__': - sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0]) - sys.exit(main()) diff --git a/venv/bin/pip3 b/venv/bin/pip3 deleted file mode 100755 index aa64131..0000000 --- a/venv/bin/pip3 +++ /dev/null @@ -1,8 +0,0 @@ -#!/Users/michael/Documents/astronomer_projects/openlineage_slack_archives/venv/bin/python3.11 -# -*- coding: utf-8 -*- -import re -import sys -from pip._internal.cli.main import main -if __name__ == '__main__': - sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0]) - sys.exit(main()) diff --git a/venv/bin/pip3.11 b/venv/bin/pip3.11 deleted file mode 100755 index aa64131..0000000 --- a/venv/bin/pip3.11 +++ /dev/null @@ -1,8 +0,0 @@ -#!/Users/michael/Documents/astronomer_projects/openlineage_slack_archives/venv/bin/python3.11 -# -*- coding: utf-8 -*- -import re -import sys -from pip._internal.cli.main import main -if __name__ == '__main__': - sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0]) - sys.exit(main()) diff --git a/venv/bin/python b/venv/bin/python deleted file mode 120000 index 6e7f3c7..0000000 --- a/venv/bin/python +++ /dev/null @@ -1 +0,0 @@ -python3.11 \ No newline at end of file diff --git a/venv/bin/python3 b/venv/bin/python3 deleted file mode 120000 index 6e7f3c7..0000000 --- a/venv/bin/python3 +++ /dev/null @@ -1 +0,0 @@ -python3.11 \ No newline at end of file diff --git a/venv/bin/python3.11 b/venv/bin/python3.11 deleted file mode 120000 index 3cf1fbd..0000000 --- a/venv/bin/python3.11 +++ /dev/null @@ -1 +0,0 @@ -/opt/homebrew/opt/python@3.11/bin/python3.11 \ No newline at end of file diff --git a/venv/lib/python3.11/site-packages/_distutils_hack/__init__.py b/venv/lib/python3.11/site-packages/_distutils_hack/__init__.py deleted file mode 100644 index b951c2d..0000000 --- a/venv/lib/python3.11/site-packages/_distutils_hack/__init__.py +++ /dev/null @@ -1,227 +0,0 @@ -# don't import any costly modules -import sys -import os - - -is_pypy = '__pypy__' in sys.builtin_module_names - - -def warn_distutils_present(): - if 'distutils' not in sys.modules: - return - if is_pypy and sys.version_info < (3, 7): - # PyPy for 3.6 unconditionally imports distutils, so bypass the warning - # https://foss.heptapod.net/pypy/pypy/-/blob/be829135bc0d758997b3566062999ee8b23872b4/lib-python/3/site.py#L250 - return - import warnings - - warnings.warn( - "Distutils was imported before Setuptools, but importing Setuptools " - "also replaces the `distutils` module in `sys.modules`. This may lead " - "to undesirable behaviors or errors. To avoid these issues, avoid " - "using distutils directly, ensure that setuptools is installed in the " - "traditional way (e.g. not an editable install), and/or make sure " - "that setuptools is always imported before distutils." - ) - - -def clear_distutils(): - if 'distutils' not in sys.modules: - return - import warnings - - warnings.warn("Setuptools is replacing distutils.") - mods = [ - name - for name in sys.modules - if name == "distutils" or name.startswith("distutils.") - ] - for name in mods: - del sys.modules[name] - - -def enabled(): - """ - Allow selection of distutils by environment variable. - """ - which = os.environ.get('SETUPTOOLS_USE_DISTUTILS', 'local') - return which == 'local' - - -def ensure_local_distutils(): - import importlib - - clear_distutils() - - # With the DistutilsMetaFinder in place, - # perform an import to cause distutils to be - # loaded from setuptools._distutils. Ref #2906. - with shim(): - importlib.import_module('distutils') - - # check that submodules load as expected - core = importlib.import_module('distutils.core') - assert '_distutils' in core.__file__, core.__file__ - assert 'setuptools._distutils.log' not in sys.modules - - -def do_override(): - """ - Ensure that the local copy of distutils is preferred over stdlib. - - See https://github.com/pypa/setuptools/issues/417#issuecomment-392298401 - for more motivation. - """ - if enabled(): - warn_distutils_present() - ensure_local_distutils() - - -class _TrivialRe: - def __init__(self, *patterns): - self._patterns = patterns - - def match(self, string): - return all(pat in string for pat in self._patterns) - - -class DistutilsMetaFinder: - def find_spec(self, fullname, path, target=None): - # optimization: only consider top level modules and those - # found in the CPython test suite. - if path is not None and not fullname.startswith('test.'): - return - - method_name = 'spec_for_{fullname}'.format(**locals()) - method = getattr(self, method_name, lambda: None) - return method() - - def spec_for_distutils(self): - if self.is_cpython(): - return - - import importlib - import importlib.abc - import importlib.util - - try: - mod = importlib.import_module('setuptools._distutils') - except Exception: - # There are a couple of cases where setuptools._distutils - # may not be present: - # - An older Setuptools without a local distutils is - # taking precedence. Ref #2957. - # - Path manipulation during sitecustomize removes - # setuptools from the path but only after the hook - # has been loaded. Ref #2980. - # In either case, fall back to stdlib behavior. - return - - class DistutilsLoader(importlib.abc.Loader): - def create_module(self, spec): - mod.__name__ = 'distutils' - return mod - - def exec_module(self, module): - pass - - return importlib.util.spec_from_loader( - 'distutils', DistutilsLoader(), origin=mod.__file__ - ) - - @staticmethod - def is_cpython(): - """ - Suppress supplying distutils for CPython (build and tests). - Ref #2965 and #3007. - """ - return os.path.isfile('pybuilddir.txt') - - def spec_for_pip(self): - """ - Ensure stdlib distutils when running under pip. - See pypa/pip#8761 for rationale. - """ - if sys.version_info >= (3, 12) or self.pip_imported_during_build(): - return - clear_distutils() - self.spec_for_distutils = lambda: None - - @classmethod - def pip_imported_during_build(cls): - """ - Detect if pip is being imported in a build script. Ref #2355. - """ - import traceback - - return any( - cls.frame_file_is_setup(frame) for frame, line in traceback.walk_stack(None) - ) - - @staticmethod - def frame_file_is_setup(frame): - """ - Return True if the indicated frame suggests a setup.py file. - """ - # some frames may not have __file__ (#2940) - return frame.f_globals.get('__file__', '').endswith('setup.py') - - def spec_for_sensitive_tests(self): - """ - Ensure stdlib distutils when running select tests under CPython. - - python/cpython#91169 - """ - clear_distutils() - self.spec_for_distutils = lambda: None - - sensitive_tests = ( - [ - 'test.test_distutils', - 'test.test_peg_generator', - 'test.test_importlib', - ] - if sys.version_info < (3, 10) - else [ - 'test.test_distutils', - ] - ) - - -for name in DistutilsMetaFinder.sensitive_tests: - setattr( - DistutilsMetaFinder, - f'spec_for_{name}', - DistutilsMetaFinder.spec_for_sensitive_tests, - ) - - -DISTUTILS_FINDER = DistutilsMetaFinder() - - -def add_shim(): - DISTUTILS_FINDER in sys.meta_path or insert_shim() - - -class shim: - def __enter__(self): - insert_shim() - - def __exit__(self, exc, value, tb): - _remove_shim() - - -def insert_shim(): - sys.meta_path.insert(0, DISTUTILS_FINDER) - - -def _remove_shim(): - try: - sys.meta_path.remove(DISTUTILS_FINDER) - except ValueError: - pass - - -if sys.version_info < (3, 12): - # DistutilsMetaFinder can only be disabled in Python < 3.12 (PEP 632) - remove_shim = _remove_shim diff --git a/venv/lib/python3.11/site-packages/_distutils_hack/__pycache__/__init__.cpython-311.pyc b/venv/lib/python3.11/site-packages/_distutils_hack/__pycache__/__init__.cpython-311.pyc deleted file mode 100644 index ea268e2..0000000 Binary files a/venv/lib/python3.11/site-packages/_distutils_hack/__pycache__/__init__.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/_distutils_hack/__pycache__/override.cpython-311.pyc b/venv/lib/python3.11/site-packages/_distutils_hack/__pycache__/override.cpython-311.pyc deleted file mode 100644 index 672bc9b..0000000 Binary files a/venv/lib/python3.11/site-packages/_distutils_hack/__pycache__/override.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/_distutils_hack/override.py b/venv/lib/python3.11/site-packages/_distutils_hack/override.py deleted file mode 100644 index 2cc433a..0000000 --- a/venv/lib/python3.11/site-packages/_distutils_hack/override.py +++ /dev/null @@ -1 +0,0 @@ -__import__('_distutils_hack').do_override() diff --git a/venv/lib/python3.11/site-packages/distutils-precedence.pth b/venv/lib/python3.11/site-packages/distutils-precedence.pth deleted file mode 100644 index 7f009fe..0000000 --- a/venv/lib/python3.11/site-packages/distutils-precedence.pth +++ /dev/null @@ -1 +0,0 @@ -import os; var = 'SETUPTOOLS_USE_DISTUTILS'; enabled = os.environ.get(var, 'local') == 'local'; enabled and __import__('_distutils_hack').add_shim(); diff --git a/venv/lib/python3.11/site-packages/pip-23.3.1.dist-info/AUTHORS.txt b/venv/lib/python3.11/site-packages/pip-23.3.1.dist-info/AUTHORS.txt deleted file mode 100644 index 49e30f6..0000000 --- a/venv/lib/python3.11/site-packages/pip-23.3.1.dist-info/AUTHORS.txt +++ /dev/null @@ -1,751 +0,0 @@ -@Switch01 -A_Rog -Aakanksha Agrawal -Abhinav Sagar -ABHYUDAY PRATAP SINGH -abs51295 -AceGentile -Adam Chainz -Adam Tse -Adam Wentz -admin -Adrien Morison -ahayrapetyan -Ahilya -AinsworthK -Akash Srivastava -Alan Yee -Albert Tugushev -Albert-Guan -albertg -Alberto Sottile -Aleks Bunin -Ales Erjavec -Alethea Flowers -Alex Gaynor -Alex Grönholm -Alex Hedges -Alex Loosley -Alex Morega -Alex Stachowiak -Alexander Shtyrov -Alexandre Conrad -Alexey Popravka -Aleš Erjavec -Alli -Ami Fischman -Ananya Maiti -Anatoly Techtonik -Anders Kaseorg -Andre Aguiar -Andreas Lutro -Andrei Geacar -Andrew Gaul -Andrew Shymanel -Andrey Bienkowski -Andrey Bulgakov -Andrés Delfino -Andy Freeland -Andy Kluger -Ani Hayrapetyan -Aniruddha Basak -Anish Tambe -Anrs Hu -Anthony Sottile -Antoine Musso -Anton Ovchinnikov -Anton Patrushev -Antonio Alvarado Hernandez -Antony Lee -Antti Kaihola -Anubhav Patel -Anudit Nagar -Anuj Godase -AQNOUCH Mohammed -AraHaan -Arindam Choudhury -Armin Ronacher -Artem -Arun Babu Neelicattu -Ashley Manton -Ashwin Ramaswami -atse -Atsushi Odagiri -Avinash Karhana -Avner Cohen -Awit (Ah-Wit) Ghirmai -Baptiste Mispelon -Barney Gale -barneygale -Bartek Ogryczak -Bastian Venthur -Ben Bodenmiller -Ben Darnell -Ben Hoyt -Ben Mares -Ben Rosser -Bence Nagy -Benjamin Peterson -Benjamin VanEvery -Benoit Pierre -Berker Peksag -Bernard -Bernard Tyers -Bernardo B. Marques -Bernhard M. Wiedemann -Bertil Hatt -Bhavam Vidyarthi -Blazej Michalik -Bogdan Opanchuk -BorisZZZ -Brad Erickson -Bradley Ayers -Brandon L. Reiss -Brandt Bucher -Brett Randall -Brett Rosen -Brian Cristante -Brian Rosner -briantracy -BrownTruck -Bruno Oliveira -Bruno Renié -Bruno S -Bstrdsmkr -Buck Golemon -burrows -Bussonnier Matthias -bwoodsend -c22 -Caleb Martinez -Calvin Smith -Carl Meyer -Carlos Liam -Carol Willing -Carter Thayer -Cass -Chandrasekhar Atina -Chih-Hsuan Yen -Chris Brinker -Chris Hunt -Chris Jerdonek -Chris Kuehl -Chris McDonough -Chris Pawley -Chris Pryer -Chris Wolfe -Christian Clauss -Christian Heimes -Christian Oudard -Christoph Reiter -Christopher Hunt -Christopher Snyder -cjc7373 -Clark Boylan -Claudio Jolowicz -Clay McClure -Cody -Cody Soyland -Colin Watson -Collin Anderson -Connor Osborn -Cooper Lees -Cooper Ry Lees -Cory Benfield -Cory Wright -Craig Kerstiens -Cristian Sorinel -Cristina -Cristina Muñoz -Curtis Doty -cytolentino -Daan De Meyer -Damian -Damian Quiroga -Damian Shaw -Dan Black -Dan Savilonis -Dan Sully -Dane Hillard -daniel -Daniel Collins -Daniel Hahler -Daniel Holth -Daniel Jost -Daniel Katz -Daniel Shaulov -Daniele Esposti -Daniele Nicolodi -Daniele Procida -Daniil Konovalenko -Danny Hermes -Danny McClanahan -Darren Kavanagh -Dav Clark -Dave Abrahams -Dave Jones -David Aguilar -David Black -David Bordeynik -David Caro -David D Lowe -David Evans -David Hewitt -David Linke -David Poggi -David Pursehouse -David Runge -David Tucker -David Wales -Davidovich -ddelange -Deepak Sharma -Deepyaman Datta -Denise Yu -dependabot[bot] -derwolfe -Desetude -Devesh Kumar Singh -Diego Caraballo -Diego Ramirez -DiegoCaraballo -Dimitri Merejkowsky -Dimitri Papadopoulos -Dirk Stolle -Dmitry Gladkov -Dmitry Volodin -Domen Kožar -Dominic Davis-Foster -Donald Stufft -Dongweiming -doron zarhi -Dos Moonen -Douglas Thor -DrFeathers -Dustin Ingram -Dwayne Bailey -Ed Morley -Edgar Ramírez -Ee Durbin -Eitan Adler -ekristina -elainechan -Eli Schwartz -Elisha Hollander -Ellen Marie Dash -Emil Burzo -Emil Styrke -Emmanuel Arias -Endoh Takanao -enoch -Erdinc Mutlu -Eric Cousineau -Eric Gillingham -Eric Hanchrow -Eric Hopper -Erik M. Bray -Erik Rose -Erwin Janssen -Eugene Vereshchagin -everdimension -Federico -Felipe Peter -Felix Yan -fiber-space -Filip Kokosiński -Filipe Laíns -Finn Womack -finnagin -Florian Briand -Florian Rathgeber -Francesco -Francesco Montesano -Frost Ming -Gabriel Curio -Gabriel de Perthuis -Garry Polley -gavin -gdanielson -Geoffrey Sneddon -George Song -Georgi Valkov -Georgy Pchelkin -ghost -Giftlin Rajaiah -gizmoguy1 -gkdoc -Godefroid Chapelle -Gopinath M -GOTO Hayato -gousaiyang -gpiks -Greg Roodt -Greg Ward -Guilherme Espada -Guillaume Seguin -gutsytechster -Guy Rozendorn -Guy Tuval -gzpan123 -Hanjun Kim -Hari Charan -Harsh Vardhan -harupy -Harutaka Kawamura -hauntsaninja -Henrich Hartzer -Henry Schreiner -Herbert Pfennig -Holly Stotelmyer -Honnix -Hsiaoming Yang -Hugo Lopes Tavares -Hugo van Kemenade -Hugues Bruant -Hynek Schlawack -Ian Bicking -Ian Cordasco -Ian Lee -Ian Stapleton Cordasco -Ian Wienand -Igor Kuzmitshov -Igor Sobreira -Ilan Schnell -Illia Volochii -Ilya Baryshev -Inada Naoki -Ionel Cristian Mărieș -Ionel Maries Cristian -Itamar Turner-Trauring -Ivan Pozdeev -Jacob Kim -Jacob Walls -Jaime Sanz -jakirkham -Jakub Kuczys -Jakub Stasiak -Jakub Vysoky -Jakub Wilk -James Cleveland -James Curtin -James Firth -James Gerity -James Polley -Jan Pokorný -Jannis Leidel -Jarek Potiuk -jarondl -Jason Curtis -Jason R. Coombs -JasonMo -JasonMo1 -Jay Graves -Jean-Christophe Fillion-Robin -Jeff Barber -Jeff Dairiki -Jeff Widman -Jelmer Vernooij -jenix21 -Jeremy Stanley -Jeremy Zafran -Jesse Rittner -Jiashuo Li -Jim Fisher -Jim Garrison -Jiun Bae -Jivan Amara -Joe Bylund -Joe Michelini -John Paton -John T. Wodder II -John-Scott Atlakson -johnthagen -Jon Banafato -Jon Dufresne -Jon Parise -Jonas Nockert -Jonathan Herbert -Joonatan Partanen -Joost Molenaar -Jorge Niedbalski -Joseph Bylund -Joseph Long -Josh Bronson -Josh Hansen -Josh Schneier -Joshua -Juan Luis Cano Rodríguez -Juanjo Bazán -Judah Rand -Julian Berman -Julian Gethmann -Julien Demoor -Jussi Kukkonen -jwg4 -Jyrki Pulliainen -Kai Chen -Kai Mueller -Kamal Bin Mustafa -kasium -kaustav haldar -keanemind -Keith Maxwell -Kelsey Hightower -Kenneth Belitzky -Kenneth Reitz -Kevin Burke -Kevin Carter -Kevin Frommelt -Kevin R Patterson -Kexuan Sun -Kit Randel -Klaas van Schelven -KOLANICH -kpinc -Krishna Oza -Kumar McMillan -Kurt McKee -Kyle Persohn -lakshmanaram -Laszlo Kiss-Kollar -Laurent Bristiel -Laurent LAPORTE -Laurie O -Laurie Opperman -layday -Leon Sasson -Lev Givon -Lincoln de Sousa -Lipis -lorddavidiii -Loren Carvalho -Lucas Cimon -Ludovic Gasc -Lukas Geiger -Lukas Juhrich -Luke Macken -Luo Jiebin -luojiebin -luz.paz -László Kiss Kollár -M00nL1ght -Marc Abramowitz -Marc Tamlyn -Marcus Smith -Mariatta -Mark Kohler -Mark Williams -Markus Hametner -Martey Dodoo -Martin Fischer -Martin Häcker -Martin Pavlasek -Masaki -Masklinn -Matej Stuchlik -Mathew Jennings -Mathieu Bridon -Mathieu Kniewallner -Matt Bacchi -Matt Good -Matt Maker -Matt Robenolt -matthew -Matthew Einhorn -Matthew Feickert -Matthew Gilliard -Matthew Iversen -Matthew Treinish -Matthew Trumbell -Matthew Willson -Matthias Bussonnier -mattip -Maurits van Rees -Max W Chase -Maxim Kurnikov -Maxime Rouyrre -mayeut -mbaluna -mdebi -memoselyk -meowmeowcat -Michael -Michael Aquilina -Michael E. Karpeles -Michael Klich -Michael Mintz -Michael Williamson -michaelpacer -Michał Górny -Mickaël Schoentgen -Miguel Araujo Perez -Mihir Singh -Mike -Mike Hendricks -Min RK -MinRK -Miro Hrončok -Monica Baluna -montefra -Monty Taylor -Muha Ajjan‮ -Nadav Wexler -Nahuel Ambrosini -Nate Coraor -Nate Prewitt -Nathan Houghton -Nathaniel J. Smith -Nehal J Wani -Neil Botelho -Nguyễn Gia Phong -Nicholas Serra -Nick Coghlan -Nick Stenning -Nick Timkovich -Nicolas Bock -Nicole Harris -Nikhil Benesch -Nikhil Ladha -Nikita Chepanov -Nikolay Korolev -Nipunn Koorapati -Nitesh Sharma -Niyas Sait -Noah -Noah Gorny -Nowell Strite -NtaleGrey -nvdv -OBITORASU -Ofek Lev -ofrinevo -Oliver Freund -Oliver Jeeves -Oliver Mannion -Oliver Tonnhofer -Olivier Girardot -Olivier Grisel -Ollie Rutherfurd -OMOTO Kenji -Omry Yadan -onlinejudge95 -Oren Held -Oscar Benjamin -Oz N Tiram -Pachwenko -Patrick Dubroy -Patrick Jenkins -Patrick Lawson -patricktokeeffe -Patrik Kopkan -Paul Ganssle -Paul Kehrer -Paul Moore -Paul Nasrat -Paul Oswald -Paul van der Linden -Paulus Schoutsen -Pavel Safronov -Pavithra Eswaramoorthy -Pawel Jasinski -Paweł Szramowski -Pekka Klärck -Peter Gessler -Peter Lisák -Peter Waller -petr-tik -Phaneendra Chiruvella -Phil Elson -Phil Freo -Phil Pennock -Phil Whelan -Philip Jägenstedt -Philip Molloy -Philippe Ombredanne -Pi Delport -Pierre-Yves Rofes -Pieter Degroote -pip -Prabakaran Kumaresshan -Prabhjyotsing Surjit Singh Sodhi -Prabhu Marappan -Pradyun Gedam -Prashant Sharma -Pratik Mallya -pre-commit-ci[bot] -Preet Thakkar -Preston Holmes -Przemek Wrzos -Pulkit Goyal -q0w -Qiangning Hong -Quentin Lee -Quentin Pradet -R. David Murray -Rafael Caricio -Ralf Schmitt -Razzi Abuissa -rdb -Reece Dunham -Remi Rampin -Rene Dudfield -Riccardo Magliocchetti -Riccardo Schirone -Richard Jones -Richard Si -Ricky Ng-Adam -Rishi -RobberPhex -Robert Collins -Robert McGibbon -Robert Pollak -Robert T. McGibbon -robin elisha robinson -Roey Berman -Rohan Jain -Roman Bogorodskiy -Roman Donchenko -Romuald Brunet -ronaudinho -Ronny Pfannschmidt -Rory McCann -Ross Brattain -Roy Wellington Ⅳ -Ruairidh MacLeod -Russell Keith-Magee -Ryan Shepherd -Ryan Wooden -ryneeverett -Sachi King -Salvatore Rinchiera -sandeepkiran-js -Sander Van Balen -Savio Jomton -schlamar -Scott Kitterman -Sean -seanj -Sebastian Jordan -Sebastian Schaetz -Segev Finer -SeongSoo Cho -Sergey Vasilyev -Seth Michael Larson -Seth Woodworth -Shahar Epstein -Shantanu -shireenrao -Shivansh-007 -Shlomi Fish -Shovan Maity -Simeon Visser -Simon Cross -Simon Pichugin -sinoroc -sinscary -snook92 -socketubs -Sorin Sbarnea -Srinivas Nyayapati -Stavros Korokithakis -Stefan Scherfke -Stefano Rivera -Stephan Erb -Stephen Rosen -stepshal -Steve (Gadget) Barnes -Steve Barnes -Steve Dower -Steve Kowalik -Steven Myint -Steven Silvester -stonebig -studioj -Stéphane Bidoul -Stéphane Bidoul (ACSONE) -Stéphane Klein -Sumana Harihareswara -Surbhi Sharma -Sviatoslav Sydorenko -Swat009 -Sylvain -Takayuki SHIMIZUKAWA -Taneli Hukkinen -tbeswick -Thiago -Thijs Triemstra -Thomas Fenzl -Thomas Grainger -Thomas Guettler -Thomas Johansson -Thomas Kluyver -Thomas Smith -Thomas VINCENT -Tim D. Smith -Tim Gates -Tim Harder -Tim Heap -tim smith -tinruufu -Tobias Hermann -Tom Forbes -Tom Freudenheim -Tom V -Tomas Hrnciar -Tomas Orsava -Tomer Chachamu -Tommi Enenkel | AnB -Tomáš Hrnčiar -Tony Beswick -Tony Narlock -Tony Zhaocheng Tan -TonyBeswick -toonarmycaptain -Toshio Kuratomi -toxinu -Travis Swicegood -Tushar Sadhwani -Tzu-ping Chung -Valentin Haenel -Victor Stinner -victorvpaulo -Vikram - Google -Viktor Szépe -Ville Skyttä -Vinay Sajip -Vincent Philippon -Vinicyus Macedo -Vipul Kumar -Vitaly Babiy -Vladimir Rutsky -W. Trevor King -Wil Tan -Wilfred Hughes -William Edwards -William ML Leslie -William T Olson -William Woodruff -Wilson Mo -wim glenn -Winson Luk -Wolfgang Maier -Wu Zhenyu -XAMES3 -Xavier Fernandez -xoviat -xtreak -YAMAMOTO Takashi -Yen Chi Hsuan -Yeray Diaz Diaz -Yoval P -Yu Jian -Yuan Jing Vincent Yan -Yusuke Hayashi -Zearin -Zhiping Deng -ziebam -Zvezdan Petkovic -Łukasz Langa -Роман Донченко -Семён Марьясин -‮rekcäH nitraM‮ diff --git a/venv/lib/python3.11/site-packages/pip-23.3.1.dist-info/INSTALLER b/venv/lib/python3.11/site-packages/pip-23.3.1.dist-info/INSTALLER deleted file mode 100644 index a1b589e..0000000 --- a/venv/lib/python3.11/site-packages/pip-23.3.1.dist-info/INSTALLER +++ /dev/null @@ -1 +0,0 @@ -pip diff --git a/venv/lib/python3.11/site-packages/pip-23.3.1.dist-info/LICENSE.txt b/venv/lib/python3.11/site-packages/pip-23.3.1.dist-info/LICENSE.txt deleted file mode 100644 index 8e7b65e..0000000 --- a/venv/lib/python3.11/site-packages/pip-23.3.1.dist-info/LICENSE.txt +++ /dev/null @@ -1,20 +0,0 @@ -Copyright (c) 2008-present The pip developers (see AUTHORS.txt file) - -Permission is hereby granted, free of charge, to any person obtaining -a copy of this software and associated documentation files (the -"Software"), to deal in the Software without restriction, including -without limitation the rights to use, copy, modify, merge, publish, -distribute, sublicense, and/or sell copies of the Software, and to -permit persons to whom the Software is furnished to do so, subject to -the following conditions: - -The above copyright notice and this permission notice shall be -included in all copies or substantial portions of the Software. - -THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, -EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF -MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND -NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE -LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION -OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION -WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. diff --git a/venv/lib/python3.11/site-packages/pip-23.3.1.dist-info/METADATA b/venv/lib/python3.11/site-packages/pip-23.3.1.dist-info/METADATA deleted file mode 100644 index f631c19..0000000 --- a/venv/lib/python3.11/site-packages/pip-23.3.1.dist-info/METADATA +++ /dev/null @@ -1,88 +0,0 @@ -Metadata-Version: 2.1 -Name: pip -Version: 23.3.1 -Summary: The PyPA recommended tool for installing Python packages. -Home-page: https://pip.pypa.io/ -Author: The pip developers -Author-email: distutils-sig@python.org -License: MIT -Project-URL: Documentation, https://pip.pypa.io -Project-URL: Source, https://github.com/pypa/pip -Project-URL: Changelog, https://pip.pypa.io/en/stable/news/ -Classifier: Development Status :: 5 - Production/Stable -Classifier: Intended Audience :: Developers -Classifier: License :: OSI Approved :: MIT License -Classifier: Topic :: Software Development :: Build Tools -Classifier: Programming Language :: Python -Classifier: Programming Language :: Python :: 3 -Classifier: Programming Language :: Python :: 3 :: Only -Classifier: Programming Language :: Python :: 3.7 -Classifier: Programming Language :: Python :: 3.8 -Classifier: Programming Language :: Python :: 3.9 -Classifier: Programming Language :: Python :: 3.10 -Classifier: Programming Language :: Python :: 3.11 -Classifier: Programming Language :: Python :: 3.12 -Classifier: Programming Language :: Python :: Implementation :: CPython -Classifier: Programming Language :: Python :: Implementation :: PyPy -Requires-Python: >=3.7 -License-File: LICENSE.txt -License-File: AUTHORS.txt - -pip - The Python Package Installer -================================== - -.. image:: https://img.shields.io/pypi/v/pip.svg - :target: https://pypi.org/project/pip/ - :alt: PyPI - -.. image:: https://img.shields.io/pypi/pyversions/pip - :target: https://pypi.org/project/pip - :alt: PyPI - Python Version - -.. image:: https://readthedocs.org/projects/pip/badge/?version=latest - :target: https://pip.pypa.io/en/latest - :alt: Documentation - -pip is the `package installer`_ for Python. You can use pip to install packages from the `Python Package Index`_ and other indexes. - -Please take a look at our documentation for how to install and use pip: - -* `Installation`_ -* `Usage`_ - -We release updates regularly, with a new version every 3 months. Find more details in our documentation: - -* `Release notes`_ -* `Release process`_ - -If you find bugs, need help, or want to talk to the developers, please use our mailing lists or chat rooms: - -* `Issue tracking`_ -* `Discourse channel`_ -* `User IRC`_ - -If you want to get involved head over to GitHub to get the source code, look at our development documentation and feel free to jump on the developer mailing lists and chat rooms: - -* `GitHub page`_ -* `Development documentation`_ -* `Development IRC`_ - -Code of Conduct ---------------- - -Everyone interacting in the pip project's codebases, issue trackers, chat -rooms, and mailing lists is expected to follow the `PSF Code of Conduct`_. - -.. _package installer: https://packaging.python.org/guides/tool-recommendations/ -.. _Python Package Index: https://pypi.org -.. _Installation: https://pip.pypa.io/en/stable/installation/ -.. _Usage: https://pip.pypa.io/en/stable/ -.. _Release notes: https://pip.pypa.io/en/stable/news.html -.. _Release process: https://pip.pypa.io/en/latest/development/release-process/ -.. _GitHub page: https://github.com/pypa/pip -.. _Development documentation: https://pip.pypa.io/en/latest/development -.. _Issue tracking: https://github.com/pypa/pip/issues -.. _Discourse channel: https://discuss.python.org/c/packaging -.. _User IRC: https://kiwiirc.com/nextclient/#ircs://irc.libera.chat:+6697/pypa -.. _Development IRC: https://kiwiirc.com/nextclient/#ircs://irc.libera.chat:+6697/pypa-dev -.. _PSF Code of Conduct: https://github.com/pypa/.github/blob/main/CODE_OF_CONDUCT.md diff --git a/venv/lib/python3.11/site-packages/pip-23.3.1.dist-info/RECORD b/venv/lib/python3.11/site-packages/pip-23.3.1.dist-info/RECORD deleted file mode 100644 index 333eee8..0000000 --- a/venv/lib/python3.11/site-packages/pip-23.3.1.dist-info/RECORD +++ /dev/null @@ -1,1011 +0,0 @@ -../../../bin/pip,sha256=obI6gfMv0gSqwt3b63R2uzpPNS-SXpTRBEMdN2cP4K4,296 -../../../bin/pip3,sha256=obI6gfMv0gSqwt3b63R2uzpPNS-SXpTRBEMdN2cP4K4,296 -../../../bin/pip3.11,sha256=obI6gfMv0gSqwt3b63R2uzpPNS-SXpTRBEMdN2cP4K4,296 -pip-23.3.1.dist-info/AUTHORS.txt,sha256=HOVK0m4Fk7uZrqt9MhiBlBTdmUbMIxXJziTWeMc_Jxc,10253 -pip-23.3.1.dist-info/INSTALLER,sha256=zuuue4knoyJ-UwPPXg8fezS7VCrXJQrAP7zeNuwvFQg,4 -pip-23.3.1.dist-info/LICENSE.txt,sha256=Y0MApmnUmurmWxLGxIySTFGkzfPR_whtw0VtyLyqIQQ,1093 -pip-23.3.1.dist-info/METADATA,sha256=ePd4oJwtCOg7e5hjeRczRRgaxHUSasxlmRPNHMtKToE,3540 -pip-23.3.1.dist-info/RECORD,, -pip-23.3.1.dist-info/REQUESTED,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0 -pip-23.3.1.dist-info/WHEEL,sha256=Xo9-1PvkuimrydujYJAjF7pCkriuXBpUPEjma1nZyJ0,92 -pip-23.3.1.dist-info/entry_points.txt,sha256=xg35gOct0aY8S3ftLtweJ0uw3KBAIVyW4k-0Jx1rkNE,125 -pip-23.3.1.dist-info/top_level.txt,sha256=zuuue4knoyJ-UwPPXg8fezS7VCrXJQrAP7zeNuwvFQg,4 -pip/__init__.py,sha256=MSbZQYwV5U4mAXP2fBQh70QhM71N-1vh7T4CRREqVog,357 -pip/__main__.py,sha256=WzbhHXTbSE6gBY19mNN9m4s5o_365LOvTYSgqgbdBhE,854 -pip/__pip-runner__.py,sha256=EnrfKmKMzWAdqg_JicLCOP9Y95Ux7zHh4ObvqLtQcjo,1444 -pip/__pycache__/__init__.cpython-311.pyc,, -pip/__pycache__/__main__.cpython-311.pyc,, -pip/__pycache__/__pip-runner__.cpython-311.pyc,, -pip/_internal/__init__.py,sha256=iqZ5-YQsQV08tkUc7L806Reop6tguLFWf70ySF6be0Y,515 -pip/_internal/__pycache__/__init__.cpython-311.pyc,, -pip/_internal/__pycache__/build_env.cpython-311.pyc,, -pip/_internal/__pycache__/cache.cpython-311.pyc,, -pip/_internal/__pycache__/configuration.cpython-311.pyc,, -pip/_internal/__pycache__/exceptions.cpython-311.pyc,, -pip/_internal/__pycache__/main.cpython-311.pyc,, -pip/_internal/__pycache__/pyproject.cpython-311.pyc,, -pip/_internal/__pycache__/self_outdated_check.cpython-311.pyc,, -pip/_internal/__pycache__/wheel_builder.cpython-311.pyc,, -pip/_internal/build_env.py,sha256=1ESpqw0iupS_K7phZK5zshVE5Czy9BtGLFU4W6Enva8,10243 -pip/_internal/cache.py,sha256=uiYD-9F0Bv1C8ZyWE85lpzDmQf7hcUkgL99GmI8I41Q,10370 -pip/_internal/cli/__init__.py,sha256=FkHBgpxxb-_gd6r1FjnNhfMOzAUYyXoXKJ6abijfcFU,132 -pip/_internal/cli/__pycache__/__init__.cpython-311.pyc,, -pip/_internal/cli/__pycache__/autocompletion.cpython-311.pyc,, -pip/_internal/cli/__pycache__/base_command.cpython-311.pyc,, -pip/_internal/cli/__pycache__/cmdoptions.cpython-311.pyc,, -pip/_internal/cli/__pycache__/command_context.cpython-311.pyc,, -pip/_internal/cli/__pycache__/main.cpython-311.pyc,, -pip/_internal/cli/__pycache__/main_parser.cpython-311.pyc,, -pip/_internal/cli/__pycache__/parser.cpython-311.pyc,, -pip/_internal/cli/__pycache__/progress_bars.cpython-311.pyc,, -pip/_internal/cli/__pycache__/req_command.cpython-311.pyc,, -pip/_internal/cli/__pycache__/spinners.cpython-311.pyc,, -pip/_internal/cli/__pycache__/status_codes.cpython-311.pyc,, -pip/_internal/cli/autocompletion.py,sha256=_br_5NgSxSuvPjMF0MLHzS5s6BpSkQAQHKrLK89VauM,6690 -pip/_internal/cli/base_command.py,sha256=iuVWGa2oTq7gBReo0er3Z0tXJ2oqBIC6QjDHcnDhKXY,8733 -pip/_internal/cli/cmdoptions.py,sha256=fAi5GzWuM9mKUesJZO56LcPCVMDtm64c2tC_YUpI1qs,30117 -pip/_internal/cli/command_context.py,sha256=RHgIPwtObh5KhMrd3YZTkl8zbVG-6Okml7YbFX4Ehg0,774 -pip/_internal/cli/main.py,sha256=Uzxt_YD1hIvB1AW5mxt6IVcht5G712AtMqdo51UMhmQ,2816 -pip/_internal/cli/main_parser.py,sha256=laDpsuBDl6kyfywp9eMMA9s84jfH2TJJn-vmL0GG90w,4338 -pip/_internal/cli/parser.py,sha256=o4esYgG-rvPsf6FBpF3fSLGHa4ndDvJtwxBgeckGyfI,10801 -pip/_internal/cli/progress_bars.py,sha256=So4mPoSjXkXiSHiTzzquH3VVyVD_njXlHJSExYPXAow,1968 -pip/_internal/cli/req_command.py,sha256=c7_XHABnXmD3_qlK9-r37KqdKBAcgmVKvQ2WcTrNLfc,18369 -pip/_internal/cli/spinners.py,sha256=hIJ83GerdFgFCdobIA23Jggetegl_uC4Sp586nzFbPE,5118 -pip/_internal/cli/status_codes.py,sha256=sEFHUaUJbqv8iArL3HAtcztWZmGOFX01hTesSytDEh0,116 -pip/_internal/commands/__init__.py,sha256=5oRO9O3dM2vGuh0bFw4HOVletryrz5HHMmmPWwJrH9U,3882 -pip/_internal/commands/__pycache__/__init__.cpython-311.pyc,, -pip/_internal/commands/__pycache__/cache.cpython-311.pyc,, -pip/_internal/commands/__pycache__/check.cpython-311.pyc,, -pip/_internal/commands/__pycache__/completion.cpython-311.pyc,, -pip/_internal/commands/__pycache__/configuration.cpython-311.pyc,, -pip/_internal/commands/__pycache__/debug.cpython-311.pyc,, -pip/_internal/commands/__pycache__/download.cpython-311.pyc,, -pip/_internal/commands/__pycache__/freeze.cpython-311.pyc,, -pip/_internal/commands/__pycache__/hash.cpython-311.pyc,, -pip/_internal/commands/__pycache__/help.cpython-311.pyc,, -pip/_internal/commands/__pycache__/index.cpython-311.pyc,, -pip/_internal/commands/__pycache__/inspect.cpython-311.pyc,, -pip/_internal/commands/__pycache__/install.cpython-311.pyc,, -pip/_internal/commands/__pycache__/list.cpython-311.pyc,, -pip/_internal/commands/__pycache__/search.cpython-311.pyc,, -pip/_internal/commands/__pycache__/show.cpython-311.pyc,, -pip/_internal/commands/__pycache__/uninstall.cpython-311.pyc,, -pip/_internal/commands/__pycache__/wheel.cpython-311.pyc,, -pip/_internal/commands/cache.py,sha256=LfPA8wNcgZtjiI5faeFFCR2Zp-ugaj7XX--FmKxx4_4,7952 -pip/_internal/commands/check.py,sha256=Rb13Q28yoLh0j1gpx5SU0jlResNct21eQCRsnaO9xKA,1782 -pip/_internal/commands/completion.py,sha256=HT4lD0bgsflHq2IDgYfiEdp7IGGtE7s6MgI3xn0VQEw,4287 -pip/_internal/commands/configuration.py,sha256=NB5uf8HIX8-li95YLoZO09nALIWlLCHDF5aifSKcBn8,9815 -pip/_internal/commands/debug.py,sha256=L15rfN8DwORQln-QW3ihBaVdCfV7Iba-lwlcyw1f_Vk,6854 -pip/_internal/commands/download.py,sha256=e4hw088zGo26WmJaMIRvCniLlLmoOjqolGyfHjsCkCQ,5335 -pip/_internal/commands/freeze.py,sha256=2qjQrH9KWi5Roav0CuR7vc7hWm4uOi_0l6tp3ESKDHM,3172 -pip/_internal/commands/hash.py,sha256=EVVOuvGtoPEdFi8SNnmdqlCQrhCxV-kJsdwtdcCnXGQ,1703 -pip/_internal/commands/help.py,sha256=gcc6QDkcgHMOuAn5UxaZwAStsRBrnGSn_yxjS57JIoM,1132 -pip/_internal/commands/index.py,sha256=cGQVSA5dAs7caQ9sz4kllYvaI4ZpGiq1WhCgaImXNSA,4793 -pip/_internal/commands/inspect.py,sha256=2wSPt9yfr3r6g-s2S5L6PvRtaHNVyb4TuodMStJ39cw,3188 -pip/_internal/commands/install.py,sha256=KTHT8EASlPfbNx428tcvnGhN8D9jlfBwcRa5lxEhFsA,28920 -pip/_internal/commands/list.py,sha256=7wRUUmdyyOknl-WZYbO_LtFQxHlWod3pjOY9yYH435o,12450 -pip/_internal/commands/search.py,sha256=sbBZiARRc050QquOKcCvOr2K3XLsoYebLKZGRi__iUI,5697 -pip/_internal/commands/show.py,sha256=t5jia4zcYJRJZy4U_Von7zMl03hJmmcofj6oDNTnj7Y,6419 -pip/_internal/commands/uninstall.py,sha256=OIqO9tqadY8kM4HwhFf1Q62fUIp7v8KDrTRo8yWMz7Y,3886 -pip/_internal/commands/wheel.py,sha256=CSnX8Pmf1oPCnd7j7bn1_f58G9KHNiAblvVJ5zykN-A,6476 -pip/_internal/configuration.py,sha256=i_dePJKndPAy7hf48Sl6ZuPyl3tFPCE67z0SNatwuwE,13839 -pip/_internal/distributions/__init__.py,sha256=Hq6kt6gXBgjNit5hTTWLAzeCNOKoB-N0pGYSqehrli8,858 -pip/_internal/distributions/__pycache__/__init__.cpython-311.pyc,, -pip/_internal/distributions/__pycache__/base.cpython-311.pyc,, -pip/_internal/distributions/__pycache__/installed.cpython-311.pyc,, -pip/_internal/distributions/__pycache__/sdist.cpython-311.pyc,, -pip/_internal/distributions/__pycache__/wheel.cpython-311.pyc,, -pip/_internal/distributions/base.py,sha256=oRSEvnv2ZjBnargamnv2fcJa1n6gUDKaW0g6CWSEpWs,1743 -pip/_internal/distributions/installed.py,sha256=QinHFbWAQ8oE0pbD8MFZWkwlnfU1QYTccA1vnhrlYOU,842 -pip/_internal/distributions/sdist.py,sha256=4K3V0VNMllHbBzCJibjwd_tylUKpmIdu2AQyhplvCQo,6709 -pip/_internal/distributions/wheel.py,sha256=-ma3sOtUQj0AxXCEb6_Fhmjl3nh4k3A0HC2taAb2N-4,1277 -pip/_internal/exceptions.py,sha256=LyTVY2dANx-i_TEk5Yr9YcwUtiy0HOEFCAQq1F_46co,23737 -pip/_internal/index/__init__.py,sha256=vpt-JeTZefh8a-FC22ZeBSXFVbuBcXSGiILhQZJaNpQ,30 -pip/_internal/index/__pycache__/__init__.cpython-311.pyc,, -pip/_internal/index/__pycache__/collector.cpython-311.pyc,, -pip/_internal/index/__pycache__/package_finder.cpython-311.pyc,, -pip/_internal/index/__pycache__/sources.cpython-311.pyc,, -pip/_internal/index/collector.py,sha256=3OmYZ3tCoRPGOrELSgQWG-03M-bQHa2-VCA3R_nJAaU,16504 -pip/_internal/index/package_finder.py,sha256=uA354-mHjHvTwxDmk9HvpAkq_7KyGvEd7_9aZFqu0HY,37889 -pip/_internal/index/sources.py,sha256=7jw9XSeeQA5K-H4I5a5034Ks2gkQqm4zPXjrhwnP1S4,6556 -pip/_internal/locations/__init__.py,sha256=Dh8LJWG8LRlDK4JIj9sfRF96TREzE--N_AIlx7Tqoe4,15365 -pip/_internal/locations/__pycache__/__init__.cpython-311.pyc,, -pip/_internal/locations/__pycache__/_distutils.cpython-311.pyc,, -pip/_internal/locations/__pycache__/_sysconfig.cpython-311.pyc,, -pip/_internal/locations/__pycache__/base.cpython-311.pyc,, -pip/_internal/locations/_distutils.py,sha256=DXL6H3xERLF76BjcYanV4j-4Sw-qcPdO2qeZhLN30WQ,6102 -pip/_internal/locations/_sysconfig.py,sha256=jyNVtUfMIf0mtyY-Xp1m9yQ8iwECozSVVFmjkN9a2yw,7680 -pip/_internal/locations/base.py,sha256=RQiPi1d4FVM2Bxk04dQhXZ2PqkeljEL2fZZ9SYqIQ78,2556 -pip/_internal/main.py,sha256=r-UnUe8HLo5XFJz8inTcOOTiu_sxNhgHb6VwlGUllOI,340 -pip/_internal/metadata/__init__.py,sha256=9pU3W3s-6HtjFuYhWcLTYVmSaziklPv7k2x8p7X1GmA,4339 -pip/_internal/metadata/__pycache__/__init__.cpython-311.pyc,, -pip/_internal/metadata/__pycache__/_json.cpython-311.pyc,, -pip/_internal/metadata/__pycache__/base.cpython-311.pyc,, -pip/_internal/metadata/__pycache__/pkg_resources.cpython-311.pyc,, -pip/_internal/metadata/_json.py,sha256=BTkWfFDrWFwuSodImjtbAh8wCL3isecbnjTb5E6UUDI,2595 -pip/_internal/metadata/base.py,sha256=l3Wgku4xlgr8s4p6fS-3qQ4QKOpPbWLRwi5d9omEFG4,25907 -pip/_internal/metadata/importlib/__init__.py,sha256=jUUidoxnHcfITHHaAWG1G2i5fdBYklv_uJcjo2x7VYE,135 -pip/_internal/metadata/importlib/__pycache__/__init__.cpython-311.pyc,, -pip/_internal/metadata/importlib/__pycache__/_compat.cpython-311.pyc,, -pip/_internal/metadata/importlib/__pycache__/_dists.cpython-311.pyc,, -pip/_internal/metadata/importlib/__pycache__/_envs.cpython-311.pyc,, -pip/_internal/metadata/importlib/_compat.py,sha256=GAe_prIfCE4iUylrnr_2dJRlkkBVRUbOidEoID7LPoE,1882 -pip/_internal/metadata/importlib/_dists.py,sha256=UPl1wUujFqiwiltRJ1tMF42WRINO1sSpNNlYQ2mX0mk,8297 -pip/_internal/metadata/importlib/_envs.py,sha256=XTaFIYERP2JF0QUZuPx2ETiugXbPEcZ8q8ZKeht6Lpc,7456 -pip/_internal/metadata/pkg_resources.py,sha256=opjw4IBSqHvie6sXJ_cbT42meygoPEUfNURJuWZY7sk,10035 -pip/_internal/models/__init__.py,sha256=3DHUd_qxpPozfzouoqa9g9ts1Czr5qaHfFxbnxriepM,63 -pip/_internal/models/__pycache__/__init__.cpython-311.pyc,, -pip/_internal/models/__pycache__/candidate.cpython-311.pyc,, -pip/_internal/models/__pycache__/direct_url.cpython-311.pyc,, -pip/_internal/models/__pycache__/format_control.cpython-311.pyc,, -pip/_internal/models/__pycache__/index.cpython-311.pyc,, -pip/_internal/models/__pycache__/installation_report.cpython-311.pyc,, -pip/_internal/models/__pycache__/link.cpython-311.pyc,, -pip/_internal/models/__pycache__/scheme.cpython-311.pyc,, -pip/_internal/models/__pycache__/search_scope.cpython-311.pyc,, -pip/_internal/models/__pycache__/selection_prefs.cpython-311.pyc,, -pip/_internal/models/__pycache__/target_python.cpython-311.pyc,, -pip/_internal/models/__pycache__/wheel.cpython-311.pyc,, -pip/_internal/models/candidate.py,sha256=6pcABsaR7CfIHlbJbr2_kMkVJFL_yrYjTx6SVWUnCPQ,990 -pip/_internal/models/direct_url.py,sha256=EepBxI97j7wSZ3AmRETYyVTmR9NoTas15vc8popxVTg,6931 -pip/_internal/models/format_control.py,sha256=DJpMYjxeYKKQdwNcML2_F0vtAh-qnKTYe-CpTxQe-4g,2520 -pip/_internal/models/index.py,sha256=tYnL8oxGi4aSNWur0mG8DAP7rC6yuha_MwJO8xw0crI,1030 -pip/_internal/models/installation_report.py,sha256=zRVZoaz-2vsrezj_H3hLOhMZCK9c7TbzWgC-jOalD00,2818 -pip/_internal/models/link.py,sha256=6OEk3bt41WU7QZoiyuoVPGsKOU-J_BbDDhouKbIXm0Y,20819 -pip/_internal/models/scheme.py,sha256=3EFQp_ICu_shH1-TBqhl0QAusKCPDFOlgHFeN4XowWs,738 -pip/_internal/models/search_scope.py,sha256=ASVyyZxiJILw7bTIVVpJx8J293M3Hk5F33ilGn0e80c,4643 -pip/_internal/models/selection_prefs.py,sha256=KZdi66gsR-_RUXUr9uejssk3rmTHrQVJWeNA2sV-VSY,1907 -pip/_internal/models/target_python.py,sha256=34EkorrMuRvRp-bjqHKJ-bOO71m9xdjN2b8WWFEC2HU,4272 -pip/_internal/models/wheel.py,sha256=YqazoIZyma_Q1ejFa1C7NHKQRRWlvWkdK96VRKmDBeI,3600 -pip/_internal/network/__init__.py,sha256=jf6Tt5nV_7zkARBrKojIXItgejvoegVJVKUbhAa5Ioc,50 -pip/_internal/network/__pycache__/__init__.cpython-311.pyc,, -pip/_internal/network/__pycache__/auth.cpython-311.pyc,, -pip/_internal/network/__pycache__/cache.cpython-311.pyc,, -pip/_internal/network/__pycache__/download.cpython-311.pyc,, -pip/_internal/network/__pycache__/lazy_wheel.cpython-311.pyc,, -pip/_internal/network/__pycache__/session.cpython-311.pyc,, -pip/_internal/network/__pycache__/utils.cpython-311.pyc,, -pip/_internal/network/__pycache__/xmlrpc.cpython-311.pyc,, -pip/_internal/network/auth.py,sha256=TC-OcW2KU4W6R1hU4qPgQXvVH54adACpZz6sWq-R9NA,20541 -pip/_internal/network/cache.py,sha256=48A971qCzKNFvkb57uGEk7-0xaqPS0HWj2711QNTxkU,3935 -pip/_internal/network/download.py,sha256=HvDDq9bVqaN3jcS3DyVJHP7uTqFzbShdkf7NFSoHfkw,6096 -pip/_internal/network/lazy_wheel.py,sha256=2PXVduYZPCPZkkQFe1J1GbfHJWeCU--FXonGyIfw9eU,7638 -pip/_internal/network/session.py,sha256=uhovd4J7abd0Yr2g426yC4aC6Uw1VKrQfpzalsEBEMw,18607 -pip/_internal/network/utils.py,sha256=6A5SrUJEEUHxbGtbscwU2NpCyz-3ztiDlGWHpRRhsJ8,4073 -pip/_internal/network/xmlrpc.py,sha256=AzQgG4GgS152_cqmGr_Oz2MIXsCal-xfsis7fA7nmU0,1791 -pip/_internal/operations/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0 -pip/_internal/operations/__pycache__/__init__.cpython-311.pyc,, -pip/_internal/operations/__pycache__/check.cpython-311.pyc,, -pip/_internal/operations/__pycache__/freeze.cpython-311.pyc,, -pip/_internal/operations/__pycache__/prepare.cpython-311.pyc,, -pip/_internal/operations/build/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0 -pip/_internal/operations/build/__pycache__/__init__.cpython-311.pyc,, -pip/_internal/operations/build/__pycache__/build_tracker.cpython-311.pyc,, -pip/_internal/operations/build/__pycache__/metadata.cpython-311.pyc,, -pip/_internal/operations/build/__pycache__/metadata_editable.cpython-311.pyc,, -pip/_internal/operations/build/__pycache__/metadata_legacy.cpython-311.pyc,, -pip/_internal/operations/build/__pycache__/wheel.cpython-311.pyc,, -pip/_internal/operations/build/__pycache__/wheel_editable.cpython-311.pyc,, -pip/_internal/operations/build/__pycache__/wheel_legacy.cpython-311.pyc,, -pip/_internal/operations/build/build_tracker.py,sha256=z-H5DOknZdBa3dh2Vq6VBMY5qLYIKmlj2p6CGZK5Lc8,4832 -pip/_internal/operations/build/metadata.py,sha256=9S0CUD8U3QqZeXp-Zyt8HxwU90lE4QrnYDgrqZDzBnc,1422 -pip/_internal/operations/build/metadata_editable.py,sha256=VLL7LvntKE8qxdhUdEJhcotFzUsOSI8NNS043xULKew,1474 -pip/_internal/operations/build/metadata_legacy.py,sha256=o-eU21As175hDC7dluM1fJJ_FqokTIShyWpjKaIpHZw,2198 -pip/_internal/operations/build/wheel.py,sha256=sT12FBLAxDC6wyrDorh8kvcZ1jG5qInCRWzzP-UkJiQ,1075 -pip/_internal/operations/build/wheel_editable.py,sha256=yOtoH6zpAkoKYEUtr8FhzrYnkNHQaQBjWQ2HYae1MQg,1417 -pip/_internal/operations/build/wheel_legacy.py,sha256=C9j6rukgQI1n_JeQLoZGuDdfUwzCXShyIdPTp6edbMQ,3064 -pip/_internal/operations/check.py,sha256=Hgz0wQJ4fGi8aAVfmdShviNc7XM_2j8oMQJUsVv6AqY,6806 -pip/_internal/operations/freeze.py,sha256=uqoeTAf6HOYVMR2UgAT8N85UZoGEVEoQdan_Ao6SOfk,9816 -pip/_internal/operations/install/__init__.py,sha256=mX7hyD2GNBO2mFGokDQ30r_GXv7Y_PLdtxcUv144e-s,51 -pip/_internal/operations/install/__pycache__/__init__.cpython-311.pyc,, -pip/_internal/operations/install/__pycache__/editable_legacy.cpython-311.pyc,, -pip/_internal/operations/install/__pycache__/wheel.cpython-311.pyc,, -pip/_internal/operations/install/editable_legacy.py,sha256=YeR0KadWXw_ZheC1NtAG1qVIEkOgRGHc23x-YtGW7NU,1282 -pip/_internal/operations/install/wheel.py,sha256=a5KnguJ9uQRo7Ikq4YJEno0fFltXYlud-0DpRj3zLr0,27457 -pip/_internal/operations/prepare.py,sha256=NWkGkNOjrnnUbHgJPTms_5usKF0M8JlaHL3nyIHABMk,28155 -pip/_internal/pyproject.py,sha256=ltmrXWaMXjiJHbYyzWplTdBvPYPdKk99GjKuQVypGZU,7161 -pip/_internal/req/__init__.py,sha256=TELFgZOof3lhMmaICVWL9U7PlhXo9OufokbMAJ6J2GI,2738 -pip/_internal/req/__pycache__/__init__.cpython-311.pyc,, -pip/_internal/req/__pycache__/constructors.cpython-311.pyc,, -pip/_internal/req/__pycache__/req_file.cpython-311.pyc,, -pip/_internal/req/__pycache__/req_install.cpython-311.pyc,, -pip/_internal/req/__pycache__/req_set.cpython-311.pyc,, -pip/_internal/req/__pycache__/req_uninstall.cpython-311.pyc,, -pip/_internal/req/constructors.py,sha256=PgLoQlsZ_ErZORw5M1mgnxW5V4mKZC0-gyj_3k4hCe0,19028 -pip/_internal/req/req_file.py,sha256=5PCO4GnDEnUENiFj4vD_1QmAMjHNtvN6HXbETZ9UGok,17872 -pip/_internal/req/req_install.py,sha256=XvoTWTF7STk9EUqIphdOI0ZtQOplw44PIl9TCb-HtXw,35130 -pip/_internal/req/req_set.py,sha256=nM-CetUtESEH31fdugrOl20GV5-pCUYAvu65FwYDJeI,4704 -pip/_internal/req/req_uninstall.py,sha256=m9GlbQ3rzLORTSa6NPFFCmONmC5zTw2lY_0fLOkLYCk,24676 -pip/_internal/resolution/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0 -pip/_internal/resolution/__pycache__/__init__.cpython-311.pyc,, -pip/_internal/resolution/__pycache__/base.cpython-311.pyc,, -pip/_internal/resolution/base.py,sha256=qlmh325SBVfvG6Me9gc5Nsh5sdwHBwzHBq6aEXtKsLA,583 -pip/_internal/resolution/legacy/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0 -pip/_internal/resolution/legacy/__pycache__/__init__.cpython-311.pyc,, -pip/_internal/resolution/legacy/__pycache__/resolver.cpython-311.pyc,, -pip/_internal/resolution/legacy/resolver.py,sha256=th-eTPIvbecfJaUsdrbH1aHQvDV2yCE-RhrrpsJhKbE,24128 -pip/_internal/resolution/resolvelib/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0 -pip/_internal/resolution/resolvelib/__pycache__/__init__.cpython-311.pyc,, -pip/_internal/resolution/resolvelib/__pycache__/base.cpython-311.pyc,, -pip/_internal/resolution/resolvelib/__pycache__/candidates.cpython-311.pyc,, -pip/_internal/resolution/resolvelib/__pycache__/factory.cpython-311.pyc,, -pip/_internal/resolution/resolvelib/__pycache__/found_candidates.cpython-311.pyc,, -pip/_internal/resolution/resolvelib/__pycache__/provider.cpython-311.pyc,, -pip/_internal/resolution/resolvelib/__pycache__/reporter.cpython-311.pyc,, -pip/_internal/resolution/resolvelib/__pycache__/requirements.cpython-311.pyc,, -pip/_internal/resolution/resolvelib/__pycache__/resolver.cpython-311.pyc,, -pip/_internal/resolution/resolvelib/base.py,sha256=jg5COmHLhmBIKOR-4spdJD3jyULYa1BdsqiBu2YJnJ4,5173 -pip/_internal/resolution/resolvelib/candidates.py,sha256=IAcXcBj-LLzJwwfBXFGyhpxir42CMBW64oCc4zEgLYo,21320 -pip/_internal/resolution/resolvelib/factory.py,sha256=FIOXvrdEGo6DMtLF9gqhUd4IQphPUkAYUe8ZIQ0ThMY,31317 -pip/_internal/resolution/resolvelib/found_candidates.py,sha256=hvL3Hoa9VaYo-qEOZkBi2Iqw251UDxPz-uMHVaWmLpE,5705 -pip/_internal/resolution/resolvelib/provider.py,sha256=4t23ivjruqM6hKBX1KpGiTt-M4HGhRcZnGLV0c01K7U,9824 -pip/_internal/resolution/resolvelib/reporter.py,sha256=YFm9hQvz4DFCbjZeFTQ56hTz3Ac-mDBnHkeNRVvMHLY,3100 -pip/_internal/resolution/resolvelib/requirements.py,sha256=SZh98hbSVbHiHBkgjrSLtdrrZB1zqRIUqFdXptS-aVY,6030 -pip/_internal/resolution/resolvelib/resolver.py,sha256=nLJOsVMEVi2gQUVJoUFKMZAeu2f7GRMjGMvNSWyz0Bc,12592 -pip/_internal/self_outdated_check.py,sha256=saxQLB8UzIFtMScquytG10TOTsYVFJQ_mkW1NY-46wE,8378 -pip/_internal/utils/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0 -pip/_internal/utils/__pycache__/__init__.cpython-311.pyc,, -pip/_internal/utils/__pycache__/_jaraco_text.cpython-311.pyc,, -pip/_internal/utils/__pycache__/_log.cpython-311.pyc,, -pip/_internal/utils/__pycache__/appdirs.cpython-311.pyc,, -pip/_internal/utils/__pycache__/compat.cpython-311.pyc,, -pip/_internal/utils/__pycache__/compatibility_tags.cpython-311.pyc,, -pip/_internal/utils/__pycache__/datetime.cpython-311.pyc,, -pip/_internal/utils/__pycache__/deprecation.cpython-311.pyc,, -pip/_internal/utils/__pycache__/direct_url_helpers.cpython-311.pyc,, -pip/_internal/utils/__pycache__/egg_link.cpython-311.pyc,, -pip/_internal/utils/__pycache__/encoding.cpython-311.pyc,, -pip/_internal/utils/__pycache__/entrypoints.cpython-311.pyc,, -pip/_internal/utils/__pycache__/filesystem.cpython-311.pyc,, -pip/_internal/utils/__pycache__/filetypes.cpython-311.pyc,, -pip/_internal/utils/__pycache__/glibc.cpython-311.pyc,, -pip/_internal/utils/__pycache__/hashes.cpython-311.pyc,, -pip/_internal/utils/__pycache__/logging.cpython-311.pyc,, -pip/_internal/utils/__pycache__/misc.cpython-311.pyc,, -pip/_internal/utils/__pycache__/models.cpython-311.pyc,, -pip/_internal/utils/__pycache__/packaging.cpython-311.pyc,, -pip/_internal/utils/__pycache__/setuptools_build.cpython-311.pyc,, -pip/_internal/utils/__pycache__/subprocess.cpython-311.pyc,, -pip/_internal/utils/__pycache__/temp_dir.cpython-311.pyc,, -pip/_internal/utils/__pycache__/unpacking.cpython-311.pyc,, -pip/_internal/utils/__pycache__/urls.cpython-311.pyc,, -pip/_internal/utils/__pycache__/virtualenv.cpython-311.pyc,, -pip/_internal/utils/__pycache__/wheel.cpython-311.pyc,, -pip/_internal/utils/_jaraco_text.py,sha256=yvDGelTVugRayPaOF2k4ab0Ky4d3uOkAfuOQjASjImY,3351 -pip/_internal/utils/_log.py,sha256=-jHLOE_THaZz5BFcCnoSL9EYAtJ0nXem49s9of4jvKw,1015 -pip/_internal/utils/appdirs.py,sha256=swgcTKOm3daLeXTW6v5BUS2Ti2RvEnGRQYH_yDXklAo,1665 -pip/_internal/utils/compat.py,sha256=ACyBfLgj3_XG-iA5omEDrXqDM0cQKzi8h8HRBInzG6Q,1884 -pip/_internal/utils/compatibility_tags.py,sha256=ydin8QG8BHqYRsPY4OL6cmb44CbqXl1T0xxS97VhHkk,5377 -pip/_internal/utils/datetime.py,sha256=m21Y3wAtQc-ji6Veb6k_M5g6A0ZyFI4egchTdnwh-pQ,242 -pip/_internal/utils/deprecation.py,sha256=NKo8VqLioJ4nnXXGmW4KdasxF90EFHkZaHeX1fT08C8,3627 -pip/_internal/utils/direct_url_helpers.py,sha256=6F1tc2rcKaCZmgfVwsE6ObIe_Pux23mUVYA-2D9wCFc,3206 -pip/_internal/utils/egg_link.py,sha256=ZryCchR_yQSCsdsMkCpxQjjLbQxObA5GDtLG0RR5mGc,2118 -pip/_internal/utils/encoding.py,sha256=qqsXDtiwMIjXMEiIVSaOjwH5YmirCaK-dIzb6-XJsL0,1169 -pip/_internal/utils/entrypoints.py,sha256=YlhLTRl2oHBAuqhc-zmL7USS67TPWVHImjeAQHreZTQ,3064 -pip/_internal/utils/filesystem.py,sha256=RhMIXUaNVMGjc3rhsDahWQ4MavvEQDdqXqgq-F6fpw8,5122 -pip/_internal/utils/filetypes.py,sha256=i8XAQ0eFCog26Fw9yV0Yb1ygAqKYB1w9Cz9n0fj8gZU,716 -pip/_internal/utils/glibc.py,sha256=Mesxxgg3BLxheLZx-dSf30b6gKpOgdVXw6W--uHSszQ,3113 -pip/_internal/utils/hashes.py,sha256=MjOigC75z6qoRMkgHiHqot7eqxfwDZSrEflJMPm-bHE,5118 -pip/_internal/utils/logging.py,sha256=fdtuZJ-AKkqwDTANDvGcBEpssL8el7T1jnwk1CnZl3Y,11603 -pip/_internal/utils/misc.py,sha256=96DVNJQIeMi0vWrNp0C0v3xjk2r7Zcay5yDoruIm_Js,23739 -pip/_internal/utils/models.py,sha256=5GoYU586SrxURMvDn_jBMJInitviJg4O5-iOU-6I0WY,1193 -pip/_internal/utils/packaging.py,sha256=5Wm6_x7lKrlqVjPI5MBN_RurcRHwVYoQ7Ksrs84de7s,2108 -pip/_internal/utils/setuptools_build.py,sha256=ouXpud-jeS8xPyTPsXJ-m34NPvK5os45otAzdSV_IJE,4435 -pip/_internal/utils/subprocess.py,sha256=zzdimb75jVLE1GU4WlTZ055gczhD7n1y1xTcNc7vNZQ,9207 -pip/_internal/utils/temp_dir.py,sha256=DUAw22uFruQdK43i2L2K53C-CDjRCPeAsBKJpu-rHQ4,9312 -pip/_internal/utils/unpacking.py,sha256=SBb2iV1crb89MDRTEKY86R4A_UOWApTQn9VQVcMDOlE,8821 -pip/_internal/utils/urls.py,sha256=AhaesUGl-9it6uvG6fsFPOr9ynFpGaTMk4t5XTX7Z_Q,1759 -pip/_internal/utils/virtualenv.py,sha256=S6f7csYorRpiD6cvn3jISZYc3I8PJC43H5iMFpRAEDU,3456 -pip/_internal/utils/wheel.py,sha256=lXOgZyTlOm5HmK8tw5iw0A3_5A6wRzsXHOaQkIvvloU,4549 -pip/_internal/vcs/__init__.py,sha256=UAqvzpbi0VbZo3Ub6skEeZAw-ooIZR-zX_WpCbxyCoU,596 -pip/_internal/vcs/__pycache__/__init__.cpython-311.pyc,, -pip/_internal/vcs/__pycache__/bazaar.cpython-311.pyc,, -pip/_internal/vcs/__pycache__/git.cpython-311.pyc,, -pip/_internal/vcs/__pycache__/mercurial.cpython-311.pyc,, -pip/_internal/vcs/__pycache__/subversion.cpython-311.pyc,, -pip/_internal/vcs/__pycache__/versioncontrol.cpython-311.pyc,, -pip/_internal/vcs/bazaar.py,sha256=j0oin0fpGRHcCFCxEcpPCQoFEvA-DMLULKdGP8Nv76o,3519 -pip/_internal/vcs/git.py,sha256=CeKBGJnl6uskvvjkAUXrJVxbHJrpS_B_pyfFdjL3CRc,18121 -pip/_internal/vcs/mercurial.py,sha256=ytRnzmP5CkLM2RfdiS4mVJx4jQcmB3FjXeLOPPFEjG8,5246 -pip/_internal/vcs/subversion.py,sha256=vhZs8L-TNggXqM1bbhl-FpbxE3TrIB6Tgnx8fh3S2HE,11729 -pip/_internal/vcs/versioncontrol.py,sha256=KUOc-hN51em9jrqxKwUR3JnkgSE-xSOqMiiJcSaL6B8,22811 -pip/_internal/wheel_builder.py,sha256=3UlHfxQi7_AAXI7ur8aPpPbmqHhecCsubmkHEl-00KU,11842 -pip/_vendor/__init__.py,sha256=U51NPwXdA-wXOiANIQncYjcMp6txgeOL5nHxksJeyas,4993 -pip/_vendor/__pycache__/__init__.cpython-311.pyc,, -pip/_vendor/__pycache__/six.cpython-311.pyc,, -pip/_vendor/__pycache__/typing_extensions.cpython-311.pyc,, -pip/_vendor/cachecontrol/__init__.py,sha256=ctHagMhQXuvQDdm4TirZrwDOT5H8oBNAJqzdKI6sovk,676 -pip/_vendor/cachecontrol/__pycache__/__init__.cpython-311.pyc,, -pip/_vendor/cachecontrol/__pycache__/_cmd.cpython-311.pyc,, -pip/_vendor/cachecontrol/__pycache__/adapter.cpython-311.pyc,, -pip/_vendor/cachecontrol/__pycache__/cache.cpython-311.pyc,, -pip/_vendor/cachecontrol/__pycache__/controller.cpython-311.pyc,, -pip/_vendor/cachecontrol/__pycache__/filewrapper.cpython-311.pyc,, -pip/_vendor/cachecontrol/__pycache__/heuristics.cpython-311.pyc,, -pip/_vendor/cachecontrol/__pycache__/serialize.cpython-311.pyc,, -pip/_vendor/cachecontrol/__pycache__/wrapper.cpython-311.pyc,, -pip/_vendor/cachecontrol/_cmd.py,sha256=iist2EpzJvDVIhMAxXq8iFnTBsiZAd6iplxfmNboNyk,1737 -pip/_vendor/cachecontrol/adapter.py,sha256=_CcWvUP9048qAZjsNqViaHbdcLs9mmFNixVfpO7oebE,6392 -pip/_vendor/cachecontrol/cache.py,sha256=OTQj72tUf8C1uEgczdl3Gc8vkldSzsTITKtDGKMx4z8,1952 -pip/_vendor/cachecontrol/caches/__init__.py,sha256=dtrrroK5BnADR1GWjCZ19aZ0tFsMfvFBtLQQU1sp_ag,303 -pip/_vendor/cachecontrol/caches/__pycache__/__init__.cpython-311.pyc,, -pip/_vendor/cachecontrol/caches/__pycache__/file_cache.cpython-311.pyc,, -pip/_vendor/cachecontrol/caches/__pycache__/redis_cache.cpython-311.pyc,, -pip/_vendor/cachecontrol/caches/file_cache.py,sha256=3z8AWKD-vfKeiJqIzLmJyIYtR2yd6Tsh3u1TyLRQoIQ,5352 -pip/_vendor/cachecontrol/caches/redis_cache.py,sha256=9rmqwtYu_ljVkW6_oLqbC7EaX_a8YT_yLuna-eS0dgo,1386 -pip/_vendor/cachecontrol/controller.py,sha256=keCFA3ZaNVaWTwHd6F1zqWhb4vyvNx_UvZuo5iIYMfo,18384 -pip/_vendor/cachecontrol/filewrapper.py,sha256=STttGmIPBvZzt2b51dUOwoWX5crcMCpKZOisM3f5BNc,4292 -pip/_vendor/cachecontrol/heuristics.py,sha256=fdFbk9W8IeLrjteIz_fK4mj2HD_Y7COXF2Uc8TgjT1c,4828 -pip/_vendor/cachecontrol/serialize.py,sha256=0dHeMaDwysVAAnGVlhMOP4tDliohgNK0Jxk_zsOiWxw,7173 -pip/_vendor/cachecontrol/wrapper.py,sha256=hsGc7g8QGQTT-4f8tgz3AM5qwScg6FO0BSdLSRdEvpU,1417 -pip/_vendor/certifi/__init__.py,sha256=L_j-d0kYuA_MzA2_2hraF1ovf6KT6DTquRdV3paQwOk,94 -pip/_vendor/certifi/__main__.py,sha256=1k3Cr95vCxxGRGDljrW3wMdpZdL3Nhf0u1n-k2qdsCY,255 -pip/_vendor/certifi/__pycache__/__init__.cpython-311.pyc,, -pip/_vendor/certifi/__pycache__/__main__.cpython-311.pyc,, -pip/_vendor/certifi/__pycache__/core.cpython-311.pyc,, -pip/_vendor/certifi/cacert.pem,sha256=eU0Dn_3yd8BH4m8sfVj4Glhl2KDrcCSg-sEWT-pNJ88,281617 -pip/_vendor/certifi/core.py,sha256=ZwiOsv-sD_ouU1ft8wy_xZ3LQ7UbcVzyqj2XNyrsZis,4279 -pip/_vendor/chardet/__init__.py,sha256=57R-HSxj0PWmILMN0GFmUNqEMfrEVSamXyjD-W6_fbs,4797 -pip/_vendor/chardet/__pycache__/__init__.cpython-311.pyc,, -pip/_vendor/chardet/__pycache__/big5freq.cpython-311.pyc,, -pip/_vendor/chardet/__pycache__/big5prober.cpython-311.pyc,, -pip/_vendor/chardet/__pycache__/chardistribution.cpython-311.pyc,, -pip/_vendor/chardet/__pycache__/charsetgroupprober.cpython-311.pyc,, -pip/_vendor/chardet/__pycache__/charsetprober.cpython-311.pyc,, -pip/_vendor/chardet/__pycache__/codingstatemachine.cpython-311.pyc,, -pip/_vendor/chardet/__pycache__/codingstatemachinedict.cpython-311.pyc,, -pip/_vendor/chardet/__pycache__/cp949prober.cpython-311.pyc,, -pip/_vendor/chardet/__pycache__/enums.cpython-311.pyc,, -pip/_vendor/chardet/__pycache__/escprober.cpython-311.pyc,, -pip/_vendor/chardet/__pycache__/escsm.cpython-311.pyc,, -pip/_vendor/chardet/__pycache__/eucjpprober.cpython-311.pyc,, -pip/_vendor/chardet/__pycache__/euckrfreq.cpython-311.pyc,, -pip/_vendor/chardet/__pycache__/euckrprober.cpython-311.pyc,, -pip/_vendor/chardet/__pycache__/euctwfreq.cpython-311.pyc,, -pip/_vendor/chardet/__pycache__/euctwprober.cpython-311.pyc,, -pip/_vendor/chardet/__pycache__/gb2312freq.cpython-311.pyc,, -pip/_vendor/chardet/__pycache__/gb2312prober.cpython-311.pyc,, -pip/_vendor/chardet/__pycache__/hebrewprober.cpython-311.pyc,, -pip/_vendor/chardet/__pycache__/jisfreq.cpython-311.pyc,, -pip/_vendor/chardet/__pycache__/johabfreq.cpython-311.pyc,, -pip/_vendor/chardet/__pycache__/johabprober.cpython-311.pyc,, -pip/_vendor/chardet/__pycache__/jpcntx.cpython-311.pyc,, -pip/_vendor/chardet/__pycache__/langbulgarianmodel.cpython-311.pyc,, -pip/_vendor/chardet/__pycache__/langgreekmodel.cpython-311.pyc,, -pip/_vendor/chardet/__pycache__/langhebrewmodel.cpython-311.pyc,, -pip/_vendor/chardet/__pycache__/langhungarianmodel.cpython-311.pyc,, -pip/_vendor/chardet/__pycache__/langrussianmodel.cpython-311.pyc,, -pip/_vendor/chardet/__pycache__/langthaimodel.cpython-311.pyc,, -pip/_vendor/chardet/__pycache__/langturkishmodel.cpython-311.pyc,, -pip/_vendor/chardet/__pycache__/latin1prober.cpython-311.pyc,, -pip/_vendor/chardet/__pycache__/macromanprober.cpython-311.pyc,, -pip/_vendor/chardet/__pycache__/mbcharsetprober.cpython-311.pyc,, -pip/_vendor/chardet/__pycache__/mbcsgroupprober.cpython-311.pyc,, -pip/_vendor/chardet/__pycache__/mbcssm.cpython-311.pyc,, -pip/_vendor/chardet/__pycache__/resultdict.cpython-311.pyc,, -pip/_vendor/chardet/__pycache__/sbcharsetprober.cpython-311.pyc,, -pip/_vendor/chardet/__pycache__/sbcsgroupprober.cpython-311.pyc,, -pip/_vendor/chardet/__pycache__/sjisprober.cpython-311.pyc,, -pip/_vendor/chardet/__pycache__/universaldetector.cpython-311.pyc,, -pip/_vendor/chardet/__pycache__/utf1632prober.cpython-311.pyc,, -pip/_vendor/chardet/__pycache__/utf8prober.cpython-311.pyc,, -pip/_vendor/chardet/__pycache__/version.cpython-311.pyc,, -pip/_vendor/chardet/big5freq.py,sha256=ltcfP-3PjlNHCoo5e4a7C4z-2DhBTXRfY6jbMbB7P30,31274 -pip/_vendor/chardet/big5prober.py,sha256=lPMfwCX6v2AaPgvFh_cSWZcgLDbWiFCHLZ_p9RQ9uxE,1763 -pip/_vendor/chardet/chardistribution.py,sha256=13B8XUG4oXDuLdXvfbIWwLFeR-ZU21AqTS1zcdON8bU,10032 -pip/_vendor/chardet/charsetgroupprober.py,sha256=UKK3SaIZB2PCdKSIS0gnvMtLR9JJX62M-fZJu3OlWyg,3915 -pip/_vendor/chardet/charsetprober.py,sha256=L3t8_wIOov8em-vZWOcbkdsrwe43N6_gqNh5pH7WPd4,5420 -pip/_vendor/chardet/cli/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0 -pip/_vendor/chardet/cli/__pycache__/__init__.cpython-311.pyc,, -pip/_vendor/chardet/cli/__pycache__/chardetect.cpython-311.pyc,, -pip/_vendor/chardet/cli/chardetect.py,sha256=zibMVg5RpKb-ME9_7EYG4ZM2Sf07NHcQzZ12U-rYJho,3242 -pip/_vendor/chardet/codingstatemachine.py,sha256=K7k69sw3jY5DmTXoSJQVsUtFIQKYPQVOSJJhBuGv_yE,3732 -pip/_vendor/chardet/codingstatemachinedict.py,sha256=0GY3Hi2qIZvDrOOJ3AtqppM1RsYxr_66ER4EHjuMiMc,542 -pip/_vendor/chardet/cp949prober.py,sha256=0jKRV7fECuWI16rNnks0ZECKA1iZYCIEaP8A1ZvjUSI,1860 -pip/_vendor/chardet/enums.py,sha256=TzECiZoCKNMqgwU76cPCeKWFBqaWvAdLMev5_bCkhY8,1683 -pip/_vendor/chardet/escprober.py,sha256=Kho48X65xE0scFylIdeJjM2bcbvRvv0h0WUbMWrJD3A,4006 -pip/_vendor/chardet/escsm.py,sha256=AqyXpA2FQFD7k-buBty_7itGEYkhmVa8X09NLRul3QM,12176 -pip/_vendor/chardet/eucjpprober.py,sha256=5KYaM9fsxkRYzw1b5k0fL-j_-ezIw-ij9r97a9MHxLY,3934 -pip/_vendor/chardet/euckrfreq.py,sha256=3mHuRvXfsq_QcQysDQFb8qSudvTiol71C6Ic2w57tKM,13566 -pip/_vendor/chardet/euckrprober.py,sha256=hiFT6wM174GIwRvqDsIcuOc-dDsq2uPKMKbyV8-1Xnc,1753 -pip/_vendor/chardet/euctwfreq.py,sha256=2alILE1Lh5eqiFJZjzRkMQXolNJRHY5oBQd-vmZYFFM,36913 -pip/_vendor/chardet/euctwprober.py,sha256=NxbpNdBtU0VFI0bKfGfDkpP7S2_8_6FlO87dVH0ogws,1753 -pip/_vendor/chardet/gb2312freq.py,sha256=49OrdXzD-HXqwavkqjo8Z7gvs58hONNzDhAyMENNkvY,20735 -pip/_vendor/chardet/gb2312prober.py,sha256=KPEBueaSLSvBpFeINMu0D6TgHcR90e5PaQawifzF4o0,1759 -pip/_vendor/chardet/hebrewprober.py,sha256=96T_Lj_OmW-fK7JrSHojYjyG3fsGgbzkoTNleZ3kfYE,14537 -pip/_vendor/chardet/jisfreq.py,sha256=mm8tfrwqhpOd3wzZKS4NJqkYBQVcDfTM2JiQ5aW932E,25796 -pip/_vendor/chardet/johabfreq.py,sha256=dBpOYG34GRX6SL8k_LbS9rxZPMjLjoMlgZ03Pz5Hmqc,42498 -pip/_vendor/chardet/johabprober.py,sha256=O1Qw9nVzRnun7vZp4UZM7wvJSv9W941mEU9uDMnY3DU,1752 -pip/_vendor/chardet/jpcntx.py,sha256=uhHrYWkLxE_rF5OkHKInm0HUsrjgKHHVQvtt3UcvotA,27055 -pip/_vendor/chardet/langbulgarianmodel.py,sha256=vmbvYFP8SZkSxoBvLkFqKiH1sjma5ihk3PTpdy71Rr4,104562 -pip/_vendor/chardet/langgreekmodel.py,sha256=JfB7bupjjJH2w3X_mYnQr9cJA_7EuITC2cRW13fUjeI,98484 -pip/_vendor/chardet/langhebrewmodel.py,sha256=3HXHaLQPNAGcXnJjkIJfozNZLTvTJmf4W5Awi6zRRKc,98196 -pip/_vendor/chardet/langhungarianmodel.py,sha256=WxbeQIxkv8YtApiNqxQcvj-tMycsoI4Xy-fwkDHpP_Y,101363 -pip/_vendor/chardet/langrussianmodel.py,sha256=s395bTZ87ESTrZCOdgXbEjZ9P1iGPwCl_8xSsac_DLY,128035 -pip/_vendor/chardet/langthaimodel.py,sha256=7bJlQitRpTnVGABmbSznHnJwOHDy3InkTvtFUx13WQI,102774 -pip/_vendor/chardet/langturkishmodel.py,sha256=XY0eGdTIy4eQ9Xg1LVPZacb-UBhHBR-cq0IpPVHowKc,95372 -pip/_vendor/chardet/latin1prober.py,sha256=p15EEmFbmQUwbKLC7lOJVGHEZwcG45ubEZYTGu01J5g,5380 -pip/_vendor/chardet/macromanprober.py,sha256=9anfzmY6TBfUPDyBDOdY07kqmTHpZ1tK0jL-p1JWcOY,6077 -pip/_vendor/chardet/mbcharsetprober.py,sha256=Wr04WNI4F3X_VxEverNG-H25g7u-MDDKlNt-JGj-_uU,3715 -pip/_vendor/chardet/mbcsgroupprober.py,sha256=iRpaNBjV0DNwYPu_z6TiHgRpwYahiM7ztI_4kZ4Uz9A,2131 -pip/_vendor/chardet/mbcssm.py,sha256=hUtPvDYgWDaA2dWdgLsshbwRfm3Q5YRlRogdmeRUNQw,30391 -pip/_vendor/chardet/metadata/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0 -pip/_vendor/chardet/metadata/__pycache__/__init__.cpython-311.pyc,, -pip/_vendor/chardet/metadata/__pycache__/languages.cpython-311.pyc,, -pip/_vendor/chardet/metadata/languages.py,sha256=FhvBIdZFxRQ-dTwkb_0madRKgVBCaUMQz9I5xqjE5iQ,13560 -pip/_vendor/chardet/resultdict.py,sha256=ez4FRvN5KaSosJeJ2WzUyKdDdg35HDy_SSLPXKCdt5M,402 -pip/_vendor/chardet/sbcharsetprober.py,sha256=-nd3F90i7GpXLjehLVHqVBE0KlWzGvQUPETLBNn4o6U,6400 -pip/_vendor/chardet/sbcsgroupprober.py,sha256=gcgI0fOfgw_3YTClpbra_MNxwyEyJ3eUXraoLHYb59E,4137 -pip/_vendor/chardet/sjisprober.py,sha256=aqQufMzRw46ZpFlzmYaYeT2-nzmKb-hmcrApppJ862k,4007 -pip/_vendor/chardet/universaldetector.py,sha256=xYBrg4x0dd9WnT8qclfADVD9ondrUNkqPmvte1pa520,14848 -pip/_vendor/chardet/utf1632prober.py,sha256=pw1epGdMj1hDGiCu1AHqqzOEfjX8MVdiW7O1BlT8-eQ,8505 -pip/_vendor/chardet/utf8prober.py,sha256=8m08Ub5490H4jQ6LYXvFysGtgKoKsHUd2zH_i8_TnVw,2812 -pip/_vendor/chardet/version.py,sha256=lGtJcxGM44Qz4Cbk4rbbmrKxnNr1-97U25TameLehZw,244 -pip/_vendor/colorama/__init__.py,sha256=wePQA4U20tKgYARySLEC047ucNX-g8pRLpYBuiHlLb8,266 -pip/_vendor/colorama/__pycache__/__init__.cpython-311.pyc,, -pip/_vendor/colorama/__pycache__/ansi.cpython-311.pyc,, -pip/_vendor/colorama/__pycache__/ansitowin32.cpython-311.pyc,, -pip/_vendor/colorama/__pycache__/initialise.cpython-311.pyc,, -pip/_vendor/colorama/__pycache__/win32.cpython-311.pyc,, -pip/_vendor/colorama/__pycache__/winterm.cpython-311.pyc,, -pip/_vendor/colorama/ansi.py,sha256=Top4EeEuaQdBWdteKMEcGOTeKeF19Q-Wo_6_Cj5kOzQ,2522 -pip/_vendor/colorama/ansitowin32.py,sha256=vPNYa3OZbxjbuFyaVo0Tmhmy1FZ1lKMWCnT7odXpItk,11128 -pip/_vendor/colorama/initialise.py,sha256=-hIny86ClXo39ixh5iSCfUIa2f_h_bgKRDW7gqs-KLU,3325 -pip/_vendor/colorama/tests/__init__.py,sha256=MkgPAEzGQd-Rq0w0PZXSX2LadRWhUECcisJY8lSrm4Q,75 -pip/_vendor/colorama/tests/__pycache__/__init__.cpython-311.pyc,, -pip/_vendor/colorama/tests/__pycache__/ansi_test.cpython-311.pyc,, -pip/_vendor/colorama/tests/__pycache__/ansitowin32_test.cpython-311.pyc,, -pip/_vendor/colorama/tests/__pycache__/initialise_test.cpython-311.pyc,, -pip/_vendor/colorama/tests/__pycache__/isatty_test.cpython-311.pyc,, -pip/_vendor/colorama/tests/__pycache__/utils.cpython-311.pyc,, -pip/_vendor/colorama/tests/__pycache__/winterm_test.cpython-311.pyc,, -pip/_vendor/colorama/tests/ansi_test.py,sha256=FeViDrUINIZcr505PAxvU4AjXz1asEiALs9GXMhwRaE,2839 -pip/_vendor/colorama/tests/ansitowin32_test.py,sha256=RN7AIhMJ5EqDsYaCjVo-o4u8JzDD4ukJbmevWKS70rY,10678 -pip/_vendor/colorama/tests/initialise_test.py,sha256=BbPy-XfyHwJ6zKozuQOvNvQZzsx9vdb_0bYXn7hsBTc,6741 -pip/_vendor/colorama/tests/isatty_test.py,sha256=Pg26LRpv0yQDB5Ac-sxgVXG7hsA1NYvapFgApZfYzZg,1866 -pip/_vendor/colorama/tests/utils.py,sha256=1IIRylG39z5-dzq09R_ngufxyPZxgldNbrxKxUGwGKE,1079 -pip/_vendor/colorama/tests/winterm_test.py,sha256=qoWFPEjym5gm2RuMwpf3pOis3a5r_PJZFCzK254JL8A,3709 -pip/_vendor/colorama/win32.py,sha256=YQOKwMTwtGBbsY4dL5HYTvwTeP9wIQra5MvPNddpxZs,6181 -pip/_vendor/colorama/winterm.py,sha256=XCQFDHjPi6AHYNdZwy0tA02H-Jh48Jp-HvCjeLeLp3U,7134 -pip/_vendor/distlib/__init__.py,sha256=acgfseOC55dNrVAzaBKpUiH3Z6V7Q1CaxsiQ3K7pC-E,581 -pip/_vendor/distlib/__pycache__/__init__.cpython-311.pyc,, -pip/_vendor/distlib/__pycache__/compat.cpython-311.pyc,, -pip/_vendor/distlib/__pycache__/database.cpython-311.pyc,, -pip/_vendor/distlib/__pycache__/index.cpython-311.pyc,, -pip/_vendor/distlib/__pycache__/locators.cpython-311.pyc,, -pip/_vendor/distlib/__pycache__/manifest.cpython-311.pyc,, -pip/_vendor/distlib/__pycache__/markers.cpython-311.pyc,, -pip/_vendor/distlib/__pycache__/metadata.cpython-311.pyc,, -pip/_vendor/distlib/__pycache__/resources.cpython-311.pyc,, -pip/_vendor/distlib/__pycache__/scripts.cpython-311.pyc,, -pip/_vendor/distlib/__pycache__/util.cpython-311.pyc,, -pip/_vendor/distlib/__pycache__/version.cpython-311.pyc,, -pip/_vendor/distlib/__pycache__/wheel.cpython-311.pyc,, -pip/_vendor/distlib/compat.py,sha256=tfoMrj6tujk7G4UC2owL6ArgDuCKabgBxuJRGZSmpko,41259 -pip/_vendor/distlib/database.py,sha256=o_mw0fAr93NDAHHHfqG54Y1Hi9Rkfrp2BX15XWZYK50,51697 -pip/_vendor/distlib/index.py,sha256=HFiDG7LMoaBs829WuotrfIwcErOOExUOR_AeBtw_TCU,20834 -pip/_vendor/distlib/locators.py,sha256=wNzG-zERzS_XGls-nBPVVyLRHa2skUlkn0-5n0trMWA,51991 -pip/_vendor/distlib/manifest.py,sha256=nQEhYmgoreaBZzyFzwYsXxJARu3fo4EkunU163U16iE,14811 -pip/_vendor/distlib/markers.py,sha256=TpHHHLgkzyT7YHbwj-2i6weRaq-Ivy2-MUnrDkjau-U,5058 -pip/_vendor/distlib/metadata.py,sha256=g_DIiu8nBXRzA-mWPRpatHGbmFZqaFoss7z9TG7QSUU,39801 -pip/_vendor/distlib/resources.py,sha256=LwbPksc0A1JMbi6XnuPdMBUn83X7BPuFNWqPGEKI698,10820 -pip/_vendor/distlib/scripts.py,sha256=BmkTKmiTk4m2cj-iueliatwz3ut_9SsABBW51vnQnZU,18102 -pip/_vendor/distlib/t32.exe,sha256=a0GV5kCoWsMutvliiCKmIgV98eRZ33wXoS-XrqvJQVs,97792 -pip/_vendor/distlib/t64-arm.exe,sha256=68TAa32V504xVBnufojh0PcenpR3U4wAqTqf-MZqbPw,182784 -pip/_vendor/distlib/t64.exe,sha256=gaYY8hy4fbkHYTTnA4i26ct8IQZzkBG2pRdy0iyuBrc,108032 -pip/_vendor/distlib/util.py,sha256=31dPXn3Rfat0xZLeVoFpuniyhe6vsbl9_QN-qd9Lhlk,66262 -pip/_vendor/distlib/version.py,sha256=WG__LyAa2GwmA6qSoEJtvJE8REA1LZpbSizy8WvhJLk,23513 -pip/_vendor/distlib/w32.exe,sha256=R4csx3-OGM9kL4aPIzQKRo5TfmRSHZo6QWyLhDhNBks,91648 -pip/_vendor/distlib/w64-arm.exe,sha256=xdyYhKj0WDcVUOCb05blQYvzdYIKMbmJn2SZvzkcey4,168448 -pip/_vendor/distlib/w64.exe,sha256=ejGf-rojoBfXseGLpya6bFTFPWRG21X5KvU8J5iU-K0,101888 -pip/_vendor/distlib/wheel.py,sha256=Rgqs658VsJ3R2845qwnZD8XQryV2CzWw2mghwLvxxsI,43898 -pip/_vendor/distro/__init__.py,sha256=2fHjF-SfgPvjyNZ1iHh_wjqWdR_Yo5ODHwZC0jLBPhc,981 -pip/_vendor/distro/__main__.py,sha256=bu9d3TifoKciZFcqRBuygV3GSuThnVD_m2IK4cz96Vs,64 -pip/_vendor/distro/__pycache__/__init__.cpython-311.pyc,, -pip/_vendor/distro/__pycache__/__main__.cpython-311.pyc,, -pip/_vendor/distro/__pycache__/distro.cpython-311.pyc,, -pip/_vendor/distro/distro.py,sha256=UZO1LjIhtFCMdlbiz39gj3raV-Amf3SBwzGzfApiMHw,49330 -pip/_vendor/idna/__init__.py,sha256=KJQN1eQBr8iIK5SKrJ47lXvxG0BJ7Lm38W4zT0v_8lk,849 -pip/_vendor/idna/__pycache__/__init__.cpython-311.pyc,, -pip/_vendor/idna/__pycache__/codec.cpython-311.pyc,, -pip/_vendor/idna/__pycache__/compat.cpython-311.pyc,, -pip/_vendor/idna/__pycache__/core.cpython-311.pyc,, -pip/_vendor/idna/__pycache__/idnadata.cpython-311.pyc,, -pip/_vendor/idna/__pycache__/intranges.cpython-311.pyc,, -pip/_vendor/idna/__pycache__/package_data.cpython-311.pyc,, -pip/_vendor/idna/__pycache__/uts46data.cpython-311.pyc,, -pip/_vendor/idna/codec.py,sha256=6ly5odKfqrytKT9_7UrlGklHnf1DSK2r9C6cSM4sa28,3374 -pip/_vendor/idna/compat.py,sha256=0_sOEUMT4CVw9doD3vyRhX80X19PwqFoUBs7gWsFME4,321 -pip/_vendor/idna/core.py,sha256=1JxchwKzkxBSn7R_oCE12oBu3eVux0VzdxolmIad24M,12950 -pip/_vendor/idna/idnadata.py,sha256=xUjqKqiJV8Ho_XzBpAtv5JFoVPSupK-SUXvtjygUHqw,44375 -pip/_vendor/idna/intranges.py,sha256=YBr4fRYuWH7kTKS2tXlFjM24ZF1Pdvcir-aywniInqg,1881 -pip/_vendor/idna/package_data.py,sha256=C_jHJzmX8PI4xq0jpzmcTMxpb5lDsq4o5VyxQzlVrZE,21 -pip/_vendor/idna/uts46data.py,sha256=zvjZU24s58_uAS850Mcd0NnD0X7_gCMAMjzWNIeUJdc,206539 -pip/_vendor/msgpack/__init__.py,sha256=hyGhlnmcJkxryJBKC3X5FnEph375kQoL_mG8LZUuXgY,1132 -pip/_vendor/msgpack/__pycache__/__init__.cpython-311.pyc,, -pip/_vendor/msgpack/__pycache__/exceptions.cpython-311.pyc,, -pip/_vendor/msgpack/__pycache__/ext.cpython-311.pyc,, -pip/_vendor/msgpack/__pycache__/fallback.cpython-311.pyc,, -pip/_vendor/msgpack/exceptions.py,sha256=dCTWei8dpkrMsQDcjQk74ATl9HsIBH0ybt8zOPNqMYc,1081 -pip/_vendor/msgpack/ext.py,sha256=C5MK8JhVYGYFWPvxsORsqZAnvOXefYQ57m1Ym0luW5M,6079 -pip/_vendor/msgpack/fallback.py,sha256=tvNBHyxxFbuVlC8GZShETClJxjLiDMOja4XwwyvNm2g,34544 -pip/_vendor/packaging/__about__.py,sha256=ugASIO2w1oUyH8_COqQ2X_s0rDhjbhQC3yJocD03h2c,661 -pip/_vendor/packaging/__init__.py,sha256=b9Kk5MF7KxhhLgcDmiUWukN-LatWFxPdNug0joPhHSk,497 -pip/_vendor/packaging/__pycache__/__about__.cpython-311.pyc,, -pip/_vendor/packaging/__pycache__/__init__.cpython-311.pyc,, -pip/_vendor/packaging/__pycache__/_manylinux.cpython-311.pyc,, -pip/_vendor/packaging/__pycache__/_musllinux.cpython-311.pyc,, -pip/_vendor/packaging/__pycache__/_structures.cpython-311.pyc,, -pip/_vendor/packaging/__pycache__/markers.cpython-311.pyc,, -pip/_vendor/packaging/__pycache__/requirements.cpython-311.pyc,, -pip/_vendor/packaging/__pycache__/specifiers.cpython-311.pyc,, -pip/_vendor/packaging/__pycache__/tags.cpython-311.pyc,, -pip/_vendor/packaging/__pycache__/utils.cpython-311.pyc,, -pip/_vendor/packaging/__pycache__/version.cpython-311.pyc,, -pip/_vendor/packaging/_manylinux.py,sha256=XcbiXB-qcjv3bcohp6N98TMpOP4_j3m-iOA8ptK2GWY,11488 -pip/_vendor/packaging/_musllinux.py,sha256=_KGgY_qc7vhMGpoqss25n2hiLCNKRtvz9mCrS7gkqyc,4378 -pip/_vendor/packaging/_structures.py,sha256=q3eVNmbWJGG_S0Dit_S3Ao8qQqz_5PYTXFAKBZe5yr4,1431 -pip/_vendor/packaging/markers.py,sha256=AJBOcY8Oq0kYc570KuuPTkvuqjAlhufaE2c9sCUbm64,8487 -pip/_vendor/packaging/requirements.py,sha256=NtDlPBtojpn1IUC85iMjPNsUmufjpSlwnNA-Xb4m5NA,4676 -pip/_vendor/packaging/specifiers.py,sha256=LRQ0kFsHrl5qfcFNEEJrIFYsnIHQUJXY9fIsakTrrqE,30110 -pip/_vendor/packaging/tags.py,sha256=lmsnGNiJ8C4D_Pf9PbM0qgbZvD9kmB9lpZBQUZa3R_Y,15699 -pip/_vendor/packaging/utils.py,sha256=dJjeat3BS-TYn1RrUFVwufUMasbtzLfYRoy_HXENeFQ,4200 -pip/_vendor/packaging/version.py,sha256=_fLRNrFrxYcHVfyo8vk9j8s6JM8N_xsSxVFr6RJyco8,14665 -pip/_vendor/pkg_resources/__init__.py,sha256=hTAeJCNYb7dJseIDVsYK3mPQep_gphj4tQh-bspX8bg,109364 -pip/_vendor/pkg_resources/__pycache__/__init__.cpython-311.pyc,, -pip/_vendor/platformdirs/__init__.py,sha256=SkhEYVyC_HUHC6KX7n4M_6coyRMtEB38QMyOYIAX6Yk,20155 -pip/_vendor/platformdirs/__main__.py,sha256=fVvSiTzr2-RM6IsjWjj4fkaOtDOgDhUWv6sA99do4CQ,1476 -pip/_vendor/platformdirs/__pycache__/__init__.cpython-311.pyc,, -pip/_vendor/platformdirs/__pycache__/__main__.cpython-311.pyc,, -pip/_vendor/platformdirs/__pycache__/android.cpython-311.pyc,, -pip/_vendor/platformdirs/__pycache__/api.cpython-311.pyc,, -pip/_vendor/platformdirs/__pycache__/macos.cpython-311.pyc,, -pip/_vendor/platformdirs/__pycache__/unix.cpython-311.pyc,, -pip/_vendor/platformdirs/__pycache__/version.cpython-311.pyc,, -pip/_vendor/platformdirs/__pycache__/windows.cpython-311.pyc,, -pip/_vendor/platformdirs/android.py,sha256=y_EEMKwYl2-bzYBDovksSn8m76on0Lda8eyJksVQE9U,7211 -pip/_vendor/platformdirs/api.py,sha256=jWtX06jAJytYrkJDOqEls97mCkyHRSZkoqUlbMK5Qew,7132 -pip/_vendor/platformdirs/macos.py,sha256=LueVOoVgGWDBwQb8OFwXkVKfVn33CM1Lkwf1-A86tRQ,3678 -pip/_vendor/platformdirs/unix.py,sha256=22JhR8ZY0aLxSVCFnKrc6f1iz6Gv42K24Daj7aTjfSg,8809 -pip/_vendor/platformdirs/version.py,sha256=mavZTQIJIXfdewEaSTn7EWrNfPZWeRofb-74xqW5f2M,160 -pip/_vendor/platformdirs/windows.py,sha256=4TtbPGoWG2PRgI11uquDa7eRk8TcxvnUNuuMGZItnXc,9573 -pip/_vendor/pygments/__init__.py,sha256=6AuDljQtvf89DTNUyWM7k3oUlP_lq70NU-INKKteOBY,2983 -pip/_vendor/pygments/__main__.py,sha256=es8EKMvXj5yToIfQ-pf3Dv5TnIeeM6sME0LW-n4ecHo,353 -pip/_vendor/pygments/__pycache__/__init__.cpython-311.pyc,, -pip/_vendor/pygments/__pycache__/__main__.cpython-311.pyc,, -pip/_vendor/pygments/__pycache__/cmdline.cpython-311.pyc,, -pip/_vendor/pygments/__pycache__/console.cpython-311.pyc,, -pip/_vendor/pygments/__pycache__/filter.cpython-311.pyc,, -pip/_vendor/pygments/__pycache__/formatter.cpython-311.pyc,, -pip/_vendor/pygments/__pycache__/lexer.cpython-311.pyc,, -pip/_vendor/pygments/__pycache__/modeline.cpython-311.pyc,, -pip/_vendor/pygments/__pycache__/plugin.cpython-311.pyc,, -pip/_vendor/pygments/__pycache__/regexopt.cpython-311.pyc,, -pip/_vendor/pygments/__pycache__/scanner.cpython-311.pyc,, -pip/_vendor/pygments/__pycache__/sphinxext.cpython-311.pyc,, -pip/_vendor/pygments/__pycache__/style.cpython-311.pyc,, -pip/_vendor/pygments/__pycache__/token.cpython-311.pyc,, -pip/_vendor/pygments/__pycache__/unistring.cpython-311.pyc,, -pip/_vendor/pygments/__pycache__/util.cpython-311.pyc,, -pip/_vendor/pygments/cmdline.py,sha256=byxYJp9gnjVeyhRlZ3UTMgo_LhkXh1afvN8wJBtAcc8,23685 -pip/_vendor/pygments/console.py,sha256=2wZ5W-U6TudJD1_NLUwjclMpbomFM91lNv11_60sfGY,1697 -pip/_vendor/pygments/filter.py,sha256=j5aLM9a9wSx6eH1oy473oSkJ02hGWNptBlVo4s1g_30,1938 -pip/_vendor/pygments/filters/__init__.py,sha256=h_koYkUFo-FFUxjs564JHUAz7O3yJpVwI6fKN3MYzG0,40386 -pip/_vendor/pygments/filters/__pycache__/__init__.cpython-311.pyc,, -pip/_vendor/pygments/formatter.py,sha256=J9OL9hXLJKZk7moUgKwpjW9HNf4WlJFg_o_-Z_S_tTY,4178 -pip/_vendor/pygments/formatters/__init__.py,sha256=_xgAcdFKr0QNYwh_i98AU9hvfP3X2wAkhElFcRRF3Uo,5424 -pip/_vendor/pygments/formatters/__pycache__/__init__.cpython-311.pyc,, -pip/_vendor/pygments/formatters/__pycache__/_mapping.cpython-311.pyc,, -pip/_vendor/pygments/formatters/__pycache__/bbcode.cpython-311.pyc,, -pip/_vendor/pygments/formatters/__pycache__/groff.cpython-311.pyc,, -pip/_vendor/pygments/formatters/__pycache__/html.cpython-311.pyc,, -pip/_vendor/pygments/formatters/__pycache__/img.cpython-311.pyc,, -pip/_vendor/pygments/formatters/__pycache__/irc.cpython-311.pyc,, -pip/_vendor/pygments/formatters/__pycache__/latex.cpython-311.pyc,, -pip/_vendor/pygments/formatters/__pycache__/other.cpython-311.pyc,, -pip/_vendor/pygments/formatters/__pycache__/pangomarkup.cpython-311.pyc,, -pip/_vendor/pygments/formatters/__pycache__/rtf.cpython-311.pyc,, -pip/_vendor/pygments/formatters/__pycache__/svg.cpython-311.pyc,, -pip/_vendor/pygments/formatters/__pycache__/terminal.cpython-311.pyc,, -pip/_vendor/pygments/formatters/__pycache__/terminal256.cpython-311.pyc,, -pip/_vendor/pygments/formatters/_mapping.py,sha256=1Cw37FuQlNacnxRKmtlPX4nyLoX9_ttko5ZwscNUZZ4,4176 -pip/_vendor/pygments/formatters/bbcode.py,sha256=r1b7wzWTJouADDLh-Z11iRi4iQxD0JKJ1qHl6mOYxsA,3314 -pip/_vendor/pygments/formatters/groff.py,sha256=xy8Zf3tXOo6MWrXh7yPGWx3lVEkg_DhY4CxmsDb0IVo,5094 -pip/_vendor/pygments/formatters/html.py,sha256=PIzAyilNqaTzSSP2slDG2VDLE3qNioWy2rgtSSoviuI,35610 -pip/_vendor/pygments/formatters/img.py,sha256=XKXmg2_XONrR4mtq2jfEU8XCsoln3VSGTw-UYiEokys,21938 -pip/_vendor/pygments/formatters/irc.py,sha256=Ep-m8jd3voFO6Fv57cUGFmz6JVA67IEgyiBOwv0N4a0,4981 -pip/_vendor/pygments/formatters/latex.py,sha256=FGzJ-YqSTE8z_voWPdzvLY5Tq8jE_ygjGjM6dXZJ8-k,19351 -pip/_vendor/pygments/formatters/other.py,sha256=gPxkk5BdAzWTCgbEHg1lpLi-1F6ZPh5A_aotgLXHnzg,5073 -pip/_vendor/pygments/formatters/pangomarkup.py,sha256=6LKnQc8yh49f802bF0sPvbzck4QivMYqqoXAPaYP8uU,2212 -pip/_vendor/pygments/formatters/rtf.py,sha256=aA0v_psW6KZI3N18TKDifxeL6mcF8EDXcPXDWI4vhVQ,5014 -pip/_vendor/pygments/formatters/svg.py,sha256=dQONWypbzfvzGCDtdp3M_NJawScJvM2DiHbx1k-ww7g,7335 -pip/_vendor/pygments/formatters/terminal.py,sha256=FG-rpjRpFmNpiGB4NzIucvxq6sQIXB3HOTo2meTKtrU,4674 -pip/_vendor/pygments/formatters/terminal256.py,sha256=13SJ3D5pFdqZ9zROE6HbWnBDwHvOGE8GlsmqGhprRp4,11753 -pip/_vendor/pygments/lexer.py,sha256=2BpqLlT2ExvOOi7vnjK5nB4Fp-m52ldiPaXMox5uwug,34618 -pip/_vendor/pygments/lexers/__init__.py,sha256=j5KEi5O_VQ5GS59H49l-10gzUOkWKxlwGeVMlGO2MMk,12130 -pip/_vendor/pygments/lexers/__pycache__/__init__.cpython-311.pyc,, -pip/_vendor/pygments/lexers/__pycache__/_mapping.cpython-311.pyc,, -pip/_vendor/pygments/lexers/__pycache__/python.cpython-311.pyc,, -pip/_vendor/pygments/lexers/_mapping.py,sha256=Hts4r_ZQ8icftGM7gkBPeED5lyVSv4affFgXYE6Ap04,72281 -pip/_vendor/pygments/lexers/python.py,sha256=c7jnmKFU9DLxTJW0UbwXt6Z9FJqbBlVsWA1Qr9xSA_w,53424 -pip/_vendor/pygments/modeline.py,sha256=eF2vO4LpOGoPvIKKkbPfnyut8hT4UiebZPpb-BYGQdI,986 -pip/_vendor/pygments/plugin.py,sha256=j1Fh310RbV2DQ9nvkmkqvlj38gdyuYKllLnGxbc8sJM,2591 -pip/_vendor/pygments/regexopt.py,sha256=jg1ALogcYGU96TQS9isBl6dCrvw5y5--BP_K-uFk_8s,3072 -pip/_vendor/pygments/scanner.py,sha256=b_nu5_f3HCgSdp5S_aNRBQ1MSCm4ZjDwec2OmTRickw,3092 -pip/_vendor/pygments/sphinxext.py,sha256=wBFYm180qea9JKt__UzhRlNRNhczPDFDaqGD21sbuso,6882 -pip/_vendor/pygments/style.py,sha256=C4qyoJrUTkq-OV3iO-8Vz3UtWYpJwSTdh5_vlGCGdNQ,6257 -pip/_vendor/pygments/styles/__init__.py,sha256=he7HjQx7sC0d2kfTVLjUs0J15mtToJM6M1brwIm9--Q,3700 -pip/_vendor/pygments/styles/__pycache__/__init__.cpython-311.pyc,, -pip/_vendor/pygments/token.py,sha256=seNsmcch9OEHXYirh8Ool7w8xDhfNTbLj5rHAC-gc_o,6184 -pip/_vendor/pygments/unistring.py,sha256=FaUfG14NBJEKLQoY9qj6JYeXrpYcLmKulghdxOGFaOc,63223 -pip/_vendor/pygments/util.py,sha256=AEVY0qonyyEMgv4Do2dINrrqUAwUk2XYSqHM650uzek,10230 -pip/_vendor/pyparsing/__init__.py,sha256=9m1JbE2JTLdBG0Mb6B0lEaZj181Wx5cuPXZpsbHEYgE,9116 -pip/_vendor/pyparsing/__pycache__/__init__.cpython-311.pyc,, -pip/_vendor/pyparsing/__pycache__/actions.cpython-311.pyc,, -pip/_vendor/pyparsing/__pycache__/common.cpython-311.pyc,, -pip/_vendor/pyparsing/__pycache__/core.cpython-311.pyc,, -pip/_vendor/pyparsing/__pycache__/exceptions.cpython-311.pyc,, -pip/_vendor/pyparsing/__pycache__/helpers.cpython-311.pyc,, -pip/_vendor/pyparsing/__pycache__/results.cpython-311.pyc,, -pip/_vendor/pyparsing/__pycache__/testing.cpython-311.pyc,, -pip/_vendor/pyparsing/__pycache__/unicode.cpython-311.pyc,, -pip/_vendor/pyparsing/__pycache__/util.cpython-311.pyc,, -pip/_vendor/pyparsing/actions.py,sha256=05uaIPOznJPQ7VgRdmGCmG4sDnUPtwgv5qOYIqbL2UY,6567 -pip/_vendor/pyparsing/common.py,sha256=p-3c83E5-DjlkF35G0O9-kjQRpoejP-2_z0hxZ-eol4,13387 -pip/_vendor/pyparsing/core.py,sha256=yvuRlLpXSF8mgk-QhiW3OVLqD9T0rsj9tbibhRH4Yaw,224445 -pip/_vendor/pyparsing/diagram/__init__.py,sha256=nxmDOoYF9NXuLaGYy01tKFjkNReWJlrGFuJNWEiTo84,24215 -pip/_vendor/pyparsing/diagram/__pycache__/__init__.cpython-311.pyc,, -pip/_vendor/pyparsing/exceptions.py,sha256=6Jc6W1eDZBzyFu1J0YrcdNFVBC-RINujZmveSnB8Rxw,9523 -pip/_vendor/pyparsing/helpers.py,sha256=BZJHCA8SS0pYio30KGQTc9w2qMOaK4YpZ7hcvHbnTgk,38646 -pip/_vendor/pyparsing/results.py,sha256=9dyqQ-w3MjfmxWbFt8KEPU6IfXeyRdoWp2Og802rUQY,26692 -pip/_vendor/pyparsing/testing.py,sha256=eJncg0p83zm1FTPvM9auNT6oavIvXaibmRFDf1qmwkY,13488 -pip/_vendor/pyparsing/unicode.py,sha256=fAPdsJiARFbkPAih6NkYry0dpj4jPqelGVMlE4wWFW8,10646 -pip/_vendor/pyparsing/util.py,sha256=vTMzTdwSDyV8d_dSgquUTdWgBFoA_W30nfxEJDsshRQ,8670 -pip/_vendor/pyproject_hooks/__init__.py,sha256=kCehmy0UaBa9oVMD7ZIZrnswfnP3LXZ5lvnNJAL5JBM,491 -pip/_vendor/pyproject_hooks/__pycache__/__init__.cpython-311.pyc,, -pip/_vendor/pyproject_hooks/__pycache__/_compat.cpython-311.pyc,, -pip/_vendor/pyproject_hooks/__pycache__/_impl.cpython-311.pyc,, -pip/_vendor/pyproject_hooks/_compat.py,sha256=by6evrYnqkisiM-MQcvOKs5bgDMzlOSgZqRHNqf04zE,138 -pip/_vendor/pyproject_hooks/_impl.py,sha256=61GJxzQip0IInhuO69ZI5GbNQ82XEDUB_1Gg5_KtUoc,11920 -pip/_vendor/pyproject_hooks/_in_process/__init__.py,sha256=9gQATptbFkelkIy0OfWFEACzqxXJMQDWCH9rBOAZVwQ,546 -pip/_vendor/pyproject_hooks/_in_process/__pycache__/__init__.cpython-311.pyc,, -pip/_vendor/pyproject_hooks/_in_process/__pycache__/_in_process.cpython-311.pyc,, -pip/_vendor/pyproject_hooks/_in_process/_in_process.py,sha256=m2b34c917IW5o-Q_6TYIHlsK9lSUlNiyrITTUH_zwew,10927 -pip/_vendor/requests/__init__.py,sha256=owujob4dk45Siy4EYtbCKR6wcFph7E04a_v_OuAacBA,5169 -pip/_vendor/requests/__pycache__/__init__.cpython-311.pyc,, -pip/_vendor/requests/__pycache__/__version__.cpython-311.pyc,, -pip/_vendor/requests/__pycache__/_internal_utils.cpython-311.pyc,, -pip/_vendor/requests/__pycache__/adapters.cpython-311.pyc,, -pip/_vendor/requests/__pycache__/api.cpython-311.pyc,, -pip/_vendor/requests/__pycache__/auth.cpython-311.pyc,, -pip/_vendor/requests/__pycache__/certs.cpython-311.pyc,, -pip/_vendor/requests/__pycache__/compat.cpython-311.pyc,, -pip/_vendor/requests/__pycache__/cookies.cpython-311.pyc,, -pip/_vendor/requests/__pycache__/exceptions.cpython-311.pyc,, -pip/_vendor/requests/__pycache__/help.cpython-311.pyc,, -pip/_vendor/requests/__pycache__/hooks.cpython-311.pyc,, -pip/_vendor/requests/__pycache__/models.cpython-311.pyc,, -pip/_vendor/requests/__pycache__/packages.cpython-311.pyc,, -pip/_vendor/requests/__pycache__/sessions.cpython-311.pyc,, -pip/_vendor/requests/__pycache__/status_codes.cpython-311.pyc,, -pip/_vendor/requests/__pycache__/structures.cpython-311.pyc,, -pip/_vendor/requests/__pycache__/utils.cpython-311.pyc,, -pip/_vendor/requests/__version__.py,sha256=ssI3Ezt7PaxgkOW45GhtwPUclo_SO_ygtIm4A74IOfw,435 -pip/_vendor/requests/_internal_utils.py,sha256=nMQymr4hs32TqVo5AbCrmcJEhvPUh7xXlluyqwslLiQ,1495 -pip/_vendor/requests/adapters.py,sha256=idj6cZcId3L5xNNeJ7ieOLtw3awJk5A64xUfetHwq3M,19697 -pip/_vendor/requests/api.py,sha256=q61xcXq4tmiImrvcSVLTbFyCiD2F-L_-hWKGbz4y8vg,6449 -pip/_vendor/requests/auth.py,sha256=h-HLlVx9j8rKV5hfSAycP2ApOSglTz77R0tz7qCbbEE,10187 -pip/_vendor/requests/certs.py,sha256=PVPooB0jP5hkZEULSCwC074532UFbR2Ptgu0I5zwmCs,575 -pip/_vendor/requests/compat.py,sha256=IhK9quyX0RRuWTNcg6d2JGSAOUbM6mym2p_2XjLTwf4,1286 -pip/_vendor/requests/cookies.py,sha256=kD3kNEcCj-mxbtf5fJsSaT86eGoEYpD3X0CSgpzl7BM,18560 -pip/_vendor/requests/exceptions.py,sha256=FA-_kVwBZ2jhXauRctN_ewHVK25b-fj0Azyz1THQ0Kk,3823 -pip/_vendor/requests/help.py,sha256=FnAAklv8MGm_qb2UilDQgS6l0cUttiCFKUjx0zn2XNA,3879 -pip/_vendor/requests/hooks.py,sha256=CiuysiHA39V5UfcCBXFIx83IrDpuwfN9RcTUgv28ftQ,733 -pip/_vendor/requests/models.py,sha256=dDZ-iThotky-Noq9yy97cUEJhr3wnY6mv-xR_ePg_lk,35288 -pip/_vendor/requests/packages.py,sha256=njJmVifY4aSctuW3PP5EFRCxjEwMRDO6J_feG2dKWsI,695 -pip/_vendor/requests/sessions.py,sha256=-LvTzrPtetSTrR3buxu4XhdgMrJFLB1q5D7P--L2Xhw,30373 -pip/_vendor/requests/status_codes.py,sha256=FvHmT5uH-_uimtRz5hH9VCbt7VV-Nei2J9upbej6j8g,4235 -pip/_vendor/requests/structures.py,sha256=-IbmhVz06S-5aPSZuUthZ6-6D9XOjRuTXHOabY041XM,2912 -pip/_vendor/requests/utils.py,sha256=kOPn0qYD6xRTzaxbqTdYiSInBZHl6379AJsyIgzYGLY,33460 -pip/_vendor/resolvelib/__init__.py,sha256=h509TdEcpb5-44JonaU3ex2TM15GVBLjM9CNCPwnTTs,537 -pip/_vendor/resolvelib/__pycache__/__init__.cpython-311.pyc,, -pip/_vendor/resolvelib/__pycache__/providers.cpython-311.pyc,, -pip/_vendor/resolvelib/__pycache__/reporters.cpython-311.pyc,, -pip/_vendor/resolvelib/__pycache__/resolvers.cpython-311.pyc,, -pip/_vendor/resolvelib/__pycache__/structs.cpython-311.pyc,, -pip/_vendor/resolvelib/compat/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0 -pip/_vendor/resolvelib/compat/__pycache__/__init__.cpython-311.pyc,, -pip/_vendor/resolvelib/compat/__pycache__/collections_abc.cpython-311.pyc,, -pip/_vendor/resolvelib/compat/collections_abc.py,sha256=uy8xUZ-NDEw916tugUXm8HgwCGiMO0f-RcdnpkfXfOs,156 -pip/_vendor/resolvelib/providers.py,sha256=fuuvVrCetu5gsxPB43ERyjfO8aReS3rFQHpDgiItbs4,5871 -pip/_vendor/resolvelib/reporters.py,sha256=TSbRmWzTc26w0ggsV1bxVpeWDB8QNIre6twYl7GIZBE,1601 -pip/_vendor/resolvelib/resolvers.py,sha256=G8rsLZSq64g5VmIq-lB7UcIJ1gjAxIQJmTF4REZleQ0,20511 -pip/_vendor/resolvelib/structs.py,sha256=0_1_XO8z_CLhegP3Vpf9VJ3zJcfLm0NOHRM-i0Ykz3o,4963 -pip/_vendor/rich/__init__.py,sha256=dRxjIL-SbFVY0q3IjSMrfgBTHrm1LZDgLOygVBwiYZc,6090 -pip/_vendor/rich/__main__.py,sha256=TT8sb9PTnsnKhhrGuHkLN0jdN0dtKhtPkEr9CidDbPM,8478 -pip/_vendor/rich/__pycache__/__init__.cpython-311.pyc,, -pip/_vendor/rich/__pycache__/__main__.cpython-311.pyc,, -pip/_vendor/rich/__pycache__/_cell_widths.cpython-311.pyc,, -pip/_vendor/rich/__pycache__/_emoji_codes.cpython-311.pyc,, -pip/_vendor/rich/__pycache__/_emoji_replace.cpython-311.pyc,, -pip/_vendor/rich/__pycache__/_export_format.cpython-311.pyc,, -pip/_vendor/rich/__pycache__/_extension.cpython-311.pyc,, -pip/_vendor/rich/__pycache__/_fileno.cpython-311.pyc,, -pip/_vendor/rich/__pycache__/_inspect.cpython-311.pyc,, -pip/_vendor/rich/__pycache__/_log_render.cpython-311.pyc,, -pip/_vendor/rich/__pycache__/_loop.cpython-311.pyc,, -pip/_vendor/rich/__pycache__/_null_file.cpython-311.pyc,, -pip/_vendor/rich/__pycache__/_palettes.cpython-311.pyc,, -pip/_vendor/rich/__pycache__/_pick.cpython-311.pyc,, -pip/_vendor/rich/__pycache__/_ratio.cpython-311.pyc,, -pip/_vendor/rich/__pycache__/_spinners.cpython-311.pyc,, -pip/_vendor/rich/__pycache__/_stack.cpython-311.pyc,, -pip/_vendor/rich/__pycache__/_timer.cpython-311.pyc,, -pip/_vendor/rich/__pycache__/_win32_console.cpython-311.pyc,, -pip/_vendor/rich/__pycache__/_windows.cpython-311.pyc,, -pip/_vendor/rich/__pycache__/_windows_renderer.cpython-311.pyc,, -pip/_vendor/rich/__pycache__/_wrap.cpython-311.pyc,, -pip/_vendor/rich/__pycache__/abc.cpython-311.pyc,, -pip/_vendor/rich/__pycache__/align.cpython-311.pyc,, -pip/_vendor/rich/__pycache__/ansi.cpython-311.pyc,, -pip/_vendor/rich/__pycache__/bar.cpython-311.pyc,, -pip/_vendor/rich/__pycache__/box.cpython-311.pyc,, -pip/_vendor/rich/__pycache__/cells.cpython-311.pyc,, -pip/_vendor/rich/__pycache__/color.cpython-311.pyc,, -pip/_vendor/rich/__pycache__/color_triplet.cpython-311.pyc,, -pip/_vendor/rich/__pycache__/columns.cpython-311.pyc,, -pip/_vendor/rich/__pycache__/console.cpython-311.pyc,, -pip/_vendor/rich/__pycache__/constrain.cpython-311.pyc,, -pip/_vendor/rich/__pycache__/containers.cpython-311.pyc,, -pip/_vendor/rich/__pycache__/control.cpython-311.pyc,, -pip/_vendor/rich/__pycache__/default_styles.cpython-311.pyc,, -pip/_vendor/rich/__pycache__/diagnose.cpython-311.pyc,, -pip/_vendor/rich/__pycache__/emoji.cpython-311.pyc,, -pip/_vendor/rich/__pycache__/errors.cpython-311.pyc,, -pip/_vendor/rich/__pycache__/file_proxy.cpython-311.pyc,, -pip/_vendor/rich/__pycache__/filesize.cpython-311.pyc,, -pip/_vendor/rich/__pycache__/highlighter.cpython-311.pyc,, -pip/_vendor/rich/__pycache__/json.cpython-311.pyc,, -pip/_vendor/rich/__pycache__/jupyter.cpython-311.pyc,, -pip/_vendor/rich/__pycache__/layout.cpython-311.pyc,, -pip/_vendor/rich/__pycache__/live.cpython-311.pyc,, -pip/_vendor/rich/__pycache__/live_render.cpython-311.pyc,, -pip/_vendor/rich/__pycache__/logging.cpython-311.pyc,, -pip/_vendor/rich/__pycache__/markup.cpython-311.pyc,, -pip/_vendor/rich/__pycache__/measure.cpython-311.pyc,, -pip/_vendor/rich/__pycache__/padding.cpython-311.pyc,, -pip/_vendor/rich/__pycache__/pager.cpython-311.pyc,, -pip/_vendor/rich/__pycache__/palette.cpython-311.pyc,, -pip/_vendor/rich/__pycache__/panel.cpython-311.pyc,, -pip/_vendor/rich/__pycache__/pretty.cpython-311.pyc,, -pip/_vendor/rich/__pycache__/progress.cpython-311.pyc,, -pip/_vendor/rich/__pycache__/progress_bar.cpython-311.pyc,, -pip/_vendor/rich/__pycache__/prompt.cpython-311.pyc,, -pip/_vendor/rich/__pycache__/protocol.cpython-311.pyc,, -pip/_vendor/rich/__pycache__/region.cpython-311.pyc,, -pip/_vendor/rich/__pycache__/repr.cpython-311.pyc,, -pip/_vendor/rich/__pycache__/rule.cpython-311.pyc,, -pip/_vendor/rich/__pycache__/scope.cpython-311.pyc,, -pip/_vendor/rich/__pycache__/screen.cpython-311.pyc,, -pip/_vendor/rich/__pycache__/segment.cpython-311.pyc,, -pip/_vendor/rich/__pycache__/spinner.cpython-311.pyc,, -pip/_vendor/rich/__pycache__/status.cpython-311.pyc,, -pip/_vendor/rich/__pycache__/style.cpython-311.pyc,, -pip/_vendor/rich/__pycache__/styled.cpython-311.pyc,, -pip/_vendor/rich/__pycache__/syntax.cpython-311.pyc,, -pip/_vendor/rich/__pycache__/table.cpython-311.pyc,, -pip/_vendor/rich/__pycache__/terminal_theme.cpython-311.pyc,, -pip/_vendor/rich/__pycache__/text.cpython-311.pyc,, -pip/_vendor/rich/__pycache__/theme.cpython-311.pyc,, -pip/_vendor/rich/__pycache__/themes.cpython-311.pyc,, -pip/_vendor/rich/__pycache__/traceback.cpython-311.pyc,, -pip/_vendor/rich/__pycache__/tree.cpython-311.pyc,, -pip/_vendor/rich/_cell_widths.py,sha256=2n4EiJi3X9sqIq0O16kUZ_zy6UYMd3xFfChlKfnW1Hc,10096 -pip/_vendor/rich/_emoji_codes.py,sha256=hu1VL9nbVdppJrVoijVshRlcRRe_v3dju3Mmd2sKZdY,140235 -pip/_vendor/rich/_emoji_replace.py,sha256=n-kcetsEUx2ZUmhQrfeMNc-teeGhpuSQ5F8VPBsyvDo,1064 -pip/_vendor/rich/_export_format.py,sha256=qxgV3nKnXQu1hfbnRVswPYy-AwIg1X0LSC47cK5s8jk,2100 -pip/_vendor/rich/_extension.py,sha256=Xt47QacCKwYruzjDi-gOBq724JReDj9Cm9xUi5fr-34,265 -pip/_vendor/rich/_fileno.py,sha256=HWZxP5C2ajMbHryvAQZseflVfQoGzsKOHzKGsLD8ynQ,799 -pip/_vendor/rich/_inspect.py,sha256=oZJGw31e64dwXSCmrDnvZbwVb1ZKhWfU8wI3VWohjJk,9695 -pip/_vendor/rich/_log_render.py,sha256=1ByI0PA1ZpxZY3CGJOK54hjlq4X-Bz_boIjIqCd8Kns,3225 -pip/_vendor/rich/_loop.py,sha256=hV_6CLdoPm0va22Wpw4zKqM0RYsz3TZxXj0PoS-9eDQ,1236 -pip/_vendor/rich/_null_file.py,sha256=tGSXk_v-IZmbj1GAzHit8A3kYIQMiCpVsCFfsC-_KJ4,1387 -pip/_vendor/rich/_palettes.py,sha256=cdev1JQKZ0JvlguV9ipHgznTdnvlIzUFDBb0It2PzjI,7063 -pip/_vendor/rich/_pick.py,sha256=evDt8QN4lF5CiwrUIXlOJCntitBCOsI3ZLPEIAVRLJU,423 -pip/_vendor/rich/_ratio.py,sha256=2lLSliL025Y-YMfdfGbutkQDevhcyDqc-DtUYW9mU70,5472 -pip/_vendor/rich/_spinners.py,sha256=U2r1_g_1zSjsjiUdAESc2iAMc3i4ri_S8PYP6kQ5z1I,19919 -pip/_vendor/rich/_stack.py,sha256=-C8OK7rxn3sIUdVwxZBBpeHhIzX0eI-VM3MemYfaXm0,351 -pip/_vendor/rich/_timer.py,sha256=zelxbT6oPFZnNrwWPpc1ktUeAT-Vc4fuFcRZLQGLtMI,417 -pip/_vendor/rich/_win32_console.py,sha256=P0vxI2fcndym1UU1S37XAzQzQnkyY7YqAKmxm24_gug,22820 -pip/_vendor/rich/_windows.py,sha256=dvNl9TmfPzNVxiKk5WDFihErZ5796g2UC9-KGGyfXmk,1926 -pip/_vendor/rich/_windows_renderer.py,sha256=t74ZL3xuDCP3nmTp9pH1L5LiI2cakJuQRQleHCJerlk,2783 -pip/_vendor/rich/_wrap.py,sha256=xfV_9t0Sg6rzimmrDru8fCVmUlalYAcHLDfrJZnbbwQ,1840 -pip/_vendor/rich/abc.py,sha256=ON-E-ZqSSheZ88VrKX2M3PXpFbGEUUZPMa_Af0l-4f0,890 -pip/_vendor/rich/align.py,sha256=Ji-Yokfkhnfe_xMmr4ISjZB07TJXggBCOYoYa-HDAr8,10368 -pip/_vendor/rich/ansi.py,sha256=iD6532QYqnBm6hADulKjrV8l8kFJ-9fEVooHJHH3hMg,6906 -pip/_vendor/rich/bar.py,sha256=a7UD303BccRCrEhGjfMElpv5RFYIinaAhAuqYqhUvmw,3264 -pip/_vendor/rich/box.py,sha256=FJ6nI3jD7h2XNFU138bJUt2HYmWOlRbltoCEuIAZhew,9842 -pip/_vendor/rich/cells.py,sha256=627ztJs9zOL-38HJ7kXBerR-gT8KBfYC8UzEwMJDYYo,4509 -pip/_vendor/rich/color.py,sha256=9Gh958U3f75WVdLTeC0U9nkGTn2n0wnojKpJ6jQEkIE,18224 -pip/_vendor/rich/color_triplet.py,sha256=3lhQkdJbvWPoLDO-AnYImAWmJvV5dlgYNCVZ97ORaN4,1054 -pip/_vendor/rich/columns.py,sha256=HUX0KcMm9dsKNi11fTbiM_h2iDtl8ySCaVcxlalEzq8,7131 -pip/_vendor/rich/console.py,sha256=pDvkbLkvtZIMIwQx_jkZ-seyNl4zGBLviXoWXte9fwg,99218 -pip/_vendor/rich/constrain.py,sha256=1VIPuC8AgtKWrcncQrjBdYqA3JVWysu6jZo1rrh7c7Q,1288 -pip/_vendor/rich/containers.py,sha256=aKgm5UDHn5Nmui6IJaKdsZhbHClh_X7D-_Wg8Ehrr7s,5497 -pip/_vendor/rich/control.py,sha256=DSkHTUQLorfSERAKE_oTAEUFefZnZp4bQb4q8rHbKws,6630 -pip/_vendor/rich/default_styles.py,sha256=-Fe318kMVI_IwciK5POpThcO0-9DYJ67TZAN6DlmlmM,8082 -pip/_vendor/rich/diagnose.py,sha256=an6uouwhKPAlvQhYpNNpGq9EJysfMIOvvCbO3oSoR24,972 -pip/_vendor/rich/emoji.py,sha256=omTF9asaAnsM4yLY94eR_9dgRRSm1lHUszX20D1yYCQ,2501 -pip/_vendor/rich/errors.py,sha256=5pP3Kc5d4QJ_c0KFsxrfyhjiPVe7J1zOqSFbFAzcV-Y,642 -pip/_vendor/rich/file_proxy.py,sha256=Tl9THMDZ-Pk5Wm8sI1gGg_U5DhusmxD-FZ0fUbcU0W0,1683 -pip/_vendor/rich/filesize.py,sha256=9fTLAPCAwHmBXdRv7KZU194jSgNrRb6Wx7RIoBgqeKY,2508 -pip/_vendor/rich/highlighter.py,sha256=p3C1g4QYzezFKdR7NF9EhPbzQDvdPUhGRgSyGGEmPko,9584 -pip/_vendor/rich/json.py,sha256=EYp9ucj-nDjYDkHCV6Mk1ve8nUOpuFLaW76X50Mis2M,5032 -pip/_vendor/rich/jupyter.py,sha256=QyoKoE_8IdCbrtiSHp9TsTSNyTHY0FO5whE7jOTd9UE,3252 -pip/_vendor/rich/layout.py,sha256=RFYL6HdCFsHf9WRpcvi3w-fpj-8O5dMZ8W96VdKNdbI,14007 -pip/_vendor/rich/live.py,sha256=vZzYvu7fqwlv3Gthl2xiw1Dc_O80VlGcCV0DOHwCyDM,14273 -pip/_vendor/rich/live_render.py,sha256=zElm3PrfSIvjOce28zETHMIUf9pFYSUA5o0AflgUP64,3667 -pip/_vendor/rich/logging.py,sha256=uB-cB-3Q4bmXDLLpbOWkmFviw-Fde39zyMV6tKJ2WHQ,11903 -pip/_vendor/rich/markup.py,sha256=xzF4uAafiEeEYDJYt_vUnJOGoTU8RrH-PH7WcWYXjCg,8198 -pip/_vendor/rich/measure.py,sha256=HmrIJX8sWRTHbgh8MxEay_83VkqNW_70s8aKP5ZcYI8,5305 -pip/_vendor/rich/padding.py,sha256=kTFGsdGe0os7tXLnHKpwTI90CXEvrceeZGCshmJy5zw,4970 -pip/_vendor/rich/pager.py,sha256=SO_ETBFKbg3n_AgOzXm41Sv36YxXAyI3_R-KOY2_uSc,828 -pip/_vendor/rich/palette.py,sha256=lInvR1ODDT2f3UZMfL1grq7dY_pDdKHw4bdUgOGaM4Y,3396 -pip/_vendor/rich/panel.py,sha256=wGMe40J8KCGgQoM0LyjRErmGIkv2bsYA71RCXThD0xE,10574 -pip/_vendor/rich/pretty.py,sha256=eLEYN9xVaMNuA6EJVYm4li7HdOHxCqmVKvnOqJpyFt0,35852 -pip/_vendor/rich/progress.py,sha256=n4KF9vky8_5iYeXcyZPEvzyLplWlDvFLkM5JI0Bs08A,59706 -pip/_vendor/rich/progress_bar.py,sha256=cEoBfkc3lLwqba4XKsUpy4vSQKDh2QQ5J2J94-ACFoo,8165 -pip/_vendor/rich/prompt.py,sha256=x0mW-pIPodJM4ry6grgmmLrl8VZp99kqcmdnBe70YYA,11303 -pip/_vendor/rich/protocol.py,sha256=5hHHDDNHckdk8iWH5zEbi-zuIVSF5hbU2jIo47R7lTE,1391 -pip/_vendor/rich/region.py,sha256=rNT9xZrVZTYIXZC0NYn41CJQwYNbR-KecPOxTgQvB8Y,166 -pip/_vendor/rich/repr.py,sha256=9Z8otOmM-tyxnyTodvXlectP60lwahjGiDTrbrxPSTg,4431 -pip/_vendor/rich/rule.py,sha256=0fNaS_aERa3UMRc3T5WMpN_sumtDxfaor2y3of1ftBk,4602 -pip/_vendor/rich/scope.py,sha256=TMUU8qo17thyqQCPqjDLYpg_UU1k5qVd-WwiJvnJVas,2843 -pip/_vendor/rich/screen.py,sha256=YoeReESUhx74grqb0mSSb9lghhysWmFHYhsbMVQjXO8,1591 -pip/_vendor/rich/segment.py,sha256=XLnJEFvcV3bjaVzMNUJiem3n8lvvI9TJ5PTu-IG2uTg,24247 -pip/_vendor/rich/spinner.py,sha256=15koCmF0DQeD8-k28Lpt6X_zJQUlzEhgo_6A6uy47lc,4339 -pip/_vendor/rich/status.py,sha256=gJsIXIZeSo3urOyxRUjs6VrhX5CZrA0NxIQ-dxhCnwo,4425 -pip/_vendor/rich/style.py,sha256=3hiocH_4N8vwRm3-8yFWzM7tSwjjEven69XqWasSQwM,27073 -pip/_vendor/rich/styled.py,sha256=eZNnzGrI4ki_54pgY3Oj0T-x3lxdXTYh4_ryDB24wBU,1258 -pip/_vendor/rich/syntax.py,sha256=jgDiVCK6cpR0NmBOpZmIu-Ud4eaW7fHvjJZkDbjpcSA,35173 -pip/_vendor/rich/table.py,sha256=-WzesL-VJKsaiDU3uyczpJMHy6VCaSewBYJwx8RudI8,39684 -pip/_vendor/rich/terminal_theme.py,sha256=1j5-ufJfnvlAo5Qsi_ACZiXDmwMXzqgmFByObT9-yJY,3370 -pip/_vendor/rich/text.py,sha256=_8JBlSau0c2z8ENOZMi1hJ7M1ZGY408E4-hXjHyyg1A,45525 -pip/_vendor/rich/theme.py,sha256=belFJogzA0W0HysQabKaHOc3RWH2ko3fQAJhoN-AFdo,3777 -pip/_vendor/rich/themes.py,sha256=0xgTLozfabebYtcJtDdC5QkX5IVUEaviqDUJJh4YVFk,102 -pip/_vendor/rich/traceback.py,sha256=yCLVrCtyoFNENd9mkm2xeG3KmqkTwH9xpFOO7p2Bq0A,29604 -pip/_vendor/rich/tree.py,sha256=BMbUYNjS9uodNPfvtY_odmU09GA5QzcMbQ5cJZhllQI,9169 -pip/_vendor/six.py,sha256=TOOfQi7nFGfMrIvtdr6wX4wyHH8M7aknmuLfo2cBBrM,34549 -pip/_vendor/tenacity/__init__.py,sha256=3kvAL6KClq8GFo2KFhmOzskRKSDQI-ubrlfZ8AQEEI0,20493 -pip/_vendor/tenacity/__pycache__/__init__.cpython-311.pyc,, -pip/_vendor/tenacity/__pycache__/_asyncio.cpython-311.pyc,, -pip/_vendor/tenacity/__pycache__/_utils.cpython-311.pyc,, -pip/_vendor/tenacity/__pycache__/after.cpython-311.pyc,, -pip/_vendor/tenacity/__pycache__/before.cpython-311.pyc,, -pip/_vendor/tenacity/__pycache__/before_sleep.cpython-311.pyc,, -pip/_vendor/tenacity/__pycache__/nap.cpython-311.pyc,, -pip/_vendor/tenacity/__pycache__/retry.cpython-311.pyc,, -pip/_vendor/tenacity/__pycache__/stop.cpython-311.pyc,, -pip/_vendor/tenacity/__pycache__/tornadoweb.cpython-311.pyc,, -pip/_vendor/tenacity/__pycache__/wait.cpython-311.pyc,, -pip/_vendor/tenacity/_asyncio.py,sha256=Qi6wgQsGa9MQibYRy3OXqcDQswIZZ00dLOoSUGN-6o8,3551 -pip/_vendor/tenacity/_utils.py,sha256=ubs6a7sxj3JDNRKWCyCU2j5r1CB7rgyONgZzYZq6D_4,2179 -pip/_vendor/tenacity/after.py,sha256=S5NCISScPeIrKwIeXRwdJl3kV9Q4nqZfnNPDx6Hf__g,1682 -pip/_vendor/tenacity/before.py,sha256=dIZE9gmBTffisfwNkK0F1xFwGPV41u5GK70UY4Pi5Kc,1562 -pip/_vendor/tenacity/before_sleep.py,sha256=YmpgN9Y7HGlH97U24vvq_YWb5deaK4_DbiD8ZuFmy-E,2372 -pip/_vendor/tenacity/nap.py,sha256=fRWvnz1aIzbIq9Ap3gAkAZgDH6oo5zxMrU6ZOVByq0I,1383 -pip/_vendor/tenacity/retry.py,sha256=jrzD_mxA5mSTUEdiYB7SHpxltjhPSYZSnSRATb-ggRc,8746 -pip/_vendor/tenacity/stop.py,sha256=YMJs7ZgZfND65PRLqlGB_agpfGXlemx_5Hm4PKnBqpQ,3086 -pip/_vendor/tenacity/tornadoweb.py,sha256=po29_F1Mt8qZpsFjX7EVwAT0ydC_NbVia9gVi7R_wXA,2142 -pip/_vendor/tenacity/wait.py,sha256=3FcBJoCDgym12_dN6xfK8C1gROY0Hn4NSI2u8xv50uE,8024 -pip/_vendor/tomli/__init__.py,sha256=JhUwV66DB1g4Hvt1UQCVMdfCu-IgAV8FXmvDU9onxd4,396 -pip/_vendor/tomli/__pycache__/__init__.cpython-311.pyc,, -pip/_vendor/tomli/__pycache__/_parser.cpython-311.pyc,, -pip/_vendor/tomli/__pycache__/_re.cpython-311.pyc,, -pip/_vendor/tomli/__pycache__/_types.cpython-311.pyc,, -pip/_vendor/tomli/_parser.py,sha256=g9-ENaALS-B8dokYpCuzUFalWlog7T-SIYMjLZSWrtM,22633 -pip/_vendor/tomli/_re.py,sha256=dbjg5ChZT23Ka9z9DHOXfdtSpPwUfdgMXnj8NOoly-w,2943 -pip/_vendor/tomli/_types.py,sha256=-GTG2VUqkpxwMqzmVO4F7ybKddIbAnuAHXfmWQcTi3Q,254 -pip/_vendor/truststore/__init__.py,sha256=qzTLSH8PvAkY1fr6QQ2vV-KwE_M83wdXugtpJaP_AbM,403 -pip/_vendor/truststore/__pycache__/__init__.cpython-311.pyc,, -pip/_vendor/truststore/__pycache__/_api.cpython-311.pyc,, -pip/_vendor/truststore/__pycache__/_macos.cpython-311.pyc,, -pip/_vendor/truststore/__pycache__/_openssl.cpython-311.pyc,, -pip/_vendor/truststore/__pycache__/_ssl_constants.cpython-311.pyc,, -pip/_vendor/truststore/__pycache__/_windows.cpython-311.pyc,, -pip/_vendor/truststore/_api.py,sha256=xjuEu_rlH4hcdJTROImEyOEqdw-F8t5vO2H2BToY0Ro,9893 -pip/_vendor/truststore/_macos.py,sha256=BjvAKoAjXhdIPuxpY124HJIFswDb0pq8DjynzJOVwqc,17694 -pip/_vendor/truststore/_openssl.py,sha256=LLUZ7ZGaio-i5dpKKjKCSeSufmn6T8pi9lDcFnvSyq0,2324 -pip/_vendor/truststore/_ssl_constants.py,sha256=NUD4fVKdSD02ri7-db0tnO0VqLP9aHuzmStcW7tAl08,1130 -pip/_vendor/truststore/_windows.py,sha256=1x_EhROeJ9QK1sMAjfnZC7awYI8UnBJYL-TjACUYI4A,17468 -pip/_vendor/typing_extensions.py,sha256=EWpcpyQnVmc48E9fSyPGs-vXgHcAk9tQABQIxmMsCGk,111130 -pip/_vendor/urllib3/__init__.py,sha256=iXLcYiJySn0GNbWOOZDDApgBL1JgP44EZ8i1760S8Mc,3333 -pip/_vendor/urllib3/__pycache__/__init__.cpython-311.pyc,, -pip/_vendor/urllib3/__pycache__/_collections.cpython-311.pyc,, -pip/_vendor/urllib3/__pycache__/_version.cpython-311.pyc,, -pip/_vendor/urllib3/__pycache__/connection.cpython-311.pyc,, -pip/_vendor/urllib3/__pycache__/connectionpool.cpython-311.pyc,, -pip/_vendor/urllib3/__pycache__/exceptions.cpython-311.pyc,, -pip/_vendor/urllib3/__pycache__/fields.cpython-311.pyc,, -pip/_vendor/urllib3/__pycache__/filepost.cpython-311.pyc,, -pip/_vendor/urllib3/__pycache__/poolmanager.cpython-311.pyc,, -pip/_vendor/urllib3/__pycache__/request.cpython-311.pyc,, -pip/_vendor/urllib3/__pycache__/response.cpython-311.pyc,, -pip/_vendor/urllib3/_collections.py,sha256=Rp1mVyBgc_UlAcp6M3at1skJBXR5J43NawRTvW2g_XY,10811 -pip/_vendor/urllib3/_version.py,sha256=azoM7M7BUADl2kBhMVR6PPf2GhBDI90me1fcnzTwdcw,64 -pip/_vendor/urllib3/connection.py,sha256=92k9td_y4PEiTIjNufCUa1NzMB3J3w0LEdyokYgXnW8,20300 -pip/_vendor/urllib3/connectionpool.py,sha256=ItVDasDnPRPP9R8bNxY7tPBlC724nJ9nlxVgXG_SLbI,39990 -pip/_vendor/urllib3/contrib/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0 -pip/_vendor/urllib3/contrib/__pycache__/__init__.cpython-311.pyc,, -pip/_vendor/urllib3/contrib/__pycache__/_appengine_environ.cpython-311.pyc,, -pip/_vendor/urllib3/contrib/__pycache__/appengine.cpython-311.pyc,, -pip/_vendor/urllib3/contrib/__pycache__/ntlmpool.cpython-311.pyc,, -pip/_vendor/urllib3/contrib/__pycache__/pyopenssl.cpython-311.pyc,, -pip/_vendor/urllib3/contrib/__pycache__/securetransport.cpython-311.pyc,, -pip/_vendor/urllib3/contrib/__pycache__/socks.cpython-311.pyc,, -pip/_vendor/urllib3/contrib/_appengine_environ.py,sha256=bDbyOEhW2CKLJcQqAKAyrEHN-aklsyHFKq6vF8ZFsmk,957 -pip/_vendor/urllib3/contrib/_securetransport/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0 -pip/_vendor/urllib3/contrib/_securetransport/__pycache__/__init__.cpython-311.pyc,, -pip/_vendor/urllib3/contrib/_securetransport/__pycache__/bindings.cpython-311.pyc,, -pip/_vendor/urllib3/contrib/_securetransport/__pycache__/low_level.cpython-311.pyc,, -pip/_vendor/urllib3/contrib/_securetransport/bindings.py,sha256=4Xk64qIkPBt09A5q-RIFUuDhNc9mXilVapm7WnYnzRw,17632 -pip/_vendor/urllib3/contrib/_securetransport/low_level.py,sha256=B2JBB2_NRP02xK6DCa1Pa9IuxrPwxzDzZbixQkb7U9M,13922 -pip/_vendor/urllib3/contrib/appengine.py,sha256=VR68eAVE137lxTgjBDwCna5UiBZTOKa01Aj_-5BaCz4,11036 -pip/_vendor/urllib3/contrib/ntlmpool.py,sha256=NlfkW7WMdW8ziqudopjHoW299og1BTWi0IeIibquFwk,4528 -pip/_vendor/urllib3/contrib/pyopenssl.py,sha256=hDJh4MhyY_p-oKlFcYcQaVQRDv6GMmBGuW9yjxyeejM,17081 -pip/_vendor/urllib3/contrib/securetransport.py,sha256=yhZdmVjY6PI6EeFbp7qYOp6-vp1Rkv2NMuOGaEj7pmc,34448 -pip/_vendor/urllib3/contrib/socks.py,sha256=aRi9eWXo9ZEb95XUxef4Z21CFlnnjbEiAo9HOseoMt4,7097 -pip/_vendor/urllib3/exceptions.py,sha256=0Mnno3KHTNfXRfY7638NufOPkUb6mXOm-Lqj-4x2w8A,8217 -pip/_vendor/urllib3/fields.py,sha256=kvLDCg_JmH1lLjUUEY_FLS8UhY7hBvDPuVETbY8mdrM,8579 -pip/_vendor/urllib3/filepost.py,sha256=5b_qqgRHVlL7uLtdAYBzBh-GHmU5AfJVt_2N0XS3PeY,2440 -pip/_vendor/urllib3/packages/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0 -pip/_vendor/urllib3/packages/__pycache__/__init__.cpython-311.pyc,, -pip/_vendor/urllib3/packages/__pycache__/six.cpython-311.pyc,, -pip/_vendor/urllib3/packages/backports/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0 -pip/_vendor/urllib3/packages/backports/__pycache__/__init__.cpython-311.pyc,, -pip/_vendor/urllib3/packages/backports/__pycache__/makefile.cpython-311.pyc,, -pip/_vendor/urllib3/packages/backports/__pycache__/weakref_finalize.cpython-311.pyc,, -pip/_vendor/urllib3/packages/backports/makefile.py,sha256=nbzt3i0agPVP07jqqgjhaYjMmuAi_W5E0EywZivVO8E,1417 -pip/_vendor/urllib3/packages/backports/weakref_finalize.py,sha256=tRCal5OAhNSRyb0DhHp-38AtIlCsRP8BxF3NX-6rqIA,5343 -pip/_vendor/urllib3/packages/six.py,sha256=b9LM0wBXv7E7SrbCjAm4wwN-hrH-iNxv18LgWNMMKPo,34665 -pip/_vendor/urllib3/poolmanager.py,sha256=0i8cJgrqupza67IBPZ_u9jXvnSxr5UBlVEiUqdkPtYI,19752 -pip/_vendor/urllib3/request.py,sha256=YTWFNr7QIwh7E1W9dde9LM77v2VWTJ5V78XuTTw7D1A,6691 -pip/_vendor/urllib3/response.py,sha256=fmDJAFkG71uFTn-sVSTh2Iw0WmcXQYqkbRjihvwBjU8,30641 -pip/_vendor/urllib3/util/__init__.py,sha256=JEmSmmqqLyaw8P51gUImZh8Gwg9i1zSe-DoqAitn2nc,1155 -pip/_vendor/urllib3/util/__pycache__/__init__.cpython-311.pyc,, -pip/_vendor/urllib3/util/__pycache__/connection.cpython-311.pyc,, -pip/_vendor/urllib3/util/__pycache__/proxy.cpython-311.pyc,, -pip/_vendor/urllib3/util/__pycache__/queue.cpython-311.pyc,, -pip/_vendor/urllib3/util/__pycache__/request.cpython-311.pyc,, -pip/_vendor/urllib3/util/__pycache__/response.cpython-311.pyc,, -pip/_vendor/urllib3/util/__pycache__/retry.cpython-311.pyc,, -pip/_vendor/urllib3/util/__pycache__/ssl_.cpython-311.pyc,, -pip/_vendor/urllib3/util/__pycache__/ssl_match_hostname.cpython-311.pyc,, -pip/_vendor/urllib3/util/__pycache__/ssltransport.cpython-311.pyc,, -pip/_vendor/urllib3/util/__pycache__/timeout.cpython-311.pyc,, -pip/_vendor/urllib3/util/__pycache__/url.cpython-311.pyc,, -pip/_vendor/urllib3/util/__pycache__/wait.cpython-311.pyc,, -pip/_vendor/urllib3/util/connection.py,sha256=5Lx2B1PW29KxBn2T0xkN1CBgRBa3gGVJBKoQoRogEVk,4901 -pip/_vendor/urllib3/util/proxy.py,sha256=zUvPPCJrp6dOF0N4GAVbOcl6o-4uXKSrGiTkkr5vUS4,1605 -pip/_vendor/urllib3/util/queue.py,sha256=nRgX8_eX-_VkvxoX096QWoz8Ps0QHUAExILCY_7PncM,498 -pip/_vendor/urllib3/util/request.py,sha256=C0OUt2tcU6LRiQJ7YYNP9GvPrSvl7ziIBekQ-5nlBZk,3997 -pip/_vendor/urllib3/util/response.py,sha256=GJpg3Egi9qaJXRwBh5wv-MNuRWan5BIu40oReoxWP28,3510 -pip/_vendor/urllib3/util/retry.py,sha256=Z6WEf518eTOXP5jr5QSQ9gqJI0DVYt3Xs3EKnYaTmus,22013 -pip/_vendor/urllib3/util/ssl_.py,sha256=X4-AqW91aYPhPx6-xbf66yHFQKbqqfC_5Zt4WkLX1Hc,17177 -pip/_vendor/urllib3/util/ssl_match_hostname.py,sha256=Ir4cZVEjmAk8gUAIHWSi7wtOO83UCYABY2xFD1Ql_WA,5758 -pip/_vendor/urllib3/util/ssltransport.py,sha256=NA-u5rMTrDFDFC8QzRKUEKMG0561hOD4qBTr3Z4pv6E,6895 -pip/_vendor/urllib3/util/timeout.py,sha256=cwq4dMk87mJHSBktK1miYJ-85G-3T3RmT20v7SFCpno,10168 -pip/_vendor/urllib3/util/url.py,sha256=lCAE7M5myA8EDdW0sJuyyZhVB9K_j38ljWhHAnFaWoE,14296 -pip/_vendor/urllib3/util/wait.py,sha256=fOX0_faozG2P7iVojQoE1mbydweNyTcm-hXEfFrTtLI,5403 -pip/_vendor/vendor.txt,sha256=epuLpe-n1shCqP5BzC97iMIAIeOeDHdtNKFgcxax-9A,493 -pip/_vendor/webencodings/__init__.py,sha256=qOBJIuPy_4ByYH6W_bNgJF-qYQ2DoU-dKsDu5yRWCXg,10579 -pip/_vendor/webencodings/__pycache__/__init__.cpython-311.pyc,, -pip/_vendor/webencodings/__pycache__/labels.cpython-311.pyc,, -pip/_vendor/webencodings/__pycache__/mklabels.cpython-311.pyc,, -pip/_vendor/webencodings/__pycache__/tests.cpython-311.pyc,, -pip/_vendor/webencodings/__pycache__/x_user_defined.cpython-311.pyc,, -pip/_vendor/webencodings/labels.py,sha256=4AO_KxTddqGtrL9ns7kAPjb0CcN6xsCIxbK37HY9r3E,8979 -pip/_vendor/webencodings/mklabels.py,sha256=GYIeywnpaLnP0GSic8LFWgd0UVvO_l1Nc6YoF-87R_4,1305 -pip/_vendor/webencodings/tests.py,sha256=OtGLyjhNY1fvkW1GvLJ_FV9ZoqC9Anyjr7q3kxTbzNs,6563 -pip/_vendor/webencodings/x_user_defined.py,sha256=yOqWSdmpytGfUgh_Z6JYgDNhoc-BAHyyeeT15Fr42tM,4307 -pip/py.typed,sha256=EBVvvPRTn_eIpz5e5QztSCdrMX7Qwd7VP93RSoIlZ2I,286 diff --git a/venv/lib/python3.11/site-packages/pip-23.3.1.dist-info/REQUESTED b/venv/lib/python3.11/site-packages/pip-23.3.1.dist-info/REQUESTED deleted file mode 100644 index e69de29..0000000 diff --git a/venv/lib/python3.11/site-packages/pip-23.3.1.dist-info/WHEEL b/venv/lib/python3.11/site-packages/pip-23.3.1.dist-info/WHEEL deleted file mode 100644 index ba48cbc..0000000 --- a/venv/lib/python3.11/site-packages/pip-23.3.1.dist-info/WHEEL +++ /dev/null @@ -1,5 +0,0 @@ -Wheel-Version: 1.0 -Generator: bdist_wheel (0.41.3) -Root-Is-Purelib: true -Tag: py3-none-any - diff --git a/venv/lib/python3.11/site-packages/pip-23.3.1.dist-info/entry_points.txt b/venv/lib/python3.11/site-packages/pip-23.3.1.dist-info/entry_points.txt deleted file mode 100644 index bcf704d..0000000 --- a/venv/lib/python3.11/site-packages/pip-23.3.1.dist-info/entry_points.txt +++ /dev/null @@ -1,4 +0,0 @@ -[console_scripts] -pip = pip._internal.cli.main:main -pip3 = pip._internal.cli.main:main -pip3.11 = pip._internal.cli.main:main diff --git a/venv/lib/python3.11/site-packages/pip-23.3.1.dist-info/top_level.txt b/venv/lib/python3.11/site-packages/pip-23.3.1.dist-info/top_level.txt deleted file mode 100644 index a1b589e..0000000 --- a/venv/lib/python3.11/site-packages/pip-23.3.1.dist-info/top_level.txt +++ /dev/null @@ -1 +0,0 @@ -pip diff --git a/venv/lib/python3.11/site-packages/pip/__init__.py b/venv/lib/python3.11/site-packages/pip/__init__.py deleted file mode 100644 index f1263cd..0000000 --- a/venv/lib/python3.11/site-packages/pip/__init__.py +++ /dev/null @@ -1,13 +0,0 @@ -from typing import List, Optional - -__version__ = "23.3.1" - - -def main(args: Optional[List[str]] = None) -> int: - """This is an internal API only meant for use by pip's own console scripts. - - For additional details, see https://github.com/pypa/pip/issues/7498. - """ - from pip._internal.utils.entrypoints import _wrapper - - return _wrapper(args) diff --git a/venv/lib/python3.11/site-packages/pip/__main__.py b/venv/lib/python3.11/site-packages/pip/__main__.py deleted file mode 100644 index 5991326..0000000 --- a/venv/lib/python3.11/site-packages/pip/__main__.py +++ /dev/null @@ -1,24 +0,0 @@ -import os -import sys - -# Remove '' and current working directory from the first entry -# of sys.path, if present to avoid using current directory -# in pip commands check, freeze, install, list and show, -# when invoked as python -m pip -if sys.path[0] in ("", os.getcwd()): - sys.path.pop(0) - -# If we are running from a wheel, add the wheel to sys.path -# This allows the usage python pip-*.whl/pip install pip-*.whl -if __package__ == "": - # __file__ is pip-*.whl/pip/__main__.py - # first dirname call strips of '/__main__.py', second strips off '/pip' - # Resulting path is the name of the wheel itself - # Add that to sys.path so we can import pip - path = os.path.dirname(os.path.dirname(__file__)) - sys.path.insert(0, path) - -if __name__ == "__main__": - from pip._internal.cli.main import main as _main - - sys.exit(_main()) diff --git a/venv/lib/python3.11/site-packages/pip/__pip-runner__.py b/venv/lib/python3.11/site-packages/pip/__pip-runner__.py deleted file mode 100644 index 49a148a..0000000 --- a/venv/lib/python3.11/site-packages/pip/__pip-runner__.py +++ /dev/null @@ -1,50 +0,0 @@ -"""Execute exactly this copy of pip, within a different environment. - -This file is named as it is, to ensure that this module can't be imported via -an import statement. -""" - -# /!\ This version compatibility check section must be Python 2 compatible. /!\ - -import sys - -# Copied from setup.py -PYTHON_REQUIRES = (3, 7) - - -def version_str(version): # type: ignore - return ".".join(str(v) for v in version) - - -if sys.version_info[:2] < PYTHON_REQUIRES: - raise SystemExit( - "This version of pip does not support python {} (requires >={}).".format( - version_str(sys.version_info[:2]), version_str(PYTHON_REQUIRES) - ) - ) - -# From here on, we can use Python 3 features, but the syntax must remain -# Python 2 compatible. - -import runpy # noqa: E402 -from importlib.machinery import PathFinder # noqa: E402 -from os.path import dirname # noqa: E402 - -PIP_SOURCES_ROOT = dirname(dirname(__file__)) - - -class PipImportRedirectingFinder: - @classmethod - def find_spec(self, fullname, path=None, target=None): # type: ignore - if fullname != "pip": - return None - - spec = PathFinder.find_spec(fullname, [PIP_SOURCES_ROOT], target) - assert spec, (PIP_SOURCES_ROOT, fullname) - return spec - - -sys.meta_path.insert(0, PipImportRedirectingFinder()) - -assert __name__ == "__main__", "Cannot run __pip-runner__.py as a non-main module" -runpy.run_module("pip", run_name="__main__", alter_sys=True) diff --git a/venv/lib/python3.11/site-packages/pip/__pycache__/__init__.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/__pycache__/__init__.cpython-311.pyc deleted file mode 100644 index ac9fee0..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/__pycache__/__init__.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/__pycache__/__main__.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/__pycache__/__main__.cpython-311.pyc deleted file mode 100644 index 92e75b0..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/__pycache__/__main__.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/__pycache__/__pip-runner__.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/__pycache__/__pip-runner__.cpython-311.pyc deleted file mode 100644 index 7166ae3..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/__pycache__/__pip-runner__.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/__init__.py b/venv/lib/python3.11/site-packages/pip/_internal/__init__.py deleted file mode 100644 index 96c6b88..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/__init__.py +++ /dev/null @@ -1,18 +0,0 @@ -from typing import List, Optional - -from pip._internal.utils import _log - -# init_logging() must be called before any call to logging.getLogger() -# which happens at import of most modules. -_log.init_logging() - - -def main(args: (Optional[List[str]]) = None) -> int: - """This is preserved for old console scripts that may still be referencing - it. - - For additional details, see https://github.com/pypa/pip/issues/7498. - """ - from pip._internal.utils.entrypoints import _wrapper - - return _wrapper(args) diff --git a/venv/lib/python3.11/site-packages/pip/_internal/__pycache__/__init__.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/__pycache__/__init__.cpython-311.pyc deleted file mode 100644 index 2ad00aa..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/__pycache__/__init__.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/__pycache__/build_env.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/__pycache__/build_env.cpython-311.pyc deleted file mode 100644 index a1d20f8..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/__pycache__/build_env.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/__pycache__/cache.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/__pycache__/cache.cpython-311.pyc deleted file mode 100644 index e4cb82e..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/__pycache__/cache.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/__pycache__/configuration.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/__pycache__/configuration.cpython-311.pyc deleted file mode 100644 index 54ad6b0..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/__pycache__/configuration.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/__pycache__/exceptions.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/__pycache__/exceptions.cpython-311.pyc deleted file mode 100644 index 4a9396d..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/__pycache__/exceptions.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/__pycache__/main.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/__pycache__/main.cpython-311.pyc deleted file mode 100644 index e87ae91..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/__pycache__/main.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/__pycache__/pyproject.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/__pycache__/pyproject.cpython-311.pyc deleted file mode 100644 index 0f260db..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/__pycache__/pyproject.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/__pycache__/self_outdated_check.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/__pycache__/self_outdated_check.cpython-311.pyc deleted file mode 100644 index 10429e5..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/__pycache__/self_outdated_check.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/__pycache__/wheel_builder.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/__pycache__/wheel_builder.cpython-311.pyc deleted file mode 100644 index 6f81c5f..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/__pycache__/wheel_builder.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/build_env.py b/venv/lib/python3.11/site-packages/pip/_internal/build_env.py deleted file mode 100644 index 4f704a3..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/build_env.py +++ /dev/null @@ -1,311 +0,0 @@ -"""Build Environment used for isolation during sdist building -""" - -import logging -import os -import pathlib -import site -import sys -import textwrap -from collections import OrderedDict -from types import TracebackType -from typing import TYPE_CHECKING, Iterable, List, Optional, Set, Tuple, Type, Union - -from pip._vendor.certifi import where -from pip._vendor.packaging.requirements import Requirement -from pip._vendor.packaging.version import Version - -from pip import __file__ as pip_location -from pip._internal.cli.spinners import open_spinner -from pip._internal.locations import get_platlib, get_purelib, get_scheme -from pip._internal.metadata import get_default_environment, get_environment -from pip._internal.utils.subprocess import call_subprocess -from pip._internal.utils.temp_dir import TempDirectory, tempdir_kinds - -if TYPE_CHECKING: - from pip._internal.index.package_finder import PackageFinder - -logger = logging.getLogger(__name__) - - -def _dedup(a: str, b: str) -> Union[Tuple[str], Tuple[str, str]]: - return (a, b) if a != b else (a,) - - -class _Prefix: - def __init__(self, path: str) -> None: - self.path = path - self.setup = False - scheme = get_scheme("", prefix=path) - self.bin_dir = scheme.scripts - self.lib_dirs = _dedup(scheme.purelib, scheme.platlib) - - -def get_runnable_pip() -> str: - """Get a file to pass to a Python executable, to run the currently-running pip. - - This is used to run a pip subprocess, for installing requirements into the build - environment. - """ - source = pathlib.Path(pip_location).resolve().parent - - if not source.is_dir(): - # This would happen if someone is using pip from inside a zip file. In that - # case, we can use that directly. - return str(source) - - return os.fsdecode(source / "__pip-runner__.py") - - -def _get_system_sitepackages() -> Set[str]: - """Get system site packages - - Usually from site.getsitepackages, - but fallback on `get_purelib()/get_platlib()` if unavailable - (e.g. in a virtualenv created by virtualenv<20) - - Returns normalized set of strings. - """ - if hasattr(site, "getsitepackages"): - system_sites = site.getsitepackages() - else: - # virtualenv < 20 overwrites site.py without getsitepackages - # fallback on get_purelib/get_platlib. - # this is known to miss things, but shouldn't in the cases - # where getsitepackages() has been removed (inside a virtualenv) - system_sites = [get_purelib(), get_platlib()] - return {os.path.normcase(path) for path in system_sites} - - -class BuildEnvironment: - """Creates and manages an isolated environment to install build deps""" - - def __init__(self) -> None: - temp_dir = TempDirectory(kind=tempdir_kinds.BUILD_ENV, globally_managed=True) - - self._prefixes = OrderedDict( - (name, _Prefix(os.path.join(temp_dir.path, name))) - for name in ("normal", "overlay") - ) - - self._bin_dirs: List[str] = [] - self._lib_dirs: List[str] = [] - for prefix in reversed(list(self._prefixes.values())): - self._bin_dirs.append(prefix.bin_dir) - self._lib_dirs.extend(prefix.lib_dirs) - - # Customize site to: - # - ensure .pth files are honored - # - prevent access to system site packages - system_sites = _get_system_sitepackages() - - self._site_dir = os.path.join(temp_dir.path, "site") - if not os.path.exists(self._site_dir): - os.mkdir(self._site_dir) - with open( - os.path.join(self._site_dir, "sitecustomize.py"), "w", encoding="utf-8" - ) as fp: - fp.write( - textwrap.dedent( - """ - import os, site, sys - - # First, drop system-sites related paths. - original_sys_path = sys.path[:] - known_paths = set() - for path in {system_sites!r}: - site.addsitedir(path, known_paths=known_paths) - system_paths = set( - os.path.normcase(path) - for path in sys.path[len(original_sys_path):] - ) - original_sys_path = [ - path for path in original_sys_path - if os.path.normcase(path) not in system_paths - ] - sys.path = original_sys_path - - # Second, add lib directories. - # ensuring .pth file are processed. - for path in {lib_dirs!r}: - assert not path in sys.path - site.addsitedir(path) - """ - ).format(system_sites=system_sites, lib_dirs=self._lib_dirs) - ) - - def __enter__(self) -> None: - self._save_env = { - name: os.environ.get(name, None) - for name in ("PATH", "PYTHONNOUSERSITE", "PYTHONPATH") - } - - path = self._bin_dirs[:] - old_path = self._save_env["PATH"] - if old_path: - path.extend(old_path.split(os.pathsep)) - - pythonpath = [self._site_dir] - - os.environ.update( - { - "PATH": os.pathsep.join(path), - "PYTHONNOUSERSITE": "1", - "PYTHONPATH": os.pathsep.join(pythonpath), - } - ) - - def __exit__( - self, - exc_type: Optional[Type[BaseException]], - exc_val: Optional[BaseException], - exc_tb: Optional[TracebackType], - ) -> None: - for varname, old_value in self._save_env.items(): - if old_value is None: - os.environ.pop(varname, None) - else: - os.environ[varname] = old_value - - def check_requirements( - self, reqs: Iterable[str] - ) -> Tuple[Set[Tuple[str, str]], Set[str]]: - """Return 2 sets: - - conflicting requirements: set of (installed, wanted) reqs tuples - - missing requirements: set of reqs - """ - missing = set() - conflicting = set() - if reqs: - env = ( - get_environment(self._lib_dirs) - if hasattr(self, "_lib_dirs") - else get_default_environment() - ) - for req_str in reqs: - req = Requirement(req_str) - # We're explicitly evaluating with an empty extra value, since build - # environments are not provided any mechanism to select specific extras. - if req.marker is not None and not req.marker.evaluate({"extra": ""}): - continue - dist = env.get_distribution(req.name) - if not dist: - missing.add(req_str) - continue - if isinstance(dist.version, Version): - installed_req_str = f"{req.name}=={dist.version}" - else: - installed_req_str = f"{req.name}==={dist.version}" - if not req.specifier.contains(dist.version, prereleases=True): - conflicting.add((installed_req_str, req_str)) - # FIXME: Consider direct URL? - return conflicting, missing - - def install_requirements( - self, - finder: "PackageFinder", - requirements: Iterable[str], - prefix_as_string: str, - *, - kind: str, - ) -> None: - prefix = self._prefixes[prefix_as_string] - assert not prefix.setup - prefix.setup = True - if not requirements: - return - self._install_requirements( - get_runnable_pip(), - finder, - requirements, - prefix, - kind=kind, - ) - - @staticmethod - def _install_requirements( - pip_runnable: str, - finder: "PackageFinder", - requirements: Iterable[str], - prefix: _Prefix, - *, - kind: str, - ) -> None: - args: List[str] = [ - sys.executable, - pip_runnable, - "install", - "--ignore-installed", - "--no-user", - "--prefix", - prefix.path, - "--no-warn-script-location", - ] - if logger.getEffectiveLevel() <= logging.DEBUG: - args.append("-v") - for format_control in ("no_binary", "only_binary"): - formats = getattr(finder.format_control, format_control) - args.extend( - ( - "--" + format_control.replace("_", "-"), - ",".join(sorted(formats or {":none:"})), - ) - ) - - index_urls = finder.index_urls - if index_urls: - args.extend(["-i", index_urls[0]]) - for extra_index in index_urls[1:]: - args.extend(["--extra-index-url", extra_index]) - else: - args.append("--no-index") - for link in finder.find_links: - args.extend(["--find-links", link]) - - for host in finder.trusted_hosts: - args.extend(["--trusted-host", host]) - if finder.allow_all_prereleases: - args.append("--pre") - if finder.prefer_binary: - args.append("--prefer-binary") - args.append("--") - args.extend(requirements) - extra_environ = {"_PIP_STANDALONE_CERT": where()} - with open_spinner(f"Installing {kind}") as spinner: - call_subprocess( - args, - command_desc=f"pip subprocess to install {kind}", - spinner=spinner, - extra_environ=extra_environ, - ) - - -class NoOpBuildEnvironment(BuildEnvironment): - """A no-op drop-in replacement for BuildEnvironment""" - - def __init__(self) -> None: - pass - - def __enter__(self) -> None: - pass - - def __exit__( - self, - exc_type: Optional[Type[BaseException]], - exc_val: Optional[BaseException], - exc_tb: Optional[TracebackType], - ) -> None: - pass - - def cleanup(self) -> None: - pass - - def install_requirements( - self, - finder: "PackageFinder", - requirements: Iterable[str], - prefix_as_string: str, - *, - kind: str, - ) -> None: - raise NotImplementedError() diff --git a/venv/lib/python3.11/site-packages/pip/_internal/cache.py b/venv/lib/python3.11/site-packages/pip/_internal/cache.py deleted file mode 100644 index f45ac23..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/cache.py +++ /dev/null @@ -1,290 +0,0 @@ -"""Cache Management -""" - -import hashlib -import json -import logging -import os -from pathlib import Path -from typing import Any, Dict, List, Optional - -from pip._vendor.packaging.tags import Tag, interpreter_name, interpreter_version -from pip._vendor.packaging.utils import canonicalize_name - -from pip._internal.exceptions import InvalidWheelFilename -from pip._internal.models.direct_url import DirectUrl -from pip._internal.models.link import Link -from pip._internal.models.wheel import Wheel -from pip._internal.utils.temp_dir import TempDirectory, tempdir_kinds -from pip._internal.utils.urls import path_to_url - -logger = logging.getLogger(__name__) - -ORIGIN_JSON_NAME = "origin.json" - - -def _hash_dict(d: Dict[str, str]) -> str: - """Return a stable sha224 of a dictionary.""" - s = json.dumps(d, sort_keys=True, separators=(",", ":"), ensure_ascii=True) - return hashlib.sha224(s.encode("ascii")).hexdigest() - - -class Cache: - """An abstract class - provides cache directories for data from links - - :param cache_dir: The root of the cache. - """ - - def __init__(self, cache_dir: str) -> None: - super().__init__() - assert not cache_dir or os.path.isabs(cache_dir) - self.cache_dir = cache_dir or None - - def _get_cache_path_parts(self, link: Link) -> List[str]: - """Get parts of part that must be os.path.joined with cache_dir""" - - # We want to generate an url to use as our cache key, we don't want to - # just re-use the URL because it might have other items in the fragment - # and we don't care about those. - key_parts = {"url": link.url_without_fragment} - if link.hash_name is not None and link.hash is not None: - key_parts[link.hash_name] = link.hash - if link.subdirectory_fragment: - key_parts["subdirectory"] = link.subdirectory_fragment - - # Include interpreter name, major and minor version in cache key - # to cope with ill-behaved sdists that build a different wheel - # depending on the python version their setup.py is being run on, - # and don't encode the difference in compatibility tags. - # https://github.com/pypa/pip/issues/7296 - key_parts["interpreter_name"] = interpreter_name() - key_parts["interpreter_version"] = interpreter_version() - - # Encode our key url with sha224, we'll use this because it has similar - # security properties to sha256, but with a shorter total output (and - # thus less secure). However the differences don't make a lot of - # difference for our use case here. - hashed = _hash_dict(key_parts) - - # We want to nest the directories some to prevent having a ton of top - # level directories where we might run out of sub directories on some - # FS. - parts = [hashed[:2], hashed[2:4], hashed[4:6], hashed[6:]] - - return parts - - def _get_candidates(self, link: Link, canonical_package_name: str) -> List[Any]: - can_not_cache = not self.cache_dir or not canonical_package_name or not link - if can_not_cache: - return [] - - path = self.get_path_for_link(link) - if os.path.isdir(path): - return [(candidate, path) for candidate in os.listdir(path)] - return [] - - def get_path_for_link(self, link: Link) -> str: - """Return a directory to store cached items in for link.""" - raise NotImplementedError() - - def get( - self, - link: Link, - package_name: Optional[str], - supported_tags: List[Tag], - ) -> Link: - """Returns a link to a cached item if it exists, otherwise returns the - passed link. - """ - raise NotImplementedError() - - -class SimpleWheelCache(Cache): - """A cache of wheels for future installs.""" - - def __init__(self, cache_dir: str) -> None: - super().__init__(cache_dir) - - def get_path_for_link(self, link: Link) -> str: - """Return a directory to store cached wheels for link - - Because there are M wheels for any one sdist, we provide a directory - to cache them in, and then consult that directory when looking up - cache hits. - - We only insert things into the cache if they have plausible version - numbers, so that we don't contaminate the cache with things that were - not unique. E.g. ./package might have dozens of installs done for it - and build a version of 0.0...and if we built and cached a wheel, we'd - end up using the same wheel even if the source has been edited. - - :param link: The link of the sdist for which this will cache wheels. - """ - parts = self._get_cache_path_parts(link) - assert self.cache_dir - # Store wheels within the root cache_dir - return os.path.join(self.cache_dir, "wheels", *parts) - - def get( - self, - link: Link, - package_name: Optional[str], - supported_tags: List[Tag], - ) -> Link: - candidates = [] - - if not package_name: - return link - - canonical_package_name = canonicalize_name(package_name) - for wheel_name, wheel_dir in self._get_candidates(link, canonical_package_name): - try: - wheel = Wheel(wheel_name) - except InvalidWheelFilename: - continue - if canonicalize_name(wheel.name) != canonical_package_name: - logger.debug( - "Ignoring cached wheel %s for %s as it " - "does not match the expected distribution name %s.", - wheel_name, - link, - package_name, - ) - continue - if not wheel.supported(supported_tags): - # Built for a different python/arch/etc - continue - candidates.append( - ( - wheel.support_index_min(supported_tags), - wheel_name, - wheel_dir, - ) - ) - - if not candidates: - return link - - _, wheel_name, wheel_dir = min(candidates) - return Link(path_to_url(os.path.join(wheel_dir, wheel_name))) - - -class EphemWheelCache(SimpleWheelCache): - """A SimpleWheelCache that creates it's own temporary cache directory""" - - def __init__(self) -> None: - self._temp_dir = TempDirectory( - kind=tempdir_kinds.EPHEM_WHEEL_CACHE, - globally_managed=True, - ) - - super().__init__(self._temp_dir.path) - - -class CacheEntry: - def __init__( - self, - link: Link, - persistent: bool, - ): - self.link = link - self.persistent = persistent - self.origin: Optional[DirectUrl] = None - origin_direct_url_path = Path(self.link.file_path).parent / ORIGIN_JSON_NAME - if origin_direct_url_path.exists(): - try: - self.origin = DirectUrl.from_json( - origin_direct_url_path.read_text(encoding="utf-8") - ) - except Exception as e: - logger.warning( - "Ignoring invalid cache entry origin file %s for %s (%s)", - origin_direct_url_path, - link.filename, - e, - ) - - -class WheelCache(Cache): - """Wraps EphemWheelCache and SimpleWheelCache into a single Cache - - This Cache allows for gracefully degradation, using the ephem wheel cache - when a certain link is not found in the simple wheel cache first. - """ - - def __init__(self, cache_dir: str) -> None: - super().__init__(cache_dir) - self._wheel_cache = SimpleWheelCache(cache_dir) - self._ephem_cache = EphemWheelCache() - - def get_path_for_link(self, link: Link) -> str: - return self._wheel_cache.get_path_for_link(link) - - def get_ephem_path_for_link(self, link: Link) -> str: - return self._ephem_cache.get_path_for_link(link) - - def get( - self, - link: Link, - package_name: Optional[str], - supported_tags: List[Tag], - ) -> Link: - cache_entry = self.get_cache_entry(link, package_name, supported_tags) - if cache_entry is None: - return link - return cache_entry.link - - def get_cache_entry( - self, - link: Link, - package_name: Optional[str], - supported_tags: List[Tag], - ) -> Optional[CacheEntry]: - """Returns a CacheEntry with a link to a cached item if it exists or - None. The cache entry indicates if the item was found in the persistent - or ephemeral cache. - """ - retval = self._wheel_cache.get( - link=link, - package_name=package_name, - supported_tags=supported_tags, - ) - if retval is not link: - return CacheEntry(retval, persistent=True) - - retval = self._ephem_cache.get( - link=link, - package_name=package_name, - supported_tags=supported_tags, - ) - if retval is not link: - return CacheEntry(retval, persistent=False) - - return None - - @staticmethod - def record_download_origin(cache_dir: str, download_info: DirectUrl) -> None: - origin_path = Path(cache_dir) / ORIGIN_JSON_NAME - if origin_path.exists(): - try: - origin = DirectUrl.from_json(origin_path.read_text(encoding="utf-8")) - except Exception as e: - logger.warning( - "Could not read origin file %s in cache entry (%s). " - "Will attempt to overwrite it.", - origin_path, - e, - ) - else: - # TODO: use DirectUrl.equivalent when - # https://github.com/pypa/pip/pull/10564 is merged. - if origin.url != download_info.url: - logger.warning( - "Origin URL %s in cache entry %s does not match download URL " - "%s. This is likely a pip bug or a cache corruption issue. " - "Will overwrite it with the new value.", - origin.url, - cache_dir, - download_info.url, - ) - origin_path.write_text(download_info.to_json(), encoding="utf-8") diff --git a/venv/lib/python3.11/site-packages/pip/_internal/cli/__init__.py b/venv/lib/python3.11/site-packages/pip/_internal/cli/__init__.py deleted file mode 100644 index e589bb9..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/cli/__init__.py +++ /dev/null @@ -1,4 +0,0 @@ -"""Subpackage containing all of pip's command line interface related code -""" - -# This file intentionally does not import submodules diff --git a/venv/lib/python3.11/site-packages/pip/_internal/cli/__pycache__/__init__.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/cli/__pycache__/__init__.cpython-311.pyc deleted file mode 100644 index 1c6ed33..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/cli/__pycache__/__init__.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/cli/__pycache__/autocompletion.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/cli/__pycache__/autocompletion.cpython-311.pyc deleted file mode 100644 index 1394636..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/cli/__pycache__/autocompletion.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/cli/__pycache__/base_command.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/cli/__pycache__/base_command.cpython-311.pyc deleted file mode 100644 index 0dfd988..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/cli/__pycache__/base_command.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/cli/__pycache__/cmdoptions.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/cli/__pycache__/cmdoptions.cpython-311.pyc deleted file mode 100644 index c54649c..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/cli/__pycache__/cmdoptions.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/cli/__pycache__/command_context.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/cli/__pycache__/command_context.cpython-311.pyc deleted file mode 100644 index 6f4fdcf..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/cli/__pycache__/command_context.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/cli/__pycache__/main.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/cli/__pycache__/main.cpython-311.pyc deleted file mode 100644 index eaf8b53..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/cli/__pycache__/main.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/cli/__pycache__/main_parser.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/cli/__pycache__/main_parser.cpython-311.pyc deleted file mode 100644 index 35483d3..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/cli/__pycache__/main_parser.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/cli/__pycache__/parser.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/cli/__pycache__/parser.cpython-311.pyc deleted file mode 100644 index cac470a..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/cli/__pycache__/parser.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/cli/__pycache__/progress_bars.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/cli/__pycache__/progress_bars.cpython-311.pyc deleted file mode 100644 index f9361dd..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/cli/__pycache__/progress_bars.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/cli/__pycache__/req_command.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/cli/__pycache__/req_command.cpython-311.pyc deleted file mode 100644 index b632cf8..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/cli/__pycache__/req_command.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/cli/__pycache__/spinners.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/cli/__pycache__/spinners.cpython-311.pyc deleted file mode 100644 index e43214b..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/cli/__pycache__/spinners.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/cli/__pycache__/status_codes.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/cli/__pycache__/status_codes.cpython-311.pyc deleted file mode 100644 index d5059f7..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/cli/__pycache__/status_codes.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/cli/autocompletion.py b/venv/lib/python3.11/site-packages/pip/_internal/cli/autocompletion.py deleted file mode 100644 index e5950b9..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/cli/autocompletion.py +++ /dev/null @@ -1,172 +0,0 @@ -"""Logic that powers autocompletion installed by ``pip completion``. -""" - -import optparse -import os -import sys -from itertools import chain -from typing import Any, Iterable, List, Optional - -from pip._internal.cli.main_parser import create_main_parser -from pip._internal.commands import commands_dict, create_command -from pip._internal.metadata import get_default_environment - - -def autocomplete() -> None: - """Entry Point for completion of main and subcommand options.""" - # Don't complete if user hasn't sourced bash_completion file. - if "PIP_AUTO_COMPLETE" not in os.environ: - return - cwords = os.environ["COMP_WORDS"].split()[1:] - cword = int(os.environ["COMP_CWORD"]) - try: - current = cwords[cword - 1] - except IndexError: - current = "" - - parser = create_main_parser() - subcommands = list(commands_dict) - options = [] - - # subcommand - subcommand_name: Optional[str] = None - for word in cwords: - if word in subcommands: - subcommand_name = word - break - # subcommand options - if subcommand_name is not None: - # special case: 'help' subcommand has no options - if subcommand_name == "help": - sys.exit(1) - # special case: list locally installed dists for show and uninstall - should_list_installed = not current.startswith("-") and subcommand_name in [ - "show", - "uninstall", - ] - if should_list_installed: - env = get_default_environment() - lc = current.lower() - installed = [ - dist.canonical_name - for dist in env.iter_installed_distributions(local_only=True) - if dist.canonical_name.startswith(lc) - and dist.canonical_name not in cwords[1:] - ] - # if there are no dists installed, fall back to option completion - if installed: - for dist in installed: - print(dist) - sys.exit(1) - - should_list_installables = ( - not current.startswith("-") and subcommand_name == "install" - ) - if should_list_installables: - for path in auto_complete_paths(current, "path"): - print(path) - sys.exit(1) - - subcommand = create_command(subcommand_name) - - for opt in subcommand.parser.option_list_all: - if opt.help != optparse.SUPPRESS_HELP: - options += [ - (opt_str, opt.nargs) for opt_str in opt._long_opts + opt._short_opts - ] - - # filter out previously specified options from available options - prev_opts = [x.split("=")[0] for x in cwords[1 : cword - 1]] - options = [(x, v) for (x, v) in options if x not in prev_opts] - # filter options by current input - options = [(k, v) for k, v in options if k.startswith(current)] - # get completion type given cwords and available subcommand options - completion_type = get_path_completion_type( - cwords, - cword, - subcommand.parser.option_list_all, - ) - # get completion files and directories if ``completion_type`` is - # ````, ```` or ```` - if completion_type: - paths = auto_complete_paths(current, completion_type) - options = [(path, 0) for path in paths] - for option in options: - opt_label = option[0] - # append '=' to options which require args - if option[1] and option[0][:2] == "--": - opt_label += "=" - print(opt_label) - else: - # show main parser options only when necessary - - opts = [i.option_list for i in parser.option_groups] - opts.append(parser.option_list) - flattened_opts = chain.from_iterable(opts) - if current.startswith("-"): - for opt in flattened_opts: - if opt.help != optparse.SUPPRESS_HELP: - subcommands += opt._long_opts + opt._short_opts - else: - # get completion type given cwords and all available options - completion_type = get_path_completion_type(cwords, cword, flattened_opts) - if completion_type: - subcommands = list(auto_complete_paths(current, completion_type)) - - print(" ".join([x for x in subcommands if x.startswith(current)])) - sys.exit(1) - - -def get_path_completion_type( - cwords: List[str], cword: int, opts: Iterable[Any] -) -> Optional[str]: - """Get the type of path completion (``file``, ``dir``, ``path`` or None) - - :param cwords: same as the environmental variable ``COMP_WORDS`` - :param cword: same as the environmental variable ``COMP_CWORD`` - :param opts: The available options to check - :return: path completion type (``file``, ``dir``, ``path`` or None) - """ - if cword < 2 or not cwords[cword - 2].startswith("-"): - return None - for opt in opts: - if opt.help == optparse.SUPPRESS_HELP: - continue - for o in str(opt).split("/"): - if cwords[cword - 2].split("=")[0] == o: - if not opt.metavar or any( - x in ("path", "file", "dir") for x in opt.metavar.split("/") - ): - return opt.metavar - return None - - -def auto_complete_paths(current: str, completion_type: str) -> Iterable[str]: - """If ``completion_type`` is ``file`` or ``path``, list all regular files - and directories starting with ``current``; otherwise only list directories - starting with ``current``. - - :param current: The word to be completed - :param completion_type: path completion type(``file``, ``path`` or ``dir``) - :return: A generator of regular files and/or directories - """ - directory, filename = os.path.split(current) - current_path = os.path.abspath(directory) - # Don't complete paths if they can't be accessed - if not os.access(current_path, os.R_OK): - return - filename = os.path.normcase(filename) - # list all files that start with ``filename`` - file_list = ( - x for x in os.listdir(current_path) if os.path.normcase(x).startswith(filename) - ) - for f in file_list: - opt = os.path.join(current_path, f) - comp_file = os.path.normcase(os.path.join(directory, f)) - # complete regular files when there is not ```` after option - # complete directories when there is ````, ```` or - # ````after option - if completion_type != "dir" and os.path.isfile(opt): - yield comp_file - elif os.path.isdir(opt): - yield os.path.join(comp_file, "") diff --git a/venv/lib/python3.11/site-packages/pip/_internal/cli/base_command.py b/venv/lib/python3.11/site-packages/pip/_internal/cli/base_command.py deleted file mode 100644 index db9d5cc..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/cli/base_command.py +++ /dev/null @@ -1,236 +0,0 @@ -"""Base Command class, and related routines""" - -import functools -import logging -import logging.config -import optparse -import os -import sys -import traceback -from optparse import Values -from typing import Any, Callable, List, Optional, Tuple - -from pip._vendor.rich import traceback as rich_traceback - -from pip._internal.cli import cmdoptions -from pip._internal.cli.command_context import CommandContextMixIn -from pip._internal.cli.parser import ConfigOptionParser, UpdatingDefaultsHelpFormatter -from pip._internal.cli.status_codes import ( - ERROR, - PREVIOUS_BUILD_DIR_ERROR, - UNKNOWN_ERROR, - VIRTUALENV_NOT_FOUND, -) -from pip._internal.exceptions import ( - BadCommand, - CommandError, - DiagnosticPipError, - InstallationError, - NetworkConnectionError, - PreviousBuildDirError, - UninstallationError, -) -from pip._internal.utils.filesystem import check_path_owner -from pip._internal.utils.logging import BrokenStdoutLoggingError, setup_logging -from pip._internal.utils.misc import get_prog, normalize_path -from pip._internal.utils.temp_dir import TempDirectoryTypeRegistry as TempDirRegistry -from pip._internal.utils.temp_dir import global_tempdir_manager, tempdir_registry -from pip._internal.utils.virtualenv import running_under_virtualenv - -__all__ = ["Command"] - -logger = logging.getLogger(__name__) - - -class Command(CommandContextMixIn): - usage: str = "" - ignore_require_venv: bool = False - - def __init__(self, name: str, summary: str, isolated: bool = False) -> None: - super().__init__() - - self.name = name - self.summary = summary - self.parser = ConfigOptionParser( - usage=self.usage, - prog=f"{get_prog()} {name}", - formatter=UpdatingDefaultsHelpFormatter(), - add_help_option=False, - name=name, - description=self.__doc__, - isolated=isolated, - ) - - self.tempdir_registry: Optional[TempDirRegistry] = None - - # Commands should add options to this option group - optgroup_name = f"{self.name.capitalize()} Options" - self.cmd_opts = optparse.OptionGroup(self.parser, optgroup_name) - - # Add the general options - gen_opts = cmdoptions.make_option_group( - cmdoptions.general_group, - self.parser, - ) - self.parser.add_option_group(gen_opts) - - self.add_options() - - def add_options(self) -> None: - pass - - def handle_pip_version_check(self, options: Values) -> None: - """ - This is a no-op so that commands by default do not do the pip version - check. - """ - # Make sure we do the pip version check if the index_group options - # are present. - assert not hasattr(options, "no_index") - - def run(self, options: Values, args: List[str]) -> int: - raise NotImplementedError - - def parse_args(self, args: List[str]) -> Tuple[Values, List[str]]: - # factored out for testability - return self.parser.parse_args(args) - - def main(self, args: List[str]) -> int: - try: - with self.main_context(): - return self._main(args) - finally: - logging.shutdown() - - def _main(self, args: List[str]) -> int: - # We must initialize this before the tempdir manager, otherwise the - # configuration would not be accessible by the time we clean up the - # tempdir manager. - self.tempdir_registry = self.enter_context(tempdir_registry()) - # Intentionally set as early as possible so globally-managed temporary - # directories are available to the rest of the code. - self.enter_context(global_tempdir_manager()) - - options, args = self.parse_args(args) - - # Set verbosity so that it can be used elsewhere. - self.verbosity = options.verbose - options.quiet - - level_number = setup_logging( - verbosity=self.verbosity, - no_color=options.no_color, - user_log_file=options.log, - ) - - always_enabled_features = set(options.features_enabled) & set( - cmdoptions.ALWAYS_ENABLED_FEATURES - ) - if always_enabled_features: - logger.warning( - "The following features are always enabled: %s. ", - ", ".join(sorted(always_enabled_features)), - ) - - # Make sure that the --python argument isn't specified after the - # subcommand. We can tell, because if --python was specified, - # we should only reach this point if we're running in the created - # subprocess, which has the _PIP_RUNNING_IN_SUBPROCESS environment - # variable set. - if options.python and "_PIP_RUNNING_IN_SUBPROCESS" not in os.environ: - logger.critical( - "The --python option must be placed before the pip subcommand name" - ) - sys.exit(ERROR) - - # TODO: Try to get these passing down from the command? - # without resorting to os.environ to hold these. - # This also affects isolated builds and it should. - - if options.no_input: - os.environ["PIP_NO_INPUT"] = "1" - - if options.exists_action: - os.environ["PIP_EXISTS_ACTION"] = " ".join(options.exists_action) - - if options.require_venv and not self.ignore_require_venv: - # If a venv is required check if it can really be found - if not running_under_virtualenv(): - logger.critical("Could not find an activated virtualenv (required).") - sys.exit(VIRTUALENV_NOT_FOUND) - - if options.cache_dir: - options.cache_dir = normalize_path(options.cache_dir) - if not check_path_owner(options.cache_dir): - logger.warning( - "The directory '%s' or its parent directory is not owned " - "or is not writable by the current user. The cache " - "has been disabled. Check the permissions and owner of " - "that directory. If executing pip with sudo, you should " - "use sudo's -H flag.", - options.cache_dir, - ) - options.cache_dir = None - - def intercepts_unhandled_exc( - run_func: Callable[..., int] - ) -> Callable[..., int]: - @functools.wraps(run_func) - def exc_logging_wrapper(*args: Any) -> int: - try: - status = run_func(*args) - assert isinstance(status, int) - return status - except DiagnosticPipError as exc: - logger.error("%s", exc, extra={"rich": True}) - logger.debug("Exception information:", exc_info=True) - - return ERROR - except PreviousBuildDirError as exc: - logger.critical(str(exc)) - logger.debug("Exception information:", exc_info=True) - - return PREVIOUS_BUILD_DIR_ERROR - except ( - InstallationError, - UninstallationError, - BadCommand, - NetworkConnectionError, - ) as exc: - logger.critical(str(exc)) - logger.debug("Exception information:", exc_info=True) - - return ERROR - except CommandError as exc: - logger.critical("%s", exc) - logger.debug("Exception information:", exc_info=True) - - return ERROR - except BrokenStdoutLoggingError: - # Bypass our logger and write any remaining messages to - # stderr because stdout no longer works. - print("ERROR: Pipe to stdout was broken", file=sys.stderr) - if level_number <= logging.DEBUG: - traceback.print_exc(file=sys.stderr) - - return ERROR - except KeyboardInterrupt: - logger.critical("Operation cancelled by user") - logger.debug("Exception information:", exc_info=True) - - return ERROR - except BaseException: - logger.critical("Exception:", exc_info=True) - - return UNKNOWN_ERROR - - return exc_logging_wrapper - - try: - if not options.debug_mode: - run = intercepts_unhandled_exc(self.run) - else: - run = self.run - rich_traceback.install(show_locals=True) - return run(options, args) - finally: - self.handle_pip_version_check(options) diff --git a/venv/lib/python3.11/site-packages/pip/_internal/cli/cmdoptions.py b/venv/lib/python3.11/site-packages/pip/_internal/cli/cmdoptions.py deleted file mode 100644 index 8fb16dc..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/cli/cmdoptions.py +++ /dev/null @@ -1,1077 +0,0 @@ -""" -shared options and groups - -The principle here is to define options once, but *not* instantiate them -globally. One reason being that options with action='append' can carry state -between parses. pip parses general options twice internally, and shouldn't -pass on state. To be consistent, all options will follow this design. -""" - -# The following comment should be removed at some point in the future. -# mypy: strict-optional=False - -import importlib.util -import logging -import os -import textwrap -from functools import partial -from optparse import SUPPRESS_HELP, Option, OptionGroup, OptionParser, Values -from textwrap import dedent -from typing import Any, Callable, Dict, Optional, Tuple - -from pip._vendor.packaging.utils import canonicalize_name - -from pip._internal.cli.parser import ConfigOptionParser -from pip._internal.exceptions import CommandError -from pip._internal.locations import USER_CACHE_DIR, get_src_prefix -from pip._internal.models.format_control import FormatControl -from pip._internal.models.index import PyPI -from pip._internal.models.target_python import TargetPython -from pip._internal.utils.hashes import STRONG_HASHES -from pip._internal.utils.misc import strtobool - -logger = logging.getLogger(__name__) - - -def raise_option_error(parser: OptionParser, option: Option, msg: str) -> None: - """ - Raise an option parsing error using parser.error(). - - Args: - parser: an OptionParser instance. - option: an Option instance. - msg: the error text. - """ - msg = f"{option} error: {msg}" - msg = textwrap.fill(" ".join(msg.split())) - parser.error(msg) - - -def make_option_group(group: Dict[str, Any], parser: ConfigOptionParser) -> OptionGroup: - """ - Return an OptionGroup object - group -- assumed to be dict with 'name' and 'options' keys - parser -- an optparse Parser - """ - option_group = OptionGroup(parser, group["name"]) - for option in group["options"]: - option_group.add_option(option()) - return option_group - - -def check_dist_restriction(options: Values, check_target: bool = False) -> None: - """Function for determining if custom platform options are allowed. - - :param options: The OptionParser options. - :param check_target: Whether or not to check if --target is being used. - """ - dist_restriction_set = any( - [ - options.python_version, - options.platforms, - options.abis, - options.implementation, - ] - ) - - binary_only = FormatControl(set(), {":all:"}) - sdist_dependencies_allowed = ( - options.format_control != binary_only and not options.ignore_dependencies - ) - - # Installations or downloads using dist restrictions must not combine - # source distributions and dist-specific wheels, as they are not - # guaranteed to be locally compatible. - if dist_restriction_set and sdist_dependencies_allowed: - raise CommandError( - "When restricting platform and interpreter constraints using " - "--python-version, --platform, --abi, or --implementation, " - "either --no-deps must be set, or --only-binary=:all: must be " - "set and --no-binary must not be set (or must be set to " - ":none:)." - ) - - if check_target: - if not options.dry_run and dist_restriction_set and not options.target_dir: - raise CommandError( - "Can not use any platform or abi specific options unless " - "installing via '--target' or using '--dry-run'" - ) - - -def _path_option_check(option: Option, opt: str, value: str) -> str: - return os.path.expanduser(value) - - -def _package_name_option_check(option: Option, opt: str, value: str) -> str: - return canonicalize_name(value) - - -class PipOption(Option): - TYPES = Option.TYPES + ("path", "package_name") - TYPE_CHECKER = Option.TYPE_CHECKER.copy() - TYPE_CHECKER["package_name"] = _package_name_option_check - TYPE_CHECKER["path"] = _path_option_check - - -########### -# options # -########### - -help_: Callable[..., Option] = partial( - Option, - "-h", - "--help", - dest="help", - action="help", - help="Show help.", -) - -debug_mode: Callable[..., Option] = partial( - Option, - "--debug", - dest="debug_mode", - action="store_true", - default=False, - help=( - "Let unhandled exceptions propagate outside the main subroutine, " - "instead of logging them to stderr." - ), -) - -isolated_mode: Callable[..., Option] = partial( - Option, - "--isolated", - dest="isolated_mode", - action="store_true", - default=False, - help=( - "Run pip in an isolated mode, ignoring environment variables and user " - "configuration." - ), -) - -require_virtualenv: Callable[..., Option] = partial( - Option, - "--require-virtualenv", - "--require-venv", - dest="require_venv", - action="store_true", - default=False, - help=( - "Allow pip to only run in a virtual environment; " - "exit with an error otherwise." - ), -) - -override_externally_managed: Callable[..., Option] = partial( - Option, - "--break-system-packages", - dest="override_externally_managed", - action="store_true", - help="Allow pip to modify an EXTERNALLY-MANAGED Python installation", -) - -python: Callable[..., Option] = partial( - Option, - "--python", - dest="python", - help="Run pip with the specified Python interpreter.", -) - -verbose: Callable[..., Option] = partial( - Option, - "-v", - "--verbose", - dest="verbose", - action="count", - default=0, - help="Give more output. Option is additive, and can be used up to 3 times.", -) - -no_color: Callable[..., Option] = partial( - Option, - "--no-color", - dest="no_color", - action="store_true", - default=False, - help="Suppress colored output.", -) - -version: Callable[..., Option] = partial( - Option, - "-V", - "--version", - dest="version", - action="store_true", - help="Show version and exit.", -) - -quiet: Callable[..., Option] = partial( - Option, - "-q", - "--quiet", - dest="quiet", - action="count", - default=0, - help=( - "Give less output. Option is additive, and can be used up to 3" - " times (corresponding to WARNING, ERROR, and CRITICAL logging" - " levels)." - ), -) - -progress_bar: Callable[..., Option] = partial( - Option, - "--progress-bar", - dest="progress_bar", - type="choice", - choices=["on", "off"], - default="on", - help="Specify whether the progress bar should be used [on, off] (default: on)", -) - -log: Callable[..., Option] = partial( - PipOption, - "--log", - "--log-file", - "--local-log", - dest="log", - metavar="path", - type="path", - help="Path to a verbose appending log.", -) - -no_input: Callable[..., Option] = partial( - Option, - # Don't ask for input - "--no-input", - dest="no_input", - action="store_true", - default=False, - help="Disable prompting for input.", -) - -keyring_provider: Callable[..., Option] = partial( - Option, - "--keyring-provider", - dest="keyring_provider", - choices=["auto", "disabled", "import", "subprocess"], - default="auto", - help=( - "Enable the credential lookup via the keyring library if user input is allowed." - " Specify which mechanism to use [disabled, import, subprocess]." - " (default: disabled)" - ), -) - -proxy: Callable[..., Option] = partial( - Option, - "--proxy", - dest="proxy", - type="str", - default="", - help="Specify a proxy in the form scheme://[user:passwd@]proxy.server:port.", -) - -retries: Callable[..., Option] = partial( - Option, - "--retries", - dest="retries", - type="int", - default=5, - help="Maximum number of retries each connection should attempt " - "(default %default times).", -) - -timeout: Callable[..., Option] = partial( - Option, - "--timeout", - "--default-timeout", - metavar="sec", - dest="timeout", - type="float", - default=15, - help="Set the socket timeout (default %default seconds).", -) - - -def exists_action() -> Option: - return Option( - # Option when path already exist - "--exists-action", - dest="exists_action", - type="choice", - choices=["s", "i", "w", "b", "a"], - default=[], - action="append", - metavar="action", - help="Default action when a path already exists: " - "(s)witch, (i)gnore, (w)ipe, (b)ackup, (a)bort.", - ) - - -cert: Callable[..., Option] = partial( - PipOption, - "--cert", - dest="cert", - type="path", - metavar="path", - help=( - "Path to PEM-encoded CA certificate bundle. " - "If provided, overrides the default. " - "See 'SSL Certificate Verification' in pip documentation " - "for more information." - ), -) - -client_cert: Callable[..., Option] = partial( - PipOption, - "--client-cert", - dest="client_cert", - type="path", - default=None, - metavar="path", - help="Path to SSL client certificate, a single file containing the " - "private key and the certificate in PEM format.", -) - -index_url: Callable[..., Option] = partial( - Option, - "-i", - "--index-url", - "--pypi-url", - dest="index_url", - metavar="URL", - default=PyPI.simple_url, - help="Base URL of the Python Package Index (default %default). " - "This should point to a repository compliant with PEP 503 " - "(the simple repository API) or a local directory laid out " - "in the same format.", -) - - -def extra_index_url() -> Option: - return Option( - "--extra-index-url", - dest="extra_index_urls", - metavar="URL", - action="append", - default=[], - help="Extra URLs of package indexes to use in addition to " - "--index-url. Should follow the same rules as " - "--index-url.", - ) - - -no_index: Callable[..., Option] = partial( - Option, - "--no-index", - dest="no_index", - action="store_true", - default=False, - help="Ignore package index (only looking at --find-links URLs instead).", -) - - -def find_links() -> Option: - return Option( - "-f", - "--find-links", - dest="find_links", - action="append", - default=[], - metavar="url", - help="If a URL or path to an html file, then parse for links to " - "archives such as sdist (.tar.gz) or wheel (.whl) files. " - "If a local path or file:// URL that's a directory, " - "then look for archives in the directory listing. " - "Links to VCS project URLs are not supported.", - ) - - -def trusted_host() -> Option: - return Option( - "--trusted-host", - dest="trusted_hosts", - action="append", - metavar="HOSTNAME", - default=[], - help="Mark this host or host:port pair as trusted, even though it " - "does not have valid or any HTTPS.", - ) - - -def constraints() -> Option: - return Option( - "-c", - "--constraint", - dest="constraints", - action="append", - default=[], - metavar="file", - help="Constrain versions using the given constraints file. " - "This option can be used multiple times.", - ) - - -def requirements() -> Option: - return Option( - "-r", - "--requirement", - dest="requirements", - action="append", - default=[], - metavar="file", - help="Install from the given requirements file. " - "This option can be used multiple times.", - ) - - -def editable() -> Option: - return Option( - "-e", - "--editable", - dest="editables", - action="append", - default=[], - metavar="path/url", - help=( - "Install a project in editable mode (i.e. setuptools " - '"develop mode") from a local project path or a VCS url.' - ), - ) - - -def _handle_src(option: Option, opt_str: str, value: str, parser: OptionParser) -> None: - value = os.path.abspath(value) - setattr(parser.values, option.dest, value) - - -src: Callable[..., Option] = partial( - PipOption, - "--src", - "--source", - "--source-dir", - "--source-directory", - dest="src_dir", - type="path", - metavar="dir", - default=get_src_prefix(), - action="callback", - callback=_handle_src, - help="Directory to check out editable projects into. " - 'The default in a virtualenv is "/src". ' - 'The default for global installs is "/src".', -) - - -def _get_format_control(values: Values, option: Option) -> Any: - """Get a format_control object.""" - return getattr(values, option.dest) - - -def _handle_no_binary( - option: Option, opt_str: str, value: str, parser: OptionParser -) -> None: - existing = _get_format_control(parser.values, option) - FormatControl.handle_mutual_excludes( - value, - existing.no_binary, - existing.only_binary, - ) - - -def _handle_only_binary( - option: Option, opt_str: str, value: str, parser: OptionParser -) -> None: - existing = _get_format_control(parser.values, option) - FormatControl.handle_mutual_excludes( - value, - existing.only_binary, - existing.no_binary, - ) - - -def no_binary() -> Option: - format_control = FormatControl(set(), set()) - return Option( - "--no-binary", - dest="format_control", - action="callback", - callback=_handle_no_binary, - type="str", - default=format_control, - help="Do not use binary packages. Can be supplied multiple times, and " - 'each time adds to the existing value. Accepts either ":all:" to ' - 'disable all binary packages, ":none:" to empty the set (notice ' - "the colons), or one or more package names with commas between " - "them (no colons). Note that some packages are tricky to compile " - "and may fail to install when this option is used on them.", - ) - - -def only_binary() -> Option: - format_control = FormatControl(set(), set()) - return Option( - "--only-binary", - dest="format_control", - action="callback", - callback=_handle_only_binary, - type="str", - default=format_control, - help="Do not use source packages. Can be supplied multiple times, and " - 'each time adds to the existing value. Accepts either ":all:" to ' - 'disable all source packages, ":none:" to empty the set, or one ' - "or more package names with commas between them. Packages " - "without binary distributions will fail to install when this " - "option is used on them.", - ) - - -platforms: Callable[..., Option] = partial( - Option, - "--platform", - dest="platforms", - metavar="platform", - action="append", - default=None, - help=( - "Only use wheels compatible with . Defaults to the " - "platform of the running system. Use this option multiple times to " - "specify multiple platforms supported by the target interpreter." - ), -) - - -# This was made a separate function for unit-testing purposes. -def _convert_python_version(value: str) -> Tuple[Tuple[int, ...], Optional[str]]: - """ - Convert a version string like "3", "37", or "3.7.3" into a tuple of ints. - - :return: A 2-tuple (version_info, error_msg), where `error_msg` is - non-None if and only if there was a parsing error. - """ - if not value: - # The empty string is the same as not providing a value. - return (None, None) - - parts = value.split(".") - if len(parts) > 3: - return ((), "at most three version parts are allowed") - - if len(parts) == 1: - # Then we are in the case of "3" or "37". - value = parts[0] - if len(value) > 1: - parts = [value[0], value[1:]] - - try: - version_info = tuple(int(part) for part in parts) - except ValueError: - return ((), "each version part must be an integer") - - return (version_info, None) - - -def _handle_python_version( - option: Option, opt_str: str, value: str, parser: OptionParser -) -> None: - """ - Handle a provided --python-version value. - """ - version_info, error_msg = _convert_python_version(value) - if error_msg is not None: - msg = "invalid --python-version value: {!r}: {}".format( - value, - error_msg, - ) - raise_option_error(parser, option=option, msg=msg) - - parser.values.python_version = version_info - - -python_version: Callable[..., Option] = partial( - Option, - "--python-version", - dest="python_version", - metavar="python_version", - action="callback", - callback=_handle_python_version, - type="str", - default=None, - help=dedent( - """\ - The Python interpreter version to use for wheel and "Requires-Python" - compatibility checks. Defaults to a version derived from the running - interpreter. The version can be specified using up to three dot-separated - integers (e.g. "3" for 3.0.0, "3.7" for 3.7.0, or "3.7.3"). A major-minor - version can also be given as a string without dots (e.g. "37" for 3.7.0). - """ - ), -) - - -implementation: Callable[..., Option] = partial( - Option, - "--implementation", - dest="implementation", - metavar="implementation", - default=None, - help=( - "Only use wheels compatible with Python " - "implementation , e.g. 'pp', 'jy', 'cp', " - " or 'ip'. If not specified, then the current " - "interpreter implementation is used. Use 'py' to force " - "implementation-agnostic wheels." - ), -) - - -abis: Callable[..., Option] = partial( - Option, - "--abi", - dest="abis", - metavar="abi", - action="append", - default=None, - help=( - "Only use wheels compatible with Python abi , e.g. 'pypy_41'. " - "If not specified, then the current interpreter abi tag is used. " - "Use this option multiple times to specify multiple abis supported " - "by the target interpreter. Generally you will need to specify " - "--implementation, --platform, and --python-version when using this " - "option." - ), -) - - -def add_target_python_options(cmd_opts: OptionGroup) -> None: - cmd_opts.add_option(platforms()) - cmd_opts.add_option(python_version()) - cmd_opts.add_option(implementation()) - cmd_opts.add_option(abis()) - - -def make_target_python(options: Values) -> TargetPython: - target_python = TargetPython( - platforms=options.platforms, - py_version_info=options.python_version, - abis=options.abis, - implementation=options.implementation, - ) - - return target_python - - -def prefer_binary() -> Option: - return Option( - "--prefer-binary", - dest="prefer_binary", - action="store_true", - default=False, - help=( - "Prefer binary packages over source packages, even if the " - "source packages are newer." - ), - ) - - -cache_dir: Callable[..., Option] = partial( - PipOption, - "--cache-dir", - dest="cache_dir", - default=USER_CACHE_DIR, - metavar="dir", - type="path", - help="Store the cache data in .", -) - - -def _handle_no_cache_dir( - option: Option, opt: str, value: str, parser: OptionParser -) -> None: - """ - Process a value provided for the --no-cache-dir option. - - This is an optparse.Option callback for the --no-cache-dir option. - """ - # The value argument will be None if --no-cache-dir is passed via the - # command-line, since the option doesn't accept arguments. However, - # the value can be non-None if the option is triggered e.g. by an - # environment variable, like PIP_NO_CACHE_DIR=true. - if value is not None: - # Then parse the string value to get argument error-checking. - try: - strtobool(value) - except ValueError as exc: - raise_option_error(parser, option=option, msg=str(exc)) - - # Originally, setting PIP_NO_CACHE_DIR to a value that strtobool() - # converted to 0 (like "false" or "no") caused cache_dir to be disabled - # rather than enabled (logic would say the latter). Thus, we disable - # the cache directory not just on values that parse to True, but (for - # backwards compatibility reasons) also on values that parse to False. - # In other words, always set it to False if the option is provided in - # some (valid) form. - parser.values.cache_dir = False - - -no_cache: Callable[..., Option] = partial( - Option, - "--no-cache-dir", - dest="cache_dir", - action="callback", - callback=_handle_no_cache_dir, - help="Disable the cache.", -) - -no_deps: Callable[..., Option] = partial( - Option, - "--no-deps", - "--no-dependencies", - dest="ignore_dependencies", - action="store_true", - default=False, - help="Don't install package dependencies.", -) - -ignore_requires_python: Callable[..., Option] = partial( - Option, - "--ignore-requires-python", - dest="ignore_requires_python", - action="store_true", - help="Ignore the Requires-Python information.", -) - -no_build_isolation: Callable[..., Option] = partial( - Option, - "--no-build-isolation", - dest="build_isolation", - action="store_false", - default=True, - help="Disable isolation when building a modern source distribution. " - "Build dependencies specified by PEP 518 must be already installed " - "if this option is used.", -) - -check_build_deps: Callable[..., Option] = partial( - Option, - "--check-build-dependencies", - dest="check_build_deps", - action="store_true", - default=False, - help="Check the build dependencies when PEP517 is used.", -) - - -def _handle_no_use_pep517( - option: Option, opt: str, value: str, parser: OptionParser -) -> None: - """ - Process a value provided for the --no-use-pep517 option. - - This is an optparse.Option callback for the no_use_pep517 option. - """ - # Since --no-use-pep517 doesn't accept arguments, the value argument - # will be None if --no-use-pep517 is passed via the command-line. - # However, the value can be non-None if the option is triggered e.g. - # by an environment variable, for example "PIP_NO_USE_PEP517=true". - if value is not None: - msg = """A value was passed for --no-use-pep517, - probably using either the PIP_NO_USE_PEP517 environment variable - or the "no-use-pep517" config file option. Use an appropriate value - of the PIP_USE_PEP517 environment variable or the "use-pep517" - config file option instead. - """ - raise_option_error(parser, option=option, msg=msg) - - # If user doesn't wish to use pep517, we check if setuptools and wheel are installed - # and raise error if it is not. - packages = ("setuptools", "wheel") - if not all(importlib.util.find_spec(package) for package in packages): - msg = ( - f"It is not possible to use --no-use-pep517 " - f"without {' and '.join(packages)} installed." - ) - raise_option_error(parser, option=option, msg=msg) - - # Otherwise, --no-use-pep517 was passed via the command-line. - parser.values.use_pep517 = False - - -use_pep517: Any = partial( - Option, - "--use-pep517", - dest="use_pep517", - action="store_true", - default=None, - help="Use PEP 517 for building source distributions " - "(use --no-use-pep517 to force legacy behaviour).", -) - -no_use_pep517: Any = partial( - Option, - "--no-use-pep517", - dest="use_pep517", - action="callback", - callback=_handle_no_use_pep517, - default=None, - help=SUPPRESS_HELP, -) - - -def _handle_config_settings( - option: Option, opt_str: str, value: str, parser: OptionParser -) -> None: - key, sep, val = value.partition("=") - if sep != "=": - parser.error(f"Arguments to {opt_str} must be of the form KEY=VAL") - dest = getattr(parser.values, option.dest) - if dest is None: - dest = {} - setattr(parser.values, option.dest, dest) - if key in dest: - if isinstance(dest[key], list): - dest[key].append(val) - else: - dest[key] = [dest[key], val] - else: - dest[key] = val - - -config_settings: Callable[..., Option] = partial( - Option, - "-C", - "--config-settings", - dest="config_settings", - type=str, - action="callback", - callback=_handle_config_settings, - metavar="settings", - help="Configuration settings to be passed to the PEP 517 build backend. " - "Settings take the form KEY=VALUE. Use multiple --config-settings options " - "to pass multiple keys to the backend.", -) - -build_options: Callable[..., Option] = partial( - Option, - "--build-option", - dest="build_options", - metavar="options", - action="append", - help="Extra arguments to be supplied to 'setup.py bdist_wheel'.", -) - -global_options: Callable[..., Option] = partial( - Option, - "--global-option", - dest="global_options", - action="append", - metavar="options", - help="Extra global options to be supplied to the setup.py " - "call before the install or bdist_wheel command.", -) - -no_clean: Callable[..., Option] = partial( - Option, - "--no-clean", - action="store_true", - default=False, - help="Don't clean up build directories.", -) - -pre: Callable[..., Option] = partial( - Option, - "--pre", - action="store_true", - default=False, - help="Include pre-release and development versions. By default, " - "pip only finds stable versions.", -) - -disable_pip_version_check: Callable[..., Option] = partial( - Option, - "--disable-pip-version-check", - dest="disable_pip_version_check", - action="store_true", - default=False, - help="Don't periodically check PyPI to determine whether a new version " - "of pip is available for download. Implied with --no-index.", -) - -root_user_action: Callable[..., Option] = partial( - Option, - "--root-user-action", - dest="root_user_action", - default="warn", - choices=["warn", "ignore"], - help="Action if pip is run as a root user. By default, a warning message is shown.", -) - - -def _handle_merge_hash( - option: Option, opt_str: str, value: str, parser: OptionParser -) -> None: - """Given a value spelled "algo:digest", append the digest to a list - pointed to in a dict by the algo name.""" - if not parser.values.hashes: - parser.values.hashes = {} - try: - algo, digest = value.split(":", 1) - except ValueError: - parser.error( - "Arguments to {} must be a hash name " - "followed by a value, like --hash=sha256:" - "abcde...".format(opt_str) - ) - if algo not in STRONG_HASHES: - parser.error( - "Allowed hash algorithms for {} are {}.".format( - opt_str, ", ".join(STRONG_HASHES) - ) - ) - parser.values.hashes.setdefault(algo, []).append(digest) - - -hash: Callable[..., Option] = partial( - Option, - "--hash", - # Hash values eventually end up in InstallRequirement.hashes due to - # __dict__ copying in process_line(). - dest="hashes", - action="callback", - callback=_handle_merge_hash, - type="string", - help="Verify that the package's archive matches this " - "hash before installing. Example: --hash=sha256:abcdef...", -) - - -require_hashes: Callable[..., Option] = partial( - Option, - "--require-hashes", - dest="require_hashes", - action="store_true", - default=False, - help="Require a hash to check each requirement against, for " - "repeatable installs. This option is implied when any package in a " - "requirements file has a --hash option.", -) - - -list_path: Callable[..., Option] = partial( - PipOption, - "--path", - dest="path", - type="path", - action="append", - help="Restrict to the specified installation path for listing " - "packages (can be used multiple times).", -) - - -def check_list_path_option(options: Values) -> None: - if options.path and (options.user or options.local): - raise CommandError("Cannot combine '--path' with '--user' or '--local'") - - -list_exclude: Callable[..., Option] = partial( - PipOption, - "--exclude", - dest="excludes", - action="append", - metavar="package", - type="package_name", - help="Exclude specified package from the output", -) - - -no_python_version_warning: Callable[..., Option] = partial( - Option, - "--no-python-version-warning", - dest="no_python_version_warning", - action="store_true", - default=False, - help="Silence deprecation warnings for upcoming unsupported Pythons.", -) - - -# Features that are now always on. A warning is printed if they are used. -ALWAYS_ENABLED_FEATURES = [ - "no-binary-enable-wheel-cache", # always on since 23.1 -] - -use_new_feature: Callable[..., Option] = partial( - Option, - "--use-feature", - dest="features_enabled", - metavar="feature", - action="append", - default=[], - choices=[ - "fast-deps", - "truststore", - ] - + ALWAYS_ENABLED_FEATURES, - help="Enable new functionality, that may be backward incompatible.", -) - -use_deprecated_feature: Callable[..., Option] = partial( - Option, - "--use-deprecated", - dest="deprecated_features_enabled", - metavar="feature", - action="append", - default=[], - choices=[ - "legacy-resolver", - ], - help=("Enable deprecated functionality, that will be removed in the future."), -) - - -########## -# groups # -########## - -general_group: Dict[str, Any] = { - "name": "General Options", - "options": [ - help_, - debug_mode, - isolated_mode, - require_virtualenv, - python, - verbose, - version, - quiet, - log, - no_input, - keyring_provider, - proxy, - retries, - timeout, - exists_action, - trusted_host, - cert, - client_cert, - cache_dir, - no_cache, - disable_pip_version_check, - no_color, - no_python_version_warning, - use_new_feature, - use_deprecated_feature, - ], -} - -index_group: Dict[str, Any] = { - "name": "Package Index Options", - "options": [ - index_url, - extra_index_url, - no_index, - find_links, - ], -} diff --git a/venv/lib/python3.11/site-packages/pip/_internal/cli/command_context.py b/venv/lib/python3.11/site-packages/pip/_internal/cli/command_context.py deleted file mode 100644 index 139995a..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/cli/command_context.py +++ /dev/null @@ -1,27 +0,0 @@ -from contextlib import ExitStack, contextmanager -from typing import ContextManager, Generator, TypeVar - -_T = TypeVar("_T", covariant=True) - - -class CommandContextMixIn: - def __init__(self) -> None: - super().__init__() - self._in_main_context = False - self._main_context = ExitStack() - - @contextmanager - def main_context(self) -> Generator[None, None, None]: - assert not self._in_main_context - - self._in_main_context = True - try: - with self._main_context: - yield - finally: - self._in_main_context = False - - def enter_context(self, context_provider: ContextManager[_T]) -> _T: - assert self._in_main_context - - return self._main_context.enter_context(context_provider) diff --git a/venv/lib/python3.11/site-packages/pip/_internal/cli/main.py b/venv/lib/python3.11/site-packages/pip/_internal/cli/main.py deleted file mode 100644 index 7e061f5..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/cli/main.py +++ /dev/null @@ -1,79 +0,0 @@ -"""Primary application entrypoint. -""" -import locale -import logging -import os -import sys -import warnings -from typing import List, Optional - -from pip._internal.cli.autocompletion import autocomplete -from pip._internal.cli.main_parser import parse_command -from pip._internal.commands import create_command -from pip._internal.exceptions import PipError -from pip._internal.utils import deprecation - -logger = logging.getLogger(__name__) - - -# Do not import and use main() directly! Using it directly is actively -# discouraged by pip's maintainers. The name, location and behavior of -# this function is subject to change, so calling it directly is not -# portable across different pip versions. - -# In addition, running pip in-process is unsupported and unsafe. This is -# elaborated in detail at -# https://pip.pypa.io/en/stable/user_guide/#using-pip-from-your-program. -# That document also provides suggestions that should work for nearly -# all users that are considering importing and using main() directly. - -# However, we know that certain users will still want to invoke pip -# in-process. If you understand and accept the implications of using pip -# in an unsupported manner, the best approach is to use runpy to avoid -# depending on the exact location of this entry point. - -# The following example shows how to use runpy to invoke pip in that -# case: -# -# sys.argv = ["pip", your, args, here] -# runpy.run_module("pip", run_name="__main__") -# -# Note that this will exit the process after running, unlike a direct -# call to main. As it is not safe to do any processing after calling -# main, this should not be an issue in practice. - - -def main(args: Optional[List[str]] = None) -> int: - if args is None: - args = sys.argv[1:] - - # Suppress the pkg_resources deprecation warning - # Note - we use a module of .*pkg_resources to cover - # the normal case (pip._vendor.pkg_resources) and the - # devendored case (a bare pkg_resources) - warnings.filterwarnings( - action="ignore", category=DeprecationWarning, module=".*pkg_resources" - ) - - # Configure our deprecation warnings to be sent through loggers - deprecation.install_warning_logger() - - autocomplete() - - try: - cmd_name, cmd_args = parse_command(args) - except PipError as exc: - sys.stderr.write(f"ERROR: {exc}") - sys.stderr.write(os.linesep) - sys.exit(1) - - # Needed for locale.getpreferredencoding(False) to work - # in pip._internal.utils.encoding.auto_decode - try: - locale.setlocale(locale.LC_ALL, "") - except locale.Error as e: - # setlocale can apparently crash if locale are uninitialized - logger.debug("Ignoring error %s when setting locale", e) - command = create_command(cmd_name, isolated=("--isolated" in cmd_args)) - - return command.main(cmd_args) diff --git a/venv/lib/python3.11/site-packages/pip/_internal/cli/main_parser.py b/venv/lib/python3.11/site-packages/pip/_internal/cli/main_parser.py deleted file mode 100644 index 5ade356..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/cli/main_parser.py +++ /dev/null @@ -1,134 +0,0 @@ -"""A single place for constructing and exposing the main parser -""" - -import os -import subprocess -import sys -from typing import List, Optional, Tuple - -from pip._internal.build_env import get_runnable_pip -from pip._internal.cli import cmdoptions -from pip._internal.cli.parser import ConfigOptionParser, UpdatingDefaultsHelpFormatter -from pip._internal.commands import commands_dict, get_similar_commands -from pip._internal.exceptions import CommandError -from pip._internal.utils.misc import get_pip_version, get_prog - -__all__ = ["create_main_parser", "parse_command"] - - -def create_main_parser() -> ConfigOptionParser: - """Creates and returns the main parser for pip's CLI""" - - parser = ConfigOptionParser( - usage="\n%prog [options]", - add_help_option=False, - formatter=UpdatingDefaultsHelpFormatter(), - name="global", - prog=get_prog(), - ) - parser.disable_interspersed_args() - - parser.version = get_pip_version() - - # add the general options - gen_opts = cmdoptions.make_option_group(cmdoptions.general_group, parser) - parser.add_option_group(gen_opts) - - # so the help formatter knows - parser.main = True # type: ignore - - # create command listing for description - description = [""] + [ - f"{name:27} {command_info.summary}" - for name, command_info in commands_dict.items() - ] - parser.description = "\n".join(description) - - return parser - - -def identify_python_interpreter(python: str) -> Optional[str]: - # If the named file exists, use it. - # If it's a directory, assume it's a virtual environment and - # look for the environment's Python executable. - if os.path.exists(python): - if os.path.isdir(python): - # bin/python for Unix, Scripts/python.exe for Windows - # Try both in case of odd cases like cygwin. - for exe in ("bin/python", "Scripts/python.exe"): - py = os.path.join(python, exe) - if os.path.exists(py): - return py - else: - return python - - # Could not find the interpreter specified - return None - - -def parse_command(args: List[str]) -> Tuple[str, List[str]]: - parser = create_main_parser() - - # Note: parser calls disable_interspersed_args(), so the result of this - # call is to split the initial args into the general options before the - # subcommand and everything else. - # For example: - # args: ['--timeout=5', 'install', '--user', 'INITools'] - # general_options: ['--timeout==5'] - # args_else: ['install', '--user', 'INITools'] - general_options, args_else = parser.parse_args(args) - - # --python - if general_options.python and "_PIP_RUNNING_IN_SUBPROCESS" not in os.environ: - # Re-invoke pip using the specified Python interpreter - interpreter = identify_python_interpreter(general_options.python) - if interpreter is None: - raise CommandError( - f"Could not locate Python interpreter {general_options.python}" - ) - - pip_cmd = [ - interpreter, - get_runnable_pip(), - ] - pip_cmd.extend(args) - - # Set a flag so the child doesn't re-invoke itself, causing - # an infinite loop. - os.environ["_PIP_RUNNING_IN_SUBPROCESS"] = "1" - returncode = 0 - try: - proc = subprocess.run(pip_cmd) - returncode = proc.returncode - except (subprocess.SubprocessError, OSError) as exc: - raise CommandError(f"Failed to run pip under {interpreter}: {exc}") - sys.exit(returncode) - - # --version - if general_options.version: - sys.stdout.write(parser.version) - sys.stdout.write(os.linesep) - sys.exit() - - # pip || pip help -> print_help() - if not args_else or (args_else[0] == "help" and len(args_else) == 1): - parser.print_help() - sys.exit() - - # the subcommand name - cmd_name = args_else[0] - - if cmd_name not in commands_dict: - guess = get_similar_commands(cmd_name) - - msg = [f'unknown command "{cmd_name}"'] - if guess: - msg.append(f'maybe you meant "{guess}"') - - raise CommandError(" - ".join(msg)) - - # all the args without the subcommand - cmd_args = args[:] - cmd_args.remove(cmd_name) - - return cmd_name, cmd_args diff --git a/venv/lib/python3.11/site-packages/pip/_internal/cli/parser.py b/venv/lib/python3.11/site-packages/pip/_internal/cli/parser.py deleted file mode 100644 index 64cf971..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/cli/parser.py +++ /dev/null @@ -1,294 +0,0 @@ -"""Base option parser setup""" - -import logging -import optparse -import shutil -import sys -import textwrap -from contextlib import suppress -from typing import Any, Dict, Generator, List, Tuple - -from pip._internal.cli.status_codes import UNKNOWN_ERROR -from pip._internal.configuration import Configuration, ConfigurationError -from pip._internal.utils.misc import redact_auth_from_url, strtobool - -logger = logging.getLogger(__name__) - - -class PrettyHelpFormatter(optparse.IndentedHelpFormatter): - """A prettier/less verbose help formatter for optparse.""" - - def __init__(self, *args: Any, **kwargs: Any) -> None: - # help position must be aligned with __init__.parseopts.description - kwargs["max_help_position"] = 30 - kwargs["indent_increment"] = 1 - kwargs["width"] = shutil.get_terminal_size()[0] - 2 - super().__init__(*args, **kwargs) - - def format_option_strings(self, option: optparse.Option) -> str: - return self._format_option_strings(option) - - def _format_option_strings( - self, option: optparse.Option, mvarfmt: str = " <{}>", optsep: str = ", " - ) -> str: - """ - Return a comma-separated list of option strings and metavars. - - :param option: tuple of (short opt, long opt), e.g: ('-f', '--format') - :param mvarfmt: metavar format string - :param optsep: separator - """ - opts = [] - - if option._short_opts: - opts.append(option._short_opts[0]) - if option._long_opts: - opts.append(option._long_opts[0]) - if len(opts) > 1: - opts.insert(1, optsep) - - if option.takes_value(): - assert option.dest is not None - metavar = option.metavar or option.dest.lower() - opts.append(mvarfmt.format(metavar.lower())) - - return "".join(opts) - - def format_heading(self, heading: str) -> str: - if heading == "Options": - return "" - return heading + ":\n" - - def format_usage(self, usage: str) -> str: - """ - Ensure there is only one newline between usage and the first heading - if there is no description. - """ - msg = "\nUsage: {}\n".format(self.indent_lines(textwrap.dedent(usage), " ")) - return msg - - def format_description(self, description: str) -> str: - # leave full control over description to us - if description: - if hasattr(self.parser, "main"): - label = "Commands" - else: - label = "Description" - # some doc strings have initial newlines, some don't - description = description.lstrip("\n") - # some doc strings have final newlines and spaces, some don't - description = description.rstrip() - # dedent, then reindent - description = self.indent_lines(textwrap.dedent(description), " ") - description = f"{label}:\n{description}\n" - return description - else: - return "" - - def format_epilog(self, epilog: str) -> str: - # leave full control over epilog to us - if epilog: - return epilog - else: - return "" - - def indent_lines(self, text: str, indent: str) -> str: - new_lines = [indent + line for line in text.split("\n")] - return "\n".join(new_lines) - - -class UpdatingDefaultsHelpFormatter(PrettyHelpFormatter): - """Custom help formatter for use in ConfigOptionParser. - - This is updates the defaults before expanding them, allowing - them to show up correctly in the help listing. - - Also redact auth from url type options - """ - - def expand_default(self, option: optparse.Option) -> str: - default_values = None - if self.parser is not None: - assert isinstance(self.parser, ConfigOptionParser) - self.parser._update_defaults(self.parser.defaults) - assert option.dest is not None - default_values = self.parser.defaults.get(option.dest) - help_text = super().expand_default(option) - - if default_values and option.metavar == "URL": - if isinstance(default_values, str): - default_values = [default_values] - - # If its not a list, we should abort and just return the help text - if not isinstance(default_values, list): - default_values = [] - - for val in default_values: - help_text = help_text.replace(val, redact_auth_from_url(val)) - - return help_text - - -class CustomOptionParser(optparse.OptionParser): - def insert_option_group( - self, idx: int, *args: Any, **kwargs: Any - ) -> optparse.OptionGroup: - """Insert an OptionGroup at a given position.""" - group = self.add_option_group(*args, **kwargs) - - self.option_groups.pop() - self.option_groups.insert(idx, group) - - return group - - @property - def option_list_all(self) -> List[optparse.Option]: - """Get a list of all options, including those in option groups.""" - res = self.option_list[:] - for i in self.option_groups: - res.extend(i.option_list) - - return res - - -class ConfigOptionParser(CustomOptionParser): - """Custom option parser which updates its defaults by checking the - configuration files and environmental variables""" - - def __init__( - self, - *args: Any, - name: str, - isolated: bool = False, - **kwargs: Any, - ) -> None: - self.name = name - self.config = Configuration(isolated) - - assert self.name - super().__init__(*args, **kwargs) - - def check_default(self, option: optparse.Option, key: str, val: Any) -> Any: - try: - return option.check_value(key, val) - except optparse.OptionValueError as exc: - print(f"An error occurred during configuration: {exc}") - sys.exit(3) - - def _get_ordered_configuration_items( - self, - ) -> Generator[Tuple[str, Any], None, None]: - # Configuration gives keys in an unordered manner. Order them. - override_order = ["global", self.name, ":env:"] - - # Pool the options into different groups - section_items: Dict[str, List[Tuple[str, Any]]] = { - name: [] for name in override_order - } - for section_key, val in self.config.items(): - # ignore empty values - if not val: - logger.debug( - "Ignoring configuration key '%s' as it's value is empty.", - section_key, - ) - continue - - section, key = section_key.split(".", 1) - if section in override_order: - section_items[section].append((key, val)) - - # Yield each group in their override order - for section in override_order: - for key, val in section_items[section]: - yield key, val - - def _update_defaults(self, defaults: Dict[str, Any]) -> Dict[str, Any]: - """Updates the given defaults with values from the config files and - the environ. Does a little special handling for certain types of - options (lists).""" - - # Accumulate complex default state. - self.values = optparse.Values(self.defaults) - late_eval = set() - # Then set the options with those values - for key, val in self._get_ordered_configuration_items(): - # '--' because configuration supports only long names - option = self.get_option("--" + key) - - # Ignore options not present in this parser. E.g. non-globals put - # in [global] by users that want them to apply to all applicable - # commands. - if option is None: - continue - - assert option.dest is not None - - if option.action in ("store_true", "store_false"): - try: - val = strtobool(val) - except ValueError: - self.error( - "{} is not a valid value for {} option, " - "please specify a boolean value like yes/no, " - "true/false or 1/0 instead.".format(val, key) - ) - elif option.action == "count": - with suppress(ValueError): - val = strtobool(val) - with suppress(ValueError): - val = int(val) - if not isinstance(val, int) or val < 0: - self.error( - "{} is not a valid value for {} option, " - "please instead specify either a non-negative integer " - "or a boolean value like yes/no or false/true " - "which is equivalent to 1/0.".format(val, key) - ) - elif option.action == "append": - val = val.split() - val = [self.check_default(option, key, v) for v in val] - elif option.action == "callback": - assert option.callback is not None - late_eval.add(option.dest) - opt_str = option.get_opt_string() - val = option.convert_value(opt_str, val) - # From take_action - args = option.callback_args or () - kwargs = option.callback_kwargs or {} - option.callback(option, opt_str, val, self, *args, **kwargs) - else: - val = self.check_default(option, key, val) - - defaults[option.dest] = val - - for key in late_eval: - defaults[key] = getattr(self.values, key) - self.values = None - return defaults - - def get_default_values(self) -> optparse.Values: - """Overriding to make updating the defaults after instantiation of - the option parser possible, _update_defaults() does the dirty work.""" - if not self.process_default_values: - # Old, pre-Optik 1.5 behaviour. - return optparse.Values(self.defaults) - - # Load the configuration, or error out in case of an error - try: - self.config.load() - except ConfigurationError as err: - self.exit(UNKNOWN_ERROR, str(err)) - - defaults = self._update_defaults(self.defaults.copy()) # ours - for option in self._get_all_options(): - assert option.dest is not None - default = defaults.get(option.dest) - if isinstance(default, str): - opt_str = option.get_opt_string() - defaults[option.dest] = option.check_value(opt_str, default) - return optparse.Values(defaults) - - def error(self, msg: str) -> None: - self.print_usage(sys.stderr) - self.exit(UNKNOWN_ERROR, f"{msg}\n") diff --git a/venv/lib/python3.11/site-packages/pip/_internal/cli/progress_bars.py b/venv/lib/python3.11/site-packages/pip/_internal/cli/progress_bars.py deleted file mode 100644 index 0ad1403..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/cli/progress_bars.py +++ /dev/null @@ -1,68 +0,0 @@ -import functools -from typing import Callable, Generator, Iterable, Iterator, Optional, Tuple - -from pip._vendor.rich.progress import ( - BarColumn, - DownloadColumn, - FileSizeColumn, - Progress, - ProgressColumn, - SpinnerColumn, - TextColumn, - TimeElapsedColumn, - TimeRemainingColumn, - TransferSpeedColumn, -) - -from pip._internal.utils.logging import get_indentation - -DownloadProgressRenderer = Callable[[Iterable[bytes]], Iterator[bytes]] - - -def _rich_progress_bar( - iterable: Iterable[bytes], - *, - bar_type: str, - size: int, -) -> Generator[bytes, None, None]: - assert bar_type == "on", "This should only be used in the default mode." - - if not size: - total = float("inf") - columns: Tuple[ProgressColumn, ...] = ( - TextColumn("[progress.description]{task.description}"), - SpinnerColumn("line", speed=1.5), - FileSizeColumn(), - TransferSpeedColumn(), - TimeElapsedColumn(), - ) - else: - total = size - columns = ( - TextColumn("[progress.description]{task.description}"), - BarColumn(), - DownloadColumn(), - TransferSpeedColumn(), - TextColumn("eta"), - TimeRemainingColumn(), - ) - - progress = Progress(*columns, refresh_per_second=30) - task_id = progress.add_task(" " * (get_indentation() + 2), total=total) - with progress: - for chunk in iterable: - yield chunk - progress.update(task_id, advance=len(chunk)) - - -def get_download_progress_renderer( - *, bar_type: str, size: Optional[int] = None -) -> DownloadProgressRenderer: - """Get an object that can be used to render the download progress. - - Returns a callable, that takes an iterable to "wrap". - """ - if bar_type == "on": - return functools.partial(_rich_progress_bar, bar_type=bar_type, size=size) - else: - return iter # no-op, when passed an iterator diff --git a/venv/lib/python3.11/site-packages/pip/_internal/cli/req_command.py b/venv/lib/python3.11/site-packages/pip/_internal/cli/req_command.py deleted file mode 100644 index 6f2f79c..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/cli/req_command.py +++ /dev/null @@ -1,505 +0,0 @@ -"""Contains the Command base classes that depend on PipSession. - -The classes in this module are in a separate module so the commands not -needing download / PackageFinder capability don't unnecessarily import the -PackageFinder machinery and all its vendored dependencies, etc. -""" - -import logging -import os -import sys -from functools import partial -from optparse import Values -from typing import TYPE_CHECKING, Any, List, Optional, Tuple - -from pip._internal.cache import WheelCache -from pip._internal.cli import cmdoptions -from pip._internal.cli.base_command import Command -from pip._internal.cli.command_context import CommandContextMixIn -from pip._internal.exceptions import CommandError, PreviousBuildDirError -from pip._internal.index.collector import LinkCollector -from pip._internal.index.package_finder import PackageFinder -from pip._internal.models.selection_prefs import SelectionPreferences -from pip._internal.models.target_python import TargetPython -from pip._internal.network.session import PipSession -from pip._internal.operations.build.build_tracker import BuildTracker -from pip._internal.operations.prepare import RequirementPreparer -from pip._internal.req.constructors import ( - install_req_from_editable, - install_req_from_line, - install_req_from_parsed_requirement, - install_req_from_req_string, -) -from pip._internal.req.req_file import parse_requirements -from pip._internal.req.req_install import InstallRequirement -from pip._internal.resolution.base import BaseResolver -from pip._internal.self_outdated_check import pip_self_version_check -from pip._internal.utils.temp_dir import ( - TempDirectory, - TempDirectoryTypeRegistry, - tempdir_kinds, -) -from pip._internal.utils.virtualenv import running_under_virtualenv - -if TYPE_CHECKING: - from ssl import SSLContext - -logger = logging.getLogger(__name__) - - -def _create_truststore_ssl_context() -> Optional["SSLContext"]: - if sys.version_info < (3, 10): - raise CommandError("The truststore feature is only available for Python 3.10+") - - try: - import ssl - except ImportError: - logger.warning("Disabling truststore since ssl support is missing") - return None - - try: - from pip._vendor import truststore - except ImportError as e: - raise CommandError(f"The truststore feature is unavailable: {e}") - - return truststore.SSLContext(ssl.PROTOCOL_TLS_CLIENT) - - -class SessionCommandMixin(CommandContextMixIn): - - """ - A class mixin for command classes needing _build_session(). - """ - - def __init__(self) -> None: - super().__init__() - self._session: Optional[PipSession] = None - - @classmethod - def _get_index_urls(cls, options: Values) -> Optional[List[str]]: - """Return a list of index urls from user-provided options.""" - index_urls = [] - if not getattr(options, "no_index", False): - url = getattr(options, "index_url", None) - if url: - index_urls.append(url) - urls = getattr(options, "extra_index_urls", None) - if urls: - index_urls.extend(urls) - # Return None rather than an empty list - return index_urls or None - - def get_default_session(self, options: Values) -> PipSession: - """Get a default-managed session.""" - if self._session is None: - self._session = self.enter_context(self._build_session(options)) - # there's no type annotation on requests.Session, so it's - # automatically ContextManager[Any] and self._session becomes Any, - # then https://github.com/python/mypy/issues/7696 kicks in - assert self._session is not None - return self._session - - def _build_session( - self, - options: Values, - retries: Optional[int] = None, - timeout: Optional[int] = None, - fallback_to_certifi: bool = False, - ) -> PipSession: - cache_dir = options.cache_dir - assert not cache_dir or os.path.isabs(cache_dir) - - if "truststore" in options.features_enabled: - try: - ssl_context = _create_truststore_ssl_context() - except Exception: - if not fallback_to_certifi: - raise - ssl_context = None - else: - ssl_context = None - - session = PipSession( - cache=os.path.join(cache_dir, "http-v2") if cache_dir else None, - retries=retries if retries is not None else options.retries, - trusted_hosts=options.trusted_hosts, - index_urls=self._get_index_urls(options), - ssl_context=ssl_context, - ) - - # Handle custom ca-bundles from the user - if options.cert: - session.verify = options.cert - - # Handle SSL client certificate - if options.client_cert: - session.cert = options.client_cert - - # Handle timeouts - if options.timeout or timeout: - session.timeout = timeout if timeout is not None else options.timeout - - # Handle configured proxies - if options.proxy: - session.proxies = { - "http": options.proxy, - "https": options.proxy, - } - - # Determine if we can prompt the user for authentication or not - session.auth.prompting = not options.no_input - session.auth.keyring_provider = options.keyring_provider - - return session - - -class IndexGroupCommand(Command, SessionCommandMixin): - - """ - Abstract base class for commands with the index_group options. - - This also corresponds to the commands that permit the pip version check. - """ - - def handle_pip_version_check(self, options: Values) -> None: - """ - Do the pip version check if not disabled. - - This overrides the default behavior of not doing the check. - """ - # Make sure the index_group options are present. - assert hasattr(options, "no_index") - - if options.disable_pip_version_check or options.no_index: - return - - # Otherwise, check if we're using the latest version of pip available. - session = self._build_session( - options, - retries=0, - timeout=min(5, options.timeout), - # This is set to ensure the function does not fail when truststore is - # specified in use-feature but cannot be loaded. This usually raises a - # CommandError and shows a nice user-facing error, but this function is not - # called in that try-except block. - fallback_to_certifi=True, - ) - with session: - pip_self_version_check(session, options) - - -KEEPABLE_TEMPDIR_TYPES = [ - tempdir_kinds.BUILD_ENV, - tempdir_kinds.EPHEM_WHEEL_CACHE, - tempdir_kinds.REQ_BUILD, -] - - -def warn_if_run_as_root() -> None: - """Output a warning for sudo users on Unix. - - In a virtual environment, sudo pip still writes to virtualenv. - On Windows, users may run pip as Administrator without issues. - This warning only applies to Unix root users outside of virtualenv. - """ - if running_under_virtualenv(): - return - if not hasattr(os, "getuid"): - return - # On Windows, there are no "system managed" Python packages. Installing as - # Administrator via pip is the correct way of updating system environments. - # - # We choose sys.platform over utils.compat.WINDOWS here to enable Mypy platform - # checks: https://mypy.readthedocs.io/en/stable/common_issues.html - if sys.platform == "win32" or sys.platform == "cygwin": - return - - if os.getuid() != 0: - return - - logger.warning( - "Running pip as the 'root' user can result in broken permissions and " - "conflicting behaviour with the system package manager. " - "It is recommended to use a virtual environment instead: " - "https://pip.pypa.io/warnings/venv" - ) - - -def with_cleanup(func: Any) -> Any: - """Decorator for common logic related to managing temporary - directories. - """ - - def configure_tempdir_registry(registry: TempDirectoryTypeRegistry) -> None: - for t in KEEPABLE_TEMPDIR_TYPES: - registry.set_delete(t, False) - - def wrapper( - self: RequirementCommand, options: Values, args: List[Any] - ) -> Optional[int]: - assert self.tempdir_registry is not None - if options.no_clean: - configure_tempdir_registry(self.tempdir_registry) - - try: - return func(self, options, args) - except PreviousBuildDirError: - # This kind of conflict can occur when the user passes an explicit - # build directory with a pre-existing folder. In that case we do - # not want to accidentally remove it. - configure_tempdir_registry(self.tempdir_registry) - raise - - return wrapper - - -class RequirementCommand(IndexGroupCommand): - def __init__(self, *args: Any, **kw: Any) -> None: - super().__init__(*args, **kw) - - self.cmd_opts.add_option(cmdoptions.no_clean()) - - @staticmethod - def determine_resolver_variant(options: Values) -> str: - """Determines which resolver should be used, based on the given options.""" - if "legacy-resolver" in options.deprecated_features_enabled: - return "legacy" - - return "resolvelib" - - @classmethod - def make_requirement_preparer( - cls, - temp_build_dir: TempDirectory, - options: Values, - build_tracker: BuildTracker, - session: PipSession, - finder: PackageFinder, - use_user_site: bool, - download_dir: Optional[str] = None, - verbosity: int = 0, - ) -> RequirementPreparer: - """ - Create a RequirementPreparer instance for the given parameters. - """ - temp_build_dir_path = temp_build_dir.path - assert temp_build_dir_path is not None - legacy_resolver = False - - resolver_variant = cls.determine_resolver_variant(options) - if resolver_variant == "resolvelib": - lazy_wheel = "fast-deps" in options.features_enabled - if lazy_wheel: - logger.warning( - "pip is using lazily downloaded wheels using HTTP " - "range requests to obtain dependency information. " - "This experimental feature is enabled through " - "--use-feature=fast-deps and it is not ready for " - "production." - ) - else: - legacy_resolver = True - lazy_wheel = False - if "fast-deps" in options.features_enabled: - logger.warning( - "fast-deps has no effect when used with the legacy resolver." - ) - - return RequirementPreparer( - build_dir=temp_build_dir_path, - src_dir=options.src_dir, - download_dir=download_dir, - build_isolation=options.build_isolation, - check_build_deps=options.check_build_deps, - build_tracker=build_tracker, - session=session, - progress_bar=options.progress_bar, - finder=finder, - require_hashes=options.require_hashes, - use_user_site=use_user_site, - lazy_wheel=lazy_wheel, - verbosity=verbosity, - legacy_resolver=legacy_resolver, - ) - - @classmethod - def make_resolver( - cls, - preparer: RequirementPreparer, - finder: PackageFinder, - options: Values, - wheel_cache: Optional[WheelCache] = None, - use_user_site: bool = False, - ignore_installed: bool = True, - ignore_requires_python: bool = False, - force_reinstall: bool = False, - upgrade_strategy: str = "to-satisfy-only", - use_pep517: Optional[bool] = None, - py_version_info: Optional[Tuple[int, ...]] = None, - ) -> BaseResolver: - """ - Create a Resolver instance for the given parameters. - """ - make_install_req = partial( - install_req_from_req_string, - isolated=options.isolated_mode, - use_pep517=use_pep517, - ) - resolver_variant = cls.determine_resolver_variant(options) - # The long import name and duplicated invocation is needed to convince - # Mypy into correctly typechecking. Otherwise it would complain the - # "Resolver" class being redefined. - if resolver_variant == "resolvelib": - import pip._internal.resolution.resolvelib.resolver - - return pip._internal.resolution.resolvelib.resolver.Resolver( - preparer=preparer, - finder=finder, - wheel_cache=wheel_cache, - make_install_req=make_install_req, - use_user_site=use_user_site, - ignore_dependencies=options.ignore_dependencies, - ignore_installed=ignore_installed, - ignore_requires_python=ignore_requires_python, - force_reinstall=force_reinstall, - upgrade_strategy=upgrade_strategy, - py_version_info=py_version_info, - ) - import pip._internal.resolution.legacy.resolver - - return pip._internal.resolution.legacy.resolver.Resolver( - preparer=preparer, - finder=finder, - wheel_cache=wheel_cache, - make_install_req=make_install_req, - use_user_site=use_user_site, - ignore_dependencies=options.ignore_dependencies, - ignore_installed=ignore_installed, - ignore_requires_python=ignore_requires_python, - force_reinstall=force_reinstall, - upgrade_strategy=upgrade_strategy, - py_version_info=py_version_info, - ) - - def get_requirements( - self, - args: List[str], - options: Values, - finder: PackageFinder, - session: PipSession, - ) -> List[InstallRequirement]: - """ - Parse command-line arguments into the corresponding requirements. - """ - requirements: List[InstallRequirement] = [] - for filename in options.constraints: - for parsed_req in parse_requirements( - filename, - constraint=True, - finder=finder, - options=options, - session=session, - ): - req_to_add = install_req_from_parsed_requirement( - parsed_req, - isolated=options.isolated_mode, - user_supplied=False, - ) - requirements.append(req_to_add) - - for req in args: - req_to_add = install_req_from_line( - req, - comes_from=None, - isolated=options.isolated_mode, - use_pep517=options.use_pep517, - user_supplied=True, - config_settings=getattr(options, "config_settings", None), - ) - requirements.append(req_to_add) - - for req in options.editables: - req_to_add = install_req_from_editable( - req, - user_supplied=True, - isolated=options.isolated_mode, - use_pep517=options.use_pep517, - config_settings=getattr(options, "config_settings", None), - ) - requirements.append(req_to_add) - - # NOTE: options.require_hashes may be set if --require-hashes is True - for filename in options.requirements: - for parsed_req in parse_requirements( - filename, finder=finder, options=options, session=session - ): - req_to_add = install_req_from_parsed_requirement( - parsed_req, - isolated=options.isolated_mode, - use_pep517=options.use_pep517, - user_supplied=True, - config_settings=parsed_req.options.get("config_settings") - if parsed_req.options - else None, - ) - requirements.append(req_to_add) - - # If any requirement has hash options, enable hash checking. - if any(req.has_hash_options for req in requirements): - options.require_hashes = True - - if not (args or options.editables or options.requirements): - opts = {"name": self.name} - if options.find_links: - raise CommandError( - "You must give at least one requirement to {name} " - '(maybe you meant "pip {name} {links}"?)'.format( - **dict(opts, links=" ".join(options.find_links)) - ) - ) - else: - raise CommandError( - "You must give at least one requirement to {name} " - '(see "pip help {name}")'.format(**opts) - ) - - return requirements - - @staticmethod - def trace_basic_info(finder: PackageFinder) -> None: - """ - Trace basic information about the provided objects. - """ - # Display where finder is looking for packages - search_scope = finder.search_scope - locations = search_scope.get_formatted_locations() - if locations: - logger.info(locations) - - def _build_package_finder( - self, - options: Values, - session: PipSession, - target_python: Optional[TargetPython] = None, - ignore_requires_python: Optional[bool] = None, - ) -> PackageFinder: - """ - Create a package finder appropriate to this requirement command. - - :param ignore_requires_python: Whether to ignore incompatible - "Requires-Python" values in links. Defaults to False. - """ - link_collector = LinkCollector.create(session, options=options) - selection_prefs = SelectionPreferences( - allow_yanked=True, - format_control=options.format_control, - allow_all_prereleases=options.pre, - prefer_binary=options.prefer_binary, - ignore_requires_python=ignore_requires_python, - ) - - return PackageFinder.create( - link_collector=link_collector, - selection_prefs=selection_prefs, - target_python=target_python, - ) diff --git a/venv/lib/python3.11/site-packages/pip/_internal/cli/spinners.py b/venv/lib/python3.11/site-packages/pip/_internal/cli/spinners.py deleted file mode 100644 index cf2b976..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/cli/spinners.py +++ /dev/null @@ -1,159 +0,0 @@ -import contextlib -import itertools -import logging -import sys -import time -from typing import IO, Generator, Optional - -from pip._internal.utils.compat import WINDOWS -from pip._internal.utils.logging import get_indentation - -logger = logging.getLogger(__name__) - - -class SpinnerInterface: - def spin(self) -> None: - raise NotImplementedError() - - def finish(self, final_status: str) -> None: - raise NotImplementedError() - - -class InteractiveSpinner(SpinnerInterface): - def __init__( - self, - message: str, - file: Optional[IO[str]] = None, - spin_chars: str = "-\\|/", - # Empirically, 8 updates/second looks nice - min_update_interval_seconds: float = 0.125, - ): - self._message = message - if file is None: - file = sys.stdout - self._file = file - self._rate_limiter = RateLimiter(min_update_interval_seconds) - self._finished = False - - self._spin_cycle = itertools.cycle(spin_chars) - - self._file.write(" " * get_indentation() + self._message + " ... ") - self._width = 0 - - def _write(self, status: str) -> None: - assert not self._finished - # Erase what we wrote before by backspacing to the beginning, writing - # spaces to overwrite the old text, and then backspacing again - backup = "\b" * self._width - self._file.write(backup + " " * self._width + backup) - # Now we have a blank slate to add our status - self._file.write(status) - self._width = len(status) - self._file.flush() - self._rate_limiter.reset() - - def spin(self) -> None: - if self._finished: - return - if not self._rate_limiter.ready(): - return - self._write(next(self._spin_cycle)) - - def finish(self, final_status: str) -> None: - if self._finished: - return - self._write(final_status) - self._file.write("\n") - self._file.flush() - self._finished = True - - -# Used for dumb terminals, non-interactive installs (no tty), etc. -# We still print updates occasionally (once every 60 seconds by default) to -# act as a keep-alive for systems like Travis-CI that take lack-of-output as -# an indication that a task has frozen. -class NonInteractiveSpinner(SpinnerInterface): - def __init__(self, message: str, min_update_interval_seconds: float = 60.0) -> None: - self._message = message - self._finished = False - self._rate_limiter = RateLimiter(min_update_interval_seconds) - self._update("started") - - def _update(self, status: str) -> None: - assert not self._finished - self._rate_limiter.reset() - logger.info("%s: %s", self._message, status) - - def spin(self) -> None: - if self._finished: - return - if not self._rate_limiter.ready(): - return - self._update("still running...") - - def finish(self, final_status: str) -> None: - if self._finished: - return - self._update(f"finished with status '{final_status}'") - self._finished = True - - -class RateLimiter: - def __init__(self, min_update_interval_seconds: float) -> None: - self._min_update_interval_seconds = min_update_interval_seconds - self._last_update: float = 0 - - def ready(self) -> bool: - now = time.time() - delta = now - self._last_update - return delta >= self._min_update_interval_seconds - - def reset(self) -> None: - self._last_update = time.time() - - -@contextlib.contextmanager -def open_spinner(message: str) -> Generator[SpinnerInterface, None, None]: - # Interactive spinner goes directly to sys.stdout rather than being routed - # through the logging system, but it acts like it has level INFO, - # i.e. it's only displayed if we're at level INFO or better. - # Non-interactive spinner goes through the logging system, so it is always - # in sync with logging configuration. - if sys.stdout.isatty() and logger.getEffectiveLevel() <= logging.INFO: - spinner: SpinnerInterface = InteractiveSpinner(message) - else: - spinner = NonInteractiveSpinner(message) - try: - with hidden_cursor(sys.stdout): - yield spinner - except KeyboardInterrupt: - spinner.finish("canceled") - raise - except Exception: - spinner.finish("error") - raise - else: - spinner.finish("done") - - -HIDE_CURSOR = "\x1b[?25l" -SHOW_CURSOR = "\x1b[?25h" - - -@contextlib.contextmanager -def hidden_cursor(file: IO[str]) -> Generator[None, None, None]: - # The Windows terminal does not support the hide/show cursor ANSI codes, - # even via colorama. So don't even try. - if WINDOWS: - yield - # We don't want to clutter the output with control characters if we're - # writing to a file, or if the user is running with --quiet. - # See https://github.com/pypa/pip/issues/3418 - elif not file.isatty() or logger.getEffectiveLevel() > logging.INFO: - yield - else: - file.write(HIDE_CURSOR) - try: - yield - finally: - file.write(SHOW_CURSOR) diff --git a/venv/lib/python3.11/site-packages/pip/_internal/cli/status_codes.py b/venv/lib/python3.11/site-packages/pip/_internal/cli/status_codes.py deleted file mode 100644 index 5e29502..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/cli/status_codes.py +++ /dev/null @@ -1,6 +0,0 @@ -SUCCESS = 0 -ERROR = 1 -UNKNOWN_ERROR = 2 -VIRTUALENV_NOT_FOUND = 3 -PREVIOUS_BUILD_DIR_ERROR = 4 -NO_MATCHES_FOUND = 23 diff --git a/venv/lib/python3.11/site-packages/pip/_internal/commands/__init__.py b/venv/lib/python3.11/site-packages/pip/_internal/commands/__init__.py deleted file mode 100644 index 858a410..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/commands/__init__.py +++ /dev/null @@ -1,132 +0,0 @@ -""" -Package containing all pip commands -""" - -import importlib -from collections import namedtuple -from typing import Any, Dict, Optional - -from pip._internal.cli.base_command import Command - -CommandInfo = namedtuple("CommandInfo", "module_path, class_name, summary") - -# This dictionary does a bunch of heavy lifting for help output: -# - Enables avoiding additional (costly) imports for presenting `--help`. -# - The ordering matters for help display. -# -# Even though the module path starts with the same "pip._internal.commands" -# prefix, the full path makes testing easier (specifically when modifying -# `commands_dict` in test setup / teardown). -commands_dict: Dict[str, CommandInfo] = { - "install": CommandInfo( - "pip._internal.commands.install", - "InstallCommand", - "Install packages.", - ), - "download": CommandInfo( - "pip._internal.commands.download", - "DownloadCommand", - "Download packages.", - ), - "uninstall": CommandInfo( - "pip._internal.commands.uninstall", - "UninstallCommand", - "Uninstall packages.", - ), - "freeze": CommandInfo( - "pip._internal.commands.freeze", - "FreezeCommand", - "Output installed packages in requirements format.", - ), - "inspect": CommandInfo( - "pip._internal.commands.inspect", - "InspectCommand", - "Inspect the python environment.", - ), - "list": CommandInfo( - "pip._internal.commands.list", - "ListCommand", - "List installed packages.", - ), - "show": CommandInfo( - "pip._internal.commands.show", - "ShowCommand", - "Show information about installed packages.", - ), - "check": CommandInfo( - "pip._internal.commands.check", - "CheckCommand", - "Verify installed packages have compatible dependencies.", - ), - "config": CommandInfo( - "pip._internal.commands.configuration", - "ConfigurationCommand", - "Manage local and global configuration.", - ), - "search": CommandInfo( - "pip._internal.commands.search", - "SearchCommand", - "Search PyPI for packages.", - ), - "cache": CommandInfo( - "pip._internal.commands.cache", - "CacheCommand", - "Inspect and manage pip's wheel cache.", - ), - "index": CommandInfo( - "pip._internal.commands.index", - "IndexCommand", - "Inspect information available from package indexes.", - ), - "wheel": CommandInfo( - "pip._internal.commands.wheel", - "WheelCommand", - "Build wheels from your requirements.", - ), - "hash": CommandInfo( - "pip._internal.commands.hash", - "HashCommand", - "Compute hashes of package archives.", - ), - "completion": CommandInfo( - "pip._internal.commands.completion", - "CompletionCommand", - "A helper command used for command completion.", - ), - "debug": CommandInfo( - "pip._internal.commands.debug", - "DebugCommand", - "Show information useful for debugging.", - ), - "help": CommandInfo( - "pip._internal.commands.help", - "HelpCommand", - "Show help for commands.", - ), -} - - -def create_command(name: str, **kwargs: Any) -> Command: - """ - Create an instance of the Command class with the given name. - """ - module_path, class_name, summary = commands_dict[name] - module = importlib.import_module(module_path) - command_class = getattr(module, class_name) - command = command_class(name=name, summary=summary, **kwargs) - - return command - - -def get_similar_commands(name: str) -> Optional[str]: - """Command name auto-correct.""" - from difflib import get_close_matches - - name = name.lower() - - close_commands = get_close_matches(name, commands_dict.keys()) - - if close_commands: - return close_commands[0] - else: - return None diff --git a/venv/lib/python3.11/site-packages/pip/_internal/commands/__pycache__/__init__.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/commands/__pycache__/__init__.cpython-311.pyc deleted file mode 100644 index 6b3c948..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/commands/__pycache__/__init__.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/commands/__pycache__/cache.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/commands/__pycache__/cache.cpython-311.pyc deleted file mode 100644 index 6ec85d5..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/commands/__pycache__/cache.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/commands/__pycache__/check.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/commands/__pycache__/check.cpython-311.pyc deleted file mode 100644 index e50b209..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/commands/__pycache__/check.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/commands/__pycache__/completion.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/commands/__pycache__/completion.cpython-311.pyc deleted file mode 100644 index 10bd21c..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/commands/__pycache__/completion.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/commands/__pycache__/configuration.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/commands/__pycache__/configuration.cpython-311.pyc deleted file mode 100644 index 67999fe..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/commands/__pycache__/configuration.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/commands/__pycache__/debug.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/commands/__pycache__/debug.cpython-311.pyc deleted file mode 100644 index 2f639f1..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/commands/__pycache__/debug.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/commands/__pycache__/download.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/commands/__pycache__/download.cpython-311.pyc deleted file mode 100644 index 5a6f40f..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/commands/__pycache__/download.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/commands/__pycache__/freeze.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/commands/__pycache__/freeze.cpython-311.pyc deleted file mode 100644 index 740623f..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/commands/__pycache__/freeze.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/commands/__pycache__/hash.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/commands/__pycache__/hash.cpython-311.pyc deleted file mode 100644 index 587ca60..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/commands/__pycache__/hash.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/commands/__pycache__/help.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/commands/__pycache__/help.cpython-311.pyc deleted file mode 100644 index 5f4d8c7..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/commands/__pycache__/help.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/commands/__pycache__/index.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/commands/__pycache__/index.cpython-311.pyc deleted file mode 100644 index 2402894..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/commands/__pycache__/index.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/commands/__pycache__/inspect.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/commands/__pycache__/inspect.cpython-311.pyc deleted file mode 100644 index 8cff10a..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/commands/__pycache__/inspect.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/commands/__pycache__/install.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/commands/__pycache__/install.cpython-311.pyc deleted file mode 100644 index fbaade8..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/commands/__pycache__/install.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/commands/__pycache__/list.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/commands/__pycache__/list.cpython-311.pyc deleted file mode 100644 index 45542bb..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/commands/__pycache__/list.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/commands/__pycache__/search.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/commands/__pycache__/search.cpython-311.pyc deleted file mode 100644 index e05be93..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/commands/__pycache__/search.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/commands/__pycache__/show.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/commands/__pycache__/show.cpython-311.pyc deleted file mode 100644 index 24eaa1d..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/commands/__pycache__/show.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/commands/__pycache__/uninstall.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/commands/__pycache__/uninstall.cpython-311.pyc deleted file mode 100644 index 4553b58..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/commands/__pycache__/uninstall.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/commands/__pycache__/wheel.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/commands/__pycache__/wheel.cpython-311.pyc deleted file mode 100644 index 87bc6d8..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/commands/__pycache__/wheel.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/commands/cache.py b/venv/lib/python3.11/site-packages/pip/_internal/commands/cache.py deleted file mode 100644 index 1f3b5fe..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/commands/cache.py +++ /dev/null @@ -1,225 +0,0 @@ -import os -import textwrap -from optparse import Values -from typing import Any, List - -from pip._internal.cli.base_command import Command -from pip._internal.cli.status_codes import ERROR, SUCCESS -from pip._internal.exceptions import CommandError, PipError -from pip._internal.utils import filesystem -from pip._internal.utils.logging import getLogger - -logger = getLogger(__name__) - - -class CacheCommand(Command): - """ - Inspect and manage pip's wheel cache. - - Subcommands: - - - dir: Show the cache directory. - - info: Show information about the cache. - - list: List filenames of packages stored in the cache. - - remove: Remove one or more package from the cache. - - purge: Remove all items from the cache. - - ```` can be a glob expression or a package name. - """ - - ignore_require_venv = True - usage = """ - %prog dir - %prog info - %prog list [] [--format=[human, abspath]] - %prog remove - %prog purge - """ - - def add_options(self) -> None: - self.cmd_opts.add_option( - "--format", - action="store", - dest="list_format", - default="human", - choices=("human", "abspath"), - help="Select the output format among: human (default) or abspath", - ) - - self.parser.insert_option_group(0, self.cmd_opts) - - def run(self, options: Values, args: List[str]) -> int: - handlers = { - "dir": self.get_cache_dir, - "info": self.get_cache_info, - "list": self.list_cache_items, - "remove": self.remove_cache_items, - "purge": self.purge_cache, - } - - if not options.cache_dir: - logger.error("pip cache commands can not function since cache is disabled.") - return ERROR - - # Determine action - if not args or args[0] not in handlers: - logger.error( - "Need an action (%s) to perform.", - ", ".join(sorted(handlers)), - ) - return ERROR - - action = args[0] - - # Error handling happens here, not in the action-handlers. - try: - handlers[action](options, args[1:]) - except PipError as e: - logger.error(e.args[0]) - return ERROR - - return SUCCESS - - def get_cache_dir(self, options: Values, args: List[Any]) -> None: - if args: - raise CommandError("Too many arguments") - - logger.info(options.cache_dir) - - def get_cache_info(self, options: Values, args: List[Any]) -> None: - if args: - raise CommandError("Too many arguments") - - num_http_files = len(self._find_http_files(options)) - num_packages = len(self._find_wheels(options, "*")) - - http_cache_location = self._cache_dir(options, "http-v2") - old_http_cache_location = self._cache_dir(options, "http") - wheels_cache_location = self._cache_dir(options, "wheels") - http_cache_size = filesystem.format_size( - filesystem.directory_size(http_cache_location) - + filesystem.directory_size(old_http_cache_location) - ) - wheels_cache_size = filesystem.format_directory_size(wheels_cache_location) - - message = ( - textwrap.dedent( - """ - Package index page cache location (pip v23.3+): {http_cache_location} - Package index page cache location (older pips): {old_http_cache_location} - Package index page cache size: {http_cache_size} - Number of HTTP files: {num_http_files} - Locally built wheels location: {wheels_cache_location} - Locally built wheels size: {wheels_cache_size} - Number of locally built wheels: {package_count} - """ # noqa: E501 - ) - .format( - http_cache_location=http_cache_location, - old_http_cache_location=old_http_cache_location, - http_cache_size=http_cache_size, - num_http_files=num_http_files, - wheels_cache_location=wheels_cache_location, - package_count=num_packages, - wheels_cache_size=wheels_cache_size, - ) - .strip() - ) - - logger.info(message) - - def list_cache_items(self, options: Values, args: List[Any]) -> None: - if len(args) > 1: - raise CommandError("Too many arguments") - - if args: - pattern = args[0] - else: - pattern = "*" - - files = self._find_wheels(options, pattern) - if options.list_format == "human": - self.format_for_human(files) - else: - self.format_for_abspath(files) - - def format_for_human(self, files: List[str]) -> None: - if not files: - logger.info("No locally built wheels cached.") - return - - results = [] - for filename in files: - wheel = os.path.basename(filename) - size = filesystem.format_file_size(filename) - results.append(f" - {wheel} ({size})") - logger.info("Cache contents:\n") - logger.info("\n".join(sorted(results))) - - def format_for_abspath(self, files: List[str]) -> None: - if files: - logger.info("\n".join(sorted(files))) - - def remove_cache_items(self, options: Values, args: List[Any]) -> None: - if len(args) > 1: - raise CommandError("Too many arguments") - - if not args: - raise CommandError("Please provide a pattern") - - files = self._find_wheels(options, args[0]) - - no_matching_msg = "No matching packages" - if args[0] == "*": - # Only fetch http files if no specific pattern given - files += self._find_http_files(options) - else: - # Add the pattern to the log message - no_matching_msg += ' for pattern "{}"'.format(args[0]) - - if not files: - logger.warning(no_matching_msg) - - for filename in files: - os.unlink(filename) - logger.verbose("Removed %s", filename) - logger.info("Files removed: %s", len(files)) - - def purge_cache(self, options: Values, args: List[Any]) -> None: - if args: - raise CommandError("Too many arguments") - - return self.remove_cache_items(options, ["*"]) - - def _cache_dir(self, options: Values, subdir: str) -> str: - return os.path.join(options.cache_dir, subdir) - - def _find_http_files(self, options: Values) -> List[str]: - old_http_dir = self._cache_dir(options, "http") - new_http_dir = self._cache_dir(options, "http-v2") - return filesystem.find_files(old_http_dir, "*") + filesystem.find_files( - new_http_dir, "*" - ) - - def _find_wheels(self, options: Values, pattern: str) -> List[str]: - wheel_dir = self._cache_dir(options, "wheels") - - # The wheel filename format, as specified in PEP 427, is: - # {distribution}-{version}(-{build})?-{python}-{abi}-{platform}.whl - # - # Additionally, non-alphanumeric values in the distribution are - # normalized to underscores (_), meaning hyphens can never occur - # before `-{version}`. - # - # Given that information: - # - If the pattern we're given contains a hyphen (-), the user is - # providing at least the version. Thus, we can just append `*.whl` - # to match the rest of it. - # - If the pattern we're given doesn't contain a hyphen (-), the - # user is only providing the name. Thus, we append `-*.whl` to - # match the hyphen before the version, followed by anything else. - # - # PEP 427: https://www.python.org/dev/peps/pep-0427/ - pattern = pattern + ("*.whl" if "-" in pattern else "-*.whl") - - return filesystem.find_files(wheel_dir, pattern) diff --git a/venv/lib/python3.11/site-packages/pip/_internal/commands/check.py b/venv/lib/python3.11/site-packages/pip/_internal/commands/check.py deleted file mode 100644 index 5efd0a3..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/commands/check.py +++ /dev/null @@ -1,54 +0,0 @@ -import logging -from optparse import Values -from typing import List - -from pip._internal.cli.base_command import Command -from pip._internal.cli.status_codes import ERROR, SUCCESS -from pip._internal.operations.check import ( - check_package_set, - create_package_set_from_installed, - warn_legacy_versions_and_specifiers, -) -from pip._internal.utils.misc import write_output - -logger = logging.getLogger(__name__) - - -class CheckCommand(Command): - """Verify installed packages have compatible dependencies.""" - - usage = """ - %prog [options]""" - - def run(self, options: Values, args: List[str]) -> int: - package_set, parsing_probs = create_package_set_from_installed() - warn_legacy_versions_and_specifiers(package_set) - missing, conflicting = check_package_set(package_set) - - for project_name in missing: - version = package_set[project_name].version - for dependency in missing[project_name]: - write_output( - "%s %s requires %s, which is not installed.", - project_name, - version, - dependency[0], - ) - - for project_name in conflicting: - version = package_set[project_name].version - for dep_name, dep_version, req in conflicting[project_name]: - write_output( - "%s %s has requirement %s, but you have %s %s.", - project_name, - version, - req, - dep_name, - dep_version, - ) - - if missing or conflicting or parsing_probs: - return ERROR - else: - write_output("No broken requirements found.") - return SUCCESS diff --git a/venv/lib/python3.11/site-packages/pip/_internal/commands/completion.py b/venv/lib/python3.11/site-packages/pip/_internal/commands/completion.py deleted file mode 100644 index 9e89e27..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/commands/completion.py +++ /dev/null @@ -1,130 +0,0 @@ -import sys -import textwrap -from optparse import Values -from typing import List - -from pip._internal.cli.base_command import Command -from pip._internal.cli.status_codes import SUCCESS -from pip._internal.utils.misc import get_prog - -BASE_COMPLETION = """ -# pip {shell} completion start{script}# pip {shell} completion end -""" - -COMPLETION_SCRIPTS = { - "bash": """ - _pip_completion() - {{ - COMPREPLY=( $( COMP_WORDS="${{COMP_WORDS[*]}}" \\ - COMP_CWORD=$COMP_CWORD \\ - PIP_AUTO_COMPLETE=1 $1 2>/dev/null ) ) - }} - complete -o default -F _pip_completion {prog} - """, - "zsh": """ - #compdef -P pip[0-9.]# - __pip() {{ - compadd $( COMP_WORDS="$words[*]" \\ - COMP_CWORD=$((CURRENT-1)) \\ - PIP_AUTO_COMPLETE=1 $words[1] 2>/dev/null ) - }} - if [[ $zsh_eval_context[-1] == loadautofunc ]]; then - # autoload from fpath, call function directly - __pip "$@" - else - # eval/source/. command, register function for later - compdef __pip -P 'pip[0-9.]#' - fi - """, - "fish": """ - function __fish_complete_pip - set -lx COMP_WORDS (commandline -o) "" - set -lx COMP_CWORD ( \\ - math (contains -i -- (commandline -t) $COMP_WORDS)-1 \\ - ) - set -lx PIP_AUTO_COMPLETE 1 - string split \\ -- (eval $COMP_WORDS[1]) - end - complete -fa "(__fish_complete_pip)" -c {prog} - """, - "powershell": """ - if ((Test-Path Function:\\TabExpansion) -and -not ` - (Test-Path Function:\\_pip_completeBackup)) {{ - Rename-Item Function:\\TabExpansion _pip_completeBackup - }} - function TabExpansion($line, $lastWord) {{ - $lastBlock = [regex]::Split($line, '[|;]')[-1].TrimStart() - if ($lastBlock.StartsWith("{prog} ")) {{ - $Env:COMP_WORDS=$lastBlock - $Env:COMP_CWORD=$lastBlock.Split().Length - 1 - $Env:PIP_AUTO_COMPLETE=1 - (& {prog}).Split() - Remove-Item Env:COMP_WORDS - Remove-Item Env:COMP_CWORD - Remove-Item Env:PIP_AUTO_COMPLETE - }} - elseif (Test-Path Function:\\_pip_completeBackup) {{ - # Fall back on existing tab expansion - _pip_completeBackup $line $lastWord - }} - }} - """, -} - - -class CompletionCommand(Command): - """A helper command to be used for command completion.""" - - ignore_require_venv = True - - def add_options(self) -> None: - self.cmd_opts.add_option( - "--bash", - "-b", - action="store_const", - const="bash", - dest="shell", - help="Emit completion code for bash", - ) - self.cmd_opts.add_option( - "--zsh", - "-z", - action="store_const", - const="zsh", - dest="shell", - help="Emit completion code for zsh", - ) - self.cmd_opts.add_option( - "--fish", - "-f", - action="store_const", - const="fish", - dest="shell", - help="Emit completion code for fish", - ) - self.cmd_opts.add_option( - "--powershell", - "-p", - action="store_const", - const="powershell", - dest="shell", - help="Emit completion code for powershell", - ) - - self.parser.insert_option_group(0, self.cmd_opts) - - def run(self, options: Values, args: List[str]) -> int: - """Prints the completion code of the given shell""" - shells = COMPLETION_SCRIPTS.keys() - shell_options = ["--" + shell for shell in sorted(shells)] - if options.shell in shells: - script = textwrap.dedent( - COMPLETION_SCRIPTS.get(options.shell, "").format(prog=get_prog()) - ) - print(BASE_COMPLETION.format(script=script, shell=options.shell)) - return SUCCESS - else: - sys.stderr.write( - "ERROR: You must pass {}\n".format(" or ".join(shell_options)) - ) - return SUCCESS diff --git a/venv/lib/python3.11/site-packages/pip/_internal/commands/configuration.py b/venv/lib/python3.11/site-packages/pip/_internal/commands/configuration.py deleted file mode 100644 index 84b134e..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/commands/configuration.py +++ /dev/null @@ -1,282 +0,0 @@ -import logging -import os -import subprocess -from optparse import Values -from typing import Any, List, Optional - -from pip._internal.cli.base_command import Command -from pip._internal.cli.status_codes import ERROR, SUCCESS -from pip._internal.configuration import ( - Configuration, - Kind, - get_configuration_files, - kinds, -) -from pip._internal.exceptions import PipError -from pip._internal.utils.logging import indent_log -from pip._internal.utils.misc import get_prog, write_output - -logger = logging.getLogger(__name__) - - -class ConfigurationCommand(Command): - """ - Manage local and global configuration. - - Subcommands: - - - list: List the active configuration (or from the file specified) - - edit: Edit the configuration file in an editor - - get: Get the value associated with command.option - - set: Set the command.option=value - - unset: Unset the value associated with command.option - - debug: List the configuration files and values defined under them - - Configuration keys should be dot separated command and option name, - with the special prefix "global" affecting any command. For example, - "pip config set global.index-url https://example.org/" would configure - the index url for all commands, but "pip config set download.timeout 10" - would configure a 10 second timeout only for "pip download" commands. - - If none of --user, --global and --site are passed, a virtual - environment configuration file is used if one is active and the file - exists. Otherwise, all modifications happen to the user file by - default. - """ - - ignore_require_venv = True - usage = """ - %prog [] list - %prog [] [--editor ] edit - - %prog [] get command.option - %prog [] set command.option value - %prog [] unset command.option - %prog [] debug - """ - - def add_options(self) -> None: - self.cmd_opts.add_option( - "--editor", - dest="editor", - action="store", - default=None, - help=( - "Editor to use to edit the file. Uses VISUAL or EDITOR " - "environment variables if not provided." - ), - ) - - self.cmd_opts.add_option( - "--global", - dest="global_file", - action="store_true", - default=False, - help="Use the system-wide configuration file only", - ) - - self.cmd_opts.add_option( - "--user", - dest="user_file", - action="store_true", - default=False, - help="Use the user configuration file only", - ) - - self.cmd_opts.add_option( - "--site", - dest="site_file", - action="store_true", - default=False, - help="Use the current environment configuration file only", - ) - - self.parser.insert_option_group(0, self.cmd_opts) - - def run(self, options: Values, args: List[str]) -> int: - handlers = { - "list": self.list_values, - "edit": self.open_in_editor, - "get": self.get_name, - "set": self.set_name_value, - "unset": self.unset_name, - "debug": self.list_config_values, - } - - # Determine action - if not args or args[0] not in handlers: - logger.error( - "Need an action (%s) to perform.", - ", ".join(sorted(handlers)), - ) - return ERROR - - action = args[0] - - # Determine which configuration files are to be loaded - # Depends on whether the command is modifying. - try: - load_only = self._determine_file( - options, need_value=(action in ["get", "set", "unset", "edit"]) - ) - except PipError as e: - logger.error(e.args[0]) - return ERROR - - # Load a new configuration - self.configuration = Configuration( - isolated=options.isolated_mode, load_only=load_only - ) - self.configuration.load() - - # Error handling happens here, not in the action-handlers. - try: - handlers[action](options, args[1:]) - except PipError as e: - logger.error(e.args[0]) - return ERROR - - return SUCCESS - - def _determine_file(self, options: Values, need_value: bool) -> Optional[Kind]: - file_options = [ - key - for key, value in ( - (kinds.USER, options.user_file), - (kinds.GLOBAL, options.global_file), - (kinds.SITE, options.site_file), - ) - if value - ] - - if not file_options: - if not need_value: - return None - # Default to user, unless there's a site file. - elif any( - os.path.exists(site_config_file) - for site_config_file in get_configuration_files()[kinds.SITE] - ): - return kinds.SITE - else: - return kinds.USER - elif len(file_options) == 1: - return file_options[0] - - raise PipError( - "Need exactly one file to operate upon " - "(--user, --site, --global) to perform." - ) - - def list_values(self, options: Values, args: List[str]) -> None: - self._get_n_args(args, "list", n=0) - - for key, value in sorted(self.configuration.items()): - write_output("%s=%r", key, value) - - def get_name(self, options: Values, args: List[str]) -> None: - key = self._get_n_args(args, "get [name]", n=1) - value = self.configuration.get_value(key) - - write_output("%s", value) - - def set_name_value(self, options: Values, args: List[str]) -> None: - key, value = self._get_n_args(args, "set [name] [value]", n=2) - self.configuration.set_value(key, value) - - self._save_configuration() - - def unset_name(self, options: Values, args: List[str]) -> None: - key = self._get_n_args(args, "unset [name]", n=1) - self.configuration.unset_value(key) - - self._save_configuration() - - def list_config_values(self, options: Values, args: List[str]) -> None: - """List config key-value pairs across different config files""" - self._get_n_args(args, "debug", n=0) - - self.print_env_var_values() - # Iterate over config files and print if they exist, and the - # key-value pairs present in them if they do - for variant, files in sorted(self.configuration.iter_config_files()): - write_output("%s:", variant) - for fname in files: - with indent_log(): - file_exists = os.path.exists(fname) - write_output("%s, exists: %r", fname, file_exists) - if file_exists: - self.print_config_file_values(variant) - - def print_config_file_values(self, variant: Kind) -> None: - """Get key-value pairs from the file of a variant""" - for name, value in self.configuration.get_values_in_config(variant).items(): - with indent_log(): - write_output("%s: %s", name, value) - - def print_env_var_values(self) -> None: - """Get key-values pairs present as environment variables""" - write_output("%s:", "env_var") - with indent_log(): - for key, value in sorted(self.configuration.get_environ_vars()): - env_var = f"PIP_{key.upper()}" - write_output("%s=%r", env_var, value) - - def open_in_editor(self, options: Values, args: List[str]) -> None: - editor = self._determine_editor(options) - - fname = self.configuration.get_file_to_edit() - if fname is None: - raise PipError("Could not determine appropriate file.") - elif '"' in fname: - # This shouldn't happen, unless we see a username like that. - # If that happens, we'd appreciate a pull request fixing this. - raise PipError( - f'Can not open an editor for a file name containing "\n{fname}' - ) - - try: - subprocess.check_call(f'{editor} "{fname}"', shell=True) - except FileNotFoundError as e: - if not e.filename: - e.filename = editor - raise - except subprocess.CalledProcessError as e: - raise PipError( - "Editor Subprocess exited with exit code {}".format(e.returncode) - ) - - def _get_n_args(self, args: List[str], example: str, n: int) -> Any: - """Helper to make sure the command got the right number of arguments""" - if len(args) != n: - msg = ( - "Got unexpected number of arguments, expected {}. " - '(example: "{} config {}")' - ).format(n, get_prog(), example) - raise PipError(msg) - - if n == 1: - return args[0] - else: - return args - - def _save_configuration(self) -> None: - # We successfully ran a modifying command. Need to save the - # configuration. - try: - self.configuration.save() - except Exception: - logger.exception( - "Unable to save configuration. Please report this as a bug." - ) - raise PipError("Internal Error.") - - def _determine_editor(self, options: Values) -> str: - if options.editor is not None: - return options.editor - elif "VISUAL" in os.environ: - return os.environ["VISUAL"] - elif "EDITOR" in os.environ: - return os.environ["EDITOR"] - else: - raise PipError("Could not determine editor to use.") diff --git a/venv/lib/python3.11/site-packages/pip/_internal/commands/debug.py b/venv/lib/python3.11/site-packages/pip/_internal/commands/debug.py deleted file mode 100644 index 5dc91bf..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/commands/debug.py +++ /dev/null @@ -1,203 +0,0 @@ -import importlib.resources -import locale -import logging -import os -import sys -from optparse import Values -from types import ModuleType -from typing import Any, Dict, List, Optional - -import pip._vendor -from pip._vendor.certifi import where -from pip._vendor.packaging.version import parse as parse_version - -from pip._internal.cli import cmdoptions -from pip._internal.cli.base_command import Command -from pip._internal.cli.cmdoptions import make_target_python -from pip._internal.cli.status_codes import SUCCESS -from pip._internal.configuration import Configuration -from pip._internal.metadata import get_environment -from pip._internal.utils.logging import indent_log -from pip._internal.utils.misc import get_pip_version - -logger = logging.getLogger(__name__) - - -def show_value(name: str, value: Any) -> None: - logger.info("%s: %s", name, value) - - -def show_sys_implementation() -> None: - logger.info("sys.implementation:") - implementation_name = sys.implementation.name - with indent_log(): - show_value("name", implementation_name) - - -def create_vendor_txt_map() -> Dict[str, str]: - with importlib.resources.open_text("pip._vendor", "vendor.txt") as f: - # Purge non version specifying lines. - # Also, remove any space prefix or suffixes (including comments). - lines = [ - line.strip().split(" ", 1)[0] for line in f.readlines() if "==" in line - ] - - # Transform into "module" -> version dict. - return dict(line.split("==", 1) for line in lines) - - -def get_module_from_module_name(module_name: str) -> Optional[ModuleType]: - # Module name can be uppercase in vendor.txt for some reason... - module_name = module_name.lower().replace("-", "_") - # PATCH: setuptools is actually only pkg_resources. - if module_name == "setuptools": - module_name = "pkg_resources" - - try: - __import__(f"pip._vendor.{module_name}", globals(), locals(), level=0) - return getattr(pip._vendor, module_name) - except ImportError: - # We allow 'truststore' to fail to import due - # to being unavailable on Python 3.9 and earlier. - if module_name == "truststore" and sys.version_info < (3, 10): - return None - raise - - -def get_vendor_version_from_module(module_name: str) -> Optional[str]: - module = get_module_from_module_name(module_name) - version = getattr(module, "__version__", None) - - if module and not version: - # Try to find version in debundled module info. - assert module.__file__ is not None - env = get_environment([os.path.dirname(module.__file__)]) - dist = env.get_distribution(module_name) - if dist: - version = str(dist.version) - - return version - - -def show_actual_vendor_versions(vendor_txt_versions: Dict[str, str]) -> None: - """Log the actual version and print extra info if there is - a conflict or if the actual version could not be imported. - """ - for module_name, expected_version in vendor_txt_versions.items(): - extra_message = "" - actual_version = get_vendor_version_from_module(module_name) - if not actual_version: - extra_message = ( - " (Unable to locate actual module version, using" - " vendor.txt specified version)" - ) - actual_version = expected_version - elif parse_version(actual_version) != parse_version(expected_version): - extra_message = ( - " (CONFLICT: vendor.txt suggests version should" - " be {})".format(expected_version) - ) - logger.info("%s==%s%s", module_name, actual_version, extra_message) - - -def show_vendor_versions() -> None: - logger.info("vendored library versions:") - - vendor_txt_versions = create_vendor_txt_map() - with indent_log(): - show_actual_vendor_versions(vendor_txt_versions) - - -def show_tags(options: Values) -> None: - tag_limit = 10 - - target_python = make_target_python(options) - tags = target_python.get_sorted_tags() - - # Display the target options that were explicitly provided. - formatted_target = target_python.format_given() - suffix = "" - if formatted_target: - suffix = f" (target: {formatted_target})" - - msg = "Compatible tags: {}{}".format(len(tags), suffix) - logger.info(msg) - - if options.verbose < 1 and len(tags) > tag_limit: - tags_limited = True - tags = tags[:tag_limit] - else: - tags_limited = False - - with indent_log(): - for tag in tags: - logger.info(str(tag)) - - if tags_limited: - msg = ( - "...\n[First {tag_limit} tags shown. Pass --verbose to show all.]" - ).format(tag_limit=tag_limit) - logger.info(msg) - - -def ca_bundle_info(config: Configuration) -> str: - levels = {key.split(".", 1)[0] for key, _ in config.items()} - if not levels: - return "Not specified" - - levels_that_override_global = ["install", "wheel", "download"] - global_overriding_level = [ - level for level in levels if level in levels_that_override_global - ] - if not global_overriding_level: - return "global" - - if "global" in levels: - levels.remove("global") - return ", ".join(levels) - - -class DebugCommand(Command): - """ - Display debug information. - """ - - usage = """ - %prog """ - ignore_require_venv = True - - def add_options(self) -> None: - cmdoptions.add_target_python_options(self.cmd_opts) - self.parser.insert_option_group(0, self.cmd_opts) - self.parser.config.load() - - def run(self, options: Values, args: List[str]) -> int: - logger.warning( - "This command is only meant for debugging. " - "Do not use this with automation for parsing and getting these " - "details, since the output and options of this command may " - "change without notice." - ) - show_value("pip version", get_pip_version()) - show_value("sys.version", sys.version) - show_value("sys.executable", sys.executable) - show_value("sys.getdefaultencoding", sys.getdefaultencoding()) - show_value("sys.getfilesystemencoding", sys.getfilesystemencoding()) - show_value( - "locale.getpreferredencoding", - locale.getpreferredencoding(), - ) - show_value("sys.platform", sys.platform) - show_sys_implementation() - - show_value("'cert' config value", ca_bundle_info(self.parser.config)) - show_value("REQUESTS_CA_BUNDLE", os.environ.get("REQUESTS_CA_BUNDLE")) - show_value("CURL_CA_BUNDLE", os.environ.get("CURL_CA_BUNDLE")) - show_value("pip._vendor.certifi.where()", where()) - show_value("pip._vendor.DEBUNDLED", pip._vendor.DEBUNDLED) - - show_vendor_versions() - - show_tags(options) - - return SUCCESS diff --git a/venv/lib/python3.11/site-packages/pip/_internal/commands/download.py b/venv/lib/python3.11/site-packages/pip/_internal/commands/download.py deleted file mode 100644 index 54247a7..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/commands/download.py +++ /dev/null @@ -1,147 +0,0 @@ -import logging -import os -from optparse import Values -from typing import List - -from pip._internal.cli import cmdoptions -from pip._internal.cli.cmdoptions import make_target_python -from pip._internal.cli.req_command import RequirementCommand, with_cleanup -from pip._internal.cli.status_codes import SUCCESS -from pip._internal.operations.build.build_tracker import get_build_tracker -from pip._internal.req.req_install import check_legacy_setup_py_options -from pip._internal.utils.misc import ensure_dir, normalize_path, write_output -from pip._internal.utils.temp_dir import TempDirectory - -logger = logging.getLogger(__name__) - - -class DownloadCommand(RequirementCommand): - """ - Download packages from: - - - PyPI (and other indexes) using requirement specifiers. - - VCS project urls. - - Local project directories. - - Local or remote source archives. - - pip also supports downloading from "requirements files", which provide - an easy way to specify a whole environment to be downloaded. - """ - - usage = """ - %prog [options] [package-index-options] ... - %prog [options] -r [package-index-options] ... - %prog [options] ... - %prog [options] ... - %prog [options] ...""" - - def add_options(self) -> None: - self.cmd_opts.add_option(cmdoptions.constraints()) - self.cmd_opts.add_option(cmdoptions.requirements()) - self.cmd_opts.add_option(cmdoptions.no_deps()) - self.cmd_opts.add_option(cmdoptions.global_options()) - self.cmd_opts.add_option(cmdoptions.no_binary()) - self.cmd_opts.add_option(cmdoptions.only_binary()) - self.cmd_opts.add_option(cmdoptions.prefer_binary()) - self.cmd_opts.add_option(cmdoptions.src()) - self.cmd_opts.add_option(cmdoptions.pre()) - self.cmd_opts.add_option(cmdoptions.require_hashes()) - self.cmd_opts.add_option(cmdoptions.progress_bar()) - self.cmd_opts.add_option(cmdoptions.no_build_isolation()) - self.cmd_opts.add_option(cmdoptions.use_pep517()) - self.cmd_opts.add_option(cmdoptions.no_use_pep517()) - self.cmd_opts.add_option(cmdoptions.check_build_deps()) - self.cmd_opts.add_option(cmdoptions.ignore_requires_python()) - - self.cmd_opts.add_option( - "-d", - "--dest", - "--destination-dir", - "--destination-directory", - dest="download_dir", - metavar="dir", - default=os.curdir, - help="Download packages into .", - ) - - cmdoptions.add_target_python_options(self.cmd_opts) - - index_opts = cmdoptions.make_option_group( - cmdoptions.index_group, - self.parser, - ) - - self.parser.insert_option_group(0, index_opts) - self.parser.insert_option_group(0, self.cmd_opts) - - @with_cleanup - def run(self, options: Values, args: List[str]) -> int: - options.ignore_installed = True - # editable doesn't really make sense for `pip download`, but the bowels - # of the RequirementSet code require that property. - options.editables = [] - - cmdoptions.check_dist_restriction(options) - - options.download_dir = normalize_path(options.download_dir) - ensure_dir(options.download_dir) - - session = self.get_default_session(options) - - target_python = make_target_python(options) - finder = self._build_package_finder( - options=options, - session=session, - target_python=target_python, - ignore_requires_python=options.ignore_requires_python, - ) - - build_tracker = self.enter_context(get_build_tracker()) - - directory = TempDirectory( - delete=not options.no_clean, - kind="download", - globally_managed=True, - ) - - reqs = self.get_requirements(args, options, finder, session) - check_legacy_setup_py_options(options, reqs) - - preparer = self.make_requirement_preparer( - temp_build_dir=directory, - options=options, - build_tracker=build_tracker, - session=session, - finder=finder, - download_dir=options.download_dir, - use_user_site=False, - verbosity=self.verbosity, - ) - - resolver = self.make_resolver( - preparer=preparer, - finder=finder, - options=options, - ignore_requires_python=options.ignore_requires_python, - use_pep517=options.use_pep517, - py_version_info=options.python_version, - ) - - self.trace_basic_info(finder) - - requirement_set = resolver.resolve(reqs, check_supported_wheels=True) - - downloaded: List[str] = [] - for req in requirement_set.requirements.values(): - if req.satisfied_by is None: - assert req.name is not None - preparer.save_linked_requirement(req) - downloaded.append(req.name) - - preparer.prepare_linked_requirements_more(requirement_set.requirements.values()) - requirement_set.warn_legacy_versions_and_specifiers() - - if downloaded: - write_output("Successfully downloaded %s", " ".join(downloaded)) - - return SUCCESS diff --git a/venv/lib/python3.11/site-packages/pip/_internal/commands/freeze.py b/venv/lib/python3.11/site-packages/pip/_internal/commands/freeze.py deleted file mode 100644 index fd9d88a..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/commands/freeze.py +++ /dev/null @@ -1,108 +0,0 @@ -import sys -from optparse import Values -from typing import AbstractSet, List - -from pip._internal.cli import cmdoptions -from pip._internal.cli.base_command import Command -from pip._internal.cli.status_codes import SUCCESS -from pip._internal.operations.freeze import freeze -from pip._internal.utils.compat import stdlib_pkgs - - -def _should_suppress_build_backends() -> bool: - return sys.version_info < (3, 12) - - -def _dev_pkgs() -> AbstractSet[str]: - pkgs = {"pip"} - - if _should_suppress_build_backends(): - pkgs |= {"setuptools", "distribute", "wheel"} - - return pkgs - - -class FreezeCommand(Command): - """ - Output installed packages in requirements format. - - packages are listed in a case-insensitive sorted order. - """ - - usage = """ - %prog [options]""" - log_streams = ("ext://sys.stderr", "ext://sys.stderr") - - def add_options(self) -> None: - self.cmd_opts.add_option( - "-r", - "--requirement", - dest="requirements", - action="append", - default=[], - metavar="file", - help=( - "Use the order in the given requirements file and its " - "comments when generating output. This option can be " - "used multiple times." - ), - ) - self.cmd_opts.add_option( - "-l", - "--local", - dest="local", - action="store_true", - default=False, - help=( - "If in a virtualenv that has global access, do not output " - "globally-installed packages." - ), - ) - self.cmd_opts.add_option( - "--user", - dest="user", - action="store_true", - default=False, - help="Only output packages installed in user-site.", - ) - self.cmd_opts.add_option(cmdoptions.list_path()) - self.cmd_opts.add_option( - "--all", - dest="freeze_all", - action="store_true", - help=( - "Do not skip these packages in the output:" - " {}".format(", ".join(_dev_pkgs())) - ), - ) - self.cmd_opts.add_option( - "--exclude-editable", - dest="exclude_editable", - action="store_true", - help="Exclude editable package from output.", - ) - self.cmd_opts.add_option(cmdoptions.list_exclude()) - - self.parser.insert_option_group(0, self.cmd_opts) - - def run(self, options: Values, args: List[str]) -> int: - skip = set(stdlib_pkgs) - if not options.freeze_all: - skip.update(_dev_pkgs()) - - if options.excludes: - skip.update(options.excludes) - - cmdoptions.check_list_path_option(options) - - for line in freeze( - requirement=options.requirements, - local_only=options.local, - user_only=options.user, - paths=options.path, - isolated=options.isolated_mode, - skip=skip, - exclude_editable=options.exclude_editable, - ): - sys.stdout.write(line + "\n") - return SUCCESS diff --git a/venv/lib/python3.11/site-packages/pip/_internal/commands/hash.py b/venv/lib/python3.11/site-packages/pip/_internal/commands/hash.py deleted file mode 100644 index 042dac8..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/commands/hash.py +++ /dev/null @@ -1,59 +0,0 @@ -import hashlib -import logging -import sys -from optparse import Values -from typing import List - -from pip._internal.cli.base_command import Command -from pip._internal.cli.status_codes import ERROR, SUCCESS -from pip._internal.utils.hashes import FAVORITE_HASH, STRONG_HASHES -from pip._internal.utils.misc import read_chunks, write_output - -logger = logging.getLogger(__name__) - - -class HashCommand(Command): - """ - Compute a hash of a local package archive. - - These can be used with --hash in a requirements file to do repeatable - installs. - """ - - usage = "%prog [options] ..." - ignore_require_venv = True - - def add_options(self) -> None: - self.cmd_opts.add_option( - "-a", - "--algorithm", - dest="algorithm", - choices=STRONG_HASHES, - action="store", - default=FAVORITE_HASH, - help="The hash algorithm to use: one of {}".format( - ", ".join(STRONG_HASHES) - ), - ) - self.parser.insert_option_group(0, self.cmd_opts) - - def run(self, options: Values, args: List[str]) -> int: - if not args: - self.parser.print_usage(sys.stderr) - return ERROR - - algorithm = options.algorithm - for path in args: - write_output( - "%s:\n--hash=%s:%s", path, algorithm, _hash_of_file(path, algorithm) - ) - return SUCCESS - - -def _hash_of_file(path: str, algorithm: str) -> str: - """Return the hash digest of a file.""" - with open(path, "rb") as archive: - hash = hashlib.new(algorithm) - for chunk in read_chunks(archive): - hash.update(chunk) - return hash.hexdigest() diff --git a/venv/lib/python3.11/site-packages/pip/_internal/commands/help.py b/venv/lib/python3.11/site-packages/pip/_internal/commands/help.py deleted file mode 100644 index 6206631..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/commands/help.py +++ /dev/null @@ -1,41 +0,0 @@ -from optparse import Values -from typing import List - -from pip._internal.cli.base_command import Command -from pip._internal.cli.status_codes import SUCCESS -from pip._internal.exceptions import CommandError - - -class HelpCommand(Command): - """Show help for commands""" - - usage = """ - %prog """ - ignore_require_venv = True - - def run(self, options: Values, args: List[str]) -> int: - from pip._internal.commands import ( - commands_dict, - create_command, - get_similar_commands, - ) - - try: - # 'pip help' with no args is handled by pip.__init__.parseopt() - cmd_name = args[0] # the command we need help for - except IndexError: - return SUCCESS - - if cmd_name not in commands_dict: - guess = get_similar_commands(cmd_name) - - msg = [f'unknown command "{cmd_name}"'] - if guess: - msg.append(f'maybe you meant "{guess}"') - - raise CommandError(" - ".join(msg)) - - command = create_command(cmd_name) - command.parser.print_help() - - return SUCCESS diff --git a/venv/lib/python3.11/site-packages/pip/_internal/commands/index.py b/venv/lib/python3.11/site-packages/pip/_internal/commands/index.py deleted file mode 100644 index 7267eff..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/commands/index.py +++ /dev/null @@ -1,139 +0,0 @@ -import logging -from optparse import Values -from typing import Any, Iterable, List, Optional, Union - -from pip._vendor.packaging.version import LegacyVersion, Version - -from pip._internal.cli import cmdoptions -from pip._internal.cli.req_command import IndexGroupCommand -from pip._internal.cli.status_codes import ERROR, SUCCESS -from pip._internal.commands.search import print_dist_installation_info -from pip._internal.exceptions import CommandError, DistributionNotFound, PipError -from pip._internal.index.collector import LinkCollector -from pip._internal.index.package_finder import PackageFinder -from pip._internal.models.selection_prefs import SelectionPreferences -from pip._internal.models.target_python import TargetPython -from pip._internal.network.session import PipSession -from pip._internal.utils.misc import write_output - -logger = logging.getLogger(__name__) - - -class IndexCommand(IndexGroupCommand): - """ - Inspect information available from package indexes. - """ - - ignore_require_venv = True - usage = """ - %prog versions - """ - - def add_options(self) -> None: - cmdoptions.add_target_python_options(self.cmd_opts) - - self.cmd_opts.add_option(cmdoptions.ignore_requires_python()) - self.cmd_opts.add_option(cmdoptions.pre()) - self.cmd_opts.add_option(cmdoptions.no_binary()) - self.cmd_opts.add_option(cmdoptions.only_binary()) - - index_opts = cmdoptions.make_option_group( - cmdoptions.index_group, - self.parser, - ) - - self.parser.insert_option_group(0, index_opts) - self.parser.insert_option_group(0, self.cmd_opts) - - def run(self, options: Values, args: List[str]) -> int: - handlers = { - "versions": self.get_available_package_versions, - } - - logger.warning( - "pip index is currently an experimental command. " - "It may be removed/changed in a future release " - "without prior warning." - ) - - # Determine action - if not args or args[0] not in handlers: - logger.error( - "Need an action (%s) to perform.", - ", ".join(sorted(handlers)), - ) - return ERROR - - action = args[0] - - # Error handling happens here, not in the action-handlers. - try: - handlers[action](options, args[1:]) - except PipError as e: - logger.error(e.args[0]) - return ERROR - - return SUCCESS - - def _build_package_finder( - self, - options: Values, - session: PipSession, - target_python: Optional[TargetPython] = None, - ignore_requires_python: Optional[bool] = None, - ) -> PackageFinder: - """ - Create a package finder appropriate to the index command. - """ - link_collector = LinkCollector.create(session, options=options) - - # Pass allow_yanked=False to ignore yanked versions. - selection_prefs = SelectionPreferences( - allow_yanked=False, - allow_all_prereleases=options.pre, - ignore_requires_python=ignore_requires_python, - ) - - return PackageFinder.create( - link_collector=link_collector, - selection_prefs=selection_prefs, - target_python=target_python, - ) - - def get_available_package_versions(self, options: Values, args: List[Any]) -> None: - if len(args) != 1: - raise CommandError("You need to specify exactly one argument") - - target_python = cmdoptions.make_target_python(options) - query = args[0] - - with self._build_session(options) as session: - finder = self._build_package_finder( - options=options, - session=session, - target_python=target_python, - ignore_requires_python=options.ignore_requires_python, - ) - - versions: Iterable[Union[LegacyVersion, Version]] = ( - candidate.version for candidate in finder.find_all_candidates(query) - ) - - if not options.pre: - # Remove prereleases - versions = ( - version for version in versions if not version.is_prerelease - ) - versions = set(versions) - - if not versions: - raise DistributionNotFound( - "No matching distribution found for {}".format(query) - ) - - formatted_versions = [str(ver) for ver in sorted(versions, reverse=True)] - latest = formatted_versions[0] - - write_output("{} ({})".format(query, latest)) - write_output("Available versions: {}".format(", ".join(formatted_versions))) - print_dist_installation_info(query, latest) diff --git a/venv/lib/python3.11/site-packages/pip/_internal/commands/inspect.py b/venv/lib/python3.11/site-packages/pip/_internal/commands/inspect.py deleted file mode 100644 index 27c8fa3..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/commands/inspect.py +++ /dev/null @@ -1,92 +0,0 @@ -import logging -from optparse import Values -from typing import Any, Dict, List - -from pip._vendor.packaging.markers import default_environment -from pip._vendor.rich import print_json - -from pip import __version__ -from pip._internal.cli import cmdoptions -from pip._internal.cli.req_command import Command -from pip._internal.cli.status_codes import SUCCESS -from pip._internal.metadata import BaseDistribution, get_environment -from pip._internal.utils.compat import stdlib_pkgs -from pip._internal.utils.urls import path_to_url - -logger = logging.getLogger(__name__) - - -class InspectCommand(Command): - """ - Inspect the content of a Python environment and produce a report in JSON format. - """ - - ignore_require_venv = True - usage = """ - %prog [options]""" - - def add_options(self) -> None: - self.cmd_opts.add_option( - "--local", - action="store_true", - default=False, - help=( - "If in a virtualenv that has global access, do not list " - "globally-installed packages." - ), - ) - self.cmd_opts.add_option( - "--user", - dest="user", - action="store_true", - default=False, - help="Only output packages installed in user-site.", - ) - self.cmd_opts.add_option(cmdoptions.list_path()) - self.parser.insert_option_group(0, self.cmd_opts) - - def run(self, options: Values, args: List[str]) -> int: - cmdoptions.check_list_path_option(options) - dists = get_environment(options.path).iter_installed_distributions( - local_only=options.local, - user_only=options.user, - skip=set(stdlib_pkgs), - ) - output = { - "version": "1", - "pip_version": __version__, - "installed": [self._dist_to_dict(dist) for dist in dists], - "environment": default_environment(), - # TODO tags? scheme? - } - print_json(data=output) - return SUCCESS - - def _dist_to_dict(self, dist: BaseDistribution) -> Dict[str, Any]: - res: Dict[str, Any] = { - "metadata": dist.metadata_dict, - "metadata_location": dist.info_location, - } - # direct_url. Note that we don't have download_info (as in the installation - # report) since it is not recorded in installed metadata. - direct_url = dist.direct_url - if direct_url is not None: - res["direct_url"] = direct_url.to_dict() - else: - # Emulate direct_url for legacy editable installs. - editable_project_location = dist.editable_project_location - if editable_project_location is not None: - res["direct_url"] = { - "url": path_to_url(editable_project_location), - "dir_info": { - "editable": True, - }, - } - # installer - installer = dist.installer - if dist.installer: - res["installer"] = installer - # requested - if dist.installed_with_dist_info: - res["requested"] = dist.requested - return res diff --git a/venv/lib/python3.11/site-packages/pip/_internal/commands/install.py b/venv/lib/python3.11/site-packages/pip/_internal/commands/install.py deleted file mode 100644 index 365764f..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/commands/install.py +++ /dev/null @@ -1,778 +0,0 @@ -import errno -import json -import operator -import os -import shutil -import site -from optparse import SUPPRESS_HELP, Values -from typing import List, Optional - -from pip._vendor.rich import print_json - -from pip._internal.cache import WheelCache -from pip._internal.cli import cmdoptions -from pip._internal.cli.cmdoptions import make_target_python -from pip._internal.cli.req_command import ( - RequirementCommand, - warn_if_run_as_root, - with_cleanup, -) -from pip._internal.cli.status_codes import ERROR, SUCCESS -from pip._internal.exceptions import CommandError, InstallationError -from pip._internal.locations import get_scheme -from pip._internal.metadata import get_environment -from pip._internal.models.installation_report import InstallationReport -from pip._internal.operations.build.build_tracker import get_build_tracker -from pip._internal.operations.check import ConflictDetails, check_install_conflicts -from pip._internal.req import install_given_reqs -from pip._internal.req.req_install import ( - InstallRequirement, - check_legacy_setup_py_options, -) -from pip._internal.utils.compat import WINDOWS -from pip._internal.utils.filesystem import test_writable_dir -from pip._internal.utils.logging import getLogger -from pip._internal.utils.misc import ( - check_externally_managed, - ensure_dir, - get_pip_version, - protect_pip_from_modification_on_windows, - write_output, -) -from pip._internal.utils.temp_dir import TempDirectory -from pip._internal.utils.virtualenv import ( - running_under_virtualenv, - virtualenv_no_global, -) -from pip._internal.wheel_builder import build, should_build_for_install_command - -logger = getLogger(__name__) - - -class InstallCommand(RequirementCommand): - """ - Install packages from: - - - PyPI (and other indexes) using requirement specifiers. - - VCS project urls. - - Local project directories. - - Local or remote source archives. - - pip also supports installing from "requirements files", which provide - an easy way to specify a whole environment to be installed. - """ - - usage = """ - %prog [options] [package-index-options] ... - %prog [options] -r [package-index-options] ... - %prog [options] [-e] ... - %prog [options] [-e] ... - %prog [options] ...""" - - def add_options(self) -> None: - self.cmd_opts.add_option(cmdoptions.requirements()) - self.cmd_opts.add_option(cmdoptions.constraints()) - self.cmd_opts.add_option(cmdoptions.no_deps()) - self.cmd_opts.add_option(cmdoptions.pre()) - - self.cmd_opts.add_option(cmdoptions.editable()) - self.cmd_opts.add_option( - "--dry-run", - action="store_true", - dest="dry_run", - default=False, - help=( - "Don't actually install anything, just print what would be. " - "Can be used in combination with --ignore-installed " - "to 'resolve' the requirements." - ), - ) - self.cmd_opts.add_option( - "-t", - "--target", - dest="target_dir", - metavar="dir", - default=None, - help=( - "Install packages into . " - "By default this will not replace existing files/folders in " - ". Use --upgrade to replace existing packages in " - "with new versions." - ), - ) - cmdoptions.add_target_python_options(self.cmd_opts) - - self.cmd_opts.add_option( - "--user", - dest="use_user_site", - action="store_true", - help=( - "Install to the Python user install directory for your " - "platform. Typically ~/.local/, or %APPDATA%\\Python on " - "Windows. (See the Python documentation for site.USER_BASE " - "for full details.)" - ), - ) - self.cmd_opts.add_option( - "--no-user", - dest="use_user_site", - action="store_false", - help=SUPPRESS_HELP, - ) - self.cmd_opts.add_option( - "--root", - dest="root_path", - metavar="dir", - default=None, - help="Install everything relative to this alternate root directory.", - ) - self.cmd_opts.add_option( - "--prefix", - dest="prefix_path", - metavar="dir", - default=None, - help=( - "Installation prefix where lib, bin and other top-level " - "folders are placed. Note that the resulting installation may " - "contain scripts and other resources which reference the " - "Python interpreter of pip, and not that of ``--prefix``. " - "See also the ``--python`` option if the intention is to " - "install packages into another (possibly pip-free) " - "environment." - ), - ) - - self.cmd_opts.add_option(cmdoptions.src()) - - self.cmd_opts.add_option( - "-U", - "--upgrade", - dest="upgrade", - action="store_true", - help=( - "Upgrade all specified packages to the newest available " - "version. The handling of dependencies depends on the " - "upgrade-strategy used." - ), - ) - - self.cmd_opts.add_option( - "--upgrade-strategy", - dest="upgrade_strategy", - default="only-if-needed", - choices=["only-if-needed", "eager"], - help=( - "Determines how dependency upgrading should be handled " - "[default: %default]. " - '"eager" - dependencies are upgraded regardless of ' - "whether the currently installed version satisfies the " - "requirements of the upgraded package(s). " - '"only-if-needed" - are upgraded only when they do not ' - "satisfy the requirements of the upgraded package(s)." - ), - ) - - self.cmd_opts.add_option( - "--force-reinstall", - dest="force_reinstall", - action="store_true", - help="Reinstall all packages even if they are already up-to-date.", - ) - - self.cmd_opts.add_option( - "-I", - "--ignore-installed", - dest="ignore_installed", - action="store_true", - help=( - "Ignore the installed packages, overwriting them. " - "This can break your system if the existing package " - "is of a different version or was installed " - "with a different package manager!" - ), - ) - - self.cmd_opts.add_option(cmdoptions.ignore_requires_python()) - self.cmd_opts.add_option(cmdoptions.no_build_isolation()) - self.cmd_opts.add_option(cmdoptions.use_pep517()) - self.cmd_opts.add_option(cmdoptions.no_use_pep517()) - self.cmd_opts.add_option(cmdoptions.check_build_deps()) - self.cmd_opts.add_option(cmdoptions.override_externally_managed()) - - self.cmd_opts.add_option(cmdoptions.config_settings()) - self.cmd_opts.add_option(cmdoptions.global_options()) - - self.cmd_opts.add_option( - "--compile", - action="store_true", - dest="compile", - default=True, - help="Compile Python source files to bytecode", - ) - - self.cmd_opts.add_option( - "--no-compile", - action="store_false", - dest="compile", - help="Do not compile Python source files to bytecode", - ) - - self.cmd_opts.add_option( - "--no-warn-script-location", - action="store_false", - dest="warn_script_location", - default=True, - help="Do not warn when installing scripts outside PATH", - ) - self.cmd_opts.add_option( - "--no-warn-conflicts", - action="store_false", - dest="warn_about_conflicts", - default=True, - help="Do not warn about broken dependencies", - ) - self.cmd_opts.add_option(cmdoptions.no_binary()) - self.cmd_opts.add_option(cmdoptions.only_binary()) - self.cmd_opts.add_option(cmdoptions.prefer_binary()) - self.cmd_opts.add_option(cmdoptions.require_hashes()) - self.cmd_opts.add_option(cmdoptions.progress_bar()) - self.cmd_opts.add_option(cmdoptions.root_user_action()) - - index_opts = cmdoptions.make_option_group( - cmdoptions.index_group, - self.parser, - ) - - self.parser.insert_option_group(0, index_opts) - self.parser.insert_option_group(0, self.cmd_opts) - - self.cmd_opts.add_option( - "--report", - dest="json_report_file", - metavar="file", - default=None, - help=( - "Generate a JSON file describing what pip did to install " - "the provided requirements. " - "Can be used in combination with --dry-run and --ignore-installed " - "to 'resolve' the requirements. " - "When - is used as file name it writes to stdout. " - "When writing to stdout, please combine with the --quiet option " - "to avoid mixing pip logging output with JSON output." - ), - ) - - @with_cleanup - def run(self, options: Values, args: List[str]) -> int: - if options.use_user_site and options.target_dir is not None: - raise CommandError("Can not combine '--user' and '--target'") - - # Check whether the environment we're installing into is externally - # managed, as specified in PEP 668. Specifying --root, --target, or - # --prefix disables the check, since there's no reliable way to locate - # the EXTERNALLY-MANAGED file for those cases. An exception is also - # made specifically for "--dry-run --report" for convenience. - installing_into_current_environment = ( - not (options.dry_run and options.json_report_file) - and options.root_path is None - and options.target_dir is None - and options.prefix_path is None - ) - if ( - installing_into_current_environment - and not options.override_externally_managed - ): - check_externally_managed() - - upgrade_strategy = "to-satisfy-only" - if options.upgrade: - upgrade_strategy = options.upgrade_strategy - - cmdoptions.check_dist_restriction(options, check_target=True) - - logger.verbose("Using %s", get_pip_version()) - options.use_user_site = decide_user_install( - options.use_user_site, - prefix_path=options.prefix_path, - target_dir=options.target_dir, - root_path=options.root_path, - isolated_mode=options.isolated_mode, - ) - - target_temp_dir: Optional[TempDirectory] = None - target_temp_dir_path: Optional[str] = None - if options.target_dir: - options.ignore_installed = True - options.target_dir = os.path.abspath(options.target_dir) - if ( - # fmt: off - os.path.exists(options.target_dir) and - not os.path.isdir(options.target_dir) - # fmt: on - ): - raise CommandError( - "Target path exists but is not a directory, will not continue." - ) - - # Create a target directory for using with the target option - target_temp_dir = TempDirectory(kind="target") - target_temp_dir_path = target_temp_dir.path - self.enter_context(target_temp_dir) - - global_options = options.global_options or [] - - session = self.get_default_session(options) - - target_python = make_target_python(options) - finder = self._build_package_finder( - options=options, - session=session, - target_python=target_python, - ignore_requires_python=options.ignore_requires_python, - ) - build_tracker = self.enter_context(get_build_tracker()) - - directory = TempDirectory( - delete=not options.no_clean, - kind="install", - globally_managed=True, - ) - - try: - reqs = self.get_requirements(args, options, finder, session) - check_legacy_setup_py_options(options, reqs) - - wheel_cache = WheelCache(options.cache_dir) - - # Only when installing is it permitted to use PEP 660. - # In other circumstances (pip wheel, pip download) we generate - # regular (i.e. non editable) metadata and wheels. - for req in reqs: - req.permit_editable_wheels = True - - preparer = self.make_requirement_preparer( - temp_build_dir=directory, - options=options, - build_tracker=build_tracker, - session=session, - finder=finder, - use_user_site=options.use_user_site, - verbosity=self.verbosity, - ) - resolver = self.make_resolver( - preparer=preparer, - finder=finder, - options=options, - wheel_cache=wheel_cache, - use_user_site=options.use_user_site, - ignore_installed=options.ignore_installed, - ignore_requires_python=options.ignore_requires_python, - force_reinstall=options.force_reinstall, - upgrade_strategy=upgrade_strategy, - use_pep517=options.use_pep517, - ) - - self.trace_basic_info(finder) - - requirement_set = resolver.resolve( - reqs, check_supported_wheels=not options.target_dir - ) - - if options.json_report_file: - report = InstallationReport(requirement_set.requirements_to_install) - if options.json_report_file == "-": - print_json(data=report.to_dict()) - else: - with open(options.json_report_file, "w", encoding="utf-8") as f: - json.dump(report.to_dict(), f, indent=2, ensure_ascii=False) - - if options.dry_run: - # In non dry-run mode, the legacy versions and specifiers check - # will be done as part of conflict detection. - requirement_set.warn_legacy_versions_and_specifiers() - would_install_items = sorted( - (r.metadata["name"], r.metadata["version"]) - for r in requirement_set.requirements_to_install - ) - if would_install_items: - write_output( - "Would install %s", - " ".join("-".join(item) for item in would_install_items), - ) - return SUCCESS - - try: - pip_req = requirement_set.get_requirement("pip") - except KeyError: - modifying_pip = False - else: - # If we're not replacing an already installed pip, - # we're not modifying it. - modifying_pip = pip_req.satisfied_by is None - protect_pip_from_modification_on_windows(modifying_pip=modifying_pip) - - reqs_to_build = [ - r - for r in requirement_set.requirements.values() - if should_build_for_install_command(r) - ] - - _, build_failures = build( - reqs_to_build, - wheel_cache=wheel_cache, - verify=True, - build_options=[], - global_options=global_options, - ) - - if build_failures: - raise InstallationError( - "Could not build wheels for {}, which is required to " - "install pyproject.toml-based projects".format( - ", ".join(r.name for r in build_failures) # type: ignore - ) - ) - - to_install = resolver.get_installation_order(requirement_set) - - # Check for conflicts in the package set we're installing. - conflicts: Optional[ConflictDetails] = None - should_warn_about_conflicts = ( - not options.ignore_dependencies and options.warn_about_conflicts - ) - if should_warn_about_conflicts: - conflicts = self._determine_conflicts(to_install) - - # Don't warn about script install locations if - # --target or --prefix has been specified - warn_script_location = options.warn_script_location - if options.target_dir or options.prefix_path: - warn_script_location = False - - installed = install_given_reqs( - to_install, - global_options, - root=options.root_path, - home=target_temp_dir_path, - prefix=options.prefix_path, - warn_script_location=warn_script_location, - use_user_site=options.use_user_site, - pycompile=options.compile, - ) - - lib_locations = get_lib_location_guesses( - user=options.use_user_site, - home=target_temp_dir_path, - root=options.root_path, - prefix=options.prefix_path, - isolated=options.isolated_mode, - ) - env = get_environment(lib_locations) - - installed.sort(key=operator.attrgetter("name")) - items = [] - for result in installed: - item = result.name - try: - installed_dist = env.get_distribution(item) - if installed_dist is not None: - item = f"{item}-{installed_dist.version}" - except Exception: - pass - items.append(item) - - if conflicts is not None: - self._warn_about_conflicts( - conflicts, - resolver_variant=self.determine_resolver_variant(options), - ) - - installed_desc = " ".join(items) - if installed_desc: - write_output( - "Successfully installed %s", - installed_desc, - ) - except OSError as error: - show_traceback = self.verbosity >= 1 - - message = create_os_error_message( - error, - show_traceback, - options.use_user_site, - ) - logger.error(message, exc_info=show_traceback) - - return ERROR - - if options.target_dir: - assert target_temp_dir - self._handle_target_dir( - options.target_dir, target_temp_dir, options.upgrade - ) - if options.root_user_action == "warn": - warn_if_run_as_root() - return SUCCESS - - def _handle_target_dir( - self, target_dir: str, target_temp_dir: TempDirectory, upgrade: bool - ) -> None: - ensure_dir(target_dir) - - # Checking both purelib and platlib directories for installed - # packages to be moved to target directory - lib_dir_list = [] - - # Checking both purelib and platlib directories for installed - # packages to be moved to target directory - scheme = get_scheme("", home=target_temp_dir.path) - purelib_dir = scheme.purelib - platlib_dir = scheme.platlib - data_dir = scheme.data - - if os.path.exists(purelib_dir): - lib_dir_list.append(purelib_dir) - if os.path.exists(platlib_dir) and platlib_dir != purelib_dir: - lib_dir_list.append(platlib_dir) - if os.path.exists(data_dir): - lib_dir_list.append(data_dir) - - for lib_dir in lib_dir_list: - for item in os.listdir(lib_dir): - if lib_dir == data_dir: - ddir = os.path.join(data_dir, item) - if any(s.startswith(ddir) for s in lib_dir_list[:-1]): - continue - target_item_dir = os.path.join(target_dir, item) - if os.path.exists(target_item_dir): - if not upgrade: - logger.warning( - "Target directory %s already exists. Specify " - "--upgrade to force replacement.", - target_item_dir, - ) - continue - if os.path.islink(target_item_dir): - logger.warning( - "Target directory %s already exists and is " - "a link. pip will not automatically replace " - "links, please remove if replacement is " - "desired.", - target_item_dir, - ) - continue - if os.path.isdir(target_item_dir): - shutil.rmtree(target_item_dir) - else: - os.remove(target_item_dir) - - shutil.move(os.path.join(lib_dir, item), target_item_dir) - - def _determine_conflicts( - self, to_install: List[InstallRequirement] - ) -> Optional[ConflictDetails]: - try: - return check_install_conflicts(to_install) - except Exception: - logger.exception( - "Error while checking for conflicts. Please file an issue on " - "pip's issue tracker: https://github.com/pypa/pip/issues/new" - ) - return None - - def _warn_about_conflicts( - self, conflict_details: ConflictDetails, resolver_variant: str - ) -> None: - package_set, (missing, conflicting) = conflict_details - if not missing and not conflicting: - return - - parts: List[str] = [] - if resolver_variant == "legacy": - parts.append( - "pip's legacy dependency resolver does not consider dependency " - "conflicts when selecting packages. This behaviour is the " - "source of the following dependency conflicts." - ) - else: - assert resolver_variant == "resolvelib" - parts.append( - "pip's dependency resolver does not currently take into account " - "all the packages that are installed. This behaviour is the " - "source of the following dependency conflicts." - ) - - # NOTE: There is some duplication here, with commands/check.py - for project_name in missing: - version = package_set[project_name][0] - for dependency in missing[project_name]: - message = ( - "{name} {version} requires {requirement}, " - "which is not installed." - ).format( - name=project_name, - version=version, - requirement=dependency[1], - ) - parts.append(message) - - for project_name in conflicting: - version = package_set[project_name][0] - for dep_name, dep_version, req in conflicting[project_name]: - message = ( - "{name} {version} requires {requirement}, but {you} have " - "{dep_name} {dep_version} which is incompatible." - ).format( - name=project_name, - version=version, - requirement=req, - dep_name=dep_name, - dep_version=dep_version, - you=("you" if resolver_variant == "resolvelib" else "you'll"), - ) - parts.append(message) - - logger.critical("\n".join(parts)) - - -def get_lib_location_guesses( - user: bool = False, - home: Optional[str] = None, - root: Optional[str] = None, - isolated: bool = False, - prefix: Optional[str] = None, -) -> List[str]: - scheme = get_scheme( - "", - user=user, - home=home, - root=root, - isolated=isolated, - prefix=prefix, - ) - return [scheme.purelib, scheme.platlib] - - -def site_packages_writable(root: Optional[str], isolated: bool) -> bool: - return all( - test_writable_dir(d) - for d in set(get_lib_location_guesses(root=root, isolated=isolated)) - ) - - -def decide_user_install( - use_user_site: Optional[bool], - prefix_path: Optional[str] = None, - target_dir: Optional[str] = None, - root_path: Optional[str] = None, - isolated_mode: bool = False, -) -> bool: - """Determine whether to do a user install based on the input options. - - If use_user_site is False, no additional checks are done. - If use_user_site is True, it is checked for compatibility with other - options. - If use_user_site is None, the default behaviour depends on the environment, - which is provided by the other arguments. - """ - # In some cases (config from tox), use_user_site can be set to an integer - # rather than a bool, which 'use_user_site is False' wouldn't catch. - if (use_user_site is not None) and (not use_user_site): - logger.debug("Non-user install by explicit request") - return False - - if use_user_site: - if prefix_path: - raise CommandError( - "Can not combine '--user' and '--prefix' as they imply " - "different installation locations" - ) - if virtualenv_no_global(): - raise InstallationError( - "Can not perform a '--user' install. User site-packages " - "are not visible in this virtualenv." - ) - logger.debug("User install by explicit request") - return True - - # If we are here, user installs have not been explicitly requested/avoided - assert use_user_site is None - - # user install incompatible with --prefix/--target - if prefix_path or target_dir: - logger.debug("Non-user install due to --prefix or --target option") - return False - - # If user installs are not enabled, choose a non-user install - if not site.ENABLE_USER_SITE: - logger.debug("Non-user install because user site-packages disabled") - return False - - # If we have permission for a non-user install, do that, - # otherwise do a user install. - if site_packages_writable(root=root_path, isolated=isolated_mode): - logger.debug("Non-user install because site-packages writeable") - return False - - logger.info( - "Defaulting to user installation because normal site-packages " - "is not writeable" - ) - return True - - -def create_os_error_message( - error: OSError, show_traceback: bool, using_user_site: bool -) -> str: - """Format an error message for an OSError - - It may occur anytime during the execution of the install command. - """ - parts = [] - - # Mention the error if we are not going to show a traceback - parts.append("Could not install packages due to an OSError") - if not show_traceback: - parts.append(": ") - parts.append(str(error)) - else: - parts.append(".") - - # Spilt the error indication from a helper message (if any) - parts[-1] += "\n" - - # Suggest useful actions to the user: - # (1) using user site-packages or (2) verifying the permissions - if error.errno == errno.EACCES: - user_option_part = "Consider using the `--user` option" - permissions_part = "Check the permissions" - - if not running_under_virtualenv() and not using_user_site: - parts.extend( - [ - user_option_part, - " or ", - permissions_part.lower(), - ] - ) - else: - parts.append(permissions_part) - parts.append(".\n") - - # Suggest the user to enable Long Paths if path length is - # more than 260 - if ( - WINDOWS - and error.errno == errno.ENOENT - and error.filename - and len(error.filename) > 260 - ): - parts.append( - "HINT: This error might have occurred since " - "this system does not have Windows Long Path " - "support enabled. You can find information on " - "how to enable this at " - "https://pip.pypa.io/warnings/enable-long-paths\n" - ) - - return "".join(parts).strip() + "\n" diff --git a/venv/lib/python3.11/site-packages/pip/_internal/commands/list.py b/venv/lib/python3.11/site-packages/pip/_internal/commands/list.py deleted file mode 100644 index e551dda..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/commands/list.py +++ /dev/null @@ -1,368 +0,0 @@ -import json -import logging -from optparse import Values -from typing import TYPE_CHECKING, Generator, List, Optional, Sequence, Tuple, cast - -from pip._vendor.packaging.utils import canonicalize_name - -from pip._internal.cli import cmdoptions -from pip._internal.cli.req_command import IndexGroupCommand -from pip._internal.cli.status_codes import SUCCESS -from pip._internal.exceptions import CommandError -from pip._internal.index.collector import LinkCollector -from pip._internal.index.package_finder import PackageFinder -from pip._internal.metadata import BaseDistribution, get_environment -from pip._internal.models.selection_prefs import SelectionPreferences -from pip._internal.network.session import PipSession -from pip._internal.utils.compat import stdlib_pkgs -from pip._internal.utils.misc import tabulate, write_output - -if TYPE_CHECKING: - from pip._internal.metadata.base import DistributionVersion - - class _DistWithLatestInfo(BaseDistribution): - """Give the distribution object a couple of extra fields. - - These will be populated during ``get_outdated()``. This is dirty but - makes the rest of the code much cleaner. - """ - - latest_version: DistributionVersion - latest_filetype: str - - _ProcessedDists = Sequence[_DistWithLatestInfo] - - -logger = logging.getLogger(__name__) - - -class ListCommand(IndexGroupCommand): - """ - List installed packages, including editables. - - Packages are listed in a case-insensitive sorted order. - """ - - ignore_require_venv = True - usage = """ - %prog [options]""" - - def add_options(self) -> None: - self.cmd_opts.add_option( - "-o", - "--outdated", - action="store_true", - default=False, - help="List outdated packages", - ) - self.cmd_opts.add_option( - "-u", - "--uptodate", - action="store_true", - default=False, - help="List uptodate packages", - ) - self.cmd_opts.add_option( - "-e", - "--editable", - action="store_true", - default=False, - help="List editable projects.", - ) - self.cmd_opts.add_option( - "-l", - "--local", - action="store_true", - default=False, - help=( - "If in a virtualenv that has global access, do not list " - "globally-installed packages." - ), - ) - self.cmd_opts.add_option( - "--user", - dest="user", - action="store_true", - default=False, - help="Only output packages installed in user-site.", - ) - self.cmd_opts.add_option(cmdoptions.list_path()) - self.cmd_opts.add_option( - "--pre", - action="store_true", - default=False, - help=( - "Include pre-release and development versions. By default, " - "pip only finds stable versions." - ), - ) - - self.cmd_opts.add_option( - "--format", - action="store", - dest="list_format", - default="columns", - choices=("columns", "freeze", "json"), - help=( - "Select the output format among: columns (default), freeze, or json. " - "The 'freeze' format cannot be used with the --outdated option." - ), - ) - - self.cmd_opts.add_option( - "--not-required", - action="store_true", - dest="not_required", - help="List packages that are not dependencies of installed packages.", - ) - - self.cmd_opts.add_option( - "--exclude-editable", - action="store_false", - dest="include_editable", - help="Exclude editable package from output.", - ) - self.cmd_opts.add_option( - "--include-editable", - action="store_true", - dest="include_editable", - help="Include editable package from output.", - default=True, - ) - self.cmd_opts.add_option(cmdoptions.list_exclude()) - index_opts = cmdoptions.make_option_group(cmdoptions.index_group, self.parser) - - self.parser.insert_option_group(0, index_opts) - self.parser.insert_option_group(0, self.cmd_opts) - - def _build_package_finder( - self, options: Values, session: PipSession - ) -> PackageFinder: - """ - Create a package finder appropriate to this list command. - """ - link_collector = LinkCollector.create(session, options=options) - - # Pass allow_yanked=False to ignore yanked versions. - selection_prefs = SelectionPreferences( - allow_yanked=False, - allow_all_prereleases=options.pre, - ) - - return PackageFinder.create( - link_collector=link_collector, - selection_prefs=selection_prefs, - ) - - def run(self, options: Values, args: List[str]) -> int: - if options.outdated and options.uptodate: - raise CommandError("Options --outdated and --uptodate cannot be combined.") - - if options.outdated and options.list_format == "freeze": - raise CommandError( - "List format 'freeze' cannot be used with the --outdated option." - ) - - cmdoptions.check_list_path_option(options) - - skip = set(stdlib_pkgs) - if options.excludes: - skip.update(canonicalize_name(n) for n in options.excludes) - - packages: "_ProcessedDists" = [ - cast("_DistWithLatestInfo", d) - for d in get_environment(options.path).iter_installed_distributions( - local_only=options.local, - user_only=options.user, - editables_only=options.editable, - include_editables=options.include_editable, - skip=skip, - ) - ] - - # get_not_required must be called firstly in order to find and - # filter out all dependencies correctly. Otherwise a package - # can't be identified as requirement because some parent packages - # could be filtered out before. - if options.not_required: - packages = self.get_not_required(packages, options) - - if options.outdated: - packages = self.get_outdated(packages, options) - elif options.uptodate: - packages = self.get_uptodate(packages, options) - - self.output_package_listing(packages, options) - return SUCCESS - - def get_outdated( - self, packages: "_ProcessedDists", options: Values - ) -> "_ProcessedDists": - return [ - dist - for dist in self.iter_packages_latest_infos(packages, options) - if dist.latest_version > dist.version - ] - - def get_uptodate( - self, packages: "_ProcessedDists", options: Values - ) -> "_ProcessedDists": - return [ - dist - for dist in self.iter_packages_latest_infos(packages, options) - if dist.latest_version == dist.version - ] - - def get_not_required( - self, packages: "_ProcessedDists", options: Values - ) -> "_ProcessedDists": - dep_keys = { - canonicalize_name(dep.name) - for dist in packages - for dep in (dist.iter_dependencies() or ()) - } - - # Create a set to remove duplicate packages, and cast it to a list - # to keep the return type consistent with get_outdated and - # get_uptodate - return list({pkg for pkg in packages if pkg.canonical_name not in dep_keys}) - - def iter_packages_latest_infos( - self, packages: "_ProcessedDists", options: Values - ) -> Generator["_DistWithLatestInfo", None, None]: - with self._build_session(options) as session: - finder = self._build_package_finder(options, session) - - def latest_info( - dist: "_DistWithLatestInfo", - ) -> Optional["_DistWithLatestInfo"]: - all_candidates = finder.find_all_candidates(dist.canonical_name) - if not options.pre: - # Remove prereleases - all_candidates = [ - candidate - for candidate in all_candidates - if not candidate.version.is_prerelease - ] - - evaluator = finder.make_candidate_evaluator( - project_name=dist.canonical_name, - ) - best_candidate = evaluator.sort_best_candidate(all_candidates) - if best_candidate is None: - return None - - remote_version = best_candidate.version - if best_candidate.link.is_wheel: - typ = "wheel" - else: - typ = "sdist" - dist.latest_version = remote_version - dist.latest_filetype = typ - return dist - - for dist in map(latest_info, packages): - if dist is not None: - yield dist - - def output_package_listing( - self, packages: "_ProcessedDists", options: Values - ) -> None: - packages = sorted( - packages, - key=lambda dist: dist.canonical_name, - ) - if options.list_format == "columns" and packages: - data, header = format_for_columns(packages, options) - self.output_package_listing_columns(data, header) - elif options.list_format == "freeze": - for dist in packages: - if options.verbose >= 1: - write_output( - "%s==%s (%s)", dist.raw_name, dist.version, dist.location - ) - else: - write_output("%s==%s", dist.raw_name, dist.version) - elif options.list_format == "json": - write_output(format_for_json(packages, options)) - - def output_package_listing_columns( - self, data: List[List[str]], header: List[str] - ) -> None: - # insert the header first: we need to know the size of column names - if len(data) > 0: - data.insert(0, header) - - pkg_strings, sizes = tabulate(data) - - # Create and add a separator. - if len(data) > 0: - pkg_strings.insert(1, " ".join("-" * x for x in sizes)) - - for val in pkg_strings: - write_output(val) - - -def format_for_columns( - pkgs: "_ProcessedDists", options: Values -) -> Tuple[List[List[str]], List[str]]: - """ - Convert the package data into something usable - by output_package_listing_columns. - """ - header = ["Package", "Version"] - - running_outdated = options.outdated - if running_outdated: - header.extend(["Latest", "Type"]) - - has_editables = any(x.editable for x in pkgs) - if has_editables: - header.append("Editable project location") - - if options.verbose >= 1: - header.append("Location") - if options.verbose >= 1: - header.append("Installer") - - data = [] - for proj in pkgs: - # if we're working on the 'outdated' list, separate out the - # latest_version and type - row = [proj.raw_name, str(proj.version)] - - if running_outdated: - row.append(str(proj.latest_version)) - row.append(proj.latest_filetype) - - if has_editables: - row.append(proj.editable_project_location or "") - - if options.verbose >= 1: - row.append(proj.location or "") - if options.verbose >= 1: - row.append(proj.installer) - - data.append(row) - - return data, header - - -def format_for_json(packages: "_ProcessedDists", options: Values) -> str: - data = [] - for dist in packages: - info = { - "name": dist.raw_name, - "version": str(dist.version), - } - if options.verbose >= 1: - info["location"] = dist.location or "" - info["installer"] = dist.installer - if options.outdated: - info["latest_version"] = str(dist.latest_version) - info["latest_filetype"] = dist.latest_filetype - editable_project_location = dist.editable_project_location - if editable_project_location: - info["editable_project_location"] = editable_project_location - data.append(info) - return json.dumps(data) diff --git a/venv/lib/python3.11/site-packages/pip/_internal/commands/search.py b/venv/lib/python3.11/site-packages/pip/_internal/commands/search.py deleted file mode 100644 index 03ed925..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/commands/search.py +++ /dev/null @@ -1,174 +0,0 @@ -import logging -import shutil -import sys -import textwrap -import xmlrpc.client -from collections import OrderedDict -from optparse import Values -from typing import TYPE_CHECKING, Dict, List, Optional - -from pip._vendor.packaging.version import parse as parse_version - -from pip._internal.cli.base_command import Command -from pip._internal.cli.req_command import SessionCommandMixin -from pip._internal.cli.status_codes import NO_MATCHES_FOUND, SUCCESS -from pip._internal.exceptions import CommandError -from pip._internal.metadata import get_default_environment -from pip._internal.models.index import PyPI -from pip._internal.network.xmlrpc import PipXmlrpcTransport -from pip._internal.utils.logging import indent_log -from pip._internal.utils.misc import write_output - -if TYPE_CHECKING: - from typing import TypedDict - - class TransformedHit(TypedDict): - name: str - summary: str - versions: List[str] - - -logger = logging.getLogger(__name__) - - -class SearchCommand(Command, SessionCommandMixin): - """Search for PyPI packages whose name or summary contains .""" - - usage = """ - %prog [options] """ - ignore_require_venv = True - - def add_options(self) -> None: - self.cmd_opts.add_option( - "-i", - "--index", - dest="index", - metavar="URL", - default=PyPI.pypi_url, - help="Base URL of Python Package Index (default %default)", - ) - - self.parser.insert_option_group(0, self.cmd_opts) - - def run(self, options: Values, args: List[str]) -> int: - if not args: - raise CommandError("Missing required argument (search query).") - query = args - pypi_hits = self.search(query, options) - hits = transform_hits(pypi_hits) - - terminal_width = None - if sys.stdout.isatty(): - terminal_width = shutil.get_terminal_size()[0] - - print_results(hits, terminal_width=terminal_width) - if pypi_hits: - return SUCCESS - return NO_MATCHES_FOUND - - def search(self, query: List[str], options: Values) -> List[Dict[str, str]]: - index_url = options.index - - session = self.get_default_session(options) - - transport = PipXmlrpcTransport(index_url, session) - pypi = xmlrpc.client.ServerProxy(index_url, transport) - try: - hits = pypi.search({"name": query, "summary": query}, "or") - except xmlrpc.client.Fault as fault: - message = "XMLRPC request failed [code: {code}]\n{string}".format( - code=fault.faultCode, - string=fault.faultString, - ) - raise CommandError(message) - assert isinstance(hits, list) - return hits - - -def transform_hits(hits: List[Dict[str, str]]) -> List["TransformedHit"]: - """ - The list from pypi is really a list of versions. We want a list of - packages with the list of versions stored inline. This converts the - list from pypi into one we can use. - """ - packages: Dict[str, "TransformedHit"] = OrderedDict() - for hit in hits: - name = hit["name"] - summary = hit["summary"] - version = hit["version"] - - if name not in packages.keys(): - packages[name] = { - "name": name, - "summary": summary, - "versions": [version], - } - else: - packages[name]["versions"].append(version) - - # if this is the highest version, replace summary and score - if version == highest_version(packages[name]["versions"]): - packages[name]["summary"] = summary - - return list(packages.values()) - - -def print_dist_installation_info(name: str, latest: str) -> None: - env = get_default_environment() - dist = env.get_distribution(name) - if dist is not None: - with indent_log(): - if dist.version == latest: - write_output("INSTALLED: %s (latest)", dist.version) - else: - write_output("INSTALLED: %s", dist.version) - if parse_version(latest).pre: - write_output( - "LATEST: %s (pre-release; install" - " with `pip install --pre`)", - latest, - ) - else: - write_output("LATEST: %s", latest) - - -def print_results( - hits: List["TransformedHit"], - name_column_width: Optional[int] = None, - terminal_width: Optional[int] = None, -) -> None: - if not hits: - return - if name_column_width is None: - name_column_width = ( - max( - [ - len(hit["name"]) + len(highest_version(hit.get("versions", ["-"]))) - for hit in hits - ] - ) - + 4 - ) - - for hit in hits: - name = hit["name"] - summary = hit["summary"] or "" - latest = highest_version(hit.get("versions", ["-"])) - if terminal_width is not None: - target_width = terminal_width - name_column_width - 5 - if target_width > 10: - # wrap and indent summary to fit terminal - summary_lines = textwrap.wrap(summary, target_width) - summary = ("\n" + " " * (name_column_width + 3)).join(summary_lines) - - name_latest = f"{name} ({latest})" - line = f"{name_latest:{name_column_width}} - {summary}" - try: - write_output(line) - print_dist_installation_info(name, latest) - except UnicodeEncodeError: - pass - - -def highest_version(versions: List[str]) -> str: - return max(versions, key=parse_version) diff --git a/venv/lib/python3.11/site-packages/pip/_internal/commands/show.py b/venv/lib/python3.11/site-packages/pip/_internal/commands/show.py deleted file mode 100644 index 3f10701..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/commands/show.py +++ /dev/null @@ -1,189 +0,0 @@ -import logging -from optparse import Values -from typing import Generator, Iterable, Iterator, List, NamedTuple, Optional - -from pip._vendor.packaging.utils import canonicalize_name - -from pip._internal.cli.base_command import Command -from pip._internal.cli.status_codes import ERROR, SUCCESS -from pip._internal.metadata import BaseDistribution, get_default_environment -from pip._internal.utils.misc import write_output - -logger = logging.getLogger(__name__) - - -class ShowCommand(Command): - """ - Show information about one or more installed packages. - - The output is in RFC-compliant mail header format. - """ - - usage = """ - %prog [options] ...""" - ignore_require_venv = True - - def add_options(self) -> None: - self.cmd_opts.add_option( - "-f", - "--files", - dest="files", - action="store_true", - default=False, - help="Show the full list of installed files for each package.", - ) - - self.parser.insert_option_group(0, self.cmd_opts) - - def run(self, options: Values, args: List[str]) -> int: - if not args: - logger.warning("ERROR: Please provide a package name or names.") - return ERROR - query = args - - results = search_packages_info(query) - if not print_results( - results, list_files=options.files, verbose=options.verbose - ): - return ERROR - return SUCCESS - - -class _PackageInfo(NamedTuple): - name: str - version: str - location: str - editable_project_location: Optional[str] - requires: List[str] - required_by: List[str] - installer: str - metadata_version: str - classifiers: List[str] - summary: str - homepage: str - project_urls: List[str] - author: str - author_email: str - license: str - entry_points: List[str] - files: Optional[List[str]] - - -def search_packages_info(query: List[str]) -> Generator[_PackageInfo, None, None]: - """ - Gather details from installed distributions. Print distribution name, - version, location, and installed files. Installed files requires a - pip generated 'installed-files.txt' in the distributions '.egg-info' - directory. - """ - env = get_default_environment() - - installed = {dist.canonical_name: dist for dist in env.iter_all_distributions()} - query_names = [canonicalize_name(name) for name in query] - missing = sorted( - [name for name, pkg in zip(query, query_names) if pkg not in installed] - ) - if missing: - logger.warning("Package(s) not found: %s", ", ".join(missing)) - - def _get_requiring_packages(current_dist: BaseDistribution) -> Iterator[str]: - return ( - dist.metadata["Name"] or "UNKNOWN" - for dist in installed.values() - if current_dist.canonical_name - in {canonicalize_name(d.name) for d in dist.iter_dependencies()} - ) - - for query_name in query_names: - try: - dist = installed[query_name] - except KeyError: - continue - - requires = sorted((req.name for req in dist.iter_dependencies()), key=str.lower) - required_by = sorted(_get_requiring_packages(dist), key=str.lower) - - try: - entry_points_text = dist.read_text("entry_points.txt") - entry_points = entry_points_text.splitlines(keepends=False) - except FileNotFoundError: - entry_points = [] - - files_iter = dist.iter_declared_entries() - if files_iter is None: - files: Optional[List[str]] = None - else: - files = sorted(files_iter) - - metadata = dist.metadata - - yield _PackageInfo( - name=dist.raw_name, - version=str(dist.version), - location=dist.location or "", - editable_project_location=dist.editable_project_location, - requires=requires, - required_by=required_by, - installer=dist.installer, - metadata_version=dist.metadata_version or "", - classifiers=metadata.get_all("Classifier", []), - summary=metadata.get("Summary", ""), - homepage=metadata.get("Home-page", ""), - project_urls=metadata.get_all("Project-URL", []), - author=metadata.get("Author", ""), - author_email=metadata.get("Author-email", ""), - license=metadata.get("License", ""), - entry_points=entry_points, - files=files, - ) - - -def print_results( - distributions: Iterable[_PackageInfo], - list_files: bool, - verbose: bool, -) -> bool: - """ - Print the information from installed distributions found. - """ - results_printed = False - for i, dist in enumerate(distributions): - results_printed = True - if i > 0: - write_output("---") - - write_output("Name: %s", dist.name) - write_output("Version: %s", dist.version) - write_output("Summary: %s", dist.summary) - write_output("Home-page: %s", dist.homepage) - write_output("Author: %s", dist.author) - write_output("Author-email: %s", dist.author_email) - write_output("License: %s", dist.license) - write_output("Location: %s", dist.location) - if dist.editable_project_location is not None: - write_output( - "Editable project location: %s", dist.editable_project_location - ) - write_output("Requires: %s", ", ".join(dist.requires)) - write_output("Required-by: %s", ", ".join(dist.required_by)) - - if verbose: - write_output("Metadata-Version: %s", dist.metadata_version) - write_output("Installer: %s", dist.installer) - write_output("Classifiers:") - for classifier in dist.classifiers: - write_output(" %s", classifier) - write_output("Entry-points:") - for entry in dist.entry_points: - write_output(" %s", entry.strip()) - write_output("Project-URLs:") - for project_url in dist.project_urls: - write_output(" %s", project_url) - if list_files: - write_output("Files:") - if dist.files is None: - write_output("Cannot locate RECORD or installed-files.txt") - else: - for line in dist.files: - write_output(" %s", line.strip()) - return results_printed diff --git a/venv/lib/python3.11/site-packages/pip/_internal/commands/uninstall.py b/venv/lib/python3.11/site-packages/pip/_internal/commands/uninstall.py deleted file mode 100644 index f198fc3..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/commands/uninstall.py +++ /dev/null @@ -1,113 +0,0 @@ -import logging -from optparse import Values -from typing import List - -from pip._vendor.packaging.utils import canonicalize_name - -from pip._internal.cli import cmdoptions -from pip._internal.cli.base_command import Command -from pip._internal.cli.req_command import SessionCommandMixin, warn_if_run_as_root -from pip._internal.cli.status_codes import SUCCESS -from pip._internal.exceptions import InstallationError -from pip._internal.req import parse_requirements -from pip._internal.req.constructors import ( - install_req_from_line, - install_req_from_parsed_requirement, -) -from pip._internal.utils.misc import ( - check_externally_managed, - protect_pip_from_modification_on_windows, -) - -logger = logging.getLogger(__name__) - - -class UninstallCommand(Command, SessionCommandMixin): - """ - Uninstall packages. - - pip is able to uninstall most installed packages. Known exceptions are: - - - Pure distutils packages installed with ``python setup.py install``, which - leave behind no metadata to determine what files were installed. - - Script wrappers installed by ``python setup.py develop``. - """ - - usage = """ - %prog [options] ... - %prog [options] -r ...""" - - def add_options(self) -> None: - self.cmd_opts.add_option( - "-r", - "--requirement", - dest="requirements", - action="append", - default=[], - metavar="file", - help=( - "Uninstall all the packages listed in the given requirements " - "file. This option can be used multiple times." - ), - ) - self.cmd_opts.add_option( - "-y", - "--yes", - dest="yes", - action="store_true", - help="Don't ask for confirmation of uninstall deletions.", - ) - self.cmd_opts.add_option(cmdoptions.root_user_action()) - self.cmd_opts.add_option(cmdoptions.override_externally_managed()) - self.parser.insert_option_group(0, self.cmd_opts) - - def run(self, options: Values, args: List[str]) -> int: - session = self.get_default_session(options) - - reqs_to_uninstall = {} - for name in args: - req = install_req_from_line( - name, - isolated=options.isolated_mode, - ) - if req.name: - reqs_to_uninstall[canonicalize_name(req.name)] = req - else: - logger.warning( - "Invalid requirement: %r ignored -" - " the uninstall command expects named" - " requirements.", - name, - ) - for filename in options.requirements: - for parsed_req in parse_requirements( - filename, options=options, session=session - ): - req = install_req_from_parsed_requirement( - parsed_req, isolated=options.isolated_mode - ) - if req.name: - reqs_to_uninstall[canonicalize_name(req.name)] = req - if not reqs_to_uninstall: - raise InstallationError( - f"You must give at least one requirement to {self.name} (see " - f'"pip help {self.name}")' - ) - - if not options.override_externally_managed: - check_externally_managed() - - protect_pip_from_modification_on_windows( - modifying_pip="pip" in reqs_to_uninstall - ) - - for req in reqs_to_uninstall.values(): - uninstall_pathset = req.uninstall( - auto_confirm=options.yes, - verbose=self.verbosity > 0, - ) - if uninstall_pathset: - uninstall_pathset.commit() - if options.root_user_action == "warn": - warn_if_run_as_root() - return SUCCESS diff --git a/venv/lib/python3.11/site-packages/pip/_internal/commands/wheel.py b/venv/lib/python3.11/site-packages/pip/_internal/commands/wheel.py deleted file mode 100644 index ed578aa..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/commands/wheel.py +++ /dev/null @@ -1,183 +0,0 @@ -import logging -import os -import shutil -from optparse import Values -from typing import List - -from pip._internal.cache import WheelCache -from pip._internal.cli import cmdoptions -from pip._internal.cli.req_command import RequirementCommand, with_cleanup -from pip._internal.cli.status_codes import SUCCESS -from pip._internal.exceptions import CommandError -from pip._internal.operations.build.build_tracker import get_build_tracker -from pip._internal.req.req_install import ( - InstallRequirement, - check_legacy_setup_py_options, -) -from pip._internal.utils.misc import ensure_dir, normalize_path -from pip._internal.utils.temp_dir import TempDirectory -from pip._internal.wheel_builder import build, should_build_for_wheel_command - -logger = logging.getLogger(__name__) - - -class WheelCommand(RequirementCommand): - """ - Build Wheel archives for your requirements and dependencies. - - Wheel is a built-package format, and offers the advantage of not - recompiling your software during every install. For more details, see the - wheel docs: https://wheel.readthedocs.io/en/latest/ - - 'pip wheel' uses the build system interface as described here: - https://pip.pypa.io/en/stable/reference/build-system/ - - """ - - usage = """ - %prog [options] ... - %prog [options] -r ... - %prog [options] [-e] ... - %prog [options] [-e] ... - %prog [options] ...""" - - def add_options(self) -> None: - self.cmd_opts.add_option( - "-w", - "--wheel-dir", - dest="wheel_dir", - metavar="dir", - default=os.curdir, - help=( - "Build wheels into , where the default is the " - "current working directory." - ), - ) - self.cmd_opts.add_option(cmdoptions.no_binary()) - self.cmd_opts.add_option(cmdoptions.only_binary()) - self.cmd_opts.add_option(cmdoptions.prefer_binary()) - self.cmd_opts.add_option(cmdoptions.no_build_isolation()) - self.cmd_opts.add_option(cmdoptions.use_pep517()) - self.cmd_opts.add_option(cmdoptions.no_use_pep517()) - self.cmd_opts.add_option(cmdoptions.check_build_deps()) - self.cmd_opts.add_option(cmdoptions.constraints()) - self.cmd_opts.add_option(cmdoptions.editable()) - self.cmd_opts.add_option(cmdoptions.requirements()) - self.cmd_opts.add_option(cmdoptions.src()) - self.cmd_opts.add_option(cmdoptions.ignore_requires_python()) - self.cmd_opts.add_option(cmdoptions.no_deps()) - self.cmd_opts.add_option(cmdoptions.progress_bar()) - - self.cmd_opts.add_option( - "--no-verify", - dest="no_verify", - action="store_true", - default=False, - help="Don't verify if built wheel is valid.", - ) - - self.cmd_opts.add_option(cmdoptions.config_settings()) - self.cmd_opts.add_option(cmdoptions.build_options()) - self.cmd_opts.add_option(cmdoptions.global_options()) - - self.cmd_opts.add_option( - "--pre", - action="store_true", - default=False, - help=( - "Include pre-release and development versions. By default, " - "pip only finds stable versions." - ), - ) - - self.cmd_opts.add_option(cmdoptions.require_hashes()) - - index_opts = cmdoptions.make_option_group( - cmdoptions.index_group, - self.parser, - ) - - self.parser.insert_option_group(0, index_opts) - self.parser.insert_option_group(0, self.cmd_opts) - - @with_cleanup - def run(self, options: Values, args: List[str]) -> int: - session = self.get_default_session(options) - - finder = self._build_package_finder(options, session) - - options.wheel_dir = normalize_path(options.wheel_dir) - ensure_dir(options.wheel_dir) - - build_tracker = self.enter_context(get_build_tracker()) - - directory = TempDirectory( - delete=not options.no_clean, - kind="wheel", - globally_managed=True, - ) - - reqs = self.get_requirements(args, options, finder, session) - check_legacy_setup_py_options(options, reqs) - - wheel_cache = WheelCache(options.cache_dir) - - preparer = self.make_requirement_preparer( - temp_build_dir=directory, - options=options, - build_tracker=build_tracker, - session=session, - finder=finder, - download_dir=options.wheel_dir, - use_user_site=False, - verbosity=self.verbosity, - ) - - resolver = self.make_resolver( - preparer=preparer, - finder=finder, - options=options, - wheel_cache=wheel_cache, - ignore_requires_python=options.ignore_requires_python, - use_pep517=options.use_pep517, - ) - - self.trace_basic_info(finder) - - requirement_set = resolver.resolve(reqs, check_supported_wheels=True) - - reqs_to_build: List[InstallRequirement] = [] - for req in requirement_set.requirements.values(): - if req.is_wheel: - preparer.save_linked_requirement(req) - elif should_build_for_wheel_command(req): - reqs_to_build.append(req) - - preparer.prepare_linked_requirements_more(requirement_set.requirements.values()) - requirement_set.warn_legacy_versions_and_specifiers() - - # build wheels - build_successes, build_failures = build( - reqs_to_build, - wheel_cache=wheel_cache, - verify=(not options.no_verify), - build_options=options.build_options or [], - global_options=options.global_options or [], - ) - for req in build_successes: - assert req.link and req.link.is_wheel - assert req.local_file_path - # copy from cache to target directory - try: - shutil.copy(req.local_file_path, options.wheel_dir) - except OSError as e: - logger.warning( - "Building wheel for %s failed: %s", - req.name, - e, - ) - build_failures.append(req) - if len(build_failures) != 0: - raise CommandError("Failed to build one or more wheels") - - return SUCCESS diff --git a/venv/lib/python3.11/site-packages/pip/_internal/configuration.py b/venv/lib/python3.11/site-packages/pip/_internal/configuration.py deleted file mode 100644 index 96f8249..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/configuration.py +++ /dev/null @@ -1,381 +0,0 @@ -"""Configuration management setup - -Some terminology: -- name - As written in config files. -- value - Value associated with a name -- key - Name combined with it's section (section.name) -- variant - A single word describing where the configuration key-value pair came from -""" - -import configparser -import locale -import os -import sys -from typing import Any, Dict, Iterable, List, NewType, Optional, Tuple - -from pip._internal.exceptions import ( - ConfigurationError, - ConfigurationFileCouldNotBeLoaded, -) -from pip._internal.utils import appdirs -from pip._internal.utils.compat import WINDOWS -from pip._internal.utils.logging import getLogger -from pip._internal.utils.misc import ensure_dir, enum - -RawConfigParser = configparser.RawConfigParser # Shorthand -Kind = NewType("Kind", str) - -CONFIG_BASENAME = "pip.ini" if WINDOWS else "pip.conf" -ENV_NAMES_IGNORED = "version", "help" - -# The kinds of configurations there are. -kinds = enum( - USER="user", # User Specific - GLOBAL="global", # System Wide - SITE="site", # [Virtual] Environment Specific - ENV="env", # from PIP_CONFIG_FILE - ENV_VAR="env-var", # from Environment Variables -) -OVERRIDE_ORDER = kinds.GLOBAL, kinds.USER, kinds.SITE, kinds.ENV, kinds.ENV_VAR -VALID_LOAD_ONLY = kinds.USER, kinds.GLOBAL, kinds.SITE - -logger = getLogger(__name__) - - -# NOTE: Maybe use the optionx attribute to normalize keynames. -def _normalize_name(name: str) -> str: - """Make a name consistent regardless of source (environment or file)""" - name = name.lower().replace("_", "-") - if name.startswith("--"): - name = name[2:] # only prefer long opts - return name - - -def _disassemble_key(name: str) -> List[str]: - if "." not in name: - error_message = ( - "Key does not contain dot separated section and key. " - "Perhaps you wanted to use 'global.{}' instead?" - ).format(name) - raise ConfigurationError(error_message) - return name.split(".", 1) - - -def get_configuration_files() -> Dict[Kind, List[str]]: - global_config_files = [ - os.path.join(path, CONFIG_BASENAME) for path in appdirs.site_config_dirs("pip") - ] - - site_config_file = os.path.join(sys.prefix, CONFIG_BASENAME) - legacy_config_file = os.path.join( - os.path.expanduser("~"), - "pip" if WINDOWS else ".pip", - CONFIG_BASENAME, - ) - new_config_file = os.path.join(appdirs.user_config_dir("pip"), CONFIG_BASENAME) - return { - kinds.GLOBAL: global_config_files, - kinds.SITE: [site_config_file], - kinds.USER: [legacy_config_file, new_config_file], - } - - -class Configuration: - """Handles management of configuration. - - Provides an interface to accessing and managing configuration files. - - This class converts provides an API that takes "section.key-name" style - keys and stores the value associated with it as "key-name" under the - section "section". - - This allows for a clean interface wherein the both the section and the - key-name are preserved in an easy to manage form in the configuration files - and the data stored is also nice. - """ - - def __init__(self, isolated: bool, load_only: Optional[Kind] = None) -> None: - super().__init__() - - if load_only is not None and load_only not in VALID_LOAD_ONLY: - raise ConfigurationError( - "Got invalid value for load_only - should be one of {}".format( - ", ".join(map(repr, VALID_LOAD_ONLY)) - ) - ) - self.isolated = isolated - self.load_only = load_only - - # Because we keep track of where we got the data from - self._parsers: Dict[Kind, List[Tuple[str, RawConfigParser]]] = { - variant: [] for variant in OVERRIDE_ORDER - } - self._config: Dict[Kind, Dict[str, Any]] = { - variant: {} for variant in OVERRIDE_ORDER - } - self._modified_parsers: List[Tuple[str, RawConfigParser]] = [] - - def load(self) -> None: - """Loads configuration from configuration files and environment""" - self._load_config_files() - if not self.isolated: - self._load_environment_vars() - - def get_file_to_edit(self) -> Optional[str]: - """Returns the file with highest priority in configuration""" - assert self.load_only is not None, "Need to be specified a file to be editing" - - try: - return self._get_parser_to_modify()[0] - except IndexError: - return None - - def items(self) -> Iterable[Tuple[str, Any]]: - """Returns key-value pairs like dict.items() representing the loaded - configuration - """ - return self._dictionary.items() - - def get_value(self, key: str) -> Any: - """Get a value from the configuration.""" - orig_key = key - key = _normalize_name(key) - try: - return self._dictionary[key] - except KeyError: - # disassembling triggers a more useful error message than simply - # "No such key" in the case that the key isn't in the form command.option - _disassemble_key(key) - raise ConfigurationError(f"No such key - {orig_key}") - - def set_value(self, key: str, value: Any) -> None: - """Modify a value in the configuration.""" - key = _normalize_name(key) - self._ensure_have_load_only() - - assert self.load_only - fname, parser = self._get_parser_to_modify() - - if parser is not None: - section, name = _disassemble_key(key) - - # Modify the parser and the configuration - if not parser.has_section(section): - parser.add_section(section) - parser.set(section, name, value) - - self._config[self.load_only][key] = value - self._mark_as_modified(fname, parser) - - def unset_value(self, key: str) -> None: - """Unset a value in the configuration.""" - orig_key = key - key = _normalize_name(key) - self._ensure_have_load_only() - - assert self.load_only - if key not in self._config[self.load_only]: - raise ConfigurationError(f"No such key - {orig_key}") - - fname, parser = self._get_parser_to_modify() - - if parser is not None: - section, name = _disassemble_key(key) - if not ( - parser.has_section(section) and parser.remove_option(section, name) - ): - # The option was not removed. - raise ConfigurationError( - "Fatal Internal error [id=1]. Please report as a bug." - ) - - # The section may be empty after the option was removed. - if not parser.items(section): - parser.remove_section(section) - self._mark_as_modified(fname, parser) - - del self._config[self.load_only][key] - - def save(self) -> None: - """Save the current in-memory state.""" - self._ensure_have_load_only() - - for fname, parser in self._modified_parsers: - logger.info("Writing to %s", fname) - - # Ensure directory exists. - ensure_dir(os.path.dirname(fname)) - - # Ensure directory's permission(need to be writeable) - try: - with open(fname, "w") as f: - parser.write(f) - except OSError as error: - raise ConfigurationError( - f"An error occurred while writing to the configuration file " - f"{fname}: {error}" - ) - - # - # Private routines - # - - def _ensure_have_load_only(self) -> None: - if self.load_only is None: - raise ConfigurationError("Needed a specific file to be modifying.") - logger.debug("Will be working with %s variant only", self.load_only) - - @property - def _dictionary(self) -> Dict[str, Any]: - """A dictionary representing the loaded configuration.""" - # NOTE: Dictionaries are not populated if not loaded. So, conditionals - # are not needed here. - retval = {} - - for variant in OVERRIDE_ORDER: - retval.update(self._config[variant]) - - return retval - - def _load_config_files(self) -> None: - """Loads configuration from configuration files""" - config_files = dict(self.iter_config_files()) - if config_files[kinds.ENV][0:1] == [os.devnull]: - logger.debug( - "Skipping loading configuration files due to " - "environment's PIP_CONFIG_FILE being os.devnull" - ) - return - - for variant, files in config_files.items(): - for fname in files: - # If there's specific variant set in `load_only`, load only - # that variant, not the others. - if self.load_only is not None and variant != self.load_only: - logger.debug("Skipping file '%s' (variant: %s)", fname, variant) - continue - - parser = self._load_file(variant, fname) - - # Keeping track of the parsers used - self._parsers[variant].append((fname, parser)) - - def _load_file(self, variant: Kind, fname: str) -> RawConfigParser: - logger.verbose("For variant '%s', will try loading '%s'", variant, fname) - parser = self._construct_parser(fname) - - for section in parser.sections(): - items = parser.items(section) - self._config[variant].update(self._normalized_keys(section, items)) - - return parser - - def _construct_parser(self, fname: str) -> RawConfigParser: - parser = configparser.RawConfigParser() - # If there is no such file, don't bother reading it but create the - # parser anyway, to hold the data. - # Doing this is useful when modifying and saving files, where we don't - # need to construct a parser. - if os.path.exists(fname): - locale_encoding = locale.getpreferredencoding(False) - try: - parser.read(fname, encoding=locale_encoding) - except UnicodeDecodeError: - # See https://github.com/pypa/pip/issues/4963 - raise ConfigurationFileCouldNotBeLoaded( - reason=f"contains invalid {locale_encoding} characters", - fname=fname, - ) - except configparser.Error as error: - # See https://github.com/pypa/pip/issues/4893 - raise ConfigurationFileCouldNotBeLoaded(error=error) - return parser - - def _load_environment_vars(self) -> None: - """Loads configuration from environment variables""" - self._config[kinds.ENV_VAR].update( - self._normalized_keys(":env:", self.get_environ_vars()) - ) - - def _normalized_keys( - self, section: str, items: Iterable[Tuple[str, Any]] - ) -> Dict[str, Any]: - """Normalizes items to construct a dictionary with normalized keys. - - This routine is where the names become keys and are made the same - regardless of source - configuration files or environment. - """ - normalized = {} - for name, val in items: - key = section + "." + _normalize_name(name) - normalized[key] = val - return normalized - - def get_environ_vars(self) -> Iterable[Tuple[str, str]]: - """Returns a generator with all environmental vars with prefix PIP_""" - for key, val in os.environ.items(): - if key.startswith("PIP_"): - name = key[4:].lower() - if name not in ENV_NAMES_IGNORED: - yield name, val - - # XXX: This is patched in the tests. - def iter_config_files(self) -> Iterable[Tuple[Kind, List[str]]]: - """Yields variant and configuration files associated with it. - - This should be treated like items of a dictionary. - """ - # SMELL: Move the conditions out of this function - - # environment variables have the lowest priority - config_file = os.environ.get("PIP_CONFIG_FILE", None) - if config_file is not None: - yield kinds.ENV, [config_file] - else: - yield kinds.ENV, [] - - config_files = get_configuration_files() - - # at the base we have any global configuration - yield kinds.GLOBAL, config_files[kinds.GLOBAL] - - # per-user configuration next - should_load_user_config = not self.isolated and not ( - config_file and os.path.exists(config_file) - ) - if should_load_user_config: - # The legacy config file is overridden by the new config file - yield kinds.USER, config_files[kinds.USER] - - # finally virtualenv configuration first trumping others - yield kinds.SITE, config_files[kinds.SITE] - - def get_values_in_config(self, variant: Kind) -> Dict[str, Any]: - """Get values present in a config file""" - return self._config[variant] - - def _get_parser_to_modify(self) -> Tuple[str, RawConfigParser]: - # Determine which parser to modify - assert self.load_only - parsers = self._parsers[self.load_only] - if not parsers: - # This should not happen if everything works correctly. - raise ConfigurationError( - "Fatal Internal error [id=2]. Please report as a bug." - ) - - # Use the highest priority parser. - return parsers[-1] - - # XXX: This is patched in the tests. - def _mark_as_modified(self, fname: str, parser: RawConfigParser) -> None: - file_parser_tuple = (fname, parser) - if file_parser_tuple not in self._modified_parsers: - self._modified_parsers.append(file_parser_tuple) - - def __repr__(self) -> str: - return f"{self.__class__.__name__}({self._dictionary!r})" diff --git a/venv/lib/python3.11/site-packages/pip/_internal/distributions/__init__.py b/venv/lib/python3.11/site-packages/pip/_internal/distributions/__init__.py deleted file mode 100644 index 9a89a83..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/distributions/__init__.py +++ /dev/null @@ -1,21 +0,0 @@ -from pip._internal.distributions.base import AbstractDistribution -from pip._internal.distributions.sdist import SourceDistribution -from pip._internal.distributions.wheel import WheelDistribution -from pip._internal.req.req_install import InstallRequirement - - -def make_distribution_for_install_requirement( - install_req: InstallRequirement, -) -> AbstractDistribution: - """Returns a Distribution for the given InstallRequirement""" - # Editable requirements will always be source distributions. They use the - # legacy logic until we create a modern standard for them. - if install_req.editable: - return SourceDistribution(install_req) - - # If it's a wheel, it's a WheelDistribution - if install_req.is_wheel: - return WheelDistribution(install_req) - - # Otherwise, a SourceDistribution - return SourceDistribution(install_req) diff --git a/venv/lib/python3.11/site-packages/pip/_internal/distributions/__pycache__/__init__.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/distributions/__pycache__/__init__.cpython-311.pyc deleted file mode 100644 index 7fe347b..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/distributions/__pycache__/__init__.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/distributions/__pycache__/base.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/distributions/__pycache__/base.cpython-311.pyc deleted file mode 100644 index eb7bc28..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/distributions/__pycache__/base.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/distributions/__pycache__/installed.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/distributions/__pycache__/installed.cpython-311.pyc deleted file mode 100644 index 6400c02..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/distributions/__pycache__/installed.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/distributions/__pycache__/sdist.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/distributions/__pycache__/sdist.cpython-311.pyc deleted file mode 100644 index 92f0bdf..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/distributions/__pycache__/sdist.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/distributions/__pycache__/wheel.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/distributions/__pycache__/wheel.cpython-311.pyc deleted file mode 100644 index afac423..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/distributions/__pycache__/wheel.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/distributions/base.py b/venv/lib/python3.11/site-packages/pip/_internal/distributions/base.py deleted file mode 100644 index 6fb0d7b..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/distributions/base.py +++ /dev/null @@ -1,51 +0,0 @@ -import abc -from typing import Optional - -from pip._internal.index.package_finder import PackageFinder -from pip._internal.metadata.base import BaseDistribution -from pip._internal.req import InstallRequirement - - -class AbstractDistribution(metaclass=abc.ABCMeta): - """A base class for handling installable artifacts. - - The requirements for anything installable are as follows: - - - we must be able to determine the requirement name - (or we can't correctly handle the non-upgrade case). - - - for packages with setup requirements, we must also be able - to determine their requirements without installing additional - packages (for the same reason as run-time dependencies) - - - we must be able to create a Distribution object exposing the - above metadata. - - - if we need to do work in the build tracker, we must be able to generate a unique - string to identify the requirement in the build tracker. - """ - - def __init__(self, req: InstallRequirement) -> None: - super().__init__() - self.req = req - - @abc.abstractproperty - def build_tracker_id(self) -> Optional[str]: - """A string that uniquely identifies this requirement to the build tracker. - - If None, then this dist has no work to do in the build tracker, and - ``.prepare_distribution_metadata()`` will not be called.""" - raise NotImplementedError() - - @abc.abstractmethod - def get_metadata_distribution(self) -> BaseDistribution: - raise NotImplementedError() - - @abc.abstractmethod - def prepare_distribution_metadata( - self, - finder: PackageFinder, - build_isolation: bool, - check_build_deps: bool, - ) -> None: - raise NotImplementedError() diff --git a/venv/lib/python3.11/site-packages/pip/_internal/distributions/installed.py b/venv/lib/python3.11/site-packages/pip/_internal/distributions/installed.py deleted file mode 100644 index ab8d53b..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/distributions/installed.py +++ /dev/null @@ -1,29 +0,0 @@ -from typing import Optional - -from pip._internal.distributions.base import AbstractDistribution -from pip._internal.index.package_finder import PackageFinder -from pip._internal.metadata import BaseDistribution - - -class InstalledDistribution(AbstractDistribution): - """Represents an installed package. - - This does not need any preparation as the required information has already - been computed. - """ - - @property - def build_tracker_id(self) -> Optional[str]: - return None - - def get_metadata_distribution(self) -> BaseDistribution: - assert self.req.satisfied_by is not None, "not actually installed" - return self.req.satisfied_by - - def prepare_distribution_metadata( - self, - finder: PackageFinder, - build_isolation: bool, - check_build_deps: bool, - ) -> None: - pass diff --git a/venv/lib/python3.11/site-packages/pip/_internal/distributions/sdist.py b/venv/lib/python3.11/site-packages/pip/_internal/distributions/sdist.py deleted file mode 100644 index 15ff42b..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/distributions/sdist.py +++ /dev/null @@ -1,156 +0,0 @@ -import logging -from typing import Iterable, Optional, Set, Tuple - -from pip._internal.build_env import BuildEnvironment -from pip._internal.distributions.base import AbstractDistribution -from pip._internal.exceptions import InstallationError -from pip._internal.index.package_finder import PackageFinder -from pip._internal.metadata import BaseDistribution -from pip._internal.utils.subprocess import runner_with_spinner_message - -logger = logging.getLogger(__name__) - - -class SourceDistribution(AbstractDistribution): - """Represents a source distribution. - - The preparation step for these needs metadata for the packages to be - generated, either using PEP 517 or using the legacy `setup.py egg_info`. - """ - - @property - def build_tracker_id(self) -> Optional[str]: - """Identify this requirement uniquely by its link.""" - assert self.req.link - return self.req.link.url_without_fragment - - def get_metadata_distribution(self) -> BaseDistribution: - return self.req.get_dist() - - def prepare_distribution_metadata( - self, - finder: PackageFinder, - build_isolation: bool, - check_build_deps: bool, - ) -> None: - # Load pyproject.toml, to determine whether PEP 517 is to be used - self.req.load_pyproject_toml() - - # Set up the build isolation, if this requirement should be isolated - should_isolate = self.req.use_pep517 and build_isolation - if should_isolate: - # Setup an isolated environment and install the build backend static - # requirements in it. - self._prepare_build_backend(finder) - # Check that if the requirement is editable, it either supports PEP 660 or - # has a setup.py or a setup.cfg. This cannot be done earlier because we need - # to setup the build backend to verify it supports build_editable, nor can - # it be done later, because we want to avoid installing build requirements - # needlessly. Doing it here also works around setuptools generating - # UNKNOWN.egg-info when running get_requires_for_build_wheel on a directory - # without setup.py nor setup.cfg. - self.req.isolated_editable_sanity_check() - # Install the dynamic build requirements. - self._install_build_reqs(finder) - # Check if the current environment provides build dependencies - should_check_deps = self.req.use_pep517 and check_build_deps - if should_check_deps: - pyproject_requires = self.req.pyproject_requires - assert pyproject_requires is not None - conflicting, missing = self.req.build_env.check_requirements( - pyproject_requires - ) - if conflicting: - self._raise_conflicts("the backend dependencies", conflicting) - if missing: - self._raise_missing_reqs(missing) - self.req.prepare_metadata() - - def _prepare_build_backend(self, finder: PackageFinder) -> None: - # Isolate in a BuildEnvironment and install the build-time - # requirements. - pyproject_requires = self.req.pyproject_requires - assert pyproject_requires is not None - - self.req.build_env = BuildEnvironment() - self.req.build_env.install_requirements( - finder, pyproject_requires, "overlay", kind="build dependencies" - ) - conflicting, missing = self.req.build_env.check_requirements( - self.req.requirements_to_check - ) - if conflicting: - self._raise_conflicts("PEP 517/518 supported requirements", conflicting) - if missing: - logger.warning( - "Missing build requirements in pyproject.toml for %s.", - self.req, - ) - logger.warning( - "The project does not specify a build backend, and " - "pip cannot fall back to setuptools without %s.", - " and ".join(map(repr, sorted(missing))), - ) - - def _get_build_requires_wheel(self) -> Iterable[str]: - with self.req.build_env: - runner = runner_with_spinner_message("Getting requirements to build wheel") - backend = self.req.pep517_backend - assert backend is not None - with backend.subprocess_runner(runner): - return backend.get_requires_for_build_wheel() - - def _get_build_requires_editable(self) -> Iterable[str]: - with self.req.build_env: - runner = runner_with_spinner_message( - "Getting requirements to build editable" - ) - backend = self.req.pep517_backend - assert backend is not None - with backend.subprocess_runner(runner): - return backend.get_requires_for_build_editable() - - def _install_build_reqs(self, finder: PackageFinder) -> None: - # Install any extra build dependencies that the backend requests. - # This must be done in a second pass, as the pyproject.toml - # dependencies must be installed before we can call the backend. - if ( - self.req.editable - and self.req.permit_editable_wheels - and self.req.supports_pyproject_editable() - ): - build_reqs = self._get_build_requires_editable() - else: - build_reqs = self._get_build_requires_wheel() - conflicting, missing = self.req.build_env.check_requirements(build_reqs) - if conflicting: - self._raise_conflicts("the backend dependencies", conflicting) - self.req.build_env.install_requirements( - finder, missing, "normal", kind="backend dependencies" - ) - - def _raise_conflicts( - self, conflicting_with: str, conflicting_reqs: Set[Tuple[str, str]] - ) -> None: - format_string = ( - "Some build dependencies for {requirement} " - "conflict with {conflicting_with}: {description}." - ) - error_message = format_string.format( - requirement=self.req, - conflicting_with=conflicting_with, - description=", ".join( - f"{installed} is incompatible with {wanted}" - for installed, wanted in sorted(conflicting_reqs) - ), - ) - raise InstallationError(error_message) - - def _raise_missing_reqs(self, missing: Set[str]) -> None: - format_string = ( - "Some build dependencies for {requirement} are missing: {missing}." - ) - error_message = format_string.format( - requirement=self.req, missing=", ".join(map(repr, sorted(missing))) - ) - raise InstallationError(error_message) diff --git a/venv/lib/python3.11/site-packages/pip/_internal/distributions/wheel.py b/venv/lib/python3.11/site-packages/pip/_internal/distributions/wheel.py deleted file mode 100644 index eb16e25..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/distributions/wheel.py +++ /dev/null @@ -1,40 +0,0 @@ -from typing import Optional - -from pip._vendor.packaging.utils import canonicalize_name - -from pip._internal.distributions.base import AbstractDistribution -from pip._internal.index.package_finder import PackageFinder -from pip._internal.metadata import ( - BaseDistribution, - FilesystemWheel, - get_wheel_distribution, -) - - -class WheelDistribution(AbstractDistribution): - """Represents a wheel distribution. - - This does not need any preparation as wheels can be directly unpacked. - """ - - @property - def build_tracker_id(self) -> Optional[str]: - return None - - def get_metadata_distribution(self) -> BaseDistribution: - """Loads the metadata from the wheel file into memory and returns a - Distribution that uses it, not relying on the wheel file or - requirement. - """ - assert self.req.local_file_path, "Set as part of preparation during download" - assert self.req.name, "Wheels are never unnamed" - wheel = FilesystemWheel(self.req.local_file_path) - return get_wheel_distribution(wheel, canonicalize_name(self.req.name)) - - def prepare_distribution_metadata( - self, - finder: PackageFinder, - build_isolation: bool, - check_build_deps: bool, - ) -> None: - pass diff --git a/venv/lib/python3.11/site-packages/pip/_internal/exceptions.py b/venv/lib/python3.11/site-packages/pip/_internal/exceptions.py deleted file mode 100644 index d95fe44..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/exceptions.py +++ /dev/null @@ -1,733 +0,0 @@ -"""Exceptions used throughout package. - -This module MUST NOT try to import from anything within `pip._internal` to -operate. This is expected to be importable from any/all files within the -subpackage and, thus, should not depend on them. -""" - -import configparser -import contextlib -import locale -import logging -import pathlib -import re -import sys -from itertools import chain, groupby, repeat -from typing import TYPE_CHECKING, Dict, Iterator, List, Optional, Union - -from pip._vendor.requests.models import Request, Response -from pip._vendor.rich.console import Console, ConsoleOptions, RenderResult -from pip._vendor.rich.markup import escape -from pip._vendor.rich.text import Text - -if TYPE_CHECKING: - from hashlib import _Hash - from typing import Literal - - from pip._internal.metadata import BaseDistribution - from pip._internal.req.req_install import InstallRequirement - -logger = logging.getLogger(__name__) - - -# -# Scaffolding -# -def _is_kebab_case(s: str) -> bool: - return re.match(r"^[a-z]+(-[a-z]+)*$", s) is not None - - -def _prefix_with_indent( - s: Union[Text, str], - console: Console, - *, - prefix: str, - indent: str, -) -> Text: - if isinstance(s, Text): - text = s - else: - text = console.render_str(s) - - return console.render_str(prefix, overflow="ignore") + console.render_str( - f"\n{indent}", overflow="ignore" - ).join(text.split(allow_blank=True)) - - -class PipError(Exception): - """The base pip error.""" - - -class DiagnosticPipError(PipError): - """An error, that presents diagnostic information to the user. - - This contains a bunch of logic, to enable pretty presentation of our error - messages. Each error gets a unique reference. Each error can also include - additional context, a hint and/or a note -- which are presented with the - main error message in a consistent style. - - This is adapted from the error output styling in `sphinx-theme-builder`. - """ - - reference: str - - def __init__( - self, - *, - kind: 'Literal["error", "warning"]' = "error", - reference: Optional[str] = None, - message: Union[str, Text], - context: Optional[Union[str, Text]], - hint_stmt: Optional[Union[str, Text]], - note_stmt: Optional[Union[str, Text]] = None, - link: Optional[str] = None, - ) -> None: - # Ensure a proper reference is provided. - if reference is None: - assert hasattr(self, "reference"), "error reference not provided!" - reference = self.reference - assert _is_kebab_case(reference), "error reference must be kebab-case!" - - self.kind = kind - self.reference = reference - - self.message = message - self.context = context - - self.note_stmt = note_stmt - self.hint_stmt = hint_stmt - - self.link = link - - super().__init__(f"<{self.__class__.__name__}: {self.reference}>") - - def __repr__(self) -> str: - return ( - f"<{self.__class__.__name__}(" - f"reference={self.reference!r}, " - f"message={self.message!r}, " - f"context={self.context!r}, " - f"note_stmt={self.note_stmt!r}, " - f"hint_stmt={self.hint_stmt!r}" - ")>" - ) - - def __rich_console__( - self, - console: Console, - options: ConsoleOptions, - ) -> RenderResult: - colour = "red" if self.kind == "error" else "yellow" - - yield f"[{colour} bold]{self.kind}[/]: [bold]{self.reference}[/]" - yield "" - - if not options.ascii_only: - # Present the main message, with relevant context indented. - if self.context is not None: - yield _prefix_with_indent( - self.message, - console, - prefix=f"[{colour}]×[/] ", - indent=f"[{colour}]│[/] ", - ) - yield _prefix_with_indent( - self.context, - console, - prefix=f"[{colour}]╰─>[/] ", - indent=f"[{colour}] [/] ", - ) - else: - yield _prefix_with_indent( - self.message, - console, - prefix="[red]×[/] ", - indent=" ", - ) - else: - yield self.message - if self.context is not None: - yield "" - yield self.context - - if self.note_stmt is not None or self.hint_stmt is not None: - yield "" - - if self.note_stmt is not None: - yield _prefix_with_indent( - self.note_stmt, - console, - prefix="[magenta bold]note[/]: ", - indent=" ", - ) - if self.hint_stmt is not None: - yield _prefix_with_indent( - self.hint_stmt, - console, - prefix="[cyan bold]hint[/]: ", - indent=" ", - ) - - if self.link is not None: - yield "" - yield f"Link: {self.link}" - - -# -# Actual Errors -# -class ConfigurationError(PipError): - """General exception in configuration""" - - -class InstallationError(PipError): - """General exception during installation""" - - -class UninstallationError(PipError): - """General exception during uninstallation""" - - -class MissingPyProjectBuildRequires(DiagnosticPipError): - """Raised when pyproject.toml has `build-system`, but no `build-system.requires`.""" - - reference = "missing-pyproject-build-system-requires" - - def __init__(self, *, package: str) -> None: - super().__init__( - message=f"Can not process {escape(package)}", - context=Text( - "This package has an invalid pyproject.toml file.\n" - "The [build-system] table is missing the mandatory `requires` key." - ), - note_stmt="This is an issue with the package mentioned above, not pip.", - hint_stmt=Text("See PEP 518 for the detailed specification."), - ) - - -class InvalidPyProjectBuildRequires(DiagnosticPipError): - """Raised when pyproject.toml an invalid `build-system.requires`.""" - - reference = "invalid-pyproject-build-system-requires" - - def __init__(self, *, package: str, reason: str) -> None: - super().__init__( - message=f"Can not process {escape(package)}", - context=Text( - "This package has an invalid `build-system.requires` key in " - f"pyproject.toml.\n{reason}" - ), - note_stmt="This is an issue with the package mentioned above, not pip.", - hint_stmt=Text("See PEP 518 for the detailed specification."), - ) - - -class NoneMetadataError(PipError): - """Raised when accessing a Distribution's "METADATA" or "PKG-INFO". - - This signifies an inconsistency, when the Distribution claims to have - the metadata file (if not, raise ``FileNotFoundError`` instead), but is - not actually able to produce its content. This may be due to permission - errors. - """ - - def __init__( - self, - dist: "BaseDistribution", - metadata_name: str, - ) -> None: - """ - :param dist: A Distribution object. - :param metadata_name: The name of the metadata being accessed - (can be "METADATA" or "PKG-INFO"). - """ - self.dist = dist - self.metadata_name = metadata_name - - def __str__(self) -> str: - # Use `dist` in the error message because its stringification - # includes more information, like the version and location. - return "None {} metadata found for distribution: {}".format( - self.metadata_name, - self.dist, - ) - - -class UserInstallationInvalid(InstallationError): - """A --user install is requested on an environment without user site.""" - - def __str__(self) -> str: - return "User base directory is not specified" - - -class InvalidSchemeCombination(InstallationError): - def __str__(self) -> str: - before = ", ".join(str(a) for a in self.args[:-1]) - return f"Cannot set {before} and {self.args[-1]} together" - - -class DistributionNotFound(InstallationError): - """Raised when a distribution cannot be found to satisfy a requirement""" - - -class RequirementsFileParseError(InstallationError): - """Raised when a general error occurs parsing a requirements file line.""" - - -class BestVersionAlreadyInstalled(PipError): - """Raised when the most up-to-date version of a package is already - installed.""" - - -class BadCommand(PipError): - """Raised when virtualenv or a command is not found""" - - -class CommandError(PipError): - """Raised when there is an error in command-line arguments""" - - -class PreviousBuildDirError(PipError): - """Raised when there's a previous conflicting build directory""" - - -class NetworkConnectionError(PipError): - """HTTP connection error""" - - def __init__( - self, - error_msg: str, - response: Optional[Response] = None, - request: Optional[Request] = None, - ) -> None: - """ - Initialize NetworkConnectionError with `request` and `response` - objects. - """ - self.response = response - self.request = request - self.error_msg = error_msg - if ( - self.response is not None - and not self.request - and hasattr(response, "request") - ): - self.request = self.response.request - super().__init__(error_msg, response, request) - - def __str__(self) -> str: - return str(self.error_msg) - - -class InvalidWheelFilename(InstallationError): - """Invalid wheel filename.""" - - -class UnsupportedWheel(InstallationError): - """Unsupported wheel.""" - - -class InvalidWheel(InstallationError): - """Invalid (e.g. corrupt) wheel.""" - - def __init__(self, location: str, name: str): - self.location = location - self.name = name - - def __str__(self) -> str: - return f"Wheel '{self.name}' located at {self.location} is invalid." - - -class MetadataInconsistent(InstallationError): - """Built metadata contains inconsistent information. - - This is raised when the metadata contains values (e.g. name and version) - that do not match the information previously obtained from sdist filename, - user-supplied ``#egg=`` value, or an install requirement name. - """ - - def __init__( - self, ireq: "InstallRequirement", field: str, f_val: str, m_val: str - ) -> None: - self.ireq = ireq - self.field = field - self.f_val = f_val - self.m_val = m_val - - def __str__(self) -> str: - return ( - f"Requested {self.ireq} has inconsistent {self.field}: " - f"expected {self.f_val!r}, but metadata has {self.m_val!r}" - ) - - -class InstallationSubprocessError(DiagnosticPipError, InstallationError): - """A subprocess call failed.""" - - reference = "subprocess-exited-with-error" - - def __init__( - self, - *, - command_description: str, - exit_code: int, - output_lines: Optional[List[str]], - ) -> None: - if output_lines is None: - output_prompt = Text("See above for output.") - else: - output_prompt = ( - Text.from_markup(f"[red][{len(output_lines)} lines of output][/]\n") - + Text("".join(output_lines)) - + Text.from_markup(R"[red]\[end of output][/]") - ) - - super().__init__( - message=( - f"[green]{escape(command_description)}[/] did not run successfully.\n" - f"exit code: {exit_code}" - ), - context=output_prompt, - hint_stmt=None, - note_stmt=( - "This error originates from a subprocess, and is likely not a " - "problem with pip." - ), - ) - - self.command_description = command_description - self.exit_code = exit_code - - def __str__(self) -> str: - return f"{self.command_description} exited with {self.exit_code}" - - -class MetadataGenerationFailed(InstallationSubprocessError, InstallationError): - reference = "metadata-generation-failed" - - def __init__( - self, - *, - package_details: str, - ) -> None: - super(InstallationSubprocessError, self).__init__( - message="Encountered error while generating package metadata.", - context=escape(package_details), - hint_stmt="See above for details.", - note_stmt="This is an issue with the package mentioned above, not pip.", - ) - - def __str__(self) -> str: - return "metadata generation failed" - - -class HashErrors(InstallationError): - """Multiple HashError instances rolled into one for reporting""" - - def __init__(self) -> None: - self.errors: List["HashError"] = [] - - def append(self, error: "HashError") -> None: - self.errors.append(error) - - def __str__(self) -> str: - lines = [] - self.errors.sort(key=lambda e: e.order) - for cls, errors_of_cls in groupby(self.errors, lambda e: e.__class__): - lines.append(cls.head) - lines.extend(e.body() for e in errors_of_cls) - if lines: - return "\n".join(lines) - return "" - - def __bool__(self) -> bool: - return bool(self.errors) - - -class HashError(InstallationError): - """ - A failure to verify a package against known-good hashes - - :cvar order: An int sorting hash exception classes by difficulty of - recovery (lower being harder), so the user doesn't bother fretting - about unpinned packages when he has deeper issues, like VCS - dependencies, to deal with. Also keeps error reports in a - deterministic order. - :cvar head: A section heading for display above potentially many - exceptions of this kind - :ivar req: The InstallRequirement that triggered this error. This is - pasted on after the exception is instantiated, because it's not - typically available earlier. - - """ - - req: Optional["InstallRequirement"] = None - head = "" - order: int = -1 - - def body(self) -> str: - """Return a summary of me for display under the heading. - - This default implementation simply prints a description of the - triggering requirement. - - :param req: The InstallRequirement that provoked this error, with - its link already populated by the resolver's _populate_link(). - - """ - return f" {self._requirement_name()}" - - def __str__(self) -> str: - return f"{self.head}\n{self.body()}" - - def _requirement_name(self) -> str: - """Return a description of the requirement that triggered me. - - This default implementation returns long description of the req, with - line numbers - - """ - return str(self.req) if self.req else "unknown package" - - -class VcsHashUnsupported(HashError): - """A hash was provided for a version-control-system-based requirement, but - we don't have a method for hashing those.""" - - order = 0 - head = ( - "Can't verify hashes for these requirements because we don't " - "have a way to hash version control repositories:" - ) - - -class DirectoryUrlHashUnsupported(HashError): - """A hash was provided for a version-control-system-based requirement, but - we don't have a method for hashing those.""" - - order = 1 - head = ( - "Can't verify hashes for these file:// requirements because they " - "point to directories:" - ) - - -class HashMissing(HashError): - """A hash was needed for a requirement but is absent.""" - - order = 2 - head = ( - "Hashes are required in --require-hashes mode, but they are " - "missing from some requirements. Here is a list of those " - "requirements along with the hashes their downloaded archives " - "actually had. Add lines like these to your requirements files to " - "prevent tampering. (If you did not enable --require-hashes " - "manually, note that it turns on automatically when any package " - "has a hash.)" - ) - - def __init__(self, gotten_hash: str) -> None: - """ - :param gotten_hash: The hash of the (possibly malicious) archive we - just downloaded - """ - self.gotten_hash = gotten_hash - - def body(self) -> str: - # Dodge circular import. - from pip._internal.utils.hashes import FAVORITE_HASH - - package = None - if self.req: - # In the case of URL-based requirements, display the original URL - # seen in the requirements file rather than the package name, - # so the output can be directly copied into the requirements file. - package = ( - self.req.original_link - if self.req.is_direct - # In case someone feeds something downright stupid - # to InstallRequirement's constructor. - else getattr(self.req, "req", None) - ) - return " {} --hash={}:{}".format( - package or "unknown package", FAVORITE_HASH, self.gotten_hash - ) - - -class HashUnpinned(HashError): - """A requirement had a hash specified but was not pinned to a specific - version.""" - - order = 3 - head = ( - "In --require-hashes mode, all requirements must have their " - "versions pinned with ==. These do not:" - ) - - -class HashMismatch(HashError): - """ - Distribution file hash values don't match. - - :ivar package_name: The name of the package that triggered the hash - mismatch. Feel free to write to this after the exception is raise to - improve its error message. - - """ - - order = 4 - head = ( - "THESE PACKAGES DO NOT MATCH THE HASHES FROM THE REQUIREMENTS " - "FILE. If you have updated the package versions, please update " - "the hashes. Otherwise, examine the package contents carefully; " - "someone may have tampered with them." - ) - - def __init__(self, allowed: Dict[str, List[str]], gots: Dict[str, "_Hash"]) -> None: - """ - :param allowed: A dict of algorithm names pointing to lists of allowed - hex digests - :param gots: A dict of algorithm names pointing to hashes we - actually got from the files under suspicion - """ - self.allowed = allowed - self.gots = gots - - def body(self) -> str: - return " {}:\n{}".format(self._requirement_name(), self._hash_comparison()) - - def _hash_comparison(self) -> str: - """ - Return a comparison of actual and expected hash values. - - Example:: - - Expected sha256 abcdeabcdeabcdeabcdeabcdeabcdeabcdeabcdeabcde - or 123451234512345123451234512345123451234512345 - Got bcdefbcdefbcdefbcdefbcdefbcdefbcdefbcdefbcdef - - """ - - def hash_then_or(hash_name: str) -> "chain[str]": - # For now, all the decent hashes have 6-char names, so we can get - # away with hard-coding space literals. - return chain([hash_name], repeat(" or")) - - lines: List[str] = [] - for hash_name, expecteds in self.allowed.items(): - prefix = hash_then_or(hash_name) - lines.extend( - (" Expected {} {}".format(next(prefix), e)) for e in expecteds - ) - lines.append( - " Got {}\n".format(self.gots[hash_name].hexdigest()) - ) - return "\n".join(lines) - - -class UnsupportedPythonVersion(InstallationError): - """Unsupported python version according to Requires-Python package - metadata.""" - - -class ConfigurationFileCouldNotBeLoaded(ConfigurationError): - """When there are errors while loading a configuration file""" - - def __init__( - self, - reason: str = "could not be loaded", - fname: Optional[str] = None, - error: Optional[configparser.Error] = None, - ) -> None: - super().__init__(error) - self.reason = reason - self.fname = fname - self.error = error - - def __str__(self) -> str: - if self.fname is not None: - message_part = f" in {self.fname}." - else: - assert self.error is not None - message_part = f".\n{self.error}\n" - return f"Configuration file {self.reason}{message_part}" - - -_DEFAULT_EXTERNALLY_MANAGED_ERROR = f"""\ -The Python environment under {sys.prefix} is managed externally, and may not be -manipulated by the user. Please use specific tooling from the distributor of -the Python installation to interact with this environment instead. -""" - - -class ExternallyManagedEnvironment(DiagnosticPipError): - """The current environment is externally managed. - - This is raised when the current environment is externally managed, as - defined by `PEP 668`_. The ``EXTERNALLY-MANAGED`` configuration is checked - and displayed when the error is bubbled up to the user. - - :param error: The error message read from ``EXTERNALLY-MANAGED``. - """ - - reference = "externally-managed-environment" - - def __init__(self, error: Optional[str]) -> None: - if error is None: - context = Text(_DEFAULT_EXTERNALLY_MANAGED_ERROR) - else: - context = Text(error) - super().__init__( - message="This environment is externally managed", - context=context, - note_stmt=( - "If you believe this is a mistake, please contact your " - "Python installation or OS distribution provider. " - "You can override this, at the risk of breaking your Python " - "installation or OS, by passing --break-system-packages." - ), - hint_stmt=Text("See PEP 668 for the detailed specification."), - ) - - @staticmethod - def _iter_externally_managed_error_keys() -> Iterator[str]: - # LC_MESSAGES is in POSIX, but not the C standard. The most common - # platform that does not implement this category is Windows, where - # using other categories for console message localization is equally - # unreliable, so we fall back to the locale-less vendor message. This - # can always be re-evaluated when a vendor proposes a new alternative. - try: - category = locale.LC_MESSAGES - except AttributeError: - lang: Optional[str] = None - else: - lang, _ = locale.getlocale(category) - if lang is not None: - yield f"Error-{lang}" - for sep in ("-", "_"): - before, found, _ = lang.partition(sep) - if not found: - continue - yield f"Error-{before}" - yield "Error" - - @classmethod - def from_config( - cls, - config: Union[pathlib.Path, str], - ) -> "ExternallyManagedEnvironment": - parser = configparser.ConfigParser(interpolation=None) - try: - parser.read(config, encoding="utf-8") - section = parser["externally-managed"] - for key in cls._iter_externally_managed_error_keys(): - with contextlib.suppress(KeyError): - return cls(section[key]) - except KeyError: - pass - except (OSError, UnicodeDecodeError, configparser.ParsingError): - from pip._internal.utils._log import VERBOSE - - exc_info = logger.isEnabledFor(VERBOSE) - logger.warning("Failed to read %s", config, exc_info=exc_info) - return cls(None) diff --git a/venv/lib/python3.11/site-packages/pip/_internal/index/__init__.py b/venv/lib/python3.11/site-packages/pip/_internal/index/__init__.py deleted file mode 100644 index 7a17b7b..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/index/__init__.py +++ /dev/null @@ -1,2 +0,0 @@ -"""Index interaction code -""" diff --git a/venv/lib/python3.11/site-packages/pip/_internal/index/__pycache__/__init__.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/index/__pycache__/__init__.cpython-311.pyc deleted file mode 100644 index e2f688a..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/index/__pycache__/__init__.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/index/__pycache__/collector.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/index/__pycache__/collector.cpython-311.pyc deleted file mode 100644 index 56a6a9f..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/index/__pycache__/collector.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/index/__pycache__/package_finder.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/index/__pycache__/package_finder.cpython-311.pyc deleted file mode 100644 index fea1300..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/index/__pycache__/package_finder.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/index/__pycache__/sources.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/index/__pycache__/sources.cpython-311.pyc deleted file mode 100644 index a3c96f4..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/index/__pycache__/sources.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/index/collector.py b/venv/lib/python3.11/site-packages/pip/_internal/index/collector.py deleted file mode 100644 index b3e293e..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/index/collector.py +++ /dev/null @@ -1,505 +0,0 @@ -""" -The main purpose of this module is to expose LinkCollector.collect_sources(). -""" - -import collections -import email.message -import functools -import itertools -import json -import logging -import os -import urllib.parse -import urllib.request -from html.parser import HTMLParser -from optparse import Values -from typing import ( - TYPE_CHECKING, - Callable, - Dict, - Iterable, - List, - MutableMapping, - NamedTuple, - Optional, - Sequence, - Tuple, - Union, -) - -from pip._vendor import requests -from pip._vendor.requests import Response -from pip._vendor.requests.exceptions import RetryError, SSLError - -from pip._internal.exceptions import NetworkConnectionError -from pip._internal.models.link import Link -from pip._internal.models.search_scope import SearchScope -from pip._internal.network.session import PipSession -from pip._internal.network.utils import raise_for_status -from pip._internal.utils.filetypes import is_archive_file -from pip._internal.utils.misc import redact_auth_from_url -from pip._internal.vcs import vcs - -from .sources import CandidatesFromPage, LinkSource, build_source - -if TYPE_CHECKING: - from typing import Protocol -else: - Protocol = object - -logger = logging.getLogger(__name__) - -ResponseHeaders = MutableMapping[str, str] - - -def _match_vcs_scheme(url: str) -> Optional[str]: - """Look for VCS schemes in the URL. - - Returns the matched VCS scheme, or None if there's no match. - """ - for scheme in vcs.schemes: - if url.lower().startswith(scheme) and url[len(scheme)] in "+:": - return scheme - return None - - -class _NotAPIContent(Exception): - def __init__(self, content_type: str, request_desc: str) -> None: - super().__init__(content_type, request_desc) - self.content_type = content_type - self.request_desc = request_desc - - -def _ensure_api_header(response: Response) -> None: - """ - Check the Content-Type header to ensure the response contains a Simple - API Response. - - Raises `_NotAPIContent` if the content type is not a valid content-type. - """ - content_type = response.headers.get("Content-Type", "Unknown") - - content_type_l = content_type.lower() - if content_type_l.startswith( - ( - "text/html", - "application/vnd.pypi.simple.v1+html", - "application/vnd.pypi.simple.v1+json", - ) - ): - return - - raise _NotAPIContent(content_type, response.request.method) - - -class _NotHTTP(Exception): - pass - - -def _ensure_api_response(url: str, session: PipSession) -> None: - """ - Send a HEAD request to the URL, and ensure the response contains a simple - API Response. - - Raises `_NotHTTP` if the URL is not available for a HEAD request, or - `_NotAPIContent` if the content type is not a valid content type. - """ - scheme, netloc, path, query, fragment = urllib.parse.urlsplit(url) - if scheme not in {"http", "https"}: - raise _NotHTTP() - - resp = session.head(url, allow_redirects=True) - raise_for_status(resp) - - _ensure_api_header(resp) - - -def _get_simple_response(url: str, session: PipSession) -> Response: - """Access an Simple API response with GET, and return the response. - - This consists of three parts: - - 1. If the URL looks suspiciously like an archive, send a HEAD first to - check the Content-Type is HTML or Simple API, to avoid downloading a - large file. Raise `_NotHTTP` if the content type cannot be determined, or - `_NotAPIContent` if it is not HTML or a Simple API. - 2. Actually perform the request. Raise HTTP exceptions on network failures. - 3. Check the Content-Type header to make sure we got a Simple API response, - and raise `_NotAPIContent` otherwise. - """ - if is_archive_file(Link(url).filename): - _ensure_api_response(url, session=session) - - logger.debug("Getting page %s", redact_auth_from_url(url)) - - resp = session.get( - url, - headers={ - "Accept": ", ".join( - [ - "application/vnd.pypi.simple.v1+json", - "application/vnd.pypi.simple.v1+html; q=0.1", - "text/html; q=0.01", - ] - ), - # We don't want to blindly returned cached data for - # /simple/, because authors generally expecting that - # twine upload && pip install will function, but if - # they've done a pip install in the last ~10 minutes - # it won't. Thus by setting this to zero we will not - # blindly use any cached data, however the benefit of - # using max-age=0 instead of no-cache, is that we will - # still support conditional requests, so we will still - # minimize traffic sent in cases where the page hasn't - # changed at all, we will just always incur the round - # trip for the conditional GET now instead of only - # once per 10 minutes. - # For more information, please see pypa/pip#5670. - "Cache-Control": "max-age=0", - }, - ) - raise_for_status(resp) - - # The check for archives above only works if the url ends with - # something that looks like an archive. However that is not a - # requirement of an url. Unless we issue a HEAD request on every - # url we cannot know ahead of time for sure if something is a - # Simple API response or not. However we can check after we've - # downloaded it. - _ensure_api_header(resp) - - logger.debug( - "Fetched page %s as %s", - redact_auth_from_url(url), - resp.headers.get("Content-Type", "Unknown"), - ) - - return resp - - -def _get_encoding_from_headers(headers: ResponseHeaders) -> Optional[str]: - """Determine if we have any encoding information in our headers.""" - if headers and "Content-Type" in headers: - m = email.message.Message() - m["content-type"] = headers["Content-Type"] - charset = m.get_param("charset") - if charset: - return str(charset) - return None - - -class CacheablePageContent: - def __init__(self, page: "IndexContent") -> None: - assert page.cache_link_parsing - self.page = page - - def __eq__(self, other: object) -> bool: - return isinstance(other, type(self)) and self.page.url == other.page.url - - def __hash__(self) -> int: - return hash(self.page.url) - - -class ParseLinks(Protocol): - def __call__(self, page: "IndexContent") -> Iterable[Link]: - ... - - -def with_cached_index_content(fn: ParseLinks) -> ParseLinks: - """ - Given a function that parses an Iterable[Link] from an IndexContent, cache the - function's result (keyed by CacheablePageContent), unless the IndexContent - `page` has `page.cache_link_parsing == False`. - """ - - @functools.lru_cache(maxsize=None) - def wrapper(cacheable_page: CacheablePageContent) -> List[Link]: - return list(fn(cacheable_page.page)) - - @functools.wraps(fn) - def wrapper_wrapper(page: "IndexContent") -> List[Link]: - if page.cache_link_parsing: - return wrapper(CacheablePageContent(page)) - return list(fn(page)) - - return wrapper_wrapper - - -@with_cached_index_content -def parse_links(page: "IndexContent") -> Iterable[Link]: - """ - Parse a Simple API's Index Content, and yield its anchor elements as Link objects. - """ - - content_type_l = page.content_type.lower() - if content_type_l.startswith("application/vnd.pypi.simple.v1+json"): - data = json.loads(page.content) - for file in data.get("files", []): - link = Link.from_json(file, page.url) - if link is None: - continue - yield link - return - - parser = HTMLLinkParser(page.url) - encoding = page.encoding or "utf-8" - parser.feed(page.content.decode(encoding)) - - url = page.url - base_url = parser.base_url or url - for anchor in parser.anchors: - link = Link.from_element(anchor, page_url=url, base_url=base_url) - if link is None: - continue - yield link - - -class IndexContent: - """Represents one response (or page), along with its URL""" - - def __init__( - self, - content: bytes, - content_type: str, - encoding: Optional[str], - url: str, - cache_link_parsing: bool = True, - ) -> None: - """ - :param encoding: the encoding to decode the given content. - :param url: the URL from which the HTML was downloaded. - :param cache_link_parsing: whether links parsed from this page's url - should be cached. PyPI index urls should - have this set to False, for example. - """ - self.content = content - self.content_type = content_type - self.encoding = encoding - self.url = url - self.cache_link_parsing = cache_link_parsing - - def __str__(self) -> str: - return redact_auth_from_url(self.url) - - -class HTMLLinkParser(HTMLParser): - """ - HTMLParser that keeps the first base HREF and a list of all anchor - elements' attributes. - """ - - def __init__(self, url: str) -> None: - super().__init__(convert_charrefs=True) - - self.url: str = url - self.base_url: Optional[str] = None - self.anchors: List[Dict[str, Optional[str]]] = [] - - def handle_starttag(self, tag: str, attrs: List[Tuple[str, Optional[str]]]) -> None: - if tag == "base" and self.base_url is None: - href = self.get_href(attrs) - if href is not None: - self.base_url = href - elif tag == "a": - self.anchors.append(dict(attrs)) - - def get_href(self, attrs: List[Tuple[str, Optional[str]]]) -> Optional[str]: - for name, value in attrs: - if name == "href": - return value - return None - - -def _handle_get_simple_fail( - link: Link, - reason: Union[str, Exception], - meth: Optional[Callable[..., None]] = None, -) -> None: - if meth is None: - meth = logger.debug - meth("Could not fetch URL %s: %s - skipping", link, reason) - - -def _make_index_content( - response: Response, cache_link_parsing: bool = True -) -> IndexContent: - encoding = _get_encoding_from_headers(response.headers) - return IndexContent( - response.content, - response.headers["Content-Type"], - encoding=encoding, - url=response.url, - cache_link_parsing=cache_link_parsing, - ) - - -def _get_index_content(link: Link, *, session: PipSession) -> Optional["IndexContent"]: - url = link.url.split("#", 1)[0] - - # Check for VCS schemes that do not support lookup as web pages. - vcs_scheme = _match_vcs_scheme(url) - if vcs_scheme: - logger.warning( - "Cannot look at %s URL %s because it does not support lookup as web pages.", - vcs_scheme, - link, - ) - return None - - # Tack index.html onto file:// URLs that point to directories - scheme, _, path, _, _, _ = urllib.parse.urlparse(url) - if scheme == "file" and os.path.isdir(urllib.request.url2pathname(path)): - # add trailing slash if not present so urljoin doesn't trim - # final segment - if not url.endswith("/"): - url += "/" - # TODO: In the future, it would be nice if pip supported PEP 691 - # style responses in the file:// URLs, however there's no - # standard file extension for application/vnd.pypi.simple.v1+json - # so we'll need to come up with something on our own. - url = urllib.parse.urljoin(url, "index.html") - logger.debug(" file: URL is directory, getting %s", url) - - try: - resp = _get_simple_response(url, session=session) - except _NotHTTP: - logger.warning( - "Skipping page %s because it looks like an archive, and cannot " - "be checked by a HTTP HEAD request.", - link, - ) - except _NotAPIContent as exc: - logger.warning( - "Skipping page %s because the %s request got Content-Type: %s. " - "The only supported Content-Types are application/vnd.pypi.simple.v1+json, " - "application/vnd.pypi.simple.v1+html, and text/html", - link, - exc.request_desc, - exc.content_type, - ) - except NetworkConnectionError as exc: - _handle_get_simple_fail(link, exc) - except RetryError as exc: - _handle_get_simple_fail(link, exc) - except SSLError as exc: - reason = "There was a problem confirming the ssl certificate: " - reason += str(exc) - _handle_get_simple_fail(link, reason, meth=logger.info) - except requests.ConnectionError as exc: - _handle_get_simple_fail(link, f"connection error: {exc}") - except requests.Timeout: - _handle_get_simple_fail(link, "timed out") - else: - return _make_index_content(resp, cache_link_parsing=link.cache_link_parsing) - return None - - -class CollectedSources(NamedTuple): - find_links: Sequence[Optional[LinkSource]] - index_urls: Sequence[Optional[LinkSource]] - - -class LinkCollector: - - """ - Responsible for collecting Link objects from all configured locations, - making network requests as needed. - - The class's main method is its collect_sources() method. - """ - - def __init__( - self, - session: PipSession, - search_scope: SearchScope, - ) -> None: - self.search_scope = search_scope - self.session = session - - @classmethod - def create( - cls, - session: PipSession, - options: Values, - suppress_no_index: bool = False, - ) -> "LinkCollector": - """ - :param session: The Session to use to make requests. - :param suppress_no_index: Whether to ignore the --no-index option - when constructing the SearchScope object. - """ - index_urls = [options.index_url] + options.extra_index_urls - if options.no_index and not suppress_no_index: - logger.debug( - "Ignoring indexes: %s", - ",".join(redact_auth_from_url(url) for url in index_urls), - ) - index_urls = [] - - # Make sure find_links is a list before passing to create(). - find_links = options.find_links or [] - - search_scope = SearchScope.create( - find_links=find_links, - index_urls=index_urls, - no_index=options.no_index, - ) - link_collector = LinkCollector( - session=session, - search_scope=search_scope, - ) - return link_collector - - @property - def find_links(self) -> List[str]: - return self.search_scope.find_links - - def fetch_response(self, location: Link) -> Optional[IndexContent]: - """ - Fetch an HTML page containing package links. - """ - return _get_index_content(location, session=self.session) - - def collect_sources( - self, - project_name: str, - candidates_from_page: CandidatesFromPage, - ) -> CollectedSources: - # The OrderedDict calls deduplicate sources by URL. - index_url_sources = collections.OrderedDict( - build_source( - loc, - candidates_from_page=candidates_from_page, - page_validator=self.session.is_secure_origin, - expand_dir=False, - cache_link_parsing=False, - ) - for loc in self.search_scope.get_index_urls_locations(project_name) - ).values() - find_links_sources = collections.OrderedDict( - build_source( - loc, - candidates_from_page=candidates_from_page, - page_validator=self.session.is_secure_origin, - expand_dir=True, - cache_link_parsing=True, - ) - for loc in self.find_links - ).values() - - if logger.isEnabledFor(logging.DEBUG): - lines = [ - f"* {s.link}" - for s in itertools.chain(find_links_sources, index_url_sources) - if s is not None and s.link is not None - ] - lines = [ - f"{len(lines)} location(s) to search " - f"for versions of {project_name}:" - ] + lines - logger.debug("\n".join(lines)) - - return CollectedSources( - find_links=list(find_links_sources), - index_urls=list(index_url_sources), - ) diff --git a/venv/lib/python3.11/site-packages/pip/_internal/index/package_finder.py b/venv/lib/python3.11/site-packages/pip/_internal/index/package_finder.py deleted file mode 100644 index 2121ca3..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/index/package_finder.py +++ /dev/null @@ -1,1029 +0,0 @@ -"""Routines related to PyPI, indexes""" - -import enum -import functools -import itertools -import logging -import re -from typing import TYPE_CHECKING, FrozenSet, Iterable, List, Optional, Set, Tuple, Union - -from pip._vendor.packaging import specifiers -from pip._vendor.packaging.tags import Tag -from pip._vendor.packaging.utils import canonicalize_name -from pip._vendor.packaging.version import _BaseVersion -from pip._vendor.packaging.version import parse as parse_version - -from pip._internal.exceptions import ( - BestVersionAlreadyInstalled, - DistributionNotFound, - InvalidWheelFilename, - UnsupportedWheel, -) -from pip._internal.index.collector import LinkCollector, parse_links -from pip._internal.models.candidate import InstallationCandidate -from pip._internal.models.format_control import FormatControl -from pip._internal.models.link import Link -from pip._internal.models.search_scope import SearchScope -from pip._internal.models.selection_prefs import SelectionPreferences -from pip._internal.models.target_python import TargetPython -from pip._internal.models.wheel import Wheel -from pip._internal.req import InstallRequirement -from pip._internal.utils._log import getLogger -from pip._internal.utils.filetypes import WHEEL_EXTENSION -from pip._internal.utils.hashes import Hashes -from pip._internal.utils.logging import indent_log -from pip._internal.utils.misc import build_netloc -from pip._internal.utils.packaging import check_requires_python -from pip._internal.utils.unpacking import SUPPORTED_EXTENSIONS - -if TYPE_CHECKING: - from pip._vendor.typing_extensions import TypeGuard - -__all__ = ["FormatControl", "BestCandidateResult", "PackageFinder"] - - -logger = getLogger(__name__) - -BuildTag = Union[Tuple[()], Tuple[int, str]] -CandidateSortingKey = Tuple[int, int, int, _BaseVersion, Optional[int], BuildTag] - - -def _check_link_requires_python( - link: Link, - version_info: Tuple[int, int, int], - ignore_requires_python: bool = False, -) -> bool: - """ - Return whether the given Python version is compatible with a link's - "Requires-Python" value. - - :param version_info: A 3-tuple of ints representing the Python - major-minor-micro version to check. - :param ignore_requires_python: Whether to ignore the "Requires-Python" - value if the given Python version isn't compatible. - """ - try: - is_compatible = check_requires_python( - link.requires_python, - version_info=version_info, - ) - except specifiers.InvalidSpecifier: - logger.debug( - "Ignoring invalid Requires-Python (%r) for link: %s", - link.requires_python, - link, - ) - else: - if not is_compatible: - version = ".".join(map(str, version_info)) - if not ignore_requires_python: - logger.verbose( - "Link requires a different Python (%s not in: %r): %s", - version, - link.requires_python, - link, - ) - return False - - logger.debug( - "Ignoring failed Requires-Python check (%s not in: %r) for link: %s", - version, - link.requires_python, - link, - ) - - return True - - -class LinkType(enum.Enum): - candidate = enum.auto() - different_project = enum.auto() - yanked = enum.auto() - format_unsupported = enum.auto() - format_invalid = enum.auto() - platform_mismatch = enum.auto() - requires_python_mismatch = enum.auto() - - -class LinkEvaluator: - - """ - Responsible for evaluating links for a particular project. - """ - - _py_version_re = re.compile(r"-py([123]\.?[0-9]?)$") - - # Don't include an allow_yanked default value to make sure each call - # site considers whether yanked releases are allowed. This also causes - # that decision to be made explicit in the calling code, which helps - # people when reading the code. - def __init__( - self, - project_name: str, - canonical_name: str, - formats: FrozenSet[str], - target_python: TargetPython, - allow_yanked: bool, - ignore_requires_python: Optional[bool] = None, - ) -> None: - """ - :param project_name: The user supplied package name. - :param canonical_name: The canonical package name. - :param formats: The formats allowed for this package. Should be a set - with 'binary' or 'source' or both in it. - :param target_python: The target Python interpreter to use when - evaluating link compatibility. This is used, for example, to - check wheel compatibility, as well as when checking the Python - version, e.g. the Python version embedded in a link filename - (or egg fragment) and against an HTML link's optional PEP 503 - "data-requires-python" attribute. - :param allow_yanked: Whether files marked as yanked (in the sense - of PEP 592) are permitted to be candidates for install. - :param ignore_requires_python: Whether to ignore incompatible - PEP 503 "data-requires-python" values in HTML links. Defaults - to False. - """ - if ignore_requires_python is None: - ignore_requires_python = False - - self._allow_yanked = allow_yanked - self._canonical_name = canonical_name - self._ignore_requires_python = ignore_requires_python - self._formats = formats - self._target_python = target_python - - self.project_name = project_name - - def evaluate_link(self, link: Link) -> Tuple[LinkType, str]: - """ - Determine whether a link is a candidate for installation. - - :return: A tuple (result, detail), where *result* is an enum - representing whether the evaluation found a candidate, or the reason - why one is not found. If a candidate is found, *detail* will be the - candidate's version string; if one is not found, it contains the - reason the link fails to qualify. - """ - version = None - if link.is_yanked and not self._allow_yanked: - reason = link.yanked_reason or "" - return (LinkType.yanked, f"yanked for reason: {reason}") - - if link.egg_fragment: - egg_info = link.egg_fragment - ext = link.ext - else: - egg_info, ext = link.splitext() - if not ext: - return (LinkType.format_unsupported, "not a file") - if ext not in SUPPORTED_EXTENSIONS: - return ( - LinkType.format_unsupported, - f"unsupported archive format: {ext}", - ) - if "binary" not in self._formats and ext == WHEEL_EXTENSION: - reason = f"No binaries permitted for {self.project_name}" - return (LinkType.format_unsupported, reason) - if "macosx10" in link.path and ext == ".zip": - return (LinkType.format_unsupported, "macosx10 one") - if ext == WHEEL_EXTENSION: - try: - wheel = Wheel(link.filename) - except InvalidWheelFilename: - return ( - LinkType.format_invalid, - "invalid wheel filename", - ) - if canonicalize_name(wheel.name) != self._canonical_name: - reason = f"wrong project name (not {self.project_name})" - return (LinkType.different_project, reason) - - supported_tags = self._target_python.get_unsorted_tags() - if not wheel.supported(supported_tags): - # Include the wheel's tags in the reason string to - # simplify troubleshooting compatibility issues. - file_tags = ", ".join(wheel.get_formatted_file_tags()) - reason = ( - f"none of the wheel's tags ({file_tags}) are compatible " - f"(run pip debug --verbose to show compatible tags)" - ) - return (LinkType.platform_mismatch, reason) - - version = wheel.version - - # This should be up by the self.ok_binary check, but see issue 2700. - if "source" not in self._formats and ext != WHEEL_EXTENSION: - reason = f"No sources permitted for {self.project_name}" - return (LinkType.format_unsupported, reason) - - if not version: - version = _extract_version_from_fragment( - egg_info, - self._canonical_name, - ) - if not version: - reason = f"Missing project version for {self.project_name}" - return (LinkType.format_invalid, reason) - - match = self._py_version_re.search(version) - if match: - version = version[: match.start()] - py_version = match.group(1) - if py_version != self._target_python.py_version: - return ( - LinkType.platform_mismatch, - "Python version is incorrect", - ) - - supports_python = _check_link_requires_python( - link, - version_info=self._target_python.py_version_info, - ignore_requires_python=self._ignore_requires_python, - ) - if not supports_python: - reason = f"{version} Requires-Python {link.requires_python}" - return (LinkType.requires_python_mismatch, reason) - - logger.debug("Found link %s, version: %s", link, version) - - return (LinkType.candidate, version) - - -def filter_unallowed_hashes( - candidates: List[InstallationCandidate], - hashes: Optional[Hashes], - project_name: str, -) -> List[InstallationCandidate]: - """ - Filter out candidates whose hashes aren't allowed, and return a new - list of candidates. - - If at least one candidate has an allowed hash, then all candidates with - either an allowed hash or no hash specified are returned. Otherwise, - the given candidates are returned. - - Including the candidates with no hash specified when there is a match - allows a warning to be logged if there is a more preferred candidate - with no hash specified. Returning all candidates in the case of no - matches lets pip report the hash of the candidate that would otherwise - have been installed (e.g. permitting the user to more easily update - their requirements file with the desired hash). - """ - if not hashes: - logger.debug( - "Given no hashes to check %s links for project %r: " - "discarding no candidates", - len(candidates), - project_name, - ) - # Make sure we're not returning back the given value. - return list(candidates) - - matches_or_no_digest = [] - # Collect the non-matches for logging purposes. - non_matches = [] - match_count = 0 - for candidate in candidates: - link = candidate.link - if not link.has_hash: - pass - elif link.is_hash_allowed(hashes=hashes): - match_count += 1 - else: - non_matches.append(candidate) - continue - - matches_or_no_digest.append(candidate) - - if match_count: - filtered = matches_or_no_digest - else: - # Make sure we're not returning back the given value. - filtered = list(candidates) - - if len(filtered) == len(candidates): - discard_message = "discarding no candidates" - else: - discard_message = "discarding {} non-matches:\n {}".format( - len(non_matches), - "\n ".join(str(candidate.link) for candidate in non_matches), - ) - - logger.debug( - "Checked %s links for project %r against %s hashes " - "(%s matches, %s no digest): %s", - len(candidates), - project_name, - hashes.digest_count, - match_count, - len(matches_or_no_digest) - match_count, - discard_message, - ) - - return filtered - - -class CandidatePreferences: - - """ - Encapsulates some of the preferences for filtering and sorting - InstallationCandidate objects. - """ - - def __init__( - self, - prefer_binary: bool = False, - allow_all_prereleases: bool = False, - ) -> None: - """ - :param allow_all_prereleases: Whether to allow all pre-releases. - """ - self.allow_all_prereleases = allow_all_prereleases - self.prefer_binary = prefer_binary - - -class BestCandidateResult: - """A collection of candidates, returned by `PackageFinder.find_best_candidate`. - - This class is only intended to be instantiated by CandidateEvaluator's - `compute_best_candidate()` method. - """ - - def __init__( - self, - candidates: List[InstallationCandidate], - applicable_candidates: List[InstallationCandidate], - best_candidate: Optional[InstallationCandidate], - ) -> None: - """ - :param candidates: A sequence of all available candidates found. - :param applicable_candidates: The applicable candidates. - :param best_candidate: The most preferred candidate found, or None - if no applicable candidates were found. - """ - assert set(applicable_candidates) <= set(candidates) - - if best_candidate is None: - assert not applicable_candidates - else: - assert best_candidate in applicable_candidates - - self._applicable_candidates = applicable_candidates - self._candidates = candidates - - self.best_candidate = best_candidate - - def iter_all(self) -> Iterable[InstallationCandidate]: - """Iterate through all candidates.""" - return iter(self._candidates) - - def iter_applicable(self) -> Iterable[InstallationCandidate]: - """Iterate through the applicable candidates.""" - return iter(self._applicable_candidates) - - -class CandidateEvaluator: - - """ - Responsible for filtering and sorting candidates for installation based - on what tags are valid. - """ - - @classmethod - def create( - cls, - project_name: str, - target_python: Optional[TargetPython] = None, - prefer_binary: bool = False, - allow_all_prereleases: bool = False, - specifier: Optional[specifiers.BaseSpecifier] = None, - hashes: Optional[Hashes] = None, - ) -> "CandidateEvaluator": - """Create a CandidateEvaluator object. - - :param target_python: The target Python interpreter to use when - checking compatibility. If None (the default), a TargetPython - object will be constructed from the running Python. - :param specifier: An optional object implementing `filter` - (e.g. `packaging.specifiers.SpecifierSet`) to filter applicable - versions. - :param hashes: An optional collection of allowed hashes. - """ - if target_python is None: - target_python = TargetPython() - if specifier is None: - specifier = specifiers.SpecifierSet() - - supported_tags = target_python.get_sorted_tags() - - return cls( - project_name=project_name, - supported_tags=supported_tags, - specifier=specifier, - prefer_binary=prefer_binary, - allow_all_prereleases=allow_all_prereleases, - hashes=hashes, - ) - - def __init__( - self, - project_name: str, - supported_tags: List[Tag], - specifier: specifiers.BaseSpecifier, - prefer_binary: bool = False, - allow_all_prereleases: bool = False, - hashes: Optional[Hashes] = None, - ) -> None: - """ - :param supported_tags: The PEP 425 tags supported by the target - Python in order of preference (most preferred first). - """ - self._allow_all_prereleases = allow_all_prereleases - self._hashes = hashes - self._prefer_binary = prefer_binary - self._project_name = project_name - self._specifier = specifier - self._supported_tags = supported_tags - # Since the index of the tag in the _supported_tags list is used - # as a priority, precompute a map from tag to index/priority to be - # used in wheel.find_most_preferred_tag. - self._wheel_tag_preferences = { - tag: idx for idx, tag in enumerate(supported_tags) - } - - def get_applicable_candidates( - self, - candidates: List[InstallationCandidate], - ) -> List[InstallationCandidate]: - """ - Return the applicable candidates from a list of candidates. - """ - # Using None infers from the specifier instead. - allow_prereleases = self._allow_all_prereleases or None - specifier = self._specifier - versions = { - str(v) - for v in specifier.filter( - # We turn the version object into a str here because otherwise - # when we're debundled but setuptools isn't, Python will see - # packaging.version.Version and - # pkg_resources._vendor.packaging.version.Version as different - # types. This way we'll use a str as a common data interchange - # format. If we stop using the pkg_resources provided specifier - # and start using our own, we can drop the cast to str(). - (str(c.version) for c in candidates), - prereleases=allow_prereleases, - ) - } - - # Again, converting version to str to deal with debundling. - applicable_candidates = [c for c in candidates if str(c.version) in versions] - - filtered_applicable_candidates = filter_unallowed_hashes( - candidates=applicable_candidates, - hashes=self._hashes, - project_name=self._project_name, - ) - - return sorted(filtered_applicable_candidates, key=self._sort_key) - - def _sort_key(self, candidate: InstallationCandidate) -> CandidateSortingKey: - """ - Function to pass as the `key` argument to a call to sorted() to sort - InstallationCandidates by preference. - - Returns a tuple such that tuples sorting as greater using Python's - default comparison operator are more preferred. - - The preference is as follows: - - First and foremost, candidates with allowed (matching) hashes are - always preferred over candidates without matching hashes. This is - because e.g. if the only candidate with an allowed hash is yanked, - we still want to use that candidate. - - Second, excepting hash considerations, candidates that have been - yanked (in the sense of PEP 592) are always less preferred than - candidates that haven't been yanked. Then: - - If not finding wheels, they are sorted by version only. - If finding wheels, then the sort order is by version, then: - 1. existing installs - 2. wheels ordered via Wheel.support_index_min(self._supported_tags) - 3. source archives - If prefer_binary was set, then all wheels are sorted above sources. - - Note: it was considered to embed this logic into the Link - comparison operators, but then different sdist links - with the same version, would have to be considered equal - """ - valid_tags = self._supported_tags - support_num = len(valid_tags) - build_tag: BuildTag = () - binary_preference = 0 - link = candidate.link - if link.is_wheel: - # can raise InvalidWheelFilename - wheel = Wheel(link.filename) - try: - pri = -( - wheel.find_most_preferred_tag( - valid_tags, self._wheel_tag_preferences - ) - ) - except ValueError: - raise UnsupportedWheel( - "{} is not a supported wheel for this platform. It " - "can't be sorted.".format(wheel.filename) - ) - if self._prefer_binary: - binary_preference = 1 - if wheel.build_tag is not None: - match = re.match(r"^(\d+)(.*)$", wheel.build_tag) - assert match is not None, "guaranteed by filename validation" - build_tag_groups = match.groups() - build_tag = (int(build_tag_groups[0]), build_tag_groups[1]) - else: # sdist - pri = -(support_num) - has_allowed_hash = int(link.is_hash_allowed(self._hashes)) - yank_value = -1 * int(link.is_yanked) # -1 for yanked. - return ( - has_allowed_hash, - yank_value, - binary_preference, - candidate.version, - pri, - build_tag, - ) - - def sort_best_candidate( - self, - candidates: List[InstallationCandidate], - ) -> Optional[InstallationCandidate]: - """ - Return the best candidate per the instance's sort order, or None if - no candidate is acceptable. - """ - if not candidates: - return None - best_candidate = max(candidates, key=self._sort_key) - return best_candidate - - def compute_best_candidate( - self, - candidates: List[InstallationCandidate], - ) -> BestCandidateResult: - """ - Compute and return a `BestCandidateResult` instance. - """ - applicable_candidates = self.get_applicable_candidates(candidates) - - best_candidate = self.sort_best_candidate(applicable_candidates) - - return BestCandidateResult( - candidates, - applicable_candidates=applicable_candidates, - best_candidate=best_candidate, - ) - - -class PackageFinder: - """This finds packages. - - This is meant to match easy_install's technique for looking for - packages, by reading pages and looking for appropriate links. - """ - - def __init__( - self, - link_collector: LinkCollector, - target_python: TargetPython, - allow_yanked: bool, - format_control: Optional[FormatControl] = None, - candidate_prefs: Optional[CandidatePreferences] = None, - ignore_requires_python: Optional[bool] = None, - ) -> None: - """ - This constructor is primarily meant to be used by the create() class - method and from tests. - - :param format_control: A FormatControl object, used to control - the selection of source packages / binary packages when consulting - the index and links. - :param candidate_prefs: Options to use when creating a - CandidateEvaluator object. - """ - if candidate_prefs is None: - candidate_prefs = CandidatePreferences() - - format_control = format_control or FormatControl(set(), set()) - - self._allow_yanked = allow_yanked - self._candidate_prefs = candidate_prefs - self._ignore_requires_python = ignore_requires_python - self._link_collector = link_collector - self._target_python = target_python - - self.format_control = format_control - - # These are boring links that have already been logged somehow. - self._logged_links: Set[Tuple[Link, LinkType, str]] = set() - - # Don't include an allow_yanked default value to make sure each call - # site considers whether yanked releases are allowed. This also causes - # that decision to be made explicit in the calling code, which helps - # people when reading the code. - @classmethod - def create( - cls, - link_collector: LinkCollector, - selection_prefs: SelectionPreferences, - target_python: Optional[TargetPython] = None, - ) -> "PackageFinder": - """Create a PackageFinder. - - :param selection_prefs: The candidate selection preferences, as a - SelectionPreferences object. - :param target_python: The target Python interpreter to use when - checking compatibility. If None (the default), a TargetPython - object will be constructed from the running Python. - """ - if target_python is None: - target_python = TargetPython() - - candidate_prefs = CandidatePreferences( - prefer_binary=selection_prefs.prefer_binary, - allow_all_prereleases=selection_prefs.allow_all_prereleases, - ) - - return cls( - candidate_prefs=candidate_prefs, - link_collector=link_collector, - target_python=target_python, - allow_yanked=selection_prefs.allow_yanked, - format_control=selection_prefs.format_control, - ignore_requires_python=selection_prefs.ignore_requires_python, - ) - - @property - def target_python(self) -> TargetPython: - return self._target_python - - @property - def search_scope(self) -> SearchScope: - return self._link_collector.search_scope - - @search_scope.setter - def search_scope(self, search_scope: SearchScope) -> None: - self._link_collector.search_scope = search_scope - - @property - def find_links(self) -> List[str]: - return self._link_collector.find_links - - @property - def index_urls(self) -> List[str]: - return self.search_scope.index_urls - - @property - def trusted_hosts(self) -> Iterable[str]: - for host_port in self._link_collector.session.pip_trusted_origins: - yield build_netloc(*host_port) - - @property - def allow_all_prereleases(self) -> bool: - return self._candidate_prefs.allow_all_prereleases - - def set_allow_all_prereleases(self) -> None: - self._candidate_prefs.allow_all_prereleases = True - - @property - def prefer_binary(self) -> bool: - return self._candidate_prefs.prefer_binary - - def set_prefer_binary(self) -> None: - self._candidate_prefs.prefer_binary = True - - def requires_python_skipped_reasons(self) -> List[str]: - reasons = { - detail - for _, result, detail in self._logged_links - if result == LinkType.requires_python_mismatch - } - return sorted(reasons) - - def make_link_evaluator(self, project_name: str) -> LinkEvaluator: - canonical_name = canonicalize_name(project_name) - formats = self.format_control.get_allowed_formats(canonical_name) - - return LinkEvaluator( - project_name=project_name, - canonical_name=canonical_name, - formats=formats, - target_python=self._target_python, - allow_yanked=self._allow_yanked, - ignore_requires_python=self._ignore_requires_python, - ) - - def _sort_links(self, links: Iterable[Link]) -> List[Link]: - """ - Returns elements of links in order, non-egg links first, egg links - second, while eliminating duplicates - """ - eggs, no_eggs = [], [] - seen: Set[Link] = set() - for link in links: - if link not in seen: - seen.add(link) - if link.egg_fragment: - eggs.append(link) - else: - no_eggs.append(link) - return no_eggs + eggs - - def _log_skipped_link(self, link: Link, result: LinkType, detail: str) -> None: - entry = (link, result, detail) - if entry not in self._logged_links: - # Put the link at the end so the reason is more visible and because - # the link string is usually very long. - logger.debug("Skipping link: %s: %s", detail, link) - self._logged_links.add(entry) - - def get_install_candidate( - self, link_evaluator: LinkEvaluator, link: Link - ) -> Optional[InstallationCandidate]: - """ - If the link is a candidate for install, convert it to an - InstallationCandidate and return it. Otherwise, return None. - """ - result, detail = link_evaluator.evaluate_link(link) - if result != LinkType.candidate: - self._log_skipped_link(link, result, detail) - return None - - return InstallationCandidate( - name=link_evaluator.project_name, - link=link, - version=detail, - ) - - def evaluate_links( - self, link_evaluator: LinkEvaluator, links: Iterable[Link] - ) -> List[InstallationCandidate]: - """ - Convert links that are candidates to InstallationCandidate objects. - """ - candidates = [] - for link in self._sort_links(links): - candidate = self.get_install_candidate(link_evaluator, link) - if candidate is not None: - candidates.append(candidate) - - return candidates - - def process_project_url( - self, project_url: Link, link_evaluator: LinkEvaluator - ) -> List[InstallationCandidate]: - logger.debug( - "Fetching project page and analyzing links: %s", - project_url, - ) - index_response = self._link_collector.fetch_response(project_url) - if index_response is None: - return [] - - page_links = list(parse_links(index_response)) - - with indent_log(): - package_links = self.evaluate_links( - link_evaluator, - links=page_links, - ) - - return package_links - - @functools.lru_cache(maxsize=None) - def find_all_candidates(self, project_name: str) -> List[InstallationCandidate]: - """Find all available InstallationCandidate for project_name - - This checks index_urls and find_links. - All versions found are returned as an InstallationCandidate list. - - See LinkEvaluator.evaluate_link() for details on which files - are accepted. - """ - link_evaluator = self.make_link_evaluator(project_name) - - collected_sources = self._link_collector.collect_sources( - project_name=project_name, - candidates_from_page=functools.partial( - self.process_project_url, - link_evaluator=link_evaluator, - ), - ) - - page_candidates_it = itertools.chain.from_iterable( - source.page_candidates() - for sources in collected_sources - for source in sources - if source is not None - ) - page_candidates = list(page_candidates_it) - - file_links_it = itertools.chain.from_iterable( - source.file_links() - for sources in collected_sources - for source in sources - if source is not None - ) - file_candidates = self.evaluate_links( - link_evaluator, - sorted(file_links_it, reverse=True), - ) - - if logger.isEnabledFor(logging.DEBUG) and file_candidates: - paths = [] - for candidate in file_candidates: - assert candidate.link.url # we need to have a URL - try: - paths.append(candidate.link.file_path) - except Exception: - paths.append(candidate.link.url) # it's not a local file - - logger.debug("Local files found: %s", ", ".join(paths)) - - # This is an intentional priority ordering - return file_candidates + page_candidates - - def make_candidate_evaluator( - self, - project_name: str, - specifier: Optional[specifiers.BaseSpecifier] = None, - hashes: Optional[Hashes] = None, - ) -> CandidateEvaluator: - """Create a CandidateEvaluator object to use.""" - candidate_prefs = self._candidate_prefs - return CandidateEvaluator.create( - project_name=project_name, - target_python=self._target_python, - prefer_binary=candidate_prefs.prefer_binary, - allow_all_prereleases=candidate_prefs.allow_all_prereleases, - specifier=specifier, - hashes=hashes, - ) - - @functools.lru_cache(maxsize=None) - def find_best_candidate( - self, - project_name: str, - specifier: Optional[specifiers.BaseSpecifier] = None, - hashes: Optional[Hashes] = None, - ) -> BestCandidateResult: - """Find matches for the given project and specifier. - - :param specifier: An optional object implementing `filter` - (e.g. `packaging.specifiers.SpecifierSet`) to filter applicable - versions. - - :return: A `BestCandidateResult` instance. - """ - candidates = self.find_all_candidates(project_name) - candidate_evaluator = self.make_candidate_evaluator( - project_name=project_name, - specifier=specifier, - hashes=hashes, - ) - return candidate_evaluator.compute_best_candidate(candidates) - - def find_requirement( - self, req: InstallRequirement, upgrade: bool - ) -> Optional[InstallationCandidate]: - """Try to find a Link matching req - - Expects req, an InstallRequirement and upgrade, a boolean - Returns a InstallationCandidate if found, - Raises DistributionNotFound or BestVersionAlreadyInstalled otherwise - """ - hashes = req.hashes(trust_internet=False) - best_candidate_result = self.find_best_candidate( - req.name, - specifier=req.specifier, - hashes=hashes, - ) - best_candidate = best_candidate_result.best_candidate - - installed_version: Optional[_BaseVersion] = None - if req.satisfied_by is not None: - installed_version = req.satisfied_by.version - - def _format_versions(cand_iter: Iterable[InstallationCandidate]) -> str: - # This repeated parse_version and str() conversion is needed to - # handle different vendoring sources from pip and pkg_resources. - # If we stop using the pkg_resources provided specifier and start - # using our own, we can drop the cast to str(). - return ( - ", ".join( - sorted( - {str(c.version) for c in cand_iter}, - key=parse_version, - ) - ) - or "none" - ) - - if installed_version is None and best_candidate is None: - logger.critical( - "Could not find a version that satisfies the requirement %s " - "(from versions: %s)", - req, - _format_versions(best_candidate_result.iter_all()), - ) - - raise DistributionNotFound( - "No matching distribution found for {}".format(req) - ) - - def _should_install_candidate( - candidate: Optional[InstallationCandidate], - ) -> "TypeGuard[InstallationCandidate]": - if installed_version is None: - return True - if best_candidate is None: - return False - return best_candidate.version > installed_version - - if not upgrade and installed_version is not None: - if _should_install_candidate(best_candidate): - logger.debug( - "Existing installed version (%s) satisfies requirement " - "(most up-to-date version is %s)", - installed_version, - best_candidate.version, - ) - else: - logger.debug( - "Existing installed version (%s) is most up-to-date and " - "satisfies requirement", - installed_version, - ) - return None - - if _should_install_candidate(best_candidate): - logger.debug( - "Using version %s (newest of versions: %s)", - best_candidate.version, - _format_versions(best_candidate_result.iter_applicable()), - ) - return best_candidate - - # We have an existing version, and its the best version - logger.debug( - "Installed version (%s) is most up-to-date (past versions: %s)", - installed_version, - _format_versions(best_candidate_result.iter_applicable()), - ) - raise BestVersionAlreadyInstalled - - -def _find_name_version_sep(fragment: str, canonical_name: str) -> int: - """Find the separator's index based on the package's canonical name. - - :param fragment: A + filename "fragment" (stem) or - egg fragment. - :param canonical_name: The package's canonical name. - - This function is needed since the canonicalized name does not necessarily - have the same length as the egg info's name part. An example:: - - >>> fragment = 'foo__bar-1.0' - >>> canonical_name = 'foo-bar' - >>> _find_name_version_sep(fragment, canonical_name) - 8 - """ - # Project name and version must be separated by one single dash. Find all - # occurrences of dashes; if the string in front of it matches the canonical - # name, this is the one separating the name and version parts. - for i, c in enumerate(fragment): - if c != "-": - continue - if canonicalize_name(fragment[:i]) == canonical_name: - return i - raise ValueError(f"{fragment} does not match {canonical_name}") - - -def _extract_version_from_fragment(fragment: str, canonical_name: str) -> Optional[str]: - """Parse the version string from a + filename - "fragment" (stem) or egg fragment. - - :param fragment: The string to parse. E.g. foo-2.1 - :param canonical_name: The canonicalized name of the package this - belongs to. - """ - try: - version_start = _find_name_version_sep(fragment, canonical_name) + 1 - except ValueError: - return None - version = fragment[version_start:] - if not version: - return None - return version diff --git a/venv/lib/python3.11/site-packages/pip/_internal/index/sources.py b/venv/lib/python3.11/site-packages/pip/_internal/index/sources.py deleted file mode 100644 index cd9cb8d..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/index/sources.py +++ /dev/null @@ -1,223 +0,0 @@ -import logging -import mimetypes -import os -import pathlib -from typing import Callable, Iterable, Optional, Tuple - -from pip._internal.models.candidate import InstallationCandidate -from pip._internal.models.link import Link -from pip._internal.utils.urls import path_to_url, url_to_path -from pip._internal.vcs import is_url - -logger = logging.getLogger(__name__) - -FoundCandidates = Iterable[InstallationCandidate] -FoundLinks = Iterable[Link] -CandidatesFromPage = Callable[[Link], Iterable[InstallationCandidate]] -PageValidator = Callable[[Link], bool] - - -class LinkSource: - @property - def link(self) -> Optional[Link]: - """Returns the underlying link, if there's one.""" - raise NotImplementedError() - - def page_candidates(self) -> FoundCandidates: - """Candidates found by parsing an archive listing HTML file.""" - raise NotImplementedError() - - def file_links(self) -> FoundLinks: - """Links found by specifying archives directly.""" - raise NotImplementedError() - - -def _is_html_file(file_url: str) -> bool: - return mimetypes.guess_type(file_url, strict=False)[0] == "text/html" - - -class _FlatDirectorySource(LinkSource): - """Link source specified by ``--find-links=``. - - This looks the content of the directory, and returns: - - * ``page_candidates``: Links listed on each HTML file in the directory. - * ``file_candidates``: Archives in the directory. - """ - - def __init__( - self, - candidates_from_page: CandidatesFromPage, - path: str, - ) -> None: - self._candidates_from_page = candidates_from_page - self._path = pathlib.Path(os.path.realpath(path)) - - @property - def link(self) -> Optional[Link]: - return None - - def page_candidates(self) -> FoundCandidates: - for path in self._path.iterdir(): - url = path_to_url(str(path)) - if not _is_html_file(url): - continue - yield from self._candidates_from_page(Link(url)) - - def file_links(self) -> FoundLinks: - for path in self._path.iterdir(): - url = path_to_url(str(path)) - if _is_html_file(url): - continue - yield Link(url) - - -class _LocalFileSource(LinkSource): - """``--find-links=`` or ``--[extra-]index-url=``. - - If a URL is supplied, it must be a ``file:`` URL. If a path is supplied to - the option, it is converted to a URL first. This returns: - - * ``page_candidates``: Links listed on an HTML file. - * ``file_candidates``: The non-HTML file. - """ - - def __init__( - self, - candidates_from_page: CandidatesFromPage, - link: Link, - ) -> None: - self._candidates_from_page = candidates_from_page - self._link = link - - @property - def link(self) -> Optional[Link]: - return self._link - - def page_candidates(self) -> FoundCandidates: - if not _is_html_file(self._link.url): - return - yield from self._candidates_from_page(self._link) - - def file_links(self) -> FoundLinks: - if _is_html_file(self._link.url): - return - yield self._link - - -class _RemoteFileSource(LinkSource): - """``--find-links=`` or ``--[extra-]index-url=``. - - This returns: - - * ``page_candidates``: Links listed on an HTML file. - * ``file_candidates``: The non-HTML file. - """ - - def __init__( - self, - candidates_from_page: CandidatesFromPage, - page_validator: PageValidator, - link: Link, - ) -> None: - self._candidates_from_page = candidates_from_page - self._page_validator = page_validator - self._link = link - - @property - def link(self) -> Optional[Link]: - return self._link - - def page_candidates(self) -> FoundCandidates: - if not self._page_validator(self._link): - return - yield from self._candidates_from_page(self._link) - - def file_links(self) -> FoundLinks: - yield self._link - - -class _IndexDirectorySource(LinkSource): - """``--[extra-]index-url=``. - - This is treated like a remote URL; ``candidates_from_page`` contains logic - for this by appending ``index.html`` to the link. - """ - - def __init__( - self, - candidates_from_page: CandidatesFromPage, - link: Link, - ) -> None: - self._candidates_from_page = candidates_from_page - self._link = link - - @property - def link(self) -> Optional[Link]: - return self._link - - def page_candidates(self) -> FoundCandidates: - yield from self._candidates_from_page(self._link) - - def file_links(self) -> FoundLinks: - return () - - -def build_source( - location: str, - *, - candidates_from_page: CandidatesFromPage, - page_validator: PageValidator, - expand_dir: bool, - cache_link_parsing: bool, -) -> Tuple[Optional[str], Optional[LinkSource]]: - path: Optional[str] = None - url: Optional[str] = None - if os.path.exists(location): # Is a local path. - url = path_to_url(location) - path = location - elif location.startswith("file:"): # A file: URL. - url = location - path = url_to_path(location) - elif is_url(location): - url = location - - if url is None: - msg = ( - "Location '%s' is ignored: " - "it is either a non-existing path or lacks a specific scheme." - ) - logger.warning(msg, location) - return (None, None) - - if path is None: - source: LinkSource = _RemoteFileSource( - candidates_from_page=candidates_from_page, - page_validator=page_validator, - link=Link(url, cache_link_parsing=cache_link_parsing), - ) - return (url, source) - - if os.path.isdir(path): - if expand_dir: - source = _FlatDirectorySource( - candidates_from_page=candidates_from_page, - path=path, - ) - else: - source = _IndexDirectorySource( - candidates_from_page=candidates_from_page, - link=Link(url, cache_link_parsing=cache_link_parsing), - ) - return (url, source) - elif os.path.isfile(path): - source = _LocalFileSource( - candidates_from_page=candidates_from_page, - link=Link(url, cache_link_parsing=cache_link_parsing), - ) - return (url, source) - logger.warning( - "Location '%s' is ignored: it is neither a file nor a directory.", - location, - ) - return (url, None) diff --git a/venv/lib/python3.11/site-packages/pip/_internal/locations/__init__.py b/venv/lib/python3.11/site-packages/pip/_internal/locations/__init__.py deleted file mode 100644 index d54bc63..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/locations/__init__.py +++ /dev/null @@ -1,467 +0,0 @@ -import functools -import logging -import os -import pathlib -import sys -import sysconfig -from typing import Any, Dict, Generator, Optional, Tuple - -from pip._internal.models.scheme import SCHEME_KEYS, Scheme -from pip._internal.utils.compat import WINDOWS -from pip._internal.utils.deprecation import deprecated -from pip._internal.utils.virtualenv import running_under_virtualenv - -from . import _sysconfig -from .base import ( - USER_CACHE_DIR, - get_major_minor_version, - get_src_prefix, - is_osx_framework, - site_packages, - user_site, -) - -__all__ = [ - "USER_CACHE_DIR", - "get_bin_prefix", - "get_bin_user", - "get_major_minor_version", - "get_platlib", - "get_purelib", - "get_scheme", - "get_src_prefix", - "site_packages", - "user_site", -] - - -logger = logging.getLogger(__name__) - - -_PLATLIBDIR: str = getattr(sys, "platlibdir", "lib") - -_USE_SYSCONFIG_DEFAULT = sys.version_info >= (3, 10) - - -def _should_use_sysconfig() -> bool: - """This function determines the value of _USE_SYSCONFIG. - - By default, pip uses sysconfig on Python 3.10+. - But Python distributors can override this decision by setting: - sysconfig._PIP_USE_SYSCONFIG = True / False - Rationale in https://github.com/pypa/pip/issues/10647 - - This is a function for testability, but should be constant during any one - run. - """ - return bool(getattr(sysconfig, "_PIP_USE_SYSCONFIG", _USE_SYSCONFIG_DEFAULT)) - - -_USE_SYSCONFIG = _should_use_sysconfig() - -if not _USE_SYSCONFIG: - # Import distutils lazily to avoid deprecation warnings, - # but import it soon enough that it is in memory and available during - # a pip reinstall. - from . import _distutils - -# Be noisy about incompatibilities if this platforms "should" be using -# sysconfig, but is explicitly opting out and using distutils instead. -if _USE_SYSCONFIG_DEFAULT and not _USE_SYSCONFIG: - _MISMATCH_LEVEL = logging.WARNING -else: - _MISMATCH_LEVEL = logging.DEBUG - - -def _looks_like_bpo_44860() -> bool: - """The resolution to bpo-44860 will change this incorrect platlib. - - See . - """ - from distutils.command.install import INSTALL_SCHEMES - - try: - unix_user_platlib = INSTALL_SCHEMES["unix_user"]["platlib"] - except KeyError: - return False - return unix_user_platlib == "$usersite" - - -def _looks_like_red_hat_patched_platlib_purelib(scheme: Dict[str, str]) -> bool: - platlib = scheme["platlib"] - if "/$platlibdir/" in platlib: - platlib = platlib.replace("/$platlibdir/", f"/{_PLATLIBDIR}/") - if "/lib64/" not in platlib: - return False - unpatched = platlib.replace("/lib64/", "/lib/") - return unpatched.replace("$platbase/", "$base/") == scheme["purelib"] - - -@functools.lru_cache(maxsize=None) -def _looks_like_red_hat_lib() -> bool: - """Red Hat patches platlib in unix_prefix and unix_home, but not purelib. - - This is the only way I can see to tell a Red Hat-patched Python. - """ - from distutils.command.install import INSTALL_SCHEMES - - return all( - k in INSTALL_SCHEMES - and _looks_like_red_hat_patched_platlib_purelib(INSTALL_SCHEMES[k]) - for k in ("unix_prefix", "unix_home") - ) - - -@functools.lru_cache(maxsize=None) -def _looks_like_debian_scheme() -> bool: - """Debian adds two additional schemes.""" - from distutils.command.install import INSTALL_SCHEMES - - return "deb_system" in INSTALL_SCHEMES and "unix_local" in INSTALL_SCHEMES - - -@functools.lru_cache(maxsize=None) -def _looks_like_red_hat_scheme() -> bool: - """Red Hat patches ``sys.prefix`` and ``sys.exec_prefix``. - - Red Hat's ``00251-change-user-install-location.patch`` changes the install - command's ``prefix`` and ``exec_prefix`` to append ``"/local"``. This is - (fortunately?) done quite unconditionally, so we create a default command - object without any configuration to detect this. - """ - from distutils.command.install import install - from distutils.dist import Distribution - - cmd: Any = install(Distribution()) - cmd.finalize_options() - return ( - cmd.exec_prefix == f"{os.path.normpath(sys.exec_prefix)}/local" - and cmd.prefix == f"{os.path.normpath(sys.prefix)}/local" - ) - - -@functools.lru_cache(maxsize=None) -def _looks_like_slackware_scheme() -> bool: - """Slackware patches sysconfig but fails to patch distutils and site. - - Slackware changes sysconfig's user scheme to use ``"lib64"`` for the lib - path, but does not do the same to the site module. - """ - if user_site is None: # User-site not available. - return False - try: - paths = sysconfig.get_paths(scheme="posix_user", expand=False) - except KeyError: # User-site not available. - return False - return "/lib64/" in paths["purelib"] and "/lib64/" not in user_site - - -@functools.lru_cache(maxsize=None) -def _looks_like_msys2_mingw_scheme() -> bool: - """MSYS2 patches distutils and sysconfig to use a UNIX-like scheme. - - However, MSYS2 incorrectly patches sysconfig ``nt`` scheme. The fix is - likely going to be included in their 3.10 release, so we ignore the warning. - See msys2/MINGW-packages#9319. - - MSYS2 MINGW's patch uses lowercase ``"lib"`` instead of the usual uppercase, - and is missing the final ``"site-packages"``. - """ - paths = sysconfig.get_paths("nt", expand=False) - return all( - "Lib" not in p and "lib" in p and not p.endswith("site-packages") - for p in (paths[key] for key in ("platlib", "purelib")) - ) - - -def _fix_abiflags(parts: Tuple[str]) -> Generator[str, None, None]: - ldversion = sysconfig.get_config_var("LDVERSION") - abiflags = getattr(sys, "abiflags", None) - - # LDVERSION does not end with sys.abiflags. Just return the path unchanged. - if not ldversion or not abiflags or not ldversion.endswith(abiflags): - yield from parts - return - - # Strip sys.abiflags from LDVERSION-based path components. - for part in parts: - if part.endswith(ldversion): - part = part[: (0 - len(abiflags))] - yield part - - -@functools.lru_cache(maxsize=None) -def _warn_mismatched(old: pathlib.Path, new: pathlib.Path, *, key: str) -> None: - issue_url = "https://github.com/pypa/pip/issues/10151" - message = ( - "Value for %s does not match. Please report this to <%s>" - "\ndistutils: %s" - "\nsysconfig: %s" - ) - logger.log(_MISMATCH_LEVEL, message, key, issue_url, old, new) - - -def _warn_if_mismatch(old: pathlib.Path, new: pathlib.Path, *, key: str) -> bool: - if old == new: - return False - _warn_mismatched(old, new, key=key) - return True - - -@functools.lru_cache(maxsize=None) -def _log_context( - *, - user: bool = False, - home: Optional[str] = None, - root: Optional[str] = None, - prefix: Optional[str] = None, -) -> None: - parts = [ - "Additional context:", - "user = %r", - "home = %r", - "root = %r", - "prefix = %r", - ] - - logger.log(_MISMATCH_LEVEL, "\n".join(parts), user, home, root, prefix) - - -def get_scheme( - dist_name: str, - user: bool = False, - home: Optional[str] = None, - root: Optional[str] = None, - isolated: bool = False, - prefix: Optional[str] = None, -) -> Scheme: - new = _sysconfig.get_scheme( - dist_name, - user=user, - home=home, - root=root, - isolated=isolated, - prefix=prefix, - ) - if _USE_SYSCONFIG: - return new - - old = _distutils.get_scheme( - dist_name, - user=user, - home=home, - root=root, - isolated=isolated, - prefix=prefix, - ) - - warning_contexts = [] - for k in SCHEME_KEYS: - old_v = pathlib.Path(getattr(old, k)) - new_v = pathlib.Path(getattr(new, k)) - - if old_v == new_v: - continue - - # distutils incorrectly put PyPy packages under ``site-packages/python`` - # in the ``posix_home`` scheme, but PyPy devs said they expect the - # directory name to be ``pypy`` instead. So we treat this as a bug fix - # and not warn about it. See bpo-43307 and python/cpython#24628. - skip_pypy_special_case = ( - sys.implementation.name == "pypy" - and home is not None - and k in ("platlib", "purelib") - and old_v.parent == new_v.parent - and old_v.name.startswith("python") - and new_v.name.startswith("pypy") - ) - if skip_pypy_special_case: - continue - - # sysconfig's ``osx_framework_user`` does not include ``pythonX.Y`` in - # the ``include`` value, but distutils's ``headers`` does. We'll let - # CPython decide whether this is a bug or feature. See bpo-43948. - skip_osx_framework_user_special_case = ( - user - and is_osx_framework() - and k == "headers" - and old_v.parent.parent == new_v.parent - and old_v.parent.name.startswith("python") - ) - if skip_osx_framework_user_special_case: - continue - - # On Red Hat and derived Linux distributions, distutils is patched to - # use "lib64" instead of "lib" for platlib. - if k == "platlib" and _looks_like_red_hat_lib(): - continue - - # On Python 3.9+, sysconfig's posix_user scheme sets platlib against - # sys.platlibdir, but distutils's unix_user incorrectly coninutes - # using the same $usersite for both platlib and purelib. This creates a - # mismatch when sys.platlibdir is not "lib". - skip_bpo_44860 = ( - user - and k == "platlib" - and not WINDOWS - and sys.version_info >= (3, 9) - and _PLATLIBDIR != "lib" - and _looks_like_bpo_44860() - ) - if skip_bpo_44860: - continue - - # Slackware incorrectly patches posix_user to use lib64 instead of lib, - # but not usersite to match the location. - skip_slackware_user_scheme = ( - user - and k in ("platlib", "purelib") - and not WINDOWS - and _looks_like_slackware_scheme() - ) - if skip_slackware_user_scheme: - continue - - # Both Debian and Red Hat patch Python to place the system site under - # /usr/local instead of /usr. Debian also places lib in dist-packages - # instead of site-packages, but the /usr/local check should cover it. - skip_linux_system_special_case = ( - not (user or home or prefix or running_under_virtualenv()) - and old_v.parts[1:3] == ("usr", "local") - and len(new_v.parts) > 1 - and new_v.parts[1] == "usr" - and (len(new_v.parts) < 3 or new_v.parts[2] != "local") - and (_looks_like_red_hat_scheme() or _looks_like_debian_scheme()) - ) - if skip_linux_system_special_case: - continue - - # On Python 3.7 and earlier, sysconfig does not include sys.abiflags in - # the "pythonX.Y" part of the path, but distutils does. - skip_sysconfig_abiflag_bug = ( - sys.version_info < (3, 8) - and not WINDOWS - and k in ("headers", "platlib", "purelib") - and tuple(_fix_abiflags(old_v.parts)) == new_v.parts - ) - if skip_sysconfig_abiflag_bug: - continue - - # MSYS2 MINGW's sysconfig patch does not include the "site-packages" - # part of the path. This is incorrect and will be fixed in MSYS. - skip_msys2_mingw_bug = ( - WINDOWS and k in ("platlib", "purelib") and _looks_like_msys2_mingw_scheme() - ) - if skip_msys2_mingw_bug: - continue - - # CPython's POSIX install script invokes pip (via ensurepip) against the - # interpreter located in the source tree, not the install site. This - # triggers special logic in sysconfig that's not present in distutils. - # https://github.com/python/cpython/blob/8c21941ddaf/Lib/sysconfig.py#L178-L194 - skip_cpython_build = ( - sysconfig.is_python_build(check_home=True) - and not WINDOWS - and k in ("headers", "include", "platinclude") - ) - if skip_cpython_build: - continue - - warning_contexts.append((old_v, new_v, f"scheme.{k}")) - - if not warning_contexts: - return old - - # Check if this path mismatch is caused by distutils config files. Those - # files will no longer work once we switch to sysconfig, so this raises a - # deprecation message for them. - default_old = _distutils.distutils_scheme( - dist_name, - user, - home, - root, - isolated, - prefix, - ignore_config_files=True, - ) - if any(default_old[k] != getattr(old, k) for k in SCHEME_KEYS): - deprecated( - reason=( - "Configuring installation scheme with distutils config files " - "is deprecated and will no longer work in the near future. If you " - "are using a Homebrew or Linuxbrew Python, please see discussion " - "at https://github.com/Homebrew/homebrew-core/issues/76621" - ), - replacement=None, - gone_in=None, - ) - return old - - # Post warnings about this mismatch so user can report them back. - for old_v, new_v, key in warning_contexts: - _warn_mismatched(old_v, new_v, key=key) - _log_context(user=user, home=home, root=root, prefix=prefix) - - return old - - -def get_bin_prefix() -> str: - new = _sysconfig.get_bin_prefix() - if _USE_SYSCONFIG: - return new - - old = _distutils.get_bin_prefix() - if _warn_if_mismatch(pathlib.Path(old), pathlib.Path(new), key="bin_prefix"): - _log_context() - return old - - -def get_bin_user() -> str: - return _sysconfig.get_scheme("", user=True).scripts - - -def _looks_like_deb_system_dist_packages(value: str) -> bool: - """Check if the value is Debian's APT-controlled dist-packages. - - Debian's ``distutils.sysconfig.get_python_lib()`` implementation returns the - default package path controlled by APT, but does not patch ``sysconfig`` to - do the same. This is similar to the bug worked around in ``get_scheme()``, - but here the default is ``deb_system`` instead of ``unix_local``. Ultimately - we can't do anything about this Debian bug, and this detection allows us to - skip the warning when needed. - """ - if not _looks_like_debian_scheme(): - return False - if value == "/usr/lib/python3/dist-packages": - return True - return False - - -def get_purelib() -> str: - """Return the default pure-Python lib location.""" - new = _sysconfig.get_purelib() - if _USE_SYSCONFIG: - return new - - old = _distutils.get_purelib() - if _looks_like_deb_system_dist_packages(old): - return old - if _warn_if_mismatch(pathlib.Path(old), pathlib.Path(new), key="purelib"): - _log_context() - return old - - -def get_platlib() -> str: - """Return the default platform-shared lib location.""" - new = _sysconfig.get_platlib() - if _USE_SYSCONFIG: - return new - - from . import _distutils - - old = _distutils.get_platlib() - if _looks_like_deb_system_dist_packages(old): - return old - if _warn_if_mismatch(pathlib.Path(old), pathlib.Path(new), key="platlib"): - _log_context() - return old diff --git a/venv/lib/python3.11/site-packages/pip/_internal/locations/__pycache__/__init__.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/locations/__pycache__/__init__.cpython-311.pyc deleted file mode 100644 index 5a137aa..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/locations/__pycache__/__init__.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/locations/__pycache__/_distutils.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/locations/__pycache__/_distutils.cpython-311.pyc deleted file mode 100644 index d28f783..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/locations/__pycache__/_distutils.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/locations/__pycache__/_sysconfig.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/locations/__pycache__/_sysconfig.cpython-311.pyc deleted file mode 100644 index cbd9904..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/locations/__pycache__/_sysconfig.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/locations/__pycache__/base.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/locations/__pycache__/base.cpython-311.pyc deleted file mode 100644 index 94bd800..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/locations/__pycache__/base.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/locations/_distutils.py b/venv/lib/python3.11/site-packages/pip/_internal/locations/_distutils.py deleted file mode 100644 index 48689f5..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/locations/_distutils.py +++ /dev/null @@ -1,173 +0,0 @@ -"""Locations where we look for configs, install stuff, etc""" - -# The following comment should be removed at some point in the future. -# mypy: strict-optional=False - -# If pip's going to use distutils, it should not be using the copy that setuptools -# might have injected into the environment. This is done by removing the injected -# shim, if it's injected. -# -# See https://github.com/pypa/pip/issues/8761 for the original discussion and -# rationale for why this is done within pip. -try: - __import__("_distutils_hack").remove_shim() -except (ImportError, AttributeError): - pass - -import logging -import os -import sys -from distutils.cmd import Command as DistutilsCommand -from distutils.command.install import SCHEME_KEYS -from distutils.command.install import install as distutils_install_command -from distutils.sysconfig import get_python_lib -from typing import Dict, List, Optional, Union, cast - -from pip._internal.models.scheme import Scheme -from pip._internal.utils.compat import WINDOWS -from pip._internal.utils.virtualenv import running_under_virtualenv - -from .base import get_major_minor_version - -logger = logging.getLogger(__name__) - - -def distutils_scheme( - dist_name: str, - user: bool = False, - home: Optional[str] = None, - root: Optional[str] = None, - isolated: bool = False, - prefix: Optional[str] = None, - *, - ignore_config_files: bool = False, -) -> Dict[str, str]: - """ - Return a distutils install scheme - """ - from distutils.dist import Distribution - - dist_args: Dict[str, Union[str, List[str]]] = {"name": dist_name} - if isolated: - dist_args["script_args"] = ["--no-user-cfg"] - - d = Distribution(dist_args) - if not ignore_config_files: - try: - d.parse_config_files() - except UnicodeDecodeError: - # Typeshed does not include find_config_files() for some reason. - paths = d.find_config_files() # type: ignore - logger.warning( - "Ignore distutils configs in %s due to encoding errors.", - ", ".join(os.path.basename(p) for p in paths), - ) - obj: Optional[DistutilsCommand] = None - obj = d.get_command_obj("install", create=True) - assert obj is not None - i = cast(distutils_install_command, obj) - # NOTE: setting user or home has the side-effect of creating the home dir - # or user base for installations during finalize_options() - # ideally, we'd prefer a scheme class that has no side-effects. - assert not (user and prefix), f"user={user} prefix={prefix}" - assert not (home and prefix), f"home={home} prefix={prefix}" - i.user = user or i.user - if user or home: - i.prefix = "" - i.prefix = prefix or i.prefix - i.home = home or i.home - i.root = root or i.root - i.finalize_options() - - scheme = {} - for key in SCHEME_KEYS: - scheme[key] = getattr(i, "install_" + key) - - # install_lib specified in setup.cfg should install *everything* - # into there (i.e. it takes precedence over both purelib and - # platlib). Note, i.install_lib is *always* set after - # finalize_options(); we only want to override here if the user - # has explicitly requested it hence going back to the config - if "install_lib" in d.get_option_dict("install"): - scheme.update({"purelib": i.install_lib, "platlib": i.install_lib}) - - if running_under_virtualenv(): - if home: - prefix = home - elif user: - prefix = i.install_userbase - else: - prefix = i.prefix - scheme["headers"] = os.path.join( - prefix, - "include", - "site", - f"python{get_major_minor_version()}", - dist_name, - ) - - if root is not None: - path_no_drive = os.path.splitdrive(os.path.abspath(scheme["headers"]))[1] - scheme["headers"] = os.path.join(root, path_no_drive[1:]) - - return scheme - - -def get_scheme( - dist_name: str, - user: bool = False, - home: Optional[str] = None, - root: Optional[str] = None, - isolated: bool = False, - prefix: Optional[str] = None, -) -> Scheme: - """ - Get the "scheme" corresponding to the input parameters. The distutils - documentation provides the context for the available schemes: - https://docs.python.org/3/install/index.html#alternate-installation - - :param dist_name: the name of the package to retrieve the scheme for, used - in the headers scheme path - :param user: indicates to use the "user" scheme - :param home: indicates to use the "home" scheme and provides the base - directory for the same - :param root: root under which other directories are re-based - :param isolated: equivalent to --no-user-cfg, i.e. do not consider - ~/.pydistutils.cfg (posix) or ~/pydistutils.cfg (non-posix) for - scheme paths - :param prefix: indicates to use the "prefix" scheme and provides the - base directory for the same - """ - scheme = distutils_scheme(dist_name, user, home, root, isolated, prefix) - return Scheme( - platlib=scheme["platlib"], - purelib=scheme["purelib"], - headers=scheme["headers"], - scripts=scheme["scripts"], - data=scheme["data"], - ) - - -def get_bin_prefix() -> str: - # XXX: In old virtualenv versions, sys.prefix can contain '..' components, - # so we need to call normpath to eliminate them. - prefix = os.path.normpath(sys.prefix) - if WINDOWS: - bin_py = os.path.join(prefix, "Scripts") - # buildout uses 'bin' on Windows too? - if not os.path.exists(bin_py): - bin_py = os.path.join(prefix, "bin") - return bin_py - # Forcing to use /usr/local/bin for standard macOS framework installs - # Also log to ~/Library/Logs/ for use with the Console.app log viewer - if sys.platform[:6] == "darwin" and prefix[:16] == "/System/Library/": - return "/usr/local/bin" - return os.path.join(prefix, "bin") - - -def get_purelib() -> str: - return get_python_lib(plat_specific=False) - - -def get_platlib() -> str: - return get_python_lib(plat_specific=True) diff --git a/venv/lib/python3.11/site-packages/pip/_internal/locations/_sysconfig.py b/venv/lib/python3.11/site-packages/pip/_internal/locations/_sysconfig.py deleted file mode 100644 index 97aef1f..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/locations/_sysconfig.py +++ /dev/null @@ -1,213 +0,0 @@ -import logging -import os -import sys -import sysconfig -import typing - -from pip._internal.exceptions import InvalidSchemeCombination, UserInstallationInvalid -from pip._internal.models.scheme import SCHEME_KEYS, Scheme -from pip._internal.utils.virtualenv import running_under_virtualenv - -from .base import change_root, get_major_minor_version, is_osx_framework - -logger = logging.getLogger(__name__) - - -# Notes on _infer_* functions. -# Unfortunately ``get_default_scheme()`` didn't exist before 3.10, so there's no -# way to ask things like "what is the '_prefix' scheme on this platform". These -# functions try to answer that with some heuristics while accounting for ad-hoc -# platforms not covered by CPython's default sysconfig implementation. If the -# ad-hoc implementation does not fully implement sysconfig, we'll fall back to -# a POSIX scheme. - -_AVAILABLE_SCHEMES = set(sysconfig.get_scheme_names()) - -_PREFERRED_SCHEME_API = getattr(sysconfig, "get_preferred_scheme", None) - - -def _should_use_osx_framework_prefix() -> bool: - """Check for Apple's ``osx_framework_library`` scheme. - - Python distributed by Apple's Command Line Tools has this special scheme - that's used when: - - * This is a framework build. - * We are installing into the system prefix. - - This does not account for ``pip install --prefix`` (also means we're not - installing to the system prefix), which should use ``posix_prefix``, but - logic here means ``_infer_prefix()`` outputs ``osx_framework_library``. But - since ``prefix`` is not available for ``sysconfig.get_default_scheme()``, - which is the stdlib replacement for ``_infer_prefix()``, presumably Apple - wouldn't be able to magically switch between ``osx_framework_library`` and - ``posix_prefix``. ``_infer_prefix()`` returning ``osx_framework_library`` - means its behavior is consistent whether we use the stdlib implementation - or our own, and we deal with this special case in ``get_scheme()`` instead. - """ - return ( - "osx_framework_library" in _AVAILABLE_SCHEMES - and not running_under_virtualenv() - and is_osx_framework() - ) - - -def _infer_prefix() -> str: - """Try to find a prefix scheme for the current platform. - - This tries: - - * A special ``osx_framework_library`` for Python distributed by Apple's - Command Line Tools, when not running in a virtual environment. - * Implementation + OS, used by PyPy on Windows (``pypy_nt``). - * Implementation without OS, used by PyPy on POSIX (``pypy``). - * OS + "prefix", used by CPython on POSIX (``posix_prefix``). - * Just the OS name, used by CPython on Windows (``nt``). - - If none of the above works, fall back to ``posix_prefix``. - """ - if _PREFERRED_SCHEME_API: - return _PREFERRED_SCHEME_API("prefix") - if _should_use_osx_framework_prefix(): - return "osx_framework_library" - implementation_suffixed = f"{sys.implementation.name}_{os.name}" - if implementation_suffixed in _AVAILABLE_SCHEMES: - return implementation_suffixed - if sys.implementation.name in _AVAILABLE_SCHEMES: - return sys.implementation.name - suffixed = f"{os.name}_prefix" - if suffixed in _AVAILABLE_SCHEMES: - return suffixed - if os.name in _AVAILABLE_SCHEMES: # On Windows, prefx is just called "nt". - return os.name - return "posix_prefix" - - -def _infer_user() -> str: - """Try to find a user scheme for the current platform.""" - if _PREFERRED_SCHEME_API: - return _PREFERRED_SCHEME_API("user") - if is_osx_framework() and not running_under_virtualenv(): - suffixed = "osx_framework_user" - else: - suffixed = f"{os.name}_user" - if suffixed in _AVAILABLE_SCHEMES: - return suffixed - if "posix_user" not in _AVAILABLE_SCHEMES: # User scheme unavailable. - raise UserInstallationInvalid() - return "posix_user" - - -def _infer_home() -> str: - """Try to find a home for the current platform.""" - if _PREFERRED_SCHEME_API: - return _PREFERRED_SCHEME_API("home") - suffixed = f"{os.name}_home" - if suffixed in _AVAILABLE_SCHEMES: - return suffixed - return "posix_home" - - -# Update these keys if the user sets a custom home. -_HOME_KEYS = [ - "installed_base", - "base", - "installed_platbase", - "platbase", - "prefix", - "exec_prefix", -] -if sysconfig.get_config_var("userbase") is not None: - _HOME_KEYS.append("userbase") - - -def get_scheme( - dist_name: str, - user: bool = False, - home: typing.Optional[str] = None, - root: typing.Optional[str] = None, - isolated: bool = False, - prefix: typing.Optional[str] = None, -) -> Scheme: - """ - Get the "scheme" corresponding to the input parameters. - - :param dist_name: the name of the package to retrieve the scheme for, used - in the headers scheme path - :param user: indicates to use the "user" scheme - :param home: indicates to use the "home" scheme - :param root: root under which other directories are re-based - :param isolated: ignored, but kept for distutils compatibility (where - this controls whether the user-site pydistutils.cfg is honored) - :param prefix: indicates to use the "prefix" scheme and provides the - base directory for the same - """ - if user and prefix: - raise InvalidSchemeCombination("--user", "--prefix") - if home and prefix: - raise InvalidSchemeCombination("--home", "--prefix") - - if home is not None: - scheme_name = _infer_home() - elif user: - scheme_name = _infer_user() - else: - scheme_name = _infer_prefix() - - # Special case: When installing into a custom prefix, use posix_prefix - # instead of osx_framework_library. See _should_use_osx_framework_prefix() - # docstring for details. - if prefix is not None and scheme_name == "osx_framework_library": - scheme_name = "posix_prefix" - - if home is not None: - variables = {k: home for k in _HOME_KEYS} - elif prefix is not None: - variables = {k: prefix for k in _HOME_KEYS} - else: - variables = {} - - paths = sysconfig.get_paths(scheme=scheme_name, vars=variables) - - # Logic here is very arbitrary, we're doing it for compatibility, don't ask. - # 1. Pip historically uses a special header path in virtual environments. - # 2. If the distribution name is not known, distutils uses 'UNKNOWN'. We - # only do the same when not running in a virtual environment because - # pip's historical header path logic (see point 1) did not do this. - if running_under_virtualenv(): - if user: - base = variables.get("userbase", sys.prefix) - else: - base = variables.get("base", sys.prefix) - python_xy = f"python{get_major_minor_version()}" - paths["include"] = os.path.join(base, "include", "site", python_xy) - elif not dist_name: - dist_name = "UNKNOWN" - - scheme = Scheme( - platlib=paths["platlib"], - purelib=paths["purelib"], - headers=os.path.join(paths["include"], dist_name), - scripts=paths["scripts"], - data=paths["data"], - ) - if root is not None: - for key in SCHEME_KEYS: - value = change_root(root, getattr(scheme, key)) - setattr(scheme, key, value) - return scheme - - -def get_bin_prefix() -> str: - # Forcing to use /usr/local/bin for standard macOS framework installs. - if sys.platform[:6] == "darwin" and sys.prefix[:16] == "/System/Library/": - return "/usr/local/bin" - return sysconfig.get_paths()["scripts"] - - -def get_purelib() -> str: - return sysconfig.get_paths()["purelib"] - - -def get_platlib() -> str: - return sysconfig.get_paths()["platlib"] diff --git a/venv/lib/python3.11/site-packages/pip/_internal/locations/base.py b/venv/lib/python3.11/site-packages/pip/_internal/locations/base.py deleted file mode 100644 index 3f9f896..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/locations/base.py +++ /dev/null @@ -1,81 +0,0 @@ -import functools -import os -import site -import sys -import sysconfig -import typing - -from pip._internal.exceptions import InstallationError -from pip._internal.utils import appdirs -from pip._internal.utils.virtualenv import running_under_virtualenv - -# Application Directories -USER_CACHE_DIR = appdirs.user_cache_dir("pip") - -# FIXME doesn't account for venv linked to global site-packages -site_packages: str = sysconfig.get_path("purelib") - - -def get_major_minor_version() -> str: - """ - Return the major-minor version of the current Python as a string, e.g. - "3.7" or "3.10". - """ - return "{}.{}".format(*sys.version_info) - - -def change_root(new_root: str, pathname: str) -> str: - """Return 'pathname' with 'new_root' prepended. - - If 'pathname' is relative, this is equivalent to os.path.join(new_root, pathname). - Otherwise, it requires making 'pathname' relative and then joining the - two, which is tricky on DOS/Windows and Mac OS. - - This is borrowed from Python's standard library's distutils module. - """ - if os.name == "posix": - if not os.path.isabs(pathname): - return os.path.join(new_root, pathname) - else: - return os.path.join(new_root, pathname[1:]) - - elif os.name == "nt": - (drive, path) = os.path.splitdrive(pathname) - if path[0] == "\\": - path = path[1:] - return os.path.join(new_root, path) - - else: - raise InstallationError( - f"Unknown platform: {os.name}\n" - "Can not change root path prefix on unknown platform." - ) - - -def get_src_prefix() -> str: - if running_under_virtualenv(): - src_prefix = os.path.join(sys.prefix, "src") - else: - # FIXME: keep src in cwd for now (it is not a temporary folder) - try: - src_prefix = os.path.join(os.getcwd(), "src") - except OSError: - # In case the current working directory has been renamed or deleted - sys.exit("The folder you are executing pip from can no longer be found.") - - # under macOS + virtualenv sys.prefix is not properly resolved - # it is something like /path/to/python/bin/.. - return os.path.abspath(src_prefix) - - -try: - # Use getusersitepackages if this is present, as it ensures that the - # value is initialised properly. - user_site: typing.Optional[str] = site.getusersitepackages() -except AttributeError: - user_site = site.USER_SITE - - -@functools.lru_cache(maxsize=None) -def is_osx_framework() -> bool: - return bool(sysconfig.get_config_var("PYTHONFRAMEWORK")) diff --git a/venv/lib/python3.11/site-packages/pip/_internal/main.py b/venv/lib/python3.11/site-packages/pip/_internal/main.py deleted file mode 100644 index 33c6d24..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/main.py +++ /dev/null @@ -1,12 +0,0 @@ -from typing import List, Optional - - -def main(args: Optional[List[str]] = None) -> int: - """This is preserved for old console scripts that may still be referencing - it. - - For additional details, see https://github.com/pypa/pip/issues/7498. - """ - from pip._internal.utils.entrypoints import _wrapper - - return _wrapper(args) diff --git a/venv/lib/python3.11/site-packages/pip/_internal/metadata/__init__.py b/venv/lib/python3.11/site-packages/pip/_internal/metadata/__init__.py deleted file mode 100644 index aa232b6..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/metadata/__init__.py +++ /dev/null @@ -1,128 +0,0 @@ -import contextlib -import functools -import os -import sys -from typing import TYPE_CHECKING, List, Optional, Type, cast - -from pip._internal.utils.misc import strtobool - -from .base import BaseDistribution, BaseEnvironment, FilesystemWheel, MemoryWheel, Wheel - -if TYPE_CHECKING: - from typing import Literal, Protocol -else: - Protocol = object - -__all__ = [ - "BaseDistribution", - "BaseEnvironment", - "FilesystemWheel", - "MemoryWheel", - "Wheel", - "get_default_environment", - "get_environment", - "get_wheel_distribution", - "select_backend", -] - - -def _should_use_importlib_metadata() -> bool: - """Whether to use the ``importlib.metadata`` or ``pkg_resources`` backend. - - By default, pip uses ``importlib.metadata`` on Python 3.11+, and - ``pkg_resourcess`` otherwise. This can be overridden by a couple of ways: - - * If environment variable ``_PIP_USE_IMPORTLIB_METADATA`` is set, it - dictates whether ``importlib.metadata`` is used, regardless of Python - version. - * On Python 3.11+, Python distributors can patch ``importlib.metadata`` - to add a global constant ``_PIP_USE_IMPORTLIB_METADATA = False``. This - makes pip use ``pkg_resources`` (unless the user set the aforementioned - environment variable to *True*). - """ - with contextlib.suppress(KeyError, ValueError): - return bool(strtobool(os.environ["_PIP_USE_IMPORTLIB_METADATA"])) - if sys.version_info < (3, 11): - return False - import importlib.metadata - - return bool(getattr(importlib.metadata, "_PIP_USE_IMPORTLIB_METADATA", True)) - - -class Backend(Protocol): - NAME: 'Literal["importlib", "pkg_resources"]' - Distribution: Type[BaseDistribution] - Environment: Type[BaseEnvironment] - - -@functools.lru_cache(maxsize=None) -def select_backend() -> Backend: - if _should_use_importlib_metadata(): - from . import importlib - - return cast(Backend, importlib) - from . import pkg_resources - - return cast(Backend, pkg_resources) - - -def get_default_environment() -> BaseEnvironment: - """Get the default representation for the current environment. - - This returns an Environment instance from the chosen backend. The default - Environment instance should be built from ``sys.path`` and may use caching - to share instance state accorss calls. - """ - return select_backend().Environment.default() - - -def get_environment(paths: Optional[List[str]]) -> BaseEnvironment: - """Get a representation of the environment specified by ``paths``. - - This returns an Environment instance from the chosen backend based on the - given import paths. The backend must build a fresh instance representing - the state of installed distributions when this function is called. - """ - return select_backend().Environment.from_paths(paths) - - -def get_directory_distribution(directory: str) -> BaseDistribution: - """Get the distribution metadata representation in the specified directory. - - This returns a Distribution instance from the chosen backend based on - the given on-disk ``.dist-info`` directory. - """ - return select_backend().Distribution.from_directory(directory) - - -def get_wheel_distribution(wheel: Wheel, canonical_name: str) -> BaseDistribution: - """Get the representation of the specified wheel's distribution metadata. - - This returns a Distribution instance from the chosen backend based on - the given wheel's ``.dist-info`` directory. - - :param canonical_name: Normalized project name of the given wheel. - """ - return select_backend().Distribution.from_wheel(wheel, canonical_name) - - -def get_metadata_distribution( - metadata_contents: bytes, - filename: str, - canonical_name: str, -) -> BaseDistribution: - """Get the dist representation of the specified METADATA file contents. - - This returns a Distribution instance from the chosen backend sourced from the data - in `metadata_contents`. - - :param metadata_contents: Contents of a METADATA file within a dist, or one served - via PEP 658. - :param filename: Filename for the dist this metadata represents. - :param canonical_name: Normalized project name of the given dist. - """ - return select_backend().Distribution.from_metadata_file_contents( - metadata_contents, - filename, - canonical_name, - ) diff --git a/venv/lib/python3.11/site-packages/pip/_internal/metadata/__pycache__/__init__.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/metadata/__pycache__/__init__.cpython-311.pyc deleted file mode 100644 index 73dd88b..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/metadata/__pycache__/__init__.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/metadata/__pycache__/_json.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/metadata/__pycache__/_json.cpython-311.pyc deleted file mode 100644 index 0b3a177..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/metadata/__pycache__/_json.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/metadata/__pycache__/base.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/metadata/__pycache__/base.cpython-311.pyc deleted file mode 100644 index 470553d..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/metadata/__pycache__/base.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/metadata/__pycache__/pkg_resources.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/metadata/__pycache__/pkg_resources.cpython-311.pyc deleted file mode 100644 index 7252d00..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/metadata/__pycache__/pkg_resources.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/metadata/_json.py b/venv/lib/python3.11/site-packages/pip/_internal/metadata/_json.py deleted file mode 100644 index 336b52f..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/metadata/_json.py +++ /dev/null @@ -1,84 +0,0 @@ -# Extracted from https://github.com/pfmoore/pkg_metadata - -from email.header import Header, decode_header, make_header -from email.message import Message -from typing import Any, Dict, List, Union - -METADATA_FIELDS = [ - # Name, Multiple-Use - ("Metadata-Version", False), - ("Name", False), - ("Version", False), - ("Dynamic", True), - ("Platform", True), - ("Supported-Platform", True), - ("Summary", False), - ("Description", False), - ("Description-Content-Type", False), - ("Keywords", False), - ("Home-page", False), - ("Download-URL", False), - ("Author", False), - ("Author-email", False), - ("Maintainer", False), - ("Maintainer-email", False), - ("License", False), - ("Classifier", True), - ("Requires-Dist", True), - ("Requires-Python", False), - ("Requires-External", True), - ("Project-URL", True), - ("Provides-Extra", True), - ("Provides-Dist", True), - ("Obsoletes-Dist", True), -] - - -def json_name(field: str) -> str: - return field.lower().replace("-", "_") - - -def msg_to_json(msg: Message) -> Dict[str, Any]: - """Convert a Message object into a JSON-compatible dictionary.""" - - def sanitise_header(h: Union[Header, str]) -> str: - if isinstance(h, Header): - chunks = [] - for bytes, encoding in decode_header(h): - if encoding == "unknown-8bit": - try: - # See if UTF-8 works - bytes.decode("utf-8") - encoding = "utf-8" - except UnicodeDecodeError: - # If not, latin1 at least won't fail - encoding = "latin1" - chunks.append((bytes, encoding)) - return str(make_header(chunks)) - return str(h) - - result = {} - for field, multi in METADATA_FIELDS: - if field not in msg: - continue - key = json_name(field) - if multi: - value: Union[str, List[str]] = [ - sanitise_header(v) for v in msg.get_all(field) - ] - else: - value = sanitise_header(msg.get(field)) - if key == "keywords": - # Accept both comma-separated and space-separated - # forms, for better compatibility with old data. - if "," in value: - value = [v.strip() for v in value.split(",")] - else: - value = value.split() - result[key] = value - - payload = msg.get_payload() - if payload: - result["description"] = payload - - return result diff --git a/venv/lib/python3.11/site-packages/pip/_internal/metadata/base.py b/venv/lib/python3.11/site-packages/pip/_internal/metadata/base.py deleted file mode 100644 index 9249124..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/metadata/base.py +++ /dev/null @@ -1,702 +0,0 @@ -import csv -import email.message -import functools -import json -import logging -import pathlib -import re -import zipfile -from typing import ( - IO, - TYPE_CHECKING, - Any, - Collection, - Container, - Dict, - Iterable, - Iterator, - List, - NamedTuple, - Optional, - Tuple, - Union, -) - -from pip._vendor.packaging.requirements import Requirement -from pip._vendor.packaging.specifiers import InvalidSpecifier, SpecifierSet -from pip._vendor.packaging.utils import NormalizedName, canonicalize_name -from pip._vendor.packaging.version import LegacyVersion, Version - -from pip._internal.exceptions import NoneMetadataError -from pip._internal.locations import site_packages, user_site -from pip._internal.models.direct_url import ( - DIRECT_URL_METADATA_NAME, - DirectUrl, - DirectUrlValidationError, -) -from pip._internal.utils.compat import stdlib_pkgs # TODO: Move definition here. -from pip._internal.utils.egg_link import egg_link_path_from_sys_path -from pip._internal.utils.misc import is_local, normalize_path -from pip._internal.utils.urls import url_to_path - -from ._json import msg_to_json - -if TYPE_CHECKING: - from typing import Protocol -else: - Protocol = object - -DistributionVersion = Union[LegacyVersion, Version] - -InfoPath = Union[str, pathlib.PurePath] - -logger = logging.getLogger(__name__) - - -class BaseEntryPoint(Protocol): - @property - def name(self) -> str: - raise NotImplementedError() - - @property - def value(self) -> str: - raise NotImplementedError() - - @property - def group(self) -> str: - raise NotImplementedError() - - -def _convert_installed_files_path( - entry: Tuple[str, ...], - info: Tuple[str, ...], -) -> str: - """Convert a legacy installed-files.txt path into modern RECORD path. - - The legacy format stores paths relative to the info directory, while the - modern format stores paths relative to the package root, e.g. the - site-packages directory. - - :param entry: Path parts of the installed-files.txt entry. - :param info: Path parts of the egg-info directory relative to package root. - :returns: The converted entry. - - For best compatibility with symlinks, this does not use ``abspath()`` or - ``Path.resolve()``, but tries to work with path parts: - - 1. While ``entry`` starts with ``..``, remove the equal amounts of parts - from ``info``; if ``info`` is empty, start appending ``..`` instead. - 2. Join the two directly. - """ - while entry and entry[0] == "..": - if not info or info[-1] == "..": - info += ("..",) - else: - info = info[:-1] - entry = entry[1:] - return str(pathlib.Path(*info, *entry)) - - -class RequiresEntry(NamedTuple): - requirement: str - extra: str - marker: str - - -class BaseDistribution(Protocol): - @classmethod - def from_directory(cls, directory: str) -> "BaseDistribution": - """Load the distribution from a metadata directory. - - :param directory: Path to a metadata directory, e.g. ``.dist-info``. - """ - raise NotImplementedError() - - @classmethod - def from_metadata_file_contents( - cls, - metadata_contents: bytes, - filename: str, - project_name: str, - ) -> "BaseDistribution": - """Load the distribution from the contents of a METADATA file. - - This is used to implement PEP 658 by generating a "shallow" dist object that can - be used for resolution without downloading or building the actual dist yet. - - :param metadata_contents: The contents of a METADATA file. - :param filename: File name for the dist with this metadata. - :param project_name: Name of the project this dist represents. - """ - raise NotImplementedError() - - @classmethod - def from_wheel(cls, wheel: "Wheel", name: str) -> "BaseDistribution": - """Load the distribution from a given wheel. - - :param wheel: A concrete wheel definition. - :param name: File name of the wheel. - - :raises InvalidWheel: Whenever loading of the wheel causes a - :py:exc:`zipfile.BadZipFile` exception to be thrown. - :raises UnsupportedWheel: If the wheel is a valid zip, but malformed - internally. - """ - raise NotImplementedError() - - def __repr__(self) -> str: - return f"{self.raw_name} {self.version} ({self.location})" - - def __str__(self) -> str: - return f"{self.raw_name} {self.version}" - - @property - def location(self) -> Optional[str]: - """Where the distribution is loaded from. - - A string value is not necessarily a filesystem path, since distributions - can be loaded from other sources, e.g. arbitrary zip archives. ``None`` - means the distribution is created in-memory. - - Do not canonicalize this value with e.g. ``pathlib.Path.resolve()``. If - this is a symbolic link, we want to preserve the relative path between - it and files in the distribution. - """ - raise NotImplementedError() - - @property - def editable_project_location(self) -> Optional[str]: - """The project location for editable distributions. - - This is the directory where pyproject.toml or setup.py is located. - None if the distribution is not installed in editable mode. - """ - # TODO: this property is relatively costly to compute, memoize it ? - direct_url = self.direct_url - if direct_url: - if direct_url.is_local_editable(): - return url_to_path(direct_url.url) - else: - # Search for an .egg-link file by walking sys.path, as it was - # done before by dist_is_editable(). - egg_link_path = egg_link_path_from_sys_path(self.raw_name) - if egg_link_path: - # TODO: get project location from second line of egg_link file - # (https://github.com/pypa/pip/issues/10243) - return self.location - return None - - @property - def installed_location(self) -> Optional[str]: - """The distribution's "installed" location. - - This should generally be a ``site-packages`` directory. This is - usually ``dist.location``, except for legacy develop-installed packages, - where ``dist.location`` is the source code location, and this is where - the ``.egg-link`` file is. - - The returned location is normalized (in particular, with symlinks removed). - """ - raise NotImplementedError() - - @property - def info_location(self) -> Optional[str]: - """Location of the .[egg|dist]-info directory or file. - - Similarly to ``location``, a string value is not necessarily a - filesystem path. ``None`` means the distribution is created in-memory. - - For a modern .dist-info installation on disk, this should be something - like ``{location}/{raw_name}-{version}.dist-info``. - - Do not canonicalize this value with e.g. ``pathlib.Path.resolve()``. If - this is a symbolic link, we want to preserve the relative path between - it and other files in the distribution. - """ - raise NotImplementedError() - - @property - def installed_by_distutils(self) -> bool: - """Whether this distribution is installed with legacy distutils format. - - A distribution installed with "raw" distutils not patched by setuptools - uses one single file at ``info_location`` to store metadata. We need to - treat this specially on uninstallation. - """ - info_location = self.info_location - if not info_location: - return False - return pathlib.Path(info_location).is_file() - - @property - def installed_as_egg(self) -> bool: - """Whether this distribution is installed as an egg. - - This usually indicates the distribution was installed by (older versions - of) easy_install. - """ - location = self.location - if not location: - return False - return location.endswith(".egg") - - @property - def installed_with_setuptools_egg_info(self) -> bool: - """Whether this distribution is installed with the ``.egg-info`` format. - - This usually indicates the distribution was installed with setuptools - with an old pip version or with ``single-version-externally-managed``. - - Note that this ensure the metadata store is a directory. distutils can - also installs an ``.egg-info``, but as a file, not a directory. This - property is *False* for that case. Also see ``installed_by_distutils``. - """ - info_location = self.info_location - if not info_location: - return False - if not info_location.endswith(".egg-info"): - return False - return pathlib.Path(info_location).is_dir() - - @property - def installed_with_dist_info(self) -> bool: - """Whether this distribution is installed with the "modern format". - - This indicates a "modern" installation, e.g. storing metadata in the - ``.dist-info`` directory. This applies to installations made by - setuptools (but through pip, not directly), or anything using the - standardized build backend interface (PEP 517). - """ - info_location = self.info_location - if not info_location: - return False - if not info_location.endswith(".dist-info"): - return False - return pathlib.Path(info_location).is_dir() - - @property - def canonical_name(self) -> NormalizedName: - raise NotImplementedError() - - @property - def version(self) -> DistributionVersion: - raise NotImplementedError() - - @property - def setuptools_filename(self) -> str: - """Convert a project name to its setuptools-compatible filename. - - This is a copy of ``pkg_resources.to_filename()`` for compatibility. - """ - return self.raw_name.replace("-", "_") - - @property - def direct_url(self) -> Optional[DirectUrl]: - """Obtain a DirectUrl from this distribution. - - Returns None if the distribution has no `direct_url.json` metadata, - or if `direct_url.json` is invalid. - """ - try: - content = self.read_text(DIRECT_URL_METADATA_NAME) - except FileNotFoundError: - return None - try: - return DirectUrl.from_json(content) - except ( - UnicodeDecodeError, - json.JSONDecodeError, - DirectUrlValidationError, - ) as e: - logger.warning( - "Error parsing %s for %s: %s", - DIRECT_URL_METADATA_NAME, - self.canonical_name, - e, - ) - return None - - @property - def installer(self) -> str: - try: - installer_text = self.read_text("INSTALLER") - except (OSError, ValueError, NoneMetadataError): - return "" # Fail silently if the installer file cannot be read. - for line in installer_text.splitlines(): - cleaned_line = line.strip() - if cleaned_line: - return cleaned_line - return "" - - @property - def requested(self) -> bool: - return self.is_file("REQUESTED") - - @property - def editable(self) -> bool: - return bool(self.editable_project_location) - - @property - def local(self) -> bool: - """If distribution is installed in the current virtual environment. - - Always True if we're not in a virtualenv. - """ - if self.installed_location is None: - return False - return is_local(self.installed_location) - - @property - def in_usersite(self) -> bool: - if self.installed_location is None or user_site is None: - return False - return self.installed_location.startswith(normalize_path(user_site)) - - @property - def in_site_packages(self) -> bool: - if self.installed_location is None or site_packages is None: - return False - return self.installed_location.startswith(normalize_path(site_packages)) - - def is_file(self, path: InfoPath) -> bool: - """Check whether an entry in the info directory is a file.""" - raise NotImplementedError() - - def iter_distutils_script_names(self) -> Iterator[str]: - """Find distutils 'scripts' entries metadata. - - If 'scripts' is supplied in ``setup.py``, distutils records those in the - installed distribution's ``scripts`` directory, a file for each script. - """ - raise NotImplementedError() - - def read_text(self, path: InfoPath) -> str: - """Read a file in the info directory. - - :raise FileNotFoundError: If ``path`` does not exist in the directory. - :raise NoneMetadataError: If ``path`` exists in the info directory, but - cannot be read. - """ - raise NotImplementedError() - - def iter_entry_points(self) -> Iterable[BaseEntryPoint]: - raise NotImplementedError() - - def _metadata_impl(self) -> email.message.Message: - raise NotImplementedError() - - @functools.lru_cache(maxsize=1) - def _metadata_cached(self) -> email.message.Message: - # When we drop python 3.7 support, move this to the metadata property and use - # functools.cached_property instead of lru_cache. - metadata = self._metadata_impl() - self._add_egg_info_requires(metadata) - return metadata - - @property - def metadata(self) -> email.message.Message: - """Metadata of distribution parsed from e.g. METADATA or PKG-INFO. - - This should return an empty message if the metadata file is unavailable. - - :raises NoneMetadataError: If the metadata file is available, but does - not contain valid metadata. - """ - return self._metadata_cached() - - @property - def metadata_dict(self) -> Dict[str, Any]: - """PEP 566 compliant JSON-serializable representation of METADATA or PKG-INFO. - - This should return an empty dict if the metadata file is unavailable. - - :raises NoneMetadataError: If the metadata file is available, but does - not contain valid metadata. - """ - return msg_to_json(self.metadata) - - @property - def metadata_version(self) -> Optional[str]: - """Value of "Metadata-Version:" in distribution metadata, if available.""" - return self.metadata.get("Metadata-Version") - - @property - def raw_name(self) -> str: - """Value of "Name:" in distribution metadata.""" - # The metadata should NEVER be missing the Name: key, but if it somehow - # does, fall back to the known canonical name. - return self.metadata.get("Name", self.canonical_name) - - @property - def requires_python(self) -> SpecifierSet: - """Value of "Requires-Python:" in distribution metadata. - - If the key does not exist or contains an invalid value, an empty - SpecifierSet should be returned. - """ - value = self.metadata.get("Requires-Python") - if value is None: - return SpecifierSet() - try: - # Convert to str to satisfy the type checker; this can be a Header object. - spec = SpecifierSet(str(value)) - except InvalidSpecifier as e: - message = "Package %r has an invalid Requires-Python: %s" - logger.warning(message, self.raw_name, e) - return SpecifierSet() - return spec - - def iter_dependencies(self, extras: Collection[str] = ()) -> Iterable[Requirement]: - """Dependencies of this distribution. - - For modern .dist-info distributions, this is the collection of - "Requires-Dist:" entries in distribution metadata. - """ - raise NotImplementedError() - - def iter_provided_extras(self) -> Iterable[str]: - """Extras provided by this distribution. - - For modern .dist-info distributions, this is the collection of - "Provides-Extra:" entries in distribution metadata. - - The return value of this function is not particularly useful other than - display purposes due to backward compatibility issues and the extra - names being poorly normalized prior to PEP 685. If you want to perform - logic operations on extras, use :func:`is_extra_provided` instead. - """ - raise NotImplementedError() - - def is_extra_provided(self, extra: str) -> bool: - """Check whether an extra is provided by this distribution. - - This is needed mostly for compatibility issues with pkg_resources not - following the extra normalization rules defined in PEP 685. - """ - raise NotImplementedError() - - def _iter_declared_entries_from_record(self) -> Optional[Iterator[str]]: - try: - text = self.read_text("RECORD") - except FileNotFoundError: - return None - # This extra Path-str cast normalizes entries. - return (str(pathlib.Path(row[0])) for row in csv.reader(text.splitlines())) - - def _iter_declared_entries_from_legacy(self) -> Optional[Iterator[str]]: - try: - text = self.read_text("installed-files.txt") - except FileNotFoundError: - return None - paths = (p for p in text.splitlines(keepends=False) if p) - root = self.location - info = self.info_location - if root is None or info is None: - return paths - try: - info_rel = pathlib.Path(info).relative_to(root) - except ValueError: # info is not relative to root. - return paths - if not info_rel.parts: # info *is* root. - return paths - return ( - _convert_installed_files_path(pathlib.Path(p).parts, info_rel.parts) - for p in paths - ) - - def iter_declared_entries(self) -> Optional[Iterator[str]]: - """Iterate through file entries declared in this distribution. - - For modern .dist-info distributions, this is the files listed in the - ``RECORD`` metadata file. For legacy setuptools distributions, this - comes from ``installed-files.txt``, with entries normalized to be - compatible with the format used by ``RECORD``. - - :return: An iterator for listed entries, or None if the distribution - contains neither ``RECORD`` nor ``installed-files.txt``. - """ - return ( - self._iter_declared_entries_from_record() - or self._iter_declared_entries_from_legacy() - ) - - def _iter_requires_txt_entries(self) -> Iterator[RequiresEntry]: - """Parse a ``requires.txt`` in an egg-info directory. - - This is an INI-ish format where an egg-info stores dependencies. A - section name describes extra other environment markers, while each entry - is an arbitrary string (not a key-value pair) representing a dependency - as a requirement string (no markers). - - There is a construct in ``importlib.metadata`` called ``Sectioned`` that - does mostly the same, but the format is currently considered private. - """ - try: - content = self.read_text("requires.txt") - except FileNotFoundError: - return - extra = marker = "" # Section-less entries don't have markers. - for line in content.splitlines(): - line = line.strip() - if not line or line.startswith("#"): # Comment; ignored. - continue - if line.startswith("[") and line.endswith("]"): # A section header. - extra, _, marker = line.strip("[]").partition(":") - continue - yield RequiresEntry(requirement=line, extra=extra, marker=marker) - - def _iter_egg_info_extras(self) -> Iterable[str]: - """Get extras from the egg-info directory.""" - known_extras = {""} - for entry in self._iter_requires_txt_entries(): - extra = canonicalize_name(entry.extra) - if extra in known_extras: - continue - known_extras.add(extra) - yield extra - - def _iter_egg_info_dependencies(self) -> Iterable[str]: - """Get distribution dependencies from the egg-info directory. - - To ease parsing, this converts a legacy dependency entry into a PEP 508 - requirement string. Like ``_iter_requires_txt_entries()``, there is code - in ``importlib.metadata`` that does mostly the same, but not do exactly - what we need. - - Namely, ``importlib.metadata`` does not normalize the extra name before - putting it into the requirement string, which causes marker comparison - to fail because the dist-info format do normalize. This is consistent in - all currently available PEP 517 backends, although not standardized. - """ - for entry in self._iter_requires_txt_entries(): - extra = canonicalize_name(entry.extra) - if extra and entry.marker: - marker = f'({entry.marker}) and extra == "{extra}"' - elif extra: - marker = f'extra == "{extra}"' - elif entry.marker: - marker = entry.marker - else: - marker = "" - if marker: - yield f"{entry.requirement} ; {marker}" - else: - yield entry.requirement - - def _add_egg_info_requires(self, metadata: email.message.Message) -> None: - """Add egg-info requires.txt information to the metadata.""" - if not metadata.get_all("Requires-Dist"): - for dep in self._iter_egg_info_dependencies(): - metadata["Requires-Dist"] = dep - if not metadata.get_all("Provides-Extra"): - for extra in self._iter_egg_info_extras(): - metadata["Provides-Extra"] = extra - - -class BaseEnvironment: - """An environment containing distributions to introspect.""" - - @classmethod - def default(cls) -> "BaseEnvironment": - raise NotImplementedError() - - @classmethod - def from_paths(cls, paths: Optional[List[str]]) -> "BaseEnvironment": - raise NotImplementedError() - - def get_distribution(self, name: str) -> Optional["BaseDistribution"]: - """Given a requirement name, return the installed distributions. - - The name may not be normalized. The implementation must canonicalize - it for lookup. - """ - raise NotImplementedError() - - def _iter_distributions(self) -> Iterator["BaseDistribution"]: - """Iterate through installed distributions. - - This function should be implemented by subclass, but never called - directly. Use the public ``iter_distribution()`` instead, which - implements additional logic to make sure the distributions are valid. - """ - raise NotImplementedError() - - def iter_all_distributions(self) -> Iterator[BaseDistribution]: - """Iterate through all installed distributions without any filtering.""" - for dist in self._iter_distributions(): - # Make sure the distribution actually comes from a valid Python - # packaging distribution. Pip's AdjacentTempDirectory leaves folders - # e.g. ``~atplotlib.dist-info`` if cleanup was interrupted. The - # valid project name pattern is taken from PEP 508. - project_name_valid = re.match( - r"^([A-Z0-9]|[A-Z0-9][A-Z0-9._-]*[A-Z0-9])$", - dist.canonical_name, - flags=re.IGNORECASE, - ) - if not project_name_valid: - logger.warning( - "Ignoring invalid distribution %s (%s)", - dist.canonical_name, - dist.location, - ) - continue - yield dist - - def iter_installed_distributions( - self, - local_only: bool = True, - skip: Container[str] = stdlib_pkgs, - include_editables: bool = True, - editables_only: bool = False, - user_only: bool = False, - ) -> Iterator[BaseDistribution]: - """Return a list of installed distributions. - - This is based on ``iter_all_distributions()`` with additional filtering - options. Note that ``iter_installed_distributions()`` without arguments - is *not* equal to ``iter_all_distributions()``, since some of the - configurations exclude packages by default. - - :param local_only: If True (default), only return installations - local to the current virtualenv, if in a virtualenv. - :param skip: An iterable of canonicalized project names to ignore; - defaults to ``stdlib_pkgs``. - :param include_editables: If False, don't report editables. - :param editables_only: If True, only report editables. - :param user_only: If True, only report installations in the user - site directory. - """ - it = self.iter_all_distributions() - if local_only: - it = (d for d in it if d.local) - if not include_editables: - it = (d for d in it if not d.editable) - if editables_only: - it = (d for d in it if d.editable) - if user_only: - it = (d for d in it if d.in_usersite) - return (d for d in it if d.canonical_name not in skip) - - -class Wheel(Protocol): - location: str - - def as_zipfile(self) -> zipfile.ZipFile: - raise NotImplementedError() - - -class FilesystemWheel(Wheel): - def __init__(self, location: str) -> None: - self.location = location - - def as_zipfile(self) -> zipfile.ZipFile: - return zipfile.ZipFile(self.location, allowZip64=True) - - -class MemoryWheel(Wheel): - def __init__(self, location: str, stream: IO[bytes]) -> None: - self.location = location - self.stream = stream - - def as_zipfile(self) -> zipfile.ZipFile: - return zipfile.ZipFile(self.stream, allowZip64=True) diff --git a/venv/lib/python3.11/site-packages/pip/_internal/metadata/importlib/__init__.py b/venv/lib/python3.11/site-packages/pip/_internal/metadata/importlib/__init__.py deleted file mode 100644 index a779138..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/metadata/importlib/__init__.py +++ /dev/null @@ -1,6 +0,0 @@ -from ._dists import Distribution -from ._envs import Environment - -__all__ = ["NAME", "Distribution", "Environment"] - -NAME = "importlib" diff --git a/venv/lib/python3.11/site-packages/pip/_internal/metadata/importlib/__pycache__/__init__.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/metadata/importlib/__pycache__/__init__.cpython-311.pyc deleted file mode 100644 index 6e8cc8f..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/metadata/importlib/__pycache__/__init__.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/metadata/importlib/__pycache__/_compat.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/metadata/importlib/__pycache__/_compat.cpython-311.pyc deleted file mode 100644 index 10e5660..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/metadata/importlib/__pycache__/_compat.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/metadata/importlib/__pycache__/_dists.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/metadata/importlib/__pycache__/_dists.cpython-311.pyc deleted file mode 100644 index ef2623a..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/metadata/importlib/__pycache__/_dists.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/metadata/importlib/__pycache__/_envs.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/metadata/importlib/__pycache__/_envs.cpython-311.pyc deleted file mode 100644 index b269500..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/metadata/importlib/__pycache__/_envs.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/metadata/importlib/_compat.py b/venv/lib/python3.11/site-packages/pip/_internal/metadata/importlib/_compat.py deleted file mode 100644 index 593bff2..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/metadata/importlib/_compat.py +++ /dev/null @@ -1,55 +0,0 @@ -import importlib.metadata -from typing import Any, Optional, Protocol, cast - - -class BadMetadata(ValueError): - def __init__(self, dist: importlib.metadata.Distribution, *, reason: str) -> None: - self.dist = dist - self.reason = reason - - def __str__(self) -> str: - return f"Bad metadata in {self.dist} ({self.reason})" - - -class BasePath(Protocol): - """A protocol that various path objects conform. - - This exists because importlib.metadata uses both ``pathlib.Path`` and - ``zipfile.Path``, and we need a common base for type hints (Union does not - work well since ``zipfile.Path`` is too new for our linter setup). - - This does not mean to be exhaustive, but only contains things that present - in both classes *that we need*. - """ - - @property - def name(self) -> str: - raise NotImplementedError() - - @property - def parent(self) -> "BasePath": - raise NotImplementedError() - - -def get_info_location(d: importlib.metadata.Distribution) -> Optional[BasePath]: - """Find the path to the distribution's metadata directory. - - HACK: This relies on importlib.metadata's private ``_path`` attribute. Not - all distributions exist on disk, so importlib.metadata is correct to not - expose the attribute as public. But pip's code base is old and not as clean, - so we do this to avoid having to rewrite too many things. Hopefully we can - eliminate this some day. - """ - return getattr(d, "_path", None) - - -def get_dist_name(dist: importlib.metadata.Distribution) -> str: - """Get the distribution's project name. - - The ``name`` attribute is only available in Python 3.10 or later. We are - targeting exactly that, but Mypy does not know this. - """ - name = cast(Any, dist).name - if not isinstance(name, str): - raise BadMetadata(dist, reason="invalid metadata entry 'name'") - return name diff --git a/venv/lib/python3.11/site-packages/pip/_internal/metadata/importlib/_dists.py b/venv/lib/python3.11/site-packages/pip/_internal/metadata/importlib/_dists.py deleted file mode 100644 index 26370fa..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/metadata/importlib/_dists.py +++ /dev/null @@ -1,227 +0,0 @@ -import email.message -import importlib.metadata -import os -import pathlib -import zipfile -from typing import ( - Collection, - Dict, - Iterable, - Iterator, - Mapping, - Optional, - Sequence, - cast, -) - -from pip._vendor.packaging.requirements import Requirement -from pip._vendor.packaging.utils import NormalizedName, canonicalize_name -from pip._vendor.packaging.version import parse as parse_version - -from pip._internal.exceptions import InvalidWheel, UnsupportedWheel -from pip._internal.metadata.base import ( - BaseDistribution, - BaseEntryPoint, - DistributionVersion, - InfoPath, - Wheel, -) -from pip._internal.utils.misc import normalize_path -from pip._internal.utils.temp_dir import TempDirectory -from pip._internal.utils.wheel import parse_wheel, read_wheel_metadata_file - -from ._compat import BasePath, get_dist_name - - -class WheelDistribution(importlib.metadata.Distribution): - """An ``importlib.metadata.Distribution`` read from a wheel. - - Although ``importlib.metadata.PathDistribution`` accepts ``zipfile.Path``, - its implementation is too "lazy" for pip's needs (we can't keep the ZipFile - handle open for the entire lifetime of the distribution object). - - This implementation eagerly reads the entire metadata directory into the - memory instead, and operates from that. - """ - - def __init__( - self, - files: Mapping[pathlib.PurePosixPath, bytes], - info_location: pathlib.PurePosixPath, - ) -> None: - self._files = files - self.info_location = info_location - - @classmethod - def from_zipfile( - cls, - zf: zipfile.ZipFile, - name: str, - location: str, - ) -> "WheelDistribution": - info_dir, _ = parse_wheel(zf, name) - paths = ( - (name, pathlib.PurePosixPath(name.split("/", 1)[-1])) - for name in zf.namelist() - if name.startswith(f"{info_dir}/") - ) - files = { - relpath: read_wheel_metadata_file(zf, fullpath) - for fullpath, relpath in paths - } - info_location = pathlib.PurePosixPath(location, info_dir) - return cls(files, info_location) - - def iterdir(self, path: InfoPath) -> Iterator[pathlib.PurePosixPath]: - # Only allow iterating through the metadata directory. - if pathlib.PurePosixPath(str(path)) in self._files: - return iter(self._files) - raise FileNotFoundError(path) - - def read_text(self, filename: str) -> Optional[str]: - try: - data = self._files[pathlib.PurePosixPath(filename)] - except KeyError: - return None - try: - text = data.decode("utf-8") - except UnicodeDecodeError as e: - wheel = self.info_location.parent - error = f"Error decoding metadata for {wheel}: {e} in {filename} file" - raise UnsupportedWheel(error) - return text - - -class Distribution(BaseDistribution): - def __init__( - self, - dist: importlib.metadata.Distribution, - info_location: Optional[BasePath], - installed_location: Optional[BasePath], - ) -> None: - self._dist = dist - self._info_location = info_location - self._installed_location = installed_location - - @classmethod - def from_directory(cls, directory: str) -> BaseDistribution: - info_location = pathlib.Path(directory) - dist = importlib.metadata.Distribution.at(info_location) - return cls(dist, info_location, info_location.parent) - - @classmethod - def from_metadata_file_contents( - cls, - metadata_contents: bytes, - filename: str, - project_name: str, - ) -> BaseDistribution: - # Generate temp dir to contain the metadata file, and write the file contents. - temp_dir = pathlib.Path( - TempDirectory(kind="metadata", globally_managed=True).path - ) - metadata_path = temp_dir / "METADATA" - metadata_path.write_bytes(metadata_contents) - # Construct dist pointing to the newly created directory. - dist = importlib.metadata.Distribution.at(metadata_path.parent) - return cls(dist, metadata_path.parent, None) - - @classmethod - def from_wheel(cls, wheel: Wheel, name: str) -> BaseDistribution: - try: - with wheel.as_zipfile() as zf: - dist = WheelDistribution.from_zipfile(zf, name, wheel.location) - except zipfile.BadZipFile as e: - raise InvalidWheel(wheel.location, name) from e - except UnsupportedWheel as e: - raise UnsupportedWheel(f"{name} has an invalid wheel, {e}") - return cls(dist, dist.info_location, pathlib.PurePosixPath(wheel.location)) - - @property - def location(self) -> Optional[str]: - if self._info_location is None: - return None - return str(self._info_location.parent) - - @property - def info_location(self) -> Optional[str]: - if self._info_location is None: - return None - return str(self._info_location) - - @property - def installed_location(self) -> Optional[str]: - if self._installed_location is None: - return None - return normalize_path(str(self._installed_location)) - - def _get_dist_name_from_location(self) -> Optional[str]: - """Try to get the name from the metadata directory name. - - This is much faster than reading metadata. - """ - if self._info_location is None: - return None - stem, suffix = os.path.splitext(self._info_location.name) - if suffix not in (".dist-info", ".egg-info"): - return None - return stem.split("-", 1)[0] - - @property - def canonical_name(self) -> NormalizedName: - name = self._get_dist_name_from_location() or get_dist_name(self._dist) - return canonicalize_name(name) - - @property - def version(self) -> DistributionVersion: - return parse_version(self._dist.version) - - def is_file(self, path: InfoPath) -> bool: - return self._dist.read_text(str(path)) is not None - - def iter_distutils_script_names(self) -> Iterator[str]: - # A distutils installation is always "flat" (not in e.g. egg form), so - # if this distribution's info location is NOT a pathlib.Path (but e.g. - # zipfile.Path), it can never contain any distutils scripts. - if not isinstance(self._info_location, pathlib.Path): - return - for child in self._info_location.joinpath("scripts").iterdir(): - yield child.name - - def read_text(self, path: InfoPath) -> str: - content = self._dist.read_text(str(path)) - if content is None: - raise FileNotFoundError(path) - return content - - def iter_entry_points(self) -> Iterable[BaseEntryPoint]: - # importlib.metadata's EntryPoint structure sasitfies BaseEntryPoint. - return self._dist.entry_points - - def _metadata_impl(self) -> email.message.Message: - # From Python 3.10+, importlib.metadata declares PackageMetadata as the - # return type. This protocol is unfortunately a disaster now and misses - # a ton of fields that we need, including get() and get_payload(). We - # rely on the implementation that the object is actually a Message now, - # until upstream can improve the protocol. (python/cpython#94952) - return cast(email.message.Message, self._dist.metadata) - - def iter_provided_extras(self) -> Iterable[str]: - return self.metadata.get_all("Provides-Extra", []) - - def is_extra_provided(self, extra: str) -> bool: - return any( - canonicalize_name(provided_extra) == canonicalize_name(extra) - for provided_extra in self.metadata.get_all("Provides-Extra", []) - ) - - def iter_dependencies(self, extras: Collection[str] = ()) -> Iterable[Requirement]: - contexts: Sequence[Dict[str, str]] = [{"extra": e} for e in extras] - for req_string in self.metadata.get_all("Requires-Dist", []): - req = Requirement(req_string) - if not req.marker: - yield req - elif not extras and req.marker.evaluate({"extra": ""}): - yield req - elif any(req.marker.evaluate(context) for context in contexts): - yield req diff --git a/venv/lib/python3.11/site-packages/pip/_internal/metadata/importlib/_envs.py b/venv/lib/python3.11/site-packages/pip/_internal/metadata/importlib/_envs.py deleted file mode 100644 index 048dc55..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/metadata/importlib/_envs.py +++ /dev/null @@ -1,189 +0,0 @@ -import functools -import importlib.metadata -import logging -import os -import pathlib -import sys -import zipfile -import zipimport -from typing import Iterator, List, Optional, Sequence, Set, Tuple - -from pip._vendor.packaging.utils import NormalizedName, canonicalize_name - -from pip._internal.metadata.base import BaseDistribution, BaseEnvironment -from pip._internal.models.wheel import Wheel -from pip._internal.utils.deprecation import deprecated -from pip._internal.utils.filetypes import WHEEL_EXTENSION - -from ._compat import BadMetadata, BasePath, get_dist_name, get_info_location -from ._dists import Distribution - -logger = logging.getLogger(__name__) - - -def _looks_like_wheel(location: str) -> bool: - if not location.endswith(WHEEL_EXTENSION): - return False - if not os.path.isfile(location): - return False - if not Wheel.wheel_file_re.match(os.path.basename(location)): - return False - return zipfile.is_zipfile(location) - - -class _DistributionFinder: - """Finder to locate distributions. - - The main purpose of this class is to memoize found distributions' names, so - only one distribution is returned for each package name. At lot of pip code - assumes this (because it is setuptools's behavior), and not doing the same - can potentially cause a distribution in lower precedence path to override a - higher precedence one if the caller is not careful. - - Eventually we probably want to make it possible to see lower precedence - installations as well. It's useful feature, after all. - """ - - FoundResult = Tuple[importlib.metadata.Distribution, Optional[BasePath]] - - def __init__(self) -> None: - self._found_names: Set[NormalizedName] = set() - - def _find_impl(self, location: str) -> Iterator[FoundResult]: - """Find distributions in a location.""" - # Skip looking inside a wheel. Since a package inside a wheel is not - # always valid (due to .data directories etc.), its .dist-info entry - # should not be considered an installed distribution. - if _looks_like_wheel(location): - return - # To know exactly where we find a distribution, we have to feed in the - # paths one by one, instead of dumping the list to importlib.metadata. - for dist in importlib.metadata.distributions(path=[location]): - info_location = get_info_location(dist) - try: - raw_name = get_dist_name(dist) - except BadMetadata as e: - logger.warning("Skipping %s due to %s", info_location, e.reason) - continue - normalized_name = canonicalize_name(raw_name) - if normalized_name in self._found_names: - continue - self._found_names.add(normalized_name) - yield dist, info_location - - def find(self, location: str) -> Iterator[BaseDistribution]: - """Find distributions in a location. - - The path can be either a directory, or a ZIP archive. - """ - for dist, info_location in self._find_impl(location): - if info_location is None: - installed_location: Optional[BasePath] = None - else: - installed_location = info_location.parent - yield Distribution(dist, info_location, installed_location) - - def find_linked(self, location: str) -> Iterator[BaseDistribution]: - """Read location in egg-link files and return distributions in there. - - The path should be a directory; otherwise this returns nothing. This - follows how setuptools does this for compatibility. The first non-empty - line in the egg-link is read as a path (resolved against the egg-link's - containing directory if relative). Distributions found at that linked - location are returned. - """ - path = pathlib.Path(location) - if not path.is_dir(): - return - for child in path.iterdir(): - if child.suffix != ".egg-link": - continue - with child.open() as f: - lines = (line.strip() for line in f) - target_rel = next((line for line in lines if line), "") - if not target_rel: - continue - target_location = str(path.joinpath(target_rel)) - for dist, info_location in self._find_impl(target_location): - yield Distribution(dist, info_location, path) - - def _find_eggs_in_dir(self, location: str) -> Iterator[BaseDistribution]: - from pip._vendor.pkg_resources import find_distributions - - from pip._internal.metadata import pkg_resources as legacy - - with os.scandir(location) as it: - for entry in it: - if not entry.name.endswith(".egg"): - continue - for dist in find_distributions(entry.path): - yield legacy.Distribution(dist) - - def _find_eggs_in_zip(self, location: str) -> Iterator[BaseDistribution]: - from pip._vendor.pkg_resources import find_eggs_in_zip - - from pip._internal.metadata import pkg_resources as legacy - - try: - importer = zipimport.zipimporter(location) - except zipimport.ZipImportError: - return - for dist in find_eggs_in_zip(importer, location): - yield legacy.Distribution(dist) - - def find_eggs(self, location: str) -> Iterator[BaseDistribution]: - """Find eggs in a location. - - This actually uses the old *pkg_resources* backend. We likely want to - deprecate this so we can eventually remove the *pkg_resources* - dependency entirely. Before that, this should first emit a deprecation - warning for some versions when using the fallback since importing - *pkg_resources* is slow for those who don't need it. - """ - if os.path.isdir(location): - yield from self._find_eggs_in_dir(location) - if zipfile.is_zipfile(location): - yield from self._find_eggs_in_zip(location) - - -@functools.lru_cache(maxsize=None) # Warn a distribution exactly once. -def _emit_egg_deprecation(location: Optional[str]) -> None: - deprecated( - reason=f"Loading egg at {location} is deprecated.", - replacement="to use pip for package installation.", - gone_in="24.3", - issue=12330, - ) - - -class Environment(BaseEnvironment): - def __init__(self, paths: Sequence[str]) -> None: - self._paths = paths - - @classmethod - def default(cls) -> BaseEnvironment: - return cls(sys.path) - - @classmethod - def from_paths(cls, paths: Optional[List[str]]) -> BaseEnvironment: - if paths is None: - return cls(sys.path) - return cls(paths) - - def _iter_distributions(self) -> Iterator[BaseDistribution]: - finder = _DistributionFinder() - for location in self._paths: - yield from finder.find(location) - for dist in finder.find_eggs(location): - _emit_egg_deprecation(dist.location) - yield dist - # This must go last because that's how pkg_resources tie-breaks. - yield from finder.find_linked(location) - - def get_distribution(self, name: str) -> Optional[BaseDistribution]: - matches = ( - distribution - for distribution in self.iter_all_distributions() - if distribution.canonical_name == canonicalize_name(name) - ) - return next(matches, None) diff --git a/venv/lib/python3.11/site-packages/pip/_internal/metadata/pkg_resources.py b/venv/lib/python3.11/site-packages/pip/_internal/metadata/pkg_resources.py deleted file mode 100644 index bb11e5b..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/metadata/pkg_resources.py +++ /dev/null @@ -1,278 +0,0 @@ -import email.message -import email.parser -import logging -import os -import zipfile -from typing import Collection, Iterable, Iterator, List, Mapping, NamedTuple, Optional - -from pip._vendor import pkg_resources -from pip._vendor.packaging.requirements import Requirement -from pip._vendor.packaging.utils import NormalizedName, canonicalize_name -from pip._vendor.packaging.version import parse as parse_version - -from pip._internal.exceptions import InvalidWheel, NoneMetadataError, UnsupportedWheel -from pip._internal.utils.egg_link import egg_link_path_from_location -from pip._internal.utils.misc import display_path, normalize_path -from pip._internal.utils.wheel import parse_wheel, read_wheel_metadata_file - -from .base import ( - BaseDistribution, - BaseEntryPoint, - BaseEnvironment, - DistributionVersion, - InfoPath, - Wheel, -) - -__all__ = ["NAME", "Distribution", "Environment"] - -logger = logging.getLogger(__name__) - -NAME = "pkg_resources" - - -class EntryPoint(NamedTuple): - name: str - value: str - group: str - - -class InMemoryMetadata: - """IMetadataProvider that reads metadata files from a dictionary. - - This also maps metadata decoding exceptions to our internal exception type. - """ - - def __init__(self, metadata: Mapping[str, bytes], wheel_name: str) -> None: - self._metadata = metadata - self._wheel_name = wheel_name - - def has_metadata(self, name: str) -> bool: - return name in self._metadata - - def get_metadata(self, name: str) -> str: - try: - return self._metadata[name].decode() - except UnicodeDecodeError as e: - # Augment the default error with the origin of the file. - raise UnsupportedWheel( - f"Error decoding metadata for {self._wheel_name}: {e} in {name} file" - ) - - def get_metadata_lines(self, name: str) -> Iterable[str]: - return pkg_resources.yield_lines(self.get_metadata(name)) - - def metadata_isdir(self, name: str) -> bool: - return False - - def metadata_listdir(self, name: str) -> List[str]: - return [] - - def run_script(self, script_name: str, namespace: str) -> None: - pass - - -class Distribution(BaseDistribution): - def __init__(self, dist: pkg_resources.Distribution) -> None: - self._dist = dist - - @classmethod - def from_directory(cls, directory: str) -> BaseDistribution: - dist_dir = directory.rstrip(os.sep) - - # Build a PathMetadata object, from path to metadata. :wink: - base_dir, dist_dir_name = os.path.split(dist_dir) - metadata = pkg_resources.PathMetadata(base_dir, dist_dir) - - # Determine the correct Distribution object type. - if dist_dir.endswith(".egg-info"): - dist_cls = pkg_resources.Distribution - dist_name = os.path.splitext(dist_dir_name)[0] - else: - assert dist_dir.endswith(".dist-info") - dist_cls = pkg_resources.DistInfoDistribution - dist_name = os.path.splitext(dist_dir_name)[0].split("-")[0] - - dist = dist_cls(base_dir, project_name=dist_name, metadata=metadata) - return cls(dist) - - @classmethod - def from_metadata_file_contents( - cls, - metadata_contents: bytes, - filename: str, - project_name: str, - ) -> BaseDistribution: - metadata_dict = { - "METADATA": metadata_contents, - } - dist = pkg_resources.DistInfoDistribution( - location=filename, - metadata=InMemoryMetadata(metadata_dict, filename), - project_name=project_name, - ) - return cls(dist) - - @classmethod - def from_wheel(cls, wheel: Wheel, name: str) -> BaseDistribution: - try: - with wheel.as_zipfile() as zf: - info_dir, _ = parse_wheel(zf, name) - metadata_dict = { - path.split("/", 1)[-1]: read_wheel_metadata_file(zf, path) - for path in zf.namelist() - if path.startswith(f"{info_dir}/") - } - except zipfile.BadZipFile as e: - raise InvalidWheel(wheel.location, name) from e - except UnsupportedWheel as e: - raise UnsupportedWheel(f"{name} has an invalid wheel, {e}") - dist = pkg_resources.DistInfoDistribution( - location=wheel.location, - metadata=InMemoryMetadata(metadata_dict, wheel.location), - project_name=name, - ) - return cls(dist) - - @property - def location(self) -> Optional[str]: - return self._dist.location - - @property - def installed_location(self) -> Optional[str]: - egg_link = egg_link_path_from_location(self.raw_name) - if egg_link: - location = egg_link - elif self.location: - location = self.location - else: - return None - return normalize_path(location) - - @property - def info_location(self) -> Optional[str]: - return self._dist.egg_info - - @property - def installed_by_distutils(self) -> bool: - # A distutils-installed distribution is provided by FileMetadata. This - # provider has a "path" attribute not present anywhere else. Not the - # best introspection logic, but pip has been doing this for a long time. - try: - return bool(self._dist._provider.path) - except AttributeError: - return False - - @property - def canonical_name(self) -> NormalizedName: - return canonicalize_name(self._dist.project_name) - - @property - def version(self) -> DistributionVersion: - return parse_version(self._dist.version) - - def is_file(self, path: InfoPath) -> bool: - return self._dist.has_metadata(str(path)) - - def iter_distutils_script_names(self) -> Iterator[str]: - yield from self._dist.metadata_listdir("scripts") - - def read_text(self, path: InfoPath) -> str: - name = str(path) - if not self._dist.has_metadata(name): - raise FileNotFoundError(name) - content = self._dist.get_metadata(name) - if content is None: - raise NoneMetadataError(self, name) - return content - - def iter_entry_points(self) -> Iterable[BaseEntryPoint]: - for group, entries in self._dist.get_entry_map().items(): - for name, entry_point in entries.items(): - name, _, value = str(entry_point).partition("=") - yield EntryPoint(name=name.strip(), value=value.strip(), group=group) - - def _metadata_impl(self) -> email.message.Message: - """ - :raises NoneMetadataError: if the distribution reports `has_metadata()` - True but `get_metadata()` returns None. - """ - if isinstance(self._dist, pkg_resources.DistInfoDistribution): - metadata_name = "METADATA" - else: - metadata_name = "PKG-INFO" - try: - metadata = self.read_text(metadata_name) - except FileNotFoundError: - if self.location: - displaying_path = display_path(self.location) - else: - displaying_path = repr(self.location) - logger.warning("No metadata found in %s", displaying_path) - metadata = "" - feed_parser = email.parser.FeedParser() - feed_parser.feed(metadata) - return feed_parser.close() - - def iter_dependencies(self, extras: Collection[str] = ()) -> Iterable[Requirement]: - if extras: # pkg_resources raises on invalid extras, so we sanitize. - extras = frozenset(pkg_resources.safe_extra(e) for e in extras) - extras = extras.intersection(self._dist.extras) - return self._dist.requires(extras) - - def iter_provided_extras(self) -> Iterable[str]: - return self._dist.extras - - def is_extra_provided(self, extra: str) -> bool: - return pkg_resources.safe_extra(extra) in self._dist.extras - - -class Environment(BaseEnvironment): - def __init__(self, ws: pkg_resources.WorkingSet) -> None: - self._ws = ws - - @classmethod - def default(cls) -> BaseEnvironment: - return cls(pkg_resources.working_set) - - @classmethod - def from_paths(cls, paths: Optional[List[str]]) -> BaseEnvironment: - return cls(pkg_resources.WorkingSet(paths)) - - def _iter_distributions(self) -> Iterator[BaseDistribution]: - for dist in self._ws: - yield Distribution(dist) - - def _search_distribution(self, name: str) -> Optional[BaseDistribution]: - """Find a distribution matching the ``name`` in the environment. - - This searches from *all* distributions available in the environment, to - match the behavior of ``pkg_resources.get_distribution()``. - """ - canonical_name = canonicalize_name(name) - for dist in self.iter_all_distributions(): - if dist.canonical_name == canonical_name: - return dist - return None - - def get_distribution(self, name: str) -> Optional[BaseDistribution]: - # Search the distribution by looking through the working set. - dist = self._search_distribution(name) - if dist: - return dist - - # If distribution could not be found, call working_set.require to - # update the working set, and try to find the distribution again. - # This might happen for e.g. when you install a package twice, once - # using setup.py develop and again using setup.py install. Now when - # running pip uninstall twice, the package gets removed from the - # working set in the first uninstall, so we have to populate the - # working set again so that pip knows about it and the packages gets - # picked up and is successfully uninstalled the second time too. - try: - # We didn't pass in any version specifiers, so this can never - # raise pkg_resources.VersionConflict. - self._ws.require(name) - except pkg_resources.DistributionNotFound: - return None - return self._search_distribution(name) diff --git a/venv/lib/python3.11/site-packages/pip/_internal/models/__init__.py b/venv/lib/python3.11/site-packages/pip/_internal/models/__init__.py deleted file mode 100644 index 7855226..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/models/__init__.py +++ /dev/null @@ -1,2 +0,0 @@ -"""A package that contains models that represent entities. -""" diff --git a/venv/lib/python3.11/site-packages/pip/_internal/models/__pycache__/__init__.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/models/__pycache__/__init__.cpython-311.pyc deleted file mode 100644 index 16f65b7..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/models/__pycache__/__init__.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/models/__pycache__/candidate.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/models/__pycache__/candidate.cpython-311.pyc deleted file mode 100644 index fe60b17..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/models/__pycache__/candidate.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/models/__pycache__/direct_url.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/models/__pycache__/direct_url.cpython-311.pyc deleted file mode 100644 index 78c190b..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/models/__pycache__/direct_url.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/models/__pycache__/format_control.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/models/__pycache__/format_control.cpython-311.pyc deleted file mode 100644 index c6ecff1..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/models/__pycache__/format_control.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/models/__pycache__/index.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/models/__pycache__/index.cpython-311.pyc deleted file mode 100644 index 18209a1..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/models/__pycache__/index.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/models/__pycache__/installation_report.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/models/__pycache__/installation_report.cpython-311.pyc deleted file mode 100644 index 31a93bc..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/models/__pycache__/installation_report.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/models/__pycache__/link.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/models/__pycache__/link.cpython-311.pyc deleted file mode 100644 index ea1cd09..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/models/__pycache__/link.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/models/__pycache__/scheme.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/models/__pycache__/scheme.cpython-311.pyc deleted file mode 100644 index cab78cb..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/models/__pycache__/scheme.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/models/__pycache__/search_scope.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/models/__pycache__/search_scope.cpython-311.pyc deleted file mode 100644 index bdaa351..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/models/__pycache__/search_scope.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/models/__pycache__/selection_prefs.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/models/__pycache__/selection_prefs.cpython-311.pyc deleted file mode 100644 index 535a67b..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/models/__pycache__/selection_prefs.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/models/__pycache__/target_python.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/models/__pycache__/target_python.cpython-311.pyc deleted file mode 100644 index d02e0d7..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/models/__pycache__/target_python.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/models/__pycache__/wheel.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/models/__pycache__/wheel.cpython-311.pyc deleted file mode 100644 index a48cfdb..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/models/__pycache__/wheel.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/models/candidate.py b/venv/lib/python3.11/site-packages/pip/_internal/models/candidate.py deleted file mode 100644 index a4963ae..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/models/candidate.py +++ /dev/null @@ -1,34 +0,0 @@ -from pip._vendor.packaging.version import parse as parse_version - -from pip._internal.models.link import Link -from pip._internal.utils.models import KeyBasedCompareMixin - - -class InstallationCandidate(KeyBasedCompareMixin): - """Represents a potential "candidate" for installation.""" - - __slots__ = ["name", "version", "link"] - - def __init__(self, name: str, version: str, link: Link) -> None: - self.name = name - self.version = parse_version(version) - self.link = link - - super().__init__( - key=(self.name, self.version, self.link), - defining_class=InstallationCandidate, - ) - - def __repr__(self) -> str: - return "".format( - self.name, - self.version, - self.link, - ) - - def __str__(self) -> str: - return "{!r} candidate (version {} at {})".format( - self.name, - self.version, - self.link, - ) diff --git a/venv/lib/python3.11/site-packages/pip/_internal/models/direct_url.py b/venv/lib/python3.11/site-packages/pip/_internal/models/direct_url.py deleted file mode 100644 index e219d73..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/models/direct_url.py +++ /dev/null @@ -1,237 +0,0 @@ -""" PEP 610 """ -import json -import re -import urllib.parse -from typing import Any, Dict, Iterable, Optional, Type, TypeVar, Union - -__all__ = [ - "DirectUrl", - "DirectUrlValidationError", - "DirInfo", - "ArchiveInfo", - "VcsInfo", -] - -T = TypeVar("T") - -DIRECT_URL_METADATA_NAME = "direct_url.json" -ENV_VAR_RE = re.compile(r"^\$\{[A-Za-z0-9-_]+\}(:\$\{[A-Za-z0-9-_]+\})?$") - - -class DirectUrlValidationError(Exception): - pass - - -def _get( - d: Dict[str, Any], expected_type: Type[T], key: str, default: Optional[T] = None -) -> Optional[T]: - """Get value from dictionary and verify expected type.""" - if key not in d: - return default - value = d[key] - if not isinstance(value, expected_type): - raise DirectUrlValidationError( - "{!r} has unexpected type for {} (expected {})".format( - value, key, expected_type - ) - ) - return value - - -def _get_required( - d: Dict[str, Any], expected_type: Type[T], key: str, default: Optional[T] = None -) -> T: - value = _get(d, expected_type, key, default) - if value is None: - raise DirectUrlValidationError(f"{key} must have a value") - return value - - -def _exactly_one_of(infos: Iterable[Optional["InfoType"]]) -> "InfoType": - infos = [info for info in infos if info is not None] - if not infos: - raise DirectUrlValidationError( - "missing one of archive_info, dir_info, vcs_info" - ) - if len(infos) > 1: - raise DirectUrlValidationError( - "more than one of archive_info, dir_info, vcs_info" - ) - assert infos[0] is not None - return infos[0] - - -def _filter_none(**kwargs: Any) -> Dict[str, Any]: - """Make dict excluding None values.""" - return {k: v for k, v in kwargs.items() if v is not None} - - -class VcsInfo: - name = "vcs_info" - - def __init__( - self, - vcs: str, - commit_id: str, - requested_revision: Optional[str] = None, - ) -> None: - self.vcs = vcs - self.requested_revision = requested_revision - self.commit_id = commit_id - - @classmethod - def _from_dict(cls, d: Optional[Dict[str, Any]]) -> Optional["VcsInfo"]: - if d is None: - return None - return cls( - vcs=_get_required(d, str, "vcs"), - commit_id=_get_required(d, str, "commit_id"), - requested_revision=_get(d, str, "requested_revision"), - ) - - def _to_dict(self) -> Dict[str, Any]: - return _filter_none( - vcs=self.vcs, - requested_revision=self.requested_revision, - commit_id=self.commit_id, - ) - - -class ArchiveInfo: - name = "archive_info" - - def __init__( - self, - hash: Optional[str] = None, - hashes: Optional[Dict[str, str]] = None, - ) -> None: - # set hashes before hash, since the hash setter will further populate hashes - self.hashes = hashes - self.hash = hash - - @property - def hash(self) -> Optional[str]: - return self._hash - - @hash.setter - def hash(self, value: Optional[str]) -> None: - if value is not None: - # Auto-populate the hashes key to upgrade to the new format automatically. - # We don't back-populate the legacy hash key from hashes. - try: - hash_name, hash_value = value.split("=", 1) - except ValueError: - raise DirectUrlValidationError( - f"invalid archive_info.hash format: {value!r}" - ) - if self.hashes is None: - self.hashes = {hash_name: hash_value} - elif hash_name not in self.hashes: - self.hashes = self.hashes.copy() - self.hashes[hash_name] = hash_value - self._hash = value - - @classmethod - def _from_dict(cls, d: Optional[Dict[str, Any]]) -> Optional["ArchiveInfo"]: - if d is None: - return None - return cls(hash=_get(d, str, "hash"), hashes=_get(d, dict, "hashes")) - - def _to_dict(self) -> Dict[str, Any]: - return _filter_none(hash=self.hash, hashes=self.hashes) - - -class DirInfo: - name = "dir_info" - - def __init__( - self, - editable: bool = False, - ) -> None: - self.editable = editable - - @classmethod - def _from_dict(cls, d: Optional[Dict[str, Any]]) -> Optional["DirInfo"]: - if d is None: - return None - return cls(editable=_get_required(d, bool, "editable", default=False)) - - def _to_dict(self) -> Dict[str, Any]: - return _filter_none(editable=self.editable or None) - - -InfoType = Union[ArchiveInfo, DirInfo, VcsInfo] - - -class DirectUrl: - def __init__( - self, - url: str, - info: InfoType, - subdirectory: Optional[str] = None, - ) -> None: - self.url = url - self.info = info - self.subdirectory = subdirectory - - def _remove_auth_from_netloc(self, netloc: str) -> str: - if "@" not in netloc: - return netloc - user_pass, netloc_no_user_pass = netloc.split("@", 1) - if ( - isinstance(self.info, VcsInfo) - and self.info.vcs == "git" - and user_pass == "git" - ): - return netloc - if ENV_VAR_RE.match(user_pass): - return netloc - return netloc_no_user_pass - - @property - def redacted_url(self) -> str: - """url with user:password part removed unless it is formed with - environment variables as specified in PEP 610, or it is ``git`` - in the case of a git URL. - """ - purl = urllib.parse.urlsplit(self.url) - netloc = self._remove_auth_from_netloc(purl.netloc) - surl = urllib.parse.urlunsplit( - (purl.scheme, netloc, purl.path, purl.query, purl.fragment) - ) - return surl - - def validate(self) -> None: - self.from_dict(self.to_dict()) - - @classmethod - def from_dict(cls, d: Dict[str, Any]) -> "DirectUrl": - return DirectUrl( - url=_get_required(d, str, "url"), - subdirectory=_get(d, str, "subdirectory"), - info=_exactly_one_of( - [ - ArchiveInfo._from_dict(_get(d, dict, "archive_info")), - DirInfo._from_dict(_get(d, dict, "dir_info")), - VcsInfo._from_dict(_get(d, dict, "vcs_info")), - ] - ), - ) - - def to_dict(self) -> Dict[str, Any]: - res = _filter_none( - url=self.redacted_url, - subdirectory=self.subdirectory, - ) - res[self.info.name] = self.info._to_dict() - return res - - @classmethod - def from_json(cls, s: str) -> "DirectUrl": - return cls.from_dict(json.loads(s)) - - def to_json(self) -> str: - return json.dumps(self.to_dict(), sort_keys=True) - - def is_local_editable(self) -> bool: - return isinstance(self.info, DirInfo) and self.info.editable diff --git a/venv/lib/python3.11/site-packages/pip/_internal/models/format_control.py b/venv/lib/python3.11/site-packages/pip/_internal/models/format_control.py deleted file mode 100644 index db3995e..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/models/format_control.py +++ /dev/null @@ -1,80 +0,0 @@ -from typing import FrozenSet, Optional, Set - -from pip._vendor.packaging.utils import canonicalize_name - -from pip._internal.exceptions import CommandError - - -class FormatControl: - """Helper for managing formats from which a package can be installed.""" - - __slots__ = ["no_binary", "only_binary"] - - def __init__( - self, - no_binary: Optional[Set[str]] = None, - only_binary: Optional[Set[str]] = None, - ) -> None: - if no_binary is None: - no_binary = set() - if only_binary is None: - only_binary = set() - - self.no_binary = no_binary - self.only_binary = only_binary - - def __eq__(self, other: object) -> bool: - if not isinstance(other, self.__class__): - return NotImplemented - - if self.__slots__ != other.__slots__: - return False - - return all(getattr(self, k) == getattr(other, k) for k in self.__slots__) - - def __repr__(self) -> str: - return "{}({}, {})".format( - self.__class__.__name__, self.no_binary, self.only_binary - ) - - @staticmethod - def handle_mutual_excludes(value: str, target: Set[str], other: Set[str]) -> None: - if value.startswith("-"): - raise CommandError( - "--no-binary / --only-binary option requires 1 argument." - ) - new = value.split(",") - while ":all:" in new: - other.clear() - target.clear() - target.add(":all:") - del new[: new.index(":all:") + 1] - # Without a none, we want to discard everything as :all: covers it - if ":none:" not in new: - return - for name in new: - if name == ":none:": - target.clear() - continue - name = canonicalize_name(name) - other.discard(name) - target.add(name) - - def get_allowed_formats(self, canonical_name: str) -> FrozenSet[str]: - result = {"binary", "source"} - if canonical_name in self.only_binary: - result.discard("source") - elif canonical_name in self.no_binary: - result.discard("binary") - elif ":all:" in self.only_binary: - result.discard("source") - elif ":all:" in self.no_binary: - result.discard("binary") - return frozenset(result) - - def disallow_binaries(self) -> None: - self.handle_mutual_excludes( - ":all:", - self.no_binary, - self.only_binary, - ) diff --git a/venv/lib/python3.11/site-packages/pip/_internal/models/index.py b/venv/lib/python3.11/site-packages/pip/_internal/models/index.py deleted file mode 100644 index b94c325..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/models/index.py +++ /dev/null @@ -1,28 +0,0 @@ -import urllib.parse - - -class PackageIndex: - """Represents a Package Index and provides easier access to endpoints""" - - __slots__ = ["url", "netloc", "simple_url", "pypi_url", "file_storage_domain"] - - def __init__(self, url: str, file_storage_domain: str) -> None: - super().__init__() - self.url = url - self.netloc = urllib.parse.urlsplit(url).netloc - self.simple_url = self._url_for_path("simple") - self.pypi_url = self._url_for_path("pypi") - - # This is part of a temporary hack used to block installs of PyPI - # packages which depend on external urls only necessary until PyPI can - # block such packages themselves - self.file_storage_domain = file_storage_domain - - def _url_for_path(self, path: str) -> str: - return urllib.parse.urljoin(self.url, path) - - -PyPI = PackageIndex("https://pypi.org/", file_storage_domain="files.pythonhosted.org") -TestPyPI = PackageIndex( - "https://test.pypi.org/", file_storage_domain="test-files.pythonhosted.org" -) diff --git a/venv/lib/python3.11/site-packages/pip/_internal/models/installation_report.py b/venv/lib/python3.11/site-packages/pip/_internal/models/installation_report.py deleted file mode 100644 index b9c6330..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/models/installation_report.py +++ /dev/null @@ -1,56 +0,0 @@ -from typing import Any, Dict, Sequence - -from pip._vendor.packaging.markers import default_environment - -from pip import __version__ -from pip._internal.req.req_install import InstallRequirement - - -class InstallationReport: - def __init__(self, install_requirements: Sequence[InstallRequirement]): - self._install_requirements = install_requirements - - @classmethod - def _install_req_to_dict(cls, ireq: InstallRequirement) -> Dict[str, Any]: - assert ireq.download_info, f"No download_info for {ireq}" - res = { - # PEP 610 json for the download URL. download_info.archive_info.hashes may - # be absent when the requirement was installed from the wheel cache - # and the cache entry was populated by an older pip version that did not - # record origin.json. - "download_info": ireq.download_info.to_dict(), - # is_direct is true if the requirement was a direct URL reference (which - # includes editable requirements), and false if the requirement was - # downloaded from a PEP 503 index or --find-links. - "is_direct": ireq.is_direct, - # is_yanked is true if the requirement was yanked from the index, but - # was still selected by pip to conform to PEP 592. - "is_yanked": ireq.link.is_yanked if ireq.link else False, - # requested is true if the requirement was specified by the user (aka - # top level requirement), and false if it was installed as a dependency of a - # requirement. https://peps.python.org/pep-0376/#requested - "requested": ireq.user_supplied, - # PEP 566 json encoding for metadata - # https://www.python.org/dev/peps/pep-0566/#json-compatible-metadata - "metadata": ireq.get_dist().metadata_dict, - } - if ireq.user_supplied and ireq.extras: - # For top level requirements, the list of requested extras, if any. - res["requested_extras"] = sorted(ireq.extras) - return res - - def to_dict(self) -> Dict[str, Any]: - return { - "version": "1", - "pip_version": __version__, - "install": [ - self._install_req_to_dict(ireq) for ireq in self._install_requirements - ], - # https://peps.python.org/pep-0508/#environment-markers - # TODO: currently, the resolver uses the default environment to evaluate - # environment markers, so that is what we report here. In the future, it - # should also take into account options such as --python-version or - # --platform, perhaps under the form of an environment_override field? - # https://github.com/pypa/pip/issues/11198 - "environment": default_environment(), - } diff --git a/venv/lib/python3.11/site-packages/pip/_internal/models/link.py b/venv/lib/python3.11/site-packages/pip/_internal/models/link.py deleted file mode 100644 index 4453519..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/models/link.py +++ /dev/null @@ -1,581 +0,0 @@ -import functools -import itertools -import logging -import os -import posixpath -import re -import urllib.parse -from dataclasses import dataclass -from typing import ( - TYPE_CHECKING, - Any, - Dict, - List, - Mapping, - NamedTuple, - Optional, - Tuple, - Union, -) - -from pip._internal.utils.deprecation import deprecated -from pip._internal.utils.filetypes import WHEEL_EXTENSION -from pip._internal.utils.hashes import Hashes -from pip._internal.utils.misc import ( - pairwise, - redact_auth_from_url, - split_auth_from_netloc, - splitext, -) -from pip._internal.utils.models import KeyBasedCompareMixin -from pip._internal.utils.urls import path_to_url, url_to_path - -if TYPE_CHECKING: - from pip._internal.index.collector import IndexContent - -logger = logging.getLogger(__name__) - - -# Order matters, earlier hashes have a precedence over later hashes for what -# we will pick to use. -_SUPPORTED_HASHES = ("sha512", "sha384", "sha256", "sha224", "sha1", "md5") - - -@dataclass(frozen=True) -class LinkHash: - """Links to content may have embedded hash values. This class parses those. - - `name` must be any member of `_SUPPORTED_HASHES`. - - This class can be converted to and from `ArchiveInfo`. While ArchiveInfo intends to - be JSON-serializable to conform to PEP 610, this class contains the logic for - parsing a hash name and value for correctness, and then checking whether that hash - conforms to a schema with `.is_hash_allowed()`.""" - - name: str - value: str - - _hash_url_fragment_re = re.compile( - # NB: we do not validate that the second group (.*) is a valid hex - # digest. Instead, we simply keep that string in this class, and then check it - # against Hashes when hash-checking is needed. This is easier to debug than - # proactively discarding an invalid hex digest, as we handle incorrect hashes - # and malformed hashes in the same place. - r"[#&]({choices})=([^&]*)".format( - choices="|".join(re.escape(hash_name) for hash_name in _SUPPORTED_HASHES) - ), - ) - - def __post_init__(self) -> None: - assert self.name in _SUPPORTED_HASHES - - @classmethod - @functools.lru_cache(maxsize=None) - def find_hash_url_fragment(cls, url: str) -> Optional["LinkHash"]: - """Search a string for a checksum algorithm name and encoded output value.""" - match = cls._hash_url_fragment_re.search(url) - if match is None: - return None - name, value = match.groups() - return cls(name=name, value=value) - - def as_dict(self) -> Dict[str, str]: - return {self.name: self.value} - - def as_hashes(self) -> Hashes: - """Return a Hashes instance which checks only for the current hash.""" - return Hashes({self.name: [self.value]}) - - def is_hash_allowed(self, hashes: Optional[Hashes]) -> bool: - """ - Return True if the current hash is allowed by `hashes`. - """ - if hashes is None: - return False - return hashes.is_hash_allowed(self.name, hex_digest=self.value) - - -@dataclass(frozen=True) -class MetadataFile: - """Information about a core metadata file associated with a distribution.""" - - hashes: Optional[Dict[str, str]] - - def __post_init__(self) -> None: - if self.hashes is not None: - assert all(name in _SUPPORTED_HASHES for name in self.hashes) - - -def supported_hashes(hashes: Optional[Dict[str, str]]) -> Optional[Dict[str, str]]: - # Remove any unsupported hash types from the mapping. If this leaves no - # supported hashes, return None - if hashes is None: - return None - hashes = {n: v for n, v in hashes.items() if n in _SUPPORTED_HASHES} - if not hashes: - return None - return hashes - - -def _clean_url_path_part(part: str) -> str: - """ - Clean a "part" of a URL path (i.e. after splitting on "@" characters). - """ - # We unquote prior to quoting to make sure nothing is double quoted. - return urllib.parse.quote(urllib.parse.unquote(part)) - - -def _clean_file_url_path(part: str) -> str: - """ - Clean the first part of a URL path that corresponds to a local - filesystem path (i.e. the first part after splitting on "@" characters). - """ - # We unquote prior to quoting to make sure nothing is double quoted. - # Also, on Windows the path part might contain a drive letter which - # should not be quoted. On Linux where drive letters do not - # exist, the colon should be quoted. We rely on urllib.request - # to do the right thing here. - return urllib.request.pathname2url(urllib.request.url2pathname(part)) - - -# percent-encoded: / -_reserved_chars_re = re.compile("(@|%2F)", re.IGNORECASE) - - -def _clean_url_path(path: str, is_local_path: bool) -> str: - """ - Clean the path portion of a URL. - """ - if is_local_path: - clean_func = _clean_file_url_path - else: - clean_func = _clean_url_path_part - - # Split on the reserved characters prior to cleaning so that - # revision strings in VCS URLs are properly preserved. - parts = _reserved_chars_re.split(path) - - cleaned_parts = [] - for to_clean, reserved in pairwise(itertools.chain(parts, [""])): - cleaned_parts.append(clean_func(to_clean)) - # Normalize %xx escapes (e.g. %2f -> %2F) - cleaned_parts.append(reserved.upper()) - - return "".join(cleaned_parts) - - -def _ensure_quoted_url(url: str) -> str: - """ - Make sure a link is fully quoted. - For example, if ' ' occurs in the URL, it will be replaced with "%20", - and without double-quoting other characters. - """ - # Split the URL into parts according to the general structure - # `scheme://netloc/path;parameters?query#fragment`. - result = urllib.parse.urlparse(url) - # If the netloc is empty, then the URL refers to a local filesystem path. - is_local_path = not result.netloc - path = _clean_url_path(result.path, is_local_path=is_local_path) - return urllib.parse.urlunparse(result._replace(path=path)) - - -class Link(KeyBasedCompareMixin): - """Represents a parsed link from a Package Index's simple URL""" - - __slots__ = [ - "_parsed_url", - "_url", - "_hashes", - "comes_from", - "requires_python", - "yanked_reason", - "metadata_file_data", - "cache_link_parsing", - "egg_fragment", - ] - - def __init__( - self, - url: str, - comes_from: Optional[Union[str, "IndexContent"]] = None, - requires_python: Optional[str] = None, - yanked_reason: Optional[str] = None, - metadata_file_data: Optional[MetadataFile] = None, - cache_link_parsing: bool = True, - hashes: Optional[Mapping[str, str]] = None, - ) -> None: - """ - :param url: url of the resource pointed to (href of the link) - :param comes_from: instance of IndexContent where the link was found, - or string. - :param requires_python: String containing the `Requires-Python` - metadata field, specified in PEP 345. This may be specified by - a data-requires-python attribute in the HTML link tag, as - described in PEP 503. - :param yanked_reason: the reason the file has been yanked, if the - file has been yanked, or None if the file hasn't been yanked. - This is the value of the "data-yanked" attribute, if present, in - a simple repository HTML link. If the file has been yanked but - no reason was provided, this should be the empty string. See - PEP 592 for more information and the specification. - :param metadata_file_data: the metadata attached to the file, or None if - no such metadata is provided. This argument, if not None, indicates - that a separate metadata file exists, and also optionally supplies - hashes for that file. - :param cache_link_parsing: A flag that is used elsewhere to determine - whether resources retrieved from this link should be cached. PyPI - URLs should generally have this set to False, for example. - :param hashes: A mapping of hash names to digests to allow us to - determine the validity of a download. - """ - - # The comes_from, requires_python, and metadata_file_data arguments are - # only used by classmethods of this class, and are not used in client - # code directly. - - # url can be a UNC windows share - if url.startswith("\\\\"): - url = path_to_url(url) - - self._parsed_url = urllib.parse.urlsplit(url) - # Store the url as a private attribute to prevent accidentally - # trying to set a new value. - self._url = url - - link_hash = LinkHash.find_hash_url_fragment(url) - hashes_from_link = {} if link_hash is None else link_hash.as_dict() - if hashes is None: - self._hashes = hashes_from_link - else: - self._hashes = {**hashes, **hashes_from_link} - - self.comes_from = comes_from - self.requires_python = requires_python if requires_python else None - self.yanked_reason = yanked_reason - self.metadata_file_data = metadata_file_data - - super().__init__(key=url, defining_class=Link) - - self.cache_link_parsing = cache_link_parsing - self.egg_fragment = self._egg_fragment() - - @classmethod - def from_json( - cls, - file_data: Dict[str, Any], - page_url: str, - ) -> Optional["Link"]: - """ - Convert an pypi json document from a simple repository page into a Link. - """ - file_url = file_data.get("url") - if file_url is None: - return None - - url = _ensure_quoted_url(urllib.parse.urljoin(page_url, file_url)) - pyrequire = file_data.get("requires-python") - yanked_reason = file_data.get("yanked") - hashes = file_data.get("hashes", {}) - - # PEP 714: Indexes must use the name core-metadata, but - # clients should support the old name as a fallback for compatibility. - metadata_info = file_data.get("core-metadata") - if metadata_info is None: - metadata_info = file_data.get("dist-info-metadata") - - # The metadata info value may be a boolean, or a dict of hashes. - if isinstance(metadata_info, dict): - # The file exists, and hashes have been supplied - metadata_file_data = MetadataFile(supported_hashes(metadata_info)) - elif metadata_info: - # The file exists, but there are no hashes - metadata_file_data = MetadataFile(None) - else: - # False or not present: the file does not exist - metadata_file_data = None - - # The Link.yanked_reason expects an empty string instead of a boolean. - if yanked_reason and not isinstance(yanked_reason, str): - yanked_reason = "" - # The Link.yanked_reason expects None instead of False. - elif not yanked_reason: - yanked_reason = None - - return cls( - url, - comes_from=page_url, - requires_python=pyrequire, - yanked_reason=yanked_reason, - hashes=hashes, - metadata_file_data=metadata_file_data, - ) - - @classmethod - def from_element( - cls, - anchor_attribs: Dict[str, Optional[str]], - page_url: str, - base_url: str, - ) -> Optional["Link"]: - """ - Convert an anchor element's attributes in a simple repository page to a Link. - """ - href = anchor_attribs.get("href") - if not href: - return None - - url = _ensure_quoted_url(urllib.parse.urljoin(base_url, href)) - pyrequire = anchor_attribs.get("data-requires-python") - yanked_reason = anchor_attribs.get("data-yanked") - - # PEP 714: Indexes must use the name data-core-metadata, but - # clients should support the old name as a fallback for compatibility. - metadata_info = anchor_attribs.get("data-core-metadata") - if metadata_info is None: - metadata_info = anchor_attribs.get("data-dist-info-metadata") - # The metadata info value may be the string "true", or a string of - # the form "hashname=hashval" - if metadata_info == "true": - # The file exists, but there are no hashes - metadata_file_data = MetadataFile(None) - elif metadata_info is None: - # The file does not exist - metadata_file_data = None - else: - # The file exists, and hashes have been supplied - hashname, sep, hashval = metadata_info.partition("=") - if sep == "=": - metadata_file_data = MetadataFile(supported_hashes({hashname: hashval})) - else: - # Error - data is wrong. Treat as no hashes supplied. - logger.debug( - "Index returned invalid data-dist-info-metadata value: %s", - metadata_info, - ) - metadata_file_data = MetadataFile(None) - - return cls( - url, - comes_from=page_url, - requires_python=pyrequire, - yanked_reason=yanked_reason, - metadata_file_data=metadata_file_data, - ) - - def __str__(self) -> str: - if self.requires_python: - rp = f" (requires-python:{self.requires_python})" - else: - rp = "" - if self.comes_from: - return "{} (from {}){}".format( - redact_auth_from_url(self._url), self.comes_from, rp - ) - else: - return redact_auth_from_url(str(self._url)) - - def __repr__(self) -> str: - return f"" - - @property - def url(self) -> str: - return self._url - - @property - def filename(self) -> str: - path = self.path.rstrip("/") - name = posixpath.basename(path) - if not name: - # Make sure we don't leak auth information if the netloc - # includes a username and password. - netloc, user_pass = split_auth_from_netloc(self.netloc) - return netloc - - name = urllib.parse.unquote(name) - assert name, f"URL {self._url!r} produced no filename" - return name - - @property - def file_path(self) -> str: - return url_to_path(self.url) - - @property - def scheme(self) -> str: - return self._parsed_url.scheme - - @property - def netloc(self) -> str: - """ - This can contain auth information. - """ - return self._parsed_url.netloc - - @property - def path(self) -> str: - return urllib.parse.unquote(self._parsed_url.path) - - def splitext(self) -> Tuple[str, str]: - return splitext(posixpath.basename(self.path.rstrip("/"))) - - @property - def ext(self) -> str: - return self.splitext()[1] - - @property - def url_without_fragment(self) -> str: - scheme, netloc, path, query, fragment = self._parsed_url - return urllib.parse.urlunsplit((scheme, netloc, path, query, "")) - - _egg_fragment_re = re.compile(r"[#&]egg=([^&]*)") - - # Per PEP 508. - _project_name_re = re.compile( - r"^([A-Z0-9]|[A-Z0-9][A-Z0-9._-]*[A-Z0-9])$", re.IGNORECASE - ) - - def _egg_fragment(self) -> Optional[str]: - match = self._egg_fragment_re.search(self._url) - if not match: - return None - - # An egg fragment looks like a PEP 508 project name, along with - # an optional extras specifier. Anything else is invalid. - project_name = match.group(1) - if not self._project_name_re.match(project_name): - deprecated( - reason=f"{self} contains an egg fragment with a non-PEP 508 name", - replacement="to use the req @ url syntax, and remove the egg fragment", - gone_in="25.0", - issue=11617, - ) - - return project_name - - _subdirectory_fragment_re = re.compile(r"[#&]subdirectory=([^&]*)") - - @property - def subdirectory_fragment(self) -> Optional[str]: - match = self._subdirectory_fragment_re.search(self._url) - if not match: - return None - return match.group(1) - - def metadata_link(self) -> Optional["Link"]: - """Return a link to the associated core metadata file (if any).""" - if self.metadata_file_data is None: - return None - metadata_url = f"{self.url_without_fragment}.metadata" - if self.metadata_file_data.hashes is None: - return Link(metadata_url) - return Link(metadata_url, hashes=self.metadata_file_data.hashes) - - def as_hashes(self) -> Hashes: - return Hashes({k: [v] for k, v in self._hashes.items()}) - - @property - def hash(self) -> Optional[str]: - return next(iter(self._hashes.values()), None) - - @property - def hash_name(self) -> Optional[str]: - return next(iter(self._hashes), None) - - @property - def show_url(self) -> str: - return posixpath.basename(self._url.split("#", 1)[0].split("?", 1)[0]) - - @property - def is_file(self) -> bool: - return self.scheme == "file" - - def is_existing_dir(self) -> bool: - return self.is_file and os.path.isdir(self.file_path) - - @property - def is_wheel(self) -> bool: - return self.ext == WHEEL_EXTENSION - - @property - def is_vcs(self) -> bool: - from pip._internal.vcs import vcs - - return self.scheme in vcs.all_schemes - - @property - def is_yanked(self) -> bool: - return self.yanked_reason is not None - - @property - def has_hash(self) -> bool: - return bool(self._hashes) - - def is_hash_allowed(self, hashes: Optional[Hashes]) -> bool: - """ - Return True if the link has a hash and it is allowed by `hashes`. - """ - if hashes is None: - return False - return any(hashes.is_hash_allowed(k, v) for k, v in self._hashes.items()) - - -class _CleanResult(NamedTuple): - """Convert link for equivalency check. - - This is used in the resolver to check whether two URL-specified requirements - likely point to the same distribution and can be considered equivalent. This - equivalency logic avoids comparing URLs literally, which can be too strict - (e.g. "a=1&b=2" vs "b=2&a=1") and produce conflicts unexpecting to users. - - Currently this does three things: - - 1. Drop the basic auth part. This is technically wrong since a server can - serve different content based on auth, but if it does that, it is even - impossible to guarantee two URLs without auth are equivalent, since - the user can input different auth information when prompted. So the - practical solution is to assume the auth doesn't affect the response. - 2. Parse the query to avoid the ordering issue. Note that ordering under the - same key in the query are NOT cleaned; i.e. "a=1&a=2" and "a=2&a=1" are - still considered different. - 3. Explicitly drop most of the fragment part, except ``subdirectory=`` and - hash values, since it should have no impact the downloaded content. Note - that this drops the "egg=" part historically used to denote the requested - project (and extras), which is wrong in the strictest sense, but too many - people are supplying it inconsistently to cause superfluous resolution - conflicts, so we choose to also ignore them. - """ - - parsed: urllib.parse.SplitResult - query: Dict[str, List[str]] - subdirectory: str - hashes: Dict[str, str] - - -def _clean_link(link: Link) -> _CleanResult: - parsed = link._parsed_url - netloc = parsed.netloc.rsplit("@", 1)[-1] - # According to RFC 8089, an empty host in file: means localhost. - if parsed.scheme == "file" and not netloc: - netloc = "localhost" - fragment = urllib.parse.parse_qs(parsed.fragment) - if "egg" in fragment: - logger.debug("Ignoring egg= fragment in %s", link) - try: - # If there are multiple subdirectory values, use the first one. - # This matches the behavior of Link.subdirectory_fragment. - subdirectory = fragment["subdirectory"][0] - except (IndexError, KeyError): - subdirectory = "" - # If there are multiple hash values under the same algorithm, use the - # first one. This matches the behavior of Link.hash_value. - hashes = {k: fragment[k][0] for k in _SUPPORTED_HASHES if k in fragment} - return _CleanResult( - parsed=parsed._replace(netloc=netloc, query="", fragment=""), - query=urllib.parse.parse_qs(parsed.query), - subdirectory=subdirectory, - hashes=hashes, - ) - - -@functools.lru_cache(maxsize=None) -def links_equivalent(link1: Link, link2: Link) -> bool: - return _clean_link(link1) == _clean_link(link2) diff --git a/venv/lib/python3.11/site-packages/pip/_internal/models/scheme.py b/venv/lib/python3.11/site-packages/pip/_internal/models/scheme.py deleted file mode 100644 index f51190a..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/models/scheme.py +++ /dev/null @@ -1,31 +0,0 @@ -""" -For types associated with installation schemes. - -For a general overview of available schemes and their context, see -https://docs.python.org/3/install/index.html#alternate-installation. -""" - - -SCHEME_KEYS = ["platlib", "purelib", "headers", "scripts", "data"] - - -class Scheme: - """A Scheme holds paths which are used as the base directories for - artifacts associated with a Python package. - """ - - __slots__ = SCHEME_KEYS - - def __init__( - self, - platlib: str, - purelib: str, - headers: str, - scripts: str, - data: str, - ) -> None: - self.platlib = platlib - self.purelib = purelib - self.headers = headers - self.scripts = scripts - self.data = data diff --git a/venv/lib/python3.11/site-packages/pip/_internal/models/search_scope.py b/venv/lib/python3.11/site-packages/pip/_internal/models/search_scope.py deleted file mode 100644 index fe61e81..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/models/search_scope.py +++ /dev/null @@ -1,132 +0,0 @@ -import itertools -import logging -import os -import posixpath -import urllib.parse -from typing import List - -from pip._vendor.packaging.utils import canonicalize_name - -from pip._internal.models.index import PyPI -from pip._internal.utils.compat import has_tls -from pip._internal.utils.misc import normalize_path, redact_auth_from_url - -logger = logging.getLogger(__name__) - - -class SearchScope: - - """ - Encapsulates the locations that pip is configured to search. - """ - - __slots__ = ["find_links", "index_urls", "no_index"] - - @classmethod - def create( - cls, - find_links: List[str], - index_urls: List[str], - no_index: bool, - ) -> "SearchScope": - """ - Create a SearchScope object after normalizing the `find_links`. - """ - # Build find_links. If an argument starts with ~, it may be - # a local file relative to a home directory. So try normalizing - # it and if it exists, use the normalized version. - # This is deliberately conservative - it might be fine just to - # blindly normalize anything starting with a ~... - built_find_links: List[str] = [] - for link in find_links: - if link.startswith("~"): - new_link = normalize_path(link) - if os.path.exists(new_link): - link = new_link - built_find_links.append(link) - - # If we don't have TLS enabled, then WARN if anyplace we're looking - # relies on TLS. - if not has_tls(): - for link in itertools.chain(index_urls, built_find_links): - parsed = urllib.parse.urlparse(link) - if parsed.scheme == "https": - logger.warning( - "pip is configured with locations that require " - "TLS/SSL, however the ssl module in Python is not " - "available." - ) - break - - return cls( - find_links=built_find_links, - index_urls=index_urls, - no_index=no_index, - ) - - def __init__( - self, - find_links: List[str], - index_urls: List[str], - no_index: bool, - ) -> None: - self.find_links = find_links - self.index_urls = index_urls - self.no_index = no_index - - def get_formatted_locations(self) -> str: - lines = [] - redacted_index_urls = [] - if self.index_urls and self.index_urls != [PyPI.simple_url]: - for url in self.index_urls: - redacted_index_url = redact_auth_from_url(url) - - # Parse the URL - purl = urllib.parse.urlsplit(redacted_index_url) - - # URL is generally invalid if scheme and netloc is missing - # there are issues with Python and URL parsing, so this test - # is a bit crude. See bpo-20271, bpo-23505. Python doesn't - # always parse invalid URLs correctly - it should raise - # exceptions for malformed URLs - if not purl.scheme and not purl.netloc: - logger.warning( - 'The index url "%s" seems invalid, please provide a scheme.', - redacted_index_url, - ) - - redacted_index_urls.append(redacted_index_url) - - lines.append( - "Looking in indexes: {}".format(", ".join(redacted_index_urls)) - ) - - if self.find_links: - lines.append( - "Looking in links: {}".format( - ", ".join(redact_auth_from_url(url) for url in self.find_links) - ) - ) - return "\n".join(lines) - - def get_index_urls_locations(self, project_name: str) -> List[str]: - """Returns the locations found via self.index_urls - - Checks the url_name on the main (first in the list) index and - use this url_name to produce all locations - """ - - def mkurl_pypi_url(url: str) -> str: - loc = posixpath.join( - url, urllib.parse.quote(canonicalize_name(project_name)) - ) - # For maximum compatibility with easy_install, ensure the path - # ends in a trailing slash. Although this isn't in the spec - # (and PyPI can handle it without the slash) some other index - # implementations might break if they relied on easy_install's - # behavior. - if not loc.endswith("/"): - loc = loc + "/" - return loc - - return [mkurl_pypi_url(url) for url in self.index_urls] diff --git a/venv/lib/python3.11/site-packages/pip/_internal/models/selection_prefs.py b/venv/lib/python3.11/site-packages/pip/_internal/models/selection_prefs.py deleted file mode 100644 index 977bc4c..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/models/selection_prefs.py +++ /dev/null @@ -1,51 +0,0 @@ -from typing import Optional - -from pip._internal.models.format_control import FormatControl - - -class SelectionPreferences: - """ - Encapsulates the candidate selection preferences for downloading - and installing files. - """ - - __slots__ = [ - "allow_yanked", - "allow_all_prereleases", - "format_control", - "prefer_binary", - "ignore_requires_python", - ] - - # Don't include an allow_yanked default value to make sure each call - # site considers whether yanked releases are allowed. This also causes - # that decision to be made explicit in the calling code, which helps - # people when reading the code. - def __init__( - self, - allow_yanked: bool, - allow_all_prereleases: bool = False, - format_control: Optional[FormatControl] = None, - prefer_binary: bool = False, - ignore_requires_python: Optional[bool] = None, - ) -> None: - """Create a SelectionPreferences object. - - :param allow_yanked: Whether files marked as yanked (in the sense - of PEP 592) are permitted to be candidates for install. - :param format_control: A FormatControl object or None. Used to control - the selection of source packages / binary packages when consulting - the index and links. - :param prefer_binary: Whether to prefer an old, but valid, binary - dist over a new source dist. - :param ignore_requires_python: Whether to ignore incompatible - "Requires-Python" values in links. Defaults to False. - """ - if ignore_requires_python is None: - ignore_requires_python = False - - self.allow_yanked = allow_yanked - self.allow_all_prereleases = allow_all_prereleases - self.format_control = format_control - self.prefer_binary = prefer_binary - self.ignore_requires_python = ignore_requires_python diff --git a/venv/lib/python3.11/site-packages/pip/_internal/models/target_python.py b/venv/lib/python3.11/site-packages/pip/_internal/models/target_python.py deleted file mode 100644 index 67ea5da..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/models/target_python.py +++ /dev/null @@ -1,122 +0,0 @@ -import sys -from typing import List, Optional, Set, Tuple - -from pip._vendor.packaging.tags import Tag - -from pip._internal.utils.compatibility_tags import get_supported, version_info_to_nodot -from pip._internal.utils.misc import normalize_version_info - - -class TargetPython: - - """ - Encapsulates the properties of a Python interpreter one is targeting - for a package install, download, etc. - """ - - __slots__ = [ - "_given_py_version_info", - "abis", - "implementation", - "platforms", - "py_version", - "py_version_info", - "_valid_tags", - "_valid_tags_set", - ] - - def __init__( - self, - platforms: Optional[List[str]] = None, - py_version_info: Optional[Tuple[int, ...]] = None, - abis: Optional[List[str]] = None, - implementation: Optional[str] = None, - ) -> None: - """ - :param platforms: A list of strings or None. If None, searches for - packages that are supported by the current system. Otherwise, will - find packages that can be built on the platforms passed in. These - packages will only be downloaded for distribution: they will - not be built locally. - :param py_version_info: An optional tuple of ints representing the - Python version information to use (e.g. `sys.version_info[:3]`). - This can have length 1, 2, or 3 when provided. - :param abis: A list of strings or None. This is passed to - compatibility_tags.py's get_supported() function as is. - :param implementation: A string or None. This is passed to - compatibility_tags.py's get_supported() function as is. - """ - # Store the given py_version_info for when we call get_supported(). - self._given_py_version_info = py_version_info - - if py_version_info is None: - py_version_info = sys.version_info[:3] - else: - py_version_info = normalize_version_info(py_version_info) - - py_version = ".".join(map(str, py_version_info[:2])) - - self.abis = abis - self.implementation = implementation - self.platforms = platforms - self.py_version = py_version - self.py_version_info = py_version_info - - # This is used to cache the return value of get_(un)sorted_tags. - self._valid_tags: Optional[List[Tag]] = None - self._valid_tags_set: Optional[Set[Tag]] = None - - def format_given(self) -> str: - """ - Format the given, non-None attributes for display. - """ - display_version = None - if self._given_py_version_info is not None: - display_version = ".".join( - str(part) for part in self._given_py_version_info - ) - - key_values = [ - ("platforms", self.platforms), - ("version_info", display_version), - ("abis", self.abis), - ("implementation", self.implementation), - ] - return " ".join( - f"{key}={value!r}" for key, value in key_values if value is not None - ) - - def get_sorted_tags(self) -> List[Tag]: - """ - Return the supported PEP 425 tags to check wheel candidates against. - - The tags are returned in order of preference (most preferred first). - """ - if self._valid_tags is None: - # Pass versions=None if no py_version_info was given since - # versions=None uses special default logic. - py_version_info = self._given_py_version_info - if py_version_info is None: - version = None - else: - version = version_info_to_nodot(py_version_info) - - tags = get_supported( - version=version, - platforms=self.platforms, - abis=self.abis, - impl=self.implementation, - ) - self._valid_tags = tags - - return self._valid_tags - - def get_unsorted_tags(self) -> Set[Tag]: - """Exactly the same as get_sorted_tags, but returns a set. - - This is important for performance. - """ - if self._valid_tags_set is None: - self._valid_tags_set = set(self.get_sorted_tags()) - - return self._valid_tags_set diff --git a/venv/lib/python3.11/site-packages/pip/_internal/models/wheel.py b/venv/lib/python3.11/site-packages/pip/_internal/models/wheel.py deleted file mode 100644 index a5dc12b..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/models/wheel.py +++ /dev/null @@ -1,92 +0,0 @@ -"""Represents a wheel file and provides access to the various parts of the -name that have meaning. -""" -import re -from typing import Dict, Iterable, List - -from pip._vendor.packaging.tags import Tag - -from pip._internal.exceptions import InvalidWheelFilename - - -class Wheel: - """A wheel file""" - - wheel_file_re = re.compile( - r"""^(?P(?P[^\s-]+?)-(?P[^\s-]*?)) - ((-(?P\d[^-]*?))?-(?P[^\s-]+?)-(?P[^\s-]+?)-(?P[^\s-]+?) - \.whl|\.dist-info)$""", - re.VERBOSE, - ) - - def __init__(self, filename: str) -> None: - """ - :raises InvalidWheelFilename: when the filename is invalid for a wheel - """ - wheel_info = self.wheel_file_re.match(filename) - if not wheel_info: - raise InvalidWheelFilename(f"{filename} is not a valid wheel filename.") - self.filename = filename - self.name = wheel_info.group("name").replace("_", "-") - # we'll assume "_" means "-" due to wheel naming scheme - # (https://github.com/pypa/pip/issues/1150) - self.version = wheel_info.group("ver").replace("_", "-") - self.build_tag = wheel_info.group("build") - self.pyversions = wheel_info.group("pyver").split(".") - self.abis = wheel_info.group("abi").split(".") - self.plats = wheel_info.group("plat").split(".") - - # All the tag combinations from this file - self.file_tags = { - Tag(x, y, z) for x in self.pyversions for y in self.abis for z in self.plats - } - - def get_formatted_file_tags(self) -> List[str]: - """Return the wheel's tags as a sorted list of strings.""" - return sorted(str(tag) for tag in self.file_tags) - - def support_index_min(self, tags: List[Tag]) -> int: - """Return the lowest index that one of the wheel's file_tag combinations - achieves in the given list of supported tags. - - For example, if there are 8 supported tags and one of the file tags - is first in the list, then return 0. - - :param tags: the PEP 425 tags to check the wheel against, in order - with most preferred first. - - :raises ValueError: If none of the wheel's file tags match one of - the supported tags. - """ - try: - return next(i for i, t in enumerate(tags) if t in self.file_tags) - except StopIteration: - raise ValueError() - - def find_most_preferred_tag( - self, tags: List[Tag], tag_to_priority: Dict[Tag, int] - ) -> int: - """Return the priority of the most preferred tag that one of the wheel's file - tag combinations achieves in the given list of supported tags using the given - tag_to_priority mapping, where lower priorities are more-preferred. - - This is used in place of support_index_min in some cases in order to avoid - an expensive linear scan of a large list of tags. - - :param tags: the PEP 425 tags to check the wheel against. - :param tag_to_priority: a mapping from tag to priority of that tag, where - lower is more preferred. - - :raises ValueError: If none of the wheel's file tags match one of - the supported tags. - """ - return min( - tag_to_priority[tag] for tag in self.file_tags if tag in tag_to_priority - ) - - def supported(self, tags: Iterable[Tag]) -> bool: - """Return whether the wheel is compatible with one of the given tags. - - :param tags: the PEP 425 tags to check the wheel against. - """ - return not self.file_tags.isdisjoint(tags) diff --git a/venv/lib/python3.11/site-packages/pip/_internal/network/__init__.py b/venv/lib/python3.11/site-packages/pip/_internal/network/__init__.py deleted file mode 100644 index b51bde9..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/network/__init__.py +++ /dev/null @@ -1,2 +0,0 @@ -"""Contains purely network-related utilities. -""" diff --git a/venv/lib/python3.11/site-packages/pip/_internal/network/__pycache__/__init__.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/network/__pycache__/__init__.cpython-311.pyc deleted file mode 100644 index 524264a..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/network/__pycache__/__init__.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/network/__pycache__/auth.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/network/__pycache__/auth.cpython-311.pyc deleted file mode 100644 index c5064c7..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/network/__pycache__/auth.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/network/__pycache__/cache.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/network/__pycache__/cache.cpython-311.pyc deleted file mode 100644 index 8465ca8..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/network/__pycache__/cache.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/network/__pycache__/download.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/network/__pycache__/download.cpython-311.pyc deleted file mode 100644 index c607a0d..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/network/__pycache__/download.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/network/__pycache__/lazy_wheel.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/network/__pycache__/lazy_wheel.cpython-311.pyc deleted file mode 100644 index 3a22de8..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/network/__pycache__/lazy_wheel.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/network/__pycache__/session.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/network/__pycache__/session.cpython-311.pyc deleted file mode 100644 index d899547..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/network/__pycache__/session.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/network/__pycache__/utils.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/network/__pycache__/utils.cpython-311.pyc deleted file mode 100644 index efd9075..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/network/__pycache__/utils.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/network/__pycache__/xmlrpc.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/network/__pycache__/xmlrpc.cpython-311.pyc deleted file mode 100644 index 39df188..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/network/__pycache__/xmlrpc.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/network/auth.py b/venv/lib/python3.11/site-packages/pip/_internal/network/auth.py deleted file mode 100644 index 94a82fa..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/network/auth.py +++ /dev/null @@ -1,561 +0,0 @@ -"""Network Authentication Helpers - -Contains interface (MultiDomainBasicAuth) and associated glue code for -providing credentials in the context of network requests. -""" -import logging -import os -import shutil -import subprocess -import sysconfig -import typing -import urllib.parse -from abc import ABC, abstractmethod -from functools import lru_cache -from os.path import commonprefix -from pathlib import Path -from typing import Any, Dict, List, NamedTuple, Optional, Tuple - -from pip._vendor.requests.auth import AuthBase, HTTPBasicAuth -from pip._vendor.requests.models import Request, Response -from pip._vendor.requests.utils import get_netrc_auth - -from pip._internal.utils.logging import getLogger -from pip._internal.utils.misc import ( - ask, - ask_input, - ask_password, - remove_auth_from_url, - split_auth_netloc_from_url, -) -from pip._internal.vcs.versioncontrol import AuthInfo - -logger = getLogger(__name__) - -KEYRING_DISABLED = False - - -class Credentials(NamedTuple): - url: str - username: str - password: str - - -class KeyRingBaseProvider(ABC): - """Keyring base provider interface""" - - has_keyring: bool - - @abstractmethod - def get_auth_info(self, url: str, username: Optional[str]) -> Optional[AuthInfo]: - ... - - @abstractmethod - def save_auth_info(self, url: str, username: str, password: str) -> None: - ... - - -class KeyRingNullProvider(KeyRingBaseProvider): - """Keyring null provider""" - - has_keyring = False - - def get_auth_info(self, url: str, username: Optional[str]) -> Optional[AuthInfo]: - return None - - def save_auth_info(self, url: str, username: str, password: str) -> None: - return None - - -class KeyRingPythonProvider(KeyRingBaseProvider): - """Keyring interface which uses locally imported `keyring`""" - - has_keyring = True - - def __init__(self) -> None: - import keyring - - self.keyring = keyring - - def get_auth_info(self, url: str, username: Optional[str]) -> Optional[AuthInfo]: - # Support keyring's get_credential interface which supports getting - # credentials without a username. This is only available for - # keyring>=15.2.0. - if hasattr(self.keyring, "get_credential"): - logger.debug("Getting credentials from keyring for %s", url) - cred = self.keyring.get_credential(url, username) - if cred is not None: - return cred.username, cred.password - return None - - if username is not None: - logger.debug("Getting password from keyring for %s", url) - password = self.keyring.get_password(url, username) - if password: - return username, password - return None - - def save_auth_info(self, url: str, username: str, password: str) -> None: - self.keyring.set_password(url, username, password) - - -class KeyRingCliProvider(KeyRingBaseProvider): - """Provider which uses `keyring` cli - - Instead of calling the keyring package installed alongside pip - we call keyring on the command line which will enable pip to - use which ever installation of keyring is available first in - PATH. - """ - - has_keyring = True - - def __init__(self, cmd: str) -> None: - self.keyring = cmd - - def get_auth_info(self, url: str, username: Optional[str]) -> Optional[AuthInfo]: - # This is the default implementation of keyring.get_credential - # https://github.com/jaraco/keyring/blob/97689324abcf01bd1793d49063e7ca01e03d7d07/keyring/backend.py#L134-L139 - if username is not None: - password = self._get_password(url, username) - if password is not None: - return username, password - return None - - def save_auth_info(self, url: str, username: str, password: str) -> None: - return self._set_password(url, username, password) - - def _get_password(self, service_name: str, username: str) -> Optional[str]: - """Mirror the implementation of keyring.get_password using cli""" - if self.keyring is None: - return None - - cmd = [self.keyring, "get", service_name, username] - env = os.environ.copy() - env["PYTHONIOENCODING"] = "utf-8" - res = subprocess.run( - cmd, - stdin=subprocess.DEVNULL, - stdout=subprocess.PIPE, - env=env, - ) - if res.returncode: - return None - return res.stdout.decode("utf-8").strip(os.linesep) - - def _set_password(self, service_name: str, username: str, password: str) -> None: - """Mirror the implementation of keyring.set_password using cli""" - if self.keyring is None: - return None - env = os.environ.copy() - env["PYTHONIOENCODING"] = "utf-8" - subprocess.run( - [self.keyring, "set", service_name, username], - input=f"{password}{os.linesep}".encode("utf-8"), - env=env, - check=True, - ) - return None - - -@lru_cache(maxsize=None) -def get_keyring_provider(provider: str) -> KeyRingBaseProvider: - logger.verbose("Keyring provider requested: %s", provider) - - # keyring has previously failed and been disabled - if KEYRING_DISABLED: - provider = "disabled" - if provider in ["import", "auto"]: - try: - impl = KeyRingPythonProvider() - logger.verbose("Keyring provider set: import") - return impl - except ImportError: - pass - except Exception as exc: - # In the event of an unexpected exception - # we should warn the user - msg = "Installed copy of keyring fails with exception %s" - if provider == "auto": - msg = msg + ", trying to find a keyring executable as a fallback" - logger.warning(msg, exc, exc_info=logger.isEnabledFor(logging.DEBUG)) - if provider in ["subprocess", "auto"]: - cli = shutil.which("keyring") - if cli and cli.startswith(sysconfig.get_path("scripts")): - # all code within this function is stolen from shutil.which implementation - @typing.no_type_check - def PATH_as_shutil_which_determines_it() -> str: - path = os.environ.get("PATH", None) - if path is None: - try: - path = os.confstr("CS_PATH") - except (AttributeError, ValueError): - # os.confstr() or CS_PATH is not available - path = os.defpath - # bpo-35755: Don't use os.defpath if the PATH environment variable is - # set to an empty string - - return path - - scripts = Path(sysconfig.get_path("scripts")) - - paths = [] - for path in PATH_as_shutil_which_determines_it().split(os.pathsep): - p = Path(path) - try: - if not p.samefile(scripts): - paths.append(path) - except FileNotFoundError: - pass - - path = os.pathsep.join(paths) - - cli = shutil.which("keyring", path=path) - - if cli: - logger.verbose("Keyring provider set: subprocess with executable %s", cli) - return KeyRingCliProvider(cli) - - logger.verbose("Keyring provider set: disabled") - return KeyRingNullProvider() - - -class MultiDomainBasicAuth(AuthBase): - def __init__( - self, - prompting: bool = True, - index_urls: Optional[List[str]] = None, - keyring_provider: str = "auto", - ) -> None: - self.prompting = prompting - self.index_urls = index_urls - self.keyring_provider = keyring_provider # type: ignore[assignment] - self.passwords: Dict[str, AuthInfo] = {} - # When the user is prompted to enter credentials and keyring is - # available, we will offer to save them. If the user accepts, - # this value is set to the credentials they entered. After the - # request authenticates, the caller should call - # ``save_credentials`` to save these. - self._credentials_to_save: Optional[Credentials] = None - - @property - def keyring_provider(self) -> KeyRingBaseProvider: - return get_keyring_provider(self._keyring_provider) - - @keyring_provider.setter - def keyring_provider(self, provider: str) -> None: - # The free function get_keyring_provider has been decorated with - # functools.cache. If an exception occurs in get_keyring_auth that - # cache will be cleared and keyring disabled, take that into account - # if you want to remove this indirection. - self._keyring_provider = provider - - @property - def use_keyring(self) -> bool: - # We won't use keyring when --no-input is passed unless - # a specific provider is requested because it might require - # user interaction - return self.prompting or self._keyring_provider not in ["auto", "disabled"] - - def _get_keyring_auth( - self, - url: Optional[str], - username: Optional[str], - ) -> Optional[AuthInfo]: - """Return the tuple auth for a given url from keyring.""" - # Do nothing if no url was provided - if not url: - return None - - try: - return self.keyring_provider.get_auth_info(url, username) - except Exception as exc: - logger.warning( - "Keyring is skipped due to an exception: %s", - str(exc), - ) - global KEYRING_DISABLED - KEYRING_DISABLED = True - get_keyring_provider.cache_clear() - return None - - def _get_index_url(self, url: str) -> Optional[str]: - """Return the original index URL matching the requested URL. - - Cached or dynamically generated credentials may work against - the original index URL rather than just the netloc. - - The provided url should have had its username and password - removed already. If the original index url had credentials then - they will be included in the return value. - - Returns None if no matching index was found, or if --no-index - was specified by the user. - """ - if not url or not self.index_urls: - return None - - url = remove_auth_from_url(url).rstrip("/") + "/" - parsed_url = urllib.parse.urlsplit(url) - - candidates = [] - - for index in self.index_urls: - index = index.rstrip("/") + "/" - parsed_index = urllib.parse.urlsplit(remove_auth_from_url(index)) - if parsed_url == parsed_index: - return index - - if parsed_url.netloc != parsed_index.netloc: - continue - - candidate = urllib.parse.urlsplit(index) - candidates.append(candidate) - - if not candidates: - return None - - candidates.sort( - reverse=True, - key=lambda candidate: commonprefix( - [ - parsed_url.path, - candidate.path, - ] - ).rfind("/"), - ) - - return urllib.parse.urlunsplit(candidates[0]) - - def _get_new_credentials( - self, - original_url: str, - *, - allow_netrc: bool = True, - allow_keyring: bool = False, - ) -> AuthInfo: - """Find and return credentials for the specified URL.""" - # Split the credentials and netloc from the url. - url, netloc, url_user_password = split_auth_netloc_from_url( - original_url, - ) - - # Start with the credentials embedded in the url - username, password = url_user_password - if username is not None and password is not None: - logger.debug("Found credentials in url for %s", netloc) - return url_user_password - - # Find a matching index url for this request - index_url = self._get_index_url(url) - if index_url: - # Split the credentials from the url. - index_info = split_auth_netloc_from_url(index_url) - if index_info: - index_url, _, index_url_user_password = index_info - logger.debug("Found index url %s", index_url) - - # If an index URL was found, try its embedded credentials - if index_url and index_url_user_password[0] is not None: - username, password = index_url_user_password - if username is not None and password is not None: - logger.debug("Found credentials in index url for %s", netloc) - return index_url_user_password - - # Get creds from netrc if we still don't have them - if allow_netrc: - netrc_auth = get_netrc_auth(original_url) - if netrc_auth: - logger.debug("Found credentials in netrc for %s", netloc) - return netrc_auth - - # If we don't have a password and keyring is available, use it. - if allow_keyring: - # The index url is more specific than the netloc, so try it first - # fmt: off - kr_auth = ( - self._get_keyring_auth(index_url, username) or - self._get_keyring_auth(netloc, username) - ) - # fmt: on - if kr_auth: - logger.debug("Found credentials in keyring for %s", netloc) - return kr_auth - - return username, password - - def _get_url_and_credentials( - self, original_url: str - ) -> Tuple[str, Optional[str], Optional[str]]: - """Return the credentials to use for the provided URL. - - If allowed, netrc and keyring may be used to obtain the - correct credentials. - - Returns (url_without_credentials, username, password). Note - that even if the original URL contains credentials, this - function may return a different username and password. - """ - url, netloc, _ = split_auth_netloc_from_url(original_url) - - # Try to get credentials from original url - username, password = self._get_new_credentials(original_url) - - # If credentials not found, use any stored credentials for this netloc. - # Do this if either the username or the password is missing. - # This accounts for the situation in which the user has specified - # the username in the index url, but the password comes from keyring. - if (username is None or password is None) and netloc in self.passwords: - un, pw = self.passwords[netloc] - # It is possible that the cached credentials are for a different username, - # in which case the cache should be ignored. - if username is None or username == un: - username, password = un, pw - - if username is not None or password is not None: - # Convert the username and password if they're None, so that - # this netloc will show up as "cached" in the conditional above. - # Further, HTTPBasicAuth doesn't accept None, so it makes sense to - # cache the value that is going to be used. - username = username or "" - password = password or "" - - # Store any acquired credentials. - self.passwords[netloc] = (username, password) - - assert ( - # Credentials were found - (username is not None and password is not None) - # Credentials were not found - or (username is None and password is None) - ), f"Could not load credentials from url: {original_url}" - - return url, username, password - - def __call__(self, req: Request) -> Request: - # Get credentials for this request - url, username, password = self._get_url_and_credentials(req.url) - - # Set the url of the request to the url without any credentials - req.url = url - - if username is not None and password is not None: - # Send the basic auth with this request - req = HTTPBasicAuth(username, password)(req) - - # Attach a hook to handle 401 responses - req.register_hook("response", self.handle_401) - - return req - - # Factored out to allow for easy patching in tests - def _prompt_for_password( - self, netloc: str - ) -> Tuple[Optional[str], Optional[str], bool]: - username = ask_input(f"User for {netloc}: ") if self.prompting else None - if not username: - return None, None, False - if self.use_keyring: - auth = self._get_keyring_auth(netloc, username) - if auth and auth[0] is not None and auth[1] is not None: - return auth[0], auth[1], False - password = ask_password("Password: ") - return username, password, True - - # Factored out to allow for easy patching in tests - def _should_save_password_to_keyring(self) -> bool: - if ( - not self.prompting - or not self.use_keyring - or not self.keyring_provider.has_keyring - ): - return False - return ask("Save credentials to keyring [y/N]: ", ["y", "n"]) == "y" - - def handle_401(self, resp: Response, **kwargs: Any) -> Response: - # We only care about 401 responses, anything else we want to just - # pass through the actual response - if resp.status_code != 401: - return resp - - username, password = None, None - - # Query the keyring for credentials: - if self.use_keyring: - username, password = self._get_new_credentials( - resp.url, - allow_netrc=False, - allow_keyring=True, - ) - - # We are not able to prompt the user so simply return the response - if not self.prompting and not username and not password: - return resp - - parsed = urllib.parse.urlparse(resp.url) - - # Prompt the user for a new username and password - save = False - if not username and not password: - username, password, save = self._prompt_for_password(parsed.netloc) - - # Store the new username and password to use for future requests - self._credentials_to_save = None - if username is not None and password is not None: - self.passwords[parsed.netloc] = (username, password) - - # Prompt to save the password to keyring - if save and self._should_save_password_to_keyring(): - self._credentials_to_save = Credentials( - url=parsed.netloc, - username=username, - password=password, - ) - - # Consume content and release the original connection to allow our new - # request to reuse the same one. - # The result of the assignment isn't used, it's just needed to consume - # the content. - _ = resp.content - resp.raw.release_conn() - - # Add our new username and password to the request - req = HTTPBasicAuth(username or "", password or "")(resp.request) - req.register_hook("response", self.warn_on_401) - - # On successful request, save the credentials that were used to - # keyring. (Note that if the user responded "no" above, this member - # is not set and nothing will be saved.) - if self._credentials_to_save: - req.register_hook("response", self.save_credentials) - - # Send our new request - new_resp = resp.connection.send(req, **kwargs) - new_resp.history.append(resp) - - return new_resp - - def warn_on_401(self, resp: Response, **kwargs: Any) -> None: - """Response callback to warn about incorrect credentials.""" - if resp.status_code == 401: - logger.warning( - "401 Error, Credentials not correct for %s", - resp.request.url, - ) - - def save_credentials(self, resp: Response, **kwargs: Any) -> None: - """Response callback to save credentials on success.""" - assert ( - self.keyring_provider.has_keyring - ), "should never reach here without keyring" - - creds = self._credentials_to_save - self._credentials_to_save = None - if creds and resp.status_code < 400: - try: - logger.info("Saving credentials to keyring") - self.keyring_provider.save_auth_info( - creds.url, creds.username, creds.password - ) - except Exception: - logger.exception("Failed to save credentials") diff --git a/venv/lib/python3.11/site-packages/pip/_internal/network/cache.py b/venv/lib/python3.11/site-packages/pip/_internal/network/cache.py deleted file mode 100644 index 4d0fb54..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/network/cache.py +++ /dev/null @@ -1,106 +0,0 @@ -"""HTTP cache implementation. -""" - -import os -from contextlib import contextmanager -from datetime import datetime -from typing import BinaryIO, Generator, Optional, Union - -from pip._vendor.cachecontrol.cache import SeparateBodyBaseCache -from pip._vendor.cachecontrol.caches import SeparateBodyFileCache -from pip._vendor.requests.models import Response - -from pip._internal.utils.filesystem import adjacent_tmp_file, replace -from pip._internal.utils.misc import ensure_dir - - -def is_from_cache(response: Response) -> bool: - return getattr(response, "from_cache", False) - - -@contextmanager -def suppressed_cache_errors() -> Generator[None, None, None]: - """If we can't access the cache then we can just skip caching and process - requests as if caching wasn't enabled. - """ - try: - yield - except OSError: - pass - - -class SafeFileCache(SeparateBodyBaseCache): - """ - A file based cache which is safe to use even when the target directory may - not be accessible or writable. - - There is a race condition when two processes try to write and/or read the - same entry at the same time, since each entry consists of two separate - files (https://github.com/psf/cachecontrol/issues/324). We therefore have - additional logic that makes sure that both files to be present before - returning an entry; this fixes the read side of the race condition. - - For the write side, we assume that the server will only ever return the - same data for the same URL, which ought to be the case for files pip is - downloading. PyPI does not have a mechanism to swap out a wheel for - another wheel, for example. If this assumption is not true, the - CacheControl issue will need to be fixed. - """ - - def __init__(self, directory: str) -> None: - assert directory is not None, "Cache directory must not be None." - super().__init__() - self.directory = directory - - def _get_cache_path(self, name: str) -> str: - # From cachecontrol.caches.file_cache.FileCache._fn, brought into our - # class for backwards-compatibility and to avoid using a non-public - # method. - hashed = SeparateBodyFileCache.encode(name) - parts = list(hashed[:5]) + [hashed] - return os.path.join(self.directory, *parts) - - def get(self, key: str) -> Optional[bytes]: - # The cache entry is only valid if both metadata and body exist. - metadata_path = self._get_cache_path(key) - body_path = metadata_path + ".body" - if not (os.path.exists(metadata_path) and os.path.exists(body_path)): - return None - with suppressed_cache_errors(): - with open(metadata_path, "rb") as f: - return f.read() - - def _write(self, path: str, data: bytes) -> None: - with suppressed_cache_errors(): - ensure_dir(os.path.dirname(path)) - - with adjacent_tmp_file(path) as f: - f.write(data) - - replace(f.name, path) - - def set( - self, key: str, value: bytes, expires: Union[int, datetime, None] = None - ) -> None: - path = self._get_cache_path(key) - self._write(path, value) - - def delete(self, key: str) -> None: - path = self._get_cache_path(key) - with suppressed_cache_errors(): - os.remove(path) - with suppressed_cache_errors(): - os.remove(path + ".body") - - def get_body(self, key: str) -> Optional[BinaryIO]: - # The cache entry is only valid if both metadata and body exist. - metadata_path = self._get_cache_path(key) - body_path = metadata_path + ".body" - if not (os.path.exists(metadata_path) and os.path.exists(body_path)): - return None - with suppressed_cache_errors(): - return open(body_path, "rb") - - def set_body(self, key: str, body: bytes) -> None: - path = self._get_cache_path(key) + ".body" - self._write(path, body) diff --git a/venv/lib/python3.11/site-packages/pip/_internal/network/download.py b/venv/lib/python3.11/site-packages/pip/_internal/network/download.py deleted file mode 100644 index 79b82a5..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/network/download.py +++ /dev/null @@ -1,186 +0,0 @@ -"""Download files with progress indicators. -""" -import email.message -import logging -import mimetypes -import os -from typing import Iterable, Optional, Tuple - -from pip._vendor.requests.models import CONTENT_CHUNK_SIZE, Response - -from pip._internal.cli.progress_bars import get_download_progress_renderer -from pip._internal.exceptions import NetworkConnectionError -from pip._internal.models.index import PyPI -from pip._internal.models.link import Link -from pip._internal.network.cache import is_from_cache -from pip._internal.network.session import PipSession -from pip._internal.network.utils import HEADERS, raise_for_status, response_chunks -from pip._internal.utils.misc import format_size, redact_auth_from_url, splitext - -logger = logging.getLogger(__name__) - - -def _get_http_response_size(resp: Response) -> Optional[int]: - try: - return int(resp.headers["content-length"]) - except (ValueError, KeyError, TypeError): - return None - - -def _prepare_download( - resp: Response, - link: Link, - progress_bar: str, -) -> Iterable[bytes]: - total_length = _get_http_response_size(resp) - - if link.netloc == PyPI.file_storage_domain: - url = link.show_url - else: - url = link.url_without_fragment - - logged_url = redact_auth_from_url(url) - - if total_length: - logged_url = "{} ({})".format(logged_url, format_size(total_length)) - - if is_from_cache(resp): - logger.info("Using cached %s", logged_url) - else: - logger.info("Downloading %s", logged_url) - - if logger.getEffectiveLevel() > logging.INFO: - show_progress = False - elif is_from_cache(resp): - show_progress = False - elif not total_length: - show_progress = True - elif total_length > (40 * 1000): - show_progress = True - else: - show_progress = False - - chunks = response_chunks(resp, CONTENT_CHUNK_SIZE) - - if not show_progress: - return chunks - - renderer = get_download_progress_renderer(bar_type=progress_bar, size=total_length) - return renderer(chunks) - - -def sanitize_content_filename(filename: str) -> str: - """ - Sanitize the "filename" value from a Content-Disposition header. - """ - return os.path.basename(filename) - - -def parse_content_disposition(content_disposition: str, default_filename: str) -> str: - """ - Parse the "filename" value from a Content-Disposition header, and - return the default filename if the result is empty. - """ - m = email.message.Message() - m["content-type"] = content_disposition - filename = m.get_param("filename") - if filename: - # We need to sanitize the filename to prevent directory traversal - # in case the filename contains ".." path parts. - filename = sanitize_content_filename(str(filename)) - return filename or default_filename - - -def _get_http_response_filename(resp: Response, link: Link) -> str: - """Get an ideal filename from the given HTTP response, falling back to - the link filename if not provided. - """ - filename = link.filename # fallback - # Have a look at the Content-Disposition header for a better guess - content_disposition = resp.headers.get("content-disposition") - if content_disposition: - filename = parse_content_disposition(content_disposition, filename) - ext: Optional[str] = splitext(filename)[1] - if not ext: - ext = mimetypes.guess_extension(resp.headers.get("content-type", "")) - if ext: - filename += ext - if not ext and link.url != resp.url: - ext = os.path.splitext(resp.url)[1] - if ext: - filename += ext - return filename - - -def _http_get_download(session: PipSession, link: Link) -> Response: - target_url = link.url.split("#", 1)[0] - resp = session.get(target_url, headers=HEADERS, stream=True) - raise_for_status(resp) - return resp - - -class Downloader: - def __init__( - self, - session: PipSession, - progress_bar: str, - ) -> None: - self._session = session - self._progress_bar = progress_bar - - def __call__(self, link: Link, location: str) -> Tuple[str, str]: - """Download the file given by link into location.""" - try: - resp = _http_get_download(self._session, link) - except NetworkConnectionError as e: - assert e.response is not None - logger.critical( - "HTTP error %s while getting %s", e.response.status_code, link - ) - raise - - filename = _get_http_response_filename(resp, link) - filepath = os.path.join(location, filename) - - chunks = _prepare_download(resp, link, self._progress_bar) - with open(filepath, "wb") as content_file: - for chunk in chunks: - content_file.write(chunk) - content_type = resp.headers.get("Content-Type", "") - return filepath, content_type - - -class BatchDownloader: - def __init__( - self, - session: PipSession, - progress_bar: str, - ) -> None: - self._session = session - self._progress_bar = progress_bar - - def __call__( - self, links: Iterable[Link], location: str - ) -> Iterable[Tuple[Link, Tuple[str, str]]]: - """Download the files given by links into location.""" - for link in links: - try: - resp = _http_get_download(self._session, link) - except NetworkConnectionError as e: - assert e.response is not None - logger.critical( - "HTTP error %s while getting %s", - e.response.status_code, - link, - ) - raise - - filename = _get_http_response_filename(resp, link) - filepath = os.path.join(location, filename) - - chunks = _prepare_download(resp, link, self._progress_bar) - with open(filepath, "wb") as content_file: - for chunk in chunks: - content_file.write(chunk) - content_type = resp.headers.get("Content-Type", "") - yield link, (filepath, content_type) diff --git a/venv/lib/python3.11/site-packages/pip/_internal/network/lazy_wheel.py b/venv/lib/python3.11/site-packages/pip/_internal/network/lazy_wheel.py deleted file mode 100644 index 82ec50d..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/network/lazy_wheel.py +++ /dev/null @@ -1,210 +0,0 @@ -"""Lazy ZIP over HTTP""" - -__all__ = ["HTTPRangeRequestUnsupported", "dist_from_wheel_url"] - -from bisect import bisect_left, bisect_right -from contextlib import contextmanager -from tempfile import NamedTemporaryFile -from typing import Any, Dict, Generator, List, Optional, Tuple -from zipfile import BadZipFile, ZipFile - -from pip._vendor.packaging.utils import canonicalize_name -from pip._vendor.requests.models import CONTENT_CHUNK_SIZE, Response - -from pip._internal.metadata import BaseDistribution, MemoryWheel, get_wheel_distribution -from pip._internal.network.session import PipSession -from pip._internal.network.utils import HEADERS, raise_for_status, response_chunks - - -class HTTPRangeRequestUnsupported(Exception): - pass - - -def dist_from_wheel_url(name: str, url: str, session: PipSession) -> BaseDistribution: - """Return a distribution object from the given wheel URL. - - This uses HTTP range requests to only fetch the portion of the wheel - containing metadata, just enough for the object to be constructed. - If such requests are not supported, HTTPRangeRequestUnsupported - is raised. - """ - with LazyZipOverHTTP(url, session) as zf: - # For read-only ZIP files, ZipFile only needs methods read, - # seek, seekable and tell, not the whole IO protocol. - wheel = MemoryWheel(zf.name, zf) # type: ignore - # After context manager exit, wheel.name - # is an invalid file by intention. - return get_wheel_distribution(wheel, canonicalize_name(name)) - - -class LazyZipOverHTTP: - """File-like object mapped to a ZIP file over HTTP. - - This uses HTTP range requests to lazily fetch the file's content, - which is supposed to be fed to ZipFile. If such requests are not - supported by the server, raise HTTPRangeRequestUnsupported - during initialization. - """ - - def __init__( - self, url: str, session: PipSession, chunk_size: int = CONTENT_CHUNK_SIZE - ) -> None: - head = session.head(url, headers=HEADERS) - raise_for_status(head) - assert head.status_code == 200 - self._session, self._url, self._chunk_size = session, url, chunk_size - self._length = int(head.headers["Content-Length"]) - self._file = NamedTemporaryFile() - self.truncate(self._length) - self._left: List[int] = [] - self._right: List[int] = [] - if "bytes" not in head.headers.get("Accept-Ranges", "none"): - raise HTTPRangeRequestUnsupported("range request is not supported") - self._check_zip() - - @property - def mode(self) -> str: - """Opening mode, which is always rb.""" - return "rb" - - @property - def name(self) -> str: - """Path to the underlying file.""" - return self._file.name - - def seekable(self) -> bool: - """Return whether random access is supported, which is True.""" - return True - - def close(self) -> None: - """Close the file.""" - self._file.close() - - @property - def closed(self) -> bool: - """Whether the file is closed.""" - return self._file.closed - - def read(self, size: int = -1) -> bytes: - """Read up to size bytes from the object and return them. - - As a convenience, if size is unspecified or -1, - all bytes until EOF are returned. Fewer than - size bytes may be returned if EOF is reached. - """ - download_size = max(size, self._chunk_size) - start, length = self.tell(), self._length - stop = length if size < 0 else min(start + download_size, length) - start = max(0, stop - download_size) - self._download(start, stop - 1) - return self._file.read(size) - - def readable(self) -> bool: - """Return whether the file is readable, which is True.""" - return True - - def seek(self, offset: int, whence: int = 0) -> int: - """Change stream position and return the new absolute position. - - Seek to offset relative position indicated by whence: - * 0: Start of stream (the default). pos should be >= 0; - * 1: Current position - pos may be negative; - * 2: End of stream - pos usually negative. - """ - return self._file.seek(offset, whence) - - def tell(self) -> int: - """Return the current position.""" - return self._file.tell() - - def truncate(self, size: Optional[int] = None) -> int: - """Resize the stream to the given size in bytes. - - If size is unspecified resize to the current position. - The current stream position isn't changed. - - Return the new file size. - """ - return self._file.truncate(size) - - def writable(self) -> bool: - """Return False.""" - return False - - def __enter__(self) -> "LazyZipOverHTTP": - self._file.__enter__() - return self - - def __exit__(self, *exc: Any) -> None: - self._file.__exit__(*exc) - - @contextmanager - def _stay(self) -> Generator[None, None, None]: - """Return a context manager keeping the position. - - At the end of the block, seek back to original position. - """ - pos = self.tell() - try: - yield - finally: - self.seek(pos) - - def _check_zip(self) -> None: - """Check and download until the file is a valid ZIP.""" - end = self._length - 1 - for start in reversed(range(0, end, self._chunk_size)): - self._download(start, end) - with self._stay(): - try: - # For read-only ZIP files, ZipFile only needs - # methods read, seek, seekable and tell. - ZipFile(self) # type: ignore - except BadZipFile: - pass - else: - break - - def _stream_response( - self, start: int, end: int, base_headers: Dict[str, str] = HEADERS - ) -> Response: - """Return HTTP response to a range request from start to end.""" - headers = base_headers.copy() - headers["Range"] = f"bytes={start}-{end}" - # TODO: Get range requests to be correctly cached - headers["Cache-Control"] = "no-cache" - return self._session.get(self._url, headers=headers, stream=True) - - def _merge( - self, start: int, end: int, left: int, right: int - ) -> Generator[Tuple[int, int], None, None]: - """Return a generator of intervals to be fetched. - - Args: - start (int): Start of needed interval - end (int): End of needed interval - left (int): Index of first overlapping downloaded data - right (int): Index after last overlapping downloaded data - """ - lslice, rslice = self._left[left:right], self._right[left:right] - i = start = min([start] + lslice[:1]) - end = max([end] + rslice[-1:]) - for j, k in zip(lslice, rslice): - if j > i: - yield i, j - 1 - i = k + 1 - if i <= end: - yield i, end - self._left[left:right], self._right[left:right] = [start], [end] - - def _download(self, start: int, end: int) -> None: - """Download bytes from start to end inclusively.""" - with self._stay(): - left = bisect_left(self._right, start) - right = bisect_right(self._left, end) - for start, end in self._merge(start, end, left, right): - response = self._stream_response(start, end) - response.raise_for_status() - self.seek(start) - for chunk in response_chunks(response, self._chunk_size): - self._file.write(chunk) diff --git a/venv/lib/python3.11/site-packages/pip/_internal/network/session.py b/venv/lib/python3.11/site-packages/pip/_internal/network/session.py deleted file mode 100644 index 887dc14..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/network/session.py +++ /dev/null @@ -1,519 +0,0 @@ -"""PipSession and supporting code, containing all pip-specific -network request configuration and behavior. -""" - -import email.utils -import io -import ipaddress -import json -import logging -import mimetypes -import os -import platform -import shutil -import subprocess -import sys -import urllib.parse -import warnings -from typing import ( - TYPE_CHECKING, - Any, - Dict, - Generator, - List, - Mapping, - Optional, - Sequence, - Tuple, - Union, -) - -from pip._vendor import requests, urllib3 -from pip._vendor.cachecontrol import CacheControlAdapter as _BaseCacheControlAdapter -from pip._vendor.requests.adapters import DEFAULT_POOLBLOCK, BaseAdapter -from pip._vendor.requests.adapters import HTTPAdapter as _BaseHTTPAdapter -from pip._vendor.requests.models import PreparedRequest, Response -from pip._vendor.requests.structures import CaseInsensitiveDict -from pip._vendor.urllib3.connectionpool import ConnectionPool -from pip._vendor.urllib3.exceptions import InsecureRequestWarning - -from pip import __version__ -from pip._internal.metadata import get_default_environment -from pip._internal.models.link import Link -from pip._internal.network.auth import MultiDomainBasicAuth -from pip._internal.network.cache import SafeFileCache - -# Import ssl from compat so the initial import occurs in only one place. -from pip._internal.utils.compat import has_tls -from pip._internal.utils.glibc import libc_ver -from pip._internal.utils.misc import build_url_from_netloc, parse_netloc -from pip._internal.utils.urls import url_to_path - -if TYPE_CHECKING: - from ssl import SSLContext - - from pip._vendor.urllib3.poolmanager import PoolManager - - -logger = logging.getLogger(__name__) - -SecureOrigin = Tuple[str, str, Optional[Union[int, str]]] - - -# Ignore warning raised when using --trusted-host. -warnings.filterwarnings("ignore", category=InsecureRequestWarning) - - -SECURE_ORIGINS: List[SecureOrigin] = [ - # protocol, hostname, port - # Taken from Chrome's list of secure origins (See: http://bit.ly/1qrySKC) - ("https", "*", "*"), - ("*", "localhost", "*"), - ("*", "127.0.0.0/8", "*"), - ("*", "::1/128", "*"), - ("file", "*", None), - # ssh is always secure. - ("ssh", "*", "*"), -] - - -# These are environment variables present when running under various -# CI systems. For each variable, some CI systems that use the variable -# are indicated. The collection was chosen so that for each of a number -# of popular systems, at least one of the environment variables is used. -# This list is used to provide some indication of and lower bound for -# CI traffic to PyPI. Thus, it is okay if the list is not comprehensive. -# For more background, see: https://github.com/pypa/pip/issues/5499 -CI_ENVIRONMENT_VARIABLES = ( - # Azure Pipelines - "BUILD_BUILDID", - # Jenkins - "BUILD_ID", - # AppVeyor, CircleCI, Codeship, Gitlab CI, Shippable, Travis CI - "CI", - # Explicit environment variable. - "PIP_IS_CI", -) - - -def looks_like_ci() -> bool: - """ - Return whether it looks like pip is running under CI. - """ - # We don't use the method of checking for a tty (e.g. using isatty()) - # because some CI systems mimic a tty (e.g. Travis CI). Thus that - # method doesn't provide definitive information in either direction. - return any(name in os.environ for name in CI_ENVIRONMENT_VARIABLES) - - -def user_agent() -> str: - """ - Return a string representing the user agent. - """ - data: Dict[str, Any] = { - "installer": {"name": "pip", "version": __version__}, - "python": platform.python_version(), - "implementation": { - "name": platform.python_implementation(), - }, - } - - if data["implementation"]["name"] == "CPython": - data["implementation"]["version"] = platform.python_version() - elif data["implementation"]["name"] == "PyPy": - pypy_version_info = sys.pypy_version_info # type: ignore - if pypy_version_info.releaselevel == "final": - pypy_version_info = pypy_version_info[:3] - data["implementation"]["version"] = ".".join( - [str(x) for x in pypy_version_info] - ) - elif data["implementation"]["name"] == "Jython": - # Complete Guess - data["implementation"]["version"] = platform.python_version() - elif data["implementation"]["name"] == "IronPython": - # Complete Guess - data["implementation"]["version"] = platform.python_version() - - if sys.platform.startswith("linux"): - from pip._vendor import distro - - linux_distribution = distro.name(), distro.version(), distro.codename() - distro_infos: Dict[str, Any] = dict( - filter( - lambda x: x[1], - zip(["name", "version", "id"], linux_distribution), - ) - ) - libc = dict( - filter( - lambda x: x[1], - zip(["lib", "version"], libc_ver()), - ) - ) - if libc: - distro_infos["libc"] = libc - if distro_infos: - data["distro"] = distro_infos - - if sys.platform.startswith("darwin") and platform.mac_ver()[0]: - data["distro"] = {"name": "macOS", "version": platform.mac_ver()[0]} - - if platform.system(): - data.setdefault("system", {})["name"] = platform.system() - - if platform.release(): - data.setdefault("system", {})["release"] = platform.release() - - if platform.machine(): - data["cpu"] = platform.machine() - - if has_tls(): - import _ssl as ssl - - data["openssl_version"] = ssl.OPENSSL_VERSION - - setuptools_dist = get_default_environment().get_distribution("setuptools") - if setuptools_dist is not None: - data["setuptools_version"] = str(setuptools_dist.version) - - if shutil.which("rustc") is not None: - # If for any reason `rustc --version` fails, silently ignore it - try: - rustc_output = subprocess.check_output( - ["rustc", "--version"], stderr=subprocess.STDOUT, timeout=0.5 - ) - except Exception: - pass - else: - if rustc_output.startswith(b"rustc "): - # The format of `rustc --version` is: - # `b'rustc 1.52.1 (9bc8c42bb 2021-05-09)\n'` - # We extract just the middle (1.52.1) part - data["rustc_version"] = rustc_output.split(b" ")[1].decode() - - # Use None rather than False so as not to give the impression that - # pip knows it is not being run under CI. Rather, it is a null or - # inconclusive result. Also, we include some value rather than no - # value to make it easier to know that the check has been run. - data["ci"] = True if looks_like_ci() else None - - user_data = os.environ.get("PIP_USER_AGENT_USER_DATA") - if user_data is not None: - data["user_data"] = user_data - - return "{data[installer][name]}/{data[installer][version]} {json}".format( - data=data, - json=json.dumps(data, separators=(",", ":"), sort_keys=True), - ) - - -class LocalFSAdapter(BaseAdapter): - def send( - self, - request: PreparedRequest, - stream: bool = False, - timeout: Optional[Union[float, Tuple[float, float]]] = None, - verify: Union[bool, str] = True, - cert: Optional[Union[str, Tuple[str, str]]] = None, - proxies: Optional[Mapping[str, str]] = None, - ) -> Response: - pathname = url_to_path(request.url) - - resp = Response() - resp.status_code = 200 - resp.url = request.url - - try: - stats = os.stat(pathname) - except OSError as exc: - # format the exception raised as a io.BytesIO object, - # to return a better error message: - resp.status_code = 404 - resp.reason = type(exc).__name__ - resp.raw = io.BytesIO(f"{resp.reason}: {exc}".encode("utf8")) - else: - modified = email.utils.formatdate(stats.st_mtime, usegmt=True) - content_type = mimetypes.guess_type(pathname)[0] or "text/plain" - resp.headers = CaseInsensitiveDict( - { - "Content-Type": content_type, - "Content-Length": stats.st_size, - "Last-Modified": modified, - } - ) - - resp.raw = open(pathname, "rb") - resp.close = resp.raw.close - - return resp - - def close(self) -> None: - pass - - -class _SSLContextAdapterMixin: - """Mixin to add the ``ssl_context`` constructor argument to HTTP adapters. - - The additional argument is forwarded directly to the pool manager. This allows us - to dynamically decide what SSL store to use at runtime, which is used to implement - the optional ``truststore`` backend. - """ - - def __init__( - self, - *, - ssl_context: Optional["SSLContext"] = None, - **kwargs: Any, - ) -> None: - self._ssl_context = ssl_context - super().__init__(**kwargs) - - def init_poolmanager( - self, - connections: int, - maxsize: int, - block: bool = DEFAULT_POOLBLOCK, - **pool_kwargs: Any, - ) -> "PoolManager": - if self._ssl_context is not None: - pool_kwargs.setdefault("ssl_context", self._ssl_context) - return super().init_poolmanager( # type: ignore[misc] - connections=connections, - maxsize=maxsize, - block=block, - **pool_kwargs, - ) - - -class HTTPAdapter(_SSLContextAdapterMixin, _BaseHTTPAdapter): - pass - - -class CacheControlAdapter(_SSLContextAdapterMixin, _BaseCacheControlAdapter): - pass - - -class InsecureHTTPAdapter(HTTPAdapter): - def cert_verify( - self, - conn: ConnectionPool, - url: str, - verify: Union[bool, str], - cert: Optional[Union[str, Tuple[str, str]]], - ) -> None: - super().cert_verify(conn=conn, url=url, verify=False, cert=cert) - - -class InsecureCacheControlAdapter(CacheControlAdapter): - def cert_verify( - self, - conn: ConnectionPool, - url: str, - verify: Union[bool, str], - cert: Optional[Union[str, Tuple[str, str]]], - ) -> None: - super().cert_verify(conn=conn, url=url, verify=False, cert=cert) - - -class PipSession(requests.Session): - timeout: Optional[int] = None - - def __init__( - self, - *args: Any, - retries: int = 0, - cache: Optional[str] = None, - trusted_hosts: Sequence[str] = (), - index_urls: Optional[List[str]] = None, - ssl_context: Optional["SSLContext"] = None, - **kwargs: Any, - ) -> None: - """ - :param trusted_hosts: Domains not to emit warnings for when not using - HTTPS. - """ - super().__init__(*args, **kwargs) - - # Namespace the attribute with "pip_" just in case to prevent - # possible conflicts with the base class. - self.pip_trusted_origins: List[Tuple[str, Optional[int]]] = [] - - # Attach our User Agent to the request - self.headers["User-Agent"] = user_agent() - - # Attach our Authentication handler to the session - self.auth = MultiDomainBasicAuth(index_urls=index_urls) - - # Create our urllib3.Retry instance which will allow us to customize - # how we handle retries. - retries = urllib3.Retry( - # Set the total number of retries that a particular request can - # have. - total=retries, - # A 503 error from PyPI typically means that the Fastly -> Origin - # connection got interrupted in some way. A 503 error in general - # is typically considered a transient error so we'll go ahead and - # retry it. - # A 500 may indicate transient error in Amazon S3 - # A 520 or 527 - may indicate transient error in CloudFlare - status_forcelist=[500, 503, 520, 527], - # Add a small amount of back off between failed requests in - # order to prevent hammering the service. - backoff_factor=0.25, - ) # type: ignore - - # Our Insecure HTTPAdapter disables HTTPS validation. It does not - # support caching so we'll use it for all http:// URLs. - # If caching is disabled, we will also use it for - # https:// hosts that we've marked as ignoring - # TLS errors for (trusted-hosts). - insecure_adapter = InsecureHTTPAdapter(max_retries=retries) - - # We want to _only_ cache responses on securely fetched origins or when - # the host is specified as trusted. We do this because - # we can't validate the response of an insecurely/untrusted fetched - # origin, and we don't want someone to be able to poison the cache and - # require manual eviction from the cache to fix it. - if cache: - secure_adapter = CacheControlAdapter( - cache=SafeFileCache(cache), - max_retries=retries, - ssl_context=ssl_context, - ) - self._trusted_host_adapter = InsecureCacheControlAdapter( - cache=SafeFileCache(cache), - max_retries=retries, - ) - else: - secure_adapter = HTTPAdapter(max_retries=retries, ssl_context=ssl_context) - self._trusted_host_adapter = insecure_adapter - - self.mount("https://", secure_adapter) - self.mount("http://", insecure_adapter) - - # Enable file:// urls - self.mount("file://", LocalFSAdapter()) - - for host in trusted_hosts: - self.add_trusted_host(host, suppress_logging=True) - - def update_index_urls(self, new_index_urls: List[str]) -> None: - """ - :param new_index_urls: New index urls to update the authentication - handler with. - """ - self.auth.index_urls = new_index_urls - - def add_trusted_host( - self, host: str, source: Optional[str] = None, suppress_logging: bool = False - ) -> None: - """ - :param host: It is okay to provide a host that has previously been - added. - :param source: An optional source string, for logging where the host - string came from. - """ - if not suppress_logging: - msg = f"adding trusted host: {host!r}" - if source is not None: - msg += f" (from {source})" - logger.info(msg) - - parsed_host, parsed_port = parse_netloc(host) - if parsed_host is None: - raise ValueError(f"Trusted host URL must include a host part: {host!r}") - if (parsed_host, parsed_port) not in self.pip_trusted_origins: - self.pip_trusted_origins.append((parsed_host, parsed_port)) - - self.mount( - build_url_from_netloc(host, scheme="http") + "/", self._trusted_host_adapter - ) - self.mount(build_url_from_netloc(host) + "/", self._trusted_host_adapter) - if not parsed_port: - self.mount( - build_url_from_netloc(host, scheme="http") + ":", - self._trusted_host_adapter, - ) - # Mount wildcard ports for the same host. - self.mount(build_url_from_netloc(host) + ":", self._trusted_host_adapter) - - def iter_secure_origins(self) -> Generator[SecureOrigin, None, None]: - yield from SECURE_ORIGINS - for host, port in self.pip_trusted_origins: - yield ("*", host, "*" if port is None else port) - - def is_secure_origin(self, location: Link) -> bool: - # Determine if this url used a secure transport mechanism - parsed = urllib.parse.urlparse(str(location)) - origin_protocol, origin_host, origin_port = ( - parsed.scheme, - parsed.hostname, - parsed.port, - ) - - # The protocol to use to see if the protocol matches. - # Don't count the repository type as part of the protocol: in - # cases such as "git+ssh", only use "ssh". (I.e., Only verify against - # the last scheme.) - origin_protocol = origin_protocol.rsplit("+", 1)[-1] - - # Determine if our origin is a secure origin by looking through our - # hardcoded list of secure origins, as well as any additional ones - # configured on this PackageFinder instance. - for secure_origin in self.iter_secure_origins(): - secure_protocol, secure_host, secure_port = secure_origin - if origin_protocol != secure_protocol and secure_protocol != "*": - continue - - try: - addr = ipaddress.ip_address(origin_host or "") - network = ipaddress.ip_network(secure_host) - except ValueError: - # We don't have both a valid address or a valid network, so - # we'll check this origin against hostnames. - if ( - origin_host - and origin_host.lower() != secure_host.lower() - and secure_host != "*" - ): - continue - else: - # We have a valid address and network, so see if the address - # is contained within the network. - if addr not in network: - continue - - # Check to see if the port matches. - if ( - origin_port != secure_port - and secure_port != "*" - and secure_port is not None - ): - continue - - # If we've gotten here, then this origin matches the current - # secure origin and we should return True - return True - - # If we've gotten to this point, then the origin isn't secure and we - # will not accept it as a valid location to search. We will however - # log a warning that we are ignoring it. - logger.warning( - "The repository located at %s is not a trusted or secure host and " - "is being ignored. If this repository is available via HTTPS we " - "recommend you use HTTPS instead, otherwise you may silence " - "this warning and allow it anyway with '--trusted-host %s'.", - origin_host, - origin_host, - ) - - return False - - def request(self, method: str, url: str, *args: Any, **kwargs: Any) -> Response: - # Allow setting a default timeout on a session - kwargs.setdefault("timeout", self.timeout) - # Allow setting a default proxies on a session - kwargs.setdefault("proxies", self.proxies) - - # Dispatch the actual request - return super().request(method, url, *args, **kwargs) diff --git a/venv/lib/python3.11/site-packages/pip/_internal/network/utils.py b/venv/lib/python3.11/site-packages/pip/_internal/network/utils.py deleted file mode 100644 index 134848a..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/network/utils.py +++ /dev/null @@ -1,96 +0,0 @@ -from typing import Dict, Generator - -from pip._vendor.requests.models import CONTENT_CHUNK_SIZE, Response - -from pip._internal.exceptions import NetworkConnectionError - -# The following comments and HTTP headers were originally added by -# Donald Stufft in git commit 22c562429a61bb77172039e480873fb239dd8c03. -# -# We use Accept-Encoding: identity here because requests defaults to -# accepting compressed responses. This breaks in a variety of ways -# depending on how the server is configured. -# - Some servers will notice that the file isn't a compressible file -# and will leave the file alone and with an empty Content-Encoding -# - Some servers will notice that the file is already compressed and -# will leave the file alone, adding a Content-Encoding: gzip header -# - Some servers won't notice anything at all and will take a file -# that's already been compressed and compress it again, and set -# the Content-Encoding: gzip header -# By setting this to request only the identity encoding we're hoping -# to eliminate the third case. Hopefully there does not exist a server -# which when given a file will notice it is already compressed and that -# you're not asking for a compressed file and will then decompress it -# before sending because if that's the case I don't think it'll ever be -# possible to make this work. -HEADERS: Dict[str, str] = {"Accept-Encoding": "identity"} - - -def raise_for_status(resp: Response) -> None: - http_error_msg = "" - if isinstance(resp.reason, bytes): - # We attempt to decode utf-8 first because some servers - # choose to localize their reason strings. If the string - # isn't utf-8, we fall back to iso-8859-1 for all other - # encodings. - try: - reason = resp.reason.decode("utf-8") - except UnicodeDecodeError: - reason = resp.reason.decode("iso-8859-1") - else: - reason = resp.reason - - if 400 <= resp.status_code < 500: - http_error_msg = ( - f"{resp.status_code} Client Error: {reason} for url: {resp.url}" - ) - - elif 500 <= resp.status_code < 600: - http_error_msg = ( - f"{resp.status_code} Server Error: {reason} for url: {resp.url}" - ) - - if http_error_msg: - raise NetworkConnectionError(http_error_msg, response=resp) - - -def response_chunks( - response: Response, chunk_size: int = CONTENT_CHUNK_SIZE -) -> Generator[bytes, None, None]: - """Given a requests Response, provide the data chunks.""" - try: - # Special case for urllib3. - for chunk in response.raw.stream( - chunk_size, - # We use decode_content=False here because we don't - # want urllib3 to mess with the raw bytes we get - # from the server. If we decompress inside of - # urllib3 then we cannot verify the checksum - # because the checksum will be of the compressed - # file. This breakage will only occur if the - # server adds a Content-Encoding header, which - # depends on how the server was configured: - # - Some servers will notice that the file isn't a - # compressible file and will leave the file alone - # and with an empty Content-Encoding - # - Some servers will notice that the file is - # already compressed and will leave the file - # alone and will add a Content-Encoding: gzip - # header - # - Some servers won't notice anything at all and - # will take a file that's already been compressed - # and compress it again and set the - # Content-Encoding: gzip header - # - # By setting this not to decode automatically we - # hope to eliminate problems with the second case. - decode_content=False, - ): - yield chunk - except AttributeError: - # Standard file-like object. - while True: - chunk = response.raw.read(chunk_size) - if not chunk: - break - yield chunk diff --git a/venv/lib/python3.11/site-packages/pip/_internal/network/xmlrpc.py b/venv/lib/python3.11/site-packages/pip/_internal/network/xmlrpc.py deleted file mode 100644 index 4a7d55d..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/network/xmlrpc.py +++ /dev/null @@ -1,60 +0,0 @@ -"""xmlrpclib.Transport implementation -""" - -import logging -import urllib.parse -import xmlrpc.client -from typing import TYPE_CHECKING, Tuple - -from pip._internal.exceptions import NetworkConnectionError -from pip._internal.network.session import PipSession -from pip._internal.network.utils import raise_for_status - -if TYPE_CHECKING: - from xmlrpc.client import _HostType, _Marshallable - -logger = logging.getLogger(__name__) - - -class PipXmlrpcTransport(xmlrpc.client.Transport): - """Provide a `xmlrpclib.Transport` implementation via a `PipSession` - object. - """ - - def __init__( - self, index_url: str, session: PipSession, use_datetime: bool = False - ) -> None: - super().__init__(use_datetime) - index_parts = urllib.parse.urlparse(index_url) - self._scheme = index_parts.scheme - self._session = session - - def request( - self, - host: "_HostType", - handler: str, - request_body: bytes, - verbose: bool = False, - ) -> Tuple["_Marshallable", ...]: - assert isinstance(host, str) - parts = (self._scheme, host, handler, None, None, None) - url = urllib.parse.urlunparse(parts) - try: - headers = {"Content-Type": "text/xml"} - response = self._session.post( - url, - data=request_body, - headers=headers, - stream=True, - ) - raise_for_status(response) - self.verbose = verbose - return self.parse_response(response.raw) - except NetworkConnectionError as exc: - assert exc.response - logger.critical( - "HTTP error %s while getting %s", - exc.response.status_code, - url, - ) - raise diff --git a/venv/lib/python3.11/site-packages/pip/_internal/operations/__init__.py b/venv/lib/python3.11/site-packages/pip/_internal/operations/__init__.py deleted file mode 100644 index e69de29..0000000 diff --git a/venv/lib/python3.11/site-packages/pip/_internal/operations/__pycache__/__init__.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/operations/__pycache__/__init__.cpython-311.pyc deleted file mode 100644 index dc94bec..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/operations/__pycache__/__init__.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/operations/__pycache__/check.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/operations/__pycache__/check.cpython-311.pyc deleted file mode 100644 index 3ca603a..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/operations/__pycache__/check.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/operations/__pycache__/freeze.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/operations/__pycache__/freeze.cpython-311.pyc deleted file mode 100644 index b7fac90..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/operations/__pycache__/freeze.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/operations/__pycache__/prepare.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/operations/__pycache__/prepare.cpython-311.pyc deleted file mode 100644 index 0813c01..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/operations/__pycache__/prepare.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/operations/build/__init__.py b/venv/lib/python3.11/site-packages/pip/_internal/operations/build/__init__.py deleted file mode 100644 index e69de29..0000000 diff --git a/venv/lib/python3.11/site-packages/pip/_internal/operations/build/__pycache__/__init__.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/operations/build/__pycache__/__init__.cpython-311.pyc deleted file mode 100644 index a299be4..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/operations/build/__pycache__/__init__.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/operations/build/__pycache__/build_tracker.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/operations/build/__pycache__/build_tracker.cpython-311.pyc deleted file mode 100644 index 964e7cf..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/operations/build/__pycache__/build_tracker.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/operations/build/__pycache__/metadata.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/operations/build/__pycache__/metadata.cpython-311.pyc deleted file mode 100644 index 6b335a1..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/operations/build/__pycache__/metadata.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/operations/build/__pycache__/metadata_editable.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/operations/build/__pycache__/metadata_editable.cpython-311.pyc deleted file mode 100644 index 07e386c..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/operations/build/__pycache__/metadata_editable.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/operations/build/__pycache__/metadata_legacy.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/operations/build/__pycache__/metadata_legacy.cpython-311.pyc deleted file mode 100644 index f97a8cb..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/operations/build/__pycache__/metadata_legacy.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/operations/build/__pycache__/wheel.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/operations/build/__pycache__/wheel.cpython-311.pyc deleted file mode 100644 index b848217..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/operations/build/__pycache__/wheel.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/operations/build/__pycache__/wheel_editable.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/operations/build/__pycache__/wheel_editable.cpython-311.pyc deleted file mode 100644 index d7fe604..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/operations/build/__pycache__/wheel_editable.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/operations/build/__pycache__/wheel_legacy.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/operations/build/__pycache__/wheel_legacy.cpython-311.pyc deleted file mode 100644 index c34abea..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/operations/build/__pycache__/wheel_legacy.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/operations/build/build_tracker.py b/venv/lib/python3.11/site-packages/pip/_internal/operations/build/build_tracker.py deleted file mode 100644 index 3791932..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/operations/build/build_tracker.py +++ /dev/null @@ -1,139 +0,0 @@ -import contextlib -import hashlib -import logging -import os -from types import TracebackType -from typing import Dict, Generator, Optional, Set, Type, Union - -from pip._internal.models.link import Link -from pip._internal.req.req_install import InstallRequirement -from pip._internal.utils.temp_dir import TempDirectory - -logger = logging.getLogger(__name__) - - -@contextlib.contextmanager -def update_env_context_manager(**changes: str) -> Generator[None, None, None]: - target = os.environ - - # Save values from the target and change them. - non_existent_marker = object() - saved_values: Dict[str, Union[object, str]] = {} - for name, new_value in changes.items(): - try: - saved_values[name] = target[name] - except KeyError: - saved_values[name] = non_existent_marker - target[name] = new_value - - try: - yield - finally: - # Restore original values in the target. - for name, original_value in saved_values.items(): - if original_value is non_existent_marker: - del target[name] - else: - assert isinstance(original_value, str) # for mypy - target[name] = original_value - - -@contextlib.contextmanager -def get_build_tracker() -> Generator["BuildTracker", None, None]: - root = os.environ.get("PIP_BUILD_TRACKER") - with contextlib.ExitStack() as ctx: - if root is None: - root = ctx.enter_context(TempDirectory(kind="build-tracker")).path - ctx.enter_context(update_env_context_manager(PIP_BUILD_TRACKER=root)) - logger.debug("Initialized build tracking at %s", root) - - with BuildTracker(root) as tracker: - yield tracker - - -class TrackerId(str): - """Uniquely identifying string provided to the build tracker.""" - - -class BuildTracker: - """Ensure that an sdist cannot request itself as a setup requirement. - - When an sdist is prepared, it identifies its setup requirements in the - context of ``BuildTracker.track()``. If a requirement shows up recursively, this - raises an exception. - - This stops fork bombs embedded in malicious packages.""" - - def __init__(self, root: str) -> None: - self._root = root - self._entries: Dict[TrackerId, InstallRequirement] = {} - logger.debug("Created build tracker: %s", self._root) - - def __enter__(self) -> "BuildTracker": - logger.debug("Entered build tracker: %s", self._root) - return self - - def __exit__( - self, - exc_type: Optional[Type[BaseException]], - exc_val: Optional[BaseException], - exc_tb: Optional[TracebackType], - ) -> None: - self.cleanup() - - def _entry_path(self, key: TrackerId) -> str: - hashed = hashlib.sha224(key.encode()).hexdigest() - return os.path.join(self._root, hashed) - - def add(self, req: InstallRequirement, key: TrackerId) -> None: - """Add an InstallRequirement to build tracking.""" - - # Get the file to write information about this requirement. - entry_path = self._entry_path(key) - - # Try reading from the file. If it exists and can be read from, a build - # is already in progress, so a LookupError is raised. - try: - with open(entry_path) as fp: - contents = fp.read() - except FileNotFoundError: - pass - else: - message = "{} is already being built: {}".format(req.link, contents) - raise LookupError(message) - - # If we're here, req should really not be building already. - assert key not in self._entries - - # Start tracking this requirement. - with open(entry_path, "w", encoding="utf-8") as fp: - fp.write(str(req)) - self._entries[key] = req - - logger.debug("Added %s to build tracker %r", req, self._root) - - def remove(self, req: InstallRequirement, key: TrackerId) -> None: - """Remove an InstallRequirement from build tracking.""" - - # Delete the created file and the corresponding entry. - os.unlink(self._entry_path(key)) - del self._entries[key] - - logger.debug("Removed %s from build tracker %r", req, self._root) - - def cleanup(self) -> None: - for key, req in list(self._entries.items()): - self.remove(req, key) - - logger.debug("Removed build tracker: %r", self._root) - - @contextlib.contextmanager - def track(self, req: InstallRequirement, key: str) -> Generator[None, None, None]: - """Ensure that `key` cannot install itself as a setup requirement. - - :raises LookupError: If `key` was already provided in a parent invocation of - the context introduced by this method.""" - tracker_id = TrackerId(key) - self.add(req, tracker_id) - yield - self.remove(req, tracker_id) diff --git a/venv/lib/python3.11/site-packages/pip/_internal/operations/build/metadata.py b/venv/lib/python3.11/site-packages/pip/_internal/operations/build/metadata.py deleted file mode 100644 index c66ac35..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/operations/build/metadata.py +++ /dev/null @@ -1,39 +0,0 @@ -"""Metadata generation logic for source distributions. -""" - -import os - -from pip._vendor.pyproject_hooks import BuildBackendHookCaller - -from pip._internal.build_env import BuildEnvironment -from pip._internal.exceptions import ( - InstallationSubprocessError, - MetadataGenerationFailed, -) -from pip._internal.utils.subprocess import runner_with_spinner_message -from pip._internal.utils.temp_dir import TempDirectory - - -def generate_metadata( - build_env: BuildEnvironment, backend: BuildBackendHookCaller, details: str -) -> str: - """Generate metadata using mechanisms described in PEP 517. - - Returns the generated metadata directory. - """ - metadata_tmpdir = TempDirectory(kind="modern-metadata", globally_managed=True) - - metadata_dir = metadata_tmpdir.path - - with build_env: - # Note that BuildBackendHookCaller implements a fallback for - # prepare_metadata_for_build_wheel, so we don't have to - # consider the possibility that this hook doesn't exist. - runner = runner_with_spinner_message("Preparing metadata (pyproject.toml)") - with backend.subprocess_runner(runner): - try: - distinfo_dir = backend.prepare_metadata_for_build_wheel(metadata_dir) - except InstallationSubprocessError as error: - raise MetadataGenerationFailed(package_details=details) from error - - return os.path.join(metadata_dir, distinfo_dir) diff --git a/venv/lib/python3.11/site-packages/pip/_internal/operations/build/metadata_editable.py b/venv/lib/python3.11/site-packages/pip/_internal/operations/build/metadata_editable.py deleted file mode 100644 index 27c69f0..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/operations/build/metadata_editable.py +++ /dev/null @@ -1,41 +0,0 @@ -"""Metadata generation logic for source distributions. -""" - -import os - -from pip._vendor.pyproject_hooks import BuildBackendHookCaller - -from pip._internal.build_env import BuildEnvironment -from pip._internal.exceptions import ( - InstallationSubprocessError, - MetadataGenerationFailed, -) -from pip._internal.utils.subprocess import runner_with_spinner_message -from pip._internal.utils.temp_dir import TempDirectory - - -def generate_editable_metadata( - build_env: BuildEnvironment, backend: BuildBackendHookCaller, details: str -) -> str: - """Generate metadata using mechanisms described in PEP 660. - - Returns the generated metadata directory. - """ - metadata_tmpdir = TempDirectory(kind="modern-metadata", globally_managed=True) - - metadata_dir = metadata_tmpdir.path - - with build_env: - # Note that BuildBackendHookCaller implements a fallback for - # prepare_metadata_for_build_wheel/editable, so we don't have to - # consider the possibility that this hook doesn't exist. - runner = runner_with_spinner_message( - "Preparing editable metadata (pyproject.toml)" - ) - with backend.subprocess_runner(runner): - try: - distinfo_dir = backend.prepare_metadata_for_build_editable(metadata_dir) - except InstallationSubprocessError as error: - raise MetadataGenerationFailed(package_details=details) from error - - return os.path.join(metadata_dir, distinfo_dir) diff --git a/venv/lib/python3.11/site-packages/pip/_internal/operations/build/metadata_legacy.py b/venv/lib/python3.11/site-packages/pip/_internal/operations/build/metadata_legacy.py deleted file mode 100644 index e60988d..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/operations/build/metadata_legacy.py +++ /dev/null @@ -1,74 +0,0 @@ -"""Metadata generation logic for legacy source distributions. -""" - -import logging -import os - -from pip._internal.build_env import BuildEnvironment -from pip._internal.cli.spinners import open_spinner -from pip._internal.exceptions import ( - InstallationError, - InstallationSubprocessError, - MetadataGenerationFailed, -) -from pip._internal.utils.setuptools_build import make_setuptools_egg_info_args -from pip._internal.utils.subprocess import call_subprocess -from pip._internal.utils.temp_dir import TempDirectory - -logger = logging.getLogger(__name__) - - -def _find_egg_info(directory: str) -> str: - """Find an .egg-info subdirectory in `directory`.""" - filenames = [f for f in os.listdir(directory) if f.endswith(".egg-info")] - - if not filenames: - raise InstallationError(f"No .egg-info directory found in {directory}") - - if len(filenames) > 1: - raise InstallationError( - "More than one .egg-info directory found in {}".format(directory) - ) - - return os.path.join(directory, filenames[0]) - - -def generate_metadata( - build_env: BuildEnvironment, - setup_py_path: str, - source_dir: str, - isolated: bool, - details: str, -) -> str: - """Generate metadata using setup.py-based defacto mechanisms. - - Returns the generated metadata directory. - """ - logger.debug( - "Running setup.py (path:%s) egg_info for package %s", - setup_py_path, - details, - ) - - egg_info_dir = TempDirectory(kind="pip-egg-info", globally_managed=True).path - - args = make_setuptools_egg_info_args( - setup_py_path, - egg_info_dir=egg_info_dir, - no_user_config=isolated, - ) - - with build_env: - with open_spinner("Preparing metadata (setup.py)") as spinner: - try: - call_subprocess( - args, - cwd=source_dir, - command_desc="python setup.py egg_info", - spinner=spinner, - ) - except InstallationSubprocessError as error: - raise MetadataGenerationFailed(package_details=details) from error - - # Return the .egg-info directory. - return _find_egg_info(egg_info_dir) diff --git a/venv/lib/python3.11/site-packages/pip/_internal/operations/build/wheel.py b/venv/lib/python3.11/site-packages/pip/_internal/operations/build/wheel.py deleted file mode 100644 index 064811a..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/operations/build/wheel.py +++ /dev/null @@ -1,37 +0,0 @@ -import logging -import os -from typing import Optional - -from pip._vendor.pyproject_hooks import BuildBackendHookCaller - -from pip._internal.utils.subprocess import runner_with_spinner_message - -logger = logging.getLogger(__name__) - - -def build_wheel_pep517( - name: str, - backend: BuildBackendHookCaller, - metadata_directory: str, - tempd: str, -) -> Optional[str]: - """Build one InstallRequirement using the PEP 517 build process. - - Returns path to wheel if successfully built. Otherwise, returns None. - """ - assert metadata_directory is not None - try: - logger.debug("Destination directory: %s", tempd) - - runner = runner_with_spinner_message( - f"Building wheel for {name} (pyproject.toml)" - ) - with backend.subprocess_runner(runner): - wheel_name = backend.build_wheel( - tempd, - metadata_directory=metadata_directory, - ) - except Exception: - logger.error("Failed building wheel for %s", name) - return None - return os.path.join(tempd, wheel_name) diff --git a/venv/lib/python3.11/site-packages/pip/_internal/operations/build/wheel_editable.py b/venv/lib/python3.11/site-packages/pip/_internal/operations/build/wheel_editable.py deleted file mode 100644 index 719d69d..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/operations/build/wheel_editable.py +++ /dev/null @@ -1,46 +0,0 @@ -import logging -import os -from typing import Optional - -from pip._vendor.pyproject_hooks import BuildBackendHookCaller, HookMissing - -from pip._internal.utils.subprocess import runner_with_spinner_message - -logger = logging.getLogger(__name__) - - -def build_wheel_editable( - name: str, - backend: BuildBackendHookCaller, - metadata_directory: str, - tempd: str, -) -> Optional[str]: - """Build one InstallRequirement using the PEP 660 build process. - - Returns path to wheel if successfully built. Otherwise, returns None. - """ - assert metadata_directory is not None - try: - logger.debug("Destination directory: %s", tempd) - - runner = runner_with_spinner_message( - f"Building editable for {name} (pyproject.toml)" - ) - with backend.subprocess_runner(runner): - try: - wheel_name = backend.build_editable( - tempd, - metadata_directory=metadata_directory, - ) - except HookMissing as e: - logger.error( - "Cannot build editable %s because the build " - "backend does not have the %s hook", - name, - e, - ) - return None - except Exception: - logger.error("Failed building editable for %s", name) - return None - return os.path.join(tempd, wheel_name) diff --git a/venv/lib/python3.11/site-packages/pip/_internal/operations/build/wheel_legacy.py b/venv/lib/python3.11/site-packages/pip/_internal/operations/build/wheel_legacy.py deleted file mode 100644 index c5f0492..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/operations/build/wheel_legacy.py +++ /dev/null @@ -1,102 +0,0 @@ -import logging -import os.path -from typing import List, Optional - -from pip._internal.cli.spinners import open_spinner -from pip._internal.utils.setuptools_build import make_setuptools_bdist_wheel_args -from pip._internal.utils.subprocess import call_subprocess, format_command_args - -logger = logging.getLogger(__name__) - - -def format_command_result( - command_args: List[str], - command_output: str, -) -> str: - """Format command information for logging.""" - command_desc = format_command_args(command_args) - text = f"Command arguments: {command_desc}\n" - - if not command_output: - text += "Command output: None" - elif logger.getEffectiveLevel() > logging.DEBUG: - text += "Command output: [use --verbose to show]" - else: - if not command_output.endswith("\n"): - command_output += "\n" - text += f"Command output:\n{command_output}" - - return text - - -def get_legacy_build_wheel_path( - names: List[str], - temp_dir: str, - name: str, - command_args: List[str], - command_output: str, -) -> Optional[str]: - """Return the path to the wheel in the temporary build directory.""" - # Sort for determinism. - names = sorted(names) - if not names: - msg = ("Legacy build of wheel for {!r} created no files.\n").format(name) - msg += format_command_result(command_args, command_output) - logger.warning(msg) - return None - - if len(names) > 1: - msg = ( - "Legacy build of wheel for {!r} created more than one file.\n" - "Filenames (choosing first): {}\n" - ).format(name, names) - msg += format_command_result(command_args, command_output) - logger.warning(msg) - - return os.path.join(temp_dir, names[0]) - - -def build_wheel_legacy( - name: str, - setup_py_path: str, - source_dir: str, - global_options: List[str], - build_options: List[str], - tempd: str, -) -> Optional[str]: - """Build one unpacked package using the "legacy" build process. - - Returns path to wheel if successfully built. Otherwise, returns None. - """ - wheel_args = make_setuptools_bdist_wheel_args( - setup_py_path, - global_options=global_options, - build_options=build_options, - destination_dir=tempd, - ) - - spin_message = f"Building wheel for {name} (setup.py)" - with open_spinner(spin_message) as spinner: - logger.debug("Destination directory: %s", tempd) - - try: - output = call_subprocess( - wheel_args, - command_desc="python setup.py bdist_wheel", - cwd=source_dir, - spinner=spinner, - ) - except Exception: - spinner.finish("error") - logger.error("Failed building wheel for %s", name) - return None - - names = os.listdir(tempd) - wheel_path = get_legacy_build_wheel_path( - names=names, - temp_dir=tempd, - name=name, - command_args=wheel_args, - command_output=output, - ) - return wheel_path diff --git a/venv/lib/python3.11/site-packages/pip/_internal/operations/check.py b/venv/lib/python3.11/site-packages/pip/_internal/operations/check.py deleted file mode 100644 index 1b7fd7a..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/operations/check.py +++ /dev/null @@ -1,187 +0,0 @@ -"""Validation of dependencies of packages -""" - -import logging -from typing import Callable, Dict, List, NamedTuple, Optional, Set, Tuple - -from pip._vendor.packaging.requirements import Requirement -from pip._vendor.packaging.specifiers import LegacySpecifier -from pip._vendor.packaging.utils import NormalizedName, canonicalize_name -from pip._vendor.packaging.version import LegacyVersion - -from pip._internal.distributions import make_distribution_for_install_requirement -from pip._internal.metadata import get_default_environment -from pip._internal.metadata.base import DistributionVersion -from pip._internal.req.req_install import InstallRequirement -from pip._internal.utils.deprecation import deprecated - -logger = logging.getLogger(__name__) - - -class PackageDetails(NamedTuple): - version: DistributionVersion - dependencies: List[Requirement] - - -# Shorthands -PackageSet = Dict[NormalizedName, PackageDetails] -Missing = Tuple[NormalizedName, Requirement] -Conflicting = Tuple[NormalizedName, DistributionVersion, Requirement] - -MissingDict = Dict[NormalizedName, List[Missing]] -ConflictingDict = Dict[NormalizedName, List[Conflicting]] -CheckResult = Tuple[MissingDict, ConflictingDict] -ConflictDetails = Tuple[PackageSet, CheckResult] - - -def create_package_set_from_installed() -> Tuple[PackageSet, bool]: - """Converts a list of distributions into a PackageSet.""" - package_set = {} - problems = False - env = get_default_environment() - for dist in env.iter_installed_distributions(local_only=False, skip=()): - name = dist.canonical_name - try: - dependencies = list(dist.iter_dependencies()) - package_set[name] = PackageDetails(dist.version, dependencies) - except (OSError, ValueError) as e: - # Don't crash on unreadable or broken metadata. - logger.warning("Error parsing requirements for %s: %s", name, e) - problems = True - return package_set, problems - - -def check_package_set( - package_set: PackageSet, should_ignore: Optional[Callable[[str], bool]] = None -) -> CheckResult: - """Check if a package set is consistent - - If should_ignore is passed, it should be a callable that takes a - package name and returns a boolean. - """ - - warn_legacy_versions_and_specifiers(package_set) - - missing = {} - conflicting = {} - - for package_name, package_detail in package_set.items(): - # Info about dependencies of package_name - missing_deps: Set[Missing] = set() - conflicting_deps: Set[Conflicting] = set() - - if should_ignore and should_ignore(package_name): - continue - - for req in package_detail.dependencies: - name = canonicalize_name(req.name) - - # Check if it's missing - if name not in package_set: - missed = True - if req.marker is not None: - missed = req.marker.evaluate({"extra": ""}) - if missed: - missing_deps.add((name, req)) - continue - - # Check if there's a conflict - version = package_set[name].version - if not req.specifier.contains(version, prereleases=True): - conflicting_deps.add((name, version, req)) - - if missing_deps: - missing[package_name] = sorted(missing_deps, key=str) - if conflicting_deps: - conflicting[package_name] = sorted(conflicting_deps, key=str) - - return missing, conflicting - - -def check_install_conflicts(to_install: List[InstallRequirement]) -> ConflictDetails: - """For checking if the dependency graph would be consistent after \ - installing given requirements - """ - # Start from the current state - package_set, _ = create_package_set_from_installed() - # Install packages - would_be_installed = _simulate_installation_of(to_install, package_set) - - # Only warn about directly-dependent packages; create a whitelist of them - whitelist = _create_whitelist(would_be_installed, package_set) - - return ( - package_set, - check_package_set( - package_set, should_ignore=lambda name: name not in whitelist - ), - ) - - -def _simulate_installation_of( - to_install: List[InstallRequirement], package_set: PackageSet -) -> Set[NormalizedName]: - """Computes the version of packages after installing to_install.""" - # Keep track of packages that were installed - installed = set() - - # Modify it as installing requirement_set would (assuming no errors) - for inst_req in to_install: - abstract_dist = make_distribution_for_install_requirement(inst_req) - dist = abstract_dist.get_metadata_distribution() - name = dist.canonical_name - package_set[name] = PackageDetails(dist.version, list(dist.iter_dependencies())) - - installed.add(name) - - return installed - - -def _create_whitelist( - would_be_installed: Set[NormalizedName], package_set: PackageSet -) -> Set[NormalizedName]: - packages_affected = set(would_be_installed) - - for package_name in package_set: - if package_name in packages_affected: - continue - - for req in package_set[package_name].dependencies: - if canonicalize_name(req.name) in packages_affected: - packages_affected.add(package_name) - break - - return packages_affected - - -def warn_legacy_versions_and_specifiers(package_set: PackageSet) -> None: - for project_name, package_details in package_set.items(): - if isinstance(package_details.version, LegacyVersion): - deprecated( - reason=( - f"{project_name} {package_details.version} " - f"has a non-standard version number." - ), - replacement=( - f"to upgrade to a newer version of {project_name} " - f"or contact the author to suggest that they " - f"release a version with a conforming version number" - ), - issue=12063, - gone_in="24.0", - ) - for dep in package_details.dependencies: - if any(isinstance(spec, LegacySpecifier) for spec in dep.specifier): - deprecated( - reason=( - f"{project_name} {package_details.version} " - f"has a non-standard dependency specifier {dep}." - ), - replacement=( - f"to upgrade to a newer version of {project_name} " - f"or contact the author to suggest that they " - f"release a version with a conforming dependency specifiers" - ), - issue=12063, - gone_in="24.0", - ) diff --git a/venv/lib/python3.11/site-packages/pip/_internal/operations/freeze.py b/venv/lib/python3.11/site-packages/pip/_internal/operations/freeze.py deleted file mode 100644 index 3544568..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/operations/freeze.py +++ /dev/null @@ -1,255 +0,0 @@ -import collections -import logging -import os -from typing import Container, Dict, Generator, Iterable, List, NamedTuple, Optional, Set - -from pip._vendor.packaging.utils import canonicalize_name -from pip._vendor.packaging.version import Version - -from pip._internal.exceptions import BadCommand, InstallationError -from pip._internal.metadata import BaseDistribution, get_environment -from pip._internal.req.constructors import ( - install_req_from_editable, - install_req_from_line, -) -from pip._internal.req.req_file import COMMENT_RE -from pip._internal.utils.direct_url_helpers import direct_url_as_pep440_direct_reference - -logger = logging.getLogger(__name__) - - -class _EditableInfo(NamedTuple): - requirement: str - comments: List[str] - - -def freeze( - requirement: Optional[List[str]] = None, - local_only: bool = False, - user_only: bool = False, - paths: Optional[List[str]] = None, - isolated: bool = False, - exclude_editable: bool = False, - skip: Container[str] = (), -) -> Generator[str, None, None]: - installations: Dict[str, FrozenRequirement] = {} - - dists = get_environment(paths).iter_installed_distributions( - local_only=local_only, - skip=(), - user_only=user_only, - ) - for dist in dists: - req = FrozenRequirement.from_dist(dist) - if exclude_editable and req.editable: - continue - installations[req.canonical_name] = req - - if requirement: - # the options that don't get turned into an InstallRequirement - # should only be emitted once, even if the same option is in multiple - # requirements files, so we need to keep track of what has been emitted - # so that we don't emit it again if it's seen again - emitted_options: Set[str] = set() - # keep track of which files a requirement is in so that we can - # give an accurate warning if a requirement appears multiple times. - req_files: Dict[str, List[str]] = collections.defaultdict(list) - for req_file_path in requirement: - with open(req_file_path) as req_file: - for line in req_file: - if ( - not line.strip() - or line.strip().startswith("#") - or line.startswith( - ( - "-r", - "--requirement", - "-f", - "--find-links", - "-i", - "--index-url", - "--pre", - "--trusted-host", - "--process-dependency-links", - "--extra-index-url", - "--use-feature", - ) - ) - ): - line = line.rstrip() - if line not in emitted_options: - emitted_options.add(line) - yield line - continue - - if line.startswith("-e") or line.startswith("--editable"): - if line.startswith("-e"): - line = line[2:].strip() - else: - line = line[len("--editable") :].strip().lstrip("=") - line_req = install_req_from_editable( - line, - isolated=isolated, - ) - else: - line_req = install_req_from_line( - COMMENT_RE.sub("", line).strip(), - isolated=isolated, - ) - - if not line_req.name: - logger.info( - "Skipping line in requirement file [%s] because " - "it's not clear what it would install: %s", - req_file_path, - line.strip(), - ) - logger.info( - " (add #egg=PackageName to the URL to avoid" - " this warning)" - ) - else: - line_req_canonical_name = canonicalize_name(line_req.name) - if line_req_canonical_name not in installations: - # either it's not installed, or it is installed - # but has been processed already - if not req_files[line_req.name]: - logger.warning( - "Requirement file [%s] contains %s, but " - "package %r is not installed", - req_file_path, - COMMENT_RE.sub("", line).strip(), - line_req.name, - ) - else: - req_files[line_req.name].append(req_file_path) - else: - yield str(installations[line_req_canonical_name]).rstrip() - del installations[line_req_canonical_name] - req_files[line_req.name].append(req_file_path) - - # Warn about requirements that were included multiple times (in a - # single requirements file or in different requirements files). - for name, files in req_files.items(): - if len(files) > 1: - logger.warning( - "Requirement %s included multiple times [%s]", - name, - ", ".join(sorted(set(files))), - ) - - yield ("## The following requirements were added by pip freeze:") - for installation in sorted(installations.values(), key=lambda x: x.name.lower()): - if installation.canonical_name not in skip: - yield str(installation).rstrip() - - -def _format_as_name_version(dist: BaseDistribution) -> str: - dist_version = dist.version - if isinstance(dist_version, Version): - return f"{dist.raw_name}=={dist_version}" - return f"{dist.raw_name}==={dist_version}" - - -def _get_editable_info(dist: BaseDistribution) -> _EditableInfo: - """ - Compute and return values (req, comments) for use in - FrozenRequirement.from_dist(). - """ - editable_project_location = dist.editable_project_location - assert editable_project_location - location = os.path.normcase(os.path.abspath(editable_project_location)) - - from pip._internal.vcs import RemoteNotFoundError, RemoteNotValidError, vcs - - vcs_backend = vcs.get_backend_for_dir(location) - - if vcs_backend is None: - display = _format_as_name_version(dist) - logger.debug( - 'No VCS found for editable requirement "%s" in: %r', - display, - location, - ) - return _EditableInfo( - requirement=location, - comments=[f"# Editable install with no version control ({display})"], - ) - - vcs_name = type(vcs_backend).__name__ - - try: - req = vcs_backend.get_src_requirement(location, dist.raw_name) - except RemoteNotFoundError: - display = _format_as_name_version(dist) - return _EditableInfo( - requirement=location, - comments=[f"# Editable {vcs_name} install with no remote ({display})"], - ) - except RemoteNotValidError as ex: - display = _format_as_name_version(dist) - return _EditableInfo( - requirement=location, - comments=[ - f"# Editable {vcs_name} install ({display}) with either a deleted " - f"local remote or invalid URI:", - f"# '{ex.url}'", - ], - ) - except BadCommand: - logger.warning( - "cannot determine version of editable source in %s " - "(%s command not found in path)", - location, - vcs_backend.name, - ) - return _EditableInfo(requirement=location, comments=[]) - except InstallationError as exc: - logger.warning("Error when trying to get requirement for VCS system %s", exc) - else: - return _EditableInfo(requirement=req, comments=[]) - - logger.warning("Could not determine repository location of %s", location) - - return _EditableInfo( - requirement=location, - comments=["## !! Could not determine repository location"], - ) - - -class FrozenRequirement: - def __init__( - self, - name: str, - req: str, - editable: bool, - comments: Iterable[str] = (), - ) -> None: - self.name = name - self.canonical_name = canonicalize_name(name) - self.req = req - self.editable = editable - self.comments = comments - - @classmethod - def from_dist(cls, dist: BaseDistribution) -> "FrozenRequirement": - editable = dist.editable - if editable: - req, comments = _get_editable_info(dist) - else: - comments = [] - direct_url = dist.direct_url - if direct_url: - # if PEP 610 metadata is present, use it - req = direct_url_as_pep440_direct_reference(direct_url, dist.raw_name) - else: - # name==version requirement - req = _format_as_name_version(dist) - - return cls(dist.raw_name, req, editable, comments=comments) - - def __str__(self) -> str: - req = self.req - if self.editable: - req = f"-e {req}" - return "\n".join(list(self.comments) + [str(req)]) + "\n" diff --git a/venv/lib/python3.11/site-packages/pip/_internal/operations/install/__init__.py b/venv/lib/python3.11/site-packages/pip/_internal/operations/install/__init__.py deleted file mode 100644 index 24d6a5d..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/operations/install/__init__.py +++ /dev/null @@ -1,2 +0,0 @@ -"""For modules related to installing packages. -""" diff --git a/venv/lib/python3.11/site-packages/pip/_internal/operations/install/__pycache__/__init__.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/operations/install/__pycache__/__init__.cpython-311.pyc deleted file mode 100644 index df5b12f..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/operations/install/__pycache__/__init__.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/operations/install/__pycache__/editable_legacy.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/operations/install/__pycache__/editable_legacy.cpython-311.pyc deleted file mode 100644 index 561ed00..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/operations/install/__pycache__/editable_legacy.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/operations/install/__pycache__/wheel.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/operations/install/__pycache__/wheel.cpython-311.pyc deleted file mode 100644 index 0d5ba89..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/operations/install/__pycache__/wheel.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/operations/install/editable_legacy.py b/venv/lib/python3.11/site-packages/pip/_internal/operations/install/editable_legacy.py deleted file mode 100644 index bebe24e..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/operations/install/editable_legacy.py +++ /dev/null @@ -1,46 +0,0 @@ -"""Legacy editable installation process, i.e. `setup.py develop`. -""" -import logging -from typing import Optional, Sequence - -from pip._internal.build_env import BuildEnvironment -from pip._internal.utils.logging import indent_log -from pip._internal.utils.setuptools_build import make_setuptools_develop_args -from pip._internal.utils.subprocess import call_subprocess - -logger = logging.getLogger(__name__) - - -def install_editable( - *, - global_options: Sequence[str], - prefix: Optional[str], - home: Optional[str], - use_user_site: bool, - name: str, - setup_py_path: str, - isolated: bool, - build_env: BuildEnvironment, - unpacked_source_directory: str, -) -> None: - """Install a package in editable mode. Most arguments are pass-through - to setuptools. - """ - logger.info("Running setup.py develop for %s", name) - - args = make_setuptools_develop_args( - setup_py_path, - global_options=global_options, - no_user_config=isolated, - prefix=prefix, - home=home, - use_user_site=use_user_site, - ) - - with indent_log(): - with build_env: - call_subprocess( - args, - command_desc="python setup.py develop", - cwd=unpacked_source_directory, - ) diff --git a/venv/lib/python3.11/site-packages/pip/_internal/operations/install/wheel.py b/venv/lib/python3.11/site-packages/pip/_internal/operations/install/wheel.py deleted file mode 100644 index 58a7730..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/operations/install/wheel.py +++ /dev/null @@ -1,740 +0,0 @@ -"""Support for installing and building the "wheel" binary package format. -""" - -import collections -import compileall -import contextlib -import csv -import importlib -import logging -import os.path -import re -import shutil -import sys -import warnings -from base64 import urlsafe_b64encode -from email.message import Message -from itertools import chain, filterfalse, starmap -from typing import ( - IO, - TYPE_CHECKING, - Any, - BinaryIO, - Callable, - Dict, - Generator, - Iterable, - Iterator, - List, - NewType, - Optional, - Sequence, - Set, - Tuple, - Union, - cast, -) -from zipfile import ZipFile, ZipInfo - -from pip._vendor.distlib.scripts import ScriptMaker -from pip._vendor.distlib.util import get_export_entry -from pip._vendor.packaging.utils import canonicalize_name - -from pip._internal.exceptions import InstallationError -from pip._internal.locations import get_major_minor_version -from pip._internal.metadata import ( - BaseDistribution, - FilesystemWheel, - get_wheel_distribution, -) -from pip._internal.models.direct_url import DIRECT_URL_METADATA_NAME, DirectUrl -from pip._internal.models.scheme import SCHEME_KEYS, Scheme -from pip._internal.utils.filesystem import adjacent_tmp_file, replace -from pip._internal.utils.misc import captured_stdout, ensure_dir, hash_file, partition -from pip._internal.utils.unpacking import ( - current_umask, - is_within_directory, - set_extracted_file_to_default_mode_plus_executable, - zip_item_is_executable, -) -from pip._internal.utils.wheel import parse_wheel - -if TYPE_CHECKING: - from typing import Protocol - - class File(Protocol): - src_record_path: "RecordPath" - dest_path: str - changed: bool - - def save(self) -> None: - pass - - -logger = logging.getLogger(__name__) - -RecordPath = NewType("RecordPath", str) -InstalledCSVRow = Tuple[RecordPath, str, Union[int, str]] - - -def rehash(path: str, blocksize: int = 1 << 20) -> Tuple[str, str]: - """Return (encoded_digest, length) for path using hashlib.sha256()""" - h, length = hash_file(path, blocksize) - digest = "sha256=" + urlsafe_b64encode(h.digest()).decode("latin1").rstrip("=") - return (digest, str(length)) - - -def csv_io_kwargs(mode: str) -> Dict[str, Any]: - """Return keyword arguments to properly open a CSV file - in the given mode. - """ - return {"mode": mode, "newline": "", "encoding": "utf-8"} - - -def fix_script(path: str) -> bool: - """Replace #!python with #!/path/to/python - Return True if file was changed. - """ - # XXX RECORD hashes will need to be updated - assert os.path.isfile(path) - - with open(path, "rb") as script: - firstline = script.readline() - if not firstline.startswith(b"#!python"): - return False - exename = sys.executable.encode(sys.getfilesystemencoding()) - firstline = b"#!" + exename + os.linesep.encode("ascii") - rest = script.read() - with open(path, "wb") as script: - script.write(firstline) - script.write(rest) - return True - - -def wheel_root_is_purelib(metadata: Message) -> bool: - return metadata.get("Root-Is-Purelib", "").lower() == "true" - - -def get_entrypoints(dist: BaseDistribution) -> Tuple[Dict[str, str], Dict[str, str]]: - console_scripts = {} - gui_scripts = {} - for entry_point in dist.iter_entry_points(): - if entry_point.group == "console_scripts": - console_scripts[entry_point.name] = entry_point.value - elif entry_point.group == "gui_scripts": - gui_scripts[entry_point.name] = entry_point.value - return console_scripts, gui_scripts - - -def message_about_scripts_not_on_PATH(scripts: Sequence[str]) -> Optional[str]: - """Determine if any scripts are not on PATH and format a warning. - Returns a warning message if one or more scripts are not on PATH, - otherwise None. - """ - if not scripts: - return None - - # Group scripts by the path they were installed in - grouped_by_dir: Dict[str, Set[str]] = collections.defaultdict(set) - for destfile in scripts: - parent_dir = os.path.dirname(destfile) - script_name = os.path.basename(destfile) - grouped_by_dir[parent_dir].add(script_name) - - # We don't want to warn for directories that are on PATH. - not_warn_dirs = [ - os.path.normcase(os.path.normpath(i)).rstrip(os.sep) - for i in os.environ.get("PATH", "").split(os.pathsep) - ] - # If an executable sits with sys.executable, we don't warn for it. - # This covers the case of venv invocations without activating the venv. - not_warn_dirs.append( - os.path.normcase(os.path.normpath(os.path.dirname(sys.executable))) - ) - warn_for: Dict[str, Set[str]] = { - parent_dir: scripts - for parent_dir, scripts in grouped_by_dir.items() - if os.path.normcase(os.path.normpath(parent_dir)) not in not_warn_dirs - } - if not warn_for: - return None - - # Format a message - msg_lines = [] - for parent_dir, dir_scripts in warn_for.items(): - sorted_scripts: List[str] = sorted(dir_scripts) - if len(sorted_scripts) == 1: - start_text = "script {} is".format(sorted_scripts[0]) - else: - start_text = "scripts {} are".format( - ", ".join(sorted_scripts[:-1]) + " and " + sorted_scripts[-1] - ) - - msg_lines.append( - "The {} installed in '{}' which is not on PATH.".format( - start_text, parent_dir - ) - ) - - last_line_fmt = ( - "Consider adding {} to PATH or, if you prefer " - "to suppress this warning, use --no-warn-script-location." - ) - if len(msg_lines) == 1: - msg_lines.append(last_line_fmt.format("this directory")) - else: - msg_lines.append(last_line_fmt.format("these directories")) - - # Add a note if any directory starts with ~ - warn_for_tilde = any( - i[0] == "~" for i in os.environ.get("PATH", "").split(os.pathsep) if i - ) - if warn_for_tilde: - tilde_warning_msg = ( - "NOTE: The current PATH contains path(s) starting with `~`, " - "which may not be expanded by all applications." - ) - msg_lines.append(tilde_warning_msg) - - # Returns the formatted multiline message - return "\n".join(msg_lines) - - -def _normalized_outrows( - outrows: Iterable[InstalledCSVRow], -) -> List[Tuple[str, str, str]]: - """Normalize the given rows of a RECORD file. - - Items in each row are converted into str. Rows are then sorted to make - the value more predictable for tests. - - Each row is a 3-tuple (path, hash, size) and corresponds to a record of - a RECORD file (see PEP 376 and PEP 427 for details). For the rows - passed to this function, the size can be an integer as an int or string, - or the empty string. - """ - # Normally, there should only be one row per path, in which case the - # second and third elements don't come into play when sorting. - # However, in cases in the wild where a path might happen to occur twice, - # we don't want the sort operation to trigger an error (but still want - # determinism). Since the third element can be an int or string, we - # coerce each element to a string to avoid a TypeError in this case. - # For additional background, see-- - # https://github.com/pypa/pip/issues/5868 - return sorted( - (record_path, hash_, str(size)) for record_path, hash_, size in outrows - ) - - -def _record_to_fs_path(record_path: RecordPath, lib_dir: str) -> str: - return os.path.join(lib_dir, record_path) - - -def _fs_to_record_path(path: str, lib_dir: str) -> RecordPath: - # On Windows, do not handle relative paths if they belong to different - # logical disks - if os.path.splitdrive(path)[0].lower() == os.path.splitdrive(lib_dir)[0].lower(): - path = os.path.relpath(path, lib_dir) - - path = path.replace(os.path.sep, "/") - return cast("RecordPath", path) - - -def get_csv_rows_for_installed( - old_csv_rows: List[List[str]], - installed: Dict[RecordPath, RecordPath], - changed: Set[RecordPath], - generated: List[str], - lib_dir: str, -) -> List[InstalledCSVRow]: - """ - :param installed: A map from archive RECORD path to installation RECORD - path. - """ - installed_rows: List[InstalledCSVRow] = [] - for row in old_csv_rows: - if len(row) > 3: - logger.warning("RECORD line has more than three elements: %s", row) - old_record_path = cast("RecordPath", row[0]) - new_record_path = installed.pop(old_record_path, old_record_path) - if new_record_path in changed: - digest, length = rehash(_record_to_fs_path(new_record_path, lib_dir)) - else: - digest = row[1] if len(row) > 1 else "" - length = row[2] if len(row) > 2 else "" - installed_rows.append((new_record_path, digest, length)) - for f in generated: - path = _fs_to_record_path(f, lib_dir) - digest, length = rehash(f) - installed_rows.append((path, digest, length)) - return installed_rows + [ - (installed_record_path, "", "") for installed_record_path in installed.values() - ] - - -def get_console_script_specs(console: Dict[str, str]) -> List[str]: - """ - Given the mapping from entrypoint name to callable, return the relevant - console script specs. - """ - # Don't mutate caller's version - console = console.copy() - - scripts_to_generate = [] - - # Special case pip and setuptools to generate versioned wrappers - # - # The issue is that some projects (specifically, pip and setuptools) use - # code in setup.py to create "versioned" entry points - pip2.7 on Python - # 2.7, pip3.3 on Python 3.3, etc. But these entry points are baked into - # the wheel metadata at build time, and so if the wheel is installed with - # a *different* version of Python the entry points will be wrong. The - # correct fix for this is to enhance the metadata to be able to describe - # such versioned entry points, but that won't happen till Metadata 2.0 is - # available. - # In the meantime, projects using versioned entry points will either have - # incorrect versioned entry points, or they will not be able to distribute - # "universal" wheels (i.e., they will need a wheel per Python version). - # - # Because setuptools and pip are bundled with _ensurepip and virtualenv, - # we need to use universal wheels. So, as a stopgap until Metadata 2.0, we - # override the versioned entry points in the wheel and generate the - # correct ones. This code is purely a short-term measure until Metadata 2.0 - # is available. - # - # To add the level of hack in this section of code, in order to support - # ensurepip this code will look for an ``ENSUREPIP_OPTIONS`` environment - # variable which will control which version scripts get installed. - # - # ENSUREPIP_OPTIONS=altinstall - # - Only pipX.Y and easy_install-X.Y will be generated and installed - # ENSUREPIP_OPTIONS=install - # - pipX.Y, pipX, easy_install-X.Y will be generated and installed. Note - # that this option is technically if ENSUREPIP_OPTIONS is set and is - # not altinstall - # DEFAULT - # - The default behavior is to install pip, pipX, pipX.Y, easy_install - # and easy_install-X.Y. - pip_script = console.pop("pip", None) - if pip_script: - if "ENSUREPIP_OPTIONS" not in os.environ: - scripts_to_generate.append("pip = " + pip_script) - - if os.environ.get("ENSUREPIP_OPTIONS", "") != "altinstall": - scripts_to_generate.append( - "pip{} = {}".format(sys.version_info[0], pip_script) - ) - - scripts_to_generate.append(f"pip{get_major_minor_version()} = {pip_script}") - # Delete any other versioned pip entry points - pip_ep = [k for k in console if re.match(r"pip(\d+(\.\d+)?)?$", k)] - for k in pip_ep: - del console[k] - easy_install_script = console.pop("easy_install", None) - if easy_install_script: - if "ENSUREPIP_OPTIONS" not in os.environ: - scripts_to_generate.append("easy_install = " + easy_install_script) - - scripts_to_generate.append( - "easy_install-{} = {}".format( - get_major_minor_version(), easy_install_script - ) - ) - # Delete any other versioned easy_install entry points - easy_install_ep = [ - k for k in console if re.match(r"easy_install(-\d+\.\d+)?$", k) - ] - for k in easy_install_ep: - del console[k] - - # Generate the console entry points specified in the wheel - scripts_to_generate.extend(starmap("{} = {}".format, console.items())) - - return scripts_to_generate - - -class ZipBackedFile: - def __init__( - self, src_record_path: RecordPath, dest_path: str, zip_file: ZipFile - ) -> None: - self.src_record_path = src_record_path - self.dest_path = dest_path - self._zip_file = zip_file - self.changed = False - - def _getinfo(self) -> ZipInfo: - return self._zip_file.getinfo(self.src_record_path) - - def save(self) -> None: - # directory creation is lazy and after file filtering - # to ensure we don't install empty dirs; empty dirs can't be - # uninstalled. - parent_dir = os.path.dirname(self.dest_path) - ensure_dir(parent_dir) - - # When we open the output file below, any existing file is truncated - # before we start writing the new contents. This is fine in most - # cases, but can cause a segfault if pip has loaded a shared - # object (e.g. from pyopenssl through its vendored urllib3) - # Since the shared object is mmap'd an attempt to call a - # symbol in it will then cause a segfault. Unlinking the file - # allows writing of new contents while allowing the process to - # continue to use the old copy. - if os.path.exists(self.dest_path): - os.unlink(self.dest_path) - - zipinfo = self._getinfo() - - with self._zip_file.open(zipinfo) as f: - with open(self.dest_path, "wb") as dest: - shutil.copyfileobj(f, dest) - - if zip_item_is_executable(zipinfo): - set_extracted_file_to_default_mode_plus_executable(self.dest_path) - - -class ScriptFile: - def __init__(self, file: "File") -> None: - self._file = file - self.src_record_path = self._file.src_record_path - self.dest_path = self._file.dest_path - self.changed = False - - def save(self) -> None: - self._file.save() - self.changed = fix_script(self.dest_path) - - -class MissingCallableSuffix(InstallationError): - def __init__(self, entry_point: str) -> None: - super().__init__( - "Invalid script entry point: {} - A callable " - "suffix is required. Cf https://packaging.python.org/" - "specifications/entry-points/#use-for-scripts for more " - "information.".format(entry_point) - ) - - -def _raise_for_invalid_entrypoint(specification: str) -> None: - entry = get_export_entry(specification) - if entry is not None and entry.suffix is None: - raise MissingCallableSuffix(str(entry)) - - -class PipScriptMaker(ScriptMaker): - def make( - self, specification: str, options: Optional[Dict[str, Any]] = None - ) -> List[str]: - _raise_for_invalid_entrypoint(specification) - return super().make(specification, options) - - -def _install_wheel( - name: str, - wheel_zip: ZipFile, - wheel_path: str, - scheme: Scheme, - pycompile: bool = True, - warn_script_location: bool = True, - direct_url: Optional[DirectUrl] = None, - requested: bool = False, -) -> None: - """Install a wheel. - - :param name: Name of the project to install - :param wheel_zip: open ZipFile for wheel being installed - :param scheme: Distutils scheme dictating the install directories - :param req_description: String used in place of the requirement, for - logging - :param pycompile: Whether to byte-compile installed Python files - :param warn_script_location: Whether to check that scripts are installed - into a directory on PATH - :raises UnsupportedWheel: - * when the directory holds an unpacked wheel with incompatible - Wheel-Version - * when the .dist-info dir does not match the wheel - """ - info_dir, metadata = parse_wheel(wheel_zip, name) - - if wheel_root_is_purelib(metadata): - lib_dir = scheme.purelib - else: - lib_dir = scheme.platlib - - # Record details of the files moved - # installed = files copied from the wheel to the destination - # changed = files changed while installing (scripts #! line typically) - # generated = files newly generated during the install (script wrappers) - installed: Dict[RecordPath, RecordPath] = {} - changed: Set[RecordPath] = set() - generated: List[str] = [] - - def record_installed( - srcfile: RecordPath, destfile: str, modified: bool = False - ) -> None: - """Map archive RECORD paths to installation RECORD paths.""" - newpath = _fs_to_record_path(destfile, lib_dir) - installed[srcfile] = newpath - if modified: - changed.add(newpath) - - def is_dir_path(path: RecordPath) -> bool: - return path.endswith("/") - - def assert_no_path_traversal(dest_dir_path: str, target_path: str) -> None: - if not is_within_directory(dest_dir_path, target_path): - message = ( - "The wheel {!r} has a file {!r} trying to install" - " outside the target directory {!r}" - ) - raise InstallationError( - message.format(wheel_path, target_path, dest_dir_path) - ) - - def root_scheme_file_maker( - zip_file: ZipFile, dest: str - ) -> Callable[[RecordPath], "File"]: - def make_root_scheme_file(record_path: RecordPath) -> "File": - normed_path = os.path.normpath(record_path) - dest_path = os.path.join(dest, normed_path) - assert_no_path_traversal(dest, dest_path) - return ZipBackedFile(record_path, dest_path, zip_file) - - return make_root_scheme_file - - def data_scheme_file_maker( - zip_file: ZipFile, scheme: Scheme - ) -> Callable[[RecordPath], "File"]: - scheme_paths = {key: getattr(scheme, key) for key in SCHEME_KEYS} - - def make_data_scheme_file(record_path: RecordPath) -> "File": - normed_path = os.path.normpath(record_path) - try: - _, scheme_key, dest_subpath = normed_path.split(os.path.sep, 2) - except ValueError: - message = ( - "Unexpected file in {}: {!r}. .data directory contents" - " should be named like: '/'." - ).format(wheel_path, record_path) - raise InstallationError(message) - - try: - scheme_path = scheme_paths[scheme_key] - except KeyError: - valid_scheme_keys = ", ".join(sorted(scheme_paths)) - message = ( - "Unknown scheme key used in {}: {} (for file {!r}). .data" - " directory contents should be in subdirectories named" - " with a valid scheme key ({})" - ).format(wheel_path, scheme_key, record_path, valid_scheme_keys) - raise InstallationError(message) - - dest_path = os.path.join(scheme_path, dest_subpath) - assert_no_path_traversal(scheme_path, dest_path) - return ZipBackedFile(record_path, dest_path, zip_file) - - return make_data_scheme_file - - def is_data_scheme_path(path: RecordPath) -> bool: - return path.split("/", 1)[0].endswith(".data") - - paths = cast(List[RecordPath], wheel_zip.namelist()) - file_paths = filterfalse(is_dir_path, paths) - root_scheme_paths, data_scheme_paths = partition(is_data_scheme_path, file_paths) - - make_root_scheme_file = root_scheme_file_maker(wheel_zip, lib_dir) - files: Iterator[File] = map(make_root_scheme_file, root_scheme_paths) - - def is_script_scheme_path(path: RecordPath) -> bool: - parts = path.split("/", 2) - return len(parts) > 2 and parts[0].endswith(".data") and parts[1] == "scripts" - - other_scheme_paths, script_scheme_paths = partition( - is_script_scheme_path, data_scheme_paths - ) - - make_data_scheme_file = data_scheme_file_maker(wheel_zip, scheme) - other_scheme_files = map(make_data_scheme_file, other_scheme_paths) - files = chain(files, other_scheme_files) - - # Get the defined entry points - distribution = get_wheel_distribution( - FilesystemWheel(wheel_path), - canonicalize_name(name), - ) - console, gui = get_entrypoints(distribution) - - def is_entrypoint_wrapper(file: "File") -> bool: - # EP, EP.exe and EP-script.py are scripts generated for - # entry point EP by setuptools - path = file.dest_path - name = os.path.basename(path) - if name.lower().endswith(".exe"): - matchname = name[:-4] - elif name.lower().endswith("-script.py"): - matchname = name[:-10] - elif name.lower().endswith(".pya"): - matchname = name[:-4] - else: - matchname = name - # Ignore setuptools-generated scripts - return matchname in console or matchname in gui - - script_scheme_files: Iterator[File] = map( - make_data_scheme_file, script_scheme_paths - ) - script_scheme_files = filterfalse(is_entrypoint_wrapper, script_scheme_files) - script_scheme_files = map(ScriptFile, script_scheme_files) - files = chain(files, script_scheme_files) - - for file in files: - file.save() - record_installed(file.src_record_path, file.dest_path, file.changed) - - def pyc_source_file_paths() -> Generator[str, None, None]: - # We de-duplicate installation paths, since there can be overlap (e.g. - # file in .data maps to same location as file in wheel root). - # Sorting installation paths makes it easier to reproduce and debug - # issues related to permissions on existing files. - for installed_path in sorted(set(installed.values())): - full_installed_path = os.path.join(lib_dir, installed_path) - if not os.path.isfile(full_installed_path): - continue - if not full_installed_path.endswith(".py"): - continue - yield full_installed_path - - def pyc_output_path(path: str) -> str: - """Return the path the pyc file would have been written to.""" - return importlib.util.cache_from_source(path) - - # Compile all of the pyc files for the installed files - if pycompile: - with captured_stdout() as stdout: - with warnings.catch_warnings(): - warnings.filterwarnings("ignore") - for path in pyc_source_file_paths(): - success = compileall.compile_file(path, force=True, quiet=True) - if success: - pyc_path = pyc_output_path(path) - assert os.path.exists(pyc_path) - pyc_record_path = cast( - "RecordPath", pyc_path.replace(os.path.sep, "/") - ) - record_installed(pyc_record_path, pyc_path) - logger.debug(stdout.getvalue()) - - maker = PipScriptMaker(None, scheme.scripts) - - # Ensure old scripts are overwritten. - # See https://github.com/pypa/pip/issues/1800 - maker.clobber = True - - # Ensure we don't generate any variants for scripts because this is almost - # never what somebody wants. - # See https://bitbucket.org/pypa/distlib/issue/35/ - maker.variants = {""} - - # This is required because otherwise distlib creates scripts that are not - # executable. - # See https://bitbucket.org/pypa/distlib/issue/32/ - maker.set_mode = True - - # Generate the console and GUI entry points specified in the wheel - scripts_to_generate = get_console_script_specs(console) - - gui_scripts_to_generate = list(starmap("{} = {}".format, gui.items())) - - generated_console_scripts = maker.make_multiple(scripts_to_generate) - generated.extend(generated_console_scripts) - - generated.extend(maker.make_multiple(gui_scripts_to_generate, {"gui": True})) - - if warn_script_location: - msg = message_about_scripts_not_on_PATH(generated_console_scripts) - if msg is not None: - logger.warning(msg) - - generated_file_mode = 0o666 & ~current_umask() - - @contextlib.contextmanager - def _generate_file(path: str, **kwargs: Any) -> Generator[BinaryIO, None, None]: - with adjacent_tmp_file(path, **kwargs) as f: - yield f - os.chmod(f.name, generated_file_mode) - replace(f.name, path) - - dest_info_dir = os.path.join(lib_dir, info_dir) - - # Record pip as the installer - installer_path = os.path.join(dest_info_dir, "INSTALLER") - with _generate_file(installer_path) as installer_file: - installer_file.write(b"pip\n") - generated.append(installer_path) - - # Record the PEP 610 direct URL reference - if direct_url is not None: - direct_url_path = os.path.join(dest_info_dir, DIRECT_URL_METADATA_NAME) - with _generate_file(direct_url_path) as direct_url_file: - direct_url_file.write(direct_url.to_json().encode("utf-8")) - generated.append(direct_url_path) - - # Record the REQUESTED file - if requested: - requested_path = os.path.join(dest_info_dir, "REQUESTED") - with open(requested_path, "wb"): - pass - generated.append(requested_path) - - record_text = distribution.read_text("RECORD") - record_rows = list(csv.reader(record_text.splitlines())) - - rows = get_csv_rows_for_installed( - record_rows, - installed=installed, - changed=changed, - generated=generated, - lib_dir=lib_dir, - ) - - # Record details of all files installed - record_path = os.path.join(dest_info_dir, "RECORD") - - with _generate_file(record_path, **csv_io_kwargs("w")) as record_file: - # Explicitly cast to typing.IO[str] as a workaround for the mypy error: - # "writer" has incompatible type "BinaryIO"; expected "_Writer" - writer = csv.writer(cast("IO[str]", record_file)) - writer.writerows(_normalized_outrows(rows)) - - -@contextlib.contextmanager -def req_error_context(req_description: str) -> Generator[None, None, None]: - try: - yield - except InstallationError as e: - message = "For req: {}. {}".format(req_description, e.args[0]) - raise InstallationError(message) from e - - -def install_wheel( - name: str, - wheel_path: str, - scheme: Scheme, - req_description: str, - pycompile: bool = True, - warn_script_location: bool = True, - direct_url: Optional[DirectUrl] = None, - requested: bool = False, -) -> None: - with ZipFile(wheel_path, allowZip64=True) as z: - with req_error_context(req_description): - _install_wheel( - name=name, - wheel_zip=z, - wheel_path=wheel_path, - scheme=scheme, - pycompile=pycompile, - warn_script_location=warn_script_location, - direct_url=direct_url, - requested=requested, - ) diff --git a/venv/lib/python3.11/site-packages/pip/_internal/operations/prepare.py b/venv/lib/python3.11/site-packages/pip/_internal/operations/prepare.py deleted file mode 100644 index 488e763..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/operations/prepare.py +++ /dev/null @@ -1,730 +0,0 @@ -"""Prepares a distribution for installation -""" - -# The following comment should be removed at some point in the future. -# mypy: strict-optional=False - -import mimetypes -import os -import shutil -from pathlib import Path -from typing import Dict, Iterable, List, Optional - -from pip._vendor.packaging.utils import canonicalize_name - -from pip._internal.distributions import make_distribution_for_install_requirement -from pip._internal.distributions.installed import InstalledDistribution -from pip._internal.exceptions import ( - DirectoryUrlHashUnsupported, - HashMismatch, - HashUnpinned, - InstallationError, - MetadataInconsistent, - NetworkConnectionError, - VcsHashUnsupported, -) -from pip._internal.index.package_finder import PackageFinder -from pip._internal.metadata import BaseDistribution, get_metadata_distribution -from pip._internal.models.direct_url import ArchiveInfo -from pip._internal.models.link import Link -from pip._internal.models.wheel import Wheel -from pip._internal.network.download import BatchDownloader, Downloader -from pip._internal.network.lazy_wheel import ( - HTTPRangeRequestUnsupported, - dist_from_wheel_url, -) -from pip._internal.network.session import PipSession -from pip._internal.operations.build.build_tracker import BuildTracker -from pip._internal.req.req_install import InstallRequirement -from pip._internal.utils._log import getLogger -from pip._internal.utils.direct_url_helpers import ( - direct_url_for_editable, - direct_url_from_link, -) -from pip._internal.utils.hashes import Hashes, MissingHashes -from pip._internal.utils.logging import indent_log -from pip._internal.utils.misc import ( - display_path, - hash_file, - hide_url, - redact_auth_from_requirement, -) -from pip._internal.utils.temp_dir import TempDirectory -from pip._internal.utils.unpacking import unpack_file -from pip._internal.vcs import vcs - -logger = getLogger(__name__) - - -def _get_prepared_distribution( - req: InstallRequirement, - build_tracker: BuildTracker, - finder: PackageFinder, - build_isolation: bool, - check_build_deps: bool, -) -> BaseDistribution: - """Prepare a distribution for installation.""" - abstract_dist = make_distribution_for_install_requirement(req) - tracker_id = abstract_dist.build_tracker_id - if tracker_id is not None: - with build_tracker.track(req, tracker_id): - abstract_dist.prepare_distribution_metadata( - finder, build_isolation, check_build_deps - ) - return abstract_dist.get_metadata_distribution() - - -def unpack_vcs_link(link: Link, location: str, verbosity: int) -> None: - vcs_backend = vcs.get_backend_for_scheme(link.scheme) - assert vcs_backend is not None - vcs_backend.unpack(location, url=hide_url(link.url), verbosity=verbosity) - - -class File: - def __init__(self, path: str, content_type: Optional[str]) -> None: - self.path = path - if content_type is None: - self.content_type = mimetypes.guess_type(path)[0] - else: - self.content_type = content_type - - -def get_http_url( - link: Link, - download: Downloader, - download_dir: Optional[str] = None, - hashes: Optional[Hashes] = None, -) -> File: - temp_dir = TempDirectory(kind="unpack", globally_managed=True) - # If a download dir is specified, is the file already downloaded there? - already_downloaded_path = None - if download_dir: - already_downloaded_path = _check_download_dir(link, download_dir, hashes) - - if already_downloaded_path: - from_path = already_downloaded_path - content_type = None - else: - # let's download to a tmp dir - from_path, content_type = download(link, temp_dir.path) - if hashes: - hashes.check_against_path(from_path) - - return File(from_path, content_type) - - -def get_file_url( - link: Link, download_dir: Optional[str] = None, hashes: Optional[Hashes] = None -) -> File: - """Get file and optionally check its hash.""" - # If a download dir is specified, is the file already there and valid? - already_downloaded_path = None - if download_dir: - already_downloaded_path = _check_download_dir(link, download_dir, hashes) - - if already_downloaded_path: - from_path = already_downloaded_path - else: - from_path = link.file_path - - # If --require-hashes is off, `hashes` is either empty, the - # link's embedded hash, or MissingHashes; it is required to - # match. If --require-hashes is on, we are satisfied by any - # hash in `hashes` matching: a URL-based or an option-based - # one; no internet-sourced hash will be in `hashes`. - if hashes: - hashes.check_against_path(from_path) - return File(from_path, None) - - -def unpack_url( - link: Link, - location: str, - download: Downloader, - verbosity: int, - download_dir: Optional[str] = None, - hashes: Optional[Hashes] = None, -) -> Optional[File]: - """Unpack link into location, downloading if required. - - :param hashes: A Hashes object, one of whose embedded hashes must match, - or HashMismatch will be raised. If the Hashes is empty, no matches are - required, and unhashable types of requirements (like VCS ones, which - would ordinarily raise HashUnsupported) are allowed. - """ - # non-editable vcs urls - if link.is_vcs: - unpack_vcs_link(link, location, verbosity=verbosity) - return None - - assert not link.is_existing_dir() - - # file urls - if link.is_file: - file = get_file_url(link, download_dir, hashes=hashes) - - # http urls - else: - file = get_http_url( - link, - download, - download_dir, - hashes=hashes, - ) - - # unpack the archive to the build dir location. even when only downloading - # archives, they have to be unpacked to parse dependencies, except wheels - if not link.is_wheel: - unpack_file(file.path, location, file.content_type) - - return file - - -def _check_download_dir( - link: Link, - download_dir: str, - hashes: Optional[Hashes], - warn_on_hash_mismatch: bool = True, -) -> Optional[str]: - """Check download_dir for previously downloaded file with correct hash - If a correct file is found return its path else None - """ - download_path = os.path.join(download_dir, link.filename) - - if not os.path.exists(download_path): - return None - - # If already downloaded, does its hash match? - logger.info("File was already downloaded %s", download_path) - if hashes: - try: - hashes.check_against_path(download_path) - except HashMismatch: - if warn_on_hash_mismatch: - logger.warning( - "Previously-downloaded file %s has bad hash. Re-downloading.", - download_path, - ) - os.unlink(download_path) - return None - return download_path - - -class RequirementPreparer: - """Prepares a Requirement""" - - def __init__( - self, - build_dir: str, - download_dir: Optional[str], - src_dir: str, - build_isolation: bool, - check_build_deps: bool, - build_tracker: BuildTracker, - session: PipSession, - progress_bar: str, - finder: PackageFinder, - require_hashes: bool, - use_user_site: bool, - lazy_wheel: bool, - verbosity: int, - legacy_resolver: bool, - ) -> None: - super().__init__() - - self.src_dir = src_dir - self.build_dir = build_dir - self.build_tracker = build_tracker - self._session = session - self._download = Downloader(session, progress_bar) - self._batch_download = BatchDownloader(session, progress_bar) - self.finder = finder - - # Where still-packed archives should be written to. If None, they are - # not saved, and are deleted immediately after unpacking. - self.download_dir = download_dir - - # Is build isolation allowed? - self.build_isolation = build_isolation - - # Should check build dependencies? - self.check_build_deps = check_build_deps - - # Should hash-checking be required? - self.require_hashes = require_hashes - - # Should install in user site-packages? - self.use_user_site = use_user_site - - # Should wheels be downloaded lazily? - self.use_lazy_wheel = lazy_wheel - - # How verbose should underlying tooling be? - self.verbosity = verbosity - - # Are we using the legacy resolver? - self.legacy_resolver = legacy_resolver - - # Memoized downloaded files, as mapping of url: path. - self._downloaded: Dict[str, str] = {} - - # Previous "header" printed for a link-based InstallRequirement - self._previous_requirement_header = ("", "") - - def _log_preparing_link(self, req: InstallRequirement) -> None: - """Provide context for the requirement being prepared.""" - if req.link.is_file and not req.is_wheel_from_cache: - message = "Processing %s" - information = str(display_path(req.link.file_path)) - else: - message = "Collecting %s" - information = redact_auth_from_requirement(req.req) if req.req else str(req) - - # If we used req.req, inject requirement source if available (this - # would already be included if we used req directly) - if req.req and req.comes_from: - if isinstance(req.comes_from, str): - comes_from: Optional[str] = req.comes_from - else: - comes_from = req.comes_from.from_path() - if comes_from: - information += f" (from {comes_from})" - - if (message, information) != self._previous_requirement_header: - self._previous_requirement_header = (message, information) - logger.info(message, information) - - if req.is_wheel_from_cache: - with indent_log(): - logger.info("Using cached %s", req.link.filename) - - def _ensure_link_req_src_dir( - self, req: InstallRequirement, parallel_builds: bool - ) -> None: - """Ensure source_dir of a linked InstallRequirement.""" - # Since source_dir is only set for editable requirements. - if req.link.is_wheel: - # We don't need to unpack wheels, so no need for a source - # directory. - return - assert req.source_dir is None - if req.link.is_existing_dir(): - # build local directories in-tree - req.source_dir = req.link.file_path - return - - # We always delete unpacked sdists after pip runs. - req.ensure_has_source_dir( - self.build_dir, - autodelete=True, - parallel_builds=parallel_builds, - ) - req.ensure_pristine_source_checkout() - - def _get_linked_req_hashes(self, req: InstallRequirement) -> Hashes: - # By the time this is called, the requirement's link should have - # been checked so we can tell what kind of requirements req is - # and raise some more informative errors than otherwise. - # (For example, we can raise VcsHashUnsupported for a VCS URL - # rather than HashMissing.) - if not self.require_hashes: - return req.hashes(trust_internet=True) - - # We could check these first 2 conditions inside unpack_url - # and save repetition of conditions, but then we would - # report less-useful error messages for unhashable - # requirements, complaining that there's no hash provided. - if req.link.is_vcs: - raise VcsHashUnsupported() - if req.link.is_existing_dir(): - raise DirectoryUrlHashUnsupported() - - # Unpinned packages are asking for trouble when a new version - # is uploaded. This isn't a security check, but it saves users - # a surprising hash mismatch in the future. - # file:/// URLs aren't pinnable, so don't complain about them - # not being pinned. - if not req.is_direct and not req.is_pinned: - raise HashUnpinned() - - # If known-good hashes are missing for this requirement, - # shim it with a facade object that will provoke hash - # computation and then raise a HashMissing exception - # showing the user what the hash should be. - return req.hashes(trust_internet=False) or MissingHashes() - - def _fetch_metadata_only( - self, - req: InstallRequirement, - ) -> Optional[BaseDistribution]: - if self.legacy_resolver: - logger.debug( - "Metadata-only fetching is not used in the legacy resolver", - ) - return None - if self.require_hashes: - logger.debug( - "Metadata-only fetching is not used as hash checking is required", - ) - return None - # Try PEP 658 metadata first, then fall back to lazy wheel if unavailable. - return self._fetch_metadata_using_link_data_attr( - req - ) or self._fetch_metadata_using_lazy_wheel(req.link) - - def _fetch_metadata_using_link_data_attr( - self, - req: InstallRequirement, - ) -> Optional[BaseDistribution]: - """Fetch metadata from the data-dist-info-metadata attribute, if possible.""" - # (1) Get the link to the metadata file, if provided by the backend. - metadata_link = req.link.metadata_link() - if metadata_link is None: - return None - assert req.req is not None - logger.verbose( - "Obtaining dependency information for %s from %s", - req.req, - metadata_link, - ) - # (2) Download the contents of the METADATA file, separate from the dist itself. - metadata_file = get_http_url( - metadata_link, - self._download, - hashes=metadata_link.as_hashes(), - ) - with open(metadata_file.path, "rb") as f: - metadata_contents = f.read() - # (3) Generate a dist just from those file contents. - metadata_dist = get_metadata_distribution( - metadata_contents, - req.link.filename, - req.req.name, - ) - # (4) Ensure the Name: field from the METADATA file matches the name from the - # install requirement. - # - # NB: raw_name will fall back to the name from the install requirement if - # the Name: field is not present, but it's noted in the raw_name docstring - # that that should NEVER happen anyway. - if canonicalize_name(metadata_dist.raw_name) != canonicalize_name(req.req.name): - raise MetadataInconsistent( - req, "Name", req.req.name, metadata_dist.raw_name - ) - return metadata_dist - - def _fetch_metadata_using_lazy_wheel( - self, - link: Link, - ) -> Optional[BaseDistribution]: - """Fetch metadata using lazy wheel, if possible.""" - # --use-feature=fast-deps must be provided. - if not self.use_lazy_wheel: - return None - if link.is_file or not link.is_wheel: - logger.debug( - "Lazy wheel is not used as %r does not point to a remote wheel", - link, - ) - return None - - wheel = Wheel(link.filename) - name = canonicalize_name(wheel.name) - logger.info( - "Obtaining dependency information from %s %s", - name, - wheel.version, - ) - url = link.url.split("#", 1)[0] - try: - return dist_from_wheel_url(name, url, self._session) - except HTTPRangeRequestUnsupported: - logger.debug("%s does not support range requests", url) - return None - - def _complete_partial_requirements( - self, - partially_downloaded_reqs: Iterable[InstallRequirement], - parallel_builds: bool = False, - ) -> None: - """Download any requirements which were only fetched by metadata.""" - # Download to a temporary directory. These will be copied over as - # needed for downstream 'download', 'wheel', and 'install' commands. - temp_dir = TempDirectory(kind="unpack", globally_managed=True).path - - # Map each link to the requirement that owns it. This allows us to set - # `req.local_file_path` on the appropriate requirement after passing - # all the links at once into BatchDownloader. - links_to_fully_download: Dict[Link, InstallRequirement] = {} - for req in partially_downloaded_reqs: - assert req.link - links_to_fully_download[req.link] = req - - batch_download = self._batch_download( - links_to_fully_download.keys(), - temp_dir, - ) - for link, (filepath, _) in batch_download: - logger.debug("Downloading link %s to %s", link, filepath) - req = links_to_fully_download[link] - # Record the downloaded file path so wheel reqs can extract a Distribution - # in .get_dist(). - req.local_file_path = filepath - # Record that the file is downloaded so we don't do it again in - # _prepare_linked_requirement(). - self._downloaded[req.link.url] = filepath - - # If this is an sdist, we need to unpack it after downloading, but the - # .source_dir won't be set up until we are in _prepare_linked_requirement(). - # Add the downloaded archive to the install requirement to unpack after - # preparing the source dir. - if not req.is_wheel: - req.needs_unpacked_archive(Path(filepath)) - - # This step is necessary to ensure all lazy wheels are processed - # successfully by the 'download', 'wheel', and 'install' commands. - for req in partially_downloaded_reqs: - self._prepare_linked_requirement(req, parallel_builds) - - def prepare_linked_requirement( - self, req: InstallRequirement, parallel_builds: bool = False - ) -> BaseDistribution: - """Prepare a requirement to be obtained from req.link.""" - assert req.link - self._log_preparing_link(req) - with indent_log(): - # Check if the relevant file is already available - # in the download directory - file_path = None - if self.download_dir is not None and req.link.is_wheel: - hashes = self._get_linked_req_hashes(req) - file_path = _check_download_dir( - req.link, - self.download_dir, - hashes, - # When a locally built wheel has been found in cache, we don't warn - # about re-downloading when the already downloaded wheel hash does - # not match. This is because the hash must be checked against the - # original link, not the cached link. It that case the already - # downloaded file will be removed and re-fetched from cache (which - # implies a hash check against the cache entry's origin.json). - warn_on_hash_mismatch=not req.is_wheel_from_cache, - ) - - if file_path is not None: - # The file is already available, so mark it as downloaded - self._downloaded[req.link.url] = file_path - else: - # The file is not available, attempt to fetch only metadata - metadata_dist = self._fetch_metadata_only(req) - if metadata_dist is not None: - req.needs_more_preparation = True - return metadata_dist - - # None of the optimizations worked, fully prepare the requirement - return self._prepare_linked_requirement(req, parallel_builds) - - def prepare_linked_requirements_more( - self, reqs: Iterable[InstallRequirement], parallel_builds: bool = False - ) -> None: - """Prepare linked requirements more, if needed.""" - reqs = [req for req in reqs if req.needs_more_preparation] - for req in reqs: - # Determine if any of these requirements were already downloaded. - if self.download_dir is not None and req.link.is_wheel: - hashes = self._get_linked_req_hashes(req) - file_path = _check_download_dir(req.link, self.download_dir, hashes) - if file_path is not None: - self._downloaded[req.link.url] = file_path - req.needs_more_preparation = False - - # Prepare requirements we found were already downloaded for some - # reason. The other downloads will be completed separately. - partially_downloaded_reqs: List[InstallRequirement] = [] - for req in reqs: - if req.needs_more_preparation: - partially_downloaded_reqs.append(req) - else: - self._prepare_linked_requirement(req, parallel_builds) - - # TODO: separate this part out from RequirementPreparer when the v1 - # resolver can be removed! - self._complete_partial_requirements( - partially_downloaded_reqs, - parallel_builds=parallel_builds, - ) - - def _prepare_linked_requirement( - self, req: InstallRequirement, parallel_builds: bool - ) -> BaseDistribution: - assert req.link - link = req.link - - hashes = self._get_linked_req_hashes(req) - - if hashes and req.is_wheel_from_cache: - assert req.download_info is not None - assert link.is_wheel - assert link.is_file - # We need to verify hashes, and we have found the requirement in the cache - # of locally built wheels. - if ( - isinstance(req.download_info.info, ArchiveInfo) - and req.download_info.info.hashes - and hashes.has_one_of(req.download_info.info.hashes) - ): - # At this point we know the requirement was built from a hashable source - # artifact, and we verified that the cache entry's hash of the original - # artifact matches one of the hashes we expect. We don't verify hashes - # against the cached wheel, because the wheel is not the original. - hashes = None - else: - logger.warning( - "The hashes of the source archive found in cache entry " - "don't match, ignoring cached built wheel " - "and re-downloading source." - ) - req.link = req.cached_wheel_source_link - link = req.link - - self._ensure_link_req_src_dir(req, parallel_builds) - - if link.is_existing_dir(): - local_file = None - elif link.url not in self._downloaded: - try: - local_file = unpack_url( - link, - req.source_dir, - self._download, - self.verbosity, - self.download_dir, - hashes, - ) - except NetworkConnectionError as exc: - raise InstallationError( - "Could not install requirement {} because of HTTP " - "error {} for URL {}".format(req, exc, link) - ) - else: - file_path = self._downloaded[link.url] - if hashes: - hashes.check_against_path(file_path) - local_file = File(file_path, content_type=None) - - # If download_info is set, we got it from the wheel cache. - if req.download_info is None: - # Editables don't go through this function (see - # prepare_editable_requirement). - assert not req.editable - req.download_info = direct_url_from_link(link, req.source_dir) - # Make sure we have a hash in download_info. If we got it as part of the - # URL, it will have been verified and we can rely on it. Otherwise we - # compute it from the downloaded file. - # FIXME: https://github.com/pypa/pip/issues/11943 - if ( - isinstance(req.download_info.info, ArchiveInfo) - and not req.download_info.info.hashes - and local_file - ): - hash = hash_file(local_file.path)[0].hexdigest() - # We populate info.hash for backward compatibility. - # This will automatically populate info.hashes. - req.download_info.info.hash = f"sha256={hash}" - - # For use in later processing, - # preserve the file path on the requirement. - if local_file: - req.local_file_path = local_file.path - - dist = _get_prepared_distribution( - req, - self.build_tracker, - self.finder, - self.build_isolation, - self.check_build_deps, - ) - return dist - - def save_linked_requirement(self, req: InstallRequirement) -> None: - assert self.download_dir is not None - assert req.link is not None - link = req.link - if link.is_vcs or (link.is_existing_dir() and req.editable): - # Make a .zip of the source_dir we already created. - req.archive(self.download_dir) - return - - if link.is_existing_dir(): - logger.debug( - "Not copying link to destination directory " - "since it is a directory: %s", - link, - ) - return - if req.local_file_path is None: - # No distribution was downloaded for this requirement. - return - - download_location = os.path.join(self.download_dir, link.filename) - if not os.path.exists(download_location): - shutil.copy(req.local_file_path, download_location) - download_path = display_path(download_location) - logger.info("Saved %s", download_path) - - def prepare_editable_requirement( - self, - req: InstallRequirement, - ) -> BaseDistribution: - """Prepare an editable requirement.""" - assert req.editable, "cannot prepare a non-editable req as editable" - - logger.info("Obtaining %s", req) - - with indent_log(): - if self.require_hashes: - raise InstallationError( - "The editable requirement {} cannot be installed when " - "requiring hashes, because there is no single file to " - "hash.".format(req) - ) - req.ensure_has_source_dir(self.src_dir) - req.update_editable() - assert req.source_dir - req.download_info = direct_url_for_editable(req.unpacked_source_directory) - - dist = _get_prepared_distribution( - req, - self.build_tracker, - self.finder, - self.build_isolation, - self.check_build_deps, - ) - - req.check_if_exists(self.use_user_site) - - return dist - - def prepare_installed_requirement( - self, - req: InstallRequirement, - skip_reason: str, - ) -> BaseDistribution: - """Prepare an already-installed requirement.""" - assert req.satisfied_by, "req should have been satisfied but isn't" - assert skip_reason is not None, ( - "did not get skip reason skipped but req.satisfied_by " - "is set to {}".format(req.satisfied_by) - ) - logger.info( - "Requirement %s: %s (%s)", skip_reason, req, req.satisfied_by.version - ) - with indent_log(): - if self.require_hashes: - logger.debug( - "Since it is already installed, we are trusting this " - "package without checking its hash. To ensure a " - "completely repeatable environment, install into an " - "empty virtualenv." - ) - return InstalledDistribution(req).get_metadata_distribution() diff --git a/venv/lib/python3.11/site-packages/pip/_internal/pyproject.py b/venv/lib/python3.11/site-packages/pip/_internal/pyproject.py deleted file mode 100644 index eb8e12b..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/pyproject.py +++ /dev/null @@ -1,179 +0,0 @@ -import importlib.util -import os -from collections import namedtuple -from typing import Any, List, Optional - -from pip._vendor import tomli -from pip._vendor.packaging.requirements import InvalidRequirement, Requirement - -from pip._internal.exceptions import ( - InstallationError, - InvalidPyProjectBuildRequires, - MissingPyProjectBuildRequires, -) - - -def _is_list_of_str(obj: Any) -> bool: - return isinstance(obj, list) and all(isinstance(item, str) for item in obj) - - -def make_pyproject_path(unpacked_source_directory: str) -> str: - return os.path.join(unpacked_source_directory, "pyproject.toml") - - -BuildSystemDetails = namedtuple( - "BuildSystemDetails", ["requires", "backend", "check", "backend_path"] -) - - -def load_pyproject_toml( - use_pep517: Optional[bool], pyproject_toml: str, setup_py: str, req_name: str -) -> Optional[BuildSystemDetails]: - """Load the pyproject.toml file. - - Parameters: - use_pep517 - Has the user requested PEP 517 processing? None - means the user hasn't explicitly specified. - pyproject_toml - Location of the project's pyproject.toml file - setup_py - Location of the project's setup.py file - req_name - The name of the requirement we're processing (for - error reporting) - - Returns: - None if we should use the legacy code path, otherwise a tuple - ( - requirements from pyproject.toml, - name of PEP 517 backend, - requirements we should check are installed after setting - up the build environment - directory paths to import the backend from (backend-path), - relative to the project root. - ) - """ - has_pyproject = os.path.isfile(pyproject_toml) - has_setup = os.path.isfile(setup_py) - - if not has_pyproject and not has_setup: - raise InstallationError( - f"{req_name} does not appear to be a Python project: " - f"neither 'setup.py' nor 'pyproject.toml' found." - ) - - if has_pyproject: - with open(pyproject_toml, encoding="utf-8") as f: - pp_toml = tomli.loads(f.read()) - build_system = pp_toml.get("build-system") - else: - build_system = None - - # The following cases must use PEP 517 - # We check for use_pep517 being non-None and falsey because that means - # the user explicitly requested --no-use-pep517. The value 0 as - # opposed to False can occur when the value is provided via an - # environment variable or config file option (due to the quirk of - # strtobool() returning an integer in pip's configuration code). - if has_pyproject and not has_setup: - if use_pep517 is not None and not use_pep517: - raise InstallationError( - "Disabling PEP 517 processing is invalid: " - "project does not have a setup.py" - ) - use_pep517 = True - elif build_system and "build-backend" in build_system: - if use_pep517 is not None and not use_pep517: - raise InstallationError( - "Disabling PEP 517 processing is invalid: " - "project specifies a build backend of {} " - "in pyproject.toml".format(build_system["build-backend"]) - ) - use_pep517 = True - - # If we haven't worked out whether to use PEP 517 yet, - # and the user hasn't explicitly stated a preference, - # we do so if the project has a pyproject.toml file - # or if we cannot import setuptools or wheels. - - # We fallback to PEP 517 when without setuptools or without the wheel package, - # so setuptools can be installed as a default build backend. - # For more info see: - # https://discuss.python.org/t/pip-without-setuptools-could-the-experience-be-improved/11810/9 - # https://github.com/pypa/pip/issues/8559 - elif use_pep517 is None: - use_pep517 = ( - has_pyproject - or not importlib.util.find_spec("setuptools") - or not importlib.util.find_spec("wheel") - ) - - # At this point, we know whether we're going to use PEP 517. - assert use_pep517 is not None - - # If we're using the legacy code path, there is nothing further - # for us to do here. - if not use_pep517: - return None - - if build_system is None: - # Either the user has a pyproject.toml with no build-system - # section, or the user has no pyproject.toml, but has opted in - # explicitly via --use-pep517. - # In the absence of any explicit backend specification, we - # assume the setuptools backend that most closely emulates the - # traditional direct setup.py execution, and require wheel and - # a version of setuptools that supports that backend. - - build_system = { - "requires": ["setuptools>=40.8.0", "wheel"], - "build-backend": "setuptools.build_meta:__legacy__", - } - - # If we're using PEP 517, we have build system information (either - # from pyproject.toml, or defaulted by the code above). - # Note that at this point, we do not know if the user has actually - # specified a backend, though. - assert build_system is not None - - # Ensure that the build-system section in pyproject.toml conforms - # to PEP 518. - - # Specifying the build-system table but not the requires key is invalid - if "requires" not in build_system: - raise MissingPyProjectBuildRequires(package=req_name) - - # Error out if requires is not a list of strings - requires = build_system["requires"] - if not _is_list_of_str(requires): - raise InvalidPyProjectBuildRequires( - package=req_name, - reason="It is not a list of strings.", - ) - - # Each requirement must be valid as per PEP 508 - for requirement in requires: - try: - Requirement(requirement) - except InvalidRequirement as error: - raise InvalidPyProjectBuildRequires( - package=req_name, - reason=f"It contains an invalid requirement: {requirement!r}", - ) from error - - backend = build_system.get("build-backend") - backend_path = build_system.get("backend-path", []) - check: List[str] = [] - if backend is None: - # If the user didn't specify a backend, we assume they want to use - # the setuptools backend. But we can't be sure they have included - # a version of setuptools which supplies the backend. So we - # make a note to check that this requirement is present once - # we have set up the environment. - # This is quite a lot of work to check for a very specific case. But - # the problem is, that case is potentially quite common - projects that - # adopted PEP 518 early for the ability to specify requirements to - # execute setup.py, but never considered needing to mention the build - # tools themselves. The original PEP 518 code had a similar check (but - # implemented in a different way). - backend = "setuptools.build_meta:__legacy__" - check = ["setuptools>=40.8.0"] - - return BuildSystemDetails(requires, backend, check, backend_path) diff --git a/venv/lib/python3.11/site-packages/pip/_internal/req/__init__.py b/venv/lib/python3.11/site-packages/pip/_internal/req/__init__.py deleted file mode 100644 index 16de903..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/req/__init__.py +++ /dev/null @@ -1,92 +0,0 @@ -import collections -import logging -from typing import Generator, List, Optional, Sequence, Tuple - -from pip._internal.utils.logging import indent_log - -from .req_file import parse_requirements -from .req_install import InstallRequirement -from .req_set import RequirementSet - -__all__ = [ - "RequirementSet", - "InstallRequirement", - "parse_requirements", - "install_given_reqs", -] - -logger = logging.getLogger(__name__) - - -class InstallationResult: - def __init__(self, name: str) -> None: - self.name = name - - def __repr__(self) -> str: - return f"InstallationResult(name={self.name!r})" - - -def _validate_requirements( - requirements: List[InstallRequirement], -) -> Generator[Tuple[str, InstallRequirement], None, None]: - for req in requirements: - assert req.name, f"invalid to-be-installed requirement: {req}" - yield req.name, req - - -def install_given_reqs( - requirements: List[InstallRequirement], - global_options: Sequence[str], - root: Optional[str], - home: Optional[str], - prefix: Optional[str], - warn_script_location: bool, - use_user_site: bool, - pycompile: bool, -) -> List[InstallationResult]: - """ - Install everything in the given list. - - (to be called after having downloaded and unpacked the packages) - """ - to_install = collections.OrderedDict(_validate_requirements(requirements)) - - if to_install: - logger.info( - "Installing collected packages: %s", - ", ".join(to_install.keys()), - ) - - installed = [] - - with indent_log(): - for req_name, requirement in to_install.items(): - if requirement.should_reinstall: - logger.info("Attempting uninstall: %s", req_name) - with indent_log(): - uninstalled_pathset = requirement.uninstall(auto_confirm=True) - else: - uninstalled_pathset = None - - try: - requirement.install( - global_options, - root=root, - home=home, - prefix=prefix, - warn_script_location=warn_script_location, - use_user_site=use_user_site, - pycompile=pycompile, - ) - except Exception: - # if install did not succeed, rollback previous uninstall - if uninstalled_pathset and not requirement.install_succeeded: - uninstalled_pathset.rollback() - raise - else: - if uninstalled_pathset and requirement.install_succeeded: - uninstalled_pathset.commit() - - installed.append(InstallationResult(req_name)) - - return installed diff --git a/venv/lib/python3.11/site-packages/pip/_internal/req/__pycache__/__init__.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/req/__pycache__/__init__.cpython-311.pyc deleted file mode 100644 index 5b764ac..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/req/__pycache__/__init__.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/req/__pycache__/constructors.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/req/__pycache__/constructors.cpython-311.pyc deleted file mode 100644 index 9f1ce7c..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/req/__pycache__/constructors.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/req/__pycache__/req_file.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/req/__pycache__/req_file.cpython-311.pyc deleted file mode 100644 index a4bae27..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/req/__pycache__/req_file.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/req/__pycache__/req_install.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/req/__pycache__/req_install.cpython-311.pyc deleted file mode 100644 index a163d04..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/req/__pycache__/req_install.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/req/__pycache__/req_set.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/req/__pycache__/req_set.cpython-311.pyc deleted file mode 100644 index 19c5b44..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/req/__pycache__/req_set.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/req/__pycache__/req_uninstall.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/req/__pycache__/req_uninstall.cpython-311.pyc deleted file mode 100644 index 6ee66f1..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/req/__pycache__/req_uninstall.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/req/constructors.py b/venv/lib/python3.11/site-packages/pip/_internal/req/constructors.py deleted file mode 100644 index b52c9a4..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/req/constructors.py +++ /dev/null @@ -1,576 +0,0 @@ -"""Backing implementation for InstallRequirement's various constructors - -The idea here is that these formed a major chunk of InstallRequirement's size -so, moving them and support code dedicated to them outside of that class -helps creates for better understandability for the rest of the code. - -These are meant to be used elsewhere within pip to create instances of -InstallRequirement. -""" - -import copy -import logging -import os -import re -from typing import Collection, Dict, List, Optional, Set, Tuple, Union - -from pip._vendor.packaging.markers import Marker -from pip._vendor.packaging.requirements import InvalidRequirement, Requirement -from pip._vendor.packaging.specifiers import Specifier - -from pip._internal.exceptions import InstallationError -from pip._internal.models.index import PyPI, TestPyPI -from pip._internal.models.link import Link -from pip._internal.models.wheel import Wheel -from pip._internal.req.req_file import ParsedRequirement -from pip._internal.req.req_install import InstallRequirement -from pip._internal.utils.filetypes import is_archive_file -from pip._internal.utils.misc import is_installable_dir -from pip._internal.utils.packaging import get_requirement -from pip._internal.utils.urls import path_to_url -from pip._internal.vcs import is_url, vcs - -__all__ = [ - "install_req_from_editable", - "install_req_from_line", - "parse_editable", -] - -logger = logging.getLogger(__name__) -operators = Specifier._operators.keys() - - -def _strip_extras(path: str) -> Tuple[str, Optional[str]]: - m = re.match(r"^(.+)(\[[^\]]+\])$", path) - extras = None - if m: - path_no_extras = m.group(1) - extras = m.group(2) - else: - path_no_extras = path - - return path_no_extras, extras - - -def convert_extras(extras: Optional[str]) -> Set[str]: - if not extras: - return set() - return get_requirement("placeholder" + extras.lower()).extras - - -def _set_requirement_extras(req: Requirement, new_extras: Set[str]) -> Requirement: - """ - Returns a new requirement based on the given one, with the supplied extras. If the - given requirement already has extras those are replaced (or dropped if no new extras - are given). - """ - match: Optional[re.Match[str]] = re.fullmatch( - # see https://peps.python.org/pep-0508/#complete-grammar - r"([\w\t .-]+)(\[[^\]]*\])?(.*)", - str(req), - flags=re.ASCII, - ) - # ireq.req is a valid requirement so the regex should always match - assert ( - match is not None - ), f"regex match on requirement {req} failed, this should never happen" - pre: Optional[str] = match.group(1) - post: Optional[str] = match.group(3) - assert ( - pre is not None and post is not None - ), f"regex group selection for requirement {req} failed, this should never happen" - extras: str = "[%s]" % ",".join(sorted(new_extras)) if new_extras else "" - return Requirement(f"{pre}{extras}{post}") - - -def parse_editable(editable_req: str) -> Tuple[Optional[str], str, Set[str]]: - """Parses an editable requirement into: - - a requirement name - - an URL - - extras - - editable options - Accepted requirements: - svn+http://blahblah@rev#egg=Foobar[baz]&subdirectory=version_subdir - .[some_extra] - """ - - url = editable_req - - # If a file path is specified with extras, strip off the extras. - url_no_extras, extras = _strip_extras(url) - - if os.path.isdir(url_no_extras): - # Treating it as code that has already been checked out - url_no_extras = path_to_url(url_no_extras) - - if url_no_extras.lower().startswith("file:"): - package_name = Link(url_no_extras).egg_fragment - if extras: - return ( - package_name, - url_no_extras, - get_requirement("placeholder" + extras.lower()).extras, - ) - else: - return package_name, url_no_extras, set() - - for version_control in vcs: - if url.lower().startswith(f"{version_control}:"): - url = f"{version_control}+{url}" - break - - link = Link(url) - - if not link.is_vcs: - backends = ", ".join(vcs.all_schemes) - raise InstallationError( - f"{editable_req} is not a valid editable requirement. " - f"It should either be a path to a local project or a VCS URL " - f"(beginning with {backends})." - ) - - package_name = link.egg_fragment - if not package_name: - raise InstallationError( - "Could not detect requirement name for '{}', please specify one " - "with #egg=your_package_name".format(editable_req) - ) - return package_name, url, set() - - -def check_first_requirement_in_file(filename: str) -> None: - """Check if file is parsable as a requirements file. - - This is heavily based on ``pkg_resources.parse_requirements``, but - simplified to just check the first meaningful line. - - :raises InvalidRequirement: If the first meaningful line cannot be parsed - as an requirement. - """ - with open(filename, encoding="utf-8", errors="ignore") as f: - # Create a steppable iterator, so we can handle \-continuations. - lines = ( - line - for line in (line.strip() for line in f) - if line and not line.startswith("#") # Skip blank lines/comments. - ) - - for line in lines: - # Drop comments -- a hash without a space may be in a URL. - if " #" in line: - line = line[: line.find(" #")] - # If there is a line continuation, drop it, and append the next line. - if line.endswith("\\"): - line = line[:-2].strip() + next(lines, "") - Requirement(line) - return - - -def deduce_helpful_msg(req: str) -> str: - """Returns helpful msg in case requirements file does not exist, - or cannot be parsed. - - :params req: Requirements file path - """ - if not os.path.exists(req): - return f" File '{req}' does not exist." - msg = " The path does exist. " - # Try to parse and check if it is a requirements file. - try: - check_first_requirement_in_file(req) - except InvalidRequirement: - logger.debug("Cannot parse '%s' as requirements file", req) - else: - msg += ( - f"The argument you provided " - f"({req}) appears to be a" - f" requirements file. If that is the" - f" case, use the '-r' flag to install" - f" the packages specified within it." - ) - return msg - - -class RequirementParts: - def __init__( - self, - requirement: Optional[Requirement], - link: Optional[Link], - markers: Optional[Marker], - extras: Set[str], - ): - self.requirement = requirement - self.link = link - self.markers = markers - self.extras = extras - - -def parse_req_from_editable(editable_req: str) -> RequirementParts: - name, url, extras_override = parse_editable(editable_req) - - if name is not None: - try: - req: Optional[Requirement] = Requirement(name) - except InvalidRequirement: - raise InstallationError(f"Invalid requirement: '{name}'") - else: - req = None - - link = Link(url) - - return RequirementParts(req, link, None, extras_override) - - -# ---- The actual constructors follow ---- - - -def install_req_from_editable( - editable_req: str, - comes_from: Optional[Union[InstallRequirement, str]] = None, - *, - use_pep517: Optional[bool] = None, - isolated: bool = False, - global_options: Optional[List[str]] = None, - hash_options: Optional[Dict[str, List[str]]] = None, - constraint: bool = False, - user_supplied: bool = False, - permit_editable_wheels: bool = False, - config_settings: Optional[Dict[str, Union[str, List[str]]]] = None, -) -> InstallRequirement: - parts = parse_req_from_editable(editable_req) - - return InstallRequirement( - parts.requirement, - comes_from=comes_from, - user_supplied=user_supplied, - editable=True, - permit_editable_wheels=permit_editable_wheels, - link=parts.link, - constraint=constraint, - use_pep517=use_pep517, - isolated=isolated, - global_options=global_options, - hash_options=hash_options, - config_settings=config_settings, - extras=parts.extras, - ) - - -def _looks_like_path(name: str) -> bool: - """Checks whether the string "looks like" a path on the filesystem. - - This does not check whether the target actually exists, only judge from the - appearance. - - Returns true if any of the following conditions is true: - * a path separator is found (either os.path.sep or os.path.altsep); - * a dot is found (which represents the current directory). - """ - if os.path.sep in name: - return True - if os.path.altsep is not None and os.path.altsep in name: - return True - if name.startswith("."): - return True - return False - - -def _get_url_from_path(path: str, name: str) -> Optional[str]: - """ - First, it checks whether a provided path is an installable directory. If it - is, returns the path. - - If false, check if the path is an archive file (such as a .whl). - The function checks if the path is a file. If false, if the path has - an @, it will treat it as a PEP 440 URL requirement and return the path. - """ - if _looks_like_path(name) and os.path.isdir(path): - if is_installable_dir(path): - return path_to_url(path) - # TODO: The is_installable_dir test here might not be necessary - # now that it is done in load_pyproject_toml too. - raise InstallationError( - f"Directory {name!r} is not installable. Neither 'setup.py' " - "nor 'pyproject.toml' found." - ) - if not is_archive_file(path): - return None - if os.path.isfile(path): - return path_to_url(path) - urlreq_parts = name.split("@", 1) - if len(urlreq_parts) >= 2 and not _looks_like_path(urlreq_parts[0]): - # If the path contains '@' and the part before it does not look - # like a path, try to treat it as a PEP 440 URL req instead. - return None - logger.warning( - "Requirement %r looks like a filename, but the file does not exist", - name, - ) - return path_to_url(path) - - -def parse_req_from_line(name: str, line_source: Optional[str]) -> RequirementParts: - if is_url(name): - marker_sep = "; " - else: - marker_sep = ";" - if marker_sep in name: - name, markers_as_string = name.split(marker_sep, 1) - markers_as_string = markers_as_string.strip() - if not markers_as_string: - markers = None - else: - markers = Marker(markers_as_string) - else: - markers = None - name = name.strip() - req_as_string = None - path = os.path.normpath(os.path.abspath(name)) - link = None - extras_as_string = None - - if is_url(name): - link = Link(name) - else: - p, extras_as_string = _strip_extras(path) - url = _get_url_from_path(p, name) - if url is not None: - link = Link(url) - - # it's a local file, dir, or url - if link: - # Handle relative file URLs - if link.scheme == "file" and re.search(r"\.\./", link.url): - link = Link(path_to_url(os.path.normpath(os.path.abspath(link.path)))) - # wheel file - if link.is_wheel: - wheel = Wheel(link.filename) # can raise InvalidWheelFilename - req_as_string = f"{wheel.name}=={wheel.version}" - else: - # set the req to the egg fragment. when it's not there, this - # will become an 'unnamed' requirement - req_as_string = link.egg_fragment - - # a requirement specifier - else: - req_as_string = name - - extras = convert_extras(extras_as_string) - - def with_source(text: str) -> str: - if not line_source: - return text - return f"{text} (from {line_source})" - - def _parse_req_string(req_as_string: str) -> Requirement: - try: - req = get_requirement(req_as_string) - except InvalidRequirement: - if os.path.sep in req_as_string: - add_msg = "It looks like a path." - add_msg += deduce_helpful_msg(req_as_string) - elif "=" in req_as_string and not any( - op in req_as_string for op in operators - ): - add_msg = "= is not a valid operator. Did you mean == ?" - else: - add_msg = "" - msg = with_source(f"Invalid requirement: {req_as_string!r}") - if add_msg: - msg += f"\nHint: {add_msg}" - raise InstallationError(msg) - else: - # Deprecate extras after specifiers: "name>=1.0[extras]" - # This currently works by accident because _strip_extras() parses - # any extras in the end of the string and those are saved in - # RequirementParts - for spec in req.specifier: - spec_str = str(spec) - if spec_str.endswith("]"): - msg = f"Extras after version '{spec_str}'." - raise InstallationError(msg) - return req - - if req_as_string is not None: - req: Optional[Requirement] = _parse_req_string(req_as_string) - else: - req = None - - return RequirementParts(req, link, markers, extras) - - -def install_req_from_line( - name: str, - comes_from: Optional[Union[str, InstallRequirement]] = None, - *, - use_pep517: Optional[bool] = None, - isolated: bool = False, - global_options: Optional[List[str]] = None, - hash_options: Optional[Dict[str, List[str]]] = None, - constraint: bool = False, - line_source: Optional[str] = None, - user_supplied: bool = False, - config_settings: Optional[Dict[str, Union[str, List[str]]]] = None, -) -> InstallRequirement: - """Creates an InstallRequirement from a name, which might be a - requirement, directory containing 'setup.py', filename, or URL. - - :param line_source: An optional string describing where the line is from, - for logging purposes in case of an error. - """ - parts = parse_req_from_line(name, line_source) - - return InstallRequirement( - parts.requirement, - comes_from, - link=parts.link, - markers=parts.markers, - use_pep517=use_pep517, - isolated=isolated, - global_options=global_options, - hash_options=hash_options, - config_settings=config_settings, - constraint=constraint, - extras=parts.extras, - user_supplied=user_supplied, - ) - - -def install_req_from_req_string( - req_string: str, - comes_from: Optional[InstallRequirement] = None, - isolated: bool = False, - use_pep517: Optional[bool] = None, - user_supplied: bool = False, -) -> InstallRequirement: - try: - req = get_requirement(req_string) - except InvalidRequirement: - raise InstallationError(f"Invalid requirement: '{req_string}'") - - domains_not_allowed = [ - PyPI.file_storage_domain, - TestPyPI.file_storage_domain, - ] - if ( - req.url - and comes_from - and comes_from.link - and comes_from.link.netloc in domains_not_allowed - ): - # Explicitly disallow pypi packages that depend on external urls - raise InstallationError( - "Packages installed from PyPI cannot depend on packages " - "which are not also hosted on PyPI.\n" - "{} depends on {} ".format(comes_from.name, req) - ) - - return InstallRequirement( - req, - comes_from, - isolated=isolated, - use_pep517=use_pep517, - user_supplied=user_supplied, - ) - - -def install_req_from_parsed_requirement( - parsed_req: ParsedRequirement, - isolated: bool = False, - use_pep517: Optional[bool] = None, - user_supplied: bool = False, - config_settings: Optional[Dict[str, Union[str, List[str]]]] = None, -) -> InstallRequirement: - if parsed_req.is_editable: - req = install_req_from_editable( - parsed_req.requirement, - comes_from=parsed_req.comes_from, - use_pep517=use_pep517, - constraint=parsed_req.constraint, - isolated=isolated, - user_supplied=user_supplied, - config_settings=config_settings, - ) - - else: - req = install_req_from_line( - parsed_req.requirement, - comes_from=parsed_req.comes_from, - use_pep517=use_pep517, - isolated=isolated, - global_options=( - parsed_req.options.get("global_options", []) - if parsed_req.options - else [] - ), - hash_options=( - parsed_req.options.get("hashes", {}) if parsed_req.options else {} - ), - constraint=parsed_req.constraint, - line_source=parsed_req.line_source, - user_supplied=user_supplied, - config_settings=config_settings, - ) - return req - - -def install_req_from_link_and_ireq( - link: Link, ireq: InstallRequirement -) -> InstallRequirement: - return InstallRequirement( - req=ireq.req, - comes_from=ireq.comes_from, - editable=ireq.editable, - link=link, - markers=ireq.markers, - use_pep517=ireq.use_pep517, - isolated=ireq.isolated, - global_options=ireq.global_options, - hash_options=ireq.hash_options, - config_settings=ireq.config_settings, - user_supplied=ireq.user_supplied, - ) - - -def install_req_drop_extras(ireq: InstallRequirement) -> InstallRequirement: - """ - Creates a new InstallationRequirement using the given template but without - any extras. Sets the original requirement as the new one's parent - (comes_from). - """ - return InstallRequirement( - req=( - _set_requirement_extras(ireq.req, set()) if ireq.req is not None else None - ), - comes_from=ireq, - editable=ireq.editable, - link=ireq.link, - markers=ireq.markers, - use_pep517=ireq.use_pep517, - isolated=ireq.isolated, - global_options=ireq.global_options, - hash_options=ireq.hash_options, - constraint=ireq.constraint, - extras=[], - config_settings=ireq.config_settings, - user_supplied=ireq.user_supplied, - permit_editable_wheels=ireq.permit_editable_wheels, - ) - - -def install_req_extend_extras( - ireq: InstallRequirement, - extras: Collection[str], -) -> InstallRequirement: - """ - Returns a copy of an installation requirement with some additional extras. - Makes a shallow copy of the ireq object. - """ - result = copy.copy(ireq) - result.extras = {*ireq.extras, *extras} - result.req = ( - _set_requirement_extras(ireq.req, result.extras) - if ireq.req is not None - else None - ) - return result diff --git a/venv/lib/python3.11/site-packages/pip/_internal/req/req_file.py b/venv/lib/python3.11/site-packages/pip/_internal/req/req_file.py deleted file mode 100644 index f717c1c..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/req/req_file.py +++ /dev/null @@ -1,552 +0,0 @@ -""" -Requirements file parsing -""" - -import logging -import optparse -import os -import re -import shlex -import urllib.parse -from optparse import Values -from typing import ( - TYPE_CHECKING, - Any, - Callable, - Dict, - Generator, - Iterable, - List, - Optional, - Tuple, -) - -from pip._internal.cli import cmdoptions -from pip._internal.exceptions import InstallationError, RequirementsFileParseError -from pip._internal.models.search_scope import SearchScope -from pip._internal.network.session import PipSession -from pip._internal.network.utils import raise_for_status -from pip._internal.utils.encoding import auto_decode -from pip._internal.utils.urls import get_url_scheme - -if TYPE_CHECKING: - # NoReturn introduced in 3.6.2; imported only for type checking to maintain - # pip compatibility with older patch versions of Python 3.6 - from typing import NoReturn - - from pip._internal.index.package_finder import PackageFinder - -__all__ = ["parse_requirements"] - -ReqFileLines = Iterable[Tuple[int, str]] - -LineParser = Callable[[str], Tuple[str, Values]] - -SCHEME_RE = re.compile(r"^(http|https|file):", re.I) -COMMENT_RE = re.compile(r"(^|\s+)#.*$") - -# Matches environment variable-style values in '${MY_VARIABLE_1}' with the -# variable name consisting of only uppercase letters, digits or the '_' -# (underscore). This follows the POSIX standard defined in IEEE Std 1003.1, -# 2013 Edition. -ENV_VAR_RE = re.compile(r"(?P\$\{(?P[A-Z0-9_]+)\})") - -SUPPORTED_OPTIONS: List[Callable[..., optparse.Option]] = [ - cmdoptions.index_url, - cmdoptions.extra_index_url, - cmdoptions.no_index, - cmdoptions.constraints, - cmdoptions.requirements, - cmdoptions.editable, - cmdoptions.find_links, - cmdoptions.no_binary, - cmdoptions.only_binary, - cmdoptions.prefer_binary, - cmdoptions.require_hashes, - cmdoptions.pre, - cmdoptions.trusted_host, - cmdoptions.use_new_feature, -] - -# options to be passed to requirements -SUPPORTED_OPTIONS_REQ: List[Callable[..., optparse.Option]] = [ - cmdoptions.global_options, - cmdoptions.hash, - cmdoptions.config_settings, -] - -# the 'dest' string values -SUPPORTED_OPTIONS_REQ_DEST = [str(o().dest) for o in SUPPORTED_OPTIONS_REQ] - -logger = logging.getLogger(__name__) - - -class ParsedRequirement: - def __init__( - self, - requirement: str, - is_editable: bool, - comes_from: str, - constraint: bool, - options: Optional[Dict[str, Any]] = None, - line_source: Optional[str] = None, - ) -> None: - self.requirement = requirement - self.is_editable = is_editable - self.comes_from = comes_from - self.options = options - self.constraint = constraint - self.line_source = line_source - - -class ParsedLine: - def __init__( - self, - filename: str, - lineno: int, - args: str, - opts: Values, - constraint: bool, - ) -> None: - self.filename = filename - self.lineno = lineno - self.opts = opts - self.constraint = constraint - - if args: - self.is_requirement = True - self.is_editable = False - self.requirement = args - elif opts.editables: - self.is_requirement = True - self.is_editable = True - # We don't support multiple -e on one line - self.requirement = opts.editables[0] - else: - self.is_requirement = False - - -def parse_requirements( - filename: str, - session: PipSession, - finder: Optional["PackageFinder"] = None, - options: Optional[optparse.Values] = None, - constraint: bool = False, -) -> Generator[ParsedRequirement, None, None]: - """Parse a requirements file and yield ParsedRequirement instances. - - :param filename: Path or url of requirements file. - :param session: PipSession instance. - :param finder: Instance of pip.index.PackageFinder. - :param options: cli options. - :param constraint: If true, parsing a constraint file rather than - requirements file. - """ - line_parser = get_line_parser(finder) - parser = RequirementsFileParser(session, line_parser) - - for parsed_line in parser.parse(filename, constraint): - parsed_req = handle_line( - parsed_line, options=options, finder=finder, session=session - ) - if parsed_req is not None: - yield parsed_req - - -def preprocess(content: str) -> ReqFileLines: - """Split, filter, and join lines, and return a line iterator - - :param content: the content of the requirements file - """ - lines_enum: ReqFileLines = enumerate(content.splitlines(), start=1) - lines_enum = join_lines(lines_enum) - lines_enum = ignore_comments(lines_enum) - lines_enum = expand_env_variables(lines_enum) - return lines_enum - - -def handle_requirement_line( - line: ParsedLine, - options: Optional[optparse.Values] = None, -) -> ParsedRequirement: - # preserve for the nested code path - line_comes_from = "{} {} (line {})".format( - "-c" if line.constraint else "-r", - line.filename, - line.lineno, - ) - - assert line.is_requirement - - if line.is_editable: - # For editable requirements, we don't support per-requirement - # options, so just return the parsed requirement. - return ParsedRequirement( - requirement=line.requirement, - is_editable=line.is_editable, - comes_from=line_comes_from, - constraint=line.constraint, - ) - else: - # get the options that apply to requirements - req_options = {} - for dest in SUPPORTED_OPTIONS_REQ_DEST: - if dest in line.opts.__dict__ and line.opts.__dict__[dest]: - req_options[dest] = line.opts.__dict__[dest] - - line_source = f"line {line.lineno} of {line.filename}" - return ParsedRequirement( - requirement=line.requirement, - is_editable=line.is_editable, - comes_from=line_comes_from, - constraint=line.constraint, - options=req_options, - line_source=line_source, - ) - - -def handle_option_line( - opts: Values, - filename: str, - lineno: int, - finder: Optional["PackageFinder"] = None, - options: Optional[optparse.Values] = None, - session: Optional[PipSession] = None, -) -> None: - if opts.hashes: - logger.warning( - "%s line %s has --hash but no requirement, and will be ignored.", - filename, - lineno, - ) - - if options: - # percolate options upward - if opts.require_hashes: - options.require_hashes = opts.require_hashes - if opts.features_enabled: - options.features_enabled.extend( - f for f in opts.features_enabled if f not in options.features_enabled - ) - - # set finder options - if finder: - find_links = finder.find_links - index_urls = finder.index_urls - no_index = finder.search_scope.no_index - if opts.no_index is True: - no_index = True - index_urls = [] - if opts.index_url and not no_index: - index_urls = [opts.index_url] - if opts.extra_index_urls and not no_index: - index_urls.extend(opts.extra_index_urls) - if opts.find_links: - # FIXME: it would be nice to keep track of the source - # of the find_links: support a find-links local path - # relative to a requirements file. - value = opts.find_links[0] - req_dir = os.path.dirname(os.path.abspath(filename)) - relative_to_reqs_file = os.path.join(req_dir, value) - if os.path.exists(relative_to_reqs_file): - value = relative_to_reqs_file - find_links.append(value) - - if session: - # We need to update the auth urls in session - session.update_index_urls(index_urls) - - search_scope = SearchScope( - find_links=find_links, - index_urls=index_urls, - no_index=no_index, - ) - finder.search_scope = search_scope - - if opts.pre: - finder.set_allow_all_prereleases() - - if opts.prefer_binary: - finder.set_prefer_binary() - - if session: - for host in opts.trusted_hosts or []: - source = f"line {lineno} of {filename}" - session.add_trusted_host(host, source=source) - - -def handle_line( - line: ParsedLine, - options: Optional[optparse.Values] = None, - finder: Optional["PackageFinder"] = None, - session: Optional[PipSession] = None, -) -> Optional[ParsedRequirement]: - """Handle a single parsed requirements line; This can result in - creating/yielding requirements, or updating the finder. - - :param line: The parsed line to be processed. - :param options: CLI options. - :param finder: The finder - updated by non-requirement lines. - :param session: The session - updated by non-requirement lines. - - Returns a ParsedRequirement object if the line is a requirement line, - otherwise returns None. - - For lines that contain requirements, the only options that have an effect - are from SUPPORTED_OPTIONS_REQ, and they are scoped to the - requirement. Other options from SUPPORTED_OPTIONS may be present, but are - ignored. - - For lines that do not contain requirements, the only options that have an - effect are from SUPPORTED_OPTIONS. Options from SUPPORTED_OPTIONS_REQ may - be present, but are ignored. These lines may contain multiple options - (although our docs imply only one is supported), and all our parsed and - affect the finder. - """ - - if line.is_requirement: - parsed_req = handle_requirement_line(line, options) - return parsed_req - else: - handle_option_line( - line.opts, - line.filename, - line.lineno, - finder, - options, - session, - ) - return None - - -class RequirementsFileParser: - def __init__( - self, - session: PipSession, - line_parser: LineParser, - ) -> None: - self._session = session - self._line_parser = line_parser - - def parse( - self, filename: str, constraint: bool - ) -> Generator[ParsedLine, None, None]: - """Parse a given file, yielding parsed lines.""" - yield from self._parse_and_recurse(filename, constraint) - - def _parse_and_recurse( - self, filename: str, constraint: bool - ) -> Generator[ParsedLine, None, None]: - for line in self._parse_file(filename, constraint): - if not line.is_requirement and ( - line.opts.requirements or line.opts.constraints - ): - # parse a nested requirements file - if line.opts.requirements: - req_path = line.opts.requirements[0] - nested_constraint = False - else: - req_path = line.opts.constraints[0] - nested_constraint = True - - # original file is over http - if SCHEME_RE.search(filename): - # do a url join so relative paths work - req_path = urllib.parse.urljoin(filename, req_path) - # original file and nested file are paths - elif not SCHEME_RE.search(req_path): - # do a join so relative paths work - req_path = os.path.join( - os.path.dirname(filename), - req_path, - ) - - yield from self._parse_and_recurse(req_path, nested_constraint) - else: - yield line - - def _parse_file( - self, filename: str, constraint: bool - ) -> Generator[ParsedLine, None, None]: - _, content = get_file_content(filename, self._session) - - lines_enum = preprocess(content) - - for line_number, line in lines_enum: - try: - args_str, opts = self._line_parser(line) - except OptionParsingError as e: - # add offending line - msg = f"Invalid requirement: {line}\n{e.msg}" - raise RequirementsFileParseError(msg) - - yield ParsedLine( - filename, - line_number, - args_str, - opts, - constraint, - ) - - -def get_line_parser(finder: Optional["PackageFinder"]) -> LineParser: - def parse_line(line: str) -> Tuple[str, Values]: - # Build new parser for each line since it accumulates appendable - # options. - parser = build_parser() - defaults = parser.get_default_values() - defaults.index_url = None - if finder: - defaults.format_control = finder.format_control - - args_str, options_str = break_args_options(line) - - try: - options = shlex.split(options_str) - except ValueError as e: - raise OptionParsingError(f"Could not split options: {options_str}") from e - - opts, _ = parser.parse_args(options, defaults) - - return args_str, opts - - return parse_line - - -def break_args_options(line: str) -> Tuple[str, str]: - """Break up the line into an args and options string. We only want to shlex - (and then optparse) the options, not the args. args can contain markers - which are corrupted by shlex. - """ - tokens = line.split(" ") - args = [] - options = tokens[:] - for token in tokens: - if token.startswith("-") or token.startswith("--"): - break - else: - args.append(token) - options.pop(0) - return " ".join(args), " ".join(options) - - -class OptionParsingError(Exception): - def __init__(self, msg: str) -> None: - self.msg = msg - - -def build_parser() -> optparse.OptionParser: - """ - Return a parser for parsing requirement lines - """ - parser = optparse.OptionParser(add_help_option=False) - - option_factories = SUPPORTED_OPTIONS + SUPPORTED_OPTIONS_REQ - for option_factory in option_factories: - option = option_factory() - parser.add_option(option) - - # By default optparse sys.exits on parsing errors. We want to wrap - # that in our own exception. - def parser_exit(self: Any, msg: str) -> "NoReturn": - raise OptionParsingError(msg) - - # NOTE: mypy disallows assigning to a method - # https://github.com/python/mypy/issues/2427 - parser.exit = parser_exit # type: ignore - - return parser - - -def join_lines(lines_enum: ReqFileLines) -> ReqFileLines: - """Joins a line ending in '\' with the previous line (except when following - comments). The joined line takes on the index of the first line. - """ - primary_line_number = None - new_line: List[str] = [] - for line_number, line in lines_enum: - if not line.endswith("\\") or COMMENT_RE.match(line): - if COMMENT_RE.match(line): - # this ensures comments are always matched later - line = " " + line - if new_line: - new_line.append(line) - assert primary_line_number is not None - yield primary_line_number, "".join(new_line) - new_line = [] - else: - yield line_number, line - else: - if not new_line: - primary_line_number = line_number - new_line.append(line.strip("\\")) - - # last line contains \ - if new_line: - assert primary_line_number is not None - yield primary_line_number, "".join(new_line) - - # TODO: handle space after '\'. - - -def ignore_comments(lines_enum: ReqFileLines) -> ReqFileLines: - """ - Strips comments and filter empty lines. - """ - for line_number, line in lines_enum: - line = COMMENT_RE.sub("", line) - line = line.strip() - if line: - yield line_number, line - - -def expand_env_variables(lines_enum: ReqFileLines) -> ReqFileLines: - """Replace all environment variables that can be retrieved via `os.getenv`. - - The only allowed format for environment variables defined in the - requirement file is `${MY_VARIABLE_1}` to ensure two things: - - 1. Strings that contain a `$` aren't accidentally (partially) expanded. - 2. Ensure consistency across platforms for requirement files. - - These points are the result of a discussion on the `github pull - request #3514 `_. - - Valid characters in variable names follow the `POSIX standard - `_ and are limited - to uppercase letter, digits and the `_` (underscore). - """ - for line_number, line in lines_enum: - for env_var, var_name in ENV_VAR_RE.findall(line): - value = os.getenv(var_name) - if not value: - continue - - line = line.replace(env_var, value) - - yield line_number, line - - -def get_file_content(url: str, session: PipSession) -> Tuple[str, str]: - """Gets the content of a file; it may be a filename, file: URL, or - http: URL. Returns (location, content). Content is unicode. - Respects # -*- coding: declarations on the retrieved files. - - :param url: File path or url. - :param session: PipSession instance. - """ - scheme = get_url_scheme(url) - - # Pip has special support for file:// URLs (LocalFSAdapter). - if scheme in ["http", "https", "file"]: - resp = session.get(url) - raise_for_status(resp) - return resp.url, resp.text - - # Assume this is a bare path. - try: - with open(url, "rb") as f: - content = auto_decode(f.read()) - except OSError as exc: - raise InstallationError(f"Could not open requirements file: {exc}") - return url, content diff --git a/venv/lib/python3.11/site-packages/pip/_internal/req/req_install.py b/venv/lib/python3.11/site-packages/pip/_internal/req/req_install.py deleted file mode 100644 index e556be2..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/req/req_install.py +++ /dev/null @@ -1,914 +0,0 @@ -import functools -import logging -import os -import shutil -import sys -import uuid -import zipfile -from optparse import Values -from pathlib import Path -from typing import Any, Collection, Dict, Iterable, List, Optional, Sequence, Union - -from pip._vendor.packaging.markers import Marker -from pip._vendor.packaging.requirements import Requirement -from pip._vendor.packaging.specifiers import SpecifierSet -from pip._vendor.packaging.utils import canonicalize_name -from pip._vendor.packaging.version import Version -from pip._vendor.packaging.version import parse as parse_version -from pip._vendor.pyproject_hooks import BuildBackendHookCaller - -from pip._internal.build_env import BuildEnvironment, NoOpBuildEnvironment -from pip._internal.exceptions import InstallationError, PreviousBuildDirError -from pip._internal.locations import get_scheme -from pip._internal.metadata import ( - BaseDistribution, - get_default_environment, - get_directory_distribution, - get_wheel_distribution, -) -from pip._internal.metadata.base import FilesystemWheel -from pip._internal.models.direct_url import DirectUrl -from pip._internal.models.link import Link -from pip._internal.operations.build.metadata import generate_metadata -from pip._internal.operations.build.metadata_editable import generate_editable_metadata -from pip._internal.operations.build.metadata_legacy import ( - generate_metadata as generate_metadata_legacy, -) -from pip._internal.operations.install.editable_legacy import ( - install_editable as install_editable_legacy, -) -from pip._internal.operations.install.wheel import install_wheel -from pip._internal.pyproject import load_pyproject_toml, make_pyproject_path -from pip._internal.req.req_uninstall import UninstallPathSet -from pip._internal.utils.deprecation import deprecated -from pip._internal.utils.hashes import Hashes -from pip._internal.utils.misc import ( - ConfiguredBuildBackendHookCaller, - ask_path_exists, - backup_dir, - display_path, - hide_url, - is_installable_dir, - redact_auth_from_requirement, - redact_auth_from_url, -) -from pip._internal.utils.packaging import safe_extra -from pip._internal.utils.subprocess import runner_with_spinner_message -from pip._internal.utils.temp_dir import TempDirectory, tempdir_kinds -from pip._internal.utils.unpacking import unpack_file -from pip._internal.utils.virtualenv import running_under_virtualenv -from pip._internal.vcs import vcs - -logger = logging.getLogger(__name__) - - -class InstallRequirement: - """ - Represents something that may be installed later on, may have information - about where to fetch the relevant requirement and also contains logic for - installing the said requirement. - """ - - def __init__( - self, - req: Optional[Requirement], - comes_from: Optional[Union[str, "InstallRequirement"]], - editable: bool = False, - link: Optional[Link] = None, - markers: Optional[Marker] = None, - use_pep517: Optional[bool] = None, - isolated: bool = False, - *, - global_options: Optional[List[str]] = None, - hash_options: Optional[Dict[str, List[str]]] = None, - config_settings: Optional[Dict[str, Union[str, List[str]]]] = None, - constraint: bool = False, - extras: Collection[str] = (), - user_supplied: bool = False, - permit_editable_wheels: bool = False, - ) -> None: - assert req is None or isinstance(req, Requirement), req - self.req = req - self.comes_from = comes_from - self.constraint = constraint - self.editable = editable - self.permit_editable_wheels = permit_editable_wheels - - # source_dir is the local directory where the linked requirement is - # located, or unpacked. In case unpacking is needed, creating and - # populating source_dir is done by the RequirementPreparer. Note this - # is not necessarily the directory where pyproject.toml or setup.py is - # located - that one is obtained via unpacked_source_directory. - self.source_dir: Optional[str] = None - if self.editable: - assert link - if link.is_file: - self.source_dir = os.path.normpath(os.path.abspath(link.file_path)) - - # original_link is the direct URL that was provided by the user for the - # requirement, either directly or via a constraints file. - if link is None and req and req.url: - # PEP 508 URL requirement - link = Link(req.url) - self.link = self.original_link = link - - # When this InstallRequirement is a wheel obtained from the cache of locally - # built wheels, this is the source link corresponding to the cache entry, which - # was used to download and build the cached wheel. - self.cached_wheel_source_link: Optional[Link] = None - - # Information about the location of the artifact that was downloaded . This - # property is guaranteed to be set in resolver results. - self.download_info: Optional[DirectUrl] = None - - # Path to any downloaded or already-existing package. - self.local_file_path: Optional[str] = None - if self.link and self.link.is_file: - self.local_file_path = self.link.file_path - - if extras: - self.extras = extras - elif req: - self.extras = req.extras - else: - self.extras = set() - if markers is None and req: - markers = req.marker - self.markers = markers - - # This holds the Distribution object if this requirement is already installed. - self.satisfied_by: Optional[BaseDistribution] = None - # Whether the installation process should try to uninstall an existing - # distribution before installing this requirement. - self.should_reinstall = False - # Temporary build location - self._temp_build_dir: Optional[TempDirectory] = None - # Set to True after successful installation - self.install_succeeded: Optional[bool] = None - # Supplied options - self.global_options = global_options if global_options else [] - self.hash_options = hash_options if hash_options else {} - self.config_settings = config_settings - # Set to True after successful preparation of this requirement - self.prepared = False - # User supplied requirement are explicitly requested for installation - # by the user via CLI arguments or requirements files, as opposed to, - # e.g. dependencies, extras or constraints. - self.user_supplied = user_supplied - - self.isolated = isolated - self.build_env: BuildEnvironment = NoOpBuildEnvironment() - - # For PEP 517, the directory where we request the project metadata - # gets stored. We need this to pass to build_wheel, so the backend - # can ensure that the wheel matches the metadata (see the PEP for - # details). - self.metadata_directory: Optional[str] = None - - # The static build requirements (from pyproject.toml) - self.pyproject_requires: Optional[List[str]] = None - - # Build requirements that we will check are available - self.requirements_to_check: List[str] = [] - - # The PEP 517 backend we should use to build the project - self.pep517_backend: Optional[BuildBackendHookCaller] = None - - # Are we using PEP 517 for this requirement? - # After pyproject.toml has been loaded, the only valid values are True - # and False. Before loading, None is valid (meaning "use the default"). - # Setting an explicit value before loading pyproject.toml is supported, - # but after loading this flag should be treated as read only. - self.use_pep517 = use_pep517 - - # This requirement needs more preparation before it can be built - self.needs_more_preparation = False - - # This requirement needs to be unpacked before it can be installed. - self._archive_source: Optional[Path] = None - - def __str__(self) -> str: - if self.req: - s = redact_auth_from_requirement(self.req) - if self.link: - s += " from {}".format(redact_auth_from_url(self.link.url)) - elif self.link: - s = redact_auth_from_url(self.link.url) - else: - s = "" - if self.satisfied_by is not None: - if self.satisfied_by.location is not None: - location = display_path(self.satisfied_by.location) - else: - location = "" - s += f" in {location}" - if self.comes_from: - if isinstance(self.comes_from, str): - comes_from: Optional[str] = self.comes_from - else: - comes_from = self.comes_from.from_path() - if comes_from: - s += f" (from {comes_from})" - return s - - def __repr__(self) -> str: - return "<{} object: {} editable={!r}>".format( - self.__class__.__name__, str(self), self.editable - ) - - def format_debug(self) -> str: - """An un-tested helper for getting state, for debugging.""" - attributes = vars(self) - names = sorted(attributes) - - state = ("{}={!r}".format(attr, attributes[attr]) for attr in sorted(names)) - return "<{name} object: {{{state}}}>".format( - name=self.__class__.__name__, - state=", ".join(state), - ) - - # Things that are valid for all kinds of requirements? - @property - def name(self) -> Optional[str]: - if self.req is None: - return None - return self.req.name - - @functools.lru_cache() # use cached_property in python 3.8+ - def supports_pyproject_editable(self) -> bool: - if not self.use_pep517: - return False - assert self.pep517_backend - with self.build_env: - runner = runner_with_spinner_message( - "Checking if build backend supports build_editable" - ) - with self.pep517_backend.subprocess_runner(runner): - return "build_editable" in self.pep517_backend._supported_features() - - @property - def specifier(self) -> SpecifierSet: - assert self.req is not None - return self.req.specifier - - @property - def is_direct(self) -> bool: - """Whether this requirement was specified as a direct URL.""" - return self.original_link is not None - - @property - def is_pinned(self) -> bool: - """Return whether I am pinned to an exact version. - - For example, some-package==1.2 is pinned; some-package>1.2 is not. - """ - assert self.req is not None - specifiers = self.req.specifier - return len(specifiers) == 1 and next(iter(specifiers)).operator in {"==", "==="} - - def match_markers(self, extras_requested: Optional[Iterable[str]] = None) -> bool: - if not extras_requested: - # Provide an extra to safely evaluate the markers - # without matching any extra - extras_requested = ("",) - if self.markers is not None: - return any( - self.markers.evaluate({"extra": extra}) - # TODO: Remove these two variants when packaging is upgraded to - # support the marker comparison logic specified in PEP 685. - or self.markers.evaluate({"extra": safe_extra(extra)}) - or self.markers.evaluate({"extra": canonicalize_name(extra)}) - for extra in extras_requested - ) - else: - return True - - @property - def has_hash_options(self) -> bool: - """Return whether any known-good hashes are specified as options. - - These activate --require-hashes mode; hashes specified as part of a - URL do not. - - """ - return bool(self.hash_options) - - def hashes(self, trust_internet: bool = True) -> Hashes: - """Return a hash-comparer that considers my option- and URL-based - hashes to be known-good. - - Hashes in URLs--ones embedded in the requirements file, not ones - downloaded from an index server--are almost peers with ones from - flags. They satisfy --require-hashes (whether it was implicitly or - explicitly activated) but do not activate it. md5 and sha224 are not - allowed in flags, which should nudge people toward good algos. We - always OR all hashes together, even ones from URLs. - - :param trust_internet: Whether to trust URL-based (#md5=...) hashes - downloaded from the internet, as by populate_link() - - """ - good_hashes = self.hash_options.copy() - if trust_internet: - link = self.link - elif self.is_direct and self.user_supplied: - link = self.original_link - else: - link = None - if link and link.hash: - assert link.hash_name is not None - good_hashes.setdefault(link.hash_name, []).append(link.hash) - return Hashes(good_hashes) - - def from_path(self) -> Optional[str]: - """Format a nice indicator to show where this "comes from" """ - if self.req is None: - return None - s = str(self.req) - if self.comes_from: - comes_from: Optional[str] - if isinstance(self.comes_from, str): - comes_from = self.comes_from - else: - comes_from = self.comes_from.from_path() - if comes_from: - s += "->" + comes_from - return s - - def ensure_build_location( - self, build_dir: str, autodelete: bool, parallel_builds: bool - ) -> str: - assert build_dir is not None - if self._temp_build_dir is not None: - assert self._temp_build_dir.path - return self._temp_build_dir.path - if self.req is None: - # Some systems have /tmp as a symlink which confuses custom - # builds (such as numpy). Thus, we ensure that the real path - # is returned. - self._temp_build_dir = TempDirectory( - kind=tempdir_kinds.REQ_BUILD, globally_managed=True - ) - - return self._temp_build_dir.path - - # This is the only remaining place where we manually determine the path - # for the temporary directory. It is only needed for editables where - # it is the value of the --src option. - - # When parallel builds are enabled, add a UUID to the build directory - # name so multiple builds do not interfere with each other. - dir_name: str = canonicalize_name(self.req.name) - if parallel_builds: - dir_name = f"{dir_name}_{uuid.uuid4().hex}" - - # FIXME: Is there a better place to create the build_dir? (hg and bzr - # need this) - if not os.path.exists(build_dir): - logger.debug("Creating directory %s", build_dir) - os.makedirs(build_dir) - actual_build_dir = os.path.join(build_dir, dir_name) - # `None` indicates that we respect the globally-configured deletion - # settings, which is what we actually want when auto-deleting. - delete_arg = None if autodelete else False - return TempDirectory( - path=actual_build_dir, - delete=delete_arg, - kind=tempdir_kinds.REQ_BUILD, - globally_managed=True, - ).path - - def _set_requirement(self) -> None: - """Set requirement after generating metadata.""" - assert self.req is None - assert self.metadata is not None - assert self.source_dir is not None - - # Construct a Requirement object from the generated metadata - if isinstance(parse_version(self.metadata["Version"]), Version): - op = "==" - else: - op = "===" - - self.req = Requirement( - "".join( - [ - self.metadata["Name"], - op, - self.metadata["Version"], - ] - ) - ) - - def warn_on_mismatching_name(self) -> None: - assert self.req is not None - metadata_name = canonicalize_name(self.metadata["Name"]) - if canonicalize_name(self.req.name) == metadata_name: - # Everything is fine. - return - - # If we're here, there's a mismatch. Log a warning about it. - logger.warning( - "Generating metadata for package %s " - "produced metadata for project name %s. Fix your " - "#egg=%s fragments.", - self.name, - metadata_name, - self.name, - ) - self.req = Requirement(metadata_name) - - def check_if_exists(self, use_user_site: bool) -> None: - """Find an installed distribution that satisfies or conflicts - with this requirement, and set self.satisfied_by or - self.should_reinstall appropriately. - """ - if self.req is None: - return - existing_dist = get_default_environment().get_distribution(self.req.name) - if not existing_dist: - return - - version_compatible = self.req.specifier.contains( - existing_dist.version, - prereleases=True, - ) - if not version_compatible: - self.satisfied_by = None - if use_user_site: - if existing_dist.in_usersite: - self.should_reinstall = True - elif running_under_virtualenv() and existing_dist.in_site_packages: - raise InstallationError( - f"Will not install to the user site because it will " - f"lack sys.path precedence to {existing_dist.raw_name} " - f"in {existing_dist.location}" - ) - else: - self.should_reinstall = True - else: - if self.editable: - self.should_reinstall = True - # when installing editables, nothing pre-existing should ever - # satisfy - self.satisfied_by = None - else: - self.satisfied_by = existing_dist - - # Things valid for wheels - @property - def is_wheel(self) -> bool: - if not self.link: - return False - return self.link.is_wheel - - @property - def is_wheel_from_cache(self) -> bool: - # When True, it means that this InstallRequirement is a local wheel file in the - # cache of locally built wheels. - return self.cached_wheel_source_link is not None - - # Things valid for sdists - @property - def unpacked_source_directory(self) -> str: - assert self.source_dir, f"No source dir for {self}" - return os.path.join( - self.source_dir, self.link and self.link.subdirectory_fragment or "" - ) - - @property - def setup_py_path(self) -> str: - assert self.source_dir, f"No source dir for {self}" - setup_py = os.path.join(self.unpacked_source_directory, "setup.py") - - return setup_py - - @property - def setup_cfg_path(self) -> str: - assert self.source_dir, f"No source dir for {self}" - setup_cfg = os.path.join(self.unpacked_source_directory, "setup.cfg") - - return setup_cfg - - @property - def pyproject_toml_path(self) -> str: - assert self.source_dir, f"No source dir for {self}" - return make_pyproject_path(self.unpacked_source_directory) - - def load_pyproject_toml(self) -> None: - """Load the pyproject.toml file. - - After calling this routine, all of the attributes related to PEP 517 - processing for this requirement have been set. In particular, the - use_pep517 attribute can be used to determine whether we should - follow the PEP 517 or legacy (setup.py) code path. - """ - pyproject_toml_data = load_pyproject_toml( - self.use_pep517, self.pyproject_toml_path, self.setup_py_path, str(self) - ) - - if pyproject_toml_data is None: - if self.config_settings: - deprecated( - reason=f"Config settings are ignored for project {self}.", - replacement=( - "to use --use-pep517 or add a " - "pyproject.toml file to the project" - ), - gone_in="24.0", - ) - self.use_pep517 = False - return - - self.use_pep517 = True - requires, backend, check, backend_path = pyproject_toml_data - self.requirements_to_check = check - self.pyproject_requires = requires - self.pep517_backend = ConfiguredBuildBackendHookCaller( - self, - self.unpacked_source_directory, - backend, - backend_path=backend_path, - ) - - def isolated_editable_sanity_check(self) -> None: - """Check that an editable requirement if valid for use with PEP 517/518. - - This verifies that an editable that has a pyproject.toml either supports PEP 660 - or as a setup.py or a setup.cfg - """ - if ( - self.editable - and self.use_pep517 - and not self.supports_pyproject_editable() - and not os.path.isfile(self.setup_py_path) - and not os.path.isfile(self.setup_cfg_path) - ): - raise InstallationError( - f"Project {self} has a 'pyproject.toml' and its build " - f"backend is missing the 'build_editable' hook. Since it does not " - f"have a 'setup.py' nor a 'setup.cfg', " - f"it cannot be installed in editable mode. " - f"Consider using a build backend that supports PEP 660." - ) - - def prepare_metadata(self) -> None: - """Ensure that project metadata is available. - - Under PEP 517 and PEP 660, call the backend hook to prepare the metadata. - Under legacy processing, call setup.py egg-info. - """ - assert self.source_dir, f"No source dir for {self}" - details = self.name or f"from {self.link}" - - if self.use_pep517: - assert self.pep517_backend is not None - if ( - self.editable - and self.permit_editable_wheels - and self.supports_pyproject_editable() - ): - self.metadata_directory = generate_editable_metadata( - build_env=self.build_env, - backend=self.pep517_backend, - details=details, - ) - else: - self.metadata_directory = generate_metadata( - build_env=self.build_env, - backend=self.pep517_backend, - details=details, - ) - else: - self.metadata_directory = generate_metadata_legacy( - build_env=self.build_env, - setup_py_path=self.setup_py_path, - source_dir=self.unpacked_source_directory, - isolated=self.isolated, - details=details, - ) - - # Act on the newly generated metadata, based on the name and version. - if not self.name: - self._set_requirement() - else: - self.warn_on_mismatching_name() - - self.assert_source_matches_version() - - @property - def metadata(self) -> Any: - if not hasattr(self, "_metadata"): - self._metadata = self.get_dist().metadata - - return self._metadata - - def get_dist(self) -> BaseDistribution: - if self.metadata_directory: - return get_directory_distribution(self.metadata_directory) - elif self.local_file_path and self.is_wheel: - assert self.req is not None - return get_wheel_distribution( - FilesystemWheel(self.local_file_path), - canonicalize_name(self.req.name), - ) - raise AssertionError( - f"InstallRequirement {self} has no metadata directory and no wheel: " - f"can't make a distribution." - ) - - def assert_source_matches_version(self) -> None: - assert self.source_dir, f"No source dir for {self}" - version = self.metadata["version"] - if self.req and self.req.specifier and version not in self.req.specifier: - logger.warning( - "Requested %s, but installing version %s", - self, - version, - ) - else: - logger.debug( - "Source in %s has version %s, which satisfies requirement %s", - display_path(self.source_dir), - version, - self, - ) - - # For both source distributions and editables - def ensure_has_source_dir( - self, - parent_dir: str, - autodelete: bool = False, - parallel_builds: bool = False, - ) -> None: - """Ensure that a source_dir is set. - - This will create a temporary build dir if the name of the requirement - isn't known yet. - - :param parent_dir: The ideal pip parent_dir for the source_dir. - Generally src_dir for editables and build_dir for sdists. - :return: self.source_dir - """ - if self.source_dir is None: - self.source_dir = self.ensure_build_location( - parent_dir, - autodelete=autodelete, - parallel_builds=parallel_builds, - ) - - def needs_unpacked_archive(self, archive_source: Path) -> None: - assert self._archive_source is None - self._archive_source = archive_source - - def ensure_pristine_source_checkout(self) -> None: - """Ensure the source directory has not yet been built in.""" - assert self.source_dir is not None - if self._archive_source is not None: - unpack_file(str(self._archive_source), self.source_dir) - elif is_installable_dir(self.source_dir): - # If a checkout exists, it's unwise to keep going. - # version inconsistencies are logged later, but do not fail - # the installation. - raise PreviousBuildDirError( - f"pip can't proceed with requirements '{self}' due to a " - f"pre-existing build directory ({self.source_dir}). This is likely " - "due to a previous installation that failed . pip is " - "being responsible and not assuming it can delete this. " - "Please delete it and try again." - ) - - # For editable installations - def update_editable(self) -> None: - if not self.link: - logger.debug( - "Cannot update repository at %s; repository location is unknown", - self.source_dir, - ) - return - assert self.editable - assert self.source_dir - if self.link.scheme == "file": - # Static paths don't get updated - return - vcs_backend = vcs.get_backend_for_scheme(self.link.scheme) - # Editable requirements are validated in Requirement constructors. - # So here, if it's neither a path nor a valid VCS URL, it's a bug. - assert vcs_backend, f"Unsupported VCS URL {self.link.url}" - hidden_url = hide_url(self.link.url) - vcs_backend.obtain(self.source_dir, url=hidden_url, verbosity=0) - - # Top-level Actions - def uninstall( - self, auto_confirm: bool = False, verbose: bool = False - ) -> Optional[UninstallPathSet]: - """ - Uninstall the distribution currently satisfying this requirement. - - Prompts before removing or modifying files unless - ``auto_confirm`` is True. - - Refuses to delete or modify files outside of ``sys.prefix`` - - thus uninstallation within a virtual environment can only - modify that virtual environment, even if the virtualenv is - linked to global site-packages. - - """ - assert self.req - dist = get_default_environment().get_distribution(self.req.name) - if not dist: - logger.warning("Skipping %s as it is not installed.", self.name) - return None - logger.info("Found existing installation: %s", dist) - - uninstalled_pathset = UninstallPathSet.from_dist(dist) - uninstalled_pathset.remove(auto_confirm, verbose) - return uninstalled_pathset - - def _get_archive_name(self, path: str, parentdir: str, rootdir: str) -> str: - def _clean_zip_name(name: str, prefix: str) -> str: - assert name.startswith( - prefix + os.path.sep - ), f"name {name!r} doesn't start with prefix {prefix!r}" - name = name[len(prefix) + 1 :] - name = name.replace(os.path.sep, "/") - return name - - assert self.req is not None - path = os.path.join(parentdir, path) - name = _clean_zip_name(path, rootdir) - return self.req.name + "/" + name - - def archive(self, build_dir: Optional[str]) -> None: - """Saves archive to provided build_dir. - - Used for saving downloaded VCS requirements as part of `pip download`. - """ - assert self.source_dir - if build_dir is None: - return - - create_archive = True - archive_name = "{}-{}.zip".format(self.name, self.metadata["version"]) - archive_path = os.path.join(build_dir, archive_name) - - if os.path.exists(archive_path): - response = ask_path_exists( - "The file {} exists. (i)gnore, (w)ipe, " - "(b)ackup, (a)bort ".format(display_path(archive_path)), - ("i", "w", "b", "a"), - ) - if response == "i": - create_archive = False - elif response == "w": - logger.warning("Deleting %s", display_path(archive_path)) - os.remove(archive_path) - elif response == "b": - dest_file = backup_dir(archive_path) - logger.warning( - "Backing up %s to %s", - display_path(archive_path), - display_path(dest_file), - ) - shutil.move(archive_path, dest_file) - elif response == "a": - sys.exit(-1) - - if not create_archive: - return - - zip_output = zipfile.ZipFile( - archive_path, - "w", - zipfile.ZIP_DEFLATED, - allowZip64=True, - ) - with zip_output: - dir = os.path.normcase(os.path.abspath(self.unpacked_source_directory)) - for dirpath, dirnames, filenames in os.walk(dir): - for dirname in dirnames: - dir_arcname = self._get_archive_name( - dirname, - parentdir=dirpath, - rootdir=dir, - ) - zipdir = zipfile.ZipInfo(dir_arcname + "/") - zipdir.external_attr = 0x1ED << 16 # 0o755 - zip_output.writestr(zipdir, "") - for filename in filenames: - file_arcname = self._get_archive_name( - filename, - parentdir=dirpath, - rootdir=dir, - ) - filename = os.path.join(dirpath, filename) - zip_output.write(filename, file_arcname) - - logger.info("Saved %s", display_path(archive_path)) - - def install( - self, - global_options: Optional[Sequence[str]] = None, - root: Optional[str] = None, - home: Optional[str] = None, - prefix: Optional[str] = None, - warn_script_location: bool = True, - use_user_site: bool = False, - pycompile: bool = True, - ) -> None: - assert self.req is not None - scheme = get_scheme( - self.req.name, - user=use_user_site, - home=home, - root=root, - isolated=self.isolated, - prefix=prefix, - ) - - if self.editable and not self.is_wheel: - install_editable_legacy( - global_options=global_options if global_options is not None else [], - prefix=prefix, - home=home, - use_user_site=use_user_site, - name=self.req.name, - setup_py_path=self.setup_py_path, - isolated=self.isolated, - build_env=self.build_env, - unpacked_source_directory=self.unpacked_source_directory, - ) - self.install_succeeded = True - return - - assert self.is_wheel - assert self.local_file_path - - install_wheel( - self.req.name, - self.local_file_path, - scheme=scheme, - req_description=str(self.req), - pycompile=pycompile, - warn_script_location=warn_script_location, - direct_url=self.download_info if self.is_direct else None, - requested=self.user_supplied, - ) - self.install_succeeded = True - - -def check_invalid_constraint_type(req: InstallRequirement) -> str: - # Check for unsupported forms - problem = "" - if not req.name: - problem = "Unnamed requirements are not allowed as constraints" - elif req.editable: - problem = "Editable requirements are not allowed as constraints" - elif req.extras: - problem = "Constraints cannot have extras" - - if problem: - deprecated( - reason=( - "Constraints are only allowed to take the form of a package " - "name and a version specifier. Other forms were originally " - "permitted as an accident of the implementation, but were " - "undocumented. The new implementation of the resolver no " - "longer supports these forms." - ), - replacement="replacing the constraint with a requirement", - # No plan yet for when the new resolver becomes default - gone_in=None, - issue=8210, - ) - - return problem - - -def _has_option(options: Values, reqs: List[InstallRequirement], option: str) -> bool: - if getattr(options, option, None): - return True - for req in reqs: - if getattr(req, option, None): - return True - return False - - -def check_legacy_setup_py_options( - options: Values, - reqs: List[InstallRequirement], -) -> None: - has_build_options = _has_option(options, reqs, "build_options") - has_global_options = _has_option(options, reqs, "global_options") - if has_build_options or has_global_options: - deprecated( - reason="--build-option and --global-option are deprecated.", - issue=11859, - replacement="to use --config-settings", - gone_in="24.0", - ) - logger.warning( - "Implying --no-binary=:all: due to the presence of " - "--build-option / --global-option. " - ) - options.format_control.disallow_binaries() diff --git a/venv/lib/python3.11/site-packages/pip/_internal/req/req_set.py b/venv/lib/python3.11/site-packages/pip/_internal/req/req_set.py deleted file mode 100644 index 1bf73d5..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/req/req_set.py +++ /dev/null @@ -1,119 +0,0 @@ -import logging -from collections import OrderedDict -from typing import Dict, List - -from pip._vendor.packaging.specifiers import LegacySpecifier -from pip._vendor.packaging.utils import canonicalize_name -from pip._vendor.packaging.version import LegacyVersion - -from pip._internal.req.req_install import InstallRequirement -from pip._internal.utils.deprecation import deprecated - -logger = logging.getLogger(__name__) - - -class RequirementSet: - def __init__(self, check_supported_wheels: bool = True) -> None: - """Create a RequirementSet.""" - - self.requirements: Dict[str, InstallRequirement] = OrderedDict() - self.check_supported_wheels = check_supported_wheels - - self.unnamed_requirements: List[InstallRequirement] = [] - - def __str__(self) -> str: - requirements = sorted( - (req for req in self.requirements.values() if not req.comes_from), - key=lambda req: canonicalize_name(req.name or ""), - ) - return " ".join(str(req.req) for req in requirements) - - def __repr__(self) -> str: - requirements = sorted( - self.requirements.values(), - key=lambda req: canonicalize_name(req.name or ""), - ) - - format_string = "<{classname} object; {count} requirement(s): {reqs}>" - return format_string.format( - classname=self.__class__.__name__, - count=len(requirements), - reqs=", ".join(str(req.req) for req in requirements), - ) - - def add_unnamed_requirement(self, install_req: InstallRequirement) -> None: - assert not install_req.name - self.unnamed_requirements.append(install_req) - - def add_named_requirement(self, install_req: InstallRequirement) -> None: - assert install_req.name - - project_name = canonicalize_name(install_req.name) - self.requirements[project_name] = install_req - - def has_requirement(self, name: str) -> bool: - project_name = canonicalize_name(name) - - return ( - project_name in self.requirements - and not self.requirements[project_name].constraint - ) - - def get_requirement(self, name: str) -> InstallRequirement: - project_name = canonicalize_name(name) - - if project_name in self.requirements: - return self.requirements[project_name] - - raise KeyError(f"No project with the name {name!r}") - - @property - def all_requirements(self) -> List[InstallRequirement]: - return self.unnamed_requirements + list(self.requirements.values()) - - @property - def requirements_to_install(self) -> List[InstallRequirement]: - """Return the list of requirements that need to be installed. - - TODO remove this property together with the legacy resolver, since the new - resolver only returns requirements that need to be installed. - """ - return [ - install_req - for install_req in self.all_requirements - if not install_req.constraint and not install_req.satisfied_by - ] - - def warn_legacy_versions_and_specifiers(self) -> None: - for req in self.requirements_to_install: - version = req.get_dist().version - if isinstance(version, LegacyVersion): - deprecated( - reason=( - f"pip has selected the non standard version {version} " - f"of {req}. In the future this version will be " - f"ignored as it isn't standard compliant." - ), - replacement=( - "set or update constraints to select another version " - "or contact the package author to fix the version number" - ), - issue=12063, - gone_in="24.0", - ) - for dep in req.get_dist().iter_dependencies(): - if any(isinstance(spec, LegacySpecifier) for spec in dep.specifier): - deprecated( - reason=( - f"pip has selected {req} {version} which has non " - f"standard dependency specifier {dep}. " - f"In the future this version of {req} will be " - f"ignored as it isn't standard compliant." - ), - replacement=( - "set or update constraints to select another version " - "or contact the package author to fix the version number" - ), - issue=12063, - gone_in="24.0", - ) diff --git a/venv/lib/python3.11/site-packages/pip/_internal/req/req_uninstall.py b/venv/lib/python3.11/site-packages/pip/_internal/req/req_uninstall.py deleted file mode 100644 index 861aa4f..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/req/req_uninstall.py +++ /dev/null @@ -1,650 +0,0 @@ -import functools -import os -import sys -import sysconfig -from importlib.util import cache_from_source -from typing import Any, Callable, Dict, Generator, Iterable, List, Optional, Set, Tuple - -from pip._internal.exceptions import UninstallationError -from pip._internal.locations import get_bin_prefix, get_bin_user -from pip._internal.metadata import BaseDistribution -from pip._internal.utils.compat import WINDOWS -from pip._internal.utils.egg_link import egg_link_path_from_location -from pip._internal.utils.logging import getLogger, indent_log -from pip._internal.utils.misc import ask, normalize_path, renames, rmtree -from pip._internal.utils.temp_dir import AdjacentTempDirectory, TempDirectory -from pip._internal.utils.virtualenv import running_under_virtualenv - -logger = getLogger(__name__) - - -def _script_names( - bin_dir: str, script_name: str, is_gui: bool -) -> Generator[str, None, None]: - """Create the fully qualified name of the files created by - {console,gui}_scripts for the given ``dist``. - Returns the list of file names - """ - exe_name = os.path.join(bin_dir, script_name) - yield exe_name - if not WINDOWS: - return - yield f"{exe_name}.exe" - yield f"{exe_name}.exe.manifest" - if is_gui: - yield f"{exe_name}-script.pyw" - else: - yield f"{exe_name}-script.py" - - -def _unique( - fn: Callable[..., Generator[Any, None, None]] -) -> Callable[..., Generator[Any, None, None]]: - @functools.wraps(fn) - def unique(*args: Any, **kw: Any) -> Generator[Any, None, None]: - seen: Set[Any] = set() - for item in fn(*args, **kw): - if item not in seen: - seen.add(item) - yield item - - return unique - - -@_unique -def uninstallation_paths(dist: BaseDistribution) -> Generator[str, None, None]: - """ - Yield all the uninstallation paths for dist based on RECORD-without-.py[co] - - Yield paths to all the files in RECORD. For each .py file in RECORD, add - the .pyc and .pyo in the same directory. - - UninstallPathSet.add() takes care of the __pycache__ .py[co]. - - If RECORD is not found, raises UninstallationError, - with possible information from the INSTALLER file. - - https://packaging.python.org/specifications/recording-installed-packages/ - """ - location = dist.location - assert location is not None, "not installed" - - entries = dist.iter_declared_entries() - if entries is None: - msg = "Cannot uninstall {dist}, RECORD file not found.".format(dist=dist) - installer = dist.installer - if not installer or installer == "pip": - dep = "{}=={}".format(dist.raw_name, dist.version) - msg += ( - " You might be able to recover from this via: " - "'pip install --force-reinstall --no-deps {}'.".format(dep) - ) - else: - msg += " Hint: The package was installed by {}.".format(installer) - raise UninstallationError(msg) - - for entry in entries: - path = os.path.join(location, entry) - yield path - if path.endswith(".py"): - dn, fn = os.path.split(path) - base = fn[:-3] - path = os.path.join(dn, base + ".pyc") - yield path - path = os.path.join(dn, base + ".pyo") - yield path - - -def compact(paths: Iterable[str]) -> Set[str]: - """Compact a path set to contain the minimal number of paths - necessary to contain all paths in the set. If /a/path/ and - /a/path/to/a/file.txt are both in the set, leave only the - shorter path.""" - - sep = os.path.sep - short_paths: Set[str] = set() - for path in sorted(paths, key=len): - should_skip = any( - path.startswith(shortpath.rstrip("*")) - and path[len(shortpath.rstrip("*").rstrip(sep))] == sep - for shortpath in short_paths - ) - if not should_skip: - short_paths.add(path) - return short_paths - - -def compress_for_rename(paths: Iterable[str]) -> Set[str]: - """Returns a set containing the paths that need to be renamed. - - This set may include directories when the original sequence of paths - included every file on disk. - """ - case_map = {os.path.normcase(p): p for p in paths} - remaining = set(case_map) - unchecked = sorted({os.path.split(p)[0] for p in case_map.values()}, key=len) - wildcards: Set[str] = set() - - def norm_join(*a: str) -> str: - return os.path.normcase(os.path.join(*a)) - - for root in unchecked: - if any(os.path.normcase(root).startswith(w) for w in wildcards): - # This directory has already been handled. - continue - - all_files: Set[str] = set() - all_subdirs: Set[str] = set() - for dirname, subdirs, files in os.walk(root): - all_subdirs.update(norm_join(root, dirname, d) for d in subdirs) - all_files.update(norm_join(root, dirname, f) for f in files) - # If all the files we found are in our remaining set of files to - # remove, then remove them from the latter set and add a wildcard - # for the directory. - if not (all_files - remaining): - remaining.difference_update(all_files) - wildcards.add(root + os.sep) - - return set(map(case_map.__getitem__, remaining)) | wildcards - - -def compress_for_output_listing(paths: Iterable[str]) -> Tuple[Set[str], Set[str]]: - """Returns a tuple of 2 sets of which paths to display to user - - The first set contains paths that would be deleted. Files of a package - are not added and the top-level directory of the package has a '*' added - at the end - to signify that all it's contents are removed. - - The second set contains files that would have been skipped in the above - folders. - """ - - will_remove = set(paths) - will_skip = set() - - # Determine folders and files - folders = set() - files = set() - for path in will_remove: - if path.endswith(".pyc"): - continue - if path.endswith("__init__.py") or ".dist-info" in path: - folders.add(os.path.dirname(path)) - files.add(path) - - # probably this one https://github.com/python/mypy/issues/390 - _normcased_files = set(map(os.path.normcase, files)) # type: ignore - - folders = compact(folders) - - # This walks the tree using os.walk to not miss extra folders - # that might get added. - for folder in folders: - for dirpath, _, dirfiles in os.walk(folder): - for fname in dirfiles: - if fname.endswith(".pyc"): - continue - - file_ = os.path.join(dirpath, fname) - if ( - os.path.isfile(file_) - and os.path.normcase(file_) not in _normcased_files - ): - # We are skipping this file. Add it to the set. - will_skip.add(file_) - - will_remove = files | {os.path.join(folder, "*") for folder in folders} - - return will_remove, will_skip - - -class StashedUninstallPathSet: - """A set of file rename operations to stash files while - tentatively uninstalling them.""" - - def __init__(self) -> None: - # Mapping from source file root to [Adjacent]TempDirectory - # for files under that directory. - self._save_dirs: Dict[str, TempDirectory] = {} - # (old path, new path) tuples for each move that may need - # to be undone. - self._moves: List[Tuple[str, str]] = [] - - def _get_directory_stash(self, path: str) -> str: - """Stashes a directory. - - Directories are stashed adjacent to their original location if - possible, or else moved/copied into the user's temp dir.""" - - try: - save_dir: TempDirectory = AdjacentTempDirectory(path) - except OSError: - save_dir = TempDirectory(kind="uninstall") - self._save_dirs[os.path.normcase(path)] = save_dir - - return save_dir.path - - def _get_file_stash(self, path: str) -> str: - """Stashes a file. - - If no root has been provided, one will be created for the directory - in the user's temp directory.""" - path = os.path.normcase(path) - head, old_head = os.path.dirname(path), None - save_dir = None - - while head != old_head: - try: - save_dir = self._save_dirs[head] - break - except KeyError: - pass - head, old_head = os.path.dirname(head), head - else: - # Did not find any suitable root - head = os.path.dirname(path) - save_dir = TempDirectory(kind="uninstall") - self._save_dirs[head] = save_dir - - relpath = os.path.relpath(path, head) - if relpath and relpath != os.path.curdir: - return os.path.join(save_dir.path, relpath) - return save_dir.path - - def stash(self, path: str) -> str: - """Stashes the directory or file and returns its new location. - Handle symlinks as files to avoid modifying the symlink targets. - """ - path_is_dir = os.path.isdir(path) and not os.path.islink(path) - if path_is_dir: - new_path = self._get_directory_stash(path) - else: - new_path = self._get_file_stash(path) - - self._moves.append((path, new_path)) - if path_is_dir and os.path.isdir(new_path): - # If we're moving a directory, we need to - # remove the destination first or else it will be - # moved to inside the existing directory. - # We just created new_path ourselves, so it will - # be removable. - os.rmdir(new_path) - renames(path, new_path) - return new_path - - def commit(self) -> None: - """Commits the uninstall by removing stashed files.""" - for save_dir in self._save_dirs.values(): - save_dir.cleanup() - self._moves = [] - self._save_dirs = {} - - def rollback(self) -> None: - """Undoes the uninstall by moving stashed files back.""" - for p in self._moves: - logger.info("Moving to %s\n from %s", *p) - - for new_path, path in self._moves: - try: - logger.debug("Replacing %s from %s", new_path, path) - if os.path.isfile(new_path) or os.path.islink(new_path): - os.unlink(new_path) - elif os.path.isdir(new_path): - rmtree(new_path) - renames(path, new_path) - except OSError as ex: - logger.error("Failed to restore %s", new_path) - logger.debug("Exception: %s", ex) - - self.commit() - - @property - def can_rollback(self) -> bool: - return bool(self._moves) - - -class UninstallPathSet: - """A set of file paths to be removed in the uninstallation of a - requirement.""" - - def __init__(self, dist: BaseDistribution) -> None: - self._paths: Set[str] = set() - self._refuse: Set[str] = set() - self._pth: Dict[str, UninstallPthEntries] = {} - self._dist = dist - self._moved_paths = StashedUninstallPathSet() - # Create local cache of normalize_path results. Creating an UninstallPathSet - # can result in hundreds/thousands of redundant calls to normalize_path with - # the same args, which hurts performance. - self._normalize_path_cached = functools.lru_cache()(normalize_path) - - def _permitted(self, path: str) -> bool: - """ - Return True if the given path is one we are permitted to - remove/modify, False otherwise. - - """ - # aka is_local, but caching normalized sys.prefix - if not running_under_virtualenv(): - return True - return path.startswith(self._normalize_path_cached(sys.prefix)) - - def add(self, path: str) -> None: - head, tail = os.path.split(path) - - # we normalize the head to resolve parent directory symlinks, but not - # the tail, since we only want to uninstall symlinks, not their targets - path = os.path.join(self._normalize_path_cached(head), os.path.normcase(tail)) - - if not os.path.exists(path): - return - if self._permitted(path): - self._paths.add(path) - else: - self._refuse.add(path) - - # __pycache__ files can show up after 'installed-files.txt' is created, - # due to imports - if os.path.splitext(path)[1] == ".py": - self.add(cache_from_source(path)) - - def add_pth(self, pth_file: str, entry: str) -> None: - pth_file = self._normalize_path_cached(pth_file) - if self._permitted(pth_file): - if pth_file not in self._pth: - self._pth[pth_file] = UninstallPthEntries(pth_file) - self._pth[pth_file].add(entry) - else: - self._refuse.add(pth_file) - - def remove(self, auto_confirm: bool = False, verbose: bool = False) -> None: - """Remove paths in ``self._paths`` with confirmation (unless - ``auto_confirm`` is True).""" - - if not self._paths: - logger.info( - "Can't uninstall '%s'. No files were found to uninstall.", - self._dist.raw_name, - ) - return - - dist_name_version = f"{self._dist.raw_name}-{self._dist.version}" - logger.info("Uninstalling %s:", dist_name_version) - - with indent_log(): - if auto_confirm or self._allowed_to_proceed(verbose): - moved = self._moved_paths - - for_rename = compress_for_rename(self._paths) - - for path in sorted(compact(for_rename)): - moved.stash(path) - logger.verbose("Removing file or directory %s", path) - - for pth in self._pth.values(): - pth.remove() - - logger.info("Successfully uninstalled %s", dist_name_version) - - def _allowed_to_proceed(self, verbose: bool) -> bool: - """Display which files would be deleted and prompt for confirmation""" - - def _display(msg: str, paths: Iterable[str]) -> None: - if not paths: - return - - logger.info(msg) - with indent_log(): - for path in sorted(compact(paths)): - logger.info(path) - - if not verbose: - will_remove, will_skip = compress_for_output_listing(self._paths) - else: - # In verbose mode, display all the files that are going to be - # deleted. - will_remove = set(self._paths) - will_skip = set() - - _display("Would remove:", will_remove) - _display("Would not remove (might be manually added):", will_skip) - _display("Would not remove (outside of prefix):", self._refuse) - if verbose: - _display("Will actually move:", compress_for_rename(self._paths)) - - return ask("Proceed (Y/n)? ", ("y", "n", "")) != "n" - - def rollback(self) -> None: - """Rollback the changes previously made by remove().""" - if not self._moved_paths.can_rollback: - logger.error( - "Can't roll back %s; was not uninstalled", - self._dist.raw_name, - ) - return - logger.info("Rolling back uninstall of %s", self._dist.raw_name) - self._moved_paths.rollback() - for pth in self._pth.values(): - pth.rollback() - - def commit(self) -> None: - """Remove temporary save dir: rollback will no longer be possible.""" - self._moved_paths.commit() - - @classmethod - def from_dist(cls, dist: BaseDistribution) -> "UninstallPathSet": - dist_location = dist.location - info_location = dist.info_location - if dist_location is None: - logger.info( - "Not uninstalling %s since it is not installed", - dist.canonical_name, - ) - return cls(dist) - - normalized_dist_location = normalize_path(dist_location) - if not dist.local: - logger.info( - "Not uninstalling %s at %s, outside environment %s", - dist.canonical_name, - normalized_dist_location, - sys.prefix, - ) - return cls(dist) - - if normalized_dist_location in { - p - for p in {sysconfig.get_path("stdlib"), sysconfig.get_path("platstdlib")} - if p - }: - logger.info( - "Not uninstalling %s at %s, as it is in the standard library.", - dist.canonical_name, - normalized_dist_location, - ) - return cls(dist) - - paths_to_remove = cls(dist) - develop_egg_link = egg_link_path_from_location(dist.raw_name) - - # Distribution is installed with metadata in a "flat" .egg-info - # directory. This means it is not a modern .dist-info installation, an - # egg, or legacy editable. - setuptools_flat_installation = ( - dist.installed_with_setuptools_egg_info - and info_location is not None - and os.path.exists(info_location) - # If dist is editable and the location points to a ``.egg-info``, - # we are in fact in the legacy editable case. - and not info_location.endswith(f"{dist.setuptools_filename}.egg-info") - ) - - # Uninstall cases order do matter as in the case of 2 installs of the - # same package, pip needs to uninstall the currently detected version - if setuptools_flat_installation: - if info_location is not None: - paths_to_remove.add(info_location) - installed_files = dist.iter_declared_entries() - if installed_files is not None: - for installed_file in installed_files: - paths_to_remove.add(os.path.join(dist_location, installed_file)) - # FIXME: need a test for this elif block - # occurs with --single-version-externally-managed/--record outside - # of pip - elif dist.is_file("top_level.txt"): - try: - namespace_packages = dist.read_text("namespace_packages.txt") - except FileNotFoundError: - namespaces = [] - else: - namespaces = namespace_packages.splitlines(keepends=False) - for top_level_pkg in [ - p - for p in dist.read_text("top_level.txt").splitlines() - if p and p not in namespaces - ]: - path = os.path.join(dist_location, top_level_pkg) - paths_to_remove.add(path) - paths_to_remove.add(f"{path}.py") - paths_to_remove.add(f"{path}.pyc") - paths_to_remove.add(f"{path}.pyo") - - elif dist.installed_by_distutils: - raise UninstallationError( - "Cannot uninstall {!r}. It is a distutils installed project " - "and thus we cannot accurately determine which files belong " - "to it which would lead to only a partial uninstall.".format( - dist.raw_name, - ) - ) - - elif dist.installed_as_egg: - # package installed by easy_install - # We cannot match on dist.egg_name because it can slightly vary - # i.e. setuptools-0.6c11-py2.6.egg vs setuptools-0.6rc11-py2.6.egg - paths_to_remove.add(dist_location) - easy_install_egg = os.path.split(dist_location)[1] - easy_install_pth = os.path.join( - os.path.dirname(dist_location), - "easy-install.pth", - ) - paths_to_remove.add_pth(easy_install_pth, "./" + easy_install_egg) - - elif dist.installed_with_dist_info: - for path in uninstallation_paths(dist): - paths_to_remove.add(path) - - elif develop_egg_link: - # PEP 660 modern editable is handled in the ``.dist-info`` case - # above, so this only covers the setuptools-style editable. - with open(develop_egg_link) as fh: - link_pointer = os.path.normcase(fh.readline().strip()) - normalized_link_pointer = paths_to_remove._normalize_path_cached( - link_pointer - ) - assert os.path.samefile( - normalized_link_pointer, normalized_dist_location - ), ( - f"Egg-link {develop_egg_link} (to {link_pointer}) does not match " - f"installed location of {dist.raw_name} (at {dist_location})" - ) - paths_to_remove.add(develop_egg_link) - easy_install_pth = os.path.join( - os.path.dirname(develop_egg_link), "easy-install.pth" - ) - paths_to_remove.add_pth(easy_install_pth, dist_location) - - else: - logger.debug( - "Not sure how to uninstall: %s - Check: %s", - dist, - dist_location, - ) - - if dist.in_usersite: - bin_dir = get_bin_user() - else: - bin_dir = get_bin_prefix() - - # find distutils scripts= scripts - try: - for script in dist.iter_distutils_script_names(): - paths_to_remove.add(os.path.join(bin_dir, script)) - if WINDOWS: - paths_to_remove.add(os.path.join(bin_dir, f"{script}.bat")) - except (FileNotFoundError, NotADirectoryError): - pass - - # find console_scripts and gui_scripts - def iter_scripts_to_remove( - dist: BaseDistribution, - bin_dir: str, - ) -> Generator[str, None, None]: - for entry_point in dist.iter_entry_points(): - if entry_point.group == "console_scripts": - yield from _script_names(bin_dir, entry_point.name, False) - elif entry_point.group == "gui_scripts": - yield from _script_names(bin_dir, entry_point.name, True) - - for s in iter_scripts_to_remove(dist, bin_dir): - paths_to_remove.add(s) - - return paths_to_remove - - -class UninstallPthEntries: - def __init__(self, pth_file: str) -> None: - self.file = pth_file - self.entries: Set[str] = set() - self._saved_lines: Optional[List[bytes]] = None - - def add(self, entry: str) -> None: - entry = os.path.normcase(entry) - # On Windows, os.path.normcase converts the entry to use - # backslashes. This is correct for entries that describe absolute - # paths outside of site-packages, but all the others use forward - # slashes. - # os.path.splitdrive is used instead of os.path.isabs because isabs - # treats non-absolute paths with drive letter markings like c:foo\bar - # as absolute paths. It also does not recognize UNC paths if they don't - # have more than "\\sever\share". Valid examples: "\\server\share\" or - # "\\server\share\folder". - if WINDOWS and not os.path.splitdrive(entry)[0]: - entry = entry.replace("\\", "/") - self.entries.add(entry) - - def remove(self) -> None: - logger.verbose("Removing pth entries from %s:", self.file) - - # If the file doesn't exist, log a warning and return - if not os.path.isfile(self.file): - logger.warning("Cannot remove entries from nonexistent file %s", self.file) - return - with open(self.file, "rb") as fh: - # windows uses '\r\n' with py3k, but uses '\n' with py2.x - lines = fh.readlines() - self._saved_lines = lines - if any(b"\r\n" in line for line in lines): - endline = "\r\n" - else: - endline = "\n" - # handle missing trailing newline - if lines and not lines[-1].endswith(endline.encode("utf-8")): - lines[-1] = lines[-1] + endline.encode("utf-8") - for entry in self.entries: - try: - logger.verbose("Removing entry: %s", entry) - lines.remove((entry + endline).encode("utf-8")) - except ValueError: - pass - with open(self.file, "wb") as fh: - fh.writelines(lines) - - def rollback(self) -> bool: - if self._saved_lines is None: - logger.error("Cannot roll back changes to %s, none were made", self.file) - return False - logger.debug("Rolling %s back to previous state", self.file) - with open(self.file, "wb") as fh: - fh.writelines(self._saved_lines) - return True diff --git a/venv/lib/python3.11/site-packages/pip/_internal/resolution/__init__.py b/venv/lib/python3.11/site-packages/pip/_internal/resolution/__init__.py deleted file mode 100644 index e69de29..0000000 diff --git a/venv/lib/python3.11/site-packages/pip/_internal/resolution/__pycache__/__init__.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/resolution/__pycache__/__init__.cpython-311.pyc deleted file mode 100644 index a9d944f..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/resolution/__pycache__/__init__.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/resolution/__pycache__/base.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/resolution/__pycache__/base.cpython-311.pyc deleted file mode 100644 index d333e27..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/resolution/__pycache__/base.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/resolution/base.py b/venv/lib/python3.11/site-packages/pip/_internal/resolution/base.py deleted file mode 100644 index 42dade1..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/resolution/base.py +++ /dev/null @@ -1,20 +0,0 @@ -from typing import Callable, List, Optional - -from pip._internal.req.req_install import InstallRequirement -from pip._internal.req.req_set import RequirementSet - -InstallRequirementProvider = Callable[ - [str, Optional[InstallRequirement]], InstallRequirement -] - - -class BaseResolver: - def resolve( - self, root_reqs: List[InstallRequirement], check_supported_wheels: bool - ) -> RequirementSet: - raise NotImplementedError() - - def get_installation_order( - self, req_set: RequirementSet - ) -> List[InstallRequirement]: - raise NotImplementedError() diff --git a/venv/lib/python3.11/site-packages/pip/_internal/resolution/legacy/__init__.py b/venv/lib/python3.11/site-packages/pip/_internal/resolution/legacy/__init__.py deleted file mode 100644 index e69de29..0000000 diff --git a/venv/lib/python3.11/site-packages/pip/_internal/resolution/legacy/__pycache__/__init__.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/resolution/legacy/__pycache__/__init__.cpython-311.pyc deleted file mode 100644 index c0ccd51..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/resolution/legacy/__pycache__/__init__.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/resolution/legacy/__pycache__/resolver.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/resolution/legacy/__pycache__/resolver.cpython-311.pyc deleted file mode 100644 index 449ddc5..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/resolution/legacy/__pycache__/resolver.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/resolution/legacy/resolver.py b/venv/lib/python3.11/site-packages/pip/_internal/resolution/legacy/resolver.py deleted file mode 100644 index b17b7e4..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/resolution/legacy/resolver.py +++ /dev/null @@ -1,600 +0,0 @@ -"""Dependency Resolution - -The dependency resolution in pip is performed as follows: - -for top-level requirements: - a. only one spec allowed per project, regardless of conflicts or not. - otherwise a "double requirement" exception is raised - b. they override sub-dependency requirements. -for sub-dependencies - a. "first found, wins" (where the order is breadth first) -""" - -# The following comment should be removed at some point in the future. -# mypy: strict-optional=False - -import logging -import sys -from collections import defaultdict -from itertools import chain -from typing import DefaultDict, Iterable, List, Optional, Set, Tuple - -from pip._vendor.packaging import specifiers -from pip._vendor.packaging.requirements import Requirement - -from pip._internal.cache import WheelCache -from pip._internal.exceptions import ( - BestVersionAlreadyInstalled, - DistributionNotFound, - HashError, - HashErrors, - InstallationError, - NoneMetadataError, - UnsupportedPythonVersion, -) -from pip._internal.index.package_finder import PackageFinder -from pip._internal.metadata import BaseDistribution -from pip._internal.models.link import Link -from pip._internal.models.wheel import Wheel -from pip._internal.operations.prepare import RequirementPreparer -from pip._internal.req.req_install import ( - InstallRequirement, - check_invalid_constraint_type, -) -from pip._internal.req.req_set import RequirementSet -from pip._internal.resolution.base import BaseResolver, InstallRequirementProvider -from pip._internal.utils import compatibility_tags -from pip._internal.utils.compatibility_tags import get_supported -from pip._internal.utils.direct_url_helpers import direct_url_from_link -from pip._internal.utils.logging import indent_log -from pip._internal.utils.misc import normalize_version_info -from pip._internal.utils.packaging import check_requires_python - -logger = logging.getLogger(__name__) - -DiscoveredDependencies = DefaultDict[str, List[InstallRequirement]] - - -def _check_dist_requires_python( - dist: BaseDistribution, - version_info: Tuple[int, int, int], - ignore_requires_python: bool = False, -) -> None: - """ - Check whether the given Python version is compatible with a distribution's - "Requires-Python" value. - - :param version_info: A 3-tuple of ints representing the Python - major-minor-micro version to check. - :param ignore_requires_python: Whether to ignore the "Requires-Python" - value if the given Python version isn't compatible. - - :raises UnsupportedPythonVersion: When the given Python version isn't - compatible. - """ - # This idiosyncratically converts the SpecifierSet to str and let - # check_requires_python then parse it again into SpecifierSet. But this - # is the legacy resolver so I'm just not going to bother refactoring. - try: - requires_python = str(dist.requires_python) - except FileNotFoundError as e: - raise NoneMetadataError(dist, str(e)) - try: - is_compatible = check_requires_python( - requires_python, - version_info=version_info, - ) - except specifiers.InvalidSpecifier as exc: - logger.warning( - "Package %r has an invalid Requires-Python: %s", dist.raw_name, exc - ) - return - - if is_compatible: - return - - version = ".".join(map(str, version_info)) - if ignore_requires_python: - logger.debug( - "Ignoring failed Requires-Python check for package %r: %s not in %r", - dist.raw_name, - version, - requires_python, - ) - return - - raise UnsupportedPythonVersion( - "Package {!r} requires a different Python: {} not in {!r}".format( - dist.raw_name, version, requires_python - ) - ) - - -class Resolver(BaseResolver): - """Resolves which packages need to be installed/uninstalled to perform \ - the requested operation without breaking the requirements of any package. - """ - - _allowed_strategies = {"eager", "only-if-needed", "to-satisfy-only"} - - def __init__( - self, - preparer: RequirementPreparer, - finder: PackageFinder, - wheel_cache: Optional[WheelCache], - make_install_req: InstallRequirementProvider, - use_user_site: bool, - ignore_dependencies: bool, - ignore_installed: bool, - ignore_requires_python: bool, - force_reinstall: bool, - upgrade_strategy: str, - py_version_info: Optional[Tuple[int, ...]] = None, - ) -> None: - super().__init__() - assert upgrade_strategy in self._allowed_strategies - - if py_version_info is None: - py_version_info = sys.version_info[:3] - else: - py_version_info = normalize_version_info(py_version_info) - - self._py_version_info = py_version_info - - self.preparer = preparer - self.finder = finder - self.wheel_cache = wheel_cache - - self.upgrade_strategy = upgrade_strategy - self.force_reinstall = force_reinstall - self.ignore_dependencies = ignore_dependencies - self.ignore_installed = ignore_installed - self.ignore_requires_python = ignore_requires_python - self.use_user_site = use_user_site - self._make_install_req = make_install_req - - self._discovered_dependencies: DiscoveredDependencies = defaultdict(list) - - def resolve( - self, root_reqs: List[InstallRequirement], check_supported_wheels: bool - ) -> RequirementSet: - """Resolve what operations need to be done - - As a side-effect of this method, the packages (and their dependencies) - are downloaded, unpacked and prepared for installation. This - preparation is done by ``pip.operations.prepare``. - - Once PyPI has static dependency metadata available, it would be - possible to move the preparation to become a step separated from - dependency resolution. - """ - requirement_set = RequirementSet(check_supported_wheels=check_supported_wheels) - for req in root_reqs: - if req.constraint: - check_invalid_constraint_type(req) - self._add_requirement_to_set(requirement_set, req) - - # Actually prepare the files, and collect any exceptions. Most hash - # exceptions cannot be checked ahead of time, because - # _populate_link() needs to be called before we can make decisions - # based on link type. - discovered_reqs: List[InstallRequirement] = [] - hash_errors = HashErrors() - for req in chain(requirement_set.all_requirements, discovered_reqs): - try: - discovered_reqs.extend(self._resolve_one(requirement_set, req)) - except HashError as exc: - exc.req = req - hash_errors.append(exc) - - if hash_errors: - raise hash_errors - - return requirement_set - - def _add_requirement_to_set( - self, - requirement_set: RequirementSet, - install_req: InstallRequirement, - parent_req_name: Optional[str] = None, - extras_requested: Optional[Iterable[str]] = None, - ) -> Tuple[List[InstallRequirement], Optional[InstallRequirement]]: - """Add install_req as a requirement to install. - - :param parent_req_name: The name of the requirement that needed this - added. The name is used because when multiple unnamed requirements - resolve to the same name, we could otherwise end up with dependency - links that point outside the Requirements set. parent_req must - already be added. Note that None implies that this is a user - supplied requirement, vs an inferred one. - :param extras_requested: an iterable of extras used to evaluate the - environment markers. - :return: Additional requirements to scan. That is either [] if - the requirement is not applicable, or [install_req] if the - requirement is applicable and has just been added. - """ - # If the markers do not match, ignore this requirement. - if not install_req.match_markers(extras_requested): - logger.info( - "Ignoring %s: markers '%s' don't match your environment", - install_req.name, - install_req.markers, - ) - return [], None - - # If the wheel is not supported, raise an error. - # Should check this after filtering out based on environment markers to - # allow specifying different wheels based on the environment/OS, in a - # single requirements file. - if install_req.link and install_req.link.is_wheel: - wheel = Wheel(install_req.link.filename) - tags = compatibility_tags.get_supported() - if requirement_set.check_supported_wheels and not wheel.supported(tags): - raise InstallationError( - "{} is not a supported wheel on this platform.".format( - wheel.filename - ) - ) - - # This next bit is really a sanity check. - assert ( - not install_req.user_supplied or parent_req_name is None - ), "a user supplied req shouldn't have a parent" - - # Unnamed requirements are scanned again and the requirement won't be - # added as a dependency until after scanning. - if not install_req.name: - requirement_set.add_unnamed_requirement(install_req) - return [install_req], None - - try: - existing_req: Optional[ - InstallRequirement - ] = requirement_set.get_requirement(install_req.name) - except KeyError: - existing_req = None - - has_conflicting_requirement = ( - parent_req_name is None - and existing_req - and not existing_req.constraint - and existing_req.extras == install_req.extras - and existing_req.req - and install_req.req - and existing_req.req.specifier != install_req.req.specifier - ) - if has_conflicting_requirement: - raise InstallationError( - "Double requirement given: {} (already in {}, name={!r})".format( - install_req, existing_req, install_req.name - ) - ) - - # When no existing requirement exists, add the requirement as a - # dependency and it will be scanned again after. - if not existing_req: - requirement_set.add_named_requirement(install_req) - # We'd want to rescan this requirement later - return [install_req], install_req - - # Assume there's no need to scan, and that we've already - # encountered this for scanning. - if install_req.constraint or not existing_req.constraint: - return [], existing_req - - does_not_satisfy_constraint = install_req.link and not ( - existing_req.link and install_req.link.path == existing_req.link.path - ) - if does_not_satisfy_constraint: - raise InstallationError( - "Could not satisfy constraints for '{}': " - "installation from path or url cannot be " - "constrained to a version".format(install_req.name) - ) - # If we're now installing a constraint, mark the existing - # object for real installation. - existing_req.constraint = False - # If we're now installing a user supplied requirement, - # mark the existing object as such. - if install_req.user_supplied: - existing_req.user_supplied = True - existing_req.extras = tuple( - sorted(set(existing_req.extras) | set(install_req.extras)) - ) - logger.debug( - "Setting %s extras to: %s", - existing_req, - existing_req.extras, - ) - # Return the existing requirement for addition to the parent and - # scanning again. - return [existing_req], existing_req - - def _is_upgrade_allowed(self, req: InstallRequirement) -> bool: - if self.upgrade_strategy == "to-satisfy-only": - return False - elif self.upgrade_strategy == "eager": - return True - else: - assert self.upgrade_strategy == "only-if-needed" - return req.user_supplied or req.constraint - - def _set_req_to_reinstall(self, req: InstallRequirement) -> None: - """ - Set a requirement to be installed. - """ - # Don't uninstall the conflict if doing a user install and the - # conflict is not a user install. - if not self.use_user_site or req.satisfied_by.in_usersite: - req.should_reinstall = True - req.satisfied_by = None - - def _check_skip_installed( - self, req_to_install: InstallRequirement - ) -> Optional[str]: - """Check if req_to_install should be skipped. - - This will check if the req is installed, and whether we should upgrade - or reinstall it, taking into account all the relevant user options. - - After calling this req_to_install will only have satisfied_by set to - None if the req_to_install is to be upgraded/reinstalled etc. Any - other value will be a dist recording the current thing installed that - satisfies the requirement. - - Note that for vcs urls and the like we can't assess skipping in this - routine - we simply identify that we need to pull the thing down, - then later on it is pulled down and introspected to assess upgrade/ - reinstalls etc. - - :return: A text reason for why it was skipped, or None. - """ - if self.ignore_installed: - return None - - req_to_install.check_if_exists(self.use_user_site) - if not req_to_install.satisfied_by: - return None - - if self.force_reinstall: - self._set_req_to_reinstall(req_to_install) - return None - - if not self._is_upgrade_allowed(req_to_install): - if self.upgrade_strategy == "only-if-needed": - return "already satisfied, skipping upgrade" - return "already satisfied" - - # Check for the possibility of an upgrade. For link-based - # requirements we have to pull the tree down and inspect to assess - # the version #, so it's handled way down. - if not req_to_install.link: - try: - self.finder.find_requirement(req_to_install, upgrade=True) - except BestVersionAlreadyInstalled: - # Then the best version is installed. - return "already up-to-date" - except DistributionNotFound: - # No distribution found, so we squash the error. It will - # be raised later when we re-try later to do the install. - # Why don't we just raise here? - pass - - self._set_req_to_reinstall(req_to_install) - return None - - def _find_requirement_link(self, req: InstallRequirement) -> Optional[Link]: - upgrade = self._is_upgrade_allowed(req) - best_candidate = self.finder.find_requirement(req, upgrade) - if not best_candidate: - return None - - # Log a warning per PEP 592 if necessary before returning. - link = best_candidate.link - if link.is_yanked: - reason = link.yanked_reason or "" - msg = ( - # Mark this as a unicode string to prevent - # "UnicodeEncodeError: 'ascii' codec can't encode character" - # in Python 2 when the reason contains non-ascii characters. - "The candidate selected for download or install is a " - "yanked version: {candidate}\n" - "Reason for being yanked: {reason}" - ).format(candidate=best_candidate, reason=reason) - logger.warning(msg) - - return link - - def _populate_link(self, req: InstallRequirement) -> None: - """Ensure that if a link can be found for this, that it is found. - - Note that req.link may still be None - if the requirement is already - installed and not needed to be upgraded based on the return value of - _is_upgrade_allowed(). - - If preparer.require_hashes is True, don't use the wheel cache, because - cached wheels, always built locally, have different hashes than the - files downloaded from the index server and thus throw false hash - mismatches. Furthermore, cached wheels at present have undeterministic - contents due to file modification times. - """ - if req.link is None: - req.link = self._find_requirement_link(req) - - if self.wheel_cache is None or self.preparer.require_hashes: - return - cache_entry = self.wheel_cache.get_cache_entry( - link=req.link, - package_name=req.name, - supported_tags=get_supported(), - ) - if cache_entry is not None: - logger.debug("Using cached wheel link: %s", cache_entry.link) - if req.link is req.original_link and cache_entry.persistent: - req.cached_wheel_source_link = req.link - if cache_entry.origin is not None: - req.download_info = cache_entry.origin - else: - # Legacy cache entry that does not have origin.json. - # download_info may miss the archive_info.hashes field. - req.download_info = direct_url_from_link( - req.link, link_is_in_wheel_cache=cache_entry.persistent - ) - req.link = cache_entry.link - - def _get_dist_for(self, req: InstallRequirement) -> BaseDistribution: - """Takes a InstallRequirement and returns a single AbstractDist \ - representing a prepared variant of the same. - """ - if req.editable: - return self.preparer.prepare_editable_requirement(req) - - # satisfied_by is only evaluated by calling _check_skip_installed, - # so it must be None here. - assert req.satisfied_by is None - skip_reason = self._check_skip_installed(req) - - if req.satisfied_by: - return self.preparer.prepare_installed_requirement(req, skip_reason) - - # We eagerly populate the link, since that's our "legacy" behavior. - self._populate_link(req) - dist = self.preparer.prepare_linked_requirement(req) - - # NOTE - # The following portion is for determining if a certain package is - # going to be re-installed/upgraded or not and reporting to the user. - # This should probably get cleaned up in a future refactor. - - # req.req is only avail after unpack for URL - # pkgs repeat check_if_exists to uninstall-on-upgrade - # (#14) - if not self.ignore_installed: - req.check_if_exists(self.use_user_site) - - if req.satisfied_by: - should_modify = ( - self.upgrade_strategy != "to-satisfy-only" - or self.force_reinstall - or self.ignore_installed - or req.link.scheme == "file" - ) - if should_modify: - self._set_req_to_reinstall(req) - else: - logger.info( - "Requirement already satisfied (use --upgrade to upgrade): %s", - req, - ) - return dist - - def _resolve_one( - self, - requirement_set: RequirementSet, - req_to_install: InstallRequirement, - ) -> List[InstallRequirement]: - """Prepare a single requirements file. - - :return: A list of additional InstallRequirements to also install. - """ - # Tell user what we are doing for this requirement: - # obtain (editable), skipping, processing (local url), collecting - # (remote url or package name) - if req_to_install.constraint or req_to_install.prepared: - return [] - - req_to_install.prepared = True - - # Parse and return dependencies - dist = self._get_dist_for(req_to_install) - # This will raise UnsupportedPythonVersion if the given Python - # version isn't compatible with the distribution's Requires-Python. - _check_dist_requires_python( - dist, - version_info=self._py_version_info, - ignore_requires_python=self.ignore_requires_python, - ) - - more_reqs: List[InstallRequirement] = [] - - def add_req(subreq: Requirement, extras_requested: Iterable[str]) -> None: - # This idiosyncratically converts the Requirement to str and let - # make_install_req then parse it again into Requirement. But this is - # the legacy resolver so I'm just not going to bother refactoring. - sub_install_req = self._make_install_req(str(subreq), req_to_install) - parent_req_name = req_to_install.name - to_scan_again, add_to_parent = self._add_requirement_to_set( - requirement_set, - sub_install_req, - parent_req_name=parent_req_name, - extras_requested=extras_requested, - ) - if parent_req_name and add_to_parent: - self._discovered_dependencies[parent_req_name].append(add_to_parent) - more_reqs.extend(to_scan_again) - - with indent_log(): - # We add req_to_install before its dependencies, so that we - # can refer to it when adding dependencies. - if not requirement_set.has_requirement(req_to_install.name): - # 'unnamed' requirements will get added here - # 'unnamed' requirements can only come from being directly - # provided by the user. - assert req_to_install.user_supplied - self._add_requirement_to_set( - requirement_set, req_to_install, parent_req_name=None - ) - - if not self.ignore_dependencies: - if req_to_install.extras: - logger.debug( - "Installing extra requirements: %r", - ",".join(req_to_install.extras), - ) - missing_requested = sorted( - set(req_to_install.extras) - set(dist.iter_provided_extras()) - ) - for missing in missing_requested: - logger.warning( - "%s %s does not provide the extra '%s'", - dist.raw_name, - dist.version, - missing, - ) - - available_requested = sorted( - set(dist.iter_provided_extras()) & set(req_to_install.extras) - ) - for subreq in dist.iter_dependencies(available_requested): - add_req(subreq, extras_requested=available_requested) - - return more_reqs - - def get_installation_order( - self, req_set: RequirementSet - ) -> List[InstallRequirement]: - """Create the installation order. - - The installation order is topological - requirements are installed - before the requiring thing. We break cycles at an arbitrary point, - and make no other guarantees. - """ - # The current implementation, which we may change at any point - # installs the user specified things in the order given, except when - # dependencies must come earlier to achieve topological order. - order = [] - ordered_reqs: Set[InstallRequirement] = set() - - def schedule(req: InstallRequirement) -> None: - if req.satisfied_by or req in ordered_reqs: - return - if req.constraint: - return - ordered_reqs.add(req) - for dep in self._discovered_dependencies[req.name]: - schedule(dep) - order.append(req) - - for install_req in req_set.requirements.values(): - schedule(install_req) - return order diff --git a/venv/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/__init__.py b/venv/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/__init__.py deleted file mode 100644 index e69de29..0000000 diff --git a/venv/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/__pycache__/__init__.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/__pycache__/__init__.cpython-311.pyc deleted file mode 100644 index de406e9..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/__pycache__/__init__.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/__pycache__/base.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/__pycache__/base.cpython-311.pyc deleted file mode 100644 index f01e525..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/__pycache__/base.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/__pycache__/candidates.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/__pycache__/candidates.cpython-311.pyc deleted file mode 100644 index 26d8127..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/__pycache__/candidates.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/__pycache__/factory.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/__pycache__/factory.cpython-311.pyc deleted file mode 100644 index cdc5b58..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/__pycache__/factory.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/__pycache__/found_candidates.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/__pycache__/found_candidates.cpython-311.pyc deleted file mode 100644 index 892bc41..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/__pycache__/found_candidates.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/__pycache__/provider.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/__pycache__/provider.cpython-311.pyc deleted file mode 100644 index 7ee0867..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/__pycache__/provider.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/__pycache__/reporter.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/__pycache__/reporter.cpython-311.pyc deleted file mode 100644 index 4a589b4..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/__pycache__/reporter.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/__pycache__/requirements.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/__pycache__/requirements.cpython-311.pyc deleted file mode 100644 index 5728bfd..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/__pycache__/requirements.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/__pycache__/resolver.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/__pycache__/resolver.cpython-311.pyc deleted file mode 100644 index a41c922..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/__pycache__/resolver.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/base.py b/venv/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/base.py deleted file mode 100644 index 9c0ef5c..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/base.py +++ /dev/null @@ -1,141 +0,0 @@ -from typing import FrozenSet, Iterable, Optional, Tuple, Union - -from pip._vendor.packaging.specifiers import SpecifierSet -from pip._vendor.packaging.utils import NormalizedName -from pip._vendor.packaging.version import LegacyVersion, Version - -from pip._internal.models.link import Link, links_equivalent -from pip._internal.req.req_install import InstallRequirement -from pip._internal.utils.hashes import Hashes - -CandidateLookup = Tuple[Optional["Candidate"], Optional[InstallRequirement]] -CandidateVersion = Union[LegacyVersion, Version] - - -def format_name(project: NormalizedName, extras: FrozenSet[NormalizedName]) -> str: - if not extras: - return project - extras_expr = ",".join(sorted(extras)) - return f"{project}[{extras_expr}]" - - -class Constraint: - def __init__( - self, specifier: SpecifierSet, hashes: Hashes, links: FrozenSet[Link] - ) -> None: - self.specifier = specifier - self.hashes = hashes - self.links = links - - @classmethod - def empty(cls) -> "Constraint": - return Constraint(SpecifierSet(), Hashes(), frozenset()) - - @classmethod - def from_ireq(cls, ireq: InstallRequirement) -> "Constraint": - links = frozenset([ireq.link]) if ireq.link else frozenset() - return Constraint(ireq.specifier, ireq.hashes(trust_internet=False), links) - - def __bool__(self) -> bool: - return bool(self.specifier) or bool(self.hashes) or bool(self.links) - - def __and__(self, other: InstallRequirement) -> "Constraint": - if not isinstance(other, InstallRequirement): - return NotImplemented - specifier = self.specifier & other.specifier - hashes = self.hashes & other.hashes(trust_internet=False) - links = self.links - if other.link: - links = links.union([other.link]) - return Constraint(specifier, hashes, links) - - def is_satisfied_by(self, candidate: "Candidate") -> bool: - # Reject if there are any mismatched URL constraints on this package. - if self.links and not all(_match_link(link, candidate) for link in self.links): - return False - # We can safely always allow prereleases here since PackageFinder - # already implements the prerelease logic, and would have filtered out - # prerelease candidates if the user does not expect them. - return self.specifier.contains(candidate.version, prereleases=True) - - -class Requirement: - @property - def project_name(self) -> NormalizedName: - """The "project name" of a requirement. - - This is different from ``name`` if this requirement contains extras, - in which case ``name`` would contain the ``[...]`` part, while this - refers to the name of the project. - """ - raise NotImplementedError("Subclass should override") - - @property - def name(self) -> str: - """The name identifying this requirement in the resolver. - - This is different from ``project_name`` if this requirement contains - extras, where ``project_name`` would not contain the ``[...]`` part. - """ - raise NotImplementedError("Subclass should override") - - def is_satisfied_by(self, candidate: "Candidate") -> bool: - return False - - def get_candidate_lookup(self) -> CandidateLookup: - raise NotImplementedError("Subclass should override") - - def format_for_error(self) -> str: - raise NotImplementedError("Subclass should override") - - -def _match_link(link: Link, candidate: "Candidate") -> bool: - if candidate.source_link: - return links_equivalent(link, candidate.source_link) - return False - - -class Candidate: - @property - def project_name(self) -> NormalizedName: - """The "project name" of the candidate. - - This is different from ``name`` if this candidate contains extras, - in which case ``name`` would contain the ``[...]`` part, while this - refers to the name of the project. - """ - raise NotImplementedError("Override in subclass") - - @property - def name(self) -> str: - """The name identifying this candidate in the resolver. - - This is different from ``project_name`` if this candidate contains - extras, where ``project_name`` would not contain the ``[...]`` part. - """ - raise NotImplementedError("Override in subclass") - - @property - def version(self) -> CandidateVersion: - raise NotImplementedError("Override in subclass") - - @property - def is_installed(self) -> bool: - raise NotImplementedError("Override in subclass") - - @property - def is_editable(self) -> bool: - raise NotImplementedError("Override in subclass") - - @property - def source_link(self) -> Optional[Link]: - raise NotImplementedError("Override in subclass") - - def iter_dependencies(self, with_requires: bool) -> Iterable[Optional[Requirement]]: - raise NotImplementedError("Override in subclass") - - def get_install_requirement(self) -> Optional[InstallRequirement]: - raise NotImplementedError("Override in subclass") - - def format_for_error(self) -> str: - raise NotImplementedError("Subclass should override") diff --git a/venv/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/candidates.py b/venv/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/candidates.py deleted file mode 100644 index 9754165..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/candidates.py +++ /dev/null @@ -1,607 +0,0 @@ -import logging -import sys -from typing import TYPE_CHECKING, Any, FrozenSet, Iterable, Optional, Tuple, Union, cast - -from pip._vendor.packaging.utils import NormalizedName, canonicalize_name -from pip._vendor.packaging.version import Version - -from pip._internal.exceptions import ( - HashError, - InstallationSubprocessError, - MetadataInconsistent, -) -from pip._internal.metadata import BaseDistribution -from pip._internal.models.link import Link, links_equivalent -from pip._internal.models.wheel import Wheel -from pip._internal.req.constructors import ( - install_req_from_editable, - install_req_from_line, -) -from pip._internal.req.req_install import InstallRequirement -from pip._internal.utils.direct_url_helpers import direct_url_from_link -from pip._internal.utils.misc import normalize_version_info - -from .base import Candidate, CandidateVersion, Requirement, format_name - -if TYPE_CHECKING: - from .factory import Factory - -logger = logging.getLogger(__name__) - -BaseCandidate = Union[ - "AlreadyInstalledCandidate", - "EditableCandidate", - "LinkCandidate", -] - -# Avoid conflicting with the PyPI package "Python". -REQUIRES_PYTHON_IDENTIFIER = cast(NormalizedName, "") - - -def as_base_candidate(candidate: Candidate) -> Optional[BaseCandidate]: - """The runtime version of BaseCandidate.""" - base_candidate_classes = ( - AlreadyInstalledCandidate, - EditableCandidate, - LinkCandidate, - ) - if isinstance(candidate, base_candidate_classes): - return candidate - return None - - -def make_install_req_from_link( - link: Link, template: InstallRequirement -) -> InstallRequirement: - assert not template.editable, "template is editable" - if template.req: - line = str(template.req) - else: - line = link.url - ireq = install_req_from_line( - line, - user_supplied=template.user_supplied, - comes_from=template.comes_from, - use_pep517=template.use_pep517, - isolated=template.isolated, - constraint=template.constraint, - global_options=template.global_options, - hash_options=template.hash_options, - config_settings=template.config_settings, - ) - ireq.original_link = template.original_link - ireq.link = link - ireq.extras = template.extras - return ireq - - -def make_install_req_from_editable( - link: Link, template: InstallRequirement -) -> InstallRequirement: - assert template.editable, "template not editable" - ireq = install_req_from_editable( - link.url, - user_supplied=template.user_supplied, - comes_from=template.comes_from, - use_pep517=template.use_pep517, - isolated=template.isolated, - constraint=template.constraint, - permit_editable_wheels=template.permit_editable_wheels, - global_options=template.global_options, - hash_options=template.hash_options, - config_settings=template.config_settings, - ) - ireq.extras = template.extras - return ireq - - -def _make_install_req_from_dist( - dist: BaseDistribution, template: InstallRequirement -) -> InstallRequirement: - if template.req: - line = str(template.req) - elif template.link: - line = f"{dist.canonical_name} @ {template.link.url}" - else: - line = f"{dist.canonical_name}=={dist.version}" - ireq = install_req_from_line( - line, - user_supplied=template.user_supplied, - comes_from=template.comes_from, - use_pep517=template.use_pep517, - isolated=template.isolated, - constraint=template.constraint, - global_options=template.global_options, - hash_options=template.hash_options, - config_settings=template.config_settings, - ) - ireq.satisfied_by = dist - return ireq - - -class _InstallRequirementBackedCandidate(Candidate): - """A candidate backed by an ``InstallRequirement``. - - This represents a package request with the target not being already - in the environment, and needs to be fetched and installed. The backing - ``InstallRequirement`` is responsible for most of the leg work; this - class exposes appropriate information to the resolver. - - :param link: The link passed to the ``InstallRequirement``. The backing - ``InstallRequirement`` will use this link to fetch the distribution. - :param source_link: The link this candidate "originates" from. This is - different from ``link`` when the link is found in the wheel cache. - ``link`` would point to the wheel cache, while this points to the - found remote link (e.g. from pypi.org). - """ - - dist: BaseDistribution - is_installed = False - - def __init__( - self, - link: Link, - source_link: Link, - ireq: InstallRequirement, - factory: "Factory", - name: Optional[NormalizedName] = None, - version: Optional[CandidateVersion] = None, - ) -> None: - self._link = link - self._source_link = source_link - self._factory = factory - self._ireq = ireq - self._name = name - self._version = version - self.dist = self._prepare() - - def __str__(self) -> str: - return f"{self.name} {self.version}" - - def __repr__(self) -> str: - return "{class_name}({link!r})".format( - class_name=self.__class__.__name__, - link=str(self._link), - ) - - def __hash__(self) -> int: - return hash((self.__class__, self._link)) - - def __eq__(self, other: Any) -> bool: - if isinstance(other, self.__class__): - return links_equivalent(self._link, other._link) - return False - - @property - def source_link(self) -> Optional[Link]: - return self._source_link - - @property - def project_name(self) -> NormalizedName: - """The normalised name of the project the candidate refers to""" - if self._name is None: - self._name = self.dist.canonical_name - return self._name - - @property - def name(self) -> str: - return self.project_name - - @property - def version(self) -> CandidateVersion: - if self._version is None: - self._version = self.dist.version - return self._version - - def format_for_error(self) -> str: - return "{} {} (from {})".format( - self.name, - self.version, - self._link.file_path if self._link.is_file else self._link, - ) - - def _prepare_distribution(self) -> BaseDistribution: - raise NotImplementedError("Override in subclass") - - def _check_metadata_consistency(self, dist: BaseDistribution) -> None: - """Check for consistency of project name and version of dist.""" - if self._name is not None and self._name != dist.canonical_name: - raise MetadataInconsistent( - self._ireq, - "name", - self._name, - dist.canonical_name, - ) - if self._version is not None and self._version != dist.version: - raise MetadataInconsistent( - self._ireq, - "version", - str(self._version), - str(dist.version), - ) - - def _prepare(self) -> BaseDistribution: - try: - dist = self._prepare_distribution() - except HashError as e: - # Provide HashError the underlying ireq that caused it. This - # provides context for the resulting error message to show the - # offending line to the user. - e.req = self._ireq - raise - except InstallationSubprocessError as exc: - # The output has been presented already, so don't duplicate it. - exc.context = "See above for output." - raise - - self._check_metadata_consistency(dist) - return dist - - def iter_dependencies(self, with_requires: bool) -> Iterable[Optional[Requirement]]: - requires = self.dist.iter_dependencies() if with_requires else () - for r in requires: - yield from self._factory.make_requirements_from_spec(str(r), self._ireq) - yield self._factory.make_requires_python_requirement(self.dist.requires_python) - - def get_install_requirement(self) -> Optional[InstallRequirement]: - return self._ireq - - -class LinkCandidate(_InstallRequirementBackedCandidate): - is_editable = False - - def __init__( - self, - link: Link, - template: InstallRequirement, - factory: "Factory", - name: Optional[NormalizedName] = None, - version: Optional[CandidateVersion] = None, - ) -> None: - source_link = link - cache_entry = factory.get_wheel_cache_entry(source_link, name) - if cache_entry is not None: - logger.debug("Using cached wheel link: %s", cache_entry.link) - link = cache_entry.link - ireq = make_install_req_from_link(link, template) - assert ireq.link == link - if ireq.link.is_wheel and not ireq.link.is_file: - wheel = Wheel(ireq.link.filename) - wheel_name = canonicalize_name(wheel.name) - assert name == wheel_name, f"{name!r} != {wheel_name!r} for wheel" - # Version may not be present for PEP 508 direct URLs - if version is not None: - wheel_version = Version(wheel.version) - assert version == wheel_version, "{!r} != {!r} for wheel {}".format( - version, wheel_version, name - ) - - if cache_entry is not None: - assert ireq.link.is_wheel - assert ireq.link.is_file - if cache_entry.persistent and template.link is template.original_link: - ireq.cached_wheel_source_link = source_link - if cache_entry.origin is not None: - ireq.download_info = cache_entry.origin - else: - # Legacy cache entry that does not have origin.json. - # download_info may miss the archive_info.hashes field. - ireq.download_info = direct_url_from_link( - source_link, link_is_in_wheel_cache=cache_entry.persistent - ) - - super().__init__( - link=link, - source_link=source_link, - ireq=ireq, - factory=factory, - name=name, - version=version, - ) - - def _prepare_distribution(self) -> BaseDistribution: - preparer = self._factory.preparer - return preparer.prepare_linked_requirement(self._ireq, parallel_builds=True) - - -class EditableCandidate(_InstallRequirementBackedCandidate): - is_editable = True - - def __init__( - self, - link: Link, - template: InstallRequirement, - factory: "Factory", - name: Optional[NormalizedName] = None, - version: Optional[CandidateVersion] = None, - ) -> None: - super().__init__( - link=link, - source_link=link, - ireq=make_install_req_from_editable(link, template), - factory=factory, - name=name, - version=version, - ) - - def _prepare_distribution(self) -> BaseDistribution: - return self._factory.preparer.prepare_editable_requirement(self._ireq) - - -class AlreadyInstalledCandidate(Candidate): - is_installed = True - source_link = None - - def __init__( - self, - dist: BaseDistribution, - template: InstallRequirement, - factory: "Factory", - ) -> None: - self.dist = dist - self._ireq = _make_install_req_from_dist(dist, template) - self._factory = factory - self._version = None - - # This is just logging some messages, so we can do it eagerly. - # The returned dist would be exactly the same as self.dist because we - # set satisfied_by in _make_install_req_from_dist. - # TODO: Supply reason based on force_reinstall and upgrade_strategy. - skip_reason = "already satisfied" - factory.preparer.prepare_installed_requirement(self._ireq, skip_reason) - - def __str__(self) -> str: - return str(self.dist) - - def __repr__(self) -> str: - return "{class_name}({distribution!r})".format( - class_name=self.__class__.__name__, - distribution=self.dist, - ) - - def __hash__(self) -> int: - return hash((self.__class__, self.name, self.version)) - - def __eq__(self, other: Any) -> bool: - if isinstance(other, self.__class__): - return self.name == other.name and self.version == other.version - return False - - @property - def project_name(self) -> NormalizedName: - return self.dist.canonical_name - - @property - def name(self) -> str: - return self.project_name - - @property - def version(self) -> CandidateVersion: - if self._version is None: - self._version = self.dist.version - return self._version - - @property - def is_editable(self) -> bool: - return self.dist.editable - - def format_for_error(self) -> str: - return f"{self.name} {self.version} (Installed)" - - def iter_dependencies(self, with_requires: bool) -> Iterable[Optional[Requirement]]: - if not with_requires: - return - for r in self.dist.iter_dependencies(): - yield from self._factory.make_requirements_from_spec(str(r), self._ireq) - - def get_install_requirement(self) -> Optional[InstallRequirement]: - return None - - -class ExtrasCandidate(Candidate): - """A candidate that has 'extras', indicating additional dependencies. - - Requirements can be for a project with dependencies, something like - foo[extra]. The extras don't affect the project/version being installed - directly, but indicate that we need additional dependencies. We model that - by having an artificial ExtrasCandidate that wraps the "base" candidate. - - The ExtrasCandidate differs from the base in the following ways: - - 1. It has a unique name, of the form foo[extra]. This causes the resolver - to treat it as a separate node in the dependency graph. - 2. When we're getting the candidate's dependencies, - a) We specify that we want the extra dependencies as well. - b) We add a dependency on the base candidate. - See below for why this is needed. - 3. We return None for the underlying InstallRequirement, as the base - candidate will provide it, and we don't want to end up with duplicates. - - The dependency on the base candidate is needed so that the resolver can't - decide that it should recommend foo[extra1] version 1.0 and foo[extra2] - version 2.0. Having those candidates depend on foo=1.0 and foo=2.0 - respectively forces the resolver to recognise that this is a conflict. - """ - - def __init__( - self, - base: BaseCandidate, - extras: FrozenSet[str], - *, - comes_from: Optional[InstallRequirement] = None, - ) -> None: - """ - :param comes_from: the InstallRequirement that led to this candidate if it - differs from the base's InstallRequirement. This will often be the - case in the sense that this candidate's requirement has the extras - while the base's does not. Unlike the InstallRequirement backed - candidates, this requirement is used solely for reporting purposes, - it does not do any leg work. - """ - self.base = base - self.extras = frozenset(canonicalize_name(e) for e in extras) - # If any extras are requested in their non-normalized forms, keep track - # of their raw values. This is needed when we look up dependencies - # since PEP 685 has not been implemented for marker-matching, and using - # the non-normalized extra for lookup ensures the user can select a - # non-normalized extra in a package with its non-normalized form. - # TODO: Remove this attribute when packaging is upgraded to support the - # marker comparison logic specified in PEP 685. - self._unnormalized_extras = extras.difference(self.extras) - self._comes_from = comes_from if comes_from is not None else self.base._ireq - - def __str__(self) -> str: - name, rest = str(self.base).split(" ", 1) - return "{}[{}] {}".format(name, ",".join(self.extras), rest) - - def __repr__(self) -> str: - return "{class_name}(base={base!r}, extras={extras!r})".format( - class_name=self.__class__.__name__, - base=self.base, - extras=self.extras, - ) - - def __hash__(self) -> int: - return hash((self.base, self.extras)) - - def __eq__(self, other: Any) -> bool: - if isinstance(other, self.__class__): - return self.base == other.base and self.extras == other.extras - return False - - @property - def project_name(self) -> NormalizedName: - return self.base.project_name - - @property - def name(self) -> str: - """The normalised name of the project the candidate refers to""" - return format_name(self.base.project_name, self.extras) - - @property - def version(self) -> CandidateVersion: - return self.base.version - - def format_for_error(self) -> str: - return "{} [{}]".format( - self.base.format_for_error(), ", ".join(sorted(self.extras)) - ) - - @property - def is_installed(self) -> bool: - return self.base.is_installed - - @property - def is_editable(self) -> bool: - return self.base.is_editable - - @property - def source_link(self) -> Optional[Link]: - return self.base.source_link - - def _warn_invalid_extras( - self, - requested: FrozenSet[str], - valid: FrozenSet[str], - ) -> None: - """Emit warnings for invalid extras being requested. - - This emits a warning for each requested extra that is not in the - candidate's ``Provides-Extra`` list. - """ - invalid_extras_to_warn = frozenset( - extra - for extra in requested - if extra not in valid - # If an extra is requested in an unnormalized form, skip warning - # about the normalized form being missing. - and extra in self.extras - ) - if not invalid_extras_to_warn: - return - for extra in sorted(invalid_extras_to_warn): - logger.warning( - "%s %s does not provide the extra '%s'", - self.base.name, - self.version, - extra, - ) - - def _calculate_valid_requested_extras(self) -> FrozenSet[str]: - """Get a list of valid extras requested by this candidate. - - The user (or upstream dependant) may have specified extras that the - candidate doesn't support. Any unsupported extras are dropped, and each - cause a warning to be logged here. - """ - requested_extras = self.extras.union(self._unnormalized_extras) - valid_extras = frozenset( - extra - for extra in requested_extras - if self.base.dist.is_extra_provided(extra) - ) - self._warn_invalid_extras(requested_extras, valid_extras) - return valid_extras - - def iter_dependencies(self, with_requires: bool) -> Iterable[Optional[Requirement]]: - factory = self.base._factory - - # Add a dependency on the exact base - # (See note 2b in the class docstring) - yield factory.make_requirement_from_candidate(self.base) - if not with_requires: - return - - valid_extras = self._calculate_valid_requested_extras() - for r in self.base.dist.iter_dependencies(valid_extras): - yield from factory.make_requirements_from_spec( - str(r), - self._comes_from, - valid_extras, - ) - - def get_install_requirement(self) -> Optional[InstallRequirement]: - # We don't return anything here, because we always - # depend on the base candidate, and we'll get the - # install requirement from that. - return None - - -class RequiresPythonCandidate(Candidate): - is_installed = False - source_link = None - - def __init__(self, py_version_info: Optional[Tuple[int, ...]]) -> None: - if py_version_info is not None: - version_info = normalize_version_info(py_version_info) - else: - version_info = sys.version_info[:3] - self._version = Version(".".join(str(c) for c in version_info)) - - # We don't need to implement __eq__() and __ne__() since there is always - # only one RequiresPythonCandidate in a resolution, i.e. the host Python. - # The built-in object.__eq__() and object.__ne__() do exactly what we want. - - def __str__(self) -> str: - return f"Python {self._version}" - - @property - def project_name(self) -> NormalizedName: - return REQUIRES_PYTHON_IDENTIFIER - - @property - def name(self) -> str: - return REQUIRES_PYTHON_IDENTIFIER - - @property - def version(self) -> CandidateVersion: - return self._version - - def format_for_error(self) -> str: - return f"Python {self.version}" - - def iter_dependencies(self, with_requires: bool) -> Iterable[Optional[Requirement]]: - return () - - def get_install_requirement(self) -> Optional[InstallRequirement]: - return None diff --git a/venv/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/factory.py b/venv/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/factory.py deleted file mode 100644 index 38c1994..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/factory.py +++ /dev/null @@ -1,791 +0,0 @@ -import contextlib -import functools -import logging -from typing import ( - TYPE_CHECKING, - Dict, - FrozenSet, - Iterable, - Iterator, - List, - Mapping, - NamedTuple, - Optional, - Sequence, - Set, - Tuple, - TypeVar, - cast, -) - -from pip._vendor.packaging.requirements import InvalidRequirement -from pip._vendor.packaging.specifiers import SpecifierSet -from pip._vendor.packaging.utils import NormalizedName, canonicalize_name -from pip._vendor.resolvelib import ResolutionImpossible - -from pip._internal.cache import CacheEntry, WheelCache -from pip._internal.exceptions import ( - DistributionNotFound, - InstallationError, - MetadataInconsistent, - UnsupportedPythonVersion, - UnsupportedWheel, -) -from pip._internal.index.package_finder import PackageFinder -from pip._internal.metadata import BaseDistribution, get_default_environment -from pip._internal.models.link import Link -from pip._internal.models.wheel import Wheel -from pip._internal.operations.prepare import RequirementPreparer -from pip._internal.req.constructors import install_req_from_link_and_ireq -from pip._internal.req.req_install import ( - InstallRequirement, - check_invalid_constraint_type, -) -from pip._internal.resolution.base import InstallRequirementProvider -from pip._internal.utils.compatibility_tags import get_supported -from pip._internal.utils.hashes import Hashes -from pip._internal.utils.packaging import get_requirement -from pip._internal.utils.virtualenv import running_under_virtualenv - -from .base import Candidate, CandidateVersion, Constraint, Requirement -from .candidates import ( - AlreadyInstalledCandidate, - BaseCandidate, - EditableCandidate, - ExtrasCandidate, - LinkCandidate, - RequiresPythonCandidate, - as_base_candidate, -) -from .found_candidates import FoundCandidates, IndexCandidateInfo -from .requirements import ( - ExplicitRequirement, - RequiresPythonRequirement, - SpecifierRequirement, - SpecifierWithoutExtrasRequirement, - UnsatisfiableRequirement, -) - -if TYPE_CHECKING: - from typing import Protocol - - class ConflictCause(Protocol): - requirement: RequiresPythonRequirement - parent: Candidate - - -logger = logging.getLogger(__name__) - -C = TypeVar("C") -Cache = Dict[Link, C] - - -class CollectedRootRequirements(NamedTuple): - requirements: List[Requirement] - constraints: Dict[str, Constraint] - user_requested: Dict[str, int] - - -class Factory: - def __init__( - self, - finder: PackageFinder, - preparer: RequirementPreparer, - make_install_req: InstallRequirementProvider, - wheel_cache: Optional[WheelCache], - use_user_site: bool, - force_reinstall: bool, - ignore_installed: bool, - ignore_requires_python: bool, - py_version_info: Optional[Tuple[int, ...]] = None, - ) -> None: - self._finder = finder - self.preparer = preparer - self._wheel_cache = wheel_cache - self._python_candidate = RequiresPythonCandidate(py_version_info) - self._make_install_req_from_spec = make_install_req - self._use_user_site = use_user_site - self._force_reinstall = force_reinstall - self._ignore_requires_python = ignore_requires_python - - self._build_failures: Cache[InstallationError] = {} - self._link_candidate_cache: Cache[LinkCandidate] = {} - self._editable_candidate_cache: Cache[EditableCandidate] = {} - self._installed_candidate_cache: Dict[str, AlreadyInstalledCandidate] = {} - self._extras_candidate_cache: Dict[ - Tuple[int, FrozenSet[NormalizedName]], ExtrasCandidate - ] = {} - - if not ignore_installed: - env = get_default_environment() - self._installed_dists = { - dist.canonical_name: dist - for dist in env.iter_installed_distributions(local_only=False) - } - else: - self._installed_dists = {} - - @property - def force_reinstall(self) -> bool: - return self._force_reinstall - - def _fail_if_link_is_unsupported_wheel(self, link: Link) -> None: - if not link.is_wheel: - return - wheel = Wheel(link.filename) - if wheel.supported(self._finder.target_python.get_unsorted_tags()): - return - msg = f"{link.filename} is not a supported wheel on this platform." - raise UnsupportedWheel(msg) - - def _make_extras_candidate( - self, - base: BaseCandidate, - extras: FrozenSet[str], - *, - comes_from: Optional[InstallRequirement] = None, - ) -> ExtrasCandidate: - cache_key = (id(base), frozenset(canonicalize_name(e) for e in extras)) - try: - candidate = self._extras_candidate_cache[cache_key] - except KeyError: - candidate = ExtrasCandidate(base, extras, comes_from=comes_from) - self._extras_candidate_cache[cache_key] = candidate - return candidate - - def _make_candidate_from_dist( - self, - dist: BaseDistribution, - extras: FrozenSet[str], - template: InstallRequirement, - ) -> Candidate: - try: - base = self._installed_candidate_cache[dist.canonical_name] - except KeyError: - base = AlreadyInstalledCandidate(dist, template, factory=self) - self._installed_candidate_cache[dist.canonical_name] = base - if not extras: - return base - return self._make_extras_candidate(base, extras, comes_from=template) - - def _make_candidate_from_link( - self, - link: Link, - extras: FrozenSet[str], - template: InstallRequirement, - name: Optional[NormalizedName], - version: Optional[CandidateVersion], - ) -> Optional[Candidate]: - # TODO: Check already installed candidate, and use it if the link and - # editable flag match. - - if link in self._build_failures: - # We already tried this candidate before, and it does not build. - # Don't bother trying again. - return None - - if template.editable: - if link not in self._editable_candidate_cache: - try: - self._editable_candidate_cache[link] = EditableCandidate( - link, - template, - factory=self, - name=name, - version=version, - ) - except MetadataInconsistent as e: - logger.info( - "Discarding [blue underline]%s[/]: [yellow]%s[reset]", - link, - e, - extra={"markup": True}, - ) - self._build_failures[link] = e - return None - - base: BaseCandidate = self._editable_candidate_cache[link] - else: - if link not in self._link_candidate_cache: - try: - self._link_candidate_cache[link] = LinkCandidate( - link, - template, - factory=self, - name=name, - version=version, - ) - except MetadataInconsistent as e: - logger.info( - "Discarding [blue underline]%s[/]: [yellow]%s[reset]", - link, - e, - extra={"markup": True}, - ) - self._build_failures[link] = e - return None - base = self._link_candidate_cache[link] - - if not extras: - return base - return self._make_extras_candidate(base, extras, comes_from=template) - - def _iter_found_candidates( - self, - ireqs: Sequence[InstallRequirement], - specifier: SpecifierSet, - hashes: Hashes, - prefers_installed: bool, - incompatible_ids: Set[int], - ) -> Iterable[Candidate]: - if not ireqs: - return () - - # The InstallRequirement implementation requires us to give it a - # "template". Here we just choose the first requirement to represent - # all of them. - # Hopefully the Project model can correct this mismatch in the future. - template = ireqs[0] - assert template.req, "Candidates found on index must be PEP 508" - name = canonicalize_name(template.req.name) - - extras: FrozenSet[str] = frozenset() - for ireq in ireqs: - assert ireq.req, "Candidates found on index must be PEP 508" - specifier &= ireq.req.specifier - hashes &= ireq.hashes(trust_internet=False) - extras |= frozenset(ireq.extras) - - def _get_installed_candidate() -> Optional[Candidate]: - """Get the candidate for the currently-installed version.""" - # If --force-reinstall is set, we want the version from the index - # instead, so we "pretend" there is nothing installed. - if self._force_reinstall: - return None - try: - installed_dist = self._installed_dists[name] - except KeyError: - return None - # Don't use the installed distribution if its version does not fit - # the current dependency graph. - if not specifier.contains(installed_dist.version, prereleases=True): - return None - candidate = self._make_candidate_from_dist( - dist=installed_dist, - extras=extras, - template=template, - ) - # The candidate is a known incompatibility. Don't use it. - if id(candidate) in incompatible_ids: - return None - return candidate - - def iter_index_candidate_infos() -> Iterator[IndexCandidateInfo]: - result = self._finder.find_best_candidate( - project_name=name, - specifier=specifier, - hashes=hashes, - ) - icans = list(result.iter_applicable()) - - # PEP 592: Yanked releases are ignored unless the specifier - # explicitly pins a version (via '==' or '===') that can be - # solely satisfied by a yanked release. - all_yanked = all(ican.link.is_yanked for ican in icans) - - def is_pinned(specifier: SpecifierSet) -> bool: - for sp in specifier: - if sp.operator == "===": - return True - if sp.operator != "==": - continue - if sp.version.endswith(".*"): - continue - return True - return False - - pinned = is_pinned(specifier) - - # PackageFinder returns earlier versions first, so we reverse. - for ican in reversed(icans): - if not (all_yanked and pinned) and ican.link.is_yanked: - continue - func = functools.partial( - self._make_candidate_from_link, - link=ican.link, - extras=extras, - template=template, - name=name, - version=ican.version, - ) - yield ican.version, func - - return FoundCandidates( - iter_index_candidate_infos, - _get_installed_candidate(), - prefers_installed, - incompatible_ids, - ) - - def _iter_explicit_candidates_from_base( - self, - base_requirements: Iterable[Requirement], - extras: FrozenSet[str], - ) -> Iterator[Candidate]: - """Produce explicit candidates from the base given an extra-ed package. - - :param base_requirements: Requirements known to the resolver. The - requirements are guaranteed to not have extras. - :param extras: The extras to inject into the explicit requirements' - candidates. - """ - for req in base_requirements: - lookup_cand, _ = req.get_candidate_lookup() - if lookup_cand is None: # Not explicit. - continue - # We've stripped extras from the identifier, and should always - # get a BaseCandidate here, unless there's a bug elsewhere. - base_cand = as_base_candidate(lookup_cand) - assert base_cand is not None, "no extras here" - yield self._make_extras_candidate(base_cand, extras) - - def _iter_candidates_from_constraints( - self, - identifier: str, - constraint: Constraint, - template: InstallRequirement, - ) -> Iterator[Candidate]: - """Produce explicit candidates from constraints. - - This creates "fake" InstallRequirement objects that are basically clones - of what "should" be the template, but with original_link set to link. - """ - for link in constraint.links: - self._fail_if_link_is_unsupported_wheel(link) - candidate = self._make_candidate_from_link( - link, - extras=frozenset(), - template=install_req_from_link_and_ireq(link, template), - name=canonicalize_name(identifier), - version=None, - ) - if candidate: - yield candidate - - def find_candidates( - self, - identifier: str, - requirements: Mapping[str, Iterable[Requirement]], - incompatibilities: Mapping[str, Iterator[Candidate]], - constraint: Constraint, - prefers_installed: bool, - ) -> Iterable[Candidate]: - # Collect basic lookup information from the requirements. - explicit_candidates: Set[Candidate] = set() - ireqs: List[InstallRequirement] = [] - for req in requirements[identifier]: - cand, ireq = req.get_candidate_lookup() - if cand is not None: - explicit_candidates.add(cand) - if ireq is not None: - ireqs.append(ireq) - - # If the current identifier contains extras, add requires and explicit - # candidates from entries from extra-less identifier. - with contextlib.suppress(InvalidRequirement): - parsed_requirement = get_requirement(identifier) - if parsed_requirement.name != identifier: - explicit_candidates.update( - self._iter_explicit_candidates_from_base( - requirements.get(parsed_requirement.name, ()), - frozenset(parsed_requirement.extras), - ), - ) - for req in requirements.get(parsed_requirement.name, []): - _, ireq = req.get_candidate_lookup() - if ireq is not None: - ireqs.append(ireq) - - # Add explicit candidates from constraints. We only do this if there are - # known ireqs, which represent requirements not already explicit. If - # there are no ireqs, we're constraining already-explicit requirements, - # which is handled later when we return the explicit candidates. - if ireqs: - try: - explicit_candidates.update( - self._iter_candidates_from_constraints( - identifier, - constraint, - template=ireqs[0], - ), - ) - except UnsupportedWheel: - # If we're constrained to install a wheel incompatible with the - # target architecture, no candidates will ever be valid. - return () - - # Since we cache all the candidates, incompatibility identification - # can be made quicker by comparing only the id() values. - incompat_ids = {id(c) for c in incompatibilities.get(identifier, ())} - - # If none of the requirements want an explicit candidate, we can ask - # the finder for candidates. - if not explicit_candidates: - return self._iter_found_candidates( - ireqs, - constraint.specifier, - constraint.hashes, - prefers_installed, - incompat_ids, - ) - - return ( - c - for c in explicit_candidates - if id(c) not in incompat_ids - and constraint.is_satisfied_by(c) - and all(req.is_satisfied_by(c) for req in requirements[identifier]) - ) - - def _make_requirements_from_install_req( - self, ireq: InstallRequirement, requested_extras: Iterable[str] - ) -> Iterator[Requirement]: - """ - Returns requirement objects associated with the given InstallRequirement. In - most cases this will be a single object but the following special cases exist: - - the InstallRequirement has markers that do not apply -> result is empty - - the InstallRequirement has both a constraint and extras -> result is split - in two requirement objects: one with the constraint and one with the - extra. This allows centralized constraint handling for the base, - resulting in fewer candidate rejections. - """ - if not ireq.match_markers(requested_extras): - logger.info( - "Ignoring %s: markers '%s' don't match your environment", - ireq.name, - ireq.markers, - ) - elif not ireq.link: - if ireq.extras and ireq.req is not None and ireq.req.specifier: - yield SpecifierWithoutExtrasRequirement(ireq) - yield SpecifierRequirement(ireq) - else: - self._fail_if_link_is_unsupported_wheel(ireq.link) - cand = self._make_candidate_from_link( - ireq.link, - extras=frozenset(ireq.extras), - template=ireq, - name=canonicalize_name(ireq.name) if ireq.name else None, - version=None, - ) - if cand is None: - # There's no way we can satisfy a URL requirement if the underlying - # candidate fails to build. An unnamed URL must be user-supplied, so - # we fail eagerly. If the URL is named, an unsatisfiable requirement - # can make the resolver do the right thing, either backtrack (and - # maybe find some other requirement that's buildable) or raise a - # ResolutionImpossible eventually. - if not ireq.name: - raise self._build_failures[ireq.link] - yield UnsatisfiableRequirement(canonicalize_name(ireq.name)) - else: - yield self.make_requirement_from_candidate(cand) - - def collect_root_requirements( - self, root_ireqs: List[InstallRequirement] - ) -> CollectedRootRequirements: - collected = CollectedRootRequirements([], {}, {}) - for i, ireq in enumerate(root_ireqs): - if ireq.constraint: - # Ensure we only accept valid constraints - problem = check_invalid_constraint_type(ireq) - if problem: - raise InstallationError(problem) - if not ireq.match_markers(): - continue - assert ireq.name, "Constraint must be named" - name = canonicalize_name(ireq.name) - if name in collected.constraints: - collected.constraints[name] &= ireq - else: - collected.constraints[name] = Constraint.from_ireq(ireq) - else: - reqs = list( - self._make_requirements_from_install_req( - ireq, - requested_extras=(), - ) - ) - if not reqs: - continue - template = reqs[0] - if ireq.user_supplied and template.name not in collected.user_requested: - collected.user_requested[template.name] = i - collected.requirements.extend(reqs) - # Put requirements with extras at the end of the root requires. This does not - # affect resolvelib's picking preference but it does affect its initial criteria - # population: by putting extras at the end we enable the candidate finder to - # present resolvelib with a smaller set of candidates to resolvelib, already - # taking into account any non-transient constraints on the associated base. This - # means resolvelib will have fewer candidates to visit and reject. - # Python's list sort is stable, meaning relative order is kept for objects with - # the same key. - collected.requirements.sort(key=lambda r: r.name != r.project_name) - return collected - - def make_requirement_from_candidate( - self, candidate: Candidate - ) -> ExplicitRequirement: - return ExplicitRequirement(candidate) - - def make_requirements_from_spec( - self, - specifier: str, - comes_from: Optional[InstallRequirement], - requested_extras: Iterable[str] = (), - ) -> Iterator[Requirement]: - """ - Returns requirement objects associated with the given specifier. In most cases - this will be a single object but the following special cases exist: - - the specifier has markers that do not apply -> result is empty - - the specifier has both a constraint and extras -> result is split - in two requirement objects: one with the constraint and one with the - extra. This allows centralized constraint handling for the base, - resulting in fewer candidate rejections. - """ - ireq = self._make_install_req_from_spec(specifier, comes_from) - return self._make_requirements_from_install_req(ireq, requested_extras) - - def make_requires_python_requirement( - self, - specifier: SpecifierSet, - ) -> Optional[Requirement]: - if self._ignore_requires_python: - return None - # Don't bother creating a dependency for an empty Requires-Python. - if not str(specifier): - return None - return RequiresPythonRequirement(specifier, self._python_candidate) - - def get_wheel_cache_entry( - self, link: Link, name: Optional[str] - ) -> Optional[CacheEntry]: - """Look up the link in the wheel cache. - - If ``preparer.require_hashes`` is True, don't use the wheel cache, - because cached wheels, always built locally, have different hashes - than the files downloaded from the index server and thus throw false - hash mismatches. Furthermore, cached wheels at present have - nondeterministic contents due to file modification times. - """ - if self._wheel_cache is None: - return None - return self._wheel_cache.get_cache_entry( - link=link, - package_name=name, - supported_tags=get_supported(), - ) - - def get_dist_to_uninstall(self, candidate: Candidate) -> Optional[BaseDistribution]: - # TODO: Are there more cases this needs to return True? Editable? - dist = self._installed_dists.get(candidate.project_name) - if dist is None: # Not installed, no uninstallation required. - return None - - # We're installing into global site. The current installation must - # be uninstalled, no matter it's in global or user site, because the - # user site installation has precedence over global. - if not self._use_user_site: - return dist - - # We're installing into user site. Remove the user site installation. - if dist.in_usersite: - return dist - - # We're installing into user site, but the installed incompatible - # package is in global site. We can't uninstall that, and would let - # the new user installation to "shadow" it. But shadowing won't work - # in virtual environments, so we error out. - if running_under_virtualenv() and dist.in_site_packages: - message = ( - f"Will not install to the user site because it will lack " - f"sys.path precedence to {dist.raw_name} in {dist.location}" - ) - raise InstallationError(message) - return None - - def _report_requires_python_error( - self, causes: Sequence["ConflictCause"] - ) -> UnsupportedPythonVersion: - assert causes, "Requires-Python error reported with no cause" - - version = self._python_candidate.version - - if len(causes) == 1: - specifier = str(causes[0].requirement.specifier) - message = ( - f"Package {causes[0].parent.name!r} requires a different " - f"Python: {version} not in {specifier!r}" - ) - return UnsupportedPythonVersion(message) - - message = f"Packages require a different Python. {version} not in:" - for cause in causes: - package = cause.parent.format_for_error() - specifier = str(cause.requirement.specifier) - message += f"\n{specifier!r} (required by {package})" - return UnsupportedPythonVersion(message) - - def _report_single_requirement_conflict( - self, req: Requirement, parent: Optional[Candidate] - ) -> DistributionNotFound: - if parent is None: - req_disp = str(req) - else: - req_disp = f"{req} (from {parent.name})" - - cands = self._finder.find_all_candidates(req.project_name) - skipped_by_requires_python = self._finder.requires_python_skipped_reasons() - - versions_set: Set[CandidateVersion] = set() - yanked_versions_set: Set[CandidateVersion] = set() - for c in cands: - is_yanked = c.link.is_yanked if c.link else False - if is_yanked: - yanked_versions_set.add(c.version) - else: - versions_set.add(c.version) - - versions = [str(v) for v in sorted(versions_set)] - yanked_versions = [str(v) for v in sorted(yanked_versions_set)] - - if yanked_versions: - # Saying "version X is yanked" isn't entirely accurate. - # https://github.com/pypa/pip/issues/11745#issuecomment-1402805842 - logger.critical( - "Ignored the following yanked versions: %s", - ", ".join(yanked_versions) or "none", - ) - if skipped_by_requires_python: - logger.critical( - "Ignored the following versions that require a different python " - "version: %s", - "; ".join(skipped_by_requires_python) or "none", - ) - logger.critical( - "Could not find a version that satisfies the requirement %s " - "(from versions: %s)", - req_disp, - ", ".join(versions) or "none", - ) - if str(req) == "requirements.txt": - logger.info( - "HINT: You are attempting to install a package literally " - 'named "requirements.txt" (which cannot exist). Consider ' - "using the '-r' flag to install the packages listed in " - "requirements.txt" - ) - - return DistributionNotFound(f"No matching distribution found for {req}") - - def get_installation_error( - self, - e: "ResolutionImpossible[Requirement, Candidate]", - constraints: Dict[str, Constraint], - ) -> InstallationError: - assert e.causes, "Installation error reported with no cause" - - # If one of the things we can't solve is "we need Python X.Y", - # that is what we report. - requires_python_causes = [ - cause - for cause in e.causes - if isinstance(cause.requirement, RequiresPythonRequirement) - and not cause.requirement.is_satisfied_by(self._python_candidate) - ] - if requires_python_causes: - # The comprehension above makes sure all Requirement instances are - # RequiresPythonRequirement, so let's cast for convenience. - return self._report_requires_python_error( - cast("Sequence[ConflictCause]", requires_python_causes), - ) - - # Otherwise, we have a set of causes which can't all be satisfied - # at once. - - # The simplest case is when we have *one* cause that can't be - # satisfied. We just report that case. - if len(e.causes) == 1: - req, parent = e.causes[0] - if req.name not in constraints: - return self._report_single_requirement_conflict(req, parent) - - # OK, we now have a list of requirements that can't all be - # satisfied at once. - - # A couple of formatting helpers - def text_join(parts: List[str]) -> str: - if len(parts) == 1: - return parts[0] - - return ", ".join(parts[:-1]) + " and " + parts[-1] - - def describe_trigger(parent: Candidate) -> str: - ireq = parent.get_install_requirement() - if not ireq or not ireq.comes_from: - return f"{parent.name}=={parent.version}" - if isinstance(ireq.comes_from, InstallRequirement): - return str(ireq.comes_from.name) - return str(ireq.comes_from) - - triggers = set() - for req, parent in e.causes: - if parent is None: - # This is a root requirement, so we can report it directly - trigger = req.format_for_error() - else: - trigger = describe_trigger(parent) - triggers.add(trigger) - - if triggers: - info = text_join(sorted(triggers)) - else: - info = "the requested packages" - - msg = ( - "Cannot install {} because these package versions " - "have conflicting dependencies.".format(info) - ) - logger.critical(msg) - msg = "\nThe conflict is caused by:" - - relevant_constraints = set() - for req, parent in e.causes: - if req.name in constraints: - relevant_constraints.add(req.name) - msg = msg + "\n " - if parent: - msg = msg + f"{parent.name} {parent.version} depends on " - else: - msg = msg + "The user requested " - msg = msg + req.format_for_error() - for key in relevant_constraints: - spec = constraints[key].specifier - msg += f"\n The user requested (constraint) {key}{spec}" - - msg = ( - msg - + "\n\n" - + "To fix this you could try to:\n" - + "1. loosen the range of package versions you've specified\n" - + "2. remove package versions to allow pip attempt to solve " - + "the dependency conflict\n" - ) - - logger.info(msg) - - return DistributionNotFound( - "ResolutionImpossible: for help visit " - "https://pip.pypa.io/en/latest/topics/dependency-resolution/" - "#dealing-with-dependency-conflicts" - ) diff --git a/venv/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py b/venv/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py deleted file mode 100644 index 8663097..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py +++ /dev/null @@ -1,155 +0,0 @@ -"""Utilities to lazily create and visit candidates found. - -Creating and visiting a candidate is a *very* costly operation. It involves -fetching, extracting, potentially building modules from source, and verifying -distribution metadata. It is therefore crucial for performance to keep -everything here lazy all the way down, so we only touch candidates that we -absolutely need, and not "download the world" when we only need one version of -something. -""" - -import functools -from collections.abc import Sequence -from typing import TYPE_CHECKING, Any, Callable, Iterator, Optional, Set, Tuple - -from pip._vendor.packaging.version import _BaseVersion - -from .base import Candidate - -IndexCandidateInfo = Tuple[_BaseVersion, Callable[[], Optional[Candidate]]] - -if TYPE_CHECKING: - SequenceCandidate = Sequence[Candidate] -else: - # For compatibility: Python before 3.9 does not support using [] on the - # Sequence class. - # - # >>> from collections.abc import Sequence - # >>> Sequence[str] - # Traceback (most recent call last): - # File "", line 1, in - # TypeError: 'ABCMeta' object is not subscriptable - # - # TODO: Remove this block after dropping Python 3.8 support. - SequenceCandidate = Sequence - - -def _iter_built(infos: Iterator[IndexCandidateInfo]) -> Iterator[Candidate]: - """Iterator for ``FoundCandidates``. - - This iterator is used when the package is not already installed. Candidates - from index come later in their normal ordering. - """ - versions_found: Set[_BaseVersion] = set() - for version, func in infos: - if version in versions_found: - continue - candidate = func() - if candidate is None: - continue - yield candidate - versions_found.add(version) - - -def _iter_built_with_prepended( - installed: Candidate, infos: Iterator[IndexCandidateInfo] -) -> Iterator[Candidate]: - """Iterator for ``FoundCandidates``. - - This iterator is used when the resolver prefers the already-installed - candidate and NOT to upgrade. The installed candidate is therefore - always yielded first, and candidates from index come later in their - normal ordering, except skipped when the version is already installed. - """ - yield installed - versions_found: Set[_BaseVersion] = {installed.version} - for version, func in infos: - if version in versions_found: - continue - candidate = func() - if candidate is None: - continue - yield candidate - versions_found.add(version) - - -def _iter_built_with_inserted( - installed: Candidate, infos: Iterator[IndexCandidateInfo] -) -> Iterator[Candidate]: - """Iterator for ``FoundCandidates``. - - This iterator is used when the resolver prefers to upgrade an - already-installed package. Candidates from index are returned in their - normal ordering, except replaced when the version is already installed. - - The implementation iterates through and yields other candidates, inserting - the installed candidate exactly once before we start yielding older or - equivalent candidates, or after all other candidates if they are all newer. - """ - versions_found: Set[_BaseVersion] = set() - for version, func in infos: - if version in versions_found: - continue - # If the installed candidate is better, yield it first. - if installed.version >= version: - yield installed - versions_found.add(installed.version) - candidate = func() - if candidate is None: - continue - yield candidate - versions_found.add(version) - - # If the installed candidate is older than all other candidates. - if installed.version not in versions_found: - yield installed - - -class FoundCandidates(SequenceCandidate): - """A lazy sequence to provide candidates to the resolver. - - The intended usage is to return this from `find_matches()` so the resolver - can iterate through the sequence multiple times, but only access the index - page when remote packages are actually needed. This improve performances - when suitable candidates are already installed on disk. - """ - - def __init__( - self, - get_infos: Callable[[], Iterator[IndexCandidateInfo]], - installed: Optional[Candidate], - prefers_installed: bool, - incompatible_ids: Set[int], - ): - self._get_infos = get_infos - self._installed = installed - self._prefers_installed = prefers_installed - self._incompatible_ids = incompatible_ids - - def __getitem__(self, index: Any) -> Any: - # Implemented to satisfy the ABC check. This is not needed by the - # resolver, and should not be used by the provider either (for - # performance reasons). - raise NotImplementedError("don't do this") - - def __iter__(self) -> Iterator[Candidate]: - infos = self._get_infos() - if not self._installed: - iterator = _iter_built(infos) - elif self._prefers_installed: - iterator = _iter_built_with_prepended(self._installed, infos) - else: - iterator = _iter_built_with_inserted(self._installed, infos) - return (c for c in iterator if id(c) not in self._incompatible_ids) - - def __len__(self) -> int: - # Implemented to satisfy the ABC check. This is not needed by the - # resolver, and should not be used by the provider either (for - # performance reasons). - raise NotImplementedError("don't do this") - - @functools.lru_cache(maxsize=1) - def __bool__(self) -> bool: - if self._prefers_installed and self._installed: - return True - return any(self) diff --git a/venv/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/provider.py b/venv/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/provider.py deleted file mode 100644 index 315fb9c..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/provider.py +++ /dev/null @@ -1,255 +0,0 @@ -import collections -import math -from typing import ( - TYPE_CHECKING, - Dict, - Iterable, - Iterator, - Mapping, - Sequence, - TypeVar, - Union, -) - -from pip._vendor.resolvelib.providers import AbstractProvider - -from .base import Candidate, Constraint, Requirement -from .candidates import REQUIRES_PYTHON_IDENTIFIER -from .factory import Factory - -if TYPE_CHECKING: - from pip._vendor.resolvelib.providers import Preference - from pip._vendor.resolvelib.resolvers import RequirementInformation - - PreferenceInformation = RequirementInformation[Requirement, Candidate] - - _ProviderBase = AbstractProvider[Requirement, Candidate, str] -else: - _ProviderBase = AbstractProvider - -# Notes on the relationship between the provider, the factory, and the -# candidate and requirement classes. -# -# The provider is a direct implementation of the resolvelib class. Its role -# is to deliver the API that resolvelib expects. -# -# Rather than work with completely abstract "requirement" and "candidate" -# concepts as resolvelib does, pip has concrete classes implementing these two -# ideas. The API of Requirement and Candidate objects are defined in the base -# classes, but essentially map fairly directly to the equivalent provider -# methods. In particular, `find_matches` and `is_satisfied_by` are -# requirement methods, and `get_dependencies` is a candidate method. -# -# The factory is the interface to pip's internal mechanisms. It is stateless, -# and is created by the resolver and held as a property of the provider. It is -# responsible for creating Requirement and Candidate objects, and provides -# services to those objects (access to pip's finder and preparer). - - -D = TypeVar("D") -V = TypeVar("V") - - -def _get_with_identifier( - mapping: Mapping[str, V], - identifier: str, - default: D, -) -> Union[D, V]: - """Get item from a package name lookup mapping with a resolver identifier. - - This extra logic is needed when the target mapping is keyed by package - name, which cannot be directly looked up with an identifier (which may - contain requested extras). Additional logic is added to also look up a value - by "cleaning up" the extras from the identifier. - """ - if identifier in mapping: - return mapping[identifier] - # HACK: Theoretically we should check whether this identifier is a valid - # "NAME[EXTRAS]" format, and parse out the name part with packaging or - # some regular expression. But since pip's resolver only spits out three - # kinds of identifiers: normalized PEP 503 names, normalized names plus - # extras, and Requires-Python, we can cheat a bit here. - name, open_bracket, _ = identifier.partition("[") - if open_bracket and name in mapping: - return mapping[name] - return default - - -class PipProvider(_ProviderBase): - """Pip's provider implementation for resolvelib. - - :params constraints: A mapping of constraints specified by the user. Keys - are canonicalized project names. - :params ignore_dependencies: Whether the user specified ``--no-deps``. - :params upgrade_strategy: The user-specified upgrade strategy. - :params user_requested: A set of canonicalized package names that the user - supplied for pip to install/upgrade. - """ - - def __init__( - self, - factory: Factory, - constraints: Dict[str, Constraint], - ignore_dependencies: bool, - upgrade_strategy: str, - user_requested: Dict[str, int], - ) -> None: - self._factory = factory - self._constraints = constraints - self._ignore_dependencies = ignore_dependencies - self._upgrade_strategy = upgrade_strategy - self._user_requested = user_requested - self._known_depths: Dict[str, float] = collections.defaultdict(lambda: math.inf) - - def identify(self, requirement_or_candidate: Union[Requirement, Candidate]) -> str: - return requirement_or_candidate.name - - def get_preference( - self, - identifier: str, - resolutions: Mapping[str, Candidate], - candidates: Mapping[str, Iterator[Candidate]], - information: Mapping[str, Iterable["PreferenceInformation"]], - backtrack_causes: Sequence["PreferenceInformation"], - ) -> "Preference": - """Produce a sort key for given requirement based on preference. - - The lower the return value is, the more preferred this group of - arguments is. - - Currently pip considers the following in order: - - * Prefer if any of the known requirements is "direct", e.g. points to an - explicit URL. - * If equal, prefer if any requirement is "pinned", i.e. contains - operator ``===`` or ``==``. - * If equal, calculate an approximate "depth" and resolve requirements - closer to the user-specified requirements first. If the depth cannot - by determined (eg: due to no matching parents), it is considered - infinite. - * Order user-specified requirements by the order they are specified. - * If equal, prefers "non-free" requirements, i.e. contains at least one - operator, such as ``>=`` or ``<``. - * If equal, order alphabetically for consistency (helps debuggability). - """ - try: - next(iter(information[identifier])) - except StopIteration: - # There is no information for this identifier, so there's no known - # candidates. - has_information = False - else: - has_information = True - - if has_information: - lookups = (r.get_candidate_lookup() for r, _ in information[identifier]) - candidate, ireqs = zip(*lookups) - else: - candidate, ireqs = None, () - - operators = [ - specifier.operator - for specifier_set in (ireq.specifier for ireq in ireqs if ireq) - for specifier in specifier_set - ] - - direct = candidate is not None - pinned = any(op[:2] == "==" for op in operators) - unfree = bool(operators) - - try: - requested_order: Union[int, float] = self._user_requested[identifier] - except KeyError: - requested_order = math.inf - if has_information: - parent_depths = ( - self._known_depths[parent.name] if parent is not None else 0.0 - for _, parent in information[identifier] - ) - inferred_depth = min(d for d in parent_depths) + 1.0 - else: - inferred_depth = math.inf - else: - inferred_depth = 1.0 - self._known_depths[identifier] = inferred_depth - - requested_order = self._user_requested.get(identifier, math.inf) - - # Requires-Python has only one candidate and the check is basically - # free, so we always do it first to avoid needless work if it fails. - requires_python = identifier == REQUIRES_PYTHON_IDENTIFIER - - # Prefer the causes of backtracking on the assumption that the problem - # resolving the dependency tree is related to the failures that caused - # the backtracking - backtrack_cause = self.is_backtrack_cause(identifier, backtrack_causes) - - return ( - not requires_python, - not direct, - not pinned, - not backtrack_cause, - inferred_depth, - requested_order, - not unfree, - identifier, - ) - - def find_matches( - self, - identifier: str, - requirements: Mapping[str, Iterator[Requirement]], - incompatibilities: Mapping[str, Iterator[Candidate]], - ) -> Iterable[Candidate]: - def _eligible_for_upgrade(identifier: str) -> bool: - """Are upgrades allowed for this project? - - This checks the upgrade strategy, and whether the project was one - that the user specified in the command line, in order to decide - whether we should upgrade if there's a newer version available. - - (Note that we don't need access to the `--upgrade` flag, because - an upgrade strategy of "to-satisfy-only" means that `--upgrade` - was not specified). - """ - if self._upgrade_strategy == "eager": - return True - elif self._upgrade_strategy == "only-if-needed": - user_order = _get_with_identifier( - self._user_requested, - identifier, - default=None, - ) - return user_order is not None - return False - - constraint = _get_with_identifier( - self._constraints, - identifier, - default=Constraint.empty(), - ) - return self._factory.find_candidates( - identifier=identifier, - requirements=requirements, - constraint=constraint, - prefers_installed=(not _eligible_for_upgrade(identifier)), - incompatibilities=incompatibilities, - ) - - def is_satisfied_by(self, requirement: Requirement, candidate: Candidate) -> bool: - return requirement.is_satisfied_by(candidate) - - def get_dependencies(self, candidate: Candidate) -> Sequence[Requirement]: - with_requires = not self._ignore_dependencies - return [r for r in candidate.iter_dependencies(with_requires) if r is not None] - - @staticmethod - def is_backtrack_cause( - identifier: str, backtrack_causes: Sequence["PreferenceInformation"] - ) -> bool: - for backtrack_cause in backtrack_causes: - if identifier == backtrack_cause.requirement.name: - return True - if backtrack_cause.parent and identifier == backtrack_cause.parent.name: - return True - return False diff --git a/venv/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/reporter.py b/venv/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/reporter.py deleted file mode 100644 index 12adeff..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/reporter.py +++ /dev/null @@ -1,80 +0,0 @@ -from collections import defaultdict -from logging import getLogger -from typing import Any, DefaultDict - -from pip._vendor.resolvelib.reporters import BaseReporter - -from .base import Candidate, Requirement - -logger = getLogger(__name__) - - -class PipReporter(BaseReporter): - def __init__(self) -> None: - self.reject_count_by_package: DefaultDict[str, int] = defaultdict(int) - - self._messages_at_reject_count = { - 1: ( - "pip is looking at multiple versions of {package_name} to " - "determine which version is compatible with other " - "requirements. This could take a while." - ), - 8: ( - "pip is still looking at multiple versions of {package_name} to " - "determine which version is compatible with other " - "requirements. This could take a while." - ), - 13: ( - "This is taking longer than usual. You might need to provide " - "the dependency resolver with stricter constraints to reduce " - "runtime. See https://pip.pypa.io/warnings/backtracking for " - "guidance. If you want to abort this run, press Ctrl + C." - ), - } - - def rejecting_candidate(self, criterion: Any, candidate: Candidate) -> None: - self.reject_count_by_package[candidate.name] += 1 - - count = self.reject_count_by_package[candidate.name] - if count not in self._messages_at_reject_count: - return - - message = self._messages_at_reject_count[count] - logger.info("INFO: %s", message.format(package_name=candidate.name)) - - msg = "Will try a different candidate, due to conflict:" - for req_info in criterion.information: - req, parent = req_info.requirement, req_info.parent - # Inspired by Factory.get_installation_error - msg += "\n " - if parent: - msg += f"{parent.name} {parent.version} depends on " - else: - msg += "The user requested " - msg += req.format_for_error() - logger.debug(msg) - - -class PipDebuggingReporter(BaseReporter): - """A reporter that does an info log for every event it sees.""" - - def starting(self) -> None: - logger.info("Reporter.starting()") - - def starting_round(self, index: int) -> None: - logger.info("Reporter.starting_round(%r)", index) - - def ending_round(self, index: int, state: Any) -> None: - logger.info("Reporter.ending_round(%r, state)", index) - - def ending(self, state: Any) -> None: - logger.info("Reporter.ending(%r)", state) - - def adding_requirement(self, requirement: Requirement, parent: Candidate) -> None: - logger.info("Reporter.adding_requirement(%r, %r)", requirement, parent) - - def rejecting_candidate(self, criterion: Any, candidate: Candidate) -> None: - logger.info("Reporter.rejecting_candidate(%r, %r)", criterion, candidate) - - def pinning(self, candidate: Candidate) -> None: - logger.info("Reporter.pinning(%r)", candidate) diff --git a/venv/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/requirements.py b/venv/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/requirements.py deleted file mode 100644 index 7d1e7bf..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/requirements.py +++ /dev/null @@ -1,178 +0,0 @@ -from pip._vendor.packaging.specifiers import SpecifierSet -from pip._vendor.packaging.utils import NormalizedName, canonicalize_name - -from pip._internal.req.constructors import install_req_drop_extras -from pip._internal.req.req_install import InstallRequirement - -from .base import Candidate, CandidateLookup, Requirement, format_name - - -class ExplicitRequirement(Requirement): - def __init__(self, candidate: Candidate) -> None: - self.candidate = candidate - - def __str__(self) -> str: - return str(self.candidate) - - def __repr__(self) -> str: - return "{class_name}({candidate!r})".format( - class_name=self.__class__.__name__, - candidate=self.candidate, - ) - - @property - def project_name(self) -> NormalizedName: - # No need to canonicalize - the candidate did this - return self.candidate.project_name - - @property - def name(self) -> str: - # No need to canonicalize - the candidate did this - return self.candidate.name - - def format_for_error(self) -> str: - return self.candidate.format_for_error() - - def get_candidate_lookup(self) -> CandidateLookup: - return self.candidate, None - - def is_satisfied_by(self, candidate: Candidate) -> bool: - return candidate == self.candidate - - -class SpecifierRequirement(Requirement): - def __init__(self, ireq: InstallRequirement) -> None: - assert ireq.link is None, "This is a link, not a specifier" - self._ireq = ireq - self._extras = frozenset(canonicalize_name(e) for e in self._ireq.extras) - - def __str__(self) -> str: - return str(self._ireq.req) - - def __repr__(self) -> str: - return "{class_name}({requirement!r})".format( - class_name=self.__class__.__name__, - requirement=str(self._ireq.req), - ) - - @property - def project_name(self) -> NormalizedName: - assert self._ireq.req, "Specifier-backed ireq is always PEP 508" - return canonicalize_name(self._ireq.req.name) - - @property - def name(self) -> str: - return format_name(self.project_name, self._extras) - - def format_for_error(self) -> str: - # Convert comma-separated specifiers into "A, B, ..., F and G" - # This makes the specifier a bit more "human readable", without - # risking a change in meaning. (Hopefully! Not all edge cases have - # been checked) - parts = [s.strip() for s in str(self).split(",")] - if len(parts) == 0: - return "" - elif len(parts) == 1: - return parts[0] - - return ", ".join(parts[:-1]) + " and " + parts[-1] - - def get_candidate_lookup(self) -> CandidateLookup: - return None, self._ireq - - def is_satisfied_by(self, candidate: Candidate) -> bool: - assert candidate.name == self.name, ( - f"Internal issue: Candidate is not for this requirement " - f"{candidate.name} vs {self.name}" - ) - # We can safely always allow prereleases here since PackageFinder - # already implements the prerelease logic, and would have filtered out - # prerelease candidates if the user does not expect them. - assert self._ireq.req, "Specifier-backed ireq is always PEP 508" - spec = self._ireq.req.specifier - return spec.contains(candidate.version, prereleases=True) - - -class SpecifierWithoutExtrasRequirement(SpecifierRequirement): - """ - Requirement backed by an install requirement on a base package. - Trims extras from its install requirement if there are any. - """ - - def __init__(self, ireq: InstallRequirement) -> None: - assert ireq.link is None, "This is a link, not a specifier" - self._ireq = install_req_drop_extras(ireq) - self._extras = frozenset(canonicalize_name(e) for e in self._ireq.extras) - - -class RequiresPythonRequirement(Requirement): - """A requirement representing Requires-Python metadata.""" - - def __init__(self, specifier: SpecifierSet, match: Candidate) -> None: - self.specifier = specifier - self._candidate = match - - def __str__(self) -> str: - return f"Python {self.specifier}" - - def __repr__(self) -> str: - return "{class_name}({specifier!r})".format( - class_name=self.__class__.__name__, - specifier=str(self.specifier), - ) - - @property - def project_name(self) -> NormalizedName: - return self._candidate.project_name - - @property - def name(self) -> str: - return self._candidate.name - - def format_for_error(self) -> str: - return str(self) - - def get_candidate_lookup(self) -> CandidateLookup: - if self.specifier.contains(self._candidate.version, prereleases=True): - return self._candidate, None - return None, None - - def is_satisfied_by(self, candidate: Candidate) -> bool: - assert candidate.name == self._candidate.name, "Not Python candidate" - # We can safely always allow prereleases here since PackageFinder - # already implements the prerelease logic, and would have filtered out - # prerelease candidates if the user does not expect them. - return self.specifier.contains(candidate.version, prereleases=True) - - -class UnsatisfiableRequirement(Requirement): - """A requirement that cannot be satisfied.""" - - def __init__(self, name: NormalizedName) -> None: - self._name = name - - def __str__(self) -> str: - return f"{self._name} (unavailable)" - - def __repr__(self) -> str: - return "{class_name}({name!r})".format( - class_name=self.__class__.__name__, - name=str(self._name), - ) - - @property - def project_name(self) -> NormalizedName: - return self._name - - @property - def name(self) -> str: - return self._name - - def format_for_error(self) -> str: - return str(self) - - def get_candidate_lookup(self) -> CandidateLookup: - return None, None - - def is_satisfied_by(self, candidate: Candidate) -> bool: - return False diff --git a/venv/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/resolver.py b/venv/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/resolver.py deleted file mode 100644 index c12beef..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/resolver.py +++ /dev/null @@ -1,317 +0,0 @@ -import contextlib -import functools -import logging -import os -from typing import TYPE_CHECKING, Dict, List, Optional, Set, Tuple, cast - -from pip._vendor.packaging.utils import canonicalize_name -from pip._vendor.resolvelib import BaseReporter, ResolutionImpossible -from pip._vendor.resolvelib import Resolver as RLResolver -from pip._vendor.resolvelib.structs import DirectedGraph - -from pip._internal.cache import WheelCache -from pip._internal.index.package_finder import PackageFinder -from pip._internal.operations.prepare import RequirementPreparer -from pip._internal.req.constructors import install_req_extend_extras -from pip._internal.req.req_install import InstallRequirement -from pip._internal.req.req_set import RequirementSet -from pip._internal.resolution.base import BaseResolver, InstallRequirementProvider -from pip._internal.resolution.resolvelib.provider import PipProvider -from pip._internal.resolution.resolvelib.reporter import ( - PipDebuggingReporter, - PipReporter, -) -from pip._internal.utils.packaging import get_requirement - -from .base import Candidate, Requirement -from .factory import Factory - -if TYPE_CHECKING: - from pip._vendor.resolvelib.resolvers import Result as RLResult - - Result = RLResult[Requirement, Candidate, str] - - -logger = logging.getLogger(__name__) - - -class Resolver(BaseResolver): - _allowed_strategies = {"eager", "only-if-needed", "to-satisfy-only"} - - def __init__( - self, - preparer: RequirementPreparer, - finder: PackageFinder, - wheel_cache: Optional[WheelCache], - make_install_req: InstallRequirementProvider, - use_user_site: bool, - ignore_dependencies: bool, - ignore_installed: bool, - ignore_requires_python: bool, - force_reinstall: bool, - upgrade_strategy: str, - py_version_info: Optional[Tuple[int, ...]] = None, - ): - super().__init__() - assert upgrade_strategy in self._allowed_strategies - - self.factory = Factory( - finder=finder, - preparer=preparer, - make_install_req=make_install_req, - wheel_cache=wheel_cache, - use_user_site=use_user_site, - force_reinstall=force_reinstall, - ignore_installed=ignore_installed, - ignore_requires_python=ignore_requires_python, - py_version_info=py_version_info, - ) - self.ignore_dependencies = ignore_dependencies - self.upgrade_strategy = upgrade_strategy - self._result: Optional[Result] = None - - def resolve( - self, root_reqs: List[InstallRequirement], check_supported_wheels: bool - ) -> RequirementSet: - collected = self.factory.collect_root_requirements(root_reqs) - provider = PipProvider( - factory=self.factory, - constraints=collected.constraints, - ignore_dependencies=self.ignore_dependencies, - upgrade_strategy=self.upgrade_strategy, - user_requested=collected.user_requested, - ) - if "PIP_RESOLVER_DEBUG" in os.environ: - reporter: BaseReporter = PipDebuggingReporter() - else: - reporter = PipReporter() - resolver: RLResolver[Requirement, Candidate, str] = RLResolver( - provider, - reporter, - ) - - try: - limit_how_complex_resolution_can_be = 200000 - result = self._result = resolver.resolve( - collected.requirements, max_rounds=limit_how_complex_resolution_can_be - ) - - except ResolutionImpossible as e: - error = self.factory.get_installation_error( - cast("ResolutionImpossible[Requirement, Candidate]", e), - collected.constraints, - ) - raise error from e - - req_set = RequirementSet(check_supported_wheels=check_supported_wheels) - # process candidates with extras last to ensure their base equivalent is - # already in the req_set if appropriate. - # Python's sort is stable so using a binary key function keeps relative order - # within both subsets. - for candidate in sorted( - result.mapping.values(), key=lambda c: c.name != c.project_name - ): - ireq = candidate.get_install_requirement() - if ireq is None: - if candidate.name != candidate.project_name: - # extend existing req's extras - with contextlib.suppress(KeyError): - req = req_set.get_requirement(candidate.project_name) - req_set.add_named_requirement( - install_req_extend_extras( - req, get_requirement(candidate.name).extras - ) - ) - continue - - # Check if there is already an installation under the same name, - # and set a flag for later stages to uninstall it, if needed. - installed_dist = self.factory.get_dist_to_uninstall(candidate) - if installed_dist is None: - # There is no existing installation -- nothing to uninstall. - ireq.should_reinstall = False - elif self.factory.force_reinstall: - # The --force-reinstall flag is set -- reinstall. - ireq.should_reinstall = True - elif installed_dist.version != candidate.version: - # The installation is different in version -- reinstall. - ireq.should_reinstall = True - elif candidate.is_editable or installed_dist.editable: - # The incoming distribution is editable, or different in - # editable-ness to installation -- reinstall. - ireq.should_reinstall = True - elif candidate.source_link and candidate.source_link.is_file: - # The incoming distribution is under file:// - if candidate.source_link.is_wheel: - # is a local wheel -- do nothing. - logger.info( - "%s is already installed with the same version as the " - "provided wheel. Use --force-reinstall to force an " - "installation of the wheel.", - ireq.name, - ) - continue - - # is a local sdist or path -- reinstall - ireq.should_reinstall = True - else: - continue - - link = candidate.source_link - if link and link.is_yanked: - # The reason can contain non-ASCII characters, Unicode - # is required for Python 2. - msg = ( - "The candidate selected for download or install is a " - "yanked version: {name!r} candidate (version {version} " - "at {link})\nReason for being yanked: {reason}" - ).format( - name=candidate.name, - version=candidate.version, - link=link, - reason=link.yanked_reason or "", - ) - logger.warning(msg) - - req_set.add_named_requirement(ireq) - - reqs = req_set.all_requirements - self.factory.preparer.prepare_linked_requirements_more(reqs) - for req in reqs: - req.prepared = True - req.needs_more_preparation = False - return req_set - - def get_installation_order( - self, req_set: RequirementSet - ) -> List[InstallRequirement]: - """Get order for installation of requirements in RequirementSet. - - The returned list contains a requirement before another that depends on - it. This helps ensure that the environment is kept consistent as they - get installed one-by-one. - - The current implementation creates a topological ordering of the - dependency graph, giving more weight to packages with less - or no dependencies, while breaking any cycles in the graph at - arbitrary points. We make no guarantees about where the cycle - would be broken, other than it *would* be broken. - """ - assert self._result is not None, "must call resolve() first" - - if not req_set.requirements: - # Nothing is left to install, so we do not need an order. - return [] - - graph = self._result.graph - weights = get_topological_weights(graph, set(req_set.requirements.keys())) - - sorted_items = sorted( - req_set.requirements.items(), - key=functools.partial(_req_set_item_sorter, weights=weights), - reverse=True, - ) - return [ireq for _, ireq in sorted_items] - - -def get_topological_weights( - graph: "DirectedGraph[Optional[str]]", requirement_keys: Set[str] -) -> Dict[Optional[str], int]: - """Assign weights to each node based on how "deep" they are. - - This implementation may change at any point in the future without prior - notice. - - We first simplify the dependency graph by pruning any leaves and giving them - the highest weight: a package without any dependencies should be installed - first. This is done again and again in the same way, giving ever less weight - to the newly found leaves. The loop stops when no leaves are left: all - remaining packages have at least one dependency left in the graph. - - Then we continue with the remaining graph, by taking the length for the - longest path to any node from root, ignoring any paths that contain a single - node twice (i.e. cycles). This is done through a depth-first search through - the graph, while keeping track of the path to the node. - - Cycles in the graph result would result in node being revisited while also - being on its own path. In this case, take no action. This helps ensure we - don't get stuck in a cycle. - - When assigning weight, the longer path (i.e. larger length) is preferred. - - We are only interested in the weights of packages that are in the - requirement_keys. - """ - path: Set[Optional[str]] = set() - weights: Dict[Optional[str], int] = {} - - def visit(node: Optional[str]) -> None: - if node in path: - # We hit a cycle, so we'll break it here. - return - - # Time to visit the children! - path.add(node) - for child in graph.iter_children(node): - visit(child) - path.remove(node) - - if node not in requirement_keys: - return - - last_known_parent_count = weights.get(node, 0) - weights[node] = max(last_known_parent_count, len(path)) - - # Simplify the graph, pruning leaves that have no dependencies. - # This is needed for large graphs (say over 200 packages) because the - # `visit` function is exponentially slower then, taking minutes. - # See https://github.com/pypa/pip/issues/10557 - # We will loop until we explicitly break the loop. - while True: - leaves = set() - for key in graph: - if key is None: - continue - for _child in graph.iter_children(key): - # This means we have at least one child - break - else: - # No child. - leaves.add(key) - if not leaves: - # We are done simplifying. - break - # Calculate the weight for the leaves. - weight = len(graph) - 1 - for leaf in leaves: - if leaf not in requirement_keys: - continue - weights[leaf] = weight - # Remove the leaves from the graph, making it simpler. - for leaf in leaves: - graph.remove(leaf) - - # Visit the remaining graph. - # `None` is guaranteed to be the root node by resolvelib. - visit(None) - - # Sanity check: all requirement keys should be in the weights, - # and no other keys should be in the weights. - difference = set(weights.keys()).difference(requirement_keys) - assert not difference, difference - - return weights - - -def _req_set_item_sorter( - item: Tuple[str, InstallRequirement], - weights: Dict[Optional[str], int], -) -> Tuple[int, str]: - """Key function used to sort install requirements for installation. - - Based on the "weight" mapping calculated in ``get_installation_order()``. - The canonical package name is returned as the second member as a tie- - breaker to ensure the result is predictable, which is useful in tests. - """ - name = canonicalize_name(item[0]) - return weights[name], name diff --git a/venv/lib/python3.11/site-packages/pip/_internal/self_outdated_check.py b/venv/lib/python3.11/site-packages/pip/_internal/self_outdated_check.py deleted file mode 100644 index 0f64ae0..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/self_outdated_check.py +++ /dev/null @@ -1,248 +0,0 @@ -import datetime -import functools -import hashlib -import json -import logging -import optparse -import os.path -import sys -from dataclasses import dataclass -from typing import Any, Callable, Dict, Optional - -from pip._vendor.packaging.version import parse as parse_version -from pip._vendor.rich.console import Group -from pip._vendor.rich.markup import escape -from pip._vendor.rich.text import Text - -from pip._internal.index.collector import LinkCollector -from pip._internal.index.package_finder import PackageFinder -from pip._internal.metadata import get_default_environment -from pip._internal.metadata.base import DistributionVersion -from pip._internal.models.selection_prefs import SelectionPreferences -from pip._internal.network.session import PipSession -from pip._internal.utils.compat import WINDOWS -from pip._internal.utils.entrypoints import ( - get_best_invocation_for_this_pip, - get_best_invocation_for_this_python, -) -from pip._internal.utils.filesystem import adjacent_tmp_file, check_path_owner, replace -from pip._internal.utils.misc import ensure_dir - -_WEEK = datetime.timedelta(days=7) - -logger = logging.getLogger(__name__) - - -def _get_statefile_name(key: str) -> str: - key_bytes = key.encode() - name = hashlib.sha224(key_bytes).hexdigest() - return name - - -def _convert_date(isodate: str) -> datetime.datetime: - """Convert an ISO format string to a date. - - Handles the format 2020-01-22T14:24:01Z (trailing Z) - which is not supported by older versions of fromisoformat. - """ - return datetime.datetime.fromisoformat(isodate.replace("Z", "+00:00")) - - -class SelfCheckState: - def __init__(self, cache_dir: str) -> None: - self._state: Dict[str, Any] = {} - self._statefile_path = None - - # Try to load the existing state - if cache_dir: - self._statefile_path = os.path.join( - cache_dir, "selfcheck", _get_statefile_name(self.key) - ) - try: - with open(self._statefile_path, encoding="utf-8") as statefile: - self._state = json.load(statefile) - except (OSError, ValueError, KeyError): - # Explicitly suppressing exceptions, since we don't want to - # error out if the cache file is invalid. - pass - - @property - def key(self) -> str: - return sys.prefix - - def get(self, current_time: datetime.datetime) -> Optional[str]: - """Check if we have a not-outdated version loaded already.""" - if not self._state: - return None - - if "last_check" not in self._state: - return None - - if "pypi_version" not in self._state: - return None - - # Determine if we need to refresh the state - last_check = _convert_date(self._state["last_check"]) - time_since_last_check = current_time - last_check - if time_since_last_check > _WEEK: - return None - - return self._state["pypi_version"] - - def set(self, pypi_version: str, current_time: datetime.datetime) -> None: - # If we do not have a path to cache in, don't bother saving. - if not self._statefile_path: - return - - # Check to make sure that we own the directory - if not check_path_owner(os.path.dirname(self._statefile_path)): - return - - # Now that we've ensured the directory is owned by this user, we'll go - # ahead and make sure that all our directories are created. - ensure_dir(os.path.dirname(self._statefile_path)) - - state = { - # Include the key so it's easy to tell which pip wrote the - # file. - "key": self.key, - "last_check": current_time.isoformat(), - "pypi_version": pypi_version, - } - - text = json.dumps(state, sort_keys=True, separators=(",", ":")) - - with adjacent_tmp_file(self._statefile_path) as f: - f.write(text.encode()) - - try: - # Since we have a prefix-specific state file, we can just - # overwrite whatever is there, no need to check. - replace(f.name, self._statefile_path) - except OSError: - # Best effort. - pass - - -@dataclass -class UpgradePrompt: - old: str - new: str - - def __rich__(self) -> Group: - if WINDOWS: - pip_cmd = f"{get_best_invocation_for_this_python()} -m pip" - else: - pip_cmd = get_best_invocation_for_this_pip() - - notice = "[bold][[reset][blue]notice[reset][bold]][reset]" - return Group( - Text(), - Text.from_markup( - f"{notice} A new release of pip is available: " - f"[red]{self.old}[reset] -> [green]{self.new}[reset]" - ), - Text.from_markup( - f"{notice} To update, run: " - f"[green]{escape(pip_cmd)} install --upgrade pip" - ), - ) - - -def was_installed_by_pip(pkg: str) -> bool: - """Checks whether pkg was installed by pip - - This is used not to display the upgrade message when pip is in fact - installed by system package manager, such as dnf on Fedora. - """ - dist = get_default_environment().get_distribution(pkg) - return dist is not None and "pip" == dist.installer - - -def _get_current_remote_pip_version( - session: PipSession, options: optparse.Values -) -> Optional[str]: - # Lets use PackageFinder to see what the latest pip version is - link_collector = LinkCollector.create( - session, - options=options, - suppress_no_index=True, - ) - - # Pass allow_yanked=False so we don't suggest upgrading to a - # yanked version. - selection_prefs = SelectionPreferences( - allow_yanked=False, - allow_all_prereleases=False, # Explicitly set to False - ) - - finder = PackageFinder.create( - link_collector=link_collector, - selection_prefs=selection_prefs, - ) - best_candidate = finder.find_best_candidate("pip").best_candidate - if best_candidate is None: - return None - - return str(best_candidate.version) - - -def _self_version_check_logic( - *, - state: SelfCheckState, - current_time: datetime.datetime, - local_version: DistributionVersion, - get_remote_version: Callable[[], Optional[str]], -) -> Optional[UpgradePrompt]: - remote_version_str = state.get(current_time) - if remote_version_str is None: - remote_version_str = get_remote_version() - if remote_version_str is None: - logger.debug("No remote pip version found") - return None - state.set(remote_version_str, current_time) - - remote_version = parse_version(remote_version_str) - logger.debug("Remote version of pip: %s", remote_version) - logger.debug("Local version of pip: %s", local_version) - - pip_installed_by_pip = was_installed_by_pip("pip") - logger.debug("Was pip installed by pip? %s", pip_installed_by_pip) - if not pip_installed_by_pip: - return None # Only suggest upgrade if pip is installed by pip. - - local_version_is_older = ( - local_version < remote_version - and local_version.base_version != remote_version.base_version - ) - if local_version_is_older: - return UpgradePrompt(old=str(local_version), new=remote_version_str) - - return None - - -def pip_self_version_check(session: PipSession, options: optparse.Values) -> None: - """Check for an update for pip. - - Limit the frequency of checks to once per week. State is stored either in - the active virtualenv or in the user's USER_CACHE_DIR keyed off the prefix - of the pip script path. - """ - installed_dist = get_default_environment().get_distribution("pip") - if not installed_dist: - return - - try: - upgrade_prompt = _self_version_check_logic( - state=SelfCheckState(cache_dir=options.cache_dir), - current_time=datetime.datetime.now(datetime.timezone.utc), - local_version=installed_dist.version, - get_remote_version=functools.partial( - _get_current_remote_pip_version, session, options - ), - ) - if upgrade_prompt is not None: - logger.warning("%s", upgrade_prompt, extra={"rich": True}) - except Exception: - logger.warning("There was an error checking the latest version of pip.") - logger.debug("See below for error", exc_info=True) diff --git a/venv/lib/python3.11/site-packages/pip/_internal/utils/__init__.py b/venv/lib/python3.11/site-packages/pip/_internal/utils/__init__.py deleted file mode 100644 index e69de29..0000000 diff --git a/venv/lib/python3.11/site-packages/pip/_internal/utils/__pycache__/__init__.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/utils/__pycache__/__init__.cpython-311.pyc deleted file mode 100644 index bb26bc7..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/utils/__pycache__/__init__.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/utils/__pycache__/_jaraco_text.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/utils/__pycache__/_jaraco_text.cpython-311.pyc deleted file mode 100644 index 6a985ed..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/utils/__pycache__/_jaraco_text.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/utils/__pycache__/_log.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/utils/__pycache__/_log.cpython-311.pyc deleted file mode 100644 index 0f24910..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/utils/__pycache__/_log.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/utils/__pycache__/appdirs.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/utils/__pycache__/appdirs.cpython-311.pyc deleted file mode 100644 index 99ed04d..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/utils/__pycache__/appdirs.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/utils/__pycache__/compat.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/utils/__pycache__/compat.cpython-311.pyc deleted file mode 100644 index d43819c..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/utils/__pycache__/compat.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/utils/__pycache__/compatibility_tags.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/utils/__pycache__/compatibility_tags.cpython-311.pyc deleted file mode 100644 index b98ae17..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/utils/__pycache__/compatibility_tags.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/utils/__pycache__/datetime.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/utils/__pycache__/datetime.cpython-311.pyc deleted file mode 100644 index 8dac045..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/utils/__pycache__/datetime.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/utils/__pycache__/deprecation.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/utils/__pycache__/deprecation.cpython-311.pyc deleted file mode 100644 index f10383e..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/utils/__pycache__/deprecation.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/utils/__pycache__/direct_url_helpers.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/utils/__pycache__/direct_url_helpers.cpython-311.pyc deleted file mode 100644 index 024e5af..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/utils/__pycache__/direct_url_helpers.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/utils/__pycache__/egg_link.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/utils/__pycache__/egg_link.cpython-311.pyc deleted file mode 100644 index 1105672..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/utils/__pycache__/egg_link.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/utils/__pycache__/encoding.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/utils/__pycache__/encoding.cpython-311.pyc deleted file mode 100644 index b5e1490..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/utils/__pycache__/encoding.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/utils/__pycache__/entrypoints.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/utils/__pycache__/entrypoints.cpython-311.pyc deleted file mode 100644 index 09d79cc..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/utils/__pycache__/entrypoints.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/utils/__pycache__/filesystem.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/utils/__pycache__/filesystem.cpython-311.pyc deleted file mode 100644 index 5422a38..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/utils/__pycache__/filesystem.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/utils/__pycache__/filetypes.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/utils/__pycache__/filetypes.cpython-311.pyc deleted file mode 100644 index 3fd74c5..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/utils/__pycache__/filetypes.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/utils/__pycache__/glibc.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/utils/__pycache__/glibc.cpython-311.pyc deleted file mode 100644 index 935bd80..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/utils/__pycache__/glibc.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/utils/__pycache__/hashes.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/utils/__pycache__/hashes.cpython-311.pyc deleted file mode 100644 index a97698f..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/utils/__pycache__/hashes.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/utils/__pycache__/logging.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/utils/__pycache__/logging.cpython-311.pyc deleted file mode 100644 index 59bd6cf..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/utils/__pycache__/logging.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/utils/__pycache__/misc.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/utils/__pycache__/misc.cpython-311.pyc deleted file mode 100644 index 1dbf876..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/utils/__pycache__/misc.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/utils/__pycache__/models.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/utils/__pycache__/models.cpython-311.pyc deleted file mode 100644 index 1186041..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/utils/__pycache__/models.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/utils/__pycache__/packaging.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/utils/__pycache__/packaging.cpython-311.pyc deleted file mode 100644 index bf1b3ac..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/utils/__pycache__/packaging.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/utils/__pycache__/setuptools_build.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/utils/__pycache__/setuptools_build.cpython-311.pyc deleted file mode 100644 index 2aca897..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/utils/__pycache__/setuptools_build.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/utils/__pycache__/subprocess.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/utils/__pycache__/subprocess.cpython-311.pyc deleted file mode 100644 index ca690db..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/utils/__pycache__/subprocess.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/utils/__pycache__/temp_dir.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/utils/__pycache__/temp_dir.cpython-311.pyc deleted file mode 100644 index 67f6884..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/utils/__pycache__/temp_dir.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/utils/__pycache__/unpacking.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/utils/__pycache__/unpacking.cpython-311.pyc deleted file mode 100644 index ed7a900..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/utils/__pycache__/unpacking.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/utils/__pycache__/urls.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/utils/__pycache__/urls.cpython-311.pyc deleted file mode 100644 index 62ac522..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/utils/__pycache__/urls.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/utils/__pycache__/virtualenv.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/utils/__pycache__/virtualenv.cpython-311.pyc deleted file mode 100644 index 482f275..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/utils/__pycache__/virtualenv.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/utils/__pycache__/wheel.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/utils/__pycache__/wheel.cpython-311.pyc deleted file mode 100644 index fff5ef4..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/utils/__pycache__/wheel.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/utils/_jaraco_text.py b/venv/lib/python3.11/site-packages/pip/_internal/utils/_jaraco_text.py deleted file mode 100644 index e06947c..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/utils/_jaraco_text.py +++ /dev/null @@ -1,109 +0,0 @@ -"""Functions brought over from jaraco.text. - -These functions are not supposed to be used within `pip._internal`. These are -helper functions brought over from `jaraco.text` to enable vendoring newer -copies of `pkg_resources` without having to vendor `jaraco.text` and its entire -dependency cone; something that our vendoring setup is not currently capable of -handling. - -License reproduced from original source below: - -Copyright Jason R. Coombs - -Permission is hereby granted, free of charge, to any person obtaining a copy -of this software and associated documentation files (the "Software"), to -deal in the Software without restriction, including without limitation the -rights to use, copy, modify, merge, publish, distribute, sublicense, and/or -sell copies of the Software, and to permit persons to whom the Software is -furnished to do so, subject to the following conditions: - -The above copyright notice and this permission notice shall be included in -all copies or substantial portions of the Software. - -THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE -AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING -FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS -IN THE SOFTWARE. -""" - -import functools -import itertools - - -def _nonblank(str): - return str and not str.startswith("#") - - -@functools.singledispatch -def yield_lines(iterable): - r""" - Yield valid lines of a string or iterable. - - >>> list(yield_lines('')) - [] - >>> list(yield_lines(['foo', 'bar'])) - ['foo', 'bar'] - >>> list(yield_lines('foo\nbar')) - ['foo', 'bar'] - >>> list(yield_lines('\nfoo\n#bar\nbaz #comment')) - ['foo', 'baz #comment'] - >>> list(yield_lines(['foo\nbar', 'baz', 'bing\n\n\n'])) - ['foo', 'bar', 'baz', 'bing'] - """ - return itertools.chain.from_iterable(map(yield_lines, iterable)) - - -@yield_lines.register(str) -def _(text): - return filter(_nonblank, map(str.strip, text.splitlines())) - - -def drop_comment(line): - """ - Drop comments. - - >>> drop_comment('foo # bar') - 'foo' - - A hash without a space may be in a URL. - - >>> drop_comment('http://example.com/foo#bar') - 'http://example.com/foo#bar' - """ - return line.partition(" #")[0] - - -def join_continuation(lines): - r""" - Join lines continued by a trailing backslash. - - >>> list(join_continuation(['foo \\', 'bar', 'baz'])) - ['foobar', 'baz'] - >>> list(join_continuation(['foo \\', 'bar', 'baz'])) - ['foobar', 'baz'] - >>> list(join_continuation(['foo \\', 'bar \\', 'baz'])) - ['foobarbaz'] - - Not sure why, but... - The character preceeding the backslash is also elided. - - >>> list(join_continuation(['goo\\', 'dly'])) - ['godly'] - - A terrible idea, but... - If no line is available to continue, suppress the lines. - - >>> list(join_continuation(['foo', 'bar\\', 'baz\\'])) - ['foo'] - """ - lines = iter(lines) - for item in lines: - while item.endswith("\\"): - try: - item = item[:-2].strip() + next(lines) - except StopIteration: - return - yield item diff --git a/venv/lib/python3.11/site-packages/pip/_internal/utils/_log.py b/venv/lib/python3.11/site-packages/pip/_internal/utils/_log.py deleted file mode 100644 index 92c4c6a..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/utils/_log.py +++ /dev/null @@ -1,38 +0,0 @@ -"""Customize logging - -Defines custom logger class for the `logger.verbose(...)` method. - -init_logging() must be called before any other modules that call logging.getLogger. -""" - -import logging -from typing import Any, cast - -# custom log level for `--verbose` output -# between DEBUG and INFO -VERBOSE = 15 - - -class VerboseLogger(logging.Logger): - """Custom Logger, defining a verbose log-level - - VERBOSE is between INFO and DEBUG. - """ - - def verbose(self, msg: str, *args: Any, **kwargs: Any) -> None: - return self.log(VERBOSE, msg, *args, **kwargs) - - -def getLogger(name: str) -> VerboseLogger: - """logging.getLogger, but ensures our VerboseLogger class is returned""" - return cast(VerboseLogger, logging.getLogger(name)) - - -def init_logging() -> None: - """Register our VerboseLogger and VERBOSE log level. - - Should be called before any calls to getLogger(), - i.e. in pip._internal.__init__ - """ - logging.setLoggerClass(VerboseLogger) - logging.addLevelName(VERBOSE, "VERBOSE") diff --git a/venv/lib/python3.11/site-packages/pip/_internal/utils/appdirs.py b/venv/lib/python3.11/site-packages/pip/_internal/utils/appdirs.py deleted file mode 100644 index 16933bf..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/utils/appdirs.py +++ /dev/null @@ -1,52 +0,0 @@ -""" -This code wraps the vendored appdirs module to so the return values are -compatible for the current pip code base. - -The intention is to rewrite current usages gradually, keeping the tests pass, -and eventually drop this after all usages are changed. -""" - -import os -import sys -from typing import List - -from pip._vendor import platformdirs as _appdirs - - -def user_cache_dir(appname: str) -> str: - return _appdirs.user_cache_dir(appname, appauthor=False) - - -def _macos_user_config_dir(appname: str, roaming: bool = True) -> str: - # Use ~/Application Support/pip, if the directory exists. - path = _appdirs.user_data_dir(appname, appauthor=False, roaming=roaming) - if os.path.isdir(path): - return path - - # Use a Linux-like ~/.config/pip, by default. - linux_like_path = "~/.config/" - if appname: - linux_like_path = os.path.join(linux_like_path, appname) - - return os.path.expanduser(linux_like_path) - - -def user_config_dir(appname: str, roaming: bool = True) -> str: - if sys.platform == "darwin": - return _macos_user_config_dir(appname, roaming) - - return _appdirs.user_config_dir(appname, appauthor=False, roaming=roaming) - - -# for the discussion regarding site_config_dir locations -# see -def site_config_dirs(appname: str) -> List[str]: - if sys.platform == "darwin": - return [_appdirs.site_data_dir(appname, appauthor=False, multipath=True)] - - dirval = _appdirs.site_config_dir(appname, appauthor=False, multipath=True) - if sys.platform == "win32": - return [dirval] - - # Unix-y system. Look in /etc as well. - return dirval.split(os.pathsep) + ["/etc"] diff --git a/venv/lib/python3.11/site-packages/pip/_internal/utils/compat.py b/venv/lib/python3.11/site-packages/pip/_internal/utils/compat.py deleted file mode 100644 index 3f4d300..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/utils/compat.py +++ /dev/null @@ -1,63 +0,0 @@ -"""Stuff that differs in different Python versions and platform -distributions.""" - -import logging -import os -import sys - -__all__ = ["get_path_uid", "stdlib_pkgs", "WINDOWS"] - - -logger = logging.getLogger(__name__) - - -def has_tls() -> bool: - try: - import _ssl # noqa: F401 # ignore unused - - return True - except ImportError: - pass - - from pip._vendor.urllib3.util import IS_PYOPENSSL - - return IS_PYOPENSSL - - -def get_path_uid(path: str) -> int: - """ - Return path's uid. - - Does not follow symlinks: - https://github.com/pypa/pip/pull/935#discussion_r5307003 - - Placed this function in compat due to differences on AIX and - Jython, that should eventually go away. - - :raises OSError: When path is a symlink or can't be read. - """ - if hasattr(os, "O_NOFOLLOW"): - fd = os.open(path, os.O_RDONLY | os.O_NOFOLLOW) - file_uid = os.fstat(fd).st_uid - os.close(fd) - else: # AIX and Jython - # WARNING: time of check vulnerability, but best we can do w/o NOFOLLOW - if not os.path.islink(path): - # older versions of Jython don't have `os.fstat` - file_uid = os.stat(path).st_uid - else: - # raise OSError for parity with os.O_NOFOLLOW above - raise OSError(f"{path} is a symlink; Will not return uid for symlinks") - return file_uid - - -# packages in the stdlib that may have installation metadata, but should not be -# considered 'installed'. this theoretically could be determined based on -# dist.location (py27:`sysconfig.get_paths()['stdlib']`, -# py26:sysconfig.get_config_vars('LIBDEST')), but fear platform variation may -# make this ineffective, so hard-coding -stdlib_pkgs = {"python", "wsgiref", "argparse"} - - -# windows detection, covers cpython and ironpython -WINDOWS = sys.platform.startswith("win") or (sys.platform == "cli" and os.name == "nt") diff --git a/venv/lib/python3.11/site-packages/pip/_internal/utils/compatibility_tags.py b/venv/lib/python3.11/site-packages/pip/_internal/utils/compatibility_tags.py deleted file mode 100644 index b6ed9a7..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/utils/compatibility_tags.py +++ /dev/null @@ -1,165 +0,0 @@ -"""Generate and work with PEP 425 Compatibility Tags. -""" - -import re -from typing import List, Optional, Tuple - -from pip._vendor.packaging.tags import ( - PythonVersion, - Tag, - compatible_tags, - cpython_tags, - generic_tags, - interpreter_name, - interpreter_version, - mac_platforms, -) - -_osx_arch_pat = re.compile(r"(.+)_(\d+)_(\d+)_(.+)") - - -def version_info_to_nodot(version_info: Tuple[int, ...]) -> str: - # Only use up to the first two numbers. - return "".join(map(str, version_info[:2])) - - -def _mac_platforms(arch: str) -> List[str]: - match = _osx_arch_pat.match(arch) - if match: - name, major, minor, actual_arch = match.groups() - mac_version = (int(major), int(minor)) - arches = [ - # Since we have always only checked that the platform starts - # with "macosx", for backwards-compatibility we extract the - # actual prefix provided by the user in case they provided - # something like "macosxcustom_". It may be good to remove - # this as undocumented or deprecate it in the future. - "{}_{}".format(name, arch[len("macosx_") :]) - for arch in mac_platforms(mac_version, actual_arch) - ] - else: - # arch pattern didn't match (?!) - arches = [arch] - return arches - - -def _custom_manylinux_platforms(arch: str) -> List[str]: - arches = [arch] - arch_prefix, arch_sep, arch_suffix = arch.partition("_") - if arch_prefix == "manylinux2014": - # manylinux1/manylinux2010 wheels run on most manylinux2014 systems - # with the exception of wheels depending on ncurses. PEP 599 states - # manylinux1/manylinux2010 wheels should be considered - # manylinux2014 wheels: - # https://www.python.org/dev/peps/pep-0599/#backwards-compatibility-with-manylinux2010-wheels - if arch_suffix in {"i686", "x86_64"}: - arches.append("manylinux2010" + arch_sep + arch_suffix) - arches.append("manylinux1" + arch_sep + arch_suffix) - elif arch_prefix == "manylinux2010": - # manylinux1 wheels run on most manylinux2010 systems with the - # exception of wheels depending on ncurses. PEP 571 states - # manylinux1 wheels should be considered manylinux2010 wheels: - # https://www.python.org/dev/peps/pep-0571/#backwards-compatibility-with-manylinux1-wheels - arches.append("manylinux1" + arch_sep + arch_suffix) - return arches - - -def _get_custom_platforms(arch: str) -> List[str]: - arch_prefix, arch_sep, arch_suffix = arch.partition("_") - if arch.startswith("macosx"): - arches = _mac_platforms(arch) - elif arch_prefix in ["manylinux2014", "manylinux2010"]: - arches = _custom_manylinux_platforms(arch) - else: - arches = [arch] - return arches - - -def _expand_allowed_platforms(platforms: Optional[List[str]]) -> Optional[List[str]]: - if not platforms: - return None - - seen = set() - result = [] - - for p in platforms: - if p in seen: - continue - additions = [c for c in _get_custom_platforms(p) if c not in seen] - seen.update(additions) - result.extend(additions) - - return result - - -def _get_python_version(version: str) -> PythonVersion: - if len(version) > 1: - return int(version[0]), int(version[1:]) - else: - return (int(version[0]),) - - -def _get_custom_interpreter( - implementation: Optional[str] = None, version: Optional[str] = None -) -> str: - if implementation is None: - implementation = interpreter_name() - if version is None: - version = interpreter_version() - return f"{implementation}{version}" - - -def get_supported( - version: Optional[str] = None, - platforms: Optional[List[str]] = None, - impl: Optional[str] = None, - abis: Optional[List[str]] = None, -) -> List[Tag]: - """Return a list of supported tags for each version specified in - `versions`. - - :param version: a string version, of the form "33" or "32", - or None. The version will be assumed to support our ABI. - :param platform: specify a list of platforms you want valid - tags for, or None. If None, use the local system platform. - :param impl: specify the exact implementation you want valid - tags for, or None. If None, use the local interpreter impl. - :param abis: specify a list of abis you want valid - tags for, or None. If None, use the local interpreter abi. - """ - supported: List[Tag] = [] - - python_version: Optional[PythonVersion] = None - if version is not None: - python_version = _get_python_version(version) - - interpreter = _get_custom_interpreter(impl, version) - - platforms = _expand_allowed_platforms(platforms) - - is_cpython = (impl or interpreter_name()) == "cp" - if is_cpython: - supported.extend( - cpython_tags( - python_version=python_version, - abis=abis, - platforms=platforms, - ) - ) - else: - supported.extend( - generic_tags( - interpreter=interpreter, - abis=abis, - platforms=platforms, - ) - ) - supported.extend( - compatible_tags( - python_version=python_version, - interpreter=interpreter, - platforms=platforms, - ) - ) - - return supported diff --git a/venv/lib/python3.11/site-packages/pip/_internal/utils/datetime.py b/venv/lib/python3.11/site-packages/pip/_internal/utils/datetime.py deleted file mode 100644 index 8668b3b..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/utils/datetime.py +++ /dev/null @@ -1,11 +0,0 @@ -"""For when pip wants to check the date or time. -""" - -import datetime - - -def today_is_later_than(year: int, month: int, day: int) -> bool: - today = datetime.date.today() - given = datetime.date(year, month, day) - - return today > given diff --git a/venv/lib/python3.11/site-packages/pip/_internal/utils/deprecation.py b/venv/lib/python3.11/site-packages/pip/_internal/utils/deprecation.py deleted file mode 100644 index 72bd6f2..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/utils/deprecation.py +++ /dev/null @@ -1,120 +0,0 @@ -""" -A module that implements tooling to enable easy warnings about deprecations. -""" - -import logging -import warnings -from typing import Any, Optional, TextIO, Type, Union - -from pip._vendor.packaging.version import parse - -from pip import __version__ as current_version # NOTE: tests patch this name. - -DEPRECATION_MSG_PREFIX = "DEPRECATION: " - - -class PipDeprecationWarning(Warning): - pass - - -_original_showwarning: Any = None - - -# Warnings <-> Logging Integration -def _showwarning( - message: Union[Warning, str], - category: Type[Warning], - filename: str, - lineno: int, - file: Optional[TextIO] = None, - line: Optional[str] = None, -) -> None: - if file is not None: - if _original_showwarning is not None: - _original_showwarning(message, category, filename, lineno, file, line) - elif issubclass(category, PipDeprecationWarning): - # We use a specially named logger which will handle all of the - # deprecation messages for pip. - logger = logging.getLogger("pip._internal.deprecations") - logger.warning(message) - else: - _original_showwarning(message, category, filename, lineno, file, line) - - -def install_warning_logger() -> None: - # Enable our Deprecation Warnings - warnings.simplefilter("default", PipDeprecationWarning, append=True) - - global _original_showwarning - - if _original_showwarning is None: - _original_showwarning = warnings.showwarning - warnings.showwarning = _showwarning - - -def deprecated( - *, - reason: str, - replacement: Optional[str], - gone_in: Optional[str], - feature_flag: Optional[str] = None, - issue: Optional[int] = None, -) -> None: - """Helper to deprecate existing functionality. - - reason: - Textual reason shown to the user about why this functionality has - been deprecated. Should be a complete sentence. - replacement: - Textual suggestion shown to the user about what alternative - functionality they can use. - gone_in: - The version of pip does this functionality should get removed in. - Raises an error if pip's current version is greater than or equal to - this. - feature_flag: - Command-line flag of the form --use-feature={feature_flag} for testing - upcoming functionality. - issue: - Issue number on the tracker that would serve as a useful place for - users to find related discussion and provide feedback. - """ - - # Determine whether or not the feature is already gone in this version. - is_gone = gone_in is not None and parse(current_version) >= parse(gone_in) - - message_parts = [ - (reason, f"{DEPRECATION_MSG_PREFIX}{{}}"), - ( - gone_in, - "pip {} will enforce this behaviour change." - if not is_gone - else "Since pip {}, this is no longer supported.", - ), - ( - replacement, - "A possible replacement is {}.", - ), - ( - feature_flag, - "You can use the flag --use-feature={} to test the upcoming behaviour." - if not is_gone - else None, - ), - ( - issue, - "Discussion can be found at https://github.com/pypa/pip/issues/{}", - ), - ] - - message = " ".join( - format_str.format(value) - for value, format_str in message_parts - if format_str is not None and value is not None - ) - - # Raise as an error if this behaviour is deprecated. - if is_gone: - raise PipDeprecationWarning(message) - - warnings.warn(message, category=PipDeprecationWarning, stacklevel=2) diff --git a/venv/lib/python3.11/site-packages/pip/_internal/utils/direct_url_helpers.py b/venv/lib/python3.11/site-packages/pip/_internal/utils/direct_url_helpers.py deleted file mode 100644 index 0e8e5e1..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/utils/direct_url_helpers.py +++ /dev/null @@ -1,87 +0,0 @@ -from typing import Optional - -from pip._internal.models.direct_url import ArchiveInfo, DirectUrl, DirInfo, VcsInfo -from pip._internal.models.link import Link -from pip._internal.utils.urls import path_to_url -from pip._internal.vcs import vcs - - -def direct_url_as_pep440_direct_reference(direct_url: DirectUrl, name: str) -> str: - """Convert a DirectUrl to a pip requirement string.""" - direct_url.validate() # if invalid, this is a pip bug - requirement = name + " @ " - fragments = [] - if isinstance(direct_url.info, VcsInfo): - requirement += "{}+{}@{}".format( - direct_url.info.vcs, direct_url.url, direct_url.info.commit_id - ) - elif isinstance(direct_url.info, ArchiveInfo): - requirement += direct_url.url - if direct_url.info.hash: - fragments.append(direct_url.info.hash) - else: - assert isinstance(direct_url.info, DirInfo) - requirement += direct_url.url - if direct_url.subdirectory: - fragments.append("subdirectory=" + direct_url.subdirectory) - if fragments: - requirement += "#" + "&".join(fragments) - return requirement - - -def direct_url_for_editable(source_dir: str) -> DirectUrl: - return DirectUrl( - url=path_to_url(source_dir), - info=DirInfo(editable=True), - ) - - -def direct_url_from_link( - link: Link, source_dir: Optional[str] = None, link_is_in_wheel_cache: bool = False -) -> DirectUrl: - if link.is_vcs: - vcs_backend = vcs.get_backend_for_scheme(link.scheme) - assert vcs_backend - url, requested_revision, _ = vcs_backend.get_url_rev_and_auth( - link.url_without_fragment - ) - # For VCS links, we need to find out and add commit_id. - if link_is_in_wheel_cache: - # If the requested VCS link corresponds to a cached - # wheel, it means the requested revision was an - # immutable commit hash, otherwise it would not have - # been cached. In that case we don't have a source_dir - # with the VCS checkout. - assert requested_revision - commit_id = requested_revision - else: - # If the wheel was not in cache, it means we have - # had to checkout from VCS to build and we have a source_dir - # which we can inspect to find out the commit id. - assert source_dir - commit_id = vcs_backend.get_revision(source_dir) - return DirectUrl( - url=url, - info=VcsInfo( - vcs=vcs_backend.name, - commit_id=commit_id, - requested_revision=requested_revision, - ), - subdirectory=link.subdirectory_fragment, - ) - elif link.is_existing_dir(): - return DirectUrl( - url=link.url_without_fragment, - info=DirInfo(), - subdirectory=link.subdirectory_fragment, - ) - else: - hash = None - hash_name = link.hash_name - if hash_name: - hash = f"{hash_name}={link.hash}" - return DirectUrl( - url=link.url_without_fragment, - info=ArchiveInfo(hash=hash), - subdirectory=link.subdirectory_fragment, - ) diff --git a/venv/lib/python3.11/site-packages/pip/_internal/utils/egg_link.py b/venv/lib/python3.11/site-packages/pip/_internal/utils/egg_link.py deleted file mode 100644 index eb57ed1..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/utils/egg_link.py +++ /dev/null @@ -1,72 +0,0 @@ -import os -import re -import sys -from typing import List, Optional - -from pip._internal.locations import site_packages, user_site -from pip._internal.utils.virtualenv import ( - running_under_virtualenv, - virtualenv_no_global, -) - -__all__ = [ - "egg_link_path_from_sys_path", - "egg_link_path_from_location", -] - - -def _egg_link_name(raw_name: str) -> str: - """ - Convert a Name metadata value to a .egg-link name, by applying - the same substitution as pkg_resources's safe_name function. - Note: we cannot use canonicalize_name because it has a different logic. - """ - return re.sub("[^A-Za-z0-9.]+", "-", raw_name) + ".egg-link" - - -def egg_link_path_from_sys_path(raw_name: str) -> Optional[str]: - """ - Look for a .egg-link file for project name, by walking sys.path. - """ - egg_link_name = _egg_link_name(raw_name) - for path_item in sys.path: - egg_link = os.path.join(path_item, egg_link_name) - if os.path.isfile(egg_link): - return egg_link - return None - - -def egg_link_path_from_location(raw_name: str) -> Optional[str]: - """ - Return the path for the .egg-link file if it exists, otherwise, None. - - There's 3 scenarios: - 1) not in a virtualenv - try to find in site.USER_SITE, then site_packages - 2) in a no-global virtualenv - try to find in site_packages - 3) in a yes-global virtualenv - try to find in site_packages, then site.USER_SITE - (don't look in global location) - - For #1 and #3, there could be odd cases, where there's an egg-link in 2 - locations. - - This method will just return the first one found. - """ - sites: List[str] = [] - if running_under_virtualenv(): - sites.append(site_packages) - if not virtualenv_no_global() and user_site: - sites.append(user_site) - else: - if user_site: - sites.append(user_site) - sites.append(site_packages) - - egg_link_name = _egg_link_name(raw_name) - for site in sites: - egglink = os.path.join(site, egg_link_name) - if os.path.isfile(egglink): - return egglink - return None diff --git a/venv/lib/python3.11/site-packages/pip/_internal/utils/encoding.py b/venv/lib/python3.11/site-packages/pip/_internal/utils/encoding.py deleted file mode 100644 index 008f06a..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/utils/encoding.py +++ /dev/null @@ -1,36 +0,0 @@ -import codecs -import locale -import re -import sys -from typing import List, Tuple - -BOMS: List[Tuple[bytes, str]] = [ - (codecs.BOM_UTF8, "utf-8"), - (codecs.BOM_UTF16, "utf-16"), - (codecs.BOM_UTF16_BE, "utf-16-be"), - (codecs.BOM_UTF16_LE, "utf-16-le"), - (codecs.BOM_UTF32, "utf-32"), - (codecs.BOM_UTF32_BE, "utf-32-be"), - (codecs.BOM_UTF32_LE, "utf-32-le"), -] - -ENCODING_RE = re.compile(rb"coding[:=]\s*([-\w.]+)") - - -def auto_decode(data: bytes) -> str: - """Check a bytes string for a BOM to correctly detect the encoding - - Fallback to locale.getpreferredencoding(False) like open() on Python3""" - for bom, encoding in BOMS: - if data.startswith(bom): - return data[len(bom) :].decode(encoding) - # Lets check the first two lines as in PEP263 - for line in data.split(b"\n")[:2]: - if line[0:1] == b"#" and ENCODING_RE.search(line): - result = ENCODING_RE.search(line) - assert result is not None - encoding = result.groups()[0].decode("ascii") - return data.decode(encoding) - return data.decode( - locale.getpreferredencoding(False) or sys.getdefaultencoding(), - ) diff --git a/venv/lib/python3.11/site-packages/pip/_internal/utils/entrypoints.py b/venv/lib/python3.11/site-packages/pip/_internal/utils/entrypoints.py deleted file mode 100644 index 1501369..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/utils/entrypoints.py +++ /dev/null @@ -1,84 +0,0 @@ -import itertools -import os -import shutil -import sys -from typing import List, Optional - -from pip._internal.cli.main import main -from pip._internal.utils.compat import WINDOWS - -_EXECUTABLE_NAMES = [ - "pip", - f"pip{sys.version_info.major}", - f"pip{sys.version_info.major}.{sys.version_info.minor}", -] -if WINDOWS: - _allowed_extensions = {"", ".exe"} - _EXECUTABLE_NAMES = [ - "".join(parts) - for parts in itertools.product(_EXECUTABLE_NAMES, _allowed_extensions) - ] - - -def _wrapper(args: Optional[List[str]] = None) -> int: - """Central wrapper for all old entrypoints. - - Historically pip has had several entrypoints defined. Because of issues - arising from PATH, sys.path, multiple Pythons, their interactions, and most - of them having a pip installed, users suffer every time an entrypoint gets - moved. - - To alleviate this pain, and provide a mechanism for warning users and - directing them to an appropriate place for help, we now define all of - our old entrypoints as wrappers for the current one. - """ - sys.stderr.write( - "WARNING: pip is being invoked by an old script wrapper. This will " - "fail in a future version of pip.\n" - "Please see https://github.com/pypa/pip/issues/5599 for advice on " - "fixing the underlying issue.\n" - "To avoid this problem you can invoke Python with '-m pip' instead of " - "running pip directly.\n" - ) - return main(args) - - -def get_best_invocation_for_this_pip() -> str: - """Try to figure out the best way to invoke pip in the current environment.""" - binary_directory = "Scripts" if WINDOWS else "bin" - binary_prefix = os.path.join(sys.prefix, binary_directory) - - # Try to use pip[X[.Y]] names, if those executables for this environment are - # the first on PATH with that name. - path_parts = os.path.normcase(os.environ.get("PATH", "")).split(os.pathsep) - exe_are_in_PATH = os.path.normcase(binary_prefix) in path_parts - if exe_are_in_PATH: - for exe_name in _EXECUTABLE_NAMES: - found_executable = shutil.which(exe_name) - binary_executable = os.path.join(binary_prefix, exe_name) - if ( - found_executable - and os.path.exists(binary_executable) - and os.path.samefile( - found_executable, - binary_executable, - ) - ): - return exe_name - - # Use the `-m` invocation, if there's no "nice" invocation. - return f"{get_best_invocation_for_this_python()} -m pip" - - -def get_best_invocation_for_this_python() -> str: - """Try to figure out the best way to invoke the current Python.""" - exe = sys.executable - exe_name = os.path.basename(exe) - - # Try to use the basename, if it's the first executable. - found_executable = shutil.which(exe_name) - if found_executable and os.path.samefile(found_executable, exe): - return exe_name - - # Use the full executable name, because we couldn't find something simpler. - return exe diff --git a/venv/lib/python3.11/site-packages/pip/_internal/utils/filesystem.py b/venv/lib/python3.11/site-packages/pip/_internal/utils/filesystem.py deleted file mode 100644 index 83c2df7..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/utils/filesystem.py +++ /dev/null @@ -1,153 +0,0 @@ -import fnmatch -import os -import os.path -import random -import sys -from contextlib import contextmanager -from tempfile import NamedTemporaryFile -from typing import Any, BinaryIO, Generator, List, Union, cast - -from pip._vendor.tenacity import retry, stop_after_delay, wait_fixed - -from pip._internal.utils.compat import get_path_uid -from pip._internal.utils.misc import format_size - - -def check_path_owner(path: str) -> bool: - # If we don't have a way to check the effective uid of this process, then - # we'll just assume that we own the directory. - if sys.platform == "win32" or not hasattr(os, "geteuid"): - return True - - assert os.path.isabs(path) - - previous = None - while path != previous: - if os.path.lexists(path): - # Check if path is writable by current user. - if os.geteuid() == 0: - # Special handling for root user in order to handle properly - # cases where users use sudo without -H flag. - try: - path_uid = get_path_uid(path) - except OSError: - return False - return path_uid == 0 - else: - return os.access(path, os.W_OK) - else: - previous, path = path, os.path.dirname(path) - return False # assume we don't own the path - - -@contextmanager -def adjacent_tmp_file(path: str, **kwargs: Any) -> Generator[BinaryIO, None, None]: - """Return a file-like object pointing to a tmp file next to path. - - The file is created securely and is ensured to be written to disk - after the context reaches its end. - - kwargs will be passed to tempfile.NamedTemporaryFile to control - the way the temporary file will be opened. - """ - with NamedTemporaryFile( - delete=False, - dir=os.path.dirname(path), - prefix=os.path.basename(path), - suffix=".tmp", - **kwargs, - ) as f: - result = cast(BinaryIO, f) - try: - yield result - finally: - result.flush() - os.fsync(result.fileno()) - - -# Tenacity raises RetryError by default, explicitly raise the original exception -_replace_retry = retry(reraise=True, stop=stop_after_delay(1), wait=wait_fixed(0.25)) - -replace = _replace_retry(os.replace) - - -# test_writable_dir and _test_writable_dir_win are copied from Flit, -# with the author's agreement to also place them under pip's license. -def test_writable_dir(path: str) -> bool: - """Check if a directory is writable. - - Uses os.access() on POSIX, tries creating files on Windows. - """ - # If the directory doesn't exist, find the closest parent that does. - while not os.path.isdir(path): - parent = os.path.dirname(path) - if parent == path: - break # Should never get here, but infinite loops are bad - path = parent - - if os.name == "posix": - return os.access(path, os.W_OK) - - return _test_writable_dir_win(path) - - -def _test_writable_dir_win(path: str) -> bool: - # os.access doesn't work on Windows: http://bugs.python.org/issue2528 - # and we can't use tempfile: http://bugs.python.org/issue22107 - basename = "accesstest_deleteme_fishfingers_custard_" - alphabet = "abcdefghijklmnopqrstuvwxyz0123456789" - for _ in range(10): - name = basename + "".join(random.choice(alphabet) for _ in range(6)) - file = os.path.join(path, name) - try: - fd = os.open(file, os.O_RDWR | os.O_CREAT | os.O_EXCL) - except FileExistsError: - pass - except PermissionError: - # This could be because there's a directory with the same name. - # But it's highly unlikely there's a directory called that, - # so we'll assume it's because the parent dir is not writable. - # This could as well be because the parent dir is not readable, - # due to non-privileged user access. - return False - else: - os.close(fd) - os.unlink(file) - return True - - # This should never be reached - raise OSError("Unexpected condition testing for writable directory") - - -def find_files(path: str, pattern: str) -> List[str]: - """Returns a list of absolute paths of files beneath path, recursively, - with filenames which match the UNIX-style shell glob pattern.""" - result: List[str] = [] - for root, _, files in os.walk(path): - matches = fnmatch.filter(files, pattern) - result.extend(os.path.join(root, f) for f in matches) - return result - - -def file_size(path: str) -> Union[int, float]: - # If it's a symlink, return 0. - if os.path.islink(path): - return 0 - return os.path.getsize(path) - - -def format_file_size(path: str) -> str: - return format_size(file_size(path)) - - -def directory_size(path: str) -> Union[int, float]: - size = 0.0 - for root, _dirs, files in os.walk(path): - for filename in files: - file_path = os.path.join(root, filename) - size += file_size(file_path) - return size - - -def format_directory_size(path: str) -> str: - return format_size(directory_size(path)) diff --git a/venv/lib/python3.11/site-packages/pip/_internal/utils/filetypes.py b/venv/lib/python3.11/site-packages/pip/_internal/utils/filetypes.py deleted file mode 100644 index 5948570..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/utils/filetypes.py +++ /dev/null @@ -1,27 +0,0 @@ -"""Filetype information. -""" - -from typing import Tuple - -from pip._internal.utils.misc import splitext - -WHEEL_EXTENSION = ".whl" -BZ2_EXTENSIONS: Tuple[str, ...] = (".tar.bz2", ".tbz") -XZ_EXTENSIONS: Tuple[str, ...] = ( - ".tar.xz", - ".txz", - ".tlz", - ".tar.lz", - ".tar.lzma", -) -ZIP_EXTENSIONS: Tuple[str, ...] = (".zip", WHEEL_EXTENSION) -TAR_EXTENSIONS: Tuple[str, ...] = (".tar.gz", ".tgz", ".tar") -ARCHIVE_EXTENSIONS = ZIP_EXTENSIONS + BZ2_EXTENSIONS + TAR_EXTENSIONS + XZ_EXTENSIONS - - -def is_archive_file(name: str) -> bool: - """Return True if `name` is a considered as an archive file.""" - ext = splitext(name)[1].lower() - if ext in ARCHIVE_EXTENSIONS: - return True - return False diff --git a/venv/lib/python3.11/site-packages/pip/_internal/utils/glibc.py b/venv/lib/python3.11/site-packages/pip/_internal/utils/glibc.py deleted file mode 100644 index 81342af..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/utils/glibc.py +++ /dev/null @@ -1,88 +0,0 @@ -import os -import sys -from typing import Optional, Tuple - - -def glibc_version_string() -> Optional[str]: - "Returns glibc version string, or None if not using glibc." - return glibc_version_string_confstr() or glibc_version_string_ctypes() - - -def glibc_version_string_confstr() -> Optional[str]: - "Primary implementation of glibc_version_string using os.confstr." - # os.confstr is quite a bit faster than ctypes.DLL. It's also less likely - # to be broken or missing. This strategy is used in the standard library - # platform module: - # https://github.com/python/cpython/blob/fcf1d003bf4f0100c9d0921ff3d70e1127ca1b71/Lib/platform.py#L175-L183 - if sys.platform == "win32": - return None - try: - gnu_libc_version = os.confstr("CS_GNU_LIBC_VERSION") - if gnu_libc_version is None: - return None - # os.confstr("CS_GNU_LIBC_VERSION") returns a string like "glibc 2.17": - _, version = gnu_libc_version.split() - except (AttributeError, OSError, ValueError): - # os.confstr() or CS_GNU_LIBC_VERSION not available (or a bad value)... - return None - return version - - -def glibc_version_string_ctypes() -> Optional[str]: - "Fallback implementation of glibc_version_string using ctypes." - - try: - import ctypes - except ImportError: - return None - - # ctypes.CDLL(None) internally calls dlopen(NULL), and as the dlopen - # manpage says, "If filename is NULL, then the returned handle is for the - # main program". This way we can let the linker do the work to figure out - # which libc our process is actually using. - process_namespace = ctypes.CDLL(None) - try: - gnu_get_libc_version = process_namespace.gnu_get_libc_version - except AttributeError: - # Symbol doesn't exist -> therefore, we are not linked to - # glibc. - return None - - # Call gnu_get_libc_version, which returns a string like "2.5" - gnu_get_libc_version.restype = ctypes.c_char_p - version_str = gnu_get_libc_version() - # py2 / py3 compatibility: - if not isinstance(version_str, str): - version_str = version_str.decode("ascii") - - return version_str - - -# platform.libc_ver regularly returns completely nonsensical glibc -# versions. E.g. on my computer, platform says: -# -# ~$ python2.7 -c 'import platform; print(platform.libc_ver())' -# ('glibc', '2.7') -# ~$ python3.5 -c 'import platform; print(platform.libc_ver())' -# ('glibc', '2.9') -# -# But the truth is: -# -# ~$ ldd --version -# ldd (Debian GLIBC 2.22-11) 2.22 -# -# This is unfortunate, because it means that the linehaul data on libc -# versions that was generated by pip 8.1.2 and earlier is useless and -# misleading. Solution: instead of using platform, use our code that actually -# works. -def libc_ver() -> Tuple[str, str]: - """Try to determine the glibc version - - Returns a tuple of strings (lib, version) which default to empty strings - in case the lookup fails. - """ - glibc_version = glibc_version_string() - if glibc_version is None: - return ("", "") - else: - return ("glibc", glibc_version) diff --git a/venv/lib/python3.11/site-packages/pip/_internal/utils/hashes.py b/venv/lib/python3.11/site-packages/pip/_internal/utils/hashes.py deleted file mode 100644 index 843cffc..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/utils/hashes.py +++ /dev/null @@ -1,151 +0,0 @@ -import hashlib -from typing import TYPE_CHECKING, BinaryIO, Dict, Iterable, List, Optional - -from pip._internal.exceptions import HashMismatch, HashMissing, InstallationError -from pip._internal.utils.misc import read_chunks - -if TYPE_CHECKING: - from hashlib import _Hash - - # NoReturn introduced in 3.6.2; imported only for type checking to maintain - # pip compatibility with older patch versions of Python 3.6 - from typing import NoReturn - - -# The recommended hash algo of the moment. Change this whenever the state of -# the art changes; it won't hurt backward compatibility. -FAVORITE_HASH = "sha256" - - -# Names of hashlib algorithms allowed by the --hash option and ``pip hash`` -# Currently, those are the ones at least as collision-resistant as sha256. -STRONG_HASHES = ["sha256", "sha384", "sha512"] - - -class Hashes: - """A wrapper that builds multiple hashes at once and checks them against - known-good values - - """ - - def __init__(self, hashes: Optional[Dict[str, List[str]]] = None) -> None: - """ - :param hashes: A dict of algorithm names pointing to lists of allowed - hex digests - """ - allowed = {} - if hashes is not None: - for alg, keys in hashes.items(): - # Make sure values are always sorted (to ease equality checks) - allowed[alg] = sorted(keys) - self._allowed = allowed - - def __and__(self, other: "Hashes") -> "Hashes": - if not isinstance(other, Hashes): - return NotImplemented - - # If either of the Hashes object is entirely empty (i.e. no hash - # specified at all), all hashes from the other object are allowed. - if not other: - return self - if not self: - return other - - # Otherwise only hashes that present in both objects are allowed. - new = {} - for alg, values in other._allowed.items(): - if alg not in self._allowed: - continue - new[alg] = [v for v in values if v in self._allowed[alg]] - return Hashes(new) - - @property - def digest_count(self) -> int: - return sum(len(digests) for digests in self._allowed.values()) - - def is_hash_allowed(self, hash_name: str, hex_digest: str) -> bool: - """Return whether the given hex digest is allowed.""" - return hex_digest in self._allowed.get(hash_name, []) - - def check_against_chunks(self, chunks: Iterable[bytes]) -> None: - """Check good hashes against ones built from iterable of chunks of - data. - - Raise HashMismatch if none match. - - """ - gots = {} - for hash_name in self._allowed.keys(): - try: - gots[hash_name] = hashlib.new(hash_name) - except (ValueError, TypeError): - raise InstallationError(f"Unknown hash name: {hash_name}") - - for chunk in chunks: - for hash in gots.values(): - hash.update(chunk) - - for hash_name, got in gots.items(): - if got.hexdigest() in self._allowed[hash_name]: - return - self._raise(gots) - - def _raise(self, gots: Dict[str, "_Hash"]) -> "NoReturn": - raise HashMismatch(self._allowed, gots) - - def check_against_file(self, file: BinaryIO) -> None: - """Check good hashes against a file-like object - - Raise HashMismatch if none match. - - """ - return self.check_against_chunks(read_chunks(file)) - - def check_against_path(self, path: str) -> None: - with open(path, "rb") as file: - return self.check_against_file(file) - - def has_one_of(self, hashes: Dict[str, str]) -> bool: - """Return whether any of the given hashes are allowed.""" - for hash_name, hex_digest in hashes.items(): - if self.is_hash_allowed(hash_name, hex_digest): - return True - return False - - def __bool__(self) -> bool: - """Return whether I know any known-good hashes.""" - return bool(self._allowed) - - def __eq__(self, other: object) -> bool: - if not isinstance(other, Hashes): - return NotImplemented - return self._allowed == other._allowed - - def __hash__(self) -> int: - return hash( - ",".join( - sorted( - ":".join((alg, digest)) - for alg, digest_list in self._allowed.items() - for digest in digest_list - ) - ) - ) - - -class MissingHashes(Hashes): - """A workalike for Hashes used when we're missing a hash for a requirement - - It computes the actual hash of the requirement and raises a HashMissing - exception showing it to the user. - - """ - - def __init__(self) -> None: - """Don't offer the ``hashes`` kwarg.""" - # Pass our favorite hash in to generate a "gotten hash". With the - # empty list, it will never match, so an error will always raise. - super().__init__(hashes={FAVORITE_HASH: []}) - - def _raise(self, gots: Dict[str, "_Hash"]) -> "NoReturn": - raise HashMissing(gots[FAVORITE_HASH].hexdigest()) diff --git a/venv/lib/python3.11/site-packages/pip/_internal/utils/logging.py b/venv/lib/python3.11/site-packages/pip/_internal/utils/logging.py deleted file mode 100644 index 95982df..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/utils/logging.py +++ /dev/null @@ -1,348 +0,0 @@ -import contextlib -import errno -import logging -import logging.handlers -import os -import sys -import threading -from dataclasses import dataclass -from io import TextIOWrapper -from logging import Filter -from typing import Any, ClassVar, Generator, List, Optional, TextIO, Type - -from pip._vendor.rich.console import ( - Console, - ConsoleOptions, - ConsoleRenderable, - RenderableType, - RenderResult, - RichCast, -) -from pip._vendor.rich.highlighter import NullHighlighter -from pip._vendor.rich.logging import RichHandler -from pip._vendor.rich.segment import Segment -from pip._vendor.rich.style import Style - -from pip._internal.utils._log import VERBOSE, getLogger -from pip._internal.utils.compat import WINDOWS -from pip._internal.utils.deprecation import DEPRECATION_MSG_PREFIX -from pip._internal.utils.misc import ensure_dir - -_log_state = threading.local() -subprocess_logger = getLogger("pip.subprocessor") - - -class BrokenStdoutLoggingError(Exception): - """ - Raised if BrokenPipeError occurs for the stdout stream while logging. - """ - - -def _is_broken_pipe_error(exc_class: Type[BaseException], exc: BaseException) -> bool: - if exc_class is BrokenPipeError: - return True - - # On Windows, a broken pipe can show up as EINVAL rather than EPIPE: - # https://bugs.python.org/issue19612 - # https://bugs.python.org/issue30418 - if not WINDOWS: - return False - - return isinstance(exc, OSError) and exc.errno in (errno.EINVAL, errno.EPIPE) - - -@contextlib.contextmanager -def indent_log(num: int = 2) -> Generator[None, None, None]: - """ - A context manager which will cause the log output to be indented for any - log messages emitted inside it. - """ - # For thread-safety - _log_state.indentation = get_indentation() - _log_state.indentation += num - try: - yield - finally: - _log_state.indentation -= num - - -def get_indentation() -> int: - return getattr(_log_state, "indentation", 0) - - -class IndentingFormatter(logging.Formatter): - default_time_format = "%Y-%m-%dT%H:%M:%S" - - def __init__( - self, - *args: Any, - add_timestamp: bool = False, - **kwargs: Any, - ) -> None: - """ - A logging.Formatter that obeys the indent_log() context manager. - - :param add_timestamp: A bool indicating output lines should be prefixed - with their record's timestamp. - """ - self.add_timestamp = add_timestamp - super().__init__(*args, **kwargs) - - def get_message_start(self, formatted: str, levelno: int) -> str: - """ - Return the start of the formatted log message (not counting the - prefix to add to each line). - """ - if levelno < logging.WARNING: - return "" - if formatted.startswith(DEPRECATION_MSG_PREFIX): - # Then the message already has a prefix. We don't want it to - # look like "WARNING: DEPRECATION: ...." - return "" - if levelno < logging.ERROR: - return "WARNING: " - - return "ERROR: " - - def format(self, record: logging.LogRecord) -> str: - """ - Calls the standard formatter, but will indent all of the log message - lines by our current indentation level. - """ - formatted = super().format(record) - message_start = self.get_message_start(formatted, record.levelno) - formatted = message_start + formatted - - prefix = "" - if self.add_timestamp: - prefix = f"{self.formatTime(record)} " - prefix += " " * get_indentation() - formatted = "".join([prefix + line for line in formatted.splitlines(True)]) - return formatted - - -@dataclass -class IndentedRenderable: - renderable: RenderableType - indent: int - - def __rich_console__( - self, console: Console, options: ConsoleOptions - ) -> RenderResult: - segments = console.render(self.renderable, options) - lines = Segment.split_lines(segments) - for line in lines: - yield Segment(" " * self.indent) - yield from line - yield Segment("\n") - - -class RichPipStreamHandler(RichHandler): - KEYWORDS: ClassVar[Optional[List[str]]] = [] - - def __init__(self, stream: Optional[TextIO], no_color: bool) -> None: - super().__init__( - console=Console(file=stream, no_color=no_color, soft_wrap=True), - show_time=False, - show_level=False, - show_path=False, - highlighter=NullHighlighter(), - ) - - # Our custom override on Rich's logger, to make things work as we need them to. - def emit(self, record: logging.LogRecord) -> None: - style: Optional[Style] = None - - # If we are given a diagnostic error to present, present it with indentation. - assert isinstance(record.args, tuple) - if getattr(record, "rich", False): - (rich_renderable,) = record.args - assert isinstance( - rich_renderable, (ConsoleRenderable, RichCast, str) - ), f"{rich_renderable} is not rich-console-renderable" - - renderable: RenderableType = IndentedRenderable( - rich_renderable, indent=get_indentation() - ) - else: - message = self.format(record) - renderable = self.render_message(record, message) - if record.levelno is not None: - if record.levelno >= logging.ERROR: - style = Style(color="red") - elif record.levelno >= logging.WARNING: - style = Style(color="yellow") - - try: - self.console.print(renderable, overflow="ignore", crop=False, style=style) - except Exception: - self.handleError(record) - - def handleError(self, record: logging.LogRecord) -> None: - """Called when logging is unable to log some output.""" - - exc_class, exc = sys.exc_info()[:2] - # If a broken pipe occurred while calling write() or flush() on the - # stdout stream in logging's Handler.emit(), then raise our special - # exception so we can handle it in main() instead of logging the - # broken pipe error and continuing. - if ( - exc_class - and exc - and self.console.file is sys.stdout - and _is_broken_pipe_error(exc_class, exc) - ): - raise BrokenStdoutLoggingError() - - return super().handleError(record) - - -class BetterRotatingFileHandler(logging.handlers.RotatingFileHandler): - def _open(self) -> TextIOWrapper: - ensure_dir(os.path.dirname(self.baseFilename)) - return super()._open() - - -class MaxLevelFilter(Filter): - def __init__(self, level: int) -> None: - self.level = level - - def filter(self, record: logging.LogRecord) -> bool: - return record.levelno < self.level - - -class ExcludeLoggerFilter(Filter): - - """ - A logging Filter that excludes records from a logger (or its children). - """ - - def filter(self, record: logging.LogRecord) -> bool: - # The base Filter class allows only records from a logger (or its - # children). - return not super().filter(record) - - -def setup_logging(verbosity: int, no_color: bool, user_log_file: Optional[str]) -> int: - """Configures and sets up all of the logging - - Returns the requested logging level, as its integer value. - """ - - # Determine the level to be logging at. - if verbosity >= 2: - level_number = logging.DEBUG - elif verbosity == 1: - level_number = VERBOSE - elif verbosity == -1: - level_number = logging.WARNING - elif verbosity == -2: - level_number = logging.ERROR - elif verbosity <= -3: - level_number = logging.CRITICAL - else: - level_number = logging.INFO - - level = logging.getLevelName(level_number) - - # The "root" logger should match the "console" level *unless* we also need - # to log to a user log file. - include_user_log = user_log_file is not None - if include_user_log: - additional_log_file = user_log_file - root_level = "DEBUG" - else: - additional_log_file = "/dev/null" - root_level = level - - # Disable any logging besides WARNING unless we have DEBUG level logging - # enabled for vendored libraries. - vendored_log_level = "WARNING" if level in ["INFO", "ERROR"] else "DEBUG" - - # Shorthands for clarity - log_streams = { - "stdout": "ext://sys.stdout", - "stderr": "ext://sys.stderr", - } - handler_classes = { - "stream": "pip._internal.utils.logging.RichPipStreamHandler", - "file": "pip._internal.utils.logging.BetterRotatingFileHandler", - } - handlers = ["console", "console_errors", "console_subprocess"] + ( - ["user_log"] if include_user_log else [] - ) - - logging.config.dictConfig( - { - "version": 1, - "disable_existing_loggers": False, - "filters": { - "exclude_warnings": { - "()": "pip._internal.utils.logging.MaxLevelFilter", - "level": logging.WARNING, - }, - "restrict_to_subprocess": { - "()": "logging.Filter", - "name": subprocess_logger.name, - }, - "exclude_subprocess": { - "()": "pip._internal.utils.logging.ExcludeLoggerFilter", - "name": subprocess_logger.name, - }, - }, - "formatters": { - "indent": { - "()": IndentingFormatter, - "format": "%(message)s", - }, - "indent_with_timestamp": { - "()": IndentingFormatter, - "format": "%(message)s", - "add_timestamp": True, - }, - }, - "handlers": { - "console": { - "level": level, - "class": handler_classes["stream"], - "no_color": no_color, - "stream": log_streams["stdout"], - "filters": ["exclude_subprocess", "exclude_warnings"], - "formatter": "indent", - }, - "console_errors": { - "level": "WARNING", - "class": handler_classes["stream"], - "no_color": no_color, - "stream": log_streams["stderr"], - "filters": ["exclude_subprocess"], - "formatter": "indent", - }, - # A handler responsible for logging to the console messages - # from the "subprocessor" logger. - "console_subprocess": { - "level": level, - "class": handler_classes["stream"], - "stream": log_streams["stderr"], - "no_color": no_color, - "filters": ["restrict_to_subprocess"], - "formatter": "indent", - }, - "user_log": { - "level": "DEBUG", - "class": handler_classes["file"], - "filename": additional_log_file, - "encoding": "utf-8", - "delay": True, - "formatter": "indent_with_timestamp", - }, - }, - "root": { - "level": root_level, - "handlers": handlers, - }, - "loggers": {"pip._vendor": {"level": vendored_log_level}}, - } - ) - - return level_number diff --git a/venv/lib/python3.11/site-packages/pip/_internal/utils/misc.py b/venv/lib/python3.11/site-packages/pip/_internal/utils/misc.py deleted file mode 100644 index 78060e8..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/utils/misc.py +++ /dev/null @@ -1,789 +0,0 @@ -import contextlib -import errno -import getpass -import hashlib -import io -import logging -import os -import posixpath -import shutil -import stat -import sys -import sysconfig -import urllib.parse -from functools import partial -from io import StringIO -from itertools import filterfalse, tee, zip_longest -from pathlib import Path -from types import FunctionType, TracebackType -from typing import ( - Any, - BinaryIO, - Callable, - ContextManager, - Dict, - Generator, - Iterable, - Iterator, - List, - Optional, - TextIO, - Tuple, - Type, - TypeVar, - Union, - cast, -) - -from pip._vendor.packaging.requirements import Requirement -from pip._vendor.pyproject_hooks import BuildBackendHookCaller -from pip._vendor.tenacity import retry, stop_after_delay, wait_fixed - -from pip import __version__ -from pip._internal.exceptions import CommandError, ExternallyManagedEnvironment -from pip._internal.locations import get_major_minor_version -from pip._internal.utils.compat import WINDOWS -from pip._internal.utils.virtualenv import running_under_virtualenv - -__all__ = [ - "rmtree", - "display_path", - "backup_dir", - "ask", - "splitext", - "format_size", - "is_installable_dir", - "normalize_path", - "renames", - "get_prog", - "captured_stdout", - "ensure_dir", - "remove_auth_from_url", - "check_externally_managed", - "ConfiguredBuildBackendHookCaller", -] - -logger = logging.getLogger(__name__) - -T = TypeVar("T") -ExcInfo = Tuple[Type[BaseException], BaseException, TracebackType] -VersionInfo = Tuple[int, int, int] -NetlocTuple = Tuple[str, Tuple[Optional[str], Optional[str]]] -OnExc = Callable[[FunctionType, Path, BaseException], Any] -OnErr = Callable[[FunctionType, Path, ExcInfo], Any] - - -def get_pip_version() -> str: - pip_pkg_dir = os.path.join(os.path.dirname(__file__), "..", "..") - pip_pkg_dir = os.path.abspath(pip_pkg_dir) - - return "pip {} from {} (python {})".format( - __version__, - pip_pkg_dir, - get_major_minor_version(), - ) - - -def normalize_version_info(py_version_info: Tuple[int, ...]) -> Tuple[int, int, int]: - """ - Convert a tuple of ints representing a Python version to one of length - three. - - :param py_version_info: a tuple of ints representing a Python version, - or None to specify no version. The tuple can have any length. - - :return: a tuple of length three if `py_version_info` is non-None. - Otherwise, return `py_version_info` unchanged (i.e. None). - """ - if len(py_version_info) < 3: - py_version_info += (3 - len(py_version_info)) * (0,) - elif len(py_version_info) > 3: - py_version_info = py_version_info[:3] - - return cast("VersionInfo", py_version_info) - - -def ensure_dir(path: str) -> None: - """os.path.makedirs without EEXIST.""" - try: - os.makedirs(path) - except OSError as e: - # Windows can raise spurious ENOTEMPTY errors. See #6426. - if e.errno != errno.EEXIST and e.errno != errno.ENOTEMPTY: - raise - - -def get_prog() -> str: - try: - prog = os.path.basename(sys.argv[0]) - if prog in ("__main__.py", "-c"): - return f"{sys.executable} -m pip" - else: - return prog - except (AttributeError, TypeError, IndexError): - pass - return "pip" - - -# Retry every half second for up to 3 seconds -# Tenacity raises RetryError by default, explicitly raise the original exception -@retry(reraise=True, stop=stop_after_delay(3), wait=wait_fixed(0.5)) -def rmtree( - dir: str, - ignore_errors: bool = False, - onexc: Optional[OnExc] = None, -) -> None: - if ignore_errors: - onexc = _onerror_ignore - if onexc is None: - onexc = _onerror_reraise - handler: OnErr = partial( - # `[func, path, Union[ExcInfo, BaseException]] -> Any` is equivalent to - # `Union[([func, path, ExcInfo] -> Any), ([func, path, BaseException] -> Any)]`. - cast(Union[OnExc, OnErr], rmtree_errorhandler), - onexc=onexc, - ) - if sys.version_info >= (3, 12): - # See https://docs.python.org/3.12/whatsnew/3.12.html#shutil. - shutil.rmtree(dir, onexc=handler) - else: - shutil.rmtree(dir, onerror=handler) - - -def _onerror_ignore(*_args: Any) -> None: - pass - - -def _onerror_reraise(*_args: Any) -> None: - raise - - -def rmtree_errorhandler( - func: FunctionType, - path: Path, - exc_info: Union[ExcInfo, BaseException], - *, - onexc: OnExc = _onerror_reraise, -) -> None: - """ - `rmtree` error handler to 'force' a file remove (i.e. like `rm -f`). - - * If a file is readonly then it's write flag is set and operation is - retried. - - * `onerror` is the original callback from `rmtree(... onerror=onerror)` - that is chained at the end if the "rm -f" still fails. - """ - try: - st_mode = os.stat(path).st_mode - except OSError: - # it's equivalent to os.path.exists - return - - if not st_mode & stat.S_IWRITE: - # convert to read/write - try: - os.chmod(path, st_mode | stat.S_IWRITE) - except OSError: - pass - else: - # use the original function to repeat the operation - try: - func(path) - return - except OSError: - pass - - if not isinstance(exc_info, BaseException): - _, exc_info, _ = exc_info - onexc(func, path, exc_info) - - -def display_path(path: str) -> str: - """Gives the display value for a given path, making it relative to cwd - if possible.""" - path = os.path.normcase(os.path.abspath(path)) - if path.startswith(os.getcwd() + os.path.sep): - path = "." + path[len(os.getcwd()) :] - return path - - -def backup_dir(dir: str, ext: str = ".bak") -> str: - """Figure out the name of a directory to back up the given dir to - (adding .bak, .bak2, etc)""" - n = 1 - extension = ext - while os.path.exists(dir + extension): - n += 1 - extension = ext + str(n) - return dir + extension - - -def ask_path_exists(message: str, options: Iterable[str]) -> str: - for action in os.environ.get("PIP_EXISTS_ACTION", "").split(): - if action in options: - return action - return ask(message, options) - - -def _check_no_input(message: str) -> None: - """Raise an error if no input is allowed.""" - if os.environ.get("PIP_NO_INPUT"): - raise Exception( - f"No input was expected ($PIP_NO_INPUT set); question: {message}" - ) - - -def ask(message: str, options: Iterable[str]) -> str: - """Ask the message interactively, with the given possible responses""" - while 1: - _check_no_input(message) - response = input(message) - response = response.strip().lower() - if response not in options: - print( - "Your response ({!r}) was not one of the expected responses: " - "{}".format(response, ", ".join(options)) - ) - else: - return response - - -def ask_input(message: str) -> str: - """Ask for input interactively.""" - _check_no_input(message) - return input(message) - - -def ask_password(message: str) -> str: - """Ask for a password interactively.""" - _check_no_input(message) - return getpass.getpass(message) - - -def strtobool(val: str) -> int: - """Convert a string representation of truth to true (1) or false (0). - - True values are 'y', 'yes', 't', 'true', 'on', and '1'; false values - are 'n', 'no', 'f', 'false', 'off', and '0'. Raises ValueError if - 'val' is anything else. - """ - val = val.lower() - if val in ("y", "yes", "t", "true", "on", "1"): - return 1 - elif val in ("n", "no", "f", "false", "off", "0"): - return 0 - else: - raise ValueError(f"invalid truth value {val!r}") - - -def format_size(bytes: float) -> str: - if bytes > 1000 * 1000: - return "{:.1f} MB".format(bytes / 1000.0 / 1000) - elif bytes > 10 * 1000: - return "{} kB".format(int(bytes / 1000)) - elif bytes > 1000: - return "{:.1f} kB".format(bytes / 1000.0) - else: - return "{} bytes".format(int(bytes)) - - -def tabulate(rows: Iterable[Iterable[Any]]) -> Tuple[List[str], List[int]]: - """Return a list of formatted rows and a list of column sizes. - - For example:: - - >>> tabulate([['foobar', 2000], [0xdeadbeef]]) - (['foobar 2000', '3735928559'], [10, 4]) - """ - rows = [tuple(map(str, row)) for row in rows] - sizes = [max(map(len, col)) for col in zip_longest(*rows, fillvalue="")] - table = [" ".join(map(str.ljust, row, sizes)).rstrip() for row in rows] - return table, sizes - - -def is_installable_dir(path: str) -> bool: - """Is path is a directory containing pyproject.toml or setup.py? - - If pyproject.toml exists, this is a PEP 517 project. Otherwise we look for - a legacy setuptools layout by identifying setup.py. We don't check for the - setup.cfg because using it without setup.py is only available for PEP 517 - projects, which are already covered by the pyproject.toml check. - """ - if not os.path.isdir(path): - return False - if os.path.isfile(os.path.join(path, "pyproject.toml")): - return True - if os.path.isfile(os.path.join(path, "setup.py")): - return True - return False - - -def read_chunks( - file: BinaryIO, size: int = io.DEFAULT_BUFFER_SIZE -) -> Generator[bytes, None, None]: - """Yield pieces of data from a file-like object until EOF.""" - while True: - chunk = file.read(size) - if not chunk: - break - yield chunk - - -def normalize_path(path: str, resolve_symlinks: bool = True) -> str: - """ - Convert a path to its canonical, case-normalized, absolute version. - - """ - path = os.path.expanduser(path) - if resolve_symlinks: - path = os.path.realpath(path) - else: - path = os.path.abspath(path) - return os.path.normcase(path) - - -def splitext(path: str) -> Tuple[str, str]: - """Like os.path.splitext, but take off .tar too""" - base, ext = posixpath.splitext(path) - if base.lower().endswith(".tar"): - ext = base[-4:] + ext - base = base[:-4] - return base, ext - - -def renames(old: str, new: str) -> None: - """Like os.renames(), but handles renaming across devices.""" - # Implementation borrowed from os.renames(). - head, tail = os.path.split(new) - if head and tail and not os.path.exists(head): - os.makedirs(head) - - shutil.move(old, new) - - head, tail = os.path.split(old) - if head and tail: - try: - os.removedirs(head) - except OSError: - pass - - -def is_local(path: str) -> bool: - """ - Return True if path is within sys.prefix, if we're running in a virtualenv. - - If we're not in a virtualenv, all paths are considered "local." - - Caution: this function assumes the head of path has been normalized - with normalize_path. - """ - if not running_under_virtualenv(): - return True - return path.startswith(normalize_path(sys.prefix)) - - -def write_output(msg: Any, *args: Any) -> None: - logger.info(msg, *args) - - -class StreamWrapper(StringIO): - orig_stream: TextIO - - @classmethod - def from_stream(cls, orig_stream: TextIO) -> "StreamWrapper": - ret = cls() - ret.orig_stream = orig_stream - return ret - - # compileall.compile_dir() needs stdout.encoding to print to stdout - # type ignore is because TextIOBase.encoding is writeable - @property - def encoding(self) -> str: # type: ignore - return self.orig_stream.encoding - - -@contextlib.contextmanager -def captured_output(stream_name: str) -> Generator[StreamWrapper, None, None]: - """Return a context manager used by captured_stdout/stdin/stderr - that temporarily replaces the sys stream *stream_name* with a StringIO. - - Taken from Lib/support/__init__.py in the CPython repo. - """ - orig_stdout = getattr(sys, stream_name) - setattr(sys, stream_name, StreamWrapper.from_stream(orig_stdout)) - try: - yield getattr(sys, stream_name) - finally: - setattr(sys, stream_name, orig_stdout) - - -def captured_stdout() -> ContextManager[StreamWrapper]: - """Capture the output of sys.stdout: - - with captured_stdout() as stdout: - print('hello') - self.assertEqual(stdout.getvalue(), 'hello\n') - - Taken from Lib/support/__init__.py in the CPython repo. - """ - return captured_output("stdout") - - -def captured_stderr() -> ContextManager[StreamWrapper]: - """ - See captured_stdout(). - """ - return captured_output("stderr") - - -# Simulates an enum -def enum(*sequential: Any, **named: Any) -> Type[Any]: - enums = dict(zip(sequential, range(len(sequential))), **named) - reverse = {value: key for key, value in enums.items()} - enums["reverse_mapping"] = reverse - return type("Enum", (), enums) - - -def build_netloc(host: str, port: Optional[int]) -> str: - """ - Build a netloc from a host-port pair - """ - if port is None: - return host - if ":" in host: - # Only wrap host with square brackets when it is IPv6 - host = f"[{host}]" - return f"{host}:{port}" - - -def build_url_from_netloc(netloc: str, scheme: str = "https") -> str: - """ - Build a full URL from a netloc. - """ - if netloc.count(":") >= 2 and "@" not in netloc and "[" not in netloc: - # It must be a bare IPv6 address, so wrap it with brackets. - netloc = f"[{netloc}]" - return f"{scheme}://{netloc}" - - -def parse_netloc(netloc: str) -> Tuple[Optional[str], Optional[int]]: - """ - Return the host-port pair from a netloc. - """ - url = build_url_from_netloc(netloc) - parsed = urllib.parse.urlparse(url) - return parsed.hostname, parsed.port - - -def split_auth_from_netloc(netloc: str) -> NetlocTuple: - """ - Parse out and remove the auth information from a netloc. - - Returns: (netloc, (username, password)). - """ - if "@" not in netloc: - return netloc, (None, None) - - # Split from the right because that's how urllib.parse.urlsplit() - # behaves if more than one @ is present (which can be checked using - # the password attribute of urlsplit()'s return value). - auth, netloc = netloc.rsplit("@", 1) - pw: Optional[str] = None - if ":" in auth: - # Split from the left because that's how urllib.parse.urlsplit() - # behaves if more than one : is present (which again can be checked - # using the password attribute of the return value) - user, pw = auth.split(":", 1) - else: - user, pw = auth, None - - user = urllib.parse.unquote(user) - if pw is not None: - pw = urllib.parse.unquote(pw) - - return netloc, (user, pw) - - -def redact_netloc(netloc: str) -> str: - """ - Replace the sensitive data in a netloc with "****", if it exists. - - For example: - - "user:pass@example.com" returns "user:****@example.com" - - "accesstoken@example.com" returns "****@example.com" - """ - netloc, (user, password) = split_auth_from_netloc(netloc) - if user is None: - return netloc - if password is None: - user = "****" - password = "" - else: - user = urllib.parse.quote(user) - password = ":****" - return "{user}{password}@{netloc}".format( - user=user, password=password, netloc=netloc - ) - - -def _transform_url( - url: str, transform_netloc: Callable[[str], Tuple[Any, ...]] -) -> Tuple[str, NetlocTuple]: - """Transform and replace netloc in a url. - - transform_netloc is a function taking the netloc and returning a - tuple. The first element of this tuple is the new netloc. The - entire tuple is returned. - - Returns a tuple containing the transformed url as item 0 and the - original tuple returned by transform_netloc as item 1. - """ - purl = urllib.parse.urlsplit(url) - netloc_tuple = transform_netloc(purl.netloc) - # stripped url - url_pieces = (purl.scheme, netloc_tuple[0], purl.path, purl.query, purl.fragment) - surl = urllib.parse.urlunsplit(url_pieces) - return surl, cast("NetlocTuple", netloc_tuple) - - -def _get_netloc(netloc: str) -> NetlocTuple: - return split_auth_from_netloc(netloc) - - -def _redact_netloc(netloc: str) -> Tuple[str]: - return (redact_netloc(netloc),) - - -def split_auth_netloc_from_url( - url: str, -) -> Tuple[str, str, Tuple[Optional[str], Optional[str]]]: - """ - Parse a url into separate netloc, auth, and url with no auth. - - Returns: (url_without_auth, netloc, (username, password)) - """ - url_without_auth, (netloc, auth) = _transform_url(url, _get_netloc) - return url_without_auth, netloc, auth - - -def remove_auth_from_url(url: str) -> str: - """Return a copy of url with 'username:password@' removed.""" - # username/pass params are passed to subversion through flags - # and are not recognized in the url. - return _transform_url(url, _get_netloc)[0] - - -def redact_auth_from_url(url: str) -> str: - """Replace the password in a given url with ****.""" - return _transform_url(url, _redact_netloc)[0] - - -def redact_auth_from_requirement(req: Requirement) -> str: - """Replace the password in a given requirement url with ****.""" - if not req.url: - return str(req) - return str(req).replace(req.url, redact_auth_from_url(req.url)) - - -class HiddenText: - def __init__(self, secret: str, redacted: str) -> None: - self.secret = secret - self.redacted = redacted - - def __repr__(self) -> str: - return "".format(str(self)) - - def __str__(self) -> str: - return self.redacted - - # This is useful for testing. - def __eq__(self, other: Any) -> bool: - if type(self) != type(other): - return False - - # The string being used for redaction doesn't also have to match, - # just the raw, original string. - return self.secret == other.secret - - -def hide_value(value: str) -> HiddenText: - return HiddenText(value, redacted="****") - - -def hide_url(url: str) -> HiddenText: - redacted = redact_auth_from_url(url) - return HiddenText(url, redacted=redacted) - - -def protect_pip_from_modification_on_windows(modifying_pip: bool) -> None: - """Protection of pip.exe from modification on Windows - - On Windows, any operation modifying pip should be run as: - python -m pip ... - """ - pip_names = [ - "pip", - f"pip{sys.version_info.major}", - f"pip{sys.version_info.major}.{sys.version_info.minor}", - ] - - # See https://github.com/pypa/pip/issues/1299 for more discussion - should_show_use_python_msg = ( - modifying_pip and WINDOWS and os.path.basename(sys.argv[0]) in pip_names - ) - - if should_show_use_python_msg: - new_command = [sys.executable, "-m", "pip"] + sys.argv[1:] - raise CommandError( - "To modify pip, please run the following command:\n{}".format( - " ".join(new_command) - ) - ) - - -def check_externally_managed() -> None: - """Check whether the current environment is externally managed. - - If the ``EXTERNALLY-MANAGED`` config file is found, the current environment - is considered externally managed, and an ExternallyManagedEnvironment is - raised. - """ - if running_under_virtualenv(): - return - marker = os.path.join(sysconfig.get_path("stdlib"), "EXTERNALLY-MANAGED") - if not os.path.isfile(marker): - return - raise ExternallyManagedEnvironment.from_config(marker) - - -def is_console_interactive() -> bool: - """Is this console interactive?""" - return sys.stdin is not None and sys.stdin.isatty() - - -def hash_file(path: str, blocksize: int = 1 << 20) -> Tuple[Any, int]: - """Return (hash, length) for path using hashlib.sha256()""" - - h = hashlib.sha256() - length = 0 - with open(path, "rb") as f: - for block in read_chunks(f, size=blocksize): - length += len(block) - h.update(block) - return h, length - - -def pairwise(iterable: Iterable[Any]) -> Iterator[Tuple[Any, Any]]: - """ - Return paired elements. - - For example: - s -> (s0, s1), (s2, s3), (s4, s5), ... - """ - iterable = iter(iterable) - return zip_longest(iterable, iterable) - - -def partition( - pred: Callable[[T], bool], - iterable: Iterable[T], -) -> Tuple[Iterable[T], Iterable[T]]: - """ - Use a predicate to partition entries into false entries and true entries, - like - - partition(is_odd, range(10)) --> 0 2 4 6 8 and 1 3 5 7 9 - """ - t1, t2 = tee(iterable) - return filterfalse(pred, t1), filter(pred, t2) - - -class ConfiguredBuildBackendHookCaller(BuildBackendHookCaller): - def __init__( - self, - config_holder: Any, - source_dir: str, - build_backend: str, - backend_path: Optional[str] = None, - runner: Optional[Callable[..., None]] = None, - python_executable: Optional[str] = None, - ): - super().__init__( - source_dir, build_backend, backend_path, runner, python_executable - ) - self.config_holder = config_holder - - def build_wheel( - self, - wheel_directory: str, - config_settings: Optional[Dict[str, Union[str, List[str]]]] = None, - metadata_directory: Optional[str] = None, - ) -> str: - cs = self.config_holder.config_settings - return super().build_wheel( - wheel_directory, config_settings=cs, metadata_directory=metadata_directory - ) - - def build_sdist( - self, - sdist_directory: str, - config_settings: Optional[Dict[str, Union[str, List[str]]]] = None, - ) -> str: - cs = self.config_holder.config_settings - return super().build_sdist(sdist_directory, config_settings=cs) - - def build_editable( - self, - wheel_directory: str, - config_settings: Optional[Dict[str, Union[str, List[str]]]] = None, - metadata_directory: Optional[str] = None, - ) -> str: - cs = self.config_holder.config_settings - return super().build_editable( - wheel_directory, config_settings=cs, metadata_directory=metadata_directory - ) - - def get_requires_for_build_wheel( - self, config_settings: Optional[Dict[str, Union[str, List[str]]]] = None - ) -> List[str]: - cs = self.config_holder.config_settings - return super().get_requires_for_build_wheel(config_settings=cs) - - def get_requires_for_build_sdist( - self, config_settings: Optional[Dict[str, Union[str, List[str]]]] = None - ) -> List[str]: - cs = self.config_holder.config_settings - return super().get_requires_for_build_sdist(config_settings=cs) - - def get_requires_for_build_editable( - self, config_settings: Optional[Dict[str, Union[str, List[str]]]] = None - ) -> List[str]: - cs = self.config_holder.config_settings - return super().get_requires_for_build_editable(config_settings=cs) - - def prepare_metadata_for_build_wheel( - self, - metadata_directory: str, - config_settings: Optional[Dict[str, Union[str, List[str]]]] = None, - _allow_fallback: bool = True, - ) -> str: - cs = self.config_holder.config_settings - return super().prepare_metadata_for_build_wheel( - metadata_directory=metadata_directory, - config_settings=cs, - _allow_fallback=_allow_fallback, - ) - - def prepare_metadata_for_build_editable( - self, - metadata_directory: str, - config_settings: Optional[Dict[str, Union[str, List[str]]]] = None, - _allow_fallback: bool = True, - ) -> str: - cs = self.config_holder.config_settings - return super().prepare_metadata_for_build_editable( - metadata_directory=metadata_directory, - config_settings=cs, - _allow_fallback=_allow_fallback, - ) diff --git a/venv/lib/python3.11/site-packages/pip/_internal/utils/models.py b/venv/lib/python3.11/site-packages/pip/_internal/utils/models.py deleted file mode 100644 index b6bb21a..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/utils/models.py +++ /dev/null @@ -1,39 +0,0 @@ -"""Utilities for defining models -""" - -import operator -from typing import Any, Callable, Type - - -class KeyBasedCompareMixin: - """Provides comparison capabilities that is based on a key""" - - __slots__ = ["_compare_key", "_defining_class"] - - def __init__(self, key: Any, defining_class: Type["KeyBasedCompareMixin"]) -> None: - self._compare_key = key - self._defining_class = defining_class - - def __hash__(self) -> int: - return hash(self._compare_key) - - def __lt__(self, other: Any) -> bool: - return self._compare(other, operator.__lt__) - - def __le__(self, other: Any) -> bool: - return self._compare(other, operator.__le__) - - def __gt__(self, other: Any) -> bool: - return self._compare(other, operator.__gt__) - - def __ge__(self, other: Any) -> bool: - return self._compare(other, operator.__ge__) - - def __eq__(self, other: Any) -> bool: - return self._compare(other, operator.__eq__) - - def _compare(self, other: Any, method: Callable[[Any, Any], bool]) -> bool: - if not isinstance(other, self._defining_class): - return NotImplemented - - return method(self._compare_key, other._compare_key) diff --git a/venv/lib/python3.11/site-packages/pip/_internal/utils/packaging.py b/venv/lib/python3.11/site-packages/pip/_internal/utils/packaging.py deleted file mode 100644 index b9f6af4..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/utils/packaging.py +++ /dev/null @@ -1,57 +0,0 @@ -import functools -import logging -import re -from typing import NewType, Optional, Tuple, cast - -from pip._vendor.packaging import specifiers, version -from pip._vendor.packaging.requirements import Requirement - -NormalizedExtra = NewType("NormalizedExtra", str) - -logger = logging.getLogger(__name__) - - -def check_requires_python( - requires_python: Optional[str], version_info: Tuple[int, ...] -) -> bool: - """ - Check if the given Python version matches a "Requires-Python" specifier. - - :param version_info: A 3-tuple of ints representing a Python - major-minor-micro version to check (e.g. `sys.version_info[:3]`). - - :return: `True` if the given Python version satisfies the requirement. - Otherwise, return `False`. - - :raises InvalidSpecifier: If `requires_python` has an invalid format. - """ - if requires_python is None: - # The package provides no information - return True - requires_python_specifier = specifiers.SpecifierSet(requires_python) - - python_version = version.parse(".".join(map(str, version_info))) - return python_version in requires_python_specifier - - -@functools.lru_cache(maxsize=512) -def get_requirement(req_string: str) -> Requirement: - """Construct a packaging.Requirement object with caching""" - # Parsing requirement strings is expensive, and is also expected to happen - # with a low diversity of different arguments (at least relative the number - # constructed). This method adds a cache to requirement object creation to - # minimize repeated parsing of the same string to construct equivalent - # Requirement objects. - return Requirement(req_string) - - -def safe_extra(extra: str) -> NormalizedExtra: - """Convert an arbitrary string to a standard 'extra' name - - Any runs of non-alphanumeric characters are replaced with a single '_', - and the result is always lowercased. - - This function is duplicated from ``pkg_resources``. Note that this is not - the same to either ``canonicalize_name`` or ``_egg_link_name``. - """ - return cast(NormalizedExtra, re.sub("[^A-Za-z0-9.-]+", "_", extra).lower()) diff --git a/venv/lib/python3.11/site-packages/pip/_internal/utils/setuptools_build.py b/venv/lib/python3.11/site-packages/pip/_internal/utils/setuptools_build.py deleted file mode 100644 index 96d1b24..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/utils/setuptools_build.py +++ /dev/null @@ -1,146 +0,0 @@ -import sys -import textwrap -from typing import List, Optional, Sequence - -# Shim to wrap setup.py invocation with setuptools -# Note that __file__ is handled via two {!r} *and* %r, to ensure that paths on -# Windows are correctly handled (it should be "C:\\Users" not "C:\Users"). -_SETUPTOOLS_SHIM = textwrap.dedent( - """ - exec(compile(''' - # This is -- a caller that pip uses to run setup.py - # - # - It imports setuptools before invoking setup.py, to enable projects that directly - # import from `distutils.core` to work with newer packaging standards. - # - It provides a clear error message when setuptools is not installed. - # - It sets `sys.argv[0]` to the underlying `setup.py`, when invoking `setup.py` so - # setuptools doesn't think the script is `-c`. This avoids the following warning: - # manifest_maker: standard file '-c' not found". - # - It generates a shim setup.py, for handling setup.cfg-only projects. - import os, sys, tokenize - - try: - import setuptools - except ImportError as error: - print( - "ERROR: Can not execute `setup.py` since setuptools is not available in " - "the build environment.", - file=sys.stderr, - ) - sys.exit(1) - - __file__ = %r - sys.argv[0] = __file__ - - if os.path.exists(__file__): - filename = __file__ - with tokenize.open(__file__) as f: - setup_py_code = f.read() - else: - filename = "" - setup_py_code = "from setuptools import setup; setup()" - - exec(compile(setup_py_code, filename, "exec")) - ''' % ({!r},), "", "exec")) - """ -).rstrip() - - -def make_setuptools_shim_args( - setup_py_path: str, - global_options: Optional[Sequence[str]] = None, - no_user_config: bool = False, - unbuffered_output: bool = False, -) -> List[str]: - """ - Get setuptools command arguments with shim wrapped setup file invocation. - - :param setup_py_path: The path to setup.py to be wrapped. - :param global_options: Additional global options. - :param no_user_config: If True, disables personal user configuration. - :param unbuffered_output: If True, adds the unbuffered switch to the - argument list. - """ - args = [sys.executable] - if unbuffered_output: - args += ["-u"] - args += ["-c", _SETUPTOOLS_SHIM.format(setup_py_path)] - if global_options: - args += global_options - if no_user_config: - args += ["--no-user-cfg"] - return args - - -def make_setuptools_bdist_wheel_args( - setup_py_path: str, - global_options: Sequence[str], - build_options: Sequence[str], - destination_dir: str, -) -> List[str]: - # NOTE: Eventually, we'd want to also -S to the flags here, when we're - # isolating. Currently, it breaks Python in virtualenvs, because it - # relies on site.py to find parts of the standard library outside the - # virtualenv. - args = make_setuptools_shim_args( - setup_py_path, global_options=global_options, unbuffered_output=True - ) - args += ["bdist_wheel", "-d", destination_dir] - args += build_options - return args - - -def make_setuptools_clean_args( - setup_py_path: str, - global_options: Sequence[str], -) -> List[str]: - args = make_setuptools_shim_args( - setup_py_path, global_options=global_options, unbuffered_output=True - ) - args += ["clean", "--all"] - return args - - -def make_setuptools_develop_args( - setup_py_path: str, - *, - global_options: Sequence[str], - no_user_config: bool, - prefix: Optional[str], - home: Optional[str], - use_user_site: bool, -) -> List[str]: - assert not (use_user_site and prefix) - - args = make_setuptools_shim_args( - setup_py_path, - global_options=global_options, - no_user_config=no_user_config, - ) - - args += ["develop", "--no-deps"] - - if prefix: - args += ["--prefix", prefix] - if home is not None: - args += ["--install-dir", home] - - if use_user_site: - args += ["--user", "--prefix="] - - return args - - -def make_setuptools_egg_info_args( - setup_py_path: str, - egg_info_dir: Optional[str], - no_user_config: bool, -) -> List[str]: - args = make_setuptools_shim_args(setup_py_path, no_user_config=no_user_config) - - args += ["egg_info"] - - if egg_info_dir: - args += ["--egg-base", egg_info_dir] - - return args diff --git a/venv/lib/python3.11/site-packages/pip/_internal/utils/subprocess.py b/venv/lib/python3.11/site-packages/pip/_internal/utils/subprocess.py deleted file mode 100644 index 79580b0..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/utils/subprocess.py +++ /dev/null @@ -1,260 +0,0 @@ -import logging -import os -import shlex -import subprocess -from typing import ( - TYPE_CHECKING, - Any, - Callable, - Iterable, - List, - Mapping, - Optional, - Union, -) - -from pip._vendor.rich.markup import escape - -from pip._internal.cli.spinners import SpinnerInterface, open_spinner -from pip._internal.exceptions import InstallationSubprocessError -from pip._internal.utils.logging import VERBOSE, subprocess_logger -from pip._internal.utils.misc import HiddenText - -if TYPE_CHECKING: - # Literal was introduced in Python 3.8. - # - # TODO: Remove `if TYPE_CHECKING` when dropping support for Python 3.7. - from typing import Literal - -CommandArgs = List[Union[str, HiddenText]] - - -def make_command(*args: Union[str, HiddenText, CommandArgs]) -> CommandArgs: - """ - Create a CommandArgs object. - """ - command_args: CommandArgs = [] - for arg in args: - # Check for list instead of CommandArgs since CommandArgs is - # only known during type-checking. - if isinstance(arg, list): - command_args.extend(arg) - else: - # Otherwise, arg is str or HiddenText. - command_args.append(arg) - - return command_args - - -def format_command_args(args: Union[List[str], CommandArgs]) -> str: - """ - Format command arguments for display. - """ - # For HiddenText arguments, display the redacted form by calling str(). - # Also, we don't apply str() to arguments that aren't HiddenText since - # this can trigger a UnicodeDecodeError in Python 2 if the argument - # has type unicode and includes a non-ascii character. (The type - # checker doesn't ensure the annotations are correct in all cases.) - return " ".join( - shlex.quote(str(arg)) if isinstance(arg, HiddenText) else shlex.quote(arg) - for arg in args - ) - - -def reveal_command_args(args: Union[List[str], CommandArgs]) -> List[str]: - """ - Return the arguments in their raw, unredacted form. - """ - return [arg.secret if isinstance(arg, HiddenText) else arg for arg in args] - - -def call_subprocess( - cmd: Union[List[str], CommandArgs], - show_stdout: bool = False, - cwd: Optional[str] = None, - on_returncode: 'Literal["raise", "warn", "ignore"]' = "raise", - extra_ok_returncodes: Optional[Iterable[int]] = None, - extra_environ: Optional[Mapping[str, Any]] = None, - unset_environ: Optional[Iterable[str]] = None, - spinner: Optional[SpinnerInterface] = None, - log_failed_cmd: Optional[bool] = True, - stdout_only: Optional[bool] = False, - *, - command_desc: str, -) -> str: - """ - Args: - show_stdout: if true, use INFO to log the subprocess's stderr and - stdout streams. Otherwise, use DEBUG. Defaults to False. - extra_ok_returncodes: an iterable of integer return codes that are - acceptable, in addition to 0. Defaults to None, which means []. - unset_environ: an iterable of environment variable names to unset - prior to calling subprocess.Popen(). - log_failed_cmd: if false, failed commands are not logged, only raised. - stdout_only: if true, return only stdout, else return both. When true, - logging of both stdout and stderr occurs when the subprocess has - terminated, else logging occurs as subprocess output is produced. - """ - if extra_ok_returncodes is None: - extra_ok_returncodes = [] - if unset_environ is None: - unset_environ = [] - # Most places in pip use show_stdout=False. What this means is-- - # - # - We connect the child's output (combined stderr and stdout) to a - # single pipe, which we read. - # - We log this output to stderr at DEBUG level as it is received. - # - If DEBUG logging isn't enabled (e.g. if --verbose logging wasn't - # requested), then we show a spinner so the user can still see the - # subprocess is in progress. - # - If the subprocess exits with an error, we log the output to stderr - # at ERROR level if it hasn't already been displayed to the console - # (e.g. if --verbose logging wasn't enabled). This way we don't log - # the output to the console twice. - # - # If show_stdout=True, then the above is still done, but with DEBUG - # replaced by INFO. - if show_stdout: - # Then log the subprocess output at INFO level. - log_subprocess: Callable[..., None] = subprocess_logger.info - used_level = logging.INFO - else: - # Then log the subprocess output using VERBOSE. This also ensures - # it will be logged to the log file (aka user_log), if enabled. - log_subprocess = subprocess_logger.verbose - used_level = VERBOSE - - # Whether the subprocess will be visible in the console. - showing_subprocess = subprocess_logger.getEffectiveLevel() <= used_level - - # Only use the spinner if we're not showing the subprocess output - # and we have a spinner. - use_spinner = not showing_subprocess and spinner is not None - - log_subprocess("Running command %s", command_desc) - env = os.environ.copy() - if extra_environ: - env.update(extra_environ) - for name in unset_environ: - env.pop(name, None) - try: - proc = subprocess.Popen( - # Convert HiddenText objects to the underlying str. - reveal_command_args(cmd), - stdin=subprocess.PIPE, - stdout=subprocess.PIPE, - stderr=subprocess.STDOUT if not stdout_only else subprocess.PIPE, - cwd=cwd, - env=env, - errors="backslashreplace", - ) - except Exception as exc: - if log_failed_cmd: - subprocess_logger.critical( - "Error %s while executing command %s", - exc, - command_desc, - ) - raise - all_output = [] - if not stdout_only: - assert proc.stdout - assert proc.stdin - proc.stdin.close() - # In this mode, stdout and stderr are in the same pipe. - while True: - line: str = proc.stdout.readline() - if not line: - break - line = line.rstrip() - all_output.append(line + "\n") - - # Show the line immediately. - log_subprocess(line) - # Update the spinner. - if use_spinner: - assert spinner - spinner.spin() - try: - proc.wait() - finally: - if proc.stdout: - proc.stdout.close() - output = "".join(all_output) - else: - # In this mode, stdout and stderr are in different pipes. - # We must use communicate() which is the only safe way to read both. - out, err = proc.communicate() - # log line by line to preserve pip log indenting - for out_line in out.splitlines(): - log_subprocess(out_line) - all_output.append(out) - for err_line in err.splitlines(): - log_subprocess(err_line) - all_output.append(err) - output = out - - proc_had_error = proc.returncode and proc.returncode not in extra_ok_returncodes - if use_spinner: - assert spinner - if proc_had_error: - spinner.finish("error") - else: - spinner.finish("done") - if proc_had_error: - if on_returncode == "raise": - error = InstallationSubprocessError( - command_description=command_desc, - exit_code=proc.returncode, - output_lines=all_output if not showing_subprocess else None, - ) - if log_failed_cmd: - subprocess_logger.error("%s", error, extra={"rich": True}) - subprocess_logger.verbose( - "[bold magenta]full command[/]: [blue]%s[/]", - escape(format_command_args(cmd)), - extra={"markup": True}, - ) - subprocess_logger.verbose( - "[bold magenta]cwd[/]: %s", - escape(cwd or "[inherit]"), - extra={"markup": True}, - ) - - raise error - elif on_returncode == "warn": - subprocess_logger.warning( - 'Command "%s" had error code %s in %s', - command_desc, - proc.returncode, - cwd, - ) - elif on_returncode == "ignore": - pass - else: - raise ValueError(f"Invalid value: on_returncode={on_returncode!r}") - return output - - -def runner_with_spinner_message(message: str) -> Callable[..., None]: - """Provide a subprocess_runner that shows a spinner message. - - Intended for use with for BuildBackendHookCaller. Thus, the runner has - an API that matches what's expected by BuildBackendHookCaller.subprocess_runner. - """ - - def runner( - cmd: List[str], - cwd: Optional[str] = None, - extra_environ: Optional[Mapping[str, Any]] = None, - ) -> None: - with open_spinner(message) as spinner: - call_subprocess( - cmd, - command_desc=message, - cwd=cwd, - extra_environ=extra_environ, - spinner=spinner, - ) - - return runner diff --git a/venv/lib/python3.11/site-packages/pip/_internal/utils/temp_dir.py b/venv/lib/python3.11/site-packages/pip/_internal/utils/temp_dir.py deleted file mode 100644 index 4eec5f3..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/utils/temp_dir.py +++ /dev/null @@ -1,296 +0,0 @@ -import errno -import itertools -import logging -import os.path -import tempfile -import traceback -from contextlib import ExitStack, contextmanager -from pathlib import Path -from typing import ( - Any, - Callable, - Dict, - Generator, - List, - Optional, - TypeVar, - Union, -) - -from pip._internal.utils.misc import enum, rmtree - -logger = logging.getLogger(__name__) - -_T = TypeVar("_T", bound="TempDirectory") - - -# Kinds of temporary directories. Only needed for ones that are -# globally-managed. -tempdir_kinds = enum( - BUILD_ENV="build-env", - EPHEM_WHEEL_CACHE="ephem-wheel-cache", - REQ_BUILD="req-build", -) - - -_tempdir_manager: Optional[ExitStack] = None - - -@contextmanager -def global_tempdir_manager() -> Generator[None, None, None]: - global _tempdir_manager - with ExitStack() as stack: - old_tempdir_manager, _tempdir_manager = _tempdir_manager, stack - try: - yield - finally: - _tempdir_manager = old_tempdir_manager - - -class TempDirectoryTypeRegistry: - """Manages temp directory behavior""" - - def __init__(self) -> None: - self._should_delete: Dict[str, bool] = {} - - def set_delete(self, kind: str, value: bool) -> None: - """Indicate whether a TempDirectory of the given kind should be - auto-deleted. - """ - self._should_delete[kind] = value - - def get_delete(self, kind: str) -> bool: - """Get configured auto-delete flag for a given TempDirectory type, - default True. - """ - return self._should_delete.get(kind, True) - - -_tempdir_registry: Optional[TempDirectoryTypeRegistry] = None - - -@contextmanager -def tempdir_registry() -> Generator[TempDirectoryTypeRegistry, None, None]: - """Provides a scoped global tempdir registry that can be used to dictate - whether directories should be deleted. - """ - global _tempdir_registry - old_tempdir_registry = _tempdir_registry - _tempdir_registry = TempDirectoryTypeRegistry() - try: - yield _tempdir_registry - finally: - _tempdir_registry = old_tempdir_registry - - -class _Default: - pass - - -_default = _Default() - - -class TempDirectory: - """Helper class that owns and cleans up a temporary directory. - - This class can be used as a context manager or as an OO representation of a - temporary directory. - - Attributes: - path - Location to the created temporary directory - delete - Whether the directory should be deleted when exiting - (when used as a contextmanager) - - Methods: - cleanup() - Deletes the temporary directory - - When used as a context manager, if the delete attribute is True, on - exiting the context the temporary directory is deleted. - """ - - def __init__( - self, - path: Optional[str] = None, - delete: Union[bool, None, _Default] = _default, - kind: str = "temp", - globally_managed: bool = False, - ignore_cleanup_errors: bool = True, - ): - super().__init__() - - if delete is _default: - if path is not None: - # If we were given an explicit directory, resolve delete option - # now. - delete = False - else: - # Otherwise, we wait until cleanup and see what - # tempdir_registry says. - delete = None - - # The only time we specify path is in for editables where it - # is the value of the --src option. - if path is None: - path = self._create(kind) - - self._path = path - self._deleted = False - self.delete = delete - self.kind = kind - self.ignore_cleanup_errors = ignore_cleanup_errors - - if globally_managed: - assert _tempdir_manager is not None - _tempdir_manager.enter_context(self) - - @property - def path(self) -> str: - assert not self._deleted, f"Attempted to access deleted path: {self._path}" - return self._path - - def __repr__(self) -> str: - return f"<{self.__class__.__name__} {self.path!r}>" - - def __enter__(self: _T) -> _T: - return self - - def __exit__(self, exc: Any, value: Any, tb: Any) -> None: - if self.delete is not None: - delete = self.delete - elif _tempdir_registry: - delete = _tempdir_registry.get_delete(self.kind) - else: - delete = True - - if delete: - self.cleanup() - - def _create(self, kind: str) -> str: - """Create a temporary directory and store its path in self.path""" - # We realpath here because some systems have their default tmpdir - # symlinked to another directory. This tends to confuse build - # scripts, so we canonicalize the path by traversing potential - # symlinks here. - path = os.path.realpath(tempfile.mkdtemp(prefix=f"pip-{kind}-")) - logger.debug("Created temporary directory: %s", path) - return path - - def cleanup(self) -> None: - """Remove the temporary directory created and reset state""" - self._deleted = True - if not os.path.exists(self._path): - return - - errors: List[BaseException] = [] - - def onerror( - func: Callable[..., Any], - path: Path, - exc_val: BaseException, - ) -> None: - """Log a warning for a `rmtree` error and continue""" - formatted_exc = "\n".join( - traceback.format_exception_only(type(exc_val), exc_val) - ) - formatted_exc = formatted_exc.rstrip() # remove trailing new line - if func in (os.unlink, os.remove, os.rmdir): - logger.debug( - "Failed to remove a temporary file '%s' due to %s.\n", - path, - formatted_exc, - ) - else: - logger.debug("%s failed with %s.", func.__qualname__, formatted_exc) - errors.append(exc_val) - - if self.ignore_cleanup_errors: - try: - # first try with tenacity; retrying to handle ephemeral errors - rmtree(self._path, ignore_errors=False) - except OSError: - # last pass ignore/log all errors - rmtree(self._path, onexc=onerror) - if errors: - logger.warning( - "Failed to remove contents in a temporary directory '%s'.\n" - "You can safely remove it manually.", - self._path, - ) - else: - rmtree(self._path) - - -class AdjacentTempDirectory(TempDirectory): - """Helper class that creates a temporary directory adjacent to a real one. - - Attributes: - original - The original directory to create a temp directory for. - path - After calling create() or entering, contains the full - path to the temporary directory. - delete - Whether the directory should be deleted when exiting - (when used as a contextmanager) - - """ - - # The characters that may be used to name the temp directory - # We always prepend a ~ and then rotate through these until - # a usable name is found. - # pkg_resources raises a different error for .dist-info folder - # with leading '-' and invalid metadata - LEADING_CHARS = "-~.=%0123456789" - - def __init__(self, original: str, delete: Optional[bool] = None) -> None: - self.original = original.rstrip("/\\") - super().__init__(delete=delete) - - @classmethod - def _generate_names(cls, name: str) -> Generator[str, None, None]: - """Generates a series of temporary names. - - The algorithm replaces the leading characters in the name - with ones that are valid filesystem characters, but are not - valid package names (for both Python and pip definitions of - package). - """ - for i in range(1, len(name)): - for candidate in itertools.combinations_with_replacement( - cls.LEADING_CHARS, i - 1 - ): - new_name = "~" + "".join(candidate) + name[i:] - if new_name != name: - yield new_name - - # If we make it this far, we will have to make a longer name - for i in range(len(cls.LEADING_CHARS)): - for candidate in itertools.combinations_with_replacement( - cls.LEADING_CHARS, i - ): - new_name = "~" + "".join(candidate) + name - if new_name != name: - yield new_name - - def _create(self, kind: str) -> str: - root, name = os.path.split(self.original) - for candidate in self._generate_names(name): - path = os.path.join(root, candidate) - try: - os.mkdir(path) - except OSError as ex: - # Continue if the name exists already - if ex.errno != errno.EEXIST: - raise - else: - path = os.path.realpath(path) - break - else: - # Final fallback on the default behavior. - path = os.path.realpath(tempfile.mkdtemp(prefix=f"pip-{kind}-")) - - logger.debug("Created temporary directory: %s", path) - return path diff --git a/venv/lib/python3.11/site-packages/pip/_internal/utils/unpacking.py b/venv/lib/python3.11/site-packages/pip/_internal/utils/unpacking.py deleted file mode 100644 index 78b5c13..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/utils/unpacking.py +++ /dev/null @@ -1,257 +0,0 @@ -"""Utilities related archives. -""" - -import logging -import os -import shutil -import stat -import tarfile -import zipfile -from typing import Iterable, List, Optional -from zipfile import ZipInfo - -from pip._internal.exceptions import InstallationError -from pip._internal.utils.filetypes import ( - BZ2_EXTENSIONS, - TAR_EXTENSIONS, - XZ_EXTENSIONS, - ZIP_EXTENSIONS, -) -from pip._internal.utils.misc import ensure_dir - -logger = logging.getLogger(__name__) - - -SUPPORTED_EXTENSIONS = ZIP_EXTENSIONS + TAR_EXTENSIONS - -try: - import bz2 # noqa - - SUPPORTED_EXTENSIONS += BZ2_EXTENSIONS -except ImportError: - logger.debug("bz2 module is not available") - -try: - # Only for Python 3.3+ - import lzma # noqa - - SUPPORTED_EXTENSIONS += XZ_EXTENSIONS -except ImportError: - logger.debug("lzma module is not available") - - -def current_umask() -> int: - """Get the current umask which involves having to set it temporarily.""" - mask = os.umask(0) - os.umask(mask) - return mask - - -def split_leading_dir(path: str) -> List[str]: - path = path.lstrip("/").lstrip("\\") - if "/" in path and ( - ("\\" in path and path.find("/") < path.find("\\")) or "\\" not in path - ): - return path.split("/", 1) - elif "\\" in path: - return path.split("\\", 1) - else: - return [path, ""] - - -def has_leading_dir(paths: Iterable[str]) -> bool: - """Returns true if all the paths have the same leading path name - (i.e., everything is in one subdirectory in an archive)""" - common_prefix = None - for path in paths: - prefix, rest = split_leading_dir(path) - if not prefix: - return False - elif common_prefix is None: - common_prefix = prefix - elif prefix != common_prefix: - return False - return True - - -def is_within_directory(directory: str, target: str) -> bool: - """ - Return true if the absolute path of target is within the directory - """ - abs_directory = os.path.abspath(directory) - abs_target = os.path.abspath(target) - - prefix = os.path.commonprefix([abs_directory, abs_target]) - return prefix == abs_directory - - -def set_extracted_file_to_default_mode_plus_executable(path: str) -> None: - """ - Make file present at path have execute for user/group/world - (chmod +x) is no-op on windows per python docs - """ - os.chmod(path, (0o777 & ~current_umask() | 0o111)) - - -def zip_item_is_executable(info: ZipInfo) -> bool: - mode = info.external_attr >> 16 - # if mode and regular file and any execute permissions for - # user/group/world? - return bool(mode and stat.S_ISREG(mode) and mode & 0o111) - - -def unzip_file(filename: str, location: str, flatten: bool = True) -> None: - """ - Unzip the file (with path `filename`) to the destination `location`. All - files are written based on system defaults and umask (i.e. permissions are - not preserved), except that regular file members with any execute - permissions (user, group, or world) have "chmod +x" applied after being - written. Note that for windows, any execute changes using os.chmod are - no-ops per the python docs. - """ - ensure_dir(location) - zipfp = open(filename, "rb") - try: - zip = zipfile.ZipFile(zipfp, allowZip64=True) - leading = has_leading_dir(zip.namelist()) and flatten - for info in zip.infolist(): - name = info.filename - fn = name - if leading: - fn = split_leading_dir(name)[1] - fn = os.path.join(location, fn) - dir = os.path.dirname(fn) - if not is_within_directory(location, fn): - message = ( - "The zip file ({}) has a file ({}) trying to install " - "outside target directory ({})" - ) - raise InstallationError(message.format(filename, fn, location)) - if fn.endswith("/") or fn.endswith("\\"): - # A directory - ensure_dir(fn) - else: - ensure_dir(dir) - # Don't use read() to avoid allocating an arbitrarily large - # chunk of memory for the file's content - fp = zip.open(name) - try: - with open(fn, "wb") as destfp: - shutil.copyfileobj(fp, destfp) - finally: - fp.close() - if zip_item_is_executable(info): - set_extracted_file_to_default_mode_plus_executable(fn) - finally: - zipfp.close() - - -def untar_file(filename: str, location: str) -> None: - """ - Untar the file (with path `filename`) to the destination `location`. - All files are written based on system defaults and umask (i.e. permissions - are not preserved), except that regular file members with any execute - permissions (user, group, or world) have "chmod +x" applied after being - written. Note that for windows, any execute changes using os.chmod are - no-ops per the python docs. - """ - ensure_dir(location) - if filename.lower().endswith(".gz") or filename.lower().endswith(".tgz"): - mode = "r:gz" - elif filename.lower().endswith(BZ2_EXTENSIONS): - mode = "r:bz2" - elif filename.lower().endswith(XZ_EXTENSIONS): - mode = "r:xz" - elif filename.lower().endswith(".tar"): - mode = "r" - else: - logger.warning( - "Cannot determine compression type for file %s", - filename, - ) - mode = "r:*" - tar = tarfile.open(filename, mode, encoding="utf-8") - try: - leading = has_leading_dir([member.name for member in tar.getmembers()]) - for member in tar.getmembers(): - fn = member.name - if leading: - fn = split_leading_dir(fn)[1] - path = os.path.join(location, fn) - if not is_within_directory(location, path): - message = ( - "The tar file ({}) has a file ({}) trying to install " - "outside target directory ({})" - ) - raise InstallationError(message.format(filename, path, location)) - if member.isdir(): - ensure_dir(path) - elif member.issym(): - try: - tar._extract_member(member, path) - except Exception as exc: - # Some corrupt tar files seem to produce this - # (specifically bad symlinks) - logger.warning( - "In the tar file %s the member %s is invalid: %s", - filename, - member.name, - exc, - ) - continue - else: - try: - fp = tar.extractfile(member) - except (KeyError, AttributeError) as exc: - # Some corrupt tar files seem to produce this - # (specifically bad symlinks) - logger.warning( - "In the tar file %s the member %s is invalid: %s", - filename, - member.name, - exc, - ) - continue - ensure_dir(os.path.dirname(path)) - assert fp is not None - with open(path, "wb") as destfp: - shutil.copyfileobj(fp, destfp) - fp.close() - # Update the timestamp (useful for cython compiled files) - tar.utime(member, path) - # member have any execute permissions for user/group/world? - if member.mode & 0o111: - set_extracted_file_to_default_mode_plus_executable(path) - finally: - tar.close() - - -def unpack_file( - filename: str, - location: str, - content_type: Optional[str] = None, -) -> None: - filename = os.path.realpath(filename) - if ( - content_type == "application/zip" - or filename.lower().endswith(ZIP_EXTENSIONS) - or zipfile.is_zipfile(filename) - ): - unzip_file(filename, location, flatten=not filename.endswith(".whl")) - elif ( - content_type == "application/x-gzip" - or tarfile.is_tarfile(filename) - or filename.lower().endswith(TAR_EXTENSIONS + BZ2_EXTENSIONS + XZ_EXTENSIONS) - ): - untar_file(filename, location) - else: - # FIXME: handle? - # FIXME: magic signatures? - logger.critical( - "Cannot unpack file %s (downloaded from %s, content-type: %s); " - "cannot detect archive format", - filename, - location, - content_type, - ) - raise InstallationError(f"Cannot determine archive format of {location}") diff --git a/venv/lib/python3.11/site-packages/pip/_internal/utils/urls.py b/venv/lib/python3.11/site-packages/pip/_internal/utils/urls.py deleted file mode 100644 index 6ba2e04..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/utils/urls.py +++ /dev/null @@ -1,62 +0,0 @@ -import os -import string -import urllib.parse -import urllib.request -from typing import Optional - -from .compat import WINDOWS - - -def get_url_scheme(url: str) -> Optional[str]: - if ":" not in url: - return None - return url.split(":", 1)[0].lower() - - -def path_to_url(path: str) -> str: - """ - Convert a path to a file: URL. The path will be made absolute and have - quoted path parts. - """ - path = os.path.normpath(os.path.abspath(path)) - url = urllib.parse.urljoin("file:", urllib.request.pathname2url(path)) - return url - - -def url_to_path(url: str) -> str: - """ - Convert a file: URL to a path. - """ - assert url.startswith( - "file:" - ), f"You can only turn file: urls into filenames (not {url!r})" - - _, netloc, path, _, _ = urllib.parse.urlsplit(url) - - if not netloc or netloc == "localhost": - # According to RFC 8089, same as empty authority. - netloc = "" - elif WINDOWS: - # If we have a UNC path, prepend UNC share notation. - netloc = "\\\\" + netloc - else: - raise ValueError( - f"non-local file URIs are not supported on this platform: {url!r}" - ) - - path = urllib.request.url2pathname(netloc + path) - - # On Windows, urlsplit parses the path as something like "/C:/Users/foo". - # This creates issues for path-related functions like io.open(), so we try - # to detect and strip the leading slash. - if ( - WINDOWS - and not netloc # Not UNC. - and len(path) >= 3 - and path[0] == "/" # Leading slash to strip. - and path[1] in string.ascii_letters # Drive letter. - and path[2:4] in (":", ":/") # Colon + end of string, or colon + absolute path. - ): - path = path[1:] - - return path diff --git a/venv/lib/python3.11/site-packages/pip/_internal/utils/virtualenv.py b/venv/lib/python3.11/site-packages/pip/_internal/utils/virtualenv.py deleted file mode 100644 index 882e36f..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/utils/virtualenv.py +++ /dev/null @@ -1,104 +0,0 @@ -import logging -import os -import re -import site -import sys -from typing import List, Optional - -logger = logging.getLogger(__name__) -_INCLUDE_SYSTEM_SITE_PACKAGES_REGEX = re.compile( - r"include-system-site-packages\s*=\s*(?Ptrue|false)" -) - - -def _running_under_venv() -> bool: - """Checks if sys.base_prefix and sys.prefix match. - - This handles PEP 405 compliant virtual environments. - """ - return sys.prefix != getattr(sys, "base_prefix", sys.prefix) - - -def _running_under_legacy_virtualenv() -> bool: - """Checks if sys.real_prefix is set. - - This handles virtual environments created with pypa's virtualenv. - """ - # pypa/virtualenv case - return hasattr(sys, "real_prefix") - - -def running_under_virtualenv() -> bool: - """True if we're running inside a virtual environment, False otherwise.""" - return _running_under_venv() or _running_under_legacy_virtualenv() - - -def _get_pyvenv_cfg_lines() -> Optional[List[str]]: - """Reads {sys.prefix}/pyvenv.cfg and returns its contents as list of lines - - Returns None, if it could not read/access the file. - """ - pyvenv_cfg_file = os.path.join(sys.prefix, "pyvenv.cfg") - try: - # Although PEP 405 does not specify, the built-in venv module always - # writes with UTF-8. (pypa/pip#8717) - with open(pyvenv_cfg_file, encoding="utf-8") as f: - return f.read().splitlines() # avoids trailing newlines - except OSError: - return None - - -def _no_global_under_venv() -> bool: - """Check `{sys.prefix}/pyvenv.cfg` for system site-packages inclusion - - PEP 405 specifies that when system site-packages are not supposed to be - visible from a virtual environment, `pyvenv.cfg` must contain the following - line: - - include-system-site-packages = false - - Additionally, log a warning if accessing the file fails. - """ - cfg_lines = _get_pyvenv_cfg_lines() - if cfg_lines is None: - # We're not in a "sane" venv, so assume there is no system - # site-packages access (since that's PEP 405's default state). - logger.warning( - "Could not access 'pyvenv.cfg' despite a virtual environment " - "being active. Assuming global site-packages is not accessible " - "in this environment." - ) - return True - - for line in cfg_lines: - match = _INCLUDE_SYSTEM_SITE_PACKAGES_REGEX.match(line) - if match is not None and match.group("value") == "false": - return True - return False - - -def _no_global_under_legacy_virtualenv() -> bool: - """Check if "no-global-site-packages.txt" exists beside site.py - - This mirrors logic in pypa/virtualenv for determining whether system - site-packages are visible in the virtual environment. - """ - site_mod_dir = os.path.dirname(os.path.abspath(site.__file__)) - no_global_site_packages_file = os.path.join( - site_mod_dir, - "no-global-site-packages.txt", - ) - return os.path.exists(no_global_site_packages_file) - - -def virtualenv_no_global() -> bool: - """Returns a boolean, whether running in venv with no system site-packages.""" - # PEP 405 compliance needs to be checked first since virtualenv >=20 would - # return True for both checks, but is only able to use the PEP 405 config. - if _running_under_venv(): - return _no_global_under_venv() - - if _running_under_legacy_virtualenv(): - return _no_global_under_legacy_virtualenv() - - return False diff --git a/venv/lib/python3.11/site-packages/pip/_internal/utils/wheel.py b/venv/lib/python3.11/site-packages/pip/_internal/utils/wheel.py deleted file mode 100644 index e5e3f34..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/utils/wheel.py +++ /dev/null @@ -1,136 +0,0 @@ -"""Support functions for working with wheel files. -""" - -import logging -from email.message import Message -from email.parser import Parser -from typing import Tuple -from zipfile import BadZipFile, ZipFile - -from pip._vendor.packaging.utils import canonicalize_name - -from pip._internal.exceptions import UnsupportedWheel - -VERSION_COMPATIBLE = (1, 0) - - -logger = logging.getLogger(__name__) - - -def parse_wheel(wheel_zip: ZipFile, name: str) -> Tuple[str, Message]: - """Extract information from the provided wheel, ensuring it meets basic - standards. - - Returns the name of the .dist-info directory and the parsed WHEEL metadata. - """ - try: - info_dir = wheel_dist_info_dir(wheel_zip, name) - metadata = wheel_metadata(wheel_zip, info_dir) - version = wheel_version(metadata) - except UnsupportedWheel as e: - raise UnsupportedWheel("{} has an invalid wheel, {}".format(name, str(e))) - - check_compatibility(version, name) - - return info_dir, metadata - - -def wheel_dist_info_dir(source: ZipFile, name: str) -> str: - """Returns the name of the contained .dist-info directory. - - Raises AssertionError or UnsupportedWheel if not found, >1 found, or - it doesn't match the provided name. - """ - # Zip file path separators must be / - subdirs = {p.split("/", 1)[0] for p in source.namelist()} - - info_dirs = [s for s in subdirs if s.endswith(".dist-info")] - - if not info_dirs: - raise UnsupportedWheel(".dist-info directory not found") - - if len(info_dirs) > 1: - raise UnsupportedWheel( - "multiple .dist-info directories found: {}".format(", ".join(info_dirs)) - ) - - info_dir = info_dirs[0] - - info_dir_name = canonicalize_name(info_dir) - canonical_name = canonicalize_name(name) - if not info_dir_name.startswith(canonical_name): - raise UnsupportedWheel( - ".dist-info directory {!r} does not start with {!r}".format( - info_dir, canonical_name - ) - ) - - return info_dir - - -def read_wheel_metadata_file(source: ZipFile, path: str) -> bytes: - try: - return source.read(path) - # BadZipFile for general corruption, KeyError for missing entry, - # and RuntimeError for password-protected files - except (BadZipFile, KeyError, RuntimeError) as e: - raise UnsupportedWheel(f"could not read {path!r} file: {e!r}") - - -def wheel_metadata(source: ZipFile, dist_info_dir: str) -> Message: - """Return the WHEEL metadata of an extracted wheel, if possible. - Otherwise, raise UnsupportedWheel. - """ - path = f"{dist_info_dir}/WHEEL" - # Zip file path separators must be / - wheel_contents = read_wheel_metadata_file(source, path) - - try: - wheel_text = wheel_contents.decode() - except UnicodeDecodeError as e: - raise UnsupportedWheel(f"error decoding {path!r}: {e!r}") - - # FeedParser (used by Parser) does not raise any exceptions. The returned - # message may have .defects populated, but for backwards-compatibility we - # currently ignore them. - return Parser().parsestr(wheel_text) - - -def wheel_version(wheel_data: Message) -> Tuple[int, ...]: - """Given WHEEL metadata, return the parsed Wheel-Version. - Otherwise, raise UnsupportedWheel. - """ - version_text = wheel_data["Wheel-Version"] - if version_text is None: - raise UnsupportedWheel("WHEEL is missing Wheel-Version") - - version = version_text.strip() - - try: - return tuple(map(int, version.split("."))) - except ValueError: - raise UnsupportedWheel(f"invalid Wheel-Version: {version!r}") - - -def check_compatibility(version: Tuple[int, ...], name: str) -> None: - """Raises errors or warns if called with an incompatible Wheel-Version. - - pip should refuse to install a Wheel-Version that's a major series - ahead of what it's compatible with (e.g 2.0 > 1.1); and warn when - installing a version only minor version ahead (e.g 1.2 > 1.1). - - version: a 2-tuple representing a Wheel-Version (Major, Minor) - name: name of wheel or package to raise exception about - - :raises UnsupportedWheel: when an incompatible Wheel-Version is given - """ - if version[0] > VERSION_COMPATIBLE[0]: - raise UnsupportedWheel( - "{}'s Wheel-Version ({}) is not compatible with this version " - "of pip".format(name, ".".join(map(str, version))) - ) - elif version > VERSION_COMPATIBLE: - logger.warning( - "Installing from a newer Wheel-Version (%s)", - ".".join(map(str, version)), - ) diff --git a/venv/lib/python3.11/site-packages/pip/_internal/vcs/__init__.py b/venv/lib/python3.11/site-packages/pip/_internal/vcs/__init__.py deleted file mode 100644 index b6beddb..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/vcs/__init__.py +++ /dev/null @@ -1,15 +0,0 @@ -# Expose a limited set of classes and functions so callers outside of -# the vcs package don't need to import deeper than `pip._internal.vcs`. -# (The test directory may still need to import from a vcs sub-package.) -# Import all vcs modules to register each VCS in the VcsSupport object. -import pip._internal.vcs.bazaar -import pip._internal.vcs.git -import pip._internal.vcs.mercurial -import pip._internal.vcs.subversion # noqa: F401 -from pip._internal.vcs.versioncontrol import ( # noqa: F401 - RemoteNotFoundError, - RemoteNotValidError, - is_url, - make_vcs_requirement_url, - vcs, -) diff --git a/venv/lib/python3.11/site-packages/pip/_internal/vcs/__pycache__/__init__.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/vcs/__pycache__/__init__.cpython-311.pyc deleted file mode 100644 index aa0b33e..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/vcs/__pycache__/__init__.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/vcs/__pycache__/bazaar.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/vcs/__pycache__/bazaar.cpython-311.pyc deleted file mode 100644 index 0297baa..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/vcs/__pycache__/bazaar.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/vcs/__pycache__/git.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/vcs/__pycache__/git.cpython-311.pyc deleted file mode 100644 index f78f569..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/vcs/__pycache__/git.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/vcs/__pycache__/mercurial.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/vcs/__pycache__/mercurial.cpython-311.pyc deleted file mode 100644 index 22ea0a2..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/vcs/__pycache__/mercurial.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/vcs/__pycache__/subversion.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/vcs/__pycache__/subversion.cpython-311.pyc deleted file mode 100644 index 3f6f237..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/vcs/__pycache__/subversion.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/vcs/__pycache__/versioncontrol.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_internal/vcs/__pycache__/versioncontrol.cpython-311.pyc deleted file mode 100644 index d6c64c1..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_internal/vcs/__pycache__/versioncontrol.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_internal/vcs/bazaar.py b/venv/lib/python3.11/site-packages/pip/_internal/vcs/bazaar.py deleted file mode 100644 index 20a17ed..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/vcs/bazaar.py +++ /dev/null @@ -1,112 +0,0 @@ -import logging -from typing import List, Optional, Tuple - -from pip._internal.utils.misc import HiddenText, display_path -from pip._internal.utils.subprocess import make_command -from pip._internal.utils.urls import path_to_url -from pip._internal.vcs.versioncontrol import ( - AuthInfo, - RemoteNotFoundError, - RevOptions, - VersionControl, - vcs, -) - -logger = logging.getLogger(__name__) - - -class Bazaar(VersionControl): - name = "bzr" - dirname = ".bzr" - repo_name = "branch" - schemes = ( - "bzr+http", - "bzr+https", - "bzr+ssh", - "bzr+sftp", - "bzr+ftp", - "bzr+lp", - "bzr+file", - ) - - @staticmethod - def get_base_rev_args(rev: str) -> List[str]: - return ["-r", rev] - - def fetch_new( - self, dest: str, url: HiddenText, rev_options: RevOptions, verbosity: int - ) -> None: - rev_display = rev_options.to_display() - logger.info( - "Checking out %s%s to %s", - url, - rev_display, - display_path(dest), - ) - if verbosity <= 0: - flag = "--quiet" - elif verbosity == 1: - flag = "" - else: - flag = f"-{'v'*verbosity}" - cmd_args = make_command( - "checkout", "--lightweight", flag, rev_options.to_args(), url, dest - ) - self.run_command(cmd_args) - - def switch(self, dest: str, url: HiddenText, rev_options: RevOptions) -> None: - self.run_command(make_command("switch", url), cwd=dest) - - def update(self, dest: str, url: HiddenText, rev_options: RevOptions) -> None: - output = self.run_command( - make_command("info"), show_stdout=False, stdout_only=True, cwd=dest - ) - if output.startswith("Standalone "): - # Older versions of pip used to create standalone branches. - # Convert the standalone branch to a checkout by calling "bzr bind". - cmd_args = make_command("bind", "-q", url) - self.run_command(cmd_args, cwd=dest) - - cmd_args = make_command("update", "-q", rev_options.to_args()) - self.run_command(cmd_args, cwd=dest) - - @classmethod - def get_url_rev_and_auth(cls, url: str) -> Tuple[str, Optional[str], AuthInfo]: - # hotfix the URL scheme after removing bzr+ from bzr+ssh:// re-add it - url, rev, user_pass = super().get_url_rev_and_auth(url) - if url.startswith("ssh://"): - url = "bzr+" + url - return url, rev, user_pass - - @classmethod - def get_remote_url(cls, location: str) -> str: - urls = cls.run_command( - ["info"], show_stdout=False, stdout_only=True, cwd=location - ) - for line in urls.splitlines(): - line = line.strip() - for x in ("checkout of branch: ", "parent branch: "): - if line.startswith(x): - repo = line.split(x)[1] - if cls._is_local_repository(repo): - return path_to_url(repo) - return repo - raise RemoteNotFoundError - - @classmethod - def get_revision(cls, location: str) -> str: - revision = cls.run_command( - ["revno"], - show_stdout=False, - stdout_only=True, - cwd=location, - ) - return revision.splitlines()[-1] - - @classmethod - def is_commit_id_equal(cls, dest: str, name: Optional[str]) -> bool: - """Always assume the versions don't match""" - return False - - -vcs.register(Bazaar) diff --git a/venv/lib/python3.11/site-packages/pip/_internal/vcs/git.py b/venv/lib/python3.11/site-packages/pip/_internal/vcs/git.py deleted file mode 100644 index 8c242cf..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/vcs/git.py +++ /dev/null @@ -1,526 +0,0 @@ -import logging -import os.path -import pathlib -import re -import urllib.parse -import urllib.request -from typing import List, Optional, Tuple - -from pip._internal.exceptions import BadCommand, InstallationError -from pip._internal.utils.misc import HiddenText, display_path, hide_url -from pip._internal.utils.subprocess import make_command -from pip._internal.vcs.versioncontrol import ( - AuthInfo, - RemoteNotFoundError, - RemoteNotValidError, - RevOptions, - VersionControl, - find_path_to_project_root_from_repo_root, - vcs, -) - -urlsplit = urllib.parse.urlsplit -urlunsplit = urllib.parse.urlunsplit - - -logger = logging.getLogger(__name__) - - -GIT_VERSION_REGEX = re.compile( - r"^git version " # Prefix. - r"(\d+)" # Major. - r"\.(\d+)" # Dot, minor. - r"(?:\.(\d+))?" # Optional dot, patch. - r".*$" # Suffix, including any pre- and post-release segments we don't care about. -) - -HASH_REGEX = re.compile("^[a-fA-F0-9]{40}$") - -# SCP (Secure copy protocol) shorthand. e.g. 'git@example.com:foo/bar.git' -SCP_REGEX = re.compile( - r"""^ - # Optional user, e.g. 'git@' - (\w+@)? - # Server, e.g. 'github.com'. - ([^/:]+): - # The server-side path. e.g. 'user/project.git'. Must start with an - # alphanumeric character so as not to be confusable with a Windows paths - # like 'C:/foo/bar' or 'C:\foo\bar'. - (\w[^:]*) - $""", - re.VERBOSE, -) - - -def looks_like_hash(sha: str) -> bool: - return bool(HASH_REGEX.match(sha)) - - -class Git(VersionControl): - name = "git" - dirname = ".git" - repo_name = "clone" - schemes = ( - "git+http", - "git+https", - "git+ssh", - "git+git", - "git+file", - ) - # Prevent the user's environment variables from interfering with pip: - # https://github.com/pypa/pip/issues/1130 - unset_environ = ("GIT_DIR", "GIT_WORK_TREE") - default_arg_rev = "HEAD" - - @staticmethod - def get_base_rev_args(rev: str) -> List[str]: - return [rev] - - def is_immutable_rev_checkout(self, url: str, dest: str) -> bool: - _, rev_options = self.get_url_rev_options(hide_url(url)) - if not rev_options.rev: - return False - if not self.is_commit_id_equal(dest, rev_options.rev): - # the current commit is different from rev, - # which means rev was something else than a commit hash - return False - # return False in the rare case rev is both a commit hash - # and a tag or a branch; we don't want to cache in that case - # because that branch/tag could point to something else in the future - is_tag_or_branch = bool(self.get_revision_sha(dest, rev_options.rev)[0]) - return not is_tag_or_branch - - def get_git_version(self) -> Tuple[int, ...]: - version = self.run_command( - ["version"], - command_desc="git version", - show_stdout=False, - stdout_only=True, - ) - match = GIT_VERSION_REGEX.match(version) - if not match: - logger.warning("Can't parse git version: %s", version) - return () - return (int(match.group(1)), int(match.group(2))) - - @classmethod - def get_current_branch(cls, location: str) -> Optional[str]: - """ - Return the current branch, or None if HEAD isn't at a branch - (e.g. detached HEAD). - """ - # git-symbolic-ref exits with empty stdout if "HEAD" is a detached - # HEAD rather than a symbolic ref. In addition, the -q causes the - # command to exit with status code 1 instead of 128 in this case - # and to suppress the message to stderr. - args = ["symbolic-ref", "-q", "HEAD"] - output = cls.run_command( - args, - extra_ok_returncodes=(1,), - show_stdout=False, - stdout_only=True, - cwd=location, - ) - ref = output.strip() - - if ref.startswith("refs/heads/"): - return ref[len("refs/heads/") :] - - return None - - @classmethod - def get_revision_sha(cls, dest: str, rev: str) -> Tuple[Optional[str], bool]: - """ - Return (sha_or_none, is_branch), where sha_or_none is a commit hash - if the revision names a remote branch or tag, otherwise None. - - Args: - dest: the repository directory. - rev: the revision name. - """ - # Pass rev to pre-filter the list. - output = cls.run_command( - ["show-ref", rev], - cwd=dest, - show_stdout=False, - stdout_only=True, - on_returncode="ignore", - ) - refs = {} - # NOTE: We do not use splitlines here since that would split on other - # unicode separators, which can be maliciously used to install a - # different revision. - for line in output.strip().split("\n"): - line = line.rstrip("\r") - if not line: - continue - try: - ref_sha, ref_name = line.split(" ", maxsplit=2) - except ValueError: - # Include the offending line to simplify troubleshooting if - # this error ever occurs. - raise ValueError(f"unexpected show-ref line: {line!r}") - - refs[ref_name] = ref_sha - - branch_ref = f"refs/remotes/origin/{rev}" - tag_ref = f"refs/tags/{rev}" - - sha = refs.get(branch_ref) - if sha is not None: - return (sha, True) - - sha = refs.get(tag_ref) - - return (sha, False) - - @classmethod - def _should_fetch(cls, dest: str, rev: str) -> bool: - """ - Return true if rev is a ref or is a commit that we don't have locally. - - Branches and tags are not considered in this method because they are - assumed to be always available locally (which is a normal outcome of - ``git clone`` and ``git fetch --tags``). - """ - if rev.startswith("refs/"): - # Always fetch remote refs. - return True - - if not looks_like_hash(rev): - # Git fetch would fail with abbreviated commits. - return False - - if cls.has_commit(dest, rev): - # Don't fetch if we have the commit locally. - return False - - return True - - @classmethod - def resolve_revision( - cls, dest: str, url: HiddenText, rev_options: RevOptions - ) -> RevOptions: - """ - Resolve a revision to a new RevOptions object with the SHA1 of the - branch, tag, or ref if found. - - Args: - rev_options: a RevOptions object. - """ - rev = rev_options.arg_rev - # The arg_rev property's implementation for Git ensures that the - # rev return value is always non-None. - assert rev is not None - - sha, is_branch = cls.get_revision_sha(dest, rev) - - if sha is not None: - rev_options = rev_options.make_new(sha) - rev_options.branch_name = rev if is_branch else None - - return rev_options - - # Do not show a warning for the common case of something that has - # the form of a Git commit hash. - if not looks_like_hash(rev): - logger.warning( - "Did not find branch or tag '%s', assuming revision or ref.", - rev, - ) - - if not cls._should_fetch(dest, rev): - return rev_options - - # fetch the requested revision - cls.run_command( - make_command("fetch", "-q", url, rev_options.to_args()), - cwd=dest, - ) - # Change the revision to the SHA of the ref we fetched - sha = cls.get_revision(dest, rev="FETCH_HEAD") - rev_options = rev_options.make_new(sha) - - return rev_options - - @classmethod - def is_commit_id_equal(cls, dest: str, name: Optional[str]) -> bool: - """ - Return whether the current commit hash equals the given name. - - Args: - dest: the repository directory. - name: a string name. - """ - if not name: - # Then avoid an unnecessary subprocess call. - return False - - return cls.get_revision(dest) == name - - def fetch_new( - self, dest: str, url: HiddenText, rev_options: RevOptions, verbosity: int - ) -> None: - rev_display = rev_options.to_display() - logger.info("Cloning %s%s to %s", url, rev_display, display_path(dest)) - if verbosity <= 0: - flags: Tuple[str, ...] = ("--quiet",) - elif verbosity == 1: - flags = () - else: - flags = ("--verbose", "--progress") - if self.get_git_version() >= (2, 17): - # Git added support for partial clone in 2.17 - # https://git-scm.com/docs/partial-clone - # Speeds up cloning by functioning without a complete copy of repository - self.run_command( - make_command( - "clone", - "--filter=blob:none", - *flags, - url, - dest, - ) - ) - else: - self.run_command(make_command("clone", *flags, url, dest)) - - if rev_options.rev: - # Then a specific revision was requested. - rev_options = self.resolve_revision(dest, url, rev_options) - branch_name = getattr(rev_options, "branch_name", None) - logger.debug("Rev options %s, branch_name %s", rev_options, branch_name) - if branch_name is None: - # Only do a checkout if the current commit id doesn't match - # the requested revision. - if not self.is_commit_id_equal(dest, rev_options.rev): - cmd_args = make_command( - "checkout", - "-q", - rev_options.to_args(), - ) - self.run_command(cmd_args, cwd=dest) - elif self.get_current_branch(dest) != branch_name: - # Then a specific branch was requested, and that branch - # is not yet checked out. - track_branch = f"origin/{branch_name}" - cmd_args = [ - "checkout", - "-b", - branch_name, - "--track", - track_branch, - ] - self.run_command(cmd_args, cwd=dest) - else: - sha = self.get_revision(dest) - rev_options = rev_options.make_new(sha) - - logger.info("Resolved %s to commit %s", url, rev_options.rev) - - #: repo may contain submodules - self.update_submodules(dest) - - def switch(self, dest: str, url: HiddenText, rev_options: RevOptions) -> None: - self.run_command( - make_command("config", "remote.origin.url", url), - cwd=dest, - ) - cmd_args = make_command("checkout", "-q", rev_options.to_args()) - self.run_command(cmd_args, cwd=dest) - - self.update_submodules(dest) - - def update(self, dest: str, url: HiddenText, rev_options: RevOptions) -> None: - # First fetch changes from the default remote - if self.get_git_version() >= (1, 9): - # fetch tags in addition to everything else - self.run_command(["fetch", "-q", "--tags"], cwd=dest) - else: - self.run_command(["fetch", "-q"], cwd=dest) - # Then reset to wanted revision (maybe even origin/master) - rev_options = self.resolve_revision(dest, url, rev_options) - cmd_args = make_command("reset", "--hard", "-q", rev_options.to_args()) - self.run_command(cmd_args, cwd=dest) - #: update submodules - self.update_submodules(dest) - - @classmethod - def get_remote_url(cls, location: str) -> str: - """ - Return URL of the first remote encountered. - - Raises RemoteNotFoundError if the repository does not have a remote - url configured. - """ - # We need to pass 1 for extra_ok_returncodes since the command - # exits with return code 1 if there are no matching lines. - stdout = cls.run_command( - ["config", "--get-regexp", r"remote\..*\.url"], - extra_ok_returncodes=(1,), - show_stdout=False, - stdout_only=True, - cwd=location, - ) - remotes = stdout.splitlines() - try: - found_remote = remotes[0] - except IndexError: - raise RemoteNotFoundError - - for remote in remotes: - if remote.startswith("remote.origin.url "): - found_remote = remote - break - url = found_remote.split(" ")[1] - return cls._git_remote_to_pip_url(url.strip()) - - @staticmethod - def _git_remote_to_pip_url(url: str) -> str: - """ - Convert a remote url from what git uses to what pip accepts. - - There are 3 legal forms **url** may take: - - 1. A fully qualified url: ssh://git@example.com/foo/bar.git - 2. A local project.git folder: /path/to/bare/repository.git - 3. SCP shorthand for form 1: git@example.com:foo/bar.git - - Form 1 is output as-is. Form 2 must be converted to URI and form 3 must - be converted to form 1. - - See the corresponding test test_git_remote_url_to_pip() for examples of - sample inputs/outputs. - """ - if re.match(r"\w+://", url): - # This is already valid. Pass it though as-is. - return url - if os.path.exists(url): - # A local bare remote (git clone --mirror). - # Needs a file:// prefix. - return pathlib.PurePath(url).as_uri() - scp_match = SCP_REGEX.match(url) - if scp_match: - # Add an ssh:// prefix and replace the ':' with a '/'. - return scp_match.expand(r"ssh://\1\2/\3") - # Otherwise, bail out. - raise RemoteNotValidError(url) - - @classmethod - def has_commit(cls, location: str, rev: str) -> bool: - """ - Check if rev is a commit that is available in the local repository. - """ - try: - cls.run_command( - ["rev-parse", "-q", "--verify", "sha^" + rev], - cwd=location, - log_failed_cmd=False, - ) - except InstallationError: - return False - else: - return True - - @classmethod - def get_revision(cls, location: str, rev: Optional[str] = None) -> str: - if rev is None: - rev = "HEAD" - current_rev = cls.run_command( - ["rev-parse", rev], - show_stdout=False, - stdout_only=True, - cwd=location, - ) - return current_rev.strip() - - @classmethod - def get_subdirectory(cls, location: str) -> Optional[str]: - """ - Return the path to Python project root, relative to the repo root. - Return None if the project root is in the repo root. - """ - # find the repo root - git_dir = cls.run_command( - ["rev-parse", "--git-dir"], - show_stdout=False, - stdout_only=True, - cwd=location, - ).strip() - if not os.path.isabs(git_dir): - git_dir = os.path.join(location, git_dir) - repo_root = os.path.abspath(os.path.join(git_dir, "..")) - return find_path_to_project_root_from_repo_root(location, repo_root) - - @classmethod - def get_url_rev_and_auth(cls, url: str) -> Tuple[str, Optional[str], AuthInfo]: - """ - Prefixes stub URLs like 'user@hostname:user/repo.git' with 'ssh://'. - That's required because although they use SSH they sometimes don't - work with a ssh:// scheme (e.g. GitHub). But we need a scheme for - parsing. Hence we remove it again afterwards and return it as a stub. - """ - # Works around an apparent Git bug - # (see https://article.gmane.org/gmane.comp.version-control.git/146500) - scheme, netloc, path, query, fragment = urlsplit(url) - if scheme.endswith("file"): - initial_slashes = path[: -len(path.lstrip("/"))] - newpath = initial_slashes + urllib.request.url2pathname(path).replace( - "\\", "/" - ).lstrip("/") - after_plus = scheme.find("+") + 1 - url = scheme[:after_plus] + urlunsplit( - (scheme[after_plus:], netloc, newpath, query, fragment), - ) - - if "://" not in url: - assert "file:" not in url - url = url.replace("git+", "git+ssh://") - url, rev, user_pass = super().get_url_rev_and_auth(url) - url = url.replace("ssh://", "") - else: - url, rev, user_pass = super().get_url_rev_and_auth(url) - - return url, rev, user_pass - - @classmethod - def update_submodules(cls, location: str) -> None: - if not os.path.exists(os.path.join(location, ".gitmodules")): - return - cls.run_command( - ["submodule", "update", "--init", "--recursive", "-q"], - cwd=location, - ) - - @classmethod - def get_repository_root(cls, location: str) -> Optional[str]: - loc = super().get_repository_root(location) - if loc: - return loc - try: - r = cls.run_command( - ["rev-parse", "--show-toplevel"], - cwd=location, - show_stdout=False, - stdout_only=True, - on_returncode="raise", - log_failed_cmd=False, - ) - except BadCommand: - logger.debug( - "could not determine if %s is under git control " - "because git is not available", - location, - ) - return None - except InstallationError: - return None - return os.path.normpath(r.rstrip("\r\n")) - - @staticmethod - def should_add_vcs_url_prefix(repo_url: str) -> bool: - """In either https or ssh form, requirements must be prefixed with git+.""" - return True - - -vcs.register(Git) diff --git a/venv/lib/python3.11/site-packages/pip/_internal/vcs/mercurial.py b/venv/lib/python3.11/site-packages/pip/_internal/vcs/mercurial.py deleted file mode 100644 index e440c12..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/vcs/mercurial.py +++ /dev/null @@ -1,163 +0,0 @@ -import configparser -import logging -import os -from typing import List, Optional, Tuple - -from pip._internal.exceptions import BadCommand, InstallationError -from pip._internal.utils.misc import HiddenText, display_path -from pip._internal.utils.subprocess import make_command -from pip._internal.utils.urls import path_to_url -from pip._internal.vcs.versioncontrol import ( - RevOptions, - VersionControl, - find_path_to_project_root_from_repo_root, - vcs, -) - -logger = logging.getLogger(__name__) - - -class Mercurial(VersionControl): - name = "hg" - dirname = ".hg" - repo_name = "clone" - schemes = ( - "hg+file", - "hg+http", - "hg+https", - "hg+ssh", - "hg+static-http", - ) - - @staticmethod - def get_base_rev_args(rev: str) -> List[str]: - return [f"-r={rev}"] - - def fetch_new( - self, dest: str, url: HiddenText, rev_options: RevOptions, verbosity: int - ) -> None: - rev_display = rev_options.to_display() - logger.info( - "Cloning hg %s%s to %s", - url, - rev_display, - display_path(dest), - ) - if verbosity <= 0: - flags: Tuple[str, ...] = ("--quiet",) - elif verbosity == 1: - flags = () - elif verbosity == 2: - flags = ("--verbose",) - else: - flags = ("--verbose", "--debug") - self.run_command(make_command("clone", "--noupdate", *flags, url, dest)) - self.run_command( - make_command("update", *flags, rev_options.to_args()), - cwd=dest, - ) - - def switch(self, dest: str, url: HiddenText, rev_options: RevOptions) -> None: - repo_config = os.path.join(dest, self.dirname, "hgrc") - config = configparser.RawConfigParser() - try: - config.read(repo_config) - config.set("paths", "default", url.secret) - with open(repo_config, "w") as config_file: - config.write(config_file) - except (OSError, configparser.NoSectionError) as exc: - logger.warning("Could not switch Mercurial repository to %s: %s", url, exc) - else: - cmd_args = make_command("update", "-q", rev_options.to_args()) - self.run_command(cmd_args, cwd=dest) - - def update(self, dest: str, url: HiddenText, rev_options: RevOptions) -> None: - self.run_command(["pull", "-q"], cwd=dest) - cmd_args = make_command("update", "-q", rev_options.to_args()) - self.run_command(cmd_args, cwd=dest) - - @classmethod - def get_remote_url(cls, location: str) -> str: - url = cls.run_command( - ["showconfig", "paths.default"], - show_stdout=False, - stdout_only=True, - cwd=location, - ).strip() - if cls._is_local_repository(url): - url = path_to_url(url) - return url.strip() - - @classmethod - def get_revision(cls, location: str) -> str: - """ - Return the repository-local changeset revision number, as an integer. - """ - current_revision = cls.run_command( - ["parents", "--template={rev}"], - show_stdout=False, - stdout_only=True, - cwd=location, - ).strip() - return current_revision - - @classmethod - def get_requirement_revision(cls, location: str) -> str: - """ - Return the changeset identification hash, as a 40-character - hexadecimal string - """ - current_rev_hash = cls.run_command( - ["parents", "--template={node}"], - show_stdout=False, - stdout_only=True, - cwd=location, - ).strip() - return current_rev_hash - - @classmethod - def is_commit_id_equal(cls, dest: str, name: Optional[str]) -> bool: - """Always assume the versions don't match""" - return False - - @classmethod - def get_subdirectory(cls, location: str) -> Optional[str]: - """ - Return the path to Python project root, relative to the repo root. - Return None if the project root is in the repo root. - """ - # find the repo root - repo_root = cls.run_command( - ["root"], show_stdout=False, stdout_only=True, cwd=location - ).strip() - if not os.path.isabs(repo_root): - repo_root = os.path.abspath(os.path.join(location, repo_root)) - return find_path_to_project_root_from_repo_root(location, repo_root) - - @classmethod - def get_repository_root(cls, location: str) -> Optional[str]: - loc = super().get_repository_root(location) - if loc: - return loc - try: - r = cls.run_command( - ["root"], - cwd=location, - show_stdout=False, - stdout_only=True, - on_returncode="raise", - log_failed_cmd=False, - ) - except BadCommand: - logger.debug( - "could not determine if %s is under hg control " - "because hg is not available", - location, - ) - return None - except InstallationError: - return None - return os.path.normpath(r.rstrip("\r\n")) - - -vcs.register(Mercurial) diff --git a/venv/lib/python3.11/site-packages/pip/_internal/vcs/subversion.py b/venv/lib/python3.11/site-packages/pip/_internal/vcs/subversion.py deleted file mode 100644 index 16d93a6..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/vcs/subversion.py +++ /dev/null @@ -1,324 +0,0 @@ -import logging -import os -import re -from typing import List, Optional, Tuple - -from pip._internal.utils.misc import ( - HiddenText, - display_path, - is_console_interactive, - is_installable_dir, - split_auth_from_netloc, -) -from pip._internal.utils.subprocess import CommandArgs, make_command -from pip._internal.vcs.versioncontrol import ( - AuthInfo, - RemoteNotFoundError, - RevOptions, - VersionControl, - vcs, -) - -logger = logging.getLogger(__name__) - -_svn_xml_url_re = re.compile('url="([^"]+)"') -_svn_rev_re = re.compile(r'committed-rev="(\d+)"') -_svn_info_xml_rev_re = re.compile(r'\s*revision="(\d+)"') -_svn_info_xml_url_re = re.compile(r"(.*)") - - -class Subversion(VersionControl): - name = "svn" - dirname = ".svn" - repo_name = "checkout" - schemes = ("svn+ssh", "svn+http", "svn+https", "svn+svn", "svn+file") - - @classmethod - def should_add_vcs_url_prefix(cls, remote_url: str) -> bool: - return True - - @staticmethod - def get_base_rev_args(rev: str) -> List[str]: - return ["-r", rev] - - @classmethod - def get_revision(cls, location: str) -> str: - """ - Return the maximum revision for all files under a given location - """ - # Note: taken from setuptools.command.egg_info - revision = 0 - - for base, dirs, _ in os.walk(location): - if cls.dirname not in dirs: - dirs[:] = [] - continue # no sense walking uncontrolled subdirs - dirs.remove(cls.dirname) - entries_fn = os.path.join(base, cls.dirname, "entries") - if not os.path.exists(entries_fn): - # FIXME: should we warn? - continue - - dirurl, localrev = cls._get_svn_url_rev(base) - - if base == location: - assert dirurl is not None - base = dirurl + "/" # save the root url - elif not dirurl or not dirurl.startswith(base): - dirs[:] = [] - continue # not part of the same svn tree, skip it - revision = max(revision, localrev) - return str(revision) - - @classmethod - def get_netloc_and_auth( - cls, netloc: str, scheme: str - ) -> Tuple[str, Tuple[Optional[str], Optional[str]]]: - """ - This override allows the auth information to be passed to svn via the - --username and --password options instead of via the URL. - """ - if scheme == "ssh": - # The --username and --password options can't be used for - # svn+ssh URLs, so keep the auth information in the URL. - return super().get_netloc_and_auth(netloc, scheme) - - return split_auth_from_netloc(netloc) - - @classmethod - def get_url_rev_and_auth(cls, url: str) -> Tuple[str, Optional[str], AuthInfo]: - # hotfix the URL scheme after removing svn+ from svn+ssh:// re-add it - url, rev, user_pass = super().get_url_rev_and_auth(url) - if url.startswith("ssh://"): - url = "svn+" + url - return url, rev, user_pass - - @staticmethod - def make_rev_args( - username: Optional[str], password: Optional[HiddenText] - ) -> CommandArgs: - extra_args: CommandArgs = [] - if username: - extra_args += ["--username", username] - if password: - extra_args += ["--password", password] - - return extra_args - - @classmethod - def get_remote_url(cls, location: str) -> str: - # In cases where the source is in a subdirectory, we have to look up in - # the location until we find a valid project root. - orig_location = location - while not is_installable_dir(location): - last_location = location - location = os.path.dirname(location) - if location == last_location: - # We've traversed up to the root of the filesystem without - # finding a Python project. - logger.warning( - "Could not find Python project for directory %s (tried all " - "parent directories)", - orig_location, - ) - raise RemoteNotFoundError - - url, _rev = cls._get_svn_url_rev(location) - if url is None: - raise RemoteNotFoundError - - return url - - @classmethod - def _get_svn_url_rev(cls, location: str) -> Tuple[Optional[str], int]: - from pip._internal.exceptions import InstallationError - - entries_path = os.path.join(location, cls.dirname, "entries") - if os.path.exists(entries_path): - with open(entries_path) as f: - data = f.read() - else: # subversion >= 1.7 does not have the 'entries' file - data = "" - - url = None - if data.startswith("8") or data.startswith("9") or data.startswith("10"): - entries = list(map(str.splitlines, data.split("\n\x0c\n"))) - del entries[0][0] # get rid of the '8' - url = entries[0][3] - revs = [int(d[9]) for d in entries if len(d) > 9 and d[9]] + [0] - elif data.startswith("= 1.7 - # Note that using get_remote_call_options is not necessary here - # because `svn info` is being run against a local directory. - # We don't need to worry about making sure interactive mode - # is being used to prompt for passwords, because passwords - # are only potentially needed for remote server requests. - xml = cls.run_command( - ["info", "--xml", location], - show_stdout=False, - stdout_only=True, - ) - match = _svn_info_xml_url_re.search(xml) - assert match is not None - url = match.group(1) - revs = [int(m.group(1)) for m in _svn_info_xml_rev_re.finditer(xml)] - except InstallationError: - url, revs = None, [] - - if revs: - rev = max(revs) - else: - rev = 0 - - return url, rev - - @classmethod - def is_commit_id_equal(cls, dest: str, name: Optional[str]) -> bool: - """Always assume the versions don't match""" - return False - - def __init__(self, use_interactive: Optional[bool] = None) -> None: - if use_interactive is None: - use_interactive = is_console_interactive() - self.use_interactive = use_interactive - - # This member is used to cache the fetched version of the current - # ``svn`` client. - # Special value definitions: - # None: Not evaluated yet. - # Empty tuple: Could not parse version. - self._vcs_version: Optional[Tuple[int, ...]] = None - - super().__init__() - - def call_vcs_version(self) -> Tuple[int, ...]: - """Query the version of the currently installed Subversion client. - - :return: A tuple containing the parts of the version information or - ``()`` if the version returned from ``svn`` could not be parsed. - :raises: BadCommand: If ``svn`` is not installed. - """ - # Example versions: - # svn, version 1.10.3 (r1842928) - # compiled Feb 25 2019, 14:20:39 on x86_64-apple-darwin17.0.0 - # svn, version 1.7.14 (r1542130) - # compiled Mar 28 2018, 08:49:13 on x86_64-pc-linux-gnu - # svn, version 1.12.0-SlikSvn (SlikSvn/1.12.0) - # compiled May 28 2019, 13:44:56 on x86_64-microsoft-windows6.2 - version_prefix = "svn, version " - version = self.run_command(["--version"], show_stdout=False, stdout_only=True) - if not version.startswith(version_prefix): - return () - - version = version[len(version_prefix) :].split()[0] - version_list = version.partition("-")[0].split(".") - try: - parsed_version = tuple(map(int, version_list)) - except ValueError: - return () - - return parsed_version - - def get_vcs_version(self) -> Tuple[int, ...]: - """Return the version of the currently installed Subversion client. - - If the version of the Subversion client has already been queried, - a cached value will be used. - - :return: A tuple containing the parts of the version information or - ``()`` if the version returned from ``svn`` could not be parsed. - :raises: BadCommand: If ``svn`` is not installed. - """ - if self._vcs_version is not None: - # Use cached version, if available. - # If parsing the version failed previously (empty tuple), - # do not attempt to parse it again. - return self._vcs_version - - vcs_version = self.call_vcs_version() - self._vcs_version = vcs_version - return vcs_version - - def get_remote_call_options(self) -> CommandArgs: - """Return options to be used on calls to Subversion that contact the server. - - These options are applicable for the following ``svn`` subcommands used - in this class. - - - checkout - - switch - - update - - :return: A list of command line arguments to pass to ``svn``. - """ - if not self.use_interactive: - # --non-interactive switch is available since Subversion 0.14.4. - # Subversion < 1.8 runs in interactive mode by default. - return ["--non-interactive"] - - svn_version = self.get_vcs_version() - # By default, Subversion >= 1.8 runs in non-interactive mode if - # stdin is not a TTY. Since that is how pip invokes SVN, in - # call_subprocess(), pip must pass --force-interactive to ensure - # the user can be prompted for a password, if required. - # SVN added the --force-interactive option in SVN 1.8. Since - # e.g. RHEL/CentOS 7, which is supported until 2024, ships with - # SVN 1.7, pip should continue to support SVN 1.7. Therefore, pip - # can't safely add the option if the SVN version is < 1.8 (or unknown). - if svn_version >= (1, 8): - return ["--force-interactive"] - - return [] - - def fetch_new( - self, dest: str, url: HiddenText, rev_options: RevOptions, verbosity: int - ) -> None: - rev_display = rev_options.to_display() - logger.info( - "Checking out %s%s to %s", - url, - rev_display, - display_path(dest), - ) - if verbosity <= 0: - flag = "--quiet" - else: - flag = "" - cmd_args = make_command( - "checkout", - flag, - self.get_remote_call_options(), - rev_options.to_args(), - url, - dest, - ) - self.run_command(cmd_args) - - def switch(self, dest: str, url: HiddenText, rev_options: RevOptions) -> None: - cmd_args = make_command( - "switch", - self.get_remote_call_options(), - rev_options.to_args(), - url, - dest, - ) - self.run_command(cmd_args) - - def update(self, dest: str, url: HiddenText, rev_options: RevOptions) -> None: - cmd_args = make_command( - "update", - self.get_remote_call_options(), - rev_options.to_args(), - dest, - ) - self.run_command(cmd_args) - - -vcs.register(Subversion) diff --git a/venv/lib/python3.11/site-packages/pip/_internal/vcs/versioncontrol.py b/venv/lib/python3.11/site-packages/pip/_internal/vcs/versioncontrol.py deleted file mode 100644 index 02bbf68..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/vcs/versioncontrol.py +++ /dev/null @@ -1,705 +0,0 @@ -"""Handles all VCS (version control) support""" - -import logging -import os -import shutil -import sys -import urllib.parse -from typing import ( - TYPE_CHECKING, - Any, - Dict, - Iterable, - Iterator, - List, - Mapping, - Optional, - Tuple, - Type, - Union, -) - -from pip._internal.cli.spinners import SpinnerInterface -from pip._internal.exceptions import BadCommand, InstallationError -from pip._internal.utils.misc import ( - HiddenText, - ask_path_exists, - backup_dir, - display_path, - hide_url, - hide_value, - is_installable_dir, - rmtree, -) -from pip._internal.utils.subprocess import ( - CommandArgs, - call_subprocess, - format_command_args, - make_command, -) -from pip._internal.utils.urls import get_url_scheme - -if TYPE_CHECKING: - # Literal was introduced in Python 3.8. - # - # TODO: Remove `if TYPE_CHECKING` when dropping support for Python 3.7. - from typing import Literal - - -__all__ = ["vcs"] - - -logger = logging.getLogger(__name__) - -AuthInfo = Tuple[Optional[str], Optional[str]] - - -def is_url(name: str) -> bool: - """ - Return true if the name looks like a URL. - """ - scheme = get_url_scheme(name) - if scheme is None: - return False - return scheme in ["http", "https", "file", "ftp"] + vcs.all_schemes - - -def make_vcs_requirement_url( - repo_url: str, rev: str, project_name: str, subdir: Optional[str] = None -) -> str: - """ - Return the URL for a VCS requirement. - - Args: - repo_url: the remote VCS url, with any needed VCS prefix (e.g. "git+"). - project_name: the (unescaped) project name. - """ - egg_project_name = project_name.replace("-", "_") - req = f"{repo_url}@{rev}#egg={egg_project_name}" - if subdir: - req += f"&subdirectory={subdir}" - - return req - - -def find_path_to_project_root_from_repo_root( - location: str, repo_root: str -) -> Optional[str]: - """ - Find the the Python project's root by searching up the filesystem from - `location`. Return the path to project root relative to `repo_root`. - Return None if the project root is `repo_root`, or cannot be found. - """ - # find project root. - orig_location = location - while not is_installable_dir(location): - last_location = location - location = os.path.dirname(location) - if location == last_location: - # We've traversed up to the root of the filesystem without - # finding a Python project. - logger.warning( - "Could not find a Python project for directory %s (tried all " - "parent directories)", - orig_location, - ) - return None - - if os.path.samefile(repo_root, location): - return None - - return os.path.relpath(location, repo_root) - - -class RemoteNotFoundError(Exception): - pass - - -class RemoteNotValidError(Exception): - def __init__(self, url: str): - super().__init__(url) - self.url = url - - -class RevOptions: - - """ - Encapsulates a VCS-specific revision to install, along with any VCS - install options. - - Instances of this class should be treated as if immutable. - """ - - def __init__( - self, - vc_class: Type["VersionControl"], - rev: Optional[str] = None, - extra_args: Optional[CommandArgs] = None, - ) -> None: - """ - Args: - vc_class: a VersionControl subclass. - rev: the name of the revision to install. - extra_args: a list of extra options. - """ - if extra_args is None: - extra_args = [] - - self.extra_args = extra_args - self.rev = rev - self.vc_class = vc_class - self.branch_name: Optional[str] = None - - def __repr__(self) -> str: - return f"" - - @property - def arg_rev(self) -> Optional[str]: - if self.rev is None: - return self.vc_class.default_arg_rev - - return self.rev - - def to_args(self) -> CommandArgs: - """ - Return the VCS-specific command arguments. - """ - args: CommandArgs = [] - rev = self.arg_rev - if rev is not None: - args += self.vc_class.get_base_rev_args(rev) - args += self.extra_args - - return args - - def to_display(self) -> str: - if not self.rev: - return "" - - return f" (to revision {self.rev})" - - def make_new(self, rev: str) -> "RevOptions": - """ - Make a copy of the current instance, but with a new rev. - - Args: - rev: the name of the revision for the new object. - """ - return self.vc_class.make_rev_options(rev, extra_args=self.extra_args) - - -class VcsSupport: - _registry: Dict[str, "VersionControl"] = {} - schemes = ["ssh", "git", "hg", "bzr", "sftp", "svn"] - - def __init__(self) -> None: - # Register more schemes with urlparse for various version control - # systems - urllib.parse.uses_netloc.extend(self.schemes) - super().__init__() - - def __iter__(self) -> Iterator[str]: - return self._registry.__iter__() - - @property - def backends(self) -> List["VersionControl"]: - return list(self._registry.values()) - - @property - def dirnames(self) -> List[str]: - return [backend.dirname for backend in self.backends] - - @property - def all_schemes(self) -> List[str]: - schemes: List[str] = [] - for backend in self.backends: - schemes.extend(backend.schemes) - return schemes - - def register(self, cls: Type["VersionControl"]) -> None: - if not hasattr(cls, "name"): - logger.warning("Cannot register VCS %s", cls.__name__) - return - if cls.name not in self._registry: - self._registry[cls.name] = cls() - logger.debug("Registered VCS backend: %s", cls.name) - - def unregister(self, name: str) -> None: - if name in self._registry: - del self._registry[name] - - def get_backend_for_dir(self, location: str) -> Optional["VersionControl"]: - """ - Return a VersionControl object if a repository of that type is found - at the given directory. - """ - vcs_backends = {} - for vcs_backend in self._registry.values(): - repo_path = vcs_backend.get_repository_root(location) - if not repo_path: - continue - logger.debug("Determine that %s uses VCS: %s", location, vcs_backend.name) - vcs_backends[repo_path] = vcs_backend - - if not vcs_backends: - return None - - # Choose the VCS in the inner-most directory. Since all repository - # roots found here would be either `location` or one of its - # parents, the longest path should have the most path components, - # i.e. the backend representing the inner-most repository. - inner_most_repo_path = max(vcs_backends, key=len) - return vcs_backends[inner_most_repo_path] - - def get_backend_for_scheme(self, scheme: str) -> Optional["VersionControl"]: - """ - Return a VersionControl object or None. - """ - for vcs_backend in self._registry.values(): - if scheme in vcs_backend.schemes: - return vcs_backend - return None - - def get_backend(self, name: str) -> Optional["VersionControl"]: - """ - Return a VersionControl object or None. - """ - name = name.lower() - return self._registry.get(name) - - -vcs = VcsSupport() - - -class VersionControl: - name = "" - dirname = "" - repo_name = "" - # List of supported schemes for this Version Control - schemes: Tuple[str, ...] = () - # Iterable of environment variable names to pass to call_subprocess(). - unset_environ: Tuple[str, ...] = () - default_arg_rev: Optional[str] = None - - @classmethod - def should_add_vcs_url_prefix(cls, remote_url: str) -> bool: - """ - Return whether the vcs prefix (e.g. "git+") should be added to a - repository's remote url when used in a requirement. - """ - return not remote_url.lower().startswith(f"{cls.name}:") - - @classmethod - def get_subdirectory(cls, location: str) -> Optional[str]: - """ - Return the path to Python project root, relative to the repo root. - Return None if the project root is in the repo root. - """ - return None - - @classmethod - def get_requirement_revision(cls, repo_dir: str) -> str: - """ - Return the revision string that should be used in a requirement. - """ - return cls.get_revision(repo_dir) - - @classmethod - def get_src_requirement(cls, repo_dir: str, project_name: str) -> str: - """ - Return the requirement string to use to redownload the files - currently at the given repository directory. - - Args: - project_name: the (unescaped) project name. - - The return value has a form similar to the following: - - {repository_url}@{revision}#egg={project_name} - """ - repo_url = cls.get_remote_url(repo_dir) - - if cls.should_add_vcs_url_prefix(repo_url): - repo_url = f"{cls.name}+{repo_url}" - - revision = cls.get_requirement_revision(repo_dir) - subdir = cls.get_subdirectory(repo_dir) - req = make_vcs_requirement_url(repo_url, revision, project_name, subdir=subdir) - - return req - - @staticmethod - def get_base_rev_args(rev: str) -> List[str]: - """ - Return the base revision arguments for a vcs command. - - Args: - rev: the name of a revision to install. Cannot be None. - """ - raise NotImplementedError - - def is_immutable_rev_checkout(self, url: str, dest: str) -> bool: - """ - Return true if the commit hash checked out at dest matches - the revision in url. - - Always return False, if the VCS does not support immutable commit - hashes. - - This method does not check if there are local uncommitted changes - in dest after checkout, as pip currently has no use case for that. - """ - return False - - @classmethod - def make_rev_options( - cls, rev: Optional[str] = None, extra_args: Optional[CommandArgs] = None - ) -> RevOptions: - """ - Return a RevOptions object. - - Args: - rev: the name of a revision to install. - extra_args: a list of extra options. - """ - return RevOptions(cls, rev, extra_args=extra_args) - - @classmethod - def _is_local_repository(cls, repo: str) -> bool: - """ - posix absolute paths start with os.path.sep, - win32 ones start with drive (like c:\\folder) - """ - drive, tail = os.path.splitdrive(repo) - return repo.startswith(os.path.sep) or bool(drive) - - @classmethod - def get_netloc_and_auth( - cls, netloc: str, scheme: str - ) -> Tuple[str, Tuple[Optional[str], Optional[str]]]: - """ - Parse the repository URL's netloc, and return the new netloc to use - along with auth information. - - Args: - netloc: the original repository URL netloc. - scheme: the repository URL's scheme without the vcs prefix. - - This is mainly for the Subversion class to override, so that auth - information can be provided via the --username and --password options - instead of through the URL. For other subclasses like Git without - such an option, auth information must stay in the URL. - - Returns: (netloc, (username, password)). - """ - return netloc, (None, None) - - @classmethod - def get_url_rev_and_auth(cls, url: str) -> Tuple[str, Optional[str], AuthInfo]: - """ - Parse the repository URL to use, and return the URL, revision, - and auth info to use. - - Returns: (url, rev, (username, password)). - """ - scheme, netloc, path, query, frag = urllib.parse.urlsplit(url) - if "+" not in scheme: - raise ValueError( - "Sorry, {!r} is a malformed VCS url. " - "The format is +://, " - "e.g. svn+http://myrepo/svn/MyApp#egg=MyApp".format(url) - ) - # Remove the vcs prefix. - scheme = scheme.split("+", 1)[1] - netloc, user_pass = cls.get_netloc_and_auth(netloc, scheme) - rev = None - if "@" in path: - path, rev = path.rsplit("@", 1) - if not rev: - raise InstallationError( - "The URL {!r} has an empty revision (after @) " - "which is not supported. Include a revision after @ " - "or remove @ from the URL.".format(url) - ) - url = urllib.parse.urlunsplit((scheme, netloc, path, query, "")) - return url, rev, user_pass - - @staticmethod - def make_rev_args( - username: Optional[str], password: Optional[HiddenText] - ) -> CommandArgs: - """ - Return the RevOptions "extra arguments" to use in obtain(). - """ - return [] - - def get_url_rev_options(self, url: HiddenText) -> Tuple[HiddenText, RevOptions]: - """ - Return the URL and RevOptions object to use in obtain(), - as a tuple (url, rev_options). - """ - secret_url, rev, user_pass = self.get_url_rev_and_auth(url.secret) - username, secret_password = user_pass - password: Optional[HiddenText] = None - if secret_password is not None: - password = hide_value(secret_password) - extra_args = self.make_rev_args(username, password) - rev_options = self.make_rev_options(rev, extra_args=extra_args) - - return hide_url(secret_url), rev_options - - @staticmethod - def normalize_url(url: str) -> str: - """ - Normalize a URL for comparison by unquoting it and removing any - trailing slash. - """ - return urllib.parse.unquote(url).rstrip("/") - - @classmethod - def compare_urls(cls, url1: str, url2: str) -> bool: - """ - Compare two repo URLs for identity, ignoring incidental differences. - """ - return cls.normalize_url(url1) == cls.normalize_url(url2) - - def fetch_new( - self, dest: str, url: HiddenText, rev_options: RevOptions, verbosity: int - ) -> None: - """ - Fetch a revision from a repository, in the case that this is the - first fetch from the repository. - - Args: - dest: the directory to fetch the repository to. - rev_options: a RevOptions object. - verbosity: verbosity level. - """ - raise NotImplementedError - - def switch(self, dest: str, url: HiddenText, rev_options: RevOptions) -> None: - """ - Switch the repo at ``dest`` to point to ``URL``. - - Args: - rev_options: a RevOptions object. - """ - raise NotImplementedError - - def update(self, dest: str, url: HiddenText, rev_options: RevOptions) -> None: - """ - Update an already-existing repo to the given ``rev_options``. - - Args: - rev_options: a RevOptions object. - """ - raise NotImplementedError - - @classmethod - def is_commit_id_equal(cls, dest: str, name: Optional[str]) -> bool: - """ - Return whether the id of the current commit equals the given name. - - Args: - dest: the repository directory. - name: a string name. - """ - raise NotImplementedError - - def obtain(self, dest: str, url: HiddenText, verbosity: int) -> None: - """ - Install or update in editable mode the package represented by this - VersionControl object. - - :param dest: the repository directory in which to install or update. - :param url: the repository URL starting with a vcs prefix. - :param verbosity: verbosity level. - """ - url, rev_options = self.get_url_rev_options(url) - - if not os.path.exists(dest): - self.fetch_new(dest, url, rev_options, verbosity=verbosity) - return - - rev_display = rev_options.to_display() - if self.is_repository_directory(dest): - existing_url = self.get_remote_url(dest) - if self.compare_urls(existing_url, url.secret): - logger.debug( - "%s in %s exists, and has correct URL (%s)", - self.repo_name.title(), - display_path(dest), - url, - ) - if not self.is_commit_id_equal(dest, rev_options.rev): - logger.info( - "Updating %s %s%s", - display_path(dest), - self.repo_name, - rev_display, - ) - self.update(dest, url, rev_options) - else: - logger.info("Skipping because already up-to-date.") - return - - logger.warning( - "%s %s in %s exists with URL %s", - self.name, - self.repo_name, - display_path(dest), - existing_url, - ) - prompt = ("(s)witch, (i)gnore, (w)ipe, (b)ackup ", ("s", "i", "w", "b")) - else: - logger.warning( - "Directory %s already exists, and is not a %s %s.", - dest, - self.name, - self.repo_name, - ) - # https://github.com/python/mypy/issues/1174 - prompt = ("(i)gnore, (w)ipe, (b)ackup ", ("i", "w", "b")) # type: ignore - - logger.warning( - "The plan is to install the %s repository %s", - self.name, - url, - ) - response = ask_path_exists("What to do? {}".format(prompt[0]), prompt[1]) - - if response == "a": - sys.exit(-1) - - if response == "w": - logger.warning("Deleting %s", display_path(dest)) - rmtree(dest) - self.fetch_new(dest, url, rev_options, verbosity=verbosity) - return - - if response == "b": - dest_dir = backup_dir(dest) - logger.warning("Backing up %s to %s", display_path(dest), dest_dir) - shutil.move(dest, dest_dir) - self.fetch_new(dest, url, rev_options, verbosity=verbosity) - return - - # Do nothing if the response is "i". - if response == "s": - logger.info( - "Switching %s %s to %s%s", - self.repo_name, - display_path(dest), - url, - rev_display, - ) - self.switch(dest, url, rev_options) - - def unpack(self, location: str, url: HiddenText, verbosity: int) -> None: - """ - Clean up current location and download the url repository - (and vcs infos) into location - - :param url: the repository URL starting with a vcs prefix. - :param verbosity: verbosity level. - """ - if os.path.exists(location): - rmtree(location) - self.obtain(location, url=url, verbosity=verbosity) - - @classmethod - def get_remote_url(cls, location: str) -> str: - """ - Return the url used at location - - Raises RemoteNotFoundError if the repository does not have a remote - url configured. - """ - raise NotImplementedError - - @classmethod - def get_revision(cls, location: str) -> str: - """ - Return the current commit id of the files at the given location. - """ - raise NotImplementedError - - @classmethod - def run_command( - cls, - cmd: Union[List[str], CommandArgs], - show_stdout: bool = True, - cwd: Optional[str] = None, - on_returncode: 'Literal["raise", "warn", "ignore"]' = "raise", - extra_ok_returncodes: Optional[Iterable[int]] = None, - command_desc: Optional[str] = None, - extra_environ: Optional[Mapping[str, Any]] = None, - spinner: Optional[SpinnerInterface] = None, - log_failed_cmd: bool = True, - stdout_only: bool = False, - ) -> str: - """ - Run a VCS subcommand - This is simply a wrapper around call_subprocess that adds the VCS - command name, and checks that the VCS is available - """ - cmd = make_command(cls.name, *cmd) - if command_desc is None: - command_desc = format_command_args(cmd) - try: - return call_subprocess( - cmd, - show_stdout, - cwd, - on_returncode=on_returncode, - extra_ok_returncodes=extra_ok_returncodes, - command_desc=command_desc, - extra_environ=extra_environ, - unset_environ=cls.unset_environ, - spinner=spinner, - log_failed_cmd=log_failed_cmd, - stdout_only=stdout_only, - ) - except FileNotFoundError: - # errno.ENOENT = no such file or directory - # In other words, the VCS executable isn't available - raise BadCommand( - f"Cannot find command {cls.name!r} - do you have " - f"{cls.name!r} installed and in your PATH?" - ) - except PermissionError: - # errno.EACCES = Permission denied - # This error occurs, for instance, when the command is installed - # only for another user. So, the current user don't have - # permission to call the other user command. - raise BadCommand( - f"No permission to execute {cls.name!r} - install it " - f"locally, globally (ask admin), or check your PATH. " - f"See possible solutions at " - f"https://pip.pypa.io/en/latest/reference/pip_freeze/" - f"#fixing-permission-denied." - ) - - @classmethod - def is_repository_directory(cls, path: str) -> bool: - """ - Return whether a directory path is a repository directory. - """ - logger.debug("Checking in %s for %s (%s)...", path, cls.dirname, cls.name) - return os.path.exists(os.path.join(path, cls.dirname)) - - @classmethod - def get_repository_root(cls, location: str) -> Optional[str]: - """ - Return the "root" (top-level) directory controlled by the vcs, - or `None` if the directory is not in any. - - It is meant to be overridden to implement smarter detection - mechanisms for specific vcs. - - This can do more than is_repository_directory() alone. For - example, the Git override checks that Git is actually available. - """ - if cls.is_repository_directory(location): - return location - return None diff --git a/venv/lib/python3.11/site-packages/pip/_internal/wheel_builder.py b/venv/lib/python3.11/site-packages/pip/_internal/wheel_builder.py deleted file mode 100644 index 60d75dd..0000000 --- a/venv/lib/python3.11/site-packages/pip/_internal/wheel_builder.py +++ /dev/null @@ -1,355 +0,0 @@ -"""Orchestrator for building wheels from InstallRequirements. -""" - -import logging -import os.path -import re -import shutil -from typing import Iterable, List, Optional, Tuple - -from pip._vendor.packaging.utils import canonicalize_name, canonicalize_version -from pip._vendor.packaging.version import InvalidVersion, Version - -from pip._internal.cache import WheelCache -from pip._internal.exceptions import InvalidWheelFilename, UnsupportedWheel -from pip._internal.metadata import FilesystemWheel, get_wheel_distribution -from pip._internal.models.link import Link -from pip._internal.models.wheel import Wheel -from pip._internal.operations.build.wheel import build_wheel_pep517 -from pip._internal.operations.build.wheel_editable import build_wheel_editable -from pip._internal.operations.build.wheel_legacy import build_wheel_legacy -from pip._internal.req.req_install import InstallRequirement -from pip._internal.utils.logging import indent_log -from pip._internal.utils.misc import ensure_dir, hash_file -from pip._internal.utils.setuptools_build import make_setuptools_clean_args -from pip._internal.utils.subprocess import call_subprocess -from pip._internal.utils.temp_dir import TempDirectory -from pip._internal.utils.urls import path_to_url -from pip._internal.vcs import vcs - -logger = logging.getLogger(__name__) - -_egg_info_re = re.compile(r"([a-z0-9_.]+)-([a-z0-9_.!+-]+)", re.IGNORECASE) - -BuildResult = Tuple[List[InstallRequirement], List[InstallRequirement]] - - -def _contains_egg_info(s: str) -> bool: - """Determine whether the string looks like an egg_info. - - :param s: The string to parse. E.g. foo-2.1 - """ - return bool(_egg_info_re.search(s)) - - -def _should_build( - req: InstallRequirement, - need_wheel: bool, -) -> bool: - """Return whether an InstallRequirement should be built into a wheel.""" - if req.constraint: - # never build requirements that are merely constraints - return False - if req.is_wheel: - if need_wheel: - logger.info( - "Skipping %s, due to already being wheel.", - req.name, - ) - return False - - if need_wheel: - # i.e. pip wheel, not pip install - return True - - # From this point, this concerns the pip install command only - # (need_wheel=False). - - if not req.source_dir: - return False - - if req.editable: - # we only build PEP 660 editable requirements - return req.supports_pyproject_editable() - - return True - - -def should_build_for_wheel_command( - req: InstallRequirement, -) -> bool: - return _should_build(req, need_wheel=True) - - -def should_build_for_install_command( - req: InstallRequirement, -) -> bool: - return _should_build(req, need_wheel=False) - - -def _should_cache( - req: InstallRequirement, -) -> Optional[bool]: - """ - Return whether a built InstallRequirement can be stored in the persistent - wheel cache, assuming the wheel cache is available, and _should_build() - has determined a wheel needs to be built. - """ - if req.editable or not req.source_dir: - # never cache editable requirements - return False - - if req.link and req.link.is_vcs: - # VCS checkout. Do not cache - # unless it points to an immutable commit hash. - assert not req.editable - assert req.source_dir - vcs_backend = vcs.get_backend_for_scheme(req.link.scheme) - assert vcs_backend - if vcs_backend.is_immutable_rev_checkout(req.link.url, req.source_dir): - return True - return False - - assert req.link - base, ext = req.link.splitext() - if _contains_egg_info(base): - return True - - # Otherwise, do not cache. - return False - - -def _get_cache_dir( - req: InstallRequirement, - wheel_cache: WheelCache, -) -> str: - """Return the persistent or temporary cache directory where the built - wheel need to be stored. - """ - cache_available = bool(wheel_cache.cache_dir) - assert req.link - if cache_available and _should_cache(req): - cache_dir = wheel_cache.get_path_for_link(req.link) - else: - cache_dir = wheel_cache.get_ephem_path_for_link(req.link) - return cache_dir - - -def _verify_one(req: InstallRequirement, wheel_path: str) -> None: - canonical_name = canonicalize_name(req.name or "") - w = Wheel(os.path.basename(wheel_path)) - if canonicalize_name(w.name) != canonical_name: - raise InvalidWheelFilename( - "Wheel has unexpected file name: expected {!r}, " - "got {!r}".format(canonical_name, w.name), - ) - dist = get_wheel_distribution(FilesystemWheel(wheel_path), canonical_name) - dist_verstr = str(dist.version) - if canonicalize_version(dist_verstr) != canonicalize_version(w.version): - raise InvalidWheelFilename( - "Wheel has unexpected file name: expected {!r}, " - "got {!r}".format(dist_verstr, w.version), - ) - metadata_version_value = dist.metadata_version - if metadata_version_value is None: - raise UnsupportedWheel("Missing Metadata-Version") - try: - metadata_version = Version(metadata_version_value) - except InvalidVersion: - msg = f"Invalid Metadata-Version: {metadata_version_value}" - raise UnsupportedWheel(msg) - if metadata_version >= Version("1.2") and not isinstance(dist.version, Version): - raise UnsupportedWheel( - "Metadata 1.2 mandates PEP 440 version, " - "but {!r} is not".format(dist_verstr) - ) - - -def _build_one( - req: InstallRequirement, - output_dir: str, - verify: bool, - build_options: List[str], - global_options: List[str], - editable: bool, -) -> Optional[str]: - """Build one wheel. - - :return: The filename of the built wheel, or None if the build failed. - """ - artifact = "editable" if editable else "wheel" - try: - ensure_dir(output_dir) - except OSError as e: - logger.warning( - "Building %s for %s failed: %s", - artifact, - req.name, - e, - ) - return None - - # Install build deps into temporary directory (PEP 518) - with req.build_env: - wheel_path = _build_one_inside_env( - req, output_dir, build_options, global_options, editable - ) - if wheel_path and verify: - try: - _verify_one(req, wheel_path) - except (InvalidWheelFilename, UnsupportedWheel) as e: - logger.warning("Built %s for %s is invalid: %s", artifact, req.name, e) - return None - return wheel_path - - -def _build_one_inside_env( - req: InstallRequirement, - output_dir: str, - build_options: List[str], - global_options: List[str], - editable: bool, -) -> Optional[str]: - with TempDirectory(kind="wheel") as temp_dir: - assert req.name - if req.use_pep517: - assert req.metadata_directory - assert req.pep517_backend - if global_options: - logger.warning( - "Ignoring --global-option when building %s using PEP 517", req.name - ) - if build_options: - logger.warning( - "Ignoring --build-option when building %s using PEP 517", req.name - ) - if editable: - wheel_path = build_wheel_editable( - name=req.name, - backend=req.pep517_backend, - metadata_directory=req.metadata_directory, - tempd=temp_dir.path, - ) - else: - wheel_path = build_wheel_pep517( - name=req.name, - backend=req.pep517_backend, - metadata_directory=req.metadata_directory, - tempd=temp_dir.path, - ) - else: - wheel_path = build_wheel_legacy( - name=req.name, - setup_py_path=req.setup_py_path, - source_dir=req.unpacked_source_directory, - global_options=global_options, - build_options=build_options, - tempd=temp_dir.path, - ) - - if wheel_path is not None: - wheel_name = os.path.basename(wheel_path) - dest_path = os.path.join(output_dir, wheel_name) - try: - wheel_hash, length = hash_file(wheel_path) - shutil.move(wheel_path, dest_path) - logger.info( - "Created wheel for %s: filename=%s size=%d sha256=%s", - req.name, - wheel_name, - length, - wheel_hash.hexdigest(), - ) - logger.info("Stored in directory: %s", output_dir) - return dest_path - except Exception as e: - logger.warning( - "Building wheel for %s failed: %s", - req.name, - e, - ) - # Ignore return, we can't do anything else useful. - if not req.use_pep517: - _clean_one_legacy(req, global_options) - return None - - -def _clean_one_legacy(req: InstallRequirement, global_options: List[str]) -> bool: - clean_args = make_setuptools_clean_args( - req.setup_py_path, - global_options=global_options, - ) - - logger.info("Running setup.py clean for %s", req.name) - try: - call_subprocess( - clean_args, command_desc="python setup.py clean", cwd=req.source_dir - ) - return True - except Exception: - logger.error("Failed cleaning build dir for %s", req.name) - return False - - -def build( - requirements: Iterable[InstallRequirement], - wheel_cache: WheelCache, - verify: bool, - build_options: List[str], - global_options: List[str], -) -> BuildResult: - """Build wheels. - - :return: The list of InstallRequirement that succeeded to build and - the list of InstallRequirement that failed to build. - """ - if not requirements: - return [], [] - - # Build the wheels. - logger.info( - "Building wheels for collected packages: %s", - ", ".join(req.name for req in requirements), # type: ignore - ) - - with indent_log(): - build_successes, build_failures = [], [] - for req in requirements: - assert req.name - cache_dir = _get_cache_dir(req, wheel_cache) - wheel_file = _build_one( - req, - cache_dir, - verify, - build_options, - global_options, - req.editable and req.permit_editable_wheels, - ) - if wheel_file: - # Record the download origin in the cache - if req.download_info is not None: - # download_info is guaranteed to be set because when we build an - # InstallRequirement it has been through the preparer before, but - # let's be cautious. - wheel_cache.record_download_origin(cache_dir, req.download_info) - # Update the link for this. - req.link = Link(path_to_url(wheel_file)) - req.local_file_path = req.link.file_path - assert req.link.is_wheel - build_successes.append(req) - else: - build_failures.append(req) - - # notify success/failure - if build_successes: - logger.info( - "Successfully built %s", - " ".join([req.name for req in build_successes]), # type: ignore - ) - if build_failures: - logger.info( - "Failed to build %s", - " ".join([req.name for req in build_failures]), # type: ignore - ) - # Return a list of requirements that failed to build - return build_successes, build_failures diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/__init__.py b/venv/lib/python3.11/site-packages/pip/_vendor/__init__.py deleted file mode 100644 index c1884ba..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/__init__.py +++ /dev/null @@ -1,121 +0,0 @@ -""" -pip._vendor is for vendoring dependencies of pip to prevent needing pip to -depend on something external. - -Files inside of pip._vendor should be considered immutable and should only be -updated to versions from upstream. -""" -from __future__ import absolute_import - -import glob -import os.path -import sys - -# Downstream redistributors which have debundled our dependencies should also -# patch this value to be true. This will trigger the additional patching -# to cause things like "six" to be available as pip. -DEBUNDLED = False - -# By default, look in this directory for a bunch of .whl files which we will -# add to the beginning of sys.path before attempting to import anything. This -# is done to support downstream re-distributors like Debian and Fedora who -# wish to create their own Wheels for our dependencies to aid in debundling. -WHEEL_DIR = os.path.abspath(os.path.dirname(__file__)) - - -# Define a small helper function to alias our vendored modules to the real ones -# if the vendored ones do not exist. This idea of this was taken from -# https://github.com/kennethreitz/requests/pull/2567. -def vendored(modulename): - vendored_name = "{0}.{1}".format(__name__, modulename) - - try: - __import__(modulename, globals(), locals(), level=0) - except ImportError: - # We can just silently allow import failures to pass here. If we - # got to this point it means that ``import pip._vendor.whatever`` - # failed and so did ``import whatever``. Since we're importing this - # upfront in an attempt to alias imports, not erroring here will - # just mean we get a regular import error whenever pip *actually* - # tries to import one of these modules to use it, which actually - # gives us a better error message than we would have otherwise - # gotten. - pass - else: - sys.modules[vendored_name] = sys.modules[modulename] - base, head = vendored_name.rsplit(".", 1) - setattr(sys.modules[base], head, sys.modules[modulename]) - - -# If we're operating in a debundled setup, then we want to go ahead and trigger -# the aliasing of our vendored libraries as well as looking for wheels to add -# to our sys.path. This will cause all of this code to be a no-op typically -# however downstream redistributors can enable it in a consistent way across -# all platforms. -if DEBUNDLED: - # Actually look inside of WHEEL_DIR to find .whl files and add them to the - # front of our sys.path. - sys.path[:] = glob.glob(os.path.join(WHEEL_DIR, "*.whl")) + sys.path - - # Actually alias all of our vendored dependencies. - vendored("cachecontrol") - vendored("certifi") - vendored("colorama") - vendored("distlib") - vendored("distro") - vendored("six") - vendored("six.moves") - vendored("six.moves.urllib") - vendored("six.moves.urllib.parse") - vendored("packaging") - vendored("packaging.version") - vendored("packaging.specifiers") - vendored("pep517") - vendored("pkg_resources") - vendored("platformdirs") - vendored("progress") - vendored("requests") - vendored("requests.exceptions") - vendored("requests.packages") - vendored("requests.packages.urllib3") - vendored("requests.packages.urllib3._collections") - vendored("requests.packages.urllib3.connection") - vendored("requests.packages.urllib3.connectionpool") - vendored("requests.packages.urllib3.contrib") - vendored("requests.packages.urllib3.contrib.ntlmpool") - vendored("requests.packages.urllib3.contrib.pyopenssl") - vendored("requests.packages.urllib3.exceptions") - vendored("requests.packages.urllib3.fields") - vendored("requests.packages.urllib3.filepost") - vendored("requests.packages.urllib3.packages") - vendored("requests.packages.urllib3.packages.ordered_dict") - vendored("requests.packages.urllib3.packages.six") - vendored("requests.packages.urllib3.packages.ssl_match_hostname") - vendored("requests.packages.urllib3.packages.ssl_match_hostname." - "_implementation") - vendored("requests.packages.urllib3.poolmanager") - vendored("requests.packages.urllib3.request") - vendored("requests.packages.urllib3.response") - vendored("requests.packages.urllib3.util") - vendored("requests.packages.urllib3.util.connection") - vendored("requests.packages.urllib3.util.request") - vendored("requests.packages.urllib3.util.response") - vendored("requests.packages.urllib3.util.retry") - vendored("requests.packages.urllib3.util.ssl_") - vendored("requests.packages.urllib3.util.timeout") - vendored("requests.packages.urllib3.util.url") - vendored("resolvelib") - vendored("rich") - vendored("rich.console") - vendored("rich.highlighter") - vendored("rich.logging") - vendored("rich.markup") - vendored("rich.progress") - vendored("rich.segment") - vendored("rich.style") - vendored("rich.text") - vendored("rich.traceback") - vendored("tenacity") - vendored("tomli") - vendored("truststore") - vendored("urllib3") diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/__pycache__/__init__.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/__pycache__/__init__.cpython-311.pyc deleted file mode 100644 index 92cc009..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/__pycache__/__init__.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/__pycache__/six.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/__pycache__/six.cpython-311.pyc deleted file mode 100644 index 484a641..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/__pycache__/six.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/__pycache__/typing_extensions.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/__pycache__/typing_extensions.cpython-311.pyc deleted file mode 100644 index 0fc7899..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/__pycache__/typing_extensions.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/cachecontrol/__init__.py b/venv/lib/python3.11/site-packages/pip/_vendor/cachecontrol/__init__.py deleted file mode 100644 index 4d20bc9..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/cachecontrol/__init__.py +++ /dev/null @@ -1,28 +0,0 @@ -# SPDX-FileCopyrightText: 2015 Eric Larson -# -# SPDX-License-Identifier: Apache-2.0 - -"""CacheControl import Interface. - -Make it easy to import from cachecontrol without long namespaces. -""" -__author__ = "Eric Larson" -__email__ = "eric@ionrock.org" -__version__ = "0.13.1" - -from pip._vendor.cachecontrol.adapter import CacheControlAdapter -from pip._vendor.cachecontrol.controller import CacheController -from pip._vendor.cachecontrol.wrapper import CacheControl - -__all__ = [ - "__author__", - "__email__", - "__version__", - "CacheControlAdapter", - "CacheController", - "CacheControl", -] - -import logging - -logging.getLogger(__name__).addHandler(logging.NullHandler()) diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/cachecontrol/__pycache__/__init__.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/cachecontrol/__pycache__/__init__.cpython-311.pyc deleted file mode 100644 index 9eec919..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/cachecontrol/__pycache__/__init__.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/cachecontrol/__pycache__/_cmd.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/cachecontrol/__pycache__/_cmd.cpython-311.pyc deleted file mode 100644 index 4778396..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/cachecontrol/__pycache__/_cmd.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/cachecontrol/__pycache__/adapter.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/cachecontrol/__pycache__/adapter.cpython-311.pyc deleted file mode 100644 index 419db3f..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/cachecontrol/__pycache__/adapter.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/cachecontrol/__pycache__/cache.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/cachecontrol/__pycache__/cache.cpython-311.pyc deleted file mode 100644 index 2187de6..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/cachecontrol/__pycache__/cache.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/cachecontrol/__pycache__/controller.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/cachecontrol/__pycache__/controller.cpython-311.pyc deleted file mode 100644 index a360fb2..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/cachecontrol/__pycache__/controller.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/cachecontrol/__pycache__/filewrapper.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/cachecontrol/__pycache__/filewrapper.cpython-311.pyc deleted file mode 100644 index bedbcf3..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/cachecontrol/__pycache__/filewrapper.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/cachecontrol/__pycache__/heuristics.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/cachecontrol/__pycache__/heuristics.cpython-311.pyc deleted file mode 100644 index 15fbbf4..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/cachecontrol/__pycache__/heuristics.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/cachecontrol/__pycache__/serialize.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/cachecontrol/__pycache__/serialize.cpython-311.pyc deleted file mode 100644 index 2910ae0..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/cachecontrol/__pycache__/serialize.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/cachecontrol/__pycache__/wrapper.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/cachecontrol/__pycache__/wrapper.cpython-311.pyc deleted file mode 100644 index a9e0879..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/cachecontrol/__pycache__/wrapper.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/cachecontrol/_cmd.py b/venv/lib/python3.11/site-packages/pip/_vendor/cachecontrol/_cmd.py deleted file mode 100644 index 2c84208..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/cachecontrol/_cmd.py +++ /dev/null @@ -1,70 +0,0 @@ -# SPDX-FileCopyrightText: 2015 Eric Larson -# -# SPDX-License-Identifier: Apache-2.0 -from __future__ import annotations - -import logging -from argparse import ArgumentParser -from typing import TYPE_CHECKING - -from pip._vendor import requests - -from pip._vendor.cachecontrol.adapter import CacheControlAdapter -from pip._vendor.cachecontrol.cache import DictCache -from pip._vendor.cachecontrol.controller import logger - -if TYPE_CHECKING: - from argparse import Namespace - - from pip._vendor.cachecontrol.controller import CacheController - - -def setup_logging() -> None: - logger.setLevel(logging.DEBUG) - handler = logging.StreamHandler() - logger.addHandler(handler) - - -def get_session() -> requests.Session: - adapter = CacheControlAdapter( - DictCache(), cache_etags=True, serializer=None, heuristic=None - ) - sess = requests.Session() - sess.mount("http://", adapter) - sess.mount("https://", adapter) - - sess.cache_controller = adapter.controller # type: ignore[attr-defined] - return sess - - -def get_args() -> Namespace: - parser = ArgumentParser() - parser.add_argument("url", help="The URL to try and cache") - return parser.parse_args() - - -def main() -> None: - args = get_args() - sess = get_session() - - # Make a request to get a response - resp = sess.get(args.url) - - # Turn on logging - setup_logging() - - # try setting the cache - cache_controller: CacheController = ( - sess.cache_controller # type: ignore[attr-defined] - ) - cache_controller.cache_response(resp.request, resp.raw) - - # Now try to get it - if cache_controller.cached_request(resp.request): - print("Cached!") - else: - print("Not cached :(") - - -if __name__ == "__main__": - main() diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/cachecontrol/adapter.py b/venv/lib/python3.11/site-packages/pip/_vendor/cachecontrol/adapter.py deleted file mode 100644 index 3e83e30..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/cachecontrol/adapter.py +++ /dev/null @@ -1,161 +0,0 @@ -# SPDX-FileCopyrightText: 2015 Eric Larson -# -# SPDX-License-Identifier: Apache-2.0 -from __future__ import annotations - -import functools -import types -import zlib -from typing import TYPE_CHECKING, Any, Collection, Mapping - -from pip._vendor.requests.adapters import HTTPAdapter - -from pip._vendor.cachecontrol.cache import DictCache -from pip._vendor.cachecontrol.controller import PERMANENT_REDIRECT_STATUSES, CacheController -from pip._vendor.cachecontrol.filewrapper import CallbackFileWrapper - -if TYPE_CHECKING: - from pip._vendor.requests import PreparedRequest, Response - from pip._vendor.urllib3 import HTTPResponse - - from pip._vendor.cachecontrol.cache import BaseCache - from pip._vendor.cachecontrol.heuristics import BaseHeuristic - from pip._vendor.cachecontrol.serialize import Serializer - - -class CacheControlAdapter(HTTPAdapter): - invalidating_methods = {"PUT", "PATCH", "DELETE"} - - def __init__( - self, - cache: BaseCache | None = None, - cache_etags: bool = True, - controller_class: type[CacheController] | None = None, - serializer: Serializer | None = None, - heuristic: BaseHeuristic | None = None, - cacheable_methods: Collection[str] | None = None, - *args: Any, - **kw: Any, - ) -> None: - super().__init__(*args, **kw) - self.cache = DictCache() if cache is None else cache - self.heuristic = heuristic - self.cacheable_methods = cacheable_methods or ("GET",) - - controller_factory = controller_class or CacheController - self.controller = controller_factory( - self.cache, cache_etags=cache_etags, serializer=serializer - ) - - def send( - self, - request: PreparedRequest, - stream: bool = False, - timeout: None | float | tuple[float, float] | tuple[float, None] = None, - verify: bool | str = True, - cert: (None | bytes | str | tuple[bytes | str, bytes | str]) = None, - proxies: Mapping[str, str] | None = None, - cacheable_methods: Collection[str] | None = None, - ) -> Response: - """ - Send a request. Use the request information to see if it - exists in the cache and cache the response if we need to and can. - """ - cacheable = cacheable_methods or self.cacheable_methods - if request.method in cacheable: - try: - cached_response = self.controller.cached_request(request) - except zlib.error: - cached_response = None - if cached_response: - return self.build_response(request, cached_response, from_cache=True) - - # check for etags and add headers if appropriate - request.headers.update(self.controller.conditional_headers(request)) - - resp = super().send(request, stream, timeout, verify, cert, proxies) - - return resp - - def build_response( - self, - request: PreparedRequest, - response: HTTPResponse, - from_cache: bool = False, - cacheable_methods: Collection[str] | None = None, - ) -> Response: - """ - Build a response by making a request or using the cache. - - This will end up calling send and returning a potentially - cached response - """ - cacheable = cacheable_methods or self.cacheable_methods - if not from_cache and request.method in cacheable: - # Check for any heuristics that might update headers - # before trying to cache. - if self.heuristic: - response = self.heuristic.apply(response) - - # apply any expiration heuristics - if response.status == 304: - # We must have sent an ETag request. This could mean - # that we've been expired already or that we simply - # have an etag. In either case, we want to try and - # update the cache if that is the case. - cached_response = self.controller.update_cached_response( - request, response - ) - - if cached_response is not response: - from_cache = True - - # We are done with the server response, read a - # possible response body (compliant servers will - # not return one, but we cannot be 100% sure) and - # release the connection back to the pool. - response.read(decode_content=False) - response.release_conn() - - response = cached_response - - # We always cache the 301 responses - elif int(response.status) in PERMANENT_REDIRECT_STATUSES: - self.controller.cache_response(request, response) - else: - # Wrap the response file with a wrapper that will cache the - # response when the stream has been consumed. - response._fp = CallbackFileWrapper( # type: ignore[attr-defined] - response._fp, # type: ignore[attr-defined] - functools.partial( - self.controller.cache_response, request, response - ), - ) - if response.chunked: - super_update_chunk_length = response._update_chunk_length # type: ignore[attr-defined] - - def _update_chunk_length(self: HTTPResponse) -> None: - super_update_chunk_length() - if self.chunk_left == 0: - self._fp._close() # type: ignore[attr-defined] - - response._update_chunk_length = types.MethodType( # type: ignore[attr-defined] - _update_chunk_length, response - ) - - resp: Response = super().build_response(request, response) # type: ignore[no-untyped-call] - - # See if we should invalidate the cache. - if request.method in self.invalidating_methods and resp.ok: - assert request.url is not None - cache_url = self.controller.cache_url(request.url) - self.cache.delete(cache_url) - - # Give the request a from_cache attr to let people use it - resp.from_cache = from_cache # type: ignore[attr-defined] - - return resp - - def close(self) -> None: - self.cache.close() - super().close() # type: ignore[no-untyped-call] diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/cachecontrol/cache.py b/venv/lib/python3.11/site-packages/pip/_vendor/cachecontrol/cache.py deleted file mode 100644 index 3293b00..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/cachecontrol/cache.py +++ /dev/null @@ -1,74 +0,0 @@ -# SPDX-FileCopyrightText: 2015 Eric Larson -# -# SPDX-License-Identifier: Apache-2.0 - -""" -The cache object API for implementing caches. The default is a thread -safe in-memory dictionary. -""" -from __future__ import annotations - -from threading import Lock -from typing import IO, TYPE_CHECKING, MutableMapping - -if TYPE_CHECKING: - from datetime import datetime - - -class BaseCache: - def get(self, key: str) -> bytes | None: - raise NotImplementedError() - - def set( - self, key: str, value: bytes, expires: int | datetime | None = None - ) -> None: - raise NotImplementedError() - - def delete(self, key: str) -> None: - raise NotImplementedError() - - def close(self) -> None: - pass - - -class DictCache(BaseCache): - def __init__(self, init_dict: MutableMapping[str, bytes] | None = None) -> None: - self.lock = Lock() - self.data = init_dict or {} - - def get(self, key: str) -> bytes | None: - return self.data.get(key, None) - - def set( - self, key: str, value: bytes, expires: int | datetime | None = None - ) -> None: - with self.lock: - self.data.update({key: value}) - - def delete(self, key: str) -> None: - with self.lock: - if key in self.data: - self.data.pop(key) - - -class SeparateBodyBaseCache(BaseCache): - """ - In this variant, the body is not stored mixed in with the metadata, but is - passed in (as a bytes-like object) in a separate call to ``set_body()``. - - That is, the expected interaction pattern is:: - - cache.set(key, serialized_metadata) - cache.set_body(key) - - Similarly, the body should be loaded separately via ``get_body()``. - """ - - def set_body(self, key: str, body: bytes) -> None: - raise NotImplementedError() - - def get_body(self, key: str) -> IO[bytes] | None: - """ - Return the body as file-like object. - """ - raise NotImplementedError() diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/cachecontrol/caches/__init__.py b/venv/lib/python3.11/site-packages/pip/_vendor/cachecontrol/caches/__init__.py deleted file mode 100644 index 24ff469..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/cachecontrol/caches/__init__.py +++ /dev/null @@ -1,8 +0,0 @@ -# SPDX-FileCopyrightText: 2015 Eric Larson -# -# SPDX-License-Identifier: Apache-2.0 - -from pip._vendor.cachecontrol.caches.file_cache import FileCache, SeparateBodyFileCache -from pip._vendor.cachecontrol.caches.redis_cache import RedisCache - -__all__ = ["FileCache", "SeparateBodyFileCache", "RedisCache"] diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/cachecontrol/caches/__pycache__/__init__.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/cachecontrol/caches/__pycache__/__init__.cpython-311.pyc deleted file mode 100644 index 65b68ec..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/cachecontrol/caches/__pycache__/__init__.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/cachecontrol/caches/__pycache__/file_cache.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/cachecontrol/caches/__pycache__/file_cache.cpython-311.pyc deleted file mode 100644 index 080768a..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/cachecontrol/caches/__pycache__/file_cache.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/cachecontrol/caches/__pycache__/redis_cache.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/cachecontrol/caches/__pycache__/redis_cache.cpython-311.pyc deleted file mode 100644 index 2357ac3..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/cachecontrol/caches/__pycache__/redis_cache.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/cachecontrol/caches/file_cache.py b/venv/lib/python3.11/site-packages/pip/_vendor/cachecontrol/caches/file_cache.py deleted file mode 100644 index 1fd2801..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/cachecontrol/caches/file_cache.py +++ /dev/null @@ -1,181 +0,0 @@ -# SPDX-FileCopyrightText: 2015 Eric Larson -# -# SPDX-License-Identifier: Apache-2.0 -from __future__ import annotations - -import hashlib -import os -from textwrap import dedent -from typing import IO, TYPE_CHECKING - -from pip._vendor.cachecontrol.cache import BaseCache, SeparateBodyBaseCache -from pip._vendor.cachecontrol.controller import CacheController - -if TYPE_CHECKING: - from datetime import datetime - - from filelock import BaseFileLock - - -def _secure_open_write(filename: str, fmode: int) -> IO[bytes]: - # We only want to write to this file, so open it in write only mode - flags = os.O_WRONLY - - # os.O_CREAT | os.O_EXCL will fail if the file already exists, so we only - # will open *new* files. - # We specify this because we want to ensure that the mode we pass is the - # mode of the file. - flags |= os.O_CREAT | os.O_EXCL - - # Do not follow symlinks to prevent someone from making a symlink that - # we follow and insecurely open a cache file. - if hasattr(os, "O_NOFOLLOW"): - flags |= os.O_NOFOLLOW - - # On Windows we'll mark this file as binary - if hasattr(os, "O_BINARY"): - flags |= os.O_BINARY - - # Before we open our file, we want to delete any existing file that is - # there - try: - os.remove(filename) - except OSError: - # The file must not exist already, so we can just skip ahead to opening - pass - - # Open our file, the use of os.O_CREAT | os.O_EXCL will ensure that if a - # race condition happens between the os.remove and this line, that an - # error will be raised. Because we utilize a lockfile this should only - # happen if someone is attempting to attack us. - fd = os.open(filename, flags, fmode) - try: - return os.fdopen(fd, "wb") - - except: - # An error occurred wrapping our FD in a file object - os.close(fd) - raise - - -class _FileCacheMixin: - """Shared implementation for both FileCache variants.""" - - def __init__( - self, - directory: str, - forever: bool = False, - filemode: int = 0o0600, - dirmode: int = 0o0700, - lock_class: type[BaseFileLock] | None = None, - ) -> None: - try: - if lock_class is None: - from filelock import FileLock - - lock_class = FileLock - except ImportError: - notice = dedent( - """ - NOTE: In order to use the FileCache you must have - filelock installed. You can install it via pip: - pip install filelock - """ - ) - raise ImportError(notice) - - self.directory = directory - self.forever = forever - self.filemode = filemode - self.dirmode = dirmode - self.lock_class = lock_class - - @staticmethod - def encode(x: str) -> str: - return hashlib.sha224(x.encode()).hexdigest() - - def _fn(self, name: str) -> str: - # NOTE: This method should not change as some may depend on it. - # See: https://github.com/ionrock/cachecontrol/issues/63 - hashed = self.encode(name) - parts = list(hashed[:5]) + [hashed] - return os.path.join(self.directory, *parts) - - def get(self, key: str) -> bytes | None: - name = self._fn(key) - try: - with open(name, "rb") as fh: - return fh.read() - - except FileNotFoundError: - return None - - def set( - self, key: str, value: bytes, expires: int | datetime | None = None - ) -> None: - name = self._fn(key) - self._write(name, value) - - def _write(self, path: str, data: bytes) -> None: - """ - Safely write the data to the given path. - """ - # Make sure the directory exists - try: - os.makedirs(os.path.dirname(path), self.dirmode) - except OSError: - pass - - with self.lock_class(path + ".lock"): - # Write our actual file - with _secure_open_write(path, self.filemode) as fh: - fh.write(data) - - def _delete(self, key: str, suffix: str) -> None: - name = self._fn(key) + suffix - if not self.forever: - try: - os.remove(name) - except FileNotFoundError: - pass - - -class FileCache(_FileCacheMixin, BaseCache): - """ - Traditional FileCache: body is stored in memory, so not suitable for large - downloads. - """ - - def delete(self, key: str) -> None: - self._delete(key, "") - - -class SeparateBodyFileCache(_FileCacheMixin, SeparateBodyBaseCache): - """ - Memory-efficient FileCache: body is stored in a separate file, reducing - peak memory usage. - """ - - def get_body(self, key: str) -> IO[bytes] | None: - name = self._fn(key) + ".body" - try: - return open(name, "rb") - except FileNotFoundError: - return None - - def set_body(self, key: str, body: bytes) -> None: - name = self._fn(key) + ".body" - self._write(name, body) - - def delete(self, key: str) -> None: - self._delete(key, "") - self._delete(key, ".body") - - -def url_to_file_path(url: str, filecache: FileCache) -> str: - """Return the file cache path based on the URL. - - This does not ensure the file exists! - """ - key = CacheController.cache_url(url) - return filecache._fn(key) diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/cachecontrol/caches/redis_cache.py b/venv/lib/python3.11/site-packages/pip/_vendor/cachecontrol/caches/redis_cache.py deleted file mode 100644 index f4f68c4..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/cachecontrol/caches/redis_cache.py +++ /dev/null @@ -1,48 +0,0 @@ -# SPDX-FileCopyrightText: 2015 Eric Larson -# -# SPDX-License-Identifier: Apache-2.0 -from __future__ import annotations - - -from datetime import datetime, timezone -from typing import TYPE_CHECKING - -from pip._vendor.cachecontrol.cache import BaseCache - -if TYPE_CHECKING: - from redis import Redis - - -class RedisCache(BaseCache): - def __init__(self, conn: Redis[bytes]) -> None: - self.conn = conn - - def get(self, key: str) -> bytes | None: - return self.conn.get(key) - - def set( - self, key: str, value: bytes, expires: int | datetime | None = None - ) -> None: - if not expires: - self.conn.set(key, value) - elif isinstance(expires, datetime): - now_utc = datetime.now(timezone.utc) - if expires.tzinfo is None: - now_utc = now_utc.replace(tzinfo=None) - delta = expires - now_utc - self.conn.setex(key, int(delta.total_seconds()), value) - else: - self.conn.setex(key, expires, value) - - def delete(self, key: str) -> None: - self.conn.delete(key) - - def clear(self) -> None: - """Helper for clearing all the keys in a database. Use with - caution!""" - for key in self.conn.keys(): - self.conn.delete(key) - - def close(self) -> None: - """Redis uses connection pooling, no need to close the connection.""" - pass diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/cachecontrol/controller.py b/venv/lib/python3.11/site-packages/pip/_vendor/cachecontrol/controller.py deleted file mode 100644 index 586b9f9..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/cachecontrol/controller.py +++ /dev/null @@ -1,494 +0,0 @@ -# SPDX-FileCopyrightText: 2015 Eric Larson -# -# SPDX-License-Identifier: Apache-2.0 - -""" -The httplib2 algorithms ported for use with requests. -""" -from __future__ import annotations - -import calendar -import logging -import re -import time -from email.utils import parsedate_tz -from typing import TYPE_CHECKING, Collection, Mapping - -from pip._vendor.requests.structures import CaseInsensitiveDict - -from pip._vendor.cachecontrol.cache import DictCache, SeparateBodyBaseCache -from pip._vendor.cachecontrol.serialize import Serializer - -if TYPE_CHECKING: - from typing import Literal - - from pip._vendor.requests import PreparedRequest - from pip._vendor.urllib3 import HTTPResponse - - from pip._vendor.cachecontrol.cache import BaseCache - -logger = logging.getLogger(__name__) - -URI = re.compile(r"^(([^:/?#]+):)?(//([^/?#]*))?([^?#]*)(\?([^#]*))?(#(.*))?") - -PERMANENT_REDIRECT_STATUSES = (301, 308) - - -def parse_uri(uri: str) -> tuple[str, str, str, str, str]: - """Parses a URI using the regex given in Appendix B of RFC 3986. - - (scheme, authority, path, query, fragment) = parse_uri(uri) - """ - match = URI.match(uri) - assert match is not None - groups = match.groups() - return (groups[1], groups[3], groups[4], groups[6], groups[8]) - - -class CacheController: - """An interface to see if request should cached or not.""" - - def __init__( - self, - cache: BaseCache | None = None, - cache_etags: bool = True, - serializer: Serializer | None = None, - status_codes: Collection[int] | None = None, - ): - self.cache = DictCache() if cache is None else cache - self.cache_etags = cache_etags - self.serializer = serializer or Serializer() - self.cacheable_status_codes = status_codes or (200, 203, 300, 301, 308) - - @classmethod - def _urlnorm(cls, uri: str) -> str: - """Normalize the URL to create a safe key for the cache""" - (scheme, authority, path, query, fragment) = parse_uri(uri) - if not scheme or not authority: - raise Exception("Only absolute URIs are allowed. uri = %s" % uri) - - scheme = scheme.lower() - authority = authority.lower() - - if not path: - path = "/" - - # Could do syntax based normalization of the URI before - # computing the digest. See Section 6.2.2 of Std 66. - request_uri = query and "?".join([path, query]) or path - defrag_uri = scheme + "://" + authority + request_uri - - return defrag_uri - - @classmethod - def cache_url(cls, uri: str) -> str: - return cls._urlnorm(uri) - - def parse_cache_control(self, headers: Mapping[str, str]) -> dict[str, int | None]: - known_directives = { - # https://tools.ietf.org/html/rfc7234#section-5.2 - "max-age": (int, True), - "max-stale": (int, False), - "min-fresh": (int, True), - "no-cache": (None, False), - "no-store": (None, False), - "no-transform": (None, False), - "only-if-cached": (None, False), - "must-revalidate": (None, False), - "public": (None, False), - "private": (None, False), - "proxy-revalidate": (None, False), - "s-maxage": (int, True), - } - - cc_headers = headers.get("cache-control", headers.get("Cache-Control", "")) - - retval: dict[str, int | None] = {} - - for cc_directive in cc_headers.split(","): - if not cc_directive.strip(): - continue - - parts = cc_directive.split("=", 1) - directive = parts[0].strip() - - try: - typ, required = known_directives[directive] - except KeyError: - logger.debug("Ignoring unknown cache-control directive: %s", directive) - continue - - if not typ or not required: - retval[directive] = None - if typ: - try: - retval[directive] = typ(parts[1].strip()) - except IndexError: - if required: - logger.debug( - "Missing value for cache-control " "directive: %s", - directive, - ) - except ValueError: - logger.debug( - "Invalid value for cache-control directive " "%s, must be %s", - directive, - typ.__name__, - ) - - return retval - - def _load_from_cache(self, request: PreparedRequest) -> HTTPResponse | None: - """ - Load a cached response, or return None if it's not available. - """ - cache_url = request.url - assert cache_url is not None - cache_data = self.cache.get(cache_url) - if cache_data is None: - logger.debug("No cache entry available") - return None - - if isinstance(self.cache, SeparateBodyBaseCache): - body_file = self.cache.get_body(cache_url) - else: - body_file = None - - result = self.serializer.loads(request, cache_data, body_file) - if result is None: - logger.warning("Cache entry deserialization failed, entry ignored") - return result - - def cached_request(self, request: PreparedRequest) -> HTTPResponse | Literal[False]: - """ - Return a cached response if it exists in the cache, otherwise - return False. - """ - assert request.url is not None - cache_url = self.cache_url(request.url) - logger.debug('Looking up "%s" in the cache', cache_url) - cc = self.parse_cache_control(request.headers) - - # Bail out if the request insists on fresh data - if "no-cache" in cc: - logger.debug('Request header has "no-cache", cache bypassed') - return False - - if "max-age" in cc and cc["max-age"] == 0: - logger.debug('Request header has "max_age" as 0, cache bypassed') - return False - - # Check whether we can load the response from the cache: - resp = self._load_from_cache(request) - if not resp: - return False - - # If we have a cached permanent redirect, return it immediately. We - # don't need to test our response for other headers b/c it is - # intrinsically "cacheable" as it is Permanent. - # - # See: - # https://tools.ietf.org/html/rfc7231#section-6.4.2 - # - # Client can try to refresh the value by repeating the request - # with cache busting headers as usual (ie no-cache). - if int(resp.status) in PERMANENT_REDIRECT_STATUSES: - msg = ( - "Returning cached permanent redirect response " - "(ignoring date and etag information)" - ) - logger.debug(msg) - return resp - - headers: CaseInsensitiveDict[str] = CaseInsensitiveDict(resp.headers) - if not headers or "date" not in headers: - if "etag" not in headers: - # Without date or etag, the cached response can never be used - # and should be deleted. - logger.debug("Purging cached response: no date or etag") - self.cache.delete(cache_url) - logger.debug("Ignoring cached response: no date") - return False - - now = time.time() - time_tuple = parsedate_tz(headers["date"]) - assert time_tuple is not None - date = calendar.timegm(time_tuple[:6]) - current_age = max(0, now - date) - logger.debug("Current age based on date: %i", current_age) - - # TODO: There is an assumption that the result will be a - # urllib3 response object. This may not be best since we - # could probably avoid instantiating or constructing the - # response until we know we need it. - resp_cc = self.parse_cache_control(headers) - - # determine freshness - freshness_lifetime = 0 - - # Check the max-age pragma in the cache control header - max_age = resp_cc.get("max-age") - if max_age is not None: - freshness_lifetime = max_age - logger.debug("Freshness lifetime from max-age: %i", freshness_lifetime) - - # If there isn't a max-age, check for an expires header - elif "expires" in headers: - expires = parsedate_tz(headers["expires"]) - if expires is not None: - expire_time = calendar.timegm(expires[:6]) - date - freshness_lifetime = max(0, expire_time) - logger.debug("Freshness lifetime from expires: %i", freshness_lifetime) - - # Determine if we are setting freshness limit in the - # request. Note, this overrides what was in the response. - max_age = cc.get("max-age") - if max_age is not None: - freshness_lifetime = max_age - logger.debug( - "Freshness lifetime from request max-age: %i", freshness_lifetime - ) - - min_fresh = cc.get("min-fresh") - if min_fresh is not None: - # adjust our current age by our min fresh - current_age += min_fresh - logger.debug("Adjusted current age from min-fresh: %i", current_age) - - # Return entry if it is fresh enough - if freshness_lifetime > current_age: - logger.debug('The response is "fresh", returning cached response') - logger.debug("%i > %i", freshness_lifetime, current_age) - return resp - - # we're not fresh. If we don't have an Etag, clear it out - if "etag" not in headers: - logger.debug('The cached response is "stale" with no etag, purging') - self.cache.delete(cache_url) - - # return the original handler - return False - - def conditional_headers(self, request: PreparedRequest) -> dict[str, str]: - resp = self._load_from_cache(request) - new_headers = {} - - if resp: - headers: CaseInsensitiveDict[str] = CaseInsensitiveDict(resp.headers) - - if "etag" in headers: - new_headers["If-None-Match"] = headers["ETag"] - - if "last-modified" in headers: - new_headers["If-Modified-Since"] = headers["Last-Modified"] - - return new_headers - - def _cache_set( - self, - cache_url: str, - request: PreparedRequest, - response: HTTPResponse, - body: bytes | None = None, - expires_time: int | None = None, - ) -> None: - """ - Store the data in the cache. - """ - if isinstance(self.cache, SeparateBodyBaseCache): - # We pass in the body separately; just put a placeholder empty - # string in the metadata. - self.cache.set( - cache_url, - self.serializer.dumps(request, response, b""), - expires=expires_time, - ) - # body is None can happen when, for example, we're only updating - # headers, as is the case in update_cached_response(). - if body is not None: - self.cache.set_body(cache_url, body) - else: - self.cache.set( - cache_url, - self.serializer.dumps(request, response, body), - expires=expires_time, - ) - - def cache_response( - self, - request: PreparedRequest, - response: HTTPResponse, - body: bytes | None = None, - status_codes: Collection[int] | None = None, - ) -> None: - """ - Algorithm for caching requests. - - This assumes a requests Response object. - """ - # From httplib2: Don't cache 206's since we aren't going to - # handle byte range requests - cacheable_status_codes = status_codes or self.cacheable_status_codes - if response.status not in cacheable_status_codes: - logger.debug( - "Status code %s not in %s", response.status, cacheable_status_codes - ) - return - - response_headers: CaseInsensitiveDict[str] = CaseInsensitiveDict( - response.headers - ) - - if "date" in response_headers: - time_tuple = parsedate_tz(response_headers["date"]) - assert time_tuple is not None - date = calendar.timegm(time_tuple[:6]) - else: - date = 0 - - # If we've been given a body, our response has a Content-Length, that - # Content-Length is valid then we can check to see if the body we've - # been given matches the expected size, and if it doesn't we'll just - # skip trying to cache it. - if ( - body is not None - and "content-length" in response_headers - and response_headers["content-length"].isdigit() - and int(response_headers["content-length"]) != len(body) - ): - return - - cc_req = self.parse_cache_control(request.headers) - cc = self.parse_cache_control(response_headers) - - assert request.url is not None - cache_url = self.cache_url(request.url) - logger.debug('Updating cache with response from "%s"', cache_url) - - # Delete it from the cache if we happen to have it stored there - no_store = False - if "no-store" in cc: - no_store = True - logger.debug('Response header has "no-store"') - if "no-store" in cc_req: - no_store = True - logger.debug('Request header has "no-store"') - if no_store and self.cache.get(cache_url): - logger.debug('Purging existing cache entry to honor "no-store"') - self.cache.delete(cache_url) - if no_store: - return - - # https://tools.ietf.org/html/rfc7234#section-4.1: - # A Vary header field-value of "*" always fails to match. - # Storing such a response leads to a deserialization warning - # during cache lookup and is not allowed to ever be served, - # so storing it can be avoided. - if "*" in response_headers.get("vary", ""): - logger.debug('Response header has "Vary: *"') - return - - # If we've been given an etag, then keep the response - if self.cache_etags and "etag" in response_headers: - expires_time = 0 - if response_headers.get("expires"): - expires = parsedate_tz(response_headers["expires"]) - if expires is not None: - expires_time = calendar.timegm(expires[:6]) - date - - expires_time = max(expires_time, 14 * 86400) - - logger.debug(f"etag object cached for {expires_time} seconds") - logger.debug("Caching due to etag") - self._cache_set(cache_url, request, response, body, expires_time) - - # Add to the cache any permanent redirects. We do this before looking - # that the Date headers. - elif int(response.status) in PERMANENT_REDIRECT_STATUSES: - logger.debug("Caching permanent redirect") - self._cache_set(cache_url, request, response, b"") - - # Add to the cache if the response headers demand it. If there - # is no date header then we can't do anything about expiring - # the cache. - elif "date" in response_headers: - time_tuple = parsedate_tz(response_headers["date"]) - assert time_tuple is not None - date = calendar.timegm(time_tuple[:6]) - # cache when there is a max-age > 0 - max_age = cc.get("max-age") - if max_age is not None and max_age > 0: - logger.debug("Caching b/c date exists and max-age > 0") - expires_time = max_age - self._cache_set( - cache_url, - request, - response, - body, - expires_time, - ) - - # If the request can expire, it means we should cache it - # in the meantime. - elif "expires" in response_headers: - if response_headers["expires"]: - expires = parsedate_tz(response_headers["expires"]) - if expires is not None: - expires_time = calendar.timegm(expires[:6]) - date - else: - expires_time = None - - logger.debug( - "Caching b/c of expires header. expires in {} seconds".format( - expires_time - ) - ) - self._cache_set( - cache_url, - request, - response, - body, - expires_time, - ) - - def update_cached_response( - self, request: PreparedRequest, response: HTTPResponse - ) -> HTTPResponse: - """On a 304 we will get a new set of headers that we want to - update our cached value with, assuming we have one. - - This should only ever be called when we've sent an ETag and - gotten a 304 as the response. - """ - assert request.url is not None - cache_url = self.cache_url(request.url) - cached_response = self._load_from_cache(request) - - if not cached_response: - # we didn't have a cached response - return response - - # Lets update our headers with the headers from the new request: - # http://tools.ietf.org/html/draft-ietf-httpbis-p4-conditional-26#section-4.1 - # - # The server isn't supposed to send headers that would make - # the cached body invalid. But... just in case, we'll be sure - # to strip out ones we know that might be problmatic due to - # typical assumptions. - excluded_headers = ["content-length"] - - cached_response.headers.update( - { - k: v - for k, v in response.headers.items() # type: ignore[no-untyped-call] - if k.lower() not in excluded_headers - } - ) - - # we want a 200 b/c we have content via the cache - cached_response.status = 200 - - # update our cache - self._cache_set(cache_url, request, cached_response) - - return cached_response diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/cachecontrol/filewrapper.py b/venv/lib/python3.11/site-packages/pip/_vendor/cachecontrol/filewrapper.py deleted file mode 100644 index 2514390..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/cachecontrol/filewrapper.py +++ /dev/null @@ -1,119 +0,0 @@ -# SPDX-FileCopyrightText: 2015 Eric Larson -# -# SPDX-License-Identifier: Apache-2.0 -from __future__ import annotations - -import mmap -from tempfile import NamedTemporaryFile -from typing import TYPE_CHECKING, Any, Callable - -if TYPE_CHECKING: - from http.client import HTTPResponse - - -class CallbackFileWrapper: - """ - Small wrapper around a fp object which will tee everything read into a - buffer, and when that file is closed it will execute a callback with the - contents of that buffer. - - All attributes are proxied to the underlying file object. - - This class uses members with a double underscore (__) leading prefix so as - not to accidentally shadow an attribute. - - The data is stored in a temporary file until it is all available. As long - as the temporary files directory is disk-based (sometimes it's a - memory-backed-``tmpfs`` on Linux), data will be unloaded to disk if memory - pressure is high. For small files the disk usually won't be used at all, - it'll all be in the filesystem memory cache, so there should be no - performance impact. - """ - - def __init__( - self, fp: HTTPResponse, callback: Callable[[bytes], None] | None - ) -> None: - self.__buf = NamedTemporaryFile("rb+", delete=True) - self.__fp = fp - self.__callback = callback - - def __getattr__(self, name: str) -> Any: - # The vaguaries of garbage collection means that self.__fp is - # not always set. By using __getattribute__ and the private - # name[0] allows looking up the attribute value and raising an - # AttributeError when it doesn't exist. This stop thigns from - # infinitely recursing calls to getattr in the case where - # self.__fp hasn't been set. - # - # [0] https://docs.python.org/2/reference/expressions.html#atom-identifiers - fp = self.__getattribute__("_CallbackFileWrapper__fp") - return getattr(fp, name) - - def __is_fp_closed(self) -> bool: - try: - return self.__fp.fp is None - - except AttributeError: - pass - - try: - closed: bool = self.__fp.closed - return closed - - except AttributeError: - pass - - # We just don't cache it then. - # TODO: Add some logging here... - return False - - def _close(self) -> None: - if self.__callback: - if self.__buf.tell() == 0: - # Empty file: - result = b"" - else: - # Return the data without actually loading it into memory, - # relying on Python's buffer API and mmap(). mmap() just gives - # a view directly into the filesystem's memory cache, so it - # doesn't result in duplicate memory use. - self.__buf.seek(0, 0) - result = memoryview( - mmap.mmap(self.__buf.fileno(), 0, access=mmap.ACCESS_READ) - ) - self.__callback(result) - - # We assign this to None here, because otherwise we can get into - # really tricky problems where the CPython interpreter dead locks - # because the callback is holding a reference to something which - # has a __del__ method. Setting this to None breaks the cycle - # and allows the garbage collector to do it's thing normally. - self.__callback = None - - # Closing the temporary file releases memory and frees disk space. - # Important when caching big files. - self.__buf.close() - - def read(self, amt: int | None = None) -> bytes: - data: bytes = self.__fp.read(amt) - if data: - # We may be dealing with b'', a sign that things are over: - # it's passed e.g. after we've already closed self.__buf. - self.__buf.write(data) - if self.__is_fp_closed(): - self._close() - - return data - - def _safe_read(self, amt: int) -> bytes: - data: bytes = self.__fp._safe_read(amt) # type: ignore[attr-defined] - if amt == 2 and data == b"\r\n": - # urllib executes this read to toss the CRLF at the end - # of the chunk. - return data - - self.__buf.write(data) - if self.__is_fp_closed(): - self._close() - - return data diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/cachecontrol/heuristics.py b/venv/lib/python3.11/site-packages/pip/_vendor/cachecontrol/heuristics.py deleted file mode 100644 index b9d72ca..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/cachecontrol/heuristics.py +++ /dev/null @@ -1,154 +0,0 @@ -# SPDX-FileCopyrightText: 2015 Eric Larson -# -# SPDX-License-Identifier: Apache-2.0 -from __future__ import annotations - -import calendar -import time -from datetime import datetime, timedelta, timezone -from email.utils import formatdate, parsedate, parsedate_tz -from typing import TYPE_CHECKING, Any, Mapping - -if TYPE_CHECKING: - from pip._vendor.urllib3 import HTTPResponse - -TIME_FMT = "%a, %d %b %Y %H:%M:%S GMT" - - -def expire_after(delta: timedelta, date: datetime | None = None) -> datetime: - date = date or datetime.now(timezone.utc) - return date + delta - - -def datetime_to_header(dt: datetime) -> str: - return formatdate(calendar.timegm(dt.timetuple())) - - -class BaseHeuristic: - def warning(self, response: HTTPResponse) -> str | None: - """ - Return a valid 1xx warning header value describing the cache - adjustments. - - The response is provided too allow warnings like 113 - http://tools.ietf.org/html/rfc7234#section-5.5.4 where we need - to explicitly say response is over 24 hours old. - """ - return '110 - "Response is Stale"' - - def update_headers(self, response: HTTPResponse) -> dict[str, str]: - """Update the response headers with any new headers. - - NOTE: This SHOULD always include some Warning header to - signify that the response was cached by the client, not - by way of the provided headers. - """ - return {} - - def apply(self, response: HTTPResponse) -> HTTPResponse: - updated_headers = self.update_headers(response) - - if updated_headers: - response.headers.update(updated_headers) - warning_header_value = self.warning(response) - if warning_header_value is not None: - response.headers.update({"Warning": warning_header_value}) - - return response - - -class OneDayCache(BaseHeuristic): - """ - Cache the response by providing an expires 1 day in the - future. - """ - - def update_headers(self, response: HTTPResponse) -> dict[str, str]: - headers = {} - - if "expires" not in response.headers: - date = parsedate(response.headers["date"]) - expires = expire_after(timedelta(days=1), date=datetime(*date[:6], tzinfo=timezone.utc)) # type: ignore[misc] - headers["expires"] = datetime_to_header(expires) - headers["cache-control"] = "public" - return headers - - -class ExpiresAfter(BaseHeuristic): - """ - Cache **all** requests for a defined time period. - """ - - def __init__(self, **kw: Any) -> None: - self.delta = timedelta(**kw) - - def update_headers(self, response: HTTPResponse) -> dict[str, str]: - expires = expire_after(self.delta) - return {"expires": datetime_to_header(expires), "cache-control": "public"} - - def warning(self, response: HTTPResponse) -> str | None: - tmpl = "110 - Automatically cached for %s. Response might be stale" - return tmpl % self.delta - - -class LastModified(BaseHeuristic): - """ - If there is no Expires header already, fall back on Last-Modified - using the heuristic from - http://tools.ietf.org/html/rfc7234#section-4.2.2 - to calculate a reasonable value. - - Firefox also does something like this per - https://developer.mozilla.org/en-US/docs/Web/HTTP/Caching_FAQ - http://lxr.mozilla.org/mozilla-release/source/netwerk/protocol/http/nsHttpResponseHead.cpp#397 - Unlike mozilla we limit this to 24-hr. - """ - - cacheable_by_default_statuses = { - 200, - 203, - 204, - 206, - 300, - 301, - 404, - 405, - 410, - 414, - 501, - } - - def update_headers(self, resp: HTTPResponse) -> dict[str, str]: - headers: Mapping[str, str] = resp.headers - - if "expires" in headers: - return {} - - if "cache-control" in headers and headers["cache-control"] != "public": - return {} - - if resp.status not in self.cacheable_by_default_statuses: - return {} - - if "date" not in headers or "last-modified" not in headers: - return {} - - time_tuple = parsedate_tz(headers["date"]) - assert time_tuple is not None - date = calendar.timegm(time_tuple[:6]) - last_modified = parsedate(headers["last-modified"]) - if last_modified is None: - return {} - - now = time.time() - current_age = max(0, now - date) - delta = date - calendar.timegm(last_modified) - freshness_lifetime = max(0, min(delta / 10, 24 * 3600)) - if freshness_lifetime <= current_age: - return {} - - expires = date + freshness_lifetime - return {"expires": time.strftime(TIME_FMT, time.gmtime(expires))} - - def warning(self, resp: HTTPResponse) -> str | None: - return None diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/cachecontrol/serialize.py b/venv/lib/python3.11/site-packages/pip/_vendor/cachecontrol/serialize.py deleted file mode 100644 index f9e967c..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/cachecontrol/serialize.py +++ /dev/null @@ -1,206 +0,0 @@ -# SPDX-FileCopyrightText: 2015 Eric Larson -# -# SPDX-License-Identifier: Apache-2.0 -from __future__ import annotations - -import io -from typing import IO, TYPE_CHECKING, Any, Mapping, cast - -from pip._vendor import msgpack -from pip._vendor.requests.structures import CaseInsensitiveDict -from pip._vendor.urllib3 import HTTPResponse - -if TYPE_CHECKING: - from pip._vendor.requests import PreparedRequest - - -class Serializer: - serde_version = "4" - - def dumps( - self, - request: PreparedRequest, - response: HTTPResponse, - body: bytes | None = None, - ) -> bytes: - response_headers: CaseInsensitiveDict[str] = CaseInsensitiveDict( - response.headers - ) - - if body is None: - # When a body isn't passed in, we'll read the response. We - # also update the response with a new file handler to be - # sure it acts as though it was never read. - body = response.read(decode_content=False) - response._fp = io.BytesIO(body) # type: ignore[attr-defined] - response.length_remaining = len(body) - - data = { - "response": { - "body": body, # Empty bytestring if body is stored separately - "headers": {str(k): str(v) for k, v in response.headers.items()}, # type: ignore[no-untyped-call] - "status": response.status, - "version": response.version, - "reason": str(response.reason), - "decode_content": response.decode_content, - } - } - - # Construct our vary headers - data["vary"] = {} - if "vary" in response_headers: - varied_headers = response_headers["vary"].split(",") - for header in varied_headers: - header = str(header).strip() - header_value = request.headers.get(header, None) - if header_value is not None: - header_value = str(header_value) - data["vary"][header] = header_value - - return b",".join([f"cc={self.serde_version}".encode(), self.serialize(data)]) - - def serialize(self, data: dict[str, Any]) -> bytes: - return cast(bytes, msgpack.dumps(data, use_bin_type=True)) - - def loads( - self, - request: PreparedRequest, - data: bytes, - body_file: IO[bytes] | None = None, - ) -> HTTPResponse | None: - # Short circuit if we've been given an empty set of data - if not data: - return None - - # Determine what version of the serializer the data was serialized - # with - try: - ver, data = data.split(b",", 1) - except ValueError: - ver = b"cc=0" - - # Make sure that our "ver" is actually a version and isn't a false - # positive from a , being in the data stream. - if ver[:3] != b"cc=": - data = ver + data - ver = b"cc=0" - - # Get the version number out of the cc=N - verstr = ver.split(b"=", 1)[-1].decode("ascii") - - # Dispatch to the actual load method for the given version - try: - return getattr(self, f"_loads_v{verstr}")(request, data, body_file) # type: ignore[no-any-return] - - except AttributeError: - # This is a version we don't have a loads function for, so we'll - # just treat it as a miss and return None - return None - - def prepare_response( - self, - request: PreparedRequest, - cached: Mapping[str, Any], - body_file: IO[bytes] | None = None, - ) -> HTTPResponse | None: - """Verify our vary headers match and construct a real urllib3 - HTTPResponse object. - """ - # Special case the '*' Vary value as it means we cannot actually - # determine if the cached response is suitable for this request. - # This case is also handled in the controller code when creating - # a cache entry, but is left here for backwards compatibility. - if "*" in cached.get("vary", {}): - return None - - # Ensure that the Vary headers for the cached response match our - # request - for header, value in cached.get("vary", {}).items(): - if request.headers.get(header, None) != value: - return None - - body_raw = cached["response"].pop("body") - - headers: CaseInsensitiveDict[str] = CaseInsensitiveDict( - data=cached["response"]["headers"] - ) - if headers.get("transfer-encoding", "") == "chunked": - headers.pop("transfer-encoding") - - cached["response"]["headers"] = headers - - try: - body: IO[bytes] - if body_file is None: - body = io.BytesIO(body_raw) - else: - body = body_file - except TypeError: - # This can happen if cachecontrol serialized to v1 format (pickle) - # using Python 2. A Python 2 str(byte string) will be unpickled as - # a Python 3 str (unicode string), which will cause the above to - # fail with: - # - # TypeError: 'str' does not support the buffer interface - body = io.BytesIO(body_raw.encode("utf8")) - - # Discard any `strict` parameter serialized by older version of cachecontrol. - cached["response"].pop("strict", None) - - return HTTPResponse(body=body, preload_content=False, **cached["response"]) - - def _loads_v0( - self, - request: PreparedRequest, - data: bytes, - body_file: IO[bytes] | None = None, - ) -> None: - # The original legacy cache data. This doesn't contain enough - # information to construct everything we need, so we'll treat this as - # a miss. - return None - - def _loads_v1( - self, - request: PreparedRequest, - data: bytes, - body_file: IO[bytes] | None = None, - ) -> HTTPResponse | None: - # The "v1" pickled cache format. This is no longer supported - # for security reasons, so we treat it as a miss. - return None - - def _loads_v2( - self, - request: PreparedRequest, - data: bytes, - body_file: IO[bytes] | None = None, - ) -> HTTPResponse | None: - # The "v2" compressed base64 cache format. - # This has been removed due to age and poor size/performance - # characteristics, so we treat it as a miss. - return None - - def _loads_v3( - self, - request: PreparedRequest, - data: bytes, - body_file: IO[bytes] | None = None, - ) -> None: - # Due to Python 2 encoding issues, it's impossible to know for sure - # exactly how to load v3 entries, thus we'll treat these as a miss so - # that they get rewritten out as v4 entries. - return None - - def _loads_v4( - self, - request: PreparedRequest, - data: bytes, - body_file: IO[bytes] | None = None, - ) -> HTTPResponse | None: - try: - cached = msgpack.loads(data, raw=False) - except ValueError: - return None - - return self.prepare_response(request, cached, body_file) diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/cachecontrol/wrapper.py b/venv/lib/python3.11/site-packages/pip/_vendor/cachecontrol/wrapper.py deleted file mode 100644 index f618bc3..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/cachecontrol/wrapper.py +++ /dev/null @@ -1,43 +0,0 @@ -# SPDX-FileCopyrightText: 2015 Eric Larson -# -# SPDX-License-Identifier: Apache-2.0 -from __future__ import annotations - -from typing import TYPE_CHECKING, Collection - -from pip._vendor.cachecontrol.adapter import CacheControlAdapter -from pip._vendor.cachecontrol.cache import DictCache - -if TYPE_CHECKING: - from pip._vendor import requests - - from pip._vendor.cachecontrol.cache import BaseCache - from pip._vendor.cachecontrol.controller import CacheController - from pip._vendor.cachecontrol.heuristics import BaseHeuristic - from pip._vendor.cachecontrol.serialize import Serializer - - -def CacheControl( - sess: requests.Session, - cache: BaseCache | None = None, - cache_etags: bool = True, - serializer: Serializer | None = None, - heuristic: BaseHeuristic | None = None, - controller_class: type[CacheController] | None = None, - adapter_class: type[CacheControlAdapter] | None = None, - cacheable_methods: Collection[str] | None = None, -) -> requests.Session: - cache = DictCache() if cache is None else cache - adapter_class = adapter_class or CacheControlAdapter - adapter = adapter_class( - cache, - cache_etags=cache_etags, - serializer=serializer, - heuristic=heuristic, - controller_class=controller_class, - cacheable_methods=cacheable_methods, - ) - sess.mount("http://", adapter) - sess.mount("https://", adapter) - - return sess diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/certifi/__init__.py b/venv/lib/python3.11/site-packages/pip/_vendor/certifi/__init__.py deleted file mode 100644 index 8ce89ce..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/certifi/__init__.py +++ /dev/null @@ -1,4 +0,0 @@ -from .core import contents, where - -__all__ = ["contents", "where"] -__version__ = "2023.07.22" diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/certifi/__main__.py b/venv/lib/python3.11/site-packages/pip/_vendor/certifi/__main__.py deleted file mode 100644 index 0037634..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/certifi/__main__.py +++ /dev/null @@ -1,12 +0,0 @@ -import argparse - -from pip._vendor.certifi import contents, where - -parser = argparse.ArgumentParser() -parser.add_argument("-c", "--contents", action="store_true") -args = parser.parse_args() - -if args.contents: - print(contents()) -else: - print(where()) diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/certifi/__pycache__/__init__.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/certifi/__pycache__/__init__.cpython-311.pyc deleted file mode 100644 index 54969f4..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/certifi/__pycache__/__init__.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/certifi/__pycache__/__main__.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/certifi/__pycache__/__main__.cpython-311.pyc deleted file mode 100644 index 40a368b..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/certifi/__pycache__/__main__.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/certifi/__pycache__/core.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/certifi/__pycache__/core.cpython-311.pyc deleted file mode 100644 index 4581440..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/certifi/__pycache__/core.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/certifi/cacert.pem b/venv/lib/python3.11/site-packages/pip/_vendor/certifi/cacert.pem deleted file mode 100644 index 0212369..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/certifi/cacert.pem +++ /dev/null @@ -1,4635 +0,0 @@ - -# Issuer: CN=GlobalSign Root CA O=GlobalSign nv-sa OU=Root CA -# Subject: CN=GlobalSign Root CA O=GlobalSign nv-sa OU=Root CA -# Label: "GlobalSign Root CA" -# Serial: 4835703278459707669005204 -# MD5 Fingerprint: 3e:45:52:15:09:51:92:e1:b7:5d:37:9f:b1:87:29:8a -# SHA1 Fingerprint: b1:bc:96:8b:d4:f4:9d:62:2a:a8:9a:81:f2:15:01:52:a4:1d:82:9c -# SHA256 Fingerprint: eb:d4:10:40:e4:bb:3e:c7:42:c9:e3:81:d3:1e:f2:a4:1a:48:b6:68:5c:96:e7:ce:f3:c1:df:6c:d4:33:1c:99 ------BEGIN CERTIFICATE----- -MIIDdTCCAl2gAwIBAgILBAAAAAABFUtaw5QwDQYJKoZIhvcNAQEFBQAwVzELMAkG -A1UEBhMCQkUxGTAXBgNVBAoTEEdsb2JhbFNpZ24gbnYtc2ExEDAOBgNVBAsTB1Jv -b3QgQ0ExGzAZBgNVBAMTEkdsb2JhbFNpZ24gUm9vdCBDQTAeFw05ODA5MDExMjAw -MDBaFw0yODAxMjgxMjAwMDBaMFcxCzAJBgNVBAYTAkJFMRkwFwYDVQQKExBHbG9i -YWxTaWduIG52LXNhMRAwDgYDVQQLEwdSb290IENBMRswGQYDVQQDExJHbG9iYWxT -aWduIFJvb3QgQ0EwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDaDuaZ -jc6j40+Kfvvxi4Mla+pIH/EqsLmVEQS98GPR4mdmzxzdzxtIK+6NiY6arymAZavp -xy0Sy6scTHAHoT0KMM0VjU/43dSMUBUc71DuxC73/OlS8pF94G3VNTCOXkNz8kHp -1Wrjsok6Vjk4bwY8iGlbKk3Fp1S4bInMm/k8yuX9ifUSPJJ4ltbcdG6TRGHRjcdG -snUOhugZitVtbNV4FpWi6cgKOOvyJBNPc1STE4U6G7weNLWLBYy5d4ux2x8gkasJ -U26Qzns3dLlwR5EiUWMWea6xrkEmCMgZK9FGqkjWZCrXgzT/LCrBbBlDSgeF59N8 -9iFo7+ryUp9/k5DPAgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBBjAPBgNVHRMBAf8E -BTADAQH/MB0GA1UdDgQWBBRge2YaRQ2XyolQL30EzTSo//z9SzANBgkqhkiG9w0B -AQUFAAOCAQEA1nPnfE920I2/7LqivjTFKDK1fPxsnCwrvQmeU79rXqoRSLblCKOz -yj1hTdNGCbM+w6DjY1Ub8rrvrTnhQ7k4o+YviiY776BQVvnGCv04zcQLcFGUl5gE -38NflNUVyRRBnMRddWQVDf9VMOyGj/8N7yy5Y0b2qvzfvGn9LhJIZJrglfCm7ymP -AbEVtQwdpf5pLGkkeB6zpxxxYu7KyJesF12KwvhHhm4qxFYxldBniYUr+WymXUad -DKqC5JlR3XC321Y9YeRq4VzW9v493kHMB65jUr9TU/Qr6cf9tveCX4XSQRjbgbME -HMUfpIBvFSDJ3gyICh3WZlXi/EjJKSZp4A== ------END CERTIFICATE----- - -# Issuer: CN=Entrust.net Certification Authority (2048) O=Entrust.net OU=www.entrust.net/CPS_2048 incorp. by ref. (limits liab.)/(c) 1999 Entrust.net Limited -# Subject: CN=Entrust.net Certification Authority (2048) O=Entrust.net OU=www.entrust.net/CPS_2048 incorp. by ref. (limits liab.)/(c) 1999 Entrust.net Limited -# Label: "Entrust.net Premium 2048 Secure Server CA" -# Serial: 946069240 -# MD5 Fingerprint: ee:29:31:bc:32:7e:9a:e6:e8:b5:f7:51:b4:34:71:90 -# SHA1 Fingerprint: 50:30:06:09:1d:97:d4:f5:ae:39:f7:cb:e7:92:7d:7d:65:2d:34:31 -# SHA256 Fingerprint: 6d:c4:71:72:e0:1c:bc:b0:bf:62:58:0d:89:5f:e2:b8:ac:9a:d4:f8:73:80:1e:0c:10:b9:c8:37:d2:1e:b1:77 ------BEGIN CERTIFICATE----- -MIIEKjCCAxKgAwIBAgIEOGPe+DANBgkqhkiG9w0BAQUFADCBtDEUMBIGA1UEChML -RW50cnVzdC5uZXQxQDA+BgNVBAsUN3d3dy5lbnRydXN0Lm5ldC9DUFNfMjA0OCBp -bmNvcnAuIGJ5IHJlZi4gKGxpbWl0cyBsaWFiLikxJTAjBgNVBAsTHChjKSAxOTk5 -IEVudHJ1c3QubmV0IExpbWl0ZWQxMzAxBgNVBAMTKkVudHJ1c3QubmV0IENlcnRp -ZmljYXRpb24gQXV0aG9yaXR5ICgyMDQ4KTAeFw05OTEyMjQxNzUwNTFaFw0yOTA3 -MjQxNDE1MTJaMIG0MRQwEgYDVQQKEwtFbnRydXN0Lm5ldDFAMD4GA1UECxQ3d3d3 -LmVudHJ1c3QubmV0L0NQU18yMDQ4IGluY29ycC4gYnkgcmVmLiAobGltaXRzIGxp -YWIuKTElMCMGA1UECxMcKGMpIDE5OTkgRW50cnVzdC5uZXQgTGltaXRlZDEzMDEG -A1UEAxMqRW50cnVzdC5uZXQgQ2VydGlmaWNhdGlvbiBBdXRob3JpdHkgKDIwNDgp -MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEArU1LqRKGsuqjIAcVFmQq -K0vRvwtKTY7tgHalZ7d4QMBzQshowNtTK91euHaYNZOLGp18EzoOH1u3Hs/lJBQe -sYGpjX24zGtLA/ECDNyrpUAkAH90lKGdCCmziAv1h3edVc3kw37XamSrhRSGlVuX -MlBvPci6Zgzj/L24ScF2iUkZ/cCovYmjZy/Gn7xxGWC4LeksyZB2ZnuU4q941mVT -XTzWnLLPKQP5L6RQstRIzgUyVYr9smRMDuSYB3Xbf9+5CFVghTAp+XtIpGmG4zU/ -HoZdenoVve8AjhUiVBcAkCaTvA5JaJG/+EfTnZVCwQ5N328mz8MYIWJmQ3DW1cAH -4QIDAQABo0IwQDAOBgNVHQ8BAf8EBAMCAQYwDwYDVR0TAQH/BAUwAwEB/zAdBgNV -HQ4EFgQUVeSB0RGAvtiJuQijMfmhJAkWuXAwDQYJKoZIhvcNAQEFBQADggEBADub -j1abMOdTmXx6eadNl9cZlZD7Bh/KM3xGY4+WZiT6QBshJ8rmcnPyT/4xmf3IDExo -U8aAghOY+rat2l098c5u9hURlIIM7j+VrxGrD9cv3h8Dj1csHsm7mhpElesYT6Yf -zX1XEC+bBAlahLVu2B064dae0Wx5XnkcFMXj0EyTO2U87d89vqbllRrDtRnDvV5b -u/8j72gZyxKTJ1wDLW8w0B62GqzeWvfRqqgnpv55gcR5mTNXuhKwqeBCbJPKVt7+ -bYQLCIt+jerXmCHG8+c8eS9enNFMFY3h7CI3zJpDC5fcgJCNs2ebb0gIFVbPv/Er -fF6adulZkMV8gzURZVE= ------END CERTIFICATE----- - -# Issuer: CN=Baltimore CyberTrust Root O=Baltimore OU=CyberTrust -# Subject: CN=Baltimore CyberTrust Root O=Baltimore OU=CyberTrust -# Label: "Baltimore CyberTrust Root" -# Serial: 33554617 -# MD5 Fingerprint: ac:b6:94:a5:9c:17:e0:d7:91:52:9b:b1:97:06:a6:e4 -# SHA1 Fingerprint: d4:de:20:d0:5e:66:fc:53:fe:1a:50:88:2c:78:db:28:52:ca:e4:74 -# SHA256 Fingerprint: 16:af:57:a9:f6:76:b0:ab:12:60:95:aa:5e:ba:de:f2:2a:b3:11:19:d6:44:ac:95:cd:4b:93:db:f3:f2:6a:eb ------BEGIN CERTIFICATE----- -MIIDdzCCAl+gAwIBAgIEAgAAuTANBgkqhkiG9w0BAQUFADBaMQswCQYDVQQGEwJJ -RTESMBAGA1UEChMJQmFsdGltb3JlMRMwEQYDVQQLEwpDeWJlclRydXN0MSIwIAYD -VQQDExlCYWx0aW1vcmUgQ3liZXJUcnVzdCBSb290MB4XDTAwMDUxMjE4NDYwMFoX -DTI1MDUxMjIzNTkwMFowWjELMAkGA1UEBhMCSUUxEjAQBgNVBAoTCUJhbHRpbW9y -ZTETMBEGA1UECxMKQ3liZXJUcnVzdDEiMCAGA1UEAxMZQmFsdGltb3JlIEN5YmVy -VHJ1c3QgUm9vdDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAKMEuyKr -mD1X6CZymrV51Cni4eiVgLGw41uOKymaZN+hXe2wCQVt2yguzmKiYv60iNoS6zjr -IZ3AQSsBUnuId9Mcj8e6uYi1agnnc+gRQKfRzMpijS3ljwumUNKoUMMo6vWrJYeK -mpYcqWe4PwzV9/lSEy/CG9VwcPCPwBLKBsua4dnKM3p31vjsufFoREJIE9LAwqSu -XmD+tqYF/LTdB1kC1FkYmGP1pWPgkAx9XbIGevOF6uvUA65ehD5f/xXtabz5OTZy -dc93Uk3zyZAsuT3lySNTPx8kmCFcB5kpvcY67Oduhjprl3RjM71oGDHweI12v/ye -jl0qhqdNkNwnGjkCAwEAAaNFMEMwHQYDVR0OBBYEFOWdWTCCR1jMrPoIVDaGezq1 -BE3wMBIGA1UdEwEB/wQIMAYBAf8CAQMwDgYDVR0PAQH/BAQDAgEGMA0GCSqGSIb3 -DQEBBQUAA4IBAQCFDF2O5G9RaEIFoN27TyclhAO992T9Ldcw46QQF+vaKSm2eT92 -9hkTI7gQCvlYpNRhcL0EYWoSihfVCr3FvDB81ukMJY2GQE/szKN+OMY3EU/t3Wgx -jkzSswF07r51XgdIGn9w/xZchMB5hbgF/X++ZRGjD8ACtPhSNzkE1akxehi/oCr0 -Epn3o0WC4zxe9Z2etciefC7IpJ5OCBRLbf1wbWsaY71k5h+3zvDyny67G7fyUIhz -ksLi4xaNmjICq44Y3ekQEe5+NauQrz4wlHrQMz2nZQ/1/I6eYs9HRCwBXbsdtTLS -R9I4LtD+gdwyah617jzV/OeBHRnDJELqYzmp ------END CERTIFICATE----- - -# Issuer: CN=Entrust Root Certification Authority O=Entrust, Inc. OU=www.entrust.net/CPS is incorporated by reference/(c) 2006 Entrust, Inc. -# Subject: CN=Entrust Root Certification Authority O=Entrust, Inc. OU=www.entrust.net/CPS is incorporated by reference/(c) 2006 Entrust, Inc. -# Label: "Entrust Root Certification Authority" -# Serial: 1164660820 -# MD5 Fingerprint: d6:a5:c3:ed:5d:dd:3e:00:c1:3d:87:92:1f:1d:3f:e4 -# SHA1 Fingerprint: b3:1e:b1:b7:40:e3:6c:84:02:da:dc:37:d4:4d:f5:d4:67:49:52:f9 -# SHA256 Fingerprint: 73:c1:76:43:4f:1b:c6:d5:ad:f4:5b:0e:76:e7:27:28:7c:8d:e5:76:16:c1:e6:e6:14:1a:2b:2c:bc:7d:8e:4c ------BEGIN CERTIFICATE----- -MIIEkTCCA3mgAwIBAgIERWtQVDANBgkqhkiG9w0BAQUFADCBsDELMAkGA1UEBhMC -VVMxFjAUBgNVBAoTDUVudHJ1c3QsIEluYy4xOTA3BgNVBAsTMHd3dy5lbnRydXN0 -Lm5ldC9DUFMgaXMgaW5jb3Jwb3JhdGVkIGJ5IHJlZmVyZW5jZTEfMB0GA1UECxMW -KGMpIDIwMDYgRW50cnVzdCwgSW5jLjEtMCsGA1UEAxMkRW50cnVzdCBSb290IENl -cnRpZmljYXRpb24gQXV0aG9yaXR5MB4XDTA2MTEyNzIwMjM0MloXDTI2MTEyNzIw -NTM0MlowgbAxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1FbnRydXN0LCBJbmMuMTkw -NwYDVQQLEzB3d3cuZW50cnVzdC5uZXQvQ1BTIGlzIGluY29ycG9yYXRlZCBieSBy -ZWZlcmVuY2UxHzAdBgNVBAsTFihjKSAyMDA2IEVudHJ1c3QsIEluYy4xLTArBgNV -BAMTJEVudHJ1c3QgUm9vdCBDZXJ0aWZpY2F0aW9uIEF1dGhvcml0eTCCASIwDQYJ -KoZIhvcNAQEBBQADggEPADCCAQoCggEBALaVtkNC+sZtKm9I35RMOVcF7sN5EUFo -Nu3s/poBj6E4KPz3EEZmLk0eGrEaTsbRwJWIsMn/MYszA9u3g3s+IIRe7bJWKKf4 -4LlAcTfFy0cOlypowCKVYhXbR9n10Cv/gkvJrT7eTNuQgFA/CYqEAOwwCj0Yzfv9 -KlmaI5UXLEWeH25DeW0MXJj+SKfFI0dcXv1u5x609mhF0YaDW6KKjbHjKYD+JXGI -rb68j6xSlkuqUY3kEzEZ6E5Nn9uss2rVvDlUccp6en+Q3X0dgNmBu1kmwhH+5pPi -94DkZfs0Nw4pgHBNrziGLp5/V6+eF67rHMsoIV+2HNjnogQi+dPa2MsCAwEAAaOB -sDCBrTAOBgNVHQ8BAf8EBAMCAQYwDwYDVR0TAQH/BAUwAwEB/zArBgNVHRAEJDAi -gA8yMDA2MTEyNzIwMjM0MlqBDzIwMjYxMTI3MjA1MzQyWjAfBgNVHSMEGDAWgBRo -kORnpKZTgMeGZqTx90tD+4S9bTAdBgNVHQ4EFgQUaJDkZ6SmU4DHhmak8fdLQ/uE -vW0wHQYJKoZIhvZ9B0EABBAwDhsIVjcuMTo0LjADAgSQMA0GCSqGSIb3DQEBBQUA -A4IBAQCT1DCw1wMgKtD5Y+iRDAUgqV8ZyntyTtSx29CW+1RaGSwMCPeyvIWonX9t -O1KzKtvn1ISMY/YPyyYBkVBs9F8U4pN0wBOeMDpQ47RgxRzwIkSNcUesyBrJ6Zua -AGAT/3B+XxFNSRuzFVJ7yVTav52Vr2ua2J7p8eRDjeIRRDq/r72DQnNSi6q7pynP -9WQcCk3RvKqsnyrQ/39/2n3qse0wJcGE2jTSW3iDVuycNsMm4hH2Z0kdkquM++v/ -eu6FSqdQgPCnXEqULl8FmTxSQeDNtGPPAUO6nIPcj2A781q0tHuu2guQOHXvgR1m -0vdXcDazv/wor3ElhVsT/h5/WrQ8 ------END CERTIFICATE----- - -# Issuer: CN=AAA Certificate Services O=Comodo CA Limited -# Subject: CN=AAA Certificate Services O=Comodo CA Limited -# Label: "Comodo AAA Services root" -# Serial: 1 -# MD5 Fingerprint: 49:79:04:b0:eb:87:19:ac:47:b0:bc:11:51:9b:74:d0 -# SHA1 Fingerprint: d1:eb:23:a4:6d:17:d6:8f:d9:25:64:c2:f1:f1:60:17:64:d8:e3:49 -# SHA256 Fingerprint: d7:a7:a0:fb:5d:7e:27:31:d7:71:e9:48:4e:bc:de:f7:1d:5f:0c:3e:0a:29:48:78:2b:c8:3e:e0:ea:69:9e:f4 ------BEGIN CERTIFICATE----- -MIIEMjCCAxqgAwIBAgIBATANBgkqhkiG9w0BAQUFADB7MQswCQYDVQQGEwJHQjEb -MBkGA1UECAwSR3JlYXRlciBNYW5jaGVzdGVyMRAwDgYDVQQHDAdTYWxmb3JkMRow -GAYDVQQKDBFDb21vZG8gQ0EgTGltaXRlZDEhMB8GA1UEAwwYQUFBIENlcnRpZmlj -YXRlIFNlcnZpY2VzMB4XDTA0MDEwMTAwMDAwMFoXDTI4MTIzMTIzNTk1OVowezEL -MAkGA1UEBhMCR0IxGzAZBgNVBAgMEkdyZWF0ZXIgTWFuY2hlc3RlcjEQMA4GA1UE -BwwHU2FsZm9yZDEaMBgGA1UECgwRQ29tb2RvIENBIExpbWl0ZWQxITAfBgNVBAMM -GEFBQSBDZXJ0aWZpY2F0ZSBTZXJ2aWNlczCCASIwDQYJKoZIhvcNAQEBBQADggEP -ADCCAQoCggEBAL5AnfRu4ep2hxxNRUSOvkbIgwadwSr+GB+O5AL686tdUIoWMQua -BtDFcCLNSS1UY8y2bmhGC1Pqy0wkwLxyTurxFa70VJoSCsN6sjNg4tqJVfMiWPPe -3M/vg4aijJRPn2jymJBGhCfHdr/jzDUsi14HZGWCwEiwqJH5YZ92IFCokcdmtet4 -YgNW8IoaE+oxox6gmf049vYnMlhvB/VruPsUK6+3qszWY19zjNoFmag4qMsXeDZR -rOme9Hg6jc8P2ULimAyrL58OAd7vn5lJ8S3frHRNG5i1R8XlKdH5kBjHYpy+g8cm -ez6KJcfA3Z3mNWgQIJ2P2N7Sw4ScDV7oL8kCAwEAAaOBwDCBvTAdBgNVHQ4EFgQU -oBEKIz6W8Qfs4q8p74Klf9AwpLQwDgYDVR0PAQH/BAQDAgEGMA8GA1UdEwEB/wQF -MAMBAf8wewYDVR0fBHQwcjA4oDagNIYyaHR0cDovL2NybC5jb21vZG9jYS5jb20v -QUFBQ2VydGlmaWNhdGVTZXJ2aWNlcy5jcmwwNqA0oDKGMGh0dHA6Ly9jcmwuY29t -b2RvLm5ldC9BQUFDZXJ0aWZpY2F0ZVNlcnZpY2VzLmNybDANBgkqhkiG9w0BAQUF -AAOCAQEACFb8AvCb6P+k+tZ7xkSAzk/ExfYAWMymtrwUSWgEdujm7l3sAg9g1o1Q -GE8mTgHj5rCl7r+8dFRBv/38ErjHT1r0iWAFf2C3BUrz9vHCv8S5dIa2LX1rzNLz -Rt0vxuBqw8M0Ayx9lt1awg6nCpnBBYurDC/zXDrPbDdVCYfeU0BsWO/8tqtlbgT2 -G9w84FoVxp7Z8VlIMCFlA2zs6SFz7JsDoeA3raAVGI/6ugLOpyypEBMs1OUIJqsi -l2D4kF501KKaU73yqWjgom7C12yxow+ev+to51byrvLjKzg6CYG1a4XXvi3tPxq3 -smPi9WIsgtRqAEFQ8TmDn5XpNpaYbg== ------END CERTIFICATE----- - -# Issuer: CN=QuoVadis Root CA 2 O=QuoVadis Limited -# Subject: CN=QuoVadis Root CA 2 O=QuoVadis Limited -# Label: "QuoVadis Root CA 2" -# Serial: 1289 -# MD5 Fingerprint: 5e:39:7b:dd:f8:ba:ec:82:e9:ac:62:ba:0c:54:00:2b -# SHA1 Fingerprint: ca:3a:fb:cf:12:40:36:4b:44:b2:16:20:88:80:48:39:19:93:7c:f7 -# SHA256 Fingerprint: 85:a0:dd:7d:d7:20:ad:b7:ff:05:f8:3d:54:2b:20:9d:c7:ff:45:28:f7:d6:77:b1:83:89:fe:a5:e5:c4:9e:86 ------BEGIN CERTIFICATE----- -MIIFtzCCA5+gAwIBAgICBQkwDQYJKoZIhvcNAQEFBQAwRTELMAkGA1UEBhMCQk0x -GTAXBgNVBAoTEFF1b1ZhZGlzIExpbWl0ZWQxGzAZBgNVBAMTElF1b1ZhZGlzIFJv -b3QgQ0EgMjAeFw0wNjExMjQxODI3MDBaFw0zMTExMjQxODIzMzNaMEUxCzAJBgNV -BAYTAkJNMRkwFwYDVQQKExBRdW9WYWRpcyBMaW1pdGVkMRswGQYDVQQDExJRdW9W -YWRpcyBSb290IENBIDIwggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQCa -GMpLlA0ALa8DKYrwD4HIrkwZhR0In6spRIXzL4GtMh6QRr+jhiYaHv5+HBg6XJxg -Fyo6dIMzMH1hVBHL7avg5tKifvVrbxi3Cgst/ek+7wrGsxDp3MJGF/hd/aTa/55J -WpzmM+Yklvc/ulsrHHo1wtZn/qtmUIttKGAr79dgw8eTvI02kfN/+NsRE8Scd3bB -rrcCaoF6qUWD4gXmuVbBlDePSHFjIuwXZQeVikvfj8ZaCuWw419eaxGrDPmF60Tp -+ARz8un+XJiM9XOva7R+zdRcAitMOeGylZUtQofX1bOQQ7dsE/He3fbE+Ik/0XX1 -ksOR1YqI0JDs3G3eicJlcZaLDQP9nL9bFqyS2+r+eXyt66/3FsvbzSUr5R/7mp/i -Ucw6UwxI5g69ybR2BlLmEROFcmMDBOAENisgGQLodKcftslWZvB1JdxnwQ5hYIiz -PtGo/KPaHbDRsSNU30R2be1B2MGyIrZTHN81Hdyhdyox5C315eXbyOD/5YDXC2Og -/zOhD7osFRXql7PSorW+8oyWHhqPHWykYTe5hnMz15eWniN9gqRMgeKh0bpnX5UH -oycR7hYQe7xFSkyyBNKr79X9DFHOUGoIMfmR2gyPZFwDwzqLID9ujWc9Otb+fVuI -yV77zGHcizN300QyNQliBJIWENieJ0f7OyHj+OsdWwIDAQABo4GwMIGtMA8GA1Ud -EwEB/wQFMAMBAf8wCwYDVR0PBAQDAgEGMB0GA1UdDgQWBBQahGK8SEwzJQTU7tD2 -A8QZRtGUazBuBgNVHSMEZzBlgBQahGK8SEwzJQTU7tD2A8QZRtGUa6FJpEcwRTEL -MAkGA1UEBhMCQk0xGTAXBgNVBAoTEFF1b1ZhZGlzIExpbWl0ZWQxGzAZBgNVBAMT -ElF1b1ZhZGlzIFJvb3QgQ0EgMoICBQkwDQYJKoZIhvcNAQEFBQADggIBAD4KFk2f -BluornFdLwUvZ+YTRYPENvbzwCYMDbVHZF34tHLJRqUDGCdViXh9duqWNIAXINzn -g/iN/Ae42l9NLmeyhP3ZRPx3UIHmfLTJDQtyU/h2BwdBR5YM++CCJpNVjP4iH2Bl -fF/nJrP3MpCYUNQ3cVX2kiF495V5+vgtJodmVjB3pjd4M1IQWK4/YY7yarHvGH5K -WWPKjaJW1acvvFYfzznB4vsKqBUsfU16Y8Zsl0Q80m/DShcK+JDSV6IZUaUtl0Ha -B0+pUNqQjZRG4T7wlP0QADj1O+hA4bRuVhogzG9Yje0uRY/W6ZM/57Es3zrWIozc -hLsib9D45MY56QSIPMO661V6bYCZJPVsAfv4l7CUW+v90m/xd2gNNWQjrLhVoQPR -TUIZ3Ph1WVaj+ahJefivDrkRoHy3au000LYmYjgahwz46P0u05B/B5EqHdZ+XIWD -mbA4CD/pXvk1B+TJYm5Xf6dQlfe6yJvmjqIBxdZmv3lh8zwc4bmCXF2gw+nYSL0Z -ohEUGW6yhhtoPkg3Goi3XZZenMfvJ2II4pEZXNLxId26F0KCl3GBUzGpn/Z9Yr9y -4aOTHcyKJloJONDO1w2AFrR4pTqHTI2KpdVGl/IsELm8VCLAAVBpQ570su9t+Oza -8eOx79+Rj1QqCyXBJhnEUhAFZdWCEOrCMc0u ------END CERTIFICATE----- - -# Issuer: CN=QuoVadis Root CA 3 O=QuoVadis Limited -# Subject: CN=QuoVadis Root CA 3 O=QuoVadis Limited -# Label: "QuoVadis Root CA 3" -# Serial: 1478 -# MD5 Fingerprint: 31:85:3c:62:94:97:63:b9:aa:fd:89:4e:af:6f:e0:cf -# SHA1 Fingerprint: 1f:49:14:f7:d8:74:95:1d:dd:ae:02:c0:be:fd:3a:2d:82:75:51:85 -# SHA256 Fingerprint: 18:f1:fc:7f:20:5d:f8:ad:dd:eb:7f:e0:07:dd:57:e3:af:37:5a:9c:4d:8d:73:54:6b:f4:f1:fe:d1:e1:8d:35 ------BEGIN CERTIFICATE----- -MIIGnTCCBIWgAwIBAgICBcYwDQYJKoZIhvcNAQEFBQAwRTELMAkGA1UEBhMCQk0x -GTAXBgNVBAoTEFF1b1ZhZGlzIExpbWl0ZWQxGzAZBgNVBAMTElF1b1ZhZGlzIFJv -b3QgQ0EgMzAeFw0wNjExMjQxOTExMjNaFw0zMTExMjQxOTA2NDRaMEUxCzAJBgNV -BAYTAkJNMRkwFwYDVQQKExBRdW9WYWRpcyBMaW1pdGVkMRswGQYDVQQDExJRdW9W -YWRpcyBSb290IENBIDMwggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQDM -V0IWVJzmmNPTTe7+7cefQzlKZbPoFog02w1ZkXTPkrgEQK0CSzGrvI2RaNggDhoB -4hp7Thdd4oq3P5kazethq8Jlph+3t723j/z9cI8LoGe+AaJZz3HmDyl2/7FWeUUr -H556VOijKTVopAFPD6QuN+8bv+OPEKhyq1hX51SGyMnzW9os2l2ObjyjPtr7guXd -8lyyBTNvijbO0BNO/79KDDRMpsMhvVAEVeuxu537RR5kFd5VAYwCdrXLoT9Cabwv -vWhDFlaJKjdhkf2mrk7AyxRllDdLkgbvBNDInIjbC3uBr7E9KsRlOni27tyAsdLT -mZw67mtaa7ONt9XOnMK+pUsvFrGeaDsGb659n/je7Mwpp5ijJUMv7/FfJuGITfhe -btfZFG4ZM2mnO4SJk8RTVROhUXhA+LjJou57ulJCg54U7QVSWllWp5f8nT8KKdjc -T5EOE7zelaTfi5m+rJsziO+1ga8bxiJTyPbH7pcUsMV8eFLI8M5ud2CEpukqdiDt -WAEXMJPpGovgc2PZapKUSU60rUqFxKMiMPwJ7Wgic6aIDFUhWMXhOp8q3crhkODZ -c6tsgLjoC2SToJyMGf+z0gzskSaHirOi4XCPLArlzW1oUevaPwV/izLmE1xr/l9A -4iLItLRkT9a6fUg+qGkM17uGcclzuD87nSVL2v9A6wIDAQABo4IBlTCCAZEwDwYD -VR0TAQH/BAUwAwEB/zCB4QYDVR0gBIHZMIHWMIHTBgkrBgEEAb5YAAMwgcUwgZMG -CCsGAQUFBwICMIGGGoGDQW55IHVzZSBvZiB0aGlzIENlcnRpZmljYXRlIGNvbnN0 -aXR1dGVzIGFjY2VwdGFuY2Ugb2YgdGhlIFF1b1ZhZGlzIFJvb3QgQ0EgMyBDZXJ0 -aWZpY2F0ZSBQb2xpY3kgLyBDZXJ0aWZpY2F0aW9uIFByYWN0aWNlIFN0YXRlbWVu -dC4wLQYIKwYBBQUHAgEWIWh0dHA6Ly93d3cucXVvdmFkaXNnbG9iYWwuY29tL2Nw -czALBgNVHQ8EBAMCAQYwHQYDVR0OBBYEFPLAE+CCQz777i9nMpY1XNu4ywLQMG4G -A1UdIwRnMGWAFPLAE+CCQz777i9nMpY1XNu4ywLQoUmkRzBFMQswCQYDVQQGEwJC -TTEZMBcGA1UEChMQUXVvVmFkaXMgTGltaXRlZDEbMBkGA1UEAxMSUXVvVmFkaXMg -Um9vdCBDQSAzggIFxjANBgkqhkiG9w0BAQUFAAOCAgEAT62gLEz6wPJv92ZVqyM0 -7ucp2sNbtrCD2dDQ4iH782CnO11gUyeim/YIIirnv6By5ZwkajGxkHon24QRiSem -d1o417+shvzuXYO8BsbRd2sPbSQvS3pspweWyuOEn62Iix2rFo1bZhfZFvSLgNLd -+LJ2w/w4E6oM3kJpK27zPOuAJ9v1pkQNn1pVWQvVDVJIxa6f8i+AxeoyUDUSly7B -4f/xI4hROJ/yZlZ25w9Rl6VSDE1JUZU2Pb+iSwwQHYaZTKrzchGT5Or2m9qoXadN -t54CrnMAyNojA+j56hl0YgCUyyIgvpSnWbWCar6ZeXqp8kokUvd0/bpO5qgdAm6x -DYBEwa7TIzdfu4V8K5Iu6H6li92Z4b8nby1dqnuH/grdS/yO9SbkbnBCbjPsMZ57 -k8HkyWkaPcBrTiJt7qtYTcbQQcEr6k8Sh17rRdhs9ZgC06DYVYoGmRmioHfRMJ6s -zHXug/WwYjnPbFfiTNKRCw51KBuav/0aQ/HKd/s7j2G4aSgWQgRecCocIdiP4b0j -Wy10QJLZYxkNc91pvGJHvOB0K7Lrfb5BG7XARsWhIstfTsEokt4YutUqKLsRixeT -mJlglFwjz1onl14LBQaTNx47aTbrqZ5hHY8y2o4M1nQ+ewkk2gF3R8Q7zTSMmfXK -4SVhM7JZG+Ju1zdXtg2pEto= ------END CERTIFICATE----- - -# Issuer: O=SECOM Trust.net OU=Security Communication RootCA1 -# Subject: O=SECOM Trust.net OU=Security Communication RootCA1 -# Label: "Security Communication Root CA" -# Serial: 0 -# MD5 Fingerprint: f1:bc:63:6a:54:e0:b5:27:f5:cd:e7:1a:e3:4d:6e:4a -# SHA1 Fingerprint: 36:b1:2b:49:f9:81:9e:d7:4c:9e:bc:38:0f:c6:56:8f:5d:ac:b2:f7 -# SHA256 Fingerprint: e7:5e:72:ed:9f:56:0e:ec:6e:b4:80:00:73:a4:3f:c3:ad:19:19:5a:39:22:82:01:78:95:97:4a:99:02:6b:6c ------BEGIN CERTIFICATE----- -MIIDWjCCAkKgAwIBAgIBADANBgkqhkiG9w0BAQUFADBQMQswCQYDVQQGEwJKUDEY -MBYGA1UEChMPU0VDT00gVHJ1c3QubmV0MScwJQYDVQQLEx5TZWN1cml0eSBDb21t -dW5pY2F0aW9uIFJvb3RDQTEwHhcNMDMwOTMwMDQyMDQ5WhcNMjMwOTMwMDQyMDQ5 -WjBQMQswCQYDVQQGEwJKUDEYMBYGA1UEChMPU0VDT00gVHJ1c3QubmV0MScwJQYD -VQQLEx5TZWN1cml0eSBDb21tdW5pY2F0aW9uIFJvb3RDQTEwggEiMA0GCSqGSIb3 -DQEBAQUAA4IBDwAwggEKAoIBAQCzs/5/022x7xZ8V6UMbXaKL0u/ZPtM7orw8yl8 -9f/uKuDp6bpbZCKamm8sOiZpUQWZJtzVHGpxxpp9Hp3dfGzGjGdnSj74cbAZJ6kJ -DKaVv0uMDPpVmDvY6CKhS3E4eayXkmmziX7qIWgGmBSWh9JhNrxtJ1aeV+7AwFb9 -Ms+k2Y7CI9eNqPPYJayX5HA49LY6tJ07lyZDo6G8SVlyTCMwhwFY9k6+HGhWZq/N -QV3Is00qVUarH9oe4kA92819uZKAnDfdDJZkndwi92SL32HeFZRSFaB9UslLqCHJ -xrHty8OVYNEP8Ktw+N/LTX7s1vqr2b1/VPKl6Xn62dZ2JChzAgMBAAGjPzA9MB0G -A1UdDgQWBBSgc0mZaNyFW2XjmygvV5+9M7wHSDALBgNVHQ8EBAMCAQYwDwYDVR0T -AQH/BAUwAwEB/zANBgkqhkiG9w0BAQUFAAOCAQEAaECpqLvkT115swW1F7NgE+vG -kl3g0dNq/vu+m22/xwVtWSDEHPC32oRYAmP6SBbvT6UL90qY8j+eG61Ha2POCEfr -Uj94nK9NrvjVT8+amCoQQTlSxN3Zmw7vkwGusi7KaEIkQmywszo+zenaSMQVy+n5 -Bw+SUEmK3TGXX8npN6o7WWWXlDLJs58+OmJYxUmtYg5xpTKqL8aJdkNAExNnPaJU -JRDL8Try2frbSVa7pv6nQTXD4IhhyYjH3zYQIphZ6rBK+1YWc26sTfcioU+tHXot -RSflMMFe8toTyyVCUZVHA4xsIcx0Qu1T/zOLjw9XARYvz6buyXAiFL39vmwLAw== ------END CERTIFICATE----- - -# Issuer: CN=XRamp Global Certification Authority O=XRamp Security Services Inc OU=www.xrampsecurity.com -# Subject: CN=XRamp Global Certification Authority O=XRamp Security Services Inc OU=www.xrampsecurity.com -# Label: "XRamp Global CA Root" -# Serial: 107108908803651509692980124233745014957 -# MD5 Fingerprint: a1:0b:44:b3:ca:10:d8:00:6e:9d:0f:d8:0f:92:0a:d1 -# SHA1 Fingerprint: b8:01:86:d1:eb:9c:86:a5:41:04:cf:30:54:f3:4c:52:b7:e5:58:c6 -# SHA256 Fingerprint: ce:cd:dc:90:50:99:d8:da:df:c5:b1:d2:09:b7:37:cb:e2:c1:8c:fb:2c:10:c0:ff:0b:cf:0d:32:86:fc:1a:a2 ------BEGIN CERTIFICATE----- -MIIEMDCCAxigAwIBAgIQUJRs7Bjq1ZxN1ZfvdY+grTANBgkqhkiG9w0BAQUFADCB -gjELMAkGA1UEBhMCVVMxHjAcBgNVBAsTFXd3dy54cmFtcHNlY3VyaXR5LmNvbTEk -MCIGA1UEChMbWFJhbXAgU2VjdXJpdHkgU2VydmljZXMgSW5jMS0wKwYDVQQDEyRY -UmFtcCBHbG9iYWwgQ2VydGlmaWNhdGlvbiBBdXRob3JpdHkwHhcNMDQxMTAxMTcx -NDA0WhcNMzUwMTAxMDUzNzE5WjCBgjELMAkGA1UEBhMCVVMxHjAcBgNVBAsTFXd3 -dy54cmFtcHNlY3VyaXR5LmNvbTEkMCIGA1UEChMbWFJhbXAgU2VjdXJpdHkgU2Vy -dmljZXMgSW5jMS0wKwYDVQQDEyRYUmFtcCBHbG9iYWwgQ2VydGlmaWNhdGlvbiBB -dXRob3JpdHkwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCYJB69FbS6 -38eMpSe2OAtp87ZOqCwuIR1cRN8hXX4jdP5efrRKt6atH67gBhbim1vZZ3RrXYCP -KZ2GG9mcDZhtdhAoWORlsH9KmHmf4MMxfoArtYzAQDsRhtDLooY2YKTVMIJt2W7Q -DxIEM5dfT2Fa8OT5kavnHTu86M/0ay00fOJIYRyO82FEzG+gSqmUsE3a56k0enI4 -qEHMPJQRfevIpoy3hsvKMzvZPTeL+3o+hiznc9cKV6xkmxnr9A8ECIqsAxcZZPRa -JSKNNCyy9mgdEm3Tih4U2sSPpuIjhdV6Db1q4Ons7Be7QhtnqiXtRYMh/MHJfNVi -PvryxS3T/dRlAgMBAAGjgZ8wgZwwEwYJKwYBBAGCNxQCBAYeBABDAEEwCwYDVR0P -BAQDAgGGMA8GA1UdEwEB/wQFMAMBAf8wHQYDVR0OBBYEFMZPoj0GY4QJnM5i5ASs -jVy16bYbMDYGA1UdHwQvMC0wK6ApoCeGJWh0dHA6Ly9jcmwueHJhbXBzZWN1cml0 -eS5jb20vWEdDQS5jcmwwEAYJKwYBBAGCNxUBBAMCAQEwDQYJKoZIhvcNAQEFBQAD -ggEBAJEVOQMBG2f7Shz5CmBbodpNl2L5JFMn14JkTpAuw0kbK5rc/Kh4ZzXxHfAR -vbdI4xD2Dd8/0sm2qlWkSLoC295ZLhVbO50WfUfXN+pfTXYSNrsf16GBBEYgoyxt -qZ4Bfj8pzgCT3/3JknOJiWSe5yvkHJEs0rnOfc5vMZnT5r7SHpDwCRR5XCOrTdLa -IR9NmXmd4c8nnxCbHIgNsIpkQTG4DmyQJKSbXHGPurt+HBvbaoAPIbzp26a3QPSy -i6mx5O+aGtA9aZnuqCij4Tyz8LIRnM98QObd50N9otg6tamN8jSZxNQQ4Qb9CYQQ -O+7ETPTsJ3xCwnR8gooJybQDJbw= ------END CERTIFICATE----- - -# Issuer: O=The Go Daddy Group, Inc. OU=Go Daddy Class 2 Certification Authority -# Subject: O=The Go Daddy Group, Inc. OU=Go Daddy Class 2 Certification Authority -# Label: "Go Daddy Class 2 CA" -# Serial: 0 -# MD5 Fingerprint: 91:de:06:25:ab:da:fd:32:17:0c:bb:25:17:2a:84:67 -# SHA1 Fingerprint: 27:96:ba:e6:3f:18:01:e2:77:26:1b:a0:d7:77:70:02:8f:20:ee:e4 -# SHA256 Fingerprint: c3:84:6b:f2:4b:9e:93:ca:64:27:4c:0e:c6:7c:1e:cc:5e:02:4f:fc:ac:d2:d7:40:19:35:0e:81:fe:54:6a:e4 ------BEGIN CERTIFICATE----- -MIIEADCCAuigAwIBAgIBADANBgkqhkiG9w0BAQUFADBjMQswCQYDVQQGEwJVUzEh -MB8GA1UEChMYVGhlIEdvIERhZGR5IEdyb3VwLCBJbmMuMTEwLwYDVQQLEyhHbyBE -YWRkeSBDbGFzcyAyIENlcnRpZmljYXRpb24gQXV0aG9yaXR5MB4XDTA0MDYyOTE3 -MDYyMFoXDTM0MDYyOTE3MDYyMFowYzELMAkGA1UEBhMCVVMxITAfBgNVBAoTGFRo -ZSBHbyBEYWRkeSBHcm91cCwgSW5jLjExMC8GA1UECxMoR28gRGFkZHkgQ2xhc3Mg -MiBDZXJ0aWZpY2F0aW9uIEF1dGhvcml0eTCCASAwDQYJKoZIhvcNAQEBBQADggEN -ADCCAQgCggEBAN6d1+pXGEmhW+vXX0iG6r7d/+TvZxz0ZWizV3GgXne77ZtJ6XCA -PVYYYwhv2vLM0D9/AlQiVBDYsoHUwHU9S3/Hd8M+eKsaA7Ugay9qK7HFiH7Eux6w -wdhFJ2+qN1j3hybX2C32qRe3H3I2TqYXP2WYktsqbl2i/ojgC95/5Y0V4evLOtXi -EqITLdiOr18SPaAIBQi2XKVlOARFmR6jYGB0xUGlcmIbYsUfb18aQr4CUWWoriMY -avx4A6lNf4DD+qta/KFApMoZFv6yyO9ecw3ud72a9nmYvLEHZ6IVDd2gWMZEewo+ -YihfukEHU1jPEX44dMX4/7VpkI+EdOqXG68CAQOjgcAwgb0wHQYDVR0OBBYEFNLE -sNKR1EwRcbNhyz2h/t2oatTjMIGNBgNVHSMEgYUwgYKAFNLEsNKR1EwRcbNhyz2h -/t2oatTjoWekZTBjMQswCQYDVQQGEwJVUzEhMB8GA1UEChMYVGhlIEdvIERhZGR5 -IEdyb3VwLCBJbmMuMTEwLwYDVQQLEyhHbyBEYWRkeSBDbGFzcyAyIENlcnRpZmlj -YXRpb24gQXV0aG9yaXR5ggEAMAwGA1UdEwQFMAMBAf8wDQYJKoZIhvcNAQEFBQAD -ggEBADJL87LKPpH8EsahB4yOd6AzBhRckB4Y9wimPQoZ+YeAEW5p5JYXMP80kWNy -OO7MHAGjHZQopDH2esRU1/blMVgDoszOYtuURXO1v0XJJLXVggKtI3lpjbi2Tc7P -TMozI+gciKqdi0FuFskg5YmezTvacPd+mSYgFFQlq25zheabIZ0KbIIOqPjCDPoQ -HmyW74cNxA9hi63ugyuV+I6ShHI56yDqg+2DzZduCLzrTia2cyvk0/ZM/iZx4mER -dEr/VxqHD3VILs9RaRegAhJhldXRQLIQTO7ErBBDpqWeCtWVYpoNz4iCxTIM5Cuf -ReYNnyicsbkqWletNw+vHX/bvZ8= ------END CERTIFICATE----- - -# Issuer: O=Starfield Technologies, Inc. OU=Starfield Class 2 Certification Authority -# Subject: O=Starfield Technologies, Inc. OU=Starfield Class 2 Certification Authority -# Label: "Starfield Class 2 CA" -# Serial: 0 -# MD5 Fingerprint: 32:4a:4b:bb:c8:63:69:9b:be:74:9a:c6:dd:1d:46:24 -# SHA1 Fingerprint: ad:7e:1c:28:b0:64:ef:8f:60:03:40:20:14:c3:d0:e3:37:0e:b5:8a -# SHA256 Fingerprint: 14:65:fa:20:53:97:b8:76:fa:a6:f0:a9:95:8e:55:90:e4:0f:cc:7f:aa:4f:b7:c2:c8:67:75:21:fb:5f:b6:58 ------BEGIN CERTIFICATE----- -MIIEDzCCAvegAwIBAgIBADANBgkqhkiG9w0BAQUFADBoMQswCQYDVQQGEwJVUzEl -MCMGA1UEChMcU3RhcmZpZWxkIFRlY2hub2xvZ2llcywgSW5jLjEyMDAGA1UECxMp -U3RhcmZpZWxkIENsYXNzIDIgQ2VydGlmaWNhdGlvbiBBdXRob3JpdHkwHhcNMDQw -NjI5MTczOTE2WhcNMzQwNjI5MTczOTE2WjBoMQswCQYDVQQGEwJVUzElMCMGA1UE -ChMcU3RhcmZpZWxkIFRlY2hub2xvZ2llcywgSW5jLjEyMDAGA1UECxMpU3RhcmZp -ZWxkIENsYXNzIDIgQ2VydGlmaWNhdGlvbiBBdXRob3JpdHkwggEgMA0GCSqGSIb3 -DQEBAQUAA4IBDQAwggEIAoIBAQC3Msj+6XGmBIWtDBFk385N78gDGIc/oav7PKaf -8MOh2tTYbitTkPskpD6E8J7oX+zlJ0T1KKY/e97gKvDIr1MvnsoFAZMej2YcOadN -+lq2cwQlZut3f+dZxkqZJRRU6ybH838Z1TBwj6+wRir/resp7defqgSHo9T5iaU0 -X9tDkYI22WY8sbi5gv2cOj4QyDvvBmVmepsZGD3/cVE8MC5fvj13c7JdBmzDI1aa -K4UmkhynArPkPw2vCHmCuDY96pzTNbO8acr1zJ3o/WSNF4Azbl5KXZnJHoe0nRrA -1W4TNSNe35tfPe/W93bC6j67eA0cQmdrBNj41tpvi/JEoAGrAgEDo4HFMIHCMB0G -A1UdDgQWBBS/X7fRzt0fhvRbVazc1xDCDqmI5zCBkgYDVR0jBIGKMIGHgBS/X7fR -zt0fhvRbVazc1xDCDqmI56FspGowaDELMAkGA1UEBhMCVVMxJTAjBgNVBAoTHFN0 -YXJmaWVsZCBUZWNobm9sb2dpZXMsIEluYy4xMjAwBgNVBAsTKVN0YXJmaWVsZCBD -bGFzcyAyIENlcnRpZmljYXRpb24gQXV0aG9yaXR5ggEAMAwGA1UdEwQFMAMBAf8w -DQYJKoZIhvcNAQEFBQADggEBAAWdP4id0ckaVaGsafPzWdqbAYcaT1epoXkJKtv3 -L7IezMdeatiDh6GX70k1PncGQVhiv45YuApnP+yz3SFmH8lU+nLMPUxA2IGvd56D -eruix/U0F47ZEUD0/CwqTRV/p2JdLiXTAAsgGh1o+Re49L2L7ShZ3U0WixeDyLJl -xy16paq8U4Zt3VekyvggQQto8PT7dL5WXXp59fkdheMtlb71cZBDzI0fmgAKhynp -VSJYACPq4xJDKVtHCN2MQWplBqjlIapBtJUhlbl90TSrE9atvNziPTnNvT51cKEY -WQPJIrSPnNVeKtelttQKbfi3QBFGmh95DmK/D5fs4C8fF5Q= ------END CERTIFICATE----- - -# Issuer: CN=DigiCert Assured ID Root CA O=DigiCert Inc OU=www.digicert.com -# Subject: CN=DigiCert Assured ID Root CA O=DigiCert Inc OU=www.digicert.com -# Label: "DigiCert Assured ID Root CA" -# Serial: 17154717934120587862167794914071425081 -# MD5 Fingerprint: 87:ce:0b:7b:2a:0e:49:00:e1:58:71:9b:37:a8:93:72 -# SHA1 Fingerprint: 05:63:b8:63:0d:62:d7:5a:bb:c8:ab:1e:4b:df:b5:a8:99:b2:4d:43 -# SHA256 Fingerprint: 3e:90:99:b5:01:5e:8f:48:6c:00:bc:ea:9d:11:1e:e7:21:fa:ba:35:5a:89:bc:f1:df:69:56:1e:3d:c6:32:5c ------BEGIN CERTIFICATE----- -MIIDtzCCAp+gAwIBAgIQDOfg5RfYRv6P5WD8G/AwOTANBgkqhkiG9w0BAQUFADBl -MQswCQYDVQQGEwJVUzEVMBMGA1UEChMMRGlnaUNlcnQgSW5jMRkwFwYDVQQLExB3 -d3cuZGlnaWNlcnQuY29tMSQwIgYDVQQDExtEaWdpQ2VydCBBc3N1cmVkIElEIFJv -b3QgQ0EwHhcNMDYxMTEwMDAwMDAwWhcNMzExMTEwMDAwMDAwWjBlMQswCQYDVQQG -EwJVUzEVMBMGA1UEChMMRGlnaUNlcnQgSW5jMRkwFwYDVQQLExB3d3cuZGlnaWNl -cnQuY29tMSQwIgYDVQQDExtEaWdpQ2VydCBBc3N1cmVkIElEIFJvb3QgQ0EwggEi -MA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCtDhXO5EOAXLGH87dg+XESpa7c -JpSIqvTO9SA5KFhgDPiA2qkVlTJhPLWxKISKityfCgyDF3qPkKyK53lTXDGEKvYP -mDI2dsze3Tyoou9q+yHyUmHfnyDXH+Kx2f4YZNISW1/5WBg1vEfNoTb5a3/UsDg+ -wRvDjDPZ2C8Y/igPs6eD1sNuRMBhNZYW/lmci3Zt1/GiSw0r/wty2p5g0I6QNcZ4 -VYcgoc/lbQrISXwxmDNsIumH0DJaoroTghHtORedmTpyoeb6pNnVFzF1roV9Iq4/ -AUaG9ih5yLHa5FcXxH4cDrC0kqZWs72yl+2qp/C3xag/lRbQ/6GW6whfGHdPAgMB -AAGjYzBhMA4GA1UdDwEB/wQEAwIBhjAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQW -BBRF66Kv9JLLgjEtUYunpyGd823IDzAfBgNVHSMEGDAWgBRF66Kv9JLLgjEtUYun -pyGd823IDzANBgkqhkiG9w0BAQUFAAOCAQEAog683+Lt8ONyc3pklL/3cmbYMuRC -dWKuh+vy1dneVrOfzM4UKLkNl2BcEkxY5NM9g0lFWJc1aRqoR+pWxnmrEthngYTf -fwk8lOa4JiwgvT2zKIn3X/8i4peEH+ll74fg38FnSbNd67IJKusm7Xi+fT8r87cm -NW1fiQG2SVufAQWbqz0lwcy2f8Lxb4bG+mRo64EtlOtCt/qMHt1i8b5QZ7dsvfPx -H2sMNgcWfzd8qVttevESRmCD1ycEvkvOl77DZypoEd+A5wwzZr8TDRRu838fYxAe -+o0bJW1sj6W3YQGx0qMmoRBxna3iw/nDmVG3KwcIzi7mULKn+gpFL6Lw8g== ------END CERTIFICATE----- - -# Issuer: CN=DigiCert Global Root CA O=DigiCert Inc OU=www.digicert.com -# Subject: CN=DigiCert Global Root CA O=DigiCert Inc OU=www.digicert.com -# Label: "DigiCert Global Root CA" -# Serial: 10944719598952040374951832963794454346 -# MD5 Fingerprint: 79:e4:a9:84:0d:7d:3a:96:d7:c0:4f:e2:43:4c:89:2e -# SHA1 Fingerprint: a8:98:5d:3a:65:e5:e5:c4:b2:d7:d6:6d:40:c6:dd:2f:b1:9c:54:36 -# SHA256 Fingerprint: 43:48:a0:e9:44:4c:78:cb:26:5e:05:8d:5e:89:44:b4:d8:4f:96:62:bd:26:db:25:7f:89:34:a4:43:c7:01:61 ------BEGIN CERTIFICATE----- -MIIDrzCCApegAwIBAgIQCDvgVpBCRrGhdWrJWZHHSjANBgkqhkiG9w0BAQUFADBh -MQswCQYDVQQGEwJVUzEVMBMGA1UEChMMRGlnaUNlcnQgSW5jMRkwFwYDVQQLExB3 -d3cuZGlnaWNlcnQuY29tMSAwHgYDVQQDExdEaWdpQ2VydCBHbG9iYWwgUm9vdCBD -QTAeFw0wNjExMTAwMDAwMDBaFw0zMTExMTAwMDAwMDBaMGExCzAJBgNVBAYTAlVT -MRUwEwYDVQQKEwxEaWdpQ2VydCBJbmMxGTAXBgNVBAsTEHd3dy5kaWdpY2VydC5j -b20xIDAeBgNVBAMTF0RpZ2lDZXJ0IEdsb2JhbCBSb290IENBMIIBIjANBgkqhkiG -9w0BAQEFAAOCAQ8AMIIBCgKCAQEA4jvhEXLeqKTTo1eqUKKPC3eQyaKl7hLOllsB -CSDMAZOnTjC3U/dDxGkAV53ijSLdhwZAAIEJzs4bg7/fzTtxRuLWZscFs3YnFo97 -nh6Vfe63SKMI2tavegw5BmV/Sl0fvBf4q77uKNd0f3p4mVmFaG5cIzJLv07A6Fpt -43C/dxC//AH2hdmoRBBYMql1GNXRor5H4idq9Joz+EkIYIvUX7Q6hL+hqkpMfT7P -T19sdl6gSzeRntwi5m3OFBqOasv+zbMUZBfHWymeMr/y7vrTC0LUq7dBMtoM1O/4 -gdW7jVg/tRvoSSiicNoxBN33shbyTApOB6jtSj1etX+jkMOvJwIDAQABo2MwYTAO -BgNVHQ8BAf8EBAMCAYYwDwYDVR0TAQH/BAUwAwEB/zAdBgNVHQ4EFgQUA95QNVbR -TLtm8KPiGxvDl7I90VUwHwYDVR0jBBgwFoAUA95QNVbRTLtm8KPiGxvDl7I90VUw -DQYJKoZIhvcNAQEFBQADggEBAMucN6pIExIK+t1EnE9SsPTfrgT1eXkIoyQY/Esr -hMAtudXH/vTBH1jLuG2cenTnmCmrEbXjcKChzUyImZOMkXDiqw8cvpOp/2PV5Adg -06O/nVsJ8dWO41P0jmP6P6fbtGbfYmbW0W5BjfIttep3Sp+dWOIrWcBAI+0tKIJF -PnlUkiaY4IBIqDfv8NZ5YBberOgOzW6sRBc4L0na4UU+Krk2U886UAb3LujEV0ls -YSEY1QSteDwsOoBrp+uvFRTp2InBuThs4pFsiv9kuXclVzDAGySj4dzp30d8tbQk -CAUw7C29C79Fv1C5qfPrmAESrciIxpg0X40KPMbp1ZWVbd4= ------END CERTIFICATE----- - -# Issuer: CN=DigiCert High Assurance EV Root CA O=DigiCert Inc OU=www.digicert.com -# Subject: CN=DigiCert High Assurance EV Root CA O=DigiCert Inc OU=www.digicert.com -# Label: "DigiCert High Assurance EV Root CA" -# Serial: 3553400076410547919724730734378100087 -# MD5 Fingerprint: d4:74:de:57:5c:39:b2:d3:9c:85:83:c5:c0:65:49:8a -# SHA1 Fingerprint: 5f:b7:ee:06:33:e2:59:db:ad:0c:4c:9a:e6:d3:8f:1a:61:c7:dc:25 -# SHA256 Fingerprint: 74:31:e5:f4:c3:c1:ce:46:90:77:4f:0b:61:e0:54:40:88:3b:a9:a0:1e:d0:0b:a6:ab:d7:80:6e:d3:b1:18:cf ------BEGIN CERTIFICATE----- -MIIDxTCCAq2gAwIBAgIQAqxcJmoLQJuPC3nyrkYldzANBgkqhkiG9w0BAQUFADBs -MQswCQYDVQQGEwJVUzEVMBMGA1UEChMMRGlnaUNlcnQgSW5jMRkwFwYDVQQLExB3 -d3cuZGlnaWNlcnQuY29tMSswKQYDVQQDEyJEaWdpQ2VydCBIaWdoIEFzc3VyYW5j -ZSBFViBSb290IENBMB4XDTA2MTExMDAwMDAwMFoXDTMxMTExMDAwMDAwMFowbDEL -MAkGA1UEBhMCVVMxFTATBgNVBAoTDERpZ2lDZXJ0IEluYzEZMBcGA1UECxMQd3d3 -LmRpZ2ljZXJ0LmNvbTErMCkGA1UEAxMiRGlnaUNlcnQgSGlnaCBBc3N1cmFuY2Ug -RVYgUm9vdCBDQTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMbM5XPm -+9S75S0tMqbf5YE/yc0lSbZxKsPVlDRnogocsF9ppkCxxLeyj9CYpKlBWTrT3JTW -PNt0OKRKzE0lgvdKpVMSOO7zSW1xkX5jtqumX8OkhPhPYlG++MXs2ziS4wblCJEM -xChBVfvLWokVfnHoNb9Ncgk9vjo4UFt3MRuNs8ckRZqnrG0AFFoEt7oT61EKmEFB -Ik5lYYeBQVCmeVyJ3hlKV9Uu5l0cUyx+mM0aBhakaHPQNAQTXKFx01p8VdteZOE3 -hzBWBOURtCmAEvF5OYiiAhF8J2a3iLd48soKqDirCmTCv2ZdlYTBoSUeh10aUAsg -EsxBu24LUTi4S8sCAwEAAaNjMGEwDgYDVR0PAQH/BAQDAgGGMA8GA1UdEwEB/wQF -MAMBAf8wHQYDVR0OBBYEFLE+w2kD+L9HAdSYJhoIAu9jZCvDMB8GA1UdIwQYMBaA -FLE+w2kD+L9HAdSYJhoIAu9jZCvDMA0GCSqGSIb3DQEBBQUAA4IBAQAcGgaX3Nec -nzyIZgYIVyHbIUf4KmeqvxgydkAQV8GK83rZEWWONfqe/EW1ntlMMUu4kehDLI6z -eM7b41N5cdblIZQB2lWHmiRk9opmzN6cN82oNLFpmyPInngiK3BD41VHMWEZ71jF -hS9OMPagMRYjyOfiZRYzy78aG6A9+MpeizGLYAiJLQwGXFK3xPkKmNEVX58Svnw2 -Yzi9RKR/5CYrCsSXaQ3pjOLAEFe4yHYSkVXySGnYvCoCWw9E1CAx2/S6cCZdkGCe -vEsXCS+0yx5DaMkHJ8HSXPfqIbloEpw8nL+e/IBcm2PN7EeqJSdnoDfzAIJ9VNep -+OkuE6N36B9K ------END CERTIFICATE----- - -# Issuer: CN=SwissSign Gold CA - G2 O=SwissSign AG -# Subject: CN=SwissSign Gold CA - G2 O=SwissSign AG -# Label: "SwissSign Gold CA - G2" -# Serial: 13492815561806991280 -# MD5 Fingerprint: 24:77:d9:a8:91:d1:3b:fa:88:2d:c2:ff:f8:cd:33:93 -# SHA1 Fingerprint: d8:c5:38:8a:b7:30:1b:1b:6e:d4:7a:e6:45:25:3a:6f:9f:1a:27:61 -# SHA256 Fingerprint: 62:dd:0b:e9:b9:f5:0a:16:3e:a0:f8:e7:5c:05:3b:1e:ca:57:ea:55:c8:68:8f:64:7c:68:81:f2:c8:35:7b:95 ------BEGIN CERTIFICATE----- -MIIFujCCA6KgAwIBAgIJALtAHEP1Xk+wMA0GCSqGSIb3DQEBBQUAMEUxCzAJBgNV -BAYTAkNIMRUwEwYDVQQKEwxTd2lzc1NpZ24gQUcxHzAdBgNVBAMTFlN3aXNzU2ln -biBHb2xkIENBIC0gRzIwHhcNMDYxMDI1MDgzMDM1WhcNMzYxMDI1MDgzMDM1WjBF -MQswCQYDVQQGEwJDSDEVMBMGA1UEChMMU3dpc3NTaWduIEFHMR8wHQYDVQQDExZT -d2lzc1NpZ24gR29sZCBDQSAtIEcyMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIIC -CgKCAgEAr+TufoskDhJuqVAtFkQ7kpJcyrhdhJJCEyq8ZVeCQD5XJM1QiyUqt2/8 -76LQwB8CJEoTlo8jE+YoWACjR8cGp4QjK7u9lit/VcyLwVcfDmJlD909Vopz2q5+ -bbqBHH5CjCA12UNNhPqE21Is8w4ndwtrvxEvcnifLtg+5hg3Wipy+dpikJKVyh+c -6bM8K8vzARO/Ws/BtQpgvd21mWRTuKCWs2/iJneRjOBiEAKfNA+k1ZIzUd6+jbqE -emA8atufK+ze3gE/bk3lUIbLtK/tREDFylqM2tIrfKjuvqblCqoOpd8FUrdVxyJd -MmqXl2MT28nbeTZ7hTpKxVKJ+STnnXepgv9VHKVxaSvRAiTysybUa9oEVeXBCsdt -MDeQKuSeFDNeFhdVxVu1yzSJkvGdJo+hB9TGsnhQ2wwMC3wLjEHXuendjIj3o02y -MszYF9rNt85mndT9Xv+9lz4pded+p2JYryU0pUHHPbwNUMoDAw8IWh+Vc3hiv69y -FGkOpeUDDniOJihC8AcLYiAQZzlG+qkDzAQ4embvIIO1jEpWjpEA/I5cgt6IoMPi -aG59je883WX0XaxR7ySArqpWl2/5rX3aYT+YdzylkbYcjCbaZaIJbcHiVOO5ykxM -gI93e2CaHt+28kgeDrpOVG2Y4OGiGqJ3UM/EY5LsRxmd6+ZrzsECAwEAAaOBrDCB -qTAOBgNVHQ8BAf8EBAMCAQYwDwYDVR0TAQH/BAUwAwEB/zAdBgNVHQ4EFgQUWyV7 -lqRlUX64OfPAeGZe6Drn8O4wHwYDVR0jBBgwFoAUWyV7lqRlUX64OfPAeGZe6Drn -8O4wRgYDVR0gBD8wPTA7BglghXQBWQECAQEwLjAsBggrBgEFBQcCARYgaHR0cDov -L3JlcG9zaXRvcnkuc3dpc3NzaWduLmNvbS8wDQYJKoZIhvcNAQEFBQADggIBACe6 -45R88a7A3hfm5djV9VSwg/S7zV4Fe0+fdWavPOhWfvxyeDgD2StiGwC5+OlgzczO -UYrHUDFu4Up+GC9pWbY9ZIEr44OE5iKHjn3g7gKZYbge9LgriBIWhMIxkziWMaa5 -O1M/wySTVltpkuzFwbs4AOPsF6m43Md8AYOfMke6UiI0HTJ6CVanfCU2qT1L2sCC -bwq7EsiHSycR+R4tx5M/nttfJmtS2S6K8RTGRI0Vqbe/vd6mGu6uLftIdxf+u+yv -GPUqUfA5hJeVbG4bwyvEdGB5JbAKJ9/fXtI5z0V9QkvfsywexcZdylU6oJxpmo/a -77KwPJ+HbBIrZXAVUjEaJM9vMSNQH4xPjyPDdEFjHFWoFN0+4FFQz/EbMFYOkrCC -hdiDyyJkvC24JdVUorgG6q2SpCSgwYa1ShNqR88uC1aVVMvOmttqtKay20EIhid3 -92qgQmwLOM7XdVAyksLfKzAiSNDVQTglXaTpXZ/GlHXQRf0wl0OPkKsKx4ZzYEpp -Ld6leNcG2mqeSz53OiATIgHQv2ieY2BrNU0LbbqhPcCT4H8js1WtciVORvnSFu+w -ZMEBnunKoGqYDs/YYPIvSbjkQuE4NRb0yG5P94FW6LqjviOvrv1vA+ACOzB2+htt -Qc8Bsem4yWb02ybzOqR08kkkW8mw0FfB+j564ZfJ ------END CERTIFICATE----- - -# Issuer: CN=SwissSign Silver CA - G2 O=SwissSign AG -# Subject: CN=SwissSign Silver CA - G2 O=SwissSign AG -# Label: "SwissSign Silver CA - G2" -# Serial: 5700383053117599563 -# MD5 Fingerprint: e0:06:a1:c9:7d:cf:c9:fc:0d:c0:56:75:96:d8:62:13 -# SHA1 Fingerprint: 9b:aa:e5:9f:56:ee:21:cb:43:5a:be:25:93:df:a7:f0:40:d1:1d:cb -# SHA256 Fingerprint: be:6c:4d:a2:bb:b9:ba:59:b6:f3:93:97:68:37:42:46:c3:c0:05:99:3f:a9:8f:02:0d:1d:ed:be:d4:8a:81:d5 ------BEGIN CERTIFICATE----- -MIIFvTCCA6WgAwIBAgIITxvUL1S7L0swDQYJKoZIhvcNAQEFBQAwRzELMAkGA1UE -BhMCQ0gxFTATBgNVBAoTDFN3aXNzU2lnbiBBRzEhMB8GA1UEAxMYU3dpc3NTaWdu -IFNpbHZlciBDQSAtIEcyMB4XDTA2MTAyNTA4MzI0NloXDTM2MTAyNTA4MzI0Nlow -RzELMAkGA1UEBhMCQ0gxFTATBgNVBAoTDFN3aXNzU2lnbiBBRzEhMB8GA1UEAxMY -U3dpc3NTaWduIFNpbHZlciBDQSAtIEcyMIICIjANBgkqhkiG9w0BAQEFAAOCAg8A -MIICCgKCAgEAxPGHf9N4Mfc4yfjDmUO8x/e8N+dOcbpLj6VzHVxumK4DV644N0Mv -Fz0fyM5oEMF4rhkDKxD6LHmD9ui5aLlV8gREpzn5/ASLHvGiTSf5YXu6t+WiE7br -YT7QbNHm+/pe7R20nqA1W6GSy/BJkv6FCgU+5tkL4k+73JU3/JHpMjUi0R86TieF -nbAVlDLaYQ1HTWBCrpJH6INaUFjpiou5XaHc3ZlKHzZnu0jkg7Y360g6rw9njxcH -6ATK72oxh9TAtvmUcXtnZLi2kUpCe2UuMGoM9ZDulebyzYLs2aFK7PayS+VFheZt -eJMELpyCbTapxDFkH4aDCyr0NQp4yVXPQbBH6TCfmb5hqAaEuSh6XzjZG6k4sIN/ -c8HDO0gqgg8hm7jMqDXDhBuDsz6+pJVpATqJAHgE2cn0mRmrVn5bi4Y5FZGkECwJ -MoBgs5PAKrYYC51+jUnyEEp/+dVGLxmSo5mnJqy7jDzmDrxHB9xzUfFwZC8I+bRH -HTBsROopN4WSaGa8gzj+ezku01DwH/teYLappvonQfGbGHLy9YR0SslnxFSuSGTf -jNFusB3hB48IHpmccelM2KX3RxIfdNFRnobzwqIjQAtz20um53MGjMGg6cFZrEb6 -5i/4z3GcRm25xBWNOHkDRUjvxF3XCO6HOSKGsg0PWEP3calILv3q1h8CAwEAAaOB -rDCBqTAOBgNVHQ8BAf8EBAMCAQYwDwYDVR0TAQH/BAUwAwEB/zAdBgNVHQ4EFgQU -F6DNweRBtjpbO8tFnb0cwpj6hlgwHwYDVR0jBBgwFoAUF6DNweRBtjpbO8tFnb0c -wpj6hlgwRgYDVR0gBD8wPTA7BglghXQBWQEDAQEwLjAsBggrBgEFBQcCARYgaHR0 -cDovL3JlcG9zaXRvcnkuc3dpc3NzaWduLmNvbS8wDQYJKoZIhvcNAQEFBQADggIB -AHPGgeAn0i0P4JUw4ppBf1AsX19iYamGamkYDHRJ1l2E6kFSGG9YrVBWIGrGvShp -WJHckRE1qTodvBqlYJ7YH39FkWnZfrt4csEGDyrOj4VwYaygzQu4OSlWhDJOhrs9 -xCrZ1x9y7v5RoSJBsXECYxqCsGKrXlcSH9/L3XWgwF15kIwb4FDm3jH+mHtwX6WQ -2K34ArZv02DdQEsixT2tOnqfGhpHkXkzuoLcMmkDlm4fS/Bx/uNncqCxv1yL5PqZ -IseEuRuNI5c/7SXgz2W79WEE790eslpBIlqhn10s6FvJbakMDHiqYMZWjwFaDGi8 -aRl5xB9+lwW/xekkUV7U1UtT7dkjWjYDZaPBA61BMPNGG4WQr2W11bHkFlt4dR2X -em1ZqSqPe97Dh4kQmUlzeMg9vVE1dCrV8X5pGyq7O70luJpaPXJhkGaH7gzWTdQR -dAtq/gsD/KNVV4n+SsuuWxcFyPKNIzFTONItaj+CuY0IavdeQXRuwxF+B6wpYJE/ -OMpXEA29MC/HpeZBoNquBYeaoKRlbEwJDIm6uNO5wJOKMPqN5ZprFQFOZ6raYlY+ -hAhm0sQ2fac+EPyI4NSA5QC9qvNOBqN6avlicuMJT+ubDgEj8Z+7fNzcbBGXJbLy -tGMU0gYqZ4yD9c7qB9iaah7s5Aq7KkzrCWA5zspi2C5u ------END CERTIFICATE----- - -# Issuer: CN=SecureTrust CA O=SecureTrust Corporation -# Subject: CN=SecureTrust CA O=SecureTrust Corporation -# Label: "SecureTrust CA" -# Serial: 17199774589125277788362757014266862032 -# MD5 Fingerprint: dc:32:c3:a7:6d:25:57:c7:68:09:9d:ea:2d:a9:a2:d1 -# SHA1 Fingerprint: 87:82:c6:c3:04:35:3b:cf:d2:96:92:d2:59:3e:7d:44:d9:34:ff:11 -# SHA256 Fingerprint: f1:c1:b5:0a:e5:a2:0d:d8:03:0e:c9:f6:bc:24:82:3d:d3:67:b5:25:57:59:b4:e7:1b:61:fc:e9:f7:37:5d:73 ------BEGIN CERTIFICATE----- -MIIDuDCCAqCgAwIBAgIQDPCOXAgWpa1Cf/DrJxhZ0DANBgkqhkiG9w0BAQUFADBI -MQswCQYDVQQGEwJVUzEgMB4GA1UEChMXU2VjdXJlVHJ1c3QgQ29ycG9yYXRpb24x -FzAVBgNVBAMTDlNlY3VyZVRydXN0IENBMB4XDTA2MTEwNzE5MzExOFoXDTI5MTIz -MTE5NDA1NVowSDELMAkGA1UEBhMCVVMxIDAeBgNVBAoTF1NlY3VyZVRydXN0IENv -cnBvcmF0aW9uMRcwFQYDVQQDEw5TZWN1cmVUcnVzdCBDQTCCASIwDQYJKoZIhvcN -AQEBBQADggEPADCCAQoCggEBAKukgeWVzfX2FI7CT8rU4niVWJxB4Q2ZQCQXOZEz -Zum+4YOvYlyJ0fwkW2Gz4BERQRwdbvC4u/jep4G6pkjGnx29vo6pQT64lO0pGtSO -0gMdA+9tDWccV9cGrcrI9f4Or2YlSASWC12juhbDCE/RRvgUXPLIXgGZbf2IzIao -wW8xQmxSPmjL8xk037uHGFaAJsTQ3MBv396gwpEWoGQRS0S8Hvbn+mPeZqx2pHGj -7DaUaHp3pLHnDi+BeuK1cobvomuL8A/b01k/unK8RCSc43Oz969XL0Imnal0ugBS -8kvNU3xHCzaFDmapCJcWNFfBZveA4+1wVMeT4C4oFVmHursCAwEAAaOBnTCBmjAT -BgkrBgEEAYI3FAIEBh4EAEMAQTALBgNVHQ8EBAMCAYYwDwYDVR0TAQH/BAUwAwEB -/zAdBgNVHQ4EFgQUQjK2FvoE/f5dS3rD/fdMQB1aQ68wNAYDVR0fBC0wKzApoCeg -JYYjaHR0cDovL2NybC5zZWN1cmV0cnVzdC5jb20vU1RDQS5jcmwwEAYJKwYBBAGC -NxUBBAMCAQAwDQYJKoZIhvcNAQEFBQADggEBADDtT0rhWDpSclu1pqNlGKa7UTt3 -6Z3q059c4EVlew3KW+JwULKUBRSuSceNQQcSc5R+DCMh/bwQf2AQWnL1mA6s7Ll/ -3XpvXdMc9P+IBWlCqQVxyLesJugutIxq/3HcuLHfmbx8IVQr5Fiiu1cprp6poxkm -D5kuCLDv/WnPmRoJjeOnnyvJNjR7JLN4TJUXpAYmHrZkUjZfYGfZnMUFdAvnZyPS -CPyI6a6Lf+Ew9Dd+/cYy2i2eRDAwbO4H3tI0/NL/QPZL9GZGBlSm8jIKYyYwa5vR -3ItHuuG51WLQoqD0ZwV4KWMabwTW+MZMo5qxN7SN5ShLHZ4swrhovO0C7jE= ------END CERTIFICATE----- - -# Issuer: CN=Secure Global CA O=SecureTrust Corporation -# Subject: CN=Secure Global CA O=SecureTrust Corporation -# Label: "Secure Global CA" -# Serial: 9751836167731051554232119481456978597 -# MD5 Fingerprint: cf:f4:27:0d:d4:ed:dc:65:16:49:6d:3d:da:bf:6e:de -# SHA1 Fingerprint: 3a:44:73:5a:e5:81:90:1f:24:86:61:46:1e:3b:9c:c4:5f:f5:3a:1b -# SHA256 Fingerprint: 42:00:f5:04:3a:c8:59:0e:bb:52:7d:20:9e:d1:50:30:29:fb:cb:d4:1c:a1:b5:06:ec:27:f1:5a:de:7d:ac:69 ------BEGIN CERTIFICATE----- -MIIDvDCCAqSgAwIBAgIQB1YipOjUiolN9BPI8PjqpTANBgkqhkiG9w0BAQUFADBK -MQswCQYDVQQGEwJVUzEgMB4GA1UEChMXU2VjdXJlVHJ1c3QgQ29ycG9yYXRpb24x -GTAXBgNVBAMTEFNlY3VyZSBHbG9iYWwgQ0EwHhcNMDYxMTA3MTk0MjI4WhcNMjkx -MjMxMTk1MjA2WjBKMQswCQYDVQQGEwJVUzEgMB4GA1UEChMXU2VjdXJlVHJ1c3Qg -Q29ycG9yYXRpb24xGTAXBgNVBAMTEFNlY3VyZSBHbG9iYWwgQ0EwggEiMA0GCSqG -SIb3DQEBAQUAA4IBDwAwggEKAoIBAQCvNS7YrGxVaQZx5RNoJLNP2MwhR/jxYDiJ -iQPpvepeRlMJ3Fz1Wuj3RSoC6zFh1ykzTM7HfAo3fg+6MpjhHZevj8fcyTiW89sa -/FHtaMbQbqR8JNGuQsiWUGMu4P51/pinX0kuleM5M2SOHqRfkNJnPLLZ/kG5VacJ -jnIFHovdRIWCQtBJwB1g8NEXLJXr9qXBkqPFwqcIYA1gBBCWeZ4WNOaptvolRTnI -HmX5k/Wq8VLcmZg9pYYaDDUz+kulBAYVHDGA76oYa8J719rO+TMg1fW9ajMtgQT7 -sFzUnKPiXB3jqUJ1XnvUd+85VLrJChgbEplJL4hL/VBi0XPnj3pDAgMBAAGjgZ0w -gZowEwYJKwYBBAGCNxQCBAYeBABDAEEwCwYDVR0PBAQDAgGGMA8GA1UdEwEB/wQF -MAMBAf8wHQYDVR0OBBYEFK9EBMJBfkiD2045AuzshHrmzsmkMDQGA1UdHwQtMCsw -KaAnoCWGI2h0dHA6Ly9jcmwuc2VjdXJldHJ1c3QuY29tL1NHQ0EuY3JsMBAGCSsG -AQQBgjcVAQQDAgEAMA0GCSqGSIb3DQEBBQUAA4IBAQBjGghAfaReUw132HquHw0L -URYD7xh8yOOvaliTFGCRsoTciE6+OYo68+aCiV0BN7OrJKQVDpI1WkpEXk5X+nXO -H0jOZvQ8QCaSmGwb7iRGDBezUqXbpZGRzzfTb+cnCDpOGR86p1hcF895P4vkp9Mm -I50mD1hp/Ed+stCNi5O/KU9DaXR2Z0vPB4zmAve14bRDtUstFJ/53CYNv6ZHdAbY -iNE6KTCEztI5gGIbqMdXSbxqVVFnFUq+NQfk1XWYN3kwFNspnWzFacxHVaIw98xc -f8LDmBxrThaA63p4ZUWiABqvDA1VZDRIuJK58bRQKfJPIx/abKwfROHdI3hRW8cW ------END CERTIFICATE----- - -# Issuer: CN=COMODO Certification Authority O=COMODO CA Limited -# Subject: CN=COMODO Certification Authority O=COMODO CA Limited -# Label: "COMODO Certification Authority" -# Serial: 104350513648249232941998508985834464573 -# MD5 Fingerprint: 5c:48:dc:f7:42:72:ec:56:94:6d:1c:cc:71:35:80:75 -# SHA1 Fingerprint: 66:31:bf:9e:f7:4f:9e:b6:c9:d5:a6:0c:ba:6a:be:d1:f7:bd:ef:7b -# SHA256 Fingerprint: 0c:2c:d6:3d:f7:80:6f:a3:99:ed:e8:09:11:6b:57:5b:f8:79:89:f0:65:18:f9:80:8c:86:05:03:17:8b:af:66 ------BEGIN CERTIFICATE----- -MIIEHTCCAwWgAwIBAgIQToEtioJl4AsC7j41AkblPTANBgkqhkiG9w0BAQUFADCB -gTELMAkGA1UEBhMCR0IxGzAZBgNVBAgTEkdyZWF0ZXIgTWFuY2hlc3RlcjEQMA4G -A1UEBxMHU2FsZm9yZDEaMBgGA1UEChMRQ09NT0RPIENBIExpbWl0ZWQxJzAlBgNV -BAMTHkNPTU9ETyBDZXJ0aWZpY2F0aW9uIEF1dGhvcml0eTAeFw0wNjEyMDEwMDAw -MDBaFw0yOTEyMzEyMzU5NTlaMIGBMQswCQYDVQQGEwJHQjEbMBkGA1UECBMSR3Jl -YXRlciBNYW5jaGVzdGVyMRAwDgYDVQQHEwdTYWxmb3JkMRowGAYDVQQKExFDT01P -RE8gQ0EgTGltaXRlZDEnMCUGA1UEAxMeQ09NT0RPIENlcnRpZmljYXRpb24gQXV0 -aG9yaXR5MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA0ECLi3LjkRv3 -UcEbVASY06m/weaKXTuH+7uIzg3jLz8GlvCiKVCZrts7oVewdFFxze1CkU1B/qnI -2GqGd0S7WWaXUF601CxwRM/aN5VCaTwwxHGzUvAhTaHYujl8HJ6jJJ3ygxaYqhZ8 -Q5sVW7euNJH+1GImGEaaP+vB+fGQV+useg2L23IwambV4EajcNxo2f8ESIl33rXp -+2dtQem8Ob0y2WIC8bGoPW43nOIv4tOiJovGuFVDiOEjPqXSJDlqR6sA1KGzqSX+ -DT+nHbrTUcELpNqsOO9VUCQFZUaTNE8tja3G1CEZ0o7KBWFxB3NH5YoZEr0ETc5O -nKVIrLsm9wIDAQABo4GOMIGLMB0GA1UdDgQWBBQLWOWLxkwVN6RAqTCpIb5HNlpW -/zAOBgNVHQ8BAf8EBAMCAQYwDwYDVR0TAQH/BAUwAwEB/zBJBgNVHR8EQjBAMD6g -PKA6hjhodHRwOi8vY3JsLmNvbW9kb2NhLmNvbS9DT01PRE9DZXJ0aWZpY2F0aW9u -QXV0aG9yaXR5LmNybDANBgkqhkiG9w0BAQUFAAOCAQEAPpiem/Yb6dc5t3iuHXIY -SdOH5EOC6z/JqvWote9VfCFSZfnVDeFs9D6Mk3ORLgLETgdxb8CPOGEIqB6BCsAv -IC9Bi5HcSEW88cbeunZrM8gALTFGTO3nnc+IlP8zwFboJIYmuNg4ON8qa90SzMc/ -RxdMosIGlgnW2/4/PEZB31jiVg88O8EckzXZOFKs7sjsLjBOlDW0JB9LeGna8gI4 -zJVSk/BwJVmcIGfE7vmLV2H0knZ9P4SNVbfo5azV8fUZVqZa+5Acr5Pr5RzUZ5dd -BA6+C4OmF4O5MBKgxTMVBbkN+8cFduPYSo38NBejxiEovjBFMR7HeL5YYTisO+IB -ZQ== ------END CERTIFICATE----- - -# Issuer: CN=COMODO ECC Certification Authority O=COMODO CA Limited -# Subject: CN=COMODO ECC Certification Authority O=COMODO CA Limited -# Label: "COMODO ECC Certification Authority" -# Serial: 41578283867086692638256921589707938090 -# MD5 Fingerprint: 7c:62:ff:74:9d:31:53:5e:68:4a:d5:78:aa:1e:bf:23 -# SHA1 Fingerprint: 9f:74:4e:9f:2b:4d:ba:ec:0f:31:2c:50:b6:56:3b:8e:2d:93:c3:11 -# SHA256 Fingerprint: 17:93:92:7a:06:14:54:97:89:ad:ce:2f:8f:34:f7:f0:b6:6d:0f:3a:e3:a3:b8:4d:21:ec:15:db:ba:4f:ad:c7 ------BEGIN CERTIFICATE----- -MIICiTCCAg+gAwIBAgIQH0evqmIAcFBUTAGem2OZKjAKBggqhkjOPQQDAzCBhTEL -MAkGA1UEBhMCR0IxGzAZBgNVBAgTEkdyZWF0ZXIgTWFuY2hlc3RlcjEQMA4GA1UE -BxMHU2FsZm9yZDEaMBgGA1UEChMRQ09NT0RPIENBIExpbWl0ZWQxKzApBgNVBAMT -IkNPTU9ETyBFQ0MgQ2VydGlmaWNhdGlvbiBBdXRob3JpdHkwHhcNMDgwMzA2MDAw -MDAwWhcNMzgwMTE4MjM1OTU5WjCBhTELMAkGA1UEBhMCR0IxGzAZBgNVBAgTEkdy -ZWF0ZXIgTWFuY2hlc3RlcjEQMA4GA1UEBxMHU2FsZm9yZDEaMBgGA1UEChMRQ09N -T0RPIENBIExpbWl0ZWQxKzApBgNVBAMTIkNPTU9ETyBFQ0MgQ2VydGlmaWNhdGlv -biBBdXRob3JpdHkwdjAQBgcqhkjOPQIBBgUrgQQAIgNiAAQDR3svdcmCFYX7deSR -FtSrYpn1PlILBs5BAH+X4QokPB0BBO490o0JlwzgdeT6+3eKKvUDYEs2ixYjFq0J -cfRK9ChQtP6IHG4/bC8vCVlbpVsLM5niwz2J+Wos77LTBumjQjBAMB0GA1UdDgQW -BBR1cacZSBm8nZ3qQUfflMRId5nTeTAOBgNVHQ8BAf8EBAMCAQYwDwYDVR0TAQH/ -BAUwAwEB/zAKBggqhkjOPQQDAwNoADBlAjEA7wNbeqy3eApyt4jf/7VGFAkK+qDm -fQjGGoe9GKhzvSbKYAydzpmfz1wPMOG+FDHqAjAU9JM8SaczepBGR7NjfRObTrdv -GDeAU/7dIOA1mjbRxwG55tzd8/8dLDoWV9mSOdY= ------END CERTIFICATE----- - -# Issuer: CN=Certigna O=Dhimyotis -# Subject: CN=Certigna O=Dhimyotis -# Label: "Certigna" -# Serial: 18364802974209362175 -# MD5 Fingerprint: ab:57:a6:5b:7d:42:82:19:b5:d8:58:26:28:5e:fd:ff -# SHA1 Fingerprint: b1:2e:13:63:45:86:a4:6f:1a:b2:60:68:37:58:2d:c4:ac:fd:94:97 -# SHA256 Fingerprint: e3:b6:a2:db:2e:d7:ce:48:84:2f:7a:c5:32:41:c7:b7:1d:54:14:4b:fb:40:c1:1f:3f:1d:0b:42:f5:ee:a1:2d ------BEGIN CERTIFICATE----- -MIIDqDCCApCgAwIBAgIJAP7c4wEPyUj/MA0GCSqGSIb3DQEBBQUAMDQxCzAJBgNV -BAYTAkZSMRIwEAYDVQQKDAlEaGlteW90aXMxETAPBgNVBAMMCENlcnRpZ25hMB4X -DTA3MDYyOTE1MTMwNVoXDTI3MDYyOTE1MTMwNVowNDELMAkGA1UEBhMCRlIxEjAQ -BgNVBAoMCURoaW15b3RpczERMA8GA1UEAwwIQ2VydGlnbmEwggEiMA0GCSqGSIb3 -DQEBAQUAA4IBDwAwggEKAoIBAQDIaPHJ1tazNHUmgh7stL7qXOEm7RFHYeGifBZ4 -QCHkYJ5ayGPhxLGWkv8YbWkj4Sti993iNi+RB7lIzw7sebYs5zRLcAglozyHGxny -gQcPOJAZ0xH+hrTy0V4eHpbNgGzOOzGTtvKg0KmVEn2lmsxryIRWijOp5yIVUxbw -zBfsV1/pogqYCd7jX5xv3EjjhQsVWqa6n6xI4wmy9/Qy3l40vhx4XUJbzg4ij02Q -130yGLMLLGq/jj8UEYkgDncUtT2UCIf3JR7VsmAA7G8qKCVuKj4YYxclPz5EIBb2 -JsglrgVKtOdjLPOMFlN+XPsRGgjBRmKfIrjxwo1p3Po6WAbfAgMBAAGjgbwwgbkw -DwYDVR0TAQH/BAUwAwEB/zAdBgNVHQ4EFgQUGu3+QTmQtCRZvgHyUtVF9lo53BEw -ZAYDVR0jBF0wW4AUGu3+QTmQtCRZvgHyUtVF9lo53BGhOKQ2MDQxCzAJBgNVBAYT -AkZSMRIwEAYDVQQKDAlEaGlteW90aXMxETAPBgNVBAMMCENlcnRpZ25hggkA/tzj -AQ/JSP8wDgYDVR0PAQH/BAQDAgEGMBEGCWCGSAGG+EIBAQQEAwIABzANBgkqhkiG -9w0BAQUFAAOCAQEAhQMeknH2Qq/ho2Ge6/PAD/Kl1NqV5ta+aDY9fm4fTIrv0Q8h -bV6lUmPOEvjvKtpv6zf+EwLHyzs+ImvaYS5/1HI93TDhHkxAGYwP15zRgzB7mFnc -fca5DClMoTOi62c6ZYTTluLtdkVwj7Ur3vkj1kluPBS1xp81HlDQwY9qcEQCYsuu -HWhBp6pX6FOqB9IG9tUUBguRA3UsbHK1YZWaDYu5Def131TN3ubY1gkIl2PlwS6w -t0QmwCbAr1UwnjvVNioZBPRcHv/PLLf/0P2HQBHVESO7SMAhqaQoLf0V+LBOK/Qw -WyH8EZE0vkHve52Xdf+XlcCWWC/qu0bXu+TZLg== ------END CERTIFICATE----- - -# Issuer: O=Chunghwa Telecom Co., Ltd. OU=ePKI Root Certification Authority -# Subject: O=Chunghwa Telecom Co., Ltd. OU=ePKI Root Certification Authority -# Label: "ePKI Root Certification Authority" -# Serial: 28956088682735189655030529057352760477 -# MD5 Fingerprint: 1b:2e:00:ca:26:06:90:3d:ad:fe:6f:15:68:d3:6b:b3 -# SHA1 Fingerprint: 67:65:0d:f1:7e:8e:7e:5b:82:40:a4:f4:56:4b:cf:e2:3d:69:c6:f0 -# SHA256 Fingerprint: c0:a6:f4:dc:63:a2:4b:fd:cf:54:ef:2a:6a:08:2a:0a:72:de:35:80:3e:2f:f5:ff:52:7a:e5:d8:72:06:df:d5 ------BEGIN CERTIFICATE----- -MIIFsDCCA5igAwIBAgIQFci9ZUdcr7iXAF7kBtK8nTANBgkqhkiG9w0BAQUFADBe -MQswCQYDVQQGEwJUVzEjMCEGA1UECgwaQ2h1bmdod2EgVGVsZWNvbSBDby4sIEx0 -ZC4xKjAoBgNVBAsMIWVQS0kgUm9vdCBDZXJ0aWZpY2F0aW9uIEF1dGhvcml0eTAe -Fw0wNDEyMjAwMjMxMjdaFw0zNDEyMjAwMjMxMjdaMF4xCzAJBgNVBAYTAlRXMSMw -IQYDVQQKDBpDaHVuZ2h3YSBUZWxlY29tIENvLiwgTHRkLjEqMCgGA1UECwwhZVBL -SSBSb290IENlcnRpZmljYXRpb24gQXV0aG9yaXR5MIICIjANBgkqhkiG9w0BAQEF -AAOCAg8AMIICCgKCAgEA4SUP7o3biDN1Z82tH306Tm2d0y8U82N0ywEhajfqhFAH -SyZbCUNsIZ5qyNUD9WBpj8zwIuQf5/dqIjG3LBXy4P4AakP/h2XGtRrBp0xtInAh -ijHyl3SJCRImHJ7K2RKilTza6We/CKBk49ZCt0Xvl/T29de1ShUCWH2YWEtgvM3X -DZoTM1PRYfl61dd4s5oz9wCGzh1NlDivqOx4UXCKXBCDUSH3ET00hl7lSM2XgYI1 -TBnsZfZrxQWh7kcT1rMhJ5QQCtkkO7q+RBNGMD+XPNjX12ruOzjjK9SXDrkb5wdJ -fzcq+Xd4z1TtW0ado4AOkUPB1ltfFLqfpo0kR0BZv3I4sjZsN/+Z0V0OWQqraffA -sgRFelQArr5T9rXn4fg8ozHSqf4hUmTFpmfwdQcGlBSBVcYn5AGPF8Fqcde+S/uU -WH1+ETOxQvdibBjWzwloPn9s9h6PYq2lY9sJpx8iQkEeb5mKPtf5P0B6ebClAZLS -nT0IFaUQAS2zMnaolQ2zepr7BxB4EW/hj8e6DyUadCrlHJhBmd8hh+iVBmoKs2pH -dmX2Os+PYhcZewoozRrSgx4hxyy/vv9haLdnG7t4TY3OZ+XkwY63I2binZB1NJip -NiuKmpS5nezMirH4JYlcWrYvjB9teSSnUmjDhDXiZo1jDiVN1Rmy5nk3pyKdVDEC -AwEAAaNqMGgwHQYDVR0OBBYEFB4M97Zn8uGSJglFwFU5Lnc/QkqiMAwGA1UdEwQF -MAMBAf8wOQYEZyoHAAQxMC8wLQIBADAJBgUrDgMCGgUAMAcGBWcqAwAABBRFsMLH -ClZ87lt4DJX5GFPBphzYEDANBgkqhkiG9w0BAQUFAAOCAgEACbODU1kBPpVJufGB -uvl2ICO1J2B01GqZNF5sAFPZn/KmsSQHRGoqxqWOeBLoR9lYGxMqXnmbnwoqZ6Yl -PwZpVnPDimZI+ymBV3QGypzqKOg4ZyYr8dW1P2WT+DZdjo2NQCCHGervJ8A9tDkP -JXtoUHRVnAxZfVo9QZQlUgjgRywVMRnVvwdVxrsStZf0X4OFunHB2WyBEXYKCrC/ -gpf36j36+uwtqSiUO1bd0lEursC9CBWMd1I0ltabrNMdjmEPNXubrjlpC2JgQCA2 -j6/7Nu4tCEoduL+bXPjqpRugc6bY+G7gMwRfaKonh+3ZwZCc7b3jajWvY9+rGNm6 -5ulK6lCKD2GTHuItGeIwlDWSXQ62B68ZgI9HkFFLLk3dheLSClIKF5r8GrBQAuUB -o2M3IUxExJtRmREOc5wGj1QupyheRDmHVi03vYVElOEMSyycw5KFNGHLD7ibSkNS -/jQ6fbjpKdx2qcgw+BRxgMYeNkh0IkFch4LoGHGLQYlE535YW6i4jRPpp2zDR+2z -Gp1iro2C6pSe3VkQw63d4k3jMdXH7OjysP6SHhYKGvzZ8/gntsm+HbRsZJB/9OTE -W9c3rkIO3aQab3yIVMUWbuF6aC74Or8NpDyJO3inTmODBCEIZ43ygknQW/2xzQ+D -hNQ+IIX3Sj0rnP0qCglN6oH4EZw= ------END CERTIFICATE----- - -# Issuer: O=certSIGN OU=certSIGN ROOT CA -# Subject: O=certSIGN OU=certSIGN ROOT CA -# Label: "certSIGN ROOT CA" -# Serial: 35210227249154 -# MD5 Fingerprint: 18:98:c0:d6:e9:3a:fc:f9:b0:f5:0c:f7:4b:01:44:17 -# SHA1 Fingerprint: fa:b7:ee:36:97:26:62:fb:2d:b0:2a:f6:bf:03:fd:e8:7c:4b:2f:9b -# SHA256 Fingerprint: ea:a9:62:c4:fa:4a:6b:af:eb:e4:15:19:6d:35:1c:cd:88:8d:4f:53:f3:fa:8a:e6:d7:c4:66:a9:4e:60:42:bb ------BEGIN CERTIFICATE----- -MIIDODCCAiCgAwIBAgIGIAYFFnACMA0GCSqGSIb3DQEBBQUAMDsxCzAJBgNVBAYT -AlJPMREwDwYDVQQKEwhjZXJ0U0lHTjEZMBcGA1UECxMQY2VydFNJR04gUk9PVCBD -QTAeFw0wNjA3MDQxNzIwMDRaFw0zMTA3MDQxNzIwMDRaMDsxCzAJBgNVBAYTAlJP -MREwDwYDVQQKEwhjZXJ0U0lHTjEZMBcGA1UECxMQY2VydFNJR04gUk9PVCBDQTCC -ASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALczuX7IJUqOtdu0KBuqV5Do -0SLTZLrTk+jUrIZhQGpgV2hUhE28alQCBf/fm5oqrl0Hj0rDKH/v+yv6efHHrfAQ -UySQi2bJqIirr1qjAOm+ukbuW3N7LBeCgV5iLKECZbO9xSsAfsT8AzNXDe3i+s5d -RdY4zTW2ssHQnIFKquSyAVwdj1+ZxLGt24gh65AIgoDzMKND5pCCrlUoSe1b16kQ -OA7+j0xbm0bqQfWwCHTD0IgztnzXdN/chNFDDnU5oSVAKOp4yw4sLjmdjItuFhwv -JoIQ4uNllAoEwF73XVv4EOLQunpL+943AAAaWyjj0pxzPjKHmKHJUS/X3qwzs08C -AwEAAaNCMEAwDwYDVR0TAQH/BAUwAwEB/zAOBgNVHQ8BAf8EBAMCAcYwHQYDVR0O -BBYEFOCMm9slSbPxfIbWskKHC9BroNnkMA0GCSqGSIb3DQEBBQUAA4IBAQA+0hyJ -LjX8+HXd5n9liPRyTMks1zJO890ZeUe9jjtbkw9QSSQTaxQGcu8J06Gh40CEyecY -MnQ8SG4Pn0vU9x7Tk4ZkVJdjclDVVc/6IJMCopvDI5NOFlV2oHB5bc0hH88vLbwZ -44gx+FkagQnIl6Z0x2DEW8xXjrJ1/RsCCdtZb3KTafcxQdaIOL+Hsr0Wefmq5L6I -Jd1hJyMctTEHBDa0GpC9oHRxUIltvBTjD4au8as+x6AJzKNI0eDbZOeStc+vckNw -i/nDhDwTqn6Sm1dTk/pwwpEOMfmbZ13pljheX7NzTogVZ96edhBiIL5VaZVDADlN -9u6wWk5JRFRYX0KD ------END CERTIFICATE----- - -# Issuer: CN=NetLock Arany (Class Gold) F\u0151tan\xfas\xedtv\xe1ny O=NetLock Kft. OU=Tan\xfas\xedtv\xe1nykiad\xf3k (Certification Services) -# Subject: CN=NetLock Arany (Class Gold) F\u0151tan\xfas\xedtv\xe1ny O=NetLock Kft. OU=Tan\xfas\xedtv\xe1nykiad\xf3k (Certification Services) -# Label: "NetLock Arany (Class Gold) F\u0151tan\xfas\xedtv\xe1ny" -# Serial: 80544274841616 -# MD5 Fingerprint: c5:a1:b7:ff:73:dd:d6:d7:34:32:18:df:fc:3c:ad:88 -# SHA1 Fingerprint: 06:08:3f:59:3f:15:a1:04:a0:69:a4:6b:a9:03:d0:06:b7:97:09:91 -# SHA256 Fingerprint: 6c:61:da:c3:a2:de:f0:31:50:6b:e0:36:d2:a6:fe:40:19:94:fb:d1:3d:f9:c8:d4:66:59:92:74:c4:46:ec:98 ------BEGIN CERTIFICATE----- -MIIEFTCCAv2gAwIBAgIGSUEs5AAQMA0GCSqGSIb3DQEBCwUAMIGnMQswCQYDVQQG -EwJIVTERMA8GA1UEBwwIQnVkYXBlc3QxFTATBgNVBAoMDE5ldExvY2sgS2Z0LjE3 -MDUGA1UECwwuVGFuw7pzw610dsOhbnlraWFkw7NrIChDZXJ0aWZpY2F0aW9uIFNl -cnZpY2VzKTE1MDMGA1UEAwwsTmV0TG9jayBBcmFueSAoQ2xhc3MgR29sZCkgRsWR -dGFuw7pzw610dsOhbnkwHhcNMDgxMjExMTUwODIxWhcNMjgxMjA2MTUwODIxWjCB -pzELMAkGA1UEBhMCSFUxETAPBgNVBAcMCEJ1ZGFwZXN0MRUwEwYDVQQKDAxOZXRM -b2NrIEtmdC4xNzA1BgNVBAsMLlRhbsO6c8OtdHbDoW55a2lhZMOzayAoQ2VydGlm -aWNhdGlvbiBTZXJ2aWNlcykxNTAzBgNVBAMMLE5ldExvY2sgQXJhbnkgKENsYXNz -IEdvbGQpIEbFkXRhbsO6c8OtdHbDoW55MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8A -MIIBCgKCAQEAxCRec75LbRTDofTjl5Bu0jBFHjzuZ9lk4BqKf8owyoPjIMHj9DrT -lF8afFttvzBPhCf2nx9JvMaZCpDyD/V/Q4Q3Y1GLeqVw/HpYzY6b7cNGbIRwXdrz -AZAj/E4wqX7hJ2Pn7WQ8oLjJM2P+FpD/sLj916jAwJRDC7bVWaaeVtAkH3B5r9s5 -VA1lddkVQZQBr17s9o3x/61k/iCa11zr/qYfCGSji3ZVrR47KGAuhyXoqq8fxmRG -ILdwfzzeSNuWU7c5d+Qa4scWhHaXWy+7GRWF+GmF9ZmnqfI0p6m2pgP8b4Y9VHx2 -BJtr+UBdADTHLpl1neWIA6pN+APSQnbAGwIDAKiLo0UwQzASBgNVHRMBAf8ECDAG -AQH/AgEEMA4GA1UdDwEB/wQEAwIBBjAdBgNVHQ4EFgQUzPpnk/C2uNClwB7zU/2M -U9+D15YwDQYJKoZIhvcNAQELBQADggEBAKt/7hwWqZw8UQCgwBEIBaeZ5m8BiFRh -bvG5GK1Krf6BQCOUL/t1fC8oS2IkgYIL9WHxHG64YTjrgfpioTtaYtOUZcTh5m2C -+C8lcLIhJsFyUR+MLMOEkMNaj7rP9KdlpeuY0fsFskZ1FSNqb4VjMIDw1Z4fKRzC -bLBQWV2QWzuoDTDPv31/zvGdg73JRm4gpvlhUbohL3u+pRVjodSVh/GeufOJ8z2F -uLjbvrW5KfnaNwUASZQDhETnv0Mxz3WLJdH0pmT1kvarBes96aULNmLazAZfNou2 -XjG4Kvte9nHfRCaexOYNkbQudZWAUWpLMKawYqGT8ZvYzsRjdT9ZR7E= ------END CERTIFICATE----- - -# Issuer: CN=SecureSign RootCA11 O=Japan Certification Services, Inc. -# Subject: CN=SecureSign RootCA11 O=Japan Certification Services, Inc. -# Label: "SecureSign RootCA11" -# Serial: 1 -# MD5 Fingerprint: b7:52:74:e2:92:b4:80:93:f2:75:e4:cc:d7:f2:ea:26 -# SHA1 Fingerprint: 3b:c4:9f:48:f8:f3:73:a0:9c:1e:bd:f8:5b:b1:c3:65:c7:d8:11:b3 -# SHA256 Fingerprint: bf:0f:ee:fb:9e:3a:58:1a:d5:f9:e9:db:75:89:98:57:43:d2:61:08:5c:4d:31:4f:6f:5d:72:59:aa:42:16:12 ------BEGIN CERTIFICATE----- -MIIDbTCCAlWgAwIBAgIBATANBgkqhkiG9w0BAQUFADBYMQswCQYDVQQGEwJKUDEr -MCkGA1UEChMiSmFwYW4gQ2VydGlmaWNhdGlvbiBTZXJ2aWNlcywgSW5jLjEcMBoG -A1UEAxMTU2VjdXJlU2lnbiBSb290Q0ExMTAeFw0wOTA0MDgwNDU2NDdaFw0yOTA0 -MDgwNDU2NDdaMFgxCzAJBgNVBAYTAkpQMSswKQYDVQQKEyJKYXBhbiBDZXJ0aWZp -Y2F0aW9uIFNlcnZpY2VzLCBJbmMuMRwwGgYDVQQDExNTZWN1cmVTaWduIFJvb3RD -QTExMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA/XeqpRyQBTvLTJsz -i1oURaTnkBbR31fSIRCkF/3frNYfp+TbfPfs37gD2pRY/V1yfIw/XwFndBWW4wI8 -h9uuywGOwvNmxoVF9ALGOrVisq/6nL+k5tSAMJjzDbaTj6nU2DbysPyKyiyhFTOV -MdrAG/LuYpmGYz+/3ZMqg6h2uRMft85OQoWPIucuGvKVCbIFtUROd6EgvanyTgp9 -UK31BQ1FT0Zx/Sg+U/sE2C3XZR1KG/rPO7AxmjVuyIsG0wCR8pQIZUyxNAYAeoni -8McDWc/V1uinMrPmmECGxc0nEovMe863ETxiYAcjPitAbpSACW22s293bzUIUPsC -h8U+iQIDAQABo0IwQDAdBgNVHQ4EFgQUW/hNT7KlhtQ60vFjmqC+CfZXt94wDgYD -VR0PAQH/BAQDAgEGMA8GA1UdEwEB/wQFMAMBAf8wDQYJKoZIhvcNAQEFBQADggEB -AKChOBZmLqdWHyGcBvod7bkixTgm2E5P7KN/ed5GIaGHd48HCJqypMWvDzKYC3xm -KbabfSVSSUOrTC4rbnpwrxYO4wJs+0LmGJ1F2FXI6Dvd5+H0LgscNFxsWEr7jIhQ -X5Ucv+2rIrVls4W6ng+4reV6G4pQOh29Dbx7VFALuUKvVaAYga1lme++5Jy/xIWr -QbJUb9wlze144o4MjQlJ3WN7WmmWAiGovVJZ6X01y8hSyn+B/tlr0/cR7SXf+Of5 -pPpyl4RTDaXQMhhRdlkUbA/r7F+AjHVDg8OFmP9Mni0N5HeDk061lgeLKBObjBmN -QSdJQO7e5iNEOdyhIta6A/I= ------END CERTIFICATE----- - -# Issuer: CN=Microsec e-Szigno Root CA 2009 O=Microsec Ltd. -# Subject: CN=Microsec e-Szigno Root CA 2009 O=Microsec Ltd. -# Label: "Microsec e-Szigno Root CA 2009" -# Serial: 14014712776195784473 -# MD5 Fingerprint: f8:49:f4:03:bc:44:2d:83:be:48:69:7d:29:64:fc:b1 -# SHA1 Fingerprint: 89:df:74:fe:5c:f4:0f:4a:80:f9:e3:37:7d:54:da:91:e1:01:31:8e -# SHA256 Fingerprint: 3c:5f:81:fe:a5:fa:b8:2c:64:bf:a2:ea:ec:af:cd:e8:e0:77:fc:86:20:a7:ca:e5:37:16:3d:f3:6e:db:f3:78 ------BEGIN CERTIFICATE----- -MIIECjCCAvKgAwIBAgIJAMJ+QwRORz8ZMA0GCSqGSIb3DQEBCwUAMIGCMQswCQYD -VQQGEwJIVTERMA8GA1UEBwwIQnVkYXBlc3QxFjAUBgNVBAoMDU1pY3Jvc2VjIEx0 -ZC4xJzAlBgNVBAMMHk1pY3Jvc2VjIGUtU3ppZ25vIFJvb3QgQ0EgMjAwOTEfMB0G -CSqGSIb3DQEJARYQaW5mb0BlLXN6aWduby5odTAeFw0wOTA2MTYxMTMwMThaFw0y -OTEyMzAxMTMwMThaMIGCMQswCQYDVQQGEwJIVTERMA8GA1UEBwwIQnVkYXBlc3Qx -FjAUBgNVBAoMDU1pY3Jvc2VjIEx0ZC4xJzAlBgNVBAMMHk1pY3Jvc2VjIGUtU3pp -Z25vIFJvb3QgQ0EgMjAwOTEfMB0GCSqGSIb3DQEJARYQaW5mb0BlLXN6aWduby5o -dTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAOn4j/NjrdqG2KfgQvvP -kd6mJviZpWNwrZuuyjNAfW2WbqEORO7hE52UQlKavXWFdCyoDh2Tthi3jCyoz/tc -cbna7P7ofo/kLx2yqHWH2Leh5TvPmUpG0IMZfcChEhyVbUr02MelTTMuhTlAdX4U -fIASmFDHQWe4oIBhVKZsTh/gnQ4H6cm6M+f+wFUoLAKApxn1ntxVUwOXewdI/5n7 -N4okxFnMUBBjjqqpGrCEGob5X7uxUG6k0QrM1XF+H6cbfPVTbiJfyyvm1HxdrtbC -xkzlBQHZ7Vf8wSN5/PrIJIOV87VqUQHQd9bpEqH5GoP7ghu5sJf0dgYzQ0mg/wu1 -+rUCAwEAAaOBgDB+MA8GA1UdEwEB/wQFMAMBAf8wDgYDVR0PAQH/BAQDAgEGMB0G -A1UdDgQWBBTLD8bfQkPMPcu1SCOhGnqmKrs0aDAfBgNVHSMEGDAWgBTLD8bfQkPM -Pcu1SCOhGnqmKrs0aDAbBgNVHREEFDASgRBpbmZvQGUtc3ppZ25vLmh1MA0GCSqG -SIb3DQEBCwUAA4IBAQDJ0Q5eLtXMs3w+y/w9/w0olZMEyL/azXm4Q5DwpL7v8u8h -mLzU1F0G9u5C7DBsoKqpyvGvivo/C3NqPuouQH4frlRheesuCDfXI/OMn74dseGk -ddug4lQUsbocKaQY9hK6ohQU4zE1yED/t+AFdlfBHFny+L/k7SViXITwfn4fs775 -tyERzAMBVnCnEJIeGzSBHq2cGsMEPO0CYdYeBvNfOofyK/FFh+U9rNHHV4S9a67c -2Pm2G2JwCz02yULyMtd6YebS2z3PyKnJm9zbWETXbzivf3jTo60adbocwTZ8jx5t -HMN1Rq41Bab2XD0h7lbwyYIiLXpUq3DDfSJlgnCW ------END CERTIFICATE----- - -# Issuer: CN=GlobalSign O=GlobalSign OU=GlobalSign Root CA - R3 -# Subject: CN=GlobalSign O=GlobalSign OU=GlobalSign Root CA - R3 -# Label: "GlobalSign Root CA - R3" -# Serial: 4835703278459759426209954 -# MD5 Fingerprint: c5:df:b8:49:ca:05:13:55:ee:2d:ba:1a:c3:3e:b0:28 -# SHA1 Fingerprint: d6:9b:56:11:48:f0:1c:77:c5:45:78:c1:09:26:df:5b:85:69:76:ad -# SHA256 Fingerprint: cb:b5:22:d7:b7:f1:27:ad:6a:01:13:86:5b:df:1c:d4:10:2e:7d:07:59:af:63:5a:7c:f4:72:0d:c9:63:c5:3b ------BEGIN CERTIFICATE----- -MIIDXzCCAkegAwIBAgILBAAAAAABIVhTCKIwDQYJKoZIhvcNAQELBQAwTDEgMB4G -A1UECxMXR2xvYmFsU2lnbiBSb290IENBIC0gUjMxEzARBgNVBAoTCkdsb2JhbFNp -Z24xEzARBgNVBAMTCkdsb2JhbFNpZ24wHhcNMDkwMzE4MTAwMDAwWhcNMjkwMzE4 -MTAwMDAwWjBMMSAwHgYDVQQLExdHbG9iYWxTaWduIFJvb3QgQ0EgLSBSMzETMBEG -A1UEChMKR2xvYmFsU2lnbjETMBEGA1UEAxMKR2xvYmFsU2lnbjCCASIwDQYJKoZI -hvcNAQEBBQADggEPADCCAQoCggEBAMwldpB5BngiFvXAg7aEyiie/QV2EcWtiHL8 -RgJDx7KKnQRfJMsuS+FggkbhUqsMgUdwbN1k0ev1LKMPgj0MK66X17YUhhB5uzsT -gHeMCOFJ0mpiLx9e+pZo34knlTifBtc+ycsmWQ1z3rDI6SYOgxXG71uL0gRgykmm -KPZpO/bLyCiR5Z2KYVc3rHQU3HTgOu5yLy6c+9C7v/U9AOEGM+iCK65TpjoWc4zd -QQ4gOsC0p6Hpsk+QLjJg6VfLuQSSaGjlOCZgdbKfd/+RFO+uIEn8rUAVSNECMWEZ -XriX7613t2Saer9fwRPvm2L7DWzgVGkWqQPabumDk3F2xmmFghcCAwEAAaNCMEAw -DgYDVR0PAQH/BAQDAgEGMA8GA1UdEwEB/wQFMAMBAf8wHQYDVR0OBBYEFI/wS3+o -LkUkrk1Q+mOai97i3Ru8MA0GCSqGSIb3DQEBCwUAA4IBAQBLQNvAUKr+yAzv95ZU -RUm7lgAJQayzE4aGKAczymvmdLm6AC2upArT9fHxD4q/c2dKg8dEe3jgr25sbwMp -jjM5RcOO5LlXbKr8EpbsU8Yt5CRsuZRj+9xTaGdWPoO4zzUhw8lo/s7awlOqzJCK -6fBdRoyV3XpYKBovHd7NADdBj+1EbddTKJd+82cEHhXXipa0095MJ6RMG3NzdvQX -mcIfeg7jLQitChws/zyrVQ4PkX4268NXSb7hLi18YIvDQVETI53O9zJrlAGomecs -Mx86OyXShkDOOyyGeMlhLxS67ttVb9+E7gUJTb0o2HLO02JQZR7rkpeDMdmztcpH -WD9f ------END CERTIFICATE----- - -# Issuer: CN=Autoridad de Certificacion Firmaprofesional CIF A62634068 -# Subject: CN=Autoridad de Certificacion Firmaprofesional CIF A62634068 -# Label: "Autoridad de Certificacion Firmaprofesional CIF A62634068" -# Serial: 6047274297262753887 -# MD5 Fingerprint: 73:3a:74:7a:ec:bb:a3:96:a6:c2:e4:e2:c8:9b:c0:c3 -# SHA1 Fingerprint: ae:c5:fb:3f:c8:e1:bf:c4:e5:4f:03:07:5a:9a:e8:00:b7:f7:b6:fa -# SHA256 Fingerprint: 04:04:80:28:bf:1f:28:64:d4:8f:9a:d4:d8:32:94:36:6a:82:88:56:55:3f:3b:14:30:3f:90:14:7f:5d:40:ef ------BEGIN CERTIFICATE----- -MIIGFDCCA/ygAwIBAgIIU+w77vuySF8wDQYJKoZIhvcNAQEFBQAwUTELMAkGA1UE -BhMCRVMxQjBABgNVBAMMOUF1dG9yaWRhZCBkZSBDZXJ0aWZpY2FjaW9uIEZpcm1h -cHJvZmVzaW9uYWwgQ0lGIEE2MjYzNDA2ODAeFw0wOTA1MjAwODM4MTVaFw0zMDEy -MzEwODM4MTVaMFExCzAJBgNVBAYTAkVTMUIwQAYDVQQDDDlBdXRvcmlkYWQgZGUg -Q2VydGlmaWNhY2lvbiBGaXJtYXByb2Zlc2lvbmFsIENJRiBBNjI2MzQwNjgwggIi -MA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQDKlmuO6vj78aI14H9M2uDDUtd9 -thDIAl6zQyrET2qyyhxdKJp4ERppWVevtSBC5IsP5t9bpgOSL/UR5GLXMnE42QQM -cas9UX4PB99jBVzpv5RvwSmCwLTaUbDBPLutN0pcyvFLNg4kq7/DhHf9qFD0sefG -L9ItWY16Ck6WaVICqjaY7Pz6FIMMNx/Jkjd/14Et5cS54D40/mf0PmbR0/RAz15i -NA9wBj4gGFrO93IbJWyTdBSTo3OxDqqHECNZXyAFGUftaI6SEspd/NYrspI8IM/h -X68gvqB2f3bl7BqGYTM+53u0P6APjqK5am+5hyZvQWyIplD9amML9ZMWGxmPsu2b -m8mQ9QEM3xk9Dz44I8kvjwzRAv4bVdZO0I08r0+k8/6vKtMFnXkIoctXMbScyJCy -Z/QYFpM6/EfY0XiWMR+6KwxfXZmtY4laJCB22N/9q06mIqqdXuYnin1oKaPnirja -EbsXLZmdEyRG98Xi2J+Of8ePdG1asuhy9azuJBCtLxTa/y2aRnFHvkLfuwHb9H/T -KI8xWVvTyQKmtFLKbpf7Q8UIJm+K9Lv9nyiqDdVF8xM6HdjAeI9BZzwelGSuewvF -6NkBiDkal4ZkQdU7hwxu+g/GvUgUvzlN1J5Bto+WHWOWk9mVBngxaJ43BjuAiUVh -OSPHG0SjFeUc+JIwuwIDAQABo4HvMIHsMBIGA1UdEwEB/wQIMAYBAf8CAQEwDgYD -VR0PAQH/BAQDAgEGMB0GA1UdDgQWBBRlzeurNR4APn7VdMActHNHDhpkLzCBpgYD -VR0gBIGeMIGbMIGYBgRVHSAAMIGPMC8GCCsGAQUFBwIBFiNodHRwOi8vd3d3LmZp -cm1hcHJvZmVzaW9uYWwuY29tL2NwczBcBggrBgEFBQcCAjBQHk4AUABhAHMAZQBv -ACAAZABlACAAbABhACAAQgBvAG4AYQBuAG8AdgBhACAANAA3ACAAQgBhAHIAYwBl -AGwAbwBuAGEAIAAwADgAMAAxADcwDQYJKoZIhvcNAQEFBQADggIBABd9oPm03cXF -661LJLWhAqvdpYhKsg9VSytXjDvlMd3+xDLx51tkljYyGOylMnfX40S2wBEqgLk9 -am58m9Ot/MPWo+ZkKXzR4Tgegiv/J2Wv+xYVxC5xhOW1//qkR71kMrv2JYSiJ0L1 -ILDCExARzRAVukKQKtJE4ZYm6zFIEv0q2skGz3QeqUvVhyj5eTSSPi5E6PaPT481 -PyWzOdxjKpBrIF/EUhJOlywqrJ2X3kjyo2bbwtKDlaZmp54lD+kLM5FlClrD2VQS -3a/DTg4fJl4N3LON7NWBcN7STyQF82xO9UxJZo3R/9ILJUFI/lGExkKvgATP0H5k -SeTy36LssUzAKh3ntLFlosS88Zj0qnAHY7S42jtM+kAiMFsRpvAFDsYCA0irhpuF -3dvd6qJ2gHN99ZwExEWN57kci57q13XRcrHedUTnQn3iV2t93Jm8PYMo6oCTjcVM -ZcFwgbg4/EMxsvYDNEeyrPsiBsse3RdHHF9mudMaotoRsaS8I8nkvof/uZS2+F0g -StRf571oe2XyFR7SOqkt6dhrJKyXWERHrVkY8SFlcN7ONGCoQPHzPKTDKCOM/icz -Q0CgFzzr6juwcqajuUpLXhZI9LK8yIySxZ2frHI2vDSANGupi5LAuBft7HZT9SQB -jLMi6Et8Vcad+qMUu2WFbm5PEn4KPJ2V ------END CERTIFICATE----- - -# Issuer: CN=Izenpe.com O=IZENPE S.A. -# Subject: CN=Izenpe.com O=IZENPE S.A. -# Label: "Izenpe.com" -# Serial: 917563065490389241595536686991402621 -# MD5 Fingerprint: a6:b0:cd:85:80:da:5c:50:34:a3:39:90:2f:55:67:73 -# SHA1 Fingerprint: 2f:78:3d:25:52:18:a7:4a:65:39:71:b5:2c:a2:9c:45:15:6f:e9:19 -# SHA256 Fingerprint: 25:30:cc:8e:98:32:15:02:ba:d9:6f:9b:1f:ba:1b:09:9e:2d:29:9e:0f:45:48:bb:91:4f:36:3b:c0:d4:53:1f ------BEGIN CERTIFICATE----- -MIIF8TCCA9mgAwIBAgIQALC3WhZIX7/hy/WL1xnmfTANBgkqhkiG9w0BAQsFADA4 -MQswCQYDVQQGEwJFUzEUMBIGA1UECgwLSVpFTlBFIFMuQS4xEzARBgNVBAMMCkl6 -ZW5wZS5jb20wHhcNMDcxMjEzMTMwODI4WhcNMzcxMjEzMDgyNzI1WjA4MQswCQYD -VQQGEwJFUzEUMBIGA1UECgwLSVpFTlBFIFMuQS4xEzARBgNVBAMMCkl6ZW5wZS5j -b20wggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQDJ03rKDx6sp4boFmVq -scIbRTJxldn+EFvMr+eleQGPicPK8lVx93e+d5TzcqQsRNiekpsUOqHnJJAKClaO -xdgmlOHZSOEtPtoKct2jmRXagaKH9HtuJneJWK3W6wyyQXpzbm3benhB6QiIEn6H -LmYRY2xU+zydcsC8Lv/Ct90NduM61/e0aL6i9eOBbsFGb12N4E3GVFWJGjMxCrFX -uaOKmMPsOzTFlUFpfnXCPCDFYbpRR6AgkJOhkEvzTnyFRVSa0QUmQbC1TR0zvsQD -yCV8wXDbO/QJLVQnSKwv4cSsPsjLkkxTOTcj7NMB+eAJRE1NZMDhDVqHIrytG6P+ -JrUV86f8hBnp7KGItERphIPzidF0BqnMC9bC3ieFUCbKF7jJeodWLBoBHmy+E60Q -rLUk9TiRodZL2vG70t5HtfG8gfZZa88ZU+mNFctKy6lvROUbQc/hhqfK0GqfvEyN -BjNaooXlkDWgYlwWTvDjovoDGrQscbNYLN57C9saD+veIR8GdwYDsMnvmfzAuU8L -hij+0rnq49qlw0dpEuDb8PYZi+17cNcC1u2HGCgsBCRMd+RIihrGO5rUD8r6ddIB -QFqNeb+Lz0vPqhbBleStTIo+F5HUsWLlguWABKQDfo2/2n+iD5dPDNMN+9fR5XJ+ -HMh3/1uaD7euBUbl8agW7EekFwIDAQABo4H2MIHzMIGwBgNVHREEgagwgaWBD2lu -Zm9AaXplbnBlLmNvbaSBkTCBjjFHMEUGA1UECgw+SVpFTlBFIFMuQS4gLSBDSUYg -QTAxMzM3MjYwLVJNZXJjLlZpdG9yaWEtR2FzdGVpeiBUMTA1NSBGNjIgUzgxQzBB -BgNVBAkMOkF2ZGEgZGVsIE1lZGl0ZXJyYW5lbyBFdG9yYmlkZWEgMTQgLSAwMTAx -MCBWaXRvcmlhLUdhc3RlaXowDwYDVR0TAQH/BAUwAwEB/zAOBgNVHQ8BAf8EBAMC -AQYwHQYDVR0OBBYEFB0cZQ6o8iV7tJHP5LGx5r1VdGwFMA0GCSqGSIb3DQEBCwUA -A4ICAQB4pgwWSp9MiDrAyw6lFn2fuUhfGI8NYjb2zRlrrKvV9pF9rnHzP7MOeIWb -laQnIUdCSnxIOvVFfLMMjlF4rJUT3sb9fbgakEyrkgPH7UIBzg/YsfqikuFgba56 -awmqxinuaElnMIAkejEWOVt+8Rwu3WwJrfIxwYJOubv5vr8qhT/AQKM6WfxZSzwo -JNu0FXWuDYi6LnPAvViH5ULy617uHjAimcs30cQhbIHsvm0m5hzkQiCeR7Csg1lw -LDXWrzY0tM07+DKo7+N4ifuNRSzanLh+QBxh5z6ikixL8s36mLYp//Pye6kfLqCT -VyvehQP5aTfLnnhqBbTFMXiJ7HqnheG5ezzevh55hM6fcA5ZwjUukCox2eRFekGk -LhObNA5me0mrZJfQRsN5nXJQY6aYWwa9SG3YOYNw6DXwBdGqvOPbyALqfP2C2sJb -UjWumDqtujWTI6cfSN01RpiyEGjkpTHCClguGYEQyVB1/OpaFs4R1+7vUIgtYf8/ -QnMFlEPVjjxOAToZpR9GTnfQXeWBIiGH/pR9hNiTrdZoQ0iy2+tzJOeRf1SktoA+ -naM8THLCV8Sg1Mw4J87VBp6iSNnpn86CcDaTmjvfliHjWbcM2pE38P1ZWrOZyGls -QyYBNWNgVYkDOnXYukrZVP/u3oDYLdE41V4tC5h9Pmzb/CaIxw== ------END CERTIFICATE----- - -# Issuer: CN=Go Daddy Root Certificate Authority - G2 O=GoDaddy.com, Inc. -# Subject: CN=Go Daddy Root Certificate Authority - G2 O=GoDaddy.com, Inc. -# Label: "Go Daddy Root Certificate Authority - G2" -# Serial: 0 -# MD5 Fingerprint: 80:3a:bc:22:c1:e6:fb:8d:9b:3b:27:4a:32:1b:9a:01 -# SHA1 Fingerprint: 47:be:ab:c9:22:ea:e8:0e:78:78:34:62:a7:9f:45:c2:54:fd:e6:8b -# SHA256 Fingerprint: 45:14:0b:32:47:eb:9c:c8:c5:b4:f0:d7:b5:30:91:f7:32:92:08:9e:6e:5a:63:e2:74:9d:d3:ac:a9:19:8e:da ------BEGIN CERTIFICATE----- -MIIDxTCCAq2gAwIBAgIBADANBgkqhkiG9w0BAQsFADCBgzELMAkGA1UEBhMCVVMx -EDAOBgNVBAgTB0FyaXpvbmExEzARBgNVBAcTClNjb3R0c2RhbGUxGjAYBgNVBAoT -EUdvRGFkZHkuY29tLCBJbmMuMTEwLwYDVQQDEyhHbyBEYWRkeSBSb290IENlcnRp -ZmljYXRlIEF1dGhvcml0eSAtIEcyMB4XDTA5MDkwMTAwMDAwMFoXDTM3MTIzMTIz -NTk1OVowgYMxCzAJBgNVBAYTAlVTMRAwDgYDVQQIEwdBcml6b25hMRMwEQYDVQQH -EwpTY290dHNkYWxlMRowGAYDVQQKExFHb0RhZGR5LmNvbSwgSW5jLjExMC8GA1UE -AxMoR28gRGFkZHkgUm9vdCBDZXJ0aWZpY2F0ZSBBdXRob3JpdHkgLSBHMjCCASIw -DQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAL9xYgjx+lk09xvJGKP3gElY6SKD -E6bFIEMBO4Tx5oVJnyfq9oQbTqC023CYxzIBsQU+B07u9PpPL1kwIuerGVZr4oAH -/PMWdYA5UXvl+TW2dE6pjYIT5LY/qQOD+qK+ihVqf94Lw7YZFAXK6sOoBJQ7Rnwy -DfMAZiLIjWltNowRGLfTshxgtDj6AozO091GB94KPutdfMh8+7ArU6SSYmlRJQVh -GkSBjCypQ5Yj36w6gZoOKcUcqeldHraenjAKOc7xiID7S13MMuyFYkMlNAJWJwGR -tDtwKj9useiciAF9n9T521NtYJ2/LOdYq7hfRvzOxBsDPAnrSTFcaUaz4EcCAwEA -AaNCMEAwDwYDVR0TAQH/BAUwAwEB/zAOBgNVHQ8BAf8EBAMCAQYwHQYDVR0OBBYE -FDqahQcQZyi27/a9BUFuIMGU2g/eMA0GCSqGSIb3DQEBCwUAA4IBAQCZ21151fmX -WWcDYfF+OwYxdS2hII5PZYe096acvNjpL9DbWu7PdIxztDhC2gV7+AJ1uP2lsdeu -9tfeE8tTEH6KRtGX+rcuKxGrkLAngPnon1rpN5+r5N9ss4UXnT3ZJE95kTXWXwTr -gIOrmgIttRD02JDHBHNA7XIloKmf7J6raBKZV8aPEjoJpL1E/QYVN8Gb5DKj7Tjo -2GTzLH4U/ALqn83/B2gX2yKQOC16jdFU8WnjXzPKej17CuPKf1855eJ1usV2GDPO -LPAvTK33sefOT6jEm0pUBsV/fdUID+Ic/n4XuKxe9tQWskMJDE32p2u0mYRlynqI -4uJEvlz36hz1 ------END CERTIFICATE----- - -# Issuer: CN=Starfield Root Certificate Authority - G2 O=Starfield Technologies, Inc. -# Subject: CN=Starfield Root Certificate Authority - G2 O=Starfield Technologies, Inc. -# Label: "Starfield Root Certificate Authority - G2" -# Serial: 0 -# MD5 Fingerprint: d6:39:81:c6:52:7e:96:69:fc:fc:ca:66:ed:05:f2:96 -# SHA1 Fingerprint: b5:1c:06:7c:ee:2b:0c:3d:f8:55:ab:2d:92:f4:fe:39:d4:e7:0f:0e -# SHA256 Fingerprint: 2c:e1:cb:0b:f9:d2:f9:e1:02:99:3f:be:21:51:52:c3:b2:dd:0c:ab:de:1c:68:e5:31:9b:83:91:54:db:b7:f5 ------BEGIN CERTIFICATE----- -MIID3TCCAsWgAwIBAgIBADANBgkqhkiG9w0BAQsFADCBjzELMAkGA1UEBhMCVVMx -EDAOBgNVBAgTB0FyaXpvbmExEzARBgNVBAcTClNjb3R0c2RhbGUxJTAjBgNVBAoT -HFN0YXJmaWVsZCBUZWNobm9sb2dpZXMsIEluYy4xMjAwBgNVBAMTKVN0YXJmaWVs -ZCBSb290IENlcnRpZmljYXRlIEF1dGhvcml0eSAtIEcyMB4XDTA5MDkwMTAwMDAw -MFoXDTM3MTIzMTIzNTk1OVowgY8xCzAJBgNVBAYTAlVTMRAwDgYDVQQIEwdBcml6 -b25hMRMwEQYDVQQHEwpTY290dHNkYWxlMSUwIwYDVQQKExxTdGFyZmllbGQgVGVj -aG5vbG9naWVzLCBJbmMuMTIwMAYDVQQDEylTdGFyZmllbGQgUm9vdCBDZXJ0aWZp -Y2F0ZSBBdXRob3JpdHkgLSBHMjCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoC -ggEBAL3twQP89o/8ArFvW59I2Z154qK3A2FWGMNHttfKPTUuiUP3oWmb3ooa/RMg -nLRJdzIpVv257IzdIvpy3Cdhl+72WoTsbhm5iSzchFvVdPtrX8WJpRBSiUZV9Lh1 -HOZ/5FSuS/hVclcCGfgXcVnrHigHdMWdSL5stPSksPNkN3mSwOxGXn/hbVNMYq/N -Hwtjuzqd+/x5AJhhdM8mgkBj87JyahkNmcrUDnXMN/uLicFZ8WJ/X7NfZTD4p7dN -dloedl40wOiWVpmKs/B/pM293DIxfJHP4F8R+GuqSVzRmZTRouNjWwl2tVZi4Ut0 -HZbUJtQIBFnQmA4O5t78w+wfkPECAwEAAaNCMEAwDwYDVR0TAQH/BAUwAwEB/zAO -BgNVHQ8BAf8EBAMCAQYwHQYDVR0OBBYEFHwMMh+n2TB/xH1oo2Kooc6rB1snMA0G -CSqGSIb3DQEBCwUAA4IBAQARWfolTwNvlJk7mh+ChTnUdgWUXuEok21iXQnCoKjU -sHU48TRqneSfioYmUeYs0cYtbpUgSpIB7LiKZ3sx4mcujJUDJi5DnUox9g61DLu3 -4jd/IroAow57UvtruzvE03lRTs2Q9GcHGcg8RnoNAX3FWOdt5oUwF5okxBDgBPfg -8n/Uqgr/Qh037ZTlZFkSIHc40zI+OIF1lnP6aI+xy84fxez6nH7PfrHxBy22/L/K -pL/QlwVKvOoYKAKQvVR4CSFx09F9HdkWsKlhPdAKACL8x3vLCWRFCztAgfd9fDL1 -mMpYjn0q7pBZc2T5NnReJaH1ZgUufzkVqSr7UIuOhWn0 ------END CERTIFICATE----- - -# Issuer: CN=Starfield Services Root Certificate Authority - G2 O=Starfield Technologies, Inc. -# Subject: CN=Starfield Services Root Certificate Authority - G2 O=Starfield Technologies, Inc. -# Label: "Starfield Services Root Certificate Authority - G2" -# Serial: 0 -# MD5 Fingerprint: 17:35:74:af:7b:61:1c:eb:f4:f9:3c:e2:ee:40:f9:a2 -# SHA1 Fingerprint: 92:5a:8f:8d:2c:6d:04:e0:66:5f:59:6a:ff:22:d8:63:e8:25:6f:3f -# SHA256 Fingerprint: 56:8d:69:05:a2:c8:87:08:a4:b3:02:51:90:ed:cf:ed:b1:97:4a:60:6a:13:c6:e5:29:0f:cb:2a:e6:3e:da:b5 ------BEGIN CERTIFICATE----- -MIID7zCCAtegAwIBAgIBADANBgkqhkiG9w0BAQsFADCBmDELMAkGA1UEBhMCVVMx -EDAOBgNVBAgTB0FyaXpvbmExEzARBgNVBAcTClNjb3R0c2RhbGUxJTAjBgNVBAoT -HFN0YXJmaWVsZCBUZWNobm9sb2dpZXMsIEluYy4xOzA5BgNVBAMTMlN0YXJmaWVs -ZCBTZXJ2aWNlcyBSb290IENlcnRpZmljYXRlIEF1dGhvcml0eSAtIEcyMB4XDTA5 -MDkwMTAwMDAwMFoXDTM3MTIzMTIzNTk1OVowgZgxCzAJBgNVBAYTAlVTMRAwDgYD -VQQIEwdBcml6b25hMRMwEQYDVQQHEwpTY290dHNkYWxlMSUwIwYDVQQKExxTdGFy -ZmllbGQgVGVjaG5vbG9naWVzLCBJbmMuMTswOQYDVQQDEzJTdGFyZmllbGQgU2Vy -dmljZXMgUm9vdCBDZXJ0aWZpY2F0ZSBBdXRob3JpdHkgLSBHMjCCASIwDQYJKoZI -hvcNAQEBBQADggEPADCCAQoCggEBANUMOsQq+U7i9b4Zl1+OiFOxHz/Lz58gE20p -OsgPfTz3a3Y4Y9k2YKibXlwAgLIvWX/2h/klQ4bnaRtSmpDhcePYLQ1Ob/bISdm2 -8xpWriu2dBTrz/sm4xq6HZYuajtYlIlHVv8loJNwU4PahHQUw2eeBGg6345AWh1K -Ts9DkTvnVtYAcMtS7nt9rjrnvDH5RfbCYM8TWQIrgMw0R9+53pBlbQLPLJGmpufe -hRhJfGZOozptqbXuNC66DQO4M99H67FrjSXZm86B0UVGMpZwh94CDklDhbZsc7tk -6mFBrMnUVN+HL8cisibMn1lUaJ/8viovxFUcdUBgF4UCVTmLfwUCAwEAAaNCMEAw -DwYDVR0TAQH/BAUwAwEB/zAOBgNVHQ8BAf8EBAMCAQYwHQYDVR0OBBYEFJxfAN+q -AdcwKziIorhtSpzyEZGDMA0GCSqGSIb3DQEBCwUAA4IBAQBLNqaEd2ndOxmfZyMI -bw5hyf2E3F/YNoHN2BtBLZ9g3ccaaNnRbobhiCPPE95Dz+I0swSdHynVv/heyNXB -ve6SbzJ08pGCL72CQnqtKrcgfU28elUSwhXqvfdqlS5sdJ/PHLTyxQGjhdByPq1z -qwubdQxtRbeOlKyWN7Wg0I8VRw7j6IPdj/3vQQF3zCepYoUz8jcI73HPdwbeyBkd -iEDPfUYd/x7H4c7/I9vG+o1VTqkC50cRRj70/b17KSa7qWFiNyi2LSr2EIZkyXCn -0q23KXB56jzaYyWf/Wi3MOxw+3WKt21gZ7IeyLnp2KhvAotnDU0mV3HaIPzBSlCN -sSi6 ------END CERTIFICATE----- - -# Issuer: CN=AffirmTrust Commercial O=AffirmTrust -# Subject: CN=AffirmTrust Commercial O=AffirmTrust -# Label: "AffirmTrust Commercial" -# Serial: 8608355977964138876 -# MD5 Fingerprint: 82:92:ba:5b:ef:cd:8a:6f:a6:3d:55:f9:84:f6:d6:b7 -# SHA1 Fingerprint: f9:b5:b6:32:45:5f:9c:be:ec:57:5f:80:dc:e9:6e:2c:c7:b2:78:b7 -# SHA256 Fingerprint: 03:76:ab:1d:54:c5:f9:80:3c:e4:b2:e2:01:a0:ee:7e:ef:7b:57:b6:36:e8:a9:3c:9b:8d:48:60:c9:6f:5f:a7 ------BEGIN CERTIFICATE----- -MIIDTDCCAjSgAwIBAgIId3cGJyapsXwwDQYJKoZIhvcNAQELBQAwRDELMAkGA1UE -BhMCVVMxFDASBgNVBAoMC0FmZmlybVRydXN0MR8wHQYDVQQDDBZBZmZpcm1UcnVz -dCBDb21tZXJjaWFsMB4XDTEwMDEyOTE0MDYwNloXDTMwMTIzMTE0MDYwNlowRDEL -MAkGA1UEBhMCVVMxFDASBgNVBAoMC0FmZmlybVRydXN0MR8wHQYDVQQDDBZBZmZp -cm1UcnVzdCBDb21tZXJjaWFsMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKC -AQEA9htPZwcroRX1BiLLHwGy43NFBkRJLLtJJRTWzsO3qyxPxkEylFf6EqdbDuKP -Hx6GGaeqtS25Xw2Kwq+FNXkyLbscYjfysVtKPcrNcV/pQr6U6Mje+SJIZMblq8Yr -ba0F8PrVC8+a5fBQpIs7R6UjW3p6+DM/uO+Zl+MgwdYoic+U+7lF7eNAFxHUdPAL -MeIrJmqbTFeurCA+ukV6BfO9m2kVrn1OIGPENXY6BwLJN/3HR+7o8XYdcxXyl6S1 -yHp52UKqK39c/s4mT6NmgTWvRLpUHhwwMmWd5jyTXlBOeuM61G7MGvv50jeuJCqr -VwMiKA1JdX+3KNp1v47j3A55MQIDAQABo0IwQDAdBgNVHQ4EFgQUnZPGU4teyq8/ -nx4P5ZmVvCT2lI8wDwYDVR0TAQH/BAUwAwEB/zAOBgNVHQ8BAf8EBAMCAQYwDQYJ -KoZIhvcNAQELBQADggEBAFis9AQOzcAN/wr91LoWXym9e2iZWEnStB03TX8nfUYG -XUPGhi4+c7ImfU+TqbbEKpqrIZcUsd6M06uJFdhrJNTxFq7YpFzUf1GO7RgBsZNj -vbz4YYCanrHOQnDiqX0GJX0nof5v7LMeJNrjS1UaADs1tDvZ110w/YETifLCBivt -Z8SOyUOyXGsViQK8YvxO8rUzqrJv0wqiUOP2O+guRMLbZjipM1ZI8W0bM40NjD9g -N53Tym1+NH4Nn3J2ixufcv1SNUFFApYvHLKac0khsUlHRUe072o0EclNmsxZt9YC -nlpOZbWUrhvfKbAW8b8Angc6F2S1BLUjIZkKlTuXfO8= ------END CERTIFICATE----- - -# Issuer: CN=AffirmTrust Networking O=AffirmTrust -# Subject: CN=AffirmTrust Networking O=AffirmTrust -# Label: "AffirmTrust Networking" -# Serial: 8957382827206547757 -# MD5 Fingerprint: 42:65:ca:be:01:9a:9a:4c:a9:8c:41:49:cd:c0:d5:7f -# SHA1 Fingerprint: 29:36:21:02:8b:20:ed:02:f5:66:c5:32:d1:d6:ed:90:9f:45:00:2f -# SHA256 Fingerprint: 0a:81:ec:5a:92:97:77:f1:45:90:4a:f3:8d:5d:50:9f:66:b5:e2:c5:8f:cd:b5:31:05:8b:0e:17:f3:f0:b4:1b ------BEGIN CERTIFICATE----- -MIIDTDCCAjSgAwIBAgIIfE8EORzUmS0wDQYJKoZIhvcNAQEFBQAwRDELMAkGA1UE -BhMCVVMxFDASBgNVBAoMC0FmZmlybVRydXN0MR8wHQYDVQQDDBZBZmZpcm1UcnVz -dCBOZXR3b3JraW5nMB4XDTEwMDEyOTE0MDgyNFoXDTMwMTIzMTE0MDgyNFowRDEL -MAkGA1UEBhMCVVMxFDASBgNVBAoMC0FmZmlybVRydXN0MR8wHQYDVQQDDBZBZmZp -cm1UcnVzdCBOZXR3b3JraW5nMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKC -AQEAtITMMxcua5Rsa2FSoOujz3mUTOWUgJnLVWREZY9nZOIG41w3SfYvm4SEHi3y -YJ0wTsyEheIszx6e/jarM3c1RNg1lho9Nuh6DtjVR6FqaYvZ/Ls6rnla1fTWcbua -kCNrmreIdIcMHl+5ni36q1Mr3Lt2PpNMCAiMHqIjHNRqrSK6mQEubWXLviRmVSRL -QESxG9fhwoXA3hA/Pe24/PHxI1Pcv2WXb9n5QHGNfb2V1M6+oF4nI979ptAmDgAp -6zxG8D1gvz9Q0twmQVGeFDdCBKNwV6gbh+0t+nvujArjqWaJGctB+d1ENmHP4ndG -yH329JKBNv3bNPFyfvMMFr20FQIDAQABo0IwQDAdBgNVHQ4EFgQUBx/S55zawm6i -QLSwelAQUHTEyL0wDwYDVR0TAQH/BAUwAwEB/zAOBgNVHQ8BAf8EBAMCAQYwDQYJ -KoZIhvcNAQEFBQADggEBAIlXshZ6qML91tmbmzTCnLQyFE2npN/svqe++EPbkTfO -tDIuUFUaNU52Q3Eg75N3ThVwLofDwR1t3Mu1J9QsVtFSUzpE0nPIxBsFZVpikpzu -QY0x2+c06lkh1QF612S4ZDnNye2v7UsDSKegmQGA3GWjNq5lWUhPgkvIZfFXHeVZ -Lgo/bNjR9eUJtGxUAArgFU2HdW23WJZa3W3SAKD0m0i+wzekujbgfIeFlxoVot4u -olu9rxj5kFDNcFn4J2dHy8egBzp90SxdbBk6ZrV9/ZFvgrG+CJPbFEfxojfHRZ48 -x3evZKiT3/Zpg4Jg8klCNO1aAFSFHBY2kgxc+qatv9s= ------END CERTIFICATE----- - -# Issuer: CN=AffirmTrust Premium O=AffirmTrust -# Subject: CN=AffirmTrust Premium O=AffirmTrust -# Label: "AffirmTrust Premium" -# Serial: 7893706540734352110 -# MD5 Fingerprint: c4:5d:0e:48:b6:ac:28:30:4e:0a:bc:f9:38:16:87:57 -# SHA1 Fingerprint: d8:a6:33:2c:e0:03:6f:b1:85:f6:63:4f:7d:6a:06:65:26:32:28:27 -# SHA256 Fingerprint: 70:a7:3f:7f:37:6b:60:07:42:48:90:45:34:b1:14:82:d5:bf:0e:69:8e:cc:49:8d:f5:25:77:eb:f2:e9:3b:9a ------BEGIN CERTIFICATE----- -MIIFRjCCAy6gAwIBAgIIbYwURrGmCu4wDQYJKoZIhvcNAQEMBQAwQTELMAkGA1UE -BhMCVVMxFDASBgNVBAoMC0FmZmlybVRydXN0MRwwGgYDVQQDDBNBZmZpcm1UcnVz -dCBQcmVtaXVtMB4XDTEwMDEyOTE0MTAzNloXDTQwMTIzMTE0MTAzNlowQTELMAkG -A1UEBhMCVVMxFDASBgNVBAoMC0FmZmlybVRydXN0MRwwGgYDVQQDDBNBZmZpcm1U -cnVzdCBQcmVtaXVtMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAxBLf -qV/+Qd3d9Z+K4/as4Tx4mrzY8H96oDMq3I0gW64tb+eT2TZwamjPjlGjhVtnBKAQ -JG9dKILBl1fYSCkTtuG+kU3fhQxTGJoeJKJPj/CihQvL9Cl/0qRY7iZNyaqoe5rZ -+jjeRFcV5fiMyNlI4g0WJx0eyIOFJbe6qlVBzAMiSy2RjYvmia9mx+n/K+k8rNrS -s8PhaJyJ+HoAVt70VZVs+7pk3WKL3wt3MutizCaam7uqYoNMtAZ6MMgpv+0GTZe5 -HMQxK9VfvFMSF5yZVylmd2EhMQcuJUmdGPLu8ytxjLW6OQdJd/zvLpKQBY0tL3d7 -70O/Nbua2Plzpyzy0FfuKE4mX4+QaAkvuPjcBukumj5Rp9EixAqnOEhss/n/fauG -V+O61oV4d7pD6kh/9ti+I20ev9E2bFhc8e6kGVQa9QPSdubhjL08s9NIS+LI+H+S -qHZGnEJlPqQewQcDWkYtuJfzt9WyVSHvutxMAJf7FJUnM7/oQ0dG0giZFmA7mn7S -5u046uwBHjxIVkkJx0w3AJ6IDsBz4W9m6XJHMD4Q5QsDyZpCAGzFlH5hxIrff4Ia -C1nEWTJ3s7xgaVY5/bQGeyzWZDbZvUjthB9+pSKPKrhC9IK31FOQeE4tGv2Bb0TX -OwF0lkLgAOIua+rF7nKsu7/+6qqo+Nz2snmKtmcCAwEAAaNCMEAwHQYDVR0OBBYE -FJ3AZ6YMItkm9UWrpmVSESfYRaxjMA8GA1UdEwEB/wQFMAMBAf8wDgYDVR0PAQH/ -BAQDAgEGMA0GCSqGSIb3DQEBDAUAA4ICAQCzV00QYk465KzquByvMiPIs0laUZx2 -KI15qldGF9X1Uva3ROgIRL8YhNILgM3FEv0AVQVhh0HctSSePMTYyPtwni94loMg -Nt58D2kTiKV1NpgIpsbfrM7jWNa3Pt668+s0QNiigfV4Py/VpfzZotReBA4Xrf5B -8OWycvpEgjNC6C1Y91aMYj+6QrCcDFx+LmUmXFNPALJ4fqENmS2NuB2OosSw/WDQ -MKSOyARiqcTtNd56l+0OOF6SL5Nwpamcb6d9Ex1+xghIsV5n61EIJenmJWtSKZGc -0jlzCFfemQa0W50QBuHCAKi4HEoCChTQwUHK+4w1IX2COPKpVJEZNZOUbWo6xbLQ -u4mGk+ibyQ86p3q4ofB4Rvr8Ny/lioTz3/4E2aFooC8k4gmVBtWVyuEklut89pMF -u+1z6S3RdTnX5yTb2E5fQ4+e0BQ5v1VwSJlXMbSc7kqYA5YwH2AG7hsj/oFgIxpH -YoWlzBk0gG+zrBrjn/B7SK3VAdlntqlyk+otZrWyuOQ9PLLvTIzq6we/qzWaVYa8 -GKa1qF60g2xraUDTn9zxw2lrueFtCfTxqlB2Cnp9ehehVZZCmTEJ3WARjQUwfuaO -RtGdFNrHF+QFlozEJLUbzxQHskD4o55BhrwE0GuWyCqANP2/7waj3VjFhT0+j/6e -KeC2uAloGRwYQw== ------END CERTIFICATE----- - -# Issuer: CN=AffirmTrust Premium ECC O=AffirmTrust -# Subject: CN=AffirmTrust Premium ECC O=AffirmTrust -# Label: "AffirmTrust Premium ECC" -# Serial: 8401224907861490260 -# MD5 Fingerprint: 64:b0:09:55:cf:b1:d5:99:e2:be:13:ab:a6:5d:ea:4d -# SHA1 Fingerprint: b8:23:6b:00:2f:1d:16:86:53:01:55:6c:11:a4:37:ca:eb:ff:c3:bb -# SHA256 Fingerprint: bd:71:fd:f6:da:97:e4:cf:62:d1:64:7a:dd:25:81:b0:7d:79:ad:f8:39:7e:b4:ec:ba:9c:5e:84:88:82:14:23 ------BEGIN CERTIFICATE----- -MIIB/jCCAYWgAwIBAgIIdJclisc/elQwCgYIKoZIzj0EAwMwRTELMAkGA1UEBhMC -VVMxFDASBgNVBAoMC0FmZmlybVRydXN0MSAwHgYDVQQDDBdBZmZpcm1UcnVzdCBQ -cmVtaXVtIEVDQzAeFw0xMDAxMjkxNDIwMjRaFw00MDEyMzExNDIwMjRaMEUxCzAJ -BgNVBAYTAlVTMRQwEgYDVQQKDAtBZmZpcm1UcnVzdDEgMB4GA1UEAwwXQWZmaXJt -VHJ1c3QgUHJlbWl1bSBFQ0MwdjAQBgcqhkjOPQIBBgUrgQQAIgNiAAQNMF4bFZ0D -0KF5Nbc6PJJ6yhUczWLznCZcBz3lVPqj1swS6vQUX+iOGasvLkjmrBhDeKzQN8O9 -ss0s5kfiGuZjuD0uL3jET9v0D6RoTFVya5UdThhClXjMNzyR4ptlKymjQjBAMB0G -A1UdDgQWBBSaryl6wBE1NSZRMADDav5A1a7WPDAPBgNVHRMBAf8EBTADAQH/MA4G -A1UdDwEB/wQEAwIBBjAKBggqhkjOPQQDAwNnADBkAjAXCfOHiFBar8jAQr9HX/Vs -aobgxCd05DhT1wV/GzTjxi+zygk8N53X57hG8f2h4nECMEJZh0PUUd+60wkyWs6I -flc9nF9Ca/UHLbXwgpP5WW+uZPpY5Yse42O+tYHNbwKMeQ== ------END CERTIFICATE----- - -# Issuer: CN=Certum Trusted Network CA O=Unizeto Technologies S.A. OU=Certum Certification Authority -# Subject: CN=Certum Trusted Network CA O=Unizeto Technologies S.A. OU=Certum Certification Authority -# Label: "Certum Trusted Network CA" -# Serial: 279744 -# MD5 Fingerprint: d5:e9:81:40:c5:18:69:fc:46:2c:89:75:62:0f:aa:78 -# SHA1 Fingerprint: 07:e0:32:e0:20:b7:2c:3f:19:2f:06:28:a2:59:3a:19:a7:0f:06:9e -# SHA256 Fingerprint: 5c:58:46:8d:55:f5:8e:49:7e:74:39:82:d2:b5:00:10:b6:d1:65:37:4a:cf:83:a7:d4:a3:2d:b7:68:c4:40:8e ------BEGIN CERTIFICATE----- -MIIDuzCCAqOgAwIBAgIDBETAMA0GCSqGSIb3DQEBBQUAMH4xCzAJBgNVBAYTAlBM -MSIwIAYDVQQKExlVbml6ZXRvIFRlY2hub2xvZ2llcyBTLkEuMScwJQYDVQQLEx5D -ZXJ0dW0gQ2VydGlmaWNhdGlvbiBBdXRob3JpdHkxIjAgBgNVBAMTGUNlcnR1bSBU -cnVzdGVkIE5ldHdvcmsgQ0EwHhcNMDgxMDIyMTIwNzM3WhcNMjkxMjMxMTIwNzM3 -WjB+MQswCQYDVQQGEwJQTDEiMCAGA1UEChMZVW5pemV0byBUZWNobm9sb2dpZXMg -Uy5BLjEnMCUGA1UECxMeQ2VydHVtIENlcnRpZmljYXRpb24gQXV0aG9yaXR5MSIw -IAYDVQQDExlDZXJ0dW0gVHJ1c3RlZCBOZXR3b3JrIENBMIIBIjANBgkqhkiG9w0B -AQEFAAOCAQ8AMIIBCgKCAQEA4/t9o3K6wvDJFIf1awFO4W5AB7ptJ11/91sts1rH -UV+rpDKmYYe2bg+G0jACl/jXaVehGDldamR5xgFZrDwxSjh80gTSSyjoIF87B6LM -TXPb865Px1bVWqeWifrzq2jUI4ZZJ88JJ7ysbnKDHDBy3+Ci6dLhdHUZvSqeexVU -BBvXQzmtVSjF4hq79MDkrjhJM8x2hZ85RdKknvISjFH4fOQtf/WsX+sWn7Et0brM -kUJ3TCXJkDhv2/DM+44el1k+1WBO5gUo7Ul5E0u6SNsv+XLTOcr+H9g0cvW0QM8x -AcPs3hEtF10fuFDRXhmnad4HMyjKUJX5p1TLVIZQRan5SQIDAQABo0IwQDAPBgNV -HRMBAf8EBTADAQH/MB0GA1UdDgQWBBQIds3LB/8k9sXN7buQvOKEN0Z19zAOBgNV -HQ8BAf8EBAMCAQYwDQYJKoZIhvcNAQEFBQADggEBAKaorSLOAT2mo/9i0Eidi15y -sHhE49wcrwn9I0j6vSrEuVUEtRCjjSfeC4Jj0O7eDDd5QVsisrCaQVymcODU0HfL -I9MA4GxWL+FpDQ3Zqr8hgVDZBqWo/5U30Kr+4rP1mS1FhIrlQgnXdAIv94nYmem8 -J9RHjboNRhx3zxSkHLmkMcScKHQDNP8zGSal6Q10tz6XxnboJ5ajZt3hrvJBW8qY -VoNzcOSGGtIxQbovvi0TWnZvTuhOgQ4/WwMioBK+ZlgRSssDxLQqKi2WF+A5VLxI -03YnnZotBqbJ7DnSq9ufmgsnAjUpsUCV5/nonFWIGUbWtzT1fs45mtk48VH3Tyw= ------END CERTIFICATE----- - -# Issuer: CN=TWCA Root Certification Authority O=TAIWAN-CA OU=Root CA -# Subject: CN=TWCA Root Certification Authority O=TAIWAN-CA OU=Root CA -# Label: "TWCA Root Certification Authority" -# Serial: 1 -# MD5 Fingerprint: aa:08:8f:f6:f9:7b:b7:f2:b1:a7:1e:9b:ea:ea:bd:79 -# SHA1 Fingerprint: cf:9e:87:6d:d3:eb:fc:42:26:97:a3:b5:a3:7a:a0:76:a9:06:23:48 -# SHA256 Fingerprint: bf:d8:8f:e1:10:1c:41:ae:3e:80:1b:f8:be:56:35:0e:e9:ba:d1:a6:b9:bd:51:5e:dc:5c:6d:5b:87:11:ac:44 ------BEGIN CERTIFICATE----- -MIIDezCCAmOgAwIBAgIBATANBgkqhkiG9w0BAQUFADBfMQswCQYDVQQGEwJUVzES -MBAGA1UECgwJVEFJV0FOLUNBMRAwDgYDVQQLDAdSb290IENBMSowKAYDVQQDDCFU -V0NBIFJvb3QgQ2VydGlmaWNhdGlvbiBBdXRob3JpdHkwHhcNMDgwODI4MDcyNDMz -WhcNMzAxMjMxMTU1OTU5WjBfMQswCQYDVQQGEwJUVzESMBAGA1UECgwJVEFJV0FO -LUNBMRAwDgYDVQQLDAdSb290IENBMSowKAYDVQQDDCFUV0NBIFJvb3QgQ2VydGlm -aWNhdGlvbiBBdXRob3JpdHkwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIB -AQCwfnK4pAOU5qfeCTiRShFAh6d8WWQUe7UREN3+v9XAu1bihSX0NXIP+FPQQeFE -AcK0HMMxQhZHhTMidrIKbw/lJVBPhYa+v5guEGcevhEFhgWQxFnQfHgQsIBct+HH -K3XLfJ+utdGdIzdjp9xCoi2SBBtQwXu4PhvJVgSLL1KbralW6cH/ralYhzC2gfeX -RfwZVzsrb+RH9JlF/h3x+JejiB03HFyP4HYlmlD4oFT/RJB2I9IyxsOrBr/8+7/z -rX2SYgJbKdM1o5OaQ2RgXbL6Mv87BK9NQGr5x+PvI/1ry+UPizgN7gr8/g+YnzAx -3WxSZfmLgb4i4RxYA7qRG4kHAgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBBjAPBgNV -HRMBAf8EBTADAQH/MB0GA1UdDgQWBBRqOFsmjd6LWvJPelSDGRjjCDWmujANBgkq -hkiG9w0BAQUFAAOCAQEAPNV3PdrfibqHDAhUaiBQkr6wQT25JmSDCi/oQMCXKCeC -MErJk/9q56YAf4lCmtYR5VPOL8zy2gXE/uJQxDqGfczafhAJO5I1KlOy/usrBdls -XebQ79NqZp4VKIV66IIArB6nCWlWQtNoURi+VJq/REG6Sb4gumlc7rh3zc5sH62D -lhh9DrUUOYTxKOkto557HnpyWoOzeW/vtPzQCqVYT0bf+215WfKEIlKuD8z7fDvn -aspHYcN6+NOSBB+4IIThNlQWx0DeO4pz3N/GCUzf7Nr/1FNCocnyYh0igzyXxfkZ -YiesZSLX0zzG5Y6yU8xJzrww/nsOM5D77dIUkR8Hrw== ------END CERTIFICATE----- - -# Issuer: O=SECOM Trust Systems CO.,LTD. OU=Security Communication RootCA2 -# Subject: O=SECOM Trust Systems CO.,LTD. OU=Security Communication RootCA2 -# Label: "Security Communication RootCA2" -# Serial: 0 -# MD5 Fingerprint: 6c:39:7d:a4:0e:55:59:b2:3f:d6:41:b1:12:50:de:43 -# SHA1 Fingerprint: 5f:3b:8c:f2:f8:10:b3:7d:78:b4:ce:ec:19:19:c3:73:34:b9:c7:74 -# SHA256 Fingerprint: 51:3b:2c:ec:b8:10:d4:cd:e5:dd:85:39:1a:df:c6:c2:dd:60:d8:7b:b7:36:d2:b5:21:48:4a:a4:7a:0e:be:f6 ------BEGIN CERTIFICATE----- -MIIDdzCCAl+gAwIBAgIBADANBgkqhkiG9w0BAQsFADBdMQswCQYDVQQGEwJKUDEl -MCMGA1UEChMcU0VDT00gVHJ1c3QgU3lzdGVtcyBDTy4sTFRELjEnMCUGA1UECxMe -U2VjdXJpdHkgQ29tbXVuaWNhdGlvbiBSb290Q0EyMB4XDTA5MDUyOTA1MDAzOVoX -DTI5MDUyOTA1MDAzOVowXTELMAkGA1UEBhMCSlAxJTAjBgNVBAoTHFNFQ09NIFRy -dXN0IFN5c3RlbXMgQ08uLExURC4xJzAlBgNVBAsTHlNlY3VyaXR5IENvbW11bmlj -YXRpb24gUm9vdENBMjCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBANAV -OVKxUrO6xVmCxF1SrjpDZYBLx/KWvNs2l9amZIyoXvDjChz335c9S672XewhtUGr -zbl+dp+++T42NKA7wfYxEUV0kz1XgMX5iZnK5atq1LXaQZAQwdbWQonCv/Q4EpVM -VAX3NuRFg3sUZdbcDE3R3n4MqzvEFb46VqZab3ZpUql6ucjrappdUtAtCms1FgkQ -hNBqyjoGADdH5H5XTz+L62e4iKrFvlNVspHEfbmwhRkGeC7bYRr6hfVKkaHnFtWO -ojnflLhwHyg/i/xAXmODPIMqGplrz95Zajv8bxbXH/1KEOtOghY6rCcMU/Gt1SSw -awNQwS08Ft1ENCcadfsCAwEAAaNCMEAwHQYDVR0OBBYEFAqFqXdlBZh8QIH4D5cs -OPEK7DzPMA4GA1UdDwEB/wQEAwIBBjAPBgNVHRMBAf8EBTADAQH/MA0GCSqGSIb3 -DQEBCwUAA4IBAQBMOqNErLlFsceTfsgLCkLfZOoc7llsCLqJX2rKSpWeeo8HxdpF -coJxDjrSzG+ntKEju/Ykn8sX/oymzsLS28yN/HH8AynBbF0zX2S2ZTuJbxh2ePXc -okgfGT+Ok+vx+hfuzU7jBBJV1uXk3fs+BXziHV7Gp7yXT2g69ekuCkO2r1dcYmh8 -t/2jioSgrGK+KwmHNPBqAbubKVY8/gA3zyNs8U6qtnRGEmyR7jTV7JqR50S+kDFy -1UkC9gLl9B/rfNmWVan/7Ir5mUf/NVoCqgTLiluHcSmRvaS0eg29mvVXIwAHIRc/ -SjnRBUkLp7Y3gaVdjKozXoEofKd9J+sAro03 ------END CERTIFICATE----- - -# Issuer: CN=Actalis Authentication Root CA O=Actalis S.p.A./03358520967 -# Subject: CN=Actalis Authentication Root CA O=Actalis S.p.A./03358520967 -# Label: "Actalis Authentication Root CA" -# Serial: 6271844772424770508 -# MD5 Fingerprint: 69:c1:0d:4f:07:a3:1b:c3:fe:56:3d:04:bc:11:f6:a6 -# SHA1 Fingerprint: f3:73:b3:87:06:5a:28:84:8a:f2:f3:4a:ce:19:2b:dd:c7:8e:9c:ac -# SHA256 Fingerprint: 55:92:60:84:ec:96:3a:64:b9:6e:2a:be:01:ce:0b:a8:6a:64:fb:fe:bc:c7:aa:b5:af:c1:55:b3:7f:d7:60:66 ------BEGIN CERTIFICATE----- -MIIFuzCCA6OgAwIBAgIIVwoRl0LE48wwDQYJKoZIhvcNAQELBQAwazELMAkGA1UE -BhMCSVQxDjAMBgNVBAcMBU1pbGFuMSMwIQYDVQQKDBpBY3RhbGlzIFMucC5BLi8w -MzM1ODUyMDk2NzEnMCUGA1UEAwweQWN0YWxpcyBBdXRoZW50aWNhdGlvbiBSb290 -IENBMB4XDTExMDkyMjExMjIwMloXDTMwMDkyMjExMjIwMlowazELMAkGA1UEBhMC -SVQxDjAMBgNVBAcMBU1pbGFuMSMwIQYDVQQKDBpBY3RhbGlzIFMucC5BLi8wMzM1 -ODUyMDk2NzEnMCUGA1UEAwweQWN0YWxpcyBBdXRoZW50aWNhdGlvbiBSb290IENB -MIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAp8bEpSmkLO/lGMWwUKNv -UTufClrJwkg4CsIcoBh/kbWHuUA/3R1oHwiD1S0eiKD4j1aPbZkCkpAW1V8IbInX -4ay8IMKx4INRimlNAJZaby/ARH6jDuSRzVju3PvHHkVH3Se5CAGfpiEd9UEtL0z9 -KK3giq0itFZljoZUj5NDKd45RnijMCO6zfB9E1fAXdKDa0hMxKufgFpbOr3JpyI/ -gCczWw63igxdBzcIy2zSekciRDXFzMwujt0q7bd9Zg1fYVEiVRvjRuPjPdA1Yprb -rxTIW6HMiRvhMCb8oJsfgadHHwTrozmSBp+Z07/T6k9QnBn+locePGX2oxgkg4YQ -51Q+qDp2JE+BIcXjDwL4k5RHILv+1A7TaLndxHqEguNTVHnd25zS8gebLra8Pu2F -be8lEfKXGkJh90qX6IuxEAf6ZYGyojnP9zz/GPvG8VqLWeICrHuS0E4UT1lF9gxe -KF+w6D9Fz8+vm2/7hNN3WpVvrJSEnu68wEqPSpP4RCHiMUVhUE4Q2OM1fEwZtN4F -v6MGn8i1zeQf1xcGDXqVdFUNaBr8EBtiZJ1t4JWgw5QHVw0U5r0F+7if5t+L4sbn -fpb2U8WANFAoWPASUHEXMLrmeGO89LKtmyuy/uE5jF66CyCU3nuDuP/jVo23Eek7 -jPKxwV2dpAtMK9myGPW1n0sCAwEAAaNjMGEwHQYDVR0OBBYEFFLYiDrIn3hm7Ynz -ezhwlMkCAjbQMA8GA1UdEwEB/wQFMAMBAf8wHwYDVR0jBBgwFoAUUtiIOsifeGbt -ifN7OHCUyQICNtAwDgYDVR0PAQH/BAQDAgEGMA0GCSqGSIb3DQEBCwUAA4ICAQAL -e3KHwGCmSUyIWOYdiPcUZEim2FgKDk8TNd81HdTtBjHIgT5q1d07GjLukD0R0i70 -jsNjLiNmsGe+b7bAEzlgqqI0JZN1Ut6nna0Oh4lScWoWPBkdg/iaKWW+9D+a2fDz -WochcYBNy+A4mz+7+uAwTc+G02UQGRjRlwKxK3JCaKygvU5a2hi/a5iB0P2avl4V -SM0RFbnAKVy06Ij3Pjaut2L9HmLecHgQHEhb2rykOLpn7VU+Xlff1ANATIGk0k9j -pwlCCRT8AKnCgHNPLsBA2RF7SOp6AsDT6ygBJlh0wcBzIm2Tlf05fbsq4/aC4yyX -X04fkZT6/iyj2HYauE2yOE+b+h1IYHkm4vP9qdCa6HCPSXrW5b0KDtst842/6+Ok -fcvHlXHo2qN8xcL4dJIEG4aspCJTQLas/kx2z/uUMsA1n3Y/buWQbqCmJqK4LL7R -K4X9p2jIugErsWx0Hbhzlefut8cl8ABMALJ+tguLHPPAUJ4lueAI3jZm/zel0btU -ZCzJJ7VLkn5l/9Mt4blOvH+kQSGQQXemOR/qnuOf0GZvBeyqdn6/axag67XH/JJU -LysRJyU3eExRarDzzFhdFPFqSBX/wge2sY0PjlxQRrM9vwGYT7JZVEc+NHt4bVaT -LnPqZih4zR0Uv6CPLy64Lo7yFIrM6bV8+2ydDKXhlg== ------END CERTIFICATE----- - -# Issuer: CN=Buypass Class 2 Root CA O=Buypass AS-983163327 -# Subject: CN=Buypass Class 2 Root CA O=Buypass AS-983163327 -# Label: "Buypass Class 2 Root CA" -# Serial: 2 -# MD5 Fingerprint: 46:a7:d2:fe:45:fb:64:5a:a8:59:90:9b:78:44:9b:29 -# SHA1 Fingerprint: 49:0a:75:74:de:87:0a:47:fe:58:ee:f6:c7:6b:eb:c6:0b:12:40:99 -# SHA256 Fingerprint: 9a:11:40:25:19:7c:5b:b9:5d:94:e6:3d:55:cd:43:79:08:47:b6:46:b2:3c:df:11:ad:a4:a0:0e:ff:15:fb:48 ------BEGIN CERTIFICATE----- -MIIFWTCCA0GgAwIBAgIBAjANBgkqhkiG9w0BAQsFADBOMQswCQYDVQQGEwJOTzEd -MBsGA1UECgwUQnV5cGFzcyBBUy05ODMxNjMzMjcxIDAeBgNVBAMMF0J1eXBhc3Mg -Q2xhc3MgMiBSb290IENBMB4XDTEwMTAyNjA4MzgwM1oXDTQwMTAyNjA4MzgwM1ow -TjELMAkGA1UEBhMCTk8xHTAbBgNVBAoMFEJ1eXBhc3MgQVMtOTgzMTYzMzI3MSAw -HgYDVQQDDBdCdXlwYXNzIENsYXNzIDIgUm9vdCBDQTCCAiIwDQYJKoZIhvcNAQEB -BQADggIPADCCAgoCggIBANfHXvfBB9R3+0Mh9PT1aeTuMgHbo4Yf5FkNuud1g1Lr -6hxhFUi7HQfKjK6w3Jad6sNgkoaCKHOcVgb/S2TwDCo3SbXlzwx87vFKu3MwZfPV -L4O2fuPn9Z6rYPnT8Z2SdIrkHJasW4DptfQxh6NR/Md+oW+OU3fUl8FVM5I+GC91 -1K2GScuVr1QGbNgGE41b/+EmGVnAJLqBcXmQRFBoJJRfuLMR8SlBYaNByyM21cHx -MlAQTn/0hpPshNOOvEu/XAFOBz3cFIqUCqTqc/sLUegTBxj6DvEr0VQVfTzh97QZ -QmdiXnfgolXsttlpF9U6r0TtSsWe5HonfOV116rLJeffawrbD02TTqigzXsu8lkB -arcNuAeBfos4GzjmCleZPe4h6KP1DBbdi+w0jpwqHAAVF41og9JwnxgIzRFo1clr -Us3ERo/ctfPYV3Me6ZQ5BL/T3jjetFPsaRyifsSP5BtwrfKi+fv3FmRmaZ9JUaLi -FRhnBkp/1Wy1TbMz4GHrXb7pmA8y1x1LPC5aAVKRCfLf6o3YBkBjqhHk/sM3nhRS -P/TizPJhk9H9Z2vXUq6/aKtAQ6BXNVN48FP4YUIHZMbXb5tMOA1jrGKvNouicwoN -9SG9dKpN6nIDSdvHXx1iY8f93ZHsM+71bbRuMGjeyNYmsHVee7QHIJihdjK4TWxP -AgMBAAGjQjBAMA8GA1UdEwEB/wQFMAMBAf8wHQYDVR0OBBYEFMmAd+BikoL1Rpzz -uvdMw964o605MA4GA1UdDwEB/wQEAwIBBjANBgkqhkiG9w0BAQsFAAOCAgEAU18h -9bqwOlI5LJKwbADJ784g7wbylp7ppHR/ehb8t/W2+xUbP6umwHJdELFx7rxP462s -A20ucS6vxOOto70MEae0/0qyexAQH6dXQbLArvQsWdZHEIjzIVEpMMpghq9Gqx3t -OluwlN5E40EIosHsHdb9T7bWR9AUC8rmyrV7d35BH16Dx7aMOZawP5aBQW9gkOLo -+fsicdl9sz1Gv7SEr5AcD48Saq/v7h56rgJKihcrdv6sVIkkLE8/trKnToyokZf7 -KcZ7XC25y2a2t6hbElGFtQl+Ynhw/qlqYLYdDnkM/crqJIByw5c/8nerQyIKx+u2 -DISCLIBrQYoIwOula9+ZEsuK1V6ADJHgJgg2SMX6OBE1/yWDLfJ6v9r9jv6ly0Us -H8SIU653DtmadsWOLB2jutXsMq7Aqqz30XpN69QH4kj3Io6wpJ9qzo6ysmD0oyLQ -I+uUWnpp3Q+/QFesa1lQ2aOZ4W7+jQF5JyMV3pKdewlNWudLSDBaGOYKbeaP4NK7 -5t98biGCwWg5TbSYWGZizEqQXsP6JwSxeRV0mcy+rSDeJmAc61ZRpqPq5KM/p/9h -3PFaTWwyI0PurKju7koSCTxdccK+efrCh2gdC/1cacwG0Jp9VJkqyTkaGa9LKkPz -Y11aWOIv4x3kqdbQCtCev9eBCfHJxyYNrJgWVqA= ------END CERTIFICATE----- - -# Issuer: CN=Buypass Class 3 Root CA O=Buypass AS-983163327 -# Subject: CN=Buypass Class 3 Root CA O=Buypass AS-983163327 -# Label: "Buypass Class 3 Root CA" -# Serial: 2 -# MD5 Fingerprint: 3d:3b:18:9e:2c:64:5a:e8:d5:88:ce:0e:f9:37:c2:ec -# SHA1 Fingerprint: da:fa:f7:fa:66:84:ec:06:8f:14:50:bd:c7:c2:81:a5:bc:a9:64:57 -# SHA256 Fingerprint: ed:f7:eb:bc:a2:7a:2a:38:4d:38:7b:7d:40:10:c6:66:e2:ed:b4:84:3e:4c:29:b4:ae:1d:5b:93:32:e6:b2:4d ------BEGIN CERTIFICATE----- -MIIFWTCCA0GgAwIBAgIBAjANBgkqhkiG9w0BAQsFADBOMQswCQYDVQQGEwJOTzEd -MBsGA1UECgwUQnV5cGFzcyBBUy05ODMxNjMzMjcxIDAeBgNVBAMMF0J1eXBhc3Mg -Q2xhc3MgMyBSb290IENBMB4XDTEwMTAyNjA4Mjg1OFoXDTQwMTAyNjA4Mjg1OFow -TjELMAkGA1UEBhMCTk8xHTAbBgNVBAoMFEJ1eXBhc3MgQVMtOTgzMTYzMzI3MSAw -HgYDVQQDDBdCdXlwYXNzIENsYXNzIDMgUm9vdCBDQTCCAiIwDQYJKoZIhvcNAQEB -BQADggIPADCCAgoCggIBAKXaCpUWUOOV8l6ddjEGMnqb8RB2uACatVI2zSRHsJ8Y -ZLya9vrVediQYkwiL944PdbgqOkcLNt4EemOaFEVcsfzM4fkoF0LXOBXByow9c3E -N3coTRiR5r/VUv1xLXA+58bEiuPwKAv0dpihi4dVsjoT/Lc+JzeOIuOoTyrvYLs9 -tznDDgFHmV0ST9tD+leh7fmdvhFHJlsTmKtdFoqwNxxXnUX/iJY2v7vKB3tvh2PX -0DJq1l1sDPGzbjniazEuOQAnFN44wOwZZoYS6J1yFhNkUsepNxz9gjDthBgd9K5c -/3ATAOux9TN6S9ZV+AWNS2mw9bMoNlwUxFFzTWsL8TQH2xc519woe2v1n/MuwU8X -KhDzzMro6/1rqy6any2CbgTUUgGTLT2G/H783+9CHaZr77kgxve9oKeV/afmiSTY -zIw0bOIjL9kSGiG5VZFvC5F5GQytQIgLcOJ60g7YaEi7ghM5EFjp2CoHxhLbWNvS -O1UQRwUVZ2J+GGOmRj8JDlQyXr8NYnon74Do29lLBlo3WiXQCBJ31G8JUJc9yB3D -34xFMFbG02SrZvPAXpacw8Tvw3xrizp5f7NJzz3iiZ+gMEuFuZyUJHmPfWupRWgP -K9Dx2hzLabjKSWJtyNBjYt1gD1iqj6G8BaVmos8bdrKEZLFMOVLAMLrwjEsCsLa3 -AgMBAAGjQjBAMA8GA1UdEwEB/wQFMAMBAf8wHQYDVR0OBBYEFEe4zf/lb+74suwv -Tg75JbCOPGvDMA4GA1UdDwEB/wQEAwIBBjANBgkqhkiG9w0BAQsFAAOCAgEAACAj -QTUEkMJAYmDv4jVM1z+s4jSQuKFvdvoWFqRINyzpkMLyPPgKn9iB5btb2iUspKdV -cSQy9sgL8rxq+JOssgfCX5/bzMiKqr5qb+FJEMwx14C7u8jYog5kV+qi9cKpMRXS -IGrs/CIBKM+GuIAeqcwRpTzyFrNHnfzSgCHEy9BHcEGhyoMZCCxt8l13nIoUE9Q2 -HJLw5QY33KbmkJs4j1xrG0aGQ0JfPgEHU1RdZX33inOhmlRaHylDFCfChQ+1iHsa -O5S3HWCntZznKWlXWpuTekMwGwPXYshApqr8ZORK15FTAaggiG6cX0S5y2CBNOxv -033aSF/rtJC8LakcC6wc1aJoIIAE1vyxjy+7SjENSoYc6+I2KSb12tjE8nVhz36u -dmNKekBlk4f4HoCMhuWG1o8O/FMsYOgWYRqiPkN7zTlgVGr18okmAWiDSKIz6MkE -kbIRNBE+6tBDGR8Dk5AM/1E9V/RBbuHLoL7ryWPNbczk+DaqaJ3tvV2XcEQNtg41 -3OEMXbugUZTLfhbrES+jkkXITHHZvMmZUldGL1DPvTVp9D0VzgalLA8+9oG6lLvD -u79leNKGef9JOxqDDPDeeOzI8k1MGt6CKfjBWtrt7uYnXuhF0J0cUahoq0Tj0Itq -4/g7u9xN12TyUb7mqqta6THuBrxzvxNiCp/HuZc= ------END CERTIFICATE----- - -# Issuer: CN=T-TeleSec GlobalRoot Class 3 O=T-Systems Enterprise Services GmbH OU=T-Systems Trust Center -# Subject: CN=T-TeleSec GlobalRoot Class 3 O=T-Systems Enterprise Services GmbH OU=T-Systems Trust Center -# Label: "T-TeleSec GlobalRoot Class 3" -# Serial: 1 -# MD5 Fingerprint: ca:fb:40:a8:4e:39:92:8a:1d:fe:8e:2f:c4:27:ea:ef -# SHA1 Fingerprint: 55:a6:72:3e:cb:f2:ec:cd:c3:23:74:70:19:9d:2a:be:11:e3:81:d1 -# SHA256 Fingerprint: fd:73:da:d3:1c:64:4f:f1:b4:3b:ef:0c:cd:da:96:71:0b:9c:d9:87:5e:ca:7e:31:70:7a:f3:e9:6d:52:2b:bd ------BEGIN CERTIFICATE----- -MIIDwzCCAqugAwIBAgIBATANBgkqhkiG9w0BAQsFADCBgjELMAkGA1UEBhMCREUx -KzApBgNVBAoMIlQtU3lzdGVtcyBFbnRlcnByaXNlIFNlcnZpY2VzIEdtYkgxHzAd -BgNVBAsMFlQtU3lzdGVtcyBUcnVzdCBDZW50ZXIxJTAjBgNVBAMMHFQtVGVsZVNl -YyBHbG9iYWxSb290IENsYXNzIDMwHhcNMDgxMDAxMTAyOTU2WhcNMzMxMDAxMjM1 -OTU5WjCBgjELMAkGA1UEBhMCREUxKzApBgNVBAoMIlQtU3lzdGVtcyBFbnRlcnBy -aXNlIFNlcnZpY2VzIEdtYkgxHzAdBgNVBAsMFlQtU3lzdGVtcyBUcnVzdCBDZW50 -ZXIxJTAjBgNVBAMMHFQtVGVsZVNlYyBHbG9iYWxSb290IENsYXNzIDMwggEiMA0G -CSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC9dZPwYiJvJK7genasfb3ZJNW4t/zN -8ELg63iIVl6bmlQdTQyK9tPPcPRStdiTBONGhnFBSivwKixVA9ZIw+A5OO3yXDw/ -RLyTPWGrTs0NvvAgJ1gORH8EGoel15YUNpDQSXuhdfsaa3Ox+M6pCSzyU9XDFES4 -hqX2iys52qMzVNn6chr3IhUciJFrf2blw2qAsCTz34ZFiP0Zf3WHHx+xGwpzJFu5 -ZeAsVMhg02YXP+HMVDNzkQI6pn97djmiH5a2OK61yJN0HZ65tOVgnS9W0eDrXltM -EnAMbEQgqxHY9Bn20pxSN+f6tsIxO0rUFJmtxxr1XV/6B7h8DR/Wgx6zAgMBAAGj -QjBAMA8GA1UdEwEB/wQFMAMBAf8wDgYDVR0PAQH/BAQDAgEGMB0GA1UdDgQWBBS1 -A/d2O2GCahKqGFPrAyGUv/7OyjANBgkqhkiG9w0BAQsFAAOCAQEAVj3vlNW92nOy -WL6ukK2YJ5f+AbGwUgC4TeQbIXQbfsDuXmkqJa9c1h3a0nnJ85cp4IaH3gRZD/FZ -1GSFS5mvJQQeyUapl96Cshtwn5z2r3Ex3XsFpSzTucpH9sry9uetuUg/vBa3wW30 -6gmv7PO15wWeph6KU1HWk4HMdJP2udqmJQV0eVp+QD6CSyYRMG7hP0HHRwA11fXT -91Q+gT3aSWqas+8QPebrb9HIIkfLzM8BMZLZGOMivgkeGj5asuRrDFR6fUNOuIml -e9eiPZaGzPImNC1qkp2aGtAw4l1OBLBfiyB+d8E9lYLRRpo7PHi4b6HQDWSieB4p -TpPDpFQUWw== ------END CERTIFICATE----- - -# Issuer: CN=D-TRUST Root Class 3 CA 2 2009 O=D-Trust GmbH -# Subject: CN=D-TRUST Root Class 3 CA 2 2009 O=D-Trust GmbH -# Label: "D-TRUST Root Class 3 CA 2 2009" -# Serial: 623603 -# MD5 Fingerprint: cd:e0:25:69:8d:47:ac:9c:89:35:90:f7:fd:51:3d:2f -# SHA1 Fingerprint: 58:e8:ab:b0:36:15:33:fb:80:f7:9b:1b:6d:29:d3:ff:8d:5f:00:f0 -# SHA256 Fingerprint: 49:e7:a4:42:ac:f0:ea:62:87:05:00:54:b5:25:64:b6:50:e4:f4:9e:42:e3:48:d6:aa:38:e0:39:e9:57:b1:c1 ------BEGIN CERTIFICATE----- -MIIEMzCCAxugAwIBAgIDCYPzMA0GCSqGSIb3DQEBCwUAME0xCzAJBgNVBAYTAkRF -MRUwEwYDVQQKDAxELVRydXN0IEdtYkgxJzAlBgNVBAMMHkQtVFJVU1QgUm9vdCBD -bGFzcyAzIENBIDIgMjAwOTAeFw0wOTExMDUwODM1NThaFw0yOTExMDUwODM1NTha -ME0xCzAJBgNVBAYTAkRFMRUwEwYDVQQKDAxELVRydXN0IEdtYkgxJzAlBgNVBAMM -HkQtVFJVU1QgUm9vdCBDbGFzcyAzIENBIDIgMjAwOTCCASIwDQYJKoZIhvcNAQEB -BQADggEPADCCAQoCggEBANOySs96R+91myP6Oi/WUEWJNTrGa9v+2wBoqOADER03 -UAifTUpolDWzU9GUY6cgVq/eUXjsKj3zSEhQPgrfRlWLJ23DEE0NkVJD2IfgXU42 -tSHKXzlABF9bfsyjxiupQB7ZNoTWSPOSHjRGICTBpFGOShrvUD9pXRl/RcPHAY9R -ySPocq60vFYJfxLLHLGvKZAKyVXMD9O0Gu1HNVpK7ZxzBCHQqr0ME7UAyiZsxGsM -lFqVlNpQmvH/pStmMaTJOKDfHR+4CS7zp+hnUquVH+BGPtikw8paxTGA6Eian5Rp -/hnd2HN8gcqW3o7tszIFZYQ05ub9VxC1X3a/L7AQDcUCAwEAAaOCARowggEWMA8G -A1UdEwEB/wQFMAMBAf8wHQYDVR0OBBYEFP3aFMSfMN4hvR5COfyrYyNJ4PGEMA4G -A1UdDwEB/wQEAwIBBjCB0wYDVR0fBIHLMIHIMIGAoH6gfIZ6bGRhcDovL2RpcmVj -dG9yeS5kLXRydXN0Lm5ldC9DTj1ELVRSVVNUJTIwUm9vdCUyMENsYXNzJTIwMyUy -MENBJTIwMiUyMDIwMDksTz1ELVRydXN0JTIwR21iSCxDPURFP2NlcnRpZmljYXRl -cmV2b2NhdGlvbmxpc3QwQ6BBoD+GPWh0dHA6Ly93d3cuZC10cnVzdC5uZXQvY3Js -L2QtdHJ1c3Rfcm9vdF9jbGFzc18zX2NhXzJfMjAwOS5jcmwwDQYJKoZIhvcNAQEL -BQADggEBAH+X2zDI36ScfSF6gHDOFBJpiBSVYEQBrLLpME+bUMJm2H6NMLVwMeni -acfzcNsgFYbQDfC+rAF1hM5+n02/t2A7nPPKHeJeaNijnZflQGDSNiH+0LS4F9p0 -o3/U37CYAqxva2ssJSRyoWXuJVrl5jLn8t+rSfrzkGkj2wTZ51xY/GXUl77M/C4K -zCUqNQT4YJEVdT1B/yMfGchs64JTBKbkTCJNjYy6zltz7GRUUG3RnFX7acM2w4y8 -PIWmawomDeCTmGCufsYkl4phX5GOZpIJhzbNi5stPvZR1FDUWSi9g/LMKHtThm3Y -Johw1+qRzT65ysCQblrGXnRl11z+o+I= ------END CERTIFICATE----- - -# Issuer: CN=D-TRUST Root Class 3 CA 2 EV 2009 O=D-Trust GmbH -# Subject: CN=D-TRUST Root Class 3 CA 2 EV 2009 O=D-Trust GmbH -# Label: "D-TRUST Root Class 3 CA 2 EV 2009" -# Serial: 623604 -# MD5 Fingerprint: aa:c6:43:2c:5e:2d:cd:c4:34:c0:50:4f:11:02:4f:b6 -# SHA1 Fingerprint: 96:c9:1b:0b:95:b4:10:98:42:fa:d0:d8:22:79:fe:60:fa:b9:16:83 -# SHA256 Fingerprint: ee:c5:49:6b:98:8c:e9:86:25:b9:34:09:2e:ec:29:08:be:d0:b0:f3:16:c2:d4:73:0c:84:ea:f1:f3:d3:48:81 ------BEGIN CERTIFICATE----- -MIIEQzCCAyugAwIBAgIDCYP0MA0GCSqGSIb3DQEBCwUAMFAxCzAJBgNVBAYTAkRF -MRUwEwYDVQQKDAxELVRydXN0IEdtYkgxKjAoBgNVBAMMIUQtVFJVU1QgUm9vdCBD -bGFzcyAzIENBIDIgRVYgMjAwOTAeFw0wOTExMDUwODUwNDZaFw0yOTExMDUwODUw -NDZaMFAxCzAJBgNVBAYTAkRFMRUwEwYDVQQKDAxELVRydXN0IEdtYkgxKjAoBgNV -BAMMIUQtVFJVU1QgUm9vdCBDbGFzcyAzIENBIDIgRVYgMjAwOTCCASIwDQYJKoZI -hvcNAQEBBQADggEPADCCAQoCggEBAJnxhDRwui+3MKCOvXwEz75ivJn9gpfSegpn -ljgJ9hBOlSJzmY3aFS3nBfwZcyK3jpgAvDw9rKFs+9Z5JUut8Mxk2og+KbgPCdM0 -3TP1YtHhzRnp7hhPTFiu4h7WDFsVWtg6uMQYZB7jM7K1iXdODL/ZlGsTl28So/6Z -qQTMFexgaDbtCHu39b+T7WYxg4zGcTSHThfqr4uRjRxWQa4iN1438h3Z0S0NL2lR -p75mpoo6Kr3HGrHhFPC+Oh25z1uxav60sUYgovseO3Dvk5h9jHOW8sXvhXCtKSb8 -HgQ+HKDYD8tSg2J87otTlZCpV6LqYQXY+U3EJ/pure3511H3a6UCAwEAAaOCASQw -ggEgMA8GA1UdEwEB/wQFMAMBAf8wHQYDVR0OBBYEFNOUikxiEyoZLsyvcop9Ntea -HNxnMA4GA1UdDwEB/wQEAwIBBjCB3QYDVR0fBIHVMIHSMIGHoIGEoIGBhn9sZGFw -Oi8vZGlyZWN0b3J5LmQtdHJ1c3QubmV0L0NOPUQtVFJVU1QlMjBSb290JTIwQ2xh -c3MlMjAzJTIwQ0ElMjAyJTIwRVYlMjAyMDA5LE89RC1UcnVzdCUyMEdtYkgsQz1E -RT9jZXJ0aWZpY2F0ZXJldm9jYXRpb25saXN0MEagRKBChkBodHRwOi8vd3d3LmQt -dHJ1c3QubmV0L2NybC9kLXRydXN0X3Jvb3RfY2xhc3NfM19jYV8yX2V2XzIwMDku -Y3JsMA0GCSqGSIb3DQEBCwUAA4IBAQA07XtaPKSUiO8aEXUHL7P+PPoeUSbrh/Yp -3uDx1MYkCenBz1UbtDDZzhr+BlGmFaQt77JLvyAoJUnRpjZ3NOhk31KxEcdzes05 -nsKtjHEh8lprr988TlWvsoRlFIm5d8sqMb7Po23Pb0iUMkZv53GMoKaEGTcH8gNF -CSuGdXzfX2lXANtu2KZyIktQ1HWYVt+3GP9DQ1CuekR78HlR10M9p9OB0/DJT7na -xpeG0ILD5EJt/rDiZE4OJudANCa1CInXCGNjOCd1HjPqbqjdn5lPdE2BiYBL3ZqX -KVwvvoFBuYz/6n1gBp7N1z3TLqMVvKjmJuVvw9y4AyHqnxbxLFS1 ------END CERTIFICATE----- - -# Issuer: CN=CA Disig Root R2 O=Disig a.s. -# Subject: CN=CA Disig Root R2 O=Disig a.s. -# Label: "CA Disig Root R2" -# Serial: 10572350602393338211 -# MD5 Fingerprint: 26:01:fb:d8:27:a7:17:9a:45:54:38:1a:43:01:3b:03 -# SHA1 Fingerprint: b5:61:eb:ea:a4:de:e4:25:4b:69:1a:98:a5:57:47:c2:34:c7:d9:71 -# SHA256 Fingerprint: e2:3d:4a:03:6d:7b:70:e9:f5:95:b1:42:20:79:d2:b9:1e:df:bb:1f:b6:51:a0:63:3e:aa:8a:9d:c5:f8:07:03 ------BEGIN CERTIFICATE----- -MIIFaTCCA1GgAwIBAgIJAJK4iNuwisFjMA0GCSqGSIb3DQEBCwUAMFIxCzAJBgNV -BAYTAlNLMRMwEQYDVQQHEwpCcmF0aXNsYXZhMRMwEQYDVQQKEwpEaXNpZyBhLnMu -MRkwFwYDVQQDExBDQSBEaXNpZyBSb290IFIyMB4XDTEyMDcxOTA5MTUzMFoXDTQy -MDcxOTA5MTUzMFowUjELMAkGA1UEBhMCU0sxEzARBgNVBAcTCkJyYXRpc2xhdmEx -EzARBgNVBAoTCkRpc2lnIGEucy4xGTAXBgNVBAMTEENBIERpc2lnIFJvb3QgUjIw -ggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQCio8QACdaFXS1tFPbCw3Oe -NcJxVX6B+6tGUODBfEl45qt5WDza/3wcn9iXAng+a0EE6UG9vgMsRfYvZNSrXaNH -PWSb6WiaxswbP7q+sos0Ai6YVRn8jG+qX9pMzk0DIaPY0jSTVpbLTAwAFjxfGs3I -x2ymrdMxp7zo5eFm1tL7A7RBZckQrg4FY8aAamkw/dLukO8NJ9+flXP04SXabBbe -QTg06ov80egEFGEtQX6sx3dOy1FU+16SGBsEWmjGycT6txOgmLcRK7fWV8x8nhfR -yyX+hk4kLlYMeE2eARKmK6cBZW58Yh2EhN/qwGu1pSqVg8NTEQxzHQuyRpDRQjrO -QG6Vrf/GlK1ul4SOfW+eioANSW1z4nuSHsPzwfPrLgVv2RvPN3YEyLRa5Beny912 -H9AZdugsBbPWnDTYltxhh5EF5EQIM8HauQhl1K6yNg3ruji6DOWbnuuNZt2Zz9aJ -QfYEkoopKW1rOhzndX0CcQ7zwOe9yxndnWCywmZgtrEE7snmhrmaZkCo5xHtgUUD -i/ZnWejBBhG93c+AAk9lQHhcR1DIm+YfgXvkRKhbhZri3lrVx/k6RGZL5DJUfORs -nLMOPReisjQS1n6yqEm70XooQL6iFh/f5DcfEXP7kAplQ6INfPgGAVUzfbANuPT1 -rqVCV3w2EYx7XsQDnYx5nQIDAQABo0IwQDAPBgNVHRMBAf8EBTADAQH/MA4GA1Ud -DwEB/wQEAwIBBjAdBgNVHQ4EFgQUtZn4r7CU9eMg1gqtzk5WpC5uQu0wDQYJKoZI -hvcNAQELBQADggIBACYGXnDnZTPIgm7ZnBc6G3pmsgH2eDtpXi/q/075KMOYKmFM -tCQSin1tERT3nLXK5ryeJ45MGcipvXrA1zYObYVybqjGom32+nNjf7xueQgcnYqf -GopTpti72TVVsRHFqQOzVju5hJMiXn7B9hJSi+osZ7z+Nkz1uM/Rs0mSO9MpDpkb -lvdhuDvEK7Z4bLQjb/D907JedR+Zlais9trhxTF7+9FGs9K8Z7RiVLoJ92Owk6Ka -+elSLotgEqv89WBW7xBci8QaQtyDW2QOy7W81k/BfDxujRNt+3vrMNDcTa/F1bal -TFtxyegxvug4BkihGuLq0t4SOVga/4AOgnXmt8kHbA7v/zjxmHHEt38OFdAlab0i -nSvtBfZGR6ztwPDUO+Ls7pZbkBNOHlY667DvlruWIxG68kOGdGSVyCh13x01utI3 -gzhTODY7z2zp+WsO0PsE6E9312UBeIYMej4hYvF/Y3EMyZ9E26gnonW+boE+18Dr -G5gPcFw0sorMwIUY6256s/daoQe/qUKS82Ail+QUoQebTnbAjn39pCXHR+3/H3Os -zMOl6W8KjptlwlCFtaOgUxLMVYdh84GuEEZhvUQhuMI9dM9+JDX6HAcOmz0iyu8x -L4ysEr3vQCj8KWefshNPZiTEUxnpHikV7+ZtsH8tZ/3zbBt1RqPlShfppNcL ------END CERTIFICATE----- - -# Issuer: CN=ACCVRAIZ1 O=ACCV OU=PKIACCV -# Subject: CN=ACCVRAIZ1 O=ACCV OU=PKIACCV -# Label: "ACCVRAIZ1" -# Serial: 6828503384748696800 -# MD5 Fingerprint: d0:a0:5a:ee:05:b6:09:94:21:a1:7d:f1:b2:29:82:02 -# SHA1 Fingerprint: 93:05:7a:88:15:c6:4f:ce:88:2f:fa:91:16:52:28:78:bc:53:64:17 -# SHA256 Fingerprint: 9a:6e:c0:12:e1:a7:da:9d:be:34:19:4d:47:8a:d7:c0:db:18:22:fb:07:1d:f1:29:81:49:6e:d1:04:38:41:13 ------BEGIN CERTIFICATE----- -MIIH0zCCBbugAwIBAgIIXsO3pkN/pOAwDQYJKoZIhvcNAQEFBQAwQjESMBAGA1UE -AwwJQUNDVlJBSVoxMRAwDgYDVQQLDAdQS0lBQ0NWMQ0wCwYDVQQKDARBQ0NWMQsw -CQYDVQQGEwJFUzAeFw0xMTA1MDUwOTM3MzdaFw0zMDEyMzEwOTM3MzdaMEIxEjAQ -BgNVBAMMCUFDQ1ZSQUlaMTEQMA4GA1UECwwHUEtJQUNDVjENMAsGA1UECgwEQUND -VjELMAkGA1UEBhMCRVMwggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQCb -qau/YUqXry+XZpp0X9DZlv3P4uRm7x8fRzPCRKPfmt4ftVTdFXxpNRFvu8gMjmoY -HtiP2Ra8EEg2XPBjs5BaXCQ316PWywlxufEBcoSwfdtNgM3802/J+Nq2DoLSRYWo -G2ioPej0RGy9ocLLA76MPhMAhN9KSMDjIgro6TenGEyxCQ0jVn8ETdkXhBilyNpA -lHPrzg5XPAOBOp0KoVdDaaxXbXmQeOW1tDvYvEyNKKGno6e6Ak4l0Squ7a4DIrhr -IA8wKFSVf+DuzgpmndFALW4ir50awQUZ0m/A8p/4e7MCQvtQqR0tkw8jq8bBD5L/ -0KIV9VMJcRz/RROE5iZe+OCIHAr8Fraocwa48GOEAqDGWuzndN9wrqODJerWx5eH -k6fGioozl2A3ED6XPm4pFdahD9GILBKfb6qkxkLrQaLjlUPTAYVtjrs78yM2x/47 -4KElB0iryYl0/wiPgL/AlmXz7uxLaL2diMMxs0Dx6M/2OLuc5NF/1OVYm3z61PMO -m3WR5LpSLhl+0fXNWhn8ugb2+1KoS5kE3fj5tItQo05iifCHJPqDQsGH+tUtKSpa -cXpkatcnYGMN285J9Y0fkIkyF/hzQ7jSWpOGYdbhdQrqeWZ2iE9x6wQl1gpaepPl -uUsXQA+xtrn13k/c4LOsOxFwYIRKQ26ZIMApcQrAZQIDAQABo4ICyzCCAscwfQYI -KwYBBQUHAQEEcTBvMEwGCCsGAQUFBzAChkBodHRwOi8vd3d3LmFjY3YuZXMvZmls -ZWFkbWluL0FyY2hpdm9zL2NlcnRpZmljYWRvcy9yYWl6YWNjdjEuY3J0MB8GCCsG -AQUFBzABhhNodHRwOi8vb2NzcC5hY2N2LmVzMB0GA1UdDgQWBBTSh7Tj3zcnk1X2 -VuqB5TbMjB4/vTAPBgNVHRMBAf8EBTADAQH/MB8GA1UdIwQYMBaAFNKHtOPfNyeT -VfZW6oHlNsyMHj+9MIIBcwYDVR0gBIIBajCCAWYwggFiBgRVHSAAMIIBWDCCASIG -CCsGAQUFBwICMIIBFB6CARAAQQB1AHQAbwByAGkAZABhAGQAIABkAGUAIABDAGUA -cgB0AGkAZgBpAGMAYQBjAGkA8wBuACAAUgBhAO0AegAgAGQAZQAgAGwAYQAgAEEA -QwBDAFYAIAAoAEEAZwBlAG4AYwBpAGEAIABkAGUAIABUAGUAYwBuAG8AbABvAGcA -7QBhACAAeQAgAEMAZQByAHQAaQBmAGkAYwBhAGMAaQDzAG4AIABFAGwAZQBjAHQA -cgDzAG4AaQBjAGEALAAgAEMASQBGACAAUQA0ADYAMAAxADEANQA2AEUAKQAuACAA -QwBQAFMAIABlAG4AIABoAHQAdABwADoALwAvAHcAdwB3AC4AYQBjAGMAdgAuAGUA -czAwBggrBgEFBQcCARYkaHR0cDovL3d3dy5hY2N2LmVzL2xlZ2lzbGFjaW9uX2Mu -aHRtMFUGA1UdHwROMEwwSqBIoEaGRGh0dHA6Ly93d3cuYWNjdi5lcy9maWxlYWRt -aW4vQXJjaGl2b3MvY2VydGlmaWNhZG9zL3JhaXphY2N2MV9kZXIuY3JsMA4GA1Ud -DwEB/wQEAwIBBjAXBgNVHREEEDAOgQxhY2N2QGFjY3YuZXMwDQYJKoZIhvcNAQEF -BQADggIBAJcxAp/n/UNnSEQU5CmH7UwoZtCPNdpNYbdKl02125DgBS4OxnnQ8pdp -D70ER9m+27Up2pvZrqmZ1dM8MJP1jaGo/AaNRPTKFpV8M9xii6g3+CfYCS0b78gU -JyCpZET/LtZ1qmxNYEAZSUNUY9rizLpm5U9EelvZaoErQNV/+QEnWCzI7UiRfD+m -AM/EKXMRNt6GGT6d7hmKG9Ww7Y49nCrADdg9ZuM8Db3VlFzi4qc1GwQA9j9ajepD -vV+JHanBsMyZ4k0ACtrJJ1vnE5Bc5PUzolVt3OAJTS+xJlsndQAJxGJ3KQhfnlms -tn6tn1QwIgPBHnFk/vk4CpYY3QIUrCPLBhwepH2NDd4nQeit2hW3sCPdK6jT2iWH -7ehVRE2I9DZ+hJp4rPcOVkkO1jMl1oRQQmwgEh0q1b688nCBpHBgvgW1m54ERL5h -I6zppSSMEYCUWqKiuUnSwdzRp+0xESyeGabu4VXhwOrPDYTkF7eifKXeVSUG7szA -h1xA2syVP1XgNce4hL60Xc16gwFy7ofmXx2utYXGJt/mwZrpHgJHnyqobalbz+xF -d3+YJ5oyXSrjhO7FmGYvliAd3djDJ9ew+f7Zfc3Qn48LFFhRny+Lwzgt3uiP1o2H -pPVWQxaZLPSkVrQ0uGE3ycJYgBugl6H8WY3pEfbRD0tVNEYqi4Y7 ------END CERTIFICATE----- - -# Issuer: CN=TWCA Global Root CA O=TAIWAN-CA OU=Root CA -# Subject: CN=TWCA Global Root CA O=TAIWAN-CA OU=Root CA -# Label: "TWCA Global Root CA" -# Serial: 3262 -# MD5 Fingerprint: f9:03:7e:cf:e6:9e:3c:73:7a:2a:90:07:69:ff:2b:96 -# SHA1 Fingerprint: 9c:bb:48:53:f6:a4:f6:d3:52:a4:e8:32:52:55:60:13:f5:ad:af:65 -# SHA256 Fingerprint: 59:76:90:07:f7:68:5d:0f:cd:50:87:2f:9f:95:d5:75:5a:5b:2b:45:7d:81:f3:69:2b:61:0a:98:67:2f:0e:1b ------BEGIN CERTIFICATE----- -MIIFQTCCAymgAwIBAgICDL4wDQYJKoZIhvcNAQELBQAwUTELMAkGA1UEBhMCVFcx -EjAQBgNVBAoTCVRBSVdBTi1DQTEQMA4GA1UECxMHUm9vdCBDQTEcMBoGA1UEAxMT -VFdDQSBHbG9iYWwgUm9vdCBDQTAeFw0xMjA2MjcwNjI4MzNaFw0zMDEyMzExNTU5 -NTlaMFExCzAJBgNVBAYTAlRXMRIwEAYDVQQKEwlUQUlXQU4tQ0ExEDAOBgNVBAsT -B1Jvb3QgQ0ExHDAaBgNVBAMTE1RXQ0EgR2xvYmFsIFJvb3QgQ0EwggIiMA0GCSqG -SIb3DQEBAQUAA4ICDwAwggIKAoICAQCwBdvI64zEbooh745NnHEKH1Jw7W2CnJfF -10xORUnLQEK1EjRsGcJ0pDFfhQKX7EMzClPSnIyOt7h52yvVavKOZsTuKwEHktSz -0ALfUPZVr2YOy+BHYC8rMjk1Ujoog/h7FsYYuGLWRyWRzvAZEk2tY/XTP3VfKfCh -MBwqoJimFb3u/Rk28OKRQ4/6ytYQJ0lM793B8YVwm8rqqFpD/G2Gb3PpN0Wp8DbH -zIh1HrtsBv+baz4X7GGqcXzGHaL3SekVtTzWoWH1EfcFbx39Eb7QMAfCKbAJTibc -46KokWofwpFFiFzlmLhxpRUZyXx1EcxwdE8tmx2RRP1WKKD+u4ZqyPpcC1jcxkt2 -yKsi2XMPpfRaAok/T54igu6idFMqPVMnaR1sjjIsZAAmY2E2TqNGtz99sy2sbZCi -laLOz9qC5wc0GZbpuCGqKX6mOL6OKUohZnkfs8O1CWfe1tQHRvMq2uYiN2DLgbYP -oA/pyJV/v1WRBXrPPRXAb94JlAGD1zQbzECl8LibZ9WYkTunhHiVJqRaCPgrdLQA -BDzfuBSO6N+pjWxnkjMdwLfS7JLIvgm/LCkFbwJrnu+8vyq8W8BQj0FwcYeyTbcE -qYSjMq+u7msXi7Kx/mzhkIyIqJdIzshNy/MGz19qCkKxHh53L46g5pIOBvwFItIm -4TFRfTLcDwIDAQABoyMwITAOBgNVHQ8BAf8EBAMCAQYwDwYDVR0TAQH/BAUwAwEB -/zANBgkqhkiG9w0BAQsFAAOCAgEAXzSBdu+WHdXltdkCY4QWwa6gcFGn90xHNcgL -1yg9iXHZqjNB6hQbbCEAwGxCGX6faVsgQt+i0trEfJdLjbDorMjupWkEmQqSpqsn -LhpNgb+E1HAerUf+/UqdM+DyucRFCCEK2mlpc3INvjT+lIutwx4116KD7+U4x6WF -H6vPNOw/KP4M8VeGTslV9xzU2KV9Bnpv1d8Q34FOIWWxtuEXeZVFBs5fzNxGiWNo -RI2T9GRwoD2dKAXDOXC4Ynsg/eTb6QihuJ49CcdP+yz4k3ZB3lLg4VfSnQO8d57+ -nile98FRYB/e2guyLXW3Q0iT5/Z5xoRdgFlglPx4mI88k1HtQJAH32RjJMtOcQWh -15QaiDLxInQirqWm2BJpTGCjAu4r7NRjkgtevi92a6O2JryPA9gK8kxkRr05YuWW -6zRjESjMlfGt7+/cgFhI6Uu46mWs6fyAtbXIRfmswZ/ZuepiiI7E8UuDEq3mi4TW -nsLrgxifarsbJGAzcMzs9zLzXNl5fe+epP7JI8Mk7hWSsT2RTyaGvWZzJBPqpK5j -wa19hAM8EHiGG3njxPPyBJUgriOCxLM6AGK/5jYk4Ve6xx6QddVfP5VhK8E7zeWz -aGHQRiapIVJpLesux+t3zqY6tQMzT3bR51xUAV3LePTJDL/PEo4XLSNolOer/qmy -KwbQBM0= ------END CERTIFICATE----- - -# Issuer: CN=TeliaSonera Root CA v1 O=TeliaSonera -# Subject: CN=TeliaSonera Root CA v1 O=TeliaSonera -# Label: "TeliaSonera Root CA v1" -# Serial: 199041966741090107964904287217786801558 -# MD5 Fingerprint: 37:41:49:1b:18:56:9a:26:f5:ad:c2:66:fb:40:a5:4c -# SHA1 Fingerprint: 43:13:bb:96:f1:d5:86:9b:c1:4e:6a:92:f6:cf:f6:34:69:87:82:37 -# SHA256 Fingerprint: dd:69:36:fe:21:f8:f0:77:c1:23:a1:a5:21:c1:22:24:f7:22:55:b7:3e:03:a7:26:06:93:e8:a2:4b:0f:a3:89 ------BEGIN CERTIFICATE----- -MIIFODCCAyCgAwIBAgIRAJW+FqD3LkbxezmCcvqLzZYwDQYJKoZIhvcNAQEFBQAw -NzEUMBIGA1UECgwLVGVsaWFTb25lcmExHzAdBgNVBAMMFlRlbGlhU29uZXJhIFJv -b3QgQ0EgdjEwHhcNMDcxMDE4MTIwMDUwWhcNMzIxMDE4MTIwMDUwWjA3MRQwEgYD -VQQKDAtUZWxpYVNvbmVyYTEfMB0GA1UEAwwWVGVsaWFTb25lcmEgUm9vdCBDQSB2 -MTCCAiIwDQYJKoZIhvcNAQEBBQADggIPADCCAgoCggIBAMK+6yfwIaPzaSZVfp3F -VRaRXP3vIb9TgHot0pGMYzHw7CTww6XScnwQbfQ3t+XmfHnqjLWCi65ItqwA3GV1 -7CpNX8GH9SBlK4GoRz6JI5UwFpB/6FcHSOcZrr9FZ7E3GwYq/t75rH2D+1665I+X -Z75Ljo1kB1c4VWk0Nj0TSO9P4tNmHqTPGrdeNjPUtAa9GAH9d4RQAEX1jF3oI7x+ -/jXh7VB7qTCNGdMJjmhnXb88lxhTuylixcpecsHHltTbLaC0H2kD7OriUPEMPPCs -81Mt8Bz17Ww5OXOAFshSsCPN4D7c3TxHoLs1iuKYaIu+5b9y7tL6pe0S7fyYGKkm -dtwoSxAgHNN/Fnct7W+A90m7UwW7XWjH1Mh1Fj+JWov3F0fUTPHSiXk+TT2YqGHe -Oh7S+F4D4MHJHIzTjU3TlTazN19jY5szFPAtJmtTfImMMsJu7D0hADnJoWjiUIMu -sDor8zagrC/kb2HCUQk5PotTubtn2txTuXZZNp1D5SDgPTJghSJRt8czu90VL6R4 -pgd7gUY2BIbdeTXHlSw7sKMXNeVzH7RcWe/a6hBle3rQf5+ztCo3O3CLm1u5K7fs -slESl1MpWtTwEhDcTwK7EpIvYtQ/aUN8Ddb8WHUBiJ1YFkveupD/RwGJBmr2X7KQ -arMCpgKIv7NHfirZ1fpoeDVNAgMBAAGjPzA9MA8GA1UdEwEB/wQFMAMBAf8wCwYD -VR0PBAQDAgEGMB0GA1UdDgQWBBTwj1k4ALP1j5qWDNXr+nuqF+gTEjANBgkqhkiG -9w0BAQUFAAOCAgEAvuRcYk4k9AwI//DTDGjkk0kiP0Qnb7tt3oNmzqjMDfz1mgbl -dxSR651Be5kqhOX//CHBXfDkH1e3damhXwIm/9fH907eT/j3HEbAek9ALCI18Bmx -0GtnLLCo4MBANzX2hFxc469CeP6nyQ1Q6g2EdvZR74NTxnr/DlZJLo961gzmJ1Tj -TQpgcmLNkQfWpb/ImWvtxBnmq0wROMVvMeJuScg/doAmAyYp4Db29iBT4xdwNBed -Y2gea+zDTYa4EzAvXUYNR0PVG6pZDrlcjQZIrXSHX8f8MVRBE+LHIQ6e4B4N4cB7 -Q4WQxYpYxmUKeFfyxiMPAdkgS94P+5KFdSpcc41teyWRyu5FrgZLAMzTsVlQ2jqI -OylDRl6XK1TOU2+NSueW+r9xDkKLfP0ooNBIytrEgUy7onOTJsjrDNYmiLbAJM+7 -vVvrdX3pCI6GMyx5dwlppYn8s3CQh3aP0yK7Qs69cwsgJirQmz1wHiRszYd2qReW -t88NkvuOGKmYSdGe/mBEciG5Ge3C9THxOUiIkCR1VBatzvT4aRRkOfujuLpwQMcn -HL/EVlP6Y2XQ8xwOFvVrhlhNGNTkDY6lnVuR3HYkUD/GKvvZt5y11ubQ2egZixVx -SK236thZiNSQvxaz2emsWWFUyBy6ysHK4bkgTI86k4mloMy/0/Z1pHWWbVY= ------END CERTIFICATE----- - -# Issuer: CN=T-TeleSec GlobalRoot Class 2 O=T-Systems Enterprise Services GmbH OU=T-Systems Trust Center -# Subject: CN=T-TeleSec GlobalRoot Class 2 O=T-Systems Enterprise Services GmbH OU=T-Systems Trust Center -# Label: "T-TeleSec GlobalRoot Class 2" -# Serial: 1 -# MD5 Fingerprint: 2b:9b:9e:e4:7b:6c:1f:00:72:1a:cc:c1:77:79:df:6a -# SHA1 Fingerprint: 59:0d:2d:7d:88:4f:40:2e:61:7e:a5:62:32:17:65:cf:17:d8:94:e9 -# SHA256 Fingerprint: 91:e2:f5:78:8d:58:10:eb:a7:ba:58:73:7d:e1:54:8a:8e:ca:cd:01:45:98:bc:0b:14:3e:04:1b:17:05:25:52 ------BEGIN CERTIFICATE----- -MIIDwzCCAqugAwIBAgIBATANBgkqhkiG9w0BAQsFADCBgjELMAkGA1UEBhMCREUx -KzApBgNVBAoMIlQtU3lzdGVtcyBFbnRlcnByaXNlIFNlcnZpY2VzIEdtYkgxHzAd -BgNVBAsMFlQtU3lzdGVtcyBUcnVzdCBDZW50ZXIxJTAjBgNVBAMMHFQtVGVsZVNl -YyBHbG9iYWxSb290IENsYXNzIDIwHhcNMDgxMDAxMTA0MDE0WhcNMzMxMDAxMjM1 -OTU5WjCBgjELMAkGA1UEBhMCREUxKzApBgNVBAoMIlQtU3lzdGVtcyBFbnRlcnBy -aXNlIFNlcnZpY2VzIEdtYkgxHzAdBgNVBAsMFlQtU3lzdGVtcyBUcnVzdCBDZW50 -ZXIxJTAjBgNVBAMMHFQtVGVsZVNlYyBHbG9iYWxSb290IENsYXNzIDIwggEiMA0G -CSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCqX9obX+hzkeXaXPSi5kfl82hVYAUd -AqSzm1nzHoqvNK38DcLZSBnuaY/JIPwhqgcZ7bBcrGXHX+0CfHt8LRvWurmAwhiC -FoT6ZrAIxlQjgeTNuUk/9k9uN0goOA/FvudocP05l03Sx5iRUKrERLMjfTlH6VJi -1hKTXrcxlkIF+3anHqP1wvzpesVsqXFP6st4vGCvx9702cu+fjOlbpSD8DT6Iavq -jnKgP6TeMFvvhk1qlVtDRKgQFRzlAVfFmPHmBiiRqiDFt1MmUUOyCxGVWOHAD3bZ -wI18gfNycJ5v/hqO2V81xrJvNHy+SE/iWjnX2J14np+GPgNeGYtEotXHAgMBAAGj -QjBAMA8GA1UdEwEB/wQFMAMBAf8wDgYDVR0PAQH/BAQDAgEGMB0GA1UdDgQWBBS/ -WSA2AHmgoCJrjNXyYdK4LMuCSjANBgkqhkiG9w0BAQsFAAOCAQEAMQOiYQsfdOhy -NsZt+U2e+iKo4YFWz827n+qrkRk4r6p8FU3ztqONpfSO9kSpp+ghla0+AGIWiPAC -uvxhI+YzmzB6azZie60EI4RYZeLbK4rnJVM3YlNfvNoBYimipidx5joifsFvHZVw -IEoHNN/q/xWA5brXethbdXwFeilHfkCoMRN3zUA7tFFHei4R40cR3p1m0IvVVGb6 -g1XqfMIpiRvpb7PO4gWEyS8+eIVibslfwXhjdFjASBgMmTnrpMwatXlajRWc2BQN -9noHV8cigwUtPJslJj0Ys6lDfMjIq2SPDqO/nBudMNva0Bkuqjzx+zOAduTNrRlP -BSeOE6Fuwg== ------END CERTIFICATE----- - -# Issuer: CN=Atos TrustedRoot 2011 O=Atos -# Subject: CN=Atos TrustedRoot 2011 O=Atos -# Label: "Atos TrustedRoot 2011" -# Serial: 6643877497813316402 -# MD5 Fingerprint: ae:b9:c4:32:4b:ac:7f:5d:66:cc:77:94:bb:2a:77:56 -# SHA1 Fingerprint: 2b:b1:f5:3e:55:0c:1d:c5:f1:d4:e6:b7:6a:46:4b:55:06:02:ac:21 -# SHA256 Fingerprint: f3:56:be:a2:44:b7:a9:1e:b3:5d:53:ca:9a:d7:86:4a:ce:01:8e:2d:35:d5:f8:f9:6d:df:68:a6:f4:1a:a4:74 ------BEGIN CERTIFICATE----- -MIIDdzCCAl+gAwIBAgIIXDPLYixfszIwDQYJKoZIhvcNAQELBQAwPDEeMBwGA1UE -AwwVQXRvcyBUcnVzdGVkUm9vdCAyMDExMQ0wCwYDVQQKDARBdG9zMQswCQYDVQQG -EwJERTAeFw0xMTA3MDcxNDU4MzBaFw0zMDEyMzEyMzU5NTlaMDwxHjAcBgNVBAMM -FUF0b3MgVHJ1c3RlZFJvb3QgMjAxMTENMAsGA1UECgwEQXRvczELMAkGA1UEBhMC -REUwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCVhTuXbyo7LjvPpvMp -Nb7PGKw+qtn4TaA+Gke5vJrf8v7MPkfoepbCJI419KkM/IL9bcFyYie96mvr54rM -VD6QUM+A1JX76LWC1BTFtqlVJVfbsVD2sGBkWXppzwO3bw2+yj5vdHLqqjAqc2K+ -SZFhyBH+DgMq92og3AIVDV4VavzjgsG1xZ1kCWyjWZgHJ8cblithdHFsQ/H3NYkQ -4J7sVaE3IqKHBAUsR320HLliKWYoyrfhk/WklAOZuXCFteZI6o1Q/NnezG8HDt0L -cp2AMBYHlT8oDv3FdU9T1nSatCQujgKRz3bFmx5VdJx4IbHwLfELn8LVlhgf8FQi -eowHAgMBAAGjfTB7MB0GA1UdDgQWBBSnpQaxLKYJYO7Rl+lwrrw7GWzbITAPBgNV -HRMBAf8EBTADAQH/MB8GA1UdIwQYMBaAFKelBrEspglg7tGX6XCuvDsZbNshMBgG -A1UdIAQRMA8wDQYLKwYBBAGwLQMEAQEwDgYDVR0PAQH/BAQDAgGGMA0GCSqGSIb3 -DQEBCwUAA4IBAQAmdzTblEiGKkGdLD4GkGDEjKwLVLgfuXvTBznk+j57sj1O7Z8j -vZfza1zv7v1Apt+hk6EKhqzvINB5Ab149xnYJDE0BAGmuhWawyfc2E8PzBhj/5kP -DpFrdRbhIfzYJsdHt6bPWHJxfrrhTZVHO8mvbaG0weyJ9rQPOLXiZNwlz6bb65pc -maHFCN795trV1lpFDMS3wrUU77QR/w4VtfX128a961qn8FYiqTxlVMYVqL2Gns2D -lmh6cYGJ4Qvh6hEbaAjMaZ7snkGeRDImeuKHCnE96+RapNLbxc3G3mB/ufNPRJLv -KrcYPqcZ2Qt9sTdBQrC6YB3y/gkRsPCHe6ed ------END CERTIFICATE----- - -# Issuer: CN=QuoVadis Root CA 1 G3 O=QuoVadis Limited -# Subject: CN=QuoVadis Root CA 1 G3 O=QuoVadis Limited -# Label: "QuoVadis Root CA 1 G3" -# Serial: 687049649626669250736271037606554624078720034195 -# MD5 Fingerprint: a4:bc:5b:3f:fe:37:9a:fa:64:f0:e2:fa:05:3d:0b:ab -# SHA1 Fingerprint: 1b:8e:ea:57:96:29:1a:c9:39:ea:b8:0a:81:1a:73:73:c0:93:79:67 -# SHA256 Fingerprint: 8a:86:6f:d1:b2:76:b5:7e:57:8e:92:1c:65:82:8a:2b:ed:58:e9:f2:f2:88:05:41:34:b7:f1:f4:bf:c9:cc:74 ------BEGIN CERTIFICATE----- -MIIFYDCCA0igAwIBAgIUeFhfLq0sGUvjNwc1NBMotZbUZZMwDQYJKoZIhvcNAQEL -BQAwSDELMAkGA1UEBhMCQk0xGTAXBgNVBAoTEFF1b1ZhZGlzIExpbWl0ZWQxHjAc -BgNVBAMTFVF1b1ZhZGlzIFJvb3QgQ0EgMSBHMzAeFw0xMjAxMTIxNzI3NDRaFw00 -MjAxMTIxNzI3NDRaMEgxCzAJBgNVBAYTAkJNMRkwFwYDVQQKExBRdW9WYWRpcyBM -aW1pdGVkMR4wHAYDVQQDExVRdW9WYWRpcyBSb290IENBIDEgRzMwggIiMA0GCSqG -SIb3DQEBAQUAA4ICDwAwggIKAoICAQCgvlAQjunybEC0BJyFuTHK3C3kEakEPBtV -wedYMB0ktMPvhd6MLOHBPd+C5k+tR4ds7FtJwUrVu4/sh6x/gpqG7D0DmVIB0jWe -rNrwU8lmPNSsAgHaJNM7qAJGr6Qc4/hzWHa39g6QDbXwz8z6+cZM5cOGMAqNF341 -68Xfuw6cwI2H44g4hWf6Pser4BOcBRiYz5P1sZK0/CPTz9XEJ0ngnjybCKOLXSoh -4Pw5qlPafX7PGglTvF0FBM+hSo+LdoINofjSxxR3W5A2B4GbPgb6Ul5jxaYA/qXp -UhtStZI5cgMJYr2wYBZupt0lwgNm3fME0UDiTouG9G/lg6AnhF4EwfWQvTA9xO+o -abw4m6SkltFi2mnAAZauy8RRNOoMqv8hjlmPSlzkYZqn0ukqeI1RPToV7qJZjqlc -3sX5kCLliEVx3ZGZbHqfPT2YfF72vhZooF6uCyP8Wg+qInYtyaEQHeTTRCOQiJ/G -KubX9ZqzWB4vMIkIG1SitZgj7Ah3HJVdYdHLiZxfokqRmu8hqkkWCKi9YSgxyXSt -hfbZxbGL0eUQMk1fiyA6PEkfM4VZDdvLCXVDaXP7a3F98N/ETH3Goy7IlXnLc6KO -Tk0k+17kBL5yG6YnLUlamXrXXAkgt3+UuU/xDRxeiEIbEbfnkduebPRq34wGmAOt -zCjvpUfzUwIDAQABo0IwQDAPBgNVHRMBAf8EBTADAQH/MA4GA1UdDwEB/wQEAwIB -BjAdBgNVHQ4EFgQUo5fW816iEOGrRZ88F2Q87gFwnMwwDQYJKoZIhvcNAQELBQAD -ggIBABj6W3X8PnrHX3fHyt/PX8MSxEBd1DKquGrX1RUVRpgjpeaQWxiZTOOtQqOC -MTaIzen7xASWSIsBx40Bz1szBpZGZnQdT+3Btrm0DWHMY37XLneMlhwqI2hrhVd2 -cDMT/uFPpiN3GPoajOi9ZcnPP/TJF9zrx7zABC4tRi9pZsMbj/7sPtPKlL92CiUN -qXsCHKnQO18LwIE6PWThv6ctTr1NxNgpxiIY0MWscgKCP6o6ojoilzHdCGPDdRS5 -YCgtW2jgFqlmgiNR9etT2DGbe+m3nUvriBbP+V04ikkwj+3x6xn0dxoxGE1nVGwv -b2X52z3sIexe9PSLymBlVNFxZPT5pqOBMzYzcfCkeF9OrYMh3jRJjehZrJ3ydlo2 -8hP0r+AJx2EqbPfgna67hkooby7utHnNkDPDs3b69fBsnQGQ+p6Q9pxyz0fawx/k -NSBT8lTR32GDpgLiJTjehTItXnOQUl1CxM49S+H5GYQd1aJQzEH7QRTDvdbJWqNj -ZgKAvQU6O0ec7AAmTPWIUb+oI38YB7AL7YsmoWTTYUrrXJ/es69nA7Mf3W1daWhp -q1467HxpvMc7hU6eFbm0FU/DlXpY18ls6Wy58yljXrQs8C097Vpl4KlbQMJImYFt -nh8GKjwStIsPm6Ik8KaN1nrgS7ZklmOVhMJKzRwuJIczYOXD ------END CERTIFICATE----- - -# Issuer: CN=QuoVadis Root CA 2 G3 O=QuoVadis Limited -# Subject: CN=QuoVadis Root CA 2 G3 O=QuoVadis Limited -# Label: "QuoVadis Root CA 2 G3" -# Serial: 390156079458959257446133169266079962026824725800 -# MD5 Fingerprint: af:0c:86:6e:bf:40:2d:7f:0b:3e:12:50:ba:12:3d:06 -# SHA1 Fingerprint: 09:3c:61:f3:8b:8b:dc:7d:55:df:75:38:02:05:00:e1:25:f5:c8:36 -# SHA256 Fingerprint: 8f:e4:fb:0a:f9:3a:4d:0d:67:db:0b:eb:b2:3e:37:c7:1b:f3:25:dc:bc:dd:24:0e:a0:4d:af:58:b4:7e:18:40 ------BEGIN CERTIFICATE----- -MIIFYDCCA0igAwIBAgIURFc0JFuBiZs18s64KztbpybwdSgwDQYJKoZIhvcNAQEL -BQAwSDELMAkGA1UEBhMCQk0xGTAXBgNVBAoTEFF1b1ZhZGlzIExpbWl0ZWQxHjAc -BgNVBAMTFVF1b1ZhZGlzIFJvb3QgQ0EgMiBHMzAeFw0xMjAxMTIxODU5MzJaFw00 -MjAxMTIxODU5MzJaMEgxCzAJBgNVBAYTAkJNMRkwFwYDVQQKExBRdW9WYWRpcyBM -aW1pdGVkMR4wHAYDVQQDExVRdW9WYWRpcyBSb290IENBIDIgRzMwggIiMA0GCSqG -SIb3DQEBAQUAA4ICDwAwggIKAoICAQChriWyARjcV4g/Ruv5r+LrI3HimtFhZiFf -qq8nUeVuGxbULX1QsFN3vXg6YOJkApt8hpvWGo6t/x8Vf9WVHhLL5hSEBMHfNrMW -n4rjyduYNM7YMxcoRvynyfDStNVNCXJJ+fKH46nafaF9a7I6JaltUkSs+L5u+9ym -c5GQYaYDFCDy54ejiK2toIz/pgslUiXnFgHVy7g1gQyjO/Dh4fxaXc6AcW34Sas+ -O7q414AB+6XrW7PFXmAqMaCvN+ggOp+oMiwMzAkd056OXbxMmO7FGmh77FOm6RQ1 -o9/NgJ8MSPsc9PG/Srj61YxxSscfrf5BmrODXfKEVu+lV0POKa2Mq1W/xPtbAd0j -IaFYAI7D0GoT7RPjEiuA3GfmlbLNHiJuKvhB1PLKFAeNilUSxmn1uIZoL1NesNKq -IcGY5jDjZ1XHm26sGahVpkUG0CM62+tlXSoREfA7T8pt9DTEceT/AFr2XK4jYIVz -8eQQsSWu1ZK7E8EM4DnatDlXtas1qnIhO4M15zHfeiFuuDIIfR0ykRVKYnLP43eh -vNURG3YBZwjgQQvD6xVu+KQZ2aKrr+InUlYrAoosFCT5v0ICvybIxo/gbjh9Uy3l -7ZizlWNof/k19N+IxWA1ksB8aRxhlRbQ694Lrz4EEEVlWFA4r0jyWbYW8jwNkALG -cC4BrTwV1wIDAQABo0IwQDAPBgNVHRMBAf8EBTADAQH/MA4GA1UdDwEB/wQEAwIB -BjAdBgNVHQ4EFgQU7edvdlq/YOxJW8ald7tyFnGbxD0wDQYJKoZIhvcNAQELBQAD -ggIBAJHfgD9DCX5xwvfrs4iP4VGyvD11+ShdyLyZm3tdquXK4Qr36LLTn91nMX66 -AarHakE7kNQIXLJgapDwyM4DYvmL7ftuKtwGTTwpD4kWilhMSA/ohGHqPHKmd+RC -roijQ1h5fq7KpVMNqT1wvSAZYaRsOPxDMuHBR//47PERIjKWnML2W2mWeyAMQ0Ga -W/ZZGYjeVYg3UQt4XAoeo0L9x52ID8DyeAIkVJOviYeIyUqAHerQbj5hLja7NQ4n -lv1mNDthcnPxFlxHBlRJAHpYErAK74X9sbgzdWqTHBLmYF5vHX/JHyPLhGGfHoJE -+V+tYlUkmlKY7VHnoX6XOuYvHxHaU4AshZ6rNRDbIl9qxV6XU/IyAgkwo1jwDQHV -csaxfGl7w/U2Rcxhbl5MlMVerugOXou/983g7aEOGzPuVBj+D77vfoRrQ+NwmNtd -dbINWQeFFSM51vHfqSYP1kjHs6Yi9TM3WpVHn3u6GBVv/9YUZINJ0gpnIdsPNWNg -KCLjsZWDzYWm3S8P52dSbrsvhXz1SnPnxT7AvSESBT/8twNJAlvIJebiVDj1eYeM -HVOyToV7BjjHLPj4sHKNJeV3UvQDHEimUF+IIDBu8oJDqz2XhOdT+yHBTw8imoa4 -WSr2Rz0ZiC3oheGe7IUIarFsNMkd7EgrO3jtZsSOeWmD3n+M ------END CERTIFICATE----- - -# Issuer: CN=QuoVadis Root CA 3 G3 O=QuoVadis Limited -# Subject: CN=QuoVadis Root CA 3 G3 O=QuoVadis Limited -# Label: "QuoVadis Root CA 3 G3" -# Serial: 268090761170461462463995952157327242137089239581 -# MD5 Fingerprint: df:7d:b9:ad:54:6f:68:a1:df:89:57:03:97:43:b0:d7 -# SHA1 Fingerprint: 48:12:bd:92:3c:a8:c4:39:06:e7:30:6d:27:96:e6:a4:cf:22:2e:7d -# SHA256 Fingerprint: 88:ef:81:de:20:2e:b0:18:45:2e:43:f8:64:72:5c:ea:5f:bd:1f:c2:d9:d2:05:73:07:09:c5:d8:b8:69:0f:46 ------BEGIN CERTIFICATE----- -MIIFYDCCA0igAwIBAgIULvWbAiin23r/1aOp7r0DoM8Sah0wDQYJKoZIhvcNAQEL -BQAwSDELMAkGA1UEBhMCQk0xGTAXBgNVBAoTEFF1b1ZhZGlzIExpbWl0ZWQxHjAc -BgNVBAMTFVF1b1ZhZGlzIFJvb3QgQ0EgMyBHMzAeFw0xMjAxMTIyMDI2MzJaFw00 -MjAxMTIyMDI2MzJaMEgxCzAJBgNVBAYTAkJNMRkwFwYDVQQKExBRdW9WYWRpcyBM -aW1pdGVkMR4wHAYDVQQDExVRdW9WYWRpcyBSb290IENBIDMgRzMwggIiMA0GCSqG -SIb3DQEBAQUAA4ICDwAwggIKAoICAQCzyw4QZ47qFJenMioKVjZ/aEzHs286IxSR -/xl/pcqs7rN2nXrpixurazHb+gtTTK/FpRp5PIpM/6zfJd5O2YIyC0TeytuMrKNu -FoM7pmRLMon7FhY4futD4tN0SsJiCnMK3UmzV9KwCoWdcTzeo8vAMvMBOSBDGzXR -U7Ox7sWTaYI+FrUoRqHe6okJ7UO4BUaKhvVZR74bbwEhELn9qdIoyhA5CcoTNs+c -ra1AdHkrAj80//ogaX3T7mH1urPnMNA3I4ZyYUUpSFlob3emLoG+B01vr87ERROR -FHAGjx+f+IdpsQ7vw4kZ6+ocYfx6bIrc1gMLnia6Et3UVDmrJqMz6nWB2i3ND0/k -A9HvFZcba5DFApCTZgIhsUfei5pKgLlVj7WiL8DWM2fafsSntARE60f75li59wzw -eyuxwHApw0BiLTtIadwjPEjrewl5qW3aqDCYz4ByA4imW0aucnl8CAMhZa634Ryl -sSqiMd5mBPfAdOhx3v89WcyWJhKLhZVXGqtrdQtEPREoPHtht+KPZ0/l7DxMYIBp -VzgeAVuNVejH38DMdyM0SXV89pgR6y3e7UEuFAUCf+D+IOs15xGsIs5XPd7JMG0Q -A4XN8f+MFrXBsj6IbGB/kE+V9/YtrQE5BwT6dYB9v0lQ7e/JxHwc64B+27bQ3RP+ -ydOc17KXqQIDAQABo0IwQDAPBgNVHRMBAf8EBTADAQH/MA4GA1UdDwEB/wQEAwIB -BjAdBgNVHQ4EFgQUxhfQvKjqAkPyGwaZXSuQILnXnOQwDQYJKoZIhvcNAQELBQAD -ggIBADRh2Va1EodVTd2jNTFGu6QHcrxfYWLopfsLN7E8trP6KZ1/AvWkyaiTt3px -KGmPc+FSkNrVvjrlt3ZqVoAh313m6Tqe5T72omnHKgqwGEfcIHB9UqM+WXzBusnI -FUBhynLWcKzSt/Ac5IYp8M7vaGPQtSCKFWGafoaYtMnCdvvMujAWzKNhxnQT5Wvv -oxXqA/4Ti2Tk08HS6IT7SdEQTXlm66r99I0xHnAUrdzeZxNMgRVhvLfZkXdxGYFg -u/BYpbWcC/ePIlUnwEsBbTuZDdQdm2NnL9DuDcpmvJRPpq3t/O5jrFc/ZSXPsoaP -0Aj/uHYUbt7lJ+yreLVTubY/6CD50qi+YUbKh4yE8/nxoGibIh6BJpsQBJFxwAYf -3KDTuVan45gtf4Od34wrnDKOMpTwATwiKp9Dwi7DmDkHOHv8XgBCH/MyJnmDhPbl -8MFREsALHgQjDFSlTC9JxUrRtm5gDWv8a4uFJGS3iQ6rJUdbPM9+Sb3H6QrG2vd+ -DhcI00iX0HGS8A85PjRqHH3Y8iKuu2n0M7SmSFXRDw4m6Oy2Cy2nhTXN/VnIn9HN -PlopNLk9hM6xZdRZkZFWdSHBd575euFgndOtBBj0fOtek49TSiIp+EgrPk2GrFt/ -ywaZWWDYWGWVjUTR939+J399roD1B0y2PpxxVJkES/1Y+Zj0 ------END CERTIFICATE----- - -# Issuer: CN=DigiCert Assured ID Root G2 O=DigiCert Inc OU=www.digicert.com -# Subject: CN=DigiCert Assured ID Root G2 O=DigiCert Inc OU=www.digicert.com -# Label: "DigiCert Assured ID Root G2" -# Serial: 15385348160840213938643033620894905419 -# MD5 Fingerprint: 92:38:b9:f8:63:24:82:65:2c:57:33:e6:fe:81:8f:9d -# SHA1 Fingerprint: a1:4b:48:d9:43:ee:0a:0e:40:90:4f:3c:e0:a4:c0:91:93:51:5d:3f -# SHA256 Fingerprint: 7d:05:eb:b6:82:33:9f:8c:94:51:ee:09:4e:eb:fe:fa:79:53:a1:14:ed:b2:f4:49:49:45:2f:ab:7d:2f:c1:85 ------BEGIN CERTIFICATE----- -MIIDljCCAn6gAwIBAgIQC5McOtY5Z+pnI7/Dr5r0SzANBgkqhkiG9w0BAQsFADBl -MQswCQYDVQQGEwJVUzEVMBMGA1UEChMMRGlnaUNlcnQgSW5jMRkwFwYDVQQLExB3 -d3cuZGlnaWNlcnQuY29tMSQwIgYDVQQDExtEaWdpQ2VydCBBc3N1cmVkIElEIFJv -b3QgRzIwHhcNMTMwODAxMTIwMDAwWhcNMzgwMTE1MTIwMDAwWjBlMQswCQYDVQQG -EwJVUzEVMBMGA1UEChMMRGlnaUNlcnQgSW5jMRkwFwYDVQQLExB3d3cuZGlnaWNl -cnQuY29tMSQwIgYDVQQDExtEaWdpQ2VydCBBc3N1cmVkIElEIFJvb3QgRzIwggEi -MA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDZ5ygvUj82ckmIkzTz+GoeMVSA -n61UQbVH35ao1K+ALbkKz3X9iaV9JPrjIgwrvJUXCzO/GU1BBpAAvQxNEP4Htecc -biJVMWWXvdMX0h5i89vqbFCMP4QMls+3ywPgym2hFEwbid3tALBSfK+RbLE4E9Hp -EgjAALAcKxHad3A2m67OeYfcgnDmCXRwVWmvo2ifv922ebPynXApVfSr/5Vh88lA -bx3RvpO704gqu52/clpWcTs/1PPRCv4o76Pu2ZmvA9OPYLfykqGxvYmJHzDNw6Yu -YjOuFgJ3RFrngQo8p0Quebg/BLxcoIfhG69Rjs3sLPr4/m3wOnyqi+RnlTGNAgMB -AAGjQjBAMA8GA1UdEwEB/wQFMAMBAf8wDgYDVR0PAQH/BAQDAgGGMB0GA1UdDgQW -BBTOw0q5mVXyuNtgv6l+vVa1lzan1jANBgkqhkiG9w0BAQsFAAOCAQEAyqVVjOPI -QW5pJ6d1Ee88hjZv0p3GeDgdaZaikmkuOGybfQTUiaWxMTeKySHMq2zNixya1r9I -0jJmwYrA8y8678Dj1JGG0VDjA9tzd29KOVPt3ibHtX2vK0LRdWLjSisCx1BL4Gni -lmwORGYQRI+tBev4eaymG+g3NJ1TyWGqolKvSnAWhsI6yLETcDbYz+70CjTVW0z9 -B5yiutkBclzzTcHdDrEcDcRjvq30FPuJ7KJBDkzMyFdA0G4Dqs0MjomZmWzwPDCv -ON9vvKO+KSAnq3T/EyJ43pdSVR6DtVQgA+6uwE9W3jfMw3+qBCe703e4YtsXfJwo -IhNzbM8m9Yop5w== ------END CERTIFICATE----- - -# Issuer: CN=DigiCert Assured ID Root G3 O=DigiCert Inc OU=www.digicert.com -# Subject: CN=DigiCert Assured ID Root G3 O=DigiCert Inc OU=www.digicert.com -# Label: "DigiCert Assured ID Root G3" -# Serial: 15459312981008553731928384953135426796 -# MD5 Fingerprint: 7c:7f:65:31:0c:81:df:8d:ba:3e:99:e2:5c:ad:6e:fb -# SHA1 Fingerprint: f5:17:a2:4f:9a:48:c6:c9:f8:a2:00:26:9f:dc:0f:48:2c:ab:30:89 -# SHA256 Fingerprint: 7e:37:cb:8b:4c:47:09:0c:ab:36:55:1b:a6:f4:5d:b8:40:68:0f:ba:16:6a:95:2d:b1:00:71:7f:43:05:3f:c2 ------BEGIN CERTIFICATE----- -MIICRjCCAc2gAwIBAgIQC6Fa+h3foLVJRK/NJKBs7DAKBggqhkjOPQQDAzBlMQsw -CQYDVQQGEwJVUzEVMBMGA1UEChMMRGlnaUNlcnQgSW5jMRkwFwYDVQQLExB3d3cu -ZGlnaWNlcnQuY29tMSQwIgYDVQQDExtEaWdpQ2VydCBBc3N1cmVkIElEIFJvb3Qg -RzMwHhcNMTMwODAxMTIwMDAwWhcNMzgwMTE1MTIwMDAwWjBlMQswCQYDVQQGEwJV -UzEVMBMGA1UEChMMRGlnaUNlcnQgSW5jMRkwFwYDVQQLExB3d3cuZGlnaWNlcnQu -Y29tMSQwIgYDVQQDExtEaWdpQ2VydCBBc3N1cmVkIElEIFJvb3QgRzMwdjAQBgcq -hkjOPQIBBgUrgQQAIgNiAAQZ57ysRGXtzbg/WPuNsVepRC0FFfLvC/8QdJ+1YlJf -Zn4f5dwbRXkLzMZTCp2NXQLZqVneAlr2lSoOjThKiknGvMYDOAdfVdp+CW7if17Q -RSAPWXYQ1qAk8C3eNvJsKTmjQjBAMA8GA1UdEwEB/wQFMAMBAf8wDgYDVR0PAQH/ -BAQDAgGGMB0GA1UdDgQWBBTL0L2p4ZgFUaFNN6KDec6NHSrkhDAKBggqhkjOPQQD -AwNnADBkAjAlpIFFAmsSS3V0T8gj43DydXLefInwz5FyYZ5eEJJZVrmDxxDnOOlY -JjZ91eQ0hjkCMHw2U/Aw5WJjOpnitqM7mzT6HtoQknFekROn3aRukswy1vUhZscv -6pZjamVFkpUBtA== ------END CERTIFICATE----- - -# Issuer: CN=DigiCert Global Root G2 O=DigiCert Inc OU=www.digicert.com -# Subject: CN=DigiCert Global Root G2 O=DigiCert Inc OU=www.digicert.com -# Label: "DigiCert Global Root G2" -# Serial: 4293743540046975378534879503202253541 -# MD5 Fingerprint: e4:a6:8a:c8:54:ac:52:42:46:0a:fd:72:48:1b:2a:44 -# SHA1 Fingerprint: df:3c:24:f9:bf:d6:66:76:1b:26:80:73:fe:06:d1:cc:8d:4f:82:a4 -# SHA256 Fingerprint: cb:3c:cb:b7:60:31:e5:e0:13:8f:8d:d3:9a:23:f9:de:47:ff:c3:5e:43:c1:14:4c:ea:27:d4:6a:5a:b1:cb:5f ------BEGIN CERTIFICATE----- -MIIDjjCCAnagAwIBAgIQAzrx5qcRqaC7KGSxHQn65TANBgkqhkiG9w0BAQsFADBh -MQswCQYDVQQGEwJVUzEVMBMGA1UEChMMRGlnaUNlcnQgSW5jMRkwFwYDVQQLExB3 -d3cuZGlnaWNlcnQuY29tMSAwHgYDVQQDExdEaWdpQ2VydCBHbG9iYWwgUm9vdCBH -MjAeFw0xMzA4MDExMjAwMDBaFw0zODAxMTUxMjAwMDBaMGExCzAJBgNVBAYTAlVT -MRUwEwYDVQQKEwxEaWdpQ2VydCBJbmMxGTAXBgNVBAsTEHd3dy5kaWdpY2VydC5j -b20xIDAeBgNVBAMTF0RpZ2lDZXJ0IEdsb2JhbCBSb290IEcyMIIBIjANBgkqhkiG -9w0BAQEFAAOCAQ8AMIIBCgKCAQEAuzfNNNx7a8myaJCtSnX/RrohCgiN9RlUyfuI -2/Ou8jqJkTx65qsGGmvPrC3oXgkkRLpimn7Wo6h+4FR1IAWsULecYxpsMNzaHxmx -1x7e/dfgy5SDN67sH0NO3Xss0r0upS/kqbitOtSZpLYl6ZtrAGCSYP9PIUkY92eQ -q2EGnI/yuum06ZIya7XzV+hdG82MHauVBJVJ8zUtluNJbd134/tJS7SsVQepj5Wz -tCO7TG1F8PapspUwtP1MVYwnSlcUfIKdzXOS0xZKBgyMUNGPHgm+F6HmIcr9g+UQ -vIOlCsRnKPZzFBQ9RnbDhxSJITRNrw9FDKZJobq7nMWxM4MphQIDAQABo0IwQDAP -BgNVHRMBAf8EBTADAQH/MA4GA1UdDwEB/wQEAwIBhjAdBgNVHQ4EFgQUTiJUIBiV -5uNu5g/6+rkS7QYXjzkwDQYJKoZIhvcNAQELBQADggEBAGBnKJRvDkhj6zHd6mcY -1Yl9PMWLSn/pvtsrF9+wX3N3KjITOYFnQoQj8kVnNeyIv/iPsGEMNKSuIEyExtv4 -NeF22d+mQrvHRAiGfzZ0JFrabA0UWTW98kndth/Jsw1HKj2ZL7tcu7XUIOGZX1NG -Fdtom/DzMNU+MeKNhJ7jitralj41E6Vf8PlwUHBHQRFXGU7Aj64GxJUTFy8bJZ91 -8rGOmaFvE7FBcf6IKshPECBV1/MUReXgRPTqh5Uykw7+U0b6LJ3/iyK5S9kJRaTe -pLiaWN0bfVKfjllDiIGknibVb63dDcY3fe0Dkhvld1927jyNxF1WW6LZZm6zNTfl -MrY= ------END CERTIFICATE----- - -# Issuer: CN=DigiCert Global Root G3 O=DigiCert Inc OU=www.digicert.com -# Subject: CN=DigiCert Global Root G3 O=DigiCert Inc OU=www.digicert.com -# Label: "DigiCert Global Root G3" -# Serial: 7089244469030293291760083333884364146 -# MD5 Fingerprint: f5:5d:a4:50:a5:fb:28:7e:1e:0f:0d:cc:96:57:56:ca -# SHA1 Fingerprint: 7e:04:de:89:6a:3e:66:6d:00:e6:87:d3:3f:fa:d9:3b:e8:3d:34:9e -# SHA256 Fingerprint: 31:ad:66:48:f8:10:41:38:c7:38:f3:9e:a4:32:01:33:39:3e:3a:18:cc:02:29:6e:f9:7c:2a:c9:ef:67:31:d0 ------BEGIN CERTIFICATE----- -MIICPzCCAcWgAwIBAgIQBVVWvPJepDU1w6QP1atFcjAKBggqhkjOPQQDAzBhMQsw -CQYDVQQGEwJVUzEVMBMGA1UEChMMRGlnaUNlcnQgSW5jMRkwFwYDVQQLExB3d3cu -ZGlnaWNlcnQuY29tMSAwHgYDVQQDExdEaWdpQ2VydCBHbG9iYWwgUm9vdCBHMzAe -Fw0xMzA4MDExMjAwMDBaFw0zODAxMTUxMjAwMDBaMGExCzAJBgNVBAYTAlVTMRUw -EwYDVQQKEwxEaWdpQ2VydCBJbmMxGTAXBgNVBAsTEHd3dy5kaWdpY2VydC5jb20x -IDAeBgNVBAMTF0RpZ2lDZXJ0IEdsb2JhbCBSb290IEczMHYwEAYHKoZIzj0CAQYF -K4EEACIDYgAE3afZu4q4C/sLfyHS8L6+c/MzXRq8NOrexpu80JX28MzQC7phW1FG -fp4tn+6OYwwX7Adw9c+ELkCDnOg/QW07rdOkFFk2eJ0DQ+4QE2xy3q6Ip6FrtUPO -Z9wj/wMco+I+o0IwQDAPBgNVHRMBAf8EBTADAQH/MA4GA1UdDwEB/wQEAwIBhjAd -BgNVHQ4EFgQUs9tIpPmhxdiuNkHMEWNpYim8S8YwCgYIKoZIzj0EAwMDaAAwZQIx -AK288mw/EkrRLTnDCgmXc/SINoyIJ7vmiI1Qhadj+Z4y3maTD/HMsQmP3Wyr+mt/ -oAIwOWZbwmSNuJ5Q3KjVSaLtx9zRSX8XAbjIho9OjIgrqJqpisXRAL34VOKa5Vt8 -sycX ------END CERTIFICATE----- - -# Issuer: CN=DigiCert Trusted Root G4 O=DigiCert Inc OU=www.digicert.com -# Subject: CN=DigiCert Trusted Root G4 O=DigiCert Inc OU=www.digicert.com -# Label: "DigiCert Trusted Root G4" -# Serial: 7451500558977370777930084869016614236 -# MD5 Fingerprint: 78:f2:fc:aa:60:1f:2f:b4:eb:c9:37:ba:53:2e:75:49 -# SHA1 Fingerprint: dd:fb:16:cd:49:31:c9:73:a2:03:7d:3f:c8:3a:4d:7d:77:5d:05:e4 -# SHA256 Fingerprint: 55:2f:7b:dc:f1:a7:af:9e:6c:e6:72:01:7f:4f:12:ab:f7:72:40:c7:8e:76:1a:c2:03:d1:d9:d2:0a:c8:99:88 ------BEGIN CERTIFICATE----- -MIIFkDCCA3igAwIBAgIQBZsbV56OITLiOQe9p3d1XDANBgkqhkiG9w0BAQwFADBi -MQswCQYDVQQGEwJVUzEVMBMGA1UEChMMRGlnaUNlcnQgSW5jMRkwFwYDVQQLExB3 -d3cuZGlnaWNlcnQuY29tMSEwHwYDVQQDExhEaWdpQ2VydCBUcnVzdGVkIFJvb3Qg -RzQwHhcNMTMwODAxMTIwMDAwWhcNMzgwMTE1MTIwMDAwWjBiMQswCQYDVQQGEwJV -UzEVMBMGA1UEChMMRGlnaUNlcnQgSW5jMRkwFwYDVQQLExB3d3cuZGlnaWNlcnQu -Y29tMSEwHwYDVQQDExhEaWdpQ2VydCBUcnVzdGVkIFJvb3QgRzQwggIiMA0GCSqG -SIb3DQEBAQUAA4ICDwAwggIKAoICAQC/5pBzaN675F1KPDAiMGkz7MKnJS7JIT3y -ithZwuEppz1Yq3aaza57G4QNxDAf8xukOBbrVsaXbR2rsnnyyhHS5F/WBTxSD1If -xp4VpX6+n6lXFllVcq9ok3DCsrp1mWpzMpTREEQQLt+C8weE5nQ7bXHiLQwb7iDV -ySAdYyktzuxeTsiT+CFhmzTrBcZe7FsavOvJz82sNEBfsXpm7nfISKhmV1efVFiO -DCu3T6cw2Vbuyntd463JT17lNecxy9qTXtyOj4DatpGYQJB5w3jHtrHEtWoYOAMQ -jdjUN6QuBX2I9YI+EJFwq1WCQTLX2wRzKm6RAXwhTNS8rhsDdV14Ztk6MUSaM0C/ -CNdaSaTC5qmgZ92kJ7yhTzm1EVgX9yRcRo9k98FpiHaYdj1ZXUJ2h4mXaXpI8OCi -EhtmmnTK3kse5w5jrubU75KSOp493ADkRSWJtppEGSt+wJS00mFt6zPZxd9LBADM -fRyVw4/3IbKyEbe7f/LVjHAsQWCqsWMYRJUadmJ+9oCw++hkpjPRiQfhvbfmQ6QY -uKZ3AeEPlAwhHbJUKSWJbOUOUlFHdL4mrLZBdd56rF+NP8m800ERElvlEFDrMcXK -chYiCd98THU/Y+whX8QgUWtvsauGi0/C1kVfnSD8oR7FwI+isX4KJpn15GkvmB0t -9dmpsh3lGwIDAQABo0IwQDAPBgNVHRMBAf8EBTADAQH/MA4GA1UdDwEB/wQEAwIB -hjAdBgNVHQ4EFgQU7NfjgtJxXWRM3y5nP+e6mK4cD08wDQYJKoZIhvcNAQEMBQAD -ggIBALth2X2pbL4XxJEbw6GiAI3jZGgPVs93rnD5/ZpKmbnJeFwMDF/k5hQpVgs2 -SV1EY+CtnJYYZhsjDT156W1r1lT40jzBQ0CuHVD1UvyQO7uYmWlrx8GnqGikJ9yd -+SeuMIW59mdNOj6PWTkiU0TryF0Dyu1Qen1iIQqAyHNm0aAFYF/opbSnr6j3bTWc -fFqK1qI4mfN4i/RN0iAL3gTujJtHgXINwBQy7zBZLq7gcfJW5GqXb5JQbZaNaHqa -sjYUegbyJLkJEVDXCLG4iXqEI2FCKeWjzaIgQdfRnGTZ6iahixTXTBmyUEFxPT9N -cCOGDErcgdLMMpSEDQgJlxxPwO5rIHQw0uA5NBCFIRUBCOhVMt5xSdkoF1BN5r5N -0XWs0Mr7QbhDparTwwVETyw2m+L64kW4I1NsBm9nVX9GtUw/bihaeSbSpKhil9Ie -4u1Ki7wb/UdKDd9nZn6yW0HQO+T0O/QEY+nvwlQAUaCKKsnOeMzV6ocEGLPOr0mI -r/OSmbaz5mEP0oUA51Aa5BuVnRmhuZyxm7EAHu/QD09CbMkKvO5D+jpxpchNJqU1 -/YldvIViHTLSoCtU7ZpXwdv6EM8Zt4tKG48BtieVU+i2iW1bvGjUI+iLUaJW+fCm -gKDWHrO8Dw9TdSmq6hN35N6MgSGtBxBHEa2HPQfRdbzP82Z+ ------END CERTIFICATE----- - -# Issuer: CN=COMODO RSA Certification Authority O=COMODO CA Limited -# Subject: CN=COMODO RSA Certification Authority O=COMODO CA Limited -# Label: "COMODO RSA Certification Authority" -# Serial: 101909084537582093308941363524873193117 -# MD5 Fingerprint: 1b:31:b0:71:40:36:cc:14:36:91:ad:c4:3e:fd:ec:18 -# SHA1 Fingerprint: af:e5:d2:44:a8:d1:19:42:30:ff:47:9f:e2:f8:97:bb:cd:7a:8c:b4 -# SHA256 Fingerprint: 52:f0:e1:c4:e5:8e:c6:29:29:1b:60:31:7f:07:46:71:b8:5d:7e:a8:0d:5b:07:27:34:63:53:4b:32:b4:02:34 ------BEGIN CERTIFICATE----- -MIIF2DCCA8CgAwIBAgIQTKr5yttjb+Af907YWwOGnTANBgkqhkiG9w0BAQwFADCB -hTELMAkGA1UEBhMCR0IxGzAZBgNVBAgTEkdyZWF0ZXIgTWFuY2hlc3RlcjEQMA4G -A1UEBxMHU2FsZm9yZDEaMBgGA1UEChMRQ09NT0RPIENBIExpbWl0ZWQxKzApBgNV -BAMTIkNPTU9ETyBSU0EgQ2VydGlmaWNhdGlvbiBBdXRob3JpdHkwHhcNMTAwMTE5 -MDAwMDAwWhcNMzgwMTE4MjM1OTU5WjCBhTELMAkGA1UEBhMCR0IxGzAZBgNVBAgT -EkdyZWF0ZXIgTWFuY2hlc3RlcjEQMA4GA1UEBxMHU2FsZm9yZDEaMBgGA1UEChMR -Q09NT0RPIENBIExpbWl0ZWQxKzApBgNVBAMTIkNPTU9ETyBSU0EgQ2VydGlmaWNh -dGlvbiBBdXRob3JpdHkwggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQCR -6FSS0gpWsawNJN3Fz0RndJkrN6N9I3AAcbxT38T6KhKPS38QVr2fcHK3YX/JSw8X -pz3jsARh7v8Rl8f0hj4K+j5c+ZPmNHrZFGvnnLOFoIJ6dq9xkNfs/Q36nGz637CC -9BR++b7Epi9Pf5l/tfxnQ3K9DADWietrLNPtj5gcFKt+5eNu/Nio5JIk2kNrYrhV -/erBvGy2i/MOjZrkm2xpmfh4SDBF1a3hDTxFYPwyllEnvGfDyi62a+pGx8cgoLEf -Zd5ICLqkTqnyg0Y3hOvozIFIQ2dOciqbXL1MGyiKXCJ7tKuY2e7gUYPDCUZObT6Z -+pUX2nwzV0E8jVHtC7ZcryxjGt9XyD+86V3Em69FmeKjWiS0uqlWPc9vqv9JWL7w -qP/0uK3pN/u6uPQLOvnoQ0IeidiEyxPx2bvhiWC4jChWrBQdnArncevPDt09qZah -SL0896+1DSJMwBGB7FY79tOi4lu3sgQiUpWAk2nojkxl8ZEDLXB0AuqLZxUpaVIC -u9ffUGpVRr+goyhhf3DQw6KqLCGqR84onAZFdr+CGCe01a60y1Dma/RMhnEw6abf -Fobg2P9A3fvQQoh/ozM6LlweQRGBY84YcWsr7KaKtzFcOmpH4MN5WdYgGq/yapiq -crxXStJLnbsQ/LBMQeXtHT1eKJ2czL+zUdqnR+WEUwIDAQABo0IwQDAdBgNVHQ4E -FgQUu69+Aj36pvE8hI6t7jiY7NkyMtQwDgYDVR0PAQH/BAQDAgEGMA8GA1UdEwEB -/wQFMAMBAf8wDQYJKoZIhvcNAQEMBQADggIBAArx1UaEt65Ru2yyTUEUAJNMnMvl -wFTPoCWOAvn9sKIN9SCYPBMtrFaisNZ+EZLpLrqeLppysb0ZRGxhNaKatBYSaVqM -4dc+pBroLwP0rmEdEBsqpIt6xf4FpuHA1sj+nq6PK7o9mfjYcwlYRm6mnPTXJ9OV -2jeDchzTc+CiR5kDOF3VSXkAKRzH7JsgHAckaVd4sjn8OoSgtZx8jb8uk2Intzna -FxiuvTwJaP+EmzzV1gsD41eeFPfR60/IvYcjt7ZJQ3mFXLrrkguhxuhoqEwWsRqZ -CuhTLJK7oQkYdQxlqHvLI7cawiiFwxv/0Cti76R7CZGYZ4wUAc1oBmpjIXUDgIiK -boHGhfKppC3n9KUkEEeDys30jXlYsQab5xoq2Z0B15R97QNKyvDb6KkBPvVWmcke -jkk9u+UJueBPSZI9FoJAzMxZxuY67RIuaTxslbH9qh17f4a+Hg4yRvv7E491f0yL -S0Zj/gA0QHDBw7mh3aZw4gSzQbzpgJHqZJx64SIDqZxubw5lT2yHh17zbqD5daWb -QOhTsiedSrnAdyGN/4fy3ryM7xfft0kL0fJuMAsaDk527RH89elWsn2/x20Kk4yl -0MC2Hb46TpSi125sC8KKfPog88Tk5c0NqMuRkrF8hey1FGlmDoLnzc7ILaZRfyHB -NVOFBkpdn627G190 ------END CERTIFICATE----- - -# Issuer: CN=USERTrust RSA Certification Authority O=The USERTRUST Network -# Subject: CN=USERTrust RSA Certification Authority O=The USERTRUST Network -# Label: "USERTrust RSA Certification Authority" -# Serial: 2645093764781058787591871645665788717 -# MD5 Fingerprint: 1b:fe:69:d1:91:b7:19:33:a3:72:a8:0f:e1:55:e5:b5 -# SHA1 Fingerprint: 2b:8f:1b:57:33:0d:bb:a2:d0:7a:6c:51:f7:0e:e9:0d:da:b9:ad:8e -# SHA256 Fingerprint: e7:93:c9:b0:2f:d8:aa:13:e2:1c:31:22:8a:cc:b0:81:19:64:3b:74:9c:89:89:64:b1:74:6d:46:c3:d4:cb:d2 ------BEGIN CERTIFICATE----- -MIIF3jCCA8agAwIBAgIQAf1tMPyjylGoG7xkDjUDLTANBgkqhkiG9w0BAQwFADCB -iDELMAkGA1UEBhMCVVMxEzARBgNVBAgTCk5ldyBKZXJzZXkxFDASBgNVBAcTC0pl -cnNleSBDaXR5MR4wHAYDVQQKExVUaGUgVVNFUlRSVVNUIE5ldHdvcmsxLjAsBgNV -BAMTJVVTRVJUcnVzdCBSU0EgQ2VydGlmaWNhdGlvbiBBdXRob3JpdHkwHhcNMTAw -MjAxMDAwMDAwWhcNMzgwMTE4MjM1OTU5WjCBiDELMAkGA1UEBhMCVVMxEzARBgNV -BAgTCk5ldyBKZXJzZXkxFDASBgNVBAcTC0plcnNleSBDaXR5MR4wHAYDVQQKExVU -aGUgVVNFUlRSVVNUIE5ldHdvcmsxLjAsBgNVBAMTJVVTRVJUcnVzdCBSU0EgQ2Vy -dGlmaWNhdGlvbiBBdXRob3JpdHkwggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIK -AoICAQCAEmUXNg7D2wiz0KxXDXbtzSfTTK1Qg2HiqiBNCS1kCdzOiZ/MPans9s/B -3PHTsdZ7NygRK0faOca8Ohm0X6a9fZ2jY0K2dvKpOyuR+OJv0OwWIJAJPuLodMkY -tJHUYmTbf6MG8YgYapAiPLz+E/CHFHv25B+O1ORRxhFnRghRy4YUVD+8M/5+bJz/ -Fp0YvVGONaanZshyZ9shZrHUm3gDwFA66Mzw3LyeTP6vBZY1H1dat//O+T23LLb2 -VN3I5xI6Ta5MirdcmrS3ID3KfyI0rn47aGYBROcBTkZTmzNg95S+UzeQc0PzMsNT -79uq/nROacdrjGCT3sTHDN/hMq7MkztReJVni+49Vv4M0GkPGw/zJSZrM233bkf6 -c0Plfg6lZrEpfDKEY1WJxA3Bk1QwGROs0303p+tdOmw1XNtB1xLaqUkL39iAigmT -Yo61Zs8liM2EuLE/pDkP2QKe6xJMlXzzawWpXhaDzLhn4ugTncxbgtNMs+1b/97l -c6wjOy0AvzVVdAlJ2ElYGn+SNuZRkg7zJn0cTRe8yexDJtC/QV9AqURE9JnnV4ee -UB9XVKg+/XRjL7FQZQnmWEIuQxpMtPAlR1n6BB6T1CZGSlCBst6+eLf8ZxXhyVeE -Hg9j1uliutZfVS7qXMYoCAQlObgOK6nyTJccBz8NUvXt7y+CDwIDAQABo0IwQDAd -BgNVHQ4EFgQUU3m/WqorSs9UgOHYm8Cd8rIDZsswDgYDVR0PAQH/BAQDAgEGMA8G -A1UdEwEB/wQFMAMBAf8wDQYJKoZIhvcNAQEMBQADggIBAFzUfA3P9wF9QZllDHPF -Up/L+M+ZBn8b2kMVn54CVVeWFPFSPCeHlCjtHzoBN6J2/FNQwISbxmtOuowhT6KO -VWKR82kV2LyI48SqC/3vqOlLVSoGIG1VeCkZ7l8wXEskEVX/JJpuXior7gtNn3/3 -ATiUFJVDBwn7YKnuHKsSjKCaXqeYalltiz8I+8jRRa8YFWSQEg9zKC7F4iRO/Fjs -8PRF/iKz6y+O0tlFYQXBl2+odnKPi4w2r78NBc5xjeambx9spnFixdjQg3IM8WcR -iQycE0xyNN+81XHfqnHd4blsjDwSXWXavVcStkNr/+XeTWYRUc+ZruwXtuhxkYze -Sf7dNXGiFSeUHM9h4ya7b6NnJSFd5t0dCy5oGzuCr+yDZ4XUmFF0sbmZgIn/f3gZ -XHlKYC6SQK5MNyosycdiyA5d9zZbyuAlJQG03RoHnHcAP9Dc1ew91Pq7P8yF1m9/ -qS3fuQL39ZeatTXaw2ewh0qpKJ4jjv9cJ2vhsE/zB+4ALtRZh8tSQZXq9EfX7mRB -VXyNWQKV3WKdwrnuWih0hKWbt5DHDAff9Yk2dDLWKMGwsAvgnEzDHNb842m1R0aB -L6KCq9NjRHDEjf8tM7qtj3u1cIiuPhnPQCjY/MiQu12ZIvVS5ljFH4gxQ+6IHdfG -jjxDah2nGN59PRbxYvnKkKj9 ------END CERTIFICATE----- - -# Issuer: CN=USERTrust ECC Certification Authority O=The USERTRUST Network -# Subject: CN=USERTrust ECC Certification Authority O=The USERTRUST Network -# Label: "USERTrust ECC Certification Authority" -# Serial: 123013823720199481456569720443997572134 -# MD5 Fingerprint: fa:68:bc:d9:b5:7f:ad:fd:c9:1d:06:83:28:cc:24:c1 -# SHA1 Fingerprint: d1:cb:ca:5d:b2:d5:2a:7f:69:3b:67:4d:e5:f0:5a:1d:0c:95:7d:f0 -# SHA256 Fingerprint: 4f:f4:60:d5:4b:9c:86:da:bf:bc:fc:57:12:e0:40:0d:2b:ed:3f:bc:4d:4f:bd:aa:86:e0:6a:dc:d2:a9:ad:7a ------BEGIN CERTIFICATE----- -MIICjzCCAhWgAwIBAgIQXIuZxVqUxdJxVt7NiYDMJjAKBggqhkjOPQQDAzCBiDEL -MAkGA1UEBhMCVVMxEzARBgNVBAgTCk5ldyBKZXJzZXkxFDASBgNVBAcTC0plcnNl -eSBDaXR5MR4wHAYDVQQKExVUaGUgVVNFUlRSVVNUIE5ldHdvcmsxLjAsBgNVBAMT -JVVTRVJUcnVzdCBFQ0MgQ2VydGlmaWNhdGlvbiBBdXRob3JpdHkwHhcNMTAwMjAx -MDAwMDAwWhcNMzgwMTE4MjM1OTU5WjCBiDELMAkGA1UEBhMCVVMxEzARBgNVBAgT -Ck5ldyBKZXJzZXkxFDASBgNVBAcTC0plcnNleSBDaXR5MR4wHAYDVQQKExVUaGUg -VVNFUlRSVVNUIE5ldHdvcmsxLjAsBgNVBAMTJVVTRVJUcnVzdCBFQ0MgQ2VydGlm -aWNhdGlvbiBBdXRob3JpdHkwdjAQBgcqhkjOPQIBBgUrgQQAIgNiAAQarFRaqflo -I+d61SRvU8Za2EurxtW20eZzca7dnNYMYf3boIkDuAUU7FfO7l0/4iGzzvfUinng -o4N+LZfQYcTxmdwlkWOrfzCjtHDix6EznPO/LlxTsV+zfTJ/ijTjeXmjQjBAMB0G -A1UdDgQWBBQ64QmG1M8ZwpZ2dEl23OA1xmNjmjAOBgNVHQ8BAf8EBAMCAQYwDwYD -VR0TAQH/BAUwAwEB/zAKBggqhkjOPQQDAwNoADBlAjA2Z6EWCNzklwBBHU6+4WMB -zzuqQhFkoJ2UOQIReVx7Hfpkue4WQrO/isIJxOzksU0CMQDpKmFHjFJKS04YcPbW -RNZu9YO6bVi9JNlWSOrvxKJGgYhqOkbRqZtNyWHa0V1Xahg= ------END CERTIFICATE----- - -# Issuer: CN=GlobalSign O=GlobalSign OU=GlobalSign ECC Root CA - R5 -# Subject: CN=GlobalSign O=GlobalSign OU=GlobalSign ECC Root CA - R5 -# Label: "GlobalSign ECC Root CA - R5" -# Serial: 32785792099990507226680698011560947931244 -# MD5 Fingerprint: 9f:ad:3b:1c:02:1e:8a:ba:17:74:38:81:0c:a2:bc:08 -# SHA1 Fingerprint: 1f:24:c6:30:cd:a4:18:ef:20:69:ff:ad:4f:dd:5f:46:3a:1b:69:aa -# SHA256 Fingerprint: 17:9f:bc:14:8a:3d:d0:0f:d2:4e:a1:34:58:cc:43:bf:a7:f5:9c:81:82:d7:83:a5:13:f6:eb:ec:10:0c:89:24 ------BEGIN CERTIFICATE----- -MIICHjCCAaSgAwIBAgIRYFlJ4CYuu1X5CneKcflK2GwwCgYIKoZIzj0EAwMwUDEk -MCIGA1UECxMbR2xvYmFsU2lnbiBFQ0MgUm9vdCBDQSAtIFI1MRMwEQYDVQQKEwpH -bG9iYWxTaWduMRMwEQYDVQQDEwpHbG9iYWxTaWduMB4XDTEyMTExMzAwMDAwMFoX -DTM4MDExOTAzMTQwN1owUDEkMCIGA1UECxMbR2xvYmFsU2lnbiBFQ0MgUm9vdCBD -QSAtIFI1MRMwEQYDVQQKEwpHbG9iYWxTaWduMRMwEQYDVQQDEwpHbG9iYWxTaWdu -MHYwEAYHKoZIzj0CAQYFK4EEACIDYgAER0UOlvt9Xb/pOdEh+J8LttV7HpI6SFkc -8GIxLcB6KP4ap1yztsyX50XUWPrRd21DosCHZTQKH3rd6zwzocWdTaRvQZU4f8ke -hOvRnkmSh5SHDDqFSmafnVmTTZdhBoZKo0IwQDAOBgNVHQ8BAf8EBAMCAQYwDwYD -VR0TAQH/BAUwAwEB/zAdBgNVHQ4EFgQUPeYpSJvqB8ohREom3m7e0oPQn1kwCgYI -KoZIzj0EAwMDaAAwZQIxAOVpEslu28YxuglB4Zf4+/2a4n0Sye18ZNPLBSWLVtmg -515dTguDnFt2KaAJJiFqYgIwcdK1j1zqO+F4CYWodZI7yFz9SO8NdCKoCOJuxUnO -xwy8p2Fp8fc74SrL+SvzZpA3 ------END CERTIFICATE----- - -# Issuer: CN=IdenTrust Commercial Root CA 1 O=IdenTrust -# Subject: CN=IdenTrust Commercial Root CA 1 O=IdenTrust -# Label: "IdenTrust Commercial Root CA 1" -# Serial: 13298821034946342390520003877796839426 -# MD5 Fingerprint: b3:3e:77:73:75:ee:a0:d3:e3:7e:49:63:49:59:bb:c7 -# SHA1 Fingerprint: df:71:7e:aa:4a:d9:4e:c9:55:84:99:60:2d:48:de:5f:bc:f0:3a:25 -# SHA256 Fingerprint: 5d:56:49:9b:e4:d2:e0:8b:cf:ca:d0:8a:3e:38:72:3d:50:50:3b:de:70:69:48:e4:2f:55:60:30:19:e5:28:ae ------BEGIN CERTIFICATE----- -MIIFYDCCA0igAwIBAgIQCgFCgAAAAUUjyES1AAAAAjANBgkqhkiG9w0BAQsFADBK -MQswCQYDVQQGEwJVUzESMBAGA1UEChMJSWRlblRydXN0MScwJQYDVQQDEx5JZGVu -VHJ1c3QgQ29tbWVyY2lhbCBSb290IENBIDEwHhcNMTQwMTE2MTgxMjIzWhcNMzQw -MTE2MTgxMjIzWjBKMQswCQYDVQQGEwJVUzESMBAGA1UEChMJSWRlblRydXN0MScw -JQYDVQQDEx5JZGVuVHJ1c3QgQ29tbWVyY2lhbCBSb290IENBIDEwggIiMA0GCSqG -SIb3DQEBAQUAA4ICDwAwggIKAoICAQCnUBneP5k91DNG8W9RYYKyqU+PZ4ldhNlT -3Qwo2dfw/66VQ3KZ+bVdfIrBQuExUHTRgQ18zZshq0PirK1ehm7zCYofWjK9ouuU -+ehcCuz/mNKvcbO0U59Oh++SvL3sTzIwiEsXXlfEU8L2ApeN2WIrvyQfYo3fw7gp -S0l4PJNgiCL8mdo2yMKi1CxUAGc1bnO/AljwpN3lsKImesrgNqUZFvX9t++uP0D1 -bVoE/c40yiTcdCMbXTMTEl3EASX2MN0CXZ/g1Ue9tOsbobtJSdifWwLziuQkkORi -T0/Br4sOdBeo0XKIanoBScy0RnnGF7HamB4HWfp1IYVl3ZBWzvurpWCdxJ35UrCL -vYf5jysjCiN2O/cz4ckA82n5S6LgTrx+kzmEB/dEcH7+B1rlsazRGMzyNeVJSQjK -Vsk9+w8YfYs7wRPCTY/JTw436R+hDmrfYi7LNQZReSzIJTj0+kuniVyc0uMNOYZK -dHzVWYfCP04MXFL0PfdSgvHqo6z9STQaKPNBiDoT7uje/5kdX7rL6B7yuVBgwDHT -c+XvvqDtMwt0viAgxGds8AgDelWAf0ZOlqf0Hj7h9tgJ4TNkK2PXMl6f+cB7D3hv -l7yTmvmcEpB4eoCHFddydJxVdHixuuFucAS6T6C6aMN7/zHwcz09lCqxC0EOoP5N -iGVreTO01wIDAQABo0IwQDAOBgNVHQ8BAf8EBAMCAQYwDwYDVR0TAQH/BAUwAwEB -/zAdBgNVHQ4EFgQU7UQZwNPwBovupHu+QucmVMiONnYwDQYJKoZIhvcNAQELBQAD -ggIBAA2ukDL2pkt8RHYZYR4nKM1eVO8lvOMIkPkp165oCOGUAFjvLi5+U1KMtlwH -6oi6mYtQlNeCgN9hCQCTrQ0U5s7B8jeUeLBfnLOic7iPBZM4zY0+sLj7wM+x8uwt -LRvM7Kqas6pgghstO8OEPVeKlh6cdbjTMM1gCIOQ045U8U1mwF10A0Cj7oV+wh93 -nAbowacYXVKV7cndJZ5t+qntozo00Fl72u1Q8zW/7esUTTHHYPTa8Yec4kjixsU3 -+wYQ+nVZZjFHKdp2mhzpgq7vmrlR94gjmmmVYjzlVYA211QC//G5Xc7UI2/YRYRK -W2XviQzdFKcgyxilJbQN+QHwotL0AMh0jqEqSI5l2xPE4iUXfeu+h1sXIFRRk0pT -AwvsXcoz7WL9RccvW9xYoIA55vrX/hMUpu09lEpCdNTDd1lzzY9GvlU47/rokTLq -l1gEIt44w8y8bckzOmoKaT+gyOpyj4xjhiO9bTyWnpXgSUyqorkqG5w2gXjtw+hG -4iZZRHUe2XWJUc0QhJ1hYMtd+ZciTY6Y5uN/9lu7rs3KSoFrXgvzUeF0K+l+J6fZ -mUlO+KWA2yUPHGNiiskzZ2s8EIPGrd6ozRaOjfAHN3Gf8qv8QfXBi+wAN10J5U6A -7/qxXDgGpRtK4dw4LTzcqx+QGtVKnO7RcGzM7vRX+Bi6hG6H ------END CERTIFICATE----- - -# Issuer: CN=IdenTrust Public Sector Root CA 1 O=IdenTrust -# Subject: CN=IdenTrust Public Sector Root CA 1 O=IdenTrust -# Label: "IdenTrust Public Sector Root CA 1" -# Serial: 13298821034946342390521976156843933698 -# MD5 Fingerprint: 37:06:a5:b0:fc:89:9d:ba:f4:6b:8c:1a:64:cd:d5:ba -# SHA1 Fingerprint: ba:29:41:60:77:98:3f:f4:f3:ef:f2:31:05:3b:2e:ea:6d:4d:45:fd -# SHA256 Fingerprint: 30:d0:89:5a:9a:44:8a:26:20:91:63:55:22:d1:f5:20:10:b5:86:7a:ca:e1:2c:78:ef:95:8f:d4:f4:38:9f:2f ------BEGIN CERTIFICATE----- -MIIFZjCCA06gAwIBAgIQCgFCgAAAAUUjz0Z8AAAAAjANBgkqhkiG9w0BAQsFADBN -MQswCQYDVQQGEwJVUzESMBAGA1UEChMJSWRlblRydXN0MSowKAYDVQQDEyFJZGVu -VHJ1c3QgUHVibGljIFNlY3RvciBSb290IENBIDEwHhcNMTQwMTE2MTc1MzMyWhcN -MzQwMTE2MTc1MzMyWjBNMQswCQYDVQQGEwJVUzESMBAGA1UEChMJSWRlblRydXN0 -MSowKAYDVQQDEyFJZGVuVHJ1c3QgUHVibGljIFNlY3RvciBSb290IENBIDEwggIi -MA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQC2IpT8pEiv6EdrCvsnduTyP4o7 -ekosMSqMjbCpwzFrqHd2hCa2rIFCDQjrVVi7evi8ZX3yoG2LqEfpYnYeEe4IFNGy -RBb06tD6Hi9e28tzQa68ALBKK0CyrOE7S8ItneShm+waOh7wCLPQ5CQ1B5+ctMlS -bdsHyo+1W/CD80/HLaXIrcuVIKQxKFdYWuSNG5qrng0M8gozOSI5Cpcu81N3uURF -/YTLNiCBWS2ab21ISGHKTN9T0a9SvESfqy9rg3LvdYDaBjMbXcjaY8ZNzaxmMc3R -3j6HEDbhuaR672BQssvKplbgN6+rNBM5Jeg5ZuSYeqoSmJxZZoY+rfGwyj4GD3vw -EUs3oERte8uojHH01bWRNszwFcYr3lEXsZdMUD2xlVl8BX0tIdUAvwFnol57plzy -9yLxkA2T26pEUWbMfXYD62qoKjgZl3YNa4ph+bz27nb9cCvdKTz4Ch5bQhyLVi9V -GxyhLrXHFub4qjySjmm2AcG1hp2JDws4lFTo6tyePSW8Uybt1as5qsVATFSrsrTZ -2fjXctscvG29ZV/viDUqZi/u9rNl8DONfJhBaUYPQxxp+pu10GFqzcpL2UyQRqsV -WaFHVCkugyhfHMKiq3IXAAaOReyL4jM9f9oZRORicsPfIsbyVtTdX5Vy7W1f90gD -W/3FKqD2cyOEEBsB5wIDAQABo0IwQDAOBgNVHQ8BAf8EBAMCAQYwDwYDVR0TAQH/ -BAUwAwEB/zAdBgNVHQ4EFgQU43HgntinQtnbcZFrlJPrw6PRFKMwDQYJKoZIhvcN -AQELBQADggIBAEf63QqwEZE4rU1d9+UOl1QZgkiHVIyqZJnYWv6IAcVYpZmxI1Qj -t2odIFflAWJBF9MJ23XLblSQdf4an4EKwt3X9wnQW3IV5B4Jaj0z8yGa5hV+rVHV -DRDtfULAj+7AmgjVQdZcDiFpboBhDhXAuM/FSRJSzL46zNQuOAXeNf0fb7iAaJg9 -TaDKQGXSc3z1i9kKlT/YPyNtGtEqJBnZhbMX73huqVjRI9PHE+1yJX9dsXNw0H8G -lwmEKYBhHfpe/3OsoOOJuBxxFcbeMX8S3OFtm6/n6J91eEyrRjuazr8FGF1NFTwW -mhlQBJqymm9li1JfPFgEKCXAZmExfrngdbkaqIHWchezxQMxNRF4eKLg6TCMf4Df -WN88uieW4oA0beOY02QnrEh+KHdcxiVhJfiFDGX6xDIvpZgF5PgLZxYWxoK4Mhn5 -+bl53B/N66+rDt0b20XkeucC4pVd/GnwU2lhlXV5C15V5jgclKlZM57IcXR5f1GJ -tshquDDIajjDbp7hNxbqBWJMWxJH7ae0s1hWx0nzfxJoCTFx8G34Tkf71oXuxVhA -GaQdp/lLQzfcaFpPz+vCZHTetBXZ9FRUGi8c15dxVJCO2SCdUyt/q4/i6jC8UDfv -8Ue1fXwsBOxonbRJRBD0ckscZOf85muQ3Wl9af0AVqW3rLatt8o+Ae+c ------END CERTIFICATE----- - -# Issuer: CN=Entrust Root Certification Authority - G2 O=Entrust, Inc. OU=See www.entrust.net/legal-terms/(c) 2009 Entrust, Inc. - for authorized use only -# Subject: CN=Entrust Root Certification Authority - G2 O=Entrust, Inc. OU=See www.entrust.net/legal-terms/(c) 2009 Entrust, Inc. - for authorized use only -# Label: "Entrust Root Certification Authority - G2" -# Serial: 1246989352 -# MD5 Fingerprint: 4b:e2:c9:91:96:65:0c:f4:0e:5a:93:92:a0:0a:fe:b2 -# SHA1 Fingerprint: 8c:f4:27:fd:79:0c:3a:d1:66:06:8d:e8:1e:57:ef:bb:93:22:72:d4 -# SHA256 Fingerprint: 43:df:57:74:b0:3e:7f:ef:5f:e4:0d:93:1a:7b:ed:f1:bb:2e:6b:42:73:8c:4e:6d:38:41:10:3d:3a:a7:f3:39 ------BEGIN CERTIFICATE----- -MIIEPjCCAyagAwIBAgIESlOMKDANBgkqhkiG9w0BAQsFADCBvjELMAkGA1UEBhMC -VVMxFjAUBgNVBAoTDUVudHJ1c3QsIEluYy4xKDAmBgNVBAsTH1NlZSB3d3cuZW50 -cnVzdC5uZXQvbGVnYWwtdGVybXMxOTA3BgNVBAsTMChjKSAyMDA5IEVudHJ1c3Qs -IEluYy4gLSBmb3IgYXV0aG9yaXplZCB1c2Ugb25seTEyMDAGA1UEAxMpRW50cnVz -dCBSb290IENlcnRpZmljYXRpb24gQXV0aG9yaXR5IC0gRzIwHhcNMDkwNzA3MTcy -NTU0WhcNMzAxMjA3MTc1NTU0WjCBvjELMAkGA1UEBhMCVVMxFjAUBgNVBAoTDUVu -dHJ1c3QsIEluYy4xKDAmBgNVBAsTH1NlZSB3d3cuZW50cnVzdC5uZXQvbGVnYWwt -dGVybXMxOTA3BgNVBAsTMChjKSAyMDA5IEVudHJ1c3QsIEluYy4gLSBmb3IgYXV0 -aG9yaXplZCB1c2Ugb25seTEyMDAGA1UEAxMpRW50cnVzdCBSb290IENlcnRpZmlj -YXRpb24gQXV0aG9yaXR5IC0gRzIwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEK -AoIBAQC6hLZy254Ma+KZ6TABp3bqMriVQRrJ2mFOWHLP/vaCeb9zYQYKpSfYs1/T -RU4cctZOMvJyig/3gxnQaoCAAEUesMfnmr8SVycco2gvCoe9amsOXmXzHHfV1IWN -cCG0szLni6LVhjkCsbjSR87kyUnEO6fe+1R9V77w6G7CebI6C1XiUJgWMhNcL3hW -wcKUs/Ja5CeanyTXxuzQmyWC48zCxEXFjJd6BmsqEZ+pCm5IO2/b1BEZQvePB7/1 -U1+cPvQXLOZprE4yTGJ36rfo5bs0vBmLrpxR57d+tVOxMyLlbc9wPBr64ptntoP0 -jaWvYkxN4FisZDQSA/i2jZRjJKRxAgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBBjAP -BgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBRqciZ60B7vfec7aVHUbI2fkBJmqzAN -BgkqhkiG9w0BAQsFAAOCAQEAeZ8dlsa2eT8ijYfThwMEYGprmi5ZiXMRrEPR9RP/ -jTkrwPK9T3CMqS/qF8QLVJ7UG5aYMzyorWKiAHarWWluBh1+xLlEjZivEtRh2woZ -Rkfz6/djwUAFQKXSt/S1mja/qYh2iARVBCuch38aNzx+LaUa2NSJXsq9rD1s2G2v -1fN2D807iDginWyTmsQ9v4IbZT+mD12q/OWyFcq1rca8PdCE6OoGcrBNOTJ4vz4R -nAuknZoh8/CbCzB428Hch0P+vGOaysXCHMnHjf87ElgI5rY97HosTvuDls4MPGmH -VHOkc8KT/1EQrBVUAdj8BbGJoX90g5pJ19xOe4pIb4tF9g== ------END CERTIFICATE----- - -# Issuer: CN=Entrust Root Certification Authority - EC1 O=Entrust, Inc. OU=See www.entrust.net/legal-terms/(c) 2012 Entrust, Inc. - for authorized use only -# Subject: CN=Entrust Root Certification Authority - EC1 O=Entrust, Inc. OU=See www.entrust.net/legal-terms/(c) 2012 Entrust, Inc. - for authorized use only -# Label: "Entrust Root Certification Authority - EC1" -# Serial: 51543124481930649114116133369 -# MD5 Fingerprint: b6:7e:1d:f0:58:c5:49:6c:24:3b:3d:ed:98:18:ed:bc -# SHA1 Fingerprint: 20:d8:06:40:df:9b:25:f5:12:25:3a:11:ea:f7:59:8a:eb:14:b5:47 -# SHA256 Fingerprint: 02:ed:0e:b2:8c:14:da:45:16:5c:56:67:91:70:0d:64:51:d7:fb:56:f0:b2:ab:1d:3b:8e:b0:70:e5:6e:df:f5 ------BEGIN CERTIFICATE----- -MIIC+TCCAoCgAwIBAgINAKaLeSkAAAAAUNCR+TAKBggqhkjOPQQDAzCBvzELMAkG -A1UEBhMCVVMxFjAUBgNVBAoTDUVudHJ1c3QsIEluYy4xKDAmBgNVBAsTH1NlZSB3 -d3cuZW50cnVzdC5uZXQvbGVnYWwtdGVybXMxOTA3BgNVBAsTMChjKSAyMDEyIEVu -dHJ1c3QsIEluYy4gLSBmb3IgYXV0aG9yaXplZCB1c2Ugb25seTEzMDEGA1UEAxMq -RW50cnVzdCBSb290IENlcnRpZmljYXRpb24gQXV0aG9yaXR5IC0gRUMxMB4XDTEy -MTIxODE1MjUzNloXDTM3MTIxODE1NTUzNlowgb8xCzAJBgNVBAYTAlVTMRYwFAYD -VQQKEw1FbnRydXN0LCBJbmMuMSgwJgYDVQQLEx9TZWUgd3d3LmVudHJ1c3QubmV0 -L2xlZ2FsLXRlcm1zMTkwNwYDVQQLEzAoYykgMjAxMiBFbnRydXN0LCBJbmMuIC0g -Zm9yIGF1dGhvcml6ZWQgdXNlIG9ubHkxMzAxBgNVBAMTKkVudHJ1c3QgUm9vdCBD -ZXJ0aWZpY2F0aW9uIEF1dGhvcml0eSAtIEVDMTB2MBAGByqGSM49AgEGBSuBBAAi -A2IABIQTydC6bUF74mzQ61VfZgIaJPRbiWlH47jCffHyAsWfoPZb1YsGGYZPUxBt -ByQnoaD41UcZYUx9ypMn6nQM72+WCf5j7HBdNq1nd67JnXxVRDqiY1Ef9eNi1KlH -Bz7MIKNCMEAwDgYDVR0PAQH/BAQDAgEGMA8GA1UdEwEB/wQFMAMBAf8wHQYDVR0O -BBYEFLdj5xrdjekIplWDpOBqUEFlEUJJMAoGCCqGSM49BAMDA2cAMGQCMGF52OVC -R98crlOZF7ZvHH3hvxGU0QOIdeSNiaSKd0bebWHvAvX7td/M/k7//qnmpwIwW5nX -hTcGtXsI/esni0qU+eH6p44mCOh8kmhtc9hvJqwhAriZtyZBWyVgrtBIGu4G ------END CERTIFICATE----- - -# Issuer: CN=CFCA EV ROOT O=China Financial Certification Authority -# Subject: CN=CFCA EV ROOT O=China Financial Certification Authority -# Label: "CFCA EV ROOT" -# Serial: 407555286 -# MD5 Fingerprint: 74:e1:b6:ed:26:7a:7a:44:30:33:94:ab:7b:27:81:30 -# SHA1 Fingerprint: e2:b8:29:4b:55:84:ab:6b:58:c2:90:46:6c:ac:3f:b8:39:8f:84:83 -# SHA256 Fingerprint: 5c:c3:d7:8e:4e:1d:5e:45:54:7a:04:e6:87:3e:64:f9:0c:f9:53:6d:1c:cc:2e:f8:00:f3:55:c4:c5:fd:70:fd ------BEGIN CERTIFICATE----- -MIIFjTCCA3WgAwIBAgIEGErM1jANBgkqhkiG9w0BAQsFADBWMQswCQYDVQQGEwJD -TjEwMC4GA1UECgwnQ2hpbmEgRmluYW5jaWFsIENlcnRpZmljYXRpb24gQXV0aG9y -aXR5MRUwEwYDVQQDDAxDRkNBIEVWIFJPT1QwHhcNMTIwODA4MDMwNzAxWhcNMjkx -MjMxMDMwNzAxWjBWMQswCQYDVQQGEwJDTjEwMC4GA1UECgwnQ2hpbmEgRmluYW5j -aWFsIENlcnRpZmljYXRpb24gQXV0aG9yaXR5MRUwEwYDVQQDDAxDRkNBIEVWIFJP -T1QwggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQDXXWvNED8fBVnVBU03 -sQ7smCuOFR36k0sXgiFxEFLXUWRwFsJVaU2OFW2fvwwbwuCjZ9YMrM8irq93VCpL -TIpTUnrD7i7es3ElweldPe6hL6P3KjzJIx1qqx2hp/Hz7KDVRM8Vz3IvHWOX6Jn5 -/ZOkVIBMUtRSqy5J35DNuF++P96hyk0g1CXohClTt7GIH//62pCfCqktQT+x8Rgp -7hZZLDRJGqgG16iI0gNyejLi6mhNbiyWZXvKWfry4t3uMCz7zEasxGPrb382KzRz -EpR/38wmnvFyXVBlWY9ps4deMm/DGIq1lY+wejfeWkU7xzbh72fROdOXW3NiGUgt -hxwG+3SYIElz8AXSG7Ggo7cbcNOIabla1jj0Ytwli3i/+Oh+uFzJlU9fpy25IGvP -a931DfSCt/SyZi4QKPaXWnuWFo8BGS1sbn85WAZkgwGDg8NNkt0yxoekN+kWzqot -aK8KgWU6cMGbrU1tVMoqLUuFG7OA5nBFDWteNfB/O7ic5ARwiRIlk9oKmSJgamNg -TnYGmE69g60dWIolhdLHZR4tjsbftsbhf4oEIRUpdPA+nJCdDC7xij5aqgwJHsfV -PKPtl8MeNPo4+QgO48BdK4PRVmrJtqhUUy54Mmc9gn900PvhtgVguXDbjgv5E1hv -cWAQUhC5wUEJ73IfZzF4/5YFjQIDAQABo2MwYTAfBgNVHSMEGDAWgBTj/i39KNAL -tbq2osS/BqoFjJP7LzAPBgNVHRMBAf8EBTADAQH/MA4GA1UdDwEB/wQEAwIBBjAd -BgNVHQ4EFgQU4/4t/SjQC7W6tqLEvwaqBYyT+y8wDQYJKoZIhvcNAQELBQADggIB -ACXGumvrh8vegjmWPfBEp2uEcwPenStPuiB/vHiyz5ewG5zz13ku9Ui20vsXiObT -ej/tUxPQ4i9qecsAIyjmHjdXNYmEwnZPNDatZ8POQQaIxffu2Bq41gt/UP+TqhdL -jOztUmCypAbqTuv0axn96/Ua4CUqmtzHQTb3yHQFhDmVOdYLO6Qn+gjYXB74BGBS -ESgoA//vU2YApUo0FmZ8/Qmkrp5nGm9BC2sGE5uPhnEFtC+NiWYzKXZUmhH4J/qy -P5Hgzg0b8zAarb8iXRvTvyUFTeGSGn+ZnzxEk8rUQElsgIfXBDrDMlI1Dlb4pd19 -xIsNER9Tyx6yF7Zod1rg1MvIB671Oi6ON7fQAUtDKXeMOZePglr4UeWJoBjnaH9d -Ci77o0cOPaYjesYBx4/IXr9tgFa+iiS6M+qf4TIRnvHST4D2G0CvOJ4RUHlzEhLN -5mydLIhyPDCBBpEi6lmt2hkuIsKNuYyH4Ga8cyNfIWRjgEj1oDwYPZTISEEdQLpe -/v5WOaHIz16eGWRGENoXkbcFgKyLmZJ956LYBws2J+dIeWCKw9cTXPhyQN9Ky8+Z -AAoACxGV2lZFA4gKn2fQ1XmxqI1AbQ3CekD6819kR5LLU7m7Wc5P/dAVUwHY3+vZ -5nbv0CO7O6l5s9UCKc2Jo5YPSjXnTkLAdc0Hz+Ys63su ------END CERTIFICATE----- - -# Issuer: CN=OISTE WISeKey Global Root GB CA O=WISeKey OU=OISTE Foundation Endorsed -# Subject: CN=OISTE WISeKey Global Root GB CA O=WISeKey OU=OISTE Foundation Endorsed -# Label: "OISTE WISeKey Global Root GB CA" -# Serial: 157768595616588414422159278966750757568 -# MD5 Fingerprint: a4:eb:b9:61:28:2e:b7:2f:98:b0:35:26:90:99:51:1d -# SHA1 Fingerprint: 0f:f9:40:76:18:d3:d7:6a:4b:98:f0:a8:35:9e:0c:fd:27:ac:cc:ed -# SHA256 Fingerprint: 6b:9c:08:e8:6e:b0:f7:67:cf:ad:65:cd:98:b6:21:49:e5:49:4a:67:f5:84:5e:7b:d1:ed:01:9f:27:b8:6b:d6 ------BEGIN CERTIFICATE----- -MIIDtTCCAp2gAwIBAgIQdrEgUnTwhYdGs/gjGvbCwDANBgkqhkiG9w0BAQsFADBt -MQswCQYDVQQGEwJDSDEQMA4GA1UEChMHV0lTZUtleTEiMCAGA1UECxMZT0lTVEUg -Rm91bmRhdGlvbiBFbmRvcnNlZDEoMCYGA1UEAxMfT0lTVEUgV0lTZUtleSBHbG9i -YWwgUm9vdCBHQiBDQTAeFw0xNDEyMDExNTAwMzJaFw0zOTEyMDExNTEwMzFaMG0x -CzAJBgNVBAYTAkNIMRAwDgYDVQQKEwdXSVNlS2V5MSIwIAYDVQQLExlPSVNURSBG -b3VuZGF0aW9uIEVuZG9yc2VkMSgwJgYDVQQDEx9PSVNURSBXSVNlS2V5IEdsb2Jh -bCBSb290IEdCIENBMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA2Be3 -HEokKtaXscriHvt9OO+Y9bI5mE4nuBFde9IllIiCFSZqGzG7qFshISvYD06fWvGx -WuR51jIjK+FTzJlFXHtPrby/h0oLS5daqPZI7H17Dc0hBt+eFf1Biki3IPShehtX -1F1Q/7pn2COZH8g/497/b1t3sWtuuMlk9+HKQUYOKXHQuSP8yYFfTvdv37+ErXNk -u7dCjmn21HYdfp2nuFeKUWdy19SouJVUQHMD9ur06/4oQnc/nSMbsrY9gBQHTC5P -99UKFg29ZkM3fiNDecNAhvVMKdqOmq0NpQSHiB6F4+lT1ZvIiwNjeOvgGUpuuy9r -M2RYk61pv48b74JIxwIDAQABo1EwTzALBgNVHQ8EBAMCAYYwDwYDVR0TAQH/BAUw -AwEB/zAdBgNVHQ4EFgQUNQ/INmNe4qPs+TtmFc5RUuORmj0wEAYJKwYBBAGCNxUB -BAMCAQAwDQYJKoZIhvcNAQELBQADggEBAEBM+4eymYGQfp3FsLAmzYh7KzKNbrgh -cViXfa43FK8+5/ea4n32cZiZBKpDdHij40lhPnOMTZTg+XHEthYOU3gf1qKHLwI5 -gSk8rxWYITD+KJAAjNHhy/peyP34EEY7onhCkRd0VQreUGdNZtGn//3ZwLWoo4rO -ZvUPQ82nK1d7Y0Zqqi5S2PTt4W2tKZB4SLrhI6qjiey1q5bAtEuiHZeeevJuQHHf -aPFlTc58Bd9TZaml8LGXBHAVRgOY1NK/VLSgWH1Sb9pWJmLU2NuJMW8c8CLC02Ic -Nc1MaRVUGpCY3useX8p3x8uOPUNpnJpY0CQ73xtAln41rYHHTnG6iBM= ------END CERTIFICATE----- - -# Issuer: CN=SZAFIR ROOT CA2 O=Krajowa Izba Rozliczeniowa S.A. -# Subject: CN=SZAFIR ROOT CA2 O=Krajowa Izba Rozliczeniowa S.A. -# Label: "SZAFIR ROOT CA2" -# Serial: 357043034767186914217277344587386743377558296292 -# MD5 Fingerprint: 11:64:c1:89:b0:24:b1:8c:b1:07:7e:89:9e:51:9e:99 -# SHA1 Fingerprint: e2:52:fa:95:3f:ed:db:24:60:bd:6e:28:f3:9c:cc:cf:5e:b3:3f:de -# SHA256 Fingerprint: a1:33:9d:33:28:1a:0b:56:e5:57:d3:d3:2b:1c:e7:f9:36:7e:b0:94:bd:5f:a7:2a:7e:50:04:c8:de:d7:ca:fe ------BEGIN CERTIFICATE----- -MIIDcjCCAlqgAwIBAgIUPopdB+xV0jLVt+O2XwHrLdzk1uQwDQYJKoZIhvcNAQEL -BQAwUTELMAkGA1UEBhMCUEwxKDAmBgNVBAoMH0tyYWpvd2EgSXpiYSBSb3psaWN6 -ZW5pb3dhIFMuQS4xGDAWBgNVBAMMD1NaQUZJUiBST09UIENBMjAeFw0xNTEwMTkw -NzQzMzBaFw0zNTEwMTkwNzQzMzBaMFExCzAJBgNVBAYTAlBMMSgwJgYDVQQKDB9L -cmFqb3dhIEl6YmEgUm96bGljemVuaW93YSBTLkEuMRgwFgYDVQQDDA9TWkFGSVIg -Uk9PVCBDQTIwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC3vD5QqEvN -QLXOYeeWyrSh2gwisPq1e3YAd4wLz32ohswmUeQgPYUM1ljj5/QqGJ3a0a4m7utT -3PSQ1hNKDJA8w/Ta0o4NkjrcsbH/ON7Dui1fgLkCvUqdGw+0w8LBZwPd3BucPbOw -3gAeqDRHu5rr/gsUvTaE2g0gv/pby6kWIK05YO4vdbbnl5z5Pv1+TW9NL++IDWr6 -3fE9biCloBK0TXC5ztdyO4mTp4CEHCdJckm1/zuVnsHMyAHs6A6KCpbns6aH5db5 -BSsNl0BwPLqsdVqc1U2dAgrSS5tmS0YHF2Wtn2yIANwiieDhZNRnvDF5YTy7ykHN -XGoAyDw4jlivAgMBAAGjQjBAMA8GA1UdEwEB/wQFMAMBAf8wDgYDVR0PAQH/BAQD -AgEGMB0GA1UdDgQWBBQuFqlKGLXLzPVvUPMjX/hd56zwyDANBgkqhkiG9w0BAQsF -AAOCAQEAtXP4A9xZWx126aMqe5Aosk3AM0+qmrHUuOQn/6mWmc5G4G18TKI4pAZw -8PRBEew/R40/cof5O/2kbytTAOD/OblqBw7rHRz2onKQy4I9EYKL0rufKq8h5mOG -nXkZ7/e7DDWQw4rtTw/1zBLZpD67oPwglV9PJi8RI4NOdQcPv5vRtB3pEAT+ymCP -oky4rc/hkA/NrgrHXXu3UNLUYfrVFdvXn4dRVOul4+vJhaAlIDf7js4MNIThPIGy -d05DpYhfhmehPea0XGG2Ptv+tyjFogeutcrKjSoS75ftwjCkySp6+/NNIxuZMzSg -LvWpCz/UXeHPhJ/iGcJfitYgHuNztw== ------END CERTIFICATE----- - -# Issuer: CN=Certum Trusted Network CA 2 O=Unizeto Technologies S.A. OU=Certum Certification Authority -# Subject: CN=Certum Trusted Network CA 2 O=Unizeto Technologies S.A. OU=Certum Certification Authority -# Label: "Certum Trusted Network CA 2" -# Serial: 44979900017204383099463764357512596969 -# MD5 Fingerprint: 6d:46:9e:d9:25:6d:08:23:5b:5e:74:7d:1e:27:db:f2 -# SHA1 Fingerprint: d3:dd:48:3e:2b:bf:4c:05:e8:af:10:f5:fa:76:26:cf:d3:dc:30:92 -# SHA256 Fingerprint: b6:76:f2:ed:da:e8:77:5c:d3:6c:b0:f6:3c:d1:d4:60:39:61:f4:9e:62:65:ba:01:3a:2f:03:07:b6:d0:b8:04 ------BEGIN CERTIFICATE----- -MIIF0jCCA7qgAwIBAgIQIdbQSk8lD8kyN/yqXhKN6TANBgkqhkiG9w0BAQ0FADCB -gDELMAkGA1UEBhMCUEwxIjAgBgNVBAoTGVVuaXpldG8gVGVjaG5vbG9naWVzIFMu -QS4xJzAlBgNVBAsTHkNlcnR1bSBDZXJ0aWZpY2F0aW9uIEF1dGhvcml0eTEkMCIG -A1UEAxMbQ2VydHVtIFRydXN0ZWQgTmV0d29yayBDQSAyMCIYDzIwMTExMDA2MDgz -OTU2WhgPMjA0NjEwMDYwODM5NTZaMIGAMQswCQYDVQQGEwJQTDEiMCAGA1UEChMZ -VW5pemV0byBUZWNobm9sb2dpZXMgUy5BLjEnMCUGA1UECxMeQ2VydHVtIENlcnRp -ZmljYXRpb24gQXV0aG9yaXR5MSQwIgYDVQQDExtDZXJ0dW0gVHJ1c3RlZCBOZXR3 -b3JrIENBIDIwggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQC9+Xj45tWA -DGSdhhuWZGc/IjoedQF97/tcZ4zJzFxrqZHmuULlIEub2pt7uZld2ZuAS9eEQCsn -0+i6MLs+CRqnSZXvK0AkwpfHp+6bJe+oCgCXhVqqndwpyeI1B+twTUrWwbNWuKFB -OJvR+zF/j+Bf4bE/D44WSWDXBo0Y+aomEKsq09DRZ40bRr5HMNUuctHFY9rnY3lE -fktjJImGLjQ/KUxSiyqnwOKRKIm5wFv5HdnnJ63/mgKXwcZQkpsCLL2puTRZCr+E -Sv/f/rOf69me4Jgj7KZrdxYq28ytOxykh9xGc14ZYmhFV+SQgkK7QtbwYeDBoz1m -o130GO6IyY0XRSmZMnUCMe4pJshrAua1YkV/NxVaI2iJ1D7eTiew8EAMvE0Xy02i -sx7QBlrd9pPPV3WZ9fqGGmd4s7+W/jTcvedSVuWz5XV710GRBdxdaeOVDUO5/IOW -OZV7bIBaTxNyxtd9KXpEulKkKtVBRgkg/iKgtlswjbyJDNXXcPiHUv3a76xRLgez -Tv7QCdpw75j6VuZt27VXS9zlLCUVyJ4ueE742pyehizKV/Ma5ciSixqClnrDvFAS -adgOWkaLOusm+iPJtrCBvkIApPjW/jAux9JG9uWOdf3yzLnQh1vMBhBgu4M1t15n -3kfsmUjxpKEV/q2MYo45VU85FrmxY53/twIDAQABo0IwQDAPBgNVHRMBAf8EBTAD -AQH/MB0GA1UdDgQWBBS2oVQ5AsOgP46KvPrU+Bym0ToO/TAOBgNVHQ8BAf8EBAMC -AQYwDQYJKoZIhvcNAQENBQADggIBAHGlDs7k6b8/ONWJWsQCYftMxRQXLYtPU2sQ -F/xlhMcQSZDe28cmk4gmb3DWAl45oPePq5a1pRNcgRRtDoGCERuKTsZPpd1iHkTf -CVn0W3cLN+mLIMb4Ck4uWBzrM9DPhmDJ2vuAL55MYIR4PSFk1vtBHxgP58l1cb29 -XN40hz5BsA72udY/CROWFC/emh1auVbONTqwX3BNXuMp8SMoclm2q8KMZiYcdywm -djWLKKdpoPk79SPdhRB0yZADVpHnr7pH1BKXESLjokmUbOe3lEu6LaTaM4tMpkT/ -WjzGHWTYtTHkpjx6qFcL2+1hGsvxznN3Y6SHb0xRONbkX8eftoEq5IVIeVheO/jb -AoJnwTnbw3RLPTYe+SmTiGhbqEQZIfCn6IENLOiTNrQ3ssqwGyZ6miUfmpqAnksq -P/ujmv5zMnHCnsZy4YpoJ/HkD7TETKVhk/iXEAcqMCWpuchxuO9ozC1+9eB+D4Ko -b7a6bINDd82Kkhehnlt4Fj1F4jNy3eFmypnTycUm/Q1oBEauttmbjL4ZvrHG8hnj -XALKLNhvSgfZyTXaQHXyxKcZb55CEJh15pWLYLztxRLXis7VmFxWlgPF7ncGNf/P -5O4/E2Hu29othfDNrp2yGAlFw5Khchf8R7agCyzxxN5DaAhqXzvwdmP7zAYspsbi -DrW5viSP ------END CERTIFICATE----- - -# Issuer: CN=Hellenic Academic and Research Institutions RootCA 2015 O=Hellenic Academic and Research Institutions Cert. Authority -# Subject: CN=Hellenic Academic and Research Institutions RootCA 2015 O=Hellenic Academic and Research Institutions Cert. Authority -# Label: "Hellenic Academic and Research Institutions RootCA 2015" -# Serial: 0 -# MD5 Fingerprint: ca:ff:e2:db:03:d9:cb:4b:e9:0f:ad:84:fd:7b:18:ce -# SHA1 Fingerprint: 01:0c:06:95:a6:98:19:14:ff:bf:5f:c6:b0:b6:95:ea:29:e9:12:a6 -# SHA256 Fingerprint: a0:40:92:9a:02:ce:53:b4:ac:f4:f2:ff:c6:98:1c:e4:49:6f:75:5e:6d:45:fe:0b:2a:69:2b:cd:52:52:3f:36 ------BEGIN CERTIFICATE----- -MIIGCzCCA/OgAwIBAgIBADANBgkqhkiG9w0BAQsFADCBpjELMAkGA1UEBhMCR1Ix -DzANBgNVBAcTBkF0aGVuczFEMEIGA1UEChM7SGVsbGVuaWMgQWNhZGVtaWMgYW5k -IFJlc2VhcmNoIEluc3RpdHV0aW9ucyBDZXJ0LiBBdXRob3JpdHkxQDA+BgNVBAMT -N0hlbGxlbmljIEFjYWRlbWljIGFuZCBSZXNlYXJjaCBJbnN0aXR1dGlvbnMgUm9v -dENBIDIwMTUwHhcNMTUwNzA3MTAxMTIxWhcNNDAwNjMwMTAxMTIxWjCBpjELMAkG -A1UEBhMCR1IxDzANBgNVBAcTBkF0aGVuczFEMEIGA1UEChM7SGVsbGVuaWMgQWNh -ZGVtaWMgYW5kIFJlc2VhcmNoIEluc3RpdHV0aW9ucyBDZXJ0LiBBdXRob3JpdHkx -QDA+BgNVBAMTN0hlbGxlbmljIEFjYWRlbWljIGFuZCBSZXNlYXJjaCBJbnN0aXR1 -dGlvbnMgUm9vdENBIDIwMTUwggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoIC -AQDC+Kk/G4n8PDwEXT2QNrCROnk8ZlrvbTkBSRq0t89/TSNTt5AA4xMqKKYx8ZEA -4yjsriFBzh/a/X0SWwGDD7mwX5nh8hKDgE0GPt+sr+ehiGsxr/CL0BgzuNtFajT0 -AoAkKAoCFZVedioNmToUW/bLy1O8E00BiDeUJRtCvCLYjqOWXjrZMts+6PAQZe10 -4S+nfK8nNLspfZu2zwnI5dMK/IhlZXQK3HMcXM1AsRzUtoSMTFDPaI6oWa7CJ06C -ojXdFPQf/7J31Ycvqm59JCfnxssm5uX+Zwdj2EUN3TpZZTlYepKZcj2chF6IIbjV -9Cz82XBST3i4vTwri5WY9bPRaM8gFH5MXF/ni+X1NYEZN9cRCLdmvtNKzoNXADrD -gfgXy5I2XdGj2HUb4Ysn6npIQf1FGQatJ5lOwXBH3bWfgVMS5bGMSF0xQxfjjMZ6 -Y5ZLKTBOhE5iGV48zpeQpX8B653g+IuJ3SWYPZK2fu/Z8VFRfS0myGlZYeCsargq -NhEEelC9MoS+L9xy1dcdFkfkR2YgP/SWxa+OAXqlD3pk9Q0Yh9muiNX6hME6wGko -LfINaFGq46V3xqSQDqE3izEjR8EJCOtu93ib14L8hCCZSRm2Ekax+0VVFqmjZayc -Bw/qa9wfLgZy7IaIEuQt218FL+TwA9MmM+eAws1CoRc0CwIDAQABo0IwQDAPBgNV -HRMBAf8EBTADAQH/MA4GA1UdDwEB/wQEAwIBBjAdBgNVHQ4EFgQUcRVnyMjJvXVd -ctA4GGqd83EkVAswDQYJKoZIhvcNAQELBQADggIBAHW7bVRLqhBYRjTyYtcWNl0I -XtVsyIe9tC5G8jH4fOpCtZMWVdyhDBKg2mF+D1hYc2Ryx+hFjtyp8iY/xnmMsVMI -M4GwVhO+5lFc2JsKT0ucVlMC6U/2DWDqTUJV6HwbISHTGzrMd/K4kPFox/la/vot -9L/J9UUbzjgQKjeKeaO04wlshYaT/4mWJ3iBj2fjRnRUjtkNaeJK9E10A/+yd+2V -Z5fkscWrv2oj6NSU4kQoYsRL4vDY4ilrGnB+JGGTe08DMiUNRSQrlrRGar9KC/ea -j8GsGsVn82800vpzY4zvFrCopEYq+OsS7HK07/grfoxSwIuEVPkvPuNVqNxmsdnh -X9izjFk0WaSrT2y7HxjbdavYy5LNlDhhDgcGH0tGEPEVvo2FXDtKK4F5D7Rpn0lQ -l033DlZdwJVqwjbDG2jJ9SrcR5q+ss7FJej6A7na+RZukYT1HCjI/CbM1xyQVqdf -bzoEvM14iQuODy+jqk+iGxI9FghAD/FGTNeqewjBCvVtJ94Cj8rDtSvK6evIIVM4 -pcw72Hc3MKJP2W/R8kCtQXoXxdZKNYm3QdV8hn9VTYNKpXMgwDqvkPGaJI7ZjnHK -e7iG2rKPmT4dEw0SEe7Uq/DpFXYC5ODfqiAeW2GFZECpkJcNrVPSWh2HagCXZWK0 -vm9qp/UsQu0yrbYhnr68 ------END CERTIFICATE----- - -# Issuer: CN=Hellenic Academic and Research Institutions ECC RootCA 2015 O=Hellenic Academic and Research Institutions Cert. Authority -# Subject: CN=Hellenic Academic and Research Institutions ECC RootCA 2015 O=Hellenic Academic and Research Institutions Cert. Authority -# Label: "Hellenic Academic and Research Institutions ECC RootCA 2015" -# Serial: 0 -# MD5 Fingerprint: 81:e5:b4:17:eb:c2:f5:e1:4b:0d:41:7b:49:92:fe:ef -# SHA1 Fingerprint: 9f:f1:71:8d:92:d5:9a:f3:7d:74:97:b4:bc:6f:84:68:0b:ba:b6:66 -# SHA256 Fingerprint: 44:b5:45:aa:8a:25:e6:5a:73:ca:15:dc:27:fc:36:d2:4c:1c:b9:95:3a:06:65:39:b1:15:82:dc:48:7b:48:33 ------BEGIN CERTIFICATE----- -MIICwzCCAkqgAwIBAgIBADAKBggqhkjOPQQDAjCBqjELMAkGA1UEBhMCR1IxDzAN -BgNVBAcTBkF0aGVuczFEMEIGA1UEChM7SGVsbGVuaWMgQWNhZGVtaWMgYW5kIFJl -c2VhcmNoIEluc3RpdHV0aW9ucyBDZXJ0LiBBdXRob3JpdHkxRDBCBgNVBAMTO0hl -bGxlbmljIEFjYWRlbWljIGFuZCBSZXNlYXJjaCBJbnN0aXR1dGlvbnMgRUNDIFJv -b3RDQSAyMDE1MB4XDTE1MDcwNzEwMzcxMloXDTQwMDYzMDEwMzcxMlowgaoxCzAJ -BgNVBAYTAkdSMQ8wDQYDVQQHEwZBdGhlbnMxRDBCBgNVBAoTO0hlbGxlbmljIEFj -YWRlbWljIGFuZCBSZXNlYXJjaCBJbnN0aXR1dGlvbnMgQ2VydC4gQXV0aG9yaXR5 -MUQwQgYDVQQDEztIZWxsZW5pYyBBY2FkZW1pYyBhbmQgUmVzZWFyY2ggSW5zdGl0 -dXRpb25zIEVDQyBSb290Q0EgMjAxNTB2MBAGByqGSM49AgEGBSuBBAAiA2IABJKg -QehLgoRc4vgxEZmGZE4JJS+dQS8KrjVPdJWyUWRrjWvmP3CV8AVER6ZyOFB2lQJa -jq4onvktTpnvLEhvTCUp6NFxW98dwXU3tNf6e3pCnGoKVlp8aQuqgAkkbH7BRqNC -MEAwDwYDVR0TAQH/BAUwAwEB/zAOBgNVHQ8BAf8EBAMCAQYwHQYDVR0OBBYEFLQi -C4KZJAEOnLvkDv2/+5cgk5kqMAoGCCqGSM49BAMCA2cAMGQCMGfOFmI4oqxiRaep -lSTAGiecMjvAwNW6qef4BENThe5SId6d9SWDPp5YSy/XZxMOIQIwBeF1Ad5o7Sof -TUwJCA3sS61kFyjndc5FZXIhF8siQQ6ME5g4mlRtm8rifOoCWCKR ------END CERTIFICATE----- - -# Issuer: CN=ISRG Root X1 O=Internet Security Research Group -# Subject: CN=ISRG Root X1 O=Internet Security Research Group -# Label: "ISRG Root X1" -# Serial: 172886928669790476064670243504169061120 -# MD5 Fingerprint: 0c:d2:f9:e0:da:17:73:e9:ed:86:4d:a5:e3:70:e7:4e -# SHA1 Fingerprint: ca:bd:2a:79:a1:07:6a:31:f2:1d:25:36:35:cb:03:9d:43:29:a5:e8 -# SHA256 Fingerprint: 96:bc:ec:06:26:49:76:f3:74:60:77:9a:cf:28:c5:a7:cf:e8:a3:c0:aa:e1:1a:8f:fc:ee:05:c0:bd:df:08:c6 ------BEGIN CERTIFICATE----- -MIIFazCCA1OgAwIBAgIRAIIQz7DSQONZRGPgu2OCiwAwDQYJKoZIhvcNAQELBQAw -TzELMAkGA1UEBhMCVVMxKTAnBgNVBAoTIEludGVybmV0IFNlY3VyaXR5IFJlc2Vh -cmNoIEdyb3VwMRUwEwYDVQQDEwxJU1JHIFJvb3QgWDEwHhcNMTUwNjA0MTEwNDM4 -WhcNMzUwNjA0MTEwNDM4WjBPMQswCQYDVQQGEwJVUzEpMCcGA1UEChMgSW50ZXJu -ZXQgU2VjdXJpdHkgUmVzZWFyY2ggR3JvdXAxFTATBgNVBAMTDElTUkcgUm9vdCBY -MTCCAiIwDQYJKoZIhvcNAQEBBQADggIPADCCAgoCggIBAK3oJHP0FDfzm54rVygc -h77ct984kIxuPOZXoHj3dcKi/vVqbvYATyjb3miGbESTtrFj/RQSa78f0uoxmyF+ -0TM8ukj13Xnfs7j/EvEhmkvBioZxaUpmZmyPfjxwv60pIgbz5MDmgK7iS4+3mX6U -A5/TR5d8mUgjU+g4rk8Kb4Mu0UlXjIB0ttov0DiNewNwIRt18jA8+o+u3dpjq+sW -T8KOEUt+zwvo/7V3LvSye0rgTBIlDHCNAymg4VMk7BPZ7hm/ELNKjD+Jo2FR3qyH -B5T0Y3HsLuJvW5iB4YlcNHlsdu87kGJ55tukmi8mxdAQ4Q7e2RCOFvu396j3x+UC -B5iPNgiV5+I3lg02dZ77DnKxHZu8A/lJBdiB3QW0KtZB6awBdpUKD9jf1b0SHzUv -KBds0pjBqAlkd25HN7rOrFleaJ1/ctaJxQZBKT5ZPt0m9STJEadao0xAH0ahmbWn -OlFuhjuefXKnEgV4We0+UXgVCwOPjdAvBbI+e0ocS3MFEvzG6uBQE3xDk3SzynTn -jh8BCNAw1FtxNrQHusEwMFxIt4I7mKZ9YIqioymCzLq9gwQbooMDQaHWBfEbwrbw -qHyGO0aoSCqI3Haadr8faqU9GY/rOPNk3sgrDQoo//fb4hVC1CLQJ13hef4Y53CI -rU7m2Ys6xt0nUW7/vGT1M0NPAgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBBjAPBgNV -HRMBAf8EBTADAQH/MB0GA1UdDgQWBBR5tFnme7bl5AFzgAiIyBpY9umbbjANBgkq -hkiG9w0BAQsFAAOCAgEAVR9YqbyyqFDQDLHYGmkgJykIrGF1XIpu+ILlaS/V9lZL -ubhzEFnTIZd+50xx+7LSYK05qAvqFyFWhfFQDlnrzuBZ6brJFe+GnY+EgPbk6ZGQ -3BebYhtF8GaV0nxvwuo77x/Py9auJ/GpsMiu/X1+mvoiBOv/2X/qkSsisRcOj/KK -NFtY2PwByVS5uCbMiogziUwthDyC3+6WVwW6LLv3xLfHTjuCvjHIInNzktHCgKQ5 -ORAzI4JMPJ+GslWYHb4phowim57iaztXOoJwTdwJx4nLCgdNbOhdjsnvzqvHu7Ur -TkXWStAmzOVyyghqpZXjFaH3pO3JLF+l+/+sKAIuvtd7u+Nxe5AW0wdeRlN8NwdC -jNPElpzVmbUq4JUagEiuTDkHzsxHpFKVK7q4+63SM1N95R1NbdWhscdCb+ZAJzVc -oyi3B43njTOQ5yOf+1CceWxG1bQVs5ZufpsMljq4Ui0/1lvh+wjChP4kqKOJ2qxq -4RgqsahDYVvTH9w7jXbyLeiNdd8XM2w9U/t7y0Ff/9yi0GE44Za4rF2LN9d11TPA -mRGunUHBcnWEvgJBQl9nJEiU0Zsnvgc/ubhPgXRR4Xq37Z0j4r7g1SgEEzwxA57d -emyPxgcYxn/eR44/KJ4EBs+lVDR3veyJm+kXQ99b21/+jh5Xos1AnX5iItreGCc= ------END CERTIFICATE----- - -# Issuer: O=FNMT-RCM OU=AC RAIZ FNMT-RCM -# Subject: O=FNMT-RCM OU=AC RAIZ FNMT-RCM -# Label: "AC RAIZ FNMT-RCM" -# Serial: 485876308206448804701554682760554759 -# MD5 Fingerprint: e2:09:04:b4:d3:bd:d1:a0:14:fd:1a:d2:47:c4:57:1d -# SHA1 Fingerprint: ec:50:35:07:b2:15:c4:95:62:19:e2:a8:9a:5b:42:99:2c:4c:2c:20 -# SHA256 Fingerprint: eb:c5:57:0c:29:01:8c:4d:67:b1:aa:12:7b:af:12:f7:03:b4:61:1e:bc:17:b7:da:b5:57:38:94:17:9b:93:fa ------BEGIN CERTIFICATE----- -MIIFgzCCA2ugAwIBAgIPXZONMGc2yAYdGsdUhGkHMA0GCSqGSIb3DQEBCwUAMDsx -CzAJBgNVBAYTAkVTMREwDwYDVQQKDAhGTk1ULVJDTTEZMBcGA1UECwwQQUMgUkFJ -WiBGTk1ULVJDTTAeFw0wODEwMjkxNTU5NTZaFw0zMDAxMDEwMDAwMDBaMDsxCzAJ -BgNVBAYTAkVTMREwDwYDVQQKDAhGTk1ULVJDTTEZMBcGA1UECwwQQUMgUkFJWiBG -Tk1ULVJDTTCCAiIwDQYJKoZIhvcNAQEBBQADggIPADCCAgoCggIBALpxgHpMhm5/ -yBNtwMZ9HACXjywMI7sQmkCpGreHiPibVmr75nuOi5KOpyVdWRHbNi63URcfqQgf -BBckWKo3Shjf5TnUV/3XwSyRAZHiItQDwFj8d0fsjz50Q7qsNI1NOHZnjrDIbzAz -WHFctPVrbtQBULgTfmxKo0nRIBnuvMApGGWn3v7v3QqQIecaZ5JCEJhfTzC8PhxF -tBDXaEAUwED653cXeuYLj2VbPNmaUtu1vZ5Gzz3rkQUCwJaydkxNEJY7kvqcfw+Z -374jNUUeAlz+taibmSXaXvMiwzn15Cou08YfxGyqxRxqAQVKL9LFwag0Jl1mpdIC -IfkYtwb1TplvqKtMUejPUBjFd8g5CSxJkjKZqLsXF3mwWsXmo8RZZUc1g16p6DUL -mbvkzSDGm0oGObVo/CK67lWMK07q87Hj/LaZmtVC+nFNCM+HHmpxffnTtOmlcYF7 -wk5HlqX2doWjKI/pgG6BU6VtX7hI+cL5NqYuSf+4lsKMB7ObiFj86xsc3i1w4peS -MKGJ47xVqCfWS+2QrYv6YyVZLag13cqXM7zlzced0ezvXg5KkAYmY6252TUtB7p2 -ZSysV4999AeU14ECll2jB0nVetBX+RvnU0Z1qrB5QstocQjpYL05ac70r8NWQMet -UqIJ5G+GR4of6ygnXYMgrwTJbFaai0b1AgMBAAGjgYMwgYAwDwYDVR0TAQH/BAUw -AwEB/zAOBgNVHQ8BAf8EBAMCAQYwHQYDVR0OBBYEFPd9xf3E6Jobd2Sn9R2gzL+H -YJptMD4GA1UdIAQ3MDUwMwYEVR0gADArMCkGCCsGAQUFBwIBFh1odHRwOi8vd3d3 -LmNlcnQuZm5tdC5lcy9kcGNzLzANBgkqhkiG9w0BAQsFAAOCAgEAB5BK3/MjTvDD -nFFlm5wioooMhfNzKWtN/gHiqQxjAb8EZ6WdmF/9ARP67Jpi6Yb+tmLSbkyU+8B1 -RXxlDPiyN8+sD8+Nb/kZ94/sHvJwnvDKuO+3/3Y3dlv2bojzr2IyIpMNOmqOFGYM -LVN0V2Ue1bLdI4E7pWYjJ2cJj+F3qkPNZVEI7VFY/uY5+ctHhKQV8Xa7pO6kO8Rf -77IzlhEYt8llvhjho6Tc+hj507wTmzl6NLrTQfv6MooqtyuGC2mDOL7Nii4LcK2N -JpLuHvUBKwrZ1pebbuCoGRw6IYsMHkCtA+fdZn71uSANA+iW+YJF1DngoABd15jm -fZ5nc8OaKveri6E6FO80vFIOiZiaBECEHX5FaZNXzuvO+FB8TxxuBEOb+dY7Ixjp -6o7RTUaN8Tvkasq6+yO3m/qZASlaWFot4/nUbQ4mrcFuNLwy+AwF+mWj2zs3gyLp -1txyM/1d8iC9djwj2ij3+RvrWWTV3F9yfiD8zYm1kGdNYno/Tq0dwzn+evQoFt9B -9kiABdcPUXmsEKvU7ANm5mqwujGSQkBqvjrTcuFqN1W8rB2Vt2lh8kORdOag0wok -RqEIr9baRRmW1FMdW4R58MD3R++Lj8UGrp1MYp3/RgT408m2ECVAdf4WqslKYIYv -uu8wd+RU4riEmViAqhOLUTpPSPaLtrM= ------END CERTIFICATE----- - -# Issuer: CN=Amazon Root CA 1 O=Amazon -# Subject: CN=Amazon Root CA 1 O=Amazon -# Label: "Amazon Root CA 1" -# Serial: 143266978916655856878034712317230054538369994 -# MD5 Fingerprint: 43:c6:bf:ae:ec:fe:ad:2f:18:c6:88:68:30:fc:c8:e6 -# SHA1 Fingerprint: 8d:a7:f9:65:ec:5e:fc:37:91:0f:1c:6e:59:fd:c1:cc:6a:6e:de:16 -# SHA256 Fingerprint: 8e:cd:e6:88:4f:3d:87:b1:12:5b:a3:1a:c3:fc:b1:3d:70:16:de:7f:57:cc:90:4f:e1:cb:97:c6:ae:98:19:6e ------BEGIN CERTIFICATE----- -MIIDQTCCAimgAwIBAgITBmyfz5m/jAo54vB4ikPmljZbyjANBgkqhkiG9w0BAQsF -ADA5MQswCQYDVQQGEwJVUzEPMA0GA1UEChMGQW1hem9uMRkwFwYDVQQDExBBbWF6 -b24gUm9vdCBDQSAxMB4XDTE1MDUyNjAwMDAwMFoXDTM4MDExNzAwMDAwMFowOTEL -MAkGA1UEBhMCVVMxDzANBgNVBAoTBkFtYXpvbjEZMBcGA1UEAxMQQW1hem9uIFJv -b3QgQ0EgMTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALJ4gHHKeNXj -ca9HgFB0fW7Y14h29Jlo91ghYPl0hAEvrAIthtOgQ3pOsqTQNroBvo3bSMgHFzZM -9O6II8c+6zf1tRn4SWiw3te5djgdYZ6k/oI2peVKVuRF4fn9tBb6dNqcmzU5L/qw -IFAGbHrQgLKm+a/sRxmPUDgH3KKHOVj4utWp+UhnMJbulHheb4mjUcAwhmahRWa6 -VOujw5H5SNz/0egwLX0tdHA114gk957EWW67c4cX8jJGKLhD+rcdqsq08p8kDi1L -93FcXmn/6pUCyziKrlA4b9v7LWIbxcceVOF34GfID5yHI9Y/QCB/IIDEgEw+OyQm -jgSubJrIqg0CAwEAAaNCMEAwDwYDVR0TAQH/BAUwAwEB/zAOBgNVHQ8BAf8EBAMC -AYYwHQYDVR0OBBYEFIQYzIU07LwMlJQuCFmcx7IQTgoIMA0GCSqGSIb3DQEBCwUA -A4IBAQCY8jdaQZChGsV2USggNiMOruYou6r4lK5IpDB/G/wkjUu0yKGX9rbxenDI -U5PMCCjjmCXPI6T53iHTfIUJrU6adTrCC2qJeHZERxhlbI1Bjjt/msv0tadQ1wUs -N+gDS63pYaACbvXy8MWy7Vu33PqUXHeeE6V/Uq2V8viTO96LXFvKWlJbYK8U90vv -o/ufQJVtMVT8QtPHRh8jrdkPSHCa2XV4cdFyQzR1bldZwgJcJmApzyMZFo6IQ6XU -5MsI+yMRQ+hDKXJioaldXgjUkK642M4UwtBV8ob2xJNDd2ZhwLnoQdeXeGADbkpy -rqXRfboQnoZsG4q5WTP468SQvvG5 ------END CERTIFICATE----- - -# Issuer: CN=Amazon Root CA 2 O=Amazon -# Subject: CN=Amazon Root CA 2 O=Amazon -# Label: "Amazon Root CA 2" -# Serial: 143266982885963551818349160658925006970653239 -# MD5 Fingerprint: c8:e5:8d:ce:a8:42:e2:7a:c0:2a:5c:7c:9e:26:bf:66 -# SHA1 Fingerprint: 5a:8c:ef:45:d7:a6:98:59:76:7a:8c:8b:44:96:b5:78:cf:47:4b:1a -# SHA256 Fingerprint: 1b:a5:b2:aa:8c:65:40:1a:82:96:01:18:f8:0b:ec:4f:62:30:4d:83:ce:c4:71:3a:19:c3:9c:01:1e:a4:6d:b4 ------BEGIN CERTIFICATE----- -MIIFQTCCAymgAwIBAgITBmyf0pY1hp8KD+WGePhbJruKNzANBgkqhkiG9w0BAQwF -ADA5MQswCQYDVQQGEwJVUzEPMA0GA1UEChMGQW1hem9uMRkwFwYDVQQDExBBbWF6 -b24gUm9vdCBDQSAyMB4XDTE1MDUyNjAwMDAwMFoXDTQwMDUyNjAwMDAwMFowOTEL -MAkGA1UEBhMCVVMxDzANBgNVBAoTBkFtYXpvbjEZMBcGA1UEAxMQQW1hem9uIFJv -b3QgQ0EgMjCCAiIwDQYJKoZIhvcNAQEBBQADggIPADCCAgoCggIBAK2Wny2cSkxK -gXlRmeyKy2tgURO8TW0G/LAIjd0ZEGrHJgw12MBvIITplLGbhQPDW9tK6Mj4kHbZ -W0/jTOgGNk3Mmqw9DJArktQGGWCsN0R5hYGCrVo34A3MnaZMUnbqQ523BNFQ9lXg -1dKmSYXpN+nKfq5clU1Imj+uIFptiJXZNLhSGkOQsL9sBbm2eLfq0OQ6PBJTYv9K -8nu+NQWpEjTj82R0Yiw9AElaKP4yRLuH3WUnAnE72kr3H9rN9yFVkE8P7K6C4Z9r -2UXTu/Bfh+08LDmG2j/e7HJV63mjrdvdfLC6HM783k81ds8P+HgfajZRRidhW+me -z/CiVX18JYpvL7TFz4QuK/0NURBs+18bvBt+xa47mAExkv8LV/SasrlX6avvDXbR -8O70zoan4G7ptGmh32n2M8ZpLpcTnqWHsFcQgTfJU7O7f/aS0ZzQGPSSbtqDT6Zj -mUyl+17vIWR6IF9sZIUVyzfpYgwLKhbcAS4y2j5L9Z469hdAlO+ekQiG+r5jqFoz -7Mt0Q5X5bGlSNscpb/xVA1wf+5+9R+vnSUeVC06JIglJ4PVhHvG/LopyboBZ/1c6 -+XUyo05f7O0oYtlNc/LMgRdg7c3r3NunysV+Ar3yVAhU/bQtCSwXVEqY0VThUWcI -0u1ufm8/0i2BWSlmy5A5lREedCf+3euvAgMBAAGjQjBAMA8GA1UdEwEB/wQFMAMB -Af8wDgYDVR0PAQH/BAQDAgGGMB0GA1UdDgQWBBSwDPBMMPQFWAJI/TPlUq9LhONm -UjANBgkqhkiG9w0BAQwFAAOCAgEAqqiAjw54o+Ci1M3m9Zh6O+oAA7CXDpO8Wqj2 -LIxyh6mx/H9z/WNxeKWHWc8w4Q0QshNabYL1auaAn6AFC2jkR2vHat+2/XcycuUY -+gn0oJMsXdKMdYV2ZZAMA3m3MSNjrXiDCYZohMr/+c8mmpJ5581LxedhpxfL86kS -k5Nrp+gvU5LEYFiwzAJRGFuFjWJZY7attN6a+yb3ACfAXVU3dJnJUH/jWS5E4ywl -7uxMMne0nxrpS10gxdr9HIcWxkPo1LsmmkVwXqkLN1PiRnsn/eBG8om3zEK2yygm -btmlyTrIQRNg91CMFa6ybRoVGld45pIq2WWQgj9sAq+uEjonljYE1x2igGOpm/Hl -urR8FLBOybEfdF849lHqm/osohHUqS0nGkWxr7JOcQ3AWEbWaQbLU8uz/mtBzUF+ -fUwPfHJ5elnNXkoOrJupmHN5fLT0zLm4BwyydFy4x2+IoZCn9Kr5v2c69BoVYh63 -n749sSmvZ6ES8lgQGVMDMBu4Gon2nL2XA46jCfMdiyHxtN/kHNGfZQIG6lzWE7OE -76KlXIx3KadowGuuQNKotOrN8I1LOJwZmhsoVLiJkO/KdYE+HvJkJMcYr07/R54H -9jVlpNMKVv/1F2Rs76giJUmTtt8AF9pYfl3uxRuw0dFfIRDH+fO6AgonB8Xx1sfT -4PsJYGw= ------END CERTIFICATE----- - -# Issuer: CN=Amazon Root CA 3 O=Amazon -# Subject: CN=Amazon Root CA 3 O=Amazon -# Label: "Amazon Root CA 3" -# Serial: 143266986699090766294700635381230934788665930 -# MD5 Fingerprint: a0:d4:ef:0b:f7:b5:d8:49:95:2a:ec:f5:c4:fc:81:87 -# SHA1 Fingerprint: 0d:44:dd:8c:3c:8c:1a:1a:58:75:64:81:e9:0f:2e:2a:ff:b3:d2:6e -# SHA256 Fingerprint: 18:ce:6c:fe:7b:f1:4e:60:b2:e3:47:b8:df:e8:68:cb:31:d0:2e:bb:3a:da:27:15:69:f5:03:43:b4:6d:b3:a4 ------BEGIN CERTIFICATE----- -MIIBtjCCAVugAwIBAgITBmyf1XSXNmY/Owua2eiedgPySjAKBggqhkjOPQQDAjA5 -MQswCQYDVQQGEwJVUzEPMA0GA1UEChMGQW1hem9uMRkwFwYDVQQDExBBbWF6b24g -Um9vdCBDQSAzMB4XDTE1MDUyNjAwMDAwMFoXDTQwMDUyNjAwMDAwMFowOTELMAkG -A1UEBhMCVVMxDzANBgNVBAoTBkFtYXpvbjEZMBcGA1UEAxMQQW1hem9uIFJvb3Qg -Q0EgMzBZMBMGByqGSM49AgEGCCqGSM49AwEHA0IABCmXp8ZBf8ANm+gBG1bG8lKl -ui2yEujSLtf6ycXYqm0fc4E7O5hrOXwzpcVOho6AF2hiRVd9RFgdszflZwjrZt6j -QjBAMA8GA1UdEwEB/wQFMAMBAf8wDgYDVR0PAQH/BAQDAgGGMB0GA1UdDgQWBBSr -ttvXBp43rDCGB5Fwx5zEGbF4wDAKBggqhkjOPQQDAgNJADBGAiEA4IWSoxe3jfkr -BqWTrBqYaGFy+uGh0PsceGCmQ5nFuMQCIQCcAu/xlJyzlvnrxir4tiz+OpAUFteM -YyRIHN8wfdVoOw== ------END CERTIFICATE----- - -# Issuer: CN=Amazon Root CA 4 O=Amazon -# Subject: CN=Amazon Root CA 4 O=Amazon -# Label: "Amazon Root CA 4" -# Serial: 143266989758080763974105200630763877849284878 -# MD5 Fingerprint: 89:bc:27:d5:eb:17:8d:06:6a:69:d5:fd:89:47:b4:cd -# SHA1 Fingerprint: f6:10:84:07:d6:f8:bb:67:98:0c:c2:e2:44:c2:eb:ae:1c:ef:63:be -# SHA256 Fingerprint: e3:5d:28:41:9e:d0:20:25:cf:a6:90:38:cd:62:39:62:45:8d:a5:c6:95:fb:de:a3:c2:2b:0b:fb:25:89:70:92 ------BEGIN CERTIFICATE----- -MIIB8jCCAXigAwIBAgITBmyf18G7EEwpQ+Vxe3ssyBrBDjAKBggqhkjOPQQDAzA5 -MQswCQYDVQQGEwJVUzEPMA0GA1UEChMGQW1hem9uMRkwFwYDVQQDExBBbWF6b24g -Um9vdCBDQSA0MB4XDTE1MDUyNjAwMDAwMFoXDTQwMDUyNjAwMDAwMFowOTELMAkG -A1UEBhMCVVMxDzANBgNVBAoTBkFtYXpvbjEZMBcGA1UEAxMQQW1hem9uIFJvb3Qg -Q0EgNDB2MBAGByqGSM49AgEGBSuBBAAiA2IABNKrijdPo1MN/sGKe0uoe0ZLY7Bi -9i0b2whxIdIA6GO9mif78DluXeo9pcmBqqNbIJhFXRbb/egQbeOc4OO9X4Ri83Bk -M6DLJC9wuoihKqB1+IGuYgbEgds5bimwHvouXKNCMEAwDwYDVR0TAQH/BAUwAwEB -/zAOBgNVHQ8BAf8EBAMCAYYwHQYDVR0OBBYEFNPsxzplbszh2naaVvuc84ZtV+WB -MAoGCCqGSM49BAMDA2gAMGUCMDqLIfG9fhGt0O9Yli/W651+kI0rz2ZVwyzjKKlw -CkcO8DdZEv8tmZQoTipPNU0zWgIxAOp1AE47xDqUEpHJWEadIRNyp4iciuRMStuW -1KyLa2tJElMzrdfkviT8tQp21KW8EA== ------END CERTIFICATE----- - -# Issuer: CN=TUBITAK Kamu SM SSL Kok Sertifikasi - Surum 1 O=Turkiye Bilimsel ve Teknolojik Arastirma Kurumu - TUBITAK OU=Kamu Sertifikasyon Merkezi - Kamu SM -# Subject: CN=TUBITAK Kamu SM SSL Kok Sertifikasi - Surum 1 O=Turkiye Bilimsel ve Teknolojik Arastirma Kurumu - TUBITAK OU=Kamu Sertifikasyon Merkezi - Kamu SM -# Label: "TUBITAK Kamu SM SSL Kok Sertifikasi - Surum 1" -# Serial: 1 -# MD5 Fingerprint: dc:00:81:dc:69:2f:3e:2f:b0:3b:f6:3d:5a:91:8e:49 -# SHA1 Fingerprint: 31:43:64:9b:ec:ce:27:ec:ed:3a:3f:0b:8f:0d:e4:e8:91:dd:ee:ca -# SHA256 Fingerprint: 46:ed:c3:68:90:46:d5:3a:45:3f:b3:10:4a:b8:0d:ca:ec:65:8b:26:60:ea:16:29:dd:7e:86:79:90:64:87:16 ------BEGIN CERTIFICATE----- -MIIEYzCCA0ugAwIBAgIBATANBgkqhkiG9w0BAQsFADCB0jELMAkGA1UEBhMCVFIx -GDAWBgNVBAcTD0dlYnplIC0gS29jYWVsaTFCMEAGA1UEChM5VHVya2l5ZSBCaWxp -bXNlbCB2ZSBUZWtub2xvamlrIEFyYXN0aXJtYSBLdXJ1bXUgLSBUVUJJVEFLMS0w -KwYDVQQLEyRLYW11IFNlcnRpZmlrYXN5b24gTWVya2V6aSAtIEthbXUgU00xNjA0 -BgNVBAMTLVRVQklUQUsgS2FtdSBTTSBTU0wgS29rIFNlcnRpZmlrYXNpIC0gU3Vy -dW0gMTAeFw0xMzExMjUwODI1NTVaFw00MzEwMjUwODI1NTVaMIHSMQswCQYDVQQG -EwJUUjEYMBYGA1UEBxMPR2ViemUgLSBLb2NhZWxpMUIwQAYDVQQKEzlUdXJraXll -IEJpbGltc2VsIHZlIFRla25vbG9qaWsgQXJhc3Rpcm1hIEt1cnVtdSAtIFRVQklU -QUsxLTArBgNVBAsTJEthbXUgU2VydGlmaWthc3lvbiBNZXJrZXppIC0gS2FtdSBT -TTE2MDQGA1UEAxMtVFVCSVRBSyBLYW11IFNNIFNTTCBLb2sgU2VydGlmaWthc2kg -LSBTdXJ1bSAxMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAr3UwM6q7 -a9OZLBI3hNmNe5eA027n/5tQlT6QlVZC1xl8JoSNkvoBHToP4mQ4t4y86Ij5iySr -LqP1N+RAjhgleYN1Hzv/bKjFxlb4tO2KRKOrbEz8HdDc72i9z+SqzvBV96I01INr -N3wcwv61A+xXzry0tcXtAA9TNypN9E8Mg/uGz8v+jE69h/mniyFXnHrfA2eJLJ2X -YacQuFWQfw4tJzh03+f92k4S400VIgLI4OD8D62K18lUUMw7D8oWgITQUVbDjlZ/ -iSIzL+aFCr2lqBs23tPcLG07xxO9WSMs5uWk99gL7eqQQESolbuT1dCANLZGeA4f -AJNG4e7p+exPFwIDAQABo0IwQDAdBgNVHQ4EFgQUZT/HiobGPN08VFw1+DrtUgxH -V8gwDgYDVR0PAQH/BAQDAgEGMA8GA1UdEwEB/wQFMAMBAf8wDQYJKoZIhvcNAQEL -BQADggEBACo/4fEyjq7hmFxLXs9rHmoJ0iKpEsdeV31zVmSAhHqT5Am5EM2fKifh -AHe+SMg1qIGf5LgsyX8OsNJLN13qudULXjS99HMpw+0mFZx+CFOKWI3QSyjfwbPf -IPP54+M638yclNhOT8NrF7f3cuitZjO1JVOr4PhMqZ398g26rrnZqsZr+ZO7rqu4 -lzwDGrpDxpa5RXI4s6ehlj2Re37AIVNMh+3yC1SVUZPVIqUNivGTDj5UDrDYyU7c -8jEyVupk+eq1nRZmQnLzf9OxMUP8pI4X8W0jq5Rm+K37DwhuJi1/FwcJsoz7UMCf -lo3Ptv0AnVoUmr8CRPXBwp8iXqIPoeM= ------END CERTIFICATE----- - -# Issuer: CN=GDCA TrustAUTH R5 ROOT O=GUANG DONG CERTIFICATE AUTHORITY CO.,LTD. -# Subject: CN=GDCA TrustAUTH R5 ROOT O=GUANG DONG CERTIFICATE AUTHORITY CO.,LTD. -# Label: "GDCA TrustAUTH R5 ROOT" -# Serial: 9009899650740120186 -# MD5 Fingerprint: 63:cc:d9:3d:34:35:5c:6f:53:a3:e2:08:70:48:1f:b4 -# SHA1 Fingerprint: 0f:36:38:5b:81:1a:25:c3:9b:31:4e:83:ca:e9:34:66:70:cc:74:b4 -# SHA256 Fingerprint: bf:ff:8f:d0:44:33:48:7d:6a:8a:a6:0c:1a:29:76:7a:9f:c2:bb:b0:5e:42:0f:71:3a:13:b9:92:89:1d:38:93 ------BEGIN CERTIFICATE----- -MIIFiDCCA3CgAwIBAgIIfQmX/vBH6nowDQYJKoZIhvcNAQELBQAwYjELMAkGA1UE -BhMCQ04xMjAwBgNVBAoMKUdVQU5HIERPTkcgQ0VSVElGSUNBVEUgQVVUSE9SSVRZ -IENPLixMVEQuMR8wHQYDVQQDDBZHRENBIFRydXN0QVVUSCBSNSBST09UMB4XDTE0 -MTEyNjA1MTMxNVoXDTQwMTIzMTE1NTk1OVowYjELMAkGA1UEBhMCQ04xMjAwBgNV -BAoMKUdVQU5HIERPTkcgQ0VSVElGSUNBVEUgQVVUSE9SSVRZIENPLixMVEQuMR8w -HQYDVQQDDBZHRENBIFRydXN0QVVUSCBSNSBST09UMIICIjANBgkqhkiG9w0BAQEF -AAOCAg8AMIICCgKCAgEA2aMW8Mh0dHeb7zMNOwZ+Vfy1YI92hhJCfVZmPoiC7XJj -Dp6L3TQsAlFRwxn9WVSEyfFrs0yw6ehGXTjGoqcuEVe6ghWinI9tsJlKCvLriXBj -TnnEt1u9ol2x8kECK62pOqPseQrsXzrj/e+APK00mxqriCZ7VqKChh/rNYmDf1+u -KU49tm7srsHwJ5uu4/Ts765/94Y9cnrrpftZTqfrlYwiOXnhLQiPzLyRuEH3FMEj -qcOtmkVEs7LXLM3GKeJQEK5cy4KOFxg2fZfmiJqwTTQJ9Cy5WmYqsBebnh52nUpm -MUHfP/vFBu8btn4aRjb3ZGM74zkYI+dndRTVdVeSN72+ahsmUPI2JgaQxXABZG12 -ZuGR224HwGGALrIuL4xwp9E7PLOR5G62xDtw8mySlwnNR30YwPO7ng/Wi64HtloP -zgsMR6flPri9fcebNaBhlzpBdRfMK5Z3KpIhHtmVdiBnaM8Nvd/WHwlqmuLMc3Gk -L30SgLdTMEZeS1SZD2fJpcjyIMGC7J0R38IC+xo70e0gmu9lZJIQDSri3nDxGGeC -jGHeuLzRL5z7D9Ar7Rt2ueQ5Vfj4oR24qoAATILnsn8JuLwwoC8N9VKejveSswoA -HQBUlwbgsQfZxw9cZX08bVlX5O2ljelAU58VS6Bx9hoh49pwBiFYFIeFd3mqgnkC -AwEAAaNCMEAwHQYDVR0OBBYEFOLJQJ9NzuiaoXzPDj9lxSmIahlRMA8GA1UdEwEB -/wQFMAMBAf8wDgYDVR0PAQH/BAQDAgGGMA0GCSqGSIb3DQEBCwUAA4ICAQDRSVfg -p8xoWLoBDysZzY2wYUWsEe1jUGn4H3++Fo/9nesLqjJHdtJnJO29fDMylyrHBYZm -DRd9FBUb1Ov9H5r2XpdptxolpAqzkT9fNqyL7FeoPueBihhXOYV0GkLH6VsTX4/5 -COmSdI31R9KrO9b7eGZONn356ZLpBN79SWP8bfsUcZNnL0dKt7n/HipzcEYwv1ry -L3ml4Y0M2fmyYzeMN2WFcGpcWwlyua1jPLHd+PwyvzeG5LuOmCd+uh8W4XAR8gPf -JWIyJyYYMoSf/wA6E7qaTfRPuBRwIrHKK5DOKcFw9C+df/KQHtZa37dG/OaG+svg -IHZ6uqbL9XzeYqWxi+7egmaKTjowHz+Ay60nugxe19CxVsp3cbK1daFQqUBDF8Io -2c9Si1vIY9RCPqAzekYu9wogRlR+ak8x8YF+QnQ4ZXMn7sZ8uI7XpTrXmKGcjBBV -09tL7ECQ8s1uV9JiDnxXk7Gnbc2dg7sq5+W2O3FYrf3RRbxake5TFW/TRQl1brqQ -XR4EzzffHqhmsYzmIGrv/EhOdJhCrylvLmrH+33RZjEizIYAfmaDDEL0vTSSwxrq -T8p+ck0LcIymSLumoRT2+1hEmRSuqguTaaApJUqlyyvdimYHFngVV3Eb7PVHhPOe -MTd61X8kreS8/f3MboPoDKi3QWwH3b08hpcv0g== ------END CERTIFICATE----- - -# Issuer: CN=SSL.com Root Certification Authority RSA O=SSL Corporation -# Subject: CN=SSL.com Root Certification Authority RSA O=SSL Corporation -# Label: "SSL.com Root Certification Authority RSA" -# Serial: 8875640296558310041 -# MD5 Fingerprint: 86:69:12:c0:70:f1:ec:ac:ac:c2:d5:bc:a5:5b:a1:29 -# SHA1 Fingerprint: b7:ab:33:08:d1:ea:44:77:ba:14:80:12:5a:6f:bd:a9:36:49:0c:bb -# SHA256 Fingerprint: 85:66:6a:56:2e:e0:be:5c:e9:25:c1:d8:89:0a:6f:76:a8:7e:c1:6d:4d:7d:5f:29:ea:74:19:cf:20:12:3b:69 ------BEGIN CERTIFICATE----- -MIIF3TCCA8WgAwIBAgIIeyyb0xaAMpkwDQYJKoZIhvcNAQELBQAwfDELMAkGA1UE -BhMCVVMxDjAMBgNVBAgMBVRleGFzMRAwDgYDVQQHDAdIb3VzdG9uMRgwFgYDVQQK -DA9TU0wgQ29ycG9yYXRpb24xMTAvBgNVBAMMKFNTTC5jb20gUm9vdCBDZXJ0aWZp -Y2F0aW9uIEF1dGhvcml0eSBSU0EwHhcNMTYwMjEyMTczOTM5WhcNNDEwMjEyMTcz -OTM5WjB8MQswCQYDVQQGEwJVUzEOMAwGA1UECAwFVGV4YXMxEDAOBgNVBAcMB0hv -dXN0b24xGDAWBgNVBAoMD1NTTCBDb3Jwb3JhdGlvbjExMC8GA1UEAwwoU1NMLmNv -bSBSb290IENlcnRpZmljYXRpb24gQXV0aG9yaXR5IFJTQTCCAiIwDQYJKoZIhvcN -AQEBBQADggIPADCCAgoCggIBAPkP3aMrfcvQKv7sZ4Wm5y4bunfh4/WvpOz6Sl2R -xFdHaxh3a3by/ZPkPQ/CFp4LZsNWlJ4Xg4XOVu/yFv0AYvUiCVToZRdOQbngT0aX -qhvIuG5iXmmxX9sqAn78bMrzQdjt0Oj8P2FI7bADFB0QDksZ4LtO7IZl/zbzXmcC -C52GVWH9ejjt/uIZALdvoVBidXQ8oPrIJZK0bnoix/geoeOy3ZExqysdBP+lSgQ3 -6YWkMyv94tZVNHwZpEpox7Ko07fKoZOI68GXvIz5HdkihCR0xwQ9aqkpk8zruFvh -/l8lqjRYyMEjVJ0bmBHDOJx+PYZspQ9AhnwC9FwCTyjLrnGfDzrIM/4RJTXq/LrF -YD3ZfBjVsqnTdXgDciLKOsMf7yzlLqn6niy2UUb9rwPW6mBo6oUWNmuF6R7As93E -JNyAKoFBbZQ+yODJgUEAnl6/f8UImKIYLEJAs/lvOCdLToD0PYFH4Ih86hzOtXVc -US4cK38acijnALXRdMbX5J+tB5O2UzU1/Dfkw/ZdFr4hc96SCvigY2q8lpJqPvi8 -ZVWb3vUNiSYE/CUapiVpy8JtynziWV+XrOvvLsi81xtZPCvM8hnIk2snYxnP/Okm -+Mpxm3+T/jRnhE6Z6/yzeAkzcLpmpnbtG3PrGqUNxCITIJRWCk4sbE6x/c+cCbqi -M+2HAgMBAAGjYzBhMB0GA1UdDgQWBBTdBAkHovV6fVJTEpKV7jiAJQ2mWTAPBgNV -HRMBAf8EBTADAQH/MB8GA1UdIwQYMBaAFN0ECQei9Xp9UlMSkpXuOIAlDaZZMA4G -A1UdDwEB/wQEAwIBhjANBgkqhkiG9w0BAQsFAAOCAgEAIBgRlCn7Jp0cHh5wYfGV -cpNxJK1ok1iOMq8bs3AD/CUrdIWQPXhq9LmLpZc7tRiRux6n+UBbkflVma8eEdBc -Hadm47GUBwwyOabqG7B52B2ccETjit3E+ZUfijhDPwGFpUenPUayvOUiaPd7nNgs -PgohyC0zrL/FgZkxdMF1ccW+sfAjRfSda/wZY52jvATGGAslu1OJD7OAUN5F7kR/ -q5R4ZJjT9ijdh9hwZXT7DrkT66cPYakylszeu+1jTBi7qUD3oFRuIIhxdRjqerQ0 -cuAjJ3dctpDqhiVAq+8zD8ufgr6iIPv2tS0a5sKFsXQP+8hlAqRSAUfdSSLBv9jr -a6x+3uxjMxW3IwiPxg+NQVrdjsW5j+VFP3jbutIbQLH+cU0/4IGiul607BXgk90I -H37hVZkLId6Tngr75qNJvTYw/ud3sqB1l7UtgYgXZSD32pAAn8lSzDLKNXz1PQ/Y -K9f1JmzJBjSWFupwWRoyeXkLtoh/D1JIPb9s2KJELtFOt3JY04kTlf5Eq/jXixtu -nLwsoFvVagCvXzfh1foQC5ichucmj87w7G6KVwuA406ywKBjYZC6VWg3dGq2ktuf -oYYitmUnDuy2n0Jg5GfCtdpBC8TTi2EbvPofkSvXRAdeuims2cXp71NIWuuA8ShY -Ic2wBlX7Jz9TkHCpBB5XJ7k= ------END CERTIFICATE----- - -# Issuer: CN=SSL.com Root Certification Authority ECC O=SSL Corporation -# Subject: CN=SSL.com Root Certification Authority ECC O=SSL Corporation -# Label: "SSL.com Root Certification Authority ECC" -# Serial: 8495723813297216424 -# MD5 Fingerprint: 2e:da:e4:39:7f:9c:8f:37:d1:70:9f:26:17:51:3a:8e -# SHA1 Fingerprint: c3:19:7c:39:24:e6:54:af:1b:c4:ab:20:95:7a:e2:c3:0e:13:02:6a -# SHA256 Fingerprint: 34:17:bb:06:cc:60:07:da:1b:96:1c:92:0b:8a:b4:ce:3f:ad:82:0e:4a:a3:0b:9a:cb:c4:a7:4e:bd:ce:bc:65 ------BEGIN CERTIFICATE----- -MIICjTCCAhSgAwIBAgIIdebfy8FoW6gwCgYIKoZIzj0EAwIwfDELMAkGA1UEBhMC -VVMxDjAMBgNVBAgMBVRleGFzMRAwDgYDVQQHDAdIb3VzdG9uMRgwFgYDVQQKDA9T -U0wgQ29ycG9yYXRpb24xMTAvBgNVBAMMKFNTTC5jb20gUm9vdCBDZXJ0aWZpY2F0 -aW9uIEF1dGhvcml0eSBFQ0MwHhcNMTYwMjEyMTgxNDAzWhcNNDEwMjEyMTgxNDAz -WjB8MQswCQYDVQQGEwJVUzEOMAwGA1UECAwFVGV4YXMxEDAOBgNVBAcMB0hvdXN0 -b24xGDAWBgNVBAoMD1NTTCBDb3Jwb3JhdGlvbjExMC8GA1UEAwwoU1NMLmNvbSBS -b290IENlcnRpZmljYXRpb24gQXV0aG9yaXR5IEVDQzB2MBAGByqGSM49AgEGBSuB -BAAiA2IABEVuqVDEpiM2nl8ojRfLliJkP9x6jh3MCLOicSS6jkm5BBtHllirLZXI -7Z4INcgn64mMU1jrYor+8FsPazFSY0E7ic3s7LaNGdM0B9y7xgZ/wkWV7Mt/qCPg -CemB+vNH06NjMGEwHQYDVR0OBBYEFILRhXMw5zUE044CkvvlpNHEIejNMA8GA1Ud -EwEB/wQFMAMBAf8wHwYDVR0jBBgwFoAUgtGFczDnNQTTjgKS++Wk0cQh6M0wDgYD -VR0PAQH/BAQDAgGGMAoGCCqGSM49BAMCA2cAMGQCMG/n61kRpGDPYbCWe+0F+S8T -kdzt5fxQaxFGRrMcIQBiu77D5+jNB5n5DQtdcj7EqgIwH7y6C+IwJPt8bYBVCpk+ -gA0z5Wajs6O7pdWLjwkspl1+4vAHCGht0nxpbl/f5Wpl ------END CERTIFICATE----- - -# Issuer: CN=SSL.com EV Root Certification Authority RSA R2 O=SSL Corporation -# Subject: CN=SSL.com EV Root Certification Authority RSA R2 O=SSL Corporation -# Label: "SSL.com EV Root Certification Authority RSA R2" -# Serial: 6248227494352943350 -# MD5 Fingerprint: e1:1e:31:58:1a:ae:54:53:02:f6:17:6a:11:7b:4d:95 -# SHA1 Fingerprint: 74:3a:f0:52:9b:d0:32:a0:f4:4a:83:cd:d4:ba:a9:7b:7c:2e:c4:9a -# SHA256 Fingerprint: 2e:7b:f1:6c:c2:24:85:a7:bb:e2:aa:86:96:75:07:61:b0:ae:39:be:3b:2f:e9:d0:cc:6d:4e:f7:34:91:42:5c ------BEGIN CERTIFICATE----- -MIIF6zCCA9OgAwIBAgIIVrYpzTS8ePYwDQYJKoZIhvcNAQELBQAwgYIxCzAJBgNV -BAYTAlVTMQ4wDAYDVQQIDAVUZXhhczEQMA4GA1UEBwwHSG91c3RvbjEYMBYGA1UE -CgwPU1NMIENvcnBvcmF0aW9uMTcwNQYDVQQDDC5TU0wuY29tIEVWIFJvb3QgQ2Vy -dGlmaWNhdGlvbiBBdXRob3JpdHkgUlNBIFIyMB4XDTE3MDUzMTE4MTQzN1oXDTQy -MDUzMDE4MTQzN1owgYIxCzAJBgNVBAYTAlVTMQ4wDAYDVQQIDAVUZXhhczEQMA4G -A1UEBwwHSG91c3RvbjEYMBYGA1UECgwPU1NMIENvcnBvcmF0aW9uMTcwNQYDVQQD -DC5TU0wuY29tIEVWIFJvb3QgQ2VydGlmaWNhdGlvbiBBdXRob3JpdHkgUlNBIFIy -MIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAjzZlQOHWTcDXtOlG2mvq -M0fNTPl9fb69LT3w23jhhqXZuglXaO1XPqDQCEGD5yhBJB/jchXQARr7XnAjssuf -OePPxU7Gkm0mxnu7s9onnQqG6YE3Bf7wcXHswxzpY6IXFJ3vG2fThVUCAtZJycxa -4bH3bzKfydQ7iEGonL3Lq9ttewkfokxykNorCPzPPFTOZw+oz12WGQvE43LrrdF9 -HSfvkusQv1vrO6/PgN3B0pYEW3p+pKk8OHakYo6gOV7qd89dAFmPZiw+B6KjBSYR -aZfqhbcPlgtLyEDhULouisv3D5oi53+aNxPN8k0TayHRwMwi8qFG9kRpnMphNQcA -b9ZhCBHqurj26bNg5U257J8UZslXWNvNh2n4ioYSA0e/ZhN2rHd9NCSFg83XqpyQ -Gp8hLH94t2S42Oim9HizVcuE0jLEeK6jj2HdzghTreyI/BXkmg3mnxp3zkyPuBQV -PWKchjgGAGYS5Fl2WlPAApiiECtoRHuOec4zSnaqW4EWG7WK2NAAe15itAnWhmMO -pgWVSbooi4iTsjQc2KRVbrcc0N6ZVTsj9CLg+SlmJuwgUHfbSguPvuUCYHBBXtSu -UDkiFCbLsjtzdFVHB3mBOagwE0TlBIqulhMlQg+5U8Sb/M3kHN48+qvWBkofZ6aY -MBzdLNvcGJVXZsb/XItW9XcCAwEAAaNjMGEwDwYDVR0TAQH/BAUwAwEB/zAfBgNV -HSMEGDAWgBT5YLvU49U09rj1BoAlp3PbRmmonjAdBgNVHQ4EFgQU+WC71OPVNPa4 -9QaAJadz20ZpqJ4wDgYDVR0PAQH/BAQDAgGGMA0GCSqGSIb3DQEBCwUAA4ICAQBW -s47LCp1Jjr+kxJG7ZhcFUZh1++VQLHqe8RT6q9OKPv+RKY9ji9i0qVQBDb6Thi/5 -Sm3HXvVX+cpVHBK+Rw82xd9qt9t1wkclf7nxY/hoLVUE0fKNsKTPvDxeH3jnpaAg -cLAExbf3cqfeIg29MyVGjGSSJuM+LmOW2puMPfgYCdcDzH2GguDKBAdRUNf/ktUM -79qGn5nX67evaOI5JpS6aLe/g9Pqemc9YmeuJeVy6OLk7K4S9ksrPJ/psEDzOFSz -/bdoyNrGj1E8svuR3Bznm53htw1yj+KkxKl4+esUrMZDBcJlOSgYAsOCsp0FvmXt -ll9ldDz7CTUue5wT/RsPXcdtgTpWD8w74a8CLyKsRspGPKAcTNZEtF4uXBVmCeEm -Kf7GUmG6sXP/wwyc5WxqlD8UykAWlYTzWamsX0xhk23RO8yilQwipmdnRC652dKK -QbNmC1r7fSOl8hqw/96bg5Qu0T/fkreRrwU7ZcegbLHNYhLDkBvjJc40vG93drEQ -w/cFGsDWr3RiSBd3kmmQYRzelYB0VI8YHMPzA9C/pEN1hlMYegouCRw2n5H9gooi -S9EOUCXdywMMF8mDAAhONU2Ki+3wApRmLER/y5UnlhetCTCstnEXbosX9hwJ1C07 -mKVx01QT2WDz9UtmT/rx7iASjbSsV7FFY6GsdqnC+w== ------END CERTIFICATE----- - -# Issuer: CN=SSL.com EV Root Certification Authority ECC O=SSL Corporation -# Subject: CN=SSL.com EV Root Certification Authority ECC O=SSL Corporation -# Label: "SSL.com EV Root Certification Authority ECC" -# Serial: 3182246526754555285 -# MD5 Fingerprint: 59:53:22:65:83:42:01:54:c0:ce:42:b9:5a:7c:f2:90 -# SHA1 Fingerprint: 4c:dd:51:a3:d1:f5:20:32:14:b0:c6:c5:32:23:03:91:c7:46:42:6d -# SHA256 Fingerprint: 22:a2:c1:f7:bd:ed:70:4c:c1:e7:01:b5:f4:08:c3:10:88:0f:e9:56:b5:de:2a:4a:44:f9:9c:87:3a:25:a7:c8 ------BEGIN CERTIFICATE----- -MIIClDCCAhqgAwIBAgIILCmcWxbtBZUwCgYIKoZIzj0EAwIwfzELMAkGA1UEBhMC -VVMxDjAMBgNVBAgMBVRleGFzMRAwDgYDVQQHDAdIb3VzdG9uMRgwFgYDVQQKDA9T -U0wgQ29ycG9yYXRpb24xNDAyBgNVBAMMK1NTTC5jb20gRVYgUm9vdCBDZXJ0aWZp -Y2F0aW9uIEF1dGhvcml0eSBFQ0MwHhcNMTYwMjEyMTgxNTIzWhcNNDEwMjEyMTgx -NTIzWjB/MQswCQYDVQQGEwJVUzEOMAwGA1UECAwFVGV4YXMxEDAOBgNVBAcMB0hv -dXN0b24xGDAWBgNVBAoMD1NTTCBDb3Jwb3JhdGlvbjE0MDIGA1UEAwwrU1NMLmNv -bSBFViBSb290IENlcnRpZmljYXRpb24gQXV0aG9yaXR5IEVDQzB2MBAGByqGSM49 -AgEGBSuBBAAiA2IABKoSR5CYG/vvw0AHgyBO8TCCogbR8pKGYfL2IWjKAMTH6kMA -VIbc/R/fALhBYlzccBYy3h+Z1MzFB8gIH2EWB1E9fVwHU+M1OIzfzZ/ZLg1Kthku -WnBaBu2+8KGwytAJKaNjMGEwHQYDVR0OBBYEFFvKXuXe0oGqzagtZFG22XKbl+ZP -MA8GA1UdEwEB/wQFMAMBAf8wHwYDVR0jBBgwFoAUW8pe5d7SgarNqC1kUbbZcpuX -5k8wDgYDVR0PAQH/BAQDAgGGMAoGCCqGSM49BAMCA2gAMGUCMQCK5kCJN+vp1RPZ -ytRrJPOwPYdGWBrssd9v+1a6cGvHOMzosYxPD/fxZ3YOg9AeUY8CMD32IygmTMZg -h5Mmm7I1HrrW9zzRHM76JTymGoEVW/MSD2zuZYrJh6j5B+BimoxcSg== ------END CERTIFICATE----- - -# Issuer: CN=GlobalSign O=GlobalSign OU=GlobalSign Root CA - R6 -# Subject: CN=GlobalSign O=GlobalSign OU=GlobalSign Root CA - R6 -# Label: "GlobalSign Root CA - R6" -# Serial: 1417766617973444989252670301619537 -# MD5 Fingerprint: 4f:dd:07:e4:d4:22:64:39:1e:0c:37:42:ea:d1:c6:ae -# SHA1 Fingerprint: 80:94:64:0e:b5:a7:a1:ca:11:9c:1f:dd:d5:9f:81:02:63:a7:fb:d1 -# SHA256 Fingerprint: 2c:ab:ea:fe:37:d0:6c:a2:2a:ba:73:91:c0:03:3d:25:98:29:52:c4:53:64:73:49:76:3a:3a:b5:ad:6c:cf:69 ------BEGIN CERTIFICATE----- -MIIFgzCCA2ugAwIBAgIORea7A4Mzw4VlSOb/RVEwDQYJKoZIhvcNAQEMBQAwTDEg -MB4GA1UECxMXR2xvYmFsU2lnbiBSb290IENBIC0gUjYxEzARBgNVBAoTCkdsb2Jh -bFNpZ24xEzARBgNVBAMTCkdsb2JhbFNpZ24wHhcNMTQxMjEwMDAwMDAwWhcNMzQx -MjEwMDAwMDAwWjBMMSAwHgYDVQQLExdHbG9iYWxTaWduIFJvb3QgQ0EgLSBSNjET -MBEGA1UEChMKR2xvYmFsU2lnbjETMBEGA1UEAxMKR2xvYmFsU2lnbjCCAiIwDQYJ -KoZIhvcNAQEBBQADggIPADCCAgoCggIBAJUH6HPKZvnsFMp7PPcNCPG0RQssgrRI -xutbPK6DuEGSMxSkb3/pKszGsIhrxbaJ0cay/xTOURQh7ErdG1rG1ofuTToVBu1k -ZguSgMpE3nOUTvOniX9PeGMIyBJQbUJmL025eShNUhqKGoC3GYEOfsSKvGRMIRxD -aNc9PIrFsmbVkJq3MQbFvuJtMgamHvm566qjuL++gmNQ0PAYid/kD3n16qIfKtJw -LnvnvJO7bVPiSHyMEAc4/2ayd2F+4OqMPKq0pPbzlUoSB239jLKJz9CgYXfIWHSw -1CM69106yqLbnQneXUQtkPGBzVeS+n68UARjNN9rkxi+azayOeSsJDa38O+2HBNX -k7besvjihbdzorg1qkXy4J02oW9UivFyVm4uiMVRQkQVlO6jxTiWm05OWgtH8wY2 -SXcwvHE35absIQh1/OZhFj931dmRl4QKbNQCTXTAFO39OfuD8l4UoQSwC+n+7o/h -bguyCLNhZglqsQY6ZZZZwPA1/cnaKI0aEYdwgQqomnUdnjqGBQCe24DWJfncBZ4n -WUx2OVvq+aWh2IMP0f/fMBH5hc8zSPXKbWQULHpYT9NLCEnFlWQaYw55PfWzjMpY -rZxCRXluDocZXFSxZba/jJvcE+kNb7gu3GduyYsRtYQUigAZcIN5kZeR1Bonvzce -MgfYFGM8KEyvAgMBAAGjYzBhMA4GA1UdDwEB/wQEAwIBBjAPBgNVHRMBAf8EBTAD -AQH/MB0GA1UdDgQWBBSubAWjkxPioufi1xzWx/B/yGdToDAfBgNVHSMEGDAWgBSu -bAWjkxPioufi1xzWx/B/yGdToDANBgkqhkiG9w0BAQwFAAOCAgEAgyXt6NH9lVLN -nsAEoJFp5lzQhN7craJP6Ed41mWYqVuoPId8AorRbrcWc+ZfwFSY1XS+wc3iEZGt -Ixg93eFyRJa0lV7Ae46ZeBZDE1ZXs6KzO7V33EByrKPrmzU+sQghoefEQzd5Mr61 -55wsTLxDKZmOMNOsIeDjHfrYBzN2VAAiKrlNIC5waNrlU/yDXNOd8v9EDERm8tLj -vUYAGm0CuiVdjaExUd1URhxN25mW7xocBFymFe944Hn+Xds+qkxV/ZoVqW/hpvvf -cDDpw+5CRu3CkwWJ+n1jez/QcYF8AOiYrg54NMMl+68KnyBr3TsTjxKM4kEaSHpz -oHdpx7Zcf4LIHv5YGygrqGytXm3ABdJ7t+uA/iU3/gKbaKxCXcPu9czc8FB10jZp -nOZ7BN9uBmm23goJSFmH63sUYHpkqmlD75HHTOwY3WzvUy2MmeFe8nI+z1TIvWfs -pA9MRf/TuTAjB0yPEL+GltmZWrSZVxykzLsViVO6LAUP5MSeGbEYNNVMnbrt9x+v -JJUEeKgDu+6B5dpffItKoZB0JaezPkvILFa9x8jvOOJckvB595yEunQtYQEgfn7R -8k8HWV+LLUNS60YMlOH1Zkd5d9VUWx+tJDfLRVpOoERIyNiwmcUVhAn21klJwGW4 -5hpxbqCo8YLoRT5s1gLXCmeDBVrJpBA= ------END CERTIFICATE----- - -# Issuer: CN=OISTE WISeKey Global Root GC CA O=WISeKey OU=OISTE Foundation Endorsed -# Subject: CN=OISTE WISeKey Global Root GC CA O=WISeKey OU=OISTE Foundation Endorsed -# Label: "OISTE WISeKey Global Root GC CA" -# Serial: 44084345621038548146064804565436152554 -# MD5 Fingerprint: a9:d6:b9:2d:2f:93:64:f8:a5:69:ca:91:e9:68:07:23 -# SHA1 Fingerprint: e0:11:84:5e:34:de:be:88:81:b9:9c:f6:16:26:d1:96:1f:c3:b9:31 -# SHA256 Fingerprint: 85:60:f9:1c:36:24:da:ba:95:70:b5:fe:a0:db:e3:6f:f1:1a:83:23:be:94:86:85:4f:b3:f3:4a:55:71:19:8d ------BEGIN CERTIFICATE----- -MIICaTCCAe+gAwIBAgIQISpWDK7aDKtARb8roi066jAKBggqhkjOPQQDAzBtMQsw -CQYDVQQGEwJDSDEQMA4GA1UEChMHV0lTZUtleTEiMCAGA1UECxMZT0lTVEUgRm91 -bmRhdGlvbiBFbmRvcnNlZDEoMCYGA1UEAxMfT0lTVEUgV0lTZUtleSBHbG9iYWwg -Um9vdCBHQyBDQTAeFw0xNzA1MDkwOTQ4MzRaFw00MjA1MDkwOTU4MzNaMG0xCzAJ -BgNVBAYTAkNIMRAwDgYDVQQKEwdXSVNlS2V5MSIwIAYDVQQLExlPSVNURSBGb3Vu -ZGF0aW9uIEVuZG9yc2VkMSgwJgYDVQQDEx9PSVNURSBXSVNlS2V5IEdsb2JhbCBS -b290IEdDIENBMHYwEAYHKoZIzj0CAQYFK4EEACIDYgAETOlQwMYPchi82PG6s4ni -eUqjFqdrVCTbUf/q9Akkwwsin8tqJ4KBDdLArzHkdIJuyiXZjHWd8dvQmqJLIX4W -p2OQ0jnUsYd4XxiWD1AbNTcPasbc2RNNpI6QN+a9WzGRo1QwUjAOBgNVHQ8BAf8E -BAMCAQYwDwYDVR0TAQH/BAUwAwEB/zAdBgNVHQ4EFgQUSIcUrOPDnpBgOtfKie7T -rYy0UGYwEAYJKwYBBAGCNxUBBAMCAQAwCgYIKoZIzj0EAwMDaAAwZQIwJsdpW9zV -57LnyAyMjMPdeYwbY9XJUpROTYJKcx6ygISpJcBMWm1JKWB4E+J+SOtkAjEA2zQg -Mgj/mkkCtojeFK9dbJlxjRo/i9fgojaGHAeCOnZT/cKi7e97sIBPWA9LUzm9 ------END CERTIFICATE----- - -# Issuer: CN=UCA Global G2 Root O=UniTrust -# Subject: CN=UCA Global G2 Root O=UniTrust -# Label: "UCA Global G2 Root" -# Serial: 124779693093741543919145257850076631279 -# MD5 Fingerprint: 80:fe:f0:c4:4a:f0:5c:62:32:9f:1c:ba:78:a9:50:f8 -# SHA1 Fingerprint: 28:f9:78:16:19:7a:ff:18:25:18:aa:44:fe:c1:a0:ce:5c:b6:4c:8a -# SHA256 Fingerprint: 9b:ea:11:c9:76:fe:01:47:64:c1:be:56:a6:f9:14:b5:a5:60:31:7a:bd:99:88:39:33:82:e5:16:1a:a0:49:3c ------BEGIN CERTIFICATE----- -MIIFRjCCAy6gAwIBAgIQXd+x2lqj7V2+WmUgZQOQ7zANBgkqhkiG9w0BAQsFADA9 -MQswCQYDVQQGEwJDTjERMA8GA1UECgwIVW5pVHJ1c3QxGzAZBgNVBAMMElVDQSBH -bG9iYWwgRzIgUm9vdDAeFw0xNjAzMTEwMDAwMDBaFw00MDEyMzEwMDAwMDBaMD0x -CzAJBgNVBAYTAkNOMREwDwYDVQQKDAhVbmlUcnVzdDEbMBkGA1UEAwwSVUNBIEds -b2JhbCBHMiBSb290MIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAxeYr -b3zvJgUno4Ek2m/LAfmZmqkywiKHYUGRO8vDaBsGxUypK8FnFyIdK+35KYmToni9 -kmugow2ifsqTs6bRjDXVdfkX9s9FxeV67HeToI8jrg4aA3++1NDtLnurRiNb/yzm -VHqUwCoV8MmNsHo7JOHXaOIxPAYzRrZUEaalLyJUKlgNAQLx+hVRZ2zA+te2G3/R -VogvGjqNO7uCEeBHANBSh6v7hn4PJGtAnTRnvI3HLYZveT6OqTwXS3+wmeOwcWDc -C/Vkw85DvG1xudLeJ1uK6NjGruFZfc8oLTW4lVYa8bJYS7cSN8h8s+1LgOGN+jIj -tm+3SJUIsUROhYw6AlQgL9+/V087OpAh18EmNVQg7Mc/R+zvWr9LesGtOxdQXGLY -D0tK3Cv6brxzks3sx1DoQZbXqX5t2Okdj4q1uViSukqSKwxW/YDrCPBeKW4bHAyv -j5OJrdu9o54hyokZ7N+1wxrrFv54NkzWbtA+FxyQF2smuvt6L78RHBgOLXMDj6Dl -NaBa4kx1HXHhOThTeEDMg5PXCp6dW4+K5OXgSORIskfNTip1KnvyIvbJvgmRlld6 -iIis7nCs+dwp4wwcOxJORNanTrAmyPPZGpeRaOrvjUYG0lZFWJo8DA+DuAUlwznP -O6Q0ibd5Ei9Hxeepl2n8pndntd978XplFeRhVmUCAwEAAaNCMEAwDgYDVR0PAQH/ -BAQDAgEGMA8GA1UdEwEB/wQFMAMBAf8wHQYDVR0OBBYEFIHEjMz15DD/pQwIX4wV -ZyF0Ad/fMA0GCSqGSIb3DQEBCwUAA4ICAQATZSL1jiutROTL/7lo5sOASD0Ee/oj -L3rtNtqyzm325p7lX1iPyzcyochltq44PTUbPrw7tgTQvPlJ9Zv3hcU2tsu8+Mg5 -1eRfB70VVJd0ysrtT7q6ZHafgbiERUlMjW+i67HM0cOU2kTC5uLqGOiiHycFutfl -1qnN3e92mI0ADs0b+gO3joBYDic/UvuUospeZcnWhNq5NXHzJsBPd+aBJ9J3O5oU -b3n09tDh05S60FdRvScFDcH9yBIw7m+NESsIndTUv4BFFJqIRNow6rSn4+7vW4LV -PtateJLbXDzz2K36uGt/xDYotgIVilQsnLAXc47QN6MUPJiVAAwpBVueSUmxX8fj -y88nZY41F7dXyDDZQVu5FLbowg+UMaeUmMxq67XhJ/UQqAHojhJi6IjMtX9Gl8Cb -EGY4GjZGXyJoPd/JxhMnq1MGrKI8hgZlb7F+sSlEmqO6SWkoaY/X5V+tBIZkbxqg -DMUIYs6Ao9Dz7GjevjPHF1t/gMRMTLGmhIrDO7gJzRSBuhjjVFc2/tsvfEehOjPI -+Vg7RE+xygKJBJYoaMVLuCaJu9YzL1DV/pqJuhgyklTGW+Cd+V7lDSKb9triyCGy -YiGqhkCyLmTTX8jjfhFnRR8F/uOi77Oos/N9j/gMHyIfLXC0uAE0djAA5SN4p1bX -UB+K+wb1whnw0A== ------END CERTIFICATE----- - -# Issuer: CN=UCA Extended Validation Root O=UniTrust -# Subject: CN=UCA Extended Validation Root O=UniTrust -# Label: "UCA Extended Validation Root" -# Serial: 106100277556486529736699587978573607008 -# MD5 Fingerprint: a1:f3:5f:43:c6:34:9b:da:bf:8c:7e:05:53:ad:96:e2 -# SHA1 Fingerprint: a3:a1:b0:6f:24:61:23:4a:e3:36:a5:c2:37:fc:a6:ff:dd:f0:d7:3a -# SHA256 Fingerprint: d4:3a:f9:b3:54:73:75:5c:96:84:fc:06:d7:d8:cb:70:ee:5c:28:e7:73:fb:29:4e:b4:1e:e7:17:22:92:4d:24 ------BEGIN CERTIFICATE----- -MIIFWjCCA0KgAwIBAgIQT9Irj/VkyDOeTzRYZiNwYDANBgkqhkiG9w0BAQsFADBH -MQswCQYDVQQGEwJDTjERMA8GA1UECgwIVW5pVHJ1c3QxJTAjBgNVBAMMHFVDQSBF -eHRlbmRlZCBWYWxpZGF0aW9uIFJvb3QwHhcNMTUwMzEzMDAwMDAwWhcNMzgxMjMx -MDAwMDAwWjBHMQswCQYDVQQGEwJDTjERMA8GA1UECgwIVW5pVHJ1c3QxJTAjBgNV -BAMMHFVDQSBFeHRlbmRlZCBWYWxpZGF0aW9uIFJvb3QwggIiMA0GCSqGSIb3DQEB -AQUAA4ICDwAwggIKAoICAQCpCQcoEwKwmeBkqh5DFnpzsZGgdT6o+uM4AHrsiWog -D4vFsJszA1qGxliG1cGFu0/GnEBNyr7uaZa4rYEwmnySBesFK5pI0Lh2PpbIILvS -sPGP2KxFRv+qZ2C0d35qHzwaUnoEPQc8hQ2E0B92CvdqFN9y4zR8V05WAT558aop -O2z6+I9tTcg1367r3CTueUWnhbYFiN6IXSV8l2RnCdm/WhUFhvMJHuxYMjMR83dk -sHYf5BA1FxvyDrFspCqjc/wJHx4yGVMR59mzLC52LqGj3n5qiAno8geK+LLNEOfi -c0CTuwjRP+H8C5SzJe98ptfRr5//lpr1kXuYC3fUfugH0mK1lTnj8/FtDw5lhIpj -VMWAtuCeS31HJqcBCF3RiJ7XwzJE+oJKCmhUfzhTA8ykADNkUVkLo4KRel7sFsLz -KuZi2irbWWIQJUoqgQtHB0MGcIfS+pMRKXpITeuUx3BNr2fVUbGAIAEBtHoIppB/ -TuDvB0GHr2qlXov7z1CymlSvw4m6WC31MJixNnI5fkkE/SmnTHnkBVfblLkWU41G -sx2VYVdWf6/wFlthWG82UBEL2KwrlRYaDh8IzTY0ZRBiZtWAXxQgXy0MoHgKaNYs -1+lvK9JKBZP8nm9rZ/+I8U6laUpSNwXqxhaN0sSZ0YIrO7o1dfdRUVjzyAfd5LQD -fwIDAQABo0IwQDAdBgNVHQ4EFgQU2XQ65DA9DfcS3H5aBZ8eNJr34RQwDwYDVR0T -AQH/BAUwAwEB/zAOBgNVHQ8BAf8EBAMCAYYwDQYJKoZIhvcNAQELBQADggIBADaN -l8xCFWQpN5smLNb7rhVpLGsaGvdftvkHTFnq88nIua7Mui563MD1sC3AO6+fcAUR -ap8lTwEpcOPlDOHqWnzcSbvBHiqB9RZLcpHIojG5qtr8nR/zXUACE/xOHAbKsxSQ -VBcZEhrxH9cMaVr2cXj0lH2RC47skFSOvG+hTKv8dGT9cZr4QQehzZHkPJrgmzI5 -c6sq1WnIeJEmMX3ixzDx/BR4dxIOE/TdFpS/S2d7cFOFyrC78zhNLJA5wA3CXWvp -4uXViI3WLL+rG761KIcSF3Ru/H38j9CHJrAb+7lsq+KePRXBOy5nAliRn+/4Qh8s -t2j1da3Ptfb/EX3C8CSlrdP6oDyp+l3cpaDvRKS+1ujl5BOWF3sGPjLtx7dCvHaj -2GU4Kzg1USEODm8uNBNA4StnDG1KQTAYI1oyVZnJF+A83vbsea0rWBmirSwiGpWO -vpaQXUJXxPkUAzUrHC1RVwinOt4/5Mi0A3PCwSaAuwtCH60NryZy2sy+s6ODWA2C -xR9GUeOcGMyNm43sSet1UNWMKFnKdDTajAshqx7qG+XH/RU+wBeq+yNuJkbL+vmx -cmtpzyKEC2IPrNkZAJSidjzULZrtBJ4tBmIQN1IchXIbJ+XMxjHsN+xjWZsLHXbM -fjKaiJUINlK73nZfdklJrX+9ZSCyycErdhh2n1ax ------END CERTIFICATE----- - -# Issuer: CN=Certigna Root CA O=Dhimyotis OU=0002 48146308100036 -# Subject: CN=Certigna Root CA O=Dhimyotis OU=0002 48146308100036 -# Label: "Certigna Root CA" -# Serial: 269714418870597844693661054334862075617 -# MD5 Fingerprint: 0e:5c:30:62:27:eb:5b:bc:d7:ae:62:ba:e9:d5:df:77 -# SHA1 Fingerprint: 2d:0d:52:14:ff:9e:ad:99:24:01:74:20:47:6e:6c:85:27:27:f5:43 -# SHA256 Fingerprint: d4:8d:3d:23:ee:db:50:a4:59:e5:51:97:60:1c:27:77:4b:9d:7b:18:c9:4d:5a:05:95:11:a1:02:50:b9:31:68 ------BEGIN CERTIFICATE----- -MIIGWzCCBEOgAwIBAgIRAMrpG4nxVQMNo+ZBbcTjpuEwDQYJKoZIhvcNAQELBQAw -WjELMAkGA1UEBhMCRlIxEjAQBgNVBAoMCURoaW15b3RpczEcMBoGA1UECwwTMDAw -MiA0ODE0NjMwODEwMDAzNjEZMBcGA1UEAwwQQ2VydGlnbmEgUm9vdCBDQTAeFw0x -MzEwMDEwODMyMjdaFw0zMzEwMDEwODMyMjdaMFoxCzAJBgNVBAYTAkZSMRIwEAYD -VQQKDAlEaGlteW90aXMxHDAaBgNVBAsMEzAwMDIgNDgxNDYzMDgxMDAwMzYxGTAX -BgNVBAMMEENlcnRpZ25hIFJvb3QgQ0EwggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAw -ggIKAoICAQDNGDllGlmx6mQWDoyUJJV8g9PFOSbcDO8WV43X2KyjQn+Cyu3NW9sO -ty3tRQgXstmzy9YXUnIo245Onoq2C/mehJpNdt4iKVzSs9IGPjA5qXSjklYcoW9M -CiBtnyN6tMbaLOQdLNyzKNAT8kxOAkmhVECe5uUFoC2EyP+YbNDrihqECB63aCPu -I9Vwzm1RaRDuoXrC0SIxwoKF0vJVdlB8JXrJhFwLrN1CTivngqIkicuQstDuI7pm -TLtipPlTWmR7fJj6o0ieD5Wupxj0auwuA0Wv8HT4Ks16XdG+RCYyKfHx9WzMfgIh -C59vpD++nVPiz32pLHxYGpfhPTc3GGYo0kDFUYqMwy3OU4gkWGQwFsWq4NYKpkDf -ePb1BHxpE4S80dGnBs8B92jAqFe7OmGtBIyT46388NtEbVncSVmurJqZNjBBe3Yz -IoejwpKGbvlw7q6Hh5UbxHq9MfPU0uWZ/75I7HX1eBYdpnDBfzwboZL7z8g81sWT -Co/1VTp2lc5ZmIoJlXcymoO6LAQ6l73UL77XbJuiyn1tJslV1c/DeVIICZkHJC1k -JWumIWmbat10TWuXekG9qxf5kBdIjzb5LdXF2+6qhUVB+s06RbFo5jZMm5BX7CO5 -hwjCxAnxl4YqKE3idMDaxIzb3+KhF1nOJFl0Mdp//TBt2dzhauH8XwIDAQABo4IB -GjCCARYwDwYDVR0TAQH/BAUwAwEB/zAOBgNVHQ8BAf8EBAMCAQYwHQYDVR0OBBYE -FBiHVuBud+4kNTxOc5of1uHieX4rMB8GA1UdIwQYMBaAFBiHVuBud+4kNTxOc5of -1uHieX4rMEQGA1UdIAQ9MDswOQYEVR0gADAxMC8GCCsGAQUFBwIBFiNodHRwczov -L3d3d3cuY2VydGlnbmEuZnIvYXV0b3JpdGVzLzBtBgNVHR8EZjBkMC+gLaArhilo -dHRwOi8vY3JsLmNlcnRpZ25hLmZyL2NlcnRpZ25hcm9vdGNhLmNybDAxoC+gLYYr -aHR0cDovL2NybC5kaGlteW90aXMuY29tL2NlcnRpZ25hcm9vdGNhLmNybDANBgkq -hkiG9w0BAQsFAAOCAgEAlLieT/DjlQgi581oQfccVdV8AOItOoldaDgvUSILSo3L -6btdPrtcPbEo/uRTVRPPoZAbAh1fZkYJMyjhDSSXcNMQH+pkV5a7XdrnxIxPTGRG -HVyH41neQtGbqH6mid2PHMkwgu07nM3A6RngatgCdTer9zQoKJHyBApPNeNgJgH6 -0BGM+RFq7q89w1DTj18zeTyGqHNFkIwgtnJzFyO+B2XleJINugHA64wcZr+shncB -lA2c5uk5jR+mUYyZDDl34bSb+hxnV29qao6pK0xXeXpXIs/NX2NGjVxZOob4Mkdi -o2cNGJHc+6Zr9UhhcyNZjgKnvETq9Emd8VRY+WCv2hikLyhF3HqgiIZd8zvn/yk1 -gPxkQ5Tm4xxvvq0OKmOZK8l+hfZx6AYDlf7ej0gcWtSS6Cvu5zHbugRqh5jnxV/v -faci9wHYTfmJ0A6aBVmknpjZbyvKcL5kwlWj9Omvw5Ip3IgWJJk8jSaYtlu3zM63 -Nwf9JtmYhST/WSMDmu2dnajkXjjO11INb9I/bbEFa0nOipFGc/T2L/Coc3cOZayh -jWZSaX5LaAzHHjcng6WMxwLkFM1JAbBzs/3GkDpv0mztO+7skb6iQ12LAEpmJURw -3kAP+HwV96LOPNdeE4yBFxgX0b3xdxA61GU5wSesVywlVP+i2k+KYTlerj1KjL0= ------END CERTIFICATE----- - -# Issuer: CN=emSign Root CA - G1 O=eMudhra Technologies Limited OU=emSign PKI -# Subject: CN=emSign Root CA - G1 O=eMudhra Technologies Limited OU=emSign PKI -# Label: "emSign Root CA - G1" -# Serial: 235931866688319308814040 -# MD5 Fingerprint: 9c:42:84:57:dd:cb:0b:a7:2e:95:ad:b6:f3:da:bc:ac -# SHA1 Fingerprint: 8a:c7:ad:8f:73:ac:4e:c1:b5:75:4d:a5:40:f4:fc:cf:7c:b5:8e:8c -# SHA256 Fingerprint: 40:f6:af:03:46:a9:9a:a1:cd:1d:55:5a:4e:9c:ce:62:c7:f9:63:46:03:ee:40:66:15:83:3d:c8:c8:d0:03:67 ------BEGIN CERTIFICATE----- -MIIDlDCCAnygAwIBAgIKMfXkYgxsWO3W2DANBgkqhkiG9w0BAQsFADBnMQswCQYD -VQQGEwJJTjETMBEGA1UECxMKZW1TaWduIFBLSTElMCMGA1UEChMcZU11ZGhyYSBU -ZWNobm9sb2dpZXMgTGltaXRlZDEcMBoGA1UEAxMTZW1TaWduIFJvb3QgQ0EgLSBH -MTAeFw0xODAyMTgxODMwMDBaFw00MzAyMTgxODMwMDBaMGcxCzAJBgNVBAYTAklO -MRMwEQYDVQQLEwplbVNpZ24gUEtJMSUwIwYDVQQKExxlTXVkaHJhIFRlY2hub2xv -Z2llcyBMaW1pdGVkMRwwGgYDVQQDExNlbVNpZ24gUm9vdCBDQSAtIEcxMIIBIjAN -BgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAk0u76WaK7p1b1TST0Bsew+eeuGQz -f2N4aLTNLnF115sgxk0pvLZoYIr3IZpWNVrzdr3YzZr/k1ZLpVkGoZM0Kd0WNHVO -8oG0x5ZOrRkVUkr+PHB1cM2vK6sVmjM8qrOLqs1D/fXqcP/tzxE7lM5OMhbTI0Aq -d7OvPAEsbO2ZLIvZTmmYsvePQbAyeGHWDV/D+qJAkh1cF+ZwPjXnorfCYuKrpDhM -tTk1b+oDafo6VGiFbdbyL0NVHpENDtjVaqSW0RM8LHhQ6DqS0hdW5TUaQBw+jSzt -Od9C4INBdN+jzcKGYEho42kLVACL5HZpIQ15TjQIXhTCzLG3rdd8cIrHhQIDAQAB -o0IwQDAdBgNVHQ4EFgQU++8Nhp6w492pufEhF38+/PB3KxowDgYDVR0PAQH/BAQD -AgEGMA8GA1UdEwEB/wQFMAMBAf8wDQYJKoZIhvcNAQELBQADggEBAFn/8oz1h31x -PaOfG1vR2vjTnGs2vZupYeveFix0PZ7mddrXuqe8QhfnPZHr5X3dPpzxz5KsbEjM -wiI/aTvFthUvozXGaCocV685743QNcMYDHsAVhzNixl03r4PEuDQqqE/AjSxcM6d -GNYIAwlG7mDgfrbESQRRfXBgvKqy/3lyeqYdPV8q+Mri/Tm3R7nrft8EI6/6nAYH -6ftjk4BAtcZsCjEozgyfz7MjNYBBjWzEN3uBL4ChQEKF6dk4jeihU80Bv2noWgby -RQuQ+q7hv53yrlc8pa6yVvSLZUDp/TGBLPQ5Cdjua6e0ph0VpZj3AYHYhX3zUVxx -iN66zB+Afko= ------END CERTIFICATE----- - -# Issuer: CN=emSign ECC Root CA - G3 O=eMudhra Technologies Limited OU=emSign PKI -# Subject: CN=emSign ECC Root CA - G3 O=eMudhra Technologies Limited OU=emSign PKI -# Label: "emSign ECC Root CA - G3" -# Serial: 287880440101571086945156 -# MD5 Fingerprint: ce:0b:72:d1:9f:88:8e:d0:50:03:e8:e3:b8:8b:67:40 -# SHA1 Fingerprint: 30:43:fa:4f:f2:57:dc:a0:c3:80:ee:2e:58:ea:78:b2:3f:e6:bb:c1 -# SHA256 Fingerprint: 86:a1:ec:ba:08:9c:4a:8d:3b:be:27:34:c6:12:ba:34:1d:81:3e:04:3c:f9:e8:a8:62:cd:5c:57:a3:6b:be:6b ------BEGIN CERTIFICATE----- -MIICTjCCAdOgAwIBAgIKPPYHqWhwDtqLhDAKBggqhkjOPQQDAzBrMQswCQYDVQQG -EwJJTjETMBEGA1UECxMKZW1TaWduIFBLSTElMCMGA1UEChMcZU11ZGhyYSBUZWNo -bm9sb2dpZXMgTGltaXRlZDEgMB4GA1UEAxMXZW1TaWduIEVDQyBSb290IENBIC0g -RzMwHhcNMTgwMjE4MTgzMDAwWhcNNDMwMjE4MTgzMDAwWjBrMQswCQYDVQQGEwJJ -TjETMBEGA1UECxMKZW1TaWduIFBLSTElMCMGA1UEChMcZU11ZGhyYSBUZWNobm9s -b2dpZXMgTGltaXRlZDEgMB4GA1UEAxMXZW1TaWduIEVDQyBSb290IENBIC0gRzMw -djAQBgcqhkjOPQIBBgUrgQQAIgNiAAQjpQy4LRL1KPOxst3iAhKAnjlfSU2fySU0 -WXTsuwYc58Byr+iuL+FBVIcUqEqy6HyC5ltqtdyzdc6LBtCGI79G1Y4PPwT01xyS -fvalY8L1X44uT6EYGQIrMgqCZH0Wk9GjQjBAMB0GA1UdDgQWBBR8XQKEE9TMipuB -zhccLikenEhjQjAOBgNVHQ8BAf8EBAMCAQYwDwYDVR0TAQH/BAUwAwEB/zAKBggq -hkjOPQQDAwNpADBmAjEAvvNhzwIQHWSVB7gYboiFBS+DCBeQyh+KTOgNG3qxrdWB -CUfvO6wIBHxcmbHtRwfSAjEAnbpV/KlK6O3t5nYBQnvI+GDZjVGLVTv7jHvrZQnD -+JbNR6iC8hZVdyR+EhCVBCyj ------END CERTIFICATE----- - -# Issuer: CN=emSign Root CA - C1 O=eMudhra Inc OU=emSign PKI -# Subject: CN=emSign Root CA - C1 O=eMudhra Inc OU=emSign PKI -# Label: "emSign Root CA - C1" -# Serial: 825510296613316004955058 -# MD5 Fingerprint: d8:e3:5d:01:21:fa:78:5a:b0:df:ba:d2:ee:2a:5f:68 -# SHA1 Fingerprint: e7:2e:f1:df:fc:b2:09:28:cf:5d:d4:d5:67:37:b1:51:cb:86:4f:01 -# SHA256 Fingerprint: 12:56:09:aa:30:1d:a0:a2:49:b9:7a:82:39:cb:6a:34:21:6f:44:dc:ac:9f:39:54:b1:42:92:f2:e8:c8:60:8f ------BEGIN CERTIFICATE----- -MIIDczCCAlugAwIBAgILAK7PALrEzzL4Q7IwDQYJKoZIhvcNAQELBQAwVjELMAkG -A1UEBhMCVVMxEzARBgNVBAsTCmVtU2lnbiBQS0kxFDASBgNVBAoTC2VNdWRocmEg -SW5jMRwwGgYDVQQDExNlbVNpZ24gUm9vdCBDQSAtIEMxMB4XDTE4MDIxODE4MzAw -MFoXDTQzMDIxODE4MzAwMFowVjELMAkGA1UEBhMCVVMxEzARBgNVBAsTCmVtU2ln -biBQS0kxFDASBgNVBAoTC2VNdWRocmEgSW5jMRwwGgYDVQQDExNlbVNpZ24gUm9v -dCBDQSAtIEMxMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAz+upufGZ -BczYKCFK83M0UYRWEPWgTywS4/oTmifQz/l5GnRfHXk5/Fv4cI7gklL35CX5VIPZ -HdPIWoU/Xse2B+4+wM6ar6xWQio5JXDWv7V7Nq2s9nPczdcdioOl+yuQFTdrHCZH -3DspVpNqs8FqOp099cGXOFgFixwR4+S0uF2FHYP+eF8LRWgYSKVGczQ7/g/IdrvH -GPMF0Ybzhe3nudkyrVWIzqa2kbBPrH4VI5b2P/AgNBbeCsbEBEV5f6f9vtKppa+c -xSMq9zwhbL2vj07FOrLzNBL834AaSaTUqZX3noleoomslMuoaJuvimUnzYnu3Yy1 -aylwQ6BpC+S5DwIDAQABo0IwQDAdBgNVHQ4EFgQU/qHgcB4qAzlSWkK+XJGFehiq -TbUwDgYDVR0PAQH/BAQDAgEGMA8GA1UdEwEB/wQFMAMBAf8wDQYJKoZIhvcNAQEL -BQADggEBAMJKVvoVIXsoounlHfv4LcQ5lkFMOycsxGwYFYDGrK9HWS8mC+M2sO87 -/kOXSTKZEhVb3xEp/6tT+LvBeA+snFOvV71ojD1pM/CjoCNjO2RnIkSt1XHLVip4 -kqNPEjE2NuLe/gDEo2APJ62gsIq1NnpSob0n9CAnYuhNlCQT5AoE6TyrLshDCUrG -YQTlSTR+08TI9Q/Aqum6VF7zYytPT1DU/rl7mYw9wC68AivTxEDkigcxHpvOJpkT -+xHqmiIMERnHXhuBUDDIlhJu58tBf5E7oke3VIAb3ADMmpDqw8NQBmIMMMAVSKeo -WXzhriKi4gp6D/piq1JM4fHfyr6DDUI= ------END CERTIFICATE----- - -# Issuer: CN=emSign ECC Root CA - C3 O=eMudhra Inc OU=emSign PKI -# Subject: CN=emSign ECC Root CA - C3 O=eMudhra Inc OU=emSign PKI -# Label: "emSign ECC Root CA - C3" -# Serial: 582948710642506000014504 -# MD5 Fingerprint: 3e:53:b3:a3:81:ee:d7:10:f8:d3:b0:1d:17:92:f5:d5 -# SHA1 Fingerprint: b6:af:43:c2:9b:81:53:7d:f6:ef:6b:c3:1f:1f:60:15:0c:ee:48:66 -# SHA256 Fingerprint: bc:4d:80:9b:15:18:9d:78:db:3e:1d:8c:f4:f9:72:6a:79:5d:a1:64:3c:a5:f1:35:8e:1d:db:0e:dc:0d:7e:b3 ------BEGIN CERTIFICATE----- -MIICKzCCAbGgAwIBAgIKe3G2gla4EnycqDAKBggqhkjOPQQDAzBaMQswCQYDVQQG -EwJVUzETMBEGA1UECxMKZW1TaWduIFBLSTEUMBIGA1UEChMLZU11ZGhyYSBJbmMx -IDAeBgNVBAMTF2VtU2lnbiBFQ0MgUm9vdCBDQSAtIEMzMB4XDTE4MDIxODE4MzAw -MFoXDTQzMDIxODE4MzAwMFowWjELMAkGA1UEBhMCVVMxEzARBgNVBAsTCmVtU2ln -biBQS0kxFDASBgNVBAoTC2VNdWRocmEgSW5jMSAwHgYDVQQDExdlbVNpZ24gRUND -IFJvb3QgQ0EgLSBDMzB2MBAGByqGSM49AgEGBSuBBAAiA2IABP2lYa57JhAd6bci -MK4G9IGzsUJxlTm801Ljr6/58pc1kjZGDoeVjbk5Wum739D+yAdBPLtVb4Ojavti -sIGJAnB9SMVK4+kiVCJNk7tCDK93nCOmfddhEc5lx/h//vXyqaNCMEAwHQYDVR0O -BBYEFPtaSNCAIEDyqOkAB2kZd6fmw/TPMA4GA1UdDwEB/wQEAwIBBjAPBgNVHRMB -Af8EBTADAQH/MAoGCCqGSM49BAMDA2gAMGUCMQC02C8Cif22TGK6Q04ThHK1rt0c -3ta13FaPWEBaLd4gTCKDypOofu4SQMfWh0/434UCMBwUZOR8loMRnLDRWmFLpg9J -0wD8ofzkpf9/rdcw0Md3f76BB1UwUCAU9Vc4CqgxUQ== ------END CERTIFICATE----- - -# Issuer: CN=Hongkong Post Root CA 3 O=Hongkong Post -# Subject: CN=Hongkong Post Root CA 3 O=Hongkong Post -# Label: "Hongkong Post Root CA 3" -# Serial: 46170865288971385588281144162979347873371282084 -# MD5 Fingerprint: 11:fc:9f:bd:73:30:02:8a:fd:3f:f3:58:b9:cb:20:f0 -# SHA1 Fingerprint: 58:a2:d0:ec:20:52:81:5b:c1:f3:f8:64:02:24:4e:c2:8e:02:4b:02 -# SHA256 Fingerprint: 5a:2f:c0:3f:0c:83:b0:90:bb:fa:40:60:4b:09:88:44:6c:76:36:18:3d:f9:84:6e:17:10:1a:44:7f:b8:ef:d6 ------BEGIN CERTIFICATE----- -MIIFzzCCA7egAwIBAgIUCBZfikyl7ADJk0DfxMauI7gcWqQwDQYJKoZIhvcNAQEL -BQAwbzELMAkGA1UEBhMCSEsxEjAQBgNVBAgTCUhvbmcgS29uZzESMBAGA1UEBxMJ -SG9uZyBLb25nMRYwFAYDVQQKEw1Ib25na29uZyBQb3N0MSAwHgYDVQQDExdIb25n -a29uZyBQb3N0IFJvb3QgQ0EgMzAeFw0xNzA2MDMwMjI5NDZaFw00MjA2MDMwMjI5 -NDZaMG8xCzAJBgNVBAYTAkhLMRIwEAYDVQQIEwlIb25nIEtvbmcxEjAQBgNVBAcT -CUhvbmcgS29uZzEWMBQGA1UEChMNSG9uZ2tvbmcgUG9zdDEgMB4GA1UEAxMXSG9u -Z2tvbmcgUG9zdCBSb290IENBIDMwggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIK -AoICAQCziNfqzg8gTr7m1gNt7ln8wlffKWihgw4+aMdoWJwcYEuJQwy51BWy7sFO -dem1p+/l6TWZ5Mwc50tfjTMwIDNT2aa71T4Tjukfh0mtUC1Qyhi+AViiE3CWu4mI -VoBc+L0sPOFMV4i707mV78vH9toxdCim5lSJ9UExyuUmGs2C4HDaOym71QP1mbpV -9WTRYA6ziUm4ii8F0oRFKHyPaFASePwLtVPLwpgchKOesL4jpNrcyCse2m5FHomY -2vkALgbpDDtw1VAliJnLzXNg99X/NWfFobxeq81KuEXryGgeDQ0URhLj0mRiikKY -vLTGCAj4/ahMZJx2Ab0vqWwzD9g/KLg8aQFChn5pwckGyuV6RmXpwtZQQS4/t+Tt -bNe/JgERohYpSms0BpDsE9K2+2p20jzt8NYt3eEV7KObLyzJPivkaTv/ciWxNoZb -x39ri1UbSsUgYT2uy1DhCDq+sI9jQVMwCFk8mB13umOResoQUGC/8Ne8lYePl8X+ -l2oBlKN8W4UdKjk60FSh0Tlxnf0h+bV78OLgAo9uliQlLKAeLKjEiafv7ZkGL7YK -TE/bosw3Gq9HhS2KX8Q0NEwA/RiTZxPRN+ZItIsGxVd7GYYKecsAyVKvQv83j+Gj -Hno9UKtjBucVtT+2RTeUN7F+8kjDf8V1/peNRY8apxpyKBpADwIDAQABo2MwYTAP -BgNVHRMBAf8EBTADAQH/MA4GA1UdDwEB/wQEAwIBBjAfBgNVHSMEGDAWgBQXnc0e -i9Y5K3DTXNSguB+wAPzFYTAdBgNVHQ4EFgQUF53NHovWOStw01zUoLgfsAD8xWEw -DQYJKoZIhvcNAQELBQADggIBAFbVe27mIgHSQpsY1Q7XZiNc4/6gx5LS6ZStS6LG -7BJ8dNVI0lkUmcDrudHr9EgwW62nV3OZqdPlt9EuWSRY3GguLmLYauRwCy0gUCCk -MpXRAJi70/33MvJJrsZ64Ee+bs7Lo3I6LWldy8joRTnU+kLBEUx3XZL7av9YROXr -gZ6voJmtvqkBZss4HTzfQx/0TW60uhdG/H39h4F5ag0zD/ov+BS5gLNdTaqX4fnk -GMX41TiMJjz98iji7lpJiCzfeT2OnpA8vUFKOt1b9pq0zj8lMH8yfaIDlNDceqFS -3m6TjRgm/VWsvY+b0s+v54Ysyx8Jb6NvqYTUc79NoXQbTiNg8swOqn+knEwlqLJm -Ozj/2ZQw9nKEvmhVEA/GcywWaZMH/rFF7buiVWqw2rVKAiUnhde3t4ZEFolsgCs+ -l6mc1X5VTMbeRRAc6uk7nwNT7u56AQIWeNTowr5GdogTPyK7SBIdUgC0An4hGh6c -JfTzPV4e0hz5sy229zdcxsshTrD3mUcYhcErulWuBurQB7Lcq9CClnXO0lD+mefP -L5/ndtFhKvshuzHQqp9HpLIiyhY6UFfEW0NnxWViA0kB60PZ2Pierc+xYw5F9KBa -LJstxabArahH9CdMOA0uG0k7UvToiIMrVCjU8jVStDKDYmlkDJGcn5fqdBb9HxEG -mpv0 ------END CERTIFICATE----- - -# Issuer: CN=Entrust Root Certification Authority - G4 O=Entrust, Inc. OU=See www.entrust.net/legal-terms/(c) 2015 Entrust, Inc. - for authorized use only -# Subject: CN=Entrust Root Certification Authority - G4 O=Entrust, Inc. OU=See www.entrust.net/legal-terms/(c) 2015 Entrust, Inc. - for authorized use only -# Label: "Entrust Root Certification Authority - G4" -# Serial: 289383649854506086828220374796556676440 -# MD5 Fingerprint: 89:53:f1:83:23:b7:7c:8e:05:f1:8c:71:38:4e:1f:88 -# SHA1 Fingerprint: 14:88:4e:86:26:37:b0:26:af:59:62:5c:40:77:ec:35:29:ba:96:01 -# SHA256 Fingerprint: db:35:17:d1:f6:73:2a:2d:5a:b9:7c:53:3e:c7:07:79:ee:32:70:a6:2f:b4:ac:42:38:37:24:60:e6:f0:1e:88 ------BEGIN CERTIFICATE----- -MIIGSzCCBDOgAwIBAgIRANm1Q3+vqTkPAAAAAFVlrVgwDQYJKoZIhvcNAQELBQAw -gb4xCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1FbnRydXN0LCBJbmMuMSgwJgYDVQQL -Ex9TZWUgd3d3LmVudHJ1c3QubmV0L2xlZ2FsLXRlcm1zMTkwNwYDVQQLEzAoYykg -MjAxNSBFbnRydXN0LCBJbmMuIC0gZm9yIGF1dGhvcml6ZWQgdXNlIG9ubHkxMjAw -BgNVBAMTKUVudHJ1c3QgUm9vdCBDZXJ0aWZpY2F0aW9uIEF1dGhvcml0eSAtIEc0 -MB4XDTE1MDUyNzExMTExNloXDTM3MTIyNzExNDExNlowgb4xCzAJBgNVBAYTAlVT -MRYwFAYDVQQKEw1FbnRydXN0LCBJbmMuMSgwJgYDVQQLEx9TZWUgd3d3LmVudHJ1 -c3QubmV0L2xlZ2FsLXRlcm1zMTkwNwYDVQQLEzAoYykgMjAxNSBFbnRydXN0LCBJ -bmMuIC0gZm9yIGF1dGhvcml6ZWQgdXNlIG9ubHkxMjAwBgNVBAMTKUVudHJ1c3Qg -Um9vdCBDZXJ0aWZpY2F0aW9uIEF1dGhvcml0eSAtIEc0MIICIjANBgkqhkiG9w0B -AQEFAAOCAg8AMIICCgKCAgEAsewsQu7i0TD/pZJH4i3DumSXbcr3DbVZwbPLqGgZ -2K+EbTBwXX7zLtJTmeH+H17ZSK9dE43b/2MzTdMAArzE+NEGCJR5WIoV3imz/f3E -T+iq4qA7ec2/a0My3dl0ELn39GjUu9CH1apLiipvKgS1sqbHoHrmSKvS0VnM1n4j -5pds8ELl3FFLFUHtSUrJ3hCX1nbB76W1NhSXNdh4IjVS70O92yfbYVaCNNzLiGAM -C1rlLAHGVK/XqsEQe9IFWrhAnoanw5CGAlZSCXqc0ieCU0plUmr1POeo8pyvi73T -DtTUXm6Hnmo9RR3RXRv06QqsYJn7ibT/mCzPfB3pAqoEmh643IhuJbNsZvc8kPNX -wbMv9W3y+8qh+CmdRouzavbmZwe+LGcKKh9asj5XxNMhIWNlUpEbsZmOeX7m640A -2Vqq6nPopIICR5b+W45UYaPrL0swsIsjdXJ8ITzI9vF01Bx7owVV7rtNOzK+mndm -nqxpkCIHH2E6lr7lmk/MBTwoWdPBDFSoWWG9yHJM6Nyfh3+9nEg2XpWjDrk4JFX8 -dWbrAuMINClKxuMrLzOg2qOGpRKX/YAr2hRC45K9PvJdXmd0LhyIRyk0X+IyqJwl -N4y6mACXi0mWHv0liqzc2thddG5msP9E36EYxr5ILzeUePiVSj9/E15dWf10hkNj -c0kCAwEAAaNCMEAwDwYDVR0TAQH/BAUwAwEB/zAOBgNVHQ8BAf8EBAMCAQYwHQYD -VR0OBBYEFJ84xFYjwznooHFs6FRM5Og6sb9nMA0GCSqGSIb3DQEBCwUAA4ICAQAS -5UKme4sPDORGpbZgQIeMJX6tuGguW8ZAdjwD+MlZ9POrYs4QjbRaZIxowLByQzTS -Gwv2LFPSypBLhmb8qoMi9IsabyZIrHZ3CL/FmFz0Jomee8O5ZDIBf9PD3Vht7LGr -hFV0d4QEJ1JrhkzO3bll/9bGXp+aEJlLdWr+aumXIOTkdnrG0CSqkM0gkLpHZPt/ -B7NTeLUKYvJzQ85BK4FqLoUWlFPUa19yIqtRLULVAJyZv967lDtX/Zr1hstWO1uI -AeV8KEsD+UmDfLJ/fOPtjqF/YFOOVZ1QNBIPt5d7bIdKROf1beyAN/BYGW5KaHbw -H5Lk6rWS02FREAutp9lfx1/cH6NcjKF+m7ee01ZvZl4HliDtC3T7Zk6LERXpgUl+ -b7DUUH8i119lAg2m9IUe2K4GS0qn0jFmwvjO5QimpAKWRGhXxNUzzxkvFMSUHHuk -2fCfDrGA4tGeEWSpiBE6doLlYsKA2KSD7ZPvfC+QsDJMlhVoSFLUmQjAJOgc47Ol -IQ6SwJAfzyBfyjs4x7dtOvPmRLgOMWuIjnDrnBdSqEGULoe256YSxXXfW8AKbnuk -5F6G+TaU33fD6Q3AOfF5u0aOq0NZJ7cguyPpVkAh7DE9ZapD8j3fcEThuk0mEDuY -n/PIjhs4ViFqUZPTkcpG2om3PVODLAgfi49T3f+sHw== ------END CERTIFICATE----- - -# Issuer: CN=Microsoft ECC Root Certificate Authority 2017 O=Microsoft Corporation -# Subject: CN=Microsoft ECC Root Certificate Authority 2017 O=Microsoft Corporation -# Label: "Microsoft ECC Root Certificate Authority 2017" -# Serial: 136839042543790627607696632466672567020 -# MD5 Fingerprint: dd:a1:03:e6:4a:93:10:d1:bf:f0:19:42:cb:fe:ed:67 -# SHA1 Fingerprint: 99:9a:64:c3:7f:f4:7d:9f:ab:95:f1:47:69:89:14:60:ee:c4:c3:c5 -# SHA256 Fingerprint: 35:8d:f3:9d:76:4a:f9:e1:b7:66:e9:c9:72:df:35:2e:e1:5c:fa:c2:27:af:6a:d1:d7:0e:8e:4a:6e:dc:ba:02 ------BEGIN CERTIFICATE----- -MIICWTCCAd+gAwIBAgIQZvI9r4fei7FK6gxXMQHC7DAKBggqhkjOPQQDAzBlMQsw -CQYDVQQGEwJVUzEeMBwGA1UEChMVTWljcm9zb2Z0IENvcnBvcmF0aW9uMTYwNAYD -VQQDEy1NaWNyb3NvZnQgRUNDIFJvb3QgQ2VydGlmaWNhdGUgQXV0aG9yaXR5IDIw -MTcwHhcNMTkxMjE4MjMwNjQ1WhcNNDIwNzE4MjMxNjA0WjBlMQswCQYDVQQGEwJV -UzEeMBwGA1UEChMVTWljcm9zb2Z0IENvcnBvcmF0aW9uMTYwNAYDVQQDEy1NaWNy -b3NvZnQgRUNDIFJvb3QgQ2VydGlmaWNhdGUgQXV0aG9yaXR5IDIwMTcwdjAQBgcq -hkjOPQIBBgUrgQQAIgNiAATUvD0CQnVBEyPNgASGAlEvaqiBYgtlzPbKnR5vSmZR -ogPZnZH6thaxjG7efM3beaYvzrvOcS/lpaso7GMEZpn4+vKTEAXhgShC48Zo9OYb -hGBKia/teQ87zvH2RPUBeMCjVDBSMA4GA1UdDwEB/wQEAwIBhjAPBgNVHRMBAf8E -BTADAQH/MB0GA1UdDgQWBBTIy5lycFIM+Oa+sgRXKSrPQhDtNTAQBgkrBgEEAYI3 -FQEEAwIBADAKBggqhkjOPQQDAwNoADBlAjBY8k3qDPlfXu5gKcs68tvWMoQZP3zV -L8KxzJOuULsJMsbG7X7JNpQS5GiFBqIb0C8CMQCZ6Ra0DvpWSNSkMBaReNtUjGUB -iudQZsIxtzm6uBoiB078a1QWIP8rtedMDE2mT3M= ------END CERTIFICATE----- - -# Issuer: CN=Microsoft RSA Root Certificate Authority 2017 O=Microsoft Corporation -# Subject: CN=Microsoft RSA Root Certificate Authority 2017 O=Microsoft Corporation -# Label: "Microsoft RSA Root Certificate Authority 2017" -# Serial: 40975477897264996090493496164228220339 -# MD5 Fingerprint: 10:ff:00:ff:cf:c9:f8:c7:7a:c0:ee:35:8e:c9:0f:47 -# SHA1 Fingerprint: 73:a5:e6:4a:3b:ff:83:16:ff:0e:dc:cc:61:8a:90:6e:4e:ae:4d:74 -# SHA256 Fingerprint: c7:41:f7:0f:4b:2a:8d:88:bf:2e:71:c1:41:22:ef:53:ef:10:eb:a0:cf:a5:e6:4c:fa:20:f4:18:85:30:73:e0 ------BEGIN CERTIFICATE----- -MIIFqDCCA5CgAwIBAgIQHtOXCV/YtLNHcB6qvn9FszANBgkqhkiG9w0BAQwFADBl -MQswCQYDVQQGEwJVUzEeMBwGA1UEChMVTWljcm9zb2Z0IENvcnBvcmF0aW9uMTYw -NAYDVQQDEy1NaWNyb3NvZnQgUlNBIFJvb3QgQ2VydGlmaWNhdGUgQXV0aG9yaXR5 -IDIwMTcwHhcNMTkxMjE4MjI1MTIyWhcNNDIwNzE4MjMwMDIzWjBlMQswCQYDVQQG -EwJVUzEeMBwGA1UEChMVTWljcm9zb2Z0IENvcnBvcmF0aW9uMTYwNAYDVQQDEy1N -aWNyb3NvZnQgUlNBIFJvb3QgQ2VydGlmaWNhdGUgQXV0aG9yaXR5IDIwMTcwggIi -MA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQDKW76UM4wplZEWCpW9R2LBifOZ -Nt9GkMml7Xhqb0eRaPgnZ1AzHaGm++DlQ6OEAlcBXZxIQIJTELy/xztokLaCLeX0 -ZdDMbRnMlfl7rEqUrQ7eS0MdhweSE5CAg2Q1OQT85elss7YfUJQ4ZVBcF0a5toW1 -HLUX6NZFndiyJrDKxHBKrmCk3bPZ7Pw71VdyvD/IybLeS2v4I2wDwAW9lcfNcztm -gGTjGqwu+UcF8ga2m3P1eDNbx6H7JyqhtJqRjJHTOoI+dkC0zVJhUXAoP8XFWvLJ -jEm7FFtNyP9nTUwSlq31/niol4fX/V4ggNyhSyL71Imtus5Hl0dVe49FyGcohJUc -aDDv70ngNXtk55iwlNpNhTs+VcQor1fznhPbRiefHqJeRIOkpcrVE7NLP8TjwuaG -YaRSMLl6IE9vDzhTyzMMEyuP1pq9KsgtsRx9S1HKR9FIJ3Jdh+vVReZIZZ2vUpC6 -W6IYZVcSn2i51BVrlMRpIpj0M+Dt+VGOQVDJNE92kKz8OMHY4Xu54+OU4UZpyw4K -UGsTuqwPN1q3ErWQgR5WrlcihtnJ0tHXUeOrO8ZV/R4O03QK0dqq6mm4lyiPSMQH -+FJDOvTKVTUssKZqwJz58oHhEmrARdlns87/I6KJClTUFLkqqNfs+avNJVgyeY+Q -W5g5xAgGwax/Dj0ApQIDAQABo1QwUjAOBgNVHQ8BAf8EBAMCAYYwDwYDVR0TAQH/ -BAUwAwEB/zAdBgNVHQ4EFgQUCctZf4aycI8awznjwNnpv7tNsiMwEAYJKwYBBAGC -NxUBBAMCAQAwDQYJKoZIhvcNAQEMBQADggIBAKyvPl3CEZaJjqPnktaXFbgToqZC -LgLNFgVZJ8og6Lq46BrsTaiXVq5lQ7GPAJtSzVXNUzltYkyLDVt8LkS/gxCP81OC -gMNPOsduET/m4xaRhPtthH80dK2Jp86519efhGSSvpWhrQlTM93uCupKUY5vVau6 -tZRGrox/2KJQJWVggEbbMwSubLWYdFQl3JPk+ONVFT24bcMKpBLBaYVu32TxU5nh -SnUgnZUP5NbcA/FZGOhHibJXWpS2qdgXKxdJ5XbLwVaZOjex/2kskZGT4d9Mozd2 -TaGf+G0eHdP67Pv0RR0Tbc/3WeUiJ3IrhvNXuzDtJE3cfVa7o7P4NHmJweDyAmH3 -pvwPuxwXC65B2Xy9J6P9LjrRk5Sxcx0ki69bIImtt2dmefU6xqaWM/5TkshGsRGR -xpl/j8nWZjEgQRCHLQzWwa80mMpkg/sTV9HB8Dx6jKXB/ZUhoHHBk2dxEuqPiApp -GWSZI1b7rCoucL5mxAyE7+WL85MB+GqQk2dLsmijtWKP6T+MejteD+eMuMZ87zf9 -dOLITzNy4ZQ5bb0Sr74MTnB8G2+NszKTc0QWbej09+CVgI+WXTik9KveCjCHk9hN -AHFiRSdLOkKEW39lt2c0Ui2cFmuqqNh7o0JMcccMyj6D5KbvtwEwXlGjefVwaaZB -RA+GsCyRxj3qrg+E ------END CERTIFICATE----- - -# Issuer: CN=e-Szigno Root CA 2017 O=Microsec Ltd. -# Subject: CN=e-Szigno Root CA 2017 O=Microsec Ltd. -# Label: "e-Szigno Root CA 2017" -# Serial: 411379200276854331539784714 -# MD5 Fingerprint: de:1f:f6:9e:84:ae:a7:b4:21:ce:1e:58:7d:d1:84:98 -# SHA1 Fingerprint: 89:d4:83:03:4f:9e:9a:48:80:5f:72:37:d4:a9:a6:ef:cb:7c:1f:d1 -# SHA256 Fingerprint: be:b0:0b:30:83:9b:9b:c3:2c:32:e4:44:79:05:95:06:41:f2:64:21:b1:5e:d0:89:19:8b:51:8a:e2:ea:1b:99 ------BEGIN CERTIFICATE----- -MIICQDCCAeWgAwIBAgIMAVRI7yH9l1kN9QQKMAoGCCqGSM49BAMCMHExCzAJBgNV -BAYTAkhVMREwDwYDVQQHDAhCdWRhcGVzdDEWMBQGA1UECgwNTWljcm9zZWMgTHRk -LjEXMBUGA1UEYQwOVkFUSFUtMjM1ODQ0OTcxHjAcBgNVBAMMFWUtU3ppZ25vIFJv -b3QgQ0EgMjAxNzAeFw0xNzA4MjIxMjA3MDZaFw00MjA4MjIxMjA3MDZaMHExCzAJ -BgNVBAYTAkhVMREwDwYDVQQHDAhCdWRhcGVzdDEWMBQGA1UECgwNTWljcm9zZWMg -THRkLjEXMBUGA1UEYQwOVkFUSFUtMjM1ODQ0OTcxHjAcBgNVBAMMFWUtU3ppZ25v -IFJvb3QgQ0EgMjAxNzBZMBMGByqGSM49AgEGCCqGSM49AwEHA0IABJbcPYrYsHtv -xie+RJCxs1YVe45DJH0ahFnuY2iyxl6H0BVIHqiQrb1TotreOpCmYF9oMrWGQd+H -Wyx7xf58etqjYzBhMA8GA1UdEwEB/wQFMAMBAf8wDgYDVR0PAQH/BAQDAgEGMB0G -A1UdDgQWBBSHERUI0arBeAyxr87GyZDvvzAEwDAfBgNVHSMEGDAWgBSHERUI0arB -eAyxr87GyZDvvzAEwDAKBggqhkjOPQQDAgNJADBGAiEAtVfd14pVCzbhhkT61Nlo -jbjcI4qKDdQvfepz7L9NbKgCIQDLpbQS+ue16M9+k/zzNY9vTlp8tLxOsvxyqltZ -+efcMQ== ------END CERTIFICATE----- - -# Issuer: O=CERTSIGN SA OU=certSIGN ROOT CA G2 -# Subject: O=CERTSIGN SA OU=certSIGN ROOT CA G2 -# Label: "certSIGN Root CA G2" -# Serial: 313609486401300475190 -# MD5 Fingerprint: 8c:f1:75:8a:c6:19:cf:94:b7:f7:65:20:87:c3:97:c7 -# SHA1 Fingerprint: 26:f9:93:b4:ed:3d:28:27:b0:b9:4b:a7:e9:15:1d:a3:8d:92:e5:32 -# SHA256 Fingerprint: 65:7c:fe:2f:a7:3f:aa:38:46:25:71:f3:32:a2:36:3a:46:fc:e7:02:09:51:71:07:02:cd:fb:b6:ee:da:33:05 ------BEGIN CERTIFICATE----- -MIIFRzCCAy+gAwIBAgIJEQA0tk7GNi02MA0GCSqGSIb3DQEBCwUAMEExCzAJBgNV -BAYTAlJPMRQwEgYDVQQKEwtDRVJUU0lHTiBTQTEcMBoGA1UECxMTY2VydFNJR04g -Uk9PVCBDQSBHMjAeFw0xNzAyMDYwOTI3MzVaFw00MjAyMDYwOTI3MzVaMEExCzAJ -BgNVBAYTAlJPMRQwEgYDVQQKEwtDRVJUU0lHTiBTQTEcMBoGA1UECxMTY2VydFNJ -R04gUk9PVCBDQSBHMjCCAiIwDQYJKoZIhvcNAQEBBQADggIPADCCAgoCggIBAMDF -dRmRfUR0dIf+DjuW3NgBFszuY5HnC2/OOwppGnzC46+CjobXXo9X69MhWf05N0Iw -vlDqtg+piNguLWkh59E3GE59kdUWX2tbAMI5Qw02hVK5U2UPHULlj88F0+7cDBrZ -uIt4ImfkabBoxTzkbFpG583H+u/E7Eu9aqSs/cwoUe+StCmrqzWaTOTECMYmzPhp -n+Sc8CnTXPnGFiWeI8MgwT0PPzhAsP6CRDiqWhqKa2NYOLQV07YRaXseVO6MGiKs -cpc/I1mbySKEwQdPzH/iV8oScLumZfNpdWO9lfsbl83kqK/20U6o2YpxJM02PbyW -xPFsqa7lzw1uKA2wDrXKUXt4FMMgL3/7FFXhEZn91QqhngLjYl/rNUssuHLoPj1P -rCy7Lobio3aP5ZMqz6WryFyNSwb/EkaseMsUBzXgqd+L6a8VTxaJW732jcZZroiF -DsGJ6x9nxUWO/203Nit4ZoORUSs9/1F3dmKh7Gc+PoGD4FapUB8fepmrY7+EF3fx -DTvf95xhszWYijqy7DwaNz9+j5LP2RIUZNoQAhVB/0/E6xyjyfqZ90bp4RjZsbgy -LcsUDFDYg2WD7rlcz8sFWkz6GZdr1l0T08JcVLwyc6B49fFtHsufpaafItzRUZ6C -eWRgKRM+o/1Pcmqr4tTluCRVLERLiohEnMqE0yo7AgMBAAGjQjBAMA8GA1UdEwEB -/wQFMAMBAf8wDgYDVR0PAQH/BAQDAgEGMB0GA1UdDgQWBBSCIS1mxteg4BXrzkwJ -d8RgnlRuAzANBgkqhkiG9w0BAQsFAAOCAgEAYN4auOfyYILVAzOBywaK8SJJ6ejq -kX/GM15oGQOGO0MBzwdw5AgeZYWR5hEit/UCI46uuR59H35s5r0l1ZUa8gWmr4UC -b6741jH/JclKyMeKqdmfS0mbEVeZkkMR3rYzpMzXjWR91M08KCy0mpbqTfXERMQl -qiCA2ClV9+BB/AYm/7k29UMUA2Z44RGx2iBfRgB4ACGlHgAoYXhvqAEBj500mv/0 -OJD7uNGzcgbJceaBxXntC6Z58hMLnPddDnskk7RI24Zf3lCGeOdA5jGokHZwYa+c -NywRtYK3qq4kNFtyDGkNzVmf9nGvnAvRCjj5BiKDUyUM/FHE5r7iOZULJK2v0ZXk -ltd0ZGtxTgI8qoXzIKNDOXZbbFD+mpwUHmUUihW9o4JFWklWatKcsWMy5WHgUyIO -pwpJ6st+H6jiYoD2EEVSmAYY3qXNL3+q1Ok+CHLsIwMCPKaq2LxndD0UF/tUSxfj -03k9bWtJySgOLnRQvwzZRjoQhsmnP+mg7H/rpXdYaXHmgwo38oZJar55CJD2AhZk -PuXaTH4MNMn5X7azKFGnpyuqSfqNZSlO42sTp5SjLVFteAxEy9/eCG/Oo2Sr05WE -1LlSVHJ7liXMvGnjSG4N0MedJ5qq+BOS3R7fY581qRY27Iy4g/Q9iY/NtBde17MX -QRBdJ3NghVdJIgc= ------END CERTIFICATE----- - -# Issuer: CN=Trustwave Global Certification Authority O=Trustwave Holdings, Inc. -# Subject: CN=Trustwave Global Certification Authority O=Trustwave Holdings, Inc. -# Label: "Trustwave Global Certification Authority" -# Serial: 1846098327275375458322922162 -# MD5 Fingerprint: f8:1c:18:2d:2f:ba:5f:6d:a1:6c:bc:c7:ab:91:c7:0e -# SHA1 Fingerprint: 2f:8f:36:4f:e1:58:97:44:21:59:87:a5:2a:9a:d0:69:95:26:7f:b5 -# SHA256 Fingerprint: 97:55:20:15:f5:dd:fc:3c:87:88:c0:06:94:45:55:40:88:94:45:00:84:f1:00:86:70:86:bc:1a:2b:b5:8d:c8 ------BEGIN CERTIFICATE----- -MIIF2jCCA8KgAwIBAgIMBfcOhtpJ80Y1LrqyMA0GCSqGSIb3DQEBCwUAMIGIMQsw -CQYDVQQGEwJVUzERMA8GA1UECAwISWxsaW5vaXMxEDAOBgNVBAcMB0NoaWNhZ28x -ITAfBgNVBAoMGFRydXN0d2F2ZSBIb2xkaW5ncywgSW5jLjExMC8GA1UEAwwoVHJ1 -c3R3YXZlIEdsb2JhbCBDZXJ0aWZpY2F0aW9uIEF1dGhvcml0eTAeFw0xNzA4MjMx -OTM0MTJaFw00MjA4MjMxOTM0MTJaMIGIMQswCQYDVQQGEwJVUzERMA8GA1UECAwI -SWxsaW5vaXMxEDAOBgNVBAcMB0NoaWNhZ28xITAfBgNVBAoMGFRydXN0d2F2ZSBI -b2xkaW5ncywgSW5jLjExMC8GA1UEAwwoVHJ1c3R3YXZlIEdsb2JhbCBDZXJ0aWZp -Y2F0aW9uIEF1dGhvcml0eTCCAiIwDQYJKoZIhvcNAQEBBQADggIPADCCAgoCggIB -ALldUShLPDeS0YLOvR29zd24q88KPuFd5dyqCblXAj7mY2Hf8g+CY66j96xz0Xzn -swuvCAAJWX/NKSqIk4cXGIDtiLK0thAfLdZfVaITXdHG6wZWiYj+rDKd/VzDBcdu -7oaJuogDnXIhhpCujwOl3J+IKMujkkkP7NAP4m1ET4BqstTnoApTAbqOl5F2brz8 -1Ws25kCI1nsvXwXoLG0R8+eyvpJETNKXpP7ScoFDB5zpET71ixpZfR9oWN0EACyW -80OzfpgZdNmcc9kYvkHHNHnZ9GLCQ7mzJ7Aiy/k9UscwR7PJPrhq4ufogXBeQotP -JqX+OsIgbrv4Fo7NDKm0G2x2EOFYeUY+VM6AqFcJNykbmROPDMjWLBz7BegIlT1l -RtzuzWniTY+HKE40Cz7PFNm73bZQmq131BnW2hqIyE4bJ3XYsgjxroMwuREOzYfw -hI0Vcnyh78zyiGG69Gm7DIwLdVcEuE4qFC49DxweMqZiNu5m4iK4BUBjECLzMx10 -coos9TkpoNPnG4CELcU9402x/RpvumUHO1jsQkUm+9jaJXLE9gCxInm943xZYkqc -BW89zubWR2OZxiRvchLIrH+QtAuRcOi35hYQcRfO3gZPSEF9NUqjifLJS3tBEW1n -twiYTOURGa5CgNz7kAXU+FDKvuStx8KU1xad5hePrzb7AgMBAAGjQjBAMA8GA1Ud -EwEB/wQFMAMBAf8wHQYDVR0OBBYEFJngGWcNYtt2s9o9uFvo/ULSMQ6HMA4GA1Ud -DwEB/wQEAwIBBjANBgkqhkiG9w0BAQsFAAOCAgEAmHNw4rDT7TnsTGDZqRKGFx6W -0OhUKDtkLSGm+J1WE2pIPU/HPinbbViDVD2HfSMF1OQc3Og4ZYbFdada2zUFvXfe -uyk3QAUHw5RSn8pk3fEbK9xGChACMf1KaA0HZJDmHvUqoai7PF35owgLEQzxPy0Q -lG/+4jSHg9bP5Rs1bdID4bANqKCqRieCNqcVtgimQlRXtpla4gt5kNdXElE1GYhB -aCXUNxeEFfsBctyV3lImIJgm4nb1J2/6ADtKYdkNy1GTKv0WBpanI5ojSP5RvbbE -sLFUzt5sQa0WZ37b/TjNuThOssFgy50X31ieemKyJo90lZvkWx3SD92YHJtZuSPT -MaCm/zjdzyBP6VhWOmfD0faZmZ26NraAL4hHT4a/RDqA5Dccprrql5gR0IRiR2Qe -qu5AvzSxnI9O4fKSTx+O856X3vOmeWqJcU9LJxdI/uz0UA9PSX3MReO9ekDFQdxh -VicGaeVyQYHTtgGJoC86cnn+OjC/QezHYj6RS8fZMXZC+fc8Y+wmjHMMfRod6qh8 -h6jCJ3zhM0EPz8/8AKAigJ5Kp28AsEFFtyLKaEjFQqKu3R3y4G5OBVixwJAWKqQ9 -EEC+j2Jjg6mcgn0tAumDMHzLJ8n9HmYAsC7TIS+OMxZsmO0QqAfWzJPP29FpHOTK -yeC2nOnOcXHebD8WpHk= ------END CERTIFICATE----- - -# Issuer: CN=Trustwave Global ECC P256 Certification Authority O=Trustwave Holdings, Inc. -# Subject: CN=Trustwave Global ECC P256 Certification Authority O=Trustwave Holdings, Inc. -# Label: "Trustwave Global ECC P256 Certification Authority" -# Serial: 4151900041497450638097112925 -# MD5 Fingerprint: 5b:44:e3:8d:5d:36:86:26:e8:0d:05:d2:59:a7:83:54 -# SHA1 Fingerprint: b4:90:82:dd:45:0c:be:8b:5b:b1:66:d3:e2:a4:08:26:cd:ed:42:cf -# SHA256 Fingerprint: 94:5b:bc:82:5e:a5:54:f4:89:d1:fd:51:a7:3d:df:2e:a6:24:ac:70:19:a0:52:05:22:5c:22:a7:8c:cf:a8:b4 ------BEGIN CERTIFICATE----- -MIICYDCCAgegAwIBAgIMDWpfCD8oXD5Rld9dMAoGCCqGSM49BAMCMIGRMQswCQYD -VQQGEwJVUzERMA8GA1UECBMISWxsaW5vaXMxEDAOBgNVBAcTB0NoaWNhZ28xITAf -BgNVBAoTGFRydXN0d2F2ZSBIb2xkaW5ncywgSW5jLjE6MDgGA1UEAxMxVHJ1c3R3 -YXZlIEdsb2JhbCBFQ0MgUDI1NiBDZXJ0aWZpY2F0aW9uIEF1dGhvcml0eTAeFw0x -NzA4MjMxOTM1MTBaFw00MjA4MjMxOTM1MTBaMIGRMQswCQYDVQQGEwJVUzERMA8G -A1UECBMISWxsaW5vaXMxEDAOBgNVBAcTB0NoaWNhZ28xITAfBgNVBAoTGFRydXN0 -d2F2ZSBIb2xkaW5ncywgSW5jLjE6MDgGA1UEAxMxVHJ1c3R3YXZlIEdsb2JhbCBF -Q0MgUDI1NiBDZXJ0aWZpY2F0aW9uIEF1dGhvcml0eTBZMBMGByqGSM49AgEGCCqG -SM49AwEHA0IABH77bOYj43MyCMpg5lOcunSNGLB4kFKA3TjASh3RqMyTpJcGOMoN -FWLGjgEqZZ2q3zSRLoHB5DOSMcT9CTqmP62jQzBBMA8GA1UdEwEB/wQFMAMBAf8w -DwYDVR0PAQH/BAUDAwcGADAdBgNVHQ4EFgQUo0EGrJBt0UrrdaVKEJmzsaGLSvcw -CgYIKoZIzj0EAwIDRwAwRAIgB+ZU2g6gWrKuEZ+Hxbb/ad4lvvigtwjzRM4q3wgh -DDcCIC0mA6AFvWvR9lz4ZcyGbbOcNEhjhAnFjXca4syc4XR7 ------END CERTIFICATE----- - -# Issuer: CN=Trustwave Global ECC P384 Certification Authority O=Trustwave Holdings, Inc. -# Subject: CN=Trustwave Global ECC P384 Certification Authority O=Trustwave Holdings, Inc. -# Label: "Trustwave Global ECC P384 Certification Authority" -# Serial: 2704997926503831671788816187 -# MD5 Fingerprint: ea:cf:60:c4:3b:b9:15:29:40:a1:97:ed:78:27:93:d6 -# SHA1 Fingerprint: e7:f3:a3:c8:cf:6f:c3:04:2e:6d:0e:67:32:c5:9e:68:95:0d:5e:d2 -# SHA256 Fingerprint: 55:90:38:59:c8:c0:c3:eb:b8:75:9e:ce:4e:25:57:22:5f:f5:75:8b:bd:38:eb:d4:82:76:60:1e:1b:d5:80:97 ------BEGIN CERTIFICATE----- -MIICnTCCAiSgAwIBAgIMCL2Fl2yZJ6SAaEc7MAoGCCqGSM49BAMDMIGRMQswCQYD -VQQGEwJVUzERMA8GA1UECBMISWxsaW5vaXMxEDAOBgNVBAcTB0NoaWNhZ28xITAf -BgNVBAoTGFRydXN0d2F2ZSBIb2xkaW5ncywgSW5jLjE6MDgGA1UEAxMxVHJ1c3R3 -YXZlIEdsb2JhbCBFQ0MgUDM4NCBDZXJ0aWZpY2F0aW9uIEF1dGhvcml0eTAeFw0x -NzA4MjMxOTM2NDNaFw00MjA4MjMxOTM2NDNaMIGRMQswCQYDVQQGEwJVUzERMA8G -A1UECBMISWxsaW5vaXMxEDAOBgNVBAcTB0NoaWNhZ28xITAfBgNVBAoTGFRydXN0 -d2F2ZSBIb2xkaW5ncywgSW5jLjE6MDgGA1UEAxMxVHJ1c3R3YXZlIEdsb2JhbCBF -Q0MgUDM4NCBDZXJ0aWZpY2F0aW9uIEF1dGhvcml0eTB2MBAGByqGSM49AgEGBSuB -BAAiA2IABGvaDXU1CDFHBa5FmVXxERMuSvgQMSOjfoPTfygIOiYaOs+Xgh+AtycJ -j9GOMMQKmw6sWASr9zZ9lCOkmwqKi6vr/TklZvFe/oyujUF5nQlgziip04pt89ZF -1PKYhDhloKNDMEEwDwYDVR0TAQH/BAUwAwEB/zAPBgNVHQ8BAf8EBQMDBwYAMB0G -A1UdDgQWBBRVqYSJ0sEyvRjLbKYHTsjnnb6CkDAKBggqhkjOPQQDAwNnADBkAjA3 -AZKXRRJ+oPM+rRk6ct30UJMDEr5E0k9BpIycnR+j9sKS50gU/k6bpZFXrsY3crsC -MGclCrEMXu6pY5Jv5ZAL/mYiykf9ijH3g/56vxC+GCsej/YpHpRZ744hN8tRmKVu -Sw== ------END CERTIFICATE----- - -# Issuer: CN=NAVER Global Root Certification Authority O=NAVER BUSINESS PLATFORM Corp. -# Subject: CN=NAVER Global Root Certification Authority O=NAVER BUSINESS PLATFORM Corp. -# Label: "NAVER Global Root Certification Authority" -# Serial: 9013692873798656336226253319739695165984492813 -# MD5 Fingerprint: c8:7e:41:f6:25:3b:f5:09:b3:17:e8:46:3d:bf:d0:9b -# SHA1 Fingerprint: 8f:6b:f2:a9:27:4a:da:14:a0:c4:f4:8e:61:27:f9:c0:1e:78:5d:d1 -# SHA256 Fingerprint: 88:f4:38:dc:f8:ff:d1:fa:8f:42:91:15:ff:e5:f8:2a:e1:e0:6e:0c:70:c3:75:fa:ad:71:7b:34:a4:9e:72:65 ------BEGIN CERTIFICATE----- -MIIFojCCA4qgAwIBAgIUAZQwHqIL3fXFMyqxQ0Rx+NZQTQ0wDQYJKoZIhvcNAQEM -BQAwaTELMAkGA1UEBhMCS1IxJjAkBgNVBAoMHU5BVkVSIEJVU0lORVNTIFBMQVRG -T1JNIENvcnAuMTIwMAYDVQQDDClOQVZFUiBHbG9iYWwgUm9vdCBDZXJ0aWZpY2F0 -aW9uIEF1dGhvcml0eTAeFw0xNzA4MTgwODU4NDJaFw0zNzA4MTgyMzU5NTlaMGkx -CzAJBgNVBAYTAktSMSYwJAYDVQQKDB1OQVZFUiBCVVNJTkVTUyBQTEFURk9STSBD -b3JwLjEyMDAGA1UEAwwpTkFWRVIgR2xvYmFsIFJvb3QgQ2VydGlmaWNhdGlvbiBB -dXRob3JpdHkwggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQC21PGTXLVA -iQqrDZBbUGOukJR0F0Vy1ntlWilLp1agS7gvQnXp2XskWjFlqxcX0TM62RHcQDaH -38dq6SZeWYp34+hInDEW+j6RscrJo+KfziFTowI2MMtSAuXaMl3Dxeb57hHHi8lE -HoSTGEq0n+USZGnQJoViAbbJAh2+g1G7XNr4rRVqmfeSVPc0W+m/6imBEtRTkZaz -kVrd/pBzKPswRrXKCAfHcXLJZtM0l/aM9BhK4dA9WkW2aacp+yPOiNgSnABIqKYP -szuSjXEOdMWLyEz59JuOuDxp7W87UC9Y7cSw0BwbagzivESq2M0UXZR4Yb8Obtoq -vC8MC3GmsxY/nOb5zJ9TNeIDoKAYv7vxvvTWjIcNQvcGufFt7QSUqP620wbGQGHf -nZ3zVHbOUzoBppJB7ASjjw2i1QnK1sua8e9DXcCrpUHPXFNwcMmIpi3Ua2FzUCaG -YQ5fG8Ir4ozVu53BA0K6lNpfqbDKzE0K70dpAy8i+/Eozr9dUGWokG2zdLAIx6yo -0es+nPxdGoMuK8u180SdOqcXYZaicdNwlhVNt0xz7hlcxVs+Qf6sdWA7G2POAN3a -CJBitOUt7kinaxeZVL6HSuOpXgRM6xBtVNbv8ejyYhbLgGvtPe31HzClrkvJE+2K -AQHJuFFYwGY6sWZLxNUxAmLpdIQM201GLQIDAQABo0IwQDAdBgNVHQ4EFgQU0p+I -36HNLL3s9TsBAZMzJ7LrYEswDgYDVR0PAQH/BAQDAgEGMA8GA1UdEwEB/wQFMAMB -Af8wDQYJKoZIhvcNAQEMBQADggIBADLKgLOdPVQG3dLSLvCkASELZ0jKbY7gyKoN -qo0hV4/GPnrK21HUUrPUloSlWGB/5QuOH/XcChWB5Tu2tyIvCZwTFrFsDDUIbatj -cu3cvuzHV+YwIHHW1xDBE1UBjCpD5EHxzzp6U5LOogMFDTjfArsQLtk70pt6wKGm -+LUx5vR1yblTmXVHIloUFcd4G7ad6Qz4G3bxhYTeodoS76TiEJd6eN4MUZeoIUCL -hr0N8F5OSza7OyAfikJW4Qsav3vQIkMsRIz75Sq0bBwcupTgE34h5prCy8VCZLQe -lHsIJchxzIdFV4XTnyliIoNRlwAYl3dqmJLJfGBs32x9SuRwTMKeuB330DTHD8z7 -p/8Dvq1wkNoL3chtl1+afwkyQf3NosxabUzyqkn+Zvjp2DXrDige7kgvOtB5CTh8 -piKCk5XQA76+AqAF3SAi428diDRgxuYKuQl1C/AH6GmWNcf7I4GOODm4RStDeKLR -LBT/DShycpWbXgnbiUSYqqFJu3FS8r/2/yehNq+4tneI3TqkbZs0kNwUXTC/t+sX -5Ie3cdCh13cV1ELX8vMxmV2b3RZtP+oGI/hGoiLtk/bdmuYqh7GYVPEi92tF4+KO -dh2ajcQGjTa3FPOdVGm3jjzVpG2Tgbet9r1ke8LJaDmgkpzNNIaRkPpkUZ3+/uul -9XXeifdy ------END CERTIFICATE----- - -# Issuer: CN=AC RAIZ FNMT-RCM SERVIDORES SEGUROS O=FNMT-RCM OU=Ceres -# Subject: CN=AC RAIZ FNMT-RCM SERVIDORES SEGUROS O=FNMT-RCM OU=Ceres -# Label: "AC RAIZ FNMT-RCM SERVIDORES SEGUROS" -# Serial: 131542671362353147877283741781055151509 -# MD5 Fingerprint: 19:36:9c:52:03:2f:d2:d1:bb:23:cc:dd:1e:12:55:bb -# SHA1 Fingerprint: 62:ff:d9:9e:c0:65:0d:03:ce:75:93:d2:ed:3f:2d:32:c9:e3:e5:4a -# SHA256 Fingerprint: 55:41:53:b1:3d:2c:f9:dd:b7:53:bf:be:1a:4e:0a:e0:8d:0a:a4:18:70:58:fe:60:a2:b8:62:b2:e4:b8:7b:cb ------BEGIN CERTIFICATE----- -MIICbjCCAfOgAwIBAgIQYvYybOXE42hcG2LdnC6dlTAKBggqhkjOPQQDAzB4MQsw -CQYDVQQGEwJFUzERMA8GA1UECgwIRk5NVC1SQ00xDjAMBgNVBAsMBUNlcmVzMRgw -FgYDVQRhDA9WQVRFUy1RMjgyNjAwNEoxLDAqBgNVBAMMI0FDIFJBSVogRk5NVC1S -Q00gU0VSVklET1JFUyBTRUdVUk9TMB4XDTE4MTIyMDA5MzczM1oXDTQzMTIyMDA5 -MzczM1oweDELMAkGA1UEBhMCRVMxETAPBgNVBAoMCEZOTVQtUkNNMQ4wDAYDVQQL -DAVDZXJlczEYMBYGA1UEYQwPVkFURVMtUTI4MjYwMDRKMSwwKgYDVQQDDCNBQyBS -QUlaIEZOTVQtUkNNIFNFUlZJRE9SRVMgU0VHVVJPUzB2MBAGByqGSM49AgEGBSuB -BAAiA2IABPa6V1PIyqvfNkpSIeSX0oNnnvBlUdBeh8dHsVnyV0ebAAKTRBdp20LH -sbI6GA60XYyzZl2hNPk2LEnb80b8s0RpRBNm/dfF/a82Tc4DTQdxz69qBdKiQ1oK -Um8BA06Oi6NCMEAwDwYDVR0TAQH/BAUwAwEB/zAOBgNVHQ8BAf8EBAMCAQYwHQYD -VR0OBBYEFAG5L++/EYZg8k/QQW6rcx/n0m5JMAoGCCqGSM49BAMDA2kAMGYCMQCu -SuMrQMN0EfKVrRYj3k4MGuZdpSRea0R7/DjiT8ucRRcRTBQnJlU5dUoDzBOQn5IC -MQD6SmxgiHPz7riYYqnOK8LZiqZwMR2vsJRM60/G49HzYqc8/5MuB1xJAWdpEgJy -v+c= ------END CERTIFICATE----- - -# Issuer: CN=GlobalSign Root R46 O=GlobalSign nv-sa -# Subject: CN=GlobalSign Root R46 O=GlobalSign nv-sa -# Label: "GlobalSign Root R46" -# Serial: 1552617688466950547958867513931858518042577 -# MD5 Fingerprint: c4:14:30:e4:fa:66:43:94:2a:6a:1b:24:5f:19:d0:ef -# SHA1 Fingerprint: 53:a2:b0:4b:ca:6b:d6:45:e6:39:8a:8e:c4:0d:d2:bf:77:c3:a2:90 -# SHA256 Fingerprint: 4f:a3:12:6d:8d:3a:11:d1:c4:85:5a:4f:80:7c:ba:d6:cf:91:9d:3a:5a:88:b0:3b:ea:2c:63:72:d9:3c:40:c9 ------BEGIN CERTIFICATE----- -MIIFWjCCA0KgAwIBAgISEdK7udcjGJ5AXwqdLdDfJWfRMA0GCSqGSIb3DQEBDAUA -MEYxCzAJBgNVBAYTAkJFMRkwFwYDVQQKExBHbG9iYWxTaWduIG52LXNhMRwwGgYD -VQQDExNHbG9iYWxTaWduIFJvb3QgUjQ2MB4XDTE5MDMyMDAwMDAwMFoXDTQ2MDMy -MDAwMDAwMFowRjELMAkGA1UEBhMCQkUxGTAXBgNVBAoTEEdsb2JhbFNpZ24gbnYt -c2ExHDAaBgNVBAMTE0dsb2JhbFNpZ24gUm9vdCBSNDYwggIiMA0GCSqGSIb3DQEB -AQUAA4ICDwAwggIKAoICAQCsrHQy6LNl5brtQyYdpokNRbopiLKkHWPd08EsCVeJ -OaFV6Wc0dwxu5FUdUiXSE2te4R2pt32JMl8Nnp8semNgQB+msLZ4j5lUlghYruQG -vGIFAha/r6gjA7aUD7xubMLL1aa7DOn2wQL7Id5m3RerdELv8HQvJfTqa1VbkNud -316HCkD7rRlr+/fKYIje2sGP1q7Vf9Q8g+7XFkyDRTNrJ9CG0Bwta/OrffGFqfUo -0q3v84RLHIf8E6M6cqJaESvWJ3En7YEtbWaBkoe0G1h6zD8K+kZPTXhc+CtI4wSE -y132tGqzZfxCnlEmIyDLPRT5ge1lFgBPGmSXZgjPjHvjK8Cd+RTyG/FWaha/LIWF -zXg4mutCagI0GIMXTpRW+LaCtfOW3T3zvn8gdz57GSNrLNRyc0NXfeD412lPFzYE -+cCQYDdF3uYM2HSNrpyibXRdQr4G9dlkbgIQrImwTDsHTUB+JMWKmIJ5jqSngiCN -I/onccnfxkF0oE32kRbcRoxfKWMxWXEM2G/CtjJ9++ZdU6Z+Ffy7dXxd7Pj2Fxzs -x2sZy/N78CsHpdlseVR2bJ0cpm4O6XkMqCNqo98bMDGfsVR7/mrLZqrcZdCinkqa -ByFrgY/bxFn63iLABJzjqls2k+g9vXqhnQt2sQvHnf3PmKgGwvgqo6GDoLclcqUC -4wIDAQABo0IwQDAOBgNVHQ8BAf8EBAMCAYYwDwYDVR0TAQH/BAUwAwEB/zAdBgNV -HQ4EFgQUA1yrc4GHqMywptWU4jaWSf8FmSwwDQYJKoZIhvcNAQEMBQADggIBAHx4 -7PYCLLtbfpIrXTncvtgdokIzTfnvpCo7RGkerNlFo048p9gkUbJUHJNOxO97k4Vg -JuoJSOD1u8fpaNK7ajFxzHmuEajwmf3lH7wvqMxX63bEIaZHU1VNaL8FpO7XJqti -2kM3S+LGteWygxk6x9PbTZ4IevPuzz5i+6zoYMzRx6Fcg0XERczzF2sUyQQCPtIk -pnnpHs6i58FZFZ8d4kuaPp92CC1r2LpXFNqD6v6MVenQTqnMdzGxRBF6XLE+0xRF -FRhiJBPSy03OXIPBNvIQtQ6IbbjhVp+J3pZmOUdkLG5NrmJ7v2B0GbhWrJKsFjLt -rWhV/pi60zTe9Mlhww6G9kuEYO4Ne7UyWHmRVSyBQ7N0H3qqJZ4d16GLuc1CLgSk -ZoNNiTW2bKg2SnkheCLQQrzRQDGQob4Ez8pn7fXwgNNgyYMqIgXQBztSvwyeqiv5 -u+YfjyW6hY0XHgL+XVAEV8/+LbzvXMAaq7afJMbfc2hIkCwU9D9SGuTSyxTDYWnP -4vkYxboznxSjBF25cfe1lNj2M8FawTSLfJvdkzrnE6JwYZ+vj+vYxXX4M2bUdGc6 -N3ec592kD3ZDZopD8p/7DEJ4Y9HiD2971KE9dJeFt0g5QdYg/NA6s/rob8SKunE3 -vouXsXgxT7PntgMTzlSdriVZzH81Xwj3QEUxeCp6 ------END CERTIFICATE----- - -# Issuer: CN=GlobalSign Root E46 O=GlobalSign nv-sa -# Subject: CN=GlobalSign Root E46 O=GlobalSign nv-sa -# Label: "GlobalSign Root E46" -# Serial: 1552617690338932563915843282459653771421763 -# MD5 Fingerprint: b5:b8:66:ed:de:08:83:e3:c9:e2:01:34:06:ac:51:6f -# SHA1 Fingerprint: 39:b4:6c:d5:fe:80:06:eb:e2:2f:4a:bb:08:33:a0:af:db:b9:dd:84 -# SHA256 Fingerprint: cb:b9:c4:4d:84:b8:04:3e:10:50:ea:31:a6:9f:51:49:55:d7:bf:d2:e2:c6:b4:93:01:01:9a:d6:1d:9f:50:58 ------BEGIN CERTIFICATE----- -MIICCzCCAZGgAwIBAgISEdK7ujNu1LzmJGjFDYQdmOhDMAoGCCqGSM49BAMDMEYx -CzAJBgNVBAYTAkJFMRkwFwYDVQQKExBHbG9iYWxTaWduIG52LXNhMRwwGgYDVQQD -ExNHbG9iYWxTaWduIFJvb3QgRTQ2MB4XDTE5MDMyMDAwMDAwMFoXDTQ2MDMyMDAw -MDAwMFowRjELMAkGA1UEBhMCQkUxGTAXBgNVBAoTEEdsb2JhbFNpZ24gbnYtc2Ex -HDAaBgNVBAMTE0dsb2JhbFNpZ24gUm9vdCBFNDYwdjAQBgcqhkjOPQIBBgUrgQQA -IgNiAAScDrHPt+ieUnd1NPqlRqetMhkytAepJ8qUuwzSChDH2omwlwxwEwkBjtjq -R+q+soArzfwoDdusvKSGN+1wCAB16pMLey5SnCNoIwZD7JIvU4Tb+0cUB+hflGdd -yXqBPCCjQjBAMA4GA1UdDwEB/wQEAwIBhjAPBgNVHRMBAf8EBTADAQH/MB0GA1Ud -DgQWBBQxCpCPtsad0kRLgLWi5h+xEk8blTAKBggqhkjOPQQDAwNoADBlAjEA31SQ -7Zvvi5QCkxeCmb6zniz2C5GMn0oUsfZkvLtoURMMA/cVi4RguYv/Uo7njLwcAjA8 -+RHUjE7AwWHCFUyqqx0LMV87HOIAl0Qx5v5zli/altP+CAezNIm8BZ/3Hobui3A= ------END CERTIFICATE----- - -# Issuer: CN=GLOBALTRUST 2020 O=e-commerce monitoring GmbH -# Subject: CN=GLOBALTRUST 2020 O=e-commerce monitoring GmbH -# Label: "GLOBALTRUST 2020" -# Serial: 109160994242082918454945253 -# MD5 Fingerprint: 8a:c7:6f:cb:6d:e3:cc:a2:f1:7c:83:fa:0e:78:d7:e8 -# SHA1 Fingerprint: d0:67:c1:13:51:01:0c:aa:d0:c7:6a:65:37:31:16:26:4f:53:71:a2 -# SHA256 Fingerprint: 9a:29:6a:51:82:d1:d4:51:a2:e3:7f:43:9b:74:da:af:a2:67:52:33:29:f9:0f:9a:0d:20:07:c3:34:e2:3c:9a ------BEGIN CERTIFICATE----- -MIIFgjCCA2qgAwIBAgILWku9WvtPilv6ZeUwDQYJKoZIhvcNAQELBQAwTTELMAkG -A1UEBhMCQVQxIzAhBgNVBAoTGmUtY29tbWVyY2UgbW9uaXRvcmluZyBHbWJIMRkw -FwYDVQQDExBHTE9CQUxUUlVTVCAyMDIwMB4XDTIwMDIxMDAwMDAwMFoXDTQwMDYx -MDAwMDAwMFowTTELMAkGA1UEBhMCQVQxIzAhBgNVBAoTGmUtY29tbWVyY2UgbW9u -aXRvcmluZyBHbWJIMRkwFwYDVQQDExBHTE9CQUxUUlVTVCAyMDIwMIICIjANBgkq -hkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAri5WrRsc7/aVj6B3GyvTY4+ETUWiD59b -RatZe1E0+eyLinjF3WuvvcTfk0Uev5E4C64OFudBc/jbu9G4UeDLgztzOG53ig9Z -YybNpyrOVPu44sB8R85gfD+yc/LAGbaKkoc1DZAoouQVBGM+uq/ufF7MpotQsjj3 -QWPKzv9pj2gOlTblzLmMCcpL3TGQlsjMH/1WljTbjhzqLL6FLmPdqqmV0/0plRPw -yJiT2S0WR5ARg6I6IqIoV6Lr/sCMKKCmfecqQjuCgGOlYx8ZzHyyZqjC0203b+J+ -BlHZRYQfEs4kUmSFC0iAToexIiIwquuuvuAC4EDosEKAA1GqtH6qRNdDYfOiaxaJ -SaSjpCuKAsR49GiKweR6NrFvG5Ybd0mN1MkGco/PU+PcF4UgStyYJ9ORJitHHmkH -r96i5OTUawuzXnzUJIBHKWk7buis/UDr2O1xcSvy6Fgd60GXIsUf1DnQJ4+H4xj0 -4KlGDfV0OoIu0G4skaMxXDtG6nsEEFZegB31pWXogvziB4xiRfUg3kZwhqG8k9Me -dKZssCz3AwyIDMvUclOGvGBG85hqwvG/Q/lwIHfKN0F5VVJjjVsSn8VoxIidrPIw -q7ejMZdnrY8XD2zHc+0klGvIg5rQmjdJBKuxFshsSUktq6HQjJLyQUp5ISXbY9e2 -nKd+Qmn7OmMCAwEAAaNjMGEwDwYDVR0TAQH/BAUwAwEB/zAOBgNVHQ8BAf8EBAMC -AQYwHQYDVR0OBBYEFNwuH9FhN3nkq9XVsxJxaD1qaJwiMB8GA1UdIwQYMBaAFNwu -H9FhN3nkq9XVsxJxaD1qaJwiMA0GCSqGSIb3DQEBCwUAA4ICAQCR8EICaEDuw2jA -VC/f7GLDw56KoDEoqoOOpFaWEhCGVrqXctJUMHytGdUdaG/7FELYjQ7ztdGl4wJC -XtzoRlgHNQIw4Lx0SsFDKv/bGtCwr2zD/cuz9X9tAy5ZVp0tLTWMstZDFyySCstd -6IwPS3BD0IL/qMy/pJTAvoe9iuOTe8aPmxadJ2W8esVCgmxcB9CpwYhgROmYhRZf -+I/KARDOJcP5YBugxZfD0yyIMaK9MOzQ0MAS8cE54+X1+NZK3TTN+2/BT+MAi1bi -kvcoskJ3ciNnxz8RFbLEAwW+uxF7Cr+obuf/WEPPm2eggAe2HcqtbepBEX4tdJP7 -wry+UUTF72glJ4DjyKDUEuzZpTcdN3y0kcra1LGWge9oXHYQSa9+pTeAsRxSvTOB -TI/53WXZFM2KJVj04sWDpQmQ1GwUY7VA3+vA/MRYfg0UFodUJ25W5HCEuGwyEn6C -MUO+1918oa2u1qsgEu8KwxCMSZY13At1XrFP1U80DhEgB3VDRemjEdqso5nCtnkn -4rnvyOL2NSl6dPrFf4IFYqYK6miyeUcGbvJXqBUzxvd4Sj1Ce2t+/vdG6tHrju+I -aFvowdlxfv1k7/9nR4hYJS8+hge9+6jlgqispdNpQ80xiEmEU5LAsTkbOYMBMMTy -qfrQA71yN2BWHzZ8vTmR9W0Nv3vXkg== ------END CERTIFICATE----- - -# Issuer: CN=ANF Secure Server Root CA O=ANF Autoridad de Certificacion OU=ANF CA Raiz -# Subject: CN=ANF Secure Server Root CA O=ANF Autoridad de Certificacion OU=ANF CA Raiz -# Label: "ANF Secure Server Root CA" -# Serial: 996390341000653745 -# MD5 Fingerprint: 26:a6:44:5a:d9:af:4e:2f:b2:1d:b6:65:b0:4e:e8:96 -# SHA1 Fingerprint: 5b:6e:68:d0:cc:15:b6:a0:5f:1e:c1:5f:ae:02:fc:6b:2f:5d:6f:74 -# SHA256 Fingerprint: fb:8f:ec:75:91:69:b9:10:6b:1e:51:16:44:c6:18:c5:13:04:37:3f:6c:06:43:08:8d:8b:ef:fd:1b:99:75:99 ------BEGIN CERTIFICATE----- -MIIF7zCCA9egAwIBAgIIDdPjvGz5a7EwDQYJKoZIhvcNAQELBQAwgYQxEjAQBgNV -BAUTCUc2MzI4NzUxMDELMAkGA1UEBhMCRVMxJzAlBgNVBAoTHkFORiBBdXRvcmlk -YWQgZGUgQ2VydGlmaWNhY2lvbjEUMBIGA1UECxMLQU5GIENBIFJhaXoxIjAgBgNV -BAMTGUFORiBTZWN1cmUgU2VydmVyIFJvb3QgQ0EwHhcNMTkwOTA0MTAwMDM4WhcN -MzkwODMwMTAwMDM4WjCBhDESMBAGA1UEBRMJRzYzMjg3NTEwMQswCQYDVQQGEwJF -UzEnMCUGA1UEChMeQU5GIEF1dG9yaWRhZCBkZSBDZXJ0aWZpY2FjaW9uMRQwEgYD -VQQLEwtBTkYgQ0EgUmFpejEiMCAGA1UEAxMZQU5GIFNlY3VyZSBTZXJ2ZXIgUm9v -dCBDQTCCAiIwDQYJKoZIhvcNAQEBBQADggIPADCCAgoCggIBANvrayvmZFSVgpCj -cqQZAZ2cC4Ffc0m6p6zzBE57lgvsEeBbphzOG9INgxwruJ4dfkUyYA8H6XdYfp9q -yGFOtibBTI3/TO80sh9l2Ll49a2pcbnvT1gdpd50IJeh7WhM3pIXS7yr/2WanvtH -2Vdy8wmhrnZEE26cLUQ5vPnHO6RYPUG9tMJJo8gN0pcvB2VSAKduyK9o7PQUlrZX -H1bDOZ8rbeTzPvY1ZNoMHKGESy9LS+IsJJ1tk0DrtSOOMspvRdOoiXsezx76W0OL -zc2oD2rKDF65nkeP8Nm2CgtYZRczuSPkdxl9y0oukntPLxB3sY0vaJxizOBQ+OyR -p1RMVwnVdmPF6GUe7m1qzwmd+nxPrWAI/VaZDxUse6mAq4xhj0oHdkLePfTdsiQz -W7i1o0TJrH93PB0j7IKppuLIBkwC/qxcmZkLLxCKpvR/1Yd0DVlJRfbwcVw5Kda/ -SiOL9V8BY9KHcyi1Swr1+KuCLH5zJTIdC2MKF4EA/7Z2Xue0sUDKIbvVgFHlSFJn -LNJhiQcND85Cd8BEc5xEUKDbEAotlRyBr+Qc5RQe8TZBAQIvfXOn3kLMTOmJDVb3 -n5HUA8ZsyY/b2BzgQJhdZpmYgG4t/wHFzstGH6wCxkPmrqKEPMVOHj1tyRRM4y5B -u8o5vzY8KhmqQYdOpc5LMnndkEl/AgMBAAGjYzBhMB8GA1UdIwQYMBaAFJxf0Gxj -o1+TypOYCK2Mh6UsXME3MB0GA1UdDgQWBBScX9BsY6Nfk8qTmAitjIelLFzBNzAO -BgNVHQ8BAf8EBAMCAYYwDwYDVR0TAQH/BAUwAwEB/zANBgkqhkiG9w0BAQsFAAOC -AgEATh65isagmD9uw2nAalxJUqzLK114OMHVVISfk/CHGT0sZonrDUL8zPB1hT+L -9IBdeeUXZ701guLyPI59WzbLWoAAKfLOKyzxj6ptBZNscsdW699QIyjlRRA96Gej -rw5VD5AJYu9LWaL2U/HANeQvwSS9eS9OICI7/RogsKQOLHDtdD+4E5UGUcjohybK -pFtqFiGS3XNgnhAY3jyB6ugYw3yJ8otQPr0R4hUDqDZ9MwFsSBXXiJCZBMXM5gf0 -vPSQ7RPi6ovDj6MzD8EpTBNO2hVWcXNyglD2mjN8orGoGjR0ZVzO0eurU+AagNjq -OknkJjCb5RyKqKkVMoaZkgoQI1YS4PbOTOK7vtuNknMBZi9iPrJyJ0U27U1W45eZ -/zo1PqVUSlJZS2Db7v54EX9K3BR5YLZrZAPbFYPhor72I5dQ8AkzNqdxliXzuUJ9 -2zg/LFis6ELhDtjTO0wugumDLmsx2d1Hhk9tl5EuT+IocTUW0fJz/iUrB0ckYyfI -+PbZa/wSMVYIwFNCr5zQM378BvAxRAMU8Vjq8moNqRGyg77FGr8H6lnco4g175x2 -MjxNBiLOFeXdntiP2t7SxDnlF4HPOEfrf4htWRvfn0IUrn7PqLBmZdo3r5+qPeoo -tt7VMVgWglvquxl1AnMaykgaIZOQCo6ThKd9OyMYkomgjaw= ------END CERTIFICATE----- - -# Issuer: CN=Certum EC-384 CA O=Asseco Data Systems S.A. OU=Certum Certification Authority -# Subject: CN=Certum EC-384 CA O=Asseco Data Systems S.A. OU=Certum Certification Authority -# Label: "Certum EC-384 CA" -# Serial: 160250656287871593594747141429395092468 -# MD5 Fingerprint: b6:65:b3:96:60:97:12:a1:ec:4e:e1:3d:a3:c6:c9:f1 -# SHA1 Fingerprint: f3:3e:78:3c:ac:df:f4:a2:cc:ac:67:55:69:56:d7:e5:16:3c:e1:ed -# SHA256 Fingerprint: 6b:32:80:85:62:53:18:aa:50:d1:73:c9:8d:8b:da:09:d5:7e:27:41:3d:11:4c:f7:87:a0:f5:d0:6c:03:0c:f6 ------BEGIN CERTIFICATE----- -MIICZTCCAeugAwIBAgIQeI8nXIESUiClBNAt3bpz9DAKBggqhkjOPQQDAzB0MQsw -CQYDVQQGEwJQTDEhMB8GA1UEChMYQXNzZWNvIERhdGEgU3lzdGVtcyBTLkEuMScw -JQYDVQQLEx5DZXJ0dW0gQ2VydGlmaWNhdGlvbiBBdXRob3JpdHkxGTAXBgNVBAMT -EENlcnR1bSBFQy0zODQgQ0EwHhcNMTgwMzI2MDcyNDU0WhcNNDMwMzI2MDcyNDU0 -WjB0MQswCQYDVQQGEwJQTDEhMB8GA1UEChMYQXNzZWNvIERhdGEgU3lzdGVtcyBT -LkEuMScwJQYDVQQLEx5DZXJ0dW0gQ2VydGlmaWNhdGlvbiBBdXRob3JpdHkxGTAX -BgNVBAMTEENlcnR1bSBFQy0zODQgQ0EwdjAQBgcqhkjOPQIBBgUrgQQAIgNiAATE -KI6rGFtqvm5kN2PkzeyrOvfMobgOgknXhimfoZTy42B4mIF4Bk3y7JoOV2CDn7Tm -Fy8as10CW4kjPMIRBSqniBMY81CE1700LCeJVf/OTOffph8oxPBUw7l8t1Ot68Kj -QjBAMA8GA1UdEwEB/wQFMAMBAf8wHQYDVR0OBBYEFI0GZnQkdjrzife81r1HfS+8 -EF9LMA4GA1UdDwEB/wQEAwIBBjAKBggqhkjOPQQDAwNoADBlAjADVS2m5hjEfO/J -UG7BJw+ch69u1RsIGL2SKcHvlJF40jocVYli5RsJHrpka/F2tNQCMQC0QoSZ/6vn -nvuRlydd3LBbMHHOXjgaatkl5+r3YZJW+OraNsKHZZYuciUvf9/DE8k= ------END CERTIFICATE----- - -# Issuer: CN=Certum Trusted Root CA O=Asseco Data Systems S.A. OU=Certum Certification Authority -# Subject: CN=Certum Trusted Root CA O=Asseco Data Systems S.A. OU=Certum Certification Authority -# Label: "Certum Trusted Root CA" -# Serial: 40870380103424195783807378461123655149 -# MD5 Fingerprint: 51:e1:c2:e7:fe:4c:84:af:59:0e:2f:f4:54:6f:ea:29 -# SHA1 Fingerprint: c8:83:44:c0:18:ae:9f:cc:f1:87:b7:8f:22:d1:c5:d7:45:84:ba:e5 -# SHA256 Fingerprint: fe:76:96:57:38:55:77:3e:37:a9:5e:7a:d4:d9:cc:96:c3:01:57:c1:5d:31:76:5b:a9:b1:57:04:e1:ae:78:fd ------BEGIN CERTIFICATE----- -MIIFwDCCA6igAwIBAgIQHr9ZULjJgDdMBvfrVU+17TANBgkqhkiG9w0BAQ0FADB6 -MQswCQYDVQQGEwJQTDEhMB8GA1UEChMYQXNzZWNvIERhdGEgU3lzdGVtcyBTLkEu -MScwJQYDVQQLEx5DZXJ0dW0gQ2VydGlmaWNhdGlvbiBBdXRob3JpdHkxHzAdBgNV -BAMTFkNlcnR1bSBUcnVzdGVkIFJvb3QgQ0EwHhcNMTgwMzE2MTIxMDEzWhcNNDMw -MzE2MTIxMDEzWjB6MQswCQYDVQQGEwJQTDEhMB8GA1UEChMYQXNzZWNvIERhdGEg -U3lzdGVtcyBTLkEuMScwJQYDVQQLEx5DZXJ0dW0gQ2VydGlmaWNhdGlvbiBBdXRo -b3JpdHkxHzAdBgNVBAMTFkNlcnR1bSBUcnVzdGVkIFJvb3QgQ0EwggIiMA0GCSqG -SIb3DQEBAQUAA4ICDwAwggIKAoICAQDRLY67tzbqbTeRn06TpwXkKQMlzhyC93yZ -n0EGze2jusDbCSzBfN8pfktlL5On1AFrAygYo9idBcEq2EXxkd7fO9CAAozPOA/q -p1x4EaTByIVcJdPTsuclzxFUl6s1wB52HO8AU5853BSlLCIls3Jy/I2z5T4IHhQq -NwuIPMqw9MjCoa68wb4pZ1Xi/K1ZXP69VyywkI3C7Te2fJmItdUDmj0VDT06qKhF -8JVOJVkdzZhpu9PMMsmN74H+rX2Ju7pgE8pllWeg8xn2A1bUatMn4qGtg/BKEiJ3 -HAVz4hlxQsDsdUaakFjgao4rpUYwBI4Zshfjvqm6f1bxJAPXsiEodg42MEx51UGa -mqi4NboMOvJEGyCI98Ul1z3G4z5D3Yf+xOr1Uz5MZf87Sst4WmsXXw3Hw09Omiqi -7VdNIuJGmj8PkTQkfVXjjJU30xrwCSss0smNtA0Aq2cpKNgB9RkEth2+dv5yXMSF -ytKAQd8FqKPVhJBPC/PgP5sZ0jeJP/J7UhyM9uH3PAeXjA6iWYEMspA90+NZRu0P -qafegGtaqge2Gcu8V/OXIXoMsSt0Puvap2ctTMSYnjYJdmZm/Bo/6khUHL4wvYBQ -v3y1zgD2DGHZ5yQD4OMBgQ692IU0iL2yNqh7XAjlRICMb/gv1SHKHRzQ+8S1h9E6 -Tsd2tTVItQIDAQABo0IwQDAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBSM+xx1 -vALTn04uSNn5YFSqxLNP+jAOBgNVHQ8BAf8EBAMCAQYwDQYJKoZIhvcNAQENBQAD -ggIBAEii1QALLtA/vBzVtVRJHlpr9OTy4EA34MwUe7nJ+jW1dReTagVphZzNTxl4 -WxmB82M+w85bj/UvXgF2Ez8sALnNllI5SW0ETsXpD4YN4fqzX4IS8TrOZgYkNCvo -zMrnadyHncI013nR03e4qllY/p0m+jiGPp2Kh2RX5Rc64vmNueMzeMGQ2Ljdt4NR -5MTMI9UGfOZR0800McD2RrsLrfw9EAUqO0qRJe6M1ISHgCq8CYyqOhNf6DR5UMEQ -GfnTKB7U0VEwKbOukGfWHwpjscWpxkIxYxeU72nLL/qMFH3EQxiJ2fAyQOaA4kZf -5ePBAFmo+eggvIksDkc0C+pXwlM2/KfUrzHN/gLldfq5Jwn58/U7yn2fqSLLiMmq -0Uc9NneoWWRrJ8/vJ8HjJLWG965+Mk2weWjROeiQWMODvA8s1pfrzgzhIMfatz7D -P78v3DSk+yshzWePS/Tj6tQ/50+6uaWTRRxmHyH6ZF5v4HaUMst19W7l9o/HuKTM -qJZ9ZPskWkoDbGs4xugDQ5r3V7mzKWmTOPQD8rv7gmsHINFSH5pkAnuYZttcTVoP -0ISVoDwUQwbKytu4QTbaakRnh6+v40URFWkIsr4WOZckbxJF0WddCajJFdr60qZf -E2Efv4WstK2tBZQIgx51F9NxO5NQI1mg7TyRVJ12AMXDuDjb ------END CERTIFICATE----- - -# Issuer: CN=TunTrust Root CA O=Agence Nationale de Certification Electronique -# Subject: CN=TunTrust Root CA O=Agence Nationale de Certification Electronique -# Label: "TunTrust Root CA" -# Serial: 108534058042236574382096126452369648152337120275 -# MD5 Fingerprint: 85:13:b9:90:5b:36:5c:b6:5e:b8:5a:f8:e0:31:57:b4 -# SHA1 Fingerprint: cf:e9:70:84:0f:e0:73:0f:9d:f6:0c:7f:2c:4b:ee:20:46:34:9c:bb -# SHA256 Fingerprint: 2e:44:10:2a:b5:8c:b8:54:19:45:1c:8e:19:d9:ac:f3:66:2c:af:bc:61:4b:6a:53:96:0a:30:f7:d0:e2:eb:41 ------BEGIN CERTIFICATE----- -MIIFszCCA5ugAwIBAgIUEwLV4kBMkkaGFmddtLu7sms+/BMwDQYJKoZIhvcNAQEL -BQAwYTELMAkGA1UEBhMCVE4xNzA1BgNVBAoMLkFnZW5jZSBOYXRpb25hbGUgZGUg -Q2VydGlmaWNhdGlvbiBFbGVjdHJvbmlxdWUxGTAXBgNVBAMMEFR1blRydXN0IFJv -b3QgQ0EwHhcNMTkwNDI2MDg1NzU2WhcNNDQwNDI2MDg1NzU2WjBhMQswCQYDVQQG -EwJUTjE3MDUGA1UECgwuQWdlbmNlIE5hdGlvbmFsZSBkZSBDZXJ0aWZpY2F0aW9u -IEVsZWN0cm9uaXF1ZTEZMBcGA1UEAwwQVHVuVHJ1c3QgUm9vdCBDQTCCAiIwDQYJ -KoZIhvcNAQEBBQADggIPADCCAgoCggIBAMPN0/y9BFPdDCA61YguBUtB9YOCfvdZ -n56eY+hz2vYGqU8ftPkLHzmMmiDQfgbU7DTZhrx1W4eI8NLZ1KMKsmwb60ksPqxd -2JQDoOw05TDENX37Jk0bbjBU2PWARZw5rZzJJQRNmpA+TkBuimvNKWfGzC3gdOgF -VwpIUPp6Q9p+7FuaDmJ2/uqdHYVy7BG7NegfJ7/Boce7SBbdVtfMTqDhuazb1YMZ -GoXRlJfXyqNlC/M4+QKu3fZnz8k/9YosRxqZbwUN/dAdgjH8KcwAWJeRTIAAHDOF -li/LQcKLEITDCSSJH7UP2dl3RxiSlGBcx5kDPP73lad9UKGAwqmDrViWVSHbhlnU -r8a83YFuB9tgYv7sEG7aaAH0gxupPqJbI9dkxt/con3YS7qC0lH4Zr8GRuR5KiY2 -eY8fTpkdso8MDhz/yV3A/ZAQprE38806JG60hZC/gLkMjNWb1sjxVj8agIl6qeIb -MlEsPvLfe/ZdeikZjuXIvTZxi11Mwh0/rViizz1wTaZQmCXcI/m4WEEIcb9PuISg -jwBUFfyRbVinljvrS5YnzWuioYasDXxU5mZMZl+QviGaAkYt5IPCgLnPSz7ofzwB -7I9ezX/SKEIBlYrilz0QIX32nRzFNKHsLA4KUiwSVXAkPcvCFDVDXSdOvsC9qnyW -5/yeYa1E0wCXAgMBAAGjYzBhMB0GA1UdDgQWBBQGmpsfU33x9aTI04Y+oXNZtPdE -ITAPBgNVHRMBAf8EBTADAQH/MB8GA1UdIwQYMBaAFAaamx9TffH1pMjThj6hc1m0 -90QhMA4GA1UdDwEB/wQEAwIBBjANBgkqhkiG9w0BAQsFAAOCAgEAqgVutt0Vyb+z -xiD2BkewhpMl0425yAA/l/VSJ4hxyXT968pk21vvHl26v9Hr7lxpuhbI87mP0zYu -QEkHDVneixCwSQXi/5E/S7fdAo74gShczNxtr18UnH1YeA32gAm56Q6XKRm4t+v4 -FstVEuTGfbvE7Pi1HE4+Z7/FXxttbUcoqgRYYdZ2vyJ/0Adqp2RT8JeNnYA/u8EH -22Wv5psymsNUk8QcCMNE+3tjEUPRahphanltkE8pjkcFwRJpadbGNjHh/PqAulxP -xOu3Mqz4dWEX1xAZufHSCe96Qp1bWgvUxpVOKs7/B9dPfhgGiPEZtdmYu65xxBzn -dFlY7wyJz4sfdZMaBBSSSFCp61cpABbjNhzI+L/wM9VBD8TMPN3pM0MBkRArHtG5 -Xc0yGYuPjCB31yLEQtyEFpslbei0VXF/sHyz03FJuc9SpAQ/3D2gu68zngowYI7b -nV2UqL1g52KAdoGDDIzMMEZJ4gzSqK/rYXHv5yJiqfdcZGyfFoxnNidF9Ql7v/YQ -CvGwjVRDjAS6oz/v4jXH+XTgbzRB0L9zZVcg+ZtnemZoJE6AZb0QmQZZ8mWvuMZH -u/2QeItBcy6vVR/cO5JyboTT0GFMDcx2V+IthSIVNg3rAZ3r2OvEhJn7wAzMMujj -d9qDRIueVSjAi1jTkD5OGwDxFa2DK5o= ------END CERTIFICATE----- - -# Issuer: CN=HARICA TLS RSA Root CA 2021 O=Hellenic Academic and Research Institutions CA -# Subject: CN=HARICA TLS RSA Root CA 2021 O=Hellenic Academic and Research Institutions CA -# Label: "HARICA TLS RSA Root CA 2021" -# Serial: 76817823531813593706434026085292783742 -# MD5 Fingerprint: 65:47:9b:58:86:dd:2c:f0:fc:a2:84:1f:1e:96:c4:91 -# SHA1 Fingerprint: 02:2d:05:82:fa:88:ce:14:0c:06:79:de:7f:14:10:e9:45:d7:a5:6d -# SHA256 Fingerprint: d9:5d:0e:8e:da:79:52:5b:f9:be:b1:1b:14:d2:10:0d:32:94:98:5f:0c:62:d9:fa:bd:9c:d9:99:ec:cb:7b:1d ------BEGIN CERTIFICATE----- -MIIFpDCCA4ygAwIBAgIQOcqTHO9D88aOk8f0ZIk4fjANBgkqhkiG9w0BAQsFADBs -MQswCQYDVQQGEwJHUjE3MDUGA1UECgwuSGVsbGVuaWMgQWNhZGVtaWMgYW5kIFJl -c2VhcmNoIEluc3RpdHV0aW9ucyBDQTEkMCIGA1UEAwwbSEFSSUNBIFRMUyBSU0Eg -Um9vdCBDQSAyMDIxMB4XDTIxMDIxOTEwNTUzOFoXDTQ1MDIxMzEwNTUzN1owbDEL -MAkGA1UEBhMCR1IxNzA1BgNVBAoMLkhlbGxlbmljIEFjYWRlbWljIGFuZCBSZXNl -YXJjaCBJbnN0aXR1dGlvbnMgQ0ExJDAiBgNVBAMMG0hBUklDQSBUTFMgUlNBIFJv -b3QgQ0EgMjAyMTCCAiIwDQYJKoZIhvcNAQEBBQADggIPADCCAgoCggIBAIvC569l -mwVnlskNJLnQDmT8zuIkGCyEf3dRywQRNrhe7Wlxp57kJQmXZ8FHws+RFjZiPTgE -4VGC/6zStGndLuwRo0Xua2s7TL+MjaQenRG56Tj5eg4MmOIjHdFOY9TnuEFE+2uv -a9of08WRiFukiZLRgeaMOVig1mlDqa2YUlhu2wr7a89o+uOkXjpFc5gH6l8Cct4M -pbOfrqkdtx2z/IpZ525yZa31MJQjB/OCFks1mJxTuy/K5FrZx40d/JiZ+yykgmvw -Kh+OC19xXFyuQnspiYHLA6OZyoieC0AJQTPb5lh6/a6ZcMBaD9YThnEvdmn8kN3b -LW7R8pv1GmuebxWMevBLKKAiOIAkbDakO/IwkfN4E8/BPzWr8R0RI7VDIp4BkrcY -AuUR0YLbFQDMYTfBKnya4dC6s1BG7oKsnTH4+yPiAwBIcKMJJnkVU2DzOFytOOqB -AGMUuTNe3QvboEUHGjMJ+E20pwKmafTCWQWIZYVWrkvL4N48fS0ayOn7H6NhStYq -E613TBoYm5EPWNgGVMWX+Ko/IIqmhaZ39qb8HOLubpQzKoNQhArlT4b4UEV4AIHr -W2jjJo3Me1xR9BQsQL4aYB16cmEdH2MtiKrOokWQCPxrvrNQKlr9qEgYRtaQQJKQ -CoReaDH46+0N0x3GfZkYVVYnZS6NRcUk7M7jAgMBAAGjQjBAMA8GA1UdEwEB/wQF -MAMBAf8wHQYDVR0OBBYEFApII6ZgpJIKM+qTW8VX6iVNvRLuMA4GA1UdDwEB/wQE -AwIBhjANBgkqhkiG9w0BAQsFAAOCAgEAPpBIqm5iFSVmewzVjIuJndftTgfvnNAU -X15QvWiWkKQUEapobQk1OUAJ2vQJLDSle1mESSmXdMgHHkdt8s4cUCbjnj1AUz/3 -f5Z2EMVGpdAgS1D0NTsY9FVqQRtHBmg8uwkIYtlfVUKqrFOFrJVWNlar5AWMxaja -H6NpvVMPxP/cyuN+8kyIhkdGGvMA9YCRotxDQpSbIPDRzbLrLFPCU3hKTwSUQZqP -JzLB5UkZv/HywouoCjkxKLR9YjYsTewfM7Z+d21+UPCfDtcRj88YxeMn/ibvBZ3P -zzfF0HvaO7AWhAw6k9a+F9sPPg4ZeAnHqQJyIkv3N3a6dcSFA1pj1bF1BcK5vZSt -jBWZp5N99sXzqnTPBIWUmAD04vnKJGW/4GKvyMX6ssmeVkjaef2WdhW+o45WxLM0 -/L5H9MG0qPzVMIho7suuyWPEdr6sOBjhXlzPrjoiUevRi7PzKzMHVIf6tLITe7pT -BGIBnfHAT+7hOtSLIBD6Alfm78ELt5BGnBkpjNxvoEppaZS3JGWg/6w/zgH7IS79 -aPib8qXPMThcFarmlwDB31qlpzmq6YR/PFGoOtmUW4y/Twhx5duoXNTSpv4Ao8YW -xw/ogM4cKGR0GQjTQuPOAF1/sdwTsOEFy9EgqoZ0njnnkf3/W9b3raYvAwtt41dU -63ZTGI0RmLo= ------END CERTIFICATE----- - -# Issuer: CN=HARICA TLS ECC Root CA 2021 O=Hellenic Academic and Research Institutions CA -# Subject: CN=HARICA TLS ECC Root CA 2021 O=Hellenic Academic and Research Institutions CA -# Label: "HARICA TLS ECC Root CA 2021" -# Serial: 137515985548005187474074462014555733966 -# MD5 Fingerprint: ae:f7:4c:e5:66:35:d1:b7:9b:8c:22:93:74:d3:4b:b0 -# SHA1 Fingerprint: bc:b0:c1:9d:e9:98:92:70:19:38:57:e9:8d:a7:b4:5d:6e:ee:01:48 -# SHA256 Fingerprint: 3f:99:cc:47:4a:cf:ce:4d:fe:d5:87:94:66:5e:47:8d:15:47:73:9f:2e:78:0f:1b:b4:ca:9b:13:30:97:d4:01 ------BEGIN CERTIFICATE----- -MIICVDCCAdugAwIBAgIQZ3SdjXfYO2rbIvT/WeK/zjAKBggqhkjOPQQDAzBsMQsw -CQYDVQQGEwJHUjE3MDUGA1UECgwuSGVsbGVuaWMgQWNhZGVtaWMgYW5kIFJlc2Vh -cmNoIEluc3RpdHV0aW9ucyBDQTEkMCIGA1UEAwwbSEFSSUNBIFRMUyBFQ0MgUm9v -dCBDQSAyMDIxMB4XDTIxMDIxOTExMDExMFoXDTQ1MDIxMzExMDEwOVowbDELMAkG -A1UEBhMCR1IxNzA1BgNVBAoMLkhlbGxlbmljIEFjYWRlbWljIGFuZCBSZXNlYXJj -aCBJbnN0aXR1dGlvbnMgQ0ExJDAiBgNVBAMMG0hBUklDQSBUTFMgRUNDIFJvb3Qg -Q0EgMjAyMTB2MBAGByqGSM49AgEGBSuBBAAiA2IABDgI/rGgltJ6rK9JOtDA4MM7 -KKrxcm1lAEeIhPyaJmuqS7psBAqIXhfyVYf8MLA04jRYVxqEU+kw2anylnTDUR9Y -STHMmE5gEYd103KUkE+bECUqqHgtvpBBWJAVcqeht6NCMEAwDwYDVR0TAQH/BAUw -AwEB/zAdBgNVHQ4EFgQUyRtTgRL+BNUW0aq8mm+3oJUZbsowDgYDVR0PAQH/BAQD -AgGGMAoGCCqGSM49BAMDA2cAMGQCMBHervjcToiwqfAircJRQO9gcS3ujwLEXQNw -SaSS6sUUiHCm0w2wqsosQJz76YJumgIwK0eaB8bRwoF8yguWGEEbo/QwCZ61IygN -nxS2PFOiTAZpffpskcYqSUXm7LcT4Tps ------END CERTIFICATE----- - -# Issuer: CN=Autoridad de Certificacion Firmaprofesional CIF A62634068 -# Subject: CN=Autoridad de Certificacion Firmaprofesional CIF A62634068 -# Label: "Autoridad de Certificacion Firmaprofesional CIF A62634068" -# Serial: 1977337328857672817 -# MD5 Fingerprint: 4e:6e:9b:54:4c:ca:b7:fa:48:e4:90:b1:15:4b:1c:a3 -# SHA1 Fingerprint: 0b:be:c2:27:22:49:cb:39:aa:db:35:5c:53:e3:8c:ae:78:ff:b6:fe -# SHA256 Fingerprint: 57:de:05:83:ef:d2:b2:6e:03:61:da:99:da:9d:f4:64:8d:ef:7e:e8:44:1c:3b:72:8a:fa:9b:cd:e0:f9:b2:6a ------BEGIN CERTIFICATE----- -MIIGFDCCA/ygAwIBAgIIG3Dp0v+ubHEwDQYJKoZIhvcNAQELBQAwUTELMAkGA1UE -BhMCRVMxQjBABgNVBAMMOUF1dG9yaWRhZCBkZSBDZXJ0aWZpY2FjaW9uIEZpcm1h -cHJvZmVzaW9uYWwgQ0lGIEE2MjYzNDA2ODAeFw0xNDA5MjMxNTIyMDdaFw0zNjA1 -MDUxNTIyMDdaMFExCzAJBgNVBAYTAkVTMUIwQAYDVQQDDDlBdXRvcmlkYWQgZGUg -Q2VydGlmaWNhY2lvbiBGaXJtYXByb2Zlc2lvbmFsIENJRiBBNjI2MzQwNjgwggIi -MA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQDKlmuO6vj78aI14H9M2uDDUtd9 -thDIAl6zQyrET2qyyhxdKJp4ERppWVevtSBC5IsP5t9bpgOSL/UR5GLXMnE42QQM -cas9UX4PB99jBVzpv5RvwSmCwLTaUbDBPLutN0pcyvFLNg4kq7/DhHf9qFD0sefG -L9ItWY16Ck6WaVICqjaY7Pz6FIMMNx/Jkjd/14Et5cS54D40/mf0PmbR0/RAz15i -NA9wBj4gGFrO93IbJWyTdBSTo3OxDqqHECNZXyAFGUftaI6SEspd/NYrspI8IM/h -X68gvqB2f3bl7BqGYTM+53u0P6APjqK5am+5hyZvQWyIplD9amML9ZMWGxmPsu2b -m8mQ9QEM3xk9Dz44I8kvjwzRAv4bVdZO0I08r0+k8/6vKtMFnXkIoctXMbScyJCy -Z/QYFpM6/EfY0XiWMR+6KwxfXZmtY4laJCB22N/9q06mIqqdXuYnin1oKaPnirja -EbsXLZmdEyRG98Xi2J+Of8ePdG1asuhy9azuJBCtLxTa/y2aRnFHvkLfuwHb9H/T -KI8xWVvTyQKmtFLKbpf7Q8UIJm+K9Lv9nyiqDdVF8xM6HdjAeI9BZzwelGSuewvF -6NkBiDkal4ZkQdU7hwxu+g/GvUgUvzlN1J5Bto+WHWOWk9mVBngxaJ43BjuAiUVh -OSPHG0SjFeUc+JIwuwIDAQABo4HvMIHsMB0GA1UdDgQWBBRlzeurNR4APn7VdMAc -tHNHDhpkLzASBgNVHRMBAf8ECDAGAQH/AgEBMIGmBgNVHSAEgZ4wgZswgZgGBFUd -IAAwgY8wLwYIKwYBBQUHAgEWI2h0dHA6Ly93d3cuZmlybWFwcm9mZXNpb25hbC5j -b20vY3BzMFwGCCsGAQUFBwICMFAeTgBQAGEAcwBlAG8AIABkAGUAIABsAGEAIABC -AG8AbgBhAG4AbwB2AGEAIAA0ADcAIABCAGEAcgBjAGUAbABvAG4AYQAgADAAOAAw -ADEANzAOBgNVHQ8BAf8EBAMCAQYwDQYJKoZIhvcNAQELBQADggIBAHSHKAIrdx9m -iWTtj3QuRhy7qPj4Cx2Dtjqn6EWKB7fgPiDL4QjbEwj4KKE1soCzC1HA01aajTNF -Sa9J8OA9B3pFE1r/yJfY0xgsfZb43aJlQ3CTkBW6kN/oGbDbLIpgD7dvlAceHabJ -hfa9NPhAeGIQcDq+fUs5gakQ1JZBu/hfHAsdCPKxsIl68veg4MSPi3i1O1ilI45P -Vf42O+AMt8oqMEEgtIDNrvx2ZnOorm7hfNoD6JQg5iKj0B+QXSBTFCZX2lSX3xZE -EAEeiGaPcjiT3SC3NL7X8e5jjkd5KAb881lFJWAiMxujX6i6KtoaPc1A6ozuBRWV -1aUsIC+nmCjuRfzxuIgALI9C2lHVnOUTaHFFQ4ueCyE8S1wF3BqfmI7avSKecs2t -CsvMo2ebKHTEm9caPARYpoKdrcd7b/+Alun4jWq9GJAd/0kakFI3ky88Al2CdgtR -5xbHV/g4+afNmyJU72OwFW1TZQNKXkqgsqeOSQBZONXH9IBk9W6VULgRfhVwOEqw -f9DEMnDAGf/JOC0ULGb0QkTmVXYbgBVX/8Cnp6o5qtjTcNAuuuuUavpfNIbnYrX9 -ivAwhZTJryQCL2/W3Wf+47BVTwSYT6RBVuKT0Gro1vP7ZeDOdcQxWQzugsgMYDNK -GbqEZycPvEJdvSRUDewdcAZfpLz6IHxV ------END CERTIFICATE----- - -# Issuer: CN=vTrus ECC Root CA O=iTrusChina Co.,Ltd. -# Subject: CN=vTrus ECC Root CA O=iTrusChina Co.,Ltd. -# Label: "vTrus ECC Root CA" -# Serial: 630369271402956006249506845124680065938238527194 -# MD5 Fingerprint: de:4b:c1:f5:52:8c:9b:43:e1:3e:8f:55:54:17:8d:85 -# SHA1 Fingerprint: f6:9c:db:b0:fc:f6:02:13:b6:52:32:a6:a3:91:3f:16:70:da:c3:e1 -# SHA256 Fingerprint: 30:fb:ba:2c:32:23:8e:2a:98:54:7a:f9:79:31:e5:50:42:8b:9b:3f:1c:8e:eb:66:33:dc:fa:86:c5:b2:7d:d3 ------BEGIN CERTIFICATE----- -MIICDzCCAZWgAwIBAgIUbmq8WapTvpg5Z6LSa6Q75m0c1towCgYIKoZIzj0EAwMw -RzELMAkGA1UEBhMCQ04xHDAaBgNVBAoTE2lUcnVzQ2hpbmEgQ28uLEx0ZC4xGjAY -BgNVBAMTEXZUcnVzIEVDQyBSb290IENBMB4XDTE4MDczMTA3MjY0NFoXDTQzMDcz -MTA3MjY0NFowRzELMAkGA1UEBhMCQ04xHDAaBgNVBAoTE2lUcnVzQ2hpbmEgQ28u -LEx0ZC4xGjAYBgNVBAMTEXZUcnVzIEVDQyBSb290IENBMHYwEAYHKoZIzj0CAQYF -K4EEACIDYgAEZVBKrox5lkqqHAjDo6LN/llWQXf9JpRCux3NCNtzslt188+cToL0 -v/hhJoVs1oVbcnDS/dtitN9Ti72xRFhiQgnH+n9bEOf+QP3A2MMrMudwpremIFUd -e4BdS49nTPEQo0IwQDAdBgNVHQ4EFgQUmDnNvtiyjPeyq+GtJK97fKHbH88wDwYD -VR0TAQH/BAUwAwEB/zAOBgNVHQ8BAf8EBAMCAQYwCgYIKoZIzj0EAwMDaAAwZQIw -V53dVvHH4+m4SVBrm2nDb+zDfSXkV5UTQJtS0zvzQBm8JsctBp61ezaf9SXUY2sA -AjEA6dPGnlaaKsyh2j/IZivTWJwghfqrkYpwcBE4YGQLYgmRWAD5Tfs0aNoJrSEG -GJTO ------END CERTIFICATE----- - -# Issuer: CN=vTrus Root CA O=iTrusChina Co.,Ltd. -# Subject: CN=vTrus Root CA O=iTrusChina Co.,Ltd. -# Label: "vTrus Root CA" -# Serial: 387574501246983434957692974888460947164905180485 -# MD5 Fingerprint: b8:c9:37:df:fa:6b:31:84:64:c5:ea:11:6a:1b:75:fc -# SHA1 Fingerprint: 84:1a:69:fb:f5:cd:1a:25:34:13:3d:e3:f8:fc:b8:99:d0:c9:14:b7 -# SHA256 Fingerprint: 8a:71:de:65:59:33:6f:42:6c:26:e5:38:80:d0:0d:88:a1:8d:a4:c6:a9:1f:0d:cb:61:94:e2:06:c5:c9:63:87 ------BEGIN CERTIFICATE----- -MIIFVjCCAz6gAwIBAgIUQ+NxE9izWRRdt86M/TX9b7wFjUUwDQYJKoZIhvcNAQEL -BQAwQzELMAkGA1UEBhMCQ04xHDAaBgNVBAoTE2lUcnVzQ2hpbmEgQ28uLEx0ZC4x -FjAUBgNVBAMTDXZUcnVzIFJvb3QgQ0EwHhcNMTgwNzMxMDcyNDA1WhcNNDMwNzMx -MDcyNDA1WjBDMQswCQYDVQQGEwJDTjEcMBoGA1UEChMTaVRydXNDaGluYSBDby4s -THRkLjEWMBQGA1UEAxMNdlRydXMgUm9vdCBDQTCCAiIwDQYJKoZIhvcNAQEBBQAD -ggIPADCCAgoCggIBAL1VfGHTuB0EYgWgrmy3cLRB6ksDXhA/kFocizuwZotsSKYc -IrrVQJLuM7IjWcmOvFjai57QGfIvWcaMY1q6n6MLsLOaXLoRuBLpDLvPbmyAhykU -AyyNJJrIZIO1aqwTLDPxn9wsYTwaP3BVm60AUn/PBLn+NvqcwBauYv6WTEN+VRS+ -GrPSbcKvdmaVayqwlHeFXgQPYh1jdfdr58tbmnDsPmcF8P4HCIDPKNsFxhQnL4Z9 -8Cfe/+Z+M0jnCx5Y0ScrUw5XSmXX+6KAYPxMvDVTAWqXcoKv8R1w6Jz1717CbMdH -flqUhSZNO7rrTOiwCcJlwp2dCZtOtZcFrPUGoPc2BX70kLJrxLT5ZOrpGgrIDajt -J8nU57O5q4IikCc9Kuh8kO+8T/3iCiSn3mUkpF3qwHYw03dQ+A0Em5Q2AXPKBlim -0zvc+gRGE1WKyURHuFE5Gi7oNOJ5y1lKCn+8pu8fA2dqWSslYpPZUxlmPCdiKYZN -pGvu/9ROutW04o5IWgAZCfEF2c6Rsffr6TlP9m8EQ5pV9T4FFL2/s1m02I4zhKOQ -UqqzApVg+QxMaPnu1RcN+HFXtSXkKe5lXa/R7jwXC1pDxaWG6iSe4gUH3DRCEpHW -OXSuTEGC2/KmSNGzm/MzqvOmwMVO9fSddmPmAsYiS8GVP1BkLFTltvA8Kc9XAgMB -AAGjQjBAMB0GA1UdDgQWBBRUYnBj8XWEQ1iO0RYgscasGrz2iTAPBgNVHRMBAf8E -BTADAQH/MA4GA1UdDwEB/wQEAwIBBjANBgkqhkiG9w0BAQsFAAOCAgEAKbqSSaet -8PFww+SX8J+pJdVrnjT+5hpk9jprUrIQeBqfTNqK2uwcN1LgQkv7bHbKJAs5EhWd -nxEt/Hlk3ODg9d3gV8mlsnZwUKT+twpw1aA08XXXTUm6EdGz2OyC/+sOxL9kLX1j -bhd47F18iMjrjld22VkE+rxSH0Ws8HqA7Oxvdq6R2xCOBNyS36D25q5J08FsEhvM -Kar5CKXiNxTKsbhm7xqC5PD48acWabfbqWE8n/Uxy+QARsIvdLGx14HuqCaVvIiv -TDUHKgLKeBRtRytAVunLKmChZwOgzoy8sHJnxDHO2zTlJQNgJXtxmOTAGytfdELS -S8VZCAeHvsXDf+eW2eHcKJfWjwXj9ZtOyh1QRwVTsMo554WgicEFOwE30z9J4nfr -I8iIZjs9OXYhRvHsXyO466JmdXTBQPfYaJqT4i2pLr0cox7IdMakLXogqzu4sEb9 -b91fUlV1YvCXoHzXOP0l382gmxDPi7g4Xl7FtKYCNqEeXxzP4padKar9mK5S4fNB -UvupLnKWnyfjqnN9+BojZns7q2WwMgFLFT49ok8MKzWixtlnEjUwzXYuFrOZnk1P -Ti07NEPhmg4NpGaXutIcSkwsKouLgU9xGqndXHt7CMUADTdA43x7VF8vhV929ven -sBxXVsFy6K2ir40zSbofitzmdHxghm+Hl3s= ------END CERTIFICATE----- - -# Issuer: CN=ISRG Root X2 O=Internet Security Research Group -# Subject: CN=ISRG Root X2 O=Internet Security Research Group -# Label: "ISRG Root X2" -# Serial: 87493402998870891108772069816698636114 -# MD5 Fingerprint: d3:9e:c4:1e:23:3c:a6:df:cf:a3:7e:6d:e0:14:e6:e5 -# SHA1 Fingerprint: bd:b1:b9:3c:d5:97:8d:45:c6:26:14:55:f8:db:95:c7:5a:d1:53:af -# SHA256 Fingerprint: 69:72:9b:8e:15:a8:6e:fc:17:7a:57:af:b7:17:1d:fc:64:ad:d2:8c:2f:ca:8c:f1:50:7e:34:45:3c:cb:14:70 ------BEGIN CERTIFICATE----- -MIICGzCCAaGgAwIBAgIQQdKd0XLq7qeAwSxs6S+HUjAKBggqhkjOPQQDAzBPMQsw -CQYDVQQGEwJVUzEpMCcGA1UEChMgSW50ZXJuZXQgU2VjdXJpdHkgUmVzZWFyY2gg -R3JvdXAxFTATBgNVBAMTDElTUkcgUm9vdCBYMjAeFw0yMDA5MDQwMDAwMDBaFw00 -MDA5MTcxNjAwMDBaME8xCzAJBgNVBAYTAlVTMSkwJwYDVQQKEyBJbnRlcm5ldCBT -ZWN1cml0eSBSZXNlYXJjaCBHcm91cDEVMBMGA1UEAxMMSVNSRyBSb290IFgyMHYw -EAYHKoZIzj0CAQYFK4EEACIDYgAEzZvVn4CDCuwJSvMWSj5cz3es3mcFDR0HttwW -+1qLFNvicWDEukWVEYmO6gbf9yoWHKS5xcUy4APgHoIYOIvXRdgKam7mAHf7AlF9 -ItgKbppbd9/w+kHsOdx1ymgHDB/qo0IwQDAOBgNVHQ8BAf8EBAMCAQYwDwYDVR0T -AQH/BAUwAwEB/zAdBgNVHQ4EFgQUfEKWrt5LSDv6kviejM9ti6lyN5UwCgYIKoZI -zj0EAwMDaAAwZQIwe3lORlCEwkSHRhtFcP9Ymd70/aTSVaYgLXTWNLxBo1BfASdW -tL4ndQavEi51mI38AjEAi/V3bNTIZargCyzuFJ0nN6T5U6VR5CmD1/iQMVtCnwr1 -/q4AaOeMSQ+2b1tbFfLn ------END CERTIFICATE----- - -# Issuer: CN=HiPKI Root CA - G1 O=Chunghwa Telecom Co., Ltd. -# Subject: CN=HiPKI Root CA - G1 O=Chunghwa Telecom Co., Ltd. -# Label: "HiPKI Root CA - G1" -# Serial: 60966262342023497858655262305426234976 -# MD5 Fingerprint: 69:45:df:16:65:4b:e8:68:9a:8f:76:5f:ff:80:9e:d3 -# SHA1 Fingerprint: 6a:92:e4:a8:ee:1b:ec:96:45:37:e3:29:57:49:cd:96:e3:e5:d2:60 -# SHA256 Fingerprint: f0:15:ce:3c:c2:39:bf:ef:06:4b:e9:f1:d2:c4:17:e1:a0:26:4a:0a:94:be:1f:0c:8d:12:18:64:eb:69:49:cc ------BEGIN CERTIFICATE----- -MIIFajCCA1KgAwIBAgIQLd2szmKXlKFD6LDNdmpeYDANBgkqhkiG9w0BAQsFADBP -MQswCQYDVQQGEwJUVzEjMCEGA1UECgwaQ2h1bmdod2EgVGVsZWNvbSBDby4sIEx0 -ZC4xGzAZBgNVBAMMEkhpUEtJIFJvb3QgQ0EgLSBHMTAeFw0xOTAyMjIwOTQ2MDRa -Fw0zNzEyMzExNTU5NTlaME8xCzAJBgNVBAYTAlRXMSMwIQYDVQQKDBpDaHVuZ2h3 -YSBUZWxlY29tIENvLiwgTHRkLjEbMBkGA1UEAwwSSGlQS0kgUm9vdCBDQSAtIEcx -MIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEA9B5/UnMyDHPkvRN0o9Qw -qNCuS9i233VHZvR85zkEHmpwINJaR3JnVfSl6J3VHiGh8Ge6zCFovkRTv4354twv -Vcg3Px+kwJyz5HdcoEb+d/oaoDjq7Zpy3iu9lFc6uux55199QmQ5eiY29yTw1S+6 -lZgRZq2XNdZ1AYDgr/SEYYwNHl98h5ZeQa/rh+r4XfEuiAU+TCK72h8q3VJGZDnz -Qs7ZngyzsHeXZJzA9KMuH5UHsBffMNsAGJZMoYFL3QRtU6M9/Aes1MU3guvklQgZ -KILSQjqj2FPseYlgSGDIcpJQ3AOPgz+yQlda22rpEZfdhSi8MEyr48KxRURHH+CK -FgeW0iEPU8DtqX7UTuybCeyvQqww1r/REEXgphaypcXTT3OUM3ECoWqj1jOXTyFj -HluP2cFeRXF3D4FdXyGarYPM+l7WjSNfGz1BryB1ZlpK9p/7qxj3ccC2HTHsOyDr -y+K49a6SsvfhhEvyovKTmiKe0xRvNlS9H15ZFblzqMF8b3ti6RZsR1pl8w4Rm0bZ -/W3c1pzAtH2lsN0/Vm+h+fbkEkj9Bn8SV7apI09bA8PgcSojt/ewsTu8mL3WmKgM -a/aOEmem8rJY5AIJEzypuxC00jBF8ez3ABHfZfjcK0NVvxaXxA/VLGGEqnKG/uY6 -fsI/fe78LxQ+5oXdUG+3Se0CAwEAAaNCMEAwDwYDVR0TAQH/BAUwAwEB/zAdBgNV -HQ4EFgQU8ncX+l6o/vY9cdVouslGDDjYr7AwDgYDVR0PAQH/BAQDAgGGMA0GCSqG -SIb3DQEBCwUAA4ICAQBQUfB13HAE4/+qddRxosuej6ip0691x1TPOhwEmSKsxBHi -7zNKpiMdDg1H2DfHb680f0+BazVP6XKlMeJ45/dOlBhbQH3PayFUhuaVevvGyuqc -SE5XCV0vrPSltJczWNWseanMX/mF+lLFjfiRFOs6DRfQUsJ748JzjkZ4Bjgs6Fza -ZsT0pPBWGTMpWmWSBUdGSquEwx4noR8RkpkndZMPvDY7l1ePJlsMu5wP1G4wB9Tc -XzZoZjmDlicmisjEOf6aIW/Vcobpf2Lll07QJNBAsNB1CI69aO4I1258EHBGG3zg -iLKecoaZAeO/n0kZtCW+VmWuF2PlHt/o/0elv+EmBYTksMCv5wiZqAxeJoBF1Pho -L5aPruJKHJwWDBNvOIf2u8g0X5IDUXlwpt/L9ZlNec1OvFefQ05rLisY+GpzjLrF -Ne85akEez3GoorKGB1s6yeHvP2UEgEcyRHCVTjFnanRbEEV16rCf0OY1/k6fi8wr -kkVbbiVghUbN0aqwdmaTd5a+g744tiROJgvM7XpWGuDpWsZkrUx6AEhEL7lAuxM+ -vhV4nYWBSipX3tUZQ9rbyltHhoMLP7YNdnhzeSJesYAfz77RP1YQmCuVh6EfnWQU -YDksswBVLuT1sw5XxJFBAJw/6KXf6vb/yPCtbVKoF6ubYfwSUTXkJf2vqmqGOQ== ------END CERTIFICATE----- - -# Issuer: CN=GlobalSign O=GlobalSign OU=GlobalSign ECC Root CA - R4 -# Subject: CN=GlobalSign O=GlobalSign OU=GlobalSign ECC Root CA - R4 -# Label: "GlobalSign ECC Root CA - R4" -# Serial: 159662223612894884239637590694 -# MD5 Fingerprint: 26:29:f8:6d:e1:88:bf:a2:65:7f:aa:c4:cd:0f:7f:fc -# SHA1 Fingerprint: 6b:a0:b0:98:e1:71:ef:5a:ad:fe:48:15:80:77:10:f4:bd:6f:0b:28 -# SHA256 Fingerprint: b0:85:d7:0b:96:4f:19:1a:73:e4:af:0d:54:ae:7a:0e:07:aa:fd:af:9b:71:dd:08:62:13:8a:b7:32:5a:24:a2 ------BEGIN CERTIFICATE----- -MIIB3DCCAYOgAwIBAgINAgPlfvU/k/2lCSGypjAKBggqhkjOPQQDAjBQMSQwIgYD -VQQLExtHbG9iYWxTaWduIEVDQyBSb290IENBIC0gUjQxEzARBgNVBAoTCkdsb2Jh -bFNpZ24xEzARBgNVBAMTCkdsb2JhbFNpZ24wHhcNMTIxMTEzMDAwMDAwWhcNMzgw -MTE5MDMxNDA3WjBQMSQwIgYDVQQLExtHbG9iYWxTaWduIEVDQyBSb290IENBIC0g -UjQxEzARBgNVBAoTCkdsb2JhbFNpZ24xEzARBgNVBAMTCkdsb2JhbFNpZ24wWTAT -BgcqhkjOPQIBBggqhkjOPQMBBwNCAAS4xnnTj2wlDp8uORkcA6SumuU5BwkWymOx -uYb4ilfBV85C+nOh92VC/x7BALJucw7/xyHlGKSq2XE/qNS5zowdo0IwQDAOBgNV -HQ8BAf8EBAMCAYYwDwYDVR0TAQH/BAUwAwEB/zAdBgNVHQ4EFgQUVLB7rUW44kB/ -+wpu+74zyTyjhNUwCgYIKoZIzj0EAwIDRwAwRAIgIk90crlgr/HmnKAWBVBfw147 -bmF0774BxL4YSFlhgjICICadVGNA3jdgUM/I2O2dgq43mLyjj0xMqTQrbO/7lZsm ------END CERTIFICATE----- - -# Issuer: CN=GTS Root R1 O=Google Trust Services LLC -# Subject: CN=GTS Root R1 O=Google Trust Services LLC -# Label: "GTS Root R1" -# Serial: 159662320309726417404178440727 -# MD5 Fingerprint: 05:fe:d0:bf:71:a8:a3:76:63:da:01:e0:d8:52:dc:40 -# SHA1 Fingerprint: e5:8c:1c:c4:91:3b:38:63:4b:e9:10:6e:e3:ad:8e:6b:9d:d9:81:4a -# SHA256 Fingerprint: d9:47:43:2a:bd:e7:b7:fa:90:fc:2e:6b:59:10:1b:12:80:e0:e1:c7:e4:e4:0f:a3:c6:88:7f:ff:57:a7:f4:cf ------BEGIN CERTIFICATE----- -MIIFVzCCAz+gAwIBAgINAgPlk28xsBNJiGuiFzANBgkqhkiG9w0BAQwFADBHMQsw -CQYDVQQGEwJVUzEiMCAGA1UEChMZR29vZ2xlIFRydXN0IFNlcnZpY2VzIExMQzEU -MBIGA1UEAxMLR1RTIFJvb3QgUjEwHhcNMTYwNjIyMDAwMDAwWhcNMzYwNjIyMDAw -MDAwWjBHMQswCQYDVQQGEwJVUzEiMCAGA1UEChMZR29vZ2xlIFRydXN0IFNlcnZp -Y2VzIExMQzEUMBIGA1UEAxMLR1RTIFJvb3QgUjEwggIiMA0GCSqGSIb3DQEBAQUA -A4ICDwAwggIKAoICAQC2EQKLHuOhd5s73L+UPreVp0A8of2C+X0yBoJx9vaMf/vo -27xqLpeXo4xL+Sv2sfnOhB2x+cWX3u+58qPpvBKJXqeqUqv4IyfLpLGcY9vXmX7w -Cl7raKb0xlpHDU0QM+NOsROjyBhsS+z8CZDfnWQpJSMHobTSPS5g4M/SCYe7zUjw -TcLCeoiKu7rPWRnWr4+wB7CeMfGCwcDfLqZtbBkOtdh+JhpFAz2weaSUKK0Pfybl -qAj+lug8aJRT7oM6iCsVlgmy4HqMLnXWnOunVmSPlk9orj2XwoSPwLxAwAtcvfaH -szVsrBhQf4TgTM2S0yDpM7xSma8ytSmzJSq0SPly4cpk9+aCEI3oncKKiPo4Zor8 -Y/kB+Xj9e1x3+naH+uzfsQ55lVe0vSbv1gHR6xYKu44LtcXFilWr06zqkUspzBmk -MiVOKvFlRNACzqrOSbTqn3yDsEB750Orp2yjj32JgfpMpf/VjsPOS+C12LOORc92 -wO1AK/1TD7Cn1TsNsYqiA94xrcx36m97PtbfkSIS5r762DL8EGMUUXLeXdYWk70p -aDPvOmbsB4om3xPXV2V4J95eSRQAogB/mqghtqmxlbCluQ0WEdrHbEg8QOB+DVrN -VjzRlwW5y0vtOUucxD/SVRNuJLDWcfr0wbrM7Rv1/oFB2ACYPTrIrnqYNxgFlQID -AQABo0IwQDAOBgNVHQ8BAf8EBAMCAYYwDwYDVR0TAQH/BAUwAwEB/zAdBgNVHQ4E -FgQU5K8rJnEaK0gnhS9SZizv8IkTcT4wDQYJKoZIhvcNAQEMBQADggIBAJ+qQibb -C5u+/x6Wki4+omVKapi6Ist9wTrYggoGxval3sBOh2Z5ofmmWJyq+bXmYOfg6LEe -QkEzCzc9zolwFcq1JKjPa7XSQCGYzyI0zzvFIoTgxQ6KfF2I5DUkzps+GlQebtuy -h6f88/qBVRRiClmpIgUxPoLW7ttXNLwzldMXG+gnoot7TiYaelpkttGsN/H9oPM4 -7HLwEXWdyzRSjeZ2axfG34arJ45JK3VmgRAhpuo+9K4l/3wV3s6MJT/KYnAK9y8J -ZgfIPxz88NtFMN9iiMG1D53Dn0reWVlHxYciNuaCp+0KueIHoI17eko8cdLiA6Ef -MgfdG+RCzgwARWGAtQsgWSl4vflVy2PFPEz0tv/bal8xa5meLMFrUKTX5hgUvYU/ -Z6tGn6D/Qqc6f1zLXbBwHSs09dR2CQzreExZBfMzQsNhFRAbd03OIozUhfJFfbdT -6u9AWpQKXCBfTkBdYiJ23//OYb2MI3jSNwLgjt7RETeJ9r/tSQdirpLsQBqvFAnZ -0E6yove+7u7Y/9waLd64NnHi/Hm3lCXRSHNboTXns5lndcEZOitHTtNCjv0xyBZm -2tIMPNuzjsmhDYAPexZ3FL//2wmUspO8IFgV6dtxQ/PeEMMA3KgqlbbC1j+Qa3bb -bP6MvPJwNQzcmRk13NfIRmPVNnGuV/u3gm3c ------END CERTIFICATE----- - -# Issuer: CN=GTS Root R2 O=Google Trust Services LLC -# Subject: CN=GTS Root R2 O=Google Trust Services LLC -# Label: "GTS Root R2" -# Serial: 159662449406622349769042896298 -# MD5 Fingerprint: 1e:39:c0:53:e6:1e:29:82:0b:ca:52:55:36:5d:57:dc -# SHA1 Fingerprint: 9a:44:49:76:32:db:de:fa:d0:bc:fb:5a:7b:17:bd:9e:56:09:24:94 -# SHA256 Fingerprint: 8d:25:cd:97:22:9d:bf:70:35:6b:da:4e:b3:cc:73:40:31:e2:4c:f0:0f:af:cf:d3:2d:c7:6e:b5:84:1c:7e:a8 ------BEGIN CERTIFICATE----- -MIIFVzCCAz+gAwIBAgINAgPlrsWNBCUaqxElqjANBgkqhkiG9w0BAQwFADBHMQsw -CQYDVQQGEwJVUzEiMCAGA1UEChMZR29vZ2xlIFRydXN0IFNlcnZpY2VzIExMQzEU -MBIGA1UEAxMLR1RTIFJvb3QgUjIwHhcNMTYwNjIyMDAwMDAwWhcNMzYwNjIyMDAw -MDAwWjBHMQswCQYDVQQGEwJVUzEiMCAGA1UEChMZR29vZ2xlIFRydXN0IFNlcnZp -Y2VzIExMQzEUMBIGA1UEAxMLR1RTIFJvb3QgUjIwggIiMA0GCSqGSIb3DQEBAQUA -A4ICDwAwggIKAoICAQDO3v2m++zsFDQ8BwZabFn3GTXd98GdVarTzTukk3LvCvpt -nfbwhYBboUhSnznFt+4orO/LdmgUud+tAWyZH8QiHZ/+cnfgLFuv5AS/T3KgGjSY -6Dlo7JUle3ah5mm5hRm9iYz+re026nO8/4Piy33B0s5Ks40FnotJk9/BW9BuXvAu -MC6C/Pq8tBcKSOWIm8Wba96wyrQD8Nr0kLhlZPdcTK3ofmZemde4wj7I0BOdre7k -RXuJVfeKH2JShBKzwkCX44ofR5GmdFrS+LFjKBC4swm4VndAoiaYecb+3yXuPuWg -f9RhD1FLPD+M2uFwdNjCaKH5wQzpoeJ/u1U8dgbuak7MkogwTZq9TwtImoS1mKPV -+3PBV2HdKFZ1E66HjucMUQkQdYhMvI35ezzUIkgfKtzra7tEscszcTJGr61K8Yzo -dDqs5xoic4DSMPclQsciOzsSrZYuxsN2B6ogtzVJV+mSSeh2FnIxZyuWfoqjx5RW -Ir9qS34BIbIjMt/kmkRtWVtd9QCgHJvGeJeNkP+byKq0rxFROV7Z+2et1VsRnTKa -G73VululycslaVNVJ1zgyjbLiGH7HrfQy+4W+9OmTN6SpdTi3/UGVN4unUu0kzCq -gc7dGtxRcw1PcOnlthYhGXmy5okLdWTK1au8CcEYof/UVKGFPP0UJAOyh9OktwID -AQABo0IwQDAOBgNVHQ8BAf8EBAMCAYYwDwYDVR0TAQH/BAUwAwEB/zAdBgNVHQ4E -FgQUu//KjiOfT5nK2+JopqUVJxce2Q4wDQYJKoZIhvcNAQEMBQADggIBAB/Kzt3H -vqGf2SdMC9wXmBFqiN495nFWcrKeGk6c1SuYJF2ba3uwM4IJvd8lRuqYnrYb/oM8 -0mJhwQTtzuDFycgTE1XnqGOtjHsB/ncw4c5omwX4Eu55MaBBRTUoCnGkJE+M3DyC -B19m3H0Q/gxhswWV7uGugQ+o+MePTagjAiZrHYNSVc61LwDKgEDg4XSsYPWHgJ2u -NmSRXbBoGOqKYcl3qJfEycel/FVL8/B/uWU9J2jQzGv6U53hkRrJXRqWbTKH7QMg -yALOWr7Z6v2yTcQvG99fevX4i8buMTolUVVnjWQye+mew4K6Ki3pHrTgSAai/Gev -HyICc/sgCq+dVEuhzf9gR7A/Xe8bVr2XIZYtCtFenTgCR2y59PYjJbigapordwj6 -xLEokCZYCDzifqrXPW+6MYgKBesntaFJ7qBFVHvmJ2WZICGoo7z7GJa7Um8M7YNR -TOlZ4iBgxcJlkoKM8xAfDoqXvneCbT+PHV28SSe9zE8P4c52hgQjxcCMElv924Sg -JPFI/2R80L5cFtHvma3AH/vLrrw4IgYmZNralw4/KBVEqE8AyvCazM90arQ+POuV -7LXTWtiBmelDGDfrs7vRWGJB82bSj6p4lVQgw1oudCvV0b4YacCs1aTPObpRhANl -6WLAYv7YTVWW4tAR+kg0Eeye7QUd5MjWHYbL ------END CERTIFICATE----- - -# Issuer: CN=GTS Root R3 O=Google Trust Services LLC -# Subject: CN=GTS Root R3 O=Google Trust Services LLC -# Label: "GTS Root R3" -# Serial: 159662495401136852707857743206 -# MD5 Fingerprint: 3e:e7:9d:58:02:94:46:51:94:e5:e0:22:4a:8b:e7:73 -# SHA1 Fingerprint: ed:e5:71:80:2b:c8:92:b9:5b:83:3c:d2:32:68:3f:09:cd:a0:1e:46 -# SHA256 Fingerprint: 34:d8:a7:3e:e2:08:d9:bc:db:0d:95:65:20:93:4b:4e:40:e6:94:82:59:6e:8b:6f:73:c8:42:6b:01:0a:6f:48 ------BEGIN CERTIFICATE----- -MIICCTCCAY6gAwIBAgINAgPluILrIPglJ209ZjAKBggqhkjOPQQDAzBHMQswCQYD -VQQGEwJVUzEiMCAGA1UEChMZR29vZ2xlIFRydXN0IFNlcnZpY2VzIExMQzEUMBIG -A1UEAxMLR1RTIFJvb3QgUjMwHhcNMTYwNjIyMDAwMDAwWhcNMzYwNjIyMDAwMDAw -WjBHMQswCQYDVQQGEwJVUzEiMCAGA1UEChMZR29vZ2xlIFRydXN0IFNlcnZpY2Vz -IExMQzEUMBIGA1UEAxMLR1RTIFJvb3QgUjMwdjAQBgcqhkjOPQIBBgUrgQQAIgNi -AAQfTzOHMymKoYTey8chWEGJ6ladK0uFxh1MJ7x/JlFyb+Kf1qPKzEUURout736G -jOyxfi//qXGdGIRFBEFVbivqJn+7kAHjSxm65FSWRQmx1WyRRK2EE46ajA2ADDL2 -4CejQjBAMA4GA1UdDwEB/wQEAwIBhjAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQW -BBTB8Sa6oC2uhYHP0/EqEr24Cmf9vDAKBggqhkjOPQQDAwNpADBmAjEA9uEglRR7 -VKOQFhG/hMjqb2sXnh5GmCCbn9MN2azTL818+FsuVbu/3ZL3pAzcMeGiAjEA/Jdm -ZuVDFhOD3cffL74UOO0BzrEXGhF16b0DjyZ+hOXJYKaV11RZt+cRLInUue4X ------END CERTIFICATE----- - -# Issuer: CN=GTS Root R4 O=Google Trust Services LLC -# Subject: CN=GTS Root R4 O=Google Trust Services LLC -# Label: "GTS Root R4" -# Serial: 159662532700760215368942768210 -# MD5 Fingerprint: 43:96:83:77:19:4d:76:b3:9d:65:52:e4:1d:22:a5:e8 -# SHA1 Fingerprint: 77:d3:03:67:b5:e0:0c:15:f6:0c:38:61:df:7c:e1:3b:92:46:4d:47 -# SHA256 Fingerprint: 34:9d:fa:40:58:c5:e2:63:12:3b:39:8a:e7:95:57:3c:4e:13:13:c8:3f:e6:8f:93:55:6c:d5:e8:03:1b:3c:7d ------BEGIN CERTIFICATE----- -MIICCTCCAY6gAwIBAgINAgPlwGjvYxqccpBQUjAKBggqhkjOPQQDAzBHMQswCQYD -VQQGEwJVUzEiMCAGA1UEChMZR29vZ2xlIFRydXN0IFNlcnZpY2VzIExMQzEUMBIG -A1UEAxMLR1RTIFJvb3QgUjQwHhcNMTYwNjIyMDAwMDAwWhcNMzYwNjIyMDAwMDAw -WjBHMQswCQYDVQQGEwJVUzEiMCAGA1UEChMZR29vZ2xlIFRydXN0IFNlcnZpY2Vz -IExMQzEUMBIGA1UEAxMLR1RTIFJvb3QgUjQwdjAQBgcqhkjOPQIBBgUrgQQAIgNi -AATzdHOnaItgrkO4NcWBMHtLSZ37wWHO5t5GvWvVYRg1rkDdc/eJkTBa6zzuhXyi -QHY7qca4R9gq55KRanPpsXI5nymfopjTX15YhmUPoYRlBtHci8nHc8iMai/lxKvR -HYqjQjBAMA4GA1UdDwEB/wQEAwIBhjAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQW -BBSATNbrdP9JNqPV2Py1PsVq8JQdjDAKBggqhkjOPQQDAwNpADBmAjEA6ED/g94D -9J+uHXqnLrmvT/aDHQ4thQEd0dlq7A/Cr8deVl5c1RxYIigL9zC2L7F8AjEA8GE8 -p/SgguMh1YQdc4acLa/KNJvxn7kjNuK8YAOdgLOaVsjh4rsUecrNIdSUtUlD ------END CERTIFICATE----- - -# Issuer: CN=Telia Root CA v2 O=Telia Finland Oyj -# Subject: CN=Telia Root CA v2 O=Telia Finland Oyj -# Label: "Telia Root CA v2" -# Serial: 7288924052977061235122729490515358 -# MD5 Fingerprint: 0e:8f:ac:aa:82:df:85:b1:f4:dc:10:1c:fc:99:d9:48 -# SHA1 Fingerprint: b9:99:cd:d1:73:50:8a:c4:47:05:08:9c:8c:88:fb:be:a0:2b:40:cd -# SHA256 Fingerprint: 24:2b:69:74:2f:cb:1e:5b:2a:bf:98:89:8b:94:57:21:87:54:4e:5b:4d:99:11:78:65:73:62:1f:6a:74:b8:2c ------BEGIN CERTIFICATE----- -MIIFdDCCA1ygAwIBAgIPAWdfJ9b+euPkrL4JWwWeMA0GCSqGSIb3DQEBCwUAMEQx -CzAJBgNVBAYTAkZJMRowGAYDVQQKDBFUZWxpYSBGaW5sYW5kIE95ajEZMBcGA1UE -AwwQVGVsaWEgUm9vdCBDQSB2MjAeFw0xODExMjkxMTU1NTRaFw00MzExMjkxMTU1 -NTRaMEQxCzAJBgNVBAYTAkZJMRowGAYDVQQKDBFUZWxpYSBGaW5sYW5kIE95ajEZ -MBcGA1UEAwwQVGVsaWEgUm9vdCBDQSB2MjCCAiIwDQYJKoZIhvcNAQEBBQADggIP -ADCCAgoCggIBALLQPwe84nvQa5n44ndp586dpAO8gm2h/oFlH0wnrI4AuhZ76zBq -AMCzdGh+sq/H1WKzej9Qyow2RCRj0jbpDIX2Q3bVTKFgcmfiKDOlyzG4OiIjNLh9 -vVYiQJ3q9HsDrWj8soFPmNB06o3lfc1jw6P23pLCWBnglrvFxKk9pXSW/q/5iaq9 -lRdU2HhE8Qx3FZLgmEKnpNaqIJLNwaCzlrI6hEKNfdWV5Nbb6WLEWLN5xYzTNTOD -n3WhUidhOPFZPY5Q4L15POdslv5e2QJltI5c0BE0312/UqeBAMN/mUWZFdUXyApT -7GPzmX3MaRKGwhfwAZ6/hLzRUssbkmbOpFPlob/E2wnW5olWK8jjfN7j/4nlNW4o -6GwLI1GpJQXrSPjdscr6bAhR77cYbETKJuFzxokGgeWKrLDiKca5JLNrRBH0pUPC -TEPlcDaMtjNXepUugqD0XBCzYYP2AgWGLnwtbNwDRm41k9V6lS/eINhbfpSQBGq6 -WT0EBXWdN6IOLj3rwaRSg/7Qa9RmjtzG6RJOHSpXqhC8fF6CfaamyfItufUXJ63R -DolUK5X6wK0dmBR4M0KGCqlztft0DbcbMBnEWg4cJ7faGND/isgFuvGqHKI3t+ZI -pEYslOqodmJHixBTB0hXbOKSTbauBcvcwUpej6w9GU7C7WB1K9vBykLVAgMBAAGj -YzBhMB8GA1UdIwQYMBaAFHKs5DN5qkWH9v2sHZ7Wxy+G2CQ5MB0GA1UdDgQWBBRy -rOQzeapFh/b9rB2e1scvhtgkOTAOBgNVHQ8BAf8EBAMCAQYwDwYDVR0TAQH/BAUw -AwEB/zANBgkqhkiG9w0BAQsFAAOCAgEAoDtZpwmUPjaE0n4vOaWWl/oRrfxn83EJ -8rKJhGdEr7nv7ZbsnGTbMjBvZ5qsfl+yqwE2foH65IRe0qw24GtixX1LDoJt0nZi -0f6X+J8wfBj5tFJ3gh1229MdqfDBmgC9bXXYfef6xzijnHDoRnkDry5023X4blMM -A8iZGok1GTzTyVR8qPAs5m4HeW9q4ebqkYJpCh3DflminmtGFZhb069GHWLIzoBS -SRE/yQQSwxN8PzuKlts8oB4KtItUsiRnDe+Cy748fdHif64W1lZYudogsYMVoe+K -TTJvQS8TUoKU1xrBeKJR3Stwbbca+few4GeXVtt8YVMJAygCQMez2P2ccGrGKMOF -6eLtGpOg3kuYooQ+BXcBlj37tCAPnHICehIv1aO6UXivKitEZU61/Qrowc15h2Er -3oBXRb9n8ZuRXqWk7FlIEA04x7D6w0RtBPV4UBySllva9bguulvP5fBqnUsvWHMt -Ty3EHD70sz+rFQ47GUGKpMFXEmZxTPpT41frYpUJnlTd0cI8Vzy9OK2YZLe4A5pT -VmBds9hCG1xLEooc6+t9xnppxyd/pPiL8uSUZodL6ZQHCRJ5irLrdATczvREWeAW -ysUsWNc8e89ihmpQfTU2Zqf7N+cox9jQraVplI/owd8k+BsHMYeB2F326CjYSlKA -rBPuUBQemMc= ------END CERTIFICATE----- - -# Issuer: CN=D-TRUST BR Root CA 1 2020 O=D-Trust GmbH -# Subject: CN=D-TRUST BR Root CA 1 2020 O=D-Trust GmbH -# Label: "D-TRUST BR Root CA 1 2020" -# Serial: 165870826978392376648679885835942448534 -# MD5 Fingerprint: b5:aa:4b:d5:ed:f7:e3:55:2e:8f:72:0a:f3:75:b8:ed -# SHA1 Fingerprint: 1f:5b:98:f0:e3:b5:f7:74:3c:ed:e6:b0:36:7d:32:cd:f4:09:41:67 -# SHA256 Fingerprint: e5:9a:aa:81:60:09:c2:2b:ff:5b:25:ba:d3:7d:f3:06:f0:49:79:7c:1f:81:d8:5a:b0:89:e6:57:bd:8f:00:44 ------BEGIN CERTIFICATE----- -MIIC2zCCAmCgAwIBAgIQfMmPK4TX3+oPyWWa00tNljAKBggqhkjOPQQDAzBIMQsw -CQYDVQQGEwJERTEVMBMGA1UEChMMRC1UcnVzdCBHbWJIMSIwIAYDVQQDExlELVRS -VVNUIEJSIFJvb3QgQ0EgMSAyMDIwMB4XDTIwMDIxMTA5NDUwMFoXDTM1MDIxMTA5 -NDQ1OVowSDELMAkGA1UEBhMCREUxFTATBgNVBAoTDEQtVHJ1c3QgR21iSDEiMCAG -A1UEAxMZRC1UUlVTVCBCUiBSb290IENBIDEgMjAyMDB2MBAGByqGSM49AgEGBSuB -BAAiA2IABMbLxyjR+4T1mu9CFCDhQ2tuda38KwOE1HaTJddZO0Flax7mNCq7dPYS -zuht56vkPE4/RAiLzRZxy7+SmfSk1zxQVFKQhYN4lGdnoxwJGT11NIXe7WB9xwy0 -QVK5buXuQqOCAQ0wggEJMA8GA1UdEwEB/wQFMAMBAf8wHQYDVR0OBBYEFHOREKv/ -VbNafAkl1bK6CKBrqx9tMA4GA1UdDwEB/wQEAwIBBjCBxgYDVR0fBIG+MIG7MD6g -PKA6hjhodHRwOi8vY3JsLmQtdHJ1c3QubmV0L2NybC9kLXRydXN0X2JyX3Jvb3Rf -Y2FfMV8yMDIwLmNybDB5oHegdYZzbGRhcDovL2RpcmVjdG9yeS5kLXRydXN0Lm5l -dC9DTj1ELVRSVVNUJTIwQlIlMjBSb290JTIwQ0ElMjAxJTIwMjAyMCxPPUQtVHJ1 -c3QlMjBHbWJILEM9REU/Y2VydGlmaWNhdGVyZXZvY2F0aW9ubGlzdDAKBggqhkjO -PQQDAwNpADBmAjEAlJAtE/rhY/hhY+ithXhUkZy4kzg+GkHaQBZTQgjKL47xPoFW -wKrY7RjEsK70PvomAjEA8yjixtsrmfu3Ubgko6SUeho/5jbiA1czijDLgsfWFBHV -dWNbFJWcHwHP2NVypw87 ------END CERTIFICATE----- - -# Issuer: CN=D-TRUST EV Root CA 1 2020 O=D-Trust GmbH -# Subject: CN=D-TRUST EV Root CA 1 2020 O=D-Trust GmbH -# Label: "D-TRUST EV Root CA 1 2020" -# Serial: 126288379621884218666039612629459926992 -# MD5 Fingerprint: 8c:2d:9d:70:9f:48:99:11:06:11:fb:e9:cb:30:c0:6e -# SHA1 Fingerprint: 61:db:8c:21:59:69:03:90:d8:7c:9c:12:86:54:cf:9d:3d:f4:dd:07 -# SHA256 Fingerprint: 08:17:0d:1a:a3:64:53:90:1a:2f:95:92:45:e3:47:db:0c:8d:37:ab:aa:bc:56:b8:1a:a1:00:dc:95:89:70:db ------BEGIN CERTIFICATE----- -MIIC2zCCAmCgAwIBAgIQXwJB13qHfEwDo6yWjfv/0DAKBggqhkjOPQQDAzBIMQsw -CQYDVQQGEwJERTEVMBMGA1UEChMMRC1UcnVzdCBHbWJIMSIwIAYDVQQDExlELVRS -VVNUIEVWIFJvb3QgQ0EgMSAyMDIwMB4XDTIwMDIxMTEwMDAwMFoXDTM1MDIxMTA5 -NTk1OVowSDELMAkGA1UEBhMCREUxFTATBgNVBAoTDEQtVHJ1c3QgR21iSDEiMCAG -A1UEAxMZRC1UUlVTVCBFViBSb290IENBIDEgMjAyMDB2MBAGByqGSM49AgEGBSuB -BAAiA2IABPEL3YZDIBnfl4XoIkqbz52Yv7QFJsnL46bSj8WeeHsxiamJrSc8ZRCC -/N/DnU7wMyPE0jL1HLDfMxddxfCxivnvubcUyilKwg+pf3VlSSowZ/Rk99Yad9rD -wpdhQntJraOCAQ0wggEJMA8GA1UdEwEB/wQFMAMBAf8wHQYDVR0OBBYEFH8QARY3 -OqQo5FD4pPfsazK2/umLMA4GA1UdDwEB/wQEAwIBBjCBxgYDVR0fBIG+MIG7MD6g -PKA6hjhodHRwOi8vY3JsLmQtdHJ1c3QubmV0L2NybC9kLXRydXN0X2V2X3Jvb3Rf -Y2FfMV8yMDIwLmNybDB5oHegdYZzbGRhcDovL2RpcmVjdG9yeS5kLXRydXN0Lm5l -dC9DTj1ELVRSVVNUJTIwRVYlMjBSb290JTIwQ0ElMjAxJTIwMjAyMCxPPUQtVHJ1 -c3QlMjBHbWJILEM9REU/Y2VydGlmaWNhdGVyZXZvY2F0aW9ubGlzdDAKBggqhkjO -PQQDAwNpADBmAjEAyjzGKnXCXnViOTYAYFqLwZOZzNnbQTs7h5kXO9XMT8oi96CA -y/m0sRtW9XLS/BnRAjEAkfcwkz8QRitxpNA7RJvAKQIFskF3UfN5Wp6OFKBOQtJb -gfM0agPnIjhQW+0ZT0MW ------END CERTIFICATE----- - -# Issuer: CN=DigiCert TLS ECC P384 Root G5 O=DigiCert, Inc. -# Subject: CN=DigiCert TLS ECC P384 Root G5 O=DigiCert, Inc. -# Label: "DigiCert TLS ECC P384 Root G5" -# Serial: 13129116028163249804115411775095713523 -# MD5 Fingerprint: d3:71:04:6a:43:1c:db:a6:59:e1:a8:a3:aa:c5:71:ed -# SHA1 Fingerprint: 17:f3:de:5e:9f:0f:19:e9:8e:f6:1f:32:26:6e:20:c4:07:ae:30:ee -# SHA256 Fingerprint: 01:8e:13:f0:77:25:32:cf:80:9b:d1:b1:72:81:86:72:83:fc:48:c6:e1:3b:e9:c6:98:12:85:4a:49:0c:1b:05 ------BEGIN CERTIFICATE----- -MIICGTCCAZ+gAwIBAgIQCeCTZaz32ci5PhwLBCou8zAKBggqhkjOPQQDAzBOMQsw -CQYDVQQGEwJVUzEXMBUGA1UEChMORGlnaUNlcnQsIEluYy4xJjAkBgNVBAMTHURp -Z2lDZXJ0IFRMUyBFQ0MgUDM4NCBSb290IEc1MB4XDTIxMDExNTAwMDAwMFoXDTQ2 -MDExNDIzNTk1OVowTjELMAkGA1UEBhMCVVMxFzAVBgNVBAoTDkRpZ2lDZXJ0LCBJ -bmMuMSYwJAYDVQQDEx1EaWdpQ2VydCBUTFMgRUNDIFAzODQgUm9vdCBHNTB2MBAG -ByqGSM49AgEGBSuBBAAiA2IABMFEoc8Rl1Ca3iOCNQfN0MsYndLxf3c1TzvdlHJS -7cI7+Oz6e2tYIOyZrsn8aLN1udsJ7MgT9U7GCh1mMEy7H0cKPGEQQil8pQgO4CLp -0zVozptjn4S1mU1YoI71VOeVyaNCMEAwHQYDVR0OBBYEFMFRRVBZqz7nLFr6ICIS -B4CIfBFqMA4GA1UdDwEB/wQEAwIBhjAPBgNVHRMBAf8EBTADAQH/MAoGCCqGSM49 -BAMDA2gAMGUCMQCJao1H5+z8blUD2WdsJk6Dxv3J+ysTvLd6jLRl0mlpYxNjOyZQ -LgGheQaRnUi/wr4CMEfDFXuxoJGZSZOoPHzoRgaLLPIxAJSdYsiJvRmEFOml+wG4 -DXZDjC5Ty3zfDBeWUA== ------END CERTIFICATE----- - -# Issuer: CN=DigiCert TLS RSA4096 Root G5 O=DigiCert, Inc. -# Subject: CN=DigiCert TLS RSA4096 Root G5 O=DigiCert, Inc. -# Label: "DigiCert TLS RSA4096 Root G5" -# Serial: 11930366277458970227240571539258396554 -# MD5 Fingerprint: ac:fe:f7:34:96:a9:f2:b3:b4:12:4b:e4:27:41:6f:e1 -# SHA1 Fingerprint: a7:88:49:dc:5d:7c:75:8c:8c:de:39:98:56:b3:aa:d0:b2:a5:71:35 -# SHA256 Fingerprint: 37:1a:00:dc:05:33:b3:72:1a:7e:eb:40:e8:41:9e:70:79:9d:2b:0a:0f:2c:1d:80:69:31:65:f7:ce:c4:ad:75 ------BEGIN CERTIFICATE----- -MIIFZjCCA06gAwIBAgIQCPm0eKj6ftpqMzeJ3nzPijANBgkqhkiG9w0BAQwFADBN -MQswCQYDVQQGEwJVUzEXMBUGA1UEChMORGlnaUNlcnQsIEluYy4xJTAjBgNVBAMT -HERpZ2lDZXJ0IFRMUyBSU0E0MDk2IFJvb3QgRzUwHhcNMjEwMTE1MDAwMDAwWhcN -NDYwMTE0MjM1OTU5WjBNMQswCQYDVQQGEwJVUzEXMBUGA1UEChMORGlnaUNlcnQs -IEluYy4xJTAjBgNVBAMTHERpZ2lDZXJ0IFRMUyBSU0E0MDk2IFJvb3QgRzUwggIi -MA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQCz0PTJeRGd/fxmgefM1eS87IE+ -ajWOLrfn3q/5B03PMJ3qCQuZvWxX2hhKuHisOjmopkisLnLlvevxGs3npAOpPxG0 -2C+JFvuUAT27L/gTBaF4HI4o4EXgg/RZG5Wzrn4DReW+wkL+7vI8toUTmDKdFqgp -wgscONyfMXdcvyej/Cestyu9dJsXLfKB2l2w4SMXPohKEiPQ6s+d3gMXsUJKoBZM -pG2T6T867jp8nVid9E6P/DsjyG244gXazOvswzH016cpVIDPRFtMbzCe88zdH5RD -nU1/cHAN1DrRN/BsnZvAFJNY781BOHW8EwOVfH/jXOnVDdXifBBiqmvwPXbzP6Po -sMH976pXTayGpxi0KcEsDr9kvimM2AItzVwv8n/vFfQMFawKsPHTDU9qTXeXAaDx -Zre3zu/O7Oyldcqs4+Fj97ihBMi8ez9dLRYiVu1ISf6nL3kwJZu6ay0/nTvEF+cd -Lvvyz6b84xQslpghjLSR6Rlgg/IwKwZzUNWYOwbpx4oMYIwo+FKbbuH2TbsGJJvX -KyY//SovcfXWJL5/MZ4PbeiPT02jP/816t9JXkGPhvnxd3lLG7SjXi/7RgLQZhNe -XoVPzthwiHvOAbWWl9fNff2C+MIkwcoBOU+NosEUQB+cZtUMCUbW8tDRSHZWOkPL -tgoRObqME2wGtZ7P6wIDAQABo0IwQDAdBgNVHQ4EFgQUUTMc7TZArxfTJc1paPKv -TiM+s0EwDgYDVR0PAQH/BAQDAgGGMA8GA1UdEwEB/wQFMAMBAf8wDQYJKoZIhvcN -AQEMBQADggIBAGCmr1tfV9qJ20tQqcQjNSH/0GEwhJG3PxDPJY7Jv0Y02cEhJhxw -GXIeo8mH/qlDZJY6yFMECrZBu8RHANmfGBg7sg7zNOok992vIGCukihfNudd5N7H -PNtQOa27PShNlnx2xlv0wdsUpasZYgcYQF+Xkdycx6u1UQ3maVNVzDl92sURVXLF -O4uJ+DQtpBflF+aZfTCIITfNMBc9uPK8qHWgQ9w+iUuQrm0D4ByjoJYJu32jtyoQ -REtGBzRj7TG5BO6jm5qu5jF49OokYTurWGT/u4cnYiWB39yhL/btp/96j1EuMPik -AdKFOV8BmZZvWltwGUb+hmA+rYAQCd05JS9Yf7vSdPD3Rh9GOUrYU9DzLjtxpdRv -/PNn5AeP3SYZ4Y1b+qOTEZvpyDrDVWiakuFSdjjo4bq9+0/V77PnSIMx8IIh47a+ -p6tv75/fTM8BuGJqIz3nCU2AG3swpMPdB380vqQmsvZB6Akd4yCYqjdP//fx4ilw -MUc/dNAUFvohigLVigmUdy7yWSiLfFCSCmZ4OIN1xLVaqBHG5cGdZlXPU8Sv13WF -qUITVuwhd4GTWgzqltlJyqEI8pc7bZsEGCREjnwB8twl2F6GmrE52/WRMmrRpnCK -ovfepEWFJqgejF0pW8hL2JpqA15w8oVPbEtoL8pU9ozaMv7Da4M/OMZ+ ------END CERTIFICATE----- - -# Issuer: CN=Certainly Root R1 O=Certainly -# Subject: CN=Certainly Root R1 O=Certainly -# Label: "Certainly Root R1" -# Serial: 188833316161142517227353805653483829216 -# MD5 Fingerprint: 07:70:d4:3e:82:87:a0:fa:33:36:13:f4:fa:33:e7:12 -# SHA1 Fingerprint: a0:50:ee:0f:28:71:f4:27:b2:12:6d:6f:50:96:25:ba:cc:86:42:af -# SHA256 Fingerprint: 77:b8:2c:d8:64:4c:43:05:f7:ac:c5:cb:15:6b:45:67:50:04:03:3d:51:c6:0c:62:02:a8:e0:c3:34:67:d3:a0 ------BEGIN CERTIFICATE----- -MIIFRzCCAy+gAwIBAgIRAI4P+UuQcWhlM1T01EQ5t+AwDQYJKoZIhvcNAQELBQAw -PTELMAkGA1UEBhMCVVMxEjAQBgNVBAoTCUNlcnRhaW5seTEaMBgGA1UEAxMRQ2Vy -dGFpbmx5IFJvb3QgUjEwHhcNMjEwNDAxMDAwMDAwWhcNNDYwNDAxMDAwMDAwWjA9 -MQswCQYDVQQGEwJVUzESMBAGA1UEChMJQ2VydGFpbmx5MRowGAYDVQQDExFDZXJ0 -YWlubHkgUm9vdCBSMTCCAiIwDQYJKoZIhvcNAQEBBQADggIPADCCAgoCggIBANA2 -1B/q3avk0bbm+yLA3RMNansiExyXPGhjZjKcA7WNpIGD2ngwEc/csiu+kr+O5MQT -vqRoTNoCaBZ0vrLdBORrKt03H2As2/X3oXyVtwxwhi7xOu9S98zTm/mLvg7fMbed -aFySpvXl8wo0tf97ouSHocavFwDvA5HtqRxOcT3Si2yJ9HiG5mpJoM610rCrm/b0 -1C7jcvk2xusVtyWMOvwlDbMicyF0yEqWYZL1LwsYpfSt4u5BvQF5+paMjRcCMLT5 -r3gajLQ2EBAHBXDQ9DGQilHFhiZ5shGIXsXwClTNSaa/ApzSRKft43jvRl5tcdF5 -cBxGX1HpyTfcX35pe0HfNEXgO4T0oYoKNp43zGJS4YkNKPl6I7ENPT2a/Z2B7yyQ -wHtETrtJ4A5KVpK8y7XdeReJkd5hiXSSqOMyhb5OhaRLWcsrxXiOcVTQAjeZjOVJ -6uBUcqQRBi8LjMFbvrWhsFNunLhgkR9Za/kt9JQKl7XsxXYDVBtlUrpMklZRNaBA -2CnbrlJ2Oy0wQJuK0EJWtLeIAaSHO1OWzaMWj/Nmqhexx2DgwUMFDO6bW2BvBlyH -Wyf5QBGenDPBt+U1VwV/J84XIIwc/PH72jEpSe31C4SnT8H2TsIonPru4K8H+zMR -eiFPCyEQtkA6qyI6BJyLm4SGcprSp6XEtHWRqSsjAgMBAAGjQjBAMA4GA1UdDwEB -/wQEAwIBBjAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBTgqj8ljZ9EXME66C6u -d0yEPmcM9DANBgkqhkiG9w0BAQsFAAOCAgEAuVevuBLaV4OPaAszHQNTVfSVcOQr -PbA56/qJYv331hgELyE03fFo8NWWWt7CgKPBjcZq91l3rhVkz1t5BXdm6ozTaw3d -8VkswTOlMIAVRQdFGjEitpIAq5lNOo93r6kiyi9jyhXWx8bwPWz8HA2YEGGeEaIi -1wrykXprOQ4vMMM2SZ/g6Q8CRFA3lFV96p/2O7qUpUzpvD5RtOjKkjZUbVwlKNrd -rRT90+7iIgXr0PK3aBLXWopBGsaSpVo7Y0VPv+E6dyIvXL9G+VoDhRNCX8reU9di -taY1BMJH/5n9hN9czulegChB8n3nHpDYT3Y+gjwN/KUD+nsa2UUeYNrEjvn8K8l7 -lcUq/6qJ34IxD3L/DCfXCh5WAFAeDJDBlrXYFIW7pw0WwfgHJBu6haEaBQmAupVj -yTrsJZ9/nbqkRxWbRHDxakvWOF5D8xh+UG7pWijmZeZ3Gzr9Hb4DJqPb1OG7fpYn -Kx3upPvaJVQTA945xsMfTZDsjxtK0hzthZU4UHlG1sGQUDGpXJpuHfUzVounmdLy -yCwzk5Iwx06MZTMQZBf9JBeW0Y3COmor6xOLRPIh80oat3df1+2IpHLlOR+Vnb5n -wXARPbv0+Em34yaXOp/SX3z7wJl8OSngex2/DaeP0ik0biQVy96QXr8axGbqwua6 -OV+KmalBWQewLK8= ------END CERTIFICATE----- - -# Issuer: CN=Certainly Root E1 O=Certainly -# Subject: CN=Certainly Root E1 O=Certainly -# Label: "Certainly Root E1" -# Serial: 8168531406727139161245376702891150584 -# MD5 Fingerprint: 0a:9e:ca:cd:3e:52:50:c6:36:f3:4b:a3:ed:a7:53:e9 -# SHA1 Fingerprint: f9:e1:6d:dc:01:89:cf:d5:82:45:63:3e:c5:37:7d:c2:eb:93:6f:2b -# SHA256 Fingerprint: b4:58:5f:22:e4:ac:75:6a:4e:86:12:a1:36:1c:5d:9d:03:1a:93:fd:84:fe:bb:77:8f:a3:06:8b:0f:c4:2d:c2 ------BEGIN CERTIFICATE----- -MIIB9zCCAX2gAwIBAgIQBiUzsUcDMydc+Y2aub/M+DAKBggqhkjOPQQDAzA9MQsw -CQYDVQQGEwJVUzESMBAGA1UEChMJQ2VydGFpbmx5MRowGAYDVQQDExFDZXJ0YWlu -bHkgUm9vdCBFMTAeFw0yMTA0MDEwMDAwMDBaFw00NjA0MDEwMDAwMDBaMD0xCzAJ -BgNVBAYTAlVTMRIwEAYDVQQKEwlDZXJ0YWlubHkxGjAYBgNVBAMTEUNlcnRhaW5s -eSBSb290IEUxMHYwEAYHKoZIzj0CAQYFK4EEACIDYgAE3m/4fxzf7flHh4axpMCK -+IKXgOqPyEpeKn2IaKcBYhSRJHpcnqMXfYqGITQYUBsQ3tA3SybHGWCA6TS9YBk2 -QNYphwk8kXr2vBMj3VlOBF7PyAIcGFPBMdjaIOlEjeR2o0IwQDAOBgNVHQ8BAf8E -BAMCAQYwDwYDVR0TAQH/BAUwAwEB/zAdBgNVHQ4EFgQU8ygYy2R17ikq6+2uI1g4 -hevIIgcwCgYIKoZIzj0EAwMDaAAwZQIxALGOWiDDshliTd6wT99u0nCK8Z9+aozm -ut6Dacpps6kFtZaSF4fC0urQe87YQVt8rgIwRt7qy12a7DLCZRawTDBcMPPaTnOG -BtjOiQRINzf43TNRnXCve1XYAS59BWQOhriR ------END CERTIFICATE----- - -# Issuer: CN=Security Communication RootCA3 O=SECOM Trust Systems CO.,LTD. -# Subject: CN=Security Communication RootCA3 O=SECOM Trust Systems CO.,LTD. -# Label: "Security Communication RootCA3" -# Serial: 16247922307909811815 -# MD5 Fingerprint: 1c:9a:16:ff:9e:5c:e0:4d:8a:14:01:f4:35:5d:29:26 -# SHA1 Fingerprint: c3:03:c8:22:74:92:e5:61:a2:9c:5f:79:91:2b:1e:44:13:91:30:3a -# SHA256 Fingerprint: 24:a5:5c:2a:b0:51:44:2d:06:17:76:65:41:23:9a:4a:d0:32:d7:c5:51:75:aa:34:ff:de:2f:bc:4f:5c:52:94 ------BEGIN CERTIFICATE----- -MIIFfzCCA2egAwIBAgIJAOF8N0D9G/5nMA0GCSqGSIb3DQEBDAUAMF0xCzAJBgNV -BAYTAkpQMSUwIwYDVQQKExxTRUNPTSBUcnVzdCBTeXN0ZW1zIENPLixMVEQuMScw -JQYDVQQDEx5TZWN1cml0eSBDb21tdW5pY2F0aW9uIFJvb3RDQTMwHhcNMTYwNjE2 -MDYxNzE2WhcNMzgwMTE4MDYxNzE2WjBdMQswCQYDVQQGEwJKUDElMCMGA1UEChMc -U0VDT00gVHJ1c3QgU3lzdGVtcyBDTy4sTFRELjEnMCUGA1UEAxMeU2VjdXJpdHkg -Q29tbXVuaWNhdGlvbiBSb290Q0EzMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIIC -CgKCAgEA48lySfcw3gl8qUCBWNO0Ot26YQ+TUG5pPDXC7ltzkBtnTCHsXzW7OT4r -CmDvu20rhvtxosis5FaU+cmvsXLUIKx00rgVrVH+hXShuRD+BYD5UpOzQD11EKzA -lrenfna84xtSGc4RHwsENPXY9Wk8d/Nk9A2qhd7gCVAEF5aEt8iKvE1y/By7z/MG -TfmfZPd+pmaGNXHIEYBMwXFAWB6+oHP2/D5Q4eAvJj1+XCO1eXDe+uDRpdYMQXF7 -9+qMHIjH7Iv10S9VlkZ8WjtYO/u62C21Jdp6Ts9EriGmnpjKIG58u4iFW/vAEGK7 -8vknR+/RiTlDxN/e4UG/VHMgly1s2vPUB6PmudhvrvyMGS7TZ2crldtYXLVqAvO4 -g160a75BflcJdURQVc1aEWEhCmHCqYj9E7wtiS/NYeCVvsq1e+F7NGcLH7YMx3we -GVPKp7FKFSBWFHA9K4IsD50VHUeAR/94mQ4xr28+j+2GaR57GIgUssL8gjMunEst -+3A7caoreyYn8xrC3PsXuKHqy6C0rtOUfnrQq8PsOC0RLoi/1D+tEjtCrI8Cbn3M -0V9hvqG8OmpI6iZVIhZdXw3/JzOfGAN0iltSIEdrRU0id4xVJ/CvHozJgyJUt5rQ -T9nO/NkuHJYosQLTA70lUhw0Zk8jq/R3gpYd0VcwCBEF/VfR2ccCAwEAAaNCMEAw -HQYDVR0OBBYEFGQUfPxYchamCik0FW8qy7z8r6irMA4GA1UdDwEB/wQEAwIBBjAP -BgNVHRMBAf8EBTADAQH/MA0GCSqGSIb3DQEBDAUAA4ICAQDcAiMI4u8hOscNtybS -YpOnpSNyByCCYN8Y11StaSWSntkUz5m5UoHPrmyKO1o5yGwBQ8IibQLwYs1OY0PA -FNr0Y/Dq9HHuTofjcan0yVflLl8cebsjqodEV+m9NU1Bu0soo5iyG9kLFwfl9+qd -9XbXv8S2gVj/yP9kaWJ5rW4OH3/uHWnlt3Jxs/6lATWUVCvAUm2PVcTJ0rjLyjQI -UYWg9by0F1jqClx6vWPGOi//lkkZhOpn2ASxYfQAW0q3nHE3GYV5v4GwxxMOdnE+ -OoAGrgYWp421wsTL/0ClXI2lyTrtcoHKXJg80jQDdwj98ClZXSEIx2C/pHF7uNke -gr4Jr2VvKKu/S7XuPghHJ6APbw+LP6yVGPO5DtxnVW5inkYO0QR4ynKudtml+LLf -iAlhi+8kTtFZP1rUPcmTPCtk9YENFpb3ksP+MW/oKjJ0DvRMmEoYDjBU1cXrvMUV -nuiZIesnKwkK2/HmcBhWuwzkvvnoEKQTkrgc4NtnHVMDpCKn3F2SEDzq//wbEBrD -2NCcnWXL0CsnMQMeNuE9dnUM/0Umud1RvCPHX9jYhxBAEg09ODfnRDwYwFMJZI// -1ZqmfHAuc1Uh6N//g7kdPjIe1qZ9LPFm6Vwdp6POXiUyK+OVrCoHzrQoeIY8Laad -TdJ0MN1kURXbg4NR16/9M51NZg== ------END CERTIFICATE----- - -# Issuer: CN=Security Communication ECC RootCA1 O=SECOM Trust Systems CO.,LTD. -# Subject: CN=Security Communication ECC RootCA1 O=SECOM Trust Systems CO.,LTD. -# Label: "Security Communication ECC RootCA1" -# Serial: 15446673492073852651 -# MD5 Fingerprint: 7e:43:b0:92:68:ec:05:43:4c:98:ab:5d:35:2e:7e:86 -# SHA1 Fingerprint: b8:0e:26:a9:bf:d2:b2:3b:c0:ef:46:c9:ba:c7:bb:f6:1d:0d:41:41 -# SHA256 Fingerprint: e7:4f:bd:a5:5b:d5:64:c4:73:a3:6b:44:1a:a7:99:c8:a6:8e:07:74:40:e8:28:8b:9f:a1:e5:0e:4b:ba:ca:11 ------BEGIN CERTIFICATE----- -MIICODCCAb6gAwIBAgIJANZdm7N4gS7rMAoGCCqGSM49BAMDMGExCzAJBgNVBAYT -AkpQMSUwIwYDVQQKExxTRUNPTSBUcnVzdCBTeXN0ZW1zIENPLixMVEQuMSswKQYD -VQQDEyJTZWN1cml0eSBDb21tdW5pY2F0aW9uIEVDQyBSb290Q0ExMB4XDTE2MDYx -NjA1MTUyOFoXDTM4MDExODA1MTUyOFowYTELMAkGA1UEBhMCSlAxJTAjBgNVBAoT -HFNFQ09NIFRydXN0IFN5c3RlbXMgQ08uLExURC4xKzApBgNVBAMTIlNlY3VyaXR5 -IENvbW11bmljYXRpb24gRUNDIFJvb3RDQTEwdjAQBgcqhkjOPQIBBgUrgQQAIgNi -AASkpW9gAwPDvTH00xecK4R1rOX9PVdu12O/5gSJko6BnOPpR27KkBLIE+Cnnfdl -dB9sELLo5OnvbYUymUSxXv3MdhDYW72ixvnWQuRXdtyQwjWpS4g8EkdtXP9JTxpK -ULGjQjBAMB0GA1UdDgQWBBSGHOf+LaVKiwj+KBH6vqNm+GBZLzAOBgNVHQ8BAf8E -BAMCAQYwDwYDVR0TAQH/BAUwAwEB/zAKBggqhkjOPQQDAwNoADBlAjAVXUI9/Lbu -9zuxNuie9sRGKEkz0FhDKmMpzE2xtHqiuQ04pV1IKv3LsnNdo4gIxwwCMQDAqy0O -be0YottT6SXbVQjgUMzfRGEWgqtJsLKB7HOHeLRMsmIbEvoWTSVLY70eN9k= ------END CERTIFICATE----- - -# Issuer: CN=BJCA Global Root CA1 O=BEIJING CERTIFICATE AUTHORITY -# Subject: CN=BJCA Global Root CA1 O=BEIJING CERTIFICATE AUTHORITY -# Label: "BJCA Global Root CA1" -# Serial: 113562791157148395269083148143378328608 -# MD5 Fingerprint: 42:32:99:76:43:33:36:24:35:07:82:9b:28:f9:d0:90 -# SHA1 Fingerprint: d5:ec:8d:7b:4c:ba:79:f4:e7:e8:cb:9d:6b:ae:77:83:10:03:21:6a -# SHA256 Fingerprint: f3:89:6f:88:fe:7c:0a:88:27:66:a7:fa:6a:d2:74:9f:b5:7a:7f:3e:98:fb:76:9c:1f:a7:b0:9c:2c:44:d5:ae ------BEGIN CERTIFICATE----- -MIIFdDCCA1ygAwIBAgIQVW9l47TZkGobCdFsPsBsIDANBgkqhkiG9w0BAQsFADBU -MQswCQYDVQQGEwJDTjEmMCQGA1UECgwdQkVJSklORyBDRVJUSUZJQ0FURSBBVVRI -T1JJVFkxHTAbBgNVBAMMFEJKQ0EgR2xvYmFsIFJvb3QgQ0ExMB4XDTE5MTIxOTAz -MTYxN1oXDTQ0MTIxMjAzMTYxN1owVDELMAkGA1UEBhMCQ04xJjAkBgNVBAoMHUJF -SUpJTkcgQ0VSVElGSUNBVEUgQVVUSE9SSVRZMR0wGwYDVQQDDBRCSkNBIEdsb2Jh -bCBSb290IENBMTCCAiIwDQYJKoZIhvcNAQEBBQADggIPADCCAgoCggIBAPFmCL3Z -xRVhy4QEQaVpN3cdwbB7+sN3SJATcmTRuHyQNZ0YeYjjlwE8R4HyDqKYDZ4/N+AZ -spDyRhySsTphzvq3Rp4Dhtczbu33RYx2N95ulpH3134rhxfVizXuhJFyV9xgw8O5 -58dnJCNPYwpj9mZ9S1WnP3hkSWkSl+BMDdMJoDIwOvqfwPKcxRIqLhy1BDPapDgR -at7GGPZHOiJBhyL8xIkoVNiMpTAK+BcWyqw3/XmnkRd4OJmtWO2y3syJfQOcs4ll -5+M7sSKGjwZteAf9kRJ/sGsciQ35uMt0WwfCyPQ10WRjeulumijWML3mG90Vr4Tq -nMfK9Q7q8l0ph49pczm+LiRvRSGsxdRpJQaDrXpIhRMsDQa4bHlW/KNnMoH1V6XK -V0Jp6VwkYe/iMBhORJhVb3rCk9gZtt58R4oRTklH2yiUAguUSiz5EtBP6DF+bHq/ -pj+bOT0CFqMYs2esWz8sgytnOYFcuX6U1WTdno9uruh8W7TXakdI136z1C2OVnZO -z2nxbkRs1CTqjSShGL+9V/6pmTW12xB3uD1IutbB5/EjPtffhZ0nPNRAvQoMvfXn -jSXWgXSHRtQpdaJCbPdzied9v3pKH9MiyRVVz99vfFXQpIsHETdfg6YmV6YBW37+ -WGgHqel62bno/1Afq8K0wM7o6v0PvY1NuLxxAgMBAAGjQjBAMB0GA1UdDgQWBBTF -7+3M2I0hxkjk49cULqcWk+WYATAPBgNVHRMBAf8EBTADAQH/MA4GA1UdDwEB/wQE -AwIBBjANBgkqhkiG9w0BAQsFAAOCAgEAUoKsITQfI/Ki2Pm4rzc2IInRNwPWaZ+4 -YRC6ojGYWUfo0Q0lHhVBDOAqVdVXUsv45Mdpox1NcQJeXyFFYEhcCY5JEMEE3Kli -awLwQ8hOnThJdMkycFRtwUf8jrQ2ntScvd0g1lPJGKm1Vrl2i5VnZu69mP6u775u -+2D2/VnGKhs/I0qUJDAnyIm860Qkmss9vk/Ves6OF8tiwdneHg56/0OGNFK8YT88 -X7vZdrRTvJez/opMEi4r89fO4aL/3Xtw+zuhTaRjAv04l5U/BXCga99igUOLtFkN -SoxUnMW7gZ/NfaXvCyUeOiDbHPwfmGcCCtRzRBPbUYQaVQNW4AB+dAb/OMRyHdOo -P2gxXdMJxy6MW2Pg6Nwe0uxhHvLe5e/2mXZgLR6UcnHGCyoyx5JO1UbXHfmpGQrI -+pXObSOYqgs4rZpWDW+N8TEAiMEXnM0ZNjX+VVOg4DwzX5Ze4jLp3zO7Bkqp2IRz -znfSxqxx4VyjHQy7Ct9f4qNx2No3WqB4K/TUfet27fJhcKVlmtOJNBir+3I+17Q9 -eVzYH6Eze9mCUAyTF6ps3MKCuwJXNq+YJyo5UOGwifUll35HaBC07HPKs5fRJNz2 -YqAo07WjuGS3iGJCz51TzZm+ZGiPTx4SSPfSKcOYKMryMguTjClPPGAyzQWWYezy -r/6zcCwupvI= ------END CERTIFICATE----- - -# Issuer: CN=BJCA Global Root CA2 O=BEIJING CERTIFICATE AUTHORITY -# Subject: CN=BJCA Global Root CA2 O=BEIJING CERTIFICATE AUTHORITY -# Label: "BJCA Global Root CA2" -# Serial: 58605626836079930195615843123109055211 -# MD5 Fingerprint: 5e:0a:f6:47:5f:a6:14:e8:11:01:95:3f:4d:01:eb:3c -# SHA1 Fingerprint: f4:27:86:eb:6e:b8:6d:88:31:67:02:fb:ba:66:a4:53:00:aa:7a:a6 -# SHA256 Fingerprint: 57:4d:f6:93:1e:27:80:39:66:7b:72:0a:fd:c1:60:0f:c2:7e:b6:6d:d3:09:29:79:fb:73:85:64:87:21:28:82 ------BEGIN CERTIFICATE----- -MIICJTCCAaugAwIBAgIQLBcIfWQqwP6FGFkGz7RK6zAKBggqhkjOPQQDAzBUMQsw -CQYDVQQGEwJDTjEmMCQGA1UECgwdQkVJSklORyBDRVJUSUZJQ0FURSBBVVRIT1JJ -VFkxHTAbBgNVBAMMFEJKQ0EgR2xvYmFsIFJvb3QgQ0EyMB4XDTE5MTIxOTAzMTgy -MVoXDTQ0MTIxMjAzMTgyMVowVDELMAkGA1UEBhMCQ04xJjAkBgNVBAoMHUJFSUpJ -TkcgQ0VSVElGSUNBVEUgQVVUSE9SSVRZMR0wGwYDVQQDDBRCSkNBIEdsb2JhbCBS -b290IENBMjB2MBAGByqGSM49AgEGBSuBBAAiA2IABJ3LgJGNU2e1uVCxA/jlSR9B -IgmwUVJY1is0j8USRhTFiy8shP8sbqjV8QnjAyEUxEM9fMEsxEtqSs3ph+B99iK+ -+kpRuDCK/eHeGBIK9ke35xe/J4rUQUyWPGCWwf0VHKNCMEAwHQYDVR0OBBYEFNJK -sVF/BvDRgh9Obl+rg/xI1LCRMA8GA1UdEwEB/wQFMAMBAf8wDgYDVR0PAQH/BAQD -AgEGMAoGCCqGSM49BAMDA2gAMGUCMBq8W9f+qdJUDkpd0m2xQNz0Q9XSSpkZElaA -94M04TVOSG0ED1cxMDAtsaqdAzjbBgIxAMvMh1PLet8gUXOQwKhbYdDFUDn9hf7B -43j4ptZLvZuHjw/l1lOWqzzIQNph91Oj9w== ------END CERTIFICATE----- - -# Issuer: CN=Sectigo Public Server Authentication Root E46 O=Sectigo Limited -# Subject: CN=Sectigo Public Server Authentication Root E46 O=Sectigo Limited -# Label: "Sectigo Public Server Authentication Root E46" -# Serial: 88989738453351742415770396670917916916 -# MD5 Fingerprint: 28:23:f8:b2:98:5c:37:16:3b:3e:46:13:4e:b0:b3:01 -# SHA1 Fingerprint: ec:8a:39:6c:40:f0:2e:bc:42:75:d4:9f:ab:1c:1a:5b:67:be:d2:9a -# SHA256 Fingerprint: c9:0f:26:f0:fb:1b:40:18:b2:22:27:51:9b:5c:a2:b5:3e:2c:a5:b3:be:5c:f1:8e:fe:1b:ef:47:38:0c:53:83 ------BEGIN CERTIFICATE----- -MIICOjCCAcGgAwIBAgIQQvLM2htpN0RfFf51KBC49DAKBggqhkjOPQQDAzBfMQsw -CQYDVQQGEwJHQjEYMBYGA1UEChMPU2VjdGlnbyBMaW1pdGVkMTYwNAYDVQQDEy1T -ZWN0aWdvIFB1YmxpYyBTZXJ2ZXIgQXV0aGVudGljYXRpb24gUm9vdCBFNDYwHhcN -MjEwMzIyMDAwMDAwWhcNNDYwMzIxMjM1OTU5WjBfMQswCQYDVQQGEwJHQjEYMBYG -A1UEChMPU2VjdGlnbyBMaW1pdGVkMTYwNAYDVQQDEy1TZWN0aWdvIFB1YmxpYyBT -ZXJ2ZXIgQXV0aGVudGljYXRpb24gUm9vdCBFNDYwdjAQBgcqhkjOPQIBBgUrgQQA -IgNiAAR2+pmpbiDt+dd34wc7qNs9Xzjoq1WmVk/WSOrsfy2qw7LFeeyZYX8QeccC -WvkEN/U0NSt3zn8gj1KjAIns1aeibVvjS5KToID1AZTc8GgHHs3u/iVStSBDHBv+ -6xnOQ6OjQjBAMB0GA1UdDgQWBBTRItpMWfFLXyY4qp3W7usNw/upYTAOBgNVHQ8B -Af8EBAMCAYYwDwYDVR0TAQH/BAUwAwEB/zAKBggqhkjOPQQDAwNnADBkAjAn7qRa -qCG76UeXlImldCBteU/IvZNeWBj7LRoAasm4PdCkT0RHlAFWovgzJQxC36oCMB3q -4S6ILuH5px0CMk7yn2xVdOOurvulGu7t0vzCAxHrRVxgED1cf5kDW21USAGKcw== ------END CERTIFICATE----- - -# Issuer: CN=Sectigo Public Server Authentication Root R46 O=Sectigo Limited -# Subject: CN=Sectigo Public Server Authentication Root R46 O=Sectigo Limited -# Label: "Sectigo Public Server Authentication Root R46" -# Serial: 156256931880233212765902055439220583700 -# MD5 Fingerprint: 32:10:09:52:00:d5:7e:6c:43:df:15:c0:b1:16:93:e5 -# SHA1 Fingerprint: ad:98:f9:f3:e4:7d:75:3b:65:d4:82:b3:a4:52:17:bb:6e:f5:e4:38 -# SHA256 Fingerprint: 7b:b6:47:a6:2a:ee:ac:88:bf:25:7a:a5:22:d0:1f:fe:a3:95:e0:ab:45:c7:3f:93:f6:56:54:ec:38:f2:5a:06 ------BEGIN CERTIFICATE----- -MIIFijCCA3KgAwIBAgIQdY39i658BwD6qSWn4cetFDANBgkqhkiG9w0BAQwFADBf -MQswCQYDVQQGEwJHQjEYMBYGA1UEChMPU2VjdGlnbyBMaW1pdGVkMTYwNAYDVQQD -Ey1TZWN0aWdvIFB1YmxpYyBTZXJ2ZXIgQXV0aGVudGljYXRpb24gUm9vdCBSNDYw -HhcNMjEwMzIyMDAwMDAwWhcNNDYwMzIxMjM1OTU5WjBfMQswCQYDVQQGEwJHQjEY -MBYGA1UEChMPU2VjdGlnbyBMaW1pdGVkMTYwNAYDVQQDEy1TZWN0aWdvIFB1Ymxp -YyBTZXJ2ZXIgQXV0aGVudGljYXRpb24gUm9vdCBSNDYwggIiMA0GCSqGSIb3DQEB -AQUAA4ICDwAwggIKAoICAQCTvtU2UnXYASOgHEdCSe5jtrch/cSV1UgrJnwUUxDa -ef0rty2k1Cz66jLdScK5vQ9IPXtamFSvnl0xdE8H/FAh3aTPaE8bEmNtJZlMKpnz -SDBh+oF8HqcIStw+KxwfGExxqjWMrfhu6DtK2eWUAtaJhBOqbchPM8xQljeSM9xf -iOefVNlI8JhD1mb9nxc4Q8UBUQvX4yMPFF1bFOdLvt30yNoDN9HWOaEhUTCDsG3X -ME6WW5HwcCSrv0WBZEMNvSE6Lzzpng3LILVCJ8zab5vuZDCQOc2TZYEhMbUjUDM3 -IuM47fgxMMxF/mL50V0yeUKH32rMVhlATc6qu/m1dkmU8Sf4kaWD5QazYw6A3OAS -VYCmO2a0OYctyPDQ0RTp5A1NDvZdV3LFOxxHVp3i1fuBYYzMTYCQNFu31xR13NgE -SJ/AwSiItOkcyqex8Va3e0lMWeUgFaiEAin6OJRpmkkGj80feRQXEgyDet4fsZfu -+Zd4KKTIRJLpfSYFplhym3kT2BFfrsU4YjRosoYwjviQYZ4ybPUHNs2iTG7sijbt -8uaZFURww3y8nDnAtOFr94MlI1fZEoDlSfB1D++N6xybVCi0ITz8fAr/73trdf+L -HaAZBav6+CuBQug4urv7qv094PPK306Xlynt8xhW6aWWrL3DkJiy4Pmi1KZHQ3xt -zwIDAQABo0IwQDAdBgNVHQ4EFgQUVnNYZJX5khqwEioEYnmhQBWIIUkwDgYDVR0P -AQH/BAQDAgGGMA8GA1UdEwEB/wQFMAMBAf8wDQYJKoZIhvcNAQEMBQADggIBAC9c -mTz8Bl6MlC5w6tIyMY208FHVvArzZJ8HXtXBc2hkeqK5Duj5XYUtqDdFqij0lgVQ -YKlJfp/imTYpE0RHap1VIDzYm/EDMrraQKFz6oOht0SmDpkBm+S8f74TlH7Kph52 -gDY9hAaLMyZlbcp+nv4fjFg4exqDsQ+8FxG75gbMY/qB8oFM2gsQa6H61SilzwZA -Fv97fRheORKkU55+MkIQpiGRqRxOF3yEvJ+M0ejf5lG5Nkc/kLnHvALcWxxPDkjB -JYOcCj+esQMzEhonrPcibCTRAUH4WAP+JWgiH5paPHxsnnVI84HxZmduTILA7rpX -DhjvLpr3Etiga+kFpaHpaPi8TD8SHkXoUsCjvxInebnMMTzD9joiFgOgyY9mpFui -TdaBJQbpdqQACj7LzTWb4OE4y2BThihCQRxEV+ioratF4yUQvNs+ZUH7G6aXD+u5 -dHn5HrwdVw1Hr8Mvn4dGp+smWg9WY7ViYG4A++MnESLn/pmPNPW56MORcr3Ywx65 -LvKRRFHQV80MNNVIIb/bE/FmJUNS0nAiNs2fxBx1IK1jcmMGDw4nztJqDby1ORrp -0XZ60Vzk50lJLVU3aPAaOpg+VBeHVOmmJ1CJeyAvP/+/oYtKR5j/K3tJPsMpRmAY -QqszKbrAKbkTidOIijlBO8n9pu0f9GBj39ItVQGL ------END CERTIFICATE----- - -# Issuer: CN=SSL.com TLS RSA Root CA 2022 O=SSL Corporation -# Subject: CN=SSL.com TLS RSA Root CA 2022 O=SSL Corporation -# Label: "SSL.com TLS RSA Root CA 2022" -# Serial: 148535279242832292258835760425842727825 -# MD5 Fingerprint: d8:4e:c6:59:30:d8:fe:a0:d6:7a:5a:2c:2c:69:78:da -# SHA1 Fingerprint: ec:2c:83:40:72:af:26:95:10:ff:0e:f2:03:ee:31:70:f6:78:9d:ca -# SHA256 Fingerprint: 8f:af:7d:2e:2c:b4:70:9b:b8:e0:b3:36:66:bf:75:a5:dd:45:b5:de:48:0f:8e:a8:d4:bf:e6:be:bc:17:f2:ed ------BEGIN CERTIFICATE----- -MIIFiTCCA3GgAwIBAgIQb77arXO9CEDii02+1PdbkTANBgkqhkiG9w0BAQsFADBO -MQswCQYDVQQGEwJVUzEYMBYGA1UECgwPU1NMIENvcnBvcmF0aW9uMSUwIwYDVQQD -DBxTU0wuY29tIFRMUyBSU0EgUm9vdCBDQSAyMDIyMB4XDTIyMDgyNTE2MzQyMloX -DTQ2MDgxOTE2MzQyMVowTjELMAkGA1UEBhMCVVMxGDAWBgNVBAoMD1NTTCBDb3Jw -b3JhdGlvbjElMCMGA1UEAwwcU1NMLmNvbSBUTFMgUlNBIFJvb3QgQ0EgMjAyMjCC -AiIwDQYJKoZIhvcNAQEBBQADggIPADCCAgoCggIBANCkCXJPQIgSYT41I57u9nTP -L3tYPc48DRAokC+X94xI2KDYJbFMsBFMF3NQ0CJKY7uB0ylu1bUJPiYYf7ISf5OY -t6/wNr/y7hienDtSxUcZXXTzZGbVXcdotL8bHAajvI9AI7YexoS9UcQbOcGV0ins -S657Lb85/bRi3pZ7QcacoOAGcvvwB5cJOYF0r/c0WRFXCsJbwST0MXMwgsadugL3 -PnxEX4MN8/HdIGkWCVDi1FW24IBydm5MR7d1VVm0U3TZlMZBrViKMWYPHqIbKUBO -L9975hYsLfy/7PO0+r4Y9ptJ1O4Fbtk085zx7AGL0SDGD6C1vBdOSHtRwvzpXGk3 -R2azaPgVKPC506QVzFpPulJwoxJF3ca6TvvC0PeoUidtbnm1jPx7jMEWTO6Af77w -dr5BUxIzrlo4QqvXDz5BjXYHMtWrifZOZ9mxQnUjbvPNQrL8VfVThxc7wDNY8VLS -+YCk8OjwO4s4zKTGkH8PnP2L0aPP2oOnaclQNtVcBdIKQXTbYxE3waWglksejBYS -d66UNHsef8JmAOSqg+qKkK3ONkRN0VHpvB/zagX9wHQfJRlAUW7qglFA35u5CCoG -AtUjHBPW6dvbxrB6y3snm/vg1UYk7RBLY0ulBY+6uB0rpvqR4pJSvezrZ5dtmi2f -gTIFZzL7SAg/2SW4BCUvAgMBAAGjYzBhMA8GA1UdEwEB/wQFMAMBAf8wHwYDVR0j -BBgwFoAU+y437uOEeicuzRk1sTN8/9REQrkwHQYDVR0OBBYEFPsuN+7jhHonLs0Z -NbEzfP/UREK5MA4GA1UdDwEB/wQEAwIBhjANBgkqhkiG9w0BAQsFAAOCAgEAjYlt -hEUY8U+zoO9opMAdrDC8Z2awms22qyIZZtM7QbUQnRC6cm4pJCAcAZli05bg4vsM -QtfhWsSWTVTNj8pDU/0quOr4ZcoBwq1gaAafORpR2eCNJvkLTqVTJXojpBzOCBvf -R4iyrT7gJ4eLSYwfqUdYe5byiB0YrrPRpgqU+tvT5TgKa3kSM/tKWTcWQA673vWJ -DPFs0/dRa1419dvAJuoSc06pkZCmF8NsLzjUo3KUQyxi4U5cMj29TH0ZR6LDSeeW -P4+a0zvkEdiLA9z2tmBVGKaBUfPhqBVq6+AL8BQx1rmMRTqoENjwuSfr98t67wVy -lrXEj5ZzxOhWc5y8aVFjvO9nHEMaX3cZHxj4HCUp+UmZKbaSPaKDN7EgkaibMOlq -bLQjk2UEqxHzDh1TJElTHaE/nUiSEeJ9DU/1172iWD54nR4fK/4huxoTtrEoZP2w -AgDHbICivRZQIA9ygV/MlP+7mea6kMvq+cYMwq7FGc4zoWtcu358NFcXrfA/rs3q -r5nsLFR+jM4uElZI7xc7P0peYNLcdDa8pUNjyw9bowJWCZ4kLOGGgYz+qxcs+sji -Mho6/4UIyYOf8kpIEFR3N+2ivEC+5BB09+Rbu7nzifmPQdjH5FCQNYA+HLhNkNPU -98OwoX6EyneSMSy4kLGCenROmxMmtNVQZlR4rmA= ------END CERTIFICATE----- - -# Issuer: CN=SSL.com TLS ECC Root CA 2022 O=SSL Corporation -# Subject: CN=SSL.com TLS ECC Root CA 2022 O=SSL Corporation -# Label: "SSL.com TLS ECC Root CA 2022" -# Serial: 26605119622390491762507526719404364228 -# MD5 Fingerprint: 99:d7:5c:f1:51:36:cc:e9:ce:d9:19:2e:77:71:56:c5 -# SHA1 Fingerprint: 9f:5f:d9:1a:54:6d:f5:0c:71:f0:ee:7a:bd:17:49:98:84:73:e2:39 -# SHA256 Fingerprint: c3:2f:fd:9f:46:f9:36:d1:6c:36:73:99:09:59:43:4b:9a:d6:0a:af:bb:9e:7c:f3:36:54:f1:44:cc:1b:a1:43 ------BEGIN CERTIFICATE----- -MIICOjCCAcCgAwIBAgIQFAP1q/s3ixdAW+JDsqXRxDAKBggqhkjOPQQDAzBOMQsw -CQYDVQQGEwJVUzEYMBYGA1UECgwPU1NMIENvcnBvcmF0aW9uMSUwIwYDVQQDDBxT -U0wuY29tIFRMUyBFQ0MgUm9vdCBDQSAyMDIyMB4XDTIyMDgyNTE2MzM0OFoXDTQ2 -MDgxOTE2MzM0N1owTjELMAkGA1UEBhMCVVMxGDAWBgNVBAoMD1NTTCBDb3Jwb3Jh -dGlvbjElMCMGA1UEAwwcU1NMLmNvbSBUTFMgRUNDIFJvb3QgQ0EgMjAyMjB2MBAG -ByqGSM49AgEGBSuBBAAiA2IABEUpNXP6wrgjzhR9qLFNoFs27iosU8NgCTWyJGYm -acCzldZdkkAZDsalE3D07xJRKF3nzL35PIXBz5SQySvOkkJYWWf9lCcQZIxPBLFN -SeR7T5v15wj4A4j3p8OSSxlUgaNjMGEwDwYDVR0TAQH/BAUwAwEB/zAfBgNVHSME -GDAWgBSJjy+j6CugFFR781a4Jl9nOAuc0DAdBgNVHQ4EFgQUiY8vo+groBRUe/NW -uCZfZzgLnNAwDgYDVR0PAQH/BAQDAgGGMAoGCCqGSM49BAMDA2gAMGUCMFXjIlbp -15IkWE8elDIPDAI2wv2sdDJO4fscgIijzPvX6yv/N33w7deedWo1dlJF4AIxAMeN -b0Igj762TVntd00pxCAgRWSGOlDGxK0tk/UYfXLtqc/ErFc2KAhl3zx5Zn6g6g== ------END CERTIFICATE----- - -# Issuer: CN=Atos TrustedRoot Root CA ECC TLS 2021 O=Atos -# Subject: CN=Atos TrustedRoot Root CA ECC TLS 2021 O=Atos -# Label: "Atos TrustedRoot Root CA ECC TLS 2021" -# Serial: 81873346711060652204712539181482831616 -# MD5 Fingerprint: 16:9f:ad:f1:70:ad:79:d6:ed:29:b4:d1:c5:79:70:a8 -# SHA1 Fingerprint: 9e:bc:75:10:42:b3:02:f3:81:f4:f7:30:62:d4:8f:c3:a7:51:b2:dd -# SHA256 Fingerprint: b2:fa:e5:3e:14:cc:d7:ab:92:12:06:47:01:ae:27:9c:1d:89:88:fa:cb:77:5f:a8:a0:08:91:4e:66:39:88:a8 ------BEGIN CERTIFICATE----- -MIICFTCCAZugAwIBAgIQPZg7pmY9kGP3fiZXOATvADAKBggqhkjOPQQDAzBMMS4w -LAYDVQQDDCVBdG9zIFRydXN0ZWRSb290IFJvb3QgQ0EgRUNDIFRMUyAyMDIxMQ0w -CwYDVQQKDARBdG9zMQswCQYDVQQGEwJERTAeFw0yMTA0MjIwOTI2MjNaFw00MTA0 -MTcwOTI2MjJaMEwxLjAsBgNVBAMMJUF0b3MgVHJ1c3RlZFJvb3QgUm9vdCBDQSBF -Q0MgVExTIDIwMjExDTALBgNVBAoMBEF0b3MxCzAJBgNVBAYTAkRFMHYwEAYHKoZI -zj0CAQYFK4EEACIDYgAEloZYKDcKZ9Cg3iQZGeHkBQcfl+3oZIK59sRxUM6KDP/X -tXa7oWyTbIOiaG6l2b4siJVBzV3dscqDY4PMwL502eCdpO5KTlbgmClBk1IQ1SQ4 -AjJn8ZQSb+/Xxd4u/RmAo0IwQDAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBR2 -KCXWfeBmmnoJsmo7jjPXNtNPojAOBgNVHQ8BAf8EBAMCAYYwCgYIKoZIzj0EAwMD -aAAwZQIwW5kp85wxtolrbNa9d+F851F+uDrNozZffPc8dz7kUK2o59JZDCaOMDtu -CCrCp1rIAjEAmeMM56PDr9NJLkaCI2ZdyQAUEv049OGYa3cpetskz2VAv9LcjBHo -9H1/IISpQuQo ------END CERTIFICATE----- - -# Issuer: CN=Atos TrustedRoot Root CA RSA TLS 2021 O=Atos -# Subject: CN=Atos TrustedRoot Root CA RSA TLS 2021 O=Atos -# Label: "Atos TrustedRoot Root CA RSA TLS 2021" -# Serial: 111436099570196163832749341232207667876 -# MD5 Fingerprint: d4:d3:46:b8:9a:c0:9c:76:5d:9e:3a:c3:b9:99:31:d2 -# SHA1 Fingerprint: 18:52:3b:0d:06:37:e4:d6:3a:df:23:e4:98:fb:5b:16:fb:86:74:48 -# SHA256 Fingerprint: 81:a9:08:8e:a5:9f:b3:64:c5:48:a6:f8:55:59:09:9b:6f:04:05:ef:bf:18:e5:32:4e:c9:f4:57:ba:00:11:2f ------BEGIN CERTIFICATE----- -MIIFZDCCA0ygAwIBAgIQU9XP5hmTC/srBRLYwiqipDANBgkqhkiG9w0BAQwFADBM -MS4wLAYDVQQDDCVBdG9zIFRydXN0ZWRSb290IFJvb3QgQ0EgUlNBIFRMUyAyMDIx -MQ0wCwYDVQQKDARBdG9zMQswCQYDVQQGEwJERTAeFw0yMTA0MjIwOTIxMTBaFw00 -MTA0MTcwOTIxMDlaMEwxLjAsBgNVBAMMJUF0b3MgVHJ1c3RlZFJvb3QgUm9vdCBD -QSBSU0EgVExTIDIwMjExDTALBgNVBAoMBEF0b3MxCzAJBgNVBAYTAkRFMIICIjAN -BgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAtoAOxHm9BYx9sKOdTSJNy/BBl01Z -4NH+VoyX8te9j2y3I49f1cTYQcvyAh5x5en2XssIKl4w8i1mx4QbZFc4nXUtVsYv -Ye+W/CBGvevUez8/fEc4BKkbqlLfEzfTFRVOvV98r61jx3ncCHvVoOX3W3WsgFWZ -kmGbzSoXfduP9LVq6hdKZChmFSlsAvFr1bqjM9xaZ6cF4r9lthawEO3NUDPJcFDs -GY6wx/J0W2tExn2WuZgIWWbeKQGb9Cpt0xU6kGpn8bRrZtkh68rZYnxGEFzedUln -nkL5/nWpo63/dgpnQOPF943HhZpZnmKaau1Fh5hnstVKPNe0OwANwI8f4UDErmwh -3El+fsqyjW22v5MvoVw+j8rtgI5Y4dtXz4U2OLJxpAmMkokIiEjxQGMYsluMWuPD -0xeqqxmjLBvk1cbiZnrXghmmOxYsL3GHX0WelXOTwkKBIROW1527k2gV+p2kHYzy -geBYBr3JtuP2iV2J+axEoctr+hbxx1A9JNr3w+SH1VbxT5Aw+kUJWdo0zuATHAR8 -ANSbhqRAvNncTFd+rrcztl524WWLZt+NyteYr842mIycg5kDcPOvdO3GDjbnvezB -c6eUWsuSZIKmAMFwoW4sKeFYV+xafJlrJaSQOoD0IJ2azsct+bJLKZWD6TWNp0lI -pw9MGZHQ9b8Q4HECAwEAAaNCMEAwDwYDVR0TAQH/BAUwAwEB/zAdBgNVHQ4EFgQU -dEmZ0f+0emhFdcN+tNzMzjkz2ggwDgYDVR0PAQH/BAQDAgGGMA0GCSqGSIb3DQEB -DAUAA4ICAQAjQ1MkYlxt/T7Cz1UAbMVWiLkO3TriJQ2VSpfKgInuKs1l+NsW4AmS -4BjHeJi78+xCUvuppILXTdiK/ORO/auQxDh1MoSf/7OwKwIzNsAQkG8dnK/haZPs -o0UvFJ/1TCplQ3IM98P4lYsU84UgYt1UU90s3BiVaU+DR3BAM1h3Egyi61IxHkzJ -qM7F78PRreBrAwA0JrRUITWXAdxfG/F851X6LWh3e9NpzNMOa7pNdkTWwhWaJuyw -xfW70Xp0wmzNxbVe9kzmWy2B27O3Opee7c9GslA9hGCZcbUztVdF5kJHdWoOsAgM -rr3e97sPWD2PAzHoPYJQyi9eDF20l74gNAf0xBLh7tew2VktafcxBPTy+av5EzH4 -AXcOPUIjJsyacmdRIXrMPIWo6iFqO9taPKU0nprALN+AnCng33eU0aKAQv9qTFsR -0PXNor6uzFFcw9VUewyu1rkGd4Di7wcaaMxZUa1+XGdrudviB0JbuAEFWDlN5LuY -o7Ey7Nmj1m+UI/87tyll5gfp77YZ6ufCOB0yiJA8EytuzO+rdwY0d4RPcuSBhPm5 -dDTedk+SKlOxJTnbPP/lPqYO5Wue/9vsL3SD3460s6neFE3/MaNFcyT6lSnMEpcE -oji2jbDwN/zIIX8/syQbPYtuzE2wFg2WHYMfRsCbvUOZ58SWLs5fyQ== ------END CERTIFICATE----- diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/certifi/core.py b/venv/lib/python3.11/site-packages/pip/_vendor/certifi/core.py deleted file mode 100644 index c3e5466..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/certifi/core.py +++ /dev/null @@ -1,108 +0,0 @@ -""" -certifi.py -~~~~~~~~~~ - -This module returns the installation location of cacert.pem or its contents. -""" -import sys - - -if sys.version_info >= (3, 11): - - from importlib.resources import as_file, files - - _CACERT_CTX = None - _CACERT_PATH = None - - def where() -> str: - # This is slightly terrible, but we want to delay extracting the file - # in cases where we're inside of a zipimport situation until someone - # actually calls where(), but we don't want to re-extract the file - # on every call of where(), so we'll do it once then store it in a - # global variable. - global _CACERT_CTX - global _CACERT_PATH - if _CACERT_PATH is None: - # This is slightly janky, the importlib.resources API wants you to - # manage the cleanup of this file, so it doesn't actually return a - # path, it returns a context manager that will give you the path - # when you enter it and will do any cleanup when you leave it. In - # the common case of not needing a temporary file, it will just - # return the file system location and the __exit__() is a no-op. - # - # We also have to hold onto the actual context manager, because - # it will do the cleanup whenever it gets garbage collected, so - # we will also store that at the global level as well. - _CACERT_CTX = as_file(files("pip._vendor.certifi").joinpath("cacert.pem")) - _CACERT_PATH = str(_CACERT_CTX.__enter__()) - - return _CACERT_PATH - - def contents() -> str: - return files("pip._vendor.certifi").joinpath("cacert.pem").read_text(encoding="ascii") - -elif sys.version_info >= (3, 7): - - from importlib.resources import path as get_path, read_text - - _CACERT_CTX = None - _CACERT_PATH = None - - def where() -> str: - # This is slightly terrible, but we want to delay extracting the - # file in cases where we're inside of a zipimport situation until - # someone actually calls where(), but we don't want to re-extract - # the file on every call of where(), so we'll do it once then store - # it in a global variable. - global _CACERT_CTX - global _CACERT_PATH - if _CACERT_PATH is None: - # This is slightly janky, the importlib.resources API wants you - # to manage the cleanup of this file, so it doesn't actually - # return a path, it returns a context manager that will give - # you the path when you enter it and will do any cleanup when - # you leave it. In the common case of not needing a temporary - # file, it will just return the file system location and the - # __exit__() is a no-op. - # - # We also have to hold onto the actual context manager, because - # it will do the cleanup whenever it gets garbage collected, so - # we will also store that at the global level as well. - _CACERT_CTX = get_path("pip._vendor.certifi", "cacert.pem") - _CACERT_PATH = str(_CACERT_CTX.__enter__()) - - return _CACERT_PATH - - def contents() -> str: - return read_text("pip._vendor.certifi", "cacert.pem", encoding="ascii") - -else: - import os - import types - from typing import Union - - Package = Union[types.ModuleType, str] - Resource = Union[str, "os.PathLike"] - - # This fallback will work for Python versions prior to 3.7 that lack the - # importlib.resources module but relies on the existing `where` function - # so won't address issues with environments like PyOxidizer that don't set - # __file__ on modules. - def read_text( - package: Package, - resource: Resource, - encoding: str = 'utf-8', - errors: str = 'strict' - ) -> str: - with open(where(), encoding=encoding) as data: - return data.read() - - # If we don't have importlib.resources, then we will just do the old logic - # of assuming we're on the filesystem and munge the path directly. - def where() -> str: - f = os.path.dirname(__file__) - - return os.path.join(f, "cacert.pem") - - def contents() -> str: - return read_text("pip._vendor.certifi", "cacert.pem", encoding="ascii") diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__init__.py b/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__init__.py deleted file mode 100644 index fe58162..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__init__.py +++ /dev/null @@ -1,115 +0,0 @@ -######################## BEGIN LICENSE BLOCK ######################## -# This library is free software; you can redistribute it and/or -# modify it under the terms of the GNU Lesser General Public -# License as published by the Free Software Foundation; either -# version 2.1 of the License, or (at your option) any later version. -# -# This library is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# Lesser General Public License for more details. -# -# You should have received a copy of the GNU Lesser General Public -# License along with this library; if not, write to the Free Software -# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA -# 02110-1301 USA -######################### END LICENSE BLOCK ######################### - -from typing import List, Union - -from .charsetgroupprober import CharSetGroupProber -from .charsetprober import CharSetProber -from .enums import InputState -from .resultdict import ResultDict -from .universaldetector import UniversalDetector -from .version import VERSION, __version__ - -__all__ = ["UniversalDetector", "detect", "detect_all", "__version__", "VERSION"] - - -def detect( - byte_str: Union[bytes, bytearray], should_rename_legacy: bool = False -) -> ResultDict: - """ - Detect the encoding of the given byte string. - - :param byte_str: The byte sequence to examine. - :type byte_str: ``bytes`` or ``bytearray`` - :param should_rename_legacy: Should we rename legacy encodings - to their more modern equivalents? - :type should_rename_legacy: ``bool`` - """ - if not isinstance(byte_str, bytearray): - if not isinstance(byte_str, bytes): - raise TypeError( - f"Expected object of type bytes or bytearray, got: {type(byte_str)}" - ) - byte_str = bytearray(byte_str) - detector = UniversalDetector(should_rename_legacy=should_rename_legacy) - detector.feed(byte_str) - return detector.close() - - -def detect_all( - byte_str: Union[bytes, bytearray], - ignore_threshold: bool = False, - should_rename_legacy: bool = False, -) -> List[ResultDict]: - """ - Detect all the possible encodings of the given byte string. - - :param byte_str: The byte sequence to examine. - :type byte_str: ``bytes`` or ``bytearray`` - :param ignore_threshold: Include encodings that are below - ``UniversalDetector.MINIMUM_THRESHOLD`` - in results. - :type ignore_threshold: ``bool`` - :param should_rename_legacy: Should we rename legacy encodings - to their more modern equivalents? - :type should_rename_legacy: ``bool`` - """ - if not isinstance(byte_str, bytearray): - if not isinstance(byte_str, bytes): - raise TypeError( - f"Expected object of type bytes or bytearray, got: {type(byte_str)}" - ) - byte_str = bytearray(byte_str) - - detector = UniversalDetector(should_rename_legacy=should_rename_legacy) - detector.feed(byte_str) - detector.close() - - if detector.input_state == InputState.HIGH_BYTE: - results: List[ResultDict] = [] - probers: List[CharSetProber] = [] - for prober in detector.charset_probers: - if isinstance(prober, CharSetGroupProber): - probers.extend(p for p in prober.probers) - else: - probers.append(prober) - for prober in probers: - if ignore_threshold or prober.get_confidence() > detector.MINIMUM_THRESHOLD: - charset_name = prober.charset_name or "" - lower_charset_name = charset_name.lower() - # Use Windows encoding name instead of ISO-8859 if we saw any - # extra Windows-specific bytes - if lower_charset_name.startswith("iso-8859") and detector.has_win_bytes: - charset_name = detector.ISO_WIN_MAP.get( - lower_charset_name, charset_name - ) - # Rename legacy encodings with superset encodings if asked - if should_rename_legacy: - charset_name = detector.LEGACY_MAP.get( - charset_name.lower(), charset_name - ) - results.append( - { - "encoding": charset_name, - "confidence": prober.get_confidence(), - "language": prober.language, - } - ) - if len(results) > 0: - return sorted(results, key=lambda result: -result["confidence"]) - - return [detector.result] diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/__init__.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/__init__.cpython-311.pyc deleted file mode 100644 index 3a8764e..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/__init__.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/big5freq.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/big5freq.cpython-311.pyc deleted file mode 100644 index c018db3..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/big5freq.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/big5prober.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/big5prober.cpython-311.pyc deleted file mode 100644 index bdfa36a..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/big5prober.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/chardistribution.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/chardistribution.cpython-311.pyc deleted file mode 100644 index b3df3a1..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/chardistribution.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/charsetgroupprober.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/charsetgroupprober.cpython-311.pyc deleted file mode 100644 index 07858ac..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/charsetgroupprober.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/charsetprober.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/charsetprober.cpython-311.pyc deleted file mode 100644 index 1ebd6ce..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/charsetprober.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/codingstatemachine.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/codingstatemachine.cpython-311.pyc deleted file mode 100644 index ad67762..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/codingstatemachine.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/codingstatemachinedict.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/codingstatemachinedict.cpython-311.pyc deleted file mode 100644 index 423e610..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/codingstatemachinedict.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/cp949prober.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/cp949prober.cpython-311.pyc deleted file mode 100644 index 8d5a988..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/cp949prober.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/enums.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/enums.cpython-311.pyc deleted file mode 100644 index 5dc1cf2..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/enums.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/escprober.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/escprober.cpython-311.pyc deleted file mode 100644 index 8107c3c..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/escprober.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/escsm.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/escsm.cpython-311.pyc deleted file mode 100644 index 0995a61..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/escsm.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/eucjpprober.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/eucjpprober.cpython-311.pyc deleted file mode 100644 index ca75aa9..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/eucjpprober.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/euckrfreq.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/euckrfreq.cpython-311.pyc deleted file mode 100644 index ce2d5e4..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/euckrfreq.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/euckrprober.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/euckrprober.cpython-311.pyc deleted file mode 100644 index aa3c96f..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/euckrprober.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/euctwfreq.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/euctwfreq.cpython-311.pyc deleted file mode 100644 index 75cc238..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/euctwfreq.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/euctwprober.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/euctwprober.cpython-311.pyc deleted file mode 100644 index f4dc3ad..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/euctwprober.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/gb2312freq.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/gb2312freq.cpython-311.pyc deleted file mode 100644 index 92ae2ad..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/gb2312freq.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/gb2312prober.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/gb2312prober.cpython-311.pyc deleted file mode 100644 index 8205b67..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/gb2312prober.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/hebrewprober.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/hebrewprober.cpython-311.pyc deleted file mode 100644 index 612929f..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/hebrewprober.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/jisfreq.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/jisfreq.cpython-311.pyc deleted file mode 100644 index c812343..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/jisfreq.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/johabfreq.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/johabfreq.cpython-311.pyc deleted file mode 100644 index 3d998dc..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/johabfreq.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/johabprober.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/johabprober.cpython-311.pyc deleted file mode 100644 index 0be6da1..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/johabprober.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/jpcntx.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/jpcntx.cpython-311.pyc deleted file mode 100644 index 122eb5e..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/jpcntx.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/langbulgarianmodel.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/langbulgarianmodel.cpython-311.pyc deleted file mode 100644 index 97688d6..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/langbulgarianmodel.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/langgreekmodel.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/langgreekmodel.cpython-311.pyc deleted file mode 100644 index a3fc43a..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/langgreekmodel.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/langhebrewmodel.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/langhebrewmodel.cpython-311.pyc deleted file mode 100644 index a23e5a8..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/langhebrewmodel.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/langhungarianmodel.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/langhungarianmodel.cpython-311.pyc deleted file mode 100644 index 44d9cd4..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/langhungarianmodel.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/langrussianmodel.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/langrussianmodel.cpython-311.pyc deleted file mode 100644 index c3f9c78..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/langrussianmodel.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/langthaimodel.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/langthaimodel.cpython-311.pyc deleted file mode 100644 index 7aba0d5..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/langthaimodel.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/langturkishmodel.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/langturkishmodel.cpython-311.pyc deleted file mode 100644 index 3dffb28..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/langturkishmodel.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/latin1prober.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/latin1prober.cpython-311.pyc deleted file mode 100644 index 8685c2a..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/latin1prober.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/macromanprober.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/macromanprober.cpython-311.pyc deleted file mode 100644 index fc45ce3..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/macromanprober.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/mbcharsetprober.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/mbcharsetprober.cpython-311.pyc deleted file mode 100644 index 28f5c21..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/mbcharsetprober.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/mbcsgroupprober.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/mbcsgroupprober.cpython-311.pyc deleted file mode 100644 index 787c118..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/mbcsgroupprober.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/mbcssm.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/mbcssm.cpython-311.pyc deleted file mode 100644 index 8dd784f..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/mbcssm.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/resultdict.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/resultdict.cpython-311.pyc deleted file mode 100644 index 2cb8949..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/resultdict.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/sbcharsetprober.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/sbcharsetprober.cpython-311.pyc deleted file mode 100644 index d3b0a67..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/sbcharsetprober.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/sbcsgroupprober.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/sbcsgroupprober.cpython-311.pyc deleted file mode 100644 index 426e88e..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/sbcsgroupprober.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/sjisprober.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/sjisprober.cpython-311.pyc deleted file mode 100644 index f1fd6f7..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/sjisprober.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/universaldetector.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/universaldetector.cpython-311.pyc deleted file mode 100644 index 4377cfc..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/universaldetector.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/utf1632prober.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/utf1632prober.cpython-311.pyc deleted file mode 100644 index 9e7c237..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/utf1632prober.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/utf8prober.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/utf8prober.cpython-311.pyc deleted file mode 100644 index e444502..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/utf8prober.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/version.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/version.cpython-311.pyc deleted file mode 100644 index 5238e7b..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__/version.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/big5freq.py b/venv/lib/python3.11/site-packages/pip/_vendor/chardet/big5freq.py deleted file mode 100644 index 87d9f97..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/big5freq.py +++ /dev/null @@ -1,386 +0,0 @@ -######################## BEGIN LICENSE BLOCK ######################## -# The Original Code is Mozilla Communicator client code. -# -# The Initial Developer of the Original Code is -# Netscape Communications Corporation. -# Portions created by the Initial Developer are Copyright (C) 1998 -# the Initial Developer. All Rights Reserved. -# -# Contributor(s): -# Mark Pilgrim - port to Python -# -# This library is free software; you can redistribute it and/or -# modify it under the terms of the GNU Lesser General Public -# License as published by the Free Software Foundation; either -# version 2.1 of the License, or (at your option) any later version. -# -# This library is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# Lesser General Public License for more details. -# -# You should have received a copy of the GNU Lesser General Public -# License along with this library; if not, write to the Free Software -# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA -# 02110-1301 USA -######################### END LICENSE BLOCK ######################### - -# Big5 frequency table -# by Taiwan's Mandarin Promotion Council -# -# -# 128 --> 0.42261 -# 256 --> 0.57851 -# 512 --> 0.74851 -# 1024 --> 0.89384 -# 2048 --> 0.97583 -# -# Ideal Distribution Ratio = 0.74851/(1-0.74851) =2.98 -# Random Distribution Ration = 512/(5401-512)=0.105 -# -# Typical Distribution Ratio about 25% of Ideal one, still much higher than RDR - -BIG5_TYPICAL_DISTRIBUTION_RATIO = 0.75 - -# Char to FreqOrder table -BIG5_TABLE_SIZE = 5376 -# fmt: off -BIG5_CHAR_TO_FREQ_ORDER = ( - 1,1801,1506, 255,1431, 198, 9, 82, 6,5008, 177, 202,3681,1256,2821, 110, # 16 -3814, 33,3274, 261, 76, 44,2114, 16,2946,2187,1176, 659,3971, 26,3451,2653, # 32 -1198,3972,3350,4202, 410,2215, 302, 590, 361,1964, 8, 204, 58,4510,5009,1932, # 48 - 63,5010,5011, 317,1614, 75, 222, 159,4203,2417,1480,5012,3555,3091, 224,2822, # 64 -3682, 3, 10,3973,1471, 29,2787,1135,2866,1940, 873, 130,3275,1123, 312,5013, # 80 -4511,2052, 507, 252, 682,5014, 142,1915, 124, 206,2947, 34,3556,3204, 64, 604, # 96 -5015,2501,1977,1978, 155,1991, 645, 641,1606,5016,3452, 337, 72, 406,5017, 80, # 112 - 630, 238,3205,1509, 263, 939,1092,2654, 756,1440,1094,3453, 449, 69,2987, 591, # 128 - 179,2096, 471, 115,2035,1844, 60, 50,2988, 134, 806,1869, 734,2036,3454, 180, # 144 - 995,1607, 156, 537,2907, 688,5018, 319,1305, 779,2145, 514,2379, 298,4512, 359, # 160 -2502, 90,2716,1338, 663, 11, 906,1099,2553, 20,2441, 182, 532,1716,5019, 732, # 176 -1376,4204,1311,1420,3206, 25,2317,1056, 113, 399, 382,1950, 242,3455,2474, 529, # 192 -3276, 475,1447,3683,5020, 117, 21, 656, 810,1297,2300,2334,3557,5021, 126,4205, # 208 - 706, 456, 150, 613,4513, 71,1118,2037,4206, 145,3092, 85, 835, 486,2115,1246, # 224 -1426, 428, 727,1285,1015, 800, 106, 623, 303,1281,5022,2128,2359, 347,3815, 221, # 240 -3558,3135,5023,1956,1153,4207, 83, 296,1199,3093, 192, 624, 93,5024, 822,1898, # 256 -2823,3136, 795,2065, 991,1554,1542,1592, 27, 43,2867, 859, 139,1456, 860,4514, # 272 - 437, 712,3974, 164,2397,3137, 695, 211,3037,2097, 195,3975,1608,3559,3560,3684, # 288 -3976, 234, 811,2989,2098,3977,2233,1441,3561,1615,2380, 668,2077,1638, 305, 228, # 304 -1664,4515, 467, 415,5025, 262,2099,1593, 239, 108, 300, 200,1033, 512,1247,2078, # 320 -5026,5027,2176,3207,3685,2682, 593, 845,1062,3277, 88,1723,2038,3978,1951, 212, # 336 - 266, 152, 149, 468,1899,4208,4516, 77, 187,5028,3038, 37, 5,2990,5029,3979, # 352 -5030,5031, 39,2524,4517,2908,3208,2079, 55, 148, 74,4518, 545, 483,1474,1029, # 368 -1665, 217,1870,1531,3138,1104,2655,4209, 24, 172,3562, 900,3980,3563,3564,4519, # 384 - 32,1408,2824,1312, 329, 487,2360,2251,2717, 784,2683, 4,3039,3351,1427,1789, # 400 - 188, 109, 499,5032,3686,1717,1790, 888,1217,3040,4520,5033,3565,5034,3352,1520, # 416 -3687,3981, 196,1034, 775,5035,5036, 929,1816, 249, 439, 38,5037,1063,5038, 794, # 432 -3982,1435,2301, 46, 178,3278,2066,5039,2381,5040, 214,1709,4521, 804, 35, 707, # 448 - 324,3688,1601,2554, 140, 459,4210,5041,5042,1365, 839, 272, 978,2262,2580,3456, # 464 -2129,1363,3689,1423, 697, 100,3094, 48, 70,1231, 495,3139,2196,5043,1294,5044, # 480 -2080, 462, 586,1042,3279, 853, 256, 988, 185,2382,3457,1698, 434,1084,5045,3458, # 496 - 314,2625,2788,4522,2335,2336, 569,2285, 637,1817,2525, 757,1162,1879,1616,3459, # 512 - 287,1577,2116, 768,4523,1671,2868,3566,2526,1321,3816, 909,2418,5046,4211, 933, # 528 -3817,4212,2053,2361,1222,4524, 765,2419,1322, 786,4525,5047,1920,1462,1677,2909, # 544 -1699,5048,4526,1424,2442,3140,3690,2600,3353,1775,1941,3460,3983,4213, 309,1369, # 560 -1130,2825, 364,2234,1653,1299,3984,3567,3985,3986,2656, 525,1085,3041, 902,2001, # 576 -1475, 964,4527, 421,1845,1415,1057,2286, 940,1364,3141, 376,4528,4529,1381, 7, # 592 -2527, 983,2383, 336,1710,2684,1846, 321,3461, 559,1131,3042,2752,1809,1132,1313, # 608 - 265,1481,1858,5049, 352,1203,2826,3280, 167,1089, 420,2827, 776, 792,1724,3568, # 624 -4214,2443,3281,5050,4215,5051, 446, 229, 333,2753, 901,3818,1200,1557,4530,2657, # 640 -1921, 395,2754,2685,3819,4216,1836, 125, 916,3209,2626,4531,5052,5053,3820,5054, # 656 -5055,5056,4532,3142,3691,1133,2555,1757,3462,1510,2318,1409,3569,5057,2146, 438, # 672 -2601,2910,2384,3354,1068, 958,3043, 461, 311,2869,2686,4217,1916,3210,4218,1979, # 688 - 383, 750,2755,2627,4219, 274, 539, 385,1278,1442,5058,1154,1965, 384, 561, 210, # 704 - 98,1295,2556,3570,5059,1711,2420,1482,3463,3987,2911,1257, 129,5060,3821, 642, # 720 - 523,2789,2790,2658,5061, 141,2235,1333, 68, 176, 441, 876, 907,4220, 603,2602, # 736 - 710, 171,3464, 404, 549, 18,3143,2398,1410,3692,1666,5062,3571,4533,2912,4534, # 752 -5063,2991, 368,5064, 146, 366, 99, 871,3693,1543, 748, 807,1586,1185, 22,2263, # 768 - 379,3822,3211,5065,3212, 505,1942,2628,1992,1382,2319,5066, 380,2362, 218, 702, # 784 -1818,1248,3465,3044,3572,3355,3282,5067,2992,3694, 930,3283,3823,5068, 59,5069, # 800 - 585, 601,4221, 497,3466,1112,1314,4535,1802,5070,1223,1472,2177,5071, 749,1837, # 816 - 690,1900,3824,1773,3988,1476, 429,1043,1791,2236,2117, 917,4222, 447,1086,1629, # 832 -5072, 556,5073,5074,2021,1654, 844,1090, 105, 550, 966,1758,2828,1008,1783, 686, # 848 -1095,5075,2287, 793,1602,5076,3573,2603,4536,4223,2948,2302,4537,3825, 980,2503, # 864 - 544, 353, 527,4538, 908,2687,2913,5077, 381,2629,1943,1348,5078,1341,1252, 560, # 880 -3095,5079,3467,2870,5080,2054, 973, 886,2081, 143,4539,5081,5082, 157,3989, 496, # 896 -4224, 57, 840, 540,2039,4540,4541,3468,2118,1445, 970,2264,1748,1966,2082,4225, # 912 -3144,1234,1776,3284,2829,3695, 773,1206,2130,1066,2040,1326,3990,1738,1725,4226, # 928 - 279,3145, 51,1544,2604, 423,1578,2131,2067, 173,4542,1880,5083,5084,1583, 264, # 944 - 610,3696,4543,2444, 280, 154,5085,5086,5087,1739, 338,1282,3096, 693,2871,1411, # 960 -1074,3826,2445,5088,4544,5089,5090,1240, 952,2399,5091,2914,1538,2688, 685,1483, # 976 -4227,2475,1436, 953,4228,2055,4545, 671,2400, 79,4229,2446,3285, 608, 567,2689, # 992 -3469,4230,4231,1691, 393,1261,1792,2401,5092,4546,5093,5094,5095,5096,1383,1672, # 1008 -3827,3213,1464, 522,1119, 661,1150, 216, 675,4547,3991,1432,3574, 609,4548,2690, # 1024 -2402,5097,5098,5099,4232,3045, 0,5100,2476, 315, 231,2447, 301,3356,4549,2385, # 1040 -5101, 233,4233,3697,1819,4550,4551,5102, 96,1777,1315,2083,5103, 257,5104,1810, # 1056 -3698,2718,1139,1820,4234,2022,1124,2164,2791,1778,2659,5105,3097, 363,1655,3214, # 1072 -5106,2993,5107,5108,5109,3992,1567,3993, 718, 103,3215, 849,1443, 341,3357,2949, # 1088 -1484,5110,1712, 127, 67, 339,4235,2403, 679,1412, 821,5111,5112, 834, 738, 351, # 1104 -2994,2147, 846, 235,1497,1881, 418,1993,3828,2719, 186,1100,2148,2756,3575,1545, # 1120 -1355,2950,2872,1377, 583,3994,4236,2581,2995,5113,1298,3699,1078,2557,3700,2363, # 1136 - 78,3829,3830, 267,1289,2100,2002,1594,4237, 348, 369,1274,2197,2178,1838,4552, # 1152 -1821,2830,3701,2757,2288,2003,4553,2951,2758, 144,3358, 882,4554,3995,2759,3470, # 1168 -4555,2915,5114,4238,1726, 320,5115,3996,3046, 788,2996,5116,2831,1774,1327,2873, # 1184 -3997,2832,5117,1306,4556,2004,1700,3831,3576,2364,2660, 787,2023, 506, 824,3702, # 1200 - 534, 323,4557,1044,3359,2024,1901, 946,3471,5118,1779,1500,1678,5119,1882,4558, # 1216 - 165, 243,4559,3703,2528, 123, 683,4239, 764,4560, 36,3998,1793, 589,2916, 816, # 1232 - 626,1667,3047,2237,1639,1555,1622,3832,3999,5120,4000,2874,1370,1228,1933, 891, # 1248 -2084,2917, 304,4240,5121, 292,2997,2720,3577, 691,2101,4241,1115,4561, 118, 662, # 1264 -5122, 611,1156, 854,2386,1316,2875, 2, 386, 515,2918,5123,5124,3286, 868,2238, # 1280 -1486, 855,2661, 785,2216,3048,5125,1040,3216,3578,5126,3146, 448,5127,1525,5128, # 1296 -2165,4562,5129,3833,5130,4242,2833,3579,3147, 503, 818,4001,3148,1568, 814, 676, # 1312 -1444, 306,1749,5131,3834,1416,1030, 197,1428, 805,2834,1501,4563,5132,5133,5134, # 1328 -1994,5135,4564,5136,5137,2198, 13,2792,3704,2998,3149,1229,1917,5138,3835,2132, # 1344 -5139,4243,4565,2404,3580,5140,2217,1511,1727,1120,5141,5142, 646,3836,2448, 307, # 1360 -5143,5144,1595,3217,5145,5146,5147,3705,1113,1356,4002,1465,2529,2530,5148, 519, # 1376 -5149, 128,2133, 92,2289,1980,5150,4003,1512, 342,3150,2199,5151,2793,2218,1981, # 1392 -3360,4244, 290,1656,1317, 789, 827,2365,5152,3837,4566, 562, 581,4004,5153, 401, # 1408 -4567,2252, 94,4568,5154,1399,2794,5155,1463,2025,4569,3218,1944,5156, 828,1105, # 1424 -4245,1262,1394,5157,4246, 605,4570,5158,1784,2876,5159,2835, 819,2102, 578,2200, # 1440 -2952,5160,1502, 436,3287,4247,3288,2836,4005,2919,3472,3473,5161,2721,2320,5162, # 1456 -5163,2337,2068, 23,4571, 193, 826,3838,2103, 699,1630,4248,3098, 390,1794,1064, # 1472 -3581,5164,1579,3099,3100,1400,5165,4249,1839,1640,2877,5166,4572,4573, 137,4250, # 1488 - 598,3101,1967, 780, 104, 974,2953,5167, 278, 899, 253, 402, 572, 504, 493,1339, # 1504 -5168,4006,1275,4574,2582,2558,5169,3706,3049,3102,2253, 565,1334,2722, 863, 41, # 1520 -5170,5171,4575,5172,1657,2338, 19, 463,2760,4251, 606,5173,2999,3289,1087,2085, # 1536 -1323,2662,3000,5174,1631,1623,1750,4252,2691,5175,2878, 791,2723,2663,2339, 232, # 1552 -2421,5176,3001,1498,5177,2664,2630, 755,1366,3707,3290,3151,2026,1609, 119,1918, # 1568 -3474, 862,1026,4253,5178,4007,3839,4576,4008,4577,2265,1952,2477,5179,1125, 817, # 1584 -4254,4255,4009,1513,1766,2041,1487,4256,3050,3291,2837,3840,3152,5180,5181,1507, # 1600 -5182,2692, 733, 40,1632,1106,2879, 345,4257, 841,2531, 230,4578,3002,1847,3292, # 1616 -3475,5183,1263, 986,3476,5184, 735, 879, 254,1137, 857, 622,1300,1180,1388,1562, # 1632 -4010,4011,2954, 967,2761,2665,1349, 592,2134,1692,3361,3003,1995,4258,1679,4012, # 1648 -1902,2188,5185, 739,3708,2724,1296,1290,5186,4259,2201,2202,1922,1563,2605,2559, # 1664 -1871,2762,3004,5187, 435,5188, 343,1108, 596, 17,1751,4579,2239,3477,3709,5189, # 1680 -4580, 294,3582,2955,1693, 477, 979, 281,2042,3583, 643,2043,3710,2631,2795,2266, # 1696 -1031,2340,2135,2303,3584,4581, 367,1249,2560,5190,3585,5191,4582,1283,3362,2005, # 1712 - 240,1762,3363,4583,4584, 836,1069,3153, 474,5192,2149,2532, 268,3586,5193,3219, # 1728 -1521,1284,5194,1658,1546,4260,5195,3587,3588,5196,4261,3364,2693,1685,4262, 961, # 1744 -1673,2632, 190,2006,2203,3841,4585,4586,5197, 570,2504,3711,1490,5198,4587,2633, # 1760 -3293,1957,4588, 584,1514, 396,1045,1945,5199,4589,1968,2449,5200,5201,4590,4013, # 1776 - 619,5202,3154,3294, 215,2007,2796,2561,3220,4591,3221,4592, 763,4263,3842,4593, # 1792 -5203,5204,1958,1767,2956,3365,3712,1174, 452,1477,4594,3366,3155,5205,2838,1253, # 1808 -2387,2189,1091,2290,4264, 492,5206, 638,1169,1825,2136,1752,4014, 648, 926,1021, # 1824 -1324,4595, 520,4596, 997, 847,1007, 892,4597,3843,2267,1872,3713,2405,1785,4598, # 1840 -1953,2957,3103,3222,1728,4265,2044,3714,4599,2008,1701,3156,1551, 30,2268,4266, # 1856 -5207,2027,4600,3589,5208, 501,5209,4267, 594,3478,2166,1822,3590,3479,3591,3223, # 1872 - 829,2839,4268,5210,1680,3157,1225,4269,5211,3295,4601,4270,3158,2341,5212,4602, # 1888 -4271,5213,4015,4016,5214,1848,2388,2606,3367,5215,4603, 374,4017, 652,4272,4273, # 1904 - 375,1140, 798,5216,5217,5218,2366,4604,2269, 546,1659, 138,3051,2450,4605,5219, # 1920 -2254, 612,1849, 910, 796,3844,1740,1371, 825,3845,3846,5220,2920,2562,5221, 692, # 1936 - 444,3052,2634, 801,4606,4274,5222,1491, 244,1053,3053,4275,4276, 340,5223,4018, # 1952 -1041,3005, 293,1168, 87,1357,5224,1539, 959,5225,2240, 721, 694,4277,3847, 219, # 1968 -1478, 644,1417,3368,2666,1413,1401,1335,1389,4019,5226,5227,3006,2367,3159,1826, # 1984 - 730,1515, 184,2840, 66,4607,5228,1660,2958, 246,3369, 378,1457, 226,3480, 975, # 2000 -4020,2959,1264,3592, 674, 696,5229, 163,5230,1141,2422,2167, 713,3593,3370,4608, # 2016 -4021,5231,5232,1186, 15,5233,1079,1070,5234,1522,3224,3594, 276,1050,2725, 758, # 2032 -1126, 653,2960,3296,5235,2342, 889,3595,4022,3104,3007, 903,1250,4609,4023,3481, # 2048 -3596,1342,1681,1718, 766,3297, 286, 89,2961,3715,5236,1713,5237,2607,3371,3008, # 2064 -5238,2962,2219,3225,2880,5239,4610,2505,2533, 181, 387,1075,4024, 731,2190,3372, # 2080 -5240,3298, 310, 313,3482,2304, 770,4278, 54,3054, 189,4611,3105,3848,4025,5241, # 2096 -1230,1617,1850, 355,3597,4279,4612,3373, 111,4280,3716,1350,3160,3483,3055,4281, # 2112 -2150,3299,3598,5242,2797,4026,4027,3009, 722,2009,5243,1071, 247,1207,2343,2478, # 2128 -1378,4613,2010, 864,1437,1214,4614, 373,3849,1142,2220, 667,4615, 442,2763,2563, # 2144 -3850,4028,1969,4282,3300,1840, 837, 170,1107, 934,1336,1883,5244,5245,2119,4283, # 2160 -2841, 743,1569,5246,4616,4284, 582,2389,1418,3484,5247,1803,5248, 357,1395,1729, # 2176 -3717,3301,2423,1564,2241,5249,3106,3851,1633,4617,1114,2086,4285,1532,5250, 482, # 2192 -2451,4618,5251,5252,1492, 833,1466,5253,2726,3599,1641,2842,5254,1526,1272,3718, # 2208 -4286,1686,1795, 416,2564,1903,1954,1804,5255,3852,2798,3853,1159,2321,5256,2881, # 2224 -4619,1610,1584,3056,2424,2764, 443,3302,1163,3161,5257,5258,4029,5259,4287,2506, # 2240 -3057,4620,4030,3162,2104,1647,3600,2011,1873,4288,5260,4289, 431,3485,5261, 250, # 2256 - 97, 81,4290,5262,1648,1851,1558, 160, 848,5263, 866, 740,1694,5264,2204,2843, # 2272 -3226,4291,4621,3719,1687, 950,2479, 426, 469,3227,3720,3721,4031,5265,5266,1188, # 2288 - 424,1996, 861,3601,4292,3854,2205,2694, 168,1235,3602,4293,5267,2087,1674,4622, # 2304 -3374,3303, 220,2565,1009,5268,3855, 670,3010, 332,1208, 717,5269,5270,3603,2452, # 2320 -4032,3375,5271, 513,5272,1209,2882,3376,3163,4623,1080,5273,5274,5275,5276,2534, # 2336 -3722,3604, 815,1587,4033,4034,5277,3605,3486,3856,1254,4624,1328,3058,1390,4035, # 2352 -1741,4036,3857,4037,5278, 236,3858,2453,3304,5279,5280,3723,3859,1273,3860,4625, # 2368 -5281, 308,5282,4626, 245,4627,1852,2480,1307,2583, 430, 715,2137,2454,5283, 270, # 2384 - 199,2883,4038,5284,3606,2727,1753, 761,1754, 725,1661,1841,4628,3487,3724,5285, # 2400 -5286, 587, 14,3305, 227,2608, 326, 480,2270, 943,2765,3607, 291, 650,1884,5287, # 2416 -1702,1226, 102,1547, 62,3488, 904,4629,3489,1164,4294,5288,5289,1224,1548,2766, # 2432 - 391, 498,1493,5290,1386,1419,5291,2056,1177,4630, 813, 880,1081,2368, 566,1145, # 2448 -4631,2291,1001,1035,2566,2609,2242, 394,1286,5292,5293,2069,5294, 86,1494,1730, # 2464 -4039, 491,1588, 745, 897,2963, 843,3377,4040,2767,2884,3306,1768, 998,2221,2070, # 2480 - 397,1827,1195,1970,3725,3011,3378, 284,5295,3861,2507,2138,2120,1904,5296,4041, # 2496 -2151,4042,4295,1036,3490,1905, 114,2567,4296, 209,1527,5297,5298,2964,2844,2635, # 2512 -2390,2728,3164, 812,2568,5299,3307,5300,1559, 737,1885,3726,1210, 885, 28,2695, # 2528 -3608,3862,5301,4297,1004,1780,4632,5302, 346,1982,2222,2696,4633,3863,1742, 797, # 2544 -1642,4043,1934,1072,1384,2152, 896,4044,3308,3727,3228,2885,3609,5303,2569,1959, # 2560 -4634,2455,1786,5304,5305,5306,4045,4298,1005,1308,3728,4299,2729,4635,4636,1528, # 2576 -2610, 161,1178,4300,1983, 987,4637,1101,4301, 631,4046,1157,3229,2425,1343,1241, # 2592 -1016,2243,2570, 372, 877,2344,2508,1160, 555,1935, 911,4047,5307, 466,1170, 169, # 2608 -1051,2921,2697,3729,2481,3012,1182,2012,2571,1251,2636,5308, 992,2345,3491,1540, # 2624 -2730,1201,2071,2406,1997,2482,5309,4638, 528,1923,2191,1503,1874,1570,2369,3379, # 2640 -3309,5310, 557,1073,5311,1828,3492,2088,2271,3165,3059,3107, 767,3108,2799,4639, # 2656 -1006,4302,4640,2346,1267,2179,3730,3230, 778,4048,3231,2731,1597,2667,5312,4641, # 2672 -5313,3493,5314,5315,5316,3310,2698,1433,3311, 131, 95,1504,4049, 723,4303,3166, # 2688 -1842,3610,2768,2192,4050,2028,2105,3731,5317,3013,4051,1218,5318,3380,3232,4052, # 2704 -4304,2584, 248,1634,3864, 912,5319,2845,3732,3060,3865, 654, 53,5320,3014,5321, # 2720 -1688,4642, 777,3494,1032,4053,1425,5322, 191, 820,2121,2846, 971,4643, 931,3233, # 2736 - 135, 664, 783,3866,1998, 772,2922,1936,4054,3867,4644,2923,3234, 282,2732, 640, # 2752 -1372,3495,1127, 922, 325,3381,5323,5324, 711,2045,5325,5326,4055,2223,2800,1937, # 2768 -4056,3382,2224,2255,3868,2305,5327,4645,3869,1258,3312,4057,3235,2139,2965,4058, # 2784 -4059,5328,2225, 258,3236,4646, 101,1227,5329,3313,1755,5330,1391,3314,5331,2924, # 2800 -2057, 893,5332,5333,5334,1402,4305,2347,5335,5336,3237,3611,5337,5338, 878,1325, # 2816 -1781,2801,4647, 259,1385,2585, 744,1183,2272,4648,5339,4060,2509,5340, 684,1024, # 2832 -4306,5341, 472,3612,3496,1165,3315,4061,4062, 322,2153, 881, 455,1695,1152,1340, # 2848 - 660, 554,2154,4649,1058,4650,4307, 830,1065,3383,4063,4651,1924,5342,1703,1919, # 2864 -5343, 932,2273, 122,5344,4652, 947, 677,5345,3870,2637, 297,1906,1925,2274,4653, # 2880 -2322,3316,5346,5347,4308,5348,4309, 84,4310, 112, 989,5349, 547,1059,4064, 701, # 2896 -3613,1019,5350,4311,5351,3497, 942, 639, 457,2306,2456, 993,2966, 407, 851, 494, # 2912 -4654,3384, 927,5352,1237,5353,2426,3385, 573,4312, 680, 921,2925,1279,1875, 285, # 2928 - 790,1448,1984, 719,2168,5354,5355,4655,4065,4066,1649,5356,1541, 563,5357,1077, # 2944 -5358,3386,3061,3498, 511,3015,4067,4068,3733,4069,1268,2572,3387,3238,4656,4657, # 2960 -5359, 535,1048,1276,1189,2926,2029,3167,1438,1373,2847,2967,1134,2013,5360,4313, # 2976 -1238,2586,3109,1259,5361, 700,5362,2968,3168,3734,4314,5363,4315,1146,1876,1907, # 2992 -4658,2611,4070, 781,2427, 132,1589, 203, 147, 273,2802,2407, 898,1787,2155,4071, # 3008 -4072,5364,3871,2803,5365,5366,4659,4660,5367,3239,5368,1635,3872, 965,5369,1805, # 3024 -2699,1516,3614,1121,1082,1329,3317,4073,1449,3873, 65,1128,2848,2927,2769,1590, # 3040 -3874,5370,5371, 12,2668, 45, 976,2587,3169,4661, 517,2535,1013,1037,3240,5372, # 3056 -3875,2849,5373,3876,5374,3499,5375,2612, 614,1999,2323,3877,3110,2733,2638,5376, # 3072 -2588,4316, 599,1269,5377,1811,3735,5378,2700,3111, 759,1060, 489,1806,3388,3318, # 3088 -1358,5379,5380,2391,1387,1215,2639,2256, 490,5381,5382,4317,1759,2392,2348,5383, # 3104 -4662,3878,1908,4074,2640,1807,3241,4663,3500,3319,2770,2349, 874,5384,5385,3501, # 3120 -3736,1859, 91,2928,3737,3062,3879,4664,5386,3170,4075,2669,5387,3502,1202,1403, # 3136 -3880,2969,2536,1517,2510,4665,3503,2511,5388,4666,5389,2701,1886,1495,1731,4076, # 3152 -2370,4667,5390,2030,5391,5392,4077,2702,1216, 237,2589,4318,2324,4078,3881,4668, # 3168 -4669,2703,3615,3504, 445,4670,5393,5394,5395,5396,2771, 61,4079,3738,1823,4080, # 3184 -5397, 687,2046, 935, 925, 405,2670, 703,1096,1860,2734,4671,4081,1877,1367,2704, # 3200 -3389, 918,2106,1782,2483, 334,3320,1611,1093,4672, 564,3171,3505,3739,3390, 945, # 3216 -2641,2058,4673,5398,1926, 872,4319,5399,3506,2705,3112, 349,4320,3740,4082,4674, # 3232 -3882,4321,3741,2156,4083,4675,4676,4322,4677,2408,2047, 782,4084, 400, 251,4323, # 3248 -1624,5400,5401, 277,3742, 299,1265, 476,1191,3883,2122,4324,4325,1109, 205,5402, # 3264 -2590,1000,2157,3616,1861,5403,5404,5405,4678,5406,4679,2573, 107,2484,2158,4085, # 3280 -3507,3172,5407,1533, 541,1301, 158, 753,4326,2886,3617,5408,1696, 370,1088,4327, # 3296 -4680,3618, 579, 327, 440, 162,2244, 269,1938,1374,3508, 968,3063, 56,1396,3113, # 3312 -2107,3321,3391,5409,1927,2159,4681,3016,5410,3619,5411,5412,3743,4682,2485,5413, # 3328 -2804,5414,1650,4683,5415,2613,5416,5417,4086,2671,3392,1149,3393,4087,3884,4088, # 3344 -5418,1076, 49,5419, 951,3242,3322,3323, 450,2850, 920,5420,1812,2805,2371,4328, # 3360 -1909,1138,2372,3885,3509,5421,3243,4684,1910,1147,1518,2428,4685,3886,5422,4686, # 3376 -2393,2614, 260,1796,3244,5423,5424,3887,3324, 708,5425,3620,1704,5426,3621,1351, # 3392 -1618,3394,3017,1887, 944,4329,3395,4330,3064,3396,4331,5427,3744, 422, 413,1714, # 3408 -3325, 500,2059,2350,4332,2486,5428,1344,1911, 954,5429,1668,5430,5431,4089,2409, # 3424 -4333,3622,3888,4334,5432,2307,1318,2512,3114, 133,3115,2887,4687, 629, 31,2851, # 3440 -2706,3889,4688, 850, 949,4689,4090,2970,1732,2089,4335,1496,1853,5433,4091, 620, # 3456 -3245, 981,1242,3745,3397,1619,3746,1643,3326,2140,2457,1971,1719,3510,2169,5434, # 3472 -3246,5435,5436,3398,1829,5437,1277,4690,1565,2048,5438,1636,3623,3116,5439, 869, # 3488 -2852, 655,3890,3891,3117,4092,3018,3892,1310,3624,4691,5440,5441,5442,1733, 558, # 3504 -4692,3747, 335,1549,3065,1756,4336,3748,1946,3511,1830,1291,1192, 470,2735,2108, # 3520 -2806, 913,1054,4093,5443,1027,5444,3066,4094,4693, 982,2672,3399,3173,3512,3247, # 3536 -3248,1947,2807,5445, 571,4694,5446,1831,5447,3625,2591,1523,2429,5448,2090, 984, # 3552 -4695,3749,1960,5449,3750, 852, 923,2808,3513,3751, 969,1519, 999,2049,2325,1705, # 3568 -5450,3118, 615,1662, 151, 597,4095,2410,2326,1049, 275,4696,3752,4337, 568,3753, # 3584 -3626,2487,4338,3754,5451,2430,2275, 409,3249,5452,1566,2888,3514,1002, 769,2853, # 3600 - 194,2091,3174,3755,2226,3327,4339, 628,1505,5453,5454,1763,2180,3019,4096, 521, # 3616 -1161,2592,1788,2206,2411,4697,4097,1625,4340,4341, 412, 42,3119, 464,5455,2642, # 3632 -4698,3400,1760,1571,2889,3515,2537,1219,2207,3893,2643,2141,2373,4699,4700,3328, # 3648 -1651,3401,3627,5456,5457,3628,2488,3516,5458,3756,5459,5460,2276,2092, 460,5461, # 3664 -4701,5462,3020, 962, 588,3629, 289,3250,2644,1116, 52,5463,3067,1797,5464,5465, # 3680 -5466,1467,5467,1598,1143,3757,4342,1985,1734,1067,4702,1280,3402, 465,4703,1572, # 3696 - 510,5468,1928,2245,1813,1644,3630,5469,4704,3758,5470,5471,2673,1573,1534,5472, # 3712 -5473, 536,1808,1761,3517,3894,3175,2645,5474,5475,5476,4705,3518,2929,1912,2809, # 3728 -5477,3329,1122, 377,3251,5478, 360,5479,5480,4343,1529, 551,5481,2060,3759,1769, # 3744 -2431,5482,2930,4344,3330,3120,2327,2109,2031,4706,1404, 136,1468,1479, 672,1171, # 3760 -3252,2308, 271,3176,5483,2772,5484,2050, 678,2736, 865,1948,4707,5485,2014,4098, # 3776 -2971,5486,2737,2227,1397,3068,3760,4708,4709,1735,2931,3403,3631,5487,3895, 509, # 3792 -2854,2458,2890,3896,5488,5489,3177,3178,4710,4345,2538,4711,2309,1166,1010, 552, # 3808 - 681,1888,5490,5491,2972,2973,4099,1287,1596,1862,3179, 358, 453, 736, 175, 478, # 3824 -1117, 905,1167,1097,5492,1854,1530,5493,1706,5494,2181,3519,2292,3761,3520,3632, # 3840 -4346,2093,4347,5495,3404,1193,2489,4348,1458,2193,2208,1863,1889,1421,3331,2932, # 3856 -3069,2182,3521, 595,2123,5496,4100,5497,5498,4349,1707,2646, 223,3762,1359, 751, # 3872 -3121, 183,3522,5499,2810,3021, 419,2374, 633, 704,3897,2394, 241,5500,5501,5502, # 3888 - 838,3022,3763,2277,2773,2459,3898,1939,2051,4101,1309,3122,2246,1181,5503,1136, # 3904 -2209,3899,2375,1446,4350,2310,4712,5504,5505,4351,1055,2615, 484,3764,5506,4102, # 3920 - 625,4352,2278,3405,1499,4353,4103,5507,4104,4354,3253,2279,2280,3523,5508,5509, # 3936 -2774, 808,2616,3765,3406,4105,4355,3123,2539, 526,3407,3900,4356, 955,5510,1620, # 3952 -4357,2647,2432,5511,1429,3766,1669,1832, 994, 928,5512,3633,1260,5513,5514,5515, # 3968 -1949,2293, 741,2933,1626,4358,2738,2460, 867,1184, 362,3408,1392,5516,5517,4106, # 3984 -4359,1770,1736,3254,2934,4713,4714,1929,2707,1459,1158,5518,3070,3409,2891,1292, # 4000 -1930,2513,2855,3767,1986,1187,2072,2015,2617,4360,5519,2574,2514,2170,3768,2490, # 4016 -3332,5520,3769,4715,5521,5522, 666,1003,3023,1022,3634,4361,5523,4716,1814,2257, # 4032 - 574,3901,1603, 295,1535, 705,3902,4362, 283, 858, 417,5524,5525,3255,4717,4718, # 4048 -3071,1220,1890,1046,2281,2461,4107,1393,1599, 689,2575, 388,4363,5526,2491, 802, # 4064 -5527,2811,3903,2061,1405,2258,5528,4719,3904,2110,1052,1345,3256,1585,5529, 809, # 4080 -5530,5531,5532, 575,2739,3524, 956,1552,1469,1144,2328,5533,2329,1560,2462,3635, # 4096 -3257,4108, 616,2210,4364,3180,2183,2294,5534,1833,5535,3525,4720,5536,1319,3770, # 4112 -3771,1211,3636,1023,3258,1293,2812,5537,5538,5539,3905, 607,2311,3906, 762,2892, # 4128 -1439,4365,1360,4721,1485,3072,5540,4722,1038,4366,1450,2062,2648,4367,1379,4723, # 4144 -2593,5541,5542,4368,1352,1414,2330,2935,1172,5543,5544,3907,3908,4724,1798,1451, # 4160 -5545,5546,5547,5548,2936,4109,4110,2492,2351, 411,4111,4112,3637,3333,3124,4725, # 4176 -1561,2674,1452,4113,1375,5549,5550, 47,2974, 316,5551,1406,1591,2937,3181,5552, # 4192 -1025,2142,3125,3182, 354,2740, 884,2228,4369,2412, 508,3772, 726,3638, 996,2433, # 4208 -3639, 729,5553, 392,2194,1453,4114,4726,3773,5554,5555,2463,3640,2618,1675,2813, # 4224 - 919,2352,2975,2353,1270,4727,4115, 73,5556,5557, 647,5558,3259,2856,2259,1550, # 4240 -1346,3024,5559,1332, 883,3526,5560,5561,5562,5563,3334,2775,5564,1212, 831,1347, # 4256 -4370,4728,2331,3909,1864,3073, 720,3910,4729,4730,3911,5565,4371,5566,5567,4731, # 4272 -5568,5569,1799,4732,3774,2619,4733,3641,1645,2376,4734,5570,2938, 669,2211,2675, # 4288 -2434,5571,2893,5572,5573,1028,3260,5574,4372,2413,5575,2260,1353,5576,5577,4735, # 4304 -3183, 518,5578,4116,5579,4373,1961,5580,2143,4374,5581,5582,3025,2354,2355,3912, # 4320 - 516,1834,1454,4117,2708,4375,4736,2229,2620,1972,1129,3642,5583,2776,5584,2976, # 4336 -1422, 577,1470,3026,1524,3410,5585,5586, 432,4376,3074,3527,5587,2594,1455,2515, # 4352 -2230,1973,1175,5588,1020,2741,4118,3528,4737,5589,2742,5590,1743,1361,3075,3529, # 4368 -2649,4119,4377,4738,2295, 895, 924,4378,2171, 331,2247,3076, 166,1627,3077,1098, # 4384 -5591,1232,2894,2231,3411,4739, 657, 403,1196,2377, 542,3775,3412,1600,4379,3530, # 4400 -5592,4740,2777,3261, 576, 530,1362,4741,4742,2540,2676,3776,4120,5593, 842,3913, # 4416 -5594,2814,2032,1014,4121, 213,2709,3413, 665, 621,4380,5595,3777,2939,2435,5596, # 4432 -2436,3335,3643,3414,4743,4381,2541,4382,4744,3644,1682,4383,3531,1380,5597, 724, # 4448 -2282, 600,1670,5598,1337,1233,4745,3126,2248,5599,1621,4746,5600, 651,4384,5601, # 4464 -1612,4385,2621,5602,2857,5603,2743,2312,3078,5604, 716,2464,3079, 174,1255,2710, # 4480 -4122,3645, 548,1320,1398, 728,4123,1574,5605,1891,1197,3080,4124,5606,3081,3082, # 4496 -3778,3646,3779, 747,5607, 635,4386,4747,5608,5609,5610,4387,5611,5612,4748,5613, # 4512 -3415,4749,2437, 451,5614,3780,2542,2073,4388,2744,4389,4125,5615,1764,4750,5616, # 4528 -4390, 350,4751,2283,2395,2493,5617,4391,4126,2249,1434,4127, 488,4752, 458,4392, # 4544 -4128,3781, 771,1330,2396,3914,2576,3184,2160,2414,1553,2677,3185,4393,5618,2494, # 4560 -2895,2622,1720,2711,4394,3416,4753,5619,2543,4395,5620,3262,4396,2778,5621,2016, # 4576 -2745,5622,1155,1017,3782,3915,5623,3336,2313, 201,1865,4397,1430,5624,4129,5625, # 4592 -5626,5627,5628,5629,4398,1604,5630, 414,1866, 371,2595,4754,4755,3532,2017,3127, # 4608 -4756,1708, 960,4399, 887, 389,2172,1536,1663,1721,5631,2232,4130,2356,2940,1580, # 4624 -5632,5633,1744,4757,2544,4758,4759,5634,4760,5635,2074,5636,4761,3647,3417,2896, # 4640 -4400,5637,4401,2650,3418,2815, 673,2712,2465, 709,3533,4131,3648,4402,5638,1148, # 4656 - 502, 634,5639,5640,1204,4762,3649,1575,4763,2623,3783,5641,3784,3128, 948,3263, # 4672 - 121,1745,3916,1110,5642,4403,3083,2516,3027,4132,3785,1151,1771,3917,1488,4133, # 4688 -1987,5643,2438,3534,5644,5645,2094,5646,4404,3918,1213,1407,2816, 531,2746,2545, # 4704 -3264,1011,1537,4764,2779,4405,3129,1061,5647,3786,3787,1867,2897,5648,2018, 120, # 4720 -4406,4407,2063,3650,3265,2314,3919,2678,3419,1955,4765,4134,5649,3535,1047,2713, # 4736 -1266,5650,1368,4766,2858, 649,3420,3920,2546,2747,1102,2859,2679,5651,5652,2000, # 4752 -5653,1111,3651,2977,5654,2495,3921,3652,2817,1855,3421,3788,5655,5656,3422,2415, # 4768 -2898,3337,3266,3653,5657,2577,5658,3654,2818,4135,1460, 856,5659,3655,5660,2899, # 4784 -2978,5661,2900,3922,5662,4408, 632,2517, 875,3923,1697,3924,2296,5663,5664,4767, # 4800 -3028,1239, 580,4768,4409,5665, 914, 936,2075,1190,4136,1039,2124,5666,5667,5668, # 4816 -5669,3423,1473,5670,1354,4410,3925,4769,2173,3084,4137, 915,3338,4411,4412,3339, # 4832 -1605,1835,5671,2748, 398,3656,4413,3926,4138, 328,1913,2860,4139,3927,1331,4414, # 4848 -3029, 937,4415,5672,3657,4140,4141,3424,2161,4770,3425, 524, 742, 538,3085,1012, # 4864 -5673,5674,3928,2466,5675, 658,1103, 225,3929,5676,5677,4771,5678,4772,5679,3267, # 4880 -1243,5680,4142, 963,2250,4773,5681,2714,3658,3186,5682,5683,2596,2332,5684,4774, # 4896 -5685,5686,5687,3536, 957,3426,2547,2033,1931,2941,2467, 870,2019,3659,1746,2780, # 4912 -2781,2439,2468,5688,3930,5689,3789,3130,3790,3537,3427,3791,5690,1179,3086,5691, # 4928 -3187,2378,4416,3792,2548,3188,3131,2749,4143,5692,3428,1556,2549,2297, 977,2901, # 4944 -2034,4144,1205,3429,5693,1765,3430,3189,2125,1271, 714,1689,4775,3538,5694,2333, # 4960 -3931, 533,4417,3660,2184, 617,5695,2469,3340,3539,2315,5696,5697,3190,5698,5699, # 4976 -3932,1988, 618, 427,2651,3540,3431,5700,5701,1244,1690,5702,2819,4418,4776,5703, # 4992 -3541,4777,5704,2284,1576, 473,3661,4419,3432, 972,5705,3662,5706,3087,5707,5708, # 5008 -4778,4779,5709,3793,4145,4146,5710, 153,4780, 356,5711,1892,2902,4420,2144, 408, # 5024 - 803,2357,5712,3933,5713,4421,1646,2578,2518,4781,4782,3934,5714,3935,4422,5715, # 5040 -2416,3433, 752,5716,5717,1962,3341,2979,5718, 746,3030,2470,4783,4423,3794, 698, # 5056 -4784,1893,4424,3663,2550,4785,3664,3936,5719,3191,3434,5720,1824,1302,4147,2715, # 5072 -3937,1974,4425,5721,4426,3192, 823,1303,1288,1236,2861,3542,4148,3435, 774,3938, # 5088 -5722,1581,4786,1304,2862,3939,4787,5723,2440,2162,1083,3268,4427,4149,4428, 344, # 5104 -1173, 288,2316, 454,1683,5724,5725,1461,4788,4150,2597,5726,5727,4789, 985, 894, # 5120 -5728,3436,3193,5729,1914,2942,3795,1989,5730,2111,1975,5731,4151,5732,2579,1194, # 5136 - 425,5733,4790,3194,1245,3796,4429,5734,5735,2863,5736, 636,4791,1856,3940, 760, # 5152 -1800,5737,4430,2212,1508,4792,4152,1894,1684,2298,5738,5739,4793,4431,4432,2213, # 5168 - 479,5740,5741, 832,5742,4153,2496,5743,2980,2497,3797, 990,3132, 627,1815,2652, # 5184 -4433,1582,4434,2126,2112,3543,4794,5744, 799,4435,3195,5745,4795,2113,1737,3031, # 5200 -1018, 543, 754,4436,3342,1676,4796,4797,4154,4798,1489,5746,3544,5747,2624,2903, # 5216 -4155,5748,5749,2981,5750,5751,5752,5753,3196,4799,4800,2185,1722,5754,3269,3270, # 5232 -1843,3665,1715, 481, 365,1976,1857,5755,5756,1963,2498,4801,5757,2127,3666,3271, # 5248 - 433,1895,2064,2076,5758, 602,2750,5759,5760,5761,5762,5763,3032,1628,3437,5764, # 5264 -3197,4802,4156,2904,4803,2519,5765,2551,2782,5766,5767,5768,3343,4804,2905,5769, # 5280 -4805,5770,2864,4806,4807,1221,2982,4157,2520,5771,5772,5773,1868,1990,5774,5775, # 5296 -5776,1896,5777,5778,4808,1897,4158, 318,5779,2095,4159,4437,5780,5781, 485,5782, # 5312 - 938,3941, 553,2680, 116,5783,3942,3667,5784,3545,2681,2783,3438,3344,2820,5785, # 5328 -3668,2943,4160,1747,2944,2983,5786,5787, 207,5788,4809,5789,4810,2521,5790,3033, # 5344 - 890,3669,3943,5791,1878,3798,3439,5792,2186,2358,3440,1652,5793,5794,5795, 941, # 5360 -2299, 208,3546,4161,2020, 330,4438,3944,2906,2499,3799,4439,4811,5796,5797,5798, # 5376 -) -# fmt: on diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/big5prober.py b/venv/lib/python3.11/site-packages/pip/_vendor/chardet/big5prober.py deleted file mode 100644 index ef09c60..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/big5prober.py +++ /dev/null @@ -1,47 +0,0 @@ -######################## BEGIN LICENSE BLOCK ######################## -# The Original Code is Mozilla Communicator client code. -# -# The Initial Developer of the Original Code is -# Netscape Communications Corporation. -# Portions created by the Initial Developer are Copyright (C) 1998 -# the Initial Developer. All Rights Reserved. -# -# Contributor(s): -# Mark Pilgrim - port to Python -# -# This library is free software; you can redistribute it and/or -# modify it under the terms of the GNU Lesser General Public -# License as published by the Free Software Foundation; either -# version 2.1 of the License, or (at your option) any later version. -# -# This library is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# Lesser General Public License for more details. -# -# You should have received a copy of the GNU Lesser General Public -# License along with this library; if not, write to the Free Software -# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA -# 02110-1301 USA -######################### END LICENSE BLOCK ######################### - -from .chardistribution import Big5DistributionAnalysis -from .codingstatemachine import CodingStateMachine -from .mbcharsetprober import MultiByteCharSetProber -from .mbcssm import BIG5_SM_MODEL - - -class Big5Prober(MultiByteCharSetProber): - def __init__(self) -> None: - super().__init__() - self.coding_sm = CodingStateMachine(BIG5_SM_MODEL) - self.distribution_analyzer = Big5DistributionAnalysis() - self.reset() - - @property - def charset_name(self) -> str: - return "Big5" - - @property - def language(self) -> str: - return "Chinese" diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/chardistribution.py b/venv/lib/python3.11/site-packages/pip/_vendor/chardet/chardistribution.py deleted file mode 100644 index 176cb99..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/chardistribution.py +++ /dev/null @@ -1,261 +0,0 @@ -######################## BEGIN LICENSE BLOCK ######################## -# The Original Code is Mozilla Communicator client code. -# -# The Initial Developer of the Original Code is -# Netscape Communications Corporation. -# Portions created by the Initial Developer are Copyright (C) 1998 -# the Initial Developer. All Rights Reserved. -# -# Contributor(s): -# Mark Pilgrim - port to Python -# -# This library is free software; you can redistribute it and/or -# modify it under the terms of the GNU Lesser General Public -# License as published by the Free Software Foundation; either -# version 2.1 of the License, or (at your option) any later version. -# -# This library is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# Lesser General Public License for more details. -# -# You should have received a copy of the GNU Lesser General Public -# License along with this library; if not, write to the Free Software -# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA -# 02110-1301 USA -######################### END LICENSE BLOCK ######################### - -from typing import Tuple, Union - -from .big5freq import ( - BIG5_CHAR_TO_FREQ_ORDER, - BIG5_TABLE_SIZE, - BIG5_TYPICAL_DISTRIBUTION_RATIO, -) -from .euckrfreq import ( - EUCKR_CHAR_TO_FREQ_ORDER, - EUCKR_TABLE_SIZE, - EUCKR_TYPICAL_DISTRIBUTION_RATIO, -) -from .euctwfreq import ( - EUCTW_CHAR_TO_FREQ_ORDER, - EUCTW_TABLE_SIZE, - EUCTW_TYPICAL_DISTRIBUTION_RATIO, -) -from .gb2312freq import ( - GB2312_CHAR_TO_FREQ_ORDER, - GB2312_TABLE_SIZE, - GB2312_TYPICAL_DISTRIBUTION_RATIO, -) -from .jisfreq import ( - JIS_CHAR_TO_FREQ_ORDER, - JIS_TABLE_SIZE, - JIS_TYPICAL_DISTRIBUTION_RATIO, -) -from .johabfreq import JOHAB_TO_EUCKR_ORDER_TABLE - - -class CharDistributionAnalysis: - ENOUGH_DATA_THRESHOLD = 1024 - SURE_YES = 0.99 - SURE_NO = 0.01 - MINIMUM_DATA_THRESHOLD = 3 - - def __init__(self) -> None: - # Mapping table to get frequency order from char order (get from - # GetOrder()) - self._char_to_freq_order: Tuple[int, ...] = tuple() - self._table_size = 0 # Size of above table - # This is a constant value which varies from language to language, - # used in calculating confidence. See - # http://www.mozilla.org/projects/intl/UniversalCharsetDetection.html - # for further detail. - self.typical_distribution_ratio = 0.0 - self._done = False - self._total_chars = 0 - self._freq_chars = 0 - self.reset() - - def reset(self) -> None: - """reset analyser, clear any state""" - # If this flag is set to True, detection is done and conclusion has - # been made - self._done = False - self._total_chars = 0 # Total characters encountered - # The number of characters whose frequency order is less than 512 - self._freq_chars = 0 - - def feed(self, char: Union[bytes, bytearray], char_len: int) -> None: - """feed a character with known length""" - if char_len == 2: - # we only care about 2-bytes character in our distribution analysis - order = self.get_order(char) - else: - order = -1 - if order >= 0: - self._total_chars += 1 - # order is valid - if order < self._table_size: - if 512 > self._char_to_freq_order[order]: - self._freq_chars += 1 - - def get_confidence(self) -> float: - """return confidence based on existing data""" - # if we didn't receive any character in our consideration range, - # return negative answer - if self._total_chars <= 0 or self._freq_chars <= self.MINIMUM_DATA_THRESHOLD: - return self.SURE_NO - - if self._total_chars != self._freq_chars: - r = self._freq_chars / ( - (self._total_chars - self._freq_chars) * self.typical_distribution_ratio - ) - if r < self.SURE_YES: - return r - - # normalize confidence (we don't want to be 100% sure) - return self.SURE_YES - - def got_enough_data(self) -> bool: - # It is not necessary to receive all data to draw conclusion. - # For charset detection, certain amount of data is enough - return self._total_chars > self.ENOUGH_DATA_THRESHOLD - - def get_order(self, _: Union[bytes, bytearray]) -> int: - # We do not handle characters based on the original encoding string, - # but convert this encoding string to a number, here called order. - # This allows multiple encodings of a language to share one frequency - # table. - return -1 - - -class EUCTWDistributionAnalysis(CharDistributionAnalysis): - def __init__(self) -> None: - super().__init__() - self._char_to_freq_order = EUCTW_CHAR_TO_FREQ_ORDER - self._table_size = EUCTW_TABLE_SIZE - self.typical_distribution_ratio = EUCTW_TYPICAL_DISTRIBUTION_RATIO - - def get_order(self, byte_str: Union[bytes, bytearray]) -> int: - # for euc-TW encoding, we are interested - # first byte range: 0xc4 -- 0xfe - # second byte range: 0xa1 -- 0xfe - # no validation needed here. State machine has done that - first_char = byte_str[0] - if first_char >= 0xC4: - return 94 * (first_char - 0xC4) + byte_str[1] - 0xA1 - return -1 - - -class EUCKRDistributionAnalysis(CharDistributionAnalysis): - def __init__(self) -> None: - super().__init__() - self._char_to_freq_order = EUCKR_CHAR_TO_FREQ_ORDER - self._table_size = EUCKR_TABLE_SIZE - self.typical_distribution_ratio = EUCKR_TYPICAL_DISTRIBUTION_RATIO - - def get_order(self, byte_str: Union[bytes, bytearray]) -> int: - # for euc-KR encoding, we are interested - # first byte range: 0xb0 -- 0xfe - # second byte range: 0xa1 -- 0xfe - # no validation needed here. State machine has done that - first_char = byte_str[0] - if first_char >= 0xB0: - return 94 * (first_char - 0xB0) + byte_str[1] - 0xA1 - return -1 - - -class JOHABDistributionAnalysis(CharDistributionAnalysis): - def __init__(self) -> None: - super().__init__() - self._char_to_freq_order = EUCKR_CHAR_TO_FREQ_ORDER - self._table_size = EUCKR_TABLE_SIZE - self.typical_distribution_ratio = EUCKR_TYPICAL_DISTRIBUTION_RATIO - - def get_order(self, byte_str: Union[bytes, bytearray]) -> int: - first_char = byte_str[0] - if 0x88 <= first_char < 0xD4: - code = first_char * 256 + byte_str[1] - return JOHAB_TO_EUCKR_ORDER_TABLE.get(code, -1) - return -1 - - -class GB2312DistributionAnalysis(CharDistributionAnalysis): - def __init__(self) -> None: - super().__init__() - self._char_to_freq_order = GB2312_CHAR_TO_FREQ_ORDER - self._table_size = GB2312_TABLE_SIZE - self.typical_distribution_ratio = GB2312_TYPICAL_DISTRIBUTION_RATIO - - def get_order(self, byte_str: Union[bytes, bytearray]) -> int: - # for GB2312 encoding, we are interested - # first byte range: 0xb0 -- 0xfe - # second byte range: 0xa1 -- 0xfe - # no validation needed here. State machine has done that - first_char, second_char = byte_str[0], byte_str[1] - if (first_char >= 0xB0) and (second_char >= 0xA1): - return 94 * (first_char - 0xB0) + second_char - 0xA1 - return -1 - - -class Big5DistributionAnalysis(CharDistributionAnalysis): - def __init__(self) -> None: - super().__init__() - self._char_to_freq_order = BIG5_CHAR_TO_FREQ_ORDER - self._table_size = BIG5_TABLE_SIZE - self.typical_distribution_ratio = BIG5_TYPICAL_DISTRIBUTION_RATIO - - def get_order(self, byte_str: Union[bytes, bytearray]) -> int: - # for big5 encoding, we are interested - # first byte range: 0xa4 -- 0xfe - # second byte range: 0x40 -- 0x7e , 0xa1 -- 0xfe - # no validation needed here. State machine has done that - first_char, second_char = byte_str[0], byte_str[1] - if first_char >= 0xA4: - if second_char >= 0xA1: - return 157 * (first_char - 0xA4) + second_char - 0xA1 + 63 - return 157 * (first_char - 0xA4) + second_char - 0x40 - return -1 - - -class SJISDistributionAnalysis(CharDistributionAnalysis): - def __init__(self) -> None: - super().__init__() - self._char_to_freq_order = JIS_CHAR_TO_FREQ_ORDER - self._table_size = JIS_TABLE_SIZE - self.typical_distribution_ratio = JIS_TYPICAL_DISTRIBUTION_RATIO - - def get_order(self, byte_str: Union[bytes, bytearray]) -> int: - # for sjis encoding, we are interested - # first byte range: 0x81 -- 0x9f , 0xe0 -- 0xfe - # second byte range: 0x40 -- 0x7e, 0x81 -- oxfe - # no validation needed here. State machine has done that - first_char, second_char = byte_str[0], byte_str[1] - if 0x81 <= first_char <= 0x9F: - order = 188 * (first_char - 0x81) - elif 0xE0 <= first_char <= 0xEF: - order = 188 * (first_char - 0xE0 + 31) - else: - return -1 - order = order + second_char - 0x40 - if second_char > 0x7F: - order = -1 - return order - - -class EUCJPDistributionAnalysis(CharDistributionAnalysis): - def __init__(self) -> None: - super().__init__() - self._char_to_freq_order = JIS_CHAR_TO_FREQ_ORDER - self._table_size = JIS_TABLE_SIZE - self.typical_distribution_ratio = JIS_TYPICAL_DISTRIBUTION_RATIO - - def get_order(self, byte_str: Union[bytes, bytearray]) -> int: - # for euc-JP encoding, we are interested - # first byte range: 0xa0 -- 0xfe - # second byte range: 0xa1 -- 0xfe - # no validation needed here. State machine has done that - char = byte_str[0] - if char >= 0xA0: - return 94 * (char - 0xA1) + byte_str[1] - 0xA1 - return -1 diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/charsetgroupprober.py b/venv/lib/python3.11/site-packages/pip/_vendor/chardet/charsetgroupprober.py deleted file mode 100644 index 6def56b..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/charsetgroupprober.py +++ /dev/null @@ -1,106 +0,0 @@ -######################## BEGIN LICENSE BLOCK ######################## -# The Original Code is Mozilla Communicator client code. -# -# The Initial Developer of the Original Code is -# Netscape Communications Corporation. -# Portions created by the Initial Developer are Copyright (C) 1998 -# the Initial Developer. All Rights Reserved. -# -# Contributor(s): -# Mark Pilgrim - port to Python -# -# This library is free software; you can redistribute it and/or -# modify it under the terms of the GNU Lesser General Public -# License as published by the Free Software Foundation; either -# version 2.1 of the License, or (at your option) any later version. -# -# This library is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# Lesser General Public License for more details. -# -# You should have received a copy of the GNU Lesser General Public -# License along with this library; if not, write to the Free Software -# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA -# 02110-1301 USA -######################### END LICENSE BLOCK ######################### - -from typing import List, Optional, Union - -from .charsetprober import CharSetProber -from .enums import LanguageFilter, ProbingState - - -class CharSetGroupProber(CharSetProber): - def __init__(self, lang_filter: LanguageFilter = LanguageFilter.NONE) -> None: - super().__init__(lang_filter=lang_filter) - self._active_num = 0 - self.probers: List[CharSetProber] = [] - self._best_guess_prober: Optional[CharSetProber] = None - - def reset(self) -> None: - super().reset() - self._active_num = 0 - for prober in self.probers: - prober.reset() - prober.active = True - self._active_num += 1 - self._best_guess_prober = None - - @property - def charset_name(self) -> Optional[str]: - if not self._best_guess_prober: - self.get_confidence() - if not self._best_guess_prober: - return None - return self._best_guess_prober.charset_name - - @property - def language(self) -> Optional[str]: - if not self._best_guess_prober: - self.get_confidence() - if not self._best_guess_prober: - return None - return self._best_guess_prober.language - - def feed(self, byte_str: Union[bytes, bytearray]) -> ProbingState: - for prober in self.probers: - if not prober.active: - continue - state = prober.feed(byte_str) - if not state: - continue - if state == ProbingState.FOUND_IT: - self._best_guess_prober = prober - self._state = ProbingState.FOUND_IT - return self.state - if state == ProbingState.NOT_ME: - prober.active = False - self._active_num -= 1 - if self._active_num <= 0: - self._state = ProbingState.NOT_ME - return self.state - return self.state - - def get_confidence(self) -> float: - state = self.state - if state == ProbingState.FOUND_IT: - return 0.99 - if state == ProbingState.NOT_ME: - return 0.01 - best_conf = 0.0 - self._best_guess_prober = None - for prober in self.probers: - if not prober.active: - self.logger.debug("%s not active", prober.charset_name) - continue - conf = prober.get_confidence() - self.logger.debug( - "%s %s confidence = %s", prober.charset_name, prober.language, conf - ) - if best_conf < conf: - best_conf = conf - self._best_guess_prober = prober - if not self._best_guess_prober: - return 0.0 - return best_conf diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/charsetprober.py b/venv/lib/python3.11/site-packages/pip/_vendor/chardet/charsetprober.py deleted file mode 100644 index a103ca1..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/charsetprober.py +++ /dev/null @@ -1,147 +0,0 @@ -######################## BEGIN LICENSE BLOCK ######################## -# The Original Code is Mozilla Universal charset detector code. -# -# The Initial Developer of the Original Code is -# Netscape Communications Corporation. -# Portions created by the Initial Developer are Copyright (C) 2001 -# the Initial Developer. All Rights Reserved. -# -# Contributor(s): -# Mark Pilgrim - port to Python -# Shy Shalom - original C code -# -# This library is free software; you can redistribute it and/or -# modify it under the terms of the GNU Lesser General Public -# License as published by the Free Software Foundation; either -# version 2.1 of the License, or (at your option) any later version. -# -# This library is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# Lesser General Public License for more details. -# -# You should have received a copy of the GNU Lesser General Public -# License along with this library; if not, write to the Free Software -# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA -# 02110-1301 USA -######################### END LICENSE BLOCK ######################### - -import logging -import re -from typing import Optional, Union - -from .enums import LanguageFilter, ProbingState - -INTERNATIONAL_WORDS_PATTERN = re.compile( - b"[a-zA-Z]*[\x80-\xFF]+[a-zA-Z]*[^a-zA-Z\x80-\xFF]?" -) - - -class CharSetProber: - - SHORTCUT_THRESHOLD = 0.95 - - def __init__(self, lang_filter: LanguageFilter = LanguageFilter.NONE) -> None: - self._state = ProbingState.DETECTING - self.active = True - self.lang_filter = lang_filter - self.logger = logging.getLogger(__name__) - - def reset(self) -> None: - self._state = ProbingState.DETECTING - - @property - def charset_name(self) -> Optional[str]: - return None - - @property - def language(self) -> Optional[str]: - raise NotImplementedError - - def feed(self, byte_str: Union[bytes, bytearray]) -> ProbingState: - raise NotImplementedError - - @property - def state(self) -> ProbingState: - return self._state - - def get_confidence(self) -> float: - return 0.0 - - @staticmethod - def filter_high_byte_only(buf: Union[bytes, bytearray]) -> bytes: - buf = re.sub(b"([\x00-\x7F])+", b" ", buf) - return buf - - @staticmethod - def filter_international_words(buf: Union[bytes, bytearray]) -> bytearray: - """ - We define three types of bytes: - alphabet: english alphabets [a-zA-Z] - international: international characters [\x80-\xFF] - marker: everything else [^a-zA-Z\x80-\xFF] - The input buffer can be thought to contain a series of words delimited - by markers. This function works to filter all words that contain at - least one international character. All contiguous sequences of markers - are replaced by a single space ascii character. - This filter applies to all scripts which do not use English characters. - """ - filtered = bytearray() - - # This regex expression filters out only words that have at-least one - # international character. The word may include one marker character at - # the end. - words = INTERNATIONAL_WORDS_PATTERN.findall(buf) - - for word in words: - filtered.extend(word[:-1]) - - # If the last character in the word is a marker, replace it with a - # space as markers shouldn't affect our analysis (they are used - # similarly across all languages and may thus have similar - # frequencies). - last_char = word[-1:] - if not last_char.isalpha() and last_char < b"\x80": - last_char = b" " - filtered.extend(last_char) - - return filtered - - @staticmethod - def remove_xml_tags(buf: Union[bytes, bytearray]) -> bytes: - """ - Returns a copy of ``buf`` that retains only the sequences of English - alphabet and high byte characters that are not between <> characters. - This filter can be applied to all scripts which contain both English - characters and extended ASCII characters, but is currently only used by - ``Latin1Prober``. - """ - filtered = bytearray() - in_tag = False - prev = 0 - buf = memoryview(buf).cast("c") - - for curr, buf_char in enumerate(buf): - # Check if we're coming out of or entering an XML tag - - # https://github.com/python/typeshed/issues/8182 - if buf_char == b">": # type: ignore[comparison-overlap] - prev = curr + 1 - in_tag = False - # https://github.com/python/typeshed/issues/8182 - elif buf_char == b"<": # type: ignore[comparison-overlap] - if curr > prev and not in_tag: - # Keep everything after last non-extended-ASCII, - # non-alphabetic character - filtered.extend(buf[prev:curr]) - # Output a space to delimit stretch we kept - filtered.extend(b" ") - in_tag = True - - # If we're not in a tag... - if not in_tag: - # Keep everything after last non-extended-ASCII, non-alphabetic - # character - filtered.extend(buf[prev:]) - - return filtered diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/cli/__init__.py b/venv/lib/python3.11/site-packages/pip/_vendor/chardet/cli/__init__.py deleted file mode 100644 index e69de29..0000000 diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/cli/__pycache__/__init__.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/chardet/cli/__pycache__/__init__.cpython-311.pyc deleted file mode 100644 index 57e5527..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/cli/__pycache__/__init__.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/cli/__pycache__/chardetect.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/chardet/cli/__pycache__/chardetect.cpython-311.pyc deleted file mode 100644 index a109c23..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/cli/__pycache__/chardetect.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/cli/chardetect.py b/venv/lib/python3.11/site-packages/pip/_vendor/chardet/cli/chardetect.py deleted file mode 100644 index 43f6e14..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/cli/chardetect.py +++ /dev/null @@ -1,112 +0,0 @@ -""" -Script which takes one or more file paths and reports on their detected -encodings - -Example:: - - % chardetect somefile someotherfile - somefile: windows-1252 with confidence 0.5 - someotherfile: ascii with confidence 1.0 - -If no paths are provided, it takes its input from stdin. - -""" - - -import argparse -import sys -from typing import Iterable, List, Optional - -from .. import __version__ -from ..universaldetector import UniversalDetector - - -def description_of( - lines: Iterable[bytes], - name: str = "stdin", - minimal: bool = False, - should_rename_legacy: bool = False, -) -> Optional[str]: - """ - Return a string describing the probable encoding of a file or - list of strings. - - :param lines: The lines to get the encoding of. - :type lines: Iterable of bytes - :param name: Name of file or collection of lines - :type name: str - :param should_rename_legacy: Should we rename legacy encodings to - their more modern equivalents? - :type should_rename_legacy: ``bool`` - """ - u = UniversalDetector(should_rename_legacy=should_rename_legacy) - for line in lines: - line = bytearray(line) - u.feed(line) - # shortcut out of the loop to save reading further - particularly useful if we read a BOM. - if u.done: - break - u.close() - result = u.result - if minimal: - return result["encoding"] - if result["encoding"]: - return f'{name}: {result["encoding"]} with confidence {result["confidence"]}' - return f"{name}: no result" - - -def main(argv: Optional[List[str]] = None) -> None: - """ - Handles command line arguments and gets things started. - - :param argv: List of arguments, as if specified on the command-line. - If None, ``sys.argv[1:]`` is used instead. - :type argv: list of str - """ - # Get command line arguments - parser = argparse.ArgumentParser( - description=( - "Takes one or more file paths and reports their detected encodings" - ) - ) - parser.add_argument( - "input", - help="File whose encoding we would like to determine. (default: stdin)", - type=argparse.FileType("rb"), - nargs="*", - default=[sys.stdin.buffer], - ) - parser.add_argument( - "--minimal", - help="Print only the encoding to standard output", - action="store_true", - ) - parser.add_argument( - "-l", - "--legacy", - help="Rename legacy encodings to more modern ones.", - action="store_true", - ) - parser.add_argument( - "--version", action="version", version=f"%(prog)s {__version__}" - ) - args = parser.parse_args(argv) - - for f in args.input: - if f.isatty(): - print( - "You are running chardetect interactively. Press " - "CTRL-D twice at the start of a blank line to signal the " - "end of your input. If you want help, run chardetect " - "--help\n", - file=sys.stderr, - ) - print( - description_of( - f, f.name, minimal=args.minimal, should_rename_legacy=args.legacy - ) - ) - - -if __name__ == "__main__": - main() diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/codingstatemachine.py b/venv/lib/python3.11/site-packages/pip/_vendor/chardet/codingstatemachine.py deleted file mode 100644 index 8ed4a87..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/codingstatemachine.py +++ /dev/null @@ -1,90 +0,0 @@ -######################## BEGIN LICENSE BLOCK ######################## -# The Original Code is mozilla.org code. -# -# The Initial Developer of the Original Code is -# Netscape Communications Corporation. -# Portions created by the Initial Developer are Copyright (C) 1998 -# the Initial Developer. All Rights Reserved. -# -# Contributor(s): -# Mark Pilgrim - port to Python -# -# This library is free software; you can redistribute it and/or -# modify it under the terms of the GNU Lesser General Public -# License as published by the Free Software Foundation; either -# version 2.1 of the License, or (at your option) any later version. -# -# This library is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# Lesser General Public License for more details. -# -# You should have received a copy of the GNU Lesser General Public -# License along with this library; if not, write to the Free Software -# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA -# 02110-1301 USA -######################### END LICENSE BLOCK ######################### - -import logging - -from .codingstatemachinedict import CodingStateMachineDict -from .enums import MachineState - - -class CodingStateMachine: - """ - A state machine to verify a byte sequence for a particular encoding. For - each byte the detector receives, it will feed that byte to every active - state machine available, one byte at a time. The state machine changes its - state based on its previous state and the byte it receives. There are 3 - states in a state machine that are of interest to an auto-detector: - - START state: This is the state to start with, or a legal byte sequence - (i.e. a valid code point) for character has been identified. - - ME state: This indicates that the state machine identified a byte sequence - that is specific to the charset it is designed for and that - there is no other possible encoding which can contain this byte - sequence. This will to lead to an immediate positive answer for - the detector. - - ERROR state: This indicates the state machine identified an illegal byte - sequence for that encoding. This will lead to an immediate - negative answer for this encoding. Detector will exclude this - encoding from consideration from here on. - """ - - def __init__(self, sm: CodingStateMachineDict) -> None: - self._model = sm - self._curr_byte_pos = 0 - self._curr_char_len = 0 - self._curr_state = MachineState.START - self.active = True - self.logger = logging.getLogger(__name__) - self.reset() - - def reset(self) -> None: - self._curr_state = MachineState.START - - def next_state(self, c: int) -> int: - # for each byte we get its class - # if it is first byte, we also get byte length - byte_class = self._model["class_table"][c] - if self._curr_state == MachineState.START: - self._curr_byte_pos = 0 - self._curr_char_len = self._model["char_len_table"][byte_class] - # from byte's class and state_table, we get its next state - curr_state = self._curr_state * self._model["class_factor"] + byte_class - self._curr_state = self._model["state_table"][curr_state] - self._curr_byte_pos += 1 - return self._curr_state - - def get_current_charlen(self) -> int: - return self._curr_char_len - - def get_coding_state_machine(self) -> str: - return self._model["name"] - - @property - def language(self) -> str: - return self._model["language"] diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/codingstatemachinedict.py b/venv/lib/python3.11/site-packages/pip/_vendor/chardet/codingstatemachinedict.py deleted file mode 100644 index 7a3c4c7..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/codingstatemachinedict.py +++ /dev/null @@ -1,19 +0,0 @@ -from typing import TYPE_CHECKING, Tuple - -if TYPE_CHECKING: - # TypedDict was introduced in Python 3.8. - # - # TODO: Remove the else block and TYPE_CHECKING check when dropping support - # for Python 3.7. - from typing import TypedDict - - class CodingStateMachineDict(TypedDict, total=False): - class_table: Tuple[int, ...] - class_factor: int - state_table: Tuple[int, ...] - char_len_table: Tuple[int, ...] - name: str - language: str # Optional key - -else: - CodingStateMachineDict = dict diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/cp949prober.py b/venv/lib/python3.11/site-packages/pip/_vendor/chardet/cp949prober.py deleted file mode 100644 index fa7307e..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/cp949prober.py +++ /dev/null @@ -1,49 +0,0 @@ -######################## BEGIN LICENSE BLOCK ######################## -# The Original Code is mozilla.org code. -# -# The Initial Developer of the Original Code is -# Netscape Communications Corporation. -# Portions created by the Initial Developer are Copyright (C) 1998 -# the Initial Developer. All Rights Reserved. -# -# Contributor(s): -# Mark Pilgrim - port to Python -# -# This library is free software; you can redistribute it and/or -# modify it under the terms of the GNU Lesser General Public -# License as published by the Free Software Foundation; either -# version 2.1 of the License, or (at your option) any later version. -# -# This library is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# Lesser General Public License for more details. -# -# You should have received a copy of the GNU Lesser General Public -# License along with this library; if not, write to the Free Software -# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA -# 02110-1301 USA -######################### END LICENSE BLOCK ######################### - -from .chardistribution import EUCKRDistributionAnalysis -from .codingstatemachine import CodingStateMachine -from .mbcharsetprober import MultiByteCharSetProber -from .mbcssm import CP949_SM_MODEL - - -class CP949Prober(MultiByteCharSetProber): - def __init__(self) -> None: - super().__init__() - self.coding_sm = CodingStateMachine(CP949_SM_MODEL) - # NOTE: CP949 is a superset of EUC-KR, so the distribution should be - # not different. - self.distribution_analyzer = EUCKRDistributionAnalysis() - self.reset() - - @property - def charset_name(self) -> str: - return "CP949" - - @property - def language(self) -> str: - return "Korean" diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/enums.py b/venv/lib/python3.11/site-packages/pip/_vendor/chardet/enums.py deleted file mode 100644 index 5e3e198..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/enums.py +++ /dev/null @@ -1,85 +0,0 @@ -""" -All of the Enums that are used throughout the chardet package. - -:author: Dan Blanchard (dan.blanchard@gmail.com) -""" - -from enum import Enum, Flag - - -class InputState: - """ - This enum represents the different states a universal detector can be in. - """ - - PURE_ASCII = 0 - ESC_ASCII = 1 - HIGH_BYTE = 2 - - -class LanguageFilter(Flag): - """ - This enum represents the different language filters we can apply to a - ``UniversalDetector``. - """ - - NONE = 0x00 - CHINESE_SIMPLIFIED = 0x01 - CHINESE_TRADITIONAL = 0x02 - JAPANESE = 0x04 - KOREAN = 0x08 - NON_CJK = 0x10 - ALL = 0x1F - CHINESE = CHINESE_SIMPLIFIED | CHINESE_TRADITIONAL - CJK = CHINESE | JAPANESE | KOREAN - - -class ProbingState(Enum): - """ - This enum represents the different states a prober can be in. - """ - - DETECTING = 0 - FOUND_IT = 1 - NOT_ME = 2 - - -class MachineState: - """ - This enum represents the different states a state machine can be in. - """ - - START = 0 - ERROR = 1 - ITS_ME = 2 - - -class SequenceLikelihood: - """ - This enum represents the likelihood of a character following the previous one. - """ - - NEGATIVE = 0 - UNLIKELY = 1 - LIKELY = 2 - POSITIVE = 3 - - @classmethod - def get_num_categories(cls) -> int: - """:returns: The number of likelihood categories in the enum.""" - return 4 - - -class CharacterCategory: - """ - This enum represents the different categories language models for - ``SingleByteCharsetProber`` put characters into. - - Anything less than CONTROL is considered a letter. - """ - - UNDEFINED = 255 - LINE_BREAK = 254 - SYMBOL = 253 - DIGIT = 252 - CONTROL = 251 diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/escprober.py b/venv/lib/python3.11/site-packages/pip/_vendor/chardet/escprober.py deleted file mode 100644 index fd71383..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/escprober.py +++ /dev/null @@ -1,102 +0,0 @@ -######################## BEGIN LICENSE BLOCK ######################## -# The Original Code is mozilla.org code. -# -# The Initial Developer of the Original Code is -# Netscape Communications Corporation. -# Portions created by the Initial Developer are Copyright (C) 1998 -# the Initial Developer. All Rights Reserved. -# -# Contributor(s): -# Mark Pilgrim - port to Python -# -# This library is free software; you can redistribute it and/or -# modify it under the terms of the GNU Lesser General Public -# License as published by the Free Software Foundation; either -# version 2.1 of the License, or (at your option) any later version. -# -# This library is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# Lesser General Public License for more details. -# -# You should have received a copy of the GNU Lesser General Public -# License along with this library; if not, write to the Free Software -# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA -# 02110-1301 USA -######################### END LICENSE BLOCK ######################### - -from typing import Optional, Union - -from .charsetprober import CharSetProber -from .codingstatemachine import CodingStateMachine -from .enums import LanguageFilter, MachineState, ProbingState -from .escsm import ( - HZ_SM_MODEL, - ISO2022CN_SM_MODEL, - ISO2022JP_SM_MODEL, - ISO2022KR_SM_MODEL, -) - - -class EscCharSetProber(CharSetProber): - """ - This CharSetProber uses a "code scheme" approach for detecting encodings, - whereby easily recognizable escape or shift sequences are relied on to - identify these encodings. - """ - - def __init__(self, lang_filter: LanguageFilter = LanguageFilter.NONE) -> None: - super().__init__(lang_filter=lang_filter) - self.coding_sm = [] - if self.lang_filter & LanguageFilter.CHINESE_SIMPLIFIED: - self.coding_sm.append(CodingStateMachine(HZ_SM_MODEL)) - self.coding_sm.append(CodingStateMachine(ISO2022CN_SM_MODEL)) - if self.lang_filter & LanguageFilter.JAPANESE: - self.coding_sm.append(CodingStateMachine(ISO2022JP_SM_MODEL)) - if self.lang_filter & LanguageFilter.KOREAN: - self.coding_sm.append(CodingStateMachine(ISO2022KR_SM_MODEL)) - self.active_sm_count = 0 - self._detected_charset: Optional[str] = None - self._detected_language: Optional[str] = None - self._state = ProbingState.DETECTING - self.reset() - - def reset(self) -> None: - super().reset() - for coding_sm in self.coding_sm: - coding_sm.active = True - coding_sm.reset() - self.active_sm_count = len(self.coding_sm) - self._detected_charset = None - self._detected_language = None - - @property - def charset_name(self) -> Optional[str]: - return self._detected_charset - - @property - def language(self) -> Optional[str]: - return self._detected_language - - def get_confidence(self) -> float: - return 0.99 if self._detected_charset else 0.00 - - def feed(self, byte_str: Union[bytes, bytearray]) -> ProbingState: - for c in byte_str: - for coding_sm in self.coding_sm: - if not coding_sm.active: - continue - coding_state = coding_sm.next_state(c) - if coding_state == MachineState.ERROR: - coding_sm.active = False - self.active_sm_count -= 1 - if self.active_sm_count <= 0: - self._state = ProbingState.NOT_ME - return self.state - elif coding_state == MachineState.ITS_ME: - self._state = ProbingState.FOUND_IT - self._detected_charset = coding_sm.get_coding_state_machine() - self._detected_language = coding_sm.language - return self.state - - return self.state diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/escsm.py b/venv/lib/python3.11/site-packages/pip/_vendor/chardet/escsm.py deleted file mode 100644 index 11d4adf..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/escsm.py +++ /dev/null @@ -1,261 +0,0 @@ -######################## BEGIN LICENSE BLOCK ######################## -# The Original Code is mozilla.org code. -# -# The Initial Developer of the Original Code is -# Netscape Communications Corporation. -# Portions created by the Initial Developer are Copyright (C) 1998 -# the Initial Developer. All Rights Reserved. -# -# Contributor(s): -# Mark Pilgrim - port to Python -# -# This library is free software; you can redistribute it and/or -# modify it under the terms of the GNU Lesser General Public -# License as published by the Free Software Foundation; either -# version 2.1 of the License, or (at your option) any later version. -# -# This library is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# Lesser General Public License for more details. -# -# You should have received a copy of the GNU Lesser General Public -# License along with this library; if not, write to the Free Software -# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA -# 02110-1301 USA -######################### END LICENSE BLOCK ######################### - -from .codingstatemachinedict import CodingStateMachineDict -from .enums import MachineState - -# fmt: off -HZ_CLS = ( - 1, 0, 0, 0, 0, 0, 0, 0, # 00 - 07 - 0, 0, 0, 0, 0, 0, 0, 0, # 08 - 0f - 0, 0, 0, 0, 0, 0, 0, 0, # 10 - 17 - 0, 0, 0, 1, 0, 0, 0, 0, # 18 - 1f - 0, 0, 0, 0, 0, 0, 0, 0, # 20 - 27 - 0, 0, 0, 0, 0, 0, 0, 0, # 28 - 2f - 0, 0, 0, 0, 0, 0, 0, 0, # 30 - 37 - 0, 0, 0, 0, 0, 0, 0, 0, # 38 - 3f - 0, 0, 0, 0, 0, 0, 0, 0, # 40 - 47 - 0, 0, 0, 0, 0, 0, 0, 0, # 48 - 4f - 0, 0, 0, 0, 0, 0, 0, 0, # 50 - 57 - 0, 0, 0, 0, 0, 0, 0, 0, # 58 - 5f - 0, 0, 0, 0, 0, 0, 0, 0, # 60 - 67 - 0, 0, 0, 0, 0, 0, 0, 0, # 68 - 6f - 0, 0, 0, 0, 0, 0, 0, 0, # 70 - 77 - 0, 0, 0, 4, 0, 5, 2, 0, # 78 - 7f - 1, 1, 1, 1, 1, 1, 1, 1, # 80 - 87 - 1, 1, 1, 1, 1, 1, 1, 1, # 88 - 8f - 1, 1, 1, 1, 1, 1, 1, 1, # 90 - 97 - 1, 1, 1, 1, 1, 1, 1, 1, # 98 - 9f - 1, 1, 1, 1, 1, 1, 1, 1, # a0 - a7 - 1, 1, 1, 1, 1, 1, 1, 1, # a8 - af - 1, 1, 1, 1, 1, 1, 1, 1, # b0 - b7 - 1, 1, 1, 1, 1, 1, 1, 1, # b8 - bf - 1, 1, 1, 1, 1, 1, 1, 1, # c0 - c7 - 1, 1, 1, 1, 1, 1, 1, 1, # c8 - cf - 1, 1, 1, 1, 1, 1, 1, 1, # d0 - d7 - 1, 1, 1, 1, 1, 1, 1, 1, # d8 - df - 1, 1, 1, 1, 1, 1, 1, 1, # e0 - e7 - 1, 1, 1, 1, 1, 1, 1, 1, # e8 - ef - 1, 1, 1, 1, 1, 1, 1, 1, # f0 - f7 - 1, 1, 1, 1, 1, 1, 1, 1, # f8 - ff -) - -HZ_ST = ( -MachineState.START, MachineState.ERROR, 3, MachineState.START, MachineState.START, MachineState.START, MachineState.ERROR, MachineState.ERROR, # 00-07 -MachineState.ERROR, MachineState.ERROR, MachineState.ERROR, MachineState.ERROR, MachineState.ITS_ME, MachineState.ITS_ME, MachineState.ITS_ME, MachineState.ITS_ME, # 08-0f -MachineState.ITS_ME, MachineState.ITS_ME, MachineState.ERROR, MachineState.ERROR, MachineState.START, MachineState.START, 4, MachineState.ERROR, # 10-17 - 5, MachineState.ERROR, 6, MachineState.ERROR, 5, 5, 4, MachineState.ERROR, # 18-1f - 4, MachineState.ERROR, 4, 4, 4, MachineState.ERROR, 4, MachineState.ERROR, # 20-27 - 4, MachineState.ITS_ME, MachineState.START, MachineState.START, MachineState.START, MachineState.START, MachineState.START, MachineState.START, # 28-2f -) -# fmt: on - -HZ_CHAR_LEN_TABLE = (0, 0, 0, 0, 0, 0) - -HZ_SM_MODEL: CodingStateMachineDict = { - "class_table": HZ_CLS, - "class_factor": 6, - "state_table": HZ_ST, - "char_len_table": HZ_CHAR_LEN_TABLE, - "name": "HZ-GB-2312", - "language": "Chinese", -} - -# fmt: off -ISO2022CN_CLS = ( - 2, 0, 0, 0, 0, 0, 0, 0, # 00 - 07 - 0, 0, 0, 0, 0, 0, 0, 0, # 08 - 0f - 0, 0, 0, 0, 0, 0, 0, 0, # 10 - 17 - 0, 0, 0, 1, 0, 0, 0, 0, # 18 - 1f - 0, 0, 0, 0, 0, 0, 0, 0, # 20 - 27 - 0, 3, 0, 0, 0, 0, 0, 0, # 28 - 2f - 0, 0, 0, 0, 0, 0, 0, 0, # 30 - 37 - 0, 0, 0, 0, 0, 0, 0, 0, # 38 - 3f - 0, 0, 0, 4, 0, 0, 0, 0, # 40 - 47 - 0, 0, 0, 0, 0, 0, 0, 0, # 48 - 4f - 0, 0, 0, 0, 0, 0, 0, 0, # 50 - 57 - 0, 0, 0, 0, 0, 0, 0, 0, # 58 - 5f - 0, 0, 0, 0, 0, 0, 0, 0, # 60 - 67 - 0, 0, 0, 0, 0, 0, 0, 0, # 68 - 6f - 0, 0, 0, 0, 0, 0, 0, 0, # 70 - 77 - 0, 0, 0, 0, 0, 0, 0, 0, # 78 - 7f - 2, 2, 2, 2, 2, 2, 2, 2, # 80 - 87 - 2, 2, 2, 2, 2, 2, 2, 2, # 88 - 8f - 2, 2, 2, 2, 2, 2, 2, 2, # 90 - 97 - 2, 2, 2, 2, 2, 2, 2, 2, # 98 - 9f - 2, 2, 2, 2, 2, 2, 2, 2, # a0 - a7 - 2, 2, 2, 2, 2, 2, 2, 2, # a8 - af - 2, 2, 2, 2, 2, 2, 2, 2, # b0 - b7 - 2, 2, 2, 2, 2, 2, 2, 2, # b8 - bf - 2, 2, 2, 2, 2, 2, 2, 2, # c0 - c7 - 2, 2, 2, 2, 2, 2, 2, 2, # c8 - cf - 2, 2, 2, 2, 2, 2, 2, 2, # d0 - d7 - 2, 2, 2, 2, 2, 2, 2, 2, # d8 - df - 2, 2, 2, 2, 2, 2, 2, 2, # e0 - e7 - 2, 2, 2, 2, 2, 2, 2, 2, # e8 - ef - 2, 2, 2, 2, 2, 2, 2, 2, # f0 - f7 - 2, 2, 2, 2, 2, 2, 2, 2, # f8 - ff -) - -ISO2022CN_ST = ( - MachineState.START, 3, MachineState.ERROR, MachineState.START, MachineState.START, MachineState.START, MachineState.START, MachineState.START, # 00-07 - MachineState.START, MachineState.ERROR, MachineState.ERROR, MachineState.ERROR, MachineState.ERROR, MachineState.ERROR, MachineState.ERROR, MachineState.ERROR, # 08-0f - MachineState.ERROR, MachineState.ERROR, MachineState.ITS_ME, MachineState.ITS_ME, MachineState.ITS_ME, MachineState.ITS_ME, MachineState.ITS_ME, MachineState.ITS_ME, # 10-17 - MachineState.ITS_ME, MachineState.ITS_ME, MachineState.ITS_ME, MachineState.ERROR, MachineState.ERROR, MachineState.ERROR, 4, MachineState.ERROR, # 18-1f - MachineState.ERROR, MachineState.ERROR, MachineState.ERROR, MachineState.ITS_ME, MachineState.ERROR, MachineState.ERROR, MachineState.ERROR, MachineState.ERROR, # 20-27 - 5, 6, MachineState.ERROR, MachineState.ERROR, MachineState.ERROR, MachineState.ERROR, MachineState.ERROR, MachineState.ERROR, # 28-2f - MachineState.ERROR, MachineState.ERROR, MachineState.ERROR, MachineState.ITS_ME, MachineState.ERROR, MachineState.ERROR, MachineState.ERROR, MachineState.ERROR, # 30-37 - MachineState.ERROR, MachineState.ERROR, MachineState.ERROR, MachineState.ERROR, MachineState.ERROR, MachineState.ITS_ME, MachineState.ERROR, MachineState.START, # 38-3f -) -# fmt: on - -ISO2022CN_CHAR_LEN_TABLE = (0, 0, 0, 0, 0, 0, 0, 0, 0) - -ISO2022CN_SM_MODEL: CodingStateMachineDict = { - "class_table": ISO2022CN_CLS, - "class_factor": 9, - "state_table": ISO2022CN_ST, - "char_len_table": ISO2022CN_CHAR_LEN_TABLE, - "name": "ISO-2022-CN", - "language": "Chinese", -} - -# fmt: off -ISO2022JP_CLS = ( - 2, 0, 0, 0, 0, 0, 0, 0, # 00 - 07 - 0, 0, 0, 0, 0, 0, 2, 2, # 08 - 0f - 0, 0, 0, 0, 0, 0, 0, 0, # 10 - 17 - 0, 0, 0, 1, 0, 0, 0, 0, # 18 - 1f - 0, 0, 0, 0, 7, 0, 0, 0, # 20 - 27 - 3, 0, 0, 0, 0, 0, 0, 0, # 28 - 2f - 0, 0, 0, 0, 0, 0, 0, 0, # 30 - 37 - 0, 0, 0, 0, 0, 0, 0, 0, # 38 - 3f - 6, 0, 4, 0, 8, 0, 0, 0, # 40 - 47 - 0, 9, 5, 0, 0, 0, 0, 0, # 48 - 4f - 0, 0, 0, 0, 0, 0, 0, 0, # 50 - 57 - 0, 0, 0, 0, 0, 0, 0, 0, # 58 - 5f - 0, 0, 0, 0, 0, 0, 0, 0, # 60 - 67 - 0, 0, 0, 0, 0, 0, 0, 0, # 68 - 6f - 0, 0, 0, 0, 0, 0, 0, 0, # 70 - 77 - 0, 0, 0, 0, 0, 0, 0, 0, # 78 - 7f - 2, 2, 2, 2, 2, 2, 2, 2, # 80 - 87 - 2, 2, 2, 2, 2, 2, 2, 2, # 88 - 8f - 2, 2, 2, 2, 2, 2, 2, 2, # 90 - 97 - 2, 2, 2, 2, 2, 2, 2, 2, # 98 - 9f - 2, 2, 2, 2, 2, 2, 2, 2, # a0 - a7 - 2, 2, 2, 2, 2, 2, 2, 2, # a8 - af - 2, 2, 2, 2, 2, 2, 2, 2, # b0 - b7 - 2, 2, 2, 2, 2, 2, 2, 2, # b8 - bf - 2, 2, 2, 2, 2, 2, 2, 2, # c0 - c7 - 2, 2, 2, 2, 2, 2, 2, 2, # c8 - cf - 2, 2, 2, 2, 2, 2, 2, 2, # d0 - d7 - 2, 2, 2, 2, 2, 2, 2, 2, # d8 - df - 2, 2, 2, 2, 2, 2, 2, 2, # e0 - e7 - 2, 2, 2, 2, 2, 2, 2, 2, # e8 - ef - 2, 2, 2, 2, 2, 2, 2, 2, # f0 - f7 - 2, 2, 2, 2, 2, 2, 2, 2, # f8 - ff -) - -ISO2022JP_ST = ( - MachineState.START, 3, MachineState.ERROR, MachineState.START, MachineState.START, MachineState.START, MachineState.START, MachineState.START, # 00-07 - MachineState.START, MachineState.START, MachineState.ERROR, MachineState.ERROR, MachineState.ERROR, MachineState.ERROR, MachineState.ERROR, MachineState.ERROR, # 08-0f - MachineState.ERROR, MachineState.ERROR, MachineState.ERROR, MachineState.ERROR, MachineState.ITS_ME, MachineState.ITS_ME, MachineState.ITS_ME, MachineState.ITS_ME, # 10-17 - MachineState.ITS_ME, MachineState.ITS_ME, MachineState.ITS_ME, MachineState.ITS_ME, MachineState.ITS_ME, MachineState.ITS_ME, MachineState.ERROR, MachineState.ERROR, # 18-1f - MachineState.ERROR, 5, MachineState.ERROR, MachineState.ERROR, MachineState.ERROR, 4, MachineState.ERROR, MachineState.ERROR, # 20-27 - MachineState.ERROR, MachineState.ERROR, MachineState.ERROR, 6, MachineState.ITS_ME, MachineState.ERROR, MachineState.ITS_ME, MachineState.ERROR, # 28-2f - MachineState.ERROR, MachineState.ERROR, MachineState.ERROR, MachineState.ERROR, MachineState.ERROR, MachineState.ERROR, MachineState.ITS_ME, MachineState.ITS_ME, # 30-37 - MachineState.ERROR, MachineState.ERROR, MachineState.ERROR, MachineState.ITS_ME, MachineState.ERROR, MachineState.ERROR, MachineState.ERROR, MachineState.ERROR, # 38-3f - MachineState.ERROR, MachineState.ERROR, MachineState.ERROR, MachineState.ERROR, MachineState.ITS_ME, MachineState.ERROR, MachineState.START, MachineState.START, # 40-47 -) -# fmt: on - -ISO2022JP_CHAR_LEN_TABLE = (0, 0, 0, 0, 0, 0, 0, 0, 0, 0) - -ISO2022JP_SM_MODEL: CodingStateMachineDict = { - "class_table": ISO2022JP_CLS, - "class_factor": 10, - "state_table": ISO2022JP_ST, - "char_len_table": ISO2022JP_CHAR_LEN_TABLE, - "name": "ISO-2022-JP", - "language": "Japanese", -} - -# fmt: off -ISO2022KR_CLS = ( - 2, 0, 0, 0, 0, 0, 0, 0, # 00 - 07 - 0, 0, 0, 0, 0, 0, 0, 0, # 08 - 0f - 0, 0, 0, 0, 0, 0, 0, 0, # 10 - 17 - 0, 0, 0, 1, 0, 0, 0, 0, # 18 - 1f - 0, 0, 0, 0, 3, 0, 0, 0, # 20 - 27 - 0, 4, 0, 0, 0, 0, 0, 0, # 28 - 2f - 0, 0, 0, 0, 0, 0, 0, 0, # 30 - 37 - 0, 0, 0, 0, 0, 0, 0, 0, # 38 - 3f - 0, 0, 0, 5, 0, 0, 0, 0, # 40 - 47 - 0, 0, 0, 0, 0, 0, 0, 0, # 48 - 4f - 0, 0, 0, 0, 0, 0, 0, 0, # 50 - 57 - 0, 0, 0, 0, 0, 0, 0, 0, # 58 - 5f - 0, 0, 0, 0, 0, 0, 0, 0, # 60 - 67 - 0, 0, 0, 0, 0, 0, 0, 0, # 68 - 6f - 0, 0, 0, 0, 0, 0, 0, 0, # 70 - 77 - 0, 0, 0, 0, 0, 0, 0, 0, # 78 - 7f - 2, 2, 2, 2, 2, 2, 2, 2, # 80 - 87 - 2, 2, 2, 2, 2, 2, 2, 2, # 88 - 8f - 2, 2, 2, 2, 2, 2, 2, 2, # 90 - 97 - 2, 2, 2, 2, 2, 2, 2, 2, # 98 - 9f - 2, 2, 2, 2, 2, 2, 2, 2, # a0 - a7 - 2, 2, 2, 2, 2, 2, 2, 2, # a8 - af - 2, 2, 2, 2, 2, 2, 2, 2, # b0 - b7 - 2, 2, 2, 2, 2, 2, 2, 2, # b8 - bf - 2, 2, 2, 2, 2, 2, 2, 2, # c0 - c7 - 2, 2, 2, 2, 2, 2, 2, 2, # c8 - cf - 2, 2, 2, 2, 2, 2, 2, 2, # d0 - d7 - 2, 2, 2, 2, 2, 2, 2, 2, # d8 - df - 2, 2, 2, 2, 2, 2, 2, 2, # e0 - e7 - 2, 2, 2, 2, 2, 2, 2, 2, # e8 - ef - 2, 2, 2, 2, 2, 2, 2, 2, # f0 - f7 - 2, 2, 2, 2, 2, 2, 2, 2, # f8 - ff -) - -ISO2022KR_ST = ( - MachineState.START, 3, MachineState.ERROR, MachineState.START, MachineState.START, MachineState.START, MachineState.ERROR, MachineState.ERROR, # 00-07 - MachineState.ERROR, MachineState.ERROR, MachineState.ERROR, MachineState.ERROR, MachineState.ITS_ME, MachineState.ITS_ME, MachineState.ITS_ME, MachineState.ITS_ME, # 08-0f - MachineState.ITS_ME, MachineState.ITS_ME, MachineState.ERROR, MachineState.ERROR, MachineState.ERROR, 4, MachineState.ERROR, MachineState.ERROR, # 10-17 - MachineState.ERROR, MachineState.ERROR, MachineState.ERROR, MachineState.ERROR, 5, MachineState.ERROR, MachineState.ERROR, MachineState.ERROR, # 18-1f - MachineState.ERROR, MachineState.ERROR, MachineState.ERROR, MachineState.ITS_ME, MachineState.START, MachineState.START, MachineState.START, MachineState.START, # 20-27 -) -# fmt: on - -ISO2022KR_CHAR_LEN_TABLE = (0, 0, 0, 0, 0, 0) - -ISO2022KR_SM_MODEL: CodingStateMachineDict = { - "class_table": ISO2022KR_CLS, - "class_factor": 6, - "state_table": ISO2022KR_ST, - "char_len_table": ISO2022KR_CHAR_LEN_TABLE, - "name": "ISO-2022-KR", - "language": "Korean", -} diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/eucjpprober.py b/venv/lib/python3.11/site-packages/pip/_vendor/chardet/eucjpprober.py deleted file mode 100644 index 39487f4..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/eucjpprober.py +++ /dev/null @@ -1,102 +0,0 @@ -######################## BEGIN LICENSE BLOCK ######################## -# The Original Code is mozilla.org code. -# -# The Initial Developer of the Original Code is -# Netscape Communications Corporation. -# Portions created by the Initial Developer are Copyright (C) 1998 -# the Initial Developer. All Rights Reserved. -# -# Contributor(s): -# Mark Pilgrim - port to Python -# -# This library is free software; you can redistribute it and/or -# modify it under the terms of the GNU Lesser General Public -# License as published by the Free Software Foundation; either -# version 2.1 of the License, or (at your option) any later version. -# -# This library is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# Lesser General Public License for more details. -# -# You should have received a copy of the GNU Lesser General Public -# License along with this library; if not, write to the Free Software -# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA -# 02110-1301 USA -######################### END LICENSE BLOCK ######################### - -from typing import Union - -from .chardistribution import EUCJPDistributionAnalysis -from .codingstatemachine import CodingStateMachine -from .enums import MachineState, ProbingState -from .jpcntx import EUCJPContextAnalysis -from .mbcharsetprober import MultiByteCharSetProber -from .mbcssm import EUCJP_SM_MODEL - - -class EUCJPProber(MultiByteCharSetProber): - def __init__(self) -> None: - super().__init__() - self.coding_sm = CodingStateMachine(EUCJP_SM_MODEL) - self.distribution_analyzer = EUCJPDistributionAnalysis() - self.context_analyzer = EUCJPContextAnalysis() - self.reset() - - def reset(self) -> None: - super().reset() - self.context_analyzer.reset() - - @property - def charset_name(self) -> str: - return "EUC-JP" - - @property - def language(self) -> str: - return "Japanese" - - def feed(self, byte_str: Union[bytes, bytearray]) -> ProbingState: - assert self.coding_sm is not None - assert self.distribution_analyzer is not None - - for i, byte in enumerate(byte_str): - # PY3K: byte_str is a byte array, so byte is an int, not a byte - coding_state = self.coding_sm.next_state(byte) - if coding_state == MachineState.ERROR: - self.logger.debug( - "%s %s prober hit error at byte %s", - self.charset_name, - self.language, - i, - ) - self._state = ProbingState.NOT_ME - break - if coding_state == MachineState.ITS_ME: - self._state = ProbingState.FOUND_IT - break - if coding_state == MachineState.START: - char_len = self.coding_sm.get_current_charlen() - if i == 0: - self._last_char[1] = byte - self.context_analyzer.feed(self._last_char, char_len) - self.distribution_analyzer.feed(self._last_char, char_len) - else: - self.context_analyzer.feed(byte_str[i - 1 : i + 1], char_len) - self.distribution_analyzer.feed(byte_str[i - 1 : i + 1], char_len) - - self._last_char[0] = byte_str[-1] - - if self.state == ProbingState.DETECTING: - if self.context_analyzer.got_enough_data() and ( - self.get_confidence() > self.SHORTCUT_THRESHOLD - ): - self._state = ProbingState.FOUND_IT - - return self.state - - def get_confidence(self) -> float: - assert self.distribution_analyzer is not None - - context_conf = self.context_analyzer.get_confidence() - distrib_conf = self.distribution_analyzer.get_confidence() - return max(context_conf, distrib_conf) diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/euckrfreq.py b/venv/lib/python3.11/site-packages/pip/_vendor/chardet/euckrfreq.py deleted file mode 100644 index 7dc3b10..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/euckrfreq.py +++ /dev/null @@ -1,196 +0,0 @@ -######################## BEGIN LICENSE BLOCK ######################## -# The Original Code is Mozilla Communicator client code. -# -# The Initial Developer of the Original Code is -# Netscape Communications Corporation. -# Portions created by the Initial Developer are Copyright (C) 1998 -# the Initial Developer. All Rights Reserved. -# -# Contributor(s): -# Mark Pilgrim - port to Python -# -# This library is free software; you can redistribute it and/or -# modify it under the terms of the GNU Lesser General Public -# License as published by the Free Software Foundation; either -# version 2.1 of the License, or (at your option) any later version. -# -# This library is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# Lesser General Public License for more details. -# -# You should have received a copy of the GNU Lesser General Public -# License along with this library; if not, write to the Free Software -# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA -# 02110-1301 USA -######################### END LICENSE BLOCK ######################### - -# Sampling from about 20M text materials include literature and computer technology - -# 128 --> 0.79 -# 256 --> 0.92 -# 512 --> 0.986 -# 1024 --> 0.99944 -# 2048 --> 0.99999 -# -# Idea Distribution Ratio = 0.98653 / (1-0.98653) = 73.24 -# Random Distribution Ration = 512 / (2350-512) = 0.279. -# -# Typical Distribution Ratio - -EUCKR_TYPICAL_DISTRIBUTION_RATIO = 6.0 - -EUCKR_TABLE_SIZE = 2352 - -# Char to FreqOrder table , -# fmt: off -EUCKR_CHAR_TO_FREQ_ORDER = ( - 13, 130, 120,1396, 481,1719,1720, 328, 609, 212,1721, 707, 400, 299,1722, 87, -1397,1723, 104, 536,1117,1203,1724,1267, 685,1268, 508,1725,1726,1727,1728,1398, -1399,1729,1730,1731, 141, 621, 326,1057, 368,1732, 267, 488, 20,1733,1269,1734, - 945,1400,1735, 47, 904,1270,1736,1737, 773, 248,1738, 409, 313, 786, 429,1739, - 116, 987, 813,1401, 683, 75,1204, 145,1740,1741,1742,1743, 16, 847, 667, 622, - 708,1744,1745,1746, 966, 787, 304, 129,1747, 60, 820, 123, 676,1748,1749,1750, -1751, 617,1752, 626,1753,1754,1755,1756, 653,1757,1758,1759,1760,1761,1762, 856, - 344,1763,1764,1765,1766, 89, 401, 418, 806, 905, 848,1767,1768,1769, 946,1205, - 709,1770,1118,1771, 241,1772,1773,1774,1271,1775, 569,1776, 999,1777,1778,1779, -1780, 337, 751,1058, 28, 628, 254,1781, 177, 906, 270, 349, 891,1079,1782, 19, -1783, 379,1784, 315,1785, 629, 754,1402, 559,1786, 636, 203,1206,1787, 710, 567, -1788, 935, 814,1789,1790,1207, 766, 528,1791,1792,1208,1793,1794,1795,1796,1797, -1403,1798,1799, 533,1059,1404,1405,1156,1406, 936, 884,1080,1800, 351,1801,1802, -1803,1804,1805, 801,1806,1807,1808,1119,1809,1157, 714, 474,1407,1810, 298, 899, - 885,1811,1120, 802,1158,1812, 892,1813,1814,1408, 659,1815,1816,1121,1817,1818, -1819,1820,1821,1822, 319,1823, 594, 545,1824, 815, 937,1209,1825,1826, 573,1409, -1022,1827,1210,1828,1829,1830,1831,1832,1833, 556, 722, 807,1122,1060,1834, 697, -1835, 900, 557, 715,1836,1410, 540,1411, 752,1159, 294, 597,1211, 976, 803, 770, -1412,1837,1838, 39, 794,1413, 358,1839, 371, 925,1840, 453, 661, 788, 531, 723, - 544,1023,1081, 869, 91,1841, 392, 430, 790, 602,1414, 677,1082, 457,1415,1416, -1842,1843, 475, 327,1024,1417, 795, 121,1844, 733, 403,1418,1845,1846,1847, 300, - 119, 711,1212, 627,1848,1272, 207,1849,1850, 796,1213, 382,1851, 519,1852,1083, - 893,1853,1854,1855, 367, 809, 487, 671,1856, 663,1857,1858, 956, 471, 306, 857, -1859,1860,1160,1084,1861,1862,1863,1864,1865,1061,1866,1867,1868,1869,1870,1871, - 282, 96, 574,1872, 502,1085,1873,1214,1874, 907,1875,1876, 827, 977,1419,1420, -1421, 268,1877,1422,1878,1879,1880, 308,1881, 2, 537,1882,1883,1215,1884,1885, - 127, 791,1886,1273,1423,1887, 34, 336, 404, 643,1888, 571, 654, 894, 840,1889, - 0, 886,1274, 122, 575, 260, 908, 938,1890,1275, 410, 316,1891,1892, 100,1893, -1894,1123, 48,1161,1124,1025,1895, 633, 901,1276,1896,1897, 115, 816,1898, 317, -1899, 694,1900, 909, 734,1424, 572, 866,1425, 691, 85, 524,1010, 543, 394, 841, -1901,1902,1903,1026,1904,1905,1906,1907,1908,1909, 30, 451, 651, 988, 310,1910, -1911,1426, 810,1216, 93,1912,1913,1277,1217,1914, 858, 759, 45, 58, 181, 610, - 269,1915,1916, 131,1062, 551, 443,1000, 821,1427, 957, 895,1086,1917,1918, 375, -1919, 359,1920, 687,1921, 822,1922, 293,1923,1924, 40, 662, 118, 692, 29, 939, - 887, 640, 482, 174,1925, 69,1162, 728,1428, 910,1926,1278,1218,1279, 386, 870, - 217, 854,1163, 823,1927,1928,1929,1930, 834,1931, 78,1932, 859,1933,1063,1934, -1935,1936,1937, 438,1164, 208, 595,1938,1939,1940,1941,1219,1125,1942, 280, 888, -1429,1430,1220,1431,1943,1944,1945,1946,1947,1280, 150, 510,1432,1948,1949,1950, -1951,1952,1953,1954,1011,1087,1955,1433,1043,1956, 881,1957, 614, 958,1064,1065, -1221,1958, 638,1001, 860, 967, 896,1434, 989, 492, 553,1281,1165,1959,1282,1002, -1283,1222,1960,1961,1962,1963, 36, 383, 228, 753, 247, 454,1964, 876, 678,1965, -1966,1284, 126, 464, 490, 835, 136, 672, 529, 940,1088,1435, 473,1967,1968, 467, - 50, 390, 227, 587, 279, 378, 598, 792, 968, 240, 151, 160, 849, 882,1126,1285, - 639,1044, 133, 140, 288, 360, 811, 563,1027, 561, 142, 523,1969,1970,1971, 7, - 103, 296, 439, 407, 506, 634, 990,1972,1973,1974,1975, 645,1976,1977,1978,1979, -1980,1981, 236,1982,1436,1983,1984,1089, 192, 828, 618, 518,1166, 333,1127,1985, - 818,1223,1986,1987,1988,1989,1990,1991,1992,1993, 342,1128,1286, 746, 842,1994, -1995, 560, 223,1287, 98, 8, 189, 650, 978,1288,1996,1437,1997, 17, 345, 250, - 423, 277, 234, 512, 226, 97, 289, 42, 167,1998, 201,1999,2000, 843, 836, 824, - 532, 338, 783,1090, 182, 576, 436,1438,1439, 527, 500,2001, 947, 889,2002,2003, -2004,2005, 262, 600, 314, 447,2006, 547,2007, 693, 738,1129,2008, 71,1440, 745, - 619, 688,2009, 829,2010,2011, 147,2012, 33, 948,2013,2014, 74, 224,2015, 61, - 191, 918, 399, 637,2016,1028,1130, 257, 902,2017,2018,2019,2020,2021,2022,2023, -2024,2025,2026, 837,2027,2028,2029,2030, 179, 874, 591, 52, 724, 246,2031,2032, -2033,2034,1167, 969,2035,1289, 630, 605, 911,1091,1168,2036,2037,2038,1441, 912, -2039, 623,2040,2041, 253,1169,1290,2042,1442, 146, 620, 611, 577, 433,2043,1224, - 719,1170, 959, 440, 437, 534, 84, 388, 480,1131, 159, 220, 198, 679,2044,1012, - 819,1066,1443, 113,1225, 194, 318,1003,1029,2045,2046,2047,2048,1067,2049,2050, -2051,2052,2053, 59, 913, 112,2054, 632,2055, 455, 144, 739,1291,2056, 273, 681, - 499,2057, 448,2058,2059, 760,2060,2061, 970, 384, 169, 245,1132,2062,2063, 414, -1444,2064,2065, 41, 235,2066, 157, 252, 877, 568, 919, 789, 580,2067, 725,2068, -2069,1292,2070,2071,1445,2072,1446,2073,2074, 55, 588, 66,1447, 271,1092,2075, -1226,2076, 960,1013, 372,2077,2078,2079,2080,2081,1293,2082,2083,2084,2085, 850, -2086,2087,2088,2089,2090, 186,2091,1068, 180,2092,2093,2094, 109,1227, 522, 606, -2095, 867,1448,1093, 991,1171, 926, 353,1133,2096, 581,2097,2098,2099,1294,1449, -1450,2100, 596,1172,1014,1228,2101,1451,1295,1173,1229,2102,2103,1296,1134,1452, - 949,1135,2104,2105,1094,1453,1454,1455,2106,1095,2107,2108,2109,2110,2111,2112, -2113,2114,2115,2116,2117, 804,2118,2119,1230,1231, 805,1456, 405,1136,2120,2121, -2122,2123,2124, 720, 701,1297, 992,1457, 927,1004,2125,2126,2127,2128,2129,2130, - 22, 417,2131, 303,2132, 385,2133, 971, 520, 513,2134,1174, 73,1096, 231, 274, - 962,1458, 673,2135,1459,2136, 152,1137,2137,2138,2139,2140,1005,1138,1460,1139, -2141,2142,2143,2144, 11, 374, 844,2145, 154,1232, 46,1461,2146, 838, 830, 721, -1233, 106,2147, 90, 428, 462, 578, 566,1175, 352,2148,2149, 538,1234, 124,1298, -2150,1462, 761, 565,2151, 686,2152, 649,2153, 72, 173,2154, 460, 415,2155,1463, -2156,1235, 305,2157,2158,2159,2160,2161,2162, 579,2163,2164,2165,2166,2167, 747, -2168,2169,2170,2171,1464, 669,2172,2173,2174,2175,2176,1465,2177, 23, 530, 285, -2178, 335, 729,2179, 397,2180,2181,2182,1030,2183,2184, 698,2185,2186, 325,2187, -2188, 369,2189, 799,1097,1015, 348,2190,1069, 680,2191, 851,1466,2192,2193, 10, -2194, 613, 424,2195, 979, 108, 449, 589, 27, 172, 81,1031, 80, 774, 281, 350, -1032, 525, 301, 582,1176,2196, 674,1045,2197,2198,1467, 730, 762,2199,2200,2201, -2202,1468,2203, 993,2204,2205, 266,1070, 963,1140,2206,2207,2208, 664,1098, 972, -2209,2210,2211,1177,1469,1470, 871,2212,2213,2214,2215,2216,1471,2217,2218,2219, -2220,2221,2222,2223,2224,2225,2226,2227,1472,1236,2228,2229,2230,2231,2232,2233, -2234,2235,1299,2236,2237, 200,2238, 477, 373,2239,2240, 731, 825, 777,2241,2242, -2243, 521, 486, 548,2244,2245,2246,1473,1300, 53, 549, 137, 875, 76, 158,2247, -1301,1474, 469, 396,1016, 278, 712,2248, 321, 442, 503, 767, 744, 941,1237,1178, -1475,2249, 82, 178,1141,1179, 973,2250,1302,2251, 297,2252,2253, 570,2254,2255, -2256, 18, 450, 206,2257, 290, 292,1142,2258, 511, 162, 99, 346, 164, 735,2259, -1476,1477, 4, 554, 343, 798,1099,2260,1100,2261, 43, 171,1303, 139, 215,2262, -2263, 717, 775,2264,1033, 322, 216,2265, 831,2266, 149,2267,1304,2268,2269, 702, -1238, 135, 845, 347, 309,2270, 484,2271, 878, 655, 238,1006,1478,2272, 67,2273, - 295,2274,2275, 461,2276, 478, 942, 412,2277,1034,2278,2279,2280, 265,2281, 541, -2282,2283,2284,2285,2286, 70, 852,1071,2287,2288,2289,2290, 21, 56, 509, 117, - 432,2291,2292, 331, 980, 552,1101, 148, 284, 105, 393,1180,1239, 755,2293, 187, -2294,1046,1479,2295, 340,2296, 63,1047, 230,2297,2298,1305, 763,1306, 101, 800, - 808, 494,2299,2300,2301, 903,2302, 37,1072, 14, 5,2303, 79, 675,2304, 312, -2305,2306,2307,2308,2309,1480, 6,1307,2310,2311,2312, 1, 470, 35, 24, 229, -2313, 695, 210, 86, 778, 15, 784, 592, 779, 32, 77, 855, 964,2314, 259,2315, - 501, 380,2316,2317, 83, 981, 153, 689,1308,1481,1482,1483,2318,2319, 716,1484, -2320,2321,2322,2323,2324,2325,1485,2326,2327, 128, 57, 68, 261,1048, 211, 170, -1240, 31,2328, 51, 435, 742,2329,2330,2331, 635,2332, 264, 456,2333,2334,2335, - 425,2336,1486, 143, 507, 263, 943,2337, 363, 920,1487, 256,1488,1102, 243, 601, -1489,2338,2339,2340,2341,2342,2343,2344, 861,2345,2346,2347,2348,2349,2350, 395, -2351,1490,1491, 62, 535, 166, 225,2352,2353, 668, 419,1241, 138, 604, 928,2354, -1181,2355,1492,1493,2356,2357,2358,1143,2359, 696,2360, 387, 307,1309, 682, 476, -2361,2362, 332, 12, 222, 156,2363, 232,2364, 641, 276, 656, 517,1494,1495,1035, - 416, 736,1496,2365,1017, 586,2366,2367,2368,1497,2369, 242,2370,2371,2372,1498, -2373, 965, 713,2374,2375,2376,2377, 740, 982,1499, 944,1500,1007,2378,2379,1310, -1501,2380,2381,2382, 785, 329,2383,2384,1502,2385,2386,2387, 932,2388,1503,2389, -2390,2391,2392,1242,2393,2394,2395,2396,2397, 994, 950,2398,2399,2400,2401,1504, -1311,2402,2403,2404,2405,1049, 749,2406,2407, 853, 718,1144,1312,2408,1182,1505, -2409,2410, 255, 516, 479, 564, 550, 214,1506,1507,1313, 413, 239, 444, 339,1145, -1036,1508,1509,1314,1037,1510,1315,2411,1511,2412,2413,2414, 176, 703, 497, 624, - 593, 921, 302,2415, 341, 165,1103,1512,2416,1513,2417,2418,2419, 376,2420, 700, -2421,2422,2423, 258, 768,1316,2424,1183,2425, 995, 608,2426,2427,2428,2429, 221, -2430,2431,2432,2433,2434,2435,2436,2437, 195, 323, 726, 188, 897, 983,1317, 377, - 644,1050, 879,2438, 452,2439,2440,2441,2442,2443,2444, 914,2445,2446,2447,2448, - 915, 489,2449,1514,1184,2450,2451, 515, 64, 427, 495,2452, 583,2453, 483, 485, -1038, 562, 213,1515, 748, 666,2454,2455,2456,2457, 334,2458, 780, 996,1008, 705, -1243,2459,2460,2461,2462,2463, 114,2464, 493,1146, 366, 163,1516, 961,1104,2465, - 291,2466,1318,1105,2467,1517, 365,2468, 355, 951,1244,2469,1319,2470, 631,2471, -2472, 218,1320, 364, 320, 756,1518,1519,1321,1520,1322,2473,2474,2475,2476, 997, -2477,2478,2479,2480, 665,1185,2481, 916,1521,2482,2483,2484, 584, 684,2485,2486, - 797,2487,1051,1186,2488,2489,2490,1522,2491,2492, 370,2493,1039,1187, 65,2494, - 434, 205, 463,1188,2495, 125, 812, 391, 402, 826, 699, 286, 398, 155, 781, 771, - 585,2496, 590, 505,1073,2497, 599, 244, 219, 917,1018, 952, 646,1523,2498,1323, -2499,2500, 49, 984, 354, 741,2501, 625,2502,1324,2503,1019, 190, 357, 757, 491, - 95, 782, 868,2504,2505,2506,2507,2508,2509, 134,1524,1074, 422,1525, 898,2510, - 161,2511,2512,2513,2514, 769,2515,1526,2516,2517, 411,1325,2518, 472,1527,2519, -2520,2521,2522,2523,2524, 985,2525,2526,2527,2528,2529,2530, 764,2531,1245,2532, -2533, 25, 204, 311,2534, 496,2535,1052,2536,2537,2538,2539,2540,2541,2542, 199, - 704, 504, 468, 758, 657,1528, 196, 44, 839,1246, 272, 750,2543, 765, 862,2544, -2545,1326,2546, 132, 615, 933,2547, 732,2548,2549,2550,1189,1529,2551, 283,1247, -1053, 607, 929,2552,2553,2554, 930, 183, 872, 616,1040,1147,2555,1148,1020, 441, - 249,1075,2556,2557,2558, 466, 743,2559,2560,2561, 92, 514, 426, 420, 526,2562, -2563,2564,2565,2566,2567,2568, 185,2569,2570,2571,2572, 776,1530, 658,2573, 362, -2574, 361, 922,1076, 793,2575,2576,2577,2578,2579,2580,1531, 251,2581,2582,2583, -2584,1532, 54, 612, 237,1327,2585,2586, 275, 408, 647, 111,2587,1533,1106, 465, - 3, 458, 9, 38,2588, 107, 110, 890, 209, 26, 737, 498,2589,1534,2590, 431, - 202, 88,1535, 356, 287,1107, 660,1149,2591, 381,1536, 986,1150, 445,1248,1151, - 974,2592,2593, 846,2594, 446, 953, 184,1249,1250, 727,2595, 923, 193, 883,2596, -2597,2598, 102, 324, 539, 817,2599, 421,1041,2600, 832,2601, 94, 175, 197, 406, -2602, 459,2603,2604,2605,2606,2607, 330, 555,2608,2609,2610, 706,1108, 389,2611, -2612,2613,2614, 233,2615, 833, 558, 931, 954,1251,2616,2617,1537, 546,2618,2619, -1009,2620,2621,2622,1538, 690,1328,2623, 955,2624,1539,2625,2626, 772,2627,2628, -2629,2630,2631, 924, 648, 863, 603,2632,2633, 934,1540, 864, 865,2634, 642,1042, - 670,1190,2635,2636,2637,2638, 168,2639, 652, 873, 542,1054,1541,2640,2641,2642, # 512, 256 -) -# fmt: on diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/euckrprober.py b/venv/lib/python3.11/site-packages/pip/_vendor/chardet/euckrprober.py deleted file mode 100644 index 1fc5de0..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/euckrprober.py +++ /dev/null @@ -1,47 +0,0 @@ -######################## BEGIN LICENSE BLOCK ######################## -# The Original Code is mozilla.org code. -# -# The Initial Developer of the Original Code is -# Netscape Communications Corporation. -# Portions created by the Initial Developer are Copyright (C) 1998 -# the Initial Developer. All Rights Reserved. -# -# Contributor(s): -# Mark Pilgrim - port to Python -# -# This library is free software; you can redistribute it and/or -# modify it under the terms of the GNU Lesser General Public -# License as published by the Free Software Foundation; either -# version 2.1 of the License, or (at your option) any later version. -# -# This library is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# Lesser General Public License for more details. -# -# You should have received a copy of the GNU Lesser General Public -# License along with this library; if not, write to the Free Software -# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA -# 02110-1301 USA -######################### END LICENSE BLOCK ######################### - -from .chardistribution import EUCKRDistributionAnalysis -from .codingstatemachine import CodingStateMachine -from .mbcharsetprober import MultiByteCharSetProber -from .mbcssm import EUCKR_SM_MODEL - - -class EUCKRProber(MultiByteCharSetProber): - def __init__(self) -> None: - super().__init__() - self.coding_sm = CodingStateMachine(EUCKR_SM_MODEL) - self.distribution_analyzer = EUCKRDistributionAnalysis() - self.reset() - - @property - def charset_name(self) -> str: - return "EUC-KR" - - @property - def language(self) -> str: - return "Korean" diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/euctwfreq.py b/venv/lib/python3.11/site-packages/pip/_vendor/chardet/euctwfreq.py deleted file mode 100644 index 4900ccc..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/euctwfreq.py +++ /dev/null @@ -1,388 +0,0 @@ -######################## BEGIN LICENSE BLOCK ######################## -# The Original Code is Mozilla Communicator client code. -# -# The Initial Developer of the Original Code is -# Netscape Communications Corporation. -# Portions created by the Initial Developer are Copyright (C) 1998 -# the Initial Developer. All Rights Reserved. -# -# Contributor(s): -# Mark Pilgrim - port to Python -# -# This library is free software; you can redistribute it and/or -# modify it under the terms of the GNU Lesser General Public -# License as published by the Free Software Foundation; either -# version 2.1 of the License, or (at your option) any later version. -# -# This library is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# Lesser General Public License for more details. -# -# You should have received a copy of the GNU Lesser General Public -# License along with this library; if not, write to the Free Software -# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA -# 02110-1301 USA -######################### END LICENSE BLOCK ######################### - -# EUCTW frequency table -# Converted from big5 work -# by Taiwan's Mandarin Promotion Council -# - -# 128 --> 0.42261 -# 256 --> 0.57851 -# 512 --> 0.74851 -# 1024 --> 0.89384 -# 2048 --> 0.97583 -# -# Idea Distribution Ratio = 0.74851/(1-0.74851) =2.98 -# Random Distribution Ration = 512/(5401-512)=0.105 -# -# Typical Distribution Ratio about 25% of Ideal one, still much higher than RDR - -EUCTW_TYPICAL_DISTRIBUTION_RATIO = 0.75 - -# Char to FreqOrder table -EUCTW_TABLE_SIZE = 5376 - -# fmt: off -EUCTW_CHAR_TO_FREQ_ORDER = ( - 1, 1800, 1506, 255, 1431, 198, 9, 82, 6, 7310, 177, 202, 3615, 1256, 2808, 110, # 2742 - 3735, 33, 3241, 261, 76, 44, 2113, 16, 2931, 2184, 1176, 659, 3868, 26, 3404, 2643, # 2758 - 1198, 3869, 3313, 4060, 410, 2211, 302, 590, 361, 1963, 8, 204, 58, 4296, 7311, 1931, # 2774 - 63, 7312, 7313, 317, 1614, 75, 222, 159, 4061, 2412, 1480, 7314, 3500, 3068, 224, 2809, # 2790 - 3616, 3, 10, 3870, 1471, 29, 2774, 1135, 2852, 1939, 873, 130, 3242, 1123, 312, 7315, # 2806 - 4297, 2051, 507, 252, 682, 7316, 142, 1914, 124, 206, 2932, 34, 3501, 3173, 64, 604, # 2822 - 7317, 2494, 1976, 1977, 155, 1990, 645, 641, 1606, 7318, 3405, 337, 72, 406, 7319, 80, # 2838 - 630, 238, 3174, 1509, 263, 939, 1092, 2644, 756, 1440, 1094, 3406, 449, 69, 2969, 591, # 2854 - 179, 2095, 471, 115, 2034, 1843, 60, 50, 2970, 134, 806, 1868, 734, 2035, 3407, 180, # 2870 - 995, 1607, 156, 537, 2893, 688, 7320, 319, 1305, 779, 2144, 514, 2374, 298, 4298, 359, # 2886 - 2495, 90, 2707, 1338, 663, 11, 906, 1099, 2545, 20, 2436, 182, 532, 1716, 7321, 732, # 2902 - 1376, 4062, 1311, 1420, 3175, 25, 2312, 1056, 113, 399, 382, 1949, 242, 3408, 2467, 529, # 2918 - 3243, 475, 1447, 3617, 7322, 117, 21, 656, 810, 1297, 2295, 2329, 3502, 7323, 126, 4063, # 2934 - 706, 456, 150, 613, 4299, 71, 1118, 2036, 4064, 145, 3069, 85, 835, 486, 2114, 1246, # 2950 - 1426, 428, 727, 1285, 1015, 800, 106, 623, 303, 1281, 7324, 2127, 2354, 347, 3736, 221, # 2966 - 3503, 3110, 7325, 1955, 1153, 4065, 83, 296, 1199, 3070, 192, 624, 93, 7326, 822, 1897, # 2982 - 2810, 3111, 795, 2064, 991, 1554, 1542, 1592, 27, 43, 2853, 859, 139, 1456, 860, 4300, # 2998 - 437, 712, 3871, 164, 2392, 3112, 695, 211, 3017, 2096, 195, 3872, 1608, 3504, 3505, 3618, # 3014 - 3873, 234, 811, 2971, 2097, 3874, 2229, 1441, 3506, 1615, 2375, 668, 2076, 1638, 305, 228, # 3030 - 1664, 4301, 467, 415, 7327, 262, 2098, 1593, 239, 108, 300, 200, 1033, 512, 1247, 2077, # 3046 - 7328, 7329, 2173, 3176, 3619, 2673, 593, 845, 1062, 3244, 88, 1723, 2037, 3875, 1950, 212, # 3062 - 266, 152, 149, 468, 1898, 4066, 4302, 77, 187, 7330, 3018, 37, 5, 2972, 7331, 3876, # 3078 - 7332, 7333, 39, 2517, 4303, 2894, 3177, 2078, 55, 148, 74, 4304, 545, 483, 1474, 1029, # 3094 - 1665, 217, 1869, 1531, 3113, 1104, 2645, 4067, 24, 172, 3507, 900, 3877, 3508, 3509, 4305, # 3110 - 32, 1408, 2811, 1312, 329, 487, 2355, 2247, 2708, 784, 2674, 4, 3019, 3314, 1427, 1788, # 3126 - 188, 109, 499, 7334, 3620, 1717, 1789, 888, 1217, 3020, 4306, 7335, 3510, 7336, 3315, 1520, # 3142 - 3621, 3878, 196, 1034, 775, 7337, 7338, 929, 1815, 249, 439, 38, 7339, 1063, 7340, 794, # 3158 - 3879, 1435, 2296, 46, 178, 3245, 2065, 7341, 2376, 7342, 214, 1709, 4307, 804, 35, 707, # 3174 - 324, 3622, 1601, 2546, 140, 459, 4068, 7343, 7344, 1365, 839, 272, 978, 2257, 2572, 3409, # 3190 - 2128, 1363, 3623, 1423, 697, 100, 3071, 48, 70, 1231, 495, 3114, 2193, 7345, 1294, 7346, # 3206 - 2079, 462, 586, 1042, 3246, 853, 256, 988, 185, 2377, 3410, 1698, 434, 1084, 7347, 3411, # 3222 - 314, 2615, 2775, 4308, 2330, 2331, 569, 2280, 637, 1816, 2518, 757, 1162, 1878, 1616, 3412, # 3238 - 287, 1577, 2115, 768, 4309, 1671, 2854, 3511, 2519, 1321, 3737, 909, 2413, 7348, 4069, 933, # 3254 - 3738, 7349, 2052, 2356, 1222, 4310, 765, 2414, 1322, 786, 4311, 7350, 1919, 1462, 1677, 2895, # 3270 - 1699, 7351, 4312, 1424, 2437, 3115, 3624, 2590, 3316, 1774, 1940, 3413, 3880, 4070, 309, 1369, # 3286 - 1130, 2812, 364, 2230, 1653, 1299, 3881, 3512, 3882, 3883, 2646, 525, 1085, 3021, 902, 2000, # 3302 - 1475, 964, 4313, 421, 1844, 1415, 1057, 2281, 940, 1364, 3116, 376, 4314, 4315, 1381, 7, # 3318 - 2520, 983, 2378, 336, 1710, 2675, 1845, 321, 3414, 559, 1131, 3022, 2742, 1808, 1132, 1313, # 3334 - 265, 1481, 1857, 7352, 352, 1203, 2813, 3247, 167, 1089, 420, 2814, 776, 792, 1724, 3513, # 3350 - 4071, 2438, 3248, 7353, 4072, 7354, 446, 229, 333, 2743, 901, 3739, 1200, 1557, 4316, 2647, # 3366 - 1920, 395, 2744, 2676, 3740, 4073, 1835, 125, 916, 3178, 2616, 4317, 7355, 7356, 3741, 7357, # 3382 - 7358, 7359, 4318, 3117, 3625, 1133, 2547, 1757, 3415, 1510, 2313, 1409, 3514, 7360, 2145, 438, # 3398 - 2591, 2896, 2379, 3317, 1068, 958, 3023, 461, 311, 2855, 2677, 4074, 1915, 3179, 4075, 1978, # 3414 - 383, 750, 2745, 2617, 4076, 274, 539, 385, 1278, 1442, 7361, 1154, 1964, 384, 561, 210, # 3430 - 98, 1295, 2548, 3515, 7362, 1711, 2415, 1482, 3416, 3884, 2897, 1257, 129, 7363, 3742, 642, # 3446 - 523, 2776, 2777, 2648, 7364, 141, 2231, 1333, 68, 176, 441, 876, 907, 4077, 603, 2592, # 3462 - 710, 171, 3417, 404, 549, 18, 3118, 2393, 1410, 3626, 1666, 7365, 3516, 4319, 2898, 4320, # 3478 - 7366, 2973, 368, 7367, 146, 366, 99, 871, 3627, 1543, 748, 807, 1586, 1185, 22, 2258, # 3494 - 379, 3743, 3180, 7368, 3181, 505, 1941, 2618, 1991, 1382, 2314, 7369, 380, 2357, 218, 702, # 3510 - 1817, 1248, 3418, 3024, 3517, 3318, 3249, 7370, 2974, 3628, 930, 3250, 3744, 7371, 59, 7372, # 3526 - 585, 601, 4078, 497, 3419, 1112, 1314, 4321, 1801, 7373, 1223, 1472, 2174, 7374, 749, 1836, # 3542 - 690, 1899, 3745, 1772, 3885, 1476, 429, 1043, 1790, 2232, 2116, 917, 4079, 447, 1086, 1629, # 3558 - 7375, 556, 7376, 7377, 2020, 1654, 844, 1090, 105, 550, 966, 1758, 2815, 1008, 1782, 686, # 3574 - 1095, 7378, 2282, 793, 1602, 7379, 3518, 2593, 4322, 4080, 2933, 2297, 4323, 3746, 980, 2496, # 3590 - 544, 353, 527, 4324, 908, 2678, 2899, 7380, 381, 2619, 1942, 1348, 7381, 1341, 1252, 560, # 3606 - 3072, 7382, 3420, 2856, 7383, 2053, 973, 886, 2080, 143, 4325, 7384, 7385, 157, 3886, 496, # 3622 - 4081, 57, 840, 540, 2038, 4326, 4327, 3421, 2117, 1445, 970, 2259, 1748, 1965, 2081, 4082, # 3638 - 3119, 1234, 1775, 3251, 2816, 3629, 773, 1206, 2129, 1066, 2039, 1326, 3887, 1738, 1725, 4083, # 3654 - 279, 3120, 51, 1544, 2594, 423, 1578, 2130, 2066, 173, 4328, 1879, 7386, 7387, 1583, 264, # 3670 - 610, 3630, 4329, 2439, 280, 154, 7388, 7389, 7390, 1739, 338, 1282, 3073, 693, 2857, 1411, # 3686 - 1074, 3747, 2440, 7391, 4330, 7392, 7393, 1240, 952, 2394, 7394, 2900, 1538, 2679, 685, 1483, # 3702 - 4084, 2468, 1436, 953, 4085, 2054, 4331, 671, 2395, 79, 4086, 2441, 3252, 608, 567, 2680, # 3718 - 3422, 4087, 4088, 1691, 393, 1261, 1791, 2396, 7395, 4332, 7396, 7397, 7398, 7399, 1383, 1672, # 3734 - 3748, 3182, 1464, 522, 1119, 661, 1150, 216, 675, 4333, 3888, 1432, 3519, 609, 4334, 2681, # 3750 - 2397, 7400, 7401, 7402, 4089, 3025, 0, 7403, 2469, 315, 231, 2442, 301, 3319, 4335, 2380, # 3766 - 7404, 233, 4090, 3631, 1818, 4336, 4337, 7405, 96, 1776, 1315, 2082, 7406, 257, 7407, 1809, # 3782 - 3632, 2709, 1139, 1819, 4091, 2021, 1124, 2163, 2778, 1777, 2649, 7408, 3074, 363, 1655, 3183, # 3798 - 7409, 2975, 7410, 7411, 7412, 3889, 1567, 3890, 718, 103, 3184, 849, 1443, 341, 3320, 2934, # 3814 - 1484, 7413, 1712, 127, 67, 339, 4092, 2398, 679, 1412, 821, 7414, 7415, 834, 738, 351, # 3830 - 2976, 2146, 846, 235, 1497, 1880, 418, 1992, 3749, 2710, 186, 1100, 2147, 2746, 3520, 1545, # 3846 - 1355, 2935, 2858, 1377, 583, 3891, 4093, 2573, 2977, 7416, 1298, 3633, 1078, 2549, 3634, 2358, # 3862 - 78, 3750, 3751, 267, 1289, 2099, 2001, 1594, 4094, 348, 369, 1274, 2194, 2175, 1837, 4338, # 3878 - 1820, 2817, 3635, 2747, 2283, 2002, 4339, 2936, 2748, 144, 3321, 882, 4340, 3892, 2749, 3423, # 3894 - 4341, 2901, 7417, 4095, 1726, 320, 7418, 3893, 3026, 788, 2978, 7419, 2818, 1773, 1327, 2859, # 3910 - 3894, 2819, 7420, 1306, 4342, 2003, 1700, 3752, 3521, 2359, 2650, 787, 2022, 506, 824, 3636, # 3926 - 534, 323, 4343, 1044, 3322, 2023, 1900, 946, 3424, 7421, 1778, 1500, 1678, 7422, 1881, 4344, # 3942 - 165, 243, 4345, 3637, 2521, 123, 683, 4096, 764, 4346, 36, 3895, 1792, 589, 2902, 816, # 3958 - 626, 1667, 3027, 2233, 1639, 1555, 1622, 3753, 3896, 7423, 3897, 2860, 1370, 1228, 1932, 891, # 3974 - 2083, 2903, 304, 4097, 7424, 292, 2979, 2711, 3522, 691, 2100, 4098, 1115, 4347, 118, 662, # 3990 - 7425, 611, 1156, 854, 2381, 1316, 2861, 2, 386, 515, 2904, 7426, 7427, 3253, 868, 2234, # 4006 - 1486, 855, 2651, 785, 2212, 3028, 7428, 1040, 3185, 3523, 7429, 3121, 448, 7430, 1525, 7431, # 4022 - 2164, 4348, 7432, 3754, 7433, 4099, 2820, 3524, 3122, 503, 818, 3898, 3123, 1568, 814, 676, # 4038 - 1444, 306, 1749, 7434, 3755, 1416, 1030, 197, 1428, 805, 2821, 1501, 4349, 7435, 7436, 7437, # 4054 - 1993, 7438, 4350, 7439, 7440, 2195, 13, 2779, 3638, 2980, 3124, 1229, 1916, 7441, 3756, 2131, # 4070 - 7442, 4100, 4351, 2399, 3525, 7443, 2213, 1511, 1727, 1120, 7444, 7445, 646, 3757, 2443, 307, # 4086 - 7446, 7447, 1595, 3186, 7448, 7449, 7450, 3639, 1113, 1356, 3899, 1465, 2522, 2523, 7451, 519, # 4102 - 7452, 128, 2132, 92, 2284, 1979, 7453, 3900, 1512, 342, 3125, 2196, 7454, 2780, 2214, 1980, # 4118 - 3323, 7455, 290, 1656, 1317, 789, 827, 2360, 7456, 3758, 4352, 562, 581, 3901, 7457, 401, # 4134 - 4353, 2248, 94, 4354, 1399, 2781, 7458, 1463, 2024, 4355, 3187, 1943, 7459, 828, 1105, 4101, # 4150 - 1262, 1394, 7460, 4102, 605, 4356, 7461, 1783, 2862, 7462, 2822, 819, 2101, 578, 2197, 2937, # 4166 - 7463, 1502, 436, 3254, 4103, 3255, 2823, 3902, 2905, 3425, 3426, 7464, 2712, 2315, 7465, 7466, # 4182 - 2332, 2067, 23, 4357, 193, 826, 3759, 2102, 699, 1630, 4104, 3075, 390, 1793, 1064, 3526, # 4198 - 7467, 1579, 3076, 3077, 1400, 7468, 4105, 1838, 1640, 2863, 7469, 4358, 4359, 137, 4106, 598, # 4214 - 3078, 1966, 780, 104, 974, 2938, 7470, 278, 899, 253, 402, 572, 504, 493, 1339, 7471, # 4230 - 3903, 1275, 4360, 2574, 2550, 7472, 3640, 3029, 3079, 2249, 565, 1334, 2713, 863, 41, 7473, # 4246 - 7474, 4361, 7475, 1657, 2333, 19, 463, 2750, 4107, 606, 7476, 2981, 3256, 1087, 2084, 1323, # 4262 - 2652, 2982, 7477, 1631, 1623, 1750, 4108, 2682, 7478, 2864, 791, 2714, 2653, 2334, 232, 2416, # 4278 - 7479, 2983, 1498, 7480, 2654, 2620, 755, 1366, 3641, 3257, 3126, 2025, 1609, 119, 1917, 3427, # 4294 - 862, 1026, 4109, 7481, 3904, 3760, 4362, 3905, 4363, 2260, 1951, 2470, 7482, 1125, 817, 4110, # 4310 - 4111, 3906, 1513, 1766, 2040, 1487, 4112, 3030, 3258, 2824, 3761, 3127, 7483, 7484, 1507, 7485, # 4326 - 2683, 733, 40, 1632, 1106, 2865, 345, 4113, 841, 2524, 230, 4364, 2984, 1846, 3259, 3428, # 4342 - 7486, 1263, 986, 3429, 7487, 735, 879, 254, 1137, 857, 622, 1300, 1180, 1388, 1562, 3907, # 4358 - 3908, 2939, 967, 2751, 2655, 1349, 592, 2133, 1692, 3324, 2985, 1994, 4114, 1679, 3909, 1901, # 4374 - 2185, 7488, 739, 3642, 2715, 1296, 1290, 7489, 4115, 2198, 2199, 1921, 1563, 2595, 2551, 1870, # 4390 - 2752, 2986, 7490, 435, 7491, 343, 1108, 596, 17, 1751, 4365, 2235, 3430, 3643, 7492, 4366, # 4406 - 294, 3527, 2940, 1693, 477, 979, 281, 2041, 3528, 643, 2042, 3644, 2621, 2782, 2261, 1031, # 4422 - 2335, 2134, 2298, 3529, 4367, 367, 1249, 2552, 7493, 3530, 7494, 4368, 1283, 3325, 2004, 240, # 4438 - 1762, 3326, 4369, 4370, 836, 1069, 3128, 474, 7495, 2148, 2525, 268, 3531, 7496, 3188, 1521, # 4454 - 1284, 7497, 1658, 1546, 4116, 7498, 3532, 3533, 7499, 4117, 3327, 2684, 1685, 4118, 961, 1673, # 4470 - 2622, 190, 2005, 2200, 3762, 4371, 4372, 7500, 570, 2497, 3645, 1490, 7501, 4373, 2623, 3260, # 4486 - 1956, 4374, 584, 1514, 396, 1045, 1944, 7502, 4375, 1967, 2444, 7503, 7504, 4376, 3910, 619, # 4502 - 7505, 3129, 3261, 215, 2006, 2783, 2553, 3189, 4377, 3190, 4378, 763, 4119, 3763, 4379, 7506, # 4518 - 7507, 1957, 1767, 2941, 3328, 3646, 1174, 452, 1477, 4380, 3329, 3130, 7508, 2825, 1253, 2382, # 4534 - 2186, 1091, 2285, 4120, 492, 7509, 638, 1169, 1824, 2135, 1752, 3911, 648, 926, 1021, 1324, # 4550 - 4381, 520, 4382, 997, 847, 1007, 892, 4383, 3764, 2262, 1871, 3647, 7510, 2400, 1784, 4384, # 4566 - 1952, 2942, 3080, 3191, 1728, 4121, 2043, 3648, 4385, 2007, 1701, 3131, 1551, 30, 2263, 4122, # 4582 - 7511, 2026, 4386, 3534, 7512, 501, 7513, 4123, 594, 3431, 2165, 1821, 3535, 3432, 3536, 3192, # 4598 - 829, 2826, 4124, 7514, 1680, 3132, 1225, 4125, 7515, 3262, 4387, 4126, 3133, 2336, 7516, 4388, # 4614 - 4127, 7517, 3912, 3913, 7518, 1847, 2383, 2596, 3330, 7519, 4389, 374, 3914, 652, 4128, 4129, # 4630 - 375, 1140, 798, 7520, 7521, 7522, 2361, 4390, 2264, 546, 1659, 138, 3031, 2445, 4391, 7523, # 4646 - 2250, 612, 1848, 910, 796, 3765, 1740, 1371, 825, 3766, 3767, 7524, 2906, 2554, 7525, 692, # 4662 - 444, 3032, 2624, 801, 4392, 4130, 7526, 1491, 244, 1053, 3033, 4131, 4132, 340, 7527, 3915, # 4678 - 1041, 2987, 293, 1168, 87, 1357, 7528, 1539, 959, 7529, 2236, 721, 694, 4133, 3768, 219, # 4694 - 1478, 644, 1417, 3331, 2656, 1413, 1401, 1335, 1389, 3916, 7530, 7531, 2988, 2362, 3134, 1825, # 4710 - 730, 1515, 184, 2827, 66, 4393, 7532, 1660, 2943, 246, 3332, 378, 1457, 226, 3433, 975, # 4726 - 3917, 2944, 1264, 3537, 674, 696, 7533, 163, 7534, 1141, 2417, 2166, 713, 3538, 3333, 4394, # 4742 - 3918, 7535, 7536, 1186, 15, 7537, 1079, 1070, 7538, 1522, 3193, 3539, 276, 1050, 2716, 758, # 4758 - 1126, 653, 2945, 3263, 7539, 2337, 889, 3540, 3919, 3081, 2989, 903, 1250, 4395, 3920, 3434, # 4774 - 3541, 1342, 1681, 1718, 766, 3264, 286, 89, 2946, 3649, 7540, 1713, 7541, 2597, 3334, 2990, # 4790 - 7542, 2947, 2215, 3194, 2866, 7543, 4396, 2498, 2526, 181, 387, 1075, 3921, 731, 2187, 3335, # 4806 - 7544, 3265, 310, 313, 3435, 2299, 770, 4134, 54, 3034, 189, 4397, 3082, 3769, 3922, 7545, # 4822 - 1230, 1617, 1849, 355, 3542, 4135, 4398, 3336, 111, 4136, 3650, 1350, 3135, 3436, 3035, 4137, # 4838 - 2149, 3266, 3543, 7546, 2784, 3923, 3924, 2991, 722, 2008, 7547, 1071, 247, 1207, 2338, 2471, # 4854 - 1378, 4399, 2009, 864, 1437, 1214, 4400, 373, 3770, 1142, 2216, 667, 4401, 442, 2753, 2555, # 4870 - 3771, 3925, 1968, 4138, 3267, 1839, 837, 170, 1107, 934, 1336, 1882, 7548, 7549, 2118, 4139, # 4886 - 2828, 743, 1569, 7550, 4402, 4140, 582, 2384, 1418, 3437, 7551, 1802, 7552, 357, 1395, 1729, # 4902 - 3651, 3268, 2418, 1564, 2237, 7553, 3083, 3772, 1633, 4403, 1114, 2085, 4141, 1532, 7554, 482, # 4918 - 2446, 4404, 7555, 7556, 1492, 833, 1466, 7557, 2717, 3544, 1641, 2829, 7558, 1526, 1272, 3652, # 4934 - 4142, 1686, 1794, 416, 2556, 1902, 1953, 1803, 7559, 3773, 2785, 3774, 1159, 2316, 7560, 2867, # 4950 - 4405, 1610, 1584, 3036, 2419, 2754, 443, 3269, 1163, 3136, 7561, 7562, 3926, 7563, 4143, 2499, # 4966 - 3037, 4406, 3927, 3137, 2103, 1647, 3545, 2010, 1872, 4144, 7564, 4145, 431, 3438, 7565, 250, # 4982 - 97, 81, 4146, 7566, 1648, 1850, 1558, 160, 848, 7567, 866, 740, 1694, 7568, 2201, 2830, # 4998 - 3195, 4147, 4407, 3653, 1687, 950, 2472, 426, 469, 3196, 3654, 3655, 3928, 7569, 7570, 1188, # 5014 - 424, 1995, 861, 3546, 4148, 3775, 2202, 2685, 168, 1235, 3547, 4149, 7571, 2086, 1674, 4408, # 5030 - 3337, 3270, 220, 2557, 1009, 7572, 3776, 670, 2992, 332, 1208, 717, 7573, 7574, 3548, 2447, # 5046 - 3929, 3338, 7575, 513, 7576, 1209, 2868, 3339, 3138, 4409, 1080, 7577, 7578, 7579, 7580, 2527, # 5062 - 3656, 3549, 815, 1587, 3930, 3931, 7581, 3550, 3439, 3777, 1254, 4410, 1328, 3038, 1390, 3932, # 5078 - 1741, 3933, 3778, 3934, 7582, 236, 3779, 2448, 3271, 7583, 7584, 3657, 3780, 1273, 3781, 4411, # 5094 - 7585, 308, 7586, 4412, 245, 4413, 1851, 2473, 1307, 2575, 430, 715, 2136, 2449, 7587, 270, # 5110 - 199, 2869, 3935, 7588, 3551, 2718, 1753, 761, 1754, 725, 1661, 1840, 4414, 3440, 3658, 7589, # 5126 - 7590, 587, 14, 3272, 227, 2598, 326, 480, 2265, 943, 2755, 3552, 291, 650, 1883, 7591, # 5142 - 1702, 1226, 102, 1547, 62, 3441, 904, 4415, 3442, 1164, 4150, 7592, 7593, 1224, 1548, 2756, # 5158 - 391, 498, 1493, 7594, 1386, 1419, 7595, 2055, 1177, 4416, 813, 880, 1081, 2363, 566, 1145, # 5174 - 4417, 2286, 1001, 1035, 2558, 2599, 2238, 394, 1286, 7596, 7597, 2068, 7598, 86, 1494, 1730, # 5190 - 3936, 491, 1588, 745, 897, 2948, 843, 3340, 3937, 2757, 2870, 3273, 1768, 998, 2217, 2069, # 5206 - 397, 1826, 1195, 1969, 3659, 2993, 3341, 284, 7599, 3782, 2500, 2137, 2119, 1903, 7600, 3938, # 5222 - 2150, 3939, 4151, 1036, 3443, 1904, 114, 2559, 4152, 209, 1527, 7601, 7602, 2949, 2831, 2625, # 5238 - 2385, 2719, 3139, 812, 2560, 7603, 3274, 7604, 1559, 737, 1884, 3660, 1210, 885, 28, 2686, # 5254 - 3553, 3783, 7605, 4153, 1004, 1779, 4418, 7606, 346, 1981, 2218, 2687, 4419, 3784, 1742, 797, # 5270 - 1642, 3940, 1933, 1072, 1384, 2151, 896, 3941, 3275, 3661, 3197, 2871, 3554, 7607, 2561, 1958, # 5286 - 4420, 2450, 1785, 7608, 7609, 7610, 3942, 4154, 1005, 1308, 3662, 4155, 2720, 4421, 4422, 1528, # 5302 - 2600, 161, 1178, 4156, 1982, 987, 4423, 1101, 4157, 631, 3943, 1157, 3198, 2420, 1343, 1241, # 5318 - 1016, 2239, 2562, 372, 877, 2339, 2501, 1160, 555, 1934, 911, 3944, 7611, 466, 1170, 169, # 5334 - 1051, 2907, 2688, 3663, 2474, 2994, 1182, 2011, 2563, 1251, 2626, 7612, 992, 2340, 3444, 1540, # 5350 - 2721, 1201, 2070, 2401, 1996, 2475, 7613, 4424, 528, 1922, 2188, 1503, 1873, 1570, 2364, 3342, # 5366 - 3276, 7614, 557, 1073, 7615, 1827, 3445, 2087, 2266, 3140, 3039, 3084, 767, 3085, 2786, 4425, # 5382 - 1006, 4158, 4426, 2341, 1267, 2176, 3664, 3199, 778, 3945, 3200, 2722, 1597, 2657, 7616, 4427, # 5398 - 7617, 3446, 7618, 7619, 7620, 3277, 2689, 1433, 3278, 131, 95, 1504, 3946, 723, 4159, 3141, # 5414 - 1841, 3555, 2758, 2189, 3947, 2027, 2104, 3665, 7621, 2995, 3948, 1218, 7622, 3343, 3201, 3949, # 5430 - 4160, 2576, 248, 1634, 3785, 912, 7623, 2832, 3666, 3040, 3786, 654, 53, 7624, 2996, 7625, # 5446 - 1688, 4428, 777, 3447, 1032, 3950, 1425, 7626, 191, 820, 2120, 2833, 971, 4429, 931, 3202, # 5462 - 135, 664, 783, 3787, 1997, 772, 2908, 1935, 3951, 3788, 4430, 2909, 3203, 282, 2723, 640, # 5478 - 1372, 3448, 1127, 922, 325, 3344, 7627, 7628, 711, 2044, 7629, 7630, 3952, 2219, 2787, 1936, # 5494 - 3953, 3345, 2220, 2251, 3789, 2300, 7631, 4431, 3790, 1258, 3279, 3954, 3204, 2138, 2950, 3955, # 5510 - 3956, 7632, 2221, 258, 3205, 4432, 101, 1227, 7633, 3280, 1755, 7634, 1391, 3281, 7635, 2910, # 5526 - 2056, 893, 7636, 7637, 7638, 1402, 4161, 2342, 7639, 7640, 3206, 3556, 7641, 7642, 878, 1325, # 5542 - 1780, 2788, 4433, 259, 1385, 2577, 744, 1183, 2267, 4434, 7643, 3957, 2502, 7644, 684, 1024, # 5558 - 4162, 7645, 472, 3557, 3449, 1165, 3282, 3958, 3959, 322, 2152, 881, 455, 1695, 1152, 1340, # 5574 - 660, 554, 2153, 4435, 1058, 4436, 4163, 830, 1065, 3346, 3960, 4437, 1923, 7646, 1703, 1918, # 5590 - 7647, 932, 2268, 122, 7648, 4438, 947, 677, 7649, 3791, 2627, 297, 1905, 1924, 2269, 4439, # 5606 - 2317, 3283, 7650, 7651, 4164, 7652, 4165, 84, 4166, 112, 989, 7653, 547, 1059, 3961, 701, # 5622 - 3558, 1019, 7654, 4167, 7655, 3450, 942, 639, 457, 2301, 2451, 993, 2951, 407, 851, 494, # 5638 - 4440, 3347, 927, 7656, 1237, 7657, 2421, 3348, 573, 4168, 680, 921, 2911, 1279, 1874, 285, # 5654 - 790, 1448, 1983, 719, 2167, 7658, 7659, 4441, 3962, 3963, 1649, 7660, 1541, 563, 7661, 1077, # 5670 - 7662, 3349, 3041, 3451, 511, 2997, 3964, 3965, 3667, 3966, 1268, 2564, 3350, 3207, 4442, 4443, # 5686 - 7663, 535, 1048, 1276, 1189, 2912, 2028, 3142, 1438, 1373, 2834, 2952, 1134, 2012, 7664, 4169, # 5702 - 1238, 2578, 3086, 1259, 7665, 700, 7666, 2953, 3143, 3668, 4170, 7667, 4171, 1146, 1875, 1906, # 5718 - 4444, 2601, 3967, 781, 2422, 132, 1589, 203, 147, 273, 2789, 2402, 898, 1786, 2154, 3968, # 5734 - 3969, 7668, 3792, 2790, 7669, 7670, 4445, 4446, 7671, 3208, 7672, 1635, 3793, 965, 7673, 1804, # 5750 - 2690, 1516, 3559, 1121, 1082, 1329, 3284, 3970, 1449, 3794, 65, 1128, 2835, 2913, 2759, 1590, # 5766 - 3795, 7674, 7675, 12, 2658, 45, 976, 2579, 3144, 4447, 517, 2528, 1013, 1037, 3209, 7676, # 5782 - 3796, 2836, 7677, 3797, 7678, 3452, 7679, 2602, 614, 1998, 2318, 3798, 3087, 2724, 2628, 7680, # 5798 - 2580, 4172, 599, 1269, 7681, 1810, 3669, 7682, 2691, 3088, 759, 1060, 489, 1805, 3351, 3285, # 5814 - 1358, 7683, 7684, 2386, 1387, 1215, 2629, 2252, 490, 7685, 7686, 4173, 1759, 2387, 2343, 7687, # 5830 - 4448, 3799, 1907, 3971, 2630, 1806, 3210, 4449, 3453, 3286, 2760, 2344, 874, 7688, 7689, 3454, # 5846 - 3670, 1858, 91, 2914, 3671, 3042, 3800, 4450, 7690, 3145, 3972, 2659, 7691, 3455, 1202, 1403, # 5862 - 3801, 2954, 2529, 1517, 2503, 4451, 3456, 2504, 7692, 4452, 7693, 2692, 1885, 1495, 1731, 3973, # 5878 - 2365, 4453, 7694, 2029, 7695, 7696, 3974, 2693, 1216, 237, 2581, 4174, 2319, 3975, 3802, 4454, # 5894 - 4455, 2694, 3560, 3457, 445, 4456, 7697, 7698, 7699, 7700, 2761, 61, 3976, 3672, 1822, 3977, # 5910 - 7701, 687, 2045, 935, 925, 405, 2660, 703, 1096, 1859, 2725, 4457, 3978, 1876, 1367, 2695, # 5926 - 3352, 918, 2105, 1781, 2476, 334, 3287, 1611, 1093, 4458, 564, 3146, 3458, 3673, 3353, 945, # 5942 - 2631, 2057, 4459, 7702, 1925, 872, 4175, 7703, 3459, 2696, 3089, 349, 4176, 3674, 3979, 4460, # 5958 - 3803, 4177, 3675, 2155, 3980, 4461, 4462, 4178, 4463, 2403, 2046, 782, 3981, 400, 251, 4179, # 5974 - 1624, 7704, 7705, 277, 3676, 299, 1265, 476, 1191, 3804, 2121, 4180, 4181, 1109, 205, 7706, # 5990 - 2582, 1000, 2156, 3561, 1860, 7707, 7708, 7709, 4464, 7710, 4465, 2565, 107, 2477, 2157, 3982, # 6006 - 3460, 3147, 7711, 1533, 541, 1301, 158, 753, 4182, 2872, 3562, 7712, 1696, 370, 1088, 4183, # 6022 - 4466, 3563, 579, 327, 440, 162, 2240, 269, 1937, 1374, 3461, 968, 3043, 56, 1396, 3090, # 6038 - 2106, 3288, 3354, 7713, 1926, 2158, 4467, 2998, 7714, 3564, 7715, 7716, 3677, 4468, 2478, 7717, # 6054 - 2791, 7718, 1650, 4469, 7719, 2603, 7720, 7721, 3983, 2661, 3355, 1149, 3356, 3984, 3805, 3985, # 6070 - 7722, 1076, 49, 7723, 951, 3211, 3289, 3290, 450, 2837, 920, 7724, 1811, 2792, 2366, 4184, # 6086 - 1908, 1138, 2367, 3806, 3462, 7725, 3212, 4470, 1909, 1147, 1518, 2423, 4471, 3807, 7726, 4472, # 6102 - 2388, 2604, 260, 1795, 3213, 7727, 7728, 3808, 3291, 708, 7729, 3565, 1704, 7730, 3566, 1351, # 6118 - 1618, 3357, 2999, 1886, 944, 4185, 3358, 4186, 3044, 3359, 4187, 7731, 3678, 422, 413, 1714, # 6134 - 3292, 500, 2058, 2345, 4188, 2479, 7732, 1344, 1910, 954, 7733, 1668, 7734, 7735, 3986, 2404, # 6150 - 4189, 3567, 3809, 4190, 7736, 2302, 1318, 2505, 3091, 133, 3092, 2873, 4473, 629, 31, 2838, # 6166 - 2697, 3810, 4474, 850, 949, 4475, 3987, 2955, 1732, 2088, 4191, 1496, 1852, 7737, 3988, 620, # 6182 - 3214, 981, 1242, 3679, 3360, 1619, 3680, 1643, 3293, 2139, 2452, 1970, 1719, 3463, 2168, 7738, # 6198 - 3215, 7739, 7740, 3361, 1828, 7741, 1277, 4476, 1565, 2047, 7742, 1636, 3568, 3093, 7743, 869, # 6214 - 2839, 655, 3811, 3812, 3094, 3989, 3000, 3813, 1310, 3569, 4477, 7744, 7745, 7746, 1733, 558, # 6230 - 4478, 3681, 335, 1549, 3045, 1756, 4192, 3682, 1945, 3464, 1829, 1291, 1192, 470, 2726, 2107, # 6246 - 2793, 913, 1054, 3990, 7747, 1027, 7748, 3046, 3991, 4479, 982, 2662, 3362, 3148, 3465, 3216, # 6262 - 3217, 1946, 2794, 7749, 571, 4480, 7750, 1830, 7751, 3570, 2583, 1523, 2424, 7752, 2089, 984, # 6278 - 4481, 3683, 1959, 7753, 3684, 852, 923, 2795, 3466, 3685, 969, 1519, 999, 2048, 2320, 1705, # 6294 - 7754, 3095, 615, 1662, 151, 597, 3992, 2405, 2321, 1049, 275, 4482, 3686, 4193, 568, 3687, # 6310 - 3571, 2480, 4194, 3688, 7755, 2425, 2270, 409, 3218, 7756, 1566, 2874, 3467, 1002, 769, 2840, # 6326 - 194, 2090, 3149, 3689, 2222, 3294, 4195, 628, 1505, 7757, 7758, 1763, 2177, 3001, 3993, 521, # 6342 - 1161, 2584, 1787, 2203, 2406, 4483, 3994, 1625, 4196, 4197, 412, 42, 3096, 464, 7759, 2632, # 6358 - 4484, 3363, 1760, 1571, 2875, 3468, 2530, 1219, 2204, 3814, 2633, 2140, 2368, 4485, 4486, 3295, # 6374 - 1651, 3364, 3572, 7760, 7761, 3573, 2481, 3469, 7762, 3690, 7763, 7764, 2271, 2091, 460, 7765, # 6390 - 4487, 7766, 3002, 962, 588, 3574, 289, 3219, 2634, 1116, 52, 7767, 3047, 1796, 7768, 7769, # 6406 - 7770, 1467, 7771, 1598, 1143, 3691, 4198, 1984, 1734, 1067, 4488, 1280, 3365, 465, 4489, 1572, # 6422 - 510, 7772, 1927, 2241, 1812, 1644, 3575, 7773, 4490, 3692, 7774, 7775, 2663, 1573, 1534, 7776, # 6438 - 7777, 4199, 536, 1807, 1761, 3470, 3815, 3150, 2635, 7778, 7779, 7780, 4491, 3471, 2915, 1911, # 6454 - 2796, 7781, 3296, 1122, 377, 3220, 7782, 360, 7783, 7784, 4200, 1529, 551, 7785, 2059, 3693, # 6470 - 1769, 2426, 7786, 2916, 4201, 3297, 3097, 2322, 2108, 2030, 4492, 1404, 136, 1468, 1479, 672, # 6486 - 1171, 3221, 2303, 271, 3151, 7787, 2762, 7788, 2049, 678, 2727, 865, 1947, 4493, 7789, 2013, # 6502 - 3995, 2956, 7790, 2728, 2223, 1397, 3048, 3694, 4494, 4495, 1735, 2917, 3366, 3576, 7791, 3816, # 6518 - 509, 2841, 2453, 2876, 3817, 7792, 7793, 3152, 3153, 4496, 4202, 2531, 4497, 2304, 1166, 1010, # 6534 - 552, 681, 1887, 7794, 7795, 2957, 2958, 3996, 1287, 1596, 1861, 3154, 358, 453, 736, 175, # 6550 - 478, 1117, 905, 1167, 1097, 7796, 1853, 1530, 7797, 1706, 7798, 2178, 3472, 2287, 3695, 3473, # 6566 - 3577, 4203, 2092, 4204, 7799, 3367, 1193, 2482, 4205, 1458, 2190, 2205, 1862, 1888, 1421, 3298, # 6582 - 2918, 3049, 2179, 3474, 595, 2122, 7800, 3997, 7801, 7802, 4206, 1707, 2636, 223, 3696, 1359, # 6598 - 751, 3098, 183, 3475, 7803, 2797, 3003, 419, 2369, 633, 704, 3818, 2389, 241, 7804, 7805, # 6614 - 7806, 838, 3004, 3697, 2272, 2763, 2454, 3819, 1938, 2050, 3998, 1309, 3099, 2242, 1181, 7807, # 6630 - 1136, 2206, 3820, 2370, 1446, 4207, 2305, 4498, 7808, 7809, 4208, 1055, 2605, 484, 3698, 7810, # 6646 - 3999, 625, 4209, 2273, 3368, 1499, 4210, 4000, 7811, 4001, 4211, 3222, 2274, 2275, 3476, 7812, # 6662 - 7813, 2764, 808, 2606, 3699, 3369, 4002, 4212, 3100, 2532, 526, 3370, 3821, 4213, 955, 7814, # 6678 - 1620, 4214, 2637, 2427, 7815, 1429, 3700, 1669, 1831, 994, 928, 7816, 3578, 1260, 7817, 7818, # 6694 - 7819, 1948, 2288, 741, 2919, 1626, 4215, 2729, 2455, 867, 1184, 362, 3371, 1392, 7820, 7821, # 6710 - 4003, 4216, 1770, 1736, 3223, 2920, 4499, 4500, 1928, 2698, 1459, 1158, 7822, 3050, 3372, 2877, # 6726 - 1292, 1929, 2506, 2842, 3701, 1985, 1187, 2071, 2014, 2607, 4217, 7823, 2566, 2507, 2169, 3702, # 6742 - 2483, 3299, 7824, 3703, 4501, 7825, 7826, 666, 1003, 3005, 1022, 3579, 4218, 7827, 4502, 1813, # 6758 - 2253, 574, 3822, 1603, 295, 1535, 705, 3823, 4219, 283, 858, 417, 7828, 7829, 3224, 4503, # 6774 - 4504, 3051, 1220, 1889, 1046, 2276, 2456, 4004, 1393, 1599, 689, 2567, 388, 4220, 7830, 2484, # 6790 - 802, 7831, 2798, 3824, 2060, 1405, 2254, 7832, 4505, 3825, 2109, 1052, 1345, 3225, 1585, 7833, # 6806 - 809, 7834, 7835, 7836, 575, 2730, 3477, 956, 1552, 1469, 1144, 2323, 7837, 2324, 1560, 2457, # 6822 - 3580, 3226, 4005, 616, 2207, 3155, 2180, 2289, 7838, 1832, 7839, 3478, 4506, 7840, 1319, 3704, # 6838 - 3705, 1211, 3581, 1023, 3227, 1293, 2799, 7841, 7842, 7843, 3826, 607, 2306, 3827, 762, 2878, # 6854 - 1439, 4221, 1360, 7844, 1485, 3052, 7845, 4507, 1038, 4222, 1450, 2061, 2638, 4223, 1379, 4508, # 6870 - 2585, 7846, 7847, 4224, 1352, 1414, 2325, 2921, 1172, 7848, 7849, 3828, 3829, 7850, 1797, 1451, # 6886 - 7851, 7852, 7853, 7854, 2922, 4006, 4007, 2485, 2346, 411, 4008, 4009, 3582, 3300, 3101, 4509, # 6902 - 1561, 2664, 1452, 4010, 1375, 7855, 7856, 47, 2959, 316, 7857, 1406, 1591, 2923, 3156, 7858, # 6918 - 1025, 2141, 3102, 3157, 354, 2731, 884, 2224, 4225, 2407, 508, 3706, 726, 3583, 996, 2428, # 6934 - 3584, 729, 7859, 392, 2191, 1453, 4011, 4510, 3707, 7860, 7861, 2458, 3585, 2608, 1675, 2800, # 6950 - 919, 2347, 2960, 2348, 1270, 4511, 4012, 73, 7862, 7863, 647, 7864, 3228, 2843, 2255, 1550, # 6966 - 1346, 3006, 7865, 1332, 883, 3479, 7866, 7867, 7868, 7869, 3301, 2765, 7870, 1212, 831, 1347, # 6982 - 4226, 4512, 2326, 3830, 1863, 3053, 720, 3831, 4513, 4514, 3832, 7871, 4227, 7872, 7873, 4515, # 6998 - 7874, 7875, 1798, 4516, 3708, 2609, 4517, 3586, 1645, 2371, 7876, 7877, 2924, 669, 2208, 2665, # 7014 - 2429, 7878, 2879, 7879, 7880, 1028, 3229, 7881, 4228, 2408, 7882, 2256, 1353, 7883, 7884, 4518, # 7030 - 3158, 518, 7885, 4013, 7886, 4229, 1960, 7887, 2142, 4230, 7888, 7889, 3007, 2349, 2350, 3833, # 7046 - 516, 1833, 1454, 4014, 2699, 4231, 4519, 2225, 2610, 1971, 1129, 3587, 7890, 2766, 7891, 2961, # 7062 - 1422, 577, 1470, 3008, 1524, 3373, 7892, 7893, 432, 4232, 3054, 3480, 7894, 2586, 1455, 2508, # 7078 - 2226, 1972, 1175, 7895, 1020, 2732, 4015, 3481, 4520, 7896, 2733, 7897, 1743, 1361, 3055, 3482, # 7094 - 2639, 4016, 4233, 4521, 2290, 895, 924, 4234, 2170, 331, 2243, 3056, 166, 1627, 3057, 1098, # 7110 - 7898, 1232, 2880, 2227, 3374, 4522, 657, 403, 1196, 2372, 542, 3709, 3375, 1600, 4235, 3483, # 7126 - 7899, 4523, 2767, 3230, 576, 530, 1362, 7900, 4524, 2533, 2666, 3710, 4017, 7901, 842, 3834, # 7142 - 7902, 2801, 2031, 1014, 4018, 213, 2700, 3376, 665, 621, 4236, 7903, 3711, 2925, 2430, 7904, # 7158 - 2431, 3302, 3588, 3377, 7905, 4237, 2534, 4238, 4525, 3589, 1682, 4239, 3484, 1380, 7906, 724, # 7174 - 2277, 600, 1670, 7907, 1337, 1233, 4526, 3103, 2244, 7908, 1621, 4527, 7909, 651, 4240, 7910, # 7190 - 1612, 4241, 2611, 7911, 2844, 7912, 2734, 2307, 3058, 7913, 716, 2459, 3059, 174, 1255, 2701, # 7206 - 4019, 3590, 548, 1320, 1398, 728, 4020, 1574, 7914, 1890, 1197, 3060, 4021, 7915, 3061, 3062, # 7222 - 3712, 3591, 3713, 747, 7916, 635, 4242, 4528, 7917, 7918, 7919, 4243, 7920, 7921, 4529, 7922, # 7238 - 3378, 4530, 2432, 451, 7923, 3714, 2535, 2072, 4244, 2735, 4245, 4022, 7924, 1764, 4531, 7925, # 7254 - 4246, 350, 7926, 2278, 2390, 2486, 7927, 4247, 4023, 2245, 1434, 4024, 488, 4532, 458, 4248, # 7270 - 4025, 3715, 771, 1330, 2391, 3835, 2568, 3159, 2159, 2409, 1553, 2667, 3160, 4249, 7928, 2487, # 7286 - 2881, 2612, 1720, 2702, 4250, 3379, 4533, 7929, 2536, 4251, 7930, 3231, 4252, 2768, 7931, 2015, # 7302 - 2736, 7932, 1155, 1017, 3716, 3836, 7933, 3303, 2308, 201, 1864, 4253, 1430, 7934, 4026, 7935, # 7318 - 7936, 7937, 7938, 7939, 4254, 1604, 7940, 414, 1865, 371, 2587, 4534, 4535, 3485, 2016, 3104, # 7334 - 4536, 1708, 960, 4255, 887, 389, 2171, 1536, 1663, 1721, 7941, 2228, 4027, 2351, 2926, 1580, # 7350 - 7942, 7943, 7944, 1744, 7945, 2537, 4537, 4538, 7946, 4539, 7947, 2073, 7948, 7949, 3592, 3380, # 7366 - 2882, 4256, 7950, 4257, 2640, 3381, 2802, 673, 2703, 2460, 709, 3486, 4028, 3593, 4258, 7951, # 7382 - 1148, 502, 634, 7952, 7953, 1204, 4540, 3594, 1575, 4541, 2613, 3717, 7954, 3718, 3105, 948, # 7398 - 3232, 121, 1745, 3837, 1110, 7955, 4259, 3063, 2509, 3009, 4029, 3719, 1151, 1771, 3838, 1488, # 7414 - 4030, 1986, 7956, 2433, 3487, 7957, 7958, 2093, 7959, 4260, 3839, 1213, 1407, 2803, 531, 2737, # 7430 - 2538, 3233, 1011, 1537, 7960, 2769, 4261, 3106, 1061, 7961, 3720, 3721, 1866, 2883, 7962, 2017, # 7446 - 120, 4262, 4263, 2062, 3595, 3234, 2309, 3840, 2668, 3382, 1954, 4542, 7963, 7964, 3488, 1047, # 7462 - 2704, 1266, 7965, 1368, 4543, 2845, 649, 3383, 3841, 2539, 2738, 1102, 2846, 2669, 7966, 7967, # 7478 - 1999, 7968, 1111, 3596, 2962, 7969, 2488, 3842, 3597, 2804, 1854, 3384, 3722, 7970, 7971, 3385, # 7494 - 2410, 2884, 3304, 3235, 3598, 7972, 2569, 7973, 3599, 2805, 4031, 1460, 856, 7974, 3600, 7975, # 7510 - 2885, 2963, 7976, 2886, 3843, 7977, 4264, 632, 2510, 875, 3844, 1697, 3845, 2291, 7978, 7979, # 7526 - 4544, 3010, 1239, 580, 4545, 4265, 7980, 914, 936, 2074, 1190, 4032, 1039, 2123, 7981, 7982, # 7542 - 7983, 3386, 1473, 7984, 1354, 4266, 3846, 7985, 2172, 3064, 4033, 915, 3305, 4267, 4268, 3306, # 7558 - 1605, 1834, 7986, 2739, 398, 3601, 4269, 3847, 4034, 328, 1912, 2847, 4035, 3848, 1331, 4270, # 7574 - 3011, 937, 4271, 7987, 3602, 4036, 4037, 3387, 2160, 4546, 3388, 524, 742, 538, 3065, 1012, # 7590 - 7988, 7989, 3849, 2461, 7990, 658, 1103, 225, 3850, 7991, 7992, 4547, 7993, 4548, 7994, 3236, # 7606 - 1243, 7995, 4038, 963, 2246, 4549, 7996, 2705, 3603, 3161, 7997, 7998, 2588, 2327, 7999, 4550, # 7622 - 8000, 8001, 8002, 3489, 3307, 957, 3389, 2540, 2032, 1930, 2927, 2462, 870, 2018, 3604, 1746, # 7638 - 2770, 2771, 2434, 2463, 8003, 3851, 8004, 3723, 3107, 3724, 3490, 3390, 3725, 8005, 1179, 3066, # 7654 - 8006, 3162, 2373, 4272, 3726, 2541, 3163, 3108, 2740, 4039, 8007, 3391, 1556, 2542, 2292, 977, # 7670 - 2887, 2033, 4040, 1205, 3392, 8008, 1765, 3393, 3164, 2124, 1271, 1689, 714, 4551, 3491, 8009, # 7686 - 2328, 3852, 533, 4273, 3605, 2181, 617, 8010, 2464, 3308, 3492, 2310, 8011, 8012, 3165, 8013, # 7702 - 8014, 3853, 1987, 618, 427, 2641, 3493, 3394, 8015, 8016, 1244, 1690, 8017, 2806, 4274, 4552, # 7718 - 8018, 3494, 8019, 8020, 2279, 1576, 473, 3606, 4275, 3395, 972, 8021, 3607, 8022, 3067, 8023, # 7734 - 8024, 4553, 4554, 8025, 3727, 4041, 4042, 8026, 153, 4555, 356, 8027, 1891, 2888, 4276, 2143, # 7750 - 408, 803, 2352, 8028, 3854, 8029, 4277, 1646, 2570, 2511, 4556, 4557, 3855, 8030, 3856, 4278, # 7766 - 8031, 2411, 3396, 752, 8032, 8033, 1961, 2964, 8034, 746, 3012, 2465, 8035, 4279, 3728, 698, # 7782 - 4558, 1892, 4280, 3608, 2543, 4559, 3609, 3857, 8036, 3166, 3397, 8037, 1823, 1302, 4043, 2706, # 7798 - 3858, 1973, 4281, 8038, 4282, 3167, 823, 1303, 1288, 1236, 2848, 3495, 4044, 3398, 774, 3859, # 7814 - 8039, 1581, 4560, 1304, 2849, 3860, 4561, 8040, 2435, 2161, 1083, 3237, 4283, 4045, 4284, 344, # 7830 - 1173, 288, 2311, 454, 1683, 8041, 8042, 1461, 4562, 4046, 2589, 8043, 8044, 4563, 985, 894, # 7846 - 8045, 3399, 3168, 8046, 1913, 2928, 3729, 1988, 8047, 2110, 1974, 8048, 4047, 8049, 2571, 1194, # 7862 - 425, 8050, 4564, 3169, 1245, 3730, 4285, 8051, 8052, 2850, 8053, 636, 4565, 1855, 3861, 760, # 7878 - 1799, 8054, 4286, 2209, 1508, 4566, 4048, 1893, 1684, 2293, 8055, 8056, 8057, 4287, 4288, 2210, # 7894 - 479, 8058, 8059, 832, 8060, 4049, 2489, 8061, 2965, 2490, 3731, 990, 3109, 627, 1814, 2642, # 7910 - 4289, 1582, 4290, 2125, 2111, 3496, 4567, 8062, 799, 4291, 3170, 8063, 4568, 2112, 1737, 3013, # 7926 - 1018, 543, 754, 4292, 3309, 1676, 4569, 4570, 4050, 8064, 1489, 8065, 3497, 8066, 2614, 2889, # 7942 - 4051, 8067, 8068, 2966, 8069, 8070, 8071, 8072, 3171, 4571, 4572, 2182, 1722, 8073, 3238, 3239, # 7958 - 1842, 3610, 1715, 481, 365, 1975, 1856, 8074, 8075, 1962, 2491, 4573, 8076, 2126, 3611, 3240, # 7974 - 433, 1894, 2063, 2075, 8077, 602, 2741, 8078, 8079, 8080, 8081, 8082, 3014, 1628, 3400, 8083, # 7990 - 3172, 4574, 4052, 2890, 4575, 2512, 8084, 2544, 2772, 8085, 8086, 8087, 3310, 4576, 2891, 8088, # 8006 - 4577, 8089, 2851, 4578, 4579, 1221, 2967, 4053, 2513, 8090, 8091, 8092, 1867, 1989, 8093, 8094, # 8022 - 8095, 1895, 8096, 8097, 4580, 1896, 4054, 318, 8098, 2094, 4055, 4293, 8099, 8100, 485, 8101, # 8038 - 938, 3862, 553, 2670, 116, 8102, 3863, 3612, 8103, 3498, 2671, 2773, 3401, 3311, 2807, 8104, # 8054 - 3613, 2929, 4056, 1747, 2930, 2968, 8105, 8106, 207, 8107, 8108, 2672, 4581, 2514, 8109, 3015, # 8070 - 890, 3614, 3864, 8110, 1877, 3732, 3402, 8111, 2183, 2353, 3403, 1652, 8112, 8113, 8114, 941, # 8086 - 2294, 208, 3499, 4057, 2019, 330, 4294, 3865, 2892, 2492, 3733, 4295, 8115, 8116, 8117, 8118, # 8102 -) -# fmt: on diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/euctwprober.py b/venv/lib/python3.11/site-packages/pip/_vendor/chardet/euctwprober.py deleted file mode 100644 index a37ab18..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/euctwprober.py +++ /dev/null @@ -1,47 +0,0 @@ -######################## BEGIN LICENSE BLOCK ######################## -# The Original Code is mozilla.org code. -# -# The Initial Developer of the Original Code is -# Netscape Communications Corporation. -# Portions created by the Initial Developer are Copyright (C) 1998 -# the Initial Developer. All Rights Reserved. -# -# Contributor(s): -# Mark Pilgrim - port to Python -# -# This library is free software; you can redistribute it and/or -# modify it under the terms of the GNU Lesser General Public -# License as published by the Free Software Foundation; either -# version 2.1 of the License, or (at your option) any later version. -# -# This library is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# Lesser General Public License for more details. -# -# You should have received a copy of the GNU Lesser General Public -# License along with this library; if not, write to the Free Software -# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA -# 02110-1301 USA -######################### END LICENSE BLOCK ######################### - -from .chardistribution import EUCTWDistributionAnalysis -from .codingstatemachine import CodingStateMachine -from .mbcharsetprober import MultiByteCharSetProber -from .mbcssm import EUCTW_SM_MODEL - - -class EUCTWProber(MultiByteCharSetProber): - def __init__(self) -> None: - super().__init__() - self.coding_sm = CodingStateMachine(EUCTW_SM_MODEL) - self.distribution_analyzer = EUCTWDistributionAnalysis() - self.reset() - - @property - def charset_name(self) -> str: - return "EUC-TW" - - @property - def language(self) -> str: - return "Taiwan" diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/gb2312freq.py b/venv/lib/python3.11/site-packages/pip/_vendor/chardet/gb2312freq.py deleted file mode 100644 index b32bfc7..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/gb2312freq.py +++ /dev/null @@ -1,284 +0,0 @@ -######################## BEGIN LICENSE BLOCK ######################## -# The Original Code is Mozilla Communicator client code. -# -# The Initial Developer of the Original Code is -# Netscape Communications Corporation. -# Portions created by the Initial Developer are Copyright (C) 1998 -# the Initial Developer. All Rights Reserved. -# -# Contributor(s): -# Mark Pilgrim - port to Python -# -# This library is free software; you can redistribute it and/or -# modify it under the terms of the GNU Lesser General Public -# License as published by the Free Software Foundation; either -# version 2.1 of the License, or (at your option) any later version. -# -# This library is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# Lesser General Public License for more details. -# -# You should have received a copy of the GNU Lesser General Public -# License along with this library; if not, write to the Free Software -# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA -# 02110-1301 USA -######################### END LICENSE BLOCK ######################### - -# GB2312 most frequently used character table -# -# Char to FreqOrder table , from hz6763 - -# 512 --> 0.79 -- 0.79 -# 1024 --> 0.92 -- 0.13 -# 2048 --> 0.98 -- 0.06 -# 6768 --> 1.00 -- 0.02 -# -# Ideal Distribution Ratio = 0.79135/(1-0.79135) = 3.79 -# Random Distribution Ration = 512 / (3755 - 512) = 0.157 -# -# Typical Distribution Ratio about 25% of Ideal one, still much higher that RDR - -GB2312_TYPICAL_DISTRIBUTION_RATIO = 0.9 - -GB2312_TABLE_SIZE = 3760 - -# fmt: off -GB2312_CHAR_TO_FREQ_ORDER = ( -1671, 749,1443,2364,3924,3807,2330,3921,1704,3463,2691,1511,1515, 572,3191,2205, -2361, 224,2558, 479,1711, 963,3162, 440,4060,1905,2966,2947,3580,2647,3961,3842, -2204, 869,4207, 970,2678,5626,2944,2956,1479,4048, 514,3595, 588,1346,2820,3409, - 249,4088,1746,1873,2047,1774, 581,1813, 358,1174,3590,1014,1561,4844,2245, 670, -1636,3112, 889,1286, 953, 556,2327,3060,1290,3141, 613, 185,3477,1367, 850,3820, -1715,2428,2642,2303,2732,3041,2562,2648,3566,3946,1349, 388,3098,2091,1360,3585, - 152,1687,1539, 738,1559, 59,1232,2925,2267,1388,1249,1741,1679,2960, 151,1566, -1125,1352,4271, 924,4296, 385,3166,4459, 310,1245,2850, 70,3285,2729,3534,3575, -2398,3298,3466,1960,2265, 217,3647, 864,1909,2084,4401,2773,1010,3269,5152, 853, -3051,3121,1244,4251,1895, 364,1499,1540,2313,1180,3655,2268, 562, 715,2417,3061, - 544, 336,3768,2380,1752,4075, 950, 280,2425,4382, 183,2759,3272, 333,4297,2155, -1688,2356,1444,1039,4540, 736,1177,3349,2443,2368,2144,2225, 565, 196,1482,3406, - 927,1335,4147, 692, 878,1311,1653,3911,3622,1378,4200,1840,2969,3149,2126,1816, -2534,1546,2393,2760, 737,2494, 13, 447, 245,2747, 38,2765,2129,2589,1079, 606, - 360, 471,3755,2890, 404, 848, 699,1785,1236, 370,2221,1023,3746,2074,2026,2023, -2388,1581,2119, 812,1141,3091,2536,1519, 804,2053, 406,1596,1090, 784, 548,4414, -1806,2264,2936,1100, 343,4114,5096, 622,3358, 743,3668,1510,1626,5020,3567,2513, -3195,4115,5627,2489,2991, 24,2065,2697,1087,2719, 48,1634, 315, 68, 985,2052, - 198,2239,1347,1107,1439, 597,2366,2172, 871,3307, 919,2487,2790,1867, 236,2570, -1413,3794, 906,3365,3381,1701,1982,1818,1524,2924,1205, 616,2586,2072,2004, 575, - 253,3099, 32,1365,1182, 197,1714,2454,1201, 554,3388,3224,2748, 756,2587, 250, -2567,1507,1517,3529,1922,2761,2337,3416,1961,1677,2452,2238,3153, 615, 911,1506, -1474,2495,1265,1906,2749,3756,3280,2161, 898,2714,1759,3450,2243,2444, 563, 26, -3286,2266,3769,3344,2707,3677, 611,1402, 531,1028,2871,4548,1375, 261,2948, 835, -1190,4134, 353, 840,2684,1900,3082,1435,2109,1207,1674, 329,1872,2781,4055,2686, -2104, 608,3318,2423,2957,2768,1108,3739,3512,3271,3985,2203,1771,3520,1418,2054, -1681,1153, 225,1627,2929, 162,2050,2511,3687,1954, 124,1859,2431,1684,3032,2894, - 585,4805,3969,2869,2704,2088,2032,2095,3656,2635,4362,2209, 256, 518,2042,2105, -3777,3657, 643,2298,1148,1779, 190, 989,3544, 414, 11,2135,2063,2979,1471, 403, -3678, 126, 770,1563, 671,2499,3216,2877, 600,1179, 307,2805,4937,1268,1297,2694, - 252,4032,1448,1494,1331,1394, 127,2256, 222,1647,1035,1481,3056,1915,1048, 873, -3651, 210, 33,1608,2516, 200,1520, 415, 102, 0,3389,1287, 817, 91,3299,2940, - 836,1814, 549,2197,1396,1669,2987,3582,2297,2848,4528,1070, 687, 20,1819, 121, -1552,1364,1461,1968,2617,3540,2824,2083, 177, 948,4938,2291, 110,4549,2066, 648, -3359,1755,2110,2114,4642,4845,1693,3937,3308,1257,1869,2123, 208,1804,3159,2992, -2531,2549,3361,2418,1350,2347,2800,2568,1291,2036,2680, 72, 842,1990, 212,1233, -1154,1586, 75,2027,3410,4900,1823,1337,2710,2676, 728,2810,1522,3026,4995, 157, - 755,1050,4022, 710, 785,1936,2194,2085,1406,2777,2400, 150,1250,4049,1206, 807, -1910, 534, 529,3309,1721,1660, 274, 39,2827, 661,2670,1578, 925,3248,3815,1094, -4278,4901,4252, 41,1150,3747,2572,2227,4501,3658,4902,3813,3357,3617,2884,2258, - 887, 538,4187,3199,1294,2439,3042,2329,2343,2497,1255, 107, 543,1527, 521,3478, -3568, 194,5062, 15, 961,3870,1241,1192,2664, 66,5215,3260,2111,1295,1127,2152, -3805,4135, 901,1164,1976, 398,1278, 530,1460, 748, 904,1054,1966,1426, 53,2909, - 509, 523,2279,1534, 536,1019, 239,1685, 460,2353, 673,1065,2401,3600,4298,2272, -1272,2363, 284,1753,3679,4064,1695, 81, 815,2677,2757,2731,1386, 859, 500,4221, -2190,2566, 757,1006,2519,2068,1166,1455, 337,2654,3203,1863,1682,1914,3025,1252, -1409,1366, 847, 714,2834,2038,3209, 964,2970,1901, 885,2553,1078,1756,3049, 301, -1572,3326, 688,2130,1996,2429,1805,1648,2930,3421,2750,3652,3088, 262,1158,1254, - 389,1641,1812, 526,1719, 923,2073,1073,1902, 468, 489,4625,1140, 857,2375,3070, -3319,2863, 380, 116,1328,2693,1161,2244, 273,1212,1884,2769,3011,1775,1142, 461, -3066,1200,2147,2212, 790, 702,2695,4222,1601,1058, 434,2338,5153,3640, 67,2360, -4099,2502, 618,3472,1329, 416,1132, 830,2782,1807,2653,3211,3510,1662, 192,2124, - 296,3979,1739,1611,3684, 23, 118, 324, 446,1239,1225, 293,2520,3814,3795,2535, -3116, 17,1074, 467,2692,2201, 387,2922, 45,1326,3055,1645,3659,2817, 958, 243, -1903,2320,1339,2825,1784,3289, 356, 576, 865,2315,2381,3377,3916,1088,3122,1713, -1655, 935, 628,4689,1034,1327, 441, 800, 720, 894,1979,2183,1528,5289,2702,1071, -4046,3572,2399,1571,3281, 79, 761,1103, 327, 134, 758,1899,1371,1615, 879, 442, - 215,2605,2579, 173,2048,2485,1057,2975,3317,1097,2253,3801,4263,1403,1650,2946, - 814,4968,3487,1548,2644,1567,1285, 2, 295,2636, 97, 946,3576, 832, 141,4257, -3273, 760,3821,3521,3156,2607, 949,1024,1733,1516,1803,1920,2125,2283,2665,3180, -1501,2064,3560,2171,1592, 803,3518,1416, 732,3897,4258,1363,1362,2458, 119,1427, - 602,1525,2608,1605,1639,3175, 694,3064, 10, 465, 76,2000,4846,4208, 444,3781, -1619,3353,2206,1273,3796, 740,2483, 320,1723,2377,3660,2619,1359,1137,1762,1724, -2345,2842,1850,1862, 912, 821,1866, 612,2625,1735,2573,3369,1093, 844, 89, 937, - 930,1424,3564,2413,2972,1004,3046,3019,2011, 711,3171,1452,4178, 428, 801,1943, - 432, 445,2811, 206,4136,1472, 730, 349, 73, 397,2802,2547, 998,1637,1167, 789, - 396,3217, 154,1218, 716,1120,1780,2819,4826,1931,3334,3762,2139,1215,2627, 552, -3664,3628,3232,1405,2383,3111,1356,2652,3577,3320,3101,1703, 640,1045,1370,1246, -4996, 371,1575,2436,1621,2210, 984,4033,1734,2638, 16,4529, 663,2755,3255,1451, -3917,2257,1253,1955,2234,1263,2951, 214,1229, 617, 485, 359,1831,1969, 473,2310, - 750,2058, 165, 80,2864,2419, 361,4344,2416,2479,1134, 796,3726,1266,2943, 860, -2715, 938, 390,2734,1313,1384, 248, 202, 877,1064,2854, 522,3907, 279,1602, 297, -2357, 395,3740, 137,2075, 944,4089,2584,1267,3802, 62,1533,2285, 178, 176, 780, -2440, 201,3707, 590, 478,1560,4354,2117,1075, 30, 74,4643,4004,1635,1441,2745, - 776,2596, 238,1077,1692,1912,2844, 605, 499,1742,3947, 241,3053, 980,1749, 936, -2640,4511,2582, 515,1543,2162,5322,2892,2993, 890,2148,1924, 665,1827,3581,1032, - 968,3163, 339,1044,1896, 270, 583,1791,1720,4367,1194,3488,3669, 43,2523,1657, - 163,2167, 290,1209,1622,3378, 550, 634,2508,2510, 695,2634,2384,2512,1476,1414, - 220,1469,2341,2138,2852,3183,2900,4939,2865,3502,1211,3680, 854,3227,1299,2976, -3172, 186,2998,1459, 443,1067,3251,1495, 321,1932,3054, 909, 753,1410,1828, 436, -2441,1119,1587,3164,2186,1258, 227, 231,1425,1890,3200,3942, 247, 959, 725,5254, -2741, 577,2158,2079, 929, 120, 174, 838,2813, 591,1115, 417,2024, 40,3240,1536, -1037, 291,4151,2354, 632,1298,2406,2500,3535,1825,1846,3451, 205,1171, 345,4238, - 18,1163, 811, 685,2208,1217, 425,1312,1508,1175,4308,2552,1033, 587,1381,3059, -2984,3482, 340,1316,4023,3972, 792,3176, 519, 777,4690, 918, 933,4130,2981,3741, - 90,3360,2911,2200,5184,4550, 609,3079,2030, 272,3379,2736, 363,3881,1130,1447, - 286, 779, 357,1169,3350,3137,1630,1220,2687,2391, 747,1277,3688,2618,2682,2601, -1156,3196,5290,4034,3102,1689,3596,3128, 874, 219,2783, 798, 508,1843,2461, 269, -1658,1776,1392,1913,2983,3287,2866,2159,2372, 829,4076, 46,4253,2873,1889,1894, - 915,1834,1631,2181,2318, 298, 664,2818,3555,2735, 954,3228,3117, 527,3511,2173, - 681,2712,3033,2247,2346,3467,1652, 155,2164,3382, 113,1994, 450, 899, 494, 994, -1237,2958,1875,2336,1926,3727, 545,1577,1550, 633,3473, 204,1305,3072,2410,1956, -2471, 707,2134, 841,2195,2196,2663,3843,1026,4940, 990,3252,4997, 368,1092, 437, -3212,3258,1933,1829, 675,2977,2893, 412, 943,3723,4644,3294,3283,2230,2373,5154, -2389,2241,2661,2323,1404,2524, 593, 787, 677,3008,1275,2059, 438,2709,2609,2240, -2269,2246,1446, 36,1568,1373,3892,1574,2301,1456,3962, 693,2276,5216,2035,1143, -2720,1919,1797,1811,2763,4137,2597,1830,1699,1488,1198,2090, 424,1694, 312,3634, -3390,4179,3335,2252,1214, 561,1059,3243,2295,2561, 975,5155,2321,2751,3772, 472, -1537,3282,3398,1047,2077,2348,2878,1323,3340,3076, 690,2906, 51, 369, 170,3541, -1060,2187,2688,3670,2541,1083,1683, 928,3918, 459, 109,4427, 599,3744,4286, 143, -2101,2730,2490, 82,1588,3036,2121, 281,1860, 477,4035,1238,2812,3020,2716,3312, -1530,2188,2055,1317, 843, 636,1808,1173,3495, 649, 181,1002, 147,3641,1159,2414, -3750,2289,2795, 813,3123,2610,1136,4368, 5,3391,4541,2174, 420, 429,1728, 754, -1228,2115,2219, 347,2223,2733, 735,1518,3003,2355,3134,1764,3948,3329,1888,2424, -1001,1234,1972,3321,3363,1672,1021,1450,1584, 226, 765, 655,2526,3404,3244,2302, -3665, 731, 594,2184, 319,1576, 621, 658,2656,4299,2099,3864,1279,2071,2598,2739, - 795,3086,3699,3908,1707,2352,2402,1382,3136,2475,1465,4847,3496,3865,1085,3004, -2591,1084, 213,2287,1963,3565,2250, 822, 793,4574,3187,1772,1789,3050, 595,1484, -1959,2770,1080,2650, 456, 422,2996, 940,3322,4328,4345,3092,2742, 965,2784, 739, -4124, 952,1358,2498,2949,2565, 332,2698,2378, 660,2260,2473,4194,3856,2919, 535, -1260,2651,1208,1428,1300,1949,1303,2942, 433,2455,2450,1251,1946, 614,1269, 641, -1306,1810,2737,3078,2912, 564,2365,1419,1415,1497,4460,2367,2185,1379,3005,1307, -3218,2175,1897,3063, 682,1157,4040,4005,1712,1160,1941,1399, 394, 402,2952,1573, -1151,2986,2404, 862, 299,2033,1489,3006, 346, 171,2886,3401,1726,2932, 168,2533, - 47,2507,1030,3735,1145,3370,1395,1318,1579,3609,4560,2857,4116,1457,2529,1965, - 504,1036,2690,2988,2405, 745,5871, 849,2397,2056,3081, 863,2359,3857,2096, 99, -1397,1769,2300,4428,1643,3455,1978,1757,3718,1440, 35,4879,3742,1296,4228,2280, - 160,5063,1599,2013, 166, 520,3479,1646,3345,3012, 490,1937,1545,1264,2182,2505, -1096,1188,1369,1436,2421,1667,2792,2460,1270,2122, 727,3167,2143, 806,1706,1012, -1800,3037, 960,2218,1882, 805, 139,2456,1139,1521, 851,1052,3093,3089, 342,2039, - 744,5097,1468,1502,1585,2087, 223, 939, 326,2140,2577, 892,2481,1623,4077, 982, -3708, 135,2131, 87,2503,3114,2326,1106, 876,1616, 547,2997,2831,2093,3441,4530, -4314, 9,3256,4229,4148, 659,1462,1986,1710,2046,2913,2231,4090,4880,5255,3392, -3274,1368,3689,4645,1477, 705,3384,3635,1068,1529,2941,1458,3782,1509, 100,1656, -2548, 718,2339, 408,1590,2780,3548,1838,4117,3719,1345,3530, 717,3442,2778,3220, -2898,1892,4590,3614,3371,2043,1998,1224,3483, 891, 635, 584,2559,3355, 733,1766, -1729,1172,3789,1891,2307, 781,2982,2271,1957,1580,5773,2633,2005,4195,3097,1535, -3213,1189,1934,5693,3262, 586,3118,1324,1598, 517,1564,2217,1868,1893,4445,3728, -2703,3139,1526,1787,1992,3882,2875,1549,1199,1056,2224,1904,2711,5098,4287, 338, -1993,3129,3489,2689,1809,2815,1997, 957,1855,3898,2550,3275,3057,1105,1319, 627, -1505,1911,1883,3526, 698,3629,3456,1833,1431, 746, 77,1261,2017,2296,1977,1885, - 125,1334,1600, 525,1798,1109,2222,1470,1945, 559,2236,1186,3443,2476,1929,1411, -2411,3135,1777,3372,2621,1841,1613,3229, 668,1430,1839,2643,2916, 195,1989,2671, -2358,1387, 629,3205,2293,5256,4439, 123,1310, 888,1879,4300,3021,3605,1003,1162, -3192,2910,2010, 140,2395,2859, 55,1082,2012,2901, 662, 419,2081,1438, 680,2774, -4654,3912,1620,1731,1625,5035,4065,2328, 512,1344, 802,5443,2163,2311,2537, 524, -3399, 98,1155,2103,1918,2606,3925,2816,1393,2465,1504,3773,2177,3963,1478,4346, - 180,1113,4655,3461,2028,1698, 833,2696,1235,1322,1594,4408,3623,3013,3225,2040, -3022, 541,2881, 607,3632,2029,1665,1219, 639,1385,1686,1099,2803,3231,1938,3188, -2858, 427, 676,2772,1168,2025, 454,3253,2486,3556, 230,1950, 580, 791,1991,1280, -1086,1974,2034, 630, 257,3338,2788,4903,1017, 86,4790, 966,2789,1995,1696,1131, - 259,3095,4188,1308, 179,1463,5257, 289,4107,1248, 42,3413,1725,2288, 896,1947, - 774,4474,4254, 604,3430,4264, 392,2514,2588, 452, 237,1408,3018, 988,4531,1970, -3034,3310, 540,2370,1562,1288,2990, 502,4765,1147, 4,1853,2708, 207, 294,2814, -4078,2902,2509, 684, 34,3105,3532,2551, 644, 709,2801,2344, 573,1727,3573,3557, -2021,1081,3100,4315,2100,3681, 199,2263,1837,2385, 146,3484,1195,2776,3949, 997, -1939,3973,1008,1091,1202,1962,1847,1149,4209,5444,1076, 493, 117,5400,2521, 972, -1490,2934,1796,4542,2374,1512,2933,2657, 413,2888,1135,2762,2314,2156,1355,2369, - 766,2007,2527,2170,3124,2491,2593,2632,4757,2437, 234,3125,3591,1898,1750,1376, -1942,3468,3138, 570,2127,2145,3276,4131, 962, 132,1445,4196, 19, 941,3624,3480, -3366,1973,1374,4461,3431,2629, 283,2415,2275, 808,2887,3620,2112,2563,1353,3610, - 955,1089,3103,1053, 96, 88,4097, 823,3808,1583, 399, 292,4091,3313, 421,1128, - 642,4006, 903,2539,1877,2082, 596, 29,4066,1790, 722,2157, 130, 995,1569, 769, -1485, 464, 513,2213, 288,1923,1101,2453,4316, 133, 486,2445, 50, 625, 487,2207, - 57, 423, 481,2962, 159,3729,1558, 491, 303, 482, 501, 240,2837, 112,3648,2392, -1783, 362, 8,3433,3422, 610,2793,3277,1390,1284,1654, 21,3823, 734, 367, 623, - 193, 287, 374,1009,1483, 816, 476, 313,2255,2340,1262,2150,2899,1146,2581, 782, -2116,1659,2018,1880, 255,3586,3314,1110,2867,2137,2564, 986,2767,5185,2006, 650, - 158, 926, 762, 881,3157,2717,2362,3587, 306,3690,3245,1542,3077,2427,1691,2478, -2118,2985,3490,2438, 539,2305, 983, 129,1754, 355,4201,2386, 827,2923, 104,1773, -2838,2771, 411,2905,3919, 376, 767, 122,1114, 828,2422,1817,3506, 266,3460,1007, -1609,4998, 945,2612,4429,2274, 726,1247,1964,2914,2199,2070,4002,4108, 657,3323, -1422, 579, 455,2764,4737,1222,2895,1670, 824,1223,1487,2525, 558, 861,3080, 598, -2659,2515,1967, 752,2583,2376,2214,4180, 977, 704,2464,4999,2622,4109,1210,2961, - 819,1541, 142,2284, 44, 418, 457,1126,3730,4347,4626,1644,1876,3671,1864, 302, -1063,5694, 624, 723,1984,3745,1314,1676,2488,1610,1449,3558,3569,2166,2098, 409, -1011,2325,3704,2306, 818,1732,1383,1824,1844,3757, 999,2705,3497,1216,1423,2683, -2426,2954,2501,2726,2229,1475,2554,5064,1971,1794,1666,2014,1343, 783, 724, 191, -2434,1354,2220,5065,1763,2752,2472,4152, 131, 175,2885,3434, 92,1466,4920,2616, -3871,3872,3866, 128,1551,1632, 669,1854,3682,4691,4125,1230, 188,2973,3290,1302, -1213, 560,3266, 917, 763,3909,3249,1760, 868,1958, 764,1782,2097, 145,2277,3774, -4462, 64,1491,3062, 971,2132,3606,2442, 221,1226,1617, 218, 323,1185,3207,3147, - 571, 619,1473,1005,1744,2281, 449,1887,2396,3685, 275, 375,3816,1743,3844,3731, - 845,1983,2350,4210,1377, 773, 967,3499,3052,3743,2725,4007,1697,1022,3943,1464, -3264,2855,2722,1952,1029,2839,2467, 84,4383,2215, 820,1391,2015,2448,3672, 377, -1948,2168, 797,2545,3536,2578,2645, 94,2874,1678, 405,1259,3071, 771, 546,1315, - 470,1243,3083, 895,2468, 981, 969,2037, 846,4181, 653,1276,2928, 14,2594, 557, -3007,2474, 156, 902,1338,1740,2574, 537,2518, 973,2282,2216,2433,1928, 138,2903, -1293,2631,1612, 646,3457, 839,2935, 111, 496,2191,2847, 589,3186, 149,3994,2060, -4031,2641,4067,3145,1870, 37,3597,2136,1025,2051,3009,3383,3549,1121,1016,3261, -1301, 251,2446,2599,2153, 872,3246, 637, 334,3705, 831, 884, 921,3065,3140,4092, -2198,1944, 246,2964, 108,2045,1152,1921,2308,1031, 203,3173,4170,1907,3890, 810, -1401,2003,1690, 506, 647,1242,2828,1761,1649,3208,2249,1589,3709,2931,5156,1708, - 498, 666,2613, 834,3817,1231, 184,2851,1124, 883,3197,2261,3710,1765,1553,2658, -1178,2639,2351, 93,1193, 942,2538,2141,4402, 235,1821, 870,1591,2192,1709,1871, -3341,1618,4126,2595,2334, 603, 651, 69, 701, 268,2662,3411,2555,1380,1606, 503, - 448, 254,2371,2646, 574,1187,2309,1770, 322,2235,1292,1801, 305, 566,1133, 229, -2067,2057, 706, 167, 483,2002,2672,3295,1820,3561,3067, 316, 378,2746,3452,1112, - 136,1981, 507,1651,2917,1117, 285,4591, 182,2580,3522,1304, 335,3303,1835,2504, -1795,1792,2248, 674,1018,2106,2449,1857,2292,2845, 976,3047,1781,2600,2727,1389, -1281, 52,3152, 153, 265,3950, 672,3485,3951,4463, 430,1183, 365, 278,2169, 27, -1407,1336,2304, 209,1340,1730,2202,1852,2403,2883, 979,1737,1062, 631,2829,2542, -3876,2592, 825,2086,2226,3048,3625, 352,1417,3724, 542, 991, 431,1351,3938,1861, -2294, 826,1361,2927,3142,3503,1738, 463,2462,2723, 582,1916,1595,2808, 400,3845, -3891,2868,3621,2254, 58,2492,1123, 910,2160,2614,1372,1603,1196,1072,3385,1700, -3267,1980, 696, 480,2430, 920, 799,1570,2920,1951,2041,4047,2540,1321,4223,2469, -3562,2228,1271,2602, 401,2833,3351,2575,5157, 907,2312,1256, 410, 263,3507,1582, - 996, 678,1849,2316,1480, 908,3545,2237, 703,2322, 667,1826,2849,1531,2604,2999, -2407,3146,2151,2630,1786,3711, 469,3542, 497,3899,2409, 858, 837,4446,3393,1274, - 786, 620,1845,2001,3311, 484, 308,3367,1204,1815,3691,2332,1532,2557,1842,2020, -2724,1927,2333,4440, 567, 22,1673,2728,4475,1987,1858,1144,1597, 101,1832,3601, - 12, 974,3783,4391, 951,1412, 1,3720, 453,4608,4041, 528,1041,1027,3230,2628, -1129, 875,1051,3291,1203,2262,1069,2860,2799,2149,2615,3278, 144,1758,3040, 31, - 475,1680, 366,2685,3184, 311,1642,4008,2466,5036,1593,1493,2809, 216,1420,1668, - 233, 304,2128,3284, 232,1429,1768,1040,2008,3407,2740,2967,2543, 242,2133, 778, -1565,2022,2620, 505,2189,2756,1098,2273, 372,1614, 708, 553,2846,2094,2278, 169, -3626,2835,4161, 228,2674,3165, 809,1454,1309, 466,1705,1095, 900,3423, 880,2667, -3751,5258,2317,3109,2571,4317,2766,1503,1342, 866,4447,1118, 63,2076, 314,1881, -1348,1061, 172, 978,3515,1747, 532, 511,3970, 6, 601, 905,2699,3300,1751, 276, -1467,3725,2668, 65,4239,2544,2779,2556,1604, 578,2451,1802, 992,2331,2624,1320, -3446, 713,1513,1013, 103,2786,2447,1661, 886,1702, 916, 654,3574,2031,1556, 751, -2178,2821,2179,1498,1538,2176, 271, 914,2251,2080,1325, 638,1953,2937,3877,2432, -2754, 95,3265,1716, 260,1227,4083, 775, 106,1357,3254, 426,1607, 555,2480, 772, -1985, 244,2546, 474, 495,1046,2611,1851,2061, 71,2089,1675,2590, 742,3758,2843, -3222,1433, 267,2180,2576,2826,2233,2092,3913,2435, 956,1745,3075, 856,2113,1116, - 451, 3,1988,2896,1398, 993,2463,1878,2049,1341,2718,2721,2870,2108, 712,2904, -4363,2753,2324, 277,2872,2349,2649, 384, 987, 435, 691,3000, 922, 164,3939, 652, -1500,1184,4153,2482,3373,2165,4848,2335,3775,3508,3154,2806,2830,1554,2102,1664, -2530,1434,2408, 893,1547,2623,3447,2832,2242,2532,3169,2856,3223,2078, 49,3770, -3469, 462, 318, 656,2259,3250,3069, 679,1629,2758, 344,1138,1104,3120,1836,1283, -3115,2154,1437,4448, 934, 759,1999, 794,2862,1038, 533,2560,1722,2342, 855,2626, -1197,1663,4476,3127, 85,4240,2528, 25,1111,1181,3673, 407,3470,4561,2679,2713, - 768,1925,2841,3986,1544,1165, 932, 373,1240,2146,1930,2673, 721,4766, 354,4333, - 391,2963, 187, 61,3364,1442,1102, 330,1940,1767, 341,3809,4118, 393,2496,2062, -2211, 105, 331, 300, 439, 913,1332, 626, 379,3304,1557, 328, 689,3952, 309,1555, - 931, 317,2517,3027, 325, 569, 686,2107,3084, 60,1042,1333,2794, 264,3177,4014, -1628, 258,3712, 7,4464,1176,1043,1778, 683, 114,1975, 78,1492, 383,1886, 510, - 386, 645,5291,2891,2069,3305,4138,3867,2939,2603,2493,1935,1066,1848,3588,1015, -1282,1289,4609, 697,1453,3044,2666,3611,1856,2412, 54, 719,1330, 568,3778,2459, -1748, 788, 492, 551,1191,1000, 488,3394,3763, 282,1799, 348,2016,1523,3155,2390, -1049, 382,2019,1788,1170, 729,2968,3523, 897,3926,2785,2938,3292, 350,2319,3238, -1718,1717,2655,3453,3143,4465, 161,2889,2980,2009,1421, 56,1908,1640,2387,2232, -1917,1874,2477,4921, 148, 83,3438, 592,4245,2882,1822,1055, 741, 115,1496,1624, - 381,1638,4592,1020, 516,3214, 458, 947,4575,1432, 211,1514,2926,1865,2142, 189, - 852,1221,1400,1486, 882,2299,4036, 351, 28,1122, 700,6479,6480,6481,6482,6483, #last 512 -) -# fmt: on diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/gb2312prober.py b/venv/lib/python3.11/site-packages/pip/_vendor/chardet/gb2312prober.py deleted file mode 100644 index d423e73..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/gb2312prober.py +++ /dev/null @@ -1,47 +0,0 @@ -######################## BEGIN LICENSE BLOCK ######################## -# The Original Code is mozilla.org code. -# -# The Initial Developer of the Original Code is -# Netscape Communications Corporation. -# Portions created by the Initial Developer are Copyright (C) 1998 -# the Initial Developer. All Rights Reserved. -# -# Contributor(s): -# Mark Pilgrim - port to Python -# -# This library is free software; you can redistribute it and/or -# modify it under the terms of the GNU Lesser General Public -# License as published by the Free Software Foundation; either -# version 2.1 of the License, or (at your option) any later version. -# -# This library is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# Lesser General Public License for more details. -# -# You should have received a copy of the GNU Lesser General Public -# License along with this library; if not, write to the Free Software -# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA -# 02110-1301 USA -######################### END LICENSE BLOCK ######################### - -from .chardistribution import GB2312DistributionAnalysis -from .codingstatemachine import CodingStateMachine -from .mbcharsetprober import MultiByteCharSetProber -from .mbcssm import GB2312_SM_MODEL - - -class GB2312Prober(MultiByteCharSetProber): - def __init__(self) -> None: - super().__init__() - self.coding_sm = CodingStateMachine(GB2312_SM_MODEL) - self.distribution_analyzer = GB2312DistributionAnalysis() - self.reset() - - @property - def charset_name(self) -> str: - return "GB2312" - - @property - def language(self) -> str: - return "Chinese" diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/hebrewprober.py b/venv/lib/python3.11/site-packages/pip/_vendor/chardet/hebrewprober.py deleted file mode 100644 index 785d005..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/hebrewprober.py +++ /dev/null @@ -1,316 +0,0 @@ -######################## BEGIN LICENSE BLOCK ######################## -# The Original Code is Mozilla Universal charset detector code. -# -# The Initial Developer of the Original Code is -# Shy Shalom -# Portions created by the Initial Developer are Copyright (C) 2005 -# the Initial Developer. All Rights Reserved. -# -# Contributor(s): -# Mark Pilgrim - port to Python -# -# This library is free software; you can redistribute it and/or -# modify it under the terms of the GNU Lesser General Public -# License as published by the Free Software Foundation; either -# version 2.1 of the License, or (at your option) any later version. -# -# This library is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# Lesser General Public License for more details. -# -# You should have received a copy of the GNU Lesser General Public -# License along with this library; if not, write to the Free Software -# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA -# 02110-1301 USA -######################### END LICENSE BLOCK ######################### - -from typing import Optional, Union - -from .charsetprober import CharSetProber -from .enums import ProbingState -from .sbcharsetprober import SingleByteCharSetProber - -# This prober doesn't actually recognize a language or a charset. -# It is a helper prober for the use of the Hebrew model probers - -### General ideas of the Hebrew charset recognition ### -# -# Four main charsets exist in Hebrew: -# "ISO-8859-8" - Visual Hebrew -# "windows-1255" - Logical Hebrew -# "ISO-8859-8-I" - Logical Hebrew -# "x-mac-hebrew" - ?? Logical Hebrew ?? -# -# Both "ISO" charsets use a completely identical set of code points, whereas -# "windows-1255" and "x-mac-hebrew" are two different proper supersets of -# these code points. windows-1255 defines additional characters in the range -# 0x80-0x9F as some misc punctuation marks as well as some Hebrew-specific -# diacritics and additional 'Yiddish' ligature letters in the range 0xc0-0xd6. -# x-mac-hebrew defines similar additional code points but with a different -# mapping. -# -# As far as an average Hebrew text with no diacritics is concerned, all four -# charsets are identical with respect to code points. Meaning that for the -# main Hebrew alphabet, all four map the same values to all 27 Hebrew letters -# (including final letters). -# -# The dominant difference between these charsets is their directionality. -# "Visual" directionality means that the text is ordered as if the renderer is -# not aware of a BIDI rendering algorithm. The renderer sees the text and -# draws it from left to right. The text itself when ordered naturally is read -# backwards. A buffer of Visual Hebrew generally looks like so: -# "[last word of first line spelled backwards] [whole line ordered backwards -# and spelled backwards] [first word of first line spelled backwards] -# [end of line] [last word of second line] ... etc' " -# adding punctuation marks, numbers and English text to visual text is -# naturally also "visual" and from left to right. -# -# "Logical" directionality means the text is ordered "naturally" according to -# the order it is read. It is the responsibility of the renderer to display -# the text from right to left. A BIDI algorithm is used to place general -# punctuation marks, numbers and English text in the text. -# -# Texts in x-mac-hebrew are almost impossible to find on the Internet. From -# what little evidence I could find, it seems that its general directionality -# is Logical. -# -# To sum up all of the above, the Hebrew probing mechanism knows about two -# charsets: -# Visual Hebrew - "ISO-8859-8" - backwards text - Words and sentences are -# backwards while line order is natural. For charset recognition purposes -# the line order is unimportant (In fact, for this implementation, even -# word order is unimportant). -# Logical Hebrew - "windows-1255" - normal, naturally ordered text. -# -# "ISO-8859-8-I" is a subset of windows-1255 and doesn't need to be -# specifically identified. -# "x-mac-hebrew" is also identified as windows-1255. A text in x-mac-hebrew -# that contain special punctuation marks or diacritics is displayed with -# some unconverted characters showing as question marks. This problem might -# be corrected using another model prober for x-mac-hebrew. Due to the fact -# that x-mac-hebrew texts are so rare, writing another model prober isn't -# worth the effort and performance hit. -# -#### The Prober #### -# -# The prober is divided between two SBCharSetProbers and a HebrewProber, -# all of which are managed, created, fed data, inquired and deleted by the -# SBCSGroupProber. The two SBCharSetProbers identify that the text is in -# fact some kind of Hebrew, Logical or Visual. The final decision about which -# one is it is made by the HebrewProber by combining final-letter scores -# with the scores of the two SBCharSetProbers to produce a final answer. -# -# The SBCSGroupProber is responsible for stripping the original text of HTML -# tags, English characters, numbers, low-ASCII punctuation characters, spaces -# and new lines. It reduces any sequence of such characters to a single space. -# The buffer fed to each prober in the SBCS group prober is pure text in -# high-ASCII. -# The two SBCharSetProbers (model probers) share the same language model: -# Win1255Model. -# The first SBCharSetProber uses the model normally as any other -# SBCharSetProber does, to recognize windows-1255, upon which this model was -# built. The second SBCharSetProber is told to make the pair-of-letter -# lookup in the language model backwards. This in practice exactly simulates -# a visual Hebrew model using the windows-1255 logical Hebrew model. -# -# The HebrewProber is not using any language model. All it does is look for -# final-letter evidence suggesting the text is either logical Hebrew or visual -# Hebrew. Disjointed from the model probers, the results of the HebrewProber -# alone are meaningless. HebrewProber always returns 0.00 as confidence -# since it never identifies a charset by itself. Instead, the pointer to the -# HebrewProber is passed to the model probers as a helper "Name Prober". -# When the Group prober receives a positive identification from any prober, -# it asks for the name of the charset identified. If the prober queried is a -# Hebrew model prober, the model prober forwards the call to the -# HebrewProber to make the final decision. In the HebrewProber, the -# decision is made according to the final-letters scores maintained and Both -# model probers scores. The answer is returned in the form of the name of the -# charset identified, either "windows-1255" or "ISO-8859-8". - - -class HebrewProber(CharSetProber): - SPACE = 0x20 - # windows-1255 / ISO-8859-8 code points of interest - FINAL_KAF = 0xEA - NORMAL_KAF = 0xEB - FINAL_MEM = 0xED - NORMAL_MEM = 0xEE - FINAL_NUN = 0xEF - NORMAL_NUN = 0xF0 - FINAL_PE = 0xF3 - NORMAL_PE = 0xF4 - FINAL_TSADI = 0xF5 - NORMAL_TSADI = 0xF6 - - # Minimum Visual vs Logical final letter score difference. - # If the difference is below this, don't rely solely on the final letter score - # distance. - MIN_FINAL_CHAR_DISTANCE = 5 - - # Minimum Visual vs Logical model score difference. - # If the difference is below this, don't rely at all on the model score - # distance. - MIN_MODEL_DISTANCE = 0.01 - - VISUAL_HEBREW_NAME = "ISO-8859-8" - LOGICAL_HEBREW_NAME = "windows-1255" - - def __init__(self) -> None: - super().__init__() - self._final_char_logical_score = 0 - self._final_char_visual_score = 0 - self._prev = self.SPACE - self._before_prev = self.SPACE - self._logical_prober: Optional[SingleByteCharSetProber] = None - self._visual_prober: Optional[SingleByteCharSetProber] = None - self.reset() - - def reset(self) -> None: - self._final_char_logical_score = 0 - self._final_char_visual_score = 0 - # The two last characters seen in the previous buffer, - # mPrev and mBeforePrev are initialized to space in order to simulate - # a word delimiter at the beginning of the data - self._prev = self.SPACE - self._before_prev = self.SPACE - # These probers are owned by the group prober. - - def set_model_probers( - self, - logical_prober: SingleByteCharSetProber, - visual_prober: SingleByteCharSetProber, - ) -> None: - self._logical_prober = logical_prober - self._visual_prober = visual_prober - - def is_final(self, c: int) -> bool: - return c in [ - self.FINAL_KAF, - self.FINAL_MEM, - self.FINAL_NUN, - self.FINAL_PE, - self.FINAL_TSADI, - ] - - def is_non_final(self, c: int) -> bool: - # The normal Tsadi is not a good Non-Final letter due to words like - # 'lechotet' (to chat) containing an apostrophe after the tsadi. This - # apostrophe is converted to a space in FilterWithoutEnglishLetters - # causing the Non-Final tsadi to appear at an end of a word even - # though this is not the case in the original text. - # The letters Pe and Kaf rarely display a related behavior of not being - # a good Non-Final letter. Words like 'Pop', 'Winamp' and 'Mubarak' - # for example legally end with a Non-Final Pe or Kaf. However, the - # benefit of these letters as Non-Final letters outweighs the damage - # since these words are quite rare. - return c in [self.NORMAL_KAF, self.NORMAL_MEM, self.NORMAL_NUN, self.NORMAL_PE] - - def feed(self, byte_str: Union[bytes, bytearray]) -> ProbingState: - # Final letter analysis for logical-visual decision. - # Look for evidence that the received buffer is either logical Hebrew - # or visual Hebrew. - # The following cases are checked: - # 1) A word longer than 1 letter, ending with a final letter. This is - # an indication that the text is laid out "naturally" since the - # final letter really appears at the end. +1 for logical score. - # 2) A word longer than 1 letter, ending with a Non-Final letter. In - # normal Hebrew, words ending with Kaf, Mem, Nun, Pe or Tsadi, - # should not end with the Non-Final form of that letter. Exceptions - # to this rule are mentioned above in isNonFinal(). This is an - # indication that the text is laid out backwards. +1 for visual - # score - # 3) A word longer than 1 letter, starting with a final letter. Final - # letters should not appear at the beginning of a word. This is an - # indication that the text is laid out backwards. +1 for visual - # score. - # - # The visual score and logical score are accumulated throughout the - # text and are finally checked against each other in GetCharSetName(). - # No checking for final letters in the middle of words is done since - # that case is not an indication for either Logical or Visual text. - # - # We automatically filter out all 7-bit characters (replace them with - # spaces) so the word boundary detection works properly. [MAP] - - if self.state == ProbingState.NOT_ME: - # Both model probers say it's not them. No reason to continue. - return ProbingState.NOT_ME - - byte_str = self.filter_high_byte_only(byte_str) - - for cur in byte_str: - if cur == self.SPACE: - # We stand on a space - a word just ended - if self._before_prev != self.SPACE: - # next-to-last char was not a space so self._prev is not a - # 1 letter word - if self.is_final(self._prev): - # case (1) [-2:not space][-1:final letter][cur:space] - self._final_char_logical_score += 1 - elif self.is_non_final(self._prev): - # case (2) [-2:not space][-1:Non-Final letter][ - # cur:space] - self._final_char_visual_score += 1 - else: - # Not standing on a space - if ( - (self._before_prev == self.SPACE) - and (self.is_final(self._prev)) - and (cur != self.SPACE) - ): - # case (3) [-2:space][-1:final letter][cur:not space] - self._final_char_visual_score += 1 - self._before_prev = self._prev - self._prev = cur - - # Forever detecting, till the end or until both model probers return - # ProbingState.NOT_ME (handled above) - return ProbingState.DETECTING - - @property - def charset_name(self) -> str: - assert self._logical_prober is not None - assert self._visual_prober is not None - - # Make the decision: is it Logical or Visual? - # If the final letter score distance is dominant enough, rely on it. - finalsub = self._final_char_logical_score - self._final_char_visual_score - if finalsub >= self.MIN_FINAL_CHAR_DISTANCE: - return self.LOGICAL_HEBREW_NAME - if finalsub <= -self.MIN_FINAL_CHAR_DISTANCE: - return self.VISUAL_HEBREW_NAME - - # It's not dominant enough, try to rely on the model scores instead. - modelsub = ( - self._logical_prober.get_confidence() - self._visual_prober.get_confidence() - ) - if modelsub > self.MIN_MODEL_DISTANCE: - return self.LOGICAL_HEBREW_NAME - if modelsub < -self.MIN_MODEL_DISTANCE: - return self.VISUAL_HEBREW_NAME - - # Still no good, back to final letter distance, maybe it'll save the - # day. - if finalsub < 0.0: - return self.VISUAL_HEBREW_NAME - - # (finalsub > 0 - Logical) or (don't know what to do) default to - # Logical. - return self.LOGICAL_HEBREW_NAME - - @property - def language(self) -> str: - return "Hebrew" - - @property - def state(self) -> ProbingState: - assert self._logical_prober is not None - assert self._visual_prober is not None - - # Remain active as long as any of the model probers are active. - if (self._logical_prober.state == ProbingState.NOT_ME) and ( - self._visual_prober.state == ProbingState.NOT_ME - ): - return ProbingState.NOT_ME - return ProbingState.DETECTING diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/jisfreq.py b/venv/lib/python3.11/site-packages/pip/_vendor/chardet/jisfreq.py deleted file mode 100644 index 3293576..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/jisfreq.py +++ /dev/null @@ -1,325 +0,0 @@ -######################## BEGIN LICENSE BLOCK ######################## -# The Original Code is Mozilla Communicator client code. -# -# The Initial Developer of the Original Code is -# Netscape Communications Corporation. -# Portions created by the Initial Developer are Copyright (C) 1998 -# the Initial Developer. All Rights Reserved. -# -# Contributor(s): -# Mark Pilgrim - port to Python -# -# This library is free software; you can redistribute it and/or -# modify it under the terms of the GNU Lesser General Public -# License as published by the Free Software Foundation; either -# version 2.1 of the License, or (at your option) any later version. -# -# This library is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# Lesser General Public License for more details. -# -# You should have received a copy of the GNU Lesser General Public -# License along with this library; if not, write to the Free Software -# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA -# 02110-1301 USA -######################### END LICENSE BLOCK ######################### - -# Sampling from about 20M text materials include literature and computer technology -# -# Japanese frequency table, applied to both S-JIS and EUC-JP -# They are sorted in order. - -# 128 --> 0.77094 -# 256 --> 0.85710 -# 512 --> 0.92635 -# 1024 --> 0.97130 -# 2048 --> 0.99431 -# -# Ideal Distribution Ratio = 0.92635 / (1-0.92635) = 12.58 -# Random Distribution Ration = 512 / (2965+62+83+86-512) = 0.191 -# -# Typical Distribution Ratio, 25% of IDR - -JIS_TYPICAL_DISTRIBUTION_RATIO = 3.0 - -# Char to FreqOrder table , -JIS_TABLE_SIZE = 4368 - -# fmt: off -JIS_CHAR_TO_FREQ_ORDER = ( - 40, 1, 6, 182, 152, 180, 295,2127, 285, 381,3295,4304,3068,4606,3165,3510, # 16 -3511,1822,2785,4607,1193,2226,5070,4608, 171,2996,1247, 18, 179,5071, 856,1661, # 32 -1262,5072, 619, 127,3431,3512,3230,1899,1700, 232, 228,1294,1298, 284, 283,2041, # 48 -2042,1061,1062, 48, 49, 44, 45, 433, 434,1040,1041, 996, 787,2997,1255,4305, # 64 -2108,4609,1684,1648,5073,5074,5075,5076,5077,5078,3687,5079,4610,5080,3927,3928, # 80 -5081,3296,3432, 290,2285,1471,2187,5082,2580,2825,1303,2140,1739,1445,2691,3375, # 96 -1691,3297,4306,4307,4611, 452,3376,1182,2713,3688,3069,4308,5083,5084,5085,5086, # 112 -5087,5088,5089,5090,5091,5092,5093,5094,5095,5096,5097,5098,5099,5100,5101,5102, # 128 -5103,5104,5105,5106,5107,5108,5109,5110,5111,5112,4097,5113,5114,5115,5116,5117, # 144 -5118,5119,5120,5121,5122,5123,5124,5125,5126,5127,5128,5129,5130,5131,5132,5133, # 160 -5134,5135,5136,5137,5138,5139,5140,5141,5142,5143,5144,5145,5146,5147,5148,5149, # 176 -5150,5151,5152,4612,5153,5154,5155,5156,5157,5158,5159,5160,5161,5162,5163,5164, # 192 -5165,5166,5167,5168,5169,5170,5171,5172,5173,5174,5175,1472, 598, 618, 820,1205, # 208 -1309,1412,1858,1307,1692,5176,5177,5178,5179,5180,5181,5182,1142,1452,1234,1172, # 224 -1875,2043,2149,1793,1382,2973, 925,2404,1067,1241, 960,1377,2935,1491, 919,1217, # 240 -1865,2030,1406,1499,2749,4098,5183,5184,5185,5186,5187,5188,2561,4099,3117,1804, # 256 -2049,3689,4309,3513,1663,5189,3166,3118,3298,1587,1561,3433,5190,3119,1625,2998, # 272 -3299,4613,1766,3690,2786,4614,5191,5192,5193,5194,2161, 26,3377, 2,3929, 20, # 288 -3691, 47,4100, 50, 17, 16, 35, 268, 27, 243, 42, 155, 24, 154, 29, 184, # 304 - 4, 91, 14, 92, 53, 396, 33, 289, 9, 37, 64, 620, 21, 39, 321, 5, # 320 - 12, 11, 52, 13, 3, 208, 138, 0, 7, 60, 526, 141, 151,1069, 181, 275, # 336 -1591, 83, 132,1475, 126, 331, 829, 15, 69, 160, 59, 22, 157, 55,1079, 312, # 352 - 109, 38, 23, 25, 10, 19, 79,5195, 61, 382,1124, 8, 30,5196,5197,5198, # 368 -5199,5200,5201,5202,5203,5204,5205,5206, 89, 62, 74, 34,2416, 112, 139, 196, # 384 - 271, 149, 84, 607, 131, 765, 46, 88, 153, 683, 76, 874, 101, 258, 57, 80, # 400 - 32, 364, 121,1508, 169,1547, 68, 235, 145,2999, 41, 360,3027, 70, 63, 31, # 416 - 43, 259, 262,1383, 99, 533, 194, 66, 93, 846, 217, 192, 56, 106, 58, 565, # 432 - 280, 272, 311, 256, 146, 82, 308, 71, 100, 128, 214, 655, 110, 261, 104,1140, # 448 - 54, 51, 36, 87, 67,3070, 185,2618,2936,2020, 28,1066,2390,2059,5207,5208, # 464 -5209,5210,5211,5212,5213,5214,5215,5216,4615,5217,5218,5219,5220,5221,5222,5223, # 480 -5224,5225,5226,5227,5228,5229,5230,5231,5232,5233,5234,5235,5236,3514,5237,5238, # 496 -5239,5240,5241,5242,5243,5244,2297,2031,4616,4310,3692,5245,3071,5246,3598,5247, # 512 -4617,3231,3515,5248,4101,4311,4618,3808,4312,4102,5249,4103,4104,3599,5250,5251, # 528 -5252,5253,5254,5255,5256,5257,5258,5259,5260,5261,5262,5263,5264,5265,5266,5267, # 544 -5268,5269,5270,5271,5272,5273,5274,5275,5276,5277,5278,5279,5280,5281,5282,5283, # 560 -5284,5285,5286,5287,5288,5289,5290,5291,5292,5293,5294,5295,5296,5297,5298,5299, # 576 -5300,5301,5302,5303,5304,5305,5306,5307,5308,5309,5310,5311,5312,5313,5314,5315, # 592 -5316,5317,5318,5319,5320,5321,5322,5323,5324,5325,5326,5327,5328,5329,5330,5331, # 608 -5332,5333,5334,5335,5336,5337,5338,5339,5340,5341,5342,5343,5344,5345,5346,5347, # 624 -5348,5349,5350,5351,5352,5353,5354,5355,5356,5357,5358,5359,5360,5361,5362,5363, # 640 -5364,5365,5366,5367,5368,5369,5370,5371,5372,5373,5374,5375,5376,5377,5378,5379, # 656 -5380,5381, 363, 642,2787,2878,2788,2789,2316,3232,2317,3434,2011, 165,1942,3930, # 672 -3931,3932,3933,5382,4619,5383,4620,5384,5385,5386,5387,5388,5389,5390,5391,5392, # 688 -5393,5394,5395,5396,5397,5398,5399,5400,5401,5402,5403,5404,5405,5406,5407,5408, # 704 -5409,5410,5411,5412,5413,5414,5415,5416,5417,5418,5419,5420,5421,5422,5423,5424, # 720 -5425,5426,5427,5428,5429,5430,5431,5432,5433,5434,5435,5436,5437,5438,5439,5440, # 736 -5441,5442,5443,5444,5445,5446,5447,5448,5449,5450,5451,5452,5453,5454,5455,5456, # 752 -5457,5458,5459,5460,5461,5462,5463,5464,5465,5466,5467,5468,5469,5470,5471,5472, # 768 -5473,5474,5475,5476,5477,5478,5479,5480,5481,5482,5483,5484,5485,5486,5487,5488, # 784 -5489,5490,5491,5492,5493,5494,5495,5496,5497,5498,5499,5500,5501,5502,5503,5504, # 800 -5505,5506,5507,5508,5509,5510,5511,5512,5513,5514,5515,5516,5517,5518,5519,5520, # 816 -5521,5522,5523,5524,5525,5526,5527,5528,5529,5530,5531,5532,5533,5534,5535,5536, # 832 -5537,5538,5539,5540,5541,5542,5543,5544,5545,5546,5547,5548,5549,5550,5551,5552, # 848 -5553,5554,5555,5556,5557,5558,5559,5560,5561,5562,5563,5564,5565,5566,5567,5568, # 864 -5569,5570,5571,5572,5573,5574,5575,5576,5577,5578,5579,5580,5581,5582,5583,5584, # 880 -5585,5586,5587,5588,5589,5590,5591,5592,5593,5594,5595,5596,5597,5598,5599,5600, # 896 -5601,5602,5603,5604,5605,5606,5607,5608,5609,5610,5611,5612,5613,5614,5615,5616, # 912 -5617,5618,5619,5620,5621,5622,5623,5624,5625,5626,5627,5628,5629,5630,5631,5632, # 928 -5633,5634,5635,5636,5637,5638,5639,5640,5641,5642,5643,5644,5645,5646,5647,5648, # 944 -5649,5650,5651,5652,5653,5654,5655,5656,5657,5658,5659,5660,5661,5662,5663,5664, # 960 -5665,5666,5667,5668,5669,5670,5671,5672,5673,5674,5675,5676,5677,5678,5679,5680, # 976 -5681,5682,5683,5684,5685,5686,5687,5688,5689,5690,5691,5692,5693,5694,5695,5696, # 992 -5697,5698,5699,5700,5701,5702,5703,5704,5705,5706,5707,5708,5709,5710,5711,5712, # 1008 -5713,5714,5715,5716,5717,5718,5719,5720,5721,5722,5723,5724,5725,5726,5727,5728, # 1024 -5729,5730,5731,5732,5733,5734,5735,5736,5737,5738,5739,5740,5741,5742,5743,5744, # 1040 -5745,5746,5747,5748,5749,5750,5751,5752,5753,5754,5755,5756,5757,5758,5759,5760, # 1056 -5761,5762,5763,5764,5765,5766,5767,5768,5769,5770,5771,5772,5773,5774,5775,5776, # 1072 -5777,5778,5779,5780,5781,5782,5783,5784,5785,5786,5787,5788,5789,5790,5791,5792, # 1088 -5793,5794,5795,5796,5797,5798,5799,5800,5801,5802,5803,5804,5805,5806,5807,5808, # 1104 -5809,5810,5811,5812,5813,5814,5815,5816,5817,5818,5819,5820,5821,5822,5823,5824, # 1120 -5825,5826,5827,5828,5829,5830,5831,5832,5833,5834,5835,5836,5837,5838,5839,5840, # 1136 -5841,5842,5843,5844,5845,5846,5847,5848,5849,5850,5851,5852,5853,5854,5855,5856, # 1152 -5857,5858,5859,5860,5861,5862,5863,5864,5865,5866,5867,5868,5869,5870,5871,5872, # 1168 -5873,5874,5875,5876,5877,5878,5879,5880,5881,5882,5883,5884,5885,5886,5887,5888, # 1184 -5889,5890,5891,5892,5893,5894,5895,5896,5897,5898,5899,5900,5901,5902,5903,5904, # 1200 -5905,5906,5907,5908,5909,5910,5911,5912,5913,5914,5915,5916,5917,5918,5919,5920, # 1216 -5921,5922,5923,5924,5925,5926,5927,5928,5929,5930,5931,5932,5933,5934,5935,5936, # 1232 -5937,5938,5939,5940,5941,5942,5943,5944,5945,5946,5947,5948,5949,5950,5951,5952, # 1248 -5953,5954,5955,5956,5957,5958,5959,5960,5961,5962,5963,5964,5965,5966,5967,5968, # 1264 -5969,5970,5971,5972,5973,5974,5975,5976,5977,5978,5979,5980,5981,5982,5983,5984, # 1280 -5985,5986,5987,5988,5989,5990,5991,5992,5993,5994,5995,5996,5997,5998,5999,6000, # 1296 -6001,6002,6003,6004,6005,6006,6007,6008,6009,6010,6011,6012,6013,6014,6015,6016, # 1312 -6017,6018,6019,6020,6021,6022,6023,6024,6025,6026,6027,6028,6029,6030,6031,6032, # 1328 -6033,6034,6035,6036,6037,6038,6039,6040,6041,6042,6043,6044,6045,6046,6047,6048, # 1344 -6049,6050,6051,6052,6053,6054,6055,6056,6057,6058,6059,6060,6061,6062,6063,6064, # 1360 -6065,6066,6067,6068,6069,6070,6071,6072,6073,6074,6075,6076,6077,6078,6079,6080, # 1376 -6081,6082,6083,6084,6085,6086,6087,6088,6089,6090,6091,6092,6093,6094,6095,6096, # 1392 -6097,6098,6099,6100,6101,6102,6103,6104,6105,6106,6107,6108,6109,6110,6111,6112, # 1408 -6113,6114,2044,2060,4621, 997,1235, 473,1186,4622, 920,3378,6115,6116, 379,1108, # 1424 -4313,2657,2735,3934,6117,3809, 636,3233, 573,1026,3693,3435,2974,3300,2298,4105, # 1440 - 854,2937,2463, 393,2581,2417, 539, 752,1280,2750,2480, 140,1161, 440, 708,1569, # 1456 - 665,2497,1746,1291,1523,3000, 164,1603, 847,1331, 537,1997, 486, 508,1693,2418, # 1472 -1970,2227, 878,1220, 299,1030, 969, 652,2751, 624,1137,3301,2619, 65,3302,2045, # 1488 -1761,1859,3120,1930,3694,3516, 663,1767, 852, 835,3695, 269, 767,2826,2339,1305, # 1504 - 896,1150, 770,1616,6118, 506,1502,2075,1012,2519, 775,2520,2975,2340,2938,4314, # 1520 -3028,2086,1224,1943,2286,6119,3072,4315,2240,1273,1987,3935,1557, 175, 597, 985, # 1536 -3517,2419,2521,1416,3029, 585, 938,1931,1007,1052,1932,1685,6120,3379,4316,4623, # 1552 - 804, 599,3121,1333,2128,2539,1159,1554,2032,3810, 687,2033,2904, 952, 675,1467, # 1568 -3436,6121,2241,1096,1786,2440,1543,1924, 980,1813,2228, 781,2692,1879, 728,1918, # 1584 -3696,4624, 548,1950,4625,1809,1088,1356,3303,2522,1944, 502, 972, 373, 513,2827, # 1600 - 586,2377,2391,1003,1976,1631,6122,2464,1084, 648,1776,4626,2141, 324, 962,2012, # 1616 -2177,2076,1384, 742,2178,1448,1173,1810, 222, 102, 301, 445, 125,2420, 662,2498, # 1632 - 277, 200,1476,1165,1068, 224,2562,1378,1446, 450,1880, 659, 791, 582,4627,2939, # 1648 -3936,1516,1274, 555,2099,3697,1020,1389,1526,3380,1762,1723,1787,2229, 412,2114, # 1664 -1900,2392,3518, 512,2597, 427,1925,2341,3122,1653,1686,2465,2499, 697, 330, 273, # 1680 - 380,2162, 951, 832, 780, 991,1301,3073, 965,2270,3519, 668,2523,2636,1286, 535, # 1696 -1407, 518, 671, 957,2658,2378, 267, 611,2197,3030,6123, 248,2299, 967,1799,2356, # 1712 - 850,1418,3437,1876,1256,1480,2828,1718,6124,6125,1755,1664,2405,6126,4628,2879, # 1728 -2829, 499,2179, 676,4629, 557,2329,2214,2090, 325,3234, 464, 811,3001, 992,2342, # 1744 -2481,1232,1469, 303,2242, 466,1070,2163, 603,1777,2091,4630,2752,4631,2714, 322, # 1760 -2659,1964,1768, 481,2188,1463,2330,2857,3600,2092,3031,2421,4632,2318,2070,1849, # 1776 -2598,4633,1302,2254,1668,1701,2422,3811,2905,3032,3123,2046,4106,1763,1694,4634, # 1792 -1604, 943,1724,1454, 917, 868,2215,1169,2940, 552,1145,1800,1228,1823,1955, 316, # 1808 -1080,2510, 361,1807,2830,4107,2660,3381,1346,1423,1134,4108,6127, 541,1263,1229, # 1824 -1148,2540, 545, 465,1833,2880,3438,1901,3074,2482, 816,3937, 713,1788,2500, 122, # 1840 -1575, 195,1451,2501,1111,6128, 859, 374,1225,2243,2483,4317, 390,1033,3439,3075, # 1856 -2524,1687, 266, 793,1440,2599, 946, 779, 802, 507, 897,1081, 528,2189,1292, 711, # 1872 -1866,1725,1167,1640, 753, 398,2661,1053, 246, 348,4318, 137,1024,3440,1600,2077, # 1888 -2129, 825,4319, 698, 238, 521, 187,2300,1157,2423,1641,1605,1464,1610,1097,2541, # 1904 -1260,1436, 759,2255,1814,2150, 705,3235, 409,2563,3304, 561,3033,2005,2564, 726, # 1920 -1956,2343,3698,4109, 949,3812,3813,3520,1669, 653,1379,2525, 881,2198, 632,2256, # 1936 -1027, 778,1074, 733,1957, 514,1481,2466, 554,2180, 702,3938,1606,1017,1398,6129, # 1952 -1380,3521, 921, 993,1313, 594, 449,1489,1617,1166, 768,1426,1360, 495,1794,3601, # 1968 -1177,3602,1170,4320,2344, 476, 425,3167,4635,3168,1424, 401,2662,1171,3382,1998, # 1984 -1089,4110, 477,3169, 474,6130,1909, 596,2831,1842, 494, 693,1051,1028,1207,3076, # 2000 - 606,2115, 727,2790,1473,1115, 743,3522, 630, 805,1532,4321,2021, 366,1057, 838, # 2016 - 684,1114,2142,4322,2050,1492,1892,1808,2271,3814,2424,1971,1447,1373,3305,1090, # 2032 -1536,3939,3523,3306,1455,2199, 336, 369,2331,1035, 584,2393, 902, 718,2600,6131, # 2048 -2753, 463,2151,1149,1611,2467, 715,1308,3124,1268, 343,1413,3236,1517,1347,2663, # 2064 -2093,3940,2022,1131,1553,2100,2941,1427,3441,2942,1323,2484,6132,1980, 872,2368, # 2080 -2441,2943, 320,2369,2116,1082, 679,1933,3941,2791,3815, 625,1143,2023, 422,2200, # 2096 -3816,6133, 730,1695, 356,2257,1626,2301,2858,2637,1627,1778, 937, 883,2906,2693, # 2112 -3002,1769,1086, 400,1063,1325,3307,2792,4111,3077, 456,2345,1046, 747,6134,1524, # 2128 - 884,1094,3383,1474,2164,1059, 974,1688,2181,2258,1047, 345,1665,1187, 358, 875, # 2144 -3170, 305, 660,3524,2190,1334,1135,3171,1540,1649,2542,1527, 927, 968,2793, 885, # 2160 -1972,1850, 482, 500,2638,1218,1109,1085,2543,1654,2034, 876, 78,2287,1482,1277, # 2176 - 861,1675,1083,1779, 724,2754, 454, 397,1132,1612,2332, 893, 672,1237, 257,2259, # 2192 -2370, 135,3384, 337,2244, 547, 352, 340, 709,2485,1400, 788,1138,2511, 540, 772, # 2208 -1682,2260,2272,2544,2013,1843,1902,4636,1999,1562,2288,4637,2201,1403,1533, 407, # 2224 - 576,3308,1254,2071, 978,3385, 170, 136,1201,3125,2664,3172,2394, 213, 912, 873, # 2240 -3603,1713,2202, 699,3604,3699, 813,3442, 493, 531,1054, 468,2907,1483, 304, 281, # 2256 -4112,1726,1252,2094, 339,2319,2130,2639, 756,1563,2944, 748, 571,2976,1588,2425, # 2272 -2715,1851,1460,2426,1528,1392,1973,3237, 288,3309, 685,3386, 296, 892,2716,2216, # 2288 -1570,2245, 722,1747,2217, 905,3238,1103,6135,1893,1441,1965, 251,1805,2371,3700, # 2304 -2601,1919,1078, 75,2182,1509,1592,1270,2640,4638,2152,6136,3310,3817, 524, 706, # 2320 -1075, 292,3818,1756,2602, 317, 98,3173,3605,3525,1844,2218,3819,2502, 814, 567, # 2336 - 385,2908,1534,6137, 534,1642,3239, 797,6138,1670,1529, 953,4323, 188,1071, 538, # 2352 - 178, 729,3240,2109,1226,1374,2000,2357,2977, 731,2468,1116,2014,2051,6139,1261, # 2368 -1593, 803,2859,2736,3443, 556, 682, 823,1541,6140,1369,2289,1706,2794, 845, 462, # 2384 -2603,2665,1361, 387, 162,2358,1740, 739,1770,1720,1304,1401,3241,1049, 627,1571, # 2400 -2427,3526,1877,3942,1852,1500, 431,1910,1503, 677, 297,2795, 286,1433,1038,1198, # 2416 -2290,1133,1596,4113,4639,2469,1510,1484,3943,6141,2442, 108, 712,4640,2372, 866, # 2432 -3701,2755,3242,1348, 834,1945,1408,3527,2395,3243,1811, 824, 994,1179,2110,1548, # 2448 -1453, 790,3003, 690,4324,4325,2832,2909,3820,1860,3821, 225,1748, 310, 346,1780, # 2464 -2470, 821,1993,2717,2796, 828, 877,3528,2860,2471,1702,2165,2910,2486,1789, 453, # 2480 - 359,2291,1676, 73,1164,1461,1127,3311, 421, 604, 314,1037, 589, 116,2487, 737, # 2496 - 837,1180, 111, 244, 735,6142,2261,1861,1362, 986, 523, 418, 581,2666,3822, 103, # 2512 - 855, 503,1414,1867,2488,1091, 657,1597, 979, 605,1316,4641,1021,2443,2078,2001, # 2528 -1209, 96, 587,2166,1032, 260,1072,2153, 173, 94, 226,3244, 819,2006,4642,4114, # 2544 -2203, 231,1744, 782, 97,2667, 786,3387, 887, 391, 442,2219,4326,1425,6143,2694, # 2560 - 633,1544,1202, 483,2015, 592,2052,1958,2472,1655, 419, 129,4327,3444,3312,1714, # 2576 -1257,3078,4328,1518,1098, 865,1310,1019,1885,1512,1734, 469,2444, 148, 773, 436, # 2592 -1815,1868,1128,1055,4329,1245,2756,3445,2154,1934,1039,4643, 579,1238, 932,2320, # 2608 - 353, 205, 801, 115,2428, 944,2321,1881, 399,2565,1211, 678, 766,3944, 335,2101, # 2624 -1459,1781,1402,3945,2737,2131,1010, 844, 981,1326,1013, 550,1816,1545,2620,1335, # 2640 -1008, 371,2881, 936,1419,1613,3529,1456,1395,2273,1834,2604,1317,2738,2503, 416, # 2656 -1643,4330, 806,1126, 229, 591,3946,1314,1981,1576,1837,1666, 347,1790, 977,3313, # 2672 - 764,2861,1853, 688,2429,1920,1462, 77, 595, 415,2002,3034, 798,1192,4115,6144, # 2688 -2978,4331,3035,2695,2582,2072,2566, 430,2430,1727, 842,1396,3947,3702, 613, 377, # 2704 - 278, 236,1417,3388,3314,3174, 757,1869, 107,3530,6145,1194, 623,2262, 207,1253, # 2720 -2167,3446,3948, 492,1117,1935, 536,1838,2757,1246,4332, 696,2095,2406,1393,1572, # 2736 -3175,1782, 583, 190, 253,1390,2230, 830,3126,3389, 934,3245,1703,1749,2979,1870, # 2752 -2545,1656,2204, 869,2346,4116,3176,1817, 496,1764,4644, 942,1504, 404,1903,1122, # 2768 -1580,3606,2945,1022, 515, 372,1735, 955,2431,3036,6146,2797,1110,2302,2798, 617, # 2784 -6147, 441, 762,1771,3447,3607,3608,1904, 840,3037, 86, 939,1385, 572,1370,2445, # 2800 -1336, 114,3703, 898, 294, 203,3315, 703,1583,2274, 429, 961,4333,1854,1951,3390, # 2816 -2373,3704,4334,1318,1381, 966,1911,2322,1006,1155, 309, 989, 458,2718,1795,1372, # 2832 -1203, 252,1689,1363,3177, 517,1936, 168,1490, 562, 193,3823,1042,4117,1835, 551, # 2848 - 470,4645, 395, 489,3448,1871,1465,2583,2641, 417,1493, 279,1295, 511,1236,1119, # 2864 - 72,1231,1982,1812,3004, 871,1564, 984,3449,1667,2696,2096,4646,2347,2833,1673, # 2880 -3609, 695,3246,2668, 807,1183,4647, 890, 388,2333,1801,1457,2911,1765,1477,1031, # 2896 -3316,3317,1278,3391,2799,2292,2526, 163,3450,4335,2669,1404,1802,6148,2323,2407, # 2912 -1584,1728,1494,1824,1269, 298, 909,3318,1034,1632, 375, 776,1683,2061, 291, 210, # 2928 -1123, 809,1249,1002,2642,3038, 206,1011,2132, 144, 975, 882,1565, 342, 667, 754, # 2944 -1442,2143,1299,2303,2062, 447, 626,2205,1221,2739,2912,1144,1214,2206,2584, 760, # 2960 -1715, 614, 950,1281,2670,2621, 810, 577,1287,2546,4648, 242,2168, 250,2643, 691, # 2976 - 123,2644, 647, 313,1029, 689,1357,2946,1650, 216, 771,1339,1306, 808,2063, 549, # 2992 - 913,1371,2913,2914,6149,1466,1092,1174,1196,1311,2605,2396,1783,1796,3079, 406, # 3008 -2671,2117,3949,4649, 487,1825,2220,6150,2915, 448,2348,1073,6151,2397,1707, 130, # 3024 - 900,1598, 329, 176,1959,2527,1620,6152,2275,4336,3319,1983,2191,3705,3610,2155, # 3040 -3706,1912,1513,1614,6153,1988, 646, 392,2304,1589,3320,3039,1826,1239,1352,1340, # 3056 -2916, 505,2567,1709,1437,2408,2547, 906,6154,2672, 384,1458,1594,1100,1329, 710, # 3072 - 423,3531,2064,2231,2622,1989,2673,1087,1882, 333, 841,3005,1296,2882,2379, 580, # 3088 -1937,1827,1293,2585, 601, 574, 249,1772,4118,2079,1120, 645, 901,1176,1690, 795, # 3104 -2207, 478,1434, 516,1190,1530, 761,2080, 930,1264, 355, 435,1552, 644,1791, 987, # 3120 - 220,1364,1163,1121,1538, 306,2169,1327,1222, 546,2645, 218, 241, 610,1704,3321, # 3136 -1984,1839,1966,2528, 451,6155,2586,3707,2568, 907,3178, 254,2947, 186,1845,4650, # 3152 - 745, 432,1757, 428,1633, 888,2246,2221,2489,3611,2118,1258,1265, 956,3127,1784, # 3168 -4337,2490, 319, 510, 119, 457,3612, 274,2035,2007,4651,1409,3128, 970,2758, 590, # 3184 -2800, 661,2247,4652,2008,3950,1420,1549,3080,3322,3951,1651,1375,2111, 485,2491, # 3200 -1429,1156,6156,2548,2183,1495, 831,1840,2529,2446, 501,1657, 307,1894,3247,1341, # 3216 - 666, 899,2156,1539,2549,1559, 886, 349,2208,3081,2305,1736,3824,2170,2759,1014, # 3232 -1913,1386, 542,1397,2948, 490, 368, 716, 362, 159, 282,2569,1129,1658,1288,1750, # 3248 -2674, 276, 649,2016, 751,1496, 658,1818,1284,1862,2209,2087,2512,3451, 622,2834, # 3264 - 376, 117,1060,2053,1208,1721,1101,1443, 247,1250,3179,1792,3952,2760,2398,3953, # 3280 -6157,2144,3708, 446,2432,1151,2570,3452,2447,2761,2835,1210,2448,3082, 424,2222, # 3296 -1251,2449,2119,2836, 504,1581,4338, 602, 817, 857,3825,2349,2306, 357,3826,1470, # 3312 -1883,2883, 255, 958, 929,2917,3248, 302,4653,1050,1271,1751,2307,1952,1430,2697, # 3328 -2719,2359, 354,3180, 777, 158,2036,4339,1659,4340,4654,2308,2949,2248,1146,2232, # 3344 -3532,2720,1696,2623,3827,6158,3129,1550,2698,1485,1297,1428, 637, 931,2721,2145, # 3360 - 914,2550,2587, 81,2450, 612, 827,2646,1242,4655,1118,2884, 472,1855,3181,3533, # 3376 -3534, 569,1353,2699,1244,1758,2588,4119,2009,2762,2171,3709,1312,1531,6159,1152, # 3392 -1938, 134,1830, 471,3710,2276,1112,1535,3323,3453,3535, 982,1337,2950, 488, 826, # 3408 - 674,1058,1628,4120,2017, 522,2399, 211, 568,1367,3454, 350, 293,1872,1139,3249, # 3424 -1399,1946,3006,1300,2360,3324, 588, 736,6160,2606, 744, 669,3536,3828,6161,1358, # 3440 - 199, 723, 848, 933, 851,1939,1505,1514,1338,1618,1831,4656,1634,3613, 443,2740, # 3456 -3829, 717,1947, 491,1914,6162,2551,1542,4121,1025,6163,1099,1223, 198,3040,2722, # 3472 - 370, 410,1905,2589, 998,1248,3182,2380, 519,1449,4122,1710, 947, 928,1153,4341, # 3488 -2277, 344,2624,1511, 615, 105, 161,1212,1076,1960,3130,2054,1926,1175,1906,2473, # 3504 - 414,1873,2801,6164,2309, 315,1319,3325, 318,2018,2146,2157, 963, 631, 223,4342, # 3520 -4343,2675, 479,3711,1197,2625,3712,2676,2361,6165,4344,4123,6166,2451,3183,1886, # 3536 -2184,1674,1330,1711,1635,1506, 799, 219,3250,3083,3954,1677,3713,3326,2081,3614, # 3552 -1652,2073,4657,1147,3041,1752, 643,1961, 147,1974,3955,6167,1716,2037, 918,3007, # 3568 -1994, 120,1537, 118, 609,3184,4345, 740,3455,1219, 332,1615,3830,6168,1621,2980, # 3584 -1582, 783, 212, 553,2350,3714,1349,2433,2082,4124, 889,6169,2310,1275,1410, 973, # 3600 - 166,1320,3456,1797,1215,3185,2885,1846,2590,2763,4658, 629, 822,3008, 763, 940, # 3616 -1990,2862, 439,2409,1566,1240,1622, 926,1282,1907,2764, 654,2210,1607, 327,1130, # 3632 -3956,1678,1623,6170,2434,2192, 686, 608,3831,3715, 903,3957,3042,6171,2741,1522, # 3648 -1915,1105,1555,2552,1359, 323,3251,4346,3457, 738,1354,2553,2311,2334,1828,2003, # 3664 -3832,1753,2351,1227,6172,1887,4125,1478,6173,2410,1874,1712,1847, 520,1204,2607, # 3680 - 264,4659, 836,2677,2102, 600,4660,3833,2278,3084,6174,4347,3615,1342, 640, 532, # 3696 - 543,2608,1888,2400,2591,1009,4348,1497, 341,1737,3616,2723,1394, 529,3252,1321, # 3712 - 983,4661,1515,2120, 971,2592, 924, 287,1662,3186,4349,2700,4350,1519, 908,1948, # 3728 -2452, 156, 796,1629,1486,2223,2055, 694,4126,1259,1036,3392,1213,2249,2742,1889, # 3744 -1230,3958,1015, 910, 408, 559,3617,4662, 746, 725, 935,4663,3959,3009,1289, 563, # 3760 - 867,4664,3960,1567,2981,2038,2626, 988,2263,2381,4351, 143,2374, 704,1895,6175, # 3776 -1188,3716,2088, 673,3085,2362,4352, 484,1608,1921,2765,2918, 215, 904,3618,3537, # 3792 - 894, 509, 976,3043,2701,3961,4353,2837,2982, 498,6176,6177,1102,3538,1332,3393, # 3808 -1487,1636,1637, 233, 245,3962, 383, 650, 995,3044, 460,1520,1206,2352, 749,3327, # 3824 - 530, 700, 389,1438,1560,1773,3963,2264, 719,2951,2724,3834, 870,1832,1644,1000, # 3840 - 839,2474,3717, 197,1630,3394, 365,2886,3964,1285,2133, 734, 922, 818,1106, 732, # 3856 - 480,2083,1774,3458, 923,2279,1350, 221,3086, 85,2233,2234,3835,1585,3010,2147, # 3872 -1387,1705,2382,1619,2475, 133, 239,2802,1991,1016,2084,2383, 411,2838,1113, 651, # 3888 -1985,1160,3328, 990,1863,3087,1048,1276,2647, 265,2627,1599,3253,2056, 150, 638, # 3904 -2019, 656, 853, 326,1479, 680,1439,4354,1001,1759, 413,3459,3395,2492,1431, 459, # 3920 -4355,1125,3329,2265,1953,1450,2065,2863, 849, 351,2678,3131,3254,3255,1104,1577, # 3936 - 227,1351,1645,2453,2193,1421,2887, 812,2121, 634, 95,2435, 201,2312,4665,1646, # 3952 -1671,2743,1601,2554,2702,2648,2280,1315,1366,2089,3132,1573,3718,3965,1729,1189, # 3968 - 328,2679,1077,1940,1136, 558,1283, 964,1195, 621,2074,1199,1743,3460,3619,1896, # 3984 -1916,1890,3836,2952,1154,2112,1064, 862, 378,3011,2066,2113,2803,1568,2839,6178, # 4000 -3088,2919,1941,1660,2004,1992,2194, 142, 707,1590,1708,1624,1922,1023,1836,1233, # 4016 -1004,2313, 789, 741,3620,6179,1609,2411,1200,4127,3719,3720,4666,2057,3721, 593, # 4032 -2840, 367,2920,1878,6180,3461,1521, 628,1168, 692,2211,2649, 300, 720,2067,2571, # 4048 -2953,3396, 959,2504,3966,3539,3462,1977, 701,6181, 954,1043, 800, 681, 183,3722, # 4064 -1803,1730,3540,4128,2103, 815,2314, 174, 467, 230,2454,1093,2134, 755,3541,3397, # 4080 -1141,1162,6182,1738,2039, 270,3256,2513,1005,1647,2185,3837, 858,1679,1897,1719, # 4096 -2954,2324,1806, 402, 670, 167,4129,1498,2158,2104, 750,6183, 915, 189,1680,1551, # 4112 - 455,4356,1501,2455, 405,1095,2955, 338,1586,1266,1819, 570, 641,1324, 237,1556, # 4128 -2650,1388,3723,6184,1368,2384,1343,1978,3089,2436, 879,3724, 792,1191, 758,3012, # 4144 -1411,2135,1322,4357, 240,4667,1848,3725,1574,6185, 420,3045,1546,1391, 714,4358, # 4160 -1967, 941,1864, 863, 664, 426, 560,1731,2680,1785,2864,1949,2363, 403,3330,1415, # 4176 -1279,2136,1697,2335, 204, 721,2097,3838, 90,6186,2085,2505, 191,3967, 124,2148, # 4192 -1376,1798,1178,1107,1898,1405, 860,4359,1243,1272,2375,2983,1558,2456,1638, 113, # 4208 -3621, 578,1923,2609, 880, 386,4130, 784,2186,2266,1422,2956,2172,1722, 497, 263, # 4224 -2514,1267,2412,2610, 177,2703,3542, 774,1927,1344, 616,1432,1595,1018, 172,4360, # 4240 -2325, 911,4361, 438,1468,3622, 794,3968,2024,2173,1681,1829,2957, 945, 895,3090, # 4256 - 575,2212,2476, 475,2401,2681, 785,2744,1745,2293,2555,1975,3133,2865, 394,4668, # 4272 -3839, 635,4131, 639, 202,1507,2195,2766,1345,1435,2572,3726,1908,1184,1181,2457, # 4288 -3727,3134,4362, 843,2611, 437, 916,4669, 234, 769,1884,3046,3047,3623, 833,6187, # 4304 -1639,2250,2402,1355,1185,2010,2047, 999, 525,1732,1290,1488,2612, 948,1578,3728, # 4320 -2413,2477,1216,2725,2159, 334,3840,1328,3624,2921,1525,4132, 564,1056, 891,4363, # 4336 -1444,1698,2385,2251,3729,1365,2281,2235,1717,6188, 864,3841,2515, 444, 527,2767, # 4352 -2922,3625, 544, 461,6189, 566, 209,2437,3398,2098,1065,2068,3331,3626,3257,2137, # 4368 #last 512 -) -# fmt: on diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/johabfreq.py b/venv/lib/python3.11/site-packages/pip/_vendor/chardet/johabfreq.py deleted file mode 100644 index c129699..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/johabfreq.py +++ /dev/null @@ -1,2382 +0,0 @@ -######################## BEGIN LICENSE BLOCK ######################## -# The Original Code is Mozilla Communicator client code. -# -# The Initial Developer of the Original Code is -# Netscape Communications Corporation. -# Portions created by the Initial Developer are Copyright (C) 1998 -# the Initial Developer. All Rights Reserved. -# -# Contributor(s): -# Mark Pilgrim - port to Python -# -# This library is free software; you can redistribute it and/or -# modify it under the terms of the GNU Lesser General Public -# License as published by the Free Software Foundation; either -# version 2.1 of the License, or (at your option) any later version. -# -# This library is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# Lesser General Public License for more details. -# -# You should have received a copy of the GNU Lesser General Public -# License along with this library; if not, write to the Free Software -# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA -# 02110-1301 USA -######################### END LICENSE BLOCK ######################### - -# The frequency data itself is the same as euc-kr. -# This is just a mapping table to euc-kr. - -JOHAB_TO_EUCKR_ORDER_TABLE = { - 0x8861: 0, - 0x8862: 1, - 0x8865: 2, - 0x8868: 3, - 0x8869: 4, - 0x886A: 5, - 0x886B: 6, - 0x8871: 7, - 0x8873: 8, - 0x8874: 9, - 0x8875: 10, - 0x8876: 11, - 0x8877: 12, - 0x8878: 13, - 0x8879: 14, - 0x887B: 15, - 0x887C: 16, - 0x887D: 17, - 0x8881: 18, - 0x8882: 19, - 0x8885: 20, - 0x8889: 21, - 0x8891: 22, - 0x8893: 23, - 0x8895: 24, - 0x8896: 25, - 0x8897: 26, - 0x88A1: 27, - 0x88A2: 28, - 0x88A5: 29, - 0x88A9: 30, - 0x88B5: 31, - 0x88B7: 32, - 0x88C1: 33, - 0x88C5: 34, - 0x88C9: 35, - 0x88E1: 36, - 0x88E2: 37, - 0x88E5: 38, - 0x88E8: 39, - 0x88E9: 40, - 0x88EB: 41, - 0x88F1: 42, - 0x88F3: 43, - 0x88F5: 44, - 0x88F6: 45, - 0x88F7: 46, - 0x88F8: 47, - 0x88FB: 48, - 0x88FC: 49, - 0x88FD: 50, - 0x8941: 51, - 0x8945: 52, - 0x8949: 53, - 0x8951: 54, - 0x8953: 55, - 0x8955: 56, - 0x8956: 57, - 0x8957: 58, - 0x8961: 59, - 0x8962: 60, - 0x8963: 61, - 0x8965: 62, - 0x8968: 63, - 0x8969: 64, - 0x8971: 65, - 0x8973: 66, - 0x8975: 67, - 0x8976: 68, - 0x8977: 69, - 0x897B: 70, - 0x8981: 71, - 0x8985: 72, - 0x8989: 73, - 0x8993: 74, - 0x8995: 75, - 0x89A1: 76, - 0x89A2: 77, - 0x89A5: 78, - 0x89A8: 79, - 0x89A9: 80, - 0x89AB: 81, - 0x89AD: 82, - 0x89B0: 83, - 0x89B1: 84, - 0x89B3: 85, - 0x89B5: 86, - 0x89B7: 87, - 0x89B8: 88, - 0x89C1: 89, - 0x89C2: 90, - 0x89C5: 91, - 0x89C9: 92, - 0x89CB: 93, - 0x89D1: 94, - 0x89D3: 95, - 0x89D5: 96, - 0x89D7: 97, - 0x89E1: 98, - 0x89E5: 99, - 0x89E9: 100, - 0x89F3: 101, - 0x89F6: 102, - 0x89F7: 103, - 0x8A41: 104, - 0x8A42: 105, - 0x8A45: 106, - 0x8A49: 107, - 0x8A51: 108, - 0x8A53: 109, - 0x8A55: 110, - 0x8A57: 111, - 0x8A61: 112, - 0x8A65: 113, - 0x8A69: 114, - 0x8A73: 115, - 0x8A75: 116, - 0x8A81: 117, - 0x8A82: 118, - 0x8A85: 119, - 0x8A88: 120, - 0x8A89: 121, - 0x8A8A: 122, - 0x8A8B: 123, - 0x8A90: 124, - 0x8A91: 125, - 0x8A93: 126, - 0x8A95: 127, - 0x8A97: 128, - 0x8A98: 129, - 0x8AA1: 130, - 0x8AA2: 131, - 0x8AA5: 132, - 0x8AA9: 133, - 0x8AB6: 134, - 0x8AB7: 135, - 0x8AC1: 136, - 0x8AD5: 137, - 0x8AE1: 138, - 0x8AE2: 139, - 0x8AE5: 140, - 0x8AE9: 141, - 0x8AF1: 142, - 0x8AF3: 143, - 0x8AF5: 144, - 0x8B41: 145, - 0x8B45: 146, - 0x8B49: 147, - 0x8B61: 148, - 0x8B62: 149, - 0x8B65: 150, - 0x8B68: 151, - 0x8B69: 152, - 0x8B6A: 153, - 0x8B71: 154, - 0x8B73: 155, - 0x8B75: 156, - 0x8B77: 157, - 0x8B81: 158, - 0x8BA1: 159, - 0x8BA2: 160, - 0x8BA5: 161, - 0x8BA8: 162, - 0x8BA9: 163, - 0x8BAB: 164, - 0x8BB1: 165, - 0x8BB3: 166, - 0x8BB5: 167, - 0x8BB7: 168, - 0x8BB8: 169, - 0x8BBC: 170, - 0x8C61: 171, - 0x8C62: 172, - 0x8C63: 173, - 0x8C65: 174, - 0x8C69: 175, - 0x8C6B: 176, - 0x8C71: 177, - 0x8C73: 178, - 0x8C75: 179, - 0x8C76: 180, - 0x8C77: 181, - 0x8C7B: 182, - 0x8C81: 183, - 0x8C82: 184, - 0x8C85: 185, - 0x8C89: 186, - 0x8C91: 187, - 0x8C93: 188, - 0x8C95: 189, - 0x8C96: 190, - 0x8C97: 191, - 0x8CA1: 192, - 0x8CA2: 193, - 0x8CA9: 194, - 0x8CE1: 195, - 0x8CE2: 196, - 0x8CE3: 197, - 0x8CE5: 198, - 0x8CE9: 199, - 0x8CF1: 200, - 0x8CF3: 201, - 0x8CF5: 202, - 0x8CF6: 203, - 0x8CF7: 204, - 0x8D41: 205, - 0x8D42: 206, - 0x8D45: 207, - 0x8D51: 208, - 0x8D55: 209, - 0x8D57: 210, - 0x8D61: 211, - 0x8D65: 212, - 0x8D69: 213, - 0x8D75: 214, - 0x8D76: 215, - 0x8D7B: 216, - 0x8D81: 217, - 0x8DA1: 218, - 0x8DA2: 219, - 0x8DA5: 220, - 0x8DA7: 221, - 0x8DA9: 222, - 0x8DB1: 223, - 0x8DB3: 224, - 0x8DB5: 225, - 0x8DB7: 226, - 0x8DB8: 227, - 0x8DB9: 228, - 0x8DC1: 229, - 0x8DC2: 230, - 0x8DC9: 231, - 0x8DD6: 232, - 0x8DD7: 233, - 0x8DE1: 234, - 0x8DE2: 235, - 0x8DF7: 236, - 0x8E41: 237, - 0x8E45: 238, - 0x8E49: 239, - 0x8E51: 240, - 0x8E53: 241, - 0x8E57: 242, - 0x8E61: 243, - 0x8E81: 244, - 0x8E82: 245, - 0x8E85: 246, - 0x8E89: 247, - 0x8E90: 248, - 0x8E91: 249, - 0x8E93: 250, - 0x8E95: 251, - 0x8E97: 252, - 0x8E98: 253, - 0x8EA1: 254, - 0x8EA9: 255, - 0x8EB6: 256, - 0x8EB7: 257, - 0x8EC1: 258, - 0x8EC2: 259, - 0x8EC5: 260, - 0x8EC9: 261, - 0x8ED1: 262, - 0x8ED3: 263, - 0x8ED6: 264, - 0x8EE1: 265, - 0x8EE5: 266, - 0x8EE9: 267, - 0x8EF1: 268, - 0x8EF3: 269, - 0x8F41: 270, - 0x8F61: 271, - 0x8F62: 272, - 0x8F65: 273, - 0x8F67: 274, - 0x8F69: 275, - 0x8F6B: 276, - 0x8F70: 277, - 0x8F71: 278, - 0x8F73: 279, - 0x8F75: 280, - 0x8F77: 281, - 0x8F7B: 282, - 0x8FA1: 283, - 0x8FA2: 284, - 0x8FA5: 285, - 0x8FA9: 286, - 0x8FB1: 287, - 0x8FB3: 288, - 0x8FB5: 289, - 0x8FB7: 290, - 0x9061: 291, - 0x9062: 292, - 0x9063: 293, - 0x9065: 294, - 0x9068: 295, - 0x9069: 296, - 0x906A: 297, - 0x906B: 298, - 0x9071: 299, - 0x9073: 300, - 0x9075: 301, - 0x9076: 302, - 0x9077: 303, - 0x9078: 304, - 0x9079: 305, - 0x907B: 306, - 0x907D: 307, - 0x9081: 308, - 0x9082: 309, - 0x9085: 310, - 0x9089: 311, - 0x9091: 312, - 0x9093: 313, - 0x9095: 314, - 0x9096: 315, - 0x9097: 316, - 0x90A1: 317, - 0x90A2: 318, - 0x90A5: 319, - 0x90A9: 320, - 0x90B1: 321, - 0x90B7: 322, - 0x90E1: 323, - 0x90E2: 324, - 0x90E4: 325, - 0x90E5: 326, - 0x90E9: 327, - 0x90EB: 328, - 0x90EC: 329, - 0x90F1: 330, - 0x90F3: 331, - 0x90F5: 332, - 0x90F6: 333, - 0x90F7: 334, - 0x90FD: 335, - 0x9141: 336, - 0x9142: 337, - 0x9145: 338, - 0x9149: 339, - 0x9151: 340, - 0x9153: 341, - 0x9155: 342, - 0x9156: 343, - 0x9157: 344, - 0x9161: 345, - 0x9162: 346, - 0x9165: 347, - 0x9169: 348, - 0x9171: 349, - 0x9173: 350, - 0x9176: 351, - 0x9177: 352, - 0x917A: 353, - 0x9181: 354, - 0x9185: 355, - 0x91A1: 356, - 0x91A2: 357, - 0x91A5: 358, - 0x91A9: 359, - 0x91AB: 360, - 0x91B1: 361, - 0x91B3: 362, - 0x91B5: 363, - 0x91B7: 364, - 0x91BC: 365, - 0x91BD: 366, - 0x91C1: 367, - 0x91C5: 368, - 0x91C9: 369, - 0x91D6: 370, - 0x9241: 371, - 0x9245: 372, - 0x9249: 373, - 0x9251: 374, - 0x9253: 375, - 0x9255: 376, - 0x9261: 377, - 0x9262: 378, - 0x9265: 379, - 0x9269: 380, - 0x9273: 381, - 0x9275: 382, - 0x9277: 383, - 0x9281: 384, - 0x9282: 385, - 0x9285: 386, - 0x9288: 387, - 0x9289: 388, - 0x9291: 389, - 0x9293: 390, - 0x9295: 391, - 0x9297: 392, - 0x92A1: 393, - 0x92B6: 394, - 0x92C1: 395, - 0x92E1: 396, - 0x92E5: 397, - 0x92E9: 398, - 0x92F1: 399, - 0x92F3: 400, - 0x9341: 401, - 0x9342: 402, - 0x9349: 403, - 0x9351: 404, - 0x9353: 405, - 0x9357: 406, - 0x9361: 407, - 0x9362: 408, - 0x9365: 409, - 0x9369: 410, - 0x936A: 411, - 0x936B: 412, - 0x9371: 413, - 0x9373: 414, - 0x9375: 415, - 0x9377: 416, - 0x9378: 417, - 0x937C: 418, - 0x9381: 419, - 0x9385: 420, - 0x9389: 421, - 0x93A1: 422, - 0x93A2: 423, - 0x93A5: 424, - 0x93A9: 425, - 0x93AB: 426, - 0x93B1: 427, - 0x93B3: 428, - 0x93B5: 429, - 0x93B7: 430, - 0x93BC: 431, - 0x9461: 432, - 0x9462: 433, - 0x9463: 434, - 0x9465: 435, - 0x9468: 436, - 0x9469: 437, - 0x946A: 438, - 0x946B: 439, - 0x946C: 440, - 0x9470: 441, - 0x9471: 442, - 0x9473: 443, - 0x9475: 444, - 0x9476: 445, - 0x9477: 446, - 0x9478: 447, - 0x9479: 448, - 0x947D: 449, - 0x9481: 450, - 0x9482: 451, - 0x9485: 452, - 0x9489: 453, - 0x9491: 454, - 0x9493: 455, - 0x9495: 456, - 0x9496: 457, - 0x9497: 458, - 0x94A1: 459, - 0x94E1: 460, - 0x94E2: 461, - 0x94E3: 462, - 0x94E5: 463, - 0x94E8: 464, - 0x94E9: 465, - 0x94EB: 466, - 0x94EC: 467, - 0x94F1: 468, - 0x94F3: 469, - 0x94F5: 470, - 0x94F7: 471, - 0x94F9: 472, - 0x94FC: 473, - 0x9541: 474, - 0x9542: 475, - 0x9545: 476, - 0x9549: 477, - 0x9551: 478, - 0x9553: 479, - 0x9555: 480, - 0x9556: 481, - 0x9557: 482, - 0x9561: 483, - 0x9565: 484, - 0x9569: 485, - 0x9576: 486, - 0x9577: 487, - 0x9581: 488, - 0x9585: 489, - 0x95A1: 490, - 0x95A2: 491, - 0x95A5: 492, - 0x95A8: 493, - 0x95A9: 494, - 0x95AB: 495, - 0x95AD: 496, - 0x95B1: 497, - 0x95B3: 498, - 0x95B5: 499, - 0x95B7: 500, - 0x95B9: 501, - 0x95BB: 502, - 0x95C1: 503, - 0x95C5: 504, - 0x95C9: 505, - 0x95E1: 506, - 0x95F6: 507, - 0x9641: 508, - 0x9645: 509, - 0x9649: 510, - 0x9651: 511, - 0x9653: 512, - 0x9655: 513, - 0x9661: 514, - 0x9681: 515, - 0x9682: 516, - 0x9685: 517, - 0x9689: 518, - 0x9691: 519, - 0x9693: 520, - 0x9695: 521, - 0x9697: 522, - 0x96A1: 523, - 0x96B6: 524, - 0x96C1: 525, - 0x96D7: 526, - 0x96E1: 527, - 0x96E5: 528, - 0x96E9: 529, - 0x96F3: 530, - 0x96F5: 531, - 0x96F7: 532, - 0x9741: 533, - 0x9745: 534, - 0x9749: 535, - 0x9751: 536, - 0x9757: 537, - 0x9761: 538, - 0x9762: 539, - 0x9765: 540, - 0x9768: 541, - 0x9769: 542, - 0x976B: 543, - 0x9771: 544, - 0x9773: 545, - 0x9775: 546, - 0x9777: 547, - 0x9781: 548, - 0x97A1: 549, - 0x97A2: 550, - 0x97A5: 551, - 0x97A8: 552, - 0x97A9: 553, - 0x97B1: 554, - 0x97B3: 555, - 0x97B5: 556, - 0x97B6: 557, - 0x97B7: 558, - 0x97B8: 559, - 0x9861: 560, - 0x9862: 561, - 0x9865: 562, - 0x9869: 563, - 0x9871: 564, - 0x9873: 565, - 0x9875: 566, - 0x9876: 567, - 0x9877: 568, - 0x987D: 569, - 0x9881: 570, - 0x9882: 571, - 0x9885: 572, - 0x9889: 573, - 0x9891: 574, - 0x9893: 575, - 0x9895: 576, - 0x9896: 577, - 0x9897: 578, - 0x98E1: 579, - 0x98E2: 580, - 0x98E5: 581, - 0x98E9: 582, - 0x98EB: 583, - 0x98EC: 584, - 0x98F1: 585, - 0x98F3: 586, - 0x98F5: 587, - 0x98F6: 588, - 0x98F7: 589, - 0x98FD: 590, - 0x9941: 591, - 0x9942: 592, - 0x9945: 593, - 0x9949: 594, - 0x9951: 595, - 0x9953: 596, - 0x9955: 597, - 0x9956: 598, - 0x9957: 599, - 0x9961: 600, - 0x9976: 601, - 0x99A1: 602, - 0x99A2: 603, - 0x99A5: 604, - 0x99A9: 605, - 0x99B7: 606, - 0x99C1: 607, - 0x99C9: 608, - 0x99E1: 609, - 0x9A41: 610, - 0x9A45: 611, - 0x9A81: 612, - 0x9A82: 613, - 0x9A85: 614, - 0x9A89: 615, - 0x9A90: 616, - 0x9A91: 617, - 0x9A97: 618, - 0x9AC1: 619, - 0x9AE1: 620, - 0x9AE5: 621, - 0x9AE9: 622, - 0x9AF1: 623, - 0x9AF3: 624, - 0x9AF7: 625, - 0x9B61: 626, - 0x9B62: 627, - 0x9B65: 628, - 0x9B68: 629, - 0x9B69: 630, - 0x9B71: 631, - 0x9B73: 632, - 0x9B75: 633, - 0x9B81: 634, - 0x9B85: 635, - 0x9B89: 636, - 0x9B91: 637, - 0x9B93: 638, - 0x9BA1: 639, - 0x9BA5: 640, - 0x9BA9: 641, - 0x9BB1: 642, - 0x9BB3: 643, - 0x9BB5: 644, - 0x9BB7: 645, - 0x9C61: 646, - 0x9C62: 647, - 0x9C65: 648, - 0x9C69: 649, - 0x9C71: 650, - 0x9C73: 651, - 0x9C75: 652, - 0x9C76: 653, - 0x9C77: 654, - 0x9C78: 655, - 0x9C7C: 656, - 0x9C7D: 657, - 0x9C81: 658, - 0x9C82: 659, - 0x9C85: 660, - 0x9C89: 661, - 0x9C91: 662, - 0x9C93: 663, - 0x9C95: 664, - 0x9C96: 665, - 0x9C97: 666, - 0x9CA1: 667, - 0x9CA2: 668, - 0x9CA5: 669, - 0x9CB5: 670, - 0x9CB7: 671, - 0x9CE1: 672, - 0x9CE2: 673, - 0x9CE5: 674, - 0x9CE9: 675, - 0x9CF1: 676, - 0x9CF3: 677, - 0x9CF5: 678, - 0x9CF6: 679, - 0x9CF7: 680, - 0x9CFD: 681, - 0x9D41: 682, - 0x9D42: 683, - 0x9D45: 684, - 0x9D49: 685, - 0x9D51: 686, - 0x9D53: 687, - 0x9D55: 688, - 0x9D57: 689, - 0x9D61: 690, - 0x9D62: 691, - 0x9D65: 692, - 0x9D69: 693, - 0x9D71: 694, - 0x9D73: 695, - 0x9D75: 696, - 0x9D76: 697, - 0x9D77: 698, - 0x9D81: 699, - 0x9D85: 700, - 0x9D93: 701, - 0x9D95: 702, - 0x9DA1: 703, - 0x9DA2: 704, - 0x9DA5: 705, - 0x9DA9: 706, - 0x9DB1: 707, - 0x9DB3: 708, - 0x9DB5: 709, - 0x9DB7: 710, - 0x9DC1: 711, - 0x9DC5: 712, - 0x9DD7: 713, - 0x9DF6: 714, - 0x9E41: 715, - 0x9E45: 716, - 0x9E49: 717, - 0x9E51: 718, - 0x9E53: 719, - 0x9E55: 720, - 0x9E57: 721, - 0x9E61: 722, - 0x9E65: 723, - 0x9E69: 724, - 0x9E73: 725, - 0x9E75: 726, - 0x9E77: 727, - 0x9E81: 728, - 0x9E82: 729, - 0x9E85: 730, - 0x9E89: 731, - 0x9E91: 732, - 0x9E93: 733, - 0x9E95: 734, - 0x9E97: 735, - 0x9EA1: 736, - 0x9EB6: 737, - 0x9EC1: 738, - 0x9EE1: 739, - 0x9EE2: 740, - 0x9EE5: 741, - 0x9EE9: 742, - 0x9EF1: 743, - 0x9EF5: 744, - 0x9EF7: 745, - 0x9F41: 746, - 0x9F42: 747, - 0x9F45: 748, - 0x9F49: 749, - 0x9F51: 750, - 0x9F53: 751, - 0x9F55: 752, - 0x9F57: 753, - 0x9F61: 754, - 0x9F62: 755, - 0x9F65: 756, - 0x9F69: 757, - 0x9F71: 758, - 0x9F73: 759, - 0x9F75: 760, - 0x9F77: 761, - 0x9F78: 762, - 0x9F7B: 763, - 0x9F7C: 764, - 0x9FA1: 765, - 0x9FA2: 766, - 0x9FA5: 767, - 0x9FA9: 768, - 0x9FB1: 769, - 0x9FB3: 770, - 0x9FB5: 771, - 0x9FB7: 772, - 0xA061: 773, - 0xA062: 774, - 0xA065: 775, - 0xA067: 776, - 0xA068: 777, - 0xA069: 778, - 0xA06A: 779, - 0xA06B: 780, - 0xA071: 781, - 0xA073: 782, - 0xA075: 783, - 0xA077: 784, - 0xA078: 785, - 0xA07B: 786, - 0xA07D: 787, - 0xA081: 788, - 0xA082: 789, - 0xA085: 790, - 0xA089: 791, - 0xA091: 792, - 0xA093: 793, - 0xA095: 794, - 0xA096: 795, - 0xA097: 796, - 0xA098: 797, - 0xA0A1: 798, - 0xA0A2: 799, - 0xA0A9: 800, - 0xA0B7: 801, - 0xA0E1: 802, - 0xA0E2: 803, - 0xA0E5: 804, - 0xA0E9: 805, - 0xA0EB: 806, - 0xA0F1: 807, - 0xA0F3: 808, - 0xA0F5: 809, - 0xA0F7: 810, - 0xA0F8: 811, - 0xA0FD: 812, - 0xA141: 813, - 0xA142: 814, - 0xA145: 815, - 0xA149: 816, - 0xA151: 817, - 0xA153: 818, - 0xA155: 819, - 0xA156: 820, - 0xA157: 821, - 0xA161: 822, - 0xA162: 823, - 0xA165: 824, - 0xA169: 825, - 0xA175: 826, - 0xA176: 827, - 0xA177: 828, - 0xA179: 829, - 0xA181: 830, - 0xA1A1: 831, - 0xA1A2: 832, - 0xA1A4: 833, - 0xA1A5: 834, - 0xA1A9: 835, - 0xA1AB: 836, - 0xA1B1: 837, - 0xA1B3: 838, - 0xA1B5: 839, - 0xA1B7: 840, - 0xA1C1: 841, - 0xA1C5: 842, - 0xA1D6: 843, - 0xA1D7: 844, - 0xA241: 845, - 0xA245: 846, - 0xA249: 847, - 0xA253: 848, - 0xA255: 849, - 0xA257: 850, - 0xA261: 851, - 0xA265: 852, - 0xA269: 853, - 0xA273: 854, - 0xA275: 855, - 0xA281: 856, - 0xA282: 857, - 0xA283: 858, - 0xA285: 859, - 0xA288: 860, - 0xA289: 861, - 0xA28A: 862, - 0xA28B: 863, - 0xA291: 864, - 0xA293: 865, - 0xA295: 866, - 0xA297: 867, - 0xA29B: 868, - 0xA29D: 869, - 0xA2A1: 870, - 0xA2A5: 871, - 0xA2A9: 872, - 0xA2B3: 873, - 0xA2B5: 874, - 0xA2C1: 875, - 0xA2E1: 876, - 0xA2E5: 877, - 0xA2E9: 878, - 0xA341: 879, - 0xA345: 880, - 0xA349: 881, - 0xA351: 882, - 0xA355: 883, - 0xA361: 884, - 0xA365: 885, - 0xA369: 886, - 0xA371: 887, - 0xA375: 888, - 0xA3A1: 889, - 0xA3A2: 890, - 0xA3A5: 891, - 0xA3A8: 892, - 0xA3A9: 893, - 0xA3AB: 894, - 0xA3B1: 895, - 0xA3B3: 896, - 0xA3B5: 897, - 0xA3B6: 898, - 0xA3B7: 899, - 0xA3B9: 900, - 0xA3BB: 901, - 0xA461: 902, - 0xA462: 903, - 0xA463: 904, - 0xA464: 905, - 0xA465: 906, - 0xA468: 907, - 0xA469: 908, - 0xA46A: 909, - 0xA46B: 910, - 0xA46C: 911, - 0xA471: 912, - 0xA473: 913, - 0xA475: 914, - 0xA477: 915, - 0xA47B: 916, - 0xA481: 917, - 0xA482: 918, - 0xA485: 919, - 0xA489: 920, - 0xA491: 921, - 0xA493: 922, - 0xA495: 923, - 0xA496: 924, - 0xA497: 925, - 0xA49B: 926, - 0xA4A1: 927, - 0xA4A2: 928, - 0xA4A5: 929, - 0xA4B3: 930, - 0xA4E1: 931, - 0xA4E2: 932, - 0xA4E5: 933, - 0xA4E8: 934, - 0xA4E9: 935, - 0xA4EB: 936, - 0xA4F1: 937, - 0xA4F3: 938, - 0xA4F5: 939, - 0xA4F7: 940, - 0xA4F8: 941, - 0xA541: 942, - 0xA542: 943, - 0xA545: 944, - 0xA548: 945, - 0xA549: 946, - 0xA551: 947, - 0xA553: 948, - 0xA555: 949, - 0xA556: 950, - 0xA557: 951, - 0xA561: 952, - 0xA562: 953, - 0xA565: 954, - 0xA569: 955, - 0xA573: 956, - 0xA575: 957, - 0xA576: 958, - 0xA577: 959, - 0xA57B: 960, - 0xA581: 961, - 0xA585: 962, - 0xA5A1: 963, - 0xA5A2: 964, - 0xA5A3: 965, - 0xA5A5: 966, - 0xA5A9: 967, - 0xA5B1: 968, - 0xA5B3: 969, - 0xA5B5: 970, - 0xA5B7: 971, - 0xA5C1: 972, - 0xA5C5: 973, - 0xA5D6: 974, - 0xA5E1: 975, - 0xA5F6: 976, - 0xA641: 977, - 0xA642: 978, - 0xA645: 979, - 0xA649: 980, - 0xA651: 981, - 0xA653: 982, - 0xA661: 983, - 0xA665: 984, - 0xA681: 985, - 0xA682: 986, - 0xA685: 987, - 0xA688: 988, - 0xA689: 989, - 0xA68A: 990, - 0xA68B: 991, - 0xA691: 992, - 0xA693: 993, - 0xA695: 994, - 0xA697: 995, - 0xA69B: 996, - 0xA69C: 997, - 0xA6A1: 998, - 0xA6A9: 999, - 0xA6B6: 1000, - 0xA6C1: 1001, - 0xA6E1: 1002, - 0xA6E2: 1003, - 0xA6E5: 1004, - 0xA6E9: 1005, - 0xA6F7: 1006, - 0xA741: 1007, - 0xA745: 1008, - 0xA749: 1009, - 0xA751: 1010, - 0xA755: 1011, - 0xA757: 1012, - 0xA761: 1013, - 0xA762: 1014, - 0xA765: 1015, - 0xA769: 1016, - 0xA771: 1017, - 0xA773: 1018, - 0xA775: 1019, - 0xA7A1: 1020, - 0xA7A2: 1021, - 0xA7A5: 1022, - 0xA7A9: 1023, - 0xA7AB: 1024, - 0xA7B1: 1025, - 0xA7B3: 1026, - 0xA7B5: 1027, - 0xA7B7: 1028, - 0xA7B8: 1029, - 0xA7B9: 1030, - 0xA861: 1031, - 0xA862: 1032, - 0xA865: 1033, - 0xA869: 1034, - 0xA86B: 1035, - 0xA871: 1036, - 0xA873: 1037, - 0xA875: 1038, - 0xA876: 1039, - 0xA877: 1040, - 0xA87D: 1041, - 0xA881: 1042, - 0xA882: 1043, - 0xA885: 1044, - 0xA889: 1045, - 0xA891: 1046, - 0xA893: 1047, - 0xA895: 1048, - 0xA896: 1049, - 0xA897: 1050, - 0xA8A1: 1051, - 0xA8A2: 1052, - 0xA8B1: 1053, - 0xA8E1: 1054, - 0xA8E2: 1055, - 0xA8E5: 1056, - 0xA8E8: 1057, - 0xA8E9: 1058, - 0xA8F1: 1059, - 0xA8F5: 1060, - 0xA8F6: 1061, - 0xA8F7: 1062, - 0xA941: 1063, - 0xA957: 1064, - 0xA961: 1065, - 0xA962: 1066, - 0xA971: 1067, - 0xA973: 1068, - 0xA975: 1069, - 0xA976: 1070, - 0xA977: 1071, - 0xA9A1: 1072, - 0xA9A2: 1073, - 0xA9A5: 1074, - 0xA9A9: 1075, - 0xA9B1: 1076, - 0xA9B3: 1077, - 0xA9B7: 1078, - 0xAA41: 1079, - 0xAA61: 1080, - 0xAA77: 1081, - 0xAA81: 1082, - 0xAA82: 1083, - 0xAA85: 1084, - 0xAA89: 1085, - 0xAA91: 1086, - 0xAA95: 1087, - 0xAA97: 1088, - 0xAB41: 1089, - 0xAB57: 1090, - 0xAB61: 1091, - 0xAB65: 1092, - 0xAB69: 1093, - 0xAB71: 1094, - 0xAB73: 1095, - 0xABA1: 1096, - 0xABA2: 1097, - 0xABA5: 1098, - 0xABA9: 1099, - 0xABB1: 1100, - 0xABB3: 1101, - 0xABB5: 1102, - 0xABB7: 1103, - 0xAC61: 1104, - 0xAC62: 1105, - 0xAC64: 1106, - 0xAC65: 1107, - 0xAC68: 1108, - 0xAC69: 1109, - 0xAC6A: 1110, - 0xAC6B: 1111, - 0xAC71: 1112, - 0xAC73: 1113, - 0xAC75: 1114, - 0xAC76: 1115, - 0xAC77: 1116, - 0xAC7B: 1117, - 0xAC81: 1118, - 0xAC82: 1119, - 0xAC85: 1120, - 0xAC89: 1121, - 0xAC91: 1122, - 0xAC93: 1123, - 0xAC95: 1124, - 0xAC96: 1125, - 0xAC97: 1126, - 0xACA1: 1127, - 0xACA2: 1128, - 0xACA5: 1129, - 0xACA9: 1130, - 0xACB1: 1131, - 0xACB3: 1132, - 0xACB5: 1133, - 0xACB7: 1134, - 0xACC1: 1135, - 0xACC5: 1136, - 0xACC9: 1137, - 0xACD1: 1138, - 0xACD7: 1139, - 0xACE1: 1140, - 0xACE2: 1141, - 0xACE3: 1142, - 0xACE4: 1143, - 0xACE5: 1144, - 0xACE8: 1145, - 0xACE9: 1146, - 0xACEB: 1147, - 0xACEC: 1148, - 0xACF1: 1149, - 0xACF3: 1150, - 0xACF5: 1151, - 0xACF6: 1152, - 0xACF7: 1153, - 0xACFC: 1154, - 0xAD41: 1155, - 0xAD42: 1156, - 0xAD45: 1157, - 0xAD49: 1158, - 0xAD51: 1159, - 0xAD53: 1160, - 0xAD55: 1161, - 0xAD56: 1162, - 0xAD57: 1163, - 0xAD61: 1164, - 0xAD62: 1165, - 0xAD65: 1166, - 0xAD69: 1167, - 0xAD71: 1168, - 0xAD73: 1169, - 0xAD75: 1170, - 0xAD76: 1171, - 0xAD77: 1172, - 0xAD81: 1173, - 0xAD85: 1174, - 0xAD89: 1175, - 0xAD97: 1176, - 0xADA1: 1177, - 0xADA2: 1178, - 0xADA3: 1179, - 0xADA5: 1180, - 0xADA9: 1181, - 0xADAB: 1182, - 0xADB1: 1183, - 0xADB3: 1184, - 0xADB5: 1185, - 0xADB7: 1186, - 0xADBB: 1187, - 0xADC1: 1188, - 0xADC2: 1189, - 0xADC5: 1190, - 0xADC9: 1191, - 0xADD7: 1192, - 0xADE1: 1193, - 0xADE5: 1194, - 0xADE9: 1195, - 0xADF1: 1196, - 0xADF5: 1197, - 0xADF6: 1198, - 0xAE41: 1199, - 0xAE45: 1200, - 0xAE49: 1201, - 0xAE51: 1202, - 0xAE53: 1203, - 0xAE55: 1204, - 0xAE61: 1205, - 0xAE62: 1206, - 0xAE65: 1207, - 0xAE69: 1208, - 0xAE71: 1209, - 0xAE73: 1210, - 0xAE75: 1211, - 0xAE77: 1212, - 0xAE81: 1213, - 0xAE82: 1214, - 0xAE85: 1215, - 0xAE88: 1216, - 0xAE89: 1217, - 0xAE91: 1218, - 0xAE93: 1219, - 0xAE95: 1220, - 0xAE97: 1221, - 0xAE99: 1222, - 0xAE9B: 1223, - 0xAE9C: 1224, - 0xAEA1: 1225, - 0xAEB6: 1226, - 0xAEC1: 1227, - 0xAEC2: 1228, - 0xAEC5: 1229, - 0xAEC9: 1230, - 0xAED1: 1231, - 0xAED7: 1232, - 0xAEE1: 1233, - 0xAEE2: 1234, - 0xAEE5: 1235, - 0xAEE9: 1236, - 0xAEF1: 1237, - 0xAEF3: 1238, - 0xAEF5: 1239, - 0xAEF7: 1240, - 0xAF41: 1241, - 0xAF42: 1242, - 0xAF49: 1243, - 0xAF51: 1244, - 0xAF55: 1245, - 0xAF57: 1246, - 0xAF61: 1247, - 0xAF62: 1248, - 0xAF65: 1249, - 0xAF69: 1250, - 0xAF6A: 1251, - 0xAF71: 1252, - 0xAF73: 1253, - 0xAF75: 1254, - 0xAF77: 1255, - 0xAFA1: 1256, - 0xAFA2: 1257, - 0xAFA5: 1258, - 0xAFA8: 1259, - 0xAFA9: 1260, - 0xAFB0: 1261, - 0xAFB1: 1262, - 0xAFB3: 1263, - 0xAFB5: 1264, - 0xAFB7: 1265, - 0xAFBC: 1266, - 0xB061: 1267, - 0xB062: 1268, - 0xB064: 1269, - 0xB065: 1270, - 0xB069: 1271, - 0xB071: 1272, - 0xB073: 1273, - 0xB076: 1274, - 0xB077: 1275, - 0xB07D: 1276, - 0xB081: 1277, - 0xB082: 1278, - 0xB085: 1279, - 0xB089: 1280, - 0xB091: 1281, - 0xB093: 1282, - 0xB096: 1283, - 0xB097: 1284, - 0xB0B7: 1285, - 0xB0E1: 1286, - 0xB0E2: 1287, - 0xB0E5: 1288, - 0xB0E9: 1289, - 0xB0EB: 1290, - 0xB0F1: 1291, - 0xB0F3: 1292, - 0xB0F6: 1293, - 0xB0F7: 1294, - 0xB141: 1295, - 0xB145: 1296, - 0xB149: 1297, - 0xB185: 1298, - 0xB1A1: 1299, - 0xB1A2: 1300, - 0xB1A5: 1301, - 0xB1A8: 1302, - 0xB1A9: 1303, - 0xB1AB: 1304, - 0xB1B1: 1305, - 0xB1B3: 1306, - 0xB1B7: 1307, - 0xB1C1: 1308, - 0xB1C2: 1309, - 0xB1C5: 1310, - 0xB1D6: 1311, - 0xB1E1: 1312, - 0xB1F6: 1313, - 0xB241: 1314, - 0xB245: 1315, - 0xB249: 1316, - 0xB251: 1317, - 0xB253: 1318, - 0xB261: 1319, - 0xB281: 1320, - 0xB282: 1321, - 0xB285: 1322, - 0xB289: 1323, - 0xB291: 1324, - 0xB293: 1325, - 0xB297: 1326, - 0xB2A1: 1327, - 0xB2B6: 1328, - 0xB2C1: 1329, - 0xB2E1: 1330, - 0xB2E5: 1331, - 0xB357: 1332, - 0xB361: 1333, - 0xB362: 1334, - 0xB365: 1335, - 0xB369: 1336, - 0xB36B: 1337, - 0xB370: 1338, - 0xB371: 1339, - 0xB373: 1340, - 0xB381: 1341, - 0xB385: 1342, - 0xB389: 1343, - 0xB391: 1344, - 0xB3A1: 1345, - 0xB3A2: 1346, - 0xB3A5: 1347, - 0xB3A9: 1348, - 0xB3B1: 1349, - 0xB3B3: 1350, - 0xB3B5: 1351, - 0xB3B7: 1352, - 0xB461: 1353, - 0xB462: 1354, - 0xB465: 1355, - 0xB466: 1356, - 0xB467: 1357, - 0xB469: 1358, - 0xB46A: 1359, - 0xB46B: 1360, - 0xB470: 1361, - 0xB471: 1362, - 0xB473: 1363, - 0xB475: 1364, - 0xB476: 1365, - 0xB477: 1366, - 0xB47B: 1367, - 0xB47C: 1368, - 0xB481: 1369, - 0xB482: 1370, - 0xB485: 1371, - 0xB489: 1372, - 0xB491: 1373, - 0xB493: 1374, - 0xB495: 1375, - 0xB496: 1376, - 0xB497: 1377, - 0xB4A1: 1378, - 0xB4A2: 1379, - 0xB4A5: 1380, - 0xB4A9: 1381, - 0xB4AC: 1382, - 0xB4B1: 1383, - 0xB4B3: 1384, - 0xB4B5: 1385, - 0xB4B7: 1386, - 0xB4BB: 1387, - 0xB4BD: 1388, - 0xB4C1: 1389, - 0xB4C5: 1390, - 0xB4C9: 1391, - 0xB4D3: 1392, - 0xB4E1: 1393, - 0xB4E2: 1394, - 0xB4E5: 1395, - 0xB4E6: 1396, - 0xB4E8: 1397, - 0xB4E9: 1398, - 0xB4EA: 1399, - 0xB4EB: 1400, - 0xB4F1: 1401, - 0xB4F3: 1402, - 0xB4F4: 1403, - 0xB4F5: 1404, - 0xB4F6: 1405, - 0xB4F7: 1406, - 0xB4F8: 1407, - 0xB4FA: 1408, - 0xB4FC: 1409, - 0xB541: 1410, - 0xB542: 1411, - 0xB545: 1412, - 0xB549: 1413, - 0xB551: 1414, - 0xB553: 1415, - 0xB555: 1416, - 0xB557: 1417, - 0xB561: 1418, - 0xB562: 1419, - 0xB563: 1420, - 0xB565: 1421, - 0xB569: 1422, - 0xB56B: 1423, - 0xB56C: 1424, - 0xB571: 1425, - 0xB573: 1426, - 0xB574: 1427, - 0xB575: 1428, - 0xB576: 1429, - 0xB577: 1430, - 0xB57B: 1431, - 0xB57C: 1432, - 0xB57D: 1433, - 0xB581: 1434, - 0xB585: 1435, - 0xB589: 1436, - 0xB591: 1437, - 0xB593: 1438, - 0xB595: 1439, - 0xB596: 1440, - 0xB5A1: 1441, - 0xB5A2: 1442, - 0xB5A5: 1443, - 0xB5A9: 1444, - 0xB5AA: 1445, - 0xB5AB: 1446, - 0xB5AD: 1447, - 0xB5B0: 1448, - 0xB5B1: 1449, - 0xB5B3: 1450, - 0xB5B5: 1451, - 0xB5B7: 1452, - 0xB5B9: 1453, - 0xB5C1: 1454, - 0xB5C2: 1455, - 0xB5C5: 1456, - 0xB5C9: 1457, - 0xB5D1: 1458, - 0xB5D3: 1459, - 0xB5D5: 1460, - 0xB5D6: 1461, - 0xB5D7: 1462, - 0xB5E1: 1463, - 0xB5E2: 1464, - 0xB5E5: 1465, - 0xB5F1: 1466, - 0xB5F5: 1467, - 0xB5F7: 1468, - 0xB641: 1469, - 0xB642: 1470, - 0xB645: 1471, - 0xB649: 1472, - 0xB651: 1473, - 0xB653: 1474, - 0xB655: 1475, - 0xB657: 1476, - 0xB661: 1477, - 0xB662: 1478, - 0xB665: 1479, - 0xB669: 1480, - 0xB671: 1481, - 0xB673: 1482, - 0xB675: 1483, - 0xB677: 1484, - 0xB681: 1485, - 0xB682: 1486, - 0xB685: 1487, - 0xB689: 1488, - 0xB68A: 1489, - 0xB68B: 1490, - 0xB691: 1491, - 0xB693: 1492, - 0xB695: 1493, - 0xB697: 1494, - 0xB6A1: 1495, - 0xB6A2: 1496, - 0xB6A5: 1497, - 0xB6A9: 1498, - 0xB6B1: 1499, - 0xB6B3: 1500, - 0xB6B6: 1501, - 0xB6B7: 1502, - 0xB6C1: 1503, - 0xB6C2: 1504, - 0xB6C5: 1505, - 0xB6C9: 1506, - 0xB6D1: 1507, - 0xB6D3: 1508, - 0xB6D7: 1509, - 0xB6E1: 1510, - 0xB6E2: 1511, - 0xB6E5: 1512, - 0xB6E9: 1513, - 0xB6F1: 1514, - 0xB6F3: 1515, - 0xB6F5: 1516, - 0xB6F7: 1517, - 0xB741: 1518, - 0xB742: 1519, - 0xB745: 1520, - 0xB749: 1521, - 0xB751: 1522, - 0xB753: 1523, - 0xB755: 1524, - 0xB757: 1525, - 0xB759: 1526, - 0xB761: 1527, - 0xB762: 1528, - 0xB765: 1529, - 0xB769: 1530, - 0xB76F: 1531, - 0xB771: 1532, - 0xB773: 1533, - 0xB775: 1534, - 0xB777: 1535, - 0xB778: 1536, - 0xB779: 1537, - 0xB77A: 1538, - 0xB77B: 1539, - 0xB77C: 1540, - 0xB77D: 1541, - 0xB781: 1542, - 0xB785: 1543, - 0xB789: 1544, - 0xB791: 1545, - 0xB795: 1546, - 0xB7A1: 1547, - 0xB7A2: 1548, - 0xB7A5: 1549, - 0xB7A9: 1550, - 0xB7AA: 1551, - 0xB7AB: 1552, - 0xB7B0: 1553, - 0xB7B1: 1554, - 0xB7B3: 1555, - 0xB7B5: 1556, - 0xB7B6: 1557, - 0xB7B7: 1558, - 0xB7B8: 1559, - 0xB7BC: 1560, - 0xB861: 1561, - 0xB862: 1562, - 0xB865: 1563, - 0xB867: 1564, - 0xB868: 1565, - 0xB869: 1566, - 0xB86B: 1567, - 0xB871: 1568, - 0xB873: 1569, - 0xB875: 1570, - 0xB876: 1571, - 0xB877: 1572, - 0xB878: 1573, - 0xB881: 1574, - 0xB882: 1575, - 0xB885: 1576, - 0xB889: 1577, - 0xB891: 1578, - 0xB893: 1579, - 0xB895: 1580, - 0xB896: 1581, - 0xB897: 1582, - 0xB8A1: 1583, - 0xB8A2: 1584, - 0xB8A5: 1585, - 0xB8A7: 1586, - 0xB8A9: 1587, - 0xB8B1: 1588, - 0xB8B7: 1589, - 0xB8C1: 1590, - 0xB8C5: 1591, - 0xB8C9: 1592, - 0xB8E1: 1593, - 0xB8E2: 1594, - 0xB8E5: 1595, - 0xB8E9: 1596, - 0xB8EB: 1597, - 0xB8F1: 1598, - 0xB8F3: 1599, - 0xB8F5: 1600, - 0xB8F7: 1601, - 0xB8F8: 1602, - 0xB941: 1603, - 0xB942: 1604, - 0xB945: 1605, - 0xB949: 1606, - 0xB951: 1607, - 0xB953: 1608, - 0xB955: 1609, - 0xB957: 1610, - 0xB961: 1611, - 0xB965: 1612, - 0xB969: 1613, - 0xB971: 1614, - 0xB973: 1615, - 0xB976: 1616, - 0xB977: 1617, - 0xB981: 1618, - 0xB9A1: 1619, - 0xB9A2: 1620, - 0xB9A5: 1621, - 0xB9A9: 1622, - 0xB9AB: 1623, - 0xB9B1: 1624, - 0xB9B3: 1625, - 0xB9B5: 1626, - 0xB9B7: 1627, - 0xB9B8: 1628, - 0xB9B9: 1629, - 0xB9BD: 1630, - 0xB9C1: 1631, - 0xB9C2: 1632, - 0xB9C9: 1633, - 0xB9D3: 1634, - 0xB9D5: 1635, - 0xB9D7: 1636, - 0xB9E1: 1637, - 0xB9F6: 1638, - 0xB9F7: 1639, - 0xBA41: 1640, - 0xBA45: 1641, - 0xBA49: 1642, - 0xBA51: 1643, - 0xBA53: 1644, - 0xBA55: 1645, - 0xBA57: 1646, - 0xBA61: 1647, - 0xBA62: 1648, - 0xBA65: 1649, - 0xBA77: 1650, - 0xBA81: 1651, - 0xBA82: 1652, - 0xBA85: 1653, - 0xBA89: 1654, - 0xBA8A: 1655, - 0xBA8B: 1656, - 0xBA91: 1657, - 0xBA93: 1658, - 0xBA95: 1659, - 0xBA97: 1660, - 0xBAA1: 1661, - 0xBAB6: 1662, - 0xBAC1: 1663, - 0xBAE1: 1664, - 0xBAE2: 1665, - 0xBAE5: 1666, - 0xBAE9: 1667, - 0xBAF1: 1668, - 0xBAF3: 1669, - 0xBAF5: 1670, - 0xBB41: 1671, - 0xBB45: 1672, - 0xBB49: 1673, - 0xBB51: 1674, - 0xBB61: 1675, - 0xBB62: 1676, - 0xBB65: 1677, - 0xBB69: 1678, - 0xBB71: 1679, - 0xBB73: 1680, - 0xBB75: 1681, - 0xBB77: 1682, - 0xBBA1: 1683, - 0xBBA2: 1684, - 0xBBA5: 1685, - 0xBBA8: 1686, - 0xBBA9: 1687, - 0xBBAB: 1688, - 0xBBB1: 1689, - 0xBBB3: 1690, - 0xBBB5: 1691, - 0xBBB7: 1692, - 0xBBB8: 1693, - 0xBBBB: 1694, - 0xBBBC: 1695, - 0xBC61: 1696, - 0xBC62: 1697, - 0xBC65: 1698, - 0xBC67: 1699, - 0xBC69: 1700, - 0xBC6C: 1701, - 0xBC71: 1702, - 0xBC73: 1703, - 0xBC75: 1704, - 0xBC76: 1705, - 0xBC77: 1706, - 0xBC81: 1707, - 0xBC82: 1708, - 0xBC85: 1709, - 0xBC89: 1710, - 0xBC91: 1711, - 0xBC93: 1712, - 0xBC95: 1713, - 0xBC96: 1714, - 0xBC97: 1715, - 0xBCA1: 1716, - 0xBCA5: 1717, - 0xBCB7: 1718, - 0xBCE1: 1719, - 0xBCE2: 1720, - 0xBCE5: 1721, - 0xBCE9: 1722, - 0xBCF1: 1723, - 0xBCF3: 1724, - 0xBCF5: 1725, - 0xBCF6: 1726, - 0xBCF7: 1727, - 0xBD41: 1728, - 0xBD57: 1729, - 0xBD61: 1730, - 0xBD76: 1731, - 0xBDA1: 1732, - 0xBDA2: 1733, - 0xBDA5: 1734, - 0xBDA9: 1735, - 0xBDB1: 1736, - 0xBDB3: 1737, - 0xBDB5: 1738, - 0xBDB7: 1739, - 0xBDB9: 1740, - 0xBDC1: 1741, - 0xBDC2: 1742, - 0xBDC9: 1743, - 0xBDD6: 1744, - 0xBDE1: 1745, - 0xBDF6: 1746, - 0xBE41: 1747, - 0xBE45: 1748, - 0xBE49: 1749, - 0xBE51: 1750, - 0xBE53: 1751, - 0xBE77: 1752, - 0xBE81: 1753, - 0xBE82: 1754, - 0xBE85: 1755, - 0xBE89: 1756, - 0xBE91: 1757, - 0xBE93: 1758, - 0xBE97: 1759, - 0xBEA1: 1760, - 0xBEB6: 1761, - 0xBEB7: 1762, - 0xBEE1: 1763, - 0xBF41: 1764, - 0xBF61: 1765, - 0xBF71: 1766, - 0xBF75: 1767, - 0xBF77: 1768, - 0xBFA1: 1769, - 0xBFA2: 1770, - 0xBFA5: 1771, - 0xBFA9: 1772, - 0xBFB1: 1773, - 0xBFB3: 1774, - 0xBFB7: 1775, - 0xBFB8: 1776, - 0xBFBD: 1777, - 0xC061: 1778, - 0xC062: 1779, - 0xC065: 1780, - 0xC067: 1781, - 0xC069: 1782, - 0xC071: 1783, - 0xC073: 1784, - 0xC075: 1785, - 0xC076: 1786, - 0xC077: 1787, - 0xC078: 1788, - 0xC081: 1789, - 0xC082: 1790, - 0xC085: 1791, - 0xC089: 1792, - 0xC091: 1793, - 0xC093: 1794, - 0xC095: 1795, - 0xC096: 1796, - 0xC097: 1797, - 0xC0A1: 1798, - 0xC0A5: 1799, - 0xC0A7: 1800, - 0xC0A9: 1801, - 0xC0B1: 1802, - 0xC0B7: 1803, - 0xC0E1: 1804, - 0xC0E2: 1805, - 0xC0E5: 1806, - 0xC0E9: 1807, - 0xC0F1: 1808, - 0xC0F3: 1809, - 0xC0F5: 1810, - 0xC0F6: 1811, - 0xC0F7: 1812, - 0xC141: 1813, - 0xC142: 1814, - 0xC145: 1815, - 0xC149: 1816, - 0xC151: 1817, - 0xC153: 1818, - 0xC155: 1819, - 0xC157: 1820, - 0xC161: 1821, - 0xC165: 1822, - 0xC176: 1823, - 0xC181: 1824, - 0xC185: 1825, - 0xC197: 1826, - 0xC1A1: 1827, - 0xC1A2: 1828, - 0xC1A5: 1829, - 0xC1A9: 1830, - 0xC1B1: 1831, - 0xC1B3: 1832, - 0xC1B5: 1833, - 0xC1B7: 1834, - 0xC1C1: 1835, - 0xC1C5: 1836, - 0xC1C9: 1837, - 0xC1D7: 1838, - 0xC241: 1839, - 0xC245: 1840, - 0xC249: 1841, - 0xC251: 1842, - 0xC253: 1843, - 0xC255: 1844, - 0xC257: 1845, - 0xC261: 1846, - 0xC271: 1847, - 0xC281: 1848, - 0xC282: 1849, - 0xC285: 1850, - 0xC289: 1851, - 0xC291: 1852, - 0xC293: 1853, - 0xC295: 1854, - 0xC297: 1855, - 0xC2A1: 1856, - 0xC2B6: 1857, - 0xC2C1: 1858, - 0xC2C5: 1859, - 0xC2E1: 1860, - 0xC2E5: 1861, - 0xC2E9: 1862, - 0xC2F1: 1863, - 0xC2F3: 1864, - 0xC2F5: 1865, - 0xC2F7: 1866, - 0xC341: 1867, - 0xC345: 1868, - 0xC349: 1869, - 0xC351: 1870, - 0xC357: 1871, - 0xC361: 1872, - 0xC362: 1873, - 0xC365: 1874, - 0xC369: 1875, - 0xC371: 1876, - 0xC373: 1877, - 0xC375: 1878, - 0xC377: 1879, - 0xC3A1: 1880, - 0xC3A2: 1881, - 0xC3A5: 1882, - 0xC3A8: 1883, - 0xC3A9: 1884, - 0xC3AA: 1885, - 0xC3B1: 1886, - 0xC3B3: 1887, - 0xC3B5: 1888, - 0xC3B7: 1889, - 0xC461: 1890, - 0xC462: 1891, - 0xC465: 1892, - 0xC469: 1893, - 0xC471: 1894, - 0xC473: 1895, - 0xC475: 1896, - 0xC477: 1897, - 0xC481: 1898, - 0xC482: 1899, - 0xC485: 1900, - 0xC489: 1901, - 0xC491: 1902, - 0xC493: 1903, - 0xC495: 1904, - 0xC496: 1905, - 0xC497: 1906, - 0xC4A1: 1907, - 0xC4A2: 1908, - 0xC4B7: 1909, - 0xC4E1: 1910, - 0xC4E2: 1911, - 0xC4E5: 1912, - 0xC4E8: 1913, - 0xC4E9: 1914, - 0xC4F1: 1915, - 0xC4F3: 1916, - 0xC4F5: 1917, - 0xC4F6: 1918, - 0xC4F7: 1919, - 0xC541: 1920, - 0xC542: 1921, - 0xC545: 1922, - 0xC549: 1923, - 0xC551: 1924, - 0xC553: 1925, - 0xC555: 1926, - 0xC557: 1927, - 0xC561: 1928, - 0xC565: 1929, - 0xC569: 1930, - 0xC571: 1931, - 0xC573: 1932, - 0xC575: 1933, - 0xC576: 1934, - 0xC577: 1935, - 0xC581: 1936, - 0xC5A1: 1937, - 0xC5A2: 1938, - 0xC5A5: 1939, - 0xC5A9: 1940, - 0xC5B1: 1941, - 0xC5B3: 1942, - 0xC5B5: 1943, - 0xC5B7: 1944, - 0xC5C1: 1945, - 0xC5C2: 1946, - 0xC5C5: 1947, - 0xC5C9: 1948, - 0xC5D1: 1949, - 0xC5D7: 1950, - 0xC5E1: 1951, - 0xC5F7: 1952, - 0xC641: 1953, - 0xC649: 1954, - 0xC661: 1955, - 0xC681: 1956, - 0xC682: 1957, - 0xC685: 1958, - 0xC689: 1959, - 0xC691: 1960, - 0xC693: 1961, - 0xC695: 1962, - 0xC697: 1963, - 0xC6A1: 1964, - 0xC6A5: 1965, - 0xC6A9: 1966, - 0xC6B7: 1967, - 0xC6C1: 1968, - 0xC6D7: 1969, - 0xC6E1: 1970, - 0xC6E2: 1971, - 0xC6E5: 1972, - 0xC6E9: 1973, - 0xC6F1: 1974, - 0xC6F3: 1975, - 0xC6F5: 1976, - 0xC6F7: 1977, - 0xC741: 1978, - 0xC745: 1979, - 0xC749: 1980, - 0xC751: 1981, - 0xC761: 1982, - 0xC762: 1983, - 0xC765: 1984, - 0xC769: 1985, - 0xC771: 1986, - 0xC773: 1987, - 0xC777: 1988, - 0xC7A1: 1989, - 0xC7A2: 1990, - 0xC7A5: 1991, - 0xC7A9: 1992, - 0xC7B1: 1993, - 0xC7B3: 1994, - 0xC7B5: 1995, - 0xC7B7: 1996, - 0xC861: 1997, - 0xC862: 1998, - 0xC865: 1999, - 0xC869: 2000, - 0xC86A: 2001, - 0xC871: 2002, - 0xC873: 2003, - 0xC875: 2004, - 0xC876: 2005, - 0xC877: 2006, - 0xC881: 2007, - 0xC882: 2008, - 0xC885: 2009, - 0xC889: 2010, - 0xC891: 2011, - 0xC893: 2012, - 0xC895: 2013, - 0xC896: 2014, - 0xC897: 2015, - 0xC8A1: 2016, - 0xC8B7: 2017, - 0xC8E1: 2018, - 0xC8E2: 2019, - 0xC8E5: 2020, - 0xC8E9: 2021, - 0xC8EB: 2022, - 0xC8F1: 2023, - 0xC8F3: 2024, - 0xC8F5: 2025, - 0xC8F6: 2026, - 0xC8F7: 2027, - 0xC941: 2028, - 0xC942: 2029, - 0xC945: 2030, - 0xC949: 2031, - 0xC951: 2032, - 0xC953: 2033, - 0xC955: 2034, - 0xC957: 2035, - 0xC961: 2036, - 0xC965: 2037, - 0xC976: 2038, - 0xC981: 2039, - 0xC985: 2040, - 0xC9A1: 2041, - 0xC9A2: 2042, - 0xC9A5: 2043, - 0xC9A9: 2044, - 0xC9B1: 2045, - 0xC9B3: 2046, - 0xC9B5: 2047, - 0xC9B7: 2048, - 0xC9BC: 2049, - 0xC9C1: 2050, - 0xC9C5: 2051, - 0xC9E1: 2052, - 0xCA41: 2053, - 0xCA45: 2054, - 0xCA55: 2055, - 0xCA57: 2056, - 0xCA61: 2057, - 0xCA81: 2058, - 0xCA82: 2059, - 0xCA85: 2060, - 0xCA89: 2061, - 0xCA91: 2062, - 0xCA93: 2063, - 0xCA95: 2064, - 0xCA97: 2065, - 0xCAA1: 2066, - 0xCAB6: 2067, - 0xCAC1: 2068, - 0xCAE1: 2069, - 0xCAE2: 2070, - 0xCAE5: 2071, - 0xCAE9: 2072, - 0xCAF1: 2073, - 0xCAF3: 2074, - 0xCAF7: 2075, - 0xCB41: 2076, - 0xCB45: 2077, - 0xCB49: 2078, - 0xCB51: 2079, - 0xCB57: 2080, - 0xCB61: 2081, - 0xCB62: 2082, - 0xCB65: 2083, - 0xCB68: 2084, - 0xCB69: 2085, - 0xCB6B: 2086, - 0xCB71: 2087, - 0xCB73: 2088, - 0xCB75: 2089, - 0xCB81: 2090, - 0xCB85: 2091, - 0xCB89: 2092, - 0xCB91: 2093, - 0xCB93: 2094, - 0xCBA1: 2095, - 0xCBA2: 2096, - 0xCBA5: 2097, - 0xCBA9: 2098, - 0xCBB1: 2099, - 0xCBB3: 2100, - 0xCBB5: 2101, - 0xCBB7: 2102, - 0xCC61: 2103, - 0xCC62: 2104, - 0xCC63: 2105, - 0xCC65: 2106, - 0xCC69: 2107, - 0xCC6B: 2108, - 0xCC71: 2109, - 0xCC73: 2110, - 0xCC75: 2111, - 0xCC76: 2112, - 0xCC77: 2113, - 0xCC7B: 2114, - 0xCC81: 2115, - 0xCC82: 2116, - 0xCC85: 2117, - 0xCC89: 2118, - 0xCC91: 2119, - 0xCC93: 2120, - 0xCC95: 2121, - 0xCC96: 2122, - 0xCC97: 2123, - 0xCCA1: 2124, - 0xCCA2: 2125, - 0xCCE1: 2126, - 0xCCE2: 2127, - 0xCCE5: 2128, - 0xCCE9: 2129, - 0xCCF1: 2130, - 0xCCF3: 2131, - 0xCCF5: 2132, - 0xCCF6: 2133, - 0xCCF7: 2134, - 0xCD41: 2135, - 0xCD42: 2136, - 0xCD45: 2137, - 0xCD49: 2138, - 0xCD51: 2139, - 0xCD53: 2140, - 0xCD55: 2141, - 0xCD57: 2142, - 0xCD61: 2143, - 0xCD65: 2144, - 0xCD69: 2145, - 0xCD71: 2146, - 0xCD73: 2147, - 0xCD76: 2148, - 0xCD77: 2149, - 0xCD81: 2150, - 0xCD89: 2151, - 0xCD93: 2152, - 0xCD95: 2153, - 0xCDA1: 2154, - 0xCDA2: 2155, - 0xCDA5: 2156, - 0xCDA9: 2157, - 0xCDB1: 2158, - 0xCDB3: 2159, - 0xCDB5: 2160, - 0xCDB7: 2161, - 0xCDC1: 2162, - 0xCDD7: 2163, - 0xCE41: 2164, - 0xCE45: 2165, - 0xCE61: 2166, - 0xCE65: 2167, - 0xCE69: 2168, - 0xCE73: 2169, - 0xCE75: 2170, - 0xCE81: 2171, - 0xCE82: 2172, - 0xCE85: 2173, - 0xCE88: 2174, - 0xCE89: 2175, - 0xCE8B: 2176, - 0xCE91: 2177, - 0xCE93: 2178, - 0xCE95: 2179, - 0xCE97: 2180, - 0xCEA1: 2181, - 0xCEB7: 2182, - 0xCEE1: 2183, - 0xCEE5: 2184, - 0xCEE9: 2185, - 0xCEF1: 2186, - 0xCEF5: 2187, - 0xCF41: 2188, - 0xCF45: 2189, - 0xCF49: 2190, - 0xCF51: 2191, - 0xCF55: 2192, - 0xCF57: 2193, - 0xCF61: 2194, - 0xCF65: 2195, - 0xCF69: 2196, - 0xCF71: 2197, - 0xCF73: 2198, - 0xCF75: 2199, - 0xCFA1: 2200, - 0xCFA2: 2201, - 0xCFA5: 2202, - 0xCFA9: 2203, - 0xCFB1: 2204, - 0xCFB3: 2205, - 0xCFB5: 2206, - 0xCFB7: 2207, - 0xD061: 2208, - 0xD062: 2209, - 0xD065: 2210, - 0xD069: 2211, - 0xD06E: 2212, - 0xD071: 2213, - 0xD073: 2214, - 0xD075: 2215, - 0xD077: 2216, - 0xD081: 2217, - 0xD082: 2218, - 0xD085: 2219, - 0xD089: 2220, - 0xD091: 2221, - 0xD093: 2222, - 0xD095: 2223, - 0xD096: 2224, - 0xD097: 2225, - 0xD0A1: 2226, - 0xD0B7: 2227, - 0xD0E1: 2228, - 0xD0E2: 2229, - 0xD0E5: 2230, - 0xD0E9: 2231, - 0xD0EB: 2232, - 0xD0F1: 2233, - 0xD0F3: 2234, - 0xD0F5: 2235, - 0xD0F7: 2236, - 0xD141: 2237, - 0xD142: 2238, - 0xD145: 2239, - 0xD149: 2240, - 0xD151: 2241, - 0xD153: 2242, - 0xD155: 2243, - 0xD157: 2244, - 0xD161: 2245, - 0xD162: 2246, - 0xD165: 2247, - 0xD169: 2248, - 0xD171: 2249, - 0xD173: 2250, - 0xD175: 2251, - 0xD176: 2252, - 0xD177: 2253, - 0xD181: 2254, - 0xD185: 2255, - 0xD189: 2256, - 0xD193: 2257, - 0xD1A1: 2258, - 0xD1A2: 2259, - 0xD1A5: 2260, - 0xD1A9: 2261, - 0xD1AE: 2262, - 0xD1B1: 2263, - 0xD1B3: 2264, - 0xD1B5: 2265, - 0xD1B7: 2266, - 0xD1BB: 2267, - 0xD1C1: 2268, - 0xD1C2: 2269, - 0xD1C5: 2270, - 0xD1C9: 2271, - 0xD1D5: 2272, - 0xD1D7: 2273, - 0xD1E1: 2274, - 0xD1E2: 2275, - 0xD1E5: 2276, - 0xD1F5: 2277, - 0xD1F7: 2278, - 0xD241: 2279, - 0xD242: 2280, - 0xD245: 2281, - 0xD249: 2282, - 0xD253: 2283, - 0xD255: 2284, - 0xD257: 2285, - 0xD261: 2286, - 0xD265: 2287, - 0xD269: 2288, - 0xD273: 2289, - 0xD275: 2290, - 0xD281: 2291, - 0xD282: 2292, - 0xD285: 2293, - 0xD289: 2294, - 0xD28E: 2295, - 0xD291: 2296, - 0xD295: 2297, - 0xD297: 2298, - 0xD2A1: 2299, - 0xD2A5: 2300, - 0xD2A9: 2301, - 0xD2B1: 2302, - 0xD2B7: 2303, - 0xD2C1: 2304, - 0xD2C2: 2305, - 0xD2C5: 2306, - 0xD2C9: 2307, - 0xD2D7: 2308, - 0xD2E1: 2309, - 0xD2E2: 2310, - 0xD2E5: 2311, - 0xD2E9: 2312, - 0xD2F1: 2313, - 0xD2F3: 2314, - 0xD2F5: 2315, - 0xD2F7: 2316, - 0xD341: 2317, - 0xD342: 2318, - 0xD345: 2319, - 0xD349: 2320, - 0xD351: 2321, - 0xD355: 2322, - 0xD357: 2323, - 0xD361: 2324, - 0xD362: 2325, - 0xD365: 2326, - 0xD367: 2327, - 0xD368: 2328, - 0xD369: 2329, - 0xD36A: 2330, - 0xD371: 2331, - 0xD373: 2332, - 0xD375: 2333, - 0xD377: 2334, - 0xD37B: 2335, - 0xD381: 2336, - 0xD385: 2337, - 0xD389: 2338, - 0xD391: 2339, - 0xD393: 2340, - 0xD397: 2341, - 0xD3A1: 2342, - 0xD3A2: 2343, - 0xD3A5: 2344, - 0xD3A9: 2345, - 0xD3B1: 2346, - 0xD3B3: 2347, - 0xD3B5: 2348, - 0xD3B7: 2349, -} diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/johabprober.py b/venv/lib/python3.11/site-packages/pip/_vendor/chardet/johabprober.py deleted file mode 100644 index d7364ba..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/johabprober.py +++ /dev/null @@ -1,47 +0,0 @@ -######################## BEGIN LICENSE BLOCK ######################## -# The Original Code is mozilla.org code. -# -# The Initial Developer of the Original Code is -# Netscape Communications Corporation. -# Portions created by the Initial Developer are Copyright (C) 1998 -# the Initial Developer. All Rights Reserved. -# -# Contributor(s): -# Mark Pilgrim - port to Python -# -# This library is free software; you can redistribute it and/or -# modify it under the terms of the GNU Lesser General Public -# License as published by the Free Software Foundation; either -# version 2.1 of the License, or (at your option) any later version. -# -# This library is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# Lesser General Public License for more details. -# -# You should have received a copy of the GNU Lesser General Public -# License along with this library; if not, write to the Free Software -# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA -# 02110-1301 USA -######################### END LICENSE BLOCK ######################### - -from .chardistribution import JOHABDistributionAnalysis -from .codingstatemachine import CodingStateMachine -from .mbcharsetprober import MultiByteCharSetProber -from .mbcssm import JOHAB_SM_MODEL - - -class JOHABProber(MultiByteCharSetProber): - def __init__(self) -> None: - super().__init__() - self.coding_sm = CodingStateMachine(JOHAB_SM_MODEL) - self.distribution_analyzer = JOHABDistributionAnalysis() - self.reset() - - @property - def charset_name(self) -> str: - return "Johab" - - @property - def language(self) -> str: - return "Korean" diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/jpcntx.py b/venv/lib/python3.11/site-packages/pip/_vendor/chardet/jpcntx.py deleted file mode 100644 index 2f53bdd..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/jpcntx.py +++ /dev/null @@ -1,238 +0,0 @@ -######################## BEGIN LICENSE BLOCK ######################## -# The Original Code is Mozilla Communicator client code. -# -# The Initial Developer of the Original Code is -# Netscape Communications Corporation. -# Portions created by the Initial Developer are Copyright (C) 1998 -# the Initial Developer. All Rights Reserved. -# -# Contributor(s): -# Mark Pilgrim - port to Python -# -# This library is free software; you can redistribute it and/or -# modify it under the terms of the GNU Lesser General Public -# License as published by the Free Software Foundation; either -# version 2.1 of the License, or (at your option) any later version. -# -# This library is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# Lesser General Public License for more details. -# -# You should have received a copy of the GNU Lesser General Public -# License along with this library; if not, write to the Free Software -# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA -# 02110-1301 USA -######################### END LICENSE BLOCK ######################### - -from typing import List, Tuple, Union - -# This is hiragana 2-char sequence table, the number in each cell represents its frequency category -# fmt: off -jp2_char_context = ( - (0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1), - (2, 4, 0, 4, 0, 3, 0, 4, 0, 3, 4, 4, 4, 2, 4, 3, 3, 4, 3, 2, 3, 3, 4, 2, 3, 3, 3, 2, 4, 1, 4, 3, 3, 1, 5, 4, 3, 4, 3, 4, 3, 5, 3, 0, 3, 5, 4, 2, 0, 3, 1, 0, 3, 3, 0, 3, 3, 0, 1, 1, 0, 4, 3, 0, 3, 3, 0, 4, 0, 2, 0, 3, 5, 5, 5, 5, 4, 0, 4, 1, 0, 3, 4), - (0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2), - (0, 4, 0, 5, 0, 5, 0, 4, 0, 4, 5, 4, 4, 3, 5, 3, 5, 1, 5, 3, 4, 3, 4, 4, 3, 4, 3, 3, 4, 3, 5, 4, 4, 3, 5, 5, 3, 5, 5, 5, 3, 5, 5, 3, 4, 5, 5, 3, 1, 3, 2, 0, 3, 4, 0, 4, 2, 0, 4, 2, 1, 5, 3, 2, 3, 5, 0, 4, 0, 2, 0, 5, 4, 4, 5, 4, 5, 0, 4, 0, 0, 4, 4), - (0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0), - (0, 3, 0, 4, 0, 3, 0, 3, 0, 4, 5, 4, 3, 3, 3, 3, 4, 3, 5, 4, 4, 3, 5, 4, 4, 3, 4, 3, 4, 4, 4, 4, 5, 3, 4, 4, 3, 4, 5, 5, 4, 5, 5, 1, 4, 5, 4, 3, 0, 3, 3, 1, 3, 3, 0, 4, 4, 0, 3, 3, 1, 5, 3, 3, 3, 5, 0, 4, 0, 3, 0, 4, 4, 3, 4, 3, 3, 0, 4, 1, 1, 3, 4), - (0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0), - (0, 4, 0, 3, 0, 3, 0, 4, 0, 3, 4, 4, 3, 2, 2, 1, 2, 1, 3, 1, 3, 3, 3, 3, 3, 4, 3, 1, 3, 3, 5, 3, 3, 0, 4, 3, 0, 5, 4, 3, 3, 5, 4, 4, 3, 4, 4, 5, 0, 1, 2, 0, 1, 2, 0, 2, 2, 0, 1, 0, 0, 5, 2, 2, 1, 4, 0, 3, 0, 1, 0, 4, 4, 3, 5, 4, 3, 0, 2, 1, 0, 4, 3), - (0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0), - (0, 3, 0, 5, 0, 4, 0, 2, 1, 4, 4, 2, 4, 1, 4, 2, 4, 2, 4, 3, 3, 3, 4, 3, 3, 3, 3, 1, 4, 2, 3, 3, 3, 1, 4, 4, 1, 1, 1, 4, 3, 3, 2, 0, 2, 4, 3, 2, 0, 3, 3, 0, 3, 1, 1, 0, 0, 0, 3, 3, 0, 4, 2, 2, 3, 4, 0, 4, 0, 3, 0, 4, 4, 5, 3, 4, 4, 0, 3, 0, 0, 1, 4), - (1, 4, 0, 4, 0, 4, 0, 4, 0, 3, 5, 4, 4, 3, 4, 3, 5, 4, 3, 3, 4, 3, 5, 4, 4, 4, 4, 3, 4, 2, 4, 3, 3, 1, 5, 4, 3, 2, 4, 5, 4, 5, 5, 4, 4, 5, 4, 4, 0, 3, 2, 2, 3, 3, 0, 4, 3, 1, 3, 2, 1, 4, 3, 3, 4, 5, 0, 3, 0, 2, 0, 4, 5, 5, 4, 5, 4, 0, 4, 0, 0, 5, 4), - (0, 5, 0, 5, 0, 4, 0, 3, 0, 4, 4, 3, 4, 3, 3, 3, 4, 0, 4, 4, 4, 3, 4, 3, 4, 3, 3, 1, 4, 2, 4, 3, 4, 0, 5, 4, 1, 4, 5, 4, 4, 5, 3, 2, 4, 3, 4, 3, 2, 4, 1, 3, 3, 3, 2, 3, 2, 0, 4, 3, 3, 4, 3, 3, 3, 4, 0, 4, 0, 3, 0, 4, 5, 4, 4, 4, 3, 0, 4, 1, 0, 1, 3), - (0, 3, 1, 4, 0, 3, 0, 2, 0, 3, 4, 4, 3, 1, 4, 2, 3, 3, 4, 3, 4, 3, 4, 3, 4, 4, 3, 2, 3, 1, 5, 4, 4, 1, 4, 4, 3, 5, 4, 4, 3, 5, 5, 4, 3, 4, 4, 3, 1, 2, 3, 1, 2, 2, 0, 3, 2, 0, 3, 1, 0, 5, 3, 3, 3, 4, 3, 3, 3, 3, 4, 4, 4, 4, 5, 4, 2, 0, 3, 3, 2, 4, 3), - (0, 2, 0, 3, 0, 1, 0, 1, 0, 0, 3, 2, 0, 0, 2, 0, 1, 0, 2, 1, 3, 3, 3, 1, 2, 3, 1, 0, 1, 0, 4, 2, 1, 1, 3, 3, 0, 4, 3, 3, 1, 4, 3, 3, 0, 3, 3, 2, 0, 0, 0, 0, 1, 0, 0, 2, 0, 0, 0, 0, 0, 4, 1, 0, 2, 3, 2, 2, 2, 1, 3, 3, 3, 4, 4, 3, 2, 0, 3, 1, 0, 3, 3), - (0, 4, 0, 4, 0, 3, 0, 3, 0, 4, 4, 4, 3, 3, 3, 3, 3, 3, 4, 3, 4, 2, 4, 3, 4, 3, 3, 2, 4, 3, 4, 5, 4, 1, 4, 5, 3, 5, 4, 5, 3, 5, 4, 0, 3, 5, 5, 3, 1, 3, 3, 2, 2, 3, 0, 3, 4, 1, 3, 3, 2, 4, 3, 3, 3, 4, 0, 4, 0, 3, 0, 4, 5, 4, 4, 5, 3, 0, 4, 1, 0, 3, 4), - (0, 2, 0, 3, 0, 3, 0, 0, 0, 2, 2, 2, 1, 0, 1, 0, 0, 0, 3, 0, 3, 0, 3, 0, 1, 3, 1, 0, 3, 1, 3, 3, 3, 1, 3, 3, 3, 0, 1, 3, 1, 3, 4, 0, 0, 3, 1, 1, 0, 3, 2, 0, 0, 0, 0, 1, 3, 0, 1, 0, 0, 3, 3, 2, 0, 3, 0, 0, 0, 0, 0, 3, 4, 3, 4, 3, 3, 0, 3, 0, 0, 2, 3), - (2, 3, 0, 3, 0, 2, 0, 1, 0, 3, 3, 4, 3, 1, 3, 1, 1, 1, 3, 1, 4, 3, 4, 3, 3, 3, 0, 0, 3, 1, 5, 4, 3, 1, 4, 3, 2, 5, 5, 4, 4, 4, 4, 3, 3, 4, 4, 4, 0, 2, 1, 1, 3, 2, 0, 1, 2, 0, 0, 1, 0, 4, 1, 3, 3, 3, 0, 3, 0, 1, 0, 4, 4, 4, 5, 5, 3, 0, 2, 0, 0, 4, 4), - (0, 2, 0, 1, 0, 3, 1, 3, 0, 2, 3, 3, 3, 0, 3, 1, 0, 0, 3, 0, 3, 2, 3, 1, 3, 2, 1, 1, 0, 0, 4, 2, 1, 0, 2, 3, 1, 4, 3, 2, 0, 4, 4, 3, 1, 3, 1, 3, 0, 1, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 4, 1, 1, 1, 2, 0, 3, 0, 0, 0, 3, 4, 2, 4, 3, 2, 0, 1, 0, 0, 3, 3), - (0, 1, 0, 4, 0, 5, 0, 4, 0, 2, 4, 4, 2, 3, 3, 2, 3, 3, 5, 3, 3, 3, 4, 3, 4, 2, 3, 0, 4, 3, 3, 3, 4, 1, 4, 3, 2, 1, 5, 5, 3, 4, 5, 1, 3, 5, 4, 2, 0, 3, 3, 0, 1, 3, 0, 4, 2, 0, 1, 3, 1, 4, 3, 3, 3, 3, 0, 3, 0, 1, 0, 3, 4, 4, 4, 5, 5, 0, 3, 0, 1, 4, 5), - (0, 2, 0, 3, 0, 3, 0, 0, 0, 2, 3, 1, 3, 0, 4, 0, 1, 1, 3, 0, 3, 4, 3, 2, 3, 1, 0, 3, 3, 2, 3, 1, 3, 0, 2, 3, 0, 2, 1, 4, 1, 2, 2, 0, 0, 3, 3, 0, 0, 2, 0, 0, 0, 1, 0, 0, 0, 0, 2, 2, 0, 3, 2, 1, 3, 3, 0, 2, 0, 2, 0, 0, 3, 3, 1, 2, 4, 0, 3, 0, 2, 2, 3), - (2, 4, 0, 5, 0, 4, 0, 4, 0, 2, 4, 4, 4, 3, 4, 3, 3, 3, 1, 2, 4, 3, 4, 3, 4, 4, 5, 0, 3, 3, 3, 3, 2, 0, 4, 3, 1, 4, 3, 4, 1, 4, 4, 3, 3, 4, 4, 3, 1, 2, 3, 0, 4, 2, 0, 4, 1, 0, 3, 3, 0, 4, 3, 3, 3, 4, 0, 4, 0, 2, 0, 3, 5, 3, 4, 5, 2, 0, 3, 0, 0, 4, 5), - (0, 3, 0, 4, 0, 1, 0, 1, 0, 1, 3, 2, 2, 1, 3, 0, 3, 0, 2, 0, 2, 0, 3, 0, 2, 0, 0, 0, 1, 0, 1, 1, 0, 0, 3, 1, 0, 0, 0, 4, 0, 3, 1, 0, 2, 1, 3, 0, 0, 0, 0, 0, 0, 3, 0, 0, 0, 0, 0, 0, 0, 4, 2, 2, 3, 1, 0, 3, 0, 0, 0, 1, 4, 4, 4, 3, 0, 0, 4, 0, 0, 1, 4), - (1, 4, 1, 5, 0, 3, 0, 3, 0, 4, 5, 4, 4, 3, 5, 3, 3, 4, 4, 3, 4, 1, 3, 3, 3, 3, 2, 1, 4, 1, 5, 4, 3, 1, 4, 4, 3, 5, 4, 4, 3, 5, 4, 3, 3, 4, 4, 4, 0, 3, 3, 1, 2, 3, 0, 3, 1, 0, 3, 3, 0, 5, 4, 4, 4, 4, 4, 4, 3, 3, 5, 4, 4, 3, 3, 5, 4, 0, 3, 2, 0, 4, 4), - (0, 2, 0, 3, 0, 1, 0, 0, 0, 1, 3, 3, 3, 2, 4, 1, 3, 0, 3, 1, 3, 0, 2, 2, 1, 1, 0, 0, 2, 0, 4, 3, 1, 0, 4, 3, 0, 4, 4, 4, 1, 4, 3, 1, 1, 3, 3, 1, 0, 2, 0, 0, 1, 3, 0, 0, 0, 0, 2, 0, 0, 4, 3, 2, 4, 3, 5, 4, 3, 3, 3, 4, 3, 3, 4, 3, 3, 0, 2, 1, 0, 3, 3), - (0, 2, 0, 4, 0, 3, 0, 2, 0, 2, 5, 5, 3, 4, 4, 4, 4, 1, 4, 3, 3, 0, 4, 3, 4, 3, 1, 3, 3, 2, 4, 3, 0, 3, 4, 3, 0, 3, 4, 4, 2, 4, 4, 0, 4, 5, 3, 3, 2, 2, 1, 1, 1, 2, 0, 1, 5, 0, 3, 3, 2, 4, 3, 3, 3, 4, 0, 3, 0, 2, 0, 4, 4, 3, 5, 5, 0, 0, 3, 0, 2, 3, 3), - (0, 3, 0, 4, 0, 3, 0, 1, 0, 3, 4, 3, 3, 1, 3, 3, 3, 0, 3, 1, 3, 0, 4, 3, 3, 1, 1, 0, 3, 0, 3, 3, 0, 0, 4, 4, 0, 1, 5, 4, 3, 3, 5, 0, 3, 3, 4, 3, 0, 2, 0, 1, 1, 1, 0, 1, 3, 0, 1, 2, 1, 3, 3, 2, 3, 3, 0, 3, 0, 1, 0, 1, 3, 3, 4, 4, 1, 0, 1, 2, 2, 1, 3), - (0, 1, 0, 4, 0, 4, 0, 3, 0, 1, 3, 3, 3, 2, 3, 1, 1, 0, 3, 0, 3, 3, 4, 3, 2, 4, 2, 0, 1, 0, 4, 3, 2, 0, 4, 3, 0, 5, 3, 3, 2, 4, 4, 4, 3, 3, 3, 4, 0, 1, 3, 0, 0, 1, 0, 0, 1, 0, 0, 0, 0, 4, 2, 3, 3, 3, 0, 3, 0, 0, 0, 4, 4, 4, 5, 3, 2, 0, 3, 3, 0, 3, 5), - (0, 2, 0, 3, 0, 0, 0, 3, 0, 1, 3, 0, 2, 0, 0, 0, 1, 0, 3, 1, 1, 3, 3, 0, 0, 3, 0, 0, 3, 0, 2, 3, 1, 0, 3, 1, 0, 3, 3, 2, 0, 4, 2, 2, 0, 2, 0, 0, 0, 4, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 1, 2, 0, 1, 0, 1, 0, 0, 0, 1, 3, 1, 2, 0, 0, 0, 1, 0, 0, 1, 4), - (0, 3, 0, 3, 0, 5, 0, 1, 0, 2, 4, 3, 1, 3, 3, 2, 1, 1, 5, 2, 1, 0, 5, 1, 2, 0, 0, 0, 3, 3, 2, 2, 3, 2, 4, 3, 0, 0, 3, 3, 1, 3, 3, 0, 2, 5, 3, 4, 0, 3, 3, 0, 1, 2, 0, 2, 2, 0, 3, 2, 0, 2, 2, 3, 3, 3, 0, 2, 0, 1, 0, 3, 4, 4, 2, 5, 4, 0, 3, 0, 0, 3, 5), - (0, 3, 0, 3, 0, 3, 0, 1, 0, 3, 3, 3, 3, 0, 3, 0, 2, 0, 2, 1, 1, 0, 2, 0, 1, 0, 0, 0, 2, 1, 0, 0, 1, 0, 3, 2, 0, 0, 3, 3, 1, 2, 3, 1, 0, 3, 3, 0, 0, 1, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 2, 3, 1, 2, 3, 0, 3, 0, 1, 0, 3, 2, 1, 0, 4, 3, 0, 1, 1, 0, 3, 3), - (0, 4, 0, 5, 0, 3, 0, 3, 0, 4, 5, 5, 4, 3, 5, 3, 4, 3, 5, 3, 3, 2, 5, 3, 4, 4, 4, 3, 4, 3, 4, 5, 5, 3, 4, 4, 3, 4, 4, 5, 4, 4, 4, 3, 4, 5, 5, 4, 2, 3, 4, 2, 3, 4, 0, 3, 3, 1, 4, 3, 2, 4, 3, 3, 5, 5, 0, 3, 0, 3, 0, 5, 5, 5, 5, 4, 4, 0, 4, 0, 1, 4, 4), - (0, 4, 0, 4, 0, 3, 0, 3, 0, 3, 5, 4, 4, 2, 3, 2, 5, 1, 3, 2, 5, 1, 4, 2, 3, 2, 3, 3, 4, 3, 3, 3, 3, 2, 5, 4, 1, 3, 3, 5, 3, 4, 4, 0, 4, 4, 3, 1, 1, 3, 1, 0, 2, 3, 0, 2, 3, 0, 3, 0, 0, 4, 3, 1, 3, 4, 0, 3, 0, 2, 0, 4, 4, 4, 3, 4, 5, 0, 4, 0, 0, 3, 4), - (0, 3, 0, 3, 0, 3, 1, 2, 0, 3, 4, 4, 3, 3, 3, 0, 2, 2, 4, 3, 3, 1, 3, 3, 3, 1, 1, 0, 3, 1, 4, 3, 2, 3, 4, 4, 2, 4, 4, 4, 3, 4, 4, 3, 2, 4, 4, 3, 1, 3, 3, 1, 3, 3, 0, 4, 1, 0, 2, 2, 1, 4, 3, 2, 3, 3, 5, 4, 3, 3, 5, 4, 4, 3, 3, 0, 4, 0, 3, 2, 2, 4, 4), - (0, 2, 0, 1, 0, 0, 0, 0, 0, 1, 2, 1, 3, 0, 0, 0, 0, 0, 2, 0, 1, 2, 1, 0, 0, 1, 0, 0, 0, 0, 3, 0, 0, 1, 0, 1, 1, 3, 1, 0, 0, 0, 1, 1, 0, 1, 1, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 2, 2, 0, 3, 4, 0, 0, 0, 1, 1, 0, 0, 1, 0, 0, 0, 0, 0, 1, 1), - (0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 4, 0, 4, 1, 4, 0, 3, 0, 4, 0, 3, 0, 4, 0, 3, 0, 3, 0, 4, 1, 5, 1, 4, 0, 0, 3, 0, 5, 0, 5, 2, 0, 1, 0, 0, 0, 2, 1, 4, 0, 1, 3, 0, 0, 3, 0, 0, 3, 1, 1, 4, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0), - (1, 4, 0, 5, 0, 3, 0, 2, 0, 3, 5, 4, 4, 3, 4, 3, 5, 3, 4, 3, 3, 0, 4, 3, 3, 3, 3, 3, 3, 2, 4, 4, 3, 1, 3, 4, 4, 5, 4, 4, 3, 4, 4, 1, 3, 5, 4, 3, 3, 3, 1, 2, 2, 3, 3, 1, 3, 1, 3, 3, 3, 5, 3, 3, 4, 5, 0, 3, 0, 3, 0, 3, 4, 3, 4, 4, 3, 0, 3, 0, 2, 4, 3), - (0, 1, 0, 4, 0, 0, 0, 0, 0, 1, 4, 0, 4, 1, 4, 2, 4, 0, 3, 0, 1, 0, 1, 0, 0, 0, 0, 0, 2, 0, 3, 1, 1, 1, 0, 3, 0, 0, 0, 1, 2, 1, 0, 0, 1, 1, 1, 1, 0, 1, 0, 0, 0, 1, 0, 0, 3, 0, 0, 0, 0, 3, 2, 0, 2, 2, 0, 1, 0, 0, 0, 2, 3, 2, 3, 3, 0, 0, 0, 0, 2, 1, 0), - (0, 5, 1, 5, 0, 3, 0, 3, 0, 5, 4, 4, 5, 1, 5, 3, 3, 0, 4, 3, 4, 3, 5, 3, 4, 3, 3, 2, 4, 3, 4, 3, 3, 0, 3, 3, 1, 4, 4, 3, 4, 4, 4, 3, 4, 5, 5, 3, 2, 3, 1, 1, 3, 3, 1, 3, 1, 1, 3, 3, 2, 4, 5, 3, 3, 5, 0, 4, 0, 3, 0, 4, 4, 3, 5, 3, 3, 0, 3, 4, 0, 4, 3), - (0, 5, 0, 5, 0, 3, 0, 2, 0, 4, 4, 3, 5, 2, 4, 3, 3, 3, 4, 4, 4, 3, 5, 3, 5, 3, 3, 1, 4, 0, 4, 3, 3, 0, 3, 3, 0, 4, 4, 4, 4, 5, 4, 3, 3, 5, 5, 3, 2, 3, 1, 2, 3, 2, 0, 1, 0, 0, 3, 2, 2, 4, 4, 3, 1, 5, 0, 4, 0, 3, 0, 4, 3, 1, 3, 2, 1, 0, 3, 3, 0, 3, 3), - (0, 4, 0, 5, 0, 5, 0, 4, 0, 4, 5, 5, 5, 3, 4, 3, 3, 2, 5, 4, 4, 3, 5, 3, 5, 3, 4, 0, 4, 3, 4, 4, 3, 2, 4, 4, 3, 4, 5, 4, 4, 5, 5, 0, 3, 5, 5, 4, 1, 3, 3, 2, 3, 3, 1, 3, 1, 0, 4, 3, 1, 4, 4, 3, 4, 5, 0, 4, 0, 2, 0, 4, 3, 4, 4, 3, 3, 0, 4, 0, 0, 5, 5), - (0, 4, 0, 4, 0, 5, 0, 1, 1, 3, 3, 4, 4, 3, 4, 1, 3, 0, 5, 1, 3, 0, 3, 1, 3, 1, 1, 0, 3, 0, 3, 3, 4, 0, 4, 3, 0, 4, 4, 4, 3, 4, 4, 0, 3, 5, 4, 1, 0, 3, 0, 0, 2, 3, 0, 3, 1, 0, 3, 1, 0, 3, 2, 1, 3, 5, 0, 3, 0, 1, 0, 3, 2, 3, 3, 4, 4, 0, 2, 2, 0, 4, 4), - (2, 4, 0, 5, 0, 4, 0, 3, 0, 4, 5, 5, 4, 3, 5, 3, 5, 3, 5, 3, 5, 2, 5, 3, 4, 3, 3, 4, 3, 4, 5, 3, 2, 1, 5, 4, 3, 2, 3, 4, 5, 3, 4, 1, 2, 5, 4, 3, 0, 3, 3, 0, 3, 2, 0, 2, 3, 0, 4, 1, 0, 3, 4, 3, 3, 5, 0, 3, 0, 1, 0, 4, 5, 5, 5, 4, 3, 0, 4, 2, 0, 3, 5), - (0, 5, 0, 4, 0, 4, 0, 2, 0, 5, 4, 3, 4, 3, 4, 3, 3, 3, 4, 3, 4, 2, 5, 3, 5, 3, 4, 1, 4, 3, 4, 4, 4, 0, 3, 5, 0, 4, 4, 4, 4, 5, 3, 1, 3, 4, 5, 3, 3, 3, 3, 3, 3, 3, 0, 2, 2, 0, 3, 3, 2, 4, 3, 3, 3, 5, 3, 4, 1, 3, 3, 5, 3, 2, 0, 0, 0, 0, 4, 3, 1, 3, 3), - (0, 1, 0, 3, 0, 3, 0, 1, 0, 1, 3, 3, 3, 2, 3, 3, 3, 0, 3, 0, 0, 0, 3, 1, 3, 0, 0, 0, 2, 2, 2, 3, 0, 0, 3, 2, 0, 1, 2, 4, 1, 3, 3, 0, 0, 3, 3, 3, 0, 1, 0, 0, 2, 1, 0, 0, 3, 0, 3, 1, 0, 3, 0, 0, 1, 3, 0, 2, 0, 1, 0, 3, 3, 1, 3, 3, 0, 0, 1, 1, 0, 3, 3), - (0, 2, 0, 3, 0, 2, 1, 4, 0, 2, 2, 3, 1, 1, 3, 1, 1, 0, 2, 0, 3, 1, 2, 3, 1, 3, 0, 0, 1, 0, 4, 3, 2, 3, 3, 3, 1, 4, 2, 3, 3, 3, 3, 1, 0, 3, 1, 4, 0, 1, 1, 0, 1, 2, 0, 1, 1, 0, 1, 1, 0, 3, 1, 3, 2, 2, 0, 1, 0, 0, 0, 2, 3, 3, 3, 1, 0, 0, 0, 0, 0, 2, 3), - (0, 5, 0, 4, 0, 5, 0, 2, 0, 4, 5, 5, 3, 3, 4, 3, 3, 1, 5, 4, 4, 2, 4, 4, 4, 3, 4, 2, 4, 3, 5, 5, 4, 3, 3, 4, 3, 3, 5, 5, 4, 5, 5, 1, 3, 4, 5, 3, 1, 4, 3, 1, 3, 3, 0, 3, 3, 1, 4, 3, 1, 4, 5, 3, 3, 5, 0, 4, 0, 3, 0, 5, 3, 3, 1, 4, 3, 0, 4, 0, 1, 5, 3), - (0, 5, 0, 5, 0, 4, 0, 2, 0, 4, 4, 3, 4, 3, 3, 3, 3, 3, 5, 4, 4, 4, 4, 4, 4, 5, 3, 3, 5, 2, 4, 4, 4, 3, 4, 4, 3, 3, 4, 4, 5, 5, 3, 3, 4, 3, 4, 3, 3, 4, 3, 3, 3, 3, 1, 2, 2, 1, 4, 3, 3, 5, 4, 4, 3, 4, 0, 4, 0, 3, 0, 4, 4, 4, 4, 4, 1, 0, 4, 2, 0, 2, 4), - (0, 4, 0, 4, 0, 3, 0, 1, 0, 3, 5, 2, 3, 0, 3, 0, 2, 1, 4, 2, 3, 3, 4, 1, 4, 3, 3, 2, 4, 1, 3, 3, 3, 0, 3, 3, 0, 0, 3, 3, 3, 5, 3, 3, 3, 3, 3, 2, 0, 2, 0, 0, 2, 0, 0, 2, 0, 0, 1, 0, 0, 3, 1, 2, 2, 3, 0, 3, 0, 2, 0, 4, 4, 3, 3, 4, 1, 0, 3, 0, 0, 2, 4), - (0, 0, 0, 4, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 2, 0, 0, 0, 0, 0, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, 3, 1, 3, 0, 3, 2, 0, 0, 0, 1, 0, 3, 2, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 3, 4, 0, 2, 0, 0, 0, 0, 0, 0, 2), - (0, 2, 1, 3, 0, 2, 0, 2, 0, 3, 3, 3, 3, 1, 3, 1, 3, 3, 3, 3, 3, 3, 4, 2, 2, 1, 2, 1, 4, 0, 4, 3, 1, 3, 3, 3, 2, 4, 3, 5, 4, 3, 3, 3, 3, 3, 3, 3, 0, 1, 3, 0, 2, 0, 0, 1, 0, 0, 1, 0, 0, 4, 2, 0, 2, 3, 0, 3, 3, 0, 3, 3, 4, 2, 3, 1, 4, 0, 1, 2, 0, 2, 3), - (0, 3, 0, 3, 0, 1, 0, 3, 0, 2, 3, 3, 3, 0, 3, 1, 2, 0, 3, 3, 2, 3, 3, 2, 3, 2, 3, 1, 3, 0, 4, 3, 2, 0, 3, 3, 1, 4, 3, 3, 2, 3, 4, 3, 1, 3, 3, 1, 1, 0, 1, 1, 0, 1, 0, 1, 0, 1, 0, 0, 0, 4, 1, 1, 0, 3, 0, 3, 1, 0, 2, 3, 3, 3, 3, 3, 1, 0, 0, 2, 0, 3, 3), - (0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 3, 0, 2, 0, 3, 0, 0, 0, 0, 0, 0, 0, 3, 0, 0, 0, 0, 0, 0, 0, 3, 0, 3, 0, 3, 1, 0, 1, 0, 1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 3, 0, 2, 0, 2, 3, 0, 0, 0, 0, 0, 0, 0, 0, 3), - (0, 2, 0, 3, 1, 3, 0, 3, 0, 2, 3, 3, 3, 1, 3, 1, 3, 1, 3, 1, 3, 3, 3, 1, 3, 0, 2, 3, 1, 1, 4, 3, 3, 2, 3, 3, 1, 2, 2, 4, 1, 3, 3, 0, 1, 4, 2, 3, 0, 1, 3, 0, 3, 0, 0, 1, 3, 0, 2, 0, 0, 3, 3, 2, 1, 3, 0, 3, 0, 2, 0, 3, 4, 4, 4, 3, 1, 0, 3, 0, 0, 3, 3), - (0, 2, 0, 1, 0, 2, 0, 0, 0, 1, 3, 2, 2, 1, 3, 0, 1, 1, 3, 0, 3, 2, 3, 1, 2, 0, 2, 0, 1, 1, 3, 3, 3, 0, 3, 3, 1, 1, 2, 3, 2, 3, 3, 1, 2, 3, 2, 0, 0, 1, 0, 0, 0, 0, 0, 0, 3, 0, 1, 0, 0, 2, 1, 2, 1, 3, 0, 3, 0, 0, 0, 3, 4, 4, 4, 3, 2, 0, 2, 0, 0, 2, 4), - (0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 2, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 3, 1, 0, 0, 0, 0, 0, 0, 0, 3), - (0, 3, 0, 3, 0, 2, 0, 3, 0, 3, 3, 3, 2, 3, 2, 2, 2, 0, 3, 1, 3, 3, 3, 2, 3, 3, 0, 0, 3, 0, 3, 2, 2, 0, 2, 3, 1, 4, 3, 4, 3, 3, 2, 3, 1, 5, 4, 4, 0, 3, 1, 2, 1, 3, 0, 3, 1, 1, 2, 0, 2, 3, 1, 3, 1, 3, 0, 3, 0, 1, 0, 3, 3, 4, 4, 2, 1, 0, 2, 1, 0, 2, 4), - (0, 1, 0, 3, 0, 1, 0, 2, 0, 1, 4, 2, 5, 1, 4, 0, 2, 0, 2, 1, 3, 1, 4, 0, 2, 1, 0, 0, 2, 1, 4, 1, 1, 0, 3, 3, 0, 5, 1, 3, 2, 3, 3, 1, 0, 3, 2, 3, 0, 1, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 4, 0, 1, 0, 3, 0, 2, 0, 1, 0, 3, 3, 3, 4, 3, 3, 0, 0, 0, 0, 2, 3), - (0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 2, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 3, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 1, 0, 0, 1, 0, 0, 0, 0, 0, 3), - (0, 1, 0, 3, 0, 4, 0, 3, 0, 2, 4, 3, 1, 0, 3, 2, 2, 1, 3, 1, 2, 2, 3, 1, 1, 1, 2, 1, 3, 0, 1, 2, 0, 1, 3, 2, 1, 3, 0, 5, 5, 1, 0, 0, 1, 3, 2, 1, 0, 3, 0, 0, 1, 0, 0, 0, 0, 0, 3, 4, 0, 1, 1, 1, 3, 2, 0, 2, 0, 1, 0, 2, 3, 3, 1, 2, 3, 0, 1, 0, 1, 0, 4), - (0, 0, 0, 1, 0, 3, 0, 3, 0, 2, 2, 1, 0, 0, 4, 0, 3, 0, 3, 1, 3, 0, 3, 0, 3, 0, 1, 0, 3, 0, 3, 1, 3, 0, 3, 3, 0, 0, 1, 2, 1, 1, 1, 0, 1, 2, 0, 0, 0, 1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 2, 2, 1, 2, 0, 0, 2, 0, 0, 0, 0, 2, 3, 3, 3, 3, 0, 0, 0, 0, 1, 4), - (0, 0, 0, 3, 0, 3, 0, 0, 0, 0, 3, 1, 1, 0, 3, 0, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 3, 0, 2, 0, 2, 3, 0, 0, 2, 2, 3, 1, 2, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 3, 0, 0, 2, 0, 0, 0, 0, 2, 3), - (2, 4, 0, 5, 0, 5, 0, 4, 0, 3, 4, 3, 3, 3, 4, 3, 3, 3, 4, 3, 4, 4, 5, 4, 5, 5, 5, 2, 3, 0, 5, 5, 4, 1, 5, 4, 3, 1, 5, 4, 3, 4, 4, 3, 3, 4, 3, 3, 0, 3, 2, 0, 2, 3, 0, 3, 0, 0, 3, 3, 0, 5, 3, 2, 3, 3, 0, 3, 0, 3, 0, 3, 4, 5, 4, 5, 3, 0, 4, 3, 0, 3, 4), - (0, 3, 0, 3, 0, 3, 0, 3, 0, 3, 3, 4, 3, 2, 3, 2, 3, 0, 4, 3, 3, 3, 3, 3, 3, 3, 3, 0, 3, 2, 4, 3, 3, 1, 3, 4, 3, 4, 4, 4, 3, 4, 4, 3, 2, 4, 4, 1, 0, 2, 0, 0, 1, 1, 0, 2, 0, 0, 3, 1, 0, 5, 3, 2, 1, 3, 0, 3, 0, 1, 2, 4, 3, 2, 4, 3, 3, 0, 3, 2, 0, 4, 4), - (0, 3, 0, 3, 0, 1, 0, 0, 0, 1, 4, 3, 3, 2, 3, 1, 3, 1, 4, 2, 3, 2, 4, 2, 3, 4, 3, 0, 2, 2, 3, 3, 3, 0, 3, 3, 3, 0, 3, 4, 1, 3, 3, 0, 3, 4, 3, 3, 0, 1, 1, 0, 1, 0, 0, 0, 4, 0, 3, 0, 0, 3, 1, 2, 1, 3, 0, 4, 0, 1, 0, 4, 3, 3, 4, 3, 3, 0, 2, 0, 0, 3, 3), - (0, 3, 0, 4, 0, 1, 0, 3, 0, 3, 4, 3, 3, 0, 3, 3, 3, 1, 3, 1, 3, 3, 4, 3, 3, 3, 0, 0, 3, 1, 5, 3, 3, 1, 3, 3, 2, 5, 4, 3, 3, 4, 5, 3, 2, 5, 3, 4, 0, 1, 0, 0, 0, 0, 0, 2, 0, 0, 1, 1, 0, 4, 2, 2, 1, 3, 0, 3, 0, 2, 0, 4, 4, 3, 5, 3, 2, 0, 1, 1, 0, 3, 4), - (0, 5, 0, 4, 0, 5, 0, 2, 0, 4, 4, 3, 3, 2, 3, 3, 3, 1, 4, 3, 4, 1, 5, 3, 4, 3, 4, 0, 4, 2, 4, 3, 4, 1, 5, 4, 0, 4, 4, 4, 4, 5, 4, 1, 3, 5, 4, 2, 1, 4, 1, 1, 3, 2, 0, 3, 1, 0, 3, 2, 1, 4, 3, 3, 3, 4, 0, 4, 0, 3, 0, 4, 4, 4, 3, 3, 3, 0, 4, 2, 0, 3, 4), - (1, 4, 0, 4, 0, 3, 0, 1, 0, 3, 3, 3, 1, 1, 3, 3, 2, 2, 3, 3, 1, 0, 3, 2, 2, 1, 2, 0, 3, 1, 2, 1, 2, 0, 3, 2, 0, 2, 2, 3, 3, 4, 3, 0, 3, 3, 1, 2, 0, 1, 1, 3, 1, 2, 0, 0, 3, 0, 1, 1, 0, 3, 2, 2, 3, 3, 0, 3, 0, 0, 0, 2, 3, 3, 4, 3, 3, 0, 1, 0, 0, 1, 4), - (0, 4, 0, 4, 0, 4, 0, 0, 0, 3, 4, 4, 3, 1, 4, 2, 3, 2, 3, 3, 3, 1, 4, 3, 4, 0, 3, 0, 4, 2, 3, 3, 2, 2, 5, 4, 2, 1, 3, 4, 3, 4, 3, 1, 3, 3, 4, 2, 0, 2, 1, 0, 3, 3, 0, 0, 2, 0, 3, 1, 0, 4, 4, 3, 4, 3, 0, 4, 0, 1, 0, 2, 4, 4, 4, 4, 4, 0, 3, 2, 0, 3, 3), - (0, 0, 0, 1, 0, 4, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 3, 2, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 2), - (0, 2, 0, 3, 0, 4, 0, 4, 0, 1, 3, 3, 3, 0, 4, 0, 2, 1, 2, 1, 1, 1, 2, 0, 3, 1, 1, 0, 1, 0, 3, 1, 0, 0, 3, 3, 2, 0, 1, 1, 0, 0, 0, 0, 0, 1, 0, 2, 0, 2, 2, 0, 3, 1, 0, 0, 1, 0, 1, 1, 0, 1, 2, 0, 3, 0, 0, 0, 0, 1, 0, 0, 3, 3, 4, 3, 1, 0, 1, 0, 3, 0, 2), - (0, 0, 0, 3, 0, 5, 0, 0, 0, 0, 1, 0, 2, 0, 3, 1, 0, 1, 3, 0, 0, 0, 2, 0, 0, 0, 1, 0, 0, 0, 1, 1, 0, 0, 4, 0, 0, 0, 2, 3, 0, 1, 4, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 3, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 3, 0, 0, 0, 0, 0, 3), - (0, 2, 0, 5, 0, 5, 0, 1, 0, 2, 4, 3, 3, 2, 5, 1, 3, 2, 3, 3, 3, 0, 4, 1, 2, 0, 3, 0, 4, 0, 2, 2, 1, 1, 5, 3, 0, 0, 1, 4, 2, 3, 2, 0, 3, 3, 3, 2, 0, 2, 4, 1, 1, 2, 0, 1, 1, 0, 3, 1, 0, 1, 3, 1, 2, 3, 0, 2, 0, 0, 0, 1, 3, 5, 4, 4, 4, 0, 3, 0, 0, 1, 3), - (0, 4, 0, 5, 0, 4, 0, 4, 0, 4, 5, 4, 3, 3, 4, 3, 3, 3, 4, 3, 4, 4, 5, 3, 4, 5, 4, 2, 4, 2, 3, 4, 3, 1, 4, 4, 1, 3, 5, 4, 4, 5, 5, 4, 4, 5, 5, 5, 2, 3, 3, 1, 4, 3, 1, 3, 3, 0, 3, 3, 1, 4, 3, 4, 4, 4, 0, 3, 0, 4, 0, 3, 3, 4, 4, 5, 0, 0, 4, 3, 0, 4, 5), - (0, 4, 0, 4, 0, 3, 0, 3, 0, 3, 4, 4, 4, 3, 3, 2, 4, 3, 4, 3, 4, 3, 5, 3, 4, 3, 2, 1, 4, 2, 4, 4, 3, 1, 3, 4, 2, 4, 5, 5, 3, 4, 5, 4, 1, 5, 4, 3, 0, 3, 2, 2, 3, 2, 1, 3, 1, 0, 3, 3, 3, 5, 3, 3, 3, 5, 4, 4, 2, 3, 3, 4, 3, 3, 3, 2, 1, 0, 3, 2, 1, 4, 3), - (0, 4, 0, 5, 0, 4, 0, 3, 0, 3, 5, 5, 3, 2, 4, 3, 4, 0, 5, 4, 4, 1, 4, 4, 4, 3, 3, 3, 4, 3, 5, 5, 2, 3, 3, 4, 1, 2, 5, 5, 3, 5, 5, 2, 3, 5, 5, 4, 0, 3, 2, 0, 3, 3, 1, 1, 5, 1, 4, 1, 0, 4, 3, 2, 3, 5, 0, 4, 0, 3, 0, 5, 4, 3, 4, 3, 0, 0, 4, 1, 0, 4, 4), - (1, 3, 0, 4, 0, 2, 0, 2, 0, 2, 5, 5, 3, 3, 3, 3, 3, 0, 4, 2, 3, 4, 4, 4, 3, 4, 0, 0, 3, 4, 5, 4, 3, 3, 3, 3, 2, 5, 5, 4, 5, 5, 5, 4, 3, 5, 5, 5, 1, 3, 1, 0, 1, 0, 0, 3, 2, 0, 4, 2, 0, 5, 2, 3, 2, 4, 1, 3, 0, 3, 0, 4, 5, 4, 5, 4, 3, 0, 4, 2, 0, 5, 4), - (0, 3, 0, 4, 0, 5, 0, 3, 0, 3, 4, 4, 3, 2, 3, 2, 3, 3, 3, 3, 3, 2, 4, 3, 3, 2, 2, 0, 3, 3, 3, 3, 3, 1, 3, 3, 3, 0, 4, 4, 3, 4, 4, 1, 1, 4, 4, 2, 0, 3, 1, 0, 1, 1, 0, 4, 1, 0, 2, 3, 1, 3, 3, 1, 3, 4, 0, 3, 0, 1, 0, 3, 1, 3, 0, 0, 1, 0, 2, 0, 0, 4, 4), - (0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0), - (0, 3, 0, 3, 0, 2, 0, 3, 0, 1, 5, 4, 3, 3, 3, 1, 4, 2, 1, 2, 3, 4, 4, 2, 4, 4, 5, 0, 3, 1, 4, 3, 4, 0, 4, 3, 3, 3, 2, 3, 2, 5, 3, 4, 3, 2, 2, 3, 0, 0, 3, 0, 2, 1, 0, 1, 2, 0, 0, 0, 0, 2, 1, 1, 3, 1, 0, 2, 0, 4, 0, 3, 4, 4, 4, 5, 2, 0, 2, 0, 0, 1, 3), - (0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 1, 1, 0, 0, 0, 4, 2, 1, 1, 0, 1, 0, 3, 2, 0, 0, 3, 1, 1, 1, 2, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 3, 0, 1, 0, 0, 0, 2, 0, 0, 0, 1, 4, 0, 4, 2, 1, 0, 0, 0, 0, 0, 1), - (0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 3, 1, 0, 0, 0, 2, 0, 2, 1, 0, 0, 1, 2, 1, 0, 1, 1, 0, 0, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 3, 1, 0, 0, 0, 0, 0, 1, 0, 0, 2, 1, 0, 0, 0, 0, 0, 0, 0, 0, 2), - (0, 4, 0, 4, 0, 4, 0, 3, 0, 4, 4, 3, 4, 2, 4, 3, 2, 0, 4, 4, 4, 3, 5, 3, 5, 3, 3, 2, 4, 2, 4, 3, 4, 3, 1, 4, 0, 2, 3, 4, 4, 4, 3, 3, 3, 4, 4, 4, 3, 4, 1, 3, 4, 3, 2, 1, 2, 1, 3, 3, 3, 4, 4, 3, 3, 5, 0, 4, 0, 3, 0, 4, 3, 3, 3, 2, 1, 0, 3, 0, 0, 3, 3), - (0, 4, 0, 3, 0, 3, 0, 3, 0, 3, 5, 5, 3, 3, 3, 3, 4, 3, 4, 3, 3, 3, 4, 4, 4, 3, 3, 3, 3, 4, 3, 5, 3, 3, 1, 3, 2, 4, 5, 5, 5, 5, 4, 3, 4, 5, 5, 3, 2, 2, 3, 3, 3, 3, 2, 3, 3, 1, 2, 3, 2, 4, 3, 3, 3, 4, 0, 4, 0, 2, 0, 4, 3, 2, 2, 1, 2, 0, 3, 0, 0, 4, 1), -) -# fmt: on - - -class JapaneseContextAnalysis: - NUM_OF_CATEGORY = 6 - DONT_KNOW = -1 - ENOUGH_REL_THRESHOLD = 100 - MAX_REL_THRESHOLD = 1000 - MINIMUM_DATA_THRESHOLD = 4 - - def __init__(self) -> None: - self._total_rel = 0 - self._rel_sample: List[int] = [] - self._need_to_skip_char_num = 0 - self._last_char_order = -1 - self._done = False - self.reset() - - def reset(self) -> None: - self._total_rel = 0 # total sequence received - # category counters, each integer counts sequence in its category - self._rel_sample = [0] * self.NUM_OF_CATEGORY - # if last byte in current buffer is not the last byte of a character, - # we need to know how many bytes to skip in next buffer - self._need_to_skip_char_num = 0 - self._last_char_order = -1 # The order of previous char - # If this flag is set to True, detection is done and conclusion has - # been made - self._done = False - - def feed(self, byte_str: Union[bytes, bytearray], num_bytes: int) -> None: - if self._done: - return - - # The buffer we got is byte oriented, and a character may span in more than one - # buffers. In case the last one or two byte in last buffer is not - # complete, we record how many byte needed to complete that character - # and skip these bytes here. We can choose to record those bytes as - # well and analyse the character once it is complete, but since a - # character will not make much difference, by simply skipping - # this character will simply our logic and improve performance. - i = self._need_to_skip_char_num - while i < num_bytes: - order, char_len = self.get_order(byte_str[i : i + 2]) - i += char_len - if i > num_bytes: - self._need_to_skip_char_num = i - num_bytes - self._last_char_order = -1 - else: - if (order != -1) and (self._last_char_order != -1): - self._total_rel += 1 - if self._total_rel > self.MAX_REL_THRESHOLD: - self._done = True - break - self._rel_sample[ - jp2_char_context[self._last_char_order][order] - ] += 1 - self._last_char_order = order - - def got_enough_data(self) -> bool: - return self._total_rel > self.ENOUGH_REL_THRESHOLD - - def get_confidence(self) -> float: - # This is just one way to calculate confidence. It works well for me. - if self._total_rel > self.MINIMUM_DATA_THRESHOLD: - return (self._total_rel - self._rel_sample[0]) / self._total_rel - return self.DONT_KNOW - - def get_order(self, _: Union[bytes, bytearray]) -> Tuple[int, int]: - return -1, 1 - - -class SJISContextAnalysis(JapaneseContextAnalysis): - def __init__(self) -> None: - super().__init__() - self._charset_name = "SHIFT_JIS" - - @property - def charset_name(self) -> str: - return self._charset_name - - def get_order(self, byte_str: Union[bytes, bytearray]) -> Tuple[int, int]: - if not byte_str: - return -1, 1 - # find out current char's byte length - first_char = byte_str[0] - if (0x81 <= first_char <= 0x9F) or (0xE0 <= first_char <= 0xFC): - char_len = 2 - if (first_char == 0x87) or (0xFA <= first_char <= 0xFC): - self._charset_name = "CP932" - else: - char_len = 1 - - # return its order if it is hiragana - if len(byte_str) > 1: - second_char = byte_str[1] - if (first_char == 202) and (0x9F <= second_char <= 0xF1): - return second_char - 0x9F, char_len - - return -1, char_len - - -class EUCJPContextAnalysis(JapaneseContextAnalysis): - def get_order(self, byte_str: Union[bytes, bytearray]) -> Tuple[int, int]: - if not byte_str: - return -1, 1 - # find out current char's byte length - first_char = byte_str[0] - if (first_char == 0x8E) or (0xA1 <= first_char <= 0xFE): - char_len = 2 - elif first_char == 0x8F: - char_len = 3 - else: - char_len = 1 - - # return its order if it is hiragana - if len(byte_str) > 1: - second_char = byte_str[1] - if (first_char == 0xA4) and (0xA1 <= second_char <= 0xF3): - return second_char - 0xA1, char_len - - return -1, char_len diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/langbulgarianmodel.py b/venv/lib/python3.11/site-packages/pip/_vendor/chardet/langbulgarianmodel.py deleted file mode 100644 index 9946682..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/langbulgarianmodel.py +++ /dev/null @@ -1,4649 +0,0 @@ -from pip._vendor.chardet.sbcharsetprober import SingleByteCharSetModel - -# 3: Positive -# 2: Likely -# 1: Unlikely -# 0: Negative - -BULGARIAN_LANG_MODEL = { - 63: { # 'e' - 63: 1, # 'e' - 45: 0, # '\xad' - 31: 0, # 'А' - 32: 0, # 'Б' - 35: 0, # 'В' - 43: 0, # 'Г' - 37: 0, # 'Д' - 44: 0, # 'Е' - 55: 0, # 'Ж' - 47: 0, # 'З' - 40: 0, # 'И' - 59: 0, # 'Й' - 33: 0, # 'К' - 46: 0, # 'Л' - 38: 0, # 'М' - 36: 0, # 'Н' - 41: 0, # 'О' - 30: 0, # 'П' - 39: 0, # 'Р' - 28: 0, # 'С' - 34: 0, # 'Т' - 51: 0, # 'У' - 48: 0, # 'Ф' - 49: 0, # 'Х' - 53: 0, # 'Ц' - 50: 0, # 'Ч' - 54: 0, # 'Ш' - 57: 0, # 'Щ' - 61: 0, # 'Ъ' - 60: 0, # 'Ю' - 56: 0, # 'Я' - 1: 0, # 'а' - 18: 1, # 'б' - 9: 1, # 'в' - 20: 1, # 'г' - 11: 1, # 'д' - 3: 1, # 'е' - 23: 1, # 'ж' - 15: 1, # 'з' - 2: 0, # 'и' - 26: 1, # 'й' - 12: 1, # 'к' - 10: 1, # 'л' - 14: 1, # 'м' - 6: 1, # 'н' - 4: 1, # 'о' - 13: 1, # 'п' - 7: 1, # 'р' - 8: 1, # 'с' - 5: 1, # 'т' - 19: 0, # 'у' - 29: 1, # 'ф' - 25: 1, # 'х' - 22: 0, # 'ц' - 21: 1, # 'ч' - 27: 1, # 'ш' - 24: 1, # 'щ' - 17: 0, # 'ъ' - 52: 0, # 'ь' - 42: 0, # 'ю' - 16: 1, # 'я' - 58: 0, # 'є' - 62: 0, # '№' - }, - 45: { # '\xad' - 63: 0, # 'e' - 45: 0, # '\xad' - 31: 0, # 'А' - 32: 1, # 'Б' - 35: 1, # 'В' - 43: 0, # 'Г' - 37: 1, # 'Д' - 44: 0, # 'Е' - 55: 0, # 'Ж' - 47: 0, # 'З' - 40: 1, # 'И' - 59: 0, # 'Й' - 33: 1, # 'К' - 46: 0, # 'Л' - 38: 1, # 'М' - 36: 0, # 'Н' - 41: 1, # 'О' - 30: 1, # 'П' - 39: 1, # 'Р' - 28: 1, # 'С' - 34: 0, # 'Т' - 51: 0, # 'У' - 48: 0, # 'Ф' - 49: 1, # 'Х' - 53: 0, # 'Ц' - 50: 0, # 'Ч' - 54: 0, # 'Ш' - 57: 0, # 'Щ' - 61: 0, # 'Ъ' - 60: 0, # 'Ю' - 56: 0, # 'Я' - 1: 0, # 'а' - 18: 0, # 'б' - 9: 0, # 'в' - 20: 0, # 'г' - 11: 0, # 'д' - 3: 0, # 'е' - 23: 0, # 'ж' - 15: 0, # 'з' - 2: 0, # 'и' - 26: 0, # 'й' - 12: 0, # 'к' - 10: 0, # 'л' - 14: 0, # 'м' - 6: 0, # 'н' - 4: 0, # 'о' - 13: 0, # 'п' - 7: 0, # 'р' - 8: 0, # 'с' - 5: 0, # 'т' - 19: 0, # 'у' - 29: 0, # 'ф' - 25: 0, # 'х' - 22: 0, # 'ц' - 21: 0, # 'ч' - 27: 0, # 'ш' - 24: 0, # 'щ' - 17: 0, # 'ъ' - 52: 0, # 'ь' - 42: 0, # 'ю' - 16: 0, # 'я' - 58: 0, # 'є' - 62: 0, # '№' - }, - 31: { # 'А' - 63: 0, # 'e' - 45: 1, # '\xad' - 31: 1, # 'А' - 32: 1, # 'Б' - 35: 2, # 'В' - 43: 1, # 'Г' - 37: 2, # 'Д' - 44: 2, # 'Е' - 55: 1, # 'Ж' - 47: 2, # 'З' - 40: 1, # 'И' - 59: 1, # 'Й' - 33: 1, # 'К' - 46: 2, # 'Л' - 38: 1, # 'М' - 36: 2, # 'Н' - 41: 1, # 'О' - 30: 2, # 'П' - 39: 2, # 'Р' - 28: 2, # 'С' - 34: 2, # 'Т' - 51: 1, # 'У' - 48: 2, # 'Ф' - 49: 1, # 'Х' - 53: 1, # 'Ц' - 50: 1, # 'Ч' - 54: 1, # 'Ш' - 57: 2, # 'Щ' - 61: 0, # 'Ъ' - 60: 0, # 'Ю' - 56: 1, # 'Я' - 1: 1, # 'а' - 18: 2, # 'б' - 9: 2, # 'в' - 20: 2, # 'г' - 11: 2, # 'д' - 3: 1, # 'е' - 23: 1, # 'ж' - 15: 2, # 'з' - 2: 0, # 'и' - 26: 2, # 'й' - 12: 2, # 'к' - 10: 3, # 'л' - 14: 2, # 'м' - 6: 3, # 'н' - 4: 0, # 'о' - 13: 2, # 'п' - 7: 2, # 'р' - 8: 2, # 'с' - 5: 2, # 'т' - 19: 1, # 'у' - 29: 2, # 'ф' - 25: 1, # 'х' - 22: 1, # 'ц' - 21: 1, # 'ч' - 27: 1, # 'ш' - 24: 0, # 'щ' - 17: 0, # 'ъ' - 52: 0, # 'ь' - 42: 0, # 'ю' - 16: 1, # 'я' - 58: 0, # 'є' - 62: 0, # '№' - }, - 32: { # 'Б' - 63: 0, # 'e' - 45: 0, # '\xad' - 31: 2, # 'А' - 32: 2, # 'Б' - 35: 1, # 'В' - 43: 1, # 'Г' - 37: 2, # 'Д' - 44: 1, # 'Е' - 55: 1, # 'Ж' - 47: 2, # 'З' - 40: 1, # 'И' - 59: 0, # 'Й' - 33: 1, # 'К' - 46: 1, # 'Л' - 38: 1, # 'М' - 36: 2, # 'Н' - 41: 2, # 'О' - 30: 1, # 'П' - 39: 1, # 'Р' - 28: 2, # 'С' - 34: 2, # 'Т' - 51: 1, # 'У' - 48: 2, # 'Ф' - 49: 1, # 'Х' - 53: 1, # 'Ц' - 50: 1, # 'Ч' - 54: 0, # 'Ш' - 57: 1, # 'Щ' - 61: 2, # 'Ъ' - 60: 1, # 'Ю' - 56: 1, # 'Я' - 1: 3, # 'а' - 18: 0, # 'б' - 9: 0, # 'в' - 20: 0, # 'г' - 11: 1, # 'д' - 3: 3, # 'е' - 23: 0, # 'ж' - 15: 0, # 'з' - 2: 2, # 'и' - 26: 0, # 'й' - 12: 0, # 'к' - 10: 2, # 'л' - 14: 0, # 'м' - 6: 0, # 'н' - 4: 3, # 'о' - 13: 0, # 'п' - 7: 2, # 'р' - 8: 1, # 'с' - 5: 0, # 'т' - 19: 2, # 'у' - 29: 0, # 'ф' - 25: 1, # 'х' - 22: 0, # 'ц' - 21: 0, # 'ч' - 27: 0, # 'ш' - 24: 0, # 'щ' - 17: 3, # 'ъ' - 52: 1, # 'ь' - 42: 1, # 'ю' - 16: 2, # 'я' - 58: 0, # 'є' - 62: 0, # '№' - }, - 35: { # 'В' - 63: 0, # 'e' - 45: 0, # '\xad' - 31: 2, # 'А' - 32: 1, # 'Б' - 35: 1, # 'В' - 43: 0, # 'Г' - 37: 1, # 'Д' - 44: 2, # 'Е' - 55: 0, # 'Ж' - 47: 0, # 'З' - 40: 2, # 'И' - 59: 0, # 'Й' - 33: 1, # 'К' - 46: 1, # 'Л' - 38: 1, # 'М' - 36: 1, # 'Н' - 41: 1, # 'О' - 30: 1, # 'П' - 39: 2, # 'Р' - 28: 2, # 'С' - 34: 1, # 'Т' - 51: 1, # 'У' - 48: 2, # 'Ф' - 49: 0, # 'Х' - 53: 1, # 'Ц' - 50: 0, # 'Ч' - 54: 0, # 'Ш' - 57: 0, # 'Щ' - 61: 1, # 'Ъ' - 60: 1, # 'Ю' - 56: 2, # 'Я' - 1: 3, # 'а' - 18: 1, # 'б' - 9: 0, # 'в' - 20: 0, # 'г' - 11: 1, # 'д' - 3: 3, # 'е' - 23: 1, # 'ж' - 15: 2, # 'з' - 2: 3, # 'и' - 26: 0, # 'й' - 12: 1, # 'к' - 10: 2, # 'л' - 14: 1, # 'м' - 6: 2, # 'н' - 4: 2, # 'о' - 13: 1, # 'п' - 7: 2, # 'р' - 8: 2, # 'с' - 5: 2, # 'т' - 19: 1, # 'у' - 29: 0, # 'ф' - 25: 1, # 'х' - 22: 0, # 'ц' - 21: 2, # 'ч' - 27: 0, # 'ш' - 24: 0, # 'щ' - 17: 2, # 'ъ' - 52: 1, # 'ь' - 42: 1, # 'ю' - 16: 1, # 'я' - 58: 0, # 'є' - 62: 0, # '№' - }, - 43: { # 'Г' - 63: 0, # 'e' - 45: 0, # '\xad' - 31: 2, # 'А' - 32: 1, # 'Б' - 35: 0, # 'В' - 43: 0, # 'Г' - 37: 1, # 'Д' - 44: 2, # 'Е' - 55: 0, # 'Ж' - 47: 1, # 'З' - 40: 1, # 'И' - 59: 0, # 'Й' - 33: 1, # 'К' - 46: 1, # 'Л' - 38: 0, # 'М' - 36: 1, # 'Н' - 41: 1, # 'О' - 30: 0, # 'П' - 39: 1, # 'Р' - 28: 1, # 'С' - 34: 0, # 'Т' - 51: 1, # 'У' - 48: 1, # 'Ф' - 49: 0, # 'Х' - 53: 0, # 'Ц' - 50: 0, # 'Ч' - 54: 0, # 'Ш' - 57: 1, # 'Щ' - 61: 1, # 'Ъ' - 60: 0, # 'Ю' - 56: 0, # 'Я' - 1: 2, # 'а' - 18: 1, # 'б' - 9: 1, # 'в' - 20: 0, # 'г' - 11: 1, # 'д' - 3: 3, # 'е' - 23: 1, # 'ж' - 15: 0, # 'з' - 2: 2, # 'и' - 26: 0, # 'й' - 12: 1, # 'к' - 10: 2, # 'л' - 14: 1, # 'м' - 6: 1, # 'н' - 4: 2, # 'о' - 13: 0, # 'п' - 7: 2, # 'р' - 8: 0, # 'с' - 5: 0, # 'т' - 19: 2, # 'у' - 29: 0, # 'ф' - 25: 0, # 'х' - 22: 0, # 'ц' - 21: 0, # 'ч' - 27: 0, # 'ш' - 24: 1, # 'щ' - 17: 2, # 'ъ' - 52: 1, # 'ь' - 42: 1, # 'ю' - 16: 1, # 'я' - 58: 0, # 'є' - 62: 0, # '№' - }, - 37: { # 'Д' - 63: 0, # 'e' - 45: 0, # '\xad' - 31: 2, # 'А' - 32: 1, # 'Б' - 35: 2, # 'В' - 43: 1, # 'Г' - 37: 2, # 'Д' - 44: 2, # 'Е' - 55: 2, # 'Ж' - 47: 1, # 'З' - 40: 2, # 'И' - 59: 0, # 'Й' - 33: 1, # 'К' - 46: 1, # 'Л' - 38: 1, # 'М' - 36: 1, # 'Н' - 41: 2, # 'О' - 30: 2, # 'П' - 39: 1, # 'Р' - 28: 2, # 'С' - 34: 1, # 'Т' - 51: 1, # 'У' - 48: 1, # 'Ф' - 49: 0, # 'Х' - 53: 1, # 'Ц' - 50: 1, # 'Ч' - 54: 0, # 'Ш' - 57: 0, # 'Щ' - 61: 1, # 'Ъ' - 60: 1, # 'Ю' - 56: 1, # 'Я' - 1: 3, # 'а' - 18: 0, # 'б' - 9: 2, # 'в' - 20: 0, # 'г' - 11: 0, # 'д' - 3: 3, # 'е' - 23: 3, # 'ж' - 15: 1, # 'з' - 2: 3, # 'и' - 26: 0, # 'й' - 12: 0, # 'к' - 10: 1, # 'л' - 14: 1, # 'м' - 6: 2, # 'н' - 4: 3, # 'о' - 13: 0, # 'п' - 7: 2, # 'р' - 8: 0, # 'с' - 5: 0, # 'т' - 19: 2, # 'у' - 29: 0, # 'ф' - 25: 0, # 'х' - 22: 0, # 'ц' - 21: 0, # 'ч' - 27: 0, # 'ш' - 24: 0, # 'щ' - 17: 2, # 'ъ' - 52: 1, # 'ь' - 42: 2, # 'ю' - 16: 1, # 'я' - 58: 0, # 'є' - 62: 0, # '№' - }, - 44: { # 'Е' - 63: 0, # 'e' - 45: 0, # '\xad' - 31: 1, # 'А' - 32: 1, # 'Б' - 35: 2, # 'В' - 43: 1, # 'Г' - 37: 1, # 'Д' - 44: 1, # 'Е' - 55: 1, # 'Ж' - 47: 1, # 'З' - 40: 1, # 'И' - 59: 1, # 'Й' - 33: 2, # 'К' - 46: 2, # 'Л' - 38: 1, # 'М' - 36: 2, # 'Н' - 41: 2, # 'О' - 30: 1, # 'П' - 39: 2, # 'Р' - 28: 2, # 'С' - 34: 2, # 'Т' - 51: 1, # 'У' - 48: 2, # 'Ф' - 49: 1, # 'Х' - 53: 2, # 'Ц' - 50: 1, # 'Ч' - 54: 1, # 'Ш' - 57: 1, # 'Щ' - 61: 0, # 'Ъ' - 60: 0, # 'Ю' - 56: 1, # 'Я' - 1: 0, # 'а' - 18: 1, # 'б' - 9: 2, # 'в' - 20: 1, # 'г' - 11: 2, # 'д' - 3: 0, # 'е' - 23: 1, # 'ж' - 15: 1, # 'з' - 2: 0, # 'и' - 26: 1, # 'й' - 12: 2, # 'к' - 10: 2, # 'л' - 14: 2, # 'м' - 6: 2, # 'н' - 4: 0, # 'о' - 13: 1, # 'п' - 7: 2, # 'р' - 8: 2, # 'с' - 5: 1, # 'т' - 19: 1, # 'у' - 29: 1, # 'ф' - 25: 1, # 'х' - 22: 0, # 'ц' - 21: 1, # 'ч' - 27: 1, # 'ш' - 24: 1, # 'щ' - 17: 1, # 'ъ' - 52: 0, # 'ь' - 42: 1, # 'ю' - 16: 1, # 'я' - 58: 0, # 'є' - 62: 0, # '№' - }, - 55: { # 'Ж' - 63: 0, # 'e' - 45: 0, # '\xad' - 31: 1, # 'А' - 32: 0, # 'Б' - 35: 1, # 'В' - 43: 0, # 'Г' - 37: 1, # 'Д' - 44: 1, # 'Е' - 55: 0, # 'Ж' - 47: 0, # 'З' - 40: 1, # 'И' - 59: 0, # 'Й' - 33: 1, # 'К' - 46: 0, # 'Л' - 38: 0, # 'М' - 36: 1, # 'Н' - 41: 1, # 'О' - 30: 0, # 'П' - 39: 0, # 'Р' - 28: 0, # 'С' - 34: 0, # 'Т' - 51: 1, # 'У' - 48: 0, # 'Ф' - 49: 0, # 'Х' - 53: 0, # 'Ц' - 50: 0, # 'Ч' - 54: 0, # 'Ш' - 57: 0, # 'Щ' - 61: 0, # 'Ъ' - 60: 0, # 'Ю' - 56: 0, # 'Я' - 1: 2, # 'а' - 18: 0, # 'б' - 9: 0, # 'в' - 20: 0, # 'г' - 11: 1, # 'д' - 3: 2, # 'е' - 23: 0, # 'ж' - 15: 0, # 'з' - 2: 2, # 'и' - 26: 0, # 'й' - 12: 0, # 'к' - 10: 0, # 'л' - 14: 0, # 'м' - 6: 0, # 'н' - 4: 2, # 'о' - 13: 1, # 'п' - 7: 1, # 'р' - 8: 0, # 'с' - 5: 0, # 'т' - 19: 1, # 'у' - 29: 0, # 'ф' - 25: 0, # 'х' - 22: 0, # 'ц' - 21: 0, # 'ч' - 27: 0, # 'ш' - 24: 0, # 'щ' - 17: 1, # 'ъ' - 52: 1, # 'ь' - 42: 1, # 'ю' - 16: 0, # 'я' - 58: 0, # 'є' - 62: 0, # '№' - }, - 47: { # 'З' - 63: 0, # 'e' - 45: 0, # '\xad' - 31: 2, # 'А' - 32: 1, # 'Б' - 35: 1, # 'В' - 43: 1, # 'Г' - 37: 1, # 'Д' - 44: 1, # 'Е' - 55: 0, # 'Ж' - 47: 1, # 'З' - 40: 1, # 'И' - 59: 0, # 'Й' - 33: 1, # 'К' - 46: 1, # 'Л' - 38: 1, # 'М' - 36: 2, # 'Н' - 41: 1, # 'О' - 30: 1, # 'П' - 39: 1, # 'Р' - 28: 1, # 'С' - 34: 1, # 'Т' - 51: 1, # 'У' - 48: 0, # 'Ф' - 49: 1, # 'Х' - 53: 1, # 'Ц' - 50: 0, # 'Ч' - 54: 0, # 'Ш' - 57: 0, # 'Щ' - 61: 1, # 'Ъ' - 60: 0, # 'Ю' - 56: 1, # 'Я' - 1: 3, # 'а' - 18: 1, # 'б' - 9: 2, # 'в' - 20: 1, # 'г' - 11: 2, # 'д' - 3: 2, # 'е' - 23: 0, # 'ж' - 15: 0, # 'з' - 2: 1, # 'и' - 26: 0, # 'й' - 12: 0, # 'к' - 10: 2, # 'л' - 14: 1, # 'м' - 6: 1, # 'н' - 4: 1, # 'о' - 13: 0, # 'п' - 7: 1, # 'р' - 8: 0, # 'с' - 5: 0, # 'т' - 19: 1, # 'у' - 29: 0, # 'ф' - 25: 0, # 'х' - 22: 0, # 'ц' - 21: 0, # 'ч' - 27: 0, # 'ш' - 24: 0, # 'щ' - 17: 1, # 'ъ' - 52: 0, # 'ь' - 42: 1, # 'ю' - 16: 0, # 'я' - 58: 0, # 'є' - 62: 0, # '№' - }, - 40: { # 'И' - 63: 0, # 'e' - 45: 1, # '\xad' - 31: 1, # 'А' - 32: 1, # 'Б' - 35: 1, # 'В' - 43: 1, # 'Г' - 37: 1, # 'Д' - 44: 2, # 'Е' - 55: 1, # 'Ж' - 47: 2, # 'З' - 40: 1, # 'И' - 59: 1, # 'Й' - 33: 2, # 'К' - 46: 2, # 'Л' - 38: 2, # 'М' - 36: 2, # 'Н' - 41: 1, # 'О' - 30: 1, # 'П' - 39: 2, # 'Р' - 28: 2, # 'С' - 34: 2, # 'Т' - 51: 0, # 'У' - 48: 1, # 'Ф' - 49: 1, # 'Х' - 53: 1, # 'Ц' - 50: 1, # 'Ч' - 54: 1, # 'Ш' - 57: 1, # 'Щ' - 61: 0, # 'Ъ' - 60: 0, # 'Ю' - 56: 2, # 'Я' - 1: 1, # 'а' - 18: 1, # 'б' - 9: 3, # 'в' - 20: 2, # 'г' - 11: 1, # 'д' - 3: 1, # 'е' - 23: 0, # 'ж' - 15: 3, # 'з' - 2: 0, # 'и' - 26: 1, # 'й' - 12: 1, # 'к' - 10: 2, # 'л' - 14: 2, # 'м' - 6: 2, # 'н' - 4: 0, # 'о' - 13: 1, # 'п' - 7: 2, # 'р' - 8: 2, # 'с' - 5: 2, # 'т' - 19: 0, # 'у' - 29: 1, # 'ф' - 25: 1, # 'х' - 22: 1, # 'ц' - 21: 1, # 'ч' - 27: 1, # 'ш' - 24: 1, # 'щ' - 17: 0, # 'ъ' - 52: 0, # 'ь' - 42: 0, # 'ю' - 16: 0, # 'я' - 58: 0, # 'є' - 62: 0, # '№' - }, - 59: { # 'Й' - 63: 0, # 'e' - 45: 0, # '\xad' - 31: 0, # 'А' - 32: 0, # 'Б' - 35: 0, # 'В' - 43: 0, # 'Г' - 37: 1, # 'Д' - 44: 1, # 'Е' - 55: 0, # 'Ж' - 47: 0, # 'З' - 40: 0, # 'И' - 59: 0, # 'Й' - 33: 1, # 'К' - 46: 1, # 'Л' - 38: 1, # 'М' - 36: 1, # 'Н' - 41: 1, # 'О' - 30: 0, # 'П' - 39: 0, # 'Р' - 28: 1, # 'С' - 34: 1, # 'Т' - 51: 0, # 'У' - 48: 0, # 'Ф' - 49: 0, # 'Х' - 53: 0, # 'Ц' - 50: 1, # 'Ч' - 54: 0, # 'Ш' - 57: 0, # 'Щ' - 61: 0, # 'Ъ' - 60: 0, # 'Ю' - 56: 1, # 'Я' - 1: 0, # 'а' - 18: 0, # 'б' - 9: 0, # 'в' - 20: 0, # 'г' - 11: 0, # 'д' - 3: 1, # 'е' - 23: 0, # 'ж' - 15: 0, # 'з' - 2: 0, # 'и' - 26: 0, # 'й' - 12: 0, # 'к' - 10: 0, # 'л' - 14: 0, # 'м' - 6: 0, # 'н' - 4: 2, # 'о' - 13: 0, # 'п' - 7: 0, # 'р' - 8: 0, # 'с' - 5: 0, # 'т' - 19: 0, # 'у' - 29: 0, # 'ф' - 25: 0, # 'х' - 22: 0, # 'ц' - 21: 0, # 'ч' - 27: 0, # 'ш' - 24: 0, # 'щ' - 17: 1, # 'ъ' - 52: 0, # 'ь' - 42: 0, # 'ю' - 16: 0, # 'я' - 58: 0, # 'є' - 62: 0, # '№' - }, - 33: { # 'К' - 63: 0, # 'e' - 45: 1, # '\xad' - 31: 2, # 'А' - 32: 1, # 'Б' - 35: 1, # 'В' - 43: 1, # 'Г' - 37: 1, # 'Д' - 44: 1, # 'Е' - 55: 0, # 'Ж' - 47: 1, # 'З' - 40: 2, # 'И' - 59: 0, # 'Й' - 33: 1, # 'К' - 46: 1, # 'Л' - 38: 0, # 'М' - 36: 2, # 'Н' - 41: 2, # 'О' - 30: 2, # 'П' - 39: 1, # 'Р' - 28: 2, # 'С' - 34: 1, # 'Т' - 51: 1, # 'У' - 48: 1, # 'Ф' - 49: 1, # 'Х' - 53: 1, # 'Ц' - 50: 0, # 'Ч' - 54: 0, # 'Ш' - 57: 0, # 'Щ' - 61: 1, # 'Ъ' - 60: 1, # 'Ю' - 56: 0, # 'Я' - 1: 3, # 'а' - 18: 0, # 'б' - 9: 1, # 'в' - 20: 0, # 'г' - 11: 0, # 'д' - 3: 2, # 'е' - 23: 1, # 'ж' - 15: 0, # 'з' - 2: 2, # 'и' - 26: 0, # 'й' - 12: 0, # 'к' - 10: 2, # 'л' - 14: 1, # 'м' - 6: 2, # 'н' - 4: 3, # 'о' - 13: 0, # 'п' - 7: 3, # 'р' - 8: 1, # 'с' - 5: 0, # 'т' - 19: 2, # 'у' - 29: 0, # 'ф' - 25: 1, # 'х' - 22: 0, # 'ц' - 21: 0, # 'ч' - 27: 1, # 'ш' - 24: 0, # 'щ' - 17: 2, # 'ъ' - 52: 1, # 'ь' - 42: 2, # 'ю' - 16: 0, # 'я' - 58: 0, # 'є' - 62: 0, # '№' - }, - 46: { # 'Л' - 63: 1, # 'e' - 45: 0, # '\xad' - 31: 2, # 'А' - 32: 1, # 'Б' - 35: 1, # 'В' - 43: 2, # 'Г' - 37: 1, # 'Д' - 44: 2, # 'Е' - 55: 0, # 'Ж' - 47: 1, # 'З' - 40: 2, # 'И' - 59: 0, # 'Й' - 33: 1, # 'К' - 46: 1, # 'Л' - 38: 0, # 'М' - 36: 1, # 'Н' - 41: 2, # 'О' - 30: 1, # 'П' - 39: 0, # 'Р' - 28: 1, # 'С' - 34: 1, # 'Т' - 51: 1, # 'У' - 48: 0, # 'Ф' - 49: 1, # 'Х' - 53: 1, # 'Ц' - 50: 1, # 'Ч' - 54: 0, # 'Ш' - 57: 0, # 'Щ' - 61: 1, # 'Ъ' - 60: 1, # 'Ю' - 56: 1, # 'Я' - 1: 2, # 'а' - 18: 0, # 'б' - 9: 1, # 'в' - 20: 0, # 'г' - 11: 0, # 'д' - 3: 3, # 'е' - 23: 0, # 'ж' - 15: 0, # 'з' - 2: 2, # 'и' - 26: 0, # 'й' - 12: 0, # 'к' - 10: 0, # 'л' - 14: 0, # 'м' - 6: 0, # 'н' - 4: 2, # 'о' - 13: 0, # 'п' - 7: 0, # 'р' - 8: 0, # 'с' - 5: 0, # 'т' - 19: 2, # 'у' - 29: 0, # 'ф' - 25: 0, # 'х' - 22: 0, # 'ц' - 21: 0, # 'ч' - 27: 0, # 'ш' - 24: 0, # 'щ' - 17: 1, # 'ъ' - 52: 1, # 'ь' - 42: 2, # 'ю' - 16: 1, # 'я' - 58: 0, # 'є' - 62: 0, # '№' - }, - 38: { # 'М' - 63: 0, # 'e' - 45: 0, # '\xad' - 31: 2, # 'А' - 32: 1, # 'Б' - 35: 2, # 'В' - 43: 0, # 'Г' - 37: 1, # 'Д' - 44: 1, # 'Е' - 55: 0, # 'Ж' - 47: 1, # 'З' - 40: 2, # 'И' - 59: 0, # 'Й' - 33: 1, # 'К' - 46: 1, # 'Л' - 38: 1, # 'М' - 36: 1, # 'Н' - 41: 2, # 'О' - 30: 1, # 'П' - 39: 1, # 'Р' - 28: 2, # 'С' - 34: 1, # 'Т' - 51: 1, # 'У' - 48: 1, # 'Ф' - 49: 0, # 'Х' - 53: 1, # 'Ц' - 50: 0, # 'Ч' - 54: 0, # 'Ш' - 57: 0, # 'Щ' - 61: 1, # 'Ъ' - 60: 0, # 'Ю' - 56: 1, # 'Я' - 1: 3, # 'а' - 18: 0, # 'б' - 9: 0, # 'в' - 20: 0, # 'г' - 11: 0, # 'д' - 3: 3, # 'е' - 23: 0, # 'ж' - 15: 0, # 'з' - 2: 3, # 'и' - 26: 0, # 'й' - 12: 0, # 'к' - 10: 2, # 'л' - 14: 0, # 'м' - 6: 2, # 'н' - 4: 3, # 'о' - 13: 0, # 'п' - 7: 1, # 'р' - 8: 0, # 'с' - 5: 0, # 'т' - 19: 2, # 'у' - 29: 0, # 'ф' - 25: 0, # 'х' - 22: 0, # 'ц' - 21: 0, # 'ч' - 27: 0, # 'ш' - 24: 0, # 'щ' - 17: 2, # 'ъ' - 52: 1, # 'ь' - 42: 2, # 'ю' - 16: 1, # 'я' - 58: 0, # 'є' - 62: 0, # '№' - }, - 36: { # 'Н' - 63: 0, # 'e' - 45: 0, # '\xad' - 31: 2, # 'А' - 32: 2, # 'Б' - 35: 1, # 'В' - 43: 1, # 'Г' - 37: 2, # 'Д' - 44: 2, # 'Е' - 55: 1, # 'Ж' - 47: 1, # 'З' - 40: 2, # 'И' - 59: 1, # 'Й' - 33: 2, # 'К' - 46: 1, # 'Л' - 38: 1, # 'М' - 36: 1, # 'Н' - 41: 2, # 'О' - 30: 1, # 'П' - 39: 1, # 'Р' - 28: 2, # 'С' - 34: 2, # 'Т' - 51: 1, # 'У' - 48: 1, # 'Ф' - 49: 1, # 'Х' - 53: 1, # 'Ц' - 50: 1, # 'Ч' - 54: 1, # 'Ш' - 57: 0, # 'Щ' - 61: 1, # 'Ъ' - 60: 1, # 'Ю' - 56: 1, # 'Я' - 1: 3, # 'а' - 18: 0, # 'б' - 9: 0, # 'в' - 20: 1, # 'г' - 11: 0, # 'д' - 3: 3, # 'е' - 23: 0, # 'ж' - 15: 0, # 'з' - 2: 3, # 'и' - 26: 0, # 'й' - 12: 0, # 'к' - 10: 0, # 'л' - 14: 0, # 'м' - 6: 0, # 'н' - 4: 3, # 'о' - 13: 0, # 'п' - 7: 0, # 'р' - 8: 0, # 'с' - 5: 1, # 'т' - 19: 1, # 'у' - 29: 0, # 'ф' - 25: 0, # 'х' - 22: 0, # 'ц' - 21: 0, # 'ч' - 27: 1, # 'ш' - 24: 0, # 'щ' - 17: 0, # 'ъ' - 52: 0, # 'ь' - 42: 2, # 'ю' - 16: 2, # 'я' - 58: 0, # 'є' - 62: 0, # '№' - }, - 41: { # 'О' - 63: 0, # 'e' - 45: 0, # '\xad' - 31: 1, # 'А' - 32: 1, # 'Б' - 35: 2, # 'В' - 43: 1, # 'Г' - 37: 2, # 'Д' - 44: 1, # 'Е' - 55: 1, # 'Ж' - 47: 1, # 'З' - 40: 1, # 'И' - 59: 1, # 'Й' - 33: 2, # 'К' - 46: 2, # 'Л' - 38: 2, # 'М' - 36: 2, # 'Н' - 41: 2, # 'О' - 30: 1, # 'П' - 39: 2, # 'Р' - 28: 2, # 'С' - 34: 2, # 'Т' - 51: 1, # 'У' - 48: 1, # 'Ф' - 49: 1, # 'Х' - 53: 0, # 'Ц' - 50: 1, # 'Ч' - 54: 1, # 'Ш' - 57: 1, # 'Щ' - 61: 0, # 'Ъ' - 60: 0, # 'Ю' - 56: 1, # 'Я' - 1: 1, # 'а' - 18: 2, # 'б' - 9: 2, # 'в' - 20: 2, # 'г' - 11: 1, # 'д' - 3: 1, # 'е' - 23: 1, # 'ж' - 15: 1, # 'з' - 2: 0, # 'и' - 26: 1, # 'й' - 12: 2, # 'к' - 10: 2, # 'л' - 14: 1, # 'м' - 6: 1, # 'н' - 4: 0, # 'о' - 13: 2, # 'п' - 7: 2, # 'р' - 8: 2, # 'с' - 5: 3, # 'т' - 19: 1, # 'у' - 29: 1, # 'ф' - 25: 1, # 'х' - 22: 1, # 'ц' - 21: 2, # 'ч' - 27: 0, # 'ш' - 24: 2, # 'щ' - 17: 0, # 'ъ' - 52: 0, # 'ь' - 42: 0, # 'ю' - 16: 1, # 'я' - 58: 0, # 'є' - 62: 0, # '№' - }, - 30: { # 'П' - 63: 0, # 'e' - 45: 1, # '\xad' - 31: 2, # 'А' - 32: 1, # 'Б' - 35: 1, # 'В' - 43: 1, # 'Г' - 37: 1, # 'Д' - 44: 1, # 'Е' - 55: 0, # 'Ж' - 47: 1, # 'З' - 40: 2, # 'И' - 59: 0, # 'Й' - 33: 1, # 'К' - 46: 1, # 'Л' - 38: 1, # 'М' - 36: 1, # 'Н' - 41: 2, # 'О' - 30: 2, # 'П' - 39: 2, # 'Р' - 28: 2, # 'С' - 34: 1, # 'Т' - 51: 2, # 'У' - 48: 1, # 'Ф' - 49: 0, # 'Х' - 53: 1, # 'Ц' - 50: 1, # 'Ч' - 54: 1, # 'Ш' - 57: 0, # 'Щ' - 61: 1, # 'Ъ' - 60: 1, # 'Ю' - 56: 0, # 'Я' - 1: 3, # 'а' - 18: 0, # 'б' - 9: 0, # 'в' - 20: 0, # 'г' - 11: 2, # 'д' - 3: 3, # 'е' - 23: 0, # 'ж' - 15: 0, # 'з' - 2: 2, # 'и' - 26: 0, # 'й' - 12: 1, # 'к' - 10: 3, # 'л' - 14: 0, # 'м' - 6: 1, # 'н' - 4: 3, # 'о' - 13: 0, # 'п' - 7: 3, # 'р' - 8: 1, # 'с' - 5: 1, # 'т' - 19: 2, # 'у' - 29: 1, # 'ф' - 25: 1, # 'х' - 22: 0, # 'ц' - 21: 1, # 'ч' - 27: 1, # 'ш' - 24: 0, # 'щ' - 17: 2, # 'ъ' - 52: 1, # 'ь' - 42: 1, # 'ю' - 16: 1, # 'я' - 58: 0, # 'є' - 62: 0, # '№' - }, - 39: { # 'Р' - 63: 0, # 'e' - 45: 1, # '\xad' - 31: 2, # 'А' - 32: 1, # 'Б' - 35: 1, # 'В' - 43: 2, # 'Г' - 37: 2, # 'Д' - 44: 2, # 'Е' - 55: 0, # 'Ж' - 47: 1, # 'З' - 40: 2, # 'И' - 59: 0, # 'Й' - 33: 1, # 'К' - 46: 0, # 'Л' - 38: 1, # 'М' - 36: 1, # 'Н' - 41: 2, # 'О' - 30: 2, # 'П' - 39: 1, # 'Р' - 28: 1, # 'С' - 34: 1, # 'Т' - 51: 1, # 'У' - 48: 1, # 'Ф' - 49: 1, # 'Х' - 53: 1, # 'Ц' - 50: 1, # 'Ч' - 54: 0, # 'Ш' - 57: 0, # 'Щ' - 61: 1, # 'Ъ' - 60: 1, # 'Ю' - 56: 1, # 'Я' - 1: 3, # 'а' - 18: 0, # 'б' - 9: 0, # 'в' - 20: 0, # 'г' - 11: 0, # 'д' - 3: 2, # 'е' - 23: 0, # 'ж' - 15: 0, # 'з' - 2: 2, # 'и' - 26: 0, # 'й' - 12: 0, # 'к' - 10: 0, # 'л' - 14: 0, # 'м' - 6: 1, # 'н' - 4: 3, # 'о' - 13: 0, # 'п' - 7: 0, # 'р' - 8: 1, # 'с' - 5: 0, # 'т' - 19: 3, # 'у' - 29: 0, # 'ф' - 25: 0, # 'х' - 22: 0, # 'ц' - 21: 0, # 'ч' - 27: 0, # 'ш' - 24: 0, # 'щ' - 17: 1, # 'ъ' - 52: 0, # 'ь' - 42: 1, # 'ю' - 16: 1, # 'я' - 58: 0, # 'є' - 62: 0, # '№' - }, - 28: { # 'С' - 63: 1, # 'e' - 45: 0, # '\xad' - 31: 3, # 'А' - 32: 2, # 'Б' - 35: 2, # 'В' - 43: 1, # 'Г' - 37: 2, # 'Д' - 44: 2, # 'Е' - 55: 1, # 'Ж' - 47: 1, # 'З' - 40: 2, # 'И' - 59: 0, # 'Й' - 33: 2, # 'К' - 46: 1, # 'Л' - 38: 1, # 'М' - 36: 1, # 'Н' - 41: 2, # 'О' - 30: 2, # 'П' - 39: 1, # 'Р' - 28: 2, # 'С' - 34: 2, # 'Т' - 51: 1, # 'У' - 48: 1, # 'Ф' - 49: 0, # 'Х' - 53: 0, # 'Ц' - 50: 0, # 'Ч' - 54: 0, # 'Ш' - 57: 0, # 'Щ' - 61: 1, # 'Ъ' - 60: 1, # 'Ю' - 56: 1, # 'Я' - 1: 3, # 'а' - 18: 1, # 'б' - 9: 2, # 'в' - 20: 1, # 'г' - 11: 1, # 'д' - 3: 3, # 'е' - 23: 0, # 'ж' - 15: 0, # 'з' - 2: 3, # 'и' - 26: 0, # 'й' - 12: 2, # 'к' - 10: 3, # 'л' - 14: 2, # 'м' - 6: 1, # 'н' - 4: 3, # 'о' - 13: 3, # 'п' - 7: 2, # 'р' - 8: 0, # 'с' - 5: 3, # 'т' - 19: 2, # 'у' - 29: 2, # 'ф' - 25: 1, # 'х' - 22: 1, # 'ц' - 21: 1, # 'ч' - 27: 0, # 'ш' - 24: 0, # 'щ' - 17: 3, # 'ъ' - 52: 1, # 'ь' - 42: 1, # 'ю' - 16: 1, # 'я' - 58: 0, # 'є' - 62: 0, # '№' - }, - 34: { # 'Т' - 63: 0, # 'e' - 45: 0, # '\xad' - 31: 2, # 'А' - 32: 2, # 'Б' - 35: 1, # 'В' - 43: 0, # 'Г' - 37: 1, # 'Д' - 44: 2, # 'Е' - 55: 0, # 'Ж' - 47: 0, # 'З' - 40: 2, # 'И' - 59: 0, # 'Й' - 33: 2, # 'К' - 46: 1, # 'Л' - 38: 1, # 'М' - 36: 1, # 'Н' - 41: 2, # 'О' - 30: 1, # 'П' - 39: 2, # 'Р' - 28: 2, # 'С' - 34: 1, # 'Т' - 51: 1, # 'У' - 48: 1, # 'Ф' - 49: 0, # 'Х' - 53: 1, # 'Ц' - 50: 0, # 'Ч' - 54: 0, # 'Ш' - 57: 0, # 'Щ' - 61: 1, # 'Ъ' - 60: 0, # 'Ю' - 56: 1, # 'Я' - 1: 3, # 'а' - 18: 1, # 'б' - 9: 1, # 'в' - 20: 0, # 'г' - 11: 0, # 'д' - 3: 3, # 'е' - 23: 0, # 'ж' - 15: 0, # 'з' - 2: 2, # 'и' - 26: 0, # 'й' - 12: 1, # 'к' - 10: 1, # 'л' - 14: 0, # 'м' - 6: 0, # 'н' - 4: 3, # 'о' - 13: 0, # 'п' - 7: 3, # 'р' - 8: 0, # 'с' - 5: 0, # 'т' - 19: 2, # 'у' - 29: 0, # 'ф' - 25: 0, # 'х' - 22: 0, # 'ц' - 21: 0, # 'ч' - 27: 0, # 'ш' - 24: 0, # 'щ' - 17: 2, # 'ъ' - 52: 0, # 'ь' - 42: 1, # 'ю' - 16: 2, # 'я' - 58: 0, # 'є' - 62: 0, # '№' - }, - 51: { # 'У' - 63: 0, # 'e' - 45: 1, # '\xad' - 31: 1, # 'А' - 32: 1, # 'Б' - 35: 1, # 'В' - 43: 1, # 'Г' - 37: 1, # 'Д' - 44: 2, # 'Е' - 55: 1, # 'Ж' - 47: 1, # 'З' - 40: 1, # 'И' - 59: 0, # 'Й' - 33: 1, # 'К' - 46: 1, # 'Л' - 38: 1, # 'М' - 36: 1, # 'Н' - 41: 0, # 'О' - 30: 1, # 'П' - 39: 1, # 'Р' - 28: 1, # 'С' - 34: 2, # 'Т' - 51: 0, # 'У' - 48: 1, # 'Ф' - 49: 1, # 'Х' - 53: 1, # 'Ц' - 50: 1, # 'Ч' - 54: 1, # 'Ш' - 57: 0, # 'Щ' - 61: 0, # 'Ъ' - 60: 0, # 'Ю' - 56: 0, # 'Я' - 1: 1, # 'а' - 18: 1, # 'б' - 9: 2, # 'в' - 20: 1, # 'г' - 11: 1, # 'д' - 3: 2, # 'е' - 23: 1, # 'ж' - 15: 1, # 'з' - 2: 2, # 'и' - 26: 1, # 'й' - 12: 2, # 'к' - 10: 1, # 'л' - 14: 1, # 'м' - 6: 2, # 'н' - 4: 2, # 'о' - 13: 1, # 'п' - 7: 1, # 'р' - 8: 2, # 'с' - 5: 1, # 'т' - 19: 1, # 'у' - 29: 0, # 'ф' - 25: 1, # 'х' - 22: 0, # 'ц' - 21: 2, # 'ч' - 27: 1, # 'ш' - 24: 0, # 'щ' - 17: 1, # 'ъ' - 52: 0, # 'ь' - 42: 0, # 'ю' - 16: 0, # 'я' - 58: 0, # 'є' - 62: 0, # '№' - }, - 48: { # 'Ф' - 63: 0, # 'e' - 45: 0, # '\xad' - 31: 2, # 'А' - 32: 1, # 'Б' - 35: 1, # 'В' - 43: 0, # 'Г' - 37: 0, # 'Д' - 44: 1, # 'Е' - 55: 0, # 'Ж' - 47: 0, # 'З' - 40: 2, # 'И' - 59: 0, # 'Й' - 33: 1, # 'К' - 46: 1, # 'Л' - 38: 0, # 'М' - 36: 1, # 'Н' - 41: 1, # 'О' - 30: 2, # 'П' - 39: 1, # 'Р' - 28: 2, # 'С' - 34: 1, # 'Т' - 51: 1, # 'У' - 48: 0, # 'Ф' - 49: 0, # 'Х' - 53: 0, # 'Ц' - 50: 0, # 'Ч' - 54: 0, # 'Ш' - 57: 0, # 'Щ' - 61: 0, # 'Ъ' - 60: 0, # 'Ю' - 56: 0, # 'Я' - 1: 2, # 'а' - 18: 0, # 'б' - 9: 0, # 'в' - 20: 0, # 'г' - 11: 0, # 'д' - 3: 2, # 'е' - 23: 0, # 'ж' - 15: 0, # 'з' - 2: 2, # 'и' - 26: 0, # 'й' - 12: 0, # 'к' - 10: 2, # 'л' - 14: 0, # 'м' - 6: 0, # 'н' - 4: 2, # 'о' - 13: 0, # 'п' - 7: 2, # 'р' - 8: 0, # 'с' - 5: 0, # 'т' - 19: 1, # 'у' - 29: 0, # 'ф' - 25: 0, # 'х' - 22: 0, # 'ц' - 21: 0, # 'ч' - 27: 0, # 'ш' - 24: 0, # 'щ' - 17: 1, # 'ъ' - 52: 1, # 'ь' - 42: 1, # 'ю' - 16: 0, # 'я' - 58: 0, # 'є' - 62: 0, # '№' - }, - 49: { # 'Х' - 63: 0, # 'e' - 45: 0, # '\xad' - 31: 1, # 'А' - 32: 0, # 'Б' - 35: 1, # 'В' - 43: 1, # 'Г' - 37: 1, # 'Д' - 44: 1, # 'Е' - 55: 0, # 'Ж' - 47: 0, # 'З' - 40: 1, # 'И' - 59: 0, # 'Й' - 33: 0, # 'К' - 46: 1, # 'Л' - 38: 1, # 'М' - 36: 1, # 'Н' - 41: 1, # 'О' - 30: 1, # 'П' - 39: 1, # 'Р' - 28: 0, # 'С' - 34: 0, # 'Т' - 51: 0, # 'У' - 48: 0, # 'Ф' - 49: 1, # 'Х' - 53: 0, # 'Ц' - 50: 0, # 'Ч' - 54: 0, # 'Ш' - 57: 0, # 'Щ' - 61: 0, # 'Ъ' - 60: 0, # 'Ю' - 56: 0, # 'Я' - 1: 2, # 'а' - 18: 0, # 'б' - 9: 1, # 'в' - 20: 0, # 'г' - 11: 0, # 'д' - 3: 2, # 'е' - 23: 0, # 'ж' - 15: 0, # 'з' - 2: 2, # 'и' - 26: 0, # 'й' - 12: 0, # 'к' - 10: 1, # 'л' - 14: 1, # 'м' - 6: 0, # 'н' - 4: 2, # 'о' - 13: 0, # 'п' - 7: 2, # 'р' - 8: 0, # 'с' - 5: 0, # 'т' - 19: 2, # 'у' - 29: 0, # 'ф' - 25: 0, # 'х' - 22: 0, # 'ц' - 21: 0, # 'ч' - 27: 0, # 'ш' - 24: 0, # 'щ' - 17: 2, # 'ъ' - 52: 1, # 'ь' - 42: 1, # 'ю' - 16: 0, # 'я' - 58: 0, # 'є' - 62: 0, # '№' - }, - 53: { # 'Ц' - 63: 0, # 'e' - 45: 0, # '\xad' - 31: 1, # 'А' - 32: 0, # 'Б' - 35: 1, # 'В' - 43: 0, # 'Г' - 37: 0, # 'Д' - 44: 1, # 'Е' - 55: 0, # 'Ж' - 47: 0, # 'З' - 40: 2, # 'И' - 59: 0, # 'Й' - 33: 2, # 'К' - 46: 1, # 'Л' - 38: 1, # 'М' - 36: 0, # 'Н' - 41: 0, # 'О' - 30: 0, # 'П' - 39: 1, # 'Р' - 28: 2, # 'С' - 34: 0, # 'Т' - 51: 1, # 'У' - 48: 0, # 'Ф' - 49: 0, # 'Х' - 53: 0, # 'Ц' - 50: 0, # 'Ч' - 54: 0, # 'Ш' - 57: 0, # 'Щ' - 61: 0, # 'Ъ' - 60: 0, # 'Ю' - 56: 0, # 'Я' - 1: 2, # 'а' - 18: 0, # 'б' - 9: 2, # 'в' - 20: 0, # 'г' - 11: 0, # 'д' - 3: 2, # 'е' - 23: 0, # 'ж' - 15: 1, # 'з' - 2: 2, # 'и' - 26: 0, # 'й' - 12: 0, # 'к' - 10: 0, # 'л' - 14: 0, # 'м' - 6: 0, # 'н' - 4: 1, # 'о' - 13: 0, # 'п' - 7: 1, # 'р' - 8: 0, # 'с' - 5: 0, # 'т' - 19: 1, # 'у' - 29: 0, # 'ф' - 25: 0, # 'х' - 22: 0, # 'ц' - 21: 0, # 'ч' - 27: 0, # 'ш' - 24: 0, # 'щ' - 17: 1, # 'ъ' - 52: 0, # 'ь' - 42: 1, # 'ю' - 16: 1, # 'я' - 58: 0, # 'є' - 62: 0, # '№' - }, - 50: { # 'Ч' - 63: 0, # 'e' - 45: 0, # '\xad' - 31: 2, # 'А' - 32: 1, # 'Б' - 35: 0, # 'В' - 43: 0, # 'Г' - 37: 0, # 'Д' - 44: 1, # 'Е' - 55: 0, # 'Ж' - 47: 1, # 'З' - 40: 1, # 'И' - 59: 0, # 'Й' - 33: 1, # 'К' - 46: 1, # 'Л' - 38: 0, # 'М' - 36: 1, # 'Н' - 41: 1, # 'О' - 30: 0, # 'П' - 39: 0, # 'Р' - 28: 0, # 'С' - 34: 0, # 'Т' - 51: 1, # 'У' - 48: 0, # 'Ф' - 49: 0, # 'Х' - 53: 0, # 'Ц' - 50: 0, # 'Ч' - 54: 0, # 'Ш' - 57: 0, # 'Щ' - 61: 0, # 'Ъ' - 60: 0, # 'Ю' - 56: 0, # 'Я' - 1: 2, # 'а' - 18: 0, # 'б' - 9: 0, # 'в' - 20: 0, # 'г' - 11: 0, # 'д' - 3: 3, # 'е' - 23: 1, # 'ж' - 15: 0, # 'з' - 2: 2, # 'и' - 26: 0, # 'й' - 12: 0, # 'к' - 10: 1, # 'л' - 14: 0, # 'м' - 6: 0, # 'н' - 4: 2, # 'о' - 13: 0, # 'п' - 7: 1, # 'р' - 8: 0, # 'с' - 5: 0, # 'т' - 19: 2, # 'у' - 29: 0, # 'ф' - 25: 0, # 'х' - 22: 0, # 'ц' - 21: 0, # 'ч' - 27: 0, # 'ш' - 24: 0, # 'щ' - 17: 1, # 'ъ' - 52: 1, # 'ь' - 42: 0, # 'ю' - 16: 0, # 'я' - 58: 0, # 'є' - 62: 0, # '№' - }, - 54: { # 'Ш' - 63: 0, # 'e' - 45: 0, # '\xad' - 31: 1, # 'А' - 32: 0, # 'Б' - 35: 0, # 'В' - 43: 0, # 'Г' - 37: 0, # 'Д' - 44: 1, # 'Е' - 55: 0, # 'Ж' - 47: 1, # 'З' - 40: 1, # 'И' - 59: 0, # 'Й' - 33: 1, # 'К' - 46: 0, # 'Л' - 38: 0, # 'М' - 36: 1, # 'Н' - 41: 1, # 'О' - 30: 0, # 'П' - 39: 0, # 'Р' - 28: 0, # 'С' - 34: 0, # 'Т' - 51: 1, # 'У' - 48: 0, # 'Ф' - 49: 0, # 'Х' - 53: 0, # 'Ц' - 50: 0, # 'Ч' - 54: 0, # 'Ш' - 57: 0, # 'Щ' - 61: 0, # 'Ъ' - 60: 0, # 'Ю' - 56: 0, # 'Я' - 1: 2, # 'а' - 18: 0, # 'б' - 9: 2, # 'в' - 20: 0, # 'г' - 11: 0, # 'д' - 3: 2, # 'е' - 23: 0, # 'ж' - 15: 0, # 'з' - 2: 2, # 'и' - 26: 0, # 'й' - 12: 1, # 'к' - 10: 1, # 'л' - 14: 1, # 'м' - 6: 1, # 'н' - 4: 2, # 'о' - 13: 1, # 'п' - 7: 1, # 'р' - 8: 0, # 'с' - 5: 0, # 'т' - 19: 2, # 'у' - 29: 0, # 'ф' - 25: 0, # 'х' - 22: 0, # 'ц' - 21: 1, # 'ч' - 27: 0, # 'ш' - 24: 0, # 'щ' - 17: 1, # 'ъ' - 52: 1, # 'ь' - 42: 0, # 'ю' - 16: 0, # 'я' - 58: 0, # 'є' - 62: 0, # '№' - }, - 57: { # 'Щ' - 63: 0, # 'e' - 45: 0, # '\xad' - 31: 1, # 'А' - 32: 0, # 'Б' - 35: 0, # 'В' - 43: 0, # 'Г' - 37: 0, # 'Д' - 44: 1, # 'Е' - 55: 0, # 'Ж' - 47: 0, # 'З' - 40: 1, # 'И' - 59: 0, # 'Й' - 33: 0, # 'К' - 46: 0, # 'Л' - 38: 0, # 'М' - 36: 0, # 'Н' - 41: 1, # 'О' - 30: 0, # 'П' - 39: 0, # 'Р' - 28: 0, # 'С' - 34: 0, # 'Т' - 51: 0, # 'У' - 48: 0, # 'Ф' - 49: 0, # 'Х' - 53: 0, # 'Ц' - 50: 0, # 'Ч' - 54: 0, # 'Ш' - 57: 0, # 'Щ' - 61: 0, # 'Ъ' - 60: 0, # 'Ю' - 56: 0, # 'Я' - 1: 2, # 'а' - 18: 0, # 'б' - 9: 0, # 'в' - 20: 0, # 'г' - 11: 0, # 'д' - 3: 2, # 'е' - 23: 0, # 'ж' - 15: 0, # 'з' - 2: 1, # 'и' - 26: 0, # 'й' - 12: 0, # 'к' - 10: 0, # 'л' - 14: 0, # 'м' - 6: 0, # 'н' - 4: 1, # 'о' - 13: 0, # 'п' - 7: 1, # 'р' - 8: 0, # 'с' - 5: 0, # 'т' - 19: 1, # 'у' - 29: 0, # 'ф' - 25: 0, # 'х' - 22: 0, # 'ц' - 21: 0, # 'ч' - 27: 0, # 'ш' - 24: 0, # 'щ' - 17: 1, # 'ъ' - 52: 0, # 'ь' - 42: 0, # 'ю' - 16: 1, # 'я' - 58: 0, # 'є' - 62: 0, # '№' - }, - 61: { # 'Ъ' - 63: 0, # 'e' - 45: 0, # '\xad' - 31: 0, # 'А' - 32: 1, # 'Б' - 35: 1, # 'В' - 43: 0, # 'Г' - 37: 1, # 'Д' - 44: 0, # 'Е' - 55: 1, # 'Ж' - 47: 1, # 'З' - 40: 0, # 'И' - 59: 0, # 'Й' - 33: 1, # 'К' - 46: 2, # 'Л' - 38: 1, # 'М' - 36: 1, # 'Н' - 41: 0, # 'О' - 30: 1, # 'П' - 39: 2, # 'Р' - 28: 1, # 'С' - 34: 1, # 'Т' - 51: 0, # 'У' - 48: 0, # 'Ф' - 49: 1, # 'Х' - 53: 1, # 'Ц' - 50: 1, # 'Ч' - 54: 1, # 'Ш' - 57: 1, # 'Щ' - 61: 0, # 'Ъ' - 60: 0, # 'Ю' - 56: 0, # 'Я' - 1: 0, # 'а' - 18: 0, # 'б' - 9: 0, # 'в' - 20: 0, # 'г' - 11: 0, # 'д' - 3: 0, # 'е' - 23: 0, # 'ж' - 15: 0, # 'з' - 2: 0, # 'и' - 26: 0, # 'й' - 12: 0, # 'к' - 10: 1, # 'л' - 14: 0, # 'м' - 6: 1, # 'н' - 4: 0, # 'о' - 13: 0, # 'п' - 7: 1, # 'р' - 8: 0, # 'с' - 5: 0, # 'т' - 19: 0, # 'у' - 29: 0, # 'ф' - 25: 0, # 'х' - 22: 0, # 'ц' - 21: 0, # 'ч' - 27: 0, # 'ш' - 24: 0, # 'щ' - 17: 0, # 'ъ' - 52: 0, # 'ь' - 42: 0, # 'ю' - 16: 0, # 'я' - 58: 0, # 'є' - 62: 0, # '№' - }, - 60: { # 'Ю' - 63: 0, # 'e' - 45: 0, # '\xad' - 31: 1, # 'А' - 32: 1, # 'Б' - 35: 0, # 'В' - 43: 1, # 'Г' - 37: 1, # 'Д' - 44: 0, # 'Е' - 55: 1, # 'Ж' - 47: 0, # 'З' - 40: 0, # 'И' - 59: 0, # 'Й' - 33: 1, # 'К' - 46: 1, # 'Л' - 38: 0, # 'М' - 36: 1, # 'Н' - 41: 0, # 'О' - 30: 0, # 'П' - 39: 1, # 'Р' - 28: 1, # 'С' - 34: 0, # 'Т' - 51: 0, # 'У' - 48: 0, # 'Ф' - 49: 0, # 'Х' - 53: 0, # 'Ц' - 50: 0, # 'Ч' - 54: 0, # 'Ш' - 57: 0, # 'Щ' - 61: 0, # 'Ъ' - 60: 0, # 'Ю' - 56: 0, # 'Я' - 1: 0, # 'а' - 18: 1, # 'б' - 9: 1, # 'в' - 20: 2, # 'г' - 11: 1, # 'д' - 3: 0, # 'е' - 23: 2, # 'ж' - 15: 1, # 'з' - 2: 1, # 'и' - 26: 0, # 'й' - 12: 1, # 'к' - 10: 1, # 'л' - 14: 1, # 'м' - 6: 1, # 'н' - 4: 0, # 'о' - 13: 1, # 'п' - 7: 1, # 'р' - 8: 1, # 'с' - 5: 1, # 'т' - 19: 0, # 'у' - 29: 0, # 'ф' - 25: 1, # 'х' - 22: 0, # 'ц' - 21: 0, # 'ч' - 27: 0, # 'ш' - 24: 0, # 'щ' - 17: 0, # 'ъ' - 52: 0, # 'ь' - 42: 0, # 'ю' - 16: 0, # 'я' - 58: 0, # 'є' - 62: 0, # '№' - }, - 56: { # 'Я' - 63: 0, # 'e' - 45: 0, # '\xad' - 31: 0, # 'А' - 32: 1, # 'Б' - 35: 1, # 'В' - 43: 1, # 'Г' - 37: 1, # 'Д' - 44: 0, # 'Е' - 55: 0, # 'Ж' - 47: 0, # 'З' - 40: 0, # 'И' - 59: 0, # 'Й' - 33: 1, # 'К' - 46: 1, # 'Л' - 38: 1, # 'М' - 36: 1, # 'Н' - 41: 0, # 'О' - 30: 0, # 'П' - 39: 0, # 'Р' - 28: 1, # 'С' - 34: 2, # 'Т' - 51: 0, # 'У' - 48: 0, # 'Ф' - 49: 0, # 'Х' - 53: 0, # 'Ц' - 50: 0, # 'Ч' - 54: 0, # 'Ш' - 57: 0, # 'Щ' - 61: 0, # 'Ъ' - 60: 0, # 'Ю' - 56: 0, # 'Я' - 1: 0, # 'а' - 18: 1, # 'б' - 9: 1, # 'в' - 20: 1, # 'г' - 11: 1, # 'д' - 3: 0, # 'е' - 23: 0, # 'ж' - 15: 1, # 'з' - 2: 1, # 'и' - 26: 1, # 'й' - 12: 1, # 'к' - 10: 1, # 'л' - 14: 2, # 'м' - 6: 2, # 'н' - 4: 0, # 'о' - 13: 2, # 'п' - 7: 1, # 'р' - 8: 1, # 'с' - 5: 1, # 'т' - 19: 0, # 'у' - 29: 0, # 'ф' - 25: 1, # 'х' - 22: 0, # 'ц' - 21: 0, # 'ч' - 27: 1, # 'ш' - 24: 0, # 'щ' - 17: 0, # 'ъ' - 52: 0, # 'ь' - 42: 1, # 'ю' - 16: 0, # 'я' - 58: 0, # 'є' - 62: 0, # '№' - }, - 1: { # 'а' - 63: 1, # 'e' - 45: 1, # '\xad' - 31: 1, # 'А' - 32: 0, # 'Б' - 35: 0, # 'В' - 43: 0, # 'Г' - 37: 0, # 'Д' - 44: 1, # 'Е' - 55: 0, # 'Ж' - 47: 0, # 'З' - 40: 0, # 'И' - 59: 0, # 'Й' - 33: 0, # 'К' - 46: 0, # 'Л' - 38: 0, # 'М' - 36: 0, # 'Н' - 41: 0, # 'О' - 30: 0, # 'П' - 39: 0, # 'Р' - 28: 0, # 'С' - 34: 0, # 'Т' - 51: 0, # 'У' - 48: 0, # 'Ф' - 49: 0, # 'Х' - 53: 0, # 'Ц' - 50: 0, # 'Ч' - 54: 0, # 'Ш' - 57: 0, # 'Щ' - 61: 0, # 'Ъ' - 60: 0, # 'Ю' - 56: 0, # 'Я' - 1: 1, # 'а' - 18: 3, # 'б' - 9: 3, # 'в' - 20: 3, # 'г' - 11: 3, # 'д' - 3: 3, # 'е' - 23: 3, # 'ж' - 15: 3, # 'з' - 2: 3, # 'и' - 26: 3, # 'й' - 12: 3, # 'к' - 10: 3, # 'л' - 14: 3, # 'м' - 6: 3, # 'н' - 4: 2, # 'о' - 13: 3, # 'п' - 7: 3, # 'р' - 8: 3, # 'с' - 5: 3, # 'т' - 19: 3, # 'у' - 29: 3, # 'ф' - 25: 3, # 'х' - 22: 3, # 'ц' - 21: 3, # 'ч' - 27: 3, # 'ш' - 24: 3, # 'щ' - 17: 0, # 'ъ' - 52: 0, # 'ь' - 42: 1, # 'ю' - 16: 3, # 'я' - 58: 0, # 'є' - 62: 0, # '№' - }, - 18: { # 'б' - 63: 1, # 'e' - 45: 0, # '\xad' - 31: 0, # 'А' - 32: 0, # 'Б' - 35: 0, # 'В' - 43: 0, # 'Г' - 37: 0, # 'Д' - 44: 0, # 'Е' - 55: 0, # 'Ж' - 47: 0, # 'З' - 40: 0, # 'И' - 59: 0, # 'Й' - 33: 0, # 'К' - 46: 0, # 'Л' - 38: 0, # 'М' - 36: 0, # 'Н' - 41: 0, # 'О' - 30: 0, # 'П' - 39: 0, # 'Р' - 28: 0, # 'С' - 34: 0, # 'Т' - 51: 0, # 'У' - 48: 0, # 'Ф' - 49: 0, # 'Х' - 53: 0, # 'Ц' - 50: 0, # 'Ч' - 54: 0, # 'Ш' - 57: 0, # 'Щ' - 61: 0, # 'Ъ' - 60: 0, # 'Ю' - 56: 0, # 'Я' - 1: 3, # 'а' - 18: 0, # 'б' - 9: 3, # 'в' - 20: 1, # 'г' - 11: 2, # 'д' - 3: 3, # 'е' - 23: 1, # 'ж' - 15: 1, # 'з' - 2: 3, # 'и' - 26: 0, # 'й' - 12: 1, # 'к' - 10: 3, # 'л' - 14: 2, # 'м' - 6: 3, # 'н' - 4: 3, # 'о' - 13: 1, # 'п' - 7: 3, # 'р' - 8: 3, # 'с' - 5: 0, # 'т' - 19: 3, # 'у' - 29: 0, # 'ф' - 25: 2, # 'х' - 22: 1, # 'ц' - 21: 1, # 'ч' - 27: 1, # 'ш' - 24: 3, # 'щ' - 17: 3, # 'ъ' - 52: 1, # 'ь' - 42: 2, # 'ю' - 16: 3, # 'я' - 58: 0, # 'є' - 62: 0, # '№' - }, - 9: { # 'в' - 63: 1, # 'e' - 45: 1, # '\xad' - 31: 0, # 'А' - 32: 1, # 'Б' - 35: 0, # 'В' - 43: 0, # 'Г' - 37: 0, # 'Д' - 44: 0, # 'Е' - 55: 0, # 'Ж' - 47: 0, # 'З' - 40: 0, # 'И' - 59: 0, # 'Й' - 33: 0, # 'К' - 46: 0, # 'Л' - 38: 0, # 'М' - 36: 0, # 'Н' - 41: 0, # 'О' - 30: 0, # 'П' - 39: 0, # 'Р' - 28: 0, # 'С' - 34: 0, # 'Т' - 51: 0, # 'У' - 48: 1, # 'Ф' - 49: 0, # 'Х' - 53: 0, # 'Ц' - 50: 0, # 'Ч' - 54: 0, # 'Ш' - 57: 0, # 'Щ' - 61: 0, # 'Ъ' - 60: 0, # 'Ю' - 56: 0, # 'Я' - 1: 3, # 'а' - 18: 1, # 'б' - 9: 0, # 'в' - 20: 2, # 'г' - 11: 3, # 'д' - 3: 3, # 'е' - 23: 1, # 'ж' - 15: 3, # 'з' - 2: 3, # 'и' - 26: 0, # 'й' - 12: 3, # 'к' - 10: 3, # 'л' - 14: 2, # 'м' - 6: 3, # 'н' - 4: 3, # 'о' - 13: 2, # 'п' - 7: 3, # 'р' - 8: 3, # 'с' - 5: 3, # 'т' - 19: 2, # 'у' - 29: 0, # 'ф' - 25: 2, # 'х' - 22: 2, # 'ц' - 21: 3, # 'ч' - 27: 2, # 'ш' - 24: 1, # 'щ' - 17: 3, # 'ъ' - 52: 1, # 'ь' - 42: 2, # 'ю' - 16: 3, # 'я' - 58: 0, # 'є' - 62: 0, # '№' - }, - 20: { # 'г' - 63: 0, # 'e' - 45: 0, # '\xad' - 31: 0, # 'А' - 32: 0, # 'Б' - 35: 0, # 'В' - 43: 0, # 'Г' - 37: 0, # 'Д' - 44: 0, # 'Е' - 55: 0, # 'Ж' - 47: 0, # 'З' - 40: 0, # 'И' - 59: 0, # 'Й' - 33: 0, # 'К' - 46: 0, # 'Л' - 38: 0, # 'М' - 36: 0, # 'Н' - 41: 0, # 'О' - 30: 0, # 'П' - 39: 0, # 'Р' - 28: 0, # 'С' - 34: 0, # 'Т' - 51: 0, # 'У' - 48: 0, # 'Ф' - 49: 0, # 'Х' - 53: 0, # 'Ц' - 50: 0, # 'Ч' - 54: 0, # 'Ш' - 57: 0, # 'Щ' - 61: 0, # 'Ъ' - 60: 0, # 'Ю' - 56: 0, # 'Я' - 1: 3, # 'а' - 18: 1, # 'б' - 9: 2, # 'в' - 20: 1, # 'г' - 11: 2, # 'д' - 3: 3, # 'е' - 23: 0, # 'ж' - 15: 1, # 'з' - 2: 3, # 'и' - 26: 0, # 'й' - 12: 1, # 'к' - 10: 3, # 'л' - 14: 1, # 'м' - 6: 3, # 'н' - 4: 3, # 'о' - 13: 1, # 'п' - 7: 3, # 'р' - 8: 2, # 'с' - 5: 2, # 'т' - 19: 3, # 'у' - 29: 1, # 'ф' - 25: 1, # 'х' - 22: 0, # 'ц' - 21: 1, # 'ч' - 27: 0, # 'ш' - 24: 0, # 'щ' - 17: 3, # 'ъ' - 52: 1, # 'ь' - 42: 1, # 'ю' - 16: 1, # 'я' - 58: 0, # 'є' - 62: 0, # '№' - }, - 11: { # 'д' - 63: 1, # 'e' - 45: 0, # '\xad' - 31: 0, # 'А' - 32: 0, # 'Б' - 35: 0, # 'В' - 43: 0, # 'Г' - 37: 0, # 'Д' - 44: 0, # 'Е' - 55: 0, # 'Ж' - 47: 0, # 'З' - 40: 0, # 'И' - 59: 0, # 'Й' - 33: 0, # 'К' - 46: 0, # 'Л' - 38: 0, # 'М' - 36: 0, # 'Н' - 41: 0, # 'О' - 30: 0, # 'П' - 39: 0, # 'Р' - 28: 0, # 'С' - 34: 0, # 'Т' - 51: 0, # 'У' - 48: 0, # 'Ф' - 49: 0, # 'Х' - 53: 0, # 'Ц' - 50: 0, # 'Ч' - 54: 0, # 'Ш' - 57: 0, # 'Щ' - 61: 0, # 'Ъ' - 60: 0, # 'Ю' - 56: 0, # 'Я' - 1: 3, # 'а' - 18: 2, # 'б' - 9: 3, # 'в' - 20: 2, # 'г' - 11: 2, # 'д' - 3: 3, # 'е' - 23: 3, # 'ж' - 15: 2, # 'з' - 2: 3, # 'и' - 26: 0, # 'й' - 12: 3, # 'к' - 10: 3, # 'л' - 14: 3, # 'м' - 6: 3, # 'н' - 4: 3, # 'о' - 13: 3, # 'п' - 7: 3, # 'р' - 8: 3, # 'с' - 5: 1, # 'т' - 19: 3, # 'у' - 29: 1, # 'ф' - 25: 2, # 'х' - 22: 2, # 'ц' - 21: 2, # 'ч' - 27: 1, # 'ш' - 24: 1, # 'щ' - 17: 3, # 'ъ' - 52: 1, # 'ь' - 42: 1, # 'ю' - 16: 3, # 'я' - 58: 0, # 'є' - 62: 0, # '№' - }, - 3: { # 'е' - 63: 0, # 'e' - 45: 1, # '\xad' - 31: 0, # 'А' - 32: 0, # 'Б' - 35: 0, # 'В' - 43: 0, # 'Г' - 37: 0, # 'Д' - 44: 0, # 'Е' - 55: 0, # 'Ж' - 47: 0, # 'З' - 40: 0, # 'И' - 59: 0, # 'Й' - 33: 0, # 'К' - 46: 0, # 'Л' - 38: 0, # 'М' - 36: 0, # 'Н' - 41: 0, # 'О' - 30: 0, # 'П' - 39: 0, # 'Р' - 28: 0, # 'С' - 34: 0, # 'Т' - 51: 0, # 'У' - 48: 0, # 'Ф' - 49: 0, # 'Х' - 53: 0, # 'Ц' - 50: 0, # 'Ч' - 54: 0, # 'Ш' - 57: 0, # 'Щ' - 61: 0, # 'Ъ' - 60: 0, # 'Ю' - 56: 0, # 'Я' - 1: 2, # 'а' - 18: 3, # 'б' - 9: 3, # 'в' - 20: 3, # 'г' - 11: 3, # 'д' - 3: 2, # 'е' - 23: 3, # 'ж' - 15: 3, # 'з' - 2: 2, # 'и' - 26: 3, # 'й' - 12: 3, # 'к' - 10: 3, # 'л' - 14: 3, # 'м' - 6: 3, # 'н' - 4: 3, # 'о' - 13: 3, # 'п' - 7: 3, # 'р' - 8: 3, # 'с' - 5: 3, # 'т' - 19: 2, # 'у' - 29: 3, # 'ф' - 25: 3, # 'х' - 22: 3, # 'ц' - 21: 3, # 'ч' - 27: 3, # 'ш' - 24: 3, # 'щ' - 17: 1, # 'ъ' - 52: 0, # 'ь' - 42: 1, # 'ю' - 16: 3, # 'я' - 58: 0, # 'є' - 62: 0, # '№' - }, - 23: { # 'ж' - 63: 0, # 'e' - 45: 0, # '\xad' - 31: 0, # 'А' - 32: 0, # 'Б' - 35: 0, # 'В' - 43: 0, # 'Г' - 37: 0, # 'Д' - 44: 0, # 'Е' - 55: 0, # 'Ж' - 47: 0, # 'З' - 40: 0, # 'И' - 59: 0, # 'Й' - 33: 0, # 'К' - 46: 0, # 'Л' - 38: 0, # 'М' - 36: 0, # 'Н' - 41: 0, # 'О' - 30: 0, # 'П' - 39: 0, # 'Р' - 28: 0, # 'С' - 34: 0, # 'Т' - 51: 0, # 'У' - 48: 0, # 'Ф' - 49: 0, # 'Х' - 53: 0, # 'Ц' - 50: 0, # 'Ч' - 54: 0, # 'Ш' - 57: 0, # 'Щ' - 61: 0, # 'Ъ' - 60: 0, # 'Ю' - 56: 0, # 'Я' - 1: 3, # 'а' - 18: 3, # 'б' - 9: 2, # 'в' - 20: 1, # 'г' - 11: 3, # 'д' - 3: 3, # 'е' - 23: 0, # 'ж' - 15: 0, # 'з' - 2: 3, # 'и' - 26: 0, # 'й' - 12: 2, # 'к' - 10: 1, # 'л' - 14: 1, # 'м' - 6: 3, # 'н' - 4: 2, # 'о' - 13: 1, # 'п' - 7: 1, # 'р' - 8: 1, # 'с' - 5: 1, # 'т' - 19: 2, # 'у' - 29: 0, # 'ф' - 25: 0, # 'х' - 22: 1, # 'ц' - 21: 1, # 'ч' - 27: 0, # 'ш' - 24: 0, # 'щ' - 17: 2, # 'ъ' - 52: 0, # 'ь' - 42: 0, # 'ю' - 16: 1, # 'я' - 58: 0, # 'є' - 62: 0, # '№' - }, - 15: { # 'з' - 63: 1, # 'e' - 45: 0, # '\xad' - 31: 0, # 'А' - 32: 0, # 'Б' - 35: 0, # 'В' - 43: 0, # 'Г' - 37: 0, # 'Д' - 44: 0, # 'Е' - 55: 0, # 'Ж' - 47: 0, # 'З' - 40: 0, # 'И' - 59: 0, # 'Й' - 33: 0, # 'К' - 46: 0, # 'Л' - 38: 0, # 'М' - 36: 0, # 'Н' - 41: 0, # 'О' - 30: 0, # 'П' - 39: 0, # 'Р' - 28: 0, # 'С' - 34: 0, # 'Т' - 51: 0, # 'У' - 48: 0, # 'Ф' - 49: 0, # 'Х' - 53: 0, # 'Ц' - 50: 0, # 'Ч' - 54: 0, # 'Ш' - 57: 0, # 'Щ' - 61: 0, # 'Ъ' - 60: 0, # 'Ю' - 56: 0, # 'Я' - 1: 3, # 'а' - 18: 3, # 'б' - 9: 3, # 'в' - 20: 3, # 'г' - 11: 3, # 'д' - 3: 3, # 'е' - 23: 1, # 'ж' - 15: 1, # 'з' - 2: 3, # 'и' - 26: 0, # 'й' - 12: 3, # 'к' - 10: 3, # 'л' - 14: 3, # 'м' - 6: 3, # 'н' - 4: 3, # 'о' - 13: 3, # 'п' - 7: 3, # 'р' - 8: 3, # 'с' - 5: 3, # 'т' - 19: 3, # 'у' - 29: 1, # 'ф' - 25: 2, # 'х' - 22: 2, # 'ц' - 21: 2, # 'ч' - 27: 2, # 'ш' - 24: 1, # 'щ' - 17: 2, # 'ъ' - 52: 1, # 'ь' - 42: 1, # 'ю' - 16: 2, # 'я' - 58: 0, # 'є' - 62: 0, # '№' - }, - 2: { # 'и' - 63: 1, # 'e' - 45: 1, # '\xad' - 31: 0, # 'А' - 32: 0, # 'Б' - 35: 0, # 'В' - 43: 1, # 'Г' - 37: 0, # 'Д' - 44: 0, # 'Е' - 55: 0, # 'Ж' - 47: 0, # 'З' - 40: 0, # 'И' - 59: 0, # 'Й' - 33: 1, # 'К' - 46: 0, # 'Л' - 38: 0, # 'М' - 36: 0, # 'Н' - 41: 0, # 'О' - 30: 1, # 'П' - 39: 0, # 'Р' - 28: 0, # 'С' - 34: 0, # 'Т' - 51: 0, # 'У' - 48: 1, # 'Ф' - 49: 0, # 'Х' - 53: 0, # 'Ц' - 50: 0, # 'Ч' - 54: 0, # 'Ш' - 57: 0, # 'Щ' - 61: 0, # 'Ъ' - 60: 0, # 'Ю' - 56: 0, # 'Я' - 1: 3, # 'а' - 18: 3, # 'б' - 9: 3, # 'в' - 20: 3, # 'г' - 11: 3, # 'д' - 3: 3, # 'е' - 23: 3, # 'ж' - 15: 3, # 'з' - 2: 3, # 'и' - 26: 3, # 'й' - 12: 3, # 'к' - 10: 3, # 'л' - 14: 3, # 'м' - 6: 3, # 'н' - 4: 3, # 'о' - 13: 3, # 'п' - 7: 3, # 'р' - 8: 3, # 'с' - 5: 3, # 'т' - 19: 2, # 'у' - 29: 3, # 'ф' - 25: 3, # 'х' - 22: 3, # 'ц' - 21: 3, # 'ч' - 27: 3, # 'ш' - 24: 3, # 'щ' - 17: 2, # 'ъ' - 52: 0, # 'ь' - 42: 1, # 'ю' - 16: 3, # 'я' - 58: 0, # 'є' - 62: 0, # '№' - }, - 26: { # 'й' - 63: 0, # 'e' - 45: 0, # '\xad' - 31: 0, # 'А' - 32: 0, # 'Б' - 35: 0, # 'В' - 43: 0, # 'Г' - 37: 0, # 'Д' - 44: 0, # 'Е' - 55: 0, # 'Ж' - 47: 0, # 'З' - 40: 0, # 'И' - 59: 0, # 'Й' - 33: 0, # 'К' - 46: 0, # 'Л' - 38: 0, # 'М' - 36: 0, # 'Н' - 41: 0, # 'О' - 30: 0, # 'П' - 39: 0, # 'Р' - 28: 0, # 'С' - 34: 0, # 'Т' - 51: 0, # 'У' - 48: 0, # 'Ф' - 49: 0, # 'Х' - 53: 0, # 'Ц' - 50: 0, # 'Ч' - 54: 0, # 'Ш' - 57: 0, # 'Щ' - 61: 0, # 'Ъ' - 60: 0, # 'Ю' - 56: 0, # 'Я' - 1: 1, # 'а' - 18: 2, # 'б' - 9: 2, # 'в' - 20: 1, # 'г' - 11: 2, # 'д' - 3: 2, # 'е' - 23: 0, # 'ж' - 15: 2, # 'з' - 2: 1, # 'и' - 26: 0, # 'й' - 12: 3, # 'к' - 10: 2, # 'л' - 14: 2, # 'м' - 6: 3, # 'н' - 4: 2, # 'о' - 13: 1, # 'п' - 7: 2, # 'р' - 8: 3, # 'с' - 5: 3, # 'т' - 19: 1, # 'у' - 29: 2, # 'ф' - 25: 1, # 'х' - 22: 2, # 'ц' - 21: 2, # 'ч' - 27: 1, # 'ш' - 24: 1, # 'щ' - 17: 1, # 'ъ' - 52: 0, # 'ь' - 42: 0, # 'ю' - 16: 1, # 'я' - 58: 0, # 'є' - 62: 0, # '№' - }, - 12: { # 'к' - 63: 1, # 'e' - 45: 0, # '\xad' - 31: 0, # 'А' - 32: 0, # 'Б' - 35: 1, # 'В' - 43: 0, # 'Г' - 37: 0, # 'Д' - 44: 0, # 'Е' - 55: 0, # 'Ж' - 47: 0, # 'З' - 40: 1, # 'И' - 59: 0, # 'Й' - 33: 0, # 'К' - 46: 0, # 'Л' - 38: 0, # 'М' - 36: 0, # 'Н' - 41: 0, # 'О' - 30: 0, # 'П' - 39: 0, # 'Р' - 28: 0, # 'С' - 34: 0, # 'Т' - 51: 0, # 'У' - 48: 0, # 'Ф' - 49: 0, # 'Х' - 53: 0, # 'Ц' - 50: 0, # 'Ч' - 54: 0, # 'Ш' - 57: 0, # 'Щ' - 61: 0, # 'Ъ' - 60: 0, # 'Ю' - 56: 0, # 'Я' - 1: 3, # 'а' - 18: 1, # 'б' - 9: 3, # 'в' - 20: 2, # 'г' - 11: 1, # 'д' - 3: 3, # 'е' - 23: 0, # 'ж' - 15: 2, # 'з' - 2: 3, # 'и' - 26: 0, # 'й' - 12: 1, # 'к' - 10: 3, # 'л' - 14: 2, # 'м' - 6: 3, # 'н' - 4: 3, # 'о' - 13: 1, # 'п' - 7: 3, # 'р' - 8: 3, # 'с' - 5: 3, # 'т' - 19: 3, # 'у' - 29: 1, # 'ф' - 25: 1, # 'х' - 22: 3, # 'ц' - 21: 2, # 'ч' - 27: 1, # 'ш' - 24: 0, # 'щ' - 17: 3, # 'ъ' - 52: 1, # 'ь' - 42: 2, # 'ю' - 16: 1, # 'я' - 58: 0, # 'є' - 62: 0, # '№' - }, - 10: { # 'л' - 63: 1, # 'e' - 45: 1, # '\xad' - 31: 0, # 'А' - 32: 0, # 'Б' - 35: 0, # 'В' - 43: 0, # 'Г' - 37: 0, # 'Д' - 44: 0, # 'Е' - 55: 0, # 'Ж' - 47: 0, # 'З' - 40: 0, # 'И' - 59: 0, # 'Й' - 33: 0, # 'К' - 46: 0, # 'Л' - 38: 0, # 'М' - 36: 0, # 'Н' - 41: 0, # 'О' - 30: 0, # 'П' - 39: 0, # 'Р' - 28: 1, # 'С' - 34: 0, # 'Т' - 51: 0, # 'У' - 48: 0, # 'Ф' - 49: 0, # 'Х' - 53: 0, # 'Ц' - 50: 0, # 'Ч' - 54: 0, # 'Ш' - 57: 0, # 'Щ' - 61: 0, # 'Ъ' - 60: 0, # 'Ю' - 56: 0, # 'Я' - 1: 3, # 'а' - 18: 3, # 'б' - 9: 3, # 'в' - 20: 3, # 'г' - 11: 2, # 'д' - 3: 3, # 'е' - 23: 3, # 'ж' - 15: 2, # 'з' - 2: 3, # 'и' - 26: 0, # 'й' - 12: 3, # 'к' - 10: 1, # 'л' - 14: 2, # 'м' - 6: 3, # 'н' - 4: 3, # 'о' - 13: 2, # 'п' - 7: 2, # 'р' - 8: 3, # 'с' - 5: 3, # 'т' - 19: 3, # 'у' - 29: 2, # 'ф' - 25: 2, # 'х' - 22: 2, # 'ц' - 21: 2, # 'ч' - 27: 2, # 'ш' - 24: 1, # 'щ' - 17: 3, # 'ъ' - 52: 2, # 'ь' - 42: 3, # 'ю' - 16: 3, # 'я' - 58: 0, # 'є' - 62: 0, # '№' - }, - 14: { # 'м' - 63: 1, # 'e' - 45: 0, # '\xad' - 31: 1, # 'А' - 32: 0, # 'Б' - 35: 0, # 'В' - 43: 0, # 'Г' - 37: 0, # 'Д' - 44: 0, # 'Е' - 55: 0, # 'Ж' - 47: 0, # 'З' - 40: 0, # 'И' - 59: 0, # 'Й' - 33: 0, # 'К' - 46: 0, # 'Л' - 38: 0, # 'М' - 36: 0, # 'Н' - 41: 0, # 'О' - 30: 0, # 'П' - 39: 0, # 'Р' - 28: 0, # 'С' - 34: 0, # 'Т' - 51: 0, # 'У' - 48: 0, # 'Ф' - 49: 0, # 'Х' - 53: 0, # 'Ц' - 50: 0, # 'Ч' - 54: 0, # 'Ш' - 57: 0, # 'Щ' - 61: 0, # 'Ъ' - 60: 0, # 'Ю' - 56: 0, # 'Я' - 1: 3, # 'а' - 18: 3, # 'б' - 9: 3, # 'в' - 20: 1, # 'г' - 11: 1, # 'д' - 3: 3, # 'е' - 23: 1, # 'ж' - 15: 1, # 'з' - 2: 3, # 'и' - 26: 0, # 'й' - 12: 2, # 'к' - 10: 3, # 'л' - 14: 1, # 'м' - 6: 3, # 'н' - 4: 3, # 'о' - 13: 3, # 'п' - 7: 2, # 'р' - 8: 2, # 'с' - 5: 1, # 'т' - 19: 3, # 'у' - 29: 2, # 'ф' - 25: 1, # 'х' - 22: 2, # 'ц' - 21: 2, # 'ч' - 27: 2, # 'ш' - 24: 1, # 'щ' - 17: 3, # 'ъ' - 52: 1, # 'ь' - 42: 2, # 'ю' - 16: 3, # 'я' - 58: 0, # 'є' - 62: 0, # '№' - }, - 6: { # 'н' - 63: 1, # 'e' - 45: 0, # '\xad' - 31: 0, # 'А' - 32: 0, # 'Б' - 35: 0, # 'В' - 43: 0, # 'Г' - 37: 0, # 'Д' - 44: 0, # 'Е' - 55: 0, # 'Ж' - 47: 0, # 'З' - 40: 0, # 'И' - 59: 0, # 'Й' - 33: 0, # 'К' - 46: 0, # 'Л' - 38: 0, # 'М' - 36: 0, # 'Н' - 41: 0, # 'О' - 30: 0, # 'П' - 39: 1, # 'Р' - 28: 0, # 'С' - 34: 0, # 'Т' - 51: 0, # 'У' - 48: 0, # 'Ф' - 49: 0, # 'Х' - 53: 0, # 'Ц' - 50: 0, # 'Ч' - 54: 0, # 'Ш' - 57: 0, # 'Щ' - 61: 0, # 'Ъ' - 60: 0, # 'Ю' - 56: 0, # 'Я' - 1: 3, # 'а' - 18: 2, # 'б' - 9: 2, # 'в' - 20: 3, # 'г' - 11: 3, # 'д' - 3: 3, # 'е' - 23: 2, # 'ж' - 15: 2, # 'з' - 2: 3, # 'и' - 26: 0, # 'й' - 12: 3, # 'к' - 10: 2, # 'л' - 14: 1, # 'м' - 6: 3, # 'н' - 4: 3, # 'о' - 13: 1, # 'п' - 7: 2, # 'р' - 8: 3, # 'с' - 5: 3, # 'т' - 19: 3, # 'у' - 29: 3, # 'ф' - 25: 2, # 'х' - 22: 3, # 'ц' - 21: 3, # 'ч' - 27: 2, # 'ш' - 24: 1, # 'щ' - 17: 3, # 'ъ' - 52: 2, # 'ь' - 42: 2, # 'ю' - 16: 3, # 'я' - 58: 0, # 'є' - 62: 0, # '№' - }, - 4: { # 'о' - 63: 0, # 'e' - 45: 1, # '\xad' - 31: 0, # 'А' - 32: 0, # 'Б' - 35: 0, # 'В' - 43: 0, # 'Г' - 37: 0, # 'Д' - 44: 0, # 'Е' - 55: 0, # 'Ж' - 47: 0, # 'З' - 40: 0, # 'И' - 59: 0, # 'Й' - 33: 0, # 'К' - 46: 0, # 'Л' - 38: 0, # 'М' - 36: 0, # 'Н' - 41: 0, # 'О' - 30: 0, # 'П' - 39: 0, # 'Р' - 28: 0, # 'С' - 34: 0, # 'Т' - 51: 0, # 'У' - 48: 0, # 'Ф' - 49: 0, # 'Х' - 53: 0, # 'Ц' - 50: 0, # 'Ч' - 54: 0, # 'Ш' - 57: 0, # 'Щ' - 61: 0, # 'Ъ' - 60: 0, # 'Ю' - 56: 0, # 'Я' - 1: 2, # 'а' - 18: 3, # 'б' - 9: 3, # 'в' - 20: 3, # 'г' - 11: 3, # 'д' - 3: 3, # 'е' - 23: 3, # 'ж' - 15: 3, # 'з' - 2: 3, # 'и' - 26: 3, # 'й' - 12: 3, # 'к' - 10: 3, # 'л' - 14: 3, # 'м' - 6: 3, # 'н' - 4: 2, # 'о' - 13: 3, # 'п' - 7: 3, # 'р' - 8: 3, # 'с' - 5: 3, # 'т' - 19: 2, # 'у' - 29: 3, # 'ф' - 25: 3, # 'х' - 22: 3, # 'ц' - 21: 3, # 'ч' - 27: 3, # 'ш' - 24: 3, # 'щ' - 17: 1, # 'ъ' - 52: 0, # 'ь' - 42: 1, # 'ю' - 16: 3, # 'я' - 58: 0, # 'є' - 62: 0, # '№' - }, - 13: { # 'п' - 63: 1, # 'e' - 45: 0, # '\xad' - 31: 0, # 'А' - 32: 0, # 'Б' - 35: 0, # 'В' - 43: 0, # 'Г' - 37: 0, # 'Д' - 44: 0, # 'Е' - 55: 0, # 'Ж' - 47: 0, # 'З' - 40: 0, # 'И' - 59: 0, # 'Й' - 33: 0, # 'К' - 46: 0, # 'Л' - 38: 0, # 'М' - 36: 0, # 'Н' - 41: 0, # 'О' - 30: 0, # 'П' - 39: 0, # 'Р' - 28: 0, # 'С' - 34: 0, # 'Т' - 51: 0, # 'У' - 48: 0, # 'Ф' - 49: 0, # 'Х' - 53: 0, # 'Ц' - 50: 0, # 'Ч' - 54: 0, # 'Ш' - 57: 0, # 'Щ' - 61: 0, # 'Ъ' - 60: 0, # 'Ю' - 56: 0, # 'Я' - 1: 3, # 'а' - 18: 1, # 'б' - 9: 2, # 'в' - 20: 1, # 'г' - 11: 1, # 'д' - 3: 3, # 'е' - 23: 0, # 'ж' - 15: 1, # 'з' - 2: 3, # 'и' - 26: 1, # 'й' - 12: 2, # 'к' - 10: 3, # 'л' - 14: 1, # 'м' - 6: 2, # 'н' - 4: 3, # 'о' - 13: 1, # 'п' - 7: 3, # 'р' - 8: 2, # 'с' - 5: 2, # 'т' - 19: 3, # 'у' - 29: 1, # 'ф' - 25: 1, # 'х' - 22: 2, # 'ц' - 21: 2, # 'ч' - 27: 1, # 'ш' - 24: 1, # 'щ' - 17: 3, # 'ъ' - 52: 1, # 'ь' - 42: 2, # 'ю' - 16: 2, # 'я' - 58: 0, # 'є' - 62: 0, # '№' - }, - 7: { # 'р' - 63: 1, # 'e' - 45: 0, # '\xad' - 31: 0, # 'А' - 32: 0, # 'Б' - 35: 0, # 'В' - 43: 0, # 'Г' - 37: 0, # 'Д' - 44: 0, # 'Е' - 55: 0, # 'Ж' - 47: 0, # 'З' - 40: 0, # 'И' - 59: 0, # 'Й' - 33: 0, # 'К' - 46: 0, # 'Л' - 38: 0, # 'М' - 36: 0, # 'Н' - 41: 0, # 'О' - 30: 0, # 'П' - 39: 0, # 'Р' - 28: 0, # 'С' - 34: 0, # 'Т' - 51: 0, # 'У' - 48: 0, # 'Ф' - 49: 0, # 'Х' - 53: 0, # 'Ц' - 50: 0, # 'Ч' - 54: 0, # 'Ш' - 57: 0, # 'Щ' - 61: 0, # 'Ъ' - 60: 0, # 'Ю' - 56: 0, # 'Я' - 1: 3, # 'а' - 18: 3, # 'б' - 9: 3, # 'в' - 20: 3, # 'г' - 11: 3, # 'д' - 3: 3, # 'е' - 23: 3, # 'ж' - 15: 2, # 'з' - 2: 3, # 'и' - 26: 0, # 'й' - 12: 3, # 'к' - 10: 3, # 'л' - 14: 3, # 'м' - 6: 3, # 'н' - 4: 3, # 'о' - 13: 2, # 'п' - 7: 1, # 'р' - 8: 3, # 'с' - 5: 3, # 'т' - 19: 3, # 'у' - 29: 2, # 'ф' - 25: 3, # 'х' - 22: 3, # 'ц' - 21: 2, # 'ч' - 27: 3, # 'ш' - 24: 1, # 'щ' - 17: 3, # 'ъ' - 52: 1, # 'ь' - 42: 2, # 'ю' - 16: 3, # 'я' - 58: 0, # 'є' - 62: 0, # '№' - }, - 8: { # 'с' - 63: 1, # 'e' - 45: 0, # '\xad' - 31: 0, # 'А' - 32: 0, # 'Б' - 35: 0, # 'В' - 43: 0, # 'Г' - 37: 0, # 'Д' - 44: 0, # 'Е' - 55: 0, # 'Ж' - 47: 0, # 'З' - 40: 0, # 'И' - 59: 0, # 'Й' - 33: 0, # 'К' - 46: 0, # 'Л' - 38: 0, # 'М' - 36: 0, # 'Н' - 41: 0, # 'О' - 30: 0, # 'П' - 39: 0, # 'Р' - 28: 0, # 'С' - 34: 0, # 'Т' - 51: 0, # 'У' - 48: 0, # 'Ф' - 49: 0, # 'Х' - 53: 0, # 'Ц' - 50: 0, # 'Ч' - 54: 0, # 'Ш' - 57: 0, # 'Щ' - 61: 0, # 'Ъ' - 60: 0, # 'Ю' - 56: 0, # 'Я' - 1: 3, # 'а' - 18: 2, # 'б' - 9: 3, # 'в' - 20: 2, # 'г' - 11: 2, # 'д' - 3: 3, # 'е' - 23: 0, # 'ж' - 15: 1, # 'з' - 2: 3, # 'и' - 26: 0, # 'й' - 12: 3, # 'к' - 10: 3, # 'л' - 14: 3, # 'м' - 6: 3, # 'н' - 4: 3, # 'о' - 13: 3, # 'п' - 7: 3, # 'р' - 8: 1, # 'с' - 5: 3, # 'т' - 19: 3, # 'у' - 29: 2, # 'ф' - 25: 2, # 'х' - 22: 2, # 'ц' - 21: 2, # 'ч' - 27: 2, # 'ш' - 24: 0, # 'щ' - 17: 3, # 'ъ' - 52: 2, # 'ь' - 42: 2, # 'ю' - 16: 3, # 'я' - 58: 0, # 'є' - 62: 0, # '№' - }, - 5: { # 'т' - 63: 1, # 'e' - 45: 0, # '\xad' - 31: 0, # 'А' - 32: 0, # 'Б' - 35: 0, # 'В' - 43: 0, # 'Г' - 37: 0, # 'Д' - 44: 0, # 'Е' - 55: 0, # 'Ж' - 47: 0, # 'З' - 40: 0, # 'И' - 59: 0, # 'Й' - 33: 0, # 'К' - 46: 0, # 'Л' - 38: 0, # 'М' - 36: 0, # 'Н' - 41: 0, # 'О' - 30: 0, # 'П' - 39: 0, # 'Р' - 28: 0, # 'С' - 34: 0, # 'Т' - 51: 0, # 'У' - 48: 0, # 'Ф' - 49: 0, # 'Х' - 53: 0, # 'Ц' - 50: 0, # 'Ч' - 54: 0, # 'Ш' - 57: 0, # 'Щ' - 61: 0, # 'Ъ' - 60: 0, # 'Ю' - 56: 0, # 'Я' - 1: 3, # 'а' - 18: 3, # 'б' - 9: 3, # 'в' - 20: 2, # 'г' - 11: 2, # 'д' - 3: 3, # 'е' - 23: 1, # 'ж' - 15: 1, # 'з' - 2: 3, # 'и' - 26: 0, # 'й' - 12: 3, # 'к' - 10: 3, # 'л' - 14: 2, # 'м' - 6: 3, # 'н' - 4: 3, # 'о' - 13: 2, # 'п' - 7: 3, # 'р' - 8: 3, # 'с' - 5: 3, # 'т' - 19: 3, # 'у' - 29: 1, # 'ф' - 25: 2, # 'х' - 22: 2, # 'ц' - 21: 2, # 'ч' - 27: 1, # 'ш' - 24: 1, # 'щ' - 17: 3, # 'ъ' - 52: 2, # 'ь' - 42: 2, # 'ю' - 16: 3, # 'я' - 58: 0, # 'є' - 62: 0, # '№' - }, - 19: { # 'у' - 63: 0, # 'e' - 45: 0, # '\xad' - 31: 0, # 'А' - 32: 0, # 'Б' - 35: 0, # 'В' - 43: 0, # 'Г' - 37: 0, # 'Д' - 44: 0, # 'Е' - 55: 0, # 'Ж' - 47: 0, # 'З' - 40: 0, # 'И' - 59: 0, # 'Й' - 33: 0, # 'К' - 46: 0, # 'Л' - 38: 0, # 'М' - 36: 0, # 'Н' - 41: 0, # 'О' - 30: 0, # 'П' - 39: 0, # 'Р' - 28: 0, # 'С' - 34: 0, # 'Т' - 51: 0, # 'У' - 48: 0, # 'Ф' - 49: 0, # 'Х' - 53: 0, # 'Ц' - 50: 0, # 'Ч' - 54: 0, # 'Ш' - 57: 0, # 'Щ' - 61: 0, # 'Ъ' - 60: 0, # 'Ю' - 56: 0, # 'Я' - 1: 3, # 'а' - 18: 3, # 'б' - 9: 3, # 'в' - 20: 3, # 'г' - 11: 3, # 'д' - 3: 2, # 'е' - 23: 3, # 'ж' - 15: 3, # 'з' - 2: 2, # 'и' - 26: 2, # 'й' - 12: 3, # 'к' - 10: 3, # 'л' - 14: 3, # 'м' - 6: 3, # 'н' - 4: 2, # 'о' - 13: 3, # 'п' - 7: 3, # 'р' - 8: 3, # 'с' - 5: 3, # 'т' - 19: 1, # 'у' - 29: 2, # 'ф' - 25: 2, # 'х' - 22: 2, # 'ц' - 21: 3, # 'ч' - 27: 3, # 'ш' - 24: 2, # 'щ' - 17: 1, # 'ъ' - 52: 0, # 'ь' - 42: 1, # 'ю' - 16: 1, # 'я' - 58: 0, # 'є' - 62: 0, # '№' - }, - 29: { # 'ф' - 63: 1, # 'e' - 45: 0, # '\xad' - 31: 0, # 'А' - 32: 0, # 'Б' - 35: 0, # 'В' - 43: 0, # 'Г' - 37: 0, # 'Д' - 44: 0, # 'Е' - 55: 0, # 'Ж' - 47: 0, # 'З' - 40: 0, # 'И' - 59: 0, # 'Й' - 33: 0, # 'К' - 46: 0, # 'Л' - 38: 0, # 'М' - 36: 0, # 'Н' - 41: 0, # 'О' - 30: 0, # 'П' - 39: 0, # 'Р' - 28: 0, # 'С' - 34: 0, # 'Т' - 51: 0, # 'У' - 48: 0, # 'Ф' - 49: 0, # 'Х' - 53: 0, # 'Ц' - 50: 0, # 'Ч' - 54: 0, # 'Ш' - 57: 0, # 'Щ' - 61: 0, # 'Ъ' - 60: 0, # 'Ю' - 56: 0, # 'Я' - 1: 3, # 'а' - 18: 1, # 'б' - 9: 1, # 'в' - 20: 1, # 'г' - 11: 0, # 'д' - 3: 3, # 'е' - 23: 0, # 'ж' - 15: 0, # 'з' - 2: 3, # 'и' - 26: 0, # 'й' - 12: 2, # 'к' - 10: 2, # 'л' - 14: 1, # 'м' - 6: 1, # 'н' - 4: 3, # 'о' - 13: 0, # 'п' - 7: 2, # 'р' - 8: 2, # 'с' - 5: 2, # 'т' - 19: 2, # 'у' - 29: 0, # 'ф' - 25: 1, # 'х' - 22: 0, # 'ц' - 21: 1, # 'ч' - 27: 1, # 'ш' - 24: 0, # 'щ' - 17: 2, # 'ъ' - 52: 2, # 'ь' - 42: 1, # 'ю' - 16: 1, # 'я' - 58: 0, # 'є' - 62: 0, # '№' - }, - 25: { # 'х' - 63: 0, # 'e' - 45: 0, # '\xad' - 31: 0, # 'А' - 32: 0, # 'Б' - 35: 0, # 'В' - 43: 0, # 'Г' - 37: 0, # 'Д' - 44: 0, # 'Е' - 55: 0, # 'Ж' - 47: 0, # 'З' - 40: 0, # 'И' - 59: 0, # 'Й' - 33: 0, # 'К' - 46: 0, # 'Л' - 38: 0, # 'М' - 36: 0, # 'Н' - 41: 0, # 'О' - 30: 0, # 'П' - 39: 0, # 'Р' - 28: 0, # 'С' - 34: 0, # 'Т' - 51: 0, # 'У' - 48: 0, # 'Ф' - 49: 0, # 'Х' - 53: 0, # 'Ц' - 50: 0, # 'Ч' - 54: 0, # 'Ш' - 57: 0, # 'Щ' - 61: 0, # 'Ъ' - 60: 0, # 'Ю' - 56: 0, # 'Я' - 1: 3, # 'а' - 18: 1, # 'б' - 9: 3, # 'в' - 20: 0, # 'г' - 11: 1, # 'д' - 3: 2, # 'е' - 23: 0, # 'ж' - 15: 1, # 'з' - 2: 3, # 'и' - 26: 0, # 'й' - 12: 1, # 'к' - 10: 2, # 'л' - 14: 2, # 'м' - 6: 3, # 'н' - 4: 3, # 'о' - 13: 1, # 'п' - 7: 3, # 'р' - 8: 1, # 'с' - 5: 2, # 'т' - 19: 3, # 'у' - 29: 0, # 'ф' - 25: 1, # 'х' - 22: 0, # 'ц' - 21: 1, # 'ч' - 27: 0, # 'ш' - 24: 0, # 'щ' - 17: 2, # 'ъ' - 52: 0, # 'ь' - 42: 1, # 'ю' - 16: 1, # 'я' - 58: 0, # 'є' - 62: 0, # '№' - }, - 22: { # 'ц' - 63: 1, # 'e' - 45: 0, # '\xad' - 31: 0, # 'А' - 32: 0, # 'Б' - 35: 0, # 'В' - 43: 0, # 'Г' - 37: 0, # 'Д' - 44: 0, # 'Е' - 55: 0, # 'Ж' - 47: 0, # 'З' - 40: 0, # 'И' - 59: 0, # 'Й' - 33: 0, # 'К' - 46: 0, # 'Л' - 38: 0, # 'М' - 36: 0, # 'Н' - 41: 0, # 'О' - 30: 0, # 'П' - 39: 0, # 'Р' - 28: 0, # 'С' - 34: 0, # 'Т' - 51: 0, # 'У' - 48: 0, # 'Ф' - 49: 0, # 'Х' - 53: 0, # 'Ц' - 50: 0, # 'Ч' - 54: 0, # 'Ш' - 57: 0, # 'Щ' - 61: 0, # 'Ъ' - 60: 0, # 'Ю' - 56: 0, # 'Я' - 1: 3, # 'а' - 18: 1, # 'б' - 9: 2, # 'в' - 20: 1, # 'г' - 11: 1, # 'д' - 3: 3, # 'е' - 23: 0, # 'ж' - 15: 1, # 'з' - 2: 3, # 'и' - 26: 0, # 'й' - 12: 2, # 'к' - 10: 1, # 'л' - 14: 1, # 'м' - 6: 1, # 'н' - 4: 2, # 'о' - 13: 1, # 'п' - 7: 1, # 'р' - 8: 1, # 'с' - 5: 1, # 'т' - 19: 2, # 'у' - 29: 1, # 'ф' - 25: 1, # 'х' - 22: 1, # 'ц' - 21: 1, # 'ч' - 27: 1, # 'ш' - 24: 1, # 'щ' - 17: 2, # 'ъ' - 52: 1, # 'ь' - 42: 0, # 'ю' - 16: 2, # 'я' - 58: 0, # 'є' - 62: 0, # '№' - }, - 21: { # 'ч' - 63: 1, # 'e' - 45: 0, # '\xad' - 31: 0, # 'А' - 32: 0, # 'Б' - 35: 0, # 'В' - 43: 0, # 'Г' - 37: 0, # 'Д' - 44: 0, # 'Е' - 55: 0, # 'Ж' - 47: 0, # 'З' - 40: 0, # 'И' - 59: 0, # 'Й' - 33: 0, # 'К' - 46: 0, # 'Л' - 38: 0, # 'М' - 36: 0, # 'Н' - 41: 0, # 'О' - 30: 0, # 'П' - 39: 0, # 'Р' - 28: 0, # 'С' - 34: 0, # 'Т' - 51: 0, # 'У' - 48: 0, # 'Ф' - 49: 0, # 'Х' - 53: 0, # 'Ц' - 50: 0, # 'Ч' - 54: 0, # 'Ш' - 57: 0, # 'Щ' - 61: 0, # 'Ъ' - 60: 0, # 'Ю' - 56: 0, # 'Я' - 1: 3, # 'а' - 18: 1, # 'б' - 9: 3, # 'в' - 20: 1, # 'г' - 11: 0, # 'д' - 3: 3, # 'е' - 23: 1, # 'ж' - 15: 0, # 'з' - 2: 3, # 'и' - 26: 0, # 'й' - 12: 3, # 'к' - 10: 2, # 'л' - 14: 2, # 'м' - 6: 3, # 'н' - 4: 3, # 'о' - 13: 0, # 'п' - 7: 2, # 'р' - 8: 0, # 'с' - 5: 2, # 'т' - 19: 3, # 'у' - 29: 0, # 'ф' - 25: 0, # 'х' - 22: 0, # 'ц' - 21: 0, # 'ч' - 27: 1, # 'ш' - 24: 0, # 'щ' - 17: 2, # 'ъ' - 52: 0, # 'ь' - 42: 1, # 'ю' - 16: 0, # 'я' - 58: 0, # 'є' - 62: 0, # '№' - }, - 27: { # 'ш' - 63: 1, # 'e' - 45: 0, # '\xad' - 31: 0, # 'А' - 32: 0, # 'Б' - 35: 0, # 'В' - 43: 0, # 'Г' - 37: 0, # 'Д' - 44: 0, # 'Е' - 55: 0, # 'Ж' - 47: 0, # 'З' - 40: 0, # 'И' - 59: 0, # 'Й' - 33: 0, # 'К' - 46: 0, # 'Л' - 38: 0, # 'М' - 36: 0, # 'Н' - 41: 0, # 'О' - 30: 0, # 'П' - 39: 0, # 'Р' - 28: 0, # 'С' - 34: 0, # 'Т' - 51: 0, # 'У' - 48: 0, # 'Ф' - 49: 0, # 'Х' - 53: 0, # 'Ц' - 50: 0, # 'Ч' - 54: 0, # 'Ш' - 57: 0, # 'Щ' - 61: 0, # 'Ъ' - 60: 0, # 'Ю' - 56: 0, # 'Я' - 1: 3, # 'а' - 18: 0, # 'б' - 9: 2, # 'в' - 20: 0, # 'г' - 11: 1, # 'д' - 3: 3, # 'е' - 23: 0, # 'ж' - 15: 0, # 'з' - 2: 3, # 'и' - 26: 0, # 'й' - 12: 3, # 'к' - 10: 2, # 'л' - 14: 1, # 'м' - 6: 3, # 'н' - 4: 2, # 'о' - 13: 2, # 'п' - 7: 1, # 'р' - 8: 0, # 'с' - 5: 1, # 'т' - 19: 2, # 'у' - 29: 1, # 'ф' - 25: 0, # 'х' - 22: 0, # 'ц' - 21: 1, # 'ч' - 27: 0, # 'ш' - 24: 0, # 'щ' - 17: 2, # 'ъ' - 52: 1, # 'ь' - 42: 1, # 'ю' - 16: 0, # 'я' - 58: 0, # 'є' - 62: 0, # '№' - }, - 24: { # 'щ' - 63: 1, # 'e' - 45: 0, # '\xad' - 31: 0, # 'А' - 32: 0, # 'Б' - 35: 0, # 'В' - 43: 0, # 'Г' - 37: 0, # 'Д' - 44: 0, # 'Е' - 55: 0, # 'Ж' - 47: 0, # 'З' - 40: 0, # 'И' - 59: 0, # 'Й' - 33: 0, # 'К' - 46: 0, # 'Л' - 38: 0, # 'М' - 36: 0, # 'Н' - 41: 0, # 'О' - 30: 0, # 'П' - 39: 0, # 'Р' - 28: 0, # 'С' - 34: 0, # 'Т' - 51: 0, # 'У' - 48: 0, # 'Ф' - 49: 0, # 'Х' - 53: 0, # 'Ц' - 50: 0, # 'Ч' - 54: 0, # 'Ш' - 57: 0, # 'Щ' - 61: 0, # 'Ъ' - 60: 0, # 'Ю' - 56: 0, # 'Я' - 1: 3, # 'а' - 18: 0, # 'б' - 9: 1, # 'в' - 20: 0, # 'г' - 11: 0, # 'д' - 3: 3, # 'е' - 23: 0, # 'ж' - 15: 0, # 'з' - 2: 3, # 'и' - 26: 0, # 'й' - 12: 1, # 'к' - 10: 0, # 'л' - 14: 0, # 'м' - 6: 2, # 'н' - 4: 3, # 'о' - 13: 0, # 'п' - 7: 1, # 'р' - 8: 0, # 'с' - 5: 2, # 'т' - 19: 3, # 'у' - 29: 0, # 'ф' - 25: 0, # 'х' - 22: 1, # 'ц' - 21: 0, # 'ч' - 27: 0, # 'ш' - 24: 0, # 'щ' - 17: 1, # 'ъ' - 52: 0, # 'ь' - 42: 0, # 'ю' - 16: 2, # 'я' - 58: 0, # 'є' - 62: 0, # '№' - }, - 17: { # 'ъ' - 63: 0, # 'e' - 45: 0, # '\xad' - 31: 0, # 'А' - 32: 0, # 'Б' - 35: 0, # 'В' - 43: 0, # 'Г' - 37: 0, # 'Д' - 44: 0, # 'Е' - 55: 0, # 'Ж' - 47: 0, # 'З' - 40: 0, # 'И' - 59: 0, # 'Й' - 33: 0, # 'К' - 46: 0, # 'Л' - 38: 0, # 'М' - 36: 0, # 'Н' - 41: 0, # 'О' - 30: 0, # 'П' - 39: 0, # 'Р' - 28: 0, # 'С' - 34: 0, # 'Т' - 51: 0, # 'У' - 48: 0, # 'Ф' - 49: 0, # 'Х' - 53: 0, # 'Ц' - 50: 0, # 'Ч' - 54: 0, # 'Ш' - 57: 0, # 'Щ' - 61: 0, # 'Ъ' - 60: 0, # 'Ю' - 56: 0, # 'Я' - 1: 1, # 'а' - 18: 3, # 'б' - 9: 3, # 'в' - 20: 3, # 'г' - 11: 3, # 'д' - 3: 2, # 'е' - 23: 3, # 'ж' - 15: 3, # 'з' - 2: 1, # 'и' - 26: 2, # 'й' - 12: 3, # 'к' - 10: 3, # 'л' - 14: 3, # 'м' - 6: 3, # 'н' - 4: 3, # 'о' - 13: 3, # 'п' - 7: 3, # 'р' - 8: 3, # 'с' - 5: 3, # 'т' - 19: 1, # 'у' - 29: 1, # 'ф' - 25: 2, # 'х' - 22: 2, # 'ц' - 21: 3, # 'ч' - 27: 2, # 'ш' - 24: 3, # 'щ' - 17: 0, # 'ъ' - 52: 0, # 'ь' - 42: 2, # 'ю' - 16: 0, # 'я' - 58: 0, # 'є' - 62: 0, # '№' - }, - 52: { # 'ь' - 63: 0, # 'e' - 45: 0, # '\xad' - 31: 0, # 'А' - 32: 0, # 'Б' - 35: 0, # 'В' - 43: 0, # 'Г' - 37: 0, # 'Д' - 44: 0, # 'Е' - 55: 0, # 'Ж' - 47: 0, # 'З' - 40: 0, # 'И' - 59: 0, # 'Й' - 33: 0, # 'К' - 46: 0, # 'Л' - 38: 0, # 'М' - 36: 0, # 'Н' - 41: 0, # 'О' - 30: 0, # 'П' - 39: 0, # 'Р' - 28: 0, # 'С' - 34: 0, # 'Т' - 51: 0, # 'У' - 48: 0, # 'Ф' - 49: 0, # 'Х' - 53: 0, # 'Ц' - 50: 0, # 'Ч' - 54: 0, # 'Ш' - 57: 0, # 'Щ' - 61: 0, # 'Ъ' - 60: 0, # 'Ю' - 56: 0, # 'Я' - 1: 0, # 'а' - 18: 0, # 'б' - 9: 0, # 'в' - 20: 0, # 'г' - 11: 0, # 'д' - 3: 1, # 'е' - 23: 0, # 'ж' - 15: 0, # 'з' - 2: 0, # 'и' - 26: 0, # 'й' - 12: 1, # 'к' - 10: 0, # 'л' - 14: 0, # 'м' - 6: 1, # 'н' - 4: 3, # 'о' - 13: 0, # 'п' - 7: 0, # 'р' - 8: 0, # 'с' - 5: 1, # 'т' - 19: 0, # 'у' - 29: 0, # 'ф' - 25: 0, # 'х' - 22: 1, # 'ц' - 21: 0, # 'ч' - 27: 0, # 'ш' - 24: 0, # 'щ' - 17: 0, # 'ъ' - 52: 0, # 'ь' - 42: 1, # 'ю' - 16: 0, # 'я' - 58: 0, # 'є' - 62: 0, # '№' - }, - 42: { # 'ю' - 63: 0, # 'e' - 45: 0, # '\xad' - 31: 0, # 'А' - 32: 0, # 'Б' - 35: 0, # 'В' - 43: 0, # 'Г' - 37: 0, # 'Д' - 44: 0, # 'Е' - 55: 0, # 'Ж' - 47: 0, # 'З' - 40: 0, # 'И' - 59: 0, # 'Й' - 33: 0, # 'К' - 46: 0, # 'Л' - 38: 0, # 'М' - 36: 0, # 'Н' - 41: 0, # 'О' - 30: 0, # 'П' - 39: 0, # 'Р' - 28: 0, # 'С' - 34: 0, # 'Т' - 51: 0, # 'У' - 48: 0, # 'Ф' - 49: 0, # 'Х' - 53: 0, # 'Ц' - 50: 0, # 'Ч' - 54: 0, # 'Ш' - 57: 0, # 'Щ' - 61: 0, # 'Ъ' - 60: 0, # 'Ю' - 56: 0, # 'Я' - 1: 1, # 'а' - 18: 2, # 'б' - 9: 1, # 'в' - 20: 2, # 'г' - 11: 2, # 'д' - 3: 1, # 'е' - 23: 2, # 'ж' - 15: 2, # 'з' - 2: 1, # 'и' - 26: 1, # 'й' - 12: 2, # 'к' - 10: 2, # 'л' - 14: 2, # 'м' - 6: 2, # 'н' - 4: 1, # 'о' - 13: 1, # 'п' - 7: 2, # 'р' - 8: 2, # 'с' - 5: 2, # 'т' - 19: 1, # 'у' - 29: 1, # 'ф' - 25: 1, # 'х' - 22: 2, # 'ц' - 21: 3, # 'ч' - 27: 1, # 'ш' - 24: 1, # 'щ' - 17: 1, # 'ъ' - 52: 0, # 'ь' - 42: 0, # 'ю' - 16: 1, # 'я' - 58: 0, # 'є' - 62: 0, # '№' - }, - 16: { # 'я' - 63: 0, # 'e' - 45: 1, # '\xad' - 31: 0, # 'А' - 32: 0, # 'Б' - 35: 0, # 'В' - 43: 0, # 'Г' - 37: 0, # 'Д' - 44: 0, # 'Е' - 55: 0, # 'Ж' - 47: 0, # 'З' - 40: 0, # 'И' - 59: 0, # 'Й' - 33: 0, # 'К' - 46: 0, # 'Л' - 38: 0, # 'М' - 36: 0, # 'Н' - 41: 0, # 'О' - 30: 0, # 'П' - 39: 0, # 'Р' - 28: 0, # 'С' - 34: 0, # 'Т' - 51: 0, # 'У' - 48: 0, # 'Ф' - 49: 0, # 'Х' - 53: 0, # 'Ц' - 50: 0, # 'Ч' - 54: 0, # 'Ш' - 57: 0, # 'Щ' - 61: 0, # 'Ъ' - 60: 0, # 'Ю' - 56: 0, # 'Я' - 1: 0, # 'а' - 18: 3, # 'б' - 9: 3, # 'в' - 20: 2, # 'г' - 11: 3, # 'д' - 3: 2, # 'е' - 23: 1, # 'ж' - 15: 2, # 'з' - 2: 1, # 'и' - 26: 2, # 'й' - 12: 3, # 'к' - 10: 3, # 'л' - 14: 3, # 'м' - 6: 3, # 'н' - 4: 1, # 'о' - 13: 2, # 'п' - 7: 2, # 'р' - 8: 3, # 'с' - 5: 3, # 'т' - 19: 1, # 'у' - 29: 1, # 'ф' - 25: 3, # 'х' - 22: 2, # 'ц' - 21: 1, # 'ч' - 27: 1, # 'ш' - 24: 2, # 'щ' - 17: 0, # 'ъ' - 52: 0, # 'ь' - 42: 0, # 'ю' - 16: 1, # 'я' - 58: 0, # 'є' - 62: 0, # '№' - }, - 58: { # 'є' - 63: 0, # 'e' - 45: 0, # '\xad' - 31: 0, # 'А' - 32: 0, # 'Б' - 35: 0, # 'В' - 43: 0, # 'Г' - 37: 0, # 'Д' - 44: 0, # 'Е' - 55: 0, # 'Ж' - 47: 0, # 'З' - 40: 0, # 'И' - 59: 0, # 'Й' - 33: 0, # 'К' - 46: 0, # 'Л' - 38: 0, # 'М' - 36: 0, # 'Н' - 41: 0, # 'О' - 30: 0, # 'П' - 39: 0, # 'Р' - 28: 0, # 'С' - 34: 0, # 'Т' - 51: 0, # 'У' - 48: 0, # 'Ф' - 49: 0, # 'Х' - 53: 0, # 'Ц' - 50: 0, # 'Ч' - 54: 0, # 'Ш' - 57: 0, # 'Щ' - 61: 0, # 'Ъ' - 60: 0, # 'Ю' - 56: 0, # 'Я' - 1: 0, # 'а' - 18: 0, # 'б' - 9: 0, # 'в' - 20: 0, # 'г' - 11: 0, # 'д' - 3: 0, # 'е' - 23: 0, # 'ж' - 15: 0, # 'з' - 2: 0, # 'и' - 26: 0, # 'й' - 12: 0, # 'к' - 10: 0, # 'л' - 14: 0, # 'м' - 6: 0, # 'н' - 4: 0, # 'о' - 13: 0, # 'п' - 7: 0, # 'р' - 8: 0, # 'с' - 5: 0, # 'т' - 19: 0, # 'у' - 29: 0, # 'ф' - 25: 0, # 'х' - 22: 0, # 'ц' - 21: 0, # 'ч' - 27: 0, # 'ш' - 24: 0, # 'щ' - 17: 0, # 'ъ' - 52: 0, # 'ь' - 42: 0, # 'ю' - 16: 0, # 'я' - 58: 0, # 'є' - 62: 0, # '№' - }, - 62: { # '№' - 63: 0, # 'e' - 45: 0, # '\xad' - 31: 0, # 'А' - 32: 0, # 'Б' - 35: 0, # 'В' - 43: 0, # 'Г' - 37: 0, # 'Д' - 44: 0, # 'Е' - 55: 0, # 'Ж' - 47: 0, # 'З' - 40: 0, # 'И' - 59: 0, # 'Й' - 33: 0, # 'К' - 46: 0, # 'Л' - 38: 0, # 'М' - 36: 0, # 'Н' - 41: 0, # 'О' - 30: 0, # 'П' - 39: 0, # 'Р' - 28: 0, # 'С' - 34: 0, # 'Т' - 51: 0, # 'У' - 48: 0, # 'Ф' - 49: 0, # 'Х' - 53: 0, # 'Ц' - 50: 0, # 'Ч' - 54: 0, # 'Ш' - 57: 0, # 'Щ' - 61: 0, # 'Ъ' - 60: 0, # 'Ю' - 56: 0, # 'Я' - 1: 0, # 'а' - 18: 0, # 'б' - 9: 0, # 'в' - 20: 0, # 'г' - 11: 0, # 'д' - 3: 0, # 'е' - 23: 0, # 'ж' - 15: 0, # 'з' - 2: 0, # 'и' - 26: 0, # 'й' - 12: 0, # 'к' - 10: 0, # 'л' - 14: 0, # 'м' - 6: 0, # 'н' - 4: 0, # 'о' - 13: 0, # 'п' - 7: 0, # 'р' - 8: 0, # 'с' - 5: 0, # 'т' - 19: 0, # 'у' - 29: 0, # 'ф' - 25: 0, # 'х' - 22: 0, # 'ц' - 21: 0, # 'ч' - 27: 0, # 'ш' - 24: 0, # 'щ' - 17: 0, # 'ъ' - 52: 0, # 'ь' - 42: 0, # 'ю' - 16: 0, # 'я' - 58: 0, # 'є' - 62: 0, # '№' - }, -} - -# 255: Undefined characters that did not exist in training text -# 254: Carriage/Return -# 253: symbol (punctuation) that does not belong to word -# 252: 0 - 9 -# 251: Control characters - -# Character Mapping Table(s): -ISO_8859_5_BULGARIAN_CHAR_TO_ORDER = { - 0: 255, # '\x00' - 1: 255, # '\x01' - 2: 255, # '\x02' - 3: 255, # '\x03' - 4: 255, # '\x04' - 5: 255, # '\x05' - 6: 255, # '\x06' - 7: 255, # '\x07' - 8: 255, # '\x08' - 9: 255, # '\t' - 10: 254, # '\n' - 11: 255, # '\x0b' - 12: 255, # '\x0c' - 13: 254, # '\r' - 14: 255, # '\x0e' - 15: 255, # '\x0f' - 16: 255, # '\x10' - 17: 255, # '\x11' - 18: 255, # '\x12' - 19: 255, # '\x13' - 20: 255, # '\x14' - 21: 255, # '\x15' - 22: 255, # '\x16' - 23: 255, # '\x17' - 24: 255, # '\x18' - 25: 255, # '\x19' - 26: 255, # '\x1a' - 27: 255, # '\x1b' - 28: 255, # '\x1c' - 29: 255, # '\x1d' - 30: 255, # '\x1e' - 31: 255, # '\x1f' - 32: 253, # ' ' - 33: 253, # '!' - 34: 253, # '"' - 35: 253, # '#' - 36: 253, # '$' - 37: 253, # '%' - 38: 253, # '&' - 39: 253, # "'" - 40: 253, # '(' - 41: 253, # ')' - 42: 253, # '*' - 43: 253, # '+' - 44: 253, # ',' - 45: 253, # '-' - 46: 253, # '.' - 47: 253, # '/' - 48: 252, # '0' - 49: 252, # '1' - 50: 252, # '2' - 51: 252, # '3' - 52: 252, # '4' - 53: 252, # '5' - 54: 252, # '6' - 55: 252, # '7' - 56: 252, # '8' - 57: 252, # '9' - 58: 253, # ':' - 59: 253, # ';' - 60: 253, # '<' - 61: 253, # '=' - 62: 253, # '>' - 63: 253, # '?' - 64: 253, # '@' - 65: 77, # 'A' - 66: 90, # 'B' - 67: 99, # 'C' - 68: 100, # 'D' - 69: 72, # 'E' - 70: 109, # 'F' - 71: 107, # 'G' - 72: 101, # 'H' - 73: 79, # 'I' - 74: 185, # 'J' - 75: 81, # 'K' - 76: 102, # 'L' - 77: 76, # 'M' - 78: 94, # 'N' - 79: 82, # 'O' - 80: 110, # 'P' - 81: 186, # 'Q' - 82: 108, # 'R' - 83: 91, # 'S' - 84: 74, # 'T' - 85: 119, # 'U' - 86: 84, # 'V' - 87: 96, # 'W' - 88: 111, # 'X' - 89: 187, # 'Y' - 90: 115, # 'Z' - 91: 253, # '[' - 92: 253, # '\\' - 93: 253, # ']' - 94: 253, # '^' - 95: 253, # '_' - 96: 253, # '`' - 97: 65, # 'a' - 98: 69, # 'b' - 99: 70, # 'c' - 100: 66, # 'd' - 101: 63, # 'e' - 102: 68, # 'f' - 103: 112, # 'g' - 104: 103, # 'h' - 105: 92, # 'i' - 106: 194, # 'j' - 107: 104, # 'k' - 108: 95, # 'l' - 109: 86, # 'm' - 110: 87, # 'n' - 111: 71, # 'o' - 112: 116, # 'p' - 113: 195, # 'q' - 114: 85, # 'r' - 115: 93, # 's' - 116: 97, # 't' - 117: 113, # 'u' - 118: 196, # 'v' - 119: 197, # 'w' - 120: 198, # 'x' - 121: 199, # 'y' - 122: 200, # 'z' - 123: 253, # '{' - 124: 253, # '|' - 125: 253, # '}' - 126: 253, # '~' - 127: 253, # '\x7f' - 128: 194, # '\x80' - 129: 195, # '\x81' - 130: 196, # '\x82' - 131: 197, # '\x83' - 132: 198, # '\x84' - 133: 199, # '\x85' - 134: 200, # '\x86' - 135: 201, # '\x87' - 136: 202, # '\x88' - 137: 203, # '\x89' - 138: 204, # '\x8a' - 139: 205, # '\x8b' - 140: 206, # '\x8c' - 141: 207, # '\x8d' - 142: 208, # '\x8e' - 143: 209, # '\x8f' - 144: 210, # '\x90' - 145: 211, # '\x91' - 146: 212, # '\x92' - 147: 213, # '\x93' - 148: 214, # '\x94' - 149: 215, # '\x95' - 150: 216, # '\x96' - 151: 217, # '\x97' - 152: 218, # '\x98' - 153: 219, # '\x99' - 154: 220, # '\x9a' - 155: 221, # '\x9b' - 156: 222, # '\x9c' - 157: 223, # '\x9d' - 158: 224, # '\x9e' - 159: 225, # '\x9f' - 160: 81, # '\xa0' - 161: 226, # 'Ё' - 162: 227, # 'Ђ' - 163: 228, # 'Ѓ' - 164: 229, # 'Є' - 165: 230, # 'Ѕ' - 166: 105, # 'І' - 167: 231, # 'Ї' - 168: 232, # 'Ј' - 169: 233, # 'Љ' - 170: 234, # 'Њ' - 171: 235, # 'Ћ' - 172: 236, # 'Ќ' - 173: 45, # '\xad' - 174: 237, # 'Ў' - 175: 238, # 'Џ' - 176: 31, # 'А' - 177: 32, # 'Б' - 178: 35, # 'В' - 179: 43, # 'Г' - 180: 37, # 'Д' - 181: 44, # 'Е' - 182: 55, # 'Ж' - 183: 47, # 'З' - 184: 40, # 'И' - 185: 59, # 'Й' - 186: 33, # 'К' - 187: 46, # 'Л' - 188: 38, # 'М' - 189: 36, # 'Н' - 190: 41, # 'О' - 191: 30, # 'П' - 192: 39, # 'Р' - 193: 28, # 'С' - 194: 34, # 'Т' - 195: 51, # 'У' - 196: 48, # 'Ф' - 197: 49, # 'Х' - 198: 53, # 'Ц' - 199: 50, # 'Ч' - 200: 54, # 'Ш' - 201: 57, # 'Щ' - 202: 61, # 'Ъ' - 203: 239, # 'Ы' - 204: 67, # 'Ь' - 205: 240, # 'Э' - 206: 60, # 'Ю' - 207: 56, # 'Я' - 208: 1, # 'а' - 209: 18, # 'б' - 210: 9, # 'в' - 211: 20, # 'г' - 212: 11, # 'д' - 213: 3, # 'е' - 214: 23, # 'ж' - 215: 15, # 'з' - 216: 2, # 'и' - 217: 26, # 'й' - 218: 12, # 'к' - 219: 10, # 'л' - 220: 14, # 'м' - 221: 6, # 'н' - 222: 4, # 'о' - 223: 13, # 'п' - 224: 7, # 'р' - 225: 8, # 'с' - 226: 5, # 'т' - 227: 19, # 'у' - 228: 29, # 'ф' - 229: 25, # 'х' - 230: 22, # 'ц' - 231: 21, # 'ч' - 232: 27, # 'ш' - 233: 24, # 'щ' - 234: 17, # 'ъ' - 235: 75, # 'ы' - 236: 52, # 'ь' - 237: 241, # 'э' - 238: 42, # 'ю' - 239: 16, # 'я' - 240: 62, # '№' - 241: 242, # 'ё' - 242: 243, # 'ђ' - 243: 244, # 'ѓ' - 244: 58, # 'є' - 245: 245, # 'ѕ' - 246: 98, # 'і' - 247: 246, # 'ї' - 248: 247, # 'ј' - 249: 248, # 'љ' - 250: 249, # 'њ' - 251: 250, # 'ћ' - 252: 251, # 'ќ' - 253: 91, # '§' - 254: 252, # 'ў' - 255: 253, # 'џ' -} - -ISO_8859_5_BULGARIAN_MODEL = SingleByteCharSetModel( - charset_name="ISO-8859-5", - language="Bulgarian", - char_to_order_map=ISO_8859_5_BULGARIAN_CHAR_TO_ORDER, - language_model=BULGARIAN_LANG_MODEL, - typical_positive_ratio=0.969392, - keep_ascii_letters=False, - alphabet="АБВГДЕЖЗИЙКЛМНОПРСТУФХЦЧШЩЪЬЮЯабвгдежзийклмнопрстуфхцчшщъьюя", -) - -WINDOWS_1251_BULGARIAN_CHAR_TO_ORDER = { - 0: 255, # '\x00' - 1: 255, # '\x01' - 2: 255, # '\x02' - 3: 255, # '\x03' - 4: 255, # '\x04' - 5: 255, # '\x05' - 6: 255, # '\x06' - 7: 255, # '\x07' - 8: 255, # '\x08' - 9: 255, # '\t' - 10: 254, # '\n' - 11: 255, # '\x0b' - 12: 255, # '\x0c' - 13: 254, # '\r' - 14: 255, # '\x0e' - 15: 255, # '\x0f' - 16: 255, # '\x10' - 17: 255, # '\x11' - 18: 255, # '\x12' - 19: 255, # '\x13' - 20: 255, # '\x14' - 21: 255, # '\x15' - 22: 255, # '\x16' - 23: 255, # '\x17' - 24: 255, # '\x18' - 25: 255, # '\x19' - 26: 255, # '\x1a' - 27: 255, # '\x1b' - 28: 255, # '\x1c' - 29: 255, # '\x1d' - 30: 255, # '\x1e' - 31: 255, # '\x1f' - 32: 253, # ' ' - 33: 253, # '!' - 34: 253, # '"' - 35: 253, # '#' - 36: 253, # '$' - 37: 253, # '%' - 38: 253, # '&' - 39: 253, # "'" - 40: 253, # '(' - 41: 253, # ')' - 42: 253, # '*' - 43: 253, # '+' - 44: 253, # ',' - 45: 253, # '-' - 46: 253, # '.' - 47: 253, # '/' - 48: 252, # '0' - 49: 252, # '1' - 50: 252, # '2' - 51: 252, # '3' - 52: 252, # '4' - 53: 252, # '5' - 54: 252, # '6' - 55: 252, # '7' - 56: 252, # '8' - 57: 252, # '9' - 58: 253, # ':' - 59: 253, # ';' - 60: 253, # '<' - 61: 253, # '=' - 62: 253, # '>' - 63: 253, # '?' - 64: 253, # '@' - 65: 77, # 'A' - 66: 90, # 'B' - 67: 99, # 'C' - 68: 100, # 'D' - 69: 72, # 'E' - 70: 109, # 'F' - 71: 107, # 'G' - 72: 101, # 'H' - 73: 79, # 'I' - 74: 185, # 'J' - 75: 81, # 'K' - 76: 102, # 'L' - 77: 76, # 'M' - 78: 94, # 'N' - 79: 82, # 'O' - 80: 110, # 'P' - 81: 186, # 'Q' - 82: 108, # 'R' - 83: 91, # 'S' - 84: 74, # 'T' - 85: 119, # 'U' - 86: 84, # 'V' - 87: 96, # 'W' - 88: 111, # 'X' - 89: 187, # 'Y' - 90: 115, # 'Z' - 91: 253, # '[' - 92: 253, # '\\' - 93: 253, # ']' - 94: 253, # '^' - 95: 253, # '_' - 96: 253, # '`' - 97: 65, # 'a' - 98: 69, # 'b' - 99: 70, # 'c' - 100: 66, # 'd' - 101: 63, # 'e' - 102: 68, # 'f' - 103: 112, # 'g' - 104: 103, # 'h' - 105: 92, # 'i' - 106: 194, # 'j' - 107: 104, # 'k' - 108: 95, # 'l' - 109: 86, # 'm' - 110: 87, # 'n' - 111: 71, # 'o' - 112: 116, # 'p' - 113: 195, # 'q' - 114: 85, # 'r' - 115: 93, # 's' - 116: 97, # 't' - 117: 113, # 'u' - 118: 196, # 'v' - 119: 197, # 'w' - 120: 198, # 'x' - 121: 199, # 'y' - 122: 200, # 'z' - 123: 253, # '{' - 124: 253, # '|' - 125: 253, # '}' - 126: 253, # '~' - 127: 253, # '\x7f' - 128: 206, # 'Ђ' - 129: 207, # 'Ѓ' - 130: 208, # '‚' - 131: 209, # 'ѓ' - 132: 210, # '„' - 133: 211, # '…' - 134: 212, # '†' - 135: 213, # '‡' - 136: 120, # '€' - 137: 214, # '‰' - 138: 215, # 'Љ' - 139: 216, # '‹' - 140: 217, # 'Њ' - 141: 218, # 'Ќ' - 142: 219, # 'Ћ' - 143: 220, # 'Џ' - 144: 221, # 'ђ' - 145: 78, # '‘' - 146: 64, # '’' - 147: 83, # '“' - 148: 121, # '”' - 149: 98, # '•' - 150: 117, # '–' - 151: 105, # '—' - 152: 222, # None - 153: 223, # '™' - 154: 224, # 'љ' - 155: 225, # '›' - 156: 226, # 'њ' - 157: 227, # 'ќ' - 158: 228, # 'ћ' - 159: 229, # 'џ' - 160: 88, # '\xa0' - 161: 230, # 'Ў' - 162: 231, # 'ў' - 163: 232, # 'Ј' - 164: 233, # '¤' - 165: 122, # 'Ґ' - 166: 89, # '¦' - 167: 106, # '§' - 168: 234, # 'Ё' - 169: 235, # '©' - 170: 236, # 'Є' - 171: 237, # '«' - 172: 238, # '¬' - 173: 45, # '\xad' - 174: 239, # '®' - 175: 240, # 'Ї' - 176: 73, # '°' - 177: 80, # '±' - 178: 118, # 'І' - 179: 114, # 'і' - 180: 241, # 'ґ' - 181: 242, # 'µ' - 182: 243, # '¶' - 183: 244, # '·' - 184: 245, # 'ё' - 185: 62, # '№' - 186: 58, # 'є' - 187: 246, # '»' - 188: 247, # 'ј' - 189: 248, # 'Ѕ' - 190: 249, # 'ѕ' - 191: 250, # 'ї' - 192: 31, # 'А' - 193: 32, # 'Б' - 194: 35, # 'В' - 195: 43, # 'Г' - 196: 37, # 'Д' - 197: 44, # 'Е' - 198: 55, # 'Ж' - 199: 47, # 'З' - 200: 40, # 'И' - 201: 59, # 'Й' - 202: 33, # 'К' - 203: 46, # 'Л' - 204: 38, # 'М' - 205: 36, # 'Н' - 206: 41, # 'О' - 207: 30, # 'П' - 208: 39, # 'Р' - 209: 28, # 'С' - 210: 34, # 'Т' - 211: 51, # 'У' - 212: 48, # 'Ф' - 213: 49, # 'Х' - 214: 53, # 'Ц' - 215: 50, # 'Ч' - 216: 54, # 'Ш' - 217: 57, # 'Щ' - 218: 61, # 'Ъ' - 219: 251, # 'Ы' - 220: 67, # 'Ь' - 221: 252, # 'Э' - 222: 60, # 'Ю' - 223: 56, # 'Я' - 224: 1, # 'а' - 225: 18, # 'б' - 226: 9, # 'в' - 227: 20, # 'г' - 228: 11, # 'д' - 229: 3, # 'е' - 230: 23, # 'ж' - 231: 15, # 'з' - 232: 2, # 'и' - 233: 26, # 'й' - 234: 12, # 'к' - 235: 10, # 'л' - 236: 14, # 'м' - 237: 6, # 'н' - 238: 4, # 'о' - 239: 13, # 'п' - 240: 7, # 'р' - 241: 8, # 'с' - 242: 5, # 'т' - 243: 19, # 'у' - 244: 29, # 'ф' - 245: 25, # 'х' - 246: 22, # 'ц' - 247: 21, # 'ч' - 248: 27, # 'ш' - 249: 24, # 'щ' - 250: 17, # 'ъ' - 251: 75, # 'ы' - 252: 52, # 'ь' - 253: 253, # 'э' - 254: 42, # 'ю' - 255: 16, # 'я' -} - -WINDOWS_1251_BULGARIAN_MODEL = SingleByteCharSetModel( - charset_name="windows-1251", - language="Bulgarian", - char_to_order_map=WINDOWS_1251_BULGARIAN_CHAR_TO_ORDER, - language_model=BULGARIAN_LANG_MODEL, - typical_positive_ratio=0.969392, - keep_ascii_letters=False, - alphabet="АБВГДЕЖЗИЙКЛМНОПРСТУФХЦЧШЩЪЬЮЯабвгдежзийклмнопрстуфхцчшщъьюя", -) diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/langgreekmodel.py b/venv/lib/python3.11/site-packages/pip/_vendor/chardet/langgreekmodel.py deleted file mode 100644 index cfb8639..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/langgreekmodel.py +++ /dev/null @@ -1,4397 +0,0 @@ -from pip._vendor.chardet.sbcharsetprober import SingleByteCharSetModel - -# 3: Positive -# 2: Likely -# 1: Unlikely -# 0: Negative - -GREEK_LANG_MODEL = { - 60: { # 'e' - 60: 2, # 'e' - 55: 1, # 'o' - 58: 2, # 't' - 36: 1, # '·' - 61: 0, # 'Ά' - 46: 0, # 'Έ' - 54: 0, # 'Ό' - 31: 0, # 'Α' - 51: 0, # 'Β' - 43: 0, # 'Γ' - 41: 0, # 'Δ' - 34: 0, # 'Ε' - 40: 0, # 'Η' - 52: 0, # 'Θ' - 47: 0, # 'Ι' - 44: 0, # 'Κ' - 53: 0, # 'Λ' - 38: 0, # 'Μ' - 49: 0, # 'Ν' - 59: 0, # 'Ξ' - 39: 0, # 'Ο' - 35: 0, # 'Π' - 48: 0, # 'Ρ' - 37: 0, # 'Σ' - 33: 0, # 'Τ' - 45: 0, # 'Υ' - 56: 0, # 'Φ' - 50: 1, # 'Χ' - 57: 0, # 'Ω' - 17: 0, # 'ά' - 18: 0, # 'έ' - 22: 0, # 'ή' - 15: 0, # 'ί' - 1: 0, # 'α' - 29: 0, # 'β' - 20: 0, # 'γ' - 21: 0, # 'δ' - 3: 0, # 'ε' - 32: 0, # 'ζ' - 13: 0, # 'η' - 25: 0, # 'θ' - 5: 0, # 'ι' - 11: 0, # 'κ' - 16: 0, # 'λ' - 10: 0, # 'μ' - 6: 0, # 'ν' - 30: 0, # 'ξ' - 4: 0, # 'ο' - 9: 0, # 'π' - 8: 0, # 'ρ' - 14: 0, # 'ς' - 7: 0, # 'σ' - 2: 0, # 'τ' - 12: 0, # 'υ' - 28: 0, # 'φ' - 23: 0, # 'χ' - 42: 0, # 'ψ' - 24: 0, # 'ω' - 19: 0, # 'ό' - 26: 0, # 'ύ' - 27: 0, # 'ώ' - }, - 55: { # 'o' - 60: 0, # 'e' - 55: 2, # 'o' - 58: 2, # 't' - 36: 1, # '·' - 61: 0, # 'Ά' - 46: 0, # 'Έ' - 54: 0, # 'Ό' - 31: 0, # 'Α' - 51: 0, # 'Β' - 43: 0, # 'Γ' - 41: 0, # 'Δ' - 34: 0, # 'Ε' - 40: 0, # 'Η' - 52: 0, # 'Θ' - 47: 0, # 'Ι' - 44: 0, # 'Κ' - 53: 0, # 'Λ' - 38: 0, # 'Μ' - 49: 0, # 'Ν' - 59: 0, # 'Ξ' - 39: 0, # 'Ο' - 35: 0, # 'Π' - 48: 0, # 'Ρ' - 37: 0, # 'Σ' - 33: 0, # 'Τ' - 45: 0, # 'Υ' - 56: 0, # 'Φ' - 50: 0, # 'Χ' - 57: 0, # 'Ω' - 17: 0, # 'ά' - 18: 0, # 'έ' - 22: 0, # 'ή' - 15: 0, # 'ί' - 1: 0, # 'α' - 29: 0, # 'β' - 20: 0, # 'γ' - 21: 0, # 'δ' - 3: 0, # 'ε' - 32: 0, # 'ζ' - 13: 0, # 'η' - 25: 0, # 'θ' - 5: 0, # 'ι' - 11: 0, # 'κ' - 16: 0, # 'λ' - 10: 0, # 'μ' - 6: 1, # 'ν' - 30: 0, # 'ξ' - 4: 0, # 'ο' - 9: 0, # 'π' - 8: 0, # 'ρ' - 14: 0, # 'ς' - 7: 0, # 'σ' - 2: 0, # 'τ' - 12: 1, # 'υ' - 28: 0, # 'φ' - 23: 0, # 'χ' - 42: 0, # 'ψ' - 24: 0, # 'ω' - 19: 0, # 'ό' - 26: 0, # 'ύ' - 27: 0, # 'ώ' - }, - 58: { # 't' - 60: 2, # 'e' - 55: 1, # 'o' - 58: 1, # 't' - 36: 0, # '·' - 61: 0, # 'Ά' - 46: 0, # 'Έ' - 54: 0, # 'Ό' - 31: 0, # 'Α' - 51: 0, # 'Β' - 43: 0, # 'Γ' - 41: 0, # 'Δ' - 34: 0, # 'Ε' - 40: 0, # 'Η' - 52: 0, # 'Θ' - 47: 0, # 'Ι' - 44: 0, # 'Κ' - 53: 0, # 'Λ' - 38: 0, # 'Μ' - 49: 0, # 'Ν' - 59: 0, # 'Ξ' - 39: 0, # 'Ο' - 35: 0, # 'Π' - 48: 0, # 'Ρ' - 37: 0, # 'Σ' - 33: 0, # 'Τ' - 45: 0, # 'Υ' - 56: 0, # 'Φ' - 50: 0, # 'Χ' - 57: 0, # 'Ω' - 17: 2, # 'ά' - 18: 0, # 'έ' - 22: 0, # 'ή' - 15: 0, # 'ί' - 1: 0, # 'α' - 29: 0, # 'β' - 20: 0, # 'γ' - 21: 0, # 'δ' - 3: 0, # 'ε' - 32: 0, # 'ζ' - 13: 0, # 'η' - 25: 0, # 'θ' - 5: 0, # 'ι' - 11: 0, # 'κ' - 16: 0, # 'λ' - 10: 0, # 'μ' - 6: 0, # 'ν' - 30: 0, # 'ξ' - 4: 1, # 'ο' - 9: 0, # 'π' - 8: 0, # 'ρ' - 14: 0, # 'ς' - 7: 0, # 'σ' - 2: 0, # 'τ' - 12: 0, # 'υ' - 28: 0, # 'φ' - 23: 0, # 'χ' - 42: 0, # 'ψ' - 24: 0, # 'ω' - 19: 0, # 'ό' - 26: 0, # 'ύ' - 27: 0, # 'ώ' - }, - 36: { # '·' - 60: 0, # 'e' - 55: 0, # 'o' - 58: 0, # 't' - 36: 0, # '·' - 61: 0, # 'Ά' - 46: 0, # 'Έ' - 54: 0, # 'Ό' - 31: 0, # 'Α' - 51: 0, # 'Β' - 43: 0, # 'Γ' - 41: 0, # 'Δ' - 34: 0, # 'Ε' - 40: 0, # 'Η' - 52: 0, # 'Θ' - 47: 0, # 'Ι' - 44: 0, # 'Κ' - 53: 0, # 'Λ' - 38: 0, # 'Μ' - 49: 0, # 'Ν' - 59: 0, # 'Ξ' - 39: 0, # 'Ο' - 35: 0, # 'Π' - 48: 0, # 'Ρ' - 37: 0, # 'Σ' - 33: 0, # 'Τ' - 45: 0, # 'Υ' - 56: 0, # 'Φ' - 50: 0, # 'Χ' - 57: 0, # 'Ω' - 17: 0, # 'ά' - 18: 0, # 'έ' - 22: 0, # 'ή' - 15: 0, # 'ί' - 1: 0, # 'α' - 29: 0, # 'β' - 20: 0, # 'γ' - 21: 0, # 'δ' - 3: 0, # 'ε' - 32: 0, # 'ζ' - 13: 0, # 'η' - 25: 0, # 'θ' - 5: 0, # 'ι' - 11: 0, # 'κ' - 16: 0, # 'λ' - 10: 0, # 'μ' - 6: 0, # 'ν' - 30: 0, # 'ξ' - 4: 0, # 'ο' - 9: 0, # 'π' - 8: 0, # 'ρ' - 14: 0, # 'ς' - 7: 0, # 'σ' - 2: 0, # 'τ' - 12: 0, # 'υ' - 28: 0, # 'φ' - 23: 0, # 'χ' - 42: 0, # 'ψ' - 24: 0, # 'ω' - 19: 0, # 'ό' - 26: 0, # 'ύ' - 27: 0, # 'ώ' - }, - 61: { # 'Ά' - 60: 0, # 'e' - 55: 0, # 'o' - 58: 0, # 't' - 36: 0, # '·' - 61: 0, # 'Ά' - 46: 0, # 'Έ' - 54: 0, # 'Ό' - 31: 0, # 'Α' - 51: 0, # 'Β' - 43: 0, # 'Γ' - 41: 0, # 'Δ' - 34: 0, # 'Ε' - 40: 0, # 'Η' - 52: 0, # 'Θ' - 47: 0, # 'Ι' - 44: 0, # 'Κ' - 53: 0, # 'Λ' - 38: 0, # 'Μ' - 49: 0, # 'Ν' - 59: 0, # 'Ξ' - 39: 0, # 'Ο' - 35: 0, # 'Π' - 48: 0, # 'Ρ' - 37: 0, # 'Σ' - 33: 0, # 'Τ' - 45: 0, # 'Υ' - 56: 0, # 'Φ' - 50: 0, # 'Χ' - 57: 0, # 'Ω' - 17: 0, # 'ά' - 18: 0, # 'έ' - 22: 0, # 'ή' - 15: 0, # 'ί' - 1: 0, # 'α' - 29: 0, # 'β' - 20: 1, # 'γ' - 21: 2, # 'δ' - 3: 0, # 'ε' - 32: 0, # 'ζ' - 13: 0, # 'η' - 25: 0, # 'θ' - 5: 0, # 'ι' - 11: 0, # 'κ' - 16: 2, # 'λ' - 10: 0, # 'μ' - 6: 0, # 'ν' - 30: 0, # 'ξ' - 4: 0, # 'ο' - 9: 1, # 'π' - 8: 2, # 'ρ' - 14: 0, # 'ς' - 7: 0, # 'σ' - 2: 0, # 'τ' - 12: 0, # 'υ' - 28: 0, # 'φ' - 23: 0, # 'χ' - 42: 0, # 'ψ' - 24: 0, # 'ω' - 19: 0, # 'ό' - 26: 0, # 'ύ' - 27: 0, # 'ώ' - }, - 46: { # 'Έ' - 60: 0, # 'e' - 55: 0, # 'o' - 58: 0, # 't' - 36: 0, # '·' - 61: 0, # 'Ά' - 46: 0, # 'Έ' - 54: 0, # 'Ό' - 31: 0, # 'Α' - 51: 0, # 'Β' - 43: 0, # 'Γ' - 41: 0, # 'Δ' - 34: 0, # 'Ε' - 40: 0, # 'Η' - 52: 0, # 'Θ' - 47: 0, # 'Ι' - 44: 0, # 'Κ' - 53: 0, # 'Λ' - 38: 0, # 'Μ' - 49: 0, # 'Ν' - 59: 0, # 'Ξ' - 39: 0, # 'Ο' - 35: 0, # 'Π' - 48: 0, # 'Ρ' - 37: 0, # 'Σ' - 33: 0, # 'Τ' - 45: 0, # 'Υ' - 56: 0, # 'Φ' - 50: 0, # 'Χ' - 57: 0, # 'Ω' - 17: 0, # 'ά' - 18: 0, # 'έ' - 22: 0, # 'ή' - 15: 0, # 'ί' - 1: 0, # 'α' - 29: 2, # 'β' - 20: 2, # 'γ' - 21: 0, # 'δ' - 3: 0, # 'ε' - 32: 0, # 'ζ' - 13: 0, # 'η' - 25: 0, # 'θ' - 5: 0, # 'ι' - 11: 2, # 'κ' - 16: 2, # 'λ' - 10: 0, # 'μ' - 6: 3, # 'ν' - 30: 2, # 'ξ' - 4: 0, # 'ο' - 9: 2, # 'π' - 8: 2, # 'ρ' - 14: 0, # 'ς' - 7: 1, # 'σ' - 2: 2, # 'τ' - 12: 0, # 'υ' - 28: 2, # 'φ' - 23: 3, # 'χ' - 42: 0, # 'ψ' - 24: 0, # 'ω' - 19: 0, # 'ό' - 26: 0, # 'ύ' - 27: 0, # 'ώ' - }, - 54: { # 'Ό' - 60: 0, # 'e' - 55: 0, # 'o' - 58: 0, # 't' - 36: 0, # '·' - 61: 0, # 'Ά' - 46: 0, # 'Έ' - 54: 0, # 'Ό' - 31: 0, # 'Α' - 51: 0, # 'Β' - 43: 0, # 'Γ' - 41: 0, # 'Δ' - 34: 0, # 'Ε' - 40: 0, # 'Η' - 52: 0, # 'Θ' - 47: 0, # 'Ι' - 44: 0, # 'Κ' - 53: 0, # 'Λ' - 38: 0, # 'Μ' - 49: 0, # 'Ν' - 59: 0, # 'Ξ' - 39: 0, # 'Ο' - 35: 0, # 'Π' - 48: 0, # 'Ρ' - 37: 0, # 'Σ' - 33: 0, # 'Τ' - 45: 0, # 'Υ' - 56: 0, # 'Φ' - 50: 0, # 'Χ' - 57: 0, # 'Ω' - 17: 0, # 'ά' - 18: 0, # 'έ' - 22: 0, # 'ή' - 15: 0, # 'ί' - 1: 0, # 'α' - 29: 0, # 'β' - 20: 0, # 'γ' - 21: 0, # 'δ' - 3: 0, # 'ε' - 32: 0, # 'ζ' - 13: 0, # 'η' - 25: 0, # 'θ' - 5: 0, # 'ι' - 11: 0, # 'κ' - 16: 2, # 'λ' - 10: 2, # 'μ' - 6: 2, # 'ν' - 30: 0, # 'ξ' - 4: 0, # 'ο' - 9: 2, # 'π' - 8: 0, # 'ρ' - 14: 0, # 'ς' - 7: 2, # 'σ' - 2: 3, # 'τ' - 12: 0, # 'υ' - 28: 0, # 'φ' - 23: 2, # 'χ' - 42: 0, # 'ψ' - 24: 0, # 'ω' - 19: 0, # 'ό' - 26: 0, # 'ύ' - 27: 0, # 'ώ' - }, - 31: { # 'Α' - 60: 0, # 'e' - 55: 0, # 'o' - 58: 0, # 't' - 36: 0, # '·' - 61: 0, # 'Ά' - 46: 0, # 'Έ' - 54: 0, # 'Ό' - 31: 0, # 'Α' - 51: 2, # 'Β' - 43: 2, # 'Γ' - 41: 1, # 'Δ' - 34: 0, # 'Ε' - 40: 0, # 'Η' - 52: 2, # 'Θ' - 47: 2, # 'Ι' - 44: 2, # 'Κ' - 53: 2, # 'Λ' - 38: 2, # 'Μ' - 49: 2, # 'Ν' - 59: 1, # 'Ξ' - 39: 0, # 'Ο' - 35: 2, # 'Π' - 48: 2, # 'Ρ' - 37: 2, # 'Σ' - 33: 2, # 'Τ' - 45: 2, # 'Υ' - 56: 2, # 'Φ' - 50: 0, # 'Χ' - 57: 0, # 'Ω' - 17: 0, # 'ά' - 18: 0, # 'έ' - 22: 0, # 'ή' - 15: 0, # 'ί' - 1: 0, # 'α' - 29: 0, # 'β' - 20: 2, # 'γ' - 21: 0, # 'δ' - 3: 0, # 'ε' - 32: 0, # 'ζ' - 13: 0, # 'η' - 25: 1, # 'θ' - 5: 0, # 'ι' - 11: 2, # 'κ' - 16: 3, # 'λ' - 10: 2, # 'μ' - 6: 3, # 'ν' - 30: 2, # 'ξ' - 4: 0, # 'ο' - 9: 3, # 'π' - 8: 3, # 'ρ' - 14: 2, # 'ς' - 7: 2, # 'σ' - 2: 0, # 'τ' - 12: 3, # 'υ' - 28: 2, # 'φ' - 23: 0, # 'χ' - 42: 0, # 'ψ' - 24: 0, # 'ω' - 19: 0, # 'ό' - 26: 2, # 'ύ' - 27: 0, # 'ώ' - }, - 51: { # 'Β' - 60: 0, # 'e' - 55: 0, # 'o' - 58: 0, # 't' - 36: 0, # '·' - 61: 0, # 'Ά' - 46: 0, # 'Έ' - 54: 0, # 'Ό' - 31: 2, # 'Α' - 51: 0, # 'Β' - 43: 0, # 'Γ' - 41: 0, # 'Δ' - 34: 1, # 'Ε' - 40: 1, # 'Η' - 52: 0, # 'Θ' - 47: 1, # 'Ι' - 44: 0, # 'Κ' - 53: 1, # 'Λ' - 38: 0, # 'Μ' - 49: 0, # 'Ν' - 59: 0, # 'Ξ' - 39: 2, # 'Ο' - 35: 0, # 'Π' - 48: 0, # 'Ρ' - 37: 0, # 'Σ' - 33: 0, # 'Τ' - 45: 0, # 'Υ' - 56: 0, # 'Φ' - 50: 0, # 'Χ' - 57: 0, # 'Ω' - 17: 2, # 'ά' - 18: 2, # 'έ' - 22: 2, # 'ή' - 15: 0, # 'ί' - 1: 2, # 'α' - 29: 0, # 'β' - 20: 0, # 'γ' - 21: 0, # 'δ' - 3: 2, # 'ε' - 32: 0, # 'ζ' - 13: 0, # 'η' - 25: 0, # 'θ' - 5: 2, # 'ι' - 11: 0, # 'κ' - 16: 2, # 'λ' - 10: 0, # 'μ' - 6: 0, # 'ν' - 30: 0, # 'ξ' - 4: 2, # 'ο' - 9: 0, # 'π' - 8: 2, # 'ρ' - 14: 0, # 'ς' - 7: 0, # 'σ' - 2: 0, # 'τ' - 12: 0, # 'υ' - 28: 0, # 'φ' - 23: 0, # 'χ' - 42: 0, # 'ψ' - 24: 0, # 'ω' - 19: 0, # 'ό' - 26: 0, # 'ύ' - 27: 0, # 'ώ' - }, - 43: { # 'Γ' - 60: 0, # 'e' - 55: 0, # 'o' - 58: 0, # 't' - 36: 0, # '·' - 61: 0, # 'Ά' - 46: 0, # 'Έ' - 54: 0, # 'Ό' - 31: 1, # 'Α' - 51: 0, # 'Β' - 43: 2, # 'Γ' - 41: 0, # 'Δ' - 34: 2, # 'Ε' - 40: 1, # 'Η' - 52: 0, # 'Θ' - 47: 2, # 'Ι' - 44: 1, # 'Κ' - 53: 1, # 'Λ' - 38: 0, # 'Μ' - 49: 0, # 'Ν' - 59: 0, # 'Ξ' - 39: 1, # 'Ο' - 35: 0, # 'Π' - 48: 2, # 'Ρ' - 37: 0, # 'Σ' - 33: 0, # 'Τ' - 45: 2, # 'Υ' - 56: 0, # 'Φ' - 50: 1, # 'Χ' - 57: 2, # 'Ω' - 17: 0, # 'ά' - 18: 0, # 'έ' - 22: 0, # 'ή' - 15: 2, # 'ί' - 1: 2, # 'α' - 29: 0, # 'β' - 20: 0, # 'γ' - 21: 0, # 'δ' - 3: 2, # 'ε' - 32: 0, # 'ζ' - 13: 0, # 'η' - 25: 0, # 'θ' - 5: 3, # 'ι' - 11: 0, # 'κ' - 16: 2, # 'λ' - 10: 0, # 'μ' - 6: 2, # 'ν' - 30: 0, # 'ξ' - 4: 0, # 'ο' - 9: 0, # 'π' - 8: 2, # 'ρ' - 14: 0, # 'ς' - 7: 0, # 'σ' - 2: 0, # 'τ' - 12: 0, # 'υ' - 28: 0, # 'φ' - 23: 0, # 'χ' - 42: 0, # 'ψ' - 24: 0, # 'ω' - 19: 0, # 'ό' - 26: 0, # 'ύ' - 27: 0, # 'ώ' - }, - 41: { # 'Δ' - 60: 0, # 'e' - 55: 0, # 'o' - 58: 0, # 't' - 36: 0, # '·' - 61: 0, # 'Ά' - 46: 0, # 'Έ' - 54: 0, # 'Ό' - 31: 0, # 'Α' - 51: 0, # 'Β' - 43: 0, # 'Γ' - 41: 0, # 'Δ' - 34: 2, # 'Ε' - 40: 2, # 'Η' - 52: 0, # 'Θ' - 47: 2, # 'Ι' - 44: 0, # 'Κ' - 53: 0, # 'Λ' - 38: 0, # 'Μ' - 49: 0, # 'Ν' - 59: 0, # 'Ξ' - 39: 2, # 'Ο' - 35: 0, # 'Π' - 48: 0, # 'Ρ' - 37: 0, # 'Σ' - 33: 0, # 'Τ' - 45: 0, # 'Υ' - 56: 0, # 'Φ' - 50: 0, # 'Χ' - 57: 2, # 'Ω' - 17: 0, # 'ά' - 18: 0, # 'έ' - 22: 2, # 'ή' - 15: 2, # 'ί' - 1: 0, # 'α' - 29: 0, # 'β' - 20: 0, # 'γ' - 21: 0, # 'δ' - 3: 3, # 'ε' - 32: 0, # 'ζ' - 13: 2, # 'η' - 25: 0, # 'θ' - 5: 3, # 'ι' - 11: 0, # 'κ' - 16: 0, # 'λ' - 10: 0, # 'μ' - 6: 0, # 'ν' - 30: 0, # 'ξ' - 4: 2, # 'ο' - 9: 0, # 'π' - 8: 2, # 'ρ' - 14: 0, # 'ς' - 7: 0, # 'σ' - 2: 0, # 'τ' - 12: 2, # 'υ' - 28: 0, # 'φ' - 23: 0, # 'χ' - 42: 0, # 'ψ' - 24: 2, # 'ω' - 19: 1, # 'ό' - 26: 2, # 'ύ' - 27: 2, # 'ώ' - }, - 34: { # 'Ε' - 60: 0, # 'e' - 55: 0, # 'o' - 58: 0, # 't' - 36: 0, # '·' - 61: 0, # 'Ά' - 46: 0, # 'Έ' - 54: 0, # 'Ό' - 31: 2, # 'Α' - 51: 0, # 'Β' - 43: 2, # 'Γ' - 41: 2, # 'Δ' - 34: 0, # 'Ε' - 40: 0, # 'Η' - 52: 0, # 'Θ' - 47: 2, # 'Ι' - 44: 2, # 'Κ' - 53: 2, # 'Λ' - 38: 2, # 'Μ' - 49: 2, # 'Ν' - 59: 1, # 'Ξ' - 39: 0, # 'Ο' - 35: 2, # 'Π' - 48: 2, # 'Ρ' - 37: 2, # 'Σ' - 33: 2, # 'Τ' - 45: 2, # 'Υ' - 56: 0, # 'Φ' - 50: 2, # 'Χ' - 57: 2, # 'Ω' - 17: 3, # 'ά' - 18: 0, # 'έ' - 22: 0, # 'ή' - 15: 3, # 'ί' - 1: 0, # 'α' - 29: 0, # 'β' - 20: 3, # 'γ' - 21: 2, # 'δ' - 3: 1, # 'ε' - 32: 0, # 'ζ' - 13: 0, # 'η' - 25: 1, # 'θ' - 5: 2, # 'ι' - 11: 3, # 'κ' - 16: 3, # 'λ' - 10: 2, # 'μ' - 6: 3, # 'ν' - 30: 2, # 'ξ' - 4: 0, # 'ο' - 9: 3, # 'π' - 8: 2, # 'ρ' - 14: 0, # 'ς' - 7: 2, # 'σ' - 2: 2, # 'τ' - 12: 2, # 'υ' - 28: 2, # 'φ' - 23: 0, # 'χ' - 42: 0, # 'ψ' - 24: 0, # 'ω' - 19: 0, # 'ό' - 26: 1, # 'ύ' - 27: 0, # 'ώ' - }, - 40: { # 'Η' - 60: 0, # 'e' - 55: 0, # 'o' - 58: 0, # 't' - 36: 0, # '·' - 61: 0, # 'Ά' - 46: 0, # 'Έ' - 54: 0, # 'Ό' - 31: 0, # 'Α' - 51: 0, # 'Β' - 43: 1, # 'Γ' - 41: 0, # 'Δ' - 34: 0, # 'Ε' - 40: 0, # 'Η' - 52: 2, # 'Θ' - 47: 0, # 'Ι' - 44: 2, # 'Κ' - 53: 0, # 'Λ' - 38: 2, # 'Μ' - 49: 2, # 'Ν' - 59: 0, # 'Ξ' - 39: 0, # 'Ο' - 35: 2, # 'Π' - 48: 2, # 'Ρ' - 37: 2, # 'Σ' - 33: 2, # 'Τ' - 45: 1, # 'Υ' - 56: 0, # 'Φ' - 50: 0, # 'Χ' - 57: 0, # 'Ω' - 17: 0, # 'ά' - 18: 0, # 'έ' - 22: 0, # 'ή' - 15: 0, # 'ί' - 1: 0, # 'α' - 29: 0, # 'β' - 20: 0, # 'γ' - 21: 0, # 'δ' - 3: 0, # 'ε' - 32: 0, # 'ζ' - 13: 0, # 'η' - 25: 0, # 'θ' - 5: 0, # 'ι' - 11: 0, # 'κ' - 16: 2, # 'λ' - 10: 0, # 'μ' - 6: 1, # 'ν' - 30: 0, # 'ξ' - 4: 0, # 'ο' - 9: 0, # 'π' - 8: 0, # 'ρ' - 14: 0, # 'ς' - 7: 0, # 'σ' - 2: 0, # 'τ' - 12: 0, # 'υ' - 28: 0, # 'φ' - 23: 1, # 'χ' - 42: 0, # 'ψ' - 24: 0, # 'ω' - 19: 0, # 'ό' - 26: 0, # 'ύ' - 27: 0, # 'ώ' - }, - 52: { # 'Θ' - 60: 0, # 'e' - 55: 0, # 'o' - 58: 0, # 't' - 36: 0, # '·' - 61: 0, # 'Ά' - 46: 0, # 'Έ' - 54: 0, # 'Ό' - 31: 2, # 'Α' - 51: 0, # 'Β' - 43: 0, # 'Γ' - 41: 0, # 'Δ' - 34: 2, # 'Ε' - 40: 2, # 'Η' - 52: 0, # 'Θ' - 47: 0, # 'Ι' - 44: 0, # 'Κ' - 53: 0, # 'Λ' - 38: 0, # 'Μ' - 49: 0, # 'Ν' - 59: 0, # 'Ξ' - 39: 2, # 'Ο' - 35: 0, # 'Π' - 48: 1, # 'Ρ' - 37: 0, # 'Σ' - 33: 0, # 'Τ' - 45: 1, # 'Υ' - 56: 0, # 'Φ' - 50: 0, # 'Χ' - 57: 0, # 'Ω' - 17: 0, # 'ά' - 18: 2, # 'έ' - 22: 0, # 'ή' - 15: 0, # 'ί' - 1: 3, # 'α' - 29: 0, # 'β' - 20: 0, # 'γ' - 21: 0, # 'δ' - 3: 2, # 'ε' - 32: 0, # 'ζ' - 13: 0, # 'η' - 25: 0, # 'θ' - 5: 0, # 'ι' - 11: 0, # 'κ' - 16: 0, # 'λ' - 10: 0, # 'μ' - 6: 0, # 'ν' - 30: 0, # 'ξ' - 4: 0, # 'ο' - 9: 0, # 'π' - 8: 0, # 'ρ' - 14: 0, # 'ς' - 7: 0, # 'σ' - 2: 0, # 'τ' - 12: 2, # 'υ' - 28: 0, # 'φ' - 23: 0, # 'χ' - 42: 0, # 'ψ' - 24: 0, # 'ω' - 19: 0, # 'ό' - 26: 2, # 'ύ' - 27: 0, # 'ώ' - }, - 47: { # 'Ι' - 60: 0, # 'e' - 55: 0, # 'o' - 58: 0, # 't' - 36: 0, # '·' - 61: 0, # 'Ά' - 46: 0, # 'Έ' - 54: 0, # 'Ό' - 31: 2, # 'Α' - 51: 1, # 'Β' - 43: 1, # 'Γ' - 41: 2, # 'Δ' - 34: 2, # 'Ε' - 40: 2, # 'Η' - 52: 0, # 'Θ' - 47: 0, # 'Ι' - 44: 2, # 'Κ' - 53: 2, # 'Λ' - 38: 2, # 'Μ' - 49: 2, # 'Ν' - 59: 0, # 'Ξ' - 39: 2, # 'Ο' - 35: 0, # 'Π' - 48: 2, # 'Ρ' - 37: 2, # 'Σ' - 33: 2, # 'Τ' - 45: 0, # 'Υ' - 56: 2, # 'Φ' - 50: 0, # 'Χ' - 57: 2, # 'Ω' - 17: 0, # 'ά' - 18: 0, # 'έ' - 22: 0, # 'ή' - 15: 0, # 'ί' - 1: 2, # 'α' - 29: 0, # 'β' - 20: 0, # 'γ' - 21: 2, # 'δ' - 3: 0, # 'ε' - 32: 0, # 'ζ' - 13: 0, # 'η' - 25: 0, # 'θ' - 5: 0, # 'ι' - 11: 0, # 'κ' - 16: 0, # 'λ' - 10: 0, # 'μ' - 6: 1, # 'ν' - 30: 0, # 'ξ' - 4: 2, # 'ο' - 9: 0, # 'π' - 8: 0, # 'ρ' - 14: 0, # 'ς' - 7: 2, # 'σ' - 2: 1, # 'τ' - 12: 0, # 'υ' - 28: 0, # 'φ' - 23: 0, # 'χ' - 42: 0, # 'ψ' - 24: 1, # 'ω' - 19: 0, # 'ό' - 26: 0, # 'ύ' - 27: 0, # 'ώ' - }, - 44: { # 'Κ' - 60: 0, # 'e' - 55: 0, # 'o' - 58: 0, # 't' - 36: 0, # '·' - 61: 0, # 'Ά' - 46: 0, # 'Έ' - 54: 0, # 'Ό' - 31: 2, # 'Α' - 51: 0, # 'Β' - 43: 0, # 'Γ' - 41: 1, # 'Δ' - 34: 2, # 'Ε' - 40: 2, # 'Η' - 52: 0, # 'Θ' - 47: 0, # 'Ι' - 44: 0, # 'Κ' - 53: 0, # 'Λ' - 38: 1, # 'Μ' - 49: 0, # 'Ν' - 59: 0, # 'Ξ' - 39: 2, # 'Ο' - 35: 0, # 'Π' - 48: 2, # 'Ρ' - 37: 0, # 'Σ' - 33: 1, # 'Τ' - 45: 2, # 'Υ' - 56: 0, # 'Φ' - 50: 0, # 'Χ' - 57: 1, # 'Ω' - 17: 3, # 'ά' - 18: 0, # 'έ' - 22: 0, # 'ή' - 15: 0, # 'ί' - 1: 3, # 'α' - 29: 0, # 'β' - 20: 0, # 'γ' - 21: 0, # 'δ' - 3: 2, # 'ε' - 32: 0, # 'ζ' - 13: 0, # 'η' - 25: 0, # 'θ' - 5: 2, # 'ι' - 11: 0, # 'κ' - 16: 2, # 'λ' - 10: 0, # 'μ' - 6: 0, # 'ν' - 30: 0, # 'ξ' - 4: 2, # 'ο' - 9: 0, # 'π' - 8: 2, # 'ρ' - 14: 0, # 'ς' - 7: 0, # 'σ' - 2: 0, # 'τ' - 12: 2, # 'υ' - 28: 0, # 'φ' - 23: 0, # 'χ' - 42: 0, # 'ψ' - 24: 0, # 'ω' - 19: 2, # 'ό' - 26: 2, # 'ύ' - 27: 2, # 'ώ' - }, - 53: { # 'Λ' - 60: 0, # 'e' - 55: 0, # 'o' - 58: 0, # 't' - 36: 0, # '·' - 61: 0, # 'Ά' - 46: 0, # 'Έ' - 54: 0, # 'Ό' - 31: 2, # 'Α' - 51: 0, # 'Β' - 43: 0, # 'Γ' - 41: 0, # 'Δ' - 34: 2, # 'Ε' - 40: 2, # 'Η' - 52: 0, # 'Θ' - 47: 2, # 'Ι' - 44: 0, # 'Κ' - 53: 2, # 'Λ' - 38: 0, # 'Μ' - 49: 0, # 'Ν' - 59: 0, # 'Ξ' - 39: 2, # 'Ο' - 35: 0, # 'Π' - 48: 0, # 'Ρ' - 37: 2, # 'Σ' - 33: 0, # 'Τ' - 45: 2, # 'Υ' - 56: 0, # 'Φ' - 50: 0, # 'Χ' - 57: 2, # 'Ω' - 17: 2, # 'ά' - 18: 2, # 'έ' - 22: 0, # 'ή' - 15: 2, # 'ί' - 1: 2, # 'α' - 29: 0, # 'β' - 20: 0, # 'γ' - 21: 0, # 'δ' - 3: 2, # 'ε' - 32: 0, # 'ζ' - 13: 0, # 'η' - 25: 0, # 'θ' - 5: 1, # 'ι' - 11: 0, # 'κ' - 16: 0, # 'λ' - 10: 0, # 'μ' - 6: 0, # 'ν' - 30: 0, # 'ξ' - 4: 2, # 'ο' - 9: 0, # 'π' - 8: 0, # 'ρ' - 14: 0, # 'ς' - 7: 0, # 'σ' - 2: 0, # 'τ' - 12: 2, # 'υ' - 28: 0, # 'φ' - 23: 0, # 'χ' - 42: 0, # 'ψ' - 24: 0, # 'ω' - 19: 2, # 'ό' - 26: 2, # 'ύ' - 27: 0, # 'ώ' - }, - 38: { # 'Μ' - 60: 0, # 'e' - 55: 0, # 'o' - 58: 0, # 't' - 36: 0, # '·' - 61: 0, # 'Ά' - 46: 0, # 'Έ' - 54: 0, # 'Ό' - 31: 2, # 'Α' - 51: 2, # 'Β' - 43: 0, # 'Γ' - 41: 0, # 'Δ' - 34: 2, # 'Ε' - 40: 2, # 'Η' - 52: 0, # 'Θ' - 47: 2, # 'Ι' - 44: 0, # 'Κ' - 53: 0, # 'Λ' - 38: 2, # 'Μ' - 49: 0, # 'Ν' - 59: 0, # 'Ξ' - 39: 2, # 'Ο' - 35: 2, # 'Π' - 48: 0, # 'Ρ' - 37: 0, # 'Σ' - 33: 0, # 'Τ' - 45: 0, # 'Υ' - 56: 0, # 'Φ' - 50: 0, # 'Χ' - 57: 0, # 'Ω' - 17: 2, # 'ά' - 18: 2, # 'έ' - 22: 2, # 'ή' - 15: 2, # 'ί' - 1: 2, # 'α' - 29: 0, # 'β' - 20: 0, # 'γ' - 21: 0, # 'δ' - 3: 3, # 'ε' - 32: 0, # 'ζ' - 13: 2, # 'η' - 25: 0, # 'θ' - 5: 3, # 'ι' - 11: 0, # 'κ' - 16: 0, # 'λ' - 10: 0, # 'μ' - 6: 0, # 'ν' - 30: 0, # 'ξ' - 4: 2, # 'ο' - 9: 3, # 'π' - 8: 0, # 'ρ' - 14: 0, # 'ς' - 7: 0, # 'σ' - 2: 0, # 'τ' - 12: 2, # 'υ' - 28: 0, # 'φ' - 23: 0, # 'χ' - 42: 0, # 'ψ' - 24: 0, # 'ω' - 19: 2, # 'ό' - 26: 0, # 'ύ' - 27: 0, # 'ώ' - }, - 49: { # 'Ν' - 60: 2, # 'e' - 55: 0, # 'o' - 58: 0, # 't' - 36: 0, # '·' - 61: 0, # 'Ά' - 46: 0, # 'Έ' - 54: 0, # 'Ό' - 31: 2, # 'Α' - 51: 0, # 'Β' - 43: 0, # 'Γ' - 41: 0, # 'Δ' - 34: 2, # 'Ε' - 40: 2, # 'Η' - 52: 0, # 'Θ' - 47: 2, # 'Ι' - 44: 0, # 'Κ' - 53: 0, # 'Λ' - 38: 0, # 'Μ' - 49: 0, # 'Ν' - 59: 0, # 'Ξ' - 39: 2, # 'Ο' - 35: 0, # 'Π' - 48: 0, # 'Ρ' - 37: 0, # 'Σ' - 33: 2, # 'Τ' - 45: 0, # 'Υ' - 56: 0, # 'Φ' - 50: 0, # 'Χ' - 57: 2, # 'Ω' - 17: 0, # 'ά' - 18: 2, # 'έ' - 22: 0, # 'ή' - 15: 2, # 'ί' - 1: 2, # 'α' - 29: 0, # 'β' - 20: 0, # 'γ' - 21: 0, # 'δ' - 3: 1, # 'ε' - 32: 0, # 'ζ' - 13: 0, # 'η' - 25: 0, # 'θ' - 5: 0, # 'ι' - 11: 0, # 'κ' - 16: 0, # 'λ' - 10: 0, # 'μ' - 6: 0, # 'ν' - 30: 0, # 'ξ' - 4: 2, # 'ο' - 9: 0, # 'π' - 8: 0, # 'ρ' - 14: 0, # 'ς' - 7: 0, # 'σ' - 2: 0, # 'τ' - 12: 0, # 'υ' - 28: 0, # 'φ' - 23: 0, # 'χ' - 42: 0, # 'ψ' - 24: 1, # 'ω' - 19: 2, # 'ό' - 26: 0, # 'ύ' - 27: 0, # 'ώ' - }, - 59: { # 'Ξ' - 60: 0, # 'e' - 55: 0, # 'o' - 58: 0, # 't' - 36: 0, # '·' - 61: 0, # 'Ά' - 46: 0, # 'Έ' - 54: 0, # 'Ό' - 31: 0, # 'Α' - 51: 0, # 'Β' - 43: 0, # 'Γ' - 41: 0, # 'Δ' - 34: 1, # 'Ε' - 40: 1, # 'Η' - 52: 0, # 'Θ' - 47: 0, # 'Ι' - 44: 0, # 'Κ' - 53: 0, # 'Λ' - 38: 0, # 'Μ' - 49: 0, # 'Ν' - 59: 0, # 'Ξ' - 39: 1, # 'Ο' - 35: 0, # 'Π' - 48: 0, # 'Ρ' - 37: 0, # 'Σ' - 33: 0, # 'Τ' - 45: 0, # 'Υ' - 56: 0, # 'Φ' - 50: 0, # 'Χ' - 57: 0, # 'Ω' - 17: 0, # 'ά' - 18: 2, # 'έ' - 22: 0, # 'ή' - 15: 0, # 'ί' - 1: 2, # 'α' - 29: 0, # 'β' - 20: 0, # 'γ' - 21: 0, # 'δ' - 3: 2, # 'ε' - 32: 0, # 'ζ' - 13: 0, # 'η' - 25: 0, # 'θ' - 5: 0, # 'ι' - 11: 0, # 'κ' - 16: 0, # 'λ' - 10: 0, # 'μ' - 6: 0, # 'ν' - 30: 0, # 'ξ' - 4: 0, # 'ο' - 9: 0, # 'π' - 8: 0, # 'ρ' - 14: 0, # 'ς' - 7: 0, # 'σ' - 2: 0, # 'τ' - 12: 0, # 'υ' - 28: 0, # 'φ' - 23: 0, # 'χ' - 42: 0, # 'ψ' - 24: 0, # 'ω' - 19: 0, # 'ό' - 26: 0, # 'ύ' - 27: 0, # 'ώ' - }, - 39: { # 'Ο' - 60: 0, # 'e' - 55: 0, # 'o' - 58: 0, # 't' - 36: 0, # '·' - 61: 0, # 'Ά' - 46: 0, # 'Έ' - 54: 0, # 'Ό' - 31: 0, # 'Α' - 51: 1, # 'Β' - 43: 2, # 'Γ' - 41: 2, # 'Δ' - 34: 2, # 'Ε' - 40: 1, # 'Η' - 52: 2, # 'Θ' - 47: 2, # 'Ι' - 44: 2, # 'Κ' - 53: 2, # 'Λ' - 38: 2, # 'Μ' - 49: 2, # 'Ν' - 59: 0, # 'Ξ' - 39: 0, # 'Ο' - 35: 2, # 'Π' - 48: 2, # 'Ρ' - 37: 2, # 'Σ' - 33: 2, # 'Τ' - 45: 2, # 'Υ' - 56: 2, # 'Φ' - 50: 2, # 'Χ' - 57: 0, # 'Ω' - 17: 0, # 'ά' - 18: 0, # 'έ' - 22: 0, # 'ή' - 15: 0, # 'ί' - 1: 0, # 'α' - 29: 0, # 'β' - 20: 0, # 'γ' - 21: 2, # 'δ' - 3: 0, # 'ε' - 32: 0, # 'ζ' - 13: 0, # 'η' - 25: 0, # 'θ' - 5: 3, # 'ι' - 11: 2, # 'κ' - 16: 2, # 'λ' - 10: 2, # 'μ' - 6: 2, # 'ν' - 30: 0, # 'ξ' - 4: 0, # 'ο' - 9: 2, # 'π' - 8: 2, # 'ρ' - 14: 0, # 'ς' - 7: 0, # 'σ' - 2: 2, # 'τ' - 12: 2, # 'υ' - 28: 1, # 'φ' - 23: 1, # 'χ' - 42: 0, # 'ψ' - 24: 0, # 'ω' - 19: 0, # 'ό' - 26: 2, # 'ύ' - 27: 0, # 'ώ' - }, - 35: { # 'Π' - 60: 0, # 'e' - 55: 0, # 'o' - 58: 0, # 't' - 36: 0, # '·' - 61: 0, # 'Ά' - 46: 0, # 'Έ' - 54: 0, # 'Ό' - 31: 2, # 'Α' - 51: 0, # 'Β' - 43: 0, # 'Γ' - 41: 0, # 'Δ' - 34: 2, # 'Ε' - 40: 0, # 'Η' - 52: 0, # 'Θ' - 47: 2, # 'Ι' - 44: 0, # 'Κ' - 53: 2, # 'Λ' - 38: 1, # 'Μ' - 49: 0, # 'Ν' - 59: 0, # 'Ξ' - 39: 2, # 'Ο' - 35: 0, # 'Π' - 48: 2, # 'Ρ' - 37: 0, # 'Σ' - 33: 1, # 'Τ' - 45: 0, # 'Υ' - 56: 0, # 'Φ' - 50: 1, # 'Χ' - 57: 2, # 'Ω' - 17: 2, # 'ά' - 18: 1, # 'έ' - 22: 1, # 'ή' - 15: 2, # 'ί' - 1: 3, # 'α' - 29: 0, # 'β' - 20: 0, # 'γ' - 21: 0, # 'δ' - 3: 3, # 'ε' - 32: 0, # 'ζ' - 13: 2, # 'η' - 25: 0, # 'θ' - 5: 2, # 'ι' - 11: 0, # 'κ' - 16: 2, # 'λ' - 10: 0, # 'μ' - 6: 2, # 'ν' - 30: 0, # 'ξ' - 4: 3, # 'ο' - 9: 0, # 'π' - 8: 3, # 'ρ' - 14: 0, # 'ς' - 7: 0, # 'σ' - 2: 0, # 'τ' - 12: 2, # 'υ' - 28: 0, # 'φ' - 23: 2, # 'χ' - 42: 0, # 'ψ' - 24: 2, # 'ω' - 19: 2, # 'ό' - 26: 0, # 'ύ' - 27: 3, # 'ώ' - }, - 48: { # 'Ρ' - 60: 0, # 'e' - 55: 0, # 'o' - 58: 0, # 't' - 36: 0, # '·' - 61: 0, # 'Ά' - 46: 0, # 'Έ' - 54: 0, # 'Ό' - 31: 2, # 'Α' - 51: 0, # 'Β' - 43: 1, # 'Γ' - 41: 1, # 'Δ' - 34: 2, # 'Ε' - 40: 2, # 'Η' - 52: 0, # 'Θ' - 47: 2, # 'Ι' - 44: 0, # 'Κ' - 53: 0, # 'Λ' - 38: 0, # 'Μ' - 49: 2, # 'Ν' - 59: 0, # 'Ξ' - 39: 2, # 'Ο' - 35: 0, # 'Π' - 48: 2, # 'Ρ' - 37: 0, # 'Σ' - 33: 1, # 'Τ' - 45: 1, # 'Υ' - 56: 0, # 'Φ' - 50: 1, # 'Χ' - 57: 1, # 'Ω' - 17: 0, # 'ά' - 18: 0, # 'έ' - 22: 0, # 'ή' - 15: 2, # 'ί' - 1: 0, # 'α' - 29: 0, # 'β' - 20: 0, # 'γ' - 21: 0, # 'δ' - 3: 0, # 'ε' - 32: 0, # 'ζ' - 13: 0, # 'η' - 25: 0, # 'θ' - 5: 0, # 'ι' - 11: 0, # 'κ' - 16: 0, # 'λ' - 10: 0, # 'μ' - 6: 0, # 'ν' - 30: 0, # 'ξ' - 4: 1, # 'ο' - 9: 0, # 'π' - 8: 0, # 'ρ' - 14: 0, # 'ς' - 7: 0, # 'σ' - 2: 0, # 'τ' - 12: 3, # 'υ' - 28: 0, # 'φ' - 23: 0, # 'χ' - 42: 0, # 'ψ' - 24: 2, # 'ω' - 19: 0, # 'ό' - 26: 2, # 'ύ' - 27: 0, # 'ώ' - }, - 37: { # 'Σ' - 60: 0, # 'e' - 55: 0, # 'o' - 58: 0, # 't' - 36: 0, # '·' - 61: 0, # 'Ά' - 46: 0, # 'Έ' - 54: 0, # 'Ό' - 31: 2, # 'Α' - 51: 0, # 'Β' - 43: 0, # 'Γ' - 41: 1, # 'Δ' - 34: 2, # 'Ε' - 40: 2, # 'Η' - 52: 0, # 'Θ' - 47: 2, # 'Ι' - 44: 2, # 'Κ' - 53: 0, # 'Λ' - 38: 2, # 'Μ' - 49: 0, # 'Ν' - 59: 0, # 'Ξ' - 39: 2, # 'Ο' - 35: 0, # 'Π' - 48: 0, # 'Ρ' - 37: 2, # 'Σ' - 33: 2, # 'Τ' - 45: 2, # 'Υ' - 56: 0, # 'Φ' - 50: 2, # 'Χ' - 57: 2, # 'Ω' - 17: 0, # 'ά' - 18: 0, # 'έ' - 22: 2, # 'ή' - 15: 2, # 'ί' - 1: 2, # 'α' - 29: 2, # 'β' - 20: 0, # 'γ' - 21: 0, # 'δ' - 3: 3, # 'ε' - 32: 0, # 'ζ' - 13: 3, # 'η' - 25: 0, # 'θ' - 5: 2, # 'ι' - 11: 2, # 'κ' - 16: 0, # 'λ' - 10: 0, # 'μ' - 6: 0, # 'ν' - 30: 0, # 'ξ' - 4: 2, # 'ο' - 9: 2, # 'π' - 8: 0, # 'ρ' - 14: 0, # 'ς' - 7: 0, # 'σ' - 2: 3, # 'τ' - 12: 3, # 'υ' - 28: 0, # 'φ' - 23: 2, # 'χ' - 42: 0, # 'ψ' - 24: 2, # 'ω' - 19: 0, # 'ό' - 26: 2, # 'ύ' - 27: 2, # 'ώ' - }, - 33: { # 'Τ' - 60: 0, # 'e' - 55: 1, # 'o' - 58: 0, # 't' - 36: 0, # '·' - 61: 0, # 'Ά' - 46: 0, # 'Έ' - 54: 0, # 'Ό' - 31: 2, # 'Α' - 51: 0, # 'Β' - 43: 0, # 'Γ' - 41: 0, # 'Δ' - 34: 2, # 'Ε' - 40: 2, # 'Η' - 52: 0, # 'Θ' - 47: 2, # 'Ι' - 44: 2, # 'Κ' - 53: 0, # 'Λ' - 38: 0, # 'Μ' - 49: 0, # 'Ν' - 59: 0, # 'Ξ' - 39: 2, # 'Ο' - 35: 0, # 'Π' - 48: 2, # 'Ρ' - 37: 0, # 'Σ' - 33: 1, # 'Τ' - 45: 1, # 'Υ' - 56: 0, # 'Φ' - 50: 0, # 'Χ' - 57: 2, # 'Ω' - 17: 2, # 'ά' - 18: 2, # 'έ' - 22: 0, # 'ή' - 15: 2, # 'ί' - 1: 3, # 'α' - 29: 0, # 'β' - 20: 0, # 'γ' - 21: 0, # 'δ' - 3: 2, # 'ε' - 32: 0, # 'ζ' - 13: 2, # 'η' - 25: 0, # 'θ' - 5: 2, # 'ι' - 11: 0, # 'κ' - 16: 0, # 'λ' - 10: 2, # 'μ' - 6: 0, # 'ν' - 30: 0, # 'ξ' - 4: 3, # 'ο' - 9: 0, # 'π' - 8: 2, # 'ρ' - 14: 0, # 'ς' - 7: 2, # 'σ' - 2: 0, # 'τ' - 12: 2, # 'υ' - 28: 0, # 'φ' - 23: 0, # 'χ' - 42: 0, # 'ψ' - 24: 0, # 'ω' - 19: 2, # 'ό' - 26: 2, # 'ύ' - 27: 3, # 'ώ' - }, - 45: { # 'Υ' - 60: 0, # 'e' - 55: 0, # 'o' - 58: 0, # 't' - 36: 0, # '·' - 61: 0, # 'Ά' - 46: 0, # 'Έ' - 54: 0, # 'Ό' - 31: 0, # 'Α' - 51: 0, # 'Β' - 43: 2, # 'Γ' - 41: 0, # 'Δ' - 34: 1, # 'Ε' - 40: 2, # 'Η' - 52: 2, # 'Θ' - 47: 0, # 'Ι' - 44: 0, # 'Κ' - 53: 1, # 'Λ' - 38: 2, # 'Μ' - 49: 2, # 'Ν' - 59: 0, # 'Ξ' - 39: 0, # 'Ο' - 35: 2, # 'Π' - 48: 1, # 'Ρ' - 37: 2, # 'Σ' - 33: 2, # 'Τ' - 45: 0, # 'Υ' - 56: 0, # 'Φ' - 50: 1, # 'Χ' - 57: 0, # 'Ω' - 17: 0, # 'ά' - 18: 0, # 'έ' - 22: 0, # 'ή' - 15: 0, # 'ί' - 1: 0, # 'α' - 29: 0, # 'β' - 20: 0, # 'γ' - 21: 0, # 'δ' - 3: 0, # 'ε' - 32: 0, # 'ζ' - 13: 0, # 'η' - 25: 0, # 'θ' - 5: 0, # 'ι' - 11: 0, # 'κ' - 16: 2, # 'λ' - 10: 0, # 'μ' - 6: 0, # 'ν' - 30: 0, # 'ξ' - 4: 0, # 'ο' - 9: 3, # 'π' - 8: 0, # 'ρ' - 14: 0, # 'ς' - 7: 0, # 'σ' - 2: 0, # 'τ' - 12: 0, # 'υ' - 28: 0, # 'φ' - 23: 0, # 'χ' - 42: 0, # 'ψ' - 24: 0, # 'ω' - 19: 0, # 'ό' - 26: 0, # 'ύ' - 27: 0, # 'ώ' - }, - 56: { # 'Φ' - 60: 0, # 'e' - 55: 0, # 'o' - 58: 0, # 't' - 36: 0, # '·' - 61: 0, # 'Ά' - 46: 0, # 'Έ' - 54: 0, # 'Ό' - 31: 1, # 'Α' - 51: 0, # 'Β' - 43: 0, # 'Γ' - 41: 0, # 'Δ' - 34: 0, # 'Ε' - 40: 1, # 'Η' - 52: 0, # 'Θ' - 47: 2, # 'Ι' - 44: 0, # 'Κ' - 53: 0, # 'Λ' - 38: 0, # 'Μ' - 49: 0, # 'Ν' - 59: 0, # 'Ξ' - 39: 2, # 'Ο' - 35: 0, # 'Π' - 48: 0, # 'Ρ' - 37: 0, # 'Σ' - 33: 0, # 'Τ' - 45: 0, # 'Υ' - 56: 0, # 'Φ' - 50: 0, # 'Χ' - 57: 0, # 'Ω' - 17: 0, # 'ά' - 18: 0, # 'έ' - 22: 0, # 'ή' - 15: 0, # 'ί' - 1: 2, # 'α' - 29: 0, # 'β' - 20: 0, # 'γ' - 21: 0, # 'δ' - 3: 2, # 'ε' - 32: 0, # 'ζ' - 13: 0, # 'η' - 25: 0, # 'θ' - 5: 2, # 'ι' - 11: 0, # 'κ' - 16: 0, # 'λ' - 10: 0, # 'μ' - 6: 0, # 'ν' - 30: 0, # 'ξ' - 4: 2, # 'ο' - 9: 0, # 'π' - 8: 0, # 'ρ' - 14: 0, # 'ς' - 7: 0, # 'σ' - 2: 2, # 'τ' - 12: 2, # 'υ' - 28: 0, # 'φ' - 23: 0, # 'χ' - 42: 0, # 'ψ' - 24: 0, # 'ω' - 19: 0, # 'ό' - 26: 1, # 'ύ' - 27: 1, # 'ώ' - }, - 50: { # 'Χ' - 60: 0, # 'e' - 55: 0, # 'o' - 58: 0, # 't' - 36: 0, # '·' - 61: 0, # 'Ά' - 46: 0, # 'Έ' - 54: 0, # 'Ό' - 31: 1, # 'Α' - 51: 0, # 'Β' - 43: 0, # 'Γ' - 41: 0, # 'Δ' - 34: 2, # 'Ε' - 40: 2, # 'Η' - 52: 0, # 'Θ' - 47: 2, # 'Ι' - 44: 0, # 'Κ' - 53: 0, # 'Λ' - 38: 0, # 'Μ' - 49: 1, # 'Ν' - 59: 0, # 'Ξ' - 39: 1, # 'Ο' - 35: 0, # 'Π' - 48: 2, # 'Ρ' - 37: 0, # 'Σ' - 33: 0, # 'Τ' - 45: 0, # 'Υ' - 56: 0, # 'Φ' - 50: 1, # 'Χ' - 57: 1, # 'Ω' - 17: 2, # 'ά' - 18: 0, # 'έ' - 22: 0, # 'ή' - 15: 0, # 'ί' - 1: 2, # 'α' - 29: 0, # 'β' - 20: 0, # 'γ' - 21: 0, # 'δ' - 3: 2, # 'ε' - 32: 0, # 'ζ' - 13: 0, # 'η' - 25: 0, # 'θ' - 5: 0, # 'ι' - 11: 0, # 'κ' - 16: 0, # 'λ' - 10: 0, # 'μ' - 6: 0, # 'ν' - 30: 0, # 'ξ' - 4: 2, # 'ο' - 9: 0, # 'π' - 8: 3, # 'ρ' - 14: 0, # 'ς' - 7: 0, # 'σ' - 2: 2, # 'τ' - 12: 0, # 'υ' - 28: 0, # 'φ' - 23: 0, # 'χ' - 42: 0, # 'ψ' - 24: 2, # 'ω' - 19: 0, # 'ό' - 26: 0, # 'ύ' - 27: 0, # 'ώ' - }, - 57: { # 'Ω' - 60: 0, # 'e' - 55: 0, # 'o' - 58: 0, # 't' - 36: 0, # '·' - 61: 0, # 'Ά' - 46: 0, # 'Έ' - 54: 0, # 'Ό' - 31: 0, # 'Α' - 51: 0, # 'Β' - 43: 1, # 'Γ' - 41: 0, # 'Δ' - 34: 0, # 'Ε' - 40: 0, # 'Η' - 52: 0, # 'Θ' - 47: 0, # 'Ι' - 44: 0, # 'Κ' - 53: 1, # 'Λ' - 38: 0, # 'Μ' - 49: 2, # 'Ν' - 59: 0, # 'Ξ' - 39: 0, # 'Ο' - 35: 0, # 'Π' - 48: 2, # 'Ρ' - 37: 2, # 'Σ' - 33: 2, # 'Τ' - 45: 0, # 'Υ' - 56: 0, # 'Φ' - 50: 0, # 'Χ' - 57: 0, # 'Ω' - 17: 0, # 'ά' - 18: 0, # 'έ' - 22: 0, # 'ή' - 15: 0, # 'ί' - 1: 0, # 'α' - 29: 0, # 'β' - 20: 0, # 'γ' - 21: 0, # 'δ' - 3: 0, # 'ε' - 32: 0, # 'ζ' - 13: 0, # 'η' - 25: 0, # 'θ' - 5: 0, # 'ι' - 11: 0, # 'κ' - 16: 0, # 'λ' - 10: 0, # 'μ' - 6: 0, # 'ν' - 30: 0, # 'ξ' - 4: 0, # 'ο' - 9: 0, # 'π' - 8: 2, # 'ρ' - 14: 2, # 'ς' - 7: 2, # 'σ' - 2: 0, # 'τ' - 12: 0, # 'υ' - 28: 0, # 'φ' - 23: 1, # 'χ' - 42: 0, # 'ψ' - 24: 0, # 'ω' - 19: 0, # 'ό' - 26: 0, # 'ύ' - 27: 0, # 'ώ' - }, - 17: { # 'ά' - 60: 0, # 'e' - 55: 0, # 'o' - 58: 0, # 't' - 36: 2, # '·' - 61: 0, # 'Ά' - 46: 0, # 'Έ' - 54: 0, # 'Ό' - 31: 0, # 'Α' - 51: 0, # 'Β' - 43: 0, # 'Γ' - 41: 0, # 'Δ' - 34: 0, # 'Ε' - 40: 0, # 'Η' - 52: 0, # 'Θ' - 47: 0, # 'Ι' - 44: 0, # 'Κ' - 53: 0, # 'Λ' - 38: 0, # 'Μ' - 49: 0, # 'Ν' - 59: 0, # 'Ξ' - 39: 0, # 'Ο' - 35: 0, # 'Π' - 48: 0, # 'Ρ' - 37: 0, # 'Σ' - 33: 0, # 'Τ' - 45: 0, # 'Υ' - 56: 0, # 'Φ' - 50: 0, # 'Χ' - 57: 0, # 'Ω' - 17: 0, # 'ά' - 18: 0, # 'έ' - 22: 0, # 'ή' - 15: 0, # 'ί' - 1: 0, # 'α' - 29: 3, # 'β' - 20: 3, # 'γ' - 21: 3, # 'δ' - 3: 3, # 'ε' - 32: 3, # 'ζ' - 13: 0, # 'η' - 25: 3, # 'θ' - 5: 2, # 'ι' - 11: 3, # 'κ' - 16: 3, # 'λ' - 10: 3, # 'μ' - 6: 3, # 'ν' - 30: 3, # 'ξ' - 4: 0, # 'ο' - 9: 3, # 'π' - 8: 3, # 'ρ' - 14: 3, # 'ς' - 7: 3, # 'σ' - 2: 3, # 'τ' - 12: 0, # 'υ' - 28: 3, # 'φ' - 23: 3, # 'χ' - 42: 3, # 'ψ' - 24: 2, # 'ω' - 19: 0, # 'ό' - 26: 0, # 'ύ' - 27: 0, # 'ώ' - }, - 18: { # 'έ' - 60: 0, # 'e' - 55: 0, # 'o' - 58: 0, # 't' - 36: 0, # '·' - 61: 0, # 'Ά' - 46: 0, # 'Έ' - 54: 0, # 'Ό' - 31: 0, # 'Α' - 51: 0, # 'Β' - 43: 0, # 'Γ' - 41: 0, # 'Δ' - 34: 0, # 'Ε' - 40: 0, # 'Η' - 52: 0, # 'Θ' - 47: 0, # 'Ι' - 44: 0, # 'Κ' - 53: 0, # 'Λ' - 38: 0, # 'Μ' - 49: 0, # 'Ν' - 59: 0, # 'Ξ' - 39: 0, # 'Ο' - 35: 0, # 'Π' - 48: 0, # 'Ρ' - 37: 0, # 'Σ' - 33: 0, # 'Τ' - 45: 0, # 'Υ' - 56: 0, # 'Φ' - 50: 0, # 'Χ' - 57: 0, # 'Ω' - 17: 0, # 'ά' - 18: 0, # 'έ' - 22: 0, # 'ή' - 15: 0, # 'ί' - 1: 3, # 'α' - 29: 2, # 'β' - 20: 3, # 'γ' - 21: 2, # 'δ' - 3: 3, # 'ε' - 32: 2, # 'ζ' - 13: 0, # 'η' - 25: 3, # 'θ' - 5: 0, # 'ι' - 11: 3, # 'κ' - 16: 3, # 'λ' - 10: 3, # 'μ' - 6: 3, # 'ν' - 30: 3, # 'ξ' - 4: 3, # 'ο' - 9: 3, # 'π' - 8: 3, # 'ρ' - 14: 3, # 'ς' - 7: 3, # 'σ' - 2: 3, # 'τ' - 12: 0, # 'υ' - 28: 3, # 'φ' - 23: 3, # 'χ' - 42: 3, # 'ψ' - 24: 2, # 'ω' - 19: 0, # 'ό' - 26: 0, # 'ύ' - 27: 0, # 'ώ' - }, - 22: { # 'ή' - 60: 0, # 'e' - 55: 0, # 'o' - 58: 0, # 't' - 36: 1, # '·' - 61: 0, # 'Ά' - 46: 0, # 'Έ' - 54: 0, # 'Ό' - 31: 0, # 'Α' - 51: 0, # 'Β' - 43: 0, # 'Γ' - 41: 0, # 'Δ' - 34: 0, # 'Ε' - 40: 0, # 'Η' - 52: 0, # 'Θ' - 47: 0, # 'Ι' - 44: 0, # 'Κ' - 53: 0, # 'Λ' - 38: 0, # 'Μ' - 49: 0, # 'Ν' - 59: 0, # 'Ξ' - 39: 0, # 'Ο' - 35: 0, # 'Π' - 48: 0, # 'Ρ' - 37: 0, # 'Σ' - 33: 0, # 'Τ' - 45: 0, # 'Υ' - 56: 0, # 'Φ' - 50: 0, # 'Χ' - 57: 0, # 'Ω' - 17: 0, # 'ά' - 18: 0, # 'έ' - 22: 0, # 'ή' - 15: 0, # 'ί' - 1: 0, # 'α' - 29: 0, # 'β' - 20: 3, # 'γ' - 21: 3, # 'δ' - 3: 0, # 'ε' - 32: 0, # 'ζ' - 13: 0, # 'η' - 25: 3, # 'θ' - 5: 0, # 'ι' - 11: 3, # 'κ' - 16: 2, # 'λ' - 10: 3, # 'μ' - 6: 3, # 'ν' - 30: 2, # 'ξ' - 4: 0, # 'ο' - 9: 3, # 'π' - 8: 3, # 'ρ' - 14: 3, # 'ς' - 7: 3, # 'σ' - 2: 3, # 'τ' - 12: 0, # 'υ' - 28: 2, # 'φ' - 23: 3, # 'χ' - 42: 2, # 'ψ' - 24: 0, # 'ω' - 19: 0, # 'ό' - 26: 0, # 'ύ' - 27: 0, # 'ώ' - }, - 15: { # 'ί' - 60: 0, # 'e' - 55: 0, # 'o' - 58: 0, # 't' - 36: 0, # '·' - 61: 0, # 'Ά' - 46: 0, # 'Έ' - 54: 0, # 'Ό' - 31: 0, # 'Α' - 51: 0, # 'Β' - 43: 0, # 'Γ' - 41: 0, # 'Δ' - 34: 0, # 'Ε' - 40: 0, # 'Η' - 52: 0, # 'Θ' - 47: 0, # 'Ι' - 44: 0, # 'Κ' - 53: 0, # 'Λ' - 38: 0, # 'Μ' - 49: 0, # 'Ν' - 59: 0, # 'Ξ' - 39: 0, # 'Ο' - 35: 0, # 'Π' - 48: 0, # 'Ρ' - 37: 0, # 'Σ' - 33: 0, # 'Τ' - 45: 0, # 'Υ' - 56: 0, # 'Φ' - 50: 0, # 'Χ' - 57: 0, # 'Ω' - 17: 0, # 'ά' - 18: 0, # 'έ' - 22: 0, # 'ή' - 15: 0, # 'ί' - 1: 3, # 'α' - 29: 2, # 'β' - 20: 3, # 'γ' - 21: 3, # 'δ' - 3: 3, # 'ε' - 32: 3, # 'ζ' - 13: 3, # 'η' - 25: 3, # 'θ' - 5: 0, # 'ι' - 11: 3, # 'κ' - 16: 3, # 'λ' - 10: 3, # 'μ' - 6: 3, # 'ν' - 30: 3, # 'ξ' - 4: 3, # 'ο' - 9: 3, # 'π' - 8: 3, # 'ρ' - 14: 3, # 'ς' - 7: 3, # 'σ' - 2: 3, # 'τ' - 12: 0, # 'υ' - 28: 1, # 'φ' - 23: 3, # 'χ' - 42: 2, # 'ψ' - 24: 3, # 'ω' - 19: 0, # 'ό' - 26: 0, # 'ύ' - 27: 0, # 'ώ' - }, - 1: { # 'α' - 60: 0, # 'e' - 55: 0, # 'o' - 58: 0, # 't' - 36: 2, # '·' - 61: 0, # 'Ά' - 46: 0, # 'Έ' - 54: 0, # 'Ό' - 31: 0, # 'Α' - 51: 0, # 'Β' - 43: 0, # 'Γ' - 41: 0, # 'Δ' - 34: 0, # 'Ε' - 40: 0, # 'Η' - 52: 0, # 'Θ' - 47: 0, # 'Ι' - 44: 0, # 'Κ' - 53: 0, # 'Λ' - 38: 0, # 'Μ' - 49: 0, # 'Ν' - 59: 0, # 'Ξ' - 39: 0, # 'Ο' - 35: 0, # 'Π' - 48: 0, # 'Ρ' - 37: 0, # 'Σ' - 33: 0, # 'Τ' - 45: 0, # 'Υ' - 56: 0, # 'Φ' - 50: 0, # 'Χ' - 57: 0, # 'Ω' - 17: 0, # 'ά' - 18: 2, # 'έ' - 22: 0, # 'ή' - 15: 3, # 'ί' - 1: 0, # 'α' - 29: 3, # 'β' - 20: 3, # 'γ' - 21: 3, # 'δ' - 3: 2, # 'ε' - 32: 3, # 'ζ' - 13: 1, # 'η' - 25: 3, # 'θ' - 5: 3, # 'ι' - 11: 3, # 'κ' - 16: 3, # 'λ' - 10: 3, # 'μ' - 6: 3, # 'ν' - 30: 3, # 'ξ' - 4: 2, # 'ο' - 9: 3, # 'π' - 8: 3, # 'ρ' - 14: 3, # 'ς' - 7: 3, # 'σ' - 2: 3, # 'τ' - 12: 3, # 'υ' - 28: 3, # 'φ' - 23: 3, # 'χ' - 42: 2, # 'ψ' - 24: 0, # 'ω' - 19: 2, # 'ό' - 26: 2, # 'ύ' - 27: 0, # 'ώ' - }, - 29: { # 'β' - 60: 0, # 'e' - 55: 0, # 'o' - 58: 0, # 't' - 36: 0, # '·' - 61: 0, # 'Ά' - 46: 0, # 'Έ' - 54: 0, # 'Ό' - 31: 0, # 'Α' - 51: 0, # 'Β' - 43: 0, # 'Γ' - 41: 0, # 'Δ' - 34: 0, # 'Ε' - 40: 0, # 'Η' - 52: 0, # 'Θ' - 47: 0, # 'Ι' - 44: 0, # 'Κ' - 53: 0, # 'Λ' - 38: 0, # 'Μ' - 49: 0, # 'Ν' - 59: 0, # 'Ξ' - 39: 0, # 'Ο' - 35: 0, # 'Π' - 48: 0, # 'Ρ' - 37: 0, # 'Σ' - 33: 0, # 'Τ' - 45: 0, # 'Υ' - 56: 0, # 'Φ' - 50: 0, # 'Χ' - 57: 0, # 'Ω' - 17: 3, # 'ά' - 18: 2, # 'έ' - 22: 3, # 'ή' - 15: 2, # 'ί' - 1: 3, # 'α' - 29: 0, # 'β' - 20: 2, # 'γ' - 21: 2, # 'δ' - 3: 3, # 'ε' - 32: 0, # 'ζ' - 13: 2, # 'η' - 25: 0, # 'θ' - 5: 3, # 'ι' - 11: 0, # 'κ' - 16: 3, # 'λ' - 10: 0, # 'μ' - 6: 0, # 'ν' - 30: 0, # 'ξ' - 4: 3, # 'ο' - 9: 0, # 'π' - 8: 3, # 'ρ' - 14: 0, # 'ς' - 7: 0, # 'σ' - 2: 0, # 'τ' - 12: 0, # 'υ' - 28: 0, # 'φ' - 23: 0, # 'χ' - 42: 0, # 'ψ' - 24: 2, # 'ω' - 19: 2, # 'ό' - 26: 2, # 'ύ' - 27: 2, # 'ώ' - }, - 20: { # 'γ' - 60: 0, # 'e' - 55: 0, # 'o' - 58: 0, # 't' - 36: 0, # '·' - 61: 0, # 'Ά' - 46: 0, # 'Έ' - 54: 0, # 'Ό' - 31: 0, # 'Α' - 51: 0, # 'Β' - 43: 0, # 'Γ' - 41: 0, # 'Δ' - 34: 0, # 'Ε' - 40: 0, # 'Η' - 52: 0, # 'Θ' - 47: 0, # 'Ι' - 44: 0, # 'Κ' - 53: 0, # 'Λ' - 38: 0, # 'Μ' - 49: 0, # 'Ν' - 59: 0, # 'Ξ' - 39: 0, # 'Ο' - 35: 0, # 'Π' - 48: 0, # 'Ρ' - 37: 0, # 'Σ' - 33: 0, # 'Τ' - 45: 0, # 'Υ' - 56: 0, # 'Φ' - 50: 0, # 'Χ' - 57: 0, # 'Ω' - 17: 3, # 'ά' - 18: 3, # 'έ' - 22: 3, # 'ή' - 15: 3, # 'ί' - 1: 3, # 'α' - 29: 0, # 'β' - 20: 3, # 'γ' - 21: 0, # 'δ' - 3: 3, # 'ε' - 32: 0, # 'ζ' - 13: 3, # 'η' - 25: 0, # 'θ' - 5: 3, # 'ι' - 11: 3, # 'κ' - 16: 3, # 'λ' - 10: 3, # 'μ' - 6: 3, # 'ν' - 30: 3, # 'ξ' - 4: 3, # 'ο' - 9: 0, # 'π' - 8: 3, # 'ρ' - 14: 0, # 'ς' - 7: 0, # 'σ' - 2: 0, # 'τ' - 12: 2, # 'υ' - 28: 0, # 'φ' - 23: 3, # 'χ' - 42: 0, # 'ψ' - 24: 3, # 'ω' - 19: 3, # 'ό' - 26: 2, # 'ύ' - 27: 3, # 'ώ' - }, - 21: { # 'δ' - 60: 0, # 'e' - 55: 0, # 'o' - 58: 0, # 't' - 36: 0, # '·' - 61: 0, # 'Ά' - 46: 0, # 'Έ' - 54: 0, # 'Ό' - 31: 0, # 'Α' - 51: 0, # 'Β' - 43: 0, # 'Γ' - 41: 0, # 'Δ' - 34: 0, # 'Ε' - 40: 0, # 'Η' - 52: 0, # 'Θ' - 47: 0, # 'Ι' - 44: 0, # 'Κ' - 53: 0, # 'Λ' - 38: 0, # 'Μ' - 49: 0, # 'Ν' - 59: 0, # 'Ξ' - 39: 0, # 'Ο' - 35: 0, # 'Π' - 48: 0, # 'Ρ' - 37: 0, # 'Σ' - 33: 0, # 'Τ' - 45: 0, # 'Υ' - 56: 0, # 'Φ' - 50: 0, # 'Χ' - 57: 0, # 'Ω' - 17: 2, # 'ά' - 18: 3, # 'έ' - 22: 3, # 'ή' - 15: 3, # 'ί' - 1: 3, # 'α' - 29: 0, # 'β' - 20: 0, # 'γ' - 21: 0, # 'δ' - 3: 3, # 'ε' - 32: 0, # 'ζ' - 13: 3, # 'η' - 25: 0, # 'θ' - 5: 3, # 'ι' - 11: 0, # 'κ' - 16: 0, # 'λ' - 10: 0, # 'μ' - 6: 0, # 'ν' - 30: 0, # 'ξ' - 4: 3, # 'ο' - 9: 0, # 'π' - 8: 3, # 'ρ' - 14: 0, # 'ς' - 7: 0, # 'σ' - 2: 0, # 'τ' - 12: 3, # 'υ' - 28: 0, # 'φ' - 23: 0, # 'χ' - 42: 0, # 'ψ' - 24: 3, # 'ω' - 19: 3, # 'ό' - 26: 3, # 'ύ' - 27: 3, # 'ώ' - }, - 3: { # 'ε' - 60: 0, # 'e' - 55: 0, # 'o' - 58: 0, # 't' - 36: 2, # '·' - 61: 0, # 'Ά' - 46: 0, # 'Έ' - 54: 0, # 'Ό' - 31: 0, # 'Α' - 51: 0, # 'Β' - 43: 0, # 'Γ' - 41: 0, # 'Δ' - 34: 0, # 'Ε' - 40: 0, # 'Η' - 52: 0, # 'Θ' - 47: 0, # 'Ι' - 44: 0, # 'Κ' - 53: 0, # 'Λ' - 38: 0, # 'Μ' - 49: 0, # 'Ν' - 59: 0, # 'Ξ' - 39: 0, # 'Ο' - 35: 0, # 'Π' - 48: 0, # 'Ρ' - 37: 0, # 'Σ' - 33: 0, # 'Τ' - 45: 0, # 'Υ' - 56: 0, # 'Φ' - 50: 0, # 'Χ' - 57: 0, # 'Ω' - 17: 3, # 'ά' - 18: 0, # 'έ' - 22: 0, # 'ή' - 15: 3, # 'ί' - 1: 2, # 'α' - 29: 3, # 'β' - 20: 3, # 'γ' - 21: 3, # 'δ' - 3: 2, # 'ε' - 32: 2, # 'ζ' - 13: 0, # 'η' - 25: 3, # 'θ' - 5: 3, # 'ι' - 11: 3, # 'κ' - 16: 3, # 'λ' - 10: 3, # 'μ' - 6: 3, # 'ν' - 30: 3, # 'ξ' - 4: 2, # 'ο' - 9: 3, # 'π' - 8: 3, # 'ρ' - 14: 3, # 'ς' - 7: 3, # 'σ' - 2: 3, # 'τ' - 12: 3, # 'υ' - 28: 3, # 'φ' - 23: 3, # 'χ' - 42: 2, # 'ψ' - 24: 3, # 'ω' - 19: 2, # 'ό' - 26: 3, # 'ύ' - 27: 2, # 'ώ' - }, - 32: { # 'ζ' - 60: 0, # 'e' - 55: 0, # 'o' - 58: 0, # 't' - 36: 0, # '·' - 61: 0, # 'Ά' - 46: 0, # 'Έ' - 54: 0, # 'Ό' - 31: 0, # 'Α' - 51: 0, # 'Β' - 43: 0, # 'Γ' - 41: 0, # 'Δ' - 34: 0, # 'Ε' - 40: 0, # 'Η' - 52: 0, # 'Θ' - 47: 0, # 'Ι' - 44: 0, # 'Κ' - 53: 0, # 'Λ' - 38: 0, # 'Μ' - 49: 0, # 'Ν' - 59: 0, # 'Ξ' - 39: 0, # 'Ο' - 35: 0, # 'Π' - 48: 0, # 'Ρ' - 37: 0, # 'Σ' - 33: 0, # 'Τ' - 45: 0, # 'Υ' - 56: 0, # 'Φ' - 50: 0, # 'Χ' - 57: 0, # 'Ω' - 17: 2, # 'ά' - 18: 2, # 'έ' - 22: 2, # 'ή' - 15: 2, # 'ί' - 1: 2, # 'α' - 29: 0, # 'β' - 20: 0, # 'γ' - 21: 0, # 'δ' - 3: 3, # 'ε' - 32: 0, # 'ζ' - 13: 3, # 'η' - 25: 0, # 'θ' - 5: 2, # 'ι' - 11: 0, # 'κ' - 16: 0, # 'λ' - 10: 0, # 'μ' - 6: 0, # 'ν' - 30: 0, # 'ξ' - 4: 3, # 'ο' - 9: 0, # 'π' - 8: 0, # 'ρ' - 14: 0, # 'ς' - 7: 0, # 'σ' - 2: 0, # 'τ' - 12: 1, # 'υ' - 28: 0, # 'φ' - 23: 0, # 'χ' - 42: 0, # 'ψ' - 24: 3, # 'ω' - 19: 2, # 'ό' - 26: 0, # 'ύ' - 27: 2, # 'ώ' - }, - 13: { # 'η' - 60: 0, # 'e' - 55: 0, # 'o' - 58: 0, # 't' - 36: 2, # '·' - 61: 0, # 'Ά' - 46: 0, # 'Έ' - 54: 0, # 'Ό' - 31: 0, # 'Α' - 51: 0, # 'Β' - 43: 0, # 'Γ' - 41: 0, # 'Δ' - 34: 0, # 'Ε' - 40: 0, # 'Η' - 52: 0, # 'Θ' - 47: 0, # 'Ι' - 44: 0, # 'Κ' - 53: 0, # 'Λ' - 38: 0, # 'Μ' - 49: 0, # 'Ν' - 59: 0, # 'Ξ' - 39: 0, # 'Ο' - 35: 0, # 'Π' - 48: 0, # 'Ρ' - 37: 0, # 'Σ' - 33: 0, # 'Τ' - 45: 0, # 'Υ' - 56: 0, # 'Φ' - 50: 0, # 'Χ' - 57: 0, # 'Ω' - 17: 0, # 'ά' - 18: 0, # 'έ' - 22: 0, # 'ή' - 15: 0, # 'ί' - 1: 0, # 'α' - 29: 0, # 'β' - 20: 3, # 'γ' - 21: 2, # 'δ' - 3: 0, # 'ε' - 32: 0, # 'ζ' - 13: 0, # 'η' - 25: 3, # 'θ' - 5: 0, # 'ι' - 11: 3, # 'κ' - 16: 3, # 'λ' - 10: 3, # 'μ' - 6: 3, # 'ν' - 30: 2, # 'ξ' - 4: 0, # 'ο' - 9: 2, # 'π' - 8: 3, # 'ρ' - 14: 3, # 'ς' - 7: 3, # 'σ' - 2: 3, # 'τ' - 12: 0, # 'υ' - 28: 2, # 'φ' - 23: 3, # 'χ' - 42: 2, # 'ψ' - 24: 0, # 'ω' - 19: 0, # 'ό' - 26: 0, # 'ύ' - 27: 0, # 'ώ' - }, - 25: { # 'θ' - 60: 0, # 'e' - 55: 0, # 'o' - 58: 0, # 't' - 36: 0, # '·' - 61: 0, # 'Ά' - 46: 0, # 'Έ' - 54: 0, # 'Ό' - 31: 0, # 'Α' - 51: 0, # 'Β' - 43: 0, # 'Γ' - 41: 0, # 'Δ' - 34: 0, # 'Ε' - 40: 0, # 'Η' - 52: 0, # 'Θ' - 47: 0, # 'Ι' - 44: 0, # 'Κ' - 53: 0, # 'Λ' - 38: 0, # 'Μ' - 49: 0, # 'Ν' - 59: 0, # 'Ξ' - 39: 0, # 'Ο' - 35: 0, # 'Π' - 48: 0, # 'Ρ' - 37: 0, # 'Σ' - 33: 0, # 'Τ' - 45: 0, # 'Υ' - 56: 0, # 'Φ' - 50: 0, # 'Χ' - 57: 0, # 'Ω' - 17: 2, # 'ά' - 18: 3, # 'έ' - 22: 3, # 'ή' - 15: 2, # 'ί' - 1: 3, # 'α' - 29: 0, # 'β' - 20: 0, # 'γ' - 21: 0, # 'δ' - 3: 3, # 'ε' - 32: 0, # 'ζ' - 13: 3, # 'η' - 25: 0, # 'θ' - 5: 3, # 'ι' - 11: 0, # 'κ' - 16: 1, # 'λ' - 10: 3, # 'μ' - 6: 2, # 'ν' - 30: 0, # 'ξ' - 4: 3, # 'ο' - 9: 0, # 'π' - 8: 3, # 'ρ' - 14: 0, # 'ς' - 7: 0, # 'σ' - 2: 0, # 'τ' - 12: 3, # 'υ' - 28: 0, # 'φ' - 23: 0, # 'χ' - 42: 0, # 'ψ' - 24: 3, # 'ω' - 19: 3, # 'ό' - 26: 3, # 'ύ' - 27: 3, # 'ώ' - }, - 5: { # 'ι' - 60: 0, # 'e' - 55: 1, # 'o' - 58: 0, # 't' - 36: 2, # '·' - 61: 0, # 'Ά' - 46: 0, # 'Έ' - 54: 0, # 'Ό' - 31: 0, # 'Α' - 51: 0, # 'Β' - 43: 0, # 'Γ' - 41: 0, # 'Δ' - 34: 1, # 'Ε' - 40: 0, # 'Η' - 52: 0, # 'Θ' - 47: 0, # 'Ι' - 44: 0, # 'Κ' - 53: 0, # 'Λ' - 38: 0, # 'Μ' - 49: 0, # 'Ν' - 59: 0, # 'Ξ' - 39: 0, # 'Ο' - 35: 0, # 'Π' - 48: 0, # 'Ρ' - 37: 0, # 'Σ' - 33: 0, # 'Τ' - 45: 0, # 'Υ' - 56: 0, # 'Φ' - 50: 0, # 'Χ' - 57: 0, # 'Ω' - 17: 3, # 'ά' - 18: 3, # 'έ' - 22: 3, # 'ή' - 15: 0, # 'ί' - 1: 3, # 'α' - 29: 3, # 'β' - 20: 3, # 'γ' - 21: 3, # 'δ' - 3: 3, # 'ε' - 32: 2, # 'ζ' - 13: 3, # 'η' - 25: 3, # 'θ' - 5: 0, # 'ι' - 11: 3, # 'κ' - 16: 3, # 'λ' - 10: 3, # 'μ' - 6: 3, # 'ν' - 30: 3, # 'ξ' - 4: 3, # 'ο' - 9: 3, # 'π' - 8: 3, # 'ρ' - 14: 3, # 'ς' - 7: 3, # 'σ' - 2: 3, # 'τ' - 12: 0, # 'υ' - 28: 2, # 'φ' - 23: 3, # 'χ' - 42: 2, # 'ψ' - 24: 3, # 'ω' - 19: 3, # 'ό' - 26: 0, # 'ύ' - 27: 3, # 'ώ' - }, - 11: { # 'κ' - 60: 0, # 'e' - 55: 0, # 'o' - 58: 0, # 't' - 36: 0, # '·' - 61: 0, # 'Ά' - 46: 0, # 'Έ' - 54: 0, # 'Ό' - 31: 0, # 'Α' - 51: 0, # 'Β' - 43: 0, # 'Γ' - 41: 0, # 'Δ' - 34: 0, # 'Ε' - 40: 0, # 'Η' - 52: 0, # 'Θ' - 47: 0, # 'Ι' - 44: 0, # 'Κ' - 53: 0, # 'Λ' - 38: 0, # 'Μ' - 49: 0, # 'Ν' - 59: 0, # 'Ξ' - 39: 0, # 'Ο' - 35: 0, # 'Π' - 48: 0, # 'Ρ' - 37: 0, # 'Σ' - 33: 0, # 'Τ' - 45: 0, # 'Υ' - 56: 0, # 'Φ' - 50: 0, # 'Χ' - 57: 0, # 'Ω' - 17: 3, # 'ά' - 18: 3, # 'έ' - 22: 3, # 'ή' - 15: 3, # 'ί' - 1: 3, # 'α' - 29: 0, # 'β' - 20: 0, # 'γ' - 21: 3, # 'δ' - 3: 3, # 'ε' - 32: 0, # 'ζ' - 13: 3, # 'η' - 25: 2, # 'θ' - 5: 3, # 'ι' - 11: 3, # 'κ' - 16: 3, # 'λ' - 10: 3, # 'μ' - 6: 2, # 'ν' - 30: 0, # 'ξ' - 4: 3, # 'ο' - 9: 2, # 'π' - 8: 3, # 'ρ' - 14: 0, # 'ς' - 7: 0, # 'σ' - 2: 3, # 'τ' - 12: 3, # 'υ' - 28: 2, # 'φ' - 23: 2, # 'χ' - 42: 0, # 'ψ' - 24: 3, # 'ω' - 19: 3, # 'ό' - 26: 3, # 'ύ' - 27: 3, # 'ώ' - }, - 16: { # 'λ' - 60: 0, # 'e' - 55: 0, # 'o' - 58: 0, # 't' - 36: 0, # '·' - 61: 0, # 'Ά' - 46: 0, # 'Έ' - 54: 0, # 'Ό' - 31: 0, # 'Α' - 51: 0, # 'Β' - 43: 0, # 'Γ' - 41: 0, # 'Δ' - 34: 0, # 'Ε' - 40: 0, # 'Η' - 52: 0, # 'Θ' - 47: 0, # 'Ι' - 44: 0, # 'Κ' - 53: 0, # 'Λ' - 38: 0, # 'Μ' - 49: 0, # 'Ν' - 59: 0, # 'Ξ' - 39: 0, # 'Ο' - 35: 0, # 'Π' - 48: 0, # 'Ρ' - 37: 0, # 'Σ' - 33: 0, # 'Τ' - 45: 0, # 'Υ' - 56: 0, # 'Φ' - 50: 0, # 'Χ' - 57: 0, # 'Ω' - 17: 3, # 'ά' - 18: 3, # 'έ' - 22: 3, # 'ή' - 15: 3, # 'ί' - 1: 3, # 'α' - 29: 1, # 'β' - 20: 2, # 'γ' - 21: 1, # 'δ' - 3: 3, # 'ε' - 32: 0, # 'ζ' - 13: 3, # 'η' - 25: 2, # 'θ' - 5: 3, # 'ι' - 11: 2, # 'κ' - 16: 3, # 'λ' - 10: 2, # 'μ' - 6: 2, # 'ν' - 30: 0, # 'ξ' - 4: 3, # 'ο' - 9: 3, # 'π' - 8: 0, # 'ρ' - 14: 0, # 'ς' - 7: 0, # 'σ' - 2: 3, # 'τ' - 12: 3, # 'υ' - 28: 2, # 'φ' - 23: 0, # 'χ' - 42: 0, # 'ψ' - 24: 3, # 'ω' - 19: 3, # 'ό' - 26: 3, # 'ύ' - 27: 3, # 'ώ' - }, - 10: { # 'μ' - 60: 0, # 'e' - 55: 0, # 'o' - 58: 0, # 't' - 36: 0, # '·' - 61: 0, # 'Ά' - 46: 0, # 'Έ' - 54: 0, # 'Ό' - 31: 0, # 'Α' - 51: 0, # 'Β' - 43: 0, # 'Γ' - 41: 0, # 'Δ' - 34: 1, # 'Ε' - 40: 0, # 'Η' - 52: 0, # 'Θ' - 47: 0, # 'Ι' - 44: 0, # 'Κ' - 53: 0, # 'Λ' - 38: 0, # 'Μ' - 49: 0, # 'Ν' - 59: 0, # 'Ξ' - 39: 0, # 'Ο' - 35: 0, # 'Π' - 48: 0, # 'Ρ' - 37: 0, # 'Σ' - 33: 0, # 'Τ' - 45: 0, # 'Υ' - 56: 0, # 'Φ' - 50: 0, # 'Χ' - 57: 0, # 'Ω' - 17: 3, # 'ά' - 18: 3, # 'έ' - 22: 3, # 'ή' - 15: 3, # 'ί' - 1: 3, # 'α' - 29: 3, # 'β' - 20: 0, # 'γ' - 21: 0, # 'δ' - 3: 3, # 'ε' - 32: 0, # 'ζ' - 13: 3, # 'η' - 25: 0, # 'θ' - 5: 3, # 'ι' - 11: 0, # 'κ' - 16: 0, # 'λ' - 10: 3, # 'μ' - 6: 3, # 'ν' - 30: 0, # 'ξ' - 4: 3, # 'ο' - 9: 3, # 'π' - 8: 0, # 'ρ' - 14: 0, # 'ς' - 7: 0, # 'σ' - 2: 0, # 'τ' - 12: 2, # 'υ' - 28: 3, # 'φ' - 23: 0, # 'χ' - 42: 2, # 'ψ' - 24: 3, # 'ω' - 19: 3, # 'ό' - 26: 2, # 'ύ' - 27: 2, # 'ώ' - }, - 6: { # 'ν' - 60: 0, # 'e' - 55: 0, # 'o' - 58: 0, # 't' - 36: 2, # '·' - 61: 0, # 'Ά' - 46: 0, # 'Έ' - 54: 0, # 'Ό' - 31: 0, # 'Α' - 51: 0, # 'Β' - 43: 0, # 'Γ' - 41: 0, # 'Δ' - 34: 0, # 'Ε' - 40: 0, # 'Η' - 52: 0, # 'Θ' - 47: 0, # 'Ι' - 44: 0, # 'Κ' - 53: 0, # 'Λ' - 38: 0, # 'Μ' - 49: 0, # 'Ν' - 59: 0, # 'Ξ' - 39: 0, # 'Ο' - 35: 0, # 'Π' - 48: 0, # 'Ρ' - 37: 0, # 'Σ' - 33: 0, # 'Τ' - 45: 0, # 'Υ' - 56: 0, # 'Φ' - 50: 0, # 'Χ' - 57: 0, # 'Ω' - 17: 3, # 'ά' - 18: 3, # 'έ' - 22: 3, # 'ή' - 15: 3, # 'ί' - 1: 3, # 'α' - 29: 0, # 'β' - 20: 0, # 'γ' - 21: 3, # 'δ' - 3: 3, # 'ε' - 32: 2, # 'ζ' - 13: 3, # 'η' - 25: 3, # 'θ' - 5: 3, # 'ι' - 11: 0, # 'κ' - 16: 1, # 'λ' - 10: 0, # 'μ' - 6: 2, # 'ν' - 30: 0, # 'ξ' - 4: 3, # 'ο' - 9: 0, # 'π' - 8: 0, # 'ρ' - 14: 0, # 'ς' - 7: 3, # 'σ' - 2: 3, # 'τ' - 12: 3, # 'υ' - 28: 0, # 'φ' - 23: 0, # 'χ' - 42: 0, # 'ψ' - 24: 3, # 'ω' - 19: 3, # 'ό' - 26: 3, # 'ύ' - 27: 3, # 'ώ' - }, - 30: { # 'ξ' - 60: 0, # 'e' - 55: 0, # 'o' - 58: 0, # 't' - 36: 0, # '·' - 61: 0, # 'Ά' - 46: 0, # 'Έ' - 54: 0, # 'Ό' - 31: 0, # 'Α' - 51: 0, # 'Β' - 43: 0, # 'Γ' - 41: 0, # 'Δ' - 34: 0, # 'Ε' - 40: 0, # 'Η' - 52: 0, # 'Θ' - 47: 0, # 'Ι' - 44: 0, # 'Κ' - 53: 0, # 'Λ' - 38: 0, # 'Μ' - 49: 0, # 'Ν' - 59: 0, # 'Ξ' - 39: 0, # 'Ο' - 35: 0, # 'Π' - 48: 0, # 'Ρ' - 37: 0, # 'Σ' - 33: 0, # 'Τ' - 45: 0, # 'Υ' - 56: 0, # 'Φ' - 50: 0, # 'Χ' - 57: 0, # 'Ω' - 17: 2, # 'ά' - 18: 3, # 'έ' - 22: 3, # 'ή' - 15: 2, # 'ί' - 1: 3, # 'α' - 29: 0, # 'β' - 20: 0, # 'γ' - 21: 0, # 'δ' - 3: 3, # 'ε' - 32: 0, # 'ζ' - 13: 3, # 'η' - 25: 0, # 'θ' - 5: 2, # 'ι' - 11: 0, # 'κ' - 16: 0, # 'λ' - 10: 0, # 'μ' - 6: 0, # 'ν' - 30: 0, # 'ξ' - 4: 3, # 'ο' - 9: 0, # 'π' - 8: 0, # 'ρ' - 14: 0, # 'ς' - 7: 0, # 'σ' - 2: 3, # 'τ' - 12: 2, # 'υ' - 28: 0, # 'φ' - 23: 0, # 'χ' - 42: 0, # 'ψ' - 24: 3, # 'ω' - 19: 2, # 'ό' - 26: 3, # 'ύ' - 27: 1, # 'ώ' - }, - 4: { # 'ο' - 60: 0, # 'e' - 55: 0, # 'o' - 58: 0, # 't' - 36: 2, # '·' - 61: 0, # 'Ά' - 46: 0, # 'Έ' - 54: 0, # 'Ό' - 31: 0, # 'Α' - 51: 0, # 'Β' - 43: 0, # 'Γ' - 41: 0, # 'Δ' - 34: 0, # 'Ε' - 40: 0, # 'Η' - 52: 0, # 'Θ' - 47: 0, # 'Ι' - 44: 0, # 'Κ' - 53: 0, # 'Λ' - 38: 0, # 'Μ' - 49: 0, # 'Ν' - 59: 0, # 'Ξ' - 39: 0, # 'Ο' - 35: 0, # 'Π' - 48: 0, # 'Ρ' - 37: 0, # 'Σ' - 33: 0, # 'Τ' - 45: 0, # 'Υ' - 56: 0, # 'Φ' - 50: 0, # 'Χ' - 57: 0, # 'Ω' - 17: 0, # 'ά' - 18: 2, # 'έ' - 22: 3, # 'ή' - 15: 3, # 'ί' - 1: 2, # 'α' - 29: 3, # 'β' - 20: 3, # 'γ' - 21: 3, # 'δ' - 3: 3, # 'ε' - 32: 0, # 'ζ' - 13: 3, # 'η' - 25: 3, # 'θ' - 5: 3, # 'ι' - 11: 3, # 'κ' - 16: 3, # 'λ' - 10: 3, # 'μ' - 6: 3, # 'ν' - 30: 2, # 'ξ' - 4: 2, # 'ο' - 9: 3, # 'π' - 8: 3, # 'ρ' - 14: 3, # 'ς' - 7: 3, # 'σ' - 2: 3, # 'τ' - 12: 3, # 'υ' - 28: 3, # 'φ' - 23: 3, # 'χ' - 42: 2, # 'ψ' - 24: 2, # 'ω' - 19: 1, # 'ό' - 26: 3, # 'ύ' - 27: 2, # 'ώ' - }, - 9: { # 'π' - 60: 0, # 'e' - 55: 0, # 'o' - 58: 0, # 't' - 36: 0, # '·' - 61: 0, # 'Ά' - 46: 0, # 'Έ' - 54: 0, # 'Ό' - 31: 0, # 'Α' - 51: 0, # 'Β' - 43: 0, # 'Γ' - 41: 0, # 'Δ' - 34: 0, # 'Ε' - 40: 0, # 'Η' - 52: 0, # 'Θ' - 47: 0, # 'Ι' - 44: 0, # 'Κ' - 53: 0, # 'Λ' - 38: 0, # 'Μ' - 49: 0, # 'Ν' - 59: 0, # 'Ξ' - 39: 0, # 'Ο' - 35: 0, # 'Π' - 48: 0, # 'Ρ' - 37: 0, # 'Σ' - 33: 0, # 'Τ' - 45: 0, # 'Υ' - 56: 0, # 'Φ' - 50: 0, # 'Χ' - 57: 0, # 'Ω' - 17: 3, # 'ά' - 18: 3, # 'έ' - 22: 3, # 'ή' - 15: 3, # 'ί' - 1: 3, # 'α' - 29: 0, # 'β' - 20: 0, # 'γ' - 21: 0, # 'δ' - 3: 3, # 'ε' - 32: 0, # 'ζ' - 13: 3, # 'η' - 25: 0, # 'θ' - 5: 3, # 'ι' - 11: 0, # 'κ' - 16: 3, # 'λ' - 10: 0, # 'μ' - 6: 2, # 'ν' - 30: 0, # 'ξ' - 4: 3, # 'ο' - 9: 0, # 'π' - 8: 3, # 'ρ' - 14: 2, # 'ς' - 7: 0, # 'σ' - 2: 3, # 'τ' - 12: 3, # 'υ' - 28: 0, # 'φ' - 23: 2, # 'χ' - 42: 0, # 'ψ' - 24: 3, # 'ω' - 19: 3, # 'ό' - 26: 2, # 'ύ' - 27: 3, # 'ώ' - }, - 8: { # 'ρ' - 60: 0, # 'e' - 55: 0, # 'o' - 58: 0, # 't' - 36: 0, # '·' - 61: 0, # 'Ά' - 46: 0, # 'Έ' - 54: 0, # 'Ό' - 31: 0, # 'Α' - 51: 0, # 'Β' - 43: 0, # 'Γ' - 41: 0, # 'Δ' - 34: 0, # 'Ε' - 40: 0, # 'Η' - 52: 0, # 'Θ' - 47: 0, # 'Ι' - 44: 0, # 'Κ' - 53: 0, # 'Λ' - 38: 0, # 'Μ' - 49: 0, # 'Ν' - 59: 0, # 'Ξ' - 39: 0, # 'Ο' - 35: 0, # 'Π' - 48: 0, # 'Ρ' - 37: 0, # 'Σ' - 33: 0, # 'Τ' - 45: 0, # 'Υ' - 56: 0, # 'Φ' - 50: 0, # 'Χ' - 57: 0, # 'Ω' - 17: 3, # 'ά' - 18: 3, # 'έ' - 22: 3, # 'ή' - 15: 3, # 'ί' - 1: 3, # 'α' - 29: 2, # 'β' - 20: 3, # 'γ' - 21: 2, # 'δ' - 3: 3, # 'ε' - 32: 0, # 'ζ' - 13: 3, # 'η' - 25: 3, # 'θ' - 5: 3, # 'ι' - 11: 3, # 'κ' - 16: 1, # 'λ' - 10: 3, # 'μ' - 6: 3, # 'ν' - 30: 2, # 'ξ' - 4: 3, # 'ο' - 9: 2, # 'π' - 8: 2, # 'ρ' - 14: 0, # 'ς' - 7: 2, # 'σ' - 2: 3, # 'τ' - 12: 3, # 'υ' - 28: 3, # 'φ' - 23: 3, # 'χ' - 42: 0, # 'ψ' - 24: 3, # 'ω' - 19: 3, # 'ό' - 26: 3, # 'ύ' - 27: 3, # 'ώ' - }, - 14: { # 'ς' - 60: 0, # 'e' - 55: 0, # 'o' - 58: 0, # 't' - 36: 2, # '·' - 61: 0, # 'Ά' - 46: 0, # 'Έ' - 54: 0, # 'Ό' - 31: 0, # 'Α' - 51: 0, # 'Β' - 43: 0, # 'Γ' - 41: 0, # 'Δ' - 34: 0, # 'Ε' - 40: 0, # 'Η' - 52: 0, # 'Θ' - 47: 0, # 'Ι' - 44: 0, # 'Κ' - 53: 0, # 'Λ' - 38: 0, # 'Μ' - 49: 0, # 'Ν' - 59: 0, # 'Ξ' - 39: 0, # 'Ο' - 35: 0, # 'Π' - 48: 0, # 'Ρ' - 37: 0, # 'Σ' - 33: 0, # 'Τ' - 45: 0, # 'Υ' - 56: 0, # 'Φ' - 50: 0, # 'Χ' - 57: 0, # 'Ω' - 17: 0, # 'ά' - 18: 0, # 'έ' - 22: 0, # 'ή' - 15: 0, # 'ί' - 1: 0, # 'α' - 29: 0, # 'β' - 20: 0, # 'γ' - 21: 0, # 'δ' - 3: 0, # 'ε' - 32: 0, # 'ζ' - 13: 0, # 'η' - 25: 0, # 'θ' - 5: 0, # 'ι' - 11: 0, # 'κ' - 16: 0, # 'λ' - 10: 0, # 'μ' - 6: 0, # 'ν' - 30: 0, # 'ξ' - 4: 0, # 'ο' - 9: 0, # 'π' - 8: 0, # 'ρ' - 14: 0, # 'ς' - 7: 0, # 'σ' - 2: 0, # 'τ' - 12: 0, # 'υ' - 28: 0, # 'φ' - 23: 0, # 'χ' - 42: 0, # 'ψ' - 24: 0, # 'ω' - 19: 0, # 'ό' - 26: 0, # 'ύ' - 27: 0, # 'ώ' - }, - 7: { # 'σ' - 60: 0, # 'e' - 55: 0, # 'o' - 58: 0, # 't' - 36: 0, # '·' - 61: 0, # 'Ά' - 46: 0, # 'Έ' - 54: 0, # 'Ό' - 31: 0, # 'Α' - 51: 0, # 'Β' - 43: 0, # 'Γ' - 41: 0, # 'Δ' - 34: 0, # 'Ε' - 40: 0, # 'Η' - 52: 0, # 'Θ' - 47: 0, # 'Ι' - 44: 0, # 'Κ' - 53: 0, # 'Λ' - 38: 0, # 'Μ' - 49: 0, # 'Ν' - 59: 0, # 'Ξ' - 39: 0, # 'Ο' - 35: 0, # 'Π' - 48: 0, # 'Ρ' - 37: 0, # 'Σ' - 33: 0, # 'Τ' - 45: 0, # 'Υ' - 56: 0, # 'Φ' - 50: 0, # 'Χ' - 57: 0, # 'Ω' - 17: 2, # 'ά' - 18: 2, # 'έ' - 22: 3, # 'ή' - 15: 3, # 'ί' - 1: 3, # 'α' - 29: 3, # 'β' - 20: 0, # 'γ' - 21: 2, # 'δ' - 3: 3, # 'ε' - 32: 0, # 'ζ' - 13: 3, # 'η' - 25: 3, # 'θ' - 5: 3, # 'ι' - 11: 3, # 'κ' - 16: 2, # 'λ' - 10: 3, # 'μ' - 6: 0, # 'ν' - 30: 0, # 'ξ' - 4: 3, # 'ο' - 9: 3, # 'π' - 8: 0, # 'ρ' - 14: 0, # 'ς' - 7: 3, # 'σ' - 2: 3, # 'τ' - 12: 3, # 'υ' - 28: 3, # 'φ' - 23: 3, # 'χ' - 42: 0, # 'ψ' - 24: 3, # 'ω' - 19: 3, # 'ό' - 26: 3, # 'ύ' - 27: 2, # 'ώ' - }, - 2: { # 'τ' - 60: 0, # 'e' - 55: 2, # 'o' - 58: 0, # 't' - 36: 0, # '·' - 61: 0, # 'Ά' - 46: 0, # 'Έ' - 54: 0, # 'Ό' - 31: 0, # 'Α' - 51: 0, # 'Β' - 43: 0, # 'Γ' - 41: 0, # 'Δ' - 34: 0, # 'Ε' - 40: 0, # 'Η' - 52: 0, # 'Θ' - 47: 0, # 'Ι' - 44: 0, # 'Κ' - 53: 0, # 'Λ' - 38: 0, # 'Μ' - 49: 0, # 'Ν' - 59: 0, # 'Ξ' - 39: 0, # 'Ο' - 35: 0, # 'Π' - 48: 0, # 'Ρ' - 37: 0, # 'Σ' - 33: 0, # 'Τ' - 45: 0, # 'Υ' - 56: 0, # 'Φ' - 50: 0, # 'Χ' - 57: 0, # 'Ω' - 17: 3, # 'ά' - 18: 3, # 'έ' - 22: 3, # 'ή' - 15: 3, # 'ί' - 1: 3, # 'α' - 29: 0, # 'β' - 20: 0, # 'γ' - 21: 0, # 'δ' - 3: 3, # 'ε' - 32: 2, # 'ζ' - 13: 3, # 'η' - 25: 0, # 'θ' - 5: 3, # 'ι' - 11: 2, # 'κ' - 16: 2, # 'λ' - 10: 3, # 'μ' - 6: 0, # 'ν' - 30: 0, # 'ξ' - 4: 3, # 'ο' - 9: 0, # 'π' - 8: 3, # 'ρ' - 14: 0, # 'ς' - 7: 3, # 'σ' - 2: 3, # 'τ' - 12: 3, # 'υ' - 28: 2, # 'φ' - 23: 0, # 'χ' - 42: 0, # 'ψ' - 24: 3, # 'ω' - 19: 3, # 'ό' - 26: 3, # 'ύ' - 27: 3, # 'ώ' - }, - 12: { # 'υ' - 60: 0, # 'e' - 55: 0, # 'o' - 58: 0, # 't' - 36: 0, # '·' - 61: 0, # 'Ά' - 46: 0, # 'Έ' - 54: 0, # 'Ό' - 31: 0, # 'Α' - 51: 0, # 'Β' - 43: 0, # 'Γ' - 41: 0, # 'Δ' - 34: 0, # 'Ε' - 40: 0, # 'Η' - 52: 0, # 'Θ' - 47: 0, # 'Ι' - 44: 0, # 'Κ' - 53: 0, # 'Λ' - 38: 0, # 'Μ' - 49: 0, # 'Ν' - 59: 0, # 'Ξ' - 39: 0, # 'Ο' - 35: 0, # 'Π' - 48: 0, # 'Ρ' - 37: 0, # 'Σ' - 33: 0, # 'Τ' - 45: 0, # 'Υ' - 56: 0, # 'Φ' - 50: 0, # 'Χ' - 57: 0, # 'Ω' - 17: 2, # 'ά' - 18: 2, # 'έ' - 22: 3, # 'ή' - 15: 2, # 'ί' - 1: 3, # 'α' - 29: 2, # 'β' - 20: 3, # 'γ' - 21: 2, # 'δ' - 3: 2, # 'ε' - 32: 2, # 'ζ' - 13: 2, # 'η' - 25: 3, # 'θ' - 5: 2, # 'ι' - 11: 3, # 'κ' - 16: 3, # 'λ' - 10: 3, # 'μ' - 6: 3, # 'ν' - 30: 3, # 'ξ' - 4: 3, # 'ο' - 9: 3, # 'π' - 8: 3, # 'ρ' - 14: 3, # 'ς' - 7: 3, # 'σ' - 2: 3, # 'τ' - 12: 0, # 'υ' - 28: 2, # 'φ' - 23: 3, # 'χ' - 42: 2, # 'ψ' - 24: 2, # 'ω' - 19: 2, # 'ό' - 26: 0, # 'ύ' - 27: 2, # 'ώ' - }, - 28: { # 'φ' - 60: 0, # 'e' - 55: 0, # 'o' - 58: 0, # 't' - 36: 0, # '·' - 61: 0, # 'Ά' - 46: 0, # 'Έ' - 54: 0, # 'Ό' - 31: 0, # 'Α' - 51: 0, # 'Β' - 43: 0, # 'Γ' - 41: 0, # 'Δ' - 34: 0, # 'Ε' - 40: 0, # 'Η' - 52: 0, # 'Θ' - 47: 0, # 'Ι' - 44: 0, # 'Κ' - 53: 0, # 'Λ' - 38: 0, # 'Μ' - 49: 0, # 'Ν' - 59: 0, # 'Ξ' - 39: 0, # 'Ο' - 35: 0, # 'Π' - 48: 0, # 'Ρ' - 37: 0, # 'Σ' - 33: 0, # 'Τ' - 45: 0, # 'Υ' - 56: 0, # 'Φ' - 50: 0, # 'Χ' - 57: 0, # 'Ω' - 17: 3, # 'ά' - 18: 3, # 'έ' - 22: 3, # 'ή' - 15: 3, # 'ί' - 1: 3, # 'α' - 29: 0, # 'β' - 20: 0, # 'γ' - 21: 0, # 'δ' - 3: 3, # 'ε' - 32: 0, # 'ζ' - 13: 2, # 'η' - 25: 2, # 'θ' - 5: 3, # 'ι' - 11: 0, # 'κ' - 16: 2, # 'λ' - 10: 0, # 'μ' - 6: 1, # 'ν' - 30: 0, # 'ξ' - 4: 3, # 'ο' - 9: 0, # 'π' - 8: 3, # 'ρ' - 14: 0, # 'ς' - 7: 0, # 'σ' - 2: 3, # 'τ' - 12: 3, # 'υ' - 28: 1, # 'φ' - 23: 0, # 'χ' - 42: 0, # 'ψ' - 24: 3, # 'ω' - 19: 3, # 'ό' - 26: 2, # 'ύ' - 27: 2, # 'ώ' - }, - 23: { # 'χ' - 60: 0, # 'e' - 55: 0, # 'o' - 58: 0, # 't' - 36: 0, # '·' - 61: 0, # 'Ά' - 46: 0, # 'Έ' - 54: 0, # 'Ό' - 31: 0, # 'Α' - 51: 0, # 'Β' - 43: 0, # 'Γ' - 41: 0, # 'Δ' - 34: 0, # 'Ε' - 40: 0, # 'Η' - 52: 0, # 'Θ' - 47: 0, # 'Ι' - 44: 0, # 'Κ' - 53: 0, # 'Λ' - 38: 0, # 'Μ' - 49: 0, # 'Ν' - 59: 0, # 'Ξ' - 39: 0, # 'Ο' - 35: 0, # 'Π' - 48: 0, # 'Ρ' - 37: 0, # 'Σ' - 33: 0, # 'Τ' - 45: 0, # 'Υ' - 56: 0, # 'Φ' - 50: 0, # 'Χ' - 57: 0, # 'Ω' - 17: 3, # 'ά' - 18: 2, # 'έ' - 22: 3, # 'ή' - 15: 3, # 'ί' - 1: 3, # 'α' - 29: 0, # 'β' - 20: 0, # 'γ' - 21: 0, # 'δ' - 3: 3, # 'ε' - 32: 0, # 'ζ' - 13: 2, # 'η' - 25: 2, # 'θ' - 5: 3, # 'ι' - 11: 0, # 'κ' - 16: 2, # 'λ' - 10: 2, # 'μ' - 6: 3, # 'ν' - 30: 0, # 'ξ' - 4: 3, # 'ο' - 9: 0, # 'π' - 8: 3, # 'ρ' - 14: 0, # 'ς' - 7: 0, # 'σ' - 2: 3, # 'τ' - 12: 3, # 'υ' - 28: 0, # 'φ' - 23: 2, # 'χ' - 42: 0, # 'ψ' - 24: 3, # 'ω' - 19: 3, # 'ό' - 26: 3, # 'ύ' - 27: 3, # 'ώ' - }, - 42: { # 'ψ' - 60: 0, # 'e' - 55: 0, # 'o' - 58: 0, # 't' - 36: 0, # '·' - 61: 0, # 'Ά' - 46: 0, # 'Έ' - 54: 0, # 'Ό' - 31: 0, # 'Α' - 51: 0, # 'Β' - 43: 0, # 'Γ' - 41: 0, # 'Δ' - 34: 0, # 'Ε' - 40: 0, # 'Η' - 52: 0, # 'Θ' - 47: 0, # 'Ι' - 44: 0, # 'Κ' - 53: 0, # 'Λ' - 38: 0, # 'Μ' - 49: 0, # 'Ν' - 59: 0, # 'Ξ' - 39: 0, # 'Ο' - 35: 0, # 'Π' - 48: 0, # 'Ρ' - 37: 0, # 'Σ' - 33: 0, # 'Τ' - 45: 0, # 'Υ' - 56: 0, # 'Φ' - 50: 0, # 'Χ' - 57: 0, # 'Ω' - 17: 2, # 'ά' - 18: 2, # 'έ' - 22: 1, # 'ή' - 15: 2, # 'ί' - 1: 2, # 'α' - 29: 0, # 'β' - 20: 0, # 'γ' - 21: 0, # 'δ' - 3: 3, # 'ε' - 32: 0, # 'ζ' - 13: 3, # 'η' - 25: 0, # 'θ' - 5: 2, # 'ι' - 11: 0, # 'κ' - 16: 0, # 'λ' - 10: 0, # 'μ' - 6: 0, # 'ν' - 30: 0, # 'ξ' - 4: 2, # 'ο' - 9: 0, # 'π' - 8: 0, # 'ρ' - 14: 0, # 'ς' - 7: 0, # 'σ' - 2: 2, # 'τ' - 12: 1, # 'υ' - 28: 0, # 'φ' - 23: 0, # 'χ' - 42: 0, # 'ψ' - 24: 2, # 'ω' - 19: 0, # 'ό' - 26: 0, # 'ύ' - 27: 0, # 'ώ' - }, - 24: { # 'ω' - 60: 0, # 'e' - 55: 0, # 'o' - 58: 0, # 't' - 36: 0, # '·' - 61: 0, # 'Ά' - 46: 0, # 'Έ' - 54: 0, # 'Ό' - 31: 0, # 'Α' - 51: 0, # 'Β' - 43: 0, # 'Γ' - 41: 0, # 'Δ' - 34: 0, # 'Ε' - 40: 0, # 'Η' - 52: 0, # 'Θ' - 47: 0, # 'Ι' - 44: 0, # 'Κ' - 53: 0, # 'Λ' - 38: 0, # 'Μ' - 49: 0, # 'Ν' - 59: 0, # 'Ξ' - 39: 0, # 'Ο' - 35: 0, # 'Π' - 48: 0, # 'Ρ' - 37: 0, # 'Σ' - 33: 0, # 'Τ' - 45: 0, # 'Υ' - 56: 0, # 'Φ' - 50: 0, # 'Χ' - 57: 0, # 'Ω' - 17: 1, # 'ά' - 18: 0, # 'έ' - 22: 2, # 'ή' - 15: 0, # 'ί' - 1: 0, # 'α' - 29: 2, # 'β' - 20: 3, # 'γ' - 21: 2, # 'δ' - 3: 0, # 'ε' - 32: 0, # 'ζ' - 13: 0, # 'η' - 25: 3, # 'θ' - 5: 2, # 'ι' - 11: 0, # 'κ' - 16: 2, # 'λ' - 10: 3, # 'μ' - 6: 3, # 'ν' - 30: 0, # 'ξ' - 4: 0, # 'ο' - 9: 3, # 'π' - 8: 3, # 'ρ' - 14: 3, # 'ς' - 7: 3, # 'σ' - 2: 3, # 'τ' - 12: 0, # 'υ' - 28: 2, # 'φ' - 23: 2, # 'χ' - 42: 0, # 'ψ' - 24: 0, # 'ω' - 19: 0, # 'ό' - 26: 0, # 'ύ' - 27: 0, # 'ώ' - }, - 19: { # 'ό' - 60: 0, # 'e' - 55: 0, # 'o' - 58: 0, # 't' - 36: 0, # '·' - 61: 0, # 'Ά' - 46: 0, # 'Έ' - 54: 0, # 'Ό' - 31: 0, # 'Α' - 51: 0, # 'Β' - 43: 0, # 'Γ' - 41: 0, # 'Δ' - 34: 0, # 'Ε' - 40: 0, # 'Η' - 52: 0, # 'Θ' - 47: 0, # 'Ι' - 44: 0, # 'Κ' - 53: 0, # 'Λ' - 38: 0, # 'Μ' - 49: 0, # 'Ν' - 59: 0, # 'Ξ' - 39: 0, # 'Ο' - 35: 0, # 'Π' - 48: 0, # 'Ρ' - 37: 0, # 'Σ' - 33: 0, # 'Τ' - 45: 0, # 'Υ' - 56: 0, # 'Φ' - 50: 0, # 'Χ' - 57: 0, # 'Ω' - 17: 0, # 'ά' - 18: 0, # 'έ' - 22: 0, # 'ή' - 15: 0, # 'ί' - 1: 0, # 'α' - 29: 3, # 'β' - 20: 3, # 'γ' - 21: 3, # 'δ' - 3: 1, # 'ε' - 32: 2, # 'ζ' - 13: 2, # 'η' - 25: 2, # 'θ' - 5: 2, # 'ι' - 11: 3, # 'κ' - 16: 3, # 'λ' - 10: 3, # 'μ' - 6: 3, # 'ν' - 30: 1, # 'ξ' - 4: 2, # 'ο' - 9: 3, # 'π' - 8: 3, # 'ρ' - 14: 3, # 'ς' - 7: 3, # 'σ' - 2: 3, # 'τ' - 12: 0, # 'υ' - 28: 2, # 'φ' - 23: 3, # 'χ' - 42: 2, # 'ψ' - 24: 0, # 'ω' - 19: 0, # 'ό' - 26: 0, # 'ύ' - 27: 0, # 'ώ' - }, - 26: { # 'ύ' - 60: 0, # 'e' - 55: 0, # 'o' - 58: 0, # 't' - 36: 0, # '·' - 61: 0, # 'Ά' - 46: 0, # 'Έ' - 54: 0, # 'Ό' - 31: 0, # 'Α' - 51: 0, # 'Β' - 43: 0, # 'Γ' - 41: 0, # 'Δ' - 34: 0, # 'Ε' - 40: 0, # 'Η' - 52: 0, # 'Θ' - 47: 0, # 'Ι' - 44: 0, # 'Κ' - 53: 0, # 'Λ' - 38: 0, # 'Μ' - 49: 0, # 'Ν' - 59: 0, # 'Ξ' - 39: 0, # 'Ο' - 35: 0, # 'Π' - 48: 0, # 'Ρ' - 37: 0, # 'Σ' - 33: 0, # 'Τ' - 45: 0, # 'Υ' - 56: 0, # 'Φ' - 50: 0, # 'Χ' - 57: 0, # 'Ω' - 17: 0, # 'ά' - 18: 0, # 'έ' - 22: 0, # 'ή' - 15: 0, # 'ί' - 1: 2, # 'α' - 29: 2, # 'β' - 20: 2, # 'γ' - 21: 1, # 'δ' - 3: 3, # 'ε' - 32: 0, # 'ζ' - 13: 2, # 'η' - 25: 3, # 'θ' - 5: 0, # 'ι' - 11: 3, # 'κ' - 16: 3, # 'λ' - 10: 3, # 'μ' - 6: 3, # 'ν' - 30: 2, # 'ξ' - 4: 3, # 'ο' - 9: 3, # 'π' - 8: 3, # 'ρ' - 14: 3, # 'ς' - 7: 3, # 'σ' - 2: 3, # 'τ' - 12: 0, # 'υ' - 28: 2, # 'φ' - 23: 2, # 'χ' - 42: 2, # 'ψ' - 24: 2, # 'ω' - 19: 0, # 'ό' - 26: 0, # 'ύ' - 27: 0, # 'ώ' - }, - 27: { # 'ώ' - 60: 0, # 'e' - 55: 0, # 'o' - 58: 0, # 't' - 36: 0, # '·' - 61: 0, # 'Ά' - 46: 0, # 'Έ' - 54: 0, # 'Ό' - 31: 0, # 'Α' - 51: 0, # 'Β' - 43: 0, # 'Γ' - 41: 0, # 'Δ' - 34: 0, # 'Ε' - 40: 0, # 'Η' - 52: 0, # 'Θ' - 47: 0, # 'Ι' - 44: 0, # 'Κ' - 53: 0, # 'Λ' - 38: 0, # 'Μ' - 49: 0, # 'Ν' - 59: 0, # 'Ξ' - 39: 0, # 'Ο' - 35: 0, # 'Π' - 48: 0, # 'Ρ' - 37: 0, # 'Σ' - 33: 0, # 'Τ' - 45: 0, # 'Υ' - 56: 0, # 'Φ' - 50: 0, # 'Χ' - 57: 0, # 'Ω' - 17: 0, # 'ά' - 18: 0, # 'έ' - 22: 0, # 'ή' - 15: 0, # 'ί' - 1: 0, # 'α' - 29: 1, # 'β' - 20: 0, # 'γ' - 21: 3, # 'δ' - 3: 0, # 'ε' - 32: 0, # 'ζ' - 13: 1, # 'η' - 25: 2, # 'θ' - 5: 2, # 'ι' - 11: 0, # 'κ' - 16: 2, # 'λ' - 10: 3, # 'μ' - 6: 3, # 'ν' - 30: 1, # 'ξ' - 4: 0, # 'ο' - 9: 2, # 'π' - 8: 3, # 'ρ' - 14: 3, # 'ς' - 7: 3, # 'σ' - 2: 3, # 'τ' - 12: 0, # 'υ' - 28: 1, # 'φ' - 23: 1, # 'χ' - 42: 0, # 'ψ' - 24: 0, # 'ω' - 19: 0, # 'ό' - 26: 0, # 'ύ' - 27: 0, # 'ώ' - }, -} - -# 255: Undefined characters that did not exist in training text -# 254: Carriage/Return -# 253: symbol (punctuation) that does not belong to word -# 252: 0 - 9 -# 251: Control characters - -# Character Mapping Table(s): -WINDOWS_1253_GREEK_CHAR_TO_ORDER = { - 0: 255, # '\x00' - 1: 255, # '\x01' - 2: 255, # '\x02' - 3: 255, # '\x03' - 4: 255, # '\x04' - 5: 255, # '\x05' - 6: 255, # '\x06' - 7: 255, # '\x07' - 8: 255, # '\x08' - 9: 255, # '\t' - 10: 254, # '\n' - 11: 255, # '\x0b' - 12: 255, # '\x0c' - 13: 254, # '\r' - 14: 255, # '\x0e' - 15: 255, # '\x0f' - 16: 255, # '\x10' - 17: 255, # '\x11' - 18: 255, # '\x12' - 19: 255, # '\x13' - 20: 255, # '\x14' - 21: 255, # '\x15' - 22: 255, # '\x16' - 23: 255, # '\x17' - 24: 255, # '\x18' - 25: 255, # '\x19' - 26: 255, # '\x1a' - 27: 255, # '\x1b' - 28: 255, # '\x1c' - 29: 255, # '\x1d' - 30: 255, # '\x1e' - 31: 255, # '\x1f' - 32: 253, # ' ' - 33: 253, # '!' - 34: 253, # '"' - 35: 253, # '#' - 36: 253, # '$' - 37: 253, # '%' - 38: 253, # '&' - 39: 253, # "'" - 40: 253, # '(' - 41: 253, # ')' - 42: 253, # '*' - 43: 253, # '+' - 44: 253, # ',' - 45: 253, # '-' - 46: 253, # '.' - 47: 253, # '/' - 48: 252, # '0' - 49: 252, # '1' - 50: 252, # '2' - 51: 252, # '3' - 52: 252, # '4' - 53: 252, # '5' - 54: 252, # '6' - 55: 252, # '7' - 56: 252, # '8' - 57: 252, # '9' - 58: 253, # ':' - 59: 253, # ';' - 60: 253, # '<' - 61: 253, # '=' - 62: 253, # '>' - 63: 253, # '?' - 64: 253, # '@' - 65: 82, # 'A' - 66: 100, # 'B' - 67: 104, # 'C' - 68: 94, # 'D' - 69: 98, # 'E' - 70: 101, # 'F' - 71: 116, # 'G' - 72: 102, # 'H' - 73: 111, # 'I' - 74: 187, # 'J' - 75: 117, # 'K' - 76: 92, # 'L' - 77: 88, # 'M' - 78: 113, # 'N' - 79: 85, # 'O' - 80: 79, # 'P' - 81: 118, # 'Q' - 82: 105, # 'R' - 83: 83, # 'S' - 84: 67, # 'T' - 85: 114, # 'U' - 86: 119, # 'V' - 87: 95, # 'W' - 88: 99, # 'X' - 89: 109, # 'Y' - 90: 188, # 'Z' - 91: 253, # '[' - 92: 253, # '\\' - 93: 253, # ']' - 94: 253, # '^' - 95: 253, # '_' - 96: 253, # '`' - 97: 72, # 'a' - 98: 70, # 'b' - 99: 80, # 'c' - 100: 81, # 'd' - 101: 60, # 'e' - 102: 96, # 'f' - 103: 93, # 'g' - 104: 89, # 'h' - 105: 68, # 'i' - 106: 120, # 'j' - 107: 97, # 'k' - 108: 77, # 'l' - 109: 86, # 'm' - 110: 69, # 'n' - 111: 55, # 'o' - 112: 78, # 'p' - 113: 115, # 'q' - 114: 65, # 'r' - 115: 66, # 's' - 116: 58, # 't' - 117: 76, # 'u' - 118: 106, # 'v' - 119: 103, # 'w' - 120: 87, # 'x' - 121: 107, # 'y' - 122: 112, # 'z' - 123: 253, # '{' - 124: 253, # '|' - 125: 253, # '}' - 126: 253, # '~' - 127: 253, # '\x7f' - 128: 255, # '€' - 129: 255, # None - 130: 255, # '‚' - 131: 255, # 'ƒ' - 132: 255, # '„' - 133: 255, # '…' - 134: 255, # '†' - 135: 255, # '‡' - 136: 255, # None - 137: 255, # '‰' - 138: 255, # None - 139: 255, # '‹' - 140: 255, # None - 141: 255, # None - 142: 255, # None - 143: 255, # None - 144: 255, # None - 145: 255, # '‘' - 146: 255, # '’' - 147: 255, # '“' - 148: 255, # '”' - 149: 255, # '•' - 150: 255, # '–' - 151: 255, # '—' - 152: 255, # None - 153: 255, # '™' - 154: 255, # None - 155: 255, # '›' - 156: 255, # None - 157: 255, # None - 158: 255, # None - 159: 255, # None - 160: 253, # '\xa0' - 161: 233, # '΅' - 162: 61, # 'Ά' - 163: 253, # '£' - 164: 253, # '¤' - 165: 253, # '¥' - 166: 253, # '¦' - 167: 253, # '§' - 168: 253, # '¨' - 169: 253, # '©' - 170: 253, # None - 171: 253, # '«' - 172: 253, # '¬' - 173: 74, # '\xad' - 174: 253, # '®' - 175: 253, # '―' - 176: 253, # '°' - 177: 253, # '±' - 178: 253, # '²' - 179: 253, # '³' - 180: 247, # '΄' - 181: 253, # 'µ' - 182: 253, # '¶' - 183: 36, # '·' - 184: 46, # 'Έ' - 185: 71, # 'Ή' - 186: 73, # 'Ί' - 187: 253, # '»' - 188: 54, # 'Ό' - 189: 253, # '½' - 190: 108, # 'Ύ' - 191: 123, # 'Ώ' - 192: 110, # 'ΐ' - 193: 31, # 'Α' - 194: 51, # 'Β' - 195: 43, # 'Γ' - 196: 41, # 'Δ' - 197: 34, # 'Ε' - 198: 91, # 'Ζ' - 199: 40, # 'Η' - 200: 52, # 'Θ' - 201: 47, # 'Ι' - 202: 44, # 'Κ' - 203: 53, # 'Λ' - 204: 38, # 'Μ' - 205: 49, # 'Ν' - 206: 59, # 'Ξ' - 207: 39, # 'Ο' - 208: 35, # 'Π' - 209: 48, # 'Ρ' - 210: 250, # None - 211: 37, # 'Σ' - 212: 33, # 'Τ' - 213: 45, # 'Υ' - 214: 56, # 'Φ' - 215: 50, # 'Χ' - 216: 84, # 'Ψ' - 217: 57, # 'Ω' - 218: 120, # 'Ϊ' - 219: 121, # 'Ϋ' - 220: 17, # 'ά' - 221: 18, # 'έ' - 222: 22, # 'ή' - 223: 15, # 'ί' - 224: 124, # 'ΰ' - 225: 1, # 'α' - 226: 29, # 'β' - 227: 20, # 'γ' - 228: 21, # 'δ' - 229: 3, # 'ε' - 230: 32, # 'ζ' - 231: 13, # 'η' - 232: 25, # 'θ' - 233: 5, # 'ι' - 234: 11, # 'κ' - 235: 16, # 'λ' - 236: 10, # 'μ' - 237: 6, # 'ν' - 238: 30, # 'ξ' - 239: 4, # 'ο' - 240: 9, # 'π' - 241: 8, # 'ρ' - 242: 14, # 'ς' - 243: 7, # 'σ' - 244: 2, # 'τ' - 245: 12, # 'υ' - 246: 28, # 'φ' - 247: 23, # 'χ' - 248: 42, # 'ψ' - 249: 24, # 'ω' - 250: 64, # 'ϊ' - 251: 75, # 'ϋ' - 252: 19, # 'ό' - 253: 26, # 'ύ' - 254: 27, # 'ώ' - 255: 253, # None -} - -WINDOWS_1253_GREEK_MODEL = SingleByteCharSetModel( - charset_name="windows-1253", - language="Greek", - char_to_order_map=WINDOWS_1253_GREEK_CHAR_TO_ORDER, - language_model=GREEK_LANG_MODEL, - typical_positive_ratio=0.982851, - keep_ascii_letters=False, - alphabet="ΆΈΉΊΌΎΏΑΒΓΔΕΖΗΘΙΚΛΜΝΞΟΠΡΣΤΥΦΧΨΩάέήίαβγδεζηθικλμνξοπρςστυφχψωόύώ", -) - -ISO_8859_7_GREEK_CHAR_TO_ORDER = { - 0: 255, # '\x00' - 1: 255, # '\x01' - 2: 255, # '\x02' - 3: 255, # '\x03' - 4: 255, # '\x04' - 5: 255, # '\x05' - 6: 255, # '\x06' - 7: 255, # '\x07' - 8: 255, # '\x08' - 9: 255, # '\t' - 10: 254, # '\n' - 11: 255, # '\x0b' - 12: 255, # '\x0c' - 13: 254, # '\r' - 14: 255, # '\x0e' - 15: 255, # '\x0f' - 16: 255, # '\x10' - 17: 255, # '\x11' - 18: 255, # '\x12' - 19: 255, # '\x13' - 20: 255, # '\x14' - 21: 255, # '\x15' - 22: 255, # '\x16' - 23: 255, # '\x17' - 24: 255, # '\x18' - 25: 255, # '\x19' - 26: 255, # '\x1a' - 27: 255, # '\x1b' - 28: 255, # '\x1c' - 29: 255, # '\x1d' - 30: 255, # '\x1e' - 31: 255, # '\x1f' - 32: 253, # ' ' - 33: 253, # '!' - 34: 253, # '"' - 35: 253, # '#' - 36: 253, # '$' - 37: 253, # '%' - 38: 253, # '&' - 39: 253, # "'" - 40: 253, # '(' - 41: 253, # ')' - 42: 253, # '*' - 43: 253, # '+' - 44: 253, # ',' - 45: 253, # '-' - 46: 253, # '.' - 47: 253, # '/' - 48: 252, # '0' - 49: 252, # '1' - 50: 252, # '2' - 51: 252, # '3' - 52: 252, # '4' - 53: 252, # '5' - 54: 252, # '6' - 55: 252, # '7' - 56: 252, # '8' - 57: 252, # '9' - 58: 253, # ':' - 59: 253, # ';' - 60: 253, # '<' - 61: 253, # '=' - 62: 253, # '>' - 63: 253, # '?' - 64: 253, # '@' - 65: 82, # 'A' - 66: 100, # 'B' - 67: 104, # 'C' - 68: 94, # 'D' - 69: 98, # 'E' - 70: 101, # 'F' - 71: 116, # 'G' - 72: 102, # 'H' - 73: 111, # 'I' - 74: 187, # 'J' - 75: 117, # 'K' - 76: 92, # 'L' - 77: 88, # 'M' - 78: 113, # 'N' - 79: 85, # 'O' - 80: 79, # 'P' - 81: 118, # 'Q' - 82: 105, # 'R' - 83: 83, # 'S' - 84: 67, # 'T' - 85: 114, # 'U' - 86: 119, # 'V' - 87: 95, # 'W' - 88: 99, # 'X' - 89: 109, # 'Y' - 90: 188, # 'Z' - 91: 253, # '[' - 92: 253, # '\\' - 93: 253, # ']' - 94: 253, # '^' - 95: 253, # '_' - 96: 253, # '`' - 97: 72, # 'a' - 98: 70, # 'b' - 99: 80, # 'c' - 100: 81, # 'd' - 101: 60, # 'e' - 102: 96, # 'f' - 103: 93, # 'g' - 104: 89, # 'h' - 105: 68, # 'i' - 106: 120, # 'j' - 107: 97, # 'k' - 108: 77, # 'l' - 109: 86, # 'm' - 110: 69, # 'n' - 111: 55, # 'o' - 112: 78, # 'p' - 113: 115, # 'q' - 114: 65, # 'r' - 115: 66, # 's' - 116: 58, # 't' - 117: 76, # 'u' - 118: 106, # 'v' - 119: 103, # 'w' - 120: 87, # 'x' - 121: 107, # 'y' - 122: 112, # 'z' - 123: 253, # '{' - 124: 253, # '|' - 125: 253, # '}' - 126: 253, # '~' - 127: 253, # '\x7f' - 128: 255, # '\x80' - 129: 255, # '\x81' - 130: 255, # '\x82' - 131: 255, # '\x83' - 132: 255, # '\x84' - 133: 255, # '\x85' - 134: 255, # '\x86' - 135: 255, # '\x87' - 136: 255, # '\x88' - 137: 255, # '\x89' - 138: 255, # '\x8a' - 139: 255, # '\x8b' - 140: 255, # '\x8c' - 141: 255, # '\x8d' - 142: 255, # '\x8e' - 143: 255, # '\x8f' - 144: 255, # '\x90' - 145: 255, # '\x91' - 146: 255, # '\x92' - 147: 255, # '\x93' - 148: 255, # '\x94' - 149: 255, # '\x95' - 150: 255, # '\x96' - 151: 255, # '\x97' - 152: 255, # '\x98' - 153: 255, # '\x99' - 154: 255, # '\x9a' - 155: 255, # '\x9b' - 156: 255, # '\x9c' - 157: 255, # '\x9d' - 158: 255, # '\x9e' - 159: 255, # '\x9f' - 160: 253, # '\xa0' - 161: 233, # '‘' - 162: 90, # '’' - 163: 253, # '£' - 164: 253, # '€' - 165: 253, # '₯' - 166: 253, # '¦' - 167: 253, # '§' - 168: 253, # '¨' - 169: 253, # '©' - 170: 253, # 'ͺ' - 171: 253, # '«' - 172: 253, # '¬' - 173: 74, # '\xad' - 174: 253, # None - 175: 253, # '―' - 176: 253, # '°' - 177: 253, # '±' - 178: 253, # '²' - 179: 253, # '³' - 180: 247, # '΄' - 181: 248, # '΅' - 182: 61, # 'Ά' - 183: 36, # '·' - 184: 46, # 'Έ' - 185: 71, # 'Ή' - 186: 73, # 'Ί' - 187: 253, # '»' - 188: 54, # 'Ό' - 189: 253, # '½' - 190: 108, # 'Ύ' - 191: 123, # 'Ώ' - 192: 110, # 'ΐ' - 193: 31, # 'Α' - 194: 51, # 'Β' - 195: 43, # 'Γ' - 196: 41, # 'Δ' - 197: 34, # 'Ε' - 198: 91, # 'Ζ' - 199: 40, # 'Η' - 200: 52, # 'Θ' - 201: 47, # 'Ι' - 202: 44, # 'Κ' - 203: 53, # 'Λ' - 204: 38, # 'Μ' - 205: 49, # 'Ν' - 206: 59, # 'Ξ' - 207: 39, # 'Ο' - 208: 35, # 'Π' - 209: 48, # 'Ρ' - 210: 250, # None - 211: 37, # 'Σ' - 212: 33, # 'Τ' - 213: 45, # 'Υ' - 214: 56, # 'Φ' - 215: 50, # 'Χ' - 216: 84, # 'Ψ' - 217: 57, # 'Ω' - 218: 120, # 'Ϊ' - 219: 121, # 'Ϋ' - 220: 17, # 'ά' - 221: 18, # 'έ' - 222: 22, # 'ή' - 223: 15, # 'ί' - 224: 124, # 'ΰ' - 225: 1, # 'α' - 226: 29, # 'β' - 227: 20, # 'γ' - 228: 21, # 'δ' - 229: 3, # 'ε' - 230: 32, # 'ζ' - 231: 13, # 'η' - 232: 25, # 'θ' - 233: 5, # 'ι' - 234: 11, # 'κ' - 235: 16, # 'λ' - 236: 10, # 'μ' - 237: 6, # 'ν' - 238: 30, # 'ξ' - 239: 4, # 'ο' - 240: 9, # 'π' - 241: 8, # 'ρ' - 242: 14, # 'ς' - 243: 7, # 'σ' - 244: 2, # 'τ' - 245: 12, # 'υ' - 246: 28, # 'φ' - 247: 23, # 'χ' - 248: 42, # 'ψ' - 249: 24, # 'ω' - 250: 64, # 'ϊ' - 251: 75, # 'ϋ' - 252: 19, # 'ό' - 253: 26, # 'ύ' - 254: 27, # 'ώ' - 255: 253, # None -} - -ISO_8859_7_GREEK_MODEL = SingleByteCharSetModel( - charset_name="ISO-8859-7", - language="Greek", - char_to_order_map=ISO_8859_7_GREEK_CHAR_TO_ORDER, - language_model=GREEK_LANG_MODEL, - typical_positive_ratio=0.982851, - keep_ascii_letters=False, - alphabet="ΆΈΉΊΌΎΏΑΒΓΔΕΖΗΘΙΚΛΜΝΞΟΠΡΣΤΥΦΧΨΩάέήίαβγδεζηθικλμνξοπρςστυφχψωόύώ", -) diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/langhebrewmodel.py b/venv/lib/python3.11/site-packages/pip/_vendor/chardet/langhebrewmodel.py deleted file mode 100644 index 56d2975..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/langhebrewmodel.py +++ /dev/null @@ -1,4380 +0,0 @@ -from pip._vendor.chardet.sbcharsetprober import SingleByteCharSetModel - -# 3: Positive -# 2: Likely -# 1: Unlikely -# 0: Negative - -HEBREW_LANG_MODEL = { - 50: { # 'a' - 50: 0, # 'a' - 60: 1, # 'c' - 61: 1, # 'd' - 42: 1, # 'e' - 53: 1, # 'i' - 56: 2, # 'l' - 54: 2, # 'n' - 49: 0, # 'o' - 51: 2, # 'r' - 43: 1, # 's' - 44: 2, # 't' - 63: 1, # 'u' - 34: 0, # '\xa0' - 55: 0, # '´' - 48: 0, # '¼' - 39: 0, # '½' - 57: 0, # '¾' - 30: 0, # 'ְ' - 59: 0, # 'ֱ' - 41: 0, # 'ֲ' - 33: 0, # 'ִ' - 37: 0, # 'ֵ' - 36: 0, # 'ֶ' - 31: 0, # 'ַ' - 29: 0, # 'ָ' - 35: 0, # 'ֹ' - 62: 0, # 'ֻ' - 28: 0, # 'ּ' - 38: 0, # 'ׁ' - 45: 0, # 'ׂ' - 9: 0, # 'א' - 8: 0, # 'ב' - 20: 0, # 'ג' - 16: 0, # 'ד' - 3: 1, # 'ה' - 2: 0, # 'ו' - 24: 0, # 'ז' - 14: 0, # 'ח' - 22: 0, # 'ט' - 1: 0, # 'י' - 25: 0, # 'ך' - 15: 0, # 'כ' - 4: 0, # 'ל' - 11: 0, # 'ם' - 6: 1, # 'מ' - 23: 0, # 'ן' - 12: 0, # 'נ' - 19: 0, # 'ס' - 13: 0, # 'ע' - 26: 0, # 'ף' - 18: 0, # 'פ' - 27: 0, # 'ץ' - 21: 0, # 'צ' - 17: 1, # 'ק' - 7: 0, # 'ר' - 10: 1, # 'ש' - 5: 0, # 'ת' - 32: 0, # '–' - 52: 1, # '’' - 47: 0, # '“' - 46: 1, # '”' - 58: 0, # '†' - 40: 1, # '…' - }, - 60: { # 'c' - 50: 1, # 'a' - 60: 1, # 'c' - 61: 0, # 'd' - 42: 1, # 'e' - 53: 1, # 'i' - 56: 1, # 'l' - 54: 0, # 'n' - 49: 1, # 'o' - 51: 1, # 'r' - 43: 1, # 's' - 44: 2, # 't' - 63: 1, # 'u' - 34: 0, # '\xa0' - 55: 0, # '´' - 48: 0, # '¼' - 39: 0, # '½' - 57: 0, # '¾' - 30: 0, # 'ְ' - 59: 0, # 'ֱ' - 41: 0, # 'ֲ' - 33: 0, # 'ִ' - 37: 0, # 'ֵ' - 36: 0, # 'ֶ' - 31: 0, # 'ַ' - 29: 0, # 'ָ' - 35: 0, # 'ֹ' - 62: 0, # 'ֻ' - 28: 0, # 'ּ' - 38: 0, # 'ׁ' - 45: 0, # 'ׂ' - 9: 1, # 'א' - 8: 0, # 'ב' - 20: 0, # 'ג' - 16: 0, # 'ד' - 3: 1, # 'ה' - 2: 0, # 'ו' - 24: 0, # 'ז' - 14: 0, # 'ח' - 22: 0, # 'ט' - 1: 0, # 'י' - 25: 0, # 'ך' - 15: 0, # 'כ' - 4: 0, # 'ל' - 11: 0, # 'ם' - 6: 1, # 'מ' - 23: 0, # 'ן' - 12: 1, # 'נ' - 19: 0, # 'ס' - 13: 0, # 'ע' - 26: 0, # 'ף' - 18: 0, # 'פ' - 27: 0, # 'ץ' - 21: 0, # 'צ' - 17: 0, # 'ק' - 7: 0, # 'ר' - 10: 0, # 'ש' - 5: 0, # 'ת' - 32: 0, # '–' - 52: 0, # '’' - 47: 0, # '“' - 46: 1, # '”' - 58: 0, # '†' - 40: 1, # '…' - }, - 61: { # 'd' - 50: 1, # 'a' - 60: 0, # 'c' - 61: 1, # 'd' - 42: 1, # 'e' - 53: 1, # 'i' - 56: 1, # 'l' - 54: 1, # 'n' - 49: 2, # 'o' - 51: 1, # 'r' - 43: 1, # 's' - 44: 0, # 't' - 63: 1, # 'u' - 34: 0, # '\xa0' - 55: 0, # '´' - 48: 0, # '¼' - 39: 0, # '½' - 57: 0, # '¾' - 30: 0, # 'ְ' - 59: 0, # 'ֱ' - 41: 0, # 'ֲ' - 33: 0, # 'ִ' - 37: 0, # 'ֵ' - 36: 0, # 'ֶ' - 31: 0, # 'ַ' - 29: 0, # 'ָ' - 35: 0, # 'ֹ' - 62: 0, # 'ֻ' - 28: 0, # 'ּ' - 38: 0, # 'ׁ' - 45: 0, # 'ׂ' - 9: 0, # 'א' - 8: 0, # 'ב' - 20: 0, # 'ג' - 16: 0, # 'ד' - 3: 1, # 'ה' - 2: 0, # 'ו' - 24: 0, # 'ז' - 14: 0, # 'ח' - 22: 0, # 'ט' - 1: 0, # 'י' - 25: 0, # 'ך' - 15: 0, # 'כ' - 4: 0, # 'ל' - 11: 0, # 'ם' - 6: 0, # 'מ' - 23: 0, # 'ן' - 12: 0, # 'נ' - 19: 0, # 'ס' - 13: 0, # 'ע' - 26: 0, # 'ף' - 18: 0, # 'פ' - 27: 0, # 'ץ' - 21: 0, # 'צ' - 17: 0, # 'ק' - 7: 0, # 'ר' - 10: 0, # 'ש' - 5: 0, # 'ת' - 32: 1, # '–' - 52: 1, # '’' - 47: 0, # '“' - 46: 1, # '”' - 58: 0, # '†' - 40: 1, # '…' - }, - 42: { # 'e' - 50: 1, # 'a' - 60: 1, # 'c' - 61: 2, # 'd' - 42: 1, # 'e' - 53: 1, # 'i' - 56: 2, # 'l' - 54: 2, # 'n' - 49: 1, # 'o' - 51: 2, # 'r' - 43: 2, # 's' - 44: 2, # 't' - 63: 1, # 'u' - 34: 1, # '\xa0' - 55: 0, # '´' - 48: 0, # '¼' - 39: 0, # '½' - 57: 0, # '¾' - 30: 0, # 'ְ' - 59: 0, # 'ֱ' - 41: 0, # 'ֲ' - 33: 0, # 'ִ' - 37: 0, # 'ֵ' - 36: 0, # 'ֶ' - 31: 0, # 'ַ' - 29: 0, # 'ָ' - 35: 0, # 'ֹ' - 62: 0, # 'ֻ' - 28: 0, # 'ּ' - 38: 0, # 'ׁ' - 45: 0, # 'ׂ' - 9: 0, # 'א' - 8: 0, # 'ב' - 20: 0, # 'ג' - 16: 0, # 'ד' - 3: 0, # 'ה' - 2: 0, # 'ו' - 24: 0, # 'ז' - 14: 0, # 'ח' - 22: 0, # 'ט' - 1: 0, # 'י' - 25: 0, # 'ך' - 15: 0, # 'כ' - 4: 0, # 'ל' - 11: 0, # 'ם' - 6: 0, # 'מ' - 23: 0, # 'ן' - 12: 0, # 'נ' - 19: 0, # 'ס' - 13: 0, # 'ע' - 26: 0, # 'ף' - 18: 1, # 'פ' - 27: 0, # 'ץ' - 21: 0, # 'צ' - 17: 0, # 'ק' - 7: 0, # 'ר' - 10: 0, # 'ש' - 5: 0, # 'ת' - 32: 1, # '–' - 52: 2, # '’' - 47: 0, # '“' - 46: 1, # '”' - 58: 0, # '†' - 40: 1, # '…' - }, - 53: { # 'i' - 50: 1, # 'a' - 60: 2, # 'c' - 61: 1, # 'd' - 42: 1, # 'e' - 53: 0, # 'i' - 56: 1, # 'l' - 54: 2, # 'n' - 49: 2, # 'o' - 51: 1, # 'r' - 43: 2, # 's' - 44: 2, # 't' - 63: 1, # 'u' - 34: 0, # '\xa0' - 55: 1, # '´' - 48: 0, # '¼' - 39: 0, # '½' - 57: 0, # '¾' - 30: 0, # 'ְ' - 59: 0, # 'ֱ' - 41: 0, # 'ֲ' - 33: 0, # 'ִ' - 37: 0, # 'ֵ' - 36: 0, # 'ֶ' - 31: 0, # 'ַ' - 29: 0, # 'ָ' - 35: 0, # 'ֹ' - 62: 0, # 'ֻ' - 28: 0, # 'ּ' - 38: 0, # 'ׁ' - 45: 0, # 'ׂ' - 9: 0, # 'א' - 8: 0, # 'ב' - 20: 0, # 'ג' - 16: 0, # 'ד' - 3: 0, # 'ה' - 2: 0, # 'ו' - 24: 0, # 'ז' - 14: 0, # 'ח' - 22: 0, # 'ט' - 1: 0, # 'י' - 25: 0, # 'ך' - 15: 0, # 'כ' - 4: 0, # 'ל' - 11: 0, # 'ם' - 6: 0, # 'מ' - 23: 0, # 'ן' - 12: 0, # 'נ' - 19: 0, # 'ס' - 13: 0, # 'ע' - 26: 0, # 'ף' - 18: 0, # 'פ' - 27: 0, # 'ץ' - 21: 0, # 'צ' - 17: 0, # 'ק' - 7: 0, # 'ר' - 10: 0, # 'ש' - 5: 0, # 'ת' - 32: 0, # '–' - 52: 1, # '’' - 47: 0, # '“' - 46: 0, # '”' - 58: 0, # '†' - 40: 0, # '…' - }, - 56: { # 'l' - 50: 1, # 'a' - 60: 1, # 'c' - 61: 1, # 'd' - 42: 2, # 'e' - 53: 2, # 'i' - 56: 2, # 'l' - 54: 1, # 'n' - 49: 1, # 'o' - 51: 0, # 'r' - 43: 1, # 's' - 44: 1, # 't' - 63: 1, # 'u' - 34: 0, # '\xa0' - 55: 0, # '´' - 48: 0, # '¼' - 39: 0, # '½' - 57: 0, # '¾' - 30: 0, # 'ְ' - 59: 0, # 'ֱ' - 41: 0, # 'ֲ' - 33: 0, # 'ִ' - 37: 0, # 'ֵ' - 36: 0, # 'ֶ' - 31: 0, # 'ַ' - 29: 0, # 'ָ' - 35: 0, # 'ֹ' - 62: 0, # 'ֻ' - 28: 0, # 'ּ' - 38: 0, # 'ׁ' - 45: 0, # 'ׂ' - 9: 0, # 'א' - 8: 0, # 'ב' - 20: 0, # 'ג' - 16: 0, # 'ד' - 3: 0, # 'ה' - 2: 0, # 'ו' - 24: 0, # 'ז' - 14: 0, # 'ח' - 22: 0, # 'ט' - 1: 0, # 'י' - 25: 0, # 'ך' - 15: 0, # 'כ' - 4: 0, # 'ל' - 11: 0, # 'ם' - 6: 0, # 'מ' - 23: 0, # 'ן' - 12: 0, # 'נ' - 19: 0, # 'ס' - 13: 0, # 'ע' - 26: 0, # 'ף' - 18: 0, # 'פ' - 27: 0, # 'ץ' - 21: 0, # 'צ' - 17: 0, # 'ק' - 7: 0, # 'ר' - 10: 0, # 'ש' - 5: 0, # 'ת' - 32: 0, # '–' - 52: 1, # '’' - 47: 0, # '“' - 46: 1, # '”' - 58: 0, # '†' - 40: 1, # '…' - }, - 54: { # 'n' - 50: 1, # 'a' - 60: 1, # 'c' - 61: 1, # 'd' - 42: 1, # 'e' - 53: 1, # 'i' - 56: 1, # 'l' - 54: 1, # 'n' - 49: 1, # 'o' - 51: 0, # 'r' - 43: 1, # 's' - 44: 2, # 't' - 63: 1, # 'u' - 34: 0, # '\xa0' - 55: 0, # '´' - 48: 0, # '¼' - 39: 0, # '½' - 57: 0, # '¾' - 30: 0, # 'ְ' - 59: 0, # 'ֱ' - 41: 0, # 'ֲ' - 33: 0, # 'ִ' - 37: 0, # 'ֵ' - 36: 0, # 'ֶ' - 31: 0, # 'ַ' - 29: 0, # 'ָ' - 35: 0, # 'ֹ' - 62: 0, # 'ֻ' - 28: 0, # 'ּ' - 38: 0, # 'ׁ' - 45: 0, # 'ׂ' - 9: 0, # 'א' - 8: 0, # 'ב' - 20: 0, # 'ג' - 16: 0, # 'ד' - 3: 1, # 'ה' - 2: 0, # 'ו' - 24: 0, # 'ז' - 14: 0, # 'ח' - 22: 0, # 'ט' - 1: 0, # 'י' - 25: 0, # 'ך' - 15: 0, # 'כ' - 4: 0, # 'ל' - 11: 0, # 'ם' - 6: 0, # 'מ' - 23: 0, # 'ן' - 12: 0, # 'נ' - 19: 0, # 'ס' - 13: 0, # 'ע' - 26: 0, # 'ף' - 18: 0, # 'פ' - 27: 0, # 'ץ' - 21: 0, # 'צ' - 17: 0, # 'ק' - 7: 0, # 'ר' - 10: 0, # 'ש' - 5: 0, # 'ת' - 32: 0, # '–' - 52: 2, # '’' - 47: 0, # '“' - 46: 1, # '”' - 58: 0, # '†' - 40: 1, # '…' - }, - 49: { # 'o' - 50: 1, # 'a' - 60: 1, # 'c' - 61: 1, # 'd' - 42: 1, # 'e' - 53: 1, # 'i' - 56: 1, # 'l' - 54: 2, # 'n' - 49: 1, # 'o' - 51: 2, # 'r' - 43: 1, # 's' - 44: 1, # 't' - 63: 1, # 'u' - 34: 0, # '\xa0' - 55: 0, # '´' - 48: 0, # '¼' - 39: 0, # '½' - 57: 0, # '¾' - 30: 0, # 'ְ' - 59: 0, # 'ֱ' - 41: 0, # 'ֲ' - 33: 0, # 'ִ' - 37: 0, # 'ֵ' - 36: 0, # 'ֶ' - 31: 0, # 'ַ' - 29: 0, # 'ָ' - 35: 0, # 'ֹ' - 62: 0, # 'ֻ' - 28: 0, # 'ּ' - 38: 0, # 'ׁ' - 45: 0, # 'ׂ' - 9: 0, # 'א' - 8: 0, # 'ב' - 20: 0, # 'ג' - 16: 0, # 'ד' - 3: 0, # 'ה' - 2: 0, # 'ו' - 24: 0, # 'ז' - 14: 0, # 'ח' - 22: 0, # 'ט' - 1: 0, # 'י' - 25: 0, # 'ך' - 15: 0, # 'כ' - 4: 0, # 'ל' - 11: 0, # 'ם' - 6: 0, # 'מ' - 23: 0, # 'ן' - 12: 0, # 'נ' - 19: 0, # 'ס' - 13: 0, # 'ע' - 26: 0, # 'ף' - 18: 0, # 'פ' - 27: 0, # 'ץ' - 21: 0, # 'צ' - 17: 0, # 'ק' - 7: 0, # 'ר' - 10: 0, # 'ש' - 5: 0, # 'ת' - 32: 0, # '–' - 52: 1, # '’' - 47: 0, # '“' - 46: 1, # '”' - 58: 0, # '†' - 40: 1, # '…' - }, - 51: { # 'r' - 50: 2, # 'a' - 60: 1, # 'c' - 61: 1, # 'd' - 42: 2, # 'e' - 53: 1, # 'i' - 56: 1, # 'l' - 54: 1, # 'n' - 49: 2, # 'o' - 51: 1, # 'r' - 43: 1, # 's' - 44: 1, # 't' - 63: 1, # 'u' - 34: 0, # '\xa0' - 55: 0, # '´' - 48: 0, # '¼' - 39: 0, # '½' - 57: 0, # '¾' - 30: 0, # 'ְ' - 59: 0, # 'ֱ' - 41: 0, # 'ֲ' - 33: 0, # 'ִ' - 37: 0, # 'ֵ' - 36: 0, # 'ֶ' - 31: 0, # 'ַ' - 29: 0, # 'ָ' - 35: 0, # 'ֹ' - 62: 0, # 'ֻ' - 28: 0, # 'ּ' - 38: 0, # 'ׁ' - 45: 0, # 'ׂ' - 9: 0, # 'א' - 8: 0, # 'ב' - 20: 0, # 'ג' - 16: 0, # 'ד' - 3: 0, # 'ה' - 2: 0, # 'ו' - 24: 0, # 'ז' - 14: 0, # 'ח' - 22: 0, # 'ט' - 1: 0, # 'י' - 25: 0, # 'ך' - 15: 0, # 'כ' - 4: 0, # 'ל' - 11: 0, # 'ם' - 6: 0, # 'מ' - 23: 0, # 'ן' - 12: 0, # 'נ' - 19: 0, # 'ס' - 13: 0, # 'ע' - 26: 0, # 'ף' - 18: 0, # 'פ' - 27: 0, # 'ץ' - 21: 0, # 'צ' - 17: 0, # 'ק' - 7: 0, # 'ר' - 10: 0, # 'ש' - 5: 0, # 'ת' - 32: 0, # '–' - 52: 2, # '’' - 47: 0, # '“' - 46: 1, # '”' - 58: 0, # '†' - 40: 1, # '…' - }, - 43: { # 's' - 50: 1, # 'a' - 60: 1, # 'c' - 61: 0, # 'd' - 42: 2, # 'e' - 53: 1, # 'i' - 56: 1, # 'l' - 54: 1, # 'n' - 49: 1, # 'o' - 51: 1, # 'r' - 43: 1, # 's' - 44: 2, # 't' - 63: 1, # 'u' - 34: 0, # '\xa0' - 55: 0, # '´' - 48: 0, # '¼' - 39: 0, # '½' - 57: 0, # '¾' - 30: 0, # 'ְ' - 59: 0, # 'ֱ' - 41: 0, # 'ֲ' - 33: 0, # 'ִ' - 37: 0, # 'ֵ' - 36: 0, # 'ֶ' - 31: 0, # 'ַ' - 29: 0, # 'ָ' - 35: 0, # 'ֹ' - 62: 0, # 'ֻ' - 28: 0, # 'ּ' - 38: 0, # 'ׁ' - 45: 0, # 'ׂ' - 9: 0, # 'א' - 8: 0, # 'ב' - 20: 0, # 'ג' - 16: 0, # 'ד' - 3: 0, # 'ה' - 2: 0, # 'ו' - 24: 0, # 'ז' - 14: 0, # 'ח' - 22: 0, # 'ט' - 1: 0, # 'י' - 25: 0, # 'ך' - 15: 0, # 'כ' - 4: 0, # 'ל' - 11: 0, # 'ם' - 6: 0, # 'מ' - 23: 0, # 'ן' - 12: 0, # 'נ' - 19: 0, # 'ס' - 13: 0, # 'ע' - 26: 0, # 'ף' - 18: 0, # 'פ' - 27: 0, # 'ץ' - 21: 0, # 'צ' - 17: 0, # 'ק' - 7: 0, # 'ר' - 10: 0, # 'ש' - 5: 0, # 'ת' - 32: 0, # '–' - 52: 1, # '’' - 47: 0, # '“' - 46: 2, # '”' - 58: 0, # '†' - 40: 2, # '…' - }, - 44: { # 't' - 50: 1, # 'a' - 60: 1, # 'c' - 61: 0, # 'd' - 42: 2, # 'e' - 53: 2, # 'i' - 56: 1, # 'l' - 54: 0, # 'n' - 49: 1, # 'o' - 51: 1, # 'r' - 43: 1, # 's' - 44: 1, # 't' - 63: 1, # 'u' - 34: 1, # '\xa0' - 55: 0, # '´' - 48: 0, # '¼' - 39: 0, # '½' - 57: 0, # '¾' - 30: 0, # 'ְ' - 59: 0, # 'ֱ' - 41: 0, # 'ֲ' - 33: 0, # 'ִ' - 37: 0, # 'ֵ' - 36: 0, # 'ֶ' - 31: 0, # 'ַ' - 29: 0, # 'ָ' - 35: 0, # 'ֹ' - 62: 0, # 'ֻ' - 28: 0, # 'ּ' - 38: 0, # 'ׁ' - 45: 0, # 'ׂ' - 9: 0, # 'א' - 8: 0, # 'ב' - 20: 0, # 'ג' - 16: 0, # 'ד' - 3: 0, # 'ה' - 2: 0, # 'ו' - 24: 0, # 'ז' - 14: 0, # 'ח' - 22: 0, # 'ט' - 1: 0, # 'י' - 25: 0, # 'ך' - 15: 0, # 'כ' - 4: 0, # 'ל' - 11: 0, # 'ם' - 6: 0, # 'מ' - 23: 0, # 'ן' - 12: 0, # 'נ' - 19: 0, # 'ס' - 13: 0, # 'ע' - 26: 0, # 'ף' - 18: 0, # 'פ' - 27: 0, # 'ץ' - 21: 0, # 'צ' - 17: 0, # 'ק' - 7: 0, # 'ר' - 10: 0, # 'ש' - 5: 0, # 'ת' - 32: 0, # '–' - 52: 2, # '’' - 47: 0, # '“' - 46: 1, # '”' - 58: 0, # '†' - 40: 1, # '…' - }, - 63: { # 'u' - 50: 1, # 'a' - 60: 1, # 'c' - 61: 1, # 'd' - 42: 1, # 'e' - 53: 1, # 'i' - 56: 1, # 'l' - 54: 1, # 'n' - 49: 0, # 'o' - 51: 1, # 'r' - 43: 2, # 's' - 44: 1, # 't' - 63: 0, # 'u' - 34: 0, # '\xa0' - 55: 0, # '´' - 48: 0, # '¼' - 39: 0, # '½' - 57: 0, # '¾' - 30: 0, # 'ְ' - 59: 0, # 'ֱ' - 41: 0, # 'ֲ' - 33: 0, # 'ִ' - 37: 0, # 'ֵ' - 36: 0, # 'ֶ' - 31: 0, # 'ַ' - 29: 0, # 'ָ' - 35: 0, # 'ֹ' - 62: 0, # 'ֻ' - 28: 0, # 'ּ' - 38: 0, # 'ׁ' - 45: 0, # 'ׂ' - 9: 0, # 'א' - 8: 0, # 'ב' - 20: 0, # 'ג' - 16: 0, # 'ד' - 3: 0, # 'ה' - 2: 0, # 'ו' - 24: 0, # 'ז' - 14: 0, # 'ח' - 22: 0, # 'ט' - 1: 0, # 'י' - 25: 0, # 'ך' - 15: 0, # 'כ' - 4: 0, # 'ל' - 11: 0, # 'ם' - 6: 0, # 'מ' - 23: 0, # 'ן' - 12: 0, # 'נ' - 19: 0, # 'ס' - 13: 0, # 'ע' - 26: 0, # 'ף' - 18: 0, # 'פ' - 27: 0, # 'ץ' - 21: 0, # 'צ' - 17: 0, # 'ק' - 7: 0, # 'ר' - 10: 0, # 'ש' - 5: 0, # 'ת' - 32: 0, # '–' - 52: 1, # '’' - 47: 0, # '“' - 46: 0, # '”' - 58: 0, # '†' - 40: 0, # '…' - }, - 34: { # '\xa0' - 50: 1, # 'a' - 60: 0, # 'c' - 61: 1, # 'd' - 42: 0, # 'e' - 53: 1, # 'i' - 56: 0, # 'l' - 54: 1, # 'n' - 49: 1, # 'o' - 51: 0, # 'r' - 43: 1, # 's' - 44: 1, # 't' - 63: 0, # 'u' - 34: 2, # '\xa0' - 55: 0, # '´' - 48: 0, # '¼' - 39: 0, # '½' - 57: 0, # '¾' - 30: 0, # 'ְ' - 59: 0, # 'ֱ' - 41: 0, # 'ֲ' - 33: 0, # 'ִ' - 37: 0, # 'ֵ' - 36: 0, # 'ֶ' - 31: 0, # 'ַ' - 29: 0, # 'ָ' - 35: 0, # 'ֹ' - 62: 0, # 'ֻ' - 28: 0, # 'ּ' - 38: 0, # 'ׁ' - 45: 0, # 'ׂ' - 9: 2, # 'א' - 8: 1, # 'ב' - 20: 1, # 'ג' - 16: 1, # 'ד' - 3: 1, # 'ה' - 2: 1, # 'ו' - 24: 1, # 'ז' - 14: 1, # 'ח' - 22: 1, # 'ט' - 1: 2, # 'י' - 25: 0, # 'ך' - 15: 1, # 'כ' - 4: 1, # 'ל' - 11: 0, # 'ם' - 6: 2, # 'מ' - 23: 0, # 'ן' - 12: 1, # 'נ' - 19: 1, # 'ס' - 13: 1, # 'ע' - 26: 0, # 'ף' - 18: 1, # 'פ' - 27: 0, # 'ץ' - 21: 1, # 'צ' - 17: 1, # 'ק' - 7: 1, # 'ר' - 10: 1, # 'ש' - 5: 1, # 'ת' - 32: 0, # '–' - 52: 0, # '’' - 47: 0, # '“' - 46: 0, # '”' - 58: 0, # '†' - 40: 0, # '…' - }, - 55: { # '´' - 50: 0, # 'a' - 60: 0, # 'c' - 61: 0, # 'd' - 42: 0, # 'e' - 53: 0, # 'i' - 56: 0, # 'l' - 54: 0, # 'n' - 49: 0, # 'o' - 51: 0, # 'r' - 43: 1, # 's' - 44: 0, # 't' - 63: 0, # 'u' - 34: 0, # '\xa0' - 55: 0, # '´' - 48: 0, # '¼' - 39: 0, # '½' - 57: 0, # '¾' - 30: 0, # 'ְ' - 59: 0, # 'ֱ' - 41: 0, # 'ֲ' - 33: 0, # 'ִ' - 37: 0, # 'ֵ' - 36: 0, # 'ֶ' - 31: 0, # 'ַ' - 29: 0, # 'ָ' - 35: 0, # 'ֹ' - 62: 0, # 'ֻ' - 28: 0, # 'ּ' - 38: 0, # 'ׁ' - 45: 0, # 'ׂ' - 9: 1, # 'א' - 8: 0, # 'ב' - 20: 0, # 'ג' - 16: 0, # 'ד' - 3: 1, # 'ה' - 2: 1, # 'ו' - 24: 0, # 'ז' - 14: 0, # 'ח' - 22: 0, # 'ט' - 1: 2, # 'י' - 25: 0, # 'ך' - 15: 0, # 'כ' - 4: 1, # 'ל' - 11: 0, # 'ם' - 6: 1, # 'מ' - 23: 1, # 'ן' - 12: 1, # 'נ' - 19: 1, # 'ס' - 13: 0, # 'ע' - 26: 0, # 'ף' - 18: 0, # 'פ' - 27: 0, # 'ץ' - 21: 0, # 'צ' - 17: 0, # 'ק' - 7: 1, # 'ר' - 10: 1, # 'ש' - 5: 0, # 'ת' - 32: 0, # '–' - 52: 0, # '’' - 47: 0, # '“' - 46: 0, # '”' - 58: 0, # '†' - 40: 0, # '…' - }, - 48: { # '¼' - 50: 0, # 'a' - 60: 0, # 'c' - 61: 0, # 'd' - 42: 0, # 'e' - 53: 0, # 'i' - 56: 0, # 'l' - 54: 0, # 'n' - 49: 0, # 'o' - 51: 0, # 'r' - 43: 0, # 's' - 44: 0, # 't' - 63: 0, # 'u' - 34: 0, # '\xa0' - 55: 0, # '´' - 48: 0, # '¼' - 39: 0, # '½' - 57: 0, # '¾' - 30: 0, # 'ְ' - 59: 0, # 'ֱ' - 41: 0, # 'ֲ' - 33: 0, # 'ִ' - 37: 0, # 'ֵ' - 36: 0, # 'ֶ' - 31: 0, # 'ַ' - 29: 0, # 'ָ' - 35: 0, # 'ֹ' - 62: 0, # 'ֻ' - 28: 0, # 'ּ' - 38: 0, # 'ׁ' - 45: 0, # 'ׂ' - 9: 1, # 'א' - 8: 0, # 'ב' - 20: 0, # 'ג' - 16: 0, # 'ד' - 3: 0, # 'ה' - 2: 1, # 'ו' - 24: 0, # 'ז' - 14: 0, # 'ח' - 22: 0, # 'ט' - 1: 0, # 'י' - 25: 0, # 'ך' - 15: 1, # 'כ' - 4: 1, # 'ל' - 11: 0, # 'ם' - 6: 1, # 'מ' - 23: 0, # 'ן' - 12: 0, # 'נ' - 19: 0, # 'ס' - 13: 0, # 'ע' - 26: 0, # 'ף' - 18: 0, # 'פ' - 27: 0, # 'ץ' - 21: 0, # 'צ' - 17: 0, # 'ק' - 7: 0, # 'ר' - 10: 0, # 'ש' - 5: 0, # 'ת' - 32: 0, # '–' - 52: 0, # '’' - 47: 0, # '“' - 46: 0, # '”' - 58: 0, # '†' - 40: 0, # '…' - }, - 39: { # '½' - 50: 0, # 'a' - 60: 0, # 'c' - 61: 0, # 'd' - 42: 0, # 'e' - 53: 0, # 'i' - 56: 0, # 'l' - 54: 0, # 'n' - 49: 0, # 'o' - 51: 0, # 'r' - 43: 0, # 's' - 44: 0, # 't' - 63: 0, # 'u' - 34: 0, # '\xa0' - 55: 0, # '´' - 48: 0, # '¼' - 39: 0, # '½' - 57: 0, # '¾' - 30: 0, # 'ְ' - 59: 0, # 'ֱ' - 41: 0, # 'ֲ' - 33: 0, # 'ִ' - 37: 0, # 'ֵ' - 36: 0, # 'ֶ' - 31: 0, # 'ַ' - 29: 0, # 'ָ' - 35: 0, # 'ֹ' - 62: 0, # 'ֻ' - 28: 0, # 'ּ' - 38: 0, # 'ׁ' - 45: 0, # 'ׂ' - 9: 0, # 'א' - 8: 0, # 'ב' - 20: 0, # 'ג' - 16: 0, # 'ד' - 3: 0, # 'ה' - 2: 0, # 'ו' - 24: 0, # 'ז' - 14: 0, # 'ח' - 22: 0, # 'ט' - 1: 0, # 'י' - 25: 0, # 'ך' - 15: 1, # 'כ' - 4: 1, # 'ל' - 11: 0, # 'ם' - 6: 0, # 'מ' - 23: 0, # 'ן' - 12: 0, # 'נ' - 19: 0, # 'ס' - 13: 0, # 'ע' - 26: 0, # 'ף' - 18: 0, # 'פ' - 27: 0, # 'ץ' - 21: 1, # 'צ' - 17: 1, # 'ק' - 7: 0, # 'ר' - 10: 0, # 'ש' - 5: 0, # 'ת' - 32: 0, # '–' - 52: 0, # '’' - 47: 0, # '“' - 46: 0, # '”' - 58: 0, # '†' - 40: 0, # '…' - }, - 57: { # '¾' - 50: 0, # 'a' - 60: 0, # 'c' - 61: 0, # 'd' - 42: 0, # 'e' - 53: 0, # 'i' - 56: 0, # 'l' - 54: 0, # 'n' - 49: 0, # 'o' - 51: 0, # 'r' - 43: 0, # 's' - 44: 0, # 't' - 63: 0, # 'u' - 34: 0, # '\xa0' - 55: 0, # '´' - 48: 0, # '¼' - 39: 0, # '½' - 57: 0, # '¾' - 30: 0, # 'ְ' - 59: 0, # 'ֱ' - 41: 0, # 'ֲ' - 33: 0, # 'ִ' - 37: 0, # 'ֵ' - 36: 0, # 'ֶ' - 31: 0, # 'ַ' - 29: 0, # 'ָ' - 35: 0, # 'ֹ' - 62: 0, # 'ֻ' - 28: 0, # 'ּ' - 38: 0, # 'ׁ' - 45: 0, # 'ׂ' - 9: 0, # 'א' - 8: 0, # 'ב' - 20: 0, # 'ג' - 16: 0, # 'ד' - 3: 0, # 'ה' - 2: 0, # 'ו' - 24: 0, # 'ז' - 14: 0, # 'ח' - 22: 0, # 'ט' - 1: 0, # 'י' - 25: 0, # 'ך' - 15: 0, # 'כ' - 4: 0, # 'ל' - 11: 0, # 'ם' - 6: 0, # 'מ' - 23: 0, # 'ן' - 12: 0, # 'נ' - 19: 0, # 'ס' - 13: 0, # 'ע' - 26: 0, # 'ף' - 18: 0, # 'פ' - 27: 0, # 'ץ' - 21: 0, # 'צ' - 17: 0, # 'ק' - 7: 0, # 'ר' - 10: 0, # 'ש' - 5: 0, # 'ת' - 32: 0, # '–' - 52: 0, # '’' - 47: 0, # '“' - 46: 0, # '”' - 58: 0, # '†' - 40: 0, # '…' - }, - 30: { # 'ְ' - 50: 0, # 'a' - 60: 0, # 'c' - 61: 0, # 'd' - 42: 0, # 'e' - 53: 0, # 'i' - 56: 0, # 'l' - 54: 0, # 'n' - 49: 0, # 'o' - 51: 0, # 'r' - 43: 0, # 's' - 44: 0, # 't' - 63: 0, # 'u' - 34: 0, # '\xa0' - 55: 0, # '´' - 48: 0, # '¼' - 39: 0, # '½' - 57: 0, # '¾' - 30: 0, # 'ְ' - 59: 0, # 'ֱ' - 41: 0, # 'ֲ' - 33: 0, # 'ִ' - 37: 0, # 'ֵ' - 36: 1, # 'ֶ' - 31: 0, # 'ַ' - 29: 0, # 'ָ' - 35: 1, # 'ֹ' - 62: 0, # 'ֻ' - 28: 0, # 'ּ' - 38: 0, # 'ׁ' - 45: 0, # 'ׂ' - 9: 2, # 'א' - 8: 2, # 'ב' - 20: 2, # 'ג' - 16: 2, # 'ד' - 3: 2, # 'ה' - 2: 2, # 'ו' - 24: 2, # 'ז' - 14: 2, # 'ח' - 22: 2, # 'ט' - 1: 2, # 'י' - 25: 2, # 'ך' - 15: 2, # 'כ' - 4: 2, # 'ל' - 11: 1, # 'ם' - 6: 2, # 'מ' - 23: 0, # 'ן' - 12: 2, # 'נ' - 19: 2, # 'ס' - 13: 2, # 'ע' - 26: 0, # 'ף' - 18: 2, # 'פ' - 27: 0, # 'ץ' - 21: 2, # 'צ' - 17: 2, # 'ק' - 7: 2, # 'ר' - 10: 2, # 'ש' - 5: 2, # 'ת' - 32: 0, # '–' - 52: 0, # '’' - 47: 0, # '“' - 46: 0, # '”' - 58: 0, # '†' - 40: 0, # '…' - }, - 59: { # 'ֱ' - 50: 0, # 'a' - 60: 0, # 'c' - 61: 0, # 'd' - 42: 0, # 'e' - 53: 0, # 'i' - 56: 0, # 'l' - 54: 0, # 'n' - 49: 0, # 'o' - 51: 0, # 'r' - 43: 0, # 's' - 44: 0, # 't' - 63: 0, # 'u' - 34: 0, # '\xa0' - 55: 0, # '´' - 48: 0, # '¼' - 39: 0, # '½' - 57: 0, # '¾' - 30: 1, # 'ְ' - 59: 0, # 'ֱ' - 41: 0, # 'ֲ' - 33: 0, # 'ִ' - 37: 0, # 'ֵ' - 36: 0, # 'ֶ' - 31: 0, # 'ַ' - 29: 0, # 'ָ' - 35: 0, # 'ֹ' - 62: 0, # 'ֻ' - 28: 0, # 'ּ' - 38: 0, # 'ׁ' - 45: 0, # 'ׂ' - 9: 0, # 'א' - 8: 1, # 'ב' - 20: 1, # 'ג' - 16: 0, # 'ד' - 3: 0, # 'ה' - 2: 0, # 'ו' - 24: 1, # 'ז' - 14: 0, # 'ח' - 22: 0, # 'ט' - 1: 1, # 'י' - 25: 0, # 'ך' - 15: 1, # 'כ' - 4: 2, # 'ל' - 11: 0, # 'ם' - 6: 2, # 'מ' - 23: 0, # 'ן' - 12: 1, # 'נ' - 19: 0, # 'ס' - 13: 0, # 'ע' - 26: 0, # 'ף' - 18: 0, # 'פ' - 27: 0, # 'ץ' - 21: 0, # 'צ' - 17: 0, # 'ק' - 7: 1, # 'ר' - 10: 1, # 'ש' - 5: 0, # 'ת' - 32: 0, # '–' - 52: 0, # '’' - 47: 0, # '“' - 46: 0, # '”' - 58: 0, # '†' - 40: 0, # '…' - }, - 41: { # 'ֲ' - 50: 0, # 'a' - 60: 0, # 'c' - 61: 0, # 'd' - 42: 0, # 'e' - 53: 0, # 'i' - 56: 0, # 'l' - 54: 0, # 'n' - 49: 0, # 'o' - 51: 0, # 'r' - 43: 0, # 's' - 44: 0, # 't' - 63: 0, # 'u' - 34: 0, # '\xa0' - 55: 0, # '´' - 48: 0, # '¼' - 39: 0, # '½' - 57: 0, # '¾' - 30: 0, # 'ְ' - 59: 0, # 'ֱ' - 41: 0, # 'ֲ' - 33: 0, # 'ִ' - 37: 0, # 'ֵ' - 36: 0, # 'ֶ' - 31: 0, # 'ַ' - 29: 0, # 'ָ' - 35: 0, # 'ֹ' - 62: 0, # 'ֻ' - 28: 0, # 'ּ' - 38: 0, # 'ׁ' - 45: 0, # 'ׂ' - 9: 0, # 'א' - 8: 2, # 'ב' - 20: 1, # 'ג' - 16: 2, # 'ד' - 3: 1, # 'ה' - 2: 1, # 'ו' - 24: 1, # 'ז' - 14: 1, # 'ח' - 22: 1, # 'ט' - 1: 1, # 'י' - 25: 1, # 'ך' - 15: 1, # 'כ' - 4: 2, # 'ל' - 11: 0, # 'ם' - 6: 2, # 'מ' - 23: 0, # 'ן' - 12: 2, # 'נ' - 19: 1, # 'ס' - 13: 0, # 'ע' - 26: 0, # 'ף' - 18: 1, # 'פ' - 27: 0, # 'ץ' - 21: 2, # 'צ' - 17: 1, # 'ק' - 7: 2, # 'ר' - 10: 2, # 'ש' - 5: 1, # 'ת' - 32: 0, # '–' - 52: 0, # '’' - 47: 0, # '“' - 46: 0, # '”' - 58: 0, # '†' - 40: 0, # '…' - }, - 33: { # 'ִ' - 50: 0, # 'a' - 60: 0, # 'c' - 61: 0, # 'd' - 42: 0, # 'e' - 53: 0, # 'i' - 56: 0, # 'l' - 54: 0, # 'n' - 49: 0, # 'o' - 51: 0, # 'r' - 43: 0, # 's' - 44: 0, # 't' - 63: 0, # 'u' - 34: 0, # '\xa0' - 55: 0, # '´' - 48: 0, # '¼' - 39: 0, # '½' - 57: 0, # '¾' - 30: 1, # 'ְ' - 59: 0, # 'ֱ' - 41: 0, # 'ֲ' - 33: 1, # 'ִ' - 37: 0, # 'ֵ' - 36: 1, # 'ֶ' - 31: 0, # 'ַ' - 29: 1, # 'ָ' - 35: 0, # 'ֹ' - 62: 0, # 'ֻ' - 28: 1, # 'ּ' - 38: 0, # 'ׁ' - 45: 0, # 'ׂ' - 9: 1, # 'א' - 8: 2, # 'ב' - 20: 2, # 'ג' - 16: 2, # 'ד' - 3: 1, # 'ה' - 2: 1, # 'ו' - 24: 2, # 'ז' - 14: 1, # 'ח' - 22: 1, # 'ט' - 1: 3, # 'י' - 25: 1, # 'ך' - 15: 2, # 'כ' - 4: 2, # 'ל' - 11: 2, # 'ם' - 6: 2, # 'מ' - 23: 2, # 'ן' - 12: 2, # 'נ' - 19: 2, # 'ס' - 13: 1, # 'ע' - 26: 0, # 'ף' - 18: 2, # 'פ' - 27: 1, # 'ץ' - 21: 2, # 'צ' - 17: 2, # 'ק' - 7: 2, # 'ר' - 10: 2, # 'ש' - 5: 2, # 'ת' - 32: 0, # '–' - 52: 0, # '’' - 47: 0, # '“' - 46: 0, # '”' - 58: 0, # '†' - 40: 0, # '…' - }, - 37: { # 'ֵ' - 50: 0, # 'a' - 60: 0, # 'c' - 61: 0, # 'd' - 42: 0, # 'e' - 53: 0, # 'i' - 56: 0, # 'l' - 54: 0, # 'n' - 49: 0, # 'o' - 51: 0, # 'r' - 43: 0, # 's' - 44: 0, # 't' - 63: 0, # 'u' - 34: 0, # '\xa0' - 55: 0, # '´' - 48: 0, # '¼' - 39: 0, # '½' - 57: 0, # '¾' - 30: 0, # 'ְ' - 59: 0, # 'ֱ' - 41: 0, # 'ֲ' - 33: 0, # 'ִ' - 37: 0, # 'ֵ' - 36: 1, # 'ֶ' - 31: 1, # 'ַ' - 29: 1, # 'ָ' - 35: 0, # 'ֹ' - 62: 0, # 'ֻ' - 28: 0, # 'ּ' - 38: 0, # 'ׁ' - 45: 0, # 'ׂ' - 9: 2, # 'א' - 8: 2, # 'ב' - 20: 1, # 'ג' - 16: 2, # 'ד' - 3: 2, # 'ה' - 2: 1, # 'ו' - 24: 1, # 'ז' - 14: 2, # 'ח' - 22: 1, # 'ט' - 1: 3, # 'י' - 25: 2, # 'ך' - 15: 1, # 'כ' - 4: 2, # 'ל' - 11: 2, # 'ם' - 6: 1, # 'מ' - 23: 2, # 'ן' - 12: 2, # 'נ' - 19: 1, # 'ס' - 13: 2, # 'ע' - 26: 1, # 'ף' - 18: 1, # 'פ' - 27: 1, # 'ץ' - 21: 1, # 'צ' - 17: 1, # 'ק' - 7: 2, # 'ר' - 10: 2, # 'ש' - 5: 2, # 'ת' - 32: 0, # '–' - 52: 0, # '’' - 47: 0, # '“' - 46: 0, # '”' - 58: 0, # '†' - 40: 0, # '…' - }, - 36: { # 'ֶ' - 50: 0, # 'a' - 60: 0, # 'c' - 61: 0, # 'd' - 42: 0, # 'e' - 53: 0, # 'i' - 56: 0, # 'l' - 54: 0, # 'n' - 49: 0, # 'o' - 51: 0, # 'r' - 43: 0, # 's' - 44: 0, # 't' - 63: 0, # 'u' - 34: 0, # '\xa0' - 55: 0, # '´' - 48: 0, # '¼' - 39: 0, # '½' - 57: 0, # '¾' - 30: 0, # 'ְ' - 59: 0, # 'ֱ' - 41: 0, # 'ֲ' - 33: 0, # 'ִ' - 37: 0, # 'ֵ' - 36: 1, # 'ֶ' - 31: 1, # 'ַ' - 29: 1, # 'ָ' - 35: 0, # 'ֹ' - 62: 0, # 'ֻ' - 28: 0, # 'ּ' - 38: 0, # 'ׁ' - 45: 0, # 'ׂ' - 9: 2, # 'א' - 8: 2, # 'ב' - 20: 1, # 'ג' - 16: 2, # 'ד' - 3: 2, # 'ה' - 2: 1, # 'ו' - 24: 1, # 'ז' - 14: 2, # 'ח' - 22: 1, # 'ט' - 1: 2, # 'י' - 25: 2, # 'ך' - 15: 1, # 'כ' - 4: 2, # 'ל' - 11: 2, # 'ם' - 6: 2, # 'מ' - 23: 2, # 'ן' - 12: 2, # 'נ' - 19: 2, # 'ס' - 13: 1, # 'ע' - 26: 1, # 'ף' - 18: 1, # 'פ' - 27: 2, # 'ץ' - 21: 1, # 'צ' - 17: 1, # 'ק' - 7: 2, # 'ר' - 10: 2, # 'ש' - 5: 2, # 'ת' - 32: 0, # '–' - 52: 0, # '’' - 47: 0, # '“' - 46: 0, # '”' - 58: 0, # '†' - 40: 0, # '…' - }, - 31: { # 'ַ' - 50: 0, # 'a' - 60: 0, # 'c' - 61: 0, # 'd' - 42: 0, # 'e' - 53: 0, # 'i' - 56: 0, # 'l' - 54: 0, # 'n' - 49: 0, # 'o' - 51: 0, # 'r' - 43: 0, # 's' - 44: 0, # 't' - 63: 0, # 'u' - 34: 0, # '\xa0' - 55: 0, # '´' - 48: 0, # '¼' - 39: 0, # '½' - 57: 0, # '¾' - 30: 1, # 'ְ' - 59: 0, # 'ֱ' - 41: 0, # 'ֲ' - 33: 0, # 'ִ' - 37: 0, # 'ֵ' - 36: 1, # 'ֶ' - 31: 0, # 'ַ' - 29: 2, # 'ָ' - 35: 0, # 'ֹ' - 62: 0, # 'ֻ' - 28: 0, # 'ּ' - 38: 0, # 'ׁ' - 45: 0, # 'ׂ' - 9: 2, # 'א' - 8: 2, # 'ב' - 20: 2, # 'ג' - 16: 2, # 'ד' - 3: 2, # 'ה' - 2: 1, # 'ו' - 24: 2, # 'ז' - 14: 2, # 'ח' - 22: 2, # 'ט' - 1: 3, # 'י' - 25: 1, # 'ך' - 15: 2, # 'כ' - 4: 2, # 'ל' - 11: 2, # 'ם' - 6: 2, # 'מ' - 23: 2, # 'ן' - 12: 2, # 'נ' - 19: 2, # 'ס' - 13: 2, # 'ע' - 26: 2, # 'ף' - 18: 2, # 'פ' - 27: 1, # 'ץ' - 21: 2, # 'צ' - 17: 2, # 'ק' - 7: 2, # 'ר' - 10: 2, # 'ש' - 5: 2, # 'ת' - 32: 0, # '–' - 52: 0, # '’' - 47: 0, # '“' - 46: 0, # '”' - 58: 0, # '†' - 40: 0, # '…' - }, - 29: { # 'ָ' - 50: 0, # 'a' - 60: 0, # 'c' - 61: 0, # 'd' - 42: 0, # 'e' - 53: 0, # 'i' - 56: 0, # 'l' - 54: 0, # 'n' - 49: 0, # 'o' - 51: 0, # 'r' - 43: 0, # 's' - 44: 0, # 't' - 63: 0, # 'u' - 34: 0, # '\xa0' - 55: 0, # '´' - 48: 0, # '¼' - 39: 0, # '½' - 57: 0, # '¾' - 30: 0, # 'ְ' - 59: 0, # 'ֱ' - 41: 0, # 'ֲ' - 33: 0, # 'ִ' - 37: 0, # 'ֵ' - 36: 0, # 'ֶ' - 31: 1, # 'ַ' - 29: 2, # 'ָ' - 35: 0, # 'ֹ' - 62: 0, # 'ֻ' - 28: 1, # 'ּ' - 38: 0, # 'ׁ' - 45: 0, # 'ׂ' - 9: 2, # 'א' - 8: 2, # 'ב' - 20: 2, # 'ג' - 16: 2, # 'ד' - 3: 3, # 'ה' - 2: 2, # 'ו' - 24: 2, # 'ז' - 14: 2, # 'ח' - 22: 1, # 'ט' - 1: 2, # 'י' - 25: 2, # 'ך' - 15: 2, # 'כ' - 4: 2, # 'ל' - 11: 2, # 'ם' - 6: 2, # 'מ' - 23: 2, # 'ן' - 12: 2, # 'נ' - 19: 1, # 'ס' - 13: 2, # 'ע' - 26: 1, # 'ף' - 18: 2, # 'פ' - 27: 1, # 'ץ' - 21: 2, # 'צ' - 17: 2, # 'ק' - 7: 2, # 'ר' - 10: 2, # 'ש' - 5: 2, # 'ת' - 32: 0, # '–' - 52: 0, # '’' - 47: 0, # '“' - 46: 0, # '”' - 58: 0, # '†' - 40: 0, # '…' - }, - 35: { # 'ֹ' - 50: 0, # 'a' - 60: 0, # 'c' - 61: 0, # 'd' - 42: 0, # 'e' - 53: 0, # 'i' - 56: 0, # 'l' - 54: 0, # 'n' - 49: 0, # 'o' - 51: 0, # 'r' - 43: 0, # 's' - 44: 0, # 't' - 63: 0, # 'u' - 34: 0, # '\xa0' - 55: 0, # '´' - 48: 0, # '¼' - 39: 0, # '½' - 57: 0, # '¾' - 30: 0, # 'ְ' - 59: 0, # 'ֱ' - 41: 0, # 'ֲ' - 33: 0, # 'ִ' - 37: 0, # 'ֵ' - 36: 0, # 'ֶ' - 31: 0, # 'ַ' - 29: 0, # 'ָ' - 35: 1, # 'ֹ' - 62: 0, # 'ֻ' - 28: 0, # 'ּ' - 38: 0, # 'ׁ' - 45: 0, # 'ׂ' - 9: 2, # 'א' - 8: 2, # 'ב' - 20: 1, # 'ג' - 16: 2, # 'ד' - 3: 2, # 'ה' - 2: 1, # 'ו' - 24: 1, # 'ז' - 14: 1, # 'ח' - 22: 1, # 'ט' - 1: 1, # 'י' - 25: 1, # 'ך' - 15: 2, # 'כ' - 4: 2, # 'ל' - 11: 2, # 'ם' - 6: 2, # 'מ' - 23: 2, # 'ן' - 12: 2, # 'נ' - 19: 2, # 'ס' - 13: 2, # 'ע' - 26: 1, # 'ף' - 18: 2, # 'פ' - 27: 1, # 'ץ' - 21: 2, # 'צ' - 17: 2, # 'ק' - 7: 2, # 'ר' - 10: 2, # 'ש' - 5: 2, # 'ת' - 32: 0, # '–' - 52: 0, # '’' - 47: 0, # '“' - 46: 0, # '”' - 58: 0, # '†' - 40: 0, # '…' - }, - 62: { # 'ֻ' - 50: 0, # 'a' - 60: 0, # 'c' - 61: 0, # 'd' - 42: 0, # 'e' - 53: 0, # 'i' - 56: 0, # 'l' - 54: 0, # 'n' - 49: 0, # 'o' - 51: 0, # 'r' - 43: 0, # 's' - 44: 0, # 't' - 63: 0, # 'u' - 34: 0, # '\xa0' - 55: 0, # '´' - 48: 0, # '¼' - 39: 0, # '½' - 57: 0, # '¾' - 30: 0, # 'ְ' - 59: 0, # 'ֱ' - 41: 0, # 'ֲ' - 33: 0, # 'ִ' - 37: 0, # 'ֵ' - 36: 0, # 'ֶ' - 31: 0, # 'ַ' - 29: 0, # 'ָ' - 35: 0, # 'ֹ' - 62: 0, # 'ֻ' - 28: 0, # 'ּ' - 38: 0, # 'ׁ' - 45: 0, # 'ׂ' - 9: 0, # 'א' - 8: 1, # 'ב' - 20: 1, # 'ג' - 16: 1, # 'ד' - 3: 1, # 'ה' - 2: 1, # 'ו' - 24: 1, # 'ז' - 14: 1, # 'ח' - 22: 0, # 'ט' - 1: 1, # 'י' - 25: 0, # 'ך' - 15: 1, # 'כ' - 4: 2, # 'ל' - 11: 1, # 'ם' - 6: 1, # 'מ' - 23: 1, # 'ן' - 12: 1, # 'נ' - 19: 1, # 'ס' - 13: 1, # 'ע' - 26: 0, # 'ף' - 18: 1, # 'פ' - 27: 0, # 'ץ' - 21: 1, # 'צ' - 17: 1, # 'ק' - 7: 1, # 'ר' - 10: 1, # 'ש' - 5: 1, # 'ת' - 32: 0, # '–' - 52: 0, # '’' - 47: 0, # '“' - 46: 0, # '”' - 58: 0, # '†' - 40: 0, # '…' - }, - 28: { # 'ּ' - 50: 0, # 'a' - 60: 0, # 'c' - 61: 0, # 'd' - 42: 0, # 'e' - 53: 0, # 'i' - 56: 0, # 'l' - 54: 0, # 'n' - 49: 0, # 'o' - 51: 0, # 'r' - 43: 0, # 's' - 44: 0, # 't' - 63: 0, # 'u' - 34: 0, # '\xa0' - 55: 0, # '´' - 48: 0, # '¼' - 39: 0, # '½' - 57: 0, # '¾' - 30: 3, # 'ְ' - 59: 0, # 'ֱ' - 41: 1, # 'ֲ' - 33: 3, # 'ִ' - 37: 2, # 'ֵ' - 36: 2, # 'ֶ' - 31: 3, # 'ַ' - 29: 3, # 'ָ' - 35: 2, # 'ֹ' - 62: 1, # 'ֻ' - 28: 0, # 'ּ' - 38: 2, # 'ׁ' - 45: 1, # 'ׂ' - 9: 2, # 'א' - 8: 2, # 'ב' - 20: 1, # 'ג' - 16: 2, # 'ד' - 3: 1, # 'ה' - 2: 2, # 'ו' - 24: 1, # 'ז' - 14: 1, # 'ח' - 22: 1, # 'ט' - 1: 2, # 'י' - 25: 2, # 'ך' - 15: 2, # 'כ' - 4: 2, # 'ל' - 11: 1, # 'ם' - 6: 2, # 'מ' - 23: 1, # 'ן' - 12: 2, # 'נ' - 19: 1, # 'ס' - 13: 2, # 'ע' - 26: 1, # 'ף' - 18: 1, # 'פ' - 27: 1, # 'ץ' - 21: 1, # 'צ' - 17: 1, # 'ק' - 7: 2, # 'ר' - 10: 2, # 'ש' - 5: 2, # 'ת' - 32: 0, # '–' - 52: 0, # '’' - 47: 0, # '“' - 46: 0, # '”' - 58: 0, # '†' - 40: 0, # '…' - }, - 38: { # 'ׁ' - 50: 0, # 'a' - 60: 0, # 'c' - 61: 0, # 'd' - 42: 0, # 'e' - 53: 0, # 'i' - 56: 0, # 'l' - 54: 0, # 'n' - 49: 0, # 'o' - 51: 0, # 'r' - 43: 0, # 's' - 44: 0, # 't' - 63: 0, # 'u' - 34: 0, # '\xa0' - 55: 0, # '´' - 48: 0, # '¼' - 39: 0, # '½' - 57: 0, # '¾' - 30: 2, # 'ְ' - 59: 0, # 'ֱ' - 41: 0, # 'ֲ' - 33: 2, # 'ִ' - 37: 2, # 'ֵ' - 36: 2, # 'ֶ' - 31: 2, # 'ַ' - 29: 2, # 'ָ' - 35: 1, # 'ֹ' - 62: 1, # 'ֻ' - 28: 0, # 'ּ' - 38: 0, # 'ׁ' - 45: 0, # 'ׂ' - 9: 0, # 'א' - 8: 0, # 'ב' - 20: 0, # 'ג' - 16: 0, # 'ד' - 3: 0, # 'ה' - 2: 2, # 'ו' - 24: 0, # 'ז' - 14: 0, # 'ח' - 22: 0, # 'ט' - 1: 1, # 'י' - 25: 0, # 'ך' - 15: 0, # 'כ' - 4: 0, # 'ל' - 11: 0, # 'ם' - 6: 0, # 'מ' - 23: 0, # 'ן' - 12: 0, # 'נ' - 19: 0, # 'ס' - 13: 1, # 'ע' - 26: 0, # 'ף' - 18: 0, # 'פ' - 27: 0, # 'ץ' - 21: 0, # 'צ' - 17: 0, # 'ק' - 7: 0, # 'ר' - 10: 0, # 'ש' - 5: 0, # 'ת' - 32: 0, # '–' - 52: 0, # '’' - 47: 0, # '“' - 46: 0, # '”' - 58: 0, # '†' - 40: 0, # '…' - }, - 45: { # 'ׂ' - 50: 0, # 'a' - 60: 0, # 'c' - 61: 0, # 'd' - 42: 0, # 'e' - 53: 0, # 'i' - 56: 0, # 'l' - 54: 0, # 'n' - 49: 0, # 'o' - 51: 0, # 'r' - 43: 0, # 's' - 44: 0, # 't' - 63: 0, # 'u' - 34: 0, # '\xa0' - 55: 0, # '´' - 48: 0, # '¼' - 39: 0, # '½' - 57: 0, # '¾' - 30: 2, # 'ְ' - 59: 0, # 'ֱ' - 41: 0, # 'ֲ' - 33: 2, # 'ִ' - 37: 1, # 'ֵ' - 36: 2, # 'ֶ' - 31: 1, # 'ַ' - 29: 2, # 'ָ' - 35: 1, # 'ֹ' - 62: 0, # 'ֻ' - 28: 0, # 'ּ' - 38: 0, # 'ׁ' - 45: 0, # 'ׂ' - 9: 1, # 'א' - 8: 0, # 'ב' - 20: 1, # 'ג' - 16: 0, # 'ד' - 3: 1, # 'ה' - 2: 2, # 'ו' - 24: 0, # 'ז' - 14: 1, # 'ח' - 22: 0, # 'ט' - 1: 1, # 'י' - 25: 0, # 'ך' - 15: 0, # 'כ' - 4: 0, # 'ל' - 11: 1, # 'ם' - 6: 1, # 'מ' - 23: 0, # 'ן' - 12: 1, # 'נ' - 19: 0, # 'ס' - 13: 1, # 'ע' - 26: 0, # 'ף' - 18: 1, # 'פ' - 27: 0, # 'ץ' - 21: 0, # 'צ' - 17: 0, # 'ק' - 7: 1, # 'ר' - 10: 0, # 'ש' - 5: 1, # 'ת' - 32: 0, # '–' - 52: 0, # '’' - 47: 0, # '“' - 46: 0, # '”' - 58: 0, # '†' - 40: 0, # '…' - }, - 9: { # 'א' - 50: 0, # 'a' - 60: 0, # 'c' - 61: 0, # 'd' - 42: 0, # 'e' - 53: 0, # 'i' - 56: 0, # 'l' - 54: 0, # 'n' - 49: 0, # 'o' - 51: 0, # 'r' - 43: 0, # 's' - 44: 0, # 't' - 63: 0, # 'u' - 34: 1, # '\xa0' - 55: 1, # '´' - 48: 1, # '¼' - 39: 0, # '½' - 57: 0, # '¾' - 30: 0, # 'ְ' - 59: 2, # 'ֱ' - 41: 2, # 'ֲ' - 33: 2, # 'ִ' - 37: 2, # 'ֵ' - 36: 2, # 'ֶ' - 31: 2, # 'ַ' - 29: 2, # 'ָ' - 35: 2, # 'ֹ' - 62: 1, # 'ֻ' - 28: 0, # 'ּ' - 38: 0, # 'ׁ' - 45: 0, # 'ׂ' - 9: 2, # 'א' - 8: 3, # 'ב' - 20: 3, # 'ג' - 16: 3, # 'ד' - 3: 3, # 'ה' - 2: 3, # 'ו' - 24: 3, # 'ז' - 14: 3, # 'ח' - 22: 3, # 'ט' - 1: 3, # 'י' - 25: 3, # 'ך' - 15: 3, # 'כ' - 4: 3, # 'ל' - 11: 3, # 'ם' - 6: 3, # 'מ' - 23: 3, # 'ן' - 12: 3, # 'נ' - 19: 3, # 'ס' - 13: 2, # 'ע' - 26: 3, # 'ף' - 18: 3, # 'פ' - 27: 1, # 'ץ' - 21: 3, # 'צ' - 17: 3, # 'ק' - 7: 3, # 'ר' - 10: 3, # 'ש' - 5: 3, # 'ת' - 32: 0, # '–' - 52: 0, # '’' - 47: 0, # '“' - 46: 1, # '”' - 58: 0, # '†' - 40: 1, # '…' - }, - 8: { # 'ב' - 50: 0, # 'a' - 60: 0, # 'c' - 61: 1, # 'd' - 42: 0, # 'e' - 53: 0, # 'i' - 56: 0, # 'l' - 54: 0, # 'n' - 49: 0, # 'o' - 51: 0, # 'r' - 43: 0, # 's' - 44: 0, # 't' - 63: 0, # 'u' - 34: 1, # '\xa0' - 55: 1, # '´' - 48: 0, # '¼' - 39: 0, # '½' - 57: 0, # '¾' - 30: 2, # 'ְ' - 59: 0, # 'ֱ' - 41: 0, # 'ֲ' - 33: 2, # 'ִ' - 37: 2, # 'ֵ' - 36: 2, # 'ֶ' - 31: 2, # 'ַ' - 29: 2, # 'ָ' - 35: 2, # 'ֹ' - 62: 1, # 'ֻ' - 28: 3, # 'ּ' - 38: 0, # 'ׁ' - 45: 0, # 'ׂ' - 9: 3, # 'א' - 8: 3, # 'ב' - 20: 3, # 'ג' - 16: 3, # 'ד' - 3: 3, # 'ה' - 2: 3, # 'ו' - 24: 3, # 'ז' - 14: 3, # 'ח' - 22: 3, # 'ט' - 1: 3, # 'י' - 25: 2, # 'ך' - 15: 3, # 'כ' - 4: 3, # 'ל' - 11: 2, # 'ם' - 6: 3, # 'מ' - 23: 3, # 'ן' - 12: 3, # 'נ' - 19: 3, # 'ס' - 13: 3, # 'ע' - 26: 1, # 'ף' - 18: 3, # 'פ' - 27: 2, # 'ץ' - 21: 3, # 'צ' - 17: 3, # 'ק' - 7: 3, # 'ר' - 10: 3, # 'ש' - 5: 3, # 'ת' - 32: 1, # '–' - 52: 0, # '’' - 47: 0, # '“' - 46: 1, # '”' - 58: 0, # '†' - 40: 1, # '…' - }, - 20: { # 'ג' - 50: 0, # 'a' - 60: 0, # 'c' - 61: 0, # 'd' - 42: 0, # 'e' - 53: 0, # 'i' - 56: 0, # 'l' - 54: 0, # 'n' - 49: 0, # 'o' - 51: 0, # 'r' - 43: 0, # 's' - 44: 0, # 't' - 63: 0, # 'u' - 34: 1, # '\xa0' - 55: 2, # '´' - 48: 0, # '¼' - 39: 0, # '½' - 57: 0, # '¾' - 30: 2, # 'ְ' - 59: 0, # 'ֱ' - 41: 0, # 'ֲ' - 33: 1, # 'ִ' - 37: 1, # 'ֵ' - 36: 1, # 'ֶ' - 31: 2, # 'ַ' - 29: 2, # 'ָ' - 35: 1, # 'ֹ' - 62: 0, # 'ֻ' - 28: 2, # 'ּ' - 38: 0, # 'ׁ' - 45: 0, # 'ׂ' - 9: 2, # 'א' - 8: 3, # 'ב' - 20: 2, # 'ג' - 16: 3, # 'ד' - 3: 3, # 'ה' - 2: 3, # 'ו' - 24: 3, # 'ז' - 14: 2, # 'ח' - 22: 2, # 'ט' - 1: 3, # 'י' - 25: 1, # 'ך' - 15: 1, # 'כ' - 4: 3, # 'ל' - 11: 3, # 'ם' - 6: 3, # 'מ' - 23: 3, # 'ן' - 12: 3, # 'נ' - 19: 2, # 'ס' - 13: 3, # 'ע' - 26: 2, # 'ף' - 18: 2, # 'פ' - 27: 1, # 'ץ' - 21: 1, # 'צ' - 17: 1, # 'ק' - 7: 3, # 'ר' - 10: 3, # 'ש' - 5: 3, # 'ת' - 32: 0, # '–' - 52: 1, # '’' - 47: 0, # '“' - 46: 1, # '”' - 58: 0, # '†' - 40: 0, # '…' - }, - 16: { # 'ד' - 50: 0, # 'a' - 60: 0, # 'c' - 61: 0, # 'd' - 42: 0, # 'e' - 53: 0, # 'i' - 56: 0, # 'l' - 54: 0, # 'n' - 49: 0, # 'o' - 51: 0, # 'r' - 43: 0, # 's' - 44: 0, # 't' - 63: 0, # 'u' - 34: 0, # '\xa0' - 55: 0, # '´' - 48: 0, # '¼' - 39: 0, # '½' - 57: 0, # '¾' - 30: 2, # 'ְ' - 59: 0, # 'ֱ' - 41: 0, # 'ֲ' - 33: 2, # 'ִ' - 37: 2, # 'ֵ' - 36: 2, # 'ֶ' - 31: 2, # 'ַ' - 29: 2, # 'ָ' - 35: 2, # 'ֹ' - 62: 1, # 'ֻ' - 28: 2, # 'ּ' - 38: 0, # 'ׁ' - 45: 0, # 'ׂ' - 9: 3, # 'א' - 8: 3, # 'ב' - 20: 3, # 'ג' - 16: 3, # 'ד' - 3: 3, # 'ה' - 2: 3, # 'ו' - 24: 1, # 'ז' - 14: 2, # 'ח' - 22: 2, # 'ט' - 1: 3, # 'י' - 25: 2, # 'ך' - 15: 2, # 'כ' - 4: 3, # 'ל' - 11: 3, # 'ם' - 6: 3, # 'מ' - 23: 2, # 'ן' - 12: 3, # 'נ' - 19: 2, # 'ס' - 13: 3, # 'ע' - 26: 2, # 'ף' - 18: 3, # 'פ' - 27: 0, # 'ץ' - 21: 2, # 'צ' - 17: 3, # 'ק' - 7: 3, # 'ר' - 10: 3, # 'ש' - 5: 3, # 'ת' - 32: 0, # '–' - 52: 0, # '’' - 47: 0, # '“' - 46: 1, # '”' - 58: 0, # '†' - 40: 1, # '…' - }, - 3: { # 'ה' - 50: 0, # 'a' - 60: 0, # 'c' - 61: 1, # 'd' - 42: 0, # 'e' - 53: 0, # 'i' - 56: 0, # 'l' - 54: 0, # 'n' - 49: 0, # 'o' - 51: 0, # 'r' - 43: 0, # 's' - 44: 0, # 't' - 63: 0, # 'u' - 34: 1, # '\xa0' - 55: 0, # '´' - 48: 1, # '¼' - 39: 0, # '½' - 57: 0, # '¾' - 30: 1, # 'ְ' - 59: 1, # 'ֱ' - 41: 2, # 'ֲ' - 33: 2, # 'ִ' - 37: 2, # 'ֵ' - 36: 2, # 'ֶ' - 31: 3, # 'ַ' - 29: 2, # 'ָ' - 35: 1, # 'ֹ' - 62: 1, # 'ֻ' - 28: 2, # 'ּ' - 38: 0, # 'ׁ' - 45: 0, # 'ׂ' - 9: 3, # 'א' - 8: 3, # 'ב' - 20: 3, # 'ג' - 16: 3, # 'ד' - 3: 3, # 'ה' - 2: 3, # 'ו' - 24: 3, # 'ז' - 14: 3, # 'ח' - 22: 3, # 'ט' - 1: 3, # 'י' - 25: 1, # 'ך' - 15: 3, # 'כ' - 4: 3, # 'ל' - 11: 3, # 'ם' - 6: 3, # 'מ' - 23: 3, # 'ן' - 12: 3, # 'נ' - 19: 3, # 'ס' - 13: 3, # 'ע' - 26: 0, # 'ף' - 18: 3, # 'פ' - 27: 1, # 'ץ' - 21: 3, # 'צ' - 17: 3, # 'ק' - 7: 3, # 'ר' - 10: 3, # 'ש' - 5: 3, # 'ת' - 32: 1, # '–' - 52: 1, # '’' - 47: 0, # '“' - 46: 1, # '”' - 58: 0, # '†' - 40: 2, # '…' - }, - 2: { # 'ו' - 50: 0, # 'a' - 60: 0, # 'c' - 61: 0, # 'd' - 42: 0, # 'e' - 53: 0, # 'i' - 56: 0, # 'l' - 54: 0, # 'n' - 49: 0, # 'o' - 51: 0, # 'r' - 43: 0, # 's' - 44: 1, # 't' - 63: 0, # 'u' - 34: 1, # '\xa0' - 55: 1, # '´' - 48: 1, # '¼' - 39: 0, # '½' - 57: 0, # '¾' - 30: 2, # 'ְ' - 59: 0, # 'ֱ' - 41: 0, # 'ֲ' - 33: 2, # 'ִ' - 37: 1, # 'ֵ' - 36: 1, # 'ֶ' - 31: 2, # 'ַ' - 29: 2, # 'ָ' - 35: 3, # 'ֹ' - 62: 0, # 'ֻ' - 28: 3, # 'ּ' - 38: 0, # 'ׁ' - 45: 0, # 'ׂ' - 9: 3, # 'א' - 8: 3, # 'ב' - 20: 3, # 'ג' - 16: 3, # 'ד' - 3: 3, # 'ה' - 2: 3, # 'ו' - 24: 3, # 'ז' - 14: 3, # 'ח' - 22: 3, # 'ט' - 1: 3, # 'י' - 25: 3, # 'ך' - 15: 3, # 'כ' - 4: 3, # 'ל' - 11: 3, # 'ם' - 6: 3, # 'מ' - 23: 3, # 'ן' - 12: 3, # 'נ' - 19: 3, # 'ס' - 13: 3, # 'ע' - 26: 3, # 'ף' - 18: 3, # 'פ' - 27: 3, # 'ץ' - 21: 3, # 'צ' - 17: 3, # 'ק' - 7: 3, # 'ר' - 10: 3, # 'ש' - 5: 3, # 'ת' - 32: 1, # '–' - 52: 0, # '’' - 47: 0, # '“' - 46: 1, # '”' - 58: 0, # '†' - 40: 2, # '…' - }, - 24: { # 'ז' - 50: 0, # 'a' - 60: 0, # 'c' - 61: 0, # 'd' - 42: 0, # 'e' - 53: 0, # 'i' - 56: 0, # 'l' - 54: 0, # 'n' - 49: 0, # 'o' - 51: 0, # 'r' - 43: 0, # 's' - 44: 0, # 't' - 63: 0, # 'u' - 34: 0, # '\xa0' - 55: 1, # '´' - 48: 0, # '¼' - 39: 0, # '½' - 57: 0, # '¾' - 30: 2, # 'ְ' - 59: 0, # 'ֱ' - 41: 1, # 'ֲ' - 33: 1, # 'ִ' - 37: 2, # 'ֵ' - 36: 2, # 'ֶ' - 31: 2, # 'ַ' - 29: 2, # 'ָ' - 35: 1, # 'ֹ' - 62: 1, # 'ֻ' - 28: 2, # 'ּ' - 38: 0, # 'ׁ' - 45: 0, # 'ׂ' - 9: 3, # 'א' - 8: 2, # 'ב' - 20: 2, # 'ג' - 16: 2, # 'ד' - 3: 3, # 'ה' - 2: 3, # 'ו' - 24: 2, # 'ז' - 14: 2, # 'ח' - 22: 1, # 'ט' - 1: 3, # 'י' - 25: 1, # 'ך' - 15: 3, # 'כ' - 4: 3, # 'ל' - 11: 2, # 'ם' - 6: 3, # 'מ' - 23: 2, # 'ן' - 12: 2, # 'נ' - 19: 1, # 'ס' - 13: 2, # 'ע' - 26: 1, # 'ף' - 18: 1, # 'פ' - 27: 0, # 'ץ' - 21: 2, # 'צ' - 17: 3, # 'ק' - 7: 3, # 'ר' - 10: 1, # 'ש' - 5: 2, # 'ת' - 32: 0, # '–' - 52: 0, # '’' - 47: 0, # '“' - 46: 0, # '”' - 58: 0, # '†' - 40: 1, # '…' - }, - 14: { # 'ח' - 50: 0, # 'a' - 60: 0, # 'c' - 61: 0, # 'd' - 42: 0, # 'e' - 53: 0, # 'i' - 56: 0, # 'l' - 54: 0, # 'n' - 49: 0, # 'o' - 51: 0, # 'r' - 43: 0, # 's' - 44: 0, # 't' - 63: 0, # 'u' - 34: 1, # '\xa0' - 55: 1, # '´' - 48: 0, # '¼' - 39: 0, # '½' - 57: 0, # '¾' - 30: 2, # 'ְ' - 59: 1, # 'ֱ' - 41: 2, # 'ֲ' - 33: 2, # 'ִ' - 37: 2, # 'ֵ' - 36: 2, # 'ֶ' - 31: 2, # 'ַ' - 29: 2, # 'ָ' - 35: 2, # 'ֹ' - 62: 1, # 'ֻ' - 28: 0, # 'ּ' - 38: 0, # 'ׁ' - 45: 0, # 'ׂ' - 9: 2, # 'א' - 8: 3, # 'ב' - 20: 2, # 'ג' - 16: 3, # 'ד' - 3: 3, # 'ה' - 2: 3, # 'ו' - 24: 3, # 'ז' - 14: 2, # 'ח' - 22: 2, # 'ט' - 1: 3, # 'י' - 25: 1, # 'ך' - 15: 2, # 'כ' - 4: 3, # 'ל' - 11: 3, # 'ם' - 6: 3, # 'מ' - 23: 2, # 'ן' - 12: 3, # 'נ' - 19: 3, # 'ס' - 13: 1, # 'ע' - 26: 2, # 'ף' - 18: 2, # 'פ' - 27: 2, # 'ץ' - 21: 3, # 'צ' - 17: 3, # 'ק' - 7: 3, # 'ר' - 10: 3, # 'ש' - 5: 3, # 'ת' - 32: 0, # '–' - 52: 1, # '’' - 47: 0, # '“' - 46: 1, # '”' - 58: 0, # '†' - 40: 1, # '…' - }, - 22: { # 'ט' - 50: 0, # 'a' - 60: 0, # 'c' - 61: 0, # 'd' - 42: 0, # 'e' - 53: 0, # 'i' - 56: 0, # 'l' - 54: 0, # 'n' - 49: 0, # 'o' - 51: 0, # 'r' - 43: 0, # 's' - 44: 0, # 't' - 63: 0, # 'u' - 34: 1, # '\xa0' - 55: 0, # '´' - 48: 0, # '¼' - 39: 0, # '½' - 57: 0, # '¾' - 30: 2, # 'ְ' - 59: 0, # 'ֱ' - 41: 0, # 'ֲ' - 33: 2, # 'ִ' - 37: 1, # 'ֵ' - 36: 1, # 'ֶ' - 31: 2, # 'ַ' - 29: 1, # 'ָ' - 35: 1, # 'ֹ' - 62: 1, # 'ֻ' - 28: 1, # 'ּ' - 38: 0, # 'ׁ' - 45: 0, # 'ׂ' - 9: 3, # 'א' - 8: 3, # 'ב' - 20: 3, # 'ג' - 16: 1, # 'ד' - 3: 3, # 'ה' - 2: 3, # 'ו' - 24: 2, # 'ז' - 14: 3, # 'ח' - 22: 2, # 'ט' - 1: 3, # 'י' - 25: 1, # 'ך' - 15: 2, # 'כ' - 4: 3, # 'ל' - 11: 2, # 'ם' - 6: 2, # 'מ' - 23: 2, # 'ן' - 12: 3, # 'נ' - 19: 2, # 'ס' - 13: 3, # 'ע' - 26: 2, # 'ף' - 18: 3, # 'פ' - 27: 1, # 'ץ' - 21: 2, # 'צ' - 17: 2, # 'ק' - 7: 3, # 'ר' - 10: 2, # 'ש' - 5: 3, # 'ת' - 32: 0, # '–' - 52: 0, # '’' - 47: 0, # '“' - 46: 0, # '”' - 58: 0, # '†' - 40: 1, # '…' - }, - 1: { # 'י' - 50: 0, # 'a' - 60: 0, # 'c' - 61: 0, # 'd' - 42: 0, # 'e' - 53: 0, # 'i' - 56: 0, # 'l' - 54: 0, # 'n' - 49: 0, # 'o' - 51: 0, # 'r' - 43: 0, # 's' - 44: 0, # 't' - 63: 0, # 'u' - 34: 1, # '\xa0' - 55: 1, # '´' - 48: 1, # '¼' - 39: 0, # '½' - 57: 0, # '¾' - 30: 2, # 'ְ' - 59: 0, # 'ֱ' - 41: 0, # 'ֲ' - 33: 2, # 'ִ' - 37: 2, # 'ֵ' - 36: 1, # 'ֶ' - 31: 2, # 'ַ' - 29: 2, # 'ָ' - 35: 2, # 'ֹ' - 62: 1, # 'ֻ' - 28: 2, # 'ּ' - 38: 0, # 'ׁ' - 45: 0, # 'ׂ' - 9: 3, # 'א' - 8: 3, # 'ב' - 20: 3, # 'ג' - 16: 3, # 'ד' - 3: 3, # 'ה' - 2: 3, # 'ו' - 24: 3, # 'ז' - 14: 3, # 'ח' - 22: 3, # 'ט' - 1: 3, # 'י' - 25: 3, # 'ך' - 15: 3, # 'כ' - 4: 3, # 'ל' - 11: 3, # 'ם' - 6: 3, # 'מ' - 23: 3, # 'ן' - 12: 3, # 'נ' - 19: 3, # 'ס' - 13: 3, # 'ע' - 26: 3, # 'ף' - 18: 3, # 'פ' - 27: 3, # 'ץ' - 21: 3, # 'צ' - 17: 3, # 'ק' - 7: 3, # 'ר' - 10: 3, # 'ש' - 5: 3, # 'ת' - 32: 1, # '–' - 52: 0, # '’' - 47: 0, # '“' - 46: 1, # '”' - 58: 0, # '†' - 40: 2, # '…' - }, - 25: { # 'ך' - 50: 0, # 'a' - 60: 0, # 'c' - 61: 0, # 'd' - 42: 0, # 'e' - 53: 0, # 'i' - 56: 0, # 'l' - 54: 0, # 'n' - 49: 0, # 'o' - 51: 0, # 'r' - 43: 0, # 's' - 44: 0, # 't' - 63: 0, # 'u' - 34: 0, # '\xa0' - 55: 0, # '´' - 48: 0, # '¼' - 39: 0, # '½' - 57: 0, # '¾' - 30: 2, # 'ְ' - 59: 0, # 'ֱ' - 41: 0, # 'ֲ' - 33: 0, # 'ִ' - 37: 0, # 'ֵ' - 36: 0, # 'ֶ' - 31: 0, # 'ַ' - 29: 2, # 'ָ' - 35: 0, # 'ֹ' - 62: 0, # 'ֻ' - 28: 1, # 'ּ' - 38: 0, # 'ׁ' - 45: 0, # 'ׂ' - 9: 1, # 'א' - 8: 0, # 'ב' - 20: 0, # 'ג' - 16: 0, # 'ד' - 3: 1, # 'ה' - 2: 0, # 'ו' - 24: 0, # 'ז' - 14: 1, # 'ח' - 22: 0, # 'ט' - 1: 0, # 'י' - 25: 0, # 'ך' - 15: 0, # 'כ' - 4: 1, # 'ל' - 11: 0, # 'ם' - 6: 1, # 'מ' - 23: 0, # 'ן' - 12: 0, # 'נ' - 19: 0, # 'ס' - 13: 0, # 'ע' - 26: 0, # 'ף' - 18: 0, # 'פ' - 27: 0, # 'ץ' - 21: 0, # 'צ' - 17: 0, # 'ק' - 7: 0, # 'ר' - 10: 1, # 'ש' - 5: 0, # 'ת' - 32: 0, # '–' - 52: 0, # '’' - 47: 0, # '“' - 46: 0, # '”' - 58: 0, # '†' - 40: 1, # '…' - }, - 15: { # 'כ' - 50: 0, # 'a' - 60: 0, # 'c' - 61: 0, # 'd' - 42: 0, # 'e' - 53: 0, # 'i' - 56: 0, # 'l' - 54: 0, # 'n' - 49: 0, # 'o' - 51: 0, # 'r' - 43: 0, # 's' - 44: 0, # 't' - 63: 0, # 'u' - 34: 0, # '\xa0' - 55: 0, # '´' - 48: 0, # '¼' - 39: 0, # '½' - 57: 0, # '¾' - 30: 2, # 'ְ' - 59: 0, # 'ֱ' - 41: 0, # 'ֲ' - 33: 2, # 'ִ' - 37: 2, # 'ֵ' - 36: 2, # 'ֶ' - 31: 2, # 'ַ' - 29: 2, # 'ָ' - 35: 1, # 'ֹ' - 62: 1, # 'ֻ' - 28: 3, # 'ּ' - 38: 0, # 'ׁ' - 45: 0, # 'ׂ' - 9: 3, # 'א' - 8: 3, # 'ב' - 20: 2, # 'ג' - 16: 3, # 'ד' - 3: 3, # 'ה' - 2: 3, # 'ו' - 24: 3, # 'ז' - 14: 3, # 'ח' - 22: 2, # 'ט' - 1: 3, # 'י' - 25: 3, # 'ך' - 15: 3, # 'כ' - 4: 3, # 'ל' - 11: 3, # 'ם' - 6: 3, # 'מ' - 23: 3, # 'ן' - 12: 3, # 'נ' - 19: 3, # 'ס' - 13: 2, # 'ע' - 26: 3, # 'ף' - 18: 3, # 'פ' - 27: 1, # 'ץ' - 21: 2, # 'צ' - 17: 2, # 'ק' - 7: 3, # 'ר' - 10: 3, # 'ש' - 5: 3, # 'ת' - 32: 0, # '–' - 52: 0, # '’' - 47: 0, # '“' - 46: 0, # '”' - 58: 0, # '†' - 40: 0, # '…' - }, - 4: { # 'ל' - 50: 0, # 'a' - 60: 0, # 'c' - 61: 0, # 'd' - 42: 0, # 'e' - 53: 0, # 'i' - 56: 0, # 'l' - 54: 0, # 'n' - 49: 0, # 'o' - 51: 0, # 'r' - 43: 0, # 's' - 44: 0, # 't' - 63: 0, # 'u' - 34: 1, # '\xa0' - 55: 1, # '´' - 48: 0, # '¼' - 39: 0, # '½' - 57: 0, # '¾' - 30: 3, # 'ְ' - 59: 0, # 'ֱ' - 41: 0, # 'ֲ' - 33: 2, # 'ִ' - 37: 2, # 'ֵ' - 36: 2, # 'ֶ' - 31: 2, # 'ַ' - 29: 2, # 'ָ' - 35: 2, # 'ֹ' - 62: 1, # 'ֻ' - 28: 2, # 'ּ' - 38: 0, # 'ׁ' - 45: 0, # 'ׂ' - 9: 3, # 'א' - 8: 3, # 'ב' - 20: 3, # 'ג' - 16: 3, # 'ד' - 3: 3, # 'ה' - 2: 3, # 'ו' - 24: 3, # 'ז' - 14: 3, # 'ח' - 22: 3, # 'ט' - 1: 3, # 'י' - 25: 3, # 'ך' - 15: 3, # 'כ' - 4: 3, # 'ל' - 11: 3, # 'ם' - 6: 3, # 'מ' - 23: 2, # 'ן' - 12: 3, # 'נ' - 19: 3, # 'ס' - 13: 3, # 'ע' - 26: 2, # 'ף' - 18: 3, # 'פ' - 27: 2, # 'ץ' - 21: 3, # 'צ' - 17: 3, # 'ק' - 7: 3, # 'ר' - 10: 3, # 'ש' - 5: 3, # 'ת' - 32: 1, # '–' - 52: 0, # '’' - 47: 0, # '“' - 46: 1, # '”' - 58: 0, # '†' - 40: 1, # '…' - }, - 11: { # 'ם' - 50: 0, # 'a' - 60: 0, # 'c' - 61: 0, # 'd' - 42: 0, # 'e' - 53: 0, # 'i' - 56: 0, # 'l' - 54: 0, # 'n' - 49: 0, # 'o' - 51: 0, # 'r' - 43: 0, # 's' - 44: 0, # 't' - 63: 0, # 'u' - 34: 1, # '\xa0' - 55: 0, # '´' - 48: 0, # '¼' - 39: 0, # '½' - 57: 0, # '¾' - 30: 0, # 'ְ' - 59: 0, # 'ֱ' - 41: 0, # 'ֲ' - 33: 0, # 'ִ' - 37: 0, # 'ֵ' - 36: 0, # 'ֶ' - 31: 0, # 'ַ' - 29: 0, # 'ָ' - 35: 0, # 'ֹ' - 62: 0, # 'ֻ' - 28: 0, # 'ּ' - 38: 0, # 'ׁ' - 45: 0, # 'ׂ' - 9: 1, # 'א' - 8: 1, # 'ב' - 20: 1, # 'ג' - 16: 0, # 'ד' - 3: 1, # 'ה' - 2: 1, # 'ו' - 24: 1, # 'ז' - 14: 1, # 'ח' - 22: 0, # 'ט' - 1: 1, # 'י' - 25: 0, # 'ך' - 15: 1, # 'כ' - 4: 1, # 'ל' - 11: 1, # 'ם' - 6: 1, # 'מ' - 23: 0, # 'ן' - 12: 1, # 'נ' - 19: 0, # 'ס' - 13: 1, # 'ע' - 26: 0, # 'ף' - 18: 1, # 'פ' - 27: 1, # 'ץ' - 21: 1, # 'צ' - 17: 1, # 'ק' - 7: 1, # 'ר' - 10: 1, # 'ש' - 5: 1, # 'ת' - 32: 0, # '–' - 52: 0, # '’' - 47: 0, # '“' - 46: 1, # '”' - 58: 0, # '†' - 40: 2, # '…' - }, - 6: { # 'מ' - 50: 0, # 'a' - 60: 0, # 'c' - 61: 0, # 'd' - 42: 0, # 'e' - 53: 0, # 'i' - 56: 0, # 'l' - 54: 0, # 'n' - 49: 0, # 'o' - 51: 0, # 'r' - 43: 0, # 's' - 44: 0, # 't' - 63: 0, # 'u' - 34: 0, # '\xa0' - 55: 1, # '´' - 48: 0, # '¼' - 39: 0, # '½' - 57: 0, # '¾' - 30: 2, # 'ְ' - 59: 0, # 'ֱ' - 41: 0, # 'ֲ' - 33: 2, # 'ִ' - 37: 2, # 'ֵ' - 36: 2, # 'ֶ' - 31: 2, # 'ַ' - 29: 2, # 'ָ' - 35: 2, # 'ֹ' - 62: 1, # 'ֻ' - 28: 2, # 'ּ' - 38: 0, # 'ׁ' - 45: 0, # 'ׂ' - 9: 3, # 'א' - 8: 3, # 'ב' - 20: 3, # 'ג' - 16: 3, # 'ד' - 3: 3, # 'ה' - 2: 3, # 'ו' - 24: 3, # 'ז' - 14: 3, # 'ח' - 22: 3, # 'ט' - 1: 3, # 'י' - 25: 2, # 'ך' - 15: 3, # 'כ' - 4: 3, # 'ל' - 11: 3, # 'ם' - 6: 3, # 'מ' - 23: 3, # 'ן' - 12: 3, # 'נ' - 19: 3, # 'ס' - 13: 3, # 'ע' - 26: 0, # 'ף' - 18: 3, # 'פ' - 27: 2, # 'ץ' - 21: 3, # 'צ' - 17: 3, # 'ק' - 7: 3, # 'ר' - 10: 3, # 'ש' - 5: 3, # 'ת' - 32: 0, # '–' - 52: 0, # '’' - 47: 0, # '“' - 46: 0, # '”' - 58: 0, # '†' - 40: 1, # '…' - }, - 23: { # 'ן' - 50: 0, # 'a' - 60: 0, # 'c' - 61: 0, # 'd' - 42: 0, # 'e' - 53: 0, # 'i' - 56: 0, # 'l' - 54: 0, # 'n' - 49: 0, # 'o' - 51: 0, # 'r' - 43: 0, # 's' - 44: 0, # 't' - 63: 0, # 'u' - 34: 1, # '\xa0' - 55: 0, # '´' - 48: 1, # '¼' - 39: 0, # '½' - 57: 0, # '¾' - 30: 0, # 'ְ' - 59: 0, # 'ֱ' - 41: 0, # 'ֲ' - 33: 0, # 'ִ' - 37: 0, # 'ֵ' - 36: 0, # 'ֶ' - 31: 0, # 'ַ' - 29: 0, # 'ָ' - 35: 0, # 'ֹ' - 62: 0, # 'ֻ' - 28: 0, # 'ּ' - 38: 0, # 'ׁ' - 45: 0, # 'ׂ' - 9: 1, # 'א' - 8: 1, # 'ב' - 20: 1, # 'ג' - 16: 1, # 'ד' - 3: 1, # 'ה' - 2: 1, # 'ו' - 24: 0, # 'ז' - 14: 1, # 'ח' - 22: 1, # 'ט' - 1: 1, # 'י' - 25: 0, # 'ך' - 15: 1, # 'כ' - 4: 1, # 'ל' - 11: 1, # 'ם' - 6: 1, # 'מ' - 23: 0, # 'ן' - 12: 1, # 'נ' - 19: 1, # 'ס' - 13: 1, # 'ע' - 26: 1, # 'ף' - 18: 1, # 'פ' - 27: 0, # 'ץ' - 21: 0, # 'צ' - 17: 1, # 'ק' - 7: 1, # 'ר' - 10: 1, # 'ש' - 5: 1, # 'ת' - 32: 1, # '–' - 52: 0, # '’' - 47: 0, # '“' - 46: 1, # '”' - 58: 0, # '†' - 40: 2, # '…' - }, - 12: { # 'נ' - 50: 0, # 'a' - 60: 0, # 'c' - 61: 0, # 'd' - 42: 0, # 'e' - 53: 0, # 'i' - 56: 0, # 'l' - 54: 0, # 'n' - 49: 0, # 'o' - 51: 0, # 'r' - 43: 0, # 's' - 44: 0, # 't' - 63: 0, # 'u' - 34: 0, # '\xa0' - 55: 0, # '´' - 48: 0, # '¼' - 39: 0, # '½' - 57: 0, # '¾' - 30: 2, # 'ְ' - 59: 0, # 'ֱ' - 41: 0, # 'ֲ' - 33: 2, # 'ִ' - 37: 2, # 'ֵ' - 36: 2, # 'ֶ' - 31: 2, # 'ַ' - 29: 2, # 'ָ' - 35: 1, # 'ֹ' - 62: 1, # 'ֻ' - 28: 2, # 'ּ' - 38: 0, # 'ׁ' - 45: 0, # 'ׂ' - 9: 3, # 'א' - 8: 3, # 'ב' - 20: 3, # 'ג' - 16: 3, # 'ד' - 3: 3, # 'ה' - 2: 3, # 'ו' - 24: 3, # 'ז' - 14: 3, # 'ח' - 22: 3, # 'ט' - 1: 3, # 'י' - 25: 2, # 'ך' - 15: 3, # 'כ' - 4: 3, # 'ל' - 11: 3, # 'ם' - 6: 3, # 'מ' - 23: 3, # 'ן' - 12: 3, # 'נ' - 19: 3, # 'ס' - 13: 3, # 'ע' - 26: 2, # 'ף' - 18: 3, # 'פ' - 27: 2, # 'ץ' - 21: 3, # 'צ' - 17: 3, # 'ק' - 7: 3, # 'ר' - 10: 3, # 'ש' - 5: 3, # 'ת' - 32: 0, # '–' - 52: 0, # '’' - 47: 0, # '“' - 46: 0, # '”' - 58: 0, # '†' - 40: 0, # '…' - }, - 19: { # 'ס' - 50: 0, # 'a' - 60: 0, # 'c' - 61: 0, # 'd' - 42: 0, # 'e' - 53: 0, # 'i' - 56: 0, # 'l' - 54: 0, # 'n' - 49: 0, # 'o' - 51: 0, # 'r' - 43: 0, # 's' - 44: 0, # 't' - 63: 0, # 'u' - 34: 1, # '\xa0' - 55: 1, # '´' - 48: 0, # '¼' - 39: 0, # '½' - 57: 0, # '¾' - 30: 2, # 'ְ' - 59: 0, # 'ֱ' - 41: 0, # 'ֲ' - 33: 2, # 'ִ' - 37: 1, # 'ֵ' - 36: 2, # 'ֶ' - 31: 2, # 'ַ' - 29: 1, # 'ָ' - 35: 1, # 'ֹ' - 62: 2, # 'ֻ' - 28: 2, # 'ּ' - 38: 0, # 'ׁ' - 45: 0, # 'ׂ' - 9: 2, # 'א' - 8: 3, # 'ב' - 20: 3, # 'ג' - 16: 3, # 'ד' - 3: 3, # 'ה' - 2: 3, # 'ו' - 24: 1, # 'ז' - 14: 3, # 'ח' - 22: 3, # 'ט' - 1: 3, # 'י' - 25: 2, # 'ך' - 15: 3, # 'כ' - 4: 3, # 'ל' - 11: 2, # 'ם' - 6: 3, # 'מ' - 23: 2, # 'ן' - 12: 3, # 'נ' - 19: 2, # 'ס' - 13: 3, # 'ע' - 26: 3, # 'ף' - 18: 3, # 'פ' - 27: 0, # 'ץ' - 21: 2, # 'צ' - 17: 3, # 'ק' - 7: 3, # 'ר' - 10: 1, # 'ש' - 5: 3, # 'ת' - 32: 0, # '–' - 52: 0, # '’' - 47: 0, # '“' - 46: 1, # '”' - 58: 0, # '†' - 40: 1, # '…' - }, - 13: { # 'ע' - 50: 0, # 'a' - 60: 0, # 'c' - 61: 0, # 'd' - 42: 0, # 'e' - 53: 0, # 'i' - 56: 0, # 'l' - 54: 0, # 'n' - 49: 0, # 'o' - 51: 0, # 'r' - 43: 0, # 's' - 44: 0, # 't' - 63: 0, # 'u' - 34: 0, # '\xa0' - 55: 0, # '´' - 48: 1, # '¼' - 39: 0, # '½' - 57: 0, # '¾' - 30: 1, # 'ְ' - 59: 1, # 'ֱ' - 41: 2, # 'ֲ' - 33: 2, # 'ִ' - 37: 2, # 'ֵ' - 36: 2, # 'ֶ' - 31: 2, # 'ַ' - 29: 2, # 'ָ' - 35: 2, # 'ֹ' - 62: 1, # 'ֻ' - 28: 0, # 'ּ' - 38: 0, # 'ׁ' - 45: 0, # 'ׂ' - 9: 2, # 'א' - 8: 3, # 'ב' - 20: 3, # 'ג' - 16: 3, # 'ד' - 3: 3, # 'ה' - 2: 3, # 'ו' - 24: 3, # 'ז' - 14: 1, # 'ח' - 22: 3, # 'ט' - 1: 3, # 'י' - 25: 2, # 'ך' - 15: 2, # 'כ' - 4: 3, # 'ל' - 11: 3, # 'ם' - 6: 3, # 'מ' - 23: 2, # 'ן' - 12: 3, # 'נ' - 19: 3, # 'ס' - 13: 2, # 'ע' - 26: 1, # 'ף' - 18: 2, # 'פ' - 27: 2, # 'ץ' - 21: 3, # 'צ' - 17: 3, # 'ק' - 7: 3, # 'ר' - 10: 3, # 'ש' - 5: 3, # 'ת' - 32: 0, # '–' - 52: 0, # '’' - 47: 0, # '“' - 46: 1, # '”' - 58: 0, # '†' - 40: 1, # '…' - }, - 26: { # 'ף' - 50: 0, # 'a' - 60: 0, # 'c' - 61: 0, # 'd' - 42: 0, # 'e' - 53: 0, # 'i' - 56: 0, # 'l' - 54: 0, # 'n' - 49: 0, # 'o' - 51: 0, # 'r' - 43: 0, # 's' - 44: 0, # 't' - 63: 0, # 'u' - 34: 0, # '\xa0' - 55: 0, # '´' - 48: 0, # '¼' - 39: 0, # '½' - 57: 0, # '¾' - 30: 0, # 'ְ' - 59: 0, # 'ֱ' - 41: 0, # 'ֲ' - 33: 0, # 'ִ' - 37: 0, # 'ֵ' - 36: 0, # 'ֶ' - 31: 0, # 'ַ' - 29: 0, # 'ָ' - 35: 0, # 'ֹ' - 62: 0, # 'ֻ' - 28: 0, # 'ּ' - 38: 0, # 'ׁ' - 45: 0, # 'ׂ' - 9: 1, # 'א' - 8: 0, # 'ב' - 20: 0, # 'ג' - 16: 0, # 'ד' - 3: 0, # 'ה' - 2: 1, # 'ו' - 24: 0, # 'ז' - 14: 1, # 'ח' - 22: 0, # 'ט' - 1: 0, # 'י' - 25: 0, # 'ך' - 15: 1, # 'כ' - 4: 1, # 'ל' - 11: 0, # 'ם' - 6: 1, # 'מ' - 23: 0, # 'ן' - 12: 0, # 'נ' - 19: 1, # 'ס' - 13: 0, # 'ע' - 26: 1, # 'ף' - 18: 1, # 'פ' - 27: 0, # 'ץ' - 21: 0, # 'צ' - 17: 1, # 'ק' - 7: 1, # 'ר' - 10: 1, # 'ש' - 5: 0, # 'ת' - 32: 0, # '–' - 52: 0, # '’' - 47: 0, # '“' - 46: 0, # '”' - 58: 0, # '†' - 40: 1, # '…' - }, - 18: { # 'פ' - 50: 0, # 'a' - 60: 0, # 'c' - 61: 0, # 'd' - 42: 0, # 'e' - 53: 0, # 'i' - 56: 0, # 'l' - 54: 0, # 'n' - 49: 0, # 'o' - 51: 0, # 'r' - 43: 0, # 's' - 44: 0, # 't' - 63: 0, # 'u' - 34: 0, # '\xa0' - 55: 1, # '´' - 48: 0, # '¼' - 39: 0, # '½' - 57: 0, # '¾' - 30: 2, # 'ְ' - 59: 0, # 'ֱ' - 41: 0, # 'ֲ' - 33: 2, # 'ִ' - 37: 1, # 'ֵ' - 36: 2, # 'ֶ' - 31: 1, # 'ַ' - 29: 2, # 'ָ' - 35: 1, # 'ֹ' - 62: 1, # 'ֻ' - 28: 2, # 'ּ' - 38: 0, # 'ׁ' - 45: 0, # 'ׂ' - 9: 3, # 'א' - 8: 2, # 'ב' - 20: 3, # 'ג' - 16: 2, # 'ד' - 3: 3, # 'ה' - 2: 3, # 'ו' - 24: 2, # 'ז' - 14: 3, # 'ח' - 22: 3, # 'ט' - 1: 3, # 'י' - 25: 2, # 'ך' - 15: 3, # 'כ' - 4: 3, # 'ל' - 11: 2, # 'ם' - 6: 2, # 'מ' - 23: 3, # 'ן' - 12: 3, # 'נ' - 19: 3, # 'ס' - 13: 3, # 'ע' - 26: 2, # 'ף' - 18: 2, # 'פ' - 27: 2, # 'ץ' - 21: 3, # 'צ' - 17: 3, # 'ק' - 7: 3, # 'ר' - 10: 3, # 'ש' - 5: 3, # 'ת' - 32: 0, # '–' - 52: 0, # '’' - 47: 0, # '“' - 46: 1, # '”' - 58: 0, # '†' - 40: 0, # '…' - }, - 27: { # 'ץ' - 50: 0, # 'a' - 60: 0, # 'c' - 61: 0, # 'd' - 42: 0, # 'e' - 53: 0, # 'i' - 56: 0, # 'l' - 54: 0, # 'n' - 49: 0, # 'o' - 51: 0, # 'r' - 43: 0, # 's' - 44: 0, # 't' - 63: 0, # 'u' - 34: 0, # '\xa0' - 55: 1, # '´' - 48: 0, # '¼' - 39: 0, # '½' - 57: 0, # '¾' - 30: 0, # 'ְ' - 59: 0, # 'ֱ' - 41: 0, # 'ֲ' - 33: 0, # 'ִ' - 37: 0, # 'ֵ' - 36: 0, # 'ֶ' - 31: 0, # 'ַ' - 29: 0, # 'ָ' - 35: 0, # 'ֹ' - 62: 0, # 'ֻ' - 28: 0, # 'ּ' - 38: 0, # 'ׁ' - 45: 0, # 'ׂ' - 9: 1, # 'א' - 8: 0, # 'ב' - 20: 0, # 'ג' - 16: 0, # 'ד' - 3: 0, # 'ה' - 2: 0, # 'ו' - 24: 0, # 'ז' - 14: 0, # 'ח' - 22: 0, # 'ט' - 1: 0, # 'י' - 25: 0, # 'ך' - 15: 0, # 'כ' - 4: 1, # 'ל' - 11: 0, # 'ם' - 6: 0, # 'מ' - 23: 0, # 'ן' - 12: 0, # 'נ' - 19: 1, # 'ס' - 13: 0, # 'ע' - 26: 0, # 'ף' - 18: 0, # 'פ' - 27: 0, # 'ץ' - 21: 0, # 'צ' - 17: 0, # 'ק' - 7: 1, # 'ר' - 10: 0, # 'ש' - 5: 1, # 'ת' - 32: 0, # '–' - 52: 0, # '’' - 47: 0, # '“' - 46: 0, # '”' - 58: 0, # '†' - 40: 1, # '…' - }, - 21: { # 'צ' - 50: 0, # 'a' - 60: 0, # 'c' - 61: 0, # 'd' - 42: 0, # 'e' - 53: 0, # 'i' - 56: 0, # 'l' - 54: 0, # 'n' - 49: 0, # 'o' - 51: 0, # 'r' - 43: 0, # 's' - 44: 0, # 't' - 63: 0, # 'u' - 34: 0, # '\xa0' - 55: 1, # '´' - 48: 0, # '¼' - 39: 0, # '½' - 57: 0, # '¾' - 30: 2, # 'ְ' - 59: 0, # 'ֱ' - 41: 0, # 'ֲ' - 33: 2, # 'ִ' - 37: 2, # 'ֵ' - 36: 1, # 'ֶ' - 31: 2, # 'ַ' - 29: 2, # 'ָ' - 35: 1, # 'ֹ' - 62: 1, # 'ֻ' - 28: 2, # 'ּ' - 38: 0, # 'ׁ' - 45: 0, # 'ׂ' - 9: 3, # 'א' - 8: 3, # 'ב' - 20: 2, # 'ג' - 16: 3, # 'ד' - 3: 3, # 'ה' - 2: 3, # 'ו' - 24: 1, # 'ז' - 14: 3, # 'ח' - 22: 2, # 'ט' - 1: 3, # 'י' - 25: 1, # 'ך' - 15: 1, # 'כ' - 4: 3, # 'ל' - 11: 2, # 'ם' - 6: 3, # 'מ' - 23: 2, # 'ן' - 12: 3, # 'נ' - 19: 1, # 'ס' - 13: 3, # 'ע' - 26: 2, # 'ף' - 18: 3, # 'פ' - 27: 2, # 'ץ' - 21: 2, # 'צ' - 17: 3, # 'ק' - 7: 3, # 'ר' - 10: 0, # 'ש' - 5: 3, # 'ת' - 32: 0, # '–' - 52: 0, # '’' - 47: 0, # '“' - 46: 0, # '”' - 58: 0, # '†' - 40: 0, # '…' - }, - 17: { # 'ק' - 50: 0, # 'a' - 60: 0, # 'c' - 61: 0, # 'd' - 42: 0, # 'e' - 53: 0, # 'i' - 56: 0, # 'l' - 54: 0, # 'n' - 49: 0, # 'o' - 51: 0, # 'r' - 43: 0, # 's' - 44: 0, # 't' - 63: 0, # 'u' - 34: 1, # '\xa0' - 55: 1, # '´' - 48: 0, # '¼' - 39: 0, # '½' - 57: 0, # '¾' - 30: 2, # 'ְ' - 59: 0, # 'ֱ' - 41: 0, # 'ֲ' - 33: 2, # 'ִ' - 37: 2, # 'ֵ' - 36: 1, # 'ֶ' - 31: 2, # 'ַ' - 29: 2, # 'ָ' - 35: 2, # 'ֹ' - 62: 1, # 'ֻ' - 28: 2, # 'ּ' - 38: 0, # 'ׁ' - 45: 0, # 'ׂ' - 9: 3, # 'א' - 8: 3, # 'ב' - 20: 2, # 'ג' - 16: 3, # 'ד' - 3: 3, # 'ה' - 2: 3, # 'ו' - 24: 2, # 'ז' - 14: 3, # 'ח' - 22: 3, # 'ט' - 1: 3, # 'י' - 25: 1, # 'ך' - 15: 1, # 'כ' - 4: 3, # 'ל' - 11: 2, # 'ם' - 6: 3, # 'מ' - 23: 2, # 'ן' - 12: 3, # 'נ' - 19: 3, # 'ס' - 13: 3, # 'ע' - 26: 2, # 'ף' - 18: 3, # 'פ' - 27: 2, # 'ץ' - 21: 3, # 'צ' - 17: 2, # 'ק' - 7: 3, # 'ר' - 10: 3, # 'ש' - 5: 3, # 'ת' - 32: 0, # '–' - 52: 1, # '’' - 47: 0, # '“' - 46: 1, # '”' - 58: 0, # '†' - 40: 1, # '…' - }, - 7: { # 'ר' - 50: 0, # 'a' - 60: 0, # 'c' - 61: 0, # 'd' - 42: 0, # 'e' - 53: 0, # 'i' - 56: 0, # 'l' - 54: 0, # 'n' - 49: 0, # 'o' - 51: 0, # 'r' - 43: 0, # 's' - 44: 0, # 't' - 63: 0, # 'u' - 34: 1, # '\xa0' - 55: 2, # '´' - 48: 1, # '¼' - 39: 0, # '½' - 57: 0, # '¾' - 30: 2, # 'ְ' - 59: 0, # 'ֱ' - 41: 1, # 'ֲ' - 33: 2, # 'ִ' - 37: 2, # 'ֵ' - 36: 2, # 'ֶ' - 31: 2, # 'ַ' - 29: 2, # 'ָ' - 35: 2, # 'ֹ' - 62: 1, # 'ֻ' - 28: 0, # 'ּ' - 38: 0, # 'ׁ' - 45: 0, # 'ׂ' - 9: 3, # 'א' - 8: 3, # 'ב' - 20: 3, # 'ג' - 16: 3, # 'ד' - 3: 3, # 'ה' - 2: 3, # 'ו' - 24: 3, # 'ז' - 14: 3, # 'ח' - 22: 3, # 'ט' - 1: 3, # 'י' - 25: 3, # 'ך' - 15: 3, # 'כ' - 4: 3, # 'ל' - 11: 3, # 'ם' - 6: 3, # 'מ' - 23: 3, # 'ן' - 12: 3, # 'נ' - 19: 3, # 'ס' - 13: 3, # 'ע' - 26: 2, # 'ף' - 18: 3, # 'פ' - 27: 3, # 'ץ' - 21: 3, # 'צ' - 17: 3, # 'ק' - 7: 3, # 'ר' - 10: 3, # 'ש' - 5: 3, # 'ת' - 32: 0, # '–' - 52: 0, # '’' - 47: 0, # '“' - 46: 1, # '”' - 58: 0, # '†' - 40: 2, # '…' - }, - 10: { # 'ש' - 50: 0, # 'a' - 60: 0, # 'c' - 61: 0, # 'd' - 42: 0, # 'e' - 53: 0, # 'i' - 56: 0, # 'l' - 54: 0, # 'n' - 49: 0, # 'o' - 51: 0, # 'r' - 43: 0, # 's' - 44: 0, # 't' - 63: 0, # 'u' - 34: 1, # '\xa0' - 55: 0, # '´' - 48: 0, # '¼' - 39: 0, # '½' - 57: 0, # '¾' - 30: 1, # 'ְ' - 59: 0, # 'ֱ' - 41: 0, # 'ֲ' - 33: 1, # 'ִ' - 37: 1, # 'ֵ' - 36: 1, # 'ֶ' - 31: 1, # 'ַ' - 29: 1, # 'ָ' - 35: 1, # 'ֹ' - 62: 1, # 'ֻ' - 28: 2, # 'ּ' - 38: 3, # 'ׁ' - 45: 2, # 'ׂ' - 9: 3, # 'א' - 8: 3, # 'ב' - 20: 3, # 'ג' - 16: 3, # 'ד' - 3: 3, # 'ה' - 2: 3, # 'ו' - 24: 2, # 'ז' - 14: 3, # 'ח' - 22: 3, # 'ט' - 1: 3, # 'י' - 25: 3, # 'ך' - 15: 3, # 'כ' - 4: 3, # 'ל' - 11: 3, # 'ם' - 6: 3, # 'מ' - 23: 2, # 'ן' - 12: 3, # 'נ' - 19: 2, # 'ס' - 13: 3, # 'ע' - 26: 2, # 'ף' - 18: 3, # 'פ' - 27: 1, # 'ץ' - 21: 2, # 'צ' - 17: 3, # 'ק' - 7: 3, # 'ר' - 10: 3, # 'ש' - 5: 3, # 'ת' - 32: 0, # '–' - 52: 0, # '’' - 47: 0, # '“' - 46: 1, # '”' - 58: 0, # '†' - 40: 1, # '…' - }, - 5: { # 'ת' - 50: 0, # 'a' - 60: 0, # 'c' - 61: 0, # 'd' - 42: 0, # 'e' - 53: 0, # 'i' - 56: 0, # 'l' - 54: 0, # 'n' - 49: 0, # 'o' - 51: 0, # 'r' - 43: 0, # 's' - 44: 0, # 't' - 63: 0, # 'u' - 34: 1, # '\xa0' - 55: 0, # '´' - 48: 1, # '¼' - 39: 1, # '½' - 57: 0, # '¾' - 30: 2, # 'ְ' - 59: 0, # 'ֱ' - 41: 0, # 'ֲ' - 33: 2, # 'ִ' - 37: 2, # 'ֵ' - 36: 2, # 'ֶ' - 31: 2, # 'ַ' - 29: 2, # 'ָ' - 35: 1, # 'ֹ' - 62: 1, # 'ֻ' - 28: 2, # 'ּ' - 38: 0, # 'ׁ' - 45: 0, # 'ׂ' - 9: 3, # 'א' - 8: 3, # 'ב' - 20: 3, # 'ג' - 16: 2, # 'ד' - 3: 3, # 'ה' - 2: 3, # 'ו' - 24: 2, # 'ז' - 14: 3, # 'ח' - 22: 2, # 'ט' - 1: 3, # 'י' - 25: 2, # 'ך' - 15: 3, # 'כ' - 4: 3, # 'ל' - 11: 3, # 'ם' - 6: 3, # 'מ' - 23: 3, # 'ן' - 12: 3, # 'נ' - 19: 2, # 'ס' - 13: 3, # 'ע' - 26: 2, # 'ף' - 18: 3, # 'פ' - 27: 1, # 'ץ' - 21: 2, # 'צ' - 17: 3, # 'ק' - 7: 3, # 'ר' - 10: 3, # 'ש' - 5: 3, # 'ת' - 32: 1, # '–' - 52: 1, # '’' - 47: 0, # '“' - 46: 0, # '”' - 58: 0, # '†' - 40: 2, # '…' - }, - 32: { # '–' - 50: 0, # 'a' - 60: 0, # 'c' - 61: 0, # 'd' - 42: 0, # 'e' - 53: 0, # 'i' - 56: 0, # 'l' - 54: 1, # 'n' - 49: 0, # 'o' - 51: 0, # 'r' - 43: 0, # 's' - 44: 0, # 't' - 63: 0, # 'u' - 34: 0, # '\xa0' - 55: 0, # '´' - 48: 0, # '¼' - 39: 0, # '½' - 57: 0, # '¾' - 30: 0, # 'ְ' - 59: 0, # 'ֱ' - 41: 0, # 'ֲ' - 33: 0, # 'ִ' - 37: 0, # 'ֵ' - 36: 0, # 'ֶ' - 31: 0, # 'ַ' - 29: 0, # 'ָ' - 35: 0, # 'ֹ' - 62: 0, # 'ֻ' - 28: 0, # 'ּ' - 38: 0, # 'ׁ' - 45: 0, # 'ׂ' - 9: 1, # 'א' - 8: 1, # 'ב' - 20: 1, # 'ג' - 16: 1, # 'ד' - 3: 1, # 'ה' - 2: 1, # 'ו' - 24: 0, # 'ז' - 14: 1, # 'ח' - 22: 0, # 'ט' - 1: 1, # 'י' - 25: 0, # 'ך' - 15: 1, # 'כ' - 4: 1, # 'ל' - 11: 0, # 'ם' - 6: 1, # 'מ' - 23: 0, # 'ן' - 12: 0, # 'נ' - 19: 1, # 'ס' - 13: 1, # 'ע' - 26: 0, # 'ף' - 18: 1, # 'פ' - 27: 0, # 'ץ' - 21: 1, # 'צ' - 17: 0, # 'ק' - 7: 1, # 'ר' - 10: 1, # 'ש' - 5: 1, # 'ת' - 32: 0, # '–' - 52: 0, # '’' - 47: 0, # '“' - 46: 0, # '”' - 58: 0, # '†' - 40: 0, # '…' - }, - 52: { # '’' - 50: 1, # 'a' - 60: 0, # 'c' - 61: 1, # 'd' - 42: 1, # 'e' - 53: 1, # 'i' - 56: 1, # 'l' - 54: 0, # 'n' - 49: 0, # 'o' - 51: 1, # 'r' - 43: 2, # 's' - 44: 2, # 't' - 63: 1, # 'u' - 34: 0, # '\xa0' - 55: 0, # '´' - 48: 0, # '¼' - 39: 0, # '½' - 57: 0, # '¾' - 30: 0, # 'ְ' - 59: 0, # 'ֱ' - 41: 0, # 'ֲ' - 33: 0, # 'ִ' - 37: 0, # 'ֵ' - 36: 0, # 'ֶ' - 31: 0, # 'ַ' - 29: 0, # 'ָ' - 35: 0, # 'ֹ' - 62: 0, # 'ֻ' - 28: 0, # 'ּ' - 38: 0, # 'ׁ' - 45: 0, # 'ׂ' - 9: 0, # 'א' - 8: 0, # 'ב' - 20: 0, # 'ג' - 16: 0, # 'ד' - 3: 0, # 'ה' - 2: 1, # 'ו' - 24: 0, # 'ז' - 14: 0, # 'ח' - 22: 0, # 'ט' - 1: 0, # 'י' - 25: 0, # 'ך' - 15: 0, # 'כ' - 4: 0, # 'ל' - 11: 0, # 'ם' - 6: 1, # 'מ' - 23: 0, # 'ן' - 12: 0, # 'נ' - 19: 0, # 'ס' - 13: 0, # 'ע' - 26: 0, # 'ף' - 18: 0, # 'פ' - 27: 0, # 'ץ' - 21: 0, # 'צ' - 17: 0, # 'ק' - 7: 0, # 'ר' - 10: 0, # 'ש' - 5: 1, # 'ת' - 32: 0, # '–' - 52: 0, # '’' - 47: 0, # '“' - 46: 0, # '”' - 58: 0, # '†' - 40: 0, # '…' - }, - 47: { # '“' - 50: 1, # 'a' - 60: 1, # 'c' - 61: 1, # 'd' - 42: 1, # 'e' - 53: 1, # 'i' - 56: 1, # 'l' - 54: 1, # 'n' - 49: 1, # 'o' - 51: 1, # 'r' - 43: 1, # 's' - 44: 1, # 't' - 63: 1, # 'u' - 34: 0, # '\xa0' - 55: 0, # '´' - 48: 0, # '¼' - 39: 0, # '½' - 57: 0, # '¾' - 30: 0, # 'ְ' - 59: 0, # 'ֱ' - 41: 0, # 'ֲ' - 33: 0, # 'ִ' - 37: 0, # 'ֵ' - 36: 0, # 'ֶ' - 31: 0, # 'ַ' - 29: 0, # 'ָ' - 35: 0, # 'ֹ' - 62: 0, # 'ֻ' - 28: 0, # 'ּ' - 38: 0, # 'ׁ' - 45: 0, # 'ׂ' - 9: 2, # 'א' - 8: 1, # 'ב' - 20: 1, # 'ג' - 16: 1, # 'ד' - 3: 1, # 'ה' - 2: 1, # 'ו' - 24: 1, # 'ז' - 14: 1, # 'ח' - 22: 1, # 'ט' - 1: 1, # 'י' - 25: 0, # 'ך' - 15: 1, # 'כ' - 4: 1, # 'ל' - 11: 0, # 'ם' - 6: 1, # 'מ' - 23: 0, # 'ן' - 12: 1, # 'נ' - 19: 1, # 'ס' - 13: 1, # 'ע' - 26: 0, # 'ף' - 18: 1, # 'פ' - 27: 0, # 'ץ' - 21: 1, # 'צ' - 17: 1, # 'ק' - 7: 1, # 'ר' - 10: 1, # 'ש' - 5: 1, # 'ת' - 32: 0, # '–' - 52: 0, # '’' - 47: 0, # '“' - 46: 0, # '”' - 58: 0, # '†' - 40: 0, # '…' - }, - 46: { # '”' - 50: 0, # 'a' - 60: 0, # 'c' - 61: 0, # 'd' - 42: 0, # 'e' - 53: 0, # 'i' - 56: 0, # 'l' - 54: 0, # 'n' - 49: 0, # 'o' - 51: 0, # 'r' - 43: 0, # 's' - 44: 1, # 't' - 63: 0, # 'u' - 34: 0, # '\xa0' - 55: 0, # '´' - 48: 0, # '¼' - 39: 0, # '½' - 57: 0, # '¾' - 30: 0, # 'ְ' - 59: 0, # 'ֱ' - 41: 0, # 'ֲ' - 33: 0, # 'ִ' - 37: 0, # 'ֵ' - 36: 0, # 'ֶ' - 31: 0, # 'ַ' - 29: 0, # 'ָ' - 35: 0, # 'ֹ' - 62: 0, # 'ֻ' - 28: 0, # 'ּ' - 38: 0, # 'ׁ' - 45: 0, # 'ׂ' - 9: 1, # 'א' - 8: 1, # 'ב' - 20: 1, # 'ג' - 16: 0, # 'ד' - 3: 0, # 'ה' - 2: 0, # 'ו' - 24: 0, # 'ז' - 14: 0, # 'ח' - 22: 0, # 'ט' - 1: 1, # 'י' - 25: 0, # 'ך' - 15: 1, # 'כ' - 4: 1, # 'ל' - 11: 0, # 'ם' - 6: 1, # 'מ' - 23: 0, # 'ן' - 12: 0, # 'נ' - 19: 0, # 'ס' - 13: 0, # 'ע' - 26: 0, # 'ף' - 18: 0, # 'פ' - 27: 0, # 'ץ' - 21: 1, # 'צ' - 17: 0, # 'ק' - 7: 1, # 'ר' - 10: 0, # 'ש' - 5: 0, # 'ת' - 32: 0, # '–' - 52: 0, # '’' - 47: 0, # '“' - 46: 0, # '”' - 58: 0, # '†' - 40: 0, # '…' - }, - 58: { # '†' - 50: 0, # 'a' - 60: 0, # 'c' - 61: 0, # 'd' - 42: 0, # 'e' - 53: 0, # 'i' - 56: 0, # 'l' - 54: 0, # 'n' - 49: 0, # 'o' - 51: 0, # 'r' - 43: 0, # 's' - 44: 0, # 't' - 63: 0, # 'u' - 34: 0, # '\xa0' - 55: 0, # '´' - 48: 0, # '¼' - 39: 0, # '½' - 57: 0, # '¾' - 30: 0, # 'ְ' - 59: 0, # 'ֱ' - 41: 0, # 'ֲ' - 33: 0, # 'ִ' - 37: 0, # 'ֵ' - 36: 0, # 'ֶ' - 31: 0, # 'ַ' - 29: 0, # 'ָ' - 35: 0, # 'ֹ' - 62: 0, # 'ֻ' - 28: 0, # 'ּ' - 38: 0, # 'ׁ' - 45: 0, # 'ׂ' - 9: 0, # 'א' - 8: 0, # 'ב' - 20: 0, # 'ג' - 16: 0, # 'ד' - 3: 0, # 'ה' - 2: 0, # 'ו' - 24: 0, # 'ז' - 14: 0, # 'ח' - 22: 0, # 'ט' - 1: 0, # 'י' - 25: 0, # 'ך' - 15: 0, # 'כ' - 4: 0, # 'ל' - 11: 0, # 'ם' - 6: 0, # 'מ' - 23: 0, # 'ן' - 12: 0, # 'נ' - 19: 0, # 'ס' - 13: 0, # 'ע' - 26: 0, # 'ף' - 18: 0, # 'פ' - 27: 0, # 'ץ' - 21: 0, # 'צ' - 17: 0, # 'ק' - 7: 0, # 'ר' - 10: 0, # 'ש' - 5: 0, # 'ת' - 32: 0, # '–' - 52: 0, # '’' - 47: 0, # '“' - 46: 0, # '”' - 58: 2, # '†' - 40: 0, # '…' - }, - 40: { # '…' - 50: 1, # 'a' - 60: 1, # 'c' - 61: 1, # 'd' - 42: 1, # 'e' - 53: 1, # 'i' - 56: 0, # 'l' - 54: 1, # 'n' - 49: 0, # 'o' - 51: 1, # 'r' - 43: 1, # 's' - 44: 1, # 't' - 63: 0, # 'u' - 34: 0, # '\xa0' - 55: 0, # '´' - 48: 0, # '¼' - 39: 0, # '½' - 57: 0, # '¾' - 30: 0, # 'ְ' - 59: 0, # 'ֱ' - 41: 0, # 'ֲ' - 33: 0, # 'ִ' - 37: 0, # 'ֵ' - 36: 0, # 'ֶ' - 31: 0, # 'ַ' - 29: 0, # 'ָ' - 35: 0, # 'ֹ' - 62: 0, # 'ֻ' - 28: 0, # 'ּ' - 38: 0, # 'ׁ' - 45: 0, # 'ׂ' - 9: 1, # 'א' - 8: 0, # 'ב' - 20: 0, # 'ג' - 16: 0, # 'ד' - 3: 1, # 'ה' - 2: 1, # 'ו' - 24: 1, # 'ז' - 14: 0, # 'ח' - 22: 0, # 'ט' - 1: 1, # 'י' - 25: 0, # 'ך' - 15: 1, # 'כ' - 4: 1, # 'ל' - 11: 0, # 'ם' - 6: 1, # 'מ' - 23: 0, # 'ן' - 12: 1, # 'נ' - 19: 0, # 'ס' - 13: 0, # 'ע' - 26: 0, # 'ף' - 18: 1, # 'פ' - 27: 0, # 'ץ' - 21: 0, # 'צ' - 17: 0, # 'ק' - 7: 1, # 'ר' - 10: 1, # 'ש' - 5: 1, # 'ת' - 32: 0, # '–' - 52: 0, # '’' - 47: 0, # '“' - 46: 1, # '”' - 58: 0, # '†' - 40: 2, # '…' - }, -} - -# 255: Undefined characters that did not exist in training text -# 254: Carriage/Return -# 253: symbol (punctuation) that does not belong to word -# 252: 0 - 9 -# 251: Control characters - -# Character Mapping Table(s): -WINDOWS_1255_HEBREW_CHAR_TO_ORDER = { - 0: 255, # '\x00' - 1: 255, # '\x01' - 2: 255, # '\x02' - 3: 255, # '\x03' - 4: 255, # '\x04' - 5: 255, # '\x05' - 6: 255, # '\x06' - 7: 255, # '\x07' - 8: 255, # '\x08' - 9: 255, # '\t' - 10: 254, # '\n' - 11: 255, # '\x0b' - 12: 255, # '\x0c' - 13: 254, # '\r' - 14: 255, # '\x0e' - 15: 255, # '\x0f' - 16: 255, # '\x10' - 17: 255, # '\x11' - 18: 255, # '\x12' - 19: 255, # '\x13' - 20: 255, # '\x14' - 21: 255, # '\x15' - 22: 255, # '\x16' - 23: 255, # '\x17' - 24: 255, # '\x18' - 25: 255, # '\x19' - 26: 255, # '\x1a' - 27: 255, # '\x1b' - 28: 255, # '\x1c' - 29: 255, # '\x1d' - 30: 255, # '\x1e' - 31: 255, # '\x1f' - 32: 253, # ' ' - 33: 253, # '!' - 34: 253, # '"' - 35: 253, # '#' - 36: 253, # '$' - 37: 253, # '%' - 38: 253, # '&' - 39: 253, # "'" - 40: 253, # '(' - 41: 253, # ')' - 42: 253, # '*' - 43: 253, # '+' - 44: 253, # ',' - 45: 253, # '-' - 46: 253, # '.' - 47: 253, # '/' - 48: 252, # '0' - 49: 252, # '1' - 50: 252, # '2' - 51: 252, # '3' - 52: 252, # '4' - 53: 252, # '5' - 54: 252, # '6' - 55: 252, # '7' - 56: 252, # '8' - 57: 252, # '9' - 58: 253, # ':' - 59: 253, # ';' - 60: 253, # '<' - 61: 253, # '=' - 62: 253, # '>' - 63: 253, # '?' - 64: 253, # '@' - 65: 69, # 'A' - 66: 91, # 'B' - 67: 79, # 'C' - 68: 80, # 'D' - 69: 92, # 'E' - 70: 89, # 'F' - 71: 97, # 'G' - 72: 90, # 'H' - 73: 68, # 'I' - 74: 111, # 'J' - 75: 112, # 'K' - 76: 82, # 'L' - 77: 73, # 'M' - 78: 95, # 'N' - 79: 85, # 'O' - 80: 78, # 'P' - 81: 121, # 'Q' - 82: 86, # 'R' - 83: 71, # 'S' - 84: 67, # 'T' - 85: 102, # 'U' - 86: 107, # 'V' - 87: 84, # 'W' - 88: 114, # 'X' - 89: 103, # 'Y' - 90: 115, # 'Z' - 91: 253, # '[' - 92: 253, # '\\' - 93: 253, # ']' - 94: 253, # '^' - 95: 253, # '_' - 96: 253, # '`' - 97: 50, # 'a' - 98: 74, # 'b' - 99: 60, # 'c' - 100: 61, # 'd' - 101: 42, # 'e' - 102: 76, # 'f' - 103: 70, # 'g' - 104: 64, # 'h' - 105: 53, # 'i' - 106: 105, # 'j' - 107: 93, # 'k' - 108: 56, # 'l' - 109: 65, # 'm' - 110: 54, # 'n' - 111: 49, # 'o' - 112: 66, # 'p' - 113: 110, # 'q' - 114: 51, # 'r' - 115: 43, # 's' - 116: 44, # 't' - 117: 63, # 'u' - 118: 81, # 'v' - 119: 77, # 'w' - 120: 98, # 'x' - 121: 75, # 'y' - 122: 108, # 'z' - 123: 253, # '{' - 124: 253, # '|' - 125: 253, # '}' - 126: 253, # '~' - 127: 253, # '\x7f' - 128: 124, # '€' - 129: 202, # None - 130: 203, # '‚' - 131: 204, # 'ƒ' - 132: 205, # '„' - 133: 40, # '…' - 134: 58, # '†' - 135: 206, # '‡' - 136: 207, # 'ˆ' - 137: 208, # '‰' - 138: 209, # None - 139: 210, # '‹' - 140: 211, # None - 141: 212, # None - 142: 213, # None - 143: 214, # None - 144: 215, # None - 145: 83, # '‘' - 146: 52, # '’' - 147: 47, # '“' - 148: 46, # '”' - 149: 72, # '•' - 150: 32, # '–' - 151: 94, # '—' - 152: 216, # '˜' - 153: 113, # '™' - 154: 217, # None - 155: 109, # '›' - 156: 218, # None - 157: 219, # None - 158: 220, # None - 159: 221, # None - 160: 34, # '\xa0' - 161: 116, # '¡' - 162: 222, # '¢' - 163: 118, # '£' - 164: 100, # '₪' - 165: 223, # '¥' - 166: 224, # '¦' - 167: 117, # '§' - 168: 119, # '¨' - 169: 104, # '©' - 170: 125, # '×' - 171: 225, # '«' - 172: 226, # '¬' - 173: 87, # '\xad' - 174: 99, # '®' - 175: 227, # '¯' - 176: 106, # '°' - 177: 122, # '±' - 178: 123, # '²' - 179: 228, # '³' - 180: 55, # '´' - 181: 229, # 'µ' - 182: 230, # '¶' - 183: 101, # '·' - 184: 231, # '¸' - 185: 232, # '¹' - 186: 120, # '÷' - 187: 233, # '»' - 188: 48, # '¼' - 189: 39, # '½' - 190: 57, # '¾' - 191: 234, # '¿' - 192: 30, # 'ְ' - 193: 59, # 'ֱ' - 194: 41, # 'ֲ' - 195: 88, # 'ֳ' - 196: 33, # 'ִ' - 197: 37, # 'ֵ' - 198: 36, # 'ֶ' - 199: 31, # 'ַ' - 200: 29, # 'ָ' - 201: 35, # 'ֹ' - 202: 235, # None - 203: 62, # 'ֻ' - 204: 28, # 'ּ' - 205: 236, # 'ֽ' - 206: 126, # '־' - 207: 237, # 'ֿ' - 208: 238, # '׀' - 209: 38, # 'ׁ' - 210: 45, # 'ׂ' - 211: 239, # '׃' - 212: 240, # 'װ' - 213: 241, # 'ױ' - 214: 242, # 'ײ' - 215: 243, # '׳' - 216: 127, # '״' - 217: 244, # None - 218: 245, # None - 219: 246, # None - 220: 247, # None - 221: 248, # None - 222: 249, # None - 223: 250, # None - 224: 9, # 'א' - 225: 8, # 'ב' - 226: 20, # 'ג' - 227: 16, # 'ד' - 228: 3, # 'ה' - 229: 2, # 'ו' - 230: 24, # 'ז' - 231: 14, # 'ח' - 232: 22, # 'ט' - 233: 1, # 'י' - 234: 25, # 'ך' - 235: 15, # 'כ' - 236: 4, # 'ל' - 237: 11, # 'ם' - 238: 6, # 'מ' - 239: 23, # 'ן' - 240: 12, # 'נ' - 241: 19, # 'ס' - 242: 13, # 'ע' - 243: 26, # 'ף' - 244: 18, # 'פ' - 245: 27, # 'ץ' - 246: 21, # 'צ' - 247: 17, # 'ק' - 248: 7, # 'ר' - 249: 10, # 'ש' - 250: 5, # 'ת' - 251: 251, # None - 252: 252, # None - 253: 128, # '\u200e' - 254: 96, # '\u200f' - 255: 253, # None -} - -WINDOWS_1255_HEBREW_MODEL = SingleByteCharSetModel( - charset_name="windows-1255", - language="Hebrew", - char_to_order_map=WINDOWS_1255_HEBREW_CHAR_TO_ORDER, - language_model=HEBREW_LANG_MODEL, - typical_positive_ratio=0.984004, - keep_ascii_letters=False, - alphabet="אבגדהוזחטיךכלםמןנסעףפץצקרשתװױײ", -) diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/langhungarianmodel.py b/venv/lib/python3.11/site-packages/pip/_vendor/chardet/langhungarianmodel.py deleted file mode 100644 index 09a0d32..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/langhungarianmodel.py +++ /dev/null @@ -1,4649 +0,0 @@ -from pip._vendor.chardet.sbcharsetprober import SingleByteCharSetModel - -# 3: Positive -# 2: Likely -# 1: Unlikely -# 0: Negative - -HUNGARIAN_LANG_MODEL = { - 28: { # 'A' - 28: 0, # 'A' - 40: 1, # 'B' - 54: 1, # 'C' - 45: 2, # 'D' - 32: 1, # 'E' - 50: 1, # 'F' - 49: 2, # 'G' - 38: 1, # 'H' - 39: 2, # 'I' - 53: 1, # 'J' - 36: 2, # 'K' - 41: 2, # 'L' - 34: 1, # 'M' - 35: 2, # 'N' - 47: 1, # 'O' - 46: 2, # 'P' - 43: 2, # 'R' - 33: 2, # 'S' - 37: 2, # 'T' - 57: 1, # 'U' - 48: 1, # 'V' - 55: 1, # 'Y' - 52: 2, # 'Z' - 2: 0, # 'a' - 18: 1, # 'b' - 26: 1, # 'c' - 17: 2, # 'd' - 1: 1, # 'e' - 27: 1, # 'f' - 12: 1, # 'g' - 20: 1, # 'h' - 9: 1, # 'i' - 22: 1, # 'j' - 7: 2, # 'k' - 6: 2, # 'l' - 13: 2, # 'm' - 4: 2, # 'n' - 8: 0, # 'o' - 23: 2, # 'p' - 10: 2, # 'r' - 5: 1, # 's' - 3: 1, # 't' - 21: 1, # 'u' - 19: 1, # 'v' - 62: 1, # 'x' - 16: 0, # 'y' - 11: 3, # 'z' - 51: 1, # 'Á' - 44: 0, # 'É' - 61: 1, # 'Í' - 58: 0, # 'Ó' - 59: 0, # 'Ö' - 60: 0, # 'Ú' - 63: 0, # 'Ü' - 14: 0, # 'á' - 15: 0, # 'é' - 30: 0, # 'í' - 25: 0, # 'ó' - 24: 0, # 'ö' - 31: 0, # 'ú' - 29: 0, # 'ü' - 42: 0, # 'ő' - 56: 0, # 'ű' - }, - 40: { # 'B' - 28: 2, # 'A' - 40: 1, # 'B' - 54: 1, # 'C' - 45: 1, # 'D' - 32: 2, # 'E' - 50: 0, # 'F' - 49: 0, # 'G' - 38: 0, # 'H' - 39: 1, # 'I' - 53: 1, # 'J' - 36: 1, # 'K' - 41: 1, # 'L' - 34: 0, # 'M' - 35: 1, # 'N' - 47: 2, # 'O' - 46: 0, # 'P' - 43: 1, # 'R' - 33: 1, # 'S' - 37: 1, # 'T' - 57: 1, # 'U' - 48: 1, # 'V' - 55: 0, # 'Y' - 52: 0, # 'Z' - 2: 2, # 'a' - 18: 0, # 'b' - 26: 0, # 'c' - 17: 0, # 'd' - 1: 3, # 'e' - 27: 0, # 'f' - 12: 0, # 'g' - 20: 0, # 'h' - 9: 2, # 'i' - 22: 1, # 'j' - 7: 0, # 'k' - 6: 1, # 'l' - 13: 0, # 'm' - 4: 0, # 'n' - 8: 2, # 'o' - 23: 1, # 'p' - 10: 2, # 'r' - 5: 0, # 's' - 3: 0, # 't' - 21: 3, # 'u' - 19: 0, # 'v' - 62: 0, # 'x' - 16: 1, # 'y' - 11: 0, # 'z' - 51: 1, # 'Á' - 44: 1, # 'É' - 61: 1, # 'Í' - 58: 1, # 'Ó' - 59: 1, # 'Ö' - 60: 1, # 'Ú' - 63: 1, # 'Ü' - 14: 2, # 'á' - 15: 2, # 'é' - 30: 1, # 'í' - 25: 1, # 'ó' - 24: 1, # 'ö' - 31: 1, # 'ú' - 29: 1, # 'ü' - 42: 1, # 'ő' - 56: 1, # 'ű' - }, - 54: { # 'C' - 28: 1, # 'A' - 40: 1, # 'B' - 54: 1, # 'C' - 45: 1, # 'D' - 32: 1, # 'E' - 50: 0, # 'F' - 49: 0, # 'G' - 38: 1, # 'H' - 39: 2, # 'I' - 53: 1, # 'J' - 36: 1, # 'K' - 41: 1, # 'L' - 34: 1, # 'M' - 35: 0, # 'N' - 47: 1, # 'O' - 46: 1, # 'P' - 43: 1, # 'R' - 33: 2, # 'S' - 37: 1, # 'T' - 57: 1, # 'U' - 48: 0, # 'V' - 55: 1, # 'Y' - 52: 1, # 'Z' - 2: 2, # 'a' - 18: 0, # 'b' - 26: 0, # 'c' - 17: 0, # 'd' - 1: 1, # 'e' - 27: 0, # 'f' - 12: 0, # 'g' - 20: 1, # 'h' - 9: 1, # 'i' - 22: 0, # 'j' - 7: 0, # 'k' - 6: 1, # 'l' - 13: 0, # 'm' - 4: 0, # 'n' - 8: 2, # 'o' - 23: 0, # 'p' - 10: 1, # 'r' - 5: 3, # 's' - 3: 0, # 't' - 21: 1, # 'u' - 19: 0, # 'v' - 62: 0, # 'x' - 16: 1, # 'y' - 11: 1, # 'z' - 51: 1, # 'Á' - 44: 1, # 'É' - 61: 1, # 'Í' - 58: 0, # 'Ó' - 59: 0, # 'Ö' - 60: 0, # 'Ú' - 63: 0, # 'Ü' - 14: 1, # 'á' - 15: 1, # 'é' - 30: 1, # 'í' - 25: 1, # 'ó' - 24: 0, # 'ö' - 31: 0, # 'ú' - 29: 0, # 'ü' - 42: 0, # 'ő' - 56: 0, # 'ű' - }, - 45: { # 'D' - 28: 2, # 'A' - 40: 1, # 'B' - 54: 0, # 'C' - 45: 1, # 'D' - 32: 2, # 'E' - 50: 1, # 'F' - 49: 1, # 'G' - 38: 1, # 'H' - 39: 2, # 'I' - 53: 1, # 'J' - 36: 1, # 'K' - 41: 0, # 'L' - 34: 1, # 'M' - 35: 1, # 'N' - 47: 2, # 'O' - 46: 0, # 'P' - 43: 1, # 'R' - 33: 1, # 'S' - 37: 1, # 'T' - 57: 1, # 'U' - 48: 1, # 'V' - 55: 1, # 'Y' - 52: 1, # 'Z' - 2: 2, # 'a' - 18: 0, # 'b' - 26: 0, # 'c' - 17: 0, # 'd' - 1: 3, # 'e' - 27: 0, # 'f' - 12: 0, # 'g' - 20: 0, # 'h' - 9: 1, # 'i' - 22: 0, # 'j' - 7: 0, # 'k' - 6: 0, # 'l' - 13: 0, # 'm' - 4: 0, # 'n' - 8: 1, # 'o' - 23: 0, # 'p' - 10: 2, # 'r' - 5: 0, # 's' - 3: 0, # 't' - 21: 2, # 'u' - 19: 0, # 'v' - 62: 0, # 'x' - 16: 1, # 'y' - 11: 1, # 'z' - 51: 1, # 'Á' - 44: 1, # 'É' - 61: 1, # 'Í' - 58: 1, # 'Ó' - 59: 1, # 'Ö' - 60: 1, # 'Ú' - 63: 1, # 'Ü' - 14: 1, # 'á' - 15: 1, # 'é' - 30: 1, # 'í' - 25: 1, # 'ó' - 24: 1, # 'ö' - 31: 1, # 'ú' - 29: 1, # 'ü' - 42: 1, # 'ő' - 56: 0, # 'ű' - }, - 32: { # 'E' - 28: 1, # 'A' - 40: 1, # 'B' - 54: 1, # 'C' - 45: 1, # 'D' - 32: 1, # 'E' - 50: 1, # 'F' - 49: 2, # 'G' - 38: 1, # 'H' - 39: 1, # 'I' - 53: 1, # 'J' - 36: 2, # 'K' - 41: 2, # 'L' - 34: 2, # 'M' - 35: 2, # 'N' - 47: 1, # 'O' - 46: 1, # 'P' - 43: 2, # 'R' - 33: 2, # 'S' - 37: 2, # 'T' - 57: 1, # 'U' - 48: 1, # 'V' - 55: 1, # 'Y' - 52: 1, # 'Z' - 2: 1, # 'a' - 18: 1, # 'b' - 26: 1, # 'c' - 17: 2, # 'd' - 1: 1, # 'e' - 27: 1, # 'f' - 12: 3, # 'g' - 20: 1, # 'h' - 9: 1, # 'i' - 22: 1, # 'j' - 7: 1, # 'k' - 6: 2, # 'l' - 13: 2, # 'm' - 4: 2, # 'n' - 8: 0, # 'o' - 23: 1, # 'p' - 10: 2, # 'r' - 5: 2, # 's' - 3: 1, # 't' - 21: 2, # 'u' - 19: 1, # 'v' - 62: 1, # 'x' - 16: 0, # 'y' - 11: 3, # 'z' - 51: 1, # 'Á' - 44: 1, # 'É' - 61: 0, # 'Í' - 58: 1, # 'Ó' - 59: 1, # 'Ö' - 60: 0, # 'Ú' - 63: 1, # 'Ü' - 14: 0, # 'á' - 15: 0, # 'é' - 30: 0, # 'í' - 25: 0, # 'ó' - 24: 1, # 'ö' - 31: 0, # 'ú' - 29: 0, # 'ü' - 42: 0, # 'ő' - 56: 0, # 'ű' - }, - 50: { # 'F' - 28: 1, # 'A' - 40: 0, # 'B' - 54: 0, # 'C' - 45: 0, # 'D' - 32: 1, # 'E' - 50: 1, # 'F' - 49: 0, # 'G' - 38: 1, # 'H' - 39: 1, # 'I' - 53: 1, # 'J' - 36: 1, # 'K' - 41: 1, # 'L' - 34: 1, # 'M' - 35: 1, # 'N' - 47: 1, # 'O' - 46: 0, # 'P' - 43: 1, # 'R' - 33: 0, # 'S' - 37: 1, # 'T' - 57: 1, # 'U' - 48: 0, # 'V' - 55: 1, # 'Y' - 52: 0, # 'Z' - 2: 2, # 'a' - 18: 0, # 'b' - 26: 0, # 'c' - 17: 0, # 'd' - 1: 2, # 'e' - 27: 1, # 'f' - 12: 0, # 'g' - 20: 0, # 'h' - 9: 2, # 'i' - 22: 1, # 'j' - 7: 0, # 'k' - 6: 1, # 'l' - 13: 0, # 'm' - 4: 0, # 'n' - 8: 2, # 'o' - 23: 0, # 'p' - 10: 2, # 'r' - 5: 0, # 's' - 3: 0, # 't' - 21: 1, # 'u' - 19: 0, # 'v' - 62: 0, # 'x' - 16: 0, # 'y' - 11: 0, # 'z' - 51: 1, # 'Á' - 44: 1, # 'É' - 61: 0, # 'Í' - 58: 1, # 'Ó' - 59: 1, # 'Ö' - 60: 0, # 'Ú' - 63: 1, # 'Ü' - 14: 1, # 'á' - 15: 1, # 'é' - 30: 0, # 'í' - 25: 0, # 'ó' - 24: 2, # 'ö' - 31: 1, # 'ú' - 29: 1, # 'ü' - 42: 1, # 'ő' - 56: 1, # 'ű' - }, - 49: { # 'G' - 28: 2, # 'A' - 40: 1, # 'B' - 54: 1, # 'C' - 45: 1, # 'D' - 32: 2, # 'E' - 50: 1, # 'F' - 49: 1, # 'G' - 38: 1, # 'H' - 39: 1, # 'I' - 53: 1, # 'J' - 36: 1, # 'K' - 41: 1, # 'L' - 34: 1, # 'M' - 35: 1, # 'N' - 47: 1, # 'O' - 46: 1, # 'P' - 43: 1, # 'R' - 33: 1, # 'S' - 37: 1, # 'T' - 57: 1, # 'U' - 48: 1, # 'V' - 55: 2, # 'Y' - 52: 1, # 'Z' - 2: 2, # 'a' - 18: 0, # 'b' - 26: 0, # 'c' - 17: 0, # 'd' - 1: 2, # 'e' - 27: 0, # 'f' - 12: 0, # 'g' - 20: 0, # 'h' - 9: 1, # 'i' - 22: 0, # 'j' - 7: 0, # 'k' - 6: 1, # 'l' - 13: 0, # 'm' - 4: 0, # 'n' - 8: 2, # 'o' - 23: 0, # 'p' - 10: 2, # 'r' - 5: 0, # 's' - 3: 0, # 't' - 21: 1, # 'u' - 19: 0, # 'v' - 62: 0, # 'x' - 16: 2, # 'y' - 11: 0, # 'z' - 51: 1, # 'Á' - 44: 1, # 'É' - 61: 1, # 'Í' - 58: 1, # 'Ó' - 59: 1, # 'Ö' - 60: 1, # 'Ú' - 63: 1, # 'Ü' - 14: 1, # 'á' - 15: 1, # 'é' - 30: 0, # 'í' - 25: 1, # 'ó' - 24: 1, # 'ö' - 31: 1, # 'ú' - 29: 1, # 'ü' - 42: 1, # 'ő' - 56: 0, # 'ű' - }, - 38: { # 'H' - 28: 2, # 'A' - 40: 1, # 'B' - 54: 1, # 'C' - 45: 0, # 'D' - 32: 1, # 'E' - 50: 0, # 'F' - 49: 0, # 'G' - 38: 0, # 'H' - 39: 1, # 'I' - 53: 0, # 'J' - 36: 0, # 'K' - 41: 1, # 'L' - 34: 0, # 'M' - 35: 0, # 'N' - 47: 1, # 'O' - 46: 0, # 'P' - 43: 1, # 'R' - 33: 1, # 'S' - 37: 1, # 'T' - 57: 1, # 'U' - 48: 0, # 'V' - 55: 1, # 'Y' - 52: 0, # 'Z' - 2: 3, # 'a' - 18: 0, # 'b' - 26: 0, # 'c' - 17: 0, # 'd' - 1: 2, # 'e' - 27: 0, # 'f' - 12: 0, # 'g' - 20: 0, # 'h' - 9: 2, # 'i' - 22: 1, # 'j' - 7: 0, # 'k' - 6: 1, # 'l' - 13: 1, # 'm' - 4: 0, # 'n' - 8: 3, # 'o' - 23: 0, # 'p' - 10: 1, # 'r' - 5: 0, # 's' - 3: 0, # 't' - 21: 2, # 'u' - 19: 0, # 'v' - 62: 0, # 'x' - 16: 1, # 'y' - 11: 0, # 'z' - 51: 2, # 'Á' - 44: 2, # 'É' - 61: 1, # 'Í' - 58: 1, # 'Ó' - 59: 1, # 'Ö' - 60: 1, # 'Ú' - 63: 1, # 'Ü' - 14: 2, # 'á' - 15: 1, # 'é' - 30: 2, # 'í' - 25: 1, # 'ó' - 24: 1, # 'ö' - 31: 1, # 'ú' - 29: 1, # 'ü' - 42: 1, # 'ő' - 56: 1, # 'ű' - }, - 39: { # 'I' - 28: 2, # 'A' - 40: 1, # 'B' - 54: 1, # 'C' - 45: 1, # 'D' - 32: 1, # 'E' - 50: 1, # 'F' - 49: 1, # 'G' - 38: 1, # 'H' - 39: 2, # 'I' - 53: 1, # 'J' - 36: 2, # 'K' - 41: 2, # 'L' - 34: 1, # 'M' - 35: 2, # 'N' - 47: 1, # 'O' - 46: 1, # 'P' - 43: 1, # 'R' - 33: 2, # 'S' - 37: 1, # 'T' - 57: 1, # 'U' - 48: 1, # 'V' - 55: 0, # 'Y' - 52: 2, # 'Z' - 2: 0, # 'a' - 18: 1, # 'b' - 26: 1, # 'c' - 17: 2, # 'd' - 1: 0, # 'e' - 27: 1, # 'f' - 12: 2, # 'g' - 20: 1, # 'h' - 9: 0, # 'i' - 22: 1, # 'j' - 7: 1, # 'k' - 6: 2, # 'l' - 13: 2, # 'm' - 4: 1, # 'n' - 8: 0, # 'o' - 23: 1, # 'p' - 10: 2, # 'r' - 5: 2, # 's' - 3: 2, # 't' - 21: 0, # 'u' - 19: 1, # 'v' - 62: 0, # 'x' - 16: 0, # 'y' - 11: 1, # 'z' - 51: 1, # 'Á' - 44: 1, # 'É' - 61: 0, # 'Í' - 58: 1, # 'Ó' - 59: 1, # 'Ö' - 60: 1, # 'Ú' - 63: 1, # 'Ü' - 14: 0, # 'á' - 15: 0, # 'é' - 30: 0, # 'í' - 25: 0, # 'ó' - 24: 0, # 'ö' - 31: 0, # 'ú' - 29: 0, # 'ü' - 42: 0, # 'ő' - 56: 0, # 'ű' - }, - 53: { # 'J' - 28: 2, # 'A' - 40: 0, # 'B' - 54: 1, # 'C' - 45: 1, # 'D' - 32: 2, # 'E' - 50: 0, # 'F' - 49: 0, # 'G' - 38: 1, # 'H' - 39: 1, # 'I' - 53: 1, # 'J' - 36: 1, # 'K' - 41: 1, # 'L' - 34: 1, # 'M' - 35: 1, # 'N' - 47: 1, # 'O' - 46: 0, # 'P' - 43: 0, # 'R' - 33: 1, # 'S' - 37: 1, # 'T' - 57: 1, # 'U' - 48: 0, # 'V' - 55: 0, # 'Y' - 52: 1, # 'Z' - 2: 2, # 'a' - 18: 0, # 'b' - 26: 0, # 'c' - 17: 0, # 'd' - 1: 2, # 'e' - 27: 0, # 'f' - 12: 0, # 'g' - 20: 0, # 'h' - 9: 1, # 'i' - 22: 0, # 'j' - 7: 0, # 'k' - 6: 0, # 'l' - 13: 0, # 'm' - 4: 0, # 'n' - 8: 1, # 'o' - 23: 0, # 'p' - 10: 0, # 'r' - 5: 0, # 's' - 3: 0, # 't' - 21: 2, # 'u' - 19: 0, # 'v' - 62: 0, # 'x' - 16: 0, # 'y' - 11: 0, # 'z' - 51: 1, # 'Á' - 44: 1, # 'É' - 61: 0, # 'Í' - 58: 1, # 'Ó' - 59: 1, # 'Ö' - 60: 1, # 'Ú' - 63: 1, # 'Ü' - 14: 2, # 'á' - 15: 1, # 'é' - 30: 0, # 'í' - 25: 2, # 'ó' - 24: 2, # 'ö' - 31: 1, # 'ú' - 29: 0, # 'ü' - 42: 1, # 'ő' - 56: 0, # 'ű' - }, - 36: { # 'K' - 28: 2, # 'A' - 40: 1, # 'B' - 54: 1, # 'C' - 45: 1, # 'D' - 32: 2, # 'E' - 50: 1, # 'F' - 49: 0, # 'G' - 38: 1, # 'H' - 39: 2, # 'I' - 53: 1, # 'J' - 36: 1, # 'K' - 41: 1, # 'L' - 34: 1, # 'M' - 35: 1, # 'N' - 47: 2, # 'O' - 46: 0, # 'P' - 43: 1, # 'R' - 33: 1, # 'S' - 37: 1, # 'T' - 57: 1, # 'U' - 48: 1, # 'V' - 55: 1, # 'Y' - 52: 0, # 'Z' - 2: 2, # 'a' - 18: 0, # 'b' - 26: 0, # 'c' - 17: 0, # 'd' - 1: 2, # 'e' - 27: 1, # 'f' - 12: 0, # 'g' - 20: 1, # 'h' - 9: 3, # 'i' - 22: 0, # 'j' - 7: 0, # 'k' - 6: 1, # 'l' - 13: 1, # 'm' - 4: 1, # 'n' - 8: 2, # 'o' - 23: 0, # 'p' - 10: 2, # 'r' - 5: 0, # 's' - 3: 0, # 't' - 21: 1, # 'u' - 19: 1, # 'v' - 62: 0, # 'x' - 16: 1, # 'y' - 11: 0, # 'z' - 51: 1, # 'Á' - 44: 1, # 'É' - 61: 1, # 'Í' - 58: 1, # 'Ó' - 59: 2, # 'Ö' - 60: 1, # 'Ú' - 63: 1, # 'Ü' - 14: 2, # 'á' - 15: 2, # 'é' - 30: 1, # 'í' - 25: 1, # 'ó' - 24: 2, # 'ö' - 31: 1, # 'ú' - 29: 2, # 'ü' - 42: 1, # 'ő' - 56: 0, # 'ű' - }, - 41: { # 'L' - 28: 2, # 'A' - 40: 1, # 'B' - 54: 1, # 'C' - 45: 1, # 'D' - 32: 2, # 'E' - 50: 1, # 'F' - 49: 1, # 'G' - 38: 1, # 'H' - 39: 2, # 'I' - 53: 1, # 'J' - 36: 1, # 'K' - 41: 2, # 'L' - 34: 1, # 'M' - 35: 1, # 'N' - 47: 2, # 'O' - 46: 0, # 'P' - 43: 1, # 'R' - 33: 1, # 'S' - 37: 2, # 'T' - 57: 1, # 'U' - 48: 1, # 'V' - 55: 1, # 'Y' - 52: 1, # 'Z' - 2: 2, # 'a' - 18: 0, # 'b' - 26: 0, # 'c' - 17: 0, # 'd' - 1: 3, # 'e' - 27: 0, # 'f' - 12: 0, # 'g' - 20: 0, # 'h' - 9: 2, # 'i' - 22: 1, # 'j' - 7: 0, # 'k' - 6: 1, # 'l' - 13: 0, # 'm' - 4: 0, # 'n' - 8: 2, # 'o' - 23: 0, # 'p' - 10: 0, # 'r' - 5: 0, # 's' - 3: 0, # 't' - 21: 2, # 'u' - 19: 0, # 'v' - 62: 0, # 'x' - 16: 1, # 'y' - 11: 0, # 'z' - 51: 2, # 'Á' - 44: 1, # 'É' - 61: 1, # 'Í' - 58: 1, # 'Ó' - 59: 1, # 'Ö' - 60: 1, # 'Ú' - 63: 1, # 'Ü' - 14: 2, # 'á' - 15: 1, # 'é' - 30: 1, # 'í' - 25: 1, # 'ó' - 24: 1, # 'ö' - 31: 0, # 'ú' - 29: 1, # 'ü' - 42: 0, # 'ő' - 56: 0, # 'ű' - }, - 34: { # 'M' - 28: 2, # 'A' - 40: 1, # 'B' - 54: 0, # 'C' - 45: 0, # 'D' - 32: 2, # 'E' - 50: 1, # 'F' - 49: 0, # 'G' - 38: 1, # 'H' - 39: 2, # 'I' - 53: 1, # 'J' - 36: 1, # 'K' - 41: 1, # 'L' - 34: 1, # 'M' - 35: 1, # 'N' - 47: 1, # 'O' - 46: 1, # 'P' - 43: 1, # 'R' - 33: 1, # 'S' - 37: 1, # 'T' - 57: 1, # 'U' - 48: 1, # 'V' - 55: 1, # 'Y' - 52: 1, # 'Z' - 2: 3, # 'a' - 18: 0, # 'b' - 26: 1, # 'c' - 17: 0, # 'd' - 1: 3, # 'e' - 27: 0, # 'f' - 12: 0, # 'g' - 20: 0, # 'h' - 9: 3, # 'i' - 22: 0, # 'j' - 7: 0, # 'k' - 6: 0, # 'l' - 13: 1, # 'm' - 4: 1, # 'n' - 8: 3, # 'o' - 23: 0, # 'p' - 10: 1, # 'r' - 5: 0, # 's' - 3: 0, # 't' - 21: 2, # 'u' - 19: 0, # 'v' - 62: 0, # 'x' - 16: 1, # 'y' - 11: 0, # 'z' - 51: 2, # 'Á' - 44: 1, # 'É' - 61: 1, # 'Í' - 58: 1, # 'Ó' - 59: 1, # 'Ö' - 60: 1, # 'Ú' - 63: 1, # 'Ü' - 14: 2, # 'á' - 15: 2, # 'é' - 30: 1, # 'í' - 25: 1, # 'ó' - 24: 1, # 'ö' - 31: 1, # 'ú' - 29: 1, # 'ü' - 42: 0, # 'ő' - 56: 1, # 'ű' - }, - 35: { # 'N' - 28: 2, # 'A' - 40: 1, # 'B' - 54: 1, # 'C' - 45: 2, # 'D' - 32: 2, # 'E' - 50: 1, # 'F' - 49: 1, # 'G' - 38: 1, # 'H' - 39: 1, # 'I' - 53: 1, # 'J' - 36: 1, # 'K' - 41: 1, # 'L' - 34: 1, # 'M' - 35: 1, # 'N' - 47: 1, # 'O' - 46: 1, # 'P' - 43: 1, # 'R' - 33: 1, # 'S' - 37: 2, # 'T' - 57: 1, # 'U' - 48: 1, # 'V' - 55: 2, # 'Y' - 52: 1, # 'Z' - 2: 3, # 'a' - 18: 0, # 'b' - 26: 0, # 'c' - 17: 0, # 'd' - 1: 3, # 'e' - 27: 0, # 'f' - 12: 0, # 'g' - 20: 0, # 'h' - 9: 2, # 'i' - 22: 0, # 'j' - 7: 0, # 'k' - 6: 0, # 'l' - 13: 0, # 'm' - 4: 1, # 'n' - 8: 2, # 'o' - 23: 0, # 'p' - 10: 0, # 'r' - 5: 0, # 's' - 3: 0, # 't' - 21: 1, # 'u' - 19: 0, # 'v' - 62: 0, # 'x' - 16: 2, # 'y' - 11: 0, # 'z' - 51: 1, # 'Á' - 44: 1, # 'É' - 61: 1, # 'Í' - 58: 1, # 'Ó' - 59: 1, # 'Ö' - 60: 1, # 'Ú' - 63: 1, # 'Ü' - 14: 1, # 'á' - 15: 2, # 'é' - 30: 1, # 'í' - 25: 1, # 'ó' - 24: 1, # 'ö' - 31: 0, # 'ú' - 29: 0, # 'ü' - 42: 1, # 'ő' - 56: 0, # 'ű' - }, - 47: { # 'O' - 28: 1, # 'A' - 40: 1, # 'B' - 54: 1, # 'C' - 45: 1, # 'D' - 32: 1, # 'E' - 50: 1, # 'F' - 49: 1, # 'G' - 38: 1, # 'H' - 39: 1, # 'I' - 53: 1, # 'J' - 36: 2, # 'K' - 41: 2, # 'L' - 34: 2, # 'M' - 35: 2, # 'N' - 47: 1, # 'O' - 46: 1, # 'P' - 43: 2, # 'R' - 33: 2, # 'S' - 37: 2, # 'T' - 57: 1, # 'U' - 48: 1, # 'V' - 55: 1, # 'Y' - 52: 1, # 'Z' - 2: 0, # 'a' - 18: 1, # 'b' - 26: 1, # 'c' - 17: 1, # 'd' - 1: 1, # 'e' - 27: 1, # 'f' - 12: 1, # 'g' - 20: 1, # 'h' - 9: 1, # 'i' - 22: 1, # 'j' - 7: 2, # 'k' - 6: 2, # 'l' - 13: 1, # 'm' - 4: 1, # 'n' - 8: 1, # 'o' - 23: 1, # 'p' - 10: 2, # 'r' - 5: 1, # 's' - 3: 2, # 't' - 21: 1, # 'u' - 19: 0, # 'v' - 62: 1, # 'x' - 16: 0, # 'y' - 11: 1, # 'z' - 51: 1, # 'Á' - 44: 1, # 'É' - 61: 0, # 'Í' - 58: 1, # 'Ó' - 59: 0, # 'Ö' - 60: 0, # 'Ú' - 63: 0, # 'Ü' - 14: 0, # 'á' - 15: 0, # 'é' - 30: 0, # 'í' - 25: 0, # 'ó' - 24: 0, # 'ö' - 31: 0, # 'ú' - 29: 0, # 'ü' - 42: 0, # 'ő' - 56: 0, # 'ű' - }, - 46: { # 'P' - 28: 1, # 'A' - 40: 1, # 'B' - 54: 1, # 'C' - 45: 1, # 'D' - 32: 1, # 'E' - 50: 1, # 'F' - 49: 1, # 'G' - 38: 1, # 'H' - 39: 1, # 'I' - 53: 1, # 'J' - 36: 1, # 'K' - 41: 1, # 'L' - 34: 0, # 'M' - 35: 1, # 'N' - 47: 1, # 'O' - 46: 1, # 'P' - 43: 2, # 'R' - 33: 1, # 'S' - 37: 1, # 'T' - 57: 1, # 'U' - 48: 1, # 'V' - 55: 0, # 'Y' - 52: 1, # 'Z' - 2: 2, # 'a' - 18: 0, # 'b' - 26: 0, # 'c' - 17: 0, # 'd' - 1: 2, # 'e' - 27: 1, # 'f' - 12: 0, # 'g' - 20: 1, # 'h' - 9: 2, # 'i' - 22: 0, # 'j' - 7: 0, # 'k' - 6: 1, # 'l' - 13: 0, # 'm' - 4: 1, # 'n' - 8: 2, # 'o' - 23: 0, # 'p' - 10: 2, # 'r' - 5: 1, # 's' - 3: 0, # 't' - 21: 1, # 'u' - 19: 0, # 'v' - 62: 0, # 'x' - 16: 1, # 'y' - 11: 0, # 'z' - 51: 2, # 'Á' - 44: 1, # 'É' - 61: 1, # 'Í' - 58: 1, # 'Ó' - 59: 1, # 'Ö' - 60: 0, # 'Ú' - 63: 1, # 'Ü' - 14: 3, # 'á' - 15: 2, # 'é' - 30: 0, # 'í' - 25: 1, # 'ó' - 24: 1, # 'ö' - 31: 0, # 'ú' - 29: 1, # 'ü' - 42: 1, # 'ő' - 56: 0, # 'ű' - }, - 43: { # 'R' - 28: 2, # 'A' - 40: 1, # 'B' - 54: 1, # 'C' - 45: 1, # 'D' - 32: 2, # 'E' - 50: 1, # 'F' - 49: 1, # 'G' - 38: 1, # 'H' - 39: 2, # 'I' - 53: 1, # 'J' - 36: 1, # 'K' - 41: 1, # 'L' - 34: 1, # 'M' - 35: 1, # 'N' - 47: 2, # 'O' - 46: 1, # 'P' - 43: 1, # 'R' - 33: 2, # 'S' - 37: 2, # 'T' - 57: 1, # 'U' - 48: 1, # 'V' - 55: 1, # 'Y' - 52: 1, # 'Z' - 2: 2, # 'a' - 18: 0, # 'b' - 26: 0, # 'c' - 17: 0, # 'd' - 1: 2, # 'e' - 27: 0, # 'f' - 12: 0, # 'g' - 20: 1, # 'h' - 9: 2, # 'i' - 22: 0, # 'j' - 7: 0, # 'k' - 6: 0, # 'l' - 13: 0, # 'm' - 4: 0, # 'n' - 8: 2, # 'o' - 23: 0, # 'p' - 10: 0, # 'r' - 5: 0, # 's' - 3: 0, # 't' - 21: 1, # 'u' - 19: 0, # 'v' - 62: 0, # 'x' - 16: 1, # 'y' - 11: 0, # 'z' - 51: 2, # 'Á' - 44: 1, # 'É' - 61: 1, # 'Í' - 58: 2, # 'Ó' - 59: 1, # 'Ö' - 60: 1, # 'Ú' - 63: 1, # 'Ü' - 14: 2, # 'á' - 15: 2, # 'é' - 30: 1, # 'í' - 25: 2, # 'ó' - 24: 1, # 'ö' - 31: 1, # 'ú' - 29: 1, # 'ü' - 42: 0, # 'ő' - 56: 0, # 'ű' - }, - 33: { # 'S' - 28: 2, # 'A' - 40: 1, # 'B' - 54: 1, # 'C' - 45: 1, # 'D' - 32: 2, # 'E' - 50: 1, # 'F' - 49: 1, # 'G' - 38: 1, # 'H' - 39: 2, # 'I' - 53: 1, # 'J' - 36: 1, # 'K' - 41: 1, # 'L' - 34: 1, # 'M' - 35: 1, # 'N' - 47: 2, # 'O' - 46: 1, # 'P' - 43: 1, # 'R' - 33: 2, # 'S' - 37: 2, # 'T' - 57: 1, # 'U' - 48: 1, # 'V' - 55: 1, # 'Y' - 52: 3, # 'Z' - 2: 2, # 'a' - 18: 0, # 'b' - 26: 1, # 'c' - 17: 0, # 'd' - 1: 2, # 'e' - 27: 0, # 'f' - 12: 0, # 'g' - 20: 1, # 'h' - 9: 2, # 'i' - 22: 0, # 'j' - 7: 1, # 'k' - 6: 1, # 'l' - 13: 1, # 'm' - 4: 0, # 'n' - 8: 2, # 'o' - 23: 1, # 'p' - 10: 0, # 'r' - 5: 0, # 's' - 3: 1, # 't' - 21: 1, # 'u' - 19: 1, # 'v' - 62: 0, # 'x' - 16: 1, # 'y' - 11: 3, # 'z' - 51: 2, # 'Á' - 44: 1, # 'É' - 61: 1, # 'Í' - 58: 1, # 'Ó' - 59: 1, # 'Ö' - 60: 1, # 'Ú' - 63: 1, # 'Ü' - 14: 2, # 'á' - 15: 1, # 'é' - 30: 1, # 'í' - 25: 1, # 'ó' - 24: 1, # 'ö' - 31: 1, # 'ú' - 29: 1, # 'ü' - 42: 1, # 'ő' - 56: 1, # 'ű' - }, - 37: { # 'T' - 28: 2, # 'A' - 40: 1, # 'B' - 54: 1, # 'C' - 45: 1, # 'D' - 32: 2, # 'E' - 50: 1, # 'F' - 49: 1, # 'G' - 38: 1, # 'H' - 39: 2, # 'I' - 53: 1, # 'J' - 36: 1, # 'K' - 41: 1, # 'L' - 34: 1, # 'M' - 35: 1, # 'N' - 47: 2, # 'O' - 46: 1, # 'P' - 43: 2, # 'R' - 33: 1, # 'S' - 37: 2, # 'T' - 57: 1, # 'U' - 48: 1, # 'V' - 55: 1, # 'Y' - 52: 1, # 'Z' - 2: 2, # 'a' - 18: 0, # 'b' - 26: 0, # 'c' - 17: 0, # 'd' - 1: 2, # 'e' - 27: 0, # 'f' - 12: 0, # 'g' - 20: 1, # 'h' - 9: 2, # 'i' - 22: 0, # 'j' - 7: 0, # 'k' - 6: 0, # 'l' - 13: 0, # 'm' - 4: 0, # 'n' - 8: 2, # 'o' - 23: 0, # 'p' - 10: 1, # 'r' - 5: 1, # 's' - 3: 0, # 't' - 21: 2, # 'u' - 19: 0, # 'v' - 62: 0, # 'x' - 16: 1, # 'y' - 11: 1, # 'z' - 51: 2, # 'Á' - 44: 2, # 'É' - 61: 1, # 'Í' - 58: 1, # 'Ó' - 59: 1, # 'Ö' - 60: 1, # 'Ú' - 63: 1, # 'Ü' - 14: 2, # 'á' - 15: 1, # 'é' - 30: 1, # 'í' - 25: 1, # 'ó' - 24: 2, # 'ö' - 31: 1, # 'ú' - 29: 1, # 'ü' - 42: 1, # 'ő' - 56: 1, # 'ű' - }, - 57: { # 'U' - 28: 1, # 'A' - 40: 1, # 'B' - 54: 1, # 'C' - 45: 1, # 'D' - 32: 1, # 'E' - 50: 1, # 'F' - 49: 1, # 'G' - 38: 1, # 'H' - 39: 1, # 'I' - 53: 1, # 'J' - 36: 1, # 'K' - 41: 1, # 'L' - 34: 1, # 'M' - 35: 1, # 'N' - 47: 1, # 'O' - 46: 1, # 'P' - 43: 1, # 'R' - 33: 2, # 'S' - 37: 1, # 'T' - 57: 0, # 'U' - 48: 1, # 'V' - 55: 0, # 'Y' - 52: 1, # 'Z' - 2: 0, # 'a' - 18: 1, # 'b' - 26: 1, # 'c' - 17: 1, # 'd' - 1: 1, # 'e' - 27: 0, # 'f' - 12: 2, # 'g' - 20: 0, # 'h' - 9: 0, # 'i' - 22: 1, # 'j' - 7: 1, # 'k' - 6: 1, # 'l' - 13: 1, # 'm' - 4: 1, # 'n' - 8: 0, # 'o' - 23: 1, # 'p' - 10: 1, # 'r' - 5: 1, # 's' - 3: 1, # 't' - 21: 0, # 'u' - 19: 0, # 'v' - 62: 0, # 'x' - 16: 0, # 'y' - 11: 1, # 'z' - 51: 0, # 'Á' - 44: 0, # 'É' - 61: 1, # 'Í' - 58: 0, # 'Ó' - 59: 0, # 'Ö' - 60: 0, # 'Ú' - 63: 0, # 'Ü' - 14: 0, # 'á' - 15: 0, # 'é' - 30: 0, # 'í' - 25: 0, # 'ó' - 24: 0, # 'ö' - 31: 0, # 'ú' - 29: 0, # 'ü' - 42: 0, # 'ő' - 56: 0, # 'ű' - }, - 48: { # 'V' - 28: 2, # 'A' - 40: 0, # 'B' - 54: 0, # 'C' - 45: 1, # 'D' - 32: 2, # 'E' - 50: 1, # 'F' - 49: 0, # 'G' - 38: 0, # 'H' - 39: 2, # 'I' - 53: 1, # 'J' - 36: 1, # 'K' - 41: 0, # 'L' - 34: 1, # 'M' - 35: 1, # 'N' - 47: 1, # 'O' - 46: 1, # 'P' - 43: 1, # 'R' - 33: 1, # 'S' - 37: 1, # 'T' - 57: 1, # 'U' - 48: 1, # 'V' - 55: 1, # 'Y' - 52: 0, # 'Z' - 2: 3, # 'a' - 18: 0, # 'b' - 26: 0, # 'c' - 17: 0, # 'd' - 1: 2, # 'e' - 27: 0, # 'f' - 12: 0, # 'g' - 20: 0, # 'h' - 9: 2, # 'i' - 22: 0, # 'j' - 7: 0, # 'k' - 6: 1, # 'l' - 13: 0, # 'm' - 4: 0, # 'n' - 8: 2, # 'o' - 23: 0, # 'p' - 10: 0, # 'r' - 5: 0, # 's' - 3: 0, # 't' - 21: 1, # 'u' - 19: 0, # 'v' - 62: 0, # 'x' - 16: 0, # 'y' - 11: 0, # 'z' - 51: 2, # 'Á' - 44: 2, # 'É' - 61: 1, # 'Í' - 58: 1, # 'Ó' - 59: 1, # 'Ö' - 60: 0, # 'Ú' - 63: 1, # 'Ü' - 14: 2, # 'á' - 15: 2, # 'é' - 30: 1, # 'í' - 25: 0, # 'ó' - 24: 1, # 'ö' - 31: 0, # 'ú' - 29: 0, # 'ü' - 42: 0, # 'ő' - 56: 0, # 'ű' - }, - 55: { # 'Y' - 28: 2, # 'A' - 40: 1, # 'B' - 54: 1, # 'C' - 45: 1, # 'D' - 32: 2, # 'E' - 50: 1, # 'F' - 49: 1, # 'G' - 38: 1, # 'H' - 39: 1, # 'I' - 53: 1, # 'J' - 36: 1, # 'K' - 41: 1, # 'L' - 34: 1, # 'M' - 35: 1, # 'N' - 47: 1, # 'O' - 46: 1, # 'P' - 43: 1, # 'R' - 33: 1, # 'S' - 37: 1, # 'T' - 57: 1, # 'U' - 48: 1, # 'V' - 55: 0, # 'Y' - 52: 2, # 'Z' - 2: 1, # 'a' - 18: 0, # 'b' - 26: 0, # 'c' - 17: 1, # 'd' - 1: 1, # 'e' - 27: 0, # 'f' - 12: 0, # 'g' - 20: 0, # 'h' - 9: 0, # 'i' - 22: 0, # 'j' - 7: 0, # 'k' - 6: 0, # 'l' - 13: 0, # 'm' - 4: 0, # 'n' - 8: 1, # 'o' - 23: 1, # 'p' - 10: 0, # 'r' - 5: 0, # 's' - 3: 0, # 't' - 21: 0, # 'u' - 19: 1, # 'v' - 62: 0, # 'x' - 16: 0, # 'y' - 11: 0, # 'z' - 51: 1, # 'Á' - 44: 1, # 'É' - 61: 1, # 'Í' - 58: 1, # 'Ó' - 59: 1, # 'Ö' - 60: 1, # 'Ú' - 63: 1, # 'Ü' - 14: 0, # 'á' - 15: 0, # 'é' - 30: 0, # 'í' - 25: 0, # 'ó' - 24: 0, # 'ö' - 31: 0, # 'ú' - 29: 0, # 'ü' - 42: 0, # 'ő' - 56: 0, # 'ű' - }, - 52: { # 'Z' - 28: 2, # 'A' - 40: 1, # 'B' - 54: 0, # 'C' - 45: 1, # 'D' - 32: 2, # 'E' - 50: 1, # 'F' - 49: 1, # 'G' - 38: 1, # 'H' - 39: 2, # 'I' - 53: 1, # 'J' - 36: 1, # 'K' - 41: 1, # 'L' - 34: 1, # 'M' - 35: 1, # 'N' - 47: 2, # 'O' - 46: 1, # 'P' - 43: 1, # 'R' - 33: 2, # 'S' - 37: 1, # 'T' - 57: 1, # 'U' - 48: 1, # 'V' - 55: 1, # 'Y' - 52: 1, # 'Z' - 2: 1, # 'a' - 18: 0, # 'b' - 26: 0, # 'c' - 17: 0, # 'd' - 1: 1, # 'e' - 27: 0, # 'f' - 12: 0, # 'g' - 20: 0, # 'h' - 9: 1, # 'i' - 22: 0, # 'j' - 7: 0, # 'k' - 6: 0, # 'l' - 13: 0, # 'm' - 4: 1, # 'n' - 8: 1, # 'o' - 23: 0, # 'p' - 10: 1, # 'r' - 5: 2, # 's' - 3: 0, # 't' - 21: 1, # 'u' - 19: 0, # 'v' - 62: 0, # 'x' - 16: 0, # 'y' - 11: 0, # 'z' - 51: 2, # 'Á' - 44: 1, # 'É' - 61: 1, # 'Í' - 58: 1, # 'Ó' - 59: 1, # 'Ö' - 60: 1, # 'Ú' - 63: 1, # 'Ü' - 14: 1, # 'á' - 15: 1, # 'é' - 30: 0, # 'í' - 25: 0, # 'ó' - 24: 1, # 'ö' - 31: 1, # 'ú' - 29: 1, # 'ü' - 42: 0, # 'ő' - 56: 0, # 'ű' - }, - 2: { # 'a' - 28: 0, # 'A' - 40: 0, # 'B' - 54: 0, # 'C' - 45: 0, # 'D' - 32: 0, # 'E' - 50: 0, # 'F' - 49: 0, # 'G' - 38: 0, # 'H' - 39: 0, # 'I' - 53: 0, # 'J' - 36: 0, # 'K' - 41: 0, # 'L' - 34: 0, # 'M' - 35: 0, # 'N' - 47: 0, # 'O' - 46: 0, # 'P' - 43: 0, # 'R' - 33: 0, # 'S' - 37: 0, # 'T' - 57: 0, # 'U' - 48: 0, # 'V' - 55: 0, # 'Y' - 52: 0, # 'Z' - 2: 1, # 'a' - 18: 3, # 'b' - 26: 3, # 'c' - 17: 3, # 'd' - 1: 2, # 'e' - 27: 2, # 'f' - 12: 3, # 'g' - 20: 3, # 'h' - 9: 3, # 'i' - 22: 3, # 'j' - 7: 3, # 'k' - 6: 3, # 'l' - 13: 3, # 'm' - 4: 3, # 'n' - 8: 2, # 'o' - 23: 3, # 'p' - 10: 3, # 'r' - 5: 3, # 's' - 3: 3, # 't' - 21: 3, # 'u' - 19: 3, # 'v' - 62: 1, # 'x' - 16: 2, # 'y' - 11: 3, # 'z' - 51: 0, # 'Á' - 44: 0, # 'É' - 61: 0, # 'Í' - 58: 0, # 'Ó' - 59: 0, # 'Ö' - 60: 0, # 'Ú' - 63: 0, # 'Ü' - 14: 1, # 'á' - 15: 1, # 'é' - 30: 1, # 'í' - 25: 1, # 'ó' - 24: 1, # 'ö' - 31: 1, # 'ú' - 29: 1, # 'ü' - 42: 0, # 'ő' - 56: 0, # 'ű' - }, - 18: { # 'b' - 28: 0, # 'A' - 40: 0, # 'B' - 54: 0, # 'C' - 45: 0, # 'D' - 32: 0, # 'E' - 50: 0, # 'F' - 49: 0, # 'G' - 38: 0, # 'H' - 39: 0, # 'I' - 53: 0, # 'J' - 36: 0, # 'K' - 41: 0, # 'L' - 34: 0, # 'M' - 35: 0, # 'N' - 47: 0, # 'O' - 46: 0, # 'P' - 43: 0, # 'R' - 33: 0, # 'S' - 37: 0, # 'T' - 57: 0, # 'U' - 48: 0, # 'V' - 55: 0, # 'Y' - 52: 0, # 'Z' - 2: 3, # 'a' - 18: 3, # 'b' - 26: 1, # 'c' - 17: 1, # 'd' - 1: 3, # 'e' - 27: 1, # 'f' - 12: 1, # 'g' - 20: 1, # 'h' - 9: 3, # 'i' - 22: 2, # 'j' - 7: 2, # 'k' - 6: 2, # 'l' - 13: 1, # 'm' - 4: 2, # 'n' - 8: 3, # 'o' - 23: 1, # 'p' - 10: 3, # 'r' - 5: 2, # 's' - 3: 1, # 't' - 21: 3, # 'u' - 19: 1, # 'v' - 62: 0, # 'x' - 16: 1, # 'y' - 11: 1, # 'z' - 51: 0, # 'Á' - 44: 0, # 'É' - 61: 0, # 'Í' - 58: 0, # 'Ó' - 59: 0, # 'Ö' - 60: 0, # 'Ú' - 63: 0, # 'Ü' - 14: 3, # 'á' - 15: 3, # 'é' - 30: 2, # 'í' - 25: 3, # 'ó' - 24: 2, # 'ö' - 31: 2, # 'ú' - 29: 2, # 'ü' - 42: 2, # 'ő' - 56: 1, # 'ű' - }, - 26: { # 'c' - 28: 0, # 'A' - 40: 0, # 'B' - 54: 1, # 'C' - 45: 0, # 'D' - 32: 0, # 'E' - 50: 0, # 'F' - 49: 1, # 'G' - 38: 0, # 'H' - 39: 0, # 'I' - 53: 0, # 'J' - 36: 0, # 'K' - 41: 0, # 'L' - 34: 0, # 'M' - 35: 0, # 'N' - 47: 0, # 'O' - 46: 0, # 'P' - 43: 0, # 'R' - 33: 0, # 'S' - 37: 0, # 'T' - 57: 0, # 'U' - 48: 0, # 'V' - 55: 0, # 'Y' - 52: 0, # 'Z' - 2: 2, # 'a' - 18: 1, # 'b' - 26: 2, # 'c' - 17: 1, # 'd' - 1: 3, # 'e' - 27: 1, # 'f' - 12: 1, # 'g' - 20: 3, # 'h' - 9: 3, # 'i' - 22: 1, # 'j' - 7: 2, # 'k' - 6: 1, # 'l' - 13: 1, # 'm' - 4: 1, # 'n' - 8: 3, # 'o' - 23: 1, # 'p' - 10: 2, # 'r' - 5: 3, # 's' - 3: 2, # 't' - 21: 2, # 'u' - 19: 1, # 'v' - 62: 0, # 'x' - 16: 1, # 'y' - 11: 2, # 'z' - 51: 0, # 'Á' - 44: 0, # 'É' - 61: 0, # 'Í' - 58: 0, # 'Ó' - 59: 0, # 'Ö' - 60: 0, # 'Ú' - 63: 0, # 'Ü' - 14: 2, # 'á' - 15: 2, # 'é' - 30: 2, # 'í' - 25: 1, # 'ó' - 24: 1, # 'ö' - 31: 1, # 'ú' - 29: 1, # 'ü' - 42: 0, # 'ő' - 56: 0, # 'ű' - }, - 17: { # 'd' - 28: 0, # 'A' - 40: 0, # 'B' - 54: 0, # 'C' - 45: 0, # 'D' - 32: 0, # 'E' - 50: 0, # 'F' - 49: 0, # 'G' - 38: 0, # 'H' - 39: 0, # 'I' - 53: 0, # 'J' - 36: 0, # 'K' - 41: 0, # 'L' - 34: 0, # 'M' - 35: 0, # 'N' - 47: 0, # 'O' - 46: 0, # 'P' - 43: 0, # 'R' - 33: 0, # 'S' - 37: 0, # 'T' - 57: 0, # 'U' - 48: 0, # 'V' - 55: 0, # 'Y' - 52: 0, # 'Z' - 2: 3, # 'a' - 18: 2, # 'b' - 26: 1, # 'c' - 17: 2, # 'd' - 1: 3, # 'e' - 27: 1, # 'f' - 12: 1, # 'g' - 20: 2, # 'h' - 9: 3, # 'i' - 22: 3, # 'j' - 7: 2, # 'k' - 6: 1, # 'l' - 13: 2, # 'm' - 4: 3, # 'n' - 8: 3, # 'o' - 23: 1, # 'p' - 10: 3, # 'r' - 5: 3, # 's' - 3: 3, # 't' - 21: 3, # 'u' - 19: 3, # 'v' - 62: 0, # 'x' - 16: 2, # 'y' - 11: 2, # 'z' - 51: 0, # 'Á' - 44: 0, # 'É' - 61: 0, # 'Í' - 58: 0, # 'Ó' - 59: 0, # 'Ö' - 60: 0, # 'Ú' - 63: 0, # 'Ü' - 14: 3, # 'á' - 15: 3, # 'é' - 30: 3, # 'í' - 25: 3, # 'ó' - 24: 3, # 'ö' - 31: 2, # 'ú' - 29: 2, # 'ü' - 42: 2, # 'ő' - 56: 1, # 'ű' - }, - 1: { # 'e' - 28: 0, # 'A' - 40: 0, # 'B' - 54: 0, # 'C' - 45: 0, # 'D' - 32: 0, # 'E' - 50: 0, # 'F' - 49: 0, # 'G' - 38: 0, # 'H' - 39: 0, # 'I' - 53: 0, # 'J' - 36: 0, # 'K' - 41: 0, # 'L' - 34: 0, # 'M' - 35: 0, # 'N' - 47: 0, # 'O' - 46: 0, # 'P' - 43: 0, # 'R' - 33: 0, # 'S' - 37: 0, # 'T' - 57: 0, # 'U' - 48: 0, # 'V' - 55: 0, # 'Y' - 52: 0, # 'Z' - 2: 2, # 'a' - 18: 3, # 'b' - 26: 3, # 'c' - 17: 3, # 'd' - 1: 2, # 'e' - 27: 3, # 'f' - 12: 3, # 'g' - 20: 3, # 'h' - 9: 3, # 'i' - 22: 3, # 'j' - 7: 3, # 'k' - 6: 3, # 'l' - 13: 3, # 'm' - 4: 3, # 'n' - 8: 2, # 'o' - 23: 3, # 'p' - 10: 3, # 'r' - 5: 3, # 's' - 3: 3, # 't' - 21: 2, # 'u' - 19: 3, # 'v' - 62: 2, # 'x' - 16: 2, # 'y' - 11: 3, # 'z' - 51: 0, # 'Á' - 44: 0, # 'É' - 61: 0, # 'Í' - 58: 0, # 'Ó' - 59: 0, # 'Ö' - 60: 0, # 'Ú' - 63: 0, # 'Ü' - 14: 3, # 'á' - 15: 1, # 'é' - 30: 1, # 'í' - 25: 1, # 'ó' - 24: 1, # 'ö' - 31: 1, # 'ú' - 29: 1, # 'ü' - 42: 0, # 'ő' - 56: 0, # 'ű' - }, - 27: { # 'f' - 28: 0, # 'A' - 40: 0, # 'B' - 54: 0, # 'C' - 45: 0, # 'D' - 32: 0, # 'E' - 50: 0, # 'F' - 49: 0, # 'G' - 38: 0, # 'H' - 39: 0, # 'I' - 53: 0, # 'J' - 36: 0, # 'K' - 41: 0, # 'L' - 34: 0, # 'M' - 35: 0, # 'N' - 47: 0, # 'O' - 46: 0, # 'P' - 43: 0, # 'R' - 33: 0, # 'S' - 37: 0, # 'T' - 57: 0, # 'U' - 48: 0, # 'V' - 55: 0, # 'Y' - 52: 0, # 'Z' - 2: 3, # 'a' - 18: 1, # 'b' - 26: 1, # 'c' - 17: 1, # 'd' - 1: 3, # 'e' - 27: 2, # 'f' - 12: 1, # 'g' - 20: 1, # 'h' - 9: 3, # 'i' - 22: 2, # 'j' - 7: 1, # 'k' - 6: 1, # 'l' - 13: 1, # 'm' - 4: 1, # 'n' - 8: 3, # 'o' - 23: 0, # 'p' - 10: 3, # 'r' - 5: 1, # 's' - 3: 1, # 't' - 21: 2, # 'u' - 19: 1, # 'v' - 62: 0, # 'x' - 16: 1, # 'y' - 11: 0, # 'z' - 51: 0, # 'Á' - 44: 0, # 'É' - 61: 0, # 'Í' - 58: 0, # 'Ó' - 59: 0, # 'Ö' - 60: 0, # 'Ú' - 63: 0, # 'Ü' - 14: 3, # 'á' - 15: 3, # 'é' - 30: 1, # 'í' - 25: 1, # 'ó' - 24: 3, # 'ö' - 31: 1, # 'ú' - 29: 2, # 'ü' - 42: 1, # 'ő' - 56: 1, # 'ű' - }, - 12: { # 'g' - 28: 0, # 'A' - 40: 0, # 'B' - 54: 0, # 'C' - 45: 0, # 'D' - 32: 0, # 'E' - 50: 0, # 'F' - 49: 0, # 'G' - 38: 0, # 'H' - 39: 0, # 'I' - 53: 0, # 'J' - 36: 0, # 'K' - 41: 0, # 'L' - 34: 0, # 'M' - 35: 0, # 'N' - 47: 0, # 'O' - 46: 0, # 'P' - 43: 0, # 'R' - 33: 0, # 'S' - 37: 0, # 'T' - 57: 0, # 'U' - 48: 0, # 'V' - 55: 0, # 'Y' - 52: 0, # 'Z' - 2: 3, # 'a' - 18: 3, # 'b' - 26: 2, # 'c' - 17: 2, # 'd' - 1: 3, # 'e' - 27: 2, # 'f' - 12: 3, # 'g' - 20: 3, # 'h' - 9: 3, # 'i' - 22: 3, # 'j' - 7: 2, # 'k' - 6: 3, # 'l' - 13: 2, # 'm' - 4: 3, # 'n' - 8: 3, # 'o' - 23: 1, # 'p' - 10: 3, # 'r' - 5: 3, # 's' - 3: 3, # 't' - 21: 3, # 'u' - 19: 3, # 'v' - 62: 0, # 'x' - 16: 3, # 'y' - 11: 2, # 'z' - 51: 0, # 'Á' - 44: 0, # 'É' - 61: 0, # 'Í' - 58: 0, # 'Ó' - 59: 0, # 'Ö' - 60: 0, # 'Ú' - 63: 0, # 'Ü' - 14: 3, # 'á' - 15: 3, # 'é' - 30: 2, # 'í' - 25: 3, # 'ó' - 24: 2, # 'ö' - 31: 2, # 'ú' - 29: 2, # 'ü' - 42: 2, # 'ő' - 56: 1, # 'ű' - }, - 20: { # 'h' - 28: 0, # 'A' - 40: 0, # 'B' - 54: 0, # 'C' - 45: 0, # 'D' - 32: 0, # 'E' - 50: 0, # 'F' - 49: 0, # 'G' - 38: 0, # 'H' - 39: 0, # 'I' - 53: 0, # 'J' - 36: 0, # 'K' - 41: 0, # 'L' - 34: 0, # 'M' - 35: 0, # 'N' - 47: 0, # 'O' - 46: 0, # 'P' - 43: 0, # 'R' - 33: 0, # 'S' - 37: 0, # 'T' - 57: 0, # 'U' - 48: 0, # 'V' - 55: 0, # 'Y' - 52: 0, # 'Z' - 2: 3, # 'a' - 18: 1, # 'b' - 26: 1, # 'c' - 17: 0, # 'd' - 1: 3, # 'e' - 27: 0, # 'f' - 12: 1, # 'g' - 20: 2, # 'h' - 9: 3, # 'i' - 22: 1, # 'j' - 7: 1, # 'k' - 6: 1, # 'l' - 13: 1, # 'm' - 4: 1, # 'n' - 8: 3, # 'o' - 23: 0, # 'p' - 10: 1, # 'r' - 5: 2, # 's' - 3: 1, # 't' - 21: 3, # 'u' - 19: 1, # 'v' - 62: 0, # 'x' - 16: 2, # 'y' - 11: 0, # 'z' - 51: 0, # 'Á' - 44: 0, # 'É' - 61: 0, # 'Í' - 58: 0, # 'Ó' - 59: 0, # 'Ö' - 60: 0, # 'Ú' - 63: 0, # 'Ü' - 14: 3, # 'á' - 15: 3, # 'é' - 30: 3, # 'í' - 25: 2, # 'ó' - 24: 2, # 'ö' - 31: 2, # 'ú' - 29: 1, # 'ü' - 42: 1, # 'ő' - 56: 1, # 'ű' - }, - 9: { # 'i' - 28: 0, # 'A' - 40: 0, # 'B' - 54: 0, # 'C' - 45: 0, # 'D' - 32: 0, # 'E' - 50: 0, # 'F' - 49: 0, # 'G' - 38: 0, # 'H' - 39: 0, # 'I' - 53: 0, # 'J' - 36: 0, # 'K' - 41: 0, # 'L' - 34: 0, # 'M' - 35: 0, # 'N' - 47: 0, # 'O' - 46: 0, # 'P' - 43: 0, # 'R' - 33: 0, # 'S' - 37: 0, # 'T' - 57: 0, # 'U' - 48: 0, # 'V' - 55: 0, # 'Y' - 52: 0, # 'Z' - 2: 3, # 'a' - 18: 3, # 'b' - 26: 3, # 'c' - 17: 3, # 'd' - 1: 3, # 'e' - 27: 3, # 'f' - 12: 3, # 'g' - 20: 3, # 'h' - 9: 2, # 'i' - 22: 2, # 'j' - 7: 3, # 'k' - 6: 3, # 'l' - 13: 3, # 'm' - 4: 3, # 'n' - 8: 2, # 'o' - 23: 2, # 'p' - 10: 3, # 'r' - 5: 3, # 's' - 3: 3, # 't' - 21: 3, # 'u' - 19: 3, # 'v' - 62: 1, # 'x' - 16: 1, # 'y' - 11: 3, # 'z' - 51: 0, # 'Á' - 44: 0, # 'É' - 61: 0, # 'Í' - 58: 0, # 'Ó' - 59: 0, # 'Ö' - 60: 0, # 'Ú' - 63: 0, # 'Ü' - 14: 3, # 'á' - 15: 2, # 'é' - 30: 1, # 'í' - 25: 3, # 'ó' - 24: 1, # 'ö' - 31: 2, # 'ú' - 29: 1, # 'ü' - 42: 0, # 'ő' - 56: 1, # 'ű' - }, - 22: { # 'j' - 28: 0, # 'A' - 40: 0, # 'B' - 54: 0, # 'C' - 45: 0, # 'D' - 32: 0, # 'E' - 50: 0, # 'F' - 49: 0, # 'G' - 38: 0, # 'H' - 39: 0, # 'I' - 53: 0, # 'J' - 36: 0, # 'K' - 41: 0, # 'L' - 34: 0, # 'M' - 35: 0, # 'N' - 47: 0, # 'O' - 46: 0, # 'P' - 43: 0, # 'R' - 33: 0, # 'S' - 37: 0, # 'T' - 57: 0, # 'U' - 48: 0, # 'V' - 55: 0, # 'Y' - 52: 0, # 'Z' - 2: 3, # 'a' - 18: 2, # 'b' - 26: 1, # 'c' - 17: 3, # 'd' - 1: 3, # 'e' - 27: 1, # 'f' - 12: 1, # 'g' - 20: 2, # 'h' - 9: 1, # 'i' - 22: 2, # 'j' - 7: 2, # 'k' - 6: 2, # 'l' - 13: 1, # 'm' - 4: 2, # 'n' - 8: 3, # 'o' - 23: 1, # 'p' - 10: 2, # 'r' - 5: 2, # 's' - 3: 3, # 't' - 21: 3, # 'u' - 19: 1, # 'v' - 62: 0, # 'x' - 16: 0, # 'y' - 11: 2, # 'z' - 51: 0, # 'Á' - 44: 0, # 'É' - 61: 0, # 'Í' - 58: 0, # 'Ó' - 59: 0, # 'Ö' - 60: 0, # 'Ú' - 63: 0, # 'Ü' - 14: 3, # 'á' - 15: 3, # 'é' - 30: 1, # 'í' - 25: 3, # 'ó' - 24: 3, # 'ö' - 31: 3, # 'ú' - 29: 2, # 'ü' - 42: 1, # 'ő' - 56: 1, # 'ű' - }, - 7: { # 'k' - 28: 0, # 'A' - 40: 0, # 'B' - 54: 0, # 'C' - 45: 0, # 'D' - 32: 0, # 'E' - 50: 0, # 'F' - 49: 0, # 'G' - 38: 0, # 'H' - 39: 0, # 'I' - 53: 0, # 'J' - 36: 0, # 'K' - 41: 0, # 'L' - 34: 0, # 'M' - 35: 0, # 'N' - 47: 0, # 'O' - 46: 0, # 'P' - 43: 0, # 'R' - 33: 0, # 'S' - 37: 0, # 'T' - 57: 0, # 'U' - 48: 0, # 'V' - 55: 0, # 'Y' - 52: 0, # 'Z' - 2: 3, # 'a' - 18: 3, # 'b' - 26: 2, # 'c' - 17: 1, # 'd' - 1: 3, # 'e' - 27: 1, # 'f' - 12: 1, # 'g' - 20: 2, # 'h' - 9: 3, # 'i' - 22: 2, # 'j' - 7: 3, # 'k' - 6: 3, # 'l' - 13: 1, # 'm' - 4: 3, # 'n' - 8: 3, # 'o' - 23: 1, # 'p' - 10: 3, # 'r' - 5: 3, # 's' - 3: 3, # 't' - 21: 3, # 'u' - 19: 2, # 'v' - 62: 0, # 'x' - 16: 2, # 'y' - 11: 1, # 'z' - 51: 0, # 'Á' - 44: 0, # 'É' - 61: 0, # 'Í' - 58: 0, # 'Ó' - 59: 0, # 'Ö' - 60: 0, # 'Ú' - 63: 0, # 'Ü' - 14: 3, # 'á' - 15: 3, # 'é' - 30: 3, # 'í' - 25: 2, # 'ó' - 24: 3, # 'ö' - 31: 1, # 'ú' - 29: 3, # 'ü' - 42: 1, # 'ő' - 56: 1, # 'ű' - }, - 6: { # 'l' - 28: 0, # 'A' - 40: 0, # 'B' - 54: 0, # 'C' - 45: 0, # 'D' - 32: 0, # 'E' - 50: 0, # 'F' - 49: 0, # 'G' - 38: 0, # 'H' - 39: 0, # 'I' - 53: 0, # 'J' - 36: 1, # 'K' - 41: 0, # 'L' - 34: 0, # 'M' - 35: 1, # 'N' - 47: 0, # 'O' - 46: 0, # 'P' - 43: 0, # 'R' - 33: 0, # 'S' - 37: 0, # 'T' - 57: 0, # 'U' - 48: 0, # 'V' - 55: 0, # 'Y' - 52: 0, # 'Z' - 2: 3, # 'a' - 18: 2, # 'b' - 26: 3, # 'c' - 17: 3, # 'd' - 1: 3, # 'e' - 27: 3, # 'f' - 12: 3, # 'g' - 20: 3, # 'h' - 9: 3, # 'i' - 22: 3, # 'j' - 7: 3, # 'k' - 6: 3, # 'l' - 13: 3, # 'm' - 4: 3, # 'n' - 8: 3, # 'o' - 23: 2, # 'p' - 10: 2, # 'r' - 5: 3, # 's' - 3: 3, # 't' - 21: 3, # 'u' - 19: 3, # 'v' - 62: 0, # 'x' - 16: 3, # 'y' - 11: 2, # 'z' - 51: 0, # 'Á' - 44: 0, # 'É' - 61: 0, # 'Í' - 58: 0, # 'Ó' - 59: 0, # 'Ö' - 60: 0, # 'Ú' - 63: 0, # 'Ü' - 14: 3, # 'á' - 15: 3, # 'é' - 30: 3, # 'í' - 25: 3, # 'ó' - 24: 3, # 'ö' - 31: 2, # 'ú' - 29: 2, # 'ü' - 42: 3, # 'ő' - 56: 1, # 'ű' - }, - 13: { # 'm' - 28: 0, # 'A' - 40: 0, # 'B' - 54: 0, # 'C' - 45: 0, # 'D' - 32: 0, # 'E' - 50: 0, # 'F' - 49: 0, # 'G' - 38: 0, # 'H' - 39: 0, # 'I' - 53: 0, # 'J' - 36: 0, # 'K' - 41: 0, # 'L' - 34: 0, # 'M' - 35: 0, # 'N' - 47: 0, # 'O' - 46: 0, # 'P' - 43: 0, # 'R' - 33: 0, # 'S' - 37: 0, # 'T' - 57: 0, # 'U' - 48: 0, # 'V' - 55: 0, # 'Y' - 52: 0, # 'Z' - 2: 3, # 'a' - 18: 3, # 'b' - 26: 2, # 'c' - 17: 1, # 'd' - 1: 3, # 'e' - 27: 1, # 'f' - 12: 1, # 'g' - 20: 2, # 'h' - 9: 3, # 'i' - 22: 2, # 'j' - 7: 1, # 'k' - 6: 3, # 'l' - 13: 3, # 'm' - 4: 2, # 'n' - 8: 3, # 'o' - 23: 3, # 'p' - 10: 2, # 'r' - 5: 2, # 's' - 3: 2, # 't' - 21: 3, # 'u' - 19: 1, # 'v' - 62: 0, # 'x' - 16: 1, # 'y' - 11: 2, # 'z' - 51: 0, # 'Á' - 44: 0, # 'É' - 61: 0, # 'Í' - 58: 0, # 'Ó' - 59: 0, # 'Ö' - 60: 0, # 'Ú' - 63: 0, # 'Ü' - 14: 3, # 'á' - 15: 3, # 'é' - 30: 2, # 'í' - 25: 2, # 'ó' - 24: 2, # 'ö' - 31: 2, # 'ú' - 29: 2, # 'ü' - 42: 1, # 'ő' - 56: 2, # 'ű' - }, - 4: { # 'n' - 28: 0, # 'A' - 40: 0, # 'B' - 54: 0, # 'C' - 45: 0, # 'D' - 32: 0, # 'E' - 50: 0, # 'F' - 49: 0, # 'G' - 38: 0, # 'H' - 39: 0, # 'I' - 53: 0, # 'J' - 36: 0, # 'K' - 41: 0, # 'L' - 34: 0, # 'M' - 35: 0, # 'N' - 47: 0, # 'O' - 46: 0, # 'P' - 43: 0, # 'R' - 33: 0, # 'S' - 37: 0, # 'T' - 57: 0, # 'U' - 48: 0, # 'V' - 55: 0, # 'Y' - 52: 0, # 'Z' - 2: 3, # 'a' - 18: 3, # 'b' - 26: 3, # 'c' - 17: 3, # 'd' - 1: 3, # 'e' - 27: 2, # 'f' - 12: 3, # 'g' - 20: 3, # 'h' - 9: 3, # 'i' - 22: 2, # 'j' - 7: 3, # 'k' - 6: 2, # 'l' - 13: 2, # 'm' - 4: 3, # 'n' - 8: 3, # 'o' - 23: 2, # 'p' - 10: 2, # 'r' - 5: 3, # 's' - 3: 3, # 't' - 21: 3, # 'u' - 19: 2, # 'v' - 62: 1, # 'x' - 16: 3, # 'y' - 11: 3, # 'z' - 51: 0, # 'Á' - 44: 0, # 'É' - 61: 0, # 'Í' - 58: 0, # 'Ó' - 59: 0, # 'Ö' - 60: 0, # 'Ú' - 63: 0, # 'Ü' - 14: 3, # 'á' - 15: 3, # 'é' - 30: 2, # 'í' - 25: 2, # 'ó' - 24: 3, # 'ö' - 31: 2, # 'ú' - 29: 3, # 'ü' - 42: 2, # 'ő' - 56: 1, # 'ű' - }, - 8: { # 'o' - 28: 0, # 'A' - 40: 0, # 'B' - 54: 0, # 'C' - 45: 0, # 'D' - 32: 0, # 'E' - 50: 0, # 'F' - 49: 0, # 'G' - 38: 0, # 'H' - 39: 0, # 'I' - 53: 0, # 'J' - 36: 0, # 'K' - 41: 0, # 'L' - 34: 0, # 'M' - 35: 0, # 'N' - 47: 1, # 'O' - 46: 0, # 'P' - 43: 0, # 'R' - 33: 0, # 'S' - 37: 0, # 'T' - 57: 0, # 'U' - 48: 0, # 'V' - 55: 0, # 'Y' - 52: 0, # 'Z' - 2: 2, # 'a' - 18: 3, # 'b' - 26: 3, # 'c' - 17: 3, # 'd' - 1: 2, # 'e' - 27: 2, # 'f' - 12: 3, # 'g' - 20: 3, # 'h' - 9: 2, # 'i' - 22: 2, # 'j' - 7: 3, # 'k' - 6: 3, # 'l' - 13: 3, # 'm' - 4: 3, # 'n' - 8: 1, # 'o' - 23: 3, # 'p' - 10: 3, # 'r' - 5: 3, # 's' - 3: 3, # 't' - 21: 2, # 'u' - 19: 3, # 'v' - 62: 1, # 'x' - 16: 1, # 'y' - 11: 3, # 'z' - 51: 0, # 'Á' - 44: 0, # 'É' - 61: 0, # 'Í' - 58: 0, # 'Ó' - 59: 0, # 'Ö' - 60: 0, # 'Ú' - 63: 0, # 'Ü' - 14: 1, # 'á' - 15: 2, # 'é' - 30: 1, # 'í' - 25: 1, # 'ó' - 24: 1, # 'ö' - 31: 1, # 'ú' - 29: 1, # 'ü' - 42: 0, # 'ő' - 56: 0, # 'ű' - }, - 23: { # 'p' - 28: 0, # 'A' - 40: 0, # 'B' - 54: 0, # 'C' - 45: 0, # 'D' - 32: 0, # 'E' - 50: 0, # 'F' - 49: 0, # 'G' - 38: 0, # 'H' - 39: 0, # 'I' - 53: 0, # 'J' - 36: 0, # 'K' - 41: 0, # 'L' - 34: 0, # 'M' - 35: 0, # 'N' - 47: 0, # 'O' - 46: 0, # 'P' - 43: 0, # 'R' - 33: 0, # 'S' - 37: 0, # 'T' - 57: 0, # 'U' - 48: 0, # 'V' - 55: 0, # 'Y' - 52: 0, # 'Z' - 2: 3, # 'a' - 18: 1, # 'b' - 26: 2, # 'c' - 17: 1, # 'd' - 1: 3, # 'e' - 27: 1, # 'f' - 12: 1, # 'g' - 20: 2, # 'h' - 9: 3, # 'i' - 22: 2, # 'j' - 7: 2, # 'k' - 6: 3, # 'l' - 13: 1, # 'm' - 4: 2, # 'n' - 8: 3, # 'o' - 23: 3, # 'p' - 10: 3, # 'r' - 5: 2, # 's' - 3: 2, # 't' - 21: 3, # 'u' - 19: 2, # 'v' - 62: 0, # 'x' - 16: 1, # 'y' - 11: 2, # 'z' - 51: 0, # 'Á' - 44: 0, # 'É' - 61: 0, # 'Í' - 58: 0, # 'Ó' - 59: 0, # 'Ö' - 60: 0, # 'Ú' - 63: 0, # 'Ü' - 14: 3, # 'á' - 15: 3, # 'é' - 30: 2, # 'í' - 25: 2, # 'ó' - 24: 2, # 'ö' - 31: 1, # 'ú' - 29: 2, # 'ü' - 42: 1, # 'ő' - 56: 1, # 'ű' - }, - 10: { # 'r' - 28: 0, # 'A' - 40: 0, # 'B' - 54: 0, # 'C' - 45: 0, # 'D' - 32: 0, # 'E' - 50: 0, # 'F' - 49: 0, # 'G' - 38: 0, # 'H' - 39: 0, # 'I' - 53: 0, # 'J' - 36: 0, # 'K' - 41: 0, # 'L' - 34: 0, # 'M' - 35: 0, # 'N' - 47: 0, # 'O' - 46: 0, # 'P' - 43: 0, # 'R' - 33: 0, # 'S' - 37: 0, # 'T' - 57: 0, # 'U' - 48: 0, # 'V' - 55: 0, # 'Y' - 52: 0, # 'Z' - 2: 3, # 'a' - 18: 3, # 'b' - 26: 3, # 'c' - 17: 3, # 'd' - 1: 3, # 'e' - 27: 2, # 'f' - 12: 3, # 'g' - 20: 2, # 'h' - 9: 3, # 'i' - 22: 3, # 'j' - 7: 3, # 'k' - 6: 3, # 'l' - 13: 3, # 'm' - 4: 3, # 'n' - 8: 3, # 'o' - 23: 2, # 'p' - 10: 3, # 'r' - 5: 3, # 's' - 3: 3, # 't' - 21: 3, # 'u' - 19: 3, # 'v' - 62: 1, # 'x' - 16: 2, # 'y' - 11: 3, # 'z' - 51: 0, # 'Á' - 44: 0, # 'É' - 61: 0, # 'Í' - 58: 0, # 'Ó' - 59: 0, # 'Ö' - 60: 0, # 'Ú' - 63: 0, # 'Ü' - 14: 3, # 'á' - 15: 3, # 'é' - 30: 2, # 'í' - 25: 3, # 'ó' - 24: 3, # 'ö' - 31: 3, # 'ú' - 29: 3, # 'ü' - 42: 2, # 'ő' - 56: 2, # 'ű' - }, - 5: { # 's' - 28: 0, # 'A' - 40: 0, # 'B' - 54: 0, # 'C' - 45: 0, # 'D' - 32: 0, # 'E' - 50: 0, # 'F' - 49: 0, # 'G' - 38: 0, # 'H' - 39: 0, # 'I' - 53: 0, # 'J' - 36: 0, # 'K' - 41: 0, # 'L' - 34: 0, # 'M' - 35: 0, # 'N' - 47: 0, # 'O' - 46: 0, # 'P' - 43: 0, # 'R' - 33: 0, # 'S' - 37: 0, # 'T' - 57: 0, # 'U' - 48: 0, # 'V' - 55: 0, # 'Y' - 52: 0, # 'Z' - 2: 3, # 'a' - 18: 3, # 'b' - 26: 2, # 'c' - 17: 2, # 'd' - 1: 3, # 'e' - 27: 2, # 'f' - 12: 2, # 'g' - 20: 2, # 'h' - 9: 3, # 'i' - 22: 1, # 'j' - 7: 3, # 'k' - 6: 2, # 'l' - 13: 3, # 'm' - 4: 3, # 'n' - 8: 3, # 'o' - 23: 2, # 'p' - 10: 3, # 'r' - 5: 3, # 's' - 3: 3, # 't' - 21: 3, # 'u' - 19: 2, # 'v' - 62: 0, # 'x' - 16: 1, # 'y' - 11: 3, # 'z' - 51: 0, # 'Á' - 44: 0, # 'É' - 61: 0, # 'Í' - 58: 0, # 'Ó' - 59: 0, # 'Ö' - 60: 0, # 'Ú' - 63: 0, # 'Ü' - 14: 3, # 'á' - 15: 3, # 'é' - 30: 3, # 'í' - 25: 3, # 'ó' - 24: 3, # 'ö' - 31: 3, # 'ú' - 29: 3, # 'ü' - 42: 2, # 'ő' - 56: 1, # 'ű' - }, - 3: { # 't' - 28: 0, # 'A' - 40: 0, # 'B' - 54: 0, # 'C' - 45: 0, # 'D' - 32: 0, # 'E' - 50: 0, # 'F' - 49: 0, # 'G' - 38: 0, # 'H' - 39: 0, # 'I' - 53: 0, # 'J' - 36: 0, # 'K' - 41: 0, # 'L' - 34: 0, # 'M' - 35: 0, # 'N' - 47: 0, # 'O' - 46: 0, # 'P' - 43: 0, # 'R' - 33: 0, # 'S' - 37: 0, # 'T' - 57: 0, # 'U' - 48: 0, # 'V' - 55: 0, # 'Y' - 52: 0, # 'Z' - 2: 3, # 'a' - 18: 3, # 'b' - 26: 2, # 'c' - 17: 1, # 'd' - 1: 3, # 'e' - 27: 2, # 'f' - 12: 1, # 'g' - 20: 3, # 'h' - 9: 3, # 'i' - 22: 3, # 'j' - 7: 3, # 'k' - 6: 3, # 'l' - 13: 2, # 'm' - 4: 3, # 'n' - 8: 3, # 'o' - 23: 1, # 'p' - 10: 3, # 'r' - 5: 3, # 's' - 3: 3, # 't' - 21: 3, # 'u' - 19: 3, # 'v' - 62: 0, # 'x' - 16: 3, # 'y' - 11: 1, # 'z' - 51: 0, # 'Á' - 44: 0, # 'É' - 61: 0, # 'Í' - 58: 0, # 'Ó' - 59: 0, # 'Ö' - 60: 0, # 'Ú' - 63: 0, # 'Ü' - 14: 3, # 'á' - 15: 3, # 'é' - 30: 2, # 'í' - 25: 3, # 'ó' - 24: 3, # 'ö' - 31: 3, # 'ú' - 29: 3, # 'ü' - 42: 3, # 'ő' - 56: 2, # 'ű' - }, - 21: { # 'u' - 28: 0, # 'A' - 40: 0, # 'B' - 54: 0, # 'C' - 45: 0, # 'D' - 32: 0, # 'E' - 50: 0, # 'F' - 49: 0, # 'G' - 38: 0, # 'H' - 39: 0, # 'I' - 53: 0, # 'J' - 36: 0, # 'K' - 41: 0, # 'L' - 34: 0, # 'M' - 35: 0, # 'N' - 47: 0, # 'O' - 46: 0, # 'P' - 43: 0, # 'R' - 33: 0, # 'S' - 37: 0, # 'T' - 57: 0, # 'U' - 48: 0, # 'V' - 55: 0, # 'Y' - 52: 0, # 'Z' - 2: 1, # 'a' - 18: 2, # 'b' - 26: 2, # 'c' - 17: 3, # 'd' - 1: 2, # 'e' - 27: 1, # 'f' - 12: 3, # 'g' - 20: 2, # 'h' - 9: 2, # 'i' - 22: 2, # 'j' - 7: 3, # 'k' - 6: 3, # 'l' - 13: 3, # 'm' - 4: 3, # 'n' - 8: 1, # 'o' - 23: 2, # 'p' - 10: 3, # 'r' - 5: 3, # 's' - 3: 3, # 't' - 21: 1, # 'u' - 19: 3, # 'v' - 62: 1, # 'x' - 16: 1, # 'y' - 11: 2, # 'z' - 51: 0, # 'Á' - 44: 0, # 'É' - 61: 0, # 'Í' - 58: 0, # 'Ó' - 59: 0, # 'Ö' - 60: 0, # 'Ú' - 63: 0, # 'Ü' - 14: 2, # 'á' - 15: 1, # 'é' - 30: 1, # 'í' - 25: 1, # 'ó' - 24: 0, # 'ö' - 31: 1, # 'ú' - 29: 0, # 'ü' - 42: 0, # 'ő' - 56: 0, # 'ű' - }, - 19: { # 'v' - 28: 0, # 'A' - 40: 0, # 'B' - 54: 0, # 'C' - 45: 0, # 'D' - 32: 0, # 'E' - 50: 0, # 'F' - 49: 0, # 'G' - 38: 0, # 'H' - 39: 0, # 'I' - 53: 0, # 'J' - 36: 0, # 'K' - 41: 0, # 'L' - 34: 0, # 'M' - 35: 0, # 'N' - 47: 0, # 'O' - 46: 0, # 'P' - 43: 0, # 'R' - 33: 0, # 'S' - 37: 0, # 'T' - 57: 0, # 'U' - 48: 0, # 'V' - 55: 0, # 'Y' - 52: 0, # 'Z' - 2: 3, # 'a' - 18: 2, # 'b' - 26: 1, # 'c' - 17: 1, # 'd' - 1: 3, # 'e' - 27: 1, # 'f' - 12: 1, # 'g' - 20: 1, # 'h' - 9: 3, # 'i' - 22: 1, # 'j' - 7: 1, # 'k' - 6: 1, # 'l' - 13: 1, # 'm' - 4: 1, # 'n' - 8: 3, # 'o' - 23: 1, # 'p' - 10: 1, # 'r' - 5: 2, # 's' - 3: 2, # 't' - 21: 2, # 'u' - 19: 2, # 'v' - 62: 0, # 'x' - 16: 1, # 'y' - 11: 1, # 'z' - 51: 0, # 'Á' - 44: 0, # 'É' - 61: 0, # 'Í' - 58: 0, # 'Ó' - 59: 0, # 'Ö' - 60: 0, # 'Ú' - 63: 0, # 'Ü' - 14: 3, # 'á' - 15: 3, # 'é' - 30: 2, # 'í' - 25: 2, # 'ó' - 24: 2, # 'ö' - 31: 1, # 'ú' - 29: 2, # 'ü' - 42: 1, # 'ő' - 56: 1, # 'ű' - }, - 62: { # 'x' - 28: 0, # 'A' - 40: 0, # 'B' - 54: 0, # 'C' - 45: 0, # 'D' - 32: 0, # 'E' - 50: 0, # 'F' - 49: 0, # 'G' - 38: 0, # 'H' - 39: 0, # 'I' - 53: 0, # 'J' - 36: 0, # 'K' - 41: 0, # 'L' - 34: 0, # 'M' - 35: 0, # 'N' - 47: 0, # 'O' - 46: 0, # 'P' - 43: 0, # 'R' - 33: 0, # 'S' - 37: 0, # 'T' - 57: 0, # 'U' - 48: 0, # 'V' - 55: 0, # 'Y' - 52: 0, # 'Z' - 2: 1, # 'a' - 18: 1, # 'b' - 26: 1, # 'c' - 17: 0, # 'd' - 1: 1, # 'e' - 27: 1, # 'f' - 12: 0, # 'g' - 20: 0, # 'h' - 9: 1, # 'i' - 22: 0, # 'j' - 7: 1, # 'k' - 6: 1, # 'l' - 13: 1, # 'm' - 4: 1, # 'n' - 8: 1, # 'o' - 23: 1, # 'p' - 10: 1, # 'r' - 5: 1, # 's' - 3: 1, # 't' - 21: 1, # 'u' - 19: 0, # 'v' - 62: 0, # 'x' - 16: 0, # 'y' - 11: 0, # 'z' - 51: 0, # 'Á' - 44: 0, # 'É' - 61: 0, # 'Í' - 58: 0, # 'Ó' - 59: 0, # 'Ö' - 60: 0, # 'Ú' - 63: 0, # 'Ü' - 14: 1, # 'á' - 15: 1, # 'é' - 30: 1, # 'í' - 25: 1, # 'ó' - 24: 0, # 'ö' - 31: 0, # 'ú' - 29: 0, # 'ü' - 42: 0, # 'ő' - 56: 0, # 'ű' - }, - 16: { # 'y' - 28: 0, # 'A' - 40: 0, # 'B' - 54: 0, # 'C' - 45: 0, # 'D' - 32: 0, # 'E' - 50: 0, # 'F' - 49: 0, # 'G' - 38: 0, # 'H' - 39: 0, # 'I' - 53: 0, # 'J' - 36: 0, # 'K' - 41: 0, # 'L' - 34: 0, # 'M' - 35: 0, # 'N' - 47: 0, # 'O' - 46: 0, # 'P' - 43: 0, # 'R' - 33: 0, # 'S' - 37: 0, # 'T' - 57: 0, # 'U' - 48: 0, # 'V' - 55: 0, # 'Y' - 52: 0, # 'Z' - 2: 3, # 'a' - 18: 2, # 'b' - 26: 1, # 'c' - 17: 1, # 'd' - 1: 3, # 'e' - 27: 2, # 'f' - 12: 2, # 'g' - 20: 2, # 'h' - 9: 3, # 'i' - 22: 2, # 'j' - 7: 2, # 'k' - 6: 2, # 'l' - 13: 2, # 'm' - 4: 3, # 'n' - 8: 3, # 'o' - 23: 2, # 'p' - 10: 2, # 'r' - 5: 3, # 's' - 3: 3, # 't' - 21: 3, # 'u' - 19: 3, # 'v' - 62: 0, # 'x' - 16: 0, # 'y' - 11: 2, # 'z' - 51: 0, # 'Á' - 44: 0, # 'É' - 61: 0, # 'Í' - 58: 0, # 'Ó' - 59: 0, # 'Ö' - 60: 0, # 'Ú' - 63: 0, # 'Ü' - 14: 3, # 'á' - 15: 3, # 'é' - 30: 2, # 'í' - 25: 2, # 'ó' - 24: 3, # 'ö' - 31: 2, # 'ú' - 29: 2, # 'ü' - 42: 1, # 'ő' - 56: 2, # 'ű' - }, - 11: { # 'z' - 28: 0, # 'A' - 40: 0, # 'B' - 54: 0, # 'C' - 45: 0, # 'D' - 32: 0, # 'E' - 50: 0, # 'F' - 49: 0, # 'G' - 38: 0, # 'H' - 39: 0, # 'I' - 53: 0, # 'J' - 36: 0, # 'K' - 41: 0, # 'L' - 34: 0, # 'M' - 35: 0, # 'N' - 47: 0, # 'O' - 46: 0, # 'P' - 43: 0, # 'R' - 33: 0, # 'S' - 37: 0, # 'T' - 57: 0, # 'U' - 48: 0, # 'V' - 55: 0, # 'Y' - 52: 0, # 'Z' - 2: 3, # 'a' - 18: 2, # 'b' - 26: 1, # 'c' - 17: 3, # 'd' - 1: 3, # 'e' - 27: 1, # 'f' - 12: 2, # 'g' - 20: 2, # 'h' - 9: 3, # 'i' - 22: 1, # 'j' - 7: 3, # 'k' - 6: 2, # 'l' - 13: 3, # 'm' - 4: 3, # 'n' - 8: 3, # 'o' - 23: 1, # 'p' - 10: 2, # 'r' - 5: 3, # 's' - 3: 3, # 't' - 21: 3, # 'u' - 19: 2, # 'v' - 62: 0, # 'x' - 16: 1, # 'y' - 11: 3, # 'z' - 51: 0, # 'Á' - 44: 0, # 'É' - 61: 0, # 'Í' - 58: 0, # 'Ó' - 59: 0, # 'Ö' - 60: 0, # 'Ú' - 63: 0, # 'Ü' - 14: 3, # 'á' - 15: 3, # 'é' - 30: 3, # 'í' - 25: 3, # 'ó' - 24: 3, # 'ö' - 31: 2, # 'ú' - 29: 3, # 'ü' - 42: 2, # 'ő' - 56: 1, # 'ű' - }, - 51: { # 'Á' - 28: 0, # 'A' - 40: 1, # 'B' - 54: 1, # 'C' - 45: 1, # 'D' - 32: 0, # 'E' - 50: 1, # 'F' - 49: 2, # 'G' - 38: 1, # 'H' - 39: 1, # 'I' - 53: 1, # 'J' - 36: 1, # 'K' - 41: 2, # 'L' - 34: 1, # 'M' - 35: 2, # 'N' - 47: 0, # 'O' - 46: 1, # 'P' - 43: 2, # 'R' - 33: 2, # 'S' - 37: 1, # 'T' - 57: 0, # 'U' - 48: 1, # 'V' - 55: 0, # 'Y' - 52: 1, # 'Z' - 2: 0, # 'a' - 18: 1, # 'b' - 26: 1, # 'c' - 17: 1, # 'd' - 1: 0, # 'e' - 27: 0, # 'f' - 12: 1, # 'g' - 20: 1, # 'h' - 9: 0, # 'i' - 22: 1, # 'j' - 7: 1, # 'k' - 6: 2, # 'l' - 13: 2, # 'm' - 4: 0, # 'n' - 8: 0, # 'o' - 23: 1, # 'p' - 10: 1, # 'r' - 5: 1, # 's' - 3: 1, # 't' - 21: 0, # 'u' - 19: 0, # 'v' - 62: 0, # 'x' - 16: 0, # 'y' - 11: 1, # 'z' - 51: 0, # 'Á' - 44: 0, # 'É' - 61: 1, # 'Í' - 58: 0, # 'Ó' - 59: 0, # 'Ö' - 60: 0, # 'Ú' - 63: 0, # 'Ü' - 14: 0, # 'á' - 15: 0, # 'é' - 30: 0, # 'í' - 25: 0, # 'ó' - 24: 0, # 'ö' - 31: 0, # 'ú' - 29: 0, # 'ü' - 42: 0, # 'ő' - 56: 0, # 'ű' - }, - 44: { # 'É' - 28: 0, # 'A' - 40: 1, # 'B' - 54: 1, # 'C' - 45: 1, # 'D' - 32: 1, # 'E' - 50: 0, # 'F' - 49: 2, # 'G' - 38: 1, # 'H' - 39: 1, # 'I' - 53: 1, # 'J' - 36: 1, # 'K' - 41: 2, # 'L' - 34: 1, # 'M' - 35: 2, # 'N' - 47: 0, # 'O' - 46: 1, # 'P' - 43: 2, # 'R' - 33: 2, # 'S' - 37: 2, # 'T' - 57: 0, # 'U' - 48: 1, # 'V' - 55: 0, # 'Y' - 52: 1, # 'Z' - 2: 0, # 'a' - 18: 1, # 'b' - 26: 1, # 'c' - 17: 1, # 'd' - 1: 0, # 'e' - 27: 0, # 'f' - 12: 1, # 'g' - 20: 1, # 'h' - 9: 0, # 'i' - 22: 1, # 'j' - 7: 1, # 'k' - 6: 2, # 'l' - 13: 1, # 'm' - 4: 2, # 'n' - 8: 0, # 'o' - 23: 1, # 'p' - 10: 2, # 'r' - 5: 3, # 's' - 3: 1, # 't' - 21: 0, # 'u' - 19: 1, # 'v' - 62: 0, # 'x' - 16: 0, # 'y' - 11: 0, # 'z' - 51: 0, # 'Á' - 44: 1, # 'É' - 61: 0, # 'Í' - 58: 0, # 'Ó' - 59: 0, # 'Ö' - 60: 0, # 'Ú' - 63: 0, # 'Ü' - 14: 0, # 'á' - 15: 0, # 'é' - 30: 0, # 'í' - 25: 0, # 'ó' - 24: 0, # 'ö' - 31: 0, # 'ú' - 29: 0, # 'ü' - 42: 0, # 'ő' - 56: 0, # 'ű' - }, - 61: { # 'Í' - 28: 0, # 'A' - 40: 1, # 'B' - 54: 1, # 'C' - 45: 1, # 'D' - 32: 0, # 'E' - 50: 1, # 'F' - 49: 1, # 'G' - 38: 0, # 'H' - 39: 0, # 'I' - 53: 1, # 'J' - 36: 0, # 'K' - 41: 1, # 'L' - 34: 1, # 'M' - 35: 1, # 'N' - 47: 0, # 'O' - 46: 1, # 'P' - 43: 1, # 'R' - 33: 1, # 'S' - 37: 1, # 'T' - 57: 0, # 'U' - 48: 1, # 'V' - 55: 0, # 'Y' - 52: 1, # 'Z' - 2: 0, # 'a' - 18: 0, # 'b' - 26: 0, # 'c' - 17: 0, # 'd' - 1: 0, # 'e' - 27: 0, # 'f' - 12: 2, # 'g' - 20: 0, # 'h' - 9: 0, # 'i' - 22: 0, # 'j' - 7: 0, # 'k' - 6: 0, # 'l' - 13: 1, # 'm' - 4: 0, # 'n' - 8: 0, # 'o' - 23: 0, # 'p' - 10: 1, # 'r' - 5: 0, # 's' - 3: 1, # 't' - 21: 0, # 'u' - 19: 0, # 'v' - 62: 0, # 'x' - 16: 0, # 'y' - 11: 1, # 'z' - 51: 0, # 'Á' - 44: 0, # 'É' - 61: 0, # 'Í' - 58: 0, # 'Ó' - 59: 0, # 'Ö' - 60: 0, # 'Ú' - 63: 0, # 'Ü' - 14: 0, # 'á' - 15: 0, # 'é' - 30: 0, # 'í' - 25: 0, # 'ó' - 24: 0, # 'ö' - 31: 0, # 'ú' - 29: 0, # 'ü' - 42: 0, # 'ő' - 56: 0, # 'ű' - }, - 58: { # 'Ó' - 28: 1, # 'A' - 40: 1, # 'B' - 54: 1, # 'C' - 45: 1, # 'D' - 32: 0, # 'E' - 50: 1, # 'F' - 49: 1, # 'G' - 38: 1, # 'H' - 39: 1, # 'I' - 53: 1, # 'J' - 36: 1, # 'K' - 41: 2, # 'L' - 34: 1, # 'M' - 35: 1, # 'N' - 47: 0, # 'O' - 46: 1, # 'P' - 43: 1, # 'R' - 33: 1, # 'S' - 37: 1, # 'T' - 57: 0, # 'U' - 48: 1, # 'V' - 55: 0, # 'Y' - 52: 1, # 'Z' - 2: 0, # 'a' - 18: 1, # 'b' - 26: 1, # 'c' - 17: 1, # 'd' - 1: 0, # 'e' - 27: 0, # 'f' - 12: 0, # 'g' - 20: 2, # 'h' - 9: 0, # 'i' - 22: 0, # 'j' - 7: 1, # 'k' - 6: 1, # 'l' - 13: 0, # 'm' - 4: 1, # 'n' - 8: 0, # 'o' - 23: 1, # 'p' - 10: 1, # 'r' - 5: 1, # 's' - 3: 0, # 't' - 21: 0, # 'u' - 19: 1, # 'v' - 62: 0, # 'x' - 16: 0, # 'y' - 11: 1, # 'z' - 51: 0, # 'Á' - 44: 1, # 'É' - 61: 0, # 'Í' - 58: 0, # 'Ó' - 59: 0, # 'Ö' - 60: 0, # 'Ú' - 63: 0, # 'Ü' - 14: 0, # 'á' - 15: 0, # 'é' - 30: 0, # 'í' - 25: 0, # 'ó' - 24: 0, # 'ö' - 31: 0, # 'ú' - 29: 0, # 'ü' - 42: 0, # 'ő' - 56: 0, # 'ű' - }, - 59: { # 'Ö' - 28: 0, # 'A' - 40: 1, # 'B' - 54: 1, # 'C' - 45: 1, # 'D' - 32: 0, # 'E' - 50: 0, # 'F' - 49: 1, # 'G' - 38: 1, # 'H' - 39: 0, # 'I' - 53: 1, # 'J' - 36: 1, # 'K' - 41: 1, # 'L' - 34: 1, # 'M' - 35: 1, # 'N' - 47: 0, # 'O' - 46: 1, # 'P' - 43: 1, # 'R' - 33: 1, # 'S' - 37: 1, # 'T' - 57: 0, # 'U' - 48: 1, # 'V' - 55: 0, # 'Y' - 52: 1, # 'Z' - 2: 0, # 'a' - 18: 0, # 'b' - 26: 1, # 'c' - 17: 1, # 'd' - 1: 0, # 'e' - 27: 0, # 'f' - 12: 0, # 'g' - 20: 0, # 'h' - 9: 0, # 'i' - 22: 0, # 'j' - 7: 1, # 'k' - 6: 1, # 'l' - 13: 1, # 'm' - 4: 1, # 'n' - 8: 0, # 'o' - 23: 0, # 'p' - 10: 2, # 'r' - 5: 1, # 's' - 3: 1, # 't' - 21: 0, # 'u' - 19: 1, # 'v' - 62: 0, # 'x' - 16: 0, # 'y' - 11: 1, # 'z' - 51: 0, # 'Á' - 44: 0, # 'É' - 61: 0, # 'Í' - 58: 0, # 'Ó' - 59: 0, # 'Ö' - 60: 0, # 'Ú' - 63: 0, # 'Ü' - 14: 0, # 'á' - 15: 0, # 'é' - 30: 0, # 'í' - 25: 0, # 'ó' - 24: 0, # 'ö' - 31: 0, # 'ú' - 29: 0, # 'ü' - 42: 0, # 'ő' - 56: 0, # 'ű' - }, - 60: { # 'Ú' - 28: 0, # 'A' - 40: 1, # 'B' - 54: 1, # 'C' - 45: 1, # 'D' - 32: 0, # 'E' - 50: 1, # 'F' - 49: 1, # 'G' - 38: 0, # 'H' - 39: 0, # 'I' - 53: 1, # 'J' - 36: 1, # 'K' - 41: 1, # 'L' - 34: 1, # 'M' - 35: 1, # 'N' - 47: 0, # 'O' - 46: 0, # 'P' - 43: 1, # 'R' - 33: 1, # 'S' - 37: 1, # 'T' - 57: 0, # 'U' - 48: 1, # 'V' - 55: 0, # 'Y' - 52: 1, # 'Z' - 2: 0, # 'a' - 18: 0, # 'b' - 26: 0, # 'c' - 17: 0, # 'd' - 1: 0, # 'e' - 27: 0, # 'f' - 12: 2, # 'g' - 20: 0, # 'h' - 9: 0, # 'i' - 22: 2, # 'j' - 7: 0, # 'k' - 6: 0, # 'l' - 13: 0, # 'm' - 4: 1, # 'n' - 8: 0, # 'o' - 23: 0, # 'p' - 10: 1, # 'r' - 5: 1, # 's' - 3: 1, # 't' - 21: 0, # 'u' - 19: 0, # 'v' - 62: 0, # 'x' - 16: 0, # 'y' - 11: 0, # 'z' - 51: 0, # 'Á' - 44: 0, # 'É' - 61: 0, # 'Í' - 58: 0, # 'Ó' - 59: 0, # 'Ö' - 60: 0, # 'Ú' - 63: 0, # 'Ü' - 14: 0, # 'á' - 15: 0, # 'é' - 30: 0, # 'í' - 25: 0, # 'ó' - 24: 0, # 'ö' - 31: 0, # 'ú' - 29: 0, # 'ü' - 42: 0, # 'ő' - 56: 0, # 'ű' - }, - 63: { # 'Ü' - 28: 0, # 'A' - 40: 1, # 'B' - 54: 0, # 'C' - 45: 1, # 'D' - 32: 0, # 'E' - 50: 0, # 'F' - 49: 1, # 'G' - 38: 1, # 'H' - 39: 0, # 'I' - 53: 1, # 'J' - 36: 1, # 'K' - 41: 1, # 'L' - 34: 1, # 'M' - 35: 1, # 'N' - 47: 0, # 'O' - 46: 0, # 'P' - 43: 1, # 'R' - 33: 1, # 'S' - 37: 1, # 'T' - 57: 0, # 'U' - 48: 1, # 'V' - 55: 0, # 'Y' - 52: 1, # 'Z' - 2: 0, # 'a' - 18: 1, # 'b' - 26: 0, # 'c' - 17: 1, # 'd' - 1: 0, # 'e' - 27: 0, # 'f' - 12: 1, # 'g' - 20: 0, # 'h' - 9: 0, # 'i' - 22: 0, # 'j' - 7: 0, # 'k' - 6: 1, # 'l' - 13: 0, # 'm' - 4: 1, # 'n' - 8: 0, # 'o' - 23: 0, # 'p' - 10: 1, # 'r' - 5: 1, # 's' - 3: 1, # 't' - 21: 0, # 'u' - 19: 1, # 'v' - 62: 0, # 'x' - 16: 0, # 'y' - 11: 1, # 'z' - 51: 0, # 'Á' - 44: 0, # 'É' - 61: 0, # 'Í' - 58: 0, # 'Ó' - 59: 0, # 'Ö' - 60: 0, # 'Ú' - 63: 0, # 'Ü' - 14: 0, # 'á' - 15: 0, # 'é' - 30: 0, # 'í' - 25: 0, # 'ó' - 24: 0, # 'ö' - 31: 0, # 'ú' - 29: 0, # 'ü' - 42: 0, # 'ő' - 56: 0, # 'ű' - }, - 14: { # 'á' - 28: 0, # 'A' - 40: 0, # 'B' - 54: 0, # 'C' - 45: 0, # 'D' - 32: 0, # 'E' - 50: 0, # 'F' - 49: 0, # 'G' - 38: 0, # 'H' - 39: 0, # 'I' - 53: 0, # 'J' - 36: 0, # 'K' - 41: 0, # 'L' - 34: 0, # 'M' - 35: 0, # 'N' - 47: 0, # 'O' - 46: 0, # 'P' - 43: 0, # 'R' - 33: 0, # 'S' - 37: 0, # 'T' - 57: 0, # 'U' - 48: 0, # 'V' - 55: 0, # 'Y' - 52: 0, # 'Z' - 2: 1, # 'a' - 18: 3, # 'b' - 26: 3, # 'c' - 17: 3, # 'd' - 1: 1, # 'e' - 27: 2, # 'f' - 12: 3, # 'g' - 20: 2, # 'h' - 9: 2, # 'i' - 22: 3, # 'j' - 7: 3, # 'k' - 6: 3, # 'l' - 13: 3, # 'm' - 4: 3, # 'n' - 8: 1, # 'o' - 23: 2, # 'p' - 10: 3, # 'r' - 5: 3, # 's' - 3: 3, # 't' - 21: 2, # 'u' - 19: 3, # 'v' - 62: 0, # 'x' - 16: 1, # 'y' - 11: 3, # 'z' - 51: 0, # 'Á' - 44: 0, # 'É' - 61: 0, # 'Í' - 58: 0, # 'Ó' - 59: 0, # 'Ö' - 60: 0, # 'Ú' - 63: 0, # 'Ü' - 14: 1, # 'á' - 15: 2, # 'é' - 30: 1, # 'í' - 25: 0, # 'ó' - 24: 1, # 'ö' - 31: 0, # 'ú' - 29: 1, # 'ü' - 42: 0, # 'ő' - 56: 0, # 'ű' - }, - 15: { # 'é' - 28: 0, # 'A' - 40: 0, # 'B' - 54: 0, # 'C' - 45: 0, # 'D' - 32: 0, # 'E' - 50: 0, # 'F' - 49: 0, # 'G' - 38: 0, # 'H' - 39: 0, # 'I' - 53: 0, # 'J' - 36: 0, # 'K' - 41: 0, # 'L' - 34: 0, # 'M' - 35: 0, # 'N' - 47: 0, # 'O' - 46: 0, # 'P' - 43: 0, # 'R' - 33: 0, # 'S' - 37: 0, # 'T' - 57: 0, # 'U' - 48: 0, # 'V' - 55: 0, # 'Y' - 52: 0, # 'Z' - 2: 1, # 'a' - 18: 3, # 'b' - 26: 2, # 'c' - 17: 3, # 'd' - 1: 1, # 'e' - 27: 1, # 'f' - 12: 3, # 'g' - 20: 3, # 'h' - 9: 2, # 'i' - 22: 2, # 'j' - 7: 3, # 'k' - 6: 3, # 'l' - 13: 3, # 'm' - 4: 3, # 'n' - 8: 1, # 'o' - 23: 3, # 'p' - 10: 3, # 'r' - 5: 3, # 's' - 3: 3, # 't' - 21: 0, # 'u' - 19: 3, # 'v' - 62: 0, # 'x' - 16: 0, # 'y' - 11: 3, # 'z' - 51: 0, # 'Á' - 44: 0, # 'É' - 61: 0, # 'Í' - 58: 0, # 'Ó' - 59: 0, # 'Ö' - 60: 0, # 'Ú' - 63: 0, # 'Ü' - 14: 1, # 'á' - 15: 1, # 'é' - 30: 0, # 'í' - 25: 0, # 'ó' - 24: 0, # 'ö' - 31: 0, # 'ú' - 29: 1, # 'ü' - 42: 0, # 'ő' - 56: 0, # 'ű' - }, - 30: { # 'í' - 28: 0, # 'A' - 40: 0, # 'B' - 54: 0, # 'C' - 45: 0, # 'D' - 32: 0, # 'E' - 50: 0, # 'F' - 49: 0, # 'G' - 38: 0, # 'H' - 39: 0, # 'I' - 53: 0, # 'J' - 36: 0, # 'K' - 41: 0, # 'L' - 34: 0, # 'M' - 35: 0, # 'N' - 47: 0, # 'O' - 46: 0, # 'P' - 43: 0, # 'R' - 33: 0, # 'S' - 37: 0, # 'T' - 57: 0, # 'U' - 48: 0, # 'V' - 55: 0, # 'Y' - 52: 0, # 'Z' - 2: 0, # 'a' - 18: 1, # 'b' - 26: 2, # 'c' - 17: 1, # 'd' - 1: 0, # 'e' - 27: 1, # 'f' - 12: 3, # 'g' - 20: 0, # 'h' - 9: 0, # 'i' - 22: 1, # 'j' - 7: 1, # 'k' - 6: 2, # 'l' - 13: 2, # 'm' - 4: 3, # 'n' - 8: 0, # 'o' - 23: 1, # 'p' - 10: 3, # 'r' - 5: 2, # 's' - 3: 3, # 't' - 21: 0, # 'u' - 19: 3, # 'v' - 62: 0, # 'x' - 16: 0, # 'y' - 11: 2, # 'z' - 51: 0, # 'Á' - 44: 0, # 'É' - 61: 0, # 'Í' - 58: 0, # 'Ó' - 59: 0, # 'Ö' - 60: 0, # 'Ú' - 63: 0, # 'Ü' - 14: 0, # 'á' - 15: 0, # 'é' - 30: 0, # 'í' - 25: 0, # 'ó' - 24: 0, # 'ö' - 31: 0, # 'ú' - 29: 0, # 'ü' - 42: 0, # 'ő' - 56: 0, # 'ű' - }, - 25: { # 'ó' - 28: 0, # 'A' - 40: 0, # 'B' - 54: 0, # 'C' - 45: 0, # 'D' - 32: 0, # 'E' - 50: 0, # 'F' - 49: 0, # 'G' - 38: 0, # 'H' - 39: 0, # 'I' - 53: 0, # 'J' - 36: 0, # 'K' - 41: 0, # 'L' - 34: 0, # 'M' - 35: 0, # 'N' - 47: 0, # 'O' - 46: 0, # 'P' - 43: 0, # 'R' - 33: 0, # 'S' - 37: 0, # 'T' - 57: 0, # 'U' - 48: 0, # 'V' - 55: 0, # 'Y' - 52: 0, # 'Z' - 2: 2, # 'a' - 18: 3, # 'b' - 26: 2, # 'c' - 17: 3, # 'd' - 1: 1, # 'e' - 27: 2, # 'f' - 12: 2, # 'g' - 20: 2, # 'h' - 9: 2, # 'i' - 22: 2, # 'j' - 7: 3, # 'k' - 6: 3, # 'l' - 13: 2, # 'm' - 4: 3, # 'n' - 8: 1, # 'o' - 23: 2, # 'p' - 10: 3, # 'r' - 5: 3, # 's' - 3: 3, # 't' - 21: 1, # 'u' - 19: 2, # 'v' - 62: 0, # 'x' - 16: 0, # 'y' - 11: 3, # 'z' - 51: 0, # 'Á' - 44: 0, # 'É' - 61: 0, # 'Í' - 58: 0, # 'Ó' - 59: 0, # 'Ö' - 60: 0, # 'Ú' - 63: 0, # 'Ü' - 14: 1, # 'á' - 15: 1, # 'é' - 30: 1, # 'í' - 25: 0, # 'ó' - 24: 1, # 'ö' - 31: 1, # 'ú' - 29: 1, # 'ü' - 42: 0, # 'ő' - 56: 0, # 'ű' - }, - 24: { # 'ö' - 28: 0, # 'A' - 40: 0, # 'B' - 54: 0, # 'C' - 45: 0, # 'D' - 32: 0, # 'E' - 50: 0, # 'F' - 49: 0, # 'G' - 38: 0, # 'H' - 39: 0, # 'I' - 53: 0, # 'J' - 36: 0, # 'K' - 41: 0, # 'L' - 34: 0, # 'M' - 35: 0, # 'N' - 47: 0, # 'O' - 46: 0, # 'P' - 43: 0, # 'R' - 33: 0, # 'S' - 37: 0, # 'T' - 57: 0, # 'U' - 48: 0, # 'V' - 55: 0, # 'Y' - 52: 0, # 'Z' - 2: 0, # 'a' - 18: 3, # 'b' - 26: 1, # 'c' - 17: 2, # 'd' - 1: 0, # 'e' - 27: 1, # 'f' - 12: 2, # 'g' - 20: 1, # 'h' - 9: 0, # 'i' - 22: 1, # 'j' - 7: 3, # 'k' - 6: 3, # 'l' - 13: 3, # 'm' - 4: 3, # 'n' - 8: 0, # 'o' - 23: 2, # 'p' - 10: 3, # 'r' - 5: 3, # 's' - 3: 3, # 't' - 21: 0, # 'u' - 19: 3, # 'v' - 62: 0, # 'x' - 16: 0, # 'y' - 11: 3, # 'z' - 51: 0, # 'Á' - 44: 0, # 'É' - 61: 0, # 'Í' - 58: 0, # 'Ó' - 59: 0, # 'Ö' - 60: 0, # 'Ú' - 63: 0, # 'Ü' - 14: 0, # 'á' - 15: 0, # 'é' - 30: 0, # 'í' - 25: 0, # 'ó' - 24: 0, # 'ö' - 31: 0, # 'ú' - 29: 0, # 'ü' - 42: 0, # 'ő' - 56: 0, # 'ű' - }, - 31: { # 'ú' - 28: 0, # 'A' - 40: 0, # 'B' - 54: 0, # 'C' - 45: 0, # 'D' - 32: 0, # 'E' - 50: 0, # 'F' - 49: 0, # 'G' - 38: 0, # 'H' - 39: 0, # 'I' - 53: 0, # 'J' - 36: 0, # 'K' - 41: 0, # 'L' - 34: 0, # 'M' - 35: 0, # 'N' - 47: 0, # 'O' - 46: 0, # 'P' - 43: 0, # 'R' - 33: 0, # 'S' - 37: 0, # 'T' - 57: 0, # 'U' - 48: 0, # 'V' - 55: 0, # 'Y' - 52: 0, # 'Z' - 2: 1, # 'a' - 18: 1, # 'b' - 26: 2, # 'c' - 17: 1, # 'd' - 1: 1, # 'e' - 27: 2, # 'f' - 12: 3, # 'g' - 20: 1, # 'h' - 9: 1, # 'i' - 22: 3, # 'j' - 7: 1, # 'k' - 6: 3, # 'l' - 13: 1, # 'm' - 4: 2, # 'n' - 8: 0, # 'o' - 23: 1, # 'p' - 10: 3, # 'r' - 5: 3, # 's' - 3: 2, # 't' - 21: 1, # 'u' - 19: 1, # 'v' - 62: 0, # 'x' - 16: 0, # 'y' - 11: 2, # 'z' - 51: 0, # 'Á' - 44: 0, # 'É' - 61: 0, # 'Í' - 58: 0, # 'Ó' - 59: 0, # 'Ö' - 60: 0, # 'Ú' - 63: 0, # 'Ü' - 14: 1, # 'á' - 15: 1, # 'é' - 30: 0, # 'í' - 25: 0, # 'ó' - 24: 0, # 'ö' - 31: 0, # 'ú' - 29: 0, # 'ü' - 42: 0, # 'ő' - 56: 0, # 'ű' - }, - 29: { # 'ü' - 28: 0, # 'A' - 40: 0, # 'B' - 54: 0, # 'C' - 45: 0, # 'D' - 32: 0, # 'E' - 50: 0, # 'F' - 49: 0, # 'G' - 38: 0, # 'H' - 39: 0, # 'I' - 53: 0, # 'J' - 36: 0, # 'K' - 41: 0, # 'L' - 34: 0, # 'M' - 35: 0, # 'N' - 47: 0, # 'O' - 46: 0, # 'P' - 43: 0, # 'R' - 33: 0, # 'S' - 37: 0, # 'T' - 57: 0, # 'U' - 48: 0, # 'V' - 55: 0, # 'Y' - 52: 0, # 'Z' - 2: 1, # 'a' - 18: 1, # 'b' - 26: 1, # 'c' - 17: 2, # 'd' - 1: 1, # 'e' - 27: 1, # 'f' - 12: 3, # 'g' - 20: 2, # 'h' - 9: 1, # 'i' - 22: 1, # 'j' - 7: 3, # 'k' - 6: 3, # 'l' - 13: 1, # 'm' - 4: 3, # 'n' - 8: 0, # 'o' - 23: 1, # 'p' - 10: 2, # 'r' - 5: 2, # 's' - 3: 2, # 't' - 21: 0, # 'u' - 19: 2, # 'v' - 62: 0, # 'x' - 16: 0, # 'y' - 11: 2, # 'z' - 51: 0, # 'Á' - 44: 0, # 'É' - 61: 0, # 'Í' - 58: 0, # 'Ó' - 59: 0, # 'Ö' - 60: 0, # 'Ú' - 63: 0, # 'Ü' - 14: 0, # 'á' - 15: 1, # 'é' - 30: 0, # 'í' - 25: 0, # 'ó' - 24: 0, # 'ö' - 31: 0, # 'ú' - 29: 0, # 'ü' - 42: 0, # 'ő' - 56: 0, # 'ű' - }, - 42: { # 'ő' - 28: 0, # 'A' - 40: 0, # 'B' - 54: 0, # 'C' - 45: 0, # 'D' - 32: 0, # 'E' - 50: 0, # 'F' - 49: 0, # 'G' - 38: 0, # 'H' - 39: 0, # 'I' - 53: 0, # 'J' - 36: 0, # 'K' - 41: 0, # 'L' - 34: 0, # 'M' - 35: 0, # 'N' - 47: 0, # 'O' - 46: 0, # 'P' - 43: 0, # 'R' - 33: 0, # 'S' - 37: 0, # 'T' - 57: 0, # 'U' - 48: 0, # 'V' - 55: 0, # 'Y' - 52: 0, # 'Z' - 2: 1, # 'a' - 18: 2, # 'b' - 26: 1, # 'c' - 17: 2, # 'd' - 1: 1, # 'e' - 27: 1, # 'f' - 12: 1, # 'g' - 20: 1, # 'h' - 9: 1, # 'i' - 22: 1, # 'j' - 7: 2, # 'k' - 6: 3, # 'l' - 13: 1, # 'm' - 4: 2, # 'n' - 8: 1, # 'o' - 23: 1, # 'p' - 10: 2, # 'r' - 5: 2, # 's' - 3: 2, # 't' - 21: 1, # 'u' - 19: 1, # 'v' - 62: 0, # 'x' - 16: 0, # 'y' - 11: 2, # 'z' - 51: 0, # 'Á' - 44: 0, # 'É' - 61: 0, # 'Í' - 58: 0, # 'Ó' - 59: 0, # 'Ö' - 60: 0, # 'Ú' - 63: 0, # 'Ü' - 14: 0, # 'á' - 15: 1, # 'é' - 30: 1, # 'í' - 25: 0, # 'ó' - 24: 0, # 'ö' - 31: 0, # 'ú' - 29: 1, # 'ü' - 42: 0, # 'ő' - 56: 0, # 'ű' - }, - 56: { # 'ű' - 28: 0, # 'A' - 40: 0, # 'B' - 54: 0, # 'C' - 45: 0, # 'D' - 32: 0, # 'E' - 50: 0, # 'F' - 49: 0, # 'G' - 38: 0, # 'H' - 39: 0, # 'I' - 53: 0, # 'J' - 36: 0, # 'K' - 41: 0, # 'L' - 34: 0, # 'M' - 35: 0, # 'N' - 47: 0, # 'O' - 46: 0, # 'P' - 43: 0, # 'R' - 33: 0, # 'S' - 37: 0, # 'T' - 57: 0, # 'U' - 48: 0, # 'V' - 55: 0, # 'Y' - 52: 0, # 'Z' - 2: 1, # 'a' - 18: 1, # 'b' - 26: 0, # 'c' - 17: 1, # 'd' - 1: 1, # 'e' - 27: 1, # 'f' - 12: 1, # 'g' - 20: 1, # 'h' - 9: 1, # 'i' - 22: 1, # 'j' - 7: 1, # 'k' - 6: 1, # 'l' - 13: 0, # 'm' - 4: 2, # 'n' - 8: 0, # 'o' - 23: 0, # 'p' - 10: 1, # 'r' - 5: 1, # 's' - 3: 1, # 't' - 21: 0, # 'u' - 19: 1, # 'v' - 62: 0, # 'x' - 16: 0, # 'y' - 11: 2, # 'z' - 51: 0, # 'Á' - 44: 0, # 'É' - 61: 0, # 'Í' - 58: 0, # 'Ó' - 59: 0, # 'Ö' - 60: 0, # 'Ú' - 63: 0, # 'Ü' - 14: 0, # 'á' - 15: 0, # 'é' - 30: 0, # 'í' - 25: 0, # 'ó' - 24: 0, # 'ö' - 31: 0, # 'ú' - 29: 0, # 'ü' - 42: 0, # 'ő' - 56: 0, # 'ű' - }, -} - -# 255: Undefined characters that did not exist in training text -# 254: Carriage/Return -# 253: symbol (punctuation) that does not belong to word -# 252: 0 - 9 -# 251: Control characters - -# Character Mapping Table(s): -WINDOWS_1250_HUNGARIAN_CHAR_TO_ORDER = { - 0: 255, # '\x00' - 1: 255, # '\x01' - 2: 255, # '\x02' - 3: 255, # '\x03' - 4: 255, # '\x04' - 5: 255, # '\x05' - 6: 255, # '\x06' - 7: 255, # '\x07' - 8: 255, # '\x08' - 9: 255, # '\t' - 10: 254, # '\n' - 11: 255, # '\x0b' - 12: 255, # '\x0c' - 13: 254, # '\r' - 14: 255, # '\x0e' - 15: 255, # '\x0f' - 16: 255, # '\x10' - 17: 255, # '\x11' - 18: 255, # '\x12' - 19: 255, # '\x13' - 20: 255, # '\x14' - 21: 255, # '\x15' - 22: 255, # '\x16' - 23: 255, # '\x17' - 24: 255, # '\x18' - 25: 255, # '\x19' - 26: 255, # '\x1a' - 27: 255, # '\x1b' - 28: 255, # '\x1c' - 29: 255, # '\x1d' - 30: 255, # '\x1e' - 31: 255, # '\x1f' - 32: 253, # ' ' - 33: 253, # '!' - 34: 253, # '"' - 35: 253, # '#' - 36: 253, # '$' - 37: 253, # '%' - 38: 253, # '&' - 39: 253, # "'" - 40: 253, # '(' - 41: 253, # ')' - 42: 253, # '*' - 43: 253, # '+' - 44: 253, # ',' - 45: 253, # '-' - 46: 253, # '.' - 47: 253, # '/' - 48: 252, # '0' - 49: 252, # '1' - 50: 252, # '2' - 51: 252, # '3' - 52: 252, # '4' - 53: 252, # '5' - 54: 252, # '6' - 55: 252, # '7' - 56: 252, # '8' - 57: 252, # '9' - 58: 253, # ':' - 59: 253, # ';' - 60: 253, # '<' - 61: 253, # '=' - 62: 253, # '>' - 63: 253, # '?' - 64: 253, # '@' - 65: 28, # 'A' - 66: 40, # 'B' - 67: 54, # 'C' - 68: 45, # 'D' - 69: 32, # 'E' - 70: 50, # 'F' - 71: 49, # 'G' - 72: 38, # 'H' - 73: 39, # 'I' - 74: 53, # 'J' - 75: 36, # 'K' - 76: 41, # 'L' - 77: 34, # 'M' - 78: 35, # 'N' - 79: 47, # 'O' - 80: 46, # 'P' - 81: 72, # 'Q' - 82: 43, # 'R' - 83: 33, # 'S' - 84: 37, # 'T' - 85: 57, # 'U' - 86: 48, # 'V' - 87: 64, # 'W' - 88: 68, # 'X' - 89: 55, # 'Y' - 90: 52, # 'Z' - 91: 253, # '[' - 92: 253, # '\\' - 93: 253, # ']' - 94: 253, # '^' - 95: 253, # '_' - 96: 253, # '`' - 97: 2, # 'a' - 98: 18, # 'b' - 99: 26, # 'c' - 100: 17, # 'd' - 101: 1, # 'e' - 102: 27, # 'f' - 103: 12, # 'g' - 104: 20, # 'h' - 105: 9, # 'i' - 106: 22, # 'j' - 107: 7, # 'k' - 108: 6, # 'l' - 109: 13, # 'm' - 110: 4, # 'n' - 111: 8, # 'o' - 112: 23, # 'p' - 113: 67, # 'q' - 114: 10, # 'r' - 115: 5, # 's' - 116: 3, # 't' - 117: 21, # 'u' - 118: 19, # 'v' - 119: 65, # 'w' - 120: 62, # 'x' - 121: 16, # 'y' - 122: 11, # 'z' - 123: 253, # '{' - 124: 253, # '|' - 125: 253, # '}' - 126: 253, # '~' - 127: 253, # '\x7f' - 128: 161, # '€' - 129: 162, # None - 130: 163, # '‚' - 131: 164, # None - 132: 165, # '„' - 133: 166, # '…' - 134: 167, # '†' - 135: 168, # '‡' - 136: 169, # None - 137: 170, # '‰' - 138: 171, # 'Š' - 139: 172, # '‹' - 140: 173, # 'Ś' - 141: 174, # 'Ť' - 142: 175, # 'Ž' - 143: 176, # 'Ź' - 144: 177, # None - 145: 178, # '‘' - 146: 179, # '’' - 147: 180, # '“' - 148: 78, # '”' - 149: 181, # '•' - 150: 69, # '–' - 151: 182, # '—' - 152: 183, # None - 153: 184, # '™' - 154: 185, # 'š' - 155: 186, # '›' - 156: 187, # 'ś' - 157: 188, # 'ť' - 158: 189, # 'ž' - 159: 190, # 'ź' - 160: 191, # '\xa0' - 161: 192, # 'ˇ' - 162: 193, # '˘' - 163: 194, # 'Ł' - 164: 195, # '¤' - 165: 196, # 'Ą' - 166: 197, # '¦' - 167: 76, # '§' - 168: 198, # '¨' - 169: 199, # '©' - 170: 200, # 'Ş' - 171: 201, # '«' - 172: 202, # '¬' - 173: 203, # '\xad' - 174: 204, # '®' - 175: 205, # 'Ż' - 176: 81, # '°' - 177: 206, # '±' - 178: 207, # '˛' - 179: 208, # 'ł' - 180: 209, # '´' - 181: 210, # 'µ' - 182: 211, # '¶' - 183: 212, # '·' - 184: 213, # '¸' - 185: 214, # 'ą' - 186: 215, # 'ş' - 187: 216, # '»' - 188: 217, # 'Ľ' - 189: 218, # '˝' - 190: 219, # 'ľ' - 191: 220, # 'ż' - 192: 221, # 'Ŕ' - 193: 51, # 'Á' - 194: 83, # 'Â' - 195: 222, # 'Ă' - 196: 80, # 'Ä' - 197: 223, # 'Ĺ' - 198: 224, # 'Ć' - 199: 225, # 'Ç' - 200: 226, # 'Č' - 201: 44, # 'É' - 202: 227, # 'Ę' - 203: 228, # 'Ë' - 204: 229, # 'Ě' - 205: 61, # 'Í' - 206: 230, # 'Î' - 207: 231, # 'Ď' - 208: 232, # 'Đ' - 209: 233, # 'Ń' - 210: 234, # 'Ň' - 211: 58, # 'Ó' - 212: 235, # 'Ô' - 213: 66, # 'Ő' - 214: 59, # 'Ö' - 215: 236, # '×' - 216: 237, # 'Ř' - 217: 238, # 'Ů' - 218: 60, # 'Ú' - 219: 70, # 'Ű' - 220: 63, # 'Ü' - 221: 239, # 'Ý' - 222: 240, # 'Ţ' - 223: 241, # 'ß' - 224: 84, # 'ŕ' - 225: 14, # 'á' - 226: 75, # 'â' - 227: 242, # 'ă' - 228: 71, # 'ä' - 229: 82, # 'ĺ' - 230: 243, # 'ć' - 231: 73, # 'ç' - 232: 244, # 'č' - 233: 15, # 'é' - 234: 85, # 'ę' - 235: 79, # 'ë' - 236: 86, # 'ě' - 237: 30, # 'í' - 238: 77, # 'î' - 239: 87, # 'ď' - 240: 245, # 'đ' - 241: 246, # 'ń' - 242: 247, # 'ň' - 243: 25, # 'ó' - 244: 74, # 'ô' - 245: 42, # 'ő' - 246: 24, # 'ö' - 247: 248, # '÷' - 248: 249, # 'ř' - 249: 250, # 'ů' - 250: 31, # 'ú' - 251: 56, # 'ű' - 252: 29, # 'ü' - 253: 251, # 'ý' - 254: 252, # 'ţ' - 255: 253, # '˙' -} - -WINDOWS_1250_HUNGARIAN_MODEL = SingleByteCharSetModel( - charset_name="windows-1250", - language="Hungarian", - char_to_order_map=WINDOWS_1250_HUNGARIAN_CHAR_TO_ORDER, - language_model=HUNGARIAN_LANG_MODEL, - typical_positive_ratio=0.947368, - keep_ascii_letters=True, - alphabet="ABCDEFGHIJKLMNOPRSTUVZabcdefghijklmnoprstuvzÁÉÍÓÖÚÜáéíóöúüŐőŰű", -) - -ISO_8859_2_HUNGARIAN_CHAR_TO_ORDER = { - 0: 255, # '\x00' - 1: 255, # '\x01' - 2: 255, # '\x02' - 3: 255, # '\x03' - 4: 255, # '\x04' - 5: 255, # '\x05' - 6: 255, # '\x06' - 7: 255, # '\x07' - 8: 255, # '\x08' - 9: 255, # '\t' - 10: 254, # '\n' - 11: 255, # '\x0b' - 12: 255, # '\x0c' - 13: 254, # '\r' - 14: 255, # '\x0e' - 15: 255, # '\x0f' - 16: 255, # '\x10' - 17: 255, # '\x11' - 18: 255, # '\x12' - 19: 255, # '\x13' - 20: 255, # '\x14' - 21: 255, # '\x15' - 22: 255, # '\x16' - 23: 255, # '\x17' - 24: 255, # '\x18' - 25: 255, # '\x19' - 26: 255, # '\x1a' - 27: 255, # '\x1b' - 28: 255, # '\x1c' - 29: 255, # '\x1d' - 30: 255, # '\x1e' - 31: 255, # '\x1f' - 32: 253, # ' ' - 33: 253, # '!' - 34: 253, # '"' - 35: 253, # '#' - 36: 253, # '$' - 37: 253, # '%' - 38: 253, # '&' - 39: 253, # "'" - 40: 253, # '(' - 41: 253, # ')' - 42: 253, # '*' - 43: 253, # '+' - 44: 253, # ',' - 45: 253, # '-' - 46: 253, # '.' - 47: 253, # '/' - 48: 252, # '0' - 49: 252, # '1' - 50: 252, # '2' - 51: 252, # '3' - 52: 252, # '4' - 53: 252, # '5' - 54: 252, # '6' - 55: 252, # '7' - 56: 252, # '8' - 57: 252, # '9' - 58: 253, # ':' - 59: 253, # ';' - 60: 253, # '<' - 61: 253, # '=' - 62: 253, # '>' - 63: 253, # '?' - 64: 253, # '@' - 65: 28, # 'A' - 66: 40, # 'B' - 67: 54, # 'C' - 68: 45, # 'D' - 69: 32, # 'E' - 70: 50, # 'F' - 71: 49, # 'G' - 72: 38, # 'H' - 73: 39, # 'I' - 74: 53, # 'J' - 75: 36, # 'K' - 76: 41, # 'L' - 77: 34, # 'M' - 78: 35, # 'N' - 79: 47, # 'O' - 80: 46, # 'P' - 81: 71, # 'Q' - 82: 43, # 'R' - 83: 33, # 'S' - 84: 37, # 'T' - 85: 57, # 'U' - 86: 48, # 'V' - 87: 64, # 'W' - 88: 68, # 'X' - 89: 55, # 'Y' - 90: 52, # 'Z' - 91: 253, # '[' - 92: 253, # '\\' - 93: 253, # ']' - 94: 253, # '^' - 95: 253, # '_' - 96: 253, # '`' - 97: 2, # 'a' - 98: 18, # 'b' - 99: 26, # 'c' - 100: 17, # 'd' - 101: 1, # 'e' - 102: 27, # 'f' - 103: 12, # 'g' - 104: 20, # 'h' - 105: 9, # 'i' - 106: 22, # 'j' - 107: 7, # 'k' - 108: 6, # 'l' - 109: 13, # 'm' - 110: 4, # 'n' - 111: 8, # 'o' - 112: 23, # 'p' - 113: 67, # 'q' - 114: 10, # 'r' - 115: 5, # 's' - 116: 3, # 't' - 117: 21, # 'u' - 118: 19, # 'v' - 119: 65, # 'w' - 120: 62, # 'x' - 121: 16, # 'y' - 122: 11, # 'z' - 123: 253, # '{' - 124: 253, # '|' - 125: 253, # '}' - 126: 253, # '~' - 127: 253, # '\x7f' - 128: 159, # '\x80' - 129: 160, # '\x81' - 130: 161, # '\x82' - 131: 162, # '\x83' - 132: 163, # '\x84' - 133: 164, # '\x85' - 134: 165, # '\x86' - 135: 166, # '\x87' - 136: 167, # '\x88' - 137: 168, # '\x89' - 138: 169, # '\x8a' - 139: 170, # '\x8b' - 140: 171, # '\x8c' - 141: 172, # '\x8d' - 142: 173, # '\x8e' - 143: 174, # '\x8f' - 144: 175, # '\x90' - 145: 176, # '\x91' - 146: 177, # '\x92' - 147: 178, # '\x93' - 148: 179, # '\x94' - 149: 180, # '\x95' - 150: 181, # '\x96' - 151: 182, # '\x97' - 152: 183, # '\x98' - 153: 184, # '\x99' - 154: 185, # '\x9a' - 155: 186, # '\x9b' - 156: 187, # '\x9c' - 157: 188, # '\x9d' - 158: 189, # '\x9e' - 159: 190, # '\x9f' - 160: 191, # '\xa0' - 161: 192, # 'Ą' - 162: 193, # '˘' - 163: 194, # 'Ł' - 164: 195, # '¤' - 165: 196, # 'Ľ' - 166: 197, # 'Ś' - 167: 75, # '§' - 168: 198, # '¨' - 169: 199, # 'Š' - 170: 200, # 'Ş' - 171: 201, # 'Ť' - 172: 202, # 'Ź' - 173: 203, # '\xad' - 174: 204, # 'Ž' - 175: 205, # 'Ż' - 176: 79, # '°' - 177: 206, # 'ą' - 178: 207, # '˛' - 179: 208, # 'ł' - 180: 209, # '´' - 181: 210, # 'ľ' - 182: 211, # 'ś' - 183: 212, # 'ˇ' - 184: 213, # '¸' - 185: 214, # 'š' - 186: 215, # 'ş' - 187: 216, # 'ť' - 188: 217, # 'ź' - 189: 218, # '˝' - 190: 219, # 'ž' - 191: 220, # 'ż' - 192: 221, # 'Ŕ' - 193: 51, # 'Á' - 194: 81, # 'Â' - 195: 222, # 'Ă' - 196: 78, # 'Ä' - 197: 223, # 'Ĺ' - 198: 224, # 'Ć' - 199: 225, # 'Ç' - 200: 226, # 'Č' - 201: 44, # 'É' - 202: 227, # 'Ę' - 203: 228, # 'Ë' - 204: 229, # 'Ě' - 205: 61, # 'Í' - 206: 230, # 'Î' - 207: 231, # 'Ď' - 208: 232, # 'Đ' - 209: 233, # 'Ń' - 210: 234, # 'Ň' - 211: 58, # 'Ó' - 212: 235, # 'Ô' - 213: 66, # 'Ő' - 214: 59, # 'Ö' - 215: 236, # '×' - 216: 237, # 'Ř' - 217: 238, # 'Ů' - 218: 60, # 'Ú' - 219: 69, # 'Ű' - 220: 63, # 'Ü' - 221: 239, # 'Ý' - 222: 240, # 'Ţ' - 223: 241, # 'ß' - 224: 82, # 'ŕ' - 225: 14, # 'á' - 226: 74, # 'â' - 227: 242, # 'ă' - 228: 70, # 'ä' - 229: 80, # 'ĺ' - 230: 243, # 'ć' - 231: 72, # 'ç' - 232: 244, # 'č' - 233: 15, # 'é' - 234: 83, # 'ę' - 235: 77, # 'ë' - 236: 84, # 'ě' - 237: 30, # 'í' - 238: 76, # 'î' - 239: 85, # 'ď' - 240: 245, # 'đ' - 241: 246, # 'ń' - 242: 247, # 'ň' - 243: 25, # 'ó' - 244: 73, # 'ô' - 245: 42, # 'ő' - 246: 24, # 'ö' - 247: 248, # '÷' - 248: 249, # 'ř' - 249: 250, # 'ů' - 250: 31, # 'ú' - 251: 56, # 'ű' - 252: 29, # 'ü' - 253: 251, # 'ý' - 254: 252, # 'ţ' - 255: 253, # '˙' -} - -ISO_8859_2_HUNGARIAN_MODEL = SingleByteCharSetModel( - charset_name="ISO-8859-2", - language="Hungarian", - char_to_order_map=ISO_8859_2_HUNGARIAN_CHAR_TO_ORDER, - language_model=HUNGARIAN_LANG_MODEL, - typical_positive_ratio=0.947368, - keep_ascii_letters=True, - alphabet="ABCDEFGHIJKLMNOPRSTUVZabcdefghijklmnoprstuvzÁÉÍÓÖÚÜáéíóöúüŐőŰű", -) diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/langrussianmodel.py b/venv/lib/python3.11/site-packages/pip/_vendor/chardet/langrussianmodel.py deleted file mode 100644 index 39a5388..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/langrussianmodel.py +++ /dev/null @@ -1,5725 +0,0 @@ -from pip._vendor.chardet.sbcharsetprober import SingleByteCharSetModel - -# 3: Positive -# 2: Likely -# 1: Unlikely -# 0: Negative - -RUSSIAN_LANG_MODEL = { - 37: { # 'А' - 37: 0, # 'А' - 44: 1, # 'Б' - 33: 1, # 'В' - 46: 1, # 'Г' - 41: 1, # 'Д' - 48: 1, # 'Е' - 56: 1, # 'Ж' - 51: 1, # 'З' - 42: 1, # 'И' - 60: 1, # 'Й' - 36: 1, # 'К' - 49: 1, # 'Л' - 38: 1, # 'М' - 31: 2, # 'Н' - 34: 1, # 'О' - 35: 1, # 'П' - 45: 1, # 'Р' - 32: 1, # 'С' - 40: 1, # 'Т' - 52: 1, # 'У' - 53: 1, # 'Ф' - 55: 1, # 'Х' - 58: 1, # 'Ц' - 50: 1, # 'Ч' - 57: 1, # 'Ш' - 63: 1, # 'Щ' - 62: 0, # 'Ы' - 61: 0, # 'Ь' - 47: 0, # 'Э' - 59: 1, # 'Ю' - 43: 1, # 'Я' - 3: 1, # 'а' - 21: 2, # 'б' - 10: 2, # 'в' - 19: 2, # 'г' - 13: 2, # 'д' - 2: 0, # 'е' - 24: 1, # 'ж' - 20: 1, # 'з' - 4: 0, # 'и' - 23: 1, # 'й' - 11: 2, # 'к' - 8: 3, # 'л' - 12: 2, # 'м' - 5: 2, # 'н' - 1: 0, # 'о' - 15: 2, # 'п' - 9: 2, # 'р' - 7: 2, # 'с' - 6: 2, # 'т' - 14: 2, # 'у' - 39: 2, # 'ф' - 26: 2, # 'х' - 28: 0, # 'ц' - 22: 1, # 'ч' - 25: 2, # 'ш' - 29: 0, # 'щ' - 54: 0, # 'ъ' - 18: 0, # 'ы' - 17: 0, # 'ь' - 30: 1, # 'э' - 27: 0, # 'ю' - 16: 0, # 'я' - }, - 44: { # 'Б' - 37: 1, # 'А' - 44: 0, # 'Б' - 33: 1, # 'В' - 46: 1, # 'Г' - 41: 0, # 'Д' - 48: 1, # 'Е' - 56: 0, # 'Ж' - 51: 0, # 'З' - 42: 1, # 'И' - 60: 0, # 'Й' - 36: 0, # 'К' - 49: 1, # 'Л' - 38: 1, # 'М' - 31: 1, # 'Н' - 34: 1, # 'О' - 35: 0, # 'П' - 45: 1, # 'Р' - 32: 0, # 'С' - 40: 0, # 'Т' - 52: 1, # 'У' - 53: 0, # 'Ф' - 55: 0, # 'Х' - 58: 0, # 'Ц' - 50: 0, # 'Ч' - 57: 0, # 'Ш' - 63: 0, # 'Щ' - 62: 1, # 'Ы' - 61: 0, # 'Ь' - 47: 0, # 'Э' - 59: 0, # 'Ю' - 43: 1, # 'Я' - 3: 2, # 'а' - 21: 0, # 'б' - 10: 0, # 'в' - 19: 0, # 'г' - 13: 1, # 'д' - 2: 3, # 'е' - 24: 0, # 'ж' - 20: 0, # 'з' - 4: 2, # 'и' - 23: 0, # 'й' - 11: 0, # 'к' - 8: 2, # 'л' - 12: 0, # 'м' - 5: 0, # 'н' - 1: 3, # 'о' - 15: 0, # 'п' - 9: 2, # 'р' - 7: 0, # 'с' - 6: 0, # 'т' - 14: 2, # 'у' - 39: 0, # 'ф' - 26: 0, # 'х' - 28: 0, # 'ц' - 22: 0, # 'ч' - 25: 0, # 'ш' - 29: 0, # 'щ' - 54: 0, # 'ъ' - 18: 2, # 'ы' - 17: 1, # 'ь' - 30: 2, # 'э' - 27: 1, # 'ю' - 16: 1, # 'я' - }, - 33: { # 'В' - 37: 2, # 'А' - 44: 0, # 'Б' - 33: 1, # 'В' - 46: 0, # 'Г' - 41: 1, # 'Д' - 48: 1, # 'Е' - 56: 0, # 'Ж' - 51: 0, # 'З' - 42: 1, # 'И' - 60: 0, # 'Й' - 36: 1, # 'К' - 49: 1, # 'Л' - 38: 1, # 'М' - 31: 1, # 'Н' - 34: 1, # 'О' - 35: 1, # 'П' - 45: 1, # 'Р' - 32: 1, # 'С' - 40: 1, # 'Т' - 52: 1, # 'У' - 53: 0, # 'Ф' - 55: 0, # 'Х' - 58: 0, # 'Ц' - 50: 0, # 'Ч' - 57: 1, # 'Ш' - 63: 0, # 'Щ' - 62: 1, # 'Ы' - 61: 1, # 'Ь' - 47: 0, # 'Э' - 59: 0, # 'Ю' - 43: 1, # 'Я' - 3: 2, # 'а' - 21: 1, # 'б' - 10: 1, # 'в' - 19: 1, # 'г' - 13: 2, # 'д' - 2: 3, # 'е' - 24: 0, # 'ж' - 20: 2, # 'з' - 4: 2, # 'и' - 23: 0, # 'й' - 11: 1, # 'к' - 8: 2, # 'л' - 12: 2, # 'м' - 5: 2, # 'н' - 1: 3, # 'о' - 15: 2, # 'п' - 9: 2, # 'р' - 7: 3, # 'с' - 6: 2, # 'т' - 14: 2, # 'у' - 39: 0, # 'ф' - 26: 1, # 'х' - 28: 1, # 'ц' - 22: 2, # 'ч' - 25: 1, # 'ш' - 29: 0, # 'щ' - 54: 1, # 'ъ' - 18: 3, # 'ы' - 17: 1, # 'ь' - 30: 2, # 'э' - 27: 0, # 'ю' - 16: 1, # 'я' - }, - 46: { # 'Г' - 37: 1, # 'А' - 44: 1, # 'Б' - 33: 0, # 'В' - 46: 0, # 'Г' - 41: 1, # 'Д' - 48: 1, # 'Е' - 56: 0, # 'Ж' - 51: 0, # 'З' - 42: 1, # 'И' - 60: 0, # 'Й' - 36: 0, # 'К' - 49: 1, # 'Л' - 38: 1, # 'М' - 31: 1, # 'Н' - 34: 1, # 'О' - 35: 1, # 'П' - 45: 1, # 'Р' - 32: 0, # 'С' - 40: 0, # 'Т' - 52: 1, # 'У' - 53: 0, # 'Ф' - 55: 0, # 'Х' - 58: 0, # 'Ц' - 50: 0, # 'Ч' - 57: 0, # 'Ш' - 63: 0, # 'Щ' - 62: 0, # 'Ы' - 61: 0, # 'Ь' - 47: 0, # 'Э' - 59: 0, # 'Ю' - 43: 0, # 'Я' - 3: 2, # 'а' - 21: 0, # 'б' - 10: 1, # 'в' - 19: 0, # 'г' - 13: 2, # 'д' - 2: 2, # 'е' - 24: 0, # 'ж' - 20: 0, # 'з' - 4: 2, # 'и' - 23: 0, # 'й' - 11: 0, # 'к' - 8: 2, # 'л' - 12: 1, # 'м' - 5: 1, # 'н' - 1: 3, # 'о' - 15: 0, # 'п' - 9: 2, # 'р' - 7: 0, # 'с' - 6: 0, # 'т' - 14: 2, # 'у' - 39: 0, # 'ф' - 26: 0, # 'х' - 28: 0, # 'ц' - 22: 0, # 'ч' - 25: 0, # 'ш' - 29: 0, # 'щ' - 54: 0, # 'ъ' - 18: 0, # 'ы' - 17: 1, # 'ь' - 30: 1, # 'э' - 27: 1, # 'ю' - 16: 0, # 'я' - }, - 41: { # 'Д' - 37: 1, # 'А' - 44: 0, # 'Б' - 33: 1, # 'В' - 46: 0, # 'Г' - 41: 0, # 'Д' - 48: 2, # 'Е' - 56: 1, # 'Ж' - 51: 0, # 'З' - 42: 1, # 'И' - 60: 0, # 'Й' - 36: 1, # 'К' - 49: 1, # 'Л' - 38: 0, # 'М' - 31: 1, # 'Н' - 34: 1, # 'О' - 35: 0, # 'П' - 45: 1, # 'Р' - 32: 1, # 'С' - 40: 0, # 'Т' - 52: 1, # 'У' - 53: 0, # 'Ф' - 55: 0, # 'Х' - 58: 1, # 'Ц' - 50: 1, # 'Ч' - 57: 0, # 'Ш' - 63: 0, # 'Щ' - 62: 1, # 'Ы' - 61: 1, # 'Ь' - 47: 0, # 'Э' - 59: 0, # 'Ю' - 43: 1, # 'Я' - 3: 3, # 'а' - 21: 0, # 'б' - 10: 2, # 'в' - 19: 0, # 'г' - 13: 0, # 'д' - 2: 2, # 'е' - 24: 3, # 'ж' - 20: 1, # 'з' - 4: 2, # 'и' - 23: 0, # 'й' - 11: 0, # 'к' - 8: 2, # 'л' - 12: 1, # 'м' - 5: 1, # 'н' - 1: 3, # 'о' - 15: 0, # 'п' - 9: 2, # 'р' - 7: 0, # 'с' - 6: 0, # 'т' - 14: 2, # 'у' - 39: 0, # 'ф' - 26: 1, # 'х' - 28: 0, # 'ц' - 22: 0, # 'ч' - 25: 0, # 'ш' - 29: 0, # 'щ' - 54: 0, # 'ъ' - 18: 1, # 'ы' - 17: 1, # 'ь' - 30: 2, # 'э' - 27: 1, # 'ю' - 16: 1, # 'я' - }, - 48: { # 'Е' - 37: 1, # 'А' - 44: 1, # 'Б' - 33: 1, # 'В' - 46: 1, # 'Г' - 41: 1, # 'Д' - 48: 1, # 'Е' - 56: 1, # 'Ж' - 51: 1, # 'З' - 42: 1, # 'И' - 60: 1, # 'Й' - 36: 1, # 'К' - 49: 1, # 'Л' - 38: 1, # 'М' - 31: 2, # 'Н' - 34: 1, # 'О' - 35: 1, # 'П' - 45: 2, # 'Р' - 32: 2, # 'С' - 40: 1, # 'Т' - 52: 0, # 'У' - 53: 0, # 'Ф' - 55: 1, # 'Х' - 58: 1, # 'Ц' - 50: 1, # 'Ч' - 57: 1, # 'Ш' - 63: 1, # 'Щ' - 62: 0, # 'Ы' - 61: 0, # 'Ь' - 47: 0, # 'Э' - 59: 0, # 'Ю' - 43: 1, # 'Я' - 3: 0, # 'а' - 21: 0, # 'б' - 10: 2, # 'в' - 19: 2, # 'г' - 13: 2, # 'д' - 2: 2, # 'е' - 24: 1, # 'ж' - 20: 1, # 'з' - 4: 0, # 'и' - 23: 2, # 'й' - 11: 1, # 'к' - 8: 2, # 'л' - 12: 2, # 'м' - 5: 1, # 'н' - 1: 0, # 'о' - 15: 1, # 'п' - 9: 1, # 'р' - 7: 3, # 'с' - 6: 0, # 'т' - 14: 0, # 'у' - 39: 1, # 'ф' - 26: 1, # 'х' - 28: 0, # 'ц' - 22: 0, # 'ч' - 25: 1, # 'ш' - 29: 2, # 'щ' - 54: 0, # 'ъ' - 18: 0, # 'ы' - 17: 0, # 'ь' - 30: 0, # 'э' - 27: 1, # 'ю' - 16: 0, # 'я' - }, - 56: { # 'Ж' - 37: 1, # 'А' - 44: 0, # 'Б' - 33: 0, # 'В' - 46: 0, # 'Г' - 41: 1, # 'Д' - 48: 1, # 'Е' - 56: 0, # 'Ж' - 51: 1, # 'З' - 42: 1, # 'И' - 60: 0, # 'Й' - 36: 0, # 'К' - 49: 0, # 'Л' - 38: 0, # 'М' - 31: 1, # 'Н' - 34: 1, # 'О' - 35: 0, # 'П' - 45: 0, # 'Р' - 32: 0, # 'С' - 40: 0, # 'Т' - 52: 1, # 'У' - 53: 0, # 'Ф' - 55: 0, # 'Х' - 58: 0, # 'Ц' - 50: 0, # 'Ч' - 57: 0, # 'Ш' - 63: 0, # 'Щ' - 62: 0, # 'Ы' - 61: 0, # 'Ь' - 47: 0, # 'Э' - 59: 0, # 'Ю' - 43: 0, # 'Я' - 3: 2, # 'а' - 21: 1, # 'б' - 10: 0, # 'в' - 19: 1, # 'г' - 13: 1, # 'д' - 2: 2, # 'е' - 24: 1, # 'ж' - 20: 0, # 'з' - 4: 2, # 'и' - 23: 0, # 'й' - 11: 0, # 'к' - 8: 0, # 'л' - 12: 1, # 'м' - 5: 0, # 'н' - 1: 2, # 'о' - 15: 0, # 'п' - 9: 1, # 'р' - 7: 0, # 'с' - 6: 0, # 'т' - 14: 2, # 'у' - 39: 0, # 'ф' - 26: 0, # 'х' - 28: 0, # 'ц' - 22: 0, # 'ч' - 25: 0, # 'ш' - 29: 0, # 'щ' - 54: 0, # 'ъ' - 18: 0, # 'ы' - 17: 0, # 'ь' - 30: 0, # 'э' - 27: 2, # 'ю' - 16: 0, # 'я' - }, - 51: { # 'З' - 37: 1, # 'А' - 44: 0, # 'Б' - 33: 1, # 'В' - 46: 1, # 'Г' - 41: 1, # 'Д' - 48: 1, # 'Е' - 56: 0, # 'Ж' - 51: 0, # 'З' - 42: 1, # 'И' - 60: 0, # 'Й' - 36: 0, # 'К' - 49: 1, # 'Л' - 38: 1, # 'М' - 31: 1, # 'Н' - 34: 1, # 'О' - 35: 0, # 'П' - 45: 1, # 'Р' - 32: 0, # 'С' - 40: 0, # 'Т' - 52: 1, # 'У' - 53: 0, # 'Ф' - 55: 0, # 'Х' - 58: 0, # 'Ц' - 50: 0, # 'Ч' - 57: 0, # 'Ш' - 63: 0, # 'Щ' - 62: 1, # 'Ы' - 61: 1, # 'Ь' - 47: 0, # 'Э' - 59: 0, # 'Ю' - 43: 0, # 'Я' - 3: 3, # 'а' - 21: 1, # 'б' - 10: 2, # 'в' - 19: 0, # 'г' - 13: 2, # 'д' - 2: 2, # 'е' - 24: 0, # 'ж' - 20: 0, # 'з' - 4: 2, # 'и' - 23: 0, # 'й' - 11: 0, # 'к' - 8: 1, # 'л' - 12: 1, # 'м' - 5: 2, # 'н' - 1: 2, # 'о' - 15: 0, # 'п' - 9: 1, # 'р' - 7: 0, # 'с' - 6: 0, # 'т' - 14: 1, # 'у' - 39: 0, # 'ф' - 26: 0, # 'х' - 28: 0, # 'ц' - 22: 0, # 'ч' - 25: 0, # 'ш' - 29: 0, # 'щ' - 54: 0, # 'ъ' - 18: 1, # 'ы' - 17: 0, # 'ь' - 30: 0, # 'э' - 27: 0, # 'ю' - 16: 1, # 'я' - }, - 42: { # 'И' - 37: 1, # 'А' - 44: 1, # 'Б' - 33: 1, # 'В' - 46: 1, # 'Г' - 41: 1, # 'Д' - 48: 2, # 'Е' - 56: 1, # 'Ж' - 51: 1, # 'З' - 42: 1, # 'И' - 60: 1, # 'Й' - 36: 1, # 'К' - 49: 1, # 'Л' - 38: 1, # 'М' - 31: 1, # 'Н' - 34: 1, # 'О' - 35: 1, # 'П' - 45: 1, # 'Р' - 32: 2, # 'С' - 40: 1, # 'Т' - 52: 0, # 'У' - 53: 1, # 'Ф' - 55: 1, # 'Х' - 58: 1, # 'Ц' - 50: 1, # 'Ч' - 57: 0, # 'Ш' - 63: 1, # 'Щ' - 62: 0, # 'Ы' - 61: 0, # 'Ь' - 47: 0, # 'Э' - 59: 1, # 'Ю' - 43: 1, # 'Я' - 3: 1, # 'а' - 21: 2, # 'б' - 10: 2, # 'в' - 19: 2, # 'г' - 13: 2, # 'д' - 2: 2, # 'е' - 24: 0, # 'ж' - 20: 2, # 'з' - 4: 1, # 'и' - 23: 0, # 'й' - 11: 1, # 'к' - 8: 2, # 'л' - 12: 2, # 'м' - 5: 2, # 'н' - 1: 1, # 'о' - 15: 1, # 'п' - 9: 2, # 'р' - 7: 2, # 'с' - 6: 2, # 'т' - 14: 1, # 'у' - 39: 1, # 'ф' - 26: 2, # 'х' - 28: 0, # 'ц' - 22: 0, # 'ч' - 25: 1, # 'ш' - 29: 1, # 'щ' - 54: 0, # 'ъ' - 18: 0, # 'ы' - 17: 0, # 'ь' - 30: 0, # 'э' - 27: 1, # 'ю' - 16: 0, # 'я' - }, - 60: { # 'Й' - 37: 0, # 'А' - 44: 0, # 'Б' - 33: 0, # 'В' - 46: 0, # 'Г' - 41: 1, # 'Д' - 48: 0, # 'Е' - 56: 0, # 'Ж' - 51: 0, # 'З' - 42: 0, # 'И' - 60: 0, # 'Й' - 36: 1, # 'К' - 49: 1, # 'Л' - 38: 0, # 'М' - 31: 1, # 'Н' - 34: 0, # 'О' - 35: 0, # 'П' - 45: 0, # 'Р' - 32: 1, # 'С' - 40: 1, # 'Т' - 52: 0, # 'У' - 53: 0, # 'Ф' - 55: 1, # 'Х' - 58: 1, # 'Ц' - 50: 0, # 'Ч' - 57: 0, # 'Ш' - 63: 0, # 'Щ' - 62: 0, # 'Ы' - 61: 0, # 'Ь' - 47: 0, # 'Э' - 59: 0, # 'Ю' - 43: 0, # 'Я' - 3: 0, # 'а' - 21: 0, # 'б' - 10: 0, # 'в' - 19: 0, # 'г' - 13: 0, # 'д' - 2: 1, # 'е' - 24: 0, # 'ж' - 20: 0, # 'з' - 4: 0, # 'и' - 23: 0, # 'й' - 11: 0, # 'к' - 8: 0, # 'л' - 12: 0, # 'м' - 5: 0, # 'н' - 1: 2, # 'о' - 15: 0, # 'п' - 9: 0, # 'р' - 7: 0, # 'с' - 6: 0, # 'т' - 14: 0, # 'у' - 39: 0, # 'ф' - 26: 0, # 'х' - 28: 0, # 'ц' - 22: 0, # 'ч' - 25: 0, # 'ш' - 29: 0, # 'щ' - 54: 0, # 'ъ' - 18: 0, # 'ы' - 17: 0, # 'ь' - 30: 0, # 'э' - 27: 0, # 'ю' - 16: 0, # 'я' - }, - 36: { # 'К' - 37: 2, # 'А' - 44: 0, # 'Б' - 33: 1, # 'В' - 46: 0, # 'Г' - 41: 0, # 'Д' - 48: 1, # 'Е' - 56: 0, # 'Ж' - 51: 1, # 'З' - 42: 1, # 'И' - 60: 0, # 'Й' - 36: 0, # 'К' - 49: 1, # 'Л' - 38: 0, # 'М' - 31: 1, # 'Н' - 34: 2, # 'О' - 35: 1, # 'П' - 45: 1, # 'Р' - 32: 1, # 'С' - 40: 1, # 'Т' - 52: 1, # 'У' - 53: 0, # 'Ф' - 55: 0, # 'Х' - 58: 1, # 'Ц' - 50: 0, # 'Ч' - 57: 0, # 'Ш' - 63: 0, # 'Щ' - 62: 0, # 'Ы' - 61: 0, # 'Ь' - 47: 0, # 'Э' - 59: 0, # 'Ю' - 43: 0, # 'Я' - 3: 3, # 'а' - 21: 0, # 'б' - 10: 1, # 'в' - 19: 0, # 'г' - 13: 0, # 'д' - 2: 2, # 'е' - 24: 0, # 'ж' - 20: 0, # 'з' - 4: 2, # 'и' - 23: 0, # 'й' - 11: 0, # 'к' - 8: 2, # 'л' - 12: 0, # 'м' - 5: 1, # 'н' - 1: 3, # 'о' - 15: 0, # 'п' - 9: 2, # 'р' - 7: 2, # 'с' - 6: 2, # 'т' - 14: 2, # 'у' - 39: 0, # 'ф' - 26: 1, # 'х' - 28: 0, # 'ц' - 22: 0, # 'ч' - 25: 0, # 'ш' - 29: 0, # 'щ' - 54: 0, # 'ъ' - 18: 1, # 'ы' - 17: 1, # 'ь' - 30: 2, # 'э' - 27: 1, # 'ю' - 16: 0, # 'я' - }, - 49: { # 'Л' - 37: 2, # 'А' - 44: 0, # 'Б' - 33: 0, # 'В' - 46: 1, # 'Г' - 41: 0, # 'Д' - 48: 1, # 'Е' - 56: 1, # 'Ж' - 51: 0, # 'З' - 42: 1, # 'И' - 60: 0, # 'Й' - 36: 1, # 'К' - 49: 1, # 'Л' - 38: 1, # 'М' - 31: 0, # 'Н' - 34: 1, # 'О' - 35: 1, # 'П' - 45: 0, # 'Р' - 32: 1, # 'С' - 40: 1, # 'Т' - 52: 1, # 'У' - 53: 0, # 'Ф' - 55: 0, # 'Х' - 58: 0, # 'Ц' - 50: 1, # 'Ч' - 57: 0, # 'Ш' - 63: 0, # 'Щ' - 62: 1, # 'Ы' - 61: 1, # 'Ь' - 47: 0, # 'Э' - 59: 1, # 'Ю' - 43: 1, # 'Я' - 3: 2, # 'а' - 21: 0, # 'б' - 10: 0, # 'в' - 19: 1, # 'г' - 13: 0, # 'д' - 2: 2, # 'е' - 24: 1, # 'ж' - 20: 0, # 'з' - 4: 2, # 'и' - 23: 0, # 'й' - 11: 0, # 'к' - 8: 1, # 'л' - 12: 0, # 'м' - 5: 1, # 'н' - 1: 2, # 'о' - 15: 0, # 'п' - 9: 0, # 'р' - 7: 0, # 'с' - 6: 0, # 'т' - 14: 2, # 'у' - 39: 0, # 'ф' - 26: 1, # 'х' - 28: 0, # 'ц' - 22: 0, # 'ч' - 25: 0, # 'ш' - 29: 0, # 'щ' - 54: 0, # 'ъ' - 18: 1, # 'ы' - 17: 1, # 'ь' - 30: 2, # 'э' - 27: 2, # 'ю' - 16: 1, # 'я' - }, - 38: { # 'М' - 37: 1, # 'А' - 44: 1, # 'Б' - 33: 1, # 'В' - 46: 0, # 'Г' - 41: 0, # 'Д' - 48: 1, # 'Е' - 56: 0, # 'Ж' - 51: 0, # 'З' - 42: 1, # 'И' - 60: 0, # 'Й' - 36: 1, # 'К' - 49: 1, # 'Л' - 38: 1, # 'М' - 31: 1, # 'Н' - 34: 1, # 'О' - 35: 1, # 'П' - 45: 1, # 'Р' - 32: 1, # 'С' - 40: 1, # 'Т' - 52: 1, # 'У' - 53: 1, # 'Ф' - 55: 1, # 'Х' - 58: 0, # 'Ц' - 50: 0, # 'Ч' - 57: 0, # 'Ш' - 63: 0, # 'Щ' - 62: 1, # 'Ы' - 61: 0, # 'Ь' - 47: 1, # 'Э' - 59: 0, # 'Ю' - 43: 1, # 'Я' - 3: 3, # 'а' - 21: 0, # 'б' - 10: 0, # 'в' - 19: 1, # 'г' - 13: 0, # 'д' - 2: 2, # 'е' - 24: 0, # 'ж' - 20: 0, # 'з' - 4: 3, # 'и' - 23: 0, # 'й' - 11: 0, # 'к' - 8: 1, # 'л' - 12: 1, # 'м' - 5: 2, # 'н' - 1: 3, # 'о' - 15: 0, # 'п' - 9: 1, # 'р' - 7: 1, # 'с' - 6: 0, # 'т' - 14: 2, # 'у' - 39: 0, # 'ф' - 26: 0, # 'х' - 28: 0, # 'ц' - 22: 0, # 'ч' - 25: 0, # 'ш' - 29: 0, # 'щ' - 54: 0, # 'ъ' - 18: 3, # 'ы' - 17: 1, # 'ь' - 30: 2, # 'э' - 27: 1, # 'ю' - 16: 1, # 'я' - }, - 31: { # 'Н' - 37: 2, # 'А' - 44: 0, # 'Б' - 33: 0, # 'В' - 46: 1, # 'Г' - 41: 1, # 'Д' - 48: 1, # 'Е' - 56: 0, # 'Ж' - 51: 1, # 'З' - 42: 2, # 'И' - 60: 0, # 'Й' - 36: 1, # 'К' - 49: 0, # 'Л' - 38: 0, # 'М' - 31: 1, # 'Н' - 34: 1, # 'О' - 35: 0, # 'П' - 45: 1, # 'Р' - 32: 1, # 'С' - 40: 1, # 'Т' - 52: 1, # 'У' - 53: 1, # 'Ф' - 55: 1, # 'Х' - 58: 1, # 'Ц' - 50: 1, # 'Ч' - 57: 0, # 'Ш' - 63: 0, # 'Щ' - 62: 1, # 'Ы' - 61: 1, # 'Ь' - 47: 1, # 'Э' - 59: 0, # 'Ю' - 43: 1, # 'Я' - 3: 3, # 'а' - 21: 0, # 'б' - 10: 0, # 'в' - 19: 0, # 'г' - 13: 0, # 'д' - 2: 3, # 'е' - 24: 0, # 'ж' - 20: 0, # 'з' - 4: 3, # 'и' - 23: 0, # 'й' - 11: 0, # 'к' - 8: 0, # 'л' - 12: 0, # 'м' - 5: 0, # 'н' - 1: 3, # 'о' - 15: 0, # 'п' - 9: 1, # 'р' - 7: 0, # 'с' - 6: 0, # 'т' - 14: 3, # 'у' - 39: 0, # 'ф' - 26: 1, # 'х' - 28: 0, # 'ц' - 22: 0, # 'ч' - 25: 0, # 'ш' - 29: 0, # 'щ' - 54: 0, # 'ъ' - 18: 1, # 'ы' - 17: 2, # 'ь' - 30: 1, # 'э' - 27: 1, # 'ю' - 16: 1, # 'я' - }, - 34: { # 'О' - 37: 0, # 'А' - 44: 1, # 'Б' - 33: 1, # 'В' - 46: 1, # 'Г' - 41: 2, # 'Д' - 48: 1, # 'Е' - 56: 1, # 'Ж' - 51: 1, # 'З' - 42: 1, # 'И' - 60: 1, # 'Й' - 36: 1, # 'К' - 49: 2, # 'Л' - 38: 1, # 'М' - 31: 2, # 'Н' - 34: 1, # 'О' - 35: 1, # 'П' - 45: 2, # 'Р' - 32: 1, # 'С' - 40: 1, # 'Т' - 52: 1, # 'У' - 53: 1, # 'Ф' - 55: 1, # 'Х' - 58: 0, # 'Ц' - 50: 1, # 'Ч' - 57: 1, # 'Ш' - 63: 1, # 'Щ' - 62: 0, # 'Ы' - 61: 0, # 'Ь' - 47: 0, # 'Э' - 59: 0, # 'Ю' - 43: 1, # 'Я' - 3: 1, # 'а' - 21: 2, # 'б' - 10: 1, # 'в' - 19: 2, # 'г' - 13: 2, # 'д' - 2: 0, # 'е' - 24: 1, # 'ж' - 20: 1, # 'з' - 4: 0, # 'и' - 23: 1, # 'й' - 11: 2, # 'к' - 8: 2, # 'л' - 12: 1, # 'м' - 5: 3, # 'н' - 1: 0, # 'о' - 15: 2, # 'п' - 9: 2, # 'р' - 7: 2, # 'с' - 6: 2, # 'т' - 14: 1, # 'у' - 39: 1, # 'ф' - 26: 2, # 'х' - 28: 1, # 'ц' - 22: 2, # 'ч' - 25: 2, # 'ш' - 29: 1, # 'щ' - 54: 0, # 'ъ' - 18: 0, # 'ы' - 17: 0, # 'ь' - 30: 0, # 'э' - 27: 0, # 'ю' - 16: 0, # 'я' - }, - 35: { # 'П' - 37: 1, # 'А' - 44: 0, # 'Б' - 33: 0, # 'В' - 46: 0, # 'Г' - 41: 0, # 'Д' - 48: 1, # 'Е' - 56: 0, # 'Ж' - 51: 0, # 'З' - 42: 1, # 'И' - 60: 0, # 'Й' - 36: 0, # 'К' - 49: 1, # 'Л' - 38: 0, # 'М' - 31: 1, # 'Н' - 34: 1, # 'О' - 35: 1, # 'П' - 45: 2, # 'Р' - 32: 1, # 'С' - 40: 1, # 'Т' - 52: 1, # 'У' - 53: 0, # 'Ф' - 55: 0, # 'Х' - 58: 0, # 'Ц' - 50: 0, # 'Ч' - 57: 0, # 'Ш' - 63: 0, # 'Щ' - 62: 1, # 'Ы' - 61: 1, # 'Ь' - 47: 0, # 'Э' - 59: 0, # 'Ю' - 43: 1, # 'Я' - 3: 2, # 'а' - 21: 0, # 'б' - 10: 0, # 'в' - 19: 0, # 'г' - 13: 0, # 'д' - 2: 2, # 'е' - 24: 0, # 'ж' - 20: 0, # 'з' - 4: 2, # 'и' - 23: 0, # 'й' - 11: 0, # 'к' - 8: 2, # 'л' - 12: 0, # 'м' - 5: 1, # 'н' - 1: 3, # 'о' - 15: 0, # 'п' - 9: 3, # 'р' - 7: 1, # 'с' - 6: 1, # 'т' - 14: 2, # 'у' - 39: 1, # 'ф' - 26: 0, # 'х' - 28: 0, # 'ц' - 22: 0, # 'ч' - 25: 1, # 'ш' - 29: 0, # 'щ' - 54: 0, # 'ъ' - 18: 1, # 'ы' - 17: 2, # 'ь' - 30: 1, # 'э' - 27: 0, # 'ю' - 16: 2, # 'я' - }, - 45: { # 'Р' - 37: 2, # 'А' - 44: 1, # 'Б' - 33: 1, # 'В' - 46: 1, # 'Г' - 41: 1, # 'Д' - 48: 2, # 'Е' - 56: 1, # 'Ж' - 51: 0, # 'З' - 42: 2, # 'И' - 60: 0, # 'Й' - 36: 1, # 'К' - 49: 1, # 'Л' - 38: 1, # 'М' - 31: 1, # 'Н' - 34: 2, # 'О' - 35: 0, # 'П' - 45: 1, # 'Р' - 32: 1, # 'С' - 40: 1, # 'Т' - 52: 1, # 'У' - 53: 0, # 'Ф' - 55: 1, # 'Х' - 58: 1, # 'Ц' - 50: 1, # 'Ч' - 57: 1, # 'Ш' - 63: 0, # 'Щ' - 62: 1, # 'Ы' - 61: 1, # 'Ь' - 47: 1, # 'Э' - 59: 1, # 'Ю' - 43: 1, # 'Я' - 3: 3, # 'а' - 21: 0, # 'б' - 10: 1, # 'в' - 19: 0, # 'г' - 13: 0, # 'д' - 2: 2, # 'е' - 24: 1, # 'ж' - 20: 0, # 'з' - 4: 2, # 'и' - 23: 0, # 'й' - 11: 0, # 'к' - 8: 0, # 'л' - 12: 0, # 'м' - 5: 0, # 'н' - 1: 3, # 'о' - 15: 0, # 'п' - 9: 1, # 'р' - 7: 0, # 'с' - 6: 0, # 'т' - 14: 2, # 'у' - 39: 0, # 'ф' - 26: 0, # 'х' - 28: 0, # 'ц' - 22: 0, # 'ч' - 25: 0, # 'ш' - 29: 0, # 'щ' - 54: 0, # 'ъ' - 18: 2, # 'ы' - 17: 0, # 'ь' - 30: 1, # 'э' - 27: 1, # 'ю' - 16: 2, # 'я' - }, - 32: { # 'С' - 37: 1, # 'А' - 44: 1, # 'Б' - 33: 1, # 'В' - 46: 1, # 'Г' - 41: 1, # 'Д' - 48: 1, # 'Е' - 56: 0, # 'Ж' - 51: 0, # 'З' - 42: 1, # 'И' - 60: 0, # 'Й' - 36: 1, # 'К' - 49: 1, # 'Л' - 38: 1, # 'М' - 31: 1, # 'Н' - 34: 1, # 'О' - 35: 1, # 'П' - 45: 1, # 'Р' - 32: 1, # 'С' - 40: 2, # 'Т' - 52: 1, # 'У' - 53: 0, # 'Ф' - 55: 1, # 'Х' - 58: 1, # 'Ц' - 50: 1, # 'Ч' - 57: 1, # 'Ш' - 63: 0, # 'Щ' - 62: 1, # 'Ы' - 61: 1, # 'Ь' - 47: 1, # 'Э' - 59: 1, # 'Ю' - 43: 1, # 'Я' - 3: 2, # 'а' - 21: 1, # 'б' - 10: 2, # 'в' - 19: 1, # 'г' - 13: 2, # 'д' - 2: 3, # 'е' - 24: 1, # 'ж' - 20: 1, # 'з' - 4: 2, # 'и' - 23: 0, # 'й' - 11: 2, # 'к' - 8: 2, # 'л' - 12: 2, # 'м' - 5: 2, # 'н' - 1: 2, # 'о' - 15: 2, # 'п' - 9: 2, # 'р' - 7: 1, # 'с' - 6: 3, # 'т' - 14: 2, # 'у' - 39: 1, # 'ф' - 26: 1, # 'х' - 28: 1, # 'ц' - 22: 1, # 'ч' - 25: 0, # 'ш' - 29: 0, # 'щ' - 54: 1, # 'ъ' - 18: 1, # 'ы' - 17: 1, # 'ь' - 30: 2, # 'э' - 27: 1, # 'ю' - 16: 1, # 'я' - }, - 40: { # 'Т' - 37: 1, # 'А' - 44: 0, # 'Б' - 33: 1, # 'В' - 46: 0, # 'Г' - 41: 0, # 'Д' - 48: 1, # 'Е' - 56: 0, # 'Ж' - 51: 0, # 'З' - 42: 1, # 'И' - 60: 0, # 'Й' - 36: 1, # 'К' - 49: 1, # 'Л' - 38: 1, # 'М' - 31: 1, # 'Н' - 34: 2, # 'О' - 35: 0, # 'П' - 45: 1, # 'Р' - 32: 1, # 'С' - 40: 1, # 'Т' - 52: 1, # 'У' - 53: 0, # 'Ф' - 55: 0, # 'Х' - 58: 0, # 'Ц' - 50: 1, # 'Ч' - 57: 0, # 'Ш' - 63: 0, # 'Щ' - 62: 1, # 'Ы' - 61: 1, # 'Ь' - 47: 1, # 'Э' - 59: 1, # 'Ю' - 43: 1, # 'Я' - 3: 3, # 'а' - 21: 1, # 'б' - 10: 2, # 'в' - 19: 0, # 'г' - 13: 0, # 'д' - 2: 3, # 'е' - 24: 0, # 'ж' - 20: 0, # 'з' - 4: 2, # 'и' - 23: 0, # 'й' - 11: 1, # 'к' - 8: 1, # 'л' - 12: 0, # 'м' - 5: 0, # 'н' - 1: 3, # 'о' - 15: 0, # 'п' - 9: 2, # 'р' - 7: 1, # 'с' - 6: 0, # 'т' - 14: 2, # 'у' - 39: 0, # 'ф' - 26: 0, # 'х' - 28: 0, # 'ц' - 22: 0, # 'ч' - 25: 0, # 'ш' - 29: 1, # 'щ' - 54: 0, # 'ъ' - 18: 3, # 'ы' - 17: 1, # 'ь' - 30: 2, # 'э' - 27: 1, # 'ю' - 16: 1, # 'я' - }, - 52: { # 'У' - 37: 1, # 'А' - 44: 1, # 'Б' - 33: 1, # 'В' - 46: 1, # 'Г' - 41: 1, # 'Д' - 48: 1, # 'Е' - 56: 1, # 'Ж' - 51: 0, # 'З' - 42: 0, # 'И' - 60: 1, # 'Й' - 36: 1, # 'К' - 49: 1, # 'Л' - 38: 1, # 'М' - 31: 1, # 'Н' - 34: 1, # 'О' - 35: 1, # 'П' - 45: 1, # 'Р' - 32: 1, # 'С' - 40: 1, # 'Т' - 52: 0, # 'У' - 53: 0, # 'Ф' - 55: 1, # 'Х' - 58: 0, # 'Ц' - 50: 1, # 'Ч' - 57: 1, # 'Ш' - 63: 1, # 'Щ' - 62: 0, # 'Ы' - 61: 0, # 'Ь' - 47: 0, # 'Э' - 59: 1, # 'Ю' - 43: 0, # 'Я' - 3: 1, # 'а' - 21: 2, # 'б' - 10: 2, # 'в' - 19: 1, # 'г' - 13: 2, # 'д' - 2: 1, # 'е' - 24: 2, # 'ж' - 20: 2, # 'з' - 4: 2, # 'и' - 23: 1, # 'й' - 11: 1, # 'к' - 8: 2, # 'л' - 12: 2, # 'м' - 5: 1, # 'н' - 1: 2, # 'о' - 15: 1, # 'п' - 9: 2, # 'р' - 7: 2, # 'с' - 6: 2, # 'т' - 14: 0, # 'у' - 39: 1, # 'ф' - 26: 1, # 'х' - 28: 1, # 'ц' - 22: 2, # 'ч' - 25: 1, # 'ш' - 29: 1, # 'щ' - 54: 0, # 'ъ' - 18: 0, # 'ы' - 17: 0, # 'ь' - 30: 2, # 'э' - 27: 1, # 'ю' - 16: 0, # 'я' - }, - 53: { # 'Ф' - 37: 1, # 'А' - 44: 1, # 'Б' - 33: 0, # 'В' - 46: 0, # 'Г' - 41: 0, # 'Д' - 48: 1, # 'Е' - 56: 0, # 'Ж' - 51: 0, # 'З' - 42: 1, # 'И' - 60: 0, # 'Й' - 36: 0, # 'К' - 49: 1, # 'Л' - 38: 0, # 'М' - 31: 0, # 'Н' - 34: 1, # 'О' - 35: 0, # 'П' - 45: 1, # 'Р' - 32: 0, # 'С' - 40: 0, # 'Т' - 52: 1, # 'У' - 53: 0, # 'Ф' - 55: 0, # 'Х' - 58: 0, # 'Ц' - 50: 0, # 'Ч' - 57: 0, # 'Ш' - 63: 0, # 'Щ' - 62: 0, # 'Ы' - 61: 0, # 'Ь' - 47: 0, # 'Э' - 59: 0, # 'Ю' - 43: 0, # 'Я' - 3: 2, # 'а' - 21: 0, # 'б' - 10: 0, # 'в' - 19: 0, # 'г' - 13: 0, # 'д' - 2: 2, # 'е' - 24: 0, # 'ж' - 20: 0, # 'з' - 4: 2, # 'и' - 23: 0, # 'й' - 11: 0, # 'к' - 8: 2, # 'л' - 12: 0, # 'м' - 5: 0, # 'н' - 1: 2, # 'о' - 15: 0, # 'п' - 9: 2, # 'р' - 7: 0, # 'с' - 6: 1, # 'т' - 14: 2, # 'у' - 39: 0, # 'ф' - 26: 0, # 'х' - 28: 0, # 'ц' - 22: 0, # 'ч' - 25: 0, # 'ш' - 29: 0, # 'щ' - 54: 0, # 'ъ' - 18: 0, # 'ы' - 17: 1, # 'ь' - 30: 2, # 'э' - 27: 0, # 'ю' - 16: 0, # 'я' - }, - 55: { # 'Х' - 37: 1, # 'А' - 44: 0, # 'Б' - 33: 1, # 'В' - 46: 0, # 'Г' - 41: 0, # 'Д' - 48: 0, # 'Е' - 56: 0, # 'Ж' - 51: 0, # 'З' - 42: 1, # 'И' - 60: 0, # 'Й' - 36: 0, # 'К' - 49: 1, # 'Л' - 38: 1, # 'М' - 31: 1, # 'Н' - 34: 1, # 'О' - 35: 0, # 'П' - 45: 0, # 'Р' - 32: 0, # 'С' - 40: 0, # 'Т' - 52: 0, # 'У' - 53: 0, # 'Ф' - 55: 0, # 'Х' - 58: 0, # 'Ц' - 50: 0, # 'Ч' - 57: 0, # 'Ш' - 63: 0, # 'Щ' - 62: 0, # 'Ы' - 61: 0, # 'Ь' - 47: 0, # 'Э' - 59: 0, # 'Ю' - 43: 0, # 'Я' - 3: 2, # 'а' - 21: 0, # 'б' - 10: 2, # 'в' - 19: 0, # 'г' - 13: 0, # 'д' - 2: 2, # 'е' - 24: 0, # 'ж' - 20: 0, # 'з' - 4: 2, # 'и' - 23: 0, # 'й' - 11: 0, # 'к' - 8: 2, # 'л' - 12: 1, # 'м' - 5: 0, # 'н' - 1: 2, # 'о' - 15: 0, # 'п' - 9: 2, # 'р' - 7: 0, # 'с' - 6: 0, # 'т' - 14: 1, # 'у' - 39: 0, # 'ф' - 26: 0, # 'х' - 28: 0, # 'ц' - 22: 0, # 'ч' - 25: 0, # 'ш' - 29: 0, # 'щ' - 54: 0, # 'ъ' - 18: 0, # 'ы' - 17: 1, # 'ь' - 30: 1, # 'э' - 27: 0, # 'ю' - 16: 0, # 'я' - }, - 58: { # 'Ц' - 37: 1, # 'А' - 44: 0, # 'Б' - 33: 0, # 'В' - 46: 0, # 'Г' - 41: 0, # 'Д' - 48: 1, # 'Е' - 56: 0, # 'Ж' - 51: 0, # 'З' - 42: 1, # 'И' - 60: 0, # 'Й' - 36: 1, # 'К' - 49: 0, # 'Л' - 38: 0, # 'М' - 31: 0, # 'Н' - 34: 1, # 'О' - 35: 0, # 'П' - 45: 0, # 'Р' - 32: 0, # 'С' - 40: 0, # 'Т' - 52: 1, # 'У' - 53: 0, # 'Ф' - 55: 0, # 'Х' - 58: 0, # 'Ц' - 50: 0, # 'Ч' - 57: 0, # 'Ш' - 63: 0, # 'Щ' - 62: 1, # 'Ы' - 61: 0, # 'Ь' - 47: 0, # 'Э' - 59: 0, # 'Ю' - 43: 0, # 'Я' - 3: 1, # 'а' - 21: 0, # 'б' - 10: 1, # 'в' - 19: 0, # 'г' - 13: 0, # 'д' - 2: 2, # 'е' - 24: 0, # 'ж' - 20: 0, # 'з' - 4: 2, # 'и' - 23: 0, # 'й' - 11: 0, # 'к' - 8: 0, # 'л' - 12: 0, # 'м' - 5: 0, # 'н' - 1: 0, # 'о' - 15: 0, # 'п' - 9: 0, # 'р' - 7: 0, # 'с' - 6: 0, # 'т' - 14: 1, # 'у' - 39: 0, # 'ф' - 26: 0, # 'х' - 28: 0, # 'ц' - 22: 0, # 'ч' - 25: 0, # 'ш' - 29: 0, # 'щ' - 54: 0, # 'ъ' - 18: 1, # 'ы' - 17: 0, # 'ь' - 30: 0, # 'э' - 27: 1, # 'ю' - 16: 0, # 'я' - }, - 50: { # 'Ч' - 37: 1, # 'А' - 44: 0, # 'Б' - 33: 0, # 'В' - 46: 0, # 'Г' - 41: 0, # 'Д' - 48: 1, # 'Е' - 56: 0, # 'Ж' - 51: 0, # 'З' - 42: 1, # 'И' - 60: 0, # 'Й' - 36: 1, # 'К' - 49: 0, # 'Л' - 38: 0, # 'М' - 31: 1, # 'Н' - 34: 0, # 'О' - 35: 1, # 'П' - 45: 0, # 'Р' - 32: 0, # 'С' - 40: 1, # 'Т' - 52: 1, # 'У' - 53: 0, # 'Ф' - 55: 0, # 'Х' - 58: 0, # 'Ц' - 50: 0, # 'Ч' - 57: 0, # 'Ш' - 63: 0, # 'Щ' - 62: 0, # 'Ы' - 61: 1, # 'Ь' - 47: 0, # 'Э' - 59: 0, # 'Ю' - 43: 0, # 'Я' - 3: 2, # 'а' - 21: 0, # 'б' - 10: 0, # 'в' - 19: 0, # 'г' - 13: 0, # 'д' - 2: 2, # 'е' - 24: 0, # 'ж' - 20: 0, # 'з' - 4: 2, # 'и' - 23: 0, # 'й' - 11: 0, # 'к' - 8: 1, # 'л' - 12: 0, # 'м' - 5: 0, # 'н' - 1: 1, # 'о' - 15: 0, # 'п' - 9: 1, # 'р' - 7: 0, # 'с' - 6: 3, # 'т' - 14: 2, # 'у' - 39: 0, # 'ф' - 26: 0, # 'х' - 28: 0, # 'ц' - 22: 0, # 'ч' - 25: 0, # 'ш' - 29: 0, # 'щ' - 54: 0, # 'ъ' - 18: 0, # 'ы' - 17: 1, # 'ь' - 30: 0, # 'э' - 27: 0, # 'ю' - 16: 0, # 'я' - }, - 57: { # 'Ш' - 37: 1, # 'А' - 44: 0, # 'Б' - 33: 0, # 'В' - 46: 0, # 'Г' - 41: 0, # 'Д' - 48: 1, # 'Е' - 56: 0, # 'Ж' - 51: 0, # 'З' - 42: 1, # 'И' - 60: 0, # 'Й' - 36: 1, # 'К' - 49: 1, # 'Л' - 38: 0, # 'М' - 31: 1, # 'Н' - 34: 1, # 'О' - 35: 0, # 'П' - 45: 0, # 'Р' - 32: 0, # 'С' - 40: 0, # 'Т' - 52: 1, # 'У' - 53: 0, # 'Ф' - 55: 0, # 'Х' - 58: 0, # 'Ц' - 50: 0, # 'Ч' - 57: 0, # 'Ш' - 63: 0, # 'Щ' - 62: 0, # 'Ы' - 61: 0, # 'Ь' - 47: 0, # 'Э' - 59: 0, # 'Ю' - 43: 0, # 'Я' - 3: 2, # 'а' - 21: 0, # 'б' - 10: 1, # 'в' - 19: 0, # 'г' - 13: 0, # 'д' - 2: 2, # 'е' - 24: 0, # 'ж' - 20: 0, # 'з' - 4: 1, # 'и' - 23: 0, # 'й' - 11: 1, # 'к' - 8: 2, # 'л' - 12: 1, # 'м' - 5: 1, # 'н' - 1: 2, # 'о' - 15: 2, # 'п' - 9: 1, # 'р' - 7: 0, # 'с' - 6: 2, # 'т' - 14: 2, # 'у' - 39: 0, # 'ф' - 26: 1, # 'х' - 28: 0, # 'ц' - 22: 0, # 'ч' - 25: 1, # 'ш' - 29: 0, # 'щ' - 54: 0, # 'ъ' - 18: 0, # 'ы' - 17: 0, # 'ь' - 30: 1, # 'э' - 27: 0, # 'ю' - 16: 0, # 'я' - }, - 63: { # 'Щ' - 37: 1, # 'А' - 44: 0, # 'Б' - 33: 0, # 'В' - 46: 0, # 'Г' - 41: 0, # 'Д' - 48: 1, # 'Е' - 56: 0, # 'Ж' - 51: 0, # 'З' - 42: 1, # 'И' - 60: 0, # 'Й' - 36: 0, # 'К' - 49: 0, # 'Л' - 38: 0, # 'М' - 31: 0, # 'Н' - 34: 0, # 'О' - 35: 0, # 'П' - 45: 0, # 'Р' - 32: 0, # 'С' - 40: 0, # 'Т' - 52: 0, # 'У' - 53: 0, # 'Ф' - 55: 0, # 'Х' - 58: 0, # 'Ц' - 50: 0, # 'Ч' - 57: 0, # 'Ш' - 63: 0, # 'Щ' - 62: 0, # 'Ы' - 61: 1, # 'Ь' - 47: 0, # 'Э' - 59: 0, # 'Ю' - 43: 0, # 'Я' - 3: 1, # 'а' - 21: 0, # 'б' - 10: 0, # 'в' - 19: 0, # 'г' - 13: 0, # 'д' - 2: 1, # 'е' - 24: 0, # 'ж' - 20: 0, # 'з' - 4: 1, # 'и' - 23: 0, # 'й' - 11: 0, # 'к' - 8: 0, # 'л' - 12: 0, # 'м' - 5: 0, # 'н' - 1: 1, # 'о' - 15: 0, # 'п' - 9: 0, # 'р' - 7: 0, # 'с' - 6: 0, # 'т' - 14: 1, # 'у' - 39: 0, # 'ф' - 26: 0, # 'х' - 28: 0, # 'ц' - 22: 0, # 'ч' - 25: 0, # 'ш' - 29: 0, # 'щ' - 54: 0, # 'ъ' - 18: 0, # 'ы' - 17: 0, # 'ь' - 30: 0, # 'э' - 27: 0, # 'ю' - 16: 0, # 'я' - }, - 62: { # 'Ы' - 37: 0, # 'А' - 44: 0, # 'Б' - 33: 1, # 'В' - 46: 1, # 'Г' - 41: 0, # 'Д' - 48: 1, # 'Е' - 56: 0, # 'Ж' - 51: 0, # 'З' - 42: 0, # 'И' - 60: 1, # 'Й' - 36: 1, # 'К' - 49: 1, # 'Л' - 38: 1, # 'М' - 31: 1, # 'Н' - 34: 0, # 'О' - 35: 1, # 'П' - 45: 1, # 'Р' - 32: 1, # 'С' - 40: 1, # 'Т' - 52: 0, # 'У' - 53: 0, # 'Ф' - 55: 1, # 'Х' - 58: 1, # 'Ц' - 50: 0, # 'Ч' - 57: 1, # 'Ш' - 63: 0, # 'Щ' - 62: 0, # 'Ы' - 61: 0, # 'Ь' - 47: 0, # 'Э' - 59: 0, # 'Ю' - 43: 0, # 'Я' - 3: 0, # 'а' - 21: 0, # 'б' - 10: 0, # 'в' - 19: 0, # 'г' - 13: 0, # 'д' - 2: 0, # 'е' - 24: 0, # 'ж' - 20: 0, # 'з' - 4: 0, # 'и' - 23: 0, # 'й' - 11: 0, # 'к' - 8: 0, # 'л' - 12: 0, # 'м' - 5: 0, # 'н' - 1: 0, # 'о' - 15: 0, # 'п' - 9: 0, # 'р' - 7: 0, # 'с' - 6: 0, # 'т' - 14: 0, # 'у' - 39: 0, # 'ф' - 26: 0, # 'х' - 28: 0, # 'ц' - 22: 0, # 'ч' - 25: 0, # 'ш' - 29: 0, # 'щ' - 54: 0, # 'ъ' - 18: 0, # 'ы' - 17: 0, # 'ь' - 30: 0, # 'э' - 27: 0, # 'ю' - 16: 0, # 'я' - }, - 61: { # 'Ь' - 37: 0, # 'А' - 44: 1, # 'Б' - 33: 1, # 'В' - 46: 0, # 'Г' - 41: 1, # 'Д' - 48: 1, # 'Е' - 56: 0, # 'Ж' - 51: 0, # 'З' - 42: 1, # 'И' - 60: 0, # 'Й' - 36: 1, # 'К' - 49: 0, # 'Л' - 38: 1, # 'М' - 31: 1, # 'Н' - 34: 1, # 'О' - 35: 0, # 'П' - 45: 0, # 'Р' - 32: 1, # 'С' - 40: 0, # 'Т' - 52: 0, # 'У' - 53: 1, # 'Ф' - 55: 0, # 'Х' - 58: 0, # 'Ц' - 50: 0, # 'Ч' - 57: 1, # 'Ш' - 63: 0, # 'Щ' - 62: 0, # 'Ы' - 61: 0, # 'Ь' - 47: 0, # 'Э' - 59: 1, # 'Ю' - 43: 1, # 'Я' - 3: 0, # 'а' - 21: 0, # 'б' - 10: 0, # 'в' - 19: 0, # 'г' - 13: 0, # 'д' - 2: 0, # 'е' - 24: 0, # 'ж' - 20: 0, # 'з' - 4: 0, # 'и' - 23: 0, # 'й' - 11: 0, # 'к' - 8: 0, # 'л' - 12: 0, # 'м' - 5: 0, # 'н' - 1: 0, # 'о' - 15: 0, # 'п' - 9: 0, # 'р' - 7: 0, # 'с' - 6: 0, # 'т' - 14: 0, # 'у' - 39: 0, # 'ф' - 26: 0, # 'х' - 28: 0, # 'ц' - 22: 0, # 'ч' - 25: 0, # 'ш' - 29: 0, # 'щ' - 54: 0, # 'ъ' - 18: 0, # 'ы' - 17: 0, # 'ь' - 30: 0, # 'э' - 27: 0, # 'ю' - 16: 0, # 'я' - }, - 47: { # 'Э' - 37: 0, # 'А' - 44: 0, # 'Б' - 33: 1, # 'В' - 46: 0, # 'Г' - 41: 1, # 'Д' - 48: 0, # 'Е' - 56: 0, # 'Ж' - 51: 0, # 'З' - 42: 0, # 'И' - 60: 1, # 'Й' - 36: 1, # 'К' - 49: 1, # 'Л' - 38: 1, # 'М' - 31: 1, # 'Н' - 34: 0, # 'О' - 35: 1, # 'П' - 45: 1, # 'Р' - 32: 1, # 'С' - 40: 1, # 'Т' - 52: 0, # 'У' - 53: 0, # 'Ф' - 55: 0, # 'Х' - 58: 0, # 'Ц' - 50: 0, # 'Ч' - 57: 0, # 'Ш' - 63: 0, # 'Щ' - 62: 0, # 'Ы' - 61: 0, # 'Ь' - 47: 0, # 'Э' - 59: 0, # 'Ю' - 43: 0, # 'Я' - 3: 1, # 'а' - 21: 1, # 'б' - 10: 2, # 'в' - 19: 1, # 'г' - 13: 2, # 'д' - 2: 0, # 'е' - 24: 1, # 'ж' - 20: 0, # 'з' - 4: 0, # 'и' - 23: 2, # 'й' - 11: 2, # 'к' - 8: 2, # 'л' - 12: 2, # 'м' - 5: 2, # 'н' - 1: 0, # 'о' - 15: 1, # 'п' - 9: 2, # 'р' - 7: 1, # 'с' - 6: 3, # 'т' - 14: 1, # 'у' - 39: 1, # 'ф' - 26: 1, # 'х' - 28: 0, # 'ц' - 22: 0, # 'ч' - 25: 1, # 'ш' - 29: 0, # 'щ' - 54: 0, # 'ъ' - 18: 0, # 'ы' - 17: 0, # 'ь' - 30: 0, # 'э' - 27: 0, # 'ю' - 16: 0, # 'я' - }, - 59: { # 'Ю' - 37: 1, # 'А' - 44: 1, # 'Б' - 33: 0, # 'В' - 46: 0, # 'Г' - 41: 1, # 'Д' - 48: 0, # 'Е' - 56: 0, # 'Ж' - 51: 0, # 'З' - 42: 0, # 'И' - 60: 0, # 'Й' - 36: 0, # 'К' - 49: 0, # 'Л' - 38: 0, # 'М' - 31: 1, # 'Н' - 34: 0, # 'О' - 35: 0, # 'П' - 45: 1, # 'Р' - 32: 0, # 'С' - 40: 1, # 'Т' - 52: 0, # 'У' - 53: 0, # 'Ф' - 55: 0, # 'Х' - 58: 0, # 'Ц' - 50: 1, # 'Ч' - 57: 0, # 'Ш' - 63: 1, # 'Щ' - 62: 0, # 'Ы' - 61: 0, # 'Ь' - 47: 0, # 'Э' - 59: 0, # 'Ю' - 43: 0, # 'Я' - 3: 0, # 'а' - 21: 1, # 'б' - 10: 0, # 'в' - 19: 1, # 'г' - 13: 1, # 'д' - 2: 0, # 'е' - 24: 1, # 'ж' - 20: 0, # 'з' - 4: 0, # 'и' - 23: 0, # 'й' - 11: 1, # 'к' - 8: 2, # 'л' - 12: 1, # 'м' - 5: 2, # 'н' - 1: 0, # 'о' - 15: 1, # 'п' - 9: 1, # 'р' - 7: 1, # 'с' - 6: 0, # 'т' - 14: 0, # 'у' - 39: 0, # 'ф' - 26: 1, # 'х' - 28: 0, # 'ц' - 22: 0, # 'ч' - 25: 0, # 'ш' - 29: 0, # 'щ' - 54: 0, # 'ъ' - 18: 0, # 'ы' - 17: 0, # 'ь' - 30: 0, # 'э' - 27: 0, # 'ю' - 16: 0, # 'я' - }, - 43: { # 'Я' - 37: 0, # 'А' - 44: 0, # 'Б' - 33: 1, # 'В' - 46: 1, # 'Г' - 41: 0, # 'Д' - 48: 1, # 'Е' - 56: 0, # 'Ж' - 51: 0, # 'З' - 42: 1, # 'И' - 60: 0, # 'Й' - 36: 1, # 'К' - 49: 0, # 'Л' - 38: 0, # 'М' - 31: 1, # 'Н' - 34: 0, # 'О' - 35: 0, # 'П' - 45: 0, # 'Р' - 32: 1, # 'С' - 40: 1, # 'Т' - 52: 0, # 'У' - 53: 0, # 'Ф' - 55: 1, # 'Х' - 58: 0, # 'Ц' - 50: 1, # 'Ч' - 57: 0, # 'Ш' - 63: 1, # 'Щ' - 62: 0, # 'Ы' - 61: 0, # 'Ь' - 47: 0, # 'Э' - 59: 1, # 'Ю' - 43: 1, # 'Я' - 3: 0, # 'а' - 21: 1, # 'б' - 10: 1, # 'в' - 19: 1, # 'г' - 13: 1, # 'д' - 2: 0, # 'е' - 24: 0, # 'ж' - 20: 1, # 'з' - 4: 0, # 'и' - 23: 1, # 'й' - 11: 1, # 'к' - 8: 1, # 'л' - 12: 1, # 'м' - 5: 2, # 'н' - 1: 0, # 'о' - 15: 1, # 'п' - 9: 1, # 'р' - 7: 1, # 'с' - 6: 0, # 'т' - 14: 0, # 'у' - 39: 0, # 'ф' - 26: 1, # 'х' - 28: 0, # 'ц' - 22: 0, # 'ч' - 25: 1, # 'ш' - 29: 1, # 'щ' - 54: 0, # 'ъ' - 18: 0, # 'ы' - 17: 0, # 'ь' - 30: 0, # 'э' - 27: 0, # 'ю' - 16: 0, # 'я' - }, - 3: { # 'а' - 37: 0, # 'А' - 44: 0, # 'Б' - 33: 0, # 'В' - 46: 0, # 'Г' - 41: 0, # 'Д' - 48: 0, # 'Е' - 56: 0, # 'Ж' - 51: 0, # 'З' - 42: 1, # 'И' - 60: 0, # 'Й' - 36: 0, # 'К' - 49: 0, # 'Л' - 38: 0, # 'М' - 31: 1, # 'Н' - 34: 0, # 'О' - 35: 0, # 'П' - 45: 0, # 'Р' - 32: 0, # 'С' - 40: 0, # 'Т' - 52: 0, # 'У' - 53: 0, # 'Ф' - 55: 0, # 'Х' - 58: 0, # 'Ц' - 50: 0, # 'Ч' - 57: 0, # 'Ш' - 63: 0, # 'Щ' - 62: 0, # 'Ы' - 61: 0, # 'Ь' - 47: 0, # 'Э' - 59: 0, # 'Ю' - 43: 0, # 'Я' - 3: 2, # 'а' - 21: 3, # 'б' - 10: 3, # 'в' - 19: 3, # 'г' - 13: 3, # 'д' - 2: 3, # 'е' - 24: 3, # 'ж' - 20: 3, # 'з' - 4: 3, # 'и' - 23: 3, # 'й' - 11: 3, # 'к' - 8: 3, # 'л' - 12: 3, # 'м' - 5: 3, # 'н' - 1: 2, # 'о' - 15: 3, # 'п' - 9: 3, # 'р' - 7: 3, # 'с' - 6: 3, # 'т' - 14: 3, # 'у' - 39: 2, # 'ф' - 26: 3, # 'х' - 28: 3, # 'ц' - 22: 3, # 'ч' - 25: 3, # 'ш' - 29: 3, # 'щ' - 54: 0, # 'ъ' - 18: 0, # 'ы' - 17: 0, # 'ь' - 30: 2, # 'э' - 27: 3, # 'ю' - 16: 3, # 'я' - }, - 21: { # 'б' - 37: 0, # 'А' - 44: 0, # 'Б' - 33: 0, # 'В' - 46: 0, # 'Г' - 41: 0, # 'Д' - 48: 0, # 'Е' - 56: 0, # 'Ж' - 51: 0, # 'З' - 42: 0, # 'И' - 60: 0, # 'Й' - 36: 1, # 'К' - 49: 0, # 'Л' - 38: 0, # 'М' - 31: 0, # 'Н' - 34: 0, # 'О' - 35: 0, # 'П' - 45: 0, # 'Р' - 32: 0, # 'С' - 40: 0, # 'Т' - 52: 0, # 'У' - 53: 0, # 'Ф' - 55: 0, # 'Х' - 58: 0, # 'Ц' - 50: 0, # 'Ч' - 57: 0, # 'Ш' - 63: 0, # 'Щ' - 62: 0, # 'Ы' - 61: 0, # 'Ь' - 47: 0, # 'Э' - 59: 0, # 'Ю' - 43: 0, # 'Я' - 3: 3, # 'а' - 21: 2, # 'б' - 10: 2, # 'в' - 19: 1, # 'г' - 13: 2, # 'д' - 2: 3, # 'е' - 24: 2, # 'ж' - 20: 1, # 'з' - 4: 3, # 'и' - 23: 0, # 'й' - 11: 2, # 'к' - 8: 3, # 'л' - 12: 2, # 'м' - 5: 3, # 'н' - 1: 3, # 'о' - 15: 1, # 'п' - 9: 3, # 'р' - 7: 3, # 'с' - 6: 2, # 'т' - 14: 3, # 'у' - 39: 0, # 'ф' - 26: 2, # 'х' - 28: 1, # 'ц' - 22: 1, # 'ч' - 25: 2, # 'ш' - 29: 3, # 'щ' - 54: 2, # 'ъ' - 18: 3, # 'ы' - 17: 2, # 'ь' - 30: 1, # 'э' - 27: 2, # 'ю' - 16: 3, # 'я' - }, - 10: { # 'в' - 37: 0, # 'А' - 44: 0, # 'Б' - 33: 0, # 'В' - 46: 0, # 'Г' - 41: 0, # 'Д' - 48: 0, # 'Е' - 56: 0, # 'Ж' - 51: 0, # 'З' - 42: 0, # 'И' - 60: 0, # 'Й' - 36: 0, # 'К' - 49: 0, # 'Л' - 38: 0, # 'М' - 31: 0, # 'Н' - 34: 0, # 'О' - 35: 0, # 'П' - 45: 0, # 'Р' - 32: 0, # 'С' - 40: 0, # 'Т' - 52: 0, # 'У' - 53: 0, # 'Ф' - 55: 0, # 'Х' - 58: 0, # 'Ц' - 50: 0, # 'Ч' - 57: 0, # 'Ш' - 63: 0, # 'Щ' - 62: 0, # 'Ы' - 61: 0, # 'Ь' - 47: 0, # 'Э' - 59: 0, # 'Ю' - 43: 0, # 'Я' - 3: 3, # 'а' - 21: 2, # 'б' - 10: 2, # 'в' - 19: 2, # 'г' - 13: 3, # 'д' - 2: 3, # 'е' - 24: 1, # 'ж' - 20: 3, # 'з' - 4: 3, # 'и' - 23: 0, # 'й' - 11: 3, # 'к' - 8: 3, # 'л' - 12: 2, # 'м' - 5: 3, # 'н' - 1: 3, # 'о' - 15: 3, # 'п' - 9: 3, # 'р' - 7: 3, # 'с' - 6: 3, # 'т' - 14: 3, # 'у' - 39: 1, # 'ф' - 26: 2, # 'х' - 28: 2, # 'ц' - 22: 2, # 'ч' - 25: 3, # 'ш' - 29: 2, # 'щ' - 54: 2, # 'ъ' - 18: 3, # 'ы' - 17: 3, # 'ь' - 30: 1, # 'э' - 27: 1, # 'ю' - 16: 3, # 'я' - }, - 19: { # 'г' - 37: 0, # 'А' - 44: 0, # 'Б' - 33: 0, # 'В' - 46: 0, # 'Г' - 41: 0, # 'Д' - 48: 0, # 'Е' - 56: 0, # 'Ж' - 51: 0, # 'З' - 42: 0, # 'И' - 60: 0, # 'Й' - 36: 0, # 'К' - 49: 0, # 'Л' - 38: 0, # 'М' - 31: 0, # 'Н' - 34: 0, # 'О' - 35: 0, # 'П' - 45: 0, # 'Р' - 32: 0, # 'С' - 40: 0, # 'Т' - 52: 0, # 'У' - 53: 0, # 'Ф' - 55: 0, # 'Х' - 58: 0, # 'Ц' - 50: 0, # 'Ч' - 57: 0, # 'Ш' - 63: 0, # 'Щ' - 62: 0, # 'Ы' - 61: 0, # 'Ь' - 47: 0, # 'Э' - 59: 0, # 'Ю' - 43: 0, # 'Я' - 3: 3, # 'а' - 21: 1, # 'б' - 10: 2, # 'в' - 19: 1, # 'г' - 13: 3, # 'д' - 2: 3, # 'е' - 24: 0, # 'ж' - 20: 1, # 'з' - 4: 3, # 'и' - 23: 0, # 'й' - 11: 2, # 'к' - 8: 3, # 'л' - 12: 2, # 'м' - 5: 3, # 'н' - 1: 3, # 'о' - 15: 0, # 'п' - 9: 3, # 'р' - 7: 2, # 'с' - 6: 2, # 'т' - 14: 3, # 'у' - 39: 1, # 'ф' - 26: 1, # 'х' - 28: 1, # 'ц' - 22: 2, # 'ч' - 25: 1, # 'ш' - 29: 0, # 'щ' - 54: 0, # 'ъ' - 18: 1, # 'ы' - 17: 1, # 'ь' - 30: 1, # 'э' - 27: 1, # 'ю' - 16: 0, # 'я' - }, - 13: { # 'д' - 37: 0, # 'А' - 44: 0, # 'Б' - 33: 0, # 'В' - 46: 0, # 'Г' - 41: 0, # 'Д' - 48: 0, # 'Е' - 56: 0, # 'Ж' - 51: 0, # 'З' - 42: 0, # 'И' - 60: 0, # 'Й' - 36: 0, # 'К' - 49: 0, # 'Л' - 38: 0, # 'М' - 31: 0, # 'Н' - 34: 0, # 'О' - 35: 0, # 'П' - 45: 0, # 'Р' - 32: 0, # 'С' - 40: 0, # 'Т' - 52: 0, # 'У' - 53: 0, # 'Ф' - 55: 0, # 'Х' - 58: 0, # 'Ц' - 50: 0, # 'Ч' - 57: 0, # 'Ш' - 63: 0, # 'Щ' - 62: 0, # 'Ы' - 61: 0, # 'Ь' - 47: 0, # 'Э' - 59: 0, # 'Ю' - 43: 0, # 'Я' - 3: 3, # 'а' - 21: 2, # 'б' - 10: 3, # 'в' - 19: 2, # 'г' - 13: 2, # 'д' - 2: 3, # 'е' - 24: 2, # 'ж' - 20: 2, # 'з' - 4: 3, # 'и' - 23: 0, # 'й' - 11: 3, # 'к' - 8: 3, # 'л' - 12: 2, # 'м' - 5: 3, # 'н' - 1: 3, # 'о' - 15: 2, # 'п' - 9: 3, # 'р' - 7: 3, # 'с' - 6: 3, # 'т' - 14: 3, # 'у' - 39: 1, # 'ф' - 26: 2, # 'х' - 28: 3, # 'ц' - 22: 2, # 'ч' - 25: 2, # 'ш' - 29: 1, # 'щ' - 54: 2, # 'ъ' - 18: 3, # 'ы' - 17: 3, # 'ь' - 30: 1, # 'э' - 27: 2, # 'ю' - 16: 3, # 'я' - }, - 2: { # 'е' - 37: 0, # 'А' - 44: 0, # 'Б' - 33: 0, # 'В' - 46: 0, # 'Г' - 41: 0, # 'Д' - 48: 0, # 'Е' - 56: 0, # 'Ж' - 51: 0, # 'З' - 42: 0, # 'И' - 60: 0, # 'Й' - 36: 0, # 'К' - 49: 0, # 'Л' - 38: 0, # 'М' - 31: 0, # 'Н' - 34: 0, # 'О' - 35: 0, # 'П' - 45: 0, # 'Р' - 32: 0, # 'С' - 40: 0, # 'Т' - 52: 0, # 'У' - 53: 0, # 'Ф' - 55: 0, # 'Х' - 58: 0, # 'Ц' - 50: 0, # 'Ч' - 57: 0, # 'Ш' - 63: 0, # 'Щ' - 62: 0, # 'Ы' - 61: 0, # 'Ь' - 47: 0, # 'Э' - 59: 0, # 'Ю' - 43: 0, # 'Я' - 3: 2, # 'а' - 21: 3, # 'б' - 10: 3, # 'в' - 19: 3, # 'г' - 13: 3, # 'д' - 2: 3, # 'е' - 24: 3, # 'ж' - 20: 3, # 'з' - 4: 2, # 'и' - 23: 3, # 'й' - 11: 3, # 'к' - 8: 3, # 'л' - 12: 3, # 'м' - 5: 3, # 'н' - 1: 3, # 'о' - 15: 3, # 'п' - 9: 3, # 'р' - 7: 3, # 'с' - 6: 3, # 'т' - 14: 2, # 'у' - 39: 2, # 'ф' - 26: 3, # 'х' - 28: 3, # 'ц' - 22: 3, # 'ч' - 25: 3, # 'ш' - 29: 3, # 'щ' - 54: 0, # 'ъ' - 18: 0, # 'ы' - 17: 0, # 'ь' - 30: 1, # 'э' - 27: 2, # 'ю' - 16: 3, # 'я' - }, - 24: { # 'ж' - 37: 0, # 'А' - 44: 0, # 'Б' - 33: 0, # 'В' - 46: 0, # 'Г' - 41: 0, # 'Д' - 48: 0, # 'Е' - 56: 0, # 'Ж' - 51: 0, # 'З' - 42: 0, # 'И' - 60: 0, # 'Й' - 36: 0, # 'К' - 49: 0, # 'Л' - 38: 0, # 'М' - 31: 0, # 'Н' - 34: 0, # 'О' - 35: 0, # 'П' - 45: 0, # 'Р' - 32: 0, # 'С' - 40: 0, # 'Т' - 52: 0, # 'У' - 53: 0, # 'Ф' - 55: 0, # 'Х' - 58: 0, # 'Ц' - 50: 0, # 'Ч' - 57: 0, # 'Ш' - 63: 0, # 'Щ' - 62: 0, # 'Ы' - 61: 0, # 'Ь' - 47: 0, # 'Э' - 59: 0, # 'Ю' - 43: 0, # 'Я' - 3: 3, # 'а' - 21: 2, # 'б' - 10: 1, # 'в' - 19: 2, # 'г' - 13: 3, # 'д' - 2: 3, # 'е' - 24: 2, # 'ж' - 20: 1, # 'з' - 4: 3, # 'и' - 23: 0, # 'й' - 11: 2, # 'к' - 8: 2, # 'л' - 12: 1, # 'м' - 5: 3, # 'н' - 1: 2, # 'о' - 15: 1, # 'п' - 9: 2, # 'р' - 7: 2, # 'с' - 6: 1, # 'т' - 14: 3, # 'у' - 39: 1, # 'ф' - 26: 0, # 'х' - 28: 1, # 'ц' - 22: 2, # 'ч' - 25: 0, # 'ш' - 29: 0, # 'щ' - 54: 0, # 'ъ' - 18: 1, # 'ы' - 17: 2, # 'ь' - 30: 1, # 'э' - 27: 1, # 'ю' - 16: 1, # 'я' - }, - 20: { # 'з' - 37: 0, # 'А' - 44: 0, # 'Б' - 33: 0, # 'В' - 46: 0, # 'Г' - 41: 0, # 'Д' - 48: 0, # 'Е' - 56: 0, # 'Ж' - 51: 0, # 'З' - 42: 0, # 'И' - 60: 0, # 'Й' - 36: 0, # 'К' - 49: 0, # 'Л' - 38: 0, # 'М' - 31: 0, # 'Н' - 34: 0, # 'О' - 35: 0, # 'П' - 45: 0, # 'Р' - 32: 0, # 'С' - 40: 0, # 'Т' - 52: 0, # 'У' - 53: 0, # 'Ф' - 55: 0, # 'Х' - 58: 0, # 'Ц' - 50: 0, # 'Ч' - 57: 0, # 'Ш' - 63: 0, # 'Щ' - 62: 0, # 'Ы' - 61: 0, # 'Ь' - 47: 0, # 'Э' - 59: 0, # 'Ю' - 43: 0, # 'Я' - 3: 3, # 'а' - 21: 3, # 'б' - 10: 3, # 'в' - 19: 3, # 'г' - 13: 3, # 'д' - 2: 3, # 'е' - 24: 2, # 'ж' - 20: 2, # 'з' - 4: 3, # 'и' - 23: 0, # 'й' - 11: 3, # 'к' - 8: 3, # 'л' - 12: 3, # 'м' - 5: 3, # 'н' - 1: 3, # 'о' - 15: 0, # 'п' - 9: 3, # 'р' - 7: 2, # 'с' - 6: 2, # 'т' - 14: 3, # 'у' - 39: 0, # 'ф' - 26: 0, # 'х' - 28: 1, # 'ц' - 22: 2, # 'ч' - 25: 1, # 'ш' - 29: 0, # 'щ' - 54: 2, # 'ъ' - 18: 3, # 'ы' - 17: 2, # 'ь' - 30: 1, # 'э' - 27: 1, # 'ю' - 16: 3, # 'я' - }, - 4: { # 'и' - 37: 1, # 'А' - 44: 0, # 'Б' - 33: 0, # 'В' - 46: 0, # 'Г' - 41: 0, # 'Д' - 48: 0, # 'Е' - 56: 0, # 'Ж' - 51: 0, # 'З' - 42: 0, # 'И' - 60: 0, # 'Й' - 36: 0, # 'К' - 49: 0, # 'Л' - 38: 0, # 'М' - 31: 1, # 'Н' - 34: 0, # 'О' - 35: 0, # 'П' - 45: 0, # 'Р' - 32: 0, # 'С' - 40: 0, # 'Т' - 52: 0, # 'У' - 53: 0, # 'Ф' - 55: 0, # 'Х' - 58: 0, # 'Ц' - 50: 0, # 'Ч' - 57: 0, # 'Ш' - 63: 0, # 'Щ' - 62: 0, # 'Ы' - 61: 0, # 'Ь' - 47: 0, # 'Э' - 59: 0, # 'Ю' - 43: 0, # 'Я' - 3: 3, # 'а' - 21: 3, # 'б' - 10: 3, # 'в' - 19: 3, # 'г' - 13: 3, # 'д' - 2: 3, # 'е' - 24: 3, # 'ж' - 20: 3, # 'з' - 4: 3, # 'и' - 23: 3, # 'й' - 11: 3, # 'к' - 8: 3, # 'л' - 12: 3, # 'м' - 5: 3, # 'н' - 1: 3, # 'о' - 15: 3, # 'п' - 9: 3, # 'р' - 7: 3, # 'с' - 6: 3, # 'т' - 14: 2, # 'у' - 39: 2, # 'ф' - 26: 3, # 'х' - 28: 3, # 'ц' - 22: 3, # 'ч' - 25: 3, # 'ш' - 29: 3, # 'щ' - 54: 0, # 'ъ' - 18: 0, # 'ы' - 17: 0, # 'ь' - 30: 2, # 'э' - 27: 3, # 'ю' - 16: 3, # 'я' - }, - 23: { # 'й' - 37: 0, # 'А' - 44: 0, # 'Б' - 33: 0, # 'В' - 46: 0, # 'Г' - 41: 0, # 'Д' - 48: 0, # 'Е' - 56: 0, # 'Ж' - 51: 0, # 'З' - 42: 0, # 'И' - 60: 0, # 'Й' - 36: 0, # 'К' - 49: 0, # 'Л' - 38: 0, # 'М' - 31: 0, # 'Н' - 34: 0, # 'О' - 35: 0, # 'П' - 45: 0, # 'Р' - 32: 0, # 'С' - 40: 0, # 'Т' - 52: 0, # 'У' - 53: 0, # 'Ф' - 55: 0, # 'Х' - 58: 0, # 'Ц' - 50: 0, # 'Ч' - 57: 0, # 'Ш' - 63: 0, # 'Щ' - 62: 0, # 'Ы' - 61: 0, # 'Ь' - 47: 0, # 'Э' - 59: 0, # 'Ю' - 43: 0, # 'Я' - 3: 1, # 'а' - 21: 1, # 'б' - 10: 1, # 'в' - 19: 2, # 'г' - 13: 3, # 'д' - 2: 2, # 'е' - 24: 0, # 'ж' - 20: 2, # 'з' - 4: 1, # 'и' - 23: 0, # 'й' - 11: 2, # 'к' - 8: 2, # 'л' - 12: 2, # 'м' - 5: 3, # 'н' - 1: 2, # 'о' - 15: 1, # 'п' - 9: 2, # 'р' - 7: 3, # 'с' - 6: 3, # 'т' - 14: 1, # 'у' - 39: 2, # 'ф' - 26: 1, # 'х' - 28: 2, # 'ц' - 22: 3, # 'ч' - 25: 2, # 'ш' - 29: 1, # 'щ' - 54: 0, # 'ъ' - 18: 0, # 'ы' - 17: 0, # 'ь' - 30: 1, # 'э' - 27: 1, # 'ю' - 16: 2, # 'я' - }, - 11: { # 'к' - 37: 0, # 'А' - 44: 0, # 'Б' - 33: 0, # 'В' - 46: 0, # 'Г' - 41: 0, # 'Д' - 48: 0, # 'Е' - 56: 0, # 'Ж' - 51: 0, # 'З' - 42: 0, # 'И' - 60: 0, # 'Й' - 36: 0, # 'К' - 49: 0, # 'Л' - 38: 0, # 'М' - 31: 0, # 'Н' - 34: 0, # 'О' - 35: 0, # 'П' - 45: 0, # 'Р' - 32: 0, # 'С' - 40: 0, # 'Т' - 52: 0, # 'У' - 53: 0, # 'Ф' - 55: 0, # 'Х' - 58: 0, # 'Ц' - 50: 0, # 'Ч' - 57: 0, # 'Ш' - 63: 0, # 'Щ' - 62: 0, # 'Ы' - 61: 0, # 'Ь' - 47: 0, # 'Э' - 59: 0, # 'Ю' - 43: 0, # 'Я' - 3: 3, # 'а' - 21: 1, # 'б' - 10: 3, # 'в' - 19: 1, # 'г' - 13: 1, # 'д' - 2: 3, # 'е' - 24: 2, # 'ж' - 20: 2, # 'з' - 4: 3, # 'и' - 23: 0, # 'й' - 11: 2, # 'к' - 8: 3, # 'л' - 12: 1, # 'м' - 5: 3, # 'н' - 1: 3, # 'о' - 15: 0, # 'п' - 9: 3, # 'р' - 7: 3, # 'с' - 6: 3, # 'т' - 14: 3, # 'у' - 39: 1, # 'ф' - 26: 2, # 'х' - 28: 2, # 'ц' - 22: 1, # 'ч' - 25: 2, # 'ш' - 29: 0, # 'щ' - 54: 0, # 'ъ' - 18: 1, # 'ы' - 17: 1, # 'ь' - 30: 1, # 'э' - 27: 1, # 'ю' - 16: 1, # 'я' - }, - 8: { # 'л' - 37: 0, # 'А' - 44: 0, # 'Б' - 33: 0, # 'В' - 46: 0, # 'Г' - 41: 0, # 'Д' - 48: 0, # 'Е' - 56: 0, # 'Ж' - 51: 0, # 'З' - 42: 0, # 'И' - 60: 0, # 'Й' - 36: 0, # 'К' - 49: 0, # 'Л' - 38: 0, # 'М' - 31: 0, # 'Н' - 34: 0, # 'О' - 35: 0, # 'П' - 45: 0, # 'Р' - 32: 0, # 'С' - 40: 0, # 'Т' - 52: 0, # 'У' - 53: 0, # 'Ф' - 55: 0, # 'Х' - 58: 0, # 'Ц' - 50: 0, # 'Ч' - 57: 0, # 'Ш' - 63: 0, # 'Щ' - 62: 0, # 'Ы' - 61: 0, # 'Ь' - 47: 0, # 'Э' - 59: 0, # 'Ю' - 43: 0, # 'Я' - 3: 3, # 'а' - 21: 2, # 'б' - 10: 2, # 'в' - 19: 3, # 'г' - 13: 2, # 'д' - 2: 3, # 'е' - 24: 3, # 'ж' - 20: 2, # 'з' - 4: 3, # 'и' - 23: 0, # 'й' - 11: 3, # 'к' - 8: 3, # 'л' - 12: 2, # 'м' - 5: 3, # 'н' - 1: 3, # 'о' - 15: 2, # 'п' - 9: 1, # 'р' - 7: 3, # 'с' - 6: 2, # 'т' - 14: 3, # 'у' - 39: 2, # 'ф' - 26: 2, # 'х' - 28: 1, # 'ц' - 22: 3, # 'ч' - 25: 2, # 'ш' - 29: 1, # 'щ' - 54: 0, # 'ъ' - 18: 3, # 'ы' - 17: 3, # 'ь' - 30: 1, # 'э' - 27: 3, # 'ю' - 16: 3, # 'я' - }, - 12: { # 'м' - 37: 0, # 'А' - 44: 0, # 'Б' - 33: 0, # 'В' - 46: 0, # 'Г' - 41: 0, # 'Д' - 48: 0, # 'Е' - 56: 0, # 'Ж' - 51: 0, # 'З' - 42: 0, # 'И' - 60: 0, # 'Й' - 36: 0, # 'К' - 49: 0, # 'Л' - 38: 0, # 'М' - 31: 0, # 'Н' - 34: 0, # 'О' - 35: 0, # 'П' - 45: 0, # 'Р' - 32: 0, # 'С' - 40: 0, # 'Т' - 52: 0, # 'У' - 53: 0, # 'Ф' - 55: 0, # 'Х' - 58: 0, # 'Ц' - 50: 0, # 'Ч' - 57: 0, # 'Ш' - 63: 0, # 'Щ' - 62: 0, # 'Ы' - 61: 0, # 'Ь' - 47: 0, # 'Э' - 59: 0, # 'Ю' - 43: 0, # 'Я' - 3: 3, # 'а' - 21: 2, # 'б' - 10: 2, # 'в' - 19: 2, # 'г' - 13: 1, # 'д' - 2: 3, # 'е' - 24: 1, # 'ж' - 20: 1, # 'з' - 4: 3, # 'и' - 23: 0, # 'й' - 11: 2, # 'к' - 8: 3, # 'л' - 12: 2, # 'м' - 5: 3, # 'н' - 1: 3, # 'о' - 15: 2, # 'п' - 9: 2, # 'р' - 7: 3, # 'с' - 6: 2, # 'т' - 14: 3, # 'у' - 39: 2, # 'ф' - 26: 2, # 'х' - 28: 2, # 'ц' - 22: 2, # 'ч' - 25: 1, # 'ш' - 29: 1, # 'щ' - 54: 0, # 'ъ' - 18: 3, # 'ы' - 17: 2, # 'ь' - 30: 2, # 'э' - 27: 1, # 'ю' - 16: 3, # 'я' - }, - 5: { # 'н' - 37: 0, # 'А' - 44: 0, # 'Б' - 33: 0, # 'В' - 46: 0, # 'Г' - 41: 0, # 'Д' - 48: 0, # 'Е' - 56: 0, # 'Ж' - 51: 0, # 'З' - 42: 0, # 'И' - 60: 0, # 'Й' - 36: 0, # 'К' - 49: 0, # 'Л' - 38: 0, # 'М' - 31: 0, # 'Н' - 34: 0, # 'О' - 35: 0, # 'П' - 45: 0, # 'Р' - 32: 0, # 'С' - 40: 0, # 'Т' - 52: 0, # 'У' - 53: 0, # 'Ф' - 55: 0, # 'Х' - 58: 0, # 'Ц' - 50: 0, # 'Ч' - 57: 0, # 'Ш' - 63: 0, # 'Щ' - 62: 0, # 'Ы' - 61: 0, # 'Ь' - 47: 0, # 'Э' - 59: 0, # 'Ю' - 43: 0, # 'Я' - 3: 3, # 'а' - 21: 2, # 'б' - 10: 2, # 'в' - 19: 3, # 'г' - 13: 3, # 'д' - 2: 3, # 'е' - 24: 2, # 'ж' - 20: 2, # 'з' - 4: 3, # 'и' - 23: 0, # 'й' - 11: 3, # 'к' - 8: 2, # 'л' - 12: 1, # 'м' - 5: 3, # 'н' - 1: 3, # 'о' - 15: 1, # 'п' - 9: 2, # 'р' - 7: 3, # 'с' - 6: 3, # 'т' - 14: 3, # 'у' - 39: 2, # 'ф' - 26: 2, # 'х' - 28: 3, # 'ц' - 22: 3, # 'ч' - 25: 2, # 'ш' - 29: 2, # 'щ' - 54: 1, # 'ъ' - 18: 3, # 'ы' - 17: 3, # 'ь' - 30: 1, # 'э' - 27: 3, # 'ю' - 16: 3, # 'я' - }, - 1: { # 'о' - 37: 0, # 'А' - 44: 0, # 'Б' - 33: 0, # 'В' - 46: 0, # 'Г' - 41: 0, # 'Д' - 48: 0, # 'Е' - 56: 0, # 'Ж' - 51: 0, # 'З' - 42: 0, # 'И' - 60: 0, # 'Й' - 36: 0, # 'К' - 49: 0, # 'Л' - 38: 0, # 'М' - 31: 0, # 'Н' - 34: 0, # 'О' - 35: 0, # 'П' - 45: 0, # 'Р' - 32: 0, # 'С' - 40: 0, # 'Т' - 52: 0, # 'У' - 53: 0, # 'Ф' - 55: 0, # 'Х' - 58: 0, # 'Ц' - 50: 0, # 'Ч' - 57: 0, # 'Ш' - 63: 0, # 'Щ' - 62: 0, # 'Ы' - 61: 0, # 'Ь' - 47: 0, # 'Э' - 59: 0, # 'Ю' - 43: 0, # 'Я' - 3: 2, # 'а' - 21: 3, # 'б' - 10: 3, # 'в' - 19: 3, # 'г' - 13: 3, # 'д' - 2: 3, # 'е' - 24: 3, # 'ж' - 20: 3, # 'з' - 4: 3, # 'и' - 23: 3, # 'й' - 11: 3, # 'к' - 8: 3, # 'л' - 12: 3, # 'м' - 5: 3, # 'н' - 1: 3, # 'о' - 15: 3, # 'п' - 9: 3, # 'р' - 7: 3, # 'с' - 6: 3, # 'т' - 14: 2, # 'у' - 39: 2, # 'ф' - 26: 3, # 'х' - 28: 2, # 'ц' - 22: 3, # 'ч' - 25: 3, # 'ш' - 29: 3, # 'щ' - 54: 0, # 'ъ' - 18: 0, # 'ы' - 17: 0, # 'ь' - 30: 2, # 'э' - 27: 3, # 'ю' - 16: 3, # 'я' - }, - 15: { # 'п' - 37: 0, # 'А' - 44: 0, # 'Б' - 33: 0, # 'В' - 46: 0, # 'Г' - 41: 0, # 'Д' - 48: 0, # 'Е' - 56: 0, # 'Ж' - 51: 0, # 'З' - 42: 0, # 'И' - 60: 0, # 'Й' - 36: 0, # 'К' - 49: 0, # 'Л' - 38: 0, # 'М' - 31: 0, # 'Н' - 34: 0, # 'О' - 35: 0, # 'П' - 45: 0, # 'Р' - 32: 0, # 'С' - 40: 0, # 'Т' - 52: 0, # 'У' - 53: 0, # 'Ф' - 55: 0, # 'Х' - 58: 0, # 'Ц' - 50: 0, # 'Ч' - 57: 0, # 'Ш' - 63: 0, # 'Щ' - 62: 0, # 'Ы' - 61: 0, # 'Ь' - 47: 0, # 'Э' - 59: 0, # 'Ю' - 43: 0, # 'Я' - 3: 3, # 'а' - 21: 1, # 'б' - 10: 0, # 'в' - 19: 0, # 'г' - 13: 0, # 'д' - 2: 3, # 'е' - 24: 0, # 'ж' - 20: 0, # 'з' - 4: 3, # 'и' - 23: 0, # 'й' - 11: 2, # 'к' - 8: 3, # 'л' - 12: 1, # 'м' - 5: 3, # 'н' - 1: 3, # 'о' - 15: 2, # 'п' - 9: 3, # 'р' - 7: 2, # 'с' - 6: 2, # 'т' - 14: 3, # 'у' - 39: 1, # 'ф' - 26: 0, # 'х' - 28: 2, # 'ц' - 22: 2, # 'ч' - 25: 1, # 'ш' - 29: 1, # 'щ' - 54: 0, # 'ъ' - 18: 3, # 'ы' - 17: 2, # 'ь' - 30: 1, # 'э' - 27: 1, # 'ю' - 16: 3, # 'я' - }, - 9: { # 'р' - 37: 0, # 'А' - 44: 0, # 'Б' - 33: 0, # 'В' - 46: 0, # 'Г' - 41: 0, # 'Д' - 48: 0, # 'Е' - 56: 0, # 'Ж' - 51: 0, # 'З' - 42: 0, # 'И' - 60: 0, # 'Й' - 36: 0, # 'К' - 49: 0, # 'Л' - 38: 0, # 'М' - 31: 0, # 'Н' - 34: 0, # 'О' - 35: 0, # 'П' - 45: 0, # 'Р' - 32: 0, # 'С' - 40: 0, # 'Т' - 52: 0, # 'У' - 53: 0, # 'Ф' - 55: 0, # 'Х' - 58: 0, # 'Ц' - 50: 0, # 'Ч' - 57: 0, # 'Ш' - 63: 0, # 'Щ' - 62: 0, # 'Ы' - 61: 0, # 'Ь' - 47: 0, # 'Э' - 59: 0, # 'Ю' - 43: 0, # 'Я' - 3: 3, # 'а' - 21: 2, # 'б' - 10: 3, # 'в' - 19: 3, # 'г' - 13: 3, # 'д' - 2: 3, # 'е' - 24: 3, # 'ж' - 20: 2, # 'з' - 4: 3, # 'и' - 23: 0, # 'й' - 11: 3, # 'к' - 8: 2, # 'л' - 12: 3, # 'м' - 5: 3, # 'н' - 1: 3, # 'о' - 15: 2, # 'п' - 9: 2, # 'р' - 7: 3, # 'с' - 6: 3, # 'т' - 14: 3, # 'у' - 39: 2, # 'ф' - 26: 3, # 'х' - 28: 2, # 'ц' - 22: 2, # 'ч' - 25: 3, # 'ш' - 29: 2, # 'щ' - 54: 0, # 'ъ' - 18: 3, # 'ы' - 17: 3, # 'ь' - 30: 2, # 'э' - 27: 2, # 'ю' - 16: 3, # 'я' - }, - 7: { # 'с' - 37: 0, # 'А' - 44: 0, # 'Б' - 33: 0, # 'В' - 46: 0, # 'Г' - 41: 0, # 'Д' - 48: 0, # 'Е' - 56: 0, # 'Ж' - 51: 1, # 'З' - 42: 0, # 'И' - 60: 0, # 'Й' - 36: 0, # 'К' - 49: 0, # 'Л' - 38: 0, # 'М' - 31: 0, # 'Н' - 34: 0, # 'О' - 35: 0, # 'П' - 45: 0, # 'Р' - 32: 0, # 'С' - 40: 0, # 'Т' - 52: 0, # 'У' - 53: 0, # 'Ф' - 55: 0, # 'Х' - 58: 0, # 'Ц' - 50: 0, # 'Ч' - 57: 0, # 'Ш' - 63: 0, # 'Щ' - 62: 0, # 'Ы' - 61: 0, # 'Ь' - 47: 0, # 'Э' - 59: 0, # 'Ю' - 43: 0, # 'Я' - 3: 3, # 'а' - 21: 2, # 'б' - 10: 3, # 'в' - 19: 2, # 'г' - 13: 3, # 'д' - 2: 3, # 'е' - 24: 2, # 'ж' - 20: 2, # 'з' - 4: 3, # 'и' - 23: 0, # 'й' - 11: 3, # 'к' - 8: 3, # 'л' - 12: 3, # 'м' - 5: 3, # 'н' - 1: 3, # 'о' - 15: 3, # 'п' - 9: 3, # 'р' - 7: 3, # 'с' - 6: 3, # 'т' - 14: 3, # 'у' - 39: 2, # 'ф' - 26: 3, # 'х' - 28: 2, # 'ц' - 22: 3, # 'ч' - 25: 2, # 'ш' - 29: 1, # 'щ' - 54: 2, # 'ъ' - 18: 3, # 'ы' - 17: 3, # 'ь' - 30: 2, # 'э' - 27: 3, # 'ю' - 16: 3, # 'я' - }, - 6: { # 'т' - 37: 0, # 'А' - 44: 0, # 'Б' - 33: 0, # 'В' - 46: 0, # 'Г' - 41: 0, # 'Д' - 48: 0, # 'Е' - 56: 0, # 'Ж' - 51: 0, # 'З' - 42: 0, # 'И' - 60: 0, # 'Й' - 36: 0, # 'К' - 49: 0, # 'Л' - 38: 0, # 'М' - 31: 0, # 'Н' - 34: 0, # 'О' - 35: 0, # 'П' - 45: 0, # 'Р' - 32: 0, # 'С' - 40: 0, # 'Т' - 52: 0, # 'У' - 53: 0, # 'Ф' - 55: 0, # 'Х' - 58: 0, # 'Ц' - 50: 0, # 'Ч' - 57: 0, # 'Ш' - 63: 0, # 'Щ' - 62: 0, # 'Ы' - 61: 0, # 'Ь' - 47: 0, # 'Э' - 59: 0, # 'Ю' - 43: 0, # 'Я' - 3: 3, # 'а' - 21: 2, # 'б' - 10: 3, # 'в' - 19: 2, # 'г' - 13: 2, # 'д' - 2: 3, # 'е' - 24: 1, # 'ж' - 20: 1, # 'з' - 4: 3, # 'и' - 23: 0, # 'й' - 11: 3, # 'к' - 8: 3, # 'л' - 12: 2, # 'м' - 5: 3, # 'н' - 1: 3, # 'о' - 15: 2, # 'п' - 9: 3, # 'р' - 7: 3, # 'с' - 6: 2, # 'т' - 14: 3, # 'у' - 39: 2, # 'ф' - 26: 2, # 'х' - 28: 2, # 'ц' - 22: 2, # 'ч' - 25: 2, # 'ш' - 29: 2, # 'щ' - 54: 2, # 'ъ' - 18: 3, # 'ы' - 17: 3, # 'ь' - 30: 2, # 'э' - 27: 2, # 'ю' - 16: 3, # 'я' - }, - 14: { # 'у' - 37: 0, # 'А' - 44: 0, # 'Б' - 33: 0, # 'В' - 46: 0, # 'Г' - 41: 0, # 'Д' - 48: 0, # 'Е' - 56: 0, # 'Ж' - 51: 0, # 'З' - 42: 0, # 'И' - 60: 0, # 'Й' - 36: 0, # 'К' - 49: 0, # 'Л' - 38: 0, # 'М' - 31: 0, # 'Н' - 34: 0, # 'О' - 35: 0, # 'П' - 45: 0, # 'Р' - 32: 0, # 'С' - 40: 0, # 'Т' - 52: 0, # 'У' - 53: 0, # 'Ф' - 55: 0, # 'Х' - 58: 0, # 'Ц' - 50: 0, # 'Ч' - 57: 0, # 'Ш' - 63: 0, # 'Щ' - 62: 0, # 'Ы' - 61: 0, # 'Ь' - 47: 0, # 'Э' - 59: 0, # 'Ю' - 43: 0, # 'Я' - 3: 2, # 'а' - 21: 3, # 'б' - 10: 3, # 'в' - 19: 3, # 'г' - 13: 3, # 'д' - 2: 3, # 'е' - 24: 3, # 'ж' - 20: 3, # 'з' - 4: 2, # 'и' - 23: 2, # 'й' - 11: 3, # 'к' - 8: 3, # 'л' - 12: 3, # 'м' - 5: 3, # 'н' - 1: 2, # 'о' - 15: 3, # 'п' - 9: 3, # 'р' - 7: 3, # 'с' - 6: 3, # 'т' - 14: 1, # 'у' - 39: 2, # 'ф' - 26: 3, # 'х' - 28: 2, # 'ц' - 22: 3, # 'ч' - 25: 3, # 'ш' - 29: 3, # 'щ' - 54: 0, # 'ъ' - 18: 0, # 'ы' - 17: 0, # 'ь' - 30: 2, # 'э' - 27: 3, # 'ю' - 16: 2, # 'я' - }, - 39: { # 'ф' - 37: 0, # 'А' - 44: 0, # 'Б' - 33: 0, # 'В' - 46: 0, # 'Г' - 41: 0, # 'Д' - 48: 0, # 'Е' - 56: 0, # 'Ж' - 51: 0, # 'З' - 42: 0, # 'И' - 60: 0, # 'Й' - 36: 0, # 'К' - 49: 0, # 'Л' - 38: 0, # 'М' - 31: 0, # 'Н' - 34: 0, # 'О' - 35: 0, # 'П' - 45: 0, # 'Р' - 32: 0, # 'С' - 40: 0, # 'Т' - 52: 0, # 'У' - 53: 0, # 'Ф' - 55: 0, # 'Х' - 58: 0, # 'Ц' - 50: 0, # 'Ч' - 57: 0, # 'Ш' - 63: 0, # 'Щ' - 62: 0, # 'Ы' - 61: 0, # 'Ь' - 47: 0, # 'Э' - 59: 0, # 'Ю' - 43: 0, # 'Я' - 3: 3, # 'а' - 21: 1, # 'б' - 10: 0, # 'в' - 19: 1, # 'г' - 13: 0, # 'д' - 2: 3, # 'е' - 24: 0, # 'ж' - 20: 0, # 'з' - 4: 3, # 'и' - 23: 0, # 'й' - 11: 1, # 'к' - 8: 2, # 'л' - 12: 1, # 'м' - 5: 1, # 'н' - 1: 3, # 'о' - 15: 1, # 'п' - 9: 2, # 'р' - 7: 2, # 'с' - 6: 2, # 'т' - 14: 2, # 'у' - 39: 2, # 'ф' - 26: 0, # 'х' - 28: 0, # 'ц' - 22: 1, # 'ч' - 25: 1, # 'ш' - 29: 0, # 'щ' - 54: 0, # 'ъ' - 18: 2, # 'ы' - 17: 1, # 'ь' - 30: 2, # 'э' - 27: 1, # 'ю' - 16: 1, # 'я' - }, - 26: { # 'х' - 37: 0, # 'А' - 44: 0, # 'Б' - 33: 0, # 'В' - 46: 0, # 'Г' - 41: 0, # 'Д' - 48: 0, # 'Е' - 56: 0, # 'Ж' - 51: 0, # 'З' - 42: 0, # 'И' - 60: 0, # 'Й' - 36: 0, # 'К' - 49: 0, # 'Л' - 38: 0, # 'М' - 31: 0, # 'Н' - 34: 0, # 'О' - 35: 0, # 'П' - 45: 0, # 'Р' - 32: 0, # 'С' - 40: 0, # 'Т' - 52: 0, # 'У' - 53: 0, # 'Ф' - 55: 0, # 'Х' - 58: 0, # 'Ц' - 50: 0, # 'Ч' - 57: 0, # 'Ш' - 63: 0, # 'Щ' - 62: 0, # 'Ы' - 61: 0, # 'Ь' - 47: 0, # 'Э' - 59: 0, # 'Ю' - 43: 0, # 'Я' - 3: 3, # 'а' - 21: 0, # 'б' - 10: 3, # 'в' - 19: 1, # 'г' - 13: 1, # 'д' - 2: 2, # 'е' - 24: 0, # 'ж' - 20: 1, # 'з' - 4: 3, # 'и' - 23: 0, # 'й' - 11: 1, # 'к' - 8: 2, # 'л' - 12: 2, # 'м' - 5: 3, # 'н' - 1: 3, # 'о' - 15: 1, # 'п' - 9: 3, # 'р' - 7: 2, # 'с' - 6: 2, # 'т' - 14: 2, # 'у' - 39: 1, # 'ф' - 26: 1, # 'х' - 28: 1, # 'ц' - 22: 1, # 'ч' - 25: 2, # 'ш' - 29: 0, # 'щ' - 54: 1, # 'ъ' - 18: 0, # 'ы' - 17: 1, # 'ь' - 30: 1, # 'э' - 27: 1, # 'ю' - 16: 0, # 'я' - }, - 28: { # 'ц' - 37: 0, # 'А' - 44: 0, # 'Б' - 33: 0, # 'В' - 46: 0, # 'Г' - 41: 0, # 'Д' - 48: 0, # 'Е' - 56: 0, # 'Ж' - 51: 0, # 'З' - 42: 0, # 'И' - 60: 0, # 'Й' - 36: 0, # 'К' - 49: 0, # 'Л' - 38: 0, # 'М' - 31: 0, # 'Н' - 34: 0, # 'О' - 35: 0, # 'П' - 45: 0, # 'Р' - 32: 0, # 'С' - 40: 0, # 'Т' - 52: 0, # 'У' - 53: 0, # 'Ф' - 55: 0, # 'Х' - 58: 0, # 'Ц' - 50: 0, # 'Ч' - 57: 0, # 'Ш' - 63: 0, # 'Щ' - 62: 0, # 'Ы' - 61: 0, # 'Ь' - 47: 0, # 'Э' - 59: 0, # 'Ю' - 43: 0, # 'Я' - 3: 3, # 'а' - 21: 1, # 'б' - 10: 2, # 'в' - 19: 1, # 'г' - 13: 1, # 'д' - 2: 3, # 'е' - 24: 0, # 'ж' - 20: 1, # 'з' - 4: 3, # 'и' - 23: 0, # 'й' - 11: 2, # 'к' - 8: 1, # 'л' - 12: 1, # 'м' - 5: 1, # 'н' - 1: 3, # 'о' - 15: 0, # 'п' - 9: 1, # 'р' - 7: 0, # 'с' - 6: 1, # 'т' - 14: 3, # 'у' - 39: 0, # 'ф' - 26: 0, # 'х' - 28: 1, # 'ц' - 22: 0, # 'ч' - 25: 1, # 'ш' - 29: 0, # 'щ' - 54: 0, # 'ъ' - 18: 3, # 'ы' - 17: 1, # 'ь' - 30: 0, # 'э' - 27: 1, # 'ю' - 16: 0, # 'я' - }, - 22: { # 'ч' - 37: 0, # 'А' - 44: 0, # 'Б' - 33: 0, # 'В' - 46: 0, # 'Г' - 41: 0, # 'Д' - 48: 0, # 'Е' - 56: 0, # 'Ж' - 51: 0, # 'З' - 42: 0, # 'И' - 60: 0, # 'Й' - 36: 0, # 'К' - 49: 0, # 'Л' - 38: 0, # 'М' - 31: 0, # 'Н' - 34: 0, # 'О' - 35: 0, # 'П' - 45: 0, # 'Р' - 32: 0, # 'С' - 40: 0, # 'Т' - 52: 0, # 'У' - 53: 0, # 'Ф' - 55: 0, # 'Х' - 58: 0, # 'Ц' - 50: 0, # 'Ч' - 57: 0, # 'Ш' - 63: 0, # 'Щ' - 62: 0, # 'Ы' - 61: 0, # 'Ь' - 47: 0, # 'Э' - 59: 0, # 'Ю' - 43: 0, # 'Я' - 3: 3, # 'а' - 21: 1, # 'б' - 10: 1, # 'в' - 19: 0, # 'г' - 13: 0, # 'д' - 2: 3, # 'е' - 24: 1, # 'ж' - 20: 0, # 'з' - 4: 3, # 'и' - 23: 0, # 'й' - 11: 3, # 'к' - 8: 2, # 'л' - 12: 1, # 'м' - 5: 3, # 'н' - 1: 2, # 'о' - 15: 0, # 'п' - 9: 2, # 'р' - 7: 1, # 'с' - 6: 3, # 'т' - 14: 3, # 'у' - 39: 1, # 'ф' - 26: 1, # 'х' - 28: 0, # 'ц' - 22: 1, # 'ч' - 25: 2, # 'ш' - 29: 0, # 'щ' - 54: 0, # 'ъ' - 18: 0, # 'ы' - 17: 3, # 'ь' - 30: 0, # 'э' - 27: 0, # 'ю' - 16: 0, # 'я' - }, - 25: { # 'ш' - 37: 0, # 'А' - 44: 0, # 'Б' - 33: 0, # 'В' - 46: 0, # 'Г' - 41: 0, # 'Д' - 48: 0, # 'Е' - 56: 0, # 'Ж' - 51: 0, # 'З' - 42: 0, # 'И' - 60: 0, # 'Й' - 36: 0, # 'К' - 49: 0, # 'Л' - 38: 0, # 'М' - 31: 0, # 'Н' - 34: 0, # 'О' - 35: 0, # 'П' - 45: 0, # 'Р' - 32: 0, # 'С' - 40: 0, # 'Т' - 52: 0, # 'У' - 53: 0, # 'Ф' - 55: 0, # 'Х' - 58: 0, # 'Ц' - 50: 0, # 'Ч' - 57: 0, # 'Ш' - 63: 0, # 'Щ' - 62: 0, # 'Ы' - 61: 0, # 'Ь' - 47: 0, # 'Э' - 59: 0, # 'Ю' - 43: 0, # 'Я' - 3: 3, # 'а' - 21: 1, # 'б' - 10: 2, # 'в' - 19: 1, # 'г' - 13: 0, # 'д' - 2: 3, # 'е' - 24: 0, # 'ж' - 20: 0, # 'з' - 4: 3, # 'и' - 23: 0, # 'й' - 11: 3, # 'к' - 8: 3, # 'л' - 12: 2, # 'м' - 5: 3, # 'н' - 1: 3, # 'о' - 15: 2, # 'п' - 9: 2, # 'р' - 7: 1, # 'с' - 6: 2, # 'т' - 14: 3, # 'у' - 39: 2, # 'ф' - 26: 1, # 'х' - 28: 1, # 'ц' - 22: 1, # 'ч' - 25: 1, # 'ш' - 29: 0, # 'щ' - 54: 0, # 'ъ' - 18: 0, # 'ы' - 17: 3, # 'ь' - 30: 1, # 'э' - 27: 1, # 'ю' - 16: 0, # 'я' - }, - 29: { # 'щ' - 37: 0, # 'А' - 44: 0, # 'Б' - 33: 0, # 'В' - 46: 0, # 'Г' - 41: 0, # 'Д' - 48: 0, # 'Е' - 56: 0, # 'Ж' - 51: 0, # 'З' - 42: 0, # 'И' - 60: 0, # 'Й' - 36: 0, # 'К' - 49: 0, # 'Л' - 38: 0, # 'М' - 31: 0, # 'Н' - 34: 0, # 'О' - 35: 0, # 'П' - 45: 0, # 'Р' - 32: 0, # 'С' - 40: 0, # 'Т' - 52: 0, # 'У' - 53: 0, # 'Ф' - 55: 0, # 'Х' - 58: 0, # 'Ц' - 50: 0, # 'Ч' - 57: 0, # 'Ш' - 63: 0, # 'Щ' - 62: 0, # 'Ы' - 61: 0, # 'Ь' - 47: 0, # 'Э' - 59: 0, # 'Ю' - 43: 0, # 'Я' - 3: 3, # 'а' - 21: 0, # 'б' - 10: 1, # 'в' - 19: 0, # 'г' - 13: 0, # 'д' - 2: 3, # 'е' - 24: 0, # 'ж' - 20: 0, # 'з' - 4: 3, # 'и' - 23: 0, # 'й' - 11: 0, # 'к' - 8: 0, # 'л' - 12: 1, # 'м' - 5: 2, # 'н' - 1: 1, # 'о' - 15: 0, # 'п' - 9: 2, # 'р' - 7: 0, # 'с' - 6: 0, # 'т' - 14: 2, # 'у' - 39: 0, # 'ф' - 26: 0, # 'х' - 28: 0, # 'ц' - 22: 0, # 'ч' - 25: 0, # 'ш' - 29: 0, # 'щ' - 54: 0, # 'ъ' - 18: 0, # 'ы' - 17: 2, # 'ь' - 30: 0, # 'э' - 27: 0, # 'ю' - 16: 0, # 'я' - }, - 54: { # 'ъ' - 37: 0, # 'А' - 44: 0, # 'Б' - 33: 0, # 'В' - 46: 0, # 'Г' - 41: 0, # 'Д' - 48: 0, # 'Е' - 56: 0, # 'Ж' - 51: 0, # 'З' - 42: 0, # 'И' - 60: 0, # 'Й' - 36: 0, # 'К' - 49: 0, # 'Л' - 38: 0, # 'М' - 31: 0, # 'Н' - 34: 0, # 'О' - 35: 0, # 'П' - 45: 0, # 'Р' - 32: 0, # 'С' - 40: 0, # 'Т' - 52: 0, # 'У' - 53: 0, # 'Ф' - 55: 0, # 'Х' - 58: 0, # 'Ц' - 50: 0, # 'Ч' - 57: 0, # 'Ш' - 63: 0, # 'Щ' - 62: 0, # 'Ы' - 61: 0, # 'Ь' - 47: 0, # 'Э' - 59: 0, # 'Ю' - 43: 0, # 'Я' - 3: 0, # 'а' - 21: 0, # 'б' - 10: 0, # 'в' - 19: 0, # 'г' - 13: 0, # 'д' - 2: 2, # 'е' - 24: 0, # 'ж' - 20: 0, # 'з' - 4: 0, # 'и' - 23: 0, # 'й' - 11: 0, # 'к' - 8: 0, # 'л' - 12: 0, # 'м' - 5: 0, # 'н' - 1: 0, # 'о' - 15: 0, # 'п' - 9: 0, # 'р' - 7: 0, # 'с' - 6: 0, # 'т' - 14: 0, # 'у' - 39: 0, # 'ф' - 26: 0, # 'х' - 28: 0, # 'ц' - 22: 0, # 'ч' - 25: 0, # 'ш' - 29: 0, # 'щ' - 54: 0, # 'ъ' - 18: 0, # 'ы' - 17: 0, # 'ь' - 30: 0, # 'э' - 27: 1, # 'ю' - 16: 2, # 'я' - }, - 18: { # 'ы' - 37: 0, # 'А' - 44: 0, # 'Б' - 33: 0, # 'В' - 46: 0, # 'Г' - 41: 0, # 'Д' - 48: 0, # 'Е' - 56: 0, # 'Ж' - 51: 0, # 'З' - 42: 0, # 'И' - 60: 0, # 'Й' - 36: 0, # 'К' - 49: 0, # 'Л' - 38: 0, # 'М' - 31: 0, # 'Н' - 34: 0, # 'О' - 35: 0, # 'П' - 45: 0, # 'Р' - 32: 0, # 'С' - 40: 0, # 'Т' - 52: 0, # 'У' - 53: 0, # 'Ф' - 55: 0, # 'Х' - 58: 0, # 'Ц' - 50: 0, # 'Ч' - 57: 0, # 'Ш' - 63: 0, # 'Щ' - 62: 0, # 'Ы' - 61: 0, # 'Ь' - 47: 0, # 'Э' - 59: 0, # 'Ю' - 43: 0, # 'Я' - 3: 0, # 'а' - 21: 3, # 'б' - 10: 3, # 'в' - 19: 2, # 'г' - 13: 2, # 'д' - 2: 3, # 'е' - 24: 2, # 'ж' - 20: 2, # 'з' - 4: 2, # 'и' - 23: 3, # 'й' - 11: 3, # 'к' - 8: 3, # 'л' - 12: 3, # 'м' - 5: 3, # 'н' - 1: 1, # 'о' - 15: 3, # 'п' - 9: 3, # 'р' - 7: 3, # 'с' - 6: 3, # 'т' - 14: 1, # 'у' - 39: 0, # 'ф' - 26: 3, # 'х' - 28: 2, # 'ц' - 22: 3, # 'ч' - 25: 3, # 'ш' - 29: 2, # 'щ' - 54: 0, # 'ъ' - 18: 0, # 'ы' - 17: 0, # 'ь' - 30: 0, # 'э' - 27: 0, # 'ю' - 16: 2, # 'я' - }, - 17: { # 'ь' - 37: 0, # 'А' - 44: 0, # 'Б' - 33: 0, # 'В' - 46: 0, # 'Г' - 41: 0, # 'Д' - 48: 0, # 'Е' - 56: 0, # 'Ж' - 51: 0, # 'З' - 42: 0, # 'И' - 60: 0, # 'Й' - 36: 0, # 'К' - 49: 0, # 'Л' - 38: 0, # 'М' - 31: 0, # 'Н' - 34: 0, # 'О' - 35: 0, # 'П' - 45: 0, # 'Р' - 32: 0, # 'С' - 40: 0, # 'Т' - 52: 0, # 'У' - 53: 0, # 'Ф' - 55: 0, # 'Х' - 58: 0, # 'Ц' - 50: 0, # 'Ч' - 57: 0, # 'Ш' - 63: 0, # 'Щ' - 62: 0, # 'Ы' - 61: 0, # 'Ь' - 47: 0, # 'Э' - 59: 0, # 'Ю' - 43: 0, # 'Я' - 3: 0, # 'а' - 21: 2, # 'б' - 10: 2, # 'в' - 19: 2, # 'г' - 13: 2, # 'д' - 2: 3, # 'е' - 24: 1, # 'ж' - 20: 3, # 'з' - 4: 2, # 'и' - 23: 0, # 'й' - 11: 3, # 'к' - 8: 0, # 'л' - 12: 3, # 'м' - 5: 3, # 'н' - 1: 2, # 'о' - 15: 2, # 'п' - 9: 1, # 'р' - 7: 3, # 'с' - 6: 2, # 'т' - 14: 0, # 'у' - 39: 2, # 'ф' - 26: 1, # 'х' - 28: 2, # 'ц' - 22: 2, # 'ч' - 25: 3, # 'ш' - 29: 2, # 'щ' - 54: 0, # 'ъ' - 18: 0, # 'ы' - 17: 0, # 'ь' - 30: 1, # 'э' - 27: 3, # 'ю' - 16: 3, # 'я' - }, - 30: { # 'э' - 37: 0, # 'А' - 44: 0, # 'Б' - 33: 0, # 'В' - 46: 0, # 'Г' - 41: 0, # 'Д' - 48: 0, # 'Е' - 56: 0, # 'Ж' - 51: 0, # 'З' - 42: 0, # 'И' - 60: 0, # 'Й' - 36: 0, # 'К' - 49: 0, # 'Л' - 38: 1, # 'М' - 31: 1, # 'Н' - 34: 0, # 'О' - 35: 0, # 'П' - 45: 1, # 'Р' - 32: 1, # 'С' - 40: 0, # 'Т' - 52: 0, # 'У' - 53: 1, # 'Ф' - 55: 0, # 'Х' - 58: 0, # 'Ц' - 50: 0, # 'Ч' - 57: 0, # 'Ш' - 63: 0, # 'Щ' - 62: 0, # 'Ы' - 61: 0, # 'Ь' - 47: 0, # 'Э' - 59: 0, # 'Ю' - 43: 0, # 'Я' - 3: 0, # 'а' - 21: 1, # 'б' - 10: 1, # 'в' - 19: 1, # 'г' - 13: 2, # 'д' - 2: 1, # 'е' - 24: 0, # 'ж' - 20: 1, # 'з' - 4: 0, # 'и' - 23: 2, # 'й' - 11: 2, # 'к' - 8: 2, # 'л' - 12: 2, # 'м' - 5: 2, # 'н' - 1: 0, # 'о' - 15: 2, # 'п' - 9: 2, # 'р' - 7: 2, # 'с' - 6: 3, # 'т' - 14: 1, # 'у' - 39: 2, # 'ф' - 26: 1, # 'х' - 28: 0, # 'ц' - 22: 0, # 'ч' - 25: 1, # 'ш' - 29: 0, # 'щ' - 54: 0, # 'ъ' - 18: 0, # 'ы' - 17: 0, # 'ь' - 30: 1, # 'э' - 27: 1, # 'ю' - 16: 1, # 'я' - }, - 27: { # 'ю' - 37: 0, # 'А' - 44: 0, # 'Б' - 33: 0, # 'В' - 46: 0, # 'Г' - 41: 0, # 'Д' - 48: 0, # 'Е' - 56: 0, # 'Ж' - 51: 0, # 'З' - 42: 0, # 'И' - 60: 0, # 'Й' - 36: 0, # 'К' - 49: 0, # 'Л' - 38: 0, # 'М' - 31: 0, # 'Н' - 34: 0, # 'О' - 35: 0, # 'П' - 45: 0, # 'Р' - 32: 0, # 'С' - 40: 0, # 'Т' - 52: 0, # 'У' - 53: 0, # 'Ф' - 55: 0, # 'Х' - 58: 0, # 'Ц' - 50: 0, # 'Ч' - 57: 0, # 'Ш' - 63: 0, # 'Щ' - 62: 0, # 'Ы' - 61: 0, # 'Ь' - 47: 0, # 'Э' - 59: 0, # 'Ю' - 43: 0, # 'Я' - 3: 2, # 'а' - 21: 3, # 'б' - 10: 1, # 'в' - 19: 2, # 'г' - 13: 3, # 'д' - 2: 1, # 'е' - 24: 2, # 'ж' - 20: 2, # 'з' - 4: 1, # 'и' - 23: 1, # 'й' - 11: 2, # 'к' - 8: 2, # 'л' - 12: 2, # 'м' - 5: 2, # 'н' - 1: 1, # 'о' - 15: 2, # 'п' - 9: 2, # 'р' - 7: 3, # 'с' - 6: 3, # 'т' - 14: 0, # 'у' - 39: 1, # 'ф' - 26: 2, # 'х' - 28: 2, # 'ц' - 22: 2, # 'ч' - 25: 2, # 'ш' - 29: 3, # 'щ' - 54: 0, # 'ъ' - 18: 0, # 'ы' - 17: 0, # 'ь' - 30: 1, # 'э' - 27: 2, # 'ю' - 16: 1, # 'я' - }, - 16: { # 'я' - 37: 0, # 'А' - 44: 0, # 'Б' - 33: 0, # 'В' - 46: 0, # 'Г' - 41: 0, # 'Д' - 48: 0, # 'Е' - 56: 0, # 'Ж' - 51: 0, # 'З' - 42: 0, # 'И' - 60: 0, # 'Й' - 36: 0, # 'К' - 49: 0, # 'Л' - 38: 0, # 'М' - 31: 0, # 'Н' - 34: 0, # 'О' - 35: 0, # 'П' - 45: 0, # 'Р' - 32: 0, # 'С' - 40: 0, # 'Т' - 52: 0, # 'У' - 53: 0, # 'Ф' - 55: 0, # 'Х' - 58: 0, # 'Ц' - 50: 0, # 'Ч' - 57: 0, # 'Ш' - 63: 0, # 'Щ' - 62: 0, # 'Ы' - 61: 0, # 'Ь' - 47: 0, # 'Э' - 59: 0, # 'Ю' - 43: 0, # 'Я' - 3: 0, # 'а' - 21: 2, # 'б' - 10: 3, # 'в' - 19: 2, # 'г' - 13: 3, # 'д' - 2: 3, # 'е' - 24: 3, # 'ж' - 20: 3, # 'з' - 4: 2, # 'и' - 23: 2, # 'й' - 11: 3, # 'к' - 8: 3, # 'л' - 12: 3, # 'м' - 5: 3, # 'н' - 1: 0, # 'о' - 15: 2, # 'п' - 9: 2, # 'р' - 7: 3, # 'с' - 6: 3, # 'т' - 14: 1, # 'у' - 39: 1, # 'ф' - 26: 3, # 'х' - 28: 2, # 'ц' - 22: 2, # 'ч' - 25: 2, # 'ш' - 29: 3, # 'щ' - 54: 0, # 'ъ' - 18: 0, # 'ы' - 17: 0, # 'ь' - 30: 0, # 'э' - 27: 2, # 'ю' - 16: 2, # 'я' - }, -} - -# 255: Undefined characters that did not exist in training text -# 254: Carriage/Return -# 253: symbol (punctuation) that does not belong to word -# 252: 0 - 9 -# 251: Control characters - -# Character Mapping Table(s): -IBM866_RUSSIAN_CHAR_TO_ORDER = { - 0: 255, # '\x00' - 1: 255, # '\x01' - 2: 255, # '\x02' - 3: 255, # '\x03' - 4: 255, # '\x04' - 5: 255, # '\x05' - 6: 255, # '\x06' - 7: 255, # '\x07' - 8: 255, # '\x08' - 9: 255, # '\t' - 10: 254, # '\n' - 11: 255, # '\x0b' - 12: 255, # '\x0c' - 13: 254, # '\r' - 14: 255, # '\x0e' - 15: 255, # '\x0f' - 16: 255, # '\x10' - 17: 255, # '\x11' - 18: 255, # '\x12' - 19: 255, # '\x13' - 20: 255, # '\x14' - 21: 255, # '\x15' - 22: 255, # '\x16' - 23: 255, # '\x17' - 24: 255, # '\x18' - 25: 255, # '\x19' - 26: 255, # '\x1a' - 27: 255, # '\x1b' - 28: 255, # '\x1c' - 29: 255, # '\x1d' - 30: 255, # '\x1e' - 31: 255, # '\x1f' - 32: 253, # ' ' - 33: 253, # '!' - 34: 253, # '"' - 35: 253, # '#' - 36: 253, # '$' - 37: 253, # '%' - 38: 253, # '&' - 39: 253, # "'" - 40: 253, # '(' - 41: 253, # ')' - 42: 253, # '*' - 43: 253, # '+' - 44: 253, # ',' - 45: 253, # '-' - 46: 253, # '.' - 47: 253, # '/' - 48: 252, # '0' - 49: 252, # '1' - 50: 252, # '2' - 51: 252, # '3' - 52: 252, # '4' - 53: 252, # '5' - 54: 252, # '6' - 55: 252, # '7' - 56: 252, # '8' - 57: 252, # '9' - 58: 253, # ':' - 59: 253, # ';' - 60: 253, # '<' - 61: 253, # '=' - 62: 253, # '>' - 63: 253, # '?' - 64: 253, # '@' - 65: 142, # 'A' - 66: 143, # 'B' - 67: 144, # 'C' - 68: 145, # 'D' - 69: 146, # 'E' - 70: 147, # 'F' - 71: 148, # 'G' - 72: 149, # 'H' - 73: 150, # 'I' - 74: 151, # 'J' - 75: 152, # 'K' - 76: 74, # 'L' - 77: 153, # 'M' - 78: 75, # 'N' - 79: 154, # 'O' - 80: 155, # 'P' - 81: 156, # 'Q' - 82: 157, # 'R' - 83: 158, # 'S' - 84: 159, # 'T' - 85: 160, # 'U' - 86: 161, # 'V' - 87: 162, # 'W' - 88: 163, # 'X' - 89: 164, # 'Y' - 90: 165, # 'Z' - 91: 253, # '[' - 92: 253, # '\\' - 93: 253, # ']' - 94: 253, # '^' - 95: 253, # '_' - 96: 253, # '`' - 97: 71, # 'a' - 98: 172, # 'b' - 99: 66, # 'c' - 100: 173, # 'd' - 101: 65, # 'e' - 102: 174, # 'f' - 103: 76, # 'g' - 104: 175, # 'h' - 105: 64, # 'i' - 106: 176, # 'j' - 107: 177, # 'k' - 108: 77, # 'l' - 109: 72, # 'm' - 110: 178, # 'n' - 111: 69, # 'o' - 112: 67, # 'p' - 113: 179, # 'q' - 114: 78, # 'r' - 115: 73, # 's' - 116: 180, # 't' - 117: 181, # 'u' - 118: 79, # 'v' - 119: 182, # 'w' - 120: 183, # 'x' - 121: 184, # 'y' - 122: 185, # 'z' - 123: 253, # '{' - 124: 253, # '|' - 125: 253, # '}' - 126: 253, # '~' - 127: 253, # '\x7f' - 128: 37, # 'А' - 129: 44, # 'Б' - 130: 33, # 'В' - 131: 46, # 'Г' - 132: 41, # 'Д' - 133: 48, # 'Е' - 134: 56, # 'Ж' - 135: 51, # 'З' - 136: 42, # 'И' - 137: 60, # 'Й' - 138: 36, # 'К' - 139: 49, # 'Л' - 140: 38, # 'М' - 141: 31, # 'Н' - 142: 34, # 'О' - 143: 35, # 'П' - 144: 45, # 'Р' - 145: 32, # 'С' - 146: 40, # 'Т' - 147: 52, # 'У' - 148: 53, # 'Ф' - 149: 55, # 'Х' - 150: 58, # 'Ц' - 151: 50, # 'Ч' - 152: 57, # 'Ш' - 153: 63, # 'Щ' - 154: 70, # 'Ъ' - 155: 62, # 'Ы' - 156: 61, # 'Ь' - 157: 47, # 'Э' - 158: 59, # 'Ю' - 159: 43, # 'Я' - 160: 3, # 'а' - 161: 21, # 'б' - 162: 10, # 'в' - 163: 19, # 'г' - 164: 13, # 'д' - 165: 2, # 'е' - 166: 24, # 'ж' - 167: 20, # 'з' - 168: 4, # 'и' - 169: 23, # 'й' - 170: 11, # 'к' - 171: 8, # 'л' - 172: 12, # 'м' - 173: 5, # 'н' - 174: 1, # 'о' - 175: 15, # 'п' - 176: 191, # '░' - 177: 192, # '▒' - 178: 193, # '▓' - 179: 194, # '│' - 180: 195, # '┤' - 181: 196, # '╡' - 182: 197, # '╢' - 183: 198, # '╖' - 184: 199, # '╕' - 185: 200, # '╣' - 186: 201, # '║' - 187: 202, # '╗' - 188: 203, # '╝' - 189: 204, # '╜' - 190: 205, # '╛' - 191: 206, # '┐' - 192: 207, # '└' - 193: 208, # '┴' - 194: 209, # '┬' - 195: 210, # '├' - 196: 211, # '─' - 197: 212, # '┼' - 198: 213, # '╞' - 199: 214, # '╟' - 200: 215, # '╚' - 201: 216, # '╔' - 202: 217, # '╩' - 203: 218, # '╦' - 204: 219, # '╠' - 205: 220, # '═' - 206: 221, # '╬' - 207: 222, # '╧' - 208: 223, # '╨' - 209: 224, # '╤' - 210: 225, # '╥' - 211: 226, # '╙' - 212: 227, # '╘' - 213: 228, # '╒' - 214: 229, # '╓' - 215: 230, # '╫' - 216: 231, # '╪' - 217: 232, # '┘' - 218: 233, # '┌' - 219: 234, # '█' - 220: 235, # '▄' - 221: 236, # '▌' - 222: 237, # '▐' - 223: 238, # '▀' - 224: 9, # 'р' - 225: 7, # 'с' - 226: 6, # 'т' - 227: 14, # 'у' - 228: 39, # 'ф' - 229: 26, # 'х' - 230: 28, # 'ц' - 231: 22, # 'ч' - 232: 25, # 'ш' - 233: 29, # 'щ' - 234: 54, # 'ъ' - 235: 18, # 'ы' - 236: 17, # 'ь' - 237: 30, # 'э' - 238: 27, # 'ю' - 239: 16, # 'я' - 240: 239, # 'Ё' - 241: 68, # 'ё' - 242: 240, # 'Є' - 243: 241, # 'є' - 244: 242, # 'Ї' - 245: 243, # 'ї' - 246: 244, # 'Ў' - 247: 245, # 'ў' - 248: 246, # '°' - 249: 247, # '∙' - 250: 248, # '·' - 251: 249, # '√' - 252: 250, # '№' - 253: 251, # '¤' - 254: 252, # '■' - 255: 255, # '\xa0' -} - -IBM866_RUSSIAN_MODEL = SingleByteCharSetModel( - charset_name="IBM866", - language="Russian", - char_to_order_map=IBM866_RUSSIAN_CHAR_TO_ORDER, - language_model=RUSSIAN_LANG_MODEL, - typical_positive_ratio=0.976601, - keep_ascii_letters=False, - alphabet="ЁАБВГДЕЖЗИЙКЛМНОПРСТУФХЦЧШЩЪЫЬЭЮЯабвгдежзийклмнопрстуфхцчшщъыьэюяё", -) - -WINDOWS_1251_RUSSIAN_CHAR_TO_ORDER = { - 0: 255, # '\x00' - 1: 255, # '\x01' - 2: 255, # '\x02' - 3: 255, # '\x03' - 4: 255, # '\x04' - 5: 255, # '\x05' - 6: 255, # '\x06' - 7: 255, # '\x07' - 8: 255, # '\x08' - 9: 255, # '\t' - 10: 254, # '\n' - 11: 255, # '\x0b' - 12: 255, # '\x0c' - 13: 254, # '\r' - 14: 255, # '\x0e' - 15: 255, # '\x0f' - 16: 255, # '\x10' - 17: 255, # '\x11' - 18: 255, # '\x12' - 19: 255, # '\x13' - 20: 255, # '\x14' - 21: 255, # '\x15' - 22: 255, # '\x16' - 23: 255, # '\x17' - 24: 255, # '\x18' - 25: 255, # '\x19' - 26: 255, # '\x1a' - 27: 255, # '\x1b' - 28: 255, # '\x1c' - 29: 255, # '\x1d' - 30: 255, # '\x1e' - 31: 255, # '\x1f' - 32: 253, # ' ' - 33: 253, # '!' - 34: 253, # '"' - 35: 253, # '#' - 36: 253, # '$' - 37: 253, # '%' - 38: 253, # '&' - 39: 253, # "'" - 40: 253, # '(' - 41: 253, # ')' - 42: 253, # '*' - 43: 253, # '+' - 44: 253, # ',' - 45: 253, # '-' - 46: 253, # '.' - 47: 253, # '/' - 48: 252, # '0' - 49: 252, # '1' - 50: 252, # '2' - 51: 252, # '3' - 52: 252, # '4' - 53: 252, # '5' - 54: 252, # '6' - 55: 252, # '7' - 56: 252, # '8' - 57: 252, # '9' - 58: 253, # ':' - 59: 253, # ';' - 60: 253, # '<' - 61: 253, # '=' - 62: 253, # '>' - 63: 253, # '?' - 64: 253, # '@' - 65: 142, # 'A' - 66: 143, # 'B' - 67: 144, # 'C' - 68: 145, # 'D' - 69: 146, # 'E' - 70: 147, # 'F' - 71: 148, # 'G' - 72: 149, # 'H' - 73: 150, # 'I' - 74: 151, # 'J' - 75: 152, # 'K' - 76: 74, # 'L' - 77: 153, # 'M' - 78: 75, # 'N' - 79: 154, # 'O' - 80: 155, # 'P' - 81: 156, # 'Q' - 82: 157, # 'R' - 83: 158, # 'S' - 84: 159, # 'T' - 85: 160, # 'U' - 86: 161, # 'V' - 87: 162, # 'W' - 88: 163, # 'X' - 89: 164, # 'Y' - 90: 165, # 'Z' - 91: 253, # '[' - 92: 253, # '\\' - 93: 253, # ']' - 94: 253, # '^' - 95: 253, # '_' - 96: 253, # '`' - 97: 71, # 'a' - 98: 172, # 'b' - 99: 66, # 'c' - 100: 173, # 'd' - 101: 65, # 'e' - 102: 174, # 'f' - 103: 76, # 'g' - 104: 175, # 'h' - 105: 64, # 'i' - 106: 176, # 'j' - 107: 177, # 'k' - 108: 77, # 'l' - 109: 72, # 'm' - 110: 178, # 'n' - 111: 69, # 'o' - 112: 67, # 'p' - 113: 179, # 'q' - 114: 78, # 'r' - 115: 73, # 's' - 116: 180, # 't' - 117: 181, # 'u' - 118: 79, # 'v' - 119: 182, # 'w' - 120: 183, # 'x' - 121: 184, # 'y' - 122: 185, # 'z' - 123: 253, # '{' - 124: 253, # '|' - 125: 253, # '}' - 126: 253, # '~' - 127: 253, # '\x7f' - 128: 191, # 'Ђ' - 129: 192, # 'Ѓ' - 130: 193, # '‚' - 131: 194, # 'ѓ' - 132: 195, # '„' - 133: 196, # '…' - 134: 197, # '†' - 135: 198, # '‡' - 136: 199, # '€' - 137: 200, # '‰' - 138: 201, # 'Љ' - 139: 202, # '‹' - 140: 203, # 'Њ' - 141: 204, # 'Ќ' - 142: 205, # 'Ћ' - 143: 206, # 'Џ' - 144: 207, # 'ђ' - 145: 208, # '‘' - 146: 209, # '’' - 147: 210, # '“' - 148: 211, # '”' - 149: 212, # '•' - 150: 213, # '–' - 151: 214, # '—' - 152: 215, # None - 153: 216, # '™' - 154: 217, # 'љ' - 155: 218, # '›' - 156: 219, # 'њ' - 157: 220, # 'ќ' - 158: 221, # 'ћ' - 159: 222, # 'џ' - 160: 223, # '\xa0' - 161: 224, # 'Ў' - 162: 225, # 'ў' - 163: 226, # 'Ј' - 164: 227, # '¤' - 165: 228, # 'Ґ' - 166: 229, # '¦' - 167: 230, # '§' - 168: 231, # 'Ё' - 169: 232, # '©' - 170: 233, # 'Є' - 171: 234, # '«' - 172: 235, # '¬' - 173: 236, # '\xad' - 174: 237, # '®' - 175: 238, # 'Ї' - 176: 239, # '°' - 177: 240, # '±' - 178: 241, # 'І' - 179: 242, # 'і' - 180: 243, # 'ґ' - 181: 244, # 'µ' - 182: 245, # '¶' - 183: 246, # '·' - 184: 68, # 'ё' - 185: 247, # '№' - 186: 248, # 'є' - 187: 249, # '»' - 188: 250, # 'ј' - 189: 251, # 'Ѕ' - 190: 252, # 'ѕ' - 191: 253, # 'ї' - 192: 37, # 'А' - 193: 44, # 'Б' - 194: 33, # 'В' - 195: 46, # 'Г' - 196: 41, # 'Д' - 197: 48, # 'Е' - 198: 56, # 'Ж' - 199: 51, # 'З' - 200: 42, # 'И' - 201: 60, # 'Й' - 202: 36, # 'К' - 203: 49, # 'Л' - 204: 38, # 'М' - 205: 31, # 'Н' - 206: 34, # 'О' - 207: 35, # 'П' - 208: 45, # 'Р' - 209: 32, # 'С' - 210: 40, # 'Т' - 211: 52, # 'У' - 212: 53, # 'Ф' - 213: 55, # 'Х' - 214: 58, # 'Ц' - 215: 50, # 'Ч' - 216: 57, # 'Ш' - 217: 63, # 'Щ' - 218: 70, # 'Ъ' - 219: 62, # 'Ы' - 220: 61, # 'Ь' - 221: 47, # 'Э' - 222: 59, # 'Ю' - 223: 43, # 'Я' - 224: 3, # 'а' - 225: 21, # 'б' - 226: 10, # 'в' - 227: 19, # 'г' - 228: 13, # 'д' - 229: 2, # 'е' - 230: 24, # 'ж' - 231: 20, # 'з' - 232: 4, # 'и' - 233: 23, # 'й' - 234: 11, # 'к' - 235: 8, # 'л' - 236: 12, # 'м' - 237: 5, # 'н' - 238: 1, # 'о' - 239: 15, # 'п' - 240: 9, # 'р' - 241: 7, # 'с' - 242: 6, # 'т' - 243: 14, # 'у' - 244: 39, # 'ф' - 245: 26, # 'х' - 246: 28, # 'ц' - 247: 22, # 'ч' - 248: 25, # 'ш' - 249: 29, # 'щ' - 250: 54, # 'ъ' - 251: 18, # 'ы' - 252: 17, # 'ь' - 253: 30, # 'э' - 254: 27, # 'ю' - 255: 16, # 'я' -} - -WINDOWS_1251_RUSSIAN_MODEL = SingleByteCharSetModel( - charset_name="windows-1251", - language="Russian", - char_to_order_map=WINDOWS_1251_RUSSIAN_CHAR_TO_ORDER, - language_model=RUSSIAN_LANG_MODEL, - typical_positive_ratio=0.976601, - keep_ascii_letters=False, - alphabet="ЁАБВГДЕЖЗИЙКЛМНОПРСТУФХЦЧШЩЪЫЬЭЮЯабвгдежзийклмнопрстуфхцчшщъыьэюяё", -) - -IBM855_RUSSIAN_CHAR_TO_ORDER = { - 0: 255, # '\x00' - 1: 255, # '\x01' - 2: 255, # '\x02' - 3: 255, # '\x03' - 4: 255, # '\x04' - 5: 255, # '\x05' - 6: 255, # '\x06' - 7: 255, # '\x07' - 8: 255, # '\x08' - 9: 255, # '\t' - 10: 254, # '\n' - 11: 255, # '\x0b' - 12: 255, # '\x0c' - 13: 254, # '\r' - 14: 255, # '\x0e' - 15: 255, # '\x0f' - 16: 255, # '\x10' - 17: 255, # '\x11' - 18: 255, # '\x12' - 19: 255, # '\x13' - 20: 255, # '\x14' - 21: 255, # '\x15' - 22: 255, # '\x16' - 23: 255, # '\x17' - 24: 255, # '\x18' - 25: 255, # '\x19' - 26: 255, # '\x1a' - 27: 255, # '\x1b' - 28: 255, # '\x1c' - 29: 255, # '\x1d' - 30: 255, # '\x1e' - 31: 255, # '\x1f' - 32: 253, # ' ' - 33: 253, # '!' - 34: 253, # '"' - 35: 253, # '#' - 36: 253, # '$' - 37: 253, # '%' - 38: 253, # '&' - 39: 253, # "'" - 40: 253, # '(' - 41: 253, # ')' - 42: 253, # '*' - 43: 253, # '+' - 44: 253, # ',' - 45: 253, # '-' - 46: 253, # '.' - 47: 253, # '/' - 48: 252, # '0' - 49: 252, # '1' - 50: 252, # '2' - 51: 252, # '3' - 52: 252, # '4' - 53: 252, # '5' - 54: 252, # '6' - 55: 252, # '7' - 56: 252, # '8' - 57: 252, # '9' - 58: 253, # ':' - 59: 253, # ';' - 60: 253, # '<' - 61: 253, # '=' - 62: 253, # '>' - 63: 253, # '?' - 64: 253, # '@' - 65: 142, # 'A' - 66: 143, # 'B' - 67: 144, # 'C' - 68: 145, # 'D' - 69: 146, # 'E' - 70: 147, # 'F' - 71: 148, # 'G' - 72: 149, # 'H' - 73: 150, # 'I' - 74: 151, # 'J' - 75: 152, # 'K' - 76: 74, # 'L' - 77: 153, # 'M' - 78: 75, # 'N' - 79: 154, # 'O' - 80: 155, # 'P' - 81: 156, # 'Q' - 82: 157, # 'R' - 83: 158, # 'S' - 84: 159, # 'T' - 85: 160, # 'U' - 86: 161, # 'V' - 87: 162, # 'W' - 88: 163, # 'X' - 89: 164, # 'Y' - 90: 165, # 'Z' - 91: 253, # '[' - 92: 253, # '\\' - 93: 253, # ']' - 94: 253, # '^' - 95: 253, # '_' - 96: 253, # '`' - 97: 71, # 'a' - 98: 172, # 'b' - 99: 66, # 'c' - 100: 173, # 'd' - 101: 65, # 'e' - 102: 174, # 'f' - 103: 76, # 'g' - 104: 175, # 'h' - 105: 64, # 'i' - 106: 176, # 'j' - 107: 177, # 'k' - 108: 77, # 'l' - 109: 72, # 'm' - 110: 178, # 'n' - 111: 69, # 'o' - 112: 67, # 'p' - 113: 179, # 'q' - 114: 78, # 'r' - 115: 73, # 's' - 116: 180, # 't' - 117: 181, # 'u' - 118: 79, # 'v' - 119: 182, # 'w' - 120: 183, # 'x' - 121: 184, # 'y' - 122: 185, # 'z' - 123: 253, # '{' - 124: 253, # '|' - 125: 253, # '}' - 126: 253, # '~' - 127: 253, # '\x7f' - 128: 191, # 'ђ' - 129: 192, # 'Ђ' - 130: 193, # 'ѓ' - 131: 194, # 'Ѓ' - 132: 68, # 'ё' - 133: 195, # 'Ё' - 134: 196, # 'є' - 135: 197, # 'Є' - 136: 198, # 'ѕ' - 137: 199, # 'Ѕ' - 138: 200, # 'і' - 139: 201, # 'І' - 140: 202, # 'ї' - 141: 203, # 'Ї' - 142: 204, # 'ј' - 143: 205, # 'Ј' - 144: 206, # 'љ' - 145: 207, # 'Љ' - 146: 208, # 'њ' - 147: 209, # 'Њ' - 148: 210, # 'ћ' - 149: 211, # 'Ћ' - 150: 212, # 'ќ' - 151: 213, # 'Ќ' - 152: 214, # 'ў' - 153: 215, # 'Ў' - 154: 216, # 'џ' - 155: 217, # 'Џ' - 156: 27, # 'ю' - 157: 59, # 'Ю' - 158: 54, # 'ъ' - 159: 70, # 'Ъ' - 160: 3, # 'а' - 161: 37, # 'А' - 162: 21, # 'б' - 163: 44, # 'Б' - 164: 28, # 'ц' - 165: 58, # 'Ц' - 166: 13, # 'д' - 167: 41, # 'Д' - 168: 2, # 'е' - 169: 48, # 'Е' - 170: 39, # 'ф' - 171: 53, # 'Ф' - 172: 19, # 'г' - 173: 46, # 'Г' - 174: 218, # '«' - 175: 219, # '»' - 176: 220, # '░' - 177: 221, # '▒' - 178: 222, # '▓' - 179: 223, # '│' - 180: 224, # '┤' - 181: 26, # 'х' - 182: 55, # 'Х' - 183: 4, # 'и' - 184: 42, # 'И' - 185: 225, # '╣' - 186: 226, # '║' - 187: 227, # '╗' - 188: 228, # '╝' - 189: 23, # 'й' - 190: 60, # 'Й' - 191: 229, # '┐' - 192: 230, # '└' - 193: 231, # '┴' - 194: 232, # '┬' - 195: 233, # '├' - 196: 234, # '─' - 197: 235, # '┼' - 198: 11, # 'к' - 199: 36, # 'К' - 200: 236, # '╚' - 201: 237, # '╔' - 202: 238, # '╩' - 203: 239, # '╦' - 204: 240, # '╠' - 205: 241, # '═' - 206: 242, # '╬' - 207: 243, # '¤' - 208: 8, # 'л' - 209: 49, # 'Л' - 210: 12, # 'м' - 211: 38, # 'М' - 212: 5, # 'н' - 213: 31, # 'Н' - 214: 1, # 'о' - 215: 34, # 'О' - 216: 15, # 'п' - 217: 244, # '┘' - 218: 245, # '┌' - 219: 246, # '█' - 220: 247, # '▄' - 221: 35, # 'П' - 222: 16, # 'я' - 223: 248, # '▀' - 224: 43, # 'Я' - 225: 9, # 'р' - 226: 45, # 'Р' - 227: 7, # 'с' - 228: 32, # 'С' - 229: 6, # 'т' - 230: 40, # 'Т' - 231: 14, # 'у' - 232: 52, # 'У' - 233: 24, # 'ж' - 234: 56, # 'Ж' - 235: 10, # 'в' - 236: 33, # 'В' - 237: 17, # 'ь' - 238: 61, # 'Ь' - 239: 249, # '№' - 240: 250, # '\xad' - 241: 18, # 'ы' - 242: 62, # 'Ы' - 243: 20, # 'з' - 244: 51, # 'З' - 245: 25, # 'ш' - 246: 57, # 'Ш' - 247: 30, # 'э' - 248: 47, # 'Э' - 249: 29, # 'щ' - 250: 63, # 'Щ' - 251: 22, # 'ч' - 252: 50, # 'Ч' - 253: 251, # '§' - 254: 252, # '■' - 255: 255, # '\xa0' -} - -IBM855_RUSSIAN_MODEL = SingleByteCharSetModel( - charset_name="IBM855", - language="Russian", - char_to_order_map=IBM855_RUSSIAN_CHAR_TO_ORDER, - language_model=RUSSIAN_LANG_MODEL, - typical_positive_ratio=0.976601, - keep_ascii_letters=False, - alphabet="ЁАБВГДЕЖЗИЙКЛМНОПРСТУФХЦЧШЩЪЫЬЭЮЯабвгдежзийклмнопрстуфхцчшщъыьэюяё", -) - -KOI8_R_RUSSIAN_CHAR_TO_ORDER = { - 0: 255, # '\x00' - 1: 255, # '\x01' - 2: 255, # '\x02' - 3: 255, # '\x03' - 4: 255, # '\x04' - 5: 255, # '\x05' - 6: 255, # '\x06' - 7: 255, # '\x07' - 8: 255, # '\x08' - 9: 255, # '\t' - 10: 254, # '\n' - 11: 255, # '\x0b' - 12: 255, # '\x0c' - 13: 254, # '\r' - 14: 255, # '\x0e' - 15: 255, # '\x0f' - 16: 255, # '\x10' - 17: 255, # '\x11' - 18: 255, # '\x12' - 19: 255, # '\x13' - 20: 255, # '\x14' - 21: 255, # '\x15' - 22: 255, # '\x16' - 23: 255, # '\x17' - 24: 255, # '\x18' - 25: 255, # '\x19' - 26: 255, # '\x1a' - 27: 255, # '\x1b' - 28: 255, # '\x1c' - 29: 255, # '\x1d' - 30: 255, # '\x1e' - 31: 255, # '\x1f' - 32: 253, # ' ' - 33: 253, # '!' - 34: 253, # '"' - 35: 253, # '#' - 36: 253, # '$' - 37: 253, # '%' - 38: 253, # '&' - 39: 253, # "'" - 40: 253, # '(' - 41: 253, # ')' - 42: 253, # '*' - 43: 253, # '+' - 44: 253, # ',' - 45: 253, # '-' - 46: 253, # '.' - 47: 253, # '/' - 48: 252, # '0' - 49: 252, # '1' - 50: 252, # '2' - 51: 252, # '3' - 52: 252, # '4' - 53: 252, # '5' - 54: 252, # '6' - 55: 252, # '7' - 56: 252, # '8' - 57: 252, # '9' - 58: 253, # ':' - 59: 253, # ';' - 60: 253, # '<' - 61: 253, # '=' - 62: 253, # '>' - 63: 253, # '?' - 64: 253, # '@' - 65: 142, # 'A' - 66: 143, # 'B' - 67: 144, # 'C' - 68: 145, # 'D' - 69: 146, # 'E' - 70: 147, # 'F' - 71: 148, # 'G' - 72: 149, # 'H' - 73: 150, # 'I' - 74: 151, # 'J' - 75: 152, # 'K' - 76: 74, # 'L' - 77: 153, # 'M' - 78: 75, # 'N' - 79: 154, # 'O' - 80: 155, # 'P' - 81: 156, # 'Q' - 82: 157, # 'R' - 83: 158, # 'S' - 84: 159, # 'T' - 85: 160, # 'U' - 86: 161, # 'V' - 87: 162, # 'W' - 88: 163, # 'X' - 89: 164, # 'Y' - 90: 165, # 'Z' - 91: 253, # '[' - 92: 253, # '\\' - 93: 253, # ']' - 94: 253, # '^' - 95: 253, # '_' - 96: 253, # '`' - 97: 71, # 'a' - 98: 172, # 'b' - 99: 66, # 'c' - 100: 173, # 'd' - 101: 65, # 'e' - 102: 174, # 'f' - 103: 76, # 'g' - 104: 175, # 'h' - 105: 64, # 'i' - 106: 176, # 'j' - 107: 177, # 'k' - 108: 77, # 'l' - 109: 72, # 'm' - 110: 178, # 'n' - 111: 69, # 'o' - 112: 67, # 'p' - 113: 179, # 'q' - 114: 78, # 'r' - 115: 73, # 's' - 116: 180, # 't' - 117: 181, # 'u' - 118: 79, # 'v' - 119: 182, # 'w' - 120: 183, # 'x' - 121: 184, # 'y' - 122: 185, # 'z' - 123: 253, # '{' - 124: 253, # '|' - 125: 253, # '}' - 126: 253, # '~' - 127: 253, # '\x7f' - 128: 191, # '─' - 129: 192, # '│' - 130: 193, # '┌' - 131: 194, # '┐' - 132: 195, # '└' - 133: 196, # '┘' - 134: 197, # '├' - 135: 198, # '┤' - 136: 199, # '┬' - 137: 200, # '┴' - 138: 201, # '┼' - 139: 202, # '▀' - 140: 203, # '▄' - 141: 204, # '█' - 142: 205, # '▌' - 143: 206, # '▐' - 144: 207, # '░' - 145: 208, # '▒' - 146: 209, # '▓' - 147: 210, # '⌠' - 148: 211, # '■' - 149: 212, # '∙' - 150: 213, # '√' - 151: 214, # '≈' - 152: 215, # '≤' - 153: 216, # '≥' - 154: 217, # '\xa0' - 155: 218, # '⌡' - 156: 219, # '°' - 157: 220, # '²' - 158: 221, # '·' - 159: 222, # '÷' - 160: 223, # '═' - 161: 224, # '║' - 162: 225, # '╒' - 163: 68, # 'ё' - 164: 226, # '╓' - 165: 227, # '╔' - 166: 228, # '╕' - 167: 229, # '╖' - 168: 230, # '╗' - 169: 231, # '╘' - 170: 232, # '╙' - 171: 233, # '╚' - 172: 234, # '╛' - 173: 235, # '╜' - 174: 236, # '╝' - 175: 237, # '╞' - 176: 238, # '╟' - 177: 239, # '╠' - 178: 240, # '╡' - 179: 241, # 'Ё' - 180: 242, # '╢' - 181: 243, # '╣' - 182: 244, # '╤' - 183: 245, # '╥' - 184: 246, # '╦' - 185: 247, # '╧' - 186: 248, # '╨' - 187: 249, # '╩' - 188: 250, # '╪' - 189: 251, # '╫' - 190: 252, # '╬' - 191: 253, # '©' - 192: 27, # 'ю' - 193: 3, # 'а' - 194: 21, # 'б' - 195: 28, # 'ц' - 196: 13, # 'д' - 197: 2, # 'е' - 198: 39, # 'ф' - 199: 19, # 'г' - 200: 26, # 'х' - 201: 4, # 'и' - 202: 23, # 'й' - 203: 11, # 'к' - 204: 8, # 'л' - 205: 12, # 'м' - 206: 5, # 'н' - 207: 1, # 'о' - 208: 15, # 'п' - 209: 16, # 'я' - 210: 9, # 'р' - 211: 7, # 'с' - 212: 6, # 'т' - 213: 14, # 'у' - 214: 24, # 'ж' - 215: 10, # 'в' - 216: 17, # 'ь' - 217: 18, # 'ы' - 218: 20, # 'з' - 219: 25, # 'ш' - 220: 30, # 'э' - 221: 29, # 'щ' - 222: 22, # 'ч' - 223: 54, # 'ъ' - 224: 59, # 'Ю' - 225: 37, # 'А' - 226: 44, # 'Б' - 227: 58, # 'Ц' - 228: 41, # 'Д' - 229: 48, # 'Е' - 230: 53, # 'Ф' - 231: 46, # 'Г' - 232: 55, # 'Х' - 233: 42, # 'И' - 234: 60, # 'Й' - 235: 36, # 'К' - 236: 49, # 'Л' - 237: 38, # 'М' - 238: 31, # 'Н' - 239: 34, # 'О' - 240: 35, # 'П' - 241: 43, # 'Я' - 242: 45, # 'Р' - 243: 32, # 'С' - 244: 40, # 'Т' - 245: 52, # 'У' - 246: 56, # 'Ж' - 247: 33, # 'В' - 248: 61, # 'Ь' - 249: 62, # 'Ы' - 250: 51, # 'З' - 251: 57, # 'Ш' - 252: 47, # 'Э' - 253: 63, # 'Щ' - 254: 50, # 'Ч' - 255: 70, # 'Ъ' -} - -KOI8_R_RUSSIAN_MODEL = SingleByteCharSetModel( - charset_name="KOI8-R", - language="Russian", - char_to_order_map=KOI8_R_RUSSIAN_CHAR_TO_ORDER, - language_model=RUSSIAN_LANG_MODEL, - typical_positive_ratio=0.976601, - keep_ascii_letters=False, - alphabet="ЁАБВГДЕЖЗИЙКЛМНОПРСТУФХЦЧШЩЪЫЬЭЮЯабвгдежзийклмнопрстуфхцчшщъыьэюяё", -) - -MACCYRILLIC_RUSSIAN_CHAR_TO_ORDER = { - 0: 255, # '\x00' - 1: 255, # '\x01' - 2: 255, # '\x02' - 3: 255, # '\x03' - 4: 255, # '\x04' - 5: 255, # '\x05' - 6: 255, # '\x06' - 7: 255, # '\x07' - 8: 255, # '\x08' - 9: 255, # '\t' - 10: 254, # '\n' - 11: 255, # '\x0b' - 12: 255, # '\x0c' - 13: 254, # '\r' - 14: 255, # '\x0e' - 15: 255, # '\x0f' - 16: 255, # '\x10' - 17: 255, # '\x11' - 18: 255, # '\x12' - 19: 255, # '\x13' - 20: 255, # '\x14' - 21: 255, # '\x15' - 22: 255, # '\x16' - 23: 255, # '\x17' - 24: 255, # '\x18' - 25: 255, # '\x19' - 26: 255, # '\x1a' - 27: 255, # '\x1b' - 28: 255, # '\x1c' - 29: 255, # '\x1d' - 30: 255, # '\x1e' - 31: 255, # '\x1f' - 32: 253, # ' ' - 33: 253, # '!' - 34: 253, # '"' - 35: 253, # '#' - 36: 253, # '$' - 37: 253, # '%' - 38: 253, # '&' - 39: 253, # "'" - 40: 253, # '(' - 41: 253, # ')' - 42: 253, # '*' - 43: 253, # '+' - 44: 253, # ',' - 45: 253, # '-' - 46: 253, # '.' - 47: 253, # '/' - 48: 252, # '0' - 49: 252, # '1' - 50: 252, # '2' - 51: 252, # '3' - 52: 252, # '4' - 53: 252, # '5' - 54: 252, # '6' - 55: 252, # '7' - 56: 252, # '8' - 57: 252, # '9' - 58: 253, # ':' - 59: 253, # ';' - 60: 253, # '<' - 61: 253, # '=' - 62: 253, # '>' - 63: 253, # '?' - 64: 253, # '@' - 65: 142, # 'A' - 66: 143, # 'B' - 67: 144, # 'C' - 68: 145, # 'D' - 69: 146, # 'E' - 70: 147, # 'F' - 71: 148, # 'G' - 72: 149, # 'H' - 73: 150, # 'I' - 74: 151, # 'J' - 75: 152, # 'K' - 76: 74, # 'L' - 77: 153, # 'M' - 78: 75, # 'N' - 79: 154, # 'O' - 80: 155, # 'P' - 81: 156, # 'Q' - 82: 157, # 'R' - 83: 158, # 'S' - 84: 159, # 'T' - 85: 160, # 'U' - 86: 161, # 'V' - 87: 162, # 'W' - 88: 163, # 'X' - 89: 164, # 'Y' - 90: 165, # 'Z' - 91: 253, # '[' - 92: 253, # '\\' - 93: 253, # ']' - 94: 253, # '^' - 95: 253, # '_' - 96: 253, # '`' - 97: 71, # 'a' - 98: 172, # 'b' - 99: 66, # 'c' - 100: 173, # 'd' - 101: 65, # 'e' - 102: 174, # 'f' - 103: 76, # 'g' - 104: 175, # 'h' - 105: 64, # 'i' - 106: 176, # 'j' - 107: 177, # 'k' - 108: 77, # 'l' - 109: 72, # 'm' - 110: 178, # 'n' - 111: 69, # 'o' - 112: 67, # 'p' - 113: 179, # 'q' - 114: 78, # 'r' - 115: 73, # 's' - 116: 180, # 't' - 117: 181, # 'u' - 118: 79, # 'v' - 119: 182, # 'w' - 120: 183, # 'x' - 121: 184, # 'y' - 122: 185, # 'z' - 123: 253, # '{' - 124: 253, # '|' - 125: 253, # '}' - 126: 253, # '~' - 127: 253, # '\x7f' - 128: 37, # 'А' - 129: 44, # 'Б' - 130: 33, # 'В' - 131: 46, # 'Г' - 132: 41, # 'Д' - 133: 48, # 'Е' - 134: 56, # 'Ж' - 135: 51, # 'З' - 136: 42, # 'И' - 137: 60, # 'Й' - 138: 36, # 'К' - 139: 49, # 'Л' - 140: 38, # 'М' - 141: 31, # 'Н' - 142: 34, # 'О' - 143: 35, # 'П' - 144: 45, # 'Р' - 145: 32, # 'С' - 146: 40, # 'Т' - 147: 52, # 'У' - 148: 53, # 'Ф' - 149: 55, # 'Х' - 150: 58, # 'Ц' - 151: 50, # 'Ч' - 152: 57, # 'Ш' - 153: 63, # 'Щ' - 154: 70, # 'Ъ' - 155: 62, # 'Ы' - 156: 61, # 'Ь' - 157: 47, # 'Э' - 158: 59, # 'Ю' - 159: 43, # 'Я' - 160: 191, # '†' - 161: 192, # '°' - 162: 193, # 'Ґ' - 163: 194, # '£' - 164: 195, # '§' - 165: 196, # '•' - 166: 197, # '¶' - 167: 198, # 'І' - 168: 199, # '®' - 169: 200, # '©' - 170: 201, # '™' - 171: 202, # 'Ђ' - 172: 203, # 'ђ' - 173: 204, # '≠' - 174: 205, # 'Ѓ' - 175: 206, # 'ѓ' - 176: 207, # '∞' - 177: 208, # '±' - 178: 209, # '≤' - 179: 210, # '≥' - 180: 211, # 'і' - 181: 212, # 'µ' - 182: 213, # 'ґ' - 183: 214, # 'Ј' - 184: 215, # 'Є' - 185: 216, # 'є' - 186: 217, # 'Ї' - 187: 218, # 'ї' - 188: 219, # 'Љ' - 189: 220, # 'љ' - 190: 221, # 'Њ' - 191: 222, # 'њ' - 192: 223, # 'ј' - 193: 224, # 'Ѕ' - 194: 225, # '¬' - 195: 226, # '√' - 196: 227, # 'ƒ' - 197: 228, # '≈' - 198: 229, # '∆' - 199: 230, # '«' - 200: 231, # '»' - 201: 232, # '…' - 202: 233, # '\xa0' - 203: 234, # 'Ћ' - 204: 235, # 'ћ' - 205: 236, # 'Ќ' - 206: 237, # 'ќ' - 207: 238, # 'ѕ' - 208: 239, # '–' - 209: 240, # '—' - 210: 241, # '“' - 211: 242, # '”' - 212: 243, # '‘' - 213: 244, # '’' - 214: 245, # '÷' - 215: 246, # '„' - 216: 247, # 'Ў' - 217: 248, # 'ў' - 218: 249, # 'Џ' - 219: 250, # 'џ' - 220: 251, # '№' - 221: 252, # 'Ё' - 222: 68, # 'ё' - 223: 16, # 'я' - 224: 3, # 'а' - 225: 21, # 'б' - 226: 10, # 'в' - 227: 19, # 'г' - 228: 13, # 'д' - 229: 2, # 'е' - 230: 24, # 'ж' - 231: 20, # 'з' - 232: 4, # 'и' - 233: 23, # 'й' - 234: 11, # 'к' - 235: 8, # 'л' - 236: 12, # 'м' - 237: 5, # 'н' - 238: 1, # 'о' - 239: 15, # 'п' - 240: 9, # 'р' - 241: 7, # 'с' - 242: 6, # 'т' - 243: 14, # 'у' - 244: 39, # 'ф' - 245: 26, # 'х' - 246: 28, # 'ц' - 247: 22, # 'ч' - 248: 25, # 'ш' - 249: 29, # 'щ' - 250: 54, # 'ъ' - 251: 18, # 'ы' - 252: 17, # 'ь' - 253: 30, # 'э' - 254: 27, # 'ю' - 255: 255, # '€' -} - -MACCYRILLIC_RUSSIAN_MODEL = SingleByteCharSetModel( - charset_name="MacCyrillic", - language="Russian", - char_to_order_map=MACCYRILLIC_RUSSIAN_CHAR_TO_ORDER, - language_model=RUSSIAN_LANG_MODEL, - typical_positive_ratio=0.976601, - keep_ascii_letters=False, - alphabet="ЁАБВГДЕЖЗИЙКЛМНОПРСТУФХЦЧШЩЪЫЬЭЮЯабвгдежзийклмнопрстуфхцчшщъыьэюяё", -) - -ISO_8859_5_RUSSIAN_CHAR_TO_ORDER = { - 0: 255, # '\x00' - 1: 255, # '\x01' - 2: 255, # '\x02' - 3: 255, # '\x03' - 4: 255, # '\x04' - 5: 255, # '\x05' - 6: 255, # '\x06' - 7: 255, # '\x07' - 8: 255, # '\x08' - 9: 255, # '\t' - 10: 254, # '\n' - 11: 255, # '\x0b' - 12: 255, # '\x0c' - 13: 254, # '\r' - 14: 255, # '\x0e' - 15: 255, # '\x0f' - 16: 255, # '\x10' - 17: 255, # '\x11' - 18: 255, # '\x12' - 19: 255, # '\x13' - 20: 255, # '\x14' - 21: 255, # '\x15' - 22: 255, # '\x16' - 23: 255, # '\x17' - 24: 255, # '\x18' - 25: 255, # '\x19' - 26: 255, # '\x1a' - 27: 255, # '\x1b' - 28: 255, # '\x1c' - 29: 255, # '\x1d' - 30: 255, # '\x1e' - 31: 255, # '\x1f' - 32: 253, # ' ' - 33: 253, # '!' - 34: 253, # '"' - 35: 253, # '#' - 36: 253, # '$' - 37: 253, # '%' - 38: 253, # '&' - 39: 253, # "'" - 40: 253, # '(' - 41: 253, # ')' - 42: 253, # '*' - 43: 253, # '+' - 44: 253, # ',' - 45: 253, # '-' - 46: 253, # '.' - 47: 253, # '/' - 48: 252, # '0' - 49: 252, # '1' - 50: 252, # '2' - 51: 252, # '3' - 52: 252, # '4' - 53: 252, # '5' - 54: 252, # '6' - 55: 252, # '7' - 56: 252, # '8' - 57: 252, # '9' - 58: 253, # ':' - 59: 253, # ';' - 60: 253, # '<' - 61: 253, # '=' - 62: 253, # '>' - 63: 253, # '?' - 64: 253, # '@' - 65: 142, # 'A' - 66: 143, # 'B' - 67: 144, # 'C' - 68: 145, # 'D' - 69: 146, # 'E' - 70: 147, # 'F' - 71: 148, # 'G' - 72: 149, # 'H' - 73: 150, # 'I' - 74: 151, # 'J' - 75: 152, # 'K' - 76: 74, # 'L' - 77: 153, # 'M' - 78: 75, # 'N' - 79: 154, # 'O' - 80: 155, # 'P' - 81: 156, # 'Q' - 82: 157, # 'R' - 83: 158, # 'S' - 84: 159, # 'T' - 85: 160, # 'U' - 86: 161, # 'V' - 87: 162, # 'W' - 88: 163, # 'X' - 89: 164, # 'Y' - 90: 165, # 'Z' - 91: 253, # '[' - 92: 253, # '\\' - 93: 253, # ']' - 94: 253, # '^' - 95: 253, # '_' - 96: 253, # '`' - 97: 71, # 'a' - 98: 172, # 'b' - 99: 66, # 'c' - 100: 173, # 'd' - 101: 65, # 'e' - 102: 174, # 'f' - 103: 76, # 'g' - 104: 175, # 'h' - 105: 64, # 'i' - 106: 176, # 'j' - 107: 177, # 'k' - 108: 77, # 'l' - 109: 72, # 'm' - 110: 178, # 'n' - 111: 69, # 'o' - 112: 67, # 'p' - 113: 179, # 'q' - 114: 78, # 'r' - 115: 73, # 's' - 116: 180, # 't' - 117: 181, # 'u' - 118: 79, # 'v' - 119: 182, # 'w' - 120: 183, # 'x' - 121: 184, # 'y' - 122: 185, # 'z' - 123: 253, # '{' - 124: 253, # '|' - 125: 253, # '}' - 126: 253, # '~' - 127: 253, # '\x7f' - 128: 191, # '\x80' - 129: 192, # '\x81' - 130: 193, # '\x82' - 131: 194, # '\x83' - 132: 195, # '\x84' - 133: 196, # '\x85' - 134: 197, # '\x86' - 135: 198, # '\x87' - 136: 199, # '\x88' - 137: 200, # '\x89' - 138: 201, # '\x8a' - 139: 202, # '\x8b' - 140: 203, # '\x8c' - 141: 204, # '\x8d' - 142: 205, # '\x8e' - 143: 206, # '\x8f' - 144: 207, # '\x90' - 145: 208, # '\x91' - 146: 209, # '\x92' - 147: 210, # '\x93' - 148: 211, # '\x94' - 149: 212, # '\x95' - 150: 213, # '\x96' - 151: 214, # '\x97' - 152: 215, # '\x98' - 153: 216, # '\x99' - 154: 217, # '\x9a' - 155: 218, # '\x9b' - 156: 219, # '\x9c' - 157: 220, # '\x9d' - 158: 221, # '\x9e' - 159: 222, # '\x9f' - 160: 223, # '\xa0' - 161: 224, # 'Ё' - 162: 225, # 'Ђ' - 163: 226, # 'Ѓ' - 164: 227, # 'Є' - 165: 228, # 'Ѕ' - 166: 229, # 'І' - 167: 230, # 'Ї' - 168: 231, # 'Ј' - 169: 232, # 'Љ' - 170: 233, # 'Њ' - 171: 234, # 'Ћ' - 172: 235, # 'Ќ' - 173: 236, # '\xad' - 174: 237, # 'Ў' - 175: 238, # 'Џ' - 176: 37, # 'А' - 177: 44, # 'Б' - 178: 33, # 'В' - 179: 46, # 'Г' - 180: 41, # 'Д' - 181: 48, # 'Е' - 182: 56, # 'Ж' - 183: 51, # 'З' - 184: 42, # 'И' - 185: 60, # 'Й' - 186: 36, # 'К' - 187: 49, # 'Л' - 188: 38, # 'М' - 189: 31, # 'Н' - 190: 34, # 'О' - 191: 35, # 'П' - 192: 45, # 'Р' - 193: 32, # 'С' - 194: 40, # 'Т' - 195: 52, # 'У' - 196: 53, # 'Ф' - 197: 55, # 'Х' - 198: 58, # 'Ц' - 199: 50, # 'Ч' - 200: 57, # 'Ш' - 201: 63, # 'Щ' - 202: 70, # 'Ъ' - 203: 62, # 'Ы' - 204: 61, # 'Ь' - 205: 47, # 'Э' - 206: 59, # 'Ю' - 207: 43, # 'Я' - 208: 3, # 'а' - 209: 21, # 'б' - 210: 10, # 'в' - 211: 19, # 'г' - 212: 13, # 'д' - 213: 2, # 'е' - 214: 24, # 'ж' - 215: 20, # 'з' - 216: 4, # 'и' - 217: 23, # 'й' - 218: 11, # 'к' - 219: 8, # 'л' - 220: 12, # 'м' - 221: 5, # 'н' - 222: 1, # 'о' - 223: 15, # 'п' - 224: 9, # 'р' - 225: 7, # 'с' - 226: 6, # 'т' - 227: 14, # 'у' - 228: 39, # 'ф' - 229: 26, # 'х' - 230: 28, # 'ц' - 231: 22, # 'ч' - 232: 25, # 'ш' - 233: 29, # 'щ' - 234: 54, # 'ъ' - 235: 18, # 'ы' - 236: 17, # 'ь' - 237: 30, # 'э' - 238: 27, # 'ю' - 239: 16, # 'я' - 240: 239, # '№' - 241: 68, # 'ё' - 242: 240, # 'ђ' - 243: 241, # 'ѓ' - 244: 242, # 'є' - 245: 243, # 'ѕ' - 246: 244, # 'і' - 247: 245, # 'ї' - 248: 246, # 'ј' - 249: 247, # 'љ' - 250: 248, # 'њ' - 251: 249, # 'ћ' - 252: 250, # 'ќ' - 253: 251, # '§' - 254: 252, # 'ў' - 255: 255, # 'џ' -} - -ISO_8859_5_RUSSIAN_MODEL = SingleByteCharSetModel( - charset_name="ISO-8859-5", - language="Russian", - char_to_order_map=ISO_8859_5_RUSSIAN_CHAR_TO_ORDER, - language_model=RUSSIAN_LANG_MODEL, - typical_positive_ratio=0.976601, - keep_ascii_letters=False, - alphabet="ЁАБВГДЕЖЗИЙКЛМНОПРСТУФХЦЧШЩЪЫЬЭЮЯабвгдежзийклмнопрстуфхцчшщъыьэюяё", -) diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/langthaimodel.py b/venv/lib/python3.11/site-packages/pip/_vendor/chardet/langthaimodel.py deleted file mode 100644 index 489cad9..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/langthaimodel.py +++ /dev/null @@ -1,4380 +0,0 @@ -from pip._vendor.chardet.sbcharsetprober import SingleByteCharSetModel - -# 3: Positive -# 2: Likely -# 1: Unlikely -# 0: Negative - -THAI_LANG_MODEL = { - 5: { # 'ก' - 5: 2, # 'ก' - 30: 2, # 'ข' - 24: 2, # 'ค' - 8: 2, # 'ง' - 26: 2, # 'จ' - 52: 0, # 'ฉ' - 34: 1, # 'ช' - 51: 1, # 'ซ' - 47: 0, # 'ญ' - 58: 3, # 'ฎ' - 57: 2, # 'ฏ' - 49: 0, # 'ฐ' - 53: 0, # 'ฑ' - 55: 0, # 'ฒ' - 43: 2, # 'ณ' - 20: 2, # 'ด' - 19: 3, # 'ต' - 44: 0, # 'ถ' - 14: 2, # 'ท' - 48: 0, # 'ธ' - 3: 2, # 'น' - 17: 1, # 'บ' - 25: 2, # 'ป' - 39: 1, # 'ผ' - 62: 1, # 'ฝ' - 31: 1, # 'พ' - 54: 0, # 'ฟ' - 45: 1, # 'ภ' - 9: 2, # 'ม' - 16: 1, # 'ย' - 2: 3, # 'ร' - 61: 2, # 'ฤ' - 15: 3, # 'ล' - 12: 3, # 'ว' - 42: 2, # 'ศ' - 46: 3, # 'ษ' - 18: 2, # 'ส' - 21: 2, # 'ห' - 4: 3, # 'อ' - 63: 1, # 'ฯ' - 22: 2, # 'ะ' - 10: 3, # 'ั' - 1: 3, # 'า' - 36: 3, # 'ำ' - 23: 3, # 'ิ' - 13: 3, # 'ี' - 40: 0, # 'ึ' - 27: 2, # 'ื' - 32: 2, # 'ุ' - 35: 1, # 'ู' - 11: 2, # 'เ' - 28: 2, # 'แ' - 41: 1, # 'โ' - 29: 1, # 'ใ' - 33: 2, # 'ไ' - 50: 1, # 'ๆ' - 37: 3, # '็' - 6: 3, # '่' - 7: 3, # '้' - 38: 2, # '์' - 56: 0, # '๑' - 59: 0, # '๒' - 60: 0, # '๕' - }, - 30: { # 'ข' - 5: 1, # 'ก' - 30: 0, # 'ข' - 24: 1, # 'ค' - 8: 1, # 'ง' - 26: 1, # 'จ' - 52: 0, # 'ฉ' - 34: 0, # 'ช' - 51: 0, # 'ซ' - 47: 0, # 'ญ' - 58: 0, # 'ฎ' - 57: 0, # 'ฏ' - 49: 0, # 'ฐ' - 53: 0, # 'ฑ' - 55: 0, # 'ฒ' - 43: 2, # 'ณ' - 20: 0, # 'ด' - 19: 2, # 'ต' - 44: 0, # 'ถ' - 14: 1, # 'ท' - 48: 0, # 'ธ' - 3: 2, # 'น' - 17: 1, # 'บ' - 25: 1, # 'ป' - 39: 0, # 'ผ' - 62: 0, # 'ฝ' - 31: 0, # 'พ' - 54: 0, # 'ฟ' - 45: 0, # 'ภ' - 9: 0, # 'ม' - 16: 2, # 'ย' - 2: 1, # 'ร' - 61: 0, # 'ฤ' - 15: 0, # 'ล' - 12: 2, # 'ว' - 42: 0, # 'ศ' - 46: 0, # 'ษ' - 18: 1, # 'ส' - 21: 1, # 'ห' - 4: 3, # 'อ' - 63: 0, # 'ฯ' - 22: 0, # 'ะ' - 10: 3, # 'ั' - 1: 3, # 'า' - 36: 0, # 'ำ' - 23: 0, # 'ิ' - 13: 2, # 'ี' - 40: 3, # 'ึ' - 27: 1, # 'ื' - 32: 1, # 'ุ' - 35: 0, # 'ู' - 11: 0, # 'เ' - 28: 0, # 'แ' - 41: 0, # 'โ' - 29: 1, # 'ใ' - 33: 0, # 'ไ' - 50: 0, # 'ๆ' - 37: 1, # '็' - 6: 2, # '่' - 7: 3, # '้' - 38: 1, # '์' - 56: 0, # '๑' - 59: 0, # '๒' - 60: 0, # '๕' - }, - 24: { # 'ค' - 5: 0, # 'ก' - 30: 0, # 'ข' - 24: 2, # 'ค' - 8: 2, # 'ง' - 26: 0, # 'จ' - 52: 0, # 'ฉ' - 34: 0, # 'ช' - 51: 0, # 'ซ' - 47: 0, # 'ญ' - 58: 0, # 'ฎ' - 57: 0, # 'ฏ' - 49: 0, # 'ฐ' - 53: 0, # 'ฑ' - 55: 0, # 'ฒ' - 43: 2, # 'ณ' - 20: 2, # 'ด' - 19: 2, # 'ต' - 44: 0, # 'ถ' - 14: 1, # 'ท' - 48: 0, # 'ธ' - 3: 3, # 'น' - 17: 0, # 'บ' - 25: 1, # 'ป' - 39: 0, # 'ผ' - 62: 0, # 'ฝ' - 31: 0, # 'พ' - 54: 0, # 'ฟ' - 45: 0, # 'ภ' - 9: 2, # 'ม' - 16: 2, # 'ย' - 2: 3, # 'ร' - 61: 0, # 'ฤ' - 15: 3, # 'ล' - 12: 3, # 'ว' - 42: 0, # 'ศ' - 46: 0, # 'ษ' - 18: 1, # 'ส' - 21: 0, # 'ห' - 4: 2, # 'อ' - 63: 0, # 'ฯ' - 22: 2, # 'ะ' - 10: 3, # 'ั' - 1: 2, # 'า' - 36: 3, # 'ำ' - 23: 3, # 'ิ' - 13: 2, # 'ี' - 40: 0, # 'ึ' - 27: 3, # 'ื' - 32: 3, # 'ุ' - 35: 2, # 'ู' - 11: 1, # 'เ' - 28: 0, # 'แ' - 41: 3, # 'โ' - 29: 0, # 'ใ' - 33: 0, # 'ไ' - 50: 0, # 'ๆ' - 37: 1, # '็' - 6: 3, # '่' - 7: 3, # '้' - 38: 3, # '์' - 56: 0, # '๑' - 59: 0, # '๒' - 60: 0, # '๕' - }, - 8: { # 'ง' - 5: 3, # 'ก' - 30: 2, # 'ข' - 24: 3, # 'ค' - 8: 2, # 'ง' - 26: 2, # 'จ' - 52: 1, # 'ฉ' - 34: 2, # 'ช' - 51: 1, # 'ซ' - 47: 0, # 'ญ' - 58: 0, # 'ฎ' - 57: 0, # 'ฏ' - 49: 0, # 'ฐ' - 53: 0, # 'ฑ' - 55: 0, # 'ฒ' - 43: 0, # 'ณ' - 20: 2, # 'ด' - 19: 2, # 'ต' - 44: 1, # 'ถ' - 14: 3, # 'ท' - 48: 1, # 'ธ' - 3: 3, # 'น' - 17: 2, # 'บ' - 25: 2, # 'ป' - 39: 2, # 'ผ' - 62: 1, # 'ฝ' - 31: 2, # 'พ' - 54: 0, # 'ฟ' - 45: 1, # 'ภ' - 9: 2, # 'ม' - 16: 1, # 'ย' - 2: 2, # 'ร' - 61: 0, # 'ฤ' - 15: 2, # 'ล' - 12: 2, # 'ว' - 42: 2, # 'ศ' - 46: 1, # 'ษ' - 18: 3, # 'ส' - 21: 3, # 'ห' - 4: 2, # 'อ' - 63: 0, # 'ฯ' - 22: 0, # 'ะ' - 10: 1, # 'ั' - 1: 3, # 'า' - 36: 0, # 'ำ' - 23: 2, # 'ิ' - 13: 1, # 'ี' - 40: 0, # 'ึ' - 27: 1, # 'ื' - 32: 1, # 'ุ' - 35: 0, # 'ู' - 11: 3, # 'เ' - 28: 2, # 'แ' - 41: 1, # 'โ' - 29: 2, # 'ใ' - 33: 2, # 'ไ' - 50: 3, # 'ๆ' - 37: 0, # '็' - 6: 2, # '่' - 7: 0, # '้' - 38: 0, # '์' - 56: 0, # '๑' - 59: 0, # '๒' - 60: 0, # '๕' - }, - 26: { # 'จ' - 5: 2, # 'ก' - 30: 1, # 'ข' - 24: 0, # 'ค' - 8: 2, # 'ง' - 26: 3, # 'จ' - 52: 0, # 'ฉ' - 34: 0, # 'ช' - 51: 0, # 'ซ' - 47: 0, # 'ญ' - 58: 0, # 'ฎ' - 57: 0, # 'ฏ' - 49: 0, # 'ฐ' - 53: 0, # 'ฑ' - 55: 0, # 'ฒ' - 43: 0, # 'ณ' - 20: 2, # 'ด' - 19: 1, # 'ต' - 44: 1, # 'ถ' - 14: 2, # 'ท' - 48: 0, # 'ธ' - 3: 3, # 'น' - 17: 1, # 'บ' - 25: 0, # 'ป' - 39: 0, # 'ผ' - 62: 0, # 'ฝ' - 31: 1, # 'พ' - 54: 0, # 'ฟ' - 45: 0, # 'ภ' - 9: 1, # 'ม' - 16: 1, # 'ย' - 2: 3, # 'ร' - 61: 0, # 'ฤ' - 15: 0, # 'ล' - 12: 1, # 'ว' - 42: 0, # 'ศ' - 46: 0, # 'ษ' - 18: 2, # 'ส' - 21: 1, # 'ห' - 4: 2, # 'อ' - 63: 0, # 'ฯ' - 22: 3, # 'ะ' - 10: 3, # 'ั' - 1: 3, # 'า' - 36: 3, # 'ำ' - 23: 2, # 'ิ' - 13: 1, # 'ี' - 40: 3, # 'ึ' - 27: 1, # 'ื' - 32: 3, # 'ุ' - 35: 2, # 'ู' - 11: 1, # 'เ' - 28: 1, # 'แ' - 41: 0, # 'โ' - 29: 1, # 'ใ' - 33: 1, # 'ไ' - 50: 0, # 'ๆ' - 37: 0, # '็' - 6: 2, # '่' - 7: 2, # '้' - 38: 0, # '์' - 56: 0, # '๑' - 59: 0, # '๒' - 60: 0, # '๕' - }, - 52: { # 'ฉ' - 5: 0, # 'ก' - 30: 0, # 'ข' - 24: 0, # 'ค' - 8: 0, # 'ง' - 26: 0, # 'จ' - 52: 0, # 'ฉ' - 34: 0, # 'ช' - 51: 0, # 'ซ' - 47: 0, # 'ญ' - 58: 0, # 'ฎ' - 57: 0, # 'ฏ' - 49: 0, # 'ฐ' - 53: 0, # 'ฑ' - 55: 0, # 'ฒ' - 43: 0, # 'ณ' - 20: 0, # 'ด' - 19: 0, # 'ต' - 44: 0, # 'ถ' - 14: 0, # 'ท' - 48: 0, # 'ธ' - 3: 0, # 'น' - 17: 3, # 'บ' - 25: 0, # 'ป' - 39: 0, # 'ผ' - 62: 0, # 'ฝ' - 31: 3, # 'พ' - 54: 0, # 'ฟ' - 45: 0, # 'ภ' - 9: 1, # 'ม' - 16: 1, # 'ย' - 2: 0, # 'ร' - 61: 0, # 'ฤ' - 15: 2, # 'ล' - 12: 1, # 'ว' - 42: 0, # 'ศ' - 46: 0, # 'ษ' - 18: 0, # 'ส' - 21: 0, # 'ห' - 4: 0, # 'อ' - 63: 0, # 'ฯ' - 22: 1, # 'ะ' - 10: 1, # 'ั' - 1: 1, # 'า' - 36: 0, # 'ำ' - 23: 1, # 'ิ' - 13: 1, # 'ี' - 40: 0, # 'ึ' - 27: 0, # 'ื' - 32: 1, # 'ุ' - 35: 0, # 'ู' - 11: 0, # 'เ' - 28: 0, # 'แ' - 41: 0, # 'โ' - 29: 0, # 'ใ' - 33: 0, # 'ไ' - 50: 0, # 'ๆ' - 37: 0, # '็' - 6: 0, # '่' - 7: 0, # '้' - 38: 0, # '์' - 56: 0, # '๑' - 59: 0, # '๒' - 60: 0, # '๕' - }, - 34: { # 'ช' - 5: 1, # 'ก' - 30: 0, # 'ข' - 24: 0, # 'ค' - 8: 1, # 'ง' - 26: 0, # 'จ' - 52: 0, # 'ฉ' - 34: 0, # 'ช' - 51: 0, # 'ซ' - 47: 1, # 'ญ' - 58: 0, # 'ฎ' - 57: 0, # 'ฏ' - 49: 0, # 'ฐ' - 53: 0, # 'ฑ' - 55: 0, # 'ฒ' - 43: 0, # 'ณ' - 20: 0, # 'ด' - 19: 0, # 'ต' - 44: 0, # 'ถ' - 14: 1, # 'ท' - 48: 0, # 'ธ' - 3: 3, # 'น' - 17: 2, # 'บ' - 25: 0, # 'ป' - 39: 0, # 'ผ' - 62: 0, # 'ฝ' - 31: 0, # 'พ' - 54: 0, # 'ฟ' - 45: 0, # 'ภ' - 9: 2, # 'ม' - 16: 1, # 'ย' - 2: 1, # 'ร' - 61: 0, # 'ฤ' - 15: 0, # 'ล' - 12: 1, # 'ว' - 42: 0, # 'ศ' - 46: 0, # 'ษ' - 18: 0, # 'ส' - 21: 0, # 'ห' - 4: 2, # 'อ' - 63: 0, # 'ฯ' - 22: 0, # 'ะ' - 10: 2, # 'ั' - 1: 3, # 'า' - 36: 1, # 'ำ' - 23: 3, # 'ิ' - 13: 2, # 'ี' - 40: 0, # 'ึ' - 27: 3, # 'ื' - 32: 3, # 'ุ' - 35: 1, # 'ู' - 11: 0, # 'เ' - 28: 0, # 'แ' - 41: 0, # 'โ' - 29: 0, # 'ใ' - 33: 0, # 'ไ' - 50: 0, # 'ๆ' - 37: 1, # '็' - 6: 3, # '่' - 7: 3, # '้' - 38: 0, # '์' - 56: 0, # '๑' - 59: 0, # '๒' - 60: 0, # '๕' - }, - 51: { # 'ซ' - 5: 0, # 'ก' - 30: 0, # 'ข' - 24: 0, # 'ค' - 8: 0, # 'ง' - 26: 0, # 'จ' - 52: 0, # 'ฉ' - 34: 0, # 'ช' - 51: 0, # 'ซ' - 47: 0, # 'ญ' - 58: 0, # 'ฎ' - 57: 0, # 'ฏ' - 49: 0, # 'ฐ' - 53: 0, # 'ฑ' - 55: 0, # 'ฒ' - 43: 0, # 'ณ' - 20: 0, # 'ด' - 19: 0, # 'ต' - 44: 0, # 'ถ' - 14: 0, # 'ท' - 48: 0, # 'ธ' - 3: 1, # 'น' - 17: 0, # 'บ' - 25: 0, # 'ป' - 39: 0, # 'ผ' - 62: 0, # 'ฝ' - 31: 0, # 'พ' - 54: 0, # 'ฟ' - 45: 0, # 'ภ' - 9: 0, # 'ม' - 16: 0, # 'ย' - 2: 0, # 'ร' - 61: 0, # 'ฤ' - 15: 1, # 'ล' - 12: 0, # 'ว' - 42: 0, # 'ศ' - 46: 0, # 'ษ' - 18: 1, # 'ส' - 21: 0, # 'ห' - 4: 2, # 'อ' - 63: 0, # 'ฯ' - 22: 0, # 'ะ' - 10: 1, # 'ั' - 1: 1, # 'า' - 36: 0, # 'ำ' - 23: 1, # 'ิ' - 13: 2, # 'ี' - 40: 3, # 'ึ' - 27: 2, # 'ื' - 32: 1, # 'ุ' - 35: 1, # 'ู' - 11: 1, # 'เ' - 28: 0, # 'แ' - 41: 0, # 'โ' - 29: 0, # 'ใ' - 33: 0, # 'ไ' - 50: 0, # 'ๆ' - 37: 1, # '็' - 6: 1, # '่' - 7: 2, # '้' - 38: 1, # '์' - 56: 0, # '๑' - 59: 0, # '๒' - 60: 0, # '๕' - }, - 47: { # 'ญ' - 5: 1, # 'ก' - 30: 1, # 'ข' - 24: 0, # 'ค' - 8: 0, # 'ง' - 26: 0, # 'จ' - 52: 0, # 'ฉ' - 34: 1, # 'ช' - 51: 0, # 'ซ' - 47: 3, # 'ญ' - 58: 0, # 'ฎ' - 57: 0, # 'ฏ' - 49: 0, # 'ฐ' - 53: 0, # 'ฑ' - 55: 0, # 'ฒ' - 43: 0, # 'ณ' - 20: 0, # 'ด' - 19: 0, # 'ต' - 44: 0, # 'ถ' - 14: 1, # 'ท' - 48: 0, # 'ธ' - 3: 0, # 'น' - 17: 1, # 'บ' - 25: 1, # 'ป' - 39: 0, # 'ผ' - 62: 0, # 'ฝ' - 31: 0, # 'พ' - 54: 0, # 'ฟ' - 45: 0, # 'ภ' - 9: 1, # 'ม' - 16: 0, # 'ย' - 2: 0, # 'ร' - 61: 0, # 'ฤ' - 15: 1, # 'ล' - 12: 0, # 'ว' - 42: 0, # 'ศ' - 46: 0, # 'ษ' - 18: 1, # 'ส' - 21: 2, # 'ห' - 4: 1, # 'อ' - 63: 0, # 'ฯ' - 22: 1, # 'ะ' - 10: 2, # 'ั' - 1: 3, # 'า' - 36: 0, # 'ำ' - 23: 1, # 'ิ' - 13: 1, # 'ี' - 40: 0, # 'ึ' - 27: 0, # 'ื' - 32: 0, # 'ุ' - 35: 0, # 'ู' - 11: 1, # 'เ' - 28: 1, # 'แ' - 41: 0, # 'โ' - 29: 1, # 'ใ' - 33: 0, # 'ไ' - 50: 1, # 'ๆ' - 37: 0, # '็' - 6: 2, # '่' - 7: 0, # '้' - 38: 0, # '์' - 56: 0, # '๑' - 59: 0, # '๒' - 60: 0, # '๕' - }, - 58: { # 'ฎ' - 5: 2, # 'ก' - 30: 0, # 'ข' - 24: 0, # 'ค' - 8: 0, # 'ง' - 26: 0, # 'จ' - 52: 0, # 'ฉ' - 34: 0, # 'ช' - 51: 0, # 'ซ' - 47: 0, # 'ญ' - 58: 0, # 'ฎ' - 57: 0, # 'ฏ' - 49: 0, # 'ฐ' - 53: 0, # 'ฑ' - 55: 0, # 'ฒ' - 43: 0, # 'ณ' - 20: 0, # 'ด' - 19: 0, # 'ต' - 44: 0, # 'ถ' - 14: 0, # 'ท' - 48: 0, # 'ธ' - 3: 0, # 'น' - 17: 0, # 'บ' - 25: 0, # 'ป' - 39: 0, # 'ผ' - 62: 0, # 'ฝ' - 31: 0, # 'พ' - 54: 0, # 'ฟ' - 45: 0, # 'ภ' - 9: 0, # 'ม' - 16: 0, # 'ย' - 2: 0, # 'ร' - 61: 0, # 'ฤ' - 15: 0, # 'ล' - 12: 0, # 'ว' - 42: 0, # 'ศ' - 46: 0, # 'ษ' - 18: 0, # 'ส' - 21: 1, # 'ห' - 4: 0, # 'อ' - 63: 0, # 'ฯ' - 22: 0, # 'ะ' - 10: 0, # 'ั' - 1: 0, # 'า' - 36: 0, # 'ำ' - 23: 1, # 'ิ' - 13: 2, # 'ี' - 40: 0, # 'ึ' - 27: 0, # 'ื' - 32: 0, # 'ุ' - 35: 0, # 'ู' - 11: 0, # 'เ' - 28: 0, # 'แ' - 41: 0, # 'โ' - 29: 0, # 'ใ' - 33: 0, # 'ไ' - 50: 0, # 'ๆ' - 37: 0, # '็' - 6: 0, # '่' - 7: 0, # '้' - 38: 0, # '์' - 56: 0, # '๑' - 59: 0, # '๒' - 60: 0, # '๕' - }, - 57: { # 'ฏ' - 5: 0, # 'ก' - 30: 0, # 'ข' - 24: 0, # 'ค' - 8: 0, # 'ง' - 26: 0, # 'จ' - 52: 0, # 'ฉ' - 34: 0, # 'ช' - 51: 0, # 'ซ' - 47: 0, # 'ญ' - 58: 0, # 'ฎ' - 57: 0, # 'ฏ' - 49: 0, # 'ฐ' - 53: 0, # 'ฑ' - 55: 0, # 'ฒ' - 43: 0, # 'ณ' - 20: 0, # 'ด' - 19: 0, # 'ต' - 44: 0, # 'ถ' - 14: 0, # 'ท' - 48: 0, # 'ธ' - 3: 0, # 'น' - 17: 0, # 'บ' - 25: 0, # 'ป' - 39: 0, # 'ผ' - 62: 0, # 'ฝ' - 31: 0, # 'พ' - 54: 0, # 'ฟ' - 45: 0, # 'ภ' - 9: 0, # 'ม' - 16: 0, # 'ย' - 2: 0, # 'ร' - 61: 0, # 'ฤ' - 15: 0, # 'ล' - 12: 0, # 'ว' - 42: 0, # 'ศ' - 46: 0, # 'ษ' - 18: 0, # 'ส' - 21: 0, # 'ห' - 4: 0, # 'อ' - 63: 0, # 'ฯ' - 22: 0, # 'ะ' - 10: 0, # 'ั' - 1: 0, # 'า' - 36: 0, # 'ำ' - 23: 3, # 'ิ' - 13: 1, # 'ี' - 40: 0, # 'ึ' - 27: 0, # 'ื' - 32: 0, # 'ุ' - 35: 0, # 'ู' - 11: 0, # 'เ' - 28: 0, # 'แ' - 41: 0, # 'โ' - 29: 0, # 'ใ' - 33: 0, # 'ไ' - 50: 0, # 'ๆ' - 37: 0, # '็' - 6: 0, # '่' - 7: 0, # '้' - 38: 0, # '์' - 56: 0, # '๑' - 59: 0, # '๒' - 60: 0, # '๕' - }, - 49: { # 'ฐ' - 5: 1, # 'ก' - 30: 0, # 'ข' - 24: 0, # 'ค' - 8: 0, # 'ง' - 26: 0, # 'จ' - 52: 0, # 'ฉ' - 34: 0, # 'ช' - 51: 0, # 'ซ' - 47: 0, # 'ญ' - 58: 0, # 'ฎ' - 57: 0, # 'ฏ' - 49: 0, # 'ฐ' - 53: 0, # 'ฑ' - 55: 0, # 'ฒ' - 43: 0, # 'ณ' - 20: 0, # 'ด' - 19: 0, # 'ต' - 44: 0, # 'ถ' - 14: 0, # 'ท' - 48: 0, # 'ธ' - 3: 0, # 'น' - 17: 2, # 'บ' - 25: 0, # 'ป' - 39: 0, # 'ผ' - 62: 0, # 'ฝ' - 31: 0, # 'พ' - 54: 0, # 'ฟ' - 45: 0, # 'ภ' - 9: 2, # 'ม' - 16: 0, # 'ย' - 2: 0, # 'ร' - 61: 0, # 'ฤ' - 15: 0, # 'ล' - 12: 0, # 'ว' - 42: 1, # 'ศ' - 46: 0, # 'ษ' - 18: 0, # 'ส' - 21: 0, # 'ห' - 4: 1, # 'อ' - 63: 0, # 'ฯ' - 22: 0, # 'ะ' - 10: 0, # 'ั' - 1: 3, # 'า' - 36: 0, # 'ำ' - 23: 0, # 'ิ' - 13: 0, # 'ี' - 40: 0, # 'ึ' - 27: 0, # 'ื' - 32: 0, # 'ุ' - 35: 0, # 'ู' - 11: 0, # 'เ' - 28: 0, # 'แ' - 41: 0, # 'โ' - 29: 0, # 'ใ' - 33: 0, # 'ไ' - 50: 0, # 'ๆ' - 37: 0, # '็' - 6: 0, # '่' - 7: 0, # '้' - 38: 1, # '์' - 56: 0, # '๑' - 59: 0, # '๒' - 60: 0, # '๕' - }, - 53: { # 'ฑ' - 5: 0, # 'ก' - 30: 0, # 'ข' - 24: 0, # 'ค' - 8: 0, # 'ง' - 26: 0, # 'จ' - 52: 0, # 'ฉ' - 34: 0, # 'ช' - 51: 0, # 'ซ' - 47: 0, # 'ญ' - 58: 0, # 'ฎ' - 57: 0, # 'ฏ' - 49: 0, # 'ฐ' - 53: 0, # 'ฑ' - 55: 0, # 'ฒ' - 43: 0, # 'ณ' - 20: 0, # 'ด' - 19: 0, # 'ต' - 44: 0, # 'ถ' - 14: 0, # 'ท' - 48: 0, # 'ธ' - 3: 0, # 'น' - 17: 0, # 'บ' - 25: 0, # 'ป' - 39: 0, # 'ผ' - 62: 0, # 'ฝ' - 31: 0, # 'พ' - 54: 0, # 'ฟ' - 45: 0, # 'ภ' - 9: 0, # 'ม' - 16: 0, # 'ย' - 2: 0, # 'ร' - 61: 0, # 'ฤ' - 15: 0, # 'ล' - 12: 0, # 'ว' - 42: 0, # 'ศ' - 46: 0, # 'ษ' - 18: 0, # 'ส' - 21: 0, # 'ห' - 4: 0, # 'อ' - 63: 0, # 'ฯ' - 22: 0, # 'ะ' - 10: 0, # 'ั' - 1: 0, # 'า' - 36: 0, # 'ำ' - 23: 2, # 'ิ' - 13: 0, # 'ี' - 40: 0, # 'ึ' - 27: 0, # 'ื' - 32: 0, # 'ุ' - 35: 0, # 'ู' - 11: 0, # 'เ' - 28: 0, # 'แ' - 41: 0, # 'โ' - 29: 0, # 'ใ' - 33: 0, # 'ไ' - 50: 0, # 'ๆ' - 37: 0, # '็' - 6: 0, # '่' - 7: 0, # '้' - 38: 3, # '์' - 56: 0, # '๑' - 59: 0, # '๒' - 60: 0, # '๕' - }, - 55: { # 'ฒ' - 5: 0, # 'ก' - 30: 0, # 'ข' - 24: 0, # 'ค' - 8: 0, # 'ง' - 26: 0, # 'จ' - 52: 0, # 'ฉ' - 34: 0, # 'ช' - 51: 0, # 'ซ' - 47: 0, # 'ญ' - 58: 0, # 'ฎ' - 57: 0, # 'ฏ' - 49: 0, # 'ฐ' - 53: 0, # 'ฑ' - 55: 0, # 'ฒ' - 43: 0, # 'ณ' - 20: 0, # 'ด' - 19: 0, # 'ต' - 44: 0, # 'ถ' - 14: 0, # 'ท' - 48: 0, # 'ธ' - 3: 3, # 'น' - 17: 0, # 'บ' - 25: 0, # 'ป' - 39: 0, # 'ผ' - 62: 0, # 'ฝ' - 31: 1, # 'พ' - 54: 0, # 'ฟ' - 45: 0, # 'ภ' - 9: 0, # 'ม' - 16: 0, # 'ย' - 2: 0, # 'ร' - 61: 0, # 'ฤ' - 15: 0, # 'ล' - 12: 0, # 'ว' - 42: 0, # 'ศ' - 46: 0, # 'ษ' - 18: 0, # 'ส' - 21: 0, # 'ห' - 4: 0, # 'อ' - 63: 0, # 'ฯ' - 22: 0, # 'ะ' - 10: 0, # 'ั' - 1: 0, # 'า' - 36: 0, # 'ำ' - 23: 1, # 'ิ' - 13: 0, # 'ี' - 40: 0, # 'ึ' - 27: 0, # 'ื' - 32: 0, # 'ุ' - 35: 0, # 'ู' - 11: 0, # 'เ' - 28: 0, # 'แ' - 41: 0, # 'โ' - 29: 0, # 'ใ' - 33: 0, # 'ไ' - 50: 0, # 'ๆ' - 37: 0, # '็' - 6: 0, # '่' - 7: 0, # '้' - 38: 0, # '์' - 56: 0, # '๑' - 59: 0, # '๒' - 60: 0, # '๕' - }, - 43: { # 'ณ' - 5: 1, # 'ก' - 30: 0, # 'ข' - 24: 0, # 'ค' - 8: 0, # 'ง' - 26: 0, # 'จ' - 52: 0, # 'ฉ' - 34: 0, # 'ช' - 51: 0, # 'ซ' - 47: 0, # 'ญ' - 58: 0, # 'ฎ' - 57: 0, # 'ฏ' - 49: 0, # 'ฐ' - 53: 3, # 'ฑ' - 55: 0, # 'ฒ' - 43: 0, # 'ณ' - 20: 0, # 'ด' - 19: 0, # 'ต' - 44: 0, # 'ถ' - 14: 0, # 'ท' - 48: 0, # 'ธ' - 3: 0, # 'น' - 17: 0, # 'บ' - 25: 0, # 'ป' - 39: 0, # 'ผ' - 62: 0, # 'ฝ' - 31: 0, # 'พ' - 54: 0, # 'ฟ' - 45: 3, # 'ภ' - 9: 0, # 'ม' - 16: 0, # 'ย' - 2: 1, # 'ร' - 61: 0, # 'ฤ' - 15: 0, # 'ล' - 12: 1, # 'ว' - 42: 0, # 'ศ' - 46: 0, # 'ษ' - 18: 1, # 'ส' - 21: 1, # 'ห' - 4: 0, # 'อ' - 63: 0, # 'ฯ' - 22: 3, # 'ะ' - 10: 0, # 'ั' - 1: 3, # 'า' - 36: 0, # 'ำ' - 23: 1, # 'ิ' - 13: 2, # 'ี' - 40: 0, # 'ึ' - 27: 0, # 'ื' - 32: 0, # 'ุ' - 35: 0, # 'ู' - 11: 1, # 'เ' - 28: 1, # 'แ' - 41: 0, # 'โ' - 29: 1, # 'ใ' - 33: 1, # 'ไ' - 50: 0, # 'ๆ' - 37: 0, # '็' - 6: 0, # '่' - 7: 0, # '้' - 38: 3, # '์' - 56: 0, # '๑' - 59: 0, # '๒' - 60: 0, # '๕' - }, - 20: { # 'ด' - 5: 2, # 'ก' - 30: 2, # 'ข' - 24: 2, # 'ค' - 8: 3, # 'ง' - 26: 2, # 'จ' - 52: 0, # 'ฉ' - 34: 1, # 'ช' - 51: 0, # 'ซ' - 47: 0, # 'ญ' - 58: 0, # 'ฎ' - 57: 0, # 'ฏ' - 49: 0, # 'ฐ' - 53: 0, # 'ฑ' - 55: 0, # 'ฒ' - 43: 0, # 'ณ' - 20: 1, # 'ด' - 19: 2, # 'ต' - 44: 1, # 'ถ' - 14: 2, # 'ท' - 48: 0, # 'ธ' - 3: 1, # 'น' - 17: 1, # 'บ' - 25: 1, # 'ป' - 39: 1, # 'ผ' - 62: 0, # 'ฝ' - 31: 1, # 'พ' - 54: 0, # 'ฟ' - 45: 1, # 'ภ' - 9: 2, # 'ม' - 16: 3, # 'ย' - 2: 2, # 'ร' - 61: 0, # 'ฤ' - 15: 2, # 'ล' - 12: 2, # 'ว' - 42: 0, # 'ศ' - 46: 0, # 'ษ' - 18: 2, # 'ส' - 21: 2, # 'ห' - 4: 1, # 'อ' - 63: 0, # 'ฯ' - 22: 0, # 'ะ' - 10: 3, # 'ั' - 1: 2, # 'า' - 36: 2, # 'ำ' - 23: 3, # 'ิ' - 13: 3, # 'ี' - 40: 1, # 'ึ' - 27: 2, # 'ื' - 32: 3, # 'ุ' - 35: 2, # 'ู' - 11: 2, # 'เ' - 28: 2, # 'แ' - 41: 1, # 'โ' - 29: 2, # 'ใ' - 33: 2, # 'ไ' - 50: 2, # 'ๆ' - 37: 2, # '็' - 6: 1, # '่' - 7: 3, # '้' - 38: 1, # '์' - 56: 0, # '๑' - 59: 0, # '๒' - 60: 0, # '๕' - }, - 19: { # 'ต' - 5: 2, # 'ก' - 30: 1, # 'ข' - 24: 1, # 'ค' - 8: 0, # 'ง' - 26: 1, # 'จ' - 52: 0, # 'ฉ' - 34: 1, # 'ช' - 51: 0, # 'ซ' - 47: 0, # 'ญ' - 58: 0, # 'ฎ' - 57: 0, # 'ฏ' - 49: 0, # 'ฐ' - 53: 0, # 'ฑ' - 55: 0, # 'ฒ' - 43: 0, # 'ณ' - 20: 1, # 'ด' - 19: 1, # 'ต' - 44: 2, # 'ถ' - 14: 1, # 'ท' - 48: 0, # 'ธ' - 3: 2, # 'น' - 17: 1, # 'บ' - 25: 1, # 'ป' - 39: 1, # 'ผ' - 62: 0, # 'ฝ' - 31: 1, # 'พ' - 54: 0, # 'ฟ' - 45: 2, # 'ภ' - 9: 1, # 'ม' - 16: 1, # 'ย' - 2: 3, # 'ร' - 61: 0, # 'ฤ' - 15: 2, # 'ล' - 12: 1, # 'ว' - 42: 0, # 'ศ' - 46: 0, # 'ษ' - 18: 3, # 'ส' - 21: 0, # 'ห' - 4: 3, # 'อ' - 63: 1, # 'ฯ' - 22: 2, # 'ะ' - 10: 3, # 'ั' - 1: 3, # 'า' - 36: 2, # 'ำ' - 23: 3, # 'ิ' - 13: 2, # 'ี' - 40: 1, # 'ึ' - 27: 1, # 'ื' - 32: 3, # 'ุ' - 35: 2, # 'ู' - 11: 1, # 'เ' - 28: 1, # 'แ' - 41: 1, # 'โ' - 29: 1, # 'ใ' - 33: 1, # 'ไ' - 50: 0, # 'ๆ' - 37: 2, # '็' - 6: 3, # '่' - 7: 3, # '้' - 38: 2, # '์' - 56: 0, # '๑' - 59: 0, # '๒' - 60: 0, # '๕' - }, - 44: { # 'ถ' - 5: 1, # 'ก' - 30: 0, # 'ข' - 24: 1, # 'ค' - 8: 0, # 'ง' - 26: 1, # 'จ' - 52: 0, # 'ฉ' - 34: 0, # 'ช' - 51: 0, # 'ซ' - 47: 0, # 'ญ' - 58: 0, # 'ฎ' - 57: 0, # 'ฏ' - 49: 0, # 'ฐ' - 53: 0, # 'ฑ' - 55: 0, # 'ฒ' - 43: 0, # 'ณ' - 20: 0, # 'ด' - 19: 1, # 'ต' - 44: 0, # 'ถ' - 14: 1, # 'ท' - 48: 0, # 'ธ' - 3: 1, # 'น' - 17: 2, # 'บ' - 25: 0, # 'ป' - 39: 0, # 'ผ' - 62: 0, # 'ฝ' - 31: 1, # 'พ' - 54: 0, # 'ฟ' - 45: 0, # 'ภ' - 9: 0, # 'ม' - 16: 0, # 'ย' - 2: 1, # 'ร' - 61: 0, # 'ฤ' - 15: 1, # 'ล' - 12: 1, # 'ว' - 42: 0, # 'ศ' - 46: 0, # 'ษ' - 18: 1, # 'ส' - 21: 0, # 'ห' - 4: 1, # 'อ' - 63: 0, # 'ฯ' - 22: 0, # 'ะ' - 10: 2, # 'ั' - 1: 3, # 'า' - 36: 0, # 'ำ' - 23: 2, # 'ิ' - 13: 1, # 'ี' - 40: 3, # 'ึ' - 27: 2, # 'ื' - 32: 2, # 'ุ' - 35: 3, # 'ู' - 11: 1, # 'เ' - 28: 1, # 'แ' - 41: 0, # 'โ' - 29: 1, # 'ใ' - 33: 1, # 'ไ' - 50: 0, # 'ๆ' - 37: 0, # '็' - 6: 2, # '่' - 7: 3, # '้' - 38: 0, # '์' - 56: 0, # '๑' - 59: 0, # '๒' - 60: 0, # '๕' - }, - 14: { # 'ท' - 5: 1, # 'ก' - 30: 1, # 'ข' - 24: 3, # 'ค' - 8: 1, # 'ง' - 26: 1, # 'จ' - 52: 0, # 'ฉ' - 34: 0, # 'ช' - 51: 0, # 'ซ' - 47: 0, # 'ญ' - 58: 0, # 'ฎ' - 57: 0, # 'ฏ' - 49: 0, # 'ฐ' - 53: 0, # 'ฑ' - 55: 0, # 'ฒ' - 43: 0, # 'ณ' - 20: 2, # 'ด' - 19: 1, # 'ต' - 44: 0, # 'ถ' - 14: 1, # 'ท' - 48: 3, # 'ธ' - 3: 3, # 'น' - 17: 2, # 'บ' - 25: 2, # 'ป' - 39: 1, # 'ผ' - 62: 0, # 'ฝ' - 31: 2, # 'พ' - 54: 0, # 'ฟ' - 45: 0, # 'ภ' - 9: 1, # 'ม' - 16: 3, # 'ย' - 2: 3, # 'ร' - 61: 1, # 'ฤ' - 15: 1, # 'ล' - 12: 2, # 'ว' - 42: 3, # 'ศ' - 46: 1, # 'ษ' - 18: 1, # 'ส' - 21: 0, # 'ห' - 4: 2, # 'อ' - 63: 0, # 'ฯ' - 22: 2, # 'ะ' - 10: 3, # 'ั' - 1: 3, # 'า' - 36: 3, # 'ำ' - 23: 2, # 'ิ' - 13: 3, # 'ี' - 40: 2, # 'ึ' - 27: 1, # 'ื' - 32: 3, # 'ุ' - 35: 1, # 'ู' - 11: 0, # 'เ' - 28: 1, # 'แ' - 41: 0, # 'โ' - 29: 1, # 'ใ' - 33: 0, # 'ไ' - 50: 0, # 'ๆ' - 37: 1, # '็' - 6: 3, # '่' - 7: 3, # '้' - 38: 2, # '์' - 56: 0, # '๑' - 59: 0, # '๒' - 60: 0, # '๕' - }, - 48: { # 'ธ' - 5: 0, # 'ก' - 30: 0, # 'ข' - 24: 0, # 'ค' - 8: 1, # 'ง' - 26: 0, # 'จ' - 52: 0, # 'ฉ' - 34: 0, # 'ช' - 51: 0, # 'ซ' - 47: 0, # 'ญ' - 58: 0, # 'ฎ' - 57: 0, # 'ฏ' - 49: 0, # 'ฐ' - 53: 0, # 'ฑ' - 55: 0, # 'ฒ' - 43: 0, # 'ณ' - 20: 0, # 'ด' - 19: 0, # 'ต' - 44: 0, # 'ถ' - 14: 0, # 'ท' - 48: 0, # 'ธ' - 3: 1, # 'น' - 17: 0, # 'บ' - 25: 0, # 'ป' - 39: 0, # 'ผ' - 62: 0, # 'ฝ' - 31: 0, # 'พ' - 54: 0, # 'ฟ' - 45: 0, # 'ภ' - 9: 0, # 'ม' - 16: 0, # 'ย' - 2: 2, # 'ร' - 61: 0, # 'ฤ' - 15: 0, # 'ล' - 12: 0, # 'ว' - 42: 0, # 'ศ' - 46: 0, # 'ษ' - 18: 0, # 'ส' - 21: 0, # 'ห' - 4: 0, # 'อ' - 63: 0, # 'ฯ' - 22: 0, # 'ะ' - 10: 0, # 'ั' - 1: 2, # 'า' - 36: 0, # 'ำ' - 23: 3, # 'ิ' - 13: 3, # 'ี' - 40: 0, # 'ึ' - 27: 0, # 'ื' - 32: 2, # 'ุ' - 35: 0, # 'ู' - 11: 0, # 'เ' - 28: 0, # 'แ' - 41: 0, # 'โ' - 29: 0, # 'ใ' - 33: 0, # 'ไ' - 50: 0, # 'ๆ' - 37: 0, # '็' - 6: 0, # '่' - 7: 0, # '้' - 38: 3, # '์' - 56: 0, # '๑' - 59: 0, # '๒' - 60: 0, # '๕' - }, - 3: { # 'น' - 5: 3, # 'ก' - 30: 2, # 'ข' - 24: 3, # 'ค' - 8: 1, # 'ง' - 26: 2, # 'จ' - 52: 0, # 'ฉ' - 34: 1, # 'ช' - 51: 1, # 'ซ' - 47: 0, # 'ญ' - 58: 0, # 'ฎ' - 57: 0, # 'ฏ' - 49: 1, # 'ฐ' - 53: 0, # 'ฑ' - 55: 0, # 'ฒ' - 43: 0, # 'ณ' - 20: 3, # 'ด' - 19: 3, # 'ต' - 44: 2, # 'ถ' - 14: 3, # 'ท' - 48: 3, # 'ธ' - 3: 2, # 'น' - 17: 2, # 'บ' - 25: 2, # 'ป' - 39: 2, # 'ผ' - 62: 0, # 'ฝ' - 31: 2, # 'พ' - 54: 1, # 'ฟ' - 45: 1, # 'ภ' - 9: 2, # 'ม' - 16: 2, # 'ย' - 2: 2, # 'ร' - 61: 1, # 'ฤ' - 15: 2, # 'ล' - 12: 3, # 'ว' - 42: 1, # 'ศ' - 46: 0, # 'ษ' - 18: 2, # 'ส' - 21: 2, # 'ห' - 4: 3, # 'อ' - 63: 1, # 'ฯ' - 22: 2, # 'ะ' - 10: 3, # 'ั' - 1: 3, # 'า' - 36: 3, # 'ำ' - 23: 3, # 'ิ' - 13: 3, # 'ี' - 40: 3, # 'ึ' - 27: 3, # 'ื' - 32: 3, # 'ุ' - 35: 2, # 'ู' - 11: 3, # 'เ' - 28: 2, # 'แ' - 41: 3, # 'โ' - 29: 3, # 'ใ' - 33: 3, # 'ไ' - 50: 2, # 'ๆ' - 37: 1, # '็' - 6: 3, # '่' - 7: 3, # '้' - 38: 2, # '์' - 56: 0, # '๑' - 59: 0, # '๒' - 60: 0, # '๕' - }, - 17: { # 'บ' - 5: 3, # 'ก' - 30: 2, # 'ข' - 24: 2, # 'ค' - 8: 1, # 'ง' - 26: 1, # 'จ' - 52: 1, # 'ฉ' - 34: 1, # 'ช' - 51: 1, # 'ซ' - 47: 0, # 'ญ' - 58: 0, # 'ฎ' - 57: 0, # 'ฏ' - 49: 0, # 'ฐ' - 53: 0, # 'ฑ' - 55: 0, # 'ฒ' - 43: 0, # 'ณ' - 20: 1, # 'ด' - 19: 2, # 'ต' - 44: 1, # 'ถ' - 14: 3, # 'ท' - 48: 0, # 'ธ' - 3: 3, # 'น' - 17: 3, # 'บ' - 25: 2, # 'ป' - 39: 2, # 'ผ' - 62: 0, # 'ฝ' - 31: 1, # 'พ' - 54: 1, # 'ฟ' - 45: 1, # 'ภ' - 9: 1, # 'ม' - 16: 0, # 'ย' - 2: 3, # 'ร' - 61: 0, # 'ฤ' - 15: 2, # 'ล' - 12: 3, # 'ว' - 42: 0, # 'ศ' - 46: 0, # 'ษ' - 18: 2, # 'ส' - 21: 2, # 'ห' - 4: 2, # 'อ' - 63: 1, # 'ฯ' - 22: 0, # 'ะ' - 10: 3, # 'ั' - 1: 3, # 'า' - 36: 2, # 'ำ' - 23: 2, # 'ิ' - 13: 2, # 'ี' - 40: 0, # 'ึ' - 27: 2, # 'ื' - 32: 3, # 'ุ' - 35: 2, # 'ู' - 11: 2, # 'เ' - 28: 2, # 'แ' - 41: 1, # 'โ' - 29: 2, # 'ใ' - 33: 2, # 'ไ' - 50: 0, # 'ๆ' - 37: 1, # '็' - 6: 2, # '่' - 7: 2, # '้' - 38: 0, # '์' - 56: 0, # '๑' - 59: 0, # '๒' - 60: 0, # '๕' - }, - 25: { # 'ป' - 5: 2, # 'ก' - 30: 0, # 'ข' - 24: 1, # 'ค' - 8: 0, # 'ง' - 26: 1, # 'จ' - 52: 0, # 'ฉ' - 34: 0, # 'ช' - 51: 1, # 'ซ' - 47: 0, # 'ญ' - 58: 1, # 'ฎ' - 57: 3, # 'ฏ' - 49: 1, # 'ฐ' - 53: 0, # 'ฑ' - 55: 0, # 'ฒ' - 43: 0, # 'ณ' - 20: 1, # 'ด' - 19: 1, # 'ต' - 44: 1, # 'ถ' - 14: 1, # 'ท' - 48: 0, # 'ธ' - 3: 2, # 'น' - 17: 0, # 'บ' - 25: 1, # 'ป' - 39: 1, # 'ผ' - 62: 1, # 'ฝ' - 31: 1, # 'พ' - 54: 0, # 'ฟ' - 45: 0, # 'ภ' - 9: 1, # 'ม' - 16: 0, # 'ย' - 2: 3, # 'ร' - 61: 0, # 'ฤ' - 15: 3, # 'ล' - 12: 1, # 'ว' - 42: 0, # 'ศ' - 46: 1, # 'ษ' - 18: 2, # 'ส' - 21: 1, # 'ห' - 4: 2, # 'อ' - 63: 0, # 'ฯ' - 22: 1, # 'ะ' - 10: 3, # 'ั' - 1: 1, # 'า' - 36: 0, # 'ำ' - 23: 2, # 'ิ' - 13: 3, # 'ี' - 40: 0, # 'ึ' - 27: 0, # 'ื' - 32: 1, # 'ุ' - 35: 0, # 'ู' - 11: 1, # 'เ' - 28: 2, # 'แ' - 41: 0, # 'โ' - 29: 1, # 'ใ' - 33: 2, # 'ไ' - 50: 0, # 'ๆ' - 37: 3, # '็' - 6: 1, # '่' - 7: 2, # '้' - 38: 1, # '์' - 56: 0, # '๑' - 59: 0, # '๒' - 60: 0, # '๕' - }, - 39: { # 'ผ' - 5: 1, # 'ก' - 30: 0, # 'ข' - 24: 0, # 'ค' - 8: 1, # 'ง' - 26: 0, # 'จ' - 52: 0, # 'ฉ' - 34: 0, # 'ช' - 51: 0, # 'ซ' - 47: 0, # 'ญ' - 58: 0, # 'ฎ' - 57: 0, # 'ฏ' - 49: 0, # 'ฐ' - 53: 0, # 'ฑ' - 55: 0, # 'ฒ' - 43: 0, # 'ณ' - 20: 0, # 'ด' - 19: 0, # 'ต' - 44: 0, # 'ถ' - 14: 0, # 'ท' - 48: 0, # 'ธ' - 3: 2, # 'น' - 17: 0, # 'บ' - 25: 0, # 'ป' - 39: 0, # 'ผ' - 62: 0, # 'ฝ' - 31: 0, # 'พ' - 54: 0, # 'ฟ' - 45: 0, # 'ภ' - 9: 1, # 'ม' - 16: 2, # 'ย' - 2: 0, # 'ร' - 61: 0, # 'ฤ' - 15: 3, # 'ล' - 12: 0, # 'ว' - 42: 0, # 'ศ' - 46: 0, # 'ษ' - 18: 1, # 'ส' - 21: 0, # 'ห' - 4: 0, # 'อ' - 63: 0, # 'ฯ' - 22: 1, # 'ะ' - 10: 1, # 'ั' - 1: 0, # 'า' - 36: 0, # 'ำ' - 23: 2, # 'ิ' - 13: 0, # 'ี' - 40: 0, # 'ึ' - 27: 1, # 'ื' - 32: 0, # 'ุ' - 35: 3, # 'ู' - 11: 0, # 'เ' - 28: 0, # 'แ' - 41: 0, # 'โ' - 29: 0, # 'ใ' - 33: 0, # 'ไ' - 50: 0, # 'ๆ' - 37: 0, # '็' - 6: 3, # '่' - 7: 1, # '้' - 38: 0, # '์' - 56: 0, # '๑' - 59: 0, # '๒' - 60: 0, # '๕' - }, - 62: { # 'ฝ' - 5: 0, # 'ก' - 30: 0, # 'ข' - 24: 0, # 'ค' - 8: 0, # 'ง' - 26: 0, # 'จ' - 52: 0, # 'ฉ' - 34: 0, # 'ช' - 51: 0, # 'ซ' - 47: 0, # 'ญ' - 58: 0, # 'ฎ' - 57: 0, # 'ฏ' - 49: 0, # 'ฐ' - 53: 0, # 'ฑ' - 55: 0, # 'ฒ' - 43: 0, # 'ณ' - 20: 0, # 'ด' - 19: 0, # 'ต' - 44: 0, # 'ถ' - 14: 0, # 'ท' - 48: 0, # 'ธ' - 3: 1, # 'น' - 17: 0, # 'บ' - 25: 0, # 'ป' - 39: 0, # 'ผ' - 62: 0, # 'ฝ' - 31: 0, # 'พ' - 54: 0, # 'ฟ' - 45: 0, # 'ภ' - 9: 0, # 'ม' - 16: 0, # 'ย' - 2: 1, # 'ร' - 61: 0, # 'ฤ' - 15: 0, # 'ล' - 12: 0, # 'ว' - 42: 0, # 'ศ' - 46: 0, # 'ษ' - 18: 0, # 'ส' - 21: 0, # 'ห' - 4: 0, # 'อ' - 63: 0, # 'ฯ' - 22: 0, # 'ะ' - 10: 1, # 'ั' - 1: 0, # 'า' - 36: 0, # 'ำ' - 23: 0, # 'ิ' - 13: 1, # 'ี' - 40: 2, # 'ึ' - 27: 0, # 'ื' - 32: 0, # 'ุ' - 35: 0, # 'ู' - 11: 0, # 'เ' - 28: 0, # 'แ' - 41: 0, # 'โ' - 29: 0, # 'ใ' - 33: 0, # 'ไ' - 50: 0, # 'ๆ' - 37: 0, # '็' - 6: 2, # '่' - 7: 1, # '้' - 38: 0, # '์' - 56: 0, # '๑' - 59: 0, # '๒' - 60: 0, # '๕' - }, - 31: { # 'พ' - 5: 1, # 'ก' - 30: 1, # 'ข' - 24: 1, # 'ค' - 8: 1, # 'ง' - 26: 1, # 'จ' - 52: 0, # 'ฉ' - 34: 0, # 'ช' - 51: 0, # 'ซ' - 47: 0, # 'ญ' - 58: 0, # 'ฎ' - 57: 0, # 'ฏ' - 49: 0, # 'ฐ' - 53: 0, # 'ฑ' - 55: 0, # 'ฒ' - 43: 1, # 'ณ' - 20: 1, # 'ด' - 19: 1, # 'ต' - 44: 0, # 'ถ' - 14: 2, # 'ท' - 48: 1, # 'ธ' - 3: 3, # 'น' - 17: 2, # 'บ' - 25: 0, # 'ป' - 39: 1, # 'ผ' - 62: 0, # 'ฝ' - 31: 1, # 'พ' - 54: 0, # 'ฟ' - 45: 0, # 'ภ' - 9: 1, # 'ม' - 16: 2, # 'ย' - 2: 3, # 'ร' - 61: 2, # 'ฤ' - 15: 2, # 'ล' - 12: 2, # 'ว' - 42: 0, # 'ศ' - 46: 0, # 'ษ' - 18: 1, # 'ส' - 21: 1, # 'ห' - 4: 2, # 'อ' - 63: 1, # 'ฯ' - 22: 0, # 'ะ' - 10: 3, # 'ั' - 1: 3, # 'า' - 36: 0, # 'ำ' - 23: 3, # 'ิ' - 13: 2, # 'ี' - 40: 1, # 'ึ' - 27: 3, # 'ื' - 32: 1, # 'ุ' - 35: 2, # 'ู' - 11: 1, # 'เ' - 28: 1, # 'แ' - 41: 0, # 'โ' - 29: 1, # 'ใ' - 33: 1, # 'ไ' - 50: 0, # 'ๆ' - 37: 1, # '็' - 6: 0, # '่' - 7: 1, # '้' - 38: 3, # '์' - 56: 0, # '๑' - 59: 0, # '๒' - 60: 0, # '๕' - }, - 54: { # 'ฟ' - 5: 0, # 'ก' - 30: 0, # 'ข' - 24: 0, # 'ค' - 8: 0, # 'ง' - 26: 0, # 'จ' - 52: 0, # 'ฉ' - 34: 1, # 'ช' - 51: 0, # 'ซ' - 47: 0, # 'ญ' - 58: 0, # 'ฎ' - 57: 0, # 'ฏ' - 49: 0, # 'ฐ' - 53: 0, # 'ฑ' - 55: 0, # 'ฒ' - 43: 0, # 'ณ' - 20: 0, # 'ด' - 19: 1, # 'ต' - 44: 0, # 'ถ' - 14: 1, # 'ท' - 48: 0, # 'ธ' - 3: 0, # 'น' - 17: 0, # 'บ' - 25: 0, # 'ป' - 39: 0, # 'ผ' - 62: 0, # 'ฝ' - 31: 0, # 'พ' - 54: 2, # 'ฟ' - 45: 0, # 'ภ' - 9: 0, # 'ม' - 16: 0, # 'ย' - 2: 1, # 'ร' - 61: 0, # 'ฤ' - 15: 2, # 'ล' - 12: 0, # 'ว' - 42: 0, # 'ศ' - 46: 0, # 'ษ' - 18: 1, # 'ส' - 21: 0, # 'ห' - 4: 1, # 'อ' - 63: 0, # 'ฯ' - 22: 0, # 'ะ' - 10: 2, # 'ั' - 1: 0, # 'า' - 36: 0, # 'ำ' - 23: 1, # 'ิ' - 13: 1, # 'ี' - 40: 0, # 'ึ' - 27: 1, # 'ื' - 32: 1, # 'ุ' - 35: 0, # 'ู' - 11: 0, # 'เ' - 28: 1, # 'แ' - 41: 0, # 'โ' - 29: 0, # 'ใ' - 33: 0, # 'ไ' - 50: 0, # 'ๆ' - 37: 0, # '็' - 6: 0, # '่' - 7: 2, # '้' - 38: 0, # '์' - 56: 0, # '๑' - 59: 0, # '๒' - 60: 0, # '๕' - }, - 45: { # 'ภ' - 5: 0, # 'ก' - 30: 0, # 'ข' - 24: 1, # 'ค' - 8: 0, # 'ง' - 26: 0, # 'จ' - 52: 0, # 'ฉ' - 34: 0, # 'ช' - 51: 0, # 'ซ' - 47: 0, # 'ญ' - 58: 0, # 'ฎ' - 57: 0, # 'ฏ' - 49: 0, # 'ฐ' - 53: 0, # 'ฑ' - 55: 0, # 'ฒ' - 43: 0, # 'ณ' - 20: 0, # 'ด' - 19: 0, # 'ต' - 44: 0, # 'ถ' - 14: 3, # 'ท' - 48: 0, # 'ธ' - 3: 0, # 'น' - 17: 0, # 'บ' - 25: 0, # 'ป' - 39: 0, # 'ผ' - 62: 0, # 'ฝ' - 31: 1, # 'พ' - 54: 0, # 'ฟ' - 45: 0, # 'ภ' - 9: 0, # 'ม' - 16: 0, # 'ย' - 2: 1, # 'ร' - 61: 0, # 'ฤ' - 15: 0, # 'ล' - 12: 0, # 'ว' - 42: 0, # 'ศ' - 46: 0, # 'ษ' - 18: 0, # 'ส' - 21: 0, # 'ห' - 4: 0, # 'อ' - 63: 0, # 'ฯ' - 22: 0, # 'ะ' - 10: 3, # 'ั' - 1: 3, # 'า' - 36: 0, # 'ำ' - 23: 1, # 'ิ' - 13: 0, # 'ี' - 40: 0, # 'ึ' - 27: 0, # 'ื' - 32: 0, # 'ุ' - 35: 2, # 'ู' - 11: 0, # 'เ' - 28: 0, # 'แ' - 41: 0, # 'โ' - 29: 0, # 'ใ' - 33: 0, # 'ไ' - 50: 0, # 'ๆ' - 37: 0, # '็' - 6: 0, # '่' - 7: 0, # '้' - 38: 1, # '์' - 56: 0, # '๑' - 59: 0, # '๒' - 60: 0, # '๕' - }, - 9: { # 'ม' - 5: 2, # 'ก' - 30: 2, # 'ข' - 24: 2, # 'ค' - 8: 2, # 'ง' - 26: 2, # 'จ' - 52: 0, # 'ฉ' - 34: 1, # 'ช' - 51: 1, # 'ซ' - 47: 0, # 'ญ' - 58: 0, # 'ฎ' - 57: 0, # 'ฏ' - 49: 0, # 'ฐ' - 53: 0, # 'ฑ' - 55: 0, # 'ฒ' - 43: 1, # 'ณ' - 20: 2, # 'ด' - 19: 2, # 'ต' - 44: 1, # 'ถ' - 14: 2, # 'ท' - 48: 1, # 'ธ' - 3: 3, # 'น' - 17: 2, # 'บ' - 25: 2, # 'ป' - 39: 1, # 'ผ' - 62: 0, # 'ฝ' - 31: 3, # 'พ' - 54: 0, # 'ฟ' - 45: 1, # 'ภ' - 9: 2, # 'ม' - 16: 1, # 'ย' - 2: 2, # 'ร' - 61: 2, # 'ฤ' - 15: 2, # 'ล' - 12: 2, # 'ว' - 42: 1, # 'ศ' - 46: 1, # 'ษ' - 18: 3, # 'ส' - 21: 3, # 'ห' - 4: 3, # 'อ' - 63: 0, # 'ฯ' - 22: 1, # 'ะ' - 10: 3, # 'ั' - 1: 3, # 'า' - 36: 0, # 'ำ' - 23: 3, # 'ิ' - 13: 3, # 'ี' - 40: 0, # 'ึ' - 27: 3, # 'ื' - 32: 3, # 'ุ' - 35: 3, # 'ู' - 11: 2, # 'เ' - 28: 2, # 'แ' - 41: 2, # 'โ' - 29: 2, # 'ใ' - 33: 2, # 'ไ' - 50: 1, # 'ๆ' - 37: 1, # '็' - 6: 3, # '่' - 7: 2, # '้' - 38: 1, # '์' - 56: 0, # '๑' - 59: 0, # '๒' - 60: 0, # '๕' - }, - 16: { # 'ย' - 5: 3, # 'ก' - 30: 1, # 'ข' - 24: 2, # 'ค' - 8: 3, # 'ง' - 26: 2, # 'จ' - 52: 0, # 'ฉ' - 34: 2, # 'ช' - 51: 0, # 'ซ' - 47: 2, # 'ญ' - 58: 0, # 'ฎ' - 57: 0, # 'ฏ' - 49: 0, # 'ฐ' - 53: 0, # 'ฑ' - 55: 0, # 'ฒ' - 43: 0, # 'ณ' - 20: 2, # 'ด' - 19: 2, # 'ต' - 44: 1, # 'ถ' - 14: 2, # 'ท' - 48: 1, # 'ธ' - 3: 3, # 'น' - 17: 3, # 'บ' - 25: 1, # 'ป' - 39: 1, # 'ผ' - 62: 0, # 'ฝ' - 31: 1, # 'พ' - 54: 0, # 'ฟ' - 45: 1, # 'ภ' - 9: 2, # 'ม' - 16: 0, # 'ย' - 2: 2, # 'ร' - 61: 0, # 'ฤ' - 15: 1, # 'ล' - 12: 3, # 'ว' - 42: 1, # 'ศ' - 46: 0, # 'ษ' - 18: 2, # 'ส' - 21: 1, # 'ห' - 4: 2, # 'อ' - 63: 0, # 'ฯ' - 22: 2, # 'ะ' - 10: 3, # 'ั' - 1: 3, # 'า' - 36: 0, # 'ำ' - 23: 2, # 'ิ' - 13: 3, # 'ี' - 40: 1, # 'ึ' - 27: 2, # 'ื' - 32: 2, # 'ุ' - 35: 3, # 'ู' - 11: 2, # 'เ' - 28: 1, # 'แ' - 41: 1, # 'โ' - 29: 2, # 'ใ' - 33: 2, # 'ไ' - 50: 2, # 'ๆ' - 37: 1, # '็' - 6: 3, # '่' - 7: 2, # '้' - 38: 3, # '์' - 56: 0, # '๑' - 59: 0, # '๒' - 60: 0, # '๕' - }, - 2: { # 'ร' - 5: 3, # 'ก' - 30: 2, # 'ข' - 24: 2, # 'ค' - 8: 3, # 'ง' - 26: 2, # 'จ' - 52: 0, # 'ฉ' - 34: 2, # 'ช' - 51: 1, # 'ซ' - 47: 0, # 'ญ' - 58: 0, # 'ฎ' - 57: 0, # 'ฏ' - 49: 3, # 'ฐ' - 53: 0, # 'ฑ' - 55: 0, # 'ฒ' - 43: 3, # 'ณ' - 20: 2, # 'ด' - 19: 2, # 'ต' - 44: 3, # 'ถ' - 14: 3, # 'ท' - 48: 1, # 'ธ' - 3: 2, # 'น' - 17: 2, # 'บ' - 25: 3, # 'ป' - 39: 2, # 'ผ' - 62: 1, # 'ฝ' - 31: 2, # 'พ' - 54: 1, # 'ฟ' - 45: 1, # 'ภ' - 9: 3, # 'ม' - 16: 2, # 'ย' - 2: 3, # 'ร' - 61: 0, # 'ฤ' - 15: 2, # 'ล' - 12: 3, # 'ว' - 42: 2, # 'ศ' - 46: 2, # 'ษ' - 18: 2, # 'ส' - 21: 2, # 'ห' - 4: 3, # 'อ' - 63: 1, # 'ฯ' - 22: 3, # 'ะ' - 10: 3, # 'ั' - 1: 3, # 'า' - 36: 0, # 'ำ' - 23: 3, # 'ิ' - 13: 3, # 'ี' - 40: 2, # 'ึ' - 27: 3, # 'ื' - 32: 3, # 'ุ' - 35: 3, # 'ู' - 11: 3, # 'เ' - 28: 3, # 'แ' - 41: 1, # 'โ' - 29: 2, # 'ใ' - 33: 1, # 'ไ' - 50: 0, # 'ๆ' - 37: 3, # '็' - 6: 3, # '่' - 7: 3, # '้' - 38: 3, # '์' - 56: 0, # '๑' - 59: 0, # '๒' - 60: 0, # '๕' - }, - 61: { # 'ฤ' - 5: 0, # 'ก' - 30: 0, # 'ข' - 24: 0, # 'ค' - 8: 0, # 'ง' - 26: 0, # 'จ' - 52: 0, # 'ฉ' - 34: 0, # 'ช' - 51: 0, # 'ซ' - 47: 0, # 'ญ' - 58: 0, # 'ฎ' - 57: 0, # 'ฏ' - 49: 0, # 'ฐ' - 53: 0, # 'ฑ' - 55: 0, # 'ฒ' - 43: 0, # 'ณ' - 20: 0, # 'ด' - 19: 2, # 'ต' - 44: 0, # 'ถ' - 14: 2, # 'ท' - 48: 0, # 'ธ' - 3: 0, # 'น' - 17: 0, # 'บ' - 25: 0, # 'ป' - 39: 0, # 'ผ' - 62: 0, # 'ฝ' - 31: 0, # 'พ' - 54: 0, # 'ฟ' - 45: 0, # 'ภ' - 9: 1, # 'ม' - 16: 0, # 'ย' - 2: 0, # 'ร' - 61: 0, # 'ฤ' - 15: 0, # 'ล' - 12: 0, # 'ว' - 42: 0, # 'ศ' - 46: 2, # 'ษ' - 18: 0, # 'ส' - 21: 0, # 'ห' - 4: 0, # 'อ' - 63: 0, # 'ฯ' - 22: 0, # 'ะ' - 10: 0, # 'ั' - 1: 0, # 'า' - 36: 0, # 'ำ' - 23: 0, # 'ิ' - 13: 0, # 'ี' - 40: 0, # 'ึ' - 27: 0, # 'ื' - 32: 0, # 'ุ' - 35: 0, # 'ู' - 11: 0, # 'เ' - 28: 0, # 'แ' - 41: 0, # 'โ' - 29: 0, # 'ใ' - 33: 0, # 'ไ' - 50: 0, # 'ๆ' - 37: 0, # '็' - 6: 0, # '่' - 7: 0, # '้' - 38: 0, # '์' - 56: 0, # '๑' - 59: 0, # '๒' - 60: 0, # '๕' - }, - 15: { # 'ล' - 5: 2, # 'ก' - 30: 3, # 'ข' - 24: 1, # 'ค' - 8: 3, # 'ง' - 26: 1, # 'จ' - 52: 0, # 'ฉ' - 34: 1, # 'ช' - 51: 0, # 'ซ' - 47: 0, # 'ญ' - 58: 0, # 'ฎ' - 57: 0, # 'ฏ' - 49: 0, # 'ฐ' - 53: 0, # 'ฑ' - 55: 0, # 'ฒ' - 43: 0, # 'ณ' - 20: 2, # 'ด' - 19: 2, # 'ต' - 44: 1, # 'ถ' - 14: 2, # 'ท' - 48: 0, # 'ธ' - 3: 1, # 'น' - 17: 2, # 'บ' - 25: 2, # 'ป' - 39: 1, # 'ผ' - 62: 0, # 'ฝ' - 31: 0, # 'พ' - 54: 0, # 'ฟ' - 45: 1, # 'ภ' - 9: 1, # 'ม' - 16: 3, # 'ย' - 2: 1, # 'ร' - 61: 0, # 'ฤ' - 15: 1, # 'ล' - 12: 1, # 'ว' - 42: 0, # 'ศ' - 46: 0, # 'ษ' - 18: 2, # 'ส' - 21: 1, # 'ห' - 4: 3, # 'อ' - 63: 2, # 'ฯ' - 22: 3, # 'ะ' - 10: 3, # 'ั' - 1: 3, # 'า' - 36: 2, # 'ำ' - 23: 3, # 'ิ' - 13: 3, # 'ี' - 40: 2, # 'ึ' - 27: 3, # 'ื' - 32: 2, # 'ุ' - 35: 3, # 'ู' - 11: 2, # 'เ' - 28: 1, # 'แ' - 41: 1, # 'โ' - 29: 2, # 'ใ' - 33: 1, # 'ไ' - 50: 0, # 'ๆ' - 37: 2, # '็' - 6: 3, # '่' - 7: 3, # '้' - 38: 2, # '์' - 56: 0, # '๑' - 59: 0, # '๒' - 60: 0, # '๕' - }, - 12: { # 'ว' - 5: 3, # 'ก' - 30: 2, # 'ข' - 24: 1, # 'ค' - 8: 3, # 'ง' - 26: 2, # 'จ' - 52: 0, # 'ฉ' - 34: 1, # 'ช' - 51: 1, # 'ซ' - 47: 0, # 'ญ' - 58: 0, # 'ฎ' - 57: 0, # 'ฏ' - 49: 0, # 'ฐ' - 53: 0, # 'ฑ' - 55: 0, # 'ฒ' - 43: 1, # 'ณ' - 20: 2, # 'ด' - 19: 1, # 'ต' - 44: 1, # 'ถ' - 14: 1, # 'ท' - 48: 0, # 'ธ' - 3: 3, # 'น' - 17: 2, # 'บ' - 25: 1, # 'ป' - 39: 1, # 'ผ' - 62: 0, # 'ฝ' - 31: 1, # 'พ' - 54: 1, # 'ฟ' - 45: 0, # 'ภ' - 9: 3, # 'ม' - 16: 3, # 'ย' - 2: 3, # 'ร' - 61: 0, # 'ฤ' - 15: 3, # 'ล' - 12: 1, # 'ว' - 42: 0, # 'ศ' - 46: 0, # 'ษ' - 18: 2, # 'ส' - 21: 2, # 'ห' - 4: 2, # 'อ' - 63: 0, # 'ฯ' - 22: 2, # 'ะ' - 10: 3, # 'ั' - 1: 3, # 'า' - 36: 0, # 'ำ' - 23: 3, # 'ิ' - 13: 2, # 'ี' - 40: 0, # 'ึ' - 27: 0, # 'ื' - 32: 2, # 'ุ' - 35: 0, # 'ู' - 11: 3, # 'เ' - 28: 2, # 'แ' - 41: 1, # 'โ' - 29: 1, # 'ใ' - 33: 2, # 'ไ' - 50: 1, # 'ๆ' - 37: 0, # '็' - 6: 3, # '่' - 7: 3, # '้' - 38: 1, # '์' - 56: 0, # '๑' - 59: 0, # '๒' - 60: 0, # '๕' - }, - 42: { # 'ศ' - 5: 1, # 'ก' - 30: 0, # 'ข' - 24: 1, # 'ค' - 8: 0, # 'ง' - 26: 1, # 'จ' - 52: 0, # 'ฉ' - 34: 0, # 'ช' - 51: 0, # 'ซ' - 47: 1, # 'ญ' - 58: 0, # 'ฎ' - 57: 0, # 'ฏ' - 49: 0, # 'ฐ' - 53: 0, # 'ฑ' - 55: 0, # 'ฒ' - 43: 0, # 'ณ' - 20: 0, # 'ด' - 19: 1, # 'ต' - 44: 0, # 'ถ' - 14: 1, # 'ท' - 48: 0, # 'ธ' - 3: 2, # 'น' - 17: 0, # 'บ' - 25: 0, # 'ป' - 39: 0, # 'ผ' - 62: 0, # 'ฝ' - 31: 0, # 'พ' - 54: 0, # 'ฟ' - 45: 0, # 'ภ' - 9: 0, # 'ม' - 16: 0, # 'ย' - 2: 2, # 'ร' - 61: 0, # 'ฤ' - 15: 0, # 'ล' - 12: 2, # 'ว' - 42: 1, # 'ศ' - 46: 2, # 'ษ' - 18: 1, # 'ส' - 21: 0, # 'ห' - 4: 0, # 'อ' - 63: 0, # 'ฯ' - 22: 0, # 'ะ' - 10: 2, # 'ั' - 1: 3, # 'า' - 36: 0, # 'ำ' - 23: 2, # 'ิ' - 13: 0, # 'ี' - 40: 3, # 'ึ' - 27: 0, # 'ื' - 32: 0, # 'ุ' - 35: 2, # 'ู' - 11: 0, # 'เ' - 28: 1, # 'แ' - 41: 0, # 'โ' - 29: 1, # 'ใ' - 33: 1, # 'ไ' - 50: 0, # 'ๆ' - 37: 0, # '็' - 6: 0, # '่' - 7: 0, # '้' - 38: 1, # '์' - 56: 0, # '๑' - 59: 0, # '๒' - 60: 0, # '๕' - }, - 46: { # 'ษ' - 5: 0, # 'ก' - 30: 0, # 'ข' - 24: 0, # 'ค' - 8: 0, # 'ง' - 26: 0, # 'จ' - 52: 0, # 'ฉ' - 34: 0, # 'ช' - 51: 0, # 'ซ' - 47: 0, # 'ญ' - 58: 2, # 'ฎ' - 57: 1, # 'ฏ' - 49: 2, # 'ฐ' - 53: 0, # 'ฑ' - 55: 0, # 'ฒ' - 43: 3, # 'ณ' - 20: 0, # 'ด' - 19: 1, # 'ต' - 44: 0, # 'ถ' - 14: 1, # 'ท' - 48: 0, # 'ธ' - 3: 0, # 'น' - 17: 0, # 'บ' - 25: 0, # 'ป' - 39: 0, # 'ผ' - 62: 0, # 'ฝ' - 31: 0, # 'พ' - 54: 0, # 'ฟ' - 45: 1, # 'ภ' - 9: 1, # 'ม' - 16: 2, # 'ย' - 2: 2, # 'ร' - 61: 0, # 'ฤ' - 15: 0, # 'ล' - 12: 0, # 'ว' - 42: 1, # 'ศ' - 46: 0, # 'ษ' - 18: 0, # 'ส' - 21: 0, # 'ห' - 4: 0, # 'อ' - 63: 0, # 'ฯ' - 22: 2, # 'ะ' - 10: 2, # 'ั' - 1: 3, # 'า' - 36: 0, # 'ำ' - 23: 0, # 'ิ' - 13: 1, # 'ี' - 40: 0, # 'ึ' - 27: 0, # 'ื' - 32: 0, # 'ุ' - 35: 0, # 'ู' - 11: 1, # 'เ' - 28: 0, # 'แ' - 41: 0, # 'โ' - 29: 0, # 'ใ' - 33: 0, # 'ไ' - 50: 0, # 'ๆ' - 37: 0, # '็' - 6: 0, # '่' - 7: 0, # '้' - 38: 2, # '์' - 56: 0, # '๑' - 59: 0, # '๒' - 60: 0, # '๕' - }, - 18: { # 'ส' - 5: 2, # 'ก' - 30: 0, # 'ข' - 24: 0, # 'ค' - 8: 2, # 'ง' - 26: 1, # 'จ' - 52: 0, # 'ฉ' - 34: 0, # 'ช' - 51: 0, # 'ซ' - 47: 0, # 'ญ' - 58: 0, # 'ฎ' - 57: 0, # 'ฏ' - 49: 0, # 'ฐ' - 53: 0, # 'ฑ' - 55: 0, # 'ฒ' - 43: 0, # 'ณ' - 20: 3, # 'ด' - 19: 3, # 'ต' - 44: 3, # 'ถ' - 14: 0, # 'ท' - 48: 0, # 'ธ' - 3: 3, # 'น' - 17: 2, # 'บ' - 25: 1, # 'ป' - 39: 0, # 'ผ' - 62: 0, # 'ฝ' - 31: 0, # 'พ' - 54: 0, # 'ฟ' - 45: 2, # 'ภ' - 9: 3, # 'ม' - 16: 1, # 'ย' - 2: 3, # 'ร' - 61: 0, # 'ฤ' - 15: 1, # 'ล' - 12: 2, # 'ว' - 42: 0, # 'ศ' - 46: 0, # 'ษ' - 18: 0, # 'ส' - 21: 2, # 'ห' - 4: 3, # 'อ' - 63: 0, # 'ฯ' - 22: 2, # 'ะ' - 10: 3, # 'ั' - 1: 3, # 'า' - 36: 3, # 'ำ' - 23: 3, # 'ิ' - 13: 3, # 'ี' - 40: 2, # 'ึ' - 27: 3, # 'ื' - 32: 3, # 'ุ' - 35: 3, # 'ู' - 11: 2, # 'เ' - 28: 0, # 'แ' - 41: 1, # 'โ' - 29: 0, # 'ใ' - 33: 1, # 'ไ' - 50: 0, # 'ๆ' - 37: 0, # '็' - 6: 3, # '่' - 7: 1, # '้' - 38: 2, # '์' - 56: 0, # '๑' - 59: 0, # '๒' - 60: 0, # '๕' - }, - 21: { # 'ห' - 5: 3, # 'ก' - 30: 0, # 'ข' - 24: 0, # 'ค' - 8: 1, # 'ง' - 26: 0, # 'จ' - 52: 0, # 'ฉ' - 34: 0, # 'ช' - 51: 0, # 'ซ' - 47: 2, # 'ญ' - 58: 0, # 'ฎ' - 57: 0, # 'ฏ' - 49: 0, # 'ฐ' - 53: 0, # 'ฑ' - 55: 0, # 'ฒ' - 43: 0, # 'ณ' - 20: 1, # 'ด' - 19: 3, # 'ต' - 44: 0, # 'ถ' - 14: 0, # 'ท' - 48: 0, # 'ธ' - 3: 3, # 'น' - 17: 0, # 'บ' - 25: 1, # 'ป' - 39: 0, # 'ผ' - 62: 0, # 'ฝ' - 31: 1, # 'พ' - 54: 0, # 'ฟ' - 45: 0, # 'ภ' - 9: 3, # 'ม' - 16: 2, # 'ย' - 2: 3, # 'ร' - 61: 0, # 'ฤ' - 15: 3, # 'ล' - 12: 2, # 'ว' - 42: 0, # 'ศ' - 46: 0, # 'ษ' - 18: 0, # 'ส' - 21: 0, # 'ห' - 4: 3, # 'อ' - 63: 0, # 'ฯ' - 22: 1, # 'ะ' - 10: 3, # 'ั' - 1: 3, # 'า' - 36: 0, # 'ำ' - 23: 1, # 'ิ' - 13: 1, # 'ี' - 40: 0, # 'ึ' - 27: 0, # 'ื' - 32: 1, # 'ุ' - 35: 1, # 'ู' - 11: 0, # 'เ' - 28: 0, # 'แ' - 41: 0, # 'โ' - 29: 0, # 'ใ' - 33: 0, # 'ไ' - 50: 0, # 'ๆ' - 37: 3, # '็' - 6: 3, # '่' - 7: 3, # '้' - 38: 2, # '์' - 56: 0, # '๑' - 59: 0, # '๒' - 60: 0, # '๕' - }, - 4: { # 'อ' - 5: 3, # 'ก' - 30: 1, # 'ข' - 24: 2, # 'ค' - 8: 3, # 'ง' - 26: 1, # 'จ' - 52: 0, # 'ฉ' - 34: 1, # 'ช' - 51: 0, # 'ซ' - 47: 0, # 'ญ' - 58: 0, # 'ฎ' - 57: 0, # 'ฏ' - 49: 0, # 'ฐ' - 53: 0, # 'ฑ' - 55: 0, # 'ฒ' - 43: 0, # 'ณ' - 20: 3, # 'ด' - 19: 2, # 'ต' - 44: 1, # 'ถ' - 14: 2, # 'ท' - 48: 1, # 'ธ' - 3: 3, # 'น' - 17: 3, # 'บ' - 25: 1, # 'ป' - 39: 1, # 'ผ' - 62: 0, # 'ฝ' - 31: 1, # 'พ' - 54: 1, # 'ฟ' - 45: 1, # 'ภ' - 9: 3, # 'ม' - 16: 3, # 'ย' - 2: 3, # 'ร' - 61: 0, # 'ฤ' - 15: 2, # 'ล' - 12: 2, # 'ว' - 42: 1, # 'ศ' - 46: 0, # 'ษ' - 18: 2, # 'ส' - 21: 2, # 'ห' - 4: 3, # 'อ' - 63: 0, # 'ฯ' - 22: 2, # 'ะ' - 10: 3, # 'ั' - 1: 3, # 'า' - 36: 2, # 'ำ' - 23: 2, # 'ิ' - 13: 3, # 'ี' - 40: 0, # 'ึ' - 27: 3, # 'ื' - 32: 3, # 'ุ' - 35: 0, # 'ู' - 11: 3, # 'เ' - 28: 1, # 'แ' - 41: 1, # 'โ' - 29: 2, # 'ใ' - 33: 2, # 'ไ' - 50: 1, # 'ๆ' - 37: 1, # '็' - 6: 2, # '่' - 7: 2, # '้' - 38: 0, # '์' - 56: 0, # '๑' - 59: 0, # '๒' - 60: 0, # '๕' - }, - 63: { # 'ฯ' - 5: 0, # 'ก' - 30: 0, # 'ข' - 24: 0, # 'ค' - 8: 0, # 'ง' - 26: 0, # 'จ' - 52: 0, # 'ฉ' - 34: 0, # 'ช' - 51: 0, # 'ซ' - 47: 0, # 'ญ' - 58: 0, # 'ฎ' - 57: 0, # 'ฏ' - 49: 0, # 'ฐ' - 53: 0, # 'ฑ' - 55: 0, # 'ฒ' - 43: 0, # 'ณ' - 20: 0, # 'ด' - 19: 0, # 'ต' - 44: 0, # 'ถ' - 14: 0, # 'ท' - 48: 0, # 'ธ' - 3: 0, # 'น' - 17: 0, # 'บ' - 25: 0, # 'ป' - 39: 0, # 'ผ' - 62: 0, # 'ฝ' - 31: 0, # 'พ' - 54: 0, # 'ฟ' - 45: 0, # 'ภ' - 9: 0, # 'ม' - 16: 0, # 'ย' - 2: 0, # 'ร' - 61: 0, # 'ฤ' - 15: 2, # 'ล' - 12: 0, # 'ว' - 42: 0, # 'ศ' - 46: 0, # 'ษ' - 18: 0, # 'ส' - 21: 0, # 'ห' - 4: 0, # 'อ' - 63: 0, # 'ฯ' - 22: 0, # 'ะ' - 10: 0, # 'ั' - 1: 0, # 'า' - 36: 0, # 'ำ' - 23: 0, # 'ิ' - 13: 0, # 'ี' - 40: 0, # 'ึ' - 27: 0, # 'ื' - 32: 0, # 'ุ' - 35: 0, # 'ู' - 11: 0, # 'เ' - 28: 0, # 'แ' - 41: 0, # 'โ' - 29: 0, # 'ใ' - 33: 0, # 'ไ' - 50: 0, # 'ๆ' - 37: 0, # '็' - 6: 0, # '่' - 7: 0, # '้' - 38: 0, # '์' - 56: 0, # '๑' - 59: 0, # '๒' - 60: 0, # '๕' - }, - 22: { # 'ะ' - 5: 3, # 'ก' - 30: 1, # 'ข' - 24: 2, # 'ค' - 8: 1, # 'ง' - 26: 2, # 'จ' - 52: 0, # 'ฉ' - 34: 3, # 'ช' - 51: 0, # 'ซ' - 47: 0, # 'ญ' - 58: 0, # 'ฎ' - 57: 0, # 'ฏ' - 49: 0, # 'ฐ' - 53: 0, # 'ฑ' - 55: 0, # 'ฒ' - 43: 0, # 'ณ' - 20: 3, # 'ด' - 19: 3, # 'ต' - 44: 1, # 'ถ' - 14: 3, # 'ท' - 48: 1, # 'ธ' - 3: 2, # 'น' - 17: 3, # 'บ' - 25: 2, # 'ป' - 39: 1, # 'ผ' - 62: 0, # 'ฝ' - 31: 2, # 'พ' - 54: 0, # 'ฟ' - 45: 1, # 'ภ' - 9: 3, # 'ม' - 16: 2, # 'ย' - 2: 2, # 'ร' - 61: 0, # 'ฤ' - 15: 2, # 'ล' - 12: 2, # 'ว' - 42: 0, # 'ศ' - 46: 0, # 'ษ' - 18: 3, # 'ส' - 21: 3, # 'ห' - 4: 2, # 'อ' - 63: 1, # 'ฯ' - 22: 1, # 'ะ' - 10: 0, # 'ั' - 1: 0, # 'า' - 36: 0, # 'ำ' - 23: 0, # 'ิ' - 13: 0, # 'ี' - 40: 0, # 'ึ' - 27: 0, # 'ื' - 32: 0, # 'ุ' - 35: 0, # 'ู' - 11: 3, # 'เ' - 28: 2, # 'แ' - 41: 1, # 'โ' - 29: 2, # 'ใ' - 33: 2, # 'ไ' - 50: 0, # 'ๆ' - 37: 0, # '็' - 6: 0, # '่' - 7: 0, # '้' - 38: 0, # '์' - 56: 0, # '๑' - 59: 0, # '๒' - 60: 0, # '๕' - }, - 10: { # 'ั' - 5: 3, # 'ก' - 30: 0, # 'ข' - 24: 1, # 'ค' - 8: 3, # 'ง' - 26: 3, # 'จ' - 52: 0, # 'ฉ' - 34: 1, # 'ช' - 51: 0, # 'ซ' - 47: 3, # 'ญ' - 58: 0, # 'ฎ' - 57: 0, # 'ฏ' - 49: 2, # 'ฐ' - 53: 0, # 'ฑ' - 55: 3, # 'ฒ' - 43: 3, # 'ณ' - 20: 3, # 'ด' - 19: 3, # 'ต' - 44: 0, # 'ถ' - 14: 2, # 'ท' - 48: 0, # 'ธ' - 3: 3, # 'น' - 17: 3, # 'บ' - 25: 1, # 'ป' - 39: 0, # 'ผ' - 62: 0, # 'ฝ' - 31: 2, # 'พ' - 54: 0, # 'ฟ' - 45: 0, # 'ภ' - 9: 3, # 'ม' - 16: 3, # 'ย' - 2: 0, # 'ร' - 61: 0, # 'ฤ' - 15: 2, # 'ล' - 12: 3, # 'ว' - 42: 2, # 'ศ' - 46: 0, # 'ษ' - 18: 3, # 'ส' - 21: 0, # 'ห' - 4: 0, # 'อ' - 63: 0, # 'ฯ' - 22: 0, # 'ะ' - 10: 0, # 'ั' - 1: 0, # 'า' - 36: 0, # 'ำ' - 23: 0, # 'ิ' - 13: 0, # 'ี' - 40: 0, # 'ึ' - 27: 0, # 'ื' - 32: 0, # 'ุ' - 35: 0, # 'ู' - 11: 0, # 'เ' - 28: 0, # 'แ' - 41: 0, # 'โ' - 29: 0, # 'ใ' - 33: 0, # 'ไ' - 50: 0, # 'ๆ' - 37: 0, # '็' - 6: 3, # '่' - 7: 3, # '้' - 38: 0, # '์' - 56: 0, # '๑' - 59: 0, # '๒' - 60: 0, # '๕' - }, - 1: { # 'า' - 5: 3, # 'ก' - 30: 2, # 'ข' - 24: 3, # 'ค' - 8: 3, # 'ง' - 26: 3, # 'จ' - 52: 0, # 'ฉ' - 34: 3, # 'ช' - 51: 1, # 'ซ' - 47: 2, # 'ญ' - 58: 0, # 'ฎ' - 57: 0, # 'ฏ' - 49: 0, # 'ฐ' - 53: 0, # 'ฑ' - 55: 0, # 'ฒ' - 43: 3, # 'ณ' - 20: 3, # 'ด' - 19: 3, # 'ต' - 44: 1, # 'ถ' - 14: 3, # 'ท' - 48: 2, # 'ธ' - 3: 3, # 'น' - 17: 3, # 'บ' - 25: 2, # 'ป' - 39: 1, # 'ผ' - 62: 1, # 'ฝ' - 31: 3, # 'พ' - 54: 1, # 'ฟ' - 45: 1, # 'ภ' - 9: 3, # 'ม' - 16: 3, # 'ย' - 2: 3, # 'ร' - 61: 0, # 'ฤ' - 15: 3, # 'ล' - 12: 3, # 'ว' - 42: 2, # 'ศ' - 46: 3, # 'ษ' - 18: 3, # 'ส' - 21: 3, # 'ห' - 4: 2, # 'อ' - 63: 1, # 'ฯ' - 22: 3, # 'ะ' - 10: 0, # 'ั' - 1: 0, # 'า' - 36: 0, # 'ำ' - 23: 0, # 'ิ' - 13: 0, # 'ี' - 40: 0, # 'ึ' - 27: 0, # 'ื' - 32: 0, # 'ุ' - 35: 0, # 'ู' - 11: 3, # 'เ' - 28: 2, # 'แ' - 41: 1, # 'โ' - 29: 2, # 'ใ' - 33: 2, # 'ไ' - 50: 1, # 'ๆ' - 37: 0, # '็' - 6: 0, # '่' - 7: 0, # '้' - 38: 0, # '์' - 56: 0, # '๑' - 59: 0, # '๒' - 60: 0, # '๕' - }, - 36: { # 'ำ' - 5: 2, # 'ก' - 30: 1, # 'ข' - 24: 3, # 'ค' - 8: 2, # 'ง' - 26: 1, # 'จ' - 52: 0, # 'ฉ' - 34: 0, # 'ช' - 51: 0, # 'ซ' - 47: 0, # 'ญ' - 58: 0, # 'ฎ' - 57: 0, # 'ฏ' - 49: 1, # 'ฐ' - 53: 0, # 'ฑ' - 55: 0, # 'ฒ' - 43: 0, # 'ณ' - 20: 1, # 'ด' - 19: 1, # 'ต' - 44: 1, # 'ถ' - 14: 1, # 'ท' - 48: 0, # 'ธ' - 3: 3, # 'น' - 17: 1, # 'บ' - 25: 1, # 'ป' - 39: 1, # 'ผ' - 62: 0, # 'ฝ' - 31: 1, # 'พ' - 54: 0, # 'ฟ' - 45: 1, # 'ภ' - 9: 1, # 'ม' - 16: 0, # 'ย' - 2: 2, # 'ร' - 61: 0, # 'ฤ' - 15: 2, # 'ล' - 12: 1, # 'ว' - 42: 0, # 'ศ' - 46: 0, # 'ษ' - 18: 1, # 'ส' - 21: 3, # 'ห' - 4: 1, # 'อ' - 63: 0, # 'ฯ' - 22: 0, # 'ะ' - 10: 0, # 'ั' - 1: 0, # 'า' - 36: 0, # 'ำ' - 23: 0, # 'ิ' - 13: 0, # 'ี' - 40: 0, # 'ึ' - 27: 0, # 'ื' - 32: 0, # 'ุ' - 35: 0, # 'ู' - 11: 3, # 'เ' - 28: 2, # 'แ' - 41: 1, # 'โ' - 29: 2, # 'ใ' - 33: 2, # 'ไ' - 50: 0, # 'ๆ' - 37: 0, # '็' - 6: 0, # '่' - 7: 0, # '้' - 38: 0, # '์' - 56: 0, # '๑' - 59: 0, # '๒' - 60: 0, # '๕' - }, - 23: { # 'ิ' - 5: 3, # 'ก' - 30: 1, # 'ข' - 24: 2, # 'ค' - 8: 3, # 'ง' - 26: 3, # 'จ' - 52: 0, # 'ฉ' - 34: 3, # 'ช' - 51: 0, # 'ซ' - 47: 2, # 'ญ' - 58: 0, # 'ฎ' - 57: 0, # 'ฏ' - 49: 0, # 'ฐ' - 53: 0, # 'ฑ' - 55: 0, # 'ฒ' - 43: 0, # 'ณ' - 20: 3, # 'ด' - 19: 3, # 'ต' - 44: 1, # 'ถ' - 14: 3, # 'ท' - 48: 3, # 'ธ' - 3: 3, # 'น' - 17: 3, # 'บ' - 25: 2, # 'ป' - 39: 2, # 'ผ' - 62: 0, # 'ฝ' - 31: 3, # 'พ' - 54: 1, # 'ฟ' - 45: 2, # 'ภ' - 9: 3, # 'ม' - 16: 2, # 'ย' - 2: 2, # 'ร' - 61: 0, # 'ฤ' - 15: 2, # 'ล' - 12: 3, # 'ว' - 42: 3, # 'ศ' - 46: 2, # 'ษ' - 18: 2, # 'ส' - 21: 3, # 'ห' - 4: 1, # 'อ' - 63: 1, # 'ฯ' - 22: 0, # 'ะ' - 10: 0, # 'ั' - 1: 0, # 'า' - 36: 0, # 'ำ' - 23: 0, # 'ิ' - 13: 0, # 'ี' - 40: 0, # 'ึ' - 27: 0, # 'ื' - 32: 0, # 'ุ' - 35: 0, # 'ู' - 11: 3, # 'เ' - 28: 1, # 'แ' - 41: 1, # 'โ' - 29: 1, # 'ใ' - 33: 0, # 'ไ' - 50: 0, # 'ๆ' - 37: 0, # '็' - 6: 3, # '่' - 7: 2, # '้' - 38: 2, # '์' - 56: 0, # '๑' - 59: 0, # '๒' - 60: 0, # '๕' - }, - 13: { # 'ี' - 5: 3, # 'ก' - 30: 2, # 'ข' - 24: 2, # 'ค' - 8: 0, # 'ง' - 26: 1, # 'จ' - 52: 0, # 'ฉ' - 34: 1, # 'ช' - 51: 0, # 'ซ' - 47: 0, # 'ญ' - 58: 0, # 'ฎ' - 57: 0, # 'ฏ' - 49: 0, # 'ฐ' - 53: 0, # 'ฑ' - 55: 0, # 'ฒ' - 43: 0, # 'ณ' - 20: 2, # 'ด' - 19: 1, # 'ต' - 44: 0, # 'ถ' - 14: 2, # 'ท' - 48: 0, # 'ธ' - 3: 1, # 'น' - 17: 2, # 'บ' - 25: 2, # 'ป' - 39: 1, # 'ผ' - 62: 0, # 'ฝ' - 31: 2, # 'พ' - 54: 0, # 'ฟ' - 45: 0, # 'ภ' - 9: 2, # 'ม' - 16: 3, # 'ย' - 2: 2, # 'ร' - 61: 0, # 'ฤ' - 15: 1, # 'ล' - 12: 2, # 'ว' - 42: 1, # 'ศ' - 46: 0, # 'ษ' - 18: 2, # 'ส' - 21: 1, # 'ห' - 4: 2, # 'อ' - 63: 0, # 'ฯ' - 22: 0, # 'ะ' - 10: 0, # 'ั' - 1: 0, # 'า' - 36: 0, # 'ำ' - 23: 0, # 'ิ' - 13: 0, # 'ี' - 40: 0, # 'ึ' - 27: 0, # 'ื' - 32: 0, # 'ุ' - 35: 0, # 'ู' - 11: 2, # 'เ' - 28: 2, # 'แ' - 41: 1, # 'โ' - 29: 1, # 'ใ' - 33: 1, # 'ไ' - 50: 1, # 'ๆ' - 37: 0, # '็' - 6: 3, # '่' - 7: 3, # '้' - 38: 0, # '์' - 56: 0, # '๑' - 59: 0, # '๒' - 60: 0, # '๕' - }, - 40: { # 'ึ' - 5: 3, # 'ก' - 30: 0, # 'ข' - 24: 0, # 'ค' - 8: 3, # 'ง' - 26: 0, # 'จ' - 52: 0, # 'ฉ' - 34: 0, # 'ช' - 51: 0, # 'ซ' - 47: 0, # 'ญ' - 58: 0, # 'ฎ' - 57: 0, # 'ฏ' - 49: 0, # 'ฐ' - 53: 0, # 'ฑ' - 55: 0, # 'ฒ' - 43: 0, # 'ณ' - 20: 1, # 'ด' - 19: 0, # 'ต' - 44: 0, # 'ถ' - 14: 0, # 'ท' - 48: 0, # 'ธ' - 3: 0, # 'น' - 17: 0, # 'บ' - 25: 0, # 'ป' - 39: 0, # 'ผ' - 62: 0, # 'ฝ' - 31: 0, # 'พ' - 54: 0, # 'ฟ' - 45: 0, # 'ภ' - 9: 1, # 'ม' - 16: 0, # 'ย' - 2: 0, # 'ร' - 61: 0, # 'ฤ' - 15: 0, # 'ล' - 12: 0, # 'ว' - 42: 0, # 'ศ' - 46: 0, # 'ษ' - 18: 0, # 'ส' - 21: 0, # 'ห' - 4: 0, # 'อ' - 63: 0, # 'ฯ' - 22: 0, # 'ะ' - 10: 0, # 'ั' - 1: 0, # 'า' - 36: 0, # 'ำ' - 23: 0, # 'ิ' - 13: 0, # 'ี' - 40: 0, # 'ึ' - 27: 0, # 'ื' - 32: 0, # 'ุ' - 35: 0, # 'ู' - 11: 0, # 'เ' - 28: 0, # 'แ' - 41: 0, # 'โ' - 29: 0, # 'ใ' - 33: 0, # 'ไ' - 50: 0, # 'ๆ' - 37: 0, # '็' - 6: 3, # '่' - 7: 3, # '้' - 38: 0, # '์' - 56: 0, # '๑' - 59: 0, # '๒' - 60: 0, # '๕' - }, - 27: { # 'ื' - 5: 0, # 'ก' - 30: 0, # 'ข' - 24: 0, # 'ค' - 8: 0, # 'ง' - 26: 0, # 'จ' - 52: 0, # 'ฉ' - 34: 1, # 'ช' - 51: 0, # 'ซ' - 47: 0, # 'ญ' - 58: 0, # 'ฎ' - 57: 0, # 'ฏ' - 49: 0, # 'ฐ' - 53: 0, # 'ฑ' - 55: 0, # 'ฒ' - 43: 0, # 'ณ' - 20: 1, # 'ด' - 19: 0, # 'ต' - 44: 0, # 'ถ' - 14: 0, # 'ท' - 48: 0, # 'ธ' - 3: 2, # 'น' - 17: 3, # 'บ' - 25: 0, # 'ป' - 39: 0, # 'ผ' - 62: 0, # 'ฝ' - 31: 0, # 'พ' - 54: 0, # 'ฟ' - 45: 0, # 'ภ' - 9: 2, # 'ม' - 16: 0, # 'ย' - 2: 0, # 'ร' - 61: 0, # 'ฤ' - 15: 0, # 'ล' - 12: 0, # 'ว' - 42: 0, # 'ศ' - 46: 0, # 'ษ' - 18: 0, # 'ส' - 21: 0, # 'ห' - 4: 3, # 'อ' - 63: 0, # 'ฯ' - 22: 0, # 'ะ' - 10: 0, # 'ั' - 1: 0, # 'า' - 36: 0, # 'ำ' - 23: 0, # 'ิ' - 13: 0, # 'ี' - 40: 0, # 'ึ' - 27: 0, # 'ื' - 32: 0, # 'ุ' - 35: 0, # 'ู' - 11: 0, # 'เ' - 28: 0, # 'แ' - 41: 0, # 'โ' - 29: 0, # 'ใ' - 33: 0, # 'ไ' - 50: 0, # 'ๆ' - 37: 0, # '็' - 6: 3, # '่' - 7: 3, # '้' - 38: 0, # '์' - 56: 0, # '๑' - 59: 0, # '๒' - 60: 0, # '๕' - }, - 32: { # 'ุ' - 5: 3, # 'ก' - 30: 2, # 'ข' - 24: 3, # 'ค' - 8: 3, # 'ง' - 26: 0, # 'จ' - 52: 0, # 'ฉ' - 34: 0, # 'ช' - 51: 0, # 'ซ' - 47: 2, # 'ญ' - 58: 0, # 'ฎ' - 57: 0, # 'ฏ' - 49: 0, # 'ฐ' - 53: 0, # 'ฑ' - 55: 1, # 'ฒ' - 43: 3, # 'ณ' - 20: 3, # 'ด' - 19: 3, # 'ต' - 44: 1, # 'ถ' - 14: 2, # 'ท' - 48: 1, # 'ธ' - 3: 2, # 'น' - 17: 2, # 'บ' - 25: 2, # 'ป' - 39: 2, # 'ผ' - 62: 0, # 'ฝ' - 31: 1, # 'พ' - 54: 0, # 'ฟ' - 45: 1, # 'ภ' - 9: 3, # 'ม' - 16: 1, # 'ย' - 2: 2, # 'ร' - 61: 0, # 'ฤ' - 15: 2, # 'ล' - 12: 1, # 'ว' - 42: 1, # 'ศ' - 46: 2, # 'ษ' - 18: 1, # 'ส' - 21: 1, # 'ห' - 4: 1, # 'อ' - 63: 0, # 'ฯ' - 22: 0, # 'ะ' - 10: 0, # 'ั' - 1: 0, # 'า' - 36: 0, # 'ำ' - 23: 0, # 'ิ' - 13: 0, # 'ี' - 40: 0, # 'ึ' - 27: 0, # 'ื' - 32: 0, # 'ุ' - 35: 0, # 'ู' - 11: 1, # 'เ' - 28: 0, # 'แ' - 41: 1, # 'โ' - 29: 0, # 'ใ' - 33: 1, # 'ไ' - 50: 0, # 'ๆ' - 37: 0, # '็' - 6: 3, # '่' - 7: 2, # '้' - 38: 1, # '์' - 56: 0, # '๑' - 59: 0, # '๒' - 60: 0, # '๕' - }, - 35: { # 'ู' - 5: 3, # 'ก' - 30: 0, # 'ข' - 24: 0, # 'ค' - 8: 2, # 'ง' - 26: 1, # 'จ' - 52: 0, # 'ฉ' - 34: 0, # 'ช' - 51: 0, # 'ซ' - 47: 2, # 'ญ' - 58: 0, # 'ฎ' - 57: 0, # 'ฏ' - 49: 0, # 'ฐ' - 53: 0, # 'ฑ' - 55: 0, # 'ฒ' - 43: 1, # 'ณ' - 20: 2, # 'ด' - 19: 2, # 'ต' - 44: 0, # 'ถ' - 14: 1, # 'ท' - 48: 0, # 'ธ' - 3: 2, # 'น' - 17: 0, # 'บ' - 25: 3, # 'ป' - 39: 0, # 'ผ' - 62: 0, # 'ฝ' - 31: 0, # 'พ' - 54: 0, # 'ฟ' - 45: 0, # 'ภ' - 9: 2, # 'ม' - 16: 0, # 'ย' - 2: 1, # 'ร' - 61: 0, # 'ฤ' - 15: 3, # 'ล' - 12: 1, # 'ว' - 42: 0, # 'ศ' - 46: 0, # 'ษ' - 18: 0, # 'ส' - 21: 0, # 'ห' - 4: 0, # 'อ' - 63: 0, # 'ฯ' - 22: 0, # 'ะ' - 10: 0, # 'ั' - 1: 0, # 'า' - 36: 0, # 'ำ' - 23: 0, # 'ิ' - 13: 0, # 'ี' - 40: 0, # 'ึ' - 27: 0, # 'ื' - 32: 0, # 'ุ' - 35: 0, # 'ู' - 11: 1, # 'เ' - 28: 1, # 'แ' - 41: 1, # 'โ' - 29: 0, # 'ใ' - 33: 0, # 'ไ' - 50: 0, # 'ๆ' - 37: 0, # '็' - 6: 3, # '่' - 7: 3, # '้' - 38: 0, # '์' - 56: 0, # '๑' - 59: 0, # '๒' - 60: 0, # '๕' - }, - 11: { # 'เ' - 5: 3, # 'ก' - 30: 3, # 'ข' - 24: 3, # 'ค' - 8: 2, # 'ง' - 26: 3, # 'จ' - 52: 3, # 'ฉ' - 34: 3, # 'ช' - 51: 2, # 'ซ' - 47: 0, # 'ญ' - 58: 0, # 'ฎ' - 57: 0, # 'ฏ' - 49: 0, # 'ฐ' - 53: 0, # 'ฑ' - 55: 0, # 'ฒ' - 43: 1, # 'ณ' - 20: 3, # 'ด' - 19: 3, # 'ต' - 44: 1, # 'ถ' - 14: 3, # 'ท' - 48: 1, # 'ธ' - 3: 3, # 'น' - 17: 3, # 'บ' - 25: 3, # 'ป' - 39: 2, # 'ผ' - 62: 1, # 'ฝ' - 31: 3, # 'พ' - 54: 1, # 'ฟ' - 45: 3, # 'ภ' - 9: 3, # 'ม' - 16: 2, # 'ย' - 2: 3, # 'ร' - 61: 0, # 'ฤ' - 15: 3, # 'ล' - 12: 3, # 'ว' - 42: 2, # 'ศ' - 46: 0, # 'ษ' - 18: 3, # 'ส' - 21: 3, # 'ห' - 4: 3, # 'อ' - 63: 0, # 'ฯ' - 22: 0, # 'ะ' - 10: 0, # 'ั' - 1: 0, # 'า' - 36: 0, # 'ำ' - 23: 0, # 'ิ' - 13: 0, # 'ี' - 40: 0, # 'ึ' - 27: 0, # 'ื' - 32: 0, # 'ุ' - 35: 0, # 'ู' - 11: 0, # 'เ' - 28: 0, # 'แ' - 41: 0, # 'โ' - 29: 0, # 'ใ' - 33: 0, # 'ไ' - 50: 0, # 'ๆ' - 37: 0, # '็' - 6: 0, # '่' - 7: 0, # '้' - 38: 0, # '์' - 56: 0, # '๑' - 59: 0, # '๒' - 60: 0, # '๕' - }, - 28: { # 'แ' - 5: 3, # 'ก' - 30: 2, # 'ข' - 24: 2, # 'ค' - 8: 1, # 'ง' - 26: 2, # 'จ' - 52: 0, # 'ฉ' - 34: 1, # 'ช' - 51: 0, # 'ซ' - 47: 0, # 'ญ' - 58: 0, # 'ฎ' - 57: 0, # 'ฏ' - 49: 0, # 'ฐ' - 53: 0, # 'ฑ' - 55: 0, # 'ฒ' - 43: 0, # 'ณ' - 20: 2, # 'ด' - 19: 3, # 'ต' - 44: 2, # 'ถ' - 14: 3, # 'ท' - 48: 0, # 'ธ' - 3: 3, # 'น' - 17: 3, # 'บ' - 25: 2, # 'ป' - 39: 3, # 'ผ' - 62: 0, # 'ฝ' - 31: 2, # 'พ' - 54: 2, # 'ฟ' - 45: 0, # 'ภ' - 9: 2, # 'ม' - 16: 2, # 'ย' - 2: 2, # 'ร' - 61: 0, # 'ฤ' - 15: 3, # 'ล' - 12: 2, # 'ว' - 42: 0, # 'ศ' - 46: 0, # 'ษ' - 18: 3, # 'ส' - 21: 3, # 'ห' - 4: 1, # 'อ' - 63: 0, # 'ฯ' - 22: 0, # 'ะ' - 10: 0, # 'ั' - 1: 0, # 'า' - 36: 0, # 'ำ' - 23: 0, # 'ิ' - 13: 0, # 'ี' - 40: 0, # 'ึ' - 27: 0, # 'ื' - 32: 0, # 'ุ' - 35: 0, # 'ู' - 11: 0, # 'เ' - 28: 0, # 'แ' - 41: 0, # 'โ' - 29: 0, # 'ใ' - 33: 0, # 'ไ' - 50: 0, # 'ๆ' - 37: 0, # '็' - 6: 0, # '่' - 7: 0, # '้' - 38: 0, # '์' - 56: 0, # '๑' - 59: 0, # '๒' - 60: 0, # '๕' - }, - 41: { # 'โ' - 5: 2, # 'ก' - 30: 1, # 'ข' - 24: 2, # 'ค' - 8: 0, # 'ง' - 26: 1, # 'จ' - 52: 1, # 'ฉ' - 34: 1, # 'ช' - 51: 1, # 'ซ' - 47: 0, # 'ญ' - 58: 0, # 'ฎ' - 57: 0, # 'ฏ' - 49: 0, # 'ฐ' - 53: 0, # 'ฑ' - 55: 0, # 'ฒ' - 43: 0, # 'ณ' - 20: 3, # 'ด' - 19: 2, # 'ต' - 44: 0, # 'ถ' - 14: 2, # 'ท' - 48: 0, # 'ธ' - 3: 3, # 'น' - 17: 1, # 'บ' - 25: 3, # 'ป' - 39: 0, # 'ผ' - 62: 0, # 'ฝ' - 31: 1, # 'พ' - 54: 1, # 'ฟ' - 45: 1, # 'ภ' - 9: 1, # 'ม' - 16: 2, # 'ย' - 2: 2, # 'ร' - 61: 0, # 'ฤ' - 15: 3, # 'ล' - 12: 0, # 'ว' - 42: 1, # 'ศ' - 46: 0, # 'ษ' - 18: 2, # 'ส' - 21: 0, # 'ห' - 4: 2, # 'อ' - 63: 0, # 'ฯ' - 22: 0, # 'ะ' - 10: 0, # 'ั' - 1: 0, # 'า' - 36: 0, # 'ำ' - 23: 0, # 'ิ' - 13: 0, # 'ี' - 40: 0, # 'ึ' - 27: 0, # 'ื' - 32: 0, # 'ุ' - 35: 0, # 'ู' - 11: 0, # 'เ' - 28: 0, # 'แ' - 41: 0, # 'โ' - 29: 0, # 'ใ' - 33: 0, # 'ไ' - 50: 0, # 'ๆ' - 37: 0, # '็' - 6: 0, # '่' - 7: 0, # '้' - 38: 0, # '์' - 56: 0, # '๑' - 59: 0, # '๒' - 60: 0, # '๕' - }, - 29: { # 'ใ' - 5: 2, # 'ก' - 30: 0, # 'ข' - 24: 1, # 'ค' - 8: 0, # 'ง' - 26: 3, # 'จ' - 52: 0, # 'ฉ' - 34: 3, # 'ช' - 51: 0, # 'ซ' - 47: 0, # 'ญ' - 58: 0, # 'ฎ' - 57: 0, # 'ฏ' - 49: 0, # 'ฐ' - 53: 0, # 'ฑ' - 55: 0, # 'ฒ' - 43: 0, # 'ณ' - 20: 3, # 'ด' - 19: 1, # 'ต' - 44: 0, # 'ถ' - 14: 0, # 'ท' - 48: 0, # 'ธ' - 3: 3, # 'น' - 17: 2, # 'บ' - 25: 0, # 'ป' - 39: 0, # 'ผ' - 62: 0, # 'ฝ' - 31: 0, # 'พ' - 54: 0, # 'ฟ' - 45: 0, # 'ภ' - 9: 0, # 'ม' - 16: 1, # 'ย' - 2: 0, # 'ร' - 61: 0, # 'ฤ' - 15: 0, # 'ล' - 12: 0, # 'ว' - 42: 0, # 'ศ' - 46: 0, # 'ษ' - 18: 3, # 'ส' - 21: 3, # 'ห' - 4: 0, # 'อ' - 63: 0, # 'ฯ' - 22: 0, # 'ะ' - 10: 0, # 'ั' - 1: 0, # 'า' - 36: 0, # 'ำ' - 23: 0, # 'ิ' - 13: 0, # 'ี' - 40: 0, # 'ึ' - 27: 0, # 'ื' - 32: 0, # 'ุ' - 35: 0, # 'ู' - 11: 0, # 'เ' - 28: 0, # 'แ' - 41: 0, # 'โ' - 29: 0, # 'ใ' - 33: 0, # 'ไ' - 50: 0, # 'ๆ' - 37: 0, # '็' - 6: 0, # '่' - 7: 0, # '้' - 38: 0, # '์' - 56: 0, # '๑' - 59: 0, # '๒' - 60: 0, # '๕' - }, - 33: { # 'ไ' - 5: 1, # 'ก' - 30: 2, # 'ข' - 24: 0, # 'ค' - 8: 0, # 'ง' - 26: 0, # 'จ' - 52: 0, # 'ฉ' - 34: 1, # 'ช' - 51: 1, # 'ซ' - 47: 0, # 'ญ' - 58: 0, # 'ฎ' - 57: 0, # 'ฏ' - 49: 0, # 'ฐ' - 53: 0, # 'ฑ' - 55: 0, # 'ฒ' - 43: 0, # 'ณ' - 20: 3, # 'ด' - 19: 1, # 'ต' - 44: 0, # 'ถ' - 14: 3, # 'ท' - 48: 0, # 'ธ' - 3: 0, # 'น' - 17: 1, # 'บ' - 25: 3, # 'ป' - 39: 0, # 'ผ' - 62: 0, # 'ฝ' - 31: 0, # 'พ' - 54: 2, # 'ฟ' - 45: 0, # 'ภ' - 9: 3, # 'ม' - 16: 0, # 'ย' - 2: 3, # 'ร' - 61: 0, # 'ฤ' - 15: 1, # 'ล' - 12: 3, # 'ว' - 42: 0, # 'ศ' - 46: 0, # 'ษ' - 18: 1, # 'ส' - 21: 2, # 'ห' - 4: 0, # 'อ' - 63: 0, # 'ฯ' - 22: 0, # 'ะ' - 10: 0, # 'ั' - 1: 0, # 'า' - 36: 0, # 'ำ' - 23: 0, # 'ิ' - 13: 0, # 'ี' - 40: 0, # 'ึ' - 27: 0, # 'ื' - 32: 0, # 'ุ' - 35: 0, # 'ู' - 11: 0, # 'เ' - 28: 0, # 'แ' - 41: 0, # 'โ' - 29: 0, # 'ใ' - 33: 0, # 'ไ' - 50: 0, # 'ๆ' - 37: 0, # '็' - 6: 0, # '่' - 7: 0, # '้' - 38: 0, # '์' - 56: 0, # '๑' - 59: 0, # '๒' - 60: 0, # '๕' - }, - 50: { # 'ๆ' - 5: 0, # 'ก' - 30: 0, # 'ข' - 24: 0, # 'ค' - 8: 0, # 'ง' - 26: 0, # 'จ' - 52: 0, # 'ฉ' - 34: 0, # 'ช' - 51: 0, # 'ซ' - 47: 0, # 'ญ' - 58: 0, # 'ฎ' - 57: 0, # 'ฏ' - 49: 0, # 'ฐ' - 53: 0, # 'ฑ' - 55: 0, # 'ฒ' - 43: 0, # 'ณ' - 20: 0, # 'ด' - 19: 0, # 'ต' - 44: 0, # 'ถ' - 14: 0, # 'ท' - 48: 0, # 'ธ' - 3: 0, # 'น' - 17: 0, # 'บ' - 25: 0, # 'ป' - 39: 0, # 'ผ' - 62: 0, # 'ฝ' - 31: 0, # 'พ' - 54: 0, # 'ฟ' - 45: 0, # 'ภ' - 9: 0, # 'ม' - 16: 0, # 'ย' - 2: 0, # 'ร' - 61: 0, # 'ฤ' - 15: 0, # 'ล' - 12: 0, # 'ว' - 42: 0, # 'ศ' - 46: 0, # 'ษ' - 18: 0, # 'ส' - 21: 0, # 'ห' - 4: 0, # 'อ' - 63: 0, # 'ฯ' - 22: 0, # 'ะ' - 10: 0, # 'ั' - 1: 0, # 'า' - 36: 0, # 'ำ' - 23: 0, # 'ิ' - 13: 0, # 'ี' - 40: 0, # 'ึ' - 27: 0, # 'ื' - 32: 0, # 'ุ' - 35: 0, # 'ู' - 11: 0, # 'เ' - 28: 0, # 'แ' - 41: 0, # 'โ' - 29: 0, # 'ใ' - 33: 0, # 'ไ' - 50: 0, # 'ๆ' - 37: 0, # '็' - 6: 0, # '่' - 7: 0, # '้' - 38: 0, # '์' - 56: 0, # '๑' - 59: 0, # '๒' - 60: 0, # '๕' - }, - 37: { # '็' - 5: 2, # 'ก' - 30: 1, # 'ข' - 24: 2, # 'ค' - 8: 2, # 'ง' - 26: 3, # 'จ' - 52: 0, # 'ฉ' - 34: 0, # 'ช' - 51: 0, # 'ซ' - 47: 1, # 'ญ' - 58: 0, # 'ฎ' - 57: 0, # 'ฏ' - 49: 0, # 'ฐ' - 53: 0, # 'ฑ' - 55: 0, # 'ฒ' - 43: 0, # 'ณ' - 20: 1, # 'ด' - 19: 2, # 'ต' - 44: 0, # 'ถ' - 14: 1, # 'ท' - 48: 0, # 'ธ' - 3: 3, # 'น' - 17: 3, # 'บ' - 25: 0, # 'ป' - 39: 0, # 'ผ' - 62: 0, # 'ฝ' - 31: 0, # 'พ' - 54: 0, # 'ฟ' - 45: 0, # 'ภ' - 9: 2, # 'ม' - 16: 1, # 'ย' - 2: 0, # 'ร' - 61: 0, # 'ฤ' - 15: 0, # 'ล' - 12: 2, # 'ว' - 42: 0, # 'ศ' - 46: 0, # 'ษ' - 18: 1, # 'ส' - 21: 0, # 'ห' - 4: 1, # 'อ' - 63: 0, # 'ฯ' - 22: 0, # 'ะ' - 10: 0, # 'ั' - 1: 0, # 'า' - 36: 0, # 'ำ' - 23: 0, # 'ิ' - 13: 0, # 'ี' - 40: 0, # 'ึ' - 27: 0, # 'ื' - 32: 0, # 'ุ' - 35: 0, # 'ู' - 11: 1, # 'เ' - 28: 0, # 'แ' - 41: 0, # 'โ' - 29: 0, # 'ใ' - 33: 1, # 'ไ' - 50: 0, # 'ๆ' - 37: 0, # '็' - 6: 0, # '่' - 7: 0, # '้' - 38: 0, # '์' - 56: 0, # '๑' - 59: 0, # '๒' - 60: 0, # '๕' - }, - 6: { # '่' - 5: 2, # 'ก' - 30: 1, # 'ข' - 24: 2, # 'ค' - 8: 3, # 'ง' - 26: 2, # 'จ' - 52: 0, # 'ฉ' - 34: 1, # 'ช' - 51: 1, # 'ซ' - 47: 0, # 'ญ' - 58: 0, # 'ฎ' - 57: 0, # 'ฏ' - 49: 1, # 'ฐ' - 53: 0, # 'ฑ' - 55: 0, # 'ฒ' - 43: 0, # 'ณ' - 20: 1, # 'ด' - 19: 2, # 'ต' - 44: 1, # 'ถ' - 14: 2, # 'ท' - 48: 1, # 'ธ' - 3: 3, # 'น' - 17: 1, # 'บ' - 25: 2, # 'ป' - 39: 2, # 'ผ' - 62: 1, # 'ฝ' - 31: 1, # 'พ' - 54: 0, # 'ฟ' - 45: 0, # 'ภ' - 9: 3, # 'ม' - 16: 3, # 'ย' - 2: 2, # 'ร' - 61: 0, # 'ฤ' - 15: 2, # 'ล' - 12: 3, # 'ว' - 42: 0, # 'ศ' - 46: 0, # 'ษ' - 18: 2, # 'ส' - 21: 1, # 'ห' - 4: 3, # 'อ' - 63: 0, # 'ฯ' - 22: 1, # 'ะ' - 10: 0, # 'ั' - 1: 3, # 'า' - 36: 2, # 'ำ' - 23: 0, # 'ิ' - 13: 0, # 'ี' - 40: 0, # 'ึ' - 27: 0, # 'ื' - 32: 0, # 'ุ' - 35: 0, # 'ู' - 11: 3, # 'เ' - 28: 2, # 'แ' - 41: 1, # 'โ' - 29: 2, # 'ใ' - 33: 2, # 'ไ' - 50: 1, # 'ๆ' - 37: 0, # '็' - 6: 0, # '่' - 7: 0, # '้' - 38: 0, # '์' - 56: 0, # '๑' - 59: 0, # '๒' - 60: 0, # '๕' - }, - 7: { # '้' - 5: 2, # 'ก' - 30: 1, # 'ข' - 24: 2, # 'ค' - 8: 3, # 'ง' - 26: 2, # 'จ' - 52: 0, # 'ฉ' - 34: 1, # 'ช' - 51: 1, # 'ซ' - 47: 0, # 'ญ' - 58: 0, # 'ฎ' - 57: 0, # 'ฏ' - 49: 0, # 'ฐ' - 53: 0, # 'ฑ' - 55: 0, # 'ฒ' - 43: 0, # 'ณ' - 20: 1, # 'ด' - 19: 2, # 'ต' - 44: 1, # 'ถ' - 14: 2, # 'ท' - 48: 0, # 'ธ' - 3: 3, # 'น' - 17: 2, # 'บ' - 25: 2, # 'ป' - 39: 2, # 'ผ' - 62: 0, # 'ฝ' - 31: 1, # 'พ' - 54: 1, # 'ฟ' - 45: 0, # 'ภ' - 9: 3, # 'ม' - 16: 2, # 'ย' - 2: 2, # 'ร' - 61: 0, # 'ฤ' - 15: 1, # 'ล' - 12: 3, # 'ว' - 42: 1, # 'ศ' - 46: 0, # 'ษ' - 18: 2, # 'ส' - 21: 2, # 'ห' - 4: 3, # 'อ' - 63: 0, # 'ฯ' - 22: 0, # 'ะ' - 10: 0, # 'ั' - 1: 3, # 'า' - 36: 2, # 'ำ' - 23: 0, # 'ิ' - 13: 0, # 'ี' - 40: 0, # 'ึ' - 27: 0, # 'ื' - 32: 0, # 'ุ' - 35: 0, # 'ู' - 11: 2, # 'เ' - 28: 2, # 'แ' - 41: 1, # 'โ' - 29: 2, # 'ใ' - 33: 2, # 'ไ' - 50: 0, # 'ๆ' - 37: 0, # '็' - 6: 0, # '่' - 7: 0, # '้' - 38: 0, # '์' - 56: 0, # '๑' - 59: 0, # '๒' - 60: 0, # '๕' - }, - 38: { # '์' - 5: 2, # 'ก' - 30: 1, # 'ข' - 24: 1, # 'ค' - 8: 0, # 'ง' - 26: 1, # 'จ' - 52: 0, # 'ฉ' - 34: 1, # 'ช' - 51: 0, # 'ซ' - 47: 0, # 'ญ' - 58: 0, # 'ฎ' - 57: 0, # 'ฏ' - 49: 0, # 'ฐ' - 53: 0, # 'ฑ' - 55: 0, # 'ฒ' - 43: 0, # 'ณ' - 20: 2, # 'ด' - 19: 1, # 'ต' - 44: 1, # 'ถ' - 14: 1, # 'ท' - 48: 0, # 'ธ' - 3: 1, # 'น' - 17: 1, # 'บ' - 25: 1, # 'ป' - 39: 0, # 'ผ' - 62: 0, # 'ฝ' - 31: 1, # 'พ' - 54: 1, # 'ฟ' - 45: 0, # 'ภ' - 9: 2, # 'ม' - 16: 0, # 'ย' - 2: 1, # 'ร' - 61: 1, # 'ฤ' - 15: 1, # 'ล' - 12: 1, # 'ว' - 42: 0, # 'ศ' - 46: 0, # 'ษ' - 18: 1, # 'ส' - 21: 1, # 'ห' - 4: 2, # 'อ' - 63: 1, # 'ฯ' - 22: 0, # 'ะ' - 10: 0, # 'ั' - 1: 0, # 'า' - 36: 0, # 'ำ' - 23: 0, # 'ิ' - 13: 0, # 'ี' - 40: 0, # 'ึ' - 27: 0, # 'ื' - 32: 0, # 'ุ' - 35: 0, # 'ู' - 11: 2, # 'เ' - 28: 2, # 'แ' - 41: 1, # 'โ' - 29: 1, # 'ใ' - 33: 1, # 'ไ' - 50: 0, # 'ๆ' - 37: 0, # '็' - 6: 0, # '่' - 7: 0, # '้' - 38: 0, # '์' - 56: 0, # '๑' - 59: 0, # '๒' - 60: 0, # '๕' - }, - 56: { # '๑' - 5: 0, # 'ก' - 30: 0, # 'ข' - 24: 0, # 'ค' - 8: 0, # 'ง' - 26: 0, # 'จ' - 52: 0, # 'ฉ' - 34: 0, # 'ช' - 51: 0, # 'ซ' - 47: 0, # 'ญ' - 58: 0, # 'ฎ' - 57: 0, # 'ฏ' - 49: 0, # 'ฐ' - 53: 0, # 'ฑ' - 55: 0, # 'ฒ' - 43: 0, # 'ณ' - 20: 0, # 'ด' - 19: 0, # 'ต' - 44: 0, # 'ถ' - 14: 0, # 'ท' - 48: 0, # 'ธ' - 3: 0, # 'น' - 17: 0, # 'บ' - 25: 0, # 'ป' - 39: 0, # 'ผ' - 62: 0, # 'ฝ' - 31: 0, # 'พ' - 54: 0, # 'ฟ' - 45: 0, # 'ภ' - 9: 0, # 'ม' - 16: 0, # 'ย' - 2: 0, # 'ร' - 61: 0, # 'ฤ' - 15: 0, # 'ล' - 12: 0, # 'ว' - 42: 0, # 'ศ' - 46: 0, # 'ษ' - 18: 0, # 'ส' - 21: 0, # 'ห' - 4: 0, # 'อ' - 63: 0, # 'ฯ' - 22: 0, # 'ะ' - 10: 0, # 'ั' - 1: 0, # 'า' - 36: 0, # 'ำ' - 23: 0, # 'ิ' - 13: 0, # 'ี' - 40: 0, # 'ึ' - 27: 0, # 'ื' - 32: 0, # 'ุ' - 35: 0, # 'ู' - 11: 0, # 'เ' - 28: 0, # 'แ' - 41: 0, # 'โ' - 29: 0, # 'ใ' - 33: 0, # 'ไ' - 50: 0, # 'ๆ' - 37: 0, # '็' - 6: 0, # '่' - 7: 0, # '้' - 38: 0, # '์' - 56: 2, # '๑' - 59: 1, # '๒' - 60: 1, # '๕' - }, - 59: { # '๒' - 5: 0, # 'ก' - 30: 0, # 'ข' - 24: 0, # 'ค' - 8: 0, # 'ง' - 26: 0, # 'จ' - 52: 0, # 'ฉ' - 34: 0, # 'ช' - 51: 0, # 'ซ' - 47: 0, # 'ญ' - 58: 0, # 'ฎ' - 57: 0, # 'ฏ' - 49: 0, # 'ฐ' - 53: 0, # 'ฑ' - 55: 0, # 'ฒ' - 43: 0, # 'ณ' - 20: 0, # 'ด' - 19: 0, # 'ต' - 44: 0, # 'ถ' - 14: 0, # 'ท' - 48: 0, # 'ธ' - 3: 0, # 'น' - 17: 0, # 'บ' - 25: 0, # 'ป' - 39: 0, # 'ผ' - 62: 0, # 'ฝ' - 31: 0, # 'พ' - 54: 0, # 'ฟ' - 45: 0, # 'ภ' - 9: 0, # 'ม' - 16: 0, # 'ย' - 2: 0, # 'ร' - 61: 0, # 'ฤ' - 15: 0, # 'ล' - 12: 0, # 'ว' - 42: 0, # 'ศ' - 46: 0, # 'ษ' - 18: 0, # 'ส' - 21: 0, # 'ห' - 4: 0, # 'อ' - 63: 0, # 'ฯ' - 22: 0, # 'ะ' - 10: 0, # 'ั' - 1: 0, # 'า' - 36: 0, # 'ำ' - 23: 0, # 'ิ' - 13: 0, # 'ี' - 40: 0, # 'ึ' - 27: 0, # 'ื' - 32: 0, # 'ุ' - 35: 0, # 'ู' - 11: 0, # 'เ' - 28: 0, # 'แ' - 41: 0, # 'โ' - 29: 0, # 'ใ' - 33: 0, # 'ไ' - 50: 0, # 'ๆ' - 37: 0, # '็' - 6: 0, # '่' - 7: 0, # '้' - 38: 0, # '์' - 56: 1, # '๑' - 59: 1, # '๒' - 60: 3, # '๕' - }, - 60: { # '๕' - 5: 0, # 'ก' - 30: 0, # 'ข' - 24: 0, # 'ค' - 8: 0, # 'ง' - 26: 0, # 'จ' - 52: 0, # 'ฉ' - 34: 0, # 'ช' - 51: 0, # 'ซ' - 47: 0, # 'ญ' - 58: 0, # 'ฎ' - 57: 0, # 'ฏ' - 49: 0, # 'ฐ' - 53: 0, # 'ฑ' - 55: 0, # 'ฒ' - 43: 0, # 'ณ' - 20: 0, # 'ด' - 19: 0, # 'ต' - 44: 0, # 'ถ' - 14: 0, # 'ท' - 48: 0, # 'ธ' - 3: 0, # 'น' - 17: 0, # 'บ' - 25: 0, # 'ป' - 39: 0, # 'ผ' - 62: 0, # 'ฝ' - 31: 0, # 'พ' - 54: 0, # 'ฟ' - 45: 0, # 'ภ' - 9: 0, # 'ม' - 16: 0, # 'ย' - 2: 0, # 'ร' - 61: 0, # 'ฤ' - 15: 0, # 'ล' - 12: 0, # 'ว' - 42: 0, # 'ศ' - 46: 0, # 'ษ' - 18: 0, # 'ส' - 21: 0, # 'ห' - 4: 0, # 'อ' - 63: 0, # 'ฯ' - 22: 0, # 'ะ' - 10: 0, # 'ั' - 1: 0, # 'า' - 36: 0, # 'ำ' - 23: 0, # 'ิ' - 13: 0, # 'ี' - 40: 0, # 'ึ' - 27: 0, # 'ื' - 32: 0, # 'ุ' - 35: 0, # 'ู' - 11: 0, # 'เ' - 28: 0, # 'แ' - 41: 0, # 'โ' - 29: 0, # 'ใ' - 33: 0, # 'ไ' - 50: 0, # 'ๆ' - 37: 0, # '็' - 6: 0, # '่' - 7: 0, # '้' - 38: 0, # '์' - 56: 2, # '๑' - 59: 1, # '๒' - 60: 0, # '๕' - }, -} - -# 255: Undefined characters that did not exist in training text -# 254: Carriage/Return -# 253: symbol (punctuation) that does not belong to word -# 252: 0 - 9 -# 251: Control characters - -# Character Mapping Table(s): -TIS_620_THAI_CHAR_TO_ORDER = { - 0: 255, # '\x00' - 1: 255, # '\x01' - 2: 255, # '\x02' - 3: 255, # '\x03' - 4: 255, # '\x04' - 5: 255, # '\x05' - 6: 255, # '\x06' - 7: 255, # '\x07' - 8: 255, # '\x08' - 9: 255, # '\t' - 10: 254, # '\n' - 11: 255, # '\x0b' - 12: 255, # '\x0c' - 13: 254, # '\r' - 14: 255, # '\x0e' - 15: 255, # '\x0f' - 16: 255, # '\x10' - 17: 255, # '\x11' - 18: 255, # '\x12' - 19: 255, # '\x13' - 20: 255, # '\x14' - 21: 255, # '\x15' - 22: 255, # '\x16' - 23: 255, # '\x17' - 24: 255, # '\x18' - 25: 255, # '\x19' - 26: 255, # '\x1a' - 27: 255, # '\x1b' - 28: 255, # '\x1c' - 29: 255, # '\x1d' - 30: 255, # '\x1e' - 31: 255, # '\x1f' - 32: 253, # ' ' - 33: 253, # '!' - 34: 253, # '"' - 35: 253, # '#' - 36: 253, # '$' - 37: 253, # '%' - 38: 253, # '&' - 39: 253, # "'" - 40: 253, # '(' - 41: 253, # ')' - 42: 253, # '*' - 43: 253, # '+' - 44: 253, # ',' - 45: 253, # '-' - 46: 253, # '.' - 47: 253, # '/' - 48: 252, # '0' - 49: 252, # '1' - 50: 252, # '2' - 51: 252, # '3' - 52: 252, # '4' - 53: 252, # '5' - 54: 252, # '6' - 55: 252, # '7' - 56: 252, # '8' - 57: 252, # '9' - 58: 253, # ':' - 59: 253, # ';' - 60: 253, # '<' - 61: 253, # '=' - 62: 253, # '>' - 63: 253, # '?' - 64: 253, # '@' - 65: 182, # 'A' - 66: 106, # 'B' - 67: 107, # 'C' - 68: 100, # 'D' - 69: 183, # 'E' - 70: 184, # 'F' - 71: 185, # 'G' - 72: 101, # 'H' - 73: 94, # 'I' - 74: 186, # 'J' - 75: 187, # 'K' - 76: 108, # 'L' - 77: 109, # 'M' - 78: 110, # 'N' - 79: 111, # 'O' - 80: 188, # 'P' - 81: 189, # 'Q' - 82: 190, # 'R' - 83: 89, # 'S' - 84: 95, # 'T' - 85: 112, # 'U' - 86: 113, # 'V' - 87: 191, # 'W' - 88: 192, # 'X' - 89: 193, # 'Y' - 90: 194, # 'Z' - 91: 253, # '[' - 92: 253, # '\\' - 93: 253, # ']' - 94: 253, # '^' - 95: 253, # '_' - 96: 253, # '`' - 97: 64, # 'a' - 98: 72, # 'b' - 99: 73, # 'c' - 100: 114, # 'd' - 101: 74, # 'e' - 102: 115, # 'f' - 103: 116, # 'g' - 104: 102, # 'h' - 105: 81, # 'i' - 106: 201, # 'j' - 107: 117, # 'k' - 108: 90, # 'l' - 109: 103, # 'm' - 110: 78, # 'n' - 111: 82, # 'o' - 112: 96, # 'p' - 113: 202, # 'q' - 114: 91, # 'r' - 115: 79, # 's' - 116: 84, # 't' - 117: 104, # 'u' - 118: 105, # 'v' - 119: 97, # 'w' - 120: 98, # 'x' - 121: 92, # 'y' - 122: 203, # 'z' - 123: 253, # '{' - 124: 253, # '|' - 125: 253, # '}' - 126: 253, # '~' - 127: 253, # '\x7f' - 128: 209, # '\x80' - 129: 210, # '\x81' - 130: 211, # '\x82' - 131: 212, # '\x83' - 132: 213, # '\x84' - 133: 88, # '\x85' - 134: 214, # '\x86' - 135: 215, # '\x87' - 136: 216, # '\x88' - 137: 217, # '\x89' - 138: 218, # '\x8a' - 139: 219, # '\x8b' - 140: 220, # '\x8c' - 141: 118, # '\x8d' - 142: 221, # '\x8e' - 143: 222, # '\x8f' - 144: 223, # '\x90' - 145: 224, # '\x91' - 146: 99, # '\x92' - 147: 85, # '\x93' - 148: 83, # '\x94' - 149: 225, # '\x95' - 150: 226, # '\x96' - 151: 227, # '\x97' - 152: 228, # '\x98' - 153: 229, # '\x99' - 154: 230, # '\x9a' - 155: 231, # '\x9b' - 156: 232, # '\x9c' - 157: 233, # '\x9d' - 158: 234, # '\x9e' - 159: 235, # '\x9f' - 160: 236, # None - 161: 5, # 'ก' - 162: 30, # 'ข' - 163: 237, # 'ฃ' - 164: 24, # 'ค' - 165: 238, # 'ฅ' - 166: 75, # 'ฆ' - 167: 8, # 'ง' - 168: 26, # 'จ' - 169: 52, # 'ฉ' - 170: 34, # 'ช' - 171: 51, # 'ซ' - 172: 119, # 'ฌ' - 173: 47, # 'ญ' - 174: 58, # 'ฎ' - 175: 57, # 'ฏ' - 176: 49, # 'ฐ' - 177: 53, # 'ฑ' - 178: 55, # 'ฒ' - 179: 43, # 'ณ' - 180: 20, # 'ด' - 181: 19, # 'ต' - 182: 44, # 'ถ' - 183: 14, # 'ท' - 184: 48, # 'ธ' - 185: 3, # 'น' - 186: 17, # 'บ' - 187: 25, # 'ป' - 188: 39, # 'ผ' - 189: 62, # 'ฝ' - 190: 31, # 'พ' - 191: 54, # 'ฟ' - 192: 45, # 'ภ' - 193: 9, # 'ม' - 194: 16, # 'ย' - 195: 2, # 'ร' - 196: 61, # 'ฤ' - 197: 15, # 'ล' - 198: 239, # 'ฦ' - 199: 12, # 'ว' - 200: 42, # 'ศ' - 201: 46, # 'ษ' - 202: 18, # 'ส' - 203: 21, # 'ห' - 204: 76, # 'ฬ' - 205: 4, # 'อ' - 206: 66, # 'ฮ' - 207: 63, # 'ฯ' - 208: 22, # 'ะ' - 209: 10, # 'ั' - 210: 1, # 'า' - 211: 36, # 'ำ' - 212: 23, # 'ิ' - 213: 13, # 'ี' - 214: 40, # 'ึ' - 215: 27, # 'ื' - 216: 32, # 'ุ' - 217: 35, # 'ู' - 218: 86, # 'ฺ' - 219: 240, # None - 220: 241, # None - 221: 242, # None - 222: 243, # None - 223: 244, # '฿' - 224: 11, # 'เ' - 225: 28, # 'แ' - 226: 41, # 'โ' - 227: 29, # 'ใ' - 228: 33, # 'ไ' - 229: 245, # 'ๅ' - 230: 50, # 'ๆ' - 231: 37, # '็' - 232: 6, # '่' - 233: 7, # '้' - 234: 67, # '๊' - 235: 77, # '๋' - 236: 38, # '์' - 237: 93, # 'ํ' - 238: 246, # '๎' - 239: 247, # '๏' - 240: 68, # '๐' - 241: 56, # '๑' - 242: 59, # '๒' - 243: 65, # '๓' - 244: 69, # '๔' - 245: 60, # '๕' - 246: 70, # '๖' - 247: 80, # '๗' - 248: 71, # '๘' - 249: 87, # '๙' - 250: 248, # '๚' - 251: 249, # '๛' - 252: 250, # None - 253: 251, # None - 254: 252, # None - 255: 253, # None -} - -TIS_620_THAI_MODEL = SingleByteCharSetModel( - charset_name="TIS-620", - language="Thai", - char_to_order_map=TIS_620_THAI_CHAR_TO_ORDER, - language_model=THAI_LANG_MODEL, - typical_positive_ratio=0.926386, - keep_ascii_letters=False, - alphabet="กขฃคฅฆงจฉชซฌญฎฏฐฑฒณดตถทธนบปผฝพฟภมยรฤลฦวศษสหฬอฮฯะัาำิีึืฺุู฿เแโใไๅๆ็่้๊๋์ํ๎๏๐๑๒๓๔๕๖๗๘๙๚๛", -) diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/langturkishmodel.py b/venv/lib/python3.11/site-packages/pip/_vendor/chardet/langturkishmodel.py deleted file mode 100644 index 291857c..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/langturkishmodel.py +++ /dev/null @@ -1,4380 +0,0 @@ -from pip._vendor.chardet.sbcharsetprober import SingleByteCharSetModel - -# 3: Positive -# 2: Likely -# 1: Unlikely -# 0: Negative - -TURKISH_LANG_MODEL = { - 23: { # 'A' - 23: 0, # 'A' - 37: 0, # 'B' - 47: 0, # 'C' - 39: 0, # 'D' - 29: 0, # 'E' - 52: 0, # 'F' - 36: 0, # 'G' - 45: 0, # 'H' - 53: 0, # 'I' - 60: 0, # 'J' - 16: 0, # 'K' - 49: 0, # 'L' - 20: 0, # 'M' - 46: 0, # 'N' - 42: 0, # 'O' - 48: 0, # 'P' - 44: 0, # 'R' - 35: 0, # 'S' - 31: 0, # 'T' - 51: 0, # 'U' - 38: 0, # 'V' - 62: 0, # 'W' - 43: 0, # 'Y' - 56: 0, # 'Z' - 1: 3, # 'a' - 21: 0, # 'b' - 28: 0, # 'c' - 12: 2, # 'd' - 2: 3, # 'e' - 18: 0, # 'f' - 27: 1, # 'g' - 25: 1, # 'h' - 3: 1, # 'i' - 24: 0, # 'j' - 10: 2, # 'k' - 5: 1, # 'l' - 13: 1, # 'm' - 4: 1, # 'n' - 15: 0, # 'o' - 26: 0, # 'p' - 7: 1, # 'r' - 8: 1, # 's' - 9: 1, # 't' - 14: 1, # 'u' - 32: 0, # 'v' - 57: 0, # 'w' - 58: 0, # 'x' - 11: 3, # 'y' - 22: 0, # 'z' - 63: 0, # '·' - 54: 0, # 'Ç' - 50: 0, # 'Ö' - 55: 0, # 'Ü' - 59: 0, # 'â' - 33: 1, # 'ç' - 61: 0, # 'î' - 34: 0, # 'ö' - 17: 0, # 'ü' - 30: 0, # 'ğ' - 41: 0, # 'İ' - 6: 0, # 'ı' - 40: 0, # 'Ş' - 19: 0, # 'ş' - }, - 37: { # 'B' - 23: 0, # 'A' - 37: 0, # 'B' - 47: 2, # 'C' - 39: 0, # 'D' - 29: 0, # 'E' - 52: 2, # 'F' - 36: 0, # 'G' - 45: 0, # 'H' - 53: 0, # 'I' - 60: 0, # 'J' - 16: 1, # 'K' - 49: 0, # 'L' - 20: 0, # 'M' - 46: 0, # 'N' - 42: 0, # 'O' - 48: 1, # 'P' - 44: 0, # 'R' - 35: 1, # 'S' - 31: 0, # 'T' - 51: 0, # 'U' - 38: 1, # 'V' - 62: 0, # 'W' - 43: 1, # 'Y' - 56: 0, # 'Z' - 1: 2, # 'a' - 21: 0, # 'b' - 28: 2, # 'c' - 12: 0, # 'd' - 2: 3, # 'e' - 18: 0, # 'f' - 27: 0, # 'g' - 25: 0, # 'h' - 3: 0, # 'i' - 24: 0, # 'j' - 10: 0, # 'k' - 5: 0, # 'l' - 13: 1, # 'm' - 4: 1, # 'n' - 15: 0, # 'o' - 26: 0, # 'p' - 7: 0, # 'r' - 8: 0, # 's' - 9: 0, # 't' - 14: 2, # 'u' - 32: 0, # 'v' - 57: 0, # 'w' - 58: 0, # 'x' - 11: 0, # 'y' - 22: 1, # 'z' - 63: 0, # '·' - 54: 0, # 'Ç' - 50: 1, # 'Ö' - 55: 0, # 'Ü' - 59: 0, # 'â' - 33: 0, # 'ç' - 61: 0, # 'î' - 34: 1, # 'ö' - 17: 0, # 'ü' - 30: 0, # 'ğ' - 41: 0, # 'İ' - 6: 0, # 'ı' - 40: 1, # 'Ş' - 19: 1, # 'ş' - }, - 47: { # 'C' - 23: 0, # 'A' - 37: 0, # 'B' - 47: 0, # 'C' - 39: 0, # 'D' - 29: 0, # 'E' - 52: 1, # 'F' - 36: 0, # 'G' - 45: 0, # 'H' - 53: 0, # 'I' - 60: 0, # 'J' - 16: 0, # 'K' - 49: 1, # 'L' - 20: 0, # 'M' - 46: 1, # 'N' - 42: 0, # 'O' - 48: 1, # 'P' - 44: 1, # 'R' - 35: 0, # 'S' - 31: 0, # 'T' - 51: 0, # 'U' - 38: 1, # 'V' - 62: 0, # 'W' - 43: 1, # 'Y' - 56: 0, # 'Z' - 1: 3, # 'a' - 21: 0, # 'b' - 28: 2, # 'c' - 12: 0, # 'd' - 2: 3, # 'e' - 18: 0, # 'f' - 27: 0, # 'g' - 25: 0, # 'h' - 3: 0, # 'i' - 24: 2, # 'j' - 10: 1, # 'k' - 5: 2, # 'l' - 13: 2, # 'm' - 4: 2, # 'n' - 15: 1, # 'o' - 26: 0, # 'p' - 7: 2, # 'r' - 8: 0, # 's' - 9: 0, # 't' - 14: 3, # 'u' - 32: 0, # 'v' - 57: 0, # 'w' - 58: 0, # 'x' - 11: 0, # 'y' - 22: 2, # 'z' - 63: 0, # '·' - 54: 0, # 'Ç' - 50: 1, # 'Ö' - 55: 0, # 'Ü' - 59: 0, # 'â' - 33: 1, # 'ç' - 61: 0, # 'î' - 34: 1, # 'ö' - 17: 0, # 'ü' - 30: 0, # 'ğ' - 41: 1, # 'İ' - 6: 3, # 'ı' - 40: 0, # 'Ş' - 19: 0, # 'ş' - }, - 39: { # 'D' - 23: 0, # 'A' - 37: 0, # 'B' - 47: 0, # 'C' - 39: 0, # 'D' - 29: 0, # 'E' - 52: 1, # 'F' - 36: 0, # 'G' - 45: 0, # 'H' - 53: 0, # 'I' - 60: 0, # 'J' - 16: 1, # 'K' - 49: 0, # 'L' - 20: 0, # 'M' - 46: 0, # 'N' - 42: 0, # 'O' - 48: 1, # 'P' - 44: 0, # 'R' - 35: 0, # 'S' - 31: 0, # 'T' - 51: 0, # 'U' - 38: 0, # 'V' - 62: 0, # 'W' - 43: 0, # 'Y' - 56: 0, # 'Z' - 1: 2, # 'a' - 21: 0, # 'b' - 28: 2, # 'c' - 12: 0, # 'd' - 2: 2, # 'e' - 18: 0, # 'f' - 27: 0, # 'g' - 25: 0, # 'h' - 3: 0, # 'i' - 24: 0, # 'j' - 10: 0, # 'k' - 5: 1, # 'l' - 13: 3, # 'm' - 4: 0, # 'n' - 15: 1, # 'o' - 26: 0, # 'p' - 7: 0, # 'r' - 8: 0, # 's' - 9: 0, # 't' - 14: 1, # 'u' - 32: 0, # 'v' - 57: 0, # 'w' - 58: 0, # 'x' - 11: 0, # 'y' - 22: 1, # 'z' - 63: 0, # '·' - 54: 1, # 'Ç' - 50: 0, # 'Ö' - 55: 0, # 'Ü' - 59: 0, # 'â' - 33: 1, # 'ç' - 61: 0, # 'î' - 34: 0, # 'ö' - 17: 0, # 'ü' - 30: 1, # 'ğ' - 41: 0, # 'İ' - 6: 1, # 'ı' - 40: 1, # 'Ş' - 19: 0, # 'ş' - }, - 29: { # 'E' - 23: 0, # 'A' - 37: 0, # 'B' - 47: 0, # 'C' - 39: 0, # 'D' - 29: 1, # 'E' - 52: 0, # 'F' - 36: 0, # 'G' - 45: 0, # 'H' - 53: 0, # 'I' - 60: 0, # 'J' - 16: 3, # 'K' - 49: 0, # 'L' - 20: 1, # 'M' - 46: 0, # 'N' - 42: 0, # 'O' - 48: 0, # 'P' - 44: 0, # 'R' - 35: 0, # 'S' - 31: 0, # 'T' - 51: 0, # 'U' - 38: 0, # 'V' - 62: 0, # 'W' - 43: 0, # 'Y' - 56: 0, # 'Z' - 1: 3, # 'a' - 21: 0, # 'b' - 28: 0, # 'c' - 12: 2, # 'd' - 2: 3, # 'e' - 18: 0, # 'f' - 27: 1, # 'g' - 25: 0, # 'h' - 3: 1, # 'i' - 24: 1, # 'j' - 10: 0, # 'k' - 5: 3, # 'l' - 13: 3, # 'm' - 4: 3, # 'n' - 15: 0, # 'o' - 26: 0, # 'p' - 7: 0, # 'r' - 8: 1, # 's' - 9: 1, # 't' - 14: 1, # 'u' - 32: 1, # 'v' - 57: 0, # 'w' - 58: 0, # 'x' - 11: 2, # 'y' - 22: 0, # 'z' - 63: 0, # '·' - 54: 0, # 'Ç' - 50: 0, # 'Ö' - 55: 0, # 'Ü' - 59: 0, # 'â' - 33: 0, # 'ç' - 61: 0, # 'î' - 34: 0, # 'ö' - 17: 0, # 'ü' - 30: 0, # 'ğ' - 41: 0, # 'İ' - 6: 3, # 'ı' - 40: 0, # 'Ş' - 19: 0, # 'ş' - }, - 52: { # 'F' - 23: 0, # 'A' - 37: 1, # 'B' - 47: 1, # 'C' - 39: 1, # 'D' - 29: 1, # 'E' - 52: 2, # 'F' - 36: 0, # 'G' - 45: 2, # 'H' - 53: 1, # 'I' - 60: 0, # 'J' - 16: 0, # 'K' - 49: 0, # 'L' - 20: 1, # 'M' - 46: 1, # 'N' - 42: 1, # 'O' - 48: 2, # 'P' - 44: 1, # 'R' - 35: 1, # 'S' - 31: 1, # 'T' - 51: 1, # 'U' - 38: 1, # 'V' - 62: 0, # 'W' - 43: 2, # 'Y' - 56: 0, # 'Z' - 1: 0, # 'a' - 21: 1, # 'b' - 28: 1, # 'c' - 12: 1, # 'd' - 2: 0, # 'e' - 18: 1, # 'f' - 27: 0, # 'g' - 25: 0, # 'h' - 3: 2, # 'i' - 24: 1, # 'j' - 10: 0, # 'k' - 5: 0, # 'l' - 13: 1, # 'm' - 4: 2, # 'n' - 15: 1, # 'o' - 26: 0, # 'p' - 7: 2, # 'r' - 8: 1, # 's' - 9: 1, # 't' - 14: 1, # 'u' - 32: 0, # 'v' - 57: 0, # 'w' - 58: 0, # 'x' - 11: 1, # 'y' - 22: 1, # 'z' - 63: 0, # '·' - 54: 0, # 'Ç' - 50: 1, # 'Ö' - 55: 2, # 'Ü' - 59: 0, # 'â' - 33: 0, # 'ç' - 61: 0, # 'î' - 34: 2, # 'ö' - 17: 0, # 'ü' - 30: 1, # 'ğ' - 41: 1, # 'İ' - 6: 2, # 'ı' - 40: 0, # 'Ş' - 19: 2, # 'ş' - }, - 36: { # 'G' - 23: 1, # 'A' - 37: 0, # 'B' - 47: 1, # 'C' - 39: 0, # 'D' - 29: 0, # 'E' - 52: 1, # 'F' - 36: 2, # 'G' - 45: 0, # 'H' - 53: 0, # 'I' - 60: 0, # 'J' - 16: 2, # 'K' - 49: 0, # 'L' - 20: 0, # 'M' - 46: 2, # 'N' - 42: 1, # 'O' - 48: 1, # 'P' - 44: 1, # 'R' - 35: 1, # 'S' - 31: 0, # 'T' - 51: 1, # 'U' - 38: 2, # 'V' - 62: 0, # 'W' - 43: 0, # 'Y' - 56: 0, # 'Z' - 1: 3, # 'a' - 21: 0, # 'b' - 28: 1, # 'c' - 12: 0, # 'd' - 2: 3, # 'e' - 18: 0, # 'f' - 27: 0, # 'g' - 25: 0, # 'h' - 3: 0, # 'i' - 24: 1, # 'j' - 10: 1, # 'k' - 5: 0, # 'l' - 13: 3, # 'm' - 4: 2, # 'n' - 15: 0, # 'o' - 26: 1, # 'p' - 7: 0, # 'r' - 8: 1, # 's' - 9: 1, # 't' - 14: 3, # 'u' - 32: 0, # 'v' - 57: 0, # 'w' - 58: 1, # 'x' - 11: 0, # 'y' - 22: 2, # 'z' - 63: 0, # '·' - 54: 1, # 'Ç' - 50: 2, # 'Ö' - 55: 0, # 'Ü' - 59: 1, # 'â' - 33: 2, # 'ç' - 61: 0, # 'î' - 34: 0, # 'ö' - 17: 0, # 'ü' - 30: 1, # 'ğ' - 41: 1, # 'İ' - 6: 2, # 'ı' - 40: 2, # 'Ş' - 19: 1, # 'ş' - }, - 45: { # 'H' - 23: 0, # 'A' - 37: 1, # 'B' - 47: 0, # 'C' - 39: 0, # 'D' - 29: 0, # 'E' - 52: 2, # 'F' - 36: 2, # 'G' - 45: 1, # 'H' - 53: 1, # 'I' - 60: 0, # 'J' - 16: 2, # 'K' - 49: 1, # 'L' - 20: 0, # 'M' - 46: 1, # 'N' - 42: 1, # 'O' - 48: 1, # 'P' - 44: 0, # 'R' - 35: 2, # 'S' - 31: 0, # 'T' - 51: 1, # 'U' - 38: 2, # 'V' - 62: 0, # 'W' - 43: 0, # 'Y' - 56: 0, # 'Z' - 1: 3, # 'a' - 21: 0, # 'b' - 28: 2, # 'c' - 12: 0, # 'd' - 2: 3, # 'e' - 18: 0, # 'f' - 27: 0, # 'g' - 25: 0, # 'h' - 3: 2, # 'i' - 24: 0, # 'j' - 10: 1, # 'k' - 5: 0, # 'l' - 13: 2, # 'm' - 4: 0, # 'n' - 15: 1, # 'o' - 26: 1, # 'p' - 7: 1, # 'r' - 8: 0, # 's' - 9: 0, # 't' - 14: 3, # 'u' - 32: 0, # 'v' - 57: 0, # 'w' - 58: 0, # 'x' - 11: 0, # 'y' - 22: 2, # 'z' - 63: 0, # '·' - 54: 1, # 'Ç' - 50: 1, # 'Ö' - 55: 0, # 'Ü' - 59: 0, # 'â' - 33: 1, # 'ç' - 61: 0, # 'î' - 34: 1, # 'ö' - 17: 0, # 'ü' - 30: 2, # 'ğ' - 41: 1, # 'İ' - 6: 0, # 'ı' - 40: 2, # 'Ş' - 19: 1, # 'ş' - }, - 53: { # 'I' - 23: 0, # 'A' - 37: 0, # 'B' - 47: 0, # 'C' - 39: 0, # 'D' - 29: 0, # 'E' - 52: 1, # 'F' - 36: 0, # 'G' - 45: 0, # 'H' - 53: 0, # 'I' - 60: 0, # 'J' - 16: 2, # 'K' - 49: 0, # 'L' - 20: 0, # 'M' - 46: 0, # 'N' - 42: 0, # 'O' - 48: 1, # 'P' - 44: 0, # 'R' - 35: 0, # 'S' - 31: 0, # 'T' - 51: 0, # 'U' - 38: 0, # 'V' - 62: 0, # 'W' - 43: 0, # 'Y' - 56: 0, # 'Z' - 1: 2, # 'a' - 21: 0, # 'b' - 28: 2, # 'c' - 12: 0, # 'd' - 2: 2, # 'e' - 18: 0, # 'f' - 27: 0, # 'g' - 25: 0, # 'h' - 3: 0, # 'i' - 24: 0, # 'j' - 10: 0, # 'k' - 5: 2, # 'l' - 13: 2, # 'm' - 4: 0, # 'n' - 15: 0, # 'o' - 26: 0, # 'p' - 7: 0, # 'r' - 8: 0, # 's' - 9: 0, # 't' - 14: 2, # 'u' - 32: 0, # 'v' - 57: 0, # 'w' - 58: 0, # 'x' - 11: 0, # 'y' - 22: 2, # 'z' - 63: 0, # '·' - 54: 1, # 'Ç' - 50: 0, # 'Ö' - 55: 0, # 'Ü' - 59: 0, # 'â' - 33: 2, # 'ç' - 61: 0, # 'î' - 34: 1, # 'ö' - 17: 0, # 'ü' - 30: 0, # 'ğ' - 41: 0, # 'İ' - 6: 0, # 'ı' - 40: 1, # 'Ş' - 19: 1, # 'ş' - }, - 60: { # 'J' - 23: 0, # 'A' - 37: 0, # 'B' - 47: 0, # 'C' - 39: 0, # 'D' - 29: 0, # 'E' - 52: 0, # 'F' - 36: 0, # 'G' - 45: 0, # 'H' - 53: 0, # 'I' - 60: 0, # 'J' - 16: 0, # 'K' - 49: 0, # 'L' - 20: 1, # 'M' - 46: 0, # 'N' - 42: 0, # 'O' - 48: 0, # 'P' - 44: 0, # 'R' - 35: 0, # 'S' - 31: 0, # 'T' - 51: 0, # 'U' - 38: 0, # 'V' - 62: 0, # 'W' - 43: 0, # 'Y' - 56: 0, # 'Z' - 1: 0, # 'a' - 21: 1, # 'b' - 28: 0, # 'c' - 12: 1, # 'd' - 2: 0, # 'e' - 18: 0, # 'f' - 27: 0, # 'g' - 25: 0, # 'h' - 3: 1, # 'i' - 24: 0, # 'j' - 10: 0, # 'k' - 5: 0, # 'l' - 13: 0, # 'm' - 4: 1, # 'n' - 15: 0, # 'o' - 26: 0, # 'p' - 7: 0, # 'r' - 8: 1, # 's' - 9: 0, # 't' - 14: 0, # 'u' - 32: 0, # 'v' - 57: 0, # 'w' - 58: 0, # 'x' - 11: 0, # 'y' - 22: 0, # 'z' - 63: 0, # '·' - 54: 0, # 'Ç' - 50: 0, # 'Ö' - 55: 0, # 'Ü' - 59: 0, # 'â' - 33: 0, # 'ç' - 61: 0, # 'î' - 34: 0, # 'ö' - 17: 0, # 'ü' - 30: 0, # 'ğ' - 41: 0, # 'İ' - 6: 0, # 'ı' - 40: 0, # 'Ş' - 19: 0, # 'ş' - }, - 16: { # 'K' - 23: 0, # 'A' - 37: 0, # 'B' - 47: 0, # 'C' - 39: 0, # 'D' - 29: 3, # 'E' - 52: 0, # 'F' - 36: 0, # 'G' - 45: 0, # 'H' - 53: 0, # 'I' - 60: 0, # 'J' - 16: 0, # 'K' - 49: 0, # 'L' - 20: 2, # 'M' - 46: 0, # 'N' - 42: 0, # 'O' - 48: 0, # 'P' - 44: 0, # 'R' - 35: 0, # 'S' - 31: 2, # 'T' - 51: 0, # 'U' - 38: 0, # 'V' - 62: 0, # 'W' - 43: 0, # 'Y' - 56: 0, # 'Z' - 1: 2, # 'a' - 21: 3, # 'b' - 28: 0, # 'c' - 12: 3, # 'd' - 2: 1, # 'e' - 18: 3, # 'f' - 27: 3, # 'g' - 25: 3, # 'h' - 3: 3, # 'i' - 24: 2, # 'j' - 10: 3, # 'k' - 5: 0, # 'l' - 13: 0, # 'm' - 4: 3, # 'n' - 15: 0, # 'o' - 26: 1, # 'p' - 7: 3, # 'r' - 8: 3, # 's' - 9: 3, # 't' - 14: 0, # 'u' - 32: 3, # 'v' - 57: 0, # 'w' - 58: 0, # 'x' - 11: 2, # 'y' - 22: 1, # 'z' - 63: 0, # '·' - 54: 0, # 'Ç' - 50: 0, # 'Ö' - 55: 0, # 'Ü' - 59: 0, # 'â' - 33: 0, # 'ç' - 61: 0, # 'î' - 34: 0, # 'ö' - 17: 2, # 'ü' - 30: 0, # 'ğ' - 41: 1, # 'İ' - 6: 3, # 'ı' - 40: 0, # 'Ş' - 19: 0, # 'ş' - }, - 49: { # 'L' - 23: 0, # 'A' - 37: 0, # 'B' - 47: 0, # 'C' - 39: 0, # 'D' - 29: 2, # 'E' - 52: 0, # 'F' - 36: 1, # 'G' - 45: 1, # 'H' - 53: 0, # 'I' - 60: 0, # 'J' - 16: 0, # 'K' - 49: 0, # 'L' - 20: 1, # 'M' - 46: 0, # 'N' - 42: 2, # 'O' - 48: 0, # 'P' - 44: 0, # 'R' - 35: 0, # 'S' - 31: 0, # 'T' - 51: 0, # 'U' - 38: 0, # 'V' - 62: 0, # 'W' - 43: 1, # 'Y' - 56: 0, # 'Z' - 1: 0, # 'a' - 21: 3, # 'b' - 28: 0, # 'c' - 12: 2, # 'd' - 2: 0, # 'e' - 18: 0, # 'f' - 27: 0, # 'g' - 25: 0, # 'h' - 3: 2, # 'i' - 24: 0, # 'j' - 10: 1, # 'k' - 5: 0, # 'l' - 13: 0, # 'm' - 4: 2, # 'n' - 15: 1, # 'o' - 26: 1, # 'p' - 7: 1, # 'r' - 8: 1, # 's' - 9: 1, # 't' - 14: 0, # 'u' - 32: 0, # 'v' - 57: 0, # 'w' - 58: 0, # 'x' - 11: 2, # 'y' - 22: 0, # 'z' - 63: 0, # '·' - 54: 0, # 'Ç' - 50: 0, # 'Ö' - 55: 2, # 'Ü' - 59: 0, # 'â' - 33: 0, # 'ç' - 61: 0, # 'î' - 34: 1, # 'ö' - 17: 1, # 'ü' - 30: 1, # 'ğ' - 41: 0, # 'İ' - 6: 2, # 'ı' - 40: 0, # 'Ş' - 19: 0, # 'ş' - }, - 20: { # 'M' - 23: 1, # 'A' - 37: 0, # 'B' - 47: 0, # 'C' - 39: 0, # 'D' - 29: 0, # 'E' - 52: 0, # 'F' - 36: 0, # 'G' - 45: 0, # 'H' - 53: 0, # 'I' - 60: 1, # 'J' - 16: 3, # 'K' - 49: 0, # 'L' - 20: 2, # 'M' - 46: 0, # 'N' - 42: 0, # 'O' - 48: 0, # 'P' - 44: 0, # 'R' - 35: 0, # 'S' - 31: 1, # 'T' - 51: 0, # 'U' - 38: 0, # 'V' - 62: 0, # 'W' - 43: 0, # 'Y' - 56: 0, # 'Z' - 1: 3, # 'a' - 21: 2, # 'b' - 28: 0, # 'c' - 12: 3, # 'd' - 2: 3, # 'e' - 18: 0, # 'f' - 27: 1, # 'g' - 25: 1, # 'h' - 3: 2, # 'i' - 24: 2, # 'j' - 10: 2, # 'k' - 5: 2, # 'l' - 13: 3, # 'm' - 4: 3, # 'n' - 15: 0, # 'o' - 26: 1, # 'p' - 7: 3, # 'r' - 8: 0, # 's' - 9: 2, # 't' - 14: 3, # 'u' - 32: 0, # 'v' - 57: 0, # 'w' - 58: 0, # 'x' - 11: 2, # 'y' - 22: 0, # 'z' - 63: 0, # '·' - 54: 0, # 'Ç' - 50: 0, # 'Ö' - 55: 0, # 'Ü' - 59: 0, # 'â' - 33: 3, # 'ç' - 61: 0, # 'î' - 34: 0, # 'ö' - 17: 0, # 'ü' - 30: 0, # 'ğ' - 41: 0, # 'İ' - 6: 3, # 'ı' - 40: 0, # 'Ş' - 19: 0, # 'ş' - }, - 46: { # 'N' - 23: 0, # 'A' - 37: 1, # 'B' - 47: 0, # 'C' - 39: 0, # 'D' - 29: 0, # 'E' - 52: 1, # 'F' - 36: 1, # 'G' - 45: 1, # 'H' - 53: 0, # 'I' - 60: 0, # 'J' - 16: 2, # 'K' - 49: 0, # 'L' - 20: 0, # 'M' - 46: 1, # 'N' - 42: 0, # 'O' - 48: 0, # 'P' - 44: 1, # 'R' - 35: 1, # 'S' - 31: 0, # 'T' - 51: 1, # 'U' - 38: 2, # 'V' - 62: 0, # 'W' - 43: 1, # 'Y' - 56: 0, # 'Z' - 1: 3, # 'a' - 21: 0, # 'b' - 28: 2, # 'c' - 12: 0, # 'd' - 2: 3, # 'e' - 18: 0, # 'f' - 27: 1, # 'g' - 25: 0, # 'h' - 3: 0, # 'i' - 24: 2, # 'j' - 10: 1, # 'k' - 5: 1, # 'l' - 13: 3, # 'm' - 4: 2, # 'n' - 15: 1, # 'o' - 26: 1, # 'p' - 7: 1, # 'r' - 8: 0, # 's' - 9: 0, # 't' - 14: 3, # 'u' - 32: 0, # 'v' - 57: 0, # 'w' - 58: 1, # 'x' - 11: 1, # 'y' - 22: 2, # 'z' - 63: 0, # '·' - 54: 1, # 'Ç' - 50: 1, # 'Ö' - 55: 0, # 'Ü' - 59: 0, # 'â' - 33: 0, # 'ç' - 61: 0, # 'î' - 34: 1, # 'ö' - 17: 0, # 'ü' - 30: 0, # 'ğ' - 41: 1, # 'İ' - 6: 2, # 'ı' - 40: 1, # 'Ş' - 19: 1, # 'ş' - }, - 42: { # 'O' - 23: 0, # 'A' - 37: 0, # 'B' - 47: 0, # 'C' - 39: 0, # 'D' - 29: 0, # 'E' - 52: 1, # 'F' - 36: 0, # 'G' - 45: 1, # 'H' - 53: 0, # 'I' - 60: 0, # 'J' - 16: 2, # 'K' - 49: 1, # 'L' - 20: 0, # 'M' - 46: 0, # 'N' - 42: 0, # 'O' - 48: 2, # 'P' - 44: 1, # 'R' - 35: 1, # 'S' - 31: 0, # 'T' - 51: 1, # 'U' - 38: 1, # 'V' - 62: 0, # 'W' - 43: 0, # 'Y' - 56: 0, # 'Z' - 1: 3, # 'a' - 21: 0, # 'b' - 28: 2, # 'c' - 12: 0, # 'd' - 2: 2, # 'e' - 18: 0, # 'f' - 27: 0, # 'g' - 25: 0, # 'h' - 3: 0, # 'i' - 24: 0, # 'j' - 10: 0, # 'k' - 5: 3, # 'l' - 13: 3, # 'm' - 4: 0, # 'n' - 15: 1, # 'o' - 26: 0, # 'p' - 7: 0, # 'r' - 8: 0, # 's' - 9: 0, # 't' - 14: 2, # 'u' - 32: 0, # 'v' - 57: 0, # 'w' - 58: 0, # 'x' - 11: 0, # 'y' - 22: 2, # 'z' - 63: 0, # '·' - 54: 2, # 'Ç' - 50: 1, # 'Ö' - 55: 0, # 'Ü' - 59: 0, # 'â' - 33: 2, # 'ç' - 61: 0, # 'î' - 34: 1, # 'ö' - 17: 0, # 'ü' - 30: 1, # 'ğ' - 41: 2, # 'İ' - 6: 1, # 'ı' - 40: 1, # 'Ş' - 19: 1, # 'ş' - }, - 48: { # 'P' - 23: 0, # 'A' - 37: 0, # 'B' - 47: 2, # 'C' - 39: 0, # 'D' - 29: 0, # 'E' - 52: 2, # 'F' - 36: 1, # 'G' - 45: 1, # 'H' - 53: 0, # 'I' - 60: 0, # 'J' - 16: 2, # 'K' - 49: 0, # 'L' - 20: 0, # 'M' - 46: 1, # 'N' - 42: 1, # 'O' - 48: 1, # 'P' - 44: 0, # 'R' - 35: 1, # 'S' - 31: 0, # 'T' - 51: 0, # 'U' - 38: 1, # 'V' - 62: 0, # 'W' - 43: 0, # 'Y' - 56: 0, # 'Z' - 1: 2, # 'a' - 21: 0, # 'b' - 28: 2, # 'c' - 12: 0, # 'd' - 2: 3, # 'e' - 18: 0, # 'f' - 27: 0, # 'g' - 25: 0, # 'h' - 3: 0, # 'i' - 24: 0, # 'j' - 10: 1, # 'k' - 5: 0, # 'l' - 13: 2, # 'm' - 4: 0, # 'n' - 15: 2, # 'o' - 26: 0, # 'p' - 7: 0, # 'r' - 8: 0, # 's' - 9: 0, # 't' - 14: 2, # 'u' - 32: 0, # 'v' - 57: 0, # 'w' - 58: 2, # 'x' - 11: 0, # 'y' - 22: 2, # 'z' - 63: 0, # '·' - 54: 1, # 'Ç' - 50: 2, # 'Ö' - 55: 0, # 'Ü' - 59: 0, # 'â' - 33: 0, # 'ç' - 61: 0, # 'î' - 34: 2, # 'ö' - 17: 0, # 'ü' - 30: 1, # 'ğ' - 41: 1, # 'İ' - 6: 0, # 'ı' - 40: 2, # 'Ş' - 19: 1, # 'ş' - }, - 44: { # 'R' - 23: 0, # 'A' - 37: 0, # 'B' - 47: 1, # 'C' - 39: 0, # 'D' - 29: 0, # 'E' - 52: 1, # 'F' - 36: 0, # 'G' - 45: 0, # 'H' - 53: 0, # 'I' - 60: 0, # 'J' - 16: 3, # 'K' - 49: 0, # 'L' - 20: 0, # 'M' - 46: 0, # 'N' - 42: 0, # 'O' - 48: 1, # 'P' - 44: 0, # 'R' - 35: 0, # 'S' - 31: 0, # 'T' - 51: 0, # 'U' - 38: 0, # 'V' - 62: 0, # 'W' - 43: 1, # 'Y' - 56: 0, # 'Z' - 1: 3, # 'a' - 21: 1, # 'b' - 28: 1, # 'c' - 12: 0, # 'd' - 2: 2, # 'e' - 18: 0, # 'f' - 27: 0, # 'g' - 25: 0, # 'h' - 3: 0, # 'i' - 24: 0, # 'j' - 10: 1, # 'k' - 5: 2, # 'l' - 13: 2, # 'm' - 4: 0, # 'n' - 15: 1, # 'o' - 26: 0, # 'p' - 7: 0, # 'r' - 8: 0, # 's' - 9: 0, # 't' - 14: 2, # 'u' - 32: 0, # 'v' - 57: 0, # 'w' - 58: 0, # 'x' - 11: 1, # 'y' - 22: 2, # 'z' - 63: 0, # '·' - 54: 0, # 'Ç' - 50: 1, # 'Ö' - 55: 0, # 'Ü' - 59: 0, # 'â' - 33: 1, # 'ç' - 61: 0, # 'î' - 34: 1, # 'ö' - 17: 1, # 'ü' - 30: 1, # 'ğ' - 41: 0, # 'İ' - 6: 2, # 'ı' - 40: 1, # 'Ş' - 19: 1, # 'ş' - }, - 35: { # 'S' - 23: 0, # 'A' - 37: 0, # 'B' - 47: 1, # 'C' - 39: 0, # 'D' - 29: 0, # 'E' - 52: 1, # 'F' - 36: 1, # 'G' - 45: 1, # 'H' - 53: 0, # 'I' - 60: 0, # 'J' - 16: 3, # 'K' - 49: 1, # 'L' - 20: 1, # 'M' - 46: 0, # 'N' - 42: 0, # 'O' - 48: 1, # 'P' - 44: 0, # 'R' - 35: 0, # 'S' - 31: 0, # 'T' - 51: 1, # 'U' - 38: 1, # 'V' - 62: 0, # 'W' - 43: 1, # 'Y' - 56: 0, # 'Z' - 1: 3, # 'a' - 21: 0, # 'b' - 28: 2, # 'c' - 12: 0, # 'd' - 2: 3, # 'e' - 18: 0, # 'f' - 27: 0, # 'g' - 25: 0, # 'h' - 3: 0, # 'i' - 24: 0, # 'j' - 10: 1, # 'k' - 5: 1, # 'l' - 13: 2, # 'm' - 4: 1, # 'n' - 15: 0, # 'o' - 26: 0, # 'p' - 7: 0, # 'r' - 8: 0, # 's' - 9: 1, # 't' - 14: 2, # 'u' - 32: 0, # 'v' - 57: 0, # 'w' - 58: 0, # 'x' - 11: 0, # 'y' - 22: 1, # 'z' - 63: 0, # '·' - 54: 2, # 'Ç' - 50: 2, # 'Ö' - 55: 0, # 'Ü' - 59: 0, # 'â' - 33: 3, # 'ç' - 61: 0, # 'î' - 34: 1, # 'ö' - 17: 0, # 'ü' - 30: 0, # 'ğ' - 41: 0, # 'İ' - 6: 3, # 'ı' - 40: 2, # 'Ş' - 19: 1, # 'ş' - }, - 31: { # 'T' - 23: 0, # 'A' - 37: 0, # 'B' - 47: 0, # 'C' - 39: 0, # 'D' - 29: 0, # 'E' - 52: 0, # 'F' - 36: 0, # 'G' - 45: 0, # 'H' - 53: 0, # 'I' - 60: 1, # 'J' - 16: 2, # 'K' - 49: 0, # 'L' - 20: 1, # 'M' - 46: 0, # 'N' - 42: 0, # 'O' - 48: 0, # 'P' - 44: 0, # 'R' - 35: 0, # 'S' - 31: 2, # 'T' - 51: 0, # 'U' - 38: 0, # 'V' - 62: 0, # 'W' - 43: 0, # 'Y' - 56: 0, # 'Z' - 1: 3, # 'a' - 21: 2, # 'b' - 28: 0, # 'c' - 12: 1, # 'd' - 2: 3, # 'e' - 18: 2, # 'f' - 27: 2, # 'g' - 25: 0, # 'h' - 3: 1, # 'i' - 24: 1, # 'j' - 10: 2, # 'k' - 5: 2, # 'l' - 13: 3, # 'm' - 4: 3, # 'n' - 15: 0, # 'o' - 26: 2, # 'p' - 7: 2, # 'r' - 8: 0, # 's' - 9: 2, # 't' - 14: 2, # 'u' - 32: 1, # 'v' - 57: 1, # 'w' - 58: 1, # 'x' - 11: 2, # 'y' - 22: 0, # 'z' - 63: 0, # '·' - 54: 0, # 'Ç' - 50: 0, # 'Ö' - 55: 0, # 'Ü' - 59: 0, # 'â' - 33: 0, # 'ç' - 61: 0, # 'î' - 34: 0, # 'ö' - 17: 1, # 'ü' - 30: 0, # 'ğ' - 41: 0, # 'İ' - 6: 3, # 'ı' - 40: 0, # 'Ş' - 19: 0, # 'ş' - }, - 51: { # 'U' - 23: 0, # 'A' - 37: 0, # 'B' - 47: 0, # 'C' - 39: 0, # 'D' - 29: 0, # 'E' - 52: 1, # 'F' - 36: 1, # 'G' - 45: 0, # 'H' - 53: 0, # 'I' - 60: 0, # 'J' - 16: 1, # 'K' - 49: 0, # 'L' - 20: 0, # 'M' - 46: 1, # 'N' - 42: 0, # 'O' - 48: 1, # 'P' - 44: 0, # 'R' - 35: 0, # 'S' - 31: 0, # 'T' - 51: 1, # 'U' - 38: 1, # 'V' - 62: 0, # 'W' - 43: 0, # 'Y' - 56: 0, # 'Z' - 1: 3, # 'a' - 21: 0, # 'b' - 28: 1, # 'c' - 12: 0, # 'd' - 2: 3, # 'e' - 18: 0, # 'f' - 27: 2, # 'g' - 25: 0, # 'h' - 3: 0, # 'i' - 24: 0, # 'j' - 10: 1, # 'k' - 5: 1, # 'l' - 13: 3, # 'm' - 4: 2, # 'n' - 15: 0, # 'o' - 26: 1, # 'p' - 7: 0, # 'r' - 8: 0, # 's' - 9: 0, # 't' - 14: 2, # 'u' - 32: 0, # 'v' - 57: 0, # 'w' - 58: 0, # 'x' - 11: 0, # 'y' - 22: 2, # 'z' - 63: 0, # '·' - 54: 1, # 'Ç' - 50: 1, # 'Ö' - 55: 0, # 'Ü' - 59: 0, # 'â' - 33: 0, # 'ç' - 61: 0, # 'î' - 34: 0, # 'ö' - 17: 0, # 'ü' - 30: 1, # 'ğ' - 41: 1, # 'İ' - 6: 2, # 'ı' - 40: 0, # 'Ş' - 19: 1, # 'ş' - }, - 38: { # 'V' - 23: 1, # 'A' - 37: 1, # 'B' - 47: 1, # 'C' - 39: 0, # 'D' - 29: 0, # 'E' - 52: 2, # 'F' - 36: 0, # 'G' - 45: 0, # 'H' - 53: 0, # 'I' - 60: 0, # 'J' - 16: 3, # 'K' - 49: 0, # 'L' - 20: 3, # 'M' - 46: 0, # 'N' - 42: 0, # 'O' - 48: 1, # 'P' - 44: 1, # 'R' - 35: 0, # 'S' - 31: 0, # 'T' - 51: 1, # 'U' - 38: 1, # 'V' - 62: 0, # 'W' - 43: 0, # 'Y' - 56: 0, # 'Z' - 1: 3, # 'a' - 21: 0, # 'b' - 28: 2, # 'c' - 12: 0, # 'd' - 2: 3, # 'e' - 18: 0, # 'f' - 27: 0, # 'g' - 25: 0, # 'h' - 3: 0, # 'i' - 24: 0, # 'j' - 10: 0, # 'k' - 5: 2, # 'l' - 13: 2, # 'm' - 4: 0, # 'n' - 15: 2, # 'o' - 26: 0, # 'p' - 7: 0, # 'r' - 8: 0, # 's' - 9: 1, # 't' - 14: 3, # 'u' - 32: 0, # 'v' - 57: 0, # 'w' - 58: 0, # 'x' - 11: 1, # 'y' - 22: 2, # 'z' - 63: 0, # '·' - 54: 1, # 'Ç' - 50: 1, # 'Ö' - 55: 0, # 'Ü' - 59: 1, # 'â' - 33: 2, # 'ç' - 61: 0, # 'î' - 34: 1, # 'ö' - 17: 0, # 'ü' - 30: 1, # 'ğ' - 41: 1, # 'İ' - 6: 3, # 'ı' - 40: 2, # 'Ş' - 19: 1, # 'ş' - }, - 62: { # 'W' - 23: 0, # 'A' - 37: 0, # 'B' - 47: 0, # 'C' - 39: 0, # 'D' - 29: 0, # 'E' - 52: 0, # 'F' - 36: 0, # 'G' - 45: 0, # 'H' - 53: 0, # 'I' - 60: 0, # 'J' - 16: 0, # 'K' - 49: 0, # 'L' - 20: 0, # 'M' - 46: 0, # 'N' - 42: 0, # 'O' - 48: 0, # 'P' - 44: 0, # 'R' - 35: 0, # 'S' - 31: 0, # 'T' - 51: 0, # 'U' - 38: 0, # 'V' - 62: 0, # 'W' - 43: 0, # 'Y' - 56: 0, # 'Z' - 1: 0, # 'a' - 21: 0, # 'b' - 28: 0, # 'c' - 12: 0, # 'd' - 2: 0, # 'e' - 18: 0, # 'f' - 27: 0, # 'g' - 25: 0, # 'h' - 3: 0, # 'i' - 24: 0, # 'j' - 10: 0, # 'k' - 5: 0, # 'l' - 13: 0, # 'm' - 4: 0, # 'n' - 15: 0, # 'o' - 26: 0, # 'p' - 7: 0, # 'r' - 8: 0, # 's' - 9: 0, # 't' - 14: 0, # 'u' - 32: 0, # 'v' - 57: 0, # 'w' - 58: 0, # 'x' - 11: 0, # 'y' - 22: 0, # 'z' - 63: 0, # '·' - 54: 0, # 'Ç' - 50: 0, # 'Ö' - 55: 0, # 'Ü' - 59: 0, # 'â' - 33: 0, # 'ç' - 61: 0, # 'î' - 34: 0, # 'ö' - 17: 0, # 'ü' - 30: 0, # 'ğ' - 41: 0, # 'İ' - 6: 0, # 'ı' - 40: 0, # 'Ş' - 19: 0, # 'ş' - }, - 43: { # 'Y' - 23: 0, # 'A' - 37: 0, # 'B' - 47: 1, # 'C' - 39: 0, # 'D' - 29: 0, # 'E' - 52: 2, # 'F' - 36: 0, # 'G' - 45: 1, # 'H' - 53: 1, # 'I' - 60: 0, # 'J' - 16: 2, # 'K' - 49: 0, # 'L' - 20: 0, # 'M' - 46: 2, # 'N' - 42: 0, # 'O' - 48: 2, # 'P' - 44: 1, # 'R' - 35: 1, # 'S' - 31: 0, # 'T' - 51: 1, # 'U' - 38: 2, # 'V' - 62: 0, # 'W' - 43: 0, # 'Y' - 56: 0, # 'Z' - 1: 3, # 'a' - 21: 0, # 'b' - 28: 2, # 'c' - 12: 0, # 'd' - 2: 2, # 'e' - 18: 0, # 'f' - 27: 0, # 'g' - 25: 0, # 'h' - 3: 0, # 'i' - 24: 1, # 'j' - 10: 1, # 'k' - 5: 1, # 'l' - 13: 3, # 'm' - 4: 0, # 'n' - 15: 2, # 'o' - 26: 0, # 'p' - 7: 0, # 'r' - 8: 0, # 's' - 9: 0, # 't' - 14: 3, # 'u' - 32: 0, # 'v' - 57: 0, # 'w' - 58: 1, # 'x' - 11: 0, # 'y' - 22: 2, # 'z' - 63: 0, # '·' - 54: 1, # 'Ç' - 50: 2, # 'Ö' - 55: 1, # 'Ü' - 59: 1, # 'â' - 33: 0, # 'ç' - 61: 0, # 'î' - 34: 1, # 'ö' - 17: 0, # 'ü' - 30: 1, # 'ğ' - 41: 1, # 'İ' - 6: 0, # 'ı' - 40: 2, # 'Ş' - 19: 1, # 'ş' - }, - 56: { # 'Z' - 23: 0, # 'A' - 37: 0, # 'B' - 47: 0, # 'C' - 39: 0, # 'D' - 29: 0, # 'E' - 52: 0, # 'F' - 36: 0, # 'G' - 45: 0, # 'H' - 53: 0, # 'I' - 60: 0, # 'J' - 16: 0, # 'K' - 49: 0, # 'L' - 20: 0, # 'M' - 46: 0, # 'N' - 42: 0, # 'O' - 48: 0, # 'P' - 44: 0, # 'R' - 35: 0, # 'S' - 31: 0, # 'T' - 51: 0, # 'U' - 38: 0, # 'V' - 62: 0, # 'W' - 43: 0, # 'Y' - 56: 2, # 'Z' - 1: 2, # 'a' - 21: 1, # 'b' - 28: 0, # 'c' - 12: 0, # 'd' - 2: 2, # 'e' - 18: 0, # 'f' - 27: 0, # 'g' - 25: 0, # 'h' - 3: 2, # 'i' - 24: 1, # 'j' - 10: 0, # 'k' - 5: 0, # 'l' - 13: 1, # 'm' - 4: 1, # 'n' - 15: 0, # 'o' - 26: 0, # 'p' - 7: 1, # 'r' - 8: 1, # 's' - 9: 0, # 't' - 14: 2, # 'u' - 32: 0, # 'v' - 57: 0, # 'w' - 58: 0, # 'x' - 11: 0, # 'y' - 22: 0, # 'z' - 63: 0, # '·' - 54: 0, # 'Ç' - 50: 0, # 'Ö' - 55: 0, # 'Ü' - 59: 0, # 'â' - 33: 0, # 'ç' - 61: 0, # 'î' - 34: 0, # 'ö' - 17: 1, # 'ü' - 30: 0, # 'ğ' - 41: 0, # 'İ' - 6: 1, # 'ı' - 40: 0, # 'Ş' - 19: 0, # 'ş' - }, - 1: { # 'a' - 23: 3, # 'A' - 37: 0, # 'B' - 47: 1, # 'C' - 39: 0, # 'D' - 29: 3, # 'E' - 52: 0, # 'F' - 36: 1, # 'G' - 45: 1, # 'H' - 53: 0, # 'I' - 60: 0, # 'J' - 16: 0, # 'K' - 49: 0, # 'L' - 20: 3, # 'M' - 46: 1, # 'N' - 42: 0, # 'O' - 48: 1, # 'P' - 44: 0, # 'R' - 35: 0, # 'S' - 31: 3, # 'T' - 51: 0, # 'U' - 38: 1, # 'V' - 62: 0, # 'W' - 43: 0, # 'Y' - 56: 2, # 'Z' - 1: 2, # 'a' - 21: 3, # 'b' - 28: 0, # 'c' - 12: 3, # 'd' - 2: 2, # 'e' - 18: 3, # 'f' - 27: 3, # 'g' - 25: 3, # 'h' - 3: 3, # 'i' - 24: 3, # 'j' - 10: 3, # 'k' - 5: 0, # 'l' - 13: 2, # 'm' - 4: 3, # 'n' - 15: 1, # 'o' - 26: 3, # 'p' - 7: 3, # 'r' - 8: 3, # 's' - 9: 3, # 't' - 14: 3, # 'u' - 32: 3, # 'v' - 57: 2, # 'w' - 58: 0, # 'x' - 11: 3, # 'y' - 22: 0, # 'z' - 63: 1, # '·' - 54: 0, # 'Ç' - 50: 0, # 'Ö' - 55: 0, # 'Ü' - 59: 0, # 'â' - 33: 1, # 'ç' - 61: 1, # 'î' - 34: 1, # 'ö' - 17: 3, # 'ü' - 30: 0, # 'ğ' - 41: 0, # 'İ' - 6: 3, # 'ı' - 40: 0, # 'Ş' - 19: 1, # 'ş' - }, - 21: { # 'b' - 23: 0, # 'A' - 37: 0, # 'B' - 47: 0, # 'C' - 39: 0, # 'D' - 29: 0, # 'E' - 52: 0, # 'F' - 36: 1, # 'G' - 45: 0, # 'H' - 53: 0, # 'I' - 60: 1, # 'J' - 16: 2, # 'K' - 49: 0, # 'L' - 20: 2, # 'M' - 46: 0, # 'N' - 42: 0, # 'O' - 48: 0, # 'P' - 44: 0, # 'R' - 35: 0, # 'S' - 31: 1, # 'T' - 51: 0, # 'U' - 38: 0, # 'V' - 62: 0, # 'W' - 43: 1, # 'Y' - 56: 0, # 'Z' - 1: 3, # 'a' - 21: 2, # 'b' - 28: 0, # 'c' - 12: 3, # 'd' - 2: 3, # 'e' - 18: 0, # 'f' - 27: 3, # 'g' - 25: 1, # 'h' - 3: 3, # 'i' - 24: 2, # 'j' - 10: 3, # 'k' - 5: 3, # 'l' - 13: 3, # 'm' - 4: 3, # 'n' - 15: 0, # 'o' - 26: 3, # 'p' - 7: 1, # 'r' - 8: 2, # 's' - 9: 2, # 't' - 14: 2, # 'u' - 32: 1, # 'v' - 57: 0, # 'w' - 58: 1, # 'x' - 11: 3, # 'y' - 22: 0, # 'z' - 63: 0, # '·' - 54: 0, # 'Ç' - 50: 0, # 'Ö' - 55: 0, # 'Ü' - 59: 0, # 'â' - 33: 1, # 'ç' - 61: 0, # 'î' - 34: 0, # 'ö' - 17: 0, # 'ü' - 30: 1, # 'ğ' - 41: 0, # 'İ' - 6: 2, # 'ı' - 40: 0, # 'Ş' - 19: 0, # 'ş' - }, - 28: { # 'c' - 23: 0, # 'A' - 37: 1, # 'B' - 47: 1, # 'C' - 39: 1, # 'D' - 29: 2, # 'E' - 52: 0, # 'F' - 36: 2, # 'G' - 45: 2, # 'H' - 53: 1, # 'I' - 60: 0, # 'J' - 16: 0, # 'K' - 49: 0, # 'L' - 20: 2, # 'M' - 46: 1, # 'N' - 42: 1, # 'O' - 48: 2, # 'P' - 44: 1, # 'R' - 35: 1, # 'S' - 31: 2, # 'T' - 51: 2, # 'U' - 38: 2, # 'V' - 62: 0, # 'W' - 43: 3, # 'Y' - 56: 0, # 'Z' - 1: 1, # 'a' - 21: 1, # 'b' - 28: 2, # 'c' - 12: 2, # 'd' - 2: 1, # 'e' - 18: 1, # 'f' - 27: 2, # 'g' - 25: 2, # 'h' - 3: 3, # 'i' - 24: 1, # 'j' - 10: 3, # 'k' - 5: 0, # 'l' - 13: 2, # 'm' - 4: 3, # 'n' - 15: 2, # 'o' - 26: 2, # 'p' - 7: 3, # 'r' - 8: 3, # 's' - 9: 3, # 't' - 14: 1, # 'u' - 32: 0, # 'v' - 57: 1, # 'w' - 58: 0, # 'x' - 11: 2, # 'y' - 22: 1, # 'z' - 63: 1, # '·' - 54: 0, # 'Ç' - 50: 0, # 'Ö' - 55: 1, # 'Ü' - 59: 0, # 'â' - 33: 0, # 'ç' - 61: 1, # 'î' - 34: 2, # 'ö' - 17: 2, # 'ü' - 30: 2, # 'ğ' - 41: 1, # 'İ' - 6: 3, # 'ı' - 40: 0, # 'Ş' - 19: 2, # 'ş' - }, - 12: { # 'd' - 23: 1, # 'A' - 37: 0, # 'B' - 47: 0, # 'C' - 39: 0, # 'D' - 29: 0, # 'E' - 52: 0, # 'F' - 36: 0, # 'G' - 45: 0, # 'H' - 53: 0, # 'I' - 60: 2, # 'J' - 16: 3, # 'K' - 49: 0, # 'L' - 20: 3, # 'M' - 46: 0, # 'N' - 42: 0, # 'O' - 48: 0, # 'P' - 44: 0, # 'R' - 35: 1, # 'S' - 31: 1, # 'T' - 51: 0, # 'U' - 38: 0, # 'V' - 62: 0, # 'W' - 43: 0, # 'Y' - 56: 0, # 'Z' - 1: 3, # 'a' - 21: 2, # 'b' - 28: 1, # 'c' - 12: 3, # 'd' - 2: 3, # 'e' - 18: 1, # 'f' - 27: 3, # 'g' - 25: 3, # 'h' - 3: 2, # 'i' - 24: 3, # 'j' - 10: 2, # 'k' - 5: 3, # 'l' - 13: 3, # 'm' - 4: 3, # 'n' - 15: 1, # 'o' - 26: 2, # 'p' - 7: 3, # 'r' - 8: 2, # 's' - 9: 2, # 't' - 14: 3, # 'u' - 32: 1, # 'v' - 57: 0, # 'w' - 58: 1, # 'x' - 11: 3, # 'y' - 22: 1, # 'z' - 63: 1, # '·' - 54: 0, # 'Ç' - 50: 0, # 'Ö' - 55: 0, # 'Ü' - 59: 0, # 'â' - 33: 0, # 'ç' - 61: 0, # 'î' - 34: 0, # 'ö' - 17: 1, # 'ü' - 30: 0, # 'ğ' - 41: 0, # 'İ' - 6: 2, # 'ı' - 40: 0, # 'Ş' - 19: 0, # 'ş' - }, - 2: { # 'e' - 23: 2, # 'A' - 37: 0, # 'B' - 47: 2, # 'C' - 39: 0, # 'D' - 29: 3, # 'E' - 52: 1, # 'F' - 36: 0, # 'G' - 45: 0, # 'H' - 53: 0, # 'I' - 60: 0, # 'J' - 16: 1, # 'K' - 49: 0, # 'L' - 20: 3, # 'M' - 46: 1, # 'N' - 42: 0, # 'O' - 48: 1, # 'P' - 44: 1, # 'R' - 35: 0, # 'S' - 31: 3, # 'T' - 51: 0, # 'U' - 38: 1, # 'V' - 62: 0, # 'W' - 43: 1, # 'Y' - 56: 0, # 'Z' - 1: 3, # 'a' - 21: 3, # 'b' - 28: 0, # 'c' - 12: 3, # 'd' - 2: 2, # 'e' - 18: 3, # 'f' - 27: 3, # 'g' - 25: 3, # 'h' - 3: 3, # 'i' - 24: 3, # 'j' - 10: 3, # 'k' - 5: 0, # 'l' - 13: 2, # 'm' - 4: 3, # 'n' - 15: 1, # 'o' - 26: 3, # 'p' - 7: 3, # 'r' - 8: 3, # 's' - 9: 3, # 't' - 14: 3, # 'u' - 32: 3, # 'v' - 57: 2, # 'w' - 58: 0, # 'x' - 11: 3, # 'y' - 22: 1, # 'z' - 63: 1, # '·' - 54: 0, # 'Ç' - 50: 0, # 'Ö' - 55: 0, # 'Ü' - 59: 0, # 'â' - 33: 1, # 'ç' - 61: 0, # 'î' - 34: 1, # 'ö' - 17: 3, # 'ü' - 30: 0, # 'ğ' - 41: 0, # 'İ' - 6: 3, # 'ı' - 40: 0, # 'Ş' - 19: 0, # 'ş' - }, - 18: { # 'f' - 23: 0, # 'A' - 37: 0, # 'B' - 47: 0, # 'C' - 39: 0, # 'D' - 29: 0, # 'E' - 52: 0, # 'F' - 36: 0, # 'G' - 45: 0, # 'H' - 53: 0, # 'I' - 60: 0, # 'J' - 16: 2, # 'K' - 49: 0, # 'L' - 20: 2, # 'M' - 46: 0, # 'N' - 42: 0, # 'O' - 48: 0, # 'P' - 44: 0, # 'R' - 35: 0, # 'S' - 31: 2, # 'T' - 51: 0, # 'U' - 38: 0, # 'V' - 62: 0, # 'W' - 43: 0, # 'Y' - 56: 0, # 'Z' - 1: 3, # 'a' - 21: 1, # 'b' - 28: 0, # 'c' - 12: 3, # 'd' - 2: 3, # 'e' - 18: 2, # 'f' - 27: 1, # 'g' - 25: 1, # 'h' - 3: 1, # 'i' - 24: 1, # 'j' - 10: 1, # 'k' - 5: 3, # 'l' - 13: 3, # 'm' - 4: 3, # 'n' - 15: 0, # 'o' - 26: 2, # 'p' - 7: 1, # 'r' - 8: 3, # 's' - 9: 3, # 't' - 14: 1, # 'u' - 32: 2, # 'v' - 57: 0, # 'w' - 58: 0, # 'x' - 11: 1, # 'y' - 22: 0, # 'z' - 63: 0, # '·' - 54: 0, # 'Ç' - 50: 0, # 'Ö' - 55: 0, # 'Ü' - 59: 0, # 'â' - 33: 1, # 'ç' - 61: 0, # 'î' - 34: 0, # 'ö' - 17: 1, # 'ü' - 30: 0, # 'ğ' - 41: 0, # 'İ' - 6: 1, # 'ı' - 40: 0, # 'Ş' - 19: 0, # 'ş' - }, - 27: { # 'g' - 23: 0, # 'A' - 37: 0, # 'B' - 47: 0, # 'C' - 39: 0, # 'D' - 29: 0, # 'E' - 52: 0, # 'F' - 36: 0, # 'G' - 45: 0, # 'H' - 53: 0, # 'I' - 60: 0, # 'J' - 16: 3, # 'K' - 49: 0, # 'L' - 20: 0, # 'M' - 46: 0, # 'N' - 42: 0, # 'O' - 48: 0, # 'P' - 44: 0, # 'R' - 35: 1, # 'S' - 31: 1, # 'T' - 51: 0, # 'U' - 38: 2, # 'V' - 62: 0, # 'W' - 43: 0, # 'Y' - 56: 0, # 'Z' - 1: 3, # 'a' - 21: 1, # 'b' - 28: 0, # 'c' - 12: 1, # 'd' - 2: 3, # 'e' - 18: 0, # 'f' - 27: 2, # 'g' - 25: 1, # 'h' - 3: 2, # 'i' - 24: 3, # 'j' - 10: 2, # 'k' - 5: 3, # 'l' - 13: 3, # 'm' - 4: 2, # 'n' - 15: 0, # 'o' - 26: 1, # 'p' - 7: 2, # 'r' - 8: 2, # 's' - 9: 3, # 't' - 14: 3, # 'u' - 32: 1, # 'v' - 57: 0, # 'w' - 58: 0, # 'x' - 11: 1, # 'y' - 22: 0, # 'z' - 63: 1, # '·' - 54: 0, # 'Ç' - 50: 0, # 'Ö' - 55: 0, # 'Ü' - 59: 0, # 'â' - 33: 0, # 'ç' - 61: 0, # 'î' - 34: 0, # 'ö' - 17: 0, # 'ü' - 30: 0, # 'ğ' - 41: 0, # 'İ' - 6: 2, # 'ı' - 40: 0, # 'Ş' - 19: 0, # 'ş' - }, - 25: { # 'h' - 23: 0, # 'A' - 37: 0, # 'B' - 47: 0, # 'C' - 39: 0, # 'D' - 29: 0, # 'E' - 52: 0, # 'F' - 36: 0, # 'G' - 45: 0, # 'H' - 53: 0, # 'I' - 60: 0, # 'J' - 16: 2, # 'K' - 49: 0, # 'L' - 20: 0, # 'M' - 46: 0, # 'N' - 42: 0, # 'O' - 48: 0, # 'P' - 44: 0, # 'R' - 35: 0, # 'S' - 31: 0, # 'T' - 51: 0, # 'U' - 38: 0, # 'V' - 62: 0, # 'W' - 43: 0, # 'Y' - 56: 0, # 'Z' - 1: 3, # 'a' - 21: 0, # 'b' - 28: 0, # 'c' - 12: 2, # 'd' - 2: 3, # 'e' - 18: 0, # 'f' - 27: 1, # 'g' - 25: 2, # 'h' - 3: 2, # 'i' - 24: 3, # 'j' - 10: 3, # 'k' - 5: 3, # 'l' - 13: 3, # 'm' - 4: 3, # 'n' - 15: 1, # 'o' - 26: 1, # 'p' - 7: 3, # 'r' - 8: 3, # 's' - 9: 2, # 't' - 14: 3, # 'u' - 32: 2, # 'v' - 57: 1, # 'w' - 58: 0, # 'x' - 11: 1, # 'y' - 22: 0, # 'z' - 63: 0, # '·' - 54: 0, # 'Ç' - 50: 0, # 'Ö' - 55: 0, # 'Ü' - 59: 0, # 'â' - 33: 0, # 'ç' - 61: 0, # 'î' - 34: 0, # 'ö' - 17: 0, # 'ü' - 30: 0, # 'ğ' - 41: 0, # 'İ' - 6: 3, # 'ı' - 40: 0, # 'Ş' - 19: 0, # 'ş' - }, - 3: { # 'i' - 23: 2, # 'A' - 37: 0, # 'B' - 47: 0, # 'C' - 39: 0, # 'D' - 29: 0, # 'E' - 52: 0, # 'F' - 36: 0, # 'G' - 45: 0, # 'H' - 53: 0, # 'I' - 60: 1, # 'J' - 16: 3, # 'K' - 49: 0, # 'L' - 20: 3, # 'M' - 46: 0, # 'N' - 42: 1, # 'O' - 48: 0, # 'P' - 44: 0, # 'R' - 35: 1, # 'S' - 31: 2, # 'T' - 51: 0, # 'U' - 38: 1, # 'V' - 62: 0, # 'W' - 43: 0, # 'Y' - 56: 0, # 'Z' - 1: 3, # 'a' - 21: 2, # 'b' - 28: 0, # 'c' - 12: 3, # 'd' - 2: 3, # 'e' - 18: 2, # 'f' - 27: 3, # 'g' - 25: 1, # 'h' - 3: 3, # 'i' - 24: 2, # 'j' - 10: 3, # 'k' - 5: 3, # 'l' - 13: 3, # 'm' - 4: 3, # 'n' - 15: 1, # 'o' - 26: 3, # 'p' - 7: 3, # 'r' - 8: 3, # 's' - 9: 3, # 't' - 14: 3, # 'u' - 32: 2, # 'v' - 57: 1, # 'w' - 58: 1, # 'x' - 11: 3, # 'y' - 22: 1, # 'z' - 63: 1, # '·' - 54: 0, # 'Ç' - 50: 0, # 'Ö' - 55: 1, # 'Ü' - 59: 0, # 'â' - 33: 2, # 'ç' - 61: 0, # 'î' - 34: 0, # 'ö' - 17: 3, # 'ü' - 30: 0, # 'ğ' - 41: 1, # 'İ' - 6: 2, # 'ı' - 40: 0, # 'Ş' - 19: 0, # 'ş' - }, - 24: { # 'j' - 23: 0, # 'A' - 37: 0, # 'B' - 47: 0, # 'C' - 39: 0, # 'D' - 29: 0, # 'E' - 52: 0, # 'F' - 36: 0, # 'G' - 45: 0, # 'H' - 53: 0, # 'I' - 60: 1, # 'J' - 16: 2, # 'K' - 49: 0, # 'L' - 20: 2, # 'M' - 46: 0, # 'N' - 42: 0, # 'O' - 48: 1, # 'P' - 44: 0, # 'R' - 35: 0, # 'S' - 31: 1, # 'T' - 51: 0, # 'U' - 38: 0, # 'V' - 62: 0, # 'W' - 43: 0, # 'Y' - 56: 1, # 'Z' - 1: 3, # 'a' - 21: 1, # 'b' - 28: 1, # 'c' - 12: 3, # 'd' - 2: 3, # 'e' - 18: 2, # 'f' - 27: 1, # 'g' - 25: 1, # 'h' - 3: 2, # 'i' - 24: 1, # 'j' - 10: 2, # 'k' - 5: 2, # 'l' - 13: 3, # 'm' - 4: 2, # 'n' - 15: 0, # 'o' - 26: 1, # 'p' - 7: 2, # 'r' - 8: 3, # 's' - 9: 2, # 't' - 14: 3, # 'u' - 32: 2, # 'v' - 57: 0, # 'w' - 58: 2, # 'x' - 11: 1, # 'y' - 22: 0, # 'z' - 63: 0, # '·' - 54: 0, # 'Ç' - 50: 0, # 'Ö' - 55: 0, # 'Ü' - 59: 0, # 'â' - 33: 1, # 'ç' - 61: 0, # 'î' - 34: 0, # 'ö' - 17: 1, # 'ü' - 30: 0, # 'ğ' - 41: 0, # 'İ' - 6: 3, # 'ı' - 40: 0, # 'Ş' - 19: 0, # 'ş' - }, - 10: { # 'k' - 23: 0, # 'A' - 37: 0, # 'B' - 47: 0, # 'C' - 39: 0, # 'D' - 29: 0, # 'E' - 52: 0, # 'F' - 36: 0, # 'G' - 45: 0, # 'H' - 53: 0, # 'I' - 60: 0, # 'J' - 16: 3, # 'K' - 49: 0, # 'L' - 20: 2, # 'M' - 46: 0, # 'N' - 42: 0, # 'O' - 48: 0, # 'P' - 44: 0, # 'R' - 35: 0, # 'S' - 31: 3, # 'T' - 51: 0, # 'U' - 38: 1, # 'V' - 62: 0, # 'W' - 43: 0, # 'Y' - 56: 1, # 'Z' - 1: 3, # 'a' - 21: 2, # 'b' - 28: 0, # 'c' - 12: 2, # 'd' - 2: 3, # 'e' - 18: 1, # 'f' - 27: 2, # 'g' - 25: 2, # 'h' - 3: 3, # 'i' - 24: 2, # 'j' - 10: 2, # 'k' - 5: 3, # 'l' - 13: 3, # 'm' - 4: 3, # 'n' - 15: 0, # 'o' - 26: 3, # 'p' - 7: 2, # 'r' - 8: 2, # 's' - 9: 2, # 't' - 14: 3, # 'u' - 32: 0, # 'v' - 57: 0, # 'w' - 58: 1, # 'x' - 11: 3, # 'y' - 22: 0, # 'z' - 63: 1, # '·' - 54: 0, # 'Ç' - 50: 0, # 'Ö' - 55: 0, # 'Ü' - 59: 0, # 'â' - 33: 3, # 'ç' - 61: 0, # 'î' - 34: 1, # 'ö' - 17: 3, # 'ü' - 30: 1, # 'ğ' - 41: 0, # 'İ' - 6: 3, # 'ı' - 40: 0, # 'Ş' - 19: 1, # 'ş' - }, - 5: { # 'l' - 23: 0, # 'A' - 37: 0, # 'B' - 47: 0, # 'C' - 39: 0, # 'D' - 29: 3, # 'E' - 52: 0, # 'F' - 36: 0, # 'G' - 45: 0, # 'H' - 53: 0, # 'I' - 60: 0, # 'J' - 16: 0, # 'K' - 49: 0, # 'L' - 20: 2, # 'M' - 46: 0, # 'N' - 42: 0, # 'O' - 48: 0, # 'P' - 44: 0, # 'R' - 35: 0, # 'S' - 31: 1, # 'T' - 51: 0, # 'U' - 38: 0, # 'V' - 62: 0, # 'W' - 43: 0, # 'Y' - 56: 0, # 'Z' - 1: 0, # 'a' - 21: 3, # 'b' - 28: 0, # 'c' - 12: 3, # 'd' - 2: 1, # 'e' - 18: 3, # 'f' - 27: 3, # 'g' - 25: 2, # 'h' - 3: 3, # 'i' - 24: 2, # 'j' - 10: 3, # 'k' - 5: 1, # 'l' - 13: 1, # 'm' - 4: 3, # 'n' - 15: 0, # 'o' - 26: 2, # 'p' - 7: 3, # 'r' - 8: 3, # 's' - 9: 3, # 't' - 14: 2, # 'u' - 32: 2, # 'v' - 57: 0, # 'w' - 58: 0, # 'x' - 11: 3, # 'y' - 22: 0, # 'z' - 63: 0, # '·' - 54: 0, # 'Ç' - 50: 0, # 'Ö' - 55: 0, # 'Ü' - 59: 0, # 'â' - 33: 1, # 'ç' - 61: 0, # 'î' - 34: 0, # 'ö' - 17: 2, # 'ü' - 30: 0, # 'ğ' - 41: 0, # 'İ' - 6: 3, # 'ı' - 40: 0, # 'Ş' - 19: 0, # 'ş' - }, - 13: { # 'm' - 23: 1, # 'A' - 37: 0, # 'B' - 47: 0, # 'C' - 39: 0, # 'D' - 29: 3, # 'E' - 52: 0, # 'F' - 36: 0, # 'G' - 45: 0, # 'H' - 53: 0, # 'I' - 60: 0, # 'J' - 16: 0, # 'K' - 49: 0, # 'L' - 20: 3, # 'M' - 46: 0, # 'N' - 42: 0, # 'O' - 48: 0, # 'P' - 44: 0, # 'R' - 35: 0, # 'S' - 31: 3, # 'T' - 51: 0, # 'U' - 38: 0, # 'V' - 62: 0, # 'W' - 43: 1, # 'Y' - 56: 0, # 'Z' - 1: 2, # 'a' - 21: 3, # 'b' - 28: 0, # 'c' - 12: 3, # 'd' - 2: 2, # 'e' - 18: 3, # 'f' - 27: 3, # 'g' - 25: 3, # 'h' - 3: 3, # 'i' - 24: 3, # 'j' - 10: 3, # 'k' - 5: 0, # 'l' - 13: 2, # 'm' - 4: 3, # 'n' - 15: 1, # 'o' - 26: 2, # 'p' - 7: 3, # 'r' - 8: 3, # 's' - 9: 3, # 't' - 14: 2, # 'u' - 32: 2, # 'v' - 57: 1, # 'w' - 58: 0, # 'x' - 11: 3, # 'y' - 22: 0, # 'z' - 63: 0, # '·' - 54: 0, # 'Ç' - 50: 0, # 'Ö' - 55: 0, # 'Ü' - 59: 0, # 'â' - 33: 0, # 'ç' - 61: 0, # 'î' - 34: 0, # 'ö' - 17: 3, # 'ü' - 30: 0, # 'ğ' - 41: 0, # 'İ' - 6: 3, # 'ı' - 40: 0, # 'Ş' - 19: 1, # 'ş' - }, - 4: { # 'n' - 23: 1, # 'A' - 37: 0, # 'B' - 47: 0, # 'C' - 39: 0, # 'D' - 29: 0, # 'E' - 52: 0, # 'F' - 36: 0, # 'G' - 45: 1, # 'H' - 53: 0, # 'I' - 60: 2, # 'J' - 16: 3, # 'K' - 49: 0, # 'L' - 20: 3, # 'M' - 46: 0, # 'N' - 42: 0, # 'O' - 48: 0, # 'P' - 44: 0, # 'R' - 35: 0, # 'S' - 31: 2, # 'T' - 51: 0, # 'U' - 38: 0, # 'V' - 62: 0, # 'W' - 43: 0, # 'Y' - 56: 0, # 'Z' - 1: 3, # 'a' - 21: 2, # 'b' - 28: 1, # 'c' - 12: 3, # 'd' - 2: 3, # 'e' - 18: 1, # 'f' - 27: 2, # 'g' - 25: 3, # 'h' - 3: 2, # 'i' - 24: 2, # 'j' - 10: 3, # 'k' - 5: 3, # 'l' - 13: 3, # 'm' - 4: 3, # 'n' - 15: 1, # 'o' - 26: 3, # 'p' - 7: 2, # 'r' - 8: 3, # 's' - 9: 3, # 't' - 14: 3, # 'u' - 32: 2, # 'v' - 57: 0, # 'w' - 58: 2, # 'x' - 11: 3, # 'y' - 22: 0, # 'z' - 63: 0, # '·' - 54: 0, # 'Ç' - 50: 0, # 'Ö' - 55: 0, # 'Ü' - 59: 0, # 'â' - 33: 1, # 'ç' - 61: 0, # 'î' - 34: 0, # 'ö' - 17: 2, # 'ü' - 30: 0, # 'ğ' - 41: 0, # 'İ' - 6: 1, # 'ı' - 40: 0, # 'Ş' - 19: 0, # 'ş' - }, - 15: { # 'o' - 23: 0, # 'A' - 37: 0, # 'B' - 47: 1, # 'C' - 39: 0, # 'D' - 29: 0, # 'E' - 52: 2, # 'F' - 36: 1, # 'G' - 45: 1, # 'H' - 53: 1, # 'I' - 60: 0, # 'J' - 16: 3, # 'K' - 49: 2, # 'L' - 20: 0, # 'M' - 46: 2, # 'N' - 42: 1, # 'O' - 48: 2, # 'P' - 44: 1, # 'R' - 35: 0, # 'S' - 31: 0, # 'T' - 51: 0, # 'U' - 38: 0, # 'V' - 62: 0, # 'W' - 43: 0, # 'Y' - 56: 0, # 'Z' - 1: 3, # 'a' - 21: 0, # 'b' - 28: 2, # 'c' - 12: 0, # 'd' - 2: 3, # 'e' - 18: 0, # 'f' - 27: 0, # 'g' - 25: 0, # 'h' - 3: 1, # 'i' - 24: 2, # 'j' - 10: 1, # 'k' - 5: 3, # 'l' - 13: 3, # 'm' - 4: 2, # 'n' - 15: 2, # 'o' - 26: 0, # 'p' - 7: 1, # 'r' - 8: 0, # 's' - 9: 0, # 't' - 14: 3, # 'u' - 32: 0, # 'v' - 57: 0, # 'w' - 58: 2, # 'x' - 11: 0, # 'y' - 22: 2, # 'z' - 63: 0, # '·' - 54: 1, # 'Ç' - 50: 2, # 'Ö' - 55: 0, # 'Ü' - 59: 0, # 'â' - 33: 3, # 'ç' - 61: 0, # 'î' - 34: 1, # 'ö' - 17: 0, # 'ü' - 30: 2, # 'ğ' - 41: 2, # 'İ' - 6: 3, # 'ı' - 40: 2, # 'Ş' - 19: 2, # 'ş' - }, - 26: { # 'p' - 23: 0, # 'A' - 37: 0, # 'B' - 47: 0, # 'C' - 39: 0, # 'D' - 29: 0, # 'E' - 52: 0, # 'F' - 36: 0, # 'G' - 45: 0, # 'H' - 53: 0, # 'I' - 60: 0, # 'J' - 16: 3, # 'K' - 49: 0, # 'L' - 20: 1, # 'M' - 46: 0, # 'N' - 42: 0, # 'O' - 48: 0, # 'P' - 44: 0, # 'R' - 35: 0, # 'S' - 31: 0, # 'T' - 51: 0, # 'U' - 38: 0, # 'V' - 62: 0, # 'W' - 43: 0, # 'Y' - 56: 0, # 'Z' - 1: 3, # 'a' - 21: 1, # 'b' - 28: 0, # 'c' - 12: 1, # 'd' - 2: 3, # 'e' - 18: 0, # 'f' - 27: 1, # 'g' - 25: 1, # 'h' - 3: 2, # 'i' - 24: 3, # 'j' - 10: 1, # 'k' - 5: 3, # 'l' - 13: 3, # 'm' - 4: 2, # 'n' - 15: 0, # 'o' - 26: 2, # 'p' - 7: 2, # 'r' - 8: 1, # 's' - 9: 1, # 't' - 14: 3, # 'u' - 32: 0, # 'v' - 57: 0, # 'w' - 58: 1, # 'x' - 11: 1, # 'y' - 22: 0, # 'z' - 63: 0, # '·' - 54: 0, # 'Ç' - 50: 0, # 'Ö' - 55: 0, # 'Ü' - 59: 0, # 'â' - 33: 3, # 'ç' - 61: 0, # 'î' - 34: 0, # 'ö' - 17: 1, # 'ü' - 30: 0, # 'ğ' - 41: 0, # 'İ' - 6: 3, # 'ı' - 40: 0, # 'Ş' - 19: 0, # 'ş' - }, - 7: { # 'r' - 23: 0, # 'A' - 37: 0, # 'B' - 47: 0, # 'C' - 39: 0, # 'D' - 29: 0, # 'E' - 52: 1, # 'F' - 36: 0, # 'G' - 45: 0, # 'H' - 53: 0, # 'I' - 60: 2, # 'J' - 16: 3, # 'K' - 49: 0, # 'L' - 20: 2, # 'M' - 46: 0, # 'N' - 42: 0, # 'O' - 48: 0, # 'P' - 44: 0, # 'R' - 35: 0, # 'S' - 31: 2, # 'T' - 51: 1, # 'U' - 38: 0, # 'V' - 62: 0, # 'W' - 43: 0, # 'Y' - 56: 1, # 'Z' - 1: 3, # 'a' - 21: 1, # 'b' - 28: 0, # 'c' - 12: 3, # 'd' - 2: 3, # 'e' - 18: 0, # 'f' - 27: 2, # 'g' - 25: 3, # 'h' - 3: 2, # 'i' - 24: 2, # 'j' - 10: 3, # 'k' - 5: 3, # 'l' - 13: 3, # 'm' - 4: 3, # 'n' - 15: 0, # 'o' - 26: 2, # 'p' - 7: 3, # 'r' - 8: 3, # 's' - 9: 3, # 't' - 14: 3, # 'u' - 32: 2, # 'v' - 57: 0, # 'w' - 58: 1, # 'x' - 11: 2, # 'y' - 22: 0, # 'z' - 63: 1, # '·' - 54: 0, # 'Ç' - 50: 0, # 'Ö' - 55: 0, # 'Ü' - 59: 0, # 'â' - 33: 2, # 'ç' - 61: 0, # 'î' - 34: 0, # 'ö' - 17: 3, # 'ü' - 30: 0, # 'ğ' - 41: 0, # 'İ' - 6: 2, # 'ı' - 40: 0, # 'Ş' - 19: 0, # 'ş' - }, - 8: { # 's' - 23: 1, # 'A' - 37: 0, # 'B' - 47: 0, # 'C' - 39: 0, # 'D' - 29: 0, # 'E' - 52: 0, # 'F' - 36: 1, # 'G' - 45: 0, # 'H' - 53: 0, # 'I' - 60: 0, # 'J' - 16: 3, # 'K' - 49: 0, # 'L' - 20: 3, # 'M' - 46: 0, # 'N' - 42: 0, # 'O' - 48: 0, # 'P' - 44: 0, # 'R' - 35: 0, # 'S' - 31: 2, # 'T' - 51: 0, # 'U' - 38: 0, # 'V' - 62: 0, # 'W' - 43: 0, # 'Y' - 56: 1, # 'Z' - 1: 3, # 'a' - 21: 2, # 'b' - 28: 1, # 'c' - 12: 3, # 'd' - 2: 3, # 'e' - 18: 0, # 'f' - 27: 2, # 'g' - 25: 2, # 'h' - 3: 2, # 'i' - 24: 3, # 'j' - 10: 3, # 'k' - 5: 3, # 'l' - 13: 3, # 'm' - 4: 3, # 'n' - 15: 0, # 'o' - 26: 3, # 'p' - 7: 3, # 'r' - 8: 3, # 's' - 9: 3, # 't' - 14: 3, # 'u' - 32: 2, # 'v' - 57: 0, # 'w' - 58: 1, # 'x' - 11: 2, # 'y' - 22: 1, # 'z' - 63: 0, # '·' - 54: 0, # 'Ç' - 50: 0, # 'Ö' - 55: 0, # 'Ü' - 59: 0, # 'â' - 33: 2, # 'ç' - 61: 0, # 'î' - 34: 0, # 'ö' - 17: 2, # 'ü' - 30: 0, # 'ğ' - 41: 0, # 'İ' - 6: 3, # 'ı' - 40: 0, # 'Ş' - 19: 1, # 'ş' - }, - 9: { # 't' - 23: 0, # 'A' - 37: 0, # 'B' - 47: 0, # 'C' - 39: 0, # 'D' - 29: 0, # 'E' - 52: 0, # 'F' - 36: 0, # 'G' - 45: 0, # 'H' - 53: 0, # 'I' - 60: 1, # 'J' - 16: 3, # 'K' - 49: 0, # 'L' - 20: 2, # 'M' - 46: 0, # 'N' - 42: 0, # 'O' - 48: 0, # 'P' - 44: 0, # 'R' - 35: 0, # 'S' - 31: 2, # 'T' - 51: 0, # 'U' - 38: 0, # 'V' - 62: 0, # 'W' - 43: 0, # 'Y' - 56: 1, # 'Z' - 1: 3, # 'a' - 21: 3, # 'b' - 28: 0, # 'c' - 12: 3, # 'd' - 2: 3, # 'e' - 18: 2, # 'f' - 27: 2, # 'g' - 25: 2, # 'h' - 3: 2, # 'i' - 24: 2, # 'j' - 10: 3, # 'k' - 5: 3, # 'l' - 13: 3, # 'm' - 4: 3, # 'n' - 15: 0, # 'o' - 26: 2, # 'p' - 7: 3, # 'r' - 8: 3, # 's' - 9: 3, # 't' - 14: 3, # 'u' - 32: 3, # 'v' - 57: 0, # 'w' - 58: 2, # 'x' - 11: 2, # 'y' - 22: 0, # 'z' - 63: 0, # '·' - 54: 0, # 'Ç' - 50: 0, # 'Ö' - 55: 0, # 'Ü' - 59: 0, # 'â' - 33: 3, # 'ç' - 61: 0, # 'î' - 34: 0, # 'ö' - 17: 2, # 'ü' - 30: 0, # 'ğ' - 41: 0, # 'İ' - 6: 3, # 'ı' - 40: 0, # 'Ş' - 19: 0, # 'ş' - }, - 14: { # 'u' - 23: 3, # 'A' - 37: 0, # 'B' - 47: 0, # 'C' - 39: 0, # 'D' - 29: 3, # 'E' - 52: 0, # 'F' - 36: 0, # 'G' - 45: 1, # 'H' - 53: 0, # 'I' - 60: 1, # 'J' - 16: 0, # 'K' - 49: 0, # 'L' - 20: 3, # 'M' - 46: 2, # 'N' - 42: 0, # 'O' - 48: 1, # 'P' - 44: 0, # 'R' - 35: 0, # 'S' - 31: 3, # 'T' - 51: 0, # 'U' - 38: 0, # 'V' - 62: 0, # 'W' - 43: 1, # 'Y' - 56: 2, # 'Z' - 1: 2, # 'a' - 21: 3, # 'b' - 28: 0, # 'c' - 12: 3, # 'd' - 2: 2, # 'e' - 18: 2, # 'f' - 27: 3, # 'g' - 25: 3, # 'h' - 3: 3, # 'i' - 24: 2, # 'j' - 10: 3, # 'k' - 5: 0, # 'l' - 13: 3, # 'm' - 4: 3, # 'n' - 15: 0, # 'o' - 26: 3, # 'p' - 7: 3, # 'r' - 8: 3, # 's' - 9: 3, # 't' - 14: 3, # 'u' - 32: 2, # 'v' - 57: 2, # 'w' - 58: 0, # 'x' - 11: 3, # 'y' - 22: 0, # 'z' - 63: 1, # '·' - 54: 0, # 'Ç' - 50: 0, # 'Ö' - 55: 0, # 'Ü' - 59: 0, # 'â' - 33: 0, # 'ç' - 61: 0, # 'î' - 34: 0, # 'ö' - 17: 3, # 'ü' - 30: 1, # 'ğ' - 41: 0, # 'İ' - 6: 3, # 'ı' - 40: 0, # 'Ş' - 19: 0, # 'ş' - }, - 32: { # 'v' - 23: 0, # 'A' - 37: 0, # 'B' - 47: 0, # 'C' - 39: 0, # 'D' - 29: 0, # 'E' - 52: 0, # 'F' - 36: 0, # 'G' - 45: 0, # 'H' - 53: 0, # 'I' - 60: 0, # 'J' - 16: 3, # 'K' - 49: 0, # 'L' - 20: 1, # 'M' - 46: 0, # 'N' - 42: 0, # 'O' - 48: 0, # 'P' - 44: 0, # 'R' - 35: 0, # 'S' - 31: 0, # 'T' - 51: 0, # 'U' - 38: 0, # 'V' - 62: 0, # 'W' - 43: 0, # 'Y' - 56: 0, # 'Z' - 1: 3, # 'a' - 21: 0, # 'b' - 28: 0, # 'c' - 12: 3, # 'd' - 2: 3, # 'e' - 18: 0, # 'f' - 27: 0, # 'g' - 25: 0, # 'h' - 3: 0, # 'i' - 24: 1, # 'j' - 10: 1, # 'k' - 5: 3, # 'l' - 13: 2, # 'm' - 4: 3, # 'n' - 15: 0, # 'o' - 26: 1, # 'p' - 7: 1, # 'r' - 8: 2, # 's' - 9: 3, # 't' - 14: 3, # 'u' - 32: 1, # 'v' - 57: 0, # 'w' - 58: 0, # 'x' - 11: 0, # 'y' - 22: 0, # 'z' - 63: 0, # '·' - 54: 0, # 'Ç' - 50: 0, # 'Ö' - 55: 0, # 'Ü' - 59: 0, # 'â' - 33: 2, # 'ç' - 61: 0, # 'î' - 34: 0, # 'ö' - 17: 0, # 'ü' - 30: 0, # 'ğ' - 41: 0, # 'İ' - 6: 1, # 'ı' - 40: 0, # 'Ş' - 19: 0, # 'ş' - }, - 57: { # 'w' - 23: 0, # 'A' - 37: 0, # 'B' - 47: 0, # 'C' - 39: 0, # 'D' - 29: 0, # 'E' - 52: 0, # 'F' - 36: 0, # 'G' - 45: 0, # 'H' - 53: 0, # 'I' - 60: 0, # 'J' - 16: 0, # 'K' - 49: 0, # 'L' - 20: 0, # 'M' - 46: 0, # 'N' - 42: 0, # 'O' - 48: 0, # 'P' - 44: 0, # 'R' - 35: 0, # 'S' - 31: 0, # 'T' - 51: 1, # 'U' - 38: 0, # 'V' - 62: 0, # 'W' - 43: 0, # 'Y' - 56: 0, # 'Z' - 1: 1, # 'a' - 21: 0, # 'b' - 28: 0, # 'c' - 12: 0, # 'd' - 2: 2, # 'e' - 18: 0, # 'f' - 27: 0, # 'g' - 25: 1, # 'h' - 3: 0, # 'i' - 24: 0, # 'j' - 10: 1, # 'k' - 5: 0, # 'l' - 13: 0, # 'm' - 4: 1, # 'n' - 15: 0, # 'o' - 26: 0, # 'p' - 7: 0, # 'r' - 8: 1, # 's' - 9: 0, # 't' - 14: 1, # 'u' - 32: 0, # 'v' - 57: 2, # 'w' - 58: 0, # 'x' - 11: 0, # 'y' - 22: 0, # 'z' - 63: 1, # '·' - 54: 0, # 'Ç' - 50: 0, # 'Ö' - 55: 0, # 'Ü' - 59: 0, # 'â' - 33: 0, # 'ç' - 61: 0, # 'î' - 34: 0, # 'ö' - 17: 1, # 'ü' - 30: 0, # 'ğ' - 41: 0, # 'İ' - 6: 0, # 'ı' - 40: 0, # 'Ş' - 19: 0, # 'ş' - }, - 58: { # 'x' - 23: 0, # 'A' - 37: 0, # 'B' - 47: 0, # 'C' - 39: 0, # 'D' - 29: 1, # 'E' - 52: 0, # 'F' - 36: 0, # 'G' - 45: 0, # 'H' - 53: 0, # 'I' - 60: 1, # 'J' - 16: 0, # 'K' - 49: 0, # 'L' - 20: 1, # 'M' - 46: 0, # 'N' - 42: 0, # 'O' - 48: 0, # 'P' - 44: 0, # 'R' - 35: 0, # 'S' - 31: 0, # 'T' - 51: 0, # 'U' - 38: 0, # 'V' - 62: 0, # 'W' - 43: 0, # 'Y' - 56: 0, # 'Z' - 1: 0, # 'a' - 21: 1, # 'b' - 28: 0, # 'c' - 12: 2, # 'd' - 2: 1, # 'e' - 18: 0, # 'f' - 27: 0, # 'g' - 25: 0, # 'h' - 3: 2, # 'i' - 24: 2, # 'j' - 10: 1, # 'k' - 5: 0, # 'l' - 13: 0, # 'm' - 4: 2, # 'n' - 15: 0, # 'o' - 26: 0, # 'p' - 7: 1, # 'r' - 8: 2, # 's' - 9: 1, # 't' - 14: 0, # 'u' - 32: 0, # 'v' - 57: 0, # 'w' - 58: 0, # 'x' - 11: 2, # 'y' - 22: 0, # 'z' - 63: 0, # '·' - 54: 0, # 'Ç' - 50: 0, # 'Ö' - 55: 0, # 'Ü' - 59: 0, # 'â' - 33: 0, # 'ç' - 61: 0, # 'î' - 34: 0, # 'ö' - 17: 1, # 'ü' - 30: 0, # 'ğ' - 41: 0, # 'İ' - 6: 2, # 'ı' - 40: 0, # 'Ş' - 19: 0, # 'ş' - }, - 11: { # 'y' - 23: 1, # 'A' - 37: 0, # 'B' - 47: 0, # 'C' - 39: 0, # 'D' - 29: 0, # 'E' - 52: 0, # 'F' - 36: 0, # 'G' - 45: 0, # 'H' - 53: 0, # 'I' - 60: 1, # 'J' - 16: 3, # 'K' - 49: 0, # 'L' - 20: 1, # 'M' - 46: 0, # 'N' - 42: 0, # 'O' - 48: 0, # 'P' - 44: 0, # 'R' - 35: 0, # 'S' - 31: 1, # 'T' - 51: 0, # 'U' - 38: 0, # 'V' - 62: 0, # 'W' - 43: 1, # 'Y' - 56: 1, # 'Z' - 1: 3, # 'a' - 21: 1, # 'b' - 28: 0, # 'c' - 12: 2, # 'd' - 2: 3, # 'e' - 18: 0, # 'f' - 27: 2, # 'g' - 25: 2, # 'h' - 3: 2, # 'i' - 24: 1, # 'j' - 10: 2, # 'k' - 5: 3, # 'l' - 13: 3, # 'm' - 4: 3, # 'n' - 15: 0, # 'o' - 26: 1, # 'p' - 7: 2, # 'r' - 8: 1, # 's' - 9: 2, # 't' - 14: 3, # 'u' - 32: 0, # 'v' - 57: 0, # 'w' - 58: 1, # 'x' - 11: 3, # 'y' - 22: 0, # 'z' - 63: 0, # '·' - 54: 0, # 'Ç' - 50: 0, # 'Ö' - 55: 0, # 'Ü' - 59: 0, # 'â' - 33: 3, # 'ç' - 61: 0, # 'î' - 34: 0, # 'ö' - 17: 2, # 'ü' - 30: 0, # 'ğ' - 41: 0, # 'İ' - 6: 3, # 'ı' - 40: 0, # 'Ş' - 19: 0, # 'ş' - }, - 22: { # 'z' - 23: 2, # 'A' - 37: 2, # 'B' - 47: 1, # 'C' - 39: 2, # 'D' - 29: 3, # 'E' - 52: 1, # 'F' - 36: 2, # 'G' - 45: 2, # 'H' - 53: 1, # 'I' - 60: 0, # 'J' - 16: 0, # 'K' - 49: 0, # 'L' - 20: 3, # 'M' - 46: 2, # 'N' - 42: 2, # 'O' - 48: 2, # 'P' - 44: 1, # 'R' - 35: 1, # 'S' - 31: 3, # 'T' - 51: 2, # 'U' - 38: 2, # 'V' - 62: 0, # 'W' - 43: 2, # 'Y' - 56: 1, # 'Z' - 1: 1, # 'a' - 21: 2, # 'b' - 28: 1, # 'c' - 12: 2, # 'd' - 2: 2, # 'e' - 18: 3, # 'f' - 27: 2, # 'g' - 25: 2, # 'h' - 3: 3, # 'i' - 24: 2, # 'j' - 10: 3, # 'k' - 5: 0, # 'l' - 13: 2, # 'm' - 4: 3, # 'n' - 15: 2, # 'o' - 26: 2, # 'p' - 7: 3, # 'r' - 8: 3, # 's' - 9: 3, # 't' - 14: 0, # 'u' - 32: 2, # 'v' - 57: 0, # 'w' - 58: 0, # 'x' - 11: 3, # 'y' - 22: 2, # 'z' - 63: 1, # '·' - 54: 0, # 'Ç' - 50: 0, # 'Ö' - 55: 2, # 'Ü' - 59: 1, # 'â' - 33: 0, # 'ç' - 61: 0, # 'î' - 34: 2, # 'ö' - 17: 2, # 'ü' - 30: 2, # 'ğ' - 41: 1, # 'İ' - 6: 3, # 'ı' - 40: 1, # 'Ş' - 19: 2, # 'ş' - }, - 63: { # '·' - 23: 0, # 'A' - 37: 0, # 'B' - 47: 0, # 'C' - 39: 0, # 'D' - 29: 0, # 'E' - 52: 0, # 'F' - 36: 0, # 'G' - 45: 0, # 'H' - 53: 0, # 'I' - 60: 0, # 'J' - 16: 0, # 'K' - 49: 0, # 'L' - 20: 0, # 'M' - 46: 0, # 'N' - 42: 0, # 'O' - 48: 0, # 'P' - 44: 0, # 'R' - 35: 0, # 'S' - 31: 0, # 'T' - 51: 0, # 'U' - 38: 0, # 'V' - 62: 0, # 'W' - 43: 0, # 'Y' - 56: 0, # 'Z' - 1: 0, # 'a' - 21: 0, # 'b' - 28: 0, # 'c' - 12: 0, # 'd' - 2: 1, # 'e' - 18: 0, # 'f' - 27: 0, # 'g' - 25: 0, # 'h' - 3: 0, # 'i' - 24: 0, # 'j' - 10: 0, # 'k' - 5: 0, # 'l' - 13: 2, # 'm' - 4: 0, # 'n' - 15: 0, # 'o' - 26: 0, # 'p' - 7: 0, # 'r' - 8: 0, # 's' - 9: 0, # 't' - 14: 2, # 'u' - 32: 0, # 'v' - 57: 0, # 'w' - 58: 0, # 'x' - 11: 0, # 'y' - 22: 0, # 'z' - 63: 0, # '·' - 54: 0, # 'Ç' - 50: 0, # 'Ö' - 55: 0, # 'Ü' - 59: 0, # 'â' - 33: 0, # 'ç' - 61: 0, # 'î' - 34: 0, # 'ö' - 17: 0, # 'ü' - 30: 0, # 'ğ' - 41: 0, # 'İ' - 6: 0, # 'ı' - 40: 0, # 'Ş' - 19: 0, # 'ş' - }, - 54: { # 'Ç' - 23: 0, # 'A' - 37: 0, # 'B' - 47: 1, # 'C' - 39: 1, # 'D' - 29: 0, # 'E' - 52: 0, # 'F' - 36: 1, # 'G' - 45: 1, # 'H' - 53: 1, # 'I' - 60: 0, # 'J' - 16: 0, # 'K' - 49: 0, # 'L' - 20: 0, # 'M' - 46: 0, # 'N' - 42: 1, # 'O' - 48: 1, # 'P' - 44: 0, # 'R' - 35: 0, # 'S' - 31: 0, # 'T' - 51: 1, # 'U' - 38: 1, # 'V' - 62: 0, # 'W' - 43: 2, # 'Y' - 56: 0, # 'Z' - 1: 0, # 'a' - 21: 1, # 'b' - 28: 0, # 'c' - 12: 1, # 'd' - 2: 0, # 'e' - 18: 0, # 'f' - 27: 1, # 'g' - 25: 0, # 'h' - 3: 3, # 'i' - 24: 0, # 'j' - 10: 1, # 'k' - 5: 0, # 'l' - 13: 0, # 'm' - 4: 2, # 'n' - 15: 1, # 'o' - 26: 0, # 'p' - 7: 2, # 'r' - 8: 0, # 's' - 9: 1, # 't' - 14: 0, # 'u' - 32: 2, # 'v' - 57: 0, # 'w' - 58: 0, # 'x' - 11: 0, # 'y' - 22: 0, # 'z' - 63: 0, # '·' - 54: 0, # 'Ç' - 50: 0, # 'Ö' - 55: 2, # 'Ü' - 59: 0, # 'â' - 33: 0, # 'ç' - 61: 0, # 'î' - 34: 1, # 'ö' - 17: 0, # 'ü' - 30: 0, # 'ğ' - 41: 0, # 'İ' - 6: 2, # 'ı' - 40: 0, # 'Ş' - 19: 1, # 'ş' - }, - 50: { # 'Ö' - 23: 0, # 'A' - 37: 0, # 'B' - 47: 1, # 'C' - 39: 1, # 'D' - 29: 2, # 'E' - 52: 0, # 'F' - 36: 1, # 'G' - 45: 2, # 'H' - 53: 0, # 'I' - 60: 0, # 'J' - 16: 0, # 'K' - 49: 0, # 'L' - 20: 1, # 'M' - 46: 1, # 'N' - 42: 2, # 'O' - 48: 2, # 'P' - 44: 1, # 'R' - 35: 0, # 'S' - 31: 0, # 'T' - 51: 1, # 'U' - 38: 1, # 'V' - 62: 0, # 'W' - 43: 2, # 'Y' - 56: 0, # 'Z' - 1: 0, # 'a' - 21: 2, # 'b' - 28: 1, # 'c' - 12: 2, # 'd' - 2: 0, # 'e' - 18: 1, # 'f' - 27: 1, # 'g' - 25: 1, # 'h' - 3: 2, # 'i' - 24: 0, # 'j' - 10: 2, # 'k' - 5: 0, # 'l' - 13: 0, # 'm' - 4: 3, # 'n' - 15: 2, # 'o' - 26: 2, # 'p' - 7: 3, # 'r' - 8: 1, # 's' - 9: 2, # 't' - 14: 0, # 'u' - 32: 1, # 'v' - 57: 0, # 'w' - 58: 0, # 'x' - 11: 0, # 'y' - 22: 1, # 'z' - 63: 0, # '·' - 54: 0, # 'Ç' - 50: 0, # 'Ö' - 55: 0, # 'Ü' - 59: 0, # 'â' - 33: 0, # 'ç' - 61: 0, # 'î' - 34: 2, # 'ö' - 17: 2, # 'ü' - 30: 1, # 'ğ' - 41: 0, # 'İ' - 6: 2, # 'ı' - 40: 0, # 'Ş' - 19: 1, # 'ş' - }, - 55: { # 'Ü' - 23: 0, # 'A' - 37: 0, # 'B' - 47: 0, # 'C' - 39: 0, # 'D' - 29: 0, # 'E' - 52: 2, # 'F' - 36: 0, # 'G' - 45: 0, # 'H' - 53: 0, # 'I' - 60: 0, # 'J' - 16: 1, # 'K' - 49: 0, # 'L' - 20: 0, # 'M' - 46: 0, # 'N' - 42: 0, # 'O' - 48: 1, # 'P' - 44: 0, # 'R' - 35: 0, # 'S' - 31: 0, # 'T' - 51: 0, # 'U' - 38: 1, # 'V' - 62: 0, # 'W' - 43: 0, # 'Y' - 56: 0, # 'Z' - 1: 2, # 'a' - 21: 0, # 'b' - 28: 2, # 'c' - 12: 0, # 'd' - 2: 2, # 'e' - 18: 0, # 'f' - 27: 1, # 'g' - 25: 0, # 'h' - 3: 0, # 'i' - 24: 0, # 'j' - 10: 0, # 'k' - 5: 1, # 'l' - 13: 1, # 'm' - 4: 1, # 'n' - 15: 0, # 'o' - 26: 0, # 'p' - 7: 0, # 'r' - 8: 0, # 's' - 9: 1, # 't' - 14: 2, # 'u' - 32: 0, # 'v' - 57: 0, # 'w' - 58: 0, # 'x' - 11: 0, # 'y' - 22: 1, # 'z' - 63: 0, # '·' - 54: 0, # 'Ç' - 50: 1, # 'Ö' - 55: 0, # 'Ü' - 59: 0, # 'â' - 33: 0, # 'ç' - 61: 0, # 'î' - 34: 1, # 'ö' - 17: 0, # 'ü' - 30: 1, # 'ğ' - 41: 1, # 'İ' - 6: 0, # 'ı' - 40: 0, # 'Ş' - 19: 1, # 'ş' - }, - 59: { # 'â' - 23: 0, # 'A' - 37: 0, # 'B' - 47: 0, # 'C' - 39: 0, # 'D' - 29: 0, # 'E' - 52: 0, # 'F' - 36: 1, # 'G' - 45: 0, # 'H' - 53: 0, # 'I' - 60: 0, # 'J' - 16: 1, # 'K' - 49: 0, # 'L' - 20: 0, # 'M' - 46: 0, # 'N' - 42: 0, # 'O' - 48: 0, # 'P' - 44: 0, # 'R' - 35: 0, # 'S' - 31: 0, # 'T' - 51: 0, # 'U' - 38: 0, # 'V' - 62: 0, # 'W' - 43: 0, # 'Y' - 56: 0, # 'Z' - 1: 2, # 'a' - 21: 0, # 'b' - 28: 0, # 'c' - 12: 0, # 'd' - 2: 2, # 'e' - 18: 0, # 'f' - 27: 0, # 'g' - 25: 0, # 'h' - 3: 0, # 'i' - 24: 0, # 'j' - 10: 0, # 'k' - 5: 0, # 'l' - 13: 2, # 'm' - 4: 0, # 'n' - 15: 1, # 'o' - 26: 0, # 'p' - 7: 0, # 'r' - 8: 0, # 's' - 9: 0, # 't' - 14: 2, # 'u' - 32: 0, # 'v' - 57: 0, # 'w' - 58: 0, # 'x' - 11: 0, # 'y' - 22: 1, # 'z' - 63: 0, # '·' - 54: 0, # 'Ç' - 50: 0, # 'Ö' - 55: 0, # 'Ü' - 59: 0, # 'â' - 33: 0, # 'ç' - 61: 0, # 'î' - 34: 0, # 'ö' - 17: 0, # 'ü' - 30: 0, # 'ğ' - 41: 0, # 'İ' - 6: 1, # 'ı' - 40: 1, # 'Ş' - 19: 0, # 'ş' - }, - 33: { # 'ç' - 23: 0, # 'A' - 37: 0, # 'B' - 47: 0, # 'C' - 39: 0, # 'D' - 29: 3, # 'E' - 52: 0, # 'F' - 36: 0, # 'G' - 45: 0, # 'H' - 53: 0, # 'I' - 60: 0, # 'J' - 16: 0, # 'K' - 49: 0, # 'L' - 20: 1, # 'M' - 46: 0, # 'N' - 42: 0, # 'O' - 48: 0, # 'P' - 44: 0, # 'R' - 35: 0, # 'S' - 31: 2, # 'T' - 51: 0, # 'U' - 38: 1, # 'V' - 62: 0, # 'W' - 43: 0, # 'Y' - 56: 0, # 'Z' - 1: 0, # 'a' - 21: 3, # 'b' - 28: 0, # 'c' - 12: 2, # 'd' - 2: 0, # 'e' - 18: 2, # 'f' - 27: 1, # 'g' - 25: 3, # 'h' - 3: 3, # 'i' - 24: 0, # 'j' - 10: 3, # 'k' - 5: 0, # 'l' - 13: 0, # 'm' - 4: 3, # 'n' - 15: 0, # 'o' - 26: 1, # 'p' - 7: 3, # 'r' - 8: 2, # 's' - 9: 3, # 't' - 14: 0, # 'u' - 32: 2, # 'v' - 57: 0, # 'w' - 58: 0, # 'x' - 11: 2, # 'y' - 22: 0, # 'z' - 63: 0, # '·' - 54: 0, # 'Ç' - 50: 0, # 'Ö' - 55: 0, # 'Ü' - 59: 0, # 'â' - 33: 0, # 'ç' - 61: 0, # 'î' - 34: 0, # 'ö' - 17: 1, # 'ü' - 30: 0, # 'ğ' - 41: 0, # 'İ' - 6: 3, # 'ı' - 40: 0, # 'Ş' - 19: 0, # 'ş' - }, - 61: { # 'î' - 23: 0, # 'A' - 37: 0, # 'B' - 47: 0, # 'C' - 39: 0, # 'D' - 29: 0, # 'E' - 52: 0, # 'F' - 36: 0, # 'G' - 45: 0, # 'H' - 53: 0, # 'I' - 60: 0, # 'J' - 16: 0, # 'K' - 49: 0, # 'L' - 20: 0, # 'M' - 46: 0, # 'N' - 42: 0, # 'O' - 48: 0, # 'P' - 44: 0, # 'R' - 35: 0, # 'S' - 31: 0, # 'T' - 51: 0, # 'U' - 38: 0, # 'V' - 62: 0, # 'W' - 43: 0, # 'Y' - 56: 1, # 'Z' - 1: 2, # 'a' - 21: 0, # 'b' - 28: 0, # 'c' - 12: 0, # 'd' - 2: 2, # 'e' - 18: 0, # 'f' - 27: 0, # 'g' - 25: 0, # 'h' - 3: 0, # 'i' - 24: 1, # 'j' - 10: 0, # 'k' - 5: 0, # 'l' - 13: 1, # 'm' - 4: 1, # 'n' - 15: 0, # 'o' - 26: 0, # 'p' - 7: 0, # 'r' - 8: 0, # 's' - 9: 0, # 't' - 14: 1, # 'u' - 32: 0, # 'v' - 57: 0, # 'w' - 58: 0, # 'x' - 11: 0, # 'y' - 22: 1, # 'z' - 63: 0, # '·' - 54: 0, # 'Ç' - 50: 0, # 'Ö' - 55: 0, # 'Ü' - 59: 0, # 'â' - 33: 0, # 'ç' - 61: 1, # 'î' - 34: 0, # 'ö' - 17: 0, # 'ü' - 30: 0, # 'ğ' - 41: 0, # 'İ' - 6: 1, # 'ı' - 40: 0, # 'Ş' - 19: 0, # 'ş' - }, - 34: { # 'ö' - 23: 0, # 'A' - 37: 1, # 'B' - 47: 1, # 'C' - 39: 0, # 'D' - 29: 0, # 'E' - 52: 2, # 'F' - 36: 1, # 'G' - 45: 1, # 'H' - 53: 0, # 'I' - 60: 0, # 'J' - 16: 3, # 'K' - 49: 1, # 'L' - 20: 0, # 'M' - 46: 1, # 'N' - 42: 1, # 'O' - 48: 2, # 'P' - 44: 1, # 'R' - 35: 1, # 'S' - 31: 1, # 'T' - 51: 1, # 'U' - 38: 1, # 'V' - 62: 0, # 'W' - 43: 0, # 'Y' - 56: 1, # 'Z' - 1: 3, # 'a' - 21: 1, # 'b' - 28: 2, # 'c' - 12: 1, # 'd' - 2: 3, # 'e' - 18: 0, # 'f' - 27: 2, # 'g' - 25: 2, # 'h' - 3: 1, # 'i' - 24: 2, # 'j' - 10: 1, # 'k' - 5: 2, # 'l' - 13: 3, # 'm' - 4: 2, # 'n' - 15: 2, # 'o' - 26: 0, # 'p' - 7: 0, # 'r' - 8: 3, # 's' - 9: 1, # 't' - 14: 3, # 'u' - 32: 0, # 'v' - 57: 0, # 'w' - 58: 0, # 'x' - 11: 1, # 'y' - 22: 2, # 'z' - 63: 0, # '·' - 54: 1, # 'Ç' - 50: 2, # 'Ö' - 55: 0, # 'Ü' - 59: 0, # 'â' - 33: 2, # 'ç' - 61: 0, # 'î' - 34: 2, # 'ö' - 17: 0, # 'ü' - 30: 2, # 'ğ' - 41: 1, # 'İ' - 6: 1, # 'ı' - 40: 2, # 'Ş' - 19: 1, # 'ş' - }, - 17: { # 'ü' - 23: 0, # 'A' - 37: 0, # 'B' - 47: 1, # 'C' - 39: 0, # 'D' - 29: 0, # 'E' - 52: 0, # 'F' - 36: 0, # 'G' - 45: 0, # 'H' - 53: 0, # 'I' - 60: 1, # 'J' - 16: 1, # 'K' - 49: 0, # 'L' - 20: 1, # 'M' - 46: 0, # 'N' - 42: 0, # 'O' - 48: 0, # 'P' - 44: 0, # 'R' - 35: 0, # 'S' - 31: 1, # 'T' - 51: 0, # 'U' - 38: 0, # 'V' - 62: 0, # 'W' - 43: 0, # 'Y' - 56: 1, # 'Z' - 1: 3, # 'a' - 21: 0, # 'b' - 28: 0, # 'c' - 12: 1, # 'd' - 2: 3, # 'e' - 18: 1, # 'f' - 27: 2, # 'g' - 25: 0, # 'h' - 3: 1, # 'i' - 24: 1, # 'j' - 10: 2, # 'k' - 5: 3, # 'l' - 13: 2, # 'm' - 4: 3, # 'n' - 15: 0, # 'o' - 26: 2, # 'p' - 7: 2, # 'r' - 8: 3, # 's' - 9: 2, # 't' - 14: 3, # 'u' - 32: 1, # 'v' - 57: 1, # 'w' - 58: 0, # 'x' - 11: 0, # 'y' - 22: 0, # 'z' - 63: 0, # '·' - 54: 0, # 'Ç' - 50: 0, # 'Ö' - 55: 0, # 'Ü' - 59: 0, # 'â' - 33: 1, # 'ç' - 61: 0, # 'î' - 34: 0, # 'ö' - 17: 2, # 'ü' - 30: 0, # 'ğ' - 41: 0, # 'İ' - 6: 2, # 'ı' - 40: 0, # 'Ş' - 19: 0, # 'ş' - }, - 30: { # 'ğ' - 23: 0, # 'A' - 37: 2, # 'B' - 47: 1, # 'C' - 39: 0, # 'D' - 29: 0, # 'E' - 52: 2, # 'F' - 36: 1, # 'G' - 45: 0, # 'H' - 53: 1, # 'I' - 60: 0, # 'J' - 16: 3, # 'K' - 49: 0, # 'L' - 20: 1, # 'M' - 46: 2, # 'N' - 42: 2, # 'O' - 48: 1, # 'P' - 44: 1, # 'R' - 35: 0, # 'S' - 31: 1, # 'T' - 51: 0, # 'U' - 38: 2, # 'V' - 62: 0, # 'W' - 43: 2, # 'Y' - 56: 0, # 'Z' - 1: 3, # 'a' - 21: 0, # 'b' - 28: 2, # 'c' - 12: 0, # 'd' - 2: 2, # 'e' - 18: 0, # 'f' - 27: 0, # 'g' - 25: 0, # 'h' - 3: 0, # 'i' - 24: 3, # 'j' - 10: 1, # 'k' - 5: 2, # 'l' - 13: 3, # 'm' - 4: 0, # 'n' - 15: 1, # 'o' - 26: 0, # 'p' - 7: 1, # 'r' - 8: 0, # 's' - 9: 0, # 't' - 14: 3, # 'u' - 32: 0, # 'v' - 57: 0, # 'w' - 58: 0, # 'x' - 11: 0, # 'y' - 22: 2, # 'z' - 63: 0, # '·' - 54: 2, # 'Ç' - 50: 2, # 'Ö' - 55: 0, # 'Ü' - 59: 0, # 'â' - 33: 1, # 'ç' - 61: 0, # 'î' - 34: 2, # 'ö' - 17: 0, # 'ü' - 30: 1, # 'ğ' - 41: 2, # 'İ' - 6: 2, # 'ı' - 40: 2, # 'Ş' - 19: 1, # 'ş' - }, - 41: { # 'İ' - 23: 0, # 'A' - 37: 0, # 'B' - 47: 1, # 'C' - 39: 1, # 'D' - 29: 1, # 'E' - 52: 0, # 'F' - 36: 2, # 'G' - 45: 2, # 'H' - 53: 0, # 'I' - 60: 0, # 'J' - 16: 0, # 'K' - 49: 0, # 'L' - 20: 2, # 'M' - 46: 1, # 'N' - 42: 1, # 'O' - 48: 2, # 'P' - 44: 0, # 'R' - 35: 1, # 'S' - 31: 1, # 'T' - 51: 1, # 'U' - 38: 1, # 'V' - 62: 0, # 'W' - 43: 2, # 'Y' - 56: 0, # 'Z' - 1: 1, # 'a' - 21: 2, # 'b' - 28: 1, # 'c' - 12: 2, # 'd' - 2: 1, # 'e' - 18: 0, # 'f' - 27: 3, # 'g' - 25: 2, # 'h' - 3: 2, # 'i' - 24: 2, # 'j' - 10: 2, # 'k' - 5: 0, # 'l' - 13: 1, # 'm' - 4: 3, # 'n' - 15: 1, # 'o' - 26: 1, # 'p' - 7: 3, # 'r' - 8: 3, # 's' - 9: 2, # 't' - 14: 0, # 'u' - 32: 0, # 'v' - 57: 1, # 'w' - 58: 0, # 'x' - 11: 2, # 'y' - 22: 0, # 'z' - 63: 0, # '·' - 54: 0, # 'Ç' - 50: 0, # 'Ö' - 55: 1, # 'Ü' - 59: 1, # 'â' - 33: 0, # 'ç' - 61: 0, # 'î' - 34: 1, # 'ö' - 17: 1, # 'ü' - 30: 2, # 'ğ' - 41: 0, # 'İ' - 6: 3, # 'ı' - 40: 0, # 'Ş' - 19: 1, # 'ş' - }, - 6: { # 'ı' - 23: 2, # 'A' - 37: 0, # 'B' - 47: 0, # 'C' - 39: 0, # 'D' - 29: 0, # 'E' - 52: 0, # 'F' - 36: 1, # 'G' - 45: 0, # 'H' - 53: 0, # 'I' - 60: 2, # 'J' - 16: 3, # 'K' - 49: 0, # 'L' - 20: 3, # 'M' - 46: 1, # 'N' - 42: 0, # 'O' - 48: 0, # 'P' - 44: 0, # 'R' - 35: 0, # 'S' - 31: 2, # 'T' - 51: 0, # 'U' - 38: 0, # 'V' - 62: 0, # 'W' - 43: 2, # 'Y' - 56: 1, # 'Z' - 1: 3, # 'a' - 21: 2, # 'b' - 28: 1, # 'c' - 12: 3, # 'd' - 2: 3, # 'e' - 18: 3, # 'f' - 27: 3, # 'g' - 25: 2, # 'h' - 3: 3, # 'i' - 24: 3, # 'j' - 10: 3, # 'k' - 5: 3, # 'l' - 13: 3, # 'm' - 4: 3, # 'n' - 15: 0, # 'o' - 26: 3, # 'p' - 7: 3, # 'r' - 8: 3, # 's' - 9: 3, # 't' - 14: 3, # 'u' - 32: 3, # 'v' - 57: 1, # 'w' - 58: 1, # 'x' - 11: 3, # 'y' - 22: 0, # 'z' - 63: 1, # '·' - 54: 0, # 'Ç' - 50: 0, # 'Ö' - 55: 0, # 'Ü' - 59: 0, # 'â' - 33: 2, # 'ç' - 61: 0, # 'î' - 34: 0, # 'ö' - 17: 3, # 'ü' - 30: 0, # 'ğ' - 41: 0, # 'İ' - 6: 3, # 'ı' - 40: 0, # 'Ş' - 19: 0, # 'ş' - }, - 40: { # 'Ş' - 23: 0, # 'A' - 37: 0, # 'B' - 47: 1, # 'C' - 39: 1, # 'D' - 29: 1, # 'E' - 52: 0, # 'F' - 36: 1, # 'G' - 45: 2, # 'H' - 53: 1, # 'I' - 60: 0, # 'J' - 16: 0, # 'K' - 49: 0, # 'L' - 20: 2, # 'M' - 46: 1, # 'N' - 42: 1, # 'O' - 48: 2, # 'P' - 44: 2, # 'R' - 35: 1, # 'S' - 31: 1, # 'T' - 51: 0, # 'U' - 38: 1, # 'V' - 62: 0, # 'W' - 43: 2, # 'Y' - 56: 1, # 'Z' - 1: 0, # 'a' - 21: 2, # 'b' - 28: 0, # 'c' - 12: 2, # 'd' - 2: 0, # 'e' - 18: 3, # 'f' - 27: 0, # 'g' - 25: 2, # 'h' - 3: 3, # 'i' - 24: 2, # 'j' - 10: 1, # 'k' - 5: 0, # 'l' - 13: 1, # 'm' - 4: 3, # 'n' - 15: 2, # 'o' - 26: 0, # 'p' - 7: 3, # 'r' - 8: 2, # 's' - 9: 2, # 't' - 14: 1, # 'u' - 32: 3, # 'v' - 57: 0, # 'w' - 58: 0, # 'x' - 11: 2, # 'y' - 22: 0, # 'z' - 63: 0, # '·' - 54: 0, # 'Ç' - 50: 0, # 'Ö' - 55: 1, # 'Ü' - 59: 0, # 'â' - 33: 0, # 'ç' - 61: 0, # 'î' - 34: 2, # 'ö' - 17: 1, # 'ü' - 30: 2, # 'ğ' - 41: 0, # 'İ' - 6: 2, # 'ı' - 40: 1, # 'Ş' - 19: 2, # 'ş' - }, - 19: { # 'ş' - 23: 0, # 'A' - 37: 0, # 'B' - 47: 1, # 'C' - 39: 0, # 'D' - 29: 0, # 'E' - 52: 2, # 'F' - 36: 1, # 'G' - 45: 0, # 'H' - 53: 0, # 'I' - 60: 0, # 'J' - 16: 3, # 'K' - 49: 2, # 'L' - 20: 0, # 'M' - 46: 1, # 'N' - 42: 1, # 'O' - 48: 1, # 'P' - 44: 1, # 'R' - 35: 1, # 'S' - 31: 0, # 'T' - 51: 1, # 'U' - 38: 1, # 'V' - 62: 0, # 'W' - 43: 1, # 'Y' - 56: 0, # 'Z' - 1: 3, # 'a' - 21: 1, # 'b' - 28: 2, # 'c' - 12: 0, # 'd' - 2: 3, # 'e' - 18: 0, # 'f' - 27: 2, # 'g' - 25: 1, # 'h' - 3: 1, # 'i' - 24: 0, # 'j' - 10: 2, # 'k' - 5: 2, # 'l' - 13: 3, # 'm' - 4: 0, # 'n' - 15: 0, # 'o' - 26: 1, # 'p' - 7: 3, # 'r' - 8: 0, # 's' - 9: 0, # 't' - 14: 3, # 'u' - 32: 0, # 'v' - 57: 0, # 'w' - 58: 0, # 'x' - 11: 0, # 'y' - 22: 2, # 'z' - 63: 0, # '·' - 54: 1, # 'Ç' - 50: 2, # 'Ö' - 55: 0, # 'Ü' - 59: 0, # 'â' - 33: 1, # 'ç' - 61: 1, # 'î' - 34: 2, # 'ö' - 17: 0, # 'ü' - 30: 1, # 'ğ' - 41: 1, # 'İ' - 6: 1, # 'ı' - 40: 1, # 'Ş' - 19: 1, # 'ş' - }, -} - -# 255: Undefined characters that did not exist in training text -# 254: Carriage/Return -# 253: symbol (punctuation) that does not belong to word -# 252: 0 - 9 -# 251: Control characters - -# Character Mapping Table(s): -ISO_8859_9_TURKISH_CHAR_TO_ORDER = { - 0: 255, # '\x00' - 1: 255, # '\x01' - 2: 255, # '\x02' - 3: 255, # '\x03' - 4: 255, # '\x04' - 5: 255, # '\x05' - 6: 255, # '\x06' - 7: 255, # '\x07' - 8: 255, # '\x08' - 9: 255, # '\t' - 10: 255, # '\n' - 11: 255, # '\x0b' - 12: 255, # '\x0c' - 13: 255, # '\r' - 14: 255, # '\x0e' - 15: 255, # '\x0f' - 16: 255, # '\x10' - 17: 255, # '\x11' - 18: 255, # '\x12' - 19: 255, # '\x13' - 20: 255, # '\x14' - 21: 255, # '\x15' - 22: 255, # '\x16' - 23: 255, # '\x17' - 24: 255, # '\x18' - 25: 255, # '\x19' - 26: 255, # '\x1a' - 27: 255, # '\x1b' - 28: 255, # '\x1c' - 29: 255, # '\x1d' - 30: 255, # '\x1e' - 31: 255, # '\x1f' - 32: 255, # ' ' - 33: 255, # '!' - 34: 255, # '"' - 35: 255, # '#' - 36: 255, # '$' - 37: 255, # '%' - 38: 255, # '&' - 39: 255, # "'" - 40: 255, # '(' - 41: 255, # ')' - 42: 255, # '*' - 43: 255, # '+' - 44: 255, # ',' - 45: 255, # '-' - 46: 255, # '.' - 47: 255, # '/' - 48: 255, # '0' - 49: 255, # '1' - 50: 255, # '2' - 51: 255, # '3' - 52: 255, # '4' - 53: 255, # '5' - 54: 255, # '6' - 55: 255, # '7' - 56: 255, # '8' - 57: 255, # '9' - 58: 255, # ':' - 59: 255, # ';' - 60: 255, # '<' - 61: 255, # '=' - 62: 255, # '>' - 63: 255, # '?' - 64: 255, # '@' - 65: 23, # 'A' - 66: 37, # 'B' - 67: 47, # 'C' - 68: 39, # 'D' - 69: 29, # 'E' - 70: 52, # 'F' - 71: 36, # 'G' - 72: 45, # 'H' - 73: 53, # 'I' - 74: 60, # 'J' - 75: 16, # 'K' - 76: 49, # 'L' - 77: 20, # 'M' - 78: 46, # 'N' - 79: 42, # 'O' - 80: 48, # 'P' - 81: 69, # 'Q' - 82: 44, # 'R' - 83: 35, # 'S' - 84: 31, # 'T' - 85: 51, # 'U' - 86: 38, # 'V' - 87: 62, # 'W' - 88: 65, # 'X' - 89: 43, # 'Y' - 90: 56, # 'Z' - 91: 255, # '[' - 92: 255, # '\\' - 93: 255, # ']' - 94: 255, # '^' - 95: 255, # '_' - 96: 255, # '`' - 97: 1, # 'a' - 98: 21, # 'b' - 99: 28, # 'c' - 100: 12, # 'd' - 101: 2, # 'e' - 102: 18, # 'f' - 103: 27, # 'g' - 104: 25, # 'h' - 105: 3, # 'i' - 106: 24, # 'j' - 107: 10, # 'k' - 108: 5, # 'l' - 109: 13, # 'm' - 110: 4, # 'n' - 111: 15, # 'o' - 112: 26, # 'p' - 113: 64, # 'q' - 114: 7, # 'r' - 115: 8, # 's' - 116: 9, # 't' - 117: 14, # 'u' - 118: 32, # 'v' - 119: 57, # 'w' - 120: 58, # 'x' - 121: 11, # 'y' - 122: 22, # 'z' - 123: 255, # '{' - 124: 255, # '|' - 125: 255, # '}' - 126: 255, # '~' - 127: 255, # '\x7f' - 128: 180, # '\x80' - 129: 179, # '\x81' - 130: 178, # '\x82' - 131: 177, # '\x83' - 132: 176, # '\x84' - 133: 175, # '\x85' - 134: 174, # '\x86' - 135: 173, # '\x87' - 136: 172, # '\x88' - 137: 171, # '\x89' - 138: 170, # '\x8a' - 139: 169, # '\x8b' - 140: 168, # '\x8c' - 141: 167, # '\x8d' - 142: 166, # '\x8e' - 143: 165, # '\x8f' - 144: 164, # '\x90' - 145: 163, # '\x91' - 146: 162, # '\x92' - 147: 161, # '\x93' - 148: 160, # '\x94' - 149: 159, # '\x95' - 150: 101, # '\x96' - 151: 158, # '\x97' - 152: 157, # '\x98' - 153: 156, # '\x99' - 154: 155, # '\x9a' - 155: 154, # '\x9b' - 156: 153, # '\x9c' - 157: 152, # '\x9d' - 158: 151, # '\x9e' - 159: 106, # '\x9f' - 160: 150, # '\xa0' - 161: 149, # '¡' - 162: 148, # '¢' - 163: 147, # '£' - 164: 146, # '¤' - 165: 145, # '¥' - 166: 144, # '¦' - 167: 100, # '§' - 168: 143, # '¨' - 169: 142, # '©' - 170: 141, # 'ª' - 171: 140, # '«' - 172: 139, # '¬' - 173: 138, # '\xad' - 174: 137, # '®' - 175: 136, # '¯' - 176: 94, # '°' - 177: 80, # '±' - 178: 93, # '²' - 179: 135, # '³' - 180: 105, # '´' - 181: 134, # 'µ' - 182: 133, # '¶' - 183: 63, # '·' - 184: 132, # '¸' - 185: 131, # '¹' - 186: 130, # 'º' - 187: 129, # '»' - 188: 128, # '¼' - 189: 127, # '½' - 190: 126, # '¾' - 191: 125, # '¿' - 192: 124, # 'À' - 193: 104, # 'Á' - 194: 73, # 'Â' - 195: 99, # 'Ã' - 196: 79, # 'Ä' - 197: 85, # 'Å' - 198: 123, # 'Æ' - 199: 54, # 'Ç' - 200: 122, # 'È' - 201: 98, # 'É' - 202: 92, # 'Ê' - 203: 121, # 'Ë' - 204: 120, # 'Ì' - 205: 91, # 'Í' - 206: 103, # 'Î' - 207: 119, # 'Ï' - 208: 68, # 'Ğ' - 209: 118, # 'Ñ' - 210: 117, # 'Ò' - 211: 97, # 'Ó' - 212: 116, # 'Ô' - 213: 115, # 'Õ' - 214: 50, # 'Ö' - 215: 90, # '×' - 216: 114, # 'Ø' - 217: 113, # 'Ù' - 218: 112, # 'Ú' - 219: 111, # 'Û' - 220: 55, # 'Ü' - 221: 41, # 'İ' - 222: 40, # 'Ş' - 223: 86, # 'ß' - 224: 89, # 'à' - 225: 70, # 'á' - 226: 59, # 'â' - 227: 78, # 'ã' - 228: 71, # 'ä' - 229: 82, # 'å' - 230: 88, # 'æ' - 231: 33, # 'ç' - 232: 77, # 'è' - 233: 66, # 'é' - 234: 84, # 'ê' - 235: 83, # 'ë' - 236: 110, # 'ì' - 237: 75, # 'í' - 238: 61, # 'î' - 239: 96, # 'ï' - 240: 30, # 'ğ' - 241: 67, # 'ñ' - 242: 109, # 'ò' - 243: 74, # 'ó' - 244: 87, # 'ô' - 245: 102, # 'õ' - 246: 34, # 'ö' - 247: 95, # '÷' - 248: 81, # 'ø' - 249: 108, # 'ù' - 250: 76, # 'ú' - 251: 72, # 'û' - 252: 17, # 'ü' - 253: 6, # 'ı' - 254: 19, # 'ş' - 255: 107, # 'ÿ' -} - -ISO_8859_9_TURKISH_MODEL = SingleByteCharSetModel( - charset_name="ISO-8859-9", - language="Turkish", - char_to_order_map=ISO_8859_9_TURKISH_CHAR_TO_ORDER, - language_model=TURKISH_LANG_MODEL, - typical_positive_ratio=0.97029, - keep_ascii_letters=True, - alphabet="ABCDEFGHIJKLMNOPRSTUVYZabcdefghijklmnoprstuvyzÂÇÎÖÛÜâçîöûüĞğİıŞş", -) diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/latin1prober.py b/venv/lib/python3.11/site-packages/pip/_vendor/chardet/latin1prober.py deleted file mode 100644 index 59a01d9..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/latin1prober.py +++ /dev/null @@ -1,147 +0,0 @@ -######################## BEGIN LICENSE BLOCK ######################## -# The Original Code is Mozilla Universal charset detector code. -# -# The Initial Developer of the Original Code is -# Netscape Communications Corporation. -# Portions created by the Initial Developer are Copyright (C) 2001 -# the Initial Developer. All Rights Reserved. -# -# Contributor(s): -# Mark Pilgrim - port to Python -# Shy Shalom - original C code -# -# This library is free software; you can redistribute it and/or -# modify it under the terms of the GNU Lesser General Public -# License as published by the Free Software Foundation; either -# version 2.1 of the License, or (at your option) any later version. -# -# This library is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# Lesser General Public License for more details. -# -# You should have received a copy of the GNU Lesser General Public -# License along with this library; if not, write to the Free Software -# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA -# 02110-1301 USA -######################### END LICENSE BLOCK ######################### - -from typing import List, Union - -from .charsetprober import CharSetProber -from .enums import ProbingState - -FREQ_CAT_NUM = 4 - -UDF = 0 # undefined -OTH = 1 # other -ASC = 2 # ascii capital letter -ASS = 3 # ascii small letter -ACV = 4 # accent capital vowel -ACO = 5 # accent capital other -ASV = 6 # accent small vowel -ASO = 7 # accent small other -CLASS_NUM = 8 # total classes - -# fmt: off -Latin1_CharToClass = ( - OTH, OTH, OTH, OTH, OTH, OTH, OTH, OTH, # 00 - 07 - OTH, OTH, OTH, OTH, OTH, OTH, OTH, OTH, # 08 - 0F - OTH, OTH, OTH, OTH, OTH, OTH, OTH, OTH, # 10 - 17 - OTH, OTH, OTH, OTH, OTH, OTH, OTH, OTH, # 18 - 1F - OTH, OTH, OTH, OTH, OTH, OTH, OTH, OTH, # 20 - 27 - OTH, OTH, OTH, OTH, OTH, OTH, OTH, OTH, # 28 - 2F - OTH, OTH, OTH, OTH, OTH, OTH, OTH, OTH, # 30 - 37 - OTH, OTH, OTH, OTH, OTH, OTH, OTH, OTH, # 38 - 3F - OTH, ASC, ASC, ASC, ASC, ASC, ASC, ASC, # 40 - 47 - ASC, ASC, ASC, ASC, ASC, ASC, ASC, ASC, # 48 - 4F - ASC, ASC, ASC, ASC, ASC, ASC, ASC, ASC, # 50 - 57 - ASC, ASC, ASC, OTH, OTH, OTH, OTH, OTH, # 58 - 5F - OTH, ASS, ASS, ASS, ASS, ASS, ASS, ASS, # 60 - 67 - ASS, ASS, ASS, ASS, ASS, ASS, ASS, ASS, # 68 - 6F - ASS, ASS, ASS, ASS, ASS, ASS, ASS, ASS, # 70 - 77 - ASS, ASS, ASS, OTH, OTH, OTH, OTH, OTH, # 78 - 7F - OTH, UDF, OTH, ASO, OTH, OTH, OTH, OTH, # 80 - 87 - OTH, OTH, ACO, OTH, ACO, UDF, ACO, UDF, # 88 - 8F - UDF, OTH, OTH, OTH, OTH, OTH, OTH, OTH, # 90 - 97 - OTH, OTH, ASO, OTH, ASO, UDF, ASO, ACO, # 98 - 9F - OTH, OTH, OTH, OTH, OTH, OTH, OTH, OTH, # A0 - A7 - OTH, OTH, OTH, OTH, OTH, OTH, OTH, OTH, # A8 - AF - OTH, OTH, OTH, OTH, OTH, OTH, OTH, OTH, # B0 - B7 - OTH, OTH, OTH, OTH, OTH, OTH, OTH, OTH, # B8 - BF - ACV, ACV, ACV, ACV, ACV, ACV, ACO, ACO, # C0 - C7 - ACV, ACV, ACV, ACV, ACV, ACV, ACV, ACV, # C8 - CF - ACO, ACO, ACV, ACV, ACV, ACV, ACV, OTH, # D0 - D7 - ACV, ACV, ACV, ACV, ACV, ACO, ACO, ACO, # D8 - DF - ASV, ASV, ASV, ASV, ASV, ASV, ASO, ASO, # E0 - E7 - ASV, ASV, ASV, ASV, ASV, ASV, ASV, ASV, # E8 - EF - ASO, ASO, ASV, ASV, ASV, ASV, ASV, OTH, # F0 - F7 - ASV, ASV, ASV, ASV, ASV, ASO, ASO, ASO, # F8 - FF -) - -# 0 : illegal -# 1 : very unlikely -# 2 : normal -# 3 : very likely -Latin1ClassModel = ( -# UDF OTH ASC ASS ACV ACO ASV ASO - 0, 0, 0, 0, 0, 0, 0, 0, # UDF - 0, 3, 3, 3, 3, 3, 3, 3, # OTH - 0, 3, 3, 3, 3, 3, 3, 3, # ASC - 0, 3, 3, 3, 1, 1, 3, 3, # ASS - 0, 3, 3, 3, 1, 2, 1, 2, # ACV - 0, 3, 3, 3, 3, 3, 3, 3, # ACO - 0, 3, 1, 3, 1, 1, 1, 3, # ASV - 0, 3, 1, 3, 1, 1, 3, 3, # ASO -) -# fmt: on - - -class Latin1Prober(CharSetProber): - def __init__(self) -> None: - super().__init__() - self._last_char_class = OTH - self._freq_counter: List[int] = [] - self.reset() - - def reset(self) -> None: - self._last_char_class = OTH - self._freq_counter = [0] * FREQ_CAT_NUM - super().reset() - - @property - def charset_name(self) -> str: - return "ISO-8859-1" - - @property - def language(self) -> str: - return "" - - def feed(self, byte_str: Union[bytes, bytearray]) -> ProbingState: - byte_str = self.remove_xml_tags(byte_str) - for c in byte_str: - char_class = Latin1_CharToClass[c] - freq = Latin1ClassModel[(self._last_char_class * CLASS_NUM) + char_class] - if freq == 0: - self._state = ProbingState.NOT_ME - break - self._freq_counter[freq] += 1 - self._last_char_class = char_class - - return self.state - - def get_confidence(self) -> float: - if self.state == ProbingState.NOT_ME: - return 0.01 - - total = sum(self._freq_counter) - confidence = ( - 0.0 - if total < 0.01 - else (self._freq_counter[3] - self._freq_counter[1] * 20.0) / total - ) - confidence = max(confidence, 0.0) - # lower the confidence of latin1 so that other more accurate - # detector can take priority. - confidence *= 0.73 - return confidence diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/macromanprober.py b/venv/lib/python3.11/site-packages/pip/_vendor/chardet/macromanprober.py deleted file mode 100644 index 1425d10..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/macromanprober.py +++ /dev/null @@ -1,162 +0,0 @@ -######################## BEGIN LICENSE BLOCK ######################## -# This code was modified from latin1prober.py by Rob Speer . -# The Original Code is Mozilla Universal charset detector code. -# -# The Initial Developer of the Original Code is -# Netscape Communications Corporation. -# Portions created by the Initial Developer are Copyright (C) 2001 -# the Initial Developer. All Rights Reserved. -# -# Contributor(s): -# Rob Speer - adapt to MacRoman encoding -# Mark Pilgrim - port to Python -# Shy Shalom - original C code -# -# This library is free software; you can redistribute it and/or -# modify it under the terms of the GNU Lesser General Public -# License as published by the Free Software Foundation; either -# version 2.1 of the License, or (at your option) any later version. -# -# This library is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# Lesser General Public License for more details. -# -# You should have received a copy of the GNU Lesser General Public -# License along with this library; if not, write to the Free Software -# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA -# 02110-1301 USA -######################### END LICENSE BLOCK ######################### - -from typing import List, Union - -from .charsetprober import CharSetProber -from .enums import ProbingState - -FREQ_CAT_NUM = 4 - -UDF = 0 # undefined -OTH = 1 # other -ASC = 2 # ascii capital letter -ASS = 3 # ascii small letter -ACV = 4 # accent capital vowel -ACO = 5 # accent capital other -ASV = 6 # accent small vowel -ASO = 7 # accent small other -ODD = 8 # character that is unlikely to appear -CLASS_NUM = 9 # total classes - -# The change from Latin1 is that we explicitly look for extended characters -# that are infrequently-occurring symbols, and consider them to always be -# improbable. This should let MacRoman get out of the way of more likely -# encodings in most situations. - -# fmt: off -MacRoman_CharToClass = ( - OTH, OTH, OTH, OTH, OTH, OTH, OTH, OTH, # 00 - 07 - OTH, OTH, OTH, OTH, OTH, OTH, OTH, OTH, # 08 - 0F - OTH, OTH, OTH, OTH, OTH, OTH, OTH, OTH, # 10 - 17 - OTH, OTH, OTH, OTH, OTH, OTH, OTH, OTH, # 18 - 1F - OTH, OTH, OTH, OTH, OTH, OTH, OTH, OTH, # 20 - 27 - OTH, OTH, OTH, OTH, OTH, OTH, OTH, OTH, # 28 - 2F - OTH, OTH, OTH, OTH, OTH, OTH, OTH, OTH, # 30 - 37 - OTH, OTH, OTH, OTH, OTH, OTH, OTH, OTH, # 38 - 3F - OTH, ASC, ASC, ASC, ASC, ASC, ASC, ASC, # 40 - 47 - ASC, ASC, ASC, ASC, ASC, ASC, ASC, ASC, # 48 - 4F - ASC, ASC, ASC, ASC, ASC, ASC, ASC, ASC, # 50 - 57 - ASC, ASC, ASC, OTH, OTH, OTH, OTH, OTH, # 58 - 5F - OTH, ASS, ASS, ASS, ASS, ASS, ASS, ASS, # 60 - 67 - ASS, ASS, ASS, ASS, ASS, ASS, ASS, ASS, # 68 - 6F - ASS, ASS, ASS, ASS, ASS, ASS, ASS, ASS, # 70 - 77 - ASS, ASS, ASS, OTH, OTH, OTH, OTH, OTH, # 78 - 7F - ACV, ACV, ACO, ACV, ACO, ACV, ACV, ASV, # 80 - 87 - ASV, ASV, ASV, ASV, ASV, ASO, ASV, ASV, # 88 - 8F - ASV, ASV, ASV, ASV, ASV, ASV, ASO, ASV, # 90 - 97 - ASV, ASV, ASV, ASV, ASV, ASV, ASV, ASV, # 98 - 9F - OTH, OTH, OTH, OTH, OTH, OTH, OTH, ASO, # A0 - A7 - OTH, OTH, ODD, ODD, OTH, OTH, ACV, ACV, # A8 - AF - OTH, OTH, OTH, OTH, OTH, OTH, OTH, OTH, # B0 - B7 - OTH, OTH, OTH, OTH, OTH, OTH, ASV, ASV, # B8 - BF - OTH, OTH, ODD, OTH, ODD, OTH, OTH, OTH, # C0 - C7 - OTH, OTH, OTH, ACV, ACV, ACV, ACV, ASV, # C8 - CF - OTH, OTH, OTH, OTH, OTH, OTH, OTH, ODD, # D0 - D7 - ASV, ACV, ODD, OTH, OTH, OTH, OTH, OTH, # D8 - DF - OTH, OTH, OTH, OTH, OTH, ACV, ACV, ACV, # E0 - E7 - ACV, ACV, ACV, ACV, ACV, ACV, ACV, ACV, # E8 - EF - ODD, ACV, ACV, ACV, ACV, ASV, ODD, ODD, # F0 - F7 - ODD, ODD, ODD, ODD, ODD, ODD, ODD, ODD, # F8 - FF -) - -# 0 : illegal -# 1 : very unlikely -# 2 : normal -# 3 : very likely -MacRomanClassModel = ( -# UDF OTH ASC ASS ACV ACO ASV ASO ODD - 0, 0, 0, 0, 0, 0, 0, 0, 0, # UDF - 0, 3, 3, 3, 3, 3, 3, 3, 1, # OTH - 0, 3, 3, 3, 3, 3, 3, 3, 1, # ASC - 0, 3, 3, 3, 1, 1, 3, 3, 1, # ASS - 0, 3, 3, 3, 1, 2, 1, 2, 1, # ACV - 0, 3, 3, 3, 3, 3, 3, 3, 1, # ACO - 0, 3, 1, 3, 1, 1, 1, 3, 1, # ASV - 0, 3, 1, 3, 1, 1, 3, 3, 1, # ASO - 0, 1, 1, 1, 1, 1, 1, 1, 1, # ODD -) -# fmt: on - - -class MacRomanProber(CharSetProber): - def __init__(self) -> None: - super().__init__() - self._last_char_class = OTH - self._freq_counter: List[int] = [] - self.reset() - - def reset(self) -> None: - self._last_char_class = OTH - self._freq_counter = [0] * FREQ_CAT_NUM - - # express the prior that MacRoman is a somewhat rare encoding; - # this can be done by starting out in a slightly improbable state - # that must be overcome - self._freq_counter[2] = 10 - - super().reset() - - @property - def charset_name(self) -> str: - return "MacRoman" - - @property - def language(self) -> str: - return "" - - def feed(self, byte_str: Union[bytes, bytearray]) -> ProbingState: - byte_str = self.remove_xml_tags(byte_str) - for c in byte_str: - char_class = MacRoman_CharToClass[c] - freq = MacRomanClassModel[(self._last_char_class * CLASS_NUM) + char_class] - if freq == 0: - self._state = ProbingState.NOT_ME - break - self._freq_counter[freq] += 1 - self._last_char_class = char_class - - return self.state - - def get_confidence(self) -> float: - if self.state == ProbingState.NOT_ME: - return 0.01 - - total = sum(self._freq_counter) - confidence = ( - 0.0 - if total < 0.01 - else (self._freq_counter[3] - self._freq_counter[1] * 20.0) / total - ) - confidence = max(confidence, 0.0) - # lower the confidence of MacRoman so that other more accurate - # detector can take priority. - confidence *= 0.73 - return confidence diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/mbcharsetprober.py b/venv/lib/python3.11/site-packages/pip/_vendor/chardet/mbcharsetprober.py deleted file mode 100644 index 666307e..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/mbcharsetprober.py +++ /dev/null @@ -1,95 +0,0 @@ -######################## BEGIN LICENSE BLOCK ######################## -# The Original Code is Mozilla Universal charset detector code. -# -# The Initial Developer of the Original Code is -# Netscape Communications Corporation. -# Portions created by the Initial Developer are Copyright (C) 2001 -# the Initial Developer. All Rights Reserved. -# -# Contributor(s): -# Mark Pilgrim - port to Python -# Shy Shalom - original C code -# Proofpoint, Inc. -# -# This library is free software; you can redistribute it and/or -# modify it under the terms of the GNU Lesser General Public -# License as published by the Free Software Foundation; either -# version 2.1 of the License, or (at your option) any later version. -# -# This library is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# Lesser General Public License for more details. -# -# You should have received a copy of the GNU Lesser General Public -# License along with this library; if not, write to the Free Software -# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA -# 02110-1301 USA -######################### END LICENSE BLOCK ######################### - -from typing import Optional, Union - -from .chardistribution import CharDistributionAnalysis -from .charsetprober import CharSetProber -from .codingstatemachine import CodingStateMachine -from .enums import LanguageFilter, MachineState, ProbingState - - -class MultiByteCharSetProber(CharSetProber): - """ - MultiByteCharSetProber - """ - - def __init__(self, lang_filter: LanguageFilter = LanguageFilter.NONE) -> None: - super().__init__(lang_filter=lang_filter) - self.distribution_analyzer: Optional[CharDistributionAnalysis] = None - self.coding_sm: Optional[CodingStateMachine] = None - self._last_char = bytearray(b"\0\0") - - def reset(self) -> None: - super().reset() - if self.coding_sm: - self.coding_sm.reset() - if self.distribution_analyzer: - self.distribution_analyzer.reset() - self._last_char = bytearray(b"\0\0") - - def feed(self, byte_str: Union[bytes, bytearray]) -> ProbingState: - assert self.coding_sm is not None - assert self.distribution_analyzer is not None - - for i, byte in enumerate(byte_str): - coding_state = self.coding_sm.next_state(byte) - if coding_state == MachineState.ERROR: - self.logger.debug( - "%s %s prober hit error at byte %s", - self.charset_name, - self.language, - i, - ) - self._state = ProbingState.NOT_ME - break - if coding_state == MachineState.ITS_ME: - self._state = ProbingState.FOUND_IT - break - if coding_state == MachineState.START: - char_len = self.coding_sm.get_current_charlen() - if i == 0: - self._last_char[1] = byte - self.distribution_analyzer.feed(self._last_char, char_len) - else: - self.distribution_analyzer.feed(byte_str[i - 1 : i + 1], char_len) - - self._last_char[0] = byte_str[-1] - - if self.state == ProbingState.DETECTING: - if self.distribution_analyzer.got_enough_data() and ( - self.get_confidence() > self.SHORTCUT_THRESHOLD - ): - self._state = ProbingState.FOUND_IT - - return self.state - - def get_confidence(self) -> float: - assert self.distribution_analyzer is not None - return self.distribution_analyzer.get_confidence() diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/mbcsgroupprober.py b/venv/lib/python3.11/site-packages/pip/_vendor/chardet/mbcsgroupprober.py deleted file mode 100644 index 6cb9cc7..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/mbcsgroupprober.py +++ /dev/null @@ -1,57 +0,0 @@ -######################## BEGIN LICENSE BLOCK ######################## -# The Original Code is Mozilla Universal charset detector code. -# -# The Initial Developer of the Original Code is -# Netscape Communications Corporation. -# Portions created by the Initial Developer are Copyright (C) 2001 -# the Initial Developer. All Rights Reserved. -# -# Contributor(s): -# Mark Pilgrim - port to Python -# Shy Shalom - original C code -# Proofpoint, Inc. -# -# This library is free software; you can redistribute it and/or -# modify it under the terms of the GNU Lesser General Public -# License as published by the Free Software Foundation; either -# version 2.1 of the License, or (at your option) any later version. -# -# This library is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# Lesser General Public License for more details. -# -# You should have received a copy of the GNU Lesser General Public -# License along with this library; if not, write to the Free Software -# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA -# 02110-1301 USA -######################### END LICENSE BLOCK ######################### - -from .big5prober import Big5Prober -from .charsetgroupprober import CharSetGroupProber -from .cp949prober import CP949Prober -from .enums import LanguageFilter -from .eucjpprober import EUCJPProber -from .euckrprober import EUCKRProber -from .euctwprober import EUCTWProber -from .gb2312prober import GB2312Prober -from .johabprober import JOHABProber -from .sjisprober import SJISProber -from .utf8prober import UTF8Prober - - -class MBCSGroupProber(CharSetGroupProber): - def __init__(self, lang_filter: LanguageFilter = LanguageFilter.NONE) -> None: - super().__init__(lang_filter=lang_filter) - self.probers = [ - UTF8Prober(), - SJISProber(), - EUCJPProber(), - GB2312Prober(), - EUCKRProber(), - CP949Prober(), - Big5Prober(), - EUCTWProber(), - JOHABProber(), - ] - self.reset() diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/mbcssm.py b/venv/lib/python3.11/site-packages/pip/_vendor/chardet/mbcssm.py deleted file mode 100644 index 7bbe97e..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/mbcssm.py +++ /dev/null @@ -1,661 +0,0 @@ -######################## BEGIN LICENSE BLOCK ######################## -# The Original Code is mozilla.org code. -# -# The Initial Developer of the Original Code is -# Netscape Communications Corporation. -# Portions created by the Initial Developer are Copyright (C) 1998 -# the Initial Developer. All Rights Reserved. -# -# Contributor(s): -# Mark Pilgrim - port to Python -# -# This library is free software; you can redistribute it and/or -# modify it under the terms of the GNU Lesser General Public -# License as published by the Free Software Foundation; either -# version 2.1 of the License, or (at your option) any later version. -# -# This library is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# Lesser General Public License for more details. -# -# You should have received a copy of the GNU Lesser General Public -# License along with this library; if not, write to the Free Software -# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA -# 02110-1301 USA -######################### END LICENSE BLOCK ######################### - -from .codingstatemachinedict import CodingStateMachineDict -from .enums import MachineState - -# BIG5 - -# fmt: off -BIG5_CLS = ( - 1, 1, 1, 1, 1, 1, 1, 1, # 00 - 07 #allow 0x00 as legal value - 1, 1, 1, 1, 1, 1, 0, 0, # 08 - 0f - 1, 1, 1, 1, 1, 1, 1, 1, # 10 - 17 - 1, 1, 1, 0, 1, 1, 1, 1, # 18 - 1f - 1, 1, 1, 1, 1, 1, 1, 1, # 20 - 27 - 1, 1, 1, 1, 1, 1, 1, 1, # 28 - 2f - 1, 1, 1, 1, 1, 1, 1, 1, # 30 - 37 - 1, 1, 1, 1, 1, 1, 1, 1, # 38 - 3f - 2, 2, 2, 2, 2, 2, 2, 2, # 40 - 47 - 2, 2, 2, 2, 2, 2, 2, 2, # 48 - 4f - 2, 2, 2, 2, 2, 2, 2, 2, # 50 - 57 - 2, 2, 2, 2, 2, 2, 2, 2, # 58 - 5f - 2, 2, 2, 2, 2, 2, 2, 2, # 60 - 67 - 2, 2, 2, 2, 2, 2, 2, 2, # 68 - 6f - 2, 2, 2, 2, 2, 2, 2, 2, # 70 - 77 - 2, 2, 2, 2, 2, 2, 2, 1, # 78 - 7f - 4, 4, 4, 4, 4, 4, 4, 4, # 80 - 87 - 4, 4, 4, 4, 4, 4, 4, 4, # 88 - 8f - 4, 4, 4, 4, 4, 4, 4, 4, # 90 - 97 - 4, 4, 4, 4, 4, 4, 4, 4, # 98 - 9f - 4, 3, 3, 3, 3, 3, 3, 3, # a0 - a7 - 3, 3, 3, 3, 3, 3, 3, 3, # a8 - af - 3, 3, 3, 3, 3, 3, 3, 3, # b0 - b7 - 3, 3, 3, 3, 3, 3, 3, 3, # b8 - bf - 3, 3, 3, 3, 3, 3, 3, 3, # c0 - c7 - 3, 3, 3, 3, 3, 3, 3, 3, # c8 - cf - 3, 3, 3, 3, 3, 3, 3, 3, # d0 - d7 - 3, 3, 3, 3, 3, 3, 3, 3, # d8 - df - 3, 3, 3, 3, 3, 3, 3, 3, # e0 - e7 - 3, 3, 3, 3, 3, 3, 3, 3, # e8 - ef - 3, 3, 3, 3, 3, 3, 3, 3, # f0 - f7 - 3, 3, 3, 3, 3, 3, 3, 0 # f8 - ff -) - -BIG5_ST = ( - MachineState.ERROR,MachineState.START,MachineState.START, 3,MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,#00-07 - MachineState.ERROR,MachineState.ERROR,MachineState.ITS_ME,MachineState.ITS_ME,MachineState.ITS_ME,MachineState.ITS_ME,MachineState.ITS_ME,MachineState.ERROR,#08-0f - MachineState.ERROR,MachineState.START,MachineState.START,MachineState.START,MachineState.START,MachineState.START,MachineState.START,MachineState.START#10-17 -) -# fmt: on - -BIG5_CHAR_LEN_TABLE = (0, 1, 1, 2, 0) - -BIG5_SM_MODEL: CodingStateMachineDict = { - "class_table": BIG5_CLS, - "class_factor": 5, - "state_table": BIG5_ST, - "char_len_table": BIG5_CHAR_LEN_TABLE, - "name": "Big5", -} - -# CP949 -# fmt: off -CP949_CLS = ( - 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, # 00 - 0f - 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, # 10 - 1f - 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, # 20 - 2f - 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, # 30 - 3f - 1, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, # 40 - 4f - 4, 4, 5, 5, 5, 5, 5, 5, 5, 5, 5, 1, 1, 1, 1, 1, # 50 - 5f - 1, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, # 60 - 6f - 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 1, 1, 1, 1, 1, # 70 - 7f - 0, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, # 80 - 8f - 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, # 90 - 9f - 6, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 8, 8, 8, # a0 - af - 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, # b0 - bf - 7, 7, 7, 7, 7, 7, 9, 2, 2, 3, 2, 2, 2, 2, 2, 2, # c0 - cf - 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, # d0 - df - 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, # e0 - ef - 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 0, # f0 - ff -) - -CP949_ST = ( -#cls= 0 1 2 3 4 5 6 7 8 9 # previous state = - MachineState.ERROR,MachineState.START, 3,MachineState.ERROR,MachineState.START,MachineState.START, 4, 5,MachineState.ERROR, 6, # MachineState.START - MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,MachineState.ERROR, # MachineState.ERROR - MachineState.ITS_ME,MachineState.ITS_ME,MachineState.ITS_ME,MachineState.ITS_ME,MachineState.ITS_ME,MachineState.ITS_ME,MachineState.ITS_ME,MachineState.ITS_ME,MachineState.ITS_ME,MachineState.ITS_ME, # MachineState.ITS_ME - MachineState.ERROR,MachineState.ERROR,MachineState.START,MachineState.START,MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,MachineState.START,MachineState.START,MachineState.START, # 3 - MachineState.ERROR,MachineState.ERROR,MachineState.START,MachineState.START,MachineState.START,MachineState.START,MachineState.START,MachineState.START,MachineState.START,MachineState.START, # 4 - MachineState.ERROR,MachineState.START,MachineState.START,MachineState.START,MachineState.START,MachineState.START,MachineState.START,MachineState.START,MachineState.START,MachineState.START, # 5 - MachineState.ERROR,MachineState.START,MachineState.START,MachineState.START,MachineState.START,MachineState.ERROR,MachineState.ERROR,MachineState.START,MachineState.START,MachineState.START, # 6 -) -# fmt: on - -CP949_CHAR_LEN_TABLE = (0, 1, 2, 0, 1, 1, 2, 2, 0, 2) - -CP949_SM_MODEL: CodingStateMachineDict = { - "class_table": CP949_CLS, - "class_factor": 10, - "state_table": CP949_ST, - "char_len_table": CP949_CHAR_LEN_TABLE, - "name": "CP949", -} - -# EUC-JP -# fmt: off -EUCJP_CLS = ( - 4, 4, 4, 4, 4, 4, 4, 4, # 00 - 07 - 4, 4, 4, 4, 4, 4, 5, 5, # 08 - 0f - 4, 4, 4, 4, 4, 4, 4, 4, # 10 - 17 - 4, 4, 4, 5, 4, 4, 4, 4, # 18 - 1f - 4, 4, 4, 4, 4, 4, 4, 4, # 20 - 27 - 4, 4, 4, 4, 4, 4, 4, 4, # 28 - 2f - 4, 4, 4, 4, 4, 4, 4, 4, # 30 - 37 - 4, 4, 4, 4, 4, 4, 4, 4, # 38 - 3f - 4, 4, 4, 4, 4, 4, 4, 4, # 40 - 47 - 4, 4, 4, 4, 4, 4, 4, 4, # 48 - 4f - 4, 4, 4, 4, 4, 4, 4, 4, # 50 - 57 - 4, 4, 4, 4, 4, 4, 4, 4, # 58 - 5f - 4, 4, 4, 4, 4, 4, 4, 4, # 60 - 67 - 4, 4, 4, 4, 4, 4, 4, 4, # 68 - 6f - 4, 4, 4, 4, 4, 4, 4, 4, # 70 - 77 - 4, 4, 4, 4, 4, 4, 4, 4, # 78 - 7f - 5, 5, 5, 5, 5, 5, 5, 5, # 80 - 87 - 5, 5, 5, 5, 5, 5, 1, 3, # 88 - 8f - 5, 5, 5, 5, 5, 5, 5, 5, # 90 - 97 - 5, 5, 5, 5, 5, 5, 5, 5, # 98 - 9f - 5, 2, 2, 2, 2, 2, 2, 2, # a0 - a7 - 2, 2, 2, 2, 2, 2, 2, 2, # a8 - af - 2, 2, 2, 2, 2, 2, 2, 2, # b0 - b7 - 2, 2, 2, 2, 2, 2, 2, 2, # b8 - bf - 2, 2, 2, 2, 2, 2, 2, 2, # c0 - c7 - 2, 2, 2, 2, 2, 2, 2, 2, # c8 - cf - 2, 2, 2, 2, 2, 2, 2, 2, # d0 - d7 - 2, 2, 2, 2, 2, 2, 2, 2, # d8 - df - 0, 0, 0, 0, 0, 0, 0, 0, # e0 - e7 - 0, 0, 0, 0, 0, 0, 0, 0, # e8 - ef - 0, 0, 0, 0, 0, 0, 0, 0, # f0 - f7 - 0, 0, 0, 0, 0, 0, 0, 5 # f8 - ff -) - -EUCJP_ST = ( - 3, 4, 3, 5,MachineState.START,MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,#00-07 - MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,MachineState.ITS_ME,MachineState.ITS_ME,MachineState.ITS_ME,MachineState.ITS_ME,#08-0f - MachineState.ITS_ME,MachineState.ITS_ME,MachineState.START,MachineState.ERROR,MachineState.START,MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,#10-17 - MachineState.ERROR,MachineState.ERROR,MachineState.START,MachineState.ERROR,MachineState.ERROR,MachineState.ERROR, 3,MachineState.ERROR,#18-1f - 3,MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,MachineState.START,MachineState.START,MachineState.START,MachineState.START#20-27 -) -# fmt: on - -EUCJP_CHAR_LEN_TABLE = (2, 2, 2, 3, 1, 0) - -EUCJP_SM_MODEL: CodingStateMachineDict = { - "class_table": EUCJP_CLS, - "class_factor": 6, - "state_table": EUCJP_ST, - "char_len_table": EUCJP_CHAR_LEN_TABLE, - "name": "EUC-JP", -} - -# EUC-KR -# fmt: off -EUCKR_CLS = ( - 1, 1, 1, 1, 1, 1, 1, 1, # 00 - 07 - 1, 1, 1, 1, 1, 1, 0, 0, # 08 - 0f - 1, 1, 1, 1, 1, 1, 1, 1, # 10 - 17 - 1, 1, 1, 0, 1, 1, 1, 1, # 18 - 1f - 1, 1, 1, 1, 1, 1, 1, 1, # 20 - 27 - 1, 1, 1, 1, 1, 1, 1, 1, # 28 - 2f - 1, 1, 1, 1, 1, 1, 1, 1, # 30 - 37 - 1, 1, 1, 1, 1, 1, 1, 1, # 38 - 3f - 1, 1, 1, 1, 1, 1, 1, 1, # 40 - 47 - 1, 1, 1, 1, 1, 1, 1, 1, # 48 - 4f - 1, 1, 1, 1, 1, 1, 1, 1, # 50 - 57 - 1, 1, 1, 1, 1, 1, 1, 1, # 58 - 5f - 1, 1, 1, 1, 1, 1, 1, 1, # 60 - 67 - 1, 1, 1, 1, 1, 1, 1, 1, # 68 - 6f - 1, 1, 1, 1, 1, 1, 1, 1, # 70 - 77 - 1, 1, 1, 1, 1, 1, 1, 1, # 78 - 7f - 0, 0, 0, 0, 0, 0, 0, 0, # 80 - 87 - 0, 0, 0, 0, 0, 0, 0, 0, # 88 - 8f - 0, 0, 0, 0, 0, 0, 0, 0, # 90 - 97 - 0, 0, 0, 0, 0, 0, 0, 0, # 98 - 9f - 0, 2, 2, 2, 2, 2, 2, 2, # a0 - a7 - 2, 2, 2, 2, 2, 3, 3, 3, # a8 - af - 2, 2, 2, 2, 2, 2, 2, 2, # b0 - b7 - 2, 2, 2, 2, 2, 2, 2, 2, # b8 - bf - 2, 2, 2, 2, 2, 2, 2, 2, # c0 - c7 - 2, 3, 2, 2, 2, 2, 2, 2, # c8 - cf - 2, 2, 2, 2, 2, 2, 2, 2, # d0 - d7 - 2, 2, 2, 2, 2, 2, 2, 2, # d8 - df - 2, 2, 2, 2, 2, 2, 2, 2, # e0 - e7 - 2, 2, 2, 2, 2, 2, 2, 2, # e8 - ef - 2, 2, 2, 2, 2, 2, 2, 2, # f0 - f7 - 2, 2, 2, 2, 2, 2, 2, 0 # f8 - ff -) - -EUCKR_ST = ( - MachineState.ERROR,MachineState.START, 3,MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,#00-07 - MachineState.ITS_ME,MachineState.ITS_ME,MachineState.ITS_ME,MachineState.ITS_ME,MachineState.ERROR,MachineState.ERROR,MachineState.START,MachineState.START #08-0f -) -# fmt: on - -EUCKR_CHAR_LEN_TABLE = (0, 1, 2, 0) - -EUCKR_SM_MODEL: CodingStateMachineDict = { - "class_table": EUCKR_CLS, - "class_factor": 4, - "state_table": EUCKR_ST, - "char_len_table": EUCKR_CHAR_LEN_TABLE, - "name": "EUC-KR", -} - -# JOHAB -# fmt: off -JOHAB_CLS = ( - 4,4,4,4,4,4,4,4, # 00 - 07 - 4,4,4,4,4,4,0,0, # 08 - 0f - 4,4,4,4,4,4,4,4, # 10 - 17 - 4,4,4,0,4,4,4,4, # 18 - 1f - 4,4,4,4,4,4,4,4, # 20 - 27 - 4,4,4,4,4,4,4,4, # 28 - 2f - 4,3,3,3,3,3,3,3, # 30 - 37 - 3,3,3,3,3,3,3,3, # 38 - 3f - 3,1,1,1,1,1,1,1, # 40 - 47 - 1,1,1,1,1,1,1,1, # 48 - 4f - 1,1,1,1,1,1,1,1, # 50 - 57 - 1,1,1,1,1,1,1,1, # 58 - 5f - 1,1,1,1,1,1,1,1, # 60 - 67 - 1,1,1,1,1,1,1,1, # 68 - 6f - 1,1,1,1,1,1,1,1, # 70 - 77 - 1,1,1,1,1,1,1,2, # 78 - 7f - 6,6,6,6,8,8,8,8, # 80 - 87 - 8,8,8,8,8,8,8,8, # 88 - 8f - 8,7,7,7,7,7,7,7, # 90 - 97 - 7,7,7,7,7,7,7,7, # 98 - 9f - 7,7,7,7,7,7,7,7, # a0 - a7 - 7,7,7,7,7,7,7,7, # a8 - af - 7,7,7,7,7,7,7,7, # b0 - b7 - 7,7,7,7,7,7,7,7, # b8 - bf - 7,7,7,7,7,7,7,7, # c0 - c7 - 7,7,7,7,7,7,7,7, # c8 - cf - 7,7,7,7,5,5,5,5, # d0 - d7 - 5,9,9,9,9,9,9,5, # d8 - df - 9,9,9,9,9,9,9,9, # e0 - e7 - 9,9,9,9,9,9,9,9, # e8 - ef - 9,9,9,9,9,9,9,9, # f0 - f7 - 9,9,5,5,5,5,5,0 # f8 - ff -) - -JOHAB_ST = ( -# cls = 0 1 2 3 4 5 6 7 8 9 - MachineState.ERROR ,MachineState.START ,MachineState.START ,MachineState.START ,MachineState.START ,MachineState.ERROR ,MachineState.ERROR ,3 ,3 ,4 , # MachineState.START - MachineState.ITS_ME,MachineState.ITS_ME,MachineState.ITS_ME,MachineState.ITS_ME,MachineState.ITS_ME,MachineState.ITS_ME,MachineState.ITS_ME,MachineState.ITS_ME,MachineState.ITS_ME,MachineState.ITS_ME, # MachineState.ITS_ME - MachineState.ERROR ,MachineState.ERROR ,MachineState.ERROR ,MachineState.ERROR ,MachineState.ERROR ,MachineState.ERROR ,MachineState.ERROR ,MachineState.ERROR ,MachineState.ERROR ,MachineState.ERROR , # MachineState.ERROR - MachineState.ERROR ,MachineState.START ,MachineState.START ,MachineState.ERROR ,MachineState.ERROR ,MachineState.START ,MachineState.START ,MachineState.START ,MachineState.START ,MachineState.START , # 3 - MachineState.ERROR ,MachineState.START ,MachineState.ERROR ,MachineState.START ,MachineState.ERROR ,MachineState.START ,MachineState.ERROR ,MachineState.START ,MachineState.ERROR ,MachineState.START , # 4 -) -# fmt: on - -JOHAB_CHAR_LEN_TABLE = (0, 1, 1, 1, 1, 0, 0, 2, 2, 2) - -JOHAB_SM_MODEL: CodingStateMachineDict = { - "class_table": JOHAB_CLS, - "class_factor": 10, - "state_table": JOHAB_ST, - "char_len_table": JOHAB_CHAR_LEN_TABLE, - "name": "Johab", -} - -# EUC-TW -# fmt: off -EUCTW_CLS = ( - 2, 2, 2, 2, 2, 2, 2, 2, # 00 - 07 - 2, 2, 2, 2, 2, 2, 0, 0, # 08 - 0f - 2, 2, 2, 2, 2, 2, 2, 2, # 10 - 17 - 2, 2, 2, 0, 2, 2, 2, 2, # 18 - 1f - 2, 2, 2, 2, 2, 2, 2, 2, # 20 - 27 - 2, 2, 2, 2, 2, 2, 2, 2, # 28 - 2f - 2, 2, 2, 2, 2, 2, 2, 2, # 30 - 37 - 2, 2, 2, 2, 2, 2, 2, 2, # 38 - 3f - 2, 2, 2, 2, 2, 2, 2, 2, # 40 - 47 - 2, 2, 2, 2, 2, 2, 2, 2, # 48 - 4f - 2, 2, 2, 2, 2, 2, 2, 2, # 50 - 57 - 2, 2, 2, 2, 2, 2, 2, 2, # 58 - 5f - 2, 2, 2, 2, 2, 2, 2, 2, # 60 - 67 - 2, 2, 2, 2, 2, 2, 2, 2, # 68 - 6f - 2, 2, 2, 2, 2, 2, 2, 2, # 70 - 77 - 2, 2, 2, 2, 2, 2, 2, 2, # 78 - 7f - 0, 0, 0, 0, 0, 0, 0, 0, # 80 - 87 - 0, 0, 0, 0, 0, 0, 6, 0, # 88 - 8f - 0, 0, 0, 0, 0, 0, 0, 0, # 90 - 97 - 0, 0, 0, 0, 0, 0, 0, 0, # 98 - 9f - 0, 3, 4, 4, 4, 4, 4, 4, # a0 - a7 - 5, 5, 1, 1, 1, 1, 1, 1, # a8 - af - 1, 1, 1, 1, 1, 1, 1, 1, # b0 - b7 - 1, 1, 1, 1, 1, 1, 1, 1, # b8 - bf - 1, 1, 3, 1, 3, 3, 3, 3, # c0 - c7 - 3, 3, 3, 3, 3, 3, 3, 3, # c8 - cf - 3, 3, 3, 3, 3, 3, 3, 3, # d0 - d7 - 3, 3, 3, 3, 3, 3, 3, 3, # d8 - df - 3, 3, 3, 3, 3, 3, 3, 3, # e0 - e7 - 3, 3, 3, 3, 3, 3, 3, 3, # e8 - ef - 3, 3, 3, 3, 3, 3, 3, 3, # f0 - f7 - 3, 3, 3, 3, 3, 3, 3, 0 # f8 - ff -) - -EUCTW_ST = ( - MachineState.ERROR,MachineState.ERROR,MachineState.START, 3, 3, 3, 4,MachineState.ERROR,#00-07 - MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,MachineState.ITS_ME,MachineState.ITS_ME,#08-0f - MachineState.ITS_ME,MachineState.ITS_ME,MachineState.ITS_ME,MachineState.ITS_ME,MachineState.ITS_ME,MachineState.ERROR,MachineState.START,MachineState.ERROR,#10-17 - MachineState.START,MachineState.START,MachineState.START,MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,#18-1f - 5,MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,MachineState.START,MachineState.ERROR,MachineState.START,MachineState.START,#20-27 - MachineState.START,MachineState.ERROR,MachineState.START,MachineState.START,MachineState.START,MachineState.START,MachineState.START,MachineState.START #28-2f -) -# fmt: on - -EUCTW_CHAR_LEN_TABLE = (0, 0, 1, 2, 2, 2, 3) - -EUCTW_SM_MODEL: CodingStateMachineDict = { - "class_table": EUCTW_CLS, - "class_factor": 7, - "state_table": EUCTW_ST, - "char_len_table": EUCTW_CHAR_LEN_TABLE, - "name": "x-euc-tw", -} - -# GB2312 -# fmt: off -GB2312_CLS = ( - 1, 1, 1, 1, 1, 1, 1, 1, # 00 - 07 - 1, 1, 1, 1, 1, 1, 0, 0, # 08 - 0f - 1, 1, 1, 1, 1, 1, 1, 1, # 10 - 17 - 1, 1, 1, 0, 1, 1, 1, 1, # 18 - 1f - 1, 1, 1, 1, 1, 1, 1, 1, # 20 - 27 - 1, 1, 1, 1, 1, 1, 1, 1, # 28 - 2f - 3, 3, 3, 3, 3, 3, 3, 3, # 30 - 37 - 3, 3, 1, 1, 1, 1, 1, 1, # 38 - 3f - 2, 2, 2, 2, 2, 2, 2, 2, # 40 - 47 - 2, 2, 2, 2, 2, 2, 2, 2, # 48 - 4f - 2, 2, 2, 2, 2, 2, 2, 2, # 50 - 57 - 2, 2, 2, 2, 2, 2, 2, 2, # 58 - 5f - 2, 2, 2, 2, 2, 2, 2, 2, # 60 - 67 - 2, 2, 2, 2, 2, 2, 2, 2, # 68 - 6f - 2, 2, 2, 2, 2, 2, 2, 2, # 70 - 77 - 2, 2, 2, 2, 2, 2, 2, 4, # 78 - 7f - 5, 6, 6, 6, 6, 6, 6, 6, # 80 - 87 - 6, 6, 6, 6, 6, 6, 6, 6, # 88 - 8f - 6, 6, 6, 6, 6, 6, 6, 6, # 90 - 97 - 6, 6, 6, 6, 6, 6, 6, 6, # 98 - 9f - 6, 6, 6, 6, 6, 6, 6, 6, # a0 - a7 - 6, 6, 6, 6, 6, 6, 6, 6, # a8 - af - 6, 6, 6, 6, 6, 6, 6, 6, # b0 - b7 - 6, 6, 6, 6, 6, 6, 6, 6, # b8 - bf - 6, 6, 6, 6, 6, 6, 6, 6, # c0 - c7 - 6, 6, 6, 6, 6, 6, 6, 6, # c8 - cf - 6, 6, 6, 6, 6, 6, 6, 6, # d0 - d7 - 6, 6, 6, 6, 6, 6, 6, 6, # d8 - df - 6, 6, 6, 6, 6, 6, 6, 6, # e0 - e7 - 6, 6, 6, 6, 6, 6, 6, 6, # e8 - ef - 6, 6, 6, 6, 6, 6, 6, 6, # f0 - f7 - 6, 6, 6, 6, 6, 6, 6, 0 # f8 - ff -) - -GB2312_ST = ( - MachineState.ERROR,MachineState.START,MachineState.START,MachineState.START,MachineState.START,MachineState.START, 3,MachineState.ERROR,#00-07 - MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,MachineState.ITS_ME,MachineState.ITS_ME,#08-0f - MachineState.ITS_ME,MachineState.ITS_ME,MachineState.ITS_ME,MachineState.ITS_ME,MachineState.ITS_ME,MachineState.ERROR,MachineState.ERROR,MachineState.START,#10-17 - 4,MachineState.ERROR,MachineState.START,MachineState.START,MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,#18-1f - MachineState.ERROR,MachineState.ERROR, 5,MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,MachineState.ITS_ME,MachineState.ERROR,#20-27 - MachineState.ERROR,MachineState.ERROR,MachineState.START,MachineState.START,MachineState.START,MachineState.START,MachineState.START,MachineState.START #28-2f -) -# fmt: on - -# To be accurate, the length of class 6 can be either 2 or 4. -# But it is not necessary to discriminate between the two since -# it is used for frequency analysis only, and we are validating -# each code range there as well. So it is safe to set it to be -# 2 here. -GB2312_CHAR_LEN_TABLE = (0, 1, 1, 1, 1, 1, 2) - -GB2312_SM_MODEL: CodingStateMachineDict = { - "class_table": GB2312_CLS, - "class_factor": 7, - "state_table": GB2312_ST, - "char_len_table": GB2312_CHAR_LEN_TABLE, - "name": "GB2312", -} - -# Shift_JIS -# fmt: off -SJIS_CLS = ( - 1, 1, 1, 1, 1, 1, 1, 1, # 00 - 07 - 1, 1, 1, 1, 1, 1, 0, 0, # 08 - 0f - 1, 1, 1, 1, 1, 1, 1, 1, # 10 - 17 - 1, 1, 1, 0, 1, 1, 1, 1, # 18 - 1f - 1, 1, 1, 1, 1, 1, 1, 1, # 20 - 27 - 1, 1, 1, 1, 1, 1, 1, 1, # 28 - 2f - 1, 1, 1, 1, 1, 1, 1, 1, # 30 - 37 - 1, 1, 1, 1, 1, 1, 1, 1, # 38 - 3f - 2, 2, 2, 2, 2, 2, 2, 2, # 40 - 47 - 2, 2, 2, 2, 2, 2, 2, 2, # 48 - 4f - 2, 2, 2, 2, 2, 2, 2, 2, # 50 - 57 - 2, 2, 2, 2, 2, 2, 2, 2, # 58 - 5f - 2, 2, 2, 2, 2, 2, 2, 2, # 60 - 67 - 2, 2, 2, 2, 2, 2, 2, 2, # 68 - 6f - 2, 2, 2, 2, 2, 2, 2, 2, # 70 - 77 - 2, 2, 2, 2, 2, 2, 2, 1, # 78 - 7f - 3, 3, 3, 3, 3, 2, 2, 3, # 80 - 87 - 3, 3, 3, 3, 3, 3, 3, 3, # 88 - 8f - 3, 3, 3, 3, 3, 3, 3, 3, # 90 - 97 - 3, 3, 3, 3, 3, 3, 3, 3, # 98 - 9f - #0xa0 is illegal in sjis encoding, but some pages does - #contain such byte. We need to be more error forgiven. - 2, 2, 2, 2, 2, 2, 2, 2, # a0 - a7 - 2, 2, 2, 2, 2, 2, 2, 2, # a8 - af - 2, 2, 2, 2, 2, 2, 2, 2, # b0 - b7 - 2, 2, 2, 2, 2, 2, 2, 2, # b8 - bf - 2, 2, 2, 2, 2, 2, 2, 2, # c0 - c7 - 2, 2, 2, 2, 2, 2, 2, 2, # c8 - cf - 2, 2, 2, 2, 2, 2, 2, 2, # d0 - d7 - 2, 2, 2, 2, 2, 2, 2, 2, # d8 - df - 3, 3, 3, 3, 3, 3, 3, 3, # e0 - e7 - 3, 3, 3, 3, 3, 4, 4, 4, # e8 - ef - 3, 3, 3, 3, 3, 3, 3, 3, # f0 - f7 - 3, 3, 3, 3, 3, 0, 0, 0, # f8 - ff -) - -SJIS_ST = ( - MachineState.ERROR,MachineState.START,MachineState.START, 3,MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,#00-07 - MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,MachineState.ITS_ME,MachineState.ITS_ME,MachineState.ITS_ME,MachineState.ITS_ME,#08-0f - MachineState.ITS_ME,MachineState.ITS_ME,MachineState.ERROR,MachineState.ERROR,MachineState.START,MachineState.START,MachineState.START,MachineState.START #10-17 -) -# fmt: on - -SJIS_CHAR_LEN_TABLE = (0, 1, 1, 2, 0, 0) - -SJIS_SM_MODEL: CodingStateMachineDict = { - "class_table": SJIS_CLS, - "class_factor": 6, - "state_table": SJIS_ST, - "char_len_table": SJIS_CHAR_LEN_TABLE, - "name": "Shift_JIS", -} - -# UCS2-BE -# fmt: off -UCS2BE_CLS = ( - 0, 0, 0, 0, 0, 0, 0, 0, # 00 - 07 - 0, 0, 1, 0, 0, 2, 0, 0, # 08 - 0f - 0, 0, 0, 0, 0, 0, 0, 0, # 10 - 17 - 0, 0, 0, 3, 0, 0, 0, 0, # 18 - 1f - 0, 0, 0, 0, 0, 0, 0, 0, # 20 - 27 - 0, 3, 3, 3, 3, 3, 0, 0, # 28 - 2f - 0, 0, 0, 0, 0, 0, 0, 0, # 30 - 37 - 0, 0, 0, 0, 0, 0, 0, 0, # 38 - 3f - 0, 0, 0, 0, 0, 0, 0, 0, # 40 - 47 - 0, 0, 0, 0, 0, 0, 0, 0, # 48 - 4f - 0, 0, 0, 0, 0, 0, 0, 0, # 50 - 57 - 0, 0, 0, 0, 0, 0, 0, 0, # 58 - 5f - 0, 0, 0, 0, 0, 0, 0, 0, # 60 - 67 - 0, 0, 0, 0, 0, 0, 0, 0, # 68 - 6f - 0, 0, 0, 0, 0, 0, 0, 0, # 70 - 77 - 0, 0, 0, 0, 0, 0, 0, 0, # 78 - 7f - 0, 0, 0, 0, 0, 0, 0, 0, # 80 - 87 - 0, 0, 0, 0, 0, 0, 0, 0, # 88 - 8f - 0, 0, 0, 0, 0, 0, 0, 0, # 90 - 97 - 0, 0, 0, 0, 0, 0, 0, 0, # 98 - 9f - 0, 0, 0, 0, 0, 0, 0, 0, # a0 - a7 - 0, 0, 0, 0, 0, 0, 0, 0, # a8 - af - 0, 0, 0, 0, 0, 0, 0, 0, # b0 - b7 - 0, 0, 0, 0, 0, 0, 0, 0, # b8 - bf - 0, 0, 0, 0, 0, 0, 0, 0, # c0 - c7 - 0, 0, 0, 0, 0, 0, 0, 0, # c8 - cf - 0, 0, 0, 0, 0, 0, 0, 0, # d0 - d7 - 0, 0, 0, 0, 0, 0, 0, 0, # d8 - df - 0, 0, 0, 0, 0, 0, 0, 0, # e0 - e7 - 0, 0, 0, 0, 0, 0, 0, 0, # e8 - ef - 0, 0, 0, 0, 0, 0, 0, 0, # f0 - f7 - 0, 0, 0, 0, 0, 0, 4, 5 # f8 - ff -) - -UCS2BE_ST = ( - 5, 7, 7,MachineState.ERROR, 4, 3,MachineState.ERROR,MachineState.ERROR,#00-07 - MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,MachineState.ITS_ME,MachineState.ITS_ME,MachineState.ITS_ME,MachineState.ITS_ME,#08-0f - MachineState.ITS_ME,MachineState.ITS_ME, 6, 6, 6, 6,MachineState.ERROR,MachineState.ERROR,#10-17 - 6, 6, 6, 6, 6,MachineState.ITS_ME, 6, 6,#18-1f - 6, 6, 6, 6, 5, 7, 7,MachineState.ERROR,#20-27 - 5, 8, 6, 6,MachineState.ERROR, 6, 6, 6,#28-2f - 6, 6, 6, 6,MachineState.ERROR,MachineState.ERROR,MachineState.START,MachineState.START #30-37 -) -# fmt: on - -UCS2BE_CHAR_LEN_TABLE = (2, 2, 2, 0, 2, 2) - -UCS2BE_SM_MODEL: CodingStateMachineDict = { - "class_table": UCS2BE_CLS, - "class_factor": 6, - "state_table": UCS2BE_ST, - "char_len_table": UCS2BE_CHAR_LEN_TABLE, - "name": "UTF-16BE", -} - -# UCS2-LE -# fmt: off -UCS2LE_CLS = ( - 0, 0, 0, 0, 0, 0, 0, 0, # 00 - 07 - 0, 0, 1, 0, 0, 2, 0, 0, # 08 - 0f - 0, 0, 0, 0, 0, 0, 0, 0, # 10 - 17 - 0, 0, 0, 3, 0, 0, 0, 0, # 18 - 1f - 0, 0, 0, 0, 0, 0, 0, 0, # 20 - 27 - 0, 3, 3, 3, 3, 3, 0, 0, # 28 - 2f - 0, 0, 0, 0, 0, 0, 0, 0, # 30 - 37 - 0, 0, 0, 0, 0, 0, 0, 0, # 38 - 3f - 0, 0, 0, 0, 0, 0, 0, 0, # 40 - 47 - 0, 0, 0, 0, 0, 0, 0, 0, # 48 - 4f - 0, 0, 0, 0, 0, 0, 0, 0, # 50 - 57 - 0, 0, 0, 0, 0, 0, 0, 0, # 58 - 5f - 0, 0, 0, 0, 0, 0, 0, 0, # 60 - 67 - 0, 0, 0, 0, 0, 0, 0, 0, # 68 - 6f - 0, 0, 0, 0, 0, 0, 0, 0, # 70 - 77 - 0, 0, 0, 0, 0, 0, 0, 0, # 78 - 7f - 0, 0, 0, 0, 0, 0, 0, 0, # 80 - 87 - 0, 0, 0, 0, 0, 0, 0, 0, # 88 - 8f - 0, 0, 0, 0, 0, 0, 0, 0, # 90 - 97 - 0, 0, 0, 0, 0, 0, 0, 0, # 98 - 9f - 0, 0, 0, 0, 0, 0, 0, 0, # a0 - a7 - 0, 0, 0, 0, 0, 0, 0, 0, # a8 - af - 0, 0, 0, 0, 0, 0, 0, 0, # b0 - b7 - 0, 0, 0, 0, 0, 0, 0, 0, # b8 - bf - 0, 0, 0, 0, 0, 0, 0, 0, # c0 - c7 - 0, 0, 0, 0, 0, 0, 0, 0, # c8 - cf - 0, 0, 0, 0, 0, 0, 0, 0, # d0 - d7 - 0, 0, 0, 0, 0, 0, 0, 0, # d8 - df - 0, 0, 0, 0, 0, 0, 0, 0, # e0 - e7 - 0, 0, 0, 0, 0, 0, 0, 0, # e8 - ef - 0, 0, 0, 0, 0, 0, 0, 0, # f0 - f7 - 0, 0, 0, 0, 0, 0, 4, 5 # f8 - ff -) - -UCS2LE_ST = ( - 6, 6, 7, 6, 4, 3,MachineState.ERROR,MachineState.ERROR,#00-07 - MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,MachineState.ITS_ME,MachineState.ITS_ME,MachineState.ITS_ME,MachineState.ITS_ME,#08-0f - MachineState.ITS_ME,MachineState.ITS_ME, 5, 5, 5,MachineState.ERROR,MachineState.ITS_ME,MachineState.ERROR,#10-17 - 5, 5, 5,MachineState.ERROR, 5,MachineState.ERROR, 6, 6,#18-1f - 7, 6, 8, 8, 5, 5, 5,MachineState.ERROR,#20-27 - 5, 5, 5,MachineState.ERROR,MachineState.ERROR,MachineState.ERROR, 5, 5,#28-2f - 5, 5, 5,MachineState.ERROR, 5,MachineState.ERROR,MachineState.START,MachineState.START #30-37 -) -# fmt: on - -UCS2LE_CHAR_LEN_TABLE = (2, 2, 2, 2, 2, 2) - -UCS2LE_SM_MODEL: CodingStateMachineDict = { - "class_table": UCS2LE_CLS, - "class_factor": 6, - "state_table": UCS2LE_ST, - "char_len_table": UCS2LE_CHAR_LEN_TABLE, - "name": "UTF-16LE", -} - -# UTF-8 -# fmt: off -UTF8_CLS = ( - 1, 1, 1, 1, 1, 1, 1, 1, # 00 - 07 #allow 0x00 as a legal value - 1, 1, 1, 1, 1, 1, 0, 0, # 08 - 0f - 1, 1, 1, 1, 1, 1, 1, 1, # 10 - 17 - 1, 1, 1, 0, 1, 1, 1, 1, # 18 - 1f - 1, 1, 1, 1, 1, 1, 1, 1, # 20 - 27 - 1, 1, 1, 1, 1, 1, 1, 1, # 28 - 2f - 1, 1, 1, 1, 1, 1, 1, 1, # 30 - 37 - 1, 1, 1, 1, 1, 1, 1, 1, # 38 - 3f - 1, 1, 1, 1, 1, 1, 1, 1, # 40 - 47 - 1, 1, 1, 1, 1, 1, 1, 1, # 48 - 4f - 1, 1, 1, 1, 1, 1, 1, 1, # 50 - 57 - 1, 1, 1, 1, 1, 1, 1, 1, # 58 - 5f - 1, 1, 1, 1, 1, 1, 1, 1, # 60 - 67 - 1, 1, 1, 1, 1, 1, 1, 1, # 68 - 6f - 1, 1, 1, 1, 1, 1, 1, 1, # 70 - 77 - 1, 1, 1, 1, 1, 1, 1, 1, # 78 - 7f - 2, 2, 2, 2, 3, 3, 3, 3, # 80 - 87 - 4, 4, 4, 4, 4, 4, 4, 4, # 88 - 8f - 4, 4, 4, 4, 4, 4, 4, 4, # 90 - 97 - 4, 4, 4, 4, 4, 4, 4, 4, # 98 - 9f - 5, 5, 5, 5, 5, 5, 5, 5, # a0 - a7 - 5, 5, 5, 5, 5, 5, 5, 5, # a8 - af - 5, 5, 5, 5, 5, 5, 5, 5, # b0 - b7 - 5, 5, 5, 5, 5, 5, 5, 5, # b8 - bf - 0, 0, 6, 6, 6, 6, 6, 6, # c0 - c7 - 6, 6, 6, 6, 6, 6, 6, 6, # c8 - cf - 6, 6, 6, 6, 6, 6, 6, 6, # d0 - d7 - 6, 6, 6, 6, 6, 6, 6, 6, # d8 - df - 7, 8, 8, 8, 8, 8, 8, 8, # e0 - e7 - 8, 8, 8, 8, 8, 9, 8, 8, # e8 - ef - 10, 11, 11, 11, 11, 11, 11, 11, # f0 - f7 - 12, 13, 13, 13, 14, 15, 0, 0 # f8 - ff -) - -UTF8_ST = ( - MachineState.ERROR,MachineState.START,MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,MachineState.ERROR, 12, 10,#00-07 - 9, 11, 8, 7, 6, 5, 4, 3,#08-0f - MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,#10-17 - MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,#18-1f - MachineState.ITS_ME,MachineState.ITS_ME,MachineState.ITS_ME,MachineState.ITS_ME,MachineState.ITS_ME,MachineState.ITS_ME,MachineState.ITS_ME,MachineState.ITS_ME,#20-27 - MachineState.ITS_ME,MachineState.ITS_ME,MachineState.ITS_ME,MachineState.ITS_ME,MachineState.ITS_ME,MachineState.ITS_ME,MachineState.ITS_ME,MachineState.ITS_ME,#28-2f - MachineState.ERROR,MachineState.ERROR, 5, 5, 5, 5,MachineState.ERROR,MachineState.ERROR,#30-37 - MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,#38-3f - MachineState.ERROR,MachineState.ERROR,MachineState.ERROR, 5, 5, 5,MachineState.ERROR,MachineState.ERROR,#40-47 - MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,#48-4f - MachineState.ERROR,MachineState.ERROR, 7, 7, 7, 7,MachineState.ERROR,MachineState.ERROR,#50-57 - MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,#58-5f - MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,MachineState.ERROR, 7, 7,MachineState.ERROR,MachineState.ERROR,#60-67 - MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,#68-6f - MachineState.ERROR,MachineState.ERROR, 9, 9, 9, 9,MachineState.ERROR,MachineState.ERROR,#70-77 - MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,#78-7f - MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,MachineState.ERROR, 9,MachineState.ERROR,MachineState.ERROR,#80-87 - MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,#88-8f - MachineState.ERROR,MachineState.ERROR, 12, 12, 12, 12,MachineState.ERROR,MachineState.ERROR,#90-97 - MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,#98-9f - MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,MachineState.ERROR, 12,MachineState.ERROR,MachineState.ERROR,#a0-a7 - MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,#a8-af - MachineState.ERROR,MachineState.ERROR, 12, 12, 12,MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,#b0-b7 - MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,#b8-bf - MachineState.ERROR,MachineState.ERROR,MachineState.START,MachineState.START,MachineState.START,MachineState.START,MachineState.ERROR,MachineState.ERROR,#c0-c7 - MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,MachineState.ERROR #c8-cf -) -# fmt: on - -UTF8_CHAR_LEN_TABLE = (0, 1, 0, 0, 0, 0, 2, 3, 3, 3, 4, 4, 5, 5, 6, 6) - -UTF8_SM_MODEL: CodingStateMachineDict = { - "class_table": UTF8_CLS, - "class_factor": 16, - "state_table": UTF8_ST, - "char_len_table": UTF8_CHAR_LEN_TABLE, - "name": "UTF-8", -} diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/metadata/__init__.py b/venv/lib/python3.11/site-packages/pip/_vendor/chardet/metadata/__init__.py deleted file mode 100644 index e69de29..0000000 diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/metadata/__pycache__/__init__.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/chardet/metadata/__pycache__/__init__.cpython-311.pyc deleted file mode 100644 index 035c07b..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/metadata/__pycache__/__init__.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/metadata/__pycache__/languages.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/chardet/metadata/__pycache__/languages.cpython-311.pyc deleted file mode 100644 index bf50903..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/metadata/__pycache__/languages.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/metadata/languages.py b/venv/lib/python3.11/site-packages/pip/_vendor/chardet/metadata/languages.py deleted file mode 100644 index eb40c5f..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/metadata/languages.py +++ /dev/null @@ -1,352 +0,0 @@ -""" -Metadata about languages used by our model training code for our -SingleByteCharSetProbers. Could be used for other things in the future. - -This code is based on the language metadata from the uchardet project. -""" - -from string import ascii_letters -from typing import List, Optional - -# TODO: Add Ukrainian (KOI8-U) - - -class Language: - """Metadata about a language useful for training models - - :ivar name: The human name for the language, in English. - :type name: str - :ivar iso_code: 2-letter ISO 639-1 if possible, 3-letter ISO code otherwise, - or use another catalog as a last resort. - :type iso_code: str - :ivar use_ascii: Whether or not ASCII letters should be included in trained - models. - :type use_ascii: bool - :ivar charsets: The charsets we want to support and create data for. - :type charsets: list of str - :ivar alphabet: The characters in the language's alphabet. If `use_ascii` is - `True`, you only need to add those not in the ASCII set. - :type alphabet: str - :ivar wiki_start_pages: The Wikipedia pages to start from if we're crawling - Wikipedia for training data. - :type wiki_start_pages: list of str - """ - - def __init__( - self, - name: Optional[str] = None, - iso_code: Optional[str] = None, - use_ascii: bool = True, - charsets: Optional[List[str]] = None, - alphabet: Optional[str] = None, - wiki_start_pages: Optional[List[str]] = None, - ) -> None: - super().__init__() - self.name = name - self.iso_code = iso_code - self.use_ascii = use_ascii - self.charsets = charsets - if self.use_ascii: - if alphabet: - alphabet += ascii_letters - else: - alphabet = ascii_letters - elif not alphabet: - raise ValueError("Must supply alphabet if use_ascii is False") - self.alphabet = "".join(sorted(set(alphabet))) if alphabet else None - self.wiki_start_pages = wiki_start_pages - - def __repr__(self) -> str: - param_str = ", ".join( - f"{k}={v!r}" for k, v in self.__dict__.items() if not k.startswith("_") - ) - return f"{self.__class__.__name__}({param_str})" - - -LANGUAGES = { - "Arabic": Language( - name="Arabic", - iso_code="ar", - use_ascii=False, - # We only support encodings that use isolated - # forms, because the current recommendation is - # that the rendering system handles presentation - # forms. This means we purposefully skip IBM864. - charsets=["ISO-8859-6", "WINDOWS-1256", "CP720", "CP864"], - alphabet="ءآأؤإئابةتثجحخدذرزسشصضطظعغػؼؽؾؿـفقكلمنهوىيًٌٍَُِّ", - wiki_start_pages=["الصفحة_الرئيسية"], - ), - "Belarusian": Language( - name="Belarusian", - iso_code="be", - use_ascii=False, - charsets=["ISO-8859-5", "WINDOWS-1251", "IBM866", "MacCyrillic"], - alphabet="АБВГДЕЁЖЗІЙКЛМНОПРСТУЎФХЦЧШЫЬЭЮЯабвгдеёжзійклмнопрстуўфхцчшыьэюяʼ", - wiki_start_pages=["Галоўная_старонка"], - ), - "Bulgarian": Language( - name="Bulgarian", - iso_code="bg", - use_ascii=False, - charsets=["ISO-8859-5", "WINDOWS-1251", "IBM855"], - alphabet="АБВГДЕЖЗИЙКЛМНОПРСТУФХЦЧШЩЪЬЮЯабвгдежзийклмнопрстуфхцчшщъьюя", - wiki_start_pages=["Начална_страница"], - ), - "Czech": Language( - name="Czech", - iso_code="cz", - use_ascii=True, - charsets=["ISO-8859-2", "WINDOWS-1250"], - alphabet="áčďéěíňóřšťúůýžÁČĎÉĚÍŇÓŘŠŤÚŮÝŽ", - wiki_start_pages=["Hlavní_strana"], - ), - "Danish": Language( - name="Danish", - iso_code="da", - use_ascii=True, - charsets=["ISO-8859-1", "ISO-8859-15", "WINDOWS-1252", "MacRoman"], - alphabet="æøåÆØÅ", - wiki_start_pages=["Forside"], - ), - "German": Language( - name="German", - iso_code="de", - use_ascii=True, - charsets=["ISO-8859-1", "ISO-8859-15", "WINDOWS-1252", "MacRoman"], - alphabet="äöüßẞÄÖÜ", - wiki_start_pages=["Wikipedia:Hauptseite"], - ), - "Greek": Language( - name="Greek", - iso_code="el", - use_ascii=False, - charsets=["ISO-8859-7", "WINDOWS-1253"], - alphabet="αβγδεζηθικλμνξοπρσςτυφχψωάέήίόύώΑΒΓΔΕΖΗΘΙΚΛΜΝΞΟΠΡΣΣΤΥΦΧΨΩΆΈΉΊΌΎΏ", - wiki_start_pages=["Πύλη:Κύρια"], - ), - "English": Language( - name="English", - iso_code="en", - use_ascii=True, - charsets=["ISO-8859-1", "WINDOWS-1252", "MacRoman"], - wiki_start_pages=["Main_Page"], - ), - "Esperanto": Language( - name="Esperanto", - iso_code="eo", - # Q, W, X, and Y not used at all - use_ascii=False, - charsets=["ISO-8859-3"], - alphabet="abcĉdefgĝhĥijĵklmnoprsŝtuŭvzABCĈDEFGĜHĤIJĴKLMNOPRSŜTUŬVZ", - wiki_start_pages=["Vikipedio:Ĉefpaĝo"], - ), - "Spanish": Language( - name="Spanish", - iso_code="es", - use_ascii=True, - charsets=["ISO-8859-1", "ISO-8859-15", "WINDOWS-1252", "MacRoman"], - alphabet="ñáéíóúüÑÁÉÍÓÚÜ", - wiki_start_pages=["Wikipedia:Portada"], - ), - "Estonian": Language( - name="Estonian", - iso_code="et", - use_ascii=False, - charsets=["ISO-8859-4", "ISO-8859-13", "WINDOWS-1257"], - # C, F, Š, Q, W, X, Y, Z, Ž are only for - # loanwords - alphabet="ABDEGHIJKLMNOPRSTUVÕÄÖÜabdeghijklmnoprstuvõäöü", - wiki_start_pages=["Esileht"], - ), - "Finnish": Language( - name="Finnish", - iso_code="fi", - use_ascii=True, - charsets=["ISO-8859-1", "ISO-8859-15", "WINDOWS-1252", "MacRoman"], - alphabet="ÅÄÖŠŽåäöšž", - wiki_start_pages=["Wikipedia:Etusivu"], - ), - "French": Language( - name="French", - iso_code="fr", - use_ascii=True, - charsets=["ISO-8859-1", "ISO-8859-15", "WINDOWS-1252", "MacRoman"], - alphabet="œàâçèéîïùûêŒÀÂÇÈÉÎÏÙÛÊ", - wiki_start_pages=["Wikipédia:Accueil_principal", "Bœuf (animal)"], - ), - "Hebrew": Language( - name="Hebrew", - iso_code="he", - use_ascii=False, - charsets=["ISO-8859-8", "WINDOWS-1255"], - alphabet="אבגדהוזחטיךכלםמןנסעףפץצקרשתװױײ", - wiki_start_pages=["עמוד_ראשי"], - ), - "Croatian": Language( - name="Croatian", - iso_code="hr", - # Q, W, X, Y are only used for foreign words. - use_ascii=False, - charsets=["ISO-8859-2", "WINDOWS-1250"], - alphabet="abcčćdđefghijklmnoprsštuvzžABCČĆDĐEFGHIJKLMNOPRSŠTUVZŽ", - wiki_start_pages=["Glavna_stranica"], - ), - "Hungarian": Language( - name="Hungarian", - iso_code="hu", - # Q, W, X, Y are only used for foreign words. - use_ascii=False, - charsets=["ISO-8859-2", "WINDOWS-1250"], - alphabet="abcdefghijklmnoprstuvzáéíóöőúüűABCDEFGHIJKLMNOPRSTUVZÁÉÍÓÖŐÚÜŰ", - wiki_start_pages=["Kezdőlap"], - ), - "Italian": Language( - name="Italian", - iso_code="it", - use_ascii=True, - charsets=["ISO-8859-1", "ISO-8859-15", "WINDOWS-1252", "MacRoman"], - alphabet="ÀÈÉÌÒÓÙàèéìòóù", - wiki_start_pages=["Pagina_principale"], - ), - "Lithuanian": Language( - name="Lithuanian", - iso_code="lt", - use_ascii=False, - charsets=["ISO-8859-13", "WINDOWS-1257", "ISO-8859-4"], - # Q, W, and X not used at all - alphabet="AĄBCČDEĘĖFGHIĮYJKLMNOPRSŠTUŲŪVZŽaąbcčdeęėfghiįyjklmnoprsštuųūvzž", - wiki_start_pages=["Pagrindinis_puslapis"], - ), - "Latvian": Language( - name="Latvian", - iso_code="lv", - use_ascii=False, - charsets=["ISO-8859-13", "WINDOWS-1257", "ISO-8859-4"], - # Q, W, X, Y are only for loanwords - alphabet="AĀBCČDEĒFGĢHIĪJKĶLĻMNŅOPRSŠTUŪVZŽaābcčdeēfgģhiījkķlļmnņoprsštuūvzž", - wiki_start_pages=["Sākumlapa"], - ), - "Macedonian": Language( - name="Macedonian", - iso_code="mk", - use_ascii=False, - charsets=["ISO-8859-5", "WINDOWS-1251", "MacCyrillic", "IBM855"], - alphabet="АБВГДЃЕЖЗЅИЈКЛЉМНЊОПРСТЌУФХЦЧЏШабвгдѓежзѕијклљмнњопрстќуфхцчџш", - wiki_start_pages=["Главна_страница"], - ), - "Dutch": Language( - name="Dutch", - iso_code="nl", - use_ascii=True, - charsets=["ISO-8859-1", "WINDOWS-1252", "MacRoman"], - wiki_start_pages=["Hoofdpagina"], - ), - "Polish": Language( - name="Polish", - iso_code="pl", - # Q and X are only used for foreign words. - use_ascii=False, - charsets=["ISO-8859-2", "WINDOWS-1250"], - alphabet="AĄBCĆDEĘFGHIJKLŁMNŃOÓPRSŚTUWYZŹŻaąbcćdeęfghijklłmnńoóprsśtuwyzźż", - wiki_start_pages=["Wikipedia:Strona_główna"], - ), - "Portuguese": Language( - name="Portuguese", - iso_code="pt", - use_ascii=True, - charsets=["ISO-8859-1", "ISO-8859-15", "WINDOWS-1252", "MacRoman"], - alphabet="ÁÂÃÀÇÉÊÍÓÔÕÚáâãàçéêíóôõú", - wiki_start_pages=["Wikipédia:Página_principal"], - ), - "Romanian": Language( - name="Romanian", - iso_code="ro", - use_ascii=True, - charsets=["ISO-8859-2", "WINDOWS-1250"], - alphabet="ăâîșțĂÂÎȘȚ", - wiki_start_pages=["Pagina_principală"], - ), - "Russian": Language( - name="Russian", - iso_code="ru", - use_ascii=False, - charsets=[ - "ISO-8859-5", - "WINDOWS-1251", - "KOI8-R", - "MacCyrillic", - "IBM866", - "IBM855", - ], - alphabet="абвгдеёжзийклмнопрстуфхцчшщъыьэюяАБВГДЕЁЖЗИЙКЛМНОПРСТУФХЦЧШЩЪЫЬЭЮЯ", - wiki_start_pages=["Заглавная_страница"], - ), - "Slovak": Language( - name="Slovak", - iso_code="sk", - use_ascii=True, - charsets=["ISO-8859-2", "WINDOWS-1250"], - alphabet="áäčďéíĺľňóôŕšťúýžÁÄČĎÉÍĹĽŇÓÔŔŠŤÚÝŽ", - wiki_start_pages=["Hlavná_stránka"], - ), - "Slovene": Language( - name="Slovene", - iso_code="sl", - # Q, W, X, Y are only used for foreign words. - use_ascii=False, - charsets=["ISO-8859-2", "WINDOWS-1250"], - alphabet="abcčdefghijklmnoprsštuvzžABCČDEFGHIJKLMNOPRSŠTUVZŽ", - wiki_start_pages=["Glavna_stran"], - ), - # Serbian can be written in both Latin and Cyrillic, but there's no - # simple way to get the Latin alphabet pages from Wikipedia through - # the API, so for now we just support Cyrillic. - "Serbian": Language( - name="Serbian", - iso_code="sr", - alphabet="АБВГДЂЕЖЗИЈКЛЉМНЊОПРСТЋУФХЦЧЏШабвгдђежзијклљмнњопрстћуфхцчџш", - charsets=["ISO-8859-5", "WINDOWS-1251", "MacCyrillic", "IBM855"], - wiki_start_pages=["Главна_страна"], - ), - "Thai": Language( - name="Thai", - iso_code="th", - use_ascii=False, - charsets=["ISO-8859-11", "TIS-620", "CP874"], - alphabet="กขฃคฅฆงจฉชซฌญฎฏฐฑฒณดตถทธนบปผฝพฟภมยรฤลฦวศษสหฬอฮฯะัาำิีึืฺุู฿เแโใไๅๆ็่้๊๋์ํ๎๏๐๑๒๓๔๕๖๗๘๙๚๛", - wiki_start_pages=["หน้าหลัก"], - ), - "Turkish": Language( - name="Turkish", - iso_code="tr", - # Q, W, and X are not used by Turkish - use_ascii=False, - charsets=["ISO-8859-3", "ISO-8859-9", "WINDOWS-1254"], - alphabet="abcçdefgğhıijklmnoöprsştuüvyzâîûABCÇDEFGĞHIİJKLMNOÖPRSŞTUÜVYZÂÎÛ", - wiki_start_pages=["Ana_Sayfa"], - ), - "Vietnamese": Language( - name="Vietnamese", - iso_code="vi", - use_ascii=False, - # Windows-1258 is the only common 8-bit - # Vietnamese encoding supported by Python. - # From Wikipedia: - # For systems that lack support for Unicode, - # dozens of 8-bit Vietnamese code pages are - # available.[1] The most common are VISCII - # (TCVN 5712:1993), VPS, and Windows-1258.[3] - # Where ASCII is required, such as when - # ensuring readability in plain text e-mail, - # Vietnamese letters are often encoded - # according to Vietnamese Quoted-Readable - # (VIQR) or VSCII Mnemonic (VSCII-MNEM),[4] - # though usage of either variable-width - # scheme has declined dramatically following - # the adoption of Unicode on the World Wide - # Web. - charsets=["WINDOWS-1258"], - alphabet="aăâbcdđeêghiklmnoôơpqrstuưvxyAĂÂBCDĐEÊGHIKLMNOÔƠPQRSTUƯVXY", - wiki_start_pages=["Chữ_Quốc_ngữ"], - ), -} diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/resultdict.py b/venv/lib/python3.11/site-packages/pip/_vendor/chardet/resultdict.py deleted file mode 100644 index 7d36e64..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/resultdict.py +++ /dev/null @@ -1,16 +0,0 @@ -from typing import TYPE_CHECKING, Optional - -if TYPE_CHECKING: - # TypedDict was introduced in Python 3.8. - # - # TODO: Remove the else block and TYPE_CHECKING check when dropping support - # for Python 3.7. - from typing import TypedDict - - class ResultDict(TypedDict): - encoding: Optional[str] - confidence: float - language: Optional[str] - -else: - ResultDict = dict diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/sbcharsetprober.py b/venv/lib/python3.11/site-packages/pip/_vendor/chardet/sbcharsetprober.py deleted file mode 100644 index 0ffbcdd..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/sbcharsetprober.py +++ /dev/null @@ -1,162 +0,0 @@ -######################## BEGIN LICENSE BLOCK ######################## -# The Original Code is Mozilla Universal charset detector code. -# -# The Initial Developer of the Original Code is -# Netscape Communications Corporation. -# Portions created by the Initial Developer are Copyright (C) 2001 -# the Initial Developer. All Rights Reserved. -# -# Contributor(s): -# Mark Pilgrim - port to Python -# Shy Shalom - original C code -# -# This library is free software; you can redistribute it and/or -# modify it under the terms of the GNU Lesser General Public -# License as published by the Free Software Foundation; either -# version 2.1 of the License, or (at your option) any later version. -# -# This library is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# Lesser General Public License for more details. -# -# You should have received a copy of the GNU Lesser General Public -# License along with this library; if not, write to the Free Software -# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA -# 02110-1301 USA -######################### END LICENSE BLOCK ######################### - -from typing import Dict, List, NamedTuple, Optional, Union - -from .charsetprober import CharSetProber -from .enums import CharacterCategory, ProbingState, SequenceLikelihood - - -class SingleByteCharSetModel(NamedTuple): - charset_name: str - language: str - char_to_order_map: Dict[int, int] - language_model: Dict[int, Dict[int, int]] - typical_positive_ratio: float - keep_ascii_letters: bool - alphabet: str - - -class SingleByteCharSetProber(CharSetProber): - SAMPLE_SIZE = 64 - SB_ENOUGH_REL_THRESHOLD = 1024 # 0.25 * SAMPLE_SIZE^2 - POSITIVE_SHORTCUT_THRESHOLD = 0.95 - NEGATIVE_SHORTCUT_THRESHOLD = 0.05 - - def __init__( - self, - model: SingleByteCharSetModel, - is_reversed: bool = False, - name_prober: Optional[CharSetProber] = None, - ) -> None: - super().__init__() - self._model = model - # TRUE if we need to reverse every pair in the model lookup - self._reversed = is_reversed - # Optional auxiliary prober for name decision - self._name_prober = name_prober - self._last_order = 255 - self._seq_counters: List[int] = [] - self._total_seqs = 0 - self._total_char = 0 - self._control_char = 0 - self._freq_char = 0 - self.reset() - - def reset(self) -> None: - super().reset() - # char order of last character - self._last_order = 255 - self._seq_counters = [0] * SequenceLikelihood.get_num_categories() - self._total_seqs = 0 - self._total_char = 0 - self._control_char = 0 - # characters that fall in our sampling range - self._freq_char = 0 - - @property - def charset_name(self) -> Optional[str]: - if self._name_prober: - return self._name_prober.charset_name - return self._model.charset_name - - @property - def language(self) -> Optional[str]: - if self._name_prober: - return self._name_prober.language - return self._model.language - - def feed(self, byte_str: Union[bytes, bytearray]) -> ProbingState: - # TODO: Make filter_international_words keep things in self.alphabet - if not self._model.keep_ascii_letters: - byte_str = self.filter_international_words(byte_str) - else: - byte_str = self.remove_xml_tags(byte_str) - if not byte_str: - return self.state - char_to_order_map = self._model.char_to_order_map - language_model = self._model.language_model - for char in byte_str: - order = char_to_order_map.get(char, CharacterCategory.UNDEFINED) - # XXX: This was SYMBOL_CAT_ORDER before, with a value of 250, but - # CharacterCategory.SYMBOL is actually 253, so we use CONTROL - # to make it closer to the original intent. The only difference - # is whether or not we count digits and control characters for - # _total_char purposes. - if order < CharacterCategory.CONTROL: - self._total_char += 1 - if order < self.SAMPLE_SIZE: - self._freq_char += 1 - if self._last_order < self.SAMPLE_SIZE: - self._total_seqs += 1 - if not self._reversed: - lm_cat = language_model[self._last_order][order] - else: - lm_cat = language_model[order][self._last_order] - self._seq_counters[lm_cat] += 1 - self._last_order = order - - charset_name = self._model.charset_name - if self.state == ProbingState.DETECTING: - if self._total_seqs > self.SB_ENOUGH_REL_THRESHOLD: - confidence = self.get_confidence() - if confidence > self.POSITIVE_SHORTCUT_THRESHOLD: - self.logger.debug( - "%s confidence = %s, we have a winner", charset_name, confidence - ) - self._state = ProbingState.FOUND_IT - elif confidence < self.NEGATIVE_SHORTCUT_THRESHOLD: - self.logger.debug( - "%s confidence = %s, below negative shortcut threshold %s", - charset_name, - confidence, - self.NEGATIVE_SHORTCUT_THRESHOLD, - ) - self._state = ProbingState.NOT_ME - - return self.state - - def get_confidence(self) -> float: - r = 0.01 - if self._total_seqs > 0: - r = ( - ( - self._seq_counters[SequenceLikelihood.POSITIVE] - + 0.25 * self._seq_counters[SequenceLikelihood.LIKELY] - ) - / self._total_seqs - / self._model.typical_positive_ratio - ) - # The more control characters (proportionnaly to the size - # of the text), the less confident we become in the current - # charset. - r = r * (self._total_char - self._control_char) / self._total_char - r = r * self._freq_char / self._total_char - if r >= 1.0: - r = 0.99 - return r diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/sbcsgroupprober.py b/venv/lib/python3.11/site-packages/pip/_vendor/chardet/sbcsgroupprober.py deleted file mode 100644 index 890ae84..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/sbcsgroupprober.py +++ /dev/null @@ -1,88 +0,0 @@ -######################## BEGIN LICENSE BLOCK ######################## -# The Original Code is Mozilla Universal charset detector code. -# -# The Initial Developer of the Original Code is -# Netscape Communications Corporation. -# Portions created by the Initial Developer are Copyright (C) 2001 -# the Initial Developer. All Rights Reserved. -# -# Contributor(s): -# Mark Pilgrim - port to Python -# Shy Shalom - original C code -# -# This library is free software; you can redistribute it and/or -# modify it under the terms of the GNU Lesser General Public -# License as published by the Free Software Foundation; either -# version 2.1 of the License, or (at your option) any later version. -# -# This library is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# Lesser General Public License for more details. -# -# You should have received a copy of the GNU Lesser General Public -# License along with this library; if not, write to the Free Software -# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA -# 02110-1301 USA -######################### END LICENSE BLOCK ######################### - -from .charsetgroupprober import CharSetGroupProber -from .hebrewprober import HebrewProber -from .langbulgarianmodel import ISO_8859_5_BULGARIAN_MODEL, WINDOWS_1251_BULGARIAN_MODEL -from .langgreekmodel import ISO_8859_7_GREEK_MODEL, WINDOWS_1253_GREEK_MODEL -from .langhebrewmodel import WINDOWS_1255_HEBREW_MODEL - -# from .langhungarianmodel import (ISO_8859_2_HUNGARIAN_MODEL, -# WINDOWS_1250_HUNGARIAN_MODEL) -from .langrussianmodel import ( - IBM855_RUSSIAN_MODEL, - IBM866_RUSSIAN_MODEL, - ISO_8859_5_RUSSIAN_MODEL, - KOI8_R_RUSSIAN_MODEL, - MACCYRILLIC_RUSSIAN_MODEL, - WINDOWS_1251_RUSSIAN_MODEL, -) -from .langthaimodel import TIS_620_THAI_MODEL -from .langturkishmodel import ISO_8859_9_TURKISH_MODEL -from .sbcharsetprober import SingleByteCharSetProber - - -class SBCSGroupProber(CharSetGroupProber): - def __init__(self) -> None: - super().__init__() - hebrew_prober = HebrewProber() - logical_hebrew_prober = SingleByteCharSetProber( - WINDOWS_1255_HEBREW_MODEL, is_reversed=False, name_prober=hebrew_prober - ) - # TODO: See if using ISO-8859-8 Hebrew model works better here, since - # it's actually the visual one - visual_hebrew_prober = SingleByteCharSetProber( - WINDOWS_1255_HEBREW_MODEL, is_reversed=True, name_prober=hebrew_prober - ) - hebrew_prober.set_model_probers(logical_hebrew_prober, visual_hebrew_prober) - # TODO: ORDER MATTERS HERE. I changed the order vs what was in master - # and several tests failed that did not before. Some thought - # should be put into the ordering, and we should consider making - # order not matter here, because that is very counter-intuitive. - self.probers = [ - SingleByteCharSetProber(WINDOWS_1251_RUSSIAN_MODEL), - SingleByteCharSetProber(KOI8_R_RUSSIAN_MODEL), - SingleByteCharSetProber(ISO_8859_5_RUSSIAN_MODEL), - SingleByteCharSetProber(MACCYRILLIC_RUSSIAN_MODEL), - SingleByteCharSetProber(IBM866_RUSSIAN_MODEL), - SingleByteCharSetProber(IBM855_RUSSIAN_MODEL), - SingleByteCharSetProber(ISO_8859_7_GREEK_MODEL), - SingleByteCharSetProber(WINDOWS_1253_GREEK_MODEL), - SingleByteCharSetProber(ISO_8859_5_BULGARIAN_MODEL), - SingleByteCharSetProber(WINDOWS_1251_BULGARIAN_MODEL), - # TODO: Restore Hungarian encodings (iso-8859-2 and windows-1250) - # after we retrain model. - # SingleByteCharSetProber(ISO_8859_2_HUNGARIAN_MODEL), - # SingleByteCharSetProber(WINDOWS_1250_HUNGARIAN_MODEL), - SingleByteCharSetProber(TIS_620_THAI_MODEL), - SingleByteCharSetProber(ISO_8859_9_TURKISH_MODEL), - hebrew_prober, - logical_hebrew_prober, - visual_hebrew_prober, - ] - self.reset() diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/sjisprober.py b/venv/lib/python3.11/site-packages/pip/_vendor/chardet/sjisprober.py deleted file mode 100644 index 91df077..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/sjisprober.py +++ /dev/null @@ -1,105 +0,0 @@ -######################## BEGIN LICENSE BLOCK ######################## -# The Original Code is mozilla.org code. -# -# The Initial Developer of the Original Code is -# Netscape Communications Corporation. -# Portions created by the Initial Developer are Copyright (C) 1998 -# the Initial Developer. All Rights Reserved. -# -# Contributor(s): -# Mark Pilgrim - port to Python -# -# This library is free software; you can redistribute it and/or -# modify it under the terms of the GNU Lesser General Public -# License as published by the Free Software Foundation; either -# version 2.1 of the License, or (at your option) any later version. -# -# This library is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# Lesser General Public License for more details. -# -# You should have received a copy of the GNU Lesser General Public -# License along with this library; if not, write to the Free Software -# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA -# 02110-1301 USA -######################### END LICENSE BLOCK ######################### - -from typing import Union - -from .chardistribution import SJISDistributionAnalysis -from .codingstatemachine import CodingStateMachine -from .enums import MachineState, ProbingState -from .jpcntx import SJISContextAnalysis -from .mbcharsetprober import MultiByteCharSetProber -from .mbcssm import SJIS_SM_MODEL - - -class SJISProber(MultiByteCharSetProber): - def __init__(self) -> None: - super().__init__() - self.coding_sm = CodingStateMachine(SJIS_SM_MODEL) - self.distribution_analyzer = SJISDistributionAnalysis() - self.context_analyzer = SJISContextAnalysis() - self.reset() - - def reset(self) -> None: - super().reset() - self.context_analyzer.reset() - - @property - def charset_name(self) -> str: - return self.context_analyzer.charset_name - - @property - def language(self) -> str: - return "Japanese" - - def feed(self, byte_str: Union[bytes, bytearray]) -> ProbingState: - assert self.coding_sm is not None - assert self.distribution_analyzer is not None - - for i, byte in enumerate(byte_str): - coding_state = self.coding_sm.next_state(byte) - if coding_state == MachineState.ERROR: - self.logger.debug( - "%s %s prober hit error at byte %s", - self.charset_name, - self.language, - i, - ) - self._state = ProbingState.NOT_ME - break - if coding_state == MachineState.ITS_ME: - self._state = ProbingState.FOUND_IT - break - if coding_state == MachineState.START: - char_len = self.coding_sm.get_current_charlen() - if i == 0: - self._last_char[1] = byte - self.context_analyzer.feed( - self._last_char[2 - char_len :], char_len - ) - self.distribution_analyzer.feed(self._last_char, char_len) - else: - self.context_analyzer.feed( - byte_str[i + 1 - char_len : i + 3 - char_len], char_len - ) - self.distribution_analyzer.feed(byte_str[i - 1 : i + 1], char_len) - - self._last_char[0] = byte_str[-1] - - if self.state == ProbingState.DETECTING: - if self.context_analyzer.got_enough_data() and ( - self.get_confidence() > self.SHORTCUT_THRESHOLD - ): - self._state = ProbingState.FOUND_IT - - return self.state - - def get_confidence(self) -> float: - assert self.distribution_analyzer is not None - - context_conf = self.context_analyzer.get_confidence() - distrib_conf = self.distribution_analyzer.get_confidence() - return max(context_conf, distrib_conf) diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/universaldetector.py b/venv/lib/python3.11/site-packages/pip/_vendor/chardet/universaldetector.py deleted file mode 100644 index 30c441d..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/universaldetector.py +++ /dev/null @@ -1,362 +0,0 @@ -######################## BEGIN LICENSE BLOCK ######################## -# The Original Code is Mozilla Universal charset detector code. -# -# The Initial Developer of the Original Code is -# Netscape Communications Corporation. -# Portions created by the Initial Developer are Copyright (C) 2001 -# the Initial Developer. All Rights Reserved. -# -# Contributor(s): -# Mark Pilgrim - port to Python -# Shy Shalom - original C code -# -# This library is free software; you can redistribute it and/or -# modify it under the terms of the GNU Lesser General Public -# License as published by the Free Software Foundation; either -# version 2.1 of the License, or (at your option) any later version. -# -# This library is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# Lesser General Public License for more details. -# -# You should have received a copy of the GNU Lesser General Public -# License along with this library; if not, write to the Free Software -# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA -# 02110-1301 USA -######################### END LICENSE BLOCK ######################### -""" -Module containing the UniversalDetector detector class, which is the primary -class a user of ``chardet`` should use. - -:author: Mark Pilgrim (initial port to Python) -:author: Shy Shalom (original C code) -:author: Dan Blanchard (major refactoring for 3.0) -:author: Ian Cordasco -""" - - -import codecs -import logging -import re -from typing import List, Optional, Union - -from .charsetgroupprober import CharSetGroupProber -from .charsetprober import CharSetProber -from .enums import InputState, LanguageFilter, ProbingState -from .escprober import EscCharSetProber -from .latin1prober import Latin1Prober -from .macromanprober import MacRomanProber -from .mbcsgroupprober import MBCSGroupProber -from .resultdict import ResultDict -from .sbcsgroupprober import SBCSGroupProber -from .utf1632prober import UTF1632Prober - - -class UniversalDetector: - """ - The ``UniversalDetector`` class underlies the ``chardet.detect`` function - and coordinates all of the different charset probers. - - To get a ``dict`` containing an encoding and its confidence, you can simply - run: - - .. code:: - - u = UniversalDetector() - u.feed(some_bytes) - u.close() - detected = u.result - - """ - - MINIMUM_THRESHOLD = 0.20 - HIGH_BYTE_DETECTOR = re.compile(b"[\x80-\xFF]") - ESC_DETECTOR = re.compile(b"(\033|~{)") - WIN_BYTE_DETECTOR = re.compile(b"[\x80-\x9F]") - ISO_WIN_MAP = { - "iso-8859-1": "Windows-1252", - "iso-8859-2": "Windows-1250", - "iso-8859-5": "Windows-1251", - "iso-8859-6": "Windows-1256", - "iso-8859-7": "Windows-1253", - "iso-8859-8": "Windows-1255", - "iso-8859-9": "Windows-1254", - "iso-8859-13": "Windows-1257", - } - # Based on https://encoding.spec.whatwg.org/#names-and-labels - # but altered to match Python names for encodings and remove mappings - # that break tests. - LEGACY_MAP = { - "ascii": "Windows-1252", - "iso-8859-1": "Windows-1252", - "tis-620": "ISO-8859-11", - "iso-8859-9": "Windows-1254", - "gb2312": "GB18030", - "euc-kr": "CP949", - "utf-16le": "UTF-16", - } - - def __init__( - self, - lang_filter: LanguageFilter = LanguageFilter.ALL, - should_rename_legacy: bool = False, - ) -> None: - self._esc_charset_prober: Optional[EscCharSetProber] = None - self._utf1632_prober: Optional[UTF1632Prober] = None - self._charset_probers: List[CharSetProber] = [] - self.result: ResultDict = { - "encoding": None, - "confidence": 0.0, - "language": None, - } - self.done = False - self._got_data = False - self._input_state = InputState.PURE_ASCII - self._last_char = b"" - self.lang_filter = lang_filter - self.logger = logging.getLogger(__name__) - self._has_win_bytes = False - self.should_rename_legacy = should_rename_legacy - self.reset() - - @property - def input_state(self) -> int: - return self._input_state - - @property - def has_win_bytes(self) -> bool: - return self._has_win_bytes - - @property - def charset_probers(self) -> List[CharSetProber]: - return self._charset_probers - - def reset(self) -> None: - """ - Reset the UniversalDetector and all of its probers back to their - initial states. This is called by ``__init__``, so you only need to - call this directly in between analyses of different documents. - """ - self.result = {"encoding": None, "confidence": 0.0, "language": None} - self.done = False - self._got_data = False - self._has_win_bytes = False - self._input_state = InputState.PURE_ASCII - self._last_char = b"" - if self._esc_charset_prober: - self._esc_charset_prober.reset() - if self._utf1632_prober: - self._utf1632_prober.reset() - for prober in self._charset_probers: - prober.reset() - - def feed(self, byte_str: Union[bytes, bytearray]) -> None: - """ - Takes a chunk of a document and feeds it through all of the relevant - charset probers. - - After calling ``feed``, you can check the value of the ``done`` - attribute to see if you need to continue feeding the - ``UniversalDetector`` more data, or if it has made a prediction - (in the ``result`` attribute). - - .. note:: - You should always call ``close`` when you're done feeding in your - document if ``done`` is not already ``True``. - """ - if self.done: - return - - if not byte_str: - return - - if not isinstance(byte_str, bytearray): - byte_str = bytearray(byte_str) - - # First check for known BOMs, since these are guaranteed to be correct - if not self._got_data: - # If the data starts with BOM, we know it is UTF - if byte_str.startswith(codecs.BOM_UTF8): - # EF BB BF UTF-8 with BOM - self.result = { - "encoding": "UTF-8-SIG", - "confidence": 1.0, - "language": "", - } - elif byte_str.startswith((codecs.BOM_UTF32_LE, codecs.BOM_UTF32_BE)): - # FF FE 00 00 UTF-32, little-endian BOM - # 00 00 FE FF UTF-32, big-endian BOM - self.result = {"encoding": "UTF-32", "confidence": 1.0, "language": ""} - elif byte_str.startswith(b"\xFE\xFF\x00\x00"): - # FE FF 00 00 UCS-4, unusual octet order BOM (3412) - self.result = { - # TODO: This encoding is not supported by Python. Should remove? - "encoding": "X-ISO-10646-UCS-4-3412", - "confidence": 1.0, - "language": "", - } - elif byte_str.startswith(b"\x00\x00\xFF\xFE"): - # 00 00 FF FE UCS-4, unusual octet order BOM (2143) - self.result = { - # TODO: This encoding is not supported by Python. Should remove? - "encoding": "X-ISO-10646-UCS-4-2143", - "confidence": 1.0, - "language": "", - } - elif byte_str.startswith((codecs.BOM_LE, codecs.BOM_BE)): - # FF FE UTF-16, little endian BOM - # FE FF UTF-16, big endian BOM - self.result = {"encoding": "UTF-16", "confidence": 1.0, "language": ""} - - self._got_data = True - if self.result["encoding"] is not None: - self.done = True - return - - # If none of those matched and we've only see ASCII so far, check - # for high bytes and escape sequences - if self._input_state == InputState.PURE_ASCII: - if self.HIGH_BYTE_DETECTOR.search(byte_str): - self._input_state = InputState.HIGH_BYTE - elif ( - self._input_state == InputState.PURE_ASCII - and self.ESC_DETECTOR.search(self._last_char + byte_str) - ): - self._input_state = InputState.ESC_ASCII - - self._last_char = byte_str[-1:] - - # next we will look to see if it is appears to be either a UTF-16 or - # UTF-32 encoding - if not self._utf1632_prober: - self._utf1632_prober = UTF1632Prober() - - if self._utf1632_prober.state == ProbingState.DETECTING: - if self._utf1632_prober.feed(byte_str) == ProbingState.FOUND_IT: - self.result = { - "encoding": self._utf1632_prober.charset_name, - "confidence": self._utf1632_prober.get_confidence(), - "language": "", - } - self.done = True - return - - # If we've seen escape sequences, use the EscCharSetProber, which - # uses a simple state machine to check for known escape sequences in - # HZ and ISO-2022 encodings, since those are the only encodings that - # use such sequences. - if self._input_state == InputState.ESC_ASCII: - if not self._esc_charset_prober: - self._esc_charset_prober = EscCharSetProber(self.lang_filter) - if self._esc_charset_prober.feed(byte_str) == ProbingState.FOUND_IT: - self.result = { - "encoding": self._esc_charset_prober.charset_name, - "confidence": self._esc_charset_prober.get_confidence(), - "language": self._esc_charset_prober.language, - } - self.done = True - # If we've seen high bytes (i.e., those with values greater than 127), - # we need to do more complicated checks using all our multi-byte and - # single-byte probers that are left. The single-byte probers - # use character bigram distributions to determine the encoding, whereas - # the multi-byte probers use a combination of character unigram and - # bigram distributions. - elif self._input_state == InputState.HIGH_BYTE: - if not self._charset_probers: - self._charset_probers = [MBCSGroupProber(self.lang_filter)] - # If we're checking non-CJK encodings, use single-byte prober - if self.lang_filter & LanguageFilter.NON_CJK: - self._charset_probers.append(SBCSGroupProber()) - self._charset_probers.append(Latin1Prober()) - self._charset_probers.append(MacRomanProber()) - for prober in self._charset_probers: - if prober.feed(byte_str) == ProbingState.FOUND_IT: - self.result = { - "encoding": prober.charset_name, - "confidence": prober.get_confidence(), - "language": prober.language, - } - self.done = True - break - if self.WIN_BYTE_DETECTOR.search(byte_str): - self._has_win_bytes = True - - def close(self) -> ResultDict: - """ - Stop analyzing the current document and come up with a final - prediction. - - :returns: The ``result`` attribute, a ``dict`` with the keys - `encoding`, `confidence`, and `language`. - """ - # Don't bother with checks if we're already done - if self.done: - return self.result - self.done = True - - if not self._got_data: - self.logger.debug("no data received!") - - # Default to ASCII if it is all we've seen so far - elif self._input_state == InputState.PURE_ASCII: - self.result = {"encoding": "ascii", "confidence": 1.0, "language": ""} - - # If we have seen non-ASCII, return the best that met MINIMUM_THRESHOLD - elif self._input_state == InputState.HIGH_BYTE: - prober_confidence = None - max_prober_confidence = 0.0 - max_prober = None - for prober in self._charset_probers: - if not prober: - continue - prober_confidence = prober.get_confidence() - if prober_confidence > max_prober_confidence: - max_prober_confidence = prober_confidence - max_prober = prober - if max_prober and (max_prober_confidence > self.MINIMUM_THRESHOLD): - charset_name = max_prober.charset_name - assert charset_name is not None - lower_charset_name = charset_name.lower() - confidence = max_prober.get_confidence() - # Use Windows encoding name instead of ISO-8859 if we saw any - # extra Windows-specific bytes - if lower_charset_name.startswith("iso-8859"): - if self._has_win_bytes: - charset_name = self.ISO_WIN_MAP.get( - lower_charset_name, charset_name - ) - # Rename legacy encodings with superset encodings if asked - if self.should_rename_legacy: - charset_name = self.LEGACY_MAP.get( - (charset_name or "").lower(), charset_name - ) - self.result = { - "encoding": charset_name, - "confidence": confidence, - "language": max_prober.language, - } - - # Log all prober confidences if none met MINIMUM_THRESHOLD - if self.logger.getEffectiveLevel() <= logging.DEBUG: - if self.result["encoding"] is None: - self.logger.debug("no probers hit minimum threshold") - for group_prober in self._charset_probers: - if not group_prober: - continue - if isinstance(group_prober, CharSetGroupProber): - for prober in group_prober.probers: - self.logger.debug( - "%s %s confidence = %s", - prober.charset_name, - prober.language, - prober.get_confidence(), - ) - else: - self.logger.debug( - "%s %s confidence = %s", - group_prober.charset_name, - group_prober.language, - group_prober.get_confidence(), - ) - return self.result diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/utf1632prober.py b/venv/lib/python3.11/site-packages/pip/_vendor/chardet/utf1632prober.py deleted file mode 100644 index 6bdec63..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/utf1632prober.py +++ /dev/null @@ -1,225 +0,0 @@ -######################## BEGIN LICENSE BLOCK ######################## -# -# Contributor(s): -# Jason Zavaglia -# -# This library is free software; you can redistribute it and/or -# modify it under the terms of the GNU Lesser General Public -# License as published by the Free Software Foundation; either -# version 2.1 of the License, or (at your option) any later version. -# -# This library is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# Lesser General Public License for more details. -# -# You should have received a copy of the GNU Lesser General Public -# License along with this library; if not, write to the Free Software -# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA -# 02110-1301 USA -######################### END LICENSE BLOCK ######################### -from typing import List, Union - -from .charsetprober import CharSetProber -from .enums import ProbingState - - -class UTF1632Prober(CharSetProber): - """ - This class simply looks for occurrences of zero bytes, and infers - whether the file is UTF16 or UTF32 (low-endian or big-endian) - For instance, files looking like ( \0 \0 \0 [nonzero] )+ - have a good probability to be UTF32BE. Files looking like ( \0 [nonzero] )+ - may be guessed to be UTF16BE, and inversely for little-endian varieties. - """ - - # how many logical characters to scan before feeling confident of prediction - MIN_CHARS_FOR_DETECTION = 20 - # a fixed constant ratio of expected zeros or non-zeros in modulo-position. - EXPECTED_RATIO = 0.94 - - def __init__(self) -> None: - super().__init__() - self.position = 0 - self.zeros_at_mod = [0] * 4 - self.nonzeros_at_mod = [0] * 4 - self._state = ProbingState.DETECTING - self.quad = [0, 0, 0, 0] - self.invalid_utf16be = False - self.invalid_utf16le = False - self.invalid_utf32be = False - self.invalid_utf32le = False - self.first_half_surrogate_pair_detected_16be = False - self.first_half_surrogate_pair_detected_16le = False - self.reset() - - def reset(self) -> None: - super().reset() - self.position = 0 - self.zeros_at_mod = [0] * 4 - self.nonzeros_at_mod = [0] * 4 - self._state = ProbingState.DETECTING - self.invalid_utf16be = False - self.invalid_utf16le = False - self.invalid_utf32be = False - self.invalid_utf32le = False - self.first_half_surrogate_pair_detected_16be = False - self.first_half_surrogate_pair_detected_16le = False - self.quad = [0, 0, 0, 0] - - @property - def charset_name(self) -> str: - if self.is_likely_utf32be(): - return "utf-32be" - if self.is_likely_utf32le(): - return "utf-32le" - if self.is_likely_utf16be(): - return "utf-16be" - if self.is_likely_utf16le(): - return "utf-16le" - # default to something valid - return "utf-16" - - @property - def language(self) -> str: - return "" - - def approx_32bit_chars(self) -> float: - return max(1.0, self.position / 4.0) - - def approx_16bit_chars(self) -> float: - return max(1.0, self.position / 2.0) - - def is_likely_utf32be(self) -> bool: - approx_chars = self.approx_32bit_chars() - return approx_chars >= self.MIN_CHARS_FOR_DETECTION and ( - self.zeros_at_mod[0] / approx_chars > self.EXPECTED_RATIO - and self.zeros_at_mod[1] / approx_chars > self.EXPECTED_RATIO - and self.zeros_at_mod[2] / approx_chars > self.EXPECTED_RATIO - and self.nonzeros_at_mod[3] / approx_chars > self.EXPECTED_RATIO - and not self.invalid_utf32be - ) - - def is_likely_utf32le(self) -> bool: - approx_chars = self.approx_32bit_chars() - return approx_chars >= self.MIN_CHARS_FOR_DETECTION and ( - self.nonzeros_at_mod[0] / approx_chars > self.EXPECTED_RATIO - and self.zeros_at_mod[1] / approx_chars > self.EXPECTED_RATIO - and self.zeros_at_mod[2] / approx_chars > self.EXPECTED_RATIO - and self.zeros_at_mod[3] / approx_chars > self.EXPECTED_RATIO - and not self.invalid_utf32le - ) - - def is_likely_utf16be(self) -> bool: - approx_chars = self.approx_16bit_chars() - return approx_chars >= self.MIN_CHARS_FOR_DETECTION and ( - (self.nonzeros_at_mod[1] + self.nonzeros_at_mod[3]) / approx_chars - > self.EXPECTED_RATIO - and (self.zeros_at_mod[0] + self.zeros_at_mod[2]) / approx_chars - > self.EXPECTED_RATIO - and not self.invalid_utf16be - ) - - def is_likely_utf16le(self) -> bool: - approx_chars = self.approx_16bit_chars() - return approx_chars >= self.MIN_CHARS_FOR_DETECTION and ( - (self.nonzeros_at_mod[0] + self.nonzeros_at_mod[2]) / approx_chars - > self.EXPECTED_RATIO - and (self.zeros_at_mod[1] + self.zeros_at_mod[3]) / approx_chars - > self.EXPECTED_RATIO - and not self.invalid_utf16le - ) - - def validate_utf32_characters(self, quad: List[int]) -> None: - """ - Validate if the quad of bytes is valid UTF-32. - - UTF-32 is valid in the range 0x00000000 - 0x0010FFFF - excluding 0x0000D800 - 0x0000DFFF - - https://en.wikipedia.org/wiki/UTF-32 - """ - if ( - quad[0] != 0 - or quad[1] > 0x10 - or (quad[0] == 0 and quad[1] == 0 and 0xD8 <= quad[2] <= 0xDF) - ): - self.invalid_utf32be = True - if ( - quad[3] != 0 - or quad[2] > 0x10 - or (quad[3] == 0 and quad[2] == 0 and 0xD8 <= quad[1] <= 0xDF) - ): - self.invalid_utf32le = True - - def validate_utf16_characters(self, pair: List[int]) -> None: - """ - Validate if the pair of bytes is valid UTF-16. - - UTF-16 is valid in the range 0x0000 - 0xFFFF excluding 0xD800 - 0xFFFF - with an exception for surrogate pairs, which must be in the range - 0xD800-0xDBFF followed by 0xDC00-0xDFFF - - https://en.wikipedia.org/wiki/UTF-16 - """ - if not self.first_half_surrogate_pair_detected_16be: - if 0xD8 <= pair[0] <= 0xDB: - self.first_half_surrogate_pair_detected_16be = True - elif 0xDC <= pair[0] <= 0xDF: - self.invalid_utf16be = True - else: - if 0xDC <= pair[0] <= 0xDF: - self.first_half_surrogate_pair_detected_16be = False - else: - self.invalid_utf16be = True - - if not self.first_half_surrogate_pair_detected_16le: - if 0xD8 <= pair[1] <= 0xDB: - self.first_half_surrogate_pair_detected_16le = True - elif 0xDC <= pair[1] <= 0xDF: - self.invalid_utf16le = True - else: - if 0xDC <= pair[1] <= 0xDF: - self.first_half_surrogate_pair_detected_16le = False - else: - self.invalid_utf16le = True - - def feed(self, byte_str: Union[bytes, bytearray]) -> ProbingState: - for c in byte_str: - mod4 = self.position % 4 - self.quad[mod4] = c - if mod4 == 3: - self.validate_utf32_characters(self.quad) - self.validate_utf16_characters(self.quad[0:2]) - self.validate_utf16_characters(self.quad[2:4]) - if c == 0: - self.zeros_at_mod[mod4] += 1 - else: - self.nonzeros_at_mod[mod4] += 1 - self.position += 1 - return self.state - - @property - def state(self) -> ProbingState: - if self._state in {ProbingState.NOT_ME, ProbingState.FOUND_IT}: - # terminal, decided states - return self._state - if self.get_confidence() > 0.80: - self._state = ProbingState.FOUND_IT - elif self.position > 4 * 1024: - # if we get to 4kb into the file, and we can't conclude it's UTF, - # let's give up - self._state = ProbingState.NOT_ME - return self._state - - def get_confidence(self) -> float: - return ( - 0.85 - if ( - self.is_likely_utf16le() - or self.is_likely_utf16be() - or self.is_likely_utf32le() - or self.is_likely_utf32be() - ) - else 0.00 - ) diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/utf8prober.py b/venv/lib/python3.11/site-packages/pip/_vendor/chardet/utf8prober.py deleted file mode 100644 index d96354d..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/utf8prober.py +++ /dev/null @@ -1,82 +0,0 @@ -######################## BEGIN LICENSE BLOCK ######################## -# The Original Code is mozilla.org code. -# -# The Initial Developer of the Original Code is -# Netscape Communications Corporation. -# Portions created by the Initial Developer are Copyright (C) 1998 -# the Initial Developer. All Rights Reserved. -# -# Contributor(s): -# Mark Pilgrim - port to Python -# -# This library is free software; you can redistribute it and/or -# modify it under the terms of the GNU Lesser General Public -# License as published by the Free Software Foundation; either -# version 2.1 of the License, or (at your option) any later version. -# -# This library is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# Lesser General Public License for more details. -# -# You should have received a copy of the GNU Lesser General Public -# License along with this library; if not, write to the Free Software -# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA -# 02110-1301 USA -######################### END LICENSE BLOCK ######################### - -from typing import Union - -from .charsetprober import CharSetProber -from .codingstatemachine import CodingStateMachine -from .enums import MachineState, ProbingState -from .mbcssm import UTF8_SM_MODEL - - -class UTF8Prober(CharSetProber): - ONE_CHAR_PROB = 0.5 - - def __init__(self) -> None: - super().__init__() - self.coding_sm = CodingStateMachine(UTF8_SM_MODEL) - self._num_mb_chars = 0 - self.reset() - - def reset(self) -> None: - super().reset() - self.coding_sm.reset() - self._num_mb_chars = 0 - - @property - def charset_name(self) -> str: - return "utf-8" - - @property - def language(self) -> str: - return "" - - def feed(self, byte_str: Union[bytes, bytearray]) -> ProbingState: - for c in byte_str: - coding_state = self.coding_sm.next_state(c) - if coding_state == MachineState.ERROR: - self._state = ProbingState.NOT_ME - break - if coding_state == MachineState.ITS_ME: - self._state = ProbingState.FOUND_IT - break - if coding_state == MachineState.START: - if self.coding_sm.get_current_charlen() >= 2: - self._num_mb_chars += 1 - - if self.state == ProbingState.DETECTING: - if self.get_confidence() > self.SHORTCUT_THRESHOLD: - self._state = ProbingState.FOUND_IT - - return self.state - - def get_confidence(self) -> float: - unlike = 0.99 - if self._num_mb_chars < 6: - unlike *= self.ONE_CHAR_PROB**self._num_mb_chars - return 1.0 - unlike - return unlike diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/version.py b/venv/lib/python3.11/site-packages/pip/_vendor/chardet/version.py deleted file mode 100644 index c5e9d85..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/chardet/version.py +++ /dev/null @@ -1,9 +0,0 @@ -""" -This module exists only to simplify retrieving the version number of chardet -from within setuptools and from chardet subpackages. - -:author: Dan Blanchard (dan.blanchard@gmail.com) -""" - -__version__ = "5.1.0" -VERSION = __version__.split(".") diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/colorama/__init__.py b/venv/lib/python3.11/site-packages/pip/_vendor/colorama/__init__.py deleted file mode 100644 index 383101c..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/colorama/__init__.py +++ /dev/null @@ -1,7 +0,0 @@ -# Copyright Jonathan Hartley 2013. BSD 3-Clause license, see LICENSE file. -from .initialise import init, deinit, reinit, colorama_text, just_fix_windows_console -from .ansi import Fore, Back, Style, Cursor -from .ansitowin32 import AnsiToWin32 - -__version__ = '0.4.6' - diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/colorama/__pycache__/__init__.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/colorama/__pycache__/__init__.cpython-311.pyc deleted file mode 100644 index 3d18cc4..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/colorama/__pycache__/__init__.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/colorama/__pycache__/ansi.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/colorama/__pycache__/ansi.cpython-311.pyc deleted file mode 100644 index a9ac79e..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/colorama/__pycache__/ansi.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/colorama/__pycache__/ansitowin32.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/colorama/__pycache__/ansitowin32.cpython-311.pyc deleted file mode 100644 index 203c24a..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/colorama/__pycache__/ansitowin32.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/colorama/__pycache__/initialise.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/colorama/__pycache__/initialise.cpython-311.pyc deleted file mode 100644 index a9ca99c..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/colorama/__pycache__/initialise.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/colorama/__pycache__/win32.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/colorama/__pycache__/win32.cpython-311.pyc deleted file mode 100644 index a5b3898..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/colorama/__pycache__/win32.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/colorama/__pycache__/winterm.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/colorama/__pycache__/winterm.cpython-311.pyc deleted file mode 100644 index 9736f95..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/colorama/__pycache__/winterm.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/colorama/ansi.py b/venv/lib/python3.11/site-packages/pip/_vendor/colorama/ansi.py deleted file mode 100644 index 11ec695..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/colorama/ansi.py +++ /dev/null @@ -1,102 +0,0 @@ -# Copyright Jonathan Hartley 2013. BSD 3-Clause license, see LICENSE file. -''' -This module generates ANSI character codes to printing colors to terminals. -See: http://en.wikipedia.org/wiki/ANSI_escape_code -''' - -CSI = '\033[' -OSC = '\033]' -BEL = '\a' - - -def code_to_chars(code): - return CSI + str(code) + 'm' - -def set_title(title): - return OSC + '2;' + title + BEL - -def clear_screen(mode=2): - return CSI + str(mode) + 'J' - -def clear_line(mode=2): - return CSI + str(mode) + 'K' - - -class AnsiCodes(object): - def __init__(self): - # the subclasses declare class attributes which are numbers. - # Upon instantiation we define instance attributes, which are the same - # as the class attributes but wrapped with the ANSI escape sequence - for name in dir(self): - if not name.startswith('_'): - value = getattr(self, name) - setattr(self, name, code_to_chars(value)) - - -class AnsiCursor(object): - def UP(self, n=1): - return CSI + str(n) + 'A' - def DOWN(self, n=1): - return CSI + str(n) + 'B' - def FORWARD(self, n=1): - return CSI + str(n) + 'C' - def BACK(self, n=1): - return CSI + str(n) + 'D' - def POS(self, x=1, y=1): - return CSI + str(y) + ';' + str(x) + 'H' - - -class AnsiFore(AnsiCodes): - BLACK = 30 - RED = 31 - GREEN = 32 - YELLOW = 33 - BLUE = 34 - MAGENTA = 35 - CYAN = 36 - WHITE = 37 - RESET = 39 - - # These are fairly well supported, but not part of the standard. - LIGHTBLACK_EX = 90 - LIGHTRED_EX = 91 - LIGHTGREEN_EX = 92 - LIGHTYELLOW_EX = 93 - LIGHTBLUE_EX = 94 - LIGHTMAGENTA_EX = 95 - LIGHTCYAN_EX = 96 - LIGHTWHITE_EX = 97 - - -class AnsiBack(AnsiCodes): - BLACK = 40 - RED = 41 - GREEN = 42 - YELLOW = 43 - BLUE = 44 - MAGENTA = 45 - CYAN = 46 - WHITE = 47 - RESET = 49 - - # These are fairly well supported, but not part of the standard. - LIGHTBLACK_EX = 100 - LIGHTRED_EX = 101 - LIGHTGREEN_EX = 102 - LIGHTYELLOW_EX = 103 - LIGHTBLUE_EX = 104 - LIGHTMAGENTA_EX = 105 - LIGHTCYAN_EX = 106 - LIGHTWHITE_EX = 107 - - -class AnsiStyle(AnsiCodes): - BRIGHT = 1 - DIM = 2 - NORMAL = 22 - RESET_ALL = 0 - -Fore = AnsiFore() -Back = AnsiBack() -Style = AnsiStyle() -Cursor = AnsiCursor() diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/colorama/ansitowin32.py b/venv/lib/python3.11/site-packages/pip/_vendor/colorama/ansitowin32.py deleted file mode 100644 index abf209e..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/colorama/ansitowin32.py +++ /dev/null @@ -1,277 +0,0 @@ -# Copyright Jonathan Hartley 2013. BSD 3-Clause license, see LICENSE file. -import re -import sys -import os - -from .ansi import AnsiFore, AnsiBack, AnsiStyle, Style, BEL -from .winterm import enable_vt_processing, WinTerm, WinColor, WinStyle -from .win32 import windll, winapi_test - - -winterm = None -if windll is not None: - winterm = WinTerm() - - -class StreamWrapper(object): - ''' - Wraps a stream (such as stdout), acting as a transparent proxy for all - attribute access apart from method 'write()', which is delegated to our - Converter instance. - ''' - def __init__(self, wrapped, converter): - # double-underscore everything to prevent clashes with names of - # attributes on the wrapped stream object. - self.__wrapped = wrapped - self.__convertor = converter - - def __getattr__(self, name): - return getattr(self.__wrapped, name) - - def __enter__(self, *args, **kwargs): - # special method lookup bypasses __getattr__/__getattribute__, see - # https://stackoverflow.com/questions/12632894/why-doesnt-getattr-work-with-exit - # thus, contextlib magic methods are not proxied via __getattr__ - return self.__wrapped.__enter__(*args, **kwargs) - - def __exit__(self, *args, **kwargs): - return self.__wrapped.__exit__(*args, **kwargs) - - def __setstate__(self, state): - self.__dict__ = state - - def __getstate__(self): - return self.__dict__ - - def write(self, text): - self.__convertor.write(text) - - def isatty(self): - stream = self.__wrapped - if 'PYCHARM_HOSTED' in os.environ: - if stream is not None and (stream is sys.__stdout__ or stream is sys.__stderr__): - return True - try: - stream_isatty = stream.isatty - except AttributeError: - return False - else: - return stream_isatty() - - @property - def closed(self): - stream = self.__wrapped - try: - return stream.closed - # AttributeError in the case that the stream doesn't support being closed - # ValueError for the case that the stream has already been detached when atexit runs - except (AttributeError, ValueError): - return True - - -class AnsiToWin32(object): - ''' - Implements a 'write()' method which, on Windows, will strip ANSI character - sequences from the text, and if outputting to a tty, will convert them into - win32 function calls. - ''' - ANSI_CSI_RE = re.compile('\001?\033\\[((?:\\d|;)*)([a-zA-Z])\002?') # Control Sequence Introducer - ANSI_OSC_RE = re.compile('\001?\033\\]([^\a]*)(\a)\002?') # Operating System Command - - def __init__(self, wrapped, convert=None, strip=None, autoreset=False): - # The wrapped stream (normally sys.stdout or sys.stderr) - self.wrapped = wrapped - - # should we reset colors to defaults after every .write() - self.autoreset = autoreset - - # create the proxy wrapping our output stream - self.stream = StreamWrapper(wrapped, self) - - on_windows = os.name == 'nt' - # We test if the WinAPI works, because even if we are on Windows - # we may be using a terminal that doesn't support the WinAPI - # (e.g. Cygwin Terminal). In this case it's up to the terminal - # to support the ANSI codes. - conversion_supported = on_windows and winapi_test() - try: - fd = wrapped.fileno() - except Exception: - fd = -1 - system_has_native_ansi = not on_windows or enable_vt_processing(fd) - have_tty = not self.stream.closed and self.stream.isatty() - need_conversion = conversion_supported and not system_has_native_ansi - - # should we strip ANSI sequences from our output? - if strip is None: - strip = need_conversion or not have_tty - self.strip = strip - - # should we should convert ANSI sequences into win32 calls? - if convert is None: - convert = need_conversion and have_tty - self.convert = convert - - # dict of ansi codes to win32 functions and parameters - self.win32_calls = self.get_win32_calls() - - # are we wrapping stderr? - self.on_stderr = self.wrapped is sys.stderr - - def should_wrap(self): - ''' - True if this class is actually needed. If false, then the output - stream will not be affected, nor will win32 calls be issued, so - wrapping stdout is not actually required. This will generally be - False on non-Windows platforms, unless optional functionality like - autoreset has been requested using kwargs to init() - ''' - return self.convert or self.strip or self.autoreset - - def get_win32_calls(self): - if self.convert and winterm: - return { - AnsiStyle.RESET_ALL: (winterm.reset_all, ), - AnsiStyle.BRIGHT: (winterm.style, WinStyle.BRIGHT), - AnsiStyle.DIM: (winterm.style, WinStyle.NORMAL), - AnsiStyle.NORMAL: (winterm.style, WinStyle.NORMAL), - AnsiFore.BLACK: (winterm.fore, WinColor.BLACK), - AnsiFore.RED: (winterm.fore, WinColor.RED), - AnsiFore.GREEN: (winterm.fore, WinColor.GREEN), - AnsiFore.YELLOW: (winterm.fore, WinColor.YELLOW), - AnsiFore.BLUE: (winterm.fore, WinColor.BLUE), - AnsiFore.MAGENTA: (winterm.fore, WinColor.MAGENTA), - AnsiFore.CYAN: (winterm.fore, WinColor.CYAN), - AnsiFore.WHITE: (winterm.fore, WinColor.GREY), - AnsiFore.RESET: (winterm.fore, ), - AnsiFore.LIGHTBLACK_EX: (winterm.fore, WinColor.BLACK, True), - AnsiFore.LIGHTRED_EX: (winterm.fore, WinColor.RED, True), - AnsiFore.LIGHTGREEN_EX: (winterm.fore, WinColor.GREEN, True), - AnsiFore.LIGHTYELLOW_EX: (winterm.fore, WinColor.YELLOW, True), - AnsiFore.LIGHTBLUE_EX: (winterm.fore, WinColor.BLUE, True), - AnsiFore.LIGHTMAGENTA_EX: (winterm.fore, WinColor.MAGENTA, True), - AnsiFore.LIGHTCYAN_EX: (winterm.fore, WinColor.CYAN, True), - AnsiFore.LIGHTWHITE_EX: (winterm.fore, WinColor.GREY, True), - AnsiBack.BLACK: (winterm.back, WinColor.BLACK), - AnsiBack.RED: (winterm.back, WinColor.RED), - AnsiBack.GREEN: (winterm.back, WinColor.GREEN), - AnsiBack.YELLOW: (winterm.back, WinColor.YELLOW), - AnsiBack.BLUE: (winterm.back, WinColor.BLUE), - AnsiBack.MAGENTA: (winterm.back, WinColor.MAGENTA), - AnsiBack.CYAN: (winterm.back, WinColor.CYAN), - AnsiBack.WHITE: (winterm.back, WinColor.GREY), - AnsiBack.RESET: (winterm.back, ), - AnsiBack.LIGHTBLACK_EX: (winterm.back, WinColor.BLACK, True), - AnsiBack.LIGHTRED_EX: (winterm.back, WinColor.RED, True), - AnsiBack.LIGHTGREEN_EX: (winterm.back, WinColor.GREEN, True), - AnsiBack.LIGHTYELLOW_EX: (winterm.back, WinColor.YELLOW, True), - AnsiBack.LIGHTBLUE_EX: (winterm.back, WinColor.BLUE, True), - AnsiBack.LIGHTMAGENTA_EX: (winterm.back, WinColor.MAGENTA, True), - AnsiBack.LIGHTCYAN_EX: (winterm.back, WinColor.CYAN, True), - AnsiBack.LIGHTWHITE_EX: (winterm.back, WinColor.GREY, True), - } - return dict() - - def write(self, text): - if self.strip or self.convert: - self.write_and_convert(text) - else: - self.wrapped.write(text) - self.wrapped.flush() - if self.autoreset: - self.reset_all() - - - def reset_all(self): - if self.convert: - self.call_win32('m', (0,)) - elif not self.strip and not self.stream.closed: - self.wrapped.write(Style.RESET_ALL) - - - def write_and_convert(self, text): - ''' - Write the given text to our wrapped stream, stripping any ANSI - sequences from the text, and optionally converting them into win32 - calls. - ''' - cursor = 0 - text = self.convert_osc(text) - for match in self.ANSI_CSI_RE.finditer(text): - start, end = match.span() - self.write_plain_text(text, cursor, start) - self.convert_ansi(*match.groups()) - cursor = end - self.write_plain_text(text, cursor, len(text)) - - - def write_plain_text(self, text, start, end): - if start < end: - self.wrapped.write(text[start:end]) - self.wrapped.flush() - - - def convert_ansi(self, paramstring, command): - if self.convert: - params = self.extract_params(command, paramstring) - self.call_win32(command, params) - - - def extract_params(self, command, paramstring): - if command in 'Hf': - params = tuple(int(p) if len(p) != 0 else 1 for p in paramstring.split(';')) - while len(params) < 2: - # defaults: - params = params + (1,) - else: - params = tuple(int(p) for p in paramstring.split(';') if len(p) != 0) - if len(params) == 0: - # defaults: - if command in 'JKm': - params = (0,) - elif command in 'ABCD': - params = (1,) - - return params - - - def call_win32(self, command, params): - if command == 'm': - for param in params: - if param in self.win32_calls: - func_args = self.win32_calls[param] - func = func_args[0] - args = func_args[1:] - kwargs = dict(on_stderr=self.on_stderr) - func(*args, **kwargs) - elif command in 'J': - winterm.erase_screen(params[0], on_stderr=self.on_stderr) - elif command in 'K': - winterm.erase_line(params[0], on_stderr=self.on_stderr) - elif command in 'Hf': # cursor position - absolute - winterm.set_cursor_position(params, on_stderr=self.on_stderr) - elif command in 'ABCD': # cursor position - relative - n = params[0] - # A - up, B - down, C - forward, D - back - x, y = {'A': (0, -n), 'B': (0, n), 'C': (n, 0), 'D': (-n, 0)}[command] - winterm.cursor_adjust(x, y, on_stderr=self.on_stderr) - - - def convert_osc(self, text): - for match in self.ANSI_OSC_RE.finditer(text): - start, end = match.span() - text = text[:start] + text[end:] - paramstring, command = match.groups() - if command == BEL: - if paramstring.count(";") == 1: - params = paramstring.split(";") - # 0 - change title and icon (we will only change title) - # 1 - change icon (we don't support this) - # 2 - change title - if params[0] in '02': - winterm.set_title(params[1]) - return text - - - def flush(self): - self.wrapped.flush() diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/colorama/initialise.py b/venv/lib/python3.11/site-packages/pip/_vendor/colorama/initialise.py deleted file mode 100644 index d5fd4b7..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/colorama/initialise.py +++ /dev/null @@ -1,121 +0,0 @@ -# Copyright Jonathan Hartley 2013. BSD 3-Clause license, see LICENSE file. -import atexit -import contextlib -import sys - -from .ansitowin32 import AnsiToWin32 - - -def _wipe_internal_state_for_tests(): - global orig_stdout, orig_stderr - orig_stdout = None - orig_stderr = None - - global wrapped_stdout, wrapped_stderr - wrapped_stdout = None - wrapped_stderr = None - - global atexit_done - atexit_done = False - - global fixed_windows_console - fixed_windows_console = False - - try: - # no-op if it wasn't registered - atexit.unregister(reset_all) - except AttributeError: - # python 2: no atexit.unregister. Oh well, we did our best. - pass - - -def reset_all(): - if AnsiToWin32 is not None: # Issue #74: objects might become None at exit - AnsiToWin32(orig_stdout).reset_all() - - -def init(autoreset=False, convert=None, strip=None, wrap=True): - - if not wrap and any([autoreset, convert, strip]): - raise ValueError('wrap=False conflicts with any other arg=True') - - global wrapped_stdout, wrapped_stderr - global orig_stdout, orig_stderr - - orig_stdout = sys.stdout - orig_stderr = sys.stderr - - if sys.stdout is None: - wrapped_stdout = None - else: - sys.stdout = wrapped_stdout = \ - wrap_stream(orig_stdout, convert, strip, autoreset, wrap) - if sys.stderr is None: - wrapped_stderr = None - else: - sys.stderr = wrapped_stderr = \ - wrap_stream(orig_stderr, convert, strip, autoreset, wrap) - - global atexit_done - if not atexit_done: - atexit.register(reset_all) - atexit_done = True - - -def deinit(): - if orig_stdout is not None: - sys.stdout = orig_stdout - if orig_stderr is not None: - sys.stderr = orig_stderr - - -def just_fix_windows_console(): - global fixed_windows_console - - if sys.platform != "win32": - return - if fixed_windows_console: - return - if wrapped_stdout is not None or wrapped_stderr is not None: - # Someone already ran init() and it did stuff, so we won't second-guess them - return - - # On newer versions of Windows, AnsiToWin32.__init__ will implicitly enable the - # native ANSI support in the console as a side-effect. We only need to actually - # replace sys.stdout/stderr if we're in the old-style conversion mode. - new_stdout = AnsiToWin32(sys.stdout, convert=None, strip=None, autoreset=False) - if new_stdout.convert: - sys.stdout = new_stdout - new_stderr = AnsiToWin32(sys.stderr, convert=None, strip=None, autoreset=False) - if new_stderr.convert: - sys.stderr = new_stderr - - fixed_windows_console = True - -@contextlib.contextmanager -def colorama_text(*args, **kwargs): - init(*args, **kwargs) - try: - yield - finally: - deinit() - - -def reinit(): - if wrapped_stdout is not None: - sys.stdout = wrapped_stdout - if wrapped_stderr is not None: - sys.stderr = wrapped_stderr - - -def wrap_stream(stream, convert, strip, autoreset, wrap): - if wrap: - wrapper = AnsiToWin32(stream, - convert=convert, strip=strip, autoreset=autoreset) - if wrapper.should_wrap(): - stream = wrapper.stream - return stream - - -# Use this for initial setup as well, to reduce code duplication -_wipe_internal_state_for_tests() diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/colorama/tests/__init__.py b/venv/lib/python3.11/site-packages/pip/_vendor/colorama/tests/__init__.py deleted file mode 100644 index 8c5661e..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/colorama/tests/__init__.py +++ /dev/null @@ -1 +0,0 @@ -# Copyright Jonathan Hartley 2013. BSD 3-Clause license, see LICENSE file. diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/colorama/tests/__pycache__/__init__.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/colorama/tests/__pycache__/__init__.cpython-311.pyc deleted file mode 100644 index 52533b3..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/colorama/tests/__pycache__/__init__.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/colorama/tests/__pycache__/ansi_test.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/colorama/tests/__pycache__/ansi_test.cpython-311.pyc deleted file mode 100644 index 3f2ddd0..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/colorama/tests/__pycache__/ansi_test.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/colorama/tests/__pycache__/ansitowin32_test.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/colorama/tests/__pycache__/ansitowin32_test.cpython-311.pyc deleted file mode 100644 index a787d92..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/colorama/tests/__pycache__/ansitowin32_test.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/colorama/tests/__pycache__/initialise_test.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/colorama/tests/__pycache__/initialise_test.cpython-311.pyc deleted file mode 100644 index 4b463ec..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/colorama/tests/__pycache__/initialise_test.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/colorama/tests/__pycache__/isatty_test.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/colorama/tests/__pycache__/isatty_test.cpython-311.pyc deleted file mode 100644 index 6d8d0cb..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/colorama/tests/__pycache__/isatty_test.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/colorama/tests/__pycache__/utils.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/colorama/tests/__pycache__/utils.cpython-311.pyc deleted file mode 100644 index 088960c..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/colorama/tests/__pycache__/utils.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/colorama/tests/__pycache__/winterm_test.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/colorama/tests/__pycache__/winterm_test.cpython-311.pyc deleted file mode 100644 index 65eb421..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/colorama/tests/__pycache__/winterm_test.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/colorama/tests/ansi_test.py b/venv/lib/python3.11/site-packages/pip/_vendor/colorama/tests/ansi_test.py deleted file mode 100644 index 0a20c80..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/colorama/tests/ansi_test.py +++ /dev/null @@ -1,76 +0,0 @@ -# Copyright Jonathan Hartley 2013. BSD 3-Clause license, see LICENSE file. -import sys -from unittest import TestCase, main - -from ..ansi import Back, Fore, Style -from ..ansitowin32 import AnsiToWin32 - -stdout_orig = sys.stdout -stderr_orig = sys.stderr - - -class AnsiTest(TestCase): - - def setUp(self): - # sanity check: stdout should be a file or StringIO object. - # It will only be AnsiToWin32 if init() has previously wrapped it - self.assertNotEqual(type(sys.stdout), AnsiToWin32) - self.assertNotEqual(type(sys.stderr), AnsiToWin32) - - def tearDown(self): - sys.stdout = stdout_orig - sys.stderr = stderr_orig - - - def testForeAttributes(self): - self.assertEqual(Fore.BLACK, '\033[30m') - self.assertEqual(Fore.RED, '\033[31m') - self.assertEqual(Fore.GREEN, '\033[32m') - self.assertEqual(Fore.YELLOW, '\033[33m') - self.assertEqual(Fore.BLUE, '\033[34m') - self.assertEqual(Fore.MAGENTA, '\033[35m') - self.assertEqual(Fore.CYAN, '\033[36m') - self.assertEqual(Fore.WHITE, '\033[37m') - self.assertEqual(Fore.RESET, '\033[39m') - - # Check the light, extended versions. - self.assertEqual(Fore.LIGHTBLACK_EX, '\033[90m') - self.assertEqual(Fore.LIGHTRED_EX, '\033[91m') - self.assertEqual(Fore.LIGHTGREEN_EX, '\033[92m') - self.assertEqual(Fore.LIGHTYELLOW_EX, '\033[93m') - self.assertEqual(Fore.LIGHTBLUE_EX, '\033[94m') - self.assertEqual(Fore.LIGHTMAGENTA_EX, '\033[95m') - self.assertEqual(Fore.LIGHTCYAN_EX, '\033[96m') - self.assertEqual(Fore.LIGHTWHITE_EX, '\033[97m') - - - def testBackAttributes(self): - self.assertEqual(Back.BLACK, '\033[40m') - self.assertEqual(Back.RED, '\033[41m') - self.assertEqual(Back.GREEN, '\033[42m') - self.assertEqual(Back.YELLOW, '\033[43m') - self.assertEqual(Back.BLUE, '\033[44m') - self.assertEqual(Back.MAGENTA, '\033[45m') - self.assertEqual(Back.CYAN, '\033[46m') - self.assertEqual(Back.WHITE, '\033[47m') - self.assertEqual(Back.RESET, '\033[49m') - - # Check the light, extended versions. - self.assertEqual(Back.LIGHTBLACK_EX, '\033[100m') - self.assertEqual(Back.LIGHTRED_EX, '\033[101m') - self.assertEqual(Back.LIGHTGREEN_EX, '\033[102m') - self.assertEqual(Back.LIGHTYELLOW_EX, '\033[103m') - self.assertEqual(Back.LIGHTBLUE_EX, '\033[104m') - self.assertEqual(Back.LIGHTMAGENTA_EX, '\033[105m') - self.assertEqual(Back.LIGHTCYAN_EX, '\033[106m') - self.assertEqual(Back.LIGHTWHITE_EX, '\033[107m') - - - def testStyleAttributes(self): - self.assertEqual(Style.DIM, '\033[2m') - self.assertEqual(Style.NORMAL, '\033[22m') - self.assertEqual(Style.BRIGHT, '\033[1m') - - -if __name__ == '__main__': - main() diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/colorama/tests/ansitowin32_test.py b/venv/lib/python3.11/site-packages/pip/_vendor/colorama/tests/ansitowin32_test.py deleted file mode 100644 index 91ca551..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/colorama/tests/ansitowin32_test.py +++ /dev/null @@ -1,294 +0,0 @@ -# Copyright Jonathan Hartley 2013. BSD 3-Clause license, see LICENSE file. -from io import StringIO, TextIOWrapper -from unittest import TestCase, main -try: - from contextlib import ExitStack -except ImportError: - # python 2 - from contextlib2 import ExitStack - -try: - from unittest.mock import MagicMock, Mock, patch -except ImportError: - from mock import MagicMock, Mock, patch - -from ..ansitowin32 import AnsiToWin32, StreamWrapper -from ..win32 import ENABLE_VIRTUAL_TERMINAL_PROCESSING -from .utils import osname - - -class StreamWrapperTest(TestCase): - - def testIsAProxy(self): - mockStream = Mock() - wrapper = StreamWrapper(mockStream, None) - self.assertTrue( wrapper.random_attr is mockStream.random_attr ) - - def testDelegatesWrite(self): - mockStream = Mock() - mockConverter = Mock() - wrapper = StreamWrapper(mockStream, mockConverter) - wrapper.write('hello') - self.assertTrue(mockConverter.write.call_args, (('hello',), {})) - - def testDelegatesContext(self): - mockConverter = Mock() - s = StringIO() - with StreamWrapper(s, mockConverter) as fp: - fp.write(u'hello') - self.assertTrue(s.closed) - - def testProxyNoContextManager(self): - mockStream = MagicMock() - mockStream.__enter__.side_effect = AttributeError() - mockConverter = Mock() - with self.assertRaises(AttributeError) as excinfo: - with StreamWrapper(mockStream, mockConverter) as wrapper: - wrapper.write('hello') - - def test_closed_shouldnt_raise_on_closed_stream(self): - stream = StringIO() - stream.close() - wrapper = StreamWrapper(stream, None) - self.assertEqual(wrapper.closed, True) - - def test_closed_shouldnt_raise_on_detached_stream(self): - stream = TextIOWrapper(StringIO()) - stream.detach() - wrapper = StreamWrapper(stream, None) - self.assertEqual(wrapper.closed, True) - -class AnsiToWin32Test(TestCase): - - def testInit(self): - mockStdout = Mock() - auto = Mock() - stream = AnsiToWin32(mockStdout, autoreset=auto) - self.assertEqual(stream.wrapped, mockStdout) - self.assertEqual(stream.autoreset, auto) - - @patch('colorama.ansitowin32.winterm', None) - @patch('colorama.ansitowin32.winapi_test', lambda *_: True) - def testStripIsTrueOnWindows(self): - with osname('nt'): - mockStdout = Mock() - stream = AnsiToWin32(mockStdout) - self.assertTrue(stream.strip) - - def testStripIsFalseOffWindows(self): - with osname('posix'): - mockStdout = Mock(closed=False) - stream = AnsiToWin32(mockStdout) - self.assertFalse(stream.strip) - - def testWriteStripsAnsi(self): - mockStdout = Mock() - stream = AnsiToWin32(mockStdout) - stream.wrapped = Mock() - stream.write_and_convert = Mock() - stream.strip = True - - stream.write('abc') - - self.assertFalse(stream.wrapped.write.called) - self.assertEqual(stream.write_and_convert.call_args, (('abc',), {})) - - def testWriteDoesNotStripAnsi(self): - mockStdout = Mock() - stream = AnsiToWin32(mockStdout) - stream.wrapped = Mock() - stream.write_and_convert = Mock() - stream.strip = False - stream.convert = False - - stream.write('abc') - - self.assertFalse(stream.write_and_convert.called) - self.assertEqual(stream.wrapped.write.call_args, (('abc',), {})) - - def assert_autoresets(self, convert, autoreset=True): - stream = AnsiToWin32(Mock()) - stream.convert = convert - stream.reset_all = Mock() - stream.autoreset = autoreset - stream.winterm = Mock() - - stream.write('abc') - - self.assertEqual(stream.reset_all.called, autoreset) - - def testWriteAutoresets(self): - self.assert_autoresets(convert=True) - self.assert_autoresets(convert=False) - self.assert_autoresets(convert=True, autoreset=False) - self.assert_autoresets(convert=False, autoreset=False) - - def testWriteAndConvertWritesPlainText(self): - stream = AnsiToWin32(Mock()) - stream.write_and_convert( 'abc' ) - self.assertEqual( stream.wrapped.write.call_args, (('abc',), {}) ) - - def testWriteAndConvertStripsAllValidAnsi(self): - stream = AnsiToWin32(Mock()) - stream.call_win32 = Mock() - data = [ - 'abc\033[mdef', - 'abc\033[0mdef', - 'abc\033[2mdef', - 'abc\033[02mdef', - 'abc\033[002mdef', - 'abc\033[40mdef', - 'abc\033[040mdef', - 'abc\033[0;1mdef', - 'abc\033[40;50mdef', - 'abc\033[50;30;40mdef', - 'abc\033[Adef', - 'abc\033[0Gdef', - 'abc\033[1;20;128Hdef', - ] - for datum in data: - stream.wrapped.write.reset_mock() - stream.write_and_convert( datum ) - self.assertEqual( - [args[0] for args in stream.wrapped.write.call_args_list], - [ ('abc',), ('def',) ] - ) - - def testWriteAndConvertSkipsEmptySnippets(self): - stream = AnsiToWin32(Mock()) - stream.call_win32 = Mock() - stream.write_and_convert( '\033[40m\033[41m' ) - self.assertFalse( stream.wrapped.write.called ) - - def testWriteAndConvertCallsWin32WithParamsAndCommand(self): - stream = AnsiToWin32(Mock()) - stream.convert = True - stream.call_win32 = Mock() - stream.extract_params = Mock(return_value='params') - data = { - 'abc\033[adef': ('a', 'params'), - 'abc\033[;;bdef': ('b', 'params'), - 'abc\033[0cdef': ('c', 'params'), - 'abc\033[;;0;;Gdef': ('G', 'params'), - 'abc\033[1;20;128Hdef': ('H', 'params'), - } - for datum, expected in data.items(): - stream.call_win32.reset_mock() - stream.write_and_convert( datum ) - self.assertEqual( stream.call_win32.call_args[0], expected ) - - def test_reset_all_shouldnt_raise_on_closed_orig_stdout(self): - stream = StringIO() - converter = AnsiToWin32(stream) - stream.close() - - converter.reset_all() - - def test_wrap_shouldnt_raise_on_closed_orig_stdout(self): - stream = StringIO() - stream.close() - with \ - patch("colorama.ansitowin32.os.name", "nt"), \ - patch("colorama.ansitowin32.winapi_test", lambda: True): - converter = AnsiToWin32(stream) - self.assertTrue(converter.strip) - self.assertFalse(converter.convert) - - def test_wrap_shouldnt_raise_on_missing_closed_attr(self): - with \ - patch("colorama.ansitowin32.os.name", "nt"), \ - patch("colorama.ansitowin32.winapi_test", lambda: True): - converter = AnsiToWin32(object()) - self.assertTrue(converter.strip) - self.assertFalse(converter.convert) - - def testExtractParams(self): - stream = AnsiToWin32(Mock()) - data = { - '': (0,), - ';;': (0,), - '2': (2,), - ';;002;;': (2,), - '0;1': (0, 1), - ';;003;;456;;': (3, 456), - '11;22;33;44;55': (11, 22, 33, 44, 55), - } - for datum, expected in data.items(): - self.assertEqual(stream.extract_params('m', datum), expected) - - def testCallWin32UsesLookup(self): - listener = Mock() - stream = AnsiToWin32(listener) - stream.win32_calls = { - 1: (lambda *_, **__: listener(11),), - 2: (lambda *_, **__: listener(22),), - 3: (lambda *_, **__: listener(33),), - } - stream.call_win32('m', (3, 1, 99, 2)) - self.assertEqual( - [a[0][0] for a in listener.call_args_list], - [33, 11, 22] ) - - def test_osc_codes(self): - mockStdout = Mock() - stream = AnsiToWin32(mockStdout, convert=True) - with patch('colorama.ansitowin32.winterm') as winterm: - data = [ - '\033]0\x07', # missing arguments - '\033]0;foo\x08', # wrong OSC command - '\033]0;colorama_test_title\x07', # should work - '\033]1;colorama_test_title\x07', # wrong set command - '\033]2;colorama_test_title\x07', # should work - '\033]' + ';' * 64 + '\x08', # see issue #247 - ] - for code in data: - stream.write(code) - self.assertEqual(winterm.set_title.call_count, 2) - - def test_native_windows_ansi(self): - with ExitStack() as stack: - def p(a, b): - stack.enter_context(patch(a, b, create=True)) - # Pretend to be on Windows - p("colorama.ansitowin32.os.name", "nt") - p("colorama.ansitowin32.winapi_test", lambda: True) - p("colorama.win32.winapi_test", lambda: True) - p("colorama.winterm.win32.windll", "non-None") - p("colorama.winterm.get_osfhandle", lambda _: 1234) - - # Pretend that our mock stream has native ANSI support - p( - "colorama.winterm.win32.GetConsoleMode", - lambda _: ENABLE_VIRTUAL_TERMINAL_PROCESSING, - ) - SetConsoleMode = Mock() - p("colorama.winterm.win32.SetConsoleMode", SetConsoleMode) - - stdout = Mock() - stdout.closed = False - stdout.isatty.return_value = True - stdout.fileno.return_value = 1 - - # Our fake console says it has native vt support, so AnsiToWin32 should - # enable that support and do nothing else. - stream = AnsiToWin32(stdout) - SetConsoleMode.assert_called_with(1234, ENABLE_VIRTUAL_TERMINAL_PROCESSING) - self.assertFalse(stream.strip) - self.assertFalse(stream.convert) - self.assertFalse(stream.should_wrap()) - - # Now let's pretend we're on an old Windows console, that doesn't have - # native ANSI support. - p("colorama.winterm.win32.GetConsoleMode", lambda _: 0) - SetConsoleMode = Mock() - p("colorama.winterm.win32.SetConsoleMode", SetConsoleMode) - - stream = AnsiToWin32(stdout) - SetConsoleMode.assert_called_with(1234, ENABLE_VIRTUAL_TERMINAL_PROCESSING) - self.assertTrue(stream.strip) - self.assertTrue(stream.convert) - self.assertTrue(stream.should_wrap()) - - -if __name__ == '__main__': - main() diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/colorama/tests/initialise_test.py b/venv/lib/python3.11/site-packages/pip/_vendor/colorama/tests/initialise_test.py deleted file mode 100644 index 89f9b07..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/colorama/tests/initialise_test.py +++ /dev/null @@ -1,189 +0,0 @@ -# Copyright Jonathan Hartley 2013. BSD 3-Clause license, see LICENSE file. -import sys -from unittest import TestCase, main, skipUnless - -try: - from unittest.mock import patch, Mock -except ImportError: - from mock import patch, Mock - -from ..ansitowin32 import StreamWrapper -from ..initialise import init, just_fix_windows_console, _wipe_internal_state_for_tests -from .utils import osname, replace_by - -orig_stdout = sys.stdout -orig_stderr = sys.stderr - - -class InitTest(TestCase): - - @skipUnless(sys.stdout.isatty(), "sys.stdout is not a tty") - def setUp(self): - # sanity check - self.assertNotWrapped() - - def tearDown(self): - _wipe_internal_state_for_tests() - sys.stdout = orig_stdout - sys.stderr = orig_stderr - - def assertWrapped(self): - self.assertIsNot(sys.stdout, orig_stdout, 'stdout should be wrapped') - self.assertIsNot(sys.stderr, orig_stderr, 'stderr should be wrapped') - self.assertTrue(isinstance(sys.stdout, StreamWrapper), - 'bad stdout wrapper') - self.assertTrue(isinstance(sys.stderr, StreamWrapper), - 'bad stderr wrapper') - - def assertNotWrapped(self): - self.assertIs(sys.stdout, orig_stdout, 'stdout should not be wrapped') - self.assertIs(sys.stderr, orig_stderr, 'stderr should not be wrapped') - - @patch('colorama.initialise.reset_all') - @patch('colorama.ansitowin32.winapi_test', lambda *_: True) - @patch('colorama.ansitowin32.enable_vt_processing', lambda *_: False) - def testInitWrapsOnWindows(self, _): - with osname("nt"): - init() - self.assertWrapped() - - @patch('colorama.initialise.reset_all') - @patch('colorama.ansitowin32.winapi_test', lambda *_: False) - def testInitDoesntWrapOnEmulatedWindows(self, _): - with osname("nt"): - init() - self.assertNotWrapped() - - def testInitDoesntWrapOnNonWindows(self): - with osname("posix"): - init() - self.assertNotWrapped() - - def testInitDoesntWrapIfNone(self): - with replace_by(None): - init() - # We can't use assertNotWrapped here because replace_by(None) - # changes stdout/stderr already. - self.assertIsNone(sys.stdout) - self.assertIsNone(sys.stderr) - - def testInitAutoresetOnWrapsOnAllPlatforms(self): - with osname("posix"): - init(autoreset=True) - self.assertWrapped() - - def testInitWrapOffDoesntWrapOnWindows(self): - with osname("nt"): - init(wrap=False) - self.assertNotWrapped() - - def testInitWrapOffIncompatibleWithAutoresetOn(self): - self.assertRaises(ValueError, lambda: init(autoreset=True, wrap=False)) - - @patch('colorama.win32.SetConsoleTextAttribute') - @patch('colorama.initialise.AnsiToWin32') - def testAutoResetPassedOn(self, mockATW32, _): - with osname("nt"): - init(autoreset=True) - self.assertEqual(len(mockATW32.call_args_list), 2) - self.assertEqual(mockATW32.call_args_list[1][1]['autoreset'], True) - self.assertEqual(mockATW32.call_args_list[0][1]['autoreset'], True) - - @patch('colorama.initialise.AnsiToWin32') - def testAutoResetChangeable(self, mockATW32): - with osname("nt"): - init() - - init(autoreset=True) - self.assertEqual(len(mockATW32.call_args_list), 4) - self.assertEqual(mockATW32.call_args_list[2][1]['autoreset'], True) - self.assertEqual(mockATW32.call_args_list[3][1]['autoreset'], True) - - init() - self.assertEqual(len(mockATW32.call_args_list), 6) - self.assertEqual( - mockATW32.call_args_list[4][1]['autoreset'], False) - self.assertEqual( - mockATW32.call_args_list[5][1]['autoreset'], False) - - - @patch('colorama.initialise.atexit.register') - def testAtexitRegisteredOnlyOnce(self, mockRegister): - init() - self.assertTrue(mockRegister.called) - mockRegister.reset_mock() - init() - self.assertFalse(mockRegister.called) - - -class JustFixWindowsConsoleTest(TestCase): - def _reset(self): - _wipe_internal_state_for_tests() - sys.stdout = orig_stdout - sys.stderr = orig_stderr - - def tearDown(self): - self._reset() - - @patch("colorama.ansitowin32.winapi_test", lambda: True) - def testJustFixWindowsConsole(self): - if sys.platform != "win32": - # just_fix_windows_console should be a no-op - just_fix_windows_console() - self.assertIs(sys.stdout, orig_stdout) - self.assertIs(sys.stderr, orig_stderr) - else: - def fake_std(): - # Emulate stdout=not a tty, stderr=tty - # to check that we handle both cases correctly - stdout = Mock() - stdout.closed = False - stdout.isatty.return_value = False - stdout.fileno.return_value = 1 - sys.stdout = stdout - - stderr = Mock() - stderr.closed = False - stderr.isatty.return_value = True - stderr.fileno.return_value = 2 - sys.stderr = stderr - - for native_ansi in [False, True]: - with patch( - 'colorama.ansitowin32.enable_vt_processing', - lambda *_: native_ansi - ): - self._reset() - fake_std() - - # Regular single-call test - prev_stdout = sys.stdout - prev_stderr = sys.stderr - just_fix_windows_console() - self.assertIs(sys.stdout, prev_stdout) - if native_ansi: - self.assertIs(sys.stderr, prev_stderr) - else: - self.assertIsNot(sys.stderr, prev_stderr) - - # second call without resetting is always a no-op - prev_stdout = sys.stdout - prev_stderr = sys.stderr - just_fix_windows_console() - self.assertIs(sys.stdout, prev_stdout) - self.assertIs(sys.stderr, prev_stderr) - - self._reset() - fake_std() - - # If init() runs first, just_fix_windows_console should be a no-op - init() - prev_stdout = sys.stdout - prev_stderr = sys.stderr - just_fix_windows_console() - self.assertIs(prev_stdout, sys.stdout) - self.assertIs(prev_stderr, sys.stderr) - - -if __name__ == '__main__': - main() diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/colorama/tests/isatty_test.py b/venv/lib/python3.11/site-packages/pip/_vendor/colorama/tests/isatty_test.py deleted file mode 100644 index 0f84e4b..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/colorama/tests/isatty_test.py +++ /dev/null @@ -1,57 +0,0 @@ -# Copyright Jonathan Hartley 2013. BSD 3-Clause license, see LICENSE file. -import sys -from unittest import TestCase, main - -from ..ansitowin32 import StreamWrapper, AnsiToWin32 -from .utils import pycharm, replace_by, replace_original_by, StreamTTY, StreamNonTTY - - -def is_a_tty(stream): - return StreamWrapper(stream, None).isatty() - -class IsattyTest(TestCase): - - def test_TTY(self): - tty = StreamTTY() - self.assertTrue(is_a_tty(tty)) - with pycharm(): - self.assertTrue(is_a_tty(tty)) - - def test_nonTTY(self): - non_tty = StreamNonTTY() - self.assertFalse(is_a_tty(non_tty)) - with pycharm(): - self.assertFalse(is_a_tty(non_tty)) - - def test_withPycharm(self): - with pycharm(): - self.assertTrue(is_a_tty(sys.stderr)) - self.assertTrue(is_a_tty(sys.stdout)) - - def test_withPycharmTTYOverride(self): - tty = StreamTTY() - with pycharm(), replace_by(tty): - self.assertTrue(is_a_tty(tty)) - - def test_withPycharmNonTTYOverride(self): - non_tty = StreamNonTTY() - with pycharm(), replace_by(non_tty): - self.assertFalse(is_a_tty(non_tty)) - - def test_withPycharmNoneOverride(self): - with pycharm(): - with replace_by(None), replace_original_by(None): - self.assertFalse(is_a_tty(None)) - self.assertFalse(is_a_tty(StreamNonTTY())) - self.assertTrue(is_a_tty(StreamTTY())) - - def test_withPycharmStreamWrapped(self): - with pycharm(): - self.assertTrue(AnsiToWin32(StreamTTY()).stream.isatty()) - self.assertFalse(AnsiToWin32(StreamNonTTY()).stream.isatty()) - self.assertTrue(AnsiToWin32(sys.stdout).stream.isatty()) - self.assertTrue(AnsiToWin32(sys.stderr).stream.isatty()) - - -if __name__ == '__main__': - main() diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/colorama/tests/utils.py b/venv/lib/python3.11/site-packages/pip/_vendor/colorama/tests/utils.py deleted file mode 100644 index 472fafb..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/colorama/tests/utils.py +++ /dev/null @@ -1,49 +0,0 @@ -# Copyright Jonathan Hartley 2013. BSD 3-Clause license, see LICENSE file. -from contextlib import contextmanager -from io import StringIO -import sys -import os - - -class StreamTTY(StringIO): - def isatty(self): - return True - -class StreamNonTTY(StringIO): - def isatty(self): - return False - -@contextmanager -def osname(name): - orig = os.name - os.name = name - yield - os.name = orig - -@contextmanager -def replace_by(stream): - orig_stdout = sys.stdout - orig_stderr = sys.stderr - sys.stdout = stream - sys.stderr = stream - yield - sys.stdout = orig_stdout - sys.stderr = orig_stderr - -@contextmanager -def replace_original_by(stream): - orig_stdout = sys.__stdout__ - orig_stderr = sys.__stderr__ - sys.__stdout__ = stream - sys.__stderr__ = stream - yield - sys.__stdout__ = orig_stdout - sys.__stderr__ = orig_stderr - -@contextmanager -def pycharm(): - os.environ["PYCHARM_HOSTED"] = "1" - non_tty = StreamNonTTY() - with replace_by(non_tty), replace_original_by(non_tty): - yield - del os.environ["PYCHARM_HOSTED"] diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/colorama/tests/winterm_test.py b/venv/lib/python3.11/site-packages/pip/_vendor/colorama/tests/winterm_test.py deleted file mode 100644 index d0955f9..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/colorama/tests/winterm_test.py +++ /dev/null @@ -1,131 +0,0 @@ -# Copyright Jonathan Hartley 2013. BSD 3-Clause license, see LICENSE file. -import sys -from unittest import TestCase, main, skipUnless - -try: - from unittest.mock import Mock, patch -except ImportError: - from mock import Mock, patch - -from ..winterm import WinColor, WinStyle, WinTerm - - -class WinTermTest(TestCase): - - @patch('colorama.winterm.win32') - def testInit(self, mockWin32): - mockAttr = Mock() - mockAttr.wAttributes = 7 + 6 * 16 + 8 - mockWin32.GetConsoleScreenBufferInfo.return_value = mockAttr - term = WinTerm() - self.assertEqual(term._fore, 7) - self.assertEqual(term._back, 6) - self.assertEqual(term._style, 8) - - @skipUnless(sys.platform.startswith("win"), "requires Windows") - def testGetAttrs(self): - term = WinTerm() - - term._fore = 0 - term._back = 0 - term._style = 0 - self.assertEqual(term.get_attrs(), 0) - - term._fore = WinColor.YELLOW - self.assertEqual(term.get_attrs(), WinColor.YELLOW) - - term._back = WinColor.MAGENTA - self.assertEqual( - term.get_attrs(), - WinColor.YELLOW + WinColor.MAGENTA * 16) - - term._style = WinStyle.BRIGHT - self.assertEqual( - term.get_attrs(), - WinColor.YELLOW + WinColor.MAGENTA * 16 + WinStyle.BRIGHT) - - @patch('colorama.winterm.win32') - def testResetAll(self, mockWin32): - mockAttr = Mock() - mockAttr.wAttributes = 1 + 2 * 16 + 8 - mockWin32.GetConsoleScreenBufferInfo.return_value = mockAttr - term = WinTerm() - - term.set_console = Mock() - term._fore = -1 - term._back = -1 - term._style = -1 - - term.reset_all() - - self.assertEqual(term._fore, 1) - self.assertEqual(term._back, 2) - self.assertEqual(term._style, 8) - self.assertEqual(term.set_console.called, True) - - @skipUnless(sys.platform.startswith("win"), "requires Windows") - def testFore(self): - term = WinTerm() - term.set_console = Mock() - term._fore = 0 - - term.fore(5) - - self.assertEqual(term._fore, 5) - self.assertEqual(term.set_console.called, True) - - @skipUnless(sys.platform.startswith("win"), "requires Windows") - def testBack(self): - term = WinTerm() - term.set_console = Mock() - term._back = 0 - - term.back(5) - - self.assertEqual(term._back, 5) - self.assertEqual(term.set_console.called, True) - - @skipUnless(sys.platform.startswith("win"), "requires Windows") - def testStyle(self): - term = WinTerm() - term.set_console = Mock() - term._style = 0 - - term.style(22) - - self.assertEqual(term._style, 22) - self.assertEqual(term.set_console.called, True) - - @patch('colorama.winterm.win32') - def testSetConsole(self, mockWin32): - mockAttr = Mock() - mockAttr.wAttributes = 0 - mockWin32.GetConsoleScreenBufferInfo.return_value = mockAttr - term = WinTerm() - term.windll = Mock() - - term.set_console() - - self.assertEqual( - mockWin32.SetConsoleTextAttribute.call_args, - ((mockWin32.STDOUT, term.get_attrs()), {}) - ) - - @patch('colorama.winterm.win32') - def testSetConsoleOnStderr(self, mockWin32): - mockAttr = Mock() - mockAttr.wAttributes = 0 - mockWin32.GetConsoleScreenBufferInfo.return_value = mockAttr - term = WinTerm() - term.windll = Mock() - - term.set_console(on_stderr=True) - - self.assertEqual( - mockWin32.SetConsoleTextAttribute.call_args, - ((mockWin32.STDERR, term.get_attrs()), {}) - ) - - -if __name__ == '__main__': - main() diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/colorama/win32.py b/venv/lib/python3.11/site-packages/pip/_vendor/colorama/win32.py deleted file mode 100644 index 841b0e2..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/colorama/win32.py +++ /dev/null @@ -1,180 +0,0 @@ -# Copyright Jonathan Hartley 2013. BSD 3-Clause license, see LICENSE file. - -# from winbase.h -STDOUT = -11 -STDERR = -12 - -ENABLE_VIRTUAL_TERMINAL_PROCESSING = 0x0004 - -try: - import ctypes - from ctypes import LibraryLoader - windll = LibraryLoader(ctypes.WinDLL) - from ctypes import wintypes -except (AttributeError, ImportError): - windll = None - SetConsoleTextAttribute = lambda *_: None - winapi_test = lambda *_: None -else: - from ctypes import byref, Structure, c_char, POINTER - - COORD = wintypes._COORD - - class CONSOLE_SCREEN_BUFFER_INFO(Structure): - """struct in wincon.h.""" - _fields_ = [ - ("dwSize", COORD), - ("dwCursorPosition", COORD), - ("wAttributes", wintypes.WORD), - ("srWindow", wintypes.SMALL_RECT), - ("dwMaximumWindowSize", COORD), - ] - def __str__(self): - return '(%d,%d,%d,%d,%d,%d,%d,%d,%d,%d,%d)' % ( - self.dwSize.Y, self.dwSize.X - , self.dwCursorPosition.Y, self.dwCursorPosition.X - , self.wAttributes - , self.srWindow.Top, self.srWindow.Left, self.srWindow.Bottom, self.srWindow.Right - , self.dwMaximumWindowSize.Y, self.dwMaximumWindowSize.X - ) - - _GetStdHandle = windll.kernel32.GetStdHandle - _GetStdHandle.argtypes = [ - wintypes.DWORD, - ] - _GetStdHandle.restype = wintypes.HANDLE - - _GetConsoleScreenBufferInfo = windll.kernel32.GetConsoleScreenBufferInfo - _GetConsoleScreenBufferInfo.argtypes = [ - wintypes.HANDLE, - POINTER(CONSOLE_SCREEN_BUFFER_INFO), - ] - _GetConsoleScreenBufferInfo.restype = wintypes.BOOL - - _SetConsoleTextAttribute = windll.kernel32.SetConsoleTextAttribute - _SetConsoleTextAttribute.argtypes = [ - wintypes.HANDLE, - wintypes.WORD, - ] - _SetConsoleTextAttribute.restype = wintypes.BOOL - - _SetConsoleCursorPosition = windll.kernel32.SetConsoleCursorPosition - _SetConsoleCursorPosition.argtypes = [ - wintypes.HANDLE, - COORD, - ] - _SetConsoleCursorPosition.restype = wintypes.BOOL - - _FillConsoleOutputCharacterA = windll.kernel32.FillConsoleOutputCharacterA - _FillConsoleOutputCharacterA.argtypes = [ - wintypes.HANDLE, - c_char, - wintypes.DWORD, - COORD, - POINTER(wintypes.DWORD), - ] - _FillConsoleOutputCharacterA.restype = wintypes.BOOL - - _FillConsoleOutputAttribute = windll.kernel32.FillConsoleOutputAttribute - _FillConsoleOutputAttribute.argtypes = [ - wintypes.HANDLE, - wintypes.WORD, - wintypes.DWORD, - COORD, - POINTER(wintypes.DWORD), - ] - _FillConsoleOutputAttribute.restype = wintypes.BOOL - - _SetConsoleTitleW = windll.kernel32.SetConsoleTitleW - _SetConsoleTitleW.argtypes = [ - wintypes.LPCWSTR - ] - _SetConsoleTitleW.restype = wintypes.BOOL - - _GetConsoleMode = windll.kernel32.GetConsoleMode - _GetConsoleMode.argtypes = [ - wintypes.HANDLE, - POINTER(wintypes.DWORD) - ] - _GetConsoleMode.restype = wintypes.BOOL - - _SetConsoleMode = windll.kernel32.SetConsoleMode - _SetConsoleMode.argtypes = [ - wintypes.HANDLE, - wintypes.DWORD - ] - _SetConsoleMode.restype = wintypes.BOOL - - def _winapi_test(handle): - csbi = CONSOLE_SCREEN_BUFFER_INFO() - success = _GetConsoleScreenBufferInfo( - handle, byref(csbi)) - return bool(success) - - def winapi_test(): - return any(_winapi_test(h) for h in - (_GetStdHandle(STDOUT), _GetStdHandle(STDERR))) - - def GetConsoleScreenBufferInfo(stream_id=STDOUT): - handle = _GetStdHandle(stream_id) - csbi = CONSOLE_SCREEN_BUFFER_INFO() - success = _GetConsoleScreenBufferInfo( - handle, byref(csbi)) - return csbi - - def SetConsoleTextAttribute(stream_id, attrs): - handle = _GetStdHandle(stream_id) - return _SetConsoleTextAttribute(handle, attrs) - - def SetConsoleCursorPosition(stream_id, position, adjust=True): - position = COORD(*position) - # If the position is out of range, do nothing. - if position.Y <= 0 or position.X <= 0: - return - # Adjust for Windows' SetConsoleCursorPosition: - # 1. being 0-based, while ANSI is 1-based. - # 2. expecting (x,y), while ANSI uses (y,x). - adjusted_position = COORD(position.Y - 1, position.X - 1) - if adjust: - # Adjust for viewport's scroll position - sr = GetConsoleScreenBufferInfo(STDOUT).srWindow - adjusted_position.Y += sr.Top - adjusted_position.X += sr.Left - # Resume normal processing - handle = _GetStdHandle(stream_id) - return _SetConsoleCursorPosition(handle, adjusted_position) - - def FillConsoleOutputCharacter(stream_id, char, length, start): - handle = _GetStdHandle(stream_id) - char = c_char(char.encode()) - length = wintypes.DWORD(length) - num_written = wintypes.DWORD(0) - # Note that this is hard-coded for ANSI (vs wide) bytes. - success = _FillConsoleOutputCharacterA( - handle, char, length, start, byref(num_written)) - return num_written.value - - def FillConsoleOutputAttribute(stream_id, attr, length, start): - ''' FillConsoleOutputAttribute( hConsole, csbi.wAttributes, dwConSize, coordScreen, &cCharsWritten )''' - handle = _GetStdHandle(stream_id) - attribute = wintypes.WORD(attr) - length = wintypes.DWORD(length) - num_written = wintypes.DWORD(0) - # Note that this is hard-coded for ANSI (vs wide) bytes. - return _FillConsoleOutputAttribute( - handle, attribute, length, start, byref(num_written)) - - def SetConsoleTitle(title): - return _SetConsoleTitleW(title) - - def GetConsoleMode(handle): - mode = wintypes.DWORD() - success = _GetConsoleMode(handle, byref(mode)) - if not success: - raise ctypes.WinError() - return mode.value - - def SetConsoleMode(handle, mode): - success = _SetConsoleMode(handle, mode) - if not success: - raise ctypes.WinError() diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/colorama/winterm.py b/venv/lib/python3.11/site-packages/pip/_vendor/colorama/winterm.py deleted file mode 100644 index aad867e..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/colorama/winterm.py +++ /dev/null @@ -1,195 +0,0 @@ -# Copyright Jonathan Hartley 2013. BSD 3-Clause license, see LICENSE file. -try: - from msvcrt import get_osfhandle -except ImportError: - def get_osfhandle(_): - raise OSError("This isn't windows!") - - -from . import win32 - -# from wincon.h -class WinColor(object): - BLACK = 0 - BLUE = 1 - GREEN = 2 - CYAN = 3 - RED = 4 - MAGENTA = 5 - YELLOW = 6 - GREY = 7 - -# from wincon.h -class WinStyle(object): - NORMAL = 0x00 # dim text, dim background - BRIGHT = 0x08 # bright text, dim background - BRIGHT_BACKGROUND = 0x80 # dim text, bright background - -class WinTerm(object): - - def __init__(self): - self._default = win32.GetConsoleScreenBufferInfo(win32.STDOUT).wAttributes - self.set_attrs(self._default) - self._default_fore = self._fore - self._default_back = self._back - self._default_style = self._style - # In order to emulate LIGHT_EX in windows, we borrow the BRIGHT style. - # So that LIGHT_EX colors and BRIGHT style do not clobber each other, - # we track them separately, since LIGHT_EX is overwritten by Fore/Back - # and BRIGHT is overwritten by Style codes. - self._light = 0 - - def get_attrs(self): - return self._fore + self._back * 16 + (self._style | self._light) - - def set_attrs(self, value): - self._fore = value & 7 - self._back = (value >> 4) & 7 - self._style = value & (WinStyle.BRIGHT | WinStyle.BRIGHT_BACKGROUND) - - def reset_all(self, on_stderr=None): - self.set_attrs(self._default) - self.set_console(attrs=self._default) - self._light = 0 - - def fore(self, fore=None, light=False, on_stderr=False): - if fore is None: - fore = self._default_fore - self._fore = fore - # Emulate LIGHT_EX with BRIGHT Style - if light: - self._light |= WinStyle.BRIGHT - else: - self._light &= ~WinStyle.BRIGHT - self.set_console(on_stderr=on_stderr) - - def back(self, back=None, light=False, on_stderr=False): - if back is None: - back = self._default_back - self._back = back - # Emulate LIGHT_EX with BRIGHT_BACKGROUND Style - if light: - self._light |= WinStyle.BRIGHT_BACKGROUND - else: - self._light &= ~WinStyle.BRIGHT_BACKGROUND - self.set_console(on_stderr=on_stderr) - - def style(self, style=None, on_stderr=False): - if style is None: - style = self._default_style - self._style = style - self.set_console(on_stderr=on_stderr) - - def set_console(self, attrs=None, on_stderr=False): - if attrs is None: - attrs = self.get_attrs() - handle = win32.STDOUT - if on_stderr: - handle = win32.STDERR - win32.SetConsoleTextAttribute(handle, attrs) - - def get_position(self, handle): - position = win32.GetConsoleScreenBufferInfo(handle).dwCursorPosition - # Because Windows coordinates are 0-based, - # and win32.SetConsoleCursorPosition expects 1-based. - position.X += 1 - position.Y += 1 - return position - - def set_cursor_position(self, position=None, on_stderr=False): - if position is None: - # I'm not currently tracking the position, so there is no default. - # position = self.get_position() - return - handle = win32.STDOUT - if on_stderr: - handle = win32.STDERR - win32.SetConsoleCursorPosition(handle, position) - - def cursor_adjust(self, x, y, on_stderr=False): - handle = win32.STDOUT - if on_stderr: - handle = win32.STDERR - position = self.get_position(handle) - adjusted_position = (position.Y + y, position.X + x) - win32.SetConsoleCursorPosition(handle, adjusted_position, adjust=False) - - def erase_screen(self, mode=0, on_stderr=False): - # 0 should clear from the cursor to the end of the screen. - # 1 should clear from the cursor to the beginning of the screen. - # 2 should clear the entire screen, and move cursor to (1,1) - handle = win32.STDOUT - if on_stderr: - handle = win32.STDERR - csbi = win32.GetConsoleScreenBufferInfo(handle) - # get the number of character cells in the current buffer - cells_in_screen = csbi.dwSize.X * csbi.dwSize.Y - # get number of character cells before current cursor position - cells_before_cursor = csbi.dwSize.X * csbi.dwCursorPosition.Y + csbi.dwCursorPosition.X - if mode == 0: - from_coord = csbi.dwCursorPosition - cells_to_erase = cells_in_screen - cells_before_cursor - elif mode == 1: - from_coord = win32.COORD(0, 0) - cells_to_erase = cells_before_cursor - elif mode == 2: - from_coord = win32.COORD(0, 0) - cells_to_erase = cells_in_screen - else: - # invalid mode - return - # fill the entire screen with blanks - win32.FillConsoleOutputCharacter(handle, ' ', cells_to_erase, from_coord) - # now set the buffer's attributes accordingly - win32.FillConsoleOutputAttribute(handle, self.get_attrs(), cells_to_erase, from_coord) - if mode == 2: - # put the cursor where needed - win32.SetConsoleCursorPosition(handle, (1, 1)) - - def erase_line(self, mode=0, on_stderr=False): - # 0 should clear from the cursor to the end of the line. - # 1 should clear from the cursor to the beginning of the line. - # 2 should clear the entire line. - handle = win32.STDOUT - if on_stderr: - handle = win32.STDERR - csbi = win32.GetConsoleScreenBufferInfo(handle) - if mode == 0: - from_coord = csbi.dwCursorPosition - cells_to_erase = csbi.dwSize.X - csbi.dwCursorPosition.X - elif mode == 1: - from_coord = win32.COORD(0, csbi.dwCursorPosition.Y) - cells_to_erase = csbi.dwCursorPosition.X - elif mode == 2: - from_coord = win32.COORD(0, csbi.dwCursorPosition.Y) - cells_to_erase = csbi.dwSize.X - else: - # invalid mode - return - # fill the entire screen with blanks - win32.FillConsoleOutputCharacter(handle, ' ', cells_to_erase, from_coord) - # now set the buffer's attributes accordingly - win32.FillConsoleOutputAttribute(handle, self.get_attrs(), cells_to_erase, from_coord) - - def set_title(self, title): - win32.SetConsoleTitle(title) - - -def enable_vt_processing(fd): - if win32.windll is None or not win32.winapi_test(): - return False - - try: - handle = get_osfhandle(fd) - mode = win32.GetConsoleMode(handle) - win32.SetConsoleMode( - handle, - mode | win32.ENABLE_VIRTUAL_TERMINAL_PROCESSING, - ) - - mode = win32.GetConsoleMode(handle) - if mode & win32.ENABLE_VIRTUAL_TERMINAL_PROCESSING: - return True - # Can get TypeError in testsuite where 'fd' is a Mock() - except (OSError, TypeError): - return False diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/distlib/__init__.py b/venv/lib/python3.11/site-packages/pip/_vendor/distlib/__init__.py deleted file mode 100644 index 962173c..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/distlib/__init__.py +++ /dev/null @@ -1,23 +0,0 @@ -# -*- coding: utf-8 -*- -# -# Copyright (C) 2012-2022 Vinay Sajip. -# Licensed to the Python Software Foundation under a contributor agreement. -# See LICENSE.txt and CONTRIBUTORS.txt. -# -import logging - -__version__ = '0.3.6' - -class DistlibException(Exception): - pass - -try: - from logging import NullHandler -except ImportError: # pragma: no cover - class NullHandler(logging.Handler): - def handle(self, record): pass - def emit(self, record): pass - def createLock(self): self.lock = None - -logger = logging.getLogger(__name__) -logger.addHandler(NullHandler()) diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/distlib/__pycache__/__init__.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/distlib/__pycache__/__init__.cpython-311.pyc deleted file mode 100644 index 41e7c53..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/distlib/__pycache__/__init__.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/distlib/__pycache__/compat.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/distlib/__pycache__/compat.cpython-311.pyc deleted file mode 100644 index 63415cb..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/distlib/__pycache__/compat.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/distlib/__pycache__/database.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/distlib/__pycache__/database.cpython-311.pyc deleted file mode 100644 index 719399f..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/distlib/__pycache__/database.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/distlib/__pycache__/index.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/distlib/__pycache__/index.cpython-311.pyc deleted file mode 100644 index a97c1fc..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/distlib/__pycache__/index.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/distlib/__pycache__/locators.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/distlib/__pycache__/locators.cpython-311.pyc deleted file mode 100644 index ceec740..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/distlib/__pycache__/locators.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/distlib/__pycache__/manifest.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/distlib/__pycache__/manifest.cpython-311.pyc deleted file mode 100644 index f962324..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/distlib/__pycache__/manifest.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/distlib/__pycache__/markers.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/distlib/__pycache__/markers.cpython-311.pyc deleted file mode 100644 index 0faac04..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/distlib/__pycache__/markers.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/distlib/__pycache__/metadata.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/distlib/__pycache__/metadata.cpython-311.pyc deleted file mode 100644 index b475c50..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/distlib/__pycache__/metadata.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/distlib/__pycache__/resources.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/distlib/__pycache__/resources.cpython-311.pyc deleted file mode 100644 index a92cff4..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/distlib/__pycache__/resources.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/distlib/__pycache__/scripts.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/distlib/__pycache__/scripts.cpython-311.pyc deleted file mode 100644 index e46c9f9..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/distlib/__pycache__/scripts.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/distlib/__pycache__/util.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/distlib/__pycache__/util.cpython-311.pyc deleted file mode 100644 index b9ad2da..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/distlib/__pycache__/util.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/distlib/__pycache__/version.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/distlib/__pycache__/version.cpython-311.pyc deleted file mode 100644 index bf0cbee..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/distlib/__pycache__/version.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/distlib/__pycache__/wheel.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/distlib/__pycache__/wheel.cpython-311.pyc deleted file mode 100644 index 8db3473..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/distlib/__pycache__/wheel.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/distlib/compat.py b/venv/lib/python3.11/site-packages/pip/_vendor/distlib/compat.py deleted file mode 100644 index 1fe3d22..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/distlib/compat.py +++ /dev/null @@ -1,1116 +0,0 @@ -# -*- coding: utf-8 -*- -# -# Copyright (C) 2013-2017 Vinay Sajip. -# Licensed to the Python Software Foundation under a contributor agreement. -# See LICENSE.txt and CONTRIBUTORS.txt. -# -from __future__ import absolute_import - -import os -import re -import sys - -try: - import ssl -except ImportError: # pragma: no cover - ssl = None - -if sys.version_info[0] < 3: # pragma: no cover - from StringIO import StringIO - string_types = basestring, - text_type = unicode - from types import FileType as file_type - import __builtin__ as builtins - import ConfigParser as configparser - from urlparse import urlparse, urlunparse, urljoin, urlsplit, urlunsplit - from urllib import (urlretrieve, quote as _quote, unquote, url2pathname, - pathname2url, ContentTooShortError, splittype) - - def quote(s): - if isinstance(s, unicode): - s = s.encode('utf-8') - return _quote(s) - - import urllib2 - from urllib2 import (Request, urlopen, URLError, HTTPError, - HTTPBasicAuthHandler, HTTPPasswordMgr, - HTTPHandler, HTTPRedirectHandler, - build_opener) - if ssl: - from urllib2 import HTTPSHandler - import httplib - import xmlrpclib - import Queue as queue - from HTMLParser import HTMLParser - import htmlentitydefs - raw_input = raw_input - from itertools import ifilter as filter - from itertools import ifilterfalse as filterfalse - - # Leaving this around for now, in case it needs resurrecting in some way - # _userprog = None - # def splituser(host): - # """splituser('user[:passwd]@host[:port]') --> 'user[:passwd]', 'host[:port]'.""" - # global _userprog - # if _userprog is None: - # import re - # _userprog = re.compile('^(.*)@(.*)$') - - # match = _userprog.match(host) - # if match: return match.group(1, 2) - # return None, host - -else: # pragma: no cover - from io import StringIO - string_types = str, - text_type = str - from io import TextIOWrapper as file_type - import builtins - import configparser - import shutil - from urllib.parse import (urlparse, urlunparse, urljoin, quote, - unquote, urlsplit, urlunsplit, splittype) - from urllib.request import (urlopen, urlretrieve, Request, url2pathname, - pathname2url, - HTTPBasicAuthHandler, HTTPPasswordMgr, - HTTPHandler, HTTPRedirectHandler, - build_opener) - if ssl: - from urllib.request import HTTPSHandler - from urllib.error import HTTPError, URLError, ContentTooShortError - import http.client as httplib - import urllib.request as urllib2 - import xmlrpc.client as xmlrpclib - import queue - from html.parser import HTMLParser - import html.entities as htmlentitydefs - raw_input = input - from itertools import filterfalse - filter = filter - - -try: - from ssl import match_hostname, CertificateError -except ImportError: # pragma: no cover - class CertificateError(ValueError): - pass - - - def _dnsname_match(dn, hostname, max_wildcards=1): - """Matching according to RFC 6125, section 6.4.3 - - http://tools.ietf.org/html/rfc6125#section-6.4.3 - """ - pats = [] - if not dn: - return False - - parts = dn.split('.') - leftmost, remainder = parts[0], parts[1:] - - wildcards = leftmost.count('*') - if wildcards > max_wildcards: - # Issue #17980: avoid denials of service by refusing more - # than one wildcard per fragment. A survey of established - # policy among SSL implementations showed it to be a - # reasonable choice. - raise CertificateError( - "too many wildcards in certificate DNS name: " + repr(dn)) - - # speed up common case w/o wildcards - if not wildcards: - return dn.lower() == hostname.lower() - - # RFC 6125, section 6.4.3, subitem 1. - # The client SHOULD NOT attempt to match a presented identifier in which - # the wildcard character comprises a label other than the left-most label. - if leftmost == '*': - # When '*' is a fragment by itself, it matches a non-empty dotless - # fragment. - pats.append('[^.]+') - elif leftmost.startswith('xn--') or hostname.startswith('xn--'): - # RFC 6125, section 6.4.3, subitem 3. - # The client SHOULD NOT attempt to match a presented identifier - # where the wildcard character is embedded within an A-label or - # U-label of an internationalized domain name. - pats.append(re.escape(leftmost)) - else: - # Otherwise, '*' matches any dotless string, e.g. www* - pats.append(re.escape(leftmost).replace(r'\*', '[^.]*')) - - # add the remaining fragments, ignore any wildcards - for frag in remainder: - pats.append(re.escape(frag)) - - pat = re.compile(r'\A' + r'\.'.join(pats) + r'\Z', re.IGNORECASE) - return pat.match(hostname) - - - def match_hostname(cert, hostname): - """Verify that *cert* (in decoded format as returned by - SSLSocket.getpeercert()) matches the *hostname*. RFC 2818 and RFC 6125 - rules are followed, but IP addresses are not accepted for *hostname*. - - CertificateError is raised on failure. On success, the function - returns nothing. - """ - if not cert: - raise ValueError("empty or no certificate, match_hostname needs a " - "SSL socket or SSL context with either " - "CERT_OPTIONAL or CERT_REQUIRED") - dnsnames = [] - san = cert.get('subjectAltName', ()) - for key, value in san: - if key == 'DNS': - if _dnsname_match(value, hostname): - return - dnsnames.append(value) - if not dnsnames: - # The subject is only checked when there is no dNSName entry - # in subjectAltName - for sub in cert.get('subject', ()): - for key, value in sub: - # XXX according to RFC 2818, the most specific Common Name - # must be used. - if key == 'commonName': - if _dnsname_match(value, hostname): - return - dnsnames.append(value) - if len(dnsnames) > 1: - raise CertificateError("hostname %r " - "doesn't match either of %s" - % (hostname, ', '.join(map(repr, dnsnames)))) - elif len(dnsnames) == 1: - raise CertificateError("hostname %r " - "doesn't match %r" - % (hostname, dnsnames[0])) - else: - raise CertificateError("no appropriate commonName or " - "subjectAltName fields were found") - - -try: - from types import SimpleNamespace as Container -except ImportError: # pragma: no cover - class Container(object): - """ - A generic container for when multiple values need to be returned - """ - def __init__(self, **kwargs): - self.__dict__.update(kwargs) - - -try: - from shutil import which -except ImportError: # pragma: no cover - # Implementation from Python 3.3 - def which(cmd, mode=os.F_OK | os.X_OK, path=None): - """Given a command, mode, and a PATH string, return the path which - conforms to the given mode on the PATH, or None if there is no such - file. - - `mode` defaults to os.F_OK | os.X_OK. `path` defaults to the result - of os.environ.get("PATH"), or can be overridden with a custom search - path. - - """ - # Check that a given file can be accessed with the correct mode. - # Additionally check that `file` is not a directory, as on Windows - # directories pass the os.access check. - def _access_check(fn, mode): - return (os.path.exists(fn) and os.access(fn, mode) - and not os.path.isdir(fn)) - - # If we're given a path with a directory part, look it up directly rather - # than referring to PATH directories. This includes checking relative to the - # current directory, e.g. ./script - if os.path.dirname(cmd): - if _access_check(cmd, mode): - return cmd - return None - - if path is None: - path = os.environ.get("PATH", os.defpath) - if not path: - return None - path = path.split(os.pathsep) - - if sys.platform == "win32": - # The current directory takes precedence on Windows. - if not os.curdir in path: - path.insert(0, os.curdir) - - # PATHEXT is necessary to check on Windows. - pathext = os.environ.get("PATHEXT", "").split(os.pathsep) - # See if the given file matches any of the expected path extensions. - # This will allow us to short circuit when given "python.exe". - # If it does match, only test that one, otherwise we have to try - # others. - if any(cmd.lower().endswith(ext.lower()) for ext in pathext): - files = [cmd] - else: - files = [cmd + ext for ext in pathext] - else: - # On other platforms you don't have things like PATHEXT to tell you - # what file suffixes are executable, so just pass on cmd as-is. - files = [cmd] - - seen = set() - for dir in path: - normdir = os.path.normcase(dir) - if not normdir in seen: - seen.add(normdir) - for thefile in files: - name = os.path.join(dir, thefile) - if _access_check(name, mode): - return name - return None - - -# ZipFile is a context manager in 2.7, but not in 2.6 - -from zipfile import ZipFile as BaseZipFile - -if hasattr(BaseZipFile, '__enter__'): # pragma: no cover - ZipFile = BaseZipFile -else: # pragma: no cover - from zipfile import ZipExtFile as BaseZipExtFile - - class ZipExtFile(BaseZipExtFile): - def __init__(self, base): - self.__dict__.update(base.__dict__) - - def __enter__(self): - return self - - def __exit__(self, *exc_info): - self.close() - # return None, so if an exception occurred, it will propagate - - class ZipFile(BaseZipFile): - def __enter__(self): - return self - - def __exit__(self, *exc_info): - self.close() - # return None, so if an exception occurred, it will propagate - - def open(self, *args, **kwargs): - base = BaseZipFile.open(self, *args, **kwargs) - return ZipExtFile(base) - -try: - from platform import python_implementation -except ImportError: # pragma: no cover - def python_implementation(): - """Return a string identifying the Python implementation.""" - if 'PyPy' in sys.version: - return 'PyPy' - if os.name == 'java': - return 'Jython' - if sys.version.startswith('IronPython'): - return 'IronPython' - return 'CPython' - -import shutil -import sysconfig - -try: - callable = callable -except NameError: # pragma: no cover - from collections.abc import Callable - - def callable(obj): - return isinstance(obj, Callable) - - -try: - fsencode = os.fsencode - fsdecode = os.fsdecode -except AttributeError: # pragma: no cover - # Issue #99: on some systems (e.g. containerised), - # sys.getfilesystemencoding() returns None, and we need a real value, - # so fall back to utf-8. From the CPython 2.7 docs relating to Unix and - # sys.getfilesystemencoding(): the return value is "the user’s preference - # according to the result of nl_langinfo(CODESET), or None if the - # nl_langinfo(CODESET) failed." - _fsencoding = sys.getfilesystemencoding() or 'utf-8' - if _fsencoding == 'mbcs': - _fserrors = 'strict' - else: - _fserrors = 'surrogateescape' - - def fsencode(filename): - if isinstance(filename, bytes): - return filename - elif isinstance(filename, text_type): - return filename.encode(_fsencoding, _fserrors) - else: - raise TypeError("expect bytes or str, not %s" % - type(filename).__name__) - - def fsdecode(filename): - if isinstance(filename, text_type): - return filename - elif isinstance(filename, bytes): - return filename.decode(_fsencoding, _fserrors) - else: - raise TypeError("expect bytes or str, not %s" % - type(filename).__name__) - -try: - from tokenize import detect_encoding -except ImportError: # pragma: no cover - from codecs import BOM_UTF8, lookup - import re - - cookie_re = re.compile(r"coding[:=]\s*([-\w.]+)") - - def _get_normal_name(orig_enc): - """Imitates get_normal_name in tokenizer.c.""" - # Only care about the first 12 characters. - enc = orig_enc[:12].lower().replace("_", "-") - if enc == "utf-8" or enc.startswith("utf-8-"): - return "utf-8" - if enc in ("latin-1", "iso-8859-1", "iso-latin-1") or \ - enc.startswith(("latin-1-", "iso-8859-1-", "iso-latin-1-")): - return "iso-8859-1" - return orig_enc - - def detect_encoding(readline): - """ - The detect_encoding() function is used to detect the encoding that should - be used to decode a Python source file. It requires one argument, readline, - in the same way as the tokenize() generator. - - It will call readline a maximum of twice, and return the encoding used - (as a string) and a list of any lines (left as bytes) it has read in. - - It detects the encoding from the presence of a utf-8 bom or an encoding - cookie as specified in pep-0263. If both a bom and a cookie are present, - but disagree, a SyntaxError will be raised. If the encoding cookie is an - invalid charset, raise a SyntaxError. Note that if a utf-8 bom is found, - 'utf-8-sig' is returned. - - If no encoding is specified, then the default of 'utf-8' will be returned. - """ - try: - filename = readline.__self__.name - except AttributeError: - filename = None - bom_found = False - encoding = None - default = 'utf-8' - def read_or_stop(): - try: - return readline() - except StopIteration: - return b'' - - def find_cookie(line): - try: - # Decode as UTF-8. Either the line is an encoding declaration, - # in which case it should be pure ASCII, or it must be UTF-8 - # per default encoding. - line_string = line.decode('utf-8') - except UnicodeDecodeError: - msg = "invalid or missing encoding declaration" - if filename is not None: - msg = '{} for {!r}'.format(msg, filename) - raise SyntaxError(msg) - - matches = cookie_re.findall(line_string) - if not matches: - return None - encoding = _get_normal_name(matches[0]) - try: - codec = lookup(encoding) - except LookupError: - # This behaviour mimics the Python interpreter - if filename is None: - msg = "unknown encoding: " + encoding - else: - msg = "unknown encoding for {!r}: {}".format(filename, - encoding) - raise SyntaxError(msg) - - if bom_found: - if codec.name != 'utf-8': - # This behaviour mimics the Python interpreter - if filename is None: - msg = 'encoding problem: utf-8' - else: - msg = 'encoding problem for {!r}: utf-8'.format(filename) - raise SyntaxError(msg) - encoding += '-sig' - return encoding - - first = read_or_stop() - if first.startswith(BOM_UTF8): - bom_found = True - first = first[3:] - default = 'utf-8-sig' - if not first: - return default, [] - - encoding = find_cookie(first) - if encoding: - return encoding, [first] - - second = read_or_stop() - if not second: - return default, [first] - - encoding = find_cookie(second) - if encoding: - return encoding, [first, second] - - return default, [first, second] - -# For converting & <-> & etc. -try: - from html import escape -except ImportError: - from cgi import escape -if sys.version_info[:2] < (3, 4): - unescape = HTMLParser().unescape -else: - from html import unescape - -try: - from collections import ChainMap -except ImportError: # pragma: no cover - from collections import MutableMapping - - try: - from reprlib import recursive_repr as _recursive_repr - except ImportError: - def _recursive_repr(fillvalue='...'): - ''' - Decorator to make a repr function return fillvalue for a recursive - call - ''' - - def decorating_function(user_function): - repr_running = set() - - def wrapper(self): - key = id(self), get_ident() - if key in repr_running: - return fillvalue - repr_running.add(key) - try: - result = user_function(self) - finally: - repr_running.discard(key) - return result - - # Can't use functools.wraps() here because of bootstrap issues - wrapper.__module__ = getattr(user_function, '__module__') - wrapper.__doc__ = getattr(user_function, '__doc__') - wrapper.__name__ = getattr(user_function, '__name__') - wrapper.__annotations__ = getattr(user_function, '__annotations__', {}) - return wrapper - - return decorating_function - - class ChainMap(MutableMapping): - ''' A ChainMap groups multiple dicts (or other mappings) together - to create a single, updateable view. - - The underlying mappings are stored in a list. That list is public and can - accessed or updated using the *maps* attribute. There is no other state. - - Lookups search the underlying mappings successively until a key is found. - In contrast, writes, updates, and deletions only operate on the first - mapping. - - ''' - - def __init__(self, *maps): - '''Initialize a ChainMap by setting *maps* to the given mappings. - If no mappings are provided, a single empty dictionary is used. - - ''' - self.maps = list(maps) or [{}] # always at least one map - - def __missing__(self, key): - raise KeyError(key) - - def __getitem__(self, key): - for mapping in self.maps: - try: - return mapping[key] # can't use 'key in mapping' with defaultdict - except KeyError: - pass - return self.__missing__(key) # support subclasses that define __missing__ - - def get(self, key, default=None): - return self[key] if key in self else default - - def __len__(self): - return len(set().union(*self.maps)) # reuses stored hash values if possible - - def __iter__(self): - return iter(set().union(*self.maps)) - - def __contains__(self, key): - return any(key in m for m in self.maps) - - def __bool__(self): - return any(self.maps) - - @_recursive_repr() - def __repr__(self): - return '{0.__class__.__name__}({1})'.format( - self, ', '.join(map(repr, self.maps))) - - @classmethod - def fromkeys(cls, iterable, *args): - 'Create a ChainMap with a single dict created from the iterable.' - return cls(dict.fromkeys(iterable, *args)) - - def copy(self): - 'New ChainMap or subclass with a new copy of maps[0] and refs to maps[1:]' - return self.__class__(self.maps[0].copy(), *self.maps[1:]) - - __copy__ = copy - - def new_child(self): # like Django's Context.push() - 'New ChainMap with a new dict followed by all previous maps.' - return self.__class__({}, *self.maps) - - @property - def parents(self): # like Django's Context.pop() - 'New ChainMap from maps[1:].' - return self.__class__(*self.maps[1:]) - - def __setitem__(self, key, value): - self.maps[0][key] = value - - def __delitem__(self, key): - try: - del self.maps[0][key] - except KeyError: - raise KeyError('Key not found in the first mapping: {!r}'.format(key)) - - def popitem(self): - 'Remove and return an item pair from maps[0]. Raise KeyError is maps[0] is empty.' - try: - return self.maps[0].popitem() - except KeyError: - raise KeyError('No keys found in the first mapping.') - - def pop(self, key, *args): - 'Remove *key* from maps[0] and return its value. Raise KeyError if *key* not in maps[0].' - try: - return self.maps[0].pop(key, *args) - except KeyError: - raise KeyError('Key not found in the first mapping: {!r}'.format(key)) - - def clear(self): - 'Clear maps[0], leaving maps[1:] intact.' - self.maps[0].clear() - -try: - from importlib.util import cache_from_source # Python >= 3.4 -except ImportError: # pragma: no cover - def cache_from_source(path, debug_override=None): - assert path.endswith('.py') - if debug_override is None: - debug_override = __debug__ - if debug_override: - suffix = 'c' - else: - suffix = 'o' - return path + suffix - -try: - from collections import OrderedDict -except ImportError: # pragma: no cover -## {{{ http://code.activestate.com/recipes/576693/ (r9) -# Backport of OrderedDict() class that runs on Python 2.4, 2.5, 2.6, 2.7 and pypy. -# Passes Python2.7's test suite and incorporates all the latest updates. - try: - from thread import get_ident as _get_ident - except ImportError: - from dummy_thread import get_ident as _get_ident - - try: - from _abcoll import KeysView, ValuesView, ItemsView - except ImportError: - pass - - - class OrderedDict(dict): - 'Dictionary that remembers insertion order' - # An inherited dict maps keys to values. - # The inherited dict provides __getitem__, __len__, __contains__, and get. - # The remaining methods are order-aware. - # Big-O running times for all methods are the same as for regular dictionaries. - - # The internal self.__map dictionary maps keys to links in a doubly linked list. - # The circular doubly linked list starts and ends with a sentinel element. - # The sentinel element never gets deleted (this simplifies the algorithm). - # Each link is stored as a list of length three: [PREV, NEXT, KEY]. - - def __init__(self, *args, **kwds): - '''Initialize an ordered dictionary. Signature is the same as for - regular dictionaries, but keyword arguments are not recommended - because their insertion order is arbitrary. - - ''' - if len(args) > 1: - raise TypeError('expected at most 1 arguments, got %d' % len(args)) - try: - self.__root - except AttributeError: - self.__root = root = [] # sentinel node - root[:] = [root, root, None] - self.__map = {} - self.__update(*args, **kwds) - - def __setitem__(self, key, value, dict_setitem=dict.__setitem__): - 'od.__setitem__(i, y) <==> od[i]=y' - # Setting a new item creates a new link which goes at the end of the linked - # list, and the inherited dictionary is updated with the new key/value pair. - if key not in self: - root = self.__root - last = root[0] - last[1] = root[0] = self.__map[key] = [last, root, key] - dict_setitem(self, key, value) - - def __delitem__(self, key, dict_delitem=dict.__delitem__): - 'od.__delitem__(y) <==> del od[y]' - # Deleting an existing item uses self.__map to find the link which is - # then removed by updating the links in the predecessor and successor nodes. - dict_delitem(self, key) - link_prev, link_next, key = self.__map.pop(key) - link_prev[1] = link_next - link_next[0] = link_prev - - def __iter__(self): - 'od.__iter__() <==> iter(od)' - root = self.__root - curr = root[1] - while curr is not root: - yield curr[2] - curr = curr[1] - - def __reversed__(self): - 'od.__reversed__() <==> reversed(od)' - root = self.__root - curr = root[0] - while curr is not root: - yield curr[2] - curr = curr[0] - - def clear(self): - 'od.clear() -> None. Remove all items from od.' - try: - for node in self.__map.itervalues(): - del node[:] - root = self.__root - root[:] = [root, root, None] - self.__map.clear() - except AttributeError: - pass - dict.clear(self) - - def popitem(self, last=True): - '''od.popitem() -> (k, v), return and remove a (key, value) pair. - Pairs are returned in LIFO order if last is true or FIFO order if false. - - ''' - if not self: - raise KeyError('dictionary is empty') - root = self.__root - if last: - link = root[0] - link_prev = link[0] - link_prev[1] = root - root[0] = link_prev - else: - link = root[1] - link_next = link[1] - root[1] = link_next - link_next[0] = root - key = link[2] - del self.__map[key] - value = dict.pop(self, key) - return key, value - - # -- the following methods do not depend on the internal structure -- - - def keys(self): - 'od.keys() -> list of keys in od' - return list(self) - - def values(self): - 'od.values() -> list of values in od' - return [self[key] for key in self] - - def items(self): - 'od.items() -> list of (key, value) pairs in od' - return [(key, self[key]) for key in self] - - def iterkeys(self): - 'od.iterkeys() -> an iterator over the keys in od' - return iter(self) - - def itervalues(self): - 'od.itervalues -> an iterator over the values in od' - for k in self: - yield self[k] - - def iteritems(self): - 'od.iteritems -> an iterator over the (key, value) items in od' - for k in self: - yield (k, self[k]) - - def update(*args, **kwds): - '''od.update(E, **F) -> None. Update od from dict/iterable E and F. - - If E is a dict instance, does: for k in E: od[k] = E[k] - If E has a .keys() method, does: for k in E.keys(): od[k] = E[k] - Or if E is an iterable of items, does: for k, v in E: od[k] = v - In either case, this is followed by: for k, v in F.items(): od[k] = v - - ''' - if len(args) > 2: - raise TypeError('update() takes at most 2 positional ' - 'arguments (%d given)' % (len(args),)) - elif not args: - raise TypeError('update() takes at least 1 argument (0 given)') - self = args[0] - # Make progressively weaker assumptions about "other" - other = () - if len(args) == 2: - other = args[1] - if isinstance(other, dict): - for key in other: - self[key] = other[key] - elif hasattr(other, 'keys'): - for key in other.keys(): - self[key] = other[key] - else: - for key, value in other: - self[key] = value - for key, value in kwds.items(): - self[key] = value - - __update = update # let subclasses override update without breaking __init__ - - __marker = object() - - def pop(self, key, default=__marker): - '''od.pop(k[,d]) -> v, remove specified key and return the corresponding value. - If key is not found, d is returned if given, otherwise KeyError is raised. - - ''' - if key in self: - result = self[key] - del self[key] - return result - if default is self.__marker: - raise KeyError(key) - return default - - def setdefault(self, key, default=None): - 'od.setdefault(k[,d]) -> od.get(k,d), also set od[k]=d if k not in od' - if key in self: - return self[key] - self[key] = default - return default - - def __repr__(self, _repr_running=None): - 'od.__repr__() <==> repr(od)' - if not _repr_running: _repr_running = {} - call_key = id(self), _get_ident() - if call_key in _repr_running: - return '...' - _repr_running[call_key] = 1 - try: - if not self: - return '%s()' % (self.__class__.__name__,) - return '%s(%r)' % (self.__class__.__name__, self.items()) - finally: - del _repr_running[call_key] - - def __reduce__(self): - 'Return state information for pickling' - items = [[k, self[k]] for k in self] - inst_dict = vars(self).copy() - for k in vars(OrderedDict()): - inst_dict.pop(k, None) - if inst_dict: - return (self.__class__, (items,), inst_dict) - return self.__class__, (items,) - - def copy(self): - 'od.copy() -> a shallow copy of od' - return self.__class__(self) - - @classmethod - def fromkeys(cls, iterable, value=None): - '''OD.fromkeys(S[, v]) -> New ordered dictionary with keys from S - and values equal to v (which defaults to None). - - ''' - d = cls() - for key in iterable: - d[key] = value - return d - - def __eq__(self, other): - '''od.__eq__(y) <==> od==y. Comparison to another OD is order-sensitive - while comparison to a regular mapping is order-insensitive. - - ''' - if isinstance(other, OrderedDict): - return len(self)==len(other) and self.items() == other.items() - return dict.__eq__(self, other) - - def __ne__(self, other): - return not self == other - - # -- the following methods are only used in Python 2.7 -- - - def viewkeys(self): - "od.viewkeys() -> a set-like object providing a view on od's keys" - return KeysView(self) - - def viewvalues(self): - "od.viewvalues() -> an object providing a view on od's values" - return ValuesView(self) - - def viewitems(self): - "od.viewitems() -> a set-like object providing a view on od's items" - return ItemsView(self) - -try: - from logging.config import BaseConfigurator, valid_ident -except ImportError: # pragma: no cover - IDENTIFIER = re.compile('^[a-z_][a-z0-9_]*$', re.I) - - - def valid_ident(s): - m = IDENTIFIER.match(s) - if not m: - raise ValueError('Not a valid Python identifier: %r' % s) - return True - - - # The ConvertingXXX classes are wrappers around standard Python containers, - # and they serve to convert any suitable values in the container. The - # conversion converts base dicts, lists and tuples to their wrapped - # equivalents, whereas strings which match a conversion format are converted - # appropriately. - # - # Each wrapper should have a configurator attribute holding the actual - # configurator to use for conversion. - - class ConvertingDict(dict): - """A converting dictionary wrapper.""" - - def __getitem__(self, key): - value = dict.__getitem__(self, key) - result = self.configurator.convert(value) - #If the converted value is different, save for next time - if value is not result: - self[key] = result - if type(result) in (ConvertingDict, ConvertingList, - ConvertingTuple): - result.parent = self - result.key = key - return result - - def get(self, key, default=None): - value = dict.get(self, key, default) - result = self.configurator.convert(value) - #If the converted value is different, save for next time - if value is not result: - self[key] = result - if type(result) in (ConvertingDict, ConvertingList, - ConvertingTuple): - result.parent = self - result.key = key - return result - - def pop(self, key, default=None): - value = dict.pop(self, key, default) - result = self.configurator.convert(value) - if value is not result: - if type(result) in (ConvertingDict, ConvertingList, - ConvertingTuple): - result.parent = self - result.key = key - return result - - class ConvertingList(list): - """A converting list wrapper.""" - def __getitem__(self, key): - value = list.__getitem__(self, key) - result = self.configurator.convert(value) - #If the converted value is different, save for next time - if value is not result: - self[key] = result - if type(result) in (ConvertingDict, ConvertingList, - ConvertingTuple): - result.parent = self - result.key = key - return result - - def pop(self, idx=-1): - value = list.pop(self, idx) - result = self.configurator.convert(value) - if value is not result: - if type(result) in (ConvertingDict, ConvertingList, - ConvertingTuple): - result.parent = self - return result - - class ConvertingTuple(tuple): - """A converting tuple wrapper.""" - def __getitem__(self, key): - value = tuple.__getitem__(self, key) - result = self.configurator.convert(value) - if value is not result: - if type(result) in (ConvertingDict, ConvertingList, - ConvertingTuple): - result.parent = self - result.key = key - return result - - class BaseConfigurator(object): - """ - The configurator base class which defines some useful defaults. - """ - - CONVERT_PATTERN = re.compile(r'^(?P[a-z]+)://(?P.*)$') - - WORD_PATTERN = re.compile(r'^\s*(\w+)\s*') - DOT_PATTERN = re.compile(r'^\.\s*(\w+)\s*') - INDEX_PATTERN = re.compile(r'^\[\s*(\w+)\s*\]\s*') - DIGIT_PATTERN = re.compile(r'^\d+$') - - value_converters = { - 'ext' : 'ext_convert', - 'cfg' : 'cfg_convert', - } - - # We might want to use a different one, e.g. importlib - importer = staticmethod(__import__) - - def __init__(self, config): - self.config = ConvertingDict(config) - self.config.configurator = self - - def resolve(self, s): - """ - Resolve strings to objects using standard import and attribute - syntax. - """ - name = s.split('.') - used = name.pop(0) - try: - found = self.importer(used) - for frag in name: - used += '.' + frag - try: - found = getattr(found, frag) - except AttributeError: - self.importer(used) - found = getattr(found, frag) - return found - except ImportError: - e, tb = sys.exc_info()[1:] - v = ValueError('Cannot resolve %r: %s' % (s, e)) - v.__cause__, v.__traceback__ = e, tb - raise v - - def ext_convert(self, value): - """Default converter for the ext:// protocol.""" - return self.resolve(value) - - def cfg_convert(self, value): - """Default converter for the cfg:// protocol.""" - rest = value - m = self.WORD_PATTERN.match(rest) - if m is None: - raise ValueError("Unable to convert %r" % value) - else: - rest = rest[m.end():] - d = self.config[m.groups()[0]] - #print d, rest - while rest: - m = self.DOT_PATTERN.match(rest) - if m: - d = d[m.groups()[0]] - else: - m = self.INDEX_PATTERN.match(rest) - if m: - idx = m.groups()[0] - if not self.DIGIT_PATTERN.match(idx): - d = d[idx] - else: - try: - n = int(idx) # try as number first (most likely) - d = d[n] - except TypeError: - d = d[idx] - if m: - rest = rest[m.end():] - else: - raise ValueError('Unable to convert ' - '%r at %r' % (value, rest)) - #rest should be empty - return d - - def convert(self, value): - """ - Convert values to an appropriate type. dicts, lists and tuples are - replaced by their converting alternatives. Strings are checked to - see if they have a conversion format and are converted if they do. - """ - if not isinstance(value, ConvertingDict) and isinstance(value, dict): - value = ConvertingDict(value) - value.configurator = self - elif not isinstance(value, ConvertingList) and isinstance(value, list): - value = ConvertingList(value) - value.configurator = self - elif not isinstance(value, ConvertingTuple) and\ - isinstance(value, tuple): - value = ConvertingTuple(value) - value.configurator = self - elif isinstance(value, string_types): - m = self.CONVERT_PATTERN.match(value) - if m: - d = m.groupdict() - prefix = d['prefix'] - converter = self.value_converters.get(prefix, None) - if converter: - suffix = d['suffix'] - converter = getattr(self, converter) - value = converter(suffix) - return value - - def configure_custom(self, config): - """Configure an object with a user-supplied factory.""" - c = config.pop('()') - if not callable(c): - c = self.resolve(c) - props = config.pop('.', None) - # Check for valid identifiers - kwargs = dict([(k, config[k]) for k in config if valid_ident(k)]) - result = c(**kwargs) - if props: - for name, value in props.items(): - setattr(result, name, value) - return result - - def as_tuple(self, value): - """Utility function which converts lists to tuples.""" - if isinstance(value, list): - value = tuple(value) - return value diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/distlib/database.py b/venv/lib/python3.11/site-packages/pip/_vendor/distlib/database.py deleted file mode 100644 index 5db5d7f..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/distlib/database.py +++ /dev/null @@ -1,1350 +0,0 @@ -# -*- coding: utf-8 -*- -# -# Copyright (C) 2012-2017 The Python Software Foundation. -# See LICENSE.txt and CONTRIBUTORS.txt. -# -"""PEP 376 implementation.""" - -from __future__ import unicode_literals - -import base64 -import codecs -import contextlib -import hashlib -import logging -import os -import posixpath -import sys -import zipimport - -from . import DistlibException, resources -from .compat import StringIO -from .version import get_scheme, UnsupportedVersionError -from .metadata import (Metadata, METADATA_FILENAME, WHEEL_METADATA_FILENAME, - LEGACY_METADATA_FILENAME) -from .util import (parse_requirement, cached_property, parse_name_and_version, - read_exports, write_exports, CSVReader, CSVWriter) - - -__all__ = ['Distribution', 'BaseInstalledDistribution', - 'InstalledDistribution', 'EggInfoDistribution', - 'DistributionPath'] - - -logger = logging.getLogger(__name__) - -EXPORTS_FILENAME = 'pydist-exports.json' -COMMANDS_FILENAME = 'pydist-commands.json' - -DIST_FILES = ('INSTALLER', METADATA_FILENAME, 'RECORD', 'REQUESTED', - 'RESOURCES', EXPORTS_FILENAME, 'SHARED') - -DISTINFO_EXT = '.dist-info' - - -class _Cache(object): - """ - A simple cache mapping names and .dist-info paths to distributions - """ - def __init__(self): - """ - Initialise an instance. There is normally one for each DistributionPath. - """ - self.name = {} - self.path = {} - self.generated = False - - def clear(self): - """ - Clear the cache, setting it to its initial state. - """ - self.name.clear() - self.path.clear() - self.generated = False - - def add(self, dist): - """ - Add a distribution to the cache. - :param dist: The distribution to add. - """ - if dist.path not in self.path: - self.path[dist.path] = dist - self.name.setdefault(dist.key, []).append(dist) - - -class DistributionPath(object): - """ - Represents a set of distributions installed on a path (typically sys.path). - """ - def __init__(self, path=None, include_egg=False): - """ - Create an instance from a path, optionally including legacy (distutils/ - setuptools/distribute) distributions. - :param path: The path to use, as a list of directories. If not specified, - sys.path is used. - :param include_egg: If True, this instance will look for and return legacy - distributions as well as those based on PEP 376. - """ - if path is None: - path = sys.path - self.path = path - self._include_dist = True - self._include_egg = include_egg - - self._cache = _Cache() - self._cache_egg = _Cache() - self._cache_enabled = True - self._scheme = get_scheme('default') - - def _get_cache_enabled(self): - return self._cache_enabled - - def _set_cache_enabled(self, value): - self._cache_enabled = value - - cache_enabled = property(_get_cache_enabled, _set_cache_enabled) - - def clear_cache(self): - """ - Clears the internal cache. - """ - self._cache.clear() - self._cache_egg.clear() - - - def _yield_distributions(self): - """ - Yield .dist-info and/or .egg(-info) distributions. - """ - # We need to check if we've seen some resources already, because on - # some Linux systems (e.g. some Debian/Ubuntu variants) there are - # symlinks which alias other files in the environment. - seen = set() - for path in self.path: - finder = resources.finder_for_path(path) - if finder is None: - continue - r = finder.find('') - if not r or not r.is_container: - continue - rset = sorted(r.resources) - for entry in rset: - r = finder.find(entry) - if not r or r.path in seen: - continue - try: - if self._include_dist and entry.endswith(DISTINFO_EXT): - possible_filenames = [METADATA_FILENAME, - WHEEL_METADATA_FILENAME, - LEGACY_METADATA_FILENAME] - for metadata_filename in possible_filenames: - metadata_path = posixpath.join(entry, metadata_filename) - pydist = finder.find(metadata_path) - if pydist: - break - else: - continue - - with contextlib.closing(pydist.as_stream()) as stream: - metadata = Metadata(fileobj=stream, scheme='legacy') - logger.debug('Found %s', r.path) - seen.add(r.path) - yield new_dist_class(r.path, metadata=metadata, - env=self) - elif self._include_egg and entry.endswith(('.egg-info', - '.egg')): - logger.debug('Found %s', r.path) - seen.add(r.path) - yield old_dist_class(r.path, self) - except Exception as e: - msg = 'Unable to read distribution at %s, perhaps due to bad metadata: %s' - logger.warning(msg, r.path, e) - import warnings - warnings.warn(msg % (r.path, e), stacklevel=2) - - def _generate_cache(self): - """ - Scan the path for distributions and populate the cache with - those that are found. - """ - gen_dist = not self._cache.generated - gen_egg = self._include_egg and not self._cache_egg.generated - if gen_dist or gen_egg: - for dist in self._yield_distributions(): - if isinstance(dist, InstalledDistribution): - self._cache.add(dist) - else: - self._cache_egg.add(dist) - - if gen_dist: - self._cache.generated = True - if gen_egg: - self._cache_egg.generated = True - - @classmethod - def distinfo_dirname(cls, name, version): - """ - The *name* and *version* parameters are converted into their - filename-escaped form, i.e. any ``'-'`` characters are replaced - with ``'_'`` other than the one in ``'dist-info'`` and the one - separating the name from the version number. - - :parameter name: is converted to a standard distribution name by replacing - any runs of non- alphanumeric characters with a single - ``'-'``. - :type name: string - :parameter version: is converted to a standard version string. Spaces - become dots, and all other non-alphanumeric characters - (except dots) become dashes, with runs of multiple - dashes condensed to a single dash. - :type version: string - :returns: directory name - :rtype: string""" - name = name.replace('-', '_') - return '-'.join([name, version]) + DISTINFO_EXT - - def get_distributions(self): - """ - Provides an iterator that looks for distributions and returns - :class:`InstalledDistribution` or - :class:`EggInfoDistribution` instances for each one of them. - - :rtype: iterator of :class:`InstalledDistribution` and - :class:`EggInfoDistribution` instances - """ - if not self._cache_enabled: - for dist in self._yield_distributions(): - yield dist - else: - self._generate_cache() - - for dist in self._cache.path.values(): - yield dist - - if self._include_egg: - for dist in self._cache_egg.path.values(): - yield dist - - def get_distribution(self, name): - """ - Looks for a named distribution on the path. - - This function only returns the first result found, as no more than one - value is expected. If nothing is found, ``None`` is returned. - - :rtype: :class:`InstalledDistribution`, :class:`EggInfoDistribution` - or ``None`` - """ - result = None - name = name.lower() - if not self._cache_enabled: - for dist in self._yield_distributions(): - if dist.key == name: - result = dist - break - else: - self._generate_cache() - - if name in self._cache.name: - result = self._cache.name[name][0] - elif self._include_egg and name in self._cache_egg.name: - result = self._cache_egg.name[name][0] - return result - - def provides_distribution(self, name, version=None): - """ - Iterates over all distributions to find which distributions provide *name*. - If a *version* is provided, it will be used to filter the results. - - This function only returns the first result found, since no more than - one values are expected. If the directory is not found, returns ``None``. - - :parameter version: a version specifier that indicates the version - required, conforming to the format in ``PEP-345`` - - :type name: string - :type version: string - """ - matcher = None - if version is not None: - try: - matcher = self._scheme.matcher('%s (%s)' % (name, version)) - except ValueError: - raise DistlibException('invalid name or version: %r, %r' % - (name, version)) - - for dist in self.get_distributions(): - # We hit a problem on Travis where enum34 was installed and doesn't - # have a provides attribute ... - if not hasattr(dist, 'provides'): - logger.debug('No "provides": %s', dist) - else: - provided = dist.provides - - for p in provided: - p_name, p_ver = parse_name_and_version(p) - if matcher is None: - if p_name == name: - yield dist - break - else: - if p_name == name and matcher.match(p_ver): - yield dist - break - - def get_file_path(self, name, relative_path): - """ - Return the path to a resource file. - """ - dist = self.get_distribution(name) - if dist is None: - raise LookupError('no distribution named %r found' % name) - return dist.get_resource_path(relative_path) - - def get_exported_entries(self, category, name=None): - """ - Return all of the exported entries in a particular category. - - :param category: The category to search for entries. - :param name: If specified, only entries with that name are returned. - """ - for dist in self.get_distributions(): - r = dist.exports - if category in r: - d = r[category] - if name is not None: - if name in d: - yield d[name] - else: - for v in d.values(): - yield v - - -class Distribution(object): - """ - A base class for distributions, whether installed or from indexes. - Either way, it must have some metadata, so that's all that's needed - for construction. - """ - - build_time_dependency = False - """ - Set to True if it's known to be only a build-time dependency (i.e. - not needed after installation). - """ - - requested = False - """A boolean that indicates whether the ``REQUESTED`` metadata file is - present (in other words, whether the package was installed by user - request or it was installed as a dependency).""" - - def __init__(self, metadata): - """ - Initialise an instance. - :param metadata: The instance of :class:`Metadata` describing this - distribution. - """ - self.metadata = metadata - self.name = metadata.name - self.key = self.name.lower() # for case-insensitive comparisons - self.version = metadata.version - self.locator = None - self.digest = None - self.extras = None # additional features requested - self.context = None # environment marker overrides - self.download_urls = set() - self.digests = {} - - @property - def source_url(self): - """ - The source archive download URL for this distribution. - """ - return self.metadata.source_url - - download_url = source_url # Backward compatibility - - @property - def name_and_version(self): - """ - A utility property which displays the name and version in parentheses. - """ - return '%s (%s)' % (self.name, self.version) - - @property - def provides(self): - """ - A set of distribution names and versions provided by this distribution. - :return: A set of "name (version)" strings. - """ - plist = self.metadata.provides - s = '%s (%s)' % (self.name, self.version) - if s not in plist: - plist.append(s) - return plist - - def _get_requirements(self, req_attr): - md = self.metadata - reqts = getattr(md, req_attr) - logger.debug('%s: got requirements %r from metadata: %r', self.name, req_attr, - reqts) - return set(md.get_requirements(reqts, extras=self.extras, - env=self.context)) - - @property - def run_requires(self): - return self._get_requirements('run_requires') - - @property - def meta_requires(self): - return self._get_requirements('meta_requires') - - @property - def build_requires(self): - return self._get_requirements('build_requires') - - @property - def test_requires(self): - return self._get_requirements('test_requires') - - @property - def dev_requires(self): - return self._get_requirements('dev_requires') - - def matches_requirement(self, req): - """ - Say if this instance matches (fulfills) a requirement. - :param req: The requirement to match. - :rtype req: str - :return: True if it matches, else False. - """ - # Requirement may contain extras - parse to lose those - # from what's passed to the matcher - r = parse_requirement(req) - scheme = get_scheme(self.metadata.scheme) - try: - matcher = scheme.matcher(r.requirement) - except UnsupportedVersionError: - # XXX compat-mode if cannot read the version - logger.warning('could not read version %r - using name only', - req) - name = req.split()[0] - matcher = scheme.matcher(name) - - name = matcher.key # case-insensitive - - result = False - for p in self.provides: - p_name, p_ver = parse_name_and_version(p) - if p_name != name: - continue - try: - result = matcher.match(p_ver) - break - except UnsupportedVersionError: - pass - return result - - def __repr__(self): - """ - Return a textual representation of this instance, - """ - if self.source_url: - suffix = ' [%s]' % self.source_url - else: - suffix = '' - return '' % (self.name, self.version, suffix) - - def __eq__(self, other): - """ - See if this distribution is the same as another. - :param other: The distribution to compare with. To be equal to one - another. distributions must have the same type, name, - version and source_url. - :return: True if it is the same, else False. - """ - if type(other) is not type(self): - result = False - else: - result = (self.name == other.name and - self.version == other.version and - self.source_url == other.source_url) - return result - - def __hash__(self): - """ - Compute hash in a way which matches the equality test. - """ - return hash(self.name) + hash(self.version) + hash(self.source_url) - - -class BaseInstalledDistribution(Distribution): - """ - This is the base class for installed distributions (whether PEP 376 or - legacy). - """ - - hasher = None - - def __init__(self, metadata, path, env=None): - """ - Initialise an instance. - :param metadata: An instance of :class:`Metadata` which describes the - distribution. This will normally have been initialised - from a metadata file in the ``path``. - :param path: The path of the ``.dist-info`` or ``.egg-info`` - directory for the distribution. - :param env: This is normally the :class:`DistributionPath` - instance where this distribution was found. - """ - super(BaseInstalledDistribution, self).__init__(metadata) - self.path = path - self.dist_path = env - - def get_hash(self, data, hasher=None): - """ - Get the hash of some data, using a particular hash algorithm, if - specified. - - :param data: The data to be hashed. - :type data: bytes - :param hasher: The name of a hash implementation, supported by hashlib, - or ``None``. Examples of valid values are ``'sha1'``, - ``'sha224'``, ``'sha384'``, '``sha256'``, ``'md5'`` and - ``'sha512'``. If no hasher is specified, the ``hasher`` - attribute of the :class:`InstalledDistribution` instance - is used. If the hasher is determined to be ``None``, MD5 - is used as the hashing algorithm. - :returns: The hash of the data. If a hasher was explicitly specified, - the returned hash will be prefixed with the specified hasher - followed by '='. - :rtype: str - """ - if hasher is None: - hasher = self.hasher - if hasher is None: - hasher = hashlib.md5 - prefix = '' - else: - hasher = getattr(hashlib, hasher) - prefix = '%s=' % self.hasher - digest = hasher(data).digest() - digest = base64.urlsafe_b64encode(digest).rstrip(b'=').decode('ascii') - return '%s%s' % (prefix, digest) - - -class InstalledDistribution(BaseInstalledDistribution): - """ - Created with the *path* of the ``.dist-info`` directory provided to the - constructor. It reads the metadata contained in ``pydist.json`` when it is - instantiated., or uses a passed in Metadata instance (useful for when - dry-run mode is being used). - """ - - hasher = 'sha256' - - def __init__(self, path, metadata=None, env=None): - self.modules = [] - self.finder = finder = resources.finder_for_path(path) - if finder is None: - raise ValueError('finder unavailable for %s' % path) - if env and env._cache_enabled and path in env._cache.path: - metadata = env._cache.path[path].metadata - elif metadata is None: - r = finder.find(METADATA_FILENAME) - # Temporary - for Wheel 0.23 support - if r is None: - r = finder.find(WHEEL_METADATA_FILENAME) - # Temporary - for legacy support - if r is None: - r = finder.find(LEGACY_METADATA_FILENAME) - if r is None: - raise ValueError('no %s found in %s' % (METADATA_FILENAME, - path)) - with contextlib.closing(r.as_stream()) as stream: - metadata = Metadata(fileobj=stream, scheme='legacy') - - super(InstalledDistribution, self).__init__(metadata, path, env) - - if env and env._cache_enabled: - env._cache.add(self) - - r = finder.find('REQUESTED') - self.requested = r is not None - p = os.path.join(path, 'top_level.txt') - if os.path.exists(p): - with open(p, 'rb') as f: - data = f.read().decode('utf-8') - self.modules = data.splitlines() - - def __repr__(self): - return '' % ( - self.name, self.version, self.path) - - def __str__(self): - return "%s %s" % (self.name, self.version) - - def _get_records(self): - """ - Get the list of installed files for the distribution - :return: A list of tuples of path, hash and size. Note that hash and - size might be ``None`` for some entries. The path is exactly - as stored in the file (which is as in PEP 376). - """ - results = [] - r = self.get_distinfo_resource('RECORD') - with contextlib.closing(r.as_stream()) as stream: - with CSVReader(stream=stream) as record_reader: - # Base location is parent dir of .dist-info dir - #base_location = os.path.dirname(self.path) - #base_location = os.path.abspath(base_location) - for row in record_reader: - missing = [None for i in range(len(row), 3)] - path, checksum, size = row + missing - #if not os.path.isabs(path): - # path = path.replace('/', os.sep) - # path = os.path.join(base_location, path) - results.append((path, checksum, size)) - return results - - @cached_property - def exports(self): - """ - Return the information exported by this distribution. - :return: A dictionary of exports, mapping an export category to a dict - of :class:`ExportEntry` instances describing the individual - export entries, and keyed by name. - """ - result = {} - r = self.get_distinfo_resource(EXPORTS_FILENAME) - if r: - result = self.read_exports() - return result - - def read_exports(self): - """ - Read exports data from a file in .ini format. - - :return: A dictionary of exports, mapping an export category to a list - of :class:`ExportEntry` instances describing the individual - export entries. - """ - result = {} - r = self.get_distinfo_resource(EXPORTS_FILENAME) - if r: - with contextlib.closing(r.as_stream()) as stream: - result = read_exports(stream) - return result - - def write_exports(self, exports): - """ - Write a dictionary of exports to a file in .ini format. - :param exports: A dictionary of exports, mapping an export category to - a list of :class:`ExportEntry` instances describing the - individual export entries. - """ - rf = self.get_distinfo_file(EXPORTS_FILENAME) - with open(rf, 'w') as f: - write_exports(exports, f) - - def get_resource_path(self, relative_path): - """ - NOTE: This API may change in the future. - - Return the absolute path to a resource file with the given relative - path. - - :param relative_path: The path, relative to .dist-info, of the resource - of interest. - :return: The absolute path where the resource is to be found. - """ - r = self.get_distinfo_resource('RESOURCES') - with contextlib.closing(r.as_stream()) as stream: - with CSVReader(stream=stream) as resources_reader: - for relative, destination in resources_reader: - if relative == relative_path: - return destination - raise KeyError('no resource file with relative path %r ' - 'is installed' % relative_path) - - def list_installed_files(self): - """ - Iterates over the ``RECORD`` entries and returns a tuple - ``(path, hash, size)`` for each line. - - :returns: iterator of (path, hash, size) - """ - for result in self._get_records(): - yield result - - def write_installed_files(self, paths, prefix, dry_run=False): - """ - Writes the ``RECORD`` file, using the ``paths`` iterable passed in. Any - existing ``RECORD`` file is silently overwritten. - - prefix is used to determine when to write absolute paths. - """ - prefix = os.path.join(prefix, '') - base = os.path.dirname(self.path) - base_under_prefix = base.startswith(prefix) - base = os.path.join(base, '') - record_path = self.get_distinfo_file('RECORD') - logger.info('creating %s', record_path) - if dry_run: - return None - with CSVWriter(record_path) as writer: - for path in paths: - if os.path.isdir(path) or path.endswith(('.pyc', '.pyo')): - # do not put size and hash, as in PEP-376 - hash_value = size = '' - else: - size = '%d' % os.path.getsize(path) - with open(path, 'rb') as fp: - hash_value = self.get_hash(fp.read()) - if path.startswith(base) or (base_under_prefix and - path.startswith(prefix)): - path = os.path.relpath(path, base) - writer.writerow((path, hash_value, size)) - - # add the RECORD file itself - if record_path.startswith(base): - record_path = os.path.relpath(record_path, base) - writer.writerow((record_path, '', '')) - return record_path - - def check_installed_files(self): - """ - Checks that the hashes and sizes of the files in ``RECORD`` are - matched by the files themselves. Returns a (possibly empty) list of - mismatches. Each entry in the mismatch list will be a tuple consisting - of the path, 'exists', 'size' or 'hash' according to what didn't match - (existence is checked first, then size, then hash), the expected - value and the actual value. - """ - mismatches = [] - base = os.path.dirname(self.path) - record_path = self.get_distinfo_file('RECORD') - for path, hash_value, size in self.list_installed_files(): - if not os.path.isabs(path): - path = os.path.join(base, path) - if path == record_path: - continue - if not os.path.exists(path): - mismatches.append((path, 'exists', True, False)) - elif os.path.isfile(path): - actual_size = str(os.path.getsize(path)) - if size and actual_size != size: - mismatches.append((path, 'size', size, actual_size)) - elif hash_value: - if '=' in hash_value: - hasher = hash_value.split('=', 1)[0] - else: - hasher = None - - with open(path, 'rb') as f: - actual_hash = self.get_hash(f.read(), hasher) - if actual_hash != hash_value: - mismatches.append((path, 'hash', hash_value, actual_hash)) - return mismatches - - @cached_property - def shared_locations(self): - """ - A dictionary of shared locations whose keys are in the set 'prefix', - 'purelib', 'platlib', 'scripts', 'headers', 'data' and 'namespace'. - The corresponding value is the absolute path of that category for - this distribution, and takes into account any paths selected by the - user at installation time (e.g. via command-line arguments). In the - case of the 'namespace' key, this would be a list of absolute paths - for the roots of namespace packages in this distribution. - - The first time this property is accessed, the relevant information is - read from the SHARED file in the .dist-info directory. - """ - result = {} - shared_path = os.path.join(self.path, 'SHARED') - if os.path.isfile(shared_path): - with codecs.open(shared_path, 'r', encoding='utf-8') as f: - lines = f.read().splitlines() - for line in lines: - key, value = line.split('=', 1) - if key == 'namespace': - result.setdefault(key, []).append(value) - else: - result[key] = value - return result - - def write_shared_locations(self, paths, dry_run=False): - """ - Write shared location information to the SHARED file in .dist-info. - :param paths: A dictionary as described in the documentation for - :meth:`shared_locations`. - :param dry_run: If True, the action is logged but no file is actually - written. - :return: The path of the file written to. - """ - shared_path = os.path.join(self.path, 'SHARED') - logger.info('creating %s', shared_path) - if dry_run: - return None - lines = [] - for key in ('prefix', 'lib', 'headers', 'scripts', 'data'): - path = paths[key] - if os.path.isdir(paths[key]): - lines.append('%s=%s' % (key, path)) - for ns in paths.get('namespace', ()): - lines.append('namespace=%s' % ns) - - with codecs.open(shared_path, 'w', encoding='utf-8') as f: - f.write('\n'.join(lines)) - return shared_path - - def get_distinfo_resource(self, path): - if path not in DIST_FILES: - raise DistlibException('invalid path for a dist-info file: ' - '%r at %r' % (path, self.path)) - finder = resources.finder_for_path(self.path) - if finder is None: - raise DistlibException('Unable to get a finder for %s' % self.path) - return finder.find(path) - - def get_distinfo_file(self, path): - """ - Returns a path located under the ``.dist-info`` directory. Returns a - string representing the path. - - :parameter path: a ``'/'``-separated path relative to the - ``.dist-info`` directory or an absolute path; - If *path* is an absolute path and doesn't start - with the ``.dist-info`` directory path, - a :class:`DistlibException` is raised - :type path: str - :rtype: str - """ - # Check if it is an absolute path # XXX use relpath, add tests - if path.find(os.sep) >= 0: - # it's an absolute path? - distinfo_dirname, path = path.split(os.sep)[-2:] - if distinfo_dirname != self.path.split(os.sep)[-1]: - raise DistlibException( - 'dist-info file %r does not belong to the %r %s ' - 'distribution' % (path, self.name, self.version)) - - # The file must be relative - if path not in DIST_FILES: - raise DistlibException('invalid path for a dist-info file: ' - '%r at %r' % (path, self.path)) - - return os.path.join(self.path, path) - - def list_distinfo_files(self): - """ - Iterates over the ``RECORD`` entries and returns paths for each line if - the path is pointing to a file located in the ``.dist-info`` directory - or one of its subdirectories. - - :returns: iterator of paths - """ - base = os.path.dirname(self.path) - for path, checksum, size in self._get_records(): - # XXX add separator or use real relpath algo - if not os.path.isabs(path): - path = os.path.join(base, path) - if path.startswith(self.path): - yield path - - def __eq__(self, other): - return (isinstance(other, InstalledDistribution) and - self.path == other.path) - - # See http://docs.python.org/reference/datamodel#object.__hash__ - __hash__ = object.__hash__ - - -class EggInfoDistribution(BaseInstalledDistribution): - """Created with the *path* of the ``.egg-info`` directory or file provided - to the constructor. It reads the metadata contained in the file itself, or - if the given path happens to be a directory, the metadata is read from the - file ``PKG-INFO`` under that directory.""" - - requested = True # as we have no way of knowing, assume it was - shared_locations = {} - - def __init__(self, path, env=None): - def set_name_and_version(s, n, v): - s.name = n - s.key = n.lower() # for case-insensitive comparisons - s.version = v - - self.path = path - self.dist_path = env - if env and env._cache_enabled and path in env._cache_egg.path: - metadata = env._cache_egg.path[path].metadata - set_name_and_version(self, metadata.name, metadata.version) - else: - metadata = self._get_metadata(path) - - # Need to be set before caching - set_name_and_version(self, metadata.name, metadata.version) - - if env and env._cache_enabled: - env._cache_egg.add(self) - super(EggInfoDistribution, self).__init__(metadata, path, env) - - def _get_metadata(self, path): - requires = None - - def parse_requires_data(data): - """Create a list of dependencies from a requires.txt file. - - *data*: the contents of a setuptools-produced requires.txt file. - """ - reqs = [] - lines = data.splitlines() - for line in lines: - line = line.strip() - if line.startswith('['): - logger.warning('Unexpected line: quitting requirement scan: %r', - line) - break - r = parse_requirement(line) - if not r: - logger.warning('Not recognised as a requirement: %r', line) - continue - if r.extras: - logger.warning('extra requirements in requires.txt are ' - 'not supported') - if not r.constraints: - reqs.append(r.name) - else: - cons = ', '.join('%s%s' % c for c in r.constraints) - reqs.append('%s (%s)' % (r.name, cons)) - return reqs - - def parse_requires_path(req_path): - """Create a list of dependencies from a requires.txt file. - - *req_path*: the path to a setuptools-produced requires.txt file. - """ - - reqs = [] - try: - with codecs.open(req_path, 'r', 'utf-8') as fp: - reqs = parse_requires_data(fp.read()) - except IOError: - pass - return reqs - - tl_path = tl_data = None - if path.endswith('.egg'): - if os.path.isdir(path): - p = os.path.join(path, 'EGG-INFO') - meta_path = os.path.join(p, 'PKG-INFO') - metadata = Metadata(path=meta_path, scheme='legacy') - req_path = os.path.join(p, 'requires.txt') - tl_path = os.path.join(p, 'top_level.txt') - requires = parse_requires_path(req_path) - else: - # FIXME handle the case where zipfile is not available - zipf = zipimport.zipimporter(path) - fileobj = StringIO( - zipf.get_data('EGG-INFO/PKG-INFO').decode('utf8')) - metadata = Metadata(fileobj=fileobj, scheme='legacy') - try: - data = zipf.get_data('EGG-INFO/requires.txt') - tl_data = zipf.get_data('EGG-INFO/top_level.txt').decode('utf-8') - requires = parse_requires_data(data.decode('utf-8')) - except IOError: - requires = None - elif path.endswith('.egg-info'): - if os.path.isdir(path): - req_path = os.path.join(path, 'requires.txt') - requires = parse_requires_path(req_path) - path = os.path.join(path, 'PKG-INFO') - tl_path = os.path.join(path, 'top_level.txt') - metadata = Metadata(path=path, scheme='legacy') - else: - raise DistlibException('path must end with .egg-info or .egg, ' - 'got %r' % path) - - if requires: - metadata.add_requirements(requires) - # look for top-level modules in top_level.txt, if present - if tl_data is None: - if tl_path is not None and os.path.exists(tl_path): - with open(tl_path, 'rb') as f: - tl_data = f.read().decode('utf-8') - if not tl_data: - tl_data = [] - else: - tl_data = tl_data.splitlines() - self.modules = tl_data - return metadata - - def __repr__(self): - return '' % ( - self.name, self.version, self.path) - - def __str__(self): - return "%s %s" % (self.name, self.version) - - def check_installed_files(self): - """ - Checks that the hashes and sizes of the files in ``RECORD`` are - matched by the files themselves. Returns a (possibly empty) list of - mismatches. Each entry in the mismatch list will be a tuple consisting - of the path, 'exists', 'size' or 'hash' according to what didn't match - (existence is checked first, then size, then hash), the expected - value and the actual value. - """ - mismatches = [] - record_path = os.path.join(self.path, 'installed-files.txt') - if os.path.exists(record_path): - for path, _, _ in self.list_installed_files(): - if path == record_path: - continue - if not os.path.exists(path): - mismatches.append((path, 'exists', True, False)) - return mismatches - - def list_installed_files(self): - """ - Iterates over the ``installed-files.txt`` entries and returns a tuple - ``(path, hash, size)`` for each line. - - :returns: a list of (path, hash, size) - """ - - def _md5(path): - f = open(path, 'rb') - try: - content = f.read() - finally: - f.close() - return hashlib.md5(content).hexdigest() - - def _size(path): - return os.stat(path).st_size - - record_path = os.path.join(self.path, 'installed-files.txt') - result = [] - if os.path.exists(record_path): - with codecs.open(record_path, 'r', encoding='utf-8') as f: - for line in f: - line = line.strip() - p = os.path.normpath(os.path.join(self.path, line)) - # "./" is present as a marker between installed files - # and installation metadata files - if not os.path.exists(p): - logger.warning('Non-existent file: %s', p) - if p.endswith(('.pyc', '.pyo')): - continue - #otherwise fall through and fail - if not os.path.isdir(p): - result.append((p, _md5(p), _size(p))) - result.append((record_path, None, None)) - return result - - def list_distinfo_files(self, absolute=False): - """ - Iterates over the ``installed-files.txt`` entries and returns paths for - each line if the path is pointing to a file located in the - ``.egg-info`` directory or one of its subdirectories. - - :parameter absolute: If *absolute* is ``True``, each returned path is - transformed into a local absolute path. Otherwise the - raw value from ``installed-files.txt`` is returned. - :type absolute: boolean - :returns: iterator of paths - """ - record_path = os.path.join(self.path, 'installed-files.txt') - if os.path.exists(record_path): - skip = True - with codecs.open(record_path, 'r', encoding='utf-8') as f: - for line in f: - line = line.strip() - if line == './': - skip = False - continue - if not skip: - p = os.path.normpath(os.path.join(self.path, line)) - if p.startswith(self.path): - if absolute: - yield p - else: - yield line - - def __eq__(self, other): - return (isinstance(other, EggInfoDistribution) and - self.path == other.path) - - # See http://docs.python.org/reference/datamodel#object.__hash__ - __hash__ = object.__hash__ - -new_dist_class = InstalledDistribution -old_dist_class = EggInfoDistribution - - -class DependencyGraph(object): - """ - Represents a dependency graph between distributions. - - The dependency relationships are stored in an ``adjacency_list`` that maps - distributions to a list of ``(other, label)`` tuples where ``other`` - is a distribution and the edge is labeled with ``label`` (i.e. the version - specifier, if such was provided). Also, for more efficient traversal, for - every distribution ``x``, a list of predecessors is kept in - ``reverse_list[x]``. An edge from distribution ``a`` to - distribution ``b`` means that ``a`` depends on ``b``. If any missing - dependencies are found, they are stored in ``missing``, which is a - dictionary that maps distributions to a list of requirements that were not - provided by any other distributions. - """ - - def __init__(self): - self.adjacency_list = {} - self.reverse_list = {} - self.missing = {} - - def add_distribution(self, distribution): - """Add the *distribution* to the graph. - - :type distribution: :class:`distutils2.database.InstalledDistribution` - or :class:`distutils2.database.EggInfoDistribution` - """ - self.adjacency_list[distribution] = [] - self.reverse_list[distribution] = [] - #self.missing[distribution] = [] - - def add_edge(self, x, y, label=None): - """Add an edge from distribution *x* to distribution *y* with the given - *label*. - - :type x: :class:`distutils2.database.InstalledDistribution` or - :class:`distutils2.database.EggInfoDistribution` - :type y: :class:`distutils2.database.InstalledDistribution` or - :class:`distutils2.database.EggInfoDistribution` - :type label: ``str`` or ``None`` - """ - self.adjacency_list[x].append((y, label)) - # multiple edges are allowed, so be careful - if x not in self.reverse_list[y]: - self.reverse_list[y].append(x) - - def add_missing(self, distribution, requirement): - """ - Add a missing *requirement* for the given *distribution*. - - :type distribution: :class:`distutils2.database.InstalledDistribution` - or :class:`distutils2.database.EggInfoDistribution` - :type requirement: ``str`` - """ - logger.debug('%s missing %r', distribution, requirement) - self.missing.setdefault(distribution, []).append(requirement) - - def _repr_dist(self, dist): - return '%s %s' % (dist.name, dist.version) - - def repr_node(self, dist, level=1): - """Prints only a subgraph""" - output = [self._repr_dist(dist)] - for other, label in self.adjacency_list[dist]: - dist = self._repr_dist(other) - if label is not None: - dist = '%s [%s]' % (dist, label) - output.append(' ' * level + str(dist)) - suboutput = self.repr_node(other, level + 1) - subs = suboutput.split('\n') - output.extend(subs[1:]) - return '\n'.join(output) - - def to_dot(self, f, skip_disconnected=True): - """Writes a DOT output for the graph to the provided file *f*. - - If *skip_disconnected* is set to ``True``, then all distributions - that are not dependent on any other distribution are skipped. - - :type f: has to support ``file``-like operations - :type skip_disconnected: ``bool`` - """ - disconnected = [] - - f.write("digraph dependencies {\n") - for dist, adjs in self.adjacency_list.items(): - if len(adjs) == 0 and not skip_disconnected: - disconnected.append(dist) - for other, label in adjs: - if not label is None: - f.write('"%s" -> "%s" [label="%s"]\n' % - (dist.name, other.name, label)) - else: - f.write('"%s" -> "%s"\n' % (dist.name, other.name)) - if not skip_disconnected and len(disconnected) > 0: - f.write('subgraph disconnected {\n') - f.write('label = "Disconnected"\n') - f.write('bgcolor = red\n') - - for dist in disconnected: - f.write('"%s"' % dist.name) - f.write('\n') - f.write('}\n') - f.write('}\n') - - def topological_sort(self): - """ - Perform a topological sort of the graph. - :return: A tuple, the first element of which is a topologically sorted - list of distributions, and the second element of which is a - list of distributions that cannot be sorted because they have - circular dependencies and so form a cycle. - """ - result = [] - # Make a shallow copy of the adjacency list - alist = {} - for k, v in self.adjacency_list.items(): - alist[k] = v[:] - while True: - # See what we can remove in this run - to_remove = [] - for k, v in list(alist.items())[:]: - if not v: - to_remove.append(k) - del alist[k] - if not to_remove: - # What's left in alist (if anything) is a cycle. - break - # Remove from the adjacency list of others - for k, v in alist.items(): - alist[k] = [(d, r) for d, r in v if d not in to_remove] - logger.debug('Moving to result: %s', - ['%s (%s)' % (d.name, d.version) for d in to_remove]) - result.extend(to_remove) - return result, list(alist.keys()) - - def __repr__(self): - """Representation of the graph""" - output = [] - for dist, adjs in self.adjacency_list.items(): - output.append(self.repr_node(dist)) - return '\n'.join(output) - - -def make_graph(dists, scheme='default'): - """Makes a dependency graph from the given distributions. - - :parameter dists: a list of distributions - :type dists: list of :class:`distutils2.database.InstalledDistribution` and - :class:`distutils2.database.EggInfoDistribution` instances - :rtype: a :class:`DependencyGraph` instance - """ - scheme = get_scheme(scheme) - graph = DependencyGraph() - provided = {} # maps names to lists of (version, dist) tuples - - # first, build the graph and find out what's provided - for dist in dists: - graph.add_distribution(dist) - - for p in dist.provides: - name, version = parse_name_and_version(p) - logger.debug('Add to provided: %s, %s, %s', name, version, dist) - provided.setdefault(name, []).append((version, dist)) - - # now make the edges - for dist in dists: - requires = (dist.run_requires | dist.meta_requires | - dist.build_requires | dist.dev_requires) - for req in requires: - try: - matcher = scheme.matcher(req) - except UnsupportedVersionError: - # XXX compat-mode if cannot read the version - logger.warning('could not read version %r - using name only', - req) - name = req.split()[0] - matcher = scheme.matcher(name) - - name = matcher.key # case-insensitive - - matched = False - if name in provided: - for version, provider in provided[name]: - try: - match = matcher.match(version) - except UnsupportedVersionError: - match = False - - if match: - graph.add_edge(dist, provider, req) - matched = True - break - if not matched: - graph.add_missing(dist, req) - return graph - - -def get_dependent_dists(dists, dist): - """Recursively generate a list of distributions from *dists* that are - dependent on *dist*. - - :param dists: a list of distributions - :param dist: a distribution, member of *dists* for which we are interested - """ - if dist not in dists: - raise DistlibException('given distribution %r is not a member ' - 'of the list' % dist.name) - graph = make_graph(dists) - - dep = [dist] # dependent distributions - todo = graph.reverse_list[dist] # list of nodes we should inspect - - while todo: - d = todo.pop() - dep.append(d) - for succ in graph.reverse_list[d]: - if succ not in dep: - todo.append(succ) - - dep.pop(0) # remove dist from dep, was there to prevent infinite loops - return dep - - -def get_required_dists(dists, dist): - """Recursively generate a list of distributions from *dists* that are - required by *dist*. - - :param dists: a list of distributions - :param dist: a distribution, member of *dists* for which we are interested - in finding the dependencies. - """ - if dist not in dists: - raise DistlibException('given distribution %r is not a member ' - 'of the list' % dist.name) - graph = make_graph(dists) - - req = set() # required distributions - todo = graph.adjacency_list[dist] # list of nodes we should inspect - seen = set(t[0] for t in todo) # already added to todo - - while todo: - d = todo.pop()[0] - req.add(d) - pred_list = graph.adjacency_list[d] - for pred in pred_list: - d = pred[0] - if d not in req and d not in seen: - seen.add(d) - todo.append(pred) - return req - - -def make_dist(name, version, **kwargs): - """ - A convenience method for making a dist given just a name and version. - """ - summary = kwargs.pop('summary', 'Placeholder for summary') - md = Metadata(**kwargs) - md.name = name - md.version = version - md.summary = summary or 'Placeholder for summary' - return Distribution(md) diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/distlib/index.py b/venv/lib/python3.11/site-packages/pip/_vendor/distlib/index.py deleted file mode 100644 index 9b6d129..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/distlib/index.py +++ /dev/null @@ -1,508 +0,0 @@ -# -*- coding: utf-8 -*- -# -# Copyright (C) 2013 Vinay Sajip. -# Licensed to the Python Software Foundation under a contributor agreement. -# See LICENSE.txt and CONTRIBUTORS.txt. -# -import hashlib -import logging -import os -import shutil -import subprocess -import tempfile -try: - from threading import Thread -except ImportError: # pragma: no cover - from dummy_threading import Thread - -from . import DistlibException -from .compat import (HTTPBasicAuthHandler, Request, HTTPPasswordMgr, - urlparse, build_opener, string_types) -from .util import zip_dir, ServerProxy - -logger = logging.getLogger(__name__) - -DEFAULT_INDEX = 'https://pypi.org/pypi' -DEFAULT_REALM = 'pypi' - -class PackageIndex(object): - """ - This class represents a package index compatible with PyPI, the Python - Package Index. - """ - - boundary = b'----------ThIs_Is_tHe_distlib_index_bouNdaRY_$' - - def __init__(self, url=None): - """ - Initialise an instance. - - :param url: The URL of the index. If not specified, the URL for PyPI is - used. - """ - self.url = url or DEFAULT_INDEX - self.read_configuration() - scheme, netloc, path, params, query, frag = urlparse(self.url) - if params or query or frag or scheme not in ('http', 'https'): - raise DistlibException('invalid repository: %s' % self.url) - self.password_handler = None - self.ssl_verifier = None - self.gpg = None - self.gpg_home = None - with open(os.devnull, 'w') as sink: - # Use gpg by default rather than gpg2, as gpg2 insists on - # prompting for passwords - for s in ('gpg', 'gpg2'): - try: - rc = subprocess.check_call([s, '--version'], stdout=sink, - stderr=sink) - if rc == 0: - self.gpg = s - break - except OSError: - pass - - def _get_pypirc_command(self): - """ - Get the distutils command for interacting with PyPI configurations. - :return: the command. - """ - from .util import _get_pypirc_command as cmd - return cmd() - - def read_configuration(self): - """ - Read the PyPI access configuration as supported by distutils. This populates - ``username``, ``password``, ``realm`` and ``url`` attributes from the - configuration. - """ - from .util import _load_pypirc - cfg = _load_pypirc(self) - self.username = cfg.get('username') - self.password = cfg.get('password') - self.realm = cfg.get('realm', 'pypi') - self.url = cfg.get('repository', self.url) - - def save_configuration(self): - """ - Save the PyPI access configuration. You must have set ``username`` and - ``password`` attributes before calling this method. - """ - self.check_credentials() - from .util import _store_pypirc - _store_pypirc(self) - - def check_credentials(self): - """ - Check that ``username`` and ``password`` have been set, and raise an - exception if not. - """ - if self.username is None or self.password is None: - raise DistlibException('username and password must be set') - pm = HTTPPasswordMgr() - _, netloc, _, _, _, _ = urlparse(self.url) - pm.add_password(self.realm, netloc, self.username, self.password) - self.password_handler = HTTPBasicAuthHandler(pm) - - def register(self, metadata): # pragma: no cover - """ - Register a distribution on PyPI, using the provided metadata. - - :param metadata: A :class:`Metadata` instance defining at least a name - and version number for the distribution to be - registered. - :return: The HTTP response received from PyPI upon submission of the - request. - """ - self.check_credentials() - metadata.validate() - d = metadata.todict() - d[':action'] = 'verify' - request = self.encode_request(d.items(), []) - response = self.send_request(request) - d[':action'] = 'submit' - request = self.encode_request(d.items(), []) - return self.send_request(request) - - def _reader(self, name, stream, outbuf): - """ - Thread runner for reading lines of from a subprocess into a buffer. - - :param name: The logical name of the stream (used for logging only). - :param stream: The stream to read from. This will typically a pipe - connected to the output stream of a subprocess. - :param outbuf: The list to append the read lines to. - """ - while True: - s = stream.readline() - if not s: - break - s = s.decode('utf-8').rstrip() - outbuf.append(s) - logger.debug('%s: %s' % (name, s)) - stream.close() - - def get_sign_command(self, filename, signer, sign_password, keystore=None): # pragma: no cover - """ - Return a suitable command for signing a file. - - :param filename: The pathname to the file to be signed. - :param signer: The identifier of the signer of the file. - :param sign_password: The passphrase for the signer's - private key used for signing. - :param keystore: The path to a directory which contains the keys - used in verification. If not specified, the - instance's ``gpg_home`` attribute is used instead. - :return: The signing command as a list suitable to be - passed to :class:`subprocess.Popen`. - """ - cmd = [self.gpg, '--status-fd', '2', '--no-tty'] - if keystore is None: - keystore = self.gpg_home - if keystore: - cmd.extend(['--homedir', keystore]) - if sign_password is not None: - cmd.extend(['--batch', '--passphrase-fd', '0']) - td = tempfile.mkdtemp() - sf = os.path.join(td, os.path.basename(filename) + '.asc') - cmd.extend(['--detach-sign', '--armor', '--local-user', - signer, '--output', sf, filename]) - logger.debug('invoking: %s', ' '.join(cmd)) - return cmd, sf - - def run_command(self, cmd, input_data=None): - """ - Run a command in a child process , passing it any input data specified. - - :param cmd: The command to run. - :param input_data: If specified, this must be a byte string containing - data to be sent to the child process. - :return: A tuple consisting of the subprocess' exit code, a list of - lines read from the subprocess' ``stdout``, and a list of - lines read from the subprocess' ``stderr``. - """ - kwargs = { - 'stdout': subprocess.PIPE, - 'stderr': subprocess.PIPE, - } - if input_data is not None: - kwargs['stdin'] = subprocess.PIPE - stdout = [] - stderr = [] - p = subprocess.Popen(cmd, **kwargs) - # We don't use communicate() here because we may need to - # get clever with interacting with the command - t1 = Thread(target=self._reader, args=('stdout', p.stdout, stdout)) - t1.start() - t2 = Thread(target=self._reader, args=('stderr', p.stderr, stderr)) - t2.start() - if input_data is not None: - p.stdin.write(input_data) - p.stdin.close() - - p.wait() - t1.join() - t2.join() - return p.returncode, stdout, stderr - - def sign_file(self, filename, signer, sign_password, keystore=None): # pragma: no cover - """ - Sign a file. - - :param filename: The pathname to the file to be signed. - :param signer: The identifier of the signer of the file. - :param sign_password: The passphrase for the signer's - private key used for signing. - :param keystore: The path to a directory which contains the keys - used in signing. If not specified, the instance's - ``gpg_home`` attribute is used instead. - :return: The absolute pathname of the file where the signature is - stored. - """ - cmd, sig_file = self.get_sign_command(filename, signer, sign_password, - keystore) - rc, stdout, stderr = self.run_command(cmd, - sign_password.encode('utf-8')) - if rc != 0: - raise DistlibException('sign command failed with error ' - 'code %s' % rc) - return sig_file - - def upload_file(self, metadata, filename, signer=None, sign_password=None, - filetype='sdist', pyversion='source', keystore=None): - """ - Upload a release file to the index. - - :param metadata: A :class:`Metadata` instance defining at least a name - and version number for the file to be uploaded. - :param filename: The pathname of the file to be uploaded. - :param signer: The identifier of the signer of the file. - :param sign_password: The passphrase for the signer's - private key used for signing. - :param filetype: The type of the file being uploaded. This is the - distutils command which produced that file, e.g. - ``sdist`` or ``bdist_wheel``. - :param pyversion: The version of Python which the release relates - to. For code compatible with any Python, this would - be ``source``, otherwise it would be e.g. ``3.2``. - :param keystore: The path to a directory which contains the keys - used in signing. If not specified, the instance's - ``gpg_home`` attribute is used instead. - :return: The HTTP response received from PyPI upon submission of the - request. - """ - self.check_credentials() - if not os.path.exists(filename): - raise DistlibException('not found: %s' % filename) - metadata.validate() - d = metadata.todict() - sig_file = None - if signer: - if not self.gpg: - logger.warning('no signing program available - not signed') - else: - sig_file = self.sign_file(filename, signer, sign_password, - keystore) - with open(filename, 'rb') as f: - file_data = f.read() - md5_digest = hashlib.md5(file_data).hexdigest() - sha256_digest = hashlib.sha256(file_data).hexdigest() - d.update({ - ':action': 'file_upload', - 'protocol_version': '1', - 'filetype': filetype, - 'pyversion': pyversion, - 'md5_digest': md5_digest, - 'sha256_digest': sha256_digest, - }) - files = [('content', os.path.basename(filename), file_data)] - if sig_file: - with open(sig_file, 'rb') as f: - sig_data = f.read() - files.append(('gpg_signature', os.path.basename(sig_file), - sig_data)) - shutil.rmtree(os.path.dirname(sig_file)) - request = self.encode_request(d.items(), files) - return self.send_request(request) - - def upload_documentation(self, metadata, doc_dir): # pragma: no cover - """ - Upload documentation to the index. - - :param metadata: A :class:`Metadata` instance defining at least a name - and version number for the documentation to be - uploaded. - :param doc_dir: The pathname of the directory which contains the - documentation. This should be the directory that - contains the ``index.html`` for the documentation. - :return: The HTTP response received from PyPI upon submission of the - request. - """ - self.check_credentials() - if not os.path.isdir(doc_dir): - raise DistlibException('not a directory: %r' % doc_dir) - fn = os.path.join(doc_dir, 'index.html') - if not os.path.exists(fn): - raise DistlibException('not found: %r' % fn) - metadata.validate() - name, version = metadata.name, metadata.version - zip_data = zip_dir(doc_dir).getvalue() - fields = [(':action', 'doc_upload'), - ('name', name), ('version', version)] - files = [('content', name, zip_data)] - request = self.encode_request(fields, files) - return self.send_request(request) - - def get_verify_command(self, signature_filename, data_filename, - keystore=None): - """ - Return a suitable command for verifying a file. - - :param signature_filename: The pathname to the file containing the - signature. - :param data_filename: The pathname to the file containing the - signed data. - :param keystore: The path to a directory which contains the keys - used in verification. If not specified, the - instance's ``gpg_home`` attribute is used instead. - :return: The verifying command as a list suitable to be - passed to :class:`subprocess.Popen`. - """ - cmd = [self.gpg, '--status-fd', '2', '--no-tty'] - if keystore is None: - keystore = self.gpg_home - if keystore: - cmd.extend(['--homedir', keystore]) - cmd.extend(['--verify', signature_filename, data_filename]) - logger.debug('invoking: %s', ' '.join(cmd)) - return cmd - - def verify_signature(self, signature_filename, data_filename, - keystore=None): - """ - Verify a signature for a file. - - :param signature_filename: The pathname to the file containing the - signature. - :param data_filename: The pathname to the file containing the - signed data. - :param keystore: The path to a directory which contains the keys - used in verification. If not specified, the - instance's ``gpg_home`` attribute is used instead. - :return: True if the signature was verified, else False. - """ - if not self.gpg: - raise DistlibException('verification unavailable because gpg ' - 'unavailable') - cmd = self.get_verify_command(signature_filename, data_filename, - keystore) - rc, stdout, stderr = self.run_command(cmd) - if rc not in (0, 1): - raise DistlibException('verify command failed with error ' - 'code %s' % rc) - return rc == 0 - - def download_file(self, url, destfile, digest=None, reporthook=None): - """ - This is a convenience method for downloading a file from an URL. - Normally, this will be a file from the index, though currently - no check is made for this (i.e. a file can be downloaded from - anywhere). - - The method is just like the :func:`urlretrieve` function in the - standard library, except that it allows digest computation to be - done during download and checking that the downloaded data - matched any expected value. - - :param url: The URL of the file to be downloaded (assumed to be - available via an HTTP GET request). - :param destfile: The pathname where the downloaded file is to be - saved. - :param digest: If specified, this must be a (hasher, value) - tuple, where hasher is the algorithm used (e.g. - ``'md5'``) and ``value`` is the expected value. - :param reporthook: The same as for :func:`urlretrieve` in the - standard library. - """ - if digest is None: - digester = None - logger.debug('No digest specified') - else: - if isinstance(digest, (list, tuple)): - hasher, digest = digest - else: - hasher = 'md5' - digester = getattr(hashlib, hasher)() - logger.debug('Digest specified: %s' % digest) - # The following code is equivalent to urlretrieve. - # We need to do it this way so that we can compute the - # digest of the file as we go. - with open(destfile, 'wb') as dfp: - # addinfourl is not a context manager on 2.x - # so we have to use try/finally - sfp = self.send_request(Request(url)) - try: - headers = sfp.info() - blocksize = 8192 - size = -1 - read = 0 - blocknum = 0 - if "content-length" in headers: - size = int(headers["Content-Length"]) - if reporthook: - reporthook(blocknum, blocksize, size) - while True: - block = sfp.read(blocksize) - if not block: - break - read += len(block) - dfp.write(block) - if digester: - digester.update(block) - blocknum += 1 - if reporthook: - reporthook(blocknum, blocksize, size) - finally: - sfp.close() - - # check that we got the whole file, if we can - if size >= 0 and read < size: - raise DistlibException( - 'retrieval incomplete: got only %d out of %d bytes' - % (read, size)) - # if we have a digest, it must match. - if digester: - actual = digester.hexdigest() - if digest != actual: - raise DistlibException('%s digest mismatch for %s: expected ' - '%s, got %s' % (hasher, destfile, - digest, actual)) - logger.debug('Digest verified: %s', digest) - - def send_request(self, req): - """ - Send a standard library :class:`Request` to PyPI and return its - response. - - :param req: The request to send. - :return: The HTTP response from PyPI (a standard library HTTPResponse). - """ - handlers = [] - if self.password_handler: - handlers.append(self.password_handler) - if self.ssl_verifier: - handlers.append(self.ssl_verifier) - opener = build_opener(*handlers) - return opener.open(req) - - def encode_request(self, fields, files): - """ - Encode fields and files for posting to an HTTP server. - - :param fields: The fields to send as a list of (fieldname, value) - tuples. - :param files: The files to send as a list of (fieldname, filename, - file_bytes) tuple. - """ - # Adapted from packaging, which in turn was adapted from - # http://code.activestate.com/recipes/146306 - - parts = [] - boundary = self.boundary - for k, values in fields: - if not isinstance(values, (list, tuple)): - values = [values] - - for v in values: - parts.extend(( - b'--' + boundary, - ('Content-Disposition: form-data; name="%s"' % - k).encode('utf-8'), - b'', - v.encode('utf-8'))) - for key, filename, value in files: - parts.extend(( - b'--' + boundary, - ('Content-Disposition: form-data; name="%s"; filename="%s"' % - (key, filename)).encode('utf-8'), - b'', - value)) - - parts.extend((b'--' + boundary + b'--', b'')) - - body = b'\r\n'.join(parts) - ct = b'multipart/form-data; boundary=' + boundary - headers = { - 'Content-type': ct, - 'Content-length': str(len(body)) - } - return Request(self.url, body, headers) - - def search(self, terms, operator=None): # pragma: no cover - if isinstance(terms, string_types): - terms = {'name': terms} - rpc_proxy = ServerProxy(self.url, timeout=3.0) - try: - return rpc_proxy.search(terms, operator or 'and') - finally: - rpc_proxy('close')() diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/distlib/locators.py b/venv/lib/python3.11/site-packages/pip/_vendor/distlib/locators.py deleted file mode 100644 index 966ebc0..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/distlib/locators.py +++ /dev/null @@ -1,1300 +0,0 @@ -# -*- coding: utf-8 -*- -# -# Copyright (C) 2012-2015 Vinay Sajip. -# Licensed to the Python Software Foundation under a contributor agreement. -# See LICENSE.txt and CONTRIBUTORS.txt. -# - -import gzip -from io import BytesIO -import json -import logging -import os -import posixpath -import re -try: - import threading -except ImportError: # pragma: no cover - import dummy_threading as threading -import zlib - -from . import DistlibException -from .compat import (urljoin, urlparse, urlunparse, url2pathname, pathname2url, - queue, quote, unescape, build_opener, - HTTPRedirectHandler as BaseRedirectHandler, text_type, - Request, HTTPError, URLError) -from .database import Distribution, DistributionPath, make_dist -from .metadata import Metadata, MetadataInvalidError -from .util import (cached_property, ensure_slash, split_filename, get_project_data, - parse_requirement, parse_name_and_version, ServerProxy, - normalize_name) -from .version import get_scheme, UnsupportedVersionError -from .wheel import Wheel, is_compatible - -logger = logging.getLogger(__name__) - -HASHER_HASH = re.compile(r'^(\w+)=([a-f0-9]+)') -CHARSET = re.compile(r';\s*charset\s*=\s*(.*)\s*$', re.I) -HTML_CONTENT_TYPE = re.compile('text/html|application/x(ht)?ml') -DEFAULT_INDEX = 'https://pypi.org/pypi' - -def get_all_distribution_names(url=None): - """ - Return all distribution names known by an index. - :param url: The URL of the index. - :return: A list of all known distribution names. - """ - if url is None: - url = DEFAULT_INDEX - client = ServerProxy(url, timeout=3.0) - try: - return client.list_packages() - finally: - client('close')() - -class RedirectHandler(BaseRedirectHandler): - """ - A class to work around a bug in some Python 3.2.x releases. - """ - # There's a bug in the base version for some 3.2.x - # (e.g. 3.2.2 on Ubuntu Oneiric). If a Location header - # returns e.g. /abc, it bails because it says the scheme '' - # is bogus, when actually it should use the request's - # URL for the scheme. See Python issue #13696. - def http_error_302(self, req, fp, code, msg, headers): - # Some servers (incorrectly) return multiple Location headers - # (so probably same goes for URI). Use first header. - newurl = None - for key in ('location', 'uri'): - if key in headers: - newurl = headers[key] - break - if newurl is None: # pragma: no cover - return - urlparts = urlparse(newurl) - if urlparts.scheme == '': - newurl = urljoin(req.get_full_url(), newurl) - if hasattr(headers, 'replace_header'): - headers.replace_header(key, newurl) - else: - headers[key] = newurl - return BaseRedirectHandler.http_error_302(self, req, fp, code, msg, - headers) - - http_error_301 = http_error_303 = http_error_307 = http_error_302 - -class Locator(object): - """ - A base class for locators - things that locate distributions. - """ - source_extensions = ('.tar.gz', '.tar.bz2', '.tar', '.zip', '.tgz', '.tbz') - binary_extensions = ('.egg', '.exe', '.whl') - excluded_extensions = ('.pdf',) - - # A list of tags indicating which wheels you want to match. The default - # value of None matches against the tags compatible with the running - # Python. If you want to match other values, set wheel_tags on a locator - # instance to a list of tuples (pyver, abi, arch) which you want to match. - wheel_tags = None - - downloadable_extensions = source_extensions + ('.whl',) - - def __init__(self, scheme='default'): - """ - Initialise an instance. - :param scheme: Because locators look for most recent versions, they - need to know the version scheme to use. This specifies - the current PEP-recommended scheme - use ``'legacy'`` - if you need to support existing distributions on PyPI. - """ - self._cache = {} - self.scheme = scheme - # Because of bugs in some of the handlers on some of the platforms, - # we use our own opener rather than just using urlopen. - self.opener = build_opener(RedirectHandler()) - # If get_project() is called from locate(), the matcher instance - # is set from the requirement passed to locate(). See issue #18 for - # why this can be useful to know. - self.matcher = None - self.errors = queue.Queue() - - def get_errors(self): - """ - Return any errors which have occurred. - """ - result = [] - while not self.errors.empty(): # pragma: no cover - try: - e = self.errors.get(False) - result.append(e) - except self.errors.Empty: - continue - self.errors.task_done() - return result - - def clear_errors(self): - """ - Clear any errors which may have been logged. - """ - # Just get the errors and throw them away - self.get_errors() - - def clear_cache(self): - self._cache.clear() - - def _get_scheme(self): - return self._scheme - - def _set_scheme(self, value): - self._scheme = value - - scheme = property(_get_scheme, _set_scheme) - - def _get_project(self, name): - """ - For a given project, get a dictionary mapping available versions to Distribution - instances. - - This should be implemented in subclasses. - - If called from a locate() request, self.matcher will be set to a - matcher for the requirement to satisfy, otherwise it will be None. - """ - raise NotImplementedError('Please implement in the subclass') - - def get_distribution_names(self): - """ - Return all the distribution names known to this locator. - """ - raise NotImplementedError('Please implement in the subclass') - - def get_project(self, name): - """ - For a given project, get a dictionary mapping available versions to Distribution - instances. - - This calls _get_project to do all the work, and just implements a caching layer on top. - """ - if self._cache is None: # pragma: no cover - result = self._get_project(name) - elif name in self._cache: - result = self._cache[name] - else: - self.clear_errors() - result = self._get_project(name) - self._cache[name] = result - return result - - def score_url(self, url): - """ - Give an url a score which can be used to choose preferred URLs - for a given project release. - """ - t = urlparse(url) - basename = posixpath.basename(t.path) - compatible = True - is_wheel = basename.endswith('.whl') - is_downloadable = basename.endswith(self.downloadable_extensions) - if is_wheel: - compatible = is_compatible(Wheel(basename), self.wheel_tags) - return (t.scheme == 'https', 'pypi.org' in t.netloc, - is_downloadable, is_wheel, compatible, basename) - - def prefer_url(self, url1, url2): - """ - Choose one of two URLs where both are candidates for distribution - archives for the same version of a distribution (for example, - .tar.gz vs. zip). - - The current implementation favours https:// URLs over http://, archives - from PyPI over those from other locations, wheel compatibility (if a - wheel) and then the archive name. - """ - result = url2 - if url1: - s1 = self.score_url(url1) - s2 = self.score_url(url2) - if s1 > s2: - result = url1 - if result != url2: - logger.debug('Not replacing %r with %r', url1, url2) - else: - logger.debug('Replacing %r with %r', url1, url2) - return result - - def split_filename(self, filename, project_name): - """ - Attempt to split a filename in project name, version and Python version. - """ - return split_filename(filename, project_name) - - def convert_url_to_download_info(self, url, project_name): - """ - See if a URL is a candidate for a download URL for a project (the URL - has typically been scraped from an HTML page). - - If it is, a dictionary is returned with keys "name", "version", - "filename" and "url"; otherwise, None is returned. - """ - def same_project(name1, name2): - return normalize_name(name1) == normalize_name(name2) - - result = None - scheme, netloc, path, params, query, frag = urlparse(url) - if frag.lower().startswith('egg='): # pragma: no cover - logger.debug('%s: version hint in fragment: %r', - project_name, frag) - m = HASHER_HASH.match(frag) - if m: - algo, digest = m.groups() - else: - algo, digest = None, None - origpath = path - if path and path[-1] == '/': # pragma: no cover - path = path[:-1] - if path.endswith('.whl'): - try: - wheel = Wheel(path) - if not is_compatible(wheel, self.wheel_tags): - logger.debug('Wheel not compatible: %s', path) - else: - if project_name is None: - include = True - else: - include = same_project(wheel.name, project_name) - if include: - result = { - 'name': wheel.name, - 'version': wheel.version, - 'filename': wheel.filename, - 'url': urlunparse((scheme, netloc, origpath, - params, query, '')), - 'python-version': ', '.join( - ['.'.join(list(v[2:])) for v in wheel.pyver]), - } - except Exception as e: # pragma: no cover - logger.warning('invalid path for wheel: %s', path) - elif not path.endswith(self.downloadable_extensions): # pragma: no cover - logger.debug('Not downloadable: %s', path) - else: # downloadable extension - path = filename = posixpath.basename(path) - for ext in self.downloadable_extensions: - if path.endswith(ext): - path = path[:-len(ext)] - t = self.split_filename(path, project_name) - if not t: # pragma: no cover - logger.debug('No match for project/version: %s', path) - else: - name, version, pyver = t - if not project_name or same_project(project_name, name): - result = { - 'name': name, - 'version': version, - 'filename': filename, - 'url': urlunparse((scheme, netloc, origpath, - params, query, '')), - #'packagetype': 'sdist', - } - if pyver: # pragma: no cover - result['python-version'] = pyver - break - if result and algo: - result['%s_digest' % algo] = digest - return result - - def _get_digest(self, info): - """ - Get a digest from a dictionary by looking at a "digests" dictionary - or keys of the form 'algo_digest'. - - Returns a 2-tuple (algo, digest) if found, else None. Currently - looks only for SHA256, then MD5. - """ - result = None - if 'digests' in info: - digests = info['digests'] - for algo in ('sha256', 'md5'): - if algo in digests: - result = (algo, digests[algo]) - break - if not result: - for algo in ('sha256', 'md5'): - key = '%s_digest' % algo - if key in info: - result = (algo, info[key]) - break - return result - - def _update_version_data(self, result, info): - """ - Update a result dictionary (the final result from _get_project) with a - dictionary for a specific version, which typically holds information - gleaned from a filename or URL for an archive for the distribution. - """ - name = info.pop('name') - version = info.pop('version') - if version in result: - dist = result[version] - md = dist.metadata - else: - dist = make_dist(name, version, scheme=self.scheme) - md = dist.metadata - dist.digest = digest = self._get_digest(info) - url = info['url'] - result['digests'][url] = digest - if md.source_url != info['url']: - md.source_url = self.prefer_url(md.source_url, url) - result['urls'].setdefault(version, set()).add(url) - dist.locator = self - result[version] = dist - - def locate(self, requirement, prereleases=False): - """ - Find the most recent distribution which matches the given - requirement. - - :param requirement: A requirement of the form 'foo (1.0)' or perhaps - 'foo (>= 1.0, < 2.0, != 1.3)' - :param prereleases: If ``True``, allow pre-release versions - to be located. Otherwise, pre-release versions - are not returned. - :return: A :class:`Distribution` instance, or ``None`` if no such - distribution could be located. - """ - result = None - r = parse_requirement(requirement) - if r is None: # pragma: no cover - raise DistlibException('Not a valid requirement: %r' % requirement) - scheme = get_scheme(self.scheme) - self.matcher = matcher = scheme.matcher(r.requirement) - logger.debug('matcher: %s (%s)', matcher, type(matcher).__name__) - versions = self.get_project(r.name) - if len(versions) > 2: # urls and digests keys are present - # sometimes, versions are invalid - slist = [] - vcls = matcher.version_class - for k in versions: - if k in ('urls', 'digests'): - continue - try: - if not matcher.match(k): - pass # logger.debug('%s did not match %r', matcher, k) - else: - if prereleases or not vcls(k).is_prerelease: - slist.append(k) - # else: - # logger.debug('skipping pre-release ' - # 'version %s of %s', k, matcher.name) - except Exception: # pragma: no cover - logger.warning('error matching %s with %r', matcher, k) - pass # slist.append(k) - if len(slist) > 1: - slist = sorted(slist, key=scheme.key) - if slist: - logger.debug('sorted list: %s', slist) - version = slist[-1] - result = versions[version] - if result: - if r.extras: - result.extras = r.extras - result.download_urls = versions.get('urls', {}).get(version, set()) - d = {} - sd = versions.get('digests', {}) - for url in result.download_urls: - if url in sd: # pragma: no cover - d[url] = sd[url] - result.digests = d - self.matcher = None - return result - - -class PyPIRPCLocator(Locator): - """ - This locator uses XML-RPC to locate distributions. It therefore - cannot be used with simple mirrors (that only mirror file content). - """ - def __init__(self, url, **kwargs): - """ - Initialise an instance. - - :param url: The URL to use for XML-RPC. - :param kwargs: Passed to the superclass constructor. - """ - super(PyPIRPCLocator, self).__init__(**kwargs) - self.base_url = url - self.client = ServerProxy(url, timeout=3.0) - - def get_distribution_names(self): - """ - Return all the distribution names known to this locator. - """ - return set(self.client.list_packages()) - - def _get_project(self, name): - result = {'urls': {}, 'digests': {}} - versions = self.client.package_releases(name, True) - for v in versions: - urls = self.client.release_urls(name, v) - data = self.client.release_data(name, v) - metadata = Metadata(scheme=self.scheme) - metadata.name = data['name'] - metadata.version = data['version'] - metadata.license = data.get('license') - metadata.keywords = data.get('keywords', []) - metadata.summary = data.get('summary') - dist = Distribution(metadata) - if urls: - info = urls[0] - metadata.source_url = info['url'] - dist.digest = self._get_digest(info) - dist.locator = self - result[v] = dist - for info in urls: - url = info['url'] - digest = self._get_digest(info) - result['urls'].setdefault(v, set()).add(url) - result['digests'][url] = digest - return result - -class PyPIJSONLocator(Locator): - """ - This locator uses PyPI's JSON interface. It's very limited in functionality - and probably not worth using. - """ - def __init__(self, url, **kwargs): - super(PyPIJSONLocator, self).__init__(**kwargs) - self.base_url = ensure_slash(url) - - def get_distribution_names(self): - """ - Return all the distribution names known to this locator. - """ - raise NotImplementedError('Not available from this locator') - - def _get_project(self, name): - result = {'urls': {}, 'digests': {}} - url = urljoin(self.base_url, '%s/json' % quote(name)) - try: - resp = self.opener.open(url) - data = resp.read().decode() # for now - d = json.loads(data) - md = Metadata(scheme=self.scheme) - data = d['info'] - md.name = data['name'] - md.version = data['version'] - md.license = data.get('license') - md.keywords = data.get('keywords', []) - md.summary = data.get('summary') - dist = Distribution(md) - dist.locator = self - urls = d['urls'] - result[md.version] = dist - for info in d['urls']: - url = info['url'] - dist.download_urls.add(url) - dist.digests[url] = self._get_digest(info) - result['urls'].setdefault(md.version, set()).add(url) - result['digests'][url] = self._get_digest(info) - # Now get other releases - for version, infos in d['releases'].items(): - if version == md.version: - continue # already done - omd = Metadata(scheme=self.scheme) - omd.name = md.name - omd.version = version - odist = Distribution(omd) - odist.locator = self - result[version] = odist - for info in infos: - url = info['url'] - odist.download_urls.add(url) - odist.digests[url] = self._get_digest(info) - result['urls'].setdefault(version, set()).add(url) - result['digests'][url] = self._get_digest(info) -# for info in urls: -# md.source_url = info['url'] -# dist.digest = self._get_digest(info) -# dist.locator = self -# for info in urls: -# url = info['url'] -# result['urls'].setdefault(md.version, set()).add(url) -# result['digests'][url] = self._get_digest(info) - except Exception as e: - self.errors.put(text_type(e)) - logger.exception('JSON fetch failed: %s', e) - return result - - -class Page(object): - """ - This class represents a scraped HTML page. - """ - # The following slightly hairy-looking regex just looks for the contents of - # an anchor link, which has an attribute "href" either immediately preceded - # or immediately followed by a "rel" attribute. The attribute values can be - # declared with double quotes, single quotes or no quotes - which leads to - # the length of the expression. - _href = re.compile(""" -(rel\\s*=\\s*(?:"(?P[^"]*)"|'(?P[^']*)'|(?P[^>\\s\n]*))\\s+)? -href\\s*=\\s*(?:"(?P[^"]*)"|'(?P[^']*)'|(?P[^>\\s\n]*)) -(\\s+rel\\s*=\\s*(?:"(?P[^"]*)"|'(?P[^']*)'|(?P[^>\\s\n]*)))? -""", re.I | re.S | re.X) - _base = re.compile(r"""]+)""", re.I | re.S) - - def __init__(self, data, url): - """ - Initialise an instance with the Unicode page contents and the URL they - came from. - """ - self.data = data - self.base_url = self.url = url - m = self._base.search(self.data) - if m: - self.base_url = m.group(1) - - _clean_re = re.compile(r'[^a-z0-9$&+,/:;=?@.#%_\\|-]', re.I) - - @cached_property - def links(self): - """ - Return the URLs of all the links on a page together with information - about their "rel" attribute, for determining which ones to treat as - downloads and which ones to queue for further scraping. - """ - def clean(url): - "Tidy up an URL." - scheme, netloc, path, params, query, frag = urlparse(url) - return urlunparse((scheme, netloc, quote(path), - params, query, frag)) - - result = set() - for match in self._href.finditer(self.data): - d = match.groupdict('') - rel = (d['rel1'] or d['rel2'] or d['rel3'] or - d['rel4'] or d['rel5'] or d['rel6']) - url = d['url1'] or d['url2'] or d['url3'] - url = urljoin(self.base_url, url) - url = unescape(url) - url = self._clean_re.sub(lambda m: '%%%2x' % ord(m.group(0)), url) - result.add((url, rel)) - # We sort the result, hoping to bring the most recent versions - # to the front - result = sorted(result, key=lambda t: t[0], reverse=True) - return result - - -class SimpleScrapingLocator(Locator): - """ - A locator which scrapes HTML pages to locate downloads for a distribution. - This runs multiple threads to do the I/O; performance is at least as good - as pip's PackageFinder, which works in an analogous fashion. - """ - - # These are used to deal with various Content-Encoding schemes. - decoders = { - 'deflate': zlib.decompress, - 'gzip': lambda b: gzip.GzipFile(fileobj=BytesIO(b)).read(), - 'none': lambda b: b, - } - - def __init__(self, url, timeout=None, num_workers=10, **kwargs): - """ - Initialise an instance. - :param url: The root URL to use for scraping. - :param timeout: The timeout, in seconds, to be applied to requests. - This defaults to ``None`` (no timeout specified). - :param num_workers: The number of worker threads you want to do I/O, - This defaults to 10. - :param kwargs: Passed to the superclass. - """ - super(SimpleScrapingLocator, self).__init__(**kwargs) - self.base_url = ensure_slash(url) - self.timeout = timeout - self._page_cache = {} - self._seen = set() - self._to_fetch = queue.Queue() - self._bad_hosts = set() - self.skip_externals = False - self.num_workers = num_workers - self._lock = threading.RLock() - # See issue #45: we need to be resilient when the locator is used - # in a thread, e.g. with concurrent.futures. We can't use self._lock - # as it is for coordinating our internal threads - the ones created - # in _prepare_threads. - self._gplock = threading.RLock() - self.platform_check = False # See issue #112 - - def _prepare_threads(self): - """ - Threads are created only when get_project is called, and terminate - before it returns. They are there primarily to parallelise I/O (i.e. - fetching web pages). - """ - self._threads = [] - for i in range(self.num_workers): - t = threading.Thread(target=self._fetch) - t.daemon = True - t.start() - self._threads.append(t) - - def _wait_threads(self): - """ - Tell all the threads to terminate (by sending a sentinel value) and - wait for them to do so. - """ - # Note that you need two loops, since you can't say which - # thread will get each sentinel - for t in self._threads: - self._to_fetch.put(None) # sentinel - for t in self._threads: - t.join() - self._threads = [] - - def _get_project(self, name): - result = {'urls': {}, 'digests': {}} - with self._gplock: - self.result = result - self.project_name = name - url = urljoin(self.base_url, '%s/' % quote(name)) - self._seen.clear() - self._page_cache.clear() - self._prepare_threads() - try: - logger.debug('Queueing %s', url) - self._to_fetch.put(url) - self._to_fetch.join() - finally: - self._wait_threads() - del self.result - return result - - platform_dependent = re.compile(r'\b(linux_(i\d86|x86_64|arm\w+)|' - r'win(32|_amd64)|macosx_?\d+)\b', re.I) - - def _is_platform_dependent(self, url): - """ - Does an URL refer to a platform-specific download? - """ - return self.platform_dependent.search(url) - - def _process_download(self, url): - """ - See if an URL is a suitable download for a project. - - If it is, register information in the result dictionary (for - _get_project) about the specific version it's for. - - Note that the return value isn't actually used other than as a boolean - value. - """ - if self.platform_check and self._is_platform_dependent(url): - info = None - else: - info = self.convert_url_to_download_info(url, self.project_name) - logger.debug('process_download: %s -> %s', url, info) - if info: - with self._lock: # needed because self.result is shared - self._update_version_data(self.result, info) - return info - - def _should_queue(self, link, referrer, rel): - """ - Determine whether a link URL from a referring page and with a - particular "rel" attribute should be queued for scraping. - """ - scheme, netloc, path, _, _, _ = urlparse(link) - if path.endswith(self.source_extensions + self.binary_extensions + - self.excluded_extensions): - result = False - elif self.skip_externals and not link.startswith(self.base_url): - result = False - elif not referrer.startswith(self.base_url): - result = False - elif rel not in ('homepage', 'download'): - result = False - elif scheme not in ('http', 'https', 'ftp'): - result = False - elif self._is_platform_dependent(link): - result = False - else: - host = netloc.split(':', 1)[0] - if host.lower() == 'localhost': - result = False - else: - result = True - logger.debug('should_queue: %s (%s) from %s -> %s', link, rel, - referrer, result) - return result - - def _fetch(self): - """ - Get a URL to fetch from the work queue, get the HTML page, examine its - links for download candidates and candidates for further scraping. - - This is a handy method to run in a thread. - """ - while True: - url = self._to_fetch.get() - try: - if url: - page = self.get_page(url) - if page is None: # e.g. after an error - continue - for link, rel in page.links: - if link not in self._seen: - try: - self._seen.add(link) - if (not self._process_download(link) and - self._should_queue(link, url, rel)): - logger.debug('Queueing %s from %s', link, url) - self._to_fetch.put(link) - except MetadataInvalidError: # e.g. invalid versions - pass - except Exception as e: # pragma: no cover - self.errors.put(text_type(e)) - finally: - # always do this, to avoid hangs :-) - self._to_fetch.task_done() - if not url: - #logger.debug('Sentinel seen, quitting.') - break - - def get_page(self, url): - """ - Get the HTML for an URL, possibly from an in-memory cache. - - XXX TODO Note: this cache is never actually cleared. It's assumed that - the data won't get stale over the lifetime of a locator instance (not - necessarily true for the default_locator). - """ - # http://peak.telecommunity.com/DevCenter/EasyInstall#package-index-api - scheme, netloc, path, _, _, _ = urlparse(url) - if scheme == 'file' and os.path.isdir(url2pathname(path)): - url = urljoin(ensure_slash(url), 'index.html') - - if url in self._page_cache: - result = self._page_cache[url] - logger.debug('Returning %s from cache: %s', url, result) - else: - host = netloc.split(':', 1)[0] - result = None - if host in self._bad_hosts: - logger.debug('Skipping %s due to bad host %s', url, host) - else: - req = Request(url, headers={'Accept-encoding': 'identity'}) - try: - logger.debug('Fetching %s', url) - resp = self.opener.open(req, timeout=self.timeout) - logger.debug('Fetched %s', url) - headers = resp.info() - content_type = headers.get('Content-Type', '') - if HTML_CONTENT_TYPE.match(content_type): - final_url = resp.geturl() - data = resp.read() - encoding = headers.get('Content-Encoding') - if encoding: - decoder = self.decoders[encoding] # fail if not found - data = decoder(data) - encoding = 'utf-8' - m = CHARSET.search(content_type) - if m: - encoding = m.group(1) - try: - data = data.decode(encoding) - except UnicodeError: # pragma: no cover - data = data.decode('latin-1') # fallback - result = Page(data, final_url) - self._page_cache[final_url] = result - except HTTPError as e: - if e.code != 404: - logger.exception('Fetch failed: %s: %s', url, e) - except URLError as e: # pragma: no cover - logger.exception('Fetch failed: %s: %s', url, e) - with self._lock: - self._bad_hosts.add(host) - except Exception as e: # pragma: no cover - logger.exception('Fetch failed: %s: %s', url, e) - finally: - self._page_cache[url] = result # even if None (failure) - return result - - _distname_re = re.compile(']*>([^<]+)<') - - def get_distribution_names(self): - """ - Return all the distribution names known to this locator. - """ - result = set() - page = self.get_page(self.base_url) - if not page: - raise DistlibException('Unable to get %s' % self.base_url) - for match in self._distname_re.finditer(page.data): - result.add(match.group(1)) - return result - -class DirectoryLocator(Locator): - """ - This class locates distributions in a directory tree. - """ - - def __init__(self, path, **kwargs): - """ - Initialise an instance. - :param path: The root of the directory tree to search. - :param kwargs: Passed to the superclass constructor, - except for: - * recursive - if True (the default), subdirectories are - recursed into. If False, only the top-level directory - is searched, - """ - self.recursive = kwargs.pop('recursive', True) - super(DirectoryLocator, self).__init__(**kwargs) - path = os.path.abspath(path) - if not os.path.isdir(path): # pragma: no cover - raise DistlibException('Not a directory: %r' % path) - self.base_dir = path - - def should_include(self, filename, parent): - """ - Should a filename be considered as a candidate for a distribution - archive? As well as the filename, the directory which contains it - is provided, though not used by the current implementation. - """ - return filename.endswith(self.downloadable_extensions) - - def _get_project(self, name): - result = {'urls': {}, 'digests': {}} - for root, dirs, files in os.walk(self.base_dir): - for fn in files: - if self.should_include(fn, root): - fn = os.path.join(root, fn) - url = urlunparse(('file', '', - pathname2url(os.path.abspath(fn)), - '', '', '')) - info = self.convert_url_to_download_info(url, name) - if info: - self._update_version_data(result, info) - if not self.recursive: - break - return result - - def get_distribution_names(self): - """ - Return all the distribution names known to this locator. - """ - result = set() - for root, dirs, files in os.walk(self.base_dir): - for fn in files: - if self.should_include(fn, root): - fn = os.path.join(root, fn) - url = urlunparse(('file', '', - pathname2url(os.path.abspath(fn)), - '', '', '')) - info = self.convert_url_to_download_info(url, None) - if info: - result.add(info['name']) - if not self.recursive: - break - return result - -class JSONLocator(Locator): - """ - This locator uses special extended metadata (not available on PyPI) and is - the basis of performant dependency resolution in distlib. Other locators - require archive downloads before dependencies can be determined! As you - might imagine, that can be slow. - """ - def get_distribution_names(self): - """ - Return all the distribution names known to this locator. - """ - raise NotImplementedError('Not available from this locator') - - def _get_project(self, name): - result = {'urls': {}, 'digests': {}} - data = get_project_data(name) - if data: - for info in data.get('files', []): - if info['ptype'] != 'sdist' or info['pyversion'] != 'source': - continue - # We don't store summary in project metadata as it makes - # the data bigger for no benefit during dependency - # resolution - dist = make_dist(data['name'], info['version'], - summary=data.get('summary', - 'Placeholder for summary'), - scheme=self.scheme) - md = dist.metadata - md.source_url = info['url'] - # TODO SHA256 digest - if 'digest' in info and info['digest']: - dist.digest = ('md5', info['digest']) - md.dependencies = info.get('requirements', {}) - dist.exports = info.get('exports', {}) - result[dist.version] = dist - result['urls'].setdefault(dist.version, set()).add(info['url']) - return result - -class DistPathLocator(Locator): - """ - This locator finds installed distributions in a path. It can be useful for - adding to an :class:`AggregatingLocator`. - """ - def __init__(self, distpath, **kwargs): - """ - Initialise an instance. - - :param distpath: A :class:`DistributionPath` instance to search. - """ - super(DistPathLocator, self).__init__(**kwargs) - assert isinstance(distpath, DistributionPath) - self.distpath = distpath - - def _get_project(self, name): - dist = self.distpath.get_distribution(name) - if dist is None: - result = {'urls': {}, 'digests': {}} - else: - result = { - dist.version: dist, - 'urls': {dist.version: set([dist.source_url])}, - 'digests': {dist.version: set([None])} - } - return result - - -class AggregatingLocator(Locator): - """ - This class allows you to chain and/or merge a list of locators. - """ - def __init__(self, *locators, **kwargs): - """ - Initialise an instance. - - :param locators: The list of locators to search. - :param kwargs: Passed to the superclass constructor, - except for: - * merge - if False (the default), the first successful - search from any of the locators is returned. If True, - the results from all locators are merged (this can be - slow). - """ - self.merge = kwargs.pop('merge', False) - self.locators = locators - super(AggregatingLocator, self).__init__(**kwargs) - - def clear_cache(self): - super(AggregatingLocator, self).clear_cache() - for locator in self.locators: - locator.clear_cache() - - def _set_scheme(self, value): - self._scheme = value - for locator in self.locators: - locator.scheme = value - - scheme = property(Locator.scheme.fget, _set_scheme) - - def _get_project(self, name): - result = {} - for locator in self.locators: - d = locator.get_project(name) - if d: - if self.merge: - files = result.get('urls', {}) - digests = result.get('digests', {}) - # next line could overwrite result['urls'], result['digests'] - result.update(d) - df = result.get('urls') - if files and df: - for k, v in files.items(): - if k in df: - df[k] |= v - else: - df[k] = v - dd = result.get('digests') - if digests and dd: - dd.update(digests) - else: - # See issue #18. If any dists are found and we're looking - # for specific constraints, we only return something if - # a match is found. For example, if a DirectoryLocator - # returns just foo (1.0) while we're looking for - # foo (>= 2.0), we'll pretend there was nothing there so - # that subsequent locators can be queried. Otherwise we - # would just return foo (1.0) which would then lead to a - # failure to find foo (>= 2.0), because other locators - # weren't searched. Note that this only matters when - # merge=False. - if self.matcher is None: - found = True - else: - found = False - for k in d: - if self.matcher.match(k): - found = True - break - if found: - result = d - break - return result - - def get_distribution_names(self): - """ - Return all the distribution names known to this locator. - """ - result = set() - for locator in self.locators: - try: - result |= locator.get_distribution_names() - except NotImplementedError: - pass - return result - - -# We use a legacy scheme simply because most of the dists on PyPI use legacy -# versions which don't conform to PEP 440. -default_locator = AggregatingLocator( - # JSONLocator(), # don't use as PEP 426 is withdrawn - SimpleScrapingLocator('https://pypi.org/simple/', - timeout=3.0), - scheme='legacy') - -locate = default_locator.locate - - -class DependencyFinder(object): - """ - Locate dependencies for distributions. - """ - - def __init__(self, locator=None): - """ - Initialise an instance, using the specified locator - to locate distributions. - """ - self.locator = locator or default_locator - self.scheme = get_scheme(self.locator.scheme) - - def add_distribution(self, dist): - """ - Add a distribution to the finder. This will update internal information - about who provides what. - :param dist: The distribution to add. - """ - logger.debug('adding distribution %s', dist) - name = dist.key - self.dists_by_name[name] = dist - self.dists[(name, dist.version)] = dist - for p in dist.provides: - name, version = parse_name_and_version(p) - logger.debug('Add to provided: %s, %s, %s', name, version, dist) - self.provided.setdefault(name, set()).add((version, dist)) - - def remove_distribution(self, dist): - """ - Remove a distribution from the finder. This will update internal - information about who provides what. - :param dist: The distribution to remove. - """ - logger.debug('removing distribution %s', dist) - name = dist.key - del self.dists_by_name[name] - del self.dists[(name, dist.version)] - for p in dist.provides: - name, version = parse_name_and_version(p) - logger.debug('Remove from provided: %s, %s, %s', name, version, dist) - s = self.provided[name] - s.remove((version, dist)) - if not s: - del self.provided[name] - - def get_matcher(self, reqt): - """ - Get a version matcher for a requirement. - :param reqt: The requirement - :type reqt: str - :return: A version matcher (an instance of - :class:`distlib.version.Matcher`). - """ - try: - matcher = self.scheme.matcher(reqt) - except UnsupportedVersionError: # pragma: no cover - # XXX compat-mode if cannot read the version - name = reqt.split()[0] - matcher = self.scheme.matcher(name) - return matcher - - def find_providers(self, reqt): - """ - Find the distributions which can fulfill a requirement. - - :param reqt: The requirement. - :type reqt: str - :return: A set of distribution which can fulfill the requirement. - """ - matcher = self.get_matcher(reqt) - name = matcher.key # case-insensitive - result = set() - provided = self.provided - if name in provided: - for version, provider in provided[name]: - try: - match = matcher.match(version) - except UnsupportedVersionError: - match = False - - if match: - result.add(provider) - break - return result - - def try_to_replace(self, provider, other, problems): - """ - Attempt to replace one provider with another. This is typically used - when resolving dependencies from multiple sources, e.g. A requires - (B >= 1.0) while C requires (B >= 1.1). - - For successful replacement, ``provider`` must meet all the requirements - which ``other`` fulfills. - - :param provider: The provider we are trying to replace with. - :param other: The provider we're trying to replace. - :param problems: If False is returned, this will contain what - problems prevented replacement. This is currently - a tuple of the literal string 'cantreplace', - ``provider``, ``other`` and the set of requirements - that ``provider`` couldn't fulfill. - :return: True if we can replace ``other`` with ``provider``, else - False. - """ - rlist = self.reqts[other] - unmatched = set() - for s in rlist: - matcher = self.get_matcher(s) - if not matcher.match(provider.version): - unmatched.add(s) - if unmatched: - # can't replace other with provider - problems.add(('cantreplace', provider, other, - frozenset(unmatched))) - result = False - else: - # can replace other with provider - self.remove_distribution(other) - del self.reqts[other] - for s in rlist: - self.reqts.setdefault(provider, set()).add(s) - self.add_distribution(provider) - result = True - return result - - def find(self, requirement, meta_extras=None, prereleases=False): - """ - Find a distribution and all distributions it depends on. - - :param requirement: The requirement specifying the distribution to - find, or a Distribution instance. - :param meta_extras: A list of meta extras such as :test:, :build: and - so on. - :param prereleases: If ``True``, allow pre-release versions to be - returned - otherwise, don't return prereleases - unless they're all that's available. - - Return a set of :class:`Distribution` instances and a set of - problems. - - The distributions returned should be such that they have the - :attr:`required` attribute set to ``True`` if they were - from the ``requirement`` passed to ``find()``, and they have the - :attr:`build_time_dependency` attribute set to ``True`` unless they - are post-installation dependencies of the ``requirement``. - - The problems should be a tuple consisting of the string - ``'unsatisfied'`` and the requirement which couldn't be satisfied - by any distribution known to the locator. - """ - - self.provided = {} - self.dists = {} - self.dists_by_name = {} - self.reqts = {} - - meta_extras = set(meta_extras or []) - if ':*:' in meta_extras: - meta_extras.remove(':*:') - # :meta: and :run: are implicitly included - meta_extras |= set([':test:', ':build:', ':dev:']) - - if isinstance(requirement, Distribution): - dist = odist = requirement - logger.debug('passed %s as requirement', odist) - else: - dist = odist = self.locator.locate(requirement, - prereleases=prereleases) - if dist is None: - raise DistlibException('Unable to locate %r' % requirement) - logger.debug('located %s', odist) - dist.requested = True - problems = set() - todo = set([dist]) - install_dists = set([odist]) - while todo: - dist = todo.pop() - name = dist.key # case-insensitive - if name not in self.dists_by_name: - self.add_distribution(dist) - else: - #import pdb; pdb.set_trace() - other = self.dists_by_name[name] - if other != dist: - self.try_to_replace(dist, other, problems) - - ireqts = dist.run_requires | dist.meta_requires - sreqts = dist.build_requires - ereqts = set() - if meta_extras and dist in install_dists: - for key in ('test', 'build', 'dev'): - e = ':%s:' % key - if e in meta_extras: - ereqts |= getattr(dist, '%s_requires' % key) - all_reqts = ireqts | sreqts | ereqts - for r in all_reqts: - providers = self.find_providers(r) - if not providers: - logger.debug('No providers found for %r', r) - provider = self.locator.locate(r, prereleases=prereleases) - # If no provider is found and we didn't consider - # prereleases, consider them now. - if provider is None and not prereleases: - provider = self.locator.locate(r, prereleases=True) - if provider is None: - logger.debug('Cannot satisfy %r', r) - problems.add(('unsatisfied', r)) - else: - n, v = provider.key, provider.version - if (n, v) not in self.dists: - todo.add(provider) - providers.add(provider) - if r in ireqts and dist in install_dists: - install_dists.add(provider) - logger.debug('Adding %s to install_dists', - provider.name_and_version) - for p in providers: - name = p.key - if name not in self.dists_by_name: - self.reqts.setdefault(p, set()).add(r) - else: - other = self.dists_by_name[name] - if other != p: - # see if other can be replaced by p - self.try_to_replace(p, other, problems) - - dists = set(self.dists.values()) - for dist in dists: - dist.build_time_dependency = dist not in install_dists - if dist.build_time_dependency: - logger.debug('%s is a build-time dependency only.', - dist.name_and_version) - logger.debug('find done for %s', odist) - return dists, problems diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/distlib/manifest.py b/venv/lib/python3.11/site-packages/pip/_vendor/distlib/manifest.py deleted file mode 100644 index ca0fe44..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/distlib/manifest.py +++ /dev/null @@ -1,393 +0,0 @@ -# -*- coding: utf-8 -*- -# -# Copyright (C) 2012-2013 Python Software Foundation. -# See LICENSE.txt and CONTRIBUTORS.txt. -# -""" -Class representing the list of files in a distribution. - -Equivalent to distutils.filelist, but fixes some problems. -""" -import fnmatch -import logging -import os -import re -import sys - -from . import DistlibException -from .compat import fsdecode -from .util import convert_path - - -__all__ = ['Manifest'] - -logger = logging.getLogger(__name__) - -# a \ followed by some spaces + EOL -_COLLAPSE_PATTERN = re.compile('\\\\w*\n', re.M) -_COMMENTED_LINE = re.compile('#.*?(?=\n)|\n(?=$)', re.M | re.S) - -# -# Due to the different results returned by fnmatch.translate, we need -# to do slightly different processing for Python 2.7 and 3.2 ... this needed -# to be brought in for Python 3.6 onwards. -# -_PYTHON_VERSION = sys.version_info[:2] - -class Manifest(object): - """A list of files built by on exploring the filesystem and filtered by - applying various patterns to what we find there. - """ - - def __init__(self, base=None): - """ - Initialise an instance. - - :param base: The base directory to explore under. - """ - self.base = os.path.abspath(os.path.normpath(base or os.getcwd())) - self.prefix = self.base + os.sep - self.allfiles = None - self.files = set() - - # - # Public API - # - - def findall(self): - """Find all files under the base and set ``allfiles`` to the absolute - pathnames of files found. - """ - from stat import S_ISREG, S_ISDIR, S_ISLNK - - self.allfiles = allfiles = [] - root = self.base - stack = [root] - pop = stack.pop - push = stack.append - - while stack: - root = pop() - names = os.listdir(root) - - for name in names: - fullname = os.path.join(root, name) - - # Avoid excess stat calls -- just one will do, thank you! - stat = os.stat(fullname) - mode = stat.st_mode - if S_ISREG(mode): - allfiles.append(fsdecode(fullname)) - elif S_ISDIR(mode) and not S_ISLNK(mode): - push(fullname) - - def add(self, item): - """ - Add a file to the manifest. - - :param item: The pathname to add. This can be relative to the base. - """ - if not item.startswith(self.prefix): - item = os.path.join(self.base, item) - self.files.add(os.path.normpath(item)) - - def add_many(self, items): - """ - Add a list of files to the manifest. - - :param items: The pathnames to add. These can be relative to the base. - """ - for item in items: - self.add(item) - - def sorted(self, wantdirs=False): - """ - Return sorted files in directory order - """ - - def add_dir(dirs, d): - dirs.add(d) - logger.debug('add_dir added %s', d) - if d != self.base: - parent, _ = os.path.split(d) - assert parent not in ('', '/') - add_dir(dirs, parent) - - result = set(self.files) # make a copy! - if wantdirs: - dirs = set() - for f in result: - add_dir(dirs, os.path.dirname(f)) - result |= dirs - return [os.path.join(*path_tuple) for path_tuple in - sorted(os.path.split(path) for path in result)] - - def clear(self): - """Clear all collected files.""" - self.files = set() - self.allfiles = [] - - def process_directive(self, directive): - """ - Process a directive which either adds some files from ``allfiles`` to - ``files``, or removes some files from ``files``. - - :param directive: The directive to process. This should be in a format - compatible with distutils ``MANIFEST.in`` files: - - http://docs.python.org/distutils/sourcedist.html#commands - """ - # Parse the line: split it up, make sure the right number of words - # is there, and return the relevant words. 'action' is always - # defined: it's the first word of the line. Which of the other - # three are defined depends on the action; it'll be either - # patterns, (dir and patterns), or (dirpattern). - action, patterns, thedir, dirpattern = self._parse_directive(directive) - - # OK, now we know that the action is valid and we have the - # right number of words on the line for that action -- so we - # can proceed with minimal error-checking. - if action == 'include': - for pattern in patterns: - if not self._include_pattern(pattern, anchor=True): - logger.warning('no files found matching %r', pattern) - - elif action == 'exclude': - for pattern in patterns: - found = self._exclude_pattern(pattern, anchor=True) - #if not found: - # logger.warning('no previously-included files ' - # 'found matching %r', pattern) - - elif action == 'global-include': - for pattern in patterns: - if not self._include_pattern(pattern, anchor=False): - logger.warning('no files found matching %r ' - 'anywhere in distribution', pattern) - - elif action == 'global-exclude': - for pattern in patterns: - found = self._exclude_pattern(pattern, anchor=False) - #if not found: - # logger.warning('no previously-included files ' - # 'matching %r found anywhere in ' - # 'distribution', pattern) - - elif action == 'recursive-include': - for pattern in patterns: - if not self._include_pattern(pattern, prefix=thedir): - logger.warning('no files found matching %r ' - 'under directory %r', pattern, thedir) - - elif action == 'recursive-exclude': - for pattern in patterns: - found = self._exclude_pattern(pattern, prefix=thedir) - #if not found: - # logger.warning('no previously-included files ' - # 'matching %r found under directory %r', - # pattern, thedir) - - elif action == 'graft': - if not self._include_pattern(None, prefix=dirpattern): - logger.warning('no directories found matching %r', - dirpattern) - - elif action == 'prune': - if not self._exclude_pattern(None, prefix=dirpattern): - logger.warning('no previously-included directories found ' - 'matching %r', dirpattern) - else: # pragma: no cover - # This should never happen, as it should be caught in - # _parse_template_line - raise DistlibException( - 'invalid action %r' % action) - - # - # Private API - # - - def _parse_directive(self, directive): - """ - Validate a directive. - :param directive: The directive to validate. - :return: A tuple of action, patterns, thedir, dir_patterns - """ - words = directive.split() - if len(words) == 1 and words[0] not in ('include', 'exclude', - 'global-include', - 'global-exclude', - 'recursive-include', - 'recursive-exclude', - 'graft', 'prune'): - # no action given, let's use the default 'include' - words.insert(0, 'include') - - action = words[0] - patterns = thedir = dir_pattern = None - - if action in ('include', 'exclude', - 'global-include', 'global-exclude'): - if len(words) < 2: - raise DistlibException( - '%r expects ...' % action) - - patterns = [convert_path(word) for word in words[1:]] - - elif action in ('recursive-include', 'recursive-exclude'): - if len(words) < 3: - raise DistlibException( - '%r expects ...' % action) - - thedir = convert_path(words[1]) - patterns = [convert_path(word) for word in words[2:]] - - elif action in ('graft', 'prune'): - if len(words) != 2: - raise DistlibException( - '%r expects a single ' % action) - - dir_pattern = convert_path(words[1]) - - else: - raise DistlibException('unknown action %r' % action) - - return action, patterns, thedir, dir_pattern - - def _include_pattern(self, pattern, anchor=True, prefix=None, - is_regex=False): - """Select strings (presumably filenames) from 'self.files' that - match 'pattern', a Unix-style wildcard (glob) pattern. - - Patterns are not quite the same as implemented by the 'fnmatch' - module: '*' and '?' match non-special characters, where "special" - is platform-dependent: slash on Unix; colon, slash, and backslash on - DOS/Windows; and colon on Mac OS. - - If 'anchor' is true (the default), then the pattern match is more - stringent: "*.py" will match "foo.py" but not "foo/bar.py". If - 'anchor' is false, both of these will match. - - If 'prefix' is supplied, then only filenames starting with 'prefix' - (itself a pattern) and ending with 'pattern', with anything in between - them, will match. 'anchor' is ignored in this case. - - If 'is_regex' is true, 'anchor' and 'prefix' are ignored, and - 'pattern' is assumed to be either a string containing a regex or a - regex object -- no translation is done, the regex is just compiled - and used as-is. - - Selected strings will be added to self.files. - - Return True if files are found. - """ - # XXX docstring lying about what the special chars are? - found = False - pattern_re = self._translate_pattern(pattern, anchor, prefix, is_regex) - - # delayed loading of allfiles list - if self.allfiles is None: - self.findall() - - for name in self.allfiles: - if pattern_re.search(name): - self.files.add(name) - found = True - return found - - def _exclude_pattern(self, pattern, anchor=True, prefix=None, - is_regex=False): - """Remove strings (presumably filenames) from 'files' that match - 'pattern'. - - Other parameters are the same as for 'include_pattern()', above. - The list 'self.files' is modified in place. Return True if files are - found. - - This API is public to allow e.g. exclusion of SCM subdirs, e.g. when - packaging source distributions - """ - found = False - pattern_re = self._translate_pattern(pattern, anchor, prefix, is_regex) - for f in list(self.files): - if pattern_re.search(f): - self.files.remove(f) - found = True - return found - - def _translate_pattern(self, pattern, anchor=True, prefix=None, - is_regex=False): - """Translate a shell-like wildcard pattern to a compiled regular - expression. - - Return the compiled regex. If 'is_regex' true, - then 'pattern' is directly compiled to a regex (if it's a string) - or just returned as-is (assumes it's a regex object). - """ - if is_regex: - if isinstance(pattern, str): - return re.compile(pattern) - else: - return pattern - - if _PYTHON_VERSION > (3, 2): - # ditch start and end characters - start, _, end = self._glob_to_re('_').partition('_') - - if pattern: - pattern_re = self._glob_to_re(pattern) - if _PYTHON_VERSION > (3, 2): - assert pattern_re.startswith(start) and pattern_re.endswith(end) - else: - pattern_re = '' - - base = re.escape(os.path.join(self.base, '')) - if prefix is not None: - # ditch end of pattern character - if _PYTHON_VERSION <= (3, 2): - empty_pattern = self._glob_to_re('') - prefix_re = self._glob_to_re(prefix)[:-len(empty_pattern)] - else: - prefix_re = self._glob_to_re(prefix) - assert prefix_re.startswith(start) and prefix_re.endswith(end) - prefix_re = prefix_re[len(start): len(prefix_re) - len(end)] - sep = os.sep - if os.sep == '\\': - sep = r'\\' - if _PYTHON_VERSION <= (3, 2): - pattern_re = '^' + base + sep.join((prefix_re, - '.*' + pattern_re)) - else: - pattern_re = pattern_re[len(start): len(pattern_re) - len(end)] - pattern_re = r'%s%s%s%s.*%s%s' % (start, base, prefix_re, sep, - pattern_re, end) - else: # no prefix -- respect anchor flag - if anchor: - if _PYTHON_VERSION <= (3, 2): - pattern_re = '^' + base + pattern_re - else: - pattern_re = r'%s%s%s' % (start, base, pattern_re[len(start):]) - - return re.compile(pattern_re) - - def _glob_to_re(self, pattern): - """Translate a shell-like glob pattern to a regular expression. - - Return a string containing the regex. Differs from - 'fnmatch.translate()' in that '*' does not match "special characters" - (which are platform-specific). - """ - pattern_re = fnmatch.translate(pattern) - - # '?' and '*' in the glob pattern become '.' and '.*' in the RE, which - # IMHO is wrong -- '?' and '*' aren't supposed to match slash in Unix, - # and by extension they shouldn't match such "special characters" under - # any OS. So change all non-escaped dots in the RE to match any - # character except the special characters (currently: just os.sep). - sep = os.sep - if os.sep == '\\': - # we're using a regex to manipulate a regex, so we need - # to escape the backslash twice - sep = r'\\\\' - escaped = r'\1[^%s]' % sep - pattern_re = re.sub(r'((? y, - '!=': lambda x, y: x != y, - '<': lambda x, y: x < y, - '<=': lambda x, y: x == y or x < y, - '>': lambda x, y: x > y, - '>=': lambda x, y: x == y or x > y, - 'and': lambda x, y: x and y, - 'or': lambda x, y: x or y, - 'in': lambda x, y: x in y, - 'not in': lambda x, y: x not in y, - } - - def evaluate(self, expr, context): - """ - Evaluate a marker expression returned by the :func:`parse_requirement` - function in the specified context. - """ - if isinstance(expr, string_types): - if expr[0] in '\'"': - result = expr[1:-1] - else: - if expr not in context: - raise SyntaxError('unknown variable: %s' % expr) - result = context[expr] - else: - assert isinstance(expr, dict) - op = expr['op'] - if op not in self.operations: - raise NotImplementedError('op not implemented: %s' % op) - elhs = expr['lhs'] - erhs = expr['rhs'] - if _is_literal(expr['lhs']) and _is_literal(expr['rhs']): - raise SyntaxError('invalid comparison: %s %s %s' % (elhs, op, erhs)) - - lhs = self.evaluate(elhs, context) - rhs = self.evaluate(erhs, context) - if ((elhs == 'python_version' or erhs == 'python_version') and - op in ('<', '<=', '>', '>=', '===', '==', '!=', '~=')): - lhs = NV(lhs) - rhs = NV(rhs) - elif elhs == 'python_version' and op in ('in', 'not in'): - lhs = NV(lhs) - rhs = _get_versions(rhs) - result = self.operations[op](lhs, rhs) - return result - -_DIGITS = re.compile(r'\d+\.\d+') - -def default_context(): - def format_full_version(info): - version = '%s.%s.%s' % (info.major, info.minor, info.micro) - kind = info.releaselevel - if kind != 'final': - version += kind[0] + str(info.serial) - return version - - if hasattr(sys, 'implementation'): - implementation_version = format_full_version(sys.implementation.version) - implementation_name = sys.implementation.name - else: - implementation_version = '0' - implementation_name = '' - - ppv = platform.python_version() - m = _DIGITS.match(ppv) - pv = m.group(0) - result = { - 'implementation_name': implementation_name, - 'implementation_version': implementation_version, - 'os_name': os.name, - 'platform_machine': platform.machine(), - 'platform_python_implementation': platform.python_implementation(), - 'platform_release': platform.release(), - 'platform_system': platform.system(), - 'platform_version': platform.version(), - 'platform_in_venv': str(in_venv()), - 'python_full_version': ppv, - 'python_version': pv, - 'sys_platform': sys.platform, - } - return result - -DEFAULT_CONTEXT = default_context() -del default_context - -evaluator = Evaluator() - -def interpret(marker, execution_context=None): - """ - Interpret a marker and return a result depending on environment. - - :param marker: The marker to interpret. - :type marker: str - :param execution_context: The context used for name lookup. - :type execution_context: mapping - """ - try: - expr, rest = parse_marker(marker) - except Exception as e: - raise SyntaxError('Unable to interpret marker syntax: %s: %s' % (marker, e)) - if rest and rest[0] != '#': - raise SyntaxError('unexpected trailing data in marker: %s: %s' % (marker, rest)) - context = dict(DEFAULT_CONTEXT) - if execution_context: - context.update(execution_context) - return evaluator.evaluate(expr, context) diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/distlib/metadata.py b/venv/lib/python3.11/site-packages/pip/_vendor/distlib/metadata.py deleted file mode 100644 index c329e19..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/distlib/metadata.py +++ /dev/null @@ -1,1076 +0,0 @@ -# -*- coding: utf-8 -*- -# -# Copyright (C) 2012 The Python Software Foundation. -# See LICENSE.txt and CONTRIBUTORS.txt. -# -"""Implementation of the Metadata for Python packages PEPs. - -Supports all metadata formats (1.0, 1.1, 1.2, 1.3/2.1 and 2.2). -""" -from __future__ import unicode_literals - -import codecs -from email import message_from_file -import json -import logging -import re - - -from . import DistlibException, __version__ -from .compat import StringIO, string_types, text_type -from .markers import interpret -from .util import extract_by_key, get_extras -from .version import get_scheme, PEP440_VERSION_RE - -logger = logging.getLogger(__name__) - - -class MetadataMissingError(DistlibException): - """A required metadata is missing""" - - -class MetadataConflictError(DistlibException): - """Attempt to read or write metadata fields that are conflictual.""" - - -class MetadataUnrecognizedVersionError(DistlibException): - """Unknown metadata version number.""" - - -class MetadataInvalidError(DistlibException): - """A metadata value is invalid""" - -# public API of this module -__all__ = ['Metadata', 'PKG_INFO_ENCODING', 'PKG_INFO_PREFERRED_VERSION'] - -# Encoding used for the PKG-INFO files -PKG_INFO_ENCODING = 'utf-8' - -# preferred version. Hopefully will be changed -# to 1.2 once PEP 345 is supported everywhere -PKG_INFO_PREFERRED_VERSION = '1.1' - -_LINE_PREFIX_1_2 = re.compile('\n \\|') -_LINE_PREFIX_PRE_1_2 = re.compile('\n ') -_241_FIELDS = ('Metadata-Version', 'Name', 'Version', 'Platform', - 'Summary', 'Description', - 'Keywords', 'Home-page', 'Author', 'Author-email', - 'License') - -_314_FIELDS = ('Metadata-Version', 'Name', 'Version', 'Platform', - 'Supported-Platform', 'Summary', 'Description', - 'Keywords', 'Home-page', 'Author', 'Author-email', - 'License', 'Classifier', 'Download-URL', 'Obsoletes', - 'Provides', 'Requires') - -_314_MARKERS = ('Obsoletes', 'Provides', 'Requires', 'Classifier', - 'Download-URL') - -_345_FIELDS = ('Metadata-Version', 'Name', 'Version', 'Platform', - 'Supported-Platform', 'Summary', 'Description', - 'Keywords', 'Home-page', 'Author', 'Author-email', - 'Maintainer', 'Maintainer-email', 'License', - 'Classifier', 'Download-URL', 'Obsoletes-Dist', - 'Project-URL', 'Provides-Dist', 'Requires-Dist', - 'Requires-Python', 'Requires-External') - -_345_MARKERS = ('Provides-Dist', 'Requires-Dist', 'Requires-Python', - 'Obsoletes-Dist', 'Requires-External', 'Maintainer', - 'Maintainer-email', 'Project-URL') - -_426_FIELDS = ('Metadata-Version', 'Name', 'Version', 'Platform', - 'Supported-Platform', 'Summary', 'Description', - 'Keywords', 'Home-page', 'Author', 'Author-email', - 'Maintainer', 'Maintainer-email', 'License', - 'Classifier', 'Download-URL', 'Obsoletes-Dist', - 'Project-URL', 'Provides-Dist', 'Requires-Dist', - 'Requires-Python', 'Requires-External', 'Private-Version', - 'Obsoleted-By', 'Setup-Requires-Dist', 'Extension', - 'Provides-Extra') - -_426_MARKERS = ('Private-Version', 'Provides-Extra', 'Obsoleted-By', - 'Setup-Requires-Dist', 'Extension') - -# See issue #106: Sometimes 'Requires' and 'Provides' occur wrongly in -# the metadata. Include them in the tuple literal below to allow them -# (for now). -# Ditto for Obsoletes - see issue #140. -_566_FIELDS = _426_FIELDS + ('Description-Content-Type', - 'Requires', 'Provides', 'Obsoletes') - -_566_MARKERS = ('Description-Content-Type',) - -_643_MARKERS = ('Dynamic', 'License-File') - -_643_FIELDS = _566_FIELDS + _643_MARKERS - -_ALL_FIELDS = set() -_ALL_FIELDS.update(_241_FIELDS) -_ALL_FIELDS.update(_314_FIELDS) -_ALL_FIELDS.update(_345_FIELDS) -_ALL_FIELDS.update(_426_FIELDS) -_ALL_FIELDS.update(_566_FIELDS) -_ALL_FIELDS.update(_643_FIELDS) - -EXTRA_RE = re.compile(r'''extra\s*==\s*("([^"]+)"|'([^']+)')''') - - -def _version2fieldlist(version): - if version == '1.0': - return _241_FIELDS - elif version == '1.1': - return _314_FIELDS - elif version == '1.2': - return _345_FIELDS - elif version in ('1.3', '2.1'): - # avoid adding field names if already there - return _345_FIELDS + tuple(f for f in _566_FIELDS if f not in _345_FIELDS) - elif version == '2.0': - raise ValueError('Metadata 2.0 is withdrawn and not supported') - # return _426_FIELDS - elif version == '2.2': - return _643_FIELDS - raise MetadataUnrecognizedVersionError(version) - - -def _best_version(fields): - """Detect the best version depending on the fields used.""" - def _has_marker(keys, markers): - for marker in markers: - if marker in keys: - return True - return False - - keys = [] - for key, value in fields.items(): - if value in ([], 'UNKNOWN', None): - continue - keys.append(key) - - possible_versions = ['1.0', '1.1', '1.2', '1.3', '2.1', '2.2'] # 2.0 removed - - # first let's try to see if a field is not part of one of the version - for key in keys: - if key not in _241_FIELDS and '1.0' in possible_versions: - possible_versions.remove('1.0') - logger.debug('Removed 1.0 due to %s', key) - if key not in _314_FIELDS and '1.1' in possible_versions: - possible_versions.remove('1.1') - logger.debug('Removed 1.1 due to %s', key) - if key not in _345_FIELDS and '1.2' in possible_versions: - possible_versions.remove('1.2') - logger.debug('Removed 1.2 due to %s', key) - if key not in _566_FIELDS and '1.3' in possible_versions: - possible_versions.remove('1.3') - logger.debug('Removed 1.3 due to %s', key) - if key not in _566_FIELDS and '2.1' in possible_versions: - if key != 'Description': # In 2.1, description allowed after headers - possible_versions.remove('2.1') - logger.debug('Removed 2.1 due to %s', key) - if key not in _643_FIELDS and '2.2' in possible_versions: - possible_versions.remove('2.2') - logger.debug('Removed 2.2 due to %s', key) - # if key not in _426_FIELDS and '2.0' in possible_versions: - # possible_versions.remove('2.0') - # logger.debug('Removed 2.0 due to %s', key) - - # possible_version contains qualified versions - if len(possible_versions) == 1: - return possible_versions[0] # found ! - elif len(possible_versions) == 0: - logger.debug('Out of options - unknown metadata set: %s', fields) - raise MetadataConflictError('Unknown metadata set') - - # let's see if one unique marker is found - is_1_1 = '1.1' in possible_versions and _has_marker(keys, _314_MARKERS) - is_1_2 = '1.2' in possible_versions and _has_marker(keys, _345_MARKERS) - is_2_1 = '2.1' in possible_versions and _has_marker(keys, _566_MARKERS) - # is_2_0 = '2.0' in possible_versions and _has_marker(keys, _426_MARKERS) - is_2_2 = '2.2' in possible_versions and _has_marker(keys, _643_MARKERS) - if int(is_1_1) + int(is_1_2) + int(is_2_1) + int(is_2_2) > 1: - raise MetadataConflictError('You used incompatible 1.1/1.2/2.1/2.2 fields') - - # we have the choice, 1.0, or 1.2, 2.1 or 2.2 - # - 1.0 has a broken Summary field but works with all tools - # - 1.1 is to avoid - # - 1.2 fixes Summary but has little adoption - # - 2.1 adds more features - # - 2.2 is the latest - if not is_1_1 and not is_1_2 and not is_2_1 and not is_2_2: - # we couldn't find any specific marker - if PKG_INFO_PREFERRED_VERSION in possible_versions: - return PKG_INFO_PREFERRED_VERSION - if is_1_1: - return '1.1' - if is_1_2: - return '1.2' - if is_2_1: - return '2.1' - # if is_2_2: - # return '2.2' - - return '2.2' - -# This follows the rules about transforming keys as described in -# https://www.python.org/dev/peps/pep-0566/#id17 -_ATTR2FIELD = { - name.lower().replace("-", "_"): name for name in _ALL_FIELDS -} -_FIELD2ATTR = {field: attr for attr, field in _ATTR2FIELD.items()} - -_PREDICATE_FIELDS = ('Requires-Dist', 'Obsoletes-Dist', 'Provides-Dist') -_VERSIONS_FIELDS = ('Requires-Python',) -_VERSION_FIELDS = ('Version',) -_LISTFIELDS = ('Platform', 'Classifier', 'Obsoletes', - 'Requires', 'Provides', 'Obsoletes-Dist', - 'Provides-Dist', 'Requires-Dist', 'Requires-External', - 'Project-URL', 'Supported-Platform', 'Setup-Requires-Dist', - 'Provides-Extra', 'Extension', 'License-File') -_LISTTUPLEFIELDS = ('Project-URL',) - -_ELEMENTSFIELD = ('Keywords',) - -_UNICODEFIELDS = ('Author', 'Maintainer', 'Summary', 'Description') - -_MISSING = object() - -_FILESAFE = re.compile('[^A-Za-z0-9.]+') - - -def _get_name_and_version(name, version, for_filename=False): - """Return the distribution name with version. - - If for_filename is true, return a filename-escaped form.""" - if for_filename: - # For both name and version any runs of non-alphanumeric or '.' - # characters are replaced with a single '-'. Additionally any - # spaces in the version string become '.' - name = _FILESAFE.sub('-', name) - version = _FILESAFE.sub('-', version.replace(' ', '.')) - return '%s-%s' % (name, version) - - -class LegacyMetadata(object): - """The legacy metadata of a release. - - Supports versions 1.0, 1.1, 1.2, 2.0 and 1.3/2.1 (auto-detected). You can - instantiate the class with one of these arguments (or none): - - *path*, the path to a metadata file - - *fileobj* give a file-like object with metadata as content - - *mapping* is a dict-like object - - *scheme* is a version scheme name - """ - # TODO document the mapping API and UNKNOWN default key - - def __init__(self, path=None, fileobj=None, mapping=None, - scheme='default'): - if [path, fileobj, mapping].count(None) < 2: - raise TypeError('path, fileobj and mapping are exclusive') - self._fields = {} - self.requires_files = [] - self._dependencies = None - self.scheme = scheme - if path is not None: - self.read(path) - elif fileobj is not None: - self.read_file(fileobj) - elif mapping is not None: - self.update(mapping) - self.set_metadata_version() - - def set_metadata_version(self): - self._fields['Metadata-Version'] = _best_version(self._fields) - - def _write_field(self, fileobj, name, value): - fileobj.write('%s: %s\n' % (name, value)) - - def __getitem__(self, name): - return self.get(name) - - def __setitem__(self, name, value): - return self.set(name, value) - - def __delitem__(self, name): - field_name = self._convert_name(name) - try: - del self._fields[field_name] - except KeyError: - raise KeyError(name) - - def __contains__(self, name): - return (name in self._fields or - self._convert_name(name) in self._fields) - - def _convert_name(self, name): - if name in _ALL_FIELDS: - return name - name = name.replace('-', '_').lower() - return _ATTR2FIELD.get(name, name) - - def _default_value(self, name): - if name in _LISTFIELDS or name in _ELEMENTSFIELD: - return [] - return 'UNKNOWN' - - def _remove_line_prefix(self, value): - if self.metadata_version in ('1.0', '1.1'): - return _LINE_PREFIX_PRE_1_2.sub('\n', value) - else: - return _LINE_PREFIX_1_2.sub('\n', value) - - def __getattr__(self, name): - if name in _ATTR2FIELD: - return self[name] - raise AttributeError(name) - - # - # Public API - # - -# dependencies = property(_get_dependencies, _set_dependencies) - - def get_fullname(self, filesafe=False): - """Return the distribution name with version. - - If filesafe is true, return a filename-escaped form.""" - return _get_name_and_version(self['Name'], self['Version'], filesafe) - - def is_field(self, name): - """return True if name is a valid metadata key""" - name = self._convert_name(name) - return name in _ALL_FIELDS - - def is_multi_field(self, name): - name = self._convert_name(name) - return name in _LISTFIELDS - - def read(self, filepath): - """Read the metadata values from a file path.""" - fp = codecs.open(filepath, 'r', encoding='utf-8') - try: - self.read_file(fp) - finally: - fp.close() - - def read_file(self, fileob): - """Read the metadata values from a file object.""" - msg = message_from_file(fileob) - self._fields['Metadata-Version'] = msg['metadata-version'] - - # When reading, get all the fields we can - for field in _ALL_FIELDS: - if field not in msg: - continue - if field in _LISTFIELDS: - # we can have multiple lines - values = msg.get_all(field) - if field in _LISTTUPLEFIELDS and values is not None: - values = [tuple(value.split(',')) for value in values] - self.set(field, values) - else: - # single line - value = msg[field] - if value is not None and value != 'UNKNOWN': - self.set(field, value) - - # PEP 566 specifies that the body be used for the description, if - # available - body = msg.get_payload() - self["Description"] = body if body else self["Description"] - # logger.debug('Attempting to set metadata for %s', self) - # self.set_metadata_version() - - def write(self, filepath, skip_unknown=False): - """Write the metadata fields to filepath.""" - fp = codecs.open(filepath, 'w', encoding='utf-8') - try: - self.write_file(fp, skip_unknown) - finally: - fp.close() - - def write_file(self, fileobject, skip_unknown=False): - """Write the PKG-INFO format data to a file object.""" - self.set_metadata_version() - - for field in _version2fieldlist(self['Metadata-Version']): - values = self.get(field) - if skip_unknown and values in ('UNKNOWN', [], ['UNKNOWN']): - continue - if field in _ELEMENTSFIELD: - self._write_field(fileobject, field, ','.join(values)) - continue - if field not in _LISTFIELDS: - if field == 'Description': - if self.metadata_version in ('1.0', '1.1'): - values = values.replace('\n', '\n ') - else: - values = values.replace('\n', '\n |') - values = [values] - - if field in _LISTTUPLEFIELDS: - values = [','.join(value) for value in values] - - for value in values: - self._write_field(fileobject, field, value) - - def update(self, other=None, **kwargs): - """Set metadata values from the given iterable `other` and kwargs. - - Behavior is like `dict.update`: If `other` has a ``keys`` method, - they are looped over and ``self[key]`` is assigned ``other[key]``. - Else, ``other`` is an iterable of ``(key, value)`` iterables. - - Keys that don't match a metadata field or that have an empty value are - dropped. - """ - def _set(key, value): - if key in _ATTR2FIELD and value: - self.set(self._convert_name(key), value) - - if not other: - # other is None or empty container - pass - elif hasattr(other, 'keys'): - for k in other.keys(): - _set(k, other[k]) - else: - for k, v in other: - _set(k, v) - - if kwargs: - for k, v in kwargs.items(): - _set(k, v) - - def set(self, name, value): - """Control then set a metadata field.""" - name = self._convert_name(name) - - if ((name in _ELEMENTSFIELD or name == 'Platform') and - not isinstance(value, (list, tuple))): - if isinstance(value, string_types): - value = [v.strip() for v in value.split(',')] - else: - value = [] - elif (name in _LISTFIELDS and - not isinstance(value, (list, tuple))): - if isinstance(value, string_types): - value = [value] - else: - value = [] - - if logger.isEnabledFor(logging.WARNING): - project_name = self['Name'] - - scheme = get_scheme(self.scheme) - if name in _PREDICATE_FIELDS and value is not None: - for v in value: - # check that the values are valid - if not scheme.is_valid_matcher(v.split(';')[0]): - logger.warning( - "'%s': '%s' is not valid (field '%s')", - project_name, v, name) - # FIXME this rejects UNKNOWN, is that right? - elif name in _VERSIONS_FIELDS and value is not None: - if not scheme.is_valid_constraint_list(value): - logger.warning("'%s': '%s' is not a valid version (field '%s')", - project_name, value, name) - elif name in _VERSION_FIELDS and value is not None: - if not scheme.is_valid_version(value): - logger.warning("'%s': '%s' is not a valid version (field '%s')", - project_name, value, name) - - if name in _UNICODEFIELDS: - if name == 'Description': - value = self._remove_line_prefix(value) - - self._fields[name] = value - - def get(self, name, default=_MISSING): - """Get a metadata field.""" - name = self._convert_name(name) - if name not in self._fields: - if default is _MISSING: - default = self._default_value(name) - return default - if name in _UNICODEFIELDS: - value = self._fields[name] - return value - elif name in _LISTFIELDS: - value = self._fields[name] - if value is None: - return [] - res = [] - for val in value: - if name not in _LISTTUPLEFIELDS: - res.append(val) - else: - # That's for Project-URL - res.append((val[0], val[1])) - return res - - elif name in _ELEMENTSFIELD: - value = self._fields[name] - if isinstance(value, string_types): - return value.split(',') - return self._fields[name] - - def check(self, strict=False): - """Check if the metadata is compliant. If strict is True then raise if - no Name or Version are provided""" - self.set_metadata_version() - - # XXX should check the versions (if the file was loaded) - missing, warnings = [], [] - - for attr in ('Name', 'Version'): # required by PEP 345 - if attr not in self: - missing.append(attr) - - if strict and missing != []: - msg = 'missing required metadata: %s' % ', '.join(missing) - raise MetadataMissingError(msg) - - for attr in ('Home-page', 'Author'): - if attr not in self: - missing.append(attr) - - # checking metadata 1.2 (XXX needs to check 1.1, 1.0) - if self['Metadata-Version'] != '1.2': - return missing, warnings - - scheme = get_scheme(self.scheme) - - def are_valid_constraints(value): - for v in value: - if not scheme.is_valid_matcher(v.split(';')[0]): - return False - return True - - for fields, controller in ((_PREDICATE_FIELDS, are_valid_constraints), - (_VERSIONS_FIELDS, - scheme.is_valid_constraint_list), - (_VERSION_FIELDS, - scheme.is_valid_version)): - for field in fields: - value = self.get(field, None) - if value is not None and not controller(value): - warnings.append("Wrong value for '%s': %s" % (field, value)) - - return missing, warnings - - def todict(self, skip_missing=False): - """Return fields as a dict. - - Field names will be converted to use the underscore-lowercase style - instead of hyphen-mixed case (i.e. home_page instead of Home-page). - This is as per https://www.python.org/dev/peps/pep-0566/#id17. - """ - self.set_metadata_version() - - fields = _version2fieldlist(self['Metadata-Version']) - - data = {} - - for field_name in fields: - if not skip_missing or field_name in self._fields: - key = _FIELD2ATTR[field_name] - if key != 'project_url': - data[key] = self[field_name] - else: - data[key] = [','.join(u) for u in self[field_name]] - - return data - - def add_requirements(self, requirements): - if self['Metadata-Version'] == '1.1': - # we can't have 1.1 metadata *and* Setuptools requires - for field in ('Obsoletes', 'Requires', 'Provides'): - if field in self: - del self[field] - self['Requires-Dist'] += requirements - - # Mapping API - # TODO could add iter* variants - - def keys(self): - return list(_version2fieldlist(self['Metadata-Version'])) - - def __iter__(self): - for key in self.keys(): - yield key - - def values(self): - return [self[key] for key in self.keys()] - - def items(self): - return [(key, self[key]) for key in self.keys()] - - def __repr__(self): - return '<%s %s %s>' % (self.__class__.__name__, self.name, - self.version) - - -METADATA_FILENAME = 'pydist.json' -WHEEL_METADATA_FILENAME = 'metadata.json' -LEGACY_METADATA_FILENAME = 'METADATA' - - -class Metadata(object): - """ - The metadata of a release. This implementation uses 2.1 - metadata where possible. If not possible, it wraps a LegacyMetadata - instance which handles the key-value metadata format. - """ - - METADATA_VERSION_MATCHER = re.compile(r'^\d+(\.\d+)*$') - - NAME_MATCHER = re.compile('^[0-9A-Z]([0-9A-Z_.-]*[0-9A-Z])?$', re.I) - - FIELDNAME_MATCHER = re.compile('^[A-Z]([0-9A-Z-]*[0-9A-Z])?$', re.I) - - VERSION_MATCHER = PEP440_VERSION_RE - - SUMMARY_MATCHER = re.compile('.{1,2047}') - - METADATA_VERSION = '2.0' - - GENERATOR = 'distlib (%s)' % __version__ - - MANDATORY_KEYS = { - 'name': (), - 'version': (), - 'summary': ('legacy',), - } - - INDEX_KEYS = ('name version license summary description author ' - 'author_email keywords platform home_page classifiers ' - 'download_url') - - DEPENDENCY_KEYS = ('extras run_requires test_requires build_requires ' - 'dev_requires provides meta_requires obsoleted_by ' - 'supports_environments') - - SYNTAX_VALIDATORS = { - 'metadata_version': (METADATA_VERSION_MATCHER, ()), - 'name': (NAME_MATCHER, ('legacy',)), - 'version': (VERSION_MATCHER, ('legacy',)), - 'summary': (SUMMARY_MATCHER, ('legacy',)), - 'dynamic': (FIELDNAME_MATCHER, ('legacy',)), - } - - __slots__ = ('_legacy', '_data', 'scheme') - - def __init__(self, path=None, fileobj=None, mapping=None, - scheme='default'): - if [path, fileobj, mapping].count(None) < 2: - raise TypeError('path, fileobj and mapping are exclusive') - self._legacy = None - self._data = None - self.scheme = scheme - #import pdb; pdb.set_trace() - if mapping is not None: - try: - self._validate_mapping(mapping, scheme) - self._data = mapping - except MetadataUnrecognizedVersionError: - self._legacy = LegacyMetadata(mapping=mapping, scheme=scheme) - self.validate() - else: - data = None - if path: - with open(path, 'rb') as f: - data = f.read() - elif fileobj: - data = fileobj.read() - if data is None: - # Initialised with no args - to be added - self._data = { - 'metadata_version': self.METADATA_VERSION, - 'generator': self.GENERATOR, - } - else: - if not isinstance(data, text_type): - data = data.decode('utf-8') - try: - self._data = json.loads(data) - self._validate_mapping(self._data, scheme) - except ValueError: - # Note: MetadataUnrecognizedVersionError does not - # inherit from ValueError (it's a DistlibException, - # which should not inherit from ValueError). - # The ValueError comes from the json.load - if that - # succeeds and we get a validation error, we want - # that to propagate - self._legacy = LegacyMetadata(fileobj=StringIO(data), - scheme=scheme) - self.validate() - - common_keys = set(('name', 'version', 'license', 'keywords', 'summary')) - - none_list = (None, list) - none_dict = (None, dict) - - mapped_keys = { - 'run_requires': ('Requires-Dist', list), - 'build_requires': ('Setup-Requires-Dist', list), - 'dev_requires': none_list, - 'test_requires': none_list, - 'meta_requires': none_list, - 'extras': ('Provides-Extra', list), - 'modules': none_list, - 'namespaces': none_list, - 'exports': none_dict, - 'commands': none_dict, - 'classifiers': ('Classifier', list), - 'source_url': ('Download-URL', None), - 'metadata_version': ('Metadata-Version', None), - } - - del none_list, none_dict - - def __getattribute__(self, key): - common = object.__getattribute__(self, 'common_keys') - mapped = object.__getattribute__(self, 'mapped_keys') - if key in mapped: - lk, maker = mapped[key] - if self._legacy: - if lk is None: - result = None if maker is None else maker() - else: - result = self._legacy.get(lk) - else: - value = None if maker is None else maker() - if key not in ('commands', 'exports', 'modules', 'namespaces', - 'classifiers'): - result = self._data.get(key, value) - else: - # special cases for PEP 459 - sentinel = object() - result = sentinel - d = self._data.get('extensions') - if d: - if key == 'commands': - result = d.get('python.commands', value) - elif key == 'classifiers': - d = d.get('python.details') - if d: - result = d.get(key, value) - else: - d = d.get('python.exports') - if not d: - d = self._data.get('python.exports') - if d: - result = d.get(key, value) - if result is sentinel: - result = value - elif key not in common: - result = object.__getattribute__(self, key) - elif self._legacy: - result = self._legacy.get(key) - else: - result = self._data.get(key) - return result - - def _validate_value(self, key, value, scheme=None): - if key in self.SYNTAX_VALIDATORS: - pattern, exclusions = self.SYNTAX_VALIDATORS[key] - if (scheme or self.scheme) not in exclusions: - m = pattern.match(value) - if not m: - raise MetadataInvalidError("'%s' is an invalid value for " - "the '%s' property" % (value, - key)) - - def __setattr__(self, key, value): - self._validate_value(key, value) - common = object.__getattribute__(self, 'common_keys') - mapped = object.__getattribute__(self, 'mapped_keys') - if key in mapped: - lk, _ = mapped[key] - if self._legacy: - if lk is None: - raise NotImplementedError - self._legacy[lk] = value - elif key not in ('commands', 'exports', 'modules', 'namespaces', - 'classifiers'): - self._data[key] = value - else: - # special cases for PEP 459 - d = self._data.setdefault('extensions', {}) - if key == 'commands': - d['python.commands'] = value - elif key == 'classifiers': - d = d.setdefault('python.details', {}) - d[key] = value - else: - d = d.setdefault('python.exports', {}) - d[key] = value - elif key not in common: - object.__setattr__(self, key, value) - else: - if key == 'keywords': - if isinstance(value, string_types): - value = value.strip() - if value: - value = value.split() - else: - value = [] - if self._legacy: - self._legacy[key] = value - else: - self._data[key] = value - - @property - def name_and_version(self): - return _get_name_and_version(self.name, self.version, True) - - @property - def provides(self): - if self._legacy: - result = self._legacy['Provides-Dist'] - else: - result = self._data.setdefault('provides', []) - s = '%s (%s)' % (self.name, self.version) - if s not in result: - result.append(s) - return result - - @provides.setter - def provides(self, value): - if self._legacy: - self._legacy['Provides-Dist'] = value - else: - self._data['provides'] = value - - def get_requirements(self, reqts, extras=None, env=None): - """ - Base method to get dependencies, given a set of extras - to satisfy and an optional environment context. - :param reqts: A list of sometimes-wanted dependencies, - perhaps dependent on extras and environment. - :param extras: A list of optional components being requested. - :param env: An optional environment for marker evaluation. - """ - if self._legacy: - result = reqts - else: - result = [] - extras = get_extras(extras or [], self.extras) - for d in reqts: - if 'extra' not in d and 'environment' not in d: - # unconditional - include = True - else: - if 'extra' not in d: - # Not extra-dependent - only environment-dependent - include = True - else: - include = d.get('extra') in extras - if include: - # Not excluded because of extras, check environment - marker = d.get('environment') - if marker: - include = interpret(marker, env) - if include: - result.extend(d['requires']) - for key in ('build', 'dev', 'test'): - e = ':%s:' % key - if e in extras: - extras.remove(e) - # A recursive call, but it should terminate since 'test' - # has been removed from the extras - reqts = self._data.get('%s_requires' % key, []) - result.extend(self.get_requirements(reqts, extras=extras, - env=env)) - return result - - @property - def dictionary(self): - if self._legacy: - return self._from_legacy() - return self._data - - @property - def dependencies(self): - if self._legacy: - raise NotImplementedError - else: - return extract_by_key(self._data, self.DEPENDENCY_KEYS) - - @dependencies.setter - def dependencies(self, value): - if self._legacy: - raise NotImplementedError - else: - self._data.update(value) - - def _validate_mapping(self, mapping, scheme): - if mapping.get('metadata_version') != self.METADATA_VERSION: - raise MetadataUnrecognizedVersionError() - missing = [] - for key, exclusions in self.MANDATORY_KEYS.items(): - if key not in mapping: - if scheme not in exclusions: - missing.append(key) - if missing: - msg = 'Missing metadata items: %s' % ', '.join(missing) - raise MetadataMissingError(msg) - for k, v in mapping.items(): - self._validate_value(k, v, scheme) - - def validate(self): - if self._legacy: - missing, warnings = self._legacy.check(True) - if missing or warnings: - logger.warning('Metadata: missing: %s, warnings: %s', - missing, warnings) - else: - self._validate_mapping(self._data, self.scheme) - - def todict(self): - if self._legacy: - return self._legacy.todict(True) - else: - result = extract_by_key(self._data, self.INDEX_KEYS) - return result - - def _from_legacy(self): - assert self._legacy and not self._data - result = { - 'metadata_version': self.METADATA_VERSION, - 'generator': self.GENERATOR, - } - lmd = self._legacy.todict(True) # skip missing ones - for k in ('name', 'version', 'license', 'summary', 'description', - 'classifier'): - if k in lmd: - if k == 'classifier': - nk = 'classifiers' - else: - nk = k - result[nk] = lmd[k] - kw = lmd.get('Keywords', []) - if kw == ['']: - kw = [] - result['keywords'] = kw - keys = (('requires_dist', 'run_requires'), - ('setup_requires_dist', 'build_requires')) - for ok, nk in keys: - if ok in lmd and lmd[ok]: - result[nk] = [{'requires': lmd[ok]}] - result['provides'] = self.provides - author = {} - maintainer = {} - return result - - LEGACY_MAPPING = { - 'name': 'Name', - 'version': 'Version', - ('extensions', 'python.details', 'license'): 'License', - 'summary': 'Summary', - 'description': 'Description', - ('extensions', 'python.project', 'project_urls', 'Home'): 'Home-page', - ('extensions', 'python.project', 'contacts', 0, 'name'): 'Author', - ('extensions', 'python.project', 'contacts', 0, 'email'): 'Author-email', - 'source_url': 'Download-URL', - ('extensions', 'python.details', 'classifiers'): 'Classifier', - } - - def _to_legacy(self): - def process_entries(entries): - reqts = set() - for e in entries: - extra = e.get('extra') - env = e.get('environment') - rlist = e['requires'] - for r in rlist: - if not env and not extra: - reqts.add(r) - else: - marker = '' - if extra: - marker = 'extra == "%s"' % extra - if env: - if marker: - marker = '(%s) and %s' % (env, marker) - else: - marker = env - reqts.add(';'.join((r, marker))) - return reqts - - assert self._data and not self._legacy - result = LegacyMetadata() - nmd = self._data - # import pdb; pdb.set_trace() - for nk, ok in self.LEGACY_MAPPING.items(): - if not isinstance(nk, tuple): - if nk in nmd: - result[ok] = nmd[nk] - else: - d = nmd - found = True - for k in nk: - try: - d = d[k] - except (KeyError, IndexError): - found = False - break - if found: - result[ok] = d - r1 = process_entries(self.run_requires + self.meta_requires) - r2 = process_entries(self.build_requires + self.dev_requires) - if self.extras: - result['Provides-Extra'] = sorted(self.extras) - result['Requires-Dist'] = sorted(r1) - result['Setup-Requires-Dist'] = sorted(r2) - # TODO: any other fields wanted - return result - - def write(self, path=None, fileobj=None, legacy=False, skip_unknown=True): - if [path, fileobj].count(None) != 1: - raise ValueError('Exactly one of path and fileobj is needed') - self.validate() - if legacy: - if self._legacy: - legacy_md = self._legacy - else: - legacy_md = self._to_legacy() - if path: - legacy_md.write(path, skip_unknown=skip_unknown) - else: - legacy_md.write_file(fileobj, skip_unknown=skip_unknown) - else: - if self._legacy: - d = self._from_legacy() - else: - d = self._data - if fileobj: - json.dump(d, fileobj, ensure_ascii=True, indent=2, - sort_keys=True) - else: - with codecs.open(path, 'w', 'utf-8') as f: - json.dump(d, f, ensure_ascii=True, indent=2, - sort_keys=True) - - def add_requirements(self, requirements): - if self._legacy: - self._legacy.add_requirements(requirements) - else: - run_requires = self._data.setdefault('run_requires', []) - always = None - for entry in run_requires: - if 'environment' not in entry and 'extra' not in entry: - always = entry - break - if always is None: - always = { 'requires': requirements } - run_requires.insert(0, always) - else: - rset = set(always['requires']) | set(requirements) - always['requires'] = sorted(rset) - - def __repr__(self): - name = self.name or '(no name)' - version = self.version or 'no version' - return '<%s %s %s (%s)>' % (self.__class__.__name__, - self.metadata_version, name, version) diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/distlib/resources.py b/venv/lib/python3.11/site-packages/pip/_vendor/distlib/resources.py deleted file mode 100644 index fef52aa..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/distlib/resources.py +++ /dev/null @@ -1,358 +0,0 @@ -# -*- coding: utf-8 -*- -# -# Copyright (C) 2013-2017 Vinay Sajip. -# Licensed to the Python Software Foundation under a contributor agreement. -# See LICENSE.txt and CONTRIBUTORS.txt. -# -from __future__ import unicode_literals - -import bisect -import io -import logging -import os -import pkgutil -import sys -import types -import zipimport - -from . import DistlibException -from .util import cached_property, get_cache_base, Cache - -logger = logging.getLogger(__name__) - - -cache = None # created when needed - - -class ResourceCache(Cache): - def __init__(self, base=None): - if base is None: - # Use native string to avoid issues on 2.x: see Python #20140. - base = os.path.join(get_cache_base(), str('resource-cache')) - super(ResourceCache, self).__init__(base) - - def is_stale(self, resource, path): - """ - Is the cache stale for the given resource? - - :param resource: The :class:`Resource` being cached. - :param path: The path of the resource in the cache. - :return: True if the cache is stale. - """ - # Cache invalidation is a hard problem :-) - return True - - def get(self, resource): - """ - Get a resource into the cache, - - :param resource: A :class:`Resource` instance. - :return: The pathname of the resource in the cache. - """ - prefix, path = resource.finder.get_cache_info(resource) - if prefix is None: - result = path - else: - result = os.path.join(self.base, self.prefix_to_dir(prefix), path) - dirname = os.path.dirname(result) - if not os.path.isdir(dirname): - os.makedirs(dirname) - if not os.path.exists(result): - stale = True - else: - stale = self.is_stale(resource, path) - if stale: - # write the bytes of the resource to the cache location - with open(result, 'wb') as f: - f.write(resource.bytes) - return result - - -class ResourceBase(object): - def __init__(self, finder, name): - self.finder = finder - self.name = name - - -class Resource(ResourceBase): - """ - A class representing an in-package resource, such as a data file. This is - not normally instantiated by user code, but rather by a - :class:`ResourceFinder` which manages the resource. - """ - is_container = False # Backwards compatibility - - def as_stream(self): - """ - Get the resource as a stream. - - This is not a property to make it obvious that it returns a new stream - each time. - """ - return self.finder.get_stream(self) - - @cached_property - def file_path(self): - global cache - if cache is None: - cache = ResourceCache() - return cache.get(self) - - @cached_property - def bytes(self): - return self.finder.get_bytes(self) - - @cached_property - def size(self): - return self.finder.get_size(self) - - -class ResourceContainer(ResourceBase): - is_container = True # Backwards compatibility - - @cached_property - def resources(self): - return self.finder.get_resources(self) - - -class ResourceFinder(object): - """ - Resource finder for file system resources. - """ - - if sys.platform.startswith('java'): - skipped_extensions = ('.pyc', '.pyo', '.class') - else: - skipped_extensions = ('.pyc', '.pyo') - - def __init__(self, module): - self.module = module - self.loader = getattr(module, '__loader__', None) - self.base = os.path.dirname(getattr(module, '__file__', '')) - - def _adjust_path(self, path): - return os.path.realpath(path) - - def _make_path(self, resource_name): - # Issue #50: need to preserve type of path on Python 2.x - # like os.path._get_sep - if isinstance(resource_name, bytes): # should only happen on 2.x - sep = b'/' - else: - sep = '/' - parts = resource_name.split(sep) - parts.insert(0, self.base) - result = os.path.join(*parts) - return self._adjust_path(result) - - def _find(self, path): - return os.path.exists(path) - - def get_cache_info(self, resource): - return None, resource.path - - def find(self, resource_name): - path = self._make_path(resource_name) - if not self._find(path): - result = None - else: - if self._is_directory(path): - result = ResourceContainer(self, resource_name) - else: - result = Resource(self, resource_name) - result.path = path - return result - - def get_stream(self, resource): - return open(resource.path, 'rb') - - def get_bytes(self, resource): - with open(resource.path, 'rb') as f: - return f.read() - - def get_size(self, resource): - return os.path.getsize(resource.path) - - def get_resources(self, resource): - def allowed(f): - return (f != '__pycache__' and not - f.endswith(self.skipped_extensions)) - return set([f for f in os.listdir(resource.path) if allowed(f)]) - - def is_container(self, resource): - return self._is_directory(resource.path) - - _is_directory = staticmethod(os.path.isdir) - - def iterator(self, resource_name): - resource = self.find(resource_name) - if resource is not None: - todo = [resource] - while todo: - resource = todo.pop(0) - yield resource - if resource.is_container: - rname = resource.name - for name in resource.resources: - if not rname: - new_name = name - else: - new_name = '/'.join([rname, name]) - child = self.find(new_name) - if child.is_container: - todo.append(child) - else: - yield child - - -class ZipResourceFinder(ResourceFinder): - """ - Resource finder for resources in .zip files. - """ - def __init__(self, module): - super(ZipResourceFinder, self).__init__(module) - archive = self.loader.archive - self.prefix_len = 1 + len(archive) - # PyPy doesn't have a _files attr on zipimporter, and you can't set one - if hasattr(self.loader, '_files'): - self._files = self.loader._files - else: - self._files = zipimport._zip_directory_cache[archive] - self.index = sorted(self._files) - - def _adjust_path(self, path): - return path - - def _find(self, path): - path = path[self.prefix_len:] - if path in self._files: - result = True - else: - if path and path[-1] != os.sep: - path = path + os.sep - i = bisect.bisect(self.index, path) - try: - result = self.index[i].startswith(path) - except IndexError: - result = False - if not result: - logger.debug('_find failed: %r %r', path, self.loader.prefix) - else: - logger.debug('_find worked: %r %r', path, self.loader.prefix) - return result - - def get_cache_info(self, resource): - prefix = self.loader.archive - path = resource.path[1 + len(prefix):] - return prefix, path - - def get_bytes(self, resource): - return self.loader.get_data(resource.path) - - def get_stream(self, resource): - return io.BytesIO(self.get_bytes(resource)) - - def get_size(self, resource): - path = resource.path[self.prefix_len:] - return self._files[path][3] - - def get_resources(self, resource): - path = resource.path[self.prefix_len:] - if path and path[-1] != os.sep: - path += os.sep - plen = len(path) - result = set() - i = bisect.bisect(self.index, path) - while i < len(self.index): - if not self.index[i].startswith(path): - break - s = self.index[i][plen:] - result.add(s.split(os.sep, 1)[0]) # only immediate children - i += 1 - return result - - def _is_directory(self, path): - path = path[self.prefix_len:] - if path and path[-1] != os.sep: - path += os.sep - i = bisect.bisect(self.index, path) - try: - result = self.index[i].startswith(path) - except IndexError: - result = False - return result - - -_finder_registry = { - type(None): ResourceFinder, - zipimport.zipimporter: ZipResourceFinder -} - -try: - # In Python 3.6, _frozen_importlib -> _frozen_importlib_external - try: - import _frozen_importlib_external as _fi - except ImportError: - import _frozen_importlib as _fi - _finder_registry[_fi.SourceFileLoader] = ResourceFinder - _finder_registry[_fi.FileFinder] = ResourceFinder - # See issue #146 - _finder_registry[_fi.SourcelessFileLoader] = ResourceFinder - del _fi -except (ImportError, AttributeError): - pass - - -def register_finder(loader, finder_maker): - _finder_registry[type(loader)] = finder_maker - - -_finder_cache = {} - - -def finder(package): - """ - Return a resource finder for a package. - :param package: The name of the package. - :return: A :class:`ResourceFinder` instance for the package. - """ - if package in _finder_cache: - result = _finder_cache[package] - else: - if package not in sys.modules: - __import__(package) - module = sys.modules[package] - path = getattr(module, '__path__', None) - if path is None: - raise DistlibException('You cannot get a finder for a module, ' - 'only for a package') - loader = getattr(module, '__loader__', None) - finder_maker = _finder_registry.get(type(loader)) - if finder_maker is None: - raise DistlibException('Unable to locate finder for %r' % package) - result = finder_maker(module) - _finder_cache[package] = result - return result - - -_dummy_module = types.ModuleType(str('__dummy__')) - - -def finder_for_path(path): - """ - Return a resource finder for a path, which should represent a container. - - :param path: The path. - :return: A :class:`ResourceFinder` instance for the path. - """ - result = None - # calls any path hooks, gets importer into cache - pkgutil.get_importer(path) - loader = sys.path_importer_cache.get(path) - finder = _finder_registry.get(type(loader)) - if finder: - module = _dummy_module - module.__file__ = os.path.join(path, '') - module.__loader__ = loader - result = finder(module) - return result diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/distlib/scripts.py b/venv/lib/python3.11/site-packages/pip/_vendor/distlib/scripts.py deleted file mode 100644 index d270624..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/distlib/scripts.py +++ /dev/null @@ -1,437 +0,0 @@ -# -*- coding: utf-8 -*- -# -# Copyright (C) 2013-2015 Vinay Sajip. -# Licensed to the Python Software Foundation under a contributor agreement. -# See LICENSE.txt and CONTRIBUTORS.txt. -# -from io import BytesIO -import logging -import os -import re -import struct -import sys -import time -from zipfile import ZipInfo - -from .compat import sysconfig, detect_encoding, ZipFile -from .resources import finder -from .util import (FileOperator, get_export_entry, convert_path, - get_executable, get_platform, in_venv) - -logger = logging.getLogger(__name__) - -_DEFAULT_MANIFEST = ''' - - - - - - - - - - - - -'''.strip() - -# check if Python is called on the first line with this expression -FIRST_LINE_RE = re.compile(b'^#!.*pythonw?[0-9.]*([ \t].*)?$') -SCRIPT_TEMPLATE = r'''# -*- coding: utf-8 -*- -import re -import sys -from %(module)s import %(import_name)s -if __name__ == '__main__': - sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0]) - sys.exit(%(func)s()) -''' - - -def enquote_executable(executable): - if ' ' in executable: - # make sure we quote only the executable in case of env - # for example /usr/bin/env "/dir with spaces/bin/jython" - # instead of "/usr/bin/env /dir with spaces/bin/jython" - # otherwise whole - if executable.startswith('/usr/bin/env '): - env, _executable = executable.split(' ', 1) - if ' ' in _executable and not _executable.startswith('"'): - executable = '%s "%s"' % (env, _executable) - else: - if not executable.startswith('"'): - executable = '"%s"' % executable - return executable - -# Keep the old name around (for now), as there is at least one project using it! -_enquote_executable = enquote_executable - -class ScriptMaker(object): - """ - A class to copy or create scripts from source scripts or callable - specifications. - """ - script_template = SCRIPT_TEMPLATE - - executable = None # for shebangs - - def __init__(self, source_dir, target_dir, add_launchers=True, - dry_run=False, fileop=None): - self.source_dir = source_dir - self.target_dir = target_dir - self.add_launchers = add_launchers - self.force = False - self.clobber = False - # It only makes sense to set mode bits on POSIX. - self.set_mode = (os.name == 'posix') or (os.name == 'java' and - os._name == 'posix') - self.variants = set(('', 'X.Y')) - self._fileop = fileop or FileOperator(dry_run) - - self._is_nt = os.name == 'nt' or ( - os.name == 'java' and os._name == 'nt') - self.version_info = sys.version_info - - def _get_alternate_executable(self, executable, options): - if options.get('gui', False) and self._is_nt: # pragma: no cover - dn, fn = os.path.split(executable) - fn = fn.replace('python', 'pythonw') - executable = os.path.join(dn, fn) - return executable - - if sys.platform.startswith('java'): # pragma: no cover - def _is_shell(self, executable): - """ - Determine if the specified executable is a script - (contains a #! line) - """ - try: - with open(executable) as fp: - return fp.read(2) == '#!' - except (OSError, IOError): - logger.warning('Failed to open %s', executable) - return False - - def _fix_jython_executable(self, executable): - if self._is_shell(executable): - # Workaround for Jython is not needed on Linux systems. - import java - - if java.lang.System.getProperty('os.name') == 'Linux': - return executable - elif executable.lower().endswith('jython.exe'): - # Use wrapper exe for Jython on Windows - return executable - return '/usr/bin/env %s' % executable - - def _build_shebang(self, executable, post_interp): - """ - Build a shebang line. In the simple case (on Windows, or a shebang line - which is not too long or contains spaces) use a simple formulation for - the shebang. Otherwise, use /bin/sh as the executable, with a contrived - shebang which allows the script to run either under Python or sh, using - suitable quoting. Thanks to Harald Nordgren for his input. - - See also: http://www.in-ulm.de/~mascheck/various/shebang/#length - https://hg.mozilla.org/mozilla-central/file/tip/mach - """ - if os.name != 'posix': - simple_shebang = True - else: - # Add 3 for '#!' prefix and newline suffix. - shebang_length = len(executable) + len(post_interp) + 3 - if sys.platform == 'darwin': - max_shebang_length = 512 - else: - max_shebang_length = 127 - simple_shebang = ((b' ' not in executable) and - (shebang_length <= max_shebang_length)) - - if simple_shebang: - result = b'#!' + executable + post_interp + b'\n' - else: - result = b'#!/bin/sh\n' - result += b"'''exec' " + executable + post_interp + b' "$0" "$@"\n' - result += b"' '''" - return result - - def _get_shebang(self, encoding, post_interp=b'', options=None): - enquote = True - if self.executable: - executable = self.executable - enquote = False # assume this will be taken care of - elif not sysconfig.is_python_build(): - executable = get_executable() - elif in_venv(): # pragma: no cover - executable = os.path.join(sysconfig.get_path('scripts'), - 'python%s' % sysconfig.get_config_var('EXE')) - else: # pragma: no cover - executable = os.path.join( - sysconfig.get_config_var('BINDIR'), - 'python%s%s' % (sysconfig.get_config_var('VERSION'), - sysconfig.get_config_var('EXE'))) - if not os.path.isfile(executable): - # for Python builds from source on Windows, no Python executables with - # a version suffix are created, so we use python.exe - executable = os.path.join(sysconfig.get_config_var('BINDIR'), - 'python%s' % (sysconfig.get_config_var('EXE'))) - if options: - executable = self._get_alternate_executable(executable, options) - - if sys.platform.startswith('java'): # pragma: no cover - executable = self._fix_jython_executable(executable) - - # Normalise case for Windows - COMMENTED OUT - # executable = os.path.normcase(executable) - # N.B. The normalising operation above has been commented out: See - # issue #124. Although paths in Windows are generally case-insensitive, - # they aren't always. For example, a path containing a ẞ (which is a - # LATIN CAPITAL LETTER SHARP S - U+1E9E) is normcased to ß (which is a - # LATIN SMALL LETTER SHARP S' - U+00DF). The two are not considered by - # Windows as equivalent in path names. - - # If the user didn't specify an executable, it may be necessary to - # cater for executable paths with spaces (not uncommon on Windows) - if enquote: - executable = enquote_executable(executable) - # Issue #51: don't use fsencode, since we later try to - # check that the shebang is decodable using utf-8. - executable = executable.encode('utf-8') - # in case of IronPython, play safe and enable frames support - if (sys.platform == 'cli' and '-X:Frames' not in post_interp - and '-X:FullFrames' not in post_interp): # pragma: no cover - post_interp += b' -X:Frames' - shebang = self._build_shebang(executable, post_interp) - # Python parser starts to read a script using UTF-8 until - # it gets a #coding:xxx cookie. The shebang has to be the - # first line of a file, the #coding:xxx cookie cannot be - # written before. So the shebang has to be decodable from - # UTF-8. - try: - shebang.decode('utf-8') - except UnicodeDecodeError: # pragma: no cover - raise ValueError( - 'The shebang (%r) is not decodable from utf-8' % shebang) - # If the script is encoded to a custom encoding (use a - # #coding:xxx cookie), the shebang has to be decodable from - # the script encoding too. - if encoding != 'utf-8': - try: - shebang.decode(encoding) - except UnicodeDecodeError: # pragma: no cover - raise ValueError( - 'The shebang (%r) is not decodable ' - 'from the script encoding (%r)' % (shebang, encoding)) - return shebang - - def _get_script_text(self, entry): - return self.script_template % dict(module=entry.prefix, - import_name=entry.suffix.split('.')[0], - func=entry.suffix) - - manifest = _DEFAULT_MANIFEST - - def get_manifest(self, exename): - base = os.path.basename(exename) - return self.manifest % base - - def _write_script(self, names, shebang, script_bytes, filenames, ext): - use_launcher = self.add_launchers and self._is_nt - linesep = os.linesep.encode('utf-8') - if not shebang.endswith(linesep): - shebang += linesep - if not use_launcher: - script_bytes = shebang + script_bytes - else: # pragma: no cover - if ext == 'py': - launcher = self._get_launcher('t') - else: - launcher = self._get_launcher('w') - stream = BytesIO() - with ZipFile(stream, 'w') as zf: - source_date_epoch = os.environ.get('SOURCE_DATE_EPOCH') - if source_date_epoch: - date_time = time.gmtime(int(source_date_epoch))[:6] - zinfo = ZipInfo(filename='__main__.py', date_time=date_time) - zf.writestr(zinfo, script_bytes) - else: - zf.writestr('__main__.py', script_bytes) - zip_data = stream.getvalue() - script_bytes = launcher + shebang + zip_data - for name in names: - outname = os.path.join(self.target_dir, name) - if use_launcher: # pragma: no cover - n, e = os.path.splitext(outname) - if e.startswith('.py'): - outname = n - outname = '%s.exe' % outname - try: - self._fileop.write_binary_file(outname, script_bytes) - except Exception: - # Failed writing an executable - it might be in use. - logger.warning('Failed to write executable - trying to ' - 'use .deleteme logic') - dfname = '%s.deleteme' % outname - if os.path.exists(dfname): - os.remove(dfname) # Not allowed to fail here - os.rename(outname, dfname) # nor here - self._fileop.write_binary_file(outname, script_bytes) - logger.debug('Able to replace executable using ' - '.deleteme logic') - try: - os.remove(dfname) - except Exception: - pass # still in use - ignore error - else: - if self._is_nt and not outname.endswith('.' + ext): # pragma: no cover - outname = '%s.%s' % (outname, ext) - if os.path.exists(outname) and not self.clobber: - logger.warning('Skipping existing file %s', outname) - continue - self._fileop.write_binary_file(outname, script_bytes) - if self.set_mode: - self._fileop.set_executable_mode([outname]) - filenames.append(outname) - - variant_separator = '-' - - def get_script_filenames(self, name): - result = set() - if '' in self.variants: - result.add(name) - if 'X' in self.variants: - result.add('%s%s' % (name, self.version_info[0])) - if 'X.Y' in self.variants: - result.add('%s%s%s.%s' % (name, self.variant_separator, - self.version_info[0], self.version_info[1])) - return result - - def _make_script(self, entry, filenames, options=None): - post_interp = b'' - if options: - args = options.get('interpreter_args', []) - if args: - args = ' %s' % ' '.join(args) - post_interp = args.encode('utf-8') - shebang = self._get_shebang('utf-8', post_interp, options=options) - script = self._get_script_text(entry).encode('utf-8') - scriptnames = self.get_script_filenames(entry.name) - if options and options.get('gui', False): - ext = 'pyw' - else: - ext = 'py' - self._write_script(scriptnames, shebang, script, filenames, ext) - - def _copy_script(self, script, filenames): - adjust = False - script = os.path.join(self.source_dir, convert_path(script)) - outname = os.path.join(self.target_dir, os.path.basename(script)) - if not self.force and not self._fileop.newer(script, outname): - logger.debug('not copying %s (up-to-date)', script) - return - - # Always open the file, but ignore failures in dry-run mode -- - # that way, we'll get accurate feedback if we can read the - # script. - try: - f = open(script, 'rb') - except IOError: # pragma: no cover - if not self.dry_run: - raise - f = None - else: - first_line = f.readline() - if not first_line: # pragma: no cover - logger.warning('%s is an empty file (skipping)', script) - return - - match = FIRST_LINE_RE.match(first_line.replace(b'\r\n', b'\n')) - if match: - adjust = True - post_interp = match.group(1) or b'' - - if not adjust: - if f: - f.close() - self._fileop.copy_file(script, outname) - if self.set_mode: - self._fileop.set_executable_mode([outname]) - filenames.append(outname) - else: - logger.info('copying and adjusting %s -> %s', script, - self.target_dir) - if not self._fileop.dry_run: - encoding, lines = detect_encoding(f.readline) - f.seek(0) - shebang = self._get_shebang(encoding, post_interp) - if b'pythonw' in first_line: # pragma: no cover - ext = 'pyw' - else: - ext = 'py' - n = os.path.basename(outname) - self._write_script([n], shebang, f.read(), filenames, ext) - if f: - f.close() - - @property - def dry_run(self): - return self._fileop.dry_run - - @dry_run.setter - def dry_run(self, value): - self._fileop.dry_run = value - - if os.name == 'nt' or (os.name == 'java' and os._name == 'nt'): # pragma: no cover - # Executable launcher support. - # Launchers are from https://bitbucket.org/vinay.sajip/simple_launcher/ - - def _get_launcher(self, kind): - if struct.calcsize('P') == 8: # 64-bit - bits = '64' - else: - bits = '32' - platform_suffix = '-arm' if get_platform() == 'win-arm64' else '' - name = '%s%s%s.exe' % (kind, bits, platform_suffix) - # Issue 31: don't hardcode an absolute package name, but - # determine it relative to the current package - distlib_package = __name__.rsplit('.', 1)[0] - resource = finder(distlib_package).find(name) - if not resource: - msg = ('Unable to find resource %s in package %s' % (name, - distlib_package)) - raise ValueError(msg) - return resource.bytes - - # Public API follows - - def make(self, specification, options=None): - """ - Make a script. - - :param specification: The specification, which is either a valid export - entry specification (to make a script from a - callable) or a filename (to make a script by - copying from a source location). - :param options: A dictionary of options controlling script generation. - :return: A list of all absolute pathnames written to. - """ - filenames = [] - entry = get_export_entry(specification) - if entry is None: - self._copy_script(specification, filenames) - else: - self._make_script(entry, filenames, options=options) - return filenames - - def make_multiple(self, specifications, options=None): - """ - Take a list of specifications and make scripts from them, - :param specifications: A list of specifications. - :return: A list of all absolute pathnames written to, - """ - filenames = [] - for specification in specifications: - filenames.extend(self.make(specification, options)) - return filenames diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/distlib/t32.exe b/venv/lib/python3.11/site-packages/pip/_vendor/distlib/t32.exe deleted file mode 100644 index 52154f0..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/distlib/t32.exe and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/distlib/t64-arm.exe b/venv/lib/python3.11/site-packages/pip/_vendor/distlib/t64-arm.exe deleted file mode 100644 index e1ab8f8..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/distlib/t64-arm.exe and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/distlib/t64.exe b/venv/lib/python3.11/site-packages/pip/_vendor/distlib/t64.exe deleted file mode 100644 index e8bebdb..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/distlib/t64.exe and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/distlib/util.py b/venv/lib/python3.11/site-packages/pip/_vendor/distlib/util.py deleted file mode 100644 index dd01849..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/distlib/util.py +++ /dev/null @@ -1,1932 +0,0 @@ -# -# Copyright (C) 2012-2021 The Python Software Foundation. -# See LICENSE.txt and CONTRIBUTORS.txt. -# -import codecs -from collections import deque -import contextlib -import csv -from glob import iglob as std_iglob -import io -import json -import logging -import os -import py_compile -import re -import socket -try: - import ssl -except ImportError: # pragma: no cover - ssl = None -import subprocess -import sys -import tarfile -import tempfile -import textwrap - -try: - import threading -except ImportError: # pragma: no cover - import dummy_threading as threading -import time - -from . import DistlibException -from .compat import (string_types, text_type, shutil, raw_input, StringIO, - cache_from_source, urlopen, urljoin, httplib, xmlrpclib, - splittype, HTTPHandler, BaseConfigurator, valid_ident, - Container, configparser, URLError, ZipFile, fsdecode, - unquote, urlparse) - -logger = logging.getLogger(__name__) - -# -# Requirement parsing code as per PEP 508 -# - -IDENTIFIER = re.compile(r'^([\w\.-]+)\s*') -VERSION_IDENTIFIER = re.compile(r'^([\w\.*+-]+)\s*') -COMPARE_OP = re.compile(r'^(<=?|>=?|={2,3}|[~!]=)\s*') -MARKER_OP = re.compile(r'^((<=?)|(>=?)|={2,3}|[~!]=|in|not\s+in)\s*') -OR = re.compile(r'^or\b\s*') -AND = re.compile(r'^and\b\s*') -NON_SPACE = re.compile(r'(\S+)\s*') -STRING_CHUNK = re.compile(r'([\s\w\.{}()*+#:;,/?!~`@$%^&=|<>\[\]-]+)') - - -def parse_marker(marker_string): - """ - Parse a marker string and return a dictionary containing a marker expression. - - The dictionary will contain keys "op", "lhs" and "rhs" for non-terminals in - the expression grammar, or strings. A string contained in quotes is to be - interpreted as a literal string, and a string not contained in quotes is a - variable (such as os_name). - """ - def marker_var(remaining): - # either identifier, or literal string - m = IDENTIFIER.match(remaining) - if m: - result = m.groups()[0] - remaining = remaining[m.end():] - elif not remaining: - raise SyntaxError('unexpected end of input') - else: - q = remaining[0] - if q not in '\'"': - raise SyntaxError('invalid expression: %s' % remaining) - oq = '\'"'.replace(q, '') - remaining = remaining[1:] - parts = [q] - while remaining: - # either a string chunk, or oq, or q to terminate - if remaining[0] == q: - break - elif remaining[0] == oq: - parts.append(oq) - remaining = remaining[1:] - else: - m = STRING_CHUNK.match(remaining) - if not m: - raise SyntaxError('error in string literal: %s' % remaining) - parts.append(m.groups()[0]) - remaining = remaining[m.end():] - else: - s = ''.join(parts) - raise SyntaxError('unterminated string: %s' % s) - parts.append(q) - result = ''.join(parts) - remaining = remaining[1:].lstrip() # skip past closing quote - return result, remaining - - def marker_expr(remaining): - if remaining and remaining[0] == '(': - result, remaining = marker(remaining[1:].lstrip()) - if remaining[0] != ')': - raise SyntaxError('unterminated parenthesis: %s' % remaining) - remaining = remaining[1:].lstrip() - else: - lhs, remaining = marker_var(remaining) - while remaining: - m = MARKER_OP.match(remaining) - if not m: - break - op = m.groups()[0] - remaining = remaining[m.end():] - rhs, remaining = marker_var(remaining) - lhs = {'op': op, 'lhs': lhs, 'rhs': rhs} - result = lhs - return result, remaining - - def marker_and(remaining): - lhs, remaining = marker_expr(remaining) - while remaining: - m = AND.match(remaining) - if not m: - break - remaining = remaining[m.end():] - rhs, remaining = marker_expr(remaining) - lhs = {'op': 'and', 'lhs': lhs, 'rhs': rhs} - return lhs, remaining - - def marker(remaining): - lhs, remaining = marker_and(remaining) - while remaining: - m = OR.match(remaining) - if not m: - break - remaining = remaining[m.end():] - rhs, remaining = marker_and(remaining) - lhs = {'op': 'or', 'lhs': lhs, 'rhs': rhs} - return lhs, remaining - - return marker(marker_string) - - -def parse_requirement(req): - """ - Parse a requirement passed in as a string. Return a Container - whose attributes contain the various parts of the requirement. - """ - remaining = req.strip() - if not remaining or remaining.startswith('#'): - return None - m = IDENTIFIER.match(remaining) - if not m: - raise SyntaxError('name expected: %s' % remaining) - distname = m.groups()[0] - remaining = remaining[m.end():] - extras = mark_expr = versions = uri = None - if remaining and remaining[0] == '[': - i = remaining.find(']', 1) - if i < 0: - raise SyntaxError('unterminated extra: %s' % remaining) - s = remaining[1:i] - remaining = remaining[i + 1:].lstrip() - extras = [] - while s: - m = IDENTIFIER.match(s) - if not m: - raise SyntaxError('malformed extra: %s' % s) - extras.append(m.groups()[0]) - s = s[m.end():] - if not s: - break - if s[0] != ',': - raise SyntaxError('comma expected in extras: %s' % s) - s = s[1:].lstrip() - if not extras: - extras = None - if remaining: - if remaining[0] == '@': - # it's a URI - remaining = remaining[1:].lstrip() - m = NON_SPACE.match(remaining) - if not m: - raise SyntaxError('invalid URI: %s' % remaining) - uri = m.groups()[0] - t = urlparse(uri) - # there are issues with Python and URL parsing, so this test - # is a bit crude. See bpo-20271, bpo-23505. Python doesn't - # always parse invalid URLs correctly - it should raise - # exceptions for malformed URLs - if not (t.scheme and t.netloc): - raise SyntaxError('Invalid URL: %s' % uri) - remaining = remaining[m.end():].lstrip() - else: - - def get_versions(ver_remaining): - """ - Return a list of operator, version tuples if any are - specified, else None. - """ - m = COMPARE_OP.match(ver_remaining) - versions = None - if m: - versions = [] - while True: - op = m.groups()[0] - ver_remaining = ver_remaining[m.end():] - m = VERSION_IDENTIFIER.match(ver_remaining) - if not m: - raise SyntaxError('invalid version: %s' % ver_remaining) - v = m.groups()[0] - versions.append((op, v)) - ver_remaining = ver_remaining[m.end():] - if not ver_remaining or ver_remaining[0] != ',': - break - ver_remaining = ver_remaining[1:].lstrip() - # Some packages have a trailing comma which would break things - # See issue #148 - if not ver_remaining: - break - m = COMPARE_OP.match(ver_remaining) - if not m: - raise SyntaxError('invalid constraint: %s' % ver_remaining) - if not versions: - versions = None - return versions, ver_remaining - - if remaining[0] != '(': - versions, remaining = get_versions(remaining) - else: - i = remaining.find(')', 1) - if i < 0: - raise SyntaxError('unterminated parenthesis: %s' % remaining) - s = remaining[1:i] - remaining = remaining[i + 1:].lstrip() - # As a special diversion from PEP 508, allow a version number - # a.b.c in parentheses as a synonym for ~= a.b.c (because this - # is allowed in earlier PEPs) - if COMPARE_OP.match(s): - versions, _ = get_versions(s) - else: - m = VERSION_IDENTIFIER.match(s) - if not m: - raise SyntaxError('invalid constraint: %s' % s) - v = m.groups()[0] - s = s[m.end():].lstrip() - if s: - raise SyntaxError('invalid constraint: %s' % s) - versions = [('~=', v)] - - if remaining: - if remaining[0] != ';': - raise SyntaxError('invalid requirement: %s' % remaining) - remaining = remaining[1:].lstrip() - - mark_expr, remaining = parse_marker(remaining) - - if remaining and remaining[0] != '#': - raise SyntaxError('unexpected trailing data: %s' % remaining) - - if not versions: - rs = distname - else: - rs = '%s %s' % (distname, ', '.join(['%s %s' % con for con in versions])) - return Container(name=distname, extras=extras, constraints=versions, - marker=mark_expr, url=uri, requirement=rs) - - -def get_resources_dests(resources_root, rules): - """Find destinations for resources files""" - - def get_rel_path(root, path): - # normalizes and returns a lstripped-/-separated path - root = root.replace(os.path.sep, '/') - path = path.replace(os.path.sep, '/') - assert path.startswith(root) - return path[len(root):].lstrip('/') - - destinations = {} - for base, suffix, dest in rules: - prefix = os.path.join(resources_root, base) - for abs_base in iglob(prefix): - abs_glob = os.path.join(abs_base, suffix) - for abs_path in iglob(abs_glob): - resource_file = get_rel_path(resources_root, abs_path) - if dest is None: # remove the entry if it was here - destinations.pop(resource_file, None) - else: - rel_path = get_rel_path(abs_base, abs_path) - rel_dest = dest.replace(os.path.sep, '/').rstrip('/') - destinations[resource_file] = rel_dest + '/' + rel_path - return destinations - - -def in_venv(): - if hasattr(sys, 'real_prefix'): - # virtualenv venvs - result = True - else: - # PEP 405 venvs - result = sys.prefix != getattr(sys, 'base_prefix', sys.prefix) - return result - - -def get_executable(): -# The __PYVENV_LAUNCHER__ dance is apparently no longer needed, as -# changes to the stub launcher mean that sys.executable always points -# to the stub on OS X -# if sys.platform == 'darwin' and ('__PYVENV_LAUNCHER__' -# in os.environ): -# result = os.environ['__PYVENV_LAUNCHER__'] -# else: -# result = sys.executable -# return result - # Avoid normcasing: see issue #143 - # result = os.path.normcase(sys.executable) - result = sys.executable - if not isinstance(result, text_type): - result = fsdecode(result) - return result - - -def proceed(prompt, allowed_chars, error_prompt=None, default=None): - p = prompt - while True: - s = raw_input(p) - p = prompt - if not s and default: - s = default - if s: - c = s[0].lower() - if c in allowed_chars: - break - if error_prompt: - p = '%c: %s\n%s' % (c, error_prompt, prompt) - return c - - -def extract_by_key(d, keys): - if isinstance(keys, string_types): - keys = keys.split() - result = {} - for key in keys: - if key in d: - result[key] = d[key] - return result - -def read_exports(stream): - if sys.version_info[0] >= 3: - # needs to be a text stream - stream = codecs.getreader('utf-8')(stream) - # Try to load as JSON, falling back on legacy format - data = stream.read() - stream = StringIO(data) - try: - jdata = json.load(stream) - result = jdata['extensions']['python.exports']['exports'] - for group, entries in result.items(): - for k, v in entries.items(): - s = '%s = %s' % (k, v) - entry = get_export_entry(s) - assert entry is not None - entries[k] = entry - return result - except Exception: - stream.seek(0, 0) - - def read_stream(cp, stream): - if hasattr(cp, 'read_file'): - cp.read_file(stream) - else: - cp.readfp(stream) - - cp = configparser.ConfigParser() - try: - read_stream(cp, stream) - except configparser.MissingSectionHeaderError: - stream.close() - data = textwrap.dedent(data) - stream = StringIO(data) - read_stream(cp, stream) - - result = {} - for key in cp.sections(): - result[key] = entries = {} - for name, value in cp.items(key): - s = '%s = %s' % (name, value) - entry = get_export_entry(s) - assert entry is not None - #entry.dist = self - entries[name] = entry - return result - - -def write_exports(exports, stream): - if sys.version_info[0] >= 3: - # needs to be a text stream - stream = codecs.getwriter('utf-8')(stream) - cp = configparser.ConfigParser() - for k, v in exports.items(): - # TODO check k, v for valid values - cp.add_section(k) - for entry in v.values(): - if entry.suffix is None: - s = entry.prefix - else: - s = '%s:%s' % (entry.prefix, entry.suffix) - if entry.flags: - s = '%s [%s]' % (s, ', '.join(entry.flags)) - cp.set(k, entry.name, s) - cp.write(stream) - - -@contextlib.contextmanager -def tempdir(): - td = tempfile.mkdtemp() - try: - yield td - finally: - shutil.rmtree(td) - -@contextlib.contextmanager -def chdir(d): - cwd = os.getcwd() - try: - os.chdir(d) - yield - finally: - os.chdir(cwd) - - -@contextlib.contextmanager -def socket_timeout(seconds=15): - cto = socket.getdefaulttimeout() - try: - socket.setdefaulttimeout(seconds) - yield - finally: - socket.setdefaulttimeout(cto) - - -class cached_property(object): - def __init__(self, func): - self.func = func - #for attr in ('__name__', '__module__', '__doc__'): - # setattr(self, attr, getattr(func, attr, None)) - - def __get__(self, obj, cls=None): - if obj is None: - return self - value = self.func(obj) - object.__setattr__(obj, self.func.__name__, value) - #obj.__dict__[self.func.__name__] = value = self.func(obj) - return value - -def convert_path(pathname): - """Return 'pathname' as a name that will work on the native filesystem. - - The path is split on '/' and put back together again using the current - directory separator. Needed because filenames in the setup script are - always supplied in Unix style, and have to be converted to the local - convention before we can actually use them in the filesystem. Raises - ValueError on non-Unix-ish systems if 'pathname' either starts or - ends with a slash. - """ - if os.sep == '/': - return pathname - if not pathname: - return pathname - if pathname[0] == '/': - raise ValueError("path '%s' cannot be absolute" % pathname) - if pathname[-1] == '/': - raise ValueError("path '%s' cannot end with '/'" % pathname) - - paths = pathname.split('/') - while os.curdir in paths: - paths.remove(os.curdir) - if not paths: - return os.curdir - return os.path.join(*paths) - - -class FileOperator(object): - def __init__(self, dry_run=False): - self.dry_run = dry_run - self.ensured = set() - self._init_record() - - def _init_record(self): - self.record = False - self.files_written = set() - self.dirs_created = set() - - def record_as_written(self, path): - if self.record: - self.files_written.add(path) - - def newer(self, source, target): - """Tell if the target is newer than the source. - - Returns true if 'source' exists and is more recently modified than - 'target', or if 'source' exists and 'target' doesn't. - - Returns false if both exist and 'target' is the same age or younger - than 'source'. Raise PackagingFileError if 'source' does not exist. - - Note that this test is not very accurate: files created in the same - second will have the same "age". - """ - if not os.path.exists(source): - raise DistlibException("file '%r' does not exist" % - os.path.abspath(source)) - if not os.path.exists(target): - return True - - return os.stat(source).st_mtime > os.stat(target).st_mtime - - def copy_file(self, infile, outfile, check=True): - """Copy a file respecting dry-run and force flags. - """ - self.ensure_dir(os.path.dirname(outfile)) - logger.info('Copying %s to %s', infile, outfile) - if not self.dry_run: - msg = None - if check: - if os.path.islink(outfile): - msg = '%s is a symlink' % outfile - elif os.path.exists(outfile) and not os.path.isfile(outfile): - msg = '%s is a non-regular file' % outfile - if msg: - raise ValueError(msg + ' which would be overwritten') - shutil.copyfile(infile, outfile) - self.record_as_written(outfile) - - def copy_stream(self, instream, outfile, encoding=None): - assert not os.path.isdir(outfile) - self.ensure_dir(os.path.dirname(outfile)) - logger.info('Copying stream %s to %s', instream, outfile) - if not self.dry_run: - if encoding is None: - outstream = open(outfile, 'wb') - else: - outstream = codecs.open(outfile, 'w', encoding=encoding) - try: - shutil.copyfileobj(instream, outstream) - finally: - outstream.close() - self.record_as_written(outfile) - - def write_binary_file(self, path, data): - self.ensure_dir(os.path.dirname(path)) - if not self.dry_run: - if os.path.exists(path): - os.remove(path) - with open(path, 'wb') as f: - f.write(data) - self.record_as_written(path) - - def write_text_file(self, path, data, encoding): - self.write_binary_file(path, data.encode(encoding)) - - def set_mode(self, bits, mask, files): - if os.name == 'posix' or (os.name == 'java' and os._name == 'posix'): - # Set the executable bits (owner, group, and world) on - # all the files specified. - for f in files: - if self.dry_run: - logger.info("changing mode of %s", f) - else: - mode = (os.stat(f).st_mode | bits) & mask - logger.info("changing mode of %s to %o", f, mode) - os.chmod(f, mode) - - set_executable_mode = lambda s, f: s.set_mode(0o555, 0o7777, f) - - def ensure_dir(self, path): - path = os.path.abspath(path) - if path not in self.ensured and not os.path.exists(path): - self.ensured.add(path) - d, f = os.path.split(path) - self.ensure_dir(d) - logger.info('Creating %s' % path) - if not self.dry_run: - os.mkdir(path) - if self.record: - self.dirs_created.add(path) - - def byte_compile(self, path, optimize=False, force=False, prefix=None, hashed_invalidation=False): - dpath = cache_from_source(path, not optimize) - logger.info('Byte-compiling %s to %s', path, dpath) - if not self.dry_run: - if force or self.newer(path, dpath): - if not prefix: - diagpath = None - else: - assert path.startswith(prefix) - diagpath = path[len(prefix):] - compile_kwargs = {} - if hashed_invalidation and hasattr(py_compile, 'PycInvalidationMode'): - compile_kwargs['invalidation_mode'] = py_compile.PycInvalidationMode.CHECKED_HASH - py_compile.compile(path, dpath, diagpath, True, **compile_kwargs) # raise error - self.record_as_written(dpath) - return dpath - - def ensure_removed(self, path): - if os.path.exists(path): - if os.path.isdir(path) and not os.path.islink(path): - logger.debug('Removing directory tree at %s', path) - if not self.dry_run: - shutil.rmtree(path) - if self.record: - if path in self.dirs_created: - self.dirs_created.remove(path) - else: - if os.path.islink(path): - s = 'link' - else: - s = 'file' - logger.debug('Removing %s %s', s, path) - if not self.dry_run: - os.remove(path) - if self.record: - if path in self.files_written: - self.files_written.remove(path) - - def is_writable(self, path): - result = False - while not result: - if os.path.exists(path): - result = os.access(path, os.W_OK) - break - parent = os.path.dirname(path) - if parent == path: - break - path = parent - return result - - def commit(self): - """ - Commit recorded changes, turn off recording, return - changes. - """ - assert self.record - result = self.files_written, self.dirs_created - self._init_record() - return result - - def rollback(self): - if not self.dry_run: - for f in list(self.files_written): - if os.path.exists(f): - os.remove(f) - # dirs should all be empty now, except perhaps for - # __pycache__ subdirs - # reverse so that subdirs appear before their parents - dirs = sorted(self.dirs_created, reverse=True) - for d in dirs: - flist = os.listdir(d) - if flist: - assert flist == ['__pycache__'] - sd = os.path.join(d, flist[0]) - os.rmdir(sd) - os.rmdir(d) # should fail if non-empty - self._init_record() - -def resolve(module_name, dotted_path): - if module_name in sys.modules: - mod = sys.modules[module_name] - else: - mod = __import__(module_name) - if dotted_path is None: - result = mod - else: - parts = dotted_path.split('.') - result = getattr(mod, parts.pop(0)) - for p in parts: - result = getattr(result, p) - return result - - -class ExportEntry(object): - def __init__(self, name, prefix, suffix, flags): - self.name = name - self.prefix = prefix - self.suffix = suffix - self.flags = flags - - @cached_property - def value(self): - return resolve(self.prefix, self.suffix) - - def __repr__(self): # pragma: no cover - return '' % (self.name, self.prefix, - self.suffix, self.flags) - - def __eq__(self, other): - if not isinstance(other, ExportEntry): - result = False - else: - result = (self.name == other.name and - self.prefix == other.prefix and - self.suffix == other.suffix and - self.flags == other.flags) - return result - - __hash__ = object.__hash__ - - -ENTRY_RE = re.compile(r'''(?P(\w|[-.+])+) - \s*=\s*(?P(\w+)([:\.]\w+)*) - \s*(\[\s*(?P[\w-]+(=\w+)?(,\s*\w+(=\w+)?)*)\s*\])? - ''', re.VERBOSE) - -def get_export_entry(specification): - m = ENTRY_RE.search(specification) - if not m: - result = None - if '[' in specification or ']' in specification: - raise DistlibException("Invalid specification " - "'%s'" % specification) - else: - d = m.groupdict() - name = d['name'] - path = d['callable'] - colons = path.count(':') - if colons == 0: - prefix, suffix = path, None - else: - if colons != 1: - raise DistlibException("Invalid specification " - "'%s'" % specification) - prefix, suffix = path.split(':') - flags = d['flags'] - if flags is None: - if '[' in specification or ']' in specification: - raise DistlibException("Invalid specification " - "'%s'" % specification) - flags = [] - else: - flags = [f.strip() for f in flags.split(',')] - result = ExportEntry(name, prefix, suffix, flags) - return result - - -def get_cache_base(suffix=None): - """ - Return the default base location for distlib caches. If the directory does - not exist, it is created. Use the suffix provided for the base directory, - and default to '.distlib' if it isn't provided. - - On Windows, if LOCALAPPDATA is defined in the environment, then it is - assumed to be a directory, and will be the parent directory of the result. - On POSIX, and on Windows if LOCALAPPDATA is not defined, the user's home - directory - using os.expanduser('~') - will be the parent directory of - the result. - - The result is just the directory '.distlib' in the parent directory as - determined above, or with the name specified with ``suffix``. - """ - if suffix is None: - suffix = '.distlib' - if os.name == 'nt' and 'LOCALAPPDATA' in os.environ: - result = os.path.expandvars('$localappdata') - else: - # Assume posix, or old Windows - result = os.path.expanduser('~') - # we use 'isdir' instead of 'exists', because we want to - # fail if there's a file with that name - if os.path.isdir(result): - usable = os.access(result, os.W_OK) - if not usable: - logger.warning('Directory exists but is not writable: %s', result) - else: - try: - os.makedirs(result) - usable = True - except OSError: - logger.warning('Unable to create %s', result, exc_info=True) - usable = False - if not usable: - result = tempfile.mkdtemp() - logger.warning('Default location unusable, using %s', result) - return os.path.join(result, suffix) - - -def path_to_cache_dir(path): - """ - Convert an absolute path to a directory name for use in a cache. - - The algorithm used is: - - #. On Windows, any ``':'`` in the drive is replaced with ``'---'``. - #. Any occurrence of ``os.sep`` is replaced with ``'--'``. - #. ``'.cache'`` is appended. - """ - d, p = os.path.splitdrive(os.path.abspath(path)) - if d: - d = d.replace(':', '---') - p = p.replace(os.sep, '--') - return d + p + '.cache' - - -def ensure_slash(s): - if not s.endswith('/'): - return s + '/' - return s - - -def parse_credentials(netloc): - username = password = None - if '@' in netloc: - prefix, netloc = netloc.rsplit('@', 1) - if ':' not in prefix: - username = prefix - else: - username, password = prefix.split(':', 1) - if username: - username = unquote(username) - if password: - password = unquote(password) - return username, password, netloc - - -def get_process_umask(): - result = os.umask(0o22) - os.umask(result) - return result - -def is_string_sequence(seq): - result = True - i = None - for i, s in enumerate(seq): - if not isinstance(s, string_types): - result = False - break - assert i is not None - return result - -PROJECT_NAME_AND_VERSION = re.compile('([a-z0-9_]+([.-][a-z_][a-z0-9_]*)*)-' - '([a-z0-9_.+-]+)', re.I) -PYTHON_VERSION = re.compile(r'-py(\d\.?\d?)') - - -def split_filename(filename, project_name=None): - """ - Extract name, version, python version from a filename (no extension) - - Return name, version, pyver or None - """ - result = None - pyver = None - filename = unquote(filename).replace(' ', '-') - m = PYTHON_VERSION.search(filename) - if m: - pyver = m.group(1) - filename = filename[:m.start()] - if project_name and len(filename) > len(project_name) + 1: - m = re.match(re.escape(project_name) + r'\b', filename) - if m: - n = m.end() - result = filename[:n], filename[n + 1:], pyver - if result is None: - m = PROJECT_NAME_AND_VERSION.match(filename) - if m: - result = m.group(1), m.group(3), pyver - return result - -# Allow spaces in name because of legacy dists like "Twisted Core" -NAME_VERSION_RE = re.compile(r'(?P[\w .-]+)\s*' - r'\(\s*(?P[^\s)]+)\)$') - -def parse_name_and_version(p): - """ - A utility method used to get name and version from a string. - - From e.g. a Provides-Dist value. - - :param p: A value in a form 'foo (1.0)' - :return: The name and version as a tuple. - """ - m = NAME_VERSION_RE.match(p) - if not m: - raise DistlibException('Ill-formed name/version string: \'%s\'' % p) - d = m.groupdict() - return d['name'].strip().lower(), d['ver'] - -def get_extras(requested, available): - result = set() - requested = set(requested or []) - available = set(available or []) - if '*' in requested: - requested.remove('*') - result |= available - for r in requested: - if r == '-': - result.add(r) - elif r.startswith('-'): - unwanted = r[1:] - if unwanted not in available: - logger.warning('undeclared extra: %s' % unwanted) - if unwanted in result: - result.remove(unwanted) - else: - if r not in available: - logger.warning('undeclared extra: %s' % r) - result.add(r) - return result -# -# Extended metadata functionality -# - -def _get_external_data(url): - result = {} - try: - # urlopen might fail if it runs into redirections, - # because of Python issue #13696. Fixed in locators - # using a custom redirect handler. - resp = urlopen(url) - headers = resp.info() - ct = headers.get('Content-Type') - if not ct.startswith('application/json'): - logger.debug('Unexpected response for JSON request: %s', ct) - else: - reader = codecs.getreader('utf-8')(resp) - #data = reader.read().decode('utf-8') - #result = json.loads(data) - result = json.load(reader) - except Exception as e: - logger.exception('Failed to get external data for %s: %s', url, e) - return result - -_external_data_base_url = 'https://www.red-dove.com/pypi/projects/' - -def get_project_data(name): - url = '%s/%s/project.json' % (name[0].upper(), name) - url = urljoin(_external_data_base_url, url) - result = _get_external_data(url) - return result - -def get_package_data(name, version): - url = '%s/%s/package-%s.json' % (name[0].upper(), name, version) - url = urljoin(_external_data_base_url, url) - return _get_external_data(url) - - -class Cache(object): - """ - A class implementing a cache for resources that need to live in the file system - e.g. shared libraries. This class was moved from resources to here because it - could be used by other modules, e.g. the wheel module. - """ - - def __init__(self, base): - """ - Initialise an instance. - - :param base: The base directory where the cache should be located. - """ - # we use 'isdir' instead of 'exists', because we want to - # fail if there's a file with that name - if not os.path.isdir(base): # pragma: no cover - os.makedirs(base) - if (os.stat(base).st_mode & 0o77) != 0: - logger.warning('Directory \'%s\' is not private', base) - self.base = os.path.abspath(os.path.normpath(base)) - - def prefix_to_dir(self, prefix): - """ - Converts a resource prefix to a directory name in the cache. - """ - return path_to_cache_dir(prefix) - - def clear(self): - """ - Clear the cache. - """ - not_removed = [] - for fn in os.listdir(self.base): - fn = os.path.join(self.base, fn) - try: - if os.path.islink(fn) or os.path.isfile(fn): - os.remove(fn) - elif os.path.isdir(fn): - shutil.rmtree(fn) - except Exception: - not_removed.append(fn) - return not_removed - - -class EventMixin(object): - """ - A very simple publish/subscribe system. - """ - def __init__(self): - self._subscribers = {} - - def add(self, event, subscriber, append=True): - """ - Add a subscriber for an event. - - :param event: The name of an event. - :param subscriber: The subscriber to be added (and called when the - event is published). - :param append: Whether to append or prepend the subscriber to an - existing subscriber list for the event. - """ - subs = self._subscribers - if event not in subs: - subs[event] = deque([subscriber]) - else: - sq = subs[event] - if append: - sq.append(subscriber) - else: - sq.appendleft(subscriber) - - def remove(self, event, subscriber): - """ - Remove a subscriber for an event. - - :param event: The name of an event. - :param subscriber: The subscriber to be removed. - """ - subs = self._subscribers - if event not in subs: - raise ValueError('No subscribers: %r' % event) - subs[event].remove(subscriber) - - def get_subscribers(self, event): - """ - Return an iterator for the subscribers for an event. - :param event: The event to return subscribers for. - """ - return iter(self._subscribers.get(event, ())) - - def publish(self, event, *args, **kwargs): - """ - Publish a event and return a list of values returned by its - subscribers. - - :param event: The event to publish. - :param args: The positional arguments to pass to the event's - subscribers. - :param kwargs: The keyword arguments to pass to the event's - subscribers. - """ - result = [] - for subscriber in self.get_subscribers(event): - try: - value = subscriber(event, *args, **kwargs) - except Exception: - logger.exception('Exception during event publication') - value = None - result.append(value) - logger.debug('publish %s: args = %s, kwargs = %s, result = %s', - event, args, kwargs, result) - return result - -# -# Simple sequencing -# -class Sequencer(object): - def __init__(self): - self._preds = {} - self._succs = {} - self._nodes = set() # nodes with no preds/succs - - def add_node(self, node): - self._nodes.add(node) - - def remove_node(self, node, edges=False): - if node in self._nodes: - self._nodes.remove(node) - if edges: - for p in set(self._preds.get(node, ())): - self.remove(p, node) - for s in set(self._succs.get(node, ())): - self.remove(node, s) - # Remove empties - for k, v in list(self._preds.items()): - if not v: - del self._preds[k] - for k, v in list(self._succs.items()): - if not v: - del self._succs[k] - - def add(self, pred, succ): - assert pred != succ - self._preds.setdefault(succ, set()).add(pred) - self._succs.setdefault(pred, set()).add(succ) - - def remove(self, pred, succ): - assert pred != succ - try: - preds = self._preds[succ] - succs = self._succs[pred] - except KeyError: # pragma: no cover - raise ValueError('%r not a successor of anything' % succ) - try: - preds.remove(pred) - succs.remove(succ) - except KeyError: # pragma: no cover - raise ValueError('%r not a successor of %r' % (succ, pred)) - - def is_step(self, step): - return (step in self._preds or step in self._succs or - step in self._nodes) - - def get_steps(self, final): - if not self.is_step(final): - raise ValueError('Unknown: %r' % final) - result = [] - todo = [] - seen = set() - todo.append(final) - while todo: - step = todo.pop(0) - if step in seen: - # if a step was already seen, - # move it to the end (so it will appear earlier - # when reversed on return) ... but not for the - # final step, as that would be confusing for - # users - if step != final: - result.remove(step) - result.append(step) - else: - seen.add(step) - result.append(step) - preds = self._preds.get(step, ()) - todo.extend(preds) - return reversed(result) - - @property - def strong_connections(self): - #http://en.wikipedia.org/wiki/Tarjan%27s_strongly_connected_components_algorithm - index_counter = [0] - stack = [] - lowlinks = {} - index = {} - result = [] - - graph = self._succs - - def strongconnect(node): - # set the depth index for this node to the smallest unused index - index[node] = index_counter[0] - lowlinks[node] = index_counter[0] - index_counter[0] += 1 - stack.append(node) - - # Consider successors - try: - successors = graph[node] - except Exception: - successors = [] - for successor in successors: - if successor not in lowlinks: - # Successor has not yet been visited - strongconnect(successor) - lowlinks[node] = min(lowlinks[node],lowlinks[successor]) - elif successor in stack: - # the successor is in the stack and hence in the current - # strongly connected component (SCC) - lowlinks[node] = min(lowlinks[node],index[successor]) - - # If `node` is a root node, pop the stack and generate an SCC - if lowlinks[node] == index[node]: - connected_component = [] - - while True: - successor = stack.pop() - connected_component.append(successor) - if successor == node: break - component = tuple(connected_component) - # storing the result - result.append(component) - - for node in graph: - if node not in lowlinks: - strongconnect(node) - - return result - - @property - def dot(self): - result = ['digraph G {'] - for succ in self._preds: - preds = self._preds[succ] - for pred in preds: - result.append(' %s -> %s;' % (pred, succ)) - for node in self._nodes: - result.append(' %s;' % node) - result.append('}') - return '\n'.join(result) - -# -# Unarchiving functionality for zip, tar, tgz, tbz, whl -# - -ARCHIVE_EXTENSIONS = ('.tar.gz', '.tar.bz2', '.tar', '.zip', - '.tgz', '.tbz', '.whl') - -def unarchive(archive_filename, dest_dir, format=None, check=True): - - def check_path(path): - if not isinstance(path, text_type): - path = path.decode('utf-8') - p = os.path.abspath(os.path.join(dest_dir, path)) - if not p.startswith(dest_dir) or p[plen] != os.sep: - raise ValueError('path outside destination: %r' % p) - - dest_dir = os.path.abspath(dest_dir) - plen = len(dest_dir) - archive = None - if format is None: - if archive_filename.endswith(('.zip', '.whl')): - format = 'zip' - elif archive_filename.endswith(('.tar.gz', '.tgz')): - format = 'tgz' - mode = 'r:gz' - elif archive_filename.endswith(('.tar.bz2', '.tbz')): - format = 'tbz' - mode = 'r:bz2' - elif archive_filename.endswith('.tar'): - format = 'tar' - mode = 'r' - else: # pragma: no cover - raise ValueError('Unknown format for %r' % archive_filename) - try: - if format == 'zip': - archive = ZipFile(archive_filename, 'r') - if check: - names = archive.namelist() - for name in names: - check_path(name) - else: - archive = tarfile.open(archive_filename, mode) - if check: - names = archive.getnames() - for name in names: - check_path(name) - if format != 'zip' and sys.version_info[0] < 3: - # See Python issue 17153. If the dest path contains Unicode, - # tarfile extraction fails on Python 2.x if a member path name - # contains non-ASCII characters - it leads to an implicit - # bytes -> unicode conversion using ASCII to decode. - for tarinfo in archive.getmembers(): - if not isinstance(tarinfo.name, text_type): - tarinfo.name = tarinfo.name.decode('utf-8') - archive.extractall(dest_dir) - - finally: - if archive: - archive.close() - - -def zip_dir(directory): - """zip a directory tree into a BytesIO object""" - result = io.BytesIO() - dlen = len(directory) - with ZipFile(result, "w") as zf: - for root, dirs, files in os.walk(directory): - for name in files: - full = os.path.join(root, name) - rel = root[dlen:] - dest = os.path.join(rel, name) - zf.write(full, dest) - return result - -# -# Simple progress bar -# - -UNITS = ('', 'K', 'M', 'G','T','P') - - -class Progress(object): - unknown = 'UNKNOWN' - - def __init__(self, minval=0, maxval=100): - assert maxval is None or maxval >= minval - self.min = self.cur = minval - self.max = maxval - self.started = None - self.elapsed = 0 - self.done = False - - def update(self, curval): - assert self.min <= curval - assert self.max is None or curval <= self.max - self.cur = curval - now = time.time() - if self.started is None: - self.started = now - else: - self.elapsed = now - self.started - - def increment(self, incr): - assert incr >= 0 - self.update(self.cur + incr) - - def start(self): - self.update(self.min) - return self - - def stop(self): - if self.max is not None: - self.update(self.max) - self.done = True - - @property - def maximum(self): - return self.unknown if self.max is None else self.max - - @property - def percentage(self): - if self.done: - result = '100 %' - elif self.max is None: - result = ' ?? %' - else: - v = 100.0 * (self.cur - self.min) / (self.max - self.min) - result = '%3d %%' % v - return result - - def format_duration(self, duration): - if (duration <= 0) and self.max is None or self.cur == self.min: - result = '??:??:??' - #elif duration < 1: - # result = '--:--:--' - else: - result = time.strftime('%H:%M:%S', time.gmtime(duration)) - return result - - @property - def ETA(self): - if self.done: - prefix = 'Done' - t = self.elapsed - #import pdb; pdb.set_trace() - else: - prefix = 'ETA ' - if self.max is None: - t = -1 - elif self.elapsed == 0 or (self.cur == self.min): - t = 0 - else: - #import pdb; pdb.set_trace() - t = float(self.max - self.min) - t /= self.cur - self.min - t = (t - 1) * self.elapsed - return '%s: %s' % (prefix, self.format_duration(t)) - - @property - def speed(self): - if self.elapsed == 0: - result = 0.0 - else: - result = (self.cur - self.min) / self.elapsed - for unit in UNITS: - if result < 1000: - break - result /= 1000.0 - return '%d %sB/s' % (result, unit) - -# -# Glob functionality -# - -RICH_GLOB = re.compile(r'\{([^}]*)\}') -_CHECK_RECURSIVE_GLOB = re.compile(r'[^/\\,{]\*\*|\*\*[^/\\,}]') -_CHECK_MISMATCH_SET = re.compile(r'^[^{]*\}|\{[^}]*$') - - -def iglob(path_glob): - """Extended globbing function that supports ** and {opt1,opt2,opt3}.""" - if _CHECK_RECURSIVE_GLOB.search(path_glob): - msg = """invalid glob %r: recursive glob "**" must be used alone""" - raise ValueError(msg % path_glob) - if _CHECK_MISMATCH_SET.search(path_glob): - msg = """invalid glob %r: mismatching set marker '{' or '}'""" - raise ValueError(msg % path_glob) - return _iglob(path_glob) - - -def _iglob(path_glob): - rich_path_glob = RICH_GLOB.split(path_glob, 1) - if len(rich_path_glob) > 1: - assert len(rich_path_glob) == 3, rich_path_glob - prefix, set, suffix = rich_path_glob - for item in set.split(','): - for path in _iglob(''.join((prefix, item, suffix))): - yield path - else: - if '**' not in path_glob: - for item in std_iglob(path_glob): - yield item - else: - prefix, radical = path_glob.split('**', 1) - if prefix == '': - prefix = '.' - if radical == '': - radical = '*' - else: - # we support both - radical = radical.lstrip('/') - radical = radical.lstrip('\\') - for path, dir, files in os.walk(prefix): - path = os.path.normpath(path) - for fn in _iglob(os.path.join(path, radical)): - yield fn - -if ssl: - from .compat import (HTTPSHandler as BaseHTTPSHandler, match_hostname, - CertificateError) - - -# -# HTTPSConnection which verifies certificates/matches domains -# - - class HTTPSConnection(httplib.HTTPSConnection): - ca_certs = None # set this to the path to the certs file (.pem) - check_domain = True # only used if ca_certs is not None - - # noinspection PyPropertyAccess - def connect(self): - sock = socket.create_connection((self.host, self.port), self.timeout) - if getattr(self, '_tunnel_host', False): - self.sock = sock - self._tunnel() - - context = ssl.SSLContext(ssl.PROTOCOL_SSLv23) - if hasattr(ssl, 'OP_NO_SSLv2'): - context.options |= ssl.OP_NO_SSLv2 - if self.cert_file: - context.load_cert_chain(self.cert_file, self.key_file) - kwargs = {} - if self.ca_certs: - context.verify_mode = ssl.CERT_REQUIRED - context.load_verify_locations(cafile=self.ca_certs) - if getattr(ssl, 'HAS_SNI', False): - kwargs['server_hostname'] = self.host - - self.sock = context.wrap_socket(sock, **kwargs) - if self.ca_certs and self.check_domain: - try: - match_hostname(self.sock.getpeercert(), self.host) - logger.debug('Host verified: %s', self.host) - except CertificateError: # pragma: no cover - self.sock.shutdown(socket.SHUT_RDWR) - self.sock.close() - raise - - class HTTPSHandler(BaseHTTPSHandler): - def __init__(self, ca_certs, check_domain=True): - BaseHTTPSHandler.__init__(self) - self.ca_certs = ca_certs - self.check_domain = check_domain - - def _conn_maker(self, *args, **kwargs): - """ - This is called to create a connection instance. Normally you'd - pass a connection class to do_open, but it doesn't actually check for - a class, and just expects a callable. As long as we behave just as a - constructor would have, we should be OK. If it ever changes so that - we *must* pass a class, we'll create an UnsafeHTTPSConnection class - which just sets check_domain to False in the class definition, and - choose which one to pass to do_open. - """ - result = HTTPSConnection(*args, **kwargs) - if self.ca_certs: - result.ca_certs = self.ca_certs - result.check_domain = self.check_domain - return result - - def https_open(self, req): - try: - return self.do_open(self._conn_maker, req) - except URLError as e: - if 'certificate verify failed' in str(e.reason): - raise CertificateError('Unable to verify server certificate ' - 'for %s' % req.host) - else: - raise - - # - # To prevent against mixing HTTP traffic with HTTPS (examples: A Man-In-The- - # Middle proxy using HTTP listens on port 443, or an index mistakenly serves - # HTML containing a http://xyz link when it should be https://xyz), - # you can use the following handler class, which does not allow HTTP traffic. - # - # It works by inheriting from HTTPHandler - so build_opener won't add a - # handler for HTTP itself. - # - class HTTPSOnlyHandler(HTTPSHandler, HTTPHandler): - def http_open(self, req): - raise URLError('Unexpected HTTP request on what should be a secure ' - 'connection: %s' % req) - -# -# XML-RPC with timeouts -# -class Transport(xmlrpclib.Transport): - def __init__(self, timeout, use_datetime=0): - self.timeout = timeout - xmlrpclib.Transport.__init__(self, use_datetime) - - def make_connection(self, host): - h, eh, x509 = self.get_host_info(host) - if not self._connection or host != self._connection[0]: - self._extra_headers = eh - self._connection = host, httplib.HTTPConnection(h) - return self._connection[1] - -if ssl: - class SafeTransport(xmlrpclib.SafeTransport): - def __init__(self, timeout, use_datetime=0): - self.timeout = timeout - xmlrpclib.SafeTransport.__init__(self, use_datetime) - - def make_connection(self, host): - h, eh, kwargs = self.get_host_info(host) - if not kwargs: - kwargs = {} - kwargs['timeout'] = self.timeout - if not self._connection or host != self._connection[0]: - self._extra_headers = eh - self._connection = host, httplib.HTTPSConnection(h, None, - **kwargs) - return self._connection[1] - - -class ServerProxy(xmlrpclib.ServerProxy): - def __init__(self, uri, **kwargs): - self.timeout = timeout = kwargs.pop('timeout', None) - # The above classes only come into play if a timeout - # is specified - if timeout is not None: - # scheme = splittype(uri) # deprecated as of Python 3.8 - scheme = urlparse(uri)[0] - use_datetime = kwargs.get('use_datetime', 0) - if scheme == 'https': - tcls = SafeTransport - else: - tcls = Transport - kwargs['transport'] = t = tcls(timeout, use_datetime=use_datetime) - self.transport = t - xmlrpclib.ServerProxy.__init__(self, uri, **kwargs) - -# -# CSV functionality. This is provided because on 2.x, the csv module can't -# handle Unicode. However, we need to deal with Unicode in e.g. RECORD files. -# - -def _csv_open(fn, mode, **kwargs): - if sys.version_info[0] < 3: - mode += 'b' - else: - kwargs['newline'] = '' - # Python 3 determines encoding from locale. Force 'utf-8' - # file encoding to match other forced utf-8 encoding - kwargs['encoding'] = 'utf-8' - return open(fn, mode, **kwargs) - - -class CSVBase(object): - defaults = { - 'delimiter': str(','), # The strs are used because we need native - 'quotechar': str('"'), # str in the csv API (2.x won't take - 'lineterminator': str('\n') # Unicode) - } - - def __enter__(self): - return self - - def __exit__(self, *exc_info): - self.stream.close() - - -class CSVReader(CSVBase): - def __init__(self, **kwargs): - if 'stream' in kwargs: - stream = kwargs['stream'] - if sys.version_info[0] >= 3: - # needs to be a text stream - stream = codecs.getreader('utf-8')(stream) - self.stream = stream - else: - self.stream = _csv_open(kwargs['path'], 'r') - self.reader = csv.reader(self.stream, **self.defaults) - - def __iter__(self): - return self - - def next(self): - result = next(self.reader) - if sys.version_info[0] < 3: - for i, item in enumerate(result): - if not isinstance(item, text_type): - result[i] = item.decode('utf-8') - return result - - __next__ = next - -class CSVWriter(CSVBase): - def __init__(self, fn, **kwargs): - self.stream = _csv_open(fn, 'w') - self.writer = csv.writer(self.stream, **self.defaults) - - def writerow(self, row): - if sys.version_info[0] < 3: - r = [] - for item in row: - if isinstance(item, text_type): - item = item.encode('utf-8') - r.append(item) - row = r - self.writer.writerow(row) - -# -# Configurator functionality -# - -class Configurator(BaseConfigurator): - - value_converters = dict(BaseConfigurator.value_converters) - value_converters['inc'] = 'inc_convert' - - def __init__(self, config, base=None): - super(Configurator, self).__init__(config) - self.base = base or os.getcwd() - - def configure_custom(self, config): - def convert(o): - if isinstance(o, (list, tuple)): - result = type(o)([convert(i) for i in o]) - elif isinstance(o, dict): - if '()' in o: - result = self.configure_custom(o) - else: - result = {} - for k in o: - result[k] = convert(o[k]) - else: - result = self.convert(o) - return result - - c = config.pop('()') - if not callable(c): - c = self.resolve(c) - props = config.pop('.', None) - # Check for valid identifiers - args = config.pop('[]', ()) - if args: - args = tuple([convert(o) for o in args]) - items = [(k, convert(config[k])) for k in config if valid_ident(k)] - kwargs = dict(items) - result = c(*args, **kwargs) - if props: - for n, v in props.items(): - setattr(result, n, convert(v)) - return result - - def __getitem__(self, key): - result = self.config[key] - if isinstance(result, dict) and '()' in result: - self.config[key] = result = self.configure_custom(result) - return result - - def inc_convert(self, value): - """Default converter for the inc:// protocol.""" - if not os.path.isabs(value): - value = os.path.join(self.base, value) - with codecs.open(value, 'r', encoding='utf-8') as f: - result = json.load(f) - return result - - -class SubprocessMixin(object): - """ - Mixin for running subprocesses and capturing their output - """ - def __init__(self, verbose=False, progress=None): - self.verbose = verbose - self.progress = progress - - def reader(self, stream, context): - """ - Read lines from a subprocess' output stream and either pass to a progress - callable (if specified) or write progress information to sys.stderr. - """ - progress = self.progress - verbose = self.verbose - while True: - s = stream.readline() - if not s: - break - if progress is not None: - progress(s, context) - else: - if not verbose: - sys.stderr.write('.') - else: - sys.stderr.write(s.decode('utf-8')) - sys.stderr.flush() - stream.close() - - def run_command(self, cmd, **kwargs): - p = subprocess.Popen(cmd, stdout=subprocess.PIPE, - stderr=subprocess.PIPE, **kwargs) - t1 = threading.Thread(target=self.reader, args=(p.stdout, 'stdout')) - t1.start() - t2 = threading.Thread(target=self.reader, args=(p.stderr, 'stderr')) - t2.start() - p.wait() - t1.join() - t2.join() - if self.progress is not None: - self.progress('done.', 'main') - elif self.verbose: - sys.stderr.write('done.\n') - return p - - -def normalize_name(name): - """Normalize a python package name a la PEP 503""" - # https://www.python.org/dev/peps/pep-0503/#normalized-names - return re.sub('[-_.]+', '-', name).lower() - -# def _get_pypirc_command(): - # """ - # Get the distutils command for interacting with PyPI configurations. - # :return: the command. - # """ - # from distutils.core import Distribution - # from distutils.config import PyPIRCCommand - # d = Distribution() - # return PyPIRCCommand(d) - -class PyPIRCFile(object): - - DEFAULT_REPOSITORY = 'https://upload.pypi.org/legacy/' - DEFAULT_REALM = 'pypi' - - def __init__(self, fn=None, url=None): - if fn is None: - fn = os.path.join(os.path.expanduser('~'), '.pypirc') - self.filename = fn - self.url = url - - def read(self): - result = {} - - if os.path.exists(self.filename): - repository = self.url or self.DEFAULT_REPOSITORY - - config = configparser.RawConfigParser() - config.read(self.filename) - sections = config.sections() - if 'distutils' in sections: - # let's get the list of servers - index_servers = config.get('distutils', 'index-servers') - _servers = [server.strip() for server in - index_servers.split('\n') - if server.strip() != ''] - if _servers == []: - # nothing set, let's try to get the default pypi - if 'pypi' in sections: - _servers = ['pypi'] - else: - for server in _servers: - result = {'server': server} - result['username'] = config.get(server, 'username') - - # optional params - for key, default in (('repository', self.DEFAULT_REPOSITORY), - ('realm', self.DEFAULT_REALM), - ('password', None)): - if config.has_option(server, key): - result[key] = config.get(server, key) - else: - result[key] = default - - # work around people having "repository" for the "pypi" - # section of their config set to the HTTP (rather than - # HTTPS) URL - if (server == 'pypi' and - repository in (self.DEFAULT_REPOSITORY, 'pypi')): - result['repository'] = self.DEFAULT_REPOSITORY - elif (result['server'] != repository and - result['repository'] != repository): - result = {} - elif 'server-login' in sections: - # old format - server = 'server-login' - if config.has_option(server, 'repository'): - repository = config.get(server, 'repository') - else: - repository = self.DEFAULT_REPOSITORY - result = { - 'username': config.get(server, 'username'), - 'password': config.get(server, 'password'), - 'repository': repository, - 'server': server, - 'realm': self.DEFAULT_REALM - } - return result - - def update(self, username, password): - # import pdb; pdb.set_trace() - config = configparser.RawConfigParser() - fn = self.filename - config.read(fn) - if not config.has_section('pypi'): - config.add_section('pypi') - config.set('pypi', 'username', username) - config.set('pypi', 'password', password) - with open(fn, 'w') as f: - config.write(f) - -def _load_pypirc(index): - """ - Read the PyPI access configuration as supported by distutils. - """ - return PyPIRCFile(url=index.url).read() - -def _store_pypirc(index): - PyPIRCFile().update(index.username, index.password) - -# -# get_platform()/get_host_platform() copied from Python 3.10.a0 source, with some minor -# tweaks -# - -def get_host_platform(): - """Return a string that identifies the current platform. This is used mainly to - distinguish platform-specific build directories and platform-specific built - distributions. Typically includes the OS name and version and the - architecture (as supplied by 'os.uname()'), although the exact information - included depends on the OS; eg. on Linux, the kernel version isn't - particularly important. - - Examples of returned values: - linux-i586 - linux-alpha (?) - solaris-2.6-sun4u - - Windows will return one of: - win-amd64 (64bit Windows on AMD64 (aka x86_64, Intel64, EM64T, etc) - win32 (all others - specifically, sys.platform is returned) - - For other non-POSIX platforms, currently just returns 'sys.platform'. - - """ - if os.name == 'nt': - if 'amd64' in sys.version.lower(): - return 'win-amd64' - if '(arm)' in sys.version.lower(): - return 'win-arm32' - if '(arm64)' in sys.version.lower(): - return 'win-arm64' - return sys.platform - - # Set for cross builds explicitly - if "_PYTHON_HOST_PLATFORM" in os.environ: - return os.environ["_PYTHON_HOST_PLATFORM"] - - if os.name != 'posix' or not hasattr(os, 'uname'): - # XXX what about the architecture? NT is Intel or Alpha, - # Mac OS is M68k or PPC, etc. - return sys.platform - - # Try to distinguish various flavours of Unix - - (osname, host, release, version, machine) = os.uname() - - # Convert the OS name to lowercase, remove '/' characters, and translate - # spaces (for "Power Macintosh") - osname = osname.lower().replace('/', '') - machine = machine.replace(' ', '_').replace('/', '-') - - if osname[:5] == 'linux': - # At least on Linux/Intel, 'machine' is the processor -- - # i386, etc. - # XXX what about Alpha, SPARC, etc? - return "%s-%s" % (osname, machine) - - elif osname[:5] == 'sunos': - if release[0] >= '5': # SunOS 5 == Solaris 2 - osname = 'solaris' - release = '%d.%s' % (int(release[0]) - 3, release[2:]) - # We can't use 'platform.architecture()[0]' because a - # bootstrap problem. We use a dict to get an error - # if some suspicious happens. - bitness = {2147483647:'32bit', 9223372036854775807:'64bit'} - machine += '.%s' % bitness[sys.maxsize] - # fall through to standard osname-release-machine representation - elif osname[:3] == 'aix': - from _aix_support import aix_platform - return aix_platform() - elif osname[:6] == 'cygwin': - osname = 'cygwin' - rel_re = re.compile (r'[\d.]+', re.ASCII) - m = rel_re.match(release) - if m: - release = m.group() - elif osname[:6] == 'darwin': - import _osx_support, distutils.sysconfig - osname, release, machine = _osx_support.get_platform_osx( - distutils.sysconfig.get_config_vars(), - osname, release, machine) - - return '%s-%s-%s' % (osname, release, machine) - - -_TARGET_TO_PLAT = { - 'x86' : 'win32', - 'x64' : 'win-amd64', - 'arm' : 'win-arm32', -} - - -def get_platform(): - if os.name != 'nt': - return get_host_platform() - cross_compilation_target = os.environ.get('VSCMD_ARG_TGT_ARCH') - if cross_compilation_target not in _TARGET_TO_PLAT: - return get_host_platform() - return _TARGET_TO_PLAT[cross_compilation_target] diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/distlib/version.py b/venv/lib/python3.11/site-packages/pip/_vendor/distlib/version.py deleted file mode 100644 index c7c8bb6..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/distlib/version.py +++ /dev/null @@ -1,739 +0,0 @@ -# -*- coding: utf-8 -*- -# -# Copyright (C) 2012-2017 The Python Software Foundation. -# See LICENSE.txt and CONTRIBUTORS.txt. -# -""" -Implementation of a flexible versioning scheme providing support for PEP-440, -setuptools-compatible and semantic versioning. -""" - -import logging -import re - -from .compat import string_types -from .util import parse_requirement - -__all__ = ['NormalizedVersion', 'NormalizedMatcher', - 'LegacyVersion', 'LegacyMatcher', - 'SemanticVersion', 'SemanticMatcher', - 'UnsupportedVersionError', 'get_scheme'] - -logger = logging.getLogger(__name__) - - -class UnsupportedVersionError(ValueError): - """This is an unsupported version.""" - pass - - -class Version(object): - def __init__(self, s): - self._string = s = s.strip() - self._parts = parts = self.parse(s) - assert isinstance(parts, tuple) - assert len(parts) > 0 - - def parse(self, s): - raise NotImplementedError('please implement in a subclass') - - def _check_compatible(self, other): - if type(self) != type(other): - raise TypeError('cannot compare %r and %r' % (self, other)) - - def __eq__(self, other): - self._check_compatible(other) - return self._parts == other._parts - - def __ne__(self, other): - return not self.__eq__(other) - - def __lt__(self, other): - self._check_compatible(other) - return self._parts < other._parts - - def __gt__(self, other): - return not (self.__lt__(other) or self.__eq__(other)) - - def __le__(self, other): - return self.__lt__(other) or self.__eq__(other) - - def __ge__(self, other): - return self.__gt__(other) or self.__eq__(other) - - # See http://docs.python.org/reference/datamodel#object.__hash__ - def __hash__(self): - return hash(self._parts) - - def __repr__(self): - return "%s('%s')" % (self.__class__.__name__, self._string) - - def __str__(self): - return self._string - - @property - def is_prerelease(self): - raise NotImplementedError('Please implement in subclasses.') - - -class Matcher(object): - version_class = None - - # value is either a callable or the name of a method - _operators = { - '<': lambda v, c, p: v < c, - '>': lambda v, c, p: v > c, - '<=': lambda v, c, p: v == c or v < c, - '>=': lambda v, c, p: v == c or v > c, - '==': lambda v, c, p: v == c, - '===': lambda v, c, p: v == c, - # by default, compatible => >=. - '~=': lambda v, c, p: v == c or v > c, - '!=': lambda v, c, p: v != c, - } - - # this is a method only to support alternative implementations - # via overriding - def parse_requirement(self, s): - return parse_requirement(s) - - def __init__(self, s): - if self.version_class is None: - raise ValueError('Please specify a version class') - self._string = s = s.strip() - r = self.parse_requirement(s) - if not r: - raise ValueError('Not valid: %r' % s) - self.name = r.name - self.key = self.name.lower() # for case-insensitive comparisons - clist = [] - if r.constraints: - # import pdb; pdb.set_trace() - for op, s in r.constraints: - if s.endswith('.*'): - if op not in ('==', '!='): - raise ValueError('\'.*\' not allowed for ' - '%r constraints' % op) - # Could be a partial version (e.g. for '2.*') which - # won't parse as a version, so keep it as a string - vn, prefix = s[:-2], True - # Just to check that vn is a valid version - self.version_class(vn) - else: - # Should parse as a version, so we can create an - # instance for the comparison - vn, prefix = self.version_class(s), False - clist.append((op, vn, prefix)) - self._parts = tuple(clist) - - def match(self, version): - """ - Check if the provided version matches the constraints. - - :param version: The version to match against this instance. - :type version: String or :class:`Version` instance. - """ - if isinstance(version, string_types): - version = self.version_class(version) - for operator, constraint, prefix in self._parts: - f = self._operators.get(operator) - if isinstance(f, string_types): - f = getattr(self, f) - if not f: - msg = ('%r not implemented ' - 'for %s' % (operator, self.__class__.__name__)) - raise NotImplementedError(msg) - if not f(version, constraint, prefix): - return False - return True - - @property - def exact_version(self): - result = None - if len(self._parts) == 1 and self._parts[0][0] in ('==', '==='): - result = self._parts[0][1] - return result - - def _check_compatible(self, other): - if type(self) != type(other) or self.name != other.name: - raise TypeError('cannot compare %s and %s' % (self, other)) - - def __eq__(self, other): - self._check_compatible(other) - return self.key == other.key and self._parts == other._parts - - def __ne__(self, other): - return not self.__eq__(other) - - # See http://docs.python.org/reference/datamodel#object.__hash__ - def __hash__(self): - return hash(self.key) + hash(self._parts) - - def __repr__(self): - return "%s(%r)" % (self.__class__.__name__, self._string) - - def __str__(self): - return self._string - - -PEP440_VERSION_RE = re.compile(r'^v?(\d+!)?(\d+(\.\d+)*)((a|b|c|rc)(\d+))?' - r'(\.(post)(\d+))?(\.(dev)(\d+))?' - r'(\+([a-zA-Z\d]+(\.[a-zA-Z\d]+)?))?$') - - -def _pep_440_key(s): - s = s.strip() - m = PEP440_VERSION_RE.match(s) - if not m: - raise UnsupportedVersionError('Not a valid version: %s' % s) - groups = m.groups() - nums = tuple(int(v) for v in groups[1].split('.')) - while len(nums) > 1 and nums[-1] == 0: - nums = nums[:-1] - - if not groups[0]: - epoch = 0 - else: - epoch = int(groups[0][:-1]) - pre = groups[4:6] - post = groups[7:9] - dev = groups[10:12] - local = groups[13] - if pre == (None, None): - pre = () - else: - pre = pre[0], int(pre[1]) - if post == (None, None): - post = () - else: - post = post[0], int(post[1]) - if dev == (None, None): - dev = () - else: - dev = dev[0], int(dev[1]) - if local is None: - local = () - else: - parts = [] - for part in local.split('.'): - # to ensure that numeric compares as > lexicographic, avoid - # comparing them directly, but encode a tuple which ensures - # correct sorting - if part.isdigit(): - part = (1, int(part)) - else: - part = (0, part) - parts.append(part) - local = tuple(parts) - if not pre: - # either before pre-release, or final release and after - if not post and dev: - # before pre-release - pre = ('a', -1) # to sort before a0 - else: - pre = ('z',) # to sort after all pre-releases - # now look at the state of post and dev. - if not post: - post = ('_',) # sort before 'a' - if not dev: - dev = ('final',) - - #print('%s -> %s' % (s, m.groups())) - return epoch, nums, pre, post, dev, local - - -_normalized_key = _pep_440_key - - -class NormalizedVersion(Version): - """A rational version. - - Good: - 1.2 # equivalent to "1.2.0" - 1.2.0 - 1.2a1 - 1.2.3a2 - 1.2.3b1 - 1.2.3c1 - 1.2.3.4 - TODO: fill this out - - Bad: - 1 # minimum two numbers - 1.2a # release level must have a release serial - 1.2.3b - """ - def parse(self, s): - result = _normalized_key(s) - # _normalized_key loses trailing zeroes in the release - # clause, since that's needed to ensure that X.Y == X.Y.0 == X.Y.0.0 - # However, PEP 440 prefix matching needs it: for example, - # (~= 1.4.5.0) matches differently to (~= 1.4.5.0.0). - m = PEP440_VERSION_RE.match(s) # must succeed - groups = m.groups() - self._release_clause = tuple(int(v) for v in groups[1].split('.')) - return result - - PREREL_TAGS = set(['a', 'b', 'c', 'rc', 'dev']) - - @property - def is_prerelease(self): - return any(t[0] in self.PREREL_TAGS for t in self._parts if t) - - -def _match_prefix(x, y): - x = str(x) - y = str(y) - if x == y: - return True - if not x.startswith(y): - return False - n = len(y) - return x[n] == '.' - - -class NormalizedMatcher(Matcher): - version_class = NormalizedVersion - - # value is either a callable or the name of a method - _operators = { - '~=': '_match_compatible', - '<': '_match_lt', - '>': '_match_gt', - '<=': '_match_le', - '>=': '_match_ge', - '==': '_match_eq', - '===': '_match_arbitrary', - '!=': '_match_ne', - } - - def _adjust_local(self, version, constraint, prefix): - if prefix: - strip_local = '+' not in constraint and version._parts[-1] - else: - # both constraint and version are - # NormalizedVersion instances. - # If constraint does not have a local component, - # ensure the version doesn't, either. - strip_local = not constraint._parts[-1] and version._parts[-1] - if strip_local: - s = version._string.split('+', 1)[0] - version = self.version_class(s) - return version, constraint - - def _match_lt(self, version, constraint, prefix): - version, constraint = self._adjust_local(version, constraint, prefix) - if version >= constraint: - return False - release_clause = constraint._release_clause - pfx = '.'.join([str(i) for i in release_clause]) - return not _match_prefix(version, pfx) - - def _match_gt(self, version, constraint, prefix): - version, constraint = self._adjust_local(version, constraint, prefix) - if version <= constraint: - return False - release_clause = constraint._release_clause - pfx = '.'.join([str(i) for i in release_clause]) - return not _match_prefix(version, pfx) - - def _match_le(self, version, constraint, prefix): - version, constraint = self._adjust_local(version, constraint, prefix) - return version <= constraint - - def _match_ge(self, version, constraint, prefix): - version, constraint = self._adjust_local(version, constraint, prefix) - return version >= constraint - - def _match_eq(self, version, constraint, prefix): - version, constraint = self._adjust_local(version, constraint, prefix) - if not prefix: - result = (version == constraint) - else: - result = _match_prefix(version, constraint) - return result - - def _match_arbitrary(self, version, constraint, prefix): - return str(version) == str(constraint) - - def _match_ne(self, version, constraint, prefix): - version, constraint = self._adjust_local(version, constraint, prefix) - if not prefix: - result = (version != constraint) - else: - result = not _match_prefix(version, constraint) - return result - - def _match_compatible(self, version, constraint, prefix): - version, constraint = self._adjust_local(version, constraint, prefix) - if version == constraint: - return True - if version < constraint: - return False -# if not prefix: -# return True - release_clause = constraint._release_clause - if len(release_clause) > 1: - release_clause = release_clause[:-1] - pfx = '.'.join([str(i) for i in release_clause]) - return _match_prefix(version, pfx) - -_REPLACEMENTS = ( - (re.compile('[.+-]$'), ''), # remove trailing puncts - (re.compile(r'^[.](\d)'), r'0.\1'), # .N -> 0.N at start - (re.compile('^[.-]'), ''), # remove leading puncts - (re.compile(r'^\((.*)\)$'), r'\1'), # remove parentheses - (re.compile(r'^v(ersion)?\s*(\d+)'), r'\2'), # remove leading v(ersion) - (re.compile(r'^r(ev)?\s*(\d+)'), r'\2'), # remove leading v(ersion) - (re.compile('[.]{2,}'), '.'), # multiple runs of '.' - (re.compile(r'\b(alfa|apha)\b'), 'alpha'), # misspelt alpha - (re.compile(r'\b(pre-alpha|prealpha)\b'), - 'pre.alpha'), # standardise - (re.compile(r'\(beta\)$'), 'beta'), # remove parentheses -) - -_SUFFIX_REPLACEMENTS = ( - (re.compile('^[:~._+-]+'), ''), # remove leading puncts - (re.compile('[,*")([\\]]'), ''), # remove unwanted chars - (re.compile('[~:+_ -]'), '.'), # replace illegal chars - (re.compile('[.]{2,}'), '.'), # multiple runs of '.' - (re.compile(r'\.$'), ''), # trailing '.' -) - -_NUMERIC_PREFIX = re.compile(r'(\d+(\.\d+)*)') - - -def _suggest_semantic_version(s): - """ - Try to suggest a semantic form for a version for which - _suggest_normalized_version couldn't come up with anything. - """ - result = s.strip().lower() - for pat, repl in _REPLACEMENTS: - result = pat.sub(repl, result) - if not result: - result = '0.0.0' - - # Now look for numeric prefix, and separate it out from - # the rest. - #import pdb; pdb.set_trace() - m = _NUMERIC_PREFIX.match(result) - if not m: - prefix = '0.0.0' - suffix = result - else: - prefix = m.groups()[0].split('.') - prefix = [int(i) for i in prefix] - while len(prefix) < 3: - prefix.append(0) - if len(prefix) == 3: - suffix = result[m.end():] - else: - suffix = '.'.join([str(i) for i in prefix[3:]]) + result[m.end():] - prefix = prefix[:3] - prefix = '.'.join([str(i) for i in prefix]) - suffix = suffix.strip() - if suffix: - #import pdb; pdb.set_trace() - # massage the suffix. - for pat, repl in _SUFFIX_REPLACEMENTS: - suffix = pat.sub(repl, suffix) - - if not suffix: - result = prefix - else: - sep = '-' if 'dev' in suffix else '+' - result = prefix + sep + suffix - if not is_semver(result): - result = None - return result - - -def _suggest_normalized_version(s): - """Suggest a normalized version close to the given version string. - - If you have a version string that isn't rational (i.e. NormalizedVersion - doesn't like it) then you might be able to get an equivalent (or close) - rational version from this function. - - This does a number of simple normalizations to the given string, based - on observation of versions currently in use on PyPI. Given a dump of - those version during PyCon 2009, 4287 of them: - - 2312 (53.93%) match NormalizedVersion without change - with the automatic suggestion - - 3474 (81.04%) match when using this suggestion method - - @param s {str} An irrational version string. - @returns A rational version string, or None, if couldn't determine one. - """ - try: - _normalized_key(s) - return s # already rational - except UnsupportedVersionError: - pass - - rs = s.lower() - - # part of this could use maketrans - for orig, repl in (('-alpha', 'a'), ('-beta', 'b'), ('alpha', 'a'), - ('beta', 'b'), ('rc', 'c'), ('-final', ''), - ('-pre', 'c'), - ('-release', ''), ('.release', ''), ('-stable', ''), - ('+', '.'), ('_', '.'), (' ', ''), ('.final', ''), - ('final', '')): - rs = rs.replace(orig, repl) - - # if something ends with dev or pre, we add a 0 - rs = re.sub(r"pre$", r"pre0", rs) - rs = re.sub(r"dev$", r"dev0", rs) - - # if we have something like "b-2" or "a.2" at the end of the - # version, that is probably beta, alpha, etc - # let's remove the dash or dot - rs = re.sub(r"([abc]|rc)[\-\.](\d+)$", r"\1\2", rs) - - # 1.0-dev-r371 -> 1.0.dev371 - # 0.1-dev-r79 -> 0.1.dev79 - rs = re.sub(r"[\-\.](dev)[\-\.]?r?(\d+)$", r".\1\2", rs) - - # Clean: 2.0.a.3, 2.0.b1, 0.9.0~c1 - rs = re.sub(r"[.~]?([abc])\.?", r"\1", rs) - - # Clean: v0.3, v1.0 - if rs.startswith('v'): - rs = rs[1:] - - # Clean leading '0's on numbers. - #TODO: unintended side-effect on, e.g., "2003.05.09" - # PyPI stats: 77 (~2%) better - rs = re.sub(r"\b0+(\d+)(?!\d)", r"\1", rs) - - # Clean a/b/c with no version. E.g. "1.0a" -> "1.0a0". Setuptools infers - # zero. - # PyPI stats: 245 (7.56%) better - rs = re.sub(r"(\d+[abc])$", r"\g<1>0", rs) - - # the 'dev-rNNN' tag is a dev tag - rs = re.sub(r"\.?(dev-r|dev\.r)\.?(\d+)$", r".dev\2", rs) - - # clean the - when used as a pre delimiter - rs = re.sub(r"-(a|b|c)(\d+)$", r"\1\2", rs) - - # a terminal "dev" or "devel" can be changed into ".dev0" - rs = re.sub(r"[\.\-](dev|devel)$", r".dev0", rs) - - # a terminal "dev" can be changed into ".dev0" - rs = re.sub(r"(?![\.\-])dev$", r".dev0", rs) - - # a terminal "final" or "stable" can be removed - rs = re.sub(r"(final|stable)$", "", rs) - - # The 'r' and the '-' tags are post release tags - # 0.4a1.r10 -> 0.4a1.post10 - # 0.9.33-17222 -> 0.9.33.post17222 - # 0.9.33-r17222 -> 0.9.33.post17222 - rs = re.sub(r"\.?(r|-|-r)\.?(\d+)$", r".post\2", rs) - - # Clean 'r' instead of 'dev' usage: - # 0.9.33+r17222 -> 0.9.33.dev17222 - # 1.0dev123 -> 1.0.dev123 - # 1.0.git123 -> 1.0.dev123 - # 1.0.bzr123 -> 1.0.dev123 - # 0.1a0dev.123 -> 0.1a0.dev123 - # PyPI stats: ~150 (~4%) better - rs = re.sub(r"\.?(dev|git|bzr)\.?(\d+)$", r".dev\2", rs) - - # Clean '.pre' (normalized from '-pre' above) instead of 'c' usage: - # 0.2.pre1 -> 0.2c1 - # 0.2-c1 -> 0.2c1 - # 1.0preview123 -> 1.0c123 - # PyPI stats: ~21 (0.62%) better - rs = re.sub(r"\.?(pre|preview|-c)(\d+)$", r"c\g<2>", rs) - - # Tcl/Tk uses "px" for their post release markers - rs = re.sub(r"p(\d+)$", r".post\1", rs) - - try: - _normalized_key(rs) - except UnsupportedVersionError: - rs = None - return rs - -# -# Legacy version processing (distribute-compatible) -# - -_VERSION_PART = re.compile(r'([a-z]+|\d+|[\.-])', re.I) -_VERSION_REPLACE = { - 'pre': 'c', - 'preview': 'c', - '-': 'final-', - 'rc': 'c', - 'dev': '@', - '': None, - '.': None, -} - - -def _legacy_key(s): - def get_parts(s): - result = [] - for p in _VERSION_PART.split(s.lower()): - p = _VERSION_REPLACE.get(p, p) - if p: - if '0' <= p[:1] <= '9': - p = p.zfill(8) - else: - p = '*' + p - result.append(p) - result.append('*final') - return result - - result = [] - for p in get_parts(s): - if p.startswith('*'): - if p < '*final': - while result and result[-1] == '*final-': - result.pop() - while result and result[-1] == '00000000': - result.pop() - result.append(p) - return tuple(result) - - -class LegacyVersion(Version): - def parse(self, s): - return _legacy_key(s) - - @property - def is_prerelease(self): - result = False - for x in self._parts: - if (isinstance(x, string_types) and x.startswith('*') and - x < '*final'): - result = True - break - return result - - -class LegacyMatcher(Matcher): - version_class = LegacyVersion - - _operators = dict(Matcher._operators) - _operators['~='] = '_match_compatible' - - numeric_re = re.compile(r'^(\d+(\.\d+)*)') - - def _match_compatible(self, version, constraint, prefix): - if version < constraint: - return False - m = self.numeric_re.match(str(constraint)) - if not m: - logger.warning('Cannot compute compatible match for version %s ' - ' and constraint %s', version, constraint) - return True - s = m.groups()[0] - if '.' in s: - s = s.rsplit('.', 1)[0] - return _match_prefix(version, s) - -# -# Semantic versioning -# - -_SEMVER_RE = re.compile(r'^(\d+)\.(\d+)\.(\d+)' - r'(-[a-z0-9]+(\.[a-z0-9-]+)*)?' - r'(\+[a-z0-9]+(\.[a-z0-9-]+)*)?$', re.I) - - -def is_semver(s): - return _SEMVER_RE.match(s) - - -def _semantic_key(s): - def make_tuple(s, absent): - if s is None: - result = (absent,) - else: - parts = s[1:].split('.') - # We can't compare ints and strings on Python 3, so fudge it - # by zero-filling numeric values so simulate a numeric comparison - result = tuple([p.zfill(8) if p.isdigit() else p for p in parts]) - return result - - m = is_semver(s) - if not m: - raise UnsupportedVersionError(s) - groups = m.groups() - major, minor, patch = [int(i) for i in groups[:3]] - # choose the '|' and '*' so that versions sort correctly - pre, build = make_tuple(groups[3], '|'), make_tuple(groups[5], '*') - return (major, minor, patch), pre, build - - -class SemanticVersion(Version): - def parse(self, s): - return _semantic_key(s) - - @property - def is_prerelease(self): - return self._parts[1][0] != '|' - - -class SemanticMatcher(Matcher): - version_class = SemanticVersion - - -class VersionScheme(object): - def __init__(self, key, matcher, suggester=None): - self.key = key - self.matcher = matcher - self.suggester = suggester - - def is_valid_version(self, s): - try: - self.matcher.version_class(s) - result = True - except UnsupportedVersionError: - result = False - return result - - def is_valid_matcher(self, s): - try: - self.matcher(s) - result = True - except UnsupportedVersionError: - result = False - return result - - def is_valid_constraint_list(self, s): - """ - Used for processing some metadata fields - """ - # See issue #140. Be tolerant of a single trailing comma. - if s.endswith(','): - s = s[:-1] - return self.is_valid_matcher('dummy_name (%s)' % s) - - def suggest(self, s): - if self.suggester is None: - result = None - else: - result = self.suggester(s) - return result - -_SCHEMES = { - 'normalized': VersionScheme(_normalized_key, NormalizedMatcher, - _suggest_normalized_version), - 'legacy': VersionScheme(_legacy_key, LegacyMatcher, lambda self, s: s), - 'semantic': VersionScheme(_semantic_key, SemanticMatcher, - _suggest_semantic_version), -} - -_SCHEMES['default'] = _SCHEMES['normalized'] - - -def get_scheme(name): - if name not in _SCHEMES: - raise ValueError('unknown scheme name: %r' % name) - return _SCHEMES[name] diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/distlib/w32.exe b/venv/lib/python3.11/site-packages/pip/_vendor/distlib/w32.exe deleted file mode 100644 index 4ee2d3a..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/distlib/w32.exe and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/distlib/w64-arm.exe b/venv/lib/python3.11/site-packages/pip/_vendor/distlib/w64-arm.exe deleted file mode 100644 index 951d581..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/distlib/w64-arm.exe and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/distlib/w64.exe b/venv/lib/python3.11/site-packages/pip/_vendor/distlib/w64.exe deleted file mode 100644 index 5763076..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/distlib/w64.exe and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/distlib/wheel.py b/venv/lib/python3.11/site-packages/pip/_vendor/distlib/wheel.py deleted file mode 100644 index 028c2d9..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/distlib/wheel.py +++ /dev/null @@ -1,1082 +0,0 @@ -# -*- coding: utf-8 -*- -# -# Copyright (C) 2013-2020 Vinay Sajip. -# Licensed to the Python Software Foundation under a contributor agreement. -# See LICENSE.txt and CONTRIBUTORS.txt. -# -from __future__ import unicode_literals - -import base64 -import codecs -import datetime -from email import message_from_file -import hashlib -import json -import logging -import os -import posixpath -import re -import shutil -import sys -import tempfile -import zipfile - -from . import __version__, DistlibException -from .compat import sysconfig, ZipFile, fsdecode, text_type, filter -from .database import InstalledDistribution -from .metadata import (Metadata, METADATA_FILENAME, WHEEL_METADATA_FILENAME, - LEGACY_METADATA_FILENAME) -from .util import (FileOperator, convert_path, CSVReader, CSVWriter, Cache, - cached_property, get_cache_base, read_exports, tempdir, - get_platform) -from .version import NormalizedVersion, UnsupportedVersionError - -logger = logging.getLogger(__name__) - -cache = None # created when needed - -if hasattr(sys, 'pypy_version_info'): # pragma: no cover - IMP_PREFIX = 'pp' -elif sys.platform.startswith('java'): # pragma: no cover - IMP_PREFIX = 'jy' -elif sys.platform == 'cli': # pragma: no cover - IMP_PREFIX = 'ip' -else: - IMP_PREFIX = 'cp' - -VER_SUFFIX = sysconfig.get_config_var('py_version_nodot') -if not VER_SUFFIX: # pragma: no cover - VER_SUFFIX = '%s%s' % sys.version_info[:2] -PYVER = 'py' + VER_SUFFIX -IMPVER = IMP_PREFIX + VER_SUFFIX - -ARCH = get_platform().replace('-', '_').replace('.', '_') - -ABI = sysconfig.get_config_var('SOABI') -if ABI and ABI.startswith('cpython-'): - ABI = ABI.replace('cpython-', 'cp').split('-')[0] -else: - def _derive_abi(): - parts = ['cp', VER_SUFFIX] - if sysconfig.get_config_var('Py_DEBUG'): - parts.append('d') - if IMP_PREFIX == 'cp': - vi = sys.version_info[:2] - if vi < (3, 8): - wpm = sysconfig.get_config_var('WITH_PYMALLOC') - if wpm is None: - wpm = True - if wpm: - parts.append('m') - if vi < (3, 3): - us = sysconfig.get_config_var('Py_UNICODE_SIZE') - if us == 4 or (us is None and sys.maxunicode == 0x10FFFF): - parts.append('u') - return ''.join(parts) - ABI = _derive_abi() - del _derive_abi - -FILENAME_RE = re.compile(r''' -(?P[^-]+) --(?P\d+[^-]*) -(-(?P\d+[^-]*))? --(?P\w+\d+(\.\w+\d+)*) --(?P\w+) --(?P\w+(\.\w+)*) -\.whl$ -''', re.IGNORECASE | re.VERBOSE) - -NAME_VERSION_RE = re.compile(r''' -(?P[^-]+) --(?P\d+[^-]*) -(-(?P\d+[^-]*))?$ -''', re.IGNORECASE | re.VERBOSE) - -SHEBANG_RE = re.compile(br'\s*#![^\r\n]*') -SHEBANG_DETAIL_RE = re.compile(br'^(\s*#!("[^"]+"|\S+))\s+(.*)$') -SHEBANG_PYTHON = b'#!python' -SHEBANG_PYTHONW = b'#!pythonw' - -if os.sep == '/': - to_posix = lambda o: o -else: - to_posix = lambda o: o.replace(os.sep, '/') - -if sys.version_info[0] < 3: - import imp -else: - imp = None - import importlib.machinery - import importlib.util - -def _get_suffixes(): - if imp: - return [s[0] for s in imp.get_suffixes()] - else: - return importlib.machinery.EXTENSION_SUFFIXES - -def _load_dynamic(name, path): - # https://docs.python.org/3/library/importlib.html#importing-a-source-file-directly - if imp: - return imp.load_dynamic(name, path) - else: - spec = importlib.util.spec_from_file_location(name, path) - module = importlib.util.module_from_spec(spec) - sys.modules[name] = module - spec.loader.exec_module(module) - return module - -class Mounter(object): - def __init__(self): - self.impure_wheels = {} - self.libs = {} - - def add(self, pathname, extensions): - self.impure_wheels[pathname] = extensions - self.libs.update(extensions) - - def remove(self, pathname): - extensions = self.impure_wheels.pop(pathname) - for k, v in extensions: - if k in self.libs: - del self.libs[k] - - def find_module(self, fullname, path=None): - if fullname in self.libs: - result = self - else: - result = None - return result - - def load_module(self, fullname): - if fullname in sys.modules: - result = sys.modules[fullname] - else: - if fullname not in self.libs: - raise ImportError('unable to find extension for %s' % fullname) - result = _load_dynamic(fullname, self.libs[fullname]) - result.__loader__ = self - parts = fullname.rsplit('.', 1) - if len(parts) > 1: - result.__package__ = parts[0] - return result - -_hook = Mounter() - - -class Wheel(object): - """ - Class to build and install from Wheel files (PEP 427). - """ - - wheel_version = (1, 1) - hash_kind = 'sha256' - - def __init__(self, filename=None, sign=False, verify=False): - """ - Initialise an instance using a (valid) filename. - """ - self.sign = sign - self.should_verify = verify - self.buildver = '' - self.pyver = [PYVER] - self.abi = ['none'] - self.arch = ['any'] - self.dirname = os.getcwd() - if filename is None: - self.name = 'dummy' - self.version = '0.1' - self._filename = self.filename - else: - m = NAME_VERSION_RE.match(filename) - if m: - info = m.groupdict('') - self.name = info['nm'] - # Reinstate the local version separator - self.version = info['vn'].replace('_', '-') - self.buildver = info['bn'] - self._filename = self.filename - else: - dirname, filename = os.path.split(filename) - m = FILENAME_RE.match(filename) - if not m: - raise DistlibException('Invalid name or ' - 'filename: %r' % filename) - if dirname: - self.dirname = os.path.abspath(dirname) - self._filename = filename - info = m.groupdict('') - self.name = info['nm'] - self.version = info['vn'] - self.buildver = info['bn'] - self.pyver = info['py'].split('.') - self.abi = info['bi'].split('.') - self.arch = info['ar'].split('.') - - @property - def filename(self): - """ - Build and return a filename from the various components. - """ - if self.buildver: - buildver = '-' + self.buildver - else: - buildver = '' - pyver = '.'.join(self.pyver) - abi = '.'.join(self.abi) - arch = '.'.join(self.arch) - # replace - with _ as a local version separator - version = self.version.replace('-', '_') - return '%s-%s%s-%s-%s-%s.whl' % (self.name, version, buildver, - pyver, abi, arch) - - @property - def exists(self): - path = os.path.join(self.dirname, self.filename) - return os.path.isfile(path) - - @property - def tags(self): - for pyver in self.pyver: - for abi in self.abi: - for arch in self.arch: - yield pyver, abi, arch - - @cached_property - def metadata(self): - pathname = os.path.join(self.dirname, self.filename) - name_ver = '%s-%s' % (self.name, self.version) - info_dir = '%s.dist-info' % name_ver - wrapper = codecs.getreader('utf-8') - with ZipFile(pathname, 'r') as zf: - wheel_metadata = self.get_wheel_metadata(zf) - wv = wheel_metadata['Wheel-Version'].split('.', 1) - file_version = tuple([int(i) for i in wv]) - # if file_version < (1, 1): - # fns = [WHEEL_METADATA_FILENAME, METADATA_FILENAME, - # LEGACY_METADATA_FILENAME] - # else: - # fns = [WHEEL_METADATA_FILENAME, METADATA_FILENAME] - fns = [WHEEL_METADATA_FILENAME, LEGACY_METADATA_FILENAME] - result = None - for fn in fns: - try: - metadata_filename = posixpath.join(info_dir, fn) - with zf.open(metadata_filename) as bf: - wf = wrapper(bf) - result = Metadata(fileobj=wf) - if result: - break - except KeyError: - pass - if not result: - raise ValueError('Invalid wheel, because metadata is ' - 'missing: looked in %s' % ', '.join(fns)) - return result - - def get_wheel_metadata(self, zf): - name_ver = '%s-%s' % (self.name, self.version) - info_dir = '%s.dist-info' % name_ver - metadata_filename = posixpath.join(info_dir, 'WHEEL') - with zf.open(metadata_filename) as bf: - wf = codecs.getreader('utf-8')(bf) - message = message_from_file(wf) - return dict(message) - - @cached_property - def info(self): - pathname = os.path.join(self.dirname, self.filename) - with ZipFile(pathname, 'r') as zf: - result = self.get_wheel_metadata(zf) - return result - - def process_shebang(self, data): - m = SHEBANG_RE.match(data) - if m: - end = m.end() - shebang, data_after_shebang = data[:end], data[end:] - # Preserve any arguments after the interpreter - if b'pythonw' in shebang.lower(): - shebang_python = SHEBANG_PYTHONW - else: - shebang_python = SHEBANG_PYTHON - m = SHEBANG_DETAIL_RE.match(shebang) - if m: - args = b' ' + m.groups()[-1] - else: - args = b'' - shebang = shebang_python + args - data = shebang + data_after_shebang - else: - cr = data.find(b'\r') - lf = data.find(b'\n') - if cr < 0 or cr > lf: - term = b'\n' - else: - if data[cr:cr + 2] == b'\r\n': - term = b'\r\n' - else: - term = b'\r' - data = SHEBANG_PYTHON + term + data - return data - - def get_hash(self, data, hash_kind=None): - if hash_kind is None: - hash_kind = self.hash_kind - try: - hasher = getattr(hashlib, hash_kind) - except AttributeError: - raise DistlibException('Unsupported hash algorithm: %r' % hash_kind) - result = hasher(data).digest() - result = base64.urlsafe_b64encode(result).rstrip(b'=').decode('ascii') - return hash_kind, result - - def write_record(self, records, record_path, archive_record_path): - records = list(records) # make a copy, as mutated - records.append((archive_record_path, '', '')) - with CSVWriter(record_path) as writer: - for row in records: - writer.writerow(row) - - def write_records(self, info, libdir, archive_paths): - records = [] - distinfo, info_dir = info - hasher = getattr(hashlib, self.hash_kind) - for ap, p in archive_paths: - with open(p, 'rb') as f: - data = f.read() - digest = '%s=%s' % self.get_hash(data) - size = os.path.getsize(p) - records.append((ap, digest, size)) - - p = os.path.join(distinfo, 'RECORD') - ap = to_posix(os.path.join(info_dir, 'RECORD')) - self.write_record(records, p, ap) - archive_paths.append((ap, p)) - - def build_zip(self, pathname, archive_paths): - with ZipFile(pathname, 'w', zipfile.ZIP_DEFLATED) as zf: - for ap, p in archive_paths: - logger.debug('Wrote %s to %s in wheel', p, ap) - zf.write(p, ap) - - def build(self, paths, tags=None, wheel_version=None): - """ - Build a wheel from files in specified paths, and use any specified tags - when determining the name of the wheel. - """ - if tags is None: - tags = {} - - libkey = list(filter(lambda o: o in paths, ('purelib', 'platlib')))[0] - if libkey == 'platlib': - is_pure = 'false' - default_pyver = [IMPVER] - default_abi = [ABI] - default_arch = [ARCH] - else: - is_pure = 'true' - default_pyver = [PYVER] - default_abi = ['none'] - default_arch = ['any'] - - self.pyver = tags.get('pyver', default_pyver) - self.abi = tags.get('abi', default_abi) - self.arch = tags.get('arch', default_arch) - - libdir = paths[libkey] - - name_ver = '%s-%s' % (self.name, self.version) - data_dir = '%s.data' % name_ver - info_dir = '%s.dist-info' % name_ver - - archive_paths = [] - - # First, stuff which is not in site-packages - for key in ('data', 'headers', 'scripts'): - if key not in paths: - continue - path = paths[key] - if os.path.isdir(path): - for root, dirs, files in os.walk(path): - for fn in files: - p = fsdecode(os.path.join(root, fn)) - rp = os.path.relpath(p, path) - ap = to_posix(os.path.join(data_dir, key, rp)) - archive_paths.append((ap, p)) - if key == 'scripts' and not p.endswith('.exe'): - with open(p, 'rb') as f: - data = f.read() - data = self.process_shebang(data) - with open(p, 'wb') as f: - f.write(data) - - # Now, stuff which is in site-packages, other than the - # distinfo stuff. - path = libdir - distinfo = None - for root, dirs, files in os.walk(path): - if root == path: - # At the top level only, save distinfo for later - # and skip it for now - for i, dn in enumerate(dirs): - dn = fsdecode(dn) - if dn.endswith('.dist-info'): - distinfo = os.path.join(root, dn) - del dirs[i] - break - assert distinfo, '.dist-info directory expected, not found' - - for fn in files: - # comment out next suite to leave .pyc files in - if fsdecode(fn).endswith(('.pyc', '.pyo')): - continue - p = os.path.join(root, fn) - rp = to_posix(os.path.relpath(p, path)) - archive_paths.append((rp, p)) - - # Now distinfo. Assumed to be flat, i.e. os.listdir is enough. - files = os.listdir(distinfo) - for fn in files: - if fn not in ('RECORD', 'INSTALLER', 'SHARED', 'WHEEL'): - p = fsdecode(os.path.join(distinfo, fn)) - ap = to_posix(os.path.join(info_dir, fn)) - archive_paths.append((ap, p)) - - wheel_metadata = [ - 'Wheel-Version: %d.%d' % (wheel_version or self.wheel_version), - 'Generator: distlib %s' % __version__, - 'Root-Is-Purelib: %s' % is_pure, - ] - for pyver, abi, arch in self.tags: - wheel_metadata.append('Tag: %s-%s-%s' % (pyver, abi, arch)) - p = os.path.join(distinfo, 'WHEEL') - with open(p, 'w') as f: - f.write('\n'.join(wheel_metadata)) - ap = to_posix(os.path.join(info_dir, 'WHEEL')) - archive_paths.append((ap, p)) - - # sort the entries by archive path. Not needed by any spec, but it - # keeps the archive listing and RECORD tidier than they would otherwise - # be. Use the number of path segments to keep directory entries together, - # and keep the dist-info stuff at the end. - def sorter(t): - ap = t[0] - n = ap.count('/') - if '.dist-info' in ap: - n += 10000 - return (n, ap) - archive_paths = sorted(archive_paths, key=sorter) - - # Now, at last, RECORD. - # Paths in here are archive paths - nothing else makes sense. - self.write_records((distinfo, info_dir), libdir, archive_paths) - # Now, ready to build the zip file - pathname = os.path.join(self.dirname, self.filename) - self.build_zip(pathname, archive_paths) - return pathname - - def skip_entry(self, arcname): - """ - Determine whether an archive entry should be skipped when verifying - or installing. - """ - # The signature file won't be in RECORD, - # and we don't currently don't do anything with it - # We also skip directories, as they won't be in RECORD - # either. See: - # - # https://github.com/pypa/wheel/issues/294 - # https://github.com/pypa/wheel/issues/287 - # https://github.com/pypa/wheel/pull/289 - # - return arcname.endswith(('/', '/RECORD.jws')) - - def install(self, paths, maker, **kwargs): - """ - Install a wheel to the specified paths. If kwarg ``warner`` is - specified, it should be a callable, which will be called with two - tuples indicating the wheel version of this software and the wheel - version in the file, if there is a discrepancy in the versions. - This can be used to issue any warnings to raise any exceptions. - If kwarg ``lib_only`` is True, only the purelib/platlib files are - installed, and the headers, scripts, data and dist-info metadata are - not written. If kwarg ``bytecode_hashed_invalidation`` is True, written - bytecode will try to use file-hash based invalidation (PEP-552) on - supported interpreter versions (CPython 2.7+). - - The return value is a :class:`InstalledDistribution` instance unless - ``options.lib_only`` is True, in which case the return value is ``None``. - """ - - dry_run = maker.dry_run - warner = kwargs.get('warner') - lib_only = kwargs.get('lib_only', False) - bc_hashed_invalidation = kwargs.get('bytecode_hashed_invalidation', False) - - pathname = os.path.join(self.dirname, self.filename) - name_ver = '%s-%s' % (self.name, self.version) - data_dir = '%s.data' % name_ver - info_dir = '%s.dist-info' % name_ver - - metadata_name = posixpath.join(info_dir, LEGACY_METADATA_FILENAME) - wheel_metadata_name = posixpath.join(info_dir, 'WHEEL') - record_name = posixpath.join(info_dir, 'RECORD') - - wrapper = codecs.getreader('utf-8') - - with ZipFile(pathname, 'r') as zf: - with zf.open(wheel_metadata_name) as bwf: - wf = wrapper(bwf) - message = message_from_file(wf) - wv = message['Wheel-Version'].split('.', 1) - file_version = tuple([int(i) for i in wv]) - if (file_version != self.wheel_version) and warner: - warner(self.wheel_version, file_version) - - if message['Root-Is-Purelib'] == 'true': - libdir = paths['purelib'] - else: - libdir = paths['platlib'] - - records = {} - with zf.open(record_name) as bf: - with CSVReader(stream=bf) as reader: - for row in reader: - p = row[0] - records[p] = row - - data_pfx = posixpath.join(data_dir, '') - info_pfx = posixpath.join(info_dir, '') - script_pfx = posixpath.join(data_dir, 'scripts', '') - - # make a new instance rather than a copy of maker's, - # as we mutate it - fileop = FileOperator(dry_run=dry_run) - fileop.record = True # so we can rollback if needed - - bc = not sys.dont_write_bytecode # Double negatives. Lovely! - - outfiles = [] # for RECORD writing - - # for script copying/shebang processing - workdir = tempfile.mkdtemp() - # set target dir later - # we default add_launchers to False, as the - # Python Launcher should be used instead - maker.source_dir = workdir - maker.target_dir = None - try: - for zinfo in zf.infolist(): - arcname = zinfo.filename - if isinstance(arcname, text_type): - u_arcname = arcname - else: - u_arcname = arcname.decode('utf-8') - if self.skip_entry(u_arcname): - continue - row = records[u_arcname] - if row[2] and str(zinfo.file_size) != row[2]: - raise DistlibException('size mismatch for ' - '%s' % u_arcname) - if row[1]: - kind, value = row[1].split('=', 1) - with zf.open(arcname) as bf: - data = bf.read() - _, digest = self.get_hash(data, kind) - if digest != value: - raise DistlibException('digest mismatch for ' - '%s' % arcname) - - if lib_only and u_arcname.startswith((info_pfx, data_pfx)): - logger.debug('lib_only: skipping %s', u_arcname) - continue - is_script = (u_arcname.startswith(script_pfx) - and not u_arcname.endswith('.exe')) - - if u_arcname.startswith(data_pfx): - _, where, rp = u_arcname.split('/', 2) - outfile = os.path.join(paths[where], convert_path(rp)) - else: - # meant for site-packages. - if u_arcname in (wheel_metadata_name, record_name): - continue - outfile = os.path.join(libdir, convert_path(u_arcname)) - if not is_script: - with zf.open(arcname) as bf: - fileop.copy_stream(bf, outfile) - # Issue #147: permission bits aren't preserved. Using - # zf.extract(zinfo, libdir) should have worked, but didn't, - # see https://www.thetopsites.net/article/53834422.shtml - # So ... manually preserve permission bits as given in zinfo - if os.name == 'posix': - # just set the normal permission bits - os.chmod(outfile, (zinfo.external_attr >> 16) & 0x1FF) - outfiles.append(outfile) - # Double check the digest of the written file - if not dry_run and row[1]: - with open(outfile, 'rb') as bf: - data = bf.read() - _, newdigest = self.get_hash(data, kind) - if newdigest != digest: - raise DistlibException('digest mismatch ' - 'on write for ' - '%s' % outfile) - if bc and outfile.endswith('.py'): - try: - pyc = fileop.byte_compile(outfile, - hashed_invalidation=bc_hashed_invalidation) - outfiles.append(pyc) - except Exception: - # Don't give up if byte-compilation fails, - # but log it and perhaps warn the user - logger.warning('Byte-compilation failed', - exc_info=True) - else: - fn = os.path.basename(convert_path(arcname)) - workname = os.path.join(workdir, fn) - with zf.open(arcname) as bf: - fileop.copy_stream(bf, workname) - - dn, fn = os.path.split(outfile) - maker.target_dir = dn - filenames = maker.make(fn) - fileop.set_executable_mode(filenames) - outfiles.extend(filenames) - - if lib_only: - logger.debug('lib_only: returning None') - dist = None - else: - # Generate scripts - - # Try to get pydist.json so we can see if there are - # any commands to generate. If this fails (e.g. because - # of a legacy wheel), log a warning but don't give up. - commands = None - file_version = self.info['Wheel-Version'] - if file_version == '1.0': - # Use legacy info - ep = posixpath.join(info_dir, 'entry_points.txt') - try: - with zf.open(ep) as bwf: - epdata = read_exports(bwf) - commands = {} - for key in ('console', 'gui'): - k = '%s_scripts' % key - if k in epdata: - commands['wrap_%s' % key] = d = {} - for v in epdata[k].values(): - s = '%s:%s' % (v.prefix, v.suffix) - if v.flags: - s += ' [%s]' % ','.join(v.flags) - d[v.name] = s - except Exception: - logger.warning('Unable to read legacy script ' - 'metadata, so cannot generate ' - 'scripts') - else: - try: - with zf.open(metadata_name) as bwf: - wf = wrapper(bwf) - commands = json.load(wf).get('extensions') - if commands: - commands = commands.get('python.commands') - except Exception: - logger.warning('Unable to read JSON metadata, so ' - 'cannot generate scripts') - if commands: - console_scripts = commands.get('wrap_console', {}) - gui_scripts = commands.get('wrap_gui', {}) - if console_scripts or gui_scripts: - script_dir = paths.get('scripts', '') - if not os.path.isdir(script_dir): - raise ValueError('Valid script path not ' - 'specified') - maker.target_dir = script_dir - for k, v in console_scripts.items(): - script = '%s = %s' % (k, v) - filenames = maker.make(script) - fileop.set_executable_mode(filenames) - - if gui_scripts: - options = {'gui': True } - for k, v in gui_scripts.items(): - script = '%s = %s' % (k, v) - filenames = maker.make(script, options) - fileop.set_executable_mode(filenames) - - p = os.path.join(libdir, info_dir) - dist = InstalledDistribution(p) - - # Write SHARED - paths = dict(paths) # don't change passed in dict - del paths['purelib'] - del paths['platlib'] - paths['lib'] = libdir - p = dist.write_shared_locations(paths, dry_run) - if p: - outfiles.append(p) - - # Write RECORD - dist.write_installed_files(outfiles, paths['prefix'], - dry_run) - return dist - except Exception: # pragma: no cover - logger.exception('installation failed.') - fileop.rollback() - raise - finally: - shutil.rmtree(workdir) - - def _get_dylib_cache(self): - global cache - if cache is None: - # Use native string to avoid issues on 2.x: see Python #20140. - base = os.path.join(get_cache_base(), str('dylib-cache'), - '%s.%s' % sys.version_info[:2]) - cache = Cache(base) - return cache - - def _get_extensions(self): - pathname = os.path.join(self.dirname, self.filename) - name_ver = '%s-%s' % (self.name, self.version) - info_dir = '%s.dist-info' % name_ver - arcname = posixpath.join(info_dir, 'EXTENSIONS') - wrapper = codecs.getreader('utf-8') - result = [] - with ZipFile(pathname, 'r') as zf: - try: - with zf.open(arcname) as bf: - wf = wrapper(bf) - extensions = json.load(wf) - cache = self._get_dylib_cache() - prefix = cache.prefix_to_dir(pathname) - cache_base = os.path.join(cache.base, prefix) - if not os.path.isdir(cache_base): - os.makedirs(cache_base) - for name, relpath in extensions.items(): - dest = os.path.join(cache_base, convert_path(relpath)) - if not os.path.exists(dest): - extract = True - else: - file_time = os.stat(dest).st_mtime - file_time = datetime.datetime.fromtimestamp(file_time) - info = zf.getinfo(relpath) - wheel_time = datetime.datetime(*info.date_time) - extract = wheel_time > file_time - if extract: - zf.extract(relpath, cache_base) - result.append((name, dest)) - except KeyError: - pass - return result - - def is_compatible(self): - """ - Determine if a wheel is compatible with the running system. - """ - return is_compatible(self) - - def is_mountable(self): - """ - Determine if a wheel is asserted as mountable by its metadata. - """ - return True # for now - metadata details TBD - - def mount(self, append=False): - pathname = os.path.abspath(os.path.join(self.dirname, self.filename)) - if not self.is_compatible(): - msg = 'Wheel %s not compatible with this Python.' % pathname - raise DistlibException(msg) - if not self.is_mountable(): - msg = 'Wheel %s is marked as not mountable.' % pathname - raise DistlibException(msg) - if pathname in sys.path: - logger.debug('%s already in path', pathname) - else: - if append: - sys.path.append(pathname) - else: - sys.path.insert(0, pathname) - extensions = self._get_extensions() - if extensions: - if _hook not in sys.meta_path: - sys.meta_path.append(_hook) - _hook.add(pathname, extensions) - - def unmount(self): - pathname = os.path.abspath(os.path.join(self.dirname, self.filename)) - if pathname not in sys.path: - logger.debug('%s not in path', pathname) - else: - sys.path.remove(pathname) - if pathname in _hook.impure_wheels: - _hook.remove(pathname) - if not _hook.impure_wheels: - if _hook in sys.meta_path: - sys.meta_path.remove(_hook) - - def verify(self): - pathname = os.path.join(self.dirname, self.filename) - name_ver = '%s-%s' % (self.name, self.version) - data_dir = '%s.data' % name_ver - info_dir = '%s.dist-info' % name_ver - - metadata_name = posixpath.join(info_dir, LEGACY_METADATA_FILENAME) - wheel_metadata_name = posixpath.join(info_dir, 'WHEEL') - record_name = posixpath.join(info_dir, 'RECORD') - - wrapper = codecs.getreader('utf-8') - - with ZipFile(pathname, 'r') as zf: - with zf.open(wheel_metadata_name) as bwf: - wf = wrapper(bwf) - message = message_from_file(wf) - wv = message['Wheel-Version'].split('.', 1) - file_version = tuple([int(i) for i in wv]) - # TODO version verification - - records = {} - with zf.open(record_name) as bf: - with CSVReader(stream=bf) as reader: - for row in reader: - p = row[0] - records[p] = row - - for zinfo in zf.infolist(): - arcname = zinfo.filename - if isinstance(arcname, text_type): - u_arcname = arcname - else: - u_arcname = arcname.decode('utf-8') - # See issue #115: some wheels have .. in their entries, but - # in the filename ... e.g. __main__..py ! So the check is - # updated to look for .. in the directory portions - p = u_arcname.split('/') - if '..' in p: - raise DistlibException('invalid entry in ' - 'wheel: %r' % u_arcname) - - if self.skip_entry(u_arcname): - continue - row = records[u_arcname] - if row[2] and str(zinfo.file_size) != row[2]: - raise DistlibException('size mismatch for ' - '%s' % u_arcname) - if row[1]: - kind, value = row[1].split('=', 1) - with zf.open(arcname) as bf: - data = bf.read() - _, digest = self.get_hash(data, kind) - if digest != value: - raise DistlibException('digest mismatch for ' - '%s' % arcname) - - def update(self, modifier, dest_dir=None, **kwargs): - """ - Update the contents of a wheel in a generic way. The modifier should - be a callable which expects a dictionary argument: its keys are - archive-entry paths, and its values are absolute filesystem paths - where the contents the corresponding archive entries can be found. The - modifier is free to change the contents of the files pointed to, add - new entries and remove entries, before returning. This method will - extract the entire contents of the wheel to a temporary location, call - the modifier, and then use the passed (and possibly updated) - dictionary to write a new wheel. If ``dest_dir`` is specified, the new - wheel is written there -- otherwise, the original wheel is overwritten. - - The modifier should return True if it updated the wheel, else False. - This method returns the same value the modifier returns. - """ - - def get_version(path_map, info_dir): - version = path = None - key = '%s/%s' % (info_dir, LEGACY_METADATA_FILENAME) - if key not in path_map: - key = '%s/PKG-INFO' % info_dir - if key in path_map: - path = path_map[key] - version = Metadata(path=path).version - return version, path - - def update_version(version, path): - updated = None - try: - v = NormalizedVersion(version) - i = version.find('-') - if i < 0: - updated = '%s+1' % version - else: - parts = [int(s) for s in version[i + 1:].split('.')] - parts[-1] += 1 - updated = '%s+%s' % (version[:i], - '.'.join(str(i) for i in parts)) - except UnsupportedVersionError: - logger.debug('Cannot update non-compliant (PEP-440) ' - 'version %r', version) - if updated: - md = Metadata(path=path) - md.version = updated - legacy = path.endswith(LEGACY_METADATA_FILENAME) - md.write(path=path, legacy=legacy) - logger.debug('Version updated from %r to %r', version, - updated) - - pathname = os.path.join(self.dirname, self.filename) - name_ver = '%s-%s' % (self.name, self.version) - info_dir = '%s.dist-info' % name_ver - record_name = posixpath.join(info_dir, 'RECORD') - with tempdir() as workdir: - with ZipFile(pathname, 'r') as zf: - path_map = {} - for zinfo in zf.infolist(): - arcname = zinfo.filename - if isinstance(arcname, text_type): - u_arcname = arcname - else: - u_arcname = arcname.decode('utf-8') - if u_arcname == record_name: - continue - if '..' in u_arcname: - raise DistlibException('invalid entry in ' - 'wheel: %r' % u_arcname) - zf.extract(zinfo, workdir) - path = os.path.join(workdir, convert_path(u_arcname)) - path_map[u_arcname] = path - - # Remember the version. - original_version, _ = get_version(path_map, info_dir) - # Files extracted. Call the modifier. - modified = modifier(path_map, **kwargs) - if modified: - # Something changed - need to build a new wheel. - current_version, path = get_version(path_map, info_dir) - if current_version and (current_version == original_version): - # Add or update local version to signify changes. - update_version(current_version, path) - # Decide where the new wheel goes. - if dest_dir is None: - fd, newpath = tempfile.mkstemp(suffix='.whl', - prefix='wheel-update-', - dir=workdir) - os.close(fd) - else: - if not os.path.isdir(dest_dir): - raise DistlibException('Not a directory: %r' % dest_dir) - newpath = os.path.join(dest_dir, self.filename) - archive_paths = list(path_map.items()) - distinfo = os.path.join(workdir, info_dir) - info = distinfo, info_dir - self.write_records(info, workdir, archive_paths) - self.build_zip(newpath, archive_paths) - if dest_dir is None: - shutil.copyfile(newpath, pathname) - return modified - -def _get_glibc_version(): - import platform - ver = platform.libc_ver() - result = [] - if ver[0] == 'glibc': - for s in ver[1].split('.'): - result.append(int(s) if s.isdigit() else 0) - result = tuple(result) - return result - -def compatible_tags(): - """ - Return (pyver, abi, arch) tuples compatible with this Python. - """ - versions = [VER_SUFFIX] - major = VER_SUFFIX[0] - for minor in range(sys.version_info[1] - 1, - 1, -1): - versions.append(''.join([major, str(minor)])) - - abis = [] - for suffix in _get_suffixes(): - if suffix.startswith('.abi'): - abis.append(suffix.split('.', 2)[1]) - abis.sort() - if ABI != 'none': - abis.insert(0, ABI) - abis.append('none') - result = [] - - arches = [ARCH] - if sys.platform == 'darwin': - m = re.match(r'(\w+)_(\d+)_(\d+)_(\w+)$', ARCH) - if m: - name, major, minor, arch = m.groups() - minor = int(minor) - matches = [arch] - if arch in ('i386', 'ppc'): - matches.append('fat') - if arch in ('i386', 'ppc', 'x86_64'): - matches.append('fat3') - if arch in ('ppc64', 'x86_64'): - matches.append('fat64') - if arch in ('i386', 'x86_64'): - matches.append('intel') - if arch in ('i386', 'x86_64', 'intel', 'ppc', 'ppc64'): - matches.append('universal') - while minor >= 0: - for match in matches: - s = '%s_%s_%s_%s' % (name, major, minor, match) - if s != ARCH: # already there - arches.append(s) - minor -= 1 - - # Most specific - our Python version, ABI and arch - for abi in abis: - for arch in arches: - result.append((''.join((IMP_PREFIX, versions[0])), abi, arch)) - # manylinux - if abi != 'none' and sys.platform.startswith('linux'): - arch = arch.replace('linux_', '') - parts = _get_glibc_version() - if len(parts) == 2: - if parts >= (2, 5): - result.append((''.join((IMP_PREFIX, versions[0])), abi, - 'manylinux1_%s' % arch)) - if parts >= (2, 12): - result.append((''.join((IMP_PREFIX, versions[0])), abi, - 'manylinux2010_%s' % arch)) - if parts >= (2, 17): - result.append((''.join((IMP_PREFIX, versions[0])), abi, - 'manylinux2014_%s' % arch)) - result.append((''.join((IMP_PREFIX, versions[0])), abi, - 'manylinux_%s_%s_%s' % (parts[0], parts[1], - arch))) - - # where no ABI / arch dependency, but IMP_PREFIX dependency - for i, version in enumerate(versions): - result.append((''.join((IMP_PREFIX, version)), 'none', 'any')) - if i == 0: - result.append((''.join((IMP_PREFIX, version[0])), 'none', 'any')) - - # no IMP_PREFIX, ABI or arch dependency - for i, version in enumerate(versions): - result.append((''.join(('py', version)), 'none', 'any')) - if i == 0: - result.append((''.join(('py', version[0])), 'none', 'any')) - - return set(result) - - -COMPATIBLE_TAGS = compatible_tags() - -del compatible_tags - - -def is_compatible(wheel, tags=None): - if not isinstance(wheel, Wheel): - wheel = Wheel(wheel) # assume it's a filename - result = False - if tags is None: - tags = COMPATIBLE_TAGS - for ver, abi, arch in tags: - if ver in wheel.pyver and abi in wheel.abi and arch in wheel.arch: - result = True - break - return result diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/distro/__init__.py b/venv/lib/python3.11/site-packages/pip/_vendor/distro/__init__.py deleted file mode 100644 index 7686fe8..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/distro/__init__.py +++ /dev/null @@ -1,54 +0,0 @@ -from .distro import ( - NORMALIZED_DISTRO_ID, - NORMALIZED_LSB_ID, - NORMALIZED_OS_ID, - LinuxDistribution, - __version__, - build_number, - codename, - distro_release_attr, - distro_release_info, - id, - info, - like, - linux_distribution, - lsb_release_attr, - lsb_release_info, - major_version, - minor_version, - name, - os_release_attr, - os_release_info, - uname_attr, - uname_info, - version, - version_parts, -) - -__all__ = [ - "NORMALIZED_DISTRO_ID", - "NORMALIZED_LSB_ID", - "NORMALIZED_OS_ID", - "LinuxDistribution", - "build_number", - "codename", - "distro_release_attr", - "distro_release_info", - "id", - "info", - "like", - "linux_distribution", - "lsb_release_attr", - "lsb_release_info", - "major_version", - "minor_version", - "name", - "os_release_attr", - "os_release_info", - "uname_attr", - "uname_info", - "version", - "version_parts", -] - -__version__ = __version__ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/distro/__main__.py b/venv/lib/python3.11/site-packages/pip/_vendor/distro/__main__.py deleted file mode 100644 index 0c01d5b..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/distro/__main__.py +++ /dev/null @@ -1,4 +0,0 @@ -from .distro import main - -if __name__ == "__main__": - main() diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/distro/__pycache__/__init__.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/distro/__pycache__/__init__.cpython-311.pyc deleted file mode 100644 index 0ca9616..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/distro/__pycache__/__init__.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/distro/__pycache__/__main__.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/distro/__pycache__/__main__.cpython-311.pyc deleted file mode 100644 index 7687d76..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/distro/__pycache__/__main__.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/distro/__pycache__/distro.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/distro/__pycache__/distro.cpython-311.pyc deleted file mode 100644 index d2c2d10..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/distro/__pycache__/distro.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/distro/distro.py b/venv/lib/python3.11/site-packages/pip/_vendor/distro/distro.py deleted file mode 100644 index 89e1868..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/distro/distro.py +++ /dev/null @@ -1,1399 +0,0 @@ -#!/usr/bin/env python -# Copyright 2015,2016,2017 Nir Cohen -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -""" -The ``distro`` package (``distro`` stands for Linux Distribution) provides -information about the Linux distribution it runs on, such as a reliable -machine-readable distro ID, or version information. - -It is the recommended replacement for Python's original -:py:func:`platform.linux_distribution` function, but it provides much more -functionality. An alternative implementation became necessary because Python -3.5 deprecated this function, and Python 3.8 removed it altogether. Its -predecessor function :py:func:`platform.dist` was already deprecated since -Python 2.6 and removed in Python 3.8. Still, there are many cases in which -access to OS distribution information is needed. See `Python issue 1322 -`_ for more information. -""" - -import argparse -import json -import logging -import os -import re -import shlex -import subprocess -import sys -import warnings -from typing import ( - Any, - Callable, - Dict, - Iterable, - Optional, - Sequence, - TextIO, - Tuple, - Type, -) - -try: - from typing import TypedDict -except ImportError: - # Python 3.7 - TypedDict = dict - -__version__ = "1.8.0" - - -class VersionDict(TypedDict): - major: str - minor: str - build_number: str - - -class InfoDict(TypedDict): - id: str - version: str - version_parts: VersionDict - like: str - codename: str - - -_UNIXCONFDIR = os.environ.get("UNIXCONFDIR", "/etc") -_UNIXUSRLIBDIR = os.environ.get("UNIXUSRLIBDIR", "/usr/lib") -_OS_RELEASE_BASENAME = "os-release" - -#: Translation table for normalizing the "ID" attribute defined in os-release -#: files, for use by the :func:`distro.id` method. -#: -#: * Key: Value as defined in the os-release file, translated to lower case, -#: with blanks translated to underscores. -#: -#: * Value: Normalized value. -NORMALIZED_OS_ID = { - "ol": "oracle", # Oracle Linux - "opensuse-leap": "opensuse", # Newer versions of OpenSuSE report as opensuse-leap -} - -#: Translation table for normalizing the "Distributor ID" attribute returned by -#: the lsb_release command, for use by the :func:`distro.id` method. -#: -#: * Key: Value as returned by the lsb_release command, translated to lower -#: case, with blanks translated to underscores. -#: -#: * Value: Normalized value. -NORMALIZED_LSB_ID = { - "enterpriseenterpriseas": "oracle", # Oracle Enterprise Linux 4 - "enterpriseenterpriseserver": "oracle", # Oracle Linux 5 - "redhatenterpriseworkstation": "rhel", # RHEL 6, 7 Workstation - "redhatenterpriseserver": "rhel", # RHEL 6, 7 Server - "redhatenterprisecomputenode": "rhel", # RHEL 6 ComputeNode -} - -#: Translation table for normalizing the distro ID derived from the file name -#: of distro release files, for use by the :func:`distro.id` method. -#: -#: * Key: Value as derived from the file name of a distro release file, -#: translated to lower case, with blanks translated to underscores. -#: -#: * Value: Normalized value. -NORMALIZED_DISTRO_ID = { - "redhat": "rhel", # RHEL 6.x, 7.x -} - -# Pattern for content of distro release file (reversed) -_DISTRO_RELEASE_CONTENT_REVERSED_PATTERN = re.compile( - r"(?:[^)]*\)(.*)\()? *(?:STL )?([\d.+\-a-z]*\d) *(?:esaeler *)?(.+)" -) - -# Pattern for base file name of distro release file -_DISTRO_RELEASE_BASENAME_PATTERN = re.compile(r"(\w+)[-_](release|version)$") - -# Base file names to be looked up for if _UNIXCONFDIR is not readable. -_DISTRO_RELEASE_BASENAMES = [ - "SuSE-release", - "arch-release", - "base-release", - "centos-release", - "fedora-release", - "gentoo-release", - "mageia-release", - "mandrake-release", - "mandriva-release", - "mandrivalinux-release", - "manjaro-release", - "oracle-release", - "redhat-release", - "rocky-release", - "sl-release", - "slackware-version", -] - -# Base file names to be ignored when searching for distro release file -_DISTRO_RELEASE_IGNORE_BASENAMES = ( - "debian_version", - "lsb-release", - "oem-release", - _OS_RELEASE_BASENAME, - "system-release", - "plesk-release", - "iredmail-release", -) - - -def linux_distribution(full_distribution_name: bool = True) -> Tuple[str, str, str]: - """ - .. deprecated:: 1.6.0 - - :func:`distro.linux_distribution()` is deprecated. It should only be - used as a compatibility shim with Python's - :py:func:`platform.linux_distribution()`. Please use :func:`distro.id`, - :func:`distro.version` and :func:`distro.name` instead. - - Return information about the current OS distribution as a tuple - ``(id_name, version, codename)`` with items as follows: - - * ``id_name``: If *full_distribution_name* is false, the result of - :func:`distro.id`. Otherwise, the result of :func:`distro.name`. - - * ``version``: The result of :func:`distro.version`. - - * ``codename``: The extra item (usually in parentheses) after the - os-release version number, or the result of :func:`distro.codename`. - - The interface of this function is compatible with the original - :py:func:`platform.linux_distribution` function, supporting a subset of - its parameters. - - The data it returns may not exactly be the same, because it uses more data - sources than the original function, and that may lead to different data if - the OS distribution is not consistent across multiple data sources it - provides (there are indeed such distributions ...). - - Another reason for differences is the fact that the :func:`distro.id` - method normalizes the distro ID string to a reliable machine-readable value - for a number of popular OS distributions. - """ - warnings.warn( - "distro.linux_distribution() is deprecated. It should only be used as a " - "compatibility shim with Python's platform.linux_distribution(). Please use " - "distro.id(), distro.version() and distro.name() instead.", - DeprecationWarning, - stacklevel=2, - ) - return _distro.linux_distribution(full_distribution_name) - - -def id() -> str: - """ - Return the distro ID of the current distribution, as a - machine-readable string. - - For a number of OS distributions, the returned distro ID value is - *reliable*, in the sense that it is documented and that it does not change - across releases of the distribution. - - This package maintains the following reliable distro ID values: - - ============== ========================================= - Distro ID Distribution - ============== ========================================= - "ubuntu" Ubuntu - "debian" Debian - "rhel" RedHat Enterprise Linux - "centos" CentOS - "fedora" Fedora - "sles" SUSE Linux Enterprise Server - "opensuse" openSUSE - "amzn" Amazon Linux - "arch" Arch Linux - "buildroot" Buildroot - "cloudlinux" CloudLinux OS - "exherbo" Exherbo Linux - "gentoo" GenToo Linux - "ibm_powerkvm" IBM PowerKVM - "kvmibm" KVM for IBM z Systems - "linuxmint" Linux Mint - "mageia" Mageia - "mandriva" Mandriva Linux - "parallels" Parallels - "pidora" Pidora - "raspbian" Raspbian - "oracle" Oracle Linux (and Oracle Enterprise Linux) - "scientific" Scientific Linux - "slackware" Slackware - "xenserver" XenServer - "openbsd" OpenBSD - "netbsd" NetBSD - "freebsd" FreeBSD - "midnightbsd" MidnightBSD - "rocky" Rocky Linux - "aix" AIX - "guix" Guix System - ============== ========================================= - - If you have a need to get distros for reliable IDs added into this set, - or if you find that the :func:`distro.id` function returns a different - distro ID for one of the listed distros, please create an issue in the - `distro issue tracker`_. - - **Lookup hierarchy and transformations:** - - First, the ID is obtained from the following sources, in the specified - order. The first available and non-empty value is used: - - * the value of the "ID" attribute of the os-release file, - - * the value of the "Distributor ID" attribute returned by the lsb_release - command, - - * the first part of the file name of the distro release file, - - The so determined ID value then passes the following transformations, - before it is returned by this method: - - * it is translated to lower case, - - * blanks (which should not be there anyway) are translated to underscores, - - * a normalization of the ID is performed, based upon - `normalization tables`_. The purpose of this normalization is to ensure - that the ID is as reliable as possible, even across incompatible changes - in the OS distributions. A common reason for an incompatible change is - the addition of an os-release file, or the addition of the lsb_release - command, with ID values that differ from what was previously determined - from the distro release file name. - """ - return _distro.id() - - -def name(pretty: bool = False) -> str: - """ - Return the name of the current OS distribution, as a human-readable - string. - - If *pretty* is false, the name is returned without version or codename. - (e.g. "CentOS Linux") - - If *pretty* is true, the version and codename are appended. - (e.g. "CentOS Linux 7.1.1503 (Core)") - - **Lookup hierarchy:** - - The name is obtained from the following sources, in the specified order. - The first available and non-empty value is used: - - * If *pretty* is false: - - - the value of the "NAME" attribute of the os-release file, - - - the value of the "Distributor ID" attribute returned by the lsb_release - command, - - - the value of the "" field of the distro release file. - - * If *pretty* is true: - - - the value of the "PRETTY_NAME" attribute of the os-release file, - - - the value of the "Description" attribute returned by the lsb_release - command, - - - the value of the "" field of the distro release file, appended - with the value of the pretty version ("" and "" - fields) of the distro release file, if available. - """ - return _distro.name(pretty) - - -def version(pretty: bool = False, best: bool = False) -> str: - """ - Return the version of the current OS distribution, as a human-readable - string. - - If *pretty* is false, the version is returned without codename (e.g. - "7.0"). - - If *pretty* is true, the codename in parenthesis is appended, if the - codename is non-empty (e.g. "7.0 (Maipo)"). - - Some distributions provide version numbers with different precisions in - the different sources of distribution information. Examining the different - sources in a fixed priority order does not always yield the most precise - version (e.g. for Debian 8.2, or CentOS 7.1). - - Some other distributions may not provide this kind of information. In these - cases, an empty string would be returned. This behavior can be observed - with rolling releases distributions (e.g. Arch Linux). - - The *best* parameter can be used to control the approach for the returned - version: - - If *best* is false, the first non-empty version number in priority order of - the examined sources is returned. - - If *best* is true, the most precise version number out of all examined - sources is returned. - - **Lookup hierarchy:** - - In all cases, the version number is obtained from the following sources. - If *best* is false, this order represents the priority order: - - * the value of the "VERSION_ID" attribute of the os-release file, - * the value of the "Release" attribute returned by the lsb_release - command, - * the version number parsed from the "" field of the first line - of the distro release file, - * the version number parsed from the "PRETTY_NAME" attribute of the - os-release file, if it follows the format of the distro release files. - * the version number parsed from the "Description" attribute returned by - the lsb_release command, if it follows the format of the distro release - files. - """ - return _distro.version(pretty, best) - - -def version_parts(best: bool = False) -> Tuple[str, str, str]: - """ - Return the version of the current OS distribution as a tuple - ``(major, minor, build_number)`` with items as follows: - - * ``major``: The result of :func:`distro.major_version`. - - * ``minor``: The result of :func:`distro.minor_version`. - - * ``build_number``: The result of :func:`distro.build_number`. - - For a description of the *best* parameter, see the :func:`distro.version` - method. - """ - return _distro.version_parts(best) - - -def major_version(best: bool = False) -> str: - """ - Return the major version of the current OS distribution, as a string, - if provided. - Otherwise, the empty string is returned. The major version is the first - part of the dot-separated version string. - - For a description of the *best* parameter, see the :func:`distro.version` - method. - """ - return _distro.major_version(best) - - -def minor_version(best: bool = False) -> str: - """ - Return the minor version of the current OS distribution, as a string, - if provided. - Otherwise, the empty string is returned. The minor version is the second - part of the dot-separated version string. - - For a description of the *best* parameter, see the :func:`distro.version` - method. - """ - return _distro.minor_version(best) - - -def build_number(best: bool = False) -> str: - """ - Return the build number of the current OS distribution, as a string, - if provided. - Otherwise, the empty string is returned. The build number is the third part - of the dot-separated version string. - - For a description of the *best* parameter, see the :func:`distro.version` - method. - """ - return _distro.build_number(best) - - -def like() -> str: - """ - Return a space-separated list of distro IDs of distributions that are - closely related to the current OS distribution in regards to packaging - and programming interfaces, for example distributions the current - distribution is a derivative from. - - **Lookup hierarchy:** - - This information item is only provided by the os-release file. - For details, see the description of the "ID_LIKE" attribute in the - `os-release man page - `_. - """ - return _distro.like() - - -def codename() -> str: - """ - Return the codename for the release of the current OS distribution, - as a string. - - If the distribution does not have a codename, an empty string is returned. - - Note that the returned codename is not always really a codename. For - example, openSUSE returns "x86_64". This function does not handle such - cases in any special way and just returns the string it finds, if any. - - **Lookup hierarchy:** - - * the codename within the "VERSION" attribute of the os-release file, if - provided, - - * the value of the "Codename" attribute returned by the lsb_release - command, - - * the value of the "" field of the distro release file. - """ - return _distro.codename() - - -def info(pretty: bool = False, best: bool = False) -> InfoDict: - """ - Return certain machine-readable information items about the current OS - distribution in a dictionary, as shown in the following example: - - .. sourcecode:: python - - { - 'id': 'rhel', - 'version': '7.0', - 'version_parts': { - 'major': '7', - 'minor': '0', - 'build_number': '' - }, - 'like': 'fedora', - 'codename': 'Maipo' - } - - The dictionary structure and keys are always the same, regardless of which - information items are available in the underlying data sources. The values - for the various keys are as follows: - - * ``id``: The result of :func:`distro.id`. - - * ``version``: The result of :func:`distro.version`. - - * ``version_parts -> major``: The result of :func:`distro.major_version`. - - * ``version_parts -> minor``: The result of :func:`distro.minor_version`. - - * ``version_parts -> build_number``: The result of - :func:`distro.build_number`. - - * ``like``: The result of :func:`distro.like`. - - * ``codename``: The result of :func:`distro.codename`. - - For a description of the *pretty* and *best* parameters, see the - :func:`distro.version` method. - """ - return _distro.info(pretty, best) - - -def os_release_info() -> Dict[str, str]: - """ - Return a dictionary containing key-value pairs for the information items - from the os-release file data source of the current OS distribution. - - See `os-release file`_ for details about these information items. - """ - return _distro.os_release_info() - - -def lsb_release_info() -> Dict[str, str]: - """ - Return a dictionary containing key-value pairs for the information items - from the lsb_release command data source of the current OS distribution. - - See `lsb_release command output`_ for details about these information - items. - """ - return _distro.lsb_release_info() - - -def distro_release_info() -> Dict[str, str]: - """ - Return a dictionary containing key-value pairs for the information items - from the distro release file data source of the current OS distribution. - - See `distro release file`_ for details about these information items. - """ - return _distro.distro_release_info() - - -def uname_info() -> Dict[str, str]: - """ - Return a dictionary containing key-value pairs for the information items - from the distro release file data source of the current OS distribution. - """ - return _distro.uname_info() - - -def os_release_attr(attribute: str) -> str: - """ - Return a single named information item from the os-release file data source - of the current OS distribution. - - Parameters: - - * ``attribute`` (string): Key of the information item. - - Returns: - - * (string): Value of the information item, if the item exists. - The empty string, if the item does not exist. - - See `os-release file`_ for details about these information items. - """ - return _distro.os_release_attr(attribute) - - -def lsb_release_attr(attribute: str) -> str: - """ - Return a single named information item from the lsb_release command output - data source of the current OS distribution. - - Parameters: - - * ``attribute`` (string): Key of the information item. - - Returns: - - * (string): Value of the information item, if the item exists. - The empty string, if the item does not exist. - - See `lsb_release command output`_ for details about these information - items. - """ - return _distro.lsb_release_attr(attribute) - - -def distro_release_attr(attribute: str) -> str: - """ - Return a single named information item from the distro release file - data source of the current OS distribution. - - Parameters: - - * ``attribute`` (string): Key of the information item. - - Returns: - - * (string): Value of the information item, if the item exists. - The empty string, if the item does not exist. - - See `distro release file`_ for details about these information items. - """ - return _distro.distro_release_attr(attribute) - - -def uname_attr(attribute: str) -> str: - """ - Return a single named information item from the distro release file - data source of the current OS distribution. - - Parameters: - - * ``attribute`` (string): Key of the information item. - - Returns: - - * (string): Value of the information item, if the item exists. - The empty string, if the item does not exist. - """ - return _distro.uname_attr(attribute) - - -try: - from functools import cached_property -except ImportError: - # Python < 3.8 - class cached_property: # type: ignore - """A version of @property which caches the value. On access, it calls the - underlying function and sets the value in `__dict__` so future accesses - will not re-call the property. - """ - - def __init__(self, f: Callable[[Any], Any]) -> None: - self._fname = f.__name__ - self._f = f - - def __get__(self, obj: Any, owner: Type[Any]) -> Any: - assert obj is not None, f"call {self._fname} on an instance" - ret = obj.__dict__[self._fname] = self._f(obj) - return ret - - -class LinuxDistribution: - """ - Provides information about a OS distribution. - - This package creates a private module-global instance of this class with - default initialization arguments, that is used by the - `consolidated accessor functions`_ and `single source accessor functions`_. - By using default initialization arguments, that module-global instance - returns data about the current OS distribution (i.e. the distro this - package runs on). - - Normally, it is not necessary to create additional instances of this class. - However, in situations where control is needed over the exact data sources - that are used, instances of this class can be created with a specific - distro release file, or a specific os-release file, or without invoking the - lsb_release command. - """ - - def __init__( - self, - include_lsb: Optional[bool] = None, - os_release_file: str = "", - distro_release_file: str = "", - include_uname: Optional[bool] = None, - root_dir: Optional[str] = None, - include_oslevel: Optional[bool] = None, - ) -> None: - """ - The initialization method of this class gathers information from the - available data sources, and stores that in private instance attributes. - Subsequent access to the information items uses these private instance - attributes, so that the data sources are read only once. - - Parameters: - - * ``include_lsb`` (bool): Controls whether the - `lsb_release command output`_ is included as a data source. - - If the lsb_release command is not available in the program execution - path, the data source for the lsb_release command will be empty. - - * ``os_release_file`` (string): The path name of the - `os-release file`_ that is to be used as a data source. - - An empty string (the default) will cause the default path name to - be used (see `os-release file`_ for details). - - If the specified or defaulted os-release file does not exist, the - data source for the os-release file will be empty. - - * ``distro_release_file`` (string): The path name of the - `distro release file`_ that is to be used as a data source. - - An empty string (the default) will cause a default search algorithm - to be used (see `distro release file`_ for details). - - If the specified distro release file does not exist, or if no default - distro release file can be found, the data source for the distro - release file will be empty. - - * ``include_uname`` (bool): Controls whether uname command output is - included as a data source. If the uname command is not available in - the program execution path the data source for the uname command will - be empty. - - * ``root_dir`` (string): The absolute path to the root directory to use - to find distro-related information files. Note that ``include_*`` - parameters must not be enabled in combination with ``root_dir``. - - * ``include_oslevel`` (bool): Controls whether (AIX) oslevel command - output is included as a data source. If the oslevel command is not - available in the program execution path the data source will be - empty. - - Public instance attributes: - - * ``os_release_file`` (string): The path name of the - `os-release file`_ that is actually used as a data source. The - empty string if no distro release file is used as a data source. - - * ``distro_release_file`` (string): The path name of the - `distro release file`_ that is actually used as a data source. The - empty string if no distro release file is used as a data source. - - * ``include_lsb`` (bool): The result of the ``include_lsb`` parameter. - This controls whether the lsb information will be loaded. - - * ``include_uname`` (bool): The result of the ``include_uname`` - parameter. This controls whether the uname information will - be loaded. - - * ``include_oslevel`` (bool): The result of the ``include_oslevel`` - parameter. This controls whether (AIX) oslevel information will be - loaded. - - * ``root_dir`` (string): The result of the ``root_dir`` parameter. - The absolute path to the root directory to use to find distro-related - information files. - - Raises: - - * :py:exc:`ValueError`: Initialization parameters combination is not - supported. - - * :py:exc:`OSError`: Some I/O issue with an os-release file or distro - release file. - - * :py:exc:`UnicodeError`: A data source has unexpected characters or - uses an unexpected encoding. - """ - self.root_dir = root_dir - self.etc_dir = os.path.join(root_dir, "etc") if root_dir else _UNIXCONFDIR - self.usr_lib_dir = ( - os.path.join(root_dir, "usr/lib") if root_dir else _UNIXUSRLIBDIR - ) - - if os_release_file: - self.os_release_file = os_release_file - else: - etc_dir_os_release_file = os.path.join(self.etc_dir, _OS_RELEASE_BASENAME) - usr_lib_os_release_file = os.path.join( - self.usr_lib_dir, _OS_RELEASE_BASENAME - ) - - # NOTE: The idea is to respect order **and** have it set - # at all times for API backwards compatibility. - if os.path.isfile(etc_dir_os_release_file) or not os.path.isfile( - usr_lib_os_release_file - ): - self.os_release_file = etc_dir_os_release_file - else: - self.os_release_file = usr_lib_os_release_file - - self.distro_release_file = distro_release_file or "" # updated later - - is_root_dir_defined = root_dir is not None - if is_root_dir_defined and (include_lsb or include_uname or include_oslevel): - raise ValueError( - "Including subprocess data sources from specific root_dir is disallowed" - " to prevent false information" - ) - self.include_lsb = ( - include_lsb if include_lsb is not None else not is_root_dir_defined - ) - self.include_uname = ( - include_uname if include_uname is not None else not is_root_dir_defined - ) - self.include_oslevel = ( - include_oslevel if include_oslevel is not None else not is_root_dir_defined - ) - - def __repr__(self) -> str: - """Return repr of all info""" - return ( - "LinuxDistribution(" - "os_release_file={self.os_release_file!r}, " - "distro_release_file={self.distro_release_file!r}, " - "include_lsb={self.include_lsb!r}, " - "include_uname={self.include_uname!r}, " - "include_oslevel={self.include_oslevel!r}, " - "root_dir={self.root_dir!r}, " - "_os_release_info={self._os_release_info!r}, " - "_lsb_release_info={self._lsb_release_info!r}, " - "_distro_release_info={self._distro_release_info!r}, " - "_uname_info={self._uname_info!r}, " - "_oslevel_info={self._oslevel_info!r})".format(self=self) - ) - - def linux_distribution( - self, full_distribution_name: bool = True - ) -> Tuple[str, str, str]: - """ - Return information about the OS distribution that is compatible - with Python's :func:`platform.linux_distribution`, supporting a subset - of its parameters. - - For details, see :func:`distro.linux_distribution`. - """ - return ( - self.name() if full_distribution_name else self.id(), - self.version(), - self._os_release_info.get("release_codename") or self.codename(), - ) - - def id(self) -> str: - """Return the distro ID of the OS distribution, as a string. - - For details, see :func:`distro.id`. - """ - - def normalize(distro_id: str, table: Dict[str, str]) -> str: - distro_id = distro_id.lower().replace(" ", "_") - return table.get(distro_id, distro_id) - - distro_id = self.os_release_attr("id") - if distro_id: - return normalize(distro_id, NORMALIZED_OS_ID) - - distro_id = self.lsb_release_attr("distributor_id") - if distro_id: - return normalize(distro_id, NORMALIZED_LSB_ID) - - distro_id = self.distro_release_attr("id") - if distro_id: - return normalize(distro_id, NORMALIZED_DISTRO_ID) - - distro_id = self.uname_attr("id") - if distro_id: - return normalize(distro_id, NORMALIZED_DISTRO_ID) - - return "" - - def name(self, pretty: bool = False) -> str: - """ - Return the name of the OS distribution, as a string. - - For details, see :func:`distro.name`. - """ - name = ( - self.os_release_attr("name") - or self.lsb_release_attr("distributor_id") - or self.distro_release_attr("name") - or self.uname_attr("name") - ) - if pretty: - name = self.os_release_attr("pretty_name") or self.lsb_release_attr( - "description" - ) - if not name: - name = self.distro_release_attr("name") or self.uname_attr("name") - version = self.version(pretty=True) - if version: - name = f"{name} {version}" - return name or "" - - def version(self, pretty: bool = False, best: bool = False) -> str: - """ - Return the version of the OS distribution, as a string. - - For details, see :func:`distro.version`. - """ - versions = [ - self.os_release_attr("version_id"), - self.lsb_release_attr("release"), - self.distro_release_attr("version_id"), - self._parse_distro_release_content(self.os_release_attr("pretty_name")).get( - "version_id", "" - ), - self._parse_distro_release_content( - self.lsb_release_attr("description") - ).get("version_id", ""), - self.uname_attr("release"), - ] - if self.uname_attr("id").startswith("aix"): - # On AIX platforms, prefer oslevel command output. - versions.insert(0, self.oslevel_info()) - elif self.id() == "debian" or "debian" in self.like().split(): - # On Debian-like, add debian_version file content to candidates list. - versions.append(self._debian_version) - version = "" - if best: - # This algorithm uses the last version in priority order that has - # the best precision. If the versions are not in conflict, that - # does not matter; otherwise, using the last one instead of the - # first one might be considered a surprise. - for v in versions: - if v.count(".") > version.count(".") or version == "": - version = v - else: - for v in versions: - if v != "": - version = v - break - if pretty and version and self.codename(): - version = f"{version} ({self.codename()})" - return version - - def version_parts(self, best: bool = False) -> Tuple[str, str, str]: - """ - Return the version of the OS distribution, as a tuple of version - numbers. - - For details, see :func:`distro.version_parts`. - """ - version_str = self.version(best=best) - if version_str: - version_regex = re.compile(r"(\d+)\.?(\d+)?\.?(\d+)?") - matches = version_regex.match(version_str) - if matches: - major, minor, build_number = matches.groups() - return major, minor or "", build_number or "" - return "", "", "" - - def major_version(self, best: bool = False) -> str: - """ - Return the major version number of the current distribution. - - For details, see :func:`distro.major_version`. - """ - return self.version_parts(best)[0] - - def minor_version(self, best: bool = False) -> str: - """ - Return the minor version number of the current distribution. - - For details, see :func:`distro.minor_version`. - """ - return self.version_parts(best)[1] - - def build_number(self, best: bool = False) -> str: - """ - Return the build number of the current distribution. - - For details, see :func:`distro.build_number`. - """ - return self.version_parts(best)[2] - - def like(self) -> str: - """ - Return the IDs of distributions that are like the OS distribution. - - For details, see :func:`distro.like`. - """ - return self.os_release_attr("id_like") or "" - - def codename(self) -> str: - """ - Return the codename of the OS distribution. - - For details, see :func:`distro.codename`. - """ - try: - # Handle os_release specially since distros might purposefully set - # this to empty string to have no codename - return self._os_release_info["codename"] - except KeyError: - return ( - self.lsb_release_attr("codename") - or self.distro_release_attr("codename") - or "" - ) - - def info(self, pretty: bool = False, best: bool = False) -> InfoDict: - """ - Return certain machine-readable information about the OS - distribution. - - For details, see :func:`distro.info`. - """ - return dict( - id=self.id(), - version=self.version(pretty, best), - version_parts=dict( - major=self.major_version(best), - minor=self.minor_version(best), - build_number=self.build_number(best), - ), - like=self.like(), - codename=self.codename(), - ) - - def os_release_info(self) -> Dict[str, str]: - """ - Return a dictionary containing key-value pairs for the information - items from the os-release file data source of the OS distribution. - - For details, see :func:`distro.os_release_info`. - """ - return self._os_release_info - - def lsb_release_info(self) -> Dict[str, str]: - """ - Return a dictionary containing key-value pairs for the information - items from the lsb_release command data source of the OS - distribution. - - For details, see :func:`distro.lsb_release_info`. - """ - return self._lsb_release_info - - def distro_release_info(self) -> Dict[str, str]: - """ - Return a dictionary containing key-value pairs for the information - items from the distro release file data source of the OS - distribution. - - For details, see :func:`distro.distro_release_info`. - """ - return self._distro_release_info - - def uname_info(self) -> Dict[str, str]: - """ - Return a dictionary containing key-value pairs for the information - items from the uname command data source of the OS distribution. - - For details, see :func:`distro.uname_info`. - """ - return self._uname_info - - def oslevel_info(self) -> str: - """ - Return AIX' oslevel command output. - """ - return self._oslevel_info - - def os_release_attr(self, attribute: str) -> str: - """ - Return a single named information item from the os-release file data - source of the OS distribution. - - For details, see :func:`distro.os_release_attr`. - """ - return self._os_release_info.get(attribute, "") - - def lsb_release_attr(self, attribute: str) -> str: - """ - Return a single named information item from the lsb_release command - output data source of the OS distribution. - - For details, see :func:`distro.lsb_release_attr`. - """ - return self._lsb_release_info.get(attribute, "") - - def distro_release_attr(self, attribute: str) -> str: - """ - Return a single named information item from the distro release file - data source of the OS distribution. - - For details, see :func:`distro.distro_release_attr`. - """ - return self._distro_release_info.get(attribute, "") - - def uname_attr(self, attribute: str) -> str: - """ - Return a single named information item from the uname command - output data source of the OS distribution. - - For details, see :func:`distro.uname_attr`. - """ - return self._uname_info.get(attribute, "") - - @cached_property - def _os_release_info(self) -> Dict[str, str]: - """ - Get the information items from the specified os-release file. - - Returns: - A dictionary containing all information items. - """ - if os.path.isfile(self.os_release_file): - with open(self.os_release_file, encoding="utf-8") as release_file: - return self._parse_os_release_content(release_file) - return {} - - @staticmethod - def _parse_os_release_content(lines: TextIO) -> Dict[str, str]: - """ - Parse the lines of an os-release file. - - Parameters: - - * lines: Iterable through the lines in the os-release file. - Each line must be a unicode string or a UTF-8 encoded byte - string. - - Returns: - A dictionary containing all information items. - """ - props = {} - lexer = shlex.shlex(lines, posix=True) - lexer.whitespace_split = True - - tokens = list(lexer) - for token in tokens: - # At this point, all shell-like parsing has been done (i.e. - # comments processed, quotes and backslash escape sequences - # processed, multi-line values assembled, trailing newlines - # stripped, etc.), so the tokens are now either: - # * variable assignments: var=value - # * commands or their arguments (not allowed in os-release) - # Ignore any tokens that are not variable assignments - if "=" in token: - k, v = token.split("=", 1) - props[k.lower()] = v - - if "version" in props: - # extract release codename (if any) from version attribute - match = re.search(r"\((\D+)\)|,\s*(\D+)", props["version"]) - if match: - release_codename = match.group(1) or match.group(2) - props["codename"] = props["release_codename"] = release_codename - - if "version_codename" in props: - # os-release added a version_codename field. Use that in - # preference to anything else Note that some distros purposefully - # do not have code names. They should be setting - # version_codename="" - props["codename"] = props["version_codename"] - elif "ubuntu_codename" in props: - # Same as above but a non-standard field name used on older Ubuntus - props["codename"] = props["ubuntu_codename"] - - return props - - @cached_property - def _lsb_release_info(self) -> Dict[str, str]: - """ - Get the information items from the lsb_release command output. - - Returns: - A dictionary containing all information items. - """ - if not self.include_lsb: - return {} - try: - cmd = ("lsb_release", "-a") - stdout = subprocess.check_output(cmd, stderr=subprocess.DEVNULL) - # Command not found or lsb_release returned error - except (OSError, subprocess.CalledProcessError): - return {} - content = self._to_str(stdout).splitlines() - return self._parse_lsb_release_content(content) - - @staticmethod - def _parse_lsb_release_content(lines: Iterable[str]) -> Dict[str, str]: - """ - Parse the output of the lsb_release command. - - Parameters: - - * lines: Iterable through the lines of the lsb_release output. - Each line must be a unicode string or a UTF-8 encoded byte - string. - - Returns: - A dictionary containing all information items. - """ - props = {} - for line in lines: - kv = line.strip("\n").split(":", 1) - if len(kv) != 2: - # Ignore lines without colon. - continue - k, v = kv - props.update({k.replace(" ", "_").lower(): v.strip()}) - return props - - @cached_property - def _uname_info(self) -> Dict[str, str]: - if not self.include_uname: - return {} - try: - cmd = ("uname", "-rs") - stdout = subprocess.check_output(cmd, stderr=subprocess.DEVNULL) - except OSError: - return {} - content = self._to_str(stdout).splitlines() - return self._parse_uname_content(content) - - @cached_property - def _oslevel_info(self) -> str: - if not self.include_oslevel: - return "" - try: - stdout = subprocess.check_output("oslevel", stderr=subprocess.DEVNULL) - except (OSError, subprocess.CalledProcessError): - return "" - return self._to_str(stdout).strip() - - @cached_property - def _debian_version(self) -> str: - try: - with open( - os.path.join(self.etc_dir, "debian_version"), encoding="ascii" - ) as fp: - return fp.readline().rstrip() - except FileNotFoundError: - return "" - - @staticmethod - def _parse_uname_content(lines: Sequence[str]) -> Dict[str, str]: - if not lines: - return {} - props = {} - match = re.search(r"^([^\s]+)\s+([\d\.]+)", lines[0].strip()) - if match: - name, version = match.groups() - - # This is to prevent the Linux kernel version from - # appearing as the 'best' version on otherwise - # identifiable distributions. - if name == "Linux": - return {} - props["id"] = name.lower() - props["name"] = name - props["release"] = version - return props - - @staticmethod - def _to_str(bytestring: bytes) -> str: - encoding = sys.getfilesystemencoding() - return bytestring.decode(encoding) - - @cached_property - def _distro_release_info(self) -> Dict[str, str]: - """ - Get the information items from the specified distro release file. - - Returns: - A dictionary containing all information items. - """ - if self.distro_release_file: - # If it was specified, we use it and parse what we can, even if - # its file name or content does not match the expected pattern. - distro_info = self._parse_distro_release_file(self.distro_release_file) - basename = os.path.basename(self.distro_release_file) - # The file name pattern for user-specified distro release files - # is somewhat more tolerant (compared to when searching for the - # file), because we want to use what was specified as best as - # possible. - match = _DISTRO_RELEASE_BASENAME_PATTERN.match(basename) - else: - try: - basenames = [ - basename - for basename in os.listdir(self.etc_dir) - if basename not in _DISTRO_RELEASE_IGNORE_BASENAMES - and os.path.isfile(os.path.join(self.etc_dir, basename)) - ] - # We sort for repeatability in cases where there are multiple - # distro specific files; e.g. CentOS, Oracle, Enterprise all - # containing `redhat-release` on top of their own. - basenames.sort() - except OSError: - # This may occur when /etc is not readable but we can't be - # sure about the *-release files. Check common entries of - # /etc for information. If they turn out to not be there the - # error is handled in `_parse_distro_release_file()`. - basenames = _DISTRO_RELEASE_BASENAMES - for basename in basenames: - match = _DISTRO_RELEASE_BASENAME_PATTERN.match(basename) - if match is None: - continue - filepath = os.path.join(self.etc_dir, basename) - distro_info = self._parse_distro_release_file(filepath) - # The name is always present if the pattern matches. - if "name" not in distro_info: - continue - self.distro_release_file = filepath - break - else: # the loop didn't "break": no candidate. - return {} - - if match is not None: - distro_info["id"] = match.group(1) - - # CloudLinux < 7: manually enrich info with proper id. - if "cloudlinux" in distro_info.get("name", "").lower(): - distro_info["id"] = "cloudlinux" - - return distro_info - - def _parse_distro_release_file(self, filepath: str) -> Dict[str, str]: - """ - Parse a distro release file. - - Parameters: - - * filepath: Path name of the distro release file. - - Returns: - A dictionary containing all information items. - """ - try: - with open(filepath, encoding="utf-8") as fp: - # Only parse the first line. For instance, on SLES there - # are multiple lines. We don't want them... - return self._parse_distro_release_content(fp.readline()) - except OSError: - # Ignore not being able to read a specific, seemingly version - # related file. - # See https://github.com/python-distro/distro/issues/162 - return {} - - @staticmethod - def _parse_distro_release_content(line: str) -> Dict[str, str]: - """ - Parse a line from a distro release file. - - Parameters: - * line: Line from the distro release file. Must be a unicode string - or a UTF-8 encoded byte string. - - Returns: - A dictionary containing all information items. - """ - matches = _DISTRO_RELEASE_CONTENT_REVERSED_PATTERN.match(line.strip()[::-1]) - distro_info = {} - if matches: - # regexp ensures non-None - distro_info["name"] = matches.group(3)[::-1] - if matches.group(2): - distro_info["version_id"] = matches.group(2)[::-1] - if matches.group(1): - distro_info["codename"] = matches.group(1)[::-1] - elif line: - distro_info["name"] = line.strip() - return distro_info - - -_distro = LinuxDistribution() - - -def main() -> None: - logger = logging.getLogger(__name__) - logger.setLevel(logging.DEBUG) - logger.addHandler(logging.StreamHandler(sys.stdout)) - - parser = argparse.ArgumentParser(description="OS distro info tool") - parser.add_argument( - "--json", "-j", help="Output in machine readable format", action="store_true" - ) - - parser.add_argument( - "--root-dir", - "-r", - type=str, - dest="root_dir", - help="Path to the root filesystem directory (defaults to /)", - ) - - args = parser.parse_args() - - if args.root_dir: - dist = LinuxDistribution( - include_lsb=False, - include_uname=False, - include_oslevel=False, - root_dir=args.root_dir, - ) - else: - dist = _distro - - if args.json: - logger.info(json.dumps(dist.info(), indent=4, sort_keys=True)) - else: - logger.info("Name: %s", dist.name(pretty=True)) - distribution_version = dist.version(pretty=True) - logger.info("Version: %s", distribution_version) - distribution_codename = dist.codename() - logger.info("Codename: %s", distribution_codename) - - -if __name__ == "__main__": - main() diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/idna/__init__.py b/venv/lib/python3.11/site-packages/pip/_vendor/idna/__init__.py deleted file mode 100644 index a40eeaf..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/idna/__init__.py +++ /dev/null @@ -1,44 +0,0 @@ -from .package_data import __version__ -from .core import ( - IDNABidiError, - IDNAError, - InvalidCodepoint, - InvalidCodepointContext, - alabel, - check_bidi, - check_hyphen_ok, - check_initial_combiner, - check_label, - check_nfc, - decode, - encode, - ulabel, - uts46_remap, - valid_contextj, - valid_contexto, - valid_label_length, - valid_string_length, -) -from .intranges import intranges_contain - -__all__ = [ - "IDNABidiError", - "IDNAError", - "InvalidCodepoint", - "InvalidCodepointContext", - "alabel", - "check_bidi", - "check_hyphen_ok", - "check_initial_combiner", - "check_label", - "check_nfc", - "decode", - "encode", - "intranges_contain", - "ulabel", - "uts46_remap", - "valid_contextj", - "valid_contexto", - "valid_label_length", - "valid_string_length", -] diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/idna/__pycache__/__init__.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/idna/__pycache__/__init__.cpython-311.pyc deleted file mode 100644 index 663e18c..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/idna/__pycache__/__init__.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/idna/__pycache__/codec.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/idna/__pycache__/codec.cpython-311.pyc deleted file mode 100644 index 4b0a608..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/idna/__pycache__/codec.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/idna/__pycache__/compat.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/idna/__pycache__/compat.cpython-311.pyc deleted file mode 100644 index 9d45101..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/idna/__pycache__/compat.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/idna/__pycache__/core.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/idna/__pycache__/core.cpython-311.pyc deleted file mode 100644 index 1809358..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/idna/__pycache__/core.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/idna/__pycache__/idnadata.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/idna/__pycache__/idnadata.cpython-311.pyc deleted file mode 100644 index efaea75..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/idna/__pycache__/idnadata.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/idna/__pycache__/intranges.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/idna/__pycache__/intranges.cpython-311.pyc deleted file mode 100644 index 479da5b..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/idna/__pycache__/intranges.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/idna/__pycache__/package_data.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/idna/__pycache__/package_data.cpython-311.pyc deleted file mode 100644 index 4a2b91f..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/idna/__pycache__/package_data.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/idna/__pycache__/uts46data.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/idna/__pycache__/uts46data.cpython-311.pyc deleted file mode 100644 index deadbfa..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/idna/__pycache__/uts46data.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/idna/codec.py b/venv/lib/python3.11/site-packages/pip/_vendor/idna/codec.py deleted file mode 100644 index 1ca9ba6..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/idna/codec.py +++ /dev/null @@ -1,112 +0,0 @@ -from .core import encode, decode, alabel, ulabel, IDNAError -import codecs -import re -from typing import Tuple, Optional - -_unicode_dots_re = re.compile('[\u002e\u3002\uff0e\uff61]') - -class Codec(codecs.Codec): - - def encode(self, data: str, errors: str = 'strict') -> Tuple[bytes, int]: - if errors != 'strict': - raise IDNAError('Unsupported error handling \"{}\"'.format(errors)) - - if not data: - return b"", 0 - - return encode(data), len(data) - - def decode(self, data: bytes, errors: str = 'strict') -> Tuple[str, int]: - if errors != 'strict': - raise IDNAError('Unsupported error handling \"{}\"'.format(errors)) - - if not data: - return '', 0 - - return decode(data), len(data) - -class IncrementalEncoder(codecs.BufferedIncrementalEncoder): - def _buffer_encode(self, data: str, errors: str, final: bool) -> Tuple[str, int]: # type: ignore - if errors != 'strict': - raise IDNAError('Unsupported error handling \"{}\"'.format(errors)) - - if not data: - return "", 0 - - labels = _unicode_dots_re.split(data) - trailing_dot = '' - if labels: - if not labels[-1]: - trailing_dot = '.' - del labels[-1] - elif not final: - # Keep potentially unfinished label until the next call - del labels[-1] - if labels: - trailing_dot = '.' - - result = [] - size = 0 - for label in labels: - result.append(alabel(label)) - if size: - size += 1 - size += len(label) - - # Join with U+002E - result_str = '.'.join(result) + trailing_dot # type: ignore - size += len(trailing_dot) - return result_str, size - -class IncrementalDecoder(codecs.BufferedIncrementalDecoder): - def _buffer_decode(self, data: str, errors: str, final: bool) -> Tuple[str, int]: # type: ignore - if errors != 'strict': - raise IDNAError('Unsupported error handling \"{}\"'.format(errors)) - - if not data: - return ('', 0) - - labels = _unicode_dots_re.split(data) - trailing_dot = '' - if labels: - if not labels[-1]: - trailing_dot = '.' - del labels[-1] - elif not final: - # Keep potentially unfinished label until the next call - del labels[-1] - if labels: - trailing_dot = '.' - - result = [] - size = 0 - for label in labels: - result.append(ulabel(label)) - if size: - size += 1 - size += len(label) - - result_str = '.'.join(result) + trailing_dot - size += len(trailing_dot) - return (result_str, size) - - -class StreamWriter(Codec, codecs.StreamWriter): - pass - - -class StreamReader(Codec, codecs.StreamReader): - pass - - -def getregentry() -> codecs.CodecInfo: - # Compatibility as a search_function for codecs.register() - return codecs.CodecInfo( - name='idna', - encode=Codec().encode, # type: ignore - decode=Codec().decode, # type: ignore - incrementalencoder=IncrementalEncoder, - incrementaldecoder=IncrementalDecoder, - streamwriter=StreamWriter, - streamreader=StreamReader, - ) diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/idna/compat.py b/venv/lib/python3.11/site-packages/pip/_vendor/idna/compat.py deleted file mode 100644 index 786e6bd..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/idna/compat.py +++ /dev/null @@ -1,13 +0,0 @@ -from .core import * -from .codec import * -from typing import Any, Union - -def ToASCII(label: str) -> bytes: - return encode(label) - -def ToUnicode(label: Union[bytes, bytearray]) -> str: - return decode(label) - -def nameprep(s: Any) -> None: - raise NotImplementedError('IDNA 2008 does not utilise nameprep protocol') - diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/idna/core.py b/venv/lib/python3.11/site-packages/pip/_vendor/idna/core.py deleted file mode 100644 index 4f30037..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/idna/core.py +++ /dev/null @@ -1,400 +0,0 @@ -from . import idnadata -import bisect -import unicodedata -import re -from typing import Union, Optional -from .intranges import intranges_contain - -_virama_combining_class = 9 -_alabel_prefix = b'xn--' -_unicode_dots_re = re.compile('[\u002e\u3002\uff0e\uff61]') - -class IDNAError(UnicodeError): - """ Base exception for all IDNA-encoding related problems """ - pass - - -class IDNABidiError(IDNAError): - """ Exception when bidirectional requirements are not satisfied """ - pass - - -class InvalidCodepoint(IDNAError): - """ Exception when a disallowed or unallocated codepoint is used """ - pass - - -class InvalidCodepointContext(IDNAError): - """ Exception when the codepoint is not valid in the context it is used """ - pass - - -def _combining_class(cp: int) -> int: - v = unicodedata.combining(chr(cp)) - if v == 0: - if not unicodedata.name(chr(cp)): - raise ValueError('Unknown character in unicodedata') - return v - -def _is_script(cp: str, script: str) -> bool: - return intranges_contain(ord(cp), idnadata.scripts[script]) - -def _punycode(s: str) -> bytes: - return s.encode('punycode') - -def _unot(s: int) -> str: - return 'U+{:04X}'.format(s) - - -def valid_label_length(label: Union[bytes, str]) -> bool: - if len(label) > 63: - return False - return True - - -def valid_string_length(label: Union[bytes, str], trailing_dot: bool) -> bool: - if len(label) > (254 if trailing_dot else 253): - return False - return True - - -def check_bidi(label: str, check_ltr: bool = False) -> bool: - # Bidi rules should only be applied if string contains RTL characters - bidi_label = False - for (idx, cp) in enumerate(label, 1): - direction = unicodedata.bidirectional(cp) - if direction == '': - # String likely comes from a newer version of Unicode - raise IDNABidiError('Unknown directionality in label {} at position {}'.format(repr(label), idx)) - if direction in ['R', 'AL', 'AN']: - bidi_label = True - if not bidi_label and not check_ltr: - return True - - # Bidi rule 1 - direction = unicodedata.bidirectional(label[0]) - if direction in ['R', 'AL']: - rtl = True - elif direction == 'L': - rtl = False - else: - raise IDNABidiError('First codepoint in label {} must be directionality L, R or AL'.format(repr(label))) - - valid_ending = False - number_type = None # type: Optional[str] - for (idx, cp) in enumerate(label, 1): - direction = unicodedata.bidirectional(cp) - - if rtl: - # Bidi rule 2 - if not direction in ['R', 'AL', 'AN', 'EN', 'ES', 'CS', 'ET', 'ON', 'BN', 'NSM']: - raise IDNABidiError('Invalid direction for codepoint at position {} in a right-to-left label'.format(idx)) - # Bidi rule 3 - if direction in ['R', 'AL', 'EN', 'AN']: - valid_ending = True - elif direction != 'NSM': - valid_ending = False - # Bidi rule 4 - if direction in ['AN', 'EN']: - if not number_type: - number_type = direction - else: - if number_type != direction: - raise IDNABidiError('Can not mix numeral types in a right-to-left label') - else: - # Bidi rule 5 - if not direction in ['L', 'EN', 'ES', 'CS', 'ET', 'ON', 'BN', 'NSM']: - raise IDNABidiError('Invalid direction for codepoint at position {} in a left-to-right label'.format(idx)) - # Bidi rule 6 - if direction in ['L', 'EN']: - valid_ending = True - elif direction != 'NSM': - valid_ending = False - - if not valid_ending: - raise IDNABidiError('Label ends with illegal codepoint directionality') - - return True - - -def check_initial_combiner(label: str) -> bool: - if unicodedata.category(label[0])[0] == 'M': - raise IDNAError('Label begins with an illegal combining character') - return True - - -def check_hyphen_ok(label: str) -> bool: - if label[2:4] == '--': - raise IDNAError('Label has disallowed hyphens in 3rd and 4th position') - if label[0] == '-' or label[-1] == '-': - raise IDNAError('Label must not start or end with a hyphen') - return True - - -def check_nfc(label: str) -> None: - if unicodedata.normalize('NFC', label) != label: - raise IDNAError('Label must be in Normalization Form C') - - -def valid_contextj(label: str, pos: int) -> bool: - cp_value = ord(label[pos]) - - if cp_value == 0x200c: - - if pos > 0: - if _combining_class(ord(label[pos - 1])) == _virama_combining_class: - return True - - ok = False - for i in range(pos-1, -1, -1): - joining_type = idnadata.joining_types.get(ord(label[i])) - if joining_type == ord('T'): - continue - if joining_type in [ord('L'), ord('D')]: - ok = True - break - - if not ok: - return False - - ok = False - for i in range(pos+1, len(label)): - joining_type = idnadata.joining_types.get(ord(label[i])) - if joining_type == ord('T'): - continue - if joining_type in [ord('R'), ord('D')]: - ok = True - break - return ok - - if cp_value == 0x200d: - - if pos > 0: - if _combining_class(ord(label[pos - 1])) == _virama_combining_class: - return True - return False - - else: - - return False - - -def valid_contexto(label: str, pos: int, exception: bool = False) -> bool: - cp_value = ord(label[pos]) - - if cp_value == 0x00b7: - if 0 < pos < len(label)-1: - if ord(label[pos - 1]) == 0x006c and ord(label[pos + 1]) == 0x006c: - return True - return False - - elif cp_value == 0x0375: - if pos < len(label)-1 and len(label) > 1: - return _is_script(label[pos + 1], 'Greek') - return False - - elif cp_value == 0x05f3 or cp_value == 0x05f4: - if pos > 0: - return _is_script(label[pos - 1], 'Hebrew') - return False - - elif cp_value == 0x30fb: - for cp in label: - if cp == '\u30fb': - continue - if _is_script(cp, 'Hiragana') or _is_script(cp, 'Katakana') or _is_script(cp, 'Han'): - return True - return False - - elif 0x660 <= cp_value <= 0x669: - for cp in label: - if 0x6f0 <= ord(cp) <= 0x06f9: - return False - return True - - elif 0x6f0 <= cp_value <= 0x6f9: - for cp in label: - if 0x660 <= ord(cp) <= 0x0669: - return False - return True - - return False - - -def check_label(label: Union[str, bytes, bytearray]) -> None: - if isinstance(label, (bytes, bytearray)): - label = label.decode('utf-8') - if len(label) == 0: - raise IDNAError('Empty Label') - - check_nfc(label) - check_hyphen_ok(label) - check_initial_combiner(label) - - for (pos, cp) in enumerate(label): - cp_value = ord(cp) - if intranges_contain(cp_value, idnadata.codepoint_classes['PVALID']): - continue - elif intranges_contain(cp_value, idnadata.codepoint_classes['CONTEXTJ']): - try: - if not valid_contextj(label, pos): - raise InvalidCodepointContext('Joiner {} not allowed at position {} in {}'.format( - _unot(cp_value), pos+1, repr(label))) - except ValueError: - raise IDNAError('Unknown codepoint adjacent to joiner {} at position {} in {}'.format( - _unot(cp_value), pos+1, repr(label))) - elif intranges_contain(cp_value, idnadata.codepoint_classes['CONTEXTO']): - if not valid_contexto(label, pos): - raise InvalidCodepointContext('Codepoint {} not allowed at position {} in {}'.format(_unot(cp_value), pos+1, repr(label))) - else: - raise InvalidCodepoint('Codepoint {} at position {} of {} not allowed'.format(_unot(cp_value), pos+1, repr(label))) - - check_bidi(label) - - -def alabel(label: str) -> bytes: - try: - label_bytes = label.encode('ascii') - ulabel(label_bytes) - if not valid_label_length(label_bytes): - raise IDNAError('Label too long') - return label_bytes - except UnicodeEncodeError: - pass - - if not label: - raise IDNAError('No Input') - - label = str(label) - check_label(label) - label_bytes = _punycode(label) - label_bytes = _alabel_prefix + label_bytes - - if not valid_label_length(label_bytes): - raise IDNAError('Label too long') - - return label_bytes - - -def ulabel(label: Union[str, bytes, bytearray]) -> str: - if not isinstance(label, (bytes, bytearray)): - try: - label_bytes = label.encode('ascii') - except UnicodeEncodeError: - check_label(label) - return label - else: - label_bytes = label - - label_bytes = label_bytes.lower() - if label_bytes.startswith(_alabel_prefix): - label_bytes = label_bytes[len(_alabel_prefix):] - if not label_bytes: - raise IDNAError('Malformed A-label, no Punycode eligible content found') - if label_bytes.decode('ascii')[-1] == '-': - raise IDNAError('A-label must not end with a hyphen') - else: - check_label(label_bytes) - return label_bytes.decode('ascii') - - try: - label = label_bytes.decode('punycode') - except UnicodeError: - raise IDNAError('Invalid A-label') - check_label(label) - return label - - -def uts46_remap(domain: str, std3_rules: bool = True, transitional: bool = False) -> str: - """Re-map the characters in the string according to UTS46 processing.""" - from .uts46data import uts46data - output = '' - - for pos, char in enumerate(domain): - code_point = ord(char) - try: - uts46row = uts46data[code_point if code_point < 256 else - bisect.bisect_left(uts46data, (code_point, 'Z')) - 1] - status = uts46row[1] - replacement = None # type: Optional[str] - if len(uts46row) == 3: - replacement = uts46row[2] # type: ignore - if (status == 'V' or - (status == 'D' and not transitional) or - (status == '3' and not std3_rules and replacement is None)): - output += char - elif replacement is not None and (status == 'M' or - (status == '3' and not std3_rules) or - (status == 'D' and transitional)): - output += replacement - elif status != 'I': - raise IndexError() - except IndexError: - raise InvalidCodepoint( - 'Codepoint {} not allowed at position {} in {}'.format( - _unot(code_point), pos + 1, repr(domain))) - - return unicodedata.normalize('NFC', output) - - -def encode(s: Union[str, bytes, bytearray], strict: bool = False, uts46: bool = False, std3_rules: bool = False, transitional: bool = False) -> bytes: - if isinstance(s, (bytes, bytearray)): - try: - s = s.decode('ascii') - except UnicodeDecodeError: - raise IDNAError('should pass a unicode string to the function rather than a byte string.') - if uts46: - s = uts46_remap(s, std3_rules, transitional) - trailing_dot = False - result = [] - if strict: - labels = s.split('.') - else: - labels = _unicode_dots_re.split(s) - if not labels or labels == ['']: - raise IDNAError('Empty domain') - if labels[-1] == '': - del labels[-1] - trailing_dot = True - for label in labels: - s = alabel(label) - if s: - result.append(s) - else: - raise IDNAError('Empty label') - if trailing_dot: - result.append(b'') - s = b'.'.join(result) - if not valid_string_length(s, trailing_dot): - raise IDNAError('Domain too long') - return s - - -def decode(s: Union[str, bytes, bytearray], strict: bool = False, uts46: bool = False, std3_rules: bool = False) -> str: - try: - if isinstance(s, (bytes, bytearray)): - s = s.decode('ascii') - except UnicodeDecodeError: - raise IDNAError('Invalid ASCII in A-label') - if uts46: - s = uts46_remap(s, std3_rules, False) - trailing_dot = False - result = [] - if not strict: - labels = _unicode_dots_re.split(s) - else: - labels = s.split('.') - if not labels or labels == ['']: - raise IDNAError('Empty domain') - if not labels[-1]: - del labels[-1] - trailing_dot = True - for label in labels: - s = ulabel(label) - if s: - result.append(s) - else: - raise IDNAError('Empty label') - if trailing_dot: - result.append('') - return '.'.join(result) diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/idna/idnadata.py b/venv/lib/python3.11/site-packages/pip/_vendor/idna/idnadata.py deleted file mode 100644 index 67db462..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/idna/idnadata.py +++ /dev/null @@ -1,2151 +0,0 @@ -# This file is automatically generated by tools/idna-data - -__version__ = '15.0.0' -scripts = { - 'Greek': ( - 0x37000000374, - 0x37500000378, - 0x37a0000037e, - 0x37f00000380, - 0x38400000385, - 0x38600000387, - 0x3880000038b, - 0x38c0000038d, - 0x38e000003a2, - 0x3a3000003e2, - 0x3f000000400, - 0x1d2600001d2b, - 0x1d5d00001d62, - 0x1d6600001d6b, - 0x1dbf00001dc0, - 0x1f0000001f16, - 0x1f1800001f1e, - 0x1f2000001f46, - 0x1f4800001f4e, - 0x1f5000001f58, - 0x1f5900001f5a, - 0x1f5b00001f5c, - 0x1f5d00001f5e, - 0x1f5f00001f7e, - 0x1f8000001fb5, - 0x1fb600001fc5, - 0x1fc600001fd4, - 0x1fd600001fdc, - 0x1fdd00001ff0, - 0x1ff200001ff5, - 0x1ff600001fff, - 0x212600002127, - 0xab650000ab66, - 0x101400001018f, - 0x101a0000101a1, - 0x1d2000001d246, - ), - 'Han': ( - 0x2e8000002e9a, - 0x2e9b00002ef4, - 0x2f0000002fd6, - 0x300500003006, - 0x300700003008, - 0x30210000302a, - 0x30380000303c, - 0x340000004dc0, - 0x4e000000a000, - 0xf9000000fa6e, - 0xfa700000fada, - 0x16fe200016fe4, - 0x16ff000016ff2, - 0x200000002a6e0, - 0x2a7000002b73a, - 0x2b7400002b81e, - 0x2b8200002cea2, - 0x2ceb00002ebe1, - 0x2f8000002fa1e, - 0x300000003134b, - 0x31350000323b0, - ), - 'Hebrew': ( - 0x591000005c8, - 0x5d0000005eb, - 0x5ef000005f5, - 0xfb1d0000fb37, - 0xfb380000fb3d, - 0xfb3e0000fb3f, - 0xfb400000fb42, - 0xfb430000fb45, - 0xfb460000fb50, - ), - 'Hiragana': ( - 0x304100003097, - 0x309d000030a0, - 0x1b0010001b120, - 0x1b1320001b133, - 0x1b1500001b153, - 0x1f2000001f201, - ), - 'Katakana': ( - 0x30a1000030fb, - 0x30fd00003100, - 0x31f000003200, - 0x32d0000032ff, - 0x330000003358, - 0xff660000ff70, - 0xff710000ff9e, - 0x1aff00001aff4, - 0x1aff50001affc, - 0x1affd0001afff, - 0x1b0000001b001, - 0x1b1200001b123, - 0x1b1550001b156, - 0x1b1640001b168, - ), -} -joining_types = { - 0x600: 85, - 0x601: 85, - 0x602: 85, - 0x603: 85, - 0x604: 85, - 0x605: 85, - 0x608: 85, - 0x60b: 85, - 0x620: 68, - 0x621: 85, - 0x622: 82, - 0x623: 82, - 0x624: 82, - 0x625: 82, - 0x626: 68, - 0x627: 82, - 0x628: 68, - 0x629: 82, - 0x62a: 68, - 0x62b: 68, - 0x62c: 68, - 0x62d: 68, - 0x62e: 68, - 0x62f: 82, - 0x630: 82, - 0x631: 82, - 0x632: 82, - 0x633: 68, - 0x634: 68, - 0x635: 68, - 0x636: 68, - 0x637: 68, - 0x638: 68, - 0x639: 68, - 0x63a: 68, - 0x63b: 68, - 0x63c: 68, - 0x63d: 68, - 0x63e: 68, - 0x63f: 68, - 0x640: 67, - 0x641: 68, - 0x642: 68, - 0x643: 68, - 0x644: 68, - 0x645: 68, - 0x646: 68, - 0x647: 68, - 0x648: 82, - 0x649: 68, - 0x64a: 68, - 0x66e: 68, - 0x66f: 68, - 0x671: 82, - 0x672: 82, - 0x673: 82, - 0x674: 85, - 0x675: 82, - 0x676: 82, - 0x677: 82, - 0x678: 68, - 0x679: 68, - 0x67a: 68, - 0x67b: 68, - 0x67c: 68, - 0x67d: 68, - 0x67e: 68, - 0x67f: 68, - 0x680: 68, - 0x681: 68, - 0x682: 68, - 0x683: 68, - 0x684: 68, - 0x685: 68, - 0x686: 68, - 0x687: 68, - 0x688: 82, - 0x689: 82, - 0x68a: 82, - 0x68b: 82, - 0x68c: 82, - 0x68d: 82, - 0x68e: 82, - 0x68f: 82, - 0x690: 82, - 0x691: 82, - 0x692: 82, - 0x693: 82, - 0x694: 82, - 0x695: 82, - 0x696: 82, - 0x697: 82, - 0x698: 82, - 0x699: 82, - 0x69a: 68, - 0x69b: 68, - 0x69c: 68, - 0x69d: 68, - 0x69e: 68, - 0x69f: 68, - 0x6a0: 68, - 0x6a1: 68, - 0x6a2: 68, - 0x6a3: 68, - 0x6a4: 68, - 0x6a5: 68, - 0x6a6: 68, - 0x6a7: 68, - 0x6a8: 68, - 0x6a9: 68, - 0x6aa: 68, - 0x6ab: 68, - 0x6ac: 68, - 0x6ad: 68, - 0x6ae: 68, - 0x6af: 68, - 0x6b0: 68, - 0x6b1: 68, - 0x6b2: 68, - 0x6b3: 68, - 0x6b4: 68, - 0x6b5: 68, - 0x6b6: 68, - 0x6b7: 68, - 0x6b8: 68, - 0x6b9: 68, - 0x6ba: 68, - 0x6bb: 68, - 0x6bc: 68, - 0x6bd: 68, - 0x6be: 68, - 0x6bf: 68, - 0x6c0: 82, - 0x6c1: 68, - 0x6c2: 68, - 0x6c3: 82, - 0x6c4: 82, - 0x6c5: 82, - 0x6c6: 82, - 0x6c7: 82, - 0x6c8: 82, - 0x6c9: 82, - 0x6ca: 82, - 0x6cb: 82, - 0x6cc: 68, - 0x6cd: 82, - 0x6ce: 68, - 0x6cf: 82, - 0x6d0: 68, - 0x6d1: 68, - 0x6d2: 82, - 0x6d3: 82, - 0x6d5: 82, - 0x6dd: 85, - 0x6ee: 82, - 0x6ef: 82, - 0x6fa: 68, - 0x6fb: 68, - 0x6fc: 68, - 0x6ff: 68, - 0x70f: 84, - 0x710: 82, - 0x712: 68, - 0x713: 68, - 0x714: 68, - 0x715: 82, - 0x716: 82, - 0x717: 82, - 0x718: 82, - 0x719: 82, - 0x71a: 68, - 0x71b: 68, - 0x71c: 68, - 0x71d: 68, - 0x71e: 82, - 0x71f: 68, - 0x720: 68, - 0x721: 68, - 0x722: 68, - 0x723: 68, - 0x724: 68, - 0x725: 68, - 0x726: 68, - 0x727: 68, - 0x728: 82, - 0x729: 68, - 0x72a: 82, - 0x72b: 68, - 0x72c: 82, - 0x72d: 68, - 0x72e: 68, - 0x72f: 82, - 0x74d: 82, - 0x74e: 68, - 0x74f: 68, - 0x750: 68, - 0x751: 68, - 0x752: 68, - 0x753: 68, - 0x754: 68, - 0x755: 68, - 0x756: 68, - 0x757: 68, - 0x758: 68, - 0x759: 82, - 0x75a: 82, - 0x75b: 82, - 0x75c: 68, - 0x75d: 68, - 0x75e: 68, - 0x75f: 68, - 0x760: 68, - 0x761: 68, - 0x762: 68, - 0x763: 68, - 0x764: 68, - 0x765: 68, - 0x766: 68, - 0x767: 68, - 0x768: 68, - 0x769: 68, - 0x76a: 68, - 0x76b: 82, - 0x76c: 82, - 0x76d: 68, - 0x76e: 68, - 0x76f: 68, - 0x770: 68, - 0x771: 82, - 0x772: 68, - 0x773: 82, - 0x774: 82, - 0x775: 68, - 0x776: 68, - 0x777: 68, - 0x778: 82, - 0x779: 82, - 0x77a: 68, - 0x77b: 68, - 0x77c: 68, - 0x77d: 68, - 0x77e: 68, - 0x77f: 68, - 0x7ca: 68, - 0x7cb: 68, - 0x7cc: 68, - 0x7cd: 68, - 0x7ce: 68, - 0x7cf: 68, - 0x7d0: 68, - 0x7d1: 68, - 0x7d2: 68, - 0x7d3: 68, - 0x7d4: 68, - 0x7d5: 68, - 0x7d6: 68, - 0x7d7: 68, - 0x7d8: 68, - 0x7d9: 68, - 0x7da: 68, - 0x7db: 68, - 0x7dc: 68, - 0x7dd: 68, - 0x7de: 68, - 0x7df: 68, - 0x7e0: 68, - 0x7e1: 68, - 0x7e2: 68, - 0x7e3: 68, - 0x7e4: 68, - 0x7e5: 68, - 0x7e6: 68, - 0x7e7: 68, - 0x7e8: 68, - 0x7e9: 68, - 0x7ea: 68, - 0x7fa: 67, - 0x840: 82, - 0x841: 68, - 0x842: 68, - 0x843: 68, - 0x844: 68, - 0x845: 68, - 0x846: 82, - 0x847: 82, - 0x848: 68, - 0x849: 82, - 0x84a: 68, - 0x84b: 68, - 0x84c: 68, - 0x84d: 68, - 0x84e: 68, - 0x84f: 68, - 0x850: 68, - 0x851: 68, - 0x852: 68, - 0x853: 68, - 0x854: 82, - 0x855: 68, - 0x856: 82, - 0x857: 82, - 0x858: 82, - 0x860: 68, - 0x861: 85, - 0x862: 68, - 0x863: 68, - 0x864: 68, - 0x865: 68, - 0x866: 85, - 0x867: 82, - 0x868: 68, - 0x869: 82, - 0x86a: 82, - 0x870: 82, - 0x871: 82, - 0x872: 82, - 0x873: 82, - 0x874: 82, - 0x875: 82, - 0x876: 82, - 0x877: 82, - 0x878: 82, - 0x879: 82, - 0x87a: 82, - 0x87b: 82, - 0x87c: 82, - 0x87d: 82, - 0x87e: 82, - 0x87f: 82, - 0x880: 82, - 0x881: 82, - 0x882: 82, - 0x883: 67, - 0x884: 67, - 0x885: 67, - 0x886: 68, - 0x887: 85, - 0x888: 85, - 0x889: 68, - 0x88a: 68, - 0x88b: 68, - 0x88c: 68, - 0x88d: 68, - 0x88e: 82, - 0x890: 85, - 0x891: 85, - 0x8a0: 68, - 0x8a1: 68, - 0x8a2: 68, - 0x8a3: 68, - 0x8a4: 68, - 0x8a5: 68, - 0x8a6: 68, - 0x8a7: 68, - 0x8a8: 68, - 0x8a9: 68, - 0x8aa: 82, - 0x8ab: 82, - 0x8ac: 82, - 0x8ad: 85, - 0x8ae: 82, - 0x8af: 68, - 0x8b0: 68, - 0x8b1: 82, - 0x8b2: 82, - 0x8b3: 68, - 0x8b4: 68, - 0x8b5: 68, - 0x8b6: 68, - 0x8b7: 68, - 0x8b8: 68, - 0x8b9: 82, - 0x8ba: 68, - 0x8bb: 68, - 0x8bc: 68, - 0x8bd: 68, - 0x8be: 68, - 0x8bf: 68, - 0x8c0: 68, - 0x8c1: 68, - 0x8c2: 68, - 0x8c3: 68, - 0x8c4: 68, - 0x8c5: 68, - 0x8c6: 68, - 0x8c7: 68, - 0x8c8: 68, - 0x8e2: 85, - 0x1806: 85, - 0x1807: 68, - 0x180a: 67, - 0x180e: 85, - 0x1820: 68, - 0x1821: 68, - 0x1822: 68, - 0x1823: 68, - 0x1824: 68, - 0x1825: 68, - 0x1826: 68, - 0x1827: 68, - 0x1828: 68, - 0x1829: 68, - 0x182a: 68, - 0x182b: 68, - 0x182c: 68, - 0x182d: 68, - 0x182e: 68, - 0x182f: 68, - 0x1830: 68, - 0x1831: 68, - 0x1832: 68, - 0x1833: 68, - 0x1834: 68, - 0x1835: 68, - 0x1836: 68, - 0x1837: 68, - 0x1838: 68, - 0x1839: 68, - 0x183a: 68, - 0x183b: 68, - 0x183c: 68, - 0x183d: 68, - 0x183e: 68, - 0x183f: 68, - 0x1840: 68, - 0x1841: 68, - 0x1842: 68, - 0x1843: 68, - 0x1844: 68, - 0x1845: 68, - 0x1846: 68, - 0x1847: 68, - 0x1848: 68, - 0x1849: 68, - 0x184a: 68, - 0x184b: 68, - 0x184c: 68, - 0x184d: 68, - 0x184e: 68, - 0x184f: 68, - 0x1850: 68, - 0x1851: 68, - 0x1852: 68, - 0x1853: 68, - 0x1854: 68, - 0x1855: 68, - 0x1856: 68, - 0x1857: 68, - 0x1858: 68, - 0x1859: 68, - 0x185a: 68, - 0x185b: 68, - 0x185c: 68, - 0x185d: 68, - 0x185e: 68, - 0x185f: 68, - 0x1860: 68, - 0x1861: 68, - 0x1862: 68, - 0x1863: 68, - 0x1864: 68, - 0x1865: 68, - 0x1866: 68, - 0x1867: 68, - 0x1868: 68, - 0x1869: 68, - 0x186a: 68, - 0x186b: 68, - 0x186c: 68, - 0x186d: 68, - 0x186e: 68, - 0x186f: 68, - 0x1870: 68, - 0x1871: 68, - 0x1872: 68, - 0x1873: 68, - 0x1874: 68, - 0x1875: 68, - 0x1876: 68, - 0x1877: 68, - 0x1878: 68, - 0x1880: 85, - 0x1881: 85, - 0x1882: 85, - 0x1883: 85, - 0x1884: 85, - 0x1885: 84, - 0x1886: 84, - 0x1887: 68, - 0x1888: 68, - 0x1889: 68, - 0x188a: 68, - 0x188b: 68, - 0x188c: 68, - 0x188d: 68, - 0x188e: 68, - 0x188f: 68, - 0x1890: 68, - 0x1891: 68, - 0x1892: 68, - 0x1893: 68, - 0x1894: 68, - 0x1895: 68, - 0x1896: 68, - 0x1897: 68, - 0x1898: 68, - 0x1899: 68, - 0x189a: 68, - 0x189b: 68, - 0x189c: 68, - 0x189d: 68, - 0x189e: 68, - 0x189f: 68, - 0x18a0: 68, - 0x18a1: 68, - 0x18a2: 68, - 0x18a3: 68, - 0x18a4: 68, - 0x18a5: 68, - 0x18a6: 68, - 0x18a7: 68, - 0x18a8: 68, - 0x18aa: 68, - 0x200c: 85, - 0x200d: 67, - 0x202f: 85, - 0x2066: 85, - 0x2067: 85, - 0x2068: 85, - 0x2069: 85, - 0xa840: 68, - 0xa841: 68, - 0xa842: 68, - 0xa843: 68, - 0xa844: 68, - 0xa845: 68, - 0xa846: 68, - 0xa847: 68, - 0xa848: 68, - 0xa849: 68, - 0xa84a: 68, - 0xa84b: 68, - 0xa84c: 68, - 0xa84d: 68, - 0xa84e: 68, - 0xa84f: 68, - 0xa850: 68, - 0xa851: 68, - 0xa852: 68, - 0xa853: 68, - 0xa854: 68, - 0xa855: 68, - 0xa856: 68, - 0xa857: 68, - 0xa858: 68, - 0xa859: 68, - 0xa85a: 68, - 0xa85b: 68, - 0xa85c: 68, - 0xa85d: 68, - 0xa85e: 68, - 0xa85f: 68, - 0xa860: 68, - 0xa861: 68, - 0xa862: 68, - 0xa863: 68, - 0xa864: 68, - 0xa865: 68, - 0xa866: 68, - 0xa867: 68, - 0xa868: 68, - 0xa869: 68, - 0xa86a: 68, - 0xa86b: 68, - 0xa86c: 68, - 0xa86d: 68, - 0xa86e: 68, - 0xa86f: 68, - 0xa870: 68, - 0xa871: 68, - 0xa872: 76, - 0xa873: 85, - 0x10ac0: 68, - 0x10ac1: 68, - 0x10ac2: 68, - 0x10ac3: 68, - 0x10ac4: 68, - 0x10ac5: 82, - 0x10ac6: 85, - 0x10ac7: 82, - 0x10ac8: 85, - 0x10ac9: 82, - 0x10aca: 82, - 0x10acb: 85, - 0x10acc: 85, - 0x10acd: 76, - 0x10ace: 82, - 0x10acf: 82, - 0x10ad0: 82, - 0x10ad1: 82, - 0x10ad2: 82, - 0x10ad3: 68, - 0x10ad4: 68, - 0x10ad5: 68, - 0x10ad6: 68, - 0x10ad7: 76, - 0x10ad8: 68, - 0x10ad9: 68, - 0x10ada: 68, - 0x10adb: 68, - 0x10adc: 68, - 0x10add: 82, - 0x10ade: 68, - 0x10adf: 68, - 0x10ae0: 68, - 0x10ae1: 82, - 0x10ae2: 85, - 0x10ae3: 85, - 0x10ae4: 82, - 0x10aeb: 68, - 0x10aec: 68, - 0x10aed: 68, - 0x10aee: 68, - 0x10aef: 82, - 0x10b80: 68, - 0x10b81: 82, - 0x10b82: 68, - 0x10b83: 82, - 0x10b84: 82, - 0x10b85: 82, - 0x10b86: 68, - 0x10b87: 68, - 0x10b88: 68, - 0x10b89: 82, - 0x10b8a: 68, - 0x10b8b: 68, - 0x10b8c: 82, - 0x10b8d: 68, - 0x10b8e: 82, - 0x10b8f: 82, - 0x10b90: 68, - 0x10b91: 82, - 0x10ba9: 82, - 0x10baa: 82, - 0x10bab: 82, - 0x10bac: 82, - 0x10bad: 68, - 0x10bae: 68, - 0x10baf: 85, - 0x10d00: 76, - 0x10d01: 68, - 0x10d02: 68, - 0x10d03: 68, - 0x10d04: 68, - 0x10d05: 68, - 0x10d06: 68, - 0x10d07: 68, - 0x10d08: 68, - 0x10d09: 68, - 0x10d0a: 68, - 0x10d0b: 68, - 0x10d0c: 68, - 0x10d0d: 68, - 0x10d0e: 68, - 0x10d0f: 68, - 0x10d10: 68, - 0x10d11: 68, - 0x10d12: 68, - 0x10d13: 68, - 0x10d14: 68, - 0x10d15: 68, - 0x10d16: 68, - 0x10d17: 68, - 0x10d18: 68, - 0x10d19: 68, - 0x10d1a: 68, - 0x10d1b: 68, - 0x10d1c: 68, - 0x10d1d: 68, - 0x10d1e: 68, - 0x10d1f: 68, - 0x10d20: 68, - 0x10d21: 68, - 0x10d22: 82, - 0x10d23: 68, - 0x10f30: 68, - 0x10f31: 68, - 0x10f32: 68, - 0x10f33: 82, - 0x10f34: 68, - 0x10f35: 68, - 0x10f36: 68, - 0x10f37: 68, - 0x10f38: 68, - 0x10f39: 68, - 0x10f3a: 68, - 0x10f3b: 68, - 0x10f3c: 68, - 0x10f3d: 68, - 0x10f3e: 68, - 0x10f3f: 68, - 0x10f40: 68, - 0x10f41: 68, - 0x10f42: 68, - 0x10f43: 68, - 0x10f44: 68, - 0x10f45: 85, - 0x10f51: 68, - 0x10f52: 68, - 0x10f53: 68, - 0x10f54: 82, - 0x10f70: 68, - 0x10f71: 68, - 0x10f72: 68, - 0x10f73: 68, - 0x10f74: 82, - 0x10f75: 82, - 0x10f76: 68, - 0x10f77: 68, - 0x10f78: 68, - 0x10f79: 68, - 0x10f7a: 68, - 0x10f7b: 68, - 0x10f7c: 68, - 0x10f7d: 68, - 0x10f7e: 68, - 0x10f7f: 68, - 0x10f80: 68, - 0x10f81: 68, - 0x10fb0: 68, - 0x10fb1: 85, - 0x10fb2: 68, - 0x10fb3: 68, - 0x10fb4: 82, - 0x10fb5: 82, - 0x10fb6: 82, - 0x10fb7: 85, - 0x10fb8: 68, - 0x10fb9: 82, - 0x10fba: 82, - 0x10fbb: 68, - 0x10fbc: 68, - 0x10fbd: 82, - 0x10fbe: 68, - 0x10fbf: 68, - 0x10fc0: 85, - 0x10fc1: 68, - 0x10fc2: 82, - 0x10fc3: 82, - 0x10fc4: 68, - 0x10fc5: 85, - 0x10fc6: 85, - 0x10fc7: 85, - 0x10fc8: 85, - 0x10fc9: 82, - 0x10fca: 68, - 0x10fcb: 76, - 0x110bd: 85, - 0x110cd: 85, - 0x1e900: 68, - 0x1e901: 68, - 0x1e902: 68, - 0x1e903: 68, - 0x1e904: 68, - 0x1e905: 68, - 0x1e906: 68, - 0x1e907: 68, - 0x1e908: 68, - 0x1e909: 68, - 0x1e90a: 68, - 0x1e90b: 68, - 0x1e90c: 68, - 0x1e90d: 68, - 0x1e90e: 68, - 0x1e90f: 68, - 0x1e910: 68, - 0x1e911: 68, - 0x1e912: 68, - 0x1e913: 68, - 0x1e914: 68, - 0x1e915: 68, - 0x1e916: 68, - 0x1e917: 68, - 0x1e918: 68, - 0x1e919: 68, - 0x1e91a: 68, - 0x1e91b: 68, - 0x1e91c: 68, - 0x1e91d: 68, - 0x1e91e: 68, - 0x1e91f: 68, - 0x1e920: 68, - 0x1e921: 68, - 0x1e922: 68, - 0x1e923: 68, - 0x1e924: 68, - 0x1e925: 68, - 0x1e926: 68, - 0x1e927: 68, - 0x1e928: 68, - 0x1e929: 68, - 0x1e92a: 68, - 0x1e92b: 68, - 0x1e92c: 68, - 0x1e92d: 68, - 0x1e92e: 68, - 0x1e92f: 68, - 0x1e930: 68, - 0x1e931: 68, - 0x1e932: 68, - 0x1e933: 68, - 0x1e934: 68, - 0x1e935: 68, - 0x1e936: 68, - 0x1e937: 68, - 0x1e938: 68, - 0x1e939: 68, - 0x1e93a: 68, - 0x1e93b: 68, - 0x1e93c: 68, - 0x1e93d: 68, - 0x1e93e: 68, - 0x1e93f: 68, - 0x1e940: 68, - 0x1e941: 68, - 0x1e942: 68, - 0x1e943: 68, - 0x1e94b: 84, -} -codepoint_classes = { - 'PVALID': ( - 0x2d0000002e, - 0x300000003a, - 0x610000007b, - 0xdf000000f7, - 0xf800000100, - 0x10100000102, - 0x10300000104, - 0x10500000106, - 0x10700000108, - 0x1090000010a, - 0x10b0000010c, - 0x10d0000010e, - 0x10f00000110, - 0x11100000112, - 0x11300000114, - 0x11500000116, - 0x11700000118, - 0x1190000011a, - 0x11b0000011c, - 0x11d0000011e, - 0x11f00000120, - 0x12100000122, - 0x12300000124, - 0x12500000126, - 0x12700000128, - 0x1290000012a, - 0x12b0000012c, - 0x12d0000012e, - 0x12f00000130, - 0x13100000132, - 0x13500000136, - 0x13700000139, - 0x13a0000013b, - 0x13c0000013d, - 0x13e0000013f, - 0x14200000143, - 0x14400000145, - 0x14600000147, - 0x14800000149, - 0x14b0000014c, - 0x14d0000014e, - 0x14f00000150, - 0x15100000152, - 0x15300000154, - 0x15500000156, - 0x15700000158, - 0x1590000015a, - 0x15b0000015c, - 0x15d0000015e, - 0x15f00000160, - 0x16100000162, - 0x16300000164, - 0x16500000166, - 0x16700000168, - 0x1690000016a, - 0x16b0000016c, - 0x16d0000016e, - 0x16f00000170, - 0x17100000172, - 0x17300000174, - 0x17500000176, - 0x17700000178, - 0x17a0000017b, - 0x17c0000017d, - 0x17e0000017f, - 0x18000000181, - 0x18300000184, - 0x18500000186, - 0x18800000189, - 0x18c0000018e, - 0x19200000193, - 0x19500000196, - 0x1990000019c, - 0x19e0000019f, - 0x1a1000001a2, - 0x1a3000001a4, - 0x1a5000001a6, - 0x1a8000001a9, - 0x1aa000001ac, - 0x1ad000001ae, - 0x1b0000001b1, - 0x1b4000001b5, - 0x1b6000001b7, - 0x1b9000001bc, - 0x1bd000001c4, - 0x1ce000001cf, - 0x1d0000001d1, - 0x1d2000001d3, - 0x1d4000001d5, - 0x1d6000001d7, - 0x1d8000001d9, - 0x1da000001db, - 0x1dc000001de, - 0x1df000001e0, - 0x1e1000001e2, - 0x1e3000001e4, - 0x1e5000001e6, - 0x1e7000001e8, - 0x1e9000001ea, - 0x1eb000001ec, - 0x1ed000001ee, - 0x1ef000001f1, - 0x1f5000001f6, - 0x1f9000001fa, - 0x1fb000001fc, - 0x1fd000001fe, - 0x1ff00000200, - 0x20100000202, - 0x20300000204, - 0x20500000206, - 0x20700000208, - 0x2090000020a, - 0x20b0000020c, - 0x20d0000020e, - 0x20f00000210, - 0x21100000212, - 0x21300000214, - 0x21500000216, - 0x21700000218, - 0x2190000021a, - 0x21b0000021c, - 0x21d0000021e, - 0x21f00000220, - 0x22100000222, - 0x22300000224, - 0x22500000226, - 0x22700000228, - 0x2290000022a, - 0x22b0000022c, - 0x22d0000022e, - 0x22f00000230, - 0x23100000232, - 0x2330000023a, - 0x23c0000023d, - 0x23f00000241, - 0x24200000243, - 0x24700000248, - 0x2490000024a, - 0x24b0000024c, - 0x24d0000024e, - 0x24f000002b0, - 0x2b9000002c2, - 0x2c6000002d2, - 0x2ec000002ed, - 0x2ee000002ef, - 0x30000000340, - 0x34200000343, - 0x3460000034f, - 0x35000000370, - 0x37100000372, - 0x37300000374, - 0x37700000378, - 0x37b0000037e, - 0x39000000391, - 0x3ac000003cf, - 0x3d7000003d8, - 0x3d9000003da, - 0x3db000003dc, - 0x3dd000003de, - 0x3df000003e0, - 0x3e1000003e2, - 0x3e3000003e4, - 0x3e5000003e6, - 0x3e7000003e8, - 0x3e9000003ea, - 0x3eb000003ec, - 0x3ed000003ee, - 0x3ef000003f0, - 0x3f3000003f4, - 0x3f8000003f9, - 0x3fb000003fd, - 0x43000000460, - 0x46100000462, - 0x46300000464, - 0x46500000466, - 0x46700000468, - 0x4690000046a, - 0x46b0000046c, - 0x46d0000046e, - 0x46f00000470, - 0x47100000472, - 0x47300000474, - 0x47500000476, - 0x47700000478, - 0x4790000047a, - 0x47b0000047c, - 0x47d0000047e, - 0x47f00000480, - 0x48100000482, - 0x48300000488, - 0x48b0000048c, - 0x48d0000048e, - 0x48f00000490, - 0x49100000492, - 0x49300000494, - 0x49500000496, - 0x49700000498, - 0x4990000049a, - 0x49b0000049c, - 0x49d0000049e, - 0x49f000004a0, - 0x4a1000004a2, - 0x4a3000004a4, - 0x4a5000004a6, - 0x4a7000004a8, - 0x4a9000004aa, - 0x4ab000004ac, - 0x4ad000004ae, - 0x4af000004b0, - 0x4b1000004b2, - 0x4b3000004b4, - 0x4b5000004b6, - 0x4b7000004b8, - 0x4b9000004ba, - 0x4bb000004bc, - 0x4bd000004be, - 0x4bf000004c0, - 0x4c2000004c3, - 0x4c4000004c5, - 0x4c6000004c7, - 0x4c8000004c9, - 0x4ca000004cb, - 0x4cc000004cd, - 0x4ce000004d0, - 0x4d1000004d2, - 0x4d3000004d4, - 0x4d5000004d6, - 0x4d7000004d8, - 0x4d9000004da, - 0x4db000004dc, - 0x4dd000004de, - 0x4df000004e0, - 0x4e1000004e2, - 0x4e3000004e4, - 0x4e5000004e6, - 0x4e7000004e8, - 0x4e9000004ea, - 0x4eb000004ec, - 0x4ed000004ee, - 0x4ef000004f0, - 0x4f1000004f2, - 0x4f3000004f4, - 0x4f5000004f6, - 0x4f7000004f8, - 0x4f9000004fa, - 0x4fb000004fc, - 0x4fd000004fe, - 0x4ff00000500, - 0x50100000502, - 0x50300000504, - 0x50500000506, - 0x50700000508, - 0x5090000050a, - 0x50b0000050c, - 0x50d0000050e, - 0x50f00000510, - 0x51100000512, - 0x51300000514, - 0x51500000516, - 0x51700000518, - 0x5190000051a, - 0x51b0000051c, - 0x51d0000051e, - 0x51f00000520, - 0x52100000522, - 0x52300000524, - 0x52500000526, - 0x52700000528, - 0x5290000052a, - 0x52b0000052c, - 0x52d0000052e, - 0x52f00000530, - 0x5590000055a, - 0x56000000587, - 0x58800000589, - 0x591000005be, - 0x5bf000005c0, - 0x5c1000005c3, - 0x5c4000005c6, - 0x5c7000005c8, - 0x5d0000005eb, - 0x5ef000005f3, - 0x6100000061b, - 0x62000000640, - 0x64100000660, - 0x66e00000675, - 0x679000006d4, - 0x6d5000006dd, - 0x6df000006e9, - 0x6ea000006f0, - 0x6fa00000700, - 0x7100000074b, - 0x74d000007b2, - 0x7c0000007f6, - 0x7fd000007fe, - 0x8000000082e, - 0x8400000085c, - 0x8600000086b, - 0x87000000888, - 0x8890000088f, - 0x898000008e2, - 0x8e300000958, - 0x96000000964, - 0x96600000970, - 0x97100000984, - 0x9850000098d, - 0x98f00000991, - 0x993000009a9, - 0x9aa000009b1, - 0x9b2000009b3, - 0x9b6000009ba, - 0x9bc000009c5, - 0x9c7000009c9, - 0x9cb000009cf, - 0x9d7000009d8, - 0x9e0000009e4, - 0x9e6000009f2, - 0x9fc000009fd, - 0x9fe000009ff, - 0xa0100000a04, - 0xa0500000a0b, - 0xa0f00000a11, - 0xa1300000a29, - 0xa2a00000a31, - 0xa3200000a33, - 0xa3500000a36, - 0xa3800000a3a, - 0xa3c00000a3d, - 0xa3e00000a43, - 0xa4700000a49, - 0xa4b00000a4e, - 0xa5100000a52, - 0xa5c00000a5d, - 0xa6600000a76, - 0xa8100000a84, - 0xa8500000a8e, - 0xa8f00000a92, - 0xa9300000aa9, - 0xaaa00000ab1, - 0xab200000ab4, - 0xab500000aba, - 0xabc00000ac6, - 0xac700000aca, - 0xacb00000ace, - 0xad000000ad1, - 0xae000000ae4, - 0xae600000af0, - 0xaf900000b00, - 0xb0100000b04, - 0xb0500000b0d, - 0xb0f00000b11, - 0xb1300000b29, - 0xb2a00000b31, - 0xb3200000b34, - 0xb3500000b3a, - 0xb3c00000b45, - 0xb4700000b49, - 0xb4b00000b4e, - 0xb5500000b58, - 0xb5f00000b64, - 0xb6600000b70, - 0xb7100000b72, - 0xb8200000b84, - 0xb8500000b8b, - 0xb8e00000b91, - 0xb9200000b96, - 0xb9900000b9b, - 0xb9c00000b9d, - 0xb9e00000ba0, - 0xba300000ba5, - 0xba800000bab, - 0xbae00000bba, - 0xbbe00000bc3, - 0xbc600000bc9, - 0xbca00000bce, - 0xbd000000bd1, - 0xbd700000bd8, - 0xbe600000bf0, - 0xc0000000c0d, - 0xc0e00000c11, - 0xc1200000c29, - 0xc2a00000c3a, - 0xc3c00000c45, - 0xc4600000c49, - 0xc4a00000c4e, - 0xc5500000c57, - 0xc5800000c5b, - 0xc5d00000c5e, - 0xc6000000c64, - 0xc6600000c70, - 0xc8000000c84, - 0xc8500000c8d, - 0xc8e00000c91, - 0xc9200000ca9, - 0xcaa00000cb4, - 0xcb500000cba, - 0xcbc00000cc5, - 0xcc600000cc9, - 0xcca00000cce, - 0xcd500000cd7, - 0xcdd00000cdf, - 0xce000000ce4, - 0xce600000cf0, - 0xcf100000cf4, - 0xd0000000d0d, - 0xd0e00000d11, - 0xd1200000d45, - 0xd4600000d49, - 0xd4a00000d4f, - 0xd5400000d58, - 0xd5f00000d64, - 0xd6600000d70, - 0xd7a00000d80, - 0xd8100000d84, - 0xd8500000d97, - 0xd9a00000db2, - 0xdb300000dbc, - 0xdbd00000dbe, - 0xdc000000dc7, - 0xdca00000dcb, - 0xdcf00000dd5, - 0xdd600000dd7, - 0xdd800000de0, - 0xde600000df0, - 0xdf200000df4, - 0xe0100000e33, - 0xe3400000e3b, - 0xe4000000e4f, - 0xe5000000e5a, - 0xe8100000e83, - 0xe8400000e85, - 0xe8600000e8b, - 0xe8c00000ea4, - 0xea500000ea6, - 0xea700000eb3, - 0xeb400000ebe, - 0xec000000ec5, - 0xec600000ec7, - 0xec800000ecf, - 0xed000000eda, - 0xede00000ee0, - 0xf0000000f01, - 0xf0b00000f0c, - 0xf1800000f1a, - 0xf2000000f2a, - 0xf3500000f36, - 0xf3700000f38, - 0xf3900000f3a, - 0xf3e00000f43, - 0xf4400000f48, - 0xf4900000f4d, - 0xf4e00000f52, - 0xf5300000f57, - 0xf5800000f5c, - 0xf5d00000f69, - 0xf6a00000f6d, - 0xf7100000f73, - 0xf7400000f75, - 0xf7a00000f81, - 0xf8200000f85, - 0xf8600000f93, - 0xf9400000f98, - 0xf9900000f9d, - 0xf9e00000fa2, - 0xfa300000fa7, - 0xfa800000fac, - 0xfad00000fb9, - 0xfba00000fbd, - 0xfc600000fc7, - 0x10000000104a, - 0x10500000109e, - 0x10d0000010fb, - 0x10fd00001100, - 0x120000001249, - 0x124a0000124e, - 0x125000001257, - 0x125800001259, - 0x125a0000125e, - 0x126000001289, - 0x128a0000128e, - 0x1290000012b1, - 0x12b2000012b6, - 0x12b8000012bf, - 0x12c0000012c1, - 0x12c2000012c6, - 0x12c8000012d7, - 0x12d800001311, - 0x131200001316, - 0x13180000135b, - 0x135d00001360, - 0x138000001390, - 0x13a0000013f6, - 0x14010000166d, - 0x166f00001680, - 0x16810000169b, - 0x16a0000016eb, - 0x16f1000016f9, - 0x170000001716, - 0x171f00001735, - 0x174000001754, - 0x17600000176d, - 0x176e00001771, - 0x177200001774, - 0x1780000017b4, - 0x17b6000017d4, - 0x17d7000017d8, - 0x17dc000017de, - 0x17e0000017ea, - 0x18100000181a, - 0x182000001879, - 0x1880000018ab, - 0x18b0000018f6, - 0x19000000191f, - 0x19200000192c, - 0x19300000193c, - 0x19460000196e, - 0x197000001975, - 0x1980000019ac, - 0x19b0000019ca, - 0x19d0000019da, - 0x1a0000001a1c, - 0x1a2000001a5f, - 0x1a6000001a7d, - 0x1a7f00001a8a, - 0x1a9000001a9a, - 0x1aa700001aa8, - 0x1ab000001abe, - 0x1abf00001acf, - 0x1b0000001b4d, - 0x1b5000001b5a, - 0x1b6b00001b74, - 0x1b8000001bf4, - 0x1c0000001c38, - 0x1c4000001c4a, - 0x1c4d00001c7e, - 0x1cd000001cd3, - 0x1cd400001cfb, - 0x1d0000001d2c, - 0x1d2f00001d30, - 0x1d3b00001d3c, - 0x1d4e00001d4f, - 0x1d6b00001d78, - 0x1d7900001d9b, - 0x1dc000001e00, - 0x1e0100001e02, - 0x1e0300001e04, - 0x1e0500001e06, - 0x1e0700001e08, - 0x1e0900001e0a, - 0x1e0b00001e0c, - 0x1e0d00001e0e, - 0x1e0f00001e10, - 0x1e1100001e12, - 0x1e1300001e14, - 0x1e1500001e16, - 0x1e1700001e18, - 0x1e1900001e1a, - 0x1e1b00001e1c, - 0x1e1d00001e1e, - 0x1e1f00001e20, - 0x1e2100001e22, - 0x1e2300001e24, - 0x1e2500001e26, - 0x1e2700001e28, - 0x1e2900001e2a, - 0x1e2b00001e2c, - 0x1e2d00001e2e, - 0x1e2f00001e30, - 0x1e3100001e32, - 0x1e3300001e34, - 0x1e3500001e36, - 0x1e3700001e38, - 0x1e3900001e3a, - 0x1e3b00001e3c, - 0x1e3d00001e3e, - 0x1e3f00001e40, - 0x1e4100001e42, - 0x1e4300001e44, - 0x1e4500001e46, - 0x1e4700001e48, - 0x1e4900001e4a, - 0x1e4b00001e4c, - 0x1e4d00001e4e, - 0x1e4f00001e50, - 0x1e5100001e52, - 0x1e5300001e54, - 0x1e5500001e56, - 0x1e5700001e58, - 0x1e5900001e5a, - 0x1e5b00001e5c, - 0x1e5d00001e5e, - 0x1e5f00001e60, - 0x1e6100001e62, - 0x1e6300001e64, - 0x1e6500001e66, - 0x1e6700001e68, - 0x1e6900001e6a, - 0x1e6b00001e6c, - 0x1e6d00001e6e, - 0x1e6f00001e70, - 0x1e7100001e72, - 0x1e7300001e74, - 0x1e7500001e76, - 0x1e7700001e78, - 0x1e7900001e7a, - 0x1e7b00001e7c, - 0x1e7d00001e7e, - 0x1e7f00001e80, - 0x1e8100001e82, - 0x1e8300001e84, - 0x1e8500001e86, - 0x1e8700001e88, - 0x1e8900001e8a, - 0x1e8b00001e8c, - 0x1e8d00001e8e, - 0x1e8f00001e90, - 0x1e9100001e92, - 0x1e9300001e94, - 0x1e9500001e9a, - 0x1e9c00001e9e, - 0x1e9f00001ea0, - 0x1ea100001ea2, - 0x1ea300001ea4, - 0x1ea500001ea6, - 0x1ea700001ea8, - 0x1ea900001eaa, - 0x1eab00001eac, - 0x1ead00001eae, - 0x1eaf00001eb0, - 0x1eb100001eb2, - 0x1eb300001eb4, - 0x1eb500001eb6, - 0x1eb700001eb8, - 0x1eb900001eba, - 0x1ebb00001ebc, - 0x1ebd00001ebe, - 0x1ebf00001ec0, - 0x1ec100001ec2, - 0x1ec300001ec4, - 0x1ec500001ec6, - 0x1ec700001ec8, - 0x1ec900001eca, - 0x1ecb00001ecc, - 0x1ecd00001ece, - 0x1ecf00001ed0, - 0x1ed100001ed2, - 0x1ed300001ed4, - 0x1ed500001ed6, - 0x1ed700001ed8, - 0x1ed900001eda, - 0x1edb00001edc, - 0x1edd00001ede, - 0x1edf00001ee0, - 0x1ee100001ee2, - 0x1ee300001ee4, - 0x1ee500001ee6, - 0x1ee700001ee8, - 0x1ee900001eea, - 0x1eeb00001eec, - 0x1eed00001eee, - 0x1eef00001ef0, - 0x1ef100001ef2, - 0x1ef300001ef4, - 0x1ef500001ef6, - 0x1ef700001ef8, - 0x1ef900001efa, - 0x1efb00001efc, - 0x1efd00001efe, - 0x1eff00001f08, - 0x1f1000001f16, - 0x1f2000001f28, - 0x1f3000001f38, - 0x1f4000001f46, - 0x1f5000001f58, - 0x1f6000001f68, - 0x1f7000001f71, - 0x1f7200001f73, - 0x1f7400001f75, - 0x1f7600001f77, - 0x1f7800001f79, - 0x1f7a00001f7b, - 0x1f7c00001f7d, - 0x1fb000001fb2, - 0x1fb600001fb7, - 0x1fc600001fc7, - 0x1fd000001fd3, - 0x1fd600001fd8, - 0x1fe000001fe3, - 0x1fe400001fe8, - 0x1ff600001ff7, - 0x214e0000214f, - 0x218400002185, - 0x2c3000002c60, - 0x2c6100002c62, - 0x2c6500002c67, - 0x2c6800002c69, - 0x2c6a00002c6b, - 0x2c6c00002c6d, - 0x2c7100002c72, - 0x2c7300002c75, - 0x2c7600002c7c, - 0x2c8100002c82, - 0x2c8300002c84, - 0x2c8500002c86, - 0x2c8700002c88, - 0x2c8900002c8a, - 0x2c8b00002c8c, - 0x2c8d00002c8e, - 0x2c8f00002c90, - 0x2c9100002c92, - 0x2c9300002c94, - 0x2c9500002c96, - 0x2c9700002c98, - 0x2c9900002c9a, - 0x2c9b00002c9c, - 0x2c9d00002c9e, - 0x2c9f00002ca0, - 0x2ca100002ca2, - 0x2ca300002ca4, - 0x2ca500002ca6, - 0x2ca700002ca8, - 0x2ca900002caa, - 0x2cab00002cac, - 0x2cad00002cae, - 0x2caf00002cb0, - 0x2cb100002cb2, - 0x2cb300002cb4, - 0x2cb500002cb6, - 0x2cb700002cb8, - 0x2cb900002cba, - 0x2cbb00002cbc, - 0x2cbd00002cbe, - 0x2cbf00002cc0, - 0x2cc100002cc2, - 0x2cc300002cc4, - 0x2cc500002cc6, - 0x2cc700002cc8, - 0x2cc900002cca, - 0x2ccb00002ccc, - 0x2ccd00002cce, - 0x2ccf00002cd0, - 0x2cd100002cd2, - 0x2cd300002cd4, - 0x2cd500002cd6, - 0x2cd700002cd8, - 0x2cd900002cda, - 0x2cdb00002cdc, - 0x2cdd00002cde, - 0x2cdf00002ce0, - 0x2ce100002ce2, - 0x2ce300002ce5, - 0x2cec00002ced, - 0x2cee00002cf2, - 0x2cf300002cf4, - 0x2d0000002d26, - 0x2d2700002d28, - 0x2d2d00002d2e, - 0x2d3000002d68, - 0x2d7f00002d97, - 0x2da000002da7, - 0x2da800002daf, - 0x2db000002db7, - 0x2db800002dbf, - 0x2dc000002dc7, - 0x2dc800002dcf, - 0x2dd000002dd7, - 0x2dd800002ddf, - 0x2de000002e00, - 0x2e2f00002e30, - 0x300500003008, - 0x302a0000302e, - 0x303c0000303d, - 0x304100003097, - 0x30990000309b, - 0x309d0000309f, - 0x30a1000030fb, - 0x30fc000030ff, - 0x310500003130, - 0x31a0000031c0, - 0x31f000003200, - 0x340000004dc0, - 0x4e000000a48d, - 0xa4d00000a4fe, - 0xa5000000a60d, - 0xa6100000a62c, - 0xa6410000a642, - 0xa6430000a644, - 0xa6450000a646, - 0xa6470000a648, - 0xa6490000a64a, - 0xa64b0000a64c, - 0xa64d0000a64e, - 0xa64f0000a650, - 0xa6510000a652, - 0xa6530000a654, - 0xa6550000a656, - 0xa6570000a658, - 0xa6590000a65a, - 0xa65b0000a65c, - 0xa65d0000a65e, - 0xa65f0000a660, - 0xa6610000a662, - 0xa6630000a664, - 0xa6650000a666, - 0xa6670000a668, - 0xa6690000a66a, - 0xa66b0000a66c, - 0xa66d0000a670, - 0xa6740000a67e, - 0xa67f0000a680, - 0xa6810000a682, - 0xa6830000a684, - 0xa6850000a686, - 0xa6870000a688, - 0xa6890000a68a, - 0xa68b0000a68c, - 0xa68d0000a68e, - 0xa68f0000a690, - 0xa6910000a692, - 0xa6930000a694, - 0xa6950000a696, - 0xa6970000a698, - 0xa6990000a69a, - 0xa69b0000a69c, - 0xa69e0000a6e6, - 0xa6f00000a6f2, - 0xa7170000a720, - 0xa7230000a724, - 0xa7250000a726, - 0xa7270000a728, - 0xa7290000a72a, - 0xa72b0000a72c, - 0xa72d0000a72e, - 0xa72f0000a732, - 0xa7330000a734, - 0xa7350000a736, - 0xa7370000a738, - 0xa7390000a73a, - 0xa73b0000a73c, - 0xa73d0000a73e, - 0xa73f0000a740, - 0xa7410000a742, - 0xa7430000a744, - 0xa7450000a746, - 0xa7470000a748, - 0xa7490000a74a, - 0xa74b0000a74c, - 0xa74d0000a74e, - 0xa74f0000a750, - 0xa7510000a752, - 0xa7530000a754, - 0xa7550000a756, - 0xa7570000a758, - 0xa7590000a75a, - 0xa75b0000a75c, - 0xa75d0000a75e, - 0xa75f0000a760, - 0xa7610000a762, - 0xa7630000a764, - 0xa7650000a766, - 0xa7670000a768, - 0xa7690000a76a, - 0xa76b0000a76c, - 0xa76d0000a76e, - 0xa76f0000a770, - 0xa7710000a779, - 0xa77a0000a77b, - 0xa77c0000a77d, - 0xa77f0000a780, - 0xa7810000a782, - 0xa7830000a784, - 0xa7850000a786, - 0xa7870000a789, - 0xa78c0000a78d, - 0xa78e0000a790, - 0xa7910000a792, - 0xa7930000a796, - 0xa7970000a798, - 0xa7990000a79a, - 0xa79b0000a79c, - 0xa79d0000a79e, - 0xa79f0000a7a0, - 0xa7a10000a7a2, - 0xa7a30000a7a4, - 0xa7a50000a7a6, - 0xa7a70000a7a8, - 0xa7a90000a7aa, - 0xa7af0000a7b0, - 0xa7b50000a7b6, - 0xa7b70000a7b8, - 0xa7b90000a7ba, - 0xa7bb0000a7bc, - 0xa7bd0000a7be, - 0xa7bf0000a7c0, - 0xa7c10000a7c2, - 0xa7c30000a7c4, - 0xa7c80000a7c9, - 0xa7ca0000a7cb, - 0xa7d10000a7d2, - 0xa7d30000a7d4, - 0xa7d50000a7d6, - 0xa7d70000a7d8, - 0xa7d90000a7da, - 0xa7f20000a7f5, - 0xa7f60000a7f8, - 0xa7fa0000a828, - 0xa82c0000a82d, - 0xa8400000a874, - 0xa8800000a8c6, - 0xa8d00000a8da, - 0xa8e00000a8f8, - 0xa8fb0000a8fc, - 0xa8fd0000a92e, - 0xa9300000a954, - 0xa9800000a9c1, - 0xa9cf0000a9da, - 0xa9e00000a9ff, - 0xaa000000aa37, - 0xaa400000aa4e, - 0xaa500000aa5a, - 0xaa600000aa77, - 0xaa7a0000aac3, - 0xaadb0000aade, - 0xaae00000aaf0, - 0xaaf20000aaf7, - 0xab010000ab07, - 0xab090000ab0f, - 0xab110000ab17, - 0xab200000ab27, - 0xab280000ab2f, - 0xab300000ab5b, - 0xab600000ab69, - 0xabc00000abeb, - 0xabec0000abee, - 0xabf00000abfa, - 0xac000000d7a4, - 0xfa0e0000fa10, - 0xfa110000fa12, - 0xfa130000fa15, - 0xfa1f0000fa20, - 0xfa210000fa22, - 0xfa230000fa25, - 0xfa270000fa2a, - 0xfb1e0000fb1f, - 0xfe200000fe30, - 0xfe730000fe74, - 0x100000001000c, - 0x1000d00010027, - 0x100280001003b, - 0x1003c0001003e, - 0x1003f0001004e, - 0x100500001005e, - 0x10080000100fb, - 0x101fd000101fe, - 0x102800001029d, - 0x102a0000102d1, - 0x102e0000102e1, - 0x1030000010320, - 0x1032d00010341, - 0x103420001034a, - 0x103500001037b, - 0x103800001039e, - 0x103a0000103c4, - 0x103c8000103d0, - 0x104280001049e, - 0x104a0000104aa, - 0x104d8000104fc, - 0x1050000010528, - 0x1053000010564, - 0x10597000105a2, - 0x105a3000105b2, - 0x105b3000105ba, - 0x105bb000105bd, - 0x1060000010737, - 0x1074000010756, - 0x1076000010768, - 0x1078000010786, - 0x10787000107b1, - 0x107b2000107bb, - 0x1080000010806, - 0x1080800010809, - 0x1080a00010836, - 0x1083700010839, - 0x1083c0001083d, - 0x1083f00010856, - 0x1086000010877, - 0x108800001089f, - 0x108e0000108f3, - 0x108f4000108f6, - 0x1090000010916, - 0x109200001093a, - 0x10980000109b8, - 0x109be000109c0, - 0x10a0000010a04, - 0x10a0500010a07, - 0x10a0c00010a14, - 0x10a1500010a18, - 0x10a1900010a36, - 0x10a3800010a3b, - 0x10a3f00010a40, - 0x10a6000010a7d, - 0x10a8000010a9d, - 0x10ac000010ac8, - 0x10ac900010ae7, - 0x10b0000010b36, - 0x10b4000010b56, - 0x10b6000010b73, - 0x10b8000010b92, - 0x10c0000010c49, - 0x10cc000010cf3, - 0x10d0000010d28, - 0x10d3000010d3a, - 0x10e8000010eaa, - 0x10eab00010ead, - 0x10eb000010eb2, - 0x10efd00010f1d, - 0x10f2700010f28, - 0x10f3000010f51, - 0x10f7000010f86, - 0x10fb000010fc5, - 0x10fe000010ff7, - 0x1100000011047, - 0x1106600011076, - 0x1107f000110bb, - 0x110c2000110c3, - 0x110d0000110e9, - 0x110f0000110fa, - 0x1110000011135, - 0x1113600011140, - 0x1114400011148, - 0x1115000011174, - 0x1117600011177, - 0x11180000111c5, - 0x111c9000111cd, - 0x111ce000111db, - 0x111dc000111dd, - 0x1120000011212, - 0x1121300011238, - 0x1123e00011242, - 0x1128000011287, - 0x1128800011289, - 0x1128a0001128e, - 0x1128f0001129e, - 0x1129f000112a9, - 0x112b0000112eb, - 0x112f0000112fa, - 0x1130000011304, - 0x113050001130d, - 0x1130f00011311, - 0x1131300011329, - 0x1132a00011331, - 0x1133200011334, - 0x113350001133a, - 0x1133b00011345, - 0x1134700011349, - 0x1134b0001134e, - 0x1135000011351, - 0x1135700011358, - 0x1135d00011364, - 0x113660001136d, - 0x1137000011375, - 0x114000001144b, - 0x114500001145a, - 0x1145e00011462, - 0x11480000114c6, - 0x114c7000114c8, - 0x114d0000114da, - 0x11580000115b6, - 0x115b8000115c1, - 0x115d8000115de, - 0x1160000011641, - 0x1164400011645, - 0x116500001165a, - 0x11680000116b9, - 0x116c0000116ca, - 0x117000001171b, - 0x1171d0001172c, - 0x117300001173a, - 0x1174000011747, - 0x118000001183b, - 0x118c0000118ea, - 0x118ff00011907, - 0x119090001190a, - 0x1190c00011914, - 0x1191500011917, - 0x1191800011936, - 0x1193700011939, - 0x1193b00011944, - 0x119500001195a, - 0x119a0000119a8, - 0x119aa000119d8, - 0x119da000119e2, - 0x119e3000119e5, - 0x11a0000011a3f, - 0x11a4700011a48, - 0x11a5000011a9a, - 0x11a9d00011a9e, - 0x11ab000011af9, - 0x11c0000011c09, - 0x11c0a00011c37, - 0x11c3800011c41, - 0x11c5000011c5a, - 0x11c7200011c90, - 0x11c9200011ca8, - 0x11ca900011cb7, - 0x11d0000011d07, - 0x11d0800011d0a, - 0x11d0b00011d37, - 0x11d3a00011d3b, - 0x11d3c00011d3e, - 0x11d3f00011d48, - 0x11d5000011d5a, - 0x11d6000011d66, - 0x11d6700011d69, - 0x11d6a00011d8f, - 0x11d9000011d92, - 0x11d9300011d99, - 0x11da000011daa, - 0x11ee000011ef7, - 0x11f0000011f11, - 0x11f1200011f3b, - 0x11f3e00011f43, - 0x11f5000011f5a, - 0x11fb000011fb1, - 0x120000001239a, - 0x1248000012544, - 0x12f9000012ff1, - 0x1300000013430, - 0x1344000013456, - 0x1440000014647, - 0x1680000016a39, - 0x16a4000016a5f, - 0x16a6000016a6a, - 0x16a7000016abf, - 0x16ac000016aca, - 0x16ad000016aee, - 0x16af000016af5, - 0x16b0000016b37, - 0x16b4000016b44, - 0x16b5000016b5a, - 0x16b6300016b78, - 0x16b7d00016b90, - 0x16e6000016e80, - 0x16f0000016f4b, - 0x16f4f00016f88, - 0x16f8f00016fa0, - 0x16fe000016fe2, - 0x16fe300016fe5, - 0x16ff000016ff2, - 0x17000000187f8, - 0x1880000018cd6, - 0x18d0000018d09, - 0x1aff00001aff4, - 0x1aff50001affc, - 0x1affd0001afff, - 0x1b0000001b123, - 0x1b1320001b133, - 0x1b1500001b153, - 0x1b1550001b156, - 0x1b1640001b168, - 0x1b1700001b2fc, - 0x1bc000001bc6b, - 0x1bc700001bc7d, - 0x1bc800001bc89, - 0x1bc900001bc9a, - 0x1bc9d0001bc9f, - 0x1cf000001cf2e, - 0x1cf300001cf47, - 0x1da000001da37, - 0x1da3b0001da6d, - 0x1da750001da76, - 0x1da840001da85, - 0x1da9b0001daa0, - 0x1daa10001dab0, - 0x1df000001df1f, - 0x1df250001df2b, - 0x1e0000001e007, - 0x1e0080001e019, - 0x1e01b0001e022, - 0x1e0230001e025, - 0x1e0260001e02b, - 0x1e0300001e06e, - 0x1e08f0001e090, - 0x1e1000001e12d, - 0x1e1300001e13e, - 0x1e1400001e14a, - 0x1e14e0001e14f, - 0x1e2900001e2af, - 0x1e2c00001e2fa, - 0x1e4d00001e4fa, - 0x1e7e00001e7e7, - 0x1e7e80001e7ec, - 0x1e7ed0001e7ef, - 0x1e7f00001e7ff, - 0x1e8000001e8c5, - 0x1e8d00001e8d7, - 0x1e9220001e94c, - 0x1e9500001e95a, - 0x200000002a6e0, - 0x2a7000002b73a, - 0x2b7400002b81e, - 0x2b8200002cea2, - 0x2ceb00002ebe1, - 0x300000003134b, - 0x31350000323b0, - ), - 'CONTEXTJ': ( - 0x200c0000200e, - ), - 'CONTEXTO': ( - 0xb7000000b8, - 0x37500000376, - 0x5f3000005f5, - 0x6600000066a, - 0x6f0000006fa, - 0x30fb000030fc, - ), -} diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/idna/intranges.py b/venv/lib/python3.11/site-packages/pip/_vendor/idna/intranges.py deleted file mode 100644 index 6a43b04..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/idna/intranges.py +++ /dev/null @@ -1,54 +0,0 @@ -""" -Given a list of integers, made up of (hopefully) a small number of long runs -of consecutive integers, compute a representation of the form -((start1, end1), (start2, end2) ...). Then answer the question "was x present -in the original list?" in time O(log(# runs)). -""" - -import bisect -from typing import List, Tuple - -def intranges_from_list(list_: List[int]) -> Tuple[int, ...]: - """Represent a list of integers as a sequence of ranges: - ((start_0, end_0), (start_1, end_1), ...), such that the original - integers are exactly those x such that start_i <= x < end_i for some i. - - Ranges are encoded as single integers (start << 32 | end), not as tuples. - """ - - sorted_list = sorted(list_) - ranges = [] - last_write = -1 - for i in range(len(sorted_list)): - if i+1 < len(sorted_list): - if sorted_list[i] == sorted_list[i+1]-1: - continue - current_range = sorted_list[last_write+1:i+1] - ranges.append(_encode_range(current_range[0], current_range[-1] + 1)) - last_write = i - - return tuple(ranges) - -def _encode_range(start: int, end: int) -> int: - return (start << 32) | end - -def _decode_range(r: int) -> Tuple[int, int]: - return (r >> 32), (r & ((1 << 32) - 1)) - - -def intranges_contain(int_: int, ranges: Tuple[int, ...]) -> bool: - """Determine if `int_` falls into one of the ranges in `ranges`.""" - tuple_ = _encode_range(int_, 0) - pos = bisect.bisect_left(ranges, tuple_) - # we could be immediately ahead of a tuple (start, end) - # with start < int_ <= end - if pos > 0: - left, right = _decode_range(ranges[pos-1]) - if left <= int_ < right: - return True - # or we could be immediately behind a tuple (int_, end) - if pos < len(ranges): - left, _ = _decode_range(ranges[pos]) - if left == int_: - return True - return False diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/idna/package_data.py b/venv/lib/python3.11/site-packages/pip/_vendor/idna/package_data.py deleted file mode 100644 index 8501893..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/idna/package_data.py +++ /dev/null @@ -1,2 +0,0 @@ -__version__ = '3.4' - diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/idna/uts46data.py b/venv/lib/python3.11/site-packages/pip/_vendor/idna/uts46data.py deleted file mode 100644 index 186796c..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/idna/uts46data.py +++ /dev/null @@ -1,8600 +0,0 @@ -# This file is automatically generated by tools/idna-data -# vim: set fileencoding=utf-8 : - -from typing import List, Tuple, Union - - -"""IDNA Mapping Table from UTS46.""" - - -__version__ = '15.0.0' -def _seg_0() -> List[Union[Tuple[int, str], Tuple[int, str, str]]]: - return [ - (0x0, '3'), - (0x1, '3'), - (0x2, '3'), - (0x3, '3'), - (0x4, '3'), - (0x5, '3'), - (0x6, '3'), - (0x7, '3'), - (0x8, '3'), - (0x9, '3'), - (0xA, '3'), - (0xB, '3'), - (0xC, '3'), - (0xD, '3'), - (0xE, '3'), - (0xF, '3'), - (0x10, '3'), - (0x11, '3'), - (0x12, '3'), - (0x13, '3'), - (0x14, '3'), - (0x15, '3'), - (0x16, '3'), - (0x17, '3'), - (0x18, '3'), - (0x19, '3'), - (0x1A, '3'), - (0x1B, '3'), - (0x1C, '3'), - (0x1D, '3'), - (0x1E, '3'), - (0x1F, '3'), - (0x20, '3'), - (0x21, '3'), - (0x22, '3'), - (0x23, '3'), - (0x24, '3'), - (0x25, '3'), - (0x26, '3'), - (0x27, '3'), - (0x28, '3'), - (0x29, '3'), - (0x2A, '3'), - (0x2B, '3'), - (0x2C, '3'), - (0x2D, 'V'), - (0x2E, 'V'), - (0x2F, '3'), - (0x30, 'V'), - (0x31, 'V'), - (0x32, 'V'), - (0x33, 'V'), - (0x34, 'V'), - (0x35, 'V'), - (0x36, 'V'), - (0x37, 'V'), - (0x38, 'V'), - (0x39, 'V'), - (0x3A, '3'), - (0x3B, '3'), - (0x3C, '3'), - (0x3D, '3'), - (0x3E, '3'), - (0x3F, '3'), - (0x40, '3'), - (0x41, 'M', 'a'), - (0x42, 'M', 'b'), - (0x43, 'M', 'c'), - (0x44, 'M', 'd'), - (0x45, 'M', 'e'), - (0x46, 'M', 'f'), - (0x47, 'M', 'g'), - (0x48, 'M', 'h'), - (0x49, 'M', 'i'), - (0x4A, 'M', 'j'), - (0x4B, 'M', 'k'), - (0x4C, 'M', 'l'), - (0x4D, 'M', 'm'), - (0x4E, 'M', 'n'), - (0x4F, 'M', 'o'), - (0x50, 'M', 'p'), - (0x51, 'M', 'q'), - (0x52, 'M', 'r'), - (0x53, 'M', 's'), - (0x54, 'M', 't'), - (0x55, 'M', 'u'), - (0x56, 'M', 'v'), - (0x57, 'M', 'w'), - (0x58, 'M', 'x'), - (0x59, 'M', 'y'), - (0x5A, 'M', 'z'), - (0x5B, '3'), - (0x5C, '3'), - (0x5D, '3'), - (0x5E, '3'), - (0x5F, '3'), - (0x60, '3'), - (0x61, 'V'), - (0x62, 'V'), - (0x63, 'V'), - ] - -def _seg_1() -> List[Union[Tuple[int, str], Tuple[int, str, str]]]: - return [ - (0x64, 'V'), - (0x65, 'V'), - (0x66, 'V'), - (0x67, 'V'), - (0x68, 'V'), - (0x69, 'V'), - (0x6A, 'V'), - (0x6B, 'V'), - (0x6C, 'V'), - (0x6D, 'V'), - (0x6E, 'V'), - (0x6F, 'V'), - (0x70, 'V'), - (0x71, 'V'), - (0x72, 'V'), - (0x73, 'V'), - (0x74, 'V'), - (0x75, 'V'), - (0x76, 'V'), - (0x77, 'V'), - (0x78, 'V'), - (0x79, 'V'), - (0x7A, 'V'), - (0x7B, '3'), - (0x7C, '3'), - (0x7D, '3'), - (0x7E, '3'), - (0x7F, '3'), - (0x80, 'X'), - (0x81, 'X'), - (0x82, 'X'), - (0x83, 'X'), - (0x84, 'X'), - (0x85, 'X'), - (0x86, 'X'), - (0x87, 'X'), - (0x88, 'X'), - (0x89, 'X'), - (0x8A, 'X'), - (0x8B, 'X'), - (0x8C, 'X'), - (0x8D, 'X'), - (0x8E, 'X'), - (0x8F, 'X'), - (0x90, 'X'), - (0x91, 'X'), - (0x92, 'X'), - (0x93, 'X'), - (0x94, 'X'), - (0x95, 'X'), - (0x96, 'X'), - (0x97, 'X'), - (0x98, 'X'), - (0x99, 'X'), - (0x9A, 'X'), - (0x9B, 'X'), - (0x9C, 'X'), - (0x9D, 'X'), - (0x9E, 'X'), - (0x9F, 'X'), - (0xA0, '3', ' '), - (0xA1, 'V'), - (0xA2, 'V'), - (0xA3, 'V'), - (0xA4, 'V'), - (0xA5, 'V'), - (0xA6, 'V'), - (0xA7, 'V'), - (0xA8, '3', ' ̈'), - (0xA9, 'V'), - (0xAA, 'M', 'a'), - (0xAB, 'V'), - (0xAC, 'V'), - (0xAD, 'I'), - (0xAE, 'V'), - (0xAF, '3', ' ̄'), - (0xB0, 'V'), - (0xB1, 'V'), - (0xB2, 'M', '2'), - (0xB3, 'M', '3'), - (0xB4, '3', ' ́'), - (0xB5, 'M', 'μ'), - (0xB6, 'V'), - (0xB7, 'V'), - (0xB8, '3', ' ̧'), - (0xB9, 'M', '1'), - (0xBA, 'M', 'o'), - (0xBB, 'V'), - (0xBC, 'M', '1⁄4'), - (0xBD, 'M', '1⁄2'), - (0xBE, 'M', '3⁄4'), - (0xBF, 'V'), - (0xC0, 'M', 'à'), - (0xC1, 'M', 'á'), - (0xC2, 'M', 'â'), - (0xC3, 'M', 'ã'), - (0xC4, 'M', 'ä'), - (0xC5, 'M', 'å'), - (0xC6, 'M', 'æ'), - (0xC7, 'M', 'ç'), - ] - -def _seg_2() -> List[Union[Tuple[int, str], Tuple[int, str, str]]]: - return [ - (0xC8, 'M', 'è'), - (0xC9, 'M', 'é'), - (0xCA, 'M', 'ê'), - (0xCB, 'M', 'ë'), - (0xCC, 'M', 'ì'), - (0xCD, 'M', 'í'), - (0xCE, 'M', 'î'), - (0xCF, 'M', 'ï'), - (0xD0, 'M', 'ð'), - (0xD1, 'M', 'ñ'), - (0xD2, 'M', 'ò'), - (0xD3, 'M', 'ó'), - (0xD4, 'M', 'ô'), - (0xD5, 'M', 'õ'), - (0xD6, 'M', 'ö'), - (0xD7, 'V'), - (0xD8, 'M', 'ø'), - (0xD9, 'M', 'ù'), - (0xDA, 'M', 'ú'), - (0xDB, 'M', 'û'), - (0xDC, 'M', 'ü'), - (0xDD, 'M', 'ý'), - (0xDE, 'M', 'þ'), - (0xDF, 'D', 'ss'), - (0xE0, 'V'), - (0xE1, 'V'), - (0xE2, 'V'), - (0xE3, 'V'), - (0xE4, 'V'), - (0xE5, 'V'), - (0xE6, 'V'), - (0xE7, 'V'), - (0xE8, 'V'), - (0xE9, 'V'), - (0xEA, 'V'), - (0xEB, 'V'), - (0xEC, 'V'), - (0xED, 'V'), - (0xEE, 'V'), - (0xEF, 'V'), - (0xF0, 'V'), - (0xF1, 'V'), - (0xF2, 'V'), - (0xF3, 'V'), - (0xF4, 'V'), - (0xF5, 'V'), - (0xF6, 'V'), - (0xF7, 'V'), - (0xF8, 'V'), - (0xF9, 'V'), - (0xFA, 'V'), - (0xFB, 'V'), - (0xFC, 'V'), - (0xFD, 'V'), - (0xFE, 'V'), - (0xFF, 'V'), - (0x100, 'M', 'ā'), - (0x101, 'V'), - (0x102, 'M', 'ă'), - (0x103, 'V'), - (0x104, 'M', 'ą'), - (0x105, 'V'), - (0x106, 'M', 'ć'), - (0x107, 'V'), - (0x108, 'M', 'ĉ'), - (0x109, 'V'), - (0x10A, 'M', 'ċ'), - (0x10B, 'V'), - (0x10C, 'M', 'č'), - (0x10D, 'V'), - (0x10E, 'M', 'ď'), - (0x10F, 'V'), - (0x110, 'M', 'đ'), - (0x111, 'V'), - (0x112, 'M', 'ē'), - (0x113, 'V'), - (0x114, 'M', 'ĕ'), - (0x115, 'V'), - (0x116, 'M', 'ė'), - (0x117, 'V'), - (0x118, 'M', 'ę'), - (0x119, 'V'), - (0x11A, 'M', 'ě'), - (0x11B, 'V'), - (0x11C, 'M', 'ĝ'), - (0x11D, 'V'), - (0x11E, 'M', 'ğ'), - (0x11F, 'V'), - (0x120, 'M', 'ġ'), - (0x121, 'V'), - (0x122, 'M', 'ģ'), - (0x123, 'V'), - (0x124, 'M', 'ĥ'), - (0x125, 'V'), - (0x126, 'M', 'ħ'), - (0x127, 'V'), - (0x128, 'M', 'ĩ'), - (0x129, 'V'), - (0x12A, 'M', 'ī'), - (0x12B, 'V'), - ] - -def _seg_3() -> List[Union[Tuple[int, str], Tuple[int, str, str]]]: - return [ - (0x12C, 'M', 'ĭ'), - (0x12D, 'V'), - (0x12E, 'M', 'į'), - (0x12F, 'V'), - (0x130, 'M', 'i̇'), - (0x131, 'V'), - (0x132, 'M', 'ij'), - (0x134, 'M', 'ĵ'), - (0x135, 'V'), - (0x136, 'M', 'ķ'), - (0x137, 'V'), - (0x139, 'M', 'ĺ'), - (0x13A, 'V'), - (0x13B, 'M', 'ļ'), - (0x13C, 'V'), - (0x13D, 'M', 'ľ'), - (0x13E, 'V'), - (0x13F, 'M', 'l·'), - (0x141, 'M', 'ł'), - (0x142, 'V'), - (0x143, 'M', 'ń'), - (0x144, 'V'), - (0x145, 'M', 'ņ'), - (0x146, 'V'), - (0x147, 'M', 'ň'), - (0x148, 'V'), - (0x149, 'M', 'ʼn'), - (0x14A, 'M', 'ŋ'), - (0x14B, 'V'), - (0x14C, 'M', 'ō'), - (0x14D, 'V'), - (0x14E, 'M', 'ŏ'), - (0x14F, 'V'), - (0x150, 'M', 'ő'), - (0x151, 'V'), - (0x152, 'M', 'œ'), - (0x153, 'V'), - (0x154, 'M', 'ŕ'), - (0x155, 'V'), - (0x156, 'M', 'ŗ'), - (0x157, 'V'), - (0x158, 'M', 'ř'), - (0x159, 'V'), - (0x15A, 'M', 'ś'), - (0x15B, 'V'), - (0x15C, 'M', 'ŝ'), - (0x15D, 'V'), - (0x15E, 'M', 'ş'), - (0x15F, 'V'), - (0x160, 'M', 'š'), - (0x161, 'V'), - (0x162, 'M', 'ţ'), - (0x163, 'V'), - (0x164, 'M', 'ť'), - (0x165, 'V'), - (0x166, 'M', 'ŧ'), - (0x167, 'V'), - (0x168, 'M', 'ũ'), - (0x169, 'V'), - (0x16A, 'M', 'ū'), - (0x16B, 'V'), - (0x16C, 'M', 'ŭ'), - (0x16D, 'V'), - (0x16E, 'M', 'ů'), - (0x16F, 'V'), - (0x170, 'M', 'ű'), - (0x171, 'V'), - (0x172, 'M', 'ų'), - (0x173, 'V'), - (0x174, 'M', 'ŵ'), - (0x175, 'V'), - (0x176, 'M', 'ŷ'), - (0x177, 'V'), - (0x178, 'M', 'ÿ'), - (0x179, 'M', 'ź'), - (0x17A, 'V'), - (0x17B, 'M', 'ż'), - (0x17C, 'V'), - (0x17D, 'M', 'ž'), - (0x17E, 'V'), - (0x17F, 'M', 's'), - (0x180, 'V'), - (0x181, 'M', 'ɓ'), - (0x182, 'M', 'ƃ'), - (0x183, 'V'), - (0x184, 'M', 'ƅ'), - (0x185, 'V'), - (0x186, 'M', 'ɔ'), - (0x187, 'M', 'ƈ'), - (0x188, 'V'), - (0x189, 'M', 'ɖ'), - (0x18A, 'M', 'ɗ'), - (0x18B, 'M', 'ƌ'), - (0x18C, 'V'), - (0x18E, 'M', 'ǝ'), - (0x18F, 'M', 'ə'), - (0x190, 'M', 'ɛ'), - (0x191, 'M', 'ƒ'), - (0x192, 'V'), - (0x193, 'M', 'ɠ'), - ] - -def _seg_4() -> List[Union[Tuple[int, str], Tuple[int, str, str]]]: - return [ - (0x194, 'M', 'ɣ'), - (0x195, 'V'), - (0x196, 'M', 'ɩ'), - (0x197, 'M', 'ɨ'), - (0x198, 'M', 'ƙ'), - (0x199, 'V'), - (0x19C, 'M', 'ɯ'), - (0x19D, 'M', 'ɲ'), - (0x19E, 'V'), - (0x19F, 'M', 'ɵ'), - (0x1A0, 'M', 'ơ'), - (0x1A1, 'V'), - (0x1A2, 'M', 'ƣ'), - (0x1A3, 'V'), - (0x1A4, 'M', 'ƥ'), - (0x1A5, 'V'), - (0x1A6, 'M', 'ʀ'), - (0x1A7, 'M', 'ƨ'), - (0x1A8, 'V'), - (0x1A9, 'M', 'ʃ'), - (0x1AA, 'V'), - (0x1AC, 'M', 'ƭ'), - (0x1AD, 'V'), - (0x1AE, 'M', 'ʈ'), - (0x1AF, 'M', 'ư'), - (0x1B0, 'V'), - (0x1B1, 'M', 'ʊ'), - (0x1B2, 'M', 'ʋ'), - (0x1B3, 'M', 'ƴ'), - (0x1B4, 'V'), - (0x1B5, 'M', 'ƶ'), - (0x1B6, 'V'), - (0x1B7, 'M', 'ʒ'), - (0x1B8, 'M', 'ƹ'), - (0x1B9, 'V'), - (0x1BC, 'M', 'ƽ'), - (0x1BD, 'V'), - (0x1C4, 'M', 'dž'), - (0x1C7, 'M', 'lj'), - (0x1CA, 'M', 'nj'), - (0x1CD, 'M', 'ǎ'), - (0x1CE, 'V'), - (0x1CF, 'M', 'ǐ'), - (0x1D0, 'V'), - (0x1D1, 'M', 'ǒ'), - (0x1D2, 'V'), - (0x1D3, 'M', 'ǔ'), - (0x1D4, 'V'), - (0x1D5, 'M', 'ǖ'), - (0x1D6, 'V'), - (0x1D7, 'M', 'ǘ'), - (0x1D8, 'V'), - (0x1D9, 'M', 'ǚ'), - (0x1DA, 'V'), - (0x1DB, 'M', 'ǜ'), - (0x1DC, 'V'), - (0x1DE, 'M', 'ǟ'), - (0x1DF, 'V'), - (0x1E0, 'M', 'ǡ'), - (0x1E1, 'V'), - (0x1E2, 'M', 'ǣ'), - (0x1E3, 'V'), - (0x1E4, 'M', 'ǥ'), - (0x1E5, 'V'), - (0x1E6, 'M', 'ǧ'), - (0x1E7, 'V'), - (0x1E8, 'M', 'ǩ'), - (0x1E9, 'V'), - (0x1EA, 'M', 'ǫ'), - (0x1EB, 'V'), - (0x1EC, 'M', 'ǭ'), - (0x1ED, 'V'), - (0x1EE, 'M', 'ǯ'), - (0x1EF, 'V'), - (0x1F1, 'M', 'dz'), - (0x1F4, 'M', 'ǵ'), - (0x1F5, 'V'), - (0x1F6, 'M', 'ƕ'), - (0x1F7, 'M', 'ƿ'), - (0x1F8, 'M', 'ǹ'), - (0x1F9, 'V'), - (0x1FA, 'M', 'ǻ'), - (0x1FB, 'V'), - (0x1FC, 'M', 'ǽ'), - (0x1FD, 'V'), - (0x1FE, 'M', 'ǿ'), - (0x1FF, 'V'), - (0x200, 'M', 'ȁ'), - (0x201, 'V'), - (0x202, 'M', 'ȃ'), - (0x203, 'V'), - (0x204, 'M', 'ȅ'), - (0x205, 'V'), - (0x206, 'M', 'ȇ'), - (0x207, 'V'), - (0x208, 'M', 'ȉ'), - (0x209, 'V'), - (0x20A, 'M', 'ȋ'), - (0x20B, 'V'), - (0x20C, 'M', 'ȍ'), - ] - -def _seg_5() -> List[Union[Tuple[int, str], Tuple[int, str, str]]]: - return [ - (0x20D, 'V'), - (0x20E, 'M', 'ȏ'), - (0x20F, 'V'), - (0x210, 'M', 'ȑ'), - (0x211, 'V'), - (0x212, 'M', 'ȓ'), - (0x213, 'V'), - (0x214, 'M', 'ȕ'), - (0x215, 'V'), - (0x216, 'M', 'ȗ'), - (0x217, 'V'), - (0x218, 'M', 'ș'), - (0x219, 'V'), - (0x21A, 'M', 'ț'), - (0x21B, 'V'), - (0x21C, 'M', 'ȝ'), - (0x21D, 'V'), - (0x21E, 'M', 'ȟ'), - (0x21F, 'V'), - (0x220, 'M', 'ƞ'), - (0x221, 'V'), - (0x222, 'M', 'ȣ'), - (0x223, 'V'), - (0x224, 'M', 'ȥ'), - (0x225, 'V'), - (0x226, 'M', 'ȧ'), - (0x227, 'V'), - (0x228, 'M', 'ȩ'), - (0x229, 'V'), - (0x22A, 'M', 'ȫ'), - (0x22B, 'V'), - (0x22C, 'M', 'ȭ'), - (0x22D, 'V'), - (0x22E, 'M', 'ȯ'), - (0x22F, 'V'), - (0x230, 'M', 'ȱ'), - (0x231, 'V'), - (0x232, 'M', 'ȳ'), - (0x233, 'V'), - (0x23A, 'M', 'ⱥ'), - (0x23B, 'M', 'ȼ'), - (0x23C, 'V'), - (0x23D, 'M', 'ƚ'), - (0x23E, 'M', 'ⱦ'), - (0x23F, 'V'), - (0x241, 'M', 'ɂ'), - (0x242, 'V'), - (0x243, 'M', 'ƀ'), - (0x244, 'M', 'ʉ'), - (0x245, 'M', 'ʌ'), - (0x246, 'M', 'ɇ'), - (0x247, 'V'), - (0x248, 'M', 'ɉ'), - (0x249, 'V'), - (0x24A, 'M', 'ɋ'), - (0x24B, 'V'), - (0x24C, 'M', 'ɍ'), - (0x24D, 'V'), - (0x24E, 'M', 'ɏ'), - (0x24F, 'V'), - (0x2B0, 'M', 'h'), - (0x2B1, 'M', 'ɦ'), - (0x2B2, 'M', 'j'), - (0x2B3, 'M', 'r'), - (0x2B4, 'M', 'ɹ'), - (0x2B5, 'M', 'ɻ'), - (0x2B6, 'M', 'ʁ'), - (0x2B7, 'M', 'w'), - (0x2B8, 'M', 'y'), - (0x2B9, 'V'), - (0x2D8, '3', ' ̆'), - (0x2D9, '3', ' ̇'), - (0x2DA, '3', ' ̊'), - (0x2DB, '3', ' ̨'), - (0x2DC, '3', ' ̃'), - (0x2DD, '3', ' ̋'), - (0x2DE, 'V'), - (0x2E0, 'M', 'ɣ'), - (0x2E1, 'M', 'l'), - (0x2E2, 'M', 's'), - (0x2E3, 'M', 'x'), - (0x2E4, 'M', 'ʕ'), - (0x2E5, 'V'), - (0x340, 'M', '̀'), - (0x341, 'M', '́'), - (0x342, 'V'), - (0x343, 'M', '̓'), - (0x344, 'M', '̈́'), - (0x345, 'M', 'ι'), - (0x346, 'V'), - (0x34F, 'I'), - (0x350, 'V'), - (0x370, 'M', 'ͱ'), - (0x371, 'V'), - (0x372, 'M', 'ͳ'), - (0x373, 'V'), - (0x374, 'M', 'ʹ'), - (0x375, 'V'), - (0x376, 'M', 'ͷ'), - (0x377, 'V'), - ] - -def _seg_6() -> List[Union[Tuple[int, str], Tuple[int, str, str]]]: - return [ - (0x378, 'X'), - (0x37A, '3', ' ι'), - (0x37B, 'V'), - (0x37E, '3', ';'), - (0x37F, 'M', 'ϳ'), - (0x380, 'X'), - (0x384, '3', ' ́'), - (0x385, '3', ' ̈́'), - (0x386, 'M', 'ά'), - (0x387, 'M', '·'), - (0x388, 'M', 'έ'), - (0x389, 'M', 'ή'), - (0x38A, 'M', 'ί'), - (0x38B, 'X'), - (0x38C, 'M', 'ό'), - (0x38D, 'X'), - (0x38E, 'M', 'ύ'), - (0x38F, 'M', 'ώ'), - (0x390, 'V'), - (0x391, 'M', 'α'), - (0x392, 'M', 'β'), - (0x393, 'M', 'γ'), - (0x394, 'M', 'δ'), - (0x395, 'M', 'ε'), - (0x396, 'M', 'ζ'), - (0x397, 'M', 'η'), - (0x398, 'M', 'θ'), - (0x399, 'M', 'ι'), - (0x39A, 'M', 'κ'), - (0x39B, 'M', 'λ'), - (0x39C, 'M', 'μ'), - (0x39D, 'M', 'ν'), - (0x39E, 'M', 'ξ'), - (0x39F, 'M', 'ο'), - (0x3A0, 'M', 'π'), - (0x3A1, 'M', 'ρ'), - (0x3A2, 'X'), - (0x3A3, 'M', 'σ'), - (0x3A4, 'M', 'τ'), - (0x3A5, 'M', 'υ'), - (0x3A6, 'M', 'φ'), - (0x3A7, 'M', 'χ'), - (0x3A8, 'M', 'ψ'), - (0x3A9, 'M', 'ω'), - (0x3AA, 'M', 'ϊ'), - (0x3AB, 'M', 'ϋ'), - (0x3AC, 'V'), - (0x3C2, 'D', 'σ'), - (0x3C3, 'V'), - (0x3CF, 'M', 'ϗ'), - (0x3D0, 'M', 'β'), - (0x3D1, 'M', 'θ'), - (0x3D2, 'M', 'υ'), - (0x3D3, 'M', 'ύ'), - (0x3D4, 'M', 'ϋ'), - (0x3D5, 'M', 'φ'), - (0x3D6, 'M', 'π'), - (0x3D7, 'V'), - (0x3D8, 'M', 'ϙ'), - (0x3D9, 'V'), - (0x3DA, 'M', 'ϛ'), - (0x3DB, 'V'), - (0x3DC, 'M', 'ϝ'), - (0x3DD, 'V'), - (0x3DE, 'M', 'ϟ'), - (0x3DF, 'V'), - (0x3E0, 'M', 'ϡ'), - (0x3E1, 'V'), - (0x3E2, 'M', 'ϣ'), - (0x3E3, 'V'), - (0x3E4, 'M', 'ϥ'), - (0x3E5, 'V'), - (0x3E6, 'M', 'ϧ'), - (0x3E7, 'V'), - (0x3E8, 'M', 'ϩ'), - (0x3E9, 'V'), - (0x3EA, 'M', 'ϫ'), - (0x3EB, 'V'), - (0x3EC, 'M', 'ϭ'), - (0x3ED, 'V'), - (0x3EE, 'M', 'ϯ'), - (0x3EF, 'V'), - (0x3F0, 'M', 'κ'), - (0x3F1, 'M', 'ρ'), - (0x3F2, 'M', 'σ'), - (0x3F3, 'V'), - (0x3F4, 'M', 'θ'), - (0x3F5, 'M', 'ε'), - (0x3F6, 'V'), - (0x3F7, 'M', 'ϸ'), - (0x3F8, 'V'), - (0x3F9, 'M', 'σ'), - (0x3FA, 'M', 'ϻ'), - (0x3FB, 'V'), - (0x3FD, 'M', 'ͻ'), - (0x3FE, 'M', 'ͼ'), - (0x3FF, 'M', 'ͽ'), - (0x400, 'M', 'ѐ'), - (0x401, 'M', 'ё'), - (0x402, 'M', 'ђ'), - ] - -def _seg_7() -> List[Union[Tuple[int, str], Tuple[int, str, str]]]: - return [ - (0x403, 'M', 'ѓ'), - (0x404, 'M', 'є'), - (0x405, 'M', 'ѕ'), - (0x406, 'M', 'і'), - (0x407, 'M', 'ї'), - (0x408, 'M', 'ј'), - (0x409, 'M', 'љ'), - (0x40A, 'M', 'њ'), - (0x40B, 'M', 'ћ'), - (0x40C, 'M', 'ќ'), - (0x40D, 'M', 'ѝ'), - (0x40E, 'M', 'ў'), - (0x40F, 'M', 'џ'), - (0x410, 'M', 'а'), - (0x411, 'M', 'б'), - (0x412, 'M', 'в'), - (0x413, 'M', 'г'), - (0x414, 'M', 'д'), - (0x415, 'M', 'е'), - (0x416, 'M', 'ж'), - (0x417, 'M', 'з'), - (0x418, 'M', 'и'), - (0x419, 'M', 'й'), - (0x41A, 'M', 'к'), - (0x41B, 'M', 'л'), - (0x41C, 'M', 'м'), - (0x41D, 'M', 'н'), - (0x41E, 'M', 'о'), - (0x41F, 'M', 'п'), - (0x420, 'M', 'р'), - (0x421, 'M', 'с'), - (0x422, 'M', 'т'), - (0x423, 'M', 'у'), - (0x424, 'M', 'ф'), - (0x425, 'M', 'х'), - (0x426, 'M', 'ц'), - (0x427, 'M', 'ч'), - (0x428, 'M', 'ш'), - (0x429, 'M', 'щ'), - (0x42A, 'M', 'ъ'), - (0x42B, 'M', 'ы'), - (0x42C, 'M', 'ь'), - (0x42D, 'M', 'э'), - (0x42E, 'M', 'ю'), - (0x42F, 'M', 'я'), - (0x430, 'V'), - (0x460, 'M', 'ѡ'), - (0x461, 'V'), - (0x462, 'M', 'ѣ'), - (0x463, 'V'), - (0x464, 'M', 'ѥ'), - (0x465, 'V'), - (0x466, 'M', 'ѧ'), - (0x467, 'V'), - (0x468, 'M', 'ѩ'), - (0x469, 'V'), - (0x46A, 'M', 'ѫ'), - (0x46B, 'V'), - (0x46C, 'M', 'ѭ'), - (0x46D, 'V'), - (0x46E, 'M', 'ѯ'), - (0x46F, 'V'), - (0x470, 'M', 'ѱ'), - (0x471, 'V'), - (0x472, 'M', 'ѳ'), - (0x473, 'V'), - (0x474, 'M', 'ѵ'), - (0x475, 'V'), - (0x476, 'M', 'ѷ'), - (0x477, 'V'), - (0x478, 'M', 'ѹ'), - (0x479, 'V'), - (0x47A, 'M', 'ѻ'), - (0x47B, 'V'), - (0x47C, 'M', 'ѽ'), - (0x47D, 'V'), - (0x47E, 'M', 'ѿ'), - (0x47F, 'V'), - (0x480, 'M', 'ҁ'), - (0x481, 'V'), - (0x48A, 'M', 'ҋ'), - (0x48B, 'V'), - (0x48C, 'M', 'ҍ'), - (0x48D, 'V'), - (0x48E, 'M', 'ҏ'), - (0x48F, 'V'), - (0x490, 'M', 'ґ'), - (0x491, 'V'), - (0x492, 'M', 'ғ'), - (0x493, 'V'), - (0x494, 'M', 'ҕ'), - (0x495, 'V'), - (0x496, 'M', 'җ'), - (0x497, 'V'), - (0x498, 'M', 'ҙ'), - (0x499, 'V'), - (0x49A, 'M', 'қ'), - (0x49B, 'V'), - (0x49C, 'M', 'ҝ'), - (0x49D, 'V'), - ] - -def _seg_8() -> List[Union[Tuple[int, str], Tuple[int, str, str]]]: - return [ - (0x49E, 'M', 'ҟ'), - (0x49F, 'V'), - (0x4A0, 'M', 'ҡ'), - (0x4A1, 'V'), - (0x4A2, 'M', 'ң'), - (0x4A3, 'V'), - (0x4A4, 'M', 'ҥ'), - (0x4A5, 'V'), - (0x4A6, 'M', 'ҧ'), - (0x4A7, 'V'), - (0x4A8, 'M', 'ҩ'), - (0x4A9, 'V'), - (0x4AA, 'M', 'ҫ'), - (0x4AB, 'V'), - (0x4AC, 'M', 'ҭ'), - (0x4AD, 'V'), - (0x4AE, 'M', 'ү'), - (0x4AF, 'V'), - (0x4B0, 'M', 'ұ'), - (0x4B1, 'V'), - (0x4B2, 'M', 'ҳ'), - (0x4B3, 'V'), - (0x4B4, 'M', 'ҵ'), - (0x4B5, 'V'), - (0x4B6, 'M', 'ҷ'), - (0x4B7, 'V'), - (0x4B8, 'M', 'ҹ'), - (0x4B9, 'V'), - (0x4BA, 'M', 'һ'), - (0x4BB, 'V'), - (0x4BC, 'M', 'ҽ'), - (0x4BD, 'V'), - (0x4BE, 'M', 'ҿ'), - (0x4BF, 'V'), - (0x4C0, 'X'), - (0x4C1, 'M', 'ӂ'), - (0x4C2, 'V'), - (0x4C3, 'M', 'ӄ'), - (0x4C4, 'V'), - (0x4C5, 'M', 'ӆ'), - (0x4C6, 'V'), - (0x4C7, 'M', 'ӈ'), - (0x4C8, 'V'), - (0x4C9, 'M', 'ӊ'), - (0x4CA, 'V'), - (0x4CB, 'M', 'ӌ'), - (0x4CC, 'V'), - (0x4CD, 'M', 'ӎ'), - (0x4CE, 'V'), - (0x4D0, 'M', 'ӑ'), - (0x4D1, 'V'), - (0x4D2, 'M', 'ӓ'), - (0x4D3, 'V'), - (0x4D4, 'M', 'ӕ'), - (0x4D5, 'V'), - (0x4D6, 'M', 'ӗ'), - (0x4D7, 'V'), - (0x4D8, 'M', 'ә'), - (0x4D9, 'V'), - (0x4DA, 'M', 'ӛ'), - (0x4DB, 'V'), - (0x4DC, 'M', 'ӝ'), - (0x4DD, 'V'), - (0x4DE, 'M', 'ӟ'), - (0x4DF, 'V'), - (0x4E0, 'M', 'ӡ'), - (0x4E1, 'V'), - (0x4E2, 'M', 'ӣ'), - (0x4E3, 'V'), - (0x4E4, 'M', 'ӥ'), - (0x4E5, 'V'), - (0x4E6, 'M', 'ӧ'), - (0x4E7, 'V'), - (0x4E8, 'M', 'ө'), - (0x4E9, 'V'), - (0x4EA, 'M', 'ӫ'), - (0x4EB, 'V'), - (0x4EC, 'M', 'ӭ'), - (0x4ED, 'V'), - (0x4EE, 'M', 'ӯ'), - (0x4EF, 'V'), - (0x4F0, 'M', 'ӱ'), - (0x4F1, 'V'), - (0x4F2, 'M', 'ӳ'), - (0x4F3, 'V'), - (0x4F4, 'M', 'ӵ'), - (0x4F5, 'V'), - (0x4F6, 'M', 'ӷ'), - (0x4F7, 'V'), - (0x4F8, 'M', 'ӹ'), - (0x4F9, 'V'), - (0x4FA, 'M', 'ӻ'), - (0x4FB, 'V'), - (0x4FC, 'M', 'ӽ'), - (0x4FD, 'V'), - (0x4FE, 'M', 'ӿ'), - (0x4FF, 'V'), - (0x500, 'M', 'ԁ'), - (0x501, 'V'), - (0x502, 'M', 'ԃ'), - ] - -def _seg_9() -> List[Union[Tuple[int, str], Tuple[int, str, str]]]: - return [ - (0x503, 'V'), - (0x504, 'M', 'ԅ'), - (0x505, 'V'), - (0x506, 'M', 'ԇ'), - (0x507, 'V'), - (0x508, 'M', 'ԉ'), - (0x509, 'V'), - (0x50A, 'M', 'ԋ'), - (0x50B, 'V'), - (0x50C, 'M', 'ԍ'), - (0x50D, 'V'), - (0x50E, 'M', 'ԏ'), - (0x50F, 'V'), - (0x510, 'M', 'ԑ'), - (0x511, 'V'), - (0x512, 'M', 'ԓ'), - (0x513, 'V'), - (0x514, 'M', 'ԕ'), - (0x515, 'V'), - (0x516, 'M', 'ԗ'), - (0x517, 'V'), - (0x518, 'M', 'ԙ'), - (0x519, 'V'), - (0x51A, 'M', 'ԛ'), - (0x51B, 'V'), - (0x51C, 'M', 'ԝ'), - (0x51D, 'V'), - (0x51E, 'M', 'ԟ'), - (0x51F, 'V'), - (0x520, 'M', 'ԡ'), - (0x521, 'V'), - (0x522, 'M', 'ԣ'), - (0x523, 'V'), - (0x524, 'M', 'ԥ'), - (0x525, 'V'), - (0x526, 'M', 'ԧ'), - (0x527, 'V'), - (0x528, 'M', 'ԩ'), - (0x529, 'V'), - (0x52A, 'M', 'ԫ'), - (0x52B, 'V'), - (0x52C, 'M', 'ԭ'), - (0x52D, 'V'), - (0x52E, 'M', 'ԯ'), - (0x52F, 'V'), - (0x530, 'X'), - (0x531, 'M', 'ա'), - (0x532, 'M', 'բ'), - (0x533, 'M', 'գ'), - (0x534, 'M', 'դ'), - (0x535, 'M', 'ե'), - (0x536, 'M', 'զ'), - (0x537, 'M', 'է'), - (0x538, 'M', 'ը'), - (0x539, 'M', 'թ'), - (0x53A, 'M', 'ժ'), - (0x53B, 'M', 'ի'), - (0x53C, 'M', 'լ'), - (0x53D, 'M', 'խ'), - (0x53E, 'M', 'ծ'), - (0x53F, 'M', 'կ'), - (0x540, 'M', 'հ'), - (0x541, 'M', 'ձ'), - (0x542, 'M', 'ղ'), - (0x543, 'M', 'ճ'), - (0x544, 'M', 'մ'), - (0x545, 'M', 'յ'), - (0x546, 'M', 'ն'), - (0x547, 'M', 'շ'), - (0x548, 'M', 'ո'), - (0x549, 'M', 'չ'), - (0x54A, 'M', 'պ'), - (0x54B, 'M', 'ջ'), - (0x54C, 'M', 'ռ'), - (0x54D, 'M', 'ս'), - (0x54E, 'M', 'վ'), - (0x54F, 'M', 'տ'), - (0x550, 'M', 'ր'), - (0x551, 'M', 'ց'), - (0x552, 'M', 'ւ'), - (0x553, 'M', 'փ'), - (0x554, 'M', 'ք'), - (0x555, 'M', 'օ'), - (0x556, 'M', 'ֆ'), - (0x557, 'X'), - (0x559, 'V'), - (0x587, 'M', 'եւ'), - (0x588, 'V'), - (0x58B, 'X'), - (0x58D, 'V'), - (0x590, 'X'), - (0x591, 'V'), - (0x5C8, 'X'), - (0x5D0, 'V'), - (0x5EB, 'X'), - (0x5EF, 'V'), - (0x5F5, 'X'), - (0x606, 'V'), - (0x61C, 'X'), - (0x61D, 'V'), - ] - -def _seg_10() -> List[Union[Tuple[int, str], Tuple[int, str, str]]]: - return [ - (0x675, 'M', 'اٴ'), - (0x676, 'M', 'وٴ'), - (0x677, 'M', 'ۇٴ'), - (0x678, 'M', 'يٴ'), - (0x679, 'V'), - (0x6DD, 'X'), - (0x6DE, 'V'), - (0x70E, 'X'), - (0x710, 'V'), - (0x74B, 'X'), - (0x74D, 'V'), - (0x7B2, 'X'), - (0x7C0, 'V'), - (0x7FB, 'X'), - (0x7FD, 'V'), - (0x82E, 'X'), - (0x830, 'V'), - (0x83F, 'X'), - (0x840, 'V'), - (0x85C, 'X'), - (0x85E, 'V'), - (0x85F, 'X'), - (0x860, 'V'), - (0x86B, 'X'), - (0x870, 'V'), - (0x88F, 'X'), - (0x898, 'V'), - (0x8E2, 'X'), - (0x8E3, 'V'), - (0x958, 'M', 'क़'), - (0x959, 'M', 'ख़'), - (0x95A, 'M', 'ग़'), - (0x95B, 'M', 'ज़'), - (0x95C, 'M', 'ड़'), - (0x95D, 'M', 'ढ़'), - (0x95E, 'M', 'फ़'), - (0x95F, 'M', 'य़'), - (0x960, 'V'), - (0x984, 'X'), - (0x985, 'V'), - (0x98D, 'X'), - (0x98F, 'V'), - (0x991, 'X'), - (0x993, 'V'), - (0x9A9, 'X'), - (0x9AA, 'V'), - (0x9B1, 'X'), - (0x9B2, 'V'), - (0x9B3, 'X'), - (0x9B6, 'V'), - (0x9BA, 'X'), - (0x9BC, 'V'), - (0x9C5, 'X'), - (0x9C7, 'V'), - (0x9C9, 'X'), - (0x9CB, 'V'), - (0x9CF, 'X'), - (0x9D7, 'V'), - (0x9D8, 'X'), - (0x9DC, 'M', 'ড়'), - (0x9DD, 'M', 'ঢ়'), - (0x9DE, 'X'), - (0x9DF, 'M', 'য়'), - (0x9E0, 'V'), - (0x9E4, 'X'), - (0x9E6, 'V'), - (0x9FF, 'X'), - (0xA01, 'V'), - (0xA04, 'X'), - (0xA05, 'V'), - (0xA0B, 'X'), - (0xA0F, 'V'), - (0xA11, 'X'), - (0xA13, 'V'), - (0xA29, 'X'), - (0xA2A, 'V'), - (0xA31, 'X'), - (0xA32, 'V'), - (0xA33, 'M', 'ਲ਼'), - (0xA34, 'X'), - (0xA35, 'V'), - (0xA36, 'M', 'ਸ਼'), - (0xA37, 'X'), - (0xA38, 'V'), - (0xA3A, 'X'), - (0xA3C, 'V'), - (0xA3D, 'X'), - (0xA3E, 'V'), - (0xA43, 'X'), - (0xA47, 'V'), - (0xA49, 'X'), - (0xA4B, 'V'), - (0xA4E, 'X'), - (0xA51, 'V'), - (0xA52, 'X'), - (0xA59, 'M', 'ਖ਼'), - (0xA5A, 'M', 'ਗ਼'), - (0xA5B, 'M', 'ਜ਼'), - (0xA5C, 'V'), - (0xA5D, 'X'), - ] - -def _seg_11() -> List[Union[Tuple[int, str], Tuple[int, str, str]]]: - return [ - (0xA5E, 'M', 'ਫ਼'), - (0xA5F, 'X'), - (0xA66, 'V'), - (0xA77, 'X'), - (0xA81, 'V'), - (0xA84, 'X'), - (0xA85, 'V'), - (0xA8E, 'X'), - (0xA8F, 'V'), - (0xA92, 'X'), - (0xA93, 'V'), - (0xAA9, 'X'), - (0xAAA, 'V'), - (0xAB1, 'X'), - (0xAB2, 'V'), - (0xAB4, 'X'), - (0xAB5, 'V'), - (0xABA, 'X'), - (0xABC, 'V'), - (0xAC6, 'X'), - (0xAC7, 'V'), - (0xACA, 'X'), - (0xACB, 'V'), - (0xACE, 'X'), - (0xAD0, 'V'), - (0xAD1, 'X'), - (0xAE0, 'V'), - (0xAE4, 'X'), - (0xAE6, 'V'), - (0xAF2, 'X'), - (0xAF9, 'V'), - (0xB00, 'X'), - (0xB01, 'V'), - (0xB04, 'X'), - (0xB05, 'V'), - (0xB0D, 'X'), - (0xB0F, 'V'), - (0xB11, 'X'), - (0xB13, 'V'), - (0xB29, 'X'), - (0xB2A, 'V'), - (0xB31, 'X'), - (0xB32, 'V'), - (0xB34, 'X'), - (0xB35, 'V'), - (0xB3A, 'X'), - (0xB3C, 'V'), - (0xB45, 'X'), - (0xB47, 'V'), - (0xB49, 'X'), - (0xB4B, 'V'), - (0xB4E, 'X'), - (0xB55, 'V'), - (0xB58, 'X'), - (0xB5C, 'M', 'ଡ଼'), - (0xB5D, 'M', 'ଢ଼'), - (0xB5E, 'X'), - (0xB5F, 'V'), - (0xB64, 'X'), - (0xB66, 'V'), - (0xB78, 'X'), - (0xB82, 'V'), - (0xB84, 'X'), - (0xB85, 'V'), - (0xB8B, 'X'), - (0xB8E, 'V'), - (0xB91, 'X'), - (0xB92, 'V'), - (0xB96, 'X'), - (0xB99, 'V'), - (0xB9B, 'X'), - (0xB9C, 'V'), - (0xB9D, 'X'), - (0xB9E, 'V'), - (0xBA0, 'X'), - (0xBA3, 'V'), - (0xBA5, 'X'), - (0xBA8, 'V'), - (0xBAB, 'X'), - (0xBAE, 'V'), - (0xBBA, 'X'), - (0xBBE, 'V'), - (0xBC3, 'X'), - (0xBC6, 'V'), - (0xBC9, 'X'), - (0xBCA, 'V'), - (0xBCE, 'X'), - (0xBD0, 'V'), - (0xBD1, 'X'), - (0xBD7, 'V'), - (0xBD8, 'X'), - (0xBE6, 'V'), - (0xBFB, 'X'), - (0xC00, 'V'), - (0xC0D, 'X'), - (0xC0E, 'V'), - (0xC11, 'X'), - (0xC12, 'V'), - (0xC29, 'X'), - (0xC2A, 'V'), - ] - -def _seg_12() -> List[Union[Tuple[int, str], Tuple[int, str, str]]]: - return [ - (0xC3A, 'X'), - (0xC3C, 'V'), - (0xC45, 'X'), - (0xC46, 'V'), - (0xC49, 'X'), - (0xC4A, 'V'), - (0xC4E, 'X'), - (0xC55, 'V'), - (0xC57, 'X'), - (0xC58, 'V'), - (0xC5B, 'X'), - (0xC5D, 'V'), - (0xC5E, 'X'), - (0xC60, 'V'), - (0xC64, 'X'), - (0xC66, 'V'), - (0xC70, 'X'), - (0xC77, 'V'), - (0xC8D, 'X'), - (0xC8E, 'V'), - (0xC91, 'X'), - (0xC92, 'V'), - (0xCA9, 'X'), - (0xCAA, 'V'), - (0xCB4, 'X'), - (0xCB5, 'V'), - (0xCBA, 'X'), - (0xCBC, 'V'), - (0xCC5, 'X'), - (0xCC6, 'V'), - (0xCC9, 'X'), - (0xCCA, 'V'), - (0xCCE, 'X'), - (0xCD5, 'V'), - (0xCD7, 'X'), - (0xCDD, 'V'), - (0xCDF, 'X'), - (0xCE0, 'V'), - (0xCE4, 'X'), - (0xCE6, 'V'), - (0xCF0, 'X'), - (0xCF1, 'V'), - (0xCF4, 'X'), - (0xD00, 'V'), - (0xD0D, 'X'), - (0xD0E, 'V'), - (0xD11, 'X'), - (0xD12, 'V'), - (0xD45, 'X'), - (0xD46, 'V'), - (0xD49, 'X'), - (0xD4A, 'V'), - (0xD50, 'X'), - (0xD54, 'V'), - (0xD64, 'X'), - (0xD66, 'V'), - (0xD80, 'X'), - (0xD81, 'V'), - (0xD84, 'X'), - (0xD85, 'V'), - (0xD97, 'X'), - (0xD9A, 'V'), - (0xDB2, 'X'), - (0xDB3, 'V'), - (0xDBC, 'X'), - (0xDBD, 'V'), - (0xDBE, 'X'), - (0xDC0, 'V'), - (0xDC7, 'X'), - (0xDCA, 'V'), - (0xDCB, 'X'), - (0xDCF, 'V'), - (0xDD5, 'X'), - (0xDD6, 'V'), - (0xDD7, 'X'), - (0xDD8, 'V'), - (0xDE0, 'X'), - (0xDE6, 'V'), - (0xDF0, 'X'), - (0xDF2, 'V'), - (0xDF5, 'X'), - (0xE01, 'V'), - (0xE33, 'M', 'ํา'), - (0xE34, 'V'), - (0xE3B, 'X'), - (0xE3F, 'V'), - (0xE5C, 'X'), - (0xE81, 'V'), - (0xE83, 'X'), - (0xE84, 'V'), - (0xE85, 'X'), - (0xE86, 'V'), - (0xE8B, 'X'), - (0xE8C, 'V'), - (0xEA4, 'X'), - (0xEA5, 'V'), - (0xEA6, 'X'), - (0xEA7, 'V'), - (0xEB3, 'M', 'ໍາ'), - (0xEB4, 'V'), - ] - -def _seg_13() -> List[Union[Tuple[int, str], Tuple[int, str, str]]]: - return [ - (0xEBE, 'X'), - (0xEC0, 'V'), - (0xEC5, 'X'), - (0xEC6, 'V'), - (0xEC7, 'X'), - (0xEC8, 'V'), - (0xECF, 'X'), - (0xED0, 'V'), - (0xEDA, 'X'), - (0xEDC, 'M', 'ຫນ'), - (0xEDD, 'M', 'ຫມ'), - (0xEDE, 'V'), - (0xEE0, 'X'), - (0xF00, 'V'), - (0xF0C, 'M', '་'), - (0xF0D, 'V'), - (0xF43, 'M', 'གྷ'), - (0xF44, 'V'), - (0xF48, 'X'), - (0xF49, 'V'), - (0xF4D, 'M', 'ཌྷ'), - (0xF4E, 'V'), - (0xF52, 'M', 'དྷ'), - (0xF53, 'V'), - (0xF57, 'M', 'བྷ'), - (0xF58, 'V'), - (0xF5C, 'M', 'ཛྷ'), - (0xF5D, 'V'), - (0xF69, 'M', 'ཀྵ'), - (0xF6A, 'V'), - (0xF6D, 'X'), - (0xF71, 'V'), - (0xF73, 'M', 'ཱི'), - (0xF74, 'V'), - (0xF75, 'M', 'ཱུ'), - (0xF76, 'M', 'ྲྀ'), - (0xF77, 'M', 'ྲཱྀ'), - (0xF78, 'M', 'ླྀ'), - (0xF79, 'M', 'ླཱྀ'), - (0xF7A, 'V'), - (0xF81, 'M', 'ཱྀ'), - (0xF82, 'V'), - (0xF93, 'M', 'ྒྷ'), - (0xF94, 'V'), - (0xF98, 'X'), - (0xF99, 'V'), - (0xF9D, 'M', 'ྜྷ'), - (0xF9E, 'V'), - (0xFA2, 'M', 'ྡྷ'), - (0xFA3, 'V'), - (0xFA7, 'M', 'ྦྷ'), - (0xFA8, 'V'), - (0xFAC, 'M', 'ྫྷ'), - (0xFAD, 'V'), - (0xFB9, 'M', 'ྐྵ'), - (0xFBA, 'V'), - (0xFBD, 'X'), - (0xFBE, 'V'), - (0xFCD, 'X'), - (0xFCE, 'V'), - (0xFDB, 'X'), - (0x1000, 'V'), - (0x10A0, 'X'), - (0x10C7, 'M', 'ⴧ'), - (0x10C8, 'X'), - (0x10CD, 'M', 'ⴭ'), - (0x10CE, 'X'), - (0x10D0, 'V'), - (0x10FC, 'M', 'ნ'), - (0x10FD, 'V'), - (0x115F, 'X'), - (0x1161, 'V'), - (0x1249, 'X'), - (0x124A, 'V'), - (0x124E, 'X'), - (0x1250, 'V'), - (0x1257, 'X'), - (0x1258, 'V'), - (0x1259, 'X'), - (0x125A, 'V'), - (0x125E, 'X'), - (0x1260, 'V'), - (0x1289, 'X'), - (0x128A, 'V'), - (0x128E, 'X'), - (0x1290, 'V'), - (0x12B1, 'X'), - (0x12B2, 'V'), - (0x12B6, 'X'), - (0x12B8, 'V'), - (0x12BF, 'X'), - (0x12C0, 'V'), - (0x12C1, 'X'), - (0x12C2, 'V'), - (0x12C6, 'X'), - (0x12C8, 'V'), - (0x12D7, 'X'), - (0x12D8, 'V'), - (0x1311, 'X'), - (0x1312, 'V'), - ] - -def _seg_14() -> List[Union[Tuple[int, str], Tuple[int, str, str]]]: - return [ - (0x1316, 'X'), - (0x1318, 'V'), - (0x135B, 'X'), - (0x135D, 'V'), - (0x137D, 'X'), - (0x1380, 'V'), - (0x139A, 'X'), - (0x13A0, 'V'), - (0x13F6, 'X'), - (0x13F8, 'M', 'Ᏸ'), - (0x13F9, 'M', 'Ᏹ'), - (0x13FA, 'M', 'Ᏺ'), - (0x13FB, 'M', 'Ᏻ'), - (0x13FC, 'M', 'Ᏼ'), - (0x13FD, 'M', 'Ᏽ'), - (0x13FE, 'X'), - (0x1400, 'V'), - (0x1680, 'X'), - (0x1681, 'V'), - (0x169D, 'X'), - (0x16A0, 'V'), - (0x16F9, 'X'), - (0x1700, 'V'), - (0x1716, 'X'), - (0x171F, 'V'), - (0x1737, 'X'), - (0x1740, 'V'), - (0x1754, 'X'), - (0x1760, 'V'), - (0x176D, 'X'), - (0x176E, 'V'), - (0x1771, 'X'), - (0x1772, 'V'), - (0x1774, 'X'), - (0x1780, 'V'), - (0x17B4, 'X'), - (0x17B6, 'V'), - (0x17DE, 'X'), - (0x17E0, 'V'), - (0x17EA, 'X'), - (0x17F0, 'V'), - (0x17FA, 'X'), - (0x1800, 'V'), - (0x1806, 'X'), - (0x1807, 'V'), - (0x180B, 'I'), - (0x180E, 'X'), - (0x180F, 'I'), - (0x1810, 'V'), - (0x181A, 'X'), - (0x1820, 'V'), - (0x1879, 'X'), - (0x1880, 'V'), - (0x18AB, 'X'), - (0x18B0, 'V'), - (0x18F6, 'X'), - (0x1900, 'V'), - (0x191F, 'X'), - (0x1920, 'V'), - (0x192C, 'X'), - (0x1930, 'V'), - (0x193C, 'X'), - (0x1940, 'V'), - (0x1941, 'X'), - (0x1944, 'V'), - (0x196E, 'X'), - (0x1970, 'V'), - (0x1975, 'X'), - (0x1980, 'V'), - (0x19AC, 'X'), - (0x19B0, 'V'), - (0x19CA, 'X'), - (0x19D0, 'V'), - (0x19DB, 'X'), - (0x19DE, 'V'), - (0x1A1C, 'X'), - (0x1A1E, 'V'), - (0x1A5F, 'X'), - (0x1A60, 'V'), - (0x1A7D, 'X'), - (0x1A7F, 'V'), - (0x1A8A, 'X'), - (0x1A90, 'V'), - (0x1A9A, 'X'), - (0x1AA0, 'V'), - (0x1AAE, 'X'), - (0x1AB0, 'V'), - (0x1ACF, 'X'), - (0x1B00, 'V'), - (0x1B4D, 'X'), - (0x1B50, 'V'), - (0x1B7F, 'X'), - (0x1B80, 'V'), - (0x1BF4, 'X'), - (0x1BFC, 'V'), - (0x1C38, 'X'), - (0x1C3B, 'V'), - (0x1C4A, 'X'), - (0x1C4D, 'V'), - (0x1C80, 'M', 'в'), - ] - -def _seg_15() -> List[Union[Tuple[int, str], Tuple[int, str, str]]]: - return [ - (0x1C81, 'M', 'д'), - (0x1C82, 'M', 'о'), - (0x1C83, 'M', 'с'), - (0x1C84, 'M', 'т'), - (0x1C86, 'M', 'ъ'), - (0x1C87, 'M', 'ѣ'), - (0x1C88, 'M', 'ꙋ'), - (0x1C89, 'X'), - (0x1C90, 'M', 'ა'), - (0x1C91, 'M', 'ბ'), - (0x1C92, 'M', 'გ'), - (0x1C93, 'M', 'დ'), - (0x1C94, 'M', 'ე'), - (0x1C95, 'M', 'ვ'), - (0x1C96, 'M', 'ზ'), - (0x1C97, 'M', 'თ'), - (0x1C98, 'M', 'ი'), - (0x1C99, 'M', 'კ'), - (0x1C9A, 'M', 'ლ'), - (0x1C9B, 'M', 'მ'), - (0x1C9C, 'M', 'ნ'), - (0x1C9D, 'M', 'ო'), - (0x1C9E, 'M', 'პ'), - (0x1C9F, 'M', 'ჟ'), - (0x1CA0, 'M', 'რ'), - (0x1CA1, 'M', 'ს'), - (0x1CA2, 'M', 'ტ'), - (0x1CA3, 'M', 'უ'), - (0x1CA4, 'M', 'ფ'), - (0x1CA5, 'M', 'ქ'), - (0x1CA6, 'M', 'ღ'), - (0x1CA7, 'M', 'ყ'), - (0x1CA8, 'M', 'შ'), - (0x1CA9, 'M', 'ჩ'), - (0x1CAA, 'M', 'ც'), - (0x1CAB, 'M', 'ძ'), - (0x1CAC, 'M', 'წ'), - (0x1CAD, 'M', 'ჭ'), - (0x1CAE, 'M', 'ხ'), - (0x1CAF, 'M', 'ჯ'), - (0x1CB0, 'M', 'ჰ'), - (0x1CB1, 'M', 'ჱ'), - (0x1CB2, 'M', 'ჲ'), - (0x1CB3, 'M', 'ჳ'), - (0x1CB4, 'M', 'ჴ'), - (0x1CB5, 'M', 'ჵ'), - (0x1CB6, 'M', 'ჶ'), - (0x1CB7, 'M', 'ჷ'), - (0x1CB8, 'M', 'ჸ'), - (0x1CB9, 'M', 'ჹ'), - (0x1CBA, 'M', 'ჺ'), - (0x1CBB, 'X'), - (0x1CBD, 'M', 'ჽ'), - (0x1CBE, 'M', 'ჾ'), - (0x1CBF, 'M', 'ჿ'), - (0x1CC0, 'V'), - (0x1CC8, 'X'), - (0x1CD0, 'V'), - (0x1CFB, 'X'), - (0x1D00, 'V'), - (0x1D2C, 'M', 'a'), - (0x1D2D, 'M', 'æ'), - (0x1D2E, 'M', 'b'), - (0x1D2F, 'V'), - (0x1D30, 'M', 'd'), - (0x1D31, 'M', 'e'), - (0x1D32, 'M', 'ǝ'), - (0x1D33, 'M', 'g'), - (0x1D34, 'M', 'h'), - (0x1D35, 'M', 'i'), - (0x1D36, 'M', 'j'), - (0x1D37, 'M', 'k'), - (0x1D38, 'M', 'l'), - (0x1D39, 'M', 'm'), - (0x1D3A, 'M', 'n'), - (0x1D3B, 'V'), - (0x1D3C, 'M', 'o'), - (0x1D3D, 'M', 'ȣ'), - (0x1D3E, 'M', 'p'), - (0x1D3F, 'M', 'r'), - (0x1D40, 'M', 't'), - (0x1D41, 'M', 'u'), - (0x1D42, 'M', 'w'), - (0x1D43, 'M', 'a'), - (0x1D44, 'M', 'ɐ'), - (0x1D45, 'M', 'ɑ'), - (0x1D46, 'M', 'ᴂ'), - (0x1D47, 'M', 'b'), - (0x1D48, 'M', 'd'), - (0x1D49, 'M', 'e'), - (0x1D4A, 'M', 'ə'), - (0x1D4B, 'M', 'ɛ'), - (0x1D4C, 'M', 'ɜ'), - (0x1D4D, 'M', 'g'), - (0x1D4E, 'V'), - (0x1D4F, 'M', 'k'), - (0x1D50, 'M', 'm'), - (0x1D51, 'M', 'ŋ'), - (0x1D52, 'M', 'o'), - (0x1D53, 'M', 'ɔ'), - ] - -def _seg_16() -> List[Union[Tuple[int, str], Tuple[int, str, str]]]: - return [ - (0x1D54, 'M', 'ᴖ'), - (0x1D55, 'M', 'ᴗ'), - (0x1D56, 'M', 'p'), - (0x1D57, 'M', 't'), - (0x1D58, 'M', 'u'), - (0x1D59, 'M', 'ᴝ'), - (0x1D5A, 'M', 'ɯ'), - (0x1D5B, 'M', 'v'), - (0x1D5C, 'M', 'ᴥ'), - (0x1D5D, 'M', 'β'), - (0x1D5E, 'M', 'γ'), - (0x1D5F, 'M', 'δ'), - (0x1D60, 'M', 'φ'), - (0x1D61, 'M', 'χ'), - (0x1D62, 'M', 'i'), - (0x1D63, 'M', 'r'), - (0x1D64, 'M', 'u'), - (0x1D65, 'M', 'v'), - (0x1D66, 'M', 'β'), - (0x1D67, 'M', 'γ'), - (0x1D68, 'M', 'ρ'), - (0x1D69, 'M', 'φ'), - (0x1D6A, 'M', 'χ'), - (0x1D6B, 'V'), - (0x1D78, 'M', 'н'), - (0x1D79, 'V'), - (0x1D9B, 'M', 'ɒ'), - (0x1D9C, 'M', 'c'), - (0x1D9D, 'M', 'ɕ'), - (0x1D9E, 'M', 'ð'), - (0x1D9F, 'M', 'ɜ'), - (0x1DA0, 'M', 'f'), - (0x1DA1, 'M', 'ɟ'), - (0x1DA2, 'M', 'ɡ'), - (0x1DA3, 'M', 'ɥ'), - (0x1DA4, 'M', 'ɨ'), - (0x1DA5, 'M', 'ɩ'), - (0x1DA6, 'M', 'ɪ'), - (0x1DA7, 'M', 'ᵻ'), - (0x1DA8, 'M', 'ʝ'), - (0x1DA9, 'M', 'ɭ'), - (0x1DAA, 'M', 'ᶅ'), - (0x1DAB, 'M', 'ʟ'), - (0x1DAC, 'M', 'ɱ'), - (0x1DAD, 'M', 'ɰ'), - (0x1DAE, 'M', 'ɲ'), - (0x1DAF, 'M', 'ɳ'), - (0x1DB0, 'M', 'ɴ'), - (0x1DB1, 'M', 'ɵ'), - (0x1DB2, 'M', 'ɸ'), - (0x1DB3, 'M', 'ʂ'), - (0x1DB4, 'M', 'ʃ'), - (0x1DB5, 'M', 'ƫ'), - (0x1DB6, 'M', 'ʉ'), - (0x1DB7, 'M', 'ʊ'), - (0x1DB8, 'M', 'ᴜ'), - (0x1DB9, 'M', 'ʋ'), - (0x1DBA, 'M', 'ʌ'), - (0x1DBB, 'M', 'z'), - (0x1DBC, 'M', 'ʐ'), - (0x1DBD, 'M', 'ʑ'), - (0x1DBE, 'M', 'ʒ'), - (0x1DBF, 'M', 'θ'), - (0x1DC0, 'V'), - (0x1E00, 'M', 'ḁ'), - (0x1E01, 'V'), - (0x1E02, 'M', 'ḃ'), - (0x1E03, 'V'), - (0x1E04, 'M', 'ḅ'), - (0x1E05, 'V'), - (0x1E06, 'M', 'ḇ'), - (0x1E07, 'V'), - (0x1E08, 'M', 'ḉ'), - (0x1E09, 'V'), - (0x1E0A, 'M', 'ḋ'), - (0x1E0B, 'V'), - (0x1E0C, 'M', 'ḍ'), - (0x1E0D, 'V'), - (0x1E0E, 'M', 'ḏ'), - (0x1E0F, 'V'), - (0x1E10, 'M', 'ḑ'), - (0x1E11, 'V'), - (0x1E12, 'M', 'ḓ'), - (0x1E13, 'V'), - (0x1E14, 'M', 'ḕ'), - (0x1E15, 'V'), - (0x1E16, 'M', 'ḗ'), - (0x1E17, 'V'), - (0x1E18, 'M', 'ḙ'), - (0x1E19, 'V'), - (0x1E1A, 'M', 'ḛ'), - (0x1E1B, 'V'), - (0x1E1C, 'M', 'ḝ'), - (0x1E1D, 'V'), - (0x1E1E, 'M', 'ḟ'), - (0x1E1F, 'V'), - (0x1E20, 'M', 'ḡ'), - (0x1E21, 'V'), - (0x1E22, 'M', 'ḣ'), - (0x1E23, 'V'), - ] - -def _seg_17() -> List[Union[Tuple[int, str], Tuple[int, str, str]]]: - return [ - (0x1E24, 'M', 'ḥ'), - (0x1E25, 'V'), - (0x1E26, 'M', 'ḧ'), - (0x1E27, 'V'), - (0x1E28, 'M', 'ḩ'), - (0x1E29, 'V'), - (0x1E2A, 'M', 'ḫ'), - (0x1E2B, 'V'), - (0x1E2C, 'M', 'ḭ'), - (0x1E2D, 'V'), - (0x1E2E, 'M', 'ḯ'), - (0x1E2F, 'V'), - (0x1E30, 'M', 'ḱ'), - (0x1E31, 'V'), - (0x1E32, 'M', 'ḳ'), - (0x1E33, 'V'), - (0x1E34, 'M', 'ḵ'), - (0x1E35, 'V'), - (0x1E36, 'M', 'ḷ'), - (0x1E37, 'V'), - (0x1E38, 'M', 'ḹ'), - (0x1E39, 'V'), - (0x1E3A, 'M', 'ḻ'), - (0x1E3B, 'V'), - (0x1E3C, 'M', 'ḽ'), - (0x1E3D, 'V'), - (0x1E3E, 'M', 'ḿ'), - (0x1E3F, 'V'), - (0x1E40, 'M', 'ṁ'), - (0x1E41, 'V'), - (0x1E42, 'M', 'ṃ'), - (0x1E43, 'V'), - (0x1E44, 'M', 'ṅ'), - (0x1E45, 'V'), - (0x1E46, 'M', 'ṇ'), - (0x1E47, 'V'), - (0x1E48, 'M', 'ṉ'), - (0x1E49, 'V'), - (0x1E4A, 'M', 'ṋ'), - (0x1E4B, 'V'), - (0x1E4C, 'M', 'ṍ'), - (0x1E4D, 'V'), - (0x1E4E, 'M', 'ṏ'), - (0x1E4F, 'V'), - (0x1E50, 'M', 'ṑ'), - (0x1E51, 'V'), - (0x1E52, 'M', 'ṓ'), - (0x1E53, 'V'), - (0x1E54, 'M', 'ṕ'), - (0x1E55, 'V'), - (0x1E56, 'M', 'ṗ'), - (0x1E57, 'V'), - (0x1E58, 'M', 'ṙ'), - (0x1E59, 'V'), - (0x1E5A, 'M', 'ṛ'), - (0x1E5B, 'V'), - (0x1E5C, 'M', 'ṝ'), - (0x1E5D, 'V'), - (0x1E5E, 'M', 'ṟ'), - (0x1E5F, 'V'), - (0x1E60, 'M', 'ṡ'), - (0x1E61, 'V'), - (0x1E62, 'M', 'ṣ'), - (0x1E63, 'V'), - (0x1E64, 'M', 'ṥ'), - (0x1E65, 'V'), - (0x1E66, 'M', 'ṧ'), - (0x1E67, 'V'), - (0x1E68, 'M', 'ṩ'), - (0x1E69, 'V'), - (0x1E6A, 'M', 'ṫ'), - (0x1E6B, 'V'), - (0x1E6C, 'M', 'ṭ'), - (0x1E6D, 'V'), - (0x1E6E, 'M', 'ṯ'), - (0x1E6F, 'V'), - (0x1E70, 'M', 'ṱ'), - (0x1E71, 'V'), - (0x1E72, 'M', 'ṳ'), - (0x1E73, 'V'), - (0x1E74, 'M', 'ṵ'), - (0x1E75, 'V'), - (0x1E76, 'M', 'ṷ'), - (0x1E77, 'V'), - (0x1E78, 'M', 'ṹ'), - (0x1E79, 'V'), - (0x1E7A, 'M', 'ṻ'), - (0x1E7B, 'V'), - (0x1E7C, 'M', 'ṽ'), - (0x1E7D, 'V'), - (0x1E7E, 'M', 'ṿ'), - (0x1E7F, 'V'), - (0x1E80, 'M', 'ẁ'), - (0x1E81, 'V'), - (0x1E82, 'M', 'ẃ'), - (0x1E83, 'V'), - (0x1E84, 'M', 'ẅ'), - (0x1E85, 'V'), - (0x1E86, 'M', 'ẇ'), - (0x1E87, 'V'), - ] - -def _seg_18() -> List[Union[Tuple[int, str], Tuple[int, str, str]]]: - return [ - (0x1E88, 'M', 'ẉ'), - (0x1E89, 'V'), - (0x1E8A, 'M', 'ẋ'), - (0x1E8B, 'V'), - (0x1E8C, 'M', 'ẍ'), - (0x1E8D, 'V'), - (0x1E8E, 'M', 'ẏ'), - (0x1E8F, 'V'), - (0x1E90, 'M', 'ẑ'), - (0x1E91, 'V'), - (0x1E92, 'M', 'ẓ'), - (0x1E93, 'V'), - (0x1E94, 'M', 'ẕ'), - (0x1E95, 'V'), - (0x1E9A, 'M', 'aʾ'), - (0x1E9B, 'M', 'ṡ'), - (0x1E9C, 'V'), - (0x1E9E, 'M', 'ss'), - (0x1E9F, 'V'), - (0x1EA0, 'M', 'ạ'), - (0x1EA1, 'V'), - (0x1EA2, 'M', 'ả'), - (0x1EA3, 'V'), - (0x1EA4, 'M', 'ấ'), - (0x1EA5, 'V'), - (0x1EA6, 'M', 'ầ'), - (0x1EA7, 'V'), - (0x1EA8, 'M', 'ẩ'), - (0x1EA9, 'V'), - (0x1EAA, 'M', 'ẫ'), - (0x1EAB, 'V'), - (0x1EAC, 'M', 'ậ'), - (0x1EAD, 'V'), - (0x1EAE, 'M', 'ắ'), - (0x1EAF, 'V'), - (0x1EB0, 'M', 'ằ'), - (0x1EB1, 'V'), - (0x1EB2, 'M', 'ẳ'), - (0x1EB3, 'V'), - (0x1EB4, 'M', 'ẵ'), - (0x1EB5, 'V'), - (0x1EB6, 'M', 'ặ'), - (0x1EB7, 'V'), - (0x1EB8, 'M', 'ẹ'), - (0x1EB9, 'V'), - (0x1EBA, 'M', 'ẻ'), - (0x1EBB, 'V'), - (0x1EBC, 'M', 'ẽ'), - (0x1EBD, 'V'), - (0x1EBE, 'M', 'ế'), - (0x1EBF, 'V'), - (0x1EC0, 'M', 'ề'), - (0x1EC1, 'V'), - (0x1EC2, 'M', 'ể'), - (0x1EC3, 'V'), - (0x1EC4, 'M', 'ễ'), - (0x1EC5, 'V'), - (0x1EC6, 'M', 'ệ'), - (0x1EC7, 'V'), - (0x1EC8, 'M', 'ỉ'), - (0x1EC9, 'V'), - (0x1ECA, 'M', 'ị'), - (0x1ECB, 'V'), - (0x1ECC, 'M', 'ọ'), - (0x1ECD, 'V'), - (0x1ECE, 'M', 'ỏ'), - (0x1ECF, 'V'), - (0x1ED0, 'M', 'ố'), - (0x1ED1, 'V'), - (0x1ED2, 'M', 'ồ'), - (0x1ED3, 'V'), - (0x1ED4, 'M', 'ổ'), - (0x1ED5, 'V'), - (0x1ED6, 'M', 'ỗ'), - (0x1ED7, 'V'), - (0x1ED8, 'M', 'ộ'), - (0x1ED9, 'V'), - (0x1EDA, 'M', 'ớ'), - (0x1EDB, 'V'), - (0x1EDC, 'M', 'ờ'), - (0x1EDD, 'V'), - (0x1EDE, 'M', 'ở'), - (0x1EDF, 'V'), - (0x1EE0, 'M', 'ỡ'), - (0x1EE1, 'V'), - (0x1EE2, 'M', 'ợ'), - (0x1EE3, 'V'), - (0x1EE4, 'M', 'ụ'), - (0x1EE5, 'V'), - (0x1EE6, 'M', 'ủ'), - (0x1EE7, 'V'), - (0x1EE8, 'M', 'ứ'), - (0x1EE9, 'V'), - (0x1EEA, 'M', 'ừ'), - (0x1EEB, 'V'), - (0x1EEC, 'M', 'ử'), - (0x1EED, 'V'), - (0x1EEE, 'M', 'ữ'), - (0x1EEF, 'V'), - (0x1EF0, 'M', 'ự'), - ] - -def _seg_19() -> List[Union[Tuple[int, str], Tuple[int, str, str]]]: - return [ - (0x1EF1, 'V'), - (0x1EF2, 'M', 'ỳ'), - (0x1EF3, 'V'), - (0x1EF4, 'M', 'ỵ'), - (0x1EF5, 'V'), - (0x1EF6, 'M', 'ỷ'), - (0x1EF7, 'V'), - (0x1EF8, 'M', 'ỹ'), - (0x1EF9, 'V'), - (0x1EFA, 'M', 'ỻ'), - (0x1EFB, 'V'), - (0x1EFC, 'M', 'ỽ'), - (0x1EFD, 'V'), - (0x1EFE, 'M', 'ỿ'), - (0x1EFF, 'V'), - (0x1F08, 'M', 'ἀ'), - (0x1F09, 'M', 'ἁ'), - (0x1F0A, 'M', 'ἂ'), - (0x1F0B, 'M', 'ἃ'), - (0x1F0C, 'M', 'ἄ'), - (0x1F0D, 'M', 'ἅ'), - (0x1F0E, 'M', 'ἆ'), - (0x1F0F, 'M', 'ἇ'), - (0x1F10, 'V'), - (0x1F16, 'X'), - (0x1F18, 'M', 'ἐ'), - (0x1F19, 'M', 'ἑ'), - (0x1F1A, 'M', 'ἒ'), - (0x1F1B, 'M', 'ἓ'), - (0x1F1C, 'M', 'ἔ'), - (0x1F1D, 'M', 'ἕ'), - (0x1F1E, 'X'), - (0x1F20, 'V'), - (0x1F28, 'M', 'ἠ'), - (0x1F29, 'M', 'ἡ'), - (0x1F2A, 'M', 'ἢ'), - (0x1F2B, 'M', 'ἣ'), - (0x1F2C, 'M', 'ἤ'), - (0x1F2D, 'M', 'ἥ'), - (0x1F2E, 'M', 'ἦ'), - (0x1F2F, 'M', 'ἧ'), - (0x1F30, 'V'), - (0x1F38, 'M', 'ἰ'), - (0x1F39, 'M', 'ἱ'), - (0x1F3A, 'M', 'ἲ'), - (0x1F3B, 'M', 'ἳ'), - (0x1F3C, 'M', 'ἴ'), - (0x1F3D, 'M', 'ἵ'), - (0x1F3E, 'M', 'ἶ'), - (0x1F3F, 'M', 'ἷ'), - (0x1F40, 'V'), - (0x1F46, 'X'), - (0x1F48, 'M', 'ὀ'), - (0x1F49, 'M', 'ὁ'), - (0x1F4A, 'M', 'ὂ'), - (0x1F4B, 'M', 'ὃ'), - (0x1F4C, 'M', 'ὄ'), - (0x1F4D, 'M', 'ὅ'), - (0x1F4E, 'X'), - (0x1F50, 'V'), - (0x1F58, 'X'), - (0x1F59, 'M', 'ὑ'), - (0x1F5A, 'X'), - (0x1F5B, 'M', 'ὓ'), - (0x1F5C, 'X'), - (0x1F5D, 'M', 'ὕ'), - (0x1F5E, 'X'), - (0x1F5F, 'M', 'ὗ'), - (0x1F60, 'V'), - (0x1F68, 'M', 'ὠ'), - (0x1F69, 'M', 'ὡ'), - (0x1F6A, 'M', 'ὢ'), - (0x1F6B, 'M', 'ὣ'), - (0x1F6C, 'M', 'ὤ'), - (0x1F6D, 'M', 'ὥ'), - (0x1F6E, 'M', 'ὦ'), - (0x1F6F, 'M', 'ὧ'), - (0x1F70, 'V'), - (0x1F71, 'M', 'ά'), - (0x1F72, 'V'), - (0x1F73, 'M', 'έ'), - (0x1F74, 'V'), - (0x1F75, 'M', 'ή'), - (0x1F76, 'V'), - (0x1F77, 'M', 'ί'), - (0x1F78, 'V'), - (0x1F79, 'M', 'ό'), - (0x1F7A, 'V'), - (0x1F7B, 'M', 'ύ'), - (0x1F7C, 'V'), - (0x1F7D, 'M', 'ώ'), - (0x1F7E, 'X'), - (0x1F80, 'M', 'ἀι'), - (0x1F81, 'M', 'ἁι'), - (0x1F82, 'M', 'ἂι'), - (0x1F83, 'M', 'ἃι'), - (0x1F84, 'M', 'ἄι'), - (0x1F85, 'M', 'ἅι'), - (0x1F86, 'M', 'ἆι'), - (0x1F87, 'M', 'ἇι'), - ] - -def _seg_20() -> List[Union[Tuple[int, str], Tuple[int, str, str]]]: - return [ - (0x1F88, 'M', 'ἀι'), - (0x1F89, 'M', 'ἁι'), - (0x1F8A, 'M', 'ἂι'), - (0x1F8B, 'M', 'ἃι'), - (0x1F8C, 'M', 'ἄι'), - (0x1F8D, 'M', 'ἅι'), - (0x1F8E, 'M', 'ἆι'), - (0x1F8F, 'M', 'ἇι'), - (0x1F90, 'M', 'ἠι'), - (0x1F91, 'M', 'ἡι'), - (0x1F92, 'M', 'ἢι'), - (0x1F93, 'M', 'ἣι'), - (0x1F94, 'M', 'ἤι'), - (0x1F95, 'M', 'ἥι'), - (0x1F96, 'M', 'ἦι'), - (0x1F97, 'M', 'ἧι'), - (0x1F98, 'M', 'ἠι'), - (0x1F99, 'M', 'ἡι'), - (0x1F9A, 'M', 'ἢι'), - (0x1F9B, 'M', 'ἣι'), - (0x1F9C, 'M', 'ἤι'), - (0x1F9D, 'M', 'ἥι'), - (0x1F9E, 'M', 'ἦι'), - (0x1F9F, 'M', 'ἧι'), - (0x1FA0, 'M', 'ὠι'), - (0x1FA1, 'M', 'ὡι'), - (0x1FA2, 'M', 'ὢι'), - (0x1FA3, 'M', 'ὣι'), - (0x1FA4, 'M', 'ὤι'), - (0x1FA5, 'M', 'ὥι'), - (0x1FA6, 'M', 'ὦι'), - (0x1FA7, 'M', 'ὧι'), - (0x1FA8, 'M', 'ὠι'), - (0x1FA9, 'M', 'ὡι'), - (0x1FAA, 'M', 'ὢι'), - (0x1FAB, 'M', 'ὣι'), - (0x1FAC, 'M', 'ὤι'), - (0x1FAD, 'M', 'ὥι'), - (0x1FAE, 'M', 'ὦι'), - (0x1FAF, 'M', 'ὧι'), - (0x1FB0, 'V'), - (0x1FB2, 'M', 'ὰι'), - (0x1FB3, 'M', 'αι'), - (0x1FB4, 'M', 'άι'), - (0x1FB5, 'X'), - (0x1FB6, 'V'), - (0x1FB7, 'M', 'ᾶι'), - (0x1FB8, 'M', 'ᾰ'), - (0x1FB9, 'M', 'ᾱ'), - (0x1FBA, 'M', 'ὰ'), - (0x1FBB, 'M', 'ά'), - (0x1FBC, 'M', 'αι'), - (0x1FBD, '3', ' ̓'), - (0x1FBE, 'M', 'ι'), - (0x1FBF, '3', ' ̓'), - (0x1FC0, '3', ' ͂'), - (0x1FC1, '3', ' ̈͂'), - (0x1FC2, 'M', 'ὴι'), - (0x1FC3, 'M', 'ηι'), - (0x1FC4, 'M', 'ήι'), - (0x1FC5, 'X'), - (0x1FC6, 'V'), - (0x1FC7, 'M', 'ῆι'), - (0x1FC8, 'M', 'ὲ'), - (0x1FC9, 'M', 'έ'), - (0x1FCA, 'M', 'ὴ'), - (0x1FCB, 'M', 'ή'), - (0x1FCC, 'M', 'ηι'), - (0x1FCD, '3', ' ̓̀'), - (0x1FCE, '3', ' ̓́'), - (0x1FCF, '3', ' ̓͂'), - (0x1FD0, 'V'), - (0x1FD3, 'M', 'ΐ'), - (0x1FD4, 'X'), - (0x1FD6, 'V'), - (0x1FD8, 'M', 'ῐ'), - (0x1FD9, 'M', 'ῑ'), - (0x1FDA, 'M', 'ὶ'), - (0x1FDB, 'M', 'ί'), - (0x1FDC, 'X'), - (0x1FDD, '3', ' ̔̀'), - (0x1FDE, '3', ' ̔́'), - (0x1FDF, '3', ' ̔͂'), - (0x1FE0, 'V'), - (0x1FE3, 'M', 'ΰ'), - (0x1FE4, 'V'), - (0x1FE8, 'M', 'ῠ'), - (0x1FE9, 'M', 'ῡ'), - (0x1FEA, 'M', 'ὺ'), - (0x1FEB, 'M', 'ύ'), - (0x1FEC, 'M', 'ῥ'), - (0x1FED, '3', ' ̈̀'), - (0x1FEE, '3', ' ̈́'), - (0x1FEF, '3', '`'), - (0x1FF0, 'X'), - (0x1FF2, 'M', 'ὼι'), - (0x1FF3, 'M', 'ωι'), - (0x1FF4, 'M', 'ώι'), - (0x1FF5, 'X'), - (0x1FF6, 'V'), - ] - -def _seg_21() -> List[Union[Tuple[int, str], Tuple[int, str, str]]]: - return [ - (0x1FF7, 'M', 'ῶι'), - (0x1FF8, 'M', 'ὸ'), - (0x1FF9, 'M', 'ό'), - (0x1FFA, 'M', 'ὼ'), - (0x1FFB, 'M', 'ώ'), - (0x1FFC, 'M', 'ωι'), - (0x1FFD, '3', ' ́'), - (0x1FFE, '3', ' ̔'), - (0x1FFF, 'X'), - (0x2000, '3', ' '), - (0x200B, 'I'), - (0x200C, 'D', ''), - (0x200E, 'X'), - (0x2010, 'V'), - (0x2011, 'M', '‐'), - (0x2012, 'V'), - (0x2017, '3', ' ̳'), - (0x2018, 'V'), - (0x2024, 'X'), - (0x2027, 'V'), - (0x2028, 'X'), - (0x202F, '3', ' '), - (0x2030, 'V'), - (0x2033, 'M', '′′'), - (0x2034, 'M', '′′′'), - (0x2035, 'V'), - (0x2036, 'M', '‵‵'), - (0x2037, 'M', '‵‵‵'), - (0x2038, 'V'), - (0x203C, '3', '!!'), - (0x203D, 'V'), - (0x203E, '3', ' ̅'), - (0x203F, 'V'), - (0x2047, '3', '??'), - (0x2048, '3', '?!'), - (0x2049, '3', '!?'), - (0x204A, 'V'), - (0x2057, 'M', '′′′′'), - (0x2058, 'V'), - (0x205F, '3', ' '), - (0x2060, 'I'), - (0x2061, 'X'), - (0x2064, 'I'), - (0x2065, 'X'), - (0x2070, 'M', '0'), - (0x2071, 'M', 'i'), - (0x2072, 'X'), - (0x2074, 'M', '4'), - (0x2075, 'M', '5'), - (0x2076, 'M', '6'), - (0x2077, 'M', '7'), - (0x2078, 'M', '8'), - (0x2079, 'M', '9'), - (0x207A, '3', '+'), - (0x207B, 'M', '−'), - (0x207C, '3', '='), - (0x207D, '3', '('), - (0x207E, '3', ')'), - (0x207F, 'M', 'n'), - (0x2080, 'M', '0'), - (0x2081, 'M', '1'), - (0x2082, 'M', '2'), - (0x2083, 'M', '3'), - (0x2084, 'M', '4'), - (0x2085, 'M', '5'), - (0x2086, 'M', '6'), - (0x2087, 'M', '7'), - (0x2088, 'M', '8'), - (0x2089, 'M', '9'), - (0x208A, '3', '+'), - (0x208B, 'M', '−'), - (0x208C, '3', '='), - (0x208D, '3', '('), - (0x208E, '3', ')'), - (0x208F, 'X'), - (0x2090, 'M', 'a'), - (0x2091, 'M', 'e'), - (0x2092, 'M', 'o'), - (0x2093, 'M', 'x'), - (0x2094, 'M', 'ə'), - (0x2095, 'M', 'h'), - (0x2096, 'M', 'k'), - (0x2097, 'M', 'l'), - (0x2098, 'M', 'm'), - (0x2099, 'M', 'n'), - (0x209A, 'M', 'p'), - (0x209B, 'M', 's'), - (0x209C, 'M', 't'), - (0x209D, 'X'), - (0x20A0, 'V'), - (0x20A8, 'M', 'rs'), - (0x20A9, 'V'), - (0x20C1, 'X'), - (0x20D0, 'V'), - (0x20F1, 'X'), - (0x2100, '3', 'a/c'), - (0x2101, '3', 'a/s'), - (0x2102, 'M', 'c'), - (0x2103, 'M', '°c'), - (0x2104, 'V'), - ] - -def _seg_22() -> List[Union[Tuple[int, str], Tuple[int, str, str]]]: - return [ - (0x2105, '3', 'c/o'), - (0x2106, '3', 'c/u'), - (0x2107, 'M', 'ɛ'), - (0x2108, 'V'), - (0x2109, 'M', '°f'), - (0x210A, 'M', 'g'), - (0x210B, 'M', 'h'), - (0x210F, 'M', 'ħ'), - (0x2110, 'M', 'i'), - (0x2112, 'M', 'l'), - (0x2114, 'V'), - (0x2115, 'M', 'n'), - (0x2116, 'M', 'no'), - (0x2117, 'V'), - (0x2119, 'M', 'p'), - (0x211A, 'M', 'q'), - (0x211B, 'M', 'r'), - (0x211E, 'V'), - (0x2120, 'M', 'sm'), - (0x2121, 'M', 'tel'), - (0x2122, 'M', 'tm'), - (0x2123, 'V'), - (0x2124, 'M', 'z'), - (0x2125, 'V'), - (0x2126, 'M', 'ω'), - (0x2127, 'V'), - (0x2128, 'M', 'z'), - (0x2129, 'V'), - (0x212A, 'M', 'k'), - (0x212B, 'M', 'å'), - (0x212C, 'M', 'b'), - (0x212D, 'M', 'c'), - (0x212E, 'V'), - (0x212F, 'M', 'e'), - (0x2131, 'M', 'f'), - (0x2132, 'X'), - (0x2133, 'M', 'm'), - (0x2134, 'M', 'o'), - (0x2135, 'M', 'א'), - (0x2136, 'M', 'ב'), - (0x2137, 'M', 'ג'), - (0x2138, 'M', 'ד'), - (0x2139, 'M', 'i'), - (0x213A, 'V'), - (0x213B, 'M', 'fax'), - (0x213C, 'M', 'π'), - (0x213D, 'M', 'γ'), - (0x213F, 'M', 'π'), - (0x2140, 'M', '∑'), - (0x2141, 'V'), - (0x2145, 'M', 'd'), - (0x2147, 'M', 'e'), - (0x2148, 'M', 'i'), - (0x2149, 'M', 'j'), - (0x214A, 'V'), - (0x2150, 'M', '1⁄7'), - (0x2151, 'M', '1⁄9'), - (0x2152, 'M', '1⁄10'), - (0x2153, 'M', '1⁄3'), - (0x2154, 'M', '2⁄3'), - (0x2155, 'M', '1⁄5'), - (0x2156, 'M', '2⁄5'), - (0x2157, 'M', '3⁄5'), - (0x2158, 'M', '4⁄5'), - (0x2159, 'M', '1⁄6'), - (0x215A, 'M', '5⁄6'), - (0x215B, 'M', '1⁄8'), - (0x215C, 'M', '3⁄8'), - (0x215D, 'M', '5⁄8'), - (0x215E, 'M', '7⁄8'), - (0x215F, 'M', '1⁄'), - (0x2160, 'M', 'i'), - (0x2161, 'M', 'ii'), - (0x2162, 'M', 'iii'), - (0x2163, 'M', 'iv'), - (0x2164, 'M', 'v'), - (0x2165, 'M', 'vi'), - (0x2166, 'M', 'vii'), - (0x2167, 'M', 'viii'), - (0x2168, 'M', 'ix'), - (0x2169, 'M', 'x'), - (0x216A, 'M', 'xi'), - (0x216B, 'M', 'xii'), - (0x216C, 'M', 'l'), - (0x216D, 'M', 'c'), - (0x216E, 'M', 'd'), - (0x216F, 'M', 'm'), - (0x2170, 'M', 'i'), - (0x2171, 'M', 'ii'), - (0x2172, 'M', 'iii'), - (0x2173, 'M', 'iv'), - (0x2174, 'M', 'v'), - (0x2175, 'M', 'vi'), - (0x2176, 'M', 'vii'), - (0x2177, 'M', 'viii'), - (0x2178, 'M', 'ix'), - (0x2179, 'M', 'x'), - (0x217A, 'M', 'xi'), - (0x217B, 'M', 'xii'), - (0x217C, 'M', 'l'), - ] - -def _seg_23() -> List[Union[Tuple[int, str], Tuple[int, str, str]]]: - return [ - (0x217D, 'M', 'c'), - (0x217E, 'M', 'd'), - (0x217F, 'M', 'm'), - (0x2180, 'V'), - (0x2183, 'X'), - (0x2184, 'V'), - (0x2189, 'M', '0⁄3'), - (0x218A, 'V'), - (0x218C, 'X'), - (0x2190, 'V'), - (0x222C, 'M', '∫∫'), - (0x222D, 'M', '∫∫∫'), - (0x222E, 'V'), - (0x222F, 'M', '∮∮'), - (0x2230, 'M', '∮∮∮'), - (0x2231, 'V'), - (0x2260, '3'), - (0x2261, 'V'), - (0x226E, '3'), - (0x2270, 'V'), - (0x2329, 'M', '〈'), - (0x232A, 'M', '〉'), - (0x232B, 'V'), - (0x2427, 'X'), - (0x2440, 'V'), - (0x244B, 'X'), - (0x2460, 'M', '1'), - (0x2461, 'M', '2'), - (0x2462, 'M', '3'), - (0x2463, 'M', '4'), - (0x2464, 'M', '5'), - (0x2465, 'M', '6'), - (0x2466, 'M', '7'), - (0x2467, 'M', '8'), - (0x2468, 'M', '9'), - (0x2469, 'M', '10'), - (0x246A, 'M', '11'), - (0x246B, 'M', '12'), - (0x246C, 'M', '13'), - (0x246D, 'M', '14'), - (0x246E, 'M', '15'), - (0x246F, 'M', '16'), - (0x2470, 'M', '17'), - (0x2471, 'M', '18'), - (0x2472, 'M', '19'), - (0x2473, 'M', '20'), - (0x2474, '3', '(1)'), - (0x2475, '3', '(2)'), - (0x2476, '3', '(3)'), - (0x2477, '3', '(4)'), - (0x2478, '3', '(5)'), - (0x2479, '3', '(6)'), - (0x247A, '3', '(7)'), - (0x247B, '3', '(8)'), - (0x247C, '3', '(9)'), - (0x247D, '3', '(10)'), - (0x247E, '3', '(11)'), - (0x247F, '3', '(12)'), - (0x2480, '3', '(13)'), - (0x2481, '3', '(14)'), - (0x2482, '3', '(15)'), - (0x2483, '3', '(16)'), - (0x2484, '3', '(17)'), - (0x2485, '3', '(18)'), - (0x2486, '3', '(19)'), - (0x2487, '3', '(20)'), - (0x2488, 'X'), - (0x249C, '3', '(a)'), - (0x249D, '3', '(b)'), - (0x249E, '3', '(c)'), - (0x249F, '3', '(d)'), - (0x24A0, '3', '(e)'), - (0x24A1, '3', '(f)'), - (0x24A2, '3', '(g)'), - (0x24A3, '3', '(h)'), - (0x24A4, '3', '(i)'), - (0x24A5, '3', '(j)'), - (0x24A6, '3', '(k)'), - (0x24A7, '3', '(l)'), - (0x24A8, '3', '(m)'), - (0x24A9, '3', '(n)'), - (0x24AA, '3', '(o)'), - (0x24AB, '3', '(p)'), - (0x24AC, '3', '(q)'), - (0x24AD, '3', '(r)'), - (0x24AE, '3', '(s)'), - (0x24AF, '3', '(t)'), - (0x24B0, '3', '(u)'), - (0x24B1, '3', '(v)'), - (0x24B2, '3', '(w)'), - (0x24B3, '3', '(x)'), - (0x24B4, '3', '(y)'), - (0x24B5, '3', '(z)'), - (0x24B6, 'M', 'a'), - (0x24B7, 'M', 'b'), - (0x24B8, 'M', 'c'), - (0x24B9, 'M', 'd'), - (0x24BA, 'M', 'e'), - (0x24BB, 'M', 'f'), - (0x24BC, 'M', 'g'), - ] - -def _seg_24() -> List[Union[Tuple[int, str], Tuple[int, str, str]]]: - return [ - (0x24BD, 'M', 'h'), - (0x24BE, 'M', 'i'), - (0x24BF, 'M', 'j'), - (0x24C0, 'M', 'k'), - (0x24C1, 'M', 'l'), - (0x24C2, 'M', 'm'), - (0x24C3, 'M', 'n'), - (0x24C4, 'M', 'o'), - (0x24C5, 'M', 'p'), - (0x24C6, 'M', 'q'), - (0x24C7, 'M', 'r'), - (0x24C8, 'M', 's'), - (0x24C9, 'M', 't'), - (0x24CA, 'M', 'u'), - (0x24CB, 'M', 'v'), - (0x24CC, 'M', 'w'), - (0x24CD, 'M', 'x'), - (0x24CE, 'M', 'y'), - (0x24CF, 'M', 'z'), - (0x24D0, 'M', 'a'), - (0x24D1, 'M', 'b'), - (0x24D2, 'M', 'c'), - (0x24D3, 'M', 'd'), - (0x24D4, 'M', 'e'), - (0x24D5, 'M', 'f'), - (0x24D6, 'M', 'g'), - (0x24D7, 'M', 'h'), - (0x24D8, 'M', 'i'), - (0x24D9, 'M', 'j'), - (0x24DA, 'M', 'k'), - (0x24DB, 'M', 'l'), - (0x24DC, 'M', 'm'), - (0x24DD, 'M', 'n'), - (0x24DE, 'M', 'o'), - (0x24DF, 'M', 'p'), - (0x24E0, 'M', 'q'), - (0x24E1, 'M', 'r'), - (0x24E2, 'M', 's'), - (0x24E3, 'M', 't'), - (0x24E4, 'M', 'u'), - (0x24E5, 'M', 'v'), - (0x24E6, 'M', 'w'), - (0x24E7, 'M', 'x'), - (0x24E8, 'M', 'y'), - (0x24E9, 'M', 'z'), - (0x24EA, 'M', '0'), - (0x24EB, 'V'), - (0x2A0C, 'M', '∫∫∫∫'), - (0x2A0D, 'V'), - (0x2A74, '3', '::='), - (0x2A75, '3', '=='), - (0x2A76, '3', '==='), - (0x2A77, 'V'), - (0x2ADC, 'M', '⫝̸'), - (0x2ADD, 'V'), - (0x2B74, 'X'), - (0x2B76, 'V'), - (0x2B96, 'X'), - (0x2B97, 'V'), - (0x2C00, 'M', 'ⰰ'), - (0x2C01, 'M', 'ⰱ'), - (0x2C02, 'M', 'ⰲ'), - (0x2C03, 'M', 'ⰳ'), - (0x2C04, 'M', 'ⰴ'), - (0x2C05, 'M', 'ⰵ'), - (0x2C06, 'M', 'ⰶ'), - (0x2C07, 'M', 'ⰷ'), - (0x2C08, 'M', 'ⰸ'), - (0x2C09, 'M', 'ⰹ'), - (0x2C0A, 'M', 'ⰺ'), - (0x2C0B, 'M', 'ⰻ'), - (0x2C0C, 'M', 'ⰼ'), - (0x2C0D, 'M', 'ⰽ'), - (0x2C0E, 'M', 'ⰾ'), - (0x2C0F, 'M', 'ⰿ'), - (0x2C10, 'M', 'ⱀ'), - (0x2C11, 'M', 'ⱁ'), - (0x2C12, 'M', 'ⱂ'), - (0x2C13, 'M', 'ⱃ'), - (0x2C14, 'M', 'ⱄ'), - (0x2C15, 'M', 'ⱅ'), - (0x2C16, 'M', 'ⱆ'), - (0x2C17, 'M', 'ⱇ'), - (0x2C18, 'M', 'ⱈ'), - (0x2C19, 'M', 'ⱉ'), - (0x2C1A, 'M', 'ⱊ'), - (0x2C1B, 'M', 'ⱋ'), - (0x2C1C, 'M', 'ⱌ'), - (0x2C1D, 'M', 'ⱍ'), - (0x2C1E, 'M', 'ⱎ'), - (0x2C1F, 'M', 'ⱏ'), - (0x2C20, 'M', 'ⱐ'), - (0x2C21, 'M', 'ⱑ'), - (0x2C22, 'M', 'ⱒ'), - (0x2C23, 'M', 'ⱓ'), - (0x2C24, 'M', 'ⱔ'), - (0x2C25, 'M', 'ⱕ'), - (0x2C26, 'M', 'ⱖ'), - (0x2C27, 'M', 'ⱗ'), - (0x2C28, 'M', 'ⱘ'), - ] - -def _seg_25() -> List[Union[Tuple[int, str], Tuple[int, str, str]]]: - return [ - (0x2C29, 'M', 'ⱙ'), - (0x2C2A, 'M', 'ⱚ'), - (0x2C2B, 'M', 'ⱛ'), - (0x2C2C, 'M', 'ⱜ'), - (0x2C2D, 'M', 'ⱝ'), - (0x2C2E, 'M', 'ⱞ'), - (0x2C2F, 'M', 'ⱟ'), - (0x2C30, 'V'), - (0x2C60, 'M', 'ⱡ'), - (0x2C61, 'V'), - (0x2C62, 'M', 'ɫ'), - (0x2C63, 'M', 'ᵽ'), - (0x2C64, 'M', 'ɽ'), - (0x2C65, 'V'), - (0x2C67, 'M', 'ⱨ'), - (0x2C68, 'V'), - (0x2C69, 'M', 'ⱪ'), - (0x2C6A, 'V'), - (0x2C6B, 'M', 'ⱬ'), - (0x2C6C, 'V'), - (0x2C6D, 'M', 'ɑ'), - (0x2C6E, 'M', 'ɱ'), - (0x2C6F, 'M', 'ɐ'), - (0x2C70, 'M', 'ɒ'), - (0x2C71, 'V'), - (0x2C72, 'M', 'ⱳ'), - (0x2C73, 'V'), - (0x2C75, 'M', 'ⱶ'), - (0x2C76, 'V'), - (0x2C7C, 'M', 'j'), - (0x2C7D, 'M', 'v'), - (0x2C7E, 'M', 'ȿ'), - (0x2C7F, 'M', 'ɀ'), - (0x2C80, 'M', 'ⲁ'), - (0x2C81, 'V'), - (0x2C82, 'M', 'ⲃ'), - (0x2C83, 'V'), - (0x2C84, 'M', 'ⲅ'), - (0x2C85, 'V'), - (0x2C86, 'M', 'ⲇ'), - (0x2C87, 'V'), - (0x2C88, 'M', 'ⲉ'), - (0x2C89, 'V'), - (0x2C8A, 'M', 'ⲋ'), - (0x2C8B, 'V'), - (0x2C8C, 'M', 'ⲍ'), - (0x2C8D, 'V'), - (0x2C8E, 'M', 'ⲏ'), - (0x2C8F, 'V'), - (0x2C90, 'M', 'ⲑ'), - (0x2C91, 'V'), - (0x2C92, 'M', 'ⲓ'), - (0x2C93, 'V'), - (0x2C94, 'M', 'ⲕ'), - (0x2C95, 'V'), - (0x2C96, 'M', 'ⲗ'), - (0x2C97, 'V'), - (0x2C98, 'M', 'ⲙ'), - (0x2C99, 'V'), - (0x2C9A, 'M', 'ⲛ'), - (0x2C9B, 'V'), - (0x2C9C, 'M', 'ⲝ'), - (0x2C9D, 'V'), - (0x2C9E, 'M', 'ⲟ'), - (0x2C9F, 'V'), - (0x2CA0, 'M', 'ⲡ'), - (0x2CA1, 'V'), - (0x2CA2, 'M', 'ⲣ'), - (0x2CA3, 'V'), - (0x2CA4, 'M', 'ⲥ'), - (0x2CA5, 'V'), - (0x2CA6, 'M', 'ⲧ'), - (0x2CA7, 'V'), - (0x2CA8, 'M', 'ⲩ'), - (0x2CA9, 'V'), - (0x2CAA, 'M', 'ⲫ'), - (0x2CAB, 'V'), - (0x2CAC, 'M', 'ⲭ'), - (0x2CAD, 'V'), - (0x2CAE, 'M', 'ⲯ'), - (0x2CAF, 'V'), - (0x2CB0, 'M', 'ⲱ'), - (0x2CB1, 'V'), - (0x2CB2, 'M', 'ⲳ'), - (0x2CB3, 'V'), - (0x2CB4, 'M', 'ⲵ'), - (0x2CB5, 'V'), - (0x2CB6, 'M', 'ⲷ'), - (0x2CB7, 'V'), - (0x2CB8, 'M', 'ⲹ'), - (0x2CB9, 'V'), - (0x2CBA, 'M', 'ⲻ'), - (0x2CBB, 'V'), - (0x2CBC, 'M', 'ⲽ'), - (0x2CBD, 'V'), - (0x2CBE, 'M', 'ⲿ'), - (0x2CBF, 'V'), - (0x2CC0, 'M', 'ⳁ'), - (0x2CC1, 'V'), - (0x2CC2, 'M', 'ⳃ'), - ] - -def _seg_26() -> List[Union[Tuple[int, str], Tuple[int, str, str]]]: - return [ - (0x2CC3, 'V'), - (0x2CC4, 'M', 'ⳅ'), - (0x2CC5, 'V'), - (0x2CC6, 'M', 'ⳇ'), - (0x2CC7, 'V'), - (0x2CC8, 'M', 'ⳉ'), - (0x2CC9, 'V'), - (0x2CCA, 'M', 'ⳋ'), - (0x2CCB, 'V'), - (0x2CCC, 'M', 'ⳍ'), - (0x2CCD, 'V'), - (0x2CCE, 'M', 'ⳏ'), - (0x2CCF, 'V'), - (0x2CD0, 'M', 'ⳑ'), - (0x2CD1, 'V'), - (0x2CD2, 'M', 'ⳓ'), - (0x2CD3, 'V'), - (0x2CD4, 'M', 'ⳕ'), - (0x2CD5, 'V'), - (0x2CD6, 'M', 'ⳗ'), - (0x2CD7, 'V'), - (0x2CD8, 'M', 'ⳙ'), - (0x2CD9, 'V'), - (0x2CDA, 'M', 'ⳛ'), - (0x2CDB, 'V'), - (0x2CDC, 'M', 'ⳝ'), - (0x2CDD, 'V'), - (0x2CDE, 'M', 'ⳟ'), - (0x2CDF, 'V'), - (0x2CE0, 'M', 'ⳡ'), - (0x2CE1, 'V'), - (0x2CE2, 'M', 'ⳣ'), - (0x2CE3, 'V'), - (0x2CEB, 'M', 'ⳬ'), - (0x2CEC, 'V'), - (0x2CED, 'M', 'ⳮ'), - (0x2CEE, 'V'), - (0x2CF2, 'M', 'ⳳ'), - (0x2CF3, 'V'), - (0x2CF4, 'X'), - (0x2CF9, 'V'), - (0x2D26, 'X'), - (0x2D27, 'V'), - (0x2D28, 'X'), - (0x2D2D, 'V'), - (0x2D2E, 'X'), - (0x2D30, 'V'), - (0x2D68, 'X'), - (0x2D6F, 'M', 'ⵡ'), - (0x2D70, 'V'), - (0x2D71, 'X'), - (0x2D7F, 'V'), - (0x2D97, 'X'), - (0x2DA0, 'V'), - (0x2DA7, 'X'), - (0x2DA8, 'V'), - (0x2DAF, 'X'), - (0x2DB0, 'V'), - (0x2DB7, 'X'), - (0x2DB8, 'V'), - (0x2DBF, 'X'), - (0x2DC0, 'V'), - (0x2DC7, 'X'), - (0x2DC8, 'V'), - (0x2DCF, 'X'), - (0x2DD0, 'V'), - (0x2DD7, 'X'), - (0x2DD8, 'V'), - (0x2DDF, 'X'), - (0x2DE0, 'V'), - (0x2E5E, 'X'), - (0x2E80, 'V'), - (0x2E9A, 'X'), - (0x2E9B, 'V'), - (0x2E9F, 'M', '母'), - (0x2EA0, 'V'), - (0x2EF3, 'M', '龟'), - (0x2EF4, 'X'), - (0x2F00, 'M', '一'), - (0x2F01, 'M', '丨'), - (0x2F02, 'M', '丶'), - (0x2F03, 'M', '丿'), - (0x2F04, 'M', '乙'), - (0x2F05, 'M', '亅'), - (0x2F06, 'M', '二'), - (0x2F07, 'M', '亠'), - (0x2F08, 'M', '人'), - (0x2F09, 'M', '儿'), - (0x2F0A, 'M', '入'), - (0x2F0B, 'M', '八'), - (0x2F0C, 'M', '冂'), - (0x2F0D, 'M', '冖'), - (0x2F0E, 'M', '冫'), - (0x2F0F, 'M', '几'), - (0x2F10, 'M', '凵'), - (0x2F11, 'M', '刀'), - (0x2F12, 'M', '力'), - (0x2F13, 'M', '勹'), - (0x2F14, 'M', '匕'), - (0x2F15, 'M', '匚'), - ] - -def _seg_27() -> List[Union[Tuple[int, str], Tuple[int, str, str]]]: - return [ - (0x2F16, 'M', '匸'), - (0x2F17, 'M', '十'), - (0x2F18, 'M', '卜'), - (0x2F19, 'M', '卩'), - (0x2F1A, 'M', '厂'), - (0x2F1B, 'M', '厶'), - (0x2F1C, 'M', '又'), - (0x2F1D, 'M', '口'), - (0x2F1E, 'M', '囗'), - (0x2F1F, 'M', '土'), - (0x2F20, 'M', '士'), - (0x2F21, 'M', '夂'), - (0x2F22, 'M', '夊'), - (0x2F23, 'M', '夕'), - (0x2F24, 'M', '大'), - (0x2F25, 'M', '女'), - (0x2F26, 'M', '子'), - (0x2F27, 'M', '宀'), - (0x2F28, 'M', '寸'), - (0x2F29, 'M', '小'), - (0x2F2A, 'M', '尢'), - (0x2F2B, 'M', '尸'), - (0x2F2C, 'M', '屮'), - (0x2F2D, 'M', '山'), - (0x2F2E, 'M', '巛'), - (0x2F2F, 'M', '工'), - (0x2F30, 'M', '己'), - (0x2F31, 'M', '巾'), - (0x2F32, 'M', '干'), - (0x2F33, 'M', '幺'), - (0x2F34, 'M', '广'), - (0x2F35, 'M', '廴'), - (0x2F36, 'M', '廾'), - (0x2F37, 'M', '弋'), - (0x2F38, 'M', '弓'), - (0x2F39, 'M', '彐'), - (0x2F3A, 'M', '彡'), - (0x2F3B, 'M', '彳'), - (0x2F3C, 'M', '心'), - (0x2F3D, 'M', '戈'), - (0x2F3E, 'M', '戶'), - (0x2F3F, 'M', '手'), - (0x2F40, 'M', '支'), - (0x2F41, 'M', '攴'), - (0x2F42, 'M', '文'), - (0x2F43, 'M', '斗'), - (0x2F44, 'M', '斤'), - (0x2F45, 'M', '方'), - (0x2F46, 'M', '无'), - (0x2F47, 'M', '日'), - (0x2F48, 'M', '曰'), - (0x2F49, 'M', '月'), - (0x2F4A, 'M', '木'), - (0x2F4B, 'M', '欠'), - (0x2F4C, 'M', '止'), - (0x2F4D, 'M', '歹'), - (0x2F4E, 'M', '殳'), - (0x2F4F, 'M', '毋'), - (0x2F50, 'M', '比'), - (0x2F51, 'M', '毛'), - (0x2F52, 'M', '氏'), - (0x2F53, 'M', '气'), - (0x2F54, 'M', '水'), - (0x2F55, 'M', '火'), - (0x2F56, 'M', '爪'), - (0x2F57, 'M', '父'), - (0x2F58, 'M', '爻'), - (0x2F59, 'M', '爿'), - (0x2F5A, 'M', '片'), - (0x2F5B, 'M', '牙'), - (0x2F5C, 'M', '牛'), - (0x2F5D, 'M', '犬'), - (0x2F5E, 'M', '玄'), - (0x2F5F, 'M', '玉'), - (0x2F60, 'M', '瓜'), - (0x2F61, 'M', '瓦'), - (0x2F62, 'M', '甘'), - (0x2F63, 'M', '生'), - (0x2F64, 'M', '用'), - (0x2F65, 'M', '田'), - (0x2F66, 'M', '疋'), - (0x2F67, 'M', '疒'), - (0x2F68, 'M', '癶'), - (0x2F69, 'M', '白'), - (0x2F6A, 'M', '皮'), - (0x2F6B, 'M', '皿'), - (0x2F6C, 'M', '目'), - (0x2F6D, 'M', '矛'), - (0x2F6E, 'M', '矢'), - (0x2F6F, 'M', '石'), - (0x2F70, 'M', '示'), - (0x2F71, 'M', '禸'), - (0x2F72, 'M', '禾'), - (0x2F73, 'M', '穴'), - (0x2F74, 'M', '立'), - (0x2F75, 'M', '竹'), - (0x2F76, 'M', '米'), - (0x2F77, 'M', '糸'), - (0x2F78, 'M', '缶'), - (0x2F79, 'M', '网'), - ] - -def _seg_28() -> List[Union[Tuple[int, str], Tuple[int, str, str]]]: - return [ - (0x2F7A, 'M', '羊'), - (0x2F7B, 'M', '羽'), - (0x2F7C, 'M', '老'), - (0x2F7D, 'M', '而'), - (0x2F7E, 'M', '耒'), - (0x2F7F, 'M', '耳'), - (0x2F80, 'M', '聿'), - (0x2F81, 'M', '肉'), - (0x2F82, 'M', '臣'), - (0x2F83, 'M', '自'), - (0x2F84, 'M', '至'), - (0x2F85, 'M', '臼'), - (0x2F86, 'M', '舌'), - (0x2F87, 'M', '舛'), - (0x2F88, 'M', '舟'), - (0x2F89, 'M', '艮'), - (0x2F8A, 'M', '色'), - (0x2F8B, 'M', '艸'), - (0x2F8C, 'M', '虍'), - (0x2F8D, 'M', '虫'), - (0x2F8E, 'M', '血'), - (0x2F8F, 'M', '行'), - (0x2F90, 'M', '衣'), - (0x2F91, 'M', '襾'), - (0x2F92, 'M', '見'), - (0x2F93, 'M', '角'), - (0x2F94, 'M', '言'), - (0x2F95, 'M', '谷'), - (0x2F96, 'M', '豆'), - (0x2F97, 'M', '豕'), - (0x2F98, 'M', '豸'), - (0x2F99, 'M', '貝'), - (0x2F9A, 'M', '赤'), - (0x2F9B, 'M', '走'), - (0x2F9C, 'M', '足'), - (0x2F9D, 'M', '身'), - (0x2F9E, 'M', '車'), - (0x2F9F, 'M', '辛'), - (0x2FA0, 'M', '辰'), - (0x2FA1, 'M', '辵'), - (0x2FA2, 'M', '邑'), - (0x2FA3, 'M', '酉'), - (0x2FA4, 'M', '釆'), - (0x2FA5, 'M', '里'), - (0x2FA6, 'M', '金'), - (0x2FA7, 'M', '長'), - (0x2FA8, 'M', '門'), - (0x2FA9, 'M', '阜'), - (0x2FAA, 'M', '隶'), - (0x2FAB, 'M', '隹'), - (0x2FAC, 'M', '雨'), - (0x2FAD, 'M', '靑'), - (0x2FAE, 'M', '非'), - (0x2FAF, 'M', '面'), - (0x2FB0, 'M', '革'), - (0x2FB1, 'M', '韋'), - (0x2FB2, 'M', '韭'), - (0x2FB3, 'M', '音'), - (0x2FB4, 'M', '頁'), - (0x2FB5, 'M', '風'), - (0x2FB6, 'M', '飛'), - (0x2FB7, 'M', '食'), - (0x2FB8, 'M', '首'), - (0x2FB9, 'M', '香'), - (0x2FBA, 'M', '馬'), - (0x2FBB, 'M', '骨'), - (0x2FBC, 'M', '高'), - (0x2FBD, 'M', '髟'), - (0x2FBE, 'M', '鬥'), - (0x2FBF, 'M', '鬯'), - (0x2FC0, 'M', '鬲'), - (0x2FC1, 'M', '鬼'), - (0x2FC2, 'M', '魚'), - (0x2FC3, 'M', '鳥'), - (0x2FC4, 'M', '鹵'), - (0x2FC5, 'M', '鹿'), - (0x2FC6, 'M', '麥'), - (0x2FC7, 'M', '麻'), - (0x2FC8, 'M', '黃'), - (0x2FC9, 'M', '黍'), - (0x2FCA, 'M', '黑'), - (0x2FCB, 'M', '黹'), - (0x2FCC, 'M', '黽'), - (0x2FCD, 'M', '鼎'), - (0x2FCE, 'M', '鼓'), - (0x2FCF, 'M', '鼠'), - (0x2FD0, 'M', '鼻'), - (0x2FD1, 'M', '齊'), - (0x2FD2, 'M', '齒'), - (0x2FD3, 'M', '龍'), - (0x2FD4, 'M', '龜'), - (0x2FD5, 'M', '龠'), - (0x2FD6, 'X'), - (0x3000, '3', ' '), - (0x3001, 'V'), - (0x3002, 'M', '.'), - (0x3003, 'V'), - (0x3036, 'M', '〒'), - (0x3037, 'V'), - (0x3038, 'M', '十'), - ] - -def _seg_29() -> List[Union[Tuple[int, str], Tuple[int, str, str]]]: - return [ - (0x3039, 'M', '卄'), - (0x303A, 'M', '卅'), - (0x303B, 'V'), - (0x3040, 'X'), - (0x3041, 'V'), - (0x3097, 'X'), - (0x3099, 'V'), - (0x309B, '3', ' ゙'), - (0x309C, '3', ' ゚'), - (0x309D, 'V'), - (0x309F, 'M', 'より'), - (0x30A0, 'V'), - (0x30FF, 'M', 'コト'), - (0x3100, 'X'), - (0x3105, 'V'), - (0x3130, 'X'), - (0x3131, 'M', 'ᄀ'), - (0x3132, 'M', 'ᄁ'), - (0x3133, 'M', 'ᆪ'), - (0x3134, 'M', 'ᄂ'), - (0x3135, 'M', 'ᆬ'), - (0x3136, 'M', 'ᆭ'), - (0x3137, 'M', 'ᄃ'), - (0x3138, 'M', 'ᄄ'), - (0x3139, 'M', 'ᄅ'), - (0x313A, 'M', 'ᆰ'), - (0x313B, 'M', 'ᆱ'), - (0x313C, 'M', 'ᆲ'), - (0x313D, 'M', 'ᆳ'), - (0x313E, 'M', 'ᆴ'), - (0x313F, 'M', 'ᆵ'), - (0x3140, 'M', 'ᄚ'), - (0x3141, 'M', 'ᄆ'), - (0x3142, 'M', 'ᄇ'), - (0x3143, 'M', 'ᄈ'), - (0x3144, 'M', 'ᄡ'), - (0x3145, 'M', 'ᄉ'), - (0x3146, 'M', 'ᄊ'), - (0x3147, 'M', 'ᄋ'), - (0x3148, 'M', 'ᄌ'), - (0x3149, 'M', 'ᄍ'), - (0x314A, 'M', 'ᄎ'), - (0x314B, 'M', 'ᄏ'), - (0x314C, 'M', 'ᄐ'), - (0x314D, 'M', 'ᄑ'), - (0x314E, 'M', 'ᄒ'), - (0x314F, 'M', 'ᅡ'), - (0x3150, 'M', 'ᅢ'), - (0x3151, 'M', 'ᅣ'), - (0x3152, 'M', 'ᅤ'), - (0x3153, 'M', 'ᅥ'), - (0x3154, 'M', 'ᅦ'), - (0x3155, 'M', 'ᅧ'), - (0x3156, 'M', 'ᅨ'), - (0x3157, 'M', 'ᅩ'), - (0x3158, 'M', 'ᅪ'), - (0x3159, 'M', 'ᅫ'), - (0x315A, 'M', 'ᅬ'), - (0x315B, 'M', 'ᅭ'), - (0x315C, 'M', 'ᅮ'), - (0x315D, 'M', 'ᅯ'), - (0x315E, 'M', 'ᅰ'), - (0x315F, 'M', 'ᅱ'), - (0x3160, 'M', 'ᅲ'), - (0x3161, 'M', 'ᅳ'), - (0x3162, 'M', 'ᅴ'), - (0x3163, 'M', 'ᅵ'), - (0x3164, 'X'), - (0x3165, 'M', 'ᄔ'), - (0x3166, 'M', 'ᄕ'), - (0x3167, 'M', 'ᇇ'), - (0x3168, 'M', 'ᇈ'), - (0x3169, 'M', 'ᇌ'), - (0x316A, 'M', 'ᇎ'), - (0x316B, 'M', 'ᇓ'), - (0x316C, 'M', 'ᇗ'), - (0x316D, 'M', 'ᇙ'), - (0x316E, 'M', 'ᄜ'), - (0x316F, 'M', 'ᇝ'), - (0x3170, 'M', 'ᇟ'), - (0x3171, 'M', 'ᄝ'), - (0x3172, 'M', 'ᄞ'), - (0x3173, 'M', 'ᄠ'), - (0x3174, 'M', 'ᄢ'), - (0x3175, 'M', 'ᄣ'), - (0x3176, 'M', 'ᄧ'), - (0x3177, 'M', 'ᄩ'), - (0x3178, 'M', 'ᄫ'), - (0x3179, 'M', 'ᄬ'), - (0x317A, 'M', 'ᄭ'), - (0x317B, 'M', 'ᄮ'), - (0x317C, 'M', 'ᄯ'), - (0x317D, 'M', 'ᄲ'), - (0x317E, 'M', 'ᄶ'), - (0x317F, 'M', 'ᅀ'), - (0x3180, 'M', 'ᅇ'), - (0x3181, 'M', 'ᅌ'), - (0x3182, 'M', 'ᇱ'), - (0x3183, 'M', 'ᇲ'), - (0x3184, 'M', 'ᅗ'), - ] - -def _seg_30() -> List[Union[Tuple[int, str], Tuple[int, str, str]]]: - return [ - (0x3185, 'M', 'ᅘ'), - (0x3186, 'M', 'ᅙ'), - (0x3187, 'M', 'ᆄ'), - (0x3188, 'M', 'ᆅ'), - (0x3189, 'M', 'ᆈ'), - (0x318A, 'M', 'ᆑ'), - (0x318B, 'M', 'ᆒ'), - (0x318C, 'M', 'ᆔ'), - (0x318D, 'M', 'ᆞ'), - (0x318E, 'M', 'ᆡ'), - (0x318F, 'X'), - (0x3190, 'V'), - (0x3192, 'M', '一'), - (0x3193, 'M', '二'), - (0x3194, 'M', '三'), - (0x3195, 'M', '四'), - (0x3196, 'M', '上'), - (0x3197, 'M', '中'), - (0x3198, 'M', '下'), - (0x3199, 'M', '甲'), - (0x319A, 'M', '乙'), - (0x319B, 'M', '丙'), - (0x319C, 'M', '丁'), - (0x319D, 'M', '天'), - (0x319E, 'M', '地'), - (0x319F, 'M', '人'), - (0x31A0, 'V'), - (0x31E4, 'X'), - (0x31F0, 'V'), - (0x3200, '3', '(ᄀ)'), - (0x3201, '3', '(ᄂ)'), - (0x3202, '3', '(ᄃ)'), - (0x3203, '3', '(ᄅ)'), - (0x3204, '3', '(ᄆ)'), - (0x3205, '3', '(ᄇ)'), - (0x3206, '3', '(ᄉ)'), - (0x3207, '3', '(ᄋ)'), - (0x3208, '3', '(ᄌ)'), - (0x3209, '3', '(ᄎ)'), - (0x320A, '3', '(ᄏ)'), - (0x320B, '3', '(ᄐ)'), - (0x320C, '3', '(ᄑ)'), - (0x320D, '3', '(ᄒ)'), - (0x320E, '3', '(가)'), - (0x320F, '3', '(나)'), - (0x3210, '3', '(다)'), - (0x3211, '3', '(라)'), - (0x3212, '3', '(마)'), - (0x3213, '3', '(바)'), - (0x3214, '3', '(사)'), - (0x3215, '3', '(아)'), - (0x3216, '3', '(자)'), - (0x3217, '3', '(차)'), - (0x3218, '3', '(카)'), - (0x3219, '3', '(타)'), - (0x321A, '3', '(파)'), - (0x321B, '3', '(하)'), - (0x321C, '3', '(주)'), - (0x321D, '3', '(오전)'), - (0x321E, '3', '(오후)'), - (0x321F, 'X'), - (0x3220, '3', '(一)'), - (0x3221, '3', '(二)'), - (0x3222, '3', '(三)'), - (0x3223, '3', '(四)'), - (0x3224, '3', '(五)'), - (0x3225, '3', '(六)'), - (0x3226, '3', '(七)'), - (0x3227, '3', '(八)'), - (0x3228, '3', '(九)'), - (0x3229, '3', '(十)'), - (0x322A, '3', '(月)'), - (0x322B, '3', '(火)'), - (0x322C, '3', '(水)'), - (0x322D, '3', '(木)'), - (0x322E, '3', '(金)'), - (0x322F, '3', '(土)'), - (0x3230, '3', '(日)'), - (0x3231, '3', '(株)'), - (0x3232, '3', '(有)'), - (0x3233, '3', '(社)'), - (0x3234, '3', '(名)'), - (0x3235, '3', '(特)'), - (0x3236, '3', '(財)'), - (0x3237, '3', '(祝)'), - (0x3238, '3', '(労)'), - (0x3239, '3', '(代)'), - (0x323A, '3', '(呼)'), - (0x323B, '3', '(学)'), - (0x323C, '3', '(監)'), - (0x323D, '3', '(企)'), - (0x323E, '3', '(資)'), - (0x323F, '3', '(協)'), - (0x3240, '3', '(祭)'), - (0x3241, '3', '(休)'), - (0x3242, '3', '(自)'), - (0x3243, '3', '(至)'), - (0x3244, 'M', '問'), - (0x3245, 'M', '幼'), - (0x3246, 'M', '文'), - ] - -def _seg_31() -> List[Union[Tuple[int, str], Tuple[int, str, str]]]: - return [ - (0x3247, 'M', '箏'), - (0x3248, 'V'), - (0x3250, 'M', 'pte'), - (0x3251, 'M', '21'), - (0x3252, 'M', '22'), - (0x3253, 'M', '23'), - (0x3254, 'M', '24'), - (0x3255, 'M', '25'), - (0x3256, 'M', '26'), - (0x3257, 'M', '27'), - (0x3258, 'M', '28'), - (0x3259, 'M', '29'), - (0x325A, 'M', '30'), - (0x325B, 'M', '31'), - (0x325C, 'M', '32'), - (0x325D, 'M', '33'), - (0x325E, 'M', '34'), - (0x325F, 'M', '35'), - (0x3260, 'M', 'ᄀ'), - (0x3261, 'M', 'ᄂ'), - (0x3262, 'M', 'ᄃ'), - (0x3263, 'M', 'ᄅ'), - (0x3264, 'M', 'ᄆ'), - (0x3265, 'M', 'ᄇ'), - (0x3266, 'M', 'ᄉ'), - (0x3267, 'M', 'ᄋ'), - (0x3268, 'M', 'ᄌ'), - (0x3269, 'M', 'ᄎ'), - (0x326A, 'M', 'ᄏ'), - (0x326B, 'M', 'ᄐ'), - (0x326C, 'M', 'ᄑ'), - (0x326D, 'M', 'ᄒ'), - (0x326E, 'M', '가'), - (0x326F, 'M', '나'), - (0x3270, 'M', '다'), - (0x3271, 'M', '라'), - (0x3272, 'M', '마'), - (0x3273, 'M', '바'), - (0x3274, 'M', '사'), - (0x3275, 'M', '아'), - (0x3276, 'M', '자'), - (0x3277, 'M', '차'), - (0x3278, 'M', '카'), - (0x3279, 'M', '타'), - (0x327A, 'M', '파'), - (0x327B, 'M', '하'), - (0x327C, 'M', '참고'), - (0x327D, 'M', '주의'), - (0x327E, 'M', '우'), - (0x327F, 'V'), - (0x3280, 'M', '一'), - (0x3281, 'M', '二'), - (0x3282, 'M', '三'), - (0x3283, 'M', '四'), - (0x3284, 'M', '五'), - (0x3285, 'M', '六'), - (0x3286, 'M', '七'), - (0x3287, 'M', '八'), - (0x3288, 'M', '九'), - (0x3289, 'M', '十'), - (0x328A, 'M', '月'), - (0x328B, 'M', '火'), - (0x328C, 'M', '水'), - (0x328D, 'M', '木'), - (0x328E, 'M', '金'), - (0x328F, 'M', '土'), - (0x3290, 'M', '日'), - (0x3291, 'M', '株'), - (0x3292, 'M', '有'), - (0x3293, 'M', '社'), - (0x3294, 'M', '名'), - (0x3295, 'M', '特'), - (0x3296, 'M', '財'), - (0x3297, 'M', '祝'), - (0x3298, 'M', '労'), - (0x3299, 'M', '秘'), - (0x329A, 'M', '男'), - (0x329B, 'M', '女'), - (0x329C, 'M', '適'), - (0x329D, 'M', '優'), - (0x329E, 'M', '印'), - (0x329F, 'M', '注'), - (0x32A0, 'M', '項'), - (0x32A1, 'M', '休'), - (0x32A2, 'M', '写'), - (0x32A3, 'M', '正'), - (0x32A4, 'M', '上'), - (0x32A5, 'M', '中'), - (0x32A6, 'M', '下'), - (0x32A7, 'M', '左'), - (0x32A8, 'M', '右'), - (0x32A9, 'M', '医'), - (0x32AA, 'M', '宗'), - (0x32AB, 'M', '学'), - (0x32AC, 'M', '監'), - (0x32AD, 'M', '企'), - (0x32AE, 'M', '資'), - (0x32AF, 'M', '協'), - (0x32B0, 'M', '夜'), - (0x32B1, 'M', '36'), - ] - -def _seg_32() -> List[Union[Tuple[int, str], Tuple[int, str, str]]]: - return [ - (0x32B2, 'M', '37'), - (0x32B3, 'M', '38'), - (0x32B4, 'M', '39'), - (0x32B5, 'M', '40'), - (0x32B6, 'M', '41'), - (0x32B7, 'M', '42'), - (0x32B8, 'M', '43'), - (0x32B9, 'M', '44'), - (0x32BA, 'M', '45'), - (0x32BB, 'M', '46'), - (0x32BC, 'M', '47'), - (0x32BD, 'M', '48'), - (0x32BE, 'M', '49'), - (0x32BF, 'M', '50'), - (0x32C0, 'M', '1月'), - (0x32C1, 'M', '2月'), - (0x32C2, 'M', '3月'), - (0x32C3, 'M', '4月'), - (0x32C4, 'M', '5月'), - (0x32C5, 'M', '6月'), - (0x32C6, 'M', '7月'), - (0x32C7, 'M', '8月'), - (0x32C8, 'M', '9月'), - (0x32C9, 'M', '10月'), - (0x32CA, 'M', '11月'), - (0x32CB, 'M', '12月'), - (0x32CC, 'M', 'hg'), - (0x32CD, 'M', 'erg'), - (0x32CE, 'M', 'ev'), - (0x32CF, 'M', 'ltd'), - (0x32D0, 'M', 'ア'), - (0x32D1, 'M', 'イ'), - (0x32D2, 'M', 'ウ'), - (0x32D3, 'M', 'エ'), - (0x32D4, 'M', 'オ'), - (0x32D5, 'M', 'カ'), - (0x32D6, 'M', 'キ'), - (0x32D7, 'M', 'ク'), - (0x32D8, 'M', 'ケ'), - (0x32D9, 'M', 'コ'), - (0x32DA, 'M', 'サ'), - (0x32DB, 'M', 'シ'), - (0x32DC, 'M', 'ス'), - (0x32DD, 'M', 'セ'), - (0x32DE, 'M', 'ソ'), - (0x32DF, 'M', 'タ'), - (0x32E0, 'M', 'チ'), - (0x32E1, 'M', 'ツ'), - (0x32E2, 'M', 'テ'), - (0x32E3, 'M', 'ト'), - (0x32E4, 'M', 'ナ'), - (0x32E5, 'M', 'ニ'), - (0x32E6, 'M', 'ヌ'), - (0x32E7, 'M', 'ネ'), - (0x32E8, 'M', 'ノ'), - (0x32E9, 'M', 'ハ'), - (0x32EA, 'M', 'ヒ'), - (0x32EB, 'M', 'フ'), - (0x32EC, 'M', 'ヘ'), - (0x32ED, 'M', 'ホ'), - (0x32EE, 'M', 'マ'), - (0x32EF, 'M', 'ミ'), - (0x32F0, 'M', 'ム'), - (0x32F1, 'M', 'メ'), - (0x32F2, 'M', 'モ'), - (0x32F3, 'M', 'ヤ'), - (0x32F4, 'M', 'ユ'), - (0x32F5, 'M', 'ヨ'), - (0x32F6, 'M', 'ラ'), - (0x32F7, 'M', 'リ'), - (0x32F8, 'M', 'ル'), - (0x32F9, 'M', 'レ'), - (0x32FA, 'M', 'ロ'), - (0x32FB, 'M', 'ワ'), - (0x32FC, 'M', 'ヰ'), - (0x32FD, 'M', 'ヱ'), - (0x32FE, 'M', 'ヲ'), - (0x32FF, 'M', '令和'), - (0x3300, 'M', 'アパート'), - (0x3301, 'M', 'アルファ'), - (0x3302, 'M', 'アンペア'), - (0x3303, 'M', 'アール'), - (0x3304, 'M', 'イニング'), - (0x3305, 'M', 'インチ'), - (0x3306, 'M', 'ウォン'), - (0x3307, 'M', 'エスクード'), - (0x3308, 'M', 'エーカー'), - (0x3309, 'M', 'オンス'), - (0x330A, 'M', 'オーム'), - (0x330B, 'M', 'カイリ'), - (0x330C, 'M', 'カラット'), - (0x330D, 'M', 'カロリー'), - (0x330E, 'M', 'ガロン'), - (0x330F, 'M', 'ガンマ'), - (0x3310, 'M', 'ギガ'), - (0x3311, 'M', 'ギニー'), - (0x3312, 'M', 'キュリー'), - (0x3313, 'M', 'ギルダー'), - (0x3314, 'M', 'キロ'), - (0x3315, 'M', 'キログラム'), - ] - -def _seg_33() -> List[Union[Tuple[int, str], Tuple[int, str, str]]]: - return [ - (0x3316, 'M', 'キロメートル'), - (0x3317, 'M', 'キロワット'), - (0x3318, 'M', 'グラム'), - (0x3319, 'M', 'グラムトン'), - (0x331A, 'M', 'クルゼイロ'), - (0x331B, 'M', 'クローネ'), - (0x331C, 'M', 'ケース'), - (0x331D, 'M', 'コルナ'), - (0x331E, 'M', 'コーポ'), - (0x331F, 'M', 'サイクル'), - (0x3320, 'M', 'サンチーム'), - (0x3321, 'M', 'シリング'), - (0x3322, 'M', 'センチ'), - (0x3323, 'M', 'セント'), - (0x3324, 'M', 'ダース'), - (0x3325, 'M', 'デシ'), - (0x3326, 'M', 'ドル'), - (0x3327, 'M', 'トン'), - (0x3328, 'M', 'ナノ'), - (0x3329, 'M', 'ノット'), - (0x332A, 'M', 'ハイツ'), - (0x332B, 'M', 'パーセント'), - (0x332C, 'M', 'パーツ'), - (0x332D, 'M', 'バーレル'), - (0x332E, 'M', 'ピアストル'), - (0x332F, 'M', 'ピクル'), - (0x3330, 'M', 'ピコ'), - (0x3331, 'M', 'ビル'), - (0x3332, 'M', 'ファラッド'), - (0x3333, 'M', 'フィート'), - (0x3334, 'M', 'ブッシェル'), - (0x3335, 'M', 'フラン'), - (0x3336, 'M', 'ヘクタール'), - (0x3337, 'M', 'ペソ'), - (0x3338, 'M', 'ペニヒ'), - (0x3339, 'M', 'ヘルツ'), - (0x333A, 'M', 'ペンス'), - (0x333B, 'M', 'ページ'), - (0x333C, 'M', 'ベータ'), - (0x333D, 'M', 'ポイント'), - (0x333E, 'M', 'ボルト'), - (0x333F, 'M', 'ホン'), - (0x3340, 'M', 'ポンド'), - (0x3341, 'M', 'ホール'), - (0x3342, 'M', 'ホーン'), - (0x3343, 'M', 'マイクロ'), - (0x3344, 'M', 'マイル'), - (0x3345, 'M', 'マッハ'), - (0x3346, 'M', 'マルク'), - (0x3347, 'M', 'マンション'), - (0x3348, 'M', 'ミクロン'), - (0x3349, 'M', 'ミリ'), - (0x334A, 'M', 'ミリバール'), - (0x334B, 'M', 'メガ'), - (0x334C, 'M', 'メガトン'), - (0x334D, 'M', 'メートル'), - (0x334E, 'M', 'ヤード'), - (0x334F, 'M', 'ヤール'), - (0x3350, 'M', 'ユアン'), - (0x3351, 'M', 'リットル'), - (0x3352, 'M', 'リラ'), - (0x3353, 'M', 'ルピー'), - (0x3354, 'M', 'ルーブル'), - (0x3355, 'M', 'レム'), - (0x3356, 'M', 'レントゲン'), - (0x3357, 'M', 'ワット'), - (0x3358, 'M', '0点'), - (0x3359, 'M', '1点'), - (0x335A, 'M', '2点'), - (0x335B, 'M', '3点'), - (0x335C, 'M', '4点'), - (0x335D, 'M', '5点'), - (0x335E, 'M', '6点'), - (0x335F, 'M', '7点'), - (0x3360, 'M', '8点'), - (0x3361, 'M', '9点'), - (0x3362, 'M', '10点'), - (0x3363, 'M', '11点'), - (0x3364, 'M', '12点'), - (0x3365, 'M', '13点'), - (0x3366, 'M', '14点'), - (0x3367, 'M', '15点'), - (0x3368, 'M', '16点'), - (0x3369, 'M', '17点'), - (0x336A, 'M', '18点'), - (0x336B, 'M', '19点'), - (0x336C, 'M', '20点'), - (0x336D, 'M', '21点'), - (0x336E, 'M', '22点'), - (0x336F, 'M', '23点'), - (0x3370, 'M', '24点'), - (0x3371, 'M', 'hpa'), - (0x3372, 'M', 'da'), - (0x3373, 'M', 'au'), - (0x3374, 'M', 'bar'), - (0x3375, 'M', 'ov'), - (0x3376, 'M', 'pc'), - (0x3377, 'M', 'dm'), - (0x3378, 'M', 'dm2'), - (0x3379, 'M', 'dm3'), - ] - -def _seg_34() -> List[Union[Tuple[int, str], Tuple[int, str, str]]]: - return [ - (0x337A, 'M', 'iu'), - (0x337B, 'M', '平成'), - (0x337C, 'M', '昭和'), - (0x337D, 'M', '大正'), - (0x337E, 'M', '明治'), - (0x337F, 'M', '株式会社'), - (0x3380, 'M', 'pa'), - (0x3381, 'M', 'na'), - (0x3382, 'M', 'μa'), - (0x3383, 'M', 'ma'), - (0x3384, 'M', 'ka'), - (0x3385, 'M', 'kb'), - (0x3386, 'M', 'mb'), - (0x3387, 'M', 'gb'), - (0x3388, 'M', 'cal'), - (0x3389, 'M', 'kcal'), - (0x338A, 'M', 'pf'), - (0x338B, 'M', 'nf'), - (0x338C, 'M', 'μf'), - (0x338D, 'M', 'μg'), - (0x338E, 'M', 'mg'), - (0x338F, 'M', 'kg'), - (0x3390, 'M', 'hz'), - (0x3391, 'M', 'khz'), - (0x3392, 'M', 'mhz'), - (0x3393, 'M', 'ghz'), - (0x3394, 'M', 'thz'), - (0x3395, 'M', 'μl'), - (0x3396, 'M', 'ml'), - (0x3397, 'M', 'dl'), - (0x3398, 'M', 'kl'), - (0x3399, 'M', 'fm'), - (0x339A, 'M', 'nm'), - (0x339B, 'M', 'μm'), - (0x339C, 'M', 'mm'), - (0x339D, 'M', 'cm'), - (0x339E, 'M', 'km'), - (0x339F, 'M', 'mm2'), - (0x33A0, 'M', 'cm2'), - (0x33A1, 'M', 'm2'), - (0x33A2, 'M', 'km2'), - (0x33A3, 'M', 'mm3'), - (0x33A4, 'M', 'cm3'), - (0x33A5, 'M', 'm3'), - (0x33A6, 'M', 'km3'), - (0x33A7, 'M', 'm∕s'), - (0x33A8, 'M', 'm∕s2'), - (0x33A9, 'M', 'pa'), - (0x33AA, 'M', 'kpa'), - (0x33AB, 'M', 'mpa'), - (0x33AC, 'M', 'gpa'), - (0x33AD, 'M', 'rad'), - (0x33AE, 'M', 'rad∕s'), - (0x33AF, 'M', 'rad∕s2'), - (0x33B0, 'M', 'ps'), - (0x33B1, 'M', 'ns'), - (0x33B2, 'M', 'μs'), - (0x33B3, 'M', 'ms'), - (0x33B4, 'M', 'pv'), - (0x33B5, 'M', 'nv'), - (0x33B6, 'M', 'μv'), - (0x33B7, 'M', 'mv'), - (0x33B8, 'M', 'kv'), - (0x33B9, 'M', 'mv'), - (0x33BA, 'M', 'pw'), - (0x33BB, 'M', 'nw'), - (0x33BC, 'M', 'μw'), - (0x33BD, 'M', 'mw'), - (0x33BE, 'M', 'kw'), - (0x33BF, 'M', 'mw'), - (0x33C0, 'M', 'kω'), - (0x33C1, 'M', 'mω'), - (0x33C2, 'X'), - (0x33C3, 'M', 'bq'), - (0x33C4, 'M', 'cc'), - (0x33C5, 'M', 'cd'), - (0x33C6, 'M', 'c∕kg'), - (0x33C7, 'X'), - (0x33C8, 'M', 'db'), - (0x33C9, 'M', 'gy'), - (0x33CA, 'M', 'ha'), - (0x33CB, 'M', 'hp'), - (0x33CC, 'M', 'in'), - (0x33CD, 'M', 'kk'), - (0x33CE, 'M', 'km'), - (0x33CF, 'M', 'kt'), - (0x33D0, 'M', 'lm'), - (0x33D1, 'M', 'ln'), - (0x33D2, 'M', 'log'), - (0x33D3, 'M', 'lx'), - (0x33D4, 'M', 'mb'), - (0x33D5, 'M', 'mil'), - (0x33D6, 'M', 'mol'), - (0x33D7, 'M', 'ph'), - (0x33D8, 'X'), - (0x33D9, 'M', 'ppm'), - (0x33DA, 'M', 'pr'), - (0x33DB, 'M', 'sr'), - (0x33DC, 'M', 'sv'), - (0x33DD, 'M', 'wb'), - ] - -def _seg_35() -> List[Union[Tuple[int, str], Tuple[int, str, str]]]: - return [ - (0x33DE, 'M', 'v∕m'), - (0x33DF, 'M', 'a∕m'), - (0x33E0, 'M', '1日'), - (0x33E1, 'M', '2日'), - (0x33E2, 'M', '3日'), - (0x33E3, 'M', '4日'), - (0x33E4, 'M', '5日'), - (0x33E5, 'M', '6日'), - (0x33E6, 'M', '7日'), - (0x33E7, 'M', '8日'), - (0x33E8, 'M', '9日'), - (0x33E9, 'M', '10日'), - (0x33EA, 'M', '11日'), - (0x33EB, 'M', '12日'), - (0x33EC, 'M', '13日'), - (0x33ED, 'M', '14日'), - (0x33EE, 'M', '15日'), - (0x33EF, 'M', '16日'), - (0x33F0, 'M', '17日'), - (0x33F1, 'M', '18日'), - (0x33F2, 'M', '19日'), - (0x33F3, 'M', '20日'), - (0x33F4, 'M', '21日'), - (0x33F5, 'M', '22日'), - (0x33F6, 'M', '23日'), - (0x33F7, 'M', '24日'), - (0x33F8, 'M', '25日'), - (0x33F9, 'M', '26日'), - (0x33FA, 'M', '27日'), - (0x33FB, 'M', '28日'), - (0x33FC, 'M', '29日'), - (0x33FD, 'M', '30日'), - (0x33FE, 'M', '31日'), - (0x33FF, 'M', 'gal'), - (0x3400, 'V'), - (0xA48D, 'X'), - (0xA490, 'V'), - (0xA4C7, 'X'), - (0xA4D0, 'V'), - (0xA62C, 'X'), - (0xA640, 'M', 'ꙁ'), - (0xA641, 'V'), - (0xA642, 'M', 'ꙃ'), - (0xA643, 'V'), - (0xA644, 'M', 'ꙅ'), - (0xA645, 'V'), - (0xA646, 'M', 'ꙇ'), - (0xA647, 'V'), - (0xA648, 'M', 'ꙉ'), - (0xA649, 'V'), - (0xA64A, 'M', 'ꙋ'), - (0xA64B, 'V'), - (0xA64C, 'M', 'ꙍ'), - (0xA64D, 'V'), - (0xA64E, 'M', 'ꙏ'), - (0xA64F, 'V'), - (0xA650, 'M', 'ꙑ'), - (0xA651, 'V'), - (0xA652, 'M', 'ꙓ'), - (0xA653, 'V'), - (0xA654, 'M', 'ꙕ'), - (0xA655, 'V'), - (0xA656, 'M', 'ꙗ'), - (0xA657, 'V'), - (0xA658, 'M', 'ꙙ'), - (0xA659, 'V'), - (0xA65A, 'M', 'ꙛ'), - (0xA65B, 'V'), - (0xA65C, 'M', 'ꙝ'), - (0xA65D, 'V'), - (0xA65E, 'M', 'ꙟ'), - (0xA65F, 'V'), - (0xA660, 'M', 'ꙡ'), - (0xA661, 'V'), - (0xA662, 'M', 'ꙣ'), - (0xA663, 'V'), - (0xA664, 'M', 'ꙥ'), - (0xA665, 'V'), - (0xA666, 'M', 'ꙧ'), - (0xA667, 'V'), - (0xA668, 'M', 'ꙩ'), - (0xA669, 'V'), - (0xA66A, 'M', 'ꙫ'), - (0xA66B, 'V'), - (0xA66C, 'M', 'ꙭ'), - (0xA66D, 'V'), - (0xA680, 'M', 'ꚁ'), - (0xA681, 'V'), - (0xA682, 'M', 'ꚃ'), - (0xA683, 'V'), - (0xA684, 'M', 'ꚅ'), - (0xA685, 'V'), - (0xA686, 'M', 'ꚇ'), - (0xA687, 'V'), - (0xA688, 'M', 'ꚉ'), - (0xA689, 'V'), - (0xA68A, 'M', 'ꚋ'), - (0xA68B, 'V'), - (0xA68C, 'M', 'ꚍ'), - (0xA68D, 'V'), - ] - -def _seg_36() -> List[Union[Tuple[int, str], Tuple[int, str, str]]]: - return [ - (0xA68E, 'M', 'ꚏ'), - (0xA68F, 'V'), - (0xA690, 'M', 'ꚑ'), - (0xA691, 'V'), - (0xA692, 'M', 'ꚓ'), - (0xA693, 'V'), - (0xA694, 'M', 'ꚕ'), - (0xA695, 'V'), - (0xA696, 'M', 'ꚗ'), - (0xA697, 'V'), - (0xA698, 'M', 'ꚙ'), - (0xA699, 'V'), - (0xA69A, 'M', 'ꚛ'), - (0xA69B, 'V'), - (0xA69C, 'M', 'ъ'), - (0xA69D, 'M', 'ь'), - (0xA69E, 'V'), - (0xA6F8, 'X'), - (0xA700, 'V'), - (0xA722, 'M', 'ꜣ'), - (0xA723, 'V'), - (0xA724, 'M', 'ꜥ'), - (0xA725, 'V'), - (0xA726, 'M', 'ꜧ'), - (0xA727, 'V'), - (0xA728, 'M', 'ꜩ'), - (0xA729, 'V'), - (0xA72A, 'M', 'ꜫ'), - (0xA72B, 'V'), - (0xA72C, 'M', 'ꜭ'), - (0xA72D, 'V'), - (0xA72E, 'M', 'ꜯ'), - (0xA72F, 'V'), - (0xA732, 'M', 'ꜳ'), - (0xA733, 'V'), - (0xA734, 'M', 'ꜵ'), - (0xA735, 'V'), - (0xA736, 'M', 'ꜷ'), - (0xA737, 'V'), - (0xA738, 'M', 'ꜹ'), - (0xA739, 'V'), - (0xA73A, 'M', 'ꜻ'), - (0xA73B, 'V'), - (0xA73C, 'M', 'ꜽ'), - (0xA73D, 'V'), - (0xA73E, 'M', 'ꜿ'), - (0xA73F, 'V'), - (0xA740, 'M', 'ꝁ'), - (0xA741, 'V'), - (0xA742, 'M', 'ꝃ'), - (0xA743, 'V'), - (0xA744, 'M', 'ꝅ'), - (0xA745, 'V'), - (0xA746, 'M', 'ꝇ'), - (0xA747, 'V'), - (0xA748, 'M', 'ꝉ'), - (0xA749, 'V'), - (0xA74A, 'M', 'ꝋ'), - (0xA74B, 'V'), - (0xA74C, 'M', 'ꝍ'), - (0xA74D, 'V'), - (0xA74E, 'M', 'ꝏ'), - (0xA74F, 'V'), - (0xA750, 'M', 'ꝑ'), - (0xA751, 'V'), - (0xA752, 'M', 'ꝓ'), - (0xA753, 'V'), - (0xA754, 'M', 'ꝕ'), - (0xA755, 'V'), - (0xA756, 'M', 'ꝗ'), - (0xA757, 'V'), - (0xA758, 'M', 'ꝙ'), - (0xA759, 'V'), - (0xA75A, 'M', 'ꝛ'), - (0xA75B, 'V'), - (0xA75C, 'M', 'ꝝ'), - (0xA75D, 'V'), - (0xA75E, 'M', 'ꝟ'), - (0xA75F, 'V'), - (0xA760, 'M', 'ꝡ'), - (0xA761, 'V'), - (0xA762, 'M', 'ꝣ'), - (0xA763, 'V'), - (0xA764, 'M', 'ꝥ'), - (0xA765, 'V'), - (0xA766, 'M', 'ꝧ'), - (0xA767, 'V'), - (0xA768, 'M', 'ꝩ'), - (0xA769, 'V'), - (0xA76A, 'M', 'ꝫ'), - (0xA76B, 'V'), - (0xA76C, 'M', 'ꝭ'), - (0xA76D, 'V'), - (0xA76E, 'M', 'ꝯ'), - (0xA76F, 'V'), - (0xA770, 'M', 'ꝯ'), - (0xA771, 'V'), - (0xA779, 'M', 'ꝺ'), - (0xA77A, 'V'), - (0xA77B, 'M', 'ꝼ'), - ] - -def _seg_37() -> List[Union[Tuple[int, str], Tuple[int, str, str]]]: - return [ - (0xA77C, 'V'), - (0xA77D, 'M', 'ᵹ'), - (0xA77E, 'M', 'ꝿ'), - (0xA77F, 'V'), - (0xA780, 'M', 'ꞁ'), - (0xA781, 'V'), - (0xA782, 'M', 'ꞃ'), - (0xA783, 'V'), - (0xA784, 'M', 'ꞅ'), - (0xA785, 'V'), - (0xA786, 'M', 'ꞇ'), - (0xA787, 'V'), - (0xA78B, 'M', 'ꞌ'), - (0xA78C, 'V'), - (0xA78D, 'M', 'ɥ'), - (0xA78E, 'V'), - (0xA790, 'M', 'ꞑ'), - (0xA791, 'V'), - (0xA792, 'M', 'ꞓ'), - (0xA793, 'V'), - (0xA796, 'M', 'ꞗ'), - (0xA797, 'V'), - (0xA798, 'M', 'ꞙ'), - (0xA799, 'V'), - (0xA79A, 'M', 'ꞛ'), - (0xA79B, 'V'), - (0xA79C, 'M', 'ꞝ'), - (0xA79D, 'V'), - (0xA79E, 'M', 'ꞟ'), - (0xA79F, 'V'), - (0xA7A0, 'M', 'ꞡ'), - (0xA7A1, 'V'), - (0xA7A2, 'M', 'ꞣ'), - (0xA7A3, 'V'), - (0xA7A4, 'M', 'ꞥ'), - (0xA7A5, 'V'), - (0xA7A6, 'M', 'ꞧ'), - (0xA7A7, 'V'), - (0xA7A8, 'M', 'ꞩ'), - (0xA7A9, 'V'), - (0xA7AA, 'M', 'ɦ'), - (0xA7AB, 'M', 'ɜ'), - (0xA7AC, 'M', 'ɡ'), - (0xA7AD, 'M', 'ɬ'), - (0xA7AE, 'M', 'ɪ'), - (0xA7AF, 'V'), - (0xA7B0, 'M', 'ʞ'), - (0xA7B1, 'M', 'ʇ'), - (0xA7B2, 'M', 'ʝ'), - (0xA7B3, 'M', 'ꭓ'), - (0xA7B4, 'M', 'ꞵ'), - (0xA7B5, 'V'), - (0xA7B6, 'M', 'ꞷ'), - (0xA7B7, 'V'), - (0xA7B8, 'M', 'ꞹ'), - (0xA7B9, 'V'), - (0xA7BA, 'M', 'ꞻ'), - (0xA7BB, 'V'), - (0xA7BC, 'M', 'ꞽ'), - (0xA7BD, 'V'), - (0xA7BE, 'M', 'ꞿ'), - (0xA7BF, 'V'), - (0xA7C0, 'M', 'ꟁ'), - (0xA7C1, 'V'), - (0xA7C2, 'M', 'ꟃ'), - (0xA7C3, 'V'), - (0xA7C4, 'M', 'ꞔ'), - (0xA7C5, 'M', 'ʂ'), - (0xA7C6, 'M', 'ᶎ'), - (0xA7C7, 'M', 'ꟈ'), - (0xA7C8, 'V'), - (0xA7C9, 'M', 'ꟊ'), - (0xA7CA, 'V'), - (0xA7CB, 'X'), - (0xA7D0, 'M', 'ꟑ'), - (0xA7D1, 'V'), - (0xA7D2, 'X'), - (0xA7D3, 'V'), - (0xA7D4, 'X'), - (0xA7D5, 'V'), - (0xA7D6, 'M', 'ꟗ'), - (0xA7D7, 'V'), - (0xA7D8, 'M', 'ꟙ'), - (0xA7D9, 'V'), - (0xA7DA, 'X'), - (0xA7F2, 'M', 'c'), - (0xA7F3, 'M', 'f'), - (0xA7F4, 'M', 'q'), - (0xA7F5, 'M', 'ꟶ'), - (0xA7F6, 'V'), - (0xA7F8, 'M', 'ħ'), - (0xA7F9, 'M', 'œ'), - (0xA7FA, 'V'), - (0xA82D, 'X'), - (0xA830, 'V'), - (0xA83A, 'X'), - (0xA840, 'V'), - (0xA878, 'X'), - (0xA880, 'V'), - (0xA8C6, 'X'), - ] - -def _seg_38() -> List[Union[Tuple[int, str], Tuple[int, str, str]]]: - return [ - (0xA8CE, 'V'), - (0xA8DA, 'X'), - (0xA8E0, 'V'), - (0xA954, 'X'), - (0xA95F, 'V'), - (0xA97D, 'X'), - (0xA980, 'V'), - (0xA9CE, 'X'), - (0xA9CF, 'V'), - (0xA9DA, 'X'), - (0xA9DE, 'V'), - (0xA9FF, 'X'), - (0xAA00, 'V'), - (0xAA37, 'X'), - (0xAA40, 'V'), - (0xAA4E, 'X'), - (0xAA50, 'V'), - (0xAA5A, 'X'), - (0xAA5C, 'V'), - (0xAAC3, 'X'), - (0xAADB, 'V'), - (0xAAF7, 'X'), - (0xAB01, 'V'), - (0xAB07, 'X'), - (0xAB09, 'V'), - (0xAB0F, 'X'), - (0xAB11, 'V'), - (0xAB17, 'X'), - (0xAB20, 'V'), - (0xAB27, 'X'), - (0xAB28, 'V'), - (0xAB2F, 'X'), - (0xAB30, 'V'), - (0xAB5C, 'M', 'ꜧ'), - (0xAB5D, 'M', 'ꬷ'), - (0xAB5E, 'M', 'ɫ'), - (0xAB5F, 'M', 'ꭒ'), - (0xAB60, 'V'), - (0xAB69, 'M', 'ʍ'), - (0xAB6A, 'V'), - (0xAB6C, 'X'), - (0xAB70, 'M', 'Ꭰ'), - (0xAB71, 'M', 'Ꭱ'), - (0xAB72, 'M', 'Ꭲ'), - (0xAB73, 'M', 'Ꭳ'), - (0xAB74, 'M', 'Ꭴ'), - (0xAB75, 'M', 'Ꭵ'), - (0xAB76, 'M', 'Ꭶ'), - (0xAB77, 'M', 'Ꭷ'), - (0xAB78, 'M', 'Ꭸ'), - (0xAB79, 'M', 'Ꭹ'), - (0xAB7A, 'M', 'Ꭺ'), - (0xAB7B, 'M', 'Ꭻ'), - (0xAB7C, 'M', 'Ꭼ'), - (0xAB7D, 'M', 'Ꭽ'), - (0xAB7E, 'M', 'Ꭾ'), - (0xAB7F, 'M', 'Ꭿ'), - (0xAB80, 'M', 'Ꮀ'), - (0xAB81, 'M', 'Ꮁ'), - (0xAB82, 'M', 'Ꮂ'), - (0xAB83, 'M', 'Ꮃ'), - (0xAB84, 'M', 'Ꮄ'), - (0xAB85, 'M', 'Ꮅ'), - (0xAB86, 'M', 'Ꮆ'), - (0xAB87, 'M', 'Ꮇ'), - (0xAB88, 'M', 'Ꮈ'), - (0xAB89, 'M', 'Ꮉ'), - (0xAB8A, 'M', 'Ꮊ'), - (0xAB8B, 'M', 'Ꮋ'), - (0xAB8C, 'M', 'Ꮌ'), - (0xAB8D, 'M', 'Ꮍ'), - (0xAB8E, 'M', 'Ꮎ'), - (0xAB8F, 'M', 'Ꮏ'), - (0xAB90, 'M', 'Ꮐ'), - (0xAB91, 'M', 'Ꮑ'), - (0xAB92, 'M', 'Ꮒ'), - (0xAB93, 'M', 'Ꮓ'), - (0xAB94, 'M', 'Ꮔ'), - (0xAB95, 'M', 'Ꮕ'), - (0xAB96, 'M', 'Ꮖ'), - (0xAB97, 'M', 'Ꮗ'), - (0xAB98, 'M', 'Ꮘ'), - (0xAB99, 'M', 'Ꮙ'), - (0xAB9A, 'M', 'Ꮚ'), - (0xAB9B, 'M', 'Ꮛ'), - (0xAB9C, 'M', 'Ꮜ'), - (0xAB9D, 'M', 'Ꮝ'), - (0xAB9E, 'M', 'Ꮞ'), - (0xAB9F, 'M', 'Ꮟ'), - (0xABA0, 'M', 'Ꮠ'), - (0xABA1, 'M', 'Ꮡ'), - (0xABA2, 'M', 'Ꮢ'), - (0xABA3, 'M', 'Ꮣ'), - (0xABA4, 'M', 'Ꮤ'), - (0xABA5, 'M', 'Ꮥ'), - (0xABA6, 'M', 'Ꮦ'), - (0xABA7, 'M', 'Ꮧ'), - (0xABA8, 'M', 'Ꮨ'), - (0xABA9, 'M', 'Ꮩ'), - (0xABAA, 'M', 'Ꮪ'), - ] - -def _seg_39() -> List[Union[Tuple[int, str], Tuple[int, str, str]]]: - return [ - (0xABAB, 'M', 'Ꮫ'), - (0xABAC, 'M', 'Ꮬ'), - (0xABAD, 'M', 'Ꮭ'), - (0xABAE, 'M', 'Ꮮ'), - (0xABAF, 'M', 'Ꮯ'), - (0xABB0, 'M', 'Ꮰ'), - (0xABB1, 'M', 'Ꮱ'), - (0xABB2, 'M', 'Ꮲ'), - (0xABB3, 'M', 'Ꮳ'), - (0xABB4, 'M', 'Ꮴ'), - (0xABB5, 'M', 'Ꮵ'), - (0xABB6, 'M', 'Ꮶ'), - (0xABB7, 'M', 'Ꮷ'), - (0xABB8, 'M', 'Ꮸ'), - (0xABB9, 'M', 'Ꮹ'), - (0xABBA, 'M', 'Ꮺ'), - (0xABBB, 'M', 'Ꮻ'), - (0xABBC, 'M', 'Ꮼ'), - (0xABBD, 'M', 'Ꮽ'), - (0xABBE, 'M', 'Ꮾ'), - (0xABBF, 'M', 'Ꮿ'), - (0xABC0, 'V'), - (0xABEE, 'X'), - (0xABF0, 'V'), - (0xABFA, 'X'), - (0xAC00, 'V'), - (0xD7A4, 'X'), - (0xD7B0, 'V'), - (0xD7C7, 'X'), - (0xD7CB, 'V'), - (0xD7FC, 'X'), - (0xF900, 'M', '豈'), - (0xF901, 'M', '更'), - (0xF902, 'M', '車'), - (0xF903, 'M', '賈'), - (0xF904, 'M', '滑'), - (0xF905, 'M', '串'), - (0xF906, 'M', '句'), - (0xF907, 'M', '龜'), - (0xF909, 'M', '契'), - (0xF90A, 'M', '金'), - (0xF90B, 'M', '喇'), - (0xF90C, 'M', '奈'), - (0xF90D, 'M', '懶'), - (0xF90E, 'M', '癩'), - (0xF90F, 'M', '羅'), - (0xF910, 'M', '蘿'), - (0xF911, 'M', '螺'), - (0xF912, 'M', '裸'), - (0xF913, 'M', '邏'), - (0xF914, 'M', '樂'), - (0xF915, 'M', '洛'), - (0xF916, 'M', '烙'), - (0xF917, 'M', '珞'), - (0xF918, 'M', '落'), - (0xF919, 'M', '酪'), - (0xF91A, 'M', '駱'), - (0xF91B, 'M', '亂'), - (0xF91C, 'M', '卵'), - (0xF91D, 'M', '欄'), - (0xF91E, 'M', '爛'), - (0xF91F, 'M', '蘭'), - (0xF920, 'M', '鸞'), - (0xF921, 'M', '嵐'), - (0xF922, 'M', '濫'), - (0xF923, 'M', '藍'), - (0xF924, 'M', '襤'), - (0xF925, 'M', '拉'), - (0xF926, 'M', '臘'), - (0xF927, 'M', '蠟'), - (0xF928, 'M', '廊'), - (0xF929, 'M', '朗'), - (0xF92A, 'M', '浪'), - (0xF92B, 'M', '狼'), - (0xF92C, 'M', '郎'), - (0xF92D, 'M', '來'), - (0xF92E, 'M', '冷'), - (0xF92F, 'M', '勞'), - (0xF930, 'M', '擄'), - (0xF931, 'M', '櫓'), - (0xF932, 'M', '爐'), - (0xF933, 'M', '盧'), - (0xF934, 'M', '老'), - (0xF935, 'M', '蘆'), - (0xF936, 'M', '虜'), - (0xF937, 'M', '路'), - (0xF938, 'M', '露'), - (0xF939, 'M', '魯'), - (0xF93A, 'M', '鷺'), - (0xF93B, 'M', '碌'), - (0xF93C, 'M', '祿'), - (0xF93D, 'M', '綠'), - (0xF93E, 'M', '菉'), - (0xF93F, 'M', '錄'), - (0xF940, 'M', '鹿'), - (0xF941, 'M', '論'), - (0xF942, 'M', '壟'), - (0xF943, 'M', '弄'), - (0xF944, 'M', '籠'), - (0xF945, 'M', '聾'), - ] - -def _seg_40() -> List[Union[Tuple[int, str], Tuple[int, str, str]]]: - return [ - (0xF946, 'M', '牢'), - (0xF947, 'M', '磊'), - (0xF948, 'M', '賂'), - (0xF949, 'M', '雷'), - (0xF94A, 'M', '壘'), - (0xF94B, 'M', '屢'), - (0xF94C, 'M', '樓'), - (0xF94D, 'M', '淚'), - (0xF94E, 'M', '漏'), - (0xF94F, 'M', '累'), - (0xF950, 'M', '縷'), - (0xF951, 'M', '陋'), - (0xF952, 'M', '勒'), - (0xF953, 'M', '肋'), - (0xF954, 'M', '凜'), - (0xF955, 'M', '凌'), - (0xF956, 'M', '稜'), - (0xF957, 'M', '綾'), - (0xF958, 'M', '菱'), - (0xF959, 'M', '陵'), - (0xF95A, 'M', '讀'), - (0xF95B, 'M', '拏'), - (0xF95C, 'M', '樂'), - (0xF95D, 'M', '諾'), - (0xF95E, 'M', '丹'), - (0xF95F, 'M', '寧'), - (0xF960, 'M', '怒'), - (0xF961, 'M', '率'), - (0xF962, 'M', '異'), - (0xF963, 'M', '北'), - (0xF964, 'M', '磻'), - (0xF965, 'M', '便'), - (0xF966, 'M', '復'), - (0xF967, 'M', '不'), - (0xF968, 'M', '泌'), - (0xF969, 'M', '數'), - (0xF96A, 'M', '索'), - (0xF96B, 'M', '參'), - (0xF96C, 'M', '塞'), - (0xF96D, 'M', '省'), - (0xF96E, 'M', '葉'), - (0xF96F, 'M', '說'), - (0xF970, 'M', '殺'), - (0xF971, 'M', '辰'), - (0xF972, 'M', '沈'), - (0xF973, 'M', '拾'), - (0xF974, 'M', '若'), - (0xF975, 'M', '掠'), - (0xF976, 'M', '略'), - (0xF977, 'M', '亮'), - (0xF978, 'M', '兩'), - (0xF979, 'M', '凉'), - (0xF97A, 'M', '梁'), - (0xF97B, 'M', '糧'), - (0xF97C, 'M', '良'), - (0xF97D, 'M', '諒'), - (0xF97E, 'M', '量'), - (0xF97F, 'M', '勵'), - (0xF980, 'M', '呂'), - (0xF981, 'M', '女'), - (0xF982, 'M', '廬'), - (0xF983, 'M', '旅'), - (0xF984, 'M', '濾'), - (0xF985, 'M', '礪'), - (0xF986, 'M', '閭'), - (0xF987, 'M', '驪'), - (0xF988, 'M', '麗'), - (0xF989, 'M', '黎'), - (0xF98A, 'M', '力'), - (0xF98B, 'M', '曆'), - (0xF98C, 'M', '歷'), - (0xF98D, 'M', '轢'), - (0xF98E, 'M', '年'), - (0xF98F, 'M', '憐'), - (0xF990, 'M', '戀'), - (0xF991, 'M', '撚'), - (0xF992, 'M', '漣'), - (0xF993, 'M', '煉'), - (0xF994, 'M', '璉'), - (0xF995, 'M', '秊'), - (0xF996, 'M', '練'), - (0xF997, 'M', '聯'), - (0xF998, 'M', '輦'), - (0xF999, 'M', '蓮'), - (0xF99A, 'M', '連'), - (0xF99B, 'M', '鍊'), - (0xF99C, 'M', '列'), - (0xF99D, 'M', '劣'), - (0xF99E, 'M', '咽'), - (0xF99F, 'M', '烈'), - (0xF9A0, 'M', '裂'), - (0xF9A1, 'M', '說'), - (0xF9A2, 'M', '廉'), - (0xF9A3, 'M', '念'), - (0xF9A4, 'M', '捻'), - (0xF9A5, 'M', '殮'), - (0xF9A6, 'M', '簾'), - (0xF9A7, 'M', '獵'), - (0xF9A8, 'M', '令'), - (0xF9A9, 'M', '囹'), - ] - -def _seg_41() -> List[Union[Tuple[int, str], Tuple[int, str, str]]]: - return [ - (0xF9AA, 'M', '寧'), - (0xF9AB, 'M', '嶺'), - (0xF9AC, 'M', '怜'), - (0xF9AD, 'M', '玲'), - (0xF9AE, 'M', '瑩'), - (0xF9AF, 'M', '羚'), - (0xF9B0, 'M', '聆'), - (0xF9B1, 'M', '鈴'), - (0xF9B2, 'M', '零'), - (0xF9B3, 'M', '靈'), - (0xF9B4, 'M', '領'), - (0xF9B5, 'M', '例'), - (0xF9B6, 'M', '禮'), - (0xF9B7, 'M', '醴'), - (0xF9B8, 'M', '隸'), - (0xF9B9, 'M', '惡'), - (0xF9BA, 'M', '了'), - (0xF9BB, 'M', '僚'), - (0xF9BC, 'M', '寮'), - (0xF9BD, 'M', '尿'), - (0xF9BE, 'M', '料'), - (0xF9BF, 'M', '樂'), - (0xF9C0, 'M', '燎'), - (0xF9C1, 'M', '療'), - (0xF9C2, 'M', '蓼'), - (0xF9C3, 'M', '遼'), - (0xF9C4, 'M', '龍'), - (0xF9C5, 'M', '暈'), - (0xF9C6, 'M', '阮'), - (0xF9C7, 'M', '劉'), - (0xF9C8, 'M', '杻'), - (0xF9C9, 'M', '柳'), - (0xF9CA, 'M', '流'), - (0xF9CB, 'M', '溜'), - (0xF9CC, 'M', '琉'), - (0xF9CD, 'M', '留'), - (0xF9CE, 'M', '硫'), - (0xF9CF, 'M', '紐'), - (0xF9D0, 'M', '類'), - (0xF9D1, 'M', '六'), - (0xF9D2, 'M', '戮'), - (0xF9D3, 'M', '陸'), - (0xF9D4, 'M', '倫'), - (0xF9D5, 'M', '崙'), - (0xF9D6, 'M', '淪'), - (0xF9D7, 'M', '輪'), - (0xF9D8, 'M', '律'), - (0xF9D9, 'M', '慄'), - (0xF9DA, 'M', '栗'), - (0xF9DB, 'M', '率'), - (0xF9DC, 'M', '隆'), - (0xF9DD, 'M', '利'), - (0xF9DE, 'M', '吏'), - (0xF9DF, 'M', '履'), - (0xF9E0, 'M', '易'), - (0xF9E1, 'M', '李'), - (0xF9E2, 'M', '梨'), - (0xF9E3, 'M', '泥'), - (0xF9E4, 'M', '理'), - (0xF9E5, 'M', '痢'), - (0xF9E6, 'M', '罹'), - (0xF9E7, 'M', '裏'), - (0xF9E8, 'M', '裡'), - (0xF9E9, 'M', '里'), - (0xF9EA, 'M', '離'), - (0xF9EB, 'M', '匿'), - (0xF9EC, 'M', '溺'), - (0xF9ED, 'M', '吝'), - (0xF9EE, 'M', '燐'), - (0xF9EF, 'M', '璘'), - (0xF9F0, 'M', '藺'), - (0xF9F1, 'M', '隣'), - (0xF9F2, 'M', '鱗'), - (0xF9F3, 'M', '麟'), - (0xF9F4, 'M', '林'), - (0xF9F5, 'M', '淋'), - (0xF9F6, 'M', '臨'), - (0xF9F7, 'M', '立'), - (0xF9F8, 'M', '笠'), - (0xF9F9, 'M', '粒'), - (0xF9FA, 'M', '狀'), - (0xF9FB, 'M', '炙'), - (0xF9FC, 'M', '識'), - (0xF9FD, 'M', '什'), - (0xF9FE, 'M', '茶'), - (0xF9FF, 'M', '刺'), - (0xFA00, 'M', '切'), - (0xFA01, 'M', '度'), - (0xFA02, 'M', '拓'), - (0xFA03, 'M', '糖'), - (0xFA04, 'M', '宅'), - (0xFA05, 'M', '洞'), - (0xFA06, 'M', '暴'), - (0xFA07, 'M', '輻'), - (0xFA08, 'M', '行'), - (0xFA09, 'M', '降'), - (0xFA0A, 'M', '見'), - (0xFA0B, 'M', '廓'), - (0xFA0C, 'M', '兀'), - (0xFA0D, 'M', '嗀'), - ] - -def _seg_42() -> List[Union[Tuple[int, str], Tuple[int, str, str]]]: - return [ - (0xFA0E, 'V'), - (0xFA10, 'M', '塚'), - (0xFA11, 'V'), - (0xFA12, 'M', '晴'), - (0xFA13, 'V'), - (0xFA15, 'M', '凞'), - (0xFA16, 'M', '猪'), - (0xFA17, 'M', '益'), - (0xFA18, 'M', '礼'), - (0xFA19, 'M', '神'), - (0xFA1A, 'M', '祥'), - (0xFA1B, 'M', '福'), - (0xFA1C, 'M', '靖'), - (0xFA1D, 'M', '精'), - (0xFA1E, 'M', '羽'), - (0xFA1F, 'V'), - (0xFA20, 'M', '蘒'), - (0xFA21, 'V'), - (0xFA22, 'M', '諸'), - (0xFA23, 'V'), - (0xFA25, 'M', '逸'), - (0xFA26, 'M', '都'), - (0xFA27, 'V'), - (0xFA2A, 'M', '飯'), - (0xFA2B, 'M', '飼'), - (0xFA2C, 'M', '館'), - (0xFA2D, 'M', '鶴'), - (0xFA2E, 'M', '郞'), - (0xFA2F, 'M', '隷'), - (0xFA30, 'M', '侮'), - (0xFA31, 'M', '僧'), - (0xFA32, 'M', '免'), - (0xFA33, 'M', '勉'), - (0xFA34, 'M', '勤'), - (0xFA35, 'M', '卑'), - (0xFA36, 'M', '喝'), - (0xFA37, 'M', '嘆'), - (0xFA38, 'M', '器'), - (0xFA39, 'M', '塀'), - (0xFA3A, 'M', '墨'), - (0xFA3B, 'M', '層'), - (0xFA3C, 'M', '屮'), - (0xFA3D, 'M', '悔'), - (0xFA3E, 'M', '慨'), - (0xFA3F, 'M', '憎'), - (0xFA40, 'M', '懲'), - (0xFA41, 'M', '敏'), - (0xFA42, 'M', '既'), - (0xFA43, 'M', '暑'), - (0xFA44, 'M', '梅'), - (0xFA45, 'M', '海'), - (0xFA46, 'M', '渚'), - (0xFA47, 'M', '漢'), - (0xFA48, 'M', '煮'), - (0xFA49, 'M', '爫'), - (0xFA4A, 'M', '琢'), - (0xFA4B, 'M', '碑'), - (0xFA4C, 'M', '社'), - (0xFA4D, 'M', '祉'), - (0xFA4E, 'M', '祈'), - (0xFA4F, 'M', '祐'), - (0xFA50, 'M', '祖'), - (0xFA51, 'M', '祝'), - (0xFA52, 'M', '禍'), - (0xFA53, 'M', '禎'), - (0xFA54, 'M', '穀'), - (0xFA55, 'M', '突'), - (0xFA56, 'M', '節'), - (0xFA57, 'M', '練'), - (0xFA58, 'M', '縉'), - (0xFA59, 'M', '繁'), - (0xFA5A, 'M', '署'), - (0xFA5B, 'M', '者'), - (0xFA5C, 'M', '臭'), - (0xFA5D, 'M', '艹'), - (0xFA5F, 'M', '著'), - (0xFA60, 'M', '褐'), - (0xFA61, 'M', '視'), - (0xFA62, 'M', '謁'), - (0xFA63, 'M', '謹'), - (0xFA64, 'M', '賓'), - (0xFA65, 'M', '贈'), - (0xFA66, 'M', '辶'), - (0xFA67, 'M', '逸'), - (0xFA68, 'M', '難'), - (0xFA69, 'M', '響'), - (0xFA6A, 'M', '頻'), - (0xFA6B, 'M', '恵'), - (0xFA6C, 'M', '𤋮'), - (0xFA6D, 'M', '舘'), - (0xFA6E, 'X'), - (0xFA70, 'M', '並'), - (0xFA71, 'M', '况'), - (0xFA72, 'M', '全'), - (0xFA73, 'M', '侀'), - (0xFA74, 'M', '充'), - (0xFA75, 'M', '冀'), - (0xFA76, 'M', '勇'), - (0xFA77, 'M', '勺'), - (0xFA78, 'M', '喝'), - ] - -def _seg_43() -> List[Union[Tuple[int, str], Tuple[int, str, str]]]: - return [ - (0xFA79, 'M', '啕'), - (0xFA7A, 'M', '喙'), - (0xFA7B, 'M', '嗢'), - (0xFA7C, 'M', '塚'), - (0xFA7D, 'M', '墳'), - (0xFA7E, 'M', '奄'), - (0xFA7F, 'M', '奔'), - (0xFA80, 'M', '婢'), - (0xFA81, 'M', '嬨'), - (0xFA82, 'M', '廒'), - (0xFA83, 'M', '廙'), - (0xFA84, 'M', '彩'), - (0xFA85, 'M', '徭'), - (0xFA86, 'M', '惘'), - (0xFA87, 'M', '慎'), - (0xFA88, 'M', '愈'), - (0xFA89, 'M', '憎'), - (0xFA8A, 'M', '慠'), - (0xFA8B, 'M', '懲'), - (0xFA8C, 'M', '戴'), - (0xFA8D, 'M', '揄'), - (0xFA8E, 'M', '搜'), - (0xFA8F, 'M', '摒'), - (0xFA90, 'M', '敖'), - (0xFA91, 'M', '晴'), - (0xFA92, 'M', '朗'), - (0xFA93, 'M', '望'), - (0xFA94, 'M', '杖'), - (0xFA95, 'M', '歹'), - (0xFA96, 'M', '殺'), - (0xFA97, 'M', '流'), - (0xFA98, 'M', '滛'), - (0xFA99, 'M', '滋'), - (0xFA9A, 'M', '漢'), - (0xFA9B, 'M', '瀞'), - (0xFA9C, 'M', '煮'), - (0xFA9D, 'M', '瞧'), - (0xFA9E, 'M', '爵'), - (0xFA9F, 'M', '犯'), - (0xFAA0, 'M', '猪'), - (0xFAA1, 'M', '瑱'), - (0xFAA2, 'M', '甆'), - (0xFAA3, 'M', '画'), - (0xFAA4, 'M', '瘝'), - (0xFAA5, 'M', '瘟'), - (0xFAA6, 'M', '益'), - (0xFAA7, 'M', '盛'), - (0xFAA8, 'M', '直'), - (0xFAA9, 'M', '睊'), - (0xFAAA, 'M', '着'), - (0xFAAB, 'M', '磌'), - (0xFAAC, 'M', '窱'), - (0xFAAD, 'M', '節'), - (0xFAAE, 'M', '类'), - (0xFAAF, 'M', '絛'), - (0xFAB0, 'M', '練'), - (0xFAB1, 'M', '缾'), - (0xFAB2, 'M', '者'), - (0xFAB3, 'M', '荒'), - (0xFAB4, 'M', '華'), - (0xFAB5, 'M', '蝹'), - (0xFAB6, 'M', '襁'), - (0xFAB7, 'M', '覆'), - (0xFAB8, 'M', '視'), - (0xFAB9, 'M', '調'), - (0xFABA, 'M', '諸'), - (0xFABB, 'M', '請'), - (0xFABC, 'M', '謁'), - (0xFABD, 'M', '諾'), - (0xFABE, 'M', '諭'), - (0xFABF, 'M', '謹'), - (0xFAC0, 'M', '變'), - (0xFAC1, 'M', '贈'), - (0xFAC2, 'M', '輸'), - (0xFAC3, 'M', '遲'), - (0xFAC4, 'M', '醙'), - (0xFAC5, 'M', '鉶'), - (0xFAC6, 'M', '陼'), - (0xFAC7, 'M', '難'), - (0xFAC8, 'M', '靖'), - (0xFAC9, 'M', '韛'), - (0xFACA, 'M', '響'), - (0xFACB, 'M', '頋'), - (0xFACC, 'M', '頻'), - (0xFACD, 'M', '鬒'), - (0xFACE, 'M', '龜'), - (0xFACF, 'M', '𢡊'), - (0xFAD0, 'M', '𢡄'), - (0xFAD1, 'M', '𣏕'), - (0xFAD2, 'M', '㮝'), - (0xFAD3, 'M', '䀘'), - (0xFAD4, 'M', '䀹'), - (0xFAD5, 'M', '𥉉'), - (0xFAD6, 'M', '𥳐'), - (0xFAD7, 'M', '𧻓'), - (0xFAD8, 'M', '齃'), - (0xFAD9, 'M', '龎'), - (0xFADA, 'X'), - (0xFB00, 'M', 'ff'), - (0xFB01, 'M', 'fi'), - ] - -def _seg_44() -> List[Union[Tuple[int, str], Tuple[int, str, str]]]: - return [ - (0xFB02, 'M', 'fl'), - (0xFB03, 'M', 'ffi'), - (0xFB04, 'M', 'ffl'), - (0xFB05, 'M', 'st'), - (0xFB07, 'X'), - (0xFB13, 'M', 'մն'), - (0xFB14, 'M', 'մե'), - (0xFB15, 'M', 'մի'), - (0xFB16, 'M', 'վն'), - (0xFB17, 'M', 'մխ'), - (0xFB18, 'X'), - (0xFB1D, 'M', 'יִ'), - (0xFB1E, 'V'), - (0xFB1F, 'M', 'ײַ'), - (0xFB20, 'M', 'ע'), - (0xFB21, 'M', 'א'), - (0xFB22, 'M', 'ד'), - (0xFB23, 'M', 'ה'), - (0xFB24, 'M', 'כ'), - (0xFB25, 'M', 'ל'), - (0xFB26, 'M', 'ם'), - (0xFB27, 'M', 'ר'), - (0xFB28, 'M', 'ת'), - (0xFB29, '3', '+'), - (0xFB2A, 'M', 'שׁ'), - (0xFB2B, 'M', 'שׂ'), - (0xFB2C, 'M', 'שּׁ'), - (0xFB2D, 'M', 'שּׂ'), - (0xFB2E, 'M', 'אַ'), - (0xFB2F, 'M', 'אָ'), - (0xFB30, 'M', 'אּ'), - (0xFB31, 'M', 'בּ'), - (0xFB32, 'M', 'גּ'), - (0xFB33, 'M', 'דּ'), - (0xFB34, 'M', 'הּ'), - (0xFB35, 'M', 'וּ'), - (0xFB36, 'M', 'זּ'), - (0xFB37, 'X'), - (0xFB38, 'M', 'טּ'), - (0xFB39, 'M', 'יּ'), - (0xFB3A, 'M', 'ךּ'), - (0xFB3B, 'M', 'כּ'), - (0xFB3C, 'M', 'לּ'), - (0xFB3D, 'X'), - (0xFB3E, 'M', 'מּ'), - (0xFB3F, 'X'), - (0xFB40, 'M', 'נּ'), - (0xFB41, 'M', 'סּ'), - (0xFB42, 'X'), - (0xFB43, 'M', 'ףּ'), - (0xFB44, 'M', 'פּ'), - (0xFB45, 'X'), - (0xFB46, 'M', 'צּ'), - (0xFB47, 'M', 'קּ'), - (0xFB48, 'M', 'רּ'), - (0xFB49, 'M', 'שּ'), - (0xFB4A, 'M', 'תּ'), - (0xFB4B, 'M', 'וֹ'), - (0xFB4C, 'M', 'בֿ'), - (0xFB4D, 'M', 'כֿ'), - (0xFB4E, 'M', 'פֿ'), - (0xFB4F, 'M', 'אל'), - (0xFB50, 'M', 'ٱ'), - (0xFB52, 'M', 'ٻ'), - (0xFB56, 'M', 'پ'), - (0xFB5A, 'M', 'ڀ'), - (0xFB5E, 'M', 'ٺ'), - (0xFB62, 'M', 'ٿ'), - (0xFB66, 'M', 'ٹ'), - (0xFB6A, 'M', 'ڤ'), - (0xFB6E, 'M', 'ڦ'), - (0xFB72, 'M', 'ڄ'), - (0xFB76, 'M', 'ڃ'), - (0xFB7A, 'M', 'چ'), - (0xFB7E, 'M', 'ڇ'), - (0xFB82, 'M', 'ڍ'), - (0xFB84, 'M', 'ڌ'), - (0xFB86, 'M', 'ڎ'), - (0xFB88, 'M', 'ڈ'), - (0xFB8A, 'M', 'ژ'), - (0xFB8C, 'M', 'ڑ'), - (0xFB8E, 'M', 'ک'), - (0xFB92, 'M', 'گ'), - (0xFB96, 'M', 'ڳ'), - (0xFB9A, 'M', 'ڱ'), - (0xFB9E, 'M', 'ں'), - (0xFBA0, 'M', 'ڻ'), - (0xFBA4, 'M', 'ۀ'), - (0xFBA6, 'M', 'ہ'), - (0xFBAA, 'M', 'ھ'), - (0xFBAE, 'M', 'ے'), - (0xFBB0, 'M', 'ۓ'), - (0xFBB2, 'V'), - (0xFBC3, 'X'), - (0xFBD3, 'M', 'ڭ'), - (0xFBD7, 'M', 'ۇ'), - (0xFBD9, 'M', 'ۆ'), - (0xFBDB, 'M', 'ۈ'), - (0xFBDD, 'M', 'ۇٴ'), - (0xFBDE, 'M', 'ۋ'), - ] - -def _seg_45() -> List[Union[Tuple[int, str], Tuple[int, str, str]]]: - return [ - (0xFBE0, 'M', 'ۅ'), - (0xFBE2, 'M', 'ۉ'), - (0xFBE4, 'M', 'ې'), - (0xFBE8, 'M', 'ى'), - (0xFBEA, 'M', 'ئا'), - (0xFBEC, 'M', 'ئە'), - (0xFBEE, 'M', 'ئو'), - (0xFBF0, 'M', 'ئۇ'), - (0xFBF2, 'M', 'ئۆ'), - (0xFBF4, 'M', 'ئۈ'), - (0xFBF6, 'M', 'ئې'), - (0xFBF9, 'M', 'ئى'), - (0xFBFC, 'M', 'ی'), - (0xFC00, 'M', 'ئج'), - (0xFC01, 'M', 'ئح'), - (0xFC02, 'M', 'ئم'), - (0xFC03, 'M', 'ئى'), - (0xFC04, 'M', 'ئي'), - (0xFC05, 'M', 'بج'), - (0xFC06, 'M', 'بح'), - (0xFC07, 'M', 'بخ'), - (0xFC08, 'M', 'بم'), - (0xFC09, 'M', 'بى'), - (0xFC0A, 'M', 'بي'), - (0xFC0B, 'M', 'تج'), - (0xFC0C, 'M', 'تح'), - (0xFC0D, 'M', 'تخ'), - (0xFC0E, 'M', 'تم'), - (0xFC0F, 'M', 'تى'), - (0xFC10, 'M', 'تي'), - (0xFC11, 'M', 'ثج'), - (0xFC12, 'M', 'ثم'), - (0xFC13, 'M', 'ثى'), - (0xFC14, 'M', 'ثي'), - (0xFC15, 'M', 'جح'), - (0xFC16, 'M', 'جم'), - (0xFC17, 'M', 'حج'), - (0xFC18, 'M', 'حم'), - (0xFC19, 'M', 'خج'), - (0xFC1A, 'M', 'خح'), - (0xFC1B, 'M', 'خم'), - (0xFC1C, 'M', 'سج'), - (0xFC1D, 'M', 'سح'), - (0xFC1E, 'M', 'سخ'), - (0xFC1F, 'M', 'سم'), - (0xFC20, 'M', 'صح'), - (0xFC21, 'M', 'صم'), - (0xFC22, 'M', 'ضج'), - (0xFC23, 'M', 'ضح'), - (0xFC24, 'M', 'ضخ'), - (0xFC25, 'M', 'ضم'), - (0xFC26, 'M', 'طح'), - (0xFC27, 'M', 'طم'), - (0xFC28, 'M', 'ظم'), - (0xFC29, 'M', 'عج'), - (0xFC2A, 'M', 'عم'), - (0xFC2B, 'M', 'غج'), - (0xFC2C, 'M', 'غم'), - (0xFC2D, 'M', 'فج'), - (0xFC2E, 'M', 'فح'), - (0xFC2F, 'M', 'فخ'), - (0xFC30, 'M', 'فم'), - (0xFC31, 'M', 'فى'), - (0xFC32, 'M', 'في'), - (0xFC33, 'M', 'قح'), - (0xFC34, 'M', 'قم'), - (0xFC35, 'M', 'قى'), - (0xFC36, 'M', 'قي'), - (0xFC37, 'M', 'كا'), - (0xFC38, 'M', 'كج'), - (0xFC39, 'M', 'كح'), - (0xFC3A, 'M', 'كخ'), - (0xFC3B, 'M', 'كل'), - (0xFC3C, 'M', 'كم'), - (0xFC3D, 'M', 'كى'), - (0xFC3E, 'M', 'كي'), - (0xFC3F, 'M', 'لج'), - (0xFC40, 'M', 'لح'), - (0xFC41, 'M', 'لخ'), - (0xFC42, 'M', 'لم'), - (0xFC43, 'M', 'لى'), - (0xFC44, 'M', 'لي'), - (0xFC45, 'M', 'مج'), - (0xFC46, 'M', 'مح'), - (0xFC47, 'M', 'مخ'), - (0xFC48, 'M', 'مم'), - (0xFC49, 'M', 'مى'), - (0xFC4A, 'M', 'مي'), - (0xFC4B, 'M', 'نج'), - (0xFC4C, 'M', 'نح'), - (0xFC4D, 'M', 'نخ'), - (0xFC4E, 'M', 'نم'), - (0xFC4F, 'M', 'نى'), - (0xFC50, 'M', 'ني'), - (0xFC51, 'M', 'هج'), - (0xFC52, 'M', 'هم'), - (0xFC53, 'M', 'هى'), - (0xFC54, 'M', 'هي'), - (0xFC55, 'M', 'يج'), - (0xFC56, 'M', 'يح'), - ] - -def _seg_46() -> List[Union[Tuple[int, str], Tuple[int, str, str]]]: - return [ - (0xFC57, 'M', 'يخ'), - (0xFC58, 'M', 'يم'), - (0xFC59, 'M', 'يى'), - (0xFC5A, 'M', 'يي'), - (0xFC5B, 'M', 'ذٰ'), - (0xFC5C, 'M', 'رٰ'), - (0xFC5D, 'M', 'ىٰ'), - (0xFC5E, '3', ' ٌّ'), - (0xFC5F, '3', ' ٍّ'), - (0xFC60, '3', ' َّ'), - (0xFC61, '3', ' ُّ'), - (0xFC62, '3', ' ِّ'), - (0xFC63, '3', ' ّٰ'), - (0xFC64, 'M', 'ئر'), - (0xFC65, 'M', 'ئز'), - (0xFC66, 'M', 'ئم'), - (0xFC67, 'M', 'ئن'), - (0xFC68, 'M', 'ئى'), - (0xFC69, 'M', 'ئي'), - (0xFC6A, 'M', 'بر'), - (0xFC6B, 'M', 'بز'), - (0xFC6C, 'M', 'بم'), - (0xFC6D, 'M', 'بن'), - (0xFC6E, 'M', 'بى'), - (0xFC6F, 'M', 'بي'), - (0xFC70, 'M', 'تر'), - (0xFC71, 'M', 'تز'), - (0xFC72, 'M', 'تم'), - (0xFC73, 'M', 'تن'), - (0xFC74, 'M', 'تى'), - (0xFC75, 'M', 'تي'), - (0xFC76, 'M', 'ثر'), - (0xFC77, 'M', 'ثز'), - (0xFC78, 'M', 'ثم'), - (0xFC79, 'M', 'ثن'), - (0xFC7A, 'M', 'ثى'), - (0xFC7B, 'M', 'ثي'), - (0xFC7C, 'M', 'فى'), - (0xFC7D, 'M', 'في'), - (0xFC7E, 'M', 'قى'), - (0xFC7F, 'M', 'قي'), - (0xFC80, 'M', 'كا'), - (0xFC81, 'M', 'كل'), - (0xFC82, 'M', 'كم'), - (0xFC83, 'M', 'كى'), - (0xFC84, 'M', 'كي'), - (0xFC85, 'M', 'لم'), - (0xFC86, 'M', 'لى'), - (0xFC87, 'M', 'لي'), - (0xFC88, 'M', 'ما'), - (0xFC89, 'M', 'مم'), - (0xFC8A, 'M', 'نر'), - (0xFC8B, 'M', 'نز'), - (0xFC8C, 'M', 'نم'), - (0xFC8D, 'M', 'نن'), - (0xFC8E, 'M', 'نى'), - (0xFC8F, 'M', 'ني'), - (0xFC90, 'M', 'ىٰ'), - (0xFC91, 'M', 'ير'), - (0xFC92, 'M', 'يز'), - (0xFC93, 'M', 'يم'), - (0xFC94, 'M', 'ين'), - (0xFC95, 'M', 'يى'), - (0xFC96, 'M', 'يي'), - (0xFC97, 'M', 'ئج'), - (0xFC98, 'M', 'ئح'), - (0xFC99, 'M', 'ئخ'), - (0xFC9A, 'M', 'ئم'), - (0xFC9B, 'M', 'ئه'), - (0xFC9C, 'M', 'بج'), - (0xFC9D, 'M', 'بح'), - (0xFC9E, 'M', 'بخ'), - (0xFC9F, 'M', 'بم'), - (0xFCA0, 'M', 'به'), - (0xFCA1, 'M', 'تج'), - (0xFCA2, 'M', 'تح'), - (0xFCA3, 'M', 'تخ'), - (0xFCA4, 'M', 'تم'), - (0xFCA5, 'M', 'ته'), - (0xFCA6, 'M', 'ثم'), - (0xFCA7, 'M', 'جح'), - (0xFCA8, 'M', 'جم'), - (0xFCA9, 'M', 'حج'), - (0xFCAA, 'M', 'حم'), - (0xFCAB, 'M', 'خج'), - (0xFCAC, 'M', 'خم'), - (0xFCAD, 'M', 'سج'), - (0xFCAE, 'M', 'سح'), - (0xFCAF, 'M', 'سخ'), - (0xFCB0, 'M', 'سم'), - (0xFCB1, 'M', 'صح'), - (0xFCB2, 'M', 'صخ'), - (0xFCB3, 'M', 'صم'), - (0xFCB4, 'M', 'ضج'), - (0xFCB5, 'M', 'ضح'), - (0xFCB6, 'M', 'ضخ'), - (0xFCB7, 'M', 'ضم'), - (0xFCB8, 'M', 'طح'), - (0xFCB9, 'M', 'ظم'), - (0xFCBA, 'M', 'عج'), - ] - -def _seg_47() -> List[Union[Tuple[int, str], Tuple[int, str, str]]]: - return [ - (0xFCBB, 'M', 'عم'), - (0xFCBC, 'M', 'غج'), - (0xFCBD, 'M', 'غم'), - (0xFCBE, 'M', 'فج'), - (0xFCBF, 'M', 'فح'), - (0xFCC0, 'M', 'فخ'), - (0xFCC1, 'M', 'فم'), - (0xFCC2, 'M', 'قح'), - (0xFCC3, 'M', 'قم'), - (0xFCC4, 'M', 'كج'), - (0xFCC5, 'M', 'كح'), - (0xFCC6, 'M', 'كخ'), - (0xFCC7, 'M', 'كل'), - (0xFCC8, 'M', 'كم'), - (0xFCC9, 'M', 'لج'), - (0xFCCA, 'M', 'لح'), - (0xFCCB, 'M', 'لخ'), - (0xFCCC, 'M', 'لم'), - (0xFCCD, 'M', 'له'), - (0xFCCE, 'M', 'مج'), - (0xFCCF, 'M', 'مح'), - (0xFCD0, 'M', 'مخ'), - (0xFCD1, 'M', 'مم'), - (0xFCD2, 'M', 'نج'), - (0xFCD3, 'M', 'نح'), - (0xFCD4, 'M', 'نخ'), - (0xFCD5, 'M', 'نم'), - (0xFCD6, 'M', 'نه'), - (0xFCD7, 'M', 'هج'), - (0xFCD8, 'M', 'هم'), - (0xFCD9, 'M', 'هٰ'), - (0xFCDA, 'M', 'يج'), - (0xFCDB, 'M', 'يح'), - (0xFCDC, 'M', 'يخ'), - (0xFCDD, 'M', 'يم'), - (0xFCDE, 'M', 'يه'), - (0xFCDF, 'M', 'ئم'), - (0xFCE0, 'M', 'ئه'), - (0xFCE1, 'M', 'بم'), - (0xFCE2, 'M', 'به'), - (0xFCE3, 'M', 'تم'), - (0xFCE4, 'M', 'ته'), - (0xFCE5, 'M', 'ثم'), - (0xFCE6, 'M', 'ثه'), - (0xFCE7, 'M', 'سم'), - (0xFCE8, 'M', 'سه'), - (0xFCE9, 'M', 'شم'), - (0xFCEA, 'M', 'شه'), - (0xFCEB, 'M', 'كل'), - (0xFCEC, 'M', 'كم'), - (0xFCED, 'M', 'لم'), - (0xFCEE, 'M', 'نم'), - (0xFCEF, 'M', 'نه'), - (0xFCF0, 'M', 'يم'), - (0xFCF1, 'M', 'يه'), - (0xFCF2, 'M', 'ـَّ'), - (0xFCF3, 'M', 'ـُّ'), - (0xFCF4, 'M', 'ـِّ'), - (0xFCF5, 'M', 'طى'), - (0xFCF6, 'M', 'طي'), - (0xFCF7, 'M', 'عى'), - (0xFCF8, 'M', 'عي'), - (0xFCF9, 'M', 'غى'), - (0xFCFA, 'M', 'غي'), - (0xFCFB, 'M', 'سى'), - (0xFCFC, 'M', 'سي'), - (0xFCFD, 'M', 'شى'), - (0xFCFE, 'M', 'شي'), - (0xFCFF, 'M', 'حى'), - (0xFD00, 'M', 'حي'), - (0xFD01, 'M', 'جى'), - (0xFD02, 'M', 'جي'), - (0xFD03, 'M', 'خى'), - (0xFD04, 'M', 'خي'), - (0xFD05, 'M', 'صى'), - (0xFD06, 'M', 'صي'), - (0xFD07, 'M', 'ضى'), - (0xFD08, 'M', 'ضي'), - (0xFD09, 'M', 'شج'), - (0xFD0A, 'M', 'شح'), - (0xFD0B, 'M', 'شخ'), - (0xFD0C, 'M', 'شم'), - (0xFD0D, 'M', 'شر'), - (0xFD0E, 'M', 'سر'), - (0xFD0F, 'M', 'صر'), - (0xFD10, 'M', 'ضر'), - (0xFD11, 'M', 'طى'), - (0xFD12, 'M', 'طي'), - (0xFD13, 'M', 'عى'), - (0xFD14, 'M', 'عي'), - (0xFD15, 'M', 'غى'), - (0xFD16, 'M', 'غي'), - (0xFD17, 'M', 'سى'), - (0xFD18, 'M', 'سي'), - (0xFD19, 'M', 'شى'), - (0xFD1A, 'M', 'شي'), - (0xFD1B, 'M', 'حى'), - (0xFD1C, 'M', 'حي'), - (0xFD1D, 'M', 'جى'), - (0xFD1E, 'M', 'جي'), - ] - -def _seg_48() -> List[Union[Tuple[int, str], Tuple[int, str, str]]]: - return [ - (0xFD1F, 'M', 'خى'), - (0xFD20, 'M', 'خي'), - (0xFD21, 'M', 'صى'), - (0xFD22, 'M', 'صي'), - (0xFD23, 'M', 'ضى'), - (0xFD24, 'M', 'ضي'), - (0xFD25, 'M', 'شج'), - (0xFD26, 'M', 'شح'), - (0xFD27, 'M', 'شخ'), - (0xFD28, 'M', 'شم'), - (0xFD29, 'M', 'شر'), - (0xFD2A, 'M', 'سر'), - (0xFD2B, 'M', 'صر'), - (0xFD2C, 'M', 'ضر'), - (0xFD2D, 'M', 'شج'), - (0xFD2E, 'M', 'شح'), - (0xFD2F, 'M', 'شخ'), - (0xFD30, 'M', 'شم'), - (0xFD31, 'M', 'سه'), - (0xFD32, 'M', 'شه'), - (0xFD33, 'M', 'طم'), - (0xFD34, 'M', 'سج'), - (0xFD35, 'M', 'سح'), - (0xFD36, 'M', 'سخ'), - (0xFD37, 'M', 'شج'), - (0xFD38, 'M', 'شح'), - (0xFD39, 'M', 'شخ'), - (0xFD3A, 'M', 'طم'), - (0xFD3B, 'M', 'ظم'), - (0xFD3C, 'M', 'اً'), - (0xFD3E, 'V'), - (0xFD50, 'M', 'تجم'), - (0xFD51, 'M', 'تحج'), - (0xFD53, 'M', 'تحم'), - (0xFD54, 'M', 'تخم'), - (0xFD55, 'M', 'تمج'), - (0xFD56, 'M', 'تمح'), - (0xFD57, 'M', 'تمخ'), - (0xFD58, 'M', 'جمح'), - (0xFD5A, 'M', 'حمي'), - (0xFD5B, 'M', 'حمى'), - (0xFD5C, 'M', 'سحج'), - (0xFD5D, 'M', 'سجح'), - (0xFD5E, 'M', 'سجى'), - (0xFD5F, 'M', 'سمح'), - (0xFD61, 'M', 'سمج'), - (0xFD62, 'M', 'سمم'), - (0xFD64, 'M', 'صحح'), - (0xFD66, 'M', 'صمم'), - (0xFD67, 'M', 'شحم'), - (0xFD69, 'M', 'شجي'), - (0xFD6A, 'M', 'شمخ'), - (0xFD6C, 'M', 'شمم'), - (0xFD6E, 'M', 'ضحى'), - (0xFD6F, 'M', 'ضخم'), - (0xFD71, 'M', 'طمح'), - (0xFD73, 'M', 'طمم'), - (0xFD74, 'M', 'طمي'), - (0xFD75, 'M', 'عجم'), - (0xFD76, 'M', 'عمم'), - (0xFD78, 'M', 'عمى'), - (0xFD79, 'M', 'غمم'), - (0xFD7A, 'M', 'غمي'), - (0xFD7B, 'M', 'غمى'), - (0xFD7C, 'M', 'فخم'), - (0xFD7E, 'M', 'قمح'), - (0xFD7F, 'M', 'قمم'), - (0xFD80, 'M', 'لحم'), - (0xFD81, 'M', 'لحي'), - (0xFD82, 'M', 'لحى'), - (0xFD83, 'M', 'لجج'), - (0xFD85, 'M', 'لخم'), - (0xFD87, 'M', 'لمح'), - (0xFD89, 'M', 'محج'), - (0xFD8A, 'M', 'محم'), - (0xFD8B, 'M', 'محي'), - (0xFD8C, 'M', 'مجح'), - (0xFD8D, 'M', 'مجم'), - (0xFD8E, 'M', 'مخج'), - (0xFD8F, 'M', 'مخم'), - (0xFD90, 'X'), - (0xFD92, 'M', 'مجخ'), - (0xFD93, 'M', 'همج'), - (0xFD94, 'M', 'همم'), - (0xFD95, 'M', 'نحم'), - (0xFD96, 'M', 'نحى'), - (0xFD97, 'M', 'نجم'), - (0xFD99, 'M', 'نجى'), - (0xFD9A, 'M', 'نمي'), - (0xFD9B, 'M', 'نمى'), - (0xFD9C, 'M', 'يمم'), - (0xFD9E, 'M', 'بخي'), - (0xFD9F, 'M', 'تجي'), - (0xFDA0, 'M', 'تجى'), - (0xFDA1, 'M', 'تخي'), - (0xFDA2, 'M', 'تخى'), - (0xFDA3, 'M', 'تمي'), - (0xFDA4, 'M', 'تمى'), - (0xFDA5, 'M', 'جمي'), - (0xFDA6, 'M', 'جحى'), - ] - -def _seg_49() -> List[Union[Tuple[int, str], Tuple[int, str, str]]]: - return [ - (0xFDA7, 'M', 'جمى'), - (0xFDA8, 'M', 'سخى'), - (0xFDA9, 'M', 'صحي'), - (0xFDAA, 'M', 'شحي'), - (0xFDAB, 'M', 'ضحي'), - (0xFDAC, 'M', 'لجي'), - (0xFDAD, 'M', 'لمي'), - (0xFDAE, 'M', 'يحي'), - (0xFDAF, 'M', 'يجي'), - (0xFDB0, 'M', 'يمي'), - (0xFDB1, 'M', 'ممي'), - (0xFDB2, 'M', 'قمي'), - (0xFDB3, 'M', 'نحي'), - (0xFDB4, 'M', 'قمح'), - (0xFDB5, 'M', 'لحم'), - (0xFDB6, 'M', 'عمي'), - (0xFDB7, 'M', 'كمي'), - (0xFDB8, 'M', 'نجح'), - (0xFDB9, 'M', 'مخي'), - (0xFDBA, 'M', 'لجم'), - (0xFDBB, 'M', 'كمم'), - (0xFDBC, 'M', 'لجم'), - (0xFDBD, 'M', 'نجح'), - (0xFDBE, 'M', 'جحي'), - (0xFDBF, 'M', 'حجي'), - (0xFDC0, 'M', 'مجي'), - (0xFDC1, 'M', 'فمي'), - (0xFDC2, 'M', 'بحي'), - (0xFDC3, 'M', 'كمم'), - (0xFDC4, 'M', 'عجم'), - (0xFDC5, 'M', 'صمم'), - (0xFDC6, 'M', 'سخي'), - (0xFDC7, 'M', 'نجي'), - (0xFDC8, 'X'), - (0xFDCF, 'V'), - (0xFDD0, 'X'), - (0xFDF0, 'M', 'صلے'), - (0xFDF1, 'M', 'قلے'), - (0xFDF2, 'M', 'الله'), - (0xFDF3, 'M', 'اكبر'), - (0xFDF4, 'M', 'محمد'), - (0xFDF5, 'M', 'صلعم'), - (0xFDF6, 'M', 'رسول'), - (0xFDF7, 'M', 'عليه'), - (0xFDF8, 'M', 'وسلم'), - (0xFDF9, 'M', 'صلى'), - (0xFDFA, '3', 'صلى الله عليه وسلم'), - (0xFDFB, '3', 'جل جلاله'), - (0xFDFC, 'M', 'ریال'), - (0xFDFD, 'V'), - (0xFE00, 'I'), - (0xFE10, '3', ','), - (0xFE11, 'M', '、'), - (0xFE12, 'X'), - (0xFE13, '3', ':'), - (0xFE14, '3', ';'), - (0xFE15, '3', '!'), - (0xFE16, '3', '?'), - (0xFE17, 'M', '〖'), - (0xFE18, 'M', '〗'), - (0xFE19, 'X'), - (0xFE20, 'V'), - (0xFE30, 'X'), - (0xFE31, 'M', '—'), - (0xFE32, 'M', '–'), - (0xFE33, '3', '_'), - (0xFE35, '3', '('), - (0xFE36, '3', ')'), - (0xFE37, '3', '{'), - (0xFE38, '3', '}'), - (0xFE39, 'M', '〔'), - (0xFE3A, 'M', '〕'), - (0xFE3B, 'M', '【'), - (0xFE3C, 'M', '】'), - (0xFE3D, 'M', '《'), - (0xFE3E, 'M', '》'), - (0xFE3F, 'M', '〈'), - (0xFE40, 'M', '〉'), - (0xFE41, 'M', '「'), - (0xFE42, 'M', '」'), - (0xFE43, 'M', '『'), - (0xFE44, 'M', '』'), - (0xFE45, 'V'), - (0xFE47, '3', '['), - (0xFE48, '3', ']'), - (0xFE49, '3', ' ̅'), - (0xFE4D, '3', '_'), - (0xFE50, '3', ','), - (0xFE51, 'M', '、'), - (0xFE52, 'X'), - (0xFE54, '3', ';'), - (0xFE55, '3', ':'), - (0xFE56, '3', '?'), - (0xFE57, '3', '!'), - (0xFE58, 'M', '—'), - (0xFE59, '3', '('), - (0xFE5A, '3', ')'), - (0xFE5B, '3', '{'), - (0xFE5C, '3', '}'), - (0xFE5D, 'M', '〔'), - ] - -def _seg_50() -> List[Union[Tuple[int, str], Tuple[int, str, str]]]: - return [ - (0xFE5E, 'M', '〕'), - (0xFE5F, '3', '#'), - (0xFE60, '3', '&'), - (0xFE61, '3', '*'), - (0xFE62, '3', '+'), - (0xFE63, 'M', '-'), - (0xFE64, '3', '<'), - (0xFE65, '3', '>'), - (0xFE66, '3', '='), - (0xFE67, 'X'), - (0xFE68, '3', '\\'), - (0xFE69, '3', '$'), - (0xFE6A, '3', '%'), - (0xFE6B, '3', '@'), - (0xFE6C, 'X'), - (0xFE70, '3', ' ً'), - (0xFE71, 'M', 'ـً'), - (0xFE72, '3', ' ٌ'), - (0xFE73, 'V'), - (0xFE74, '3', ' ٍ'), - (0xFE75, 'X'), - (0xFE76, '3', ' َ'), - (0xFE77, 'M', 'ـَ'), - (0xFE78, '3', ' ُ'), - (0xFE79, 'M', 'ـُ'), - (0xFE7A, '3', ' ِ'), - (0xFE7B, 'M', 'ـِ'), - (0xFE7C, '3', ' ّ'), - (0xFE7D, 'M', 'ـّ'), - (0xFE7E, '3', ' ْ'), - (0xFE7F, 'M', 'ـْ'), - (0xFE80, 'M', 'ء'), - (0xFE81, 'M', 'آ'), - (0xFE83, 'M', 'أ'), - (0xFE85, 'M', 'ؤ'), - (0xFE87, 'M', 'إ'), - (0xFE89, 'M', 'ئ'), - (0xFE8D, 'M', 'ا'), - (0xFE8F, 'M', 'ب'), - (0xFE93, 'M', 'ة'), - (0xFE95, 'M', 'ت'), - (0xFE99, 'M', 'ث'), - (0xFE9D, 'M', 'ج'), - (0xFEA1, 'M', 'ح'), - (0xFEA5, 'M', 'خ'), - (0xFEA9, 'M', 'د'), - (0xFEAB, 'M', 'ذ'), - (0xFEAD, 'M', 'ر'), - (0xFEAF, 'M', 'ز'), - (0xFEB1, 'M', 'س'), - (0xFEB5, 'M', 'ش'), - (0xFEB9, 'M', 'ص'), - (0xFEBD, 'M', 'ض'), - (0xFEC1, 'M', 'ط'), - (0xFEC5, 'M', 'ظ'), - (0xFEC9, 'M', 'ع'), - (0xFECD, 'M', 'غ'), - (0xFED1, 'M', 'ف'), - (0xFED5, 'M', 'ق'), - (0xFED9, 'M', 'ك'), - (0xFEDD, 'M', 'ل'), - (0xFEE1, 'M', 'م'), - (0xFEE5, 'M', 'ن'), - (0xFEE9, 'M', 'ه'), - (0xFEED, 'M', 'و'), - (0xFEEF, 'M', 'ى'), - (0xFEF1, 'M', 'ي'), - (0xFEF5, 'M', 'لآ'), - (0xFEF7, 'M', 'لأ'), - (0xFEF9, 'M', 'لإ'), - (0xFEFB, 'M', 'لا'), - (0xFEFD, 'X'), - (0xFEFF, 'I'), - (0xFF00, 'X'), - (0xFF01, '3', '!'), - (0xFF02, '3', '"'), - (0xFF03, '3', '#'), - (0xFF04, '3', '$'), - (0xFF05, '3', '%'), - (0xFF06, '3', '&'), - (0xFF07, '3', '\''), - (0xFF08, '3', '('), - (0xFF09, '3', ')'), - (0xFF0A, '3', '*'), - (0xFF0B, '3', '+'), - (0xFF0C, '3', ','), - (0xFF0D, 'M', '-'), - (0xFF0E, 'M', '.'), - (0xFF0F, '3', '/'), - (0xFF10, 'M', '0'), - (0xFF11, 'M', '1'), - (0xFF12, 'M', '2'), - (0xFF13, 'M', '3'), - (0xFF14, 'M', '4'), - (0xFF15, 'M', '5'), - (0xFF16, 'M', '6'), - (0xFF17, 'M', '7'), - (0xFF18, 'M', '8'), - (0xFF19, 'M', '9'), - (0xFF1A, '3', ':'), - ] - -def _seg_51() -> List[Union[Tuple[int, str], Tuple[int, str, str]]]: - return [ - (0xFF1B, '3', ';'), - (0xFF1C, '3', '<'), - (0xFF1D, '3', '='), - (0xFF1E, '3', '>'), - (0xFF1F, '3', '?'), - (0xFF20, '3', '@'), - (0xFF21, 'M', 'a'), - (0xFF22, 'M', 'b'), - (0xFF23, 'M', 'c'), - (0xFF24, 'M', 'd'), - (0xFF25, 'M', 'e'), - (0xFF26, 'M', 'f'), - (0xFF27, 'M', 'g'), - (0xFF28, 'M', 'h'), - (0xFF29, 'M', 'i'), - (0xFF2A, 'M', 'j'), - (0xFF2B, 'M', 'k'), - (0xFF2C, 'M', 'l'), - (0xFF2D, 'M', 'm'), - (0xFF2E, 'M', 'n'), - (0xFF2F, 'M', 'o'), - (0xFF30, 'M', 'p'), - (0xFF31, 'M', 'q'), - (0xFF32, 'M', 'r'), - (0xFF33, 'M', 's'), - (0xFF34, 'M', 't'), - (0xFF35, 'M', 'u'), - (0xFF36, 'M', 'v'), - (0xFF37, 'M', 'w'), - (0xFF38, 'M', 'x'), - (0xFF39, 'M', 'y'), - (0xFF3A, 'M', 'z'), - (0xFF3B, '3', '['), - (0xFF3C, '3', '\\'), - (0xFF3D, '3', ']'), - (0xFF3E, '3', '^'), - (0xFF3F, '3', '_'), - (0xFF40, '3', '`'), - (0xFF41, 'M', 'a'), - (0xFF42, 'M', 'b'), - (0xFF43, 'M', 'c'), - (0xFF44, 'M', 'd'), - (0xFF45, 'M', 'e'), - (0xFF46, 'M', 'f'), - (0xFF47, 'M', 'g'), - (0xFF48, 'M', 'h'), - (0xFF49, 'M', 'i'), - (0xFF4A, 'M', 'j'), - (0xFF4B, 'M', 'k'), - (0xFF4C, 'M', 'l'), - (0xFF4D, 'M', 'm'), - (0xFF4E, 'M', 'n'), - (0xFF4F, 'M', 'o'), - (0xFF50, 'M', 'p'), - (0xFF51, 'M', 'q'), - (0xFF52, 'M', 'r'), - (0xFF53, 'M', 's'), - (0xFF54, 'M', 't'), - (0xFF55, 'M', 'u'), - (0xFF56, 'M', 'v'), - (0xFF57, 'M', 'w'), - (0xFF58, 'M', 'x'), - (0xFF59, 'M', 'y'), - (0xFF5A, 'M', 'z'), - (0xFF5B, '3', '{'), - (0xFF5C, '3', '|'), - (0xFF5D, '3', '}'), - (0xFF5E, '3', '~'), - (0xFF5F, 'M', '⦅'), - (0xFF60, 'M', '⦆'), - (0xFF61, 'M', '.'), - (0xFF62, 'M', '「'), - (0xFF63, 'M', '」'), - (0xFF64, 'M', '、'), - (0xFF65, 'M', '・'), - (0xFF66, 'M', 'ヲ'), - (0xFF67, 'M', 'ァ'), - (0xFF68, 'M', 'ィ'), - (0xFF69, 'M', 'ゥ'), - (0xFF6A, 'M', 'ェ'), - (0xFF6B, 'M', 'ォ'), - (0xFF6C, 'M', 'ャ'), - (0xFF6D, 'M', 'ュ'), - (0xFF6E, 'M', 'ョ'), - (0xFF6F, 'M', 'ッ'), - (0xFF70, 'M', 'ー'), - (0xFF71, 'M', 'ア'), - (0xFF72, 'M', 'イ'), - (0xFF73, 'M', 'ウ'), - (0xFF74, 'M', 'エ'), - (0xFF75, 'M', 'オ'), - (0xFF76, 'M', 'カ'), - (0xFF77, 'M', 'キ'), - (0xFF78, 'M', 'ク'), - (0xFF79, 'M', 'ケ'), - (0xFF7A, 'M', 'コ'), - (0xFF7B, 'M', 'サ'), - (0xFF7C, 'M', 'シ'), - (0xFF7D, 'M', 'ス'), - (0xFF7E, 'M', 'セ'), - ] - -def _seg_52() -> List[Union[Tuple[int, str], Tuple[int, str, str]]]: - return [ - (0xFF7F, 'M', 'ソ'), - (0xFF80, 'M', 'タ'), - (0xFF81, 'M', 'チ'), - (0xFF82, 'M', 'ツ'), - (0xFF83, 'M', 'テ'), - (0xFF84, 'M', 'ト'), - (0xFF85, 'M', 'ナ'), - (0xFF86, 'M', 'ニ'), - (0xFF87, 'M', 'ヌ'), - (0xFF88, 'M', 'ネ'), - (0xFF89, 'M', 'ノ'), - (0xFF8A, 'M', 'ハ'), - (0xFF8B, 'M', 'ヒ'), - (0xFF8C, 'M', 'フ'), - (0xFF8D, 'M', 'ヘ'), - (0xFF8E, 'M', 'ホ'), - (0xFF8F, 'M', 'マ'), - (0xFF90, 'M', 'ミ'), - (0xFF91, 'M', 'ム'), - (0xFF92, 'M', 'メ'), - (0xFF93, 'M', 'モ'), - (0xFF94, 'M', 'ヤ'), - (0xFF95, 'M', 'ユ'), - (0xFF96, 'M', 'ヨ'), - (0xFF97, 'M', 'ラ'), - (0xFF98, 'M', 'リ'), - (0xFF99, 'M', 'ル'), - (0xFF9A, 'M', 'レ'), - (0xFF9B, 'M', 'ロ'), - (0xFF9C, 'M', 'ワ'), - (0xFF9D, 'M', 'ン'), - (0xFF9E, 'M', '゙'), - (0xFF9F, 'M', '゚'), - (0xFFA0, 'X'), - (0xFFA1, 'M', 'ᄀ'), - (0xFFA2, 'M', 'ᄁ'), - (0xFFA3, 'M', 'ᆪ'), - (0xFFA4, 'M', 'ᄂ'), - (0xFFA5, 'M', 'ᆬ'), - (0xFFA6, 'M', 'ᆭ'), - (0xFFA7, 'M', 'ᄃ'), - (0xFFA8, 'M', 'ᄄ'), - (0xFFA9, 'M', 'ᄅ'), - (0xFFAA, 'M', 'ᆰ'), - (0xFFAB, 'M', 'ᆱ'), - (0xFFAC, 'M', 'ᆲ'), - (0xFFAD, 'M', 'ᆳ'), - (0xFFAE, 'M', 'ᆴ'), - (0xFFAF, 'M', 'ᆵ'), - (0xFFB0, 'M', 'ᄚ'), - (0xFFB1, 'M', 'ᄆ'), - (0xFFB2, 'M', 'ᄇ'), - (0xFFB3, 'M', 'ᄈ'), - (0xFFB4, 'M', 'ᄡ'), - (0xFFB5, 'M', 'ᄉ'), - (0xFFB6, 'M', 'ᄊ'), - (0xFFB7, 'M', 'ᄋ'), - (0xFFB8, 'M', 'ᄌ'), - (0xFFB9, 'M', 'ᄍ'), - (0xFFBA, 'M', 'ᄎ'), - (0xFFBB, 'M', 'ᄏ'), - (0xFFBC, 'M', 'ᄐ'), - (0xFFBD, 'M', 'ᄑ'), - (0xFFBE, 'M', 'ᄒ'), - (0xFFBF, 'X'), - (0xFFC2, 'M', 'ᅡ'), - (0xFFC3, 'M', 'ᅢ'), - (0xFFC4, 'M', 'ᅣ'), - (0xFFC5, 'M', 'ᅤ'), - (0xFFC6, 'M', 'ᅥ'), - (0xFFC7, 'M', 'ᅦ'), - (0xFFC8, 'X'), - (0xFFCA, 'M', 'ᅧ'), - (0xFFCB, 'M', 'ᅨ'), - (0xFFCC, 'M', 'ᅩ'), - (0xFFCD, 'M', 'ᅪ'), - (0xFFCE, 'M', 'ᅫ'), - (0xFFCF, 'M', 'ᅬ'), - (0xFFD0, 'X'), - (0xFFD2, 'M', 'ᅭ'), - (0xFFD3, 'M', 'ᅮ'), - (0xFFD4, 'M', 'ᅯ'), - (0xFFD5, 'M', 'ᅰ'), - (0xFFD6, 'M', 'ᅱ'), - (0xFFD7, 'M', 'ᅲ'), - (0xFFD8, 'X'), - (0xFFDA, 'M', 'ᅳ'), - (0xFFDB, 'M', 'ᅴ'), - (0xFFDC, 'M', 'ᅵ'), - (0xFFDD, 'X'), - (0xFFE0, 'M', '¢'), - (0xFFE1, 'M', '£'), - (0xFFE2, 'M', '¬'), - (0xFFE3, '3', ' ̄'), - (0xFFE4, 'M', '¦'), - (0xFFE5, 'M', '¥'), - (0xFFE6, 'M', '₩'), - (0xFFE7, 'X'), - (0xFFE8, 'M', '│'), - (0xFFE9, 'M', '←'), - ] - -def _seg_53() -> List[Union[Tuple[int, str], Tuple[int, str, str]]]: - return [ - (0xFFEA, 'M', '↑'), - (0xFFEB, 'M', '→'), - (0xFFEC, 'M', '↓'), - (0xFFED, 'M', '■'), - (0xFFEE, 'M', '○'), - (0xFFEF, 'X'), - (0x10000, 'V'), - (0x1000C, 'X'), - (0x1000D, 'V'), - (0x10027, 'X'), - (0x10028, 'V'), - (0x1003B, 'X'), - (0x1003C, 'V'), - (0x1003E, 'X'), - (0x1003F, 'V'), - (0x1004E, 'X'), - (0x10050, 'V'), - (0x1005E, 'X'), - (0x10080, 'V'), - (0x100FB, 'X'), - (0x10100, 'V'), - (0x10103, 'X'), - (0x10107, 'V'), - (0x10134, 'X'), - (0x10137, 'V'), - (0x1018F, 'X'), - (0x10190, 'V'), - (0x1019D, 'X'), - (0x101A0, 'V'), - (0x101A1, 'X'), - (0x101D0, 'V'), - (0x101FE, 'X'), - (0x10280, 'V'), - (0x1029D, 'X'), - (0x102A0, 'V'), - (0x102D1, 'X'), - (0x102E0, 'V'), - (0x102FC, 'X'), - (0x10300, 'V'), - (0x10324, 'X'), - (0x1032D, 'V'), - (0x1034B, 'X'), - (0x10350, 'V'), - (0x1037B, 'X'), - (0x10380, 'V'), - (0x1039E, 'X'), - (0x1039F, 'V'), - (0x103C4, 'X'), - (0x103C8, 'V'), - (0x103D6, 'X'), - (0x10400, 'M', '𐐨'), - (0x10401, 'M', '𐐩'), - (0x10402, 'M', '𐐪'), - (0x10403, 'M', '𐐫'), - (0x10404, 'M', '𐐬'), - (0x10405, 'M', '𐐭'), - (0x10406, 'M', '𐐮'), - (0x10407, 'M', '𐐯'), - (0x10408, 'M', '𐐰'), - (0x10409, 'M', '𐐱'), - (0x1040A, 'M', '𐐲'), - (0x1040B, 'M', '𐐳'), - (0x1040C, 'M', '𐐴'), - (0x1040D, 'M', '𐐵'), - (0x1040E, 'M', '𐐶'), - (0x1040F, 'M', '𐐷'), - (0x10410, 'M', '𐐸'), - (0x10411, 'M', '𐐹'), - (0x10412, 'M', '𐐺'), - (0x10413, 'M', '𐐻'), - (0x10414, 'M', '𐐼'), - (0x10415, 'M', '𐐽'), - (0x10416, 'M', '𐐾'), - (0x10417, 'M', '𐐿'), - (0x10418, 'M', '𐑀'), - (0x10419, 'M', '𐑁'), - (0x1041A, 'M', '𐑂'), - (0x1041B, 'M', '𐑃'), - (0x1041C, 'M', '𐑄'), - (0x1041D, 'M', '𐑅'), - (0x1041E, 'M', '𐑆'), - (0x1041F, 'M', '𐑇'), - (0x10420, 'M', '𐑈'), - (0x10421, 'M', '𐑉'), - (0x10422, 'M', '𐑊'), - (0x10423, 'M', '𐑋'), - (0x10424, 'M', '𐑌'), - (0x10425, 'M', '𐑍'), - (0x10426, 'M', '𐑎'), - (0x10427, 'M', '𐑏'), - (0x10428, 'V'), - (0x1049E, 'X'), - (0x104A0, 'V'), - (0x104AA, 'X'), - (0x104B0, 'M', '𐓘'), - (0x104B1, 'M', '𐓙'), - (0x104B2, 'M', '𐓚'), - (0x104B3, 'M', '𐓛'), - (0x104B4, 'M', '𐓜'), - (0x104B5, 'M', '𐓝'), - ] - -def _seg_54() -> List[Union[Tuple[int, str], Tuple[int, str, str]]]: - return [ - (0x104B6, 'M', '𐓞'), - (0x104B7, 'M', '𐓟'), - (0x104B8, 'M', '𐓠'), - (0x104B9, 'M', '𐓡'), - (0x104BA, 'M', '𐓢'), - (0x104BB, 'M', '𐓣'), - (0x104BC, 'M', '𐓤'), - (0x104BD, 'M', '𐓥'), - (0x104BE, 'M', '𐓦'), - (0x104BF, 'M', '𐓧'), - (0x104C0, 'M', '𐓨'), - (0x104C1, 'M', '𐓩'), - (0x104C2, 'M', '𐓪'), - (0x104C3, 'M', '𐓫'), - (0x104C4, 'M', '𐓬'), - (0x104C5, 'M', '𐓭'), - (0x104C6, 'M', '𐓮'), - (0x104C7, 'M', '𐓯'), - (0x104C8, 'M', '𐓰'), - (0x104C9, 'M', '𐓱'), - (0x104CA, 'M', '𐓲'), - (0x104CB, 'M', '𐓳'), - (0x104CC, 'M', '𐓴'), - (0x104CD, 'M', '𐓵'), - (0x104CE, 'M', '𐓶'), - (0x104CF, 'M', '𐓷'), - (0x104D0, 'M', '𐓸'), - (0x104D1, 'M', '𐓹'), - (0x104D2, 'M', '𐓺'), - (0x104D3, 'M', '𐓻'), - (0x104D4, 'X'), - (0x104D8, 'V'), - (0x104FC, 'X'), - (0x10500, 'V'), - (0x10528, 'X'), - (0x10530, 'V'), - (0x10564, 'X'), - (0x1056F, 'V'), - (0x10570, 'M', '𐖗'), - (0x10571, 'M', '𐖘'), - (0x10572, 'M', '𐖙'), - (0x10573, 'M', '𐖚'), - (0x10574, 'M', '𐖛'), - (0x10575, 'M', '𐖜'), - (0x10576, 'M', '𐖝'), - (0x10577, 'M', '𐖞'), - (0x10578, 'M', '𐖟'), - (0x10579, 'M', '𐖠'), - (0x1057A, 'M', '𐖡'), - (0x1057B, 'X'), - (0x1057C, 'M', '𐖣'), - (0x1057D, 'M', '𐖤'), - (0x1057E, 'M', '𐖥'), - (0x1057F, 'M', '𐖦'), - (0x10580, 'M', '𐖧'), - (0x10581, 'M', '𐖨'), - (0x10582, 'M', '𐖩'), - (0x10583, 'M', '𐖪'), - (0x10584, 'M', '𐖫'), - (0x10585, 'M', '𐖬'), - (0x10586, 'M', '𐖭'), - (0x10587, 'M', '𐖮'), - (0x10588, 'M', '𐖯'), - (0x10589, 'M', '𐖰'), - (0x1058A, 'M', '𐖱'), - (0x1058B, 'X'), - (0x1058C, 'M', '𐖳'), - (0x1058D, 'M', '𐖴'), - (0x1058E, 'M', '𐖵'), - (0x1058F, 'M', '𐖶'), - (0x10590, 'M', '𐖷'), - (0x10591, 'M', '𐖸'), - (0x10592, 'M', '𐖹'), - (0x10593, 'X'), - (0x10594, 'M', '𐖻'), - (0x10595, 'M', '𐖼'), - (0x10596, 'X'), - (0x10597, 'V'), - (0x105A2, 'X'), - (0x105A3, 'V'), - (0x105B2, 'X'), - (0x105B3, 'V'), - (0x105BA, 'X'), - (0x105BB, 'V'), - (0x105BD, 'X'), - (0x10600, 'V'), - (0x10737, 'X'), - (0x10740, 'V'), - (0x10756, 'X'), - (0x10760, 'V'), - (0x10768, 'X'), - (0x10780, 'V'), - (0x10781, 'M', 'ː'), - (0x10782, 'M', 'ˑ'), - (0x10783, 'M', 'æ'), - (0x10784, 'M', 'ʙ'), - (0x10785, 'M', 'ɓ'), - (0x10786, 'X'), - (0x10787, 'M', 'ʣ'), - (0x10788, 'M', 'ꭦ'), - ] - -def _seg_55() -> List[Union[Tuple[int, str], Tuple[int, str, str]]]: - return [ - (0x10789, 'M', 'ʥ'), - (0x1078A, 'M', 'ʤ'), - (0x1078B, 'M', 'ɖ'), - (0x1078C, 'M', 'ɗ'), - (0x1078D, 'M', 'ᶑ'), - (0x1078E, 'M', 'ɘ'), - (0x1078F, 'M', 'ɞ'), - (0x10790, 'M', 'ʩ'), - (0x10791, 'M', 'ɤ'), - (0x10792, 'M', 'ɢ'), - (0x10793, 'M', 'ɠ'), - (0x10794, 'M', 'ʛ'), - (0x10795, 'M', 'ħ'), - (0x10796, 'M', 'ʜ'), - (0x10797, 'M', 'ɧ'), - (0x10798, 'M', 'ʄ'), - (0x10799, 'M', 'ʪ'), - (0x1079A, 'M', 'ʫ'), - (0x1079B, 'M', 'ɬ'), - (0x1079C, 'M', '𝼄'), - (0x1079D, 'M', 'ꞎ'), - (0x1079E, 'M', 'ɮ'), - (0x1079F, 'M', '𝼅'), - (0x107A0, 'M', 'ʎ'), - (0x107A1, 'M', '𝼆'), - (0x107A2, 'M', 'ø'), - (0x107A3, 'M', 'ɶ'), - (0x107A4, 'M', 'ɷ'), - (0x107A5, 'M', 'q'), - (0x107A6, 'M', 'ɺ'), - (0x107A7, 'M', '𝼈'), - (0x107A8, 'M', 'ɽ'), - (0x107A9, 'M', 'ɾ'), - (0x107AA, 'M', 'ʀ'), - (0x107AB, 'M', 'ʨ'), - (0x107AC, 'M', 'ʦ'), - (0x107AD, 'M', 'ꭧ'), - (0x107AE, 'M', 'ʧ'), - (0x107AF, 'M', 'ʈ'), - (0x107B0, 'M', 'ⱱ'), - (0x107B1, 'X'), - (0x107B2, 'M', 'ʏ'), - (0x107B3, 'M', 'ʡ'), - (0x107B4, 'M', 'ʢ'), - (0x107B5, 'M', 'ʘ'), - (0x107B6, 'M', 'ǀ'), - (0x107B7, 'M', 'ǁ'), - (0x107B8, 'M', 'ǂ'), - (0x107B9, 'M', '𝼊'), - (0x107BA, 'M', '𝼞'), - (0x107BB, 'X'), - (0x10800, 'V'), - (0x10806, 'X'), - (0x10808, 'V'), - (0x10809, 'X'), - (0x1080A, 'V'), - (0x10836, 'X'), - (0x10837, 'V'), - (0x10839, 'X'), - (0x1083C, 'V'), - (0x1083D, 'X'), - (0x1083F, 'V'), - (0x10856, 'X'), - (0x10857, 'V'), - (0x1089F, 'X'), - (0x108A7, 'V'), - (0x108B0, 'X'), - (0x108E0, 'V'), - (0x108F3, 'X'), - (0x108F4, 'V'), - (0x108F6, 'X'), - (0x108FB, 'V'), - (0x1091C, 'X'), - (0x1091F, 'V'), - (0x1093A, 'X'), - (0x1093F, 'V'), - (0x10940, 'X'), - (0x10980, 'V'), - (0x109B8, 'X'), - (0x109BC, 'V'), - (0x109D0, 'X'), - (0x109D2, 'V'), - (0x10A04, 'X'), - (0x10A05, 'V'), - (0x10A07, 'X'), - (0x10A0C, 'V'), - (0x10A14, 'X'), - (0x10A15, 'V'), - (0x10A18, 'X'), - (0x10A19, 'V'), - (0x10A36, 'X'), - (0x10A38, 'V'), - (0x10A3B, 'X'), - (0x10A3F, 'V'), - (0x10A49, 'X'), - (0x10A50, 'V'), - (0x10A59, 'X'), - (0x10A60, 'V'), - (0x10AA0, 'X'), - (0x10AC0, 'V'), - ] - -def _seg_56() -> List[Union[Tuple[int, str], Tuple[int, str, str]]]: - return [ - (0x10AE7, 'X'), - (0x10AEB, 'V'), - (0x10AF7, 'X'), - (0x10B00, 'V'), - (0x10B36, 'X'), - (0x10B39, 'V'), - (0x10B56, 'X'), - (0x10B58, 'V'), - (0x10B73, 'X'), - (0x10B78, 'V'), - (0x10B92, 'X'), - (0x10B99, 'V'), - (0x10B9D, 'X'), - (0x10BA9, 'V'), - (0x10BB0, 'X'), - (0x10C00, 'V'), - (0x10C49, 'X'), - (0x10C80, 'M', '𐳀'), - (0x10C81, 'M', '𐳁'), - (0x10C82, 'M', '𐳂'), - (0x10C83, 'M', '𐳃'), - (0x10C84, 'M', '𐳄'), - (0x10C85, 'M', '𐳅'), - (0x10C86, 'M', '𐳆'), - (0x10C87, 'M', '𐳇'), - (0x10C88, 'M', '𐳈'), - (0x10C89, 'M', '𐳉'), - (0x10C8A, 'M', '𐳊'), - (0x10C8B, 'M', '𐳋'), - (0x10C8C, 'M', '𐳌'), - (0x10C8D, 'M', '𐳍'), - (0x10C8E, 'M', '𐳎'), - (0x10C8F, 'M', '𐳏'), - (0x10C90, 'M', '𐳐'), - (0x10C91, 'M', '𐳑'), - (0x10C92, 'M', '𐳒'), - (0x10C93, 'M', '𐳓'), - (0x10C94, 'M', '𐳔'), - (0x10C95, 'M', '𐳕'), - (0x10C96, 'M', '𐳖'), - (0x10C97, 'M', '𐳗'), - (0x10C98, 'M', '𐳘'), - (0x10C99, 'M', '𐳙'), - (0x10C9A, 'M', '𐳚'), - (0x10C9B, 'M', '𐳛'), - (0x10C9C, 'M', '𐳜'), - (0x10C9D, 'M', '𐳝'), - (0x10C9E, 'M', '𐳞'), - (0x10C9F, 'M', '𐳟'), - (0x10CA0, 'M', '𐳠'), - (0x10CA1, 'M', '𐳡'), - (0x10CA2, 'M', '𐳢'), - (0x10CA3, 'M', '𐳣'), - (0x10CA4, 'M', '𐳤'), - (0x10CA5, 'M', '𐳥'), - (0x10CA6, 'M', '𐳦'), - (0x10CA7, 'M', '𐳧'), - (0x10CA8, 'M', '𐳨'), - (0x10CA9, 'M', '𐳩'), - (0x10CAA, 'M', '𐳪'), - (0x10CAB, 'M', '𐳫'), - (0x10CAC, 'M', '𐳬'), - (0x10CAD, 'M', '𐳭'), - (0x10CAE, 'M', '𐳮'), - (0x10CAF, 'M', '𐳯'), - (0x10CB0, 'M', '𐳰'), - (0x10CB1, 'M', '𐳱'), - (0x10CB2, 'M', '𐳲'), - (0x10CB3, 'X'), - (0x10CC0, 'V'), - (0x10CF3, 'X'), - (0x10CFA, 'V'), - (0x10D28, 'X'), - (0x10D30, 'V'), - (0x10D3A, 'X'), - (0x10E60, 'V'), - (0x10E7F, 'X'), - (0x10E80, 'V'), - (0x10EAA, 'X'), - (0x10EAB, 'V'), - (0x10EAE, 'X'), - (0x10EB0, 'V'), - (0x10EB2, 'X'), - (0x10EFD, 'V'), - (0x10F28, 'X'), - (0x10F30, 'V'), - (0x10F5A, 'X'), - (0x10F70, 'V'), - (0x10F8A, 'X'), - (0x10FB0, 'V'), - (0x10FCC, 'X'), - (0x10FE0, 'V'), - (0x10FF7, 'X'), - (0x11000, 'V'), - (0x1104E, 'X'), - (0x11052, 'V'), - (0x11076, 'X'), - (0x1107F, 'V'), - (0x110BD, 'X'), - (0x110BE, 'V'), - ] - -def _seg_57() -> List[Union[Tuple[int, str], Tuple[int, str, str]]]: - return [ - (0x110C3, 'X'), - (0x110D0, 'V'), - (0x110E9, 'X'), - (0x110F0, 'V'), - (0x110FA, 'X'), - (0x11100, 'V'), - (0x11135, 'X'), - (0x11136, 'V'), - (0x11148, 'X'), - (0x11150, 'V'), - (0x11177, 'X'), - (0x11180, 'V'), - (0x111E0, 'X'), - (0x111E1, 'V'), - (0x111F5, 'X'), - (0x11200, 'V'), - (0x11212, 'X'), - (0x11213, 'V'), - (0x11242, 'X'), - (0x11280, 'V'), - (0x11287, 'X'), - (0x11288, 'V'), - (0x11289, 'X'), - (0x1128A, 'V'), - (0x1128E, 'X'), - (0x1128F, 'V'), - (0x1129E, 'X'), - (0x1129F, 'V'), - (0x112AA, 'X'), - (0x112B0, 'V'), - (0x112EB, 'X'), - (0x112F0, 'V'), - (0x112FA, 'X'), - (0x11300, 'V'), - (0x11304, 'X'), - (0x11305, 'V'), - (0x1130D, 'X'), - (0x1130F, 'V'), - (0x11311, 'X'), - (0x11313, 'V'), - (0x11329, 'X'), - (0x1132A, 'V'), - (0x11331, 'X'), - (0x11332, 'V'), - (0x11334, 'X'), - (0x11335, 'V'), - (0x1133A, 'X'), - (0x1133B, 'V'), - (0x11345, 'X'), - (0x11347, 'V'), - (0x11349, 'X'), - (0x1134B, 'V'), - (0x1134E, 'X'), - (0x11350, 'V'), - (0x11351, 'X'), - (0x11357, 'V'), - (0x11358, 'X'), - (0x1135D, 'V'), - (0x11364, 'X'), - (0x11366, 'V'), - (0x1136D, 'X'), - (0x11370, 'V'), - (0x11375, 'X'), - (0x11400, 'V'), - (0x1145C, 'X'), - (0x1145D, 'V'), - (0x11462, 'X'), - (0x11480, 'V'), - (0x114C8, 'X'), - (0x114D0, 'V'), - (0x114DA, 'X'), - (0x11580, 'V'), - (0x115B6, 'X'), - (0x115B8, 'V'), - (0x115DE, 'X'), - (0x11600, 'V'), - (0x11645, 'X'), - (0x11650, 'V'), - (0x1165A, 'X'), - (0x11660, 'V'), - (0x1166D, 'X'), - (0x11680, 'V'), - (0x116BA, 'X'), - (0x116C0, 'V'), - (0x116CA, 'X'), - (0x11700, 'V'), - (0x1171B, 'X'), - (0x1171D, 'V'), - (0x1172C, 'X'), - (0x11730, 'V'), - (0x11747, 'X'), - (0x11800, 'V'), - (0x1183C, 'X'), - (0x118A0, 'M', '𑣀'), - (0x118A1, 'M', '𑣁'), - (0x118A2, 'M', '𑣂'), - (0x118A3, 'M', '𑣃'), - (0x118A4, 'M', '𑣄'), - (0x118A5, 'M', '𑣅'), - (0x118A6, 'M', '𑣆'), - ] - -def _seg_58() -> List[Union[Tuple[int, str], Tuple[int, str, str]]]: - return [ - (0x118A7, 'M', '𑣇'), - (0x118A8, 'M', '𑣈'), - (0x118A9, 'M', '𑣉'), - (0x118AA, 'M', '𑣊'), - (0x118AB, 'M', '𑣋'), - (0x118AC, 'M', '𑣌'), - (0x118AD, 'M', '𑣍'), - (0x118AE, 'M', '𑣎'), - (0x118AF, 'M', '𑣏'), - (0x118B0, 'M', '𑣐'), - (0x118B1, 'M', '𑣑'), - (0x118B2, 'M', '𑣒'), - (0x118B3, 'M', '𑣓'), - (0x118B4, 'M', '𑣔'), - (0x118B5, 'M', '𑣕'), - (0x118B6, 'M', '𑣖'), - (0x118B7, 'M', '𑣗'), - (0x118B8, 'M', '𑣘'), - (0x118B9, 'M', '𑣙'), - (0x118BA, 'M', '𑣚'), - (0x118BB, 'M', '𑣛'), - (0x118BC, 'M', '𑣜'), - (0x118BD, 'M', '𑣝'), - (0x118BE, 'M', '𑣞'), - (0x118BF, 'M', '𑣟'), - (0x118C0, 'V'), - (0x118F3, 'X'), - (0x118FF, 'V'), - (0x11907, 'X'), - (0x11909, 'V'), - (0x1190A, 'X'), - (0x1190C, 'V'), - (0x11914, 'X'), - (0x11915, 'V'), - (0x11917, 'X'), - (0x11918, 'V'), - (0x11936, 'X'), - (0x11937, 'V'), - (0x11939, 'X'), - (0x1193B, 'V'), - (0x11947, 'X'), - (0x11950, 'V'), - (0x1195A, 'X'), - (0x119A0, 'V'), - (0x119A8, 'X'), - (0x119AA, 'V'), - (0x119D8, 'X'), - (0x119DA, 'V'), - (0x119E5, 'X'), - (0x11A00, 'V'), - (0x11A48, 'X'), - (0x11A50, 'V'), - (0x11AA3, 'X'), - (0x11AB0, 'V'), - (0x11AF9, 'X'), - (0x11B00, 'V'), - (0x11B0A, 'X'), - (0x11C00, 'V'), - (0x11C09, 'X'), - (0x11C0A, 'V'), - (0x11C37, 'X'), - (0x11C38, 'V'), - (0x11C46, 'X'), - (0x11C50, 'V'), - (0x11C6D, 'X'), - (0x11C70, 'V'), - (0x11C90, 'X'), - (0x11C92, 'V'), - (0x11CA8, 'X'), - (0x11CA9, 'V'), - (0x11CB7, 'X'), - (0x11D00, 'V'), - (0x11D07, 'X'), - (0x11D08, 'V'), - (0x11D0A, 'X'), - (0x11D0B, 'V'), - (0x11D37, 'X'), - (0x11D3A, 'V'), - (0x11D3B, 'X'), - (0x11D3C, 'V'), - (0x11D3E, 'X'), - (0x11D3F, 'V'), - (0x11D48, 'X'), - (0x11D50, 'V'), - (0x11D5A, 'X'), - (0x11D60, 'V'), - (0x11D66, 'X'), - (0x11D67, 'V'), - (0x11D69, 'X'), - (0x11D6A, 'V'), - (0x11D8F, 'X'), - (0x11D90, 'V'), - (0x11D92, 'X'), - (0x11D93, 'V'), - (0x11D99, 'X'), - (0x11DA0, 'V'), - (0x11DAA, 'X'), - (0x11EE0, 'V'), - (0x11EF9, 'X'), - (0x11F00, 'V'), - ] - -def _seg_59() -> List[Union[Tuple[int, str], Tuple[int, str, str]]]: - return [ - (0x11F11, 'X'), - (0x11F12, 'V'), - (0x11F3B, 'X'), - (0x11F3E, 'V'), - (0x11F5A, 'X'), - (0x11FB0, 'V'), - (0x11FB1, 'X'), - (0x11FC0, 'V'), - (0x11FF2, 'X'), - (0x11FFF, 'V'), - (0x1239A, 'X'), - (0x12400, 'V'), - (0x1246F, 'X'), - (0x12470, 'V'), - (0x12475, 'X'), - (0x12480, 'V'), - (0x12544, 'X'), - (0x12F90, 'V'), - (0x12FF3, 'X'), - (0x13000, 'V'), - (0x13430, 'X'), - (0x13440, 'V'), - (0x13456, 'X'), - (0x14400, 'V'), - (0x14647, 'X'), - (0x16800, 'V'), - (0x16A39, 'X'), - (0x16A40, 'V'), - (0x16A5F, 'X'), - (0x16A60, 'V'), - (0x16A6A, 'X'), - (0x16A6E, 'V'), - (0x16ABF, 'X'), - (0x16AC0, 'V'), - (0x16ACA, 'X'), - (0x16AD0, 'V'), - (0x16AEE, 'X'), - (0x16AF0, 'V'), - (0x16AF6, 'X'), - (0x16B00, 'V'), - (0x16B46, 'X'), - (0x16B50, 'V'), - (0x16B5A, 'X'), - (0x16B5B, 'V'), - (0x16B62, 'X'), - (0x16B63, 'V'), - (0x16B78, 'X'), - (0x16B7D, 'V'), - (0x16B90, 'X'), - (0x16E40, 'M', '𖹠'), - (0x16E41, 'M', '𖹡'), - (0x16E42, 'M', '𖹢'), - (0x16E43, 'M', '𖹣'), - (0x16E44, 'M', '𖹤'), - (0x16E45, 'M', '𖹥'), - (0x16E46, 'M', '𖹦'), - (0x16E47, 'M', '𖹧'), - (0x16E48, 'M', '𖹨'), - (0x16E49, 'M', '𖹩'), - (0x16E4A, 'M', '𖹪'), - (0x16E4B, 'M', '𖹫'), - (0x16E4C, 'M', '𖹬'), - (0x16E4D, 'M', '𖹭'), - (0x16E4E, 'M', '𖹮'), - (0x16E4F, 'M', '𖹯'), - (0x16E50, 'M', '𖹰'), - (0x16E51, 'M', '𖹱'), - (0x16E52, 'M', '𖹲'), - (0x16E53, 'M', '𖹳'), - (0x16E54, 'M', '𖹴'), - (0x16E55, 'M', '𖹵'), - (0x16E56, 'M', '𖹶'), - (0x16E57, 'M', '𖹷'), - (0x16E58, 'M', '𖹸'), - (0x16E59, 'M', '𖹹'), - (0x16E5A, 'M', '𖹺'), - (0x16E5B, 'M', '𖹻'), - (0x16E5C, 'M', '𖹼'), - (0x16E5D, 'M', '𖹽'), - (0x16E5E, 'M', '𖹾'), - (0x16E5F, 'M', '𖹿'), - (0x16E60, 'V'), - (0x16E9B, 'X'), - (0x16F00, 'V'), - (0x16F4B, 'X'), - (0x16F4F, 'V'), - (0x16F88, 'X'), - (0x16F8F, 'V'), - (0x16FA0, 'X'), - (0x16FE0, 'V'), - (0x16FE5, 'X'), - (0x16FF0, 'V'), - (0x16FF2, 'X'), - (0x17000, 'V'), - (0x187F8, 'X'), - (0x18800, 'V'), - (0x18CD6, 'X'), - (0x18D00, 'V'), - (0x18D09, 'X'), - (0x1AFF0, 'V'), - ] - -def _seg_60() -> List[Union[Tuple[int, str], Tuple[int, str, str]]]: - return [ - (0x1AFF4, 'X'), - (0x1AFF5, 'V'), - (0x1AFFC, 'X'), - (0x1AFFD, 'V'), - (0x1AFFF, 'X'), - (0x1B000, 'V'), - (0x1B123, 'X'), - (0x1B132, 'V'), - (0x1B133, 'X'), - (0x1B150, 'V'), - (0x1B153, 'X'), - (0x1B155, 'V'), - (0x1B156, 'X'), - (0x1B164, 'V'), - (0x1B168, 'X'), - (0x1B170, 'V'), - (0x1B2FC, 'X'), - (0x1BC00, 'V'), - (0x1BC6B, 'X'), - (0x1BC70, 'V'), - (0x1BC7D, 'X'), - (0x1BC80, 'V'), - (0x1BC89, 'X'), - (0x1BC90, 'V'), - (0x1BC9A, 'X'), - (0x1BC9C, 'V'), - (0x1BCA0, 'I'), - (0x1BCA4, 'X'), - (0x1CF00, 'V'), - (0x1CF2E, 'X'), - (0x1CF30, 'V'), - (0x1CF47, 'X'), - (0x1CF50, 'V'), - (0x1CFC4, 'X'), - (0x1D000, 'V'), - (0x1D0F6, 'X'), - (0x1D100, 'V'), - (0x1D127, 'X'), - (0x1D129, 'V'), - (0x1D15E, 'M', '𝅗𝅥'), - (0x1D15F, 'M', '𝅘𝅥'), - (0x1D160, 'M', '𝅘𝅥𝅮'), - (0x1D161, 'M', '𝅘𝅥𝅯'), - (0x1D162, 'M', '𝅘𝅥𝅰'), - (0x1D163, 'M', '𝅘𝅥𝅱'), - (0x1D164, 'M', '𝅘𝅥𝅲'), - (0x1D165, 'V'), - (0x1D173, 'X'), - (0x1D17B, 'V'), - (0x1D1BB, 'M', '𝆹𝅥'), - (0x1D1BC, 'M', '𝆺𝅥'), - (0x1D1BD, 'M', '𝆹𝅥𝅮'), - (0x1D1BE, 'M', '𝆺𝅥𝅮'), - (0x1D1BF, 'M', '𝆹𝅥𝅯'), - (0x1D1C0, 'M', '𝆺𝅥𝅯'), - (0x1D1C1, 'V'), - (0x1D1EB, 'X'), - (0x1D200, 'V'), - (0x1D246, 'X'), - (0x1D2C0, 'V'), - (0x1D2D4, 'X'), - (0x1D2E0, 'V'), - (0x1D2F4, 'X'), - (0x1D300, 'V'), - (0x1D357, 'X'), - (0x1D360, 'V'), - (0x1D379, 'X'), - (0x1D400, 'M', 'a'), - (0x1D401, 'M', 'b'), - (0x1D402, 'M', 'c'), - (0x1D403, 'M', 'd'), - (0x1D404, 'M', 'e'), - (0x1D405, 'M', 'f'), - (0x1D406, 'M', 'g'), - (0x1D407, 'M', 'h'), - (0x1D408, 'M', 'i'), - (0x1D409, 'M', 'j'), - (0x1D40A, 'M', 'k'), - (0x1D40B, 'M', 'l'), - (0x1D40C, 'M', 'm'), - (0x1D40D, 'M', 'n'), - (0x1D40E, 'M', 'o'), - (0x1D40F, 'M', 'p'), - (0x1D410, 'M', 'q'), - (0x1D411, 'M', 'r'), - (0x1D412, 'M', 's'), - (0x1D413, 'M', 't'), - (0x1D414, 'M', 'u'), - (0x1D415, 'M', 'v'), - (0x1D416, 'M', 'w'), - (0x1D417, 'M', 'x'), - (0x1D418, 'M', 'y'), - (0x1D419, 'M', 'z'), - (0x1D41A, 'M', 'a'), - (0x1D41B, 'M', 'b'), - (0x1D41C, 'M', 'c'), - (0x1D41D, 'M', 'd'), - (0x1D41E, 'M', 'e'), - (0x1D41F, 'M', 'f'), - (0x1D420, 'M', 'g'), - ] - -def _seg_61() -> List[Union[Tuple[int, str], Tuple[int, str, str]]]: - return [ - (0x1D421, 'M', 'h'), - (0x1D422, 'M', 'i'), - (0x1D423, 'M', 'j'), - (0x1D424, 'M', 'k'), - (0x1D425, 'M', 'l'), - (0x1D426, 'M', 'm'), - (0x1D427, 'M', 'n'), - (0x1D428, 'M', 'o'), - (0x1D429, 'M', 'p'), - (0x1D42A, 'M', 'q'), - (0x1D42B, 'M', 'r'), - (0x1D42C, 'M', 's'), - (0x1D42D, 'M', 't'), - (0x1D42E, 'M', 'u'), - (0x1D42F, 'M', 'v'), - (0x1D430, 'M', 'w'), - (0x1D431, 'M', 'x'), - (0x1D432, 'M', 'y'), - (0x1D433, 'M', 'z'), - (0x1D434, 'M', 'a'), - (0x1D435, 'M', 'b'), - (0x1D436, 'M', 'c'), - (0x1D437, 'M', 'd'), - (0x1D438, 'M', 'e'), - (0x1D439, 'M', 'f'), - (0x1D43A, 'M', 'g'), - (0x1D43B, 'M', 'h'), - (0x1D43C, 'M', 'i'), - (0x1D43D, 'M', 'j'), - (0x1D43E, 'M', 'k'), - (0x1D43F, 'M', 'l'), - (0x1D440, 'M', 'm'), - (0x1D441, 'M', 'n'), - (0x1D442, 'M', 'o'), - (0x1D443, 'M', 'p'), - (0x1D444, 'M', 'q'), - (0x1D445, 'M', 'r'), - (0x1D446, 'M', 's'), - (0x1D447, 'M', 't'), - (0x1D448, 'M', 'u'), - (0x1D449, 'M', 'v'), - (0x1D44A, 'M', 'w'), - (0x1D44B, 'M', 'x'), - (0x1D44C, 'M', 'y'), - (0x1D44D, 'M', 'z'), - (0x1D44E, 'M', 'a'), - (0x1D44F, 'M', 'b'), - (0x1D450, 'M', 'c'), - (0x1D451, 'M', 'd'), - (0x1D452, 'M', 'e'), - (0x1D453, 'M', 'f'), - (0x1D454, 'M', 'g'), - (0x1D455, 'X'), - (0x1D456, 'M', 'i'), - (0x1D457, 'M', 'j'), - (0x1D458, 'M', 'k'), - (0x1D459, 'M', 'l'), - (0x1D45A, 'M', 'm'), - (0x1D45B, 'M', 'n'), - (0x1D45C, 'M', 'o'), - (0x1D45D, 'M', 'p'), - (0x1D45E, 'M', 'q'), - (0x1D45F, 'M', 'r'), - (0x1D460, 'M', 's'), - (0x1D461, 'M', 't'), - (0x1D462, 'M', 'u'), - (0x1D463, 'M', 'v'), - (0x1D464, 'M', 'w'), - (0x1D465, 'M', 'x'), - (0x1D466, 'M', 'y'), - (0x1D467, 'M', 'z'), - (0x1D468, 'M', 'a'), - (0x1D469, 'M', 'b'), - (0x1D46A, 'M', 'c'), - (0x1D46B, 'M', 'd'), - (0x1D46C, 'M', 'e'), - (0x1D46D, 'M', 'f'), - (0x1D46E, 'M', 'g'), - (0x1D46F, 'M', 'h'), - (0x1D470, 'M', 'i'), - (0x1D471, 'M', 'j'), - (0x1D472, 'M', 'k'), - (0x1D473, 'M', 'l'), - (0x1D474, 'M', 'm'), - (0x1D475, 'M', 'n'), - (0x1D476, 'M', 'o'), - (0x1D477, 'M', 'p'), - (0x1D478, 'M', 'q'), - (0x1D479, 'M', 'r'), - (0x1D47A, 'M', 's'), - (0x1D47B, 'M', 't'), - (0x1D47C, 'M', 'u'), - (0x1D47D, 'M', 'v'), - (0x1D47E, 'M', 'w'), - (0x1D47F, 'M', 'x'), - (0x1D480, 'M', 'y'), - (0x1D481, 'M', 'z'), - (0x1D482, 'M', 'a'), - (0x1D483, 'M', 'b'), - (0x1D484, 'M', 'c'), - ] - -def _seg_62() -> List[Union[Tuple[int, str], Tuple[int, str, str]]]: - return [ - (0x1D485, 'M', 'd'), - (0x1D486, 'M', 'e'), - (0x1D487, 'M', 'f'), - (0x1D488, 'M', 'g'), - (0x1D489, 'M', 'h'), - (0x1D48A, 'M', 'i'), - (0x1D48B, 'M', 'j'), - (0x1D48C, 'M', 'k'), - (0x1D48D, 'M', 'l'), - (0x1D48E, 'M', 'm'), - (0x1D48F, 'M', 'n'), - (0x1D490, 'M', 'o'), - (0x1D491, 'M', 'p'), - (0x1D492, 'M', 'q'), - (0x1D493, 'M', 'r'), - (0x1D494, 'M', 's'), - (0x1D495, 'M', 't'), - (0x1D496, 'M', 'u'), - (0x1D497, 'M', 'v'), - (0x1D498, 'M', 'w'), - (0x1D499, 'M', 'x'), - (0x1D49A, 'M', 'y'), - (0x1D49B, 'M', 'z'), - (0x1D49C, 'M', 'a'), - (0x1D49D, 'X'), - (0x1D49E, 'M', 'c'), - (0x1D49F, 'M', 'd'), - (0x1D4A0, 'X'), - (0x1D4A2, 'M', 'g'), - (0x1D4A3, 'X'), - (0x1D4A5, 'M', 'j'), - (0x1D4A6, 'M', 'k'), - (0x1D4A7, 'X'), - (0x1D4A9, 'M', 'n'), - (0x1D4AA, 'M', 'o'), - (0x1D4AB, 'M', 'p'), - (0x1D4AC, 'M', 'q'), - (0x1D4AD, 'X'), - (0x1D4AE, 'M', 's'), - (0x1D4AF, 'M', 't'), - (0x1D4B0, 'M', 'u'), - (0x1D4B1, 'M', 'v'), - (0x1D4B2, 'M', 'w'), - (0x1D4B3, 'M', 'x'), - (0x1D4B4, 'M', 'y'), - (0x1D4B5, 'M', 'z'), - (0x1D4B6, 'M', 'a'), - (0x1D4B7, 'M', 'b'), - (0x1D4B8, 'M', 'c'), - (0x1D4B9, 'M', 'd'), - (0x1D4BA, 'X'), - (0x1D4BB, 'M', 'f'), - (0x1D4BC, 'X'), - (0x1D4BD, 'M', 'h'), - (0x1D4BE, 'M', 'i'), - (0x1D4BF, 'M', 'j'), - (0x1D4C0, 'M', 'k'), - (0x1D4C1, 'M', 'l'), - (0x1D4C2, 'M', 'm'), - (0x1D4C3, 'M', 'n'), - (0x1D4C4, 'X'), - (0x1D4C5, 'M', 'p'), - (0x1D4C6, 'M', 'q'), - (0x1D4C7, 'M', 'r'), - (0x1D4C8, 'M', 's'), - (0x1D4C9, 'M', 't'), - (0x1D4CA, 'M', 'u'), - (0x1D4CB, 'M', 'v'), - (0x1D4CC, 'M', 'w'), - (0x1D4CD, 'M', 'x'), - (0x1D4CE, 'M', 'y'), - (0x1D4CF, 'M', 'z'), - (0x1D4D0, 'M', 'a'), - (0x1D4D1, 'M', 'b'), - (0x1D4D2, 'M', 'c'), - (0x1D4D3, 'M', 'd'), - (0x1D4D4, 'M', 'e'), - (0x1D4D5, 'M', 'f'), - (0x1D4D6, 'M', 'g'), - (0x1D4D7, 'M', 'h'), - (0x1D4D8, 'M', 'i'), - (0x1D4D9, 'M', 'j'), - (0x1D4DA, 'M', 'k'), - (0x1D4DB, 'M', 'l'), - (0x1D4DC, 'M', 'm'), - (0x1D4DD, 'M', 'n'), - (0x1D4DE, 'M', 'o'), - (0x1D4DF, 'M', 'p'), - (0x1D4E0, 'M', 'q'), - (0x1D4E1, 'M', 'r'), - (0x1D4E2, 'M', 's'), - (0x1D4E3, 'M', 't'), - (0x1D4E4, 'M', 'u'), - (0x1D4E5, 'M', 'v'), - (0x1D4E6, 'M', 'w'), - (0x1D4E7, 'M', 'x'), - (0x1D4E8, 'M', 'y'), - (0x1D4E9, 'M', 'z'), - (0x1D4EA, 'M', 'a'), - (0x1D4EB, 'M', 'b'), - ] - -def _seg_63() -> List[Union[Tuple[int, str], Tuple[int, str, str]]]: - return [ - (0x1D4EC, 'M', 'c'), - (0x1D4ED, 'M', 'd'), - (0x1D4EE, 'M', 'e'), - (0x1D4EF, 'M', 'f'), - (0x1D4F0, 'M', 'g'), - (0x1D4F1, 'M', 'h'), - (0x1D4F2, 'M', 'i'), - (0x1D4F3, 'M', 'j'), - (0x1D4F4, 'M', 'k'), - (0x1D4F5, 'M', 'l'), - (0x1D4F6, 'M', 'm'), - (0x1D4F7, 'M', 'n'), - (0x1D4F8, 'M', 'o'), - (0x1D4F9, 'M', 'p'), - (0x1D4FA, 'M', 'q'), - (0x1D4FB, 'M', 'r'), - (0x1D4FC, 'M', 's'), - (0x1D4FD, 'M', 't'), - (0x1D4FE, 'M', 'u'), - (0x1D4FF, 'M', 'v'), - (0x1D500, 'M', 'w'), - (0x1D501, 'M', 'x'), - (0x1D502, 'M', 'y'), - (0x1D503, 'M', 'z'), - (0x1D504, 'M', 'a'), - (0x1D505, 'M', 'b'), - (0x1D506, 'X'), - (0x1D507, 'M', 'd'), - (0x1D508, 'M', 'e'), - (0x1D509, 'M', 'f'), - (0x1D50A, 'M', 'g'), - (0x1D50B, 'X'), - (0x1D50D, 'M', 'j'), - (0x1D50E, 'M', 'k'), - (0x1D50F, 'M', 'l'), - (0x1D510, 'M', 'm'), - (0x1D511, 'M', 'n'), - (0x1D512, 'M', 'o'), - (0x1D513, 'M', 'p'), - (0x1D514, 'M', 'q'), - (0x1D515, 'X'), - (0x1D516, 'M', 's'), - (0x1D517, 'M', 't'), - (0x1D518, 'M', 'u'), - (0x1D519, 'M', 'v'), - (0x1D51A, 'M', 'w'), - (0x1D51B, 'M', 'x'), - (0x1D51C, 'M', 'y'), - (0x1D51D, 'X'), - (0x1D51E, 'M', 'a'), - (0x1D51F, 'M', 'b'), - (0x1D520, 'M', 'c'), - (0x1D521, 'M', 'd'), - (0x1D522, 'M', 'e'), - (0x1D523, 'M', 'f'), - (0x1D524, 'M', 'g'), - (0x1D525, 'M', 'h'), - (0x1D526, 'M', 'i'), - (0x1D527, 'M', 'j'), - (0x1D528, 'M', 'k'), - (0x1D529, 'M', 'l'), - (0x1D52A, 'M', 'm'), - (0x1D52B, 'M', 'n'), - (0x1D52C, 'M', 'o'), - (0x1D52D, 'M', 'p'), - (0x1D52E, 'M', 'q'), - (0x1D52F, 'M', 'r'), - (0x1D530, 'M', 's'), - (0x1D531, 'M', 't'), - (0x1D532, 'M', 'u'), - (0x1D533, 'M', 'v'), - (0x1D534, 'M', 'w'), - (0x1D535, 'M', 'x'), - (0x1D536, 'M', 'y'), - (0x1D537, 'M', 'z'), - (0x1D538, 'M', 'a'), - (0x1D539, 'M', 'b'), - (0x1D53A, 'X'), - (0x1D53B, 'M', 'd'), - (0x1D53C, 'M', 'e'), - (0x1D53D, 'M', 'f'), - (0x1D53E, 'M', 'g'), - (0x1D53F, 'X'), - (0x1D540, 'M', 'i'), - (0x1D541, 'M', 'j'), - (0x1D542, 'M', 'k'), - (0x1D543, 'M', 'l'), - (0x1D544, 'M', 'm'), - (0x1D545, 'X'), - (0x1D546, 'M', 'o'), - (0x1D547, 'X'), - (0x1D54A, 'M', 's'), - (0x1D54B, 'M', 't'), - (0x1D54C, 'M', 'u'), - (0x1D54D, 'M', 'v'), - (0x1D54E, 'M', 'w'), - (0x1D54F, 'M', 'x'), - (0x1D550, 'M', 'y'), - (0x1D551, 'X'), - (0x1D552, 'M', 'a'), - ] - -def _seg_64() -> List[Union[Tuple[int, str], Tuple[int, str, str]]]: - return [ - (0x1D553, 'M', 'b'), - (0x1D554, 'M', 'c'), - (0x1D555, 'M', 'd'), - (0x1D556, 'M', 'e'), - (0x1D557, 'M', 'f'), - (0x1D558, 'M', 'g'), - (0x1D559, 'M', 'h'), - (0x1D55A, 'M', 'i'), - (0x1D55B, 'M', 'j'), - (0x1D55C, 'M', 'k'), - (0x1D55D, 'M', 'l'), - (0x1D55E, 'M', 'm'), - (0x1D55F, 'M', 'n'), - (0x1D560, 'M', 'o'), - (0x1D561, 'M', 'p'), - (0x1D562, 'M', 'q'), - (0x1D563, 'M', 'r'), - (0x1D564, 'M', 's'), - (0x1D565, 'M', 't'), - (0x1D566, 'M', 'u'), - (0x1D567, 'M', 'v'), - (0x1D568, 'M', 'w'), - (0x1D569, 'M', 'x'), - (0x1D56A, 'M', 'y'), - (0x1D56B, 'M', 'z'), - (0x1D56C, 'M', 'a'), - (0x1D56D, 'M', 'b'), - (0x1D56E, 'M', 'c'), - (0x1D56F, 'M', 'd'), - (0x1D570, 'M', 'e'), - (0x1D571, 'M', 'f'), - (0x1D572, 'M', 'g'), - (0x1D573, 'M', 'h'), - (0x1D574, 'M', 'i'), - (0x1D575, 'M', 'j'), - (0x1D576, 'M', 'k'), - (0x1D577, 'M', 'l'), - (0x1D578, 'M', 'm'), - (0x1D579, 'M', 'n'), - (0x1D57A, 'M', 'o'), - (0x1D57B, 'M', 'p'), - (0x1D57C, 'M', 'q'), - (0x1D57D, 'M', 'r'), - (0x1D57E, 'M', 's'), - (0x1D57F, 'M', 't'), - (0x1D580, 'M', 'u'), - (0x1D581, 'M', 'v'), - (0x1D582, 'M', 'w'), - (0x1D583, 'M', 'x'), - (0x1D584, 'M', 'y'), - (0x1D585, 'M', 'z'), - (0x1D586, 'M', 'a'), - (0x1D587, 'M', 'b'), - (0x1D588, 'M', 'c'), - (0x1D589, 'M', 'd'), - (0x1D58A, 'M', 'e'), - (0x1D58B, 'M', 'f'), - (0x1D58C, 'M', 'g'), - (0x1D58D, 'M', 'h'), - (0x1D58E, 'M', 'i'), - (0x1D58F, 'M', 'j'), - (0x1D590, 'M', 'k'), - (0x1D591, 'M', 'l'), - (0x1D592, 'M', 'm'), - (0x1D593, 'M', 'n'), - (0x1D594, 'M', 'o'), - (0x1D595, 'M', 'p'), - (0x1D596, 'M', 'q'), - (0x1D597, 'M', 'r'), - (0x1D598, 'M', 's'), - (0x1D599, 'M', 't'), - (0x1D59A, 'M', 'u'), - (0x1D59B, 'M', 'v'), - (0x1D59C, 'M', 'w'), - (0x1D59D, 'M', 'x'), - (0x1D59E, 'M', 'y'), - (0x1D59F, 'M', 'z'), - (0x1D5A0, 'M', 'a'), - (0x1D5A1, 'M', 'b'), - (0x1D5A2, 'M', 'c'), - (0x1D5A3, 'M', 'd'), - (0x1D5A4, 'M', 'e'), - (0x1D5A5, 'M', 'f'), - (0x1D5A6, 'M', 'g'), - (0x1D5A7, 'M', 'h'), - (0x1D5A8, 'M', 'i'), - (0x1D5A9, 'M', 'j'), - (0x1D5AA, 'M', 'k'), - (0x1D5AB, 'M', 'l'), - (0x1D5AC, 'M', 'm'), - (0x1D5AD, 'M', 'n'), - (0x1D5AE, 'M', 'o'), - (0x1D5AF, 'M', 'p'), - (0x1D5B0, 'M', 'q'), - (0x1D5B1, 'M', 'r'), - (0x1D5B2, 'M', 's'), - (0x1D5B3, 'M', 't'), - (0x1D5B4, 'M', 'u'), - (0x1D5B5, 'M', 'v'), - (0x1D5B6, 'M', 'w'), - ] - -def _seg_65() -> List[Union[Tuple[int, str], Tuple[int, str, str]]]: - return [ - (0x1D5B7, 'M', 'x'), - (0x1D5B8, 'M', 'y'), - (0x1D5B9, 'M', 'z'), - (0x1D5BA, 'M', 'a'), - (0x1D5BB, 'M', 'b'), - (0x1D5BC, 'M', 'c'), - (0x1D5BD, 'M', 'd'), - (0x1D5BE, 'M', 'e'), - (0x1D5BF, 'M', 'f'), - (0x1D5C0, 'M', 'g'), - (0x1D5C1, 'M', 'h'), - (0x1D5C2, 'M', 'i'), - (0x1D5C3, 'M', 'j'), - (0x1D5C4, 'M', 'k'), - (0x1D5C5, 'M', 'l'), - (0x1D5C6, 'M', 'm'), - (0x1D5C7, 'M', 'n'), - (0x1D5C8, 'M', 'o'), - (0x1D5C9, 'M', 'p'), - (0x1D5CA, 'M', 'q'), - (0x1D5CB, 'M', 'r'), - (0x1D5CC, 'M', 's'), - (0x1D5CD, 'M', 't'), - (0x1D5CE, 'M', 'u'), - (0x1D5CF, 'M', 'v'), - (0x1D5D0, 'M', 'w'), - (0x1D5D1, 'M', 'x'), - (0x1D5D2, 'M', 'y'), - (0x1D5D3, 'M', 'z'), - (0x1D5D4, 'M', 'a'), - (0x1D5D5, 'M', 'b'), - (0x1D5D6, 'M', 'c'), - (0x1D5D7, 'M', 'd'), - (0x1D5D8, 'M', 'e'), - (0x1D5D9, 'M', 'f'), - (0x1D5DA, 'M', 'g'), - (0x1D5DB, 'M', 'h'), - (0x1D5DC, 'M', 'i'), - (0x1D5DD, 'M', 'j'), - (0x1D5DE, 'M', 'k'), - (0x1D5DF, 'M', 'l'), - (0x1D5E0, 'M', 'm'), - (0x1D5E1, 'M', 'n'), - (0x1D5E2, 'M', 'o'), - (0x1D5E3, 'M', 'p'), - (0x1D5E4, 'M', 'q'), - (0x1D5E5, 'M', 'r'), - (0x1D5E6, 'M', 's'), - (0x1D5E7, 'M', 't'), - (0x1D5E8, 'M', 'u'), - (0x1D5E9, 'M', 'v'), - (0x1D5EA, 'M', 'w'), - (0x1D5EB, 'M', 'x'), - (0x1D5EC, 'M', 'y'), - (0x1D5ED, 'M', 'z'), - (0x1D5EE, 'M', 'a'), - (0x1D5EF, 'M', 'b'), - (0x1D5F0, 'M', 'c'), - (0x1D5F1, 'M', 'd'), - (0x1D5F2, 'M', 'e'), - (0x1D5F3, 'M', 'f'), - (0x1D5F4, 'M', 'g'), - (0x1D5F5, 'M', 'h'), - (0x1D5F6, 'M', 'i'), - (0x1D5F7, 'M', 'j'), - (0x1D5F8, 'M', 'k'), - (0x1D5F9, 'M', 'l'), - (0x1D5FA, 'M', 'm'), - (0x1D5FB, 'M', 'n'), - (0x1D5FC, 'M', 'o'), - (0x1D5FD, 'M', 'p'), - (0x1D5FE, 'M', 'q'), - (0x1D5FF, 'M', 'r'), - (0x1D600, 'M', 's'), - (0x1D601, 'M', 't'), - (0x1D602, 'M', 'u'), - (0x1D603, 'M', 'v'), - (0x1D604, 'M', 'w'), - (0x1D605, 'M', 'x'), - (0x1D606, 'M', 'y'), - (0x1D607, 'M', 'z'), - (0x1D608, 'M', 'a'), - (0x1D609, 'M', 'b'), - (0x1D60A, 'M', 'c'), - (0x1D60B, 'M', 'd'), - (0x1D60C, 'M', 'e'), - (0x1D60D, 'M', 'f'), - (0x1D60E, 'M', 'g'), - (0x1D60F, 'M', 'h'), - (0x1D610, 'M', 'i'), - (0x1D611, 'M', 'j'), - (0x1D612, 'M', 'k'), - (0x1D613, 'M', 'l'), - (0x1D614, 'M', 'm'), - (0x1D615, 'M', 'n'), - (0x1D616, 'M', 'o'), - (0x1D617, 'M', 'p'), - (0x1D618, 'M', 'q'), - (0x1D619, 'M', 'r'), - (0x1D61A, 'M', 's'), - ] - -def _seg_66() -> List[Union[Tuple[int, str], Tuple[int, str, str]]]: - return [ - (0x1D61B, 'M', 't'), - (0x1D61C, 'M', 'u'), - (0x1D61D, 'M', 'v'), - (0x1D61E, 'M', 'w'), - (0x1D61F, 'M', 'x'), - (0x1D620, 'M', 'y'), - (0x1D621, 'M', 'z'), - (0x1D622, 'M', 'a'), - (0x1D623, 'M', 'b'), - (0x1D624, 'M', 'c'), - (0x1D625, 'M', 'd'), - (0x1D626, 'M', 'e'), - (0x1D627, 'M', 'f'), - (0x1D628, 'M', 'g'), - (0x1D629, 'M', 'h'), - (0x1D62A, 'M', 'i'), - (0x1D62B, 'M', 'j'), - (0x1D62C, 'M', 'k'), - (0x1D62D, 'M', 'l'), - (0x1D62E, 'M', 'm'), - (0x1D62F, 'M', 'n'), - (0x1D630, 'M', 'o'), - (0x1D631, 'M', 'p'), - (0x1D632, 'M', 'q'), - (0x1D633, 'M', 'r'), - (0x1D634, 'M', 's'), - (0x1D635, 'M', 't'), - (0x1D636, 'M', 'u'), - (0x1D637, 'M', 'v'), - (0x1D638, 'M', 'w'), - (0x1D639, 'M', 'x'), - (0x1D63A, 'M', 'y'), - (0x1D63B, 'M', 'z'), - (0x1D63C, 'M', 'a'), - (0x1D63D, 'M', 'b'), - (0x1D63E, 'M', 'c'), - (0x1D63F, 'M', 'd'), - (0x1D640, 'M', 'e'), - (0x1D641, 'M', 'f'), - (0x1D642, 'M', 'g'), - (0x1D643, 'M', 'h'), - (0x1D644, 'M', 'i'), - (0x1D645, 'M', 'j'), - (0x1D646, 'M', 'k'), - (0x1D647, 'M', 'l'), - (0x1D648, 'M', 'm'), - (0x1D649, 'M', 'n'), - (0x1D64A, 'M', 'o'), - (0x1D64B, 'M', 'p'), - (0x1D64C, 'M', 'q'), - (0x1D64D, 'M', 'r'), - (0x1D64E, 'M', 's'), - (0x1D64F, 'M', 't'), - (0x1D650, 'M', 'u'), - (0x1D651, 'M', 'v'), - (0x1D652, 'M', 'w'), - (0x1D653, 'M', 'x'), - (0x1D654, 'M', 'y'), - (0x1D655, 'M', 'z'), - (0x1D656, 'M', 'a'), - (0x1D657, 'M', 'b'), - (0x1D658, 'M', 'c'), - (0x1D659, 'M', 'd'), - (0x1D65A, 'M', 'e'), - (0x1D65B, 'M', 'f'), - (0x1D65C, 'M', 'g'), - (0x1D65D, 'M', 'h'), - (0x1D65E, 'M', 'i'), - (0x1D65F, 'M', 'j'), - (0x1D660, 'M', 'k'), - (0x1D661, 'M', 'l'), - (0x1D662, 'M', 'm'), - (0x1D663, 'M', 'n'), - (0x1D664, 'M', 'o'), - (0x1D665, 'M', 'p'), - (0x1D666, 'M', 'q'), - (0x1D667, 'M', 'r'), - (0x1D668, 'M', 's'), - (0x1D669, 'M', 't'), - (0x1D66A, 'M', 'u'), - (0x1D66B, 'M', 'v'), - (0x1D66C, 'M', 'w'), - (0x1D66D, 'M', 'x'), - (0x1D66E, 'M', 'y'), - (0x1D66F, 'M', 'z'), - (0x1D670, 'M', 'a'), - (0x1D671, 'M', 'b'), - (0x1D672, 'M', 'c'), - (0x1D673, 'M', 'd'), - (0x1D674, 'M', 'e'), - (0x1D675, 'M', 'f'), - (0x1D676, 'M', 'g'), - (0x1D677, 'M', 'h'), - (0x1D678, 'M', 'i'), - (0x1D679, 'M', 'j'), - (0x1D67A, 'M', 'k'), - (0x1D67B, 'M', 'l'), - (0x1D67C, 'M', 'm'), - (0x1D67D, 'M', 'n'), - (0x1D67E, 'M', 'o'), - ] - -def _seg_67() -> List[Union[Tuple[int, str], Tuple[int, str, str]]]: - return [ - (0x1D67F, 'M', 'p'), - (0x1D680, 'M', 'q'), - (0x1D681, 'M', 'r'), - (0x1D682, 'M', 's'), - (0x1D683, 'M', 't'), - (0x1D684, 'M', 'u'), - (0x1D685, 'M', 'v'), - (0x1D686, 'M', 'w'), - (0x1D687, 'M', 'x'), - (0x1D688, 'M', 'y'), - (0x1D689, 'M', 'z'), - (0x1D68A, 'M', 'a'), - (0x1D68B, 'M', 'b'), - (0x1D68C, 'M', 'c'), - (0x1D68D, 'M', 'd'), - (0x1D68E, 'M', 'e'), - (0x1D68F, 'M', 'f'), - (0x1D690, 'M', 'g'), - (0x1D691, 'M', 'h'), - (0x1D692, 'M', 'i'), - (0x1D693, 'M', 'j'), - (0x1D694, 'M', 'k'), - (0x1D695, 'M', 'l'), - (0x1D696, 'M', 'm'), - (0x1D697, 'M', 'n'), - (0x1D698, 'M', 'o'), - (0x1D699, 'M', 'p'), - (0x1D69A, 'M', 'q'), - (0x1D69B, 'M', 'r'), - (0x1D69C, 'M', 's'), - (0x1D69D, 'M', 't'), - (0x1D69E, 'M', 'u'), - (0x1D69F, 'M', 'v'), - (0x1D6A0, 'M', 'w'), - (0x1D6A1, 'M', 'x'), - (0x1D6A2, 'M', 'y'), - (0x1D6A3, 'M', 'z'), - (0x1D6A4, 'M', 'ı'), - (0x1D6A5, 'M', 'ȷ'), - (0x1D6A6, 'X'), - (0x1D6A8, 'M', 'α'), - (0x1D6A9, 'M', 'β'), - (0x1D6AA, 'M', 'γ'), - (0x1D6AB, 'M', 'δ'), - (0x1D6AC, 'M', 'ε'), - (0x1D6AD, 'M', 'ζ'), - (0x1D6AE, 'M', 'η'), - (0x1D6AF, 'M', 'θ'), - (0x1D6B0, 'M', 'ι'), - (0x1D6B1, 'M', 'κ'), - (0x1D6B2, 'M', 'λ'), - (0x1D6B3, 'M', 'μ'), - (0x1D6B4, 'M', 'ν'), - (0x1D6B5, 'M', 'ξ'), - (0x1D6B6, 'M', 'ο'), - (0x1D6B7, 'M', 'π'), - (0x1D6B8, 'M', 'ρ'), - (0x1D6B9, 'M', 'θ'), - (0x1D6BA, 'M', 'σ'), - (0x1D6BB, 'M', 'τ'), - (0x1D6BC, 'M', 'υ'), - (0x1D6BD, 'M', 'φ'), - (0x1D6BE, 'M', 'χ'), - (0x1D6BF, 'M', 'ψ'), - (0x1D6C0, 'M', 'ω'), - (0x1D6C1, 'M', '∇'), - (0x1D6C2, 'M', 'α'), - (0x1D6C3, 'M', 'β'), - (0x1D6C4, 'M', 'γ'), - (0x1D6C5, 'M', 'δ'), - (0x1D6C6, 'M', 'ε'), - (0x1D6C7, 'M', 'ζ'), - (0x1D6C8, 'M', 'η'), - (0x1D6C9, 'M', 'θ'), - (0x1D6CA, 'M', 'ι'), - (0x1D6CB, 'M', 'κ'), - (0x1D6CC, 'M', 'λ'), - (0x1D6CD, 'M', 'μ'), - (0x1D6CE, 'M', 'ν'), - (0x1D6CF, 'M', 'ξ'), - (0x1D6D0, 'M', 'ο'), - (0x1D6D1, 'M', 'π'), - (0x1D6D2, 'M', 'ρ'), - (0x1D6D3, 'M', 'σ'), - (0x1D6D5, 'M', 'τ'), - (0x1D6D6, 'M', 'υ'), - (0x1D6D7, 'M', 'φ'), - (0x1D6D8, 'M', 'χ'), - (0x1D6D9, 'M', 'ψ'), - (0x1D6DA, 'M', 'ω'), - (0x1D6DB, 'M', '∂'), - (0x1D6DC, 'M', 'ε'), - (0x1D6DD, 'M', 'θ'), - (0x1D6DE, 'M', 'κ'), - (0x1D6DF, 'M', 'φ'), - (0x1D6E0, 'M', 'ρ'), - (0x1D6E1, 'M', 'π'), - (0x1D6E2, 'M', 'α'), - (0x1D6E3, 'M', 'β'), - (0x1D6E4, 'M', 'γ'), - ] - -def _seg_68() -> List[Union[Tuple[int, str], Tuple[int, str, str]]]: - return [ - (0x1D6E5, 'M', 'δ'), - (0x1D6E6, 'M', 'ε'), - (0x1D6E7, 'M', 'ζ'), - (0x1D6E8, 'M', 'η'), - (0x1D6E9, 'M', 'θ'), - (0x1D6EA, 'M', 'ι'), - (0x1D6EB, 'M', 'κ'), - (0x1D6EC, 'M', 'λ'), - (0x1D6ED, 'M', 'μ'), - (0x1D6EE, 'M', 'ν'), - (0x1D6EF, 'M', 'ξ'), - (0x1D6F0, 'M', 'ο'), - (0x1D6F1, 'M', 'π'), - (0x1D6F2, 'M', 'ρ'), - (0x1D6F3, 'M', 'θ'), - (0x1D6F4, 'M', 'σ'), - (0x1D6F5, 'M', 'τ'), - (0x1D6F6, 'M', 'υ'), - (0x1D6F7, 'M', 'φ'), - (0x1D6F8, 'M', 'χ'), - (0x1D6F9, 'M', 'ψ'), - (0x1D6FA, 'M', 'ω'), - (0x1D6FB, 'M', '∇'), - (0x1D6FC, 'M', 'α'), - (0x1D6FD, 'M', 'β'), - (0x1D6FE, 'M', 'γ'), - (0x1D6FF, 'M', 'δ'), - (0x1D700, 'M', 'ε'), - (0x1D701, 'M', 'ζ'), - (0x1D702, 'M', 'η'), - (0x1D703, 'M', 'θ'), - (0x1D704, 'M', 'ι'), - (0x1D705, 'M', 'κ'), - (0x1D706, 'M', 'λ'), - (0x1D707, 'M', 'μ'), - (0x1D708, 'M', 'ν'), - (0x1D709, 'M', 'ξ'), - (0x1D70A, 'M', 'ο'), - (0x1D70B, 'M', 'π'), - (0x1D70C, 'M', 'ρ'), - (0x1D70D, 'M', 'σ'), - (0x1D70F, 'M', 'τ'), - (0x1D710, 'M', 'υ'), - (0x1D711, 'M', 'φ'), - (0x1D712, 'M', 'χ'), - (0x1D713, 'M', 'ψ'), - (0x1D714, 'M', 'ω'), - (0x1D715, 'M', '∂'), - (0x1D716, 'M', 'ε'), - (0x1D717, 'M', 'θ'), - (0x1D718, 'M', 'κ'), - (0x1D719, 'M', 'φ'), - (0x1D71A, 'M', 'ρ'), - (0x1D71B, 'M', 'π'), - (0x1D71C, 'M', 'α'), - (0x1D71D, 'M', 'β'), - (0x1D71E, 'M', 'γ'), - (0x1D71F, 'M', 'δ'), - (0x1D720, 'M', 'ε'), - (0x1D721, 'M', 'ζ'), - (0x1D722, 'M', 'η'), - (0x1D723, 'M', 'θ'), - (0x1D724, 'M', 'ι'), - (0x1D725, 'M', 'κ'), - (0x1D726, 'M', 'λ'), - (0x1D727, 'M', 'μ'), - (0x1D728, 'M', 'ν'), - (0x1D729, 'M', 'ξ'), - (0x1D72A, 'M', 'ο'), - (0x1D72B, 'M', 'π'), - (0x1D72C, 'M', 'ρ'), - (0x1D72D, 'M', 'θ'), - (0x1D72E, 'M', 'σ'), - (0x1D72F, 'M', 'τ'), - (0x1D730, 'M', 'υ'), - (0x1D731, 'M', 'φ'), - (0x1D732, 'M', 'χ'), - (0x1D733, 'M', 'ψ'), - (0x1D734, 'M', 'ω'), - (0x1D735, 'M', '∇'), - (0x1D736, 'M', 'α'), - (0x1D737, 'M', 'β'), - (0x1D738, 'M', 'γ'), - (0x1D739, 'M', 'δ'), - (0x1D73A, 'M', 'ε'), - (0x1D73B, 'M', 'ζ'), - (0x1D73C, 'M', 'η'), - (0x1D73D, 'M', 'θ'), - (0x1D73E, 'M', 'ι'), - (0x1D73F, 'M', 'κ'), - (0x1D740, 'M', 'λ'), - (0x1D741, 'M', 'μ'), - (0x1D742, 'M', 'ν'), - (0x1D743, 'M', 'ξ'), - (0x1D744, 'M', 'ο'), - (0x1D745, 'M', 'π'), - (0x1D746, 'M', 'ρ'), - (0x1D747, 'M', 'σ'), - (0x1D749, 'M', 'τ'), - (0x1D74A, 'M', 'υ'), - ] - -def _seg_69() -> List[Union[Tuple[int, str], Tuple[int, str, str]]]: - return [ - (0x1D74B, 'M', 'φ'), - (0x1D74C, 'M', 'χ'), - (0x1D74D, 'M', 'ψ'), - (0x1D74E, 'M', 'ω'), - (0x1D74F, 'M', '∂'), - (0x1D750, 'M', 'ε'), - (0x1D751, 'M', 'θ'), - (0x1D752, 'M', 'κ'), - (0x1D753, 'M', 'φ'), - (0x1D754, 'M', 'ρ'), - (0x1D755, 'M', 'π'), - (0x1D756, 'M', 'α'), - (0x1D757, 'M', 'β'), - (0x1D758, 'M', 'γ'), - (0x1D759, 'M', 'δ'), - (0x1D75A, 'M', 'ε'), - (0x1D75B, 'M', 'ζ'), - (0x1D75C, 'M', 'η'), - (0x1D75D, 'M', 'θ'), - (0x1D75E, 'M', 'ι'), - (0x1D75F, 'M', 'κ'), - (0x1D760, 'M', 'λ'), - (0x1D761, 'M', 'μ'), - (0x1D762, 'M', 'ν'), - (0x1D763, 'M', 'ξ'), - (0x1D764, 'M', 'ο'), - (0x1D765, 'M', 'π'), - (0x1D766, 'M', 'ρ'), - (0x1D767, 'M', 'θ'), - (0x1D768, 'M', 'σ'), - (0x1D769, 'M', 'τ'), - (0x1D76A, 'M', 'υ'), - (0x1D76B, 'M', 'φ'), - (0x1D76C, 'M', 'χ'), - (0x1D76D, 'M', 'ψ'), - (0x1D76E, 'M', 'ω'), - (0x1D76F, 'M', '∇'), - (0x1D770, 'M', 'α'), - (0x1D771, 'M', 'β'), - (0x1D772, 'M', 'γ'), - (0x1D773, 'M', 'δ'), - (0x1D774, 'M', 'ε'), - (0x1D775, 'M', 'ζ'), - (0x1D776, 'M', 'η'), - (0x1D777, 'M', 'θ'), - (0x1D778, 'M', 'ι'), - (0x1D779, 'M', 'κ'), - (0x1D77A, 'M', 'λ'), - (0x1D77B, 'M', 'μ'), - (0x1D77C, 'M', 'ν'), - (0x1D77D, 'M', 'ξ'), - (0x1D77E, 'M', 'ο'), - (0x1D77F, 'M', 'π'), - (0x1D780, 'M', 'ρ'), - (0x1D781, 'M', 'σ'), - (0x1D783, 'M', 'τ'), - (0x1D784, 'M', 'υ'), - (0x1D785, 'M', 'φ'), - (0x1D786, 'M', 'χ'), - (0x1D787, 'M', 'ψ'), - (0x1D788, 'M', 'ω'), - (0x1D789, 'M', '∂'), - (0x1D78A, 'M', 'ε'), - (0x1D78B, 'M', 'θ'), - (0x1D78C, 'M', 'κ'), - (0x1D78D, 'M', 'φ'), - (0x1D78E, 'M', 'ρ'), - (0x1D78F, 'M', 'π'), - (0x1D790, 'M', 'α'), - (0x1D791, 'M', 'β'), - (0x1D792, 'M', 'γ'), - (0x1D793, 'M', 'δ'), - (0x1D794, 'M', 'ε'), - (0x1D795, 'M', 'ζ'), - (0x1D796, 'M', 'η'), - (0x1D797, 'M', 'θ'), - (0x1D798, 'M', 'ι'), - (0x1D799, 'M', 'κ'), - (0x1D79A, 'M', 'λ'), - (0x1D79B, 'M', 'μ'), - (0x1D79C, 'M', 'ν'), - (0x1D79D, 'M', 'ξ'), - (0x1D79E, 'M', 'ο'), - (0x1D79F, 'M', 'π'), - (0x1D7A0, 'M', 'ρ'), - (0x1D7A1, 'M', 'θ'), - (0x1D7A2, 'M', 'σ'), - (0x1D7A3, 'M', 'τ'), - (0x1D7A4, 'M', 'υ'), - (0x1D7A5, 'M', 'φ'), - (0x1D7A6, 'M', 'χ'), - (0x1D7A7, 'M', 'ψ'), - (0x1D7A8, 'M', 'ω'), - (0x1D7A9, 'M', '∇'), - (0x1D7AA, 'M', 'α'), - (0x1D7AB, 'M', 'β'), - (0x1D7AC, 'M', 'γ'), - (0x1D7AD, 'M', 'δ'), - (0x1D7AE, 'M', 'ε'), - (0x1D7AF, 'M', 'ζ'), - ] - -def _seg_70() -> List[Union[Tuple[int, str], Tuple[int, str, str]]]: - return [ - (0x1D7B0, 'M', 'η'), - (0x1D7B1, 'M', 'θ'), - (0x1D7B2, 'M', 'ι'), - (0x1D7B3, 'M', 'κ'), - (0x1D7B4, 'M', 'λ'), - (0x1D7B5, 'M', 'μ'), - (0x1D7B6, 'M', 'ν'), - (0x1D7B7, 'M', 'ξ'), - (0x1D7B8, 'M', 'ο'), - (0x1D7B9, 'M', 'π'), - (0x1D7BA, 'M', 'ρ'), - (0x1D7BB, 'M', 'σ'), - (0x1D7BD, 'M', 'τ'), - (0x1D7BE, 'M', 'υ'), - (0x1D7BF, 'M', 'φ'), - (0x1D7C0, 'M', 'χ'), - (0x1D7C1, 'M', 'ψ'), - (0x1D7C2, 'M', 'ω'), - (0x1D7C3, 'M', '∂'), - (0x1D7C4, 'M', 'ε'), - (0x1D7C5, 'M', 'θ'), - (0x1D7C6, 'M', 'κ'), - (0x1D7C7, 'M', 'φ'), - (0x1D7C8, 'M', 'ρ'), - (0x1D7C9, 'M', 'π'), - (0x1D7CA, 'M', 'ϝ'), - (0x1D7CC, 'X'), - (0x1D7CE, 'M', '0'), - (0x1D7CF, 'M', '1'), - (0x1D7D0, 'M', '2'), - (0x1D7D1, 'M', '3'), - (0x1D7D2, 'M', '4'), - (0x1D7D3, 'M', '5'), - (0x1D7D4, 'M', '6'), - (0x1D7D5, 'M', '7'), - (0x1D7D6, 'M', '8'), - (0x1D7D7, 'M', '9'), - (0x1D7D8, 'M', '0'), - (0x1D7D9, 'M', '1'), - (0x1D7DA, 'M', '2'), - (0x1D7DB, 'M', '3'), - (0x1D7DC, 'M', '4'), - (0x1D7DD, 'M', '5'), - (0x1D7DE, 'M', '6'), - (0x1D7DF, 'M', '7'), - (0x1D7E0, 'M', '8'), - (0x1D7E1, 'M', '9'), - (0x1D7E2, 'M', '0'), - (0x1D7E3, 'M', '1'), - (0x1D7E4, 'M', '2'), - (0x1D7E5, 'M', '3'), - (0x1D7E6, 'M', '4'), - (0x1D7E7, 'M', '5'), - (0x1D7E8, 'M', '6'), - (0x1D7E9, 'M', '7'), - (0x1D7EA, 'M', '8'), - (0x1D7EB, 'M', '9'), - (0x1D7EC, 'M', '0'), - (0x1D7ED, 'M', '1'), - (0x1D7EE, 'M', '2'), - (0x1D7EF, 'M', '3'), - (0x1D7F0, 'M', '4'), - (0x1D7F1, 'M', '5'), - (0x1D7F2, 'M', '6'), - (0x1D7F3, 'M', '7'), - (0x1D7F4, 'M', '8'), - (0x1D7F5, 'M', '9'), - (0x1D7F6, 'M', '0'), - (0x1D7F7, 'M', '1'), - (0x1D7F8, 'M', '2'), - (0x1D7F9, 'M', '3'), - (0x1D7FA, 'M', '4'), - (0x1D7FB, 'M', '5'), - (0x1D7FC, 'M', '6'), - (0x1D7FD, 'M', '7'), - (0x1D7FE, 'M', '8'), - (0x1D7FF, 'M', '9'), - (0x1D800, 'V'), - (0x1DA8C, 'X'), - (0x1DA9B, 'V'), - (0x1DAA0, 'X'), - (0x1DAA1, 'V'), - (0x1DAB0, 'X'), - (0x1DF00, 'V'), - (0x1DF1F, 'X'), - (0x1DF25, 'V'), - (0x1DF2B, 'X'), - (0x1E000, 'V'), - (0x1E007, 'X'), - (0x1E008, 'V'), - (0x1E019, 'X'), - (0x1E01B, 'V'), - (0x1E022, 'X'), - (0x1E023, 'V'), - (0x1E025, 'X'), - (0x1E026, 'V'), - (0x1E02B, 'X'), - (0x1E030, 'M', 'а'), - (0x1E031, 'M', 'б'), - (0x1E032, 'M', 'в'), - ] - -def _seg_71() -> List[Union[Tuple[int, str], Tuple[int, str, str]]]: - return [ - (0x1E033, 'M', 'г'), - (0x1E034, 'M', 'д'), - (0x1E035, 'M', 'е'), - (0x1E036, 'M', 'ж'), - (0x1E037, 'M', 'з'), - (0x1E038, 'M', 'и'), - (0x1E039, 'M', 'к'), - (0x1E03A, 'M', 'л'), - (0x1E03B, 'M', 'м'), - (0x1E03C, 'M', 'о'), - (0x1E03D, 'M', 'п'), - (0x1E03E, 'M', 'р'), - (0x1E03F, 'M', 'с'), - (0x1E040, 'M', 'т'), - (0x1E041, 'M', 'у'), - (0x1E042, 'M', 'ф'), - (0x1E043, 'M', 'х'), - (0x1E044, 'M', 'ц'), - (0x1E045, 'M', 'ч'), - (0x1E046, 'M', 'ш'), - (0x1E047, 'M', 'ы'), - (0x1E048, 'M', 'э'), - (0x1E049, 'M', 'ю'), - (0x1E04A, 'M', 'ꚉ'), - (0x1E04B, 'M', 'ә'), - (0x1E04C, 'M', 'і'), - (0x1E04D, 'M', 'ј'), - (0x1E04E, 'M', 'ө'), - (0x1E04F, 'M', 'ү'), - (0x1E050, 'M', 'ӏ'), - (0x1E051, 'M', 'а'), - (0x1E052, 'M', 'б'), - (0x1E053, 'M', 'в'), - (0x1E054, 'M', 'г'), - (0x1E055, 'M', 'д'), - (0x1E056, 'M', 'е'), - (0x1E057, 'M', 'ж'), - (0x1E058, 'M', 'з'), - (0x1E059, 'M', 'и'), - (0x1E05A, 'M', 'к'), - (0x1E05B, 'M', 'л'), - (0x1E05C, 'M', 'о'), - (0x1E05D, 'M', 'п'), - (0x1E05E, 'M', 'с'), - (0x1E05F, 'M', 'у'), - (0x1E060, 'M', 'ф'), - (0x1E061, 'M', 'х'), - (0x1E062, 'M', 'ц'), - (0x1E063, 'M', 'ч'), - (0x1E064, 'M', 'ш'), - (0x1E065, 'M', 'ъ'), - (0x1E066, 'M', 'ы'), - (0x1E067, 'M', 'ґ'), - (0x1E068, 'M', 'і'), - (0x1E069, 'M', 'ѕ'), - (0x1E06A, 'M', 'џ'), - (0x1E06B, 'M', 'ҫ'), - (0x1E06C, 'M', 'ꙑ'), - (0x1E06D, 'M', 'ұ'), - (0x1E06E, 'X'), - (0x1E08F, 'V'), - (0x1E090, 'X'), - (0x1E100, 'V'), - (0x1E12D, 'X'), - (0x1E130, 'V'), - (0x1E13E, 'X'), - (0x1E140, 'V'), - (0x1E14A, 'X'), - (0x1E14E, 'V'), - (0x1E150, 'X'), - (0x1E290, 'V'), - (0x1E2AF, 'X'), - (0x1E2C0, 'V'), - (0x1E2FA, 'X'), - (0x1E2FF, 'V'), - (0x1E300, 'X'), - (0x1E4D0, 'V'), - (0x1E4FA, 'X'), - (0x1E7E0, 'V'), - (0x1E7E7, 'X'), - (0x1E7E8, 'V'), - (0x1E7EC, 'X'), - (0x1E7ED, 'V'), - (0x1E7EF, 'X'), - (0x1E7F0, 'V'), - (0x1E7FF, 'X'), - (0x1E800, 'V'), - (0x1E8C5, 'X'), - (0x1E8C7, 'V'), - (0x1E8D7, 'X'), - (0x1E900, 'M', '𞤢'), - (0x1E901, 'M', '𞤣'), - (0x1E902, 'M', '𞤤'), - (0x1E903, 'M', '𞤥'), - (0x1E904, 'M', '𞤦'), - (0x1E905, 'M', '𞤧'), - (0x1E906, 'M', '𞤨'), - (0x1E907, 'M', '𞤩'), - (0x1E908, 'M', '𞤪'), - (0x1E909, 'M', '𞤫'), - ] - -def _seg_72() -> List[Union[Tuple[int, str], Tuple[int, str, str]]]: - return [ - (0x1E90A, 'M', '𞤬'), - (0x1E90B, 'M', '𞤭'), - (0x1E90C, 'M', '𞤮'), - (0x1E90D, 'M', '𞤯'), - (0x1E90E, 'M', '𞤰'), - (0x1E90F, 'M', '𞤱'), - (0x1E910, 'M', '𞤲'), - (0x1E911, 'M', '𞤳'), - (0x1E912, 'M', '𞤴'), - (0x1E913, 'M', '𞤵'), - (0x1E914, 'M', '𞤶'), - (0x1E915, 'M', '𞤷'), - (0x1E916, 'M', '𞤸'), - (0x1E917, 'M', '𞤹'), - (0x1E918, 'M', '𞤺'), - (0x1E919, 'M', '𞤻'), - (0x1E91A, 'M', '𞤼'), - (0x1E91B, 'M', '𞤽'), - (0x1E91C, 'M', '𞤾'), - (0x1E91D, 'M', '𞤿'), - (0x1E91E, 'M', '𞥀'), - (0x1E91F, 'M', '𞥁'), - (0x1E920, 'M', '𞥂'), - (0x1E921, 'M', '𞥃'), - (0x1E922, 'V'), - (0x1E94C, 'X'), - (0x1E950, 'V'), - (0x1E95A, 'X'), - (0x1E95E, 'V'), - (0x1E960, 'X'), - (0x1EC71, 'V'), - (0x1ECB5, 'X'), - (0x1ED01, 'V'), - (0x1ED3E, 'X'), - (0x1EE00, 'M', 'ا'), - (0x1EE01, 'M', 'ب'), - (0x1EE02, 'M', 'ج'), - (0x1EE03, 'M', 'د'), - (0x1EE04, 'X'), - (0x1EE05, 'M', 'و'), - (0x1EE06, 'M', 'ز'), - (0x1EE07, 'M', 'ح'), - (0x1EE08, 'M', 'ط'), - (0x1EE09, 'M', 'ي'), - (0x1EE0A, 'M', 'ك'), - (0x1EE0B, 'M', 'ل'), - (0x1EE0C, 'M', 'م'), - (0x1EE0D, 'M', 'ن'), - (0x1EE0E, 'M', 'س'), - (0x1EE0F, 'M', 'ع'), - (0x1EE10, 'M', 'ف'), - (0x1EE11, 'M', 'ص'), - (0x1EE12, 'M', 'ق'), - (0x1EE13, 'M', 'ر'), - (0x1EE14, 'M', 'ش'), - (0x1EE15, 'M', 'ت'), - (0x1EE16, 'M', 'ث'), - (0x1EE17, 'M', 'خ'), - (0x1EE18, 'M', 'ذ'), - (0x1EE19, 'M', 'ض'), - (0x1EE1A, 'M', 'ظ'), - (0x1EE1B, 'M', 'غ'), - (0x1EE1C, 'M', 'ٮ'), - (0x1EE1D, 'M', 'ں'), - (0x1EE1E, 'M', 'ڡ'), - (0x1EE1F, 'M', 'ٯ'), - (0x1EE20, 'X'), - (0x1EE21, 'M', 'ب'), - (0x1EE22, 'M', 'ج'), - (0x1EE23, 'X'), - (0x1EE24, 'M', 'ه'), - (0x1EE25, 'X'), - (0x1EE27, 'M', 'ح'), - (0x1EE28, 'X'), - (0x1EE29, 'M', 'ي'), - (0x1EE2A, 'M', 'ك'), - (0x1EE2B, 'M', 'ل'), - (0x1EE2C, 'M', 'م'), - (0x1EE2D, 'M', 'ن'), - (0x1EE2E, 'M', 'س'), - (0x1EE2F, 'M', 'ع'), - (0x1EE30, 'M', 'ف'), - (0x1EE31, 'M', 'ص'), - (0x1EE32, 'M', 'ق'), - (0x1EE33, 'X'), - (0x1EE34, 'M', 'ش'), - (0x1EE35, 'M', 'ت'), - (0x1EE36, 'M', 'ث'), - (0x1EE37, 'M', 'خ'), - (0x1EE38, 'X'), - (0x1EE39, 'M', 'ض'), - (0x1EE3A, 'X'), - (0x1EE3B, 'M', 'غ'), - (0x1EE3C, 'X'), - (0x1EE42, 'M', 'ج'), - (0x1EE43, 'X'), - (0x1EE47, 'M', 'ح'), - (0x1EE48, 'X'), - (0x1EE49, 'M', 'ي'), - (0x1EE4A, 'X'), - ] - -def _seg_73() -> List[Union[Tuple[int, str], Tuple[int, str, str]]]: - return [ - (0x1EE4B, 'M', 'ل'), - (0x1EE4C, 'X'), - (0x1EE4D, 'M', 'ن'), - (0x1EE4E, 'M', 'س'), - (0x1EE4F, 'M', 'ع'), - (0x1EE50, 'X'), - (0x1EE51, 'M', 'ص'), - (0x1EE52, 'M', 'ق'), - (0x1EE53, 'X'), - (0x1EE54, 'M', 'ش'), - (0x1EE55, 'X'), - (0x1EE57, 'M', 'خ'), - (0x1EE58, 'X'), - (0x1EE59, 'M', 'ض'), - (0x1EE5A, 'X'), - (0x1EE5B, 'M', 'غ'), - (0x1EE5C, 'X'), - (0x1EE5D, 'M', 'ں'), - (0x1EE5E, 'X'), - (0x1EE5F, 'M', 'ٯ'), - (0x1EE60, 'X'), - (0x1EE61, 'M', 'ب'), - (0x1EE62, 'M', 'ج'), - (0x1EE63, 'X'), - (0x1EE64, 'M', 'ه'), - (0x1EE65, 'X'), - (0x1EE67, 'M', 'ح'), - (0x1EE68, 'M', 'ط'), - (0x1EE69, 'M', 'ي'), - (0x1EE6A, 'M', 'ك'), - (0x1EE6B, 'X'), - (0x1EE6C, 'M', 'م'), - (0x1EE6D, 'M', 'ن'), - (0x1EE6E, 'M', 'س'), - (0x1EE6F, 'M', 'ع'), - (0x1EE70, 'M', 'ف'), - (0x1EE71, 'M', 'ص'), - (0x1EE72, 'M', 'ق'), - (0x1EE73, 'X'), - (0x1EE74, 'M', 'ش'), - (0x1EE75, 'M', 'ت'), - (0x1EE76, 'M', 'ث'), - (0x1EE77, 'M', 'خ'), - (0x1EE78, 'X'), - (0x1EE79, 'M', 'ض'), - (0x1EE7A, 'M', 'ظ'), - (0x1EE7B, 'M', 'غ'), - (0x1EE7C, 'M', 'ٮ'), - (0x1EE7D, 'X'), - (0x1EE7E, 'M', 'ڡ'), - (0x1EE7F, 'X'), - (0x1EE80, 'M', 'ا'), - (0x1EE81, 'M', 'ب'), - (0x1EE82, 'M', 'ج'), - (0x1EE83, 'M', 'د'), - (0x1EE84, 'M', 'ه'), - (0x1EE85, 'M', 'و'), - (0x1EE86, 'M', 'ز'), - (0x1EE87, 'M', 'ح'), - (0x1EE88, 'M', 'ط'), - (0x1EE89, 'M', 'ي'), - (0x1EE8A, 'X'), - (0x1EE8B, 'M', 'ل'), - (0x1EE8C, 'M', 'م'), - (0x1EE8D, 'M', 'ن'), - (0x1EE8E, 'M', 'س'), - (0x1EE8F, 'M', 'ع'), - (0x1EE90, 'M', 'ف'), - (0x1EE91, 'M', 'ص'), - (0x1EE92, 'M', 'ق'), - (0x1EE93, 'M', 'ر'), - (0x1EE94, 'M', 'ش'), - (0x1EE95, 'M', 'ت'), - (0x1EE96, 'M', 'ث'), - (0x1EE97, 'M', 'خ'), - (0x1EE98, 'M', 'ذ'), - (0x1EE99, 'M', 'ض'), - (0x1EE9A, 'M', 'ظ'), - (0x1EE9B, 'M', 'غ'), - (0x1EE9C, 'X'), - (0x1EEA1, 'M', 'ب'), - (0x1EEA2, 'M', 'ج'), - (0x1EEA3, 'M', 'د'), - (0x1EEA4, 'X'), - (0x1EEA5, 'M', 'و'), - (0x1EEA6, 'M', 'ز'), - (0x1EEA7, 'M', 'ح'), - (0x1EEA8, 'M', 'ط'), - (0x1EEA9, 'M', 'ي'), - (0x1EEAA, 'X'), - (0x1EEAB, 'M', 'ل'), - (0x1EEAC, 'M', 'م'), - (0x1EEAD, 'M', 'ن'), - (0x1EEAE, 'M', 'س'), - (0x1EEAF, 'M', 'ع'), - (0x1EEB0, 'M', 'ف'), - (0x1EEB1, 'M', 'ص'), - (0x1EEB2, 'M', 'ق'), - (0x1EEB3, 'M', 'ر'), - (0x1EEB4, 'M', 'ش'), - ] - -def _seg_74() -> List[Union[Tuple[int, str], Tuple[int, str, str]]]: - return [ - (0x1EEB5, 'M', 'ت'), - (0x1EEB6, 'M', 'ث'), - (0x1EEB7, 'M', 'خ'), - (0x1EEB8, 'M', 'ذ'), - (0x1EEB9, 'M', 'ض'), - (0x1EEBA, 'M', 'ظ'), - (0x1EEBB, 'M', 'غ'), - (0x1EEBC, 'X'), - (0x1EEF0, 'V'), - (0x1EEF2, 'X'), - (0x1F000, 'V'), - (0x1F02C, 'X'), - (0x1F030, 'V'), - (0x1F094, 'X'), - (0x1F0A0, 'V'), - (0x1F0AF, 'X'), - (0x1F0B1, 'V'), - (0x1F0C0, 'X'), - (0x1F0C1, 'V'), - (0x1F0D0, 'X'), - (0x1F0D1, 'V'), - (0x1F0F6, 'X'), - (0x1F101, '3', '0,'), - (0x1F102, '3', '1,'), - (0x1F103, '3', '2,'), - (0x1F104, '3', '3,'), - (0x1F105, '3', '4,'), - (0x1F106, '3', '5,'), - (0x1F107, '3', '6,'), - (0x1F108, '3', '7,'), - (0x1F109, '3', '8,'), - (0x1F10A, '3', '9,'), - (0x1F10B, 'V'), - (0x1F110, '3', '(a)'), - (0x1F111, '3', '(b)'), - (0x1F112, '3', '(c)'), - (0x1F113, '3', '(d)'), - (0x1F114, '3', '(e)'), - (0x1F115, '3', '(f)'), - (0x1F116, '3', '(g)'), - (0x1F117, '3', '(h)'), - (0x1F118, '3', '(i)'), - (0x1F119, '3', '(j)'), - (0x1F11A, '3', '(k)'), - (0x1F11B, '3', '(l)'), - (0x1F11C, '3', '(m)'), - (0x1F11D, '3', '(n)'), - (0x1F11E, '3', '(o)'), - (0x1F11F, '3', '(p)'), - (0x1F120, '3', '(q)'), - (0x1F121, '3', '(r)'), - (0x1F122, '3', '(s)'), - (0x1F123, '3', '(t)'), - (0x1F124, '3', '(u)'), - (0x1F125, '3', '(v)'), - (0x1F126, '3', '(w)'), - (0x1F127, '3', '(x)'), - (0x1F128, '3', '(y)'), - (0x1F129, '3', '(z)'), - (0x1F12A, 'M', '〔s〕'), - (0x1F12B, 'M', 'c'), - (0x1F12C, 'M', 'r'), - (0x1F12D, 'M', 'cd'), - (0x1F12E, 'M', 'wz'), - (0x1F12F, 'V'), - (0x1F130, 'M', 'a'), - (0x1F131, 'M', 'b'), - (0x1F132, 'M', 'c'), - (0x1F133, 'M', 'd'), - (0x1F134, 'M', 'e'), - (0x1F135, 'M', 'f'), - (0x1F136, 'M', 'g'), - (0x1F137, 'M', 'h'), - (0x1F138, 'M', 'i'), - (0x1F139, 'M', 'j'), - (0x1F13A, 'M', 'k'), - (0x1F13B, 'M', 'l'), - (0x1F13C, 'M', 'm'), - (0x1F13D, 'M', 'n'), - (0x1F13E, 'M', 'o'), - (0x1F13F, 'M', 'p'), - (0x1F140, 'M', 'q'), - (0x1F141, 'M', 'r'), - (0x1F142, 'M', 's'), - (0x1F143, 'M', 't'), - (0x1F144, 'M', 'u'), - (0x1F145, 'M', 'v'), - (0x1F146, 'M', 'w'), - (0x1F147, 'M', 'x'), - (0x1F148, 'M', 'y'), - (0x1F149, 'M', 'z'), - (0x1F14A, 'M', 'hv'), - (0x1F14B, 'M', 'mv'), - (0x1F14C, 'M', 'sd'), - (0x1F14D, 'M', 'ss'), - (0x1F14E, 'M', 'ppv'), - (0x1F14F, 'M', 'wc'), - (0x1F150, 'V'), - (0x1F16A, 'M', 'mc'), - (0x1F16B, 'M', 'md'), - ] - -def _seg_75() -> List[Union[Tuple[int, str], Tuple[int, str, str]]]: - return [ - (0x1F16C, 'M', 'mr'), - (0x1F16D, 'V'), - (0x1F190, 'M', 'dj'), - (0x1F191, 'V'), - (0x1F1AE, 'X'), - (0x1F1E6, 'V'), - (0x1F200, 'M', 'ほか'), - (0x1F201, 'M', 'ココ'), - (0x1F202, 'M', 'サ'), - (0x1F203, 'X'), - (0x1F210, 'M', '手'), - (0x1F211, 'M', '字'), - (0x1F212, 'M', '双'), - (0x1F213, 'M', 'デ'), - (0x1F214, 'M', '二'), - (0x1F215, 'M', '多'), - (0x1F216, 'M', '解'), - (0x1F217, 'M', '天'), - (0x1F218, 'M', '交'), - (0x1F219, 'M', '映'), - (0x1F21A, 'M', '無'), - (0x1F21B, 'M', '料'), - (0x1F21C, 'M', '前'), - (0x1F21D, 'M', '後'), - (0x1F21E, 'M', '再'), - (0x1F21F, 'M', '新'), - (0x1F220, 'M', '初'), - (0x1F221, 'M', '終'), - (0x1F222, 'M', '生'), - (0x1F223, 'M', '販'), - (0x1F224, 'M', '声'), - (0x1F225, 'M', '吹'), - (0x1F226, 'M', '演'), - (0x1F227, 'M', '投'), - (0x1F228, 'M', '捕'), - (0x1F229, 'M', '一'), - (0x1F22A, 'M', '三'), - (0x1F22B, 'M', '遊'), - (0x1F22C, 'M', '左'), - (0x1F22D, 'M', '中'), - (0x1F22E, 'M', '右'), - (0x1F22F, 'M', '指'), - (0x1F230, 'M', '走'), - (0x1F231, 'M', '打'), - (0x1F232, 'M', '禁'), - (0x1F233, 'M', '空'), - (0x1F234, 'M', '合'), - (0x1F235, 'M', '満'), - (0x1F236, 'M', '有'), - (0x1F237, 'M', '月'), - (0x1F238, 'M', '申'), - (0x1F239, 'M', '割'), - (0x1F23A, 'M', '営'), - (0x1F23B, 'M', '配'), - (0x1F23C, 'X'), - (0x1F240, 'M', '〔本〕'), - (0x1F241, 'M', '〔三〕'), - (0x1F242, 'M', '〔二〕'), - (0x1F243, 'M', '〔安〕'), - (0x1F244, 'M', '〔点〕'), - (0x1F245, 'M', '〔打〕'), - (0x1F246, 'M', '〔盗〕'), - (0x1F247, 'M', '〔勝〕'), - (0x1F248, 'M', '〔敗〕'), - (0x1F249, 'X'), - (0x1F250, 'M', '得'), - (0x1F251, 'M', '可'), - (0x1F252, 'X'), - (0x1F260, 'V'), - (0x1F266, 'X'), - (0x1F300, 'V'), - (0x1F6D8, 'X'), - (0x1F6DC, 'V'), - (0x1F6ED, 'X'), - (0x1F6F0, 'V'), - (0x1F6FD, 'X'), - (0x1F700, 'V'), - (0x1F777, 'X'), - (0x1F77B, 'V'), - (0x1F7DA, 'X'), - (0x1F7E0, 'V'), - (0x1F7EC, 'X'), - (0x1F7F0, 'V'), - (0x1F7F1, 'X'), - (0x1F800, 'V'), - (0x1F80C, 'X'), - (0x1F810, 'V'), - (0x1F848, 'X'), - (0x1F850, 'V'), - (0x1F85A, 'X'), - (0x1F860, 'V'), - (0x1F888, 'X'), - (0x1F890, 'V'), - (0x1F8AE, 'X'), - (0x1F8B0, 'V'), - (0x1F8B2, 'X'), - (0x1F900, 'V'), - (0x1FA54, 'X'), - (0x1FA60, 'V'), - (0x1FA6E, 'X'), - ] - -def _seg_76() -> List[Union[Tuple[int, str], Tuple[int, str, str]]]: - return [ - (0x1FA70, 'V'), - (0x1FA7D, 'X'), - (0x1FA80, 'V'), - (0x1FA89, 'X'), - (0x1FA90, 'V'), - (0x1FABE, 'X'), - (0x1FABF, 'V'), - (0x1FAC6, 'X'), - (0x1FACE, 'V'), - (0x1FADC, 'X'), - (0x1FAE0, 'V'), - (0x1FAE9, 'X'), - (0x1FAF0, 'V'), - (0x1FAF9, 'X'), - (0x1FB00, 'V'), - (0x1FB93, 'X'), - (0x1FB94, 'V'), - (0x1FBCB, 'X'), - (0x1FBF0, 'M', '0'), - (0x1FBF1, 'M', '1'), - (0x1FBF2, 'M', '2'), - (0x1FBF3, 'M', '3'), - (0x1FBF4, 'M', '4'), - (0x1FBF5, 'M', '5'), - (0x1FBF6, 'M', '6'), - (0x1FBF7, 'M', '7'), - (0x1FBF8, 'M', '8'), - (0x1FBF9, 'M', '9'), - (0x1FBFA, 'X'), - (0x20000, 'V'), - (0x2A6E0, 'X'), - (0x2A700, 'V'), - (0x2B73A, 'X'), - (0x2B740, 'V'), - (0x2B81E, 'X'), - (0x2B820, 'V'), - (0x2CEA2, 'X'), - (0x2CEB0, 'V'), - (0x2EBE1, 'X'), - (0x2F800, 'M', '丽'), - (0x2F801, 'M', '丸'), - (0x2F802, 'M', '乁'), - (0x2F803, 'M', '𠄢'), - (0x2F804, 'M', '你'), - (0x2F805, 'M', '侮'), - (0x2F806, 'M', '侻'), - (0x2F807, 'M', '倂'), - (0x2F808, 'M', '偺'), - (0x2F809, 'M', '備'), - (0x2F80A, 'M', '僧'), - (0x2F80B, 'M', '像'), - (0x2F80C, 'M', '㒞'), - (0x2F80D, 'M', '𠘺'), - (0x2F80E, 'M', '免'), - (0x2F80F, 'M', '兔'), - (0x2F810, 'M', '兤'), - (0x2F811, 'M', '具'), - (0x2F812, 'M', '𠔜'), - (0x2F813, 'M', '㒹'), - (0x2F814, 'M', '內'), - (0x2F815, 'M', '再'), - (0x2F816, 'M', '𠕋'), - (0x2F817, 'M', '冗'), - (0x2F818, 'M', '冤'), - (0x2F819, 'M', '仌'), - (0x2F81A, 'M', '冬'), - (0x2F81B, 'M', '况'), - (0x2F81C, 'M', '𩇟'), - (0x2F81D, 'M', '凵'), - (0x2F81E, 'M', '刃'), - (0x2F81F, 'M', '㓟'), - (0x2F820, 'M', '刻'), - (0x2F821, 'M', '剆'), - (0x2F822, 'M', '割'), - (0x2F823, 'M', '剷'), - (0x2F824, 'M', '㔕'), - (0x2F825, 'M', '勇'), - (0x2F826, 'M', '勉'), - (0x2F827, 'M', '勤'), - (0x2F828, 'M', '勺'), - (0x2F829, 'M', '包'), - (0x2F82A, 'M', '匆'), - (0x2F82B, 'M', '北'), - (0x2F82C, 'M', '卉'), - (0x2F82D, 'M', '卑'), - (0x2F82E, 'M', '博'), - (0x2F82F, 'M', '即'), - (0x2F830, 'M', '卽'), - (0x2F831, 'M', '卿'), - (0x2F834, 'M', '𠨬'), - (0x2F835, 'M', '灰'), - (0x2F836, 'M', '及'), - (0x2F837, 'M', '叟'), - (0x2F838, 'M', '𠭣'), - (0x2F839, 'M', '叫'), - (0x2F83A, 'M', '叱'), - (0x2F83B, 'M', '吆'), - (0x2F83C, 'M', '咞'), - (0x2F83D, 'M', '吸'), - (0x2F83E, 'M', '呈'), - ] - -def _seg_77() -> List[Union[Tuple[int, str], Tuple[int, str, str]]]: - return [ - (0x2F83F, 'M', '周'), - (0x2F840, 'M', '咢'), - (0x2F841, 'M', '哶'), - (0x2F842, 'M', '唐'), - (0x2F843, 'M', '啓'), - (0x2F844, 'M', '啣'), - (0x2F845, 'M', '善'), - (0x2F847, 'M', '喙'), - (0x2F848, 'M', '喫'), - (0x2F849, 'M', '喳'), - (0x2F84A, 'M', '嗂'), - (0x2F84B, 'M', '圖'), - (0x2F84C, 'M', '嘆'), - (0x2F84D, 'M', '圗'), - (0x2F84E, 'M', '噑'), - (0x2F84F, 'M', '噴'), - (0x2F850, 'M', '切'), - (0x2F851, 'M', '壮'), - (0x2F852, 'M', '城'), - (0x2F853, 'M', '埴'), - (0x2F854, 'M', '堍'), - (0x2F855, 'M', '型'), - (0x2F856, 'M', '堲'), - (0x2F857, 'M', '報'), - (0x2F858, 'M', '墬'), - (0x2F859, 'M', '𡓤'), - (0x2F85A, 'M', '売'), - (0x2F85B, 'M', '壷'), - (0x2F85C, 'M', '夆'), - (0x2F85D, 'M', '多'), - (0x2F85E, 'M', '夢'), - (0x2F85F, 'M', '奢'), - (0x2F860, 'M', '𡚨'), - (0x2F861, 'M', '𡛪'), - (0x2F862, 'M', '姬'), - (0x2F863, 'M', '娛'), - (0x2F864, 'M', '娧'), - (0x2F865, 'M', '姘'), - (0x2F866, 'M', '婦'), - (0x2F867, 'M', '㛮'), - (0x2F868, 'X'), - (0x2F869, 'M', '嬈'), - (0x2F86A, 'M', '嬾'), - (0x2F86C, 'M', '𡧈'), - (0x2F86D, 'M', '寃'), - (0x2F86E, 'M', '寘'), - (0x2F86F, 'M', '寧'), - (0x2F870, 'M', '寳'), - (0x2F871, 'M', '𡬘'), - (0x2F872, 'M', '寿'), - (0x2F873, 'M', '将'), - (0x2F874, 'X'), - (0x2F875, 'M', '尢'), - (0x2F876, 'M', '㞁'), - (0x2F877, 'M', '屠'), - (0x2F878, 'M', '屮'), - (0x2F879, 'M', '峀'), - (0x2F87A, 'M', '岍'), - (0x2F87B, 'M', '𡷤'), - (0x2F87C, 'M', '嵃'), - (0x2F87D, 'M', '𡷦'), - (0x2F87E, 'M', '嵮'), - (0x2F87F, 'M', '嵫'), - (0x2F880, 'M', '嵼'), - (0x2F881, 'M', '巡'), - (0x2F882, 'M', '巢'), - (0x2F883, 'M', '㠯'), - (0x2F884, 'M', '巽'), - (0x2F885, 'M', '帨'), - (0x2F886, 'M', '帽'), - (0x2F887, 'M', '幩'), - (0x2F888, 'M', '㡢'), - (0x2F889, 'M', '𢆃'), - (0x2F88A, 'M', '㡼'), - (0x2F88B, 'M', '庰'), - (0x2F88C, 'M', '庳'), - (0x2F88D, 'M', '庶'), - (0x2F88E, 'M', '廊'), - (0x2F88F, 'M', '𪎒'), - (0x2F890, 'M', '廾'), - (0x2F891, 'M', '𢌱'), - (0x2F893, 'M', '舁'), - (0x2F894, 'M', '弢'), - (0x2F896, 'M', '㣇'), - (0x2F897, 'M', '𣊸'), - (0x2F898, 'M', '𦇚'), - (0x2F899, 'M', '形'), - (0x2F89A, 'M', '彫'), - (0x2F89B, 'M', '㣣'), - (0x2F89C, 'M', '徚'), - (0x2F89D, 'M', '忍'), - (0x2F89E, 'M', '志'), - (0x2F89F, 'M', '忹'), - (0x2F8A0, 'M', '悁'), - (0x2F8A1, 'M', '㤺'), - (0x2F8A2, 'M', '㤜'), - (0x2F8A3, 'M', '悔'), - (0x2F8A4, 'M', '𢛔'), - (0x2F8A5, 'M', '惇'), - (0x2F8A6, 'M', '慈'), - ] - -def _seg_78() -> List[Union[Tuple[int, str], Tuple[int, str, str]]]: - return [ - (0x2F8A7, 'M', '慌'), - (0x2F8A8, 'M', '慎'), - (0x2F8A9, 'M', '慌'), - (0x2F8AA, 'M', '慺'), - (0x2F8AB, 'M', '憎'), - (0x2F8AC, 'M', '憲'), - (0x2F8AD, 'M', '憤'), - (0x2F8AE, 'M', '憯'), - (0x2F8AF, 'M', '懞'), - (0x2F8B0, 'M', '懲'), - (0x2F8B1, 'M', '懶'), - (0x2F8B2, 'M', '成'), - (0x2F8B3, 'M', '戛'), - (0x2F8B4, 'M', '扝'), - (0x2F8B5, 'M', '抱'), - (0x2F8B6, 'M', '拔'), - (0x2F8B7, 'M', '捐'), - (0x2F8B8, 'M', '𢬌'), - (0x2F8B9, 'M', '挽'), - (0x2F8BA, 'M', '拼'), - (0x2F8BB, 'M', '捨'), - (0x2F8BC, 'M', '掃'), - (0x2F8BD, 'M', '揤'), - (0x2F8BE, 'M', '𢯱'), - (0x2F8BF, 'M', '搢'), - (0x2F8C0, 'M', '揅'), - (0x2F8C1, 'M', '掩'), - (0x2F8C2, 'M', '㨮'), - (0x2F8C3, 'M', '摩'), - (0x2F8C4, 'M', '摾'), - (0x2F8C5, 'M', '撝'), - (0x2F8C6, 'M', '摷'), - (0x2F8C7, 'M', '㩬'), - (0x2F8C8, 'M', '敏'), - (0x2F8C9, 'M', '敬'), - (0x2F8CA, 'M', '𣀊'), - (0x2F8CB, 'M', '旣'), - (0x2F8CC, 'M', '書'), - (0x2F8CD, 'M', '晉'), - (0x2F8CE, 'M', '㬙'), - (0x2F8CF, 'M', '暑'), - (0x2F8D0, 'M', '㬈'), - (0x2F8D1, 'M', '㫤'), - (0x2F8D2, 'M', '冒'), - (0x2F8D3, 'M', '冕'), - (0x2F8D4, 'M', '最'), - (0x2F8D5, 'M', '暜'), - (0x2F8D6, 'M', '肭'), - (0x2F8D7, 'M', '䏙'), - (0x2F8D8, 'M', '朗'), - (0x2F8D9, 'M', '望'), - (0x2F8DA, 'M', '朡'), - (0x2F8DB, 'M', '杞'), - (0x2F8DC, 'M', '杓'), - (0x2F8DD, 'M', '𣏃'), - (0x2F8DE, 'M', '㭉'), - (0x2F8DF, 'M', '柺'), - (0x2F8E0, 'M', '枅'), - (0x2F8E1, 'M', '桒'), - (0x2F8E2, 'M', '梅'), - (0x2F8E3, 'M', '𣑭'), - (0x2F8E4, 'M', '梎'), - (0x2F8E5, 'M', '栟'), - (0x2F8E6, 'M', '椔'), - (0x2F8E7, 'M', '㮝'), - (0x2F8E8, 'M', '楂'), - (0x2F8E9, 'M', '榣'), - (0x2F8EA, 'M', '槪'), - (0x2F8EB, 'M', '檨'), - (0x2F8EC, 'M', '𣚣'), - (0x2F8ED, 'M', '櫛'), - (0x2F8EE, 'M', '㰘'), - (0x2F8EF, 'M', '次'), - (0x2F8F0, 'M', '𣢧'), - (0x2F8F1, 'M', '歔'), - (0x2F8F2, 'M', '㱎'), - (0x2F8F3, 'M', '歲'), - (0x2F8F4, 'M', '殟'), - (0x2F8F5, 'M', '殺'), - (0x2F8F6, 'M', '殻'), - (0x2F8F7, 'M', '𣪍'), - (0x2F8F8, 'M', '𡴋'), - (0x2F8F9, 'M', '𣫺'), - (0x2F8FA, 'M', '汎'), - (0x2F8FB, 'M', '𣲼'), - (0x2F8FC, 'M', '沿'), - (0x2F8FD, 'M', '泍'), - (0x2F8FE, 'M', '汧'), - (0x2F8FF, 'M', '洖'), - (0x2F900, 'M', '派'), - (0x2F901, 'M', '海'), - (0x2F902, 'M', '流'), - (0x2F903, 'M', '浩'), - (0x2F904, 'M', '浸'), - (0x2F905, 'M', '涅'), - (0x2F906, 'M', '𣴞'), - (0x2F907, 'M', '洴'), - (0x2F908, 'M', '港'), - (0x2F909, 'M', '湮'), - (0x2F90A, 'M', '㴳'), - ] - -def _seg_79() -> List[Union[Tuple[int, str], Tuple[int, str, str]]]: - return [ - (0x2F90B, 'M', '滋'), - (0x2F90C, 'M', '滇'), - (0x2F90D, 'M', '𣻑'), - (0x2F90E, 'M', '淹'), - (0x2F90F, 'M', '潮'), - (0x2F910, 'M', '𣽞'), - (0x2F911, 'M', '𣾎'), - (0x2F912, 'M', '濆'), - (0x2F913, 'M', '瀹'), - (0x2F914, 'M', '瀞'), - (0x2F915, 'M', '瀛'), - (0x2F916, 'M', '㶖'), - (0x2F917, 'M', '灊'), - (0x2F918, 'M', '災'), - (0x2F919, 'M', '灷'), - (0x2F91A, 'M', '炭'), - (0x2F91B, 'M', '𠔥'), - (0x2F91C, 'M', '煅'), - (0x2F91D, 'M', '𤉣'), - (0x2F91E, 'M', '熜'), - (0x2F91F, 'X'), - (0x2F920, 'M', '爨'), - (0x2F921, 'M', '爵'), - (0x2F922, 'M', '牐'), - (0x2F923, 'M', '𤘈'), - (0x2F924, 'M', '犀'), - (0x2F925, 'M', '犕'), - (0x2F926, 'M', '𤜵'), - (0x2F927, 'M', '𤠔'), - (0x2F928, 'M', '獺'), - (0x2F929, 'M', '王'), - (0x2F92A, 'M', '㺬'), - (0x2F92B, 'M', '玥'), - (0x2F92C, 'M', '㺸'), - (0x2F92E, 'M', '瑇'), - (0x2F92F, 'M', '瑜'), - (0x2F930, 'M', '瑱'), - (0x2F931, 'M', '璅'), - (0x2F932, 'M', '瓊'), - (0x2F933, 'M', '㼛'), - (0x2F934, 'M', '甤'), - (0x2F935, 'M', '𤰶'), - (0x2F936, 'M', '甾'), - (0x2F937, 'M', '𤲒'), - (0x2F938, 'M', '異'), - (0x2F939, 'M', '𢆟'), - (0x2F93A, 'M', '瘐'), - (0x2F93B, 'M', '𤾡'), - (0x2F93C, 'M', '𤾸'), - (0x2F93D, 'M', '𥁄'), - (0x2F93E, 'M', '㿼'), - (0x2F93F, 'M', '䀈'), - (0x2F940, 'M', '直'), - (0x2F941, 'M', '𥃳'), - (0x2F942, 'M', '𥃲'), - (0x2F943, 'M', '𥄙'), - (0x2F944, 'M', '𥄳'), - (0x2F945, 'M', '眞'), - (0x2F946, 'M', '真'), - (0x2F948, 'M', '睊'), - (0x2F949, 'M', '䀹'), - (0x2F94A, 'M', '瞋'), - (0x2F94B, 'M', '䁆'), - (0x2F94C, 'M', '䂖'), - (0x2F94D, 'M', '𥐝'), - (0x2F94E, 'M', '硎'), - (0x2F94F, 'M', '碌'), - (0x2F950, 'M', '磌'), - (0x2F951, 'M', '䃣'), - (0x2F952, 'M', '𥘦'), - (0x2F953, 'M', '祖'), - (0x2F954, 'M', '𥚚'), - (0x2F955, 'M', '𥛅'), - (0x2F956, 'M', '福'), - (0x2F957, 'M', '秫'), - (0x2F958, 'M', '䄯'), - (0x2F959, 'M', '穀'), - (0x2F95A, 'M', '穊'), - (0x2F95B, 'M', '穏'), - (0x2F95C, 'M', '𥥼'), - (0x2F95D, 'M', '𥪧'), - (0x2F95F, 'X'), - (0x2F960, 'M', '䈂'), - (0x2F961, 'M', '𥮫'), - (0x2F962, 'M', '篆'), - (0x2F963, 'M', '築'), - (0x2F964, 'M', '䈧'), - (0x2F965, 'M', '𥲀'), - (0x2F966, 'M', '糒'), - (0x2F967, 'M', '䊠'), - (0x2F968, 'M', '糨'), - (0x2F969, 'M', '糣'), - (0x2F96A, 'M', '紀'), - (0x2F96B, 'M', '𥾆'), - (0x2F96C, 'M', '絣'), - (0x2F96D, 'M', '䌁'), - (0x2F96E, 'M', '緇'), - (0x2F96F, 'M', '縂'), - (0x2F970, 'M', '繅'), - (0x2F971, 'M', '䌴'), - ] - -def _seg_80() -> List[Union[Tuple[int, str], Tuple[int, str, str]]]: - return [ - (0x2F972, 'M', '𦈨'), - (0x2F973, 'M', '𦉇'), - (0x2F974, 'M', '䍙'), - (0x2F975, 'M', '𦋙'), - (0x2F976, 'M', '罺'), - (0x2F977, 'M', '𦌾'), - (0x2F978, 'M', '羕'), - (0x2F979, 'M', '翺'), - (0x2F97A, 'M', '者'), - (0x2F97B, 'M', '𦓚'), - (0x2F97C, 'M', '𦔣'), - (0x2F97D, 'M', '聠'), - (0x2F97E, 'M', '𦖨'), - (0x2F97F, 'M', '聰'), - (0x2F980, 'M', '𣍟'), - (0x2F981, 'M', '䏕'), - (0x2F982, 'M', '育'), - (0x2F983, 'M', '脃'), - (0x2F984, 'M', '䐋'), - (0x2F985, 'M', '脾'), - (0x2F986, 'M', '媵'), - (0x2F987, 'M', '𦞧'), - (0x2F988, 'M', '𦞵'), - (0x2F989, 'M', '𣎓'), - (0x2F98A, 'M', '𣎜'), - (0x2F98B, 'M', '舁'), - (0x2F98C, 'M', '舄'), - (0x2F98D, 'M', '辞'), - (0x2F98E, 'M', '䑫'), - (0x2F98F, 'M', '芑'), - (0x2F990, 'M', '芋'), - (0x2F991, 'M', '芝'), - (0x2F992, 'M', '劳'), - (0x2F993, 'M', '花'), - (0x2F994, 'M', '芳'), - (0x2F995, 'M', '芽'), - (0x2F996, 'M', '苦'), - (0x2F997, 'M', '𦬼'), - (0x2F998, 'M', '若'), - (0x2F999, 'M', '茝'), - (0x2F99A, 'M', '荣'), - (0x2F99B, 'M', '莭'), - (0x2F99C, 'M', '茣'), - (0x2F99D, 'M', '莽'), - (0x2F99E, 'M', '菧'), - (0x2F99F, 'M', '著'), - (0x2F9A0, 'M', '荓'), - (0x2F9A1, 'M', '菊'), - (0x2F9A2, 'M', '菌'), - (0x2F9A3, 'M', '菜'), - (0x2F9A4, 'M', '𦰶'), - (0x2F9A5, 'M', '𦵫'), - (0x2F9A6, 'M', '𦳕'), - (0x2F9A7, 'M', '䔫'), - (0x2F9A8, 'M', '蓱'), - (0x2F9A9, 'M', '蓳'), - (0x2F9AA, 'M', '蔖'), - (0x2F9AB, 'M', '𧏊'), - (0x2F9AC, 'M', '蕤'), - (0x2F9AD, 'M', '𦼬'), - (0x2F9AE, 'M', '䕝'), - (0x2F9AF, 'M', '䕡'), - (0x2F9B0, 'M', '𦾱'), - (0x2F9B1, 'M', '𧃒'), - (0x2F9B2, 'M', '䕫'), - (0x2F9B3, 'M', '虐'), - (0x2F9B4, 'M', '虜'), - (0x2F9B5, 'M', '虧'), - (0x2F9B6, 'M', '虩'), - (0x2F9B7, 'M', '蚩'), - (0x2F9B8, 'M', '蚈'), - (0x2F9B9, 'M', '蜎'), - (0x2F9BA, 'M', '蛢'), - (0x2F9BB, 'M', '蝹'), - (0x2F9BC, 'M', '蜨'), - (0x2F9BD, 'M', '蝫'), - (0x2F9BE, 'M', '螆'), - (0x2F9BF, 'X'), - (0x2F9C0, 'M', '蟡'), - (0x2F9C1, 'M', '蠁'), - (0x2F9C2, 'M', '䗹'), - (0x2F9C3, 'M', '衠'), - (0x2F9C4, 'M', '衣'), - (0x2F9C5, 'M', '𧙧'), - (0x2F9C6, 'M', '裗'), - (0x2F9C7, 'M', '裞'), - (0x2F9C8, 'M', '䘵'), - (0x2F9C9, 'M', '裺'), - (0x2F9CA, 'M', '㒻'), - (0x2F9CB, 'M', '𧢮'), - (0x2F9CC, 'M', '𧥦'), - (0x2F9CD, 'M', '䚾'), - (0x2F9CE, 'M', '䛇'), - (0x2F9CF, 'M', '誠'), - (0x2F9D0, 'M', '諭'), - (0x2F9D1, 'M', '變'), - (0x2F9D2, 'M', '豕'), - (0x2F9D3, 'M', '𧲨'), - (0x2F9D4, 'M', '貫'), - (0x2F9D5, 'M', '賁'), - ] - -def _seg_81() -> List[Union[Tuple[int, str], Tuple[int, str, str]]]: - return [ - (0x2F9D6, 'M', '贛'), - (0x2F9D7, 'M', '起'), - (0x2F9D8, 'M', '𧼯'), - (0x2F9D9, 'M', '𠠄'), - (0x2F9DA, 'M', '跋'), - (0x2F9DB, 'M', '趼'), - (0x2F9DC, 'M', '跰'), - (0x2F9DD, 'M', '𠣞'), - (0x2F9DE, 'M', '軔'), - (0x2F9DF, 'M', '輸'), - (0x2F9E0, 'M', '𨗒'), - (0x2F9E1, 'M', '𨗭'), - (0x2F9E2, 'M', '邔'), - (0x2F9E3, 'M', '郱'), - (0x2F9E4, 'M', '鄑'), - (0x2F9E5, 'M', '𨜮'), - (0x2F9E6, 'M', '鄛'), - (0x2F9E7, 'M', '鈸'), - (0x2F9E8, 'M', '鋗'), - (0x2F9E9, 'M', '鋘'), - (0x2F9EA, 'M', '鉼'), - (0x2F9EB, 'M', '鏹'), - (0x2F9EC, 'M', '鐕'), - (0x2F9ED, 'M', '𨯺'), - (0x2F9EE, 'M', '開'), - (0x2F9EF, 'M', '䦕'), - (0x2F9F0, 'M', '閷'), - (0x2F9F1, 'M', '𨵷'), - (0x2F9F2, 'M', '䧦'), - (0x2F9F3, 'M', '雃'), - (0x2F9F4, 'M', '嶲'), - (0x2F9F5, 'M', '霣'), - (0x2F9F6, 'M', '𩅅'), - (0x2F9F7, 'M', '𩈚'), - (0x2F9F8, 'M', '䩮'), - (0x2F9F9, 'M', '䩶'), - (0x2F9FA, 'M', '韠'), - (0x2F9FB, 'M', '𩐊'), - (0x2F9FC, 'M', '䪲'), - (0x2F9FD, 'M', '𩒖'), - (0x2F9FE, 'M', '頋'), - (0x2FA00, 'M', '頩'), - (0x2FA01, 'M', '𩖶'), - (0x2FA02, 'M', '飢'), - (0x2FA03, 'M', '䬳'), - (0x2FA04, 'M', '餩'), - (0x2FA05, 'M', '馧'), - (0x2FA06, 'M', '駂'), - (0x2FA07, 'M', '駾'), - (0x2FA08, 'M', '䯎'), - (0x2FA09, 'M', '𩬰'), - (0x2FA0A, 'M', '鬒'), - (0x2FA0B, 'M', '鱀'), - (0x2FA0C, 'M', '鳽'), - (0x2FA0D, 'M', '䳎'), - (0x2FA0E, 'M', '䳭'), - (0x2FA0F, 'M', '鵧'), - (0x2FA10, 'M', '𪃎'), - (0x2FA11, 'M', '䳸'), - (0x2FA12, 'M', '𪄅'), - (0x2FA13, 'M', '𪈎'), - (0x2FA14, 'M', '𪊑'), - (0x2FA15, 'M', '麻'), - (0x2FA16, 'M', '䵖'), - (0x2FA17, 'M', '黹'), - (0x2FA18, 'M', '黾'), - (0x2FA19, 'M', '鼅'), - (0x2FA1A, 'M', '鼏'), - (0x2FA1B, 'M', '鼖'), - (0x2FA1C, 'M', '鼻'), - (0x2FA1D, 'M', '𪘀'), - (0x2FA1E, 'X'), - (0x30000, 'V'), - (0x3134B, 'X'), - (0x31350, 'V'), - (0x323B0, 'X'), - (0xE0100, 'I'), - (0xE01F0, 'X'), - ] - -uts46data = tuple( - _seg_0() - + _seg_1() - + _seg_2() - + _seg_3() - + _seg_4() - + _seg_5() - + _seg_6() - + _seg_7() - + _seg_8() - + _seg_9() - + _seg_10() - + _seg_11() - + _seg_12() - + _seg_13() - + _seg_14() - + _seg_15() - + _seg_16() - + _seg_17() - + _seg_18() - + _seg_19() - + _seg_20() - + _seg_21() - + _seg_22() - + _seg_23() - + _seg_24() - + _seg_25() - + _seg_26() - + _seg_27() - + _seg_28() - + _seg_29() - + _seg_30() - + _seg_31() - + _seg_32() - + _seg_33() - + _seg_34() - + _seg_35() - + _seg_36() - + _seg_37() - + _seg_38() - + _seg_39() - + _seg_40() - + _seg_41() - + _seg_42() - + _seg_43() - + _seg_44() - + _seg_45() - + _seg_46() - + _seg_47() - + _seg_48() - + _seg_49() - + _seg_50() - + _seg_51() - + _seg_52() - + _seg_53() - + _seg_54() - + _seg_55() - + _seg_56() - + _seg_57() - + _seg_58() - + _seg_59() - + _seg_60() - + _seg_61() - + _seg_62() - + _seg_63() - + _seg_64() - + _seg_65() - + _seg_66() - + _seg_67() - + _seg_68() - + _seg_69() - + _seg_70() - + _seg_71() - + _seg_72() - + _seg_73() - + _seg_74() - + _seg_75() - + _seg_76() - + _seg_77() - + _seg_78() - + _seg_79() - + _seg_80() - + _seg_81() -) # type: Tuple[Union[Tuple[int, str], Tuple[int, str, str]], ...] diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/msgpack/__init__.py b/venv/lib/python3.11/site-packages/pip/_vendor/msgpack/__init__.py deleted file mode 100644 index 1300b86..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/msgpack/__init__.py +++ /dev/null @@ -1,57 +0,0 @@ -# coding: utf-8 -from .exceptions import * -from .ext import ExtType, Timestamp - -import os -import sys - - -version = (1, 0, 5) -__version__ = "1.0.5" - - -if os.environ.get("MSGPACK_PUREPYTHON") or sys.version_info[0] == 2: - from .fallback import Packer, unpackb, Unpacker -else: - try: - from ._cmsgpack import Packer, unpackb, Unpacker - except ImportError: - from .fallback import Packer, unpackb, Unpacker - - -def pack(o, stream, **kwargs): - """ - Pack object `o` and write it to `stream` - - See :class:`Packer` for options. - """ - packer = Packer(**kwargs) - stream.write(packer.pack(o)) - - -def packb(o, **kwargs): - """ - Pack object `o` and return packed bytes - - See :class:`Packer` for options. - """ - return Packer(**kwargs).pack(o) - - -def unpack(stream, **kwargs): - """ - Unpack an object from `stream`. - - Raises `ExtraData` when `stream` contains extra bytes. - See :class:`Unpacker` for options. - """ - data = stream.read() - return unpackb(data, **kwargs) - - -# alias for compatibility to simplejson/marshal/pickle. -load = unpack -loads = unpackb - -dump = pack -dumps = packb diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/msgpack/__pycache__/__init__.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/msgpack/__pycache__/__init__.cpython-311.pyc deleted file mode 100644 index 54a60a4..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/msgpack/__pycache__/__init__.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/msgpack/__pycache__/exceptions.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/msgpack/__pycache__/exceptions.cpython-311.pyc deleted file mode 100644 index 4f85dd7..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/msgpack/__pycache__/exceptions.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/msgpack/__pycache__/ext.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/msgpack/__pycache__/ext.cpython-311.pyc deleted file mode 100644 index 2f9e68b..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/msgpack/__pycache__/ext.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/msgpack/__pycache__/fallback.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/msgpack/__pycache__/fallback.cpython-311.pyc deleted file mode 100644 index 422676c..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/msgpack/__pycache__/fallback.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/msgpack/exceptions.py b/venv/lib/python3.11/site-packages/pip/_vendor/msgpack/exceptions.py deleted file mode 100644 index d6d2615..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/msgpack/exceptions.py +++ /dev/null @@ -1,48 +0,0 @@ -class UnpackException(Exception): - """Base class for some exceptions raised while unpacking. - - NOTE: unpack may raise exception other than subclass of - UnpackException. If you want to catch all error, catch - Exception instead. - """ - - -class BufferFull(UnpackException): - pass - - -class OutOfData(UnpackException): - pass - - -class FormatError(ValueError, UnpackException): - """Invalid msgpack format""" - - -class StackError(ValueError, UnpackException): - """Too nested""" - - -# Deprecated. Use ValueError instead -UnpackValueError = ValueError - - -class ExtraData(UnpackValueError): - """ExtraData is raised when there is trailing data. - - This exception is raised while only one-shot (not streaming) - unpack. - """ - - def __init__(self, unpacked, extra): - self.unpacked = unpacked - self.extra = extra - - def __str__(self): - return "unpack(b) received extra data." - - -# Deprecated. Use Exception instead to catch all exception during packing. -PackException = Exception -PackValueError = ValueError -PackOverflowError = OverflowError diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/msgpack/ext.py b/venv/lib/python3.11/site-packages/pip/_vendor/msgpack/ext.py deleted file mode 100644 index 23e0d6b..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/msgpack/ext.py +++ /dev/null @@ -1,193 +0,0 @@ -# coding: utf-8 -from collections import namedtuple -import datetime -import sys -import struct - - -PY2 = sys.version_info[0] == 2 - -if PY2: - int_types = (int, long) - _utc = None -else: - int_types = int - try: - _utc = datetime.timezone.utc - except AttributeError: - _utc = datetime.timezone(datetime.timedelta(0)) - - -class ExtType(namedtuple("ExtType", "code data")): - """ExtType represents ext type in msgpack.""" - - def __new__(cls, code, data): - if not isinstance(code, int): - raise TypeError("code must be int") - if not isinstance(data, bytes): - raise TypeError("data must be bytes") - if not 0 <= code <= 127: - raise ValueError("code must be 0~127") - return super(ExtType, cls).__new__(cls, code, data) - - -class Timestamp(object): - """Timestamp represents the Timestamp extension type in msgpack. - - When built with Cython, msgpack uses C methods to pack and unpack `Timestamp`. When using pure-Python - msgpack, :func:`to_bytes` and :func:`from_bytes` are used to pack and unpack `Timestamp`. - - This class is immutable: Do not override seconds and nanoseconds. - """ - - __slots__ = ["seconds", "nanoseconds"] - - def __init__(self, seconds, nanoseconds=0): - """Initialize a Timestamp object. - - :param int seconds: - Number of seconds since the UNIX epoch (00:00:00 UTC Jan 1 1970, minus leap seconds). - May be negative. - - :param int nanoseconds: - Number of nanoseconds to add to `seconds` to get fractional time. - Maximum is 999_999_999. Default is 0. - - Note: Negative times (before the UNIX epoch) are represented as negative seconds + positive ns. - """ - if not isinstance(seconds, int_types): - raise TypeError("seconds must be an integer") - if not isinstance(nanoseconds, int_types): - raise TypeError("nanoseconds must be an integer") - if not (0 <= nanoseconds < 10**9): - raise ValueError( - "nanoseconds must be a non-negative integer less than 999999999." - ) - self.seconds = seconds - self.nanoseconds = nanoseconds - - def __repr__(self): - """String representation of Timestamp.""" - return "Timestamp(seconds={0}, nanoseconds={1})".format( - self.seconds, self.nanoseconds - ) - - def __eq__(self, other): - """Check for equality with another Timestamp object""" - if type(other) is self.__class__: - return ( - self.seconds == other.seconds and self.nanoseconds == other.nanoseconds - ) - return False - - def __ne__(self, other): - """not-equals method (see :func:`__eq__()`)""" - return not self.__eq__(other) - - def __hash__(self): - return hash((self.seconds, self.nanoseconds)) - - @staticmethod - def from_bytes(b): - """Unpack bytes into a `Timestamp` object. - - Used for pure-Python msgpack unpacking. - - :param b: Payload from msgpack ext message with code -1 - :type b: bytes - - :returns: Timestamp object unpacked from msgpack ext payload - :rtype: Timestamp - """ - if len(b) == 4: - seconds = struct.unpack("!L", b)[0] - nanoseconds = 0 - elif len(b) == 8: - data64 = struct.unpack("!Q", b)[0] - seconds = data64 & 0x00000003FFFFFFFF - nanoseconds = data64 >> 34 - elif len(b) == 12: - nanoseconds, seconds = struct.unpack("!Iq", b) - else: - raise ValueError( - "Timestamp type can only be created from 32, 64, or 96-bit byte objects" - ) - return Timestamp(seconds, nanoseconds) - - def to_bytes(self): - """Pack this Timestamp object into bytes. - - Used for pure-Python msgpack packing. - - :returns data: Payload for EXT message with code -1 (timestamp type) - :rtype: bytes - """ - if (self.seconds >> 34) == 0: # seconds is non-negative and fits in 34 bits - data64 = self.nanoseconds << 34 | self.seconds - if data64 & 0xFFFFFFFF00000000 == 0: - # nanoseconds is zero and seconds < 2**32, so timestamp 32 - data = struct.pack("!L", data64) - else: - # timestamp 64 - data = struct.pack("!Q", data64) - else: - # timestamp 96 - data = struct.pack("!Iq", self.nanoseconds, self.seconds) - return data - - @staticmethod - def from_unix(unix_sec): - """Create a Timestamp from posix timestamp in seconds. - - :param unix_float: Posix timestamp in seconds. - :type unix_float: int or float. - """ - seconds = int(unix_sec // 1) - nanoseconds = int((unix_sec % 1) * 10**9) - return Timestamp(seconds, nanoseconds) - - def to_unix(self): - """Get the timestamp as a floating-point value. - - :returns: posix timestamp - :rtype: float - """ - return self.seconds + self.nanoseconds / 1e9 - - @staticmethod - def from_unix_nano(unix_ns): - """Create a Timestamp from posix timestamp in nanoseconds. - - :param int unix_ns: Posix timestamp in nanoseconds. - :rtype: Timestamp - """ - return Timestamp(*divmod(unix_ns, 10**9)) - - def to_unix_nano(self): - """Get the timestamp as a unixtime in nanoseconds. - - :returns: posix timestamp in nanoseconds - :rtype: int - """ - return self.seconds * 10**9 + self.nanoseconds - - def to_datetime(self): - """Get the timestamp as a UTC datetime. - - Python 2 is not supported. - - :rtype: datetime. - """ - return datetime.datetime.fromtimestamp(0, _utc) + datetime.timedelta( - seconds=self.to_unix() - ) - - @staticmethod - def from_datetime(dt): - """Create a Timestamp from datetime with tzinfo. - - Python 2 is not supported. - - :rtype: Timestamp - """ - return Timestamp.from_unix(dt.timestamp()) diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/msgpack/fallback.py b/venv/lib/python3.11/site-packages/pip/_vendor/msgpack/fallback.py deleted file mode 100644 index e8cebc1..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/msgpack/fallback.py +++ /dev/null @@ -1,1010 +0,0 @@ -"""Fallback pure Python implementation of msgpack""" -from datetime import datetime as _DateTime -import sys -import struct - - -PY2 = sys.version_info[0] == 2 -if PY2: - int_types = (int, long) - - def dict_iteritems(d): - return d.iteritems() - -else: - int_types = int - unicode = str - xrange = range - - def dict_iteritems(d): - return d.items() - - -if sys.version_info < (3, 5): - # Ugly hack... - RecursionError = RuntimeError - - def _is_recursionerror(e): - return ( - len(e.args) == 1 - and isinstance(e.args[0], str) - and e.args[0].startswith("maximum recursion depth exceeded") - ) - -else: - - def _is_recursionerror(e): - return True - - -if hasattr(sys, "pypy_version_info"): - # StringIO is slow on PyPy, StringIO is faster. However: PyPy's own - # StringBuilder is fastest. - from __pypy__ import newlist_hint - - try: - from __pypy__.builders import BytesBuilder as StringBuilder - except ImportError: - from __pypy__.builders import StringBuilder - USING_STRINGBUILDER = True - - class StringIO(object): - def __init__(self, s=b""): - if s: - self.builder = StringBuilder(len(s)) - self.builder.append(s) - else: - self.builder = StringBuilder() - - def write(self, s): - if isinstance(s, memoryview): - s = s.tobytes() - elif isinstance(s, bytearray): - s = bytes(s) - self.builder.append(s) - - def getvalue(self): - return self.builder.build() - -else: - USING_STRINGBUILDER = False - from io import BytesIO as StringIO - - newlist_hint = lambda size: [] - - -from .exceptions import BufferFull, OutOfData, ExtraData, FormatError, StackError - -from .ext import ExtType, Timestamp - - -EX_SKIP = 0 -EX_CONSTRUCT = 1 -EX_READ_ARRAY_HEADER = 2 -EX_READ_MAP_HEADER = 3 - -TYPE_IMMEDIATE = 0 -TYPE_ARRAY = 1 -TYPE_MAP = 2 -TYPE_RAW = 3 -TYPE_BIN = 4 -TYPE_EXT = 5 - -DEFAULT_RECURSE_LIMIT = 511 - - -def _check_type_strict(obj, t, type=type, tuple=tuple): - if type(t) is tuple: - return type(obj) in t - else: - return type(obj) is t - - -def _get_data_from_buffer(obj): - view = memoryview(obj) - if view.itemsize != 1: - raise ValueError("cannot unpack from multi-byte object") - return view - - -def unpackb(packed, **kwargs): - """ - Unpack an object from `packed`. - - Raises ``ExtraData`` when *packed* contains extra bytes. - Raises ``ValueError`` when *packed* is incomplete. - Raises ``FormatError`` when *packed* is not valid msgpack. - Raises ``StackError`` when *packed* contains too nested. - Other exceptions can be raised during unpacking. - - See :class:`Unpacker` for options. - """ - unpacker = Unpacker(None, max_buffer_size=len(packed), **kwargs) - unpacker.feed(packed) - try: - ret = unpacker._unpack() - except OutOfData: - raise ValueError("Unpack failed: incomplete input") - except RecursionError as e: - if _is_recursionerror(e): - raise StackError - raise - if unpacker._got_extradata(): - raise ExtraData(ret, unpacker._get_extradata()) - return ret - - -if sys.version_info < (2, 7, 6): - - def _unpack_from(f, b, o=0): - """Explicit type cast for legacy struct.unpack_from""" - return struct.unpack_from(f, bytes(b), o) - -else: - _unpack_from = struct.unpack_from - -_NO_FORMAT_USED = "" -_MSGPACK_HEADERS = { - 0xC4: (1, _NO_FORMAT_USED, TYPE_BIN), - 0xC5: (2, ">H", TYPE_BIN), - 0xC6: (4, ">I", TYPE_BIN), - 0xC7: (2, "Bb", TYPE_EXT), - 0xC8: (3, ">Hb", TYPE_EXT), - 0xC9: (5, ">Ib", TYPE_EXT), - 0xCA: (4, ">f"), - 0xCB: (8, ">d"), - 0xCC: (1, _NO_FORMAT_USED), - 0xCD: (2, ">H"), - 0xCE: (4, ">I"), - 0xCF: (8, ">Q"), - 0xD0: (1, "b"), - 0xD1: (2, ">h"), - 0xD2: (4, ">i"), - 0xD3: (8, ">q"), - 0xD4: (1, "b1s", TYPE_EXT), - 0xD5: (2, "b2s", TYPE_EXT), - 0xD6: (4, "b4s", TYPE_EXT), - 0xD7: (8, "b8s", TYPE_EXT), - 0xD8: (16, "b16s", TYPE_EXT), - 0xD9: (1, _NO_FORMAT_USED, TYPE_RAW), - 0xDA: (2, ">H", TYPE_RAW), - 0xDB: (4, ">I", TYPE_RAW), - 0xDC: (2, ">H", TYPE_ARRAY), - 0xDD: (4, ">I", TYPE_ARRAY), - 0xDE: (2, ">H", TYPE_MAP), - 0xDF: (4, ">I", TYPE_MAP), -} - - -class Unpacker(object): - """Streaming unpacker. - - Arguments: - - :param file_like: - File-like object having `.read(n)` method. - If specified, unpacker reads serialized data from it and :meth:`feed()` is not usable. - - :param int read_size: - Used as `file_like.read(read_size)`. (default: `min(16*1024, max_buffer_size)`) - - :param bool use_list: - If true, unpack msgpack array to Python list. - Otherwise, unpack to Python tuple. (default: True) - - :param bool raw: - If true, unpack msgpack raw to Python bytes. - Otherwise, unpack to Python str by decoding with UTF-8 encoding (default). - - :param int timestamp: - Control how timestamp type is unpacked: - - 0 - Timestamp - 1 - float (Seconds from the EPOCH) - 2 - int (Nanoseconds from the EPOCH) - 3 - datetime.datetime (UTC). Python 2 is not supported. - - :param bool strict_map_key: - If true (default), only str or bytes are accepted for map (dict) keys. - - :param callable object_hook: - When specified, it should be callable. - Unpacker calls it with a dict argument after unpacking msgpack map. - (See also simplejson) - - :param callable object_pairs_hook: - When specified, it should be callable. - Unpacker calls it with a list of key-value pairs after unpacking msgpack map. - (See also simplejson) - - :param str unicode_errors: - The error handler for decoding unicode. (default: 'strict') - This option should be used only when you have msgpack data which - contains invalid UTF-8 string. - - :param int max_buffer_size: - Limits size of data waiting unpacked. 0 means 2**32-1. - The default value is 100*1024*1024 (100MiB). - Raises `BufferFull` exception when it is insufficient. - You should set this parameter when unpacking data from untrusted source. - - :param int max_str_len: - Deprecated, use *max_buffer_size* instead. - Limits max length of str. (default: max_buffer_size) - - :param int max_bin_len: - Deprecated, use *max_buffer_size* instead. - Limits max length of bin. (default: max_buffer_size) - - :param int max_array_len: - Limits max length of array. - (default: max_buffer_size) - - :param int max_map_len: - Limits max length of map. - (default: max_buffer_size//2) - - :param int max_ext_len: - Deprecated, use *max_buffer_size* instead. - Limits max size of ext type. (default: max_buffer_size) - - Example of streaming deserialize from file-like object:: - - unpacker = Unpacker(file_like) - for o in unpacker: - process(o) - - Example of streaming deserialize from socket:: - - unpacker = Unpacker() - while True: - buf = sock.recv(1024**2) - if not buf: - break - unpacker.feed(buf) - for o in unpacker: - process(o) - - Raises ``ExtraData`` when *packed* contains extra bytes. - Raises ``OutOfData`` when *packed* is incomplete. - Raises ``FormatError`` when *packed* is not valid msgpack. - Raises ``StackError`` when *packed* contains too nested. - Other exceptions can be raised during unpacking. - """ - - def __init__( - self, - file_like=None, - read_size=0, - use_list=True, - raw=False, - timestamp=0, - strict_map_key=True, - object_hook=None, - object_pairs_hook=None, - list_hook=None, - unicode_errors=None, - max_buffer_size=100 * 1024 * 1024, - ext_hook=ExtType, - max_str_len=-1, - max_bin_len=-1, - max_array_len=-1, - max_map_len=-1, - max_ext_len=-1, - ): - if unicode_errors is None: - unicode_errors = "strict" - - if file_like is None: - self._feeding = True - else: - if not callable(file_like.read): - raise TypeError("`file_like.read` must be callable") - self.file_like = file_like - self._feeding = False - - #: array of bytes fed. - self._buffer = bytearray() - #: Which position we currently reads - self._buff_i = 0 - - # When Unpacker is used as an iterable, between the calls to next(), - # the buffer is not "consumed" completely, for efficiency sake. - # Instead, it is done sloppily. To make sure we raise BufferFull at - # the correct moments, we have to keep track of how sloppy we were. - # Furthermore, when the buffer is incomplete (that is: in the case - # we raise an OutOfData) we need to rollback the buffer to the correct - # state, which _buf_checkpoint records. - self._buf_checkpoint = 0 - - if not max_buffer_size: - max_buffer_size = 2**31 - 1 - if max_str_len == -1: - max_str_len = max_buffer_size - if max_bin_len == -1: - max_bin_len = max_buffer_size - if max_array_len == -1: - max_array_len = max_buffer_size - if max_map_len == -1: - max_map_len = max_buffer_size // 2 - if max_ext_len == -1: - max_ext_len = max_buffer_size - - self._max_buffer_size = max_buffer_size - if read_size > self._max_buffer_size: - raise ValueError("read_size must be smaller than max_buffer_size") - self._read_size = read_size or min(self._max_buffer_size, 16 * 1024) - self._raw = bool(raw) - self._strict_map_key = bool(strict_map_key) - self._unicode_errors = unicode_errors - self._use_list = use_list - if not (0 <= timestamp <= 3): - raise ValueError("timestamp must be 0..3") - self._timestamp = timestamp - self._list_hook = list_hook - self._object_hook = object_hook - self._object_pairs_hook = object_pairs_hook - self._ext_hook = ext_hook - self._max_str_len = max_str_len - self._max_bin_len = max_bin_len - self._max_array_len = max_array_len - self._max_map_len = max_map_len - self._max_ext_len = max_ext_len - self._stream_offset = 0 - - if list_hook is not None and not callable(list_hook): - raise TypeError("`list_hook` is not callable") - if object_hook is not None and not callable(object_hook): - raise TypeError("`object_hook` is not callable") - if object_pairs_hook is not None and not callable(object_pairs_hook): - raise TypeError("`object_pairs_hook` is not callable") - if object_hook is not None and object_pairs_hook is not None: - raise TypeError( - "object_pairs_hook and object_hook are mutually " "exclusive" - ) - if not callable(ext_hook): - raise TypeError("`ext_hook` is not callable") - - def feed(self, next_bytes): - assert self._feeding - view = _get_data_from_buffer(next_bytes) - if len(self._buffer) - self._buff_i + len(view) > self._max_buffer_size: - raise BufferFull - - # Strip buffer before checkpoint before reading file. - if self._buf_checkpoint > 0: - del self._buffer[: self._buf_checkpoint] - self._buff_i -= self._buf_checkpoint - self._buf_checkpoint = 0 - - # Use extend here: INPLACE_ADD += doesn't reliably typecast memoryview in jython - self._buffer.extend(view) - - def _consume(self): - """Gets rid of the used parts of the buffer.""" - self._stream_offset += self._buff_i - self._buf_checkpoint - self._buf_checkpoint = self._buff_i - - def _got_extradata(self): - return self._buff_i < len(self._buffer) - - def _get_extradata(self): - return self._buffer[self._buff_i :] - - def read_bytes(self, n): - ret = self._read(n, raise_outofdata=False) - self._consume() - return ret - - def _read(self, n, raise_outofdata=True): - # (int) -> bytearray - self._reserve(n, raise_outofdata=raise_outofdata) - i = self._buff_i - ret = self._buffer[i : i + n] - self._buff_i = i + len(ret) - return ret - - def _reserve(self, n, raise_outofdata=True): - remain_bytes = len(self._buffer) - self._buff_i - n - - # Fast path: buffer has n bytes already - if remain_bytes >= 0: - return - - if self._feeding: - self._buff_i = self._buf_checkpoint - raise OutOfData - - # Strip buffer before checkpoint before reading file. - if self._buf_checkpoint > 0: - del self._buffer[: self._buf_checkpoint] - self._buff_i -= self._buf_checkpoint - self._buf_checkpoint = 0 - - # Read from file - remain_bytes = -remain_bytes - if remain_bytes + len(self._buffer) > self._max_buffer_size: - raise BufferFull - while remain_bytes > 0: - to_read_bytes = max(self._read_size, remain_bytes) - read_data = self.file_like.read(to_read_bytes) - if not read_data: - break - assert isinstance(read_data, bytes) - self._buffer += read_data - remain_bytes -= len(read_data) - - if len(self._buffer) < n + self._buff_i and raise_outofdata: - self._buff_i = 0 # rollback - raise OutOfData - - def _read_header(self): - typ = TYPE_IMMEDIATE - n = 0 - obj = None - self._reserve(1) - b = self._buffer[self._buff_i] - self._buff_i += 1 - if b & 0b10000000 == 0: - obj = b - elif b & 0b11100000 == 0b11100000: - obj = -1 - (b ^ 0xFF) - elif b & 0b11100000 == 0b10100000: - n = b & 0b00011111 - typ = TYPE_RAW - if n > self._max_str_len: - raise ValueError("%s exceeds max_str_len(%s)" % (n, self._max_str_len)) - obj = self._read(n) - elif b & 0b11110000 == 0b10010000: - n = b & 0b00001111 - typ = TYPE_ARRAY - if n > self._max_array_len: - raise ValueError( - "%s exceeds max_array_len(%s)" % (n, self._max_array_len) - ) - elif b & 0b11110000 == 0b10000000: - n = b & 0b00001111 - typ = TYPE_MAP - if n > self._max_map_len: - raise ValueError("%s exceeds max_map_len(%s)" % (n, self._max_map_len)) - elif b == 0xC0: - obj = None - elif b == 0xC2: - obj = False - elif b == 0xC3: - obj = True - elif 0xC4 <= b <= 0xC6: - size, fmt, typ = _MSGPACK_HEADERS[b] - self._reserve(size) - if len(fmt) > 0: - n = _unpack_from(fmt, self._buffer, self._buff_i)[0] - else: - n = self._buffer[self._buff_i] - self._buff_i += size - if n > self._max_bin_len: - raise ValueError("%s exceeds max_bin_len(%s)" % (n, self._max_bin_len)) - obj = self._read(n) - elif 0xC7 <= b <= 0xC9: - size, fmt, typ = _MSGPACK_HEADERS[b] - self._reserve(size) - L, n = _unpack_from(fmt, self._buffer, self._buff_i) - self._buff_i += size - if L > self._max_ext_len: - raise ValueError("%s exceeds max_ext_len(%s)" % (L, self._max_ext_len)) - obj = self._read(L) - elif 0xCA <= b <= 0xD3: - size, fmt = _MSGPACK_HEADERS[b] - self._reserve(size) - if len(fmt) > 0: - obj = _unpack_from(fmt, self._buffer, self._buff_i)[0] - else: - obj = self._buffer[self._buff_i] - self._buff_i += size - elif 0xD4 <= b <= 0xD8: - size, fmt, typ = _MSGPACK_HEADERS[b] - if self._max_ext_len < size: - raise ValueError( - "%s exceeds max_ext_len(%s)" % (size, self._max_ext_len) - ) - self._reserve(size + 1) - n, obj = _unpack_from(fmt, self._buffer, self._buff_i) - self._buff_i += size + 1 - elif 0xD9 <= b <= 0xDB: - size, fmt, typ = _MSGPACK_HEADERS[b] - self._reserve(size) - if len(fmt) > 0: - (n,) = _unpack_from(fmt, self._buffer, self._buff_i) - else: - n = self._buffer[self._buff_i] - self._buff_i += size - if n > self._max_str_len: - raise ValueError("%s exceeds max_str_len(%s)" % (n, self._max_str_len)) - obj = self._read(n) - elif 0xDC <= b <= 0xDD: - size, fmt, typ = _MSGPACK_HEADERS[b] - self._reserve(size) - (n,) = _unpack_from(fmt, self._buffer, self._buff_i) - self._buff_i += size - if n > self._max_array_len: - raise ValueError( - "%s exceeds max_array_len(%s)" % (n, self._max_array_len) - ) - elif 0xDE <= b <= 0xDF: - size, fmt, typ = _MSGPACK_HEADERS[b] - self._reserve(size) - (n,) = _unpack_from(fmt, self._buffer, self._buff_i) - self._buff_i += size - if n > self._max_map_len: - raise ValueError("%s exceeds max_map_len(%s)" % (n, self._max_map_len)) - else: - raise FormatError("Unknown header: 0x%x" % b) - return typ, n, obj - - def _unpack(self, execute=EX_CONSTRUCT): - typ, n, obj = self._read_header() - - if execute == EX_READ_ARRAY_HEADER: - if typ != TYPE_ARRAY: - raise ValueError("Expected array") - return n - if execute == EX_READ_MAP_HEADER: - if typ != TYPE_MAP: - raise ValueError("Expected map") - return n - # TODO should we eliminate the recursion? - if typ == TYPE_ARRAY: - if execute == EX_SKIP: - for i in xrange(n): - # TODO check whether we need to call `list_hook` - self._unpack(EX_SKIP) - return - ret = newlist_hint(n) - for i in xrange(n): - ret.append(self._unpack(EX_CONSTRUCT)) - if self._list_hook is not None: - ret = self._list_hook(ret) - # TODO is the interaction between `list_hook` and `use_list` ok? - return ret if self._use_list else tuple(ret) - if typ == TYPE_MAP: - if execute == EX_SKIP: - for i in xrange(n): - # TODO check whether we need to call hooks - self._unpack(EX_SKIP) - self._unpack(EX_SKIP) - return - if self._object_pairs_hook is not None: - ret = self._object_pairs_hook( - (self._unpack(EX_CONSTRUCT), self._unpack(EX_CONSTRUCT)) - for _ in xrange(n) - ) - else: - ret = {} - for _ in xrange(n): - key = self._unpack(EX_CONSTRUCT) - if self._strict_map_key and type(key) not in (unicode, bytes): - raise ValueError( - "%s is not allowed for map key" % str(type(key)) - ) - if not PY2 and type(key) is str: - key = sys.intern(key) - ret[key] = self._unpack(EX_CONSTRUCT) - if self._object_hook is not None: - ret = self._object_hook(ret) - return ret - if execute == EX_SKIP: - return - if typ == TYPE_RAW: - if self._raw: - obj = bytes(obj) - else: - obj = obj.decode("utf_8", self._unicode_errors) - return obj - if typ == TYPE_BIN: - return bytes(obj) - if typ == TYPE_EXT: - if n == -1: # timestamp - ts = Timestamp.from_bytes(bytes(obj)) - if self._timestamp == 1: - return ts.to_unix() - elif self._timestamp == 2: - return ts.to_unix_nano() - elif self._timestamp == 3: - return ts.to_datetime() - else: - return ts - else: - return self._ext_hook(n, bytes(obj)) - assert typ == TYPE_IMMEDIATE - return obj - - def __iter__(self): - return self - - def __next__(self): - try: - ret = self._unpack(EX_CONSTRUCT) - self._consume() - return ret - except OutOfData: - self._consume() - raise StopIteration - except RecursionError: - raise StackError - - next = __next__ - - def skip(self): - self._unpack(EX_SKIP) - self._consume() - - def unpack(self): - try: - ret = self._unpack(EX_CONSTRUCT) - except RecursionError: - raise StackError - self._consume() - return ret - - def read_array_header(self): - ret = self._unpack(EX_READ_ARRAY_HEADER) - self._consume() - return ret - - def read_map_header(self): - ret = self._unpack(EX_READ_MAP_HEADER) - self._consume() - return ret - - def tell(self): - return self._stream_offset - - -class Packer(object): - """ - MessagePack Packer - - Usage:: - - packer = Packer() - astream.write(packer.pack(a)) - astream.write(packer.pack(b)) - - Packer's constructor has some keyword arguments: - - :param callable default: - Convert user type to builtin type that Packer supports. - See also simplejson's document. - - :param bool use_single_float: - Use single precision float type for float. (default: False) - - :param bool autoreset: - Reset buffer after each pack and return its content as `bytes`. (default: True). - If set this to false, use `bytes()` to get content and `.reset()` to clear buffer. - - :param bool use_bin_type: - Use bin type introduced in msgpack spec 2.0 for bytes. - It also enables str8 type for unicode. (default: True) - - :param bool strict_types: - If set to true, types will be checked to be exact. Derived classes - from serializable types will not be serialized and will be - treated as unsupported type and forwarded to default. - Additionally tuples will not be serialized as lists. - This is useful when trying to implement accurate serialization - for python types. - - :param bool datetime: - If set to true, datetime with tzinfo is packed into Timestamp type. - Note that the tzinfo is stripped in the timestamp. - You can get UTC datetime with `timestamp=3` option of the Unpacker. - (Python 2 is not supported). - - :param str unicode_errors: - The error handler for encoding unicode. (default: 'strict') - DO NOT USE THIS!! This option is kept for very specific usage. - - Example of streaming deserialize from file-like object:: - - unpacker = Unpacker(file_like) - for o in unpacker: - process(o) - - Example of streaming deserialize from socket:: - - unpacker = Unpacker() - while True: - buf = sock.recv(1024**2) - if not buf: - break - unpacker.feed(buf) - for o in unpacker: - process(o) - - Raises ``ExtraData`` when *packed* contains extra bytes. - Raises ``OutOfData`` when *packed* is incomplete. - Raises ``FormatError`` when *packed* is not valid msgpack. - Raises ``StackError`` when *packed* contains too nested. - Other exceptions can be raised during unpacking. - """ - - def __init__( - self, - default=None, - use_single_float=False, - autoreset=True, - use_bin_type=True, - strict_types=False, - datetime=False, - unicode_errors=None, - ): - self._strict_types = strict_types - self._use_float = use_single_float - self._autoreset = autoreset - self._use_bin_type = use_bin_type - self._buffer = StringIO() - if PY2 and datetime: - raise ValueError("datetime is not supported in Python 2") - self._datetime = bool(datetime) - self._unicode_errors = unicode_errors or "strict" - if default is not None: - if not callable(default): - raise TypeError("default must be callable") - self._default = default - - def _pack( - self, - obj, - nest_limit=DEFAULT_RECURSE_LIMIT, - check=isinstance, - check_type_strict=_check_type_strict, - ): - default_used = False - if self._strict_types: - check = check_type_strict - list_types = list - else: - list_types = (list, tuple) - while True: - if nest_limit < 0: - raise ValueError("recursion limit exceeded") - if obj is None: - return self._buffer.write(b"\xc0") - if check(obj, bool): - if obj: - return self._buffer.write(b"\xc3") - return self._buffer.write(b"\xc2") - if check(obj, int_types): - if 0 <= obj < 0x80: - return self._buffer.write(struct.pack("B", obj)) - if -0x20 <= obj < 0: - return self._buffer.write(struct.pack("b", obj)) - if 0x80 <= obj <= 0xFF: - return self._buffer.write(struct.pack("BB", 0xCC, obj)) - if -0x80 <= obj < 0: - return self._buffer.write(struct.pack(">Bb", 0xD0, obj)) - if 0xFF < obj <= 0xFFFF: - return self._buffer.write(struct.pack(">BH", 0xCD, obj)) - if -0x8000 <= obj < -0x80: - return self._buffer.write(struct.pack(">Bh", 0xD1, obj)) - if 0xFFFF < obj <= 0xFFFFFFFF: - return self._buffer.write(struct.pack(">BI", 0xCE, obj)) - if -0x80000000 <= obj < -0x8000: - return self._buffer.write(struct.pack(">Bi", 0xD2, obj)) - if 0xFFFFFFFF < obj <= 0xFFFFFFFFFFFFFFFF: - return self._buffer.write(struct.pack(">BQ", 0xCF, obj)) - if -0x8000000000000000 <= obj < -0x80000000: - return self._buffer.write(struct.pack(">Bq", 0xD3, obj)) - if not default_used and self._default is not None: - obj = self._default(obj) - default_used = True - continue - raise OverflowError("Integer value out of range") - if check(obj, (bytes, bytearray)): - n = len(obj) - if n >= 2**32: - raise ValueError("%s is too large" % type(obj).__name__) - self._pack_bin_header(n) - return self._buffer.write(obj) - if check(obj, unicode): - obj = obj.encode("utf-8", self._unicode_errors) - n = len(obj) - if n >= 2**32: - raise ValueError("String is too large") - self._pack_raw_header(n) - return self._buffer.write(obj) - if check(obj, memoryview): - n = obj.nbytes - if n >= 2**32: - raise ValueError("Memoryview is too large") - self._pack_bin_header(n) - return self._buffer.write(obj) - if check(obj, float): - if self._use_float: - return self._buffer.write(struct.pack(">Bf", 0xCA, obj)) - return self._buffer.write(struct.pack(">Bd", 0xCB, obj)) - if check(obj, (ExtType, Timestamp)): - if check(obj, Timestamp): - code = -1 - data = obj.to_bytes() - else: - code = obj.code - data = obj.data - assert isinstance(code, int) - assert isinstance(data, bytes) - L = len(data) - if L == 1: - self._buffer.write(b"\xd4") - elif L == 2: - self._buffer.write(b"\xd5") - elif L == 4: - self._buffer.write(b"\xd6") - elif L == 8: - self._buffer.write(b"\xd7") - elif L == 16: - self._buffer.write(b"\xd8") - elif L <= 0xFF: - self._buffer.write(struct.pack(">BB", 0xC7, L)) - elif L <= 0xFFFF: - self._buffer.write(struct.pack(">BH", 0xC8, L)) - else: - self._buffer.write(struct.pack(">BI", 0xC9, L)) - self._buffer.write(struct.pack("b", code)) - self._buffer.write(data) - return - if check(obj, list_types): - n = len(obj) - self._pack_array_header(n) - for i in xrange(n): - self._pack(obj[i], nest_limit - 1) - return - if check(obj, dict): - return self._pack_map_pairs( - len(obj), dict_iteritems(obj), nest_limit - 1 - ) - - if self._datetime and check(obj, _DateTime) and obj.tzinfo is not None: - obj = Timestamp.from_datetime(obj) - default_used = 1 - continue - - if not default_used and self._default is not None: - obj = self._default(obj) - default_used = 1 - continue - - if self._datetime and check(obj, _DateTime): - raise ValueError("Cannot serialize %r where tzinfo=None" % (obj,)) - - raise TypeError("Cannot serialize %r" % (obj,)) - - def pack(self, obj): - try: - self._pack(obj) - except: - self._buffer = StringIO() # force reset - raise - if self._autoreset: - ret = self._buffer.getvalue() - self._buffer = StringIO() - return ret - - def pack_map_pairs(self, pairs): - self._pack_map_pairs(len(pairs), pairs) - if self._autoreset: - ret = self._buffer.getvalue() - self._buffer = StringIO() - return ret - - def pack_array_header(self, n): - if n >= 2**32: - raise ValueError - self._pack_array_header(n) - if self._autoreset: - ret = self._buffer.getvalue() - self._buffer = StringIO() - return ret - - def pack_map_header(self, n): - if n >= 2**32: - raise ValueError - self._pack_map_header(n) - if self._autoreset: - ret = self._buffer.getvalue() - self._buffer = StringIO() - return ret - - def pack_ext_type(self, typecode, data): - if not isinstance(typecode, int): - raise TypeError("typecode must have int type.") - if not 0 <= typecode <= 127: - raise ValueError("typecode should be 0-127") - if not isinstance(data, bytes): - raise TypeError("data must have bytes type") - L = len(data) - if L > 0xFFFFFFFF: - raise ValueError("Too large data") - if L == 1: - self._buffer.write(b"\xd4") - elif L == 2: - self._buffer.write(b"\xd5") - elif L == 4: - self._buffer.write(b"\xd6") - elif L == 8: - self._buffer.write(b"\xd7") - elif L == 16: - self._buffer.write(b"\xd8") - elif L <= 0xFF: - self._buffer.write(b"\xc7" + struct.pack("B", L)) - elif L <= 0xFFFF: - self._buffer.write(b"\xc8" + struct.pack(">H", L)) - else: - self._buffer.write(b"\xc9" + struct.pack(">I", L)) - self._buffer.write(struct.pack("B", typecode)) - self._buffer.write(data) - - def _pack_array_header(self, n): - if n <= 0x0F: - return self._buffer.write(struct.pack("B", 0x90 + n)) - if n <= 0xFFFF: - return self._buffer.write(struct.pack(">BH", 0xDC, n)) - if n <= 0xFFFFFFFF: - return self._buffer.write(struct.pack(">BI", 0xDD, n)) - raise ValueError("Array is too large") - - def _pack_map_header(self, n): - if n <= 0x0F: - return self._buffer.write(struct.pack("B", 0x80 + n)) - if n <= 0xFFFF: - return self._buffer.write(struct.pack(">BH", 0xDE, n)) - if n <= 0xFFFFFFFF: - return self._buffer.write(struct.pack(">BI", 0xDF, n)) - raise ValueError("Dict is too large") - - def _pack_map_pairs(self, n, pairs, nest_limit=DEFAULT_RECURSE_LIMIT): - self._pack_map_header(n) - for (k, v) in pairs: - self._pack(k, nest_limit - 1) - self._pack(v, nest_limit - 1) - - def _pack_raw_header(self, n): - if n <= 0x1F: - self._buffer.write(struct.pack("B", 0xA0 + n)) - elif self._use_bin_type and n <= 0xFF: - self._buffer.write(struct.pack(">BB", 0xD9, n)) - elif n <= 0xFFFF: - self._buffer.write(struct.pack(">BH", 0xDA, n)) - elif n <= 0xFFFFFFFF: - self._buffer.write(struct.pack(">BI", 0xDB, n)) - else: - raise ValueError("Raw is too large") - - def _pack_bin_header(self, n): - if not self._use_bin_type: - return self._pack_raw_header(n) - elif n <= 0xFF: - return self._buffer.write(struct.pack(">BB", 0xC4, n)) - elif n <= 0xFFFF: - return self._buffer.write(struct.pack(">BH", 0xC5, n)) - elif n <= 0xFFFFFFFF: - return self._buffer.write(struct.pack(">BI", 0xC6, n)) - else: - raise ValueError("Bin is too large") - - def bytes(self): - """Return internal buffer contents as bytes object""" - return self._buffer.getvalue() - - def reset(self): - """Reset internal buffer. - - This method is useful only when autoreset=False. - """ - self._buffer = StringIO() - - def getbuffer(self): - """Return view of internal buffer.""" - if USING_STRINGBUILDER or PY2: - return memoryview(self.bytes()) - else: - return self._buffer.getbuffer() diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/packaging/__about__.py b/venv/lib/python3.11/site-packages/pip/_vendor/packaging/__about__.py deleted file mode 100644 index 3551bc2..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/packaging/__about__.py +++ /dev/null @@ -1,26 +0,0 @@ -# This file is dual licensed under the terms of the Apache License, Version -# 2.0, and the BSD License. See the LICENSE file in the root of this repository -# for complete details. - -__all__ = [ - "__title__", - "__summary__", - "__uri__", - "__version__", - "__author__", - "__email__", - "__license__", - "__copyright__", -] - -__title__ = "packaging" -__summary__ = "Core utilities for Python packages" -__uri__ = "https://github.com/pypa/packaging" - -__version__ = "21.3" - -__author__ = "Donald Stufft and individual contributors" -__email__ = "donald@stufft.io" - -__license__ = "BSD-2-Clause or Apache-2.0" -__copyright__ = "2014-2019 %s" % __author__ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/packaging/__init__.py b/venv/lib/python3.11/site-packages/pip/_vendor/packaging/__init__.py deleted file mode 100644 index 3c50c5d..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/packaging/__init__.py +++ /dev/null @@ -1,25 +0,0 @@ -# This file is dual licensed under the terms of the Apache License, Version -# 2.0, and the BSD License. See the LICENSE file in the root of this repository -# for complete details. - -from .__about__ import ( - __author__, - __copyright__, - __email__, - __license__, - __summary__, - __title__, - __uri__, - __version__, -) - -__all__ = [ - "__title__", - "__summary__", - "__uri__", - "__version__", - "__author__", - "__email__", - "__license__", - "__copyright__", -] diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/packaging/__pycache__/__about__.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/packaging/__pycache__/__about__.cpython-311.pyc deleted file mode 100644 index c7e3c95..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/packaging/__pycache__/__about__.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/packaging/__pycache__/__init__.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/packaging/__pycache__/__init__.cpython-311.pyc deleted file mode 100644 index ff04ff6..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/packaging/__pycache__/__init__.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/packaging/__pycache__/_manylinux.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/packaging/__pycache__/_manylinux.cpython-311.pyc deleted file mode 100644 index bc5a280..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/packaging/__pycache__/_manylinux.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/packaging/__pycache__/_musllinux.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/packaging/__pycache__/_musllinux.cpython-311.pyc deleted file mode 100644 index bdaeca1..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/packaging/__pycache__/_musllinux.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/packaging/__pycache__/_structures.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/packaging/__pycache__/_structures.cpython-311.pyc deleted file mode 100644 index 42c771d..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/packaging/__pycache__/_structures.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/packaging/__pycache__/markers.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/packaging/__pycache__/markers.cpython-311.pyc deleted file mode 100644 index 43de188..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/packaging/__pycache__/markers.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/packaging/__pycache__/requirements.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/packaging/__pycache__/requirements.cpython-311.pyc deleted file mode 100644 index f27157c..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/packaging/__pycache__/requirements.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/packaging/__pycache__/specifiers.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/packaging/__pycache__/specifiers.cpython-311.pyc deleted file mode 100644 index b958e71..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/packaging/__pycache__/specifiers.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/packaging/__pycache__/tags.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/packaging/__pycache__/tags.cpython-311.pyc deleted file mode 100644 index 593f468..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/packaging/__pycache__/tags.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/packaging/__pycache__/utils.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/packaging/__pycache__/utils.cpython-311.pyc deleted file mode 100644 index 0fc6bf1..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/packaging/__pycache__/utils.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/packaging/__pycache__/version.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/packaging/__pycache__/version.cpython-311.pyc deleted file mode 100644 index 0b39e4f..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/packaging/__pycache__/version.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/packaging/_manylinux.py b/venv/lib/python3.11/site-packages/pip/_vendor/packaging/_manylinux.py deleted file mode 100644 index 4c379aa..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/packaging/_manylinux.py +++ /dev/null @@ -1,301 +0,0 @@ -import collections -import functools -import os -import re -import struct -import sys -import warnings -from typing import IO, Dict, Iterator, NamedTuple, Optional, Tuple - - -# Python does not provide platform information at sufficient granularity to -# identify the architecture of the running executable in some cases, so we -# determine it dynamically by reading the information from the running -# process. This only applies on Linux, which uses the ELF format. -class _ELFFileHeader: - # https://en.wikipedia.org/wiki/Executable_and_Linkable_Format#File_header - class _InvalidELFFileHeader(ValueError): - """ - An invalid ELF file header was found. - """ - - ELF_MAGIC_NUMBER = 0x7F454C46 - ELFCLASS32 = 1 - ELFCLASS64 = 2 - ELFDATA2LSB = 1 - ELFDATA2MSB = 2 - EM_386 = 3 - EM_S390 = 22 - EM_ARM = 40 - EM_X86_64 = 62 - EF_ARM_ABIMASK = 0xFF000000 - EF_ARM_ABI_VER5 = 0x05000000 - EF_ARM_ABI_FLOAT_HARD = 0x00000400 - - def __init__(self, file: IO[bytes]) -> None: - def unpack(fmt: str) -> int: - try: - data = file.read(struct.calcsize(fmt)) - result: Tuple[int, ...] = struct.unpack(fmt, data) - except struct.error: - raise _ELFFileHeader._InvalidELFFileHeader() - return result[0] - - self.e_ident_magic = unpack(">I") - if self.e_ident_magic != self.ELF_MAGIC_NUMBER: - raise _ELFFileHeader._InvalidELFFileHeader() - self.e_ident_class = unpack("B") - if self.e_ident_class not in {self.ELFCLASS32, self.ELFCLASS64}: - raise _ELFFileHeader._InvalidELFFileHeader() - self.e_ident_data = unpack("B") - if self.e_ident_data not in {self.ELFDATA2LSB, self.ELFDATA2MSB}: - raise _ELFFileHeader._InvalidELFFileHeader() - self.e_ident_version = unpack("B") - self.e_ident_osabi = unpack("B") - self.e_ident_abiversion = unpack("B") - self.e_ident_pad = file.read(7) - format_h = "H" - format_i = "I" - format_q = "Q" - format_p = format_i if self.e_ident_class == self.ELFCLASS32 else format_q - self.e_type = unpack(format_h) - self.e_machine = unpack(format_h) - self.e_version = unpack(format_i) - self.e_entry = unpack(format_p) - self.e_phoff = unpack(format_p) - self.e_shoff = unpack(format_p) - self.e_flags = unpack(format_i) - self.e_ehsize = unpack(format_h) - self.e_phentsize = unpack(format_h) - self.e_phnum = unpack(format_h) - self.e_shentsize = unpack(format_h) - self.e_shnum = unpack(format_h) - self.e_shstrndx = unpack(format_h) - - -def _get_elf_header() -> Optional[_ELFFileHeader]: - try: - with open(sys.executable, "rb") as f: - elf_header = _ELFFileHeader(f) - except (OSError, TypeError, _ELFFileHeader._InvalidELFFileHeader): - return None - return elf_header - - -def _is_linux_armhf() -> bool: - # hard-float ABI can be detected from the ELF header of the running - # process - # https://static.docs.arm.com/ihi0044/g/aaelf32.pdf - elf_header = _get_elf_header() - if elf_header is None: - return False - result = elf_header.e_ident_class == elf_header.ELFCLASS32 - result &= elf_header.e_ident_data == elf_header.ELFDATA2LSB - result &= elf_header.e_machine == elf_header.EM_ARM - result &= ( - elf_header.e_flags & elf_header.EF_ARM_ABIMASK - ) == elf_header.EF_ARM_ABI_VER5 - result &= ( - elf_header.e_flags & elf_header.EF_ARM_ABI_FLOAT_HARD - ) == elf_header.EF_ARM_ABI_FLOAT_HARD - return result - - -def _is_linux_i686() -> bool: - elf_header = _get_elf_header() - if elf_header is None: - return False - result = elf_header.e_ident_class == elf_header.ELFCLASS32 - result &= elf_header.e_ident_data == elf_header.ELFDATA2LSB - result &= elf_header.e_machine == elf_header.EM_386 - return result - - -def _have_compatible_abi(arch: str) -> bool: - if arch == "armv7l": - return _is_linux_armhf() - if arch == "i686": - return _is_linux_i686() - return arch in {"x86_64", "aarch64", "ppc64", "ppc64le", "s390x"} - - -# If glibc ever changes its major version, we need to know what the last -# minor version was, so we can build the complete list of all versions. -# For now, guess what the highest minor version might be, assume it will -# be 50 for testing. Once this actually happens, update the dictionary -# with the actual value. -_LAST_GLIBC_MINOR: Dict[int, int] = collections.defaultdict(lambda: 50) - - -class _GLibCVersion(NamedTuple): - major: int - minor: int - - -def _glibc_version_string_confstr() -> Optional[str]: - """ - Primary implementation of glibc_version_string using os.confstr. - """ - # os.confstr is quite a bit faster than ctypes.DLL. It's also less likely - # to be broken or missing. This strategy is used in the standard library - # platform module. - # https://github.com/python/cpython/blob/fcf1d003bf4f0100c/Lib/platform.py#L175-L183 - try: - # os.confstr("CS_GNU_LIBC_VERSION") returns a string like "glibc 2.17". - version_string = os.confstr("CS_GNU_LIBC_VERSION") - assert version_string is not None - _, version = version_string.split() - except (AssertionError, AttributeError, OSError, ValueError): - # os.confstr() or CS_GNU_LIBC_VERSION not available (or a bad value)... - return None - return version - - -def _glibc_version_string_ctypes() -> Optional[str]: - """ - Fallback implementation of glibc_version_string using ctypes. - """ - try: - import ctypes - except ImportError: - return None - - # ctypes.CDLL(None) internally calls dlopen(NULL), and as the dlopen - # manpage says, "If filename is NULL, then the returned handle is for the - # main program". This way we can let the linker do the work to figure out - # which libc our process is actually using. - # - # We must also handle the special case where the executable is not a - # dynamically linked executable. This can occur when using musl libc, - # for example. In this situation, dlopen() will error, leading to an - # OSError. Interestingly, at least in the case of musl, there is no - # errno set on the OSError. The single string argument used to construct - # OSError comes from libc itself and is therefore not portable to - # hard code here. In any case, failure to call dlopen() means we - # can proceed, so we bail on our attempt. - try: - process_namespace = ctypes.CDLL(None) - except OSError: - return None - - try: - gnu_get_libc_version = process_namespace.gnu_get_libc_version - except AttributeError: - # Symbol doesn't exist -> therefore, we are not linked to - # glibc. - return None - - # Call gnu_get_libc_version, which returns a string like "2.5" - gnu_get_libc_version.restype = ctypes.c_char_p - version_str: str = gnu_get_libc_version() - # py2 / py3 compatibility: - if not isinstance(version_str, str): - version_str = version_str.decode("ascii") - - return version_str - - -def _glibc_version_string() -> Optional[str]: - """Returns glibc version string, or None if not using glibc.""" - return _glibc_version_string_confstr() or _glibc_version_string_ctypes() - - -def _parse_glibc_version(version_str: str) -> Tuple[int, int]: - """Parse glibc version. - - We use a regexp instead of str.split because we want to discard any - random junk that might come after the minor version -- this might happen - in patched/forked versions of glibc (e.g. Linaro's version of glibc - uses version strings like "2.20-2014.11"). See gh-3588. - """ - m = re.match(r"(?P[0-9]+)\.(?P[0-9]+)", version_str) - if not m: - warnings.warn( - "Expected glibc version with 2 components major.minor," - " got: %s" % version_str, - RuntimeWarning, - ) - return -1, -1 - return int(m.group("major")), int(m.group("minor")) - - -@functools.lru_cache() -def _get_glibc_version() -> Tuple[int, int]: - version_str = _glibc_version_string() - if version_str is None: - return (-1, -1) - return _parse_glibc_version(version_str) - - -# From PEP 513, PEP 600 -def _is_compatible(name: str, arch: str, version: _GLibCVersion) -> bool: - sys_glibc = _get_glibc_version() - if sys_glibc < version: - return False - # Check for presence of _manylinux module. - try: - import _manylinux # noqa - except ImportError: - return True - if hasattr(_manylinux, "manylinux_compatible"): - result = _manylinux.manylinux_compatible(version[0], version[1], arch) - if result is not None: - return bool(result) - return True - if version == _GLibCVersion(2, 5): - if hasattr(_manylinux, "manylinux1_compatible"): - return bool(_manylinux.manylinux1_compatible) - if version == _GLibCVersion(2, 12): - if hasattr(_manylinux, "manylinux2010_compatible"): - return bool(_manylinux.manylinux2010_compatible) - if version == _GLibCVersion(2, 17): - if hasattr(_manylinux, "manylinux2014_compatible"): - return bool(_manylinux.manylinux2014_compatible) - return True - - -_LEGACY_MANYLINUX_MAP = { - # CentOS 7 w/ glibc 2.17 (PEP 599) - (2, 17): "manylinux2014", - # CentOS 6 w/ glibc 2.12 (PEP 571) - (2, 12): "manylinux2010", - # CentOS 5 w/ glibc 2.5 (PEP 513) - (2, 5): "manylinux1", -} - - -def platform_tags(linux: str, arch: str) -> Iterator[str]: - if not _have_compatible_abi(arch): - return - # Oldest glibc to be supported regardless of architecture is (2, 17). - too_old_glibc2 = _GLibCVersion(2, 16) - if arch in {"x86_64", "i686"}: - # On x86/i686 also oldest glibc to be supported is (2, 5). - too_old_glibc2 = _GLibCVersion(2, 4) - current_glibc = _GLibCVersion(*_get_glibc_version()) - glibc_max_list = [current_glibc] - # We can assume compatibility across glibc major versions. - # https://sourceware.org/bugzilla/show_bug.cgi?id=24636 - # - # Build a list of maximum glibc versions so that we can - # output the canonical list of all glibc from current_glibc - # down to too_old_glibc2, including all intermediary versions. - for glibc_major in range(current_glibc.major - 1, 1, -1): - glibc_minor = _LAST_GLIBC_MINOR[glibc_major] - glibc_max_list.append(_GLibCVersion(glibc_major, glibc_minor)) - for glibc_max in glibc_max_list: - if glibc_max.major == too_old_glibc2.major: - min_minor = too_old_glibc2.minor - else: - # For other glibc major versions oldest supported is (x, 0). - min_minor = -1 - for glibc_minor in range(glibc_max.minor, min_minor, -1): - glibc_version = _GLibCVersion(glibc_max.major, glibc_minor) - tag = "manylinux_{}_{}".format(*glibc_version) - if _is_compatible(tag, arch, glibc_version): - yield linux.replace("linux", tag) - # Handle the legacy manylinux1, manylinux2010, manylinux2014 tags. - if glibc_version in _LEGACY_MANYLINUX_MAP: - legacy_tag = _LEGACY_MANYLINUX_MAP[glibc_version] - if _is_compatible(legacy_tag, arch, glibc_version): - yield linux.replace("linux", legacy_tag) diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/packaging/_musllinux.py b/venv/lib/python3.11/site-packages/pip/_vendor/packaging/_musllinux.py deleted file mode 100644 index 8ac3059..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/packaging/_musllinux.py +++ /dev/null @@ -1,136 +0,0 @@ -"""PEP 656 support. - -This module implements logic to detect if the currently running Python is -linked against musl, and what musl version is used. -""" - -import contextlib -import functools -import operator -import os -import re -import struct -import subprocess -import sys -from typing import IO, Iterator, NamedTuple, Optional, Tuple - - -def _read_unpacked(f: IO[bytes], fmt: str) -> Tuple[int, ...]: - return struct.unpack(fmt, f.read(struct.calcsize(fmt))) - - -def _parse_ld_musl_from_elf(f: IO[bytes]) -> Optional[str]: - """Detect musl libc location by parsing the Python executable. - - Based on: https://gist.github.com/lyssdod/f51579ae8d93c8657a5564aefc2ffbca - ELF header: https://refspecs.linuxfoundation.org/elf/gabi4+/ch4.eheader.html - """ - f.seek(0) - try: - ident = _read_unpacked(f, "16B") - except struct.error: - return None - if ident[:4] != tuple(b"\x7fELF"): # Invalid magic, not ELF. - return None - f.seek(struct.calcsize("HHI"), 1) # Skip file type, machine, and version. - - try: - # e_fmt: Format for program header. - # p_fmt: Format for section header. - # p_idx: Indexes to find p_type, p_offset, and p_filesz. - e_fmt, p_fmt, p_idx = { - 1: ("IIIIHHH", "IIIIIIII", (0, 1, 4)), # 32-bit. - 2: ("QQQIHHH", "IIQQQQQQ", (0, 2, 5)), # 64-bit. - }[ident[4]] - except KeyError: - return None - else: - p_get = operator.itemgetter(*p_idx) - - # Find the interpreter section and return its content. - try: - _, e_phoff, _, _, _, e_phentsize, e_phnum = _read_unpacked(f, e_fmt) - except struct.error: - return None - for i in range(e_phnum + 1): - f.seek(e_phoff + e_phentsize * i) - try: - p_type, p_offset, p_filesz = p_get(_read_unpacked(f, p_fmt)) - except struct.error: - return None - if p_type != 3: # Not PT_INTERP. - continue - f.seek(p_offset) - interpreter = os.fsdecode(f.read(p_filesz)).strip("\0") - if "musl" not in interpreter: - return None - return interpreter - return None - - -class _MuslVersion(NamedTuple): - major: int - minor: int - - -def _parse_musl_version(output: str) -> Optional[_MuslVersion]: - lines = [n for n in (n.strip() for n in output.splitlines()) if n] - if len(lines) < 2 or lines[0][:4] != "musl": - return None - m = re.match(r"Version (\d+)\.(\d+)", lines[1]) - if not m: - return None - return _MuslVersion(major=int(m.group(1)), minor=int(m.group(2))) - - -@functools.lru_cache() -def _get_musl_version(executable: str) -> Optional[_MuslVersion]: - """Detect currently-running musl runtime version. - - This is done by checking the specified executable's dynamic linking - information, and invoking the loader to parse its output for a version - string. If the loader is musl, the output would be something like:: - - musl libc (x86_64) - Version 1.2.2 - Dynamic Program Loader - """ - with contextlib.ExitStack() as stack: - try: - f = stack.enter_context(open(executable, "rb")) - except OSError: - return None - ld = _parse_ld_musl_from_elf(f) - if not ld: - return None - proc = subprocess.run([ld], stderr=subprocess.PIPE, universal_newlines=True) - return _parse_musl_version(proc.stderr) - - -def platform_tags(arch: str) -> Iterator[str]: - """Generate musllinux tags compatible to the current platform. - - :param arch: Should be the part of platform tag after the ``linux_`` - prefix, e.g. ``x86_64``. The ``linux_`` prefix is assumed as a - prerequisite for the current platform to be musllinux-compatible. - - :returns: An iterator of compatible musllinux tags. - """ - sys_musl = _get_musl_version(sys.executable) - if sys_musl is None: # Python not dynamically linked against musl. - return - for minor in range(sys_musl.minor, -1, -1): - yield f"musllinux_{sys_musl.major}_{minor}_{arch}" - - -if __name__ == "__main__": # pragma: no cover - import sysconfig - - plat = sysconfig.get_platform() - assert plat.startswith("linux-"), "not linux" - - print("plat:", plat) - print("musl:", _get_musl_version(sys.executable)) - print("tags:", end=" ") - for t in platform_tags(re.sub(r"[.-]", "_", plat.split("-", 1)[-1])): - print(t, end="\n ") diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/packaging/_structures.py b/venv/lib/python3.11/site-packages/pip/_vendor/packaging/_structures.py deleted file mode 100644 index 90a6465..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/packaging/_structures.py +++ /dev/null @@ -1,61 +0,0 @@ -# This file is dual licensed under the terms of the Apache License, Version -# 2.0, and the BSD License. See the LICENSE file in the root of this repository -# for complete details. - - -class InfinityType: - def __repr__(self) -> str: - return "Infinity" - - def __hash__(self) -> int: - return hash(repr(self)) - - def __lt__(self, other: object) -> bool: - return False - - def __le__(self, other: object) -> bool: - return False - - def __eq__(self, other: object) -> bool: - return isinstance(other, self.__class__) - - def __gt__(self, other: object) -> bool: - return True - - def __ge__(self, other: object) -> bool: - return True - - def __neg__(self: object) -> "NegativeInfinityType": - return NegativeInfinity - - -Infinity = InfinityType() - - -class NegativeInfinityType: - def __repr__(self) -> str: - return "-Infinity" - - def __hash__(self) -> int: - return hash(repr(self)) - - def __lt__(self, other: object) -> bool: - return True - - def __le__(self, other: object) -> bool: - return True - - def __eq__(self, other: object) -> bool: - return isinstance(other, self.__class__) - - def __gt__(self, other: object) -> bool: - return False - - def __ge__(self, other: object) -> bool: - return False - - def __neg__(self: object) -> InfinityType: - return Infinity - - -NegativeInfinity = NegativeInfinityType() diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/packaging/markers.py b/venv/lib/python3.11/site-packages/pip/_vendor/packaging/markers.py deleted file mode 100644 index 540e7a4..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/packaging/markers.py +++ /dev/null @@ -1,304 +0,0 @@ -# This file is dual licensed under the terms of the Apache License, Version -# 2.0, and the BSD License. See the LICENSE file in the root of this repository -# for complete details. - -import operator -import os -import platform -import sys -from typing import Any, Callable, Dict, List, Optional, Tuple, Union - -from pip._vendor.pyparsing import ( # noqa: N817 - Forward, - Group, - Literal as L, - ParseException, - ParseResults, - QuotedString, - ZeroOrMore, - stringEnd, - stringStart, -) - -from .specifiers import InvalidSpecifier, Specifier - -__all__ = [ - "InvalidMarker", - "UndefinedComparison", - "UndefinedEnvironmentName", - "Marker", - "default_environment", -] - -Operator = Callable[[str, str], bool] - - -class InvalidMarker(ValueError): - """ - An invalid marker was found, users should refer to PEP 508. - """ - - -class UndefinedComparison(ValueError): - """ - An invalid operation was attempted on a value that doesn't support it. - """ - - -class UndefinedEnvironmentName(ValueError): - """ - A name was attempted to be used that does not exist inside of the - environment. - """ - - -class Node: - def __init__(self, value: Any) -> None: - self.value = value - - def __str__(self) -> str: - return str(self.value) - - def __repr__(self) -> str: - return f"<{self.__class__.__name__}('{self}')>" - - def serialize(self) -> str: - raise NotImplementedError - - -class Variable(Node): - def serialize(self) -> str: - return str(self) - - -class Value(Node): - def serialize(self) -> str: - return f'"{self}"' - - -class Op(Node): - def serialize(self) -> str: - return str(self) - - -VARIABLE = ( - L("implementation_version") - | L("platform_python_implementation") - | L("implementation_name") - | L("python_full_version") - | L("platform_release") - | L("platform_version") - | L("platform_machine") - | L("platform_system") - | L("python_version") - | L("sys_platform") - | L("os_name") - | L("os.name") # PEP-345 - | L("sys.platform") # PEP-345 - | L("platform.version") # PEP-345 - | L("platform.machine") # PEP-345 - | L("platform.python_implementation") # PEP-345 - | L("python_implementation") # undocumented setuptools legacy - | L("extra") # PEP-508 -) -ALIASES = { - "os.name": "os_name", - "sys.platform": "sys_platform", - "platform.version": "platform_version", - "platform.machine": "platform_machine", - "platform.python_implementation": "platform_python_implementation", - "python_implementation": "platform_python_implementation", -} -VARIABLE.setParseAction(lambda s, l, t: Variable(ALIASES.get(t[0], t[0]))) - -VERSION_CMP = ( - L("===") | L("==") | L(">=") | L("<=") | L("!=") | L("~=") | L(">") | L("<") -) - -MARKER_OP = VERSION_CMP | L("not in") | L("in") -MARKER_OP.setParseAction(lambda s, l, t: Op(t[0])) - -MARKER_VALUE = QuotedString("'") | QuotedString('"') -MARKER_VALUE.setParseAction(lambda s, l, t: Value(t[0])) - -BOOLOP = L("and") | L("or") - -MARKER_VAR = VARIABLE | MARKER_VALUE - -MARKER_ITEM = Group(MARKER_VAR + MARKER_OP + MARKER_VAR) -MARKER_ITEM.setParseAction(lambda s, l, t: tuple(t[0])) - -LPAREN = L("(").suppress() -RPAREN = L(")").suppress() - -MARKER_EXPR = Forward() -MARKER_ATOM = MARKER_ITEM | Group(LPAREN + MARKER_EXPR + RPAREN) -MARKER_EXPR << MARKER_ATOM + ZeroOrMore(BOOLOP + MARKER_EXPR) - -MARKER = stringStart + MARKER_EXPR + stringEnd - - -def _coerce_parse_result(results: Union[ParseResults, List[Any]]) -> List[Any]: - if isinstance(results, ParseResults): - return [_coerce_parse_result(i) for i in results] - else: - return results - - -def _format_marker( - marker: Union[List[str], Tuple[Node, ...], str], first: Optional[bool] = True -) -> str: - - assert isinstance(marker, (list, tuple, str)) - - # Sometimes we have a structure like [[...]] which is a single item list - # where the single item is itself it's own list. In that case we want skip - # the rest of this function so that we don't get extraneous () on the - # outside. - if ( - isinstance(marker, list) - and len(marker) == 1 - and isinstance(marker[0], (list, tuple)) - ): - return _format_marker(marker[0]) - - if isinstance(marker, list): - inner = (_format_marker(m, first=False) for m in marker) - if first: - return " ".join(inner) - else: - return "(" + " ".join(inner) + ")" - elif isinstance(marker, tuple): - return " ".join([m.serialize() for m in marker]) - else: - return marker - - -_operators: Dict[str, Operator] = { - "in": lambda lhs, rhs: lhs in rhs, - "not in": lambda lhs, rhs: lhs not in rhs, - "<": operator.lt, - "<=": operator.le, - "==": operator.eq, - "!=": operator.ne, - ">=": operator.ge, - ">": operator.gt, -} - - -def _eval_op(lhs: str, op: Op, rhs: str) -> bool: - try: - spec = Specifier("".join([op.serialize(), rhs])) - except InvalidSpecifier: - pass - else: - return spec.contains(lhs) - - oper: Optional[Operator] = _operators.get(op.serialize()) - if oper is None: - raise UndefinedComparison(f"Undefined {op!r} on {lhs!r} and {rhs!r}.") - - return oper(lhs, rhs) - - -class Undefined: - pass - - -_undefined = Undefined() - - -def _get_env(environment: Dict[str, str], name: str) -> str: - value: Union[str, Undefined] = environment.get(name, _undefined) - - if isinstance(value, Undefined): - raise UndefinedEnvironmentName( - f"{name!r} does not exist in evaluation environment." - ) - - return value - - -def _evaluate_markers(markers: List[Any], environment: Dict[str, str]) -> bool: - groups: List[List[bool]] = [[]] - - for marker in markers: - assert isinstance(marker, (list, tuple, str)) - - if isinstance(marker, list): - groups[-1].append(_evaluate_markers(marker, environment)) - elif isinstance(marker, tuple): - lhs, op, rhs = marker - - if isinstance(lhs, Variable): - lhs_value = _get_env(environment, lhs.value) - rhs_value = rhs.value - else: - lhs_value = lhs.value - rhs_value = _get_env(environment, rhs.value) - - groups[-1].append(_eval_op(lhs_value, op, rhs_value)) - else: - assert marker in ["and", "or"] - if marker == "or": - groups.append([]) - - return any(all(item) for item in groups) - - -def format_full_version(info: "sys._version_info") -> str: - version = "{0.major}.{0.minor}.{0.micro}".format(info) - kind = info.releaselevel - if kind != "final": - version += kind[0] + str(info.serial) - return version - - -def default_environment() -> Dict[str, str]: - iver = format_full_version(sys.implementation.version) - implementation_name = sys.implementation.name - return { - "implementation_name": implementation_name, - "implementation_version": iver, - "os_name": os.name, - "platform_machine": platform.machine(), - "platform_release": platform.release(), - "platform_system": platform.system(), - "platform_version": platform.version(), - "python_full_version": platform.python_version(), - "platform_python_implementation": platform.python_implementation(), - "python_version": ".".join(platform.python_version_tuple()[:2]), - "sys_platform": sys.platform, - } - - -class Marker: - def __init__(self, marker: str) -> None: - try: - self._markers = _coerce_parse_result(MARKER.parseString(marker)) - except ParseException as e: - raise InvalidMarker( - f"Invalid marker: {marker!r}, parse error at " - f"{marker[e.loc : e.loc + 8]!r}" - ) - - def __str__(self) -> str: - return _format_marker(self._markers) - - def __repr__(self) -> str: - return f"" - - def evaluate(self, environment: Optional[Dict[str, str]] = None) -> bool: - """Evaluate a marker. - - Return the boolean from evaluating the given marker against the - environment. environment is an optional argument to override all or - part of the determined environment. - - The environment is determined from the current Python process. - """ - current_environment = default_environment() - if environment is not None: - current_environment.update(environment) - - return _evaluate_markers(self._markers, current_environment) diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/packaging/requirements.py b/venv/lib/python3.11/site-packages/pip/_vendor/packaging/requirements.py deleted file mode 100644 index 1eab7dd..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/packaging/requirements.py +++ /dev/null @@ -1,146 +0,0 @@ -# This file is dual licensed under the terms of the Apache License, Version -# 2.0, and the BSD License. See the LICENSE file in the root of this repository -# for complete details. - -import re -import string -import urllib.parse -from typing import List, Optional as TOptional, Set - -from pip._vendor.pyparsing import ( # noqa - Combine, - Literal as L, - Optional, - ParseException, - Regex, - Word, - ZeroOrMore, - originalTextFor, - stringEnd, - stringStart, -) - -from .markers import MARKER_EXPR, Marker -from .specifiers import LegacySpecifier, Specifier, SpecifierSet - - -class InvalidRequirement(ValueError): - """ - An invalid requirement was found, users should refer to PEP 508. - """ - - -ALPHANUM = Word(string.ascii_letters + string.digits) - -LBRACKET = L("[").suppress() -RBRACKET = L("]").suppress() -LPAREN = L("(").suppress() -RPAREN = L(")").suppress() -COMMA = L(",").suppress() -SEMICOLON = L(";").suppress() -AT = L("@").suppress() - -PUNCTUATION = Word("-_.") -IDENTIFIER_END = ALPHANUM | (ZeroOrMore(PUNCTUATION) + ALPHANUM) -IDENTIFIER = Combine(ALPHANUM + ZeroOrMore(IDENTIFIER_END)) - -NAME = IDENTIFIER("name") -EXTRA = IDENTIFIER - -URI = Regex(r"[^ ]+")("url") -URL = AT + URI - -EXTRAS_LIST = EXTRA + ZeroOrMore(COMMA + EXTRA) -EXTRAS = (LBRACKET + Optional(EXTRAS_LIST) + RBRACKET)("extras") - -VERSION_PEP440 = Regex(Specifier._regex_str, re.VERBOSE | re.IGNORECASE) -VERSION_LEGACY = Regex(LegacySpecifier._regex_str, re.VERBOSE | re.IGNORECASE) - -VERSION_ONE = VERSION_PEP440 ^ VERSION_LEGACY -VERSION_MANY = Combine( - VERSION_ONE + ZeroOrMore(COMMA + VERSION_ONE), joinString=",", adjacent=False -)("_raw_spec") -_VERSION_SPEC = Optional((LPAREN + VERSION_MANY + RPAREN) | VERSION_MANY) -_VERSION_SPEC.setParseAction(lambda s, l, t: t._raw_spec or "") - -VERSION_SPEC = originalTextFor(_VERSION_SPEC)("specifier") -VERSION_SPEC.setParseAction(lambda s, l, t: t[1]) - -MARKER_EXPR = originalTextFor(MARKER_EXPR())("marker") -MARKER_EXPR.setParseAction( - lambda s, l, t: Marker(s[t._original_start : t._original_end]) -) -MARKER_SEPARATOR = SEMICOLON -MARKER = MARKER_SEPARATOR + MARKER_EXPR - -VERSION_AND_MARKER = VERSION_SPEC + Optional(MARKER) -URL_AND_MARKER = URL + Optional(MARKER) - -NAMED_REQUIREMENT = NAME + Optional(EXTRAS) + (URL_AND_MARKER | VERSION_AND_MARKER) - -REQUIREMENT = stringStart + NAMED_REQUIREMENT + stringEnd -# pyparsing isn't thread safe during initialization, so we do it eagerly, see -# issue #104 -REQUIREMENT.parseString("x[]") - - -class Requirement: - """Parse a requirement. - - Parse a given requirement string into its parts, such as name, specifier, - URL, and extras. Raises InvalidRequirement on a badly-formed requirement - string. - """ - - # TODO: Can we test whether something is contained within a requirement? - # If so how do we do that? Do we need to test against the _name_ of - # the thing as well as the version? What about the markers? - # TODO: Can we normalize the name and extra name? - - def __init__(self, requirement_string: str) -> None: - try: - req = REQUIREMENT.parseString(requirement_string) - except ParseException as e: - raise InvalidRequirement( - f'Parse error at "{ requirement_string[e.loc : e.loc + 8]!r}": {e.msg}' - ) - - self.name: str = req.name - if req.url: - parsed_url = urllib.parse.urlparse(req.url) - if parsed_url.scheme == "file": - if urllib.parse.urlunparse(parsed_url) != req.url: - raise InvalidRequirement("Invalid URL given") - elif not (parsed_url.scheme and parsed_url.netloc) or ( - not parsed_url.scheme and not parsed_url.netloc - ): - raise InvalidRequirement(f"Invalid URL: {req.url}") - self.url: TOptional[str] = req.url - else: - self.url = None - self.extras: Set[str] = set(req.extras.asList() if req.extras else []) - self.specifier: SpecifierSet = SpecifierSet(req.specifier) - self.marker: TOptional[Marker] = req.marker if req.marker else None - - def __str__(self) -> str: - parts: List[str] = [self.name] - - if self.extras: - formatted_extras = ",".join(sorted(self.extras)) - parts.append(f"[{formatted_extras}]") - - if self.specifier: - parts.append(str(self.specifier)) - - if self.url: - parts.append(f"@ {self.url}") - if self.marker: - parts.append(" ") - - if self.marker: - parts.append(f"; {self.marker}") - - return "".join(parts) - - def __repr__(self) -> str: - return f"" diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/packaging/specifiers.py b/venv/lib/python3.11/site-packages/pip/_vendor/packaging/specifiers.py deleted file mode 100644 index 0e218a6..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/packaging/specifiers.py +++ /dev/null @@ -1,802 +0,0 @@ -# This file is dual licensed under the terms of the Apache License, Version -# 2.0, and the BSD License. See the LICENSE file in the root of this repository -# for complete details. - -import abc -import functools -import itertools -import re -import warnings -from typing import ( - Callable, - Dict, - Iterable, - Iterator, - List, - Optional, - Pattern, - Set, - Tuple, - TypeVar, - Union, -) - -from .utils import canonicalize_version -from .version import LegacyVersion, Version, parse - -ParsedVersion = Union[Version, LegacyVersion] -UnparsedVersion = Union[Version, LegacyVersion, str] -VersionTypeVar = TypeVar("VersionTypeVar", bound=UnparsedVersion) -CallableOperator = Callable[[ParsedVersion, str], bool] - - -class InvalidSpecifier(ValueError): - """ - An invalid specifier was found, users should refer to PEP 440. - """ - - -class BaseSpecifier(metaclass=abc.ABCMeta): - @abc.abstractmethod - def __str__(self) -> str: - """ - Returns the str representation of this Specifier like object. This - should be representative of the Specifier itself. - """ - - @abc.abstractmethod - def __hash__(self) -> int: - """ - Returns a hash value for this Specifier like object. - """ - - @abc.abstractmethod - def __eq__(self, other: object) -> bool: - """ - Returns a boolean representing whether or not the two Specifier like - objects are equal. - """ - - @abc.abstractproperty - def prereleases(self) -> Optional[bool]: - """ - Returns whether or not pre-releases as a whole are allowed by this - specifier. - """ - - @prereleases.setter - def prereleases(self, value: bool) -> None: - """ - Sets whether or not pre-releases as a whole are allowed by this - specifier. - """ - - @abc.abstractmethod - def contains(self, item: str, prereleases: Optional[bool] = None) -> bool: - """ - Determines if the given item is contained within this specifier. - """ - - @abc.abstractmethod - def filter( - self, iterable: Iterable[VersionTypeVar], prereleases: Optional[bool] = None - ) -> Iterable[VersionTypeVar]: - """ - Takes an iterable of items and filters them so that only items which - are contained within this specifier are allowed in it. - """ - - -class _IndividualSpecifier(BaseSpecifier): - - _operators: Dict[str, str] = {} - _regex: Pattern[str] - - def __init__(self, spec: str = "", prereleases: Optional[bool] = None) -> None: - match = self._regex.search(spec) - if not match: - raise InvalidSpecifier(f"Invalid specifier: '{spec}'") - - self._spec: Tuple[str, str] = ( - match.group("operator").strip(), - match.group("version").strip(), - ) - - # Store whether or not this Specifier should accept prereleases - self._prereleases = prereleases - - def __repr__(self) -> str: - pre = ( - f", prereleases={self.prereleases!r}" - if self._prereleases is not None - else "" - ) - - return f"<{self.__class__.__name__}({str(self)!r}{pre})>" - - def __str__(self) -> str: - return "{}{}".format(*self._spec) - - @property - def _canonical_spec(self) -> Tuple[str, str]: - return self._spec[0], canonicalize_version(self._spec[1]) - - def __hash__(self) -> int: - return hash(self._canonical_spec) - - def __eq__(self, other: object) -> bool: - if isinstance(other, str): - try: - other = self.__class__(str(other)) - except InvalidSpecifier: - return NotImplemented - elif not isinstance(other, self.__class__): - return NotImplemented - - return self._canonical_spec == other._canonical_spec - - def _get_operator(self, op: str) -> CallableOperator: - operator_callable: CallableOperator = getattr( - self, f"_compare_{self._operators[op]}" - ) - return operator_callable - - def _coerce_version(self, version: UnparsedVersion) -> ParsedVersion: - if not isinstance(version, (LegacyVersion, Version)): - version = parse(version) - return version - - @property - def operator(self) -> str: - return self._spec[0] - - @property - def version(self) -> str: - return self._spec[1] - - @property - def prereleases(self) -> Optional[bool]: - return self._prereleases - - @prereleases.setter - def prereleases(self, value: bool) -> None: - self._prereleases = value - - def __contains__(self, item: str) -> bool: - return self.contains(item) - - def contains( - self, item: UnparsedVersion, prereleases: Optional[bool] = None - ) -> bool: - - # Determine if prereleases are to be allowed or not. - if prereleases is None: - prereleases = self.prereleases - - # Normalize item to a Version or LegacyVersion, this allows us to have - # a shortcut for ``"2.0" in Specifier(">=2") - normalized_item = self._coerce_version(item) - - # Determine if we should be supporting prereleases in this specifier - # or not, if we do not support prereleases than we can short circuit - # logic if this version is a prereleases. - if normalized_item.is_prerelease and not prereleases: - return False - - # Actually do the comparison to determine if this item is contained - # within this Specifier or not. - operator_callable: CallableOperator = self._get_operator(self.operator) - return operator_callable(normalized_item, self.version) - - def filter( - self, iterable: Iterable[VersionTypeVar], prereleases: Optional[bool] = None - ) -> Iterable[VersionTypeVar]: - - yielded = False - found_prereleases = [] - - kw = {"prereleases": prereleases if prereleases is not None else True} - - # Attempt to iterate over all the values in the iterable and if any of - # them match, yield them. - for version in iterable: - parsed_version = self._coerce_version(version) - - if self.contains(parsed_version, **kw): - # If our version is a prerelease, and we were not set to allow - # prereleases, then we'll store it for later in case nothing - # else matches this specifier. - if parsed_version.is_prerelease and not ( - prereleases or self.prereleases - ): - found_prereleases.append(version) - # Either this is not a prerelease, or we should have been - # accepting prereleases from the beginning. - else: - yielded = True - yield version - - # Now that we've iterated over everything, determine if we've yielded - # any values, and if we have not and we have any prereleases stored up - # then we will go ahead and yield the prereleases. - if not yielded and found_prereleases: - for version in found_prereleases: - yield version - - -class LegacySpecifier(_IndividualSpecifier): - - _regex_str = r""" - (?P(==|!=|<=|>=|<|>)) - \s* - (?P - [^,;\s)]* # Since this is a "legacy" specifier, and the version - # string can be just about anything, we match everything - # except for whitespace, a semi-colon for marker support, - # a closing paren since versions can be enclosed in - # them, and a comma since it's a version separator. - ) - """ - - _regex = re.compile(r"^\s*" + _regex_str + r"\s*$", re.VERBOSE | re.IGNORECASE) - - _operators = { - "==": "equal", - "!=": "not_equal", - "<=": "less_than_equal", - ">=": "greater_than_equal", - "<": "less_than", - ">": "greater_than", - } - - def __init__(self, spec: str = "", prereleases: Optional[bool] = None) -> None: - super().__init__(spec, prereleases) - - warnings.warn( - "Creating a LegacyVersion has been deprecated and will be " - "removed in the next major release", - DeprecationWarning, - ) - - def _coerce_version(self, version: UnparsedVersion) -> LegacyVersion: - if not isinstance(version, LegacyVersion): - version = LegacyVersion(str(version)) - return version - - def _compare_equal(self, prospective: LegacyVersion, spec: str) -> bool: - return prospective == self._coerce_version(spec) - - def _compare_not_equal(self, prospective: LegacyVersion, spec: str) -> bool: - return prospective != self._coerce_version(spec) - - def _compare_less_than_equal(self, prospective: LegacyVersion, spec: str) -> bool: - return prospective <= self._coerce_version(spec) - - def _compare_greater_than_equal( - self, prospective: LegacyVersion, spec: str - ) -> bool: - return prospective >= self._coerce_version(spec) - - def _compare_less_than(self, prospective: LegacyVersion, spec: str) -> bool: - return prospective < self._coerce_version(spec) - - def _compare_greater_than(self, prospective: LegacyVersion, spec: str) -> bool: - return prospective > self._coerce_version(spec) - - -def _require_version_compare( - fn: Callable[["Specifier", ParsedVersion, str], bool] -) -> Callable[["Specifier", ParsedVersion, str], bool]: - @functools.wraps(fn) - def wrapped(self: "Specifier", prospective: ParsedVersion, spec: str) -> bool: - if not isinstance(prospective, Version): - return False - return fn(self, prospective, spec) - - return wrapped - - -class Specifier(_IndividualSpecifier): - - _regex_str = r""" - (?P(~=|==|!=|<=|>=|<|>|===)) - (?P - (?: - # The identity operators allow for an escape hatch that will - # do an exact string match of the version you wish to install. - # This will not be parsed by PEP 440 and we cannot determine - # any semantic meaning from it. This operator is discouraged - # but included entirely as an escape hatch. - (?<====) # Only match for the identity operator - \s* - [^\s]* # We just match everything, except for whitespace - # since we are only testing for strict identity. - ) - | - (?: - # The (non)equality operators allow for wild card and local - # versions to be specified so we have to define these two - # operators separately to enable that. - (?<===|!=) # Only match for equals and not equals - - \s* - v? - (?:[0-9]+!)? # epoch - [0-9]+(?:\.[0-9]+)* # release - (?: # pre release - [-_\.]? - (a|b|c|rc|alpha|beta|pre|preview) - [-_\.]? - [0-9]* - )? - (?: # post release - (?:-[0-9]+)|(?:[-_\.]?(post|rev|r)[-_\.]?[0-9]*) - )? - - # You cannot use a wild card and a dev or local version - # together so group them with a | and make them optional. - (?: - (?:[-_\.]?dev[-_\.]?[0-9]*)? # dev release - (?:\+[a-z0-9]+(?:[-_\.][a-z0-9]+)*)? # local - | - \.\* # Wild card syntax of .* - )? - ) - | - (?: - # The compatible operator requires at least two digits in the - # release segment. - (?<=~=) # Only match for the compatible operator - - \s* - v? - (?:[0-9]+!)? # epoch - [0-9]+(?:\.[0-9]+)+ # release (We have a + instead of a *) - (?: # pre release - [-_\.]? - (a|b|c|rc|alpha|beta|pre|preview) - [-_\.]? - [0-9]* - )? - (?: # post release - (?:-[0-9]+)|(?:[-_\.]?(post|rev|r)[-_\.]?[0-9]*) - )? - (?:[-_\.]?dev[-_\.]?[0-9]*)? # dev release - ) - | - (?: - # All other operators only allow a sub set of what the - # (non)equality operators do. Specifically they do not allow - # local versions to be specified nor do they allow the prefix - # matching wild cards. - (?=": "greater_than_equal", - "<": "less_than", - ">": "greater_than", - "===": "arbitrary", - } - - @_require_version_compare - def _compare_compatible(self, prospective: ParsedVersion, spec: str) -> bool: - - # Compatible releases have an equivalent combination of >= and ==. That - # is that ~=2.2 is equivalent to >=2.2,==2.*. This allows us to - # implement this in terms of the other specifiers instead of - # implementing it ourselves. The only thing we need to do is construct - # the other specifiers. - - # We want everything but the last item in the version, but we want to - # ignore suffix segments. - prefix = ".".join( - list(itertools.takewhile(_is_not_suffix, _version_split(spec)))[:-1] - ) - - # Add the prefix notation to the end of our string - prefix += ".*" - - return self._get_operator(">=")(prospective, spec) and self._get_operator("==")( - prospective, prefix - ) - - @_require_version_compare - def _compare_equal(self, prospective: ParsedVersion, spec: str) -> bool: - - # We need special logic to handle prefix matching - if spec.endswith(".*"): - # In the case of prefix matching we want to ignore local segment. - prospective = Version(prospective.public) - # Split the spec out by dots, and pretend that there is an implicit - # dot in between a release segment and a pre-release segment. - split_spec = _version_split(spec[:-2]) # Remove the trailing .* - - # Split the prospective version out by dots, and pretend that there - # is an implicit dot in between a release segment and a pre-release - # segment. - split_prospective = _version_split(str(prospective)) - - # Shorten the prospective version to be the same length as the spec - # so that we can determine if the specifier is a prefix of the - # prospective version or not. - shortened_prospective = split_prospective[: len(split_spec)] - - # Pad out our two sides with zeros so that they both equal the same - # length. - padded_spec, padded_prospective = _pad_version( - split_spec, shortened_prospective - ) - - return padded_prospective == padded_spec - else: - # Convert our spec string into a Version - spec_version = Version(spec) - - # If the specifier does not have a local segment, then we want to - # act as if the prospective version also does not have a local - # segment. - if not spec_version.local: - prospective = Version(prospective.public) - - return prospective == spec_version - - @_require_version_compare - def _compare_not_equal(self, prospective: ParsedVersion, spec: str) -> bool: - return not self._compare_equal(prospective, spec) - - @_require_version_compare - def _compare_less_than_equal(self, prospective: ParsedVersion, spec: str) -> bool: - - # NB: Local version identifiers are NOT permitted in the version - # specifier, so local version labels can be universally removed from - # the prospective version. - return Version(prospective.public) <= Version(spec) - - @_require_version_compare - def _compare_greater_than_equal( - self, prospective: ParsedVersion, spec: str - ) -> bool: - - # NB: Local version identifiers are NOT permitted in the version - # specifier, so local version labels can be universally removed from - # the prospective version. - return Version(prospective.public) >= Version(spec) - - @_require_version_compare - def _compare_less_than(self, prospective: ParsedVersion, spec_str: str) -> bool: - - # Convert our spec to a Version instance, since we'll want to work with - # it as a version. - spec = Version(spec_str) - - # Check to see if the prospective version is less than the spec - # version. If it's not we can short circuit and just return False now - # instead of doing extra unneeded work. - if not prospective < spec: - return False - - # This special case is here so that, unless the specifier itself - # includes is a pre-release version, that we do not accept pre-release - # versions for the version mentioned in the specifier (e.g. <3.1 should - # not match 3.1.dev0, but should match 3.0.dev0). - if not spec.is_prerelease and prospective.is_prerelease: - if Version(prospective.base_version) == Version(spec.base_version): - return False - - # If we've gotten to here, it means that prospective version is both - # less than the spec version *and* it's not a pre-release of the same - # version in the spec. - return True - - @_require_version_compare - def _compare_greater_than(self, prospective: ParsedVersion, spec_str: str) -> bool: - - # Convert our spec to a Version instance, since we'll want to work with - # it as a version. - spec = Version(spec_str) - - # Check to see if the prospective version is greater than the spec - # version. If it's not we can short circuit and just return False now - # instead of doing extra unneeded work. - if not prospective > spec: - return False - - # This special case is here so that, unless the specifier itself - # includes is a post-release version, that we do not accept - # post-release versions for the version mentioned in the specifier - # (e.g. >3.1 should not match 3.0.post0, but should match 3.2.post0). - if not spec.is_postrelease and prospective.is_postrelease: - if Version(prospective.base_version) == Version(spec.base_version): - return False - - # Ensure that we do not allow a local version of the version mentioned - # in the specifier, which is technically greater than, to match. - if prospective.local is not None: - if Version(prospective.base_version) == Version(spec.base_version): - return False - - # If we've gotten to here, it means that prospective version is both - # greater than the spec version *and* it's not a pre-release of the - # same version in the spec. - return True - - def _compare_arbitrary(self, prospective: Version, spec: str) -> bool: - return str(prospective).lower() == str(spec).lower() - - @property - def prereleases(self) -> bool: - - # If there is an explicit prereleases set for this, then we'll just - # blindly use that. - if self._prereleases is not None: - return self._prereleases - - # Look at all of our specifiers and determine if they are inclusive - # operators, and if they are if they are including an explicit - # prerelease. - operator, version = self._spec - if operator in ["==", ">=", "<=", "~=", "==="]: - # The == specifier can include a trailing .*, if it does we - # want to remove before parsing. - if operator == "==" and version.endswith(".*"): - version = version[:-2] - - # Parse the version, and if it is a pre-release than this - # specifier allows pre-releases. - if parse(version).is_prerelease: - return True - - return False - - @prereleases.setter - def prereleases(self, value: bool) -> None: - self._prereleases = value - - -_prefix_regex = re.compile(r"^([0-9]+)((?:a|b|c|rc)[0-9]+)$") - - -def _version_split(version: str) -> List[str]: - result: List[str] = [] - for item in version.split("."): - match = _prefix_regex.search(item) - if match: - result.extend(match.groups()) - else: - result.append(item) - return result - - -def _is_not_suffix(segment: str) -> bool: - return not any( - segment.startswith(prefix) for prefix in ("dev", "a", "b", "rc", "post") - ) - - -def _pad_version(left: List[str], right: List[str]) -> Tuple[List[str], List[str]]: - left_split, right_split = [], [] - - # Get the release segment of our versions - left_split.append(list(itertools.takewhile(lambda x: x.isdigit(), left))) - right_split.append(list(itertools.takewhile(lambda x: x.isdigit(), right))) - - # Get the rest of our versions - left_split.append(left[len(left_split[0]) :]) - right_split.append(right[len(right_split[0]) :]) - - # Insert our padding - left_split.insert(1, ["0"] * max(0, len(right_split[0]) - len(left_split[0]))) - right_split.insert(1, ["0"] * max(0, len(left_split[0]) - len(right_split[0]))) - - return (list(itertools.chain(*left_split)), list(itertools.chain(*right_split))) - - -class SpecifierSet(BaseSpecifier): - def __init__( - self, specifiers: str = "", prereleases: Optional[bool] = None - ) -> None: - - # Split on , to break each individual specifier into it's own item, and - # strip each item to remove leading/trailing whitespace. - split_specifiers = [s.strip() for s in specifiers.split(",") if s.strip()] - - # Parsed each individual specifier, attempting first to make it a - # Specifier and falling back to a LegacySpecifier. - parsed: Set[_IndividualSpecifier] = set() - for specifier in split_specifiers: - try: - parsed.add(Specifier(specifier)) - except InvalidSpecifier: - parsed.add(LegacySpecifier(specifier)) - - # Turn our parsed specifiers into a frozen set and save them for later. - self._specs = frozenset(parsed) - - # Store our prereleases value so we can use it later to determine if - # we accept prereleases or not. - self._prereleases = prereleases - - def __repr__(self) -> str: - pre = ( - f", prereleases={self.prereleases!r}" - if self._prereleases is not None - else "" - ) - - return f"" - - def __str__(self) -> str: - return ",".join(sorted(str(s) for s in self._specs)) - - def __hash__(self) -> int: - return hash(self._specs) - - def __and__(self, other: Union["SpecifierSet", str]) -> "SpecifierSet": - if isinstance(other, str): - other = SpecifierSet(other) - elif not isinstance(other, SpecifierSet): - return NotImplemented - - specifier = SpecifierSet() - specifier._specs = frozenset(self._specs | other._specs) - - if self._prereleases is None and other._prereleases is not None: - specifier._prereleases = other._prereleases - elif self._prereleases is not None and other._prereleases is None: - specifier._prereleases = self._prereleases - elif self._prereleases == other._prereleases: - specifier._prereleases = self._prereleases - else: - raise ValueError( - "Cannot combine SpecifierSets with True and False prerelease " - "overrides." - ) - - return specifier - - def __eq__(self, other: object) -> bool: - if isinstance(other, (str, _IndividualSpecifier)): - other = SpecifierSet(str(other)) - elif not isinstance(other, SpecifierSet): - return NotImplemented - - return self._specs == other._specs - - def __len__(self) -> int: - return len(self._specs) - - def __iter__(self) -> Iterator[_IndividualSpecifier]: - return iter(self._specs) - - @property - def prereleases(self) -> Optional[bool]: - - # If we have been given an explicit prerelease modifier, then we'll - # pass that through here. - if self._prereleases is not None: - return self._prereleases - - # If we don't have any specifiers, and we don't have a forced value, - # then we'll just return None since we don't know if this should have - # pre-releases or not. - if not self._specs: - return None - - # Otherwise we'll see if any of the given specifiers accept - # prereleases, if any of them do we'll return True, otherwise False. - return any(s.prereleases for s in self._specs) - - @prereleases.setter - def prereleases(self, value: bool) -> None: - self._prereleases = value - - def __contains__(self, item: UnparsedVersion) -> bool: - return self.contains(item) - - def contains( - self, item: UnparsedVersion, prereleases: Optional[bool] = None - ) -> bool: - - # Ensure that our item is a Version or LegacyVersion instance. - if not isinstance(item, (LegacyVersion, Version)): - item = parse(item) - - # Determine if we're forcing a prerelease or not, if we're not forcing - # one for this particular filter call, then we'll use whatever the - # SpecifierSet thinks for whether or not we should support prereleases. - if prereleases is None: - prereleases = self.prereleases - - # We can determine if we're going to allow pre-releases by looking to - # see if any of the underlying items supports them. If none of them do - # and this item is a pre-release then we do not allow it and we can - # short circuit that here. - # Note: This means that 1.0.dev1 would not be contained in something - # like >=1.0.devabc however it would be in >=1.0.debabc,>0.0.dev0 - if not prereleases and item.is_prerelease: - return False - - # We simply dispatch to the underlying specs here to make sure that the - # given version is contained within all of them. - # Note: This use of all() here means that an empty set of specifiers - # will always return True, this is an explicit design decision. - return all(s.contains(item, prereleases=prereleases) for s in self._specs) - - def filter( - self, iterable: Iterable[VersionTypeVar], prereleases: Optional[bool] = None - ) -> Iterable[VersionTypeVar]: - - # Determine if we're forcing a prerelease or not, if we're not forcing - # one for this particular filter call, then we'll use whatever the - # SpecifierSet thinks for whether or not we should support prereleases. - if prereleases is None: - prereleases = self.prereleases - - # If we have any specifiers, then we want to wrap our iterable in the - # filter method for each one, this will act as a logical AND amongst - # each specifier. - if self._specs: - for spec in self._specs: - iterable = spec.filter(iterable, prereleases=bool(prereleases)) - return iterable - # If we do not have any specifiers, then we need to have a rough filter - # which will filter out any pre-releases, unless there are no final - # releases, and which will filter out LegacyVersion in general. - else: - filtered: List[VersionTypeVar] = [] - found_prereleases: List[VersionTypeVar] = [] - - item: UnparsedVersion - parsed_version: Union[Version, LegacyVersion] - - for item in iterable: - # Ensure that we some kind of Version class for this item. - if not isinstance(item, (LegacyVersion, Version)): - parsed_version = parse(item) - else: - parsed_version = item - - # Filter out any item which is parsed as a LegacyVersion - if isinstance(parsed_version, LegacyVersion): - continue - - # Store any item which is a pre-release for later unless we've - # already found a final version or we are accepting prereleases - if parsed_version.is_prerelease and not prereleases: - if not filtered: - found_prereleases.append(item) - else: - filtered.append(item) - - # If we've found no items except for pre-releases, then we'll go - # ahead and use the pre-releases - if not filtered and found_prereleases and prereleases is None: - return found_prereleases - - return filtered diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/packaging/tags.py b/venv/lib/python3.11/site-packages/pip/_vendor/packaging/tags.py deleted file mode 100644 index 9a3d25a..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/packaging/tags.py +++ /dev/null @@ -1,487 +0,0 @@ -# This file is dual licensed under the terms of the Apache License, Version -# 2.0, and the BSD License. See the LICENSE file in the root of this repository -# for complete details. - -import logging -import platform -import sys -import sysconfig -from importlib.machinery import EXTENSION_SUFFIXES -from typing import ( - Dict, - FrozenSet, - Iterable, - Iterator, - List, - Optional, - Sequence, - Tuple, - Union, - cast, -) - -from . import _manylinux, _musllinux - -logger = logging.getLogger(__name__) - -PythonVersion = Sequence[int] -MacVersion = Tuple[int, int] - -INTERPRETER_SHORT_NAMES: Dict[str, str] = { - "python": "py", # Generic. - "cpython": "cp", - "pypy": "pp", - "ironpython": "ip", - "jython": "jy", -} - - -_32_BIT_INTERPRETER = sys.maxsize <= 2 ** 32 - - -class Tag: - """ - A representation of the tag triple for a wheel. - - Instances are considered immutable and thus are hashable. Equality checking - is also supported. - """ - - __slots__ = ["_interpreter", "_abi", "_platform", "_hash"] - - def __init__(self, interpreter: str, abi: str, platform: str) -> None: - self._interpreter = interpreter.lower() - self._abi = abi.lower() - self._platform = platform.lower() - # The __hash__ of every single element in a Set[Tag] will be evaluated each time - # that a set calls its `.disjoint()` method, which may be called hundreds of - # times when scanning a page of links for packages with tags matching that - # Set[Tag]. Pre-computing the value here produces significant speedups for - # downstream consumers. - self._hash = hash((self._interpreter, self._abi, self._platform)) - - @property - def interpreter(self) -> str: - return self._interpreter - - @property - def abi(self) -> str: - return self._abi - - @property - def platform(self) -> str: - return self._platform - - def __eq__(self, other: object) -> bool: - if not isinstance(other, Tag): - return NotImplemented - - return ( - (self._hash == other._hash) # Short-circuit ASAP for perf reasons. - and (self._platform == other._platform) - and (self._abi == other._abi) - and (self._interpreter == other._interpreter) - ) - - def __hash__(self) -> int: - return self._hash - - def __str__(self) -> str: - return f"{self._interpreter}-{self._abi}-{self._platform}" - - def __repr__(self) -> str: - return f"<{self} @ {id(self)}>" - - -def parse_tag(tag: str) -> FrozenSet[Tag]: - """ - Parses the provided tag (e.g. `py3-none-any`) into a frozenset of Tag instances. - - Returning a set is required due to the possibility that the tag is a - compressed tag set. - """ - tags = set() - interpreters, abis, platforms = tag.split("-") - for interpreter in interpreters.split("."): - for abi in abis.split("."): - for platform_ in platforms.split("."): - tags.add(Tag(interpreter, abi, platform_)) - return frozenset(tags) - - -def _get_config_var(name: str, warn: bool = False) -> Union[int, str, None]: - value = sysconfig.get_config_var(name) - if value is None and warn: - logger.debug( - "Config variable '%s' is unset, Python ABI tag may be incorrect", name - ) - return value - - -def _normalize_string(string: str) -> str: - return string.replace(".", "_").replace("-", "_") - - -def _abi3_applies(python_version: PythonVersion) -> bool: - """ - Determine if the Python version supports abi3. - - PEP 384 was first implemented in Python 3.2. - """ - return len(python_version) > 1 and tuple(python_version) >= (3, 2) - - -def _cpython_abis(py_version: PythonVersion, warn: bool = False) -> List[str]: - py_version = tuple(py_version) # To allow for version comparison. - abis = [] - version = _version_nodot(py_version[:2]) - debug = pymalloc = ucs4 = "" - with_debug = _get_config_var("Py_DEBUG", warn) - has_refcount = hasattr(sys, "gettotalrefcount") - # Windows doesn't set Py_DEBUG, so checking for support of debug-compiled - # extension modules is the best option. - # https://github.com/pypa/pip/issues/3383#issuecomment-173267692 - has_ext = "_d.pyd" in EXTENSION_SUFFIXES - if with_debug or (with_debug is None and (has_refcount or has_ext)): - debug = "d" - if py_version < (3, 8): - with_pymalloc = _get_config_var("WITH_PYMALLOC", warn) - if with_pymalloc or with_pymalloc is None: - pymalloc = "m" - if py_version < (3, 3): - unicode_size = _get_config_var("Py_UNICODE_SIZE", warn) - if unicode_size == 4 or ( - unicode_size is None and sys.maxunicode == 0x10FFFF - ): - ucs4 = "u" - elif debug: - # Debug builds can also load "normal" extension modules. - # We can also assume no UCS-4 or pymalloc requirement. - abis.append(f"cp{version}") - abis.insert( - 0, - "cp{version}{debug}{pymalloc}{ucs4}".format( - version=version, debug=debug, pymalloc=pymalloc, ucs4=ucs4 - ), - ) - return abis - - -def cpython_tags( - python_version: Optional[PythonVersion] = None, - abis: Optional[Iterable[str]] = None, - platforms: Optional[Iterable[str]] = None, - *, - warn: bool = False, -) -> Iterator[Tag]: - """ - Yields the tags for a CPython interpreter. - - The tags consist of: - - cp-- - - cp-abi3- - - cp-none- - - cp-abi3- # Older Python versions down to 3.2. - - If python_version only specifies a major version then user-provided ABIs and - the 'none' ABItag will be used. - - If 'abi3' or 'none' are specified in 'abis' then they will be yielded at - their normal position and not at the beginning. - """ - if not python_version: - python_version = sys.version_info[:2] - - interpreter = f"cp{_version_nodot(python_version[:2])}" - - if abis is None: - if len(python_version) > 1: - abis = _cpython_abis(python_version, warn) - else: - abis = [] - abis = list(abis) - # 'abi3' and 'none' are explicitly handled later. - for explicit_abi in ("abi3", "none"): - try: - abis.remove(explicit_abi) - except ValueError: - pass - - platforms = list(platforms or platform_tags()) - for abi in abis: - for platform_ in platforms: - yield Tag(interpreter, abi, platform_) - if _abi3_applies(python_version): - yield from (Tag(interpreter, "abi3", platform_) for platform_ in platforms) - yield from (Tag(interpreter, "none", platform_) for platform_ in platforms) - - if _abi3_applies(python_version): - for minor_version in range(python_version[1] - 1, 1, -1): - for platform_ in platforms: - interpreter = "cp{version}".format( - version=_version_nodot((python_version[0], minor_version)) - ) - yield Tag(interpreter, "abi3", platform_) - - -def _generic_abi() -> Iterator[str]: - abi = sysconfig.get_config_var("SOABI") - if abi: - yield _normalize_string(abi) - - -def generic_tags( - interpreter: Optional[str] = None, - abis: Optional[Iterable[str]] = None, - platforms: Optional[Iterable[str]] = None, - *, - warn: bool = False, -) -> Iterator[Tag]: - """ - Yields the tags for a generic interpreter. - - The tags consist of: - - -- - - The "none" ABI will be added if it was not explicitly provided. - """ - if not interpreter: - interp_name = interpreter_name() - interp_version = interpreter_version(warn=warn) - interpreter = "".join([interp_name, interp_version]) - if abis is None: - abis = _generic_abi() - platforms = list(platforms or platform_tags()) - abis = list(abis) - if "none" not in abis: - abis.append("none") - for abi in abis: - for platform_ in platforms: - yield Tag(interpreter, abi, platform_) - - -def _py_interpreter_range(py_version: PythonVersion) -> Iterator[str]: - """ - Yields Python versions in descending order. - - After the latest version, the major-only version will be yielded, and then - all previous versions of that major version. - """ - if len(py_version) > 1: - yield f"py{_version_nodot(py_version[:2])}" - yield f"py{py_version[0]}" - if len(py_version) > 1: - for minor in range(py_version[1] - 1, -1, -1): - yield f"py{_version_nodot((py_version[0], minor))}" - - -def compatible_tags( - python_version: Optional[PythonVersion] = None, - interpreter: Optional[str] = None, - platforms: Optional[Iterable[str]] = None, -) -> Iterator[Tag]: - """ - Yields the sequence of tags that are compatible with a specific version of Python. - - The tags consist of: - - py*-none- - - -none-any # ... if `interpreter` is provided. - - py*-none-any - """ - if not python_version: - python_version = sys.version_info[:2] - platforms = list(platforms or platform_tags()) - for version in _py_interpreter_range(python_version): - for platform_ in platforms: - yield Tag(version, "none", platform_) - if interpreter: - yield Tag(interpreter, "none", "any") - for version in _py_interpreter_range(python_version): - yield Tag(version, "none", "any") - - -def _mac_arch(arch: str, is_32bit: bool = _32_BIT_INTERPRETER) -> str: - if not is_32bit: - return arch - - if arch.startswith("ppc"): - return "ppc" - - return "i386" - - -def _mac_binary_formats(version: MacVersion, cpu_arch: str) -> List[str]: - formats = [cpu_arch] - if cpu_arch == "x86_64": - if version < (10, 4): - return [] - formats.extend(["intel", "fat64", "fat32"]) - - elif cpu_arch == "i386": - if version < (10, 4): - return [] - formats.extend(["intel", "fat32", "fat"]) - - elif cpu_arch == "ppc64": - # TODO: Need to care about 32-bit PPC for ppc64 through 10.2? - if version > (10, 5) or version < (10, 4): - return [] - formats.append("fat64") - - elif cpu_arch == "ppc": - if version > (10, 6): - return [] - formats.extend(["fat32", "fat"]) - - if cpu_arch in {"arm64", "x86_64"}: - formats.append("universal2") - - if cpu_arch in {"x86_64", "i386", "ppc64", "ppc", "intel"}: - formats.append("universal") - - return formats - - -def mac_platforms( - version: Optional[MacVersion] = None, arch: Optional[str] = None -) -> Iterator[str]: - """ - Yields the platform tags for a macOS system. - - The `version` parameter is a two-item tuple specifying the macOS version to - generate platform tags for. The `arch` parameter is the CPU architecture to - generate platform tags for. Both parameters default to the appropriate value - for the current system. - """ - version_str, _, cpu_arch = platform.mac_ver() - if version is None: - version = cast("MacVersion", tuple(map(int, version_str.split(".")[:2]))) - else: - version = version - if arch is None: - arch = _mac_arch(cpu_arch) - else: - arch = arch - - if (10, 0) <= version and version < (11, 0): - # Prior to Mac OS 11, each yearly release of Mac OS bumped the - # "minor" version number. The major version was always 10. - for minor_version in range(version[1], -1, -1): - compat_version = 10, minor_version - binary_formats = _mac_binary_formats(compat_version, arch) - for binary_format in binary_formats: - yield "macosx_{major}_{minor}_{binary_format}".format( - major=10, minor=minor_version, binary_format=binary_format - ) - - if version >= (11, 0): - # Starting with Mac OS 11, each yearly release bumps the major version - # number. The minor versions are now the midyear updates. - for major_version in range(version[0], 10, -1): - compat_version = major_version, 0 - binary_formats = _mac_binary_formats(compat_version, arch) - for binary_format in binary_formats: - yield "macosx_{major}_{minor}_{binary_format}".format( - major=major_version, minor=0, binary_format=binary_format - ) - - if version >= (11, 0): - # Mac OS 11 on x86_64 is compatible with binaries from previous releases. - # Arm64 support was introduced in 11.0, so no Arm binaries from previous - # releases exist. - # - # However, the "universal2" binary format can have a - # macOS version earlier than 11.0 when the x86_64 part of the binary supports - # that version of macOS. - if arch == "x86_64": - for minor_version in range(16, 3, -1): - compat_version = 10, minor_version - binary_formats = _mac_binary_formats(compat_version, arch) - for binary_format in binary_formats: - yield "macosx_{major}_{minor}_{binary_format}".format( - major=compat_version[0], - minor=compat_version[1], - binary_format=binary_format, - ) - else: - for minor_version in range(16, 3, -1): - compat_version = 10, minor_version - binary_format = "universal2" - yield "macosx_{major}_{minor}_{binary_format}".format( - major=compat_version[0], - minor=compat_version[1], - binary_format=binary_format, - ) - - -def _linux_platforms(is_32bit: bool = _32_BIT_INTERPRETER) -> Iterator[str]: - linux = _normalize_string(sysconfig.get_platform()) - if is_32bit: - if linux == "linux_x86_64": - linux = "linux_i686" - elif linux == "linux_aarch64": - linux = "linux_armv7l" - _, arch = linux.split("_", 1) - yield from _manylinux.platform_tags(linux, arch) - yield from _musllinux.platform_tags(arch) - yield linux - - -def _generic_platforms() -> Iterator[str]: - yield _normalize_string(sysconfig.get_platform()) - - -def platform_tags() -> Iterator[str]: - """ - Provides the platform tags for this installation. - """ - if platform.system() == "Darwin": - return mac_platforms() - elif platform.system() == "Linux": - return _linux_platforms() - else: - return _generic_platforms() - - -def interpreter_name() -> str: - """ - Returns the name of the running interpreter. - """ - name = sys.implementation.name - return INTERPRETER_SHORT_NAMES.get(name) or name - - -def interpreter_version(*, warn: bool = False) -> str: - """ - Returns the version of the running interpreter. - """ - version = _get_config_var("py_version_nodot", warn=warn) - if version: - version = str(version) - else: - version = _version_nodot(sys.version_info[:2]) - return version - - -def _version_nodot(version: PythonVersion) -> str: - return "".join(map(str, version)) - - -def sys_tags(*, warn: bool = False) -> Iterator[Tag]: - """ - Returns the sequence of tag triples for the running interpreter. - - The order of the sequence corresponds to priority order for the - interpreter, from most to least important. - """ - - interp_name = interpreter_name() - if interp_name == "cp": - yield from cpython_tags(warn=warn) - else: - yield from generic_tags() - - if interp_name == "pp": - yield from compatible_tags(interpreter="pp3") - else: - yield from compatible_tags() diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/packaging/utils.py b/venv/lib/python3.11/site-packages/pip/_vendor/packaging/utils.py deleted file mode 100644 index bab11b8..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/packaging/utils.py +++ /dev/null @@ -1,136 +0,0 @@ -# This file is dual licensed under the terms of the Apache License, Version -# 2.0, and the BSD License. See the LICENSE file in the root of this repository -# for complete details. - -import re -from typing import FrozenSet, NewType, Tuple, Union, cast - -from .tags import Tag, parse_tag -from .version import InvalidVersion, Version - -BuildTag = Union[Tuple[()], Tuple[int, str]] -NormalizedName = NewType("NormalizedName", str) - - -class InvalidWheelFilename(ValueError): - """ - An invalid wheel filename was found, users should refer to PEP 427. - """ - - -class InvalidSdistFilename(ValueError): - """ - An invalid sdist filename was found, users should refer to the packaging user guide. - """ - - -_canonicalize_regex = re.compile(r"[-_.]+") -# PEP 427: The build number must start with a digit. -_build_tag_regex = re.compile(r"(\d+)(.*)") - - -def canonicalize_name(name: str) -> NormalizedName: - # This is taken from PEP 503. - value = _canonicalize_regex.sub("-", name).lower() - return cast(NormalizedName, value) - - -def canonicalize_version(version: Union[Version, str]) -> str: - """ - This is very similar to Version.__str__, but has one subtle difference - with the way it handles the release segment. - """ - if isinstance(version, str): - try: - parsed = Version(version) - except InvalidVersion: - # Legacy versions cannot be normalized - return version - else: - parsed = version - - parts = [] - - # Epoch - if parsed.epoch != 0: - parts.append(f"{parsed.epoch}!") - - # Release segment - # NB: This strips trailing '.0's to normalize - parts.append(re.sub(r"(\.0)+$", "", ".".join(str(x) for x in parsed.release))) - - # Pre-release - if parsed.pre is not None: - parts.append("".join(str(x) for x in parsed.pre)) - - # Post-release - if parsed.post is not None: - parts.append(f".post{parsed.post}") - - # Development release - if parsed.dev is not None: - parts.append(f".dev{parsed.dev}") - - # Local version segment - if parsed.local is not None: - parts.append(f"+{parsed.local}") - - return "".join(parts) - - -def parse_wheel_filename( - filename: str, -) -> Tuple[NormalizedName, Version, BuildTag, FrozenSet[Tag]]: - if not filename.endswith(".whl"): - raise InvalidWheelFilename( - f"Invalid wheel filename (extension must be '.whl'): {filename}" - ) - - filename = filename[:-4] - dashes = filename.count("-") - if dashes not in (4, 5): - raise InvalidWheelFilename( - f"Invalid wheel filename (wrong number of parts): {filename}" - ) - - parts = filename.split("-", dashes - 2) - name_part = parts[0] - # See PEP 427 for the rules on escaping the project name - if "__" in name_part or re.match(r"^[\w\d._]*$", name_part, re.UNICODE) is None: - raise InvalidWheelFilename(f"Invalid project name: {filename}") - name = canonicalize_name(name_part) - version = Version(parts[1]) - if dashes == 5: - build_part = parts[2] - build_match = _build_tag_regex.match(build_part) - if build_match is None: - raise InvalidWheelFilename( - f"Invalid build number: {build_part} in '{filename}'" - ) - build = cast(BuildTag, (int(build_match.group(1)), build_match.group(2))) - else: - build = () - tags = parse_tag(parts[-1]) - return (name, version, build, tags) - - -def parse_sdist_filename(filename: str) -> Tuple[NormalizedName, Version]: - if filename.endswith(".tar.gz"): - file_stem = filename[: -len(".tar.gz")] - elif filename.endswith(".zip"): - file_stem = filename[: -len(".zip")] - else: - raise InvalidSdistFilename( - f"Invalid sdist filename (extension must be '.tar.gz' or '.zip'):" - f" {filename}" - ) - - # We are requiring a PEP 440 version, which cannot contain dashes, - # so we split on the last dash. - name_part, sep, version_part = file_stem.rpartition("-") - if not sep: - raise InvalidSdistFilename(f"Invalid sdist filename: {filename}") - - name = canonicalize_name(name_part) - version = Version(version_part) - return (name, version) diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/packaging/version.py b/venv/lib/python3.11/site-packages/pip/_vendor/packaging/version.py deleted file mode 100644 index de9a09a..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/packaging/version.py +++ /dev/null @@ -1,504 +0,0 @@ -# This file is dual licensed under the terms of the Apache License, Version -# 2.0, and the BSD License. See the LICENSE file in the root of this repository -# for complete details. - -import collections -import itertools -import re -import warnings -from typing import Callable, Iterator, List, Optional, SupportsInt, Tuple, Union - -from ._structures import Infinity, InfinityType, NegativeInfinity, NegativeInfinityType - -__all__ = ["parse", "Version", "LegacyVersion", "InvalidVersion", "VERSION_PATTERN"] - -InfiniteTypes = Union[InfinityType, NegativeInfinityType] -PrePostDevType = Union[InfiniteTypes, Tuple[str, int]] -SubLocalType = Union[InfiniteTypes, int, str] -LocalType = Union[ - NegativeInfinityType, - Tuple[ - Union[ - SubLocalType, - Tuple[SubLocalType, str], - Tuple[NegativeInfinityType, SubLocalType], - ], - ..., - ], -] -CmpKey = Tuple[ - int, Tuple[int, ...], PrePostDevType, PrePostDevType, PrePostDevType, LocalType -] -LegacyCmpKey = Tuple[int, Tuple[str, ...]] -VersionComparisonMethod = Callable[ - [Union[CmpKey, LegacyCmpKey], Union[CmpKey, LegacyCmpKey]], bool -] - -_Version = collections.namedtuple( - "_Version", ["epoch", "release", "dev", "pre", "post", "local"] -) - - -def parse(version: str) -> Union["LegacyVersion", "Version"]: - """ - Parse the given version string and return either a :class:`Version` object - or a :class:`LegacyVersion` object depending on if the given version is - a valid PEP 440 version or a legacy version. - """ - try: - return Version(version) - except InvalidVersion: - return LegacyVersion(version) - - -class InvalidVersion(ValueError): - """ - An invalid version was found, users should refer to PEP 440. - """ - - -class _BaseVersion: - _key: Union[CmpKey, LegacyCmpKey] - - def __hash__(self) -> int: - return hash(self._key) - - # Please keep the duplicated `isinstance` check - # in the six comparisons hereunder - # unless you find a way to avoid adding overhead function calls. - def __lt__(self, other: "_BaseVersion") -> bool: - if not isinstance(other, _BaseVersion): - return NotImplemented - - return self._key < other._key - - def __le__(self, other: "_BaseVersion") -> bool: - if not isinstance(other, _BaseVersion): - return NotImplemented - - return self._key <= other._key - - def __eq__(self, other: object) -> bool: - if not isinstance(other, _BaseVersion): - return NotImplemented - - return self._key == other._key - - def __ge__(self, other: "_BaseVersion") -> bool: - if not isinstance(other, _BaseVersion): - return NotImplemented - - return self._key >= other._key - - def __gt__(self, other: "_BaseVersion") -> bool: - if not isinstance(other, _BaseVersion): - return NotImplemented - - return self._key > other._key - - def __ne__(self, other: object) -> bool: - if not isinstance(other, _BaseVersion): - return NotImplemented - - return self._key != other._key - - -class LegacyVersion(_BaseVersion): - def __init__(self, version: str) -> None: - self._version = str(version) - self._key = _legacy_cmpkey(self._version) - - warnings.warn( - "Creating a LegacyVersion has been deprecated and will be " - "removed in the next major release", - DeprecationWarning, - ) - - def __str__(self) -> str: - return self._version - - def __repr__(self) -> str: - return f"" - - @property - def public(self) -> str: - return self._version - - @property - def base_version(self) -> str: - return self._version - - @property - def epoch(self) -> int: - return -1 - - @property - def release(self) -> None: - return None - - @property - def pre(self) -> None: - return None - - @property - def post(self) -> None: - return None - - @property - def dev(self) -> None: - return None - - @property - def local(self) -> None: - return None - - @property - def is_prerelease(self) -> bool: - return False - - @property - def is_postrelease(self) -> bool: - return False - - @property - def is_devrelease(self) -> bool: - return False - - -_legacy_version_component_re = re.compile(r"(\d+ | [a-z]+ | \.| -)", re.VERBOSE) - -_legacy_version_replacement_map = { - "pre": "c", - "preview": "c", - "-": "final-", - "rc": "c", - "dev": "@", -} - - -def _parse_version_parts(s: str) -> Iterator[str]: - for part in _legacy_version_component_re.split(s): - part = _legacy_version_replacement_map.get(part, part) - - if not part or part == ".": - continue - - if part[:1] in "0123456789": - # pad for numeric comparison - yield part.zfill(8) - else: - yield "*" + part - - # ensure that alpha/beta/candidate are before final - yield "*final" - - -def _legacy_cmpkey(version: str) -> LegacyCmpKey: - - # We hardcode an epoch of -1 here. A PEP 440 version can only have a epoch - # greater than or equal to 0. This will effectively put the LegacyVersion, - # which uses the defacto standard originally implemented by setuptools, - # as before all PEP 440 versions. - epoch = -1 - - # This scheme is taken from pkg_resources.parse_version setuptools prior to - # it's adoption of the packaging library. - parts: List[str] = [] - for part in _parse_version_parts(version.lower()): - if part.startswith("*"): - # remove "-" before a prerelease tag - if part < "*final": - while parts and parts[-1] == "*final-": - parts.pop() - - # remove trailing zeros from each series of numeric parts - while parts and parts[-1] == "00000000": - parts.pop() - - parts.append(part) - - return epoch, tuple(parts) - - -# Deliberately not anchored to the start and end of the string, to make it -# easier for 3rd party code to reuse -VERSION_PATTERN = r""" - v? - (?: - (?:(?P[0-9]+)!)? # epoch - (?P[0-9]+(?:\.[0-9]+)*) # release segment - (?P
                                          # pre-release
-            [-_\.]?
-            (?P(a|b|c|rc|alpha|beta|pre|preview))
-            [-_\.]?
-            (?P[0-9]+)?
-        )?
-        (?P                                         # post release
-            (?:-(?P[0-9]+))
-            |
-            (?:
-                [-_\.]?
-                (?Ppost|rev|r)
-                [-_\.]?
-                (?P[0-9]+)?
-            )
-        )?
-        (?P                                          # dev release
-            [-_\.]?
-            (?Pdev)
-            [-_\.]?
-            (?P[0-9]+)?
-        )?
-    )
-    (?:\+(?P[a-z0-9]+(?:[-_\.][a-z0-9]+)*))?       # local version
-"""
-
-
-class Version(_BaseVersion):
-
-    _regex = re.compile(r"^\s*" + VERSION_PATTERN + r"\s*$", re.VERBOSE | re.IGNORECASE)
-
-    def __init__(self, version: str) -> None:
-
-        # Validate the version and parse it into pieces
-        match = self._regex.search(version)
-        if not match:
-            raise InvalidVersion(f"Invalid version: '{version}'")
-
-        # Store the parsed out pieces of the version
-        self._version = _Version(
-            epoch=int(match.group("epoch")) if match.group("epoch") else 0,
-            release=tuple(int(i) for i in match.group("release").split(".")),
-            pre=_parse_letter_version(match.group("pre_l"), match.group("pre_n")),
-            post=_parse_letter_version(
-                match.group("post_l"), match.group("post_n1") or match.group("post_n2")
-            ),
-            dev=_parse_letter_version(match.group("dev_l"), match.group("dev_n")),
-            local=_parse_local_version(match.group("local")),
-        )
-
-        # Generate a key which will be used for sorting
-        self._key = _cmpkey(
-            self._version.epoch,
-            self._version.release,
-            self._version.pre,
-            self._version.post,
-            self._version.dev,
-            self._version.local,
-        )
-
-    def __repr__(self) -> str:
-        return f""
-
-    def __str__(self) -> str:
-        parts = []
-
-        # Epoch
-        if self.epoch != 0:
-            parts.append(f"{self.epoch}!")
-
-        # Release segment
-        parts.append(".".join(str(x) for x in self.release))
-
-        # Pre-release
-        if self.pre is not None:
-            parts.append("".join(str(x) for x in self.pre))
-
-        # Post-release
-        if self.post is not None:
-            parts.append(f".post{self.post}")
-
-        # Development release
-        if self.dev is not None:
-            parts.append(f".dev{self.dev}")
-
-        # Local version segment
-        if self.local is not None:
-            parts.append(f"+{self.local}")
-
-        return "".join(parts)
-
-    @property
-    def epoch(self) -> int:
-        _epoch: int = self._version.epoch
-        return _epoch
-
-    @property
-    def release(self) -> Tuple[int, ...]:
-        _release: Tuple[int, ...] = self._version.release
-        return _release
-
-    @property
-    def pre(self) -> Optional[Tuple[str, int]]:
-        _pre: Optional[Tuple[str, int]] = self._version.pre
-        return _pre
-
-    @property
-    def post(self) -> Optional[int]:
-        return self._version.post[1] if self._version.post else None
-
-    @property
-    def dev(self) -> Optional[int]:
-        return self._version.dev[1] if self._version.dev else None
-
-    @property
-    def local(self) -> Optional[str]:
-        if self._version.local:
-            return ".".join(str(x) for x in self._version.local)
-        else:
-            return None
-
-    @property
-    def public(self) -> str:
-        return str(self).split("+", 1)[0]
-
-    @property
-    def base_version(self) -> str:
-        parts = []
-
-        # Epoch
-        if self.epoch != 0:
-            parts.append(f"{self.epoch}!")
-
-        # Release segment
-        parts.append(".".join(str(x) for x in self.release))
-
-        return "".join(parts)
-
-    @property
-    def is_prerelease(self) -> bool:
-        return self.dev is not None or self.pre is not None
-
-    @property
-    def is_postrelease(self) -> bool:
-        return self.post is not None
-
-    @property
-    def is_devrelease(self) -> bool:
-        return self.dev is not None
-
-    @property
-    def major(self) -> int:
-        return self.release[0] if len(self.release) >= 1 else 0
-
-    @property
-    def minor(self) -> int:
-        return self.release[1] if len(self.release) >= 2 else 0
-
-    @property
-    def micro(self) -> int:
-        return self.release[2] if len(self.release) >= 3 else 0
-
-
-def _parse_letter_version(
-    letter: str, number: Union[str, bytes, SupportsInt]
-) -> Optional[Tuple[str, int]]:
-
-    if letter:
-        # We consider there to be an implicit 0 in a pre-release if there is
-        # not a numeral associated with it.
-        if number is None:
-            number = 0
-
-        # We normalize any letters to their lower case form
-        letter = letter.lower()
-
-        # We consider some words to be alternate spellings of other words and
-        # in those cases we want to normalize the spellings to our preferred
-        # spelling.
-        if letter == "alpha":
-            letter = "a"
-        elif letter == "beta":
-            letter = "b"
-        elif letter in ["c", "pre", "preview"]:
-            letter = "rc"
-        elif letter in ["rev", "r"]:
-            letter = "post"
-
-        return letter, int(number)
-    if not letter and number:
-        # We assume if we are given a number, but we are not given a letter
-        # then this is using the implicit post release syntax (e.g. 1.0-1)
-        letter = "post"
-
-        return letter, int(number)
-
-    return None
-
-
-_local_version_separators = re.compile(r"[\._-]")
-
-
-def _parse_local_version(local: str) -> Optional[LocalType]:
-    """
-    Takes a string like abc.1.twelve and turns it into ("abc", 1, "twelve").
-    """
-    if local is not None:
-        return tuple(
-            part.lower() if not part.isdigit() else int(part)
-            for part in _local_version_separators.split(local)
-        )
-    return None
-
-
-def _cmpkey(
-    epoch: int,
-    release: Tuple[int, ...],
-    pre: Optional[Tuple[str, int]],
-    post: Optional[Tuple[str, int]],
-    dev: Optional[Tuple[str, int]],
-    local: Optional[Tuple[SubLocalType]],
-) -> CmpKey:
-
-    # When we compare a release version, we want to compare it with all of the
-    # trailing zeros removed. So we'll use a reverse the list, drop all the now
-    # leading zeros until we come to something non zero, then take the rest
-    # re-reverse it back into the correct order and make it a tuple and use
-    # that for our sorting key.
-    _release = tuple(
-        reversed(list(itertools.dropwhile(lambda x: x == 0, reversed(release))))
-    )
-
-    # We need to "trick" the sorting algorithm to put 1.0.dev0 before 1.0a0.
-    # We'll do this by abusing the pre segment, but we _only_ want to do this
-    # if there is not a pre or a post segment. If we have one of those then
-    # the normal sorting rules will handle this case correctly.
-    if pre is None and post is None and dev is not None:
-        _pre: PrePostDevType = NegativeInfinity
-    # Versions without a pre-release (except as noted above) should sort after
-    # those with one.
-    elif pre is None:
-        _pre = Infinity
-    else:
-        _pre = pre
-
-    # Versions without a post segment should sort before those with one.
-    if post is None:
-        _post: PrePostDevType = NegativeInfinity
-
-    else:
-        _post = post
-
-    # Versions without a development segment should sort after those with one.
-    if dev is None:
-        _dev: PrePostDevType = Infinity
-
-    else:
-        _dev = dev
-
-    if local is None:
-        # Versions without a local segment should sort before those with one.
-        _local: LocalType = NegativeInfinity
-    else:
-        # Versions with a local segment need that segment parsed to implement
-        # the sorting rules in PEP440.
-        # - Alpha numeric segments sort before numeric segments
-        # - Alpha numeric segments sort lexicographically
-        # - Numeric segments sort numerically
-        # - Shorter versions sort before longer versions when the prefixes
-        #   match exactly
-        _local = tuple(
-            (i, "") if isinstance(i, int) else (NegativeInfinity, i) for i in local
-        )
-
-    return epoch, _release, _pre, _post, _dev, _local
diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/pkg_resources/__init__.py b/venv/lib/python3.11/site-packages/pip/_vendor/pkg_resources/__init__.py
deleted file mode 100644
index ad27940..0000000
--- a/venv/lib/python3.11/site-packages/pip/_vendor/pkg_resources/__init__.py
+++ /dev/null
@@ -1,3361 +0,0 @@
-"""
-Package resource API
---------------------
-
-A resource is a logical file contained within a package, or a logical
-subdirectory thereof.  The package resource API expects resource names
-to have their path parts separated with ``/``, *not* whatever the local
-path separator is.  Do not use os.path operations to manipulate resource
-names being passed into the API.
-
-The package resource API is designed to work with normal filesystem packages,
-.egg files, and unpacked .egg files.  It can also work in a limited way with
-.zip files and with custom PEP 302 loaders that support the ``get_data()``
-method.
-
-This module is deprecated. Users are directed to :mod:`importlib.resources`,
-:mod:`importlib.metadata` and :pypi:`packaging` instead.
-"""
-
-import sys
-import os
-import io
-import time
-import re
-import types
-import zipfile
-import zipimport
-import warnings
-import stat
-import functools
-import pkgutil
-import operator
-import platform
-import collections
-import plistlib
-import email.parser
-import errno
-import tempfile
-import textwrap
-import inspect
-import ntpath
-import posixpath
-import importlib
-from pkgutil import get_importer
-
-try:
-    import _imp
-except ImportError:
-    # Python 3.2 compatibility
-    import imp as _imp
-
-try:
-    FileExistsError
-except NameError:
-    FileExistsError = OSError
-
-# capture these to bypass sandboxing
-from os import utime
-
-try:
-    from os import mkdir, rename, unlink
-
-    WRITE_SUPPORT = True
-except ImportError:
-    # no write support, probably under GAE
-    WRITE_SUPPORT = False
-
-from os import open as os_open
-from os.path import isdir, split
-
-try:
-    import importlib.machinery as importlib_machinery
-
-    # access attribute to force import under delayed import mechanisms.
-    importlib_machinery.__name__
-except ImportError:
-    importlib_machinery = None
-
-from pip._internal.utils._jaraco_text import (
-    yield_lines,
-    drop_comment,
-    join_continuation,
-)
-
-from pip._vendor import platformdirs
-from pip._vendor import packaging
-
-__import__('pip._vendor.packaging.version')
-__import__('pip._vendor.packaging.specifiers')
-__import__('pip._vendor.packaging.requirements')
-__import__('pip._vendor.packaging.markers')
-__import__('pip._vendor.packaging.utils')
-
-if sys.version_info < (3, 5):
-    raise RuntimeError("Python 3.5 or later is required")
-
-# declare some globals that will be defined later to
-# satisfy the linters.
-require = None
-working_set = None
-add_activation_listener = None
-resources_stream = None
-cleanup_resources = None
-resource_dir = None
-resource_stream = None
-set_extraction_path = None
-resource_isdir = None
-resource_string = None
-iter_entry_points = None
-resource_listdir = None
-resource_filename = None
-resource_exists = None
-_distribution_finders = None
-_namespace_handlers = None
-_namespace_packages = None
-
-
-warnings.warn(
-    "pkg_resources is deprecated as an API. "
-    "See https://setuptools.pypa.io/en/latest/pkg_resources.html",
-    DeprecationWarning,
-    stacklevel=2
-)
-
-
-_PEP440_FALLBACK = re.compile(r"^v?(?P(?:[0-9]+!)?[0-9]+(?:\.[0-9]+)*)", re.I)
-
-
-class PEP440Warning(RuntimeWarning):
-    """
-    Used when there is an issue with a version or specifier not complying with
-    PEP 440.
-    """
-
-
-parse_version = packaging.version.Version
-
-
-_state_vars = {}
-
-
-def _declare_state(vartype, **kw):
-    globals().update(kw)
-    _state_vars.update(dict.fromkeys(kw, vartype))
-
-
-def __getstate__():
-    state = {}
-    g = globals()
-    for k, v in _state_vars.items():
-        state[k] = g['_sget_' + v](g[k])
-    return state
-
-
-def __setstate__(state):
-    g = globals()
-    for k, v in state.items():
-        g['_sset_' + _state_vars[k]](k, g[k], v)
-    return state
-
-
-def _sget_dict(val):
-    return val.copy()
-
-
-def _sset_dict(key, ob, state):
-    ob.clear()
-    ob.update(state)
-
-
-def _sget_object(val):
-    return val.__getstate__()
-
-
-def _sset_object(key, ob, state):
-    ob.__setstate__(state)
-
-
-_sget_none = _sset_none = lambda *args: None
-
-
-def get_supported_platform():
-    """Return this platform's maximum compatible version.
-
-    distutils.util.get_platform() normally reports the minimum version
-    of macOS that would be required to *use* extensions produced by
-    distutils.  But what we want when checking compatibility is to know the
-    version of macOS that we are *running*.  To allow usage of packages that
-    explicitly require a newer version of macOS, we must also know the
-    current version of the OS.
-
-    If this condition occurs for any other platform with a version in its
-    platform strings, this function should be extended accordingly.
-    """
-    plat = get_build_platform()
-    m = macosVersionString.match(plat)
-    if m is not None and sys.platform == "darwin":
-        try:
-            plat = 'macosx-%s-%s' % ('.'.join(_macos_vers()[:2]), m.group(3))
-        except ValueError:
-            # not macOS
-            pass
-    return plat
-
-
-__all__ = [
-    # Basic resource access and distribution/entry point discovery
-    'require',
-    'run_script',
-    'get_provider',
-    'get_distribution',
-    'load_entry_point',
-    'get_entry_map',
-    'get_entry_info',
-    'iter_entry_points',
-    'resource_string',
-    'resource_stream',
-    'resource_filename',
-    'resource_listdir',
-    'resource_exists',
-    'resource_isdir',
-    # Environmental control
-    'declare_namespace',
-    'working_set',
-    'add_activation_listener',
-    'find_distributions',
-    'set_extraction_path',
-    'cleanup_resources',
-    'get_default_cache',
-    # Primary implementation classes
-    'Environment',
-    'WorkingSet',
-    'ResourceManager',
-    'Distribution',
-    'Requirement',
-    'EntryPoint',
-    # Exceptions
-    'ResolutionError',
-    'VersionConflict',
-    'DistributionNotFound',
-    'UnknownExtra',
-    'ExtractionError',
-    # Warnings
-    'PEP440Warning',
-    # Parsing functions and string utilities
-    'parse_requirements',
-    'parse_version',
-    'safe_name',
-    'safe_version',
-    'get_platform',
-    'compatible_platforms',
-    'yield_lines',
-    'split_sections',
-    'safe_extra',
-    'to_filename',
-    'invalid_marker',
-    'evaluate_marker',
-    # filesystem utilities
-    'ensure_directory',
-    'normalize_path',
-    # Distribution "precedence" constants
-    'EGG_DIST',
-    'BINARY_DIST',
-    'SOURCE_DIST',
-    'CHECKOUT_DIST',
-    'DEVELOP_DIST',
-    # "Provider" interfaces, implementations, and registration/lookup APIs
-    'IMetadataProvider',
-    'IResourceProvider',
-    'FileMetadata',
-    'PathMetadata',
-    'EggMetadata',
-    'EmptyProvider',
-    'empty_provider',
-    'NullProvider',
-    'EggProvider',
-    'DefaultProvider',
-    'ZipProvider',
-    'register_finder',
-    'register_namespace_handler',
-    'register_loader_type',
-    'fixup_namespace_packages',
-    'get_importer',
-    # Warnings
-    'PkgResourcesDeprecationWarning',
-    # Deprecated/backward compatibility only
-    'run_main',
-    'AvailableDistributions',
-]
-
-
-class ResolutionError(Exception):
-    """Abstract base for dependency resolution errors"""
-
-    def __repr__(self):
-        return self.__class__.__name__ + repr(self.args)
-
-
-class VersionConflict(ResolutionError):
-    """
-    An already-installed version conflicts with the requested version.
-
-    Should be initialized with the installed Distribution and the requested
-    Requirement.
-    """
-
-    _template = "{self.dist} is installed but {self.req} is required"
-
-    @property
-    def dist(self):
-        return self.args[0]
-
-    @property
-    def req(self):
-        return self.args[1]
-
-    def report(self):
-        return self._template.format(**locals())
-
-    def with_context(self, required_by):
-        """
-        If required_by is non-empty, return a version of self that is a
-        ContextualVersionConflict.
-        """
-        if not required_by:
-            return self
-        args = self.args + (required_by,)
-        return ContextualVersionConflict(*args)
-
-
-class ContextualVersionConflict(VersionConflict):
-    """
-    A VersionConflict that accepts a third parameter, the set of the
-    requirements that required the installed Distribution.
-    """
-
-    _template = VersionConflict._template + ' by {self.required_by}'
-
-    @property
-    def required_by(self):
-        return self.args[2]
-
-
-class DistributionNotFound(ResolutionError):
-    """A requested distribution was not found"""
-
-    _template = (
-        "The '{self.req}' distribution was not found "
-        "and is required by {self.requirers_str}"
-    )
-
-    @property
-    def req(self):
-        return self.args[0]
-
-    @property
-    def requirers(self):
-        return self.args[1]
-
-    @property
-    def requirers_str(self):
-        if not self.requirers:
-            return 'the application'
-        return ', '.join(self.requirers)
-
-    def report(self):
-        return self._template.format(**locals())
-
-    def __str__(self):
-        return self.report()
-
-
-class UnknownExtra(ResolutionError):
-    """Distribution doesn't have an "extra feature" of the given name"""
-
-
-_provider_factories = {}
-
-PY_MAJOR = '{}.{}'.format(*sys.version_info)
-EGG_DIST = 3
-BINARY_DIST = 2
-SOURCE_DIST = 1
-CHECKOUT_DIST = 0
-DEVELOP_DIST = -1
-
-
-def register_loader_type(loader_type, provider_factory):
-    """Register `provider_factory` to make providers for `loader_type`
-
-    `loader_type` is the type or class of a PEP 302 ``module.__loader__``,
-    and `provider_factory` is a function that, passed a *module* object,
-    returns an ``IResourceProvider`` for that module.
-    """
-    _provider_factories[loader_type] = provider_factory
-
-
-def get_provider(moduleOrReq):
-    """Return an IResourceProvider for the named module or requirement"""
-    if isinstance(moduleOrReq, Requirement):
-        return working_set.find(moduleOrReq) or require(str(moduleOrReq))[0]
-    try:
-        module = sys.modules[moduleOrReq]
-    except KeyError:
-        __import__(moduleOrReq)
-        module = sys.modules[moduleOrReq]
-    loader = getattr(module, '__loader__', None)
-    return _find_adapter(_provider_factories, loader)(module)
-
-
-def _macos_vers(_cache=[]):
-    if not _cache:
-        version = platform.mac_ver()[0]
-        # fallback for MacPorts
-        if version == '':
-            plist = '/System/Library/CoreServices/SystemVersion.plist'
-            if os.path.exists(plist):
-                if hasattr(plistlib, 'readPlist'):
-                    plist_content = plistlib.readPlist(plist)
-                    if 'ProductVersion' in plist_content:
-                        version = plist_content['ProductVersion']
-
-        _cache.append(version.split('.'))
-    return _cache[0]
-
-
-def _macos_arch(machine):
-    return {'PowerPC': 'ppc', 'Power_Macintosh': 'ppc'}.get(machine, machine)
-
-
-def get_build_platform():
-    """Return this platform's string for platform-specific distributions
-
-    XXX Currently this is the same as ``distutils.util.get_platform()``, but it
-    needs some hacks for Linux and macOS.
-    """
-    from sysconfig import get_platform
-
-    plat = get_platform()
-    if sys.platform == "darwin" and not plat.startswith('macosx-'):
-        try:
-            version = _macos_vers()
-            machine = os.uname()[4].replace(" ", "_")
-            return "macosx-%d.%d-%s" % (
-                int(version[0]),
-                int(version[1]),
-                _macos_arch(machine),
-            )
-        except ValueError:
-            # if someone is running a non-Mac darwin system, this will fall
-            # through to the default implementation
-            pass
-    return plat
-
-
-macosVersionString = re.compile(r"macosx-(\d+)\.(\d+)-(.*)")
-darwinVersionString = re.compile(r"darwin-(\d+)\.(\d+)\.(\d+)-(.*)")
-# XXX backward compat
-get_platform = get_build_platform
-
-
-def compatible_platforms(provided, required):
-    """Can code for the `provided` platform run on the `required` platform?
-
-    Returns true if either platform is ``None``, or the platforms are equal.
-
-    XXX Needs compatibility checks for Linux and other unixy OSes.
-    """
-    if provided is None or required is None or provided == required:
-        # easy case
-        return True
-
-    # macOS special cases
-    reqMac = macosVersionString.match(required)
-    if reqMac:
-        provMac = macosVersionString.match(provided)
-
-        # is this a Mac package?
-        if not provMac:
-            # this is backwards compatibility for packages built before
-            # setuptools 0.6. All packages built after this point will
-            # use the new macOS designation.
-            provDarwin = darwinVersionString.match(provided)
-            if provDarwin:
-                dversion = int(provDarwin.group(1))
-                macosversion = "%s.%s" % (reqMac.group(1), reqMac.group(2))
-                if (
-                    dversion == 7
-                    and macosversion >= "10.3"
-                    or dversion == 8
-                    and macosversion >= "10.4"
-                ):
-                    return True
-            # egg isn't macOS or legacy darwin
-            return False
-
-        # are they the same major version and machine type?
-        if provMac.group(1) != reqMac.group(1) or provMac.group(3) != reqMac.group(3):
-            return False
-
-        # is the required OS major update >= the provided one?
-        if int(provMac.group(2)) > int(reqMac.group(2)):
-            return False
-
-        return True
-
-    # XXX Linux and other platforms' special cases should go here
-    return False
-
-
-def run_script(dist_spec, script_name):
-    """Locate distribution `dist_spec` and run its `script_name` script"""
-    ns = sys._getframe(1).f_globals
-    name = ns['__name__']
-    ns.clear()
-    ns['__name__'] = name
-    require(dist_spec)[0].run_script(script_name, ns)
-
-
-# backward compatibility
-run_main = run_script
-
-
-def get_distribution(dist):
-    """Return a current distribution object for a Requirement or string"""
-    if isinstance(dist, str):
-        dist = Requirement.parse(dist)
-    if isinstance(dist, Requirement):
-        dist = get_provider(dist)
-    if not isinstance(dist, Distribution):
-        raise TypeError("Expected string, Requirement, or Distribution", dist)
-    return dist
-
-
-def load_entry_point(dist, group, name):
-    """Return `name` entry point of `group` for `dist` or raise ImportError"""
-    return get_distribution(dist).load_entry_point(group, name)
-
-
-def get_entry_map(dist, group=None):
-    """Return the entry point map for `group`, or the full entry map"""
-    return get_distribution(dist).get_entry_map(group)
-
-
-def get_entry_info(dist, group, name):
-    """Return the EntryPoint object for `group`+`name`, or ``None``"""
-    return get_distribution(dist).get_entry_info(group, name)
-
-
-class IMetadataProvider:
-    def has_metadata(name):
-        """Does the package's distribution contain the named metadata?"""
-
-    def get_metadata(name):
-        """The named metadata resource as a string"""
-
-    def get_metadata_lines(name):
-        """Yield named metadata resource as list of non-blank non-comment lines
-
-        Leading and trailing whitespace is stripped from each line, and lines
-        with ``#`` as the first non-blank character are omitted."""
-
-    def metadata_isdir(name):
-        """Is the named metadata a directory?  (like ``os.path.isdir()``)"""
-
-    def metadata_listdir(name):
-        """List of metadata names in the directory (like ``os.listdir()``)"""
-
-    def run_script(script_name, namespace):
-        """Execute the named script in the supplied namespace dictionary"""
-
-
-class IResourceProvider(IMetadataProvider):
-    """An object that provides access to package resources"""
-
-    def get_resource_filename(manager, resource_name):
-        """Return a true filesystem path for `resource_name`
-
-        `manager` must be an ``IResourceManager``"""
-
-    def get_resource_stream(manager, resource_name):
-        """Return a readable file-like object for `resource_name`
-
-        `manager` must be an ``IResourceManager``"""
-
-    def get_resource_string(manager, resource_name):
-        """Return a string containing the contents of `resource_name`
-
-        `manager` must be an ``IResourceManager``"""
-
-    def has_resource(resource_name):
-        """Does the package contain the named resource?"""
-
-    def resource_isdir(resource_name):
-        """Is the named resource a directory?  (like ``os.path.isdir()``)"""
-
-    def resource_listdir(resource_name):
-        """List of resource names in the directory (like ``os.listdir()``)"""
-
-
-class WorkingSet:
-    """A collection of active distributions on sys.path (or a similar list)"""
-
-    def __init__(self, entries=None):
-        """Create working set from list of path entries (default=sys.path)"""
-        self.entries = []
-        self.entry_keys = {}
-        self.by_key = {}
-        self.normalized_to_canonical_keys = {}
-        self.callbacks = []
-
-        if entries is None:
-            entries = sys.path
-
-        for entry in entries:
-            self.add_entry(entry)
-
-    @classmethod
-    def _build_master(cls):
-        """
-        Prepare the master working set.
-        """
-        ws = cls()
-        try:
-            from __main__ import __requires__
-        except ImportError:
-            # The main program does not list any requirements
-            return ws
-
-        # ensure the requirements are met
-        try:
-            ws.require(__requires__)
-        except VersionConflict:
-            return cls._build_from_requirements(__requires__)
-
-        return ws
-
-    @classmethod
-    def _build_from_requirements(cls, req_spec):
-        """
-        Build a working set from a requirement spec. Rewrites sys.path.
-        """
-        # try it without defaults already on sys.path
-        # by starting with an empty path
-        ws = cls([])
-        reqs = parse_requirements(req_spec)
-        dists = ws.resolve(reqs, Environment())
-        for dist in dists:
-            ws.add(dist)
-
-        # add any missing entries from sys.path
-        for entry in sys.path:
-            if entry not in ws.entries:
-                ws.add_entry(entry)
-
-        # then copy back to sys.path
-        sys.path[:] = ws.entries
-        return ws
-
-    def add_entry(self, entry):
-        """Add a path item to ``.entries``, finding any distributions on it
-
-        ``find_distributions(entry, True)`` is used to find distributions
-        corresponding to the path entry, and they are added.  `entry` is
-        always appended to ``.entries``, even if it is already present.
-        (This is because ``sys.path`` can contain the same value more than
-        once, and the ``.entries`` of the ``sys.path`` WorkingSet should always
-        equal ``sys.path``.)
-        """
-        self.entry_keys.setdefault(entry, [])
-        self.entries.append(entry)
-        for dist in find_distributions(entry, True):
-            self.add(dist, entry, False)
-
-    def __contains__(self, dist):
-        """True if `dist` is the active distribution for its project"""
-        return self.by_key.get(dist.key) == dist
-
-    def find(self, req):
-        """Find a distribution matching requirement `req`
-
-        If there is an active distribution for the requested project, this
-        returns it as long as it meets the version requirement specified by
-        `req`.  But, if there is an active distribution for the project and it
-        does *not* meet the `req` requirement, ``VersionConflict`` is raised.
-        If there is no active distribution for the requested project, ``None``
-        is returned.
-        """
-        dist = self.by_key.get(req.key)
-
-        if dist is None:
-            canonical_key = self.normalized_to_canonical_keys.get(req.key)
-
-            if canonical_key is not None:
-                req.key = canonical_key
-                dist = self.by_key.get(canonical_key)
-
-        if dist is not None and dist not in req:
-            # XXX add more info
-            raise VersionConflict(dist, req)
-        return dist
-
-    def iter_entry_points(self, group, name=None):
-        """Yield entry point objects from `group` matching `name`
-
-        If `name` is None, yields all entry points in `group` from all
-        distributions in the working set, otherwise only ones matching
-        both `group` and `name` are yielded (in distribution order).
-        """
-        return (
-            entry
-            for dist in self
-            for entry in dist.get_entry_map(group).values()
-            if name is None or name == entry.name
-        )
-
-    def run_script(self, requires, script_name):
-        """Locate distribution for `requires` and run `script_name` script"""
-        ns = sys._getframe(1).f_globals
-        name = ns['__name__']
-        ns.clear()
-        ns['__name__'] = name
-        self.require(requires)[0].run_script(script_name, ns)
-
-    def __iter__(self):
-        """Yield distributions for non-duplicate projects in the working set
-
-        The yield order is the order in which the items' path entries were
-        added to the working set.
-        """
-        seen = {}
-        for item in self.entries:
-            if item not in self.entry_keys:
-                # workaround a cache issue
-                continue
-
-            for key in self.entry_keys[item]:
-                if key not in seen:
-                    seen[key] = 1
-                    yield self.by_key[key]
-
-    def add(self, dist, entry=None, insert=True, replace=False):
-        """Add `dist` to working set, associated with `entry`
-
-        If `entry` is unspecified, it defaults to the ``.location`` of `dist`.
-        On exit from this routine, `entry` is added to the end of the working
-        set's ``.entries`` (if it wasn't already present).
-
-        `dist` is only added to the working set if it's for a project that
-        doesn't already have a distribution in the set, unless `replace=True`.
-        If it's added, any callbacks registered with the ``subscribe()`` method
-        will be called.
-        """
-        if insert:
-            dist.insert_on(self.entries, entry, replace=replace)
-
-        if entry is None:
-            entry = dist.location
-        keys = self.entry_keys.setdefault(entry, [])
-        keys2 = self.entry_keys.setdefault(dist.location, [])
-        if not replace and dist.key in self.by_key:
-            # ignore hidden distros
-            return
-
-        self.by_key[dist.key] = dist
-        normalized_name = packaging.utils.canonicalize_name(dist.key)
-        self.normalized_to_canonical_keys[normalized_name] = dist.key
-        if dist.key not in keys:
-            keys.append(dist.key)
-        if dist.key not in keys2:
-            keys2.append(dist.key)
-        self._added_new(dist)
-
-    def resolve(
-        self,
-        requirements,
-        env=None,
-        installer=None,
-        replace_conflicting=False,
-        extras=None,
-    ):
-        """List all distributions needed to (recursively) meet `requirements`
-
-        `requirements` must be a sequence of ``Requirement`` objects.  `env`,
-        if supplied, should be an ``Environment`` instance.  If
-        not supplied, it defaults to all distributions available within any
-        entry or distribution in the working set.  `installer`, if supplied,
-        will be invoked with each requirement that cannot be met by an
-        already-installed distribution; it should return a ``Distribution`` or
-        ``None``.
-
-        Unless `replace_conflicting=True`, raises a VersionConflict exception
-        if
-        any requirements are found on the path that have the correct name but
-        the wrong version.  Otherwise, if an `installer` is supplied it will be
-        invoked to obtain the correct version of the requirement and activate
-        it.
-
-        `extras` is a list of the extras to be used with these requirements.
-        This is important because extra requirements may look like `my_req;
-        extra = "my_extra"`, which would otherwise be interpreted as a purely
-        optional requirement.  Instead, we want to be able to assert that these
-        requirements are truly required.
-        """
-
-        # set up the stack
-        requirements = list(requirements)[::-1]
-        # set of processed requirements
-        processed = {}
-        # key -> dist
-        best = {}
-        to_activate = []
-
-        req_extras = _ReqExtras()
-
-        # Mapping of requirement to set of distributions that required it;
-        # useful for reporting info about conflicts.
-        required_by = collections.defaultdict(set)
-
-        while requirements:
-            # process dependencies breadth-first
-            req = requirements.pop(0)
-            if req in processed:
-                # Ignore cyclic or redundant dependencies
-                continue
-
-            if not req_extras.markers_pass(req, extras):
-                continue
-
-            dist = self._resolve_dist(
-                req, best, replace_conflicting, env, installer, required_by, to_activate
-            )
-
-            # push the new requirements onto the stack
-            new_requirements = dist.requires(req.extras)[::-1]
-            requirements.extend(new_requirements)
-
-            # Register the new requirements needed by req
-            for new_requirement in new_requirements:
-                required_by[new_requirement].add(req.project_name)
-                req_extras[new_requirement] = req.extras
-
-            processed[req] = True
-
-        # return list of distros to activate
-        return to_activate
-
-    def _resolve_dist(
-        self, req, best, replace_conflicting, env, installer, required_by, to_activate
-    ):
-        dist = best.get(req.key)
-        if dist is None:
-            # Find the best distribution and add it to the map
-            dist = self.by_key.get(req.key)
-            if dist is None or (dist not in req and replace_conflicting):
-                ws = self
-                if env is None:
-                    if dist is None:
-                        env = Environment(self.entries)
-                    else:
-                        # Use an empty environment and workingset to avoid
-                        # any further conflicts with the conflicting
-                        # distribution
-                        env = Environment([])
-                        ws = WorkingSet([])
-                dist = best[req.key] = env.best_match(
-                    req, ws, installer, replace_conflicting=replace_conflicting
-                )
-                if dist is None:
-                    requirers = required_by.get(req, None)
-                    raise DistributionNotFound(req, requirers)
-            to_activate.append(dist)
-        if dist not in req:
-            # Oops, the "best" so far conflicts with a dependency
-            dependent_req = required_by[req]
-            raise VersionConflict(dist, req).with_context(dependent_req)
-        return dist
-
-    def find_plugins(self, plugin_env, full_env=None, installer=None, fallback=True):
-        """Find all activatable distributions in `plugin_env`
-
-        Example usage::
-
-            distributions, errors = working_set.find_plugins(
-                Environment(plugin_dirlist)
-            )
-            # add plugins+libs to sys.path
-            map(working_set.add, distributions)
-            # display errors
-            print('Could not load', errors)
-
-        The `plugin_env` should be an ``Environment`` instance that contains
-        only distributions that are in the project's "plugin directory" or
-        directories. The `full_env`, if supplied, should be an ``Environment``
-        contains all currently-available distributions.  If `full_env` is not
-        supplied, one is created automatically from the ``WorkingSet`` this
-        method is called on, which will typically mean that every directory on
-        ``sys.path`` will be scanned for distributions.
-
-        `installer` is a standard installer callback as used by the
-        ``resolve()`` method. The `fallback` flag indicates whether we should
-        attempt to resolve older versions of a plugin if the newest version
-        cannot be resolved.
-
-        This method returns a 2-tuple: (`distributions`, `error_info`), where
-        `distributions` is a list of the distributions found in `plugin_env`
-        that were loadable, along with any other distributions that are needed
-        to resolve their dependencies.  `error_info` is a dictionary mapping
-        unloadable plugin distributions to an exception instance describing the
-        error that occurred. Usually this will be a ``DistributionNotFound`` or
-        ``VersionConflict`` instance.
-        """
-
-        plugin_projects = list(plugin_env)
-        # scan project names in alphabetic order
-        plugin_projects.sort()
-
-        error_info = {}
-        distributions = {}
-
-        if full_env is None:
-            env = Environment(self.entries)
-            env += plugin_env
-        else:
-            env = full_env + plugin_env
-
-        shadow_set = self.__class__([])
-        # put all our entries in shadow_set
-        list(map(shadow_set.add, self))
-
-        for project_name in plugin_projects:
-            for dist in plugin_env[project_name]:
-                req = [dist.as_requirement()]
-
-                try:
-                    resolvees = shadow_set.resolve(req, env, installer)
-
-                except ResolutionError as v:
-                    # save error info
-                    error_info[dist] = v
-                    if fallback:
-                        # try the next older version of project
-                        continue
-                    else:
-                        # give up on this project, keep going
-                        break
-
-                else:
-                    list(map(shadow_set.add, resolvees))
-                    distributions.update(dict.fromkeys(resolvees))
-
-                    # success, no need to try any more versions of this project
-                    break
-
-        distributions = list(distributions)
-        distributions.sort()
-
-        return distributions, error_info
-
-    def require(self, *requirements):
-        """Ensure that distributions matching `requirements` are activated
-
-        `requirements` must be a string or a (possibly-nested) sequence
-        thereof, specifying the distributions and versions required.  The
-        return value is a sequence of the distributions that needed to be
-        activated to fulfill the requirements; all relevant distributions are
-        included, even if they were already activated in this working set.
-        """
-        needed = self.resolve(parse_requirements(requirements))
-
-        for dist in needed:
-            self.add(dist)
-
-        return needed
-
-    def subscribe(self, callback, existing=True):
-        """Invoke `callback` for all distributions
-
-        If `existing=True` (default),
-        call on all existing ones, as well.
-        """
-        if callback in self.callbacks:
-            return
-        self.callbacks.append(callback)
-        if not existing:
-            return
-        for dist in self:
-            callback(dist)
-
-    def _added_new(self, dist):
-        for callback in self.callbacks:
-            callback(dist)
-
-    def __getstate__(self):
-        return (
-            self.entries[:],
-            self.entry_keys.copy(),
-            self.by_key.copy(),
-            self.normalized_to_canonical_keys.copy(),
-            self.callbacks[:],
-        )
-
-    def __setstate__(self, e_k_b_n_c):
-        entries, keys, by_key, normalized_to_canonical_keys, callbacks = e_k_b_n_c
-        self.entries = entries[:]
-        self.entry_keys = keys.copy()
-        self.by_key = by_key.copy()
-        self.normalized_to_canonical_keys = normalized_to_canonical_keys.copy()
-        self.callbacks = callbacks[:]
-
-
-class _ReqExtras(dict):
-    """
-    Map each requirement to the extras that demanded it.
-    """
-
-    def markers_pass(self, req, extras=None):
-        """
-        Evaluate markers for req against each extra that
-        demanded it.
-
-        Return False if the req has a marker and fails
-        evaluation. Otherwise, return True.
-        """
-        extra_evals = (
-            req.marker.evaluate({'extra': extra})
-            for extra in self.get(req, ()) + (extras or (None,))
-        )
-        return not req.marker or any(extra_evals)
-
-
-class Environment:
-    """Searchable snapshot of distributions on a search path"""
-
-    def __init__(
-        self, search_path=None, platform=get_supported_platform(), python=PY_MAJOR
-    ):
-        """Snapshot distributions available on a search path
-
-        Any distributions found on `search_path` are added to the environment.
-        `search_path` should be a sequence of ``sys.path`` items.  If not
-        supplied, ``sys.path`` is used.
-
-        `platform` is an optional string specifying the name of the platform
-        that platform-specific distributions must be compatible with.  If
-        unspecified, it defaults to the current platform.  `python` is an
-        optional string naming the desired version of Python (e.g. ``'3.6'``);
-        it defaults to the current version.
-
-        You may explicitly set `platform` (and/or `python`) to ``None`` if you
-        wish to map *all* distributions, not just those compatible with the
-        running platform or Python version.
-        """
-        self._distmap = {}
-        self.platform = platform
-        self.python = python
-        self.scan(search_path)
-
-    def can_add(self, dist):
-        """Is distribution `dist` acceptable for this environment?
-
-        The distribution must match the platform and python version
-        requirements specified when this environment was created, or False
-        is returned.
-        """
-        py_compat = (
-            self.python is None
-            or dist.py_version is None
-            or dist.py_version == self.python
-        )
-        return py_compat and compatible_platforms(dist.platform, self.platform)
-
-    def remove(self, dist):
-        """Remove `dist` from the environment"""
-        self._distmap[dist.key].remove(dist)
-
-    def scan(self, search_path=None):
-        """Scan `search_path` for distributions usable in this environment
-
-        Any distributions found are added to the environment.
-        `search_path` should be a sequence of ``sys.path`` items.  If not
-        supplied, ``sys.path`` is used.  Only distributions conforming to
-        the platform/python version defined at initialization are added.
-        """
-        if search_path is None:
-            search_path = sys.path
-
-        for item in search_path:
-            for dist in find_distributions(item):
-                self.add(dist)
-
-    def __getitem__(self, project_name):
-        """Return a newest-to-oldest list of distributions for `project_name`
-
-        Uses case-insensitive `project_name` comparison, assuming all the
-        project's distributions use their project's name converted to all
-        lowercase as their key.
-
-        """
-        distribution_key = project_name.lower()
-        return self._distmap.get(distribution_key, [])
-
-    def add(self, dist):
-        """Add `dist` if we ``can_add()`` it and it has not already been added"""
-        if self.can_add(dist) and dist.has_version():
-            dists = self._distmap.setdefault(dist.key, [])
-            if dist not in dists:
-                dists.append(dist)
-                dists.sort(key=operator.attrgetter('hashcmp'), reverse=True)
-
-    def best_match(self, req, working_set, installer=None, replace_conflicting=False):
-        """Find distribution best matching `req` and usable on `working_set`
-
-        This calls the ``find(req)`` method of the `working_set` to see if a
-        suitable distribution is already active.  (This may raise
-        ``VersionConflict`` if an unsuitable version of the project is already
-        active in the specified `working_set`.)  If a suitable distribution
-        isn't active, this method returns the newest distribution in the
-        environment that meets the ``Requirement`` in `req`.  If no suitable
-        distribution is found, and `installer` is supplied, then the result of
-        calling the environment's ``obtain(req, installer)`` method will be
-        returned.
-        """
-        try:
-            dist = working_set.find(req)
-        except VersionConflict:
-            if not replace_conflicting:
-                raise
-            dist = None
-        if dist is not None:
-            return dist
-        for dist in self[req.key]:
-            if dist in req:
-                return dist
-        # try to download/install
-        return self.obtain(req, installer)
-
-    def obtain(self, requirement, installer=None):
-        """Obtain a distribution matching `requirement` (e.g. via download)
-
-        Obtain a distro that matches requirement (e.g. via download).  In the
-        base ``Environment`` class, this routine just returns
-        ``installer(requirement)``, unless `installer` is None, in which case
-        None is returned instead.  This method is a hook that allows subclasses
-        to attempt other ways of obtaining a distribution before falling back
-        to the `installer` argument."""
-        if installer is not None:
-            return installer(requirement)
-
-    def __iter__(self):
-        """Yield the unique project names of the available distributions"""
-        for key in self._distmap.keys():
-            if self[key]:
-                yield key
-
-    def __iadd__(self, other):
-        """In-place addition of a distribution or environment"""
-        if isinstance(other, Distribution):
-            self.add(other)
-        elif isinstance(other, Environment):
-            for project in other:
-                for dist in other[project]:
-                    self.add(dist)
-        else:
-            raise TypeError("Can't add %r to environment" % (other,))
-        return self
-
-    def __add__(self, other):
-        """Add an environment or distribution to an environment"""
-        new = self.__class__([], platform=None, python=None)
-        for env in self, other:
-            new += env
-        return new
-
-
-# XXX backward compatibility
-AvailableDistributions = Environment
-
-
-class ExtractionError(RuntimeError):
-    """An error occurred extracting a resource
-
-    The following attributes are available from instances of this exception:
-
-    manager
-        The resource manager that raised this exception
-
-    cache_path
-        The base directory for resource extraction
-
-    original_error
-        The exception instance that caused extraction to fail
-    """
-
-
-class ResourceManager:
-    """Manage resource extraction and packages"""
-
-    extraction_path = None
-
-    def __init__(self):
-        self.cached_files = {}
-
-    def resource_exists(self, package_or_requirement, resource_name):
-        """Does the named resource exist?"""
-        return get_provider(package_or_requirement).has_resource(resource_name)
-
-    def resource_isdir(self, package_or_requirement, resource_name):
-        """Is the named resource an existing directory?"""
-        return get_provider(package_or_requirement).resource_isdir(resource_name)
-
-    def resource_filename(self, package_or_requirement, resource_name):
-        """Return a true filesystem path for specified resource"""
-        return get_provider(package_or_requirement).get_resource_filename(
-            self, resource_name
-        )
-
-    def resource_stream(self, package_or_requirement, resource_name):
-        """Return a readable file-like object for specified resource"""
-        return get_provider(package_or_requirement).get_resource_stream(
-            self, resource_name
-        )
-
-    def resource_string(self, package_or_requirement, resource_name):
-        """Return specified resource as a string"""
-        return get_provider(package_or_requirement).get_resource_string(
-            self, resource_name
-        )
-
-    def resource_listdir(self, package_or_requirement, resource_name):
-        """List the contents of the named resource directory"""
-        return get_provider(package_or_requirement).resource_listdir(resource_name)
-
-    def extraction_error(self):
-        """Give an error message for problems extracting file(s)"""
-
-        old_exc = sys.exc_info()[1]
-        cache_path = self.extraction_path or get_default_cache()
-
-        tmpl = textwrap.dedent(
-            """
-            Can't extract file(s) to egg cache
-
-            The following error occurred while trying to extract file(s)
-            to the Python egg cache:
-
-              {old_exc}
-
-            The Python egg cache directory is currently set to:
-
-              {cache_path}
-
-            Perhaps your account does not have write access to this directory?
-            You can change the cache directory by setting the PYTHON_EGG_CACHE
-            environment variable to point to an accessible directory.
-            """
-        ).lstrip()
-        err = ExtractionError(tmpl.format(**locals()))
-        err.manager = self
-        err.cache_path = cache_path
-        err.original_error = old_exc
-        raise err
-
-    def get_cache_path(self, archive_name, names=()):
-        """Return absolute location in cache for `archive_name` and `names`
-
-        The parent directory of the resulting path will be created if it does
-        not already exist.  `archive_name` should be the base filename of the
-        enclosing egg (which may not be the name of the enclosing zipfile!),
-        including its ".egg" extension.  `names`, if provided, should be a
-        sequence of path name parts "under" the egg's extraction location.
-
-        This method should only be called by resource providers that need to
-        obtain an extraction location, and only for names they intend to
-        extract, as it tracks the generated names for possible cleanup later.
-        """
-        extract_path = self.extraction_path or get_default_cache()
-        target_path = os.path.join(extract_path, archive_name + '-tmp', *names)
-        try:
-            _bypass_ensure_directory(target_path)
-        except Exception:
-            self.extraction_error()
-
-        self._warn_unsafe_extraction_path(extract_path)
-
-        self.cached_files[target_path] = 1
-        return target_path
-
-    @staticmethod
-    def _warn_unsafe_extraction_path(path):
-        """
-        If the default extraction path is overridden and set to an insecure
-        location, such as /tmp, it opens up an opportunity for an attacker to
-        replace an extracted file with an unauthorized payload. Warn the user
-        if a known insecure location is used.
-
-        See Distribute #375 for more details.
-        """
-        if os.name == 'nt' and not path.startswith(os.environ['windir']):
-            # On Windows, permissions are generally restrictive by default
-            #  and temp directories are not writable by other users, so
-            #  bypass the warning.
-            return
-        mode = os.stat(path).st_mode
-        if mode & stat.S_IWOTH or mode & stat.S_IWGRP:
-            msg = (
-                "Extraction path is writable by group/others "
-                "and vulnerable to attack when "
-                "used with get_resource_filename ({path}). "
-                "Consider a more secure "
-                "location (set with .set_extraction_path or the "
-                "PYTHON_EGG_CACHE environment variable)."
-            ).format(**locals())
-            warnings.warn(msg, UserWarning)
-
-    def postprocess(self, tempname, filename):
-        """Perform any platform-specific postprocessing of `tempname`
-
-        This is where Mac header rewrites should be done; other platforms don't
-        have anything special they should do.
-
-        Resource providers should call this method ONLY after successfully
-        extracting a compressed resource.  They must NOT call it on resources
-        that are already in the filesystem.
-
-        `tempname` is the current (temporary) name of the file, and `filename`
-        is the name it will be renamed to by the caller after this routine
-        returns.
-        """
-
-        if os.name == 'posix':
-            # Make the resource executable
-            mode = ((os.stat(tempname).st_mode) | 0o555) & 0o7777
-            os.chmod(tempname, mode)
-
-    def set_extraction_path(self, path):
-        """Set the base path where resources will be extracted to, if needed.
-
-        If you do not call this routine before any extractions take place, the
-        path defaults to the return value of ``get_default_cache()``.  (Which
-        is based on the ``PYTHON_EGG_CACHE`` environment variable, with various
-        platform-specific fallbacks.  See that routine's documentation for more
-        details.)
-
-        Resources are extracted to subdirectories of this path based upon
-        information given by the ``IResourceProvider``.  You may set this to a
-        temporary directory, but then you must call ``cleanup_resources()`` to
-        delete the extracted files when done.  There is no guarantee that
-        ``cleanup_resources()`` will be able to remove all extracted files.
-
-        (Note: you may not change the extraction path for a given resource
-        manager once resources have been extracted, unless you first call
-        ``cleanup_resources()``.)
-        """
-        if self.cached_files:
-            raise ValueError("Can't change extraction path, files already extracted")
-
-        self.extraction_path = path
-
-    def cleanup_resources(self, force=False):
-        """
-        Delete all extracted resource files and directories, returning a list
-        of the file and directory names that could not be successfully removed.
-        This function does not have any concurrency protection, so it should
-        generally only be called when the extraction path is a temporary
-        directory exclusive to a single process.  This method is not
-        automatically called; you must call it explicitly or register it as an
-        ``atexit`` function if you wish to ensure cleanup of a temporary
-        directory used for extractions.
-        """
-        # XXX
-
-
-def get_default_cache():
-    """
-    Return the ``PYTHON_EGG_CACHE`` environment variable
-    or a platform-relevant user cache dir for an app
-    named "Python-Eggs".
-    """
-    return os.environ.get('PYTHON_EGG_CACHE') or platformdirs.user_cache_dir(
-        appname='Python-Eggs'
-    )
-
-
-def safe_name(name):
-    """Convert an arbitrary string to a standard distribution name
-
-    Any runs of non-alphanumeric/. characters are replaced with a single '-'.
-    """
-    return re.sub('[^A-Za-z0-9.]+', '-', name)
-
-
-def safe_version(version):
-    """
-    Convert an arbitrary string to a standard version string
-    """
-    try:
-        # normalize the version
-        return str(packaging.version.Version(version))
-    except packaging.version.InvalidVersion:
-        version = version.replace(' ', '.')
-        return re.sub('[^A-Za-z0-9.]+', '-', version)
-
-
-def _forgiving_version(version):
-    """Fallback when ``safe_version`` is not safe enough
-    >>> parse_version(_forgiving_version('0.23ubuntu1'))
-    
-    >>> parse_version(_forgiving_version('0.23-'))
-    
-    >>> parse_version(_forgiving_version('0.-_'))
-    
-    >>> parse_version(_forgiving_version('42.+?1'))
-    
-    >>> parse_version(_forgiving_version('hello world'))
-    
-    """
-    version = version.replace(' ', '.')
-    match = _PEP440_FALLBACK.search(version)
-    if match:
-        safe = match["safe"]
-        rest = version[len(safe):]
-    else:
-        safe = "0"
-        rest = version
-    local = f"sanitized.{_safe_segment(rest)}".strip(".")
-    return f"{safe}.dev0+{local}"
-
-
-def _safe_segment(segment):
-    """Convert an arbitrary string into a safe segment"""
-    segment = re.sub('[^A-Za-z0-9.]+', '-', segment)
-    segment = re.sub('-[^A-Za-z0-9]+', '-', segment)
-    return re.sub(r'\.[^A-Za-z0-9]+', '.', segment).strip(".-")
-
-
-def safe_extra(extra):
-    """Convert an arbitrary string to a standard 'extra' name
-
-    Any runs of non-alphanumeric characters are replaced with a single '_',
-    and the result is always lowercased.
-    """
-    return re.sub('[^A-Za-z0-9.-]+', '_', extra).lower()
-
-
-def to_filename(name):
-    """Convert a project or version name to its filename-escaped form
-
-    Any '-' characters are currently replaced with '_'.
-    """
-    return name.replace('-', '_')
-
-
-def invalid_marker(text):
-    """
-    Validate text as a PEP 508 environment marker; return an exception
-    if invalid or False otherwise.
-    """
-    try:
-        evaluate_marker(text)
-    except SyntaxError as e:
-        e.filename = None
-        e.lineno = None
-        return e
-    return False
-
-
-def evaluate_marker(text, extra=None):
-    """
-    Evaluate a PEP 508 environment marker.
-    Return a boolean indicating the marker result in this environment.
-    Raise SyntaxError if marker is invalid.
-
-    This implementation uses the 'pyparsing' module.
-    """
-    try:
-        marker = packaging.markers.Marker(text)
-        return marker.evaluate()
-    except packaging.markers.InvalidMarker as e:
-        raise SyntaxError(e) from e
-
-
-class NullProvider:
-    """Try to implement resources and metadata for arbitrary PEP 302 loaders"""
-
-    egg_name = None
-    egg_info = None
-    loader = None
-
-    def __init__(self, module):
-        self.loader = getattr(module, '__loader__', None)
-        self.module_path = os.path.dirname(getattr(module, '__file__', ''))
-
-    def get_resource_filename(self, manager, resource_name):
-        return self._fn(self.module_path, resource_name)
-
-    def get_resource_stream(self, manager, resource_name):
-        return io.BytesIO(self.get_resource_string(manager, resource_name))
-
-    def get_resource_string(self, manager, resource_name):
-        return self._get(self._fn(self.module_path, resource_name))
-
-    def has_resource(self, resource_name):
-        return self._has(self._fn(self.module_path, resource_name))
-
-    def _get_metadata_path(self, name):
-        return self._fn(self.egg_info, name)
-
-    def has_metadata(self, name):
-        if not self.egg_info:
-            return self.egg_info
-
-        path = self._get_metadata_path(name)
-        return self._has(path)
-
-    def get_metadata(self, name):
-        if not self.egg_info:
-            return ""
-        path = self._get_metadata_path(name)
-        value = self._get(path)
-        try:
-            return value.decode('utf-8')
-        except UnicodeDecodeError as exc:
-            # Include the path in the error message to simplify
-            # troubleshooting, and without changing the exception type.
-            exc.reason += ' in {} file at path: {}'.format(name, path)
-            raise
-
-    def get_metadata_lines(self, name):
-        return yield_lines(self.get_metadata(name))
-
-    def resource_isdir(self, resource_name):
-        return self._isdir(self._fn(self.module_path, resource_name))
-
-    def metadata_isdir(self, name):
-        return self.egg_info and self._isdir(self._fn(self.egg_info, name))
-
-    def resource_listdir(self, resource_name):
-        return self._listdir(self._fn(self.module_path, resource_name))
-
-    def metadata_listdir(self, name):
-        if self.egg_info:
-            return self._listdir(self._fn(self.egg_info, name))
-        return []
-
-    def run_script(self, script_name, namespace):
-        script = 'scripts/' + script_name
-        if not self.has_metadata(script):
-            raise ResolutionError(
-                "Script {script!r} not found in metadata at {self.egg_info!r}".format(
-                    **locals()
-                ),
-            )
-        script_text = self.get_metadata(script).replace('\r\n', '\n')
-        script_text = script_text.replace('\r', '\n')
-        script_filename = self._fn(self.egg_info, script)
-        namespace['__file__'] = script_filename
-        if os.path.exists(script_filename):
-            with open(script_filename) as fid:
-                source = fid.read()
-            code = compile(source, script_filename, 'exec')
-            exec(code, namespace, namespace)
-        else:
-            from linecache import cache
-
-            cache[script_filename] = (
-                len(script_text),
-                0,
-                script_text.split('\n'),
-                script_filename,
-            )
-            script_code = compile(script_text, script_filename, 'exec')
-            exec(script_code, namespace, namespace)
-
-    def _has(self, path):
-        raise NotImplementedError(
-            "Can't perform this operation for unregistered loader type"
-        )
-
-    def _isdir(self, path):
-        raise NotImplementedError(
-            "Can't perform this operation for unregistered loader type"
-        )
-
-    def _listdir(self, path):
-        raise NotImplementedError(
-            "Can't perform this operation for unregistered loader type"
-        )
-
-    def _fn(self, base, resource_name):
-        self._validate_resource_path(resource_name)
-        if resource_name:
-            return os.path.join(base, *resource_name.split('/'))
-        return base
-
-    @staticmethod
-    def _validate_resource_path(path):
-        """
-        Validate the resource paths according to the docs.
-        https://setuptools.pypa.io/en/latest/pkg_resources.html#basic-resource-access
-
-        >>> warned = getfixture('recwarn')
-        >>> warnings.simplefilter('always')
-        >>> vrp = NullProvider._validate_resource_path
-        >>> vrp('foo/bar.txt')
-        >>> bool(warned)
-        False
-        >>> vrp('../foo/bar.txt')
-        >>> bool(warned)
-        True
-        >>> warned.clear()
-        >>> vrp('/foo/bar.txt')
-        >>> bool(warned)
-        True
-        >>> vrp('foo/../../bar.txt')
-        >>> bool(warned)
-        True
-        >>> warned.clear()
-        >>> vrp('foo/f../bar.txt')
-        >>> bool(warned)
-        False
-
-        Windows path separators are straight-up disallowed.
-        >>> vrp(r'\\foo/bar.txt')
-        Traceback (most recent call last):
-        ...
-        ValueError: Use of .. or absolute path in a resource path \
-is not allowed.
-
-        >>> vrp(r'C:\\foo/bar.txt')
-        Traceback (most recent call last):
-        ...
-        ValueError: Use of .. or absolute path in a resource path \
-is not allowed.
-
-        Blank values are allowed
-
-        >>> vrp('')
-        >>> bool(warned)
-        False
-
-        Non-string values are not.
-
-        >>> vrp(None)
-        Traceback (most recent call last):
-        ...
-        AttributeError: ...
-        """
-        invalid = (
-            os.path.pardir in path.split(posixpath.sep)
-            or posixpath.isabs(path)
-            or ntpath.isabs(path)
-        )
-        if not invalid:
-            return
-
-        msg = "Use of .. or absolute path in a resource path is not allowed."
-
-        # Aggressively disallow Windows absolute paths
-        if ntpath.isabs(path) and not posixpath.isabs(path):
-            raise ValueError(msg)
-
-        # for compatibility, warn; in future
-        # raise ValueError(msg)
-        issue_warning(
-            msg[:-1] + " and will raise exceptions in a future release.",
-            DeprecationWarning,
-        )
-
-    def _get(self, path):
-        if hasattr(self.loader, 'get_data'):
-            return self.loader.get_data(path)
-        raise NotImplementedError(
-            "Can't perform this operation for loaders without 'get_data()'"
-        )
-
-
-register_loader_type(object, NullProvider)
-
-
-def _parents(path):
-    """
-    yield all parents of path including path
-    """
-    last = None
-    while path != last:
-        yield path
-        last = path
-        path, _ = os.path.split(path)
-
-
-class EggProvider(NullProvider):
-    """Provider based on a virtual filesystem"""
-
-    def __init__(self, module):
-        super().__init__(module)
-        self._setup_prefix()
-
-    def _setup_prefix(self):
-        # Assume that metadata may be nested inside a "basket"
-        # of multiple eggs and use module_path instead of .archive.
-        eggs = filter(_is_egg_path, _parents(self.module_path))
-        egg = next(eggs, None)
-        egg and self._set_egg(egg)
-
-    def _set_egg(self, path):
-        self.egg_name = os.path.basename(path)
-        self.egg_info = os.path.join(path, 'EGG-INFO')
-        self.egg_root = path
-
-
-class DefaultProvider(EggProvider):
-    """Provides access to package resources in the filesystem"""
-
-    def _has(self, path):
-        return os.path.exists(path)
-
-    def _isdir(self, path):
-        return os.path.isdir(path)
-
-    def _listdir(self, path):
-        return os.listdir(path)
-
-    def get_resource_stream(self, manager, resource_name):
-        return open(self._fn(self.module_path, resource_name), 'rb')
-
-    def _get(self, path):
-        with open(path, 'rb') as stream:
-            return stream.read()
-
-    @classmethod
-    def _register(cls):
-        loader_names = (
-            'SourceFileLoader',
-            'SourcelessFileLoader',
-        )
-        for name in loader_names:
-            loader_cls = getattr(importlib_machinery, name, type(None))
-            register_loader_type(loader_cls, cls)
-
-
-DefaultProvider._register()
-
-
-class EmptyProvider(NullProvider):
-    """Provider that returns nothing for all requests"""
-
-    module_path = None
-
-    _isdir = _has = lambda self, path: False
-
-    def _get(self, path):
-        return ''
-
-    def _listdir(self, path):
-        return []
-
-    def __init__(self):
-        pass
-
-
-empty_provider = EmptyProvider()
-
-
-class ZipManifests(dict):
-    """
-    zip manifest builder
-    """
-
-    @classmethod
-    def build(cls, path):
-        """
-        Build a dictionary similar to the zipimport directory
-        caches, except instead of tuples, store ZipInfo objects.
-
-        Use a platform-specific path separator (os.sep) for the path keys
-        for compatibility with pypy on Windows.
-        """
-        with zipfile.ZipFile(path) as zfile:
-            items = (
-                (
-                    name.replace('/', os.sep),
-                    zfile.getinfo(name),
-                )
-                for name in zfile.namelist()
-            )
-            return dict(items)
-
-    load = build
-
-
-class MemoizedZipManifests(ZipManifests):
-    """
-    Memoized zipfile manifests.
-    """
-
-    manifest_mod = collections.namedtuple('manifest_mod', 'manifest mtime')
-
-    def load(self, path):
-        """
-        Load a manifest at path or return a suitable manifest already loaded.
-        """
-        path = os.path.normpath(path)
-        mtime = os.stat(path).st_mtime
-
-        if path not in self or self[path].mtime != mtime:
-            manifest = self.build(path)
-            self[path] = self.manifest_mod(manifest, mtime)
-
-        return self[path].manifest
-
-
-class ZipProvider(EggProvider):
-    """Resource support for zips and eggs"""
-
-    eagers = None
-    _zip_manifests = MemoizedZipManifests()
-
-    def __init__(self, module):
-        super().__init__(module)
-        self.zip_pre = self.loader.archive + os.sep
-
-    def _zipinfo_name(self, fspath):
-        # Convert a virtual filename (full path to file) into a zipfile subpath
-        # usable with the zipimport directory cache for our target archive
-        fspath = fspath.rstrip(os.sep)
-        if fspath == self.loader.archive:
-            return ''
-        if fspath.startswith(self.zip_pre):
-            return fspath[len(self.zip_pre) :]
-        raise AssertionError("%s is not a subpath of %s" % (fspath, self.zip_pre))
-
-    def _parts(self, zip_path):
-        # Convert a zipfile subpath into an egg-relative path part list.
-        # pseudo-fs path
-        fspath = self.zip_pre + zip_path
-        if fspath.startswith(self.egg_root + os.sep):
-            return fspath[len(self.egg_root) + 1 :].split(os.sep)
-        raise AssertionError("%s is not a subpath of %s" % (fspath, self.egg_root))
-
-    @property
-    def zipinfo(self):
-        return self._zip_manifests.load(self.loader.archive)
-
-    def get_resource_filename(self, manager, resource_name):
-        if not self.egg_name:
-            raise NotImplementedError(
-                "resource_filename() only supported for .egg, not .zip"
-            )
-        # no need to lock for extraction, since we use temp names
-        zip_path = self._resource_to_zip(resource_name)
-        eagers = self._get_eager_resources()
-        if '/'.join(self._parts(zip_path)) in eagers:
-            for name in eagers:
-                self._extract_resource(manager, self._eager_to_zip(name))
-        return self._extract_resource(manager, zip_path)
-
-    @staticmethod
-    def _get_date_and_size(zip_stat):
-        size = zip_stat.file_size
-        # ymdhms+wday, yday, dst
-        date_time = zip_stat.date_time + (0, 0, -1)
-        # 1980 offset already done
-        timestamp = time.mktime(date_time)
-        return timestamp, size
-
-    # FIXME: 'ZipProvider._extract_resource' is too complex (12)
-    def _extract_resource(self, manager, zip_path):  # noqa: C901
-        if zip_path in self._index():
-            for name in self._index()[zip_path]:
-                last = self._extract_resource(manager, os.path.join(zip_path, name))
-            # return the extracted directory name
-            return os.path.dirname(last)
-
-        timestamp, size = self._get_date_and_size(self.zipinfo[zip_path])
-
-        if not WRITE_SUPPORT:
-            raise IOError(
-                '"os.rename" and "os.unlink" are not supported ' 'on this platform'
-            )
-        try:
-            real_path = manager.get_cache_path(self.egg_name, self._parts(zip_path))
-
-            if self._is_current(real_path, zip_path):
-                return real_path
-
-            outf, tmpnam = _mkstemp(
-                ".$extract",
-                dir=os.path.dirname(real_path),
-            )
-            os.write(outf, self.loader.get_data(zip_path))
-            os.close(outf)
-            utime(tmpnam, (timestamp, timestamp))
-            manager.postprocess(tmpnam, real_path)
-
-            try:
-                rename(tmpnam, real_path)
-
-            except os.error:
-                if os.path.isfile(real_path):
-                    if self._is_current(real_path, zip_path):
-                        # the file became current since it was checked above,
-                        #  so proceed.
-                        return real_path
-                    # Windows, del old file and retry
-                    elif os.name == 'nt':
-                        unlink(real_path)
-                        rename(tmpnam, real_path)
-                        return real_path
-                raise
-
-        except os.error:
-            # report a user-friendly error
-            manager.extraction_error()
-
-        return real_path
-
-    def _is_current(self, file_path, zip_path):
-        """
-        Return True if the file_path is current for this zip_path
-        """
-        timestamp, size = self._get_date_and_size(self.zipinfo[zip_path])
-        if not os.path.isfile(file_path):
-            return False
-        stat = os.stat(file_path)
-        if stat.st_size != size or stat.st_mtime != timestamp:
-            return False
-        # check that the contents match
-        zip_contents = self.loader.get_data(zip_path)
-        with open(file_path, 'rb') as f:
-            file_contents = f.read()
-        return zip_contents == file_contents
-
-    def _get_eager_resources(self):
-        if self.eagers is None:
-            eagers = []
-            for name in ('native_libs.txt', 'eager_resources.txt'):
-                if self.has_metadata(name):
-                    eagers.extend(self.get_metadata_lines(name))
-            self.eagers = eagers
-        return self.eagers
-
-    def _index(self):
-        try:
-            return self._dirindex
-        except AttributeError:
-            ind = {}
-            for path in self.zipinfo:
-                parts = path.split(os.sep)
-                while parts:
-                    parent = os.sep.join(parts[:-1])
-                    if parent in ind:
-                        ind[parent].append(parts[-1])
-                        break
-                    else:
-                        ind[parent] = [parts.pop()]
-            self._dirindex = ind
-            return ind
-
-    def _has(self, fspath):
-        zip_path = self._zipinfo_name(fspath)
-        return zip_path in self.zipinfo or zip_path in self._index()
-
-    def _isdir(self, fspath):
-        return self._zipinfo_name(fspath) in self._index()
-
-    def _listdir(self, fspath):
-        return list(self._index().get(self._zipinfo_name(fspath), ()))
-
-    def _eager_to_zip(self, resource_name):
-        return self._zipinfo_name(self._fn(self.egg_root, resource_name))
-
-    def _resource_to_zip(self, resource_name):
-        return self._zipinfo_name(self._fn(self.module_path, resource_name))
-
-
-register_loader_type(zipimport.zipimporter, ZipProvider)
-
-
-class FileMetadata(EmptyProvider):
-    """Metadata handler for standalone PKG-INFO files
-
-    Usage::
-
-        metadata = FileMetadata("/path/to/PKG-INFO")
-
-    This provider rejects all data and metadata requests except for PKG-INFO,
-    which is treated as existing, and will be the contents of the file at
-    the provided location.
-    """
-
-    def __init__(self, path):
-        self.path = path
-
-    def _get_metadata_path(self, name):
-        return self.path
-
-    def has_metadata(self, name):
-        return name == 'PKG-INFO' and os.path.isfile(self.path)
-
-    def get_metadata(self, name):
-        if name != 'PKG-INFO':
-            raise KeyError("No metadata except PKG-INFO is available")
-
-        with io.open(self.path, encoding='utf-8', errors="replace") as f:
-            metadata = f.read()
-        self._warn_on_replacement(metadata)
-        return metadata
-
-    def _warn_on_replacement(self, metadata):
-        replacement_char = '�'
-        if replacement_char in metadata:
-            tmpl = "{self.path} could not be properly decoded in UTF-8"
-            msg = tmpl.format(**locals())
-            warnings.warn(msg)
-
-    def get_metadata_lines(self, name):
-        return yield_lines(self.get_metadata(name))
-
-
-class PathMetadata(DefaultProvider):
-    """Metadata provider for egg directories
-
-    Usage::
-
-        # Development eggs:
-
-        egg_info = "/path/to/PackageName.egg-info"
-        base_dir = os.path.dirname(egg_info)
-        metadata = PathMetadata(base_dir, egg_info)
-        dist_name = os.path.splitext(os.path.basename(egg_info))[0]
-        dist = Distribution(basedir, project_name=dist_name, metadata=metadata)
-
-        # Unpacked egg directories:
-
-        egg_path = "/path/to/PackageName-ver-pyver-etc.egg"
-        metadata = PathMetadata(egg_path, os.path.join(egg_path,'EGG-INFO'))
-        dist = Distribution.from_filename(egg_path, metadata=metadata)
-    """
-
-    def __init__(self, path, egg_info):
-        self.module_path = path
-        self.egg_info = egg_info
-
-
-class EggMetadata(ZipProvider):
-    """Metadata provider for .egg files"""
-
-    def __init__(self, importer):
-        """Create a metadata provider from a zipimporter"""
-
-        self.zip_pre = importer.archive + os.sep
-        self.loader = importer
-        if importer.prefix:
-            self.module_path = os.path.join(importer.archive, importer.prefix)
-        else:
-            self.module_path = importer.archive
-        self._setup_prefix()
-
-
-_declare_state('dict', _distribution_finders={})
-
-
-def register_finder(importer_type, distribution_finder):
-    """Register `distribution_finder` to find distributions in sys.path items
-
-    `importer_type` is the type or class of a PEP 302 "Importer" (sys.path item
-    handler), and `distribution_finder` is a callable that, passed a path
-    item and the importer instance, yields ``Distribution`` instances found on
-    that path item.  See ``pkg_resources.find_on_path`` for an example."""
-    _distribution_finders[importer_type] = distribution_finder
-
-
-def find_distributions(path_item, only=False):
-    """Yield distributions accessible via `path_item`"""
-    importer = get_importer(path_item)
-    finder = _find_adapter(_distribution_finders, importer)
-    return finder(importer, path_item, only)
-
-
-def find_eggs_in_zip(importer, path_item, only=False):
-    """
-    Find eggs in zip files; possibly multiple nested eggs.
-    """
-    if importer.archive.endswith('.whl'):
-        # wheels are not supported with this finder
-        # they don't have PKG-INFO metadata, and won't ever contain eggs
-        return
-    metadata = EggMetadata(importer)
-    if metadata.has_metadata('PKG-INFO'):
-        yield Distribution.from_filename(path_item, metadata=metadata)
-    if only:
-        # don't yield nested distros
-        return
-    for subitem in metadata.resource_listdir(''):
-        if _is_egg_path(subitem):
-            subpath = os.path.join(path_item, subitem)
-            dists = find_eggs_in_zip(zipimport.zipimporter(subpath), subpath)
-            for dist in dists:
-                yield dist
-        elif subitem.lower().endswith(('.dist-info', '.egg-info')):
-            subpath = os.path.join(path_item, subitem)
-            submeta = EggMetadata(zipimport.zipimporter(subpath))
-            submeta.egg_info = subpath
-            yield Distribution.from_location(path_item, subitem, submeta)
-
-
-register_finder(zipimport.zipimporter, find_eggs_in_zip)
-
-
-def find_nothing(importer, path_item, only=False):
-    return ()
-
-
-register_finder(object, find_nothing)
-
-
-def find_on_path(importer, path_item, only=False):
-    """Yield distributions accessible on a sys.path directory"""
-    path_item = _normalize_cached(path_item)
-
-    if _is_unpacked_egg(path_item):
-        yield Distribution.from_filename(
-            path_item,
-            metadata=PathMetadata(path_item, os.path.join(path_item, 'EGG-INFO')),
-        )
-        return
-
-    entries = (os.path.join(path_item, child) for child in safe_listdir(path_item))
-
-    # scan for .egg and .egg-info in directory
-    for entry in sorted(entries):
-        fullpath = os.path.join(path_item, entry)
-        factory = dist_factory(path_item, entry, only)
-        for dist in factory(fullpath):
-            yield dist
-
-
-def dist_factory(path_item, entry, only):
-    """Return a dist_factory for the given entry."""
-    lower = entry.lower()
-    is_egg_info = lower.endswith('.egg-info')
-    is_dist_info = lower.endswith('.dist-info') and os.path.isdir(
-        os.path.join(path_item, entry)
-    )
-    is_meta = is_egg_info or is_dist_info
-    return (
-        distributions_from_metadata
-        if is_meta
-        else find_distributions
-        if not only and _is_egg_path(entry)
-        else resolve_egg_link
-        if not only and lower.endswith('.egg-link')
-        else NoDists()
-    )
-
-
-class NoDists:
-    """
-    >>> bool(NoDists())
-    False
-
-    >>> list(NoDists()('anything'))
-    []
-    """
-
-    def __bool__(self):
-        return False
-
-    def __call__(self, fullpath):
-        return iter(())
-
-
-def safe_listdir(path):
-    """
-    Attempt to list contents of path, but suppress some exceptions.
-    """
-    try:
-        return os.listdir(path)
-    except (PermissionError, NotADirectoryError):
-        pass
-    except OSError as e:
-        # Ignore the directory if does not exist, not a directory or
-        # permission denied
-        if e.errno not in (errno.ENOTDIR, errno.EACCES, errno.ENOENT):
-            raise
-    return ()
-
-
-def distributions_from_metadata(path):
-    root = os.path.dirname(path)
-    if os.path.isdir(path):
-        if len(os.listdir(path)) == 0:
-            # empty metadata dir; skip
-            return
-        metadata = PathMetadata(root, path)
-    else:
-        metadata = FileMetadata(path)
-    entry = os.path.basename(path)
-    yield Distribution.from_location(
-        root,
-        entry,
-        metadata,
-        precedence=DEVELOP_DIST,
-    )
-
-
-def non_empty_lines(path):
-    """
-    Yield non-empty lines from file at path
-    """
-    with open(path) as f:
-        for line in f:
-            line = line.strip()
-            if line:
-                yield line
-
-
-def resolve_egg_link(path):
-    """
-    Given a path to an .egg-link, resolve distributions
-    present in the referenced path.
-    """
-    referenced_paths = non_empty_lines(path)
-    resolved_paths = (
-        os.path.join(os.path.dirname(path), ref) for ref in referenced_paths
-    )
-    dist_groups = map(find_distributions, resolved_paths)
-    return next(dist_groups, ())
-
-
-if hasattr(pkgutil, 'ImpImporter'):
-    register_finder(pkgutil.ImpImporter, find_on_path)
-
-register_finder(importlib_machinery.FileFinder, find_on_path)
-
-_declare_state('dict', _namespace_handlers={})
-_declare_state('dict', _namespace_packages={})
-
-
-def register_namespace_handler(importer_type, namespace_handler):
-    """Register `namespace_handler` to declare namespace packages
-
-    `importer_type` is the type or class of a PEP 302 "Importer" (sys.path item
-    handler), and `namespace_handler` is a callable like this::
-
-        def namespace_handler(importer, path_entry, moduleName, module):
-            # return a path_entry to use for child packages
-
-    Namespace handlers are only called if the importer object has already
-    agreed that it can handle the relevant path item, and they should only
-    return a subpath if the module __path__ does not already contain an
-    equivalent subpath.  For an example namespace handler, see
-    ``pkg_resources.file_ns_handler``.
-    """
-    _namespace_handlers[importer_type] = namespace_handler
-
-
-def _handle_ns(packageName, path_item):
-    """Ensure that named package includes a subpath of path_item (if needed)"""
-
-    importer = get_importer(path_item)
-    if importer is None:
-        return None
-
-    # use find_spec (PEP 451) and fall-back to find_module (PEP 302)
-    try:
-        spec = importer.find_spec(packageName)
-    except AttributeError:
-        # capture warnings due to #1111
-        with warnings.catch_warnings():
-            warnings.simplefilter("ignore")
-            loader = importer.find_module(packageName)
-    else:
-        loader = spec.loader if spec else None
-
-    if loader is None:
-        return None
-    module = sys.modules.get(packageName)
-    if module is None:
-        module = sys.modules[packageName] = types.ModuleType(packageName)
-        module.__path__ = []
-        _set_parent_ns(packageName)
-    elif not hasattr(module, '__path__'):
-        raise TypeError("Not a package:", packageName)
-    handler = _find_adapter(_namespace_handlers, importer)
-    subpath = handler(importer, path_item, packageName, module)
-    if subpath is not None:
-        path = module.__path__
-        path.append(subpath)
-        importlib.import_module(packageName)
-        _rebuild_mod_path(path, packageName, module)
-    return subpath
-
-
-def _rebuild_mod_path(orig_path, package_name, module):
-    """
-    Rebuild module.__path__ ensuring that all entries are ordered
-    corresponding to their sys.path order
-    """
-    sys_path = [_normalize_cached(p) for p in sys.path]
-
-    def safe_sys_path_index(entry):
-        """
-        Workaround for #520 and #513.
-        """
-        try:
-            return sys_path.index(entry)
-        except ValueError:
-            return float('inf')
-
-    def position_in_sys_path(path):
-        """
-        Return the ordinal of the path based on its position in sys.path
-        """
-        path_parts = path.split(os.sep)
-        module_parts = package_name.count('.') + 1
-        parts = path_parts[:-module_parts]
-        return safe_sys_path_index(_normalize_cached(os.sep.join(parts)))
-
-    new_path = sorted(orig_path, key=position_in_sys_path)
-    new_path = [_normalize_cached(p) for p in new_path]
-
-    if isinstance(module.__path__, list):
-        module.__path__[:] = new_path
-    else:
-        module.__path__ = new_path
-
-
-def declare_namespace(packageName):
-    """Declare that package 'packageName' is a namespace package"""
-
-    msg = (
-        f"Deprecated call to `pkg_resources.declare_namespace({packageName!r})`.\n"
-        "Implementing implicit namespace packages (as specified in PEP 420) "
-        "is preferred to `pkg_resources.declare_namespace`. "
-        "See https://setuptools.pypa.io/en/latest/references/"
-        "keywords.html#keyword-namespace-packages"
-    )
-    warnings.warn(msg, DeprecationWarning, stacklevel=2)
-
-    _imp.acquire_lock()
-    try:
-        if packageName in _namespace_packages:
-            return
-
-        path = sys.path
-        parent, _, _ = packageName.rpartition('.')
-
-        if parent:
-            declare_namespace(parent)
-            if parent not in _namespace_packages:
-                __import__(parent)
-            try:
-                path = sys.modules[parent].__path__
-            except AttributeError as e:
-                raise TypeError("Not a package:", parent) from e
-
-        # Track what packages are namespaces, so when new path items are added,
-        # they can be updated
-        _namespace_packages.setdefault(parent or None, []).append(packageName)
-        _namespace_packages.setdefault(packageName, [])
-
-        for path_item in path:
-            # Ensure all the parent's path items are reflected in the child,
-            # if they apply
-            _handle_ns(packageName, path_item)
-
-    finally:
-        _imp.release_lock()
-
-
-def fixup_namespace_packages(path_item, parent=None):
-    """Ensure that previously-declared namespace packages include path_item"""
-    _imp.acquire_lock()
-    try:
-        for package in _namespace_packages.get(parent, ()):
-            subpath = _handle_ns(package, path_item)
-            if subpath:
-                fixup_namespace_packages(subpath, package)
-    finally:
-        _imp.release_lock()
-
-
-def file_ns_handler(importer, path_item, packageName, module):
-    """Compute an ns-package subpath for a filesystem or zipfile importer"""
-
-    subpath = os.path.join(path_item, packageName.split('.')[-1])
-    normalized = _normalize_cached(subpath)
-    for item in module.__path__:
-        if _normalize_cached(item) == normalized:
-            break
-    else:
-        # Only return the path if it's not already there
-        return subpath
-
-
-if hasattr(pkgutil, 'ImpImporter'):
-    register_namespace_handler(pkgutil.ImpImporter, file_ns_handler)
-
-register_namespace_handler(zipimport.zipimporter, file_ns_handler)
-register_namespace_handler(importlib_machinery.FileFinder, file_ns_handler)
-
-
-def null_ns_handler(importer, path_item, packageName, module):
-    return None
-
-
-register_namespace_handler(object, null_ns_handler)
-
-
-def normalize_path(filename):
-    """Normalize a file/dir name for comparison purposes"""
-    return os.path.normcase(os.path.realpath(os.path.normpath(_cygwin_patch(filename))))
-
-
-def _cygwin_patch(filename):  # pragma: nocover
-    """
-    Contrary to POSIX 2008, on Cygwin, getcwd (3) contains
-    symlink components. Using
-    os.path.abspath() works around this limitation. A fix in os.getcwd()
-    would probably better, in Cygwin even more so, except
-    that this seems to be by design...
-    """
-    return os.path.abspath(filename) if sys.platform == 'cygwin' else filename
-
-
-def _normalize_cached(filename, _cache={}):
-    try:
-        return _cache[filename]
-    except KeyError:
-        _cache[filename] = result = normalize_path(filename)
-        return result
-
-
-def _is_egg_path(path):
-    """
-    Determine if given path appears to be an egg.
-    """
-    return _is_zip_egg(path) or _is_unpacked_egg(path)
-
-
-def _is_zip_egg(path):
-    return (
-        path.lower().endswith('.egg')
-        and os.path.isfile(path)
-        and zipfile.is_zipfile(path)
-    )
-
-
-def _is_unpacked_egg(path):
-    """
-    Determine if given path appears to be an unpacked egg.
-    """
-    return path.lower().endswith('.egg') and os.path.isfile(
-        os.path.join(path, 'EGG-INFO', 'PKG-INFO')
-    )
-
-
-def _set_parent_ns(packageName):
-    parts = packageName.split('.')
-    name = parts.pop()
-    if parts:
-        parent = '.'.join(parts)
-        setattr(sys.modules[parent], name, sys.modules[packageName])
-
-
-MODULE = re.compile(r"\w+(\.\w+)*$").match
-EGG_NAME = re.compile(
-    r"""
-    (?P[^-]+) (
-        -(?P[^-]+) (
-            -py(?P[^-]+) (
-                -(?P.+)
-            )?
-        )?
-    )?
-    """,
-    re.VERBOSE | re.IGNORECASE,
-).match
-
-
-class EntryPoint:
-    """Object representing an advertised importable object"""
-
-    def __init__(self, name, module_name, attrs=(), extras=(), dist=None):
-        if not MODULE(module_name):
-            raise ValueError("Invalid module name", module_name)
-        self.name = name
-        self.module_name = module_name
-        self.attrs = tuple(attrs)
-        self.extras = tuple(extras)
-        self.dist = dist
-
-    def __str__(self):
-        s = "%s = %s" % (self.name, self.module_name)
-        if self.attrs:
-            s += ':' + '.'.join(self.attrs)
-        if self.extras:
-            s += ' [%s]' % ','.join(self.extras)
-        return s
-
-    def __repr__(self):
-        return "EntryPoint.parse(%r)" % str(self)
-
-    def load(self, require=True, *args, **kwargs):
-        """
-        Require packages for this EntryPoint, then resolve it.
-        """
-        if not require or args or kwargs:
-            warnings.warn(
-                "Parameters to load are deprecated.  Call .resolve and "
-                ".require separately.",
-                PkgResourcesDeprecationWarning,
-                stacklevel=2,
-            )
-        if require:
-            self.require(*args, **kwargs)
-        return self.resolve()
-
-    def resolve(self):
-        """
-        Resolve the entry point from its module and attrs.
-        """
-        module = __import__(self.module_name, fromlist=['__name__'], level=0)
-        try:
-            return functools.reduce(getattr, self.attrs, module)
-        except AttributeError as exc:
-            raise ImportError(str(exc)) from exc
-
-    def require(self, env=None, installer=None):
-        if self.extras and not self.dist:
-            raise UnknownExtra("Can't require() without a distribution", self)
-
-        # Get the requirements for this entry point with all its extras and
-        # then resolve them. We have to pass `extras` along when resolving so
-        # that the working set knows what extras we want. Otherwise, for
-        # dist-info distributions, the working set will assume that the
-        # requirements for that extra are purely optional and skip over them.
-        reqs = self.dist.requires(self.extras)
-        items = working_set.resolve(reqs, env, installer, extras=self.extras)
-        list(map(working_set.add, items))
-
-    pattern = re.compile(
-        r'\s*'
-        r'(?P.+?)\s*'
-        r'=\s*'
-        r'(?P[\w.]+)\s*'
-        r'(:\s*(?P[\w.]+))?\s*'
-        r'(?P\[.*\])?\s*$'
-    )
-
-    @classmethod
-    def parse(cls, src, dist=None):
-        """Parse a single entry point from string `src`
-
-        Entry point syntax follows the form::
-
-            name = some.module:some.attr [extra1, extra2]
-
-        The entry name and module name are required, but the ``:attrs`` and
-        ``[extras]`` parts are optional
-        """
-        m = cls.pattern.match(src)
-        if not m:
-            msg = "EntryPoint must be in 'name=module:attrs [extras]' format"
-            raise ValueError(msg, src)
-        res = m.groupdict()
-        extras = cls._parse_extras(res['extras'])
-        attrs = res['attr'].split('.') if res['attr'] else ()
-        return cls(res['name'], res['module'], attrs, extras, dist)
-
-    @classmethod
-    def _parse_extras(cls, extras_spec):
-        if not extras_spec:
-            return ()
-        req = Requirement.parse('x' + extras_spec)
-        if req.specs:
-            raise ValueError()
-        return req.extras
-
-    @classmethod
-    def parse_group(cls, group, lines, dist=None):
-        """Parse an entry point group"""
-        if not MODULE(group):
-            raise ValueError("Invalid group name", group)
-        this = {}
-        for line in yield_lines(lines):
-            ep = cls.parse(line, dist)
-            if ep.name in this:
-                raise ValueError("Duplicate entry point", group, ep.name)
-            this[ep.name] = ep
-        return this
-
-    @classmethod
-    def parse_map(cls, data, dist=None):
-        """Parse a map of entry point groups"""
-        if isinstance(data, dict):
-            data = data.items()
-        else:
-            data = split_sections(data)
-        maps = {}
-        for group, lines in data:
-            if group is None:
-                if not lines:
-                    continue
-                raise ValueError("Entry points must be listed in groups")
-            group = group.strip()
-            if group in maps:
-                raise ValueError("Duplicate group name", group)
-            maps[group] = cls.parse_group(group, lines, dist)
-        return maps
-
-
-def _version_from_file(lines):
-    """
-    Given an iterable of lines from a Metadata file, return
-    the value of the Version field, if present, or None otherwise.
-    """
-
-    def is_version_line(line):
-        return line.lower().startswith('version:')
-
-    version_lines = filter(is_version_line, lines)
-    line = next(iter(version_lines), '')
-    _, _, value = line.partition(':')
-    return safe_version(value.strip()) or None
-
-
-class Distribution:
-    """Wrap an actual or potential sys.path entry w/metadata"""
-
-    PKG_INFO = 'PKG-INFO'
-
-    def __init__(
-        self,
-        location=None,
-        metadata=None,
-        project_name=None,
-        version=None,
-        py_version=PY_MAJOR,
-        platform=None,
-        precedence=EGG_DIST,
-    ):
-        self.project_name = safe_name(project_name or 'Unknown')
-        if version is not None:
-            self._version = safe_version(version)
-        self.py_version = py_version
-        self.platform = platform
-        self.location = location
-        self.precedence = precedence
-        self._provider = metadata or empty_provider
-
-    @classmethod
-    def from_location(cls, location, basename, metadata=None, **kw):
-        project_name, version, py_version, platform = [None] * 4
-        basename, ext = os.path.splitext(basename)
-        if ext.lower() in _distributionImpl:
-            cls = _distributionImpl[ext.lower()]
-
-            match = EGG_NAME(basename)
-            if match:
-                project_name, version, py_version, platform = match.group(
-                    'name', 'ver', 'pyver', 'plat'
-                )
-        return cls(
-            location,
-            metadata,
-            project_name=project_name,
-            version=version,
-            py_version=py_version,
-            platform=platform,
-            **kw,
-        )._reload_version()
-
-    def _reload_version(self):
-        return self
-
-    @property
-    def hashcmp(self):
-        return (
-            self._forgiving_parsed_version,
-            self.precedence,
-            self.key,
-            self.location,
-            self.py_version or '',
-            self.platform or '',
-        )
-
-    def __hash__(self):
-        return hash(self.hashcmp)
-
-    def __lt__(self, other):
-        return self.hashcmp < other.hashcmp
-
-    def __le__(self, other):
-        return self.hashcmp <= other.hashcmp
-
-    def __gt__(self, other):
-        return self.hashcmp > other.hashcmp
-
-    def __ge__(self, other):
-        return self.hashcmp >= other.hashcmp
-
-    def __eq__(self, other):
-        if not isinstance(other, self.__class__):
-            # It's not a Distribution, so they are not equal
-            return False
-        return self.hashcmp == other.hashcmp
-
-    def __ne__(self, other):
-        return not self == other
-
-    # These properties have to be lazy so that we don't have to load any
-    # metadata until/unless it's actually needed.  (i.e., some distributions
-    # may not know their name or version without loading PKG-INFO)
-
-    @property
-    def key(self):
-        try:
-            return self._key
-        except AttributeError:
-            self._key = key = self.project_name.lower()
-            return key
-
-    @property
-    def parsed_version(self):
-        if not hasattr(self, "_parsed_version"):
-            try:
-                self._parsed_version = parse_version(self.version)
-            except packaging.version.InvalidVersion as ex:
-                info = f"(package: {self.project_name})"
-                if hasattr(ex, "add_note"):
-                    ex.add_note(info)  # PEP 678
-                    raise
-                raise packaging.version.InvalidVersion(f"{str(ex)} {info}") from None
-
-        return self._parsed_version
-
-    @property
-    def _forgiving_parsed_version(self):
-        try:
-            return self.parsed_version
-        except packaging.version.InvalidVersion as ex:
-            self._parsed_version = parse_version(_forgiving_version(self.version))
-
-            notes = "\n".join(getattr(ex, "__notes__", []))  # PEP 678
-            msg = f"""!!\n\n
-            *************************************************************************
-            {str(ex)}\n{notes}
-
-            This is a long overdue deprecation.
-            For the time being, `pkg_resources` will use `{self._parsed_version}`
-            as a replacement to avoid breaking existing environments,
-            but no future compatibility is guaranteed.
-
-            If you maintain package {self.project_name} you should implement
-            the relevant changes to adequate the project to PEP 440 immediately.
-            *************************************************************************
-            \n\n!!
-            """
-            warnings.warn(msg, DeprecationWarning)
-
-            return self._parsed_version
-
-    @property
-    def version(self):
-        try:
-            return self._version
-        except AttributeError as e:
-            version = self._get_version()
-            if version is None:
-                path = self._get_metadata_path_for_display(self.PKG_INFO)
-                msg = ("Missing 'Version:' header and/or {} file at path: {}").format(
-                    self.PKG_INFO, path
-                )
-                raise ValueError(msg, self) from e
-
-            return version
-
-    @property
-    def _dep_map(self):
-        """
-        A map of extra to its list of (direct) requirements
-        for this distribution, including the null extra.
-        """
-        try:
-            return self.__dep_map
-        except AttributeError:
-            self.__dep_map = self._filter_extras(self._build_dep_map())
-        return self.__dep_map
-
-    @staticmethod
-    def _filter_extras(dm):
-        """
-        Given a mapping of extras to dependencies, strip off
-        environment markers and filter out any dependencies
-        not matching the markers.
-        """
-        for extra in list(filter(None, dm)):
-            new_extra = extra
-            reqs = dm.pop(extra)
-            new_extra, _, marker = extra.partition(':')
-            fails_marker = marker and (
-                invalid_marker(marker) or not evaluate_marker(marker)
-            )
-            if fails_marker:
-                reqs = []
-            new_extra = safe_extra(new_extra) or None
-
-            dm.setdefault(new_extra, []).extend(reqs)
-        return dm
-
-    def _build_dep_map(self):
-        dm = {}
-        for name in 'requires.txt', 'depends.txt':
-            for extra, reqs in split_sections(self._get_metadata(name)):
-                dm.setdefault(extra, []).extend(parse_requirements(reqs))
-        return dm
-
-    def requires(self, extras=()):
-        """List of Requirements needed for this distro if `extras` are used"""
-        dm = self._dep_map
-        deps = []
-        deps.extend(dm.get(None, ()))
-        for ext in extras:
-            try:
-                deps.extend(dm[safe_extra(ext)])
-            except KeyError as e:
-                raise UnknownExtra(
-                    "%s has no such extra feature %r" % (self, ext)
-                ) from e
-        return deps
-
-    def _get_metadata_path_for_display(self, name):
-        """
-        Return the path to the given metadata file, if available.
-        """
-        try:
-            # We need to access _get_metadata_path() on the provider object
-            # directly rather than through this class's __getattr__()
-            # since _get_metadata_path() is marked private.
-            path = self._provider._get_metadata_path(name)
-
-        # Handle exceptions e.g. in case the distribution's metadata
-        # provider doesn't support _get_metadata_path().
-        except Exception:
-            return '[could not detect]'
-
-        return path
-
-    def _get_metadata(self, name):
-        if self.has_metadata(name):
-            for line in self.get_metadata_lines(name):
-                yield line
-
-    def _get_version(self):
-        lines = self._get_metadata(self.PKG_INFO)
-        version = _version_from_file(lines)
-
-        return version
-
-    def activate(self, path=None, replace=False):
-        """Ensure distribution is importable on `path` (default=sys.path)"""
-        if path is None:
-            path = sys.path
-        self.insert_on(path, replace=replace)
-        if path is sys.path:
-            fixup_namespace_packages(self.location)
-            for pkg in self._get_metadata('namespace_packages.txt'):
-                if pkg in sys.modules:
-                    declare_namespace(pkg)
-
-    def egg_name(self):
-        """Return what this distribution's standard .egg filename should be"""
-        filename = "%s-%s-py%s" % (
-            to_filename(self.project_name),
-            to_filename(self.version),
-            self.py_version or PY_MAJOR,
-        )
-
-        if self.platform:
-            filename += '-' + self.platform
-        return filename
-
-    def __repr__(self):
-        if self.location:
-            return "%s (%s)" % (self, self.location)
-        else:
-            return str(self)
-
-    def __str__(self):
-        try:
-            version = getattr(self, 'version', None)
-        except ValueError:
-            version = None
-        version = version or "[unknown version]"
-        return "%s %s" % (self.project_name, version)
-
-    def __getattr__(self, attr):
-        """Delegate all unrecognized public attributes to .metadata provider"""
-        if attr.startswith('_'):
-            raise AttributeError(attr)
-        return getattr(self._provider, attr)
-
-    def __dir__(self):
-        return list(
-            set(super(Distribution, self).__dir__())
-            | set(attr for attr in self._provider.__dir__() if not attr.startswith('_'))
-        )
-
-    @classmethod
-    def from_filename(cls, filename, metadata=None, **kw):
-        return cls.from_location(
-            _normalize_cached(filename), os.path.basename(filename), metadata, **kw
-        )
-
-    def as_requirement(self):
-        """Return a ``Requirement`` that matches this distribution exactly"""
-        if isinstance(self.parsed_version, packaging.version.Version):
-            spec = "%s==%s" % (self.project_name, self.parsed_version)
-        else:
-            spec = "%s===%s" % (self.project_name, self.parsed_version)
-
-        return Requirement.parse(spec)
-
-    def load_entry_point(self, group, name):
-        """Return the `name` entry point of `group` or raise ImportError"""
-        ep = self.get_entry_info(group, name)
-        if ep is None:
-            raise ImportError("Entry point %r not found" % ((group, name),))
-        return ep.load()
-
-    def get_entry_map(self, group=None):
-        """Return the entry point map for `group`, or the full entry map"""
-        try:
-            ep_map = self._ep_map
-        except AttributeError:
-            ep_map = self._ep_map = EntryPoint.parse_map(
-                self._get_metadata('entry_points.txt'), self
-            )
-        if group is not None:
-            return ep_map.get(group, {})
-        return ep_map
-
-    def get_entry_info(self, group, name):
-        """Return the EntryPoint object for `group`+`name`, or ``None``"""
-        return self.get_entry_map(group).get(name)
-
-    # FIXME: 'Distribution.insert_on' is too complex (13)
-    def insert_on(self, path, loc=None, replace=False):  # noqa: C901
-        """Ensure self.location is on path
-
-        If replace=False (default):
-            - If location is already in path anywhere, do nothing.
-            - Else:
-              - If it's an egg and its parent directory is on path,
-                insert just ahead of the parent.
-              - Else: add to the end of path.
-        If replace=True:
-            - If location is already on path anywhere (not eggs)
-              or higher priority than its parent (eggs)
-              do nothing.
-            - Else:
-              - If it's an egg and its parent directory is on path,
-                insert just ahead of the parent,
-                removing any lower-priority entries.
-              - Else: add it to the front of path.
-        """
-
-        loc = loc or self.location
-        if not loc:
-            return
-
-        nloc = _normalize_cached(loc)
-        bdir = os.path.dirname(nloc)
-        npath = [(p and _normalize_cached(p) or p) for p in path]
-
-        for p, item in enumerate(npath):
-            if item == nloc:
-                if replace:
-                    break
-                else:
-                    # don't modify path (even removing duplicates) if
-                    # found and not replace
-                    return
-            elif item == bdir and self.precedence == EGG_DIST:
-                # if it's an .egg, give it precedence over its directory
-                # UNLESS it's already been added to sys.path and replace=False
-                if (not replace) and nloc in npath[p:]:
-                    return
-                if path is sys.path:
-                    self.check_version_conflict()
-                path.insert(p, loc)
-                npath.insert(p, nloc)
-                break
-        else:
-            if path is sys.path:
-                self.check_version_conflict()
-            if replace:
-                path.insert(0, loc)
-            else:
-                path.append(loc)
-            return
-
-        # p is the spot where we found or inserted loc; now remove duplicates
-        while True:
-            try:
-                np = npath.index(nloc, p + 1)
-            except ValueError:
-                break
-            else:
-                del npath[np], path[np]
-                # ha!
-                p = np
-
-        return
-
-    def check_version_conflict(self):
-        if self.key == 'setuptools':
-            # ignore the inevitable setuptools self-conflicts  :(
-            return
-
-        nsp = dict.fromkeys(self._get_metadata('namespace_packages.txt'))
-        loc = normalize_path(self.location)
-        for modname in self._get_metadata('top_level.txt'):
-            if (
-                modname not in sys.modules
-                or modname in nsp
-                or modname in _namespace_packages
-            ):
-                continue
-            if modname in ('pkg_resources', 'setuptools', 'site'):
-                continue
-            fn = getattr(sys.modules[modname], '__file__', None)
-            if fn and (
-                normalize_path(fn).startswith(loc) or fn.startswith(self.location)
-            ):
-                continue
-            issue_warning(
-                "Module %s was already imported from %s, but %s is being added"
-                " to sys.path" % (modname, fn, self.location),
-            )
-
-    def has_version(self):
-        try:
-            self.version
-        except ValueError:
-            issue_warning("Unbuilt egg for " + repr(self))
-            return False
-        except SystemError:
-            # TODO: remove this except clause when python/cpython#103632 is fixed.
-            return False
-        return True
-
-    def clone(self, **kw):
-        """Copy this distribution, substituting in any changed keyword args"""
-        names = 'project_name version py_version platform location precedence'
-        for attr in names.split():
-            kw.setdefault(attr, getattr(self, attr, None))
-        kw.setdefault('metadata', self._provider)
-        return self.__class__(**kw)
-
-    @property
-    def extras(self):
-        return [dep for dep in self._dep_map if dep]
-
-
-class EggInfoDistribution(Distribution):
-    def _reload_version(self):
-        """
-        Packages installed by distutils (e.g. numpy or scipy),
-        which uses an old safe_version, and so
-        their version numbers can get mangled when
-        converted to filenames (e.g., 1.11.0.dev0+2329eae to
-        1.11.0.dev0_2329eae). These distributions will not be
-        parsed properly
-        downstream by Distribution and safe_version, so
-        take an extra step and try to get the version number from
-        the metadata file itself instead of the filename.
-        """
-        md_version = self._get_version()
-        if md_version:
-            self._version = md_version
-        return self
-
-
-class DistInfoDistribution(Distribution):
-    """
-    Wrap an actual or potential sys.path entry
-    w/metadata, .dist-info style.
-    """
-
-    PKG_INFO = 'METADATA'
-    EQEQ = re.compile(r"([\(,])\s*(\d.*?)\s*([,\)])")
-
-    @property
-    def _parsed_pkg_info(self):
-        """Parse and cache metadata"""
-        try:
-            return self._pkg_info
-        except AttributeError:
-            metadata = self.get_metadata(self.PKG_INFO)
-            self._pkg_info = email.parser.Parser().parsestr(metadata)
-            return self._pkg_info
-
-    @property
-    def _dep_map(self):
-        try:
-            return self.__dep_map
-        except AttributeError:
-            self.__dep_map = self._compute_dependencies()
-            return self.__dep_map
-
-    def _compute_dependencies(self):
-        """Recompute this distribution's dependencies."""
-        dm = self.__dep_map = {None: []}
-
-        reqs = []
-        # Including any condition expressions
-        for req in self._parsed_pkg_info.get_all('Requires-Dist') or []:
-            reqs.extend(parse_requirements(req))
-
-        def reqs_for_extra(extra):
-            for req in reqs:
-                if not req.marker or req.marker.evaluate({'extra': extra}):
-                    yield req
-
-        common = types.MappingProxyType(dict.fromkeys(reqs_for_extra(None)))
-        dm[None].extend(common)
-
-        for extra in self._parsed_pkg_info.get_all('Provides-Extra') or []:
-            s_extra = safe_extra(extra.strip())
-            dm[s_extra] = [r for r in reqs_for_extra(extra) if r not in common]
-
-        return dm
-
-
-_distributionImpl = {
-    '.egg': Distribution,
-    '.egg-info': EggInfoDistribution,
-    '.dist-info': DistInfoDistribution,
-}
-
-
-def issue_warning(*args, **kw):
-    level = 1
-    g = globals()
-    try:
-        # find the first stack frame that is *not* code in
-        # the pkg_resources module, to use for the warning
-        while sys._getframe(level).f_globals is g:
-            level += 1
-    except ValueError:
-        pass
-    warnings.warn(stacklevel=level + 1, *args, **kw)
-
-
-def parse_requirements(strs):
-    """
-    Yield ``Requirement`` objects for each specification in `strs`.
-
-    `strs` must be a string, or a (possibly-nested) iterable thereof.
-    """
-    return map(Requirement, join_continuation(map(drop_comment, yield_lines(strs))))
-
-
-class RequirementParseError(packaging.requirements.InvalidRequirement):
-    "Compatibility wrapper for InvalidRequirement"
-
-
-class Requirement(packaging.requirements.Requirement):
-    def __init__(self, requirement_string):
-        """DO NOT CALL THIS UNDOCUMENTED METHOD; use Requirement.parse()!"""
-        super(Requirement, self).__init__(requirement_string)
-        self.unsafe_name = self.name
-        project_name = safe_name(self.name)
-        self.project_name, self.key = project_name, project_name.lower()
-        self.specs = [(spec.operator, spec.version) for spec in self.specifier]
-        self.extras = tuple(map(safe_extra, self.extras))
-        self.hashCmp = (
-            self.key,
-            self.url,
-            self.specifier,
-            frozenset(self.extras),
-            str(self.marker) if self.marker else None,
-        )
-        self.__hash = hash(self.hashCmp)
-
-    def __eq__(self, other):
-        return isinstance(other, Requirement) and self.hashCmp == other.hashCmp
-
-    def __ne__(self, other):
-        return not self == other
-
-    def __contains__(self, item):
-        if isinstance(item, Distribution):
-            if item.key != self.key:
-                return False
-
-            item = item.version
-
-        # Allow prereleases always in order to match the previous behavior of
-        # this method. In the future this should be smarter and follow PEP 440
-        # more accurately.
-        return self.specifier.contains(item, prereleases=True)
-
-    def __hash__(self):
-        return self.__hash
-
-    def __repr__(self):
-        return "Requirement.parse(%r)" % str(self)
-
-    @staticmethod
-    def parse(s):
-        (req,) = parse_requirements(s)
-        return req
-
-
-def _always_object(classes):
-    """
-    Ensure object appears in the mro even
-    for old-style classes.
-    """
-    if object not in classes:
-        return classes + (object,)
-    return classes
-
-
-def _find_adapter(registry, ob):
-    """Return an adapter factory for `ob` from `registry`"""
-    types = _always_object(inspect.getmro(getattr(ob, '__class__', type(ob))))
-    for t in types:
-        if t in registry:
-            return registry[t]
-
-
-def ensure_directory(path):
-    """Ensure that the parent directory of `path` exists"""
-    dirname = os.path.dirname(path)
-    os.makedirs(dirname, exist_ok=True)
-
-
-def _bypass_ensure_directory(path):
-    """Sandbox-bypassing version of ensure_directory()"""
-    if not WRITE_SUPPORT:
-        raise IOError('"os.mkdir" not supported on this platform.')
-    dirname, filename = split(path)
-    if dirname and filename and not isdir(dirname):
-        _bypass_ensure_directory(dirname)
-        try:
-            mkdir(dirname, 0o755)
-        except FileExistsError:
-            pass
-
-
-def split_sections(s):
-    """Split a string or iterable thereof into (section, content) pairs
-
-    Each ``section`` is a stripped version of the section header ("[section]")
-    and each ``content`` is a list of stripped lines excluding blank lines and
-    comment-only lines.  If there are any such lines before the first section
-    header, they're returned in a first ``section`` of ``None``.
-    """
-    section = None
-    content = []
-    for line in yield_lines(s):
-        if line.startswith("["):
-            if line.endswith("]"):
-                if section or content:
-                    yield section, content
-                section = line[1:-1].strip()
-                content = []
-            else:
-                raise ValueError("Invalid section heading", line)
-        else:
-            content.append(line)
-
-    # wrap up last segment
-    yield section, content
-
-
-def _mkstemp(*args, **kw):
-    old_open = os.open
-    try:
-        # temporarily bypass sandboxing
-        os.open = os_open
-        return tempfile.mkstemp(*args, **kw)
-    finally:
-        # and then put it back
-        os.open = old_open
-
-
-# Silence the PEP440Warning by default, so that end users don't get hit by it
-# randomly just because they use pkg_resources. We want to append the rule
-# because we want earlier uses of filterwarnings to take precedence over this
-# one.
-warnings.filterwarnings("ignore", category=PEP440Warning, append=True)
-
-
-# from jaraco.functools 1.3
-def _call_aside(f, *args, **kwargs):
-    f(*args, **kwargs)
-    return f
-
-
-@_call_aside
-def _initialize(g=globals()):
-    "Set up global resource manager (deliberately not state-saved)"
-    manager = ResourceManager()
-    g['_manager'] = manager
-    g.update(
-        (name, getattr(manager, name))
-        for name in dir(manager)
-        if not name.startswith('_')
-    )
-
-
-class PkgResourcesDeprecationWarning(Warning):
-    """
-    Base class for warning about deprecations in ``pkg_resources``
-
-    This class is not derived from ``DeprecationWarning``, and as such is
-    visible by default.
-    """
-
-
-@_call_aside
-def _initialize_master_working_set():
-    """
-    Prepare the master working set and make the ``require()``
-    API available.
-
-    This function has explicit effects on the global state
-    of pkg_resources. It is intended to be invoked once at
-    the initialization of this module.
-
-    Invocation by other packages is unsupported and done
-    at their own risk.
-    """
-    working_set = WorkingSet._build_master()
-    _declare_state('object', working_set=working_set)
-
-    require = working_set.require
-    iter_entry_points = working_set.iter_entry_points
-    add_activation_listener = working_set.subscribe
-    run_script = working_set.run_script
-    # backward compatibility
-    run_main = run_script
-    # Activate all distributions already on sys.path with replace=False and
-    # ensure that all distributions added to the working set in the future
-    # (e.g. by calling ``require()``) will get activated as well,
-    # with higher priority (replace=True).
-    tuple(dist.activate(replace=False) for dist in working_set)
-    add_activation_listener(
-        lambda dist: dist.activate(replace=True),
-        existing=False,
-    )
-    working_set.entries = []
-    # match order
-    list(map(working_set.add_entry, sys.path))
-    globals().update(locals())
diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/pkg_resources/__pycache__/__init__.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/pkg_resources/__pycache__/__init__.cpython-311.pyc
deleted file mode 100644
index ccf1389..0000000
Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/pkg_resources/__pycache__/__init__.cpython-311.pyc and /dev/null differ
diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/platformdirs/__init__.py b/venv/lib/python3.11/site-packages/pip/_vendor/platformdirs/__init__.py
deleted file mode 100644
index 5ebf595..0000000
--- a/venv/lib/python3.11/site-packages/pip/_vendor/platformdirs/__init__.py
+++ /dev/null
@@ -1,566 +0,0 @@
-"""
-Utilities for determining application-specific dirs. See  for details and
-usage.
-"""
-from __future__ import annotations
-
-import os
-import sys
-from typing import TYPE_CHECKING
-
-from .api import PlatformDirsABC
-from .version import __version__
-from .version import __version_tuple__ as __version_info__
-
-if TYPE_CHECKING:
-    from pathlib import Path
-
-    if sys.version_info >= (3, 8):  # pragma: no cover (py38+)
-        from typing import Literal
-    else:  # pragma: no cover (py38+)
-        from pip._vendor.typing_extensions import Literal
-
-
-def _set_platform_dir_class() -> type[PlatformDirsABC]:
-    if sys.platform == "win32":
-        from pip._vendor.platformdirs.windows import Windows as Result
-    elif sys.platform == "darwin":
-        from pip._vendor.platformdirs.macos import MacOS as Result
-    else:
-        from pip._vendor.platformdirs.unix import Unix as Result
-
-    if os.getenv("ANDROID_DATA") == "/data" and os.getenv("ANDROID_ROOT") == "/system":
-        if os.getenv("SHELL") or os.getenv("PREFIX"):
-            return Result
-
-        from pip._vendor.platformdirs.android import _android_folder
-
-        if _android_folder() is not None:
-            from pip._vendor.platformdirs.android import Android
-
-            return Android  # return to avoid redefinition of result
-
-    return Result
-
-
-PlatformDirs = _set_platform_dir_class()  #: Currently active platform
-AppDirs = PlatformDirs  #: Backwards compatibility with appdirs
-
-
-def user_data_dir(
-    appname: str | None = None,
-    appauthor: str | None | Literal[False] = None,
-    version: str | None = None,
-    roaming: bool = False,  # noqa: FBT001, FBT002
-    ensure_exists: bool = False,  # noqa: FBT001, FBT002
-) -> str:
-    """
-    :param appname: See `appname `.
-    :param appauthor: See `appauthor `.
-    :param version: See `version `.
-    :param roaming: See `roaming `.
-    :param ensure_exists: See `ensure_exists `.
-    :returns: data directory tied to the user
-    """
-    return PlatformDirs(
-        appname=appname,
-        appauthor=appauthor,
-        version=version,
-        roaming=roaming,
-        ensure_exists=ensure_exists,
-    ).user_data_dir
-
-
-def site_data_dir(
-    appname: str | None = None,
-    appauthor: str | None | Literal[False] = None,
-    version: str | None = None,
-    multipath: bool = False,  # noqa: FBT001, FBT002
-    ensure_exists: bool = False,  # noqa: FBT001, FBT002
-) -> str:
-    """
-    :param appname: See `appname `.
-    :param appauthor: See `appauthor `.
-    :param version: See `version `.
-    :param multipath: See `roaming `.
-    :param ensure_exists: See `ensure_exists `.
-    :returns: data directory shared by users
-    """
-    return PlatformDirs(
-        appname=appname,
-        appauthor=appauthor,
-        version=version,
-        multipath=multipath,
-        ensure_exists=ensure_exists,
-    ).site_data_dir
-
-
-def user_config_dir(
-    appname: str | None = None,
-    appauthor: str | None | Literal[False] = None,
-    version: str | None = None,
-    roaming: bool = False,  # noqa: FBT001, FBT002
-    ensure_exists: bool = False,  # noqa: FBT001, FBT002
-) -> str:
-    """
-    :param appname: See `appname `.
-    :param appauthor: See `appauthor `.
-    :param version: See `version `.
-    :param roaming: See `roaming `.
-    :param ensure_exists: See `ensure_exists `.
-    :returns: config directory tied to the user
-    """
-    return PlatformDirs(
-        appname=appname,
-        appauthor=appauthor,
-        version=version,
-        roaming=roaming,
-        ensure_exists=ensure_exists,
-    ).user_config_dir
-
-
-def site_config_dir(
-    appname: str | None = None,
-    appauthor: str | None | Literal[False] = None,
-    version: str | None = None,
-    multipath: bool = False,  # noqa: FBT001, FBT002
-    ensure_exists: bool = False,  # noqa: FBT001, FBT002
-) -> str:
-    """
-    :param appname: See `appname `.
-    :param appauthor: See `appauthor `.
-    :param version: See `version `.
-    :param multipath: See `roaming `.
-    :param ensure_exists: See `ensure_exists `.
-    :returns: config directory shared by the users
-    """
-    return PlatformDirs(
-        appname=appname,
-        appauthor=appauthor,
-        version=version,
-        multipath=multipath,
-        ensure_exists=ensure_exists,
-    ).site_config_dir
-
-
-def user_cache_dir(
-    appname: str | None = None,
-    appauthor: str | None | Literal[False] = None,
-    version: str | None = None,
-    opinion: bool = True,  # noqa: FBT001, FBT002
-    ensure_exists: bool = False,  # noqa: FBT001, FBT002
-) -> str:
-    """
-    :param appname: See `appname `.
-    :param appauthor: See `appauthor `.
-    :param version: See `version `.
-    :param opinion: See `roaming `.
-    :param ensure_exists: See `ensure_exists `.
-    :returns: cache directory tied to the user
-    """
-    return PlatformDirs(
-        appname=appname,
-        appauthor=appauthor,
-        version=version,
-        opinion=opinion,
-        ensure_exists=ensure_exists,
-    ).user_cache_dir
-
-
-def site_cache_dir(
-    appname: str | None = None,
-    appauthor: str | None | Literal[False] = None,
-    version: str | None = None,
-    opinion: bool = True,  # noqa: FBT001, FBT002
-    ensure_exists: bool = False,  # noqa: FBT001, FBT002
-) -> str:
-    """
-    :param appname: See `appname `.
-    :param appauthor: See `appauthor `.
-    :param version: See `version `.
-    :param opinion: See `opinion `.
-    :param ensure_exists: See `ensure_exists `.
-    :returns: cache directory tied to the user
-    """
-    return PlatformDirs(
-        appname=appname,
-        appauthor=appauthor,
-        version=version,
-        opinion=opinion,
-        ensure_exists=ensure_exists,
-    ).site_cache_dir
-
-
-def user_state_dir(
-    appname: str | None = None,
-    appauthor: str | None | Literal[False] = None,
-    version: str | None = None,
-    roaming: bool = False,  # noqa: FBT001, FBT002
-    ensure_exists: bool = False,  # noqa: FBT001, FBT002
-) -> str:
-    """
-    :param appname: See `appname `.
-    :param appauthor: See `appauthor `.
-    :param version: See `version `.
-    :param roaming: See `roaming `.
-    :param ensure_exists: See `ensure_exists `.
-    :returns: state directory tied to the user
-    """
-    return PlatformDirs(
-        appname=appname,
-        appauthor=appauthor,
-        version=version,
-        roaming=roaming,
-        ensure_exists=ensure_exists,
-    ).user_state_dir
-
-
-def user_log_dir(
-    appname: str | None = None,
-    appauthor: str | None | Literal[False] = None,
-    version: str | None = None,
-    opinion: bool = True,  # noqa: FBT001, FBT002
-    ensure_exists: bool = False,  # noqa: FBT001, FBT002
-) -> str:
-    """
-    :param appname: See `appname `.
-    :param appauthor: See `appauthor `.
-    :param version: See `version `.
-    :param opinion: See `roaming `.
-    :param ensure_exists: See `ensure_exists `.
-    :returns: log directory tied to the user
-    """
-    return PlatformDirs(
-        appname=appname,
-        appauthor=appauthor,
-        version=version,
-        opinion=opinion,
-        ensure_exists=ensure_exists,
-    ).user_log_dir
-
-
-def user_documents_dir() -> str:
-    """:returns: documents directory tied to the user"""
-    return PlatformDirs().user_documents_dir
-
-
-def user_downloads_dir() -> str:
-    """:returns: downloads directory tied to the user"""
-    return PlatformDirs().user_downloads_dir
-
-
-def user_pictures_dir() -> str:
-    """:returns: pictures directory tied to the user"""
-    return PlatformDirs().user_pictures_dir
-
-
-def user_videos_dir() -> str:
-    """:returns: videos directory tied to the user"""
-    return PlatformDirs().user_videos_dir
-
-
-def user_music_dir() -> str:
-    """:returns: music directory tied to the user"""
-    return PlatformDirs().user_music_dir
-
-
-def user_runtime_dir(
-    appname: str | None = None,
-    appauthor: str | None | Literal[False] = None,
-    version: str | None = None,
-    opinion: bool = True,  # noqa: FBT001, FBT002
-    ensure_exists: bool = False,  # noqa: FBT001, FBT002
-) -> str:
-    """
-    :param appname: See `appname `.
-    :param appauthor: See `appauthor `.
-    :param version: See `version `.
-    :param opinion: See `opinion `.
-    :param ensure_exists: See `ensure_exists `.
-    :returns: runtime directory tied to the user
-    """
-    return PlatformDirs(
-        appname=appname,
-        appauthor=appauthor,
-        version=version,
-        opinion=opinion,
-        ensure_exists=ensure_exists,
-    ).user_runtime_dir
-
-
-def user_data_path(
-    appname: str | None = None,
-    appauthor: str | None | Literal[False] = None,
-    version: str | None = None,
-    roaming: bool = False,  # noqa: FBT001, FBT002
-    ensure_exists: bool = False,  # noqa: FBT001, FBT002
-) -> Path:
-    """
-    :param appname: See `appname `.
-    :param appauthor: See `appauthor `.
-    :param version: See `version `.
-    :param roaming: See `roaming `.
-    :param ensure_exists: See `ensure_exists `.
-    :returns: data path tied to the user
-    """
-    return PlatformDirs(
-        appname=appname,
-        appauthor=appauthor,
-        version=version,
-        roaming=roaming,
-        ensure_exists=ensure_exists,
-    ).user_data_path
-
-
-def site_data_path(
-    appname: str | None = None,
-    appauthor: str | None | Literal[False] = None,
-    version: str | None = None,
-    multipath: bool = False,  # noqa: FBT001, FBT002
-    ensure_exists: bool = False,  # noqa: FBT001, FBT002
-) -> Path:
-    """
-    :param appname: See `appname `.
-    :param appauthor: See `appauthor `.
-    :param version: See `version `.
-    :param multipath: See `multipath `.
-    :param ensure_exists: See `ensure_exists `.
-    :returns: data path shared by users
-    """
-    return PlatformDirs(
-        appname=appname,
-        appauthor=appauthor,
-        version=version,
-        multipath=multipath,
-        ensure_exists=ensure_exists,
-    ).site_data_path
-
-
-def user_config_path(
-    appname: str | None = None,
-    appauthor: str | None | Literal[False] = None,
-    version: str | None = None,
-    roaming: bool = False,  # noqa: FBT001, FBT002
-    ensure_exists: bool = False,  # noqa: FBT001, FBT002
-) -> Path:
-    """
-    :param appname: See `appname `.
-    :param appauthor: See `appauthor `.
-    :param version: See `version `.
-    :param roaming: See `roaming `.
-    :param ensure_exists: See `ensure_exists `.
-    :returns: config path tied to the user
-    """
-    return PlatformDirs(
-        appname=appname,
-        appauthor=appauthor,
-        version=version,
-        roaming=roaming,
-        ensure_exists=ensure_exists,
-    ).user_config_path
-
-
-def site_config_path(
-    appname: str | None = None,
-    appauthor: str | None | Literal[False] = None,
-    version: str | None = None,
-    multipath: bool = False,  # noqa: FBT001, FBT002
-    ensure_exists: bool = False,  # noqa: FBT001, FBT002
-) -> Path:
-    """
-    :param appname: See `appname `.
-    :param appauthor: See `appauthor `.
-    :param version: See `version `.
-    :param multipath: See `roaming `.
-    :param ensure_exists: See `ensure_exists `.
-    :returns: config path shared by the users
-    """
-    return PlatformDirs(
-        appname=appname,
-        appauthor=appauthor,
-        version=version,
-        multipath=multipath,
-        ensure_exists=ensure_exists,
-    ).site_config_path
-
-
-def site_cache_path(
-    appname: str | None = None,
-    appauthor: str | None | Literal[False] = None,
-    version: str | None = None,
-    opinion: bool = True,  # noqa: FBT001, FBT002
-    ensure_exists: bool = False,  # noqa: FBT001, FBT002
-) -> Path:
-    """
-    :param appname: See `appname `.
-    :param appauthor: See `appauthor `.
-    :param version: See `version `.
-    :param opinion: See `opinion `.
-    :param ensure_exists: See `ensure_exists `.
-    :returns: cache directory tied to the user
-    """
-    return PlatformDirs(
-        appname=appname,
-        appauthor=appauthor,
-        version=version,
-        opinion=opinion,
-        ensure_exists=ensure_exists,
-    ).site_cache_path
-
-
-def user_cache_path(
-    appname: str | None = None,
-    appauthor: str | None | Literal[False] = None,
-    version: str | None = None,
-    opinion: bool = True,  # noqa: FBT001, FBT002
-    ensure_exists: bool = False,  # noqa: FBT001, FBT002
-) -> Path:
-    """
-    :param appname: See `appname `.
-    :param appauthor: See `appauthor `.
-    :param version: See `version `.
-    :param opinion: See `roaming `.
-    :param ensure_exists: See `ensure_exists `.
-    :returns: cache path tied to the user
-    """
-    return PlatformDirs(
-        appname=appname,
-        appauthor=appauthor,
-        version=version,
-        opinion=opinion,
-        ensure_exists=ensure_exists,
-    ).user_cache_path
-
-
-def user_state_path(
-    appname: str | None = None,
-    appauthor: str | None | Literal[False] = None,
-    version: str | None = None,
-    roaming: bool = False,  # noqa: FBT001, FBT002
-    ensure_exists: bool = False,  # noqa: FBT001, FBT002
-) -> Path:
-    """
-    :param appname: See `appname `.
-    :param appauthor: See `appauthor `.
-    :param version: See `version `.
-    :param roaming: See `roaming `.
-    :param ensure_exists: See `ensure_exists `.
-    :returns: state path tied to the user
-    """
-    return PlatformDirs(
-        appname=appname,
-        appauthor=appauthor,
-        version=version,
-        roaming=roaming,
-        ensure_exists=ensure_exists,
-    ).user_state_path
-
-
-def user_log_path(
-    appname: str | None = None,
-    appauthor: str | None | Literal[False] = None,
-    version: str | None = None,
-    opinion: bool = True,  # noqa: FBT001, FBT002
-    ensure_exists: bool = False,  # noqa: FBT001, FBT002
-) -> Path:
-    """
-    :param appname: See `appname `.
-    :param appauthor: See `appauthor `.
-    :param version: See `version `.
-    :param opinion: See `roaming `.
-    :param ensure_exists: See `ensure_exists `.
-    :returns: log path tied to the user
-    """
-    return PlatformDirs(
-        appname=appname,
-        appauthor=appauthor,
-        version=version,
-        opinion=opinion,
-        ensure_exists=ensure_exists,
-    ).user_log_path
-
-
-def user_documents_path() -> Path:
-    """:returns: documents path tied to the user"""
-    return PlatformDirs().user_documents_path
-
-
-def user_downloads_path() -> Path:
-    """:returns: downloads path tied to the user"""
-    return PlatformDirs().user_downloads_path
-
-
-def user_pictures_path() -> Path:
-    """:returns: pictures path tied to the user"""
-    return PlatformDirs().user_pictures_path
-
-
-def user_videos_path() -> Path:
-    """:returns: videos path tied to the user"""
-    return PlatformDirs().user_videos_path
-
-
-def user_music_path() -> Path:
-    """:returns: music path tied to the user"""
-    return PlatformDirs().user_music_path
-
-
-def user_runtime_path(
-    appname: str | None = None,
-    appauthor: str | None | Literal[False] = None,
-    version: str | None = None,
-    opinion: bool = True,  # noqa: FBT001, FBT002
-    ensure_exists: bool = False,  # noqa: FBT001, FBT002
-) -> Path:
-    """
-    :param appname: See `appname `.
-    :param appauthor: See `appauthor `.
-    :param version: See `version `.
-    :param opinion: See `opinion `.
-    :param ensure_exists: See `ensure_exists `.
-    :returns: runtime path tied to the user
-    """
-    return PlatformDirs(
-        appname=appname,
-        appauthor=appauthor,
-        version=version,
-        opinion=opinion,
-        ensure_exists=ensure_exists,
-    ).user_runtime_path
-
-
-__all__ = [
-    "__version__",
-    "__version_info__",
-    "PlatformDirs",
-    "AppDirs",
-    "PlatformDirsABC",
-    "user_data_dir",
-    "user_config_dir",
-    "user_cache_dir",
-    "user_state_dir",
-    "user_log_dir",
-    "user_documents_dir",
-    "user_downloads_dir",
-    "user_pictures_dir",
-    "user_videos_dir",
-    "user_music_dir",
-    "user_runtime_dir",
-    "site_data_dir",
-    "site_config_dir",
-    "site_cache_dir",
-    "user_data_path",
-    "user_config_path",
-    "user_cache_path",
-    "user_state_path",
-    "user_log_path",
-    "user_documents_path",
-    "user_downloads_path",
-    "user_pictures_path",
-    "user_videos_path",
-    "user_music_path",
-    "user_runtime_path",
-    "site_data_path",
-    "site_config_path",
-    "site_cache_path",
-]
diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/platformdirs/__main__.py b/venv/lib/python3.11/site-packages/pip/_vendor/platformdirs/__main__.py
deleted file mode 100644
index 6a0d6dd..0000000
--- a/venv/lib/python3.11/site-packages/pip/_vendor/platformdirs/__main__.py
+++ /dev/null
@@ -1,53 +0,0 @@
-"""Main entry point."""
-from __future__ import annotations
-
-from pip._vendor.platformdirs import PlatformDirs, __version__
-
-PROPS = (
-    "user_data_dir",
-    "user_config_dir",
-    "user_cache_dir",
-    "user_state_dir",
-    "user_log_dir",
-    "user_documents_dir",
-    "user_downloads_dir",
-    "user_pictures_dir",
-    "user_videos_dir",
-    "user_music_dir",
-    "user_runtime_dir",
-    "site_data_dir",
-    "site_config_dir",
-    "site_cache_dir",
-)
-
-
-def main() -> None:
-    """Run main entry point."""
-    app_name = "MyApp"
-    app_author = "MyCompany"
-
-    print(f"-- platformdirs {__version__} --")  # noqa: T201
-
-    print("-- app dirs (with optional 'version')")  # noqa: T201
-    dirs = PlatformDirs(app_name, app_author, version="1.0")
-    for prop in PROPS:
-        print(f"{prop}: {getattr(dirs, prop)}")  # noqa: T201
-
-    print("\n-- app dirs (without optional 'version')")  # noqa: T201
-    dirs = PlatformDirs(app_name, app_author)
-    for prop in PROPS:
-        print(f"{prop}: {getattr(dirs, prop)}")  # noqa: T201
-
-    print("\n-- app dirs (without optional 'appauthor')")  # noqa: T201
-    dirs = PlatformDirs(app_name)
-    for prop in PROPS:
-        print(f"{prop}: {getattr(dirs, prop)}")  # noqa: T201
-
-    print("\n-- app dirs (with disabled 'appauthor')")  # noqa: T201
-    dirs = PlatformDirs(app_name, appauthor=False)
-    for prop in PROPS:
-        print(f"{prop}: {getattr(dirs, prop)}")  # noqa: T201
-
-
-if __name__ == "__main__":
-    main()
diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/platformdirs/__pycache__/__init__.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/platformdirs/__pycache__/__init__.cpython-311.pyc
deleted file mode 100644
index 3d52bce..0000000
Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/platformdirs/__pycache__/__init__.cpython-311.pyc and /dev/null differ
diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/platformdirs/__pycache__/__main__.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/platformdirs/__pycache__/__main__.cpython-311.pyc
deleted file mode 100644
index a87afc2..0000000
Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/platformdirs/__pycache__/__main__.cpython-311.pyc and /dev/null differ
diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/platformdirs/__pycache__/android.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/platformdirs/__pycache__/android.cpython-311.pyc
deleted file mode 100644
index 81d4b43..0000000
Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/platformdirs/__pycache__/android.cpython-311.pyc and /dev/null differ
diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/platformdirs/__pycache__/api.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/platformdirs/__pycache__/api.cpython-311.pyc
deleted file mode 100644
index 0b9d365..0000000
Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/platformdirs/__pycache__/api.cpython-311.pyc and /dev/null differ
diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/platformdirs/__pycache__/macos.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/platformdirs/__pycache__/macos.cpython-311.pyc
deleted file mode 100644
index 8d5ea7a..0000000
Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/platformdirs/__pycache__/macos.cpython-311.pyc and /dev/null differ
diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/platformdirs/__pycache__/unix.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/platformdirs/__pycache__/unix.cpython-311.pyc
deleted file mode 100644
index 2ab18fa..0000000
Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/platformdirs/__pycache__/unix.cpython-311.pyc and /dev/null differ
diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/platformdirs/__pycache__/version.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/platformdirs/__pycache__/version.cpython-311.pyc
deleted file mode 100644
index 1f561e1..0000000
Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/platformdirs/__pycache__/version.cpython-311.pyc and /dev/null differ
diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/platformdirs/__pycache__/windows.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/platformdirs/__pycache__/windows.cpython-311.pyc
deleted file mode 100644
index 386a76a..0000000
Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/platformdirs/__pycache__/windows.cpython-311.pyc and /dev/null differ
diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/platformdirs/android.py b/venv/lib/python3.11/site-packages/pip/_vendor/platformdirs/android.py
deleted file mode 100644
index 76527dd..0000000
--- a/venv/lib/python3.11/site-packages/pip/_vendor/platformdirs/android.py
+++ /dev/null
@@ -1,210 +0,0 @@
-"""Android."""
-from __future__ import annotations
-
-import os
-import re
-import sys
-from functools import lru_cache
-from typing import cast
-
-from .api import PlatformDirsABC
-
-
-class Android(PlatformDirsABC):
-    """
-    Follows the guidance `from here `_. Makes use of the
-    `appname `,
-    `version `,
-    `ensure_exists `.
-    """
-
-    @property
-    def user_data_dir(self) -> str:
-        """:return: data directory tied to the user, e.g. ``/data/user///files/``"""
-        return self._append_app_name_and_version(cast(str, _android_folder()), "files")
-
-    @property
-    def site_data_dir(self) -> str:
-        """:return: data directory shared by users, same as `user_data_dir`"""
-        return self.user_data_dir
-
-    @property
-    def user_config_dir(self) -> str:
-        """
-        :return: config directory tied to the user, e.g. \
-        ``/data/user///shared_prefs/``
-        """
-        return self._append_app_name_and_version(cast(str, _android_folder()), "shared_prefs")
-
-    @property
-    def site_config_dir(self) -> str:
-        """:return: config directory shared by the users, same as `user_config_dir`"""
-        return self.user_config_dir
-
-    @property
-    def user_cache_dir(self) -> str:
-        """:return: cache directory tied to the user, e.g. e.g. ``/data/user///cache/``"""
-        return self._append_app_name_and_version(cast(str, _android_folder()), "cache")
-
-    @property
-    def site_cache_dir(self) -> str:
-        """:return: cache directory shared by users, same as `user_cache_dir`"""
-        return self.user_cache_dir
-
-    @property
-    def user_state_dir(self) -> str:
-        """:return: state directory tied to the user, same as `user_data_dir`"""
-        return self.user_data_dir
-
-    @property
-    def user_log_dir(self) -> str:
-        """
-        :return: log directory tied to the user, same as `user_cache_dir` if not opinionated else ``log`` in it,
-          e.g. ``/data/user///cache//log``
-        """
-        path = self.user_cache_dir
-        if self.opinion:
-            path = os.path.join(path, "log")  # noqa: PTH118
-        return path
-
-    @property
-    def user_documents_dir(self) -> str:
-        """:return: documents directory tied to the user e.g. ``/storage/emulated/0/Documents``"""
-        return _android_documents_folder()
-
-    @property
-    def user_downloads_dir(self) -> str:
-        """:return: downloads directory tied to the user e.g. ``/storage/emulated/0/Downloads``"""
-        return _android_downloads_folder()
-
-    @property
-    def user_pictures_dir(self) -> str:
-        """:return: pictures directory tied to the user e.g. ``/storage/emulated/0/Pictures``"""
-        return _android_pictures_folder()
-
-    @property
-    def user_videos_dir(self) -> str:
-        """:return: videos directory tied to the user e.g. ``/storage/emulated/0/DCIM/Camera``"""
-        return _android_videos_folder()
-
-    @property
-    def user_music_dir(self) -> str:
-        """:return: music directory tied to the user e.g. ``/storage/emulated/0/Music``"""
-        return _android_music_folder()
-
-    @property
-    def user_runtime_dir(self) -> str:
-        """
-        :return: runtime directory tied to the user, same as `user_cache_dir` if not opinionated else ``tmp`` in it,
-          e.g. ``/data/user///cache//tmp``
-        """
-        path = self.user_cache_dir
-        if self.opinion:
-            path = os.path.join(path, "tmp")  # noqa: PTH118
-        return path
-
-
-@lru_cache(maxsize=1)
-def _android_folder() -> str | None:
-    """:return: base folder for the Android OS or None if it cannot be found"""
-    try:
-        # First try to get path to android app via pyjnius
-        from jnius import autoclass
-
-        context = autoclass("android.content.Context")
-        result: str | None = context.getFilesDir().getParentFile().getAbsolutePath()
-    except Exception:  # noqa: BLE001
-        # if fails find an android folder looking path on the sys.path
-        pattern = re.compile(r"/data/(data|user/\d+)/(.+)/files")
-        for path in sys.path:
-            if pattern.match(path):
-                result = path.split("/files")[0]
-                break
-        else:
-            result = None
-    return result
-
-
-@lru_cache(maxsize=1)
-def _android_documents_folder() -> str:
-    """:return: documents folder for the Android OS"""
-    # Get directories with pyjnius
-    try:
-        from jnius import autoclass
-
-        context = autoclass("android.content.Context")
-        environment = autoclass("android.os.Environment")
-        documents_dir: str = context.getExternalFilesDir(environment.DIRECTORY_DOCUMENTS).getAbsolutePath()
-    except Exception:  # noqa: BLE001
-        documents_dir = "/storage/emulated/0/Documents"
-
-    return documents_dir
-
-
-@lru_cache(maxsize=1)
-def _android_downloads_folder() -> str:
-    """:return: downloads folder for the Android OS"""
-    # Get directories with pyjnius
-    try:
-        from jnius import autoclass
-
-        context = autoclass("android.content.Context")
-        environment = autoclass("android.os.Environment")
-        downloads_dir: str = context.getExternalFilesDir(environment.DIRECTORY_DOWNLOADS).getAbsolutePath()
-    except Exception:  # noqa: BLE001
-        downloads_dir = "/storage/emulated/0/Downloads"
-
-    return downloads_dir
-
-
-@lru_cache(maxsize=1)
-def _android_pictures_folder() -> str:
-    """:return: pictures folder for the Android OS"""
-    # Get directories with pyjnius
-    try:
-        from jnius import autoclass
-
-        context = autoclass("android.content.Context")
-        environment = autoclass("android.os.Environment")
-        pictures_dir: str = context.getExternalFilesDir(environment.DIRECTORY_PICTURES).getAbsolutePath()
-    except Exception:  # noqa: BLE001
-        pictures_dir = "/storage/emulated/0/Pictures"
-
-    return pictures_dir
-
-
-@lru_cache(maxsize=1)
-def _android_videos_folder() -> str:
-    """:return: videos folder for the Android OS"""
-    # Get directories with pyjnius
-    try:
-        from jnius import autoclass
-
-        context = autoclass("android.content.Context")
-        environment = autoclass("android.os.Environment")
-        videos_dir: str = context.getExternalFilesDir(environment.DIRECTORY_DCIM).getAbsolutePath()
-    except Exception:  # noqa: BLE001
-        videos_dir = "/storage/emulated/0/DCIM/Camera"
-
-    return videos_dir
-
-
-@lru_cache(maxsize=1)
-def _android_music_folder() -> str:
-    """:return: music folder for the Android OS"""
-    # Get directories with pyjnius
-    try:
-        from jnius import autoclass
-
-        context = autoclass("android.content.Context")
-        environment = autoclass("android.os.Environment")
-        music_dir: str = context.getExternalFilesDir(environment.DIRECTORY_MUSIC).getAbsolutePath()
-    except Exception:  # noqa: BLE001
-        music_dir = "/storage/emulated/0/Music"
-
-    return music_dir
-
-
-__all__ = [
-    "Android",
-]
diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/platformdirs/api.py b/venv/lib/python3.11/site-packages/pip/_vendor/platformdirs/api.py
deleted file mode 100644
index d64ebb9..0000000
--- a/venv/lib/python3.11/site-packages/pip/_vendor/platformdirs/api.py
+++ /dev/null
@@ -1,223 +0,0 @@
-"""Base API."""
-from __future__ import annotations
-
-import os
-from abc import ABC, abstractmethod
-from pathlib import Path
-from typing import TYPE_CHECKING
-
-if TYPE_CHECKING:
-    import sys
-
-    if sys.version_info >= (3, 8):  # pragma: no cover (py38+)
-        from typing import Literal
-    else:  # pragma: no cover (py38+)
-        from pip._vendor.typing_extensions import Literal
-
-
-class PlatformDirsABC(ABC):
-    """Abstract base class for platform directories."""
-
-    def __init__(  # noqa: PLR0913
-        self,
-        appname: str | None = None,
-        appauthor: str | None | Literal[False] = None,
-        version: str | None = None,
-        roaming: bool = False,  # noqa: FBT001, FBT002
-        multipath: bool = False,  # noqa: FBT001, FBT002
-        opinion: bool = True,  # noqa: FBT001, FBT002
-        ensure_exists: bool = False,  # noqa: FBT001, FBT002
-    ) -> None:
-        """
-        Create a new platform directory.
-
-        :param appname: See `appname`.
-        :param appauthor: See `appauthor`.
-        :param version: See `version`.
-        :param roaming: See `roaming`.
-        :param multipath: See `multipath`.
-        :param opinion: See `opinion`.
-        :param ensure_exists: See `ensure_exists`.
-        """
-        self.appname = appname  #: The name of application.
-        self.appauthor = appauthor
-        """
-        The name of the app author or distributing body for this application. Typically, it is the owning company name.
-        Defaults to `appname`. You may pass ``False`` to disable it.
-        """
-        self.version = version
-        """
-        An optional version path element to append to the path. You might want to use this if you want multiple versions
-        of your app to be able to run independently. If used, this would typically be ``.``.
-        """
-        self.roaming = roaming
-        """
-        Whether to use the roaming appdata directory on Windows. That means that for users on a Windows network setup
-        for roaming profiles, this user data will be synced on login (see
-        `here `_).
-        """
-        self.multipath = multipath
-        """
-        An optional parameter only applicable to Unix/Linux which indicates that the entire list of data dirs should be
-        returned. By default, the first item would only be returned.
-        """
-        self.opinion = opinion  #: A flag to indicating to use opinionated values.
-        self.ensure_exists = ensure_exists
-        """
-        Optionally create the directory (and any missing parents) upon access if it does not exist.
-        By default, no directories are created.
-        """
-
-    def _append_app_name_and_version(self, *base: str) -> str:
-        params = list(base[1:])
-        if self.appname:
-            params.append(self.appname)
-            if self.version:
-                params.append(self.version)
-        path = os.path.join(base[0], *params)  # noqa: PTH118
-        self._optionally_create_directory(path)
-        return path
-
-    def _optionally_create_directory(self, path: str) -> None:
-        if self.ensure_exists:
-            Path(path).mkdir(parents=True, exist_ok=True)
-
-    @property
-    @abstractmethod
-    def user_data_dir(self) -> str:
-        """:return: data directory tied to the user"""
-
-    @property
-    @abstractmethod
-    def site_data_dir(self) -> str:
-        """:return: data directory shared by users"""
-
-    @property
-    @abstractmethod
-    def user_config_dir(self) -> str:
-        """:return: config directory tied to the user"""
-
-    @property
-    @abstractmethod
-    def site_config_dir(self) -> str:
-        """:return: config directory shared by the users"""
-
-    @property
-    @abstractmethod
-    def user_cache_dir(self) -> str:
-        """:return: cache directory tied to the user"""
-
-    @property
-    @abstractmethod
-    def site_cache_dir(self) -> str:
-        """:return: cache directory shared by users"""
-
-    @property
-    @abstractmethod
-    def user_state_dir(self) -> str:
-        """:return: state directory tied to the user"""
-
-    @property
-    @abstractmethod
-    def user_log_dir(self) -> str:
-        """:return: log directory tied to the user"""
-
-    @property
-    @abstractmethod
-    def user_documents_dir(self) -> str:
-        """:return: documents directory tied to the user"""
-
-    @property
-    @abstractmethod
-    def user_downloads_dir(self) -> str:
-        """:return: downloads directory tied to the user"""
-
-    @property
-    @abstractmethod
-    def user_pictures_dir(self) -> str:
-        """:return: pictures directory tied to the user"""
-
-    @property
-    @abstractmethod
-    def user_videos_dir(self) -> str:
-        """:return: videos directory tied to the user"""
-
-    @property
-    @abstractmethod
-    def user_music_dir(self) -> str:
-        """:return: music directory tied to the user"""
-
-    @property
-    @abstractmethod
-    def user_runtime_dir(self) -> str:
-        """:return: runtime directory tied to the user"""
-
-    @property
-    def user_data_path(self) -> Path:
-        """:return: data path tied to the user"""
-        return Path(self.user_data_dir)
-
-    @property
-    def site_data_path(self) -> Path:
-        """:return: data path shared by users"""
-        return Path(self.site_data_dir)
-
-    @property
-    def user_config_path(self) -> Path:
-        """:return: config path tied to the user"""
-        return Path(self.user_config_dir)
-
-    @property
-    def site_config_path(self) -> Path:
-        """:return: config path shared by the users"""
-        return Path(self.site_config_dir)
-
-    @property
-    def user_cache_path(self) -> Path:
-        """:return: cache path tied to the user"""
-        return Path(self.user_cache_dir)
-
-    @property
-    def site_cache_path(self) -> Path:
-        """:return: cache path shared by users"""
-        return Path(self.site_cache_dir)
-
-    @property
-    def user_state_path(self) -> Path:
-        """:return: state path tied to the user"""
-        return Path(self.user_state_dir)
-
-    @property
-    def user_log_path(self) -> Path:
-        """:return: log path tied to the user"""
-        return Path(self.user_log_dir)
-
-    @property
-    def user_documents_path(self) -> Path:
-        """:return: documents path tied to the user"""
-        return Path(self.user_documents_dir)
-
-    @property
-    def user_downloads_path(self) -> Path:
-        """:return: downloads path tied to the user"""
-        return Path(self.user_downloads_dir)
-
-    @property
-    def user_pictures_path(self) -> Path:
-        """:return: pictures path tied to the user"""
-        return Path(self.user_pictures_dir)
-
-    @property
-    def user_videos_path(self) -> Path:
-        """:return: videos path tied to the user"""
-        return Path(self.user_videos_dir)
-
-    @property
-    def user_music_path(self) -> Path:
-        """:return: music path tied to the user"""
-        return Path(self.user_music_dir)
-
-    @property
-    def user_runtime_path(self) -> Path:
-        """:return: runtime path tied to the user"""
-        return Path(self.user_runtime_dir)
diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/platformdirs/macos.py b/venv/lib/python3.11/site-packages/pip/_vendor/platformdirs/macos.py
deleted file mode 100644
index a753e2a..0000000
--- a/venv/lib/python3.11/site-packages/pip/_vendor/platformdirs/macos.py
+++ /dev/null
@@ -1,91 +0,0 @@
-"""macOS."""
-from __future__ import annotations
-
-import os.path
-
-from .api import PlatformDirsABC
-
-
-class MacOS(PlatformDirsABC):
-    """
-    Platform directories for the macOS operating system. Follows the guidance from `Apple documentation
-    `_.
-    Makes use of the `appname `,
-    `version `,
-    `ensure_exists `.
-    """
-
-    @property
-    def user_data_dir(self) -> str:
-        """:return: data directory tied to the user, e.g. ``~/Library/Application Support/$appname/$version``"""
-        return self._append_app_name_and_version(os.path.expanduser("~/Library/Application Support"))  # noqa: PTH111
-
-    @property
-    def site_data_dir(self) -> str:
-        """:return: data directory shared by users, e.g. ``/Library/Application Support/$appname/$version``"""
-        return self._append_app_name_and_version("/Library/Application Support")
-
-    @property
-    def user_config_dir(self) -> str:
-        """:return: config directory tied to the user, same as `user_data_dir`"""
-        return self.user_data_dir
-
-    @property
-    def site_config_dir(self) -> str:
-        """:return: config directory shared by the users, same as `site_data_dir`"""
-        return self.site_data_dir
-
-    @property
-    def user_cache_dir(self) -> str:
-        """:return: cache directory tied to the user, e.g. ``~/Library/Caches/$appname/$version``"""
-        return self._append_app_name_and_version(os.path.expanduser("~/Library/Caches"))  # noqa: PTH111
-
-    @property
-    def site_cache_dir(self) -> str:
-        """:return: cache directory shared by users, e.g. ``/Library/Caches/$appname/$version``"""
-        return self._append_app_name_and_version("/Library/Caches")
-
-    @property
-    def user_state_dir(self) -> str:
-        """:return: state directory tied to the user, same as `user_data_dir`"""
-        return self.user_data_dir
-
-    @property
-    def user_log_dir(self) -> str:
-        """:return: log directory tied to the user, e.g. ``~/Library/Logs/$appname/$version``"""
-        return self._append_app_name_and_version(os.path.expanduser("~/Library/Logs"))  # noqa: PTH111
-
-    @property
-    def user_documents_dir(self) -> str:
-        """:return: documents directory tied to the user, e.g. ``~/Documents``"""
-        return os.path.expanduser("~/Documents")  # noqa: PTH111
-
-    @property
-    def user_downloads_dir(self) -> str:
-        """:return: downloads directory tied to the user, e.g. ``~/Downloads``"""
-        return os.path.expanduser("~/Downloads")  # noqa: PTH111
-
-    @property
-    def user_pictures_dir(self) -> str:
-        """:return: pictures directory tied to the user, e.g. ``~/Pictures``"""
-        return os.path.expanduser("~/Pictures")  # noqa: PTH111
-
-    @property
-    def user_videos_dir(self) -> str:
-        """:return: videos directory tied to the user, e.g. ``~/Movies``"""
-        return os.path.expanduser("~/Movies")  # noqa: PTH111
-
-    @property
-    def user_music_dir(self) -> str:
-        """:return: music directory tied to the user, e.g. ``~/Music``"""
-        return os.path.expanduser("~/Music")  # noqa: PTH111
-
-    @property
-    def user_runtime_dir(self) -> str:
-        """:return: runtime directory tied to the user, e.g. ``~/Library/Caches/TemporaryItems/$appname/$version``"""
-        return self._append_app_name_and_version(os.path.expanduser("~/Library/Caches/TemporaryItems"))  # noqa: PTH111
-
-
-__all__ = [
-    "MacOS",
-]
diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/platformdirs/unix.py b/venv/lib/python3.11/site-packages/pip/_vendor/platformdirs/unix.py
deleted file mode 100644
index 468b0ab..0000000
--- a/venv/lib/python3.11/site-packages/pip/_vendor/platformdirs/unix.py
+++ /dev/null
@@ -1,223 +0,0 @@
-"""Unix."""
-from __future__ import annotations
-
-import os
-import sys
-from configparser import ConfigParser
-from pathlib import Path
-
-from .api import PlatformDirsABC
-
-if sys.platform == "win32":
-
-    def getuid() -> int:
-        msg = "should only be used on Unix"
-        raise RuntimeError(msg)
-
-else:
-    from os import getuid
-
-
-class Unix(PlatformDirsABC):
-    """
-    On Unix/Linux, we follow the
-    `XDG Basedir Spec `_. The spec allows
-    overriding directories with environment variables. The examples show are the default values, alongside the name of
-    the environment variable that overrides them. Makes use of the
-    `appname `,
-    `version `,
-    `multipath `,
-    `opinion `,
-    `ensure_exists `.
-    """
-
-    @property
-    def user_data_dir(self) -> str:
-        """
-        :return: data directory tied to the user, e.g. ``~/.local/share/$appname/$version`` or
-         ``$XDG_DATA_HOME/$appname/$version``
-        """
-        path = os.environ.get("XDG_DATA_HOME", "")
-        if not path.strip():
-            path = os.path.expanduser("~/.local/share")  # noqa: PTH111
-        return self._append_app_name_and_version(path)
-
-    @property
-    def site_data_dir(self) -> str:
-        """
-        :return: data directories shared by users (if `multipath ` is
-         enabled and ``XDG_DATA_DIR`` is set and a multi path the response is also a multi path separated by the OS
-         path separator), e.g. ``/usr/local/share/$appname/$version`` or ``/usr/share/$appname/$version``
-        """
-        # XDG default for $XDG_DATA_DIRS; only first, if multipath is False
-        path = os.environ.get("XDG_DATA_DIRS", "")
-        if not path.strip():
-            path = f"/usr/local/share{os.pathsep}/usr/share"
-        return self._with_multi_path(path)
-
-    def _with_multi_path(self, path: str) -> str:
-        path_list = path.split(os.pathsep)
-        if not self.multipath:
-            path_list = path_list[0:1]
-        path_list = [self._append_app_name_and_version(os.path.expanduser(p)) for p in path_list]  # noqa: PTH111
-        return os.pathsep.join(path_list)
-
-    @property
-    def user_config_dir(self) -> str:
-        """
-        :return: config directory tied to the user, e.g. ``~/.config/$appname/$version`` or
-         ``$XDG_CONFIG_HOME/$appname/$version``
-        """
-        path = os.environ.get("XDG_CONFIG_HOME", "")
-        if not path.strip():
-            path = os.path.expanduser("~/.config")  # noqa: PTH111
-        return self._append_app_name_and_version(path)
-
-    @property
-    def site_config_dir(self) -> str:
-        """
-        :return: config directories shared by users (if `multipath `
-         is enabled and ``XDG_DATA_DIR`` is set and a multi path the response is also a multi path separated by the OS
-         path separator), e.g. ``/etc/xdg/$appname/$version``
-        """
-        # XDG default for $XDG_CONFIG_DIRS only first, if multipath is False
-        path = os.environ.get("XDG_CONFIG_DIRS", "")
-        if not path.strip():
-            path = "/etc/xdg"
-        return self._with_multi_path(path)
-
-    @property
-    def user_cache_dir(self) -> str:
-        """
-        :return: cache directory tied to the user, e.g. ``~/.cache/$appname/$version`` or
-         ``~/$XDG_CACHE_HOME/$appname/$version``
-        """
-        path = os.environ.get("XDG_CACHE_HOME", "")
-        if not path.strip():
-            path = os.path.expanduser("~/.cache")  # noqa: PTH111
-        return self._append_app_name_and_version(path)
-
-    @property
-    def site_cache_dir(self) -> str:
-        """:return: cache directory shared by users, e.g. ``/var/tmp/$appname/$version``"""
-        return self._append_app_name_and_version("/var/tmp")  # noqa: S108
-
-    @property
-    def user_state_dir(self) -> str:
-        """
-        :return: state directory tied to the user, e.g. ``~/.local/state/$appname/$version`` or
-         ``$XDG_STATE_HOME/$appname/$version``
-        """
-        path = os.environ.get("XDG_STATE_HOME", "")
-        if not path.strip():
-            path = os.path.expanduser("~/.local/state")  # noqa: PTH111
-        return self._append_app_name_and_version(path)
-
-    @property
-    def user_log_dir(self) -> str:
-        """:return: log directory tied to the user, same as `user_state_dir` if not opinionated else ``log`` in it"""
-        path = self.user_state_dir
-        if self.opinion:
-            path = os.path.join(path, "log")  # noqa: PTH118
-        return path
-
-    @property
-    def user_documents_dir(self) -> str:
-        """:return: documents directory tied to the user, e.g. ``~/Documents``"""
-        return _get_user_media_dir("XDG_DOCUMENTS_DIR", "~/Documents")
-
-    @property
-    def user_downloads_dir(self) -> str:
-        """:return: downloads directory tied to the user, e.g. ``~/Downloads``"""
-        return _get_user_media_dir("XDG_DOWNLOAD_DIR", "~/Downloads")
-
-    @property
-    def user_pictures_dir(self) -> str:
-        """:return: pictures directory tied to the user, e.g. ``~/Pictures``"""
-        return _get_user_media_dir("XDG_PICTURES_DIR", "~/Pictures")
-
-    @property
-    def user_videos_dir(self) -> str:
-        """:return: videos directory tied to the user, e.g. ``~/Videos``"""
-        return _get_user_media_dir("XDG_VIDEOS_DIR", "~/Videos")
-
-    @property
-    def user_music_dir(self) -> str:
-        """:return: music directory tied to the user, e.g. ``~/Music``"""
-        return _get_user_media_dir("XDG_MUSIC_DIR", "~/Music")
-
-    @property
-    def user_runtime_dir(self) -> str:
-        """
-        :return: runtime directory tied to the user, e.g. ``/run/user/$(id -u)/$appname/$version`` or
-         ``$XDG_RUNTIME_DIR/$appname/$version``.
-
-         For FreeBSD/OpenBSD/NetBSD, it would return ``/var/run/user/$(id -u)/$appname/$version`` if
-         exists, otherwise ``/tmp/runtime-$(id -u)/$appname/$version``, if``$XDG_RUNTIME_DIR``
-         is not set.
-        """
-        path = os.environ.get("XDG_RUNTIME_DIR", "")
-        if not path.strip():
-            if sys.platform.startswith(("freebsd", "openbsd", "netbsd")):
-                path = f"/var/run/user/{getuid()}"
-                if not Path(path).exists():
-                    path = f"/tmp/runtime-{getuid()}"  # noqa: S108
-            else:
-                path = f"/run/user/{getuid()}"
-        return self._append_app_name_and_version(path)
-
-    @property
-    def site_data_path(self) -> Path:
-        """:return: data path shared by users. Only return first item, even if ``multipath`` is set to ``True``"""
-        return self._first_item_as_path_if_multipath(self.site_data_dir)
-
-    @property
-    def site_config_path(self) -> Path:
-        """:return: config path shared by the users. Only return first item, even if ``multipath`` is set to ``True``"""
-        return self._first_item_as_path_if_multipath(self.site_config_dir)
-
-    @property
-    def site_cache_path(self) -> Path:
-        """:return: cache path shared by users. Only return first item, even if ``multipath`` is set to ``True``"""
-        return self._first_item_as_path_if_multipath(self.site_cache_dir)
-
-    def _first_item_as_path_if_multipath(self, directory: str) -> Path:
-        if self.multipath:
-            # If multipath is True, the first path is returned.
-            directory = directory.split(os.pathsep)[0]
-        return Path(directory)
-
-
-def _get_user_media_dir(env_var: str, fallback_tilde_path: str) -> str:
-    media_dir = _get_user_dirs_folder(env_var)
-    if media_dir is None:
-        media_dir = os.environ.get(env_var, "").strip()
-        if not media_dir:
-            media_dir = os.path.expanduser(fallback_tilde_path)  # noqa: PTH111
-
-    return media_dir
-
-
-def _get_user_dirs_folder(key: str) -> str | None:
-    """Return directory from user-dirs.dirs config file. See https://freedesktop.org/wiki/Software/xdg-user-dirs/."""
-    user_dirs_config_path = Path(Unix().user_config_dir) / "user-dirs.dirs"
-    if user_dirs_config_path.exists():
-        parser = ConfigParser()
-
-        with user_dirs_config_path.open() as stream:
-            # Add fake section header, so ConfigParser doesn't complain
-            parser.read_string(f"[top]\n{stream.read()}")
-
-        if key not in parser["top"]:
-            return None
-
-        path = parser["top"][key].strip('"')
-        # Handle relative home paths
-        return path.replace("$HOME", os.path.expanduser("~"))  # noqa: PTH111
-
-    return None
-
-
-__all__ = [
-    "Unix",
-]
diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/platformdirs/version.py b/venv/lib/python3.11/site-packages/pip/_vendor/platformdirs/version.py
deleted file mode 100644
index dc8c44c..0000000
--- a/venv/lib/python3.11/site-packages/pip/_vendor/platformdirs/version.py
+++ /dev/null
@@ -1,4 +0,0 @@
-# file generated by setuptools_scm
-# don't change, don't track in version control
-__version__ = version = '3.8.1'
-__version_tuple__ = version_tuple = (3, 8, 1)
diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/platformdirs/windows.py b/venv/lib/python3.11/site-packages/pip/_vendor/platformdirs/windows.py
deleted file mode 100644
index b52c9c6..0000000
--- a/venv/lib/python3.11/site-packages/pip/_vendor/platformdirs/windows.py
+++ /dev/null
@@ -1,255 +0,0 @@
-"""Windows."""
-from __future__ import annotations
-
-import ctypes
-import os
-import sys
-from functools import lru_cache
-from typing import TYPE_CHECKING
-
-from .api import PlatformDirsABC
-
-if TYPE_CHECKING:
-    from collections.abc import Callable
-
-
-class Windows(PlatformDirsABC):
-    """
-    `MSDN on where to store app data files
-    `_.
-    Makes use of the
-    `appname `,
-    `appauthor `,
-    `version `,
-    `roaming `,
-    `opinion `,
-    `ensure_exists `.
-    """
-
-    @property
-    def user_data_dir(self) -> str:
-        """
-        :return: data directory tied to the user, e.g.
-         ``%USERPROFILE%\\AppData\\Local\\$appauthor\\$appname`` (not roaming) or
-         ``%USERPROFILE%\\AppData\\Roaming\\$appauthor\\$appname`` (roaming)
-        """
-        const = "CSIDL_APPDATA" if self.roaming else "CSIDL_LOCAL_APPDATA"
-        path = os.path.normpath(get_win_folder(const))
-        return self._append_parts(path)
-
-    def _append_parts(self, path: str, *, opinion_value: str | None = None) -> str:
-        params = []
-        if self.appname:
-            if self.appauthor is not False:
-                author = self.appauthor or self.appname
-                params.append(author)
-            params.append(self.appname)
-            if opinion_value is not None and self.opinion:
-                params.append(opinion_value)
-            if self.version:
-                params.append(self.version)
-        path = os.path.join(path, *params)  # noqa: PTH118
-        self._optionally_create_directory(path)
-        return path
-
-    @property
-    def site_data_dir(self) -> str:
-        """:return: data directory shared by users, e.g. ``C:\\ProgramData\\$appauthor\\$appname``"""
-        path = os.path.normpath(get_win_folder("CSIDL_COMMON_APPDATA"))
-        return self._append_parts(path)
-
-    @property
-    def user_config_dir(self) -> str:
-        """:return: config directory tied to the user, same as `user_data_dir`"""
-        return self.user_data_dir
-
-    @property
-    def site_config_dir(self) -> str:
-        """:return: config directory shared by the users, same as `site_data_dir`"""
-        return self.site_data_dir
-
-    @property
-    def user_cache_dir(self) -> str:
-        """
-        :return: cache directory tied to the user (if opinionated with ``Cache`` folder within ``$appname``) e.g.
-         ``%USERPROFILE%\\AppData\\Local\\$appauthor\\$appname\\Cache\\$version``
-        """
-        path = os.path.normpath(get_win_folder("CSIDL_LOCAL_APPDATA"))
-        return self._append_parts(path, opinion_value="Cache")
-
-    @property
-    def site_cache_dir(self) -> str:
-        """:return: cache directory shared by users, e.g. ``C:\\ProgramData\\$appauthor\\$appname\\Cache\\$version``"""
-        path = os.path.normpath(get_win_folder("CSIDL_COMMON_APPDATA"))
-        return self._append_parts(path, opinion_value="Cache")
-
-    @property
-    def user_state_dir(self) -> str:
-        """:return: state directory tied to the user, same as `user_data_dir`"""
-        return self.user_data_dir
-
-    @property
-    def user_log_dir(self) -> str:
-        """:return: log directory tied to the user, same as `user_data_dir` if not opinionated else ``Logs`` in it"""
-        path = self.user_data_dir
-        if self.opinion:
-            path = os.path.join(path, "Logs")  # noqa: PTH118
-            self._optionally_create_directory(path)
-        return path
-
-    @property
-    def user_documents_dir(self) -> str:
-        """:return: documents directory tied to the user e.g. ``%USERPROFILE%\\Documents``"""
-        return os.path.normpath(get_win_folder("CSIDL_PERSONAL"))
-
-    @property
-    def user_downloads_dir(self) -> str:
-        """:return: downloads directory tied to the user e.g. ``%USERPROFILE%\\Downloads``"""
-        return os.path.normpath(get_win_folder("CSIDL_DOWNLOADS"))
-
-    @property
-    def user_pictures_dir(self) -> str:
-        """:return: pictures directory tied to the user e.g. ``%USERPROFILE%\\Pictures``"""
-        return os.path.normpath(get_win_folder("CSIDL_MYPICTURES"))
-
-    @property
-    def user_videos_dir(self) -> str:
-        """:return: videos directory tied to the user e.g. ``%USERPROFILE%\\Videos``"""
-        return os.path.normpath(get_win_folder("CSIDL_MYVIDEO"))
-
-    @property
-    def user_music_dir(self) -> str:
-        """:return: music directory tied to the user e.g. ``%USERPROFILE%\\Music``"""
-        return os.path.normpath(get_win_folder("CSIDL_MYMUSIC"))
-
-    @property
-    def user_runtime_dir(self) -> str:
-        """
-        :return: runtime directory tied to the user, e.g.
-         ``%USERPROFILE%\\AppData\\Local\\Temp\\$appauthor\\$appname``
-        """
-        path = os.path.normpath(os.path.join(get_win_folder("CSIDL_LOCAL_APPDATA"), "Temp"))  # noqa: PTH118
-        return self._append_parts(path)
-
-
-def get_win_folder_from_env_vars(csidl_name: str) -> str:
-    """Get folder from environment variables."""
-    result = get_win_folder_if_csidl_name_not_env_var(csidl_name)
-    if result is not None:
-        return result
-
-    env_var_name = {
-        "CSIDL_APPDATA": "APPDATA",
-        "CSIDL_COMMON_APPDATA": "ALLUSERSPROFILE",
-        "CSIDL_LOCAL_APPDATA": "LOCALAPPDATA",
-    }.get(csidl_name)
-    if env_var_name is None:
-        msg = f"Unknown CSIDL name: {csidl_name}"
-        raise ValueError(msg)
-    result = os.environ.get(env_var_name)
-    if result is None:
-        msg = f"Unset environment variable: {env_var_name}"
-        raise ValueError(msg)
-    return result
-
-
-def get_win_folder_if_csidl_name_not_env_var(csidl_name: str) -> str | None:
-    """Get folder for a CSIDL name that does not exist as an environment variable."""
-    if csidl_name == "CSIDL_PERSONAL":
-        return os.path.join(os.path.normpath(os.environ["USERPROFILE"]), "Documents")  # noqa: PTH118
-
-    if csidl_name == "CSIDL_DOWNLOADS":
-        return os.path.join(os.path.normpath(os.environ["USERPROFILE"]), "Downloads")  # noqa: PTH118
-
-    if csidl_name == "CSIDL_MYPICTURES":
-        return os.path.join(os.path.normpath(os.environ["USERPROFILE"]), "Pictures")  # noqa: PTH118
-
-    if csidl_name == "CSIDL_MYVIDEO":
-        return os.path.join(os.path.normpath(os.environ["USERPROFILE"]), "Videos")  # noqa: PTH118
-
-    if csidl_name == "CSIDL_MYMUSIC":
-        return os.path.join(os.path.normpath(os.environ["USERPROFILE"]), "Music")  # noqa: PTH118
-    return None
-
-
-def get_win_folder_from_registry(csidl_name: str) -> str:
-    """
-    Get folder from the registry.
-
-    This is a fallback technique at best. I'm not sure if using the registry for these guarantees us the correct answer
-    for all CSIDL_* names.
-    """
-    shell_folder_name = {
-        "CSIDL_APPDATA": "AppData",
-        "CSIDL_COMMON_APPDATA": "Common AppData",
-        "CSIDL_LOCAL_APPDATA": "Local AppData",
-        "CSIDL_PERSONAL": "Personal",
-        "CSIDL_DOWNLOADS": "{374DE290-123F-4565-9164-39C4925E467B}",
-        "CSIDL_MYPICTURES": "My Pictures",
-        "CSIDL_MYVIDEO": "My Video",
-        "CSIDL_MYMUSIC": "My Music",
-    }.get(csidl_name)
-    if shell_folder_name is None:
-        msg = f"Unknown CSIDL name: {csidl_name}"
-        raise ValueError(msg)
-    if sys.platform != "win32":  # only needed for mypy type checker to know that this code runs only on Windows
-        raise NotImplementedError
-    import winreg
-
-    key = winreg.OpenKey(winreg.HKEY_CURRENT_USER, r"Software\Microsoft\Windows\CurrentVersion\Explorer\Shell Folders")
-    directory, _ = winreg.QueryValueEx(key, shell_folder_name)
-    return str(directory)
-
-
-def get_win_folder_via_ctypes(csidl_name: str) -> str:
-    """Get folder with ctypes."""
-    # There is no 'CSIDL_DOWNLOADS'.
-    # Use 'CSIDL_PROFILE' (40) and append the default folder 'Downloads' instead.
-    # https://learn.microsoft.com/en-us/windows/win32/shell/knownfolderid
-
-    csidl_const = {
-        "CSIDL_APPDATA": 26,
-        "CSIDL_COMMON_APPDATA": 35,
-        "CSIDL_LOCAL_APPDATA": 28,
-        "CSIDL_PERSONAL": 5,
-        "CSIDL_MYPICTURES": 39,
-        "CSIDL_MYVIDEO": 14,
-        "CSIDL_MYMUSIC": 13,
-        "CSIDL_DOWNLOADS": 40,
-    }.get(csidl_name)
-    if csidl_const is None:
-        msg = f"Unknown CSIDL name: {csidl_name}"
-        raise ValueError(msg)
-
-    buf = ctypes.create_unicode_buffer(1024)
-    windll = getattr(ctypes, "windll")  # noqa: B009 # using getattr to avoid false positive with mypy type checker
-    windll.shell32.SHGetFolderPathW(None, csidl_const, None, 0, buf)
-
-    # Downgrade to short path name if it has highbit chars.
-    if any(ord(c) > 255 for c in buf):  # noqa: PLR2004
-        buf2 = ctypes.create_unicode_buffer(1024)
-        if windll.kernel32.GetShortPathNameW(buf.value, buf2, 1024):
-            buf = buf2
-
-    if csidl_name == "CSIDL_DOWNLOADS":
-        return os.path.join(buf.value, "Downloads")  # noqa: PTH118
-
-    return buf.value
-
-
-def _pick_get_win_folder() -> Callable[[str], str]:
-    if hasattr(ctypes, "windll"):
-        return get_win_folder_via_ctypes
-    try:
-        import winreg  # noqa: F401
-    except ImportError:
-        return get_win_folder_from_env_vars
-    else:
-        return get_win_folder_from_registry
-
-
-get_win_folder = lru_cache(maxsize=None)(_pick_get_win_folder())
-
-__all__ = [
-    "Windows",
-]
diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/pygments/__init__.py b/venv/lib/python3.11/site-packages/pip/_vendor/pygments/__init__.py
deleted file mode 100644
index 39c84aa..0000000
--- a/venv/lib/python3.11/site-packages/pip/_vendor/pygments/__init__.py
+++ /dev/null
@@ -1,82 +0,0 @@
-"""
-    Pygments
-    ~~~~~~~~
-
-    Pygments is a syntax highlighting package written in Python.
-
-    It is a generic syntax highlighter for general use in all kinds of software
-    such as forum systems, wikis or other applications that need to prettify
-    source code. Highlights are:
-
-    * a wide range of common languages and markup formats is supported
-    * special attention is paid to details, increasing quality by a fair amount
-    * support for new languages and formats are added easily
-    * a number of output formats, presently HTML, LaTeX, RTF, SVG, all image
-      formats that PIL supports, and ANSI sequences
-    * it is usable as a command-line tool and as a library
-    * ... and it highlights even Brainfuck!
-
-    The `Pygments master branch`_ is installable with ``easy_install Pygments==dev``.
-
-    .. _Pygments master branch:
-       https://github.com/pygments/pygments/archive/master.zip#egg=Pygments-dev
-
-    :copyright: Copyright 2006-2023 by the Pygments team, see AUTHORS.
-    :license: BSD, see LICENSE for details.
-"""
-from io import StringIO, BytesIO
-
-__version__ = '2.15.1'
-__docformat__ = 'restructuredtext'
-
-__all__ = ['lex', 'format', 'highlight']
-
-
-def lex(code, lexer):
-    """
-    Lex `code` with the `lexer` (must be a `Lexer` instance)
-    and return an iterable of tokens. Currently, this only calls
-    `lexer.get_tokens()`.
-    """
-    try:
-        return lexer.get_tokens(code)
-    except TypeError:
-        # Heuristic to catch a common mistake.
-        from pip._vendor.pygments.lexer import RegexLexer
-        if isinstance(lexer, type) and issubclass(lexer, RegexLexer):
-            raise TypeError('lex() argument must be a lexer instance, '
-                            'not a class')
-        raise
-
-
-def format(tokens, formatter, outfile=None):  # pylint: disable=redefined-builtin
-    """
-    Format ``tokens`` (an iterable of tokens) with the formatter ``formatter``
-    (a `Formatter` instance).
-
-    If ``outfile`` is given and a valid file object (an object with a
-    ``write`` method), the result will be written to it, otherwise it
-    is returned as a string.
-    """
-    try:
-        if not outfile:
-            realoutfile = getattr(formatter, 'encoding', None) and BytesIO() or StringIO()
-            formatter.format(tokens, realoutfile)
-            return realoutfile.getvalue()
-        else:
-            formatter.format(tokens, outfile)
-    except TypeError:
-        # Heuristic to catch a common mistake.
-        from pip._vendor.pygments.formatter import Formatter
-        if isinstance(formatter, type) and issubclass(formatter, Formatter):
-            raise TypeError('format() argument must be a formatter instance, '
-                            'not a class')
-        raise
-
-
-def highlight(code, lexer, formatter, outfile=None):
-    """
-    This is the most high-level highlighting function. It combines `lex` and
-    `format` in one function.
-    """
-    return format(lex(code, lexer), formatter, outfile)
diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/pygments/__main__.py b/venv/lib/python3.11/site-packages/pip/_vendor/pygments/__main__.py
deleted file mode 100644
index 2f7f8cb..0000000
--- a/venv/lib/python3.11/site-packages/pip/_vendor/pygments/__main__.py
+++ /dev/null
@@ -1,17 +0,0 @@
-"""
-    pygments.__main__
-    ~~~~~~~~~~~~~~~~~
-
-    Main entry point for ``python -m pygments``.
-
-    :copyright: Copyright 2006-2023 by the Pygments team, see AUTHORS.
-    :license: BSD, see LICENSE for details.
-"""
-
-import sys
-from pip._vendor.pygments.cmdline import main
-
-try:
-    sys.exit(main(sys.argv))
-except KeyboardInterrupt:
-    sys.exit(1)
diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/pygments/__pycache__/__init__.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/pygments/__pycache__/__init__.cpython-311.pyc
deleted file mode 100644
index 07792ed..0000000
Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/pygments/__pycache__/__init__.cpython-311.pyc and /dev/null differ
diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/pygments/__pycache__/__main__.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/pygments/__pycache__/__main__.cpython-311.pyc
deleted file mode 100644
index c677072..0000000
Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/pygments/__pycache__/__main__.cpython-311.pyc and /dev/null differ
diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/pygments/__pycache__/cmdline.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/pygments/__pycache__/cmdline.cpython-311.pyc
deleted file mode 100644
index 83efbc2..0000000
Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/pygments/__pycache__/cmdline.cpython-311.pyc and /dev/null differ
diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/pygments/__pycache__/console.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/pygments/__pycache__/console.cpython-311.pyc
deleted file mode 100644
index 9122e0f..0000000
Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/pygments/__pycache__/console.cpython-311.pyc and /dev/null differ
diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/pygments/__pycache__/filter.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/pygments/__pycache__/filter.cpython-311.pyc
deleted file mode 100644
index b085b7d..0000000
Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/pygments/__pycache__/filter.cpython-311.pyc and /dev/null differ
diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/pygments/__pycache__/formatter.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/pygments/__pycache__/formatter.cpython-311.pyc
deleted file mode 100644
index b2ebb20..0000000
Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/pygments/__pycache__/formatter.cpython-311.pyc and /dev/null differ
diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/pygments/__pycache__/lexer.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/pygments/__pycache__/lexer.cpython-311.pyc
deleted file mode 100644
index c049f22..0000000
Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/pygments/__pycache__/lexer.cpython-311.pyc and /dev/null differ
diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/pygments/__pycache__/modeline.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/pygments/__pycache__/modeline.cpython-311.pyc
deleted file mode 100644
index 7f61878..0000000
Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/pygments/__pycache__/modeline.cpython-311.pyc and /dev/null differ
diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/pygments/__pycache__/plugin.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/pygments/__pycache__/plugin.cpython-311.pyc
deleted file mode 100644
index 3fd16d4..0000000
Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/pygments/__pycache__/plugin.cpython-311.pyc and /dev/null differ
diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/pygments/__pycache__/regexopt.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/pygments/__pycache__/regexopt.cpython-311.pyc
deleted file mode 100644
index 1f47780..0000000
Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/pygments/__pycache__/regexopt.cpython-311.pyc and /dev/null differ
diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/pygments/__pycache__/scanner.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/pygments/__pycache__/scanner.cpython-311.pyc
deleted file mode 100644
index f7c54da..0000000
Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/pygments/__pycache__/scanner.cpython-311.pyc and /dev/null differ
diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/pygments/__pycache__/sphinxext.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/pygments/__pycache__/sphinxext.cpython-311.pyc
deleted file mode 100644
index ca709c5..0000000
Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/pygments/__pycache__/sphinxext.cpython-311.pyc and /dev/null differ
diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/pygments/__pycache__/style.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/pygments/__pycache__/style.cpython-311.pyc
deleted file mode 100644
index 8d13cc6..0000000
Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/pygments/__pycache__/style.cpython-311.pyc and /dev/null differ
diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/pygments/__pycache__/token.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/pygments/__pycache__/token.cpython-311.pyc
deleted file mode 100644
index 48b1fd6..0000000
Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/pygments/__pycache__/token.cpython-311.pyc and /dev/null differ
diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/pygments/__pycache__/unistring.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/pygments/__pycache__/unistring.cpython-311.pyc
deleted file mode 100644
index 8e37aa2..0000000
Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/pygments/__pycache__/unistring.cpython-311.pyc and /dev/null differ
diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/pygments/__pycache__/util.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/pygments/__pycache__/util.cpython-311.pyc
deleted file mode 100644
index cfed31f..0000000
Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/pygments/__pycache__/util.cpython-311.pyc and /dev/null differ
diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/pygments/cmdline.py b/venv/lib/python3.11/site-packages/pip/_vendor/pygments/cmdline.py
deleted file mode 100644
index eec1775..0000000
--- a/venv/lib/python3.11/site-packages/pip/_vendor/pygments/cmdline.py
+++ /dev/null
@@ -1,668 +0,0 @@
-"""
-    pygments.cmdline
-    ~~~~~~~~~~~~~~~~
-
-    Command line interface.
-
-    :copyright: Copyright 2006-2023 by the Pygments team, see AUTHORS.
-    :license: BSD, see LICENSE for details.
-"""
-
-import os
-import sys
-import shutil
-import argparse
-from textwrap import dedent
-
-from pip._vendor.pygments import __version__, highlight
-from pip._vendor.pygments.util import ClassNotFound, OptionError, docstring_headline, \
-    guess_decode, guess_decode_from_terminal, terminal_encoding, \
-    UnclosingTextIOWrapper
-from pip._vendor.pygments.lexers import get_all_lexers, get_lexer_by_name, guess_lexer, \
-    load_lexer_from_file, get_lexer_for_filename, find_lexer_class_for_filename
-from pip._vendor.pygments.lexers.special import TextLexer
-from pip._vendor.pygments.formatters.latex import LatexEmbeddedLexer, LatexFormatter
-from pip._vendor.pygments.formatters import get_all_formatters, get_formatter_by_name, \
-    load_formatter_from_file, get_formatter_for_filename, find_formatter_class
-from pip._vendor.pygments.formatters.terminal import TerminalFormatter
-from pip._vendor.pygments.formatters.terminal256 import Terminal256Formatter, TerminalTrueColorFormatter
-from pip._vendor.pygments.filters import get_all_filters, find_filter_class
-from pip._vendor.pygments.styles import get_all_styles, get_style_by_name
-
-
-def _parse_options(o_strs):
-    opts = {}
-    if not o_strs:
-        return opts
-    for o_str in o_strs:
-        if not o_str.strip():
-            continue
-        o_args = o_str.split(',')
-        for o_arg in o_args:
-            o_arg = o_arg.strip()
-            try:
-                o_key, o_val = o_arg.split('=', 1)
-                o_key = o_key.strip()
-                o_val = o_val.strip()
-            except ValueError:
-                opts[o_arg] = True
-            else:
-                opts[o_key] = o_val
-    return opts
-
-
-def _parse_filters(f_strs):
-    filters = []
-    if not f_strs:
-        return filters
-    for f_str in f_strs:
-        if ':' in f_str:
-            fname, fopts = f_str.split(':', 1)
-            filters.append((fname, _parse_options([fopts])))
-        else:
-            filters.append((f_str, {}))
-    return filters
-
-
-def _print_help(what, name):
-    try:
-        if what == 'lexer':
-            cls = get_lexer_by_name(name)
-            print("Help on the %s lexer:" % cls.name)
-            print(dedent(cls.__doc__))
-        elif what == 'formatter':
-            cls = find_formatter_class(name)
-            print("Help on the %s formatter:" % cls.name)
-            print(dedent(cls.__doc__))
-        elif what == 'filter':
-            cls = find_filter_class(name)
-            print("Help on the %s filter:" % name)
-            print(dedent(cls.__doc__))
-        return 0
-    except (AttributeError, ValueError):
-        print("%s not found!" % what, file=sys.stderr)
-        return 1
-
-
-def _print_list(what):
-    if what == 'lexer':
-        print()
-        print("Lexers:")
-        print("~~~~~~~")
-
-        info = []
-        for fullname, names, exts, _ in get_all_lexers():
-            tup = (', '.join(names)+':', fullname,
-                   exts and '(filenames ' + ', '.join(exts) + ')' or '')
-            info.append(tup)
-        info.sort()
-        for i in info:
-            print(('* %s\n    %s %s') % i)
-
-    elif what == 'formatter':
-        print()
-        print("Formatters:")
-        print("~~~~~~~~~~~")
-
-        info = []
-        for cls in get_all_formatters():
-            doc = docstring_headline(cls)
-            tup = (', '.join(cls.aliases) + ':', doc, cls.filenames and
-                   '(filenames ' + ', '.join(cls.filenames) + ')' or '')
-            info.append(tup)
-        info.sort()
-        for i in info:
-            print(('* %s\n    %s %s') % i)
-
-    elif what == 'filter':
-        print()
-        print("Filters:")
-        print("~~~~~~~~")
-
-        for name in get_all_filters():
-            cls = find_filter_class(name)
-            print("* " + name + ':')
-            print("    %s" % docstring_headline(cls))
-
-    elif what == 'style':
-        print()
-        print("Styles:")
-        print("~~~~~~~")
-
-        for name in get_all_styles():
-            cls = get_style_by_name(name)
-            print("* " + name + ':')
-            print("    %s" % docstring_headline(cls))
-
-
-def _print_list_as_json(requested_items):
-    import json
-    result = {}
-    if 'lexer' in requested_items:
-        info = {}
-        for fullname, names, filenames, mimetypes in get_all_lexers():
-            info[fullname] = {
-                'aliases': names,
-                'filenames': filenames,
-                'mimetypes': mimetypes
-            }
-        result['lexers'] = info
-
-    if 'formatter' in requested_items:
-        info = {}
-        for cls in get_all_formatters():
-            doc = docstring_headline(cls)
-            info[cls.name] = {
-                'aliases': cls.aliases,
-                'filenames': cls.filenames,
-                'doc': doc
-            }
-        result['formatters'] = info
-
-    if 'filter' in requested_items:
-        info = {}
-        for name in get_all_filters():
-            cls = find_filter_class(name)
-            info[name] = {
-                'doc': docstring_headline(cls)
-            }
-        result['filters'] = info
-
-    if 'style' in requested_items:
-        info = {}
-        for name in get_all_styles():
-            cls = get_style_by_name(name)
-            info[name] = {
-                'doc': docstring_headline(cls)
-            }
-        result['styles'] = info
-
-    json.dump(result, sys.stdout)
-
-def main_inner(parser, argns):
-    if argns.help:
-        parser.print_help()
-        return 0
-
-    if argns.V:
-        print('Pygments version %s, (c) 2006-2023 by Georg Brandl, Matthäus '
-              'Chajdas and contributors.' % __version__)
-        return 0
-
-    def is_only_option(opt):
-        return not any(v for (k, v) in vars(argns).items() if k != opt)
-
-    # handle ``pygmentize -L``
-    if argns.L is not None:
-        arg_set = set()
-        for k, v in vars(argns).items():
-            if v:
-                arg_set.add(k)
-
-        arg_set.discard('L')
-        arg_set.discard('json')
-
-        if arg_set:
-            parser.print_help(sys.stderr)
-            return 2
-
-        # print version
-        if not argns.json:
-            main(['', '-V'])
-        allowed_types = {'lexer', 'formatter', 'filter', 'style'}
-        largs = [arg.rstrip('s') for arg in argns.L]
-        if any(arg not in allowed_types for arg in largs):
-            parser.print_help(sys.stderr)
-            return 0
-        if not largs:
-            largs = allowed_types
-        if not argns.json:
-            for arg in largs:
-                _print_list(arg)
-        else:
-            _print_list_as_json(largs)
-        return 0
-
-    # handle ``pygmentize -H``
-    if argns.H:
-        if not is_only_option('H'):
-            parser.print_help(sys.stderr)
-            return 2
-        what, name = argns.H
-        if what not in ('lexer', 'formatter', 'filter'):
-            parser.print_help(sys.stderr)
-            return 2
-        return _print_help(what, name)
-
-    # parse -O options
-    parsed_opts = _parse_options(argns.O or [])
-
-    # parse -P options
-    for p_opt in argns.P or []:
-        try:
-            name, value = p_opt.split('=', 1)
-        except ValueError:
-            parsed_opts[p_opt] = True
-        else:
-            parsed_opts[name] = value
-
-    # encodings
-    inencoding = parsed_opts.get('inencoding', parsed_opts.get('encoding'))
-    outencoding = parsed_opts.get('outencoding', parsed_opts.get('encoding'))
-
-    # handle ``pygmentize -N``
-    if argns.N:
-        lexer = find_lexer_class_for_filename(argns.N)
-        if lexer is None:
-            lexer = TextLexer
-
-        print(lexer.aliases[0])
-        return 0
-
-    # handle ``pygmentize -C``
-    if argns.C:
-        inp = sys.stdin.buffer.read()
-        try:
-            lexer = guess_lexer(inp, inencoding=inencoding)
-        except ClassNotFound:
-            lexer = TextLexer
-
-        print(lexer.aliases[0])
-        return 0
-
-    # handle ``pygmentize -S``
-    S_opt = argns.S
-    a_opt = argns.a
-    if S_opt is not None:
-        f_opt = argns.f
-        if not f_opt:
-            parser.print_help(sys.stderr)
-            return 2
-        if argns.l or argns.INPUTFILE:
-            parser.print_help(sys.stderr)
-            return 2
-
-        try:
-            parsed_opts['style'] = S_opt
-            fmter = get_formatter_by_name(f_opt, **parsed_opts)
-        except ClassNotFound as err:
-            print(err, file=sys.stderr)
-            return 1
-
-        print(fmter.get_style_defs(a_opt or ''))
-        return 0
-
-    # if no -S is given, -a is not allowed
-    if argns.a is not None:
-        parser.print_help(sys.stderr)
-        return 2
-
-    # parse -F options
-    F_opts = _parse_filters(argns.F or [])
-
-    # -x: allow custom (eXternal) lexers and formatters
-    allow_custom_lexer_formatter = bool(argns.x)
-
-    # select lexer
-    lexer = None
-
-    # given by name?
-    lexername = argns.l
-    if lexername:
-        # custom lexer, located relative to user's cwd
-        if allow_custom_lexer_formatter and '.py' in lexername:
-            try:
-                filename = None
-                name = None
-                if ':' in lexername:
-                    filename, name = lexername.rsplit(':', 1)
-
-                    if '.py' in name:
-                        # This can happen on Windows: If the lexername is
-                        # C:\lexer.py -- return to normal load path in that case
-                        name = None
-
-                if filename and name:
-                    lexer = load_lexer_from_file(filename, name,
-                                                 **parsed_opts)
-                else:
-                    lexer = load_lexer_from_file(lexername, **parsed_opts)
-            except ClassNotFound as err:
-                print('Error:', err, file=sys.stderr)
-                return 1
-        else:
-            try:
-                lexer = get_lexer_by_name(lexername, **parsed_opts)
-            except (OptionError, ClassNotFound) as err:
-                print('Error:', err, file=sys.stderr)
-                return 1
-
-    # read input code
-    code = None
-
-    if argns.INPUTFILE:
-        if argns.s:
-            print('Error: -s option not usable when input file specified',
-                  file=sys.stderr)
-            return 2
-
-        infn = argns.INPUTFILE
-        try:
-            with open(infn, 'rb') as infp:
-                code = infp.read()
-        except Exception as err:
-            print('Error: cannot read infile:', err, file=sys.stderr)
-            return 1
-        if not inencoding:
-            code, inencoding = guess_decode(code)
-
-        # do we have to guess the lexer?
-        if not lexer:
-            try:
-                lexer = get_lexer_for_filename(infn, code, **parsed_opts)
-            except ClassNotFound as err:
-                if argns.g:
-                    try:
-                        lexer = guess_lexer(code, **parsed_opts)
-                    except ClassNotFound:
-                        lexer = TextLexer(**parsed_opts)
-                else:
-                    print('Error:', err, file=sys.stderr)
-                    return 1
-            except OptionError as err:
-                print('Error:', err, file=sys.stderr)
-                return 1
-
-    elif not argns.s:  # treat stdin as full file (-s support is later)
-        # read code from terminal, always in binary mode since we want to
-        # decode ourselves and be tolerant with it
-        code = sys.stdin.buffer.read()  # use .buffer to get a binary stream
-        if not inencoding:
-            code, inencoding = guess_decode_from_terminal(code, sys.stdin)
-            # else the lexer will do the decoding
-        if not lexer:
-            try:
-                lexer = guess_lexer(code, **parsed_opts)
-            except ClassNotFound:
-                lexer = TextLexer(**parsed_opts)
-
-    else:  # -s option needs a lexer with -l
-        if not lexer:
-            print('Error: when using -s a lexer has to be selected with -l',
-                  file=sys.stderr)
-            return 2
-
-    # process filters
-    for fname, fopts in F_opts:
-        try:
-            lexer.add_filter(fname, **fopts)
-        except ClassNotFound as err:
-            print('Error:', err, file=sys.stderr)
-            return 1
-
-    # select formatter
-    outfn = argns.o
-    fmter = argns.f
-    if fmter:
-        # custom formatter, located relative to user's cwd
-        if allow_custom_lexer_formatter and '.py' in fmter:
-            try:
-                filename = None
-                name = None
-                if ':' in fmter:
-                    # Same logic as above for custom lexer
-                    filename, name = fmter.rsplit(':', 1)
-
-                    if '.py' in name:
-                        name = None
-
-                if filename and name:
-                    fmter = load_formatter_from_file(filename, name,
-                                                     **parsed_opts)
-                else:
-                    fmter = load_formatter_from_file(fmter, **parsed_opts)
-            except ClassNotFound as err:
-                print('Error:', err, file=sys.stderr)
-                return 1
-        else:
-            try:
-                fmter = get_formatter_by_name(fmter, **parsed_opts)
-            except (OptionError, ClassNotFound) as err:
-                print('Error:', err, file=sys.stderr)
-                return 1
-
-    if outfn:
-        if not fmter:
-            try:
-                fmter = get_formatter_for_filename(outfn, **parsed_opts)
-            except (OptionError, ClassNotFound) as err:
-                print('Error:', err, file=sys.stderr)
-                return 1
-        try:
-            outfile = open(outfn, 'wb')
-        except Exception as err:
-            print('Error: cannot open outfile:', err, file=sys.stderr)
-            return 1
-    else:
-        if not fmter:
-            if os.environ.get('COLORTERM','') in ('truecolor', '24bit'):
-                fmter = TerminalTrueColorFormatter(**parsed_opts)
-            elif '256' in os.environ.get('TERM', ''):
-                fmter = Terminal256Formatter(**parsed_opts)
-            else:
-                fmter = TerminalFormatter(**parsed_opts)
-        outfile = sys.stdout.buffer
-
-    # determine output encoding if not explicitly selected
-    if not outencoding:
-        if outfn:
-            # output file? use lexer encoding for now (can still be None)
-            fmter.encoding = inencoding
-        else:
-            # else use terminal encoding
-            fmter.encoding = terminal_encoding(sys.stdout)
-
-    # provide coloring under Windows, if possible
-    if not outfn and sys.platform in ('win32', 'cygwin') and \
-       fmter.name in ('Terminal', 'Terminal256'):  # pragma: no cover
-        # unfortunately colorama doesn't support binary streams on Py3
-        outfile = UnclosingTextIOWrapper(outfile, encoding=fmter.encoding)
-        fmter.encoding = None
-        try:
-            import pip._vendor.colorama.initialise as colorama_initialise
-        except ImportError:
-            pass
-        else:
-            outfile = colorama_initialise.wrap_stream(
-                outfile, convert=None, strip=None, autoreset=False, wrap=True)
-
-    # When using the LaTeX formatter and the option `escapeinside` is
-    # specified, we need a special lexer which collects escaped text
-    # before running the chosen language lexer.
-    escapeinside = parsed_opts.get('escapeinside', '')
-    if len(escapeinside) == 2 and isinstance(fmter, LatexFormatter):
-        left = escapeinside[0]
-        right = escapeinside[1]
-        lexer = LatexEmbeddedLexer(left, right, lexer)
-
-    # ... and do it!
-    if not argns.s:
-        # process whole input as per normal...
-        try:
-            highlight(code, lexer, fmter, outfile)
-        finally:
-            if outfn:
-                outfile.close()
-        return 0
-    else:
-        # line by line processing of stdin (eg: for 'tail -f')...
-        try:
-            while 1:
-                line = sys.stdin.buffer.readline()
-                if not line:
-                    break
-                if not inencoding:
-                    line = guess_decode_from_terminal(line, sys.stdin)[0]
-                highlight(line, lexer, fmter, outfile)
-                if hasattr(outfile, 'flush'):
-                    outfile.flush()
-            return 0
-        except KeyboardInterrupt:  # pragma: no cover
-            return 0
-        finally:
-            if outfn:
-                outfile.close()
-
-
-class HelpFormatter(argparse.HelpFormatter):
-    def __init__(self, prog, indent_increment=2, max_help_position=16, width=None):
-        if width is None:
-            try:
-                width = shutil.get_terminal_size().columns - 2
-            except Exception:
-                pass
-        argparse.HelpFormatter.__init__(self, prog, indent_increment,
-                                        max_help_position, width)
-
-
-def main(args=sys.argv):
-    """
-    Main command line entry point.
-    """
-    desc = "Highlight an input file and write the result to an output file."
-    parser = argparse.ArgumentParser(description=desc, add_help=False,
-                                     formatter_class=HelpFormatter)
-
-    operation = parser.add_argument_group('Main operation')
-    lexersel = operation.add_mutually_exclusive_group()
-    lexersel.add_argument(
-        '-l', metavar='LEXER',
-        help='Specify the lexer to use.  (Query names with -L.)  If not '
-        'given and -g is not present, the lexer is guessed from the filename.')
-    lexersel.add_argument(
-        '-g', action='store_true',
-        help='Guess the lexer from the file contents, or pass through '
-        'as plain text if nothing can be guessed.')
-    operation.add_argument(
-        '-F', metavar='FILTER[:options]', action='append',
-        help='Add a filter to the token stream.  (Query names with -L.) '
-        'Filter options are given after a colon if necessary.')
-    operation.add_argument(
-        '-f', metavar='FORMATTER',
-        help='Specify the formatter to use.  (Query names with -L.) '
-        'If not given, the formatter is guessed from the output filename, '
-        'and defaults to the terminal formatter if the output is to the '
-        'terminal or an unknown file extension.')
-    operation.add_argument(
-        '-O', metavar='OPTION=value[,OPTION=value,...]', action='append',
-        help='Give options to the lexer and formatter as a comma-separated '
-        'list of key-value pairs. '
-        'Example: `-O bg=light,python=cool`.')
-    operation.add_argument(
-        '-P', metavar='OPTION=value', action='append',
-        help='Give a single option to the lexer and formatter - with this '
-        'you can pass options whose value contains commas and equal signs. '
-        'Example: `-P "heading=Pygments, the Python highlighter"`.')
-    operation.add_argument(
-        '-o', metavar='OUTPUTFILE',
-        help='Where to write the output.  Defaults to standard output.')
-
-    operation.add_argument(
-        'INPUTFILE', nargs='?',
-        help='Where to read the input.  Defaults to standard input.')
-
-    flags = parser.add_argument_group('Operation flags')
-    flags.add_argument(
-        '-v', action='store_true',
-        help='Print a detailed traceback on unhandled exceptions, which '
-        'is useful for debugging and bug reports.')
-    flags.add_argument(
-        '-s', action='store_true',
-        help='Process lines one at a time until EOF, rather than waiting to '
-        'process the entire file.  This only works for stdin, only for lexers '
-        'with no line-spanning constructs, and is intended for streaming '
-        'input such as you get from `tail -f`. '
-        'Example usage: `tail -f sql.log | pygmentize -s -l sql`.')
-    flags.add_argument(
-        '-x', action='store_true',
-        help='Allow custom lexers and formatters to be loaded from a .py file '
-        'relative to the current working directory. For example, '
-        '`-l ./customlexer.py -x`. By default, this option expects a file '
-        'with a class named CustomLexer or CustomFormatter; you can also '
-        'specify your own class name with a colon (`-l ./lexer.py:MyLexer`). '
-        'Users should be very careful not to use this option with untrusted '
-        'files, because it will import and run them.')
-    flags.add_argument('--json', help='Output as JSON. This can '
-        'be only used in conjunction with -L.',
-        default=False,
-        action='store_true')
-
-    special_modes_group = parser.add_argument_group(
-        'Special modes - do not do any highlighting')
-    special_modes = special_modes_group.add_mutually_exclusive_group()
-    special_modes.add_argument(
-        '-S', metavar='STYLE -f formatter',
-        help='Print style definitions for STYLE for a formatter '
-        'given with -f. The argument given by -a is formatter '
-        'dependent.')
-    special_modes.add_argument(
-        '-L', nargs='*', metavar='WHAT',
-        help='List lexers, formatters, styles or filters -- '
-        'give additional arguments for the thing(s) you want to list '
-        '(e.g. "styles"), or omit them to list everything.')
-    special_modes.add_argument(
-        '-N', metavar='FILENAME',
-        help='Guess and print out a lexer name based solely on the given '
-        'filename. Does not take input or highlight anything. If no specific '
-        'lexer can be determined, "text" is printed.')
-    special_modes.add_argument(
-        '-C', action='store_true',
-        help='Like -N, but print out a lexer name based solely on '
-        'a given content from standard input.')
-    special_modes.add_argument(
-        '-H', action='store', nargs=2, metavar=('NAME', 'TYPE'),
-        help='Print detailed help for the object  of type , '
-        'where  is one of "lexer", "formatter" or "filter".')
-    special_modes.add_argument(
-        '-V', action='store_true',
-        help='Print the package version.')
-    special_modes.add_argument(
-        '-h', '--help', action='store_true',
-        help='Print this help.')
-    special_modes_group.add_argument(
-        '-a', metavar='ARG',
-        help='Formatter-specific additional argument for the -S (print '
-        'style sheet) mode.')
-
-    argns = parser.parse_args(args[1:])
-
-    try:
-        return main_inner(parser, argns)
-    except BrokenPipeError:
-        # someone closed our stdout, e.g. by quitting a pager.
-        return 0
-    except Exception:
-        if argns.v:
-            print(file=sys.stderr)
-            print('*' * 65, file=sys.stderr)
-            print('An unhandled exception occurred while highlighting.',
-                  file=sys.stderr)
-            print('Please report the whole traceback to the issue tracker at',
-                  file=sys.stderr)
-            print('.',
-                  file=sys.stderr)
-            print('*' * 65, file=sys.stderr)
-            print(file=sys.stderr)
-            raise
-        import traceback
-        info = traceback.format_exception(*sys.exc_info())
-        msg = info[-1].strip()
-        if len(info) >= 3:
-            # extract relevant file and position info
-            msg += '\n   (f%s)' % info[-2].split('\n')[0].strip()[1:]
-        print(file=sys.stderr)
-        print('*** Error while highlighting:', file=sys.stderr)
-        print(msg, file=sys.stderr)
-        print('*** If this is a bug you want to report, please rerun with -v.',
-              file=sys.stderr)
-        return 1
diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/pygments/console.py b/venv/lib/python3.11/site-packages/pip/_vendor/pygments/console.py
deleted file mode 100644
index deb4937..0000000
--- a/venv/lib/python3.11/site-packages/pip/_vendor/pygments/console.py
+++ /dev/null
@@ -1,70 +0,0 @@
-"""
-    pygments.console
-    ~~~~~~~~~~~~~~~~
-
-    Format colored console output.
-
-    :copyright: Copyright 2006-2023 by the Pygments team, see AUTHORS.
-    :license: BSD, see LICENSE for details.
-"""
-
-esc = "\x1b["
-
-codes = {}
-codes[""] = ""
-codes["reset"] = esc + "39;49;00m"
-
-codes["bold"] = esc + "01m"
-codes["faint"] = esc + "02m"
-codes["standout"] = esc + "03m"
-codes["underline"] = esc + "04m"
-codes["blink"] = esc + "05m"
-codes["overline"] = esc + "06m"
-
-dark_colors = ["black", "red", "green", "yellow", "blue",
-               "magenta", "cyan", "gray"]
-light_colors = ["brightblack", "brightred", "brightgreen", "brightyellow", "brightblue",
-                "brightmagenta", "brightcyan", "white"]
-
-x = 30
-for d, l in zip(dark_colors, light_colors):
-    codes[d] = esc + "%im" % x
-    codes[l] = esc + "%im" % (60 + x)
-    x += 1
-
-del d, l, x
-
-codes["white"] = codes["bold"]
-
-
-def reset_color():
-    return codes["reset"]
-
-
-def colorize(color_key, text):
-    return codes[color_key] + text + codes["reset"]
-
-
-def ansiformat(attr, text):
-    """
-    Format ``text`` with a color and/or some attributes::
-
-        color       normal color
-        *color*     bold color
-        _color_     underlined color
-        +color+     blinking color
-    """
-    result = []
-    if attr[:1] == attr[-1:] == '+':
-        result.append(codes['blink'])
-        attr = attr[1:-1]
-    if attr[:1] == attr[-1:] == '*':
-        result.append(codes['bold'])
-        attr = attr[1:-1]
-    if attr[:1] == attr[-1:] == '_':
-        result.append(codes['underline'])
-        attr = attr[1:-1]
-    result.append(codes[attr])
-    result.append(text)
-    result.append(codes['reset'])
-    return ''.join(result)
diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/pygments/filter.py b/venv/lib/python3.11/site-packages/pip/_vendor/pygments/filter.py
deleted file mode 100644
index dafa08d..0000000
--- a/venv/lib/python3.11/site-packages/pip/_vendor/pygments/filter.py
+++ /dev/null
@@ -1,71 +0,0 @@
-"""
-    pygments.filter
-    ~~~~~~~~~~~~~~~
-
-    Module that implements the default filter.
-
-    :copyright: Copyright 2006-2023 by the Pygments team, see AUTHORS.
-    :license: BSD, see LICENSE for details.
-"""
-
-
-def apply_filters(stream, filters, lexer=None):
-    """
-    Use this method to apply an iterable of filters to
-    a stream. If lexer is given it's forwarded to the
-    filter, otherwise the filter receives `None`.
-    """
-    def _apply(filter_, stream):
-        yield from filter_.filter(lexer, stream)
-    for filter_ in filters:
-        stream = _apply(filter_, stream)
-    return stream
-
-
-def simplefilter(f):
-    """
-    Decorator that converts a function into a filter::
-
-        @simplefilter
-        def lowercase(self, lexer, stream, options):
-            for ttype, value in stream:
-                yield ttype, value.lower()
-    """
-    return type(f.__name__, (FunctionFilter,), {
-        '__module__': getattr(f, '__module__'),
-        '__doc__': f.__doc__,
-        'function': f,
-    })
-
-
-class Filter:
-    """
-    Default filter. Subclass this class or use the `simplefilter`
-    decorator to create own filters.
-    """
-
-    def __init__(self, **options):
-        self.options = options
-
-    def filter(self, lexer, stream):
-        raise NotImplementedError()
-
-
-class FunctionFilter(Filter):
-    """
-    Abstract class used by `simplefilter` to create simple
-    function filters on the fly. The `simplefilter` decorator
-    automatically creates subclasses of this class for
-    functions passed to it.
-    """
-    function = None
-
-    def __init__(self, **options):
-        if not hasattr(self, 'function'):
-            raise TypeError('%r used without bound function' %
-                            self.__class__.__name__)
-        Filter.__init__(self, **options)
-
-    def filter(self, lexer, stream):
-        # pylint: disable=not-callable
-        yield from self.function(lexer, stream, self.options)
diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/pygments/filters/__init__.py b/venv/lib/python3.11/site-packages/pip/_vendor/pygments/filters/__init__.py
deleted file mode 100644
index 5aa9ecb..0000000
--- a/venv/lib/python3.11/site-packages/pip/_vendor/pygments/filters/__init__.py
+++ /dev/null
@@ -1,940 +0,0 @@
-"""
-    pygments.filters
-    ~~~~~~~~~~~~~~~~
-
-    Module containing filter lookup functions and default
-    filters.
-
-    :copyright: Copyright 2006-2023 by the Pygments team, see AUTHORS.
-    :license: BSD, see LICENSE for details.
-"""
-
-import re
-
-from pip._vendor.pygments.token import String, Comment, Keyword, Name, Error, Whitespace, \
-    string_to_tokentype
-from pip._vendor.pygments.filter import Filter
-from pip._vendor.pygments.util import get_list_opt, get_int_opt, get_bool_opt, \
-    get_choice_opt, ClassNotFound, OptionError
-from pip._vendor.pygments.plugin import find_plugin_filters
-
-
-def find_filter_class(filtername):
-    """Lookup a filter by name. Return None if not found."""
-    if filtername in FILTERS:
-        return FILTERS[filtername]
-    for name, cls in find_plugin_filters():
-        if name == filtername:
-            return cls
-    return None
-
-
-def get_filter_by_name(filtername, **options):
-    """Return an instantiated filter.
-
-    Options are passed to the filter initializer if wanted.
-    Raise a ClassNotFound if not found.
-    """
-    cls = find_filter_class(filtername)
-    if cls:
-        return cls(**options)
-    else:
-        raise ClassNotFound('filter %r not found' % filtername)
-
-
-def get_all_filters():
-    """Return a generator of all filter names."""
-    yield from FILTERS
-    for name, _ in find_plugin_filters():
-        yield name
-
-
-def _replace_special(ttype, value, regex, specialttype,
-                     replacefunc=lambda x: x):
-    last = 0
-    for match in regex.finditer(value):
-        start, end = match.start(), match.end()
-        if start != last:
-            yield ttype, value[last:start]
-        yield specialttype, replacefunc(value[start:end])
-        last = end
-    if last != len(value):
-        yield ttype, value[last:]
-
-
-class CodeTagFilter(Filter):
-    """Highlight special code tags in comments and docstrings.
-
-    Options accepted:
-
-    `codetags` : list of strings
-       A list of strings that are flagged as code tags.  The default is to
-       highlight ``XXX``, ``TODO``, ``FIXME``, ``BUG`` and ``NOTE``.
-
-    .. versionchanged:: 2.13
-       Now recognizes ``FIXME`` by default.
-    """
-
-    def __init__(self, **options):
-        Filter.__init__(self, **options)
-        tags = get_list_opt(options, 'codetags',
-                            ['XXX', 'TODO', 'FIXME', 'BUG', 'NOTE'])
-        self.tag_re = re.compile(r'\b(%s)\b' % '|'.join([
-            re.escape(tag) for tag in tags if tag
-        ]))
-
-    def filter(self, lexer, stream):
-        regex = self.tag_re
-        for ttype, value in stream:
-            if ttype in String.Doc or \
-               ttype in Comment and \
-               ttype not in Comment.Preproc:
-                yield from _replace_special(ttype, value, regex, Comment.Special)
-            else:
-                yield ttype, value
-
-
-class SymbolFilter(Filter):
-    """Convert mathematical symbols such as \\ in Isabelle
-    or \\longrightarrow in LaTeX into Unicode characters.
-
-    This is mostly useful for HTML or console output when you want to
-    approximate the source rendering you'd see in an IDE.
-
-    Options accepted:
-
-    `lang` : string
-       The symbol language. Must be one of ``'isabelle'`` or
-       ``'latex'``.  The default is ``'isabelle'``.
-    """
-
-    latex_symbols = {
-        '\\alpha'                : '\U000003b1',
-        '\\beta'                 : '\U000003b2',
-        '\\gamma'                : '\U000003b3',
-        '\\delta'                : '\U000003b4',
-        '\\varepsilon'           : '\U000003b5',
-        '\\zeta'                 : '\U000003b6',
-        '\\eta'                  : '\U000003b7',
-        '\\vartheta'             : '\U000003b8',
-        '\\iota'                 : '\U000003b9',
-        '\\kappa'                : '\U000003ba',
-        '\\lambda'               : '\U000003bb',
-        '\\mu'                   : '\U000003bc',
-        '\\nu'                   : '\U000003bd',
-        '\\xi'                   : '\U000003be',
-        '\\pi'                   : '\U000003c0',
-        '\\varrho'               : '\U000003c1',
-        '\\sigma'                : '\U000003c3',
-        '\\tau'                  : '\U000003c4',
-        '\\upsilon'              : '\U000003c5',
-        '\\varphi'               : '\U000003c6',
-        '\\chi'                  : '\U000003c7',
-        '\\psi'                  : '\U000003c8',
-        '\\omega'                : '\U000003c9',
-        '\\Gamma'                : '\U00000393',
-        '\\Delta'                : '\U00000394',
-        '\\Theta'                : '\U00000398',
-        '\\Lambda'               : '\U0000039b',
-        '\\Xi'                   : '\U0000039e',
-        '\\Pi'                   : '\U000003a0',
-        '\\Sigma'                : '\U000003a3',
-        '\\Upsilon'              : '\U000003a5',
-        '\\Phi'                  : '\U000003a6',
-        '\\Psi'                  : '\U000003a8',
-        '\\Omega'                : '\U000003a9',
-        '\\leftarrow'            : '\U00002190',
-        '\\longleftarrow'        : '\U000027f5',
-        '\\rightarrow'           : '\U00002192',
-        '\\longrightarrow'       : '\U000027f6',
-        '\\Leftarrow'            : '\U000021d0',
-        '\\Longleftarrow'        : '\U000027f8',
-        '\\Rightarrow'           : '\U000021d2',
-        '\\Longrightarrow'       : '\U000027f9',
-        '\\leftrightarrow'       : '\U00002194',
-        '\\longleftrightarrow'   : '\U000027f7',
-        '\\Leftrightarrow'       : '\U000021d4',
-        '\\Longleftrightarrow'   : '\U000027fa',
-        '\\mapsto'               : '\U000021a6',
-        '\\longmapsto'           : '\U000027fc',
-        '\\relbar'               : '\U00002500',
-        '\\Relbar'               : '\U00002550',
-        '\\hookleftarrow'        : '\U000021a9',
-        '\\hookrightarrow'       : '\U000021aa',
-        '\\leftharpoondown'      : '\U000021bd',
-        '\\rightharpoondown'     : '\U000021c1',
-        '\\leftharpoonup'        : '\U000021bc',
-        '\\rightharpoonup'       : '\U000021c0',
-        '\\rightleftharpoons'    : '\U000021cc',
-        '\\leadsto'              : '\U0000219d',
-        '\\downharpoonleft'      : '\U000021c3',
-        '\\downharpoonright'     : '\U000021c2',
-        '\\upharpoonleft'        : '\U000021bf',
-        '\\upharpoonright'       : '\U000021be',
-        '\\restriction'          : '\U000021be',
-        '\\uparrow'              : '\U00002191',
-        '\\Uparrow'              : '\U000021d1',
-        '\\downarrow'            : '\U00002193',
-        '\\Downarrow'            : '\U000021d3',
-        '\\updownarrow'          : '\U00002195',
-        '\\Updownarrow'          : '\U000021d5',
-        '\\langle'               : '\U000027e8',
-        '\\rangle'               : '\U000027e9',
-        '\\lceil'                : '\U00002308',
-        '\\rceil'                : '\U00002309',
-        '\\lfloor'               : '\U0000230a',
-        '\\rfloor'               : '\U0000230b',
-        '\\flqq'                 : '\U000000ab',
-        '\\frqq'                 : '\U000000bb',
-        '\\bot'                  : '\U000022a5',
-        '\\top'                  : '\U000022a4',
-        '\\wedge'                : '\U00002227',
-        '\\bigwedge'             : '\U000022c0',
-        '\\vee'                  : '\U00002228',
-        '\\bigvee'               : '\U000022c1',
-        '\\forall'               : '\U00002200',
-        '\\exists'               : '\U00002203',
-        '\\nexists'              : '\U00002204',
-        '\\neg'                  : '\U000000ac',
-        '\\Box'                  : '\U000025a1',
-        '\\Diamond'              : '\U000025c7',
-        '\\vdash'                : '\U000022a2',
-        '\\models'               : '\U000022a8',
-        '\\dashv'                : '\U000022a3',
-        '\\surd'                 : '\U0000221a',
-        '\\le'                   : '\U00002264',
-        '\\ge'                   : '\U00002265',
-        '\\ll'                   : '\U0000226a',
-        '\\gg'                   : '\U0000226b',
-        '\\lesssim'              : '\U00002272',
-        '\\gtrsim'               : '\U00002273',
-        '\\lessapprox'           : '\U00002a85',
-        '\\gtrapprox'            : '\U00002a86',
-        '\\in'                   : '\U00002208',
-        '\\notin'                : '\U00002209',
-        '\\subset'               : '\U00002282',
-        '\\supset'               : '\U00002283',
-        '\\subseteq'             : '\U00002286',
-        '\\supseteq'             : '\U00002287',
-        '\\sqsubset'             : '\U0000228f',
-        '\\sqsupset'             : '\U00002290',
-        '\\sqsubseteq'           : '\U00002291',
-        '\\sqsupseteq'           : '\U00002292',
-        '\\cap'                  : '\U00002229',
-        '\\bigcap'               : '\U000022c2',
-        '\\cup'                  : '\U0000222a',
-        '\\bigcup'               : '\U000022c3',
-        '\\sqcup'                : '\U00002294',
-        '\\bigsqcup'             : '\U00002a06',
-        '\\sqcap'                : '\U00002293',
-        '\\Bigsqcap'             : '\U00002a05',
-        '\\setminus'             : '\U00002216',
-        '\\propto'               : '\U0000221d',
-        '\\uplus'                : '\U0000228e',
-        '\\bigplus'              : '\U00002a04',
-        '\\sim'                  : '\U0000223c',
-        '\\doteq'                : '\U00002250',
-        '\\simeq'                : '\U00002243',
-        '\\approx'               : '\U00002248',
-        '\\asymp'                : '\U0000224d',
-        '\\cong'                 : '\U00002245',
-        '\\equiv'                : '\U00002261',
-        '\\Join'                 : '\U000022c8',
-        '\\bowtie'               : '\U00002a1d',
-        '\\prec'                 : '\U0000227a',
-        '\\succ'                 : '\U0000227b',
-        '\\preceq'               : '\U0000227c',
-        '\\succeq'               : '\U0000227d',
-        '\\parallel'             : '\U00002225',
-        '\\mid'                  : '\U000000a6',
-        '\\pm'                   : '\U000000b1',
-        '\\mp'                   : '\U00002213',
-        '\\times'                : '\U000000d7',
-        '\\div'                  : '\U000000f7',
-        '\\cdot'                 : '\U000022c5',
-        '\\star'                 : '\U000022c6',
-        '\\circ'                 : '\U00002218',
-        '\\dagger'               : '\U00002020',
-        '\\ddagger'              : '\U00002021',
-        '\\lhd'                  : '\U000022b2',
-        '\\rhd'                  : '\U000022b3',
-        '\\unlhd'                : '\U000022b4',
-        '\\unrhd'                : '\U000022b5',
-        '\\triangleleft'         : '\U000025c3',
-        '\\triangleright'        : '\U000025b9',
-        '\\triangle'             : '\U000025b3',
-        '\\triangleq'            : '\U0000225c',
-        '\\oplus'                : '\U00002295',
-        '\\bigoplus'             : '\U00002a01',
-        '\\otimes'               : '\U00002297',
-        '\\bigotimes'            : '\U00002a02',
-        '\\odot'                 : '\U00002299',
-        '\\bigodot'              : '\U00002a00',
-        '\\ominus'               : '\U00002296',
-        '\\oslash'               : '\U00002298',
-        '\\dots'                 : '\U00002026',
-        '\\cdots'                : '\U000022ef',
-        '\\sum'                  : '\U00002211',
-        '\\prod'                 : '\U0000220f',
-        '\\coprod'               : '\U00002210',
-        '\\infty'                : '\U0000221e',
-        '\\int'                  : '\U0000222b',
-        '\\oint'                 : '\U0000222e',
-        '\\clubsuit'             : '\U00002663',
-        '\\diamondsuit'          : '\U00002662',
-        '\\heartsuit'            : '\U00002661',
-        '\\spadesuit'            : '\U00002660',
-        '\\aleph'                : '\U00002135',
-        '\\emptyset'             : '\U00002205',
-        '\\nabla'                : '\U00002207',
-        '\\partial'              : '\U00002202',
-        '\\flat'                 : '\U0000266d',
-        '\\natural'              : '\U0000266e',
-        '\\sharp'                : '\U0000266f',
-        '\\angle'                : '\U00002220',
-        '\\copyright'            : '\U000000a9',
-        '\\textregistered'       : '\U000000ae',
-        '\\textonequarter'       : '\U000000bc',
-        '\\textonehalf'          : '\U000000bd',
-        '\\textthreequarters'    : '\U000000be',
-        '\\textordfeminine'      : '\U000000aa',
-        '\\textordmasculine'     : '\U000000ba',
-        '\\euro'                 : '\U000020ac',
-        '\\pounds'               : '\U000000a3',
-        '\\yen'                  : '\U000000a5',
-        '\\textcent'             : '\U000000a2',
-        '\\textcurrency'         : '\U000000a4',
-        '\\textdegree'           : '\U000000b0',
-    }
-
-    isabelle_symbols = {
-        '\\'                 : '\U0001d7ec',
-        '\\'                  : '\U0001d7ed',
-        '\\'                  : '\U0001d7ee',
-        '\\'                : '\U0001d7ef',
-        '\\'                 : '\U0001d7f0',
-        '\\'                 : '\U0001d7f1',
-        '\\'                  : '\U0001d7f2',
-        '\\'                : '\U0001d7f3',
-        '\\'                : '\U0001d7f4',
-        '\\'                 : '\U0001d7f5',
-        '\\'                    : '\U0001d49c',
-        '\\'                    : '\U0000212c',
-        '\\'                    : '\U0001d49e',
-        '\\'                    : '\U0001d49f',
-        '\\'                    : '\U00002130',
-        '\\'                    : '\U00002131',
-        '\\'                    : '\U0001d4a2',
-        '\\'                    : '\U0000210b',
-        '\\'                    : '\U00002110',
-        '\\'                    : '\U0001d4a5',
-        '\\'                    : '\U0001d4a6',
-        '\\'                    : '\U00002112',
-        '\\'                    : '\U00002133',
-        '\\'                    : '\U0001d4a9',
-        '\\'                    : '\U0001d4aa',
-        '\\
'                    : '\U0001d5c9',
-        '\\'                    : '\U0001d5ca',
-        '\\'                    : '\U0001d5cb',
-        '\\'                    : '\U0001d5cc',
-        '\\'                    : '\U0001d5cd',
-        '\\'                    : '\U0001d5ce',
-        '\\'                    : '\U0001d5cf',
-        '\\'                    : '\U0001d5d0',
-        '\\'                    : '\U0001d5d1',
-        '\\'                    : '\U0001d5d2',
-        '\\'                    : '\U0001d5d3',
-        '\\'                   : '\U0001d504',
-        '\\'                   : '\U0001d505',
-        '\\'                   : '\U0000212d',
-        '\\
'                   : '\U0001d507',
-        '\\'                   : '\U0001d508',
-        '\\'                   : '\U0001d509',
-        '\\'                   : '\U0001d50a',
-        '\\'                   : '\U0000210c',
-        '\\'                   : '\U00002111',
-        '\\'                   : '\U0001d50d',
-        '\\'                   : '\U0001d50e',
-        '\\'                   : '\U0001d50f',
-        '\\'                   : '\U0001d510',
-        '\\'                   : '\U0001d511',
-        '\\'                   : '\U0001d512',
-        '\\'                   : '\U0001d513',
-        '\\'                   : '\U0001d514',
-        '\\'                   : '\U0000211c',
-        '\\'                   : '\U0001d516',
-        '\\'                   : '\U0001d517',
-        '\\'                   : '\U0001d518',
-        '\\'                   : '\U0001d519',
-        '\\'                   : '\U0001d51a',
-        '\\'                   : '\U0001d51b',
-        '\\'                   : '\U0001d51c',
-        '\\'                   : '\U00002128',
-        '\\'                   : '\U0001d51e',
-        '\\'                   : '\U0001d51f',
-        '\\'                   : '\U0001d520',
-        '\\
'                   : '\U0001d521',
-        '\\'                   : '\U0001d522',
-        '\\'                   : '\U0001d523',
-        '\\'                   : '\U0001d524',
-        '\\'                   : '\U0001d525',
-        '\\'                   : '\U0001d526',
-        '\\'                   : '\U0001d527',
-        '\\'                   : '\U0001d528',
-        '\\'                   : '\U0001d529',
-        '\\'                   : '\U0001d52a',
-        '\\'                   : '\U0001d52b',
-        '\\'                   : '\U0001d52c',
-        '\\'                   : '\U0001d52d',
-        '\\'                   : '\U0001d52e',
-        '\\'                   : '\U0001d52f',
-        '\\'                   : '\U0001d530',
-        '\\'                   : '\U0001d531',
-        '\\'                   : '\U0001d532',
-        '\\'                   : '\U0001d533',
-        '\\'                   : '\U0001d534',
-        '\\'                   : '\U0001d535',
-        '\\'                   : '\U0001d536',
-        '\\'                   : '\U0001d537',
-        '\\'                : '\U000003b1',
-        '\\'                 : '\U000003b2',
-        '\\'                : '\U000003b3',
-        '\\'                : '\U000003b4',
-        '\\'              : '\U000003b5',
-        '\\'                 : '\U000003b6',
-        '\\'                  : '\U000003b7',
-        '\\'                : '\U000003b8',
-        '\\'                 : '\U000003b9',
-        '\\'                : '\U000003ba',
-        '\\'               : '\U000003bb',
-        '\\'                   : '\U000003bc',
-        '\\'                   : '\U000003bd',
-        '\\'                   : '\U000003be',
-        '\\'                   : '\U000003c0',
-        '\\'                  : '\U000003c1',
-        '\\'                : '\U000003c3',
-        '\\'                  : '\U000003c4',
-        '\\'              : '\U000003c5',
-        '\\'                  : '\U000003c6',
-        '\\'                  : '\U000003c7',
-        '\\'                  : '\U000003c8',
-        '\\'                : '\U000003c9',
-        '\\'                : '\U00000393',
-        '\\'                : '\U00000394',
-        '\\'                : '\U00000398',
-        '\\'               : '\U0000039b',
-        '\\'                   : '\U0000039e',
-        '\\'                   : '\U000003a0',
-        '\\'                : '\U000003a3',
-        '\\'              : '\U000003a5',
-        '\\'                  : '\U000003a6',
-        '\\'                  : '\U000003a8',
-        '\\'                : '\U000003a9',
-        '\\'                 : '\U0001d539',
-        '\\'              : '\U00002102',
-        '\\'                  : '\U00002115',
-        '\\'                  : '\U0000211a',
-        '\\'                 : '\U0000211d',
-        '\\'                  : '\U00002124',
-        '\\'            : '\U00002190',
-        '\\'        : '\U000027f5',
-        '\\'           : '\U00002192',
-        '\\'       : '\U000027f6',
-        '\\'            : '\U000021d0',
-        '\\'        : '\U000027f8',
-        '\\'           : '\U000021d2',
-        '\\'       : '\U000027f9',
-        '\\'       : '\U00002194',
-        '\\'   : '\U000027f7',
-        '\\'       : '\U000021d4',
-        '\\'   : '\U000027fa',
-        '\\'               : '\U000021a6',
-        '\\'           : '\U000027fc',
-        '\\'             : '\U00002500',
-        '\\'             : '\U00002550',
-        '\\'        : '\U000021a9',
-        '\\'       : '\U000021aa',
-        '\\'      : '\U000021bd',
-        '\\'     : '\U000021c1',
-        '\\'        : '\U000021bc',
-        '\\'       : '\U000021c0',
-        '\\'    : '\U000021cc',
-        '\\'              : '\U0000219d',
-        '\\'      : '\U000021c3',
-        '\\'     : '\U000021c2',
-        '\\'        : '\U000021bf',
-        '\\'       : '\U000021be',
-        '\\'          : '\U000021be',
-        '\\'                : '\U00002237',
-        '\\'                   : '\U00002191',
-        '\\'                   : '\U000021d1',
-        '\\'                 : '\U00002193',
-        '\\'                 : '\U000021d3',
-        '\\'               : '\U00002195',
-        '\\'               : '\U000021d5',
-        '\\'               : '\U000027e8',
-        '\\'               : '\U000027e9',
-        '\\'                : '\U00002308',
-        '\\'                : '\U00002309',
-        '\\'               : '\U0000230a',
-        '\\'               : '\U0000230b',
-        '\\'                : '\U00002987',
-        '\\'                : '\U00002988',
-        '\\'               : '\U000027e6',
-        '\\'               : '\U000027e7',
-        '\\'               : '\U00002983',
-        '\\'               : '\U00002984',
-        '\\'        : '\U000000ab',
-        '\\'       : '\U000000bb',
-        '\\'               : '\U000022a5',
-        '\\'                  : '\U000022a4',
-        '\\'                  : '\U00002227',
-        '\\'                  : '\U000022c0',
-        '\\'                   : '\U00002228',
-        '\\'                   : '\U000022c1',
-        '\\'               : '\U00002200',
-        '\\'               : '\U00002203',
-        '\\'              : '\U00002204',
-        '\\'                  : '\U000000ac',
-        '\\'                  : '\U000025a1',
-        '\\'              : '\U000025c7',
-        '\\'            : '\U000022a2',
-        '\\'            : '\U000022a8',
-        '\\'           : '\U000022a9',
-        '\\'           : '\U000022ab',
-        '\\'            : '\U000022a3',
-        '\\'                 : '\U0000221a',
-        '\\'                   : '\U00002264',
-        '\\'                   : '\U00002265',
-        '\\'                : '\U0000226a',
-        '\\'             : '\U0000226b',
-        '\\'              : '\U00002272',
-        '\\'           : '\U00002273',
-        '\\'           : '\U00002a85',
-        '\\'        : '\U00002a86',
-        '\\'                   : '\U00002208',
-        '\\'                : '\U00002209',
-        '\\'               : '\U00002282',
-        '\\'               : '\U00002283',
-        '\\'             : '\U00002286',
-        '\\'             : '\U00002287',
-        '\\'             : '\U0000228f',
-        '\\'             : '\U00002290',
-        '\\'           : '\U00002291',
-        '\\'           : '\U00002292',
-        '\\'                : '\U00002229',
-        '\\'                : '\U000022c2',
-        '\\'                : '\U0000222a',
-        '\\'                : '\U000022c3',
-        '\\'              : '\U00002294',
-        '\\'              : '\U00002a06',
-        '\\'              : '\U00002293',
-        '\\'              : '\U00002a05',
-        '\\'             : '\U00002216',
-        '\\'               : '\U0000221d',
-        '\\'                : '\U0000228e',
-        '\\'                : '\U00002a04',
-        '\\'                : '\U00002260',
-        '\\'                  : '\U0000223c',
-        '\\'                : '\U00002250',
-        '\\'                : '\U00002243',
-        '\\'               : '\U00002248',
-        '\\'                : '\U0000224d',
-        '\\'                 : '\U00002245',
-        '\\'                : '\U00002323',
-        '\\'                : '\U00002261',
-        '\\'                : '\U00002322',
-        '\\'                 : '\U000022c8',
-        '\\'               : '\U00002a1d',
-        '\\'                 : '\U0000227a',
-        '\\'                 : '\U0000227b',
-        '\\'               : '\U0000227c',
-        '\\'               : '\U0000227d',
-        '\\'             : '\U00002225',
-        '\\'                  : '\U000000a6',
-        '\\'            : '\U000000b1',
-        '\\'            : '\U00002213',
-        '\\'                : '\U000000d7',
-        '\\
'                  : '\U000000f7',
-        '\\'                 : '\U000022c5',
-        '\\'                 : '\U000022c6',
-        '\\'               : '\U00002219',
-        '\\'                 : '\U00002218',
-        '\\'               : '\U00002020',
-        '\\'              : '\U00002021',
-        '\\'                  : '\U000022b2',
-        '\\'                  : '\U000022b3',
-        '\\'                : '\U000022b4',
-        '\\'                : '\U000022b5',
-        '\\'         : '\U000025c3',
-        '\\'        : '\U000025b9',
-        '\\'             : '\U000025b3',
-        '\\'            : '\U0000225c',
-        '\\'                : '\U00002295',
-        '\\'                : '\U00002a01',
-        '\\'               : '\U00002297',
-        '\\'               : '\U00002a02',
-        '\\'                 : '\U00002299',
-        '\\'                 : '\U00002a00',
-        '\\'               : '\U00002296',
-        '\\'               : '\U00002298',
-        '\\'                 : '\U00002026',
-        '\\'                : '\U000022ef',
-        '\\'                  : '\U00002211',
-        '\\'                 : '\U0000220f',
-        '\\'               : '\U00002210',
-        '\\'             : '\U0000221e',
-        '\\'             : '\U0000222b',
-        '\\'            : '\U0000222e',
-        '\\'             : '\U00002663',
-        '\\'          : '\U00002662',
-        '\\'            : '\U00002661',
-        '\\'            : '\U00002660',
-        '\\'                : '\U00002135',
-        '\\'             : '\U00002205',
-        '\\'                : '\U00002207',
-        '\\'              : '\U00002202',
-        '\\'                 : '\U0000266d',
-        '\\'              : '\U0000266e',
-        '\\'                : '\U0000266f',
-        '\\'                : '\U00002220',
-        '\\'            : '\U000000a9',
-        '\\'           : '\U000000ae',
-        '\\'               : '\U000000ad',
-        '\\'              : '\U000000af',
-        '\\'           : '\U000000bc',
-        '\\'              : '\U000000bd',
-        '\\'        : '\U000000be',
-        '\\'          : '\U000000aa',
-        '\\'         : '\U000000ba',
-        '\\
'              : '\U000000a7',
-        '\\'            : '\U000000b6',
-        '\\'           : '\U000000a1',
-        '\\'         : '\U000000bf',
-        '\\'                 : '\U000020ac',
-        '\\'               : '\U000000a3',
-        '\\'                  : '\U000000a5',
-        '\\'                 : '\U000000a2',
-        '\\'             : '\U000000a4',
-        '\\'               : '\U000000b0',
-        '\\'                : '\U00002a3f',
-        '\\'                  : '\U00002127',
-        '\\'              : '\U000025ca',
-        '\\'                   : '\U00002118',
-        '\\'                : '\U00002240',
-        '\\'               : '\U000022c4',
-        '\\'                : '\U000000b4',
-        '\\'                : '\U00000131',
-        '\\'             : '\U000000a8',
-        '\\'              : '\U000000b8',
-        '\\'         : '\U000002dd',
-        '\\'                 : '\U000003f5',
-        '\\'              : '\U000023ce',
-        '\\'                 : '\U00002039',
-        '\\'                : '\U0000203a',
-        '\\'                 : '\U00002302',
-        '\\<^sub>'                 : '\U000021e9',
-        '\\<^sup>'                 : '\U000021e7',
-        '\\<^bold>'                : '\U00002759',
-        '\\<^bsub>'                : '\U000021d8',
-        '\\<^esub>'                : '\U000021d9',
-        '\\<^bsup>'                : '\U000021d7',
-        '\\<^esup>'                : '\U000021d6',
-    }
-
-    lang_map = {'isabelle' : isabelle_symbols, 'latex' : latex_symbols}
-
-    def __init__(self, **options):
-        Filter.__init__(self, **options)
-        lang = get_choice_opt(options, 'lang',
-                              ['isabelle', 'latex'], 'isabelle')
-        self.symbols = self.lang_map[lang]
-
-    def filter(self, lexer, stream):
-        for ttype, value in stream:
-            if value in self.symbols:
-                yield ttype, self.symbols[value]
-            else:
-                yield ttype, value
-
-
-class KeywordCaseFilter(Filter):
-    """Convert keywords to lowercase or uppercase or capitalize them, which
-    means first letter uppercase, rest lowercase.
-
-    This can be useful e.g. if you highlight Pascal code and want to adapt the
-    code to your styleguide.
-
-    Options accepted:
-
-    `case` : string
-       The casing to convert keywords to. Must be one of ``'lower'``,
-       ``'upper'`` or ``'capitalize'``.  The default is ``'lower'``.
-    """
-
-    def __init__(self, **options):
-        Filter.__init__(self, **options)
-        case = get_choice_opt(options, 'case',
-                              ['lower', 'upper', 'capitalize'], 'lower')
-        self.convert = getattr(str, case)
-
-    def filter(self, lexer, stream):
-        for ttype, value in stream:
-            if ttype in Keyword:
-                yield ttype, self.convert(value)
-            else:
-                yield ttype, value
-
-
-class NameHighlightFilter(Filter):
-    """Highlight a normal Name (and Name.*) token with a different token type.
-
-    Example::
-
-        filter = NameHighlightFilter(
-            names=['foo', 'bar', 'baz'],
-            tokentype=Name.Function,
-        )
-
-    This would highlight the names "foo", "bar" and "baz"
-    as functions. `Name.Function` is the default token type.
-
-    Options accepted:
-
-    `names` : list of strings
-      A list of names that should be given the different token type.
-      There is no default.
-    `tokentype` : TokenType or string
-      A token type or a string containing a token type name that is
-      used for highlighting the strings in `names`.  The default is
-      `Name.Function`.
-    """
-
-    def __init__(self, **options):
-        Filter.__init__(self, **options)
-        self.names = set(get_list_opt(options, 'names', []))
-        tokentype = options.get('tokentype')
-        if tokentype:
-            self.tokentype = string_to_tokentype(tokentype)
-        else:
-            self.tokentype = Name.Function
-
-    def filter(self, lexer, stream):
-        for ttype, value in stream:
-            if ttype in Name and value in self.names:
-                yield self.tokentype, value
-            else:
-                yield ttype, value
-
-
-class ErrorToken(Exception):
-    pass
-
-
-class RaiseOnErrorTokenFilter(Filter):
-    """Raise an exception when the lexer generates an error token.
-
-    Options accepted:
-
-    `excclass` : Exception class
-      The exception class to raise.
-      The default is `pygments.filters.ErrorToken`.
-
-    .. versionadded:: 0.8
-    """
-
-    def __init__(self, **options):
-        Filter.__init__(self, **options)
-        self.exception = options.get('excclass', ErrorToken)
-        try:
-            # issubclass() will raise TypeError if first argument is not a class
-            if not issubclass(self.exception, Exception):
-                raise TypeError
-        except TypeError:
-            raise OptionError('excclass option is not an exception class')
-
-    def filter(self, lexer, stream):
-        for ttype, value in stream:
-            if ttype is Error:
-                raise self.exception(value)
-            yield ttype, value
-
-
-class VisibleWhitespaceFilter(Filter):
-    """Convert tabs, newlines and/or spaces to visible characters.
-
-    Options accepted:
-
-    `spaces` : string or bool
-      If this is a one-character string, spaces will be replaces by this string.
-      If it is another true value, spaces will be replaced by ``·`` (unicode
-      MIDDLE DOT).  If it is a false value, spaces will not be replaced.  The
-      default is ``False``.
-    `tabs` : string or bool
-      The same as for `spaces`, but the default replacement character is ``»``
-      (unicode RIGHT-POINTING DOUBLE ANGLE QUOTATION MARK).  The default value
-      is ``False``.  Note: this will not work if the `tabsize` option for the
-      lexer is nonzero, as tabs will already have been expanded then.
-    `tabsize` : int
-      If tabs are to be replaced by this filter (see the `tabs` option), this
-      is the total number of characters that a tab should be expanded to.
-      The default is ``8``.
-    `newlines` : string or bool
-      The same as for `spaces`, but the default replacement character is ``¶``
-      (unicode PILCROW SIGN).  The default value is ``False``.
-    `wstokentype` : bool
-      If true, give whitespace the special `Whitespace` token type.  This allows
-      styling the visible whitespace differently (e.g. greyed out), but it can
-      disrupt background colors.  The default is ``True``.
-
-    .. versionadded:: 0.8
-    """
-
-    def __init__(self, **options):
-        Filter.__init__(self, **options)
-        for name, default in [('spaces',   '·'),
-                              ('tabs',     '»'),
-                              ('newlines', '¶')]:
-            opt = options.get(name, False)
-            if isinstance(opt, str) and len(opt) == 1:
-                setattr(self, name, opt)
-            else:
-                setattr(self, name, (opt and default or ''))
-        tabsize = get_int_opt(options, 'tabsize', 8)
-        if self.tabs:
-            self.tabs += ' ' * (tabsize - 1)
-        if self.newlines:
-            self.newlines += '\n'
-        self.wstt = get_bool_opt(options, 'wstokentype', True)
-
-    def filter(self, lexer, stream):
-        if self.wstt:
-            spaces = self.spaces or ' '
-            tabs = self.tabs or '\t'
-            newlines = self.newlines or '\n'
-            regex = re.compile(r'\s')
-
-            def replacefunc(wschar):
-                if wschar == ' ':
-                    return spaces
-                elif wschar == '\t':
-                    return tabs
-                elif wschar == '\n':
-                    return newlines
-                return wschar
-
-            for ttype, value in stream:
-                yield from _replace_special(ttype, value, regex, Whitespace,
-                                            replacefunc)
-        else:
-            spaces, tabs, newlines = self.spaces, self.tabs, self.newlines
-            # simpler processing
-            for ttype, value in stream:
-                if spaces:
-                    value = value.replace(' ', spaces)
-                if tabs:
-                    value = value.replace('\t', tabs)
-                if newlines:
-                    value = value.replace('\n', newlines)
-                yield ttype, value
-
-
-class GobbleFilter(Filter):
-    """Gobbles source code lines (eats initial characters).
-
-    This filter drops the first ``n`` characters off every line of code.  This
-    may be useful when the source code fed to the lexer is indented by a fixed
-    amount of space that isn't desired in the output.
-
-    Options accepted:
-
-    `n` : int
-       The number of characters to gobble.
-
-    .. versionadded:: 1.2
-    """
-    def __init__(self, **options):
-        Filter.__init__(self, **options)
-        self.n = get_int_opt(options, 'n', 0)
-
-    def gobble(self, value, left):
-        if left < len(value):
-            return value[left:], 0
-        else:
-            return '', left - len(value)
-
-    def filter(self, lexer, stream):
-        n = self.n
-        left = n  # How many characters left to gobble.
-        for ttype, value in stream:
-            # Remove ``left`` tokens from first line, ``n`` from all others.
-            parts = value.split('\n')
-            (parts[0], left) = self.gobble(parts[0], left)
-            for i in range(1, len(parts)):
-                (parts[i], left) = self.gobble(parts[i], n)
-            value = '\n'.join(parts)
-
-            if value != '':
-                yield ttype, value
-
-
-class TokenMergeFilter(Filter):
-    """Merges consecutive tokens with the same token type in the output
-    stream of a lexer.
-
-    .. versionadded:: 1.2
-    """
-    def __init__(self, **options):
-        Filter.__init__(self, **options)
-
-    def filter(self, lexer, stream):
-        current_type = None
-        current_value = None
-        for ttype, value in stream:
-            if ttype is current_type:
-                current_value += value
-            else:
-                if current_type is not None:
-                    yield current_type, current_value
-                current_type = ttype
-                current_value = value
-        if current_type is not None:
-            yield current_type, current_value
-
-
-FILTERS = {
-    'codetagify':     CodeTagFilter,
-    'keywordcase':    KeywordCaseFilter,
-    'highlight':      NameHighlightFilter,
-    'raiseonerror':   RaiseOnErrorTokenFilter,
-    'whitespace':     VisibleWhitespaceFilter,
-    'gobble':         GobbleFilter,
-    'tokenmerge':     TokenMergeFilter,
-    'symbols':        SymbolFilter,
-}
diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/pygments/filters/__pycache__/__init__.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/pygments/filters/__pycache__/__init__.cpython-311.pyc
deleted file mode 100644
index 1c22d59..0000000
Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/pygments/filters/__pycache__/__init__.cpython-311.pyc and /dev/null differ
diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/pygments/formatter.py b/venv/lib/python3.11/site-packages/pip/_vendor/pygments/formatter.py
deleted file mode 100644
index 3ca4892..0000000
--- a/venv/lib/python3.11/site-packages/pip/_vendor/pygments/formatter.py
+++ /dev/null
@@ -1,124 +0,0 @@
-"""
-    pygments.formatter
-    ~~~~~~~~~~~~~~~~~~
-
-    Base formatter class.
-
-    :copyright: Copyright 2006-2023 by the Pygments team, see AUTHORS.
-    :license: BSD, see LICENSE for details.
-"""
-
-import codecs
-
-from pip._vendor.pygments.util import get_bool_opt
-from pip._vendor.pygments.styles import get_style_by_name
-
-__all__ = ['Formatter']
-
-
-def _lookup_style(style):
-    if isinstance(style, str):
-        return get_style_by_name(style)
-    return style
-
-
-class Formatter:
-    """
-    Converts a token stream to text.
-
-    Formatters should have attributes to help selecting them. These
-    are similar to the corresponding :class:`~pygments.lexer.Lexer`
-    attributes.
-
-    .. autoattribute:: name
-       :no-value:
-
-    .. autoattribute:: aliases
-       :no-value:
-
-    .. autoattribute:: filenames
-       :no-value:
-
-    You can pass options as keyword arguments to the constructor.
-    All formatters accept these basic options:
-
-    ``style``
-        The style to use, can be a string or a Style subclass
-        (default: "default"). Not used by e.g. the
-        TerminalFormatter.
-    ``full``
-        Tells the formatter to output a "full" document, i.e.
-        a complete self-contained document. This doesn't have
-        any effect for some formatters (default: false).
-    ``title``
-        If ``full`` is true, the title that should be used to
-        caption the document (default: '').
-    ``encoding``
-        If given, must be an encoding name. This will be used to
-        convert the Unicode token strings to byte strings in the
-        output. If it is "" or None, Unicode strings will be written
-        to the output file, which most file-like objects do not
-        support (default: None).
-    ``outencoding``
-        Overrides ``encoding`` if given.
-
-    """
-
-    #: Full name for the formatter, in human-readable form.
-    name = None
-
-    #: A list of short, unique identifiers that can be used to lookup
-    #: the formatter from a list, e.g. using :func:`.get_formatter_by_name()`.
-    aliases = []
-
-    #: A list of fnmatch patterns that match filenames for which this
-    #: formatter can produce output. The patterns in this list should be unique
-    #: among all formatters.
-    filenames = []
-
-    #: If True, this formatter outputs Unicode strings when no encoding
-    #: option is given.
-    unicodeoutput = True
-
-    def __init__(self, **options):
-        """
-        As with lexers, this constructor takes arbitrary optional arguments,
-        and if you override it, you should first process your own options, then
-        call the base class implementation.
-        """
-        self.style = _lookup_style(options.get('style', 'default'))
-        self.full = get_bool_opt(options, 'full', False)
-        self.title = options.get('title', '')
-        self.encoding = options.get('encoding', None) or None
-        if self.encoding in ('guess', 'chardet'):
-            # can happen for e.g. pygmentize -O encoding=guess
-            self.encoding = 'utf-8'
-        self.encoding = options.get('outencoding') or self.encoding
-        self.options = options
-
-    def get_style_defs(self, arg=''):
-        """
-        This method must return statements or declarations suitable to define
-        the current style for subsequent highlighted text (e.g. CSS classes
-        in the `HTMLFormatter`).
-
-        The optional argument `arg` can be used to modify the generation and
-        is formatter dependent (it is standardized because it can be given on
-        the command line).
-
-        This method is called by the ``-S`` :doc:`command-line option `,
-        the `arg` is then given by the ``-a`` option.
-        """
-        return ''
-
-    def format(self, tokensource, outfile):
-        """
-        This method must format the tokens from the `tokensource` iterable and
-        write the formatted version to the file object `outfile`.
-
-        Formatter options can control how exactly the tokens are converted.
-        """
-        if self.encoding:
-            # wrap the outfile in a StreamWriter
-            outfile = codecs.lookup(self.encoding)[3](outfile)
-        return self.format_unencoded(tokensource, outfile)
diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/pygments/formatters/__init__.py b/venv/lib/python3.11/site-packages/pip/_vendor/pygments/formatters/__init__.py
deleted file mode 100644
index 39db842..0000000
--- a/venv/lib/python3.11/site-packages/pip/_vendor/pygments/formatters/__init__.py
+++ /dev/null
@@ -1,158 +0,0 @@
-"""
-    pygments.formatters
-    ~~~~~~~~~~~~~~~~~~~
-
-    Pygments formatters.
-
-    :copyright: Copyright 2006-2023 by the Pygments team, see AUTHORS.
-    :license: BSD, see LICENSE for details.
-"""
-
-import re
-import sys
-import types
-import fnmatch
-from os.path import basename
-
-from pip._vendor.pygments.formatters._mapping import FORMATTERS
-from pip._vendor.pygments.plugin import find_plugin_formatters
-from pip._vendor.pygments.util import ClassNotFound
-
-__all__ = ['get_formatter_by_name', 'get_formatter_for_filename',
-           'get_all_formatters', 'load_formatter_from_file'] + list(FORMATTERS)
-
-_formatter_cache = {}  # classes by name
-_pattern_cache = {}
-
-
-def _fn_matches(fn, glob):
-    """Return whether the supplied file name fn matches pattern filename."""
-    if glob not in _pattern_cache:
-        pattern = _pattern_cache[glob] = re.compile(fnmatch.translate(glob))
-        return pattern.match(fn)
-    return _pattern_cache[glob].match(fn)
-
-
-def _load_formatters(module_name):
-    """Load a formatter (and all others in the module too)."""
-    mod = __import__(module_name, None, None, ['__all__'])
-    for formatter_name in mod.__all__:
-        cls = getattr(mod, formatter_name)
-        _formatter_cache[cls.name] = cls
-
-
-def get_all_formatters():
-    """Return a generator for all formatter classes."""
-    # NB: this returns formatter classes, not info like get_all_lexers().
-    for info in FORMATTERS.values():
-        if info[1] not in _formatter_cache:
-            _load_formatters(info[0])
-        yield _formatter_cache[info[1]]
-    for _, formatter in find_plugin_formatters():
-        yield formatter
-
-
-def find_formatter_class(alias):
-    """Lookup a formatter by alias.
-
-    Returns None if not found.
-    """
-    for module_name, name, aliases, _, _ in FORMATTERS.values():
-        if alias in aliases:
-            if name not in _formatter_cache:
-                _load_formatters(module_name)
-            return _formatter_cache[name]
-    for _, cls in find_plugin_formatters():
-        if alias in cls.aliases:
-            return cls
-
-
-def get_formatter_by_name(_alias, **options):
-    """
-    Return an instance of a :class:`.Formatter` subclass that has `alias` in its
-    aliases list. The formatter is given the `options` at its instantiation.
-
-    Will raise :exc:`pygments.util.ClassNotFound` if no formatter with that
-    alias is found.
-    """
-    cls = find_formatter_class(_alias)
-    if cls is None:
-        raise ClassNotFound("no formatter found for name %r" % _alias)
-    return cls(**options)
-
-
-def load_formatter_from_file(filename, formattername="CustomFormatter", **options):
-    """
-    Return a `Formatter` subclass instance loaded from the provided file, relative
-    to the current directory.
-
-    The file is expected to contain a Formatter class named ``formattername``
-    (by default, CustomFormatter). Users should be very careful with the input, because
-    this method is equivalent to running ``eval()`` on the input file. The formatter is
-    given the `options` at its instantiation.
-
-    :exc:`pygments.util.ClassNotFound` is raised if there are any errors loading
-    the formatter.
-
-    .. versionadded:: 2.2
-    """
-    try:
-        # This empty dict will contain the namespace for the exec'd file
-        custom_namespace = {}
-        with open(filename, 'rb') as f:
-            exec(f.read(), custom_namespace)
-        # Retrieve the class `formattername` from that namespace
-        if formattername not in custom_namespace:
-            raise ClassNotFound('no valid %s class found in %s' %
-                                (formattername, filename))
-        formatter_class = custom_namespace[formattername]
-        # And finally instantiate it with the options
-        return formatter_class(**options)
-    except OSError as err:
-        raise ClassNotFound('cannot read %s: %s' % (filename, err))
-    except ClassNotFound:
-        raise
-    except Exception as err:
-        raise ClassNotFound('error when loading custom formatter: %s' % err)
-
-
-def get_formatter_for_filename(fn, **options):
-    """
-    Return a :class:`.Formatter` subclass instance that has a filename pattern
-    matching `fn`. The formatter is given the `options` at its instantiation.
-
-    Will raise :exc:`pygments.util.ClassNotFound` if no formatter for that filename
-    is found.
-    """
-    fn = basename(fn)
-    for modname, name, _, filenames, _ in FORMATTERS.values():
-        for filename in filenames:
-            if _fn_matches(fn, filename):
-                if name not in _formatter_cache:
-                    _load_formatters(modname)
-                return _formatter_cache[name](**options)
-    for cls in find_plugin_formatters():
-        for filename in cls.filenames:
-            if _fn_matches(fn, filename):
-                return cls(**options)
-    raise ClassNotFound("no formatter found for file name %r" % fn)
-
-
-class _automodule(types.ModuleType):
-    """Automatically import formatters."""
-
-    def __getattr__(self, name):
-        info = FORMATTERS.get(name)
-        if info:
-            _load_formatters(info[0])
-            cls = _formatter_cache[info[1]]
-            setattr(self, name, cls)
-            return cls
-        raise AttributeError(name)
-
-
-oldmod = sys.modules[__name__]
-newmod = _automodule(__name__)
-newmod.__dict__.update(oldmod.__dict__)
-sys.modules[__name__] = newmod
-del newmod.newmod, newmod.oldmod, newmod.sys, newmod.types
diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/pygments/formatters/__pycache__/__init__.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/pygments/formatters/__pycache__/__init__.cpython-311.pyc
deleted file mode 100644
index a8a69dc..0000000
Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/pygments/formatters/__pycache__/__init__.cpython-311.pyc and /dev/null differ
diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/pygments/formatters/__pycache__/_mapping.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/pygments/formatters/__pycache__/_mapping.cpython-311.pyc
deleted file mode 100644
index df6ccdb..0000000
Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/pygments/formatters/__pycache__/_mapping.cpython-311.pyc and /dev/null differ
diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/pygments/formatters/__pycache__/bbcode.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/pygments/formatters/__pycache__/bbcode.cpython-311.pyc
deleted file mode 100644
index 0b9cb48..0000000
Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/pygments/formatters/__pycache__/bbcode.cpython-311.pyc and /dev/null differ
diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/pygments/formatters/__pycache__/groff.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/pygments/formatters/__pycache__/groff.cpython-311.pyc
deleted file mode 100644
index 4afc63f..0000000
Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/pygments/formatters/__pycache__/groff.cpython-311.pyc and /dev/null differ
diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/pygments/formatters/__pycache__/html.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/pygments/formatters/__pycache__/html.cpython-311.pyc
deleted file mode 100644
index 440109a..0000000
Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/pygments/formatters/__pycache__/html.cpython-311.pyc and /dev/null differ
diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/pygments/formatters/__pycache__/img.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/pygments/formatters/__pycache__/img.cpython-311.pyc
deleted file mode 100644
index b7fc3f8..0000000
Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/pygments/formatters/__pycache__/img.cpython-311.pyc and /dev/null differ
diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/pygments/formatters/__pycache__/irc.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/pygments/formatters/__pycache__/irc.cpython-311.pyc
deleted file mode 100644
index 1746627..0000000
Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/pygments/formatters/__pycache__/irc.cpython-311.pyc and /dev/null differ
diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/pygments/formatters/__pycache__/latex.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/pygments/formatters/__pycache__/latex.cpython-311.pyc
deleted file mode 100644
index e943157..0000000
Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/pygments/formatters/__pycache__/latex.cpython-311.pyc and /dev/null differ
diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/pygments/formatters/__pycache__/other.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/pygments/formatters/__pycache__/other.cpython-311.pyc
deleted file mode 100644
index c3e7cb8..0000000
Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/pygments/formatters/__pycache__/other.cpython-311.pyc and /dev/null differ
diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/pygments/formatters/__pycache__/pangomarkup.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/pygments/formatters/__pycache__/pangomarkup.cpython-311.pyc
deleted file mode 100644
index 1080249..0000000
Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/pygments/formatters/__pycache__/pangomarkup.cpython-311.pyc and /dev/null differ
diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/pygments/formatters/__pycache__/rtf.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/pygments/formatters/__pycache__/rtf.cpython-311.pyc
deleted file mode 100644
index 71e62ab..0000000
Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/pygments/formatters/__pycache__/rtf.cpython-311.pyc and /dev/null differ
diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/pygments/formatters/__pycache__/svg.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/pygments/formatters/__pycache__/svg.cpython-311.pyc
deleted file mode 100644
index 5fc89b1..0000000
Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/pygments/formatters/__pycache__/svg.cpython-311.pyc and /dev/null differ
diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/pygments/formatters/__pycache__/terminal.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/pygments/formatters/__pycache__/terminal.cpython-311.pyc
deleted file mode 100644
index 7eebf03..0000000
Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/pygments/formatters/__pycache__/terminal.cpython-311.pyc and /dev/null differ
diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/pygments/formatters/__pycache__/terminal256.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/pygments/formatters/__pycache__/terminal256.cpython-311.pyc
deleted file mode 100644
index c29d736..0000000
Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/pygments/formatters/__pycache__/terminal256.cpython-311.pyc and /dev/null differ
diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/pygments/formatters/_mapping.py b/venv/lib/python3.11/site-packages/pip/_vendor/pygments/formatters/_mapping.py
deleted file mode 100644
index 72ca840..0000000
--- a/venv/lib/python3.11/site-packages/pip/_vendor/pygments/formatters/_mapping.py
+++ /dev/null
@@ -1,23 +0,0 @@
-# Automatically generated by scripts/gen_mapfiles.py.
-# DO NOT EDIT BY HAND; run `tox -e mapfiles` instead.
-
-FORMATTERS = {
-    'BBCodeFormatter': ('pygments.formatters.bbcode', 'BBCode', ('bbcode', 'bb'), (), 'Format tokens with BBcodes. These formatting codes are used by many bulletin boards, so you can highlight your sourcecode with pygments before posting it there.'),
-    'BmpImageFormatter': ('pygments.formatters.img', 'img_bmp', ('bmp', 'bitmap'), ('*.bmp',), 'Create a bitmap image from source code. This uses the Python Imaging Library to generate a pixmap from the source code.'),
-    'GifImageFormatter': ('pygments.formatters.img', 'img_gif', ('gif',), ('*.gif',), 'Create a GIF image from source code. This uses the Python Imaging Library to generate a pixmap from the source code.'),
-    'GroffFormatter': ('pygments.formatters.groff', 'groff', ('groff', 'troff', 'roff'), (), 'Format tokens with groff escapes to change their color and font style.'),
-    'HtmlFormatter': ('pygments.formatters.html', 'HTML', ('html',), ('*.html', '*.htm'), "Format tokens as HTML 4 ```` tags. By default, the content is enclosed in a ``
`` tag, itself wrapped in a ``
`` tag (but see the `nowrap` option). The ``
``'s CSS class can be set by the `cssclass` option."),
-    'IRCFormatter': ('pygments.formatters.irc', 'IRC', ('irc', 'IRC'), (), 'Format tokens with IRC color sequences'),
-    'ImageFormatter': ('pygments.formatters.img', 'img', ('img', 'IMG', 'png'), ('*.png',), 'Create a PNG image from source code. This uses the Python Imaging Library to generate a pixmap from the source code.'),
-    'JpgImageFormatter': ('pygments.formatters.img', 'img_jpg', ('jpg', 'jpeg'), ('*.jpg',), 'Create a JPEG image from source code. This uses the Python Imaging Library to generate a pixmap from the source code.'),
-    'LatexFormatter': ('pygments.formatters.latex', 'LaTeX', ('latex', 'tex'), ('*.tex',), 'Format tokens as LaTeX code. This needs the `fancyvrb` and `color` standard packages.'),
-    'NullFormatter': ('pygments.formatters.other', 'Text only', ('text', 'null'), ('*.txt',), 'Output the text unchanged without any formatting.'),
-    'PangoMarkupFormatter': ('pygments.formatters.pangomarkup', 'Pango Markup', ('pango', 'pangomarkup'), (), 'Format tokens as Pango Markup code. It can then be rendered to an SVG.'),
-    'RawTokenFormatter': ('pygments.formatters.other', 'Raw tokens', ('raw', 'tokens'), ('*.raw',), 'Format tokens as a raw representation for storing token streams.'),
-    'RtfFormatter': ('pygments.formatters.rtf', 'RTF', ('rtf',), ('*.rtf',), 'Format tokens as RTF markup. This formatter automatically outputs full RTF documents with color information and other useful stuff. Perfect for Copy and Paste into Microsoft(R) Word(R) documents.'),
-    'SvgFormatter': ('pygments.formatters.svg', 'SVG', ('svg',), ('*.svg',), 'Format tokens as an SVG graphics file.  This formatter is still experimental. Each line of code is a ```` element with explicit ``x`` and ``y`` coordinates containing ```` elements with the individual token styles.'),
-    'Terminal256Formatter': ('pygments.formatters.terminal256', 'Terminal256', ('terminal256', 'console256', '256'), (), 'Format tokens with ANSI color sequences, for output in a 256-color terminal or console.  Like in `TerminalFormatter` color sequences are terminated at newlines, so that paging the output works correctly.'),
-    'TerminalFormatter': ('pygments.formatters.terminal', 'Terminal', ('terminal', 'console'), (), 'Format tokens with ANSI color sequences, for output in a text console. Color sequences are terminated at newlines, so that paging the output works correctly.'),
-    'TerminalTrueColorFormatter': ('pygments.formatters.terminal256', 'TerminalTrueColor', ('terminal16m', 'console16m', '16m'), (), 'Format tokens with ANSI color sequences, for output in a true-color terminal or console.  Like in `TerminalFormatter` color sequences are terminated at newlines, so that paging the output works correctly.'),
-    'TestcaseFormatter': ('pygments.formatters.other', 'Testcase', ('testcase',), (), 'Format tokens as appropriate for a new testcase.'),
-}
diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/pygments/formatters/bbcode.py b/venv/lib/python3.11/site-packages/pip/_vendor/pygments/formatters/bbcode.py
deleted file mode 100644
index c4db8f4..0000000
--- a/venv/lib/python3.11/site-packages/pip/_vendor/pygments/formatters/bbcode.py
+++ /dev/null
@@ -1,108 +0,0 @@
-"""
-    pygments.formatters.bbcode
-    ~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-    BBcode formatter.
-
-    :copyright: Copyright 2006-2023 by the Pygments team, see AUTHORS.
-    :license: BSD, see LICENSE for details.
-"""
-
-
-from pip._vendor.pygments.formatter import Formatter
-from pip._vendor.pygments.util import get_bool_opt
-
-__all__ = ['BBCodeFormatter']
-
-
-class BBCodeFormatter(Formatter):
-    """
-    Format tokens with BBcodes. These formatting codes are used by many
-    bulletin boards, so you can highlight your sourcecode with pygments before
-    posting it there.
-
-    This formatter has no support for background colors and borders, as there
-    are no common BBcode tags for that.
-
-    Some board systems (e.g. phpBB) don't support colors in their [code] tag,
-    so you can't use the highlighting together with that tag.
-    Text in a [code] tag usually is shown with a monospace font (which this
-    formatter can do with the ``monofont`` option) and no spaces (which you
-    need for indentation) are removed.
-
-    Additional options accepted:
-
-    `style`
-        The style to use, can be a string or a Style subclass (default:
-        ``'default'``).
-
-    `codetag`
-        If set to true, put the output into ``[code]`` tags (default:
-        ``false``)
-
-    `monofont`
-        If set to true, add a tag to show the code with a monospace font
-        (default: ``false``).
-    """
-    name = 'BBCode'
-    aliases = ['bbcode', 'bb']
-    filenames = []
-
-    def __init__(self, **options):
-        Formatter.__init__(self, **options)
-        self._code = get_bool_opt(options, 'codetag', False)
-        self._mono = get_bool_opt(options, 'monofont', False)
-
-        self.styles = {}
-        self._make_styles()
-
-    def _make_styles(self):
-        for ttype, ndef in self.style:
-            start = end = ''
-            if ndef['color']:
-                start += '[color=#%s]' % ndef['color']
-                end = '[/color]' + end
-            if ndef['bold']:
-                start += '[b]'
-                end = '[/b]' + end
-            if ndef['italic']:
-                start += '[i]'
-                end = '[/i]' + end
-            if ndef['underline']:
-                start += '[u]'
-                end = '[/u]' + end
-            # there are no common BBcodes for background-color and border
-
-            self.styles[ttype] = start, end
-
-    def format_unencoded(self, tokensource, outfile):
-        if self._code:
-            outfile.write('[code]')
-        if self._mono:
-            outfile.write('[font=monospace]')
-
-        lastval = ''
-        lasttype = None
-
-        for ttype, value in tokensource:
-            while ttype not in self.styles:
-                ttype = ttype.parent
-            if ttype == lasttype:
-                lastval += value
-            else:
-                if lastval:
-                    start, end = self.styles[lasttype]
-                    outfile.write(''.join((start, lastval, end)))
-                lastval = value
-                lasttype = ttype
-
-        if lastval:
-            start, end = self.styles[lasttype]
-            outfile.write(''.join((start, lastval, end)))
-
-        if self._mono:
-            outfile.write('[/font]')
-        if self._code:
-            outfile.write('[/code]')
-        if self._code or self._mono:
-            outfile.write('\n')
diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/pygments/formatters/groff.py b/venv/lib/python3.11/site-packages/pip/_vendor/pygments/formatters/groff.py
deleted file mode 100644
index 30a528e..0000000
--- a/venv/lib/python3.11/site-packages/pip/_vendor/pygments/formatters/groff.py
+++ /dev/null
@@ -1,170 +0,0 @@
-"""
-    pygments.formatters.groff
-    ~~~~~~~~~~~~~~~~~~~~~~~~~
-
-    Formatter for groff output.
-
-    :copyright: Copyright 2006-2023 by the Pygments team, see AUTHORS.
-    :license: BSD, see LICENSE for details.
-"""
-
-import math
-from pip._vendor.pygments.formatter import Formatter
-from pip._vendor.pygments.util import get_bool_opt, get_int_opt
-
-__all__ = ['GroffFormatter']
-
-
-class GroffFormatter(Formatter):
-    """
-    Format tokens with groff escapes to change their color and font style.
-
-    .. versionadded:: 2.11
-
-    Additional options accepted:
-
-    `style`
-        The style to use, can be a string or a Style subclass (default:
-        ``'default'``).
-
-    `monospaced`
-        If set to true, monospace font will be used (default: ``true``).
-
-    `linenos`
-        If set to true, print the line numbers (default: ``false``).
-
-    `wrap`
-        Wrap lines to the specified number of characters. Disabled if set to 0
-        (default: ``0``).
-    """
-
-    name = 'groff'
-    aliases = ['groff','troff','roff']
-    filenames = []
-
-    def __init__(self, **options):
-        Formatter.__init__(self, **options)
-
-        self.monospaced = get_bool_opt(options, 'monospaced', True)
-        self.linenos = get_bool_opt(options, 'linenos', False)
-        self._lineno = 0
-        self.wrap = get_int_opt(options, 'wrap', 0)
-        self._linelen = 0
-
-        self.styles = {}
-        self._make_styles()
-
-
-    def _make_styles(self):
-        regular = '\\f[CR]' if self.monospaced else '\\f[R]'
-        bold = '\\f[CB]' if self.monospaced else '\\f[B]'
-        italic = '\\f[CI]' if self.monospaced else '\\f[I]'
-
-        for ttype, ndef in self.style:
-            start = end = ''
-            if ndef['color']:
-                start += '\\m[%s]' % ndef['color']
-                end = '\\m[]' + end
-            if ndef['bold']:
-                start += bold
-                end = regular + end
-            if ndef['italic']:
-                start += italic
-                end = regular + end
-            if ndef['bgcolor']:
-                start += '\\M[%s]' % ndef['bgcolor']
-                end = '\\M[]' + end
-
-            self.styles[ttype] = start, end
-
-
-    def _define_colors(self, outfile):
-        colors = set()
-        for _, ndef in self.style:
-            if ndef['color'] is not None:
-                colors.add(ndef['color'])
-
-        for color in sorted(colors):
-            outfile.write('.defcolor ' + color + ' rgb #' + color + '\n')
-
-
-    def _write_lineno(self, outfile):
-        self._lineno += 1
-        outfile.write("%s% 4d " % (self._lineno != 1 and '\n' or '', self._lineno))
-
-
-    def _wrap_line(self, line):
-        length = len(line.rstrip('\n'))
-        space = '     ' if self.linenos else ''
-        newline = ''
-
-        if length > self.wrap:
-            for i in range(0, math.floor(length / self.wrap)):
-                chunk = line[i*self.wrap:i*self.wrap+self.wrap]
-                newline += (chunk + '\n' + space)
-            remainder = length % self.wrap
-            if remainder > 0:
-                newline += line[-remainder-1:]
-                self._linelen = remainder
-        elif self._linelen + length > self.wrap:
-            newline = ('\n' + space) + line
-            self._linelen = length
-        else:
-            newline = line
-            self._linelen += length
-
-        return newline
-
-
-    def _escape_chars(self, text):
-        text = text.replace('\\', '\\[u005C]'). \
-                    replace('.', '\\[char46]'). \
-                    replace('\'', '\\[u0027]'). \
-                    replace('`', '\\[u0060]'). \
-                    replace('~', '\\[u007E]')
-        copy = text
-
-        for char in copy:
-            if len(char) != len(char.encode()):
-                uni = char.encode('unicode_escape') \
-                    .decode()[1:] \
-                    .replace('x', 'u00') \
-                    .upper()
-                text = text.replace(char, '\\[u' + uni[1:] + ']')
-
-        return text
-
-
-    def format_unencoded(self, tokensource, outfile):
-        self._define_colors(outfile)
-
-        outfile.write('.nf\n\\f[CR]\n')
-
-        if self.linenos:
-            self._write_lineno(outfile)
-
-        for ttype, value in tokensource:
-            while ttype not in self.styles:
-                ttype = ttype.parent
-            start, end = self.styles[ttype]
-
-            for line in value.splitlines(True):
-                if self.wrap > 0:
-                    line = self._wrap_line(line)
-
-                if start and end:
-                    text = self._escape_chars(line.rstrip('\n'))
-                    if text != '':
-                        outfile.write(''.join((start, text, end)))
-                else:
-                    outfile.write(self._escape_chars(line.rstrip('\n')))
-
-                if line.endswith('\n'):
-                    if self.linenos:
-                        self._write_lineno(outfile)
-                        self._linelen = 0
-                    else:
-                        outfile.write('\n')
-                        self._linelen = 0
-
-        outfile.write('\n.fi')
diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/pygments/formatters/html.py b/venv/lib/python3.11/site-packages/pip/_vendor/pygments/formatters/html.py
deleted file mode 100644
index 931d7c3..0000000
--- a/venv/lib/python3.11/site-packages/pip/_vendor/pygments/formatters/html.py
+++ /dev/null
@@ -1,989 +0,0 @@
-"""
-    pygments.formatters.html
-    ~~~~~~~~~~~~~~~~~~~~~~~~
-
-    Formatter for HTML output.
-
-    :copyright: Copyright 2006-2023 by the Pygments team, see AUTHORS.
-    :license: BSD, see LICENSE for details.
-"""
-
-import functools
-import os
-import sys
-import os.path
-from io import StringIO
-
-from pip._vendor.pygments.formatter import Formatter
-from pip._vendor.pygments.token import Token, Text, STANDARD_TYPES
-from pip._vendor.pygments.util import get_bool_opt, get_int_opt, get_list_opt
-
-try:
-    import ctags
-except ImportError:
-    ctags = None
-
-__all__ = ['HtmlFormatter']
-
-
-_escape_html_table = {
-    ord('&'): '&',
-    ord('<'): '<',
-    ord('>'): '>',
-    ord('"'): '"',
-    ord("'"): ''',
-}
-
-
-def escape_html(text, table=_escape_html_table):
-    """Escape &, <, > as well as single and double quotes for HTML."""
-    return text.translate(table)
-
-
-def webify(color):
-    if color.startswith('calc') or color.startswith('var'):
-        return color
-    else:
-        return '#' + color
-
-
-def _get_ttype_class(ttype):
-    fname = STANDARD_TYPES.get(ttype)
-    if fname:
-        return fname
-    aname = ''
-    while fname is None:
-        aname = '-' + ttype[-1] + aname
-        ttype = ttype.parent
-        fname = STANDARD_TYPES.get(ttype)
-    return fname + aname
-
-
-CSSFILE_TEMPLATE = '''\
-/*
-generated by Pygments 
-Copyright 2006-2023 by the Pygments team.
-Licensed under the BSD license, see LICENSE for details.
-*/
-%(styledefs)s
-'''
-
-DOC_HEADER = '''\
-
-
-
-
-  %(title)s
-  
-  
-
-
-
%(title)s

-
-'''
-
-DOC_HEADER_EXTERNALCSS = '''\
-
-
-
-
-  %(title)s
-  
-  
-
-
-
%(title)s

-
-'''
-
-DOC_FOOTER = '''\
-
-
-'''
-
-
-class HtmlFormatter(Formatter):
-    r"""
-    Format tokens as HTML 4 ```` tags. By default, the content is enclosed
-    in a ``
`` tag, itself wrapped in a ``
`` tag (but see the `nowrap` option).
-    The ``
``'s CSS class can be set by the `cssclass` option.
-
-    If the `linenos` option is set to ``"table"``, the ``
`` is
-    additionally wrapped inside a ``
'                    : '\U0001d4ab',
-        '\\'                    : '\U0001d4ac',
-        '\\'                    : '\U0000211b',
-        '\\'                    : '\U0001d4ae',
-        '\\'                    : '\U0001d4af',
-        '\\'                    : '\U0001d4b0',
-        '\\'                    : '\U0001d4b1',
-        '\\'                    : '\U0001d4b2',
-        '\\'                    : '\U0001d4b3',
-        '\\'                    : '\U0001d4b4',
-        '\\'                    : '\U0001d4b5',
-        '\\'                    : '\U0001d5ba',
-        '\\'                    : '\U0001d5bb',
-        '\\'                    : '\U0001d5bc',
-        '\\'                    : '\U0001d5bd',
-        '\\'                    : '\U0001d5be',
-        '\\'                    : '\U0001d5bf',
-        '\\'                    : '\U0001d5c0',
-        '\\'                    : '\U0001d5c1',
-        '\\'                    : '\U0001d5c2',
-        '\\'                    : '\U0001d5c3',
-        '\\'                    : '\U0001d5c4',
-        '\\'                    : '\U0001d5c5',
-        '\\'                    : '\U0001d5c6',
-        '\\'                    : '\U0001d5c7',
-        '\\'                    : '\U0001d5c8',
-        '\\
`` which has one row and two - cells: one containing the line numbers and one containing the code. - Example: - - .. sourcecode:: html - -
-
- - -
- 
1
-            2
- 		- 
def foo(bar):
-              pass
-
-

- - (whitespace added to improve clarity). - - A list of lines can be specified using the `hl_lines` option to make these - lines highlighted (as of Pygments 0.11). - - With the `full` option, a complete HTML 4 document is output, including - the style definitions inside a `` - {% else %} - {{ head | safe }} - {% endif %} -{% if not embed %} - - -{% endif %} -{{ body | safe }} -{% for diagram in diagrams %} -
-

{{ diagram.title }}

-
{{ diagram.text }}
-
- {{ diagram.svg }} -
-
-{% endfor %} -{% if not embed %} - - -{% endif %} -""" - -template = Template(jinja2_template_source) - -# Note: ideally this would be a dataclass, but we're supporting Python 3.5+ so we can't do this yet -NamedDiagram = NamedTuple( - "NamedDiagram", - [("name", str), ("diagram", typing.Optional[railroad.DiagramItem]), ("index", int)], -) -""" -A simple structure for associating a name with a railroad diagram -""" - -T = TypeVar("T") - - -class EachItem(railroad.Group): - """ - Custom railroad item to compose a: - - Group containing a - - OneOrMore containing a - - Choice of the elements in the Each - with the group label indicating that all must be matched - """ - - all_label = "[ALL]" - - def __init__(self, *items): - choice_item = railroad.Choice(len(items) - 1, *items) - one_or_more_item = railroad.OneOrMore(item=choice_item) - super().__init__(one_or_more_item, label=self.all_label) - - -class AnnotatedItem(railroad.Group): - """ - Simple subclass of Group that creates an annotation label - """ - - def __init__(self, label: str, item): - super().__init__(item=item, label="[{}]".format(label) if label else label) - - -class EditablePartial(Generic[T]): - """ - Acts like a functools.partial, but can be edited. In other words, it represents a type that hasn't yet been - constructed. - """ - - # We need this here because the railroad constructors actually transform the data, so can't be called until the - # entire tree is assembled - - def __init__(self, func: Callable[..., T], args: list, kwargs: dict): - self.func = func - self.args = args - self.kwargs = kwargs - - @classmethod - def from_call(cls, func: Callable[..., T], *args, **kwargs) -> "EditablePartial[T]": - """ - If you call this function in the same way that you would call the constructor, it will store the arguments - as you expect. For example EditablePartial.from_call(Fraction, 1, 3)() == Fraction(1, 3) - """ - return EditablePartial(func=func, args=list(args), kwargs=kwargs) - - @property - def name(self): - return self.kwargs["name"] - - def __call__(self) -> T: - """ - Evaluate the partial and return the result - """ - args = self.args.copy() - kwargs = self.kwargs.copy() - - # This is a helpful hack to allow you to specify varargs parameters (e.g. *args) as keyword args (e.g. - # args=['list', 'of', 'things']) - arg_spec = inspect.getfullargspec(self.func) - if arg_spec.varargs in self.kwargs: - args += kwargs.pop(arg_spec.varargs) - - return self.func(*args, **kwargs) - - -def railroad_to_html(diagrams: List[NamedDiagram], embed=False, **kwargs) -> str: - """ - Given a list of NamedDiagram, produce a single HTML string that visualises those diagrams - :params kwargs: kwargs to be passed in to the template - """ - data = [] - for diagram in diagrams: - if diagram.diagram is None: - continue - io = StringIO() - try: - css = kwargs.get('css') - diagram.diagram.writeStandalone(io.write, css=css) - except AttributeError: - diagram.diagram.writeSvg(io.write) - title = diagram.name - if diagram.index == 0: - title += " (root)" - data.append({"title": title, "text": "", "svg": io.getvalue()}) - - return template.render(diagrams=data, embed=embed, **kwargs) - - -def resolve_partial(partial: "EditablePartial[T]") -> T: - """ - Recursively resolves a collection of Partials into whatever type they are - """ - if isinstance(partial, EditablePartial): - partial.args = resolve_partial(partial.args) - partial.kwargs = resolve_partial(partial.kwargs) - return partial() - elif isinstance(partial, list): - return [resolve_partial(x) for x in partial] - elif isinstance(partial, dict): - return {key: resolve_partial(x) for key, x in partial.items()} - else: - return partial - - -def to_railroad( - element: pyparsing.ParserElement, - diagram_kwargs: typing.Optional[dict] = None, - vertical: int = 3, - show_results_names: bool = False, - show_groups: bool = False, -) -> List[NamedDiagram]: - """ - Convert a pyparsing element tree into a list of diagrams. This is the recommended entrypoint to diagram - creation if you want to access the Railroad tree before it is converted to HTML - :param element: base element of the parser being diagrammed - :param diagram_kwargs: kwargs to pass to the Diagram() constructor - :param vertical: (optional) - int - limit at which number of alternatives should be - shown vertically instead of horizontally - :param show_results_names - bool to indicate whether results name annotations should be - included in the diagram - :param show_groups - bool to indicate whether groups should be highlighted with an unlabeled - surrounding box - """ - # Convert the whole tree underneath the root - lookup = ConverterState(diagram_kwargs=diagram_kwargs or {}) - _to_diagram_element( - element, - lookup=lookup, - parent=None, - vertical=vertical, - show_results_names=show_results_names, - show_groups=show_groups, - ) - - root_id = id(element) - # Convert the root if it hasn't been already - if root_id in lookup: - if not element.customName: - lookup[root_id].name = "" - lookup[root_id].mark_for_extraction(root_id, lookup, force=True) - - # Now that we're finished, we can convert from intermediate structures into Railroad elements - diags = list(lookup.diagrams.values()) - if len(diags) > 1: - # collapse out duplicate diags with the same name - seen = set() - deduped_diags = [] - for d in diags: - # don't extract SkipTo elements, they are uninformative as subdiagrams - if d.name == "...": - continue - if d.name is not None and d.name not in seen: - seen.add(d.name) - deduped_diags.append(d) - resolved = [resolve_partial(partial) for partial in deduped_diags] - else: - # special case - if just one diagram, always display it, even if - # it has no name - resolved = [resolve_partial(partial) for partial in diags] - return sorted(resolved, key=lambda diag: diag.index) - - -def _should_vertical( - specification: int, exprs: Iterable[pyparsing.ParserElement] -) -> bool: - """ - Returns true if we should return a vertical list of elements - """ - if specification is None: - return False - else: - return len(_visible_exprs(exprs)) >= specification - - -class ElementState: - """ - State recorded for an individual pyparsing Element - """ - - # Note: this should be a dataclass, but we have to support Python 3.5 - def __init__( - self, - element: pyparsing.ParserElement, - converted: EditablePartial, - parent: EditablePartial, - number: int, - name: str = None, - parent_index: typing.Optional[int] = None, - ): - #: The pyparsing element that this represents - self.element: pyparsing.ParserElement = element - #: The name of the element - self.name: typing.Optional[str] = name - #: The output Railroad element in an unconverted state - self.converted: EditablePartial = converted - #: The parent Railroad element, which we store so that we can extract this if it's duplicated - self.parent: EditablePartial = parent - #: The order in which we found this element, used for sorting diagrams if this is extracted into a diagram - self.number: int = number - #: The index of this inside its parent - self.parent_index: typing.Optional[int] = parent_index - #: If true, we should extract this out into a subdiagram - self.extract: bool = False - #: If true, all of this element's children have been filled out - self.complete: bool = False - - def mark_for_extraction( - self, el_id: int, state: "ConverterState", name: str = None, force: bool = False - ): - """ - Called when this instance has been seen twice, and thus should eventually be extracted into a sub-diagram - :param el_id: id of the element - :param state: element/diagram state tracker - :param name: name to use for this element's text - :param force: If true, force extraction now, regardless of the state of this. Only useful for extracting the - root element when we know we're finished - """ - self.extract = True - - # Set the name - if not self.name: - if name: - # Allow forcing a custom name - self.name = name - elif self.element.customName: - self.name = self.element.customName - else: - self.name = "" - - # Just because this is marked for extraction doesn't mean we can do it yet. We may have to wait for children - # to be added - # Also, if this is just a string literal etc, don't bother extracting it - if force or (self.complete and _worth_extracting(self.element)): - state.extract_into_diagram(el_id) - - -class ConverterState: - """ - Stores some state that persists between recursions into the element tree - """ - - def __init__(self, diagram_kwargs: typing.Optional[dict] = None): - #: A dictionary mapping ParserElements to state relating to them - self._element_diagram_states: Dict[int, ElementState] = {} - #: A dictionary mapping ParserElement IDs to subdiagrams generated from them - self.diagrams: Dict[int, EditablePartial[NamedDiagram]] = {} - #: The index of the next unnamed element - self.unnamed_index: int = 1 - #: The index of the next element. This is used for sorting - self.index: int = 0 - #: Shared kwargs that are used to customize the construction of diagrams - self.diagram_kwargs: dict = diagram_kwargs or {} - self.extracted_diagram_names: Set[str] = set() - - def __setitem__(self, key: int, value: ElementState): - self._element_diagram_states[key] = value - - def __getitem__(self, key: int) -> ElementState: - return self._element_diagram_states[key] - - def __delitem__(self, key: int): - del self._element_diagram_states[key] - - def __contains__(self, key: int): - return key in self._element_diagram_states - - def generate_unnamed(self) -> int: - """ - Generate a number used in the name of an otherwise unnamed diagram - """ - self.unnamed_index += 1 - return self.unnamed_index - - def generate_index(self) -> int: - """ - Generate a number used to index a diagram - """ - self.index += 1 - return self.index - - def extract_into_diagram(self, el_id: int): - """ - Used when we encounter the same token twice in the same tree. When this - happens, we replace all instances of that token with a terminal, and - create a new subdiagram for the token - """ - position = self[el_id] - - # Replace the original definition of this element with a regular block - if position.parent: - ret = EditablePartial.from_call(railroad.NonTerminal, text=position.name) - if "item" in position.parent.kwargs: - position.parent.kwargs["item"] = ret - elif "items" in position.parent.kwargs: - position.parent.kwargs["items"][position.parent_index] = ret - - # If the element we're extracting is a group, skip to its content but keep the title - if position.converted.func == railroad.Group: - content = position.converted.kwargs["item"] - else: - content = position.converted - - self.diagrams[el_id] = EditablePartial.from_call( - NamedDiagram, - name=position.name, - diagram=EditablePartial.from_call( - railroad.Diagram, content, **self.diagram_kwargs - ), - index=position.number, - ) - - del self[el_id] - - -def _worth_extracting(element: pyparsing.ParserElement) -> bool: - """ - Returns true if this element is worth having its own sub-diagram. Simply, if any of its children - themselves have children, then its complex enough to extract - """ - children = element.recurse() - return any(child.recurse() for child in children) - - -def _apply_diagram_item_enhancements(fn): - """ - decorator to ensure enhancements to a diagram item (such as results name annotations) - get applied on return from _to_diagram_element (we do this since there are several - returns in _to_diagram_element) - """ - - def _inner( - element: pyparsing.ParserElement, - parent: typing.Optional[EditablePartial], - lookup: ConverterState = None, - vertical: int = None, - index: int = 0, - name_hint: str = None, - show_results_names: bool = False, - show_groups: bool = False, - ) -> typing.Optional[EditablePartial]: - ret = fn( - element, - parent, - lookup, - vertical, - index, - name_hint, - show_results_names, - show_groups, - ) - - # apply annotation for results name, if present - if show_results_names and ret is not None: - element_results_name = element.resultsName - if element_results_name: - # add "*" to indicate if this is a "list all results" name - element_results_name += "" if element.modalResults else "*" - ret = EditablePartial.from_call( - railroad.Group, item=ret, label=element_results_name - ) - - return ret - - return _inner - - -def _visible_exprs(exprs: Iterable[pyparsing.ParserElement]): - non_diagramming_exprs = ( - pyparsing.ParseElementEnhance, - pyparsing.PositionToken, - pyparsing.And._ErrorStop, - ) - return [ - e - for e in exprs - if not (e.customName or e.resultsName or isinstance(e, non_diagramming_exprs)) - ] - - -@_apply_diagram_item_enhancements -def _to_diagram_element( - element: pyparsing.ParserElement, - parent: typing.Optional[EditablePartial], - lookup: ConverterState = None, - vertical: int = None, - index: int = 0, - name_hint: str = None, - show_results_names: bool = False, - show_groups: bool = False, -) -> typing.Optional[EditablePartial]: - """ - Recursively converts a PyParsing Element to a railroad Element - :param lookup: The shared converter state that keeps track of useful things - :param index: The index of this element within the parent - :param parent: The parent of this element in the output tree - :param vertical: Controls at what point we make a list of elements vertical. If this is an integer (the default), - it sets the threshold of the number of items before we go vertical. If True, always go vertical, if False, never - do so - :param name_hint: If provided, this will override the generated name - :param show_results_names: bool flag indicating whether to add annotations for results names - :returns: The converted version of the input element, but as a Partial that hasn't yet been constructed - :param show_groups: bool flag indicating whether to show groups using bounding box - """ - exprs = element.recurse() - name = name_hint or element.customName or element.__class__.__name__ - - # Python's id() is used to provide a unique identifier for elements - el_id = id(element) - - element_results_name = element.resultsName - - # Here we basically bypass processing certain wrapper elements if they contribute nothing to the diagram - if not element.customName: - if isinstance( - element, - ( - # pyparsing.TokenConverter, - # pyparsing.Forward, - pyparsing.Located, - ), - ): - # However, if this element has a useful custom name, and its child does not, we can pass it on to the child - if exprs: - if not exprs[0].customName: - propagated_name = name - else: - propagated_name = None - - return _to_diagram_element( - element.expr, - parent=parent, - lookup=lookup, - vertical=vertical, - index=index, - name_hint=propagated_name, - show_results_names=show_results_names, - show_groups=show_groups, - ) - - # If the element isn't worth extracting, we always treat it as the first time we say it - if _worth_extracting(element): - if el_id in lookup: - # If we've seen this element exactly once before, we are only just now finding out that it's a duplicate, - # so we have to extract it into a new diagram. - looked_up = lookup[el_id] - looked_up.mark_for_extraction(el_id, lookup, name=name_hint) - ret = EditablePartial.from_call(railroad.NonTerminal, text=looked_up.name) - return ret - - elif el_id in lookup.diagrams: - # If we have seen the element at least twice before, and have already extracted it into a subdiagram, we - # just put in a marker element that refers to the sub-diagram - ret = EditablePartial.from_call( - railroad.NonTerminal, text=lookup.diagrams[el_id].kwargs["name"] - ) - return ret - - # Recursively convert child elements - # Here we find the most relevant Railroad element for matching pyparsing Element - # We use ``items=[]`` here to hold the place for where the child elements will go once created - if isinstance(element, pyparsing.And): - # detect And's created with ``expr*N`` notation - for these use a OneOrMore with a repeat - # (all will have the same name, and resultsName) - if not exprs: - return None - if len(set((e.name, e.resultsName) for e in exprs)) == 1: - ret = EditablePartial.from_call( - railroad.OneOrMore, item="", repeat=str(len(exprs)) - ) - elif _should_vertical(vertical, exprs): - ret = EditablePartial.from_call(railroad.Stack, items=[]) - else: - ret = EditablePartial.from_call(railroad.Sequence, items=[]) - elif isinstance(element, (pyparsing.Or, pyparsing.MatchFirst)): - if not exprs: - return None - if _should_vertical(vertical, exprs): - ret = EditablePartial.from_call(railroad.Choice, 0, items=[]) - else: - ret = EditablePartial.from_call(railroad.HorizontalChoice, items=[]) - elif isinstance(element, pyparsing.Each): - if not exprs: - return None - ret = EditablePartial.from_call(EachItem, items=[]) - elif isinstance(element, pyparsing.NotAny): - ret = EditablePartial.from_call(AnnotatedItem, label="NOT", item="") - elif isinstance(element, pyparsing.FollowedBy): - ret = EditablePartial.from_call(AnnotatedItem, label="LOOKAHEAD", item="") - elif isinstance(element, pyparsing.PrecededBy): - ret = EditablePartial.from_call(AnnotatedItem, label="LOOKBEHIND", item="") - elif isinstance(element, pyparsing.Group): - if show_groups: - ret = EditablePartial.from_call(AnnotatedItem, label="", item="") - else: - ret = EditablePartial.from_call(railroad.Group, label="", item="") - elif isinstance(element, pyparsing.TokenConverter): - label = type(element).__name__.lower() - if label == "tokenconverter": - ret = EditablePartial.from_call(railroad.Sequence, items=[]) - else: - ret = EditablePartial.from_call(AnnotatedItem, label=label, item="") - elif isinstance(element, pyparsing.Opt): - ret = EditablePartial.from_call(railroad.Optional, item="") - elif isinstance(element, pyparsing.OneOrMore): - ret = EditablePartial.from_call(railroad.OneOrMore, item="") - elif isinstance(element, pyparsing.ZeroOrMore): - ret = EditablePartial.from_call(railroad.ZeroOrMore, item="") - elif isinstance(element, pyparsing.Group): - ret = EditablePartial.from_call( - railroad.Group, item=None, label=element_results_name - ) - elif isinstance(element, pyparsing.Empty) and not element.customName: - # Skip unnamed "Empty" elements - ret = None - elif isinstance(element, pyparsing.ParseElementEnhance): - ret = EditablePartial.from_call(railroad.Sequence, items=[]) - elif len(exprs) > 0 and not element_results_name: - ret = EditablePartial.from_call(railroad.Group, item="", label=name) - elif len(exprs) > 0: - ret = EditablePartial.from_call(railroad.Sequence, items=[]) - else: - terminal = EditablePartial.from_call(railroad.Terminal, element.defaultName) - ret = terminal - - if ret is None: - return - - # Indicate this element's position in the tree so we can extract it if necessary - lookup[el_id] = ElementState( - element=element, - converted=ret, - parent=parent, - parent_index=index, - number=lookup.generate_index(), - ) - if element.customName: - lookup[el_id].mark_for_extraction(el_id, lookup, element.customName) - - i = 0 - for expr in exprs: - # Add a placeholder index in case we have to extract the child before we even add it to the parent - if "items" in ret.kwargs: - ret.kwargs["items"].insert(i, None) - - item = _to_diagram_element( - expr, - parent=ret, - lookup=lookup, - vertical=vertical, - index=i, - show_results_names=show_results_names, - show_groups=show_groups, - ) - - # Some elements don't need to be shown in the diagram - if item is not None: - if "item" in ret.kwargs: - ret.kwargs["item"] = item - elif "items" in ret.kwargs: - # If we've already extracted the child, don't touch this index, since it's occupied by a nonterminal - ret.kwargs["items"][i] = item - i += 1 - elif "items" in ret.kwargs: - # If we're supposed to skip this element, remove it from the parent - del ret.kwargs["items"][i] - - # If all this items children are none, skip this item - if ret and ( - ("items" in ret.kwargs and len(ret.kwargs["items"]) == 0) - or ("item" in ret.kwargs and ret.kwargs["item"] is None) - ): - ret = EditablePartial.from_call(railroad.Terminal, name) - - # Mark this element as "complete", ie it has all of its children - if el_id in lookup: - lookup[el_id].complete = True - - if el_id in lookup and lookup[el_id].extract and lookup[el_id].complete: - lookup.extract_into_diagram(el_id) - if ret is not None: - ret = EditablePartial.from_call( - railroad.NonTerminal, text=lookup.diagrams[el_id].kwargs["name"] - ) - - return ret diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/pyparsing/diagram/__pycache__/__init__.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/pyparsing/diagram/__pycache__/__init__.cpython-311.pyc deleted file mode 100644 index adbd0b7..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/pyparsing/diagram/__pycache__/__init__.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/pyparsing/exceptions.py b/venv/lib/python3.11/site-packages/pip/_vendor/pyparsing/exceptions.py deleted file mode 100644 index 12219f1..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/pyparsing/exceptions.py +++ /dev/null @@ -1,299 +0,0 @@ -# exceptions.py - -import re -import sys -import typing - -from .util import ( - col, - line, - lineno, - _collapse_string_to_ranges, - replaced_by_pep8, -) -from .unicode import pyparsing_unicode as ppu - - -class ExceptionWordUnicode(ppu.Latin1, ppu.LatinA, ppu.LatinB, ppu.Greek, ppu.Cyrillic): - pass - - -_extract_alphanums = _collapse_string_to_ranges(ExceptionWordUnicode.alphanums) -_exception_word_extractor = re.compile("([" + _extract_alphanums + "]{1,16})|.") - - -class ParseBaseException(Exception): - """base exception class for all parsing runtime exceptions""" - - loc: int - msg: str - pstr: str - parser_element: typing.Any # "ParserElement" - args: typing.Tuple[str, int, typing.Optional[str]] - - __slots__ = ( - "loc", - "msg", - "pstr", - "parser_element", - "args", - ) - - # Performance tuning: we construct a *lot* of these, so keep this - # constructor as small and fast as possible - def __init__( - self, - pstr: str, - loc: int = 0, - msg: typing.Optional[str] = None, - elem=None, - ): - self.loc = loc - if msg is None: - self.msg = pstr - self.pstr = "" - else: - self.msg = msg - self.pstr = pstr - self.parser_element = elem - self.args = (pstr, loc, msg) - - @staticmethod - def explain_exception(exc, depth=16): - """ - Method to take an exception and translate the Python internal traceback into a list - of the pyparsing expressions that caused the exception to be raised. - - Parameters: - - - exc - exception raised during parsing (need not be a ParseException, in support - of Python exceptions that might be raised in a parse action) - - depth (default=16) - number of levels back in the stack trace to list expression - and function names; if None, the full stack trace names will be listed; if 0, only - the failing input line, marker, and exception string will be shown - - Returns a multi-line string listing the ParserElements and/or function names in the - exception's stack trace. - """ - import inspect - from .core import ParserElement - - if depth is None: - depth = sys.getrecursionlimit() - ret = [] - if isinstance(exc, ParseBaseException): - ret.append(exc.line) - ret.append(" " * (exc.column - 1) + "^") - ret.append(f"{type(exc).__name__}: {exc}") - - if depth > 0: - callers = inspect.getinnerframes(exc.__traceback__, context=depth) - seen = set() - for i, ff in enumerate(callers[-depth:]): - frm = ff[0] - - f_self = frm.f_locals.get("self", None) - if isinstance(f_self, ParserElement): - if not frm.f_code.co_name.startswith( - ("parseImpl", "_parseNoCache") - ): - continue - if id(f_self) in seen: - continue - seen.add(id(f_self)) - - self_type = type(f_self) - ret.append( - f"{self_type.__module__}.{self_type.__name__} - {f_self}" - ) - - elif f_self is not None: - self_type = type(f_self) - ret.append(f"{self_type.__module__}.{self_type.__name__}") - - else: - code = frm.f_code - if code.co_name in ("wrapper", ""): - continue - - ret.append(code.co_name) - - depth -= 1 - if not depth: - break - - return "\n".join(ret) - - @classmethod - def _from_exception(cls, pe): - """ - internal factory method to simplify creating one type of ParseException - from another - avoids having __init__ signature conflicts among subclasses - """ - return cls(pe.pstr, pe.loc, pe.msg, pe.parser_element) - - @property - def line(self) -> str: - """ - Return the line of text where the exception occurred. - """ - return line(self.loc, self.pstr) - - @property - def lineno(self) -> int: - """ - Return the 1-based line number of text where the exception occurred. - """ - return lineno(self.loc, self.pstr) - - @property - def col(self) -> int: - """ - Return the 1-based column on the line of text where the exception occurred. - """ - return col(self.loc, self.pstr) - - @property - def column(self) -> int: - """ - Return the 1-based column on the line of text where the exception occurred. - """ - return col(self.loc, self.pstr) - - # pre-PEP8 compatibility - @property - def parserElement(self): - return self.parser_element - - @parserElement.setter - def parserElement(self, elem): - self.parser_element = elem - - def __str__(self) -> str: - if self.pstr: - if self.loc >= len(self.pstr): - foundstr = ", found end of text" - else: - # pull out next word at error location - found_match = _exception_word_extractor.match(self.pstr, self.loc) - if found_match is not None: - found = found_match.group(0) - else: - found = self.pstr[self.loc : self.loc + 1] - foundstr = (", found %r" % found).replace(r"\\", "\\") - else: - foundstr = "" - return f"{self.msg}{foundstr} (at char {self.loc}), (line:{self.lineno}, col:{self.column})" - - def __repr__(self): - return str(self) - - def mark_input_line( - self, marker_string: typing.Optional[str] = None, *, markerString: str = ">!<" - ) -> str: - """ - Extracts the exception line from the input string, and marks - the location of the exception with a special symbol. - """ - markerString = marker_string if marker_string is not None else markerString - line_str = self.line - line_column = self.column - 1 - if markerString: - line_str = "".join( - (line_str[:line_column], markerString, line_str[line_column:]) - ) - return line_str.strip() - - def explain(self, depth=16) -> str: - """ - Method to translate the Python internal traceback into a list - of the pyparsing expressions that caused the exception to be raised. - - Parameters: - - - depth (default=16) - number of levels back in the stack trace to list expression - and function names; if None, the full stack trace names will be listed; if 0, only - the failing input line, marker, and exception string will be shown - - Returns a multi-line string listing the ParserElements and/or function names in the - exception's stack trace. - - Example:: - - expr = pp.Word(pp.nums) * 3 - try: - expr.parse_string("123 456 A789") - except pp.ParseException as pe: - print(pe.explain(depth=0)) - - prints:: - - 123 456 A789 - ^ - ParseException: Expected W:(0-9), found 'A' (at char 8), (line:1, col:9) - - Note: the diagnostic output will include string representations of the expressions - that failed to parse. These representations will be more helpful if you use `set_name` to - give identifiable names to your expressions. Otherwise they will use the default string - forms, which may be cryptic to read. - - Note: pyparsing's default truncation of exception tracebacks may also truncate the - stack of expressions that are displayed in the ``explain`` output. To get the full listing - of parser expressions, you may have to set ``ParserElement.verbose_stacktrace = True`` - """ - return self.explain_exception(self, depth) - - # fmt: off - @replaced_by_pep8(mark_input_line) - def markInputline(self): ... - # fmt: on - - -class ParseException(ParseBaseException): - """ - Exception thrown when a parse expression doesn't match the input string - - Example:: - - try: - Word(nums).set_name("integer").parse_string("ABC") - except ParseException as pe: - print(pe) - print("column: {}".format(pe.column)) - - prints:: - - Expected integer (at char 0), (line:1, col:1) - column: 1 - - """ - - -class ParseFatalException(ParseBaseException): - """ - User-throwable exception thrown when inconsistent parse content - is found; stops all parsing immediately - """ - - -class ParseSyntaxException(ParseFatalException): - """ - Just like :class:`ParseFatalException`, but thrown internally - when an :class:`ErrorStop` ('-' operator) indicates - that parsing is to stop immediately because an unbacktrackable - syntax error has been found. - """ - - -class RecursiveGrammarException(Exception): - """ - Exception thrown by :class:`ParserElement.validate` if the - grammar could be left-recursive; parser may need to enable - left recursion using :class:`ParserElement.enable_left_recursion` - """ - - def __init__(self, parseElementList): - self.parseElementTrace = parseElementList - - def __str__(self) -> str: - return f"RecursiveGrammarException: {self.parseElementTrace}" diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/pyparsing/helpers.py b/venv/lib/python3.11/site-packages/pip/_vendor/pyparsing/helpers.py deleted file mode 100644 index 018f0d6..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/pyparsing/helpers.py +++ /dev/null @@ -1,1100 +0,0 @@ -# helpers.py -import html.entities -import re -import sys -import typing - -from . import __diag__ -from .core import * -from .util import ( - _bslash, - _flatten, - _escape_regex_range_chars, - replaced_by_pep8, -) - - -# -# global helpers -# -def counted_array( - expr: ParserElement, - int_expr: typing.Optional[ParserElement] = None, - *, - intExpr: typing.Optional[ParserElement] = None, -) -> ParserElement: - """Helper to define a counted list of expressions. - - This helper defines a pattern of the form:: - - integer expr expr expr... - - where the leading integer tells how many expr expressions follow. - The matched tokens returns the array of expr tokens as a list - the - leading count token is suppressed. - - If ``int_expr`` is specified, it should be a pyparsing expression - that produces an integer value. - - Example:: - - counted_array(Word(alphas)).parse_string('2 ab cd ef') # -> ['ab', 'cd'] - - # in this parser, the leading integer value is given in binary, - # '10' indicating that 2 values are in the array - binary_constant = Word('01').set_parse_action(lambda t: int(t[0], 2)) - counted_array(Word(alphas), int_expr=binary_constant).parse_string('10 ab cd ef') # -> ['ab', 'cd'] - - # if other fields must be parsed after the count but before the - # list items, give the fields results names and they will - # be preserved in the returned ParseResults: - count_with_metadata = integer + Word(alphas)("type") - typed_array = counted_array(Word(alphanums), int_expr=count_with_metadata)("items") - result = typed_array.parse_string("3 bool True True False") - print(result.dump()) - - # prints - # ['True', 'True', 'False'] - # - items: ['True', 'True', 'False'] - # - type: 'bool' - """ - intExpr = intExpr or int_expr - array_expr = Forward() - - def count_field_parse_action(s, l, t): - nonlocal array_expr - n = t[0] - array_expr <<= (expr * n) if n else Empty() - # clear list contents, but keep any named results - del t[:] - - if intExpr is None: - intExpr = Word(nums).set_parse_action(lambda t: int(t[0])) - else: - intExpr = intExpr.copy() - intExpr.set_name("arrayLen") - intExpr.add_parse_action(count_field_parse_action, call_during_try=True) - return (intExpr + array_expr).set_name("(len) " + str(expr) + "...") - - -def match_previous_literal(expr: ParserElement) -> ParserElement: - """Helper to define an expression that is indirectly defined from - the tokens matched in a previous expression, that is, it looks for - a 'repeat' of a previous expression. For example:: - - first = Word(nums) - second = match_previous_literal(first) - match_expr = first + ":" + second - - will match ``"1:1"``, but not ``"1:2"``. Because this - matches a previous literal, will also match the leading - ``"1:1"`` in ``"1:10"``. If this is not desired, use - :class:`match_previous_expr`. Do *not* use with packrat parsing - enabled. - """ - rep = Forward() - - def copy_token_to_repeater(s, l, t): - if t: - if len(t) == 1: - rep << t[0] - else: - # flatten t tokens - tflat = _flatten(t.as_list()) - rep << And(Literal(tt) for tt in tflat) - else: - rep << Empty() - - expr.add_parse_action(copy_token_to_repeater, callDuringTry=True) - rep.set_name("(prev) " + str(expr)) - return rep - - -def match_previous_expr(expr: ParserElement) -> ParserElement: - """Helper to define an expression that is indirectly defined from - the tokens matched in a previous expression, that is, it looks for - a 'repeat' of a previous expression. For example:: - - first = Word(nums) - second = match_previous_expr(first) - match_expr = first + ":" + second - - will match ``"1:1"``, but not ``"1:2"``. Because this - matches by expressions, will *not* match the leading ``"1:1"`` - in ``"1:10"``; the expressions are evaluated first, and then - compared, so ``"1"`` is compared with ``"10"``. Do *not* use - with packrat parsing enabled. - """ - rep = Forward() - e2 = expr.copy() - rep <<= e2 - - def copy_token_to_repeater(s, l, t): - matchTokens = _flatten(t.as_list()) - - def must_match_these_tokens(s, l, t): - theseTokens = _flatten(t.as_list()) - if theseTokens != matchTokens: - raise ParseException( - s, l, f"Expected {matchTokens}, found{theseTokens}" - ) - - rep.set_parse_action(must_match_these_tokens, callDuringTry=True) - - expr.add_parse_action(copy_token_to_repeater, callDuringTry=True) - rep.set_name("(prev) " + str(expr)) - return rep - - -def one_of( - strs: Union[typing.Iterable[str], str], - caseless: bool = False, - use_regex: bool = True, - as_keyword: bool = False, - *, - useRegex: bool = True, - asKeyword: bool = False, -) -> ParserElement: - """Helper to quickly define a set of alternative :class:`Literal` s, - and makes sure to do longest-first testing when there is a conflict, - regardless of the input order, but returns - a :class:`MatchFirst` for best performance. - - Parameters: - - - ``strs`` - a string of space-delimited literals, or a collection of - string literals - - ``caseless`` - treat all literals as caseless - (default= ``False``) - - ``use_regex`` - as an optimization, will - generate a :class:`Regex` object; otherwise, will generate - a :class:`MatchFirst` object (if ``caseless=True`` or ``as_keyword=True``, or if - creating a :class:`Regex` raises an exception) - (default= ``True``) - - ``as_keyword`` - enforce :class:`Keyword`-style matching on the - generated expressions - (default= ``False``) - - ``asKeyword`` and ``useRegex`` are retained for pre-PEP8 compatibility, - but will be removed in a future release - - Example:: - - comp_oper = one_of("< = > <= >= !=") - var = Word(alphas) - number = Word(nums) - term = var | number - comparison_expr = term + comp_oper + term - print(comparison_expr.search_string("B = 12 AA=23 B<=AA AA>12")) - - prints:: - - [['B', '=', '12'], ['AA', '=', '23'], ['B', '<=', 'AA'], ['AA', '>', '12']] - """ - asKeyword = asKeyword or as_keyword - useRegex = useRegex and use_regex - - if ( - isinstance(caseless, str_type) - and __diag__.warn_on_multiple_string_args_to_oneof - ): - warnings.warn( - "More than one string argument passed to one_of, pass" - " choices as a list or space-delimited string", - stacklevel=2, - ) - - if caseless: - isequal = lambda a, b: a.upper() == b.upper() - masks = lambda a, b: b.upper().startswith(a.upper()) - parseElementClass = CaselessKeyword if asKeyword else CaselessLiteral - else: - isequal = lambda a, b: a == b - masks = lambda a, b: b.startswith(a) - parseElementClass = Keyword if asKeyword else Literal - - symbols: List[str] = [] - if isinstance(strs, str_type): - strs = typing.cast(str, strs) - symbols = strs.split() - elif isinstance(strs, Iterable): - symbols = list(strs) - else: - raise TypeError("Invalid argument to one_of, expected string or iterable") - if not symbols: - return NoMatch() - - # reorder given symbols to take care to avoid masking longer choices with shorter ones - # (but only if the given symbols are not just single characters) - if any(len(sym) > 1 for sym in symbols): - i = 0 - while i < len(symbols) - 1: - cur = symbols[i] - for j, other in enumerate(symbols[i + 1 :]): - if isequal(other, cur): - del symbols[i + j + 1] - break - elif masks(cur, other): - del symbols[i + j + 1] - symbols.insert(i, other) - break - else: - i += 1 - - if useRegex: - re_flags: int = re.IGNORECASE if caseless else 0 - - try: - if all(len(sym) == 1 for sym in symbols): - # symbols are just single characters, create range regex pattern - patt = f"[{''.join(_escape_regex_range_chars(sym) for sym in symbols)}]" - else: - patt = "|".join(re.escape(sym) for sym in symbols) - - # wrap with \b word break markers if defining as keywords - if asKeyword: - patt = rf"\b(?:{patt})\b" - - ret = Regex(patt, flags=re_flags).set_name(" | ".join(symbols)) - - if caseless: - # add parse action to return symbols as specified, not in random - # casing as found in input string - symbol_map = {sym.lower(): sym for sym in symbols} - ret.add_parse_action(lambda s, l, t: symbol_map[t[0].lower()]) - - return ret - - except re.error: - warnings.warn( - "Exception creating Regex for one_of, building MatchFirst", stacklevel=2 - ) - - # last resort, just use MatchFirst - return MatchFirst(parseElementClass(sym) for sym in symbols).set_name( - " | ".join(symbols) - ) - - -def dict_of(key: ParserElement, value: ParserElement) -> ParserElement: - """Helper to easily and clearly define a dictionary by specifying - the respective patterns for the key and value. Takes care of - defining the :class:`Dict`, :class:`ZeroOrMore`, and - :class:`Group` tokens in the proper order. The key pattern - can include delimiting markers or punctuation, as long as they are - suppressed, thereby leaving the significant key text. The value - pattern can include named results, so that the :class:`Dict` results - can include named token fields. - - Example:: - - text = "shape: SQUARE posn: upper left color: light blue texture: burlap" - attr_expr = (label + Suppress(':') + OneOrMore(data_word, stop_on=label).set_parse_action(' '.join)) - print(attr_expr[1, ...].parse_string(text).dump()) - - attr_label = label - attr_value = Suppress(':') + OneOrMore(data_word, stop_on=label).set_parse_action(' '.join) - - # similar to Dict, but simpler call format - result = dict_of(attr_label, attr_value).parse_string(text) - print(result.dump()) - print(result['shape']) - print(result.shape) # object attribute access works too - print(result.as_dict()) - - prints:: - - [['shape', 'SQUARE'], ['posn', 'upper left'], ['color', 'light blue'], ['texture', 'burlap']] - - color: 'light blue' - - posn: 'upper left' - - shape: 'SQUARE' - - texture: 'burlap' - SQUARE - SQUARE - {'color': 'light blue', 'shape': 'SQUARE', 'posn': 'upper left', 'texture': 'burlap'} - """ - return Dict(OneOrMore(Group(key + value))) - - -def original_text_for( - expr: ParserElement, as_string: bool = True, *, asString: bool = True -) -> ParserElement: - """Helper to return the original, untokenized text for a given - expression. Useful to restore the parsed fields of an HTML start - tag into the raw tag text itself, or to revert separate tokens with - intervening whitespace back to the original matching input text. By - default, returns a string containing the original parsed text. - - If the optional ``as_string`` argument is passed as - ``False``, then the return value is - a :class:`ParseResults` containing any results names that - were originally matched, and a single token containing the original - matched text from the input string. So if the expression passed to - :class:`original_text_for` contains expressions with defined - results names, you must set ``as_string`` to ``False`` if you - want to preserve those results name values. - - The ``asString`` pre-PEP8 argument is retained for compatibility, - but will be removed in a future release. - - Example:: - - src = "this is test bold text normal text " - for tag in ("b", "i"): - opener, closer = make_html_tags(tag) - patt = original_text_for(opener + ... + closer) - print(patt.search_string(src)[0]) - - prints:: - - [' bold text '] - ['text'] - """ - asString = asString and as_string - - locMarker = Empty().set_parse_action(lambda s, loc, t: loc) - endlocMarker = locMarker.copy() - endlocMarker.callPreparse = False - matchExpr = locMarker("_original_start") + expr + endlocMarker("_original_end") - if asString: - extractText = lambda s, l, t: s[t._original_start : t._original_end] - else: - - def extractText(s, l, t): - t[:] = [s[t.pop("_original_start") : t.pop("_original_end")]] - - matchExpr.set_parse_action(extractText) - matchExpr.ignoreExprs = expr.ignoreExprs - matchExpr.suppress_warning(Diagnostics.warn_ungrouped_named_tokens_in_collection) - return matchExpr - - -def ungroup(expr: ParserElement) -> ParserElement: - """Helper to undo pyparsing's default grouping of And expressions, - even if all but one are non-empty. - """ - return TokenConverter(expr).add_parse_action(lambda t: t[0]) - - -def locatedExpr(expr: ParserElement) -> ParserElement: - """ - (DEPRECATED - future code should use the :class:`Located` class) - Helper to decorate a returned token with its starting and ending - locations in the input string. - - This helper adds the following results names: - - - ``locn_start`` - location where matched expression begins - - ``locn_end`` - location where matched expression ends - - ``value`` - the actual parsed results - - Be careful if the input text contains ```` characters, you - may want to call :class:`ParserElement.parse_with_tabs` - - Example:: - - wd = Word(alphas) - for match in locatedExpr(wd).search_string("ljsdf123lksdjjf123lkkjj1222"): - print(match) - - prints:: - - [[0, 'ljsdf', 5]] - [[8, 'lksdjjf', 15]] - [[18, 'lkkjj', 23]] - """ - locator = Empty().set_parse_action(lambda ss, ll, tt: ll) - return Group( - locator("locn_start") - + expr("value") - + locator.copy().leaveWhitespace()("locn_end") - ) - - -def nested_expr( - opener: Union[str, ParserElement] = "(", - closer: Union[str, ParserElement] = ")", - content: typing.Optional[ParserElement] = None, - ignore_expr: ParserElement = quoted_string(), - *, - ignoreExpr: ParserElement = quoted_string(), -) -> ParserElement: - """Helper method for defining nested lists enclosed in opening and - closing delimiters (``"("`` and ``")"`` are the default). - - Parameters: - - - ``opener`` - opening character for a nested list - (default= ``"("``); can also be a pyparsing expression - - ``closer`` - closing character for a nested list - (default= ``")"``); can also be a pyparsing expression - - ``content`` - expression for items within the nested lists - (default= ``None``) - - ``ignore_expr`` - expression for ignoring opening and closing delimiters - (default= :class:`quoted_string`) - - ``ignoreExpr`` - this pre-PEP8 argument is retained for compatibility - but will be removed in a future release - - If an expression is not provided for the content argument, the - nested expression will capture all whitespace-delimited content - between delimiters as a list of separate values. - - Use the ``ignore_expr`` argument to define expressions that may - contain opening or closing characters that should not be treated as - opening or closing characters for nesting, such as quoted_string or - a comment expression. Specify multiple expressions using an - :class:`Or` or :class:`MatchFirst`. The default is - :class:`quoted_string`, but if no expressions are to be ignored, then - pass ``None`` for this argument. - - Example:: - - data_type = one_of("void int short long char float double") - decl_data_type = Combine(data_type + Opt(Word('*'))) - ident = Word(alphas+'_', alphanums+'_') - number = pyparsing_common.number - arg = Group(decl_data_type + ident) - LPAR, RPAR = map(Suppress, "()") - - code_body = nested_expr('{', '}', ignore_expr=(quoted_string | c_style_comment)) - - c_function = (decl_data_type("type") - + ident("name") - + LPAR + Opt(DelimitedList(arg), [])("args") + RPAR - + code_body("body")) - c_function.ignore(c_style_comment) - - source_code = ''' - int is_odd(int x) { - return (x%2); - } - - int dec_to_hex(char hchar) { - if (hchar >= '0' && hchar <= '9') { - return (ord(hchar)-ord('0')); - } else { - return (10+ord(hchar)-ord('A')); - } - } - ''' - for func in c_function.search_string(source_code): - print("%(name)s (%(type)s) args: %(args)s" % func) - - - prints:: - - is_odd (int) args: [['int', 'x']] - dec_to_hex (int) args: [['char', 'hchar']] - """ - if ignoreExpr != ignore_expr: - ignoreExpr = ignore_expr if ignoreExpr == quoted_string() else ignoreExpr - if opener == closer: - raise ValueError("opening and closing strings cannot be the same") - if content is None: - if isinstance(opener, str_type) and isinstance(closer, str_type): - opener = typing.cast(str, opener) - closer = typing.cast(str, closer) - if len(opener) == 1 and len(closer) == 1: - if ignoreExpr is not None: - content = Combine( - OneOrMore( - ~ignoreExpr - + CharsNotIn( - opener + closer + ParserElement.DEFAULT_WHITE_CHARS, - exact=1, - ) - ) - ).set_parse_action(lambda t: t[0].strip()) - else: - content = empty.copy() + CharsNotIn( - opener + closer + ParserElement.DEFAULT_WHITE_CHARS - ).set_parse_action(lambda t: t[0].strip()) - else: - if ignoreExpr is not None: - content = Combine( - OneOrMore( - ~ignoreExpr - + ~Literal(opener) - + ~Literal(closer) - + CharsNotIn(ParserElement.DEFAULT_WHITE_CHARS, exact=1) - ) - ).set_parse_action(lambda t: t[0].strip()) - else: - content = Combine( - OneOrMore( - ~Literal(opener) - + ~Literal(closer) - + CharsNotIn(ParserElement.DEFAULT_WHITE_CHARS, exact=1) - ) - ).set_parse_action(lambda t: t[0].strip()) - else: - raise ValueError( - "opening and closing arguments must be strings if no content expression is given" - ) - ret = Forward() - if ignoreExpr is not None: - ret <<= Group( - Suppress(opener) + ZeroOrMore(ignoreExpr | ret | content) + Suppress(closer) - ) - else: - ret <<= Group(Suppress(opener) + ZeroOrMore(ret | content) + Suppress(closer)) - ret.set_name("nested %s%s expression" % (opener, closer)) - return ret - - -def _makeTags(tagStr, xml, suppress_LT=Suppress("<"), suppress_GT=Suppress(">")): - """Internal helper to construct opening and closing tag expressions, given a tag name""" - if isinstance(tagStr, str_type): - resname = tagStr - tagStr = Keyword(tagStr, caseless=not xml) - else: - resname = tagStr.name - - tagAttrName = Word(alphas, alphanums + "_-:") - if xml: - tagAttrValue = dbl_quoted_string.copy().set_parse_action(remove_quotes) - openTag = ( - suppress_LT - + tagStr("tag") - + Dict(ZeroOrMore(Group(tagAttrName + Suppress("=") + tagAttrValue))) - + Opt("/", default=[False])("empty").set_parse_action( - lambda s, l, t: t[0] == "/" - ) - + suppress_GT - ) - else: - tagAttrValue = quoted_string.copy().set_parse_action(remove_quotes) | Word( - printables, exclude_chars=">" - ) - openTag = ( - suppress_LT - + tagStr("tag") - + Dict( - ZeroOrMore( - Group( - tagAttrName.set_parse_action(lambda t: t[0].lower()) - + Opt(Suppress("=") + tagAttrValue) - ) - ) - ) - + Opt("/", default=[False])("empty").set_parse_action( - lambda s, l, t: t[0] == "/" - ) - + suppress_GT - ) - closeTag = Combine(Literal("", adjacent=False) - - openTag.set_name("<%s>" % resname) - # add start results name in parse action now that ungrouped names are not reported at two levels - openTag.add_parse_action( - lambda t: t.__setitem__( - "start" + "".join(resname.replace(":", " ").title().split()), t.copy() - ) - ) - closeTag = closeTag( - "end" + "".join(resname.replace(":", " ").title().split()) - ).set_name("" % resname) - openTag.tag = resname - closeTag.tag = resname - openTag.tag_body = SkipTo(closeTag()) - return openTag, closeTag - - -def make_html_tags( - tag_str: Union[str, ParserElement] -) -> Tuple[ParserElement, ParserElement]: - """Helper to construct opening and closing tag expressions for HTML, - given a tag name. Matches tags in either upper or lower case, - attributes with namespaces and with quoted or unquoted values. - - Example:: - - text = 'More info at the pyparsing wiki page' - # make_html_tags returns pyparsing expressions for the opening and - # closing tags as a 2-tuple - a, a_end = make_html_tags("A") - link_expr = a + SkipTo(a_end)("link_text") + a_end - - for link in link_expr.search_string(text): - # attributes in the tag (like "href" shown here) are - # also accessible as named results - print(link.link_text, '->', link.href) - - prints:: - - pyparsing -> https://github.com/pyparsing/pyparsing/wiki - """ - return _makeTags(tag_str, False) - - -def make_xml_tags( - tag_str: Union[str, ParserElement] -) -> Tuple[ParserElement, ParserElement]: - """Helper to construct opening and closing tag expressions for XML, - given a tag name. Matches tags only in the given upper/lower case. - - Example: similar to :class:`make_html_tags` - """ - return _makeTags(tag_str, True) - - -any_open_tag: ParserElement -any_close_tag: ParserElement -any_open_tag, any_close_tag = make_html_tags( - Word(alphas, alphanums + "_:").set_name("any tag") -) - -_htmlEntityMap = {k.rstrip(";"): v for k, v in html.entities.html5.items()} -common_html_entity = Regex("&(?P" + "|".join(_htmlEntityMap) + ");").set_name( - "common HTML entity" -) - - -def replace_html_entity(s, l, t): - """Helper parser action to replace common HTML entities with their special characters""" - return _htmlEntityMap.get(t.entity) - - -class OpAssoc(Enum): - """Enumeration of operator associativity - - used in constructing InfixNotationOperatorSpec for :class:`infix_notation`""" - - LEFT = 1 - RIGHT = 2 - - -InfixNotationOperatorArgType = Union[ - ParserElement, str, Tuple[Union[ParserElement, str], Union[ParserElement, str]] -] -InfixNotationOperatorSpec = Union[ - Tuple[ - InfixNotationOperatorArgType, - int, - OpAssoc, - typing.Optional[ParseAction], - ], - Tuple[ - InfixNotationOperatorArgType, - int, - OpAssoc, - ], -] - - -def infix_notation( - base_expr: ParserElement, - op_list: List[InfixNotationOperatorSpec], - lpar: Union[str, ParserElement] = Suppress("("), - rpar: Union[str, ParserElement] = Suppress(")"), -) -> ParserElement: - """Helper method for constructing grammars of expressions made up of - operators working in a precedence hierarchy. Operators may be unary - or binary, left- or right-associative. Parse actions can also be - attached to operator expressions. The generated parser will also - recognize the use of parentheses to override operator precedences - (see example below). - - Note: if you define a deep operator list, you may see performance - issues when using infix_notation. See - :class:`ParserElement.enable_packrat` for a mechanism to potentially - improve your parser performance. - - Parameters: - - - ``base_expr`` - expression representing the most basic operand to - be used in the expression - - ``op_list`` - list of tuples, one for each operator precedence level - in the expression grammar; each tuple is of the form ``(op_expr, - num_operands, right_left_assoc, (optional)parse_action)``, where: - - - ``op_expr`` is the pyparsing expression for the operator; may also - be a string, which will be converted to a Literal; if ``num_operands`` - is 3, ``op_expr`` is a tuple of two expressions, for the two - operators separating the 3 terms - - ``num_operands`` is the number of terms for this operator (must be 1, - 2, or 3) - - ``right_left_assoc`` is the indicator whether the operator is right - or left associative, using the pyparsing-defined constants - ``OpAssoc.RIGHT`` and ``OpAssoc.LEFT``. - - ``parse_action`` is the parse action to be associated with - expressions matching this operator expression (the parse action - tuple member may be omitted); if the parse action is passed - a tuple or list of functions, this is equivalent to calling - ``set_parse_action(*fn)`` - (:class:`ParserElement.set_parse_action`) - - ``lpar`` - expression for matching left-parentheses; if passed as a - str, then will be parsed as ``Suppress(lpar)``. If lpar is passed as - an expression (such as ``Literal('(')``), then it will be kept in - the parsed results, and grouped with them. (default= ``Suppress('(')``) - - ``rpar`` - expression for matching right-parentheses; if passed as a - str, then will be parsed as ``Suppress(rpar)``. If rpar is passed as - an expression (such as ``Literal(')')``), then it will be kept in - the parsed results, and grouped with them. (default= ``Suppress(')')``) - - Example:: - - # simple example of four-function arithmetic with ints and - # variable names - integer = pyparsing_common.signed_integer - varname = pyparsing_common.identifier - - arith_expr = infix_notation(integer | varname, - [ - ('-', 1, OpAssoc.RIGHT), - (one_of('* /'), 2, OpAssoc.LEFT), - (one_of('+ -'), 2, OpAssoc.LEFT), - ]) - - arith_expr.run_tests(''' - 5+3*6 - (5+3)*6 - -2--11 - ''', full_dump=False) - - prints:: - - 5+3*6 - [[5, '+', [3, '*', 6]]] - - (5+3)*6 - [[[5, '+', 3], '*', 6]] - - (5+x)*y - [[[5, '+', 'x'], '*', 'y']] - - -2--11 - [[['-', 2], '-', ['-', 11]]] - """ - - # captive version of FollowedBy that does not do parse actions or capture results names - class _FB(FollowedBy): - def parseImpl(self, instring, loc, doActions=True): - self.expr.try_parse(instring, loc) - return loc, [] - - _FB.__name__ = "FollowedBy>" - - ret = Forward() - if isinstance(lpar, str): - lpar = Suppress(lpar) - if isinstance(rpar, str): - rpar = Suppress(rpar) - - # if lpar and rpar are not suppressed, wrap in group - if not (isinstance(rpar, Suppress) and isinstance(rpar, Suppress)): - lastExpr = base_expr | Group(lpar + ret + rpar) - else: - lastExpr = base_expr | (lpar + ret + rpar) - - arity: int - rightLeftAssoc: opAssoc - pa: typing.Optional[ParseAction] - opExpr1: ParserElement - opExpr2: ParserElement - for i, operDef in enumerate(op_list): - opExpr, arity, rightLeftAssoc, pa = (operDef + (None,))[:4] # type: ignore[assignment] - if isinstance(opExpr, str_type): - opExpr = ParserElement._literalStringClass(opExpr) - opExpr = typing.cast(ParserElement, opExpr) - if arity == 3: - if not isinstance(opExpr, (tuple, list)) or len(opExpr) != 2: - raise ValueError( - "if numterms=3, opExpr must be a tuple or list of two expressions" - ) - opExpr1, opExpr2 = opExpr - term_name = f"{opExpr1}{opExpr2} term" - else: - term_name = f"{opExpr} term" - - if not 1 <= arity <= 3: - raise ValueError("operator must be unary (1), binary (2), or ternary (3)") - - if rightLeftAssoc not in (OpAssoc.LEFT, OpAssoc.RIGHT): - raise ValueError("operator must indicate right or left associativity") - - thisExpr: ParserElement = Forward().set_name(term_name) - thisExpr = typing.cast(Forward, thisExpr) - if rightLeftAssoc is OpAssoc.LEFT: - if arity == 1: - matchExpr = _FB(lastExpr + opExpr) + Group(lastExpr + opExpr[1, ...]) - elif arity == 2: - if opExpr is not None: - matchExpr = _FB(lastExpr + opExpr + lastExpr) + Group( - lastExpr + (opExpr + lastExpr)[1, ...] - ) - else: - matchExpr = _FB(lastExpr + lastExpr) + Group(lastExpr[2, ...]) - elif arity == 3: - matchExpr = _FB( - lastExpr + opExpr1 + lastExpr + opExpr2 + lastExpr - ) + Group(lastExpr + OneOrMore(opExpr1 + lastExpr + opExpr2 + lastExpr)) - elif rightLeftAssoc is OpAssoc.RIGHT: - if arity == 1: - # try to avoid LR with this extra test - if not isinstance(opExpr, Opt): - opExpr = Opt(opExpr) - matchExpr = _FB(opExpr.expr + thisExpr) + Group(opExpr + thisExpr) - elif arity == 2: - if opExpr is not None: - matchExpr = _FB(lastExpr + opExpr + thisExpr) + Group( - lastExpr + (opExpr + thisExpr)[1, ...] - ) - else: - matchExpr = _FB(lastExpr + thisExpr) + Group( - lastExpr + thisExpr[1, ...] - ) - elif arity == 3: - matchExpr = _FB( - lastExpr + opExpr1 + thisExpr + opExpr2 + thisExpr - ) + Group(lastExpr + opExpr1 + thisExpr + opExpr2 + thisExpr) - if pa: - if isinstance(pa, (tuple, list)): - matchExpr.set_parse_action(*pa) - else: - matchExpr.set_parse_action(pa) - thisExpr <<= (matchExpr | lastExpr).setName(term_name) - lastExpr = thisExpr - ret <<= lastExpr - return ret - - -def indentedBlock(blockStatementExpr, indentStack, indent=True, backup_stacks=[]): - """ - (DEPRECATED - use :class:`IndentedBlock` class instead) - Helper method for defining space-delimited indentation blocks, - such as those used to define block statements in Python source code. - - Parameters: - - - ``blockStatementExpr`` - expression defining syntax of statement that - is repeated within the indented block - - ``indentStack`` - list created by caller to manage indentation stack - (multiple ``statementWithIndentedBlock`` expressions within a single - grammar should share a common ``indentStack``) - - ``indent`` - boolean indicating whether block must be indented beyond - the current level; set to ``False`` for block of left-most statements - (default= ``True``) - - A valid block must contain at least one ``blockStatement``. - - (Note that indentedBlock uses internal parse actions which make it - incompatible with packrat parsing.) - - Example:: - - data = ''' - def A(z): - A1 - B = 100 - G = A2 - A2 - A3 - B - def BB(a,b,c): - BB1 - def BBA(): - bba1 - bba2 - bba3 - C - D - def spam(x,y): - def eggs(z): - pass - ''' - - - indentStack = [1] - stmt = Forward() - - identifier = Word(alphas, alphanums) - funcDecl = ("def" + identifier + Group("(" + Opt(delimitedList(identifier)) + ")") + ":") - func_body = indentedBlock(stmt, indentStack) - funcDef = Group(funcDecl + func_body) - - rvalue = Forward() - funcCall = Group(identifier + "(" + Opt(delimitedList(rvalue)) + ")") - rvalue << (funcCall | identifier | Word(nums)) - assignment = Group(identifier + "=" + rvalue) - stmt << (funcDef | assignment | identifier) - - module_body = stmt[1, ...] - - parseTree = module_body.parseString(data) - parseTree.pprint() - - prints:: - - [['def', - 'A', - ['(', 'z', ')'], - ':', - [['A1'], [['B', '=', '100']], [['G', '=', 'A2']], ['A2'], ['A3']]], - 'B', - ['def', - 'BB', - ['(', 'a', 'b', 'c', ')'], - ':', - [['BB1'], [['def', 'BBA', ['(', ')'], ':', [['bba1'], ['bba2'], ['bba3']]]]]], - 'C', - 'D', - ['def', - 'spam', - ['(', 'x', 'y', ')'], - ':', - [[['def', 'eggs', ['(', 'z', ')'], ':', [['pass']]]]]]] - """ - backup_stacks.append(indentStack[:]) - - def reset_stack(): - indentStack[:] = backup_stacks[-1] - - def checkPeerIndent(s, l, t): - if l >= len(s): - return - curCol = col(l, s) - if curCol != indentStack[-1]: - if curCol > indentStack[-1]: - raise ParseException(s, l, "illegal nesting") - raise ParseException(s, l, "not a peer entry") - - def checkSubIndent(s, l, t): - curCol = col(l, s) - if curCol > indentStack[-1]: - indentStack.append(curCol) - else: - raise ParseException(s, l, "not a subentry") - - def checkUnindent(s, l, t): - if l >= len(s): - return - curCol = col(l, s) - if not (indentStack and curCol in indentStack): - raise ParseException(s, l, "not an unindent") - if curCol < indentStack[-1]: - indentStack.pop() - - NL = OneOrMore(LineEnd().set_whitespace_chars("\t ").suppress()) - INDENT = (Empty() + Empty().set_parse_action(checkSubIndent)).set_name("INDENT") - PEER = Empty().set_parse_action(checkPeerIndent).set_name("") - UNDENT = Empty().set_parse_action(checkUnindent).set_name("UNINDENT") - if indent: - smExpr = Group( - Opt(NL) - + INDENT - + OneOrMore(PEER + Group(blockStatementExpr) + Opt(NL)) - + UNDENT - ) - else: - smExpr = Group( - Opt(NL) - + OneOrMore(PEER + Group(blockStatementExpr) + Opt(NL)) - + Opt(UNDENT) - ) - - # add a parse action to remove backup_stack from list of backups - smExpr.add_parse_action( - lambda: backup_stacks.pop(-1) and None if backup_stacks else None - ) - smExpr.set_fail_action(lambda a, b, c, d: reset_stack()) - blockStatementExpr.ignore(_bslash + LineEnd()) - return smExpr.set_name("indented block") - - -# it's easy to get these comment structures wrong - they're very common, so may as well make them available -c_style_comment = Combine(Regex(r"/\*(?:[^*]|\*(?!/))*") + "*/").set_name( - "C style comment" -) -"Comment of the form ``/* ... */``" - -html_comment = Regex(r"").set_name("HTML comment") -"Comment of the form ````" - -rest_of_line = Regex(r".*").leave_whitespace().set_name("rest of line") -dbl_slash_comment = Regex(r"//(?:\\\n|[^\n])*").set_name("// comment") -"Comment of the form ``// ... (to end of line)``" - -cpp_style_comment = Combine( - Regex(r"/\*(?:[^*]|\*(?!/))*") + "*/" | dbl_slash_comment -).set_name("C++ style comment") -"Comment of either form :class:`c_style_comment` or :class:`dbl_slash_comment`" - -java_style_comment = cpp_style_comment -"Same as :class:`cpp_style_comment`" - -python_style_comment = Regex(r"#.*").set_name("Python style comment") -"Comment of the form ``# ... (to end of line)``" - - -# build list of built-in expressions, for future reference if a global default value -# gets updated -_builtin_exprs: List[ParserElement] = [ - v for v in vars().values() if isinstance(v, ParserElement) -] - - -# compatibility function, superseded by DelimitedList class -def delimited_list( - expr: Union[str, ParserElement], - delim: Union[str, ParserElement] = ",", - combine: bool = False, - min: typing.Optional[int] = None, - max: typing.Optional[int] = None, - *, - allow_trailing_delim: bool = False, -) -> ParserElement: - """(DEPRECATED - use :class:`DelimitedList` class)""" - return DelimitedList( - expr, delim, combine, min, max, allow_trailing_delim=allow_trailing_delim - ) - - -# pre-PEP8 compatible names -# fmt: off -opAssoc = OpAssoc -anyOpenTag = any_open_tag -anyCloseTag = any_close_tag -commonHTMLEntity = common_html_entity -cStyleComment = c_style_comment -htmlComment = html_comment -restOfLine = rest_of_line -dblSlashComment = dbl_slash_comment -cppStyleComment = cpp_style_comment -javaStyleComment = java_style_comment -pythonStyleComment = python_style_comment - -@replaced_by_pep8(DelimitedList) -def delimitedList(): ... - -@replaced_by_pep8(DelimitedList) -def delimited_list(): ... - -@replaced_by_pep8(counted_array) -def countedArray(): ... - -@replaced_by_pep8(match_previous_literal) -def matchPreviousLiteral(): ... - -@replaced_by_pep8(match_previous_expr) -def matchPreviousExpr(): ... - -@replaced_by_pep8(one_of) -def oneOf(): ... - -@replaced_by_pep8(dict_of) -def dictOf(): ... - -@replaced_by_pep8(original_text_for) -def originalTextFor(): ... - -@replaced_by_pep8(nested_expr) -def nestedExpr(): ... - -@replaced_by_pep8(make_html_tags) -def makeHTMLTags(): ... - -@replaced_by_pep8(make_xml_tags) -def makeXMLTags(): ... - -@replaced_by_pep8(replace_html_entity) -def replaceHTMLEntity(): ... - -@replaced_by_pep8(infix_notation) -def infixNotation(): ... -# fmt: on diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/pyparsing/results.py b/venv/lib/python3.11/site-packages/pip/_vendor/pyparsing/results.py deleted file mode 100644 index 0313049..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/pyparsing/results.py +++ /dev/null @@ -1,796 +0,0 @@ -# results.py -from collections.abc import ( - MutableMapping, - Mapping, - MutableSequence, - Iterator, - Sequence, - Container, -) -import pprint -from typing import Tuple, Any, Dict, Set, List - -str_type: Tuple[type, ...] = (str, bytes) -_generator_type = type((_ for _ in ())) - - -class _ParseResultsWithOffset: - tup: Tuple["ParseResults", int] - __slots__ = ["tup"] - - def __init__(self, p1: "ParseResults", p2: int): - self.tup: Tuple[ParseResults, int] = (p1, p2) - - def __getitem__(self, i): - return self.tup[i] - - def __getstate__(self): - return self.tup - - def __setstate__(self, *args): - self.tup = args[0] - - -class ParseResults: - """Structured parse results, to provide multiple means of access to - the parsed data: - - - as a list (``len(results)``) - - by list index (``results[0], results[1]``, etc.) - - by attribute (``results.`` - see :class:`ParserElement.set_results_name`) - - Example:: - - integer = Word(nums) - date_str = (integer.set_results_name("year") + '/' - + integer.set_results_name("month") + '/' - + integer.set_results_name("day")) - # equivalent form: - # date_str = (integer("year") + '/' - # + integer("month") + '/' - # + integer("day")) - - # parse_string returns a ParseResults object - result = date_str.parse_string("1999/12/31") - - def test(s, fn=repr): - print(f"{s} -> {fn(eval(s))}") - test("list(result)") - test("result[0]") - test("result['month']") - test("result.day") - test("'month' in result") - test("'minutes' in result") - test("result.dump()", str) - - prints:: - - list(result) -> ['1999', '/', '12', '/', '31'] - result[0] -> '1999' - result['month'] -> '12' - result.day -> '31' - 'month' in result -> True - 'minutes' in result -> False - result.dump() -> ['1999', '/', '12', '/', '31'] - - day: '31' - - month: '12' - - year: '1999' - """ - - _null_values: Tuple[Any, ...] = (None, [], ()) - - _name: str - _parent: "ParseResults" - _all_names: Set[str] - _modal: bool - _toklist: List[Any] - _tokdict: Dict[str, Any] - - __slots__ = ( - "_name", - "_parent", - "_all_names", - "_modal", - "_toklist", - "_tokdict", - ) - - class List(list): - """ - Simple wrapper class to distinguish parsed list results that should be preserved - as actual Python lists, instead of being converted to :class:`ParseResults`:: - - LBRACK, RBRACK = map(pp.Suppress, "[]") - element = pp.Forward() - item = ppc.integer - element_list = LBRACK + pp.DelimitedList(element) + RBRACK - - # add parse actions to convert from ParseResults to actual Python collection types - def as_python_list(t): - return pp.ParseResults.List(t.as_list()) - element_list.add_parse_action(as_python_list) - - element <<= item | element_list - - element.run_tests(''' - 100 - [2,3,4] - [[2, 1],3,4] - [(2, 1),3,4] - (2,3,4) - ''', post_parse=lambda s, r: (r[0], type(r[0]))) - - prints:: - - 100 - (100, ) - - [2,3,4] - ([2, 3, 4], ) - - [[2, 1],3,4] - ([[2, 1], 3, 4], ) - - (Used internally by :class:`Group` when `aslist=True`.) - """ - - def __new__(cls, contained=None): - if contained is None: - contained = [] - - if not isinstance(contained, list): - raise TypeError( - f"{cls.__name__} may only be constructed with a list, not {type(contained).__name__}" - ) - - return list.__new__(cls) - - def __new__(cls, toklist=None, name=None, **kwargs): - if isinstance(toklist, ParseResults): - return toklist - self = object.__new__(cls) - self._name = None - self._parent = None - self._all_names = set() - - if toklist is None: - self._toklist = [] - elif isinstance(toklist, (list, _generator_type)): - self._toklist = ( - [toklist[:]] - if isinstance(toklist, ParseResults.List) - else list(toklist) - ) - else: - self._toklist = [toklist] - self._tokdict = dict() - return self - - # Performance tuning: we construct a *lot* of these, so keep this - # constructor as small and fast as possible - def __init__( - self, toklist=None, name=None, asList=True, modal=True, isinstance=isinstance - ): - self._tokdict: Dict[str, _ParseResultsWithOffset] - self._modal = modal - if name is not None and name != "": - if isinstance(name, int): - name = str(name) - if not modal: - self._all_names = {name} - self._name = name - if toklist not in self._null_values: - if isinstance(toklist, (str_type, type)): - toklist = [toklist] - if asList: - if isinstance(toklist, ParseResults): - self[name] = _ParseResultsWithOffset( - ParseResults(toklist._toklist), 0 - ) - else: - self[name] = _ParseResultsWithOffset( - ParseResults(toklist[0]), 0 - ) - self[name]._name = name - else: - try: - self[name] = toklist[0] - except (KeyError, TypeError, IndexError): - if toklist is not self: - self[name] = toklist - else: - self._name = name - - def __getitem__(self, i): - if isinstance(i, (int, slice)): - return self._toklist[i] - else: - if i not in self._all_names: - return self._tokdict[i][-1][0] - else: - return ParseResults([v[0] for v in self._tokdict[i]]) - - def __setitem__(self, k, v, isinstance=isinstance): - if isinstance(v, _ParseResultsWithOffset): - self._tokdict[k] = self._tokdict.get(k, list()) + [v] - sub = v[0] - elif isinstance(k, (int, slice)): - self._toklist[k] = v - sub = v - else: - self._tokdict[k] = self._tokdict.get(k, list()) + [ - _ParseResultsWithOffset(v, 0) - ] - sub = v - if isinstance(sub, ParseResults): - sub._parent = self - - def __delitem__(self, i): - if isinstance(i, (int, slice)): - mylen = len(self._toklist) - del self._toklist[i] - - # convert int to slice - if isinstance(i, int): - if i < 0: - i += mylen - i = slice(i, i + 1) - # get removed indices - removed = list(range(*i.indices(mylen))) - removed.reverse() - # fixup indices in token dictionary - for name, occurrences in self._tokdict.items(): - for j in removed: - for k, (value, position) in enumerate(occurrences): - occurrences[k] = _ParseResultsWithOffset( - value, position - (position > j) - ) - else: - del self._tokdict[i] - - def __contains__(self, k) -> bool: - return k in self._tokdict - - def __len__(self) -> int: - return len(self._toklist) - - def __bool__(self) -> bool: - return not not (self._toklist or self._tokdict) - - def __iter__(self) -> Iterator: - return iter(self._toklist) - - def __reversed__(self) -> Iterator: - return iter(self._toklist[::-1]) - - def keys(self): - return iter(self._tokdict) - - def values(self): - return (self[k] for k in self.keys()) - - def items(self): - return ((k, self[k]) for k in self.keys()) - - def haskeys(self) -> bool: - """ - Since ``keys()`` returns an iterator, this method is helpful in bypassing - code that looks for the existence of any defined results names.""" - return not not self._tokdict - - def pop(self, *args, **kwargs): - """ - Removes and returns item at specified index (default= ``last``). - Supports both ``list`` and ``dict`` semantics for ``pop()``. If - passed no argument or an integer argument, it will use ``list`` - semantics and pop tokens from the list of parsed tokens. If passed - a non-integer argument (most likely a string), it will use ``dict`` - semantics and pop the corresponding value from any defined results - names. A second default return value argument is supported, just as in - ``dict.pop()``. - - Example:: - - numlist = Word(nums)[...] - print(numlist.parse_string("0 123 321")) # -> ['0', '123', '321'] - - def remove_first(tokens): - tokens.pop(0) - numlist.add_parse_action(remove_first) - print(numlist.parse_string("0 123 321")) # -> ['123', '321'] - - label = Word(alphas) - patt = label("LABEL") + Word(nums)[1, ...] - print(patt.parse_string("AAB 123 321").dump()) - - # Use pop() in a parse action to remove named result (note that corresponding value is not - # removed from list form of results) - def remove_LABEL(tokens): - tokens.pop("LABEL") - return tokens - patt.add_parse_action(remove_LABEL) - print(patt.parse_string("AAB 123 321").dump()) - - prints:: - - ['AAB', '123', '321'] - - LABEL: 'AAB' - - ['AAB', '123', '321'] - """ - if not args: - args = [-1] - for k, v in kwargs.items(): - if k == "default": - args = (args[0], v) - else: - raise TypeError(f"pop() got an unexpected keyword argument {k!r}") - if isinstance(args[0], int) or len(args) == 1 or args[0] in self: - index = args[0] - ret = self[index] - del self[index] - return ret - else: - defaultvalue = args[1] - return defaultvalue - - def get(self, key, default_value=None): - """ - Returns named result matching the given key, or if there is no - such name, then returns the given ``default_value`` or ``None`` if no - ``default_value`` is specified. - - Similar to ``dict.get()``. - - Example:: - - integer = Word(nums) - date_str = integer("year") + '/' + integer("month") + '/' + integer("day") - - result = date_str.parse_string("1999/12/31") - print(result.get("year")) # -> '1999' - print(result.get("hour", "not specified")) # -> 'not specified' - print(result.get("hour")) # -> None - """ - if key in self: - return self[key] - else: - return default_value - - def insert(self, index, ins_string): - """ - Inserts new element at location index in the list of parsed tokens. - - Similar to ``list.insert()``. - - Example:: - - numlist = Word(nums)[...] - print(numlist.parse_string("0 123 321")) # -> ['0', '123', '321'] - - # use a parse action to insert the parse location in the front of the parsed results - def insert_locn(locn, tokens): - tokens.insert(0, locn) - numlist.add_parse_action(insert_locn) - print(numlist.parse_string("0 123 321")) # -> [0, '0', '123', '321'] - """ - self._toklist.insert(index, ins_string) - # fixup indices in token dictionary - for name, occurrences in self._tokdict.items(): - for k, (value, position) in enumerate(occurrences): - occurrences[k] = _ParseResultsWithOffset( - value, position + (position > index) - ) - - def append(self, item): - """ - Add single element to end of ``ParseResults`` list of elements. - - Example:: - - numlist = Word(nums)[...] - print(numlist.parse_string("0 123 321")) # -> ['0', '123', '321'] - - # use a parse action to compute the sum of the parsed integers, and add it to the end - def append_sum(tokens): - tokens.append(sum(map(int, tokens))) - numlist.add_parse_action(append_sum) - print(numlist.parse_string("0 123 321")) # -> ['0', '123', '321', 444] - """ - self._toklist.append(item) - - def extend(self, itemseq): - """ - Add sequence of elements to end of ``ParseResults`` list of elements. - - Example:: - - patt = Word(alphas)[1, ...] - - # use a parse action to append the reverse of the matched strings, to make a palindrome - def make_palindrome(tokens): - tokens.extend(reversed([t[::-1] for t in tokens])) - return ''.join(tokens) - patt.add_parse_action(make_palindrome) - print(patt.parse_string("lskdj sdlkjf lksd")) # -> 'lskdjsdlkjflksddsklfjkldsjdksl' - """ - if isinstance(itemseq, ParseResults): - self.__iadd__(itemseq) - else: - self._toklist.extend(itemseq) - - def clear(self): - """ - Clear all elements and results names. - """ - del self._toklist[:] - self._tokdict.clear() - - def __getattr__(self, name): - try: - return self[name] - except KeyError: - if name.startswith("__"): - raise AttributeError(name) - return "" - - def __add__(self, other: "ParseResults") -> "ParseResults": - ret = self.copy() - ret += other - return ret - - def __iadd__(self, other: "ParseResults") -> "ParseResults": - if not other: - return self - - if other._tokdict: - offset = len(self._toklist) - addoffset = lambda a: offset if a < 0 else a + offset - otheritems = other._tokdict.items() - otherdictitems = [ - (k, _ParseResultsWithOffset(v[0], addoffset(v[1]))) - for k, vlist in otheritems - for v in vlist - ] - for k, v in otherdictitems: - self[k] = v - if isinstance(v[0], ParseResults): - v[0]._parent = self - - self._toklist += other._toklist - self._all_names |= other._all_names - return self - - def __radd__(self, other) -> "ParseResults": - if isinstance(other, int) and other == 0: - # useful for merging many ParseResults using sum() builtin - return self.copy() - else: - # this may raise a TypeError - so be it - return other + self - - def __repr__(self) -> str: - return f"{type(self).__name__}({self._toklist!r}, {self.as_dict()})" - - def __str__(self) -> str: - return ( - "[" - + ", ".join( - [ - str(i) if isinstance(i, ParseResults) else repr(i) - for i in self._toklist - ] - ) - + "]" - ) - - def _asStringList(self, sep=""): - out = [] - for item in self._toklist: - if out and sep: - out.append(sep) - if isinstance(item, ParseResults): - out += item._asStringList() - else: - out.append(str(item)) - return out - - def as_list(self) -> list: - """ - Returns the parse results as a nested list of matching tokens, all converted to strings. - - Example:: - - patt = Word(alphas)[1, ...] - result = patt.parse_string("sldkj lsdkj sldkj") - # even though the result prints in string-like form, it is actually a pyparsing ParseResults - print(type(result), result) # -> ['sldkj', 'lsdkj', 'sldkj'] - - # Use as_list() to create an actual list - result_list = result.as_list() - print(type(result_list), result_list) # -> ['sldkj', 'lsdkj', 'sldkj'] - """ - return [ - res.as_list() if isinstance(res, ParseResults) else res - for res in self._toklist - ] - - def as_dict(self) -> dict: - """ - Returns the named parse results as a nested dictionary. - - Example:: - - integer = Word(nums) - date_str = integer("year") + '/' + integer("month") + '/' + integer("day") - - result = date_str.parse_string('12/31/1999') - print(type(result), repr(result)) # -> (['12', '/', '31', '/', '1999'], {'day': [('1999', 4)], 'year': [('12', 0)], 'month': [('31', 2)]}) - - result_dict = result.as_dict() - print(type(result_dict), repr(result_dict)) # -> {'day': '1999', 'year': '12', 'month': '31'} - - # even though a ParseResults supports dict-like access, sometime you just need to have a dict - import json - print(json.dumps(result)) # -> Exception: TypeError: ... is not JSON serializable - print(json.dumps(result.as_dict())) # -> {"month": "31", "day": "1999", "year": "12"} - """ - - def to_item(obj): - if isinstance(obj, ParseResults): - return obj.as_dict() if obj.haskeys() else [to_item(v) for v in obj] - else: - return obj - - return dict((k, to_item(v)) for k, v in self.items()) - - def copy(self) -> "ParseResults": - """ - Returns a new shallow copy of a :class:`ParseResults` object. `ParseResults` - items contained within the source are shared with the copy. Use - :class:`ParseResults.deepcopy()` to create a copy with its own separate - content values. - """ - ret = ParseResults(self._toklist) - ret._tokdict = self._tokdict.copy() - ret._parent = self._parent - ret._all_names |= self._all_names - ret._name = self._name - return ret - - def deepcopy(self) -> "ParseResults": - """ - Returns a new deep copy of a :class:`ParseResults` object. - """ - ret = self.copy() - # replace values with copies if they are of known mutable types - for i, obj in enumerate(self._toklist): - if isinstance(obj, ParseResults): - self._toklist[i] = obj.deepcopy() - elif isinstance(obj, (str, bytes)): - pass - elif isinstance(obj, MutableMapping): - self._toklist[i] = dest = type(obj)() - for k, v in obj.items(): - dest[k] = v.deepcopy() if isinstance(v, ParseResults) else v - elif isinstance(obj, Container): - self._toklist[i] = type(obj)( - v.deepcopy() if isinstance(v, ParseResults) else v for v in obj - ) - return ret - - def get_name(self): - r""" - Returns the results name for this token expression. Useful when several - different expressions might match at a particular location. - - Example:: - - integer = Word(nums) - ssn_expr = Regex(r"\d\d\d-\d\d-\d\d\d\d") - house_number_expr = Suppress('#') + Word(nums, alphanums) - user_data = (Group(house_number_expr)("house_number") - | Group(ssn_expr)("ssn") - | Group(integer)("age")) - user_info = user_data[1, ...] - - result = user_info.parse_string("22 111-22-3333 #221B") - for item in result: - print(item.get_name(), ':', item[0]) - - prints:: - - age : 22 - ssn : 111-22-3333 - house_number : 221B - """ - if self._name: - return self._name - elif self._parent: - par: "ParseResults" = self._parent - parent_tokdict_items = par._tokdict.items() - return next( - ( - k - for k, vlist in parent_tokdict_items - for v, loc in vlist - if v is self - ), - None, - ) - elif ( - len(self) == 1 - and len(self._tokdict) == 1 - and next(iter(self._tokdict.values()))[0][1] in (0, -1) - ): - return next(iter(self._tokdict.keys())) - else: - return None - - def dump(self, indent="", full=True, include_list=True, _depth=0) -> str: - """ - Diagnostic method for listing out the contents of - a :class:`ParseResults`. Accepts an optional ``indent`` argument so - that this string can be embedded in a nested display of other data. - - Example:: - - integer = Word(nums) - date_str = integer("year") + '/' + integer("month") + '/' + integer("day") - - result = date_str.parse_string('1999/12/31') - print(result.dump()) - - prints:: - - ['1999', '/', '12', '/', '31'] - - day: '31' - - month: '12' - - year: '1999' - """ - out = [] - NL = "\n" - out.append(indent + str(self.as_list()) if include_list else "") - - if full: - if self.haskeys(): - items = sorted((str(k), v) for k, v in self.items()) - for k, v in items: - if out: - out.append(NL) - out.append(f"{indent}{(' ' * _depth)}- {k}: ") - if isinstance(v, ParseResults): - if v: - out.append( - v.dump( - indent=indent, - full=full, - include_list=include_list, - _depth=_depth + 1, - ) - ) - else: - out.append(str(v)) - else: - out.append(repr(v)) - if any(isinstance(vv, ParseResults) for vv in self): - v = self - for i, vv in enumerate(v): - if isinstance(vv, ParseResults): - out.append( - "\n{}{}[{}]:\n{}{}{}".format( - indent, - (" " * (_depth)), - i, - indent, - (" " * (_depth + 1)), - vv.dump( - indent=indent, - full=full, - include_list=include_list, - _depth=_depth + 1, - ), - ) - ) - else: - out.append( - "\n%s%s[%d]:\n%s%s%s" - % ( - indent, - (" " * (_depth)), - i, - indent, - (" " * (_depth + 1)), - str(vv), - ) - ) - - return "".join(out) - - def pprint(self, *args, **kwargs): - """ - Pretty-printer for parsed results as a list, using the - `pprint `_ module. - Accepts additional positional or keyword args as defined for - `pprint.pprint `_ . - - Example:: - - ident = Word(alphas, alphanums) - num = Word(nums) - func = Forward() - term = ident | num | Group('(' + func + ')') - func <<= ident + Group(Optional(DelimitedList(term))) - result = func.parse_string("fna a,b,(fnb c,d,200),100") - result.pprint(width=40) - - prints:: - - ['fna', - ['a', - 'b', - ['(', 'fnb', ['c', 'd', '200'], ')'], - '100']] - """ - pprint.pprint(self.as_list(), *args, **kwargs) - - # add support for pickle protocol - def __getstate__(self): - return ( - self._toklist, - ( - self._tokdict.copy(), - None, - self._all_names, - self._name, - ), - ) - - def __setstate__(self, state): - self._toklist, (self._tokdict, par, inAccumNames, self._name) = state - self._all_names = set(inAccumNames) - self._parent = None - - def __getnewargs__(self): - return self._toklist, self._name - - def __dir__(self): - return dir(type(self)) + list(self.keys()) - - @classmethod - def from_dict(cls, other, name=None) -> "ParseResults": - """ - Helper classmethod to construct a ``ParseResults`` from a ``dict``, preserving the - name-value relations as results names. If an optional ``name`` argument is - given, a nested ``ParseResults`` will be returned. - """ - - def is_iterable(obj): - try: - iter(obj) - except Exception: - return False - # str's are iterable, but in pyparsing, we don't want to iterate over them - else: - return not isinstance(obj, str_type) - - ret = cls([]) - for k, v in other.items(): - if isinstance(v, Mapping): - ret += cls.from_dict(v, name=k) - else: - ret += cls([v], name=k, asList=is_iterable(v)) - if name is not None: - ret = cls([ret], name=name) - return ret - - asList = as_list - """Deprecated - use :class:`as_list`""" - asDict = as_dict - """Deprecated - use :class:`as_dict`""" - getName = get_name - """Deprecated - use :class:`get_name`""" - - -MutableMapping.register(ParseResults) -MutableSequence.register(ParseResults) diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/pyparsing/testing.py b/venv/lib/python3.11/site-packages/pip/_vendor/pyparsing/testing.py deleted file mode 100644 index 6a254c1..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/pyparsing/testing.py +++ /dev/null @@ -1,331 +0,0 @@ -# testing.py - -from contextlib import contextmanager -import typing - -from .core import ( - ParserElement, - ParseException, - Keyword, - __diag__, - __compat__, -) - - -class pyparsing_test: - """ - namespace class for classes useful in writing unit tests - """ - - class reset_pyparsing_context: - """ - Context manager to be used when writing unit tests that modify pyparsing config values: - - packrat parsing - - bounded recursion parsing - - default whitespace characters. - - default keyword characters - - literal string auto-conversion class - - __diag__ settings - - Example:: - - with reset_pyparsing_context(): - # test that literals used to construct a grammar are automatically suppressed - ParserElement.inlineLiteralsUsing(Suppress) - - term = Word(alphas) | Word(nums) - group = Group('(' + term[...] + ')') - - # assert that the '()' characters are not included in the parsed tokens - self.assertParseAndCheckList(group, "(abc 123 def)", ['abc', '123', 'def']) - - # after exiting context manager, literals are converted to Literal expressions again - """ - - def __init__(self): - self._save_context = {} - - def save(self): - self._save_context["default_whitespace"] = ParserElement.DEFAULT_WHITE_CHARS - self._save_context["default_keyword_chars"] = Keyword.DEFAULT_KEYWORD_CHARS - - self._save_context[ - "literal_string_class" - ] = ParserElement._literalStringClass - - self._save_context["verbose_stacktrace"] = ParserElement.verbose_stacktrace - - self._save_context["packrat_enabled"] = ParserElement._packratEnabled - if ParserElement._packratEnabled: - self._save_context[ - "packrat_cache_size" - ] = ParserElement.packrat_cache.size - else: - self._save_context["packrat_cache_size"] = None - self._save_context["packrat_parse"] = ParserElement._parse - self._save_context[ - "recursion_enabled" - ] = ParserElement._left_recursion_enabled - - self._save_context["__diag__"] = { - name: getattr(__diag__, name) for name in __diag__._all_names - } - - self._save_context["__compat__"] = { - "collect_all_And_tokens": __compat__.collect_all_And_tokens - } - - return self - - def restore(self): - # reset pyparsing global state - if ( - ParserElement.DEFAULT_WHITE_CHARS - != self._save_context["default_whitespace"] - ): - ParserElement.set_default_whitespace_chars( - self._save_context["default_whitespace"] - ) - - ParserElement.verbose_stacktrace = self._save_context["verbose_stacktrace"] - - Keyword.DEFAULT_KEYWORD_CHARS = self._save_context["default_keyword_chars"] - ParserElement.inlineLiteralsUsing( - self._save_context["literal_string_class"] - ) - - for name, value in self._save_context["__diag__"].items(): - (__diag__.enable if value else __diag__.disable)(name) - - ParserElement._packratEnabled = False - if self._save_context["packrat_enabled"]: - ParserElement.enable_packrat(self._save_context["packrat_cache_size"]) - else: - ParserElement._parse = self._save_context["packrat_parse"] - ParserElement._left_recursion_enabled = self._save_context[ - "recursion_enabled" - ] - - __compat__.collect_all_And_tokens = self._save_context["__compat__"] - - return self - - def copy(self): - ret = type(self)() - ret._save_context.update(self._save_context) - return ret - - def __enter__(self): - return self.save() - - def __exit__(self, *args): - self.restore() - - class TestParseResultsAsserts: - """ - A mixin class to add parse results assertion methods to normal unittest.TestCase classes. - """ - - def assertParseResultsEquals( - self, result, expected_list=None, expected_dict=None, msg=None - ): - """ - Unit test assertion to compare a :class:`ParseResults` object with an optional ``expected_list``, - and compare any defined results names with an optional ``expected_dict``. - """ - if expected_list is not None: - self.assertEqual(expected_list, result.as_list(), msg=msg) - if expected_dict is not None: - self.assertEqual(expected_dict, result.as_dict(), msg=msg) - - def assertParseAndCheckList( - self, expr, test_string, expected_list, msg=None, verbose=True - ): - """ - Convenience wrapper assert to test a parser element and input string, and assert that - the resulting ``ParseResults.asList()`` is equal to the ``expected_list``. - """ - result = expr.parse_string(test_string, parse_all=True) - if verbose: - print(result.dump()) - else: - print(result.as_list()) - self.assertParseResultsEquals(result, expected_list=expected_list, msg=msg) - - def assertParseAndCheckDict( - self, expr, test_string, expected_dict, msg=None, verbose=True - ): - """ - Convenience wrapper assert to test a parser element and input string, and assert that - the resulting ``ParseResults.asDict()`` is equal to the ``expected_dict``. - """ - result = expr.parse_string(test_string, parseAll=True) - if verbose: - print(result.dump()) - else: - print(result.as_list()) - self.assertParseResultsEquals(result, expected_dict=expected_dict, msg=msg) - - def assertRunTestResults( - self, run_tests_report, expected_parse_results=None, msg=None - ): - """ - Unit test assertion to evaluate output of ``ParserElement.runTests()``. If a list of - list-dict tuples is given as the ``expected_parse_results`` argument, then these are zipped - with the report tuples returned by ``runTests`` and evaluated using ``assertParseResultsEquals``. - Finally, asserts that the overall ``runTests()`` success value is ``True``. - - :param run_tests_report: tuple(bool, [tuple(str, ParseResults or Exception)]) returned from runTests - :param expected_parse_results (optional): [tuple(str, list, dict, Exception)] - """ - run_test_success, run_test_results = run_tests_report - - if expected_parse_results is not None: - merged = [ - (*rpt, expected) - for rpt, expected in zip(run_test_results, expected_parse_results) - ] - for test_string, result, expected in merged: - # expected should be a tuple containing a list and/or a dict or an exception, - # and optional failure message string - # an empty tuple will skip any result validation - fail_msg = next( - (exp for exp in expected if isinstance(exp, str)), None - ) - expected_exception = next( - ( - exp - for exp in expected - if isinstance(exp, type) and issubclass(exp, Exception) - ), - None, - ) - if expected_exception is not None: - with self.assertRaises( - expected_exception=expected_exception, msg=fail_msg or msg - ): - if isinstance(result, Exception): - raise result - else: - expected_list = next( - (exp for exp in expected if isinstance(exp, list)), None - ) - expected_dict = next( - (exp for exp in expected if isinstance(exp, dict)), None - ) - if (expected_list, expected_dict) != (None, None): - self.assertParseResultsEquals( - result, - expected_list=expected_list, - expected_dict=expected_dict, - msg=fail_msg or msg, - ) - else: - # warning here maybe? - print(f"no validation for {test_string!r}") - - # do this last, in case some specific test results can be reported instead - self.assertTrue( - run_test_success, msg=msg if msg is not None else "failed runTests" - ) - - @contextmanager - def assertRaisesParseException(self, exc_type=ParseException, msg=None): - with self.assertRaises(exc_type, msg=msg): - yield - - @staticmethod - def with_line_numbers( - s: str, - start_line: typing.Optional[int] = None, - end_line: typing.Optional[int] = None, - expand_tabs: bool = True, - eol_mark: str = "|", - mark_spaces: typing.Optional[str] = None, - mark_control: typing.Optional[str] = None, - ) -> str: - """ - Helpful method for debugging a parser - prints a string with line and column numbers. - (Line and column numbers are 1-based.) - - :param s: tuple(bool, str - string to be printed with line and column numbers - :param start_line: int - (optional) starting line number in s to print (default=1) - :param end_line: int - (optional) ending line number in s to print (default=len(s)) - :param expand_tabs: bool - (optional) expand tabs to spaces, to match the pyparsing default - :param eol_mark: str - (optional) string to mark the end of lines, helps visualize trailing spaces (default="|") - :param mark_spaces: str - (optional) special character to display in place of spaces - :param mark_control: str - (optional) convert non-printing control characters to a placeholding - character; valid values: - - "unicode" - replaces control chars with Unicode symbols, such as "␍" and "␊" - - any single character string - replace control characters with given string - - None (default) - string is displayed as-is - - :return: str - input string with leading line numbers and column number headers - """ - if expand_tabs: - s = s.expandtabs() - if mark_control is not None: - mark_control = typing.cast(str, mark_control) - if mark_control == "unicode": - transtable_map = { - c: u for c, u in zip(range(0, 33), range(0x2400, 0x2433)) - } - transtable_map[127] = 0x2421 - tbl = str.maketrans(transtable_map) - eol_mark = "" - else: - ord_mark_control = ord(mark_control) - tbl = str.maketrans( - {c: ord_mark_control for c in list(range(0, 32)) + [127]} - ) - s = s.translate(tbl) - if mark_spaces is not None and mark_spaces != " ": - if mark_spaces == "unicode": - tbl = str.maketrans({9: 0x2409, 32: 0x2423}) - s = s.translate(tbl) - else: - s = s.replace(" ", mark_spaces) - if start_line is None: - start_line = 1 - if end_line is None: - end_line = len(s) - end_line = min(end_line, len(s)) - start_line = min(max(1, start_line), end_line) - - if mark_control != "unicode": - s_lines = s.splitlines()[start_line - 1 : end_line] - else: - s_lines = [line + "␊" for line in s.split("␊")[start_line - 1 : end_line]] - if not s_lines: - return "" - - lineno_width = len(str(end_line)) - max_line_len = max(len(line) for line in s_lines) - lead = " " * (lineno_width + 1) - if max_line_len >= 99: - header0 = ( - lead - + "".join( - f"{' ' * 99}{(i + 1) % 100}" - for i in range(max(max_line_len // 100, 1)) - ) - + "\n" - ) - else: - header0 = "" - header1 = ( - header0 - + lead - + "".join(f" {(i + 1) % 10}" for i in range(-(-max_line_len // 10))) - + "\n" - ) - header2 = lead + "1234567890" * (-(-max_line_len // 10)) + "\n" - return ( - header1 - + header2 - + "\n".join( - f"{i:{lineno_width}d}:{line}{eol_mark}" - for i, line in enumerate(s_lines, start=start_line) - ) - + "\n" - ) diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/pyparsing/unicode.py b/venv/lib/python3.11/site-packages/pip/_vendor/pyparsing/unicode.py deleted file mode 100644 index ec0b3a4..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/pyparsing/unicode.py +++ /dev/null @@ -1,361 +0,0 @@ -# unicode.py - -import sys -from itertools import filterfalse -from typing import List, Tuple, Union - - -class _lazyclassproperty: - def __init__(self, fn): - self.fn = fn - self.__doc__ = fn.__doc__ - self.__name__ = fn.__name__ - - def __get__(self, obj, cls): - if cls is None: - cls = type(obj) - if not hasattr(cls, "_intern") or any( - cls._intern is getattr(superclass, "_intern", []) - for superclass in cls.__mro__[1:] - ): - cls._intern = {} - attrname = self.fn.__name__ - if attrname not in cls._intern: - cls._intern[attrname] = self.fn(cls) - return cls._intern[attrname] - - -UnicodeRangeList = List[Union[Tuple[int, int], Tuple[int]]] - - -class unicode_set: - """ - A set of Unicode characters, for language-specific strings for - ``alphas``, ``nums``, ``alphanums``, and ``printables``. - A unicode_set is defined by a list of ranges in the Unicode character - set, in a class attribute ``_ranges``. Ranges can be specified using - 2-tuples or a 1-tuple, such as:: - - _ranges = [ - (0x0020, 0x007e), - (0x00a0, 0x00ff), - (0x0100,), - ] - - Ranges are left- and right-inclusive. A 1-tuple of (x,) is treated as (x, x). - - A unicode set can also be defined using multiple inheritance of other unicode sets:: - - class CJK(Chinese, Japanese, Korean): - pass - """ - - _ranges: UnicodeRangeList = [] - - @_lazyclassproperty - def _chars_for_ranges(cls): - ret = [] - for cc in cls.__mro__: - if cc is unicode_set: - break - for rr in getattr(cc, "_ranges", ()): - ret.extend(range(rr[0], rr[-1] + 1)) - return [chr(c) for c in sorted(set(ret))] - - @_lazyclassproperty - def printables(cls): - """all non-whitespace characters in this range""" - return "".join(filterfalse(str.isspace, cls._chars_for_ranges)) - - @_lazyclassproperty - def alphas(cls): - """all alphabetic characters in this range""" - return "".join(filter(str.isalpha, cls._chars_for_ranges)) - - @_lazyclassproperty - def nums(cls): - """all numeric digit characters in this range""" - return "".join(filter(str.isdigit, cls._chars_for_ranges)) - - @_lazyclassproperty - def alphanums(cls): - """all alphanumeric characters in this range""" - return cls.alphas + cls.nums - - @_lazyclassproperty - def identchars(cls): - """all characters in this range that are valid identifier characters, plus underscore '_'""" - return "".join( - sorted( - set( - "".join(filter(str.isidentifier, cls._chars_for_ranges)) - + "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyzªµº" - + "ÀÁÂÃÄÅÆÇÈÉÊËÌÍÎÏÐÑÒÓÔÕÖØÙÚÛÜÝÞßàáâãäåæçèéêëìíîïðñòóôõöøùúûüýþÿ" - + "_" - ) - ) - ) - - @_lazyclassproperty - def identbodychars(cls): - """ - all characters in this range that are valid identifier body characters, - plus the digits 0-9, and · (Unicode MIDDLE DOT) - """ - return "".join( - sorted( - set( - cls.identchars - + "0123456789·" - + "".join( - [c for c in cls._chars_for_ranges if ("_" + c).isidentifier()] - ) - ) - ) - ) - - @_lazyclassproperty - def identifier(cls): - """ - a pyparsing Word expression for an identifier using this range's definitions for - identchars and identbodychars - """ - from pip._vendor.pyparsing import Word - - return Word(cls.identchars, cls.identbodychars) - - -class pyparsing_unicode(unicode_set): - """ - A namespace class for defining common language unicode_sets. - """ - - # fmt: off - - # define ranges in language character sets - _ranges: UnicodeRangeList = [ - (0x0020, sys.maxunicode), - ] - - class BasicMultilingualPlane(unicode_set): - """Unicode set for the Basic Multilingual Plane""" - _ranges: UnicodeRangeList = [ - (0x0020, 0xFFFF), - ] - - class Latin1(unicode_set): - """Unicode set for Latin-1 Unicode Character Range""" - _ranges: UnicodeRangeList = [ - (0x0020, 0x007E), - (0x00A0, 0x00FF), - ] - - class LatinA(unicode_set): - """Unicode set for Latin-A Unicode Character Range""" - _ranges: UnicodeRangeList = [ - (0x0100, 0x017F), - ] - - class LatinB(unicode_set): - """Unicode set for Latin-B Unicode Character Range""" - _ranges: UnicodeRangeList = [ - (0x0180, 0x024F), - ] - - class Greek(unicode_set): - """Unicode set for Greek Unicode Character Ranges""" - _ranges: UnicodeRangeList = [ - (0x0342, 0x0345), - (0x0370, 0x0377), - (0x037A, 0x037F), - (0x0384, 0x038A), - (0x038C,), - (0x038E, 0x03A1), - (0x03A3, 0x03E1), - (0x03F0, 0x03FF), - (0x1D26, 0x1D2A), - (0x1D5E,), - (0x1D60,), - (0x1D66, 0x1D6A), - (0x1F00, 0x1F15), - (0x1F18, 0x1F1D), - (0x1F20, 0x1F45), - (0x1F48, 0x1F4D), - (0x1F50, 0x1F57), - (0x1F59,), - (0x1F5B,), - (0x1F5D,), - (0x1F5F, 0x1F7D), - (0x1F80, 0x1FB4), - (0x1FB6, 0x1FC4), - (0x1FC6, 0x1FD3), - (0x1FD6, 0x1FDB), - (0x1FDD, 0x1FEF), - (0x1FF2, 0x1FF4), - (0x1FF6, 0x1FFE), - (0x2129,), - (0x2719, 0x271A), - (0xAB65,), - (0x10140, 0x1018D), - (0x101A0,), - (0x1D200, 0x1D245), - (0x1F7A1, 0x1F7A7), - ] - - class Cyrillic(unicode_set): - """Unicode set for Cyrillic Unicode Character Range""" - _ranges: UnicodeRangeList = [ - (0x0400, 0x052F), - (0x1C80, 0x1C88), - (0x1D2B,), - (0x1D78,), - (0x2DE0, 0x2DFF), - (0xA640, 0xA672), - (0xA674, 0xA69F), - (0xFE2E, 0xFE2F), - ] - - class Chinese(unicode_set): - """Unicode set for Chinese Unicode Character Range""" - _ranges: UnicodeRangeList = [ - (0x2E80, 0x2E99), - (0x2E9B, 0x2EF3), - (0x31C0, 0x31E3), - (0x3400, 0x4DB5), - (0x4E00, 0x9FEF), - (0xA700, 0xA707), - (0xF900, 0xFA6D), - (0xFA70, 0xFAD9), - (0x16FE2, 0x16FE3), - (0x1F210, 0x1F212), - (0x1F214, 0x1F23B), - (0x1F240, 0x1F248), - (0x20000, 0x2A6D6), - (0x2A700, 0x2B734), - (0x2B740, 0x2B81D), - (0x2B820, 0x2CEA1), - (0x2CEB0, 0x2EBE0), - (0x2F800, 0x2FA1D), - ] - - class Japanese(unicode_set): - """Unicode set for Japanese Unicode Character Range, combining Kanji, Hiragana, and Katakana ranges""" - - class Kanji(unicode_set): - "Unicode set for Kanji Unicode Character Range" - _ranges: UnicodeRangeList = [ - (0x4E00, 0x9FBF), - (0x3000, 0x303F), - ] - - class Hiragana(unicode_set): - """Unicode set for Hiragana Unicode Character Range""" - _ranges: UnicodeRangeList = [ - (0x3041, 0x3096), - (0x3099, 0x30A0), - (0x30FC,), - (0xFF70,), - (0x1B001,), - (0x1B150, 0x1B152), - (0x1F200,), - ] - - class Katakana(unicode_set): - """Unicode set for Katakana Unicode Character Range""" - _ranges: UnicodeRangeList = [ - (0x3099, 0x309C), - (0x30A0, 0x30FF), - (0x31F0, 0x31FF), - (0x32D0, 0x32FE), - (0xFF65, 0xFF9F), - (0x1B000,), - (0x1B164, 0x1B167), - (0x1F201, 0x1F202), - (0x1F213,), - ] - - 漢字 = Kanji - カタカナ = Katakana - ひらがな = Hiragana - - _ranges = ( - Kanji._ranges - + Hiragana._ranges - + Katakana._ranges - ) - - class Hangul(unicode_set): - """Unicode set for Hangul (Korean) Unicode Character Range""" - _ranges: UnicodeRangeList = [ - (0x1100, 0x11FF), - (0x302E, 0x302F), - (0x3131, 0x318E), - (0x3200, 0x321C), - (0x3260, 0x327B), - (0x327E,), - (0xA960, 0xA97C), - (0xAC00, 0xD7A3), - (0xD7B0, 0xD7C6), - (0xD7CB, 0xD7FB), - (0xFFA0, 0xFFBE), - (0xFFC2, 0xFFC7), - (0xFFCA, 0xFFCF), - (0xFFD2, 0xFFD7), - (0xFFDA, 0xFFDC), - ] - - Korean = Hangul - - class CJK(Chinese, Japanese, Hangul): - """Unicode set for combined Chinese, Japanese, and Korean (CJK) Unicode Character Range""" - - class Thai(unicode_set): - """Unicode set for Thai Unicode Character Range""" - _ranges: UnicodeRangeList = [ - (0x0E01, 0x0E3A), - (0x0E3F, 0x0E5B) - ] - - class Arabic(unicode_set): - """Unicode set for Arabic Unicode Character Range""" - _ranges: UnicodeRangeList = [ - (0x0600, 0x061B), - (0x061E, 0x06FF), - (0x0700, 0x077F), - ] - - class Hebrew(unicode_set): - """Unicode set for Hebrew Unicode Character Range""" - _ranges: UnicodeRangeList = [ - (0x0591, 0x05C7), - (0x05D0, 0x05EA), - (0x05EF, 0x05F4), - (0xFB1D, 0xFB36), - (0xFB38, 0xFB3C), - (0xFB3E,), - (0xFB40, 0xFB41), - (0xFB43, 0xFB44), - (0xFB46, 0xFB4F), - ] - - class Devanagari(unicode_set): - """Unicode set for Devanagari Unicode Character Range""" - _ranges: UnicodeRangeList = [ - (0x0900, 0x097F), - (0xA8E0, 0xA8FF) - ] - - BMP = BasicMultilingualPlane - - # add language identifiers using language Unicode - العربية = Arabic - 中文 = Chinese - кириллица = Cyrillic - Ελληνικά = Greek - עִברִית = Hebrew - 日本語 = Japanese - 한국어 = Korean - ไทย = Thai - देवनागरी = Devanagari - - # fmt: on diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/pyparsing/util.py b/venv/lib/python3.11/site-packages/pip/_vendor/pyparsing/util.py deleted file mode 100644 index d8d3f41..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/pyparsing/util.py +++ /dev/null @@ -1,284 +0,0 @@ -# util.py -import inspect -import warnings -import types -import collections -import itertools -from functools import lru_cache, wraps -from typing import Callable, List, Union, Iterable, TypeVar, cast - -_bslash = chr(92) -C = TypeVar("C", bound=Callable) - - -class __config_flags: - """Internal class for defining compatibility and debugging flags""" - - _all_names: List[str] = [] - _fixed_names: List[str] = [] - _type_desc = "configuration" - - @classmethod - def _set(cls, dname, value): - if dname in cls._fixed_names: - warnings.warn( - f"{cls.__name__}.{dname} {cls._type_desc} is {str(getattr(cls, dname)).upper()}" - f" and cannot be overridden", - stacklevel=3, - ) - return - if dname in cls._all_names: - setattr(cls, dname, value) - else: - raise ValueError(f"no such {cls._type_desc} {dname!r}") - - enable = classmethod(lambda cls, name: cls._set(name, True)) - disable = classmethod(lambda cls, name: cls._set(name, False)) - - -@lru_cache(maxsize=128) -def col(loc: int, strg: str) -> int: - """ - Returns current column within a string, counting newlines as line separators. - The first column is number 1. - - Note: the default parsing behavior is to expand tabs in the input string - before starting the parsing process. See - :class:`ParserElement.parse_string` for more - information on parsing strings containing ```` s, and suggested - methods to maintain a consistent view of the parsed string, the parse - location, and line and column positions within the parsed string. - """ - s = strg - return 1 if 0 < loc < len(s) and s[loc - 1] == "\n" else loc - s.rfind("\n", 0, loc) - - -@lru_cache(maxsize=128) -def lineno(loc: int, strg: str) -> int: - """Returns current line number within a string, counting newlines as line separators. - The first line is number 1. - - Note - the default parsing behavior is to expand tabs in the input string - before starting the parsing process. See :class:`ParserElement.parse_string` - for more information on parsing strings containing ```` s, and - suggested methods to maintain a consistent view of the parsed string, the - parse location, and line and column positions within the parsed string. - """ - return strg.count("\n", 0, loc) + 1 - - -@lru_cache(maxsize=128) -def line(loc: int, strg: str) -> str: - """ - Returns the line of text containing loc within a string, counting newlines as line separators. - """ - last_cr = strg.rfind("\n", 0, loc) - next_cr = strg.find("\n", loc) - return strg[last_cr + 1 : next_cr] if next_cr >= 0 else strg[last_cr + 1 :] - - -class _UnboundedCache: - def __init__(self): - cache = {} - cache_get = cache.get - self.not_in_cache = not_in_cache = object() - - def get(_, key): - return cache_get(key, not_in_cache) - - def set_(_, key, value): - cache[key] = value - - def clear(_): - cache.clear() - - self.size = None - self.get = types.MethodType(get, self) - self.set = types.MethodType(set_, self) - self.clear = types.MethodType(clear, self) - - -class _FifoCache: - def __init__(self, size): - self.not_in_cache = not_in_cache = object() - cache = {} - keyring = [object()] * size - cache_get = cache.get - cache_pop = cache.pop - keyiter = itertools.cycle(range(size)) - - def get(_, key): - return cache_get(key, not_in_cache) - - def set_(_, key, value): - cache[key] = value - i = next(keyiter) - cache_pop(keyring[i], None) - keyring[i] = key - - def clear(_): - cache.clear() - keyring[:] = [object()] * size - - self.size = size - self.get = types.MethodType(get, self) - self.set = types.MethodType(set_, self) - self.clear = types.MethodType(clear, self) - - -class LRUMemo: - """ - A memoizing mapping that retains `capacity` deleted items - - The memo tracks retained items by their access order; once `capacity` items - are retained, the least recently used item is discarded. - """ - - def __init__(self, capacity): - self._capacity = capacity - self._active = {} - self._memory = collections.OrderedDict() - - def __getitem__(self, key): - try: - return self._active[key] - except KeyError: - self._memory.move_to_end(key) - return self._memory[key] - - def __setitem__(self, key, value): - self._memory.pop(key, None) - self._active[key] = value - - def __delitem__(self, key): - try: - value = self._active.pop(key) - except KeyError: - pass - else: - while len(self._memory) >= self._capacity: - self._memory.popitem(last=False) - self._memory[key] = value - - def clear(self): - self._active.clear() - self._memory.clear() - - -class UnboundedMemo(dict): - """ - A memoizing mapping that retains all deleted items - """ - - def __delitem__(self, key): - pass - - -def _escape_regex_range_chars(s: str) -> str: - # escape these chars: ^-[] - for c in r"\^-[]": - s = s.replace(c, _bslash + c) - s = s.replace("\n", r"\n") - s = s.replace("\t", r"\t") - return str(s) - - -def _collapse_string_to_ranges( - s: Union[str, Iterable[str]], re_escape: bool = True -) -> str: - def is_consecutive(c): - c_int = ord(c) - is_consecutive.prev, prev = c_int, is_consecutive.prev - if c_int - prev > 1: - is_consecutive.value = next(is_consecutive.counter) - return is_consecutive.value - - is_consecutive.prev = 0 # type: ignore [attr-defined] - is_consecutive.counter = itertools.count() # type: ignore [attr-defined] - is_consecutive.value = -1 # type: ignore [attr-defined] - - def escape_re_range_char(c): - return "\\" + c if c in r"\^-][" else c - - def no_escape_re_range_char(c): - return c - - if not re_escape: - escape_re_range_char = no_escape_re_range_char - - ret = [] - s = "".join(sorted(set(s))) - if len(s) > 3: - for _, chars in itertools.groupby(s, key=is_consecutive): - first = last = next(chars) - last = collections.deque( - itertools.chain(iter([last]), chars), maxlen=1 - ).pop() - if first == last: - ret.append(escape_re_range_char(first)) - else: - sep = "" if ord(last) == ord(first) + 1 else "-" - ret.append( - f"{escape_re_range_char(first)}{sep}{escape_re_range_char(last)}" - ) - else: - ret = [escape_re_range_char(c) for c in s] - - return "".join(ret) - - -def _flatten(ll: list) -> list: - ret = [] - for i in ll: - if isinstance(i, list): - ret.extend(_flatten(i)) - else: - ret.append(i) - return ret - - -def _make_synonym_function(compat_name: str, fn: C) -> C: - # In a future version, uncomment the code in the internal _inner() functions - # to begin emitting DeprecationWarnings. - - # Unwrap staticmethod/classmethod - fn = getattr(fn, "__func__", fn) - - # (Presence of 'self' arg in signature is used by explain_exception() methods, so we take - # some extra steps to add it if present in decorated function.) - if "self" == list(inspect.signature(fn).parameters)[0]: - - @wraps(fn) - def _inner(self, *args, **kwargs): - # warnings.warn( - # f"Deprecated - use {fn.__name__}", DeprecationWarning, stacklevel=3 - # ) - return fn(self, *args, **kwargs) - - else: - - @wraps(fn) - def _inner(*args, **kwargs): - # warnings.warn( - # f"Deprecated - use {fn.__name__}", DeprecationWarning, stacklevel=3 - # ) - return fn(*args, **kwargs) - - _inner.__doc__ = f"""Deprecated - use :class:`{fn.__name__}`""" - _inner.__name__ = compat_name - _inner.__annotations__ = fn.__annotations__ - if isinstance(fn, types.FunctionType): - _inner.__kwdefaults__ = fn.__kwdefaults__ - elif isinstance(fn, type) and hasattr(fn, "__init__"): - _inner.__kwdefaults__ = fn.__init__.__kwdefaults__ - else: - _inner.__kwdefaults__ = None - _inner.__qualname__ = fn.__qualname__ - return cast(C, _inner) - - -def replaced_by_pep8(fn: C) -> Callable[[Callable], C]: - """ - Decorator for pre-PEP8 compatibility synonyms, to link them to the new function. - """ - return lambda other: _make_synonym_function(other.__name__, fn) diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/__init__.py b/venv/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/__init__.py deleted file mode 100644 index ddfcf7f..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/__init__.py +++ /dev/null @@ -1,23 +0,0 @@ -"""Wrappers to call pyproject.toml-based build backend hooks. -""" - -from ._impl import ( - BackendInvalid, - BackendUnavailable, - BuildBackendHookCaller, - HookMissing, - UnsupportedOperation, - default_subprocess_runner, - quiet_subprocess_runner, -) - -__version__ = '1.0.0' -__all__ = [ - 'BackendUnavailable', - 'BackendInvalid', - 'HookMissing', - 'UnsupportedOperation', - 'default_subprocess_runner', - 'quiet_subprocess_runner', - 'BuildBackendHookCaller', -] diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/__pycache__/__init__.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/__pycache__/__init__.cpython-311.pyc deleted file mode 100644 index f6d8ff0..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/__pycache__/__init__.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/__pycache__/_compat.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/__pycache__/_compat.cpython-311.pyc deleted file mode 100644 index c0341d9..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/__pycache__/_compat.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/__pycache__/_impl.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/__pycache__/_impl.cpython-311.pyc deleted file mode 100644 index 8d27239..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/__pycache__/_impl.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_compat.py b/venv/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_compat.py deleted file mode 100644 index 95e509c..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_compat.py +++ /dev/null @@ -1,8 +0,0 @@ -__all__ = ("tomllib",) - -import sys - -if sys.version_info >= (3, 11): - import tomllib -else: - from pip._vendor import tomli as tomllib diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_impl.py b/venv/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_impl.py deleted file mode 100644 index 37b0e65..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_impl.py +++ /dev/null @@ -1,330 +0,0 @@ -import json -import os -import sys -import tempfile -from contextlib import contextmanager -from os.path import abspath -from os.path import join as pjoin -from subprocess import STDOUT, check_call, check_output - -from ._in_process import _in_proc_script_path - - -def write_json(obj, path, **kwargs): - with open(path, 'w', encoding='utf-8') as f: - json.dump(obj, f, **kwargs) - - -def read_json(path): - with open(path, encoding='utf-8') as f: - return json.load(f) - - -class BackendUnavailable(Exception): - """Will be raised if the backend cannot be imported in the hook process.""" - def __init__(self, traceback): - self.traceback = traceback - - -class BackendInvalid(Exception): - """Will be raised if the backend is invalid.""" - def __init__(self, backend_name, backend_path, message): - super().__init__(message) - self.backend_name = backend_name - self.backend_path = backend_path - - -class HookMissing(Exception): - """Will be raised on missing hooks (if a fallback can't be used).""" - def __init__(self, hook_name): - super().__init__(hook_name) - self.hook_name = hook_name - - -class UnsupportedOperation(Exception): - """May be raised by build_sdist if the backend indicates that it can't.""" - def __init__(self, traceback): - self.traceback = traceback - - -def default_subprocess_runner(cmd, cwd=None, extra_environ=None): - """The default method of calling the wrapper subprocess. - - This uses :func:`subprocess.check_call` under the hood. - """ - env = os.environ.copy() - if extra_environ: - env.update(extra_environ) - - check_call(cmd, cwd=cwd, env=env) - - -def quiet_subprocess_runner(cmd, cwd=None, extra_environ=None): - """Call the subprocess while suppressing output. - - This uses :func:`subprocess.check_output` under the hood. - """ - env = os.environ.copy() - if extra_environ: - env.update(extra_environ) - - check_output(cmd, cwd=cwd, env=env, stderr=STDOUT) - - -def norm_and_check(source_tree, requested): - """Normalise and check a backend path. - - Ensure that the requested backend path is specified as a relative path, - and resolves to a location under the given source tree. - - Return an absolute version of the requested path. - """ - if os.path.isabs(requested): - raise ValueError("paths must be relative") - - abs_source = os.path.abspath(source_tree) - abs_requested = os.path.normpath(os.path.join(abs_source, requested)) - # We have to use commonprefix for Python 2.7 compatibility. So we - # normalise case to avoid problems because commonprefix is a character - # based comparison :-( - norm_source = os.path.normcase(abs_source) - norm_requested = os.path.normcase(abs_requested) - if os.path.commonprefix([norm_source, norm_requested]) != norm_source: - raise ValueError("paths must be inside source tree") - - return abs_requested - - -class BuildBackendHookCaller: - """A wrapper to call the build backend hooks for a source directory. - """ - - def __init__( - self, - source_dir, - build_backend, - backend_path=None, - runner=None, - python_executable=None, - ): - """ - :param source_dir: The source directory to invoke the build backend for - :param build_backend: The build backend spec - :param backend_path: Additional path entries for the build backend spec - :param runner: The :ref:`subprocess runner ` to use - :param python_executable: - The Python executable used to invoke the build backend - """ - if runner is None: - runner = default_subprocess_runner - - self.source_dir = abspath(source_dir) - self.build_backend = build_backend - if backend_path: - backend_path = [ - norm_and_check(self.source_dir, p) for p in backend_path - ] - self.backend_path = backend_path - self._subprocess_runner = runner - if not python_executable: - python_executable = sys.executable - self.python_executable = python_executable - - @contextmanager - def subprocess_runner(self, runner): - """A context manager for temporarily overriding the default - :ref:`subprocess runner `. - - .. code-block:: python - - hook_caller = BuildBackendHookCaller(...) - with hook_caller.subprocess_runner(quiet_subprocess_runner): - ... - """ - prev = self._subprocess_runner - self._subprocess_runner = runner - try: - yield - finally: - self._subprocess_runner = prev - - def _supported_features(self): - """Return the list of optional features supported by the backend.""" - return self._call_hook('_supported_features', {}) - - def get_requires_for_build_wheel(self, config_settings=None): - """Get additional dependencies required for building a wheel. - - :returns: A list of :pep:`dependency specifiers <508>`. - :rtype: list[str] - - .. admonition:: Fallback - - If the build backend does not defined a hook with this name, an - empty list will be returned. - """ - return self._call_hook('get_requires_for_build_wheel', { - 'config_settings': config_settings - }) - - def prepare_metadata_for_build_wheel( - self, metadata_directory, config_settings=None, - _allow_fallback=True): - """Prepare a ``*.dist-info`` folder with metadata for this project. - - :returns: Name of the newly created subfolder within - ``metadata_directory``, containing the metadata. - :rtype: str - - .. admonition:: Fallback - - If the build backend does not define a hook with this name and - ``_allow_fallback`` is truthy, the backend will be asked to build a - wheel via the ``build_wheel`` hook and the dist-info extracted from - that will be returned. - """ - return self._call_hook('prepare_metadata_for_build_wheel', { - 'metadata_directory': abspath(metadata_directory), - 'config_settings': config_settings, - '_allow_fallback': _allow_fallback, - }) - - def build_wheel( - self, wheel_directory, config_settings=None, - metadata_directory=None): - """Build a wheel from this project. - - :returns: - The name of the newly created wheel within ``wheel_directory``. - - .. admonition:: Interaction with fallback - - If the ``build_wheel`` hook was called in the fallback for - :meth:`prepare_metadata_for_build_wheel`, the build backend would - not be invoked. Instead, the previously built wheel will be copied - to ``wheel_directory`` and the name of that file will be returned. - """ - if metadata_directory is not None: - metadata_directory = abspath(metadata_directory) - return self._call_hook('build_wheel', { - 'wheel_directory': abspath(wheel_directory), - 'config_settings': config_settings, - 'metadata_directory': metadata_directory, - }) - - def get_requires_for_build_editable(self, config_settings=None): - """Get additional dependencies required for building an editable wheel. - - :returns: A list of :pep:`dependency specifiers <508>`. - :rtype: list[str] - - .. admonition:: Fallback - - If the build backend does not defined a hook with this name, an - empty list will be returned. - """ - return self._call_hook('get_requires_for_build_editable', { - 'config_settings': config_settings - }) - - def prepare_metadata_for_build_editable( - self, metadata_directory, config_settings=None, - _allow_fallback=True): - """Prepare a ``*.dist-info`` folder with metadata for this project. - - :returns: Name of the newly created subfolder within - ``metadata_directory``, containing the metadata. - :rtype: str - - .. admonition:: Fallback - - If the build backend does not define a hook with this name and - ``_allow_fallback`` is truthy, the backend will be asked to build a - wheel via the ``build_editable`` hook and the dist-info - extracted from that will be returned. - """ - return self._call_hook('prepare_metadata_for_build_editable', { - 'metadata_directory': abspath(metadata_directory), - 'config_settings': config_settings, - '_allow_fallback': _allow_fallback, - }) - - def build_editable( - self, wheel_directory, config_settings=None, - metadata_directory=None): - """Build an editable wheel from this project. - - :returns: - The name of the newly created wheel within ``wheel_directory``. - - .. admonition:: Interaction with fallback - - If the ``build_editable`` hook was called in the fallback for - :meth:`prepare_metadata_for_build_editable`, the build backend - would not be invoked. Instead, the previously built wheel will be - copied to ``wheel_directory`` and the name of that file will be - returned. - """ - if metadata_directory is not None: - metadata_directory = abspath(metadata_directory) - return self._call_hook('build_editable', { - 'wheel_directory': abspath(wheel_directory), - 'config_settings': config_settings, - 'metadata_directory': metadata_directory, - }) - - def get_requires_for_build_sdist(self, config_settings=None): - """Get additional dependencies required for building an sdist. - - :returns: A list of :pep:`dependency specifiers <508>`. - :rtype: list[str] - """ - return self._call_hook('get_requires_for_build_sdist', { - 'config_settings': config_settings - }) - - def build_sdist(self, sdist_directory, config_settings=None): - """Build an sdist from this project. - - :returns: - The name of the newly created sdist within ``wheel_directory``. - """ - return self._call_hook('build_sdist', { - 'sdist_directory': abspath(sdist_directory), - 'config_settings': config_settings, - }) - - def _call_hook(self, hook_name, kwargs): - extra_environ = {'PEP517_BUILD_BACKEND': self.build_backend} - - if self.backend_path: - backend_path = os.pathsep.join(self.backend_path) - extra_environ['PEP517_BACKEND_PATH'] = backend_path - - with tempfile.TemporaryDirectory() as td: - hook_input = {'kwargs': kwargs} - write_json(hook_input, pjoin(td, 'input.json'), indent=2) - - # Run the hook in a subprocess - with _in_proc_script_path() as script: - python = self.python_executable - self._subprocess_runner( - [python, abspath(str(script)), hook_name, td], - cwd=self.source_dir, - extra_environ=extra_environ - ) - - data = read_json(pjoin(td, 'output.json')) - if data.get('unsupported'): - raise UnsupportedOperation(data.get('traceback', '')) - if data.get('no_backend'): - raise BackendUnavailable(data.get('traceback', '')) - if data.get('backend_invalid'): - raise BackendInvalid( - backend_name=self.build_backend, - backend_path=self.backend_path, - message=data.get('backend_error', '') - ) - if data.get('hook_missing'): - raise HookMissing(data.get('missing_hook_name') or hook_name) - return data['return_val'] diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/__init__.py b/venv/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/__init__.py deleted file mode 100644 index 917fa06..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/__init__.py +++ /dev/null @@ -1,18 +0,0 @@ -"""This is a subpackage because the directory is on sys.path for _in_process.py - -The subpackage should stay as empty as possible to avoid shadowing modules that -the backend might import. -""" - -import importlib.resources as resources - -try: - resources.files -except AttributeError: - # Python 3.8 compatibility - def _in_proc_script_path(): - return resources.path(__package__, '_in_process.py') -else: - def _in_proc_script_path(): - return resources.as_file( - resources.files(__package__).joinpath('_in_process.py')) diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/__pycache__/__init__.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/__pycache__/__init__.cpython-311.pyc deleted file mode 100644 index 90bd228..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/__pycache__/__init__.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/__pycache__/_in_process.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/__pycache__/_in_process.cpython-311.pyc deleted file mode 100644 index 4952759..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/__pycache__/_in_process.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py b/venv/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py deleted file mode 100644 index ee511ff..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py +++ /dev/null @@ -1,353 +0,0 @@ -"""This is invoked in a subprocess to call the build backend hooks. - -It expects: -- Command line args: hook_name, control_dir -- Environment variables: - PEP517_BUILD_BACKEND=entry.point:spec - PEP517_BACKEND_PATH=paths (separated with os.pathsep) -- control_dir/input.json: - - {"kwargs": {...}} - -Results: -- control_dir/output.json - - {"return_val": ...} -""" -import json -import os -import os.path -import re -import shutil -import sys -import traceback -from glob import glob -from importlib import import_module -from os.path import join as pjoin - -# This file is run as a script, and `import wrappers` is not zip-safe, so we -# include write_json() and read_json() from wrappers.py. - - -def write_json(obj, path, **kwargs): - with open(path, 'w', encoding='utf-8') as f: - json.dump(obj, f, **kwargs) - - -def read_json(path): - with open(path, encoding='utf-8') as f: - return json.load(f) - - -class BackendUnavailable(Exception): - """Raised if we cannot import the backend""" - def __init__(self, traceback): - self.traceback = traceback - - -class BackendInvalid(Exception): - """Raised if the backend is invalid""" - def __init__(self, message): - self.message = message - - -class HookMissing(Exception): - """Raised if a hook is missing and we are not executing the fallback""" - def __init__(self, hook_name=None): - super().__init__(hook_name) - self.hook_name = hook_name - - -def contained_in(filename, directory): - """Test if a file is located within the given directory.""" - filename = os.path.normcase(os.path.abspath(filename)) - directory = os.path.normcase(os.path.abspath(directory)) - return os.path.commonprefix([filename, directory]) == directory - - -def _build_backend(): - """Find and load the build backend""" - # Add in-tree backend directories to the front of sys.path. - backend_path = os.environ.get('PEP517_BACKEND_PATH') - if backend_path: - extra_pathitems = backend_path.split(os.pathsep) - sys.path[:0] = extra_pathitems - - ep = os.environ['PEP517_BUILD_BACKEND'] - mod_path, _, obj_path = ep.partition(':') - try: - obj = import_module(mod_path) - except ImportError: - raise BackendUnavailable(traceback.format_exc()) - - if backend_path: - if not any( - contained_in(obj.__file__, path) - for path in extra_pathitems - ): - raise BackendInvalid("Backend was not loaded from backend-path") - - if obj_path: - for path_part in obj_path.split('.'): - obj = getattr(obj, path_part) - return obj - - -def _supported_features(): - """Return the list of options features supported by the backend. - - Returns a list of strings. - The only possible value is 'build_editable'. - """ - backend = _build_backend() - features = [] - if hasattr(backend, "build_editable"): - features.append("build_editable") - return features - - -def get_requires_for_build_wheel(config_settings): - """Invoke the optional get_requires_for_build_wheel hook - - Returns [] if the hook is not defined. - """ - backend = _build_backend() - try: - hook = backend.get_requires_for_build_wheel - except AttributeError: - return [] - else: - return hook(config_settings) - - -def get_requires_for_build_editable(config_settings): - """Invoke the optional get_requires_for_build_editable hook - - Returns [] if the hook is not defined. - """ - backend = _build_backend() - try: - hook = backend.get_requires_for_build_editable - except AttributeError: - return [] - else: - return hook(config_settings) - - -def prepare_metadata_for_build_wheel( - metadata_directory, config_settings, _allow_fallback): - """Invoke optional prepare_metadata_for_build_wheel - - Implements a fallback by building a wheel if the hook isn't defined, - unless _allow_fallback is False in which case HookMissing is raised. - """ - backend = _build_backend() - try: - hook = backend.prepare_metadata_for_build_wheel - except AttributeError: - if not _allow_fallback: - raise HookMissing() - else: - return hook(metadata_directory, config_settings) - # fallback to build_wheel outside the try block to avoid exception chaining - # which can be confusing to users and is not relevant - whl_basename = backend.build_wheel(metadata_directory, config_settings) - return _get_wheel_metadata_from_wheel(whl_basename, metadata_directory, - config_settings) - - -def prepare_metadata_for_build_editable( - metadata_directory, config_settings, _allow_fallback): - """Invoke optional prepare_metadata_for_build_editable - - Implements a fallback by building an editable wheel if the hook isn't - defined, unless _allow_fallback is False in which case HookMissing is - raised. - """ - backend = _build_backend() - try: - hook = backend.prepare_metadata_for_build_editable - except AttributeError: - if not _allow_fallback: - raise HookMissing() - try: - build_hook = backend.build_editable - except AttributeError: - raise HookMissing(hook_name='build_editable') - else: - whl_basename = build_hook(metadata_directory, config_settings) - return _get_wheel_metadata_from_wheel(whl_basename, - metadata_directory, - config_settings) - else: - return hook(metadata_directory, config_settings) - - -WHEEL_BUILT_MARKER = 'PEP517_ALREADY_BUILT_WHEEL' - - -def _dist_info_files(whl_zip): - """Identify the .dist-info folder inside a wheel ZipFile.""" - res = [] - for path in whl_zip.namelist(): - m = re.match(r'[^/\\]+-[^/\\]+\.dist-info/', path) - if m: - res.append(path) - if res: - return res - raise Exception("No .dist-info folder found in wheel") - - -def _get_wheel_metadata_from_wheel( - whl_basename, metadata_directory, config_settings): - """Extract the metadata from a wheel. - - Fallback for when the build backend does not - define the 'get_wheel_metadata' hook. - """ - from zipfile import ZipFile - with open(os.path.join(metadata_directory, WHEEL_BUILT_MARKER), 'wb'): - pass # Touch marker file - - whl_file = os.path.join(metadata_directory, whl_basename) - with ZipFile(whl_file) as zipf: - dist_info = _dist_info_files(zipf) - zipf.extractall(path=metadata_directory, members=dist_info) - return dist_info[0].split('/')[0] - - -def _find_already_built_wheel(metadata_directory): - """Check for a wheel already built during the get_wheel_metadata hook. - """ - if not metadata_directory: - return None - metadata_parent = os.path.dirname(metadata_directory) - if not os.path.isfile(pjoin(metadata_parent, WHEEL_BUILT_MARKER)): - return None - - whl_files = glob(os.path.join(metadata_parent, '*.whl')) - if not whl_files: - print('Found wheel built marker, but no .whl files') - return None - if len(whl_files) > 1: - print('Found multiple .whl files; unspecified behaviour. ' - 'Will call build_wheel.') - return None - - # Exactly one .whl file - return whl_files[0] - - -def build_wheel(wheel_directory, config_settings, metadata_directory=None): - """Invoke the mandatory build_wheel hook. - - If a wheel was already built in the - prepare_metadata_for_build_wheel fallback, this - will copy it rather than rebuilding the wheel. - """ - prebuilt_whl = _find_already_built_wheel(metadata_directory) - if prebuilt_whl: - shutil.copy2(prebuilt_whl, wheel_directory) - return os.path.basename(prebuilt_whl) - - return _build_backend().build_wheel(wheel_directory, config_settings, - metadata_directory) - - -def build_editable(wheel_directory, config_settings, metadata_directory=None): - """Invoke the optional build_editable hook. - - If a wheel was already built in the - prepare_metadata_for_build_editable fallback, this - will copy it rather than rebuilding the wheel. - """ - backend = _build_backend() - try: - hook = backend.build_editable - except AttributeError: - raise HookMissing() - else: - prebuilt_whl = _find_already_built_wheel(metadata_directory) - if prebuilt_whl: - shutil.copy2(prebuilt_whl, wheel_directory) - return os.path.basename(prebuilt_whl) - - return hook(wheel_directory, config_settings, metadata_directory) - - -def get_requires_for_build_sdist(config_settings): - """Invoke the optional get_requires_for_build_wheel hook - - Returns [] if the hook is not defined. - """ - backend = _build_backend() - try: - hook = backend.get_requires_for_build_sdist - except AttributeError: - return [] - else: - return hook(config_settings) - - -class _DummyException(Exception): - """Nothing should ever raise this exception""" - - -class GotUnsupportedOperation(Exception): - """For internal use when backend raises UnsupportedOperation""" - def __init__(self, traceback): - self.traceback = traceback - - -def build_sdist(sdist_directory, config_settings): - """Invoke the mandatory build_sdist hook.""" - backend = _build_backend() - try: - return backend.build_sdist(sdist_directory, config_settings) - except getattr(backend, 'UnsupportedOperation', _DummyException): - raise GotUnsupportedOperation(traceback.format_exc()) - - -HOOK_NAMES = { - 'get_requires_for_build_wheel', - 'prepare_metadata_for_build_wheel', - 'build_wheel', - 'get_requires_for_build_editable', - 'prepare_metadata_for_build_editable', - 'build_editable', - 'get_requires_for_build_sdist', - 'build_sdist', - '_supported_features', -} - - -def main(): - if len(sys.argv) < 3: - sys.exit("Needs args: hook_name, control_dir") - hook_name = sys.argv[1] - control_dir = sys.argv[2] - if hook_name not in HOOK_NAMES: - sys.exit("Unknown hook: %s" % hook_name) - hook = globals()[hook_name] - - hook_input = read_json(pjoin(control_dir, 'input.json')) - - json_out = {'unsupported': False, 'return_val': None} - try: - json_out['return_val'] = hook(**hook_input['kwargs']) - except BackendUnavailable as e: - json_out['no_backend'] = True - json_out['traceback'] = e.traceback - except BackendInvalid as e: - json_out['backend_invalid'] = True - json_out['backend_error'] = e.message - except GotUnsupportedOperation as e: - json_out['unsupported'] = True - json_out['traceback'] = e.traceback - except HookMissing as e: - json_out['hook_missing'] = True - json_out['missing_hook_name'] = e.hook_name or hook_name - - write_json(json_out, pjoin(control_dir, 'output.json'), indent=2) - - -if __name__ == '__main__': - main() diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/requests/__init__.py b/venv/lib/python3.11/site-packages/pip/_vendor/requests/__init__.py deleted file mode 100644 index 10ff67f..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/requests/__init__.py +++ /dev/null @@ -1,182 +0,0 @@ -# __ -# /__) _ _ _ _ _/ _ -# / ( (- (/ (/ (- _) / _) -# / - -""" -Requests HTTP Library -~~~~~~~~~~~~~~~~~~~~~ - -Requests is an HTTP library, written in Python, for human beings. -Basic GET usage: - - >>> import requests - >>> r = requests.get('https://www.python.org') - >>> r.status_code - 200 - >>> b'Python is a programming language' in r.content - True - -... or POST: - - >>> payload = dict(key1='value1', key2='value2') - >>> r = requests.post('https://httpbin.org/post', data=payload) - >>> print(r.text) - { - ... - "form": { - "key1": "value1", - "key2": "value2" - }, - ... - } - -The other HTTP methods are supported - see `requests.api`. Full documentation -is at . - -:copyright: (c) 2017 by Kenneth Reitz. -:license: Apache 2.0, see LICENSE for more details. -""" - -import warnings - -from pip._vendor import urllib3 - -from .exceptions import RequestsDependencyWarning - -charset_normalizer_version = None - -try: - from pip._vendor.chardet import __version__ as chardet_version -except ImportError: - chardet_version = None - - -def check_compatibility(urllib3_version, chardet_version, charset_normalizer_version): - urllib3_version = urllib3_version.split(".") - assert urllib3_version != ["dev"] # Verify urllib3 isn't installed from git. - - # Sometimes, urllib3 only reports its version as 16.1. - if len(urllib3_version) == 2: - urllib3_version.append("0") - - # Check urllib3 for compatibility. - major, minor, patch = urllib3_version # noqa: F811 - major, minor, patch = int(major), int(minor), int(patch) - # urllib3 >= 1.21.1 - assert major >= 1 - if major == 1: - assert minor >= 21 - - # Check charset_normalizer for compatibility. - if chardet_version: - major, minor, patch = chardet_version.split(".")[:3] - major, minor, patch = int(major), int(minor), int(patch) - # chardet_version >= 3.0.2, < 6.0.0 - assert (3, 0, 2) <= (major, minor, patch) < (6, 0, 0) - elif charset_normalizer_version: - major, minor, patch = charset_normalizer_version.split(".")[:3] - major, minor, patch = int(major), int(minor), int(patch) - # charset_normalizer >= 2.0.0 < 4.0.0 - assert (2, 0, 0) <= (major, minor, patch) < (4, 0, 0) - else: - raise Exception("You need either charset_normalizer or chardet installed") - - -def _check_cryptography(cryptography_version): - # cryptography < 1.3.4 - try: - cryptography_version = list(map(int, cryptography_version.split("."))) - except ValueError: - return - - if cryptography_version < [1, 3, 4]: - warning = "Old version of cryptography ({}) may cause slowdown.".format( - cryptography_version - ) - warnings.warn(warning, RequestsDependencyWarning) - - -# Check imported dependencies for compatibility. -try: - check_compatibility( - urllib3.__version__, chardet_version, charset_normalizer_version - ) -except (AssertionError, ValueError): - warnings.warn( - "urllib3 ({}) or chardet ({})/charset_normalizer ({}) doesn't match a supported " - "version!".format( - urllib3.__version__, chardet_version, charset_normalizer_version - ), - RequestsDependencyWarning, - ) - -# Attempt to enable urllib3's fallback for SNI support -# if the standard library doesn't support SNI or the -# 'ssl' library isn't available. -try: - # Note: This logic prevents upgrading cryptography on Windows, if imported - # as part of pip. - from pip._internal.utils.compat import WINDOWS - if not WINDOWS: - raise ImportError("pip internals: don't import cryptography on Windows") - try: - import ssl - except ImportError: - ssl = None - - if not getattr(ssl, "HAS_SNI", False): - from pip._vendor.urllib3.contrib import pyopenssl - - pyopenssl.inject_into_urllib3() - - # Check cryptography version - from cryptography import __version__ as cryptography_version - - _check_cryptography(cryptography_version) -except ImportError: - pass - -# urllib3's DependencyWarnings should be silenced. -from pip._vendor.urllib3.exceptions import DependencyWarning - -warnings.simplefilter("ignore", DependencyWarning) - -# Set default logging handler to avoid "No handler found" warnings. -import logging -from logging import NullHandler - -from . import packages, utils -from .__version__ import ( - __author__, - __author_email__, - __build__, - __cake__, - __copyright__, - __description__, - __license__, - __title__, - __url__, - __version__, -) -from .api import delete, get, head, options, patch, post, put, request -from .exceptions import ( - ConnectionError, - ConnectTimeout, - FileModeWarning, - HTTPError, - JSONDecodeError, - ReadTimeout, - RequestException, - Timeout, - TooManyRedirects, - URLRequired, -) -from .models import PreparedRequest, Request, Response -from .sessions import Session, session -from .status_codes import codes - -logging.getLogger(__name__).addHandler(NullHandler()) - -# FileModeWarnings go off per the default. -warnings.simplefilter("default", FileModeWarning, append=True) diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/requests/__pycache__/__init__.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/requests/__pycache__/__init__.cpython-311.pyc deleted file mode 100644 index 4a1f170..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/requests/__pycache__/__init__.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/requests/__pycache__/__version__.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/requests/__pycache__/__version__.cpython-311.pyc deleted file mode 100644 index e2c9d65..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/requests/__pycache__/__version__.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/requests/__pycache__/_internal_utils.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/requests/__pycache__/_internal_utils.cpython-311.pyc deleted file mode 100644 index 7a7435e..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/requests/__pycache__/_internal_utils.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/requests/__pycache__/adapters.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/requests/__pycache__/adapters.cpython-311.pyc deleted file mode 100644 index 1137073..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/requests/__pycache__/adapters.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/requests/__pycache__/api.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/requests/__pycache__/api.cpython-311.pyc deleted file mode 100644 index 3058065..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/requests/__pycache__/api.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/requests/__pycache__/auth.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/requests/__pycache__/auth.cpython-311.pyc deleted file mode 100644 index dab031b..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/requests/__pycache__/auth.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/requests/__pycache__/certs.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/requests/__pycache__/certs.cpython-311.pyc deleted file mode 100644 index c2a7994..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/requests/__pycache__/certs.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/requests/__pycache__/compat.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/requests/__pycache__/compat.cpython-311.pyc deleted file mode 100644 index c28913d..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/requests/__pycache__/compat.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/requests/__pycache__/cookies.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/requests/__pycache__/cookies.cpython-311.pyc deleted file mode 100644 index 30dfa96..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/requests/__pycache__/cookies.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/requests/__pycache__/exceptions.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/requests/__pycache__/exceptions.cpython-311.pyc deleted file mode 100644 index 35943e1..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/requests/__pycache__/exceptions.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/requests/__pycache__/help.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/requests/__pycache__/help.cpython-311.pyc deleted file mode 100644 index df9aaee..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/requests/__pycache__/help.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/requests/__pycache__/hooks.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/requests/__pycache__/hooks.cpython-311.pyc deleted file mode 100644 index 3b75aa5..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/requests/__pycache__/hooks.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/requests/__pycache__/models.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/requests/__pycache__/models.cpython-311.pyc deleted file mode 100644 index 712414c..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/requests/__pycache__/models.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/requests/__pycache__/packages.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/requests/__pycache__/packages.cpython-311.pyc deleted file mode 100644 index 3c5fc51..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/requests/__pycache__/packages.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/requests/__pycache__/sessions.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/requests/__pycache__/sessions.cpython-311.pyc deleted file mode 100644 index b360f93..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/requests/__pycache__/sessions.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/requests/__pycache__/status_codes.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/requests/__pycache__/status_codes.cpython-311.pyc deleted file mode 100644 index 4cc8b26..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/requests/__pycache__/status_codes.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/requests/__pycache__/structures.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/requests/__pycache__/structures.cpython-311.pyc deleted file mode 100644 index 8cda797..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/requests/__pycache__/structures.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/requests/__pycache__/utils.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/requests/__pycache__/utils.cpython-311.pyc deleted file mode 100644 index 80e264c..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/requests/__pycache__/utils.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/requests/__version__.py b/venv/lib/python3.11/site-packages/pip/_vendor/requests/__version__.py deleted file mode 100644 index 5063c3f..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/requests/__version__.py +++ /dev/null @@ -1,14 +0,0 @@ -# .-. .-. .-. . . .-. .-. .-. .-. -# |( |- |.| | | |- `-. | `-. -# ' ' `-' `-`.`-' `-' `-' ' `-' - -__title__ = "requests" -__description__ = "Python HTTP for Humans." -__url__ = "https://requests.readthedocs.io" -__version__ = "2.31.0" -__build__ = 0x023100 -__author__ = "Kenneth Reitz" -__author_email__ = "me@kennethreitz.org" -__license__ = "Apache 2.0" -__copyright__ = "Copyright Kenneth Reitz" -__cake__ = "\u2728 \U0001f370 \u2728" diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/requests/_internal_utils.py b/venv/lib/python3.11/site-packages/pip/_vendor/requests/_internal_utils.py deleted file mode 100644 index f2cf635..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/requests/_internal_utils.py +++ /dev/null @@ -1,50 +0,0 @@ -""" -requests._internal_utils -~~~~~~~~~~~~~~ - -Provides utility functions that are consumed internally by Requests -which depend on extremely few external helpers (such as compat) -""" -import re - -from .compat import builtin_str - -_VALID_HEADER_NAME_RE_BYTE = re.compile(rb"^[^:\s][^:\r\n]*$") -_VALID_HEADER_NAME_RE_STR = re.compile(r"^[^:\s][^:\r\n]*$") -_VALID_HEADER_VALUE_RE_BYTE = re.compile(rb"^\S[^\r\n]*$|^$") -_VALID_HEADER_VALUE_RE_STR = re.compile(r"^\S[^\r\n]*$|^$") - -_HEADER_VALIDATORS_STR = (_VALID_HEADER_NAME_RE_STR, _VALID_HEADER_VALUE_RE_STR) -_HEADER_VALIDATORS_BYTE = (_VALID_HEADER_NAME_RE_BYTE, _VALID_HEADER_VALUE_RE_BYTE) -HEADER_VALIDATORS = { - bytes: _HEADER_VALIDATORS_BYTE, - str: _HEADER_VALIDATORS_STR, -} - - -def to_native_string(string, encoding="ascii"): - """Given a string object, regardless of type, returns a representation of - that string in the native string type, encoding and decoding where - necessary. This assumes ASCII unless told otherwise. - """ - if isinstance(string, builtin_str): - out = string - else: - out = string.decode(encoding) - - return out - - -def unicode_is_ascii(u_string): - """Determine if unicode string only contains ASCII characters. - - :param str u_string: unicode string to check. Must be unicode - and not Python 2 `str`. - :rtype: bool - """ - assert isinstance(u_string, str) - try: - u_string.encode("ascii") - return True - except UnicodeEncodeError: - return False diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/requests/adapters.py b/venv/lib/python3.11/site-packages/pip/_vendor/requests/adapters.py deleted file mode 100644 index 10c1767..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/requests/adapters.py +++ /dev/null @@ -1,538 +0,0 @@ -""" -requests.adapters -~~~~~~~~~~~~~~~~~ - -This module contains the transport adapters that Requests uses to define -and maintain connections. -""" - -import os.path -import socket # noqa: F401 - -from pip._vendor.urllib3.exceptions import ClosedPoolError, ConnectTimeoutError -from pip._vendor.urllib3.exceptions import HTTPError as _HTTPError -from pip._vendor.urllib3.exceptions import InvalidHeader as _InvalidHeader -from pip._vendor.urllib3.exceptions import ( - LocationValueError, - MaxRetryError, - NewConnectionError, - ProtocolError, -) -from pip._vendor.urllib3.exceptions import ProxyError as _ProxyError -from pip._vendor.urllib3.exceptions import ReadTimeoutError, ResponseError -from pip._vendor.urllib3.exceptions import SSLError as _SSLError -from pip._vendor.urllib3.poolmanager import PoolManager, proxy_from_url -from pip._vendor.urllib3.util import Timeout as TimeoutSauce -from pip._vendor.urllib3.util import parse_url -from pip._vendor.urllib3.util.retry import Retry - -from .auth import _basic_auth_str -from .compat import basestring, urlparse -from .cookies import extract_cookies_to_jar -from .exceptions import ( - ConnectionError, - ConnectTimeout, - InvalidHeader, - InvalidProxyURL, - InvalidSchema, - InvalidURL, - ProxyError, - ReadTimeout, - RetryError, - SSLError, -) -from .models import Response -from .structures import CaseInsensitiveDict -from .utils import ( - DEFAULT_CA_BUNDLE_PATH, - extract_zipped_paths, - get_auth_from_url, - get_encoding_from_headers, - prepend_scheme_if_needed, - select_proxy, - urldefragauth, -) - -try: - from pip._vendor.urllib3.contrib.socks import SOCKSProxyManager -except ImportError: - - def SOCKSProxyManager(*args, **kwargs): - raise InvalidSchema("Missing dependencies for SOCKS support.") - - -DEFAULT_POOLBLOCK = False -DEFAULT_POOLSIZE = 10 -DEFAULT_RETRIES = 0 -DEFAULT_POOL_TIMEOUT = None - - -class BaseAdapter: - """The Base Transport Adapter""" - - def __init__(self): - super().__init__() - - def send( - self, request, stream=False, timeout=None, verify=True, cert=None, proxies=None - ): - """Sends PreparedRequest object. Returns Response object. - - :param request: The :class:`PreparedRequest ` being sent. - :param stream: (optional) Whether to stream the request content. - :param timeout: (optional) How long to wait for the server to send - data before giving up, as a float, or a :ref:`(connect timeout, - read timeout) ` tuple. - :type timeout: float or tuple - :param verify: (optional) Either a boolean, in which case it controls whether we verify - the server's TLS certificate, or a string, in which case it must be a path - to a CA bundle to use - :param cert: (optional) Any user-provided SSL certificate to be trusted. - :param proxies: (optional) The proxies dictionary to apply to the request. - """ - raise NotImplementedError - - def close(self): - """Cleans up adapter specific items.""" - raise NotImplementedError - - -class HTTPAdapter(BaseAdapter): - """The built-in HTTP Adapter for urllib3. - - Provides a general-case interface for Requests sessions to contact HTTP and - HTTPS urls by implementing the Transport Adapter interface. This class will - usually be created by the :class:`Session ` class under the - covers. - - :param pool_connections: The number of urllib3 connection pools to cache. - :param pool_maxsize: The maximum number of connections to save in the pool. - :param max_retries: The maximum number of retries each connection - should attempt. Note, this applies only to failed DNS lookups, socket - connections and connection timeouts, never to requests where data has - made it to the server. By default, Requests does not retry failed - connections. If you need granular control over the conditions under - which we retry a request, import urllib3's ``Retry`` class and pass - that instead. - :param pool_block: Whether the connection pool should block for connections. - - Usage:: - - >>> import requests - >>> s = requests.Session() - >>> a = requests.adapters.HTTPAdapter(max_retries=3) - >>> s.mount('http://', a) - """ - - __attrs__ = [ - "max_retries", - "config", - "_pool_connections", - "_pool_maxsize", - "_pool_block", - ] - - def __init__( - self, - pool_connections=DEFAULT_POOLSIZE, - pool_maxsize=DEFAULT_POOLSIZE, - max_retries=DEFAULT_RETRIES, - pool_block=DEFAULT_POOLBLOCK, - ): - if max_retries == DEFAULT_RETRIES: - self.max_retries = Retry(0, read=False) - else: - self.max_retries = Retry.from_int(max_retries) - self.config = {} - self.proxy_manager = {} - - super().__init__() - - self._pool_connections = pool_connections - self._pool_maxsize = pool_maxsize - self._pool_block = pool_block - - self.init_poolmanager(pool_connections, pool_maxsize, block=pool_block) - - def __getstate__(self): - return {attr: getattr(self, attr, None) for attr in self.__attrs__} - - def __setstate__(self, state): - # Can't handle by adding 'proxy_manager' to self.__attrs__ because - # self.poolmanager uses a lambda function, which isn't pickleable. - self.proxy_manager = {} - self.config = {} - - for attr, value in state.items(): - setattr(self, attr, value) - - self.init_poolmanager( - self._pool_connections, self._pool_maxsize, block=self._pool_block - ) - - def init_poolmanager( - self, connections, maxsize, block=DEFAULT_POOLBLOCK, **pool_kwargs - ): - """Initializes a urllib3 PoolManager. - - This method should not be called from user code, and is only - exposed for use when subclassing the - :class:`HTTPAdapter `. - - :param connections: The number of urllib3 connection pools to cache. - :param maxsize: The maximum number of connections to save in the pool. - :param block: Block when no free connections are available. - :param pool_kwargs: Extra keyword arguments used to initialize the Pool Manager. - """ - # save these values for pickling - self._pool_connections = connections - self._pool_maxsize = maxsize - self._pool_block = block - - self.poolmanager = PoolManager( - num_pools=connections, - maxsize=maxsize, - block=block, - **pool_kwargs, - ) - - def proxy_manager_for(self, proxy, **proxy_kwargs): - """Return urllib3 ProxyManager for the given proxy. - - This method should not be called from user code, and is only - exposed for use when subclassing the - :class:`HTTPAdapter `. - - :param proxy: The proxy to return a urllib3 ProxyManager for. - :param proxy_kwargs: Extra keyword arguments used to configure the Proxy Manager. - :returns: ProxyManager - :rtype: urllib3.ProxyManager - """ - if proxy in self.proxy_manager: - manager = self.proxy_manager[proxy] - elif proxy.lower().startswith("socks"): - username, password = get_auth_from_url(proxy) - manager = self.proxy_manager[proxy] = SOCKSProxyManager( - proxy, - username=username, - password=password, - num_pools=self._pool_connections, - maxsize=self._pool_maxsize, - block=self._pool_block, - **proxy_kwargs, - ) - else: - proxy_headers = self.proxy_headers(proxy) - manager = self.proxy_manager[proxy] = proxy_from_url( - proxy, - proxy_headers=proxy_headers, - num_pools=self._pool_connections, - maxsize=self._pool_maxsize, - block=self._pool_block, - **proxy_kwargs, - ) - - return manager - - def cert_verify(self, conn, url, verify, cert): - """Verify a SSL certificate. This method should not be called from user - code, and is only exposed for use when subclassing the - :class:`HTTPAdapter `. - - :param conn: The urllib3 connection object associated with the cert. - :param url: The requested URL. - :param verify: Either a boolean, in which case it controls whether we verify - the server's TLS certificate, or a string, in which case it must be a path - to a CA bundle to use - :param cert: The SSL certificate to verify. - """ - if url.lower().startswith("https") and verify: - - cert_loc = None - - # Allow self-specified cert location. - if verify is not True: - cert_loc = verify - - if not cert_loc: - cert_loc = extract_zipped_paths(DEFAULT_CA_BUNDLE_PATH) - - if not cert_loc or not os.path.exists(cert_loc): - raise OSError( - f"Could not find a suitable TLS CA certificate bundle, " - f"invalid path: {cert_loc}" - ) - - conn.cert_reqs = "CERT_REQUIRED" - - if not os.path.isdir(cert_loc): - conn.ca_certs = cert_loc - else: - conn.ca_cert_dir = cert_loc - else: - conn.cert_reqs = "CERT_NONE" - conn.ca_certs = None - conn.ca_cert_dir = None - - if cert: - if not isinstance(cert, basestring): - conn.cert_file = cert[0] - conn.key_file = cert[1] - else: - conn.cert_file = cert - conn.key_file = None - if conn.cert_file and not os.path.exists(conn.cert_file): - raise OSError( - f"Could not find the TLS certificate file, " - f"invalid path: {conn.cert_file}" - ) - if conn.key_file and not os.path.exists(conn.key_file): - raise OSError( - f"Could not find the TLS key file, invalid path: {conn.key_file}" - ) - - def build_response(self, req, resp): - """Builds a :class:`Response ` object from a urllib3 - response. This should not be called from user code, and is only exposed - for use when subclassing the - :class:`HTTPAdapter ` - - :param req: The :class:`PreparedRequest ` used to generate the response. - :param resp: The urllib3 response object. - :rtype: requests.Response - """ - response = Response() - - # Fallback to None if there's no status_code, for whatever reason. - response.status_code = getattr(resp, "status", None) - - # Make headers case-insensitive. - response.headers = CaseInsensitiveDict(getattr(resp, "headers", {})) - - # Set encoding. - response.encoding = get_encoding_from_headers(response.headers) - response.raw = resp - response.reason = response.raw.reason - - if isinstance(req.url, bytes): - response.url = req.url.decode("utf-8") - else: - response.url = req.url - - # Add new cookies from the server. - extract_cookies_to_jar(response.cookies, req, resp) - - # Give the Response some context. - response.request = req - response.connection = self - - return response - - def get_connection(self, url, proxies=None): - """Returns a urllib3 connection for the given URL. This should not be - called from user code, and is only exposed for use when subclassing the - :class:`HTTPAdapter `. - - :param url: The URL to connect to. - :param proxies: (optional) A Requests-style dictionary of proxies used on this request. - :rtype: urllib3.ConnectionPool - """ - proxy = select_proxy(url, proxies) - - if proxy: - proxy = prepend_scheme_if_needed(proxy, "http") - proxy_url = parse_url(proxy) - if not proxy_url.host: - raise InvalidProxyURL( - "Please check proxy URL. It is malformed " - "and could be missing the host." - ) - proxy_manager = self.proxy_manager_for(proxy) - conn = proxy_manager.connection_from_url(url) - else: - # Only scheme should be lower case - parsed = urlparse(url) - url = parsed.geturl() - conn = self.poolmanager.connection_from_url(url) - - return conn - - def close(self): - """Disposes of any internal state. - - Currently, this closes the PoolManager and any active ProxyManager, - which closes any pooled connections. - """ - self.poolmanager.clear() - for proxy in self.proxy_manager.values(): - proxy.clear() - - def request_url(self, request, proxies): - """Obtain the url to use when making the final request. - - If the message is being sent through a HTTP proxy, the full URL has to - be used. Otherwise, we should only use the path portion of the URL. - - This should not be called from user code, and is only exposed for use - when subclassing the - :class:`HTTPAdapter `. - - :param request: The :class:`PreparedRequest ` being sent. - :param proxies: A dictionary of schemes or schemes and hosts to proxy URLs. - :rtype: str - """ - proxy = select_proxy(request.url, proxies) - scheme = urlparse(request.url).scheme - - is_proxied_http_request = proxy and scheme != "https" - using_socks_proxy = False - if proxy: - proxy_scheme = urlparse(proxy).scheme.lower() - using_socks_proxy = proxy_scheme.startswith("socks") - - url = request.path_url - if is_proxied_http_request and not using_socks_proxy: - url = urldefragauth(request.url) - - return url - - def add_headers(self, request, **kwargs): - """Add any headers needed by the connection. As of v2.0 this does - nothing by default, but is left for overriding by users that subclass - the :class:`HTTPAdapter `. - - This should not be called from user code, and is only exposed for use - when subclassing the - :class:`HTTPAdapter `. - - :param request: The :class:`PreparedRequest ` to add headers to. - :param kwargs: The keyword arguments from the call to send(). - """ - pass - - def proxy_headers(self, proxy): - """Returns a dictionary of the headers to add to any request sent - through a proxy. This works with urllib3 magic to ensure that they are - correctly sent to the proxy, rather than in a tunnelled request if - CONNECT is being used. - - This should not be called from user code, and is only exposed for use - when subclassing the - :class:`HTTPAdapter `. - - :param proxy: The url of the proxy being used for this request. - :rtype: dict - """ - headers = {} - username, password = get_auth_from_url(proxy) - - if username: - headers["Proxy-Authorization"] = _basic_auth_str(username, password) - - return headers - - def send( - self, request, stream=False, timeout=None, verify=True, cert=None, proxies=None - ): - """Sends PreparedRequest object. Returns Response object. - - :param request: The :class:`PreparedRequest ` being sent. - :param stream: (optional) Whether to stream the request content. - :param timeout: (optional) How long to wait for the server to send - data before giving up, as a float, or a :ref:`(connect timeout, - read timeout) ` tuple. - :type timeout: float or tuple or urllib3 Timeout object - :param verify: (optional) Either a boolean, in which case it controls whether - we verify the server's TLS certificate, or a string, in which case it - must be a path to a CA bundle to use - :param cert: (optional) Any user-provided SSL certificate to be trusted. - :param proxies: (optional) The proxies dictionary to apply to the request. - :rtype: requests.Response - """ - - try: - conn = self.get_connection(request.url, proxies) - except LocationValueError as e: - raise InvalidURL(e, request=request) - - self.cert_verify(conn, request.url, verify, cert) - url = self.request_url(request, proxies) - self.add_headers( - request, - stream=stream, - timeout=timeout, - verify=verify, - cert=cert, - proxies=proxies, - ) - - chunked = not (request.body is None or "Content-Length" in request.headers) - - if isinstance(timeout, tuple): - try: - connect, read = timeout - timeout = TimeoutSauce(connect=connect, read=read) - except ValueError: - raise ValueError( - f"Invalid timeout {timeout}. Pass a (connect, read) timeout tuple, " - f"or a single float to set both timeouts to the same value." - ) - elif isinstance(timeout, TimeoutSauce): - pass - else: - timeout = TimeoutSauce(connect=timeout, read=timeout) - - try: - resp = conn.urlopen( - method=request.method, - url=url, - body=request.body, - headers=request.headers, - redirect=False, - assert_same_host=False, - preload_content=False, - decode_content=False, - retries=self.max_retries, - timeout=timeout, - chunked=chunked, - ) - - except (ProtocolError, OSError) as err: - raise ConnectionError(err, request=request) - - except MaxRetryError as e: - if isinstance(e.reason, ConnectTimeoutError): - # TODO: Remove this in 3.0.0: see #2811 - if not isinstance(e.reason, NewConnectionError): - raise ConnectTimeout(e, request=request) - - if isinstance(e.reason, ResponseError): - raise RetryError(e, request=request) - - if isinstance(e.reason, _ProxyError): - raise ProxyError(e, request=request) - - if isinstance(e.reason, _SSLError): - # This branch is for urllib3 v1.22 and later. - raise SSLError(e, request=request) - - raise ConnectionError(e, request=request) - - except ClosedPoolError as e: - raise ConnectionError(e, request=request) - - except _ProxyError as e: - raise ProxyError(e) - - except (_SSLError, _HTTPError) as e: - if isinstance(e, _SSLError): - # This branch is for urllib3 versions earlier than v1.22 - raise SSLError(e, request=request) - elif isinstance(e, ReadTimeoutError): - raise ReadTimeout(e, request=request) - elif isinstance(e, _InvalidHeader): - raise InvalidHeader(e, request=request) - else: - raise - - return self.build_response(request, resp) diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/requests/api.py b/venv/lib/python3.11/site-packages/pip/_vendor/requests/api.py deleted file mode 100644 index cd0b3ee..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/requests/api.py +++ /dev/null @@ -1,157 +0,0 @@ -""" -requests.api -~~~~~~~~~~~~ - -This module implements the Requests API. - -:copyright: (c) 2012 by Kenneth Reitz. -:license: Apache2, see LICENSE for more details. -""" - -from . import sessions - - -def request(method, url, **kwargs): - """Constructs and sends a :class:`Request `. - - :param method: method for the new :class:`Request` object: ``GET``, ``OPTIONS``, ``HEAD``, ``POST``, ``PUT``, ``PATCH``, or ``DELETE``. - :param url: URL for the new :class:`Request` object. - :param params: (optional) Dictionary, list of tuples or bytes to send - in the query string for the :class:`Request`. - :param data: (optional) Dictionary, list of tuples, bytes, or file-like - object to send in the body of the :class:`Request`. - :param json: (optional) A JSON serializable Python object to send in the body of the :class:`Request`. - :param headers: (optional) Dictionary of HTTP Headers to send with the :class:`Request`. - :param cookies: (optional) Dict or CookieJar object to send with the :class:`Request`. - :param files: (optional) Dictionary of ``'name': file-like-objects`` (or ``{'name': file-tuple}``) for multipart encoding upload. - ``file-tuple`` can be a 2-tuple ``('filename', fileobj)``, 3-tuple ``('filename', fileobj, 'content_type')`` - or a 4-tuple ``('filename', fileobj, 'content_type', custom_headers)``, where ``'content-type'`` is a string - defining the content type of the given file and ``custom_headers`` a dict-like object containing additional headers - to add for the file. - :param auth: (optional) Auth tuple to enable Basic/Digest/Custom HTTP Auth. - :param timeout: (optional) How many seconds to wait for the server to send data - before giving up, as a float, or a :ref:`(connect timeout, read - timeout) ` tuple. - :type timeout: float or tuple - :param allow_redirects: (optional) Boolean. Enable/disable GET/OPTIONS/POST/PUT/PATCH/DELETE/HEAD redirection. Defaults to ``True``. - :type allow_redirects: bool - :param proxies: (optional) Dictionary mapping protocol to the URL of the proxy. - :param verify: (optional) Either a boolean, in which case it controls whether we verify - the server's TLS certificate, or a string, in which case it must be a path - to a CA bundle to use. Defaults to ``True``. - :param stream: (optional) if ``False``, the response content will be immediately downloaded. - :param cert: (optional) if String, path to ssl client cert file (.pem). If Tuple, ('cert', 'key') pair. - :return: :class:`Response ` object - :rtype: requests.Response - - Usage:: - - >>> import requests - >>> req = requests.request('GET', 'https://httpbin.org/get') - >>> req - - """ - - # By using the 'with' statement we are sure the session is closed, thus we - # avoid leaving sockets open which can trigger a ResourceWarning in some - # cases, and look like a memory leak in others. - with sessions.Session() as session: - return session.request(method=method, url=url, **kwargs) - - -def get(url, params=None, **kwargs): - r"""Sends a GET request. - - :param url: URL for the new :class:`Request` object. - :param params: (optional) Dictionary, list of tuples or bytes to send - in the query string for the :class:`Request`. - :param \*\*kwargs: Optional arguments that ``request`` takes. - :return: :class:`Response ` object - :rtype: requests.Response - """ - - return request("get", url, params=params, **kwargs) - - -def options(url, **kwargs): - r"""Sends an OPTIONS request. - - :param url: URL for the new :class:`Request` object. - :param \*\*kwargs: Optional arguments that ``request`` takes. - :return: :class:`Response ` object - :rtype: requests.Response - """ - - return request("options", url, **kwargs) - - -def head(url, **kwargs): - r"""Sends a HEAD request. - - :param url: URL for the new :class:`Request` object. - :param \*\*kwargs: Optional arguments that ``request`` takes. If - `allow_redirects` is not provided, it will be set to `False` (as - opposed to the default :meth:`request` behavior). - :return: :class:`Response ` object - :rtype: requests.Response - """ - - kwargs.setdefault("allow_redirects", False) - return request("head", url, **kwargs) - - -def post(url, data=None, json=None, **kwargs): - r"""Sends a POST request. - - :param url: URL for the new :class:`Request` object. - :param data: (optional) Dictionary, list of tuples, bytes, or file-like - object to send in the body of the :class:`Request`. - :param json: (optional) A JSON serializable Python object to send in the body of the :class:`Request`. - :param \*\*kwargs: Optional arguments that ``request`` takes. - :return: :class:`Response ` object - :rtype: requests.Response - """ - - return request("post", url, data=data, json=json, **kwargs) - - -def put(url, data=None, **kwargs): - r"""Sends a PUT request. - - :param url: URL for the new :class:`Request` object. - :param data: (optional) Dictionary, list of tuples, bytes, or file-like - object to send in the body of the :class:`Request`. - :param json: (optional) A JSON serializable Python object to send in the body of the :class:`Request`. - :param \*\*kwargs: Optional arguments that ``request`` takes. - :return: :class:`Response ` object - :rtype: requests.Response - """ - - return request("put", url, data=data, **kwargs) - - -def patch(url, data=None, **kwargs): - r"""Sends a PATCH request. - - :param url: URL for the new :class:`Request` object. - :param data: (optional) Dictionary, list of tuples, bytes, or file-like - object to send in the body of the :class:`Request`. - :param json: (optional) A JSON serializable Python object to send in the body of the :class:`Request`. - :param \*\*kwargs: Optional arguments that ``request`` takes. - :return: :class:`Response ` object - :rtype: requests.Response - """ - - return request("patch", url, data=data, **kwargs) - - -def delete(url, **kwargs): - r"""Sends a DELETE request. - - :param url: URL for the new :class:`Request` object. - :param \*\*kwargs: Optional arguments that ``request`` takes. - :return: :class:`Response ` object - :rtype: requests.Response - """ - - return request("delete", url, **kwargs) diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/requests/auth.py b/venv/lib/python3.11/site-packages/pip/_vendor/requests/auth.py deleted file mode 100644 index 9733686..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/requests/auth.py +++ /dev/null @@ -1,315 +0,0 @@ -""" -requests.auth -~~~~~~~~~~~~~ - -This module contains the authentication handlers for Requests. -""" - -import hashlib -import os -import re -import threading -import time -import warnings -from base64 import b64encode - -from ._internal_utils import to_native_string -from .compat import basestring, str, urlparse -from .cookies import extract_cookies_to_jar -from .utils import parse_dict_header - -CONTENT_TYPE_FORM_URLENCODED = "application/x-www-form-urlencoded" -CONTENT_TYPE_MULTI_PART = "multipart/form-data" - - -def _basic_auth_str(username, password): - """Returns a Basic Auth string.""" - - # "I want us to put a big-ol' comment on top of it that - # says that this behaviour is dumb but we need to preserve - # it because people are relying on it." - # - Lukasa - # - # These are here solely to maintain backwards compatibility - # for things like ints. This will be removed in 3.0.0. - if not isinstance(username, basestring): - warnings.warn( - "Non-string usernames will no longer be supported in Requests " - "3.0.0. Please convert the object you've passed in ({!r}) to " - "a string or bytes object in the near future to avoid " - "problems.".format(username), - category=DeprecationWarning, - ) - username = str(username) - - if not isinstance(password, basestring): - warnings.warn( - "Non-string passwords will no longer be supported in Requests " - "3.0.0. Please convert the object you've passed in ({!r}) to " - "a string or bytes object in the near future to avoid " - "problems.".format(type(password)), - category=DeprecationWarning, - ) - password = str(password) - # -- End Removal -- - - if isinstance(username, str): - username = username.encode("latin1") - - if isinstance(password, str): - password = password.encode("latin1") - - authstr = "Basic " + to_native_string( - b64encode(b":".join((username, password))).strip() - ) - - return authstr - - -class AuthBase: - """Base class that all auth implementations derive from""" - - def __call__(self, r): - raise NotImplementedError("Auth hooks must be callable.") - - -class HTTPBasicAuth(AuthBase): - """Attaches HTTP Basic Authentication to the given Request object.""" - - def __init__(self, username, password): - self.username = username - self.password = password - - def __eq__(self, other): - return all( - [ - self.username == getattr(other, "username", None), - self.password == getattr(other, "password", None), - ] - ) - - def __ne__(self, other): - return not self == other - - def __call__(self, r): - r.headers["Authorization"] = _basic_auth_str(self.username, self.password) - return r - - -class HTTPProxyAuth(HTTPBasicAuth): - """Attaches HTTP Proxy Authentication to a given Request object.""" - - def __call__(self, r): - r.headers["Proxy-Authorization"] = _basic_auth_str(self.username, self.password) - return r - - -class HTTPDigestAuth(AuthBase): - """Attaches HTTP Digest Authentication to the given Request object.""" - - def __init__(self, username, password): - self.username = username - self.password = password - # Keep state in per-thread local storage - self._thread_local = threading.local() - - def init_per_thread_state(self): - # Ensure state is initialized just once per-thread - if not hasattr(self._thread_local, "init"): - self._thread_local.init = True - self._thread_local.last_nonce = "" - self._thread_local.nonce_count = 0 - self._thread_local.chal = {} - self._thread_local.pos = None - self._thread_local.num_401_calls = None - - def build_digest_header(self, method, url): - """ - :rtype: str - """ - - realm = self._thread_local.chal["realm"] - nonce = self._thread_local.chal["nonce"] - qop = self._thread_local.chal.get("qop") - algorithm = self._thread_local.chal.get("algorithm") - opaque = self._thread_local.chal.get("opaque") - hash_utf8 = None - - if algorithm is None: - _algorithm = "MD5" - else: - _algorithm = algorithm.upper() - # lambdas assume digest modules are imported at the top level - if _algorithm == "MD5" or _algorithm == "MD5-SESS": - - def md5_utf8(x): - if isinstance(x, str): - x = x.encode("utf-8") - return hashlib.md5(x).hexdigest() - - hash_utf8 = md5_utf8 - elif _algorithm == "SHA": - - def sha_utf8(x): - if isinstance(x, str): - x = x.encode("utf-8") - return hashlib.sha1(x).hexdigest() - - hash_utf8 = sha_utf8 - elif _algorithm == "SHA-256": - - def sha256_utf8(x): - if isinstance(x, str): - x = x.encode("utf-8") - return hashlib.sha256(x).hexdigest() - - hash_utf8 = sha256_utf8 - elif _algorithm == "SHA-512": - - def sha512_utf8(x): - if isinstance(x, str): - x = x.encode("utf-8") - return hashlib.sha512(x).hexdigest() - - hash_utf8 = sha512_utf8 - - KD = lambda s, d: hash_utf8(f"{s}:{d}") # noqa:E731 - - if hash_utf8 is None: - return None - - # XXX not implemented yet - entdig = None - p_parsed = urlparse(url) - #: path is request-uri defined in RFC 2616 which should not be empty - path = p_parsed.path or "/" - if p_parsed.query: - path += f"?{p_parsed.query}" - - A1 = f"{self.username}:{realm}:{self.password}" - A2 = f"{method}:{path}" - - HA1 = hash_utf8(A1) - HA2 = hash_utf8(A2) - - if nonce == self._thread_local.last_nonce: - self._thread_local.nonce_count += 1 - else: - self._thread_local.nonce_count = 1 - ncvalue = f"{self._thread_local.nonce_count:08x}" - s = str(self._thread_local.nonce_count).encode("utf-8") - s += nonce.encode("utf-8") - s += time.ctime().encode("utf-8") - s += os.urandom(8) - - cnonce = hashlib.sha1(s).hexdigest()[:16] - if _algorithm == "MD5-SESS": - HA1 = hash_utf8(f"{HA1}:{nonce}:{cnonce}") - - if not qop: - respdig = KD(HA1, f"{nonce}:{HA2}") - elif qop == "auth" or "auth" in qop.split(","): - noncebit = f"{nonce}:{ncvalue}:{cnonce}:auth:{HA2}" - respdig = KD(HA1, noncebit) - else: - # XXX handle auth-int. - return None - - self._thread_local.last_nonce = nonce - - # XXX should the partial digests be encoded too? - base = ( - f'username="{self.username}", realm="{realm}", nonce="{nonce}", ' - f'uri="{path}", response="{respdig}"' - ) - if opaque: - base += f', opaque="{opaque}"' - if algorithm: - base += f', algorithm="{algorithm}"' - if entdig: - base += f', digest="{entdig}"' - if qop: - base += f', qop="auth", nc={ncvalue}, cnonce="{cnonce}"' - - return f"Digest {base}" - - def handle_redirect(self, r, **kwargs): - """Reset num_401_calls counter on redirects.""" - if r.is_redirect: - self._thread_local.num_401_calls = 1 - - def handle_401(self, r, **kwargs): - """ - Takes the given response and tries digest-auth, if needed. - - :rtype: requests.Response - """ - - # If response is not 4xx, do not auth - # See https://github.com/psf/requests/issues/3772 - if not 400 <= r.status_code < 500: - self._thread_local.num_401_calls = 1 - return r - - if self._thread_local.pos is not None: - # Rewind the file position indicator of the body to where - # it was to resend the request. - r.request.body.seek(self._thread_local.pos) - s_auth = r.headers.get("www-authenticate", "") - - if "digest" in s_auth.lower() and self._thread_local.num_401_calls < 2: - - self._thread_local.num_401_calls += 1 - pat = re.compile(r"digest ", flags=re.IGNORECASE) - self._thread_local.chal = parse_dict_header(pat.sub("", s_auth, count=1)) - - # Consume content and release the original connection - # to allow our new request to reuse the same one. - r.content - r.close() - prep = r.request.copy() - extract_cookies_to_jar(prep._cookies, r.request, r.raw) - prep.prepare_cookies(prep._cookies) - - prep.headers["Authorization"] = self.build_digest_header( - prep.method, prep.url - ) - _r = r.connection.send(prep, **kwargs) - _r.history.append(r) - _r.request = prep - - return _r - - self._thread_local.num_401_calls = 1 - return r - - def __call__(self, r): - # Initialize per-thread state, if needed - self.init_per_thread_state() - # If we have a saved nonce, skip the 401 - if self._thread_local.last_nonce: - r.headers["Authorization"] = self.build_digest_header(r.method, r.url) - try: - self._thread_local.pos = r.body.tell() - except AttributeError: - # In the case of HTTPDigestAuth being reused and the body of - # the previous request was a file-like object, pos has the - # file position of the previous body. Ensure it's set to - # None. - self._thread_local.pos = None - r.register_hook("response", self.handle_401) - r.register_hook("response", self.handle_redirect) - self._thread_local.num_401_calls = 1 - - return r - - def __eq__(self, other): - return all( - [ - self.username == getattr(other, "username", None), - self.password == getattr(other, "password", None), - ] - ) - - def __ne__(self, other): - return not self == other diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/requests/certs.py b/venv/lib/python3.11/site-packages/pip/_vendor/requests/certs.py deleted file mode 100644 index 38696a1..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/requests/certs.py +++ /dev/null @@ -1,24 +0,0 @@ -#!/usr/bin/env python - -""" -requests.certs -~~~~~~~~~~~~~~ - -This module returns the preferred default CA certificate bundle. There is -only one — the one from the certifi package. - -If you are packaging Requests, e.g., for a Linux distribution or a managed -environment, you can change the definition of where() to return a separately -packaged CA bundle. -""" - -import os - -if "_PIP_STANDALONE_CERT" not in os.environ: - from pip._vendor.certifi import where -else: - def where(): - return os.environ["_PIP_STANDALONE_CERT"] - -if __name__ == "__main__": - print(where()) diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/requests/compat.py b/venv/lib/python3.11/site-packages/pip/_vendor/requests/compat.py deleted file mode 100644 index 9ab2bb4..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/requests/compat.py +++ /dev/null @@ -1,67 +0,0 @@ -""" -requests.compat -~~~~~~~~~~~~~~~ - -This module previously handled import compatibility issues -between Python 2 and Python 3. It remains for backwards -compatibility until the next major version. -""" - -from pip._vendor import chardet - -import sys - -# ------- -# Pythons -# ------- - -# Syntax sugar. -_ver = sys.version_info - -#: Python 2.x? -is_py2 = _ver[0] == 2 - -#: Python 3.x? -is_py3 = _ver[0] == 3 - -# Note: We've patched out simplejson support in pip because it prevents -# upgrading simplejson on Windows. -import json -from json import JSONDecodeError - -# Keep OrderedDict for backwards compatibility. -from collections import OrderedDict -from collections.abc import Callable, Mapping, MutableMapping -from http import cookiejar as cookielib -from http.cookies import Morsel -from io import StringIO - -# -------------- -# Legacy Imports -# -------------- -from urllib.parse import ( - quote, - quote_plus, - unquote, - unquote_plus, - urldefrag, - urlencode, - urljoin, - urlparse, - urlsplit, - urlunparse, -) -from urllib.request import ( - getproxies, - getproxies_environment, - parse_http_list, - proxy_bypass, - proxy_bypass_environment, -) - -builtin_str = str -str = str -bytes = bytes -basestring = (str, bytes) -numeric_types = (int, float) -integer_types = (int,) diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/requests/cookies.py b/venv/lib/python3.11/site-packages/pip/_vendor/requests/cookies.py deleted file mode 100644 index bf54ab2..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/requests/cookies.py +++ /dev/null @@ -1,561 +0,0 @@ -""" -requests.cookies -~~~~~~~~~~~~~~~~ - -Compatibility code to be able to use `cookielib.CookieJar` with requests. - -requests.utils imports from here, so be careful with imports. -""" - -import calendar -import copy -import time - -from ._internal_utils import to_native_string -from .compat import Morsel, MutableMapping, cookielib, urlparse, urlunparse - -try: - import threading -except ImportError: - import dummy_threading as threading - - -class MockRequest: - """Wraps a `requests.Request` to mimic a `urllib2.Request`. - - The code in `cookielib.CookieJar` expects this interface in order to correctly - manage cookie policies, i.e., determine whether a cookie can be set, given the - domains of the request and the cookie. - - The original request object is read-only. The client is responsible for collecting - the new headers via `get_new_headers()` and interpreting them appropriately. You - probably want `get_cookie_header`, defined below. - """ - - def __init__(self, request): - self._r = request - self._new_headers = {} - self.type = urlparse(self._r.url).scheme - - def get_type(self): - return self.type - - def get_host(self): - return urlparse(self._r.url).netloc - - def get_origin_req_host(self): - return self.get_host() - - def get_full_url(self): - # Only return the response's URL if the user hadn't set the Host - # header - if not self._r.headers.get("Host"): - return self._r.url - # If they did set it, retrieve it and reconstruct the expected domain - host = to_native_string(self._r.headers["Host"], encoding="utf-8") - parsed = urlparse(self._r.url) - # Reconstruct the URL as we expect it - return urlunparse( - [ - parsed.scheme, - host, - parsed.path, - parsed.params, - parsed.query, - parsed.fragment, - ] - ) - - def is_unverifiable(self): - return True - - def has_header(self, name): - return name in self._r.headers or name in self._new_headers - - def get_header(self, name, default=None): - return self._r.headers.get(name, self._new_headers.get(name, default)) - - def add_header(self, key, val): - """cookielib has no legitimate use for this method; add it back if you find one.""" - raise NotImplementedError( - "Cookie headers should be added with add_unredirected_header()" - ) - - def add_unredirected_header(self, name, value): - self._new_headers[name] = value - - def get_new_headers(self): - return self._new_headers - - @property - def unverifiable(self): - return self.is_unverifiable() - - @property - def origin_req_host(self): - return self.get_origin_req_host() - - @property - def host(self): - return self.get_host() - - -class MockResponse: - """Wraps a `httplib.HTTPMessage` to mimic a `urllib.addinfourl`. - - ...what? Basically, expose the parsed HTTP headers from the server response - the way `cookielib` expects to see them. - """ - - def __init__(self, headers): - """Make a MockResponse for `cookielib` to read. - - :param headers: a httplib.HTTPMessage or analogous carrying the headers - """ - self._headers = headers - - def info(self): - return self._headers - - def getheaders(self, name): - self._headers.getheaders(name) - - -def extract_cookies_to_jar(jar, request, response): - """Extract the cookies from the response into a CookieJar. - - :param jar: cookielib.CookieJar (not necessarily a RequestsCookieJar) - :param request: our own requests.Request object - :param response: urllib3.HTTPResponse object - """ - if not (hasattr(response, "_original_response") and response._original_response): - return - # the _original_response field is the wrapped httplib.HTTPResponse object, - req = MockRequest(request) - # pull out the HTTPMessage with the headers and put it in the mock: - res = MockResponse(response._original_response.msg) - jar.extract_cookies(res, req) - - -def get_cookie_header(jar, request): - """ - Produce an appropriate Cookie header string to be sent with `request`, or None. - - :rtype: str - """ - r = MockRequest(request) - jar.add_cookie_header(r) - return r.get_new_headers().get("Cookie") - - -def remove_cookie_by_name(cookiejar, name, domain=None, path=None): - """Unsets a cookie by name, by default over all domains and paths. - - Wraps CookieJar.clear(), is O(n). - """ - clearables = [] - for cookie in cookiejar: - if cookie.name != name: - continue - if domain is not None and domain != cookie.domain: - continue - if path is not None and path != cookie.path: - continue - clearables.append((cookie.domain, cookie.path, cookie.name)) - - for domain, path, name in clearables: - cookiejar.clear(domain, path, name) - - -class CookieConflictError(RuntimeError): - """There are two cookies that meet the criteria specified in the cookie jar. - Use .get and .set and include domain and path args in order to be more specific. - """ - - -class RequestsCookieJar(cookielib.CookieJar, MutableMapping): - """Compatibility class; is a cookielib.CookieJar, but exposes a dict - interface. - - This is the CookieJar we create by default for requests and sessions that - don't specify one, since some clients may expect response.cookies and - session.cookies to support dict operations. - - Requests does not use the dict interface internally; it's just for - compatibility with external client code. All requests code should work - out of the box with externally provided instances of ``CookieJar``, e.g. - ``LWPCookieJar`` and ``FileCookieJar``. - - Unlike a regular CookieJar, this class is pickleable. - - .. warning:: dictionary operations that are normally O(1) may be O(n). - """ - - def get(self, name, default=None, domain=None, path=None): - """Dict-like get() that also supports optional domain and path args in - order to resolve naming collisions from using one cookie jar over - multiple domains. - - .. warning:: operation is O(n), not O(1). - """ - try: - return self._find_no_duplicates(name, domain, path) - except KeyError: - return default - - def set(self, name, value, **kwargs): - """Dict-like set() that also supports optional domain and path args in - order to resolve naming collisions from using one cookie jar over - multiple domains. - """ - # support client code that unsets cookies by assignment of a None value: - if value is None: - remove_cookie_by_name( - self, name, domain=kwargs.get("domain"), path=kwargs.get("path") - ) - return - - if isinstance(value, Morsel): - c = morsel_to_cookie(value) - else: - c = create_cookie(name, value, **kwargs) - self.set_cookie(c) - return c - - def iterkeys(self): - """Dict-like iterkeys() that returns an iterator of names of cookies - from the jar. - - .. seealso:: itervalues() and iteritems(). - """ - for cookie in iter(self): - yield cookie.name - - def keys(self): - """Dict-like keys() that returns a list of names of cookies from the - jar. - - .. seealso:: values() and items(). - """ - return list(self.iterkeys()) - - def itervalues(self): - """Dict-like itervalues() that returns an iterator of values of cookies - from the jar. - - .. seealso:: iterkeys() and iteritems(). - """ - for cookie in iter(self): - yield cookie.value - - def values(self): - """Dict-like values() that returns a list of values of cookies from the - jar. - - .. seealso:: keys() and items(). - """ - return list(self.itervalues()) - - def iteritems(self): - """Dict-like iteritems() that returns an iterator of name-value tuples - from the jar. - - .. seealso:: iterkeys() and itervalues(). - """ - for cookie in iter(self): - yield cookie.name, cookie.value - - def items(self): - """Dict-like items() that returns a list of name-value tuples from the - jar. Allows client-code to call ``dict(RequestsCookieJar)`` and get a - vanilla python dict of key value pairs. - - .. seealso:: keys() and values(). - """ - return list(self.iteritems()) - - def list_domains(self): - """Utility method to list all the domains in the jar.""" - domains = [] - for cookie in iter(self): - if cookie.domain not in domains: - domains.append(cookie.domain) - return domains - - def list_paths(self): - """Utility method to list all the paths in the jar.""" - paths = [] - for cookie in iter(self): - if cookie.path not in paths: - paths.append(cookie.path) - return paths - - def multiple_domains(self): - """Returns True if there are multiple domains in the jar. - Returns False otherwise. - - :rtype: bool - """ - domains = [] - for cookie in iter(self): - if cookie.domain is not None and cookie.domain in domains: - return True - domains.append(cookie.domain) - return False # there is only one domain in jar - - def get_dict(self, domain=None, path=None): - """Takes as an argument an optional domain and path and returns a plain - old Python dict of name-value pairs of cookies that meet the - requirements. - - :rtype: dict - """ - dictionary = {} - for cookie in iter(self): - if (domain is None or cookie.domain == domain) and ( - path is None or cookie.path == path - ): - dictionary[cookie.name] = cookie.value - return dictionary - - def __contains__(self, name): - try: - return super().__contains__(name) - except CookieConflictError: - return True - - def __getitem__(self, name): - """Dict-like __getitem__() for compatibility with client code. Throws - exception if there are more than one cookie with name. In that case, - use the more explicit get() method instead. - - .. warning:: operation is O(n), not O(1). - """ - return self._find_no_duplicates(name) - - def __setitem__(self, name, value): - """Dict-like __setitem__ for compatibility with client code. Throws - exception if there is already a cookie of that name in the jar. In that - case, use the more explicit set() method instead. - """ - self.set(name, value) - - def __delitem__(self, name): - """Deletes a cookie given a name. Wraps ``cookielib.CookieJar``'s - ``remove_cookie_by_name()``. - """ - remove_cookie_by_name(self, name) - - def set_cookie(self, cookie, *args, **kwargs): - if ( - hasattr(cookie.value, "startswith") - and cookie.value.startswith('"') - and cookie.value.endswith('"') - ): - cookie.value = cookie.value.replace('\\"', "") - return super().set_cookie(cookie, *args, **kwargs) - - def update(self, other): - """Updates this jar with cookies from another CookieJar or dict-like""" - if isinstance(other, cookielib.CookieJar): - for cookie in other: - self.set_cookie(copy.copy(cookie)) - else: - super().update(other) - - def _find(self, name, domain=None, path=None): - """Requests uses this method internally to get cookie values. - - If there are conflicting cookies, _find arbitrarily chooses one. - See _find_no_duplicates if you want an exception thrown if there are - conflicting cookies. - - :param name: a string containing name of cookie - :param domain: (optional) string containing domain of cookie - :param path: (optional) string containing path of cookie - :return: cookie.value - """ - for cookie in iter(self): - if cookie.name == name: - if domain is None or cookie.domain == domain: - if path is None or cookie.path == path: - return cookie.value - - raise KeyError(f"name={name!r}, domain={domain!r}, path={path!r}") - - def _find_no_duplicates(self, name, domain=None, path=None): - """Both ``__get_item__`` and ``get`` call this function: it's never - used elsewhere in Requests. - - :param name: a string containing name of cookie - :param domain: (optional) string containing domain of cookie - :param path: (optional) string containing path of cookie - :raises KeyError: if cookie is not found - :raises CookieConflictError: if there are multiple cookies - that match name and optionally domain and path - :return: cookie.value - """ - toReturn = None - for cookie in iter(self): - if cookie.name == name: - if domain is None or cookie.domain == domain: - if path is None or cookie.path == path: - if toReturn is not None: - # if there are multiple cookies that meet passed in criteria - raise CookieConflictError( - f"There are multiple cookies with name, {name!r}" - ) - # we will eventually return this as long as no cookie conflict - toReturn = cookie.value - - if toReturn: - return toReturn - raise KeyError(f"name={name!r}, domain={domain!r}, path={path!r}") - - def __getstate__(self): - """Unlike a normal CookieJar, this class is pickleable.""" - state = self.__dict__.copy() - # remove the unpickleable RLock object - state.pop("_cookies_lock") - return state - - def __setstate__(self, state): - """Unlike a normal CookieJar, this class is pickleable.""" - self.__dict__.update(state) - if "_cookies_lock" not in self.__dict__: - self._cookies_lock = threading.RLock() - - def copy(self): - """Return a copy of this RequestsCookieJar.""" - new_cj = RequestsCookieJar() - new_cj.set_policy(self.get_policy()) - new_cj.update(self) - return new_cj - - def get_policy(self): - """Return the CookiePolicy instance used.""" - return self._policy - - -def _copy_cookie_jar(jar): - if jar is None: - return None - - if hasattr(jar, "copy"): - # We're dealing with an instance of RequestsCookieJar - return jar.copy() - # We're dealing with a generic CookieJar instance - new_jar = copy.copy(jar) - new_jar.clear() - for cookie in jar: - new_jar.set_cookie(copy.copy(cookie)) - return new_jar - - -def create_cookie(name, value, **kwargs): - """Make a cookie from underspecified parameters. - - By default, the pair of `name` and `value` will be set for the domain '' - and sent on every request (this is sometimes called a "supercookie"). - """ - result = { - "version": 0, - "name": name, - "value": value, - "port": None, - "domain": "", - "path": "/", - "secure": False, - "expires": None, - "discard": True, - "comment": None, - "comment_url": None, - "rest": {"HttpOnly": None}, - "rfc2109": False, - } - - badargs = set(kwargs) - set(result) - if badargs: - raise TypeError( - f"create_cookie() got unexpected keyword arguments: {list(badargs)}" - ) - - result.update(kwargs) - result["port_specified"] = bool(result["port"]) - result["domain_specified"] = bool(result["domain"]) - result["domain_initial_dot"] = result["domain"].startswith(".") - result["path_specified"] = bool(result["path"]) - - return cookielib.Cookie(**result) - - -def morsel_to_cookie(morsel): - """Convert a Morsel object into a Cookie containing the one k/v pair.""" - - expires = None - if morsel["max-age"]: - try: - expires = int(time.time() + int(morsel["max-age"])) - except ValueError: - raise TypeError(f"max-age: {morsel['max-age']} must be integer") - elif morsel["expires"]: - time_template = "%a, %d-%b-%Y %H:%M:%S GMT" - expires = calendar.timegm(time.strptime(morsel["expires"], time_template)) - return create_cookie( - comment=morsel["comment"], - comment_url=bool(morsel["comment"]), - discard=False, - domain=morsel["domain"], - expires=expires, - name=morsel.key, - path=morsel["path"], - port=None, - rest={"HttpOnly": morsel["httponly"]}, - rfc2109=False, - secure=bool(morsel["secure"]), - value=morsel.value, - version=morsel["version"] or 0, - ) - - -def cookiejar_from_dict(cookie_dict, cookiejar=None, overwrite=True): - """Returns a CookieJar from a key/value dictionary. - - :param cookie_dict: Dict of key/values to insert into CookieJar. - :param cookiejar: (optional) A cookiejar to add the cookies to. - :param overwrite: (optional) If False, will not replace cookies - already in the jar with new ones. - :rtype: CookieJar - """ - if cookiejar is None: - cookiejar = RequestsCookieJar() - - if cookie_dict is not None: - names_from_jar = [cookie.name for cookie in cookiejar] - for name in cookie_dict: - if overwrite or (name not in names_from_jar): - cookiejar.set_cookie(create_cookie(name, cookie_dict[name])) - - return cookiejar - - -def merge_cookies(cookiejar, cookies): - """Add cookies to cookiejar and returns a merged CookieJar. - - :param cookiejar: CookieJar object to add the cookies to. - :param cookies: Dictionary or CookieJar object to be added. - :rtype: CookieJar - """ - if not isinstance(cookiejar, cookielib.CookieJar): - raise ValueError("You can only merge into CookieJar") - - if isinstance(cookies, dict): - cookiejar = cookiejar_from_dict(cookies, cookiejar=cookiejar, overwrite=False) - elif isinstance(cookies, cookielib.CookieJar): - try: - cookiejar.update(cookies) - except AttributeError: - for cookie_in_jar in cookies: - cookiejar.set_cookie(cookie_in_jar) - - return cookiejar diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/requests/exceptions.py b/venv/lib/python3.11/site-packages/pip/_vendor/requests/exceptions.py deleted file mode 100644 index 168d073..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/requests/exceptions.py +++ /dev/null @@ -1,141 +0,0 @@ -""" -requests.exceptions -~~~~~~~~~~~~~~~~~~~ - -This module contains the set of Requests' exceptions. -""" -from pip._vendor.urllib3.exceptions import HTTPError as BaseHTTPError - -from .compat import JSONDecodeError as CompatJSONDecodeError - - -class RequestException(IOError): - """There was an ambiguous exception that occurred while handling your - request. - """ - - def __init__(self, *args, **kwargs): - """Initialize RequestException with `request` and `response` objects.""" - response = kwargs.pop("response", None) - self.response = response - self.request = kwargs.pop("request", None) - if response is not None and not self.request and hasattr(response, "request"): - self.request = self.response.request - super().__init__(*args, **kwargs) - - -class InvalidJSONError(RequestException): - """A JSON error occurred.""" - - -class JSONDecodeError(InvalidJSONError, CompatJSONDecodeError): - """Couldn't decode the text into json""" - - def __init__(self, *args, **kwargs): - """ - Construct the JSONDecodeError instance first with all - args. Then use it's args to construct the IOError so that - the json specific args aren't used as IOError specific args - and the error message from JSONDecodeError is preserved. - """ - CompatJSONDecodeError.__init__(self, *args) - InvalidJSONError.__init__(self, *self.args, **kwargs) - - -class HTTPError(RequestException): - """An HTTP error occurred.""" - - -class ConnectionError(RequestException): - """A Connection error occurred.""" - - -class ProxyError(ConnectionError): - """A proxy error occurred.""" - - -class SSLError(ConnectionError): - """An SSL error occurred.""" - - -class Timeout(RequestException): - """The request timed out. - - Catching this error will catch both - :exc:`~requests.exceptions.ConnectTimeout` and - :exc:`~requests.exceptions.ReadTimeout` errors. - """ - - -class ConnectTimeout(ConnectionError, Timeout): - """The request timed out while trying to connect to the remote server. - - Requests that produced this error are safe to retry. - """ - - -class ReadTimeout(Timeout): - """The server did not send any data in the allotted amount of time.""" - - -class URLRequired(RequestException): - """A valid URL is required to make a request.""" - - -class TooManyRedirects(RequestException): - """Too many redirects.""" - - -class MissingSchema(RequestException, ValueError): - """The URL scheme (e.g. http or https) is missing.""" - - -class InvalidSchema(RequestException, ValueError): - """The URL scheme provided is either invalid or unsupported.""" - - -class InvalidURL(RequestException, ValueError): - """The URL provided was somehow invalid.""" - - -class InvalidHeader(RequestException, ValueError): - """The header value provided was somehow invalid.""" - - -class InvalidProxyURL(InvalidURL): - """The proxy URL provided is invalid.""" - - -class ChunkedEncodingError(RequestException): - """The server declared chunked encoding but sent an invalid chunk.""" - - -class ContentDecodingError(RequestException, BaseHTTPError): - """Failed to decode response content.""" - - -class StreamConsumedError(RequestException, TypeError): - """The content for this response was already consumed.""" - - -class RetryError(RequestException): - """Custom retries logic failed""" - - -class UnrewindableBodyError(RequestException): - """Requests encountered an error when trying to rewind a body.""" - - -# Warnings - - -class RequestsWarning(Warning): - """Base warning for Requests.""" - - -class FileModeWarning(RequestsWarning, DeprecationWarning): - """A file was opened in text mode, but Requests determined its binary length.""" - - -class RequestsDependencyWarning(RequestsWarning): - """An imported dependency doesn't match the expected version range.""" diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/requests/help.py b/venv/lib/python3.11/site-packages/pip/_vendor/requests/help.py deleted file mode 100644 index 2d292c2..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/requests/help.py +++ /dev/null @@ -1,131 +0,0 @@ -"""Module containing bug report helper(s).""" - -import json -import platform -import ssl -import sys - -from pip._vendor import idna -from pip._vendor import urllib3 - -from . import __version__ as requests_version - -charset_normalizer = None - -try: - from pip._vendor import chardet -except ImportError: - chardet = None - -try: - from pip._vendor.urllib3.contrib import pyopenssl -except ImportError: - pyopenssl = None - OpenSSL = None - cryptography = None -else: - import cryptography - import OpenSSL - - -def _implementation(): - """Return a dict with the Python implementation and version. - - Provide both the name and the version of the Python implementation - currently running. For example, on CPython 3.10.3 it will return - {'name': 'CPython', 'version': '3.10.3'}. - - This function works best on CPython and PyPy: in particular, it probably - doesn't work for Jython or IronPython. Future investigation should be done - to work out the correct shape of the code for those platforms. - """ - implementation = platform.python_implementation() - - if implementation == "CPython": - implementation_version = platform.python_version() - elif implementation == "PyPy": - implementation_version = "{}.{}.{}".format( - sys.pypy_version_info.major, - sys.pypy_version_info.minor, - sys.pypy_version_info.micro, - ) - if sys.pypy_version_info.releaselevel != "final": - implementation_version = "".join( - [implementation_version, sys.pypy_version_info.releaselevel] - ) - elif implementation == "Jython": - implementation_version = platform.python_version() # Complete Guess - elif implementation == "IronPython": - implementation_version = platform.python_version() # Complete Guess - else: - implementation_version = "Unknown" - - return {"name": implementation, "version": implementation_version} - - -def info(): - """Generate information for a bug report.""" - try: - platform_info = { - "system": platform.system(), - "release": platform.release(), - } - except OSError: - platform_info = { - "system": "Unknown", - "release": "Unknown", - } - - implementation_info = _implementation() - urllib3_info = {"version": urllib3.__version__} - charset_normalizer_info = {"version": None} - chardet_info = {"version": None} - if charset_normalizer: - charset_normalizer_info = {"version": charset_normalizer.__version__} - if chardet: - chardet_info = {"version": chardet.__version__} - - pyopenssl_info = { - "version": None, - "openssl_version": "", - } - if OpenSSL: - pyopenssl_info = { - "version": OpenSSL.__version__, - "openssl_version": f"{OpenSSL.SSL.OPENSSL_VERSION_NUMBER:x}", - } - cryptography_info = { - "version": getattr(cryptography, "__version__", ""), - } - idna_info = { - "version": getattr(idna, "__version__", ""), - } - - system_ssl = ssl.OPENSSL_VERSION_NUMBER - system_ssl_info = {"version": f"{system_ssl:x}" if system_ssl is not None else ""} - - return { - "platform": platform_info, - "implementation": implementation_info, - "system_ssl": system_ssl_info, - "using_pyopenssl": pyopenssl is not None, - "using_charset_normalizer": chardet is None, - "pyOpenSSL": pyopenssl_info, - "urllib3": urllib3_info, - "chardet": chardet_info, - "charset_normalizer": charset_normalizer_info, - "cryptography": cryptography_info, - "idna": idna_info, - "requests": { - "version": requests_version, - }, - } - - -def main(): - """Pretty-print the bug information as JSON.""" - print(json.dumps(info(), sort_keys=True, indent=2)) - - -if __name__ == "__main__": - main() diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/requests/hooks.py b/venv/lib/python3.11/site-packages/pip/_vendor/requests/hooks.py deleted file mode 100644 index d181ba2..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/requests/hooks.py +++ /dev/null @@ -1,33 +0,0 @@ -""" -requests.hooks -~~~~~~~~~~~~~~ - -This module provides the capabilities for the Requests hooks system. - -Available hooks: - -``response``: - The response generated from a Request. -""" -HOOKS = ["response"] - - -def default_hooks(): - return {event: [] for event in HOOKS} - - -# TODO: response is the only one - - -def dispatch_hook(key, hooks, hook_data, **kwargs): - """Dispatches a hook dictionary on a given piece of data.""" - hooks = hooks or {} - hooks = hooks.get(key) - if hooks: - if hasattr(hooks, "__call__"): - hooks = [hooks] - for hook in hooks: - _hook_data = hook(hook_data, **kwargs) - if _hook_data is not None: - hook_data = _hook_data - return hook_data diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/requests/models.py b/venv/lib/python3.11/site-packages/pip/_vendor/requests/models.py deleted file mode 100644 index 76e6f19..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/requests/models.py +++ /dev/null @@ -1,1034 +0,0 @@ -""" -requests.models -~~~~~~~~~~~~~~~ - -This module contains the primary objects that power Requests. -""" - -import datetime - -# Import encoding now, to avoid implicit import later. -# Implicit import within threads may cause LookupError when standard library is in a ZIP, -# such as in Embedded Python. See https://github.com/psf/requests/issues/3578. -import encodings.idna # noqa: F401 -from io import UnsupportedOperation - -from pip._vendor.urllib3.exceptions import ( - DecodeError, - LocationParseError, - ProtocolError, - ReadTimeoutError, - SSLError, -) -from pip._vendor.urllib3.fields import RequestField -from pip._vendor.urllib3.filepost import encode_multipart_formdata -from pip._vendor.urllib3.util import parse_url - -from ._internal_utils import to_native_string, unicode_is_ascii -from .auth import HTTPBasicAuth -from .compat import ( - Callable, - JSONDecodeError, - Mapping, - basestring, - builtin_str, - chardet, - cookielib, -) -from .compat import json as complexjson -from .compat import urlencode, urlsplit, urlunparse -from .cookies import _copy_cookie_jar, cookiejar_from_dict, get_cookie_header -from .exceptions import ( - ChunkedEncodingError, - ConnectionError, - ContentDecodingError, - HTTPError, - InvalidJSONError, - InvalidURL, -) -from .exceptions import JSONDecodeError as RequestsJSONDecodeError -from .exceptions import MissingSchema -from .exceptions import SSLError as RequestsSSLError -from .exceptions import StreamConsumedError -from .hooks import default_hooks -from .status_codes import codes -from .structures import CaseInsensitiveDict -from .utils import ( - check_header_validity, - get_auth_from_url, - guess_filename, - guess_json_utf, - iter_slices, - parse_header_links, - requote_uri, - stream_decode_response_unicode, - super_len, - to_key_val_list, -) - -#: The set of HTTP status codes that indicate an automatically -#: processable redirect. -REDIRECT_STATI = ( - codes.moved, # 301 - codes.found, # 302 - codes.other, # 303 - codes.temporary_redirect, # 307 - codes.permanent_redirect, # 308 -) - -DEFAULT_REDIRECT_LIMIT = 30 -CONTENT_CHUNK_SIZE = 10 * 1024 -ITER_CHUNK_SIZE = 512 - - -class RequestEncodingMixin: - @property - def path_url(self): - """Build the path URL to use.""" - - url = [] - - p = urlsplit(self.url) - - path = p.path - if not path: - path = "/" - - url.append(path) - - query = p.query - if query: - url.append("?") - url.append(query) - - return "".join(url) - - @staticmethod - def _encode_params(data): - """Encode parameters in a piece of data. - - Will successfully encode parameters when passed as a dict or a list of - 2-tuples. Order is retained if data is a list of 2-tuples but arbitrary - if parameters are supplied as a dict. - """ - - if isinstance(data, (str, bytes)): - return data - elif hasattr(data, "read"): - return data - elif hasattr(data, "__iter__"): - result = [] - for k, vs in to_key_val_list(data): - if isinstance(vs, basestring) or not hasattr(vs, "__iter__"): - vs = [vs] - for v in vs: - if v is not None: - result.append( - ( - k.encode("utf-8") if isinstance(k, str) else k, - v.encode("utf-8") if isinstance(v, str) else v, - ) - ) - return urlencode(result, doseq=True) - else: - return data - - @staticmethod - def _encode_files(files, data): - """Build the body for a multipart/form-data request. - - Will successfully encode files when passed as a dict or a list of - tuples. Order is retained if data is a list of tuples but arbitrary - if parameters are supplied as a dict. - The tuples may be 2-tuples (filename, fileobj), 3-tuples (filename, fileobj, contentype) - or 4-tuples (filename, fileobj, contentype, custom_headers). - """ - if not files: - raise ValueError("Files must be provided.") - elif isinstance(data, basestring): - raise ValueError("Data must not be a string.") - - new_fields = [] - fields = to_key_val_list(data or {}) - files = to_key_val_list(files or {}) - - for field, val in fields: - if isinstance(val, basestring) or not hasattr(val, "__iter__"): - val = [val] - for v in val: - if v is not None: - # Don't call str() on bytestrings: in Py3 it all goes wrong. - if not isinstance(v, bytes): - v = str(v) - - new_fields.append( - ( - field.decode("utf-8") - if isinstance(field, bytes) - else field, - v.encode("utf-8") if isinstance(v, str) else v, - ) - ) - - for (k, v) in files: - # support for explicit filename - ft = None - fh = None - if isinstance(v, (tuple, list)): - if len(v) == 2: - fn, fp = v - elif len(v) == 3: - fn, fp, ft = v - else: - fn, fp, ft, fh = v - else: - fn = guess_filename(v) or k - fp = v - - if isinstance(fp, (str, bytes, bytearray)): - fdata = fp - elif hasattr(fp, "read"): - fdata = fp.read() - elif fp is None: - continue - else: - fdata = fp - - rf = RequestField(name=k, data=fdata, filename=fn, headers=fh) - rf.make_multipart(content_type=ft) - new_fields.append(rf) - - body, content_type = encode_multipart_formdata(new_fields) - - return body, content_type - - -class RequestHooksMixin: - def register_hook(self, event, hook): - """Properly register a hook.""" - - if event not in self.hooks: - raise ValueError(f'Unsupported event specified, with event name "{event}"') - - if isinstance(hook, Callable): - self.hooks[event].append(hook) - elif hasattr(hook, "__iter__"): - self.hooks[event].extend(h for h in hook if isinstance(h, Callable)) - - def deregister_hook(self, event, hook): - """Deregister a previously registered hook. - Returns True if the hook existed, False if not. - """ - - try: - self.hooks[event].remove(hook) - return True - except ValueError: - return False - - -class Request(RequestHooksMixin): - """A user-created :class:`Request ` object. - - Used to prepare a :class:`PreparedRequest `, which is sent to the server. - - :param method: HTTP method to use. - :param url: URL to send. - :param headers: dictionary of headers to send. - :param files: dictionary of {filename: fileobject} files to multipart upload. - :param data: the body to attach to the request. If a dictionary or - list of tuples ``[(key, value)]`` is provided, form-encoding will - take place. - :param json: json for the body to attach to the request (if files or data is not specified). - :param params: URL parameters to append to the URL. If a dictionary or - list of tuples ``[(key, value)]`` is provided, form-encoding will - take place. - :param auth: Auth handler or (user, pass) tuple. - :param cookies: dictionary or CookieJar of cookies to attach to this request. - :param hooks: dictionary of callback hooks, for internal usage. - - Usage:: - - >>> import requests - >>> req = requests.Request('GET', 'https://httpbin.org/get') - >>> req.prepare() - - """ - - def __init__( - self, - method=None, - url=None, - headers=None, - files=None, - data=None, - params=None, - auth=None, - cookies=None, - hooks=None, - json=None, - ): - - # Default empty dicts for dict params. - data = [] if data is None else data - files = [] if files is None else files - headers = {} if headers is None else headers - params = {} if params is None else params - hooks = {} if hooks is None else hooks - - self.hooks = default_hooks() - for (k, v) in list(hooks.items()): - self.register_hook(event=k, hook=v) - - self.method = method - self.url = url - self.headers = headers - self.files = files - self.data = data - self.json = json - self.params = params - self.auth = auth - self.cookies = cookies - - def __repr__(self): - return f"" - - def prepare(self): - """Constructs a :class:`PreparedRequest ` for transmission and returns it.""" - p = PreparedRequest() - p.prepare( - method=self.method, - url=self.url, - headers=self.headers, - files=self.files, - data=self.data, - json=self.json, - params=self.params, - auth=self.auth, - cookies=self.cookies, - hooks=self.hooks, - ) - return p - - -class PreparedRequest(RequestEncodingMixin, RequestHooksMixin): - """The fully mutable :class:`PreparedRequest ` object, - containing the exact bytes that will be sent to the server. - - Instances are generated from a :class:`Request ` object, and - should not be instantiated manually; doing so may produce undesirable - effects. - - Usage:: - - >>> import requests - >>> req = requests.Request('GET', 'https://httpbin.org/get') - >>> r = req.prepare() - >>> r - - - >>> s = requests.Session() - >>> s.send(r) - - """ - - def __init__(self): - #: HTTP verb to send to the server. - self.method = None - #: HTTP URL to send the request to. - self.url = None - #: dictionary of HTTP headers. - self.headers = None - # The `CookieJar` used to create the Cookie header will be stored here - # after prepare_cookies is called - self._cookies = None - #: request body to send to the server. - self.body = None - #: dictionary of callback hooks, for internal usage. - self.hooks = default_hooks() - #: integer denoting starting position of a readable file-like body. - self._body_position = None - - def prepare( - self, - method=None, - url=None, - headers=None, - files=None, - data=None, - params=None, - auth=None, - cookies=None, - hooks=None, - json=None, - ): - """Prepares the entire request with the given parameters.""" - - self.prepare_method(method) - self.prepare_url(url, params) - self.prepare_headers(headers) - self.prepare_cookies(cookies) - self.prepare_body(data, files, json) - self.prepare_auth(auth, url) - - # Note that prepare_auth must be last to enable authentication schemes - # such as OAuth to work on a fully prepared request. - - # This MUST go after prepare_auth. Authenticators could add a hook - self.prepare_hooks(hooks) - - def __repr__(self): - return f"" - - def copy(self): - p = PreparedRequest() - p.method = self.method - p.url = self.url - p.headers = self.headers.copy() if self.headers is not None else None - p._cookies = _copy_cookie_jar(self._cookies) - p.body = self.body - p.hooks = self.hooks - p._body_position = self._body_position - return p - - def prepare_method(self, method): - """Prepares the given HTTP method.""" - self.method = method - if self.method is not None: - self.method = to_native_string(self.method.upper()) - - @staticmethod - def _get_idna_encoded_host(host): - from pip._vendor import idna - - try: - host = idna.encode(host, uts46=True).decode("utf-8") - except idna.IDNAError: - raise UnicodeError - return host - - def prepare_url(self, url, params): - """Prepares the given HTTP URL.""" - #: Accept objects that have string representations. - #: We're unable to blindly call unicode/str functions - #: as this will include the bytestring indicator (b'') - #: on python 3.x. - #: https://github.com/psf/requests/pull/2238 - if isinstance(url, bytes): - url = url.decode("utf8") - else: - url = str(url) - - # Remove leading whitespaces from url - url = url.lstrip() - - # Don't do any URL preparation for non-HTTP schemes like `mailto`, - # `data` etc to work around exceptions from `url_parse`, which - # handles RFC 3986 only. - if ":" in url and not url.lower().startswith("http"): - self.url = url - return - - # Support for unicode domain names and paths. - try: - scheme, auth, host, port, path, query, fragment = parse_url(url) - except LocationParseError as e: - raise InvalidURL(*e.args) - - if not scheme: - raise MissingSchema( - f"Invalid URL {url!r}: No scheme supplied. " - f"Perhaps you meant https://{url}?" - ) - - if not host: - raise InvalidURL(f"Invalid URL {url!r}: No host supplied") - - # In general, we want to try IDNA encoding the hostname if the string contains - # non-ASCII characters. This allows users to automatically get the correct IDNA - # behaviour. For strings containing only ASCII characters, we need to also verify - # it doesn't start with a wildcard (*), before allowing the unencoded hostname. - if not unicode_is_ascii(host): - try: - host = self._get_idna_encoded_host(host) - except UnicodeError: - raise InvalidURL("URL has an invalid label.") - elif host.startswith(("*", ".")): - raise InvalidURL("URL has an invalid label.") - - # Carefully reconstruct the network location - netloc = auth or "" - if netloc: - netloc += "@" - netloc += host - if port: - netloc += f":{port}" - - # Bare domains aren't valid URLs. - if not path: - path = "/" - - if isinstance(params, (str, bytes)): - params = to_native_string(params) - - enc_params = self._encode_params(params) - if enc_params: - if query: - query = f"{query}&{enc_params}" - else: - query = enc_params - - url = requote_uri(urlunparse([scheme, netloc, path, None, query, fragment])) - self.url = url - - def prepare_headers(self, headers): - """Prepares the given HTTP headers.""" - - self.headers = CaseInsensitiveDict() - if headers: - for header in headers.items(): - # Raise exception on invalid header value. - check_header_validity(header) - name, value = header - self.headers[to_native_string(name)] = value - - def prepare_body(self, data, files, json=None): - """Prepares the given HTTP body data.""" - - # Check if file, fo, generator, iterator. - # If not, run through normal process. - - # Nottin' on you. - body = None - content_type = None - - if not data and json is not None: - # urllib3 requires a bytes-like body. Python 2's json.dumps - # provides this natively, but Python 3 gives a Unicode string. - content_type = "application/json" - - try: - body = complexjson.dumps(json, allow_nan=False) - except ValueError as ve: - raise InvalidJSONError(ve, request=self) - - if not isinstance(body, bytes): - body = body.encode("utf-8") - - is_stream = all( - [ - hasattr(data, "__iter__"), - not isinstance(data, (basestring, list, tuple, Mapping)), - ] - ) - - if is_stream: - try: - length = super_len(data) - except (TypeError, AttributeError, UnsupportedOperation): - length = None - - body = data - - if getattr(body, "tell", None) is not None: - # Record the current file position before reading. - # This will allow us to rewind a file in the event - # of a redirect. - try: - self._body_position = body.tell() - except OSError: - # This differentiates from None, allowing us to catch - # a failed `tell()` later when trying to rewind the body - self._body_position = object() - - if files: - raise NotImplementedError( - "Streamed bodies and files are mutually exclusive." - ) - - if length: - self.headers["Content-Length"] = builtin_str(length) - else: - self.headers["Transfer-Encoding"] = "chunked" - else: - # Multi-part file uploads. - if files: - (body, content_type) = self._encode_files(files, data) - else: - if data: - body = self._encode_params(data) - if isinstance(data, basestring) or hasattr(data, "read"): - content_type = None - else: - content_type = "application/x-www-form-urlencoded" - - self.prepare_content_length(body) - - # Add content-type if it wasn't explicitly provided. - if content_type and ("content-type" not in self.headers): - self.headers["Content-Type"] = content_type - - self.body = body - - def prepare_content_length(self, body): - """Prepare Content-Length header based on request method and body""" - if body is not None: - length = super_len(body) - if length: - # If length exists, set it. Otherwise, we fallback - # to Transfer-Encoding: chunked. - self.headers["Content-Length"] = builtin_str(length) - elif ( - self.method not in ("GET", "HEAD") - and self.headers.get("Content-Length") is None - ): - # Set Content-Length to 0 for methods that can have a body - # but don't provide one. (i.e. not GET or HEAD) - self.headers["Content-Length"] = "0" - - def prepare_auth(self, auth, url=""): - """Prepares the given HTTP auth data.""" - - # If no Auth is explicitly provided, extract it from the URL first. - if auth is None: - url_auth = get_auth_from_url(self.url) - auth = url_auth if any(url_auth) else None - - if auth: - if isinstance(auth, tuple) and len(auth) == 2: - # special-case basic HTTP auth - auth = HTTPBasicAuth(*auth) - - # Allow auth to make its changes. - r = auth(self) - - # Update self to reflect the auth changes. - self.__dict__.update(r.__dict__) - - # Recompute Content-Length - self.prepare_content_length(self.body) - - def prepare_cookies(self, cookies): - """Prepares the given HTTP cookie data. - - This function eventually generates a ``Cookie`` header from the - given cookies using cookielib. Due to cookielib's design, the header - will not be regenerated if it already exists, meaning this function - can only be called once for the life of the - :class:`PreparedRequest ` object. Any subsequent calls - to ``prepare_cookies`` will have no actual effect, unless the "Cookie" - header is removed beforehand. - """ - if isinstance(cookies, cookielib.CookieJar): - self._cookies = cookies - else: - self._cookies = cookiejar_from_dict(cookies) - - cookie_header = get_cookie_header(self._cookies, self) - if cookie_header is not None: - self.headers["Cookie"] = cookie_header - - def prepare_hooks(self, hooks): - """Prepares the given hooks.""" - # hooks can be passed as None to the prepare method and to this - # method. To prevent iterating over None, simply use an empty list - # if hooks is False-y - hooks = hooks or [] - for event in hooks: - self.register_hook(event, hooks[event]) - - -class Response: - """The :class:`Response ` object, which contains a - server's response to an HTTP request. - """ - - __attrs__ = [ - "_content", - "status_code", - "headers", - "url", - "history", - "encoding", - "reason", - "cookies", - "elapsed", - "request", - ] - - def __init__(self): - self._content = False - self._content_consumed = False - self._next = None - - #: Integer Code of responded HTTP Status, e.g. 404 or 200. - self.status_code = None - - #: Case-insensitive Dictionary of Response Headers. - #: For example, ``headers['content-encoding']`` will return the - #: value of a ``'Content-Encoding'`` response header. - self.headers = CaseInsensitiveDict() - - #: File-like object representation of response (for advanced usage). - #: Use of ``raw`` requires that ``stream=True`` be set on the request. - #: This requirement does not apply for use internally to Requests. - self.raw = None - - #: Final URL location of Response. - self.url = None - - #: Encoding to decode with when accessing r.text. - self.encoding = None - - #: A list of :class:`Response ` objects from - #: the history of the Request. Any redirect responses will end - #: up here. The list is sorted from the oldest to the most recent request. - self.history = [] - - #: Textual reason of responded HTTP Status, e.g. "Not Found" or "OK". - self.reason = None - - #: A CookieJar of Cookies the server sent back. - self.cookies = cookiejar_from_dict({}) - - #: The amount of time elapsed between sending the request - #: and the arrival of the response (as a timedelta). - #: This property specifically measures the time taken between sending - #: the first byte of the request and finishing parsing the headers. It - #: is therefore unaffected by consuming the response content or the - #: value of the ``stream`` keyword argument. - self.elapsed = datetime.timedelta(0) - - #: The :class:`PreparedRequest ` object to which this - #: is a response. - self.request = None - - def __enter__(self): - return self - - def __exit__(self, *args): - self.close() - - def __getstate__(self): - # Consume everything; accessing the content attribute makes - # sure the content has been fully read. - if not self._content_consumed: - self.content - - return {attr: getattr(self, attr, None) for attr in self.__attrs__} - - def __setstate__(self, state): - for name, value in state.items(): - setattr(self, name, value) - - # pickled objects do not have .raw - setattr(self, "_content_consumed", True) - setattr(self, "raw", None) - - def __repr__(self): - return f"" - - def __bool__(self): - """Returns True if :attr:`status_code` is less than 400. - - This attribute checks if the status code of the response is between - 400 and 600 to see if there was a client error or a server error. If - the status code, is between 200 and 400, this will return True. This - is **not** a check to see if the response code is ``200 OK``. - """ - return self.ok - - def __nonzero__(self): - """Returns True if :attr:`status_code` is less than 400. - - This attribute checks if the status code of the response is between - 400 and 600 to see if there was a client error or a server error. If - the status code, is between 200 and 400, this will return True. This - is **not** a check to see if the response code is ``200 OK``. - """ - return self.ok - - def __iter__(self): - """Allows you to use a response as an iterator.""" - return self.iter_content(128) - - @property - def ok(self): - """Returns True if :attr:`status_code` is less than 400, False if not. - - This attribute checks if the status code of the response is between - 400 and 600 to see if there was a client error or a server error. If - the status code is between 200 and 400, this will return True. This - is **not** a check to see if the response code is ``200 OK``. - """ - try: - self.raise_for_status() - except HTTPError: - return False - return True - - @property - def is_redirect(self): - """True if this Response is a well-formed HTTP redirect that could have - been processed automatically (by :meth:`Session.resolve_redirects`). - """ - return "location" in self.headers and self.status_code in REDIRECT_STATI - - @property - def is_permanent_redirect(self): - """True if this Response one of the permanent versions of redirect.""" - return "location" in self.headers and self.status_code in ( - codes.moved_permanently, - codes.permanent_redirect, - ) - - @property - def next(self): - """Returns a PreparedRequest for the next request in a redirect chain, if there is one.""" - return self._next - - @property - def apparent_encoding(self): - """The apparent encoding, provided by the charset_normalizer or chardet libraries.""" - return chardet.detect(self.content)["encoding"] - - def iter_content(self, chunk_size=1, decode_unicode=False): - """Iterates over the response data. When stream=True is set on the - request, this avoids reading the content at once into memory for - large responses. The chunk size is the number of bytes it should - read into memory. This is not necessarily the length of each item - returned as decoding can take place. - - chunk_size must be of type int or None. A value of None will - function differently depending on the value of `stream`. - stream=True will read data as it arrives in whatever size the - chunks are received. If stream=False, data is returned as - a single chunk. - - If decode_unicode is True, content will be decoded using the best - available encoding based on the response. - """ - - def generate(): - # Special case for urllib3. - if hasattr(self.raw, "stream"): - try: - yield from self.raw.stream(chunk_size, decode_content=True) - except ProtocolError as e: - raise ChunkedEncodingError(e) - except DecodeError as e: - raise ContentDecodingError(e) - except ReadTimeoutError as e: - raise ConnectionError(e) - except SSLError as e: - raise RequestsSSLError(e) - else: - # Standard file-like object. - while True: - chunk = self.raw.read(chunk_size) - if not chunk: - break - yield chunk - - self._content_consumed = True - - if self._content_consumed and isinstance(self._content, bool): - raise StreamConsumedError() - elif chunk_size is not None and not isinstance(chunk_size, int): - raise TypeError( - f"chunk_size must be an int, it is instead a {type(chunk_size)}." - ) - # simulate reading small chunks of the content - reused_chunks = iter_slices(self._content, chunk_size) - - stream_chunks = generate() - - chunks = reused_chunks if self._content_consumed else stream_chunks - - if decode_unicode: - chunks = stream_decode_response_unicode(chunks, self) - - return chunks - - def iter_lines( - self, chunk_size=ITER_CHUNK_SIZE, decode_unicode=False, delimiter=None - ): - """Iterates over the response data, one line at a time. When - stream=True is set on the request, this avoids reading the - content at once into memory for large responses. - - .. note:: This method is not reentrant safe. - """ - - pending = None - - for chunk in self.iter_content( - chunk_size=chunk_size, decode_unicode=decode_unicode - ): - - if pending is not None: - chunk = pending + chunk - - if delimiter: - lines = chunk.split(delimiter) - else: - lines = chunk.splitlines() - - if lines and lines[-1] and chunk and lines[-1][-1] == chunk[-1]: - pending = lines.pop() - else: - pending = None - - yield from lines - - if pending is not None: - yield pending - - @property - def content(self): - """Content of the response, in bytes.""" - - if self._content is False: - # Read the contents. - if self._content_consumed: - raise RuntimeError("The content for this response was already consumed") - - if self.status_code == 0 or self.raw is None: - self._content = None - else: - self._content = b"".join(self.iter_content(CONTENT_CHUNK_SIZE)) or b"" - - self._content_consumed = True - # don't need to release the connection; that's been handled by urllib3 - # since we exhausted the data. - return self._content - - @property - def text(self): - """Content of the response, in unicode. - - If Response.encoding is None, encoding will be guessed using - ``charset_normalizer`` or ``chardet``. - - The encoding of the response content is determined based solely on HTTP - headers, following RFC 2616 to the letter. If you can take advantage of - non-HTTP knowledge to make a better guess at the encoding, you should - set ``r.encoding`` appropriately before accessing this property. - """ - - # Try charset from content-type - content = None - encoding = self.encoding - - if not self.content: - return "" - - # Fallback to auto-detected encoding. - if self.encoding is None: - encoding = self.apparent_encoding - - # Decode unicode from given encoding. - try: - content = str(self.content, encoding, errors="replace") - except (LookupError, TypeError): - # A LookupError is raised if the encoding was not found which could - # indicate a misspelling or similar mistake. - # - # A TypeError can be raised if encoding is None - # - # So we try blindly encoding. - content = str(self.content, errors="replace") - - return content - - def json(self, **kwargs): - r"""Returns the json-encoded content of a response, if any. - - :param \*\*kwargs: Optional arguments that ``json.loads`` takes. - :raises requests.exceptions.JSONDecodeError: If the response body does not - contain valid json. - """ - - if not self.encoding and self.content and len(self.content) > 3: - # No encoding set. JSON RFC 4627 section 3 states we should expect - # UTF-8, -16 or -32. Detect which one to use; If the detection or - # decoding fails, fall back to `self.text` (using charset_normalizer to make - # a best guess). - encoding = guess_json_utf(self.content) - if encoding is not None: - try: - return complexjson.loads(self.content.decode(encoding), **kwargs) - except UnicodeDecodeError: - # Wrong UTF codec detected; usually because it's not UTF-8 - # but some other 8-bit codec. This is an RFC violation, - # and the server didn't bother to tell us what codec *was* - # used. - pass - except JSONDecodeError as e: - raise RequestsJSONDecodeError(e.msg, e.doc, e.pos) - - try: - return complexjson.loads(self.text, **kwargs) - except JSONDecodeError as e: - # Catch JSON-related errors and raise as requests.JSONDecodeError - # This aliases json.JSONDecodeError and simplejson.JSONDecodeError - raise RequestsJSONDecodeError(e.msg, e.doc, e.pos) - - @property - def links(self): - """Returns the parsed header links of the response, if any.""" - - header = self.headers.get("link") - - resolved_links = {} - - if header: - links = parse_header_links(header) - - for link in links: - key = link.get("rel") or link.get("url") - resolved_links[key] = link - - return resolved_links - - def raise_for_status(self): - """Raises :class:`HTTPError`, if one occurred.""" - - http_error_msg = "" - if isinstance(self.reason, bytes): - # We attempt to decode utf-8 first because some servers - # choose to localize their reason strings. If the string - # isn't utf-8, we fall back to iso-8859-1 for all other - # encodings. (See PR #3538) - try: - reason = self.reason.decode("utf-8") - except UnicodeDecodeError: - reason = self.reason.decode("iso-8859-1") - else: - reason = self.reason - - if 400 <= self.status_code < 500: - http_error_msg = ( - f"{self.status_code} Client Error: {reason} for url: {self.url}" - ) - - elif 500 <= self.status_code < 600: - http_error_msg = ( - f"{self.status_code} Server Error: {reason} for url: {self.url}" - ) - - if http_error_msg: - raise HTTPError(http_error_msg, response=self) - - def close(self): - """Releases the connection back to the pool. Once this method has been - called the underlying ``raw`` object must not be accessed again. - - *Note: Should not normally need to be called explicitly.* - """ - if not self._content_consumed: - self.raw.close() - - release_conn = getattr(self.raw, "release_conn", None) - if release_conn is not None: - release_conn() diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/requests/packages.py b/venv/lib/python3.11/site-packages/pip/_vendor/requests/packages.py deleted file mode 100644 index 9582fa7..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/requests/packages.py +++ /dev/null @@ -1,16 +0,0 @@ -import sys - -# This code exists for backwards compatibility reasons. -# I don't like it either. Just look the other way. :) - -for package in ('urllib3', 'idna', 'chardet'): - vendored_package = "pip._vendor." + package - locals()[package] = __import__(vendored_package) - # This traversal is apparently necessary such that the identities are - # preserved (requests.packages.urllib3.* is urllib3.*) - for mod in list(sys.modules): - if mod == vendored_package or mod.startswith(vendored_package + '.'): - unprefixed_mod = mod[len("pip._vendor."):] - sys.modules['pip._vendor.requests.packages.' + unprefixed_mod] = sys.modules[mod] - -# Kinda cool, though, right? diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/requests/sessions.py b/venv/lib/python3.11/site-packages/pip/_vendor/requests/sessions.py deleted file mode 100644 index dbcf2a7..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/requests/sessions.py +++ /dev/null @@ -1,833 +0,0 @@ -""" -requests.sessions -~~~~~~~~~~~~~~~~~ - -This module provides a Session object to manage and persist settings across -requests (cookies, auth, proxies). -""" -import os -import sys -import time -from collections import OrderedDict -from datetime import timedelta - -from ._internal_utils import to_native_string -from .adapters import HTTPAdapter -from .auth import _basic_auth_str -from .compat import Mapping, cookielib, urljoin, urlparse -from .cookies import ( - RequestsCookieJar, - cookiejar_from_dict, - extract_cookies_to_jar, - merge_cookies, -) -from .exceptions import ( - ChunkedEncodingError, - ContentDecodingError, - InvalidSchema, - TooManyRedirects, -) -from .hooks import default_hooks, dispatch_hook - -# formerly defined here, reexposed here for backward compatibility -from .models import ( # noqa: F401 - DEFAULT_REDIRECT_LIMIT, - REDIRECT_STATI, - PreparedRequest, - Request, -) -from .status_codes import codes -from .structures import CaseInsensitiveDict -from .utils import ( # noqa: F401 - DEFAULT_PORTS, - default_headers, - get_auth_from_url, - get_environ_proxies, - get_netrc_auth, - requote_uri, - resolve_proxies, - rewind_body, - should_bypass_proxies, - to_key_val_list, -) - -# Preferred clock, based on which one is more accurate on a given system. -if sys.platform == "win32": - preferred_clock = time.perf_counter -else: - preferred_clock = time.time - - -def merge_setting(request_setting, session_setting, dict_class=OrderedDict): - """Determines appropriate setting for a given request, taking into account - the explicit setting on that request, and the setting in the session. If a - setting is a dictionary, they will be merged together using `dict_class` - """ - - if session_setting is None: - return request_setting - - if request_setting is None: - return session_setting - - # Bypass if not a dictionary (e.g. verify) - if not ( - isinstance(session_setting, Mapping) and isinstance(request_setting, Mapping) - ): - return request_setting - - merged_setting = dict_class(to_key_val_list(session_setting)) - merged_setting.update(to_key_val_list(request_setting)) - - # Remove keys that are set to None. Extract keys first to avoid altering - # the dictionary during iteration. - none_keys = [k for (k, v) in merged_setting.items() if v is None] - for key in none_keys: - del merged_setting[key] - - return merged_setting - - -def merge_hooks(request_hooks, session_hooks, dict_class=OrderedDict): - """Properly merges both requests and session hooks. - - This is necessary because when request_hooks == {'response': []}, the - merge breaks Session hooks entirely. - """ - if session_hooks is None or session_hooks.get("response") == []: - return request_hooks - - if request_hooks is None or request_hooks.get("response") == []: - return session_hooks - - return merge_setting(request_hooks, session_hooks, dict_class) - - -class SessionRedirectMixin: - def get_redirect_target(self, resp): - """Receives a Response. Returns a redirect URI or ``None``""" - # Due to the nature of how requests processes redirects this method will - # be called at least once upon the original response and at least twice - # on each subsequent redirect response (if any). - # If a custom mixin is used to handle this logic, it may be advantageous - # to cache the redirect location onto the response object as a private - # attribute. - if resp.is_redirect: - location = resp.headers["location"] - # Currently the underlying http module on py3 decode headers - # in latin1, but empirical evidence suggests that latin1 is very - # rarely used with non-ASCII characters in HTTP headers. - # It is more likely to get UTF8 header rather than latin1. - # This causes incorrect handling of UTF8 encoded location headers. - # To solve this, we re-encode the location in latin1. - location = location.encode("latin1") - return to_native_string(location, "utf8") - return None - - def should_strip_auth(self, old_url, new_url): - """Decide whether Authorization header should be removed when redirecting""" - old_parsed = urlparse(old_url) - new_parsed = urlparse(new_url) - if old_parsed.hostname != new_parsed.hostname: - return True - # Special case: allow http -> https redirect when using the standard - # ports. This isn't specified by RFC 7235, but is kept to avoid - # breaking backwards compatibility with older versions of requests - # that allowed any redirects on the same host. - if ( - old_parsed.scheme == "http" - and old_parsed.port in (80, None) - and new_parsed.scheme == "https" - and new_parsed.port in (443, None) - ): - return False - - # Handle default port usage corresponding to scheme. - changed_port = old_parsed.port != new_parsed.port - changed_scheme = old_parsed.scheme != new_parsed.scheme - default_port = (DEFAULT_PORTS.get(old_parsed.scheme, None), None) - if ( - not changed_scheme - and old_parsed.port in default_port - and new_parsed.port in default_port - ): - return False - - # Standard case: root URI must match - return changed_port or changed_scheme - - def resolve_redirects( - self, - resp, - req, - stream=False, - timeout=None, - verify=True, - cert=None, - proxies=None, - yield_requests=False, - **adapter_kwargs, - ): - """Receives a Response. Returns a generator of Responses or Requests.""" - - hist = [] # keep track of history - - url = self.get_redirect_target(resp) - previous_fragment = urlparse(req.url).fragment - while url: - prepared_request = req.copy() - - # Update history and keep track of redirects. - # resp.history must ignore the original request in this loop - hist.append(resp) - resp.history = hist[1:] - - try: - resp.content # Consume socket so it can be released - except (ChunkedEncodingError, ContentDecodingError, RuntimeError): - resp.raw.read(decode_content=False) - - if len(resp.history) >= self.max_redirects: - raise TooManyRedirects( - f"Exceeded {self.max_redirects} redirects.", response=resp - ) - - # Release the connection back into the pool. - resp.close() - - # Handle redirection without scheme (see: RFC 1808 Section 4) - if url.startswith("//"): - parsed_rurl = urlparse(resp.url) - url = ":".join([to_native_string(parsed_rurl.scheme), url]) - - # Normalize url case and attach previous fragment if needed (RFC 7231 7.1.2) - parsed = urlparse(url) - if parsed.fragment == "" and previous_fragment: - parsed = parsed._replace(fragment=previous_fragment) - elif parsed.fragment: - previous_fragment = parsed.fragment - url = parsed.geturl() - - # Facilitate relative 'location' headers, as allowed by RFC 7231. - # (e.g. '/path/to/resource' instead of 'http://domain.tld/path/to/resource') - # Compliant with RFC3986, we percent encode the url. - if not parsed.netloc: - url = urljoin(resp.url, requote_uri(url)) - else: - url = requote_uri(url) - - prepared_request.url = to_native_string(url) - - self.rebuild_method(prepared_request, resp) - - # https://github.com/psf/requests/issues/1084 - if resp.status_code not in ( - codes.temporary_redirect, - codes.permanent_redirect, - ): - # https://github.com/psf/requests/issues/3490 - purged_headers = ("Content-Length", "Content-Type", "Transfer-Encoding") - for header in purged_headers: - prepared_request.headers.pop(header, None) - prepared_request.body = None - - headers = prepared_request.headers - headers.pop("Cookie", None) - - # Extract any cookies sent on the response to the cookiejar - # in the new request. Because we've mutated our copied prepared - # request, use the old one that we haven't yet touched. - extract_cookies_to_jar(prepared_request._cookies, req, resp.raw) - merge_cookies(prepared_request._cookies, self.cookies) - prepared_request.prepare_cookies(prepared_request._cookies) - - # Rebuild auth and proxy information. - proxies = self.rebuild_proxies(prepared_request, proxies) - self.rebuild_auth(prepared_request, resp) - - # A failed tell() sets `_body_position` to `object()`. This non-None - # value ensures `rewindable` will be True, allowing us to raise an - # UnrewindableBodyError, instead of hanging the connection. - rewindable = prepared_request._body_position is not None and ( - "Content-Length" in headers or "Transfer-Encoding" in headers - ) - - # Attempt to rewind consumed file-like object. - if rewindable: - rewind_body(prepared_request) - - # Override the original request. - req = prepared_request - - if yield_requests: - yield req - else: - - resp = self.send( - req, - stream=stream, - timeout=timeout, - verify=verify, - cert=cert, - proxies=proxies, - allow_redirects=False, - **adapter_kwargs, - ) - - extract_cookies_to_jar(self.cookies, prepared_request, resp.raw) - - # extract redirect url, if any, for the next loop - url = self.get_redirect_target(resp) - yield resp - - def rebuild_auth(self, prepared_request, response): - """When being redirected we may want to strip authentication from the - request to avoid leaking credentials. This method intelligently removes - and reapplies authentication where possible to avoid credential loss. - """ - headers = prepared_request.headers - url = prepared_request.url - - if "Authorization" in headers and self.should_strip_auth( - response.request.url, url - ): - # If we get redirected to a new host, we should strip out any - # authentication headers. - del headers["Authorization"] - - # .netrc might have more auth for us on our new host. - new_auth = get_netrc_auth(url) if self.trust_env else None - if new_auth is not None: - prepared_request.prepare_auth(new_auth) - - def rebuild_proxies(self, prepared_request, proxies): - """This method re-evaluates the proxy configuration by considering the - environment variables. If we are redirected to a URL covered by - NO_PROXY, we strip the proxy configuration. Otherwise, we set missing - proxy keys for this URL (in case they were stripped by a previous - redirect). - - This method also replaces the Proxy-Authorization header where - necessary. - - :rtype: dict - """ - headers = prepared_request.headers - scheme = urlparse(prepared_request.url).scheme - new_proxies = resolve_proxies(prepared_request, proxies, self.trust_env) - - if "Proxy-Authorization" in headers: - del headers["Proxy-Authorization"] - - try: - username, password = get_auth_from_url(new_proxies[scheme]) - except KeyError: - username, password = None, None - - # urllib3 handles proxy authorization for us in the standard adapter. - # Avoid appending this to TLS tunneled requests where it may be leaked. - if not scheme.startswith('https') and username and password: - headers["Proxy-Authorization"] = _basic_auth_str(username, password) - - return new_proxies - - def rebuild_method(self, prepared_request, response): - """When being redirected we may want to change the method of the request - based on certain specs or browser behavior. - """ - method = prepared_request.method - - # https://tools.ietf.org/html/rfc7231#section-6.4.4 - if response.status_code == codes.see_other and method != "HEAD": - method = "GET" - - # Do what the browsers do, despite standards... - # First, turn 302s into GETs. - if response.status_code == codes.found and method != "HEAD": - method = "GET" - - # Second, if a POST is responded to with a 301, turn it into a GET. - # This bizarre behaviour is explained in Issue 1704. - if response.status_code == codes.moved and method == "POST": - method = "GET" - - prepared_request.method = method - - -class Session(SessionRedirectMixin): - """A Requests session. - - Provides cookie persistence, connection-pooling, and configuration. - - Basic Usage:: - - >>> import requests - >>> s = requests.Session() - >>> s.get('https://httpbin.org/get') - - - Or as a context manager:: - - >>> with requests.Session() as s: - ... s.get('https://httpbin.org/get') - - """ - - __attrs__ = [ - "headers", - "cookies", - "auth", - "proxies", - "hooks", - "params", - "verify", - "cert", - "adapters", - "stream", - "trust_env", - "max_redirects", - ] - - def __init__(self): - - #: A case-insensitive dictionary of headers to be sent on each - #: :class:`Request ` sent from this - #: :class:`Session `. - self.headers = default_headers() - - #: Default Authentication tuple or object to attach to - #: :class:`Request `. - self.auth = None - - #: Dictionary mapping protocol or protocol and host to the URL of the proxy - #: (e.g. {'http': 'foo.bar:3128', 'http://host.name': 'foo.bar:4012'}) to - #: be used on each :class:`Request `. - self.proxies = {} - - #: Event-handling hooks. - self.hooks = default_hooks() - - #: Dictionary of querystring data to attach to each - #: :class:`Request `. The dictionary values may be lists for - #: representing multivalued query parameters. - self.params = {} - - #: Stream response content default. - self.stream = False - - #: SSL Verification default. - #: Defaults to `True`, requiring requests to verify the TLS certificate at the - #: remote end. - #: If verify is set to `False`, requests will accept any TLS certificate - #: presented by the server, and will ignore hostname mismatches and/or - #: expired certificates, which will make your application vulnerable to - #: man-in-the-middle (MitM) attacks. - #: Only set this to `False` for testing. - self.verify = True - - #: SSL client certificate default, if String, path to ssl client - #: cert file (.pem). If Tuple, ('cert', 'key') pair. - self.cert = None - - #: Maximum number of redirects allowed. If the request exceeds this - #: limit, a :class:`TooManyRedirects` exception is raised. - #: This defaults to requests.models.DEFAULT_REDIRECT_LIMIT, which is - #: 30. - self.max_redirects = DEFAULT_REDIRECT_LIMIT - - #: Trust environment settings for proxy configuration, default - #: authentication and similar. - self.trust_env = True - - #: A CookieJar containing all currently outstanding cookies set on this - #: session. By default it is a - #: :class:`RequestsCookieJar `, but - #: may be any other ``cookielib.CookieJar`` compatible object. - self.cookies = cookiejar_from_dict({}) - - # Default connection adapters. - self.adapters = OrderedDict() - self.mount("https://", HTTPAdapter()) - self.mount("http://", HTTPAdapter()) - - def __enter__(self): - return self - - def __exit__(self, *args): - self.close() - - def prepare_request(self, request): - """Constructs a :class:`PreparedRequest ` for - transmission and returns it. The :class:`PreparedRequest` has settings - merged from the :class:`Request ` instance and those of the - :class:`Session`. - - :param request: :class:`Request` instance to prepare with this - session's settings. - :rtype: requests.PreparedRequest - """ - cookies = request.cookies or {} - - # Bootstrap CookieJar. - if not isinstance(cookies, cookielib.CookieJar): - cookies = cookiejar_from_dict(cookies) - - # Merge with session cookies - merged_cookies = merge_cookies( - merge_cookies(RequestsCookieJar(), self.cookies), cookies - ) - - # Set environment's basic authentication if not explicitly set. - auth = request.auth - if self.trust_env and not auth and not self.auth: - auth = get_netrc_auth(request.url) - - p = PreparedRequest() - p.prepare( - method=request.method.upper(), - url=request.url, - files=request.files, - data=request.data, - json=request.json, - headers=merge_setting( - request.headers, self.headers, dict_class=CaseInsensitiveDict - ), - params=merge_setting(request.params, self.params), - auth=merge_setting(auth, self.auth), - cookies=merged_cookies, - hooks=merge_hooks(request.hooks, self.hooks), - ) - return p - - def request( - self, - method, - url, - params=None, - data=None, - headers=None, - cookies=None, - files=None, - auth=None, - timeout=None, - allow_redirects=True, - proxies=None, - hooks=None, - stream=None, - verify=None, - cert=None, - json=None, - ): - """Constructs a :class:`Request `, prepares it and sends it. - Returns :class:`Response ` object. - - :param method: method for the new :class:`Request` object. - :param url: URL for the new :class:`Request` object. - :param params: (optional) Dictionary or bytes to be sent in the query - string for the :class:`Request`. - :param data: (optional) Dictionary, list of tuples, bytes, or file-like - object to send in the body of the :class:`Request`. - :param json: (optional) json to send in the body of the - :class:`Request`. - :param headers: (optional) Dictionary of HTTP Headers to send with the - :class:`Request`. - :param cookies: (optional) Dict or CookieJar object to send with the - :class:`Request`. - :param files: (optional) Dictionary of ``'filename': file-like-objects`` - for multipart encoding upload. - :param auth: (optional) Auth tuple or callable to enable - Basic/Digest/Custom HTTP Auth. - :param timeout: (optional) How long to wait for the server to send - data before giving up, as a float, or a :ref:`(connect timeout, - read timeout) ` tuple. - :type timeout: float or tuple - :param allow_redirects: (optional) Set to True by default. - :type allow_redirects: bool - :param proxies: (optional) Dictionary mapping protocol or protocol and - hostname to the URL of the proxy. - :param stream: (optional) whether to immediately download the response - content. Defaults to ``False``. - :param verify: (optional) Either a boolean, in which case it controls whether we verify - the server's TLS certificate, or a string, in which case it must be a path - to a CA bundle to use. Defaults to ``True``. When set to - ``False``, requests will accept any TLS certificate presented by - the server, and will ignore hostname mismatches and/or expired - certificates, which will make your application vulnerable to - man-in-the-middle (MitM) attacks. Setting verify to ``False`` - may be useful during local development or testing. - :param cert: (optional) if String, path to ssl client cert file (.pem). - If Tuple, ('cert', 'key') pair. - :rtype: requests.Response - """ - # Create the Request. - req = Request( - method=method.upper(), - url=url, - headers=headers, - files=files, - data=data or {}, - json=json, - params=params or {}, - auth=auth, - cookies=cookies, - hooks=hooks, - ) - prep = self.prepare_request(req) - - proxies = proxies or {} - - settings = self.merge_environment_settings( - prep.url, proxies, stream, verify, cert - ) - - # Send the request. - send_kwargs = { - "timeout": timeout, - "allow_redirects": allow_redirects, - } - send_kwargs.update(settings) - resp = self.send(prep, **send_kwargs) - - return resp - - def get(self, url, **kwargs): - r"""Sends a GET request. Returns :class:`Response` object. - - :param url: URL for the new :class:`Request` object. - :param \*\*kwargs: Optional arguments that ``request`` takes. - :rtype: requests.Response - """ - - kwargs.setdefault("allow_redirects", True) - return self.request("GET", url, **kwargs) - - def options(self, url, **kwargs): - r"""Sends a OPTIONS request. Returns :class:`Response` object. - - :param url: URL for the new :class:`Request` object. - :param \*\*kwargs: Optional arguments that ``request`` takes. - :rtype: requests.Response - """ - - kwargs.setdefault("allow_redirects", True) - return self.request("OPTIONS", url, **kwargs) - - def head(self, url, **kwargs): - r"""Sends a HEAD request. Returns :class:`Response` object. - - :param url: URL for the new :class:`Request` object. - :param \*\*kwargs: Optional arguments that ``request`` takes. - :rtype: requests.Response - """ - - kwargs.setdefault("allow_redirects", False) - return self.request("HEAD", url, **kwargs) - - def post(self, url, data=None, json=None, **kwargs): - r"""Sends a POST request. Returns :class:`Response` object. - - :param url: URL for the new :class:`Request` object. - :param data: (optional) Dictionary, list of tuples, bytes, or file-like - object to send in the body of the :class:`Request`. - :param json: (optional) json to send in the body of the :class:`Request`. - :param \*\*kwargs: Optional arguments that ``request`` takes. - :rtype: requests.Response - """ - - return self.request("POST", url, data=data, json=json, **kwargs) - - def put(self, url, data=None, **kwargs): - r"""Sends a PUT request. Returns :class:`Response` object. - - :param url: URL for the new :class:`Request` object. - :param data: (optional) Dictionary, list of tuples, bytes, or file-like - object to send in the body of the :class:`Request`. - :param \*\*kwargs: Optional arguments that ``request`` takes. - :rtype: requests.Response - """ - - return self.request("PUT", url, data=data, **kwargs) - - def patch(self, url, data=None, **kwargs): - r"""Sends a PATCH request. Returns :class:`Response` object. - - :param url: URL for the new :class:`Request` object. - :param data: (optional) Dictionary, list of tuples, bytes, or file-like - object to send in the body of the :class:`Request`. - :param \*\*kwargs: Optional arguments that ``request`` takes. - :rtype: requests.Response - """ - - return self.request("PATCH", url, data=data, **kwargs) - - def delete(self, url, **kwargs): - r"""Sends a DELETE request. Returns :class:`Response` object. - - :param url: URL for the new :class:`Request` object. - :param \*\*kwargs: Optional arguments that ``request`` takes. - :rtype: requests.Response - """ - - return self.request("DELETE", url, **kwargs) - - def send(self, request, **kwargs): - """Send a given PreparedRequest. - - :rtype: requests.Response - """ - # Set defaults that the hooks can utilize to ensure they always have - # the correct parameters to reproduce the previous request. - kwargs.setdefault("stream", self.stream) - kwargs.setdefault("verify", self.verify) - kwargs.setdefault("cert", self.cert) - if "proxies" not in kwargs: - kwargs["proxies"] = resolve_proxies(request, self.proxies, self.trust_env) - - # It's possible that users might accidentally send a Request object. - # Guard against that specific failure case. - if isinstance(request, Request): - raise ValueError("You can only send PreparedRequests.") - - # Set up variables needed for resolve_redirects and dispatching of hooks - allow_redirects = kwargs.pop("allow_redirects", True) - stream = kwargs.get("stream") - hooks = request.hooks - - # Get the appropriate adapter to use - adapter = self.get_adapter(url=request.url) - - # Start time (approximately) of the request - start = preferred_clock() - - # Send the request - r = adapter.send(request, **kwargs) - - # Total elapsed time of the request (approximately) - elapsed = preferred_clock() - start - r.elapsed = timedelta(seconds=elapsed) - - # Response manipulation hooks - r = dispatch_hook("response", hooks, r, **kwargs) - - # Persist cookies - if r.history: - - # If the hooks create history then we want those cookies too - for resp in r.history: - extract_cookies_to_jar(self.cookies, resp.request, resp.raw) - - extract_cookies_to_jar(self.cookies, request, r.raw) - - # Resolve redirects if allowed. - if allow_redirects: - # Redirect resolving generator. - gen = self.resolve_redirects(r, request, **kwargs) - history = [resp for resp in gen] - else: - history = [] - - # Shuffle things around if there's history. - if history: - # Insert the first (original) request at the start - history.insert(0, r) - # Get the last request made - r = history.pop() - r.history = history - - # If redirects aren't being followed, store the response on the Request for Response.next(). - if not allow_redirects: - try: - r._next = next( - self.resolve_redirects(r, request, yield_requests=True, **kwargs) - ) - except StopIteration: - pass - - if not stream: - r.content - - return r - - def merge_environment_settings(self, url, proxies, stream, verify, cert): - """ - Check the environment and merge it with some settings. - - :rtype: dict - """ - # Gather clues from the surrounding environment. - if self.trust_env: - # Set environment's proxies. - no_proxy = proxies.get("no_proxy") if proxies is not None else None - env_proxies = get_environ_proxies(url, no_proxy=no_proxy) - for (k, v) in env_proxies.items(): - proxies.setdefault(k, v) - - # Look for requests environment configuration - # and be compatible with cURL. - if verify is True or verify is None: - verify = ( - os.environ.get("REQUESTS_CA_BUNDLE") - or os.environ.get("CURL_CA_BUNDLE") - or verify - ) - - # Merge all the kwargs. - proxies = merge_setting(proxies, self.proxies) - stream = merge_setting(stream, self.stream) - verify = merge_setting(verify, self.verify) - cert = merge_setting(cert, self.cert) - - return {"proxies": proxies, "stream": stream, "verify": verify, "cert": cert} - - def get_adapter(self, url): - """ - Returns the appropriate connection adapter for the given URL. - - :rtype: requests.adapters.BaseAdapter - """ - for (prefix, adapter) in self.adapters.items(): - - if url.lower().startswith(prefix.lower()): - return adapter - - # Nothing matches :-/ - raise InvalidSchema(f"No connection adapters were found for {url!r}") - - def close(self): - """Closes all adapters and as such the session""" - for v in self.adapters.values(): - v.close() - - def mount(self, prefix, adapter): - """Registers a connection adapter to a prefix. - - Adapters are sorted in descending order by prefix length. - """ - self.adapters[prefix] = adapter - keys_to_move = [k for k in self.adapters if len(k) < len(prefix)] - - for key in keys_to_move: - self.adapters[key] = self.adapters.pop(key) - - def __getstate__(self): - state = {attr: getattr(self, attr, None) for attr in self.__attrs__} - return state - - def __setstate__(self, state): - for attr, value in state.items(): - setattr(self, attr, value) - - -def session(): - """ - Returns a :class:`Session` for context-management. - - .. deprecated:: 1.0.0 - - This method has been deprecated since version 1.0.0 and is only kept for - backwards compatibility. New code should use :class:`~requests.sessions.Session` - to create a session. This may be removed at a future date. - - :rtype: Session - """ - return Session() diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/requests/status_codes.py b/venv/lib/python3.11/site-packages/pip/_vendor/requests/status_codes.py deleted file mode 100644 index 4bd072b..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/requests/status_codes.py +++ /dev/null @@ -1,128 +0,0 @@ -r""" -The ``codes`` object defines a mapping from common names for HTTP statuses -to their numerical codes, accessible either as attributes or as dictionary -items. - -Example:: - - >>> import requests - >>> requests.codes['temporary_redirect'] - 307 - >>> requests.codes.teapot - 418 - >>> requests.codes['\o/'] - 200 - -Some codes have multiple names, and both upper- and lower-case versions of -the names are allowed. For example, ``codes.ok``, ``codes.OK``, and -``codes.okay`` all correspond to the HTTP status code 200. -""" - -from .structures import LookupDict - -_codes = { - # Informational. - 100: ("continue",), - 101: ("switching_protocols",), - 102: ("processing",), - 103: ("checkpoint",), - 122: ("uri_too_long", "request_uri_too_long"), - 200: ("ok", "okay", "all_ok", "all_okay", "all_good", "\\o/", "✓"), - 201: ("created",), - 202: ("accepted",), - 203: ("non_authoritative_info", "non_authoritative_information"), - 204: ("no_content",), - 205: ("reset_content", "reset"), - 206: ("partial_content", "partial"), - 207: ("multi_status", "multiple_status", "multi_stati", "multiple_stati"), - 208: ("already_reported",), - 226: ("im_used",), - # Redirection. - 300: ("multiple_choices",), - 301: ("moved_permanently", "moved", "\\o-"), - 302: ("found",), - 303: ("see_other", "other"), - 304: ("not_modified",), - 305: ("use_proxy",), - 306: ("switch_proxy",), - 307: ("temporary_redirect", "temporary_moved", "temporary"), - 308: ( - "permanent_redirect", - "resume_incomplete", - "resume", - ), # "resume" and "resume_incomplete" to be removed in 3.0 - # Client Error. - 400: ("bad_request", "bad"), - 401: ("unauthorized",), - 402: ("payment_required", "payment"), - 403: ("forbidden",), - 404: ("not_found", "-o-"), - 405: ("method_not_allowed", "not_allowed"), - 406: ("not_acceptable",), - 407: ("proxy_authentication_required", "proxy_auth", "proxy_authentication"), - 408: ("request_timeout", "timeout"), - 409: ("conflict",), - 410: ("gone",), - 411: ("length_required",), - 412: ("precondition_failed", "precondition"), - 413: ("request_entity_too_large",), - 414: ("request_uri_too_large",), - 415: ("unsupported_media_type", "unsupported_media", "media_type"), - 416: ( - "requested_range_not_satisfiable", - "requested_range", - "range_not_satisfiable", - ), - 417: ("expectation_failed",), - 418: ("im_a_teapot", "teapot", "i_am_a_teapot"), - 421: ("misdirected_request",), - 422: ("unprocessable_entity", "unprocessable"), - 423: ("locked",), - 424: ("failed_dependency", "dependency"), - 425: ("unordered_collection", "unordered"), - 426: ("upgrade_required", "upgrade"), - 428: ("precondition_required", "precondition"), - 429: ("too_many_requests", "too_many"), - 431: ("header_fields_too_large", "fields_too_large"), - 444: ("no_response", "none"), - 449: ("retry_with", "retry"), - 450: ("blocked_by_windows_parental_controls", "parental_controls"), - 451: ("unavailable_for_legal_reasons", "legal_reasons"), - 499: ("client_closed_request",), - # Server Error. - 500: ("internal_server_error", "server_error", "/o\\", "✗"), - 501: ("not_implemented",), - 502: ("bad_gateway",), - 503: ("service_unavailable", "unavailable"), - 504: ("gateway_timeout",), - 505: ("http_version_not_supported", "http_version"), - 506: ("variant_also_negotiates",), - 507: ("insufficient_storage",), - 509: ("bandwidth_limit_exceeded", "bandwidth"), - 510: ("not_extended",), - 511: ("network_authentication_required", "network_auth", "network_authentication"), -} - -codes = LookupDict(name="status_codes") - - -def _init(): - for code, titles in _codes.items(): - for title in titles: - setattr(codes, title, code) - if not title.startswith(("\\", "/")): - setattr(codes, title.upper(), code) - - def doc(code): - names = ", ".join(f"``{n}``" for n in _codes[code]) - return "* %d: %s" % (code, names) - - global __doc__ - __doc__ = ( - __doc__ + "\n" + "\n".join(doc(code) for code in sorted(_codes)) - if __doc__ is not None - else None - ) - - -_init() diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/requests/structures.py b/venv/lib/python3.11/site-packages/pip/_vendor/requests/structures.py deleted file mode 100644 index 188e13e..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/requests/structures.py +++ /dev/null @@ -1,99 +0,0 @@ -""" -requests.structures -~~~~~~~~~~~~~~~~~~~ - -Data structures that power Requests. -""" - -from collections import OrderedDict - -from .compat import Mapping, MutableMapping - - -class CaseInsensitiveDict(MutableMapping): - """A case-insensitive ``dict``-like object. - - Implements all methods and operations of - ``MutableMapping`` as well as dict's ``copy``. Also - provides ``lower_items``. - - All keys are expected to be strings. The structure remembers the - case of the last key to be set, and ``iter(instance)``, - ``keys()``, ``items()``, ``iterkeys()``, and ``iteritems()`` - will contain case-sensitive keys. However, querying and contains - testing is case insensitive:: - - cid = CaseInsensitiveDict() - cid['Accept'] = 'application/json' - cid['aCCEPT'] == 'application/json' # True - list(cid) == ['Accept'] # True - - For example, ``headers['content-encoding']`` will return the - value of a ``'Content-Encoding'`` response header, regardless - of how the header name was originally stored. - - If the constructor, ``.update``, or equality comparison - operations are given keys that have equal ``.lower()``s, the - behavior is undefined. - """ - - def __init__(self, data=None, **kwargs): - self._store = OrderedDict() - if data is None: - data = {} - self.update(data, **kwargs) - - def __setitem__(self, key, value): - # Use the lowercased key for lookups, but store the actual - # key alongside the value. - self._store[key.lower()] = (key, value) - - def __getitem__(self, key): - return self._store[key.lower()][1] - - def __delitem__(self, key): - del self._store[key.lower()] - - def __iter__(self): - return (casedkey for casedkey, mappedvalue in self._store.values()) - - def __len__(self): - return len(self._store) - - def lower_items(self): - """Like iteritems(), but with all lowercase keys.""" - return ((lowerkey, keyval[1]) for (lowerkey, keyval) in self._store.items()) - - def __eq__(self, other): - if isinstance(other, Mapping): - other = CaseInsensitiveDict(other) - else: - return NotImplemented - # Compare insensitively - return dict(self.lower_items()) == dict(other.lower_items()) - - # Copy is required - def copy(self): - return CaseInsensitiveDict(self._store.values()) - - def __repr__(self): - return str(dict(self.items())) - - -class LookupDict(dict): - """Dictionary lookup object.""" - - def __init__(self, name=None): - self.name = name - super().__init__() - - def __repr__(self): - return f"" - - def __getitem__(self, key): - # We allow fall-through here, so values default to None - - return self.__dict__.get(key, None) - - def get(self, key, default=None): - return self.__dict__.get(key, default) diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/requests/utils.py b/venv/lib/python3.11/site-packages/pip/_vendor/requests/utils.py deleted file mode 100644 index 36607ed..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/requests/utils.py +++ /dev/null @@ -1,1094 +0,0 @@ -""" -requests.utils -~~~~~~~~~~~~~~ - -This module provides utility functions that are used within Requests -that are also useful for external consumption. -""" - -import codecs -import contextlib -import io -import os -import re -import socket -import struct -import sys -import tempfile -import warnings -import zipfile -from collections import OrderedDict - -from pip._vendor.urllib3.util import make_headers, parse_url - -from . import certs -from .__version__ import __version__ - -# to_native_string is unused here, but imported here for backwards compatibility -from ._internal_utils import ( # noqa: F401 - _HEADER_VALIDATORS_BYTE, - _HEADER_VALIDATORS_STR, - HEADER_VALIDATORS, - to_native_string, -) -from .compat import ( - Mapping, - basestring, - bytes, - getproxies, - getproxies_environment, - integer_types, -) -from .compat import parse_http_list as _parse_list_header -from .compat import ( - proxy_bypass, - proxy_bypass_environment, - quote, - str, - unquote, - urlparse, - urlunparse, -) -from .cookies import cookiejar_from_dict -from .exceptions import ( - FileModeWarning, - InvalidHeader, - InvalidURL, - UnrewindableBodyError, -) -from .structures import CaseInsensitiveDict - -NETRC_FILES = (".netrc", "_netrc") - -DEFAULT_CA_BUNDLE_PATH = certs.where() - -DEFAULT_PORTS = {"http": 80, "https": 443} - -# Ensure that ', ' is used to preserve previous delimiter behavior. -DEFAULT_ACCEPT_ENCODING = ", ".join( - re.split(r",\s*", make_headers(accept_encoding=True)["accept-encoding"]) -) - - -if sys.platform == "win32": - # provide a proxy_bypass version on Windows without DNS lookups - - def proxy_bypass_registry(host): - try: - import winreg - except ImportError: - return False - - try: - internetSettings = winreg.OpenKey( - winreg.HKEY_CURRENT_USER, - r"Software\Microsoft\Windows\CurrentVersion\Internet Settings", - ) - # ProxyEnable could be REG_SZ or REG_DWORD, normalizing it - proxyEnable = int(winreg.QueryValueEx(internetSettings, "ProxyEnable")[0]) - # ProxyOverride is almost always a string - proxyOverride = winreg.QueryValueEx(internetSettings, "ProxyOverride")[0] - except (OSError, ValueError): - return False - if not proxyEnable or not proxyOverride: - return False - - # make a check value list from the registry entry: replace the - # '' string by the localhost entry and the corresponding - # canonical entry. - proxyOverride = proxyOverride.split(";") - # now check if we match one of the registry values. - for test in proxyOverride: - if test == "": - if "." not in host: - return True - test = test.replace(".", r"\.") # mask dots - test = test.replace("*", r".*") # change glob sequence - test = test.replace("?", r".") # change glob char - if re.match(test, host, re.I): - return True - return False - - def proxy_bypass(host): # noqa - """Return True, if the host should be bypassed. - - Checks proxy settings gathered from the environment, if specified, - or the registry. - """ - if getproxies_environment(): - return proxy_bypass_environment(host) - else: - return proxy_bypass_registry(host) - - -def dict_to_sequence(d): - """Returns an internal sequence dictionary update.""" - - if hasattr(d, "items"): - d = d.items() - - return d - - -def super_len(o): - total_length = None - current_position = 0 - - if hasattr(o, "__len__"): - total_length = len(o) - - elif hasattr(o, "len"): - total_length = o.len - - elif hasattr(o, "fileno"): - try: - fileno = o.fileno() - except (io.UnsupportedOperation, AttributeError): - # AttributeError is a surprising exception, seeing as how we've just checked - # that `hasattr(o, 'fileno')`. It happens for objects obtained via - # `Tarfile.extractfile()`, per issue 5229. - pass - else: - total_length = os.fstat(fileno).st_size - - # Having used fstat to determine the file length, we need to - # confirm that this file was opened up in binary mode. - if "b" not in o.mode: - warnings.warn( - ( - "Requests has determined the content-length for this " - "request using the binary size of the file: however, the " - "file has been opened in text mode (i.e. without the 'b' " - "flag in the mode). This may lead to an incorrect " - "content-length. In Requests 3.0, support will be removed " - "for files in text mode." - ), - FileModeWarning, - ) - - if hasattr(o, "tell"): - try: - current_position = o.tell() - except OSError: - # This can happen in some weird situations, such as when the file - # is actually a special file descriptor like stdin. In this - # instance, we don't know what the length is, so set it to zero and - # let requests chunk it instead. - if total_length is not None: - current_position = total_length - else: - if hasattr(o, "seek") and total_length is None: - # StringIO and BytesIO have seek but no usable fileno - try: - # seek to end of file - o.seek(0, 2) - total_length = o.tell() - - # seek back to current position to support - # partially read file-like objects - o.seek(current_position or 0) - except OSError: - total_length = 0 - - if total_length is None: - total_length = 0 - - return max(0, total_length - current_position) - - -def get_netrc_auth(url, raise_errors=False): - """Returns the Requests tuple auth for a given url from netrc.""" - - netrc_file = os.environ.get("NETRC") - if netrc_file is not None: - netrc_locations = (netrc_file,) - else: - netrc_locations = (f"~/{f}" for f in NETRC_FILES) - - try: - from netrc import NetrcParseError, netrc - - netrc_path = None - - for f in netrc_locations: - try: - loc = os.path.expanduser(f) - except KeyError: - # os.path.expanduser can fail when $HOME is undefined and - # getpwuid fails. See https://bugs.python.org/issue20164 & - # https://github.com/psf/requests/issues/1846 - return - - if os.path.exists(loc): - netrc_path = loc - break - - # Abort early if there isn't one. - if netrc_path is None: - return - - ri = urlparse(url) - - # Strip port numbers from netloc. This weird `if...encode`` dance is - # used for Python 3.2, which doesn't support unicode literals. - splitstr = b":" - if isinstance(url, str): - splitstr = splitstr.decode("ascii") - host = ri.netloc.split(splitstr)[0] - - try: - _netrc = netrc(netrc_path).authenticators(host) - if _netrc: - # Return with login / password - login_i = 0 if _netrc[0] else 1 - return (_netrc[login_i], _netrc[2]) - except (NetrcParseError, OSError): - # If there was a parsing error or a permissions issue reading the file, - # we'll just skip netrc auth unless explicitly asked to raise errors. - if raise_errors: - raise - - # App Engine hackiness. - except (ImportError, AttributeError): - pass - - -def guess_filename(obj): - """Tries to guess the filename of the given object.""" - name = getattr(obj, "name", None) - if name and isinstance(name, basestring) and name[0] != "<" and name[-1] != ">": - return os.path.basename(name) - - -def extract_zipped_paths(path): - """Replace nonexistent paths that look like they refer to a member of a zip - archive with the location of an extracted copy of the target, or else - just return the provided path unchanged. - """ - if os.path.exists(path): - # this is already a valid path, no need to do anything further - return path - - # find the first valid part of the provided path and treat that as a zip archive - # assume the rest of the path is the name of a member in the archive - archive, member = os.path.split(path) - while archive and not os.path.exists(archive): - archive, prefix = os.path.split(archive) - if not prefix: - # If we don't check for an empty prefix after the split (in other words, archive remains unchanged after the split), - # we _can_ end up in an infinite loop on a rare corner case affecting a small number of users - break - member = "/".join([prefix, member]) - - if not zipfile.is_zipfile(archive): - return path - - zip_file = zipfile.ZipFile(archive) - if member not in zip_file.namelist(): - return path - - # we have a valid zip archive and a valid member of that archive - tmp = tempfile.gettempdir() - extracted_path = os.path.join(tmp, member.split("/")[-1]) - if not os.path.exists(extracted_path): - # use read + write to avoid the creating nested folders, we only want the file, avoids mkdir racing condition - with atomic_open(extracted_path) as file_handler: - file_handler.write(zip_file.read(member)) - return extracted_path - - -@contextlib.contextmanager -def atomic_open(filename): - """Write a file to the disk in an atomic fashion""" - tmp_descriptor, tmp_name = tempfile.mkstemp(dir=os.path.dirname(filename)) - try: - with os.fdopen(tmp_descriptor, "wb") as tmp_handler: - yield tmp_handler - os.replace(tmp_name, filename) - except BaseException: - os.remove(tmp_name) - raise - - -def from_key_val_list(value): - """Take an object and test to see if it can be represented as a - dictionary. Unless it can not be represented as such, return an - OrderedDict, e.g., - - :: - - >>> from_key_val_list([('key', 'val')]) - OrderedDict([('key', 'val')]) - >>> from_key_val_list('string') - Traceback (most recent call last): - ... - ValueError: cannot encode objects that are not 2-tuples - >>> from_key_val_list({'key': 'val'}) - OrderedDict([('key', 'val')]) - - :rtype: OrderedDict - """ - if value is None: - return None - - if isinstance(value, (str, bytes, bool, int)): - raise ValueError("cannot encode objects that are not 2-tuples") - - return OrderedDict(value) - - -def to_key_val_list(value): - """Take an object and test to see if it can be represented as a - dictionary. If it can be, return a list of tuples, e.g., - - :: - - >>> to_key_val_list([('key', 'val')]) - [('key', 'val')] - >>> to_key_val_list({'key': 'val'}) - [('key', 'val')] - >>> to_key_val_list('string') - Traceback (most recent call last): - ... - ValueError: cannot encode objects that are not 2-tuples - - :rtype: list - """ - if value is None: - return None - - if isinstance(value, (str, bytes, bool, int)): - raise ValueError("cannot encode objects that are not 2-tuples") - - if isinstance(value, Mapping): - value = value.items() - - return list(value) - - -# From mitsuhiko/werkzeug (used with permission). -def parse_list_header(value): - """Parse lists as described by RFC 2068 Section 2. - - In particular, parse comma-separated lists where the elements of - the list may include quoted-strings. A quoted-string could - contain a comma. A non-quoted string could have quotes in the - middle. Quotes are removed automatically after parsing. - - It basically works like :func:`parse_set_header` just that items - may appear multiple times and case sensitivity is preserved. - - The return value is a standard :class:`list`: - - >>> parse_list_header('token, "quoted value"') - ['token', 'quoted value'] - - To create a header from the :class:`list` again, use the - :func:`dump_header` function. - - :param value: a string with a list header. - :return: :class:`list` - :rtype: list - """ - result = [] - for item in _parse_list_header(value): - if item[:1] == item[-1:] == '"': - item = unquote_header_value(item[1:-1]) - result.append(item) - return result - - -# From mitsuhiko/werkzeug (used with permission). -def parse_dict_header(value): - """Parse lists of key, value pairs as described by RFC 2068 Section 2 and - convert them into a python dict: - - >>> d = parse_dict_header('foo="is a fish", bar="as well"') - >>> type(d) is dict - True - >>> sorted(d.items()) - [('bar', 'as well'), ('foo', 'is a fish')] - - If there is no value for a key it will be `None`: - - >>> parse_dict_header('key_without_value') - {'key_without_value': None} - - To create a header from the :class:`dict` again, use the - :func:`dump_header` function. - - :param value: a string with a dict header. - :return: :class:`dict` - :rtype: dict - """ - result = {} - for item in _parse_list_header(value): - if "=" not in item: - result[item] = None - continue - name, value = item.split("=", 1) - if value[:1] == value[-1:] == '"': - value = unquote_header_value(value[1:-1]) - result[name] = value - return result - - -# From mitsuhiko/werkzeug (used with permission). -def unquote_header_value(value, is_filename=False): - r"""Unquotes a header value. (Reversal of :func:`quote_header_value`). - This does not use the real unquoting but what browsers are actually - using for quoting. - - :param value: the header value to unquote. - :rtype: str - """ - if value and value[0] == value[-1] == '"': - # this is not the real unquoting, but fixing this so that the - # RFC is met will result in bugs with internet explorer and - # probably some other browsers as well. IE for example is - # uploading files with "C:\foo\bar.txt" as filename - value = value[1:-1] - - # if this is a filename and the starting characters look like - # a UNC path, then just return the value without quotes. Using the - # replace sequence below on a UNC path has the effect of turning - # the leading double slash into a single slash and then - # _fix_ie_filename() doesn't work correctly. See #458. - if not is_filename or value[:2] != "\\\\": - return value.replace("\\\\", "\\").replace('\\"', '"') - return value - - -def dict_from_cookiejar(cj): - """Returns a key/value dictionary from a CookieJar. - - :param cj: CookieJar object to extract cookies from. - :rtype: dict - """ - - cookie_dict = {} - - for cookie in cj: - cookie_dict[cookie.name] = cookie.value - - return cookie_dict - - -def add_dict_to_cookiejar(cj, cookie_dict): - """Returns a CookieJar from a key/value dictionary. - - :param cj: CookieJar to insert cookies into. - :param cookie_dict: Dict of key/values to insert into CookieJar. - :rtype: CookieJar - """ - - return cookiejar_from_dict(cookie_dict, cj) - - -def get_encodings_from_content(content): - """Returns encodings from given content string. - - :param content: bytestring to extract encodings from. - """ - warnings.warn( - ( - "In requests 3.0, get_encodings_from_content will be removed. For " - "more information, please see the discussion on issue #2266. (This" - " warning should only appear once.)" - ), - DeprecationWarning, - ) - - charset_re = re.compile(r']', flags=re.I) - pragma_re = re.compile(r']', flags=re.I) - xml_re = re.compile(r'^<\?xml.*?encoding=["\']*(.+?)["\'>]') - - return ( - charset_re.findall(content) - + pragma_re.findall(content) - + xml_re.findall(content) - ) - - -def _parse_content_type_header(header): - """Returns content type and parameters from given header - - :param header: string - :return: tuple containing content type and dictionary of - parameters - """ - - tokens = header.split(";") - content_type, params = tokens[0].strip(), tokens[1:] - params_dict = {} - items_to_strip = "\"' " - - for param in params: - param = param.strip() - if param: - key, value = param, True - index_of_equals = param.find("=") - if index_of_equals != -1: - key = param[:index_of_equals].strip(items_to_strip) - value = param[index_of_equals + 1 :].strip(items_to_strip) - params_dict[key.lower()] = value - return content_type, params_dict - - -def get_encoding_from_headers(headers): - """Returns encodings from given HTTP Header Dict. - - :param headers: dictionary to extract encoding from. - :rtype: str - """ - - content_type = headers.get("content-type") - - if not content_type: - return None - - content_type, params = _parse_content_type_header(content_type) - - if "charset" in params: - return params["charset"].strip("'\"") - - if "text" in content_type: - return "ISO-8859-1" - - if "application/json" in content_type: - # Assume UTF-8 based on RFC 4627: https://www.ietf.org/rfc/rfc4627.txt since the charset was unset - return "utf-8" - - -def stream_decode_response_unicode(iterator, r): - """Stream decodes an iterator.""" - - if r.encoding is None: - yield from iterator - return - - decoder = codecs.getincrementaldecoder(r.encoding)(errors="replace") - for chunk in iterator: - rv = decoder.decode(chunk) - if rv: - yield rv - rv = decoder.decode(b"", final=True) - if rv: - yield rv - - -def iter_slices(string, slice_length): - """Iterate over slices of a string.""" - pos = 0 - if slice_length is None or slice_length <= 0: - slice_length = len(string) - while pos < len(string): - yield string[pos : pos + slice_length] - pos += slice_length - - -def get_unicode_from_response(r): - """Returns the requested content back in unicode. - - :param r: Response object to get unicode content from. - - Tried: - - 1. charset from content-type - 2. fall back and replace all unicode characters - - :rtype: str - """ - warnings.warn( - ( - "In requests 3.0, get_unicode_from_response will be removed. For " - "more information, please see the discussion on issue #2266. (This" - " warning should only appear once.)" - ), - DeprecationWarning, - ) - - tried_encodings = [] - - # Try charset from content-type - encoding = get_encoding_from_headers(r.headers) - - if encoding: - try: - return str(r.content, encoding) - except UnicodeError: - tried_encodings.append(encoding) - - # Fall back: - try: - return str(r.content, encoding, errors="replace") - except TypeError: - return r.content - - -# The unreserved URI characters (RFC 3986) -UNRESERVED_SET = frozenset( - "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz" + "0123456789-._~" -) - - -def unquote_unreserved(uri): - """Un-escape any percent-escape sequences in a URI that are unreserved - characters. This leaves all reserved, illegal and non-ASCII bytes encoded. - - :rtype: str - """ - parts = uri.split("%") - for i in range(1, len(parts)): - h = parts[i][0:2] - if len(h) == 2 and h.isalnum(): - try: - c = chr(int(h, 16)) - except ValueError: - raise InvalidURL(f"Invalid percent-escape sequence: '{h}'") - - if c in UNRESERVED_SET: - parts[i] = c + parts[i][2:] - else: - parts[i] = f"%{parts[i]}" - else: - parts[i] = f"%{parts[i]}" - return "".join(parts) - - -def requote_uri(uri): - """Re-quote the given URI. - - This function passes the given URI through an unquote/quote cycle to - ensure that it is fully and consistently quoted. - - :rtype: str - """ - safe_with_percent = "!#$%&'()*+,/:;=?@[]~" - safe_without_percent = "!#$&'()*+,/:;=?@[]~" - try: - # Unquote only the unreserved characters - # Then quote only illegal characters (do not quote reserved, - # unreserved, or '%') - return quote(unquote_unreserved(uri), safe=safe_with_percent) - except InvalidURL: - # We couldn't unquote the given URI, so let's try quoting it, but - # there may be unquoted '%'s in the URI. We need to make sure they're - # properly quoted so they do not cause issues elsewhere. - return quote(uri, safe=safe_without_percent) - - -def address_in_network(ip, net): - """This function allows you to check if an IP belongs to a network subnet - - Example: returns True if ip = 192.168.1.1 and net = 192.168.1.0/24 - returns False if ip = 192.168.1.1 and net = 192.168.100.0/24 - - :rtype: bool - """ - ipaddr = struct.unpack("=L", socket.inet_aton(ip))[0] - netaddr, bits = net.split("/") - netmask = struct.unpack("=L", socket.inet_aton(dotted_netmask(int(bits))))[0] - network = struct.unpack("=L", socket.inet_aton(netaddr))[0] & netmask - return (ipaddr & netmask) == (network & netmask) - - -def dotted_netmask(mask): - """Converts mask from /xx format to xxx.xxx.xxx.xxx - - Example: if mask is 24 function returns 255.255.255.0 - - :rtype: str - """ - bits = 0xFFFFFFFF ^ (1 << 32 - mask) - 1 - return socket.inet_ntoa(struct.pack(">I", bits)) - - -def is_ipv4_address(string_ip): - """ - :rtype: bool - """ - try: - socket.inet_aton(string_ip) - except OSError: - return False - return True - - -def is_valid_cidr(string_network): - """ - Very simple check of the cidr format in no_proxy variable. - - :rtype: bool - """ - if string_network.count("/") == 1: - try: - mask = int(string_network.split("/")[1]) - except ValueError: - return False - - if mask < 1 or mask > 32: - return False - - try: - socket.inet_aton(string_network.split("/")[0]) - except OSError: - return False - else: - return False - return True - - -@contextlib.contextmanager -def set_environ(env_name, value): - """Set the environment variable 'env_name' to 'value' - - Save previous value, yield, and then restore the previous value stored in - the environment variable 'env_name'. - - If 'value' is None, do nothing""" - value_changed = value is not None - if value_changed: - old_value = os.environ.get(env_name) - os.environ[env_name] = value - try: - yield - finally: - if value_changed: - if old_value is None: - del os.environ[env_name] - else: - os.environ[env_name] = old_value - - -def should_bypass_proxies(url, no_proxy): - """ - Returns whether we should bypass proxies or not. - - :rtype: bool - """ - # Prioritize lowercase environment variables over uppercase - # to keep a consistent behaviour with other http projects (curl, wget). - def get_proxy(key): - return os.environ.get(key) or os.environ.get(key.upper()) - - # First check whether no_proxy is defined. If it is, check that the URL - # we're getting isn't in the no_proxy list. - no_proxy_arg = no_proxy - if no_proxy is None: - no_proxy = get_proxy("no_proxy") - parsed = urlparse(url) - - if parsed.hostname is None: - # URLs don't always have hostnames, e.g. file:/// urls. - return True - - if no_proxy: - # We need to check whether we match here. We need to see if we match - # the end of the hostname, both with and without the port. - no_proxy = (host for host in no_proxy.replace(" ", "").split(",") if host) - - if is_ipv4_address(parsed.hostname): - for proxy_ip in no_proxy: - if is_valid_cidr(proxy_ip): - if address_in_network(parsed.hostname, proxy_ip): - return True - elif parsed.hostname == proxy_ip: - # If no_proxy ip was defined in plain IP notation instead of cidr notation & - # matches the IP of the index - return True - else: - host_with_port = parsed.hostname - if parsed.port: - host_with_port += f":{parsed.port}" - - for host in no_proxy: - if parsed.hostname.endswith(host) or host_with_port.endswith(host): - # The URL does match something in no_proxy, so we don't want - # to apply the proxies on this URL. - return True - - with set_environ("no_proxy", no_proxy_arg): - # parsed.hostname can be `None` in cases such as a file URI. - try: - bypass = proxy_bypass(parsed.hostname) - except (TypeError, socket.gaierror): - bypass = False - - if bypass: - return True - - return False - - -def get_environ_proxies(url, no_proxy=None): - """ - Return a dict of environment proxies. - - :rtype: dict - """ - if should_bypass_proxies(url, no_proxy=no_proxy): - return {} - else: - return getproxies() - - -def select_proxy(url, proxies): - """Select a proxy for the url, if applicable. - - :param url: The url being for the request - :param proxies: A dictionary of schemes or schemes and hosts to proxy URLs - """ - proxies = proxies or {} - urlparts = urlparse(url) - if urlparts.hostname is None: - return proxies.get(urlparts.scheme, proxies.get("all")) - - proxy_keys = [ - urlparts.scheme + "://" + urlparts.hostname, - urlparts.scheme, - "all://" + urlparts.hostname, - "all", - ] - proxy = None - for proxy_key in proxy_keys: - if proxy_key in proxies: - proxy = proxies[proxy_key] - break - - return proxy - - -def resolve_proxies(request, proxies, trust_env=True): - """This method takes proxy information from a request and configuration - input to resolve a mapping of target proxies. This will consider settings - such a NO_PROXY to strip proxy configurations. - - :param request: Request or PreparedRequest - :param proxies: A dictionary of schemes or schemes and hosts to proxy URLs - :param trust_env: Boolean declaring whether to trust environment configs - - :rtype: dict - """ - proxies = proxies if proxies is not None else {} - url = request.url - scheme = urlparse(url).scheme - no_proxy = proxies.get("no_proxy") - new_proxies = proxies.copy() - - if trust_env and not should_bypass_proxies(url, no_proxy=no_proxy): - environ_proxies = get_environ_proxies(url, no_proxy=no_proxy) - - proxy = environ_proxies.get(scheme, environ_proxies.get("all")) - - if proxy: - new_proxies.setdefault(scheme, proxy) - return new_proxies - - -def default_user_agent(name="python-requests"): - """ - Return a string representing the default user agent. - - :rtype: str - """ - return f"{name}/{__version__}" - - -def default_headers(): - """ - :rtype: requests.structures.CaseInsensitiveDict - """ - return CaseInsensitiveDict( - { - "User-Agent": default_user_agent(), - "Accept-Encoding": DEFAULT_ACCEPT_ENCODING, - "Accept": "*/*", - "Connection": "keep-alive", - } - ) - - -def parse_header_links(value): - """Return a list of parsed link headers proxies. - - i.e. Link: ; rel=front; type="image/jpeg",; rel=back;type="image/jpeg" - - :rtype: list - """ - - links = [] - - replace_chars = " '\"" - - value = value.strip(replace_chars) - if not value: - return links - - for val in re.split(", *<", value): - try: - url, params = val.split(";", 1) - except ValueError: - url, params = val, "" - - link = {"url": url.strip("<> '\"")} - - for param in params.split(";"): - try: - key, value = param.split("=") - except ValueError: - break - - link[key.strip(replace_chars)] = value.strip(replace_chars) - - links.append(link) - - return links - - -# Null bytes; no need to recreate these on each call to guess_json_utf -_null = "\x00".encode("ascii") # encoding to ASCII for Python 3 -_null2 = _null * 2 -_null3 = _null * 3 - - -def guess_json_utf(data): - """ - :rtype: str - """ - # JSON always starts with two ASCII characters, so detection is as - # easy as counting the nulls and from their location and count - # determine the encoding. Also detect a BOM, if present. - sample = data[:4] - if sample in (codecs.BOM_UTF32_LE, codecs.BOM_UTF32_BE): - return "utf-32" # BOM included - if sample[:3] == codecs.BOM_UTF8: - return "utf-8-sig" # BOM included, MS style (discouraged) - if sample[:2] in (codecs.BOM_UTF16_LE, codecs.BOM_UTF16_BE): - return "utf-16" # BOM included - nullcount = sample.count(_null) - if nullcount == 0: - return "utf-8" - if nullcount == 2: - if sample[::2] == _null2: # 1st and 3rd are null - return "utf-16-be" - if sample[1::2] == _null2: # 2nd and 4th are null - return "utf-16-le" - # Did not detect 2 valid UTF-16 ascii-range characters - if nullcount == 3: - if sample[:3] == _null3: - return "utf-32-be" - if sample[1:] == _null3: - return "utf-32-le" - # Did not detect a valid UTF-32 ascii-range character - return None - - -def prepend_scheme_if_needed(url, new_scheme): - """Given a URL that may or may not have a scheme, prepend the given scheme. - Does not replace a present scheme with the one provided as an argument. - - :rtype: str - """ - parsed = parse_url(url) - scheme, auth, host, port, path, query, fragment = parsed - - # A defect in urlparse determines that there isn't a netloc present in some - # urls. We previously assumed parsing was overly cautious, and swapped the - # netloc and path. Due to a lack of tests on the original defect, this is - # maintained with parse_url for backwards compatibility. - netloc = parsed.netloc - if not netloc: - netloc, path = path, netloc - - if auth: - # parse_url doesn't provide the netloc with auth - # so we'll add it ourselves. - netloc = "@".join([auth, netloc]) - if scheme is None: - scheme = new_scheme - if path is None: - path = "" - - return urlunparse((scheme, netloc, path, "", query, fragment)) - - -def get_auth_from_url(url): - """Given a url with authentication components, extract them into a tuple of - username,password. - - :rtype: (str,str) - """ - parsed = urlparse(url) - - try: - auth = (unquote(parsed.username), unquote(parsed.password)) - except (AttributeError, TypeError): - auth = ("", "") - - return auth - - -def check_header_validity(header): - """Verifies that header parts don't contain leading whitespace - reserved characters, or return characters. - - :param header: tuple, in the format (name, value). - """ - name, value = header - _validate_header_part(header, name, 0) - _validate_header_part(header, value, 1) - - -def _validate_header_part(header, header_part, header_validator_index): - if isinstance(header_part, str): - validator = _HEADER_VALIDATORS_STR[header_validator_index] - elif isinstance(header_part, bytes): - validator = _HEADER_VALIDATORS_BYTE[header_validator_index] - else: - raise InvalidHeader( - f"Header part ({header_part!r}) from {header} " - f"must be of type str or bytes, not {type(header_part)}" - ) - - if not validator.match(header_part): - header_kind = "name" if header_validator_index == 0 else "value" - raise InvalidHeader( - f"Invalid leading whitespace, reserved character(s), or return" - f"character(s) in header {header_kind}: {header_part!r}" - ) - - -def urldefragauth(url): - """ - Given a url remove the fragment and the authentication part. - - :rtype: str - """ - scheme, netloc, path, params, query, fragment = urlparse(url) - - # see func:`prepend_scheme_if_needed` - if not netloc: - netloc, path = path, netloc - - netloc = netloc.rsplit("@", 1)[-1] - - return urlunparse((scheme, netloc, path, params, query, "")) - - -def rewind_body(prepared_request): - """Move file pointer back to its recorded starting position - so it can be read again on redirect. - """ - body_seek = getattr(prepared_request.body, "seek", None) - if body_seek is not None and isinstance( - prepared_request._body_position, integer_types - ): - try: - body_seek(prepared_request._body_position) - except OSError: - raise UnrewindableBodyError( - "An error occurred when rewinding request body for redirect." - ) - else: - raise UnrewindableBodyError("Unable to rewind request body for redirect.") diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/resolvelib/__init__.py b/venv/lib/python3.11/site-packages/pip/_vendor/resolvelib/__init__.py deleted file mode 100644 index d92acc7..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/resolvelib/__init__.py +++ /dev/null @@ -1,26 +0,0 @@ -__all__ = [ - "__version__", - "AbstractProvider", - "AbstractResolver", - "BaseReporter", - "InconsistentCandidate", - "Resolver", - "RequirementsConflicted", - "ResolutionError", - "ResolutionImpossible", - "ResolutionTooDeep", -] - -__version__ = "1.0.1" - - -from .providers import AbstractProvider, AbstractResolver -from .reporters import BaseReporter -from .resolvers import ( - InconsistentCandidate, - RequirementsConflicted, - ResolutionError, - ResolutionImpossible, - ResolutionTooDeep, - Resolver, -) diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/resolvelib/__pycache__/__init__.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/resolvelib/__pycache__/__init__.cpython-311.pyc deleted file mode 100644 index e401dd9..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/resolvelib/__pycache__/__init__.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/resolvelib/__pycache__/providers.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/resolvelib/__pycache__/providers.cpython-311.pyc deleted file mode 100644 index 8a317b9..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/resolvelib/__pycache__/providers.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/resolvelib/__pycache__/reporters.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/resolvelib/__pycache__/reporters.cpython-311.pyc deleted file mode 100644 index 309542e..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/resolvelib/__pycache__/reporters.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/resolvelib/__pycache__/resolvers.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/resolvelib/__pycache__/resolvers.cpython-311.pyc deleted file mode 100644 index 14584e7..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/resolvelib/__pycache__/resolvers.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/resolvelib/__pycache__/structs.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/resolvelib/__pycache__/structs.cpython-311.pyc deleted file mode 100644 index 9caf963..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/resolvelib/__pycache__/structs.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/resolvelib/compat/__init__.py b/venv/lib/python3.11/site-packages/pip/_vendor/resolvelib/compat/__init__.py deleted file mode 100644 index e69de29..0000000 diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/resolvelib/compat/__pycache__/__init__.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/resolvelib/compat/__pycache__/__init__.cpython-311.pyc deleted file mode 100644 index 472f0af..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/resolvelib/compat/__pycache__/__init__.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/resolvelib/compat/__pycache__/collections_abc.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/resolvelib/compat/__pycache__/collections_abc.cpython-311.pyc deleted file mode 100644 index e27cd37..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/resolvelib/compat/__pycache__/collections_abc.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/resolvelib/compat/collections_abc.py b/venv/lib/python3.11/site-packages/pip/_vendor/resolvelib/compat/collections_abc.py deleted file mode 100644 index 1becc50..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/resolvelib/compat/collections_abc.py +++ /dev/null @@ -1,6 +0,0 @@ -__all__ = ["Mapping", "Sequence"] - -try: - from collections.abc import Mapping, Sequence -except ImportError: - from collections import Mapping, Sequence diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/resolvelib/providers.py b/venv/lib/python3.11/site-packages/pip/_vendor/resolvelib/providers.py deleted file mode 100644 index e99d87e..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/resolvelib/providers.py +++ /dev/null @@ -1,133 +0,0 @@ -class AbstractProvider(object): - """Delegate class to provide the required interface for the resolver.""" - - def identify(self, requirement_or_candidate): - """Given a requirement, return an identifier for it. - - This is used to identify a requirement, e.g. whether two requirements - should have their specifier parts merged. - """ - raise NotImplementedError - - def get_preference( - self, - identifier, - resolutions, - candidates, - information, - backtrack_causes, - ): - """Produce a sort key for given requirement based on preference. - - The preference is defined as "I think this requirement should be - resolved first". The lower the return value is, the more preferred - this group of arguments is. - - :param identifier: An identifier as returned by ``identify()``. This - identifies the dependency matches which should be returned. - :param resolutions: Mapping of candidates currently pinned by the - resolver. Each key is an identifier, and the value is a candidate. - The candidate may conflict with requirements from ``information``. - :param candidates: Mapping of each dependency's possible candidates. - Each value is an iterator of candidates. - :param information: Mapping of requirement information of each package. - Each value is an iterator of *requirement information*. - :param backtrack_causes: Sequence of requirement information that were - the requirements that caused the resolver to most recently backtrack. - - A *requirement information* instance is a named tuple with two members: - - * ``requirement`` specifies a requirement contributing to the current - list of candidates. - * ``parent`` specifies the candidate that provides (depended on) the - requirement, or ``None`` to indicate a root requirement. - - The preference could depend on various issues, including (not - necessarily in this order): - - * Is this package pinned in the current resolution result? - * How relaxed is the requirement? Stricter ones should probably be - worked on first? (I don't know, actually.) - * How many possibilities are there to satisfy this requirement? Those - with few left should likely be worked on first, I guess? - * Are there any known conflicts for this requirement? We should - probably work on those with the most known conflicts. - - A sortable value should be returned (this will be used as the ``key`` - parameter of the built-in sorting function). The smaller the value is, - the more preferred this requirement is (i.e. the sorting function - is called with ``reverse=False``). - """ - raise NotImplementedError - - def find_matches(self, identifier, requirements, incompatibilities): - """Find all possible candidates that satisfy the given constraints. - - :param identifier: An identifier as returned by ``identify()``. This - identifies the dependency matches of which should be returned. - :param requirements: A mapping of requirements that all returned - candidates must satisfy. Each key is an identifier, and the value - an iterator of requirements for that dependency. - :param incompatibilities: A mapping of known incompatibilities of - each dependency. Each key is an identifier, and the value an - iterator of incompatibilities known to the resolver. All - incompatibilities *must* be excluded from the return value. - - This should try to get candidates based on the requirements' types. - For VCS, local, and archive requirements, the one-and-only match is - returned, and for a "named" requirement, the index(es) should be - consulted to find concrete candidates for this requirement. - - The return value should produce candidates ordered by preference; the - most preferred candidate should come first. The return type may be one - of the following: - - * A callable that returns an iterator that yields candidates. - * An collection of candidates. - * An iterable of candidates. This will be consumed immediately into a - list of candidates. - """ - raise NotImplementedError - - def is_satisfied_by(self, requirement, candidate): - """Whether the given requirement can be satisfied by a candidate. - - The candidate is guaranteed to have been generated from the - requirement. - - A boolean should be returned to indicate whether ``candidate`` is a - viable solution to the requirement. - """ - raise NotImplementedError - - def get_dependencies(self, candidate): - """Get dependencies of a candidate. - - This should return a collection of requirements that `candidate` - specifies as its dependencies. - """ - raise NotImplementedError - - -class AbstractResolver(object): - """The thing that performs the actual resolution work.""" - - base_exception = Exception - - def __init__(self, provider, reporter): - self.provider = provider - self.reporter = reporter - - def resolve(self, requirements, **kwargs): - """Take a collection of constraints, spit out the resolution result. - - This returns a representation of the final resolution state, with one - guarenteed attribute ``mapping`` that contains resolved candidates as - values. The keys are their respective identifiers. - - :param requirements: A collection of constraints. - :param kwargs: Additional keyword arguments that subclasses may accept. - - :raises: ``self.base_exception`` or its subclass. - """ - raise NotImplementedError diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/resolvelib/reporters.py b/venv/lib/python3.11/site-packages/pip/_vendor/resolvelib/reporters.py deleted file mode 100644 index 688b5e1..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/resolvelib/reporters.py +++ /dev/null @@ -1,43 +0,0 @@ -class BaseReporter(object): - """Delegate class to provider progress reporting for the resolver.""" - - def starting(self): - """Called before the resolution actually starts.""" - - def starting_round(self, index): - """Called before each round of resolution starts. - - The index is zero-based. - """ - - def ending_round(self, index, state): - """Called before each round of resolution ends. - - This is NOT called if the resolution ends at this round. Use `ending` - if you want to report finalization. The index is zero-based. - """ - - def ending(self, state): - """Called before the resolution ends successfully.""" - - def adding_requirement(self, requirement, parent): - """Called when adding a new requirement into the resolve criteria. - - :param requirement: The additional requirement to be applied to filter - the available candidaites. - :param parent: The candidate that requires ``requirement`` as a - dependency, or None if ``requirement`` is one of the root - requirements passed in from ``Resolver.resolve()``. - """ - - def resolving_conflicts(self, causes): - """Called when starting to attempt requirement conflict resolution. - - :param causes: The information on the collision that caused the backtracking. - """ - - def rejecting_candidate(self, criterion, candidate): - """Called when rejecting a candidate during backtracking.""" - - def pinning(self, candidate): - """Called when adding a candidate to the potential solution.""" diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/resolvelib/resolvers.py b/venv/lib/python3.11/site-packages/pip/_vendor/resolvelib/resolvers.py deleted file mode 100644 index 2c3d0e3..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/resolvelib/resolvers.py +++ /dev/null @@ -1,547 +0,0 @@ -import collections -import itertools -import operator - -from .providers import AbstractResolver -from .structs import DirectedGraph, IteratorMapping, build_iter_view - -RequirementInformation = collections.namedtuple( - "RequirementInformation", ["requirement", "parent"] -) - - -class ResolverException(Exception): - """A base class for all exceptions raised by this module. - - Exceptions derived by this class should all be handled in this module. Any - bubbling pass the resolver should be treated as a bug. - """ - - -class RequirementsConflicted(ResolverException): - def __init__(self, criterion): - super(RequirementsConflicted, self).__init__(criterion) - self.criterion = criterion - - def __str__(self): - return "Requirements conflict: {}".format( - ", ".join(repr(r) for r in self.criterion.iter_requirement()), - ) - - -class InconsistentCandidate(ResolverException): - def __init__(self, candidate, criterion): - super(InconsistentCandidate, self).__init__(candidate, criterion) - self.candidate = candidate - self.criterion = criterion - - def __str__(self): - return "Provided candidate {!r} does not satisfy {}".format( - self.candidate, - ", ".join(repr(r) for r in self.criterion.iter_requirement()), - ) - - -class Criterion(object): - """Representation of possible resolution results of a package. - - This holds three attributes: - - * `information` is a collection of `RequirementInformation` pairs. - Each pair is a requirement contributing to this criterion, and the - candidate that provides the requirement. - * `incompatibilities` is a collection of all known not-to-work candidates - to exclude from consideration. - * `candidates` is a collection containing all possible candidates deducted - from the union of contributing requirements and known incompatibilities. - It should never be empty, except when the criterion is an attribute of a - raised `RequirementsConflicted` (in which case it is always empty). - - .. note:: - This class is intended to be externally immutable. **Do not** mutate - any of its attribute containers. - """ - - def __init__(self, candidates, information, incompatibilities): - self.candidates = candidates - self.information = information - self.incompatibilities = incompatibilities - - def __repr__(self): - requirements = ", ".join( - "({!r}, via={!r})".format(req, parent) - for req, parent in self.information - ) - return "Criterion({})".format(requirements) - - def iter_requirement(self): - return (i.requirement for i in self.information) - - def iter_parent(self): - return (i.parent for i in self.information) - - -class ResolutionError(ResolverException): - pass - - -class ResolutionImpossible(ResolutionError): - def __init__(self, causes): - super(ResolutionImpossible, self).__init__(causes) - # causes is a list of RequirementInformation objects - self.causes = causes - - -class ResolutionTooDeep(ResolutionError): - def __init__(self, round_count): - super(ResolutionTooDeep, self).__init__(round_count) - self.round_count = round_count - - -# Resolution state in a round. -State = collections.namedtuple("State", "mapping criteria backtrack_causes") - - -class Resolution(object): - """Stateful resolution object. - - This is designed as a one-off object that holds information to kick start - the resolution process, and holds the results afterwards. - """ - - def __init__(self, provider, reporter): - self._p = provider - self._r = reporter - self._states = [] - - @property - def state(self): - try: - return self._states[-1] - except IndexError: - raise AttributeError("state") - - def _push_new_state(self): - """Push a new state into history. - - This new state will be used to hold resolution results of the next - coming round. - """ - base = self._states[-1] - state = State( - mapping=base.mapping.copy(), - criteria=base.criteria.copy(), - backtrack_causes=base.backtrack_causes[:], - ) - self._states.append(state) - - def _add_to_criteria(self, criteria, requirement, parent): - self._r.adding_requirement(requirement=requirement, parent=parent) - - identifier = self._p.identify(requirement_or_candidate=requirement) - criterion = criteria.get(identifier) - if criterion: - incompatibilities = list(criterion.incompatibilities) - else: - incompatibilities = [] - - matches = self._p.find_matches( - identifier=identifier, - requirements=IteratorMapping( - criteria, - operator.methodcaller("iter_requirement"), - {identifier: [requirement]}, - ), - incompatibilities=IteratorMapping( - criteria, - operator.attrgetter("incompatibilities"), - {identifier: incompatibilities}, - ), - ) - - if criterion: - information = list(criterion.information) - information.append(RequirementInformation(requirement, parent)) - else: - information = [RequirementInformation(requirement, parent)] - - criterion = Criterion( - candidates=build_iter_view(matches), - information=information, - incompatibilities=incompatibilities, - ) - if not criterion.candidates: - raise RequirementsConflicted(criterion) - criteria[identifier] = criterion - - def _remove_information_from_criteria(self, criteria, parents): - """Remove information from parents of criteria. - - Concretely, removes all values from each criterion's ``information`` - field that have one of ``parents`` as provider of the requirement. - - :param criteria: The criteria to update. - :param parents: Identifiers for which to remove information from all criteria. - """ - if not parents: - return - for key, criterion in criteria.items(): - criteria[key] = Criterion( - criterion.candidates, - [ - information - for information in criterion.information - if ( - information.parent is None - or self._p.identify(information.parent) not in parents - ) - ], - criterion.incompatibilities, - ) - - def _get_preference(self, name): - return self._p.get_preference( - identifier=name, - resolutions=self.state.mapping, - candidates=IteratorMapping( - self.state.criteria, - operator.attrgetter("candidates"), - ), - information=IteratorMapping( - self.state.criteria, - operator.attrgetter("information"), - ), - backtrack_causes=self.state.backtrack_causes, - ) - - def _is_current_pin_satisfying(self, name, criterion): - try: - current_pin = self.state.mapping[name] - except KeyError: - return False - return all( - self._p.is_satisfied_by(requirement=r, candidate=current_pin) - for r in criterion.iter_requirement() - ) - - def _get_updated_criteria(self, candidate): - criteria = self.state.criteria.copy() - for requirement in self._p.get_dependencies(candidate=candidate): - self._add_to_criteria(criteria, requirement, parent=candidate) - return criteria - - def _attempt_to_pin_criterion(self, name): - criterion = self.state.criteria[name] - - causes = [] - for candidate in criterion.candidates: - try: - criteria = self._get_updated_criteria(candidate) - except RequirementsConflicted as e: - self._r.rejecting_candidate(e.criterion, candidate) - causes.append(e.criterion) - continue - - # Check the newly-pinned candidate actually works. This should - # always pass under normal circumstances, but in the case of a - # faulty provider, we will raise an error to notify the implementer - # to fix find_matches() and/or is_satisfied_by(). - satisfied = all( - self._p.is_satisfied_by(requirement=r, candidate=candidate) - for r in criterion.iter_requirement() - ) - if not satisfied: - raise InconsistentCandidate(candidate, criterion) - - self._r.pinning(candidate=candidate) - self.state.criteria.update(criteria) - - # Put newly-pinned candidate at the end. This is essential because - # backtracking looks at this mapping to get the last pin. - self.state.mapping.pop(name, None) - self.state.mapping[name] = candidate - - return [] - - # All candidates tried, nothing works. This criterion is a dead - # end, signal for backtracking. - return causes - - def _backjump(self, causes): - """Perform backjumping. - - When we enter here, the stack is like this:: - - [ state Z ] - [ state Y ] - [ state X ] - .... earlier states are irrelevant. - - 1. No pins worked for Z, so it does not have a pin. - 2. We want to reset state Y to unpinned, and pin another candidate. - 3. State X holds what state Y was before the pin, but does not - have the incompatibility information gathered in state Y. - - Each iteration of the loop will: - - 1. Identify Z. The incompatibility is not always caused by the latest - state. For example, given three requirements A, B and C, with - dependencies A1, B1 and C1, where A1 and B1 are incompatible: the - last state might be related to C, so we want to discard the - previous state. - 2. Discard Z. - 3. Discard Y but remember its incompatibility information gathered - previously, and the failure we're dealing with right now. - 4. Push a new state Y' based on X, and apply the incompatibility - information from Y to Y'. - 5a. If this causes Y' to conflict, we need to backtrack again. Make Y' - the new Z and go back to step 2. - 5b. If the incompatibilities apply cleanly, end backtracking. - """ - incompatible_reqs = itertools.chain( - (c.parent for c in causes if c.parent is not None), - (c.requirement for c in causes), - ) - incompatible_deps = {self._p.identify(r) for r in incompatible_reqs} - while len(self._states) >= 3: - # Remove the state that triggered backtracking. - del self._states[-1] - - # Ensure to backtrack to a state that caused the incompatibility - incompatible_state = False - while not incompatible_state: - # Retrieve the last candidate pin and known incompatibilities. - try: - broken_state = self._states.pop() - name, candidate = broken_state.mapping.popitem() - except (IndexError, KeyError): - raise ResolutionImpossible(causes) - current_dependencies = { - self._p.identify(d) - for d in self._p.get_dependencies(candidate) - } - incompatible_state = not current_dependencies.isdisjoint( - incompatible_deps - ) - - incompatibilities_from_broken = [ - (k, list(v.incompatibilities)) - for k, v in broken_state.criteria.items() - ] - - # Also mark the newly known incompatibility. - incompatibilities_from_broken.append((name, [candidate])) - - # Create a new state from the last known-to-work one, and apply - # the previously gathered incompatibility information. - def _patch_criteria(): - for k, incompatibilities in incompatibilities_from_broken: - if not incompatibilities: - continue - try: - criterion = self.state.criteria[k] - except KeyError: - continue - matches = self._p.find_matches( - identifier=k, - requirements=IteratorMapping( - self.state.criteria, - operator.methodcaller("iter_requirement"), - ), - incompatibilities=IteratorMapping( - self.state.criteria, - operator.attrgetter("incompatibilities"), - {k: incompatibilities}, - ), - ) - candidates = build_iter_view(matches) - if not candidates: - return False - incompatibilities.extend(criterion.incompatibilities) - self.state.criteria[k] = Criterion( - candidates=candidates, - information=list(criterion.information), - incompatibilities=incompatibilities, - ) - return True - - self._push_new_state() - success = _patch_criteria() - - # It works! Let's work on this new state. - if success: - return True - - # State does not work after applying known incompatibilities. - # Try the still previous state. - - # No way to backtrack anymore. - return False - - def resolve(self, requirements, max_rounds): - if self._states: - raise RuntimeError("already resolved") - - self._r.starting() - - # Initialize the root state. - self._states = [ - State( - mapping=collections.OrderedDict(), - criteria={}, - backtrack_causes=[], - ) - ] - for r in requirements: - try: - self._add_to_criteria(self.state.criteria, r, parent=None) - except RequirementsConflicted as e: - raise ResolutionImpossible(e.criterion.information) - - # The root state is saved as a sentinel so the first ever pin can have - # something to backtrack to if it fails. The root state is basically - # pinning the virtual "root" package in the graph. - self._push_new_state() - - for round_index in range(max_rounds): - self._r.starting_round(index=round_index) - - unsatisfied_names = [ - key - for key, criterion in self.state.criteria.items() - if not self._is_current_pin_satisfying(key, criterion) - ] - - # All criteria are accounted for. Nothing more to pin, we are done! - if not unsatisfied_names: - self._r.ending(state=self.state) - return self.state - - # keep track of satisfied names to calculate diff after pinning - satisfied_names = set(self.state.criteria.keys()) - set( - unsatisfied_names - ) - - # Choose the most preferred unpinned criterion to try. - name = min(unsatisfied_names, key=self._get_preference) - failure_causes = self._attempt_to_pin_criterion(name) - - if failure_causes: - causes = [i for c in failure_causes for i in c.information] - # Backjump if pinning fails. The backjump process puts us in - # an unpinned state, so we can work on it in the next round. - self._r.resolving_conflicts(causes=causes) - success = self._backjump(causes) - self.state.backtrack_causes[:] = causes - - # Dead ends everywhere. Give up. - if not success: - raise ResolutionImpossible(self.state.backtrack_causes) - else: - # discard as information sources any invalidated names - # (unsatisfied names that were previously satisfied) - newly_unsatisfied_names = { - key - for key, criterion in self.state.criteria.items() - if key in satisfied_names - and not self._is_current_pin_satisfying(key, criterion) - } - self._remove_information_from_criteria( - self.state.criteria, newly_unsatisfied_names - ) - # Pinning was successful. Push a new state to do another pin. - self._push_new_state() - - self._r.ending_round(index=round_index, state=self.state) - - raise ResolutionTooDeep(max_rounds) - - -def _has_route_to_root(criteria, key, all_keys, connected): - if key in connected: - return True - if key not in criteria: - return False - for p in criteria[key].iter_parent(): - try: - pkey = all_keys[id(p)] - except KeyError: - continue - if pkey in connected: - connected.add(key) - return True - if _has_route_to_root(criteria, pkey, all_keys, connected): - connected.add(key) - return True - return False - - -Result = collections.namedtuple("Result", "mapping graph criteria") - - -def _build_result(state): - mapping = state.mapping - all_keys = {id(v): k for k, v in mapping.items()} - all_keys[id(None)] = None - - graph = DirectedGraph() - graph.add(None) # Sentinel as root dependencies' parent. - - connected = {None} - for key, criterion in state.criteria.items(): - if not _has_route_to_root(state.criteria, key, all_keys, connected): - continue - if key not in graph: - graph.add(key) - for p in criterion.iter_parent(): - try: - pkey = all_keys[id(p)] - except KeyError: - continue - if pkey not in graph: - graph.add(pkey) - graph.connect(pkey, key) - - return Result( - mapping={k: v for k, v in mapping.items() if k in connected}, - graph=graph, - criteria=state.criteria, - ) - - -class Resolver(AbstractResolver): - """The thing that performs the actual resolution work.""" - - base_exception = ResolverException - - def resolve(self, requirements, max_rounds=100): - """Take a collection of constraints, spit out the resolution result. - - The return value is a representation to the final resolution result. It - is a tuple subclass with three public members: - - * `mapping`: A dict of resolved candidates. Each key is an identifier - of a requirement (as returned by the provider's `identify` method), - and the value is the resolved candidate. - * `graph`: A `DirectedGraph` instance representing the dependency tree. - The vertices are keys of `mapping`, and each edge represents *why* - a particular package is included. A special vertex `None` is - included to represent parents of user-supplied requirements. - * `criteria`: A dict of "criteria" that hold detailed information on - how edges in the graph are derived. Each key is an identifier of a - requirement, and the value is a `Criterion` instance. - - The following exceptions may be raised if a resolution cannot be found: - - * `ResolutionImpossible`: A resolution cannot be found for the given - combination of requirements. The `causes` attribute of the - exception is a list of (requirement, parent), giving the - requirements that could not be satisfied. - * `ResolutionTooDeep`: The dependency tree is too deeply nested and - the resolver gave up. This is usually caused by a circular - dependency, but you can try to resolve this by increasing the - `max_rounds` argument. - """ - resolution = Resolution(self.provider, self.reporter) - state = resolution.resolve(requirements, max_rounds=max_rounds) - return _build_result(state) diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/resolvelib/structs.py b/venv/lib/python3.11/site-packages/pip/_vendor/resolvelib/structs.py deleted file mode 100644 index 359a34f..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/resolvelib/structs.py +++ /dev/null @@ -1,170 +0,0 @@ -import itertools - -from .compat import collections_abc - - -class DirectedGraph(object): - """A graph structure with directed edges.""" - - def __init__(self): - self._vertices = set() - self._forwards = {} # -> Set[] - self._backwards = {} # -> Set[] - - def __iter__(self): - return iter(self._vertices) - - def __len__(self): - return len(self._vertices) - - def __contains__(self, key): - return key in self._vertices - - def copy(self): - """Return a shallow copy of this graph.""" - other = DirectedGraph() - other._vertices = set(self._vertices) - other._forwards = {k: set(v) for k, v in self._forwards.items()} - other._backwards = {k: set(v) for k, v in self._backwards.items()} - return other - - def add(self, key): - """Add a new vertex to the graph.""" - if key in self._vertices: - raise ValueError("vertex exists") - self._vertices.add(key) - self._forwards[key] = set() - self._backwards[key] = set() - - def remove(self, key): - """Remove a vertex from the graph, disconnecting all edges from/to it.""" - self._vertices.remove(key) - for f in self._forwards.pop(key): - self._backwards[f].remove(key) - for t in self._backwards.pop(key): - self._forwards[t].remove(key) - - def connected(self, f, t): - return f in self._backwards[t] and t in self._forwards[f] - - def connect(self, f, t): - """Connect two existing vertices. - - Nothing happens if the vertices are already connected. - """ - if t not in self._vertices: - raise KeyError(t) - self._forwards[f].add(t) - self._backwards[t].add(f) - - def iter_edges(self): - for f, children in self._forwards.items(): - for t in children: - yield f, t - - def iter_children(self, key): - return iter(self._forwards[key]) - - def iter_parents(self, key): - return iter(self._backwards[key]) - - -class IteratorMapping(collections_abc.Mapping): - def __init__(self, mapping, accessor, appends=None): - self._mapping = mapping - self._accessor = accessor - self._appends = appends or {} - - def __repr__(self): - return "IteratorMapping({!r}, {!r}, {!r})".format( - self._mapping, - self._accessor, - self._appends, - ) - - def __bool__(self): - return bool(self._mapping or self._appends) - - __nonzero__ = __bool__ # XXX: Python 2. - - def __contains__(self, key): - return key in self._mapping or key in self._appends - - def __getitem__(self, k): - try: - v = self._mapping[k] - except KeyError: - return iter(self._appends[k]) - return itertools.chain(self._accessor(v), self._appends.get(k, ())) - - def __iter__(self): - more = (k for k in self._appends if k not in self._mapping) - return itertools.chain(self._mapping, more) - - def __len__(self): - more = sum(1 for k in self._appends if k not in self._mapping) - return len(self._mapping) + more - - -class _FactoryIterableView(object): - """Wrap an iterator factory returned by `find_matches()`. - - Calling `iter()` on this class would invoke the underlying iterator - factory, making it a "collection with ordering" that can be iterated - through multiple times, but lacks random access methods presented in - built-in Python sequence types. - """ - - def __init__(self, factory): - self._factory = factory - self._iterable = None - - def __repr__(self): - return "{}({})".format(type(self).__name__, list(self)) - - def __bool__(self): - try: - next(iter(self)) - except StopIteration: - return False - return True - - __nonzero__ = __bool__ # XXX: Python 2. - - def __iter__(self): - iterable = ( - self._factory() if self._iterable is None else self._iterable - ) - self._iterable, current = itertools.tee(iterable) - return current - - -class _SequenceIterableView(object): - """Wrap an iterable returned by find_matches(). - - This is essentially just a proxy to the underlying sequence that provides - the same interface as `_FactoryIterableView`. - """ - - def __init__(self, sequence): - self._sequence = sequence - - def __repr__(self): - return "{}({})".format(type(self).__name__, self._sequence) - - def __bool__(self): - return bool(self._sequence) - - __nonzero__ = __bool__ # XXX: Python 2. - - def __iter__(self): - return iter(self._sequence) - - -def build_iter_view(matches): - """Build an iterable view from the value returned by `find_matches()`.""" - if callable(matches): - return _FactoryIterableView(matches) - if not isinstance(matches, collections_abc.Sequence): - matches = list(matches) - return _SequenceIterableView(matches) diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__init__.py b/venv/lib/python3.11/site-packages/pip/_vendor/rich/__init__.py deleted file mode 100644 index 73f58d7..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__init__.py +++ /dev/null @@ -1,177 +0,0 @@ -"""Rich text and beautiful formatting in the terminal.""" - -import os -from typing import IO, TYPE_CHECKING, Any, Callable, Optional, Union - -from ._extension import load_ipython_extension # noqa: F401 - -__all__ = ["get_console", "reconfigure", "print", "inspect", "print_json"] - -if TYPE_CHECKING: - from .console import Console - -# Global console used by alternative print -_console: Optional["Console"] = None - -try: - _IMPORT_CWD = os.path.abspath(os.getcwd()) -except FileNotFoundError: - # Can happen if the cwd has been deleted - _IMPORT_CWD = "" - - -def get_console() -> "Console": - """Get a global :class:`~rich.console.Console` instance. This function is used when Rich requires a Console, - and hasn't been explicitly given one. - - Returns: - Console: A console instance. - """ - global _console - if _console is None: - from .console import Console - - _console = Console() - - return _console - - -def reconfigure(*args: Any, **kwargs: Any) -> None: - """Reconfigures the global console by replacing it with another. - - Args: - *args (Any): Positional arguments for the replacement :class:`~rich.console.Console`. - **kwargs (Any): Keyword arguments for the replacement :class:`~rich.console.Console`. - """ - from pip._vendor.rich.console import Console - - new_console = Console(*args, **kwargs) - _console = get_console() - _console.__dict__ = new_console.__dict__ - - -def print( - *objects: Any, - sep: str = " ", - end: str = "\n", - file: Optional[IO[str]] = None, - flush: bool = False, -) -> None: - r"""Print object(s) supplied via positional arguments. - This function has an identical signature to the built-in print. - For more advanced features, see the :class:`~rich.console.Console` class. - - Args: - sep (str, optional): Separator between printed objects. Defaults to " ". - end (str, optional): Character to write at end of output. Defaults to "\\n". - file (IO[str], optional): File to write to, or None for stdout. Defaults to None. - flush (bool, optional): Has no effect as Rich always flushes output. Defaults to False. - - """ - from .console import Console - - write_console = get_console() if file is None else Console(file=file) - return write_console.print(*objects, sep=sep, end=end) - - -def print_json( - json: Optional[str] = None, - *, - data: Any = None, - indent: Union[None, int, str] = 2, - highlight: bool = True, - skip_keys: bool = False, - ensure_ascii: bool = False, - check_circular: bool = True, - allow_nan: bool = True, - default: Optional[Callable[[Any], Any]] = None, - sort_keys: bool = False, -) -> None: - """Pretty prints JSON. Output will be valid JSON. - - Args: - json (str): A string containing JSON. - data (Any): If json is not supplied, then encode this data. - indent (int, optional): Number of spaces to indent. Defaults to 2. - highlight (bool, optional): Enable highlighting of output: Defaults to True. - skip_keys (bool, optional): Skip keys not of a basic type. Defaults to False. - ensure_ascii (bool, optional): Escape all non-ascii characters. Defaults to False. - check_circular (bool, optional): Check for circular references. Defaults to True. - allow_nan (bool, optional): Allow NaN and Infinity values. Defaults to True. - default (Callable, optional): A callable that converts values that can not be encoded - in to something that can be JSON encoded. Defaults to None. - sort_keys (bool, optional): Sort dictionary keys. Defaults to False. - """ - - get_console().print_json( - json, - data=data, - indent=indent, - highlight=highlight, - skip_keys=skip_keys, - ensure_ascii=ensure_ascii, - check_circular=check_circular, - allow_nan=allow_nan, - default=default, - sort_keys=sort_keys, - ) - - -def inspect( - obj: Any, - *, - console: Optional["Console"] = None, - title: Optional[str] = None, - help: bool = False, - methods: bool = False, - docs: bool = True, - private: bool = False, - dunder: bool = False, - sort: bool = True, - all: bool = False, - value: bool = True, -) -> None: - """Inspect any Python object. - - * inspect() to see summarized info. - * inspect(, methods=True) to see methods. - * inspect(, help=True) to see full (non-abbreviated) help. - * inspect(, private=True) to see private attributes (single underscore). - * inspect(, dunder=True) to see attributes beginning with double underscore. - * inspect(, all=True) to see all attributes. - - Args: - obj (Any): An object to inspect. - title (str, optional): Title to display over inspect result, or None use type. Defaults to None. - help (bool, optional): Show full help text rather than just first paragraph. Defaults to False. - methods (bool, optional): Enable inspection of callables. Defaults to False. - docs (bool, optional): Also render doc strings. Defaults to True. - private (bool, optional): Show private attributes (beginning with underscore). Defaults to False. - dunder (bool, optional): Show attributes starting with double underscore. Defaults to False. - sort (bool, optional): Sort attributes alphabetically. Defaults to True. - all (bool, optional): Show all attributes. Defaults to False. - value (bool, optional): Pretty print value. Defaults to True. - """ - _console = console or get_console() - from pip._vendor.rich._inspect import Inspect - - # Special case for inspect(inspect) - is_inspect = obj is inspect - - _inspect = Inspect( - obj, - title=title, - help=is_inspect or help, - methods=is_inspect or methods, - docs=is_inspect or docs, - private=private, - dunder=dunder, - sort=sort, - all=all, - value=value, - ) - _console.print(_inspect) - - -if __name__ == "__main__": # pragma: no cover - print("Hello, **World**") diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__main__.py b/venv/lib/python3.11/site-packages/pip/_vendor/rich/__main__.py deleted file mode 100644 index 270629f..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__main__.py +++ /dev/null @@ -1,274 +0,0 @@ -import colorsys -import io -from time import process_time - -from pip._vendor.rich import box -from pip._vendor.rich.color import Color -from pip._vendor.rich.console import Console, ConsoleOptions, Group, RenderableType, RenderResult -from pip._vendor.rich.markdown import Markdown -from pip._vendor.rich.measure import Measurement -from pip._vendor.rich.pretty import Pretty -from pip._vendor.rich.segment import Segment -from pip._vendor.rich.style import Style -from pip._vendor.rich.syntax import Syntax -from pip._vendor.rich.table import Table -from pip._vendor.rich.text import Text - - -class ColorBox: - def __rich_console__( - self, console: Console, options: ConsoleOptions - ) -> RenderResult: - for y in range(0, 5): - for x in range(options.max_width): - h = x / options.max_width - l = 0.1 + ((y / 5) * 0.7) - r1, g1, b1 = colorsys.hls_to_rgb(h, l, 1.0) - r2, g2, b2 = colorsys.hls_to_rgb(h, l + 0.7 / 10, 1.0) - bgcolor = Color.from_rgb(r1 * 255, g1 * 255, b1 * 255) - color = Color.from_rgb(r2 * 255, g2 * 255, b2 * 255) - yield Segment("▄", Style(color=color, bgcolor=bgcolor)) - yield Segment.line() - - def __rich_measure__( - self, console: "Console", options: ConsoleOptions - ) -> Measurement: - return Measurement(1, options.max_width) - - -def make_test_card() -> Table: - """Get a renderable that demonstrates a number of features.""" - table = Table.grid(padding=1, pad_edge=True) - table.title = "Rich features" - table.add_column("Feature", no_wrap=True, justify="center", style="bold red") - table.add_column("Demonstration") - - color_table = Table( - box=None, - expand=False, - show_header=False, - show_edge=False, - pad_edge=False, - ) - color_table.add_row( - ( - "✓ [bold green]4-bit color[/]\n" - "✓ [bold blue]8-bit color[/]\n" - "✓ [bold magenta]Truecolor (16.7 million)[/]\n" - "✓ [bold yellow]Dumb terminals[/]\n" - "✓ [bold cyan]Automatic color conversion" - ), - ColorBox(), - ) - - table.add_row("Colors", color_table) - - table.add_row( - "Styles", - "All ansi styles: [bold]bold[/], [dim]dim[/], [italic]italic[/italic], [underline]underline[/], [strike]strikethrough[/], [reverse]reverse[/], and even [blink]blink[/].", - ) - - lorem = "Lorem ipsum dolor sit amet, consectetur adipiscing elit. Quisque in metus sed sapien ultricies pretium a at justo. Maecenas luctus velit et auctor maximus." - lorem_table = Table.grid(padding=1, collapse_padding=True) - lorem_table.pad_edge = False - lorem_table.add_row( - Text(lorem, justify="left", style="green"), - Text(lorem, justify="center", style="yellow"), - Text(lorem, justify="right", style="blue"), - Text(lorem, justify="full", style="red"), - ) - table.add_row( - "Text", - Group( - Text.from_markup( - """Word wrap text. Justify [green]left[/], [yellow]center[/], [blue]right[/] or [red]full[/].\n""" - ), - lorem_table, - ), - ) - - def comparison(renderable1: RenderableType, renderable2: RenderableType) -> Table: - table = Table(show_header=False, pad_edge=False, box=None, expand=True) - table.add_column("1", ratio=1) - table.add_column("2", ratio=1) - table.add_row(renderable1, renderable2) - return table - - table.add_row( - "Asian\nlanguage\nsupport", - ":flag_for_china: 该库支持中文，日文和韩文文本！\n:flag_for_japan: ライブラリは中国語、日本語、韓国語のテキストをサポートしています\n:flag_for_south_korea: 이 라이브러리는 중국어, 일본어 및 한국어 텍스트를 지원합니다", - ) - - markup_example = ( - "[bold magenta]Rich[/] supports a simple [i]bbcode[/i]-like [b]markup[/b] for [yellow]color[/], [underline]style[/], and emoji! " - ":+1: :apple: :ant: :bear: :baguette_bread: :bus: " - ) - table.add_row("Markup", markup_example) - - example_table = Table( - show_edge=False, - show_header=True, - expand=False, - row_styles=["none", "dim"], - box=box.SIMPLE, - ) - example_table.add_column("[green]Date", style="green", no_wrap=True) - example_table.add_column("[blue]Title", style="blue") - example_table.add_column( - "[cyan]Production Budget", - style="cyan", - justify="right", - no_wrap=True, - ) - example_table.add_column( - "[magenta]Box Office", - style="magenta", - justify="right", - no_wrap=True, - ) - example_table.add_row( - "Dec 20, 2019", - "Star Wars: The Rise of Skywalker", - "$275,000,000", - "$375,126,118", - ) - example_table.add_row( - "May 25, 2018", - "[b]Solo[/]: A Star Wars Story", - "$275,000,000", - "$393,151,347", - ) - example_table.add_row( - "Dec 15, 2017", - "Star Wars Ep. VIII: The Last Jedi", - "$262,000,000", - "[bold]$1,332,539,889[/bold]", - ) - example_table.add_row( - "May 19, 1999", - "Star Wars Ep. [b]I[/b]: [i]The phantom Menace", - "$115,000,000", - "$1,027,044,677", - ) - - table.add_row("Tables", example_table) - - code = '''\ -def iter_last(values: Iterable[T]) -> Iterable[Tuple[bool, T]]: - """Iterate and generate a tuple with a flag for last value.""" - iter_values = iter(values) - try: - previous_value = next(iter_values) - except StopIteration: - return - for value in iter_values: - yield False, previous_value - previous_value = value - yield True, previous_value''' - - pretty_data = { - "foo": [ - 3.1427, - ( - "Paul Atreides", - "Vladimir Harkonnen", - "Thufir Hawat", - ), - ], - "atomic": (False, True, None), - } - table.add_row( - "Syntax\nhighlighting\n&\npretty\nprinting", - comparison( - Syntax(code, "python3", line_numbers=True, indent_guides=True), - Pretty(pretty_data, indent_guides=True), - ), - ) - - markdown_example = """\ -# Markdown - -Supports much of the *markdown* __syntax__! - -- Headers -- Basic formatting: **bold**, *italic*, `code` -- Block quotes -- Lists, and more... - """ - table.add_row( - "Markdown", comparison("[cyan]" + markdown_example, Markdown(markdown_example)) - ) - - table.add_row( - "+more!", - """Progress bars, columns, styled logging handler, tracebacks, etc...""", - ) - return table - - -if __name__ == "__main__": # pragma: no cover - - console = Console( - file=io.StringIO(), - force_terminal=True, - ) - test_card = make_test_card() - - # Print once to warm cache - start = process_time() - console.print(test_card) - pre_cache_taken = round((process_time() - start) * 1000.0, 1) - - console.file = io.StringIO() - - start = process_time() - console.print(test_card) - taken = round((process_time() - start) * 1000.0, 1) - - c = Console(record=True) - c.print(test_card) - - print(f"rendered in {pre_cache_taken}ms (cold cache)") - print(f"rendered in {taken}ms (warm cache)") - - from pip._vendor.rich.panel import Panel - - console = Console() - - sponsor_message = Table.grid(padding=1) - sponsor_message.add_column(style="green", justify="right") - sponsor_message.add_column(no_wrap=True) - - sponsor_message.add_row( - "Textualize", - "[u blue link=https://github.com/textualize]https://github.com/textualize", - ) - sponsor_message.add_row( - "Twitter", - "[u blue link=https://twitter.com/willmcgugan]https://twitter.com/willmcgugan", - ) - - intro_message = Text.from_markup( - """\ -We hope you enjoy using Rich! - -Rich is maintained with [red]:heart:[/] by [link=https://www.textualize.io]Textualize.io[/] - -- Will McGugan""" - ) - - message = Table.grid(padding=2) - message.add_column() - message.add_column(no_wrap=True) - message.add_row(intro_message, sponsor_message) - - console.print( - Panel.fit( - message, - box=box.ROUNDED, - padding=(1, 2), - title="[b red]Thanks for trying out Rich!", - border_style="bright_blue", - ), - justify="center", - ) diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/__init__.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/__init__.cpython-311.pyc deleted file mode 100644 index 35eac45..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/__init__.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/__main__.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/__main__.cpython-311.pyc deleted file mode 100644 index 5d90241..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/__main__.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/_cell_widths.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/_cell_widths.cpython-311.pyc deleted file mode 100644 index 32cb8f3..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/_cell_widths.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/_emoji_codes.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/_emoji_codes.cpython-311.pyc deleted file mode 100644 index 8707f98..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/_emoji_codes.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/_emoji_replace.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/_emoji_replace.cpython-311.pyc deleted file mode 100644 index 3bef2b3..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/_emoji_replace.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/_export_format.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/_export_format.cpython-311.pyc deleted file mode 100644 index f68e584..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/_export_format.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/_extension.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/_extension.cpython-311.pyc deleted file mode 100644 index 2342723..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/_extension.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/_fileno.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/_fileno.cpython-311.pyc deleted file mode 100644 index 5b99803..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/_fileno.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/_inspect.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/_inspect.cpython-311.pyc deleted file mode 100644 index 59212f6..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/_inspect.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/_log_render.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/_log_render.cpython-311.pyc deleted file mode 100644 index 97270c0..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/_log_render.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/_loop.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/_loop.cpython-311.pyc deleted file mode 100644 index 2f8f6d6..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/_loop.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/_null_file.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/_null_file.cpython-311.pyc deleted file mode 100644 index 413e46b..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/_null_file.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/_palettes.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/_palettes.cpython-311.pyc deleted file mode 100644 index 8c00150..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/_palettes.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/_pick.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/_pick.cpython-311.pyc deleted file mode 100644 index 2088ab5..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/_pick.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/_ratio.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/_ratio.cpython-311.pyc deleted file mode 100644 index 123ec1d..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/_ratio.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/_spinners.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/_spinners.cpython-311.pyc deleted file mode 100644 index 8f2c52b..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/_spinners.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/_stack.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/_stack.cpython-311.pyc deleted file mode 100644 index 036c0e9..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/_stack.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/_timer.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/_timer.cpython-311.pyc deleted file mode 100644 index 18d9583..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/_timer.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/_win32_console.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/_win32_console.cpython-311.pyc deleted file mode 100644 index 744b857..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/_win32_console.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/_windows.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/_windows.cpython-311.pyc deleted file mode 100644 index e9c3412..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/_windows.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/_windows_renderer.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/_windows_renderer.cpython-311.pyc deleted file mode 100644 index 237e67b..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/_windows_renderer.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/_wrap.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/_wrap.cpython-311.pyc deleted file mode 100644 index 9bf4ae6..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/_wrap.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/abc.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/abc.cpython-311.pyc deleted file mode 100644 index 9aed81c..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/abc.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/align.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/align.cpython-311.pyc deleted file mode 100644 index a76253c..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/align.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/ansi.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/ansi.cpython-311.pyc deleted file mode 100644 index 3304e5c..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/ansi.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/bar.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/bar.cpython-311.pyc deleted file mode 100644 index 71672e1..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/bar.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/box.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/box.cpython-311.pyc deleted file mode 100644 index 6c61a5d..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/box.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/cells.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/cells.cpython-311.pyc deleted file mode 100644 index bfe1ea1..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/cells.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/color.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/color.cpython-311.pyc deleted file mode 100644 index b5514f0..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/color.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/color_triplet.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/color_triplet.cpython-311.pyc deleted file mode 100644 index 7a7a007..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/color_triplet.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/columns.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/columns.cpython-311.pyc deleted file mode 100644 index d5c933a..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/columns.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/console.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/console.cpython-311.pyc deleted file mode 100644 index 00ff63d..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/console.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/constrain.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/constrain.cpython-311.pyc deleted file mode 100644 index bdc1d1c..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/constrain.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/containers.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/containers.cpython-311.pyc deleted file mode 100644 index 4640751..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/containers.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/control.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/control.cpython-311.pyc deleted file mode 100644 index d89caa8..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/control.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/default_styles.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/default_styles.cpython-311.pyc deleted file mode 100644 index b35fb43..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/default_styles.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/diagnose.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/diagnose.cpython-311.pyc deleted file mode 100644 index 5ee423c..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/diagnose.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/emoji.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/emoji.cpython-311.pyc deleted file mode 100644 index 9a82f68..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/emoji.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/errors.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/errors.cpython-311.pyc deleted file mode 100644 index 93368be..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/errors.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/file_proxy.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/file_proxy.cpython-311.pyc deleted file mode 100644 index a4e3c60..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/file_proxy.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/filesize.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/filesize.cpython-311.pyc deleted file mode 100644 index 0304234..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/filesize.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/highlighter.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/highlighter.cpython-311.pyc deleted file mode 100644 index 338149e..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/highlighter.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/json.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/json.cpython-311.pyc deleted file mode 100644 index edef10f..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/json.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/jupyter.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/jupyter.cpython-311.pyc deleted file mode 100644 index b6addde..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/jupyter.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/layout.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/layout.cpython-311.pyc deleted file mode 100644 index 477208f..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/layout.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/live.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/live.cpython-311.pyc deleted file mode 100644 index d7c13a4..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/live.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/live_render.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/live_render.cpython-311.pyc deleted file mode 100644 index 85a6e96..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/live_render.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/logging.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/logging.cpython-311.pyc deleted file mode 100644 index c467d9d..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/logging.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/markup.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/markup.cpython-311.pyc deleted file mode 100644 index ddce4cc..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/markup.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/measure.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/measure.cpython-311.pyc deleted file mode 100644 index 8f37b14..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/measure.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/padding.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/padding.cpython-311.pyc deleted file mode 100644 index 244d4fa..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/padding.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/pager.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/pager.cpython-311.pyc deleted file mode 100644 index 42660e8..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/pager.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/palette.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/palette.cpython-311.pyc deleted file mode 100644 index 43cb69d..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/palette.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/panel.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/panel.cpython-311.pyc deleted file mode 100644 index c76d639..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/panel.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/pretty.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/pretty.cpython-311.pyc deleted file mode 100644 index 3217821..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/pretty.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/progress.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/progress.cpython-311.pyc deleted file mode 100644 index 9c19faf..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/progress.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/progress_bar.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/progress_bar.cpython-311.pyc deleted file mode 100644 index 16d17fb..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/progress_bar.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/prompt.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/prompt.cpython-311.pyc deleted file mode 100644 index e16a119..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/prompt.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/protocol.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/protocol.cpython-311.pyc deleted file mode 100644 index 40939dc..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/protocol.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/region.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/region.cpython-311.pyc deleted file mode 100644 index 49fe6f2..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/region.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/repr.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/repr.cpython-311.pyc deleted file mode 100644 index f966417..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/repr.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/rule.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/rule.cpython-311.pyc deleted file mode 100644 index cff1ae3..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/rule.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/scope.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/scope.cpython-311.pyc deleted file mode 100644 index 6b9d9dd..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/scope.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/screen.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/screen.cpython-311.pyc deleted file mode 100644 index dc2467e..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/screen.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/segment.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/segment.cpython-311.pyc deleted file mode 100644 index d94daad..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/segment.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/spinner.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/spinner.cpython-311.pyc deleted file mode 100644 index a931440..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/spinner.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/status.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/status.cpython-311.pyc deleted file mode 100644 index 77d1aea..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/status.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/style.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/style.cpython-311.pyc deleted file mode 100644 index 9b3f1cb..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/style.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/styled.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/styled.cpython-311.pyc deleted file mode 100644 index ca26b23..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/styled.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/syntax.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/syntax.cpython-311.pyc deleted file mode 100644 index 9e4506a..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/syntax.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/table.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/table.cpython-311.pyc deleted file mode 100644 index 132be68..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/table.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/terminal_theme.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/terminal_theme.cpython-311.pyc deleted file mode 100644 index c457a6f..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/terminal_theme.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/text.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/text.cpython-311.pyc deleted file mode 100644 index 2a81d6a..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/text.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/theme.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/theme.cpython-311.pyc deleted file mode 100644 index 7f511f7..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/theme.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/themes.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/themes.cpython-311.pyc deleted file mode 100644 index 6df469b..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/themes.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/traceback.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/traceback.cpython-311.pyc deleted file mode 100644 index 6b0bee3..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/traceback.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/tree.cpython-311.pyc b/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/tree.cpython-311.pyc deleted file mode 100644 index d0b7ae2..0000000 Binary files a/venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__/tree.cpython-311.pyc and /dev/null differ diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/rich/_cell_widths.py b/venv/lib/python3.11/site-packages/pip/_vendor/rich/_cell_widths.py deleted file mode 100644 index 36286df..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/rich/_cell_widths.py +++ /dev/null @@ -1,451 +0,0 @@ -# Auto generated by make_terminal_widths.py - -CELL_WIDTHS = [ - (0, 0, 0), - (1, 31, -1), - (127, 159, -1), - (768, 879, 0), - (1155, 1161, 0), - (1425, 1469, 0), - (1471, 1471, 0), - (1473, 1474, 0), - (1476, 1477, 0), - (1479, 1479, 0), - (1552, 1562, 0), - (1611, 1631, 0), - (1648, 1648, 0), - (1750, 1756, 0), - (1759, 1764, 0), - (1767, 1768, 0), - (1770, 1773, 0), - (1809, 1809, 0), - (1840, 1866, 0), - (1958, 1968, 0), - (2027, 2035, 0), - (2045, 2045, 0), - (2070, 2073, 0), - (2075, 2083, 0), - (2085, 2087, 0), - (2089, 2093, 0), - (2137, 2139, 0), - (2259, 2273, 0), - (2275, 2306, 0), - (2362, 2362, 0), - (2364, 2364, 0), - (2369, 2376, 0), - (2381, 2381, 0), - (2385, 2391, 0), - (2402, 2403, 0), - (2433, 2433, 0), - (2492, 2492, 0), - (2497, 2500, 0), - (2509, 2509, 0), - (2530, 2531, 0), - (2558, 2558, 0), - (2561, 2562, 0), - (2620, 2620, 0), - (2625, 2626, 0), - (2631, 2632, 0), - (2635, 2637, 0), - (2641, 2641, 0), - (2672, 2673, 0), - (2677, 2677, 0), - (2689, 2690, 0), - (2748, 2748, 0), - (2753, 2757, 0), - (2759, 2760, 0), - (2765, 2765, 0), - (2786, 2787, 0), - (2810, 2815, 0), - (2817, 2817, 0), - (2876, 2876, 0), - (2879, 2879, 0), - (2881, 2884, 0), - (2893, 2893, 0), - (2901, 2902, 0), - (2914, 2915, 0), - (2946, 2946, 0), - (3008, 3008, 0), - (3021, 3021, 0), - (3072, 3072, 0), - (3076, 3076, 0), - (3134, 3136, 0), - (3142, 3144, 0), - (3146, 3149, 0), - (3157, 3158, 0), - (3170, 3171, 0), - (3201, 3201, 0), - (3260, 3260, 0), - (3263, 3263, 0), - (3270, 3270, 0), - (3276, 3277, 0), - (3298, 3299, 0), - (3328, 3329, 0), - (3387, 3388, 0), - (3393, 3396, 0), - (3405, 3405, 0), - (3426, 3427, 0), - (3457, 3457, 0), - (3530, 3530, 0), - (3538, 3540, 0), - (3542, 3542, 0), - (3633, 3633, 0), - (3636, 3642, 0), - (3655, 3662, 0), - (3761, 3761, 0), - (3764, 3772, 0), - (3784, 3789, 0), - (3864, 3865, 0), - (3893, 3893, 0), - (3895, 3895, 0), - (3897, 3897, 0), - (3953, 3966, 0), - (3968, 3972, 0), - (3974, 3975, 0), - (3981, 3991, 0), - (3993, 4028, 0), - (4038, 4038, 0), - (4141, 4144, 0), - (4146, 4151, 0), - (4153, 4154, 0), - (4157, 4158, 0), - (4184, 4185, 0), - (4190, 4192, 0), - (4209, 4212, 0), - (4226, 4226, 0), - (4229, 4230, 0), - (4237, 4237, 0), - (4253, 4253, 0), - (4352, 4447, 2), - (4957, 4959, 0), - (5906, 5908, 0), - (5938, 5940, 0), - (5970, 5971, 0), - (6002, 6003, 0), - (6068, 6069, 0), - (6071, 6077, 0), - (6086, 6086, 0), - (6089, 6099, 0), - (6109, 6109, 0), - (6155, 6157, 0), - (6277, 6278, 0), - (6313, 6313, 0), - (6432, 6434, 0), - (6439, 6440, 0), - (6450, 6450, 0), - (6457, 6459, 0), - (6679, 6680, 0), - (6683, 6683, 0), - (6742, 6742, 0), - (6744, 6750, 0), - (6752, 6752, 0), - (6754, 6754, 0), - (6757, 6764, 0), - (6771, 6780, 0), - (6783, 6783, 0), - (6832, 6848, 0), - (6912, 6915, 0), - (6964, 6964, 0), - (6966, 6970, 0), - (6972, 6972, 0), - (6978, 6978, 0), - (7019, 7027, 0), - (7040, 7041, 0), - (7074, 7077, 0), - (7080, 7081, 0), - (7083, 7085, 0), - (7142, 7142, 0), - (7144, 7145, 0), - (7149, 7149, 0), - (7151, 7153, 0), - (7212, 7219, 0), - (7222, 7223, 0), - (7376, 7378, 0), - (7380, 7392, 0), - (7394, 7400, 0), - (7405, 7405, 0), - (7412, 7412, 0), - (7416, 7417, 0), - (7616, 7673, 0), - (7675, 7679, 0), - (8203, 8207, 0), - (8232, 8238, 0), - (8288, 8291, 0), - (8400, 8432, 0), - (8986, 8987, 2), - (9001, 9002, 2), - (9193, 9196, 2), - (9200, 9200, 2), - (9203, 9203, 2), - (9725, 9726, 2), - (9748, 9749, 2), - (9800, 9811, 2), - (9855, 9855, 2), - (9875, 9875, 2), - (9889, 9889, 2), - (9898, 9899, 2), - (9917, 9918, 2), - (9924, 9925, 2), - (9934, 9934, 2), - (9940, 9940, 2), - (9962, 9962, 2), - (9970, 9971, 2), - (9973, 9973, 2), - (9978, 9978, 2), - (9981, 9981, 2), - (9989, 9989, 2), - (9994, 9995, 2), - (10024, 10024, 2), - (10060, 10060, 2), - (10062, 10062, 2), - (10067, 10069, 2), - (10071, 10071, 2), - (10133, 10135, 2), - (10160, 10160, 2), - (10175, 10175, 2), - (11035, 11036, 2), - (11088, 11088, 2), - (11093, 11093, 2), - (11503, 11505, 0), - (11647, 11647, 0), - (11744, 11775, 0), - (11904, 11929, 2), - (11931, 12019, 2), - (12032, 12245, 2), - (12272, 12283, 2), - (12288, 12329, 2), - (12330, 12333, 0), - (12334, 12350, 2), - (12353, 12438, 2), - (12441, 12442, 0), - (12443, 12543, 2), - (12549, 12591, 2), - (12593, 12686, 2), - (12688, 12771, 2), - (12784, 12830, 2), - (12832, 12871, 2), - (12880, 19903, 2), - (19968, 42124, 2), - (42128, 42182, 2), - (42607, 42610, 0), - (42612, 42621, 0), - (42654, 42655, 0), - (42736, 42737, 0), - (43010, 43010, 0), - (43014, 43014, 0), - (43019, 43019, 0), - (43045, 43046, 0), - (43052, 43052, 0), - (43204, 43205, 0), - (43232, 43249, 0), - (43263, 43263, 0), - (43302, 43309, 0), - (43335, 43345, 0), - (43360, 43388, 2), - (43392, 43394, 0), - (43443, 43443, 0), - (43446, 43449, 0), - (43452, 43453, 0), - (43493, 43493, 0), - (43561, 43566, 0), - (43569, 43570, 0), - (43573, 43574, 0), - (43587, 43587, 0), - (43596, 43596, 0), - (43644, 43644, 0), - (43696, 43696, 0), - (43698, 43700, 0), - (43703, 43704, 0), - (43710, 43711, 0), - (43713, 43713, 0), - (43756, 43757, 0), - (43766, 43766, 0), - (44005, 44005, 0), - (44008, 44008, 0), - (44013, 44013, 0), - (44032, 55203, 2), - (63744, 64255, 2), - (64286, 64286, 0), - (65024, 65039, 0), - (65040, 65049, 2), - (65056, 65071, 0), - (65072, 65106, 2), - (65108, 65126, 2), - (65128, 65131, 2), - (65281, 65376, 2), - (65504, 65510, 2), - (66045, 66045, 0), - (66272, 66272, 0), - (66422, 66426, 0), - (68097, 68099, 0), - (68101, 68102, 0), - (68108, 68111, 0), - (68152, 68154, 0), - (68159, 68159, 0), - (68325, 68326, 0), - (68900, 68903, 0), - (69291, 69292, 0), - (69446, 69456, 0), - (69633, 69633, 0), - (69688, 69702, 0), - (69759, 69761, 0), - (69811, 69814, 0), - (69817, 69818, 0), - (69888, 69890, 0), - (69927, 69931, 0), - (69933, 69940, 0), - (70003, 70003, 0), - (70016, 70017, 0), - (70070, 70078, 0), - (70089, 70092, 0), - (70095, 70095, 0), - (70191, 70193, 0), - (70196, 70196, 0), - (70198, 70199, 0), - (70206, 70206, 0), - (70367, 70367, 0), - (70371, 70378, 0), - (70400, 70401, 0), - (70459, 70460, 0), - (70464, 70464, 0), - (70502, 70508, 0), - (70512, 70516, 0), - (70712, 70719, 0), - (70722, 70724, 0), - (70726, 70726, 0), - (70750, 70750, 0), - (70835, 70840, 0), - (70842, 70842, 0), - (70847, 70848, 0), - (70850, 70851, 0), - (71090, 71093, 0), - (71100, 71101, 0), - (71103, 71104, 0), - (71132, 71133, 0), - (71219, 71226, 0), - (71229, 71229, 0), - (71231, 71232, 0), - (71339, 71339, 0), - (71341, 71341, 0), - (71344, 71349, 0), - (71351, 71351, 0), - (71453, 71455, 0), - (71458, 71461, 0), - (71463, 71467, 0), - (71727, 71735, 0), - (71737, 71738, 0), - (71995, 71996, 0), - (71998, 71998, 0), - (72003, 72003, 0), - (72148, 72151, 0), - (72154, 72155, 0), - (72160, 72160, 0), - (72193, 72202, 0), - (72243, 72248, 0), - (72251, 72254, 0), - (72263, 72263, 0), - (72273, 72278, 0), - (72281, 72283, 0), - (72330, 72342, 0), - (72344, 72345, 0), - (72752, 72758, 0), - (72760, 72765, 0), - (72767, 72767, 0), - (72850, 72871, 0), - (72874, 72880, 0), - (72882, 72883, 0), - (72885, 72886, 0), - (73009, 73014, 0), - (73018, 73018, 0), - (73020, 73021, 0), - (73023, 73029, 0), - (73031, 73031, 0), - (73104, 73105, 0), - (73109, 73109, 0), - (73111, 73111, 0), - (73459, 73460, 0), - (92912, 92916, 0), - (92976, 92982, 0), - (94031, 94031, 0), - (94095, 94098, 0), - (94176, 94179, 2), - (94180, 94180, 0), - (94192, 94193, 2), - (94208, 100343, 2), - (100352, 101589, 2), - (101632, 101640, 2), - (110592, 110878, 2), - (110928, 110930, 2), - (110948, 110951, 2), - (110960, 111355, 2), - (113821, 113822, 0), - (119143, 119145, 0), - (119163, 119170, 0), - (119173, 119179, 0), - (119210, 119213, 0), - (119362, 119364, 0), - (121344, 121398, 0), - (121403, 121452, 0), - (121461, 121461, 0), - (121476, 121476, 0), - (121499, 121503, 0), - (121505, 121519, 0), - (122880, 122886, 0), - (122888, 122904, 0), - (122907, 122913, 0), - (122915, 122916, 0), - (122918, 122922, 0), - (123184, 123190, 0), - (123628, 123631, 0), - (125136, 125142, 0), - (125252, 125258, 0), - (126980, 126980, 2), - (127183, 127183, 2), - (127374, 127374, 2), - (127377, 127386, 2), - (127488, 127490, 2), - (127504, 127547, 2), - (127552, 127560, 2), - (127568, 127569, 2), - (127584, 127589, 2), - (127744, 127776, 2), - (127789, 127797, 2), - (127799, 127868, 2), - (127870, 127891, 2), - (127904, 127946, 2), - (127951, 127955, 2), - (127968, 127984, 2), - (127988, 127988, 2), - (127992, 128062, 2), - (128064, 128064, 2), - (128066, 128252, 2), - (128255, 128317, 2), - (128331, 128334, 2), - (128336, 128359, 2), - (128378, 128378, 2), - (128405, 128406, 2), - (128420, 128420, 2), - (128507, 128591, 2), - (128640, 128709, 2), - (128716, 128716, 2), - (128720, 128722, 2), - (128725, 128727, 2), - (128747, 128748, 2), - (128756, 128764, 2), - (128992, 129003, 2), - (129292, 129338, 2), - (129340, 129349, 2), - (129351, 129400, 2), - (129402, 129483, 2), - (129485, 129535, 2), - (129648, 129652, 2), - (129656, 129658, 2), - (129664, 129670, 2), - (129680, 129704, 2), - (129712, 129718, 2), - (129728, 129730, 2), - (129744, 129750, 2), - (131072, 196605, 2), - (196608, 262141, 2), - (917760, 917999, 0), -] diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/rich/_emoji_codes.py b/venv/lib/python3.11/site-packages/pip/_vendor/rich/_emoji_codes.py deleted file mode 100644 index 1f2877b..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/rich/_emoji_codes.py +++ /dev/null @@ -1,3610 +0,0 @@ -EMOJI = { - "1st_place_medal": "🥇", - "2nd_place_medal": "🥈", - "3rd_place_medal": "🥉", - "ab_button_(blood_type)": "🆎", - "atm_sign": "🏧", - "a_button_(blood_type)": "🅰", - "afghanistan": "🇦🇫", - "albania": "🇦🇱", - "algeria": "🇩🇿", - "american_samoa": "🇦🇸", - "andorra": "🇦🇩", - "angola": "🇦🇴", - "anguilla": "🇦🇮", - "antarctica": "🇦🇶", - "antigua_&_barbuda": "🇦🇬", - "aquarius": "♒", - "argentina": "🇦🇷", - "aries": "♈", - "armenia": "🇦🇲", - "aruba": "🇦🇼", - "ascension_island": "🇦🇨", - "australia": "🇦🇺", - "austria": "🇦🇹", - "azerbaijan": "🇦🇿", - "back_arrow": "🔙", - "b_button_(blood_type)": "🅱", - "bahamas": "🇧🇸", - "bahrain": "🇧🇭", - "bangladesh": "🇧🇩", - "barbados": "🇧🇧", - "belarus": "🇧🇾", - "belgium": "🇧🇪", - "belize": "🇧🇿", - "benin": "🇧🇯", - "bermuda": "🇧🇲", - "bhutan": "🇧🇹", - "bolivia": "🇧🇴", - "bosnia_&_herzegovina": "🇧🇦", - "botswana": "🇧🇼", - "bouvet_island": "🇧🇻", - "brazil": "🇧🇷", - "british_indian_ocean_territory": "🇮🇴", - "british_virgin_islands": "🇻🇬", - "brunei": "🇧🇳", - "bulgaria": "🇧🇬", - "burkina_faso": "🇧🇫", - "burundi": "🇧🇮", - "cl_button": "🆑", - "cool_button": "🆒", - "cambodia": "🇰🇭", - "cameroon": "🇨🇲", - "canada": "🇨🇦", - "canary_islands": "🇮🇨", - "cancer": "♋", - "cape_verde": "🇨🇻", - "capricorn": "♑", - "caribbean_netherlands": "🇧🇶", - "cayman_islands": "🇰🇾", - "central_african_republic": "🇨🇫", - "ceuta_&_melilla": "🇪🇦", - "chad": "🇹🇩", - "chile": "🇨🇱", - "china": "🇨🇳", - "christmas_island": "🇨🇽", - "christmas_tree": "🎄", - "clipperton_island": "🇨🇵", - "cocos_(keeling)_islands": "🇨🇨", - "colombia": "🇨🇴", - "comoros": "🇰🇲", - "congo_-_brazzaville": "🇨🇬", - "congo_-_kinshasa": "🇨🇩", - "cook_islands": "🇨🇰", - "costa_rica": "🇨🇷", - "croatia": "🇭🇷", - "cuba": "🇨🇺", - "curaçao": "🇨🇼", - "cyprus": "🇨🇾", - "czechia": "🇨🇿", - "côte_d’ivoire": "🇨🇮", - "denmark": "🇩🇰", - "diego_garcia": "🇩🇬", - "djibouti": "🇩🇯", - "dominica": "🇩🇲", - "dominican_republic": "🇩🇴", - "end_arrow": "🔚", - "ecuador": "🇪🇨", - "egypt": "🇪🇬", - "el_salvador": "🇸🇻", - "england": "🏴\U000e0067\U000e0062\U000e0065\U000e006e\U000e0067\U000e007f", - "equatorial_guinea": "🇬🇶", - "eritrea": "🇪🇷", - "estonia": "🇪🇪", - "ethiopia": "🇪🇹", - "european_union": "🇪🇺", - "free_button": "🆓", - "falkland_islands": "🇫🇰", - "faroe_islands": "🇫🇴", - "fiji": "🇫🇯", - "finland": "🇫🇮", - "france": "🇫🇷", - "french_guiana": "🇬🇫", - "french_polynesia": "🇵🇫", - "french_southern_territories": "🇹🇫", - "gabon": "🇬🇦", - "gambia": "🇬🇲", - "gemini": "♊", - "georgia": "🇬🇪", - "germany": "🇩🇪", - "ghana": "🇬🇭", - "gibraltar": "🇬🇮", - "greece": "🇬🇷", - "greenland": "🇬🇱", - "grenada": "🇬🇩", - "guadeloupe": "🇬🇵", - "guam": "🇬🇺", - "guatemala": "🇬🇹", - "guernsey": "🇬🇬", - "guinea": "🇬🇳", - "guinea-bissau": "🇬🇼", - "guyana": "🇬🇾", - "haiti": "🇭🇹", - "heard_&_mcdonald_islands": "🇭🇲", - "honduras": "🇭🇳", - "hong_kong_sar_china": "🇭🇰", - "hungary": "🇭🇺", - "id_button": "🆔", - "iceland": "🇮🇸", - "india": "🇮🇳", - "indonesia": "🇮🇩", - "iran": "🇮🇷", - "iraq": "🇮🇶", - "ireland": "🇮🇪", - "isle_of_man": "🇮🇲", - "israel": "🇮🇱", - "italy": "🇮🇹", - "jamaica": "🇯🇲", - "japan": "🗾", - "japanese_acceptable_button": "🉑", - "japanese_application_button": "🈸", - "japanese_bargain_button": "🉐", - "japanese_castle": "🏯", - "japanese_congratulations_button": "㊗", - "japanese_discount_button": "🈹", - "japanese_dolls": "🎎", - "japanese_free_of_charge_button": "🈚", - "japanese_here_button": "🈁", - "japanese_monthly_amount_button": "🈷", - "japanese_no_vacancy_button": "🈵", - "japanese_not_free_of_charge_button": "🈶", - "japanese_open_for_business_button": "🈺", - "japanese_passing_grade_button": "🈴", - "japanese_post_office": "🏣", - "japanese_prohibited_button": "🈲", - "japanese_reserved_button": "🈯", - "japanese_secret_button": "㊙", - "japanese_service_charge_button": "🈂", - "japanese_symbol_for_beginner": "🔰", - "japanese_vacancy_button": "🈳", - "jersey": "🇯🇪", - "jordan": "🇯🇴", - "kazakhstan": "🇰🇿", - "kenya": "🇰🇪", - "kiribati": "🇰🇮", - "kosovo": "🇽🇰", - "kuwait": "🇰🇼", - "kyrgyzstan": "🇰🇬", - "laos": "🇱🇦", - "latvia": "🇱🇻", - "lebanon": "🇱🇧", - "leo": "♌", - "lesotho": "🇱🇸", - "liberia": "🇱🇷", - "libra": "♎", - "libya": "🇱🇾", - "liechtenstein": "🇱🇮", - "lithuania": "🇱🇹", - "luxembourg": "🇱🇺", - "macau_sar_china": "🇲🇴", - "macedonia": "🇲🇰", - "madagascar": "🇲🇬", - "malawi": "🇲🇼", - "malaysia": "🇲🇾", - "maldives": "🇲🇻", - "mali": "🇲🇱", - "malta": "🇲🇹", - "marshall_islands": "🇲🇭", - "martinique": "🇲🇶", - "mauritania": "🇲🇷", - "mauritius": "🇲🇺", - "mayotte": "🇾🇹", - "mexico": "🇲🇽", - "micronesia": "🇫🇲", - "moldova": "🇲🇩", - "monaco": "🇲🇨", - "mongolia": "🇲🇳", - "montenegro": "🇲🇪", - "montserrat": "🇲🇸", - "morocco": "🇲🇦", - "mozambique": "🇲🇿", - "mrs._claus": "🤶", - "mrs._claus_dark_skin_tone": "🤶🏿", - "mrs._claus_light_skin_tone": "🤶🏻", - "mrs._claus_medium-dark_skin_tone": "🤶🏾", - "mrs._claus_medium-light_skin_tone": "🤶🏼", - "mrs._claus_medium_skin_tone": "🤶🏽", - "myanmar_(burma)": "🇲🇲", - "new_button": "🆕", - "ng_button": "🆖", - "namibia": "🇳🇦", - "nauru": "🇳🇷", - "nepal": "🇳🇵", - "netherlands": "🇳🇱", - "new_caledonia": "🇳🇨", - "new_zealand": "🇳🇿", - "nicaragua": "🇳🇮", - "niger": "🇳🇪", - "nigeria": "🇳🇬", - "niue": "🇳🇺", - "norfolk_island": "🇳🇫", - "north_korea": "🇰🇵", - "northern_mariana_islands": "🇲🇵", - "norway": "🇳🇴", - "ok_button": "🆗", - "ok_hand": "👌", - "ok_hand_dark_skin_tone": "👌🏿", - "ok_hand_light_skin_tone": "👌🏻", - "ok_hand_medium-dark_skin_tone": "👌🏾", - "ok_hand_medium-light_skin_tone": "👌🏼", - "ok_hand_medium_skin_tone": "👌🏽", - "on!_arrow": "🔛", - "o_button_(blood_type)": "🅾", - "oman": "🇴🇲", - "ophiuchus": "⛎", - "p_button": "🅿", - "pakistan": "🇵🇰", - "palau": "🇵🇼", - "palestinian_territories": "🇵🇸", - "panama": "🇵🇦", - "papua_new_guinea": "🇵🇬", - "paraguay": "🇵🇾", - "peru": "🇵🇪", - "philippines": "🇵🇭", - "pisces": "♓", - "pitcairn_islands": "🇵🇳", - "poland": "🇵🇱", - "portugal": "🇵🇹", - "puerto_rico": "🇵🇷", - "qatar": "🇶🇦", - "romania": "🇷🇴", - "russia": "🇷🇺", - "rwanda": "🇷🇼", - "réunion": "🇷🇪", - "soon_arrow": "🔜", - "sos_button": "🆘", - "sagittarius": "♐", - "samoa": "🇼🇸", - "san_marino": "🇸🇲", - "santa_claus": "🎅", - "santa_claus_dark_skin_tone": "🎅🏿", - "santa_claus_light_skin_tone": "🎅🏻", - "santa_claus_medium-dark_skin_tone": "🎅🏾", - "santa_claus_medium-light_skin_tone": "🎅🏼", - "santa_claus_medium_skin_tone": "🎅🏽", - "saudi_arabia": "🇸🇦", - "scorpio": "♏", - "scotland": "🏴\U000e0067\U000e0062\U000e0073\U000e0063\U000e0074\U000e007f", - "senegal": "🇸🇳", - "serbia": "🇷🇸", - "seychelles": "🇸🇨", - "sierra_leone": "🇸🇱", - "singapore": "🇸🇬", - "sint_maarten": "🇸🇽", - "slovakia": "🇸🇰", - "slovenia": "🇸🇮", - "solomon_islands": "🇸🇧", - "somalia": "🇸🇴", - "south_africa": "🇿🇦", - "south_georgia_&_south_sandwich_islands": "🇬🇸", - "south_korea": "🇰🇷", - "south_sudan": "🇸🇸", - "spain": "🇪🇸", - "sri_lanka": "🇱🇰", - "st._barthélemy": "🇧🇱", - "st._helena": "🇸🇭", - "st._kitts_&_nevis": "🇰🇳", - "st._lucia": "🇱🇨", - "st._martin": "🇲🇫", - "st._pierre_&_miquelon": "🇵🇲", - "st._vincent_&_grenadines": "🇻🇨", - "statue_of_liberty": "🗽", - "sudan": "🇸🇩", - "suriname": "🇸🇷", - "svalbard_&_jan_mayen": "🇸🇯", - "swaziland": "🇸🇿", - "sweden": "🇸🇪", - "switzerland": "🇨🇭", - "syria": "🇸🇾", - "são_tomé_&_príncipe": "🇸🇹", - "t-rex": "🦖", - "top_arrow": "🔝", - "taiwan": "🇹🇼", - "tajikistan": "🇹🇯", - "tanzania": "🇹🇿", - "taurus": "♉", - "thailand": "🇹🇭", - "timor-leste": "🇹🇱", - "togo": "🇹🇬", - "tokelau": "🇹🇰", - "tokyo_tower": "🗼", - "tonga": "🇹🇴", - "trinidad_&_tobago": "🇹🇹", - "tristan_da_cunha": "🇹🇦", - "tunisia": "🇹🇳", - "turkey": "🦃", - "turkmenistan": "🇹🇲", - "turks_&_caicos_islands": "🇹🇨", - "tuvalu": "🇹🇻", - "u.s._outlying_islands": "🇺🇲", - "u.s._virgin_islands": "🇻🇮", - "up!_button": "🆙", - "uganda": "🇺🇬", - "ukraine": "🇺🇦", - "united_arab_emirates": "🇦🇪", - "united_kingdom": "🇬🇧", - "united_nations": "🇺🇳", - "united_states": "🇺🇸", - "uruguay": "🇺🇾", - "uzbekistan": "🇺🇿", - "vs_button": "🆚", - "vanuatu": "🇻🇺", - "vatican_city": "🇻🇦", - "venezuela": "🇻🇪", - "vietnam": "🇻🇳", - "virgo": "♍", - "wales": "🏴\U000e0067\U000e0062\U000e0077\U000e006c\U000e0073\U000e007f", - "wallis_&_futuna": "🇼🇫", - "western_sahara": "🇪🇭", - "yemen": "🇾🇪", - "zambia": "🇿🇲", - "zimbabwe": "🇿🇼", - "abacus": "🧮", - "adhesive_bandage": "🩹", - "admission_tickets": "🎟", - "adult": "🧑", - "adult_dark_skin_tone": "🧑🏿", - "adult_light_skin_tone": "🧑🏻", - "adult_medium-dark_skin_tone": "🧑🏾", - "adult_medium-light_skin_tone": "🧑🏼", - "adult_medium_skin_tone": "🧑🏽", - "aerial_tramway": "🚡", - "airplane": "✈", - "airplane_arrival": "🛬", - "airplane_departure": "🛫", - "alarm_clock": "⏰", - "alembic": "⚗", - "alien": "👽", - "alien_monster": "👾", - "ambulance": "🚑", - "american_football": "🏈", - "amphora": "🏺", - "anchor": "⚓", - "anger_symbol": "💢", - "angry_face": "😠", - "angry_face_with_horns": "👿", - "anguished_face": "😧", - "ant": "🐜", - "antenna_bars": "📶", - "anxious_face_with_sweat": "😰", - "articulated_lorry": "🚛", - "artist_palette": "🎨", - "astonished_face": "😲", - "atom_symbol": "⚛", - "auto_rickshaw": "🛺", - "automobile": "🚗", - "avocado": "🥑", - "axe": "🪓", - "baby": "👶", - "baby_angel": "👼", - "baby_angel_dark_skin_tone": "👼🏿", - "baby_angel_light_skin_tone": "👼🏻", - "baby_angel_medium-dark_skin_tone": "👼🏾", - "baby_angel_medium-light_skin_tone": "👼🏼", - "baby_angel_medium_skin_tone": "👼🏽", - "baby_bottle": "🍼", - "baby_chick": "🐤", - "baby_dark_skin_tone": "👶🏿", - "baby_light_skin_tone": "👶🏻", - "baby_medium-dark_skin_tone": "👶🏾", - "baby_medium-light_skin_tone": "👶🏼", - "baby_medium_skin_tone": "👶🏽", - "baby_symbol": "🚼", - "backhand_index_pointing_down": "👇", - "backhand_index_pointing_down_dark_skin_tone": "👇🏿", - "backhand_index_pointing_down_light_skin_tone": "👇🏻", - "backhand_index_pointing_down_medium-dark_skin_tone": "👇🏾", - "backhand_index_pointing_down_medium-light_skin_tone": "👇🏼", - "backhand_index_pointing_down_medium_skin_tone": "👇🏽", - "backhand_index_pointing_left": "👈", - "backhand_index_pointing_left_dark_skin_tone": "👈🏿", - "backhand_index_pointing_left_light_skin_tone": "👈🏻", - "backhand_index_pointing_left_medium-dark_skin_tone": "👈🏾", - "backhand_index_pointing_left_medium-light_skin_tone": "👈🏼", - "backhand_index_pointing_left_medium_skin_tone": "👈🏽", - "backhand_index_pointing_right": "👉", - "backhand_index_pointing_right_dark_skin_tone": "👉🏿", - "backhand_index_pointing_right_light_skin_tone": "👉🏻", - "backhand_index_pointing_right_medium-dark_skin_tone": "👉🏾", - "backhand_index_pointing_right_medium-light_skin_tone": "👉🏼", - "backhand_index_pointing_right_medium_skin_tone": "👉🏽", - "backhand_index_pointing_up": "👆", - "backhand_index_pointing_up_dark_skin_tone": "👆🏿", - "backhand_index_pointing_up_light_skin_tone": "👆🏻", - "backhand_index_pointing_up_medium-dark_skin_tone": "👆🏾", - "backhand_index_pointing_up_medium-light_skin_tone": "👆🏼", - "backhand_index_pointing_up_medium_skin_tone": "👆🏽", - "bacon": "🥓", - "badger": "🦡", - "badminton": "🏸", - "bagel": "🥯", - "baggage_claim": "🛄", - "baguette_bread": "🥖", - "balance_scale": "⚖", - "bald": "🦲", - "bald_man": "👨\u200d🦲", - "bald_woman": "👩\u200d🦲", - "ballet_shoes": "🩰", - "balloon": "🎈", - "ballot_box_with_ballot": "🗳", - "ballot_box_with_check": "☑", - "banana": "🍌", - "banjo": "🪕", - "bank": "🏦", - "bar_chart": "📊", - "barber_pole": "💈", - "baseball": "⚾", - "basket": "🧺", - "basketball": "🏀", - "bat": "🦇", - "bathtub": "🛁", - "battery": "🔋", - "beach_with_umbrella": "🏖", - "beaming_face_with_smiling_eyes": "😁", - "bear_face": "🐻", - "bearded_person": "🧔", - "bearded_person_dark_skin_tone": "🧔🏿", - "bearded_person_light_skin_tone": "🧔🏻", - "bearded_person_medium-dark_skin_tone": "🧔🏾", - "bearded_person_medium-light_skin_tone": "🧔🏼", - "bearded_person_medium_skin_tone": "🧔🏽", - "beating_heart": "💓", - "bed": "🛏", - "beer_mug": "🍺", - "bell": "🔔", - "bell_with_slash": "🔕", - "bellhop_bell": "🛎", - "bento_box": "🍱", - "beverage_box": "🧃", - "bicycle": "🚲", - "bikini": "👙", - "billed_cap": "🧢", - "biohazard": "☣", - "bird": "🐦", - "birthday_cake": "🎂", - "black_circle": "⚫", - "black_flag": "🏴", - "black_heart": "🖤", - "black_large_square": "⬛", - "black_medium-small_square": "◾", - "black_medium_square": "◼", - "black_nib": "✒", - "black_small_square": "▪", - "black_square_button": "🔲", - "blond-haired_man": "👱\u200d♂️", - "blond-haired_man_dark_skin_tone": "👱🏿\u200d♂️", - "blond-haired_man_light_skin_tone": "👱🏻\u200d♂️", - "blond-haired_man_medium-dark_skin_tone": "👱🏾\u200d♂️", - "blond-haired_man_medium-light_skin_tone": "👱🏼\u200d♂️", - "blond-haired_man_medium_skin_tone": "👱🏽\u200d♂️", - "blond-haired_person": "👱", - "blond-haired_person_dark_skin_tone": "👱🏿", - "blond-haired_person_light_skin_tone": "👱🏻", - "blond-haired_person_medium-dark_skin_tone": "👱🏾", - "blond-haired_person_medium-light_skin_tone": "👱🏼", - "blond-haired_person_medium_skin_tone": "👱🏽", - "blond-haired_woman": "👱\u200d♀️", - "blond-haired_woman_dark_skin_tone": "👱🏿\u200d♀️", - "blond-haired_woman_light_skin_tone": "👱🏻\u200d♀️", - "blond-haired_woman_medium-dark_skin_tone": "👱🏾\u200d♀️", - "blond-haired_woman_medium-light_skin_tone": "👱🏼\u200d♀️", - "blond-haired_woman_medium_skin_tone": "👱🏽\u200d♀️", - "blossom": "🌼", - "blowfish": "🐡", - "blue_book": "📘", - "blue_circle": "🔵", - "blue_heart": "💙", - "blue_square": "🟦", - "boar": "🐗", - "bomb": "💣", - "bone": "🦴", - "bookmark": "🔖", - "bookmark_tabs": "📑", - "books": "📚", - "bottle_with_popping_cork": "🍾", - "bouquet": "💐", - "bow_and_arrow": "🏹", - "bowl_with_spoon": "🥣", - "bowling": "🎳", - "boxing_glove": "🥊", - "boy": "👦", - "boy_dark_skin_tone": "👦🏿", - "boy_light_skin_tone": "👦🏻", - "boy_medium-dark_skin_tone": "👦🏾", - "boy_medium-light_skin_tone": "👦🏼", - "boy_medium_skin_tone": "👦🏽", - "brain": "🧠", - "bread": "🍞", - "breast-feeding": "🤱", - "breast-feeding_dark_skin_tone": "🤱🏿", - "breast-feeding_light_skin_tone": "🤱🏻", - "breast-feeding_medium-dark_skin_tone": "🤱🏾", - "breast-feeding_medium-light_skin_tone": "🤱🏼", - "breast-feeding_medium_skin_tone": "🤱🏽", - "brick": "🧱", - "bride_with_veil": "👰", - "bride_with_veil_dark_skin_tone": "👰🏿", - "bride_with_veil_light_skin_tone": "👰🏻", - "bride_with_veil_medium-dark_skin_tone": "👰🏾", - "bride_with_veil_medium-light_skin_tone": "👰🏼", - "bride_with_veil_medium_skin_tone": "👰🏽", - "bridge_at_night": "🌉", - "briefcase": "💼", - "briefs": "🩲", - "bright_button": "🔆", - "broccoli": "🥦", - "broken_heart": "💔", - "broom": "🧹", - "brown_circle": "🟤", - "brown_heart": "🤎", - "brown_square": "🟫", - "bug": "🐛", - "building_construction": "🏗", - "bullet_train": "🚅", - "burrito": "🌯", - "bus": "🚌", - "bus_stop": "🚏", - "bust_in_silhouette": "👤", - "busts_in_silhouette": "👥", - "butter": "🧈", - "butterfly": "🦋", - "cactus": "🌵", - "calendar": "📆", - "call_me_hand": "🤙", - "call_me_hand_dark_skin_tone": "🤙🏿", - "call_me_hand_light_skin_tone": "🤙🏻", - "call_me_hand_medium-dark_skin_tone": "🤙🏾", - "call_me_hand_medium-light_skin_tone": "🤙🏼", - "call_me_hand_medium_skin_tone": "🤙🏽", - "camel": "🐫", - "camera": "📷", - "camera_with_flash": "📸", - "camping": "🏕", - "candle": "🕯", - "candy": "🍬", - "canned_food": "🥫", - "canoe": "🛶", - "card_file_box": "🗃", - "card_index": "📇", - "card_index_dividers": "🗂", - "carousel_horse": "🎠", - "carp_streamer": "🎏", - "carrot": "🥕", - "castle": "🏰", - "cat": "🐱", - "cat_face": "🐱", - "cat_face_with_tears_of_joy": "😹", - "cat_face_with_wry_smile": "😼", - "chains": "⛓", - "chair": "🪑", - "chart_decreasing": "📉", - "chart_increasing": "📈", - "chart_increasing_with_yen": "💹", - "cheese_wedge": "🧀", - "chequered_flag": "🏁", - "cherries": "🍒", - "cherry_blossom": "🌸", - "chess_pawn": "♟", - "chestnut": "🌰", - "chicken": "🐔", - "child": "🧒", - "child_dark_skin_tone": "🧒🏿", - "child_light_skin_tone": "🧒🏻", - "child_medium-dark_skin_tone": "🧒🏾", - "child_medium-light_skin_tone": "🧒🏼", - "child_medium_skin_tone": "🧒🏽", - "children_crossing": "🚸", - "chipmunk": "🐿", - "chocolate_bar": "🍫", - "chopsticks": "🥢", - "church": "⛪", - "cigarette": "🚬", - "cinema": "🎦", - "circled_m": "Ⓜ", - "circus_tent": "🎪", - "cityscape": "🏙", - "cityscape_at_dusk": "🌆", - "clamp": "🗜", - "clapper_board": "🎬", - "clapping_hands": "👏", - "clapping_hands_dark_skin_tone": "👏🏿", - "clapping_hands_light_skin_tone": "👏🏻", - "clapping_hands_medium-dark_skin_tone": "👏🏾", - "clapping_hands_medium-light_skin_tone": "👏🏼", - "clapping_hands_medium_skin_tone": "👏🏽", - "classical_building": "🏛", - "clinking_beer_mugs": "🍻", - "clinking_glasses": "🥂", - "clipboard": "📋", - "clockwise_vertical_arrows": "🔃", - "closed_book": "📕", - "closed_mailbox_with_lowered_flag": "📪", - "closed_mailbox_with_raised_flag": "📫", - "closed_umbrella": "🌂", - "cloud": "☁", - "cloud_with_lightning": "🌩", - "cloud_with_lightning_and_rain": "⛈", - "cloud_with_rain": "🌧", - "cloud_with_snow": "🌨", - "clown_face": "🤡", - "club_suit": "♣", - "clutch_bag": "👝", - "coat": "🧥", - "cocktail_glass": "🍸", - "coconut": "🥥", - "coffin": "⚰", - "cold_face": "🥶", - "collision": "💥", - "comet": "☄", - "compass": "🧭", - "computer_disk": "💽", - "computer_mouse": "🖱", - "confetti_ball": "🎊", - "confounded_face": "😖", - "confused_face": "😕", - "construction": "🚧", - "construction_worker": "👷", - "construction_worker_dark_skin_tone": "👷🏿", - "construction_worker_light_skin_tone": "👷🏻", - "construction_worker_medium-dark_skin_tone": "👷🏾", - "construction_worker_medium-light_skin_tone": "👷🏼", - "construction_worker_medium_skin_tone": "👷🏽", - "control_knobs": "🎛", - "convenience_store": "🏪", - "cooked_rice": "🍚", - "cookie": "🍪", - "cooking": "🍳", - "copyright": "©", - "couch_and_lamp": "🛋", - "counterclockwise_arrows_button": "🔄", - "couple_with_heart": "💑", - "couple_with_heart_man_man": "👨\u200d❤️\u200d👨", - "couple_with_heart_woman_man": "👩\u200d❤️\u200d👨", - "couple_with_heart_woman_woman": "👩\u200d❤️\u200d👩", - "cow": "🐮", - "cow_face": "🐮", - "cowboy_hat_face": "🤠", - "crab": "🦀", - "crayon": "🖍", - "credit_card": "💳", - "crescent_moon": "🌙", - "cricket": "🦗", - "cricket_game": "🏏", - "crocodile": "🐊", - "croissant": "🥐", - "cross_mark": "❌", - "cross_mark_button": "❎", - "crossed_fingers": "🤞", - "crossed_fingers_dark_skin_tone": "🤞🏿", - "crossed_fingers_light_skin_tone": "🤞🏻", - "crossed_fingers_medium-dark_skin_tone": "🤞🏾", - "crossed_fingers_medium-light_skin_tone": "🤞🏼", - "crossed_fingers_medium_skin_tone": "🤞🏽", - "crossed_flags": "🎌", - "crossed_swords": "⚔", - "crown": "👑", - "crying_cat_face": "😿", - "crying_face": "😢", - "crystal_ball": "🔮", - "cucumber": "🥒", - "cupcake": "🧁", - "cup_with_straw": "🥤", - "curling_stone": "🥌", - "curly_hair": "🦱", - "curly-haired_man": "👨\u200d🦱", - "curly-haired_woman": "👩\u200d🦱", - "curly_loop": "➰", - "currency_exchange": "💱", - "curry_rice": "🍛", - "custard": "🍮", - "customs": "🛃", - "cut_of_meat": "🥩", - "cyclone": "🌀", - "dagger": "🗡", - "dango": "🍡", - "dashing_away": "💨", - "deaf_person": "🧏", - "deciduous_tree": "🌳", - "deer": "🦌", - "delivery_truck": "🚚", - "department_store": "🏬", - "derelict_house": "🏚", - "desert": "🏜", - "desert_island": "🏝", - "desktop_computer": "🖥", - "detective": "🕵", - "detective_dark_skin_tone": "🕵🏿", - "detective_light_skin_tone": "🕵🏻", - "detective_medium-dark_skin_tone": "🕵🏾", - "detective_medium-light_skin_tone": "🕵🏼", - "detective_medium_skin_tone": "🕵🏽", - "diamond_suit": "♦", - "diamond_with_a_dot": "💠", - "dim_button": "🔅", - "direct_hit": "🎯", - "disappointed_face": "😞", - "diving_mask": "🤿", - "diya_lamp": "🪔", - "dizzy": "💫", - "dizzy_face": "😵", - "dna": "🧬", - "dog": "🐶", - "dog_face": "🐶", - "dollar_banknote": "💵", - "dolphin": "🐬", - "door": "🚪", - "dotted_six-pointed_star": "🔯", - "double_curly_loop": "➿", - "double_exclamation_mark": "‼", - "doughnut": "🍩", - "dove": "🕊", - "down-left_arrow": "↙", - "down-right_arrow": "↘", - "down_arrow": "⬇", - "downcast_face_with_sweat": "😓", - "downwards_button": "🔽", - "dragon": "🐉", - "dragon_face": "🐲", - "dress": "👗", - "drooling_face": "🤤", - "drop_of_blood": "🩸", - "droplet": "💧", - "drum": "🥁", - "duck": "🦆", - "dumpling": "🥟", - "dvd": "📀", - "e-mail": "📧", - "eagle": "🦅", - "ear": "👂", - "ear_dark_skin_tone": "👂🏿", - "ear_light_skin_tone": "👂🏻", - "ear_medium-dark_skin_tone": "👂🏾", - "ear_medium-light_skin_tone": "👂🏼", - "ear_medium_skin_tone": "👂🏽", - "ear_of_corn": "🌽", - "ear_with_hearing_aid": "🦻", - "egg": "🍳", - "eggplant": "🍆", - "eight-pointed_star": "✴", - "eight-spoked_asterisk": "✳", - "eight-thirty": "🕣", - "eight_o’clock": "🕗", - "eject_button": "⏏", - "electric_plug": "🔌", - "elephant": "🐘", - "eleven-thirty": "🕦", - "eleven_o’clock": "🕚", - "elf": "🧝", - "elf_dark_skin_tone": "🧝🏿", - "elf_light_skin_tone": "🧝🏻", - "elf_medium-dark_skin_tone": "🧝🏾", - "elf_medium-light_skin_tone": "🧝🏼", - "elf_medium_skin_tone": "🧝🏽", - "envelope": "✉", - "envelope_with_arrow": "📩", - "euro_banknote": "💶", - "evergreen_tree": "🌲", - "ewe": "🐑", - "exclamation_mark": "❗", - "exclamation_question_mark": "⁉", - "exploding_head": "🤯", - "expressionless_face": "😑", - "eye": "👁", - "eye_in_speech_bubble": "👁️\u200d🗨️", - "eyes": "👀", - "face_blowing_a_kiss": "😘", - "face_savoring_food": "😋", - "face_screaming_in_fear": "😱", - "face_vomiting": "🤮", - "face_with_hand_over_mouth": "🤭", - "face_with_head-bandage": "🤕", - "face_with_medical_mask": "😷", - "face_with_monocle": "🧐", - "face_with_open_mouth": "😮", - "face_with_raised_eyebrow": "🤨", - "face_with_rolling_eyes": "🙄", - "face_with_steam_from_nose": "😤", - "face_with_symbols_on_mouth": "🤬", - "face_with_tears_of_joy": "😂", - "face_with_thermometer": "🤒", - "face_with_tongue": "😛", - "face_without_mouth": "😶", - "factory": "🏭", - "fairy": "🧚", - "fairy_dark_skin_tone": "🧚🏿", - "fairy_light_skin_tone": "🧚🏻", - "fairy_medium-dark_skin_tone": "🧚🏾", - "fairy_medium-light_skin_tone": "🧚🏼", - "fairy_medium_skin_tone": "🧚🏽", - "falafel": "🧆", - "fallen_leaf": "🍂", - "family": "👪", - "family_man_boy": "👨\u200d👦", - "family_man_boy_boy": "👨\u200d👦\u200d👦", - "family_man_girl": "👨\u200d👧", - "family_man_girl_boy": "👨\u200d👧\u200d👦", - "family_man_girl_girl": "👨\u200d👧\u200d👧", - "family_man_man_boy": "👨\u200d👨\u200d👦", - "family_man_man_boy_boy": "👨\u200d👨\u200d👦\u200d👦", - "family_man_man_girl": "👨\u200d👨\u200d👧", - "family_man_man_girl_boy": "👨\u200d👨\u200d👧\u200d👦", - "family_man_man_girl_girl": "👨\u200d👨\u200d👧\u200d👧", - "family_man_woman_boy": "👨\u200d👩\u200d👦", - "family_man_woman_boy_boy": "👨\u200d👩\u200d👦\u200d👦", - "family_man_woman_girl": "👨\u200d👩\u200d👧", - "family_man_woman_girl_boy": "👨\u200d👩\u200d👧\u200d👦", - "family_man_woman_girl_girl": "👨\u200d👩\u200d👧\u200d👧", - "family_woman_boy": "👩\u200d👦", - "family_woman_boy_boy": "👩\u200d👦\u200d👦", - "family_woman_girl": "👩\u200d👧", - "family_woman_girl_boy": "👩\u200d👧\u200d👦", - "family_woman_girl_girl": "👩\u200d👧\u200d👧", - "family_woman_woman_boy": "👩\u200d👩\u200d👦", - "family_woman_woman_boy_boy": "👩\u200d👩\u200d👦\u200d👦", - "family_woman_woman_girl": "👩\u200d👩\u200d👧", - "family_woman_woman_girl_boy": "👩\u200d👩\u200d👧\u200d👦", - "family_woman_woman_girl_girl": "👩\u200d👩\u200d👧\u200d👧", - "fast-forward_button": "⏩", - "fast_down_button": "⏬", - "fast_reverse_button": "⏪", - "fast_up_button": "⏫", - "fax_machine": "📠", - "fearful_face": "😨", - "female_sign": "♀", - "ferris_wheel": "🎡", - "ferry": "⛴", - "field_hockey": "🏑", - "file_cabinet": "🗄", - "file_folder": "📁", - "film_frames": "🎞", - "film_projector": "📽", - "fire": "🔥", - "fire_extinguisher": "🧯", - "firecracker": "🧨", - "fire_engine": "🚒", - "fireworks": "🎆", - "first_quarter_moon": "🌓", - "first_quarter_moon_face": "🌛", - "fish": "🐟", - "fish_cake_with_swirl": "🍥", - "fishing_pole": "🎣", - "five-thirty": "🕠", - "five_o’clock": "🕔", - "flag_in_hole": "⛳", - "flamingo": "🦩", - "flashlight": "🔦", - "flat_shoe": "🥿", - "fleur-de-lis": "⚜", - "flexed_biceps": "💪", - "flexed_biceps_dark_skin_tone": "💪🏿", - "flexed_biceps_light_skin_tone": "💪🏻", - "flexed_biceps_medium-dark_skin_tone": "💪🏾", - "flexed_biceps_medium-light_skin_tone": "💪🏼", - "flexed_biceps_medium_skin_tone": "💪🏽", - "floppy_disk": "💾", - "flower_playing_cards": "🎴", - "flushed_face": "😳", - "flying_disc": "🥏", - "flying_saucer": "🛸", - "fog": "🌫", - "foggy": "🌁", - "folded_hands": "🙏", - "folded_hands_dark_skin_tone": "🙏🏿", - "folded_hands_light_skin_tone": "🙏🏻", - "folded_hands_medium-dark_skin_tone": "🙏🏾", - "folded_hands_medium-light_skin_tone": "🙏🏼", - "folded_hands_medium_skin_tone": "🙏🏽", - "foot": "🦶", - "footprints": "👣", - "fork_and_knife": "🍴", - "fork_and_knife_with_plate": "🍽", - "fortune_cookie": "🥠", - "fountain": "⛲", - "fountain_pen": "🖋", - "four-thirty": "🕟", - "four_leaf_clover": "🍀", - "four_o’clock": "🕓", - "fox_face": "🦊", - "framed_picture": "🖼", - "french_fries": "🍟", - "fried_shrimp": "🍤", - "frog_face": "🐸", - "front-facing_baby_chick": "🐥", - "frowning_face": "☹", - "frowning_face_with_open_mouth": "😦", - "fuel_pump": "⛽", - "full_moon": "🌕", - "full_moon_face": "🌝", - "funeral_urn": "⚱", - "game_die": "🎲", - "garlic": "🧄", - "gear": "⚙", - "gem_stone": "💎", - "genie": "🧞", - "ghost": "👻", - "giraffe": "🦒", - "girl": "👧", - "girl_dark_skin_tone": "👧🏿", - "girl_light_skin_tone": "👧🏻", - "girl_medium-dark_skin_tone": "👧🏾", - "girl_medium-light_skin_tone": "👧🏼", - "girl_medium_skin_tone": "👧🏽", - "glass_of_milk": "🥛", - "glasses": "👓", - "globe_showing_americas": "🌎", - "globe_showing_asia-australia": "🌏", - "globe_showing_europe-africa": "🌍", - "globe_with_meridians": "🌐", - "gloves": "🧤", - "glowing_star": "🌟", - "goal_net": "🥅", - "goat": "🐐", - "goblin": "👺", - "goggles": "🥽", - "gorilla": "🦍", - "graduation_cap": "🎓", - "grapes": "🍇", - "green_apple": "🍏", - "green_book": "📗", - "green_circle": "🟢", - "green_heart": "💚", - "green_salad": "🥗", - "green_square": "🟩", - "grimacing_face": "😬", - "grinning_cat_face": "😺", - "grinning_cat_face_with_smiling_eyes": "😸", - "grinning_face": "😀", - "grinning_face_with_big_eyes": "😃", - "grinning_face_with_smiling_eyes": "😄", - "grinning_face_with_sweat": "😅", - "grinning_squinting_face": "😆", - "growing_heart": "💗", - "guard": "💂", - "guard_dark_skin_tone": "💂🏿", - "guard_light_skin_tone": "💂🏻", - "guard_medium-dark_skin_tone": "💂🏾", - "guard_medium-light_skin_tone": "💂🏼", - "guard_medium_skin_tone": "💂🏽", - "guide_dog": "🦮", - "guitar": "🎸", - "hamburger": "🍔", - "hammer": "🔨", - "hammer_and_pick": "⚒", - "hammer_and_wrench": "🛠", - "hamster_face": "🐹", - "hand_with_fingers_splayed": "🖐", - "hand_with_fingers_splayed_dark_skin_tone": "🖐🏿", - "hand_with_fingers_splayed_light_skin_tone": "🖐🏻", - "hand_with_fingers_splayed_medium-dark_skin_tone": "🖐🏾", - "hand_with_fingers_splayed_medium-light_skin_tone": "🖐🏼", - "hand_with_fingers_splayed_medium_skin_tone": "🖐🏽", - "handbag": "👜", - "handshake": "🤝", - "hatching_chick": "🐣", - "headphone": "🎧", - "hear-no-evil_monkey": "🙉", - "heart_decoration": "💟", - "heart_suit": "♥", - "heart_with_arrow": "💘", - "heart_with_ribbon": "💝", - "heavy_check_mark": "✔", - "heavy_division_sign": "➗", - "heavy_dollar_sign": "💲", - "heavy_heart_exclamation": "❣", - "heavy_large_circle": "⭕", - "heavy_minus_sign": "➖", - "heavy_multiplication_x": "✖", - "heavy_plus_sign": "➕", - "hedgehog": "🦔", - "helicopter": "🚁", - "herb": "🌿", - "hibiscus": "🌺", - "high-heeled_shoe": "👠", - "high-speed_train": "🚄", - "high_voltage": "⚡", - "hiking_boot": "🥾", - "hindu_temple": "🛕", - "hippopotamus": "🦛", - "hole": "🕳", - "honey_pot": "🍯", - "honeybee": "🐝", - "horizontal_traffic_light": "🚥", - "horse": "🐴", - "horse_face": "🐴", - "horse_racing": "🏇", - "horse_racing_dark_skin_tone": "🏇🏿", - "horse_racing_light_skin_tone": "🏇🏻", - "horse_racing_medium-dark_skin_tone": "🏇🏾", - "horse_racing_medium-light_skin_tone": "🏇🏼", - "horse_racing_medium_skin_tone": "🏇🏽", - "hospital": "🏥", - "hot_beverage": "☕", - "hot_dog": "🌭", - "hot_face": "🥵", - "hot_pepper": "🌶", - "hot_springs": "♨", - "hotel": "🏨", - "hourglass_done": "⌛", - "hourglass_not_done": "⏳", - "house": "🏠", - "house_with_garden": "🏡", - "houses": "🏘", - "hugging_face": "🤗", - "hundred_points": "💯", - "hushed_face": "😯", - "ice": "🧊", - "ice_cream": "🍨", - "ice_hockey": "🏒", - "ice_skate": "⛸", - "inbox_tray": "📥", - "incoming_envelope": "📨", - "index_pointing_up": "☝", - "index_pointing_up_dark_skin_tone": "☝🏿", - "index_pointing_up_light_skin_tone": "☝🏻", - "index_pointing_up_medium-dark_skin_tone": "☝🏾", - "index_pointing_up_medium-light_skin_tone": "☝🏼", - "index_pointing_up_medium_skin_tone": "☝🏽", - "infinity": "♾", - "information": "ℹ", - "input_latin_letters": "🔤", - "input_latin_lowercase": "🔡", - "input_latin_uppercase": "🔠", - "input_numbers": "🔢", - "input_symbols": "🔣", - "jack-o-lantern": "🎃", - "jeans": "👖", - "jigsaw": "🧩", - "joker": "🃏", - "joystick": "🕹", - "kaaba": "🕋", - "kangaroo": "🦘", - "key": "🔑", - "keyboard": "⌨", - "keycap_#": "#️⃣", - "keycap_*": "*️⃣", - "keycap_0": "0️⃣", - "keycap_1": "1️⃣", - "keycap_10": "🔟", - "keycap_2": "2️⃣", - "keycap_3": "3️⃣", - "keycap_4": "4️⃣", - "keycap_5": "5️⃣", - "keycap_6": "6️⃣", - "keycap_7": "7️⃣", - "keycap_8": "8️⃣", - "keycap_9": "9️⃣", - "kick_scooter": "🛴", - "kimono": "👘", - "kiss": "💋", - "kiss_man_man": "👨\u200d❤️\u200d💋\u200d👨", - "kiss_mark": "💋", - "kiss_woman_man": "👩\u200d❤️\u200d💋\u200d👨", - "kiss_woman_woman": "👩\u200d❤️\u200d💋\u200d👩", - "kissing_cat_face": "😽", - "kissing_face": "😗", - "kissing_face_with_closed_eyes": "😚", - "kissing_face_with_smiling_eyes": "😙", - "kitchen_knife": "🔪", - "kite": "🪁", - "kiwi_fruit": "🥝", - "koala": "🐨", - "lab_coat": "🥼", - "label": "🏷", - "lacrosse": "🥍", - "lady_beetle": "🐞", - "laptop_computer": "💻", - "large_blue_diamond": "🔷", - "large_orange_diamond": "🔶", - "last_quarter_moon": "🌗", - "last_quarter_moon_face": "🌜", - "last_track_button": "⏮", - "latin_cross": "✝", - "leaf_fluttering_in_wind": "🍃", - "leafy_green": "🥬", - "ledger": "📒", - "left-facing_fist": "🤛", - "left-facing_fist_dark_skin_tone": "🤛🏿", - "left-facing_fist_light_skin_tone": "🤛🏻", - "left-facing_fist_medium-dark_skin_tone": "🤛🏾", - "left-facing_fist_medium-light_skin_tone": "🤛🏼", - "left-facing_fist_medium_skin_tone": "🤛🏽", - "left-right_arrow": "↔", - "left_arrow": "⬅", - "left_arrow_curving_right": "↪", - "left_luggage": "🛅", - "left_speech_bubble": "🗨", - "leg": "🦵", - "lemon": "🍋", - "leopard": "🐆", - "level_slider": "🎚", - "light_bulb": "💡", - "light_rail": "🚈", - "link": "🔗", - "linked_paperclips": "🖇", - "lion_face": "🦁", - "lipstick": "💄", - "litter_in_bin_sign": "🚮", - "lizard": "🦎", - "llama": "🦙", - "lobster": "🦞", - "locked": "🔒", - "locked_with_key": "🔐", - "locked_with_pen": "🔏", - "locomotive": "🚂", - "lollipop": "🍭", - "lotion_bottle": "🧴", - "loudly_crying_face": "😭", - "loudspeaker": "📢", - "love-you_gesture": "🤟", - "love-you_gesture_dark_skin_tone": "🤟🏿", - "love-you_gesture_light_skin_tone": "🤟🏻", - "love-you_gesture_medium-dark_skin_tone": "🤟🏾", - "love-you_gesture_medium-light_skin_tone": "🤟🏼", - "love-you_gesture_medium_skin_tone": "🤟🏽", - "love_hotel": "🏩", - "love_letter": "💌", - "luggage": "🧳", - "lying_face": "🤥", - "mage": "🧙", - "mage_dark_skin_tone": "🧙🏿", - "mage_light_skin_tone": "🧙🏻", - "mage_medium-dark_skin_tone": "🧙🏾", - "mage_medium-light_skin_tone": "🧙🏼", - "mage_medium_skin_tone": "🧙🏽", - "magnet": "🧲", - "magnifying_glass_tilted_left": "🔍", - "magnifying_glass_tilted_right": "🔎", - "mahjong_red_dragon": "🀄", - "male_sign": "♂", - "man": "👨", - "man_and_woman_holding_hands": "👫", - "man_artist": "👨\u200d🎨", - "man_artist_dark_skin_tone": "👨🏿\u200d🎨", - "man_artist_light_skin_tone": "👨🏻\u200d🎨", - "man_artist_medium-dark_skin_tone": "👨🏾\u200d🎨", - "man_artist_medium-light_skin_tone": "👨🏼\u200d🎨", - "man_artist_medium_skin_tone": "👨🏽\u200d🎨", - "man_astronaut": "👨\u200d🚀", - "man_astronaut_dark_skin_tone": "👨🏿\u200d🚀", - "man_astronaut_light_skin_tone": "👨🏻\u200d🚀", - "man_astronaut_medium-dark_skin_tone": "👨🏾\u200d🚀", - "man_astronaut_medium-light_skin_tone": "👨🏼\u200d🚀", - "man_astronaut_medium_skin_tone": "👨🏽\u200d🚀", - "man_biking": "🚴\u200d♂️", - "man_biking_dark_skin_tone": "🚴🏿\u200d♂️", - "man_biking_light_skin_tone": "🚴🏻\u200d♂️", - "man_biking_medium-dark_skin_tone": "🚴🏾\u200d♂️", - "man_biking_medium-light_skin_tone": "🚴🏼\u200d♂️", - "man_biking_medium_skin_tone": "🚴🏽\u200d♂️", - "man_bouncing_ball": "⛹️\u200d♂️", - "man_bouncing_ball_dark_skin_tone": "⛹🏿\u200d♂️", - "man_bouncing_ball_light_skin_tone": "⛹🏻\u200d♂️", - "man_bouncing_ball_medium-dark_skin_tone": "⛹🏾\u200d♂️", - "man_bouncing_ball_medium-light_skin_tone": "⛹🏼\u200d♂️", - "man_bouncing_ball_medium_skin_tone": "⛹🏽\u200d♂️", - "man_bowing": "🙇\u200d♂️", - "man_bowing_dark_skin_tone": "🙇🏿\u200d♂️", - "man_bowing_light_skin_tone": "🙇🏻\u200d♂️", - "man_bowing_medium-dark_skin_tone": "🙇🏾\u200d♂️", - "man_bowing_medium-light_skin_tone": "🙇🏼\u200d♂️", - "man_bowing_medium_skin_tone": "🙇🏽\u200d♂️", - "man_cartwheeling": "🤸\u200d♂️", - "man_cartwheeling_dark_skin_tone": "🤸🏿\u200d♂️", - "man_cartwheeling_light_skin_tone": "🤸🏻\u200d♂️", - "man_cartwheeling_medium-dark_skin_tone": "🤸🏾\u200d♂️", - "man_cartwheeling_medium-light_skin_tone": "🤸🏼\u200d♂️", - "man_cartwheeling_medium_skin_tone": "🤸🏽\u200d♂️", - "man_climbing": "🧗\u200d♂️", - "man_climbing_dark_skin_tone": "🧗🏿\u200d♂️", - "man_climbing_light_skin_tone": "🧗🏻\u200d♂️", - "man_climbing_medium-dark_skin_tone": "🧗🏾\u200d♂️", - "man_climbing_medium-light_skin_tone": "🧗🏼\u200d♂️", - "man_climbing_medium_skin_tone": "🧗🏽\u200d♂️", - "man_construction_worker": "👷\u200d♂️", - "man_construction_worker_dark_skin_tone": "👷🏿\u200d♂️", - "man_construction_worker_light_skin_tone": "👷🏻\u200d♂️", - "man_construction_worker_medium-dark_skin_tone": "👷🏾\u200d♂️", - "man_construction_worker_medium-light_skin_tone": "👷🏼\u200d♂️", - "man_construction_worker_medium_skin_tone": "👷🏽\u200d♂️", - "man_cook": "👨\u200d🍳", - "man_cook_dark_skin_tone": "👨🏿\u200d🍳", - "man_cook_light_skin_tone": "👨🏻\u200d🍳", - "man_cook_medium-dark_skin_tone": "👨🏾\u200d🍳", - "man_cook_medium-light_skin_tone": "👨🏼\u200d🍳", - "man_cook_medium_skin_tone": "👨🏽\u200d🍳", - "man_dancing": "🕺", - "man_dancing_dark_skin_tone": "🕺🏿", - "man_dancing_light_skin_tone": "🕺🏻", - "man_dancing_medium-dark_skin_tone": "🕺🏾", - "man_dancing_medium-light_skin_tone": "🕺🏼", - "man_dancing_medium_skin_tone": "🕺🏽", - "man_dark_skin_tone": "👨🏿", - "man_detective": "🕵️\u200d♂️", - "man_detective_dark_skin_tone": "🕵🏿\u200d♂️", - "man_detective_light_skin_tone": "🕵🏻\u200d♂️", - "man_detective_medium-dark_skin_tone": "🕵🏾\u200d♂️", - "man_detective_medium-light_skin_tone": "🕵🏼\u200d♂️", - "man_detective_medium_skin_tone": "🕵🏽\u200d♂️", - "man_elf": "🧝\u200d♂️", - "man_elf_dark_skin_tone": "🧝🏿\u200d♂️", - "man_elf_light_skin_tone": "🧝🏻\u200d♂️", - "man_elf_medium-dark_skin_tone": "🧝🏾\u200d♂️", - "man_elf_medium-light_skin_tone": "🧝🏼\u200d♂️", - "man_elf_medium_skin_tone": "🧝🏽\u200d♂️", - "man_facepalming": "🤦\u200d♂️", - "man_facepalming_dark_skin_tone": "🤦🏿\u200d♂️", - "man_facepalming_light_skin_tone": "🤦🏻\u200d♂️", - "man_facepalming_medium-dark_skin_tone": "🤦🏾\u200d♂️", - "man_facepalming_medium-light_skin_tone": "🤦🏼\u200d♂️", - "man_facepalming_medium_skin_tone": "🤦🏽\u200d♂️", - "man_factory_worker": "👨\u200d🏭", - "man_factory_worker_dark_skin_tone": "👨🏿\u200d🏭", - "man_factory_worker_light_skin_tone": "👨🏻\u200d🏭", - "man_factory_worker_medium-dark_skin_tone": "👨🏾\u200d🏭", - "man_factory_worker_medium-light_skin_tone": "👨🏼\u200d🏭", - "man_factory_worker_medium_skin_tone": "👨🏽\u200d🏭", - "man_fairy": "🧚\u200d♂️", - "man_fairy_dark_skin_tone": "🧚🏿\u200d♂️", - "man_fairy_light_skin_tone": "🧚🏻\u200d♂️", - "man_fairy_medium-dark_skin_tone": "🧚🏾\u200d♂️", - "man_fairy_medium-light_skin_tone": "🧚🏼\u200d♂️", - "man_fairy_medium_skin_tone": "🧚🏽\u200d♂️", - "man_farmer": "👨\u200d🌾", - "man_farmer_dark_skin_tone": "👨🏿\u200d🌾", - "man_farmer_light_skin_tone": "👨🏻\u200d🌾", - "man_farmer_medium-dark_skin_tone": "👨🏾\u200d🌾", - "man_farmer_medium-light_skin_tone": "👨🏼\u200d🌾", - "man_farmer_medium_skin_tone": "👨🏽\u200d🌾", - "man_firefighter": "👨\u200d🚒", - "man_firefighter_dark_skin_tone": "👨🏿\u200d🚒", - "man_firefighter_light_skin_tone": "👨🏻\u200d🚒", - "man_firefighter_medium-dark_skin_tone": "👨🏾\u200d🚒", - "man_firefighter_medium-light_skin_tone": "👨🏼\u200d🚒", - "man_firefighter_medium_skin_tone": "👨🏽\u200d🚒", - "man_frowning": "🙍\u200d♂️", - "man_frowning_dark_skin_tone": "🙍🏿\u200d♂️", - "man_frowning_light_skin_tone": "🙍🏻\u200d♂️", - "man_frowning_medium-dark_skin_tone": "🙍🏾\u200d♂️", - "man_frowning_medium-light_skin_tone": "🙍🏼\u200d♂️", - "man_frowning_medium_skin_tone": "🙍🏽\u200d♂️", - "man_genie": "🧞\u200d♂️", - "man_gesturing_no": "🙅\u200d♂️", - "man_gesturing_no_dark_skin_tone": "🙅🏿\u200d♂️", - "man_gesturing_no_light_skin_tone": "🙅🏻\u200d♂️", - "man_gesturing_no_medium-dark_skin_tone": "🙅🏾\u200d♂️", - "man_gesturing_no_medium-light_skin_tone": "🙅🏼\u200d♂️", - "man_gesturing_no_medium_skin_tone": "🙅🏽\u200d♂️", - "man_gesturing_ok": "🙆\u200d♂️", - "man_gesturing_ok_dark_skin_tone": "🙆🏿\u200d♂️", - "man_gesturing_ok_light_skin_tone": "🙆🏻\u200d♂️", - "man_gesturing_ok_medium-dark_skin_tone": "🙆🏾\u200d♂️", - "man_gesturing_ok_medium-light_skin_tone": "🙆🏼\u200d♂️", - "man_gesturing_ok_medium_skin_tone": "🙆🏽\u200d♂️", - "man_getting_haircut": "💇\u200d♂️", - "man_getting_haircut_dark_skin_tone": "💇🏿\u200d♂️", - "man_getting_haircut_light_skin_tone": "💇🏻\u200d♂️", - "man_getting_haircut_medium-dark_skin_tone": "💇🏾\u200d♂️", - "man_getting_haircut_medium-light_skin_tone": "💇🏼\u200d♂️", - "man_getting_haircut_medium_skin_tone": "💇🏽\u200d♂️", - "man_getting_massage": "💆\u200d♂️", - "man_getting_massage_dark_skin_tone": "💆🏿\u200d♂️", - "man_getting_massage_light_skin_tone": "💆🏻\u200d♂️", - "man_getting_massage_medium-dark_skin_tone": "💆🏾\u200d♂️", - "man_getting_massage_medium-light_skin_tone": "💆🏼\u200d♂️", - "man_getting_massage_medium_skin_tone": "💆🏽\u200d♂️", - "man_golfing": "🏌️\u200d♂️", - "man_golfing_dark_skin_tone": "🏌🏿\u200d♂️", - "man_golfing_light_skin_tone": "🏌🏻\u200d♂️", - "man_golfing_medium-dark_skin_tone": "🏌🏾\u200d♂️", - "man_golfing_medium-light_skin_tone": "🏌🏼\u200d♂️", - "man_golfing_medium_skin_tone": "🏌🏽\u200d♂️", - "man_guard": "💂\u200d♂️", - "man_guard_dark_skin_tone": "💂🏿\u200d♂️", - "man_guard_light_skin_tone": "💂🏻\u200d♂️", - "man_guard_medium-dark_skin_tone": "💂🏾\u200d♂️", - "man_guard_medium-light_skin_tone": "💂🏼\u200d♂️", - "man_guard_medium_skin_tone": "💂🏽\u200d♂️", - "man_health_worker": "👨\u200d⚕️", - "man_health_worker_dark_skin_tone": "👨🏿\u200d⚕️", - "man_health_worker_light_skin_tone": "👨🏻\u200d⚕️", - "man_health_worker_medium-dark_skin_tone": "👨🏾\u200d⚕️", - "man_health_worker_medium-light_skin_tone": "👨🏼\u200d⚕️", - "man_health_worker_medium_skin_tone": "👨🏽\u200d⚕️", - "man_in_lotus_position": "🧘\u200d♂️", - "man_in_lotus_position_dark_skin_tone": "🧘🏿\u200d♂️", - "man_in_lotus_position_light_skin_tone": "🧘🏻\u200d♂️", - "man_in_lotus_position_medium-dark_skin_tone": "🧘🏾\u200d♂️", - "man_in_lotus_position_medium-light_skin_tone": "🧘🏼\u200d♂️", - "man_in_lotus_position_medium_skin_tone": "🧘🏽\u200d♂️", - "man_in_manual_wheelchair": "👨\u200d🦽", - "man_in_motorized_wheelchair": "👨\u200d🦼", - "man_in_steamy_room": "🧖\u200d♂️", - "man_in_steamy_room_dark_skin_tone": "🧖🏿\u200d♂️", - "man_in_steamy_room_light_skin_tone": "🧖🏻\u200d♂️", - "man_in_steamy_room_medium-dark_skin_tone": "🧖🏾\u200d♂️", - "man_in_steamy_room_medium-light_skin_tone": "🧖🏼\u200d♂️", - "man_in_steamy_room_medium_skin_tone": "🧖🏽\u200d♂️", - "man_in_suit_levitating": "🕴", - "man_in_suit_levitating_dark_skin_tone": "🕴🏿", - "man_in_suit_levitating_light_skin_tone": "🕴🏻", - "man_in_suit_levitating_medium-dark_skin_tone": "🕴🏾", - "man_in_suit_levitating_medium-light_skin_tone": "🕴🏼", - "man_in_suit_levitating_medium_skin_tone": "🕴🏽", - "man_in_tuxedo": "🤵", - "man_in_tuxedo_dark_skin_tone": "🤵🏿", - "man_in_tuxedo_light_skin_tone": "🤵🏻", - "man_in_tuxedo_medium-dark_skin_tone": "🤵🏾", - "man_in_tuxedo_medium-light_skin_tone": "🤵🏼", - "man_in_tuxedo_medium_skin_tone": "🤵🏽", - "man_judge": "👨\u200d⚖️", - "man_judge_dark_skin_tone": "👨🏿\u200d⚖️", - "man_judge_light_skin_tone": "👨🏻\u200d⚖️", - "man_judge_medium-dark_skin_tone": "👨🏾\u200d⚖️", - "man_judge_medium-light_skin_tone": "👨🏼\u200d⚖️", - "man_judge_medium_skin_tone": "👨🏽\u200d⚖️", - "man_juggling": "🤹\u200d♂️", - "man_juggling_dark_skin_tone": "🤹🏿\u200d♂️", - "man_juggling_light_skin_tone": "🤹🏻\u200d♂️", - "man_juggling_medium-dark_skin_tone": "🤹🏾\u200d♂️", - "man_juggling_medium-light_skin_tone": "🤹🏼\u200d♂️", - "man_juggling_medium_skin_tone": "🤹🏽\u200d♂️", - "man_lifting_weights": "🏋️\u200d♂️", - "man_lifting_weights_dark_skin_tone": "🏋🏿\u200d♂️", - "man_lifting_weights_light_skin_tone": "🏋🏻\u200d♂️", - "man_lifting_weights_medium-dark_skin_tone": "🏋🏾\u200d♂️", - "man_lifting_weights_medium-light_skin_tone": "🏋🏼\u200d♂️", - "man_lifting_weights_medium_skin_tone": "🏋🏽\u200d♂️", - "man_light_skin_tone": "👨🏻", - "man_mage": "🧙\u200d♂️", - "man_mage_dark_skin_tone": "🧙🏿\u200d♂️", - "man_mage_light_skin_tone": "🧙🏻\u200d♂️", - "man_mage_medium-dark_skin_tone": "🧙🏾\u200d♂️", - "man_mage_medium-light_skin_tone": "🧙🏼\u200d♂️", - "man_mage_medium_skin_tone": "🧙🏽\u200d♂️", - "man_mechanic": "👨\u200d🔧", - "man_mechanic_dark_skin_tone": "👨🏿\u200d🔧", - "man_mechanic_light_skin_tone": "👨🏻\u200d🔧", - "man_mechanic_medium-dark_skin_tone": "👨🏾\u200d🔧", - "man_mechanic_medium-light_skin_tone": "👨🏼\u200d🔧", - "man_mechanic_medium_skin_tone": "👨🏽\u200d🔧", - "man_medium-dark_skin_tone": "👨🏾", - "man_medium-light_skin_tone": "👨🏼", - "man_medium_skin_tone": "👨🏽", - "man_mountain_biking": "🚵\u200d♂️", - "man_mountain_biking_dark_skin_tone": "🚵🏿\u200d♂️", - "man_mountain_biking_light_skin_tone": "🚵🏻\u200d♂️", - "man_mountain_biking_medium-dark_skin_tone": "🚵🏾\u200d♂️", - "man_mountain_biking_medium-light_skin_tone": "🚵🏼\u200d♂️", - "man_mountain_biking_medium_skin_tone": "🚵🏽\u200d♂️", - "man_office_worker": "👨\u200d💼", - "man_office_worker_dark_skin_tone": "👨🏿\u200d💼", - "man_office_worker_light_skin_tone": "👨🏻\u200d💼", - "man_office_worker_medium-dark_skin_tone": "👨🏾\u200d💼", - "man_office_worker_medium-light_skin_tone": "👨🏼\u200d💼", - "man_office_worker_medium_skin_tone": "👨🏽\u200d💼", - "man_pilot": "👨\u200d✈️", - "man_pilot_dark_skin_tone": "👨🏿\u200d✈️", - "man_pilot_light_skin_tone": "👨🏻\u200d✈️", - "man_pilot_medium-dark_skin_tone": "👨🏾\u200d✈️", - "man_pilot_medium-light_skin_tone": "👨🏼\u200d✈️", - "man_pilot_medium_skin_tone": "👨🏽\u200d✈️", - "man_playing_handball": "🤾\u200d♂️", - "man_playing_handball_dark_skin_tone": "🤾🏿\u200d♂️", - "man_playing_handball_light_skin_tone": "🤾🏻\u200d♂️", - "man_playing_handball_medium-dark_skin_tone": "🤾🏾\u200d♂️", - "man_playing_handball_medium-light_skin_tone": "🤾🏼\u200d♂️", - "man_playing_handball_medium_skin_tone": "🤾🏽\u200d♂️", - "man_playing_water_polo": "🤽\u200d♂️", - "man_playing_water_polo_dark_skin_tone": "🤽🏿\u200d♂️", - "man_playing_water_polo_light_skin_tone": "🤽🏻\u200d♂️", - "man_playing_water_polo_medium-dark_skin_tone": "🤽🏾\u200d♂️", - "man_playing_water_polo_medium-light_skin_tone": "🤽🏼\u200d♂️", - "man_playing_water_polo_medium_skin_tone": "🤽🏽\u200d♂️", - "man_police_officer": "👮\u200d♂️", - "man_police_officer_dark_skin_tone": "👮🏿\u200d♂️", - "man_police_officer_light_skin_tone": "👮🏻\u200d♂️", - "man_police_officer_medium-dark_skin_tone": "👮🏾\u200d♂️", - "man_police_officer_medium-light_skin_tone": "👮🏼\u200d♂️", - "man_police_officer_medium_skin_tone": "👮🏽\u200d♂️", - "man_pouting": "🙎\u200d♂️", - "man_pouting_dark_skin_tone": "🙎🏿\u200d♂️", - "man_pouting_light_skin_tone": "🙎🏻\u200d♂️", - "man_pouting_medium-dark_skin_tone": "🙎🏾\u200d♂️", - "man_pouting_medium-light_skin_tone": "🙎🏼\u200d♂️", - "man_pouting_medium_skin_tone": "🙎🏽\u200d♂️", - "man_raising_hand": "🙋\u200d♂️", - "man_raising_hand_dark_skin_tone": "🙋🏿\u200d♂️", - "man_raising_hand_light_skin_tone": "🙋🏻\u200d♂️", - "man_raising_hand_medium-dark_skin_tone": "🙋🏾\u200d♂️", - "man_raising_hand_medium-light_skin_tone": "🙋🏼\u200d♂️", - "man_raising_hand_medium_skin_tone": "🙋🏽\u200d♂️", - "man_rowing_boat": "🚣\u200d♂️", - "man_rowing_boat_dark_skin_tone": "🚣🏿\u200d♂️", - "man_rowing_boat_light_skin_tone": "🚣🏻\u200d♂️", - "man_rowing_boat_medium-dark_skin_tone": "🚣🏾\u200d♂️", - "man_rowing_boat_medium-light_skin_tone": "🚣🏼\u200d♂️", - "man_rowing_boat_medium_skin_tone": "🚣🏽\u200d♂️", - "man_running": "🏃\u200d♂️", - "man_running_dark_skin_tone": "🏃🏿\u200d♂️", - "man_running_light_skin_tone": "🏃🏻\u200d♂️", - "man_running_medium-dark_skin_tone": "🏃🏾\u200d♂️", - "man_running_medium-light_skin_tone": "🏃🏼\u200d♂️", - "man_running_medium_skin_tone": "🏃🏽\u200d♂️", - "man_scientist": "👨\u200d🔬", - "man_scientist_dark_skin_tone": "👨🏿\u200d🔬", - "man_scientist_light_skin_tone": "👨🏻\u200d🔬", - "man_scientist_medium-dark_skin_tone": "👨🏾\u200d🔬", - "man_scientist_medium-light_skin_tone": "👨🏼\u200d🔬", - "man_scientist_medium_skin_tone": "👨🏽\u200d🔬", - "man_shrugging": "🤷\u200d♂️", - "man_shrugging_dark_skin_tone": "🤷🏿\u200d♂️", - "man_shrugging_light_skin_tone": "🤷🏻\u200d♂️", - "man_shrugging_medium-dark_skin_tone": "🤷🏾\u200d♂️", - "man_shrugging_medium-light_skin_tone": "🤷🏼\u200d♂️", - "man_shrugging_medium_skin_tone": "🤷🏽\u200d♂️", - "man_singer": "👨\u200d🎤", - "man_singer_dark_skin_tone": "👨🏿\u200d🎤", - "man_singer_light_skin_tone": "👨🏻\u200d🎤", - "man_singer_medium-dark_skin_tone": "👨🏾\u200d🎤", - "man_singer_medium-light_skin_tone": "👨🏼\u200d🎤", - "man_singer_medium_skin_tone": "👨🏽\u200d🎤", - "man_student": "👨\u200d🎓", - "man_student_dark_skin_tone": "👨🏿\u200d🎓", - "man_student_light_skin_tone": "👨🏻\u200d🎓", - "man_student_medium-dark_skin_tone": "👨🏾\u200d🎓", - "man_student_medium-light_skin_tone": "👨🏼\u200d🎓", - "man_student_medium_skin_tone": "👨🏽\u200d🎓", - "man_surfing": "🏄\u200d♂️", - "man_surfing_dark_skin_tone": "🏄🏿\u200d♂️", - "man_surfing_light_skin_tone": "🏄🏻\u200d♂️", - "man_surfing_medium-dark_skin_tone": "🏄🏾\u200d♂️", - "man_surfing_medium-light_skin_tone": "🏄🏼\u200d♂️", - "man_surfing_medium_skin_tone": "🏄🏽\u200d♂️", - "man_swimming": "🏊\u200d♂️", - "man_swimming_dark_skin_tone": "🏊🏿\u200d♂️", - "man_swimming_light_skin_tone": "🏊🏻\u200d♂️", - "man_swimming_medium-dark_skin_tone": "🏊🏾\u200d♂️", - "man_swimming_medium-light_skin_tone": "🏊🏼\u200d♂️", - "man_swimming_medium_skin_tone": "🏊🏽\u200d♂️", - "man_teacher": "👨\u200d🏫", - "man_teacher_dark_skin_tone": "👨🏿\u200d🏫", - "man_teacher_light_skin_tone": "👨🏻\u200d🏫", - "man_teacher_medium-dark_skin_tone": "👨🏾\u200d🏫", - "man_teacher_medium-light_skin_tone": "👨🏼\u200d🏫", - "man_teacher_medium_skin_tone": "👨🏽\u200d🏫", - "man_technologist": "👨\u200d💻", - "man_technologist_dark_skin_tone": "👨🏿\u200d💻", - "man_technologist_light_skin_tone": "👨🏻\u200d💻", - "man_technologist_medium-dark_skin_tone": "👨🏾\u200d💻", - "man_technologist_medium-light_skin_tone": "👨🏼\u200d💻", - "man_technologist_medium_skin_tone": "👨🏽\u200d💻", - "man_tipping_hand": "💁\u200d♂️", - "man_tipping_hand_dark_skin_tone": "💁🏿\u200d♂️", - "man_tipping_hand_light_skin_tone": "💁🏻\u200d♂️", - "man_tipping_hand_medium-dark_skin_tone": "💁🏾\u200d♂️", - "man_tipping_hand_medium-light_skin_tone": "💁🏼\u200d♂️", - "man_tipping_hand_medium_skin_tone": "💁🏽\u200d♂️", - "man_vampire": "🧛\u200d♂️", - "man_vampire_dark_skin_tone": "🧛🏿\u200d♂️", - "man_vampire_light_skin_tone": "🧛🏻\u200d♂️", - "man_vampire_medium-dark_skin_tone": "🧛🏾\u200d♂️", - "man_vampire_medium-light_skin_tone": "🧛🏼\u200d♂️", - "man_vampire_medium_skin_tone": "🧛🏽\u200d♂️", - "man_walking": "🚶\u200d♂️", - "man_walking_dark_skin_tone": "🚶🏿\u200d♂️", - "man_walking_light_skin_tone": "🚶🏻\u200d♂️", - "man_walking_medium-dark_skin_tone": "🚶🏾\u200d♂️", - "man_walking_medium-light_skin_tone": "🚶🏼\u200d♂️", - "man_walking_medium_skin_tone": "🚶🏽\u200d♂️", - "man_wearing_turban": "👳\u200d♂️", - "man_wearing_turban_dark_skin_tone": "👳🏿\u200d♂️", - "man_wearing_turban_light_skin_tone": "👳🏻\u200d♂️", - "man_wearing_turban_medium-dark_skin_tone": "👳🏾\u200d♂️", - "man_wearing_turban_medium-light_skin_tone": "👳🏼\u200d♂️", - "man_wearing_turban_medium_skin_tone": "👳🏽\u200d♂️", - "man_with_probing_cane": "👨\u200d🦯", - "man_with_chinese_cap": "👲", - "man_with_chinese_cap_dark_skin_tone": "👲🏿", - "man_with_chinese_cap_light_skin_tone": "👲🏻", - "man_with_chinese_cap_medium-dark_skin_tone": "👲🏾", - "man_with_chinese_cap_medium-light_skin_tone": "👲🏼", - "man_with_chinese_cap_medium_skin_tone": "👲🏽", - "man_zombie": "🧟\u200d♂️", - "mango": "🥭", - "mantelpiece_clock": "🕰", - "manual_wheelchair": "🦽", - "man’s_shoe": "👞", - "map_of_japan": "🗾", - "maple_leaf": "🍁", - "martial_arts_uniform": "🥋", - "mate": "🧉", - "meat_on_bone": "🍖", - "mechanical_arm": "🦾", - "mechanical_leg": "🦿", - "medical_symbol": "⚕", - "megaphone": "📣", - "melon": "🍈", - "memo": "📝", - "men_with_bunny_ears": "👯\u200d♂️", - "men_wrestling": "🤼\u200d♂️", - "menorah": "🕎", - "men’s_room": "🚹", - "mermaid": "🧜\u200d♀️", - "mermaid_dark_skin_tone": "🧜🏿\u200d♀️", - "mermaid_light_skin_tone": "🧜🏻\u200d♀️", - "mermaid_medium-dark_skin_tone": "🧜🏾\u200d♀️", - "mermaid_medium-light_skin_tone": "🧜🏼\u200d♀️", - "mermaid_medium_skin_tone": "🧜🏽\u200d♀️", - "merman": "🧜\u200d♂️", - "merman_dark_skin_tone": "🧜🏿\u200d♂️", - "merman_light_skin_tone": "🧜🏻\u200d♂️", - "merman_medium-dark_skin_tone": "🧜🏾\u200d♂️", - "merman_medium-light_skin_tone": "🧜🏼\u200d♂️", - "merman_medium_skin_tone": "🧜🏽\u200d♂️", - "merperson": "🧜", - "merperson_dark_skin_tone": "🧜🏿", - "merperson_light_skin_tone": "🧜🏻", - "merperson_medium-dark_skin_tone": "🧜🏾", - "merperson_medium-light_skin_tone": "🧜🏼", - "merperson_medium_skin_tone": "🧜🏽", - "metro": "🚇", - "microbe": "🦠", - "microphone": "🎤", - "microscope": "🔬", - "middle_finger": "🖕", - "middle_finger_dark_skin_tone": "🖕🏿", - "middle_finger_light_skin_tone": "🖕🏻", - "middle_finger_medium-dark_skin_tone": "🖕🏾", - "middle_finger_medium-light_skin_tone": "🖕🏼", - "middle_finger_medium_skin_tone": "🖕🏽", - "military_medal": "🎖", - "milky_way": "🌌", - "minibus": "🚐", - "moai": "🗿", - "mobile_phone": "📱", - "mobile_phone_off": "📴", - "mobile_phone_with_arrow": "📲", - "money-mouth_face": "🤑", - "money_bag": "💰", - "money_with_wings": "💸", - "monkey": "🐒", - "monkey_face": "🐵", - "monorail": "🚝", - "moon_cake": "🥮", - "moon_viewing_ceremony": "🎑", - "mosque": "🕌", - "mosquito": "🦟", - "motor_boat": "🛥", - "motor_scooter": "🛵", - "motorcycle": "🏍", - "motorized_wheelchair": "🦼", - "motorway": "🛣", - "mount_fuji": "🗻", - "mountain": "⛰", - "mountain_cableway": "🚠", - "mountain_railway": "🚞", - "mouse": "🐭", - "mouse_face": "🐭", - "mouth": "👄", - "movie_camera": "🎥", - "mushroom": "🍄", - "musical_keyboard": "🎹", - "musical_note": "🎵", - "musical_notes": "🎶", - "musical_score": "🎼", - "muted_speaker": "🔇", - "nail_polish": "💅", - "nail_polish_dark_skin_tone": "💅🏿", - "nail_polish_light_skin_tone": "💅🏻", - "nail_polish_medium-dark_skin_tone": "💅🏾", - "nail_polish_medium-light_skin_tone": "💅🏼", - "nail_polish_medium_skin_tone": "💅🏽", - "name_badge": "📛", - "national_park": "🏞", - "nauseated_face": "🤢", - "nazar_amulet": "🧿", - "necktie": "👔", - "nerd_face": "🤓", - "neutral_face": "😐", - "new_moon": "🌑", - "new_moon_face": "🌚", - "newspaper": "📰", - "next_track_button": "⏭", - "night_with_stars": "🌃", - "nine-thirty": "🕤", - "nine_o’clock": "🕘", - "no_bicycles": "🚳", - "no_entry": "⛔", - "no_littering": "🚯", - "no_mobile_phones": "📵", - "no_one_under_eighteen": "🔞", - "no_pedestrians": "🚷", - "no_smoking": "🚭", - "non-potable_water": "🚱", - "nose": "👃", - "nose_dark_skin_tone": "👃🏿", - "nose_light_skin_tone": "👃🏻", - "nose_medium-dark_skin_tone": "👃🏾", - "nose_medium-light_skin_tone": "👃🏼", - "nose_medium_skin_tone": "👃🏽", - "notebook": "📓", - "notebook_with_decorative_cover": "📔", - "nut_and_bolt": "🔩", - "octopus": "🐙", - "oden": "🍢", - "office_building": "🏢", - "ogre": "👹", - "oil_drum": "🛢", - "old_key": "🗝", - "old_man": "👴", - "old_man_dark_skin_tone": "👴🏿", - "old_man_light_skin_tone": "👴🏻", - "old_man_medium-dark_skin_tone": "👴🏾", - "old_man_medium-light_skin_tone": "👴🏼", - "old_man_medium_skin_tone": "👴🏽", - "old_woman": "👵", - "old_woman_dark_skin_tone": "👵🏿", - "old_woman_light_skin_tone": "👵🏻", - "old_woman_medium-dark_skin_tone": "👵🏾", - "old_woman_medium-light_skin_tone": "👵🏼", - "old_woman_medium_skin_tone": "👵🏽", - "older_adult": "🧓", - "older_adult_dark_skin_tone": "🧓🏿", - "older_adult_light_skin_tone": "🧓🏻", - "older_adult_medium-dark_skin_tone": "🧓🏾", - "older_adult_medium-light_skin_tone": "🧓🏼", - "older_adult_medium_skin_tone": "🧓🏽", - "om": "🕉", - "oncoming_automobile": "🚘", - "oncoming_bus": "🚍", - "oncoming_fist": "👊", - "oncoming_fist_dark_skin_tone": "👊🏿", - "oncoming_fist_light_skin_tone": "👊🏻", - "oncoming_fist_medium-dark_skin_tone": "👊🏾", - "oncoming_fist_medium-light_skin_tone": "👊🏼", - "oncoming_fist_medium_skin_tone": "👊🏽", - "oncoming_police_car": "🚔", - "oncoming_taxi": "🚖", - "one-piece_swimsuit": "🩱", - "one-thirty": "🕜", - "one_o’clock": "🕐", - "onion": "🧅", - "open_book": "📖", - "open_file_folder": "📂", - "open_hands": "👐", - "open_hands_dark_skin_tone": "👐🏿", - "open_hands_light_skin_tone": "👐🏻", - "open_hands_medium-dark_skin_tone": "👐🏾", - "open_hands_medium-light_skin_tone": "👐🏼", - "open_hands_medium_skin_tone": "👐🏽", - "open_mailbox_with_lowered_flag": "📭", - "open_mailbox_with_raised_flag": "📬", - "optical_disk": "💿", - "orange_book": "📙", - "orange_circle": "🟠", - "orange_heart": "🧡", - "orange_square": "🟧", - "orangutan": "🦧", - "orthodox_cross": "☦", - "otter": "🦦", - "outbox_tray": "📤", - "owl": "🦉", - "ox": "🐂", - "oyster": "🦪", - "package": "📦", - "page_facing_up": "📄", - "page_with_curl": "📃", - "pager": "📟", - "paintbrush": "🖌", - "palm_tree": "🌴", - "palms_up_together": "🤲", - "palms_up_together_dark_skin_tone": "🤲🏿", - "palms_up_together_light_skin_tone": "🤲🏻", - "palms_up_together_medium-dark_skin_tone": "🤲🏾", - "palms_up_together_medium-light_skin_tone": "🤲🏼", - "palms_up_together_medium_skin_tone": "🤲🏽", - "pancakes": "🥞", - "panda_face": "🐼", - "paperclip": "📎", - "parrot": "🦜", - "part_alternation_mark": "〽", - "party_popper": "🎉", - "partying_face": "🥳", - "passenger_ship": "🛳", - "passport_control": "🛂", - "pause_button": "⏸", - "paw_prints": "🐾", - "peace_symbol": "☮", - "peach": "🍑", - "peacock": "🦚", - "peanuts": "🥜", - "pear": "🍐", - "pen": "🖊", - "pencil": "📝", - "penguin": "🐧", - "pensive_face": "😔", - "people_holding_hands": "🧑\u200d🤝\u200d🧑", - "people_with_bunny_ears": "👯", - "people_wrestling": "🤼", - "performing_arts": "🎭", - "persevering_face": "😣", - "person_biking": "🚴", - "person_biking_dark_skin_tone": "🚴🏿", - "person_biking_light_skin_tone": "🚴🏻", - "person_biking_medium-dark_skin_tone": "🚴🏾", - "person_biking_medium-light_skin_tone": "🚴🏼", - "person_biking_medium_skin_tone": "🚴🏽", - "person_bouncing_ball": "⛹", - "person_bouncing_ball_dark_skin_tone": "⛹🏿", - "person_bouncing_ball_light_skin_tone": "⛹🏻", - "person_bouncing_ball_medium-dark_skin_tone": "⛹🏾", - "person_bouncing_ball_medium-light_skin_tone": "⛹🏼", - "person_bouncing_ball_medium_skin_tone": "⛹🏽", - "person_bowing": "🙇", - "person_bowing_dark_skin_tone": "🙇🏿", - "person_bowing_light_skin_tone": "🙇🏻", - "person_bowing_medium-dark_skin_tone": "🙇🏾", - "person_bowing_medium-light_skin_tone": "🙇🏼", - "person_bowing_medium_skin_tone": "🙇🏽", - "person_cartwheeling": "🤸", - "person_cartwheeling_dark_skin_tone": "🤸🏿", - "person_cartwheeling_light_skin_tone": "🤸🏻", - "person_cartwheeling_medium-dark_skin_tone": "🤸🏾", - "person_cartwheeling_medium-light_skin_tone": "🤸🏼", - "person_cartwheeling_medium_skin_tone": "🤸🏽", - "person_climbing": "🧗", - "person_climbing_dark_skin_tone": "🧗🏿", - "person_climbing_light_skin_tone": "🧗🏻", - "person_climbing_medium-dark_skin_tone": "🧗🏾", - "person_climbing_medium-light_skin_tone": "🧗🏼", - "person_climbing_medium_skin_tone": "🧗🏽", - "person_facepalming": "🤦", - "person_facepalming_dark_skin_tone": "🤦🏿", - "person_facepalming_light_skin_tone": "🤦🏻", - "person_facepalming_medium-dark_skin_tone": "🤦🏾", - "person_facepalming_medium-light_skin_tone": "🤦🏼", - "person_facepalming_medium_skin_tone": "🤦🏽", - "person_fencing": "🤺", - "person_frowning": "🙍", - "person_frowning_dark_skin_tone": "🙍🏿", - "person_frowning_light_skin_tone": "🙍🏻", - "person_frowning_medium-dark_skin_tone": "🙍🏾", - "person_frowning_medium-light_skin_tone": "🙍🏼", - "person_frowning_medium_skin_tone": "🙍🏽", - "person_gesturing_no": "🙅", - "person_gesturing_no_dark_skin_tone": "🙅🏿", - "person_gesturing_no_light_skin_tone": "🙅🏻", - "person_gesturing_no_medium-dark_skin_tone": "🙅🏾", - "person_gesturing_no_medium-light_skin_tone": "🙅🏼", - "person_gesturing_no_medium_skin_tone": "🙅🏽", - "person_gesturing_ok": "🙆", - "person_gesturing_ok_dark_skin_tone": "🙆🏿", - "person_gesturing_ok_light_skin_tone": "🙆🏻", - "person_gesturing_ok_medium-dark_skin_tone": "🙆🏾", - "person_gesturing_ok_medium-light_skin_tone": "🙆🏼", - "person_gesturing_ok_medium_skin_tone": "🙆🏽", - "person_getting_haircut": "💇", - "person_getting_haircut_dark_skin_tone": "💇🏿", - "person_getting_haircut_light_skin_tone": "💇🏻", - "person_getting_haircut_medium-dark_skin_tone": "💇🏾", - "person_getting_haircut_medium-light_skin_tone": "💇🏼", - "person_getting_haircut_medium_skin_tone": "💇🏽", - "person_getting_massage": "💆", - "person_getting_massage_dark_skin_tone": "💆🏿", - "person_getting_massage_light_skin_tone": "💆🏻", - "person_getting_massage_medium-dark_skin_tone": "💆🏾", - "person_getting_massage_medium-light_skin_tone": "💆🏼", - "person_getting_massage_medium_skin_tone": "💆🏽", - "person_golfing": "🏌", - "person_golfing_dark_skin_tone": "🏌🏿", - "person_golfing_light_skin_tone": "🏌🏻", - "person_golfing_medium-dark_skin_tone": "🏌🏾", - "person_golfing_medium-light_skin_tone": "🏌🏼", - "person_golfing_medium_skin_tone": "🏌🏽", - "person_in_bed": "🛌", - "person_in_bed_dark_skin_tone": "🛌🏿", - "person_in_bed_light_skin_tone": "🛌🏻", - "person_in_bed_medium-dark_skin_tone": "🛌🏾", - "person_in_bed_medium-light_skin_tone": "🛌🏼", - "person_in_bed_medium_skin_tone": "🛌🏽", - "person_in_lotus_position": "🧘", - "person_in_lotus_position_dark_skin_tone": "🧘🏿", - "person_in_lotus_position_light_skin_tone": "🧘🏻", - "person_in_lotus_position_medium-dark_skin_tone": "🧘🏾", - "person_in_lotus_position_medium-light_skin_tone": "🧘🏼", - "person_in_lotus_position_medium_skin_tone": "🧘🏽", - "person_in_steamy_room": "🧖", - "person_in_steamy_room_dark_skin_tone": "🧖🏿", - "person_in_steamy_room_light_skin_tone": "🧖🏻", - "person_in_steamy_room_medium-dark_skin_tone": "🧖🏾", - "person_in_steamy_room_medium-light_skin_tone": "🧖🏼", - "person_in_steamy_room_medium_skin_tone": "🧖🏽", - "person_juggling": "🤹", - "person_juggling_dark_skin_tone": "🤹🏿", - "person_juggling_light_skin_tone": "🤹🏻", - "person_juggling_medium-dark_skin_tone": "🤹🏾", - "person_juggling_medium-light_skin_tone": "🤹🏼", - "person_juggling_medium_skin_tone": "🤹🏽", - "person_kneeling": "🧎", - "person_lifting_weights": "🏋", - "person_lifting_weights_dark_skin_tone": "🏋🏿", - "person_lifting_weights_light_skin_tone": "🏋🏻", - "person_lifting_weights_medium-dark_skin_tone": "🏋🏾", - "person_lifting_weights_medium-light_skin_tone": "🏋🏼", - "person_lifting_weights_medium_skin_tone": "🏋🏽", - "person_mountain_biking": "🚵", - "person_mountain_biking_dark_skin_tone": "🚵🏿", - "person_mountain_biking_light_skin_tone": "🚵🏻", - "person_mountain_biking_medium-dark_skin_tone": "🚵🏾", - "person_mountain_biking_medium-light_skin_tone": "🚵🏼", - "person_mountain_biking_medium_skin_tone": "🚵🏽", - "person_playing_handball": "🤾", - "person_playing_handball_dark_skin_tone": "🤾🏿", - "person_playing_handball_light_skin_tone": "🤾🏻", - "person_playing_handball_medium-dark_skin_tone": "🤾🏾", - "person_playing_handball_medium-light_skin_tone": "🤾🏼", - "person_playing_handball_medium_skin_tone": "🤾🏽", - "person_playing_water_polo": "🤽", - "person_playing_water_polo_dark_skin_tone": "🤽🏿", - "person_playing_water_polo_light_skin_tone": "🤽🏻", - "person_playing_water_polo_medium-dark_skin_tone": "🤽🏾", - "person_playing_water_polo_medium-light_skin_tone": "🤽🏼", - "person_playing_water_polo_medium_skin_tone": "🤽🏽", - "person_pouting": "🙎", - "person_pouting_dark_skin_tone": "🙎🏿", - "person_pouting_light_skin_tone": "🙎🏻", - "person_pouting_medium-dark_skin_tone": "🙎🏾", - "person_pouting_medium-light_skin_tone": "🙎🏼", - "person_pouting_medium_skin_tone": "🙎🏽", - "person_raising_hand": "🙋", - "person_raising_hand_dark_skin_tone": "🙋🏿", - "person_raising_hand_light_skin_tone": "🙋🏻", - "person_raising_hand_medium-dark_skin_tone": "🙋🏾", - "person_raising_hand_medium-light_skin_tone": "🙋🏼", - "person_raising_hand_medium_skin_tone": "🙋🏽", - "person_rowing_boat": "🚣", - "person_rowing_boat_dark_skin_tone": "🚣🏿", - "person_rowing_boat_light_skin_tone": "🚣🏻", - "person_rowing_boat_medium-dark_skin_tone": "🚣🏾", - "person_rowing_boat_medium-light_skin_tone": "🚣🏼", - "person_rowing_boat_medium_skin_tone": "🚣🏽", - "person_running": "🏃", - "person_running_dark_skin_tone": "🏃🏿", - "person_running_light_skin_tone": "🏃🏻", - "person_running_medium-dark_skin_tone": "🏃🏾", - "person_running_medium-light_skin_tone": "🏃🏼", - "person_running_medium_skin_tone": "🏃🏽", - "person_shrugging": "🤷", - "person_shrugging_dark_skin_tone": "🤷🏿", - "person_shrugging_light_skin_tone": "🤷🏻", - "person_shrugging_medium-dark_skin_tone": "🤷🏾", - "person_shrugging_medium-light_skin_tone": "🤷🏼", - "person_shrugging_medium_skin_tone": "🤷🏽", - "person_standing": "🧍", - "person_surfing": "🏄", - "person_surfing_dark_skin_tone": "🏄🏿", - "person_surfing_light_skin_tone": "🏄🏻", - "person_surfing_medium-dark_skin_tone": "🏄🏾", - "person_surfing_medium-light_skin_tone": "🏄🏼", - "person_surfing_medium_skin_tone": "🏄🏽", - "person_swimming": "🏊", - "person_swimming_dark_skin_tone": "🏊🏿", - "person_swimming_light_skin_tone": "🏊🏻", - "person_swimming_medium-dark_skin_tone": "🏊🏾", - "person_swimming_medium-light_skin_tone": "🏊🏼", - "person_swimming_medium_skin_tone": "🏊🏽", - "person_taking_bath": "🛀", - "person_taking_bath_dark_skin_tone": "🛀🏿", - "person_taking_bath_light_skin_tone": "🛀🏻", - "person_taking_bath_medium-dark_skin_tone": "🛀🏾", - "person_taking_bath_medium-light_skin_tone": "🛀🏼", - "person_taking_bath_medium_skin_tone": "🛀🏽", - "person_tipping_hand": "💁", - "person_tipping_hand_dark_skin_tone": "💁🏿", - "person_tipping_hand_light_skin_tone": "💁🏻", - "person_tipping_hand_medium-dark_skin_tone": "💁🏾", - "person_tipping_hand_medium-light_skin_tone": "💁🏼", - "person_tipping_hand_medium_skin_tone": "💁🏽", - "person_walking": "🚶", - "person_walking_dark_skin_tone": "🚶🏿", - "person_walking_light_skin_tone": "🚶🏻", - "person_walking_medium-dark_skin_tone": "🚶🏾", - "person_walking_medium-light_skin_tone": "🚶🏼", - "person_walking_medium_skin_tone": "🚶🏽", - "person_wearing_turban": "👳", - "person_wearing_turban_dark_skin_tone": "👳🏿", - "person_wearing_turban_light_skin_tone": "👳🏻", - "person_wearing_turban_medium-dark_skin_tone": "👳🏾", - "person_wearing_turban_medium-light_skin_tone": "👳🏼", - "person_wearing_turban_medium_skin_tone": "👳🏽", - "petri_dish": "🧫", - "pick": "⛏", - "pie": "🥧", - "pig": "🐷", - "pig_face": "🐷", - "pig_nose": "🐽", - "pile_of_poo": "💩", - "pill": "💊", - "pinching_hand": "🤏", - "pine_decoration": "🎍", - "pineapple": "🍍", - "ping_pong": "🏓", - "pirate_flag": "🏴\u200d☠️", - "pistol": "🔫", - "pizza": "🍕", - "place_of_worship": "🛐", - "play_button": "▶", - "play_or_pause_button": "⏯", - "pleading_face": "🥺", - "police_car": "🚓", - "police_car_light": "🚨", - "police_officer": "👮", - "police_officer_dark_skin_tone": "👮🏿", - "police_officer_light_skin_tone": "👮🏻", - "police_officer_medium-dark_skin_tone": "👮🏾", - "police_officer_medium-light_skin_tone": "👮🏼", - "police_officer_medium_skin_tone": "👮🏽", - "poodle": "🐩", - "pool_8_ball": "🎱", - "popcorn": "🍿", - "post_office": "🏣", - "postal_horn": "📯", - "postbox": "📮", - "pot_of_food": "🍲", - "potable_water": "🚰", - "potato": "🥔", - "poultry_leg": "🍗", - "pound_banknote": "💷", - "pouting_cat_face": "😾", - "pouting_face": "😡", - "prayer_beads": "📿", - "pregnant_woman": "🤰", - "pregnant_woman_dark_skin_tone": "🤰🏿", - "pregnant_woman_light_skin_tone": "🤰🏻", - "pregnant_woman_medium-dark_skin_tone": "🤰🏾", - "pregnant_woman_medium-light_skin_tone": "🤰🏼", - "pregnant_woman_medium_skin_tone": "🤰🏽", - "pretzel": "🥨", - "probing_cane": "🦯", - "prince": "🤴", - "prince_dark_skin_tone": "🤴🏿", - "prince_light_skin_tone": "🤴🏻", - "prince_medium-dark_skin_tone": "🤴🏾", - "prince_medium-light_skin_tone": "🤴🏼", - "prince_medium_skin_tone": "🤴🏽", - "princess": "👸", - "princess_dark_skin_tone": "👸🏿", - "princess_light_skin_tone": "👸🏻", - "princess_medium-dark_skin_tone": "👸🏾", - "princess_medium-light_skin_tone": "👸🏼", - "princess_medium_skin_tone": "👸🏽", - "printer": "🖨", - "prohibited": "🚫", - "purple_circle": "🟣", - "purple_heart": "💜", - "purple_square": "🟪", - "purse": "👛", - "pushpin": "📌", - "question_mark": "❓", - "rabbit": "🐰", - "rabbit_face": "🐰", - "raccoon": "🦝", - "racing_car": "🏎", - "radio": "📻", - "radio_button": "🔘", - "radioactive": "☢", - "railway_car": "🚃", - "railway_track": "🛤", - "rainbow": "🌈", - "rainbow_flag": "🏳️\u200d🌈", - "raised_back_of_hand": "🤚", - "raised_back_of_hand_dark_skin_tone": "🤚🏿", - "raised_back_of_hand_light_skin_tone": "🤚🏻", - "raised_back_of_hand_medium-dark_skin_tone": "🤚🏾", - "raised_back_of_hand_medium-light_skin_tone": "🤚🏼", - "raised_back_of_hand_medium_skin_tone": "🤚🏽", - "raised_fist": "✊", - "raised_fist_dark_skin_tone": "✊🏿", - "raised_fist_light_skin_tone": "✊🏻", - "raised_fist_medium-dark_skin_tone": "✊🏾", - "raised_fist_medium-light_skin_tone": "✊🏼", - "raised_fist_medium_skin_tone": "✊🏽", - "raised_hand": "✋", - "raised_hand_dark_skin_tone": "✋🏿", - "raised_hand_light_skin_tone": "✋🏻", - "raised_hand_medium-dark_skin_tone": "✋🏾", - "raised_hand_medium-light_skin_tone": "✋🏼", - "raised_hand_medium_skin_tone": "✋🏽", - "raising_hands": "🙌", - "raising_hands_dark_skin_tone": "🙌🏿", - "raising_hands_light_skin_tone": "🙌🏻", - "raising_hands_medium-dark_skin_tone": "🙌🏾", - "raising_hands_medium-light_skin_tone": "🙌🏼", - "raising_hands_medium_skin_tone": "🙌🏽", - "ram": "🐏", - "rat": "🐀", - "razor": "🪒", - "ringed_planet": "🪐", - "receipt": "🧾", - "record_button": "⏺", - "recycling_symbol": "♻", - "red_apple": "🍎", - "red_circle": "🔴", - "red_envelope": "🧧", - "red_hair": "🦰", - "red-haired_man": "👨\u200d🦰", - "red-haired_woman": "👩\u200d🦰", - "red_heart": "❤", - "red_paper_lantern": "🏮", - "red_square": "🟥", - "red_triangle_pointed_down": "🔻", - "red_triangle_pointed_up": "🔺", - "registered": "®", - "relieved_face": "😌", - "reminder_ribbon": "🎗", - "repeat_button": "🔁", - "repeat_single_button": "🔂", - "rescue_worker’s_helmet": "⛑", - "restroom": "🚻", - "reverse_button": "◀", - "revolving_hearts": "💞", - "rhinoceros": "🦏", - "ribbon": "🎀", - "rice_ball": "🍙", - "rice_cracker": "🍘", - "right-facing_fist": "🤜", - "right-facing_fist_dark_skin_tone": "🤜🏿", - "right-facing_fist_light_skin_tone": "🤜🏻", - "right-facing_fist_medium-dark_skin_tone": "🤜🏾", - "right-facing_fist_medium-light_skin_tone": "🤜🏼", - "right-facing_fist_medium_skin_tone": "🤜🏽", - "right_anger_bubble": "🗯", - "right_arrow": "➡", - "right_arrow_curving_down": "⤵", - "right_arrow_curving_left": "↩", - "right_arrow_curving_up": "⤴", - "ring": "💍", - "roasted_sweet_potato": "🍠", - "robot_face": "🤖", - "rocket": "🚀", - "roll_of_paper": "🧻", - "rolled-up_newspaper": "🗞", - "roller_coaster": "🎢", - "rolling_on_the_floor_laughing": "🤣", - "rooster": "🐓", - "rose": "🌹", - "rosette": "🏵", - "round_pushpin": "📍", - "rugby_football": "🏉", - "running_shirt": "🎽", - "running_shoe": "👟", - "sad_but_relieved_face": "😥", - "safety_pin": "🧷", - "safety_vest": "🦺", - "salt": "🧂", - "sailboat": "⛵", - "sake": "🍶", - "sandwich": "🥪", - "sari": "🥻", - "satellite": "📡", - "satellite_antenna": "📡", - "sauropod": "🦕", - "saxophone": "🎷", - "scarf": "🧣", - "school": "🏫", - "school_backpack": "🎒", - "scissors": "✂", - "scorpion": "🦂", - "scroll": "📜", - "seat": "💺", - "see-no-evil_monkey": "🙈", - "seedling": "🌱", - "selfie": "🤳", - "selfie_dark_skin_tone": "🤳🏿", - "selfie_light_skin_tone": "🤳🏻", - "selfie_medium-dark_skin_tone": "🤳🏾", - "selfie_medium-light_skin_tone": "🤳🏼", - "selfie_medium_skin_tone": "🤳🏽", - "service_dog": "🐕\u200d🦺", - "seven-thirty": "🕢", - "seven_o’clock": "🕖", - "shallow_pan_of_food": "🥘", - "shamrock": "☘", - "shark": "🦈", - "shaved_ice": "🍧", - "sheaf_of_rice": "🌾", - "shield": "🛡", - "shinto_shrine": "⛩", - "ship": "🚢", - "shooting_star": "🌠", - "shopping_bags": "🛍", - "shopping_cart": "🛒", - "shortcake": "🍰", - "shorts": "🩳", - "shower": "🚿", - "shrimp": "🦐", - "shuffle_tracks_button": "🔀", - "shushing_face": "🤫", - "sign_of_the_horns": "🤘", - "sign_of_the_horns_dark_skin_tone": "🤘🏿", - "sign_of_the_horns_light_skin_tone": "🤘🏻", - "sign_of_the_horns_medium-dark_skin_tone": "🤘🏾", - "sign_of_the_horns_medium-light_skin_tone": "🤘🏼", - "sign_of_the_horns_medium_skin_tone": "🤘🏽", - "six-thirty": "🕡", - "six_o’clock": "🕕", - "skateboard": "🛹", - "skier": "⛷", - "skis": "🎿", - "skull": "💀", - "skull_and_crossbones": "☠", - "skunk": "🦨", - "sled": "🛷", - "sleeping_face": "😴", - "sleepy_face": "😪", - "slightly_frowning_face": "🙁", - "slightly_smiling_face": "🙂", - "slot_machine": "🎰", - "sloth": "🦥", - "small_airplane": "🛩", - "small_blue_diamond": "🔹", - "small_orange_diamond": "🔸", - "smiling_cat_face_with_heart-eyes": "😻", - "smiling_face": "☺", - "smiling_face_with_halo": "😇", - "smiling_face_with_3_hearts": "🥰", - "smiling_face_with_heart-eyes": "😍", - "smiling_face_with_horns": "😈", - "smiling_face_with_smiling_eyes": "😊", - "smiling_face_with_sunglasses": "😎", - "smirking_face": "😏", - "snail": "🐌", - "snake": "🐍", - "sneezing_face": "🤧", - "snow-capped_mountain": "🏔", - "snowboarder": "🏂", - "snowboarder_dark_skin_tone": "🏂🏿", - "snowboarder_light_skin_tone": "🏂🏻", - "snowboarder_medium-dark_skin_tone": "🏂🏾", - "snowboarder_medium-light_skin_tone": "🏂🏼", - "snowboarder_medium_skin_tone": "🏂🏽", - "snowflake": "❄", - "snowman": "☃", - "snowman_without_snow": "⛄", - "soap": "🧼", - "soccer_ball": "⚽", - "socks": "🧦", - "softball": "🥎", - "soft_ice_cream": "🍦", - "spade_suit": "♠", - "spaghetti": "🍝", - "sparkle": "❇", - "sparkler": "🎇", - "sparkles": "✨", - "sparkling_heart": "💖", - "speak-no-evil_monkey": "🙊", - "speaker_high_volume": "🔊", - "speaker_low_volume": "🔈", - "speaker_medium_volume": "🔉", - "speaking_head": "🗣", - "speech_balloon": "💬", - "speedboat": "🚤", - "spider": "🕷", - "spider_web": "🕸", - "spiral_calendar": "🗓", - "spiral_notepad": "🗒", - "spiral_shell": "🐚", - "spoon": "🥄", - "sponge": "🧽", - "sport_utility_vehicle": "🚙", - "sports_medal": "🏅", - "spouting_whale": "🐳", - "squid": "🦑", - "squinting_face_with_tongue": "😝", - "stadium": "🏟", - "star-struck": "🤩", - "star_and_crescent": "☪", - "star_of_david": "✡", - "station": "🚉", - "steaming_bowl": "🍜", - "stethoscope": "🩺", - "stop_button": "⏹", - "stop_sign": "🛑", - "stopwatch": "⏱", - "straight_ruler": "📏", - "strawberry": "🍓", - "studio_microphone": "🎙", - "stuffed_flatbread": "🥙", - "sun": "☀", - "sun_behind_cloud": "⛅", - "sun_behind_large_cloud": "🌥", - "sun_behind_rain_cloud": "🌦", - "sun_behind_small_cloud": "🌤", - "sun_with_face": "🌞", - "sunflower": "🌻", - "sunglasses": "😎", - "sunrise": "🌅", - "sunrise_over_mountains": "🌄", - "sunset": "🌇", - "superhero": "🦸", - "supervillain": "🦹", - "sushi": "🍣", - "suspension_railway": "🚟", - "swan": "🦢", - "sweat_droplets": "💦", - "synagogue": "🕍", - "syringe": "💉", - "t-shirt": "👕", - "taco": "🌮", - "takeout_box": "🥡", - "tanabata_tree": "🎋", - "tangerine": "🍊", - "taxi": "🚕", - "teacup_without_handle": "🍵", - "tear-off_calendar": "📆", - "teddy_bear": "🧸", - "telephone": "☎", - "telephone_receiver": "📞", - "telescope": "🔭", - "television": "📺", - "ten-thirty": "🕥", - "ten_o’clock": "🕙", - "tennis": "🎾", - "tent": "⛺", - "test_tube": "🧪", - "thermometer": "🌡", - "thinking_face": "🤔", - "thought_balloon": "💭", - "thread": "🧵", - "three-thirty": "🕞", - "three_o’clock": "🕒", - "thumbs_down": "👎", - "thumbs_down_dark_skin_tone": "👎🏿", - "thumbs_down_light_skin_tone": "👎🏻", - "thumbs_down_medium-dark_skin_tone": "👎🏾", - "thumbs_down_medium-light_skin_tone": "👎🏼", - "thumbs_down_medium_skin_tone": "👎🏽", - "thumbs_up": "👍", - "thumbs_up_dark_skin_tone": "👍🏿", - "thumbs_up_light_skin_tone": "👍🏻", - "thumbs_up_medium-dark_skin_tone": "👍🏾", - "thumbs_up_medium-light_skin_tone": "👍🏼", - "thumbs_up_medium_skin_tone": "👍🏽", - "ticket": "🎫", - "tiger": "🐯", - "tiger_face": "🐯", - "timer_clock": "⏲", - "tired_face": "😫", - "toolbox": "🧰", - "toilet": "🚽", - "tomato": "🍅", - "tongue": "👅", - "tooth": "🦷", - "top_hat": "🎩", - "tornado": "🌪", - "trackball": "🖲", - "tractor": "🚜", - "trade_mark": "™", - "train": "🚋", - "tram": "🚊", - "tram_car": "🚋", - "triangular_flag": "🚩", - "triangular_ruler": "📐", - "trident_emblem": "🔱", - "trolleybus": "🚎", - "trophy": "🏆", - "tropical_drink": "🍹", - "tropical_fish": "🐠", - "trumpet": "🎺", - "tulip": "🌷", - "tumbler_glass": "🥃", - "turtle": "🐢", - "twelve-thirty": "🕧", - "twelve_o’clock": "🕛", - "two-hump_camel": "🐫", - "two-thirty": "🕝", - "two_hearts": "💕", - "two_men_holding_hands": "👬", - "two_o’clock": "🕑", - "two_women_holding_hands": "👭", - "umbrella": "☂", - "umbrella_on_ground": "⛱", - "umbrella_with_rain_drops": "☔", - "unamused_face": "😒", - "unicorn_face": "🦄", - "unlocked": "🔓", - "up-down_arrow": "↕", - "up-left_arrow": "↖", - "up-right_arrow": "↗", - "up_arrow": "⬆", - "upside-down_face": "🙃", - "upwards_button": "🔼", - "vampire": "🧛", - "vampire_dark_skin_tone": "🧛🏿", - "vampire_light_skin_tone": "🧛🏻", - "vampire_medium-dark_skin_tone": "🧛🏾", - "vampire_medium-light_skin_tone": "🧛🏼", - "vampire_medium_skin_tone": "🧛🏽", - "vertical_traffic_light": "🚦", - "vibration_mode": "📳", - "victory_hand": "✌", - "victory_hand_dark_skin_tone": "✌🏿", - "victory_hand_light_skin_tone": "✌🏻", - "victory_hand_medium-dark_skin_tone": "✌🏾", - "victory_hand_medium-light_skin_tone": "✌🏼", - "victory_hand_medium_skin_tone": "✌🏽", - "video_camera": "📹", - "video_game": "🎮", - "videocassette": "📼", - "violin": "🎻", - "volcano": "🌋", - "volleyball": "🏐", - "vulcan_salute": "🖖", - "vulcan_salute_dark_skin_tone": "🖖🏿", - "vulcan_salute_light_skin_tone": "🖖🏻", - "vulcan_salute_medium-dark_skin_tone": "🖖🏾", - "vulcan_salute_medium-light_skin_tone": "🖖🏼", - "vulcan_salute_medium_skin_tone": "🖖🏽", - "waffle": "🧇", - "waning_crescent_moon": "🌘", - "waning_gibbous_moon": "🌖", - "warning": "⚠", - "wastebasket": "🗑", - "watch": "⌚", - "water_buffalo": "🐃", - "water_closet": "🚾", - "water_wave": "🌊", - "watermelon": "🍉", - "waving_hand": "👋", - "waving_hand_dark_skin_tone": "👋🏿", - "waving_hand_light_skin_tone": "👋🏻", - "waving_hand_medium-dark_skin_tone": "👋🏾", - "waving_hand_medium-light_skin_tone": "👋🏼", - "waving_hand_medium_skin_tone": "👋🏽", - "wavy_dash": "〰", - "waxing_crescent_moon": "🌒", - "waxing_gibbous_moon": "🌔", - "weary_cat_face": "🙀", - "weary_face": "😩", - "wedding": "💒", - "whale": "🐳", - "wheel_of_dharma": "☸", - "wheelchair_symbol": "♿", - "white_circle": "⚪", - "white_exclamation_mark": "❕", - "white_flag": "🏳", - "white_flower": "💮", - "white_hair": "🦳", - "white-haired_man": "👨\u200d🦳", - "white-haired_woman": "👩\u200d🦳", - "white_heart": "🤍", - "white_heavy_check_mark": "✅", - "white_large_square": "⬜", - "white_medium-small_square": "◽", - "white_medium_square": "◻", - "white_medium_star": "⭐", - "white_question_mark": "❔", - "white_small_square": "▫", - "white_square_button": "🔳", - "wilted_flower": "🥀", - "wind_chime": "🎐", - "wind_face": "🌬", - "wine_glass": "🍷", - "winking_face": "😉", - "winking_face_with_tongue": "😜", - "wolf_face": "🐺", - "woman": "👩", - "woman_artist": "👩\u200d🎨", - "woman_artist_dark_skin_tone": "👩🏿\u200d🎨", - "woman_artist_light_skin_tone": "👩🏻\u200d🎨", - "woman_artist_medium-dark_skin_tone": "👩🏾\u200d🎨", - "woman_artist_medium-light_skin_tone": "👩🏼\u200d🎨", - "woman_artist_medium_skin_tone": "👩🏽\u200d🎨", - "woman_astronaut": "👩\u200d🚀", - "woman_astronaut_dark_skin_tone": "👩🏿\u200d🚀", - "woman_astronaut_light_skin_tone": "👩🏻\u200d🚀", - "woman_astronaut_medium-dark_skin_tone": "👩🏾\u200d🚀", - "woman_astronaut_medium-light_skin_tone": "👩🏼\u200d🚀", - "woman_astronaut_medium_skin_tone": "👩🏽\u200d🚀", - "woman_biking": "🚴\u200d♀️", - "woman_biking_dark_skin_tone": "🚴🏿\u200d♀️", - "woman_biking_light_skin_tone": "🚴🏻\u200d♀️", - "woman_biking_medium-dark_skin_tone": "🚴🏾\u200d♀️", - "woman_biking_medium-light_skin_tone": "🚴🏼\u200d♀️", - "woman_biking_medium_skin_tone": "🚴🏽\u200d♀️", - "woman_bouncing_ball": "⛹️\u200d♀️", - "woman_bouncing_ball_dark_skin_tone": "⛹🏿\u200d♀️", - "woman_bouncing_ball_light_skin_tone": "⛹🏻\u200d♀️", - "woman_bouncing_ball_medium-dark_skin_tone": "⛹🏾\u200d♀️", - "woman_bouncing_ball_medium-light_skin_tone": "⛹🏼\u200d♀️", - "woman_bouncing_ball_medium_skin_tone": "⛹🏽\u200d♀️", - "woman_bowing": "🙇\u200d♀️", - "woman_bowing_dark_skin_tone": "🙇🏿\u200d♀️", - "woman_bowing_light_skin_tone": "🙇🏻\u200d♀️", - "woman_bowing_medium-dark_skin_tone": "🙇🏾\u200d♀️", - "woman_bowing_medium-light_skin_tone": "🙇🏼\u200d♀️", - "woman_bowing_medium_skin_tone": "🙇🏽\u200d♀️", - "woman_cartwheeling": "🤸\u200d♀️", - "woman_cartwheeling_dark_skin_tone": "🤸🏿\u200d♀️", - "woman_cartwheeling_light_skin_tone": "🤸🏻\u200d♀️", - "woman_cartwheeling_medium-dark_skin_tone": "🤸🏾\u200d♀️", - "woman_cartwheeling_medium-light_skin_tone": "🤸🏼\u200d♀️", - "woman_cartwheeling_medium_skin_tone": "🤸🏽\u200d♀️", - "woman_climbing": "🧗\u200d♀️", - "woman_climbing_dark_skin_tone": "🧗🏿\u200d♀️", - "woman_climbing_light_skin_tone": "🧗🏻\u200d♀️", - "woman_climbing_medium-dark_skin_tone": "🧗🏾\u200d♀️", - "woman_climbing_medium-light_skin_tone": "🧗🏼\u200d♀️", - "woman_climbing_medium_skin_tone": "🧗🏽\u200d♀️", - "woman_construction_worker": "👷\u200d♀️", - "woman_construction_worker_dark_skin_tone": "👷🏿\u200d♀️", - "woman_construction_worker_light_skin_tone": "👷🏻\u200d♀️", - "woman_construction_worker_medium-dark_skin_tone": "👷🏾\u200d♀️", - "woman_construction_worker_medium-light_skin_tone": "👷🏼\u200d♀️", - "woman_construction_worker_medium_skin_tone": "👷🏽\u200d♀️", - "woman_cook": "👩\u200d🍳", - "woman_cook_dark_skin_tone": "👩🏿\u200d🍳", - "woman_cook_light_skin_tone": "👩🏻\u200d🍳", - "woman_cook_medium-dark_skin_tone": "👩🏾\u200d🍳", - "woman_cook_medium-light_skin_tone": "👩🏼\u200d🍳", - "woman_cook_medium_skin_tone": "👩🏽\u200d🍳", - "woman_dancing": "💃", - "woman_dancing_dark_skin_tone": "💃🏿", - "woman_dancing_light_skin_tone": "💃🏻", - "woman_dancing_medium-dark_skin_tone": "💃🏾", - "woman_dancing_medium-light_skin_tone": "💃🏼", - "woman_dancing_medium_skin_tone": "💃🏽", - "woman_dark_skin_tone": "👩🏿", - "woman_detective": "🕵️\u200d♀️", - "woman_detective_dark_skin_tone": "🕵🏿\u200d♀️", - "woman_detective_light_skin_tone": "🕵🏻\u200d♀️", - "woman_detective_medium-dark_skin_tone": "🕵🏾\u200d♀️", - "woman_detective_medium-light_skin_tone": "🕵🏼\u200d♀️", - "woman_detective_medium_skin_tone": "🕵🏽\u200d♀️", - "woman_elf": "🧝\u200d♀️", - "woman_elf_dark_skin_tone": "🧝🏿\u200d♀️", - "woman_elf_light_skin_tone": "🧝🏻\u200d♀️", - "woman_elf_medium-dark_skin_tone": "🧝🏾\u200d♀️", - "woman_elf_medium-light_skin_tone": "🧝🏼\u200d♀️", - "woman_elf_medium_skin_tone": "🧝🏽\u200d♀️", - "woman_facepalming": "🤦\u200d♀️", - "woman_facepalming_dark_skin_tone": "🤦🏿\u200d♀️", - "woman_facepalming_light_skin_tone": "🤦🏻\u200d♀️", - "woman_facepalming_medium-dark_skin_tone": "🤦🏾\u200d♀️", - "woman_facepalming_medium-light_skin_tone": "🤦🏼\u200d♀️", - "woman_facepalming_medium_skin_tone": "🤦🏽\u200d♀️", - "woman_factory_worker": "👩\u200d🏭", - "woman_factory_worker_dark_skin_tone": "👩🏿\u200d🏭", - "woman_factory_worker_light_skin_tone": "👩🏻\u200d🏭", - "woman_factory_worker_medium-dark_skin_tone": "👩🏾\u200d🏭", - "woman_factory_worker_medium-light_skin_tone": "👩🏼\u200d🏭", - "woman_factory_worker_medium_skin_tone": "👩🏽\u200d🏭", - "woman_fairy": "🧚\u200d♀️", - "woman_fairy_dark_skin_tone": "🧚🏿\u200d♀️", - "woman_fairy_light_skin_tone": "🧚🏻\u200d♀️", - "woman_fairy_medium-dark_skin_tone": "🧚🏾\u200d♀️", - "woman_fairy_medium-light_skin_tone": "🧚🏼\u200d♀️", - "woman_fairy_medium_skin_tone": "🧚🏽\u200d♀️", - "woman_farmer": "👩\u200d🌾", - "woman_farmer_dark_skin_tone": "👩🏿\u200d🌾", - "woman_farmer_light_skin_tone": "👩🏻\u200d🌾", - "woman_farmer_medium-dark_skin_tone": "👩🏾\u200d🌾", - "woman_farmer_medium-light_skin_tone": "👩🏼\u200d🌾", - "woman_farmer_medium_skin_tone": "👩🏽\u200d🌾", - "woman_firefighter": "👩\u200d🚒", - "woman_firefighter_dark_skin_tone": "👩🏿\u200d🚒", - "woman_firefighter_light_skin_tone": "👩🏻\u200d🚒", - "woman_firefighter_medium-dark_skin_tone": "👩🏾\u200d🚒", - "woman_firefighter_medium-light_skin_tone": "👩🏼\u200d🚒", - "woman_firefighter_medium_skin_tone": "👩🏽\u200d🚒", - "woman_frowning": "🙍\u200d♀️", - "woman_frowning_dark_skin_tone": "🙍🏿\u200d♀️", - "woman_frowning_light_skin_tone": "🙍🏻\u200d♀️", - "woman_frowning_medium-dark_skin_tone": "🙍🏾\u200d♀️", - "woman_frowning_medium-light_skin_tone": "🙍🏼\u200d♀️", - "woman_frowning_medium_skin_tone": "🙍🏽\u200d♀️", - "woman_genie": "🧞\u200d♀️", - "woman_gesturing_no": "🙅\u200d♀️", - "woman_gesturing_no_dark_skin_tone": "🙅🏿\u200d♀️", - "woman_gesturing_no_light_skin_tone": "🙅🏻\u200d♀️", - "woman_gesturing_no_medium-dark_skin_tone": "🙅🏾\u200d♀️", - "woman_gesturing_no_medium-light_skin_tone": "🙅🏼\u200d♀️", - "woman_gesturing_no_medium_skin_tone": "🙅🏽\u200d♀️", - "woman_gesturing_ok": "🙆\u200d♀️", - "woman_gesturing_ok_dark_skin_tone": "🙆🏿\u200d♀️", - "woman_gesturing_ok_light_skin_tone": "🙆🏻\u200d♀️", - "woman_gesturing_ok_medium-dark_skin_tone": "🙆🏾\u200d♀️", - "woman_gesturing_ok_medium-light_skin_tone": "🙆🏼\u200d♀️", - "woman_gesturing_ok_medium_skin_tone": "🙆🏽\u200d♀️", - "woman_getting_haircut": "💇\u200d♀️", - "woman_getting_haircut_dark_skin_tone": "💇🏿\u200d♀️", - "woman_getting_haircut_light_skin_tone": "💇🏻\u200d♀️", - "woman_getting_haircut_medium-dark_skin_tone": "💇🏾\u200d♀️", - "woman_getting_haircut_medium-light_skin_tone": "💇🏼\u200d♀️", - "woman_getting_haircut_medium_skin_tone": "💇🏽\u200d♀️", - "woman_getting_massage": "💆\u200d♀️", - "woman_getting_massage_dark_skin_tone": "💆🏿\u200d♀️", - "woman_getting_massage_light_skin_tone": "💆🏻\u200d♀️", - "woman_getting_massage_medium-dark_skin_tone": "💆🏾\u200d♀️", - "woman_getting_massage_medium-light_skin_tone": "💆🏼\u200d♀️", - "woman_getting_massage_medium_skin_tone": "💆🏽\u200d♀️", - "woman_golfing": "🏌️\u200d♀️", - "woman_golfing_dark_skin_tone": "🏌🏿\u200d♀️", - "woman_golfing_light_skin_tone": "🏌🏻\u200d♀️", - "woman_golfing_medium-dark_skin_tone": "🏌🏾\u200d♀️", - "woman_golfing_medium-light_skin_tone": "🏌🏼\u200d♀️", - "woman_golfing_medium_skin_tone": "🏌🏽\u200d♀️", - "woman_guard": "💂\u200d♀️", - "woman_guard_dark_skin_tone": "💂🏿\u200d♀️", - "woman_guard_light_skin_tone": "💂🏻\u200d♀️", - "woman_guard_medium-dark_skin_tone": "💂🏾\u200d♀️", - "woman_guard_medium-light_skin_tone": "💂🏼\u200d♀️", - "woman_guard_medium_skin_tone": "💂🏽\u200d♀️", - "woman_health_worker": "👩\u200d⚕️", - "woman_health_worker_dark_skin_tone": "👩🏿\u200d⚕️", - "woman_health_worker_light_skin_tone": "👩🏻\u200d⚕️", - "woman_health_worker_medium-dark_skin_tone": "👩🏾\u200d⚕️", - "woman_health_worker_medium-light_skin_tone": "👩🏼\u200d⚕️", - "woman_health_worker_medium_skin_tone": "👩🏽\u200d⚕️", - "woman_in_lotus_position": "🧘\u200d♀️", - "woman_in_lotus_position_dark_skin_tone": "🧘🏿\u200d♀️", - "woman_in_lotus_position_light_skin_tone": "🧘🏻\u200d♀️", - "woman_in_lotus_position_medium-dark_skin_tone": "🧘🏾\u200d♀️", - "woman_in_lotus_position_medium-light_skin_tone": "🧘🏼\u200d♀️", - "woman_in_lotus_position_medium_skin_tone": "🧘🏽\u200d♀️", - "woman_in_manual_wheelchair": "👩\u200d🦽", - "woman_in_motorized_wheelchair": "👩\u200d🦼", - "woman_in_steamy_room": "🧖\u200d♀️", - "woman_in_steamy_room_dark_skin_tone": "🧖🏿\u200d♀️", - "woman_in_steamy_room_light_skin_tone": "🧖🏻\u200d♀️", - "woman_in_steamy_room_medium-dark_skin_tone": "🧖🏾\u200d♀️", - "woman_in_steamy_room_medium-light_skin_tone": "🧖🏼\u200d♀️", - "woman_in_steamy_room_medium_skin_tone": "🧖🏽\u200d♀️", - "woman_judge": "👩\u200d⚖️", - "woman_judge_dark_skin_tone": "👩🏿\u200d⚖️", - "woman_judge_light_skin_tone": "👩🏻\u200d⚖️", - "woman_judge_medium-dark_skin_tone": "👩🏾\u200d⚖️", - "woman_judge_medium-light_skin_tone": "👩🏼\u200d⚖️", - "woman_judge_medium_skin_tone": "👩🏽\u200d⚖️", - "woman_juggling": "🤹\u200d♀️", - "woman_juggling_dark_skin_tone": "🤹🏿\u200d♀️", - "woman_juggling_light_skin_tone": "🤹🏻\u200d♀️", - "woman_juggling_medium-dark_skin_tone": "🤹🏾\u200d♀️", - "woman_juggling_medium-light_skin_tone": "🤹🏼\u200d♀️", - "woman_juggling_medium_skin_tone": "🤹🏽\u200d♀️", - "woman_lifting_weights": "🏋️\u200d♀️", - "woman_lifting_weights_dark_skin_tone": "🏋🏿\u200d♀️", - "woman_lifting_weights_light_skin_tone": "🏋🏻\u200d♀️", - "woman_lifting_weights_medium-dark_skin_tone": "🏋🏾\u200d♀️", - "woman_lifting_weights_medium-light_skin_tone": "🏋🏼\u200d♀️", - "woman_lifting_weights_medium_skin_tone": "🏋🏽\u200d♀️", - "woman_light_skin_tone": "👩🏻", - "woman_mage": "🧙\u200d♀️", - "woman_mage_dark_skin_tone": "🧙🏿\u200d♀️", - "woman_mage_light_skin_tone": "🧙🏻\u200d♀️", - "woman_mage_medium-dark_skin_tone": "🧙🏾\u200d♀️", - "woman_mage_medium-light_skin_tone": "🧙🏼\u200d♀️", - "woman_mage_medium_skin_tone": "🧙🏽\u200d♀️", - "woman_mechanic": "👩\u200d🔧", - "woman_mechanic_dark_skin_tone": "👩🏿\u200d🔧", - "woman_mechanic_light_skin_tone": "👩🏻\u200d🔧", - "woman_mechanic_medium-dark_skin_tone": "👩🏾\u200d🔧", - "woman_mechanic_medium-light_skin_tone": "👩🏼\u200d🔧", - "woman_mechanic_medium_skin_tone": "👩🏽\u200d🔧", - "woman_medium-dark_skin_tone": "👩🏾", - "woman_medium-light_skin_tone": "👩🏼", - "woman_medium_skin_tone": "👩🏽", - "woman_mountain_biking": "🚵\u200d♀️", - "woman_mountain_biking_dark_skin_tone": "🚵🏿\u200d♀️", - "woman_mountain_biking_light_skin_tone": "🚵🏻\u200d♀️", - "woman_mountain_biking_medium-dark_skin_tone": "🚵🏾\u200d♀️", - "woman_mountain_biking_medium-light_skin_tone": "🚵🏼\u200d♀️", - "woman_mountain_biking_medium_skin_tone": "🚵🏽\u200d♀️", - "woman_office_worker": "👩\u200d💼", - "woman_office_worker_dark_skin_tone": "👩🏿\u200d💼", - "woman_office_worker_light_skin_tone": "👩🏻\u200d💼", - "woman_office_worker_medium-dark_skin_tone": "👩🏾\u200d💼", - "woman_office_worker_medium-light_skin_tone": "👩🏼\u200d💼", - "woman_office_worker_medium_skin_tone": "👩🏽\u200d💼", - "woman_pilot": "👩\u200d✈️", - "woman_pilot_dark_skin_tone": "👩🏿\u200d✈️", - "woman_pilot_light_skin_tone": "👩🏻\u200d✈️", - "woman_pilot_medium-dark_skin_tone": "👩🏾\u200d✈️", - "woman_pilot_medium-light_skin_tone": "👩🏼\u200d✈️", - "woman_pilot_medium_skin_tone": "👩🏽\u200d✈️", - "woman_playing_handball": "🤾\u200d♀️", - "woman_playing_handball_dark_skin_tone": "🤾🏿\u200d♀️", - "woman_playing_handball_light_skin_tone": "🤾🏻\u200d♀️", - "woman_playing_handball_medium-dark_skin_tone": "🤾🏾\u200d♀️", - "woman_playing_handball_medium-light_skin_tone": "🤾🏼\u200d♀️", - "woman_playing_handball_medium_skin_tone": "🤾🏽\u200d♀️", - "woman_playing_water_polo": "🤽\u200d♀️", - "woman_playing_water_polo_dark_skin_tone": "🤽🏿\u200d♀️", - "woman_playing_water_polo_light_skin_tone": "🤽🏻\u200d♀️", - "woman_playing_water_polo_medium-dark_skin_tone": "🤽🏾\u200d♀️", - "woman_playing_water_polo_medium-light_skin_tone": "🤽🏼\u200d♀️", - "woman_playing_water_polo_medium_skin_tone": "🤽🏽\u200d♀️", - "woman_police_officer": "👮\u200d♀️", - "woman_police_officer_dark_skin_tone": "👮🏿\u200d♀️", - "woman_police_officer_light_skin_tone": "👮🏻\u200d♀️", - "woman_police_officer_medium-dark_skin_tone": "👮🏾\u200d♀️", - "woman_police_officer_medium-light_skin_tone": "👮🏼\u200d♀️", - "woman_police_officer_medium_skin_tone": "👮🏽\u200d♀️", - "woman_pouting": "🙎\u200d♀️", - "woman_pouting_dark_skin_tone": "🙎🏿\u200d♀️", - "woman_pouting_light_skin_tone": "🙎🏻\u200d♀️", - "woman_pouting_medium-dark_skin_tone": "🙎🏾\u200d♀️", - "woman_pouting_medium-light_skin_tone": "🙎🏼\u200d♀️", - "woman_pouting_medium_skin_tone": "🙎🏽\u200d♀️", - "woman_raising_hand": "🙋\u200d♀️", - "woman_raising_hand_dark_skin_tone": "🙋🏿\u200d♀️", - "woman_raising_hand_light_skin_tone": "🙋🏻\u200d♀️", - "woman_raising_hand_medium-dark_skin_tone": "🙋🏾\u200d♀️", - "woman_raising_hand_medium-light_skin_tone": "🙋🏼\u200d♀️", - "woman_raising_hand_medium_skin_tone": "🙋🏽\u200d♀️", - "woman_rowing_boat": "🚣\u200d♀️", - "woman_rowing_boat_dark_skin_tone": "🚣🏿\u200d♀️", - "woman_rowing_boat_light_skin_tone": "🚣🏻\u200d♀️", - "woman_rowing_boat_medium-dark_skin_tone": "🚣🏾\u200d♀️", - "woman_rowing_boat_medium-light_skin_tone": "🚣🏼\u200d♀️", - "woman_rowing_boat_medium_skin_tone": "🚣🏽\u200d♀️", - "woman_running": "🏃\u200d♀️", - "woman_running_dark_skin_tone": "🏃🏿\u200d♀️", - "woman_running_light_skin_tone": "🏃🏻\u200d♀️", - "woman_running_medium-dark_skin_tone": "🏃🏾\u200d♀️", - "woman_running_medium-light_skin_tone": "🏃🏼\u200d♀️", - "woman_running_medium_skin_tone": "🏃🏽\u200d♀️", - "woman_scientist": "👩\u200d🔬", - "woman_scientist_dark_skin_tone": "👩🏿\u200d🔬", - "woman_scientist_light_skin_tone": "👩🏻\u200d🔬", - "woman_scientist_medium-dark_skin_tone": "👩🏾\u200d🔬", - "woman_scientist_medium-light_skin_tone": "👩🏼\u200d🔬", - "woman_scientist_medium_skin_tone": "👩🏽\u200d🔬", - "woman_shrugging": "🤷\u200d♀️", - "woman_shrugging_dark_skin_tone": "🤷🏿\u200d♀️", - "woman_shrugging_light_skin_tone": "🤷🏻\u200d♀️", - "woman_shrugging_medium-dark_skin_tone": "🤷🏾\u200d♀️", - "woman_shrugging_medium-light_skin_tone": "🤷🏼\u200d♀️", - "woman_shrugging_medium_skin_tone": "🤷🏽\u200d♀️", - "woman_singer": "👩\u200d🎤", - "woman_singer_dark_skin_tone": "👩🏿\u200d🎤", - "woman_singer_light_skin_tone": "👩🏻\u200d🎤", - "woman_singer_medium-dark_skin_tone": "👩🏾\u200d🎤", - "woman_singer_medium-light_skin_tone": "👩🏼\u200d🎤", - "woman_singer_medium_skin_tone": "👩🏽\u200d🎤", - "woman_student": "👩\u200d🎓", - "woman_student_dark_skin_tone": "👩🏿\u200d🎓", - "woman_student_light_skin_tone": "👩🏻\u200d🎓", - "woman_student_medium-dark_skin_tone": "👩🏾\u200d🎓", - "woman_student_medium-light_skin_tone": "👩🏼\u200d🎓", - "woman_student_medium_skin_tone": "👩🏽\u200d🎓", - "woman_surfing": "🏄\u200d♀️", - "woman_surfing_dark_skin_tone": "🏄🏿\u200d♀️", - "woman_surfing_light_skin_tone": "🏄🏻\u200d♀️", - "woman_surfing_medium-dark_skin_tone": "🏄🏾\u200d♀️", - "woman_surfing_medium-light_skin_tone": "🏄🏼\u200d♀️", - "woman_surfing_medium_skin_tone": "🏄🏽\u200d♀️", - "woman_swimming": "🏊\u200d♀️", - "woman_swimming_dark_skin_tone": "🏊🏿\u200d♀️", - "woman_swimming_light_skin_tone": "🏊🏻\u200d♀️", - "woman_swimming_medium-dark_skin_tone": "🏊🏾\u200d♀️", - "woman_swimming_medium-light_skin_tone": "🏊🏼\u200d♀️", - "woman_swimming_medium_skin_tone": "🏊🏽\u200d♀️", - "woman_teacher": "👩\u200d🏫", - "woman_teacher_dark_skin_tone": "👩🏿\u200d🏫", - "woman_teacher_light_skin_tone": "👩🏻\u200d🏫", - "woman_teacher_medium-dark_skin_tone": "👩🏾\u200d🏫", - "woman_teacher_medium-light_skin_tone": "👩🏼\u200d🏫", - "woman_teacher_medium_skin_tone": "👩🏽\u200d🏫", - "woman_technologist": "👩\u200d💻", - "woman_technologist_dark_skin_tone": "👩🏿\u200d💻", - "woman_technologist_light_skin_tone": "👩🏻\u200d💻", - "woman_technologist_medium-dark_skin_tone": "👩🏾\u200d💻", - "woman_technologist_medium-light_skin_tone": "👩🏼\u200d💻", - "woman_technologist_medium_skin_tone": "👩🏽\u200d💻", - "woman_tipping_hand": "💁\u200d♀️", - "woman_tipping_hand_dark_skin_tone": "💁🏿\u200d♀️", - "woman_tipping_hand_light_skin_tone": "💁🏻\u200d♀️", - "woman_tipping_hand_medium-dark_skin_tone": "💁🏾\u200d♀️", - "woman_tipping_hand_medium-light_skin_tone": "💁🏼\u200d♀️", - "woman_tipping_hand_medium_skin_tone": "💁🏽\u200d♀️", - "woman_vampire": "🧛\u200d♀️", - "woman_vampire_dark_skin_tone": "🧛🏿\u200d♀️", - "woman_vampire_light_skin_tone": "🧛🏻\u200d♀️", - "woman_vampire_medium-dark_skin_tone": "🧛🏾\u200d♀️", - "woman_vampire_medium-light_skin_tone": "🧛🏼\u200d♀️", - "woman_vampire_medium_skin_tone": "🧛🏽\u200d♀️", - "woman_walking": "🚶\u200d♀️", - "woman_walking_dark_skin_tone": "🚶🏿\u200d♀️", - "woman_walking_light_skin_tone": "🚶🏻\u200d♀️", - "woman_walking_medium-dark_skin_tone": "🚶🏾\u200d♀️", - "woman_walking_medium-light_skin_tone": "🚶🏼\u200d♀️", - "woman_walking_medium_skin_tone": "🚶🏽\u200d♀️", - "woman_wearing_turban": "👳\u200d♀️", - "woman_wearing_turban_dark_skin_tone": "👳🏿\u200d♀️", - "woman_wearing_turban_light_skin_tone": "👳🏻\u200d♀️", - "woman_wearing_turban_medium-dark_skin_tone": "👳🏾\u200d♀️", - "woman_wearing_turban_medium-light_skin_tone": "👳🏼\u200d♀️", - "woman_wearing_turban_medium_skin_tone": "👳🏽\u200d♀️", - "woman_with_headscarf": "🧕", - "woman_with_headscarf_dark_skin_tone": "🧕🏿", - "woman_with_headscarf_light_skin_tone": "🧕🏻", - "woman_with_headscarf_medium-dark_skin_tone": "🧕🏾", - "woman_with_headscarf_medium-light_skin_tone": "🧕🏼", - "woman_with_headscarf_medium_skin_tone": "🧕🏽", - "woman_with_probing_cane": "👩\u200d🦯", - "woman_zombie": "🧟\u200d♀️", - "woman’s_boot": "👢", - "woman’s_clothes": "👚", - "woman’s_hat": "👒", - "woman’s_sandal": "👡", - "women_with_bunny_ears": "👯\u200d♀️", - "women_wrestling": "🤼\u200d♀️", - "women’s_room": "🚺", - "woozy_face": "🥴", - "world_map": "🗺", - "worried_face": "😟", - "wrapped_gift": "🎁", - "wrench": "🔧", - "writing_hand": "✍", - "writing_hand_dark_skin_tone": "✍🏿", - "writing_hand_light_skin_tone": "✍🏻", - "writing_hand_medium-dark_skin_tone": "✍🏾", - "writing_hand_medium-light_skin_tone": "✍🏼", - "writing_hand_medium_skin_tone": "✍🏽", - "yarn": "🧶", - "yawning_face": "🥱", - "yellow_circle": "🟡", - "yellow_heart": "💛", - "yellow_square": "🟨", - "yen_banknote": "💴", - "yo-yo": "🪀", - "yin_yang": "☯", - "zany_face": "🤪", - "zebra": "🦓", - "zipper-mouth_face": "🤐", - "zombie": "🧟", - "zzz": "💤", - "åland_islands": "🇦🇽", - "keycap_asterisk": "*⃣", - "keycap_digit_eight": "8⃣", - "keycap_digit_five": "5⃣", - "keycap_digit_four": "4⃣", - "keycap_digit_nine": "9⃣", - "keycap_digit_one": "1⃣", - "keycap_digit_seven": "7⃣", - "keycap_digit_six": "6⃣", - "keycap_digit_three": "3⃣", - "keycap_digit_two": "2⃣", - "keycap_digit_zero": "0⃣", - "keycap_number_sign": "#⃣", - "light_skin_tone": "🏻", - "medium_light_skin_tone": "🏼", - "medium_skin_tone": "🏽", - "medium_dark_skin_tone": "🏾", - "dark_skin_tone": "🏿", - "regional_indicator_symbol_letter_a": "🇦", - "regional_indicator_symbol_letter_b": "🇧", - "regional_indicator_symbol_letter_c": "🇨", - "regional_indicator_symbol_letter_d": "🇩", - "regional_indicator_symbol_letter_e": "🇪", - "regional_indicator_symbol_letter_f": "🇫", - "regional_indicator_symbol_letter_g": "🇬", - "regional_indicator_symbol_letter_h": "🇭", - "regional_indicator_symbol_letter_i": "🇮", - "regional_indicator_symbol_letter_j": "🇯", - "regional_indicator_symbol_letter_k": "🇰", - "regional_indicator_symbol_letter_l": "🇱", - "regional_indicator_symbol_letter_m": "🇲", - "regional_indicator_symbol_letter_n": "🇳", - "regional_indicator_symbol_letter_o": "🇴", - "regional_indicator_symbol_letter_p": "🇵", - "regional_indicator_symbol_letter_q": "🇶", - "regional_indicator_symbol_letter_r": "🇷", - "regional_indicator_symbol_letter_s": "🇸", - "regional_indicator_symbol_letter_t": "🇹", - "regional_indicator_symbol_letter_u": "🇺", - "regional_indicator_symbol_letter_v": "🇻", - "regional_indicator_symbol_letter_w": "🇼", - "regional_indicator_symbol_letter_x": "🇽", - "regional_indicator_symbol_letter_y": "🇾", - "regional_indicator_symbol_letter_z": "🇿", - "airplane_arriving": "🛬", - "space_invader": "👾", - "football": "🏈", - "anger": "💢", - "angry": "😠", - "anguished": "😧", - "signal_strength": "📶", - "arrows_counterclockwise": "🔄", - "arrow_heading_down": "⤵", - "arrow_heading_up": "⤴", - "art": "🎨", - "astonished": "😲", - "athletic_shoe": "👟", - "atm": "🏧", - "car": "🚗", - "red_car": "🚗", - "angel": "👼", - "back": "🔙", - "badminton_racquet_and_shuttlecock": "🏸", - "dollar": "💵", - "euro": "💶", - "pound": "💷", - "yen": "💴", - "barber": "💈", - "bath": "🛀", - "bear": "🐻", - "heartbeat": "💓", - "beer": "🍺", - "no_bell": "🔕", - "bento": "🍱", - "bike": "🚲", - "bicyclist": "🚴", - "8ball": "🎱", - "biohazard_sign": "☣", - "birthday": "🎂", - "black_circle_for_record": "⏺", - "clubs": "♣", - "diamonds": "♦", - "arrow_double_down": "⏬", - "hearts": "♥", - "rewind": "⏪", - "black_left__pointing_double_triangle_with_vertical_bar": "⏮", - "arrow_backward": "◀", - "black_medium_small_square": "◾", - "question": "❓", - "fast_forward": "⏩", - "black_right__pointing_double_triangle_with_vertical_bar": "⏭", - "arrow_forward": "▶", - "black_right__pointing_triangle_with_double_vertical_bar": "⏯", - "arrow_right": "➡", - "spades": "♠", - "black_square_for_stop": "⏹", - "sunny": "☀", - "phone": "☎", - "recycle": "♻", - "arrow_double_up": "⏫", - "busstop": "🚏", - "date": "📅", - "flags": "🎏", - "cat2": "🐈", - "joy_cat": "😹", - "smirk_cat": "😼", - "chart_with_downwards_trend": "📉", - "chart_with_upwards_trend": "📈", - "chart": "💹", - "mega": "📣", - "checkered_flag": "🏁", - "accept": "🉑", - "ideograph_advantage": "🉐", - "congratulations": "㊗", - "secret": "㊙", - "m": "Ⓜ", - "city_sunset": "🌆", - "clapper": "🎬", - "clap": "👏", - "beers": "🍻", - "clock830": "🕣", - "clock8": "🕗", - "clock1130": "🕦", - "clock11": "🕚", - "clock530": "🕠", - "clock5": "🕔", - "clock430": "🕟", - "clock4": "🕓", - "clock930": "🕤", - "clock9": "🕘", - "clock130": "🕜", - "clock1": "🕐", - "clock730": "🕢", - "clock7": "🕖", - "clock630": "🕡", - "clock6": "🕕", - "clock1030": "🕥", - "clock10": "🕙", - "clock330": "🕞", - "clock3": "🕒", - "clock1230": "🕧", - "clock12": "🕛", - "clock230": "🕝", - "clock2": "🕑", - "arrows_clockwise": "🔃", - "repeat": "🔁", - "repeat_one": "🔂", - "closed_lock_with_key": "🔐", - "mailbox_closed": "📪", - "mailbox": "📫", - "cloud_with_tornado": "🌪", - "cocktail": "🍸", - "boom": "💥", - "compression": "🗜", - "confounded": "😖", - "confused": "😕", - "rice": "🍚", - "cow2": "🐄", - "cricket_bat_and_ball": "🏏", - "x": "❌", - "cry": "😢", - "curry": "🍛", - "dagger_knife": "🗡", - "dancer": "💃", - "dark_sunglasses": "🕶", - "dash": "💨", - "truck": "🚚", - "derelict_house_building": "🏚", - "diamond_shape_with_a_dot_inside": "💠", - "dart": "🎯", - "disappointed_relieved": "😥", - "disappointed": "😞", - "do_not_litter": "🚯", - "dog2": "🐕", - "flipper": "🐬", - "loop": "➿", - "bangbang": "‼", - "double_vertical_bar": "⏸", - "dove_of_peace": "🕊", - "small_red_triangle_down": "🔻", - "arrow_down_small": "🔽", - "arrow_down": "⬇", - "dromedary_camel": "🐪", - "e__mail": "📧", - "corn": "🌽", - "ear_of_rice": "🌾", - "earth_americas": "🌎", - "earth_asia": "🌏", - "earth_africa": "🌍", - "eight_pointed_black_star": "✴", - "eight_spoked_asterisk": "✳", - "eject_symbol": "⏏", - "bulb": "💡", - "emoji_modifier_fitzpatrick_type__1__2": "🏻", - "emoji_modifier_fitzpatrick_type__3": "🏼", - "emoji_modifier_fitzpatrick_type__4": "🏽", - "emoji_modifier_fitzpatrick_type__5": "🏾", - "emoji_modifier_fitzpatrick_type__6": "🏿", - "end": "🔚", - "email": "✉", - "european_castle": "🏰", - "european_post_office": "🏤", - "interrobang": "⁉", - "expressionless": "😑", - "eyeglasses": "👓", - "massage": "💆", - "yum": "😋", - "scream": "😱", - "kissing_heart": "😘", - "sweat": "😓", - "face_with_head__bandage": "🤕", - "triumph": "😤", - "mask": "😷", - "no_good": "🙅", - "ok_woman": "🙆", - "open_mouth": "😮", - "cold_sweat": "😰", - "stuck_out_tongue": "😛", - "stuck_out_tongue_closed_eyes": "😝", - "stuck_out_tongue_winking_eye": "😜", - "joy": "😂", - "no_mouth": "😶", - "santa": "🎅", - "fax": "📠", - "fearful": "😨", - "field_hockey_stick_and_ball": "🏑", - "first_quarter_moon_with_face": "🌛", - "fish_cake": "🍥", - "fishing_pole_and_fish": "🎣", - "facepunch": "👊", - "punch": "👊", - "flag_for_afghanistan": "🇦🇫", - "flag_for_albania": "🇦🇱", - "flag_for_algeria": "🇩🇿", - "flag_for_american_samoa": "🇦🇸", - "flag_for_andorra": "🇦🇩", - "flag_for_angola": "🇦🇴", - "flag_for_anguilla": "🇦🇮", - "flag_for_antarctica": "🇦🇶", - "flag_for_antigua_&_barbuda": "🇦🇬", - "flag_for_argentina": "🇦🇷", - "flag_for_armenia": "🇦🇲", - "flag_for_aruba": "🇦🇼", - "flag_for_ascension_island": "🇦🇨", - "flag_for_australia": "🇦🇺", - "flag_for_austria": "🇦🇹", - "flag_for_azerbaijan": "🇦🇿", - "flag_for_bahamas": "🇧🇸", - "flag_for_bahrain": "🇧🇭", - "flag_for_bangladesh": "🇧🇩", - "flag_for_barbados": "🇧🇧", - "flag_for_belarus": "🇧🇾", - "flag_for_belgium": "🇧🇪", - "flag_for_belize": "🇧🇿", - "flag_for_benin": "🇧🇯", - "flag_for_bermuda": "🇧🇲", - "flag_for_bhutan": "🇧🇹", - "flag_for_bolivia": "🇧🇴", - "flag_for_bosnia_&_herzegovina": "🇧🇦", - "flag_for_botswana": "🇧🇼", - "flag_for_bouvet_island": "🇧🇻", - "flag_for_brazil": "🇧🇷", - "flag_for_british_indian_ocean_territory": "🇮🇴", - "flag_for_british_virgin_islands": "🇻🇬", - "flag_for_brunei": "🇧🇳", - "flag_for_bulgaria": "🇧🇬", - "flag_for_burkina_faso": "🇧🇫", - "flag_for_burundi": "🇧🇮", - "flag_for_cambodia": "🇰🇭", - "flag_for_cameroon": "🇨🇲", - "flag_for_canada": "🇨🇦", - "flag_for_canary_islands": "🇮🇨", - "flag_for_cape_verde": "🇨🇻", - "flag_for_caribbean_netherlands": "🇧🇶", - "flag_for_cayman_islands": "🇰🇾", - "flag_for_central_african_republic": "🇨🇫", - "flag_for_ceuta_&_melilla": "🇪🇦", - "flag_for_chad": "🇹🇩", - "flag_for_chile": "🇨🇱", - "flag_for_china": "🇨🇳", - "flag_for_christmas_island": "🇨🇽", - "flag_for_clipperton_island": "🇨🇵", - "flag_for_cocos__islands": "🇨🇨", - "flag_for_colombia": "🇨🇴", - "flag_for_comoros": "🇰🇲", - "flag_for_congo____brazzaville": "🇨🇬", - "flag_for_congo____kinshasa": "🇨🇩", - "flag_for_cook_islands": "🇨🇰", - "flag_for_costa_rica": "🇨🇷", - "flag_for_croatia": "🇭🇷", - "flag_for_cuba": "🇨🇺", - "flag_for_curaçao": "🇨🇼", - "flag_for_cyprus": "🇨🇾", - "flag_for_czech_republic": "🇨🇿", - "flag_for_côte_d’ivoire": "🇨🇮", - "flag_for_denmark": "🇩🇰", - "flag_for_diego_garcia": "🇩🇬", - "flag_for_djibouti": "🇩🇯", - "flag_for_dominica": "🇩🇲", - "flag_for_dominican_republic": "🇩🇴", - "flag_for_ecuador": "🇪🇨", - "flag_for_egypt": "🇪🇬", - "flag_for_el_salvador": "🇸🇻", - "flag_for_equatorial_guinea": "🇬🇶", - "flag_for_eritrea": "🇪🇷", - "flag_for_estonia": "🇪🇪", - "flag_for_ethiopia": "🇪🇹", - "flag_for_european_union": "🇪🇺", - "flag_for_falkland_islands": "🇫🇰", - "flag_for_faroe_islands": "🇫🇴", - "flag_for_fiji": "🇫🇯", - "flag_for_finland": "🇫🇮", - "flag_for_france": "🇫🇷", - "flag_for_french_guiana": "🇬🇫", - "flag_for_french_polynesia": "🇵🇫", - "flag_for_french_southern_territories": "🇹🇫", - "flag_for_gabon": "🇬🇦", - "flag_for_gambia": "🇬🇲", - "flag_for_georgia": "🇬🇪", - "flag_for_germany": "🇩🇪", - "flag_for_ghana": "🇬🇭", - "flag_for_gibraltar": "🇬🇮", - "flag_for_greece": "🇬🇷", - "flag_for_greenland": "🇬🇱", - "flag_for_grenada": "🇬🇩", - "flag_for_guadeloupe": "🇬🇵", - "flag_for_guam": "🇬🇺", - "flag_for_guatemala": "🇬🇹", - "flag_for_guernsey": "🇬🇬", - "flag_for_guinea": "🇬🇳", - "flag_for_guinea__bissau": "🇬🇼", - "flag_for_guyana": "🇬🇾", - "flag_for_haiti": "🇭🇹", - "flag_for_heard_&_mcdonald_islands": "🇭🇲", - "flag_for_honduras": "🇭🇳", - "flag_for_hong_kong": "🇭🇰", - "flag_for_hungary": "🇭🇺", - "flag_for_iceland": "🇮🇸", - "flag_for_india": "🇮🇳", - "flag_for_indonesia": "🇮🇩", - "flag_for_iran": "🇮🇷", - "flag_for_iraq": "🇮🇶", - "flag_for_ireland": "🇮🇪", - "flag_for_isle_of_man": "🇮🇲", - "flag_for_israel": "🇮🇱", - "flag_for_italy": "🇮🇹", - "flag_for_jamaica": "🇯🇲", - "flag_for_japan": "🇯🇵", - "flag_for_jersey": "🇯🇪", - "flag_for_jordan": "🇯🇴", - "flag_for_kazakhstan": "🇰🇿", - "flag_for_kenya": "🇰🇪", - "flag_for_kiribati": "🇰🇮", - "flag_for_kosovo": "🇽🇰", - "flag_for_kuwait": "🇰🇼", - "flag_for_kyrgyzstan": "🇰🇬", - "flag_for_laos": "🇱🇦", - "flag_for_latvia": "🇱🇻", - "flag_for_lebanon": "🇱🇧", - "flag_for_lesotho": "🇱🇸", - "flag_for_liberia": "🇱🇷", - "flag_for_libya": "🇱🇾", - "flag_for_liechtenstein": "🇱🇮", - "flag_for_lithuania": "🇱🇹", - "flag_for_luxembourg": "🇱🇺", - "flag_for_macau": "🇲🇴", - "flag_for_macedonia": "🇲🇰", - "flag_for_madagascar": "🇲🇬", - "flag_for_malawi": "🇲🇼", - "flag_for_malaysia": "🇲🇾", - "flag_for_maldives": "🇲🇻", - "flag_for_mali": "🇲🇱", - "flag_for_malta": "🇲🇹", - "flag_for_marshall_islands": "🇲🇭", - "flag_for_martinique": "🇲🇶", - "flag_for_mauritania": "🇲🇷", - "flag_for_mauritius": "🇲🇺", - "flag_for_mayotte": "🇾🇹", - "flag_for_mexico": "🇲🇽", - "flag_for_micronesia": "🇫🇲", - "flag_for_moldova": "🇲🇩", - "flag_for_monaco": "🇲🇨", - "flag_for_mongolia": "🇲🇳", - "flag_for_montenegro": "🇲🇪", - "flag_for_montserrat": "🇲🇸", - "flag_for_morocco": "🇲🇦", - "flag_for_mozambique": "🇲🇿", - "flag_for_myanmar": "🇲🇲", - "flag_for_namibia": "🇳🇦", - "flag_for_nauru": "🇳🇷", - "flag_for_nepal": "🇳🇵", - "flag_for_netherlands": "🇳🇱", - "flag_for_new_caledonia": "🇳🇨", - "flag_for_new_zealand": "🇳🇿", - "flag_for_nicaragua": "🇳🇮", - "flag_for_niger": "🇳🇪", - "flag_for_nigeria": "🇳🇬", - "flag_for_niue": "🇳🇺", - "flag_for_norfolk_island": "🇳🇫", - "flag_for_north_korea": "🇰🇵", - "flag_for_northern_mariana_islands": "🇲🇵", - "flag_for_norway": "🇳🇴", - "flag_for_oman": "🇴🇲", - "flag_for_pakistan": "🇵🇰", - "flag_for_palau": "🇵🇼", - "flag_for_palestinian_territories": "🇵🇸", - "flag_for_panama": "🇵🇦", - "flag_for_papua_new_guinea": "🇵🇬", - "flag_for_paraguay": "🇵🇾", - "flag_for_peru": "🇵🇪", - "flag_for_philippines": "🇵🇭", - "flag_for_pitcairn_islands": "🇵🇳", - "flag_for_poland": "🇵🇱", - "flag_for_portugal": "🇵🇹", - "flag_for_puerto_rico": "🇵🇷", - "flag_for_qatar": "🇶🇦", - "flag_for_romania": "🇷🇴", - "flag_for_russia": "🇷🇺", - "flag_for_rwanda": "🇷🇼", - "flag_for_réunion": "🇷🇪", - "flag_for_samoa": "🇼🇸", - "flag_for_san_marino": "🇸🇲", - "flag_for_saudi_arabia": "🇸🇦", - "flag_for_senegal": "🇸🇳", - "flag_for_serbia": "🇷🇸", - "flag_for_seychelles": "🇸🇨", - "flag_for_sierra_leone": "🇸🇱", - "flag_for_singapore": "🇸🇬", - "flag_for_sint_maarten": "🇸🇽", - "flag_for_slovakia": "🇸🇰", - "flag_for_slovenia": "🇸🇮", - "flag_for_solomon_islands": "🇸🇧", - "flag_for_somalia": "🇸🇴", - "flag_for_south_africa": "🇿🇦", - "flag_for_south_georgia_&_south_sandwich_islands": "🇬🇸", - "flag_for_south_korea": "🇰🇷", - "flag_for_south_sudan": "🇸🇸", - "flag_for_spain": "🇪🇸", - "flag_for_sri_lanka": "🇱🇰", - "flag_for_st._barthélemy": "🇧🇱", - "flag_for_st._helena": "🇸🇭", - "flag_for_st._kitts_&_nevis": "🇰🇳", - "flag_for_st._lucia": "🇱🇨", - "flag_for_st._martin": "🇲🇫", - "flag_for_st._pierre_&_miquelon": "🇵🇲", - "flag_for_st._vincent_&_grenadines": "🇻🇨", - "flag_for_sudan": "🇸🇩", - "flag_for_suriname": "🇸🇷", - "flag_for_svalbard_&_jan_mayen": "🇸🇯", - "flag_for_swaziland": "🇸🇿", - "flag_for_sweden": "🇸🇪", - "flag_for_switzerland": "🇨🇭", - "flag_for_syria": "🇸🇾", - "flag_for_são_tomé_&_príncipe": "🇸🇹", - "flag_for_taiwan": "🇹🇼", - "flag_for_tajikistan": "🇹🇯", - "flag_for_tanzania": "🇹🇿", - "flag_for_thailand": "🇹🇭", - "flag_for_timor__leste": "🇹🇱", - "flag_for_togo": "🇹🇬", - "flag_for_tokelau": "🇹🇰", - "flag_for_tonga": "🇹🇴", - "flag_for_trinidad_&_tobago": "🇹🇹", - "flag_for_tristan_da_cunha": "🇹🇦", - "flag_for_tunisia": "🇹🇳", - "flag_for_turkey": "🇹🇷", - "flag_for_turkmenistan": "🇹🇲", - "flag_for_turks_&_caicos_islands": "🇹🇨", - "flag_for_tuvalu": "🇹🇻", - "flag_for_u.s._outlying_islands": "🇺🇲", - "flag_for_u.s._virgin_islands": "🇻🇮", - "flag_for_uganda": "🇺🇬", - "flag_for_ukraine": "🇺🇦", - "flag_for_united_arab_emirates": "🇦🇪", - "flag_for_united_kingdom": "🇬🇧", - "flag_for_united_states": "🇺🇸", - "flag_for_uruguay": "🇺🇾", - "flag_for_uzbekistan": "🇺🇿", - "flag_for_vanuatu": "🇻🇺", - "flag_for_vatican_city": "🇻🇦", - "flag_for_venezuela": "🇻🇪", - "flag_for_vietnam": "🇻🇳", - "flag_for_wallis_&_futuna": "🇼🇫", - "flag_for_western_sahara": "🇪🇭", - "flag_for_yemen": "🇾🇪", - "flag_for_zambia": "🇿🇲", - "flag_for_zimbabwe": "🇿🇼", - "flag_for_åland_islands": "🇦🇽", - "golf": "⛳", - "fleur__de__lis": "⚜", - "muscle": "💪", - "flushed": "😳", - "frame_with_picture": "🖼", - "fries": "🍟", - "frog": "🐸", - "hatched_chick": "🐥", - "frowning": "😦", - "fuelpump": "⛽", - "full_moon_with_face": "🌝", - "gem": "💎", - "star2": "🌟", - "golfer": "🏌", - "mortar_board": "🎓", - "grimacing": "😬", - "smile_cat": "😸", - "grinning": "😀", - "grin": "😁", - "heartpulse": "💗", - "guardsman": "💂", - "haircut": "💇", - "hamster": "🐹", - "raising_hand": "🙋", - "headphones": "🎧", - "hear_no_evil": "🙉", - "cupid": "💘", - "gift_heart": "💝", - "heart": "❤", - "exclamation": "❗", - "heavy_exclamation_mark": "❗", - "heavy_heart_exclamation_mark_ornament": "❣", - "o": "⭕", - "helm_symbol": "⎈", - "helmet_with_white_cross": "⛑", - "high_heel": "👠", - "bullettrain_side": "🚄", - "bullettrain_front": "🚅", - "high_brightness": "🔆", - "zap": "⚡", - "hocho": "🔪", - "knife": "🔪", - "bee": "🐝", - "traffic_light": "🚥", - "racehorse": "🐎", - "coffee": "☕", - "hotsprings": "♨", - "hourglass": "⌛", - "hourglass_flowing_sand": "⏳", - "house_buildings": "🏘", - "100": "💯", - "hushed": "😯", - "ice_hockey_stick_and_puck": "🏒", - "imp": "👿", - "information_desk_person": "💁", - "information_source": "ℹ", - "capital_abcd": "🔠", - "abc": "🔤", - "abcd": "🔡", - "1234": "🔢", - "symbols": "🔣", - "izakaya_lantern": "🏮", - "lantern": "🏮", - "jack_o_lantern": "🎃", - "dolls": "🎎", - "japanese_goblin": "👺", - "japanese_ogre": "👹", - "beginner": "🔰", - "zero": "0️⃣", - "one": "1️⃣", - "ten": "🔟", - "two": "2️⃣", - "three": "3️⃣", - "four": "4️⃣", - "five": "5️⃣", - "six": "6️⃣", - "seven": "7️⃣", - "eight": "8️⃣", - "nine": "9️⃣", - "couplekiss": "💏", - "kissing_cat": "😽", - "kissing": "😗", - "kissing_closed_eyes": "😚", - "kissing_smiling_eyes": "😙", - "beetle": "🐞", - "large_blue_circle": "🔵", - "last_quarter_moon_with_face": "🌜", - "leaves": "🍃", - "mag": "🔍", - "left_right_arrow": "↔", - "leftwards_arrow_with_hook": "↩", - "arrow_left": "⬅", - "lock": "🔒", - "lock_with_ink_pen": "🔏", - "sob": "😭", - "low_brightness": "🔅", - "lower_left_ballpoint_pen": "🖊", - "lower_left_crayon": "🖍", - "lower_left_fountain_pen": "🖋", - "lower_left_paintbrush": "🖌", - "mahjong": "🀄", - "couple": "👫", - "man_in_business_suit_levitating": "🕴", - "man_with_gua_pi_mao": "👲", - "man_with_turban": "👳", - "mans_shoe": "👞", - "shoe": "👞", - "menorah_with_nine_branches": "🕎", - "mens": "🚹", - "minidisc": "💽", - "iphone": "📱", - "calling": "📲", - "money__mouth_face": "🤑", - "moneybag": "💰", - "rice_scene": "🎑", - "mountain_bicyclist": "🚵", - "mouse2": "🐁", - "lips": "👄", - "moyai": "🗿", - "notes": "🎶", - "nail_care": "💅", - "ab": "🆎", - "negative_squared_cross_mark": "❎", - "a": "🅰", - "b": "🅱", - "o2": "🅾", - "parking": "🅿", - "new_moon_with_face": "🌚", - "no_entry_sign": "🚫", - "underage": "🔞", - "non__potable_water": "🚱", - "arrow_upper_right": "↗", - "arrow_upper_left": "↖", - "office": "🏢", - "older_man": "👴", - "older_woman": "👵", - "om_symbol": "🕉", - "on": "🔛", - "book": "📖", - "unlock": "🔓", - "mailbox_with_no_mail": "📭", - "mailbox_with_mail": "📬", - "cd": "💿", - "tada": "🎉", - "feet": "🐾", - "walking": "🚶", - "pencil2": "✏", - "pensive": "😔", - "persevere": "😣", - "bow": "🙇", - "raised_hands": "🙌", - "person_with_ball": "⛹", - "person_with_blond_hair": "👱", - "pray": "🙏", - "person_with_pouting_face": "🙎", - "computer": "💻", - "pig2": "🐖", - "hankey": "💩", - "poop": "💩", - "shit": "💩", - "bamboo": "🎍", - "gun": "🔫", - "black_joker": "🃏", - "rotating_light": "🚨", - "cop": "👮", - "stew": "🍲", - "pouch": "👝", - "pouting_cat": "😾", - "rage": "😡", - "put_litter_in_its_place": "🚮", - "rabbit2": "🐇", - "racing_motorcycle": "🏍", - "radioactive_sign": "☢", - "fist": "✊", - "hand": "✋", - "raised_hand_with_fingers_splayed": "🖐", - "raised_hand_with_part_between_middle_and_ring_fingers": "🖖", - "blue_car": "🚙", - "apple": "🍎", - "relieved": "😌", - "reversed_hand_with_middle_finger_extended": "🖕", - "mag_right": "🔎", - "arrow_right_hook": "↪", - "sweet_potato": "🍠", - "robot": "🤖", - "rolled__up_newspaper": "🗞", - "rowboat": "🚣", - "runner": "🏃", - "running": "🏃", - "running_shirt_with_sash": "🎽", - "boat": "⛵", - "scales": "⚖", - "school_satchel": "🎒", - "scorpius": "♏", - "see_no_evil": "🙈", - "sheep": "🐑", - "stars": "🌠", - "cake": "🍰", - "six_pointed_star": "🔯", - "ski": "🎿", - "sleeping_accommodation": "🛌", - "sleeping": "😴", - "sleepy": "😪", - "sleuth_or_spy": "🕵", - "heart_eyes_cat": "😻", - "smiley_cat": "😺", - "innocent": "😇", - "heart_eyes": "😍", - "smiling_imp": "😈", - "smiley": "😃", - "sweat_smile": "😅", - "smile": "😄", - "laughing": "😆", - "satisfied": "😆", - "blush": "😊", - "smirk": "😏", - "smoking": "🚬", - "snow_capped_mountain": "🏔", - "soccer": "⚽", - "icecream": "🍦", - "soon": "🔜", - "arrow_lower_right": "↘", - "arrow_lower_left": "↙", - "speak_no_evil": "🙊", - "speaker": "🔈", - "mute": "🔇", - "sound": "🔉", - "loud_sound": "🔊", - "speaking_head_in_silhouette": "🗣", - "spiral_calendar_pad": "🗓", - "spiral_note_pad": "🗒", - "shell": "🐚", - "sweat_drops": "💦", - "u5272": "🈹", - "u5408": "🈴", - "u55b6": "🈺", - "u6307": "🈯", - "u6708": "🈷", - "u6709": "🈶", - "u6e80": "🈵", - "u7121": "🈚", - "u7533": "🈸", - "u7981": "🈲", - "u7a7a": "🈳", - "cl": "🆑", - "cool": "🆒", - "free": "🆓", - "id": "🆔", - "koko": "🈁", - "sa": "🈂", - "new": "🆕", - "ng": "🆖", - "ok": "🆗", - "sos": "🆘", - "up": "🆙", - "vs": "🆚", - "steam_locomotive": "🚂", - "ramen": "🍜", - "partly_sunny": "⛅", - "city_sunrise": "🌇", - "surfer": "🏄", - "swimmer": "🏊", - "shirt": "👕", - "tshirt": "👕", - "table_tennis_paddle_and_ball": "🏓", - "tea": "🍵", - "tv": "📺", - "three_button_mouse": "🖱", - "+1": "👍", - "thumbsup": "👍", - "__1": "👎", - "-1": "👎", - "thumbsdown": "👎", - "thunder_cloud_and_rain": "⛈", - "tiger2": "🐅", - "tophat": "🎩", - "top": "🔝", - "tm": "™", - "train2": "🚆", - "triangular_flag_on_post": "🚩", - "trident": "🔱", - "twisted_rightwards_arrows": "🔀", - "unamused": "😒", - "small_red_triangle": "🔺", - "arrow_up_small": "🔼", - "arrow_up_down": "↕", - "upside__down_face": "🙃", - "arrow_up": "⬆", - "v": "✌", - "vhs": "📼", - "wc": "🚾", - "ocean": "🌊", - "waving_black_flag": "🏴", - "wave": "👋", - "waving_white_flag": "🏳", - "moon": "🌔", - "scream_cat": "🙀", - "weary": "😩", - "weight_lifter": "🏋", - "whale2": "🐋", - "wheelchair": "♿", - "point_down": "👇", - "grey_exclamation": "❕", - "white_frowning_face": "☹", - "white_check_mark": "✅", - "point_left": "👈", - "white_medium_small_square": "◽", - "star": "⭐", - "grey_question": "❔", - "point_right": "👉", - "relaxed": "☺", - "white_sun_behind_cloud": "🌥", - "white_sun_behind_cloud_with_rain": "🌦", - "white_sun_with_small_cloud": "🌤", - "point_up_2": "👆", - "point_up": "☝", - "wind_blowing_face": "🌬", - "wink": "😉", - "wolf": "🐺", - "dancers": "👯", - "boot": "👢", - "womans_clothes": "👚", - "womans_hat": "👒", - "sandal": "👡", - "womens": "🚺", - "worried": "😟", - "gift": "🎁", - "zipper__mouth_face": "🤐", - "regional_indicator_a": "🇦", - "regional_indicator_b": "🇧", - "regional_indicator_c": "🇨", - "regional_indicator_d": "🇩", - "regional_indicator_e": "🇪", - "regional_indicator_f": "🇫", - "regional_indicator_g": "🇬", - "regional_indicator_h": "🇭", - "regional_indicator_i": "🇮", - "regional_indicator_j": "🇯", - "regional_indicator_k": "🇰", - "regional_indicator_l": "🇱", - "regional_indicator_m": "🇲", - "regional_indicator_n": "🇳", - "regional_indicator_o": "🇴", - "regional_indicator_p": "🇵", - "regional_indicator_q": "🇶", - "regional_indicator_r": "🇷", - "regional_indicator_s": "🇸", - "regional_indicator_t": "🇹", - "regional_indicator_u": "🇺", - "regional_indicator_v": "🇻", - "regional_indicator_w": "🇼", - "regional_indicator_x": "🇽", - "regional_indicator_y": "🇾", - "regional_indicator_z": "🇿", -} diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/rich/_emoji_replace.py b/venv/lib/python3.11/site-packages/pip/_vendor/rich/_emoji_replace.py deleted file mode 100644 index bb2cafa..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/rich/_emoji_replace.py +++ /dev/null @@ -1,32 +0,0 @@ -from typing import Callable, Match, Optional -import re - -from ._emoji_codes import EMOJI - - -_ReStringMatch = Match[str] # regex match object -_ReSubCallable = Callable[[_ReStringMatch], str] # Callable invoked by re.sub -_EmojiSubMethod = Callable[[_ReSubCallable, str], str] # Sub method of a compiled re - - -def _emoji_replace( - text: str, - default_variant: Optional[str] = None, - _emoji_sub: _EmojiSubMethod = re.compile(r"(:(\S*?)(?:(?:\-)(emoji|text))?:)").sub, -) -> str: - """Replace emoji code in text.""" - get_emoji = EMOJI.__getitem__ - variants = {"text": "\uFE0E", "emoji": "\uFE0F"} - get_variant = variants.get - default_variant_code = variants.get(default_variant, "") if default_variant else "" - - def do_replace(match: Match[str]) -> str: - emoji_code, emoji_name, variant = match.groups() - try: - return get_emoji(emoji_name.lower()) + get_variant( - variant, default_variant_code - ) - except KeyError: - return emoji_code - - return _emoji_sub(do_replace, text) diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/rich/_export_format.py b/venv/lib/python3.11/site-packages/pip/_vendor/rich/_export_format.py deleted file mode 100644 index 094d2dc..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/rich/_export_format.py +++ /dev/null @@ -1,76 +0,0 @@ -CONSOLE_HTML_FORMAT = """\ - - - - - - - - 
{code}
- - -""" - -CONSOLE_SVG_FORMAT = """\ - - - - - - - - - {lines} - - - {chrome} - - {backgrounds} - - {matrix} - - - -""" - -_SVG_FONT_FAMILY = "Rich Fira Code" -_SVG_CLASSES_PREFIX = "rich-svg" diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/rich/_extension.py b/venv/lib/python3.11/site-packages/pip/_vendor/rich/_extension.py deleted file mode 100644 index cbd6da9..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/rich/_extension.py +++ /dev/null @@ -1,10 +0,0 @@ -from typing import Any - - -def load_ipython_extension(ip: Any) -> None: # pragma: no cover - # prevent circular import - from pip._vendor.rich.pretty import install - from pip._vendor.rich.traceback import install as tr_install - - install() - tr_install() diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/rich/_fileno.py b/venv/lib/python3.11/site-packages/pip/_vendor/rich/_fileno.py deleted file mode 100644 index b17ee65..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/rich/_fileno.py +++ /dev/null @@ -1,24 +0,0 @@ -from __future__ import annotations - -from typing import IO, Callable - - -def get_fileno(file_like: IO[str]) -> int | None: - """Get fileno() from a file, accounting for poorly implemented file-like objects. - - Args: - file_like (IO): A file-like object. - - Returns: - int | None: The result of fileno if available, or None if operation failed. - """ - fileno: Callable[[], int] | None = getattr(file_like, "fileno", None) - if fileno is not None: - try: - return fileno() - except Exception: - # `fileno` is documented as potentially raising a OSError - # Alas, from the issues, there are so many poorly implemented file-like objects, - # that `fileno()` can raise just about anything. - return None - return None diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/rich/_inspect.py b/venv/lib/python3.11/site-packages/pip/_vendor/rich/_inspect.py deleted file mode 100644 index 30446ce..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/rich/_inspect.py +++ /dev/null @@ -1,270 +0,0 @@ -from __future__ import absolute_import - -import inspect -from inspect import cleandoc, getdoc, getfile, isclass, ismodule, signature -from typing import Any, Collection, Iterable, Optional, Tuple, Type, Union - -from .console import Group, RenderableType -from .control import escape_control_codes -from .highlighter import ReprHighlighter -from .jupyter import JupyterMixin -from .panel import Panel -from .pretty import Pretty -from .table import Table -from .text import Text, TextType - - -def _first_paragraph(doc: str) -> str: - """Get the first paragraph from a docstring.""" - paragraph, _, _ = doc.partition("\n\n") - return paragraph - - -class Inspect(JupyterMixin): - """A renderable to inspect any Python Object. - - Args: - obj (Any): An object to inspect. - title (str, optional): Title to display over inspect result, or None use type. Defaults to None. - help (bool, optional): Show full help text rather than just first paragraph. Defaults to False. - methods (bool, optional): Enable inspection of callables. Defaults to False. - docs (bool, optional): Also render doc strings. Defaults to True. - private (bool, optional): Show private attributes (beginning with underscore). Defaults to False. - dunder (bool, optional): Show attributes starting with double underscore. Defaults to False. - sort (bool, optional): Sort attributes alphabetically. Defaults to True. - all (bool, optional): Show all attributes. Defaults to False. - value (bool, optional): Pretty print value of object. Defaults to True. - """ - - def __init__( - self, - obj: Any, - *, - title: Optional[TextType] = None, - help: bool = False, - methods: bool = False, - docs: bool = True, - private: bool = False, - dunder: bool = False, - sort: bool = True, - all: bool = True, - value: bool = True, - ) -> None: - self.highlighter = ReprHighlighter() - self.obj = obj - self.title = title or self._make_title(obj) - if all: - methods = private = dunder = True - self.help = help - self.methods = methods - self.docs = docs or help - self.private = private or dunder - self.dunder = dunder - self.sort = sort - self.value = value - - def _make_title(self, obj: Any) -> Text: - """Make a default title.""" - title_str = ( - str(obj) - if (isclass(obj) or callable(obj) or ismodule(obj)) - else str(type(obj)) - ) - title_text = self.highlighter(title_str) - return title_text - - def __rich__(self) -> Panel: - return Panel.fit( - Group(*self._render()), - title=self.title, - border_style="scope.border", - padding=(0, 1), - ) - - def _get_signature(self, name: str, obj: Any) -> Optional[Text]: - """Get a signature for a callable.""" - try: - _signature = str(signature(obj)) + ":" - except ValueError: - _signature = "(...)" - except TypeError: - return None - - source_filename: Optional[str] = None - try: - source_filename = getfile(obj) - except (OSError, TypeError): - # OSError is raised if obj has no source file, e.g. when defined in REPL. - pass - - callable_name = Text(name, style="inspect.callable") - if source_filename: - callable_name.stylize(f"link file://{source_filename}") - signature_text = self.highlighter(_signature) - - qualname = name or getattr(obj, "__qualname__", name) - - # If obj is a module, there may be classes (which are callable) to display - if inspect.isclass(obj): - prefix = "class" - elif inspect.iscoroutinefunction(obj): - prefix = "async def" - else: - prefix = "def" - - qual_signature = Text.assemble( - (f"{prefix} ", f"inspect.{prefix.replace(' ', '_')}"), - (qualname, "inspect.callable"), - signature_text, - ) - - return qual_signature - - def _render(self) -> Iterable[RenderableType]: - """Render object.""" - - def sort_items(item: Tuple[str, Any]) -> Tuple[bool, str]: - key, (_error, value) = item - return (callable(value), key.strip("_").lower()) - - def safe_getattr(attr_name: str) -> Tuple[Any, Any]: - """Get attribute or any exception.""" - try: - return (None, getattr(obj, attr_name)) - except Exception as error: - return (error, None) - - obj = self.obj - keys = dir(obj) - total_items = len(keys) - if not self.dunder: - keys = [key for key in keys if not key.startswith("__")] - if not self.private: - keys = [key for key in keys if not key.startswith("_")] - not_shown_count = total_items - len(keys) - items = [(key, safe_getattr(key)) for key in keys] - if self.sort: - items.sort(key=sort_items) - - items_table = Table.grid(padding=(0, 1), expand=False) - items_table.add_column(justify="right") - add_row = items_table.add_row - highlighter = self.highlighter - - if callable(obj): - signature = self._get_signature("", obj) - if signature is not None: - yield signature - yield "" - - if self.docs: - _doc = self._get_formatted_doc(obj) - if _doc is not None: - doc_text = Text(_doc, style="inspect.help") - doc_text = highlighter(doc_text) - yield doc_text - yield "" - - if self.value and not (isclass(obj) or callable(obj) or ismodule(obj)): - yield Panel( - Pretty(obj, indent_guides=True, max_length=10, max_string=60), - border_style="inspect.value.border", - ) - yield "" - - for key, (error, value) in items: - key_text = Text.assemble( - ( - key, - "inspect.attr.dunder" if key.startswith("__") else "inspect.attr", - ), - (" =", "inspect.equals"), - ) - if error is not None: - warning = key_text.copy() - warning.stylize("inspect.error") - add_row(warning, highlighter(repr(error))) - continue - - if callable(value): - if not self.methods: - continue - - _signature_text = self._get_signature(key, value) - if _signature_text is None: - add_row(key_text, Pretty(value, highlighter=highlighter)) - else: - if self.docs: - docs = self._get_formatted_doc(value) - if docs is not None: - _signature_text.append("\n" if "\n" in docs else " ") - doc = highlighter(docs) - doc.stylize("inspect.doc") - _signature_text.append(doc) - - add_row(key_text, _signature_text) - else: - add_row(key_text, Pretty(value, highlighter=highlighter)) - if items_table.row_count: - yield items_table - elif not_shown_count: - yield Text.from_markup( - f"[b cyan]{not_shown_count}[/][i] attribute(s) not shown.[/i] " - f"Run [b][magenta]inspect[/]([not b]inspect[/])[/b] for options." - ) - - def _get_formatted_doc(self, object_: Any) -> Optional[str]: - """ - Extract the docstring of an object, process it and returns it. - The processing consists in cleaning up the doctring's indentation, - taking only its 1st paragraph if `self.help` is not True, - and escape its control codes. - - Args: - object_ (Any): the object to get the docstring from. - - Returns: - Optional[str]: the processed docstring, or None if no docstring was found. - """ - docs = getdoc(object_) - if docs is None: - return None - docs = cleandoc(docs).strip() - if not self.help: - docs = _first_paragraph(docs) - return escape_control_codes(docs) - - -def get_object_types_mro(obj: Union[object, Type[Any]]) -> Tuple[type, ...]: - """Returns the MRO of an object's class, or of the object itself if it's a class.""" - if not hasattr(obj, "__mro__"): - # N.B. we cannot use `if type(obj) is type` here because it doesn't work with - # some types of classes, such as the ones that use abc.ABCMeta. - obj = type(obj) - return getattr(obj, "__mro__", ()) - - -def get_object_types_mro_as_strings(obj: object) -> Collection[str]: - """ - Returns the MRO of an object's class as full qualified names, or of the object itself if it's a class. - - Examples: - `object_types_mro_as_strings(JSONDecoder)` will return `['json.decoder.JSONDecoder', 'builtins.object']` - """ - return [ - f'{getattr(type_, "__module__", "")}.{getattr(type_, "__qualname__", "")}' - for type_ in get_object_types_mro(obj) - ] - - -def is_object_one_of_types( - obj: object, fully_qualified_types_names: Collection[str] -) -> bool: - """ - Returns `True` if the given object's class (or the object itself, if it's a class) has one of the - fully qualified names in its MRO. - """ - for type_name in get_object_types_mro_as_strings(obj): - if type_name in fully_qualified_types_names: - return True - return False diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/rich/_log_render.py b/venv/lib/python3.11/site-packages/pip/_vendor/rich/_log_render.py deleted file mode 100644 index fc16c84..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/rich/_log_render.py +++ /dev/null @@ -1,94 +0,0 @@ -from datetime import datetime -from typing import Iterable, List, Optional, TYPE_CHECKING, Union, Callable - - -from .text import Text, TextType - -if TYPE_CHECKING: - from .console import Console, ConsoleRenderable, RenderableType - from .table import Table - -FormatTimeCallable = Callable[[datetime], Text] - - -class LogRender: - def __init__( - self, - show_time: bool = True, - show_level: bool = False, - show_path: bool = True, - time_format: Union[str, FormatTimeCallable] = "[%x %X]", - omit_repeated_times: bool = True, - level_width: Optional[int] = 8, - ) -> None: - self.show_time = show_time - self.show_level = show_level - self.show_path = show_path - self.time_format = time_format - self.omit_repeated_times = omit_repeated_times - self.level_width = level_width - self._last_time: Optional[Text] = None - - def __call__( - self, - console: "Console", - renderables: Iterable["ConsoleRenderable"], - log_time: Optional[datetime] = None, - time_format: Optional[Union[str, FormatTimeCallable]] = None, - level: TextType = "", - path: Optional[str] = None, - line_no: Optional[int] = None, - link_path: Optional[str] = None, - ) -> "Table": - from .containers import Renderables - from .table import Table - - output = Table.grid(padding=(0, 1)) - output.expand = True - if self.show_time: - output.add_column(style="log.time") - if self.show_level: - output.add_column(style="log.level", width=self.level_width) - output.add_column(ratio=1, style="log.message", overflow="fold") - if self.show_path and path: - output.add_column(style="log.path") - row: List["RenderableType"] = [] - if self.show_time: - log_time = log_time or console.get_datetime() - time_format = time_format or self.time_format - if callable(time_format): - log_time_display = time_format(log_time) - else: - log_time_display = Text(log_time.strftime(time_format)) - if log_time_display == self._last_time and self.omit_repeated_times: - row.append(Text(" " * len(log_time_display))) - else: - row.append(log_time_display) - self._last_time = log_time_display - if self.show_level: - row.append(level) - - row.append(Renderables(renderables)) - if self.show_path and path: - path_text = Text() - path_text.append( - path, style=f"link file://{link_path}" if link_path else "" - ) - if line_no: - path_text.append(":") - path_text.append( - f"{line_no}", - style=f"link file://{link_path}#{line_no}" if link_path else "", - ) - row.append(path_text) - - output.add_row(*row) - return output - - -if __name__ == "__main__": # pragma: no cover - from pip._vendor.rich.console import Console - - c = Console() - c.print("[on blue]Hello", justify="right") - c.log("[on blue]hello", justify="right") diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/rich/_loop.py b/venv/lib/python3.11/site-packages/pip/_vendor/rich/_loop.py deleted file mode 100644 index 01c6caf..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/rich/_loop.py +++ /dev/null @@ -1,43 +0,0 @@ -from typing import Iterable, Tuple, TypeVar - -T = TypeVar("T") - - -def loop_first(values: Iterable[T]) -> Iterable[Tuple[bool, T]]: - """Iterate and generate a tuple with a flag for first value.""" - iter_values = iter(values) - try: - value = next(iter_values) - except StopIteration: - return - yield True, value - for value in iter_values: - yield False, value - - -def loop_last(values: Iterable[T]) -> Iterable[Tuple[bool, T]]: - """Iterate and generate a tuple with a flag for last value.""" - iter_values = iter(values) - try: - previous_value = next(iter_values) - except StopIteration: - return - for value in iter_values: - yield False, previous_value - previous_value = value - yield True, previous_value - - -def loop_first_last(values: Iterable[T]) -> Iterable[Tuple[bool, bool, T]]: - """Iterate and generate a tuple with a flag for first and last value.""" - iter_values = iter(values) - try: - previous_value = next(iter_values) - except StopIteration: - return - first = True - for value in iter_values: - yield first, False, previous_value - first = False - previous_value = value - yield first, True, previous_value diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/rich/_null_file.py b/venv/lib/python3.11/site-packages/pip/_vendor/rich/_null_file.py deleted file mode 100644 index b659673..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/rich/_null_file.py +++ /dev/null @@ -1,69 +0,0 @@ -from types import TracebackType -from typing import IO, Iterable, Iterator, List, Optional, Type - - -class NullFile(IO[str]): - def close(self) -> None: - pass - - def isatty(self) -> bool: - return False - - def read(self, __n: int = 1) -> str: - return "" - - def readable(self) -> bool: - return False - - def readline(self, __limit: int = 1) -> str: - return "" - - def readlines(self, __hint: int = 1) -> List[str]: - return [] - - def seek(self, __offset: int, __whence: int = 1) -> int: - return 0 - - def seekable(self) -> bool: - return False - - def tell(self) -> int: - return 0 - - def truncate(self, __size: Optional[int] = 1) -> int: - return 0 - - def writable(self) -> bool: - return False - - def writelines(self, __lines: Iterable[str]) -> None: - pass - - def __next__(self) -> str: - return "" - - def __iter__(self) -> Iterator[str]: - return iter([""]) - - def __enter__(self) -> IO[str]: - pass - - def __exit__( - self, - __t: Optional[Type[BaseException]], - __value: Optional[BaseException], - __traceback: Optional[TracebackType], - ) -> None: - pass - - def write(self, text: str) -> int: - return 0 - - def flush(self) -> None: - pass - - def fileno(self) -> int: - return -1 - - -NULL_FILE = NullFile() diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/rich/_palettes.py b/venv/lib/python3.11/site-packages/pip/_vendor/rich/_palettes.py deleted file mode 100644 index 3c748d3..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/rich/_palettes.py +++ /dev/null @@ -1,309 +0,0 @@ -from .palette import Palette - - -# Taken from https://en.wikipedia.org/wiki/ANSI_escape_code (Windows 10 column) -WINDOWS_PALETTE = Palette( - [ - (12, 12, 12), - (197, 15, 31), - (19, 161, 14), - (193, 156, 0), - (0, 55, 218), - (136, 23, 152), - (58, 150, 221), - (204, 204, 204), - (118, 118, 118), - (231, 72, 86), - (22, 198, 12), - (249, 241, 165), - (59, 120, 255), - (180, 0, 158), - (97, 214, 214), - (242, 242, 242), - ] -) - -# # The standard ansi colors (including bright variants) -STANDARD_PALETTE = Palette( - [ - (0, 0, 0), - (170, 0, 0), - (0, 170, 0), - (170, 85, 0), - (0, 0, 170), - (170, 0, 170), - (0, 170, 170), - (170, 170, 170), - (85, 85, 85), - (255, 85, 85), - (85, 255, 85), - (255, 255, 85), - (85, 85, 255), - (255, 85, 255), - (85, 255, 255), - (255, 255, 255), - ] -) - - -# The 256 color palette -EIGHT_BIT_PALETTE = Palette( - [ - (0, 0, 0), - (128, 0, 0), - (0, 128, 0), - (128, 128, 0), - (0, 0, 128), - (128, 0, 128), - (0, 128, 128), - (192, 192, 192), - (128, 128, 128), - (255, 0, 0), - (0, 255, 0), - (255, 255, 0), - (0, 0, 255), - (255, 0, 255), - (0, 255, 255), - (255, 255, 255), - (0, 0, 0), - (0, 0, 95), - (0, 0, 135), - (0, 0, 175), - (0, 0, 215), - (0, 0, 255), - (0, 95, 0), - (0, 95, 95), - (0, 95, 135), - (0, 95, 175), - (0, 95, 215), - (0, 95, 255), - (0, 135, 0), - (0, 135, 95), - (0, 135, 135), - (0, 135, 175), - (0, 135, 215), - (0, 135, 255), - (0, 175, 0), - (0, 175, 95), - (0, 175, 135), - (0, 175, 175), - (0, 175, 215), - (0, 175, 255), - (0, 215, 0), - (0, 215, 95), - (0, 215, 135), - (0, 215, 175), - (0, 215, 215), - (0, 215, 255), - (0, 255, 0), - (0, 255, 95), - (0, 255, 135), - (0, 255, 175), - (0, 255, 215), - (0, 255, 255), - (95, 0, 0), - (95, 0, 95), - (95, 0, 135), - (95, 0, 175), - (95, 0, 215), - (95, 0, 255), - (95, 95, 0), - (95, 95, 95), - (95, 95, 135), - (95, 95, 175), - (95, 95, 215), - (95, 95, 255), - (95, 135, 0), - (95, 135, 95), - (95, 135, 135), - (95, 135, 175), - (95, 135, 215), - (95, 135, 255), - (95, 175, 0), - (95, 175, 95), - (95, 175, 135), - (95, 175, 175), - (95, 175, 215), - (95, 175, 255), - (95, 215, 0), - (95, 215, 95), - (95, 215, 135), - (95, 215, 175), - (95, 215, 215), - (95, 215, 255), - (95, 255, 0), - (95, 255, 95), - (95, 255, 135), - (95, 255, 175), - (95, 255, 215), - (95, 255, 255), - (135, 0, 0), - (135, 0, 95), - (135, 0, 135), - (135, 0, 175), - (135, 0, 215), - (135, 0, 255), - (135, 95, 0), - (135, 95, 95), - (135, 95, 135), - (135, 95, 175), - (135, 95, 215), - (135, 95, 255), - (135, 135, 0), - (135, 135, 95), - (135, 135, 135), - (135, 135, 175), - (135, 135, 215), - (135, 135, 255), - (135, 175, 0), - (135, 175, 95), - (135, 175, 135), - (135, 175, 175), - (135, 175, 215), - (135, 175, 255), - (135, 215, 0), - (135, 215, 95), - (135, 215, 135), - (135, 215, 175), - (135, 215, 215), - (135, 215, 255), - (135, 255, 0), - (135, 255, 95), - (135, 255, 135), - (135, 255, 175), - (135, 255, 215), - (135, 255, 255), - (175, 0, 0), - (175, 0, 95), - (175, 0, 135), - (175, 0, 175), - (175, 0, 215), - (175, 0, 255), - (175, 95, 0), - (175, 95, 95), - (175, 95, 135), - (175, 95, 175), - (175, 95, 215), - (175, 95, 255), - (175, 135, 0), - (175, 135, 95), - (175, 135, 135), - (175, 135, 175), - (175, 135, 215), - (175, 135, 255), - (175, 175, 0), - (175, 175, 95), - (175, 175, 135), - (175, 175, 175), - (175, 175, 215), - (175, 175, 255), - (175, 215, 0), - (175, 215, 95), - (175, 215, 135), - (175, 215, 175), - (175, 215, 215), - (175, 215, 255), - (175, 255, 0), - (175, 255, 95), - (175, 255, 135), - (175, 255, 175), - (175, 255, 215), - (175, 255, 255), - (215, 0, 0), - (215, 0, 95), - (215, 0, 135), - (215, 0, 175), - (215, 0, 215), - (215, 0, 255), - (215, 95, 0), - (215, 95, 95), - (215, 95, 135), - (215, 95, 175), - (215, 95, 215), - (215, 95, 255), - (215, 135, 0), - (215, 135, 95), - (215, 135, 135), - (215, 135, 175), - (215, 135, 215), - (215, 135, 255), - (215, 175, 0), - (215, 175, 95), - (215, 175, 135), - (215, 175, 175), - (215, 175, 215), - (215, 175, 255), - (215, 215, 0), - (215, 215, 95), - (215, 215, 135), - (215, 215, 175), - (215, 215, 215), - (215, 215, 255), - (215, 255, 0), - (215, 255, 95), - (215, 255, 135), - (215, 255, 175), - (215, 255, 215), - (215, 255, 255), - (255, 0, 0), - (255, 0, 95), - (255, 0, 135), - (255, 0, 175), - (255, 0, 215), - (255, 0, 255), - (255, 95, 0), - (255, 95, 95), - (255, 95, 135), - (255, 95, 175), - (255, 95, 215), - (255, 95, 255), - (255, 135, 0), - (255, 135, 95), - (255, 135, 135), - (255, 135, 175), - (255, 135, 215), - (255, 135, 255), - (255, 175, 0), - (255, 175, 95), - (255, 175, 135), - (255, 175, 175), - (255, 175, 215), - (255, 175, 255), - (255, 215, 0), - (255, 215, 95), - (255, 215, 135), - (255, 215, 175), - (255, 215, 215), - (255, 215, 255), - (255, 255, 0), - (255, 255, 95), - (255, 255, 135), - (255, 255, 175), - (255, 255, 215), - (255, 255, 255), - (8, 8, 8), - (18, 18, 18), - (28, 28, 28), - (38, 38, 38), - (48, 48, 48), - (58, 58, 58), - (68, 68, 68), - (78, 78, 78), - (88, 88, 88), - (98, 98, 98), - (108, 108, 108), - (118, 118, 118), - (128, 128, 128), - (138, 138, 138), - (148, 148, 148), - (158, 158, 158), - (168, 168, 168), - (178, 178, 178), - (188, 188, 188), - (198, 198, 198), - (208, 208, 208), - (218, 218, 218), - (228, 228, 228), - (238, 238, 238), - ] -) diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/rich/_pick.py b/venv/lib/python3.11/site-packages/pip/_vendor/rich/_pick.py deleted file mode 100644 index 4f6d8b2..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/rich/_pick.py +++ /dev/null @@ -1,17 +0,0 @@ -from typing import Optional - - -def pick_bool(*values: Optional[bool]) -> bool: - """Pick the first non-none bool or return the last value. - - Args: - *values (bool): Any number of boolean or None values. - - Returns: - bool: First non-none boolean. - """ - assert values, "1 or more values required" - for value in values: - if value is not None: - return value - return bool(value) diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/rich/_ratio.py b/venv/lib/python3.11/site-packages/pip/_vendor/rich/_ratio.py deleted file mode 100644 index e8a3a67..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/rich/_ratio.py +++ /dev/null @@ -1,160 +0,0 @@ -import sys -from fractions import Fraction -from math import ceil -from typing import cast, List, Optional, Sequence - -if sys.version_info >= (3, 8): - from typing import Protocol -else: - from pip._vendor.typing_extensions import Protocol # pragma: no cover - - -class Edge(Protocol): - """Any object that defines an edge (such as Layout).""" - - size: Optional[int] = None - ratio: int = 1 - minimum_size: int = 1 - - -def ratio_resolve(total: int, edges: Sequence[Edge]) -> List[int]: - """Divide total space to satisfy size, ratio, and minimum_size, constraints. - - The returned list of integers should add up to total in most cases, unless it is - impossible to satisfy all the constraints. For instance, if there are two edges - with a minimum size of 20 each and `total` is 30 then the returned list will be - greater than total. In practice, this would mean that a Layout object would - clip the rows that would overflow the screen height. - - Args: - total (int): Total number of characters. - edges (List[Edge]): Edges within total space. - - Returns: - List[int]: Number of characters for each edge. - """ - # Size of edge or None for yet to be determined - sizes = [(edge.size or None) for edge in edges] - - _Fraction = Fraction - - # While any edges haven't been calculated - while None in sizes: - # Get flexible edges and index to map these back on to sizes list - flexible_edges = [ - (index, edge) - for index, (size, edge) in enumerate(zip(sizes, edges)) - if size is None - ] - # Remaining space in total - remaining = total - sum(size or 0 for size in sizes) - if remaining <= 0: - # No room for flexible edges - return [ - ((edge.minimum_size or 1) if size is None else size) - for size, edge in zip(sizes, edges) - ] - # Calculate number of characters in a ratio portion - portion = _Fraction( - remaining, sum((edge.ratio or 1) for _, edge in flexible_edges) - ) - - # If any edges will be less than their minimum, replace size with the minimum - for index, edge in flexible_edges: - if portion * edge.ratio <= edge.minimum_size: - sizes[index] = edge.minimum_size - # New fixed size will invalidate calculations, so we need to repeat the process - break - else: - # Distribute flexible space and compensate for rounding error - # Since edge sizes can only be integers we need to add the remainder - # to the following line - remainder = _Fraction(0) - for index, edge in flexible_edges: - size, remainder = divmod(portion * edge.ratio + remainder, 1) - sizes[index] = size - break - # Sizes now contains integers only - return cast(List[int], sizes) - - -def ratio_reduce( - total: int, ratios: List[int], maximums: List[int], values: List[int] -) -> List[int]: - """Divide an integer total in to parts based on ratios. - - Args: - total (int): The total to divide. - ratios (List[int]): A list of integer ratios. - maximums (List[int]): List of maximums values for each slot. - values (List[int]): List of values - - Returns: - List[int]: A list of integers guaranteed to sum to total. - """ - ratios = [ratio if _max else 0 for ratio, _max in zip(ratios, maximums)] - total_ratio = sum(ratios) - if not total_ratio: - return values[:] - total_remaining = total - result: List[int] = [] - append = result.append - for ratio, maximum, value in zip(ratios, maximums, values): - if ratio and total_ratio > 0: - distributed = min(maximum, round(ratio * total_remaining / total_ratio)) - append(value - distributed) - total_remaining -= distributed - total_ratio -= ratio - else: - append(value) - return result - - -def ratio_distribute( - total: int, ratios: List[int], minimums: Optional[List[int]] = None -) -> List[int]: - """Distribute an integer total in to parts based on ratios. - - Args: - total (int): The total to divide. - ratios (List[int]): A list of integer ratios. - minimums (List[int]): List of minimum values for each slot. - - Returns: - List[int]: A list of integers guaranteed to sum to total. - """ - if minimums: - ratios = [ratio if _min else 0 for ratio, _min in zip(ratios, minimums)] - total_ratio = sum(ratios) - assert total_ratio > 0, "Sum of ratios must be > 0" - - total_remaining = total - distributed_total: List[int] = [] - append = distributed_total.append - if minimums is None: - _minimums = [0] * len(ratios) - else: - _minimums = minimums - for ratio, minimum in zip(ratios, _minimums): - if total_ratio > 0: - distributed = max(minimum, ceil(ratio * total_remaining / total_ratio)) - else: - distributed = total_remaining - append(distributed) - total_ratio -= ratio - total_remaining -= distributed - return distributed_total - - -if __name__ == "__main__": - from dataclasses import dataclass - - @dataclass - class E: - - size: Optional[int] = None - ratio: int = 1 - minimum_size: int = 1 - - resolved = ratio_resolve(110, [E(None, 1, 1), E(None, 1, 1), E(None, 1, 1)]) - print(sum(resolved)) diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/rich/_spinners.py b/venv/lib/python3.11/site-packages/pip/_vendor/rich/_spinners.py deleted file mode 100644 index d0bb1fe..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/rich/_spinners.py +++ /dev/null @@ -1,482 +0,0 @@ -""" -Spinners are from: -* cli-spinners: - MIT License - Copyright (c) Sindre Sorhus (sindresorhus.com) - Permission is hereby granted, free of charge, to any person obtaining a copy - of this software and associated documentation files (the "Software"), to deal - in the Software without restriction, including without limitation the rights to - use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of - the Software, and to permit persons to whom the Software is furnished to do so, - subject to the following conditions: - The above copyright notice and this permission notice shall be included - in all copies or substantial portions of the Software. - THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, - INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR - PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE - FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, - ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS - IN THE SOFTWARE. -""" - -SPINNERS = { - "dots": { - "interval": 80, - "frames": "⠋⠙⠹⠸⠼⠴⠦⠧⠇⠏", - }, - "dots2": {"interval": 80, "frames": "⣾⣽⣻⢿⡿⣟⣯⣷"}, - "dots3": { - "interval": 80, - "frames": "⠋⠙⠚⠞⠖⠦⠴⠲⠳⠓", - }, - "dots4": { - "interval": 80, - "frames": "⠄⠆⠇⠋⠙⠸⠰⠠⠰⠸⠙⠋⠇⠆", - }, - "dots5": { - "interval": 80, - "frames": "⠋⠙⠚⠒⠂⠂⠒⠲⠴⠦⠖⠒⠐⠐⠒⠓⠋", - }, - "dots6": { - "interval": 80, - "frames": "⠁⠉⠙⠚⠒⠂⠂⠒⠲⠴⠤⠄⠄⠤⠴⠲⠒⠂⠂⠒⠚⠙⠉⠁", - }, - "dots7": { - "interval": 80, - "frames": "⠈⠉⠋⠓⠒⠐⠐⠒⠖⠦⠤⠠⠠⠤⠦⠖⠒⠐⠐⠒⠓⠋⠉⠈", - }, - "dots8": { - "interval": 80, - "frames": "⠁⠁⠉⠙⠚⠒⠂⠂⠒⠲⠴⠤⠄⠄⠤⠠⠠⠤⠦⠖⠒⠐⠐⠒⠓⠋⠉⠈⠈", - }, - "dots9": {"interval": 80, "frames": "⢹⢺⢼⣸⣇⡧⡗⡏"}, - "dots10": {"interval": 80, "frames": "⢄⢂⢁⡁⡈⡐⡠"}, - "dots11": {"interval": 100, "frames": "⠁⠂⠄⡀⢀⠠⠐⠈"}, - "dots12": { - "interval": 80, - "frames": [ - "⢀⠀", - "⡀⠀", - "⠄⠀", - "⢂⠀", - "⡂⠀", - "⠅⠀", - "⢃⠀", - "⡃⠀", - "⠍⠀", - "⢋⠀", - "⡋⠀", - "⠍⠁", - "⢋⠁", - "⡋⠁", - "⠍⠉", - "⠋⠉", - "⠋⠉", - "⠉⠙", - "⠉⠙", - "⠉⠩", - "⠈⢙", - "⠈⡙", - "⢈⠩", - "⡀⢙", - "⠄⡙", - "⢂⠩", - "⡂⢘", - "⠅⡘", - "⢃⠨", - "⡃⢐", - "⠍⡐", - "⢋⠠", - "⡋⢀", - "⠍⡁", - "⢋⠁", - "⡋⠁", - "⠍⠉", - "⠋⠉", - "⠋⠉", - "⠉⠙", - "⠉⠙", - "⠉⠩", - "⠈⢙", - "⠈⡙", - "⠈⠩", - "⠀⢙", - "⠀⡙", - "⠀⠩", - "⠀⢘", - "⠀⡘", - "⠀⠨", - "⠀⢐", - "⠀⡐", - "⠀⠠", - "⠀⢀", - "⠀⡀", - ], - }, - "dots8Bit": { - "interval": 80, - "frames": "⠀⠁⠂⠃⠄⠅⠆⠇⡀⡁⡂⡃⡄⡅⡆⡇⠈⠉⠊⠋⠌⠍⠎⠏⡈⡉⡊⡋⡌⡍⡎⡏⠐⠑⠒⠓⠔⠕⠖⠗⡐⡑⡒⡓⡔⡕⡖⡗⠘⠙⠚⠛⠜⠝⠞⠟⡘⡙" - "⡚⡛⡜⡝⡞⡟⠠⠡⠢⠣⠤⠥⠦⠧⡠⡡⡢⡣⡤⡥⡦⡧⠨⠩⠪⠫⠬⠭⠮⠯⡨⡩⡪⡫⡬⡭⡮⡯⠰⠱⠲⠳⠴⠵⠶⠷⡰⡱⡲⡳⡴⡵⡶⡷⠸⠹⠺⠻" - "⠼⠽⠾⠿⡸⡹⡺⡻⡼⡽⡾⡿⢀⢁⢂⢃⢄⢅⢆⢇⣀⣁⣂⣃⣄⣅⣆⣇⢈⢉⢊⢋⢌⢍⢎⢏⣈⣉⣊⣋⣌⣍⣎⣏⢐⢑⢒⢓⢔⢕⢖⢗⣐⣑⣒⣓⣔⣕" - "⣖⣗⢘⢙⢚⢛⢜⢝⢞⢟⣘⣙⣚⣛⣜⣝⣞⣟⢠⢡⢢⢣⢤⢥⢦⢧⣠⣡⣢⣣⣤⣥⣦⣧⢨⢩⢪⢫⢬⢭⢮⢯⣨⣩⣪⣫⣬⣭⣮⣯⢰⢱⢲⢳⢴⢵⢶⢷" - "⣰⣱⣲⣳⣴⣵⣶⣷⢸⢹⢺⢻⢼⢽⢾⢿⣸⣹⣺⣻⣼⣽⣾⣿", - }, - "line": {"interval": 130, "frames": ["-", "\\", "|", "/"]}, - "line2": {"interval": 100, "frames": "⠂-–—–-"}, - "pipe": {"interval": 100, "frames": "┤┘┴└├┌┬┐"}, - "simpleDots": {"interval": 400, "frames": [". ", ".. ", "...", " "]}, - "simpleDotsScrolling": { - "interval": 200, - "frames": [". ", ".. ", "...", " ..", " .", " "], - }, - "star": {"interval": 70, "frames": "✶✸✹✺✹✷"}, - "star2": {"interval": 80, "frames": "+x*"}, - "flip": { - "interval": 70, - "frames": "___-``'´-___", - }, - "hamburger": {"interval": 100, "frames": "☱☲☴"}, - "growVertical": { - "interval": 120, - "frames": "▁▃▄▅▆▇▆▅▄▃", - }, - "growHorizontal": { - "interval": 120, - "frames": "▏▎▍▌▋▊▉▊▋▌▍▎", - }, - "balloon": {"interval": 140, "frames": " .oO@* "}, - "balloon2": {"interval": 120, "frames": ".oO°Oo."}, - "noise": {"interval": 100, "frames": "▓▒░"}, - "bounce": {"interval": 120, "frames": "⠁⠂⠄⠂"}, - "boxBounce": {"interval": 120, "frames": "▖▘▝▗"}, - "boxBounce2": {"interval": 100, "frames": "▌▀▐▄"}, - "triangle": {"interval": 50, "frames": "◢◣◤◥"}, - "arc": {"interval": 100, "frames": "◜◠◝◞◡◟"}, - "circle": {"interval": 120, "frames": "◡⊙◠"}, - "squareCorners": {"interval": 180, "frames": "◰◳◲◱"}, - "circleQuarters": {"interval": 120, "frames": "◴◷◶◵"}, - "circleHalves": {"interval": 50, "frames": "◐◓◑◒"}, - "squish": {"interval": 100, "frames": "╫╪"}, - "toggle": {"interval": 250, "frames": "⊶⊷"}, - "toggle2": {"interval": 80, "frames": "▫▪"}, - "toggle3": {"interval": 120, "frames": "□■"}, - "toggle4": {"interval": 100, "frames": "■□▪▫"}, - "toggle5": {"interval": 100, "frames": "▮▯"}, - "toggle6": {"interval": 300, "frames": "ဝ၀"}, - "toggle7": {"interval": 80, "frames": "⦾⦿"}, - "toggle8": {"interval": 100, "frames": "◍◌"}, - "toggle9": {"interval": 100, "frames": "◉◎"}, - "toggle10": {"interval": 100, "frames": "㊂㊀㊁"}, - "toggle11": {"interval": 50, "frames": "⧇⧆"}, - "toggle12": {"interval": 120, "frames": "☗☖"}, - "toggle13": {"interval": 80, "frames": "=*-"}, - "arrow": {"interval": 100, "frames": "←↖↑↗→↘↓↙"}, - "arrow2": { - "interval": 80, - "frames": ["⬆️ ", "↗️ ", "➡️ ", "↘️ ", "⬇️ ", "↙️ ", "⬅️ ", "↖️ "], - }, - "arrow3": { - "interval": 120, - "frames": ["▹▹▹▹▹", "▸▹▹▹▹", "▹▸▹▹▹", "▹▹▸▹▹", "▹▹▹▸▹", "▹▹▹▹▸"], - }, - "bouncingBar": { - "interval": 80, - "frames": [ - "[ ]", - "[= ]", - "[== ]", - "[=== ]", - "[ ===]", - "[ ==]", - "[ =]", - "[ ]", - "[ =]", - "[ ==]", - "[ ===]", - "[====]", - "[=== ]", - "[== ]", - "[= ]", - ], - }, - "bouncingBall": { - "interval": 80, - "frames": [ - "( ● )", - "( ● )", - "( ● )", - "( ● )", - "( ●)", - "( ● )", - "( ● )", - "( ● )", - "( ● )", - "(● )", - ], - }, - "smiley": {"interval": 200, "frames": ["😄 ", "😝 "]}, - "monkey": {"interval": 300, "frames": ["🙈 ", "🙈 ", "🙉 ", "🙊 "]}, - "hearts": {"interval": 100, "frames": ["💛 ", "💙 ", "💜 ", "💚 ", "❤️ "]}, - "clock": { - "interval": 100, - "frames": [ - "🕛 ", - "🕐 ", - "🕑 ", - "🕒 ", - "🕓 ", - "🕔 ", - "🕕 ", - "🕖 ", - "🕗 ", - "🕘 ", - "🕙 ", - "🕚 ", - ], - }, - "earth": {"interval": 180, "frames": ["🌍 ", "🌎 ", "🌏 "]}, - "material": { - "interval": 17, - "frames": [ - "█▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁", - "██▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁", - "███▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁", - "████▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁", - "██████▁▁▁▁▁▁▁▁▁▁▁▁▁▁", - "██████▁▁▁▁▁▁▁▁▁▁▁▁▁▁", - "███████▁▁▁▁▁▁▁▁▁▁▁▁▁", - "████████▁▁▁▁▁▁▁▁▁▁▁▁", - "█████████▁▁▁▁▁▁▁▁▁▁▁", - "█████████▁▁▁▁▁▁▁▁▁▁▁", - "██████████▁▁▁▁▁▁▁▁▁▁", - "███████████▁▁▁▁▁▁▁▁▁", - "█████████████▁▁▁▁▁▁▁", - "██████████████▁▁▁▁▁▁", - "██████████████▁▁▁▁▁▁", - "▁██████████████▁▁▁▁▁", - "▁██████████████▁▁▁▁▁", - "▁██████████████▁▁▁▁▁", - "▁▁██████████████▁▁▁▁", - "▁▁▁██████████████▁▁▁", - "▁▁▁▁█████████████▁▁▁", - "▁▁▁▁██████████████▁▁", - "▁▁▁▁██████████████▁▁", - "▁▁▁▁▁██████████████▁", - "▁▁▁▁▁██████████████▁", - "▁▁▁▁▁██████████████▁", - "▁▁▁▁▁▁██████████████", - "▁▁▁▁▁▁██████████████", - "▁▁▁▁▁▁▁█████████████", - "▁▁▁▁▁▁▁█████████████", - "▁▁▁▁▁▁▁▁████████████", - "▁▁▁▁▁▁▁▁████████████", - "▁▁▁▁▁▁▁▁▁███████████", - "▁▁▁▁▁▁▁▁▁███████████", - "▁▁▁▁▁▁▁▁▁▁██████████", - "▁▁▁▁▁▁▁▁▁▁██████████", - "▁▁▁▁▁▁▁▁▁▁▁▁████████", - "▁▁▁▁▁▁▁▁▁▁▁▁▁███████", - "▁▁▁▁▁▁▁▁▁▁▁▁▁▁██████", - "▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁█████", - "▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁█████", - "█▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁████", - "██▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁███", - "██▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁███", - "███▁▁▁▁▁▁▁▁▁▁▁▁▁▁███", - "████▁▁▁▁▁▁▁▁▁▁▁▁▁▁██", - "█████▁▁▁▁▁▁▁▁▁▁▁▁▁▁█", - "█████▁▁▁▁▁▁▁▁▁▁▁▁▁▁█", - "██████▁▁▁▁▁▁▁▁▁▁▁▁▁█", - "████████▁▁▁▁▁▁▁▁▁▁▁▁", - "█████████▁▁▁▁▁▁▁▁▁▁▁", - "█████████▁▁▁▁▁▁▁▁▁▁▁", - "█████████▁▁▁▁▁▁▁▁▁▁▁", - "█████████▁▁▁▁▁▁▁▁▁▁▁", - "███████████▁▁▁▁▁▁▁▁▁", - "████████████▁▁▁▁▁▁▁▁", - "████████████▁▁▁▁▁▁▁▁", - "██████████████▁▁▁▁▁▁", - "██████████████▁▁▁▁▁▁", - "▁██████████████▁▁▁▁▁", - "▁██████████████▁▁▁▁▁", - "▁▁▁█████████████▁▁▁▁", - "▁▁▁▁▁████████████▁▁▁", - "▁▁▁▁▁████████████▁▁▁", - "▁▁▁▁▁▁███████████▁▁▁", - "▁▁▁▁▁▁▁▁█████████▁▁▁", - "▁▁▁▁▁▁▁▁█████████▁▁▁", - "▁▁▁▁▁▁▁▁▁█████████▁▁", - "▁▁▁▁▁▁▁▁▁█████████▁▁", - "▁▁▁▁▁▁▁▁▁▁█████████▁", - "▁▁▁▁▁▁▁▁▁▁▁████████▁", - "▁▁▁▁▁▁▁▁▁▁▁████████▁", - "▁▁▁▁▁▁▁▁▁▁▁▁███████▁", - "▁▁▁▁▁▁▁▁▁▁▁▁███████▁", - "▁▁▁▁▁▁▁▁▁▁▁▁▁███████", - "▁▁▁▁▁▁▁▁▁▁▁▁▁███████", - "▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁█████", - "▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁████", - "▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁████", - "▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁████", - "▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁███", - "▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁███", - "▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁██", - "▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁██", - "▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁██", - "▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁█", - "▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁█", - "▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁█", - "▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁", - "▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁", - "▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁", - "▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁", - ], - }, - "moon": { - "interval": 80, - "frames": ["🌑 ", "🌒 ", "🌓 ", "🌔 ", "🌕 ", "🌖 ", "🌗 ", "🌘 "], - }, - "runner": {"interval": 140, "frames": ["🚶 ", "🏃 "]}, - "pong": { - "interval": 80, - "frames": [ - "▐⠂ ▌", - "▐⠈ ▌", - "▐ ⠂ ▌", - "▐ ⠠ ▌", - "▐ ⡀ ▌", - "▐ ⠠ ▌", - "▐ ⠂ ▌", - "▐ ⠈ ▌", - "▐ ⠂ ▌", - "▐ ⠠ ▌", - "▐ ⡀ ▌", - "▐ ⠠ ▌", - "▐ ⠂ ▌", - "▐ ⠈ ▌", - "▐ ⠂▌", - "▐ ⠠▌", - "▐ ⡀▌", - "▐ ⠠ ▌", - "▐ ⠂ ▌", - "▐ ⠈ ▌", - "▐ ⠂ ▌", - "▐ ⠠ ▌", - "▐ ⡀ ▌", - "▐ ⠠ ▌", - "▐ ⠂ ▌", - "▐ ⠈ ▌", - "▐ ⠂ ▌", - "▐ ⠠ ▌", - "▐ ⡀ ▌", - "▐⠠ ▌", - ], - }, - "shark": { - "interval": 120, - "frames": [ - "▐|\\____________▌", - "▐_|\\___________▌", - "▐__|\\__________▌", - "▐___|\\_________▌", - "▐____|\\________▌", - "▐_____|\\_______▌", - "▐______|\\______▌", - "▐_______|\\_____▌", - "▐________|\\____▌", - "▐_________|\\___▌", - "▐__________|\\__▌", - "▐___________|\\_▌", - "▐____________|\\▌", - "▐____________/|▌", - "▐___________/|_▌", - "▐__________/|__▌", - "▐_________/|___▌", - "▐________/|____▌", - "▐_______/|_____▌", - "▐______/|______▌", - "▐_____/|_______▌", - "▐____/|________▌", - "▐___/|_________▌", - "▐__/|__________▌", - "▐_/|___________▌", - "▐/|____________▌", - ], - }, - "dqpb": {"interval": 100, "frames": "dqpb"}, - "weather": { - "interval": 100, - "frames": [ - "☀️ ", - "☀️ ", - "☀️ ", - "🌤 ", - "⛅️ ", - "🌥 ", - "☁️ ", - "🌧 ", - "🌨 ", - "🌧 ", - "🌨 ", - "🌧 ", - "🌨 ", - "⛈ ", - "🌨 ", - "🌧 ", - "🌨 ", - "☁️ ", - "🌥 ", - "⛅️ ", - "🌤 ", - "☀️ ", - "☀️ ", - ], - }, - "christmas": {"interval": 400, "frames": "🌲🎄"}, - "grenade": { - "interval": 80, - "frames": [ - "، ", - "′ ", - " ´ ", - " ‾ ", - " ⸌", - " ⸊", - " |", - " ⁎", - " ⁕", - " ෴ ", - " ⁓", - " ", - " ", - " ", - ], - }, - "point": {"interval": 125, "frames": ["∙∙∙", "●∙∙", "∙●∙", "∙∙●", "∙∙∙"]}, - "layer": {"interval": 150, "frames": "-=≡"}, - "betaWave": { - "interval": 80, - "frames": [ - "ρββββββ", - "βρβββββ", - "ββρββββ", - "βββρβββ", - "ββββρββ", - "βββββρβ", - "ββββββρ", - ], - }, - "aesthetic": { - "interval": 80, - "frames": [ - "▰▱▱▱▱▱▱", - "▰▰▱▱▱▱▱", - "▰▰▰▱▱▱▱", - "▰▰▰▰▱▱▱", - "▰▰▰▰▰▱▱", - "▰▰▰▰▰▰▱", - "▰▰▰▰▰▰▰", - "▰▱▱▱▱▱▱", - ], - }, -} diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/rich/_stack.py b/venv/lib/python3.11/site-packages/pip/_vendor/rich/_stack.py deleted file mode 100644 index 194564e..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/rich/_stack.py +++ /dev/null @@ -1,16 +0,0 @@ -from typing import List, TypeVar - -T = TypeVar("T") - - -class Stack(List[T]): - """A small shim over builtin list.""" - - @property - def top(self) -> T: - """Get top of stack.""" - return self[-1] - - def push(self, item: T) -> None: - """Push an item on to the stack (append in stack nomenclature).""" - self.append(item) diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/rich/_timer.py b/venv/lib/python3.11/site-packages/pip/_vendor/rich/_timer.py deleted file mode 100644 index a2ca6be..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/rich/_timer.py +++ /dev/null @@ -1,19 +0,0 @@ -""" -Timer context manager, only used in debug. - -""" - -from time import time - -import contextlib -from typing import Generator - - -@contextlib.contextmanager -def timer(subject: str = "time") -> Generator[None, None, None]: - """print the elapsed time. (only used in debugging)""" - start = time() - yield - elapsed = time() - start - elapsed_ms = elapsed * 1000 - print(f"{subject} elapsed {elapsed_ms:.1f}ms") diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/rich/_win32_console.py b/venv/lib/python3.11/site-packages/pip/_vendor/rich/_win32_console.py deleted file mode 100644 index 81b1082..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/rich/_win32_console.py +++ /dev/null @@ -1,662 +0,0 @@ -"""Light wrapper around the Win32 Console API - this module should only be imported on Windows - -The API that this module wraps is documented at https://docs.microsoft.com/en-us/windows/console/console-functions -""" -import ctypes -import sys -from typing import Any - -windll: Any = None -if sys.platform == "win32": - windll = ctypes.LibraryLoader(ctypes.WinDLL) -else: - raise ImportError(f"{__name__} can only be imported on Windows") - -import time -from ctypes import Structure, byref, wintypes -from typing import IO, NamedTuple, Type, cast - -from pip._vendor.rich.color import ColorSystem -from pip._vendor.rich.style import Style - -STDOUT = -11 -ENABLE_VIRTUAL_TERMINAL_PROCESSING = 4 - -COORD = wintypes._COORD - - -class LegacyWindowsError(Exception): - pass - - -class WindowsCoordinates(NamedTuple): - """Coordinates in the Windows Console API are (y, x), not (x, y). - This class is intended to prevent that confusion. - Rows and columns are indexed from 0. - This class can be used in place of wintypes._COORD in arguments and argtypes. - """ - - row: int - col: int - - @classmethod - def from_param(cls, value: "WindowsCoordinates") -> COORD: - """Converts a WindowsCoordinates into a wintypes _COORD structure. - This classmethod is internally called by ctypes to perform the conversion. - - Args: - value (WindowsCoordinates): The input coordinates to convert. - - Returns: - wintypes._COORD: The converted coordinates struct. - """ - return COORD(value.col, value.row) - - -class CONSOLE_SCREEN_BUFFER_INFO(Structure): - _fields_ = [ - ("dwSize", COORD), - ("dwCursorPosition", COORD), - ("wAttributes", wintypes.WORD), - ("srWindow", wintypes.SMALL_RECT), - ("dwMaximumWindowSize", COORD), - ] - - -class CONSOLE_CURSOR_INFO(ctypes.Structure): - _fields_ = [("dwSize", wintypes.DWORD), ("bVisible", wintypes.BOOL)] - - -_GetStdHandle = windll.kernel32.GetStdHandle -_GetStdHandle.argtypes = [ - wintypes.DWORD, -] -_GetStdHandle.restype = wintypes.HANDLE - - -def GetStdHandle(handle: int = STDOUT) -> wintypes.HANDLE: - """Retrieves a handle to the specified standard device (standard input, standard output, or standard error). - - Args: - handle (int): Integer identifier for the handle. Defaults to -11 (stdout). - - Returns: - wintypes.HANDLE: The handle - """ - return cast(wintypes.HANDLE, _GetStdHandle(handle)) - - -_GetConsoleMode = windll.kernel32.GetConsoleMode -_GetConsoleMode.argtypes = [wintypes.HANDLE, wintypes.LPDWORD] -_GetConsoleMode.restype = wintypes.BOOL - - -def GetConsoleMode(std_handle: wintypes.HANDLE) -> int: - """Retrieves the current input mode of a console's input buffer - or the current output mode of a console screen buffer. - - Args: - std_handle (wintypes.HANDLE): A handle to the console input buffer or the console screen buffer. - - Raises: - LegacyWindowsError: If any error occurs while calling the Windows console API. - - Returns: - int: Value representing the current console mode as documented at - https://docs.microsoft.com/en-us/windows/console/getconsolemode#parameters - """ - - console_mode = wintypes.DWORD() - success = bool(_GetConsoleMode(std_handle, console_mode)) - if not success: - raise LegacyWindowsError("Unable to get legacy Windows Console Mode") - return console_mode.value - - -_FillConsoleOutputCharacterW = windll.kernel32.FillConsoleOutputCharacterW -_FillConsoleOutputCharacterW.argtypes = [ - wintypes.HANDLE, - ctypes.c_char, - wintypes.DWORD, - cast(Type[COORD], WindowsCoordinates), - ctypes.POINTER(wintypes.DWORD), -] -_FillConsoleOutputCharacterW.restype = wintypes.BOOL - - -def FillConsoleOutputCharacter( - std_handle: wintypes.HANDLE, - char: str, - length: int, - start: WindowsCoordinates, -) -> int: - """Writes a character to the console screen buffer a specified number of times, beginning at the specified coordinates. - - Args: - std_handle (wintypes.HANDLE): A handle to the console input buffer or the console screen buffer. - char (str): The character to write. Must be a string of length 1. - length (int): The number of times to write the character. - start (WindowsCoordinates): The coordinates to start writing at. - - Returns: - int: The number of characters written. - """ - character = ctypes.c_char(char.encode()) - num_characters = wintypes.DWORD(length) - num_written = wintypes.DWORD(0) - _FillConsoleOutputCharacterW( - std_handle, - character, - num_characters, - start, - byref(num_written), - ) - return num_written.value - - -_FillConsoleOutputAttribute = windll.kernel32.FillConsoleOutputAttribute -_FillConsoleOutputAttribute.argtypes = [ - wintypes.HANDLE, - wintypes.WORD, - wintypes.DWORD, - cast(Type[COORD], WindowsCoordinates), - ctypes.POINTER(wintypes.DWORD), -] -_FillConsoleOutputAttribute.restype = wintypes.BOOL - - -def FillConsoleOutputAttribute( - std_handle: wintypes.HANDLE, - attributes: int, - length: int, - start: WindowsCoordinates, -) -> int: - """Sets the character attributes for a specified number of character cells, - beginning at the specified coordinates in a screen buffer. - - Args: - std_handle (wintypes.HANDLE): A handle to the console input buffer or the console screen buffer. - attributes (int): Integer value representing the foreground and background colours of the cells. - length (int): The number of cells to set the output attribute of. - start (WindowsCoordinates): The coordinates of the first cell whose attributes are to be set. - - Returns: - int: The number of cells whose attributes were actually set. - """ - num_cells = wintypes.DWORD(length) - style_attrs = wintypes.WORD(attributes) - num_written = wintypes.DWORD(0) - _FillConsoleOutputAttribute( - std_handle, style_attrs, num_cells, start, byref(num_written) - ) - return num_written.value - - -_SetConsoleTextAttribute = windll.kernel32.SetConsoleTextAttribute -_SetConsoleTextAttribute.argtypes = [ - wintypes.HANDLE, - wintypes.WORD, -] -_SetConsoleTextAttribute.restype = wintypes.BOOL - - -def SetConsoleTextAttribute( - std_handle: wintypes.HANDLE, attributes: wintypes.WORD -) -> bool: - """Set the colour attributes for all text written after this function is called. - - Args: - std_handle (wintypes.HANDLE): A handle to the console input buffer or the console screen buffer. - attributes (int): Integer value representing the foreground and background colours. - - - Returns: - bool: True if the attribute was set successfully, otherwise False. - """ - return bool(_SetConsoleTextAttribute(std_handle, attributes)) - - -_GetConsoleScreenBufferInfo = windll.kernel32.GetConsoleScreenBufferInfo -_GetConsoleScreenBufferInfo.argtypes = [ - wintypes.HANDLE, - ctypes.POINTER(CONSOLE_SCREEN_BUFFER_INFO), -] -_GetConsoleScreenBufferInfo.restype = wintypes.BOOL - - -def GetConsoleScreenBufferInfo( - std_handle: wintypes.HANDLE, -) -> CONSOLE_SCREEN_BUFFER_INFO: - """Retrieves information about the specified console screen buffer. - - Args: - std_handle (wintypes.HANDLE): A handle to the console input buffer or the console screen buffer. - - Returns: - CONSOLE_SCREEN_BUFFER_INFO: A CONSOLE_SCREEN_BUFFER_INFO ctype struct contain information about - screen size, cursor position, colour attributes, and more.""" - console_screen_buffer_info = CONSOLE_SCREEN_BUFFER_INFO() - _GetConsoleScreenBufferInfo(std_handle, byref(console_screen_buffer_info)) - return console_screen_buffer_info - - -_SetConsoleCursorPosition = windll.kernel32.SetConsoleCursorPosition -_SetConsoleCursorPosition.argtypes = [ - wintypes.HANDLE, - cast(Type[COORD], WindowsCoordinates), -] -_SetConsoleCursorPosition.restype = wintypes.BOOL - - -def SetConsoleCursorPosition( - std_handle: wintypes.HANDLE, coords: WindowsCoordinates -) -> bool: - """Set the position of the cursor in the console screen - - Args: - std_handle (wintypes.HANDLE): A handle to the console input buffer or the console screen buffer. - coords (WindowsCoordinates): The coordinates to move the cursor to. - - Returns: - bool: True if the function succeeds, otherwise False. - """ - return bool(_SetConsoleCursorPosition(std_handle, coords)) - - -_GetConsoleCursorInfo = windll.kernel32.GetConsoleCursorInfo -_GetConsoleCursorInfo.argtypes = [ - wintypes.HANDLE, - ctypes.POINTER(CONSOLE_CURSOR_INFO), -] -_GetConsoleCursorInfo.restype = wintypes.BOOL - - -def GetConsoleCursorInfo( - std_handle: wintypes.HANDLE, cursor_info: CONSOLE_CURSOR_INFO -) -> bool: - """Get the cursor info - used to get cursor visibility and width - - Args: - std_handle (wintypes.HANDLE): A handle to the console input buffer or the console screen buffer. - cursor_info (CONSOLE_CURSOR_INFO): CONSOLE_CURSOR_INFO ctype struct that receives information - about the console's cursor. - - Returns: - bool: True if the function succeeds, otherwise False. - """ - return bool(_GetConsoleCursorInfo(std_handle, byref(cursor_info))) - - -_SetConsoleCursorInfo = windll.kernel32.SetConsoleCursorInfo -_SetConsoleCursorInfo.argtypes = [ - wintypes.HANDLE, - ctypes.POINTER(CONSOLE_CURSOR_INFO), -] -_SetConsoleCursorInfo.restype = wintypes.BOOL - - -def SetConsoleCursorInfo( - std_handle: wintypes.HANDLE, cursor_info: CONSOLE_CURSOR_INFO -) -> bool: - """Set the cursor info - used for adjusting cursor visibility and width - - Args: - std_handle (wintypes.HANDLE): A handle to the console input buffer or the console screen buffer. - cursor_info (CONSOLE_CURSOR_INFO): CONSOLE_CURSOR_INFO ctype struct containing the new cursor info. - - Returns: - bool: True if the function succeeds, otherwise False. - """ - return bool(_SetConsoleCursorInfo(std_handle, byref(cursor_info))) - - -_SetConsoleTitle = windll.kernel32.SetConsoleTitleW -_SetConsoleTitle.argtypes = [wintypes.LPCWSTR] -_SetConsoleTitle.restype = wintypes.BOOL - - -def SetConsoleTitle(title: str) -> bool: - """Sets the title of the current console window - - Args: - title (str): The new title of the console window. - - Returns: - bool: True if the function succeeds, otherwise False. - """ - return bool(_SetConsoleTitle(title)) - - -class LegacyWindowsTerm: - """This class allows interaction with the legacy Windows Console API. It should only be used in the context - of environments where virtual terminal processing is not available. However, if it is used in a Windows environment, - the entire API should work. - - Args: - file (IO[str]): The file which the Windows Console API HANDLE is retrieved from, defaults to sys.stdout. - """ - - BRIGHT_BIT = 8 - - # Indices are ANSI color numbers, values are the corresponding Windows Console API color numbers - ANSI_TO_WINDOWS = [ - 0, # black The Windows colours are defined in wincon.h as follows: - 4, # red define FOREGROUND_BLUE 0x0001 -- 0000 0001 - 2, # green define FOREGROUND_GREEN 0x0002 -- 0000 0010 - 6, # yellow define FOREGROUND_RED 0x0004 -- 0000 0100 - 1, # blue define FOREGROUND_INTENSITY 0x0008 -- 0000 1000 - 5, # magenta define BACKGROUND_BLUE 0x0010 -- 0001 0000 - 3, # cyan define BACKGROUND_GREEN 0x0020 -- 0010 0000 - 7, # white define BACKGROUND_RED 0x0040 -- 0100 0000 - 8, # bright black (grey) define BACKGROUND_INTENSITY 0x0080 -- 1000 0000 - 12, # bright red - 10, # bright green - 14, # bright yellow - 9, # bright blue - 13, # bright magenta - 11, # bright cyan - 15, # bright white - ] - - def __init__(self, file: "IO[str]") -> None: - handle = GetStdHandle(STDOUT) - self._handle = handle - default_text = GetConsoleScreenBufferInfo(handle).wAttributes - self._default_text = default_text - - self._default_fore = default_text & 7 - self._default_back = (default_text >> 4) & 7 - self._default_attrs = self._default_fore | (self._default_back << 4) - - self._file = file - self.write = file.write - self.flush = file.flush - - @property - def cursor_position(self) -> WindowsCoordinates: - """Returns the current position of the cursor (0-based) - - Returns: - WindowsCoordinates: The current cursor position. - """ - coord: COORD = GetConsoleScreenBufferInfo(self._handle).dwCursorPosition - return WindowsCoordinates(row=cast(int, coord.Y), col=cast(int, coord.X)) - - @property - def screen_size(self) -> WindowsCoordinates: - """Returns the current size of the console screen buffer, in character columns and rows - - Returns: - WindowsCoordinates: The width and height of the screen as WindowsCoordinates. - """ - screen_size: COORD = GetConsoleScreenBufferInfo(self._handle).dwSize - return WindowsCoordinates( - row=cast(int, screen_size.Y), col=cast(int, screen_size.X) - ) - - def write_text(self, text: str) -> None: - """Write text directly to the terminal without any modification of styles - - Args: - text (str): The text to write to the console - """ - self.write(text) - self.flush() - - def write_styled(self, text: str, style: Style) -> None: - """Write styled text to the terminal. - - Args: - text (str): The text to write - style (Style): The style of the text - """ - color = style.color - bgcolor = style.bgcolor - if style.reverse: - color, bgcolor = bgcolor, color - - if color: - fore = color.downgrade(ColorSystem.WINDOWS).number - fore = fore if fore is not None else 7 # Default to ANSI 7: White - if style.bold: - fore = fore | self.BRIGHT_BIT - if style.dim: - fore = fore & ~self.BRIGHT_BIT - fore = self.ANSI_TO_WINDOWS[fore] - else: - fore = self._default_fore - - if bgcolor: - back = bgcolor.downgrade(ColorSystem.WINDOWS).number - back = back if back is not None else 0 # Default to ANSI 0: Black - back = self.ANSI_TO_WINDOWS[back] - else: - back = self._default_back - - assert fore is not None - assert back is not None - - SetConsoleTextAttribute( - self._handle, attributes=ctypes.c_ushort(fore | (back << 4)) - ) - self.write_text(text) - SetConsoleTextAttribute(self._handle, attributes=self._default_text) - - def move_cursor_to(self, new_position: WindowsCoordinates) -> None: - """Set the position of the cursor - - Args: - new_position (WindowsCoordinates): The WindowsCoordinates representing the new position of the cursor. - """ - if new_position.col < 0 or new_position.row < 0: - return - SetConsoleCursorPosition(self._handle, coords=new_position) - - def erase_line(self) -> None: - """Erase all content on the line the cursor is currently located at""" - screen_size = self.screen_size - cursor_position = self.cursor_position - cells_to_erase = screen_size.col - start_coordinates = WindowsCoordinates(row=cursor_position.row, col=0) - FillConsoleOutputCharacter( - self._handle, " ", length=cells_to_erase, start=start_coordinates - ) - FillConsoleOutputAttribute( - self._handle, - self._default_attrs, - length=cells_to_erase, - start=start_coordinates, - ) - - def erase_end_of_line(self) -> None: - """Erase all content from the cursor position to the end of that line""" - cursor_position = self.cursor_position - cells_to_erase = self.screen_size.col - cursor_position.col - FillConsoleOutputCharacter( - self._handle, " ", length=cells_to_erase, start=cursor_position - ) - FillConsoleOutputAttribute( - self._handle, - self._default_attrs, - length=cells_to_erase, - start=cursor_position, - ) - - def erase_start_of_line(self) -> None: - """Erase all content from the cursor position to the start of that line""" - row, col = self.cursor_position - start = WindowsCoordinates(row, 0) - FillConsoleOutputCharacter(self._handle, " ", length=col, start=start) - FillConsoleOutputAttribute( - self._handle, self._default_attrs, length=col, start=start - ) - - def move_cursor_up(self) -> None: - """Move the cursor up a single cell""" - cursor_position = self.cursor_position - SetConsoleCursorPosition( - self._handle, - coords=WindowsCoordinates( - row=cursor_position.row - 1, col=cursor_position.col - ), - ) - - def move_cursor_down(self) -> None: - """Move the cursor down a single cell""" - cursor_position = self.cursor_position - SetConsoleCursorPosition( - self._handle, - coords=WindowsCoordinates( - row=cursor_position.row + 1, - col=cursor_position.col, - ), - ) - - def move_cursor_forward(self) -> None: - """Move the cursor forward a single cell. Wrap to the next line if required.""" - row, col = self.cursor_position - if col == self.screen_size.col - 1: - row += 1 - col = 0 - else: - col += 1 - SetConsoleCursorPosition( - self._handle, coords=WindowsCoordinates(row=row, col=col) - ) - - def move_cursor_to_column(self, column: int) -> None: - """Move cursor to the column specified by the zero-based column index, staying on the same row - - Args: - column (int): The zero-based column index to move the cursor to. - """ - row, _ = self.cursor_position - SetConsoleCursorPosition(self._handle, coords=WindowsCoordinates(row, column)) - - def move_cursor_backward(self) -> None: - """Move the cursor backward a single cell. Wrap to the previous line if required.""" - row, col = self.cursor_position - if col == 0: - row -= 1 - col = self.screen_size.col - 1 - else: - col -= 1 - SetConsoleCursorPosition( - self._handle, coords=WindowsCoordinates(row=row, col=col) - ) - - def hide_cursor(self) -> None: - """Hide the cursor""" - current_cursor_size = self._get_cursor_size() - invisible_cursor = CONSOLE_CURSOR_INFO(dwSize=current_cursor_size, bVisible=0) - SetConsoleCursorInfo(self._handle, cursor_info=invisible_cursor) - - def show_cursor(self) -> None: - """Show the cursor""" - current_cursor_size = self._get_cursor_size() - visible_cursor = CONSOLE_CURSOR_INFO(dwSize=current_cursor_size, bVisible=1) - SetConsoleCursorInfo(self._handle, cursor_info=visible_cursor) - - def set_title(self, title: str) -> None: - """Set the title of the terminal window - - Args: - title (str): The new title of the console window - """ - assert len(title) < 255, "Console title must be less than 255 characters" - SetConsoleTitle(title) - - def _get_cursor_size(self) -> int: - """Get the percentage of the character cell that is filled by the cursor""" - cursor_info = CONSOLE_CURSOR_INFO() - GetConsoleCursorInfo(self._handle, cursor_info=cursor_info) - return int(cursor_info.dwSize) - - -if __name__ == "__main__": - handle = GetStdHandle() - - from pip._vendor.rich.console import Console - - console = Console() - - term = LegacyWindowsTerm(sys.stdout) - term.set_title("Win32 Console Examples") - - style = Style(color="black", bgcolor="red") - - heading = Style.parse("black on green") - - # Check colour output - console.rule("Checking colour output") - console.print("[on red]on red!") - console.print("[blue]blue!") - console.print("[yellow]yellow!") - console.print("[bold yellow]bold yellow!") - console.print("[bright_yellow]bright_yellow!") - console.print("[dim bright_yellow]dim bright_yellow!") - console.print("[italic cyan]italic cyan!") - console.print("[bold white on blue]bold white on blue!") - console.print("[reverse bold white on blue]reverse bold white on blue!") - console.print("[bold black on cyan]bold black on cyan!") - console.print("[black on green]black on green!") - console.print("[blue on green]blue on green!") - console.print("[white on black]white on black!") - console.print("[black on white]black on white!") - console.print("[#1BB152 on #DA812D]#1BB152 on #DA812D!") - - # Check cursor movement - console.rule("Checking cursor movement") - console.print() - term.move_cursor_backward() - term.move_cursor_backward() - term.write_text("went back and wrapped to prev line") - time.sleep(1) - term.move_cursor_up() - term.write_text("we go up") - time.sleep(1) - term.move_cursor_down() - term.write_text("and down") - time.sleep(1) - term.move_cursor_up() - term.move_cursor_backward() - term.move_cursor_backward() - term.write_text("we went up and back 2") - time.sleep(1) - term.move_cursor_down() - term.move_cursor_backward() - term.move_cursor_backward() - term.write_text("we went down and back 2") - time.sleep(1) - - # Check erasing of lines - term.hide_cursor() - console.print() - console.rule("Checking line erasing") - console.print("\n...Deleting to the start of the line...") - term.write_text("The red arrow shows the cursor location, and direction of erase") - time.sleep(1) - term.move_cursor_to_column(16) - term.write_styled("<", Style.parse("black on red")) - term.move_cursor_backward() - time.sleep(1) - term.erase_start_of_line() - time.sleep(1) - - console.print("\n\n...And to the end of the line...") - term.write_text("The red arrow shows the cursor location, and direction of erase") - time.sleep(1) - - term.move_cursor_to_column(16) - term.write_styled(">", Style.parse("black on red")) - time.sleep(1) - term.erase_end_of_line() - time.sleep(1) - - console.print("\n\n...Now the whole line will be erased...") - term.write_styled("I'm going to disappear!", style=Style.parse("black on cyan")) - time.sleep(1) - term.erase_line() - - term.show_cursor() - print("\n") diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/rich/_windows.py b/venv/lib/python3.11/site-packages/pip/_vendor/rich/_windows.py deleted file mode 100644 index 10fc0d7..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/rich/_windows.py +++ /dev/null @@ -1,72 +0,0 @@ -import sys -from dataclasses import dataclass - - -@dataclass -class WindowsConsoleFeatures: - """Windows features available.""" - - vt: bool = False - """The console supports VT codes.""" - truecolor: bool = False - """The console supports truecolor.""" - - -try: - import ctypes - from ctypes import LibraryLoader - - if sys.platform == "win32": - windll = LibraryLoader(ctypes.WinDLL) - else: - windll = None - raise ImportError("Not windows") - - from pip._vendor.rich._win32_console import ( - ENABLE_VIRTUAL_TERMINAL_PROCESSING, - GetConsoleMode, - GetStdHandle, - LegacyWindowsError, - ) - -except (AttributeError, ImportError, ValueError): - - # Fallback if we can't load the Windows DLL - def get_windows_console_features() -> WindowsConsoleFeatures: - features = WindowsConsoleFeatures() - return features - -else: - - def get_windows_console_features() -> WindowsConsoleFeatures: - """Get windows console features. - - Returns: - WindowsConsoleFeatures: An instance of WindowsConsoleFeatures. - """ - handle = GetStdHandle() - try: - console_mode = GetConsoleMode(handle) - success = True - except LegacyWindowsError: - console_mode = 0 - success = False - vt = bool(success and console_mode & ENABLE_VIRTUAL_TERMINAL_PROCESSING) - truecolor = False - if vt: - win_version = sys.getwindowsversion() - truecolor = win_version.major > 10 or ( - win_version.major == 10 and win_version.build >= 15063 - ) - features = WindowsConsoleFeatures(vt=vt, truecolor=truecolor) - return features - - -if __name__ == "__main__": - import platform - - features = get_windows_console_features() - from pip._vendor.rich import print - - print(f'platform="{platform.system()}"') - print(repr(features)) diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/rich/_windows_renderer.py b/venv/lib/python3.11/site-packages/pip/_vendor/rich/_windows_renderer.py deleted file mode 100644 index 5ece056..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/rich/_windows_renderer.py +++ /dev/null @@ -1,56 +0,0 @@ -from typing import Iterable, Sequence, Tuple, cast - -from pip._vendor.rich._win32_console import LegacyWindowsTerm, WindowsCoordinates -from pip._vendor.rich.segment import ControlCode, ControlType, Segment - - -def legacy_windows_render(buffer: Iterable[Segment], term: LegacyWindowsTerm) -> None: - """Makes appropriate Windows Console API calls based on the segments in the buffer. - - Args: - buffer (Iterable[Segment]): Iterable of Segments to convert to Win32 API calls. - term (LegacyWindowsTerm): Used to call the Windows Console API. - """ - for text, style, control in buffer: - if not control: - if style: - term.write_styled(text, style) - else: - term.write_text(text) - else: - control_codes: Sequence[ControlCode] = control - for control_code in control_codes: - control_type = control_code[0] - if control_type == ControlType.CURSOR_MOVE_TO: - _, x, y = cast(Tuple[ControlType, int, int], control_code) - term.move_cursor_to(WindowsCoordinates(row=y - 1, col=x - 1)) - elif control_type == ControlType.CARRIAGE_RETURN: - term.write_text("\r") - elif control_type == ControlType.HOME: - term.move_cursor_to(WindowsCoordinates(0, 0)) - elif control_type == ControlType.CURSOR_UP: - term.move_cursor_up() - elif control_type == ControlType.CURSOR_DOWN: - term.move_cursor_down() - elif control_type == ControlType.CURSOR_FORWARD: - term.move_cursor_forward() - elif control_type == ControlType.CURSOR_BACKWARD: - term.move_cursor_backward() - elif control_type == ControlType.CURSOR_MOVE_TO_COLUMN: - _, column = cast(Tuple[ControlType, int], control_code) - term.move_cursor_to_column(column - 1) - elif control_type == ControlType.HIDE_CURSOR: - term.hide_cursor() - elif control_type == ControlType.SHOW_CURSOR: - term.show_cursor() - elif control_type == ControlType.ERASE_IN_LINE: - _, mode = cast(Tuple[ControlType, int], control_code) - if mode == 0: - term.erase_end_of_line() - elif mode == 1: - term.erase_start_of_line() - elif mode == 2: - term.erase_line() - elif control_type == ControlType.SET_WINDOW_TITLE: - _, title = cast(Tuple[ControlType, str], control_code) - term.set_title(title) diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/rich/_wrap.py b/venv/lib/python3.11/site-packages/pip/_vendor/rich/_wrap.py deleted file mode 100644 index c45f193..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/rich/_wrap.py +++ /dev/null @@ -1,56 +0,0 @@ -import re -from typing import Iterable, List, Tuple - -from ._loop import loop_last -from .cells import cell_len, chop_cells - -re_word = re.compile(r"\s*\S+\s*") - - -def words(text: str) -> Iterable[Tuple[int, int, str]]: - position = 0 - word_match = re_word.match(text, position) - while word_match is not None: - start, end = word_match.span() - word = word_match.group(0) - yield start, end, word - word_match = re_word.match(text, end) - - -def divide_line(text: str, width: int, fold: bool = True) -> List[int]: - divides: List[int] = [] - append = divides.append - line_position = 0 - _cell_len = cell_len - for start, _end, word in words(text): - word_length = _cell_len(word.rstrip()) - if line_position + word_length > width: - if word_length > width: - if fold: - chopped_words = chop_cells(word, max_size=width, position=0) - for last, line in loop_last(chopped_words): - if start: - append(start) - - if last: - line_position = _cell_len(line) - else: - start += len(line) - else: - if start: - append(start) - line_position = _cell_len(word) - elif line_position and start: - append(start) - line_position = _cell_len(word) - else: - line_position += _cell_len(word) - return divides - - -if __name__ == "__main__": # pragma: no cover - from .console import Console - - console = Console(width=10) - console.print("12345 abcdefghijklmnopqrstuvwyxzABCDEFGHIJKLMNOPQRSTUVWXYZ 12345") - print(chop_cells("abcdefghijklmnopqrstuvwxyz", 10, position=2)) diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/rich/abc.py b/venv/lib/python3.11/site-packages/pip/_vendor/rich/abc.py deleted file mode 100644 index e6e498e..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/rich/abc.py +++ /dev/null @@ -1,33 +0,0 @@ -from abc import ABC - - -class RichRenderable(ABC): - """An abstract base class for Rich renderables. - - Note that there is no need to extend this class, the intended use is to check if an - object supports the Rich renderable protocol. For example:: - - if isinstance(my_object, RichRenderable): - console.print(my_object) - - """ - - @classmethod - def __subclasshook__(cls, other: type) -> bool: - """Check if this class supports the rich render protocol.""" - return hasattr(other, "__rich_console__") or hasattr(other, "__rich__") - - -if __name__ == "__main__": # pragma: no cover - from pip._vendor.rich.text import Text - - t = Text() - print(isinstance(Text, RichRenderable)) - print(isinstance(t, RichRenderable)) - - class Foo: - pass - - f = Foo() - print(isinstance(f, RichRenderable)) - print(isinstance("", RichRenderable)) diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/rich/align.py b/venv/lib/python3.11/site-packages/pip/_vendor/rich/align.py deleted file mode 100644 index c310b66..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/rich/align.py +++ /dev/null @@ -1,311 +0,0 @@ -import sys -from itertools import chain -from typing import TYPE_CHECKING, Iterable, Optional - -if sys.version_info >= (3, 8): - from typing import Literal -else: - from pip._vendor.typing_extensions import Literal # pragma: no cover - -from .constrain import Constrain -from .jupyter import JupyterMixin -from .measure import Measurement -from .segment import Segment -from .style import StyleType - -if TYPE_CHECKING: - from .console import Console, ConsoleOptions, RenderableType, RenderResult - -AlignMethod = Literal["left", "center", "right"] -VerticalAlignMethod = Literal["top", "middle", "bottom"] - - -class Align(JupyterMixin): - """Align a renderable by adding spaces if necessary. - - Args: - renderable (RenderableType): A console renderable. - align (AlignMethod): One of "left", "center", or "right"" - style (StyleType, optional): An optional style to apply to the background. - vertical (Optional[VerticalAlginMethod], optional): Optional vertical align, one of "top", "middle", or "bottom". Defaults to None. - pad (bool, optional): Pad the right with spaces. Defaults to True. - width (int, optional): Restrict contents to given width, or None to use default width. Defaults to None. - height (int, optional): Set height of align renderable, or None to fit to contents. Defaults to None. - - Raises: - ValueError: if ``align`` is not one of the expected values. - """ - - def __init__( - self, - renderable: "RenderableType", - align: AlignMethod = "left", - style: Optional[StyleType] = None, - *, - vertical: Optional[VerticalAlignMethod] = None, - pad: bool = True, - width: Optional[int] = None, - height: Optional[int] = None, - ) -> None: - if align not in ("left", "center", "right"): - raise ValueError( - f'invalid value for align, expected "left", "center", or "right" (not {align!r})' - ) - if vertical is not None and vertical not in ("top", "middle", "bottom"): - raise ValueError( - f'invalid value for vertical, expected "top", "middle", or "bottom" (not {vertical!r})' - ) - self.renderable = renderable - self.align = align - self.style = style - self.vertical = vertical - self.pad = pad - self.width = width - self.height = height - - def __repr__(self) -> str: - return f"Align({self.renderable!r}, {self.align!r})" - - @classmethod - def left( - cls, - renderable: "RenderableType", - style: Optional[StyleType] = None, - *, - vertical: Optional[VerticalAlignMethod] = None, - pad: bool = True, - width: Optional[int] = None, - height: Optional[int] = None, - ) -> "Align": - """Align a renderable to the left.""" - return cls( - renderable, - "left", - style=style, - vertical=vertical, - pad=pad, - width=width, - height=height, - ) - - @classmethod - def center( - cls, - renderable: "RenderableType", - style: Optional[StyleType] = None, - *, - vertical: Optional[VerticalAlignMethod] = None, - pad: bool = True, - width: Optional[int] = None, - height: Optional[int] = None, - ) -> "Align": - """Align a renderable to the center.""" - return cls( - renderable, - "center", - style=style, - vertical=vertical, - pad=pad, - width=width, - height=height, - ) - - @classmethod - def right( - cls, - renderable: "RenderableType", - style: Optional[StyleType] = None, - *, - vertical: Optional[VerticalAlignMethod] = None, - pad: bool = True, - width: Optional[int] = None, - height: Optional[int] = None, - ) -> "Align": - """Align a renderable to the right.""" - return cls( - renderable, - "right", - style=style, - vertical=vertical, - pad=pad, - width=width, - height=height, - ) - - def __rich_console__( - self, console: "Console", options: "ConsoleOptions" - ) -> "RenderResult": - align = self.align - width = console.measure(self.renderable, options=options).maximum - rendered = console.render( - Constrain( - self.renderable, width if self.width is None else min(width, self.width) - ), - options.update(height=None), - ) - lines = list(Segment.split_lines(rendered)) - width, height = Segment.get_shape(lines) - lines = Segment.set_shape(lines, width, height) - new_line = Segment.line() - excess_space = options.max_width - width - style = console.get_style(self.style) if self.style is not None else None - - def generate_segments() -> Iterable[Segment]: - if excess_space <= 0: - # Exact fit - for line in lines: - yield from line - yield new_line - - elif align == "left": - # Pad on the right - pad = Segment(" " * excess_space, style) if self.pad else None - for line in lines: - yield from line - if pad: - yield pad - yield new_line - - elif align == "center": - # Pad left and right - left = excess_space // 2 - pad = Segment(" " * left, style) - pad_right = ( - Segment(" " * (excess_space - left), style) if self.pad else None - ) - for line in lines: - if left: - yield pad - yield from line - if pad_right: - yield pad_right - yield new_line - - elif align == "right": - # Padding on left - pad = Segment(" " * excess_space, style) - for line in lines: - yield pad - yield from line - yield new_line - - blank_line = ( - Segment(f"{' ' * (self.width or options.max_width)}\n", style) - if self.pad - else Segment("\n") - ) - - def blank_lines(count: int) -> Iterable[Segment]: - if count > 0: - for _ in range(count): - yield blank_line - - vertical_height = self.height or options.height - iter_segments: Iterable[Segment] - if self.vertical and vertical_height is not None: - if self.vertical == "top": - bottom_space = vertical_height - height - iter_segments = chain(generate_segments(), blank_lines(bottom_space)) - elif self.vertical == "middle": - top_space = (vertical_height - height) // 2 - bottom_space = vertical_height - top_space - height - iter_segments = chain( - blank_lines(top_space), - generate_segments(), - blank_lines(bottom_space), - ) - else: # self.vertical == "bottom": - top_space = vertical_height - height - iter_segments = chain(blank_lines(top_space), generate_segments()) - else: - iter_segments = generate_segments() - if self.style: - style = console.get_style(self.style) - iter_segments = Segment.apply_style(iter_segments, style) - yield from iter_segments - - def __rich_measure__( - self, console: "Console", options: "ConsoleOptions" - ) -> Measurement: - measurement = Measurement.get(console, options, self.renderable) - return measurement - - -class VerticalCenter(JupyterMixin): - """Vertically aligns a renderable. - - Warn: - This class is deprecated and may be removed in a future version. Use Align class with - `vertical="middle"`. - - Args: - renderable (RenderableType): A renderable object. - """ - - def __init__( - self, - renderable: "RenderableType", - style: Optional[StyleType] = None, - ) -> None: - self.renderable = renderable - self.style = style - - def __repr__(self) -> str: - return f"VerticalCenter({self.renderable!r})" - - def __rich_console__( - self, console: "Console", options: "ConsoleOptions" - ) -> "RenderResult": - style = console.get_style(self.style) if self.style is not None else None - lines = console.render_lines( - self.renderable, options.update(height=None), pad=False - ) - width, _height = Segment.get_shape(lines) - new_line = Segment.line() - height = options.height or options.size.height - top_space = (height - len(lines)) // 2 - bottom_space = height - top_space - len(lines) - blank_line = Segment(f"{' ' * width}", style) - - def blank_lines(count: int) -> Iterable[Segment]: - for _ in range(count): - yield blank_line - yield new_line - - if top_space > 0: - yield from blank_lines(top_space) - for line in lines: - yield from line - yield new_line - if bottom_space > 0: - yield from blank_lines(bottom_space) - - def __rich_measure__( - self, console: "Console", options: "ConsoleOptions" - ) -> Measurement: - measurement = Measurement.get(console, options, self.renderable) - return measurement - - -if __name__ == "__main__": # pragma: no cover - from pip._vendor.rich.console import Console, Group - from pip._vendor.rich.highlighter import ReprHighlighter - from pip._vendor.rich.panel import Panel - - highlighter = ReprHighlighter() - console = Console() - - panel = Panel( - Group( - Align.left(highlighter("align='left'")), - Align.center(highlighter("align='center'")), - Align.right(highlighter("align='right'")), - ), - width=60, - style="on dark_blue", - title="Align", - ) - - console.print( - Align.center(panel, vertical="middle", style="on red", height=console.height) - ) diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/rich/ansi.py b/venv/lib/python3.11/site-packages/pip/_vendor/rich/ansi.py deleted file mode 100644 index 66365e6..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/rich/ansi.py +++ /dev/null @@ -1,240 +0,0 @@ -import re -import sys -from contextlib import suppress -from typing import Iterable, NamedTuple, Optional - -from .color import Color -from .style import Style -from .text import Text - -re_ansi = re.compile( - r""" -(?:\x1b\](.*?)\x1b\\)| -(?:\x1b([(@-Z\\-_]|\[[0-?]*[ -/]*[@-~])) -""", - re.VERBOSE, -) - - -class _AnsiToken(NamedTuple): - """Result of ansi tokenized string.""" - - plain: str = "" - sgr: Optional[str] = "" - osc: Optional[str] = "" - - -def _ansi_tokenize(ansi_text: str) -> Iterable[_AnsiToken]: - """Tokenize a string in to plain text and ANSI codes. - - Args: - ansi_text (str): A String containing ANSI codes. - - Yields: - AnsiToken: A named tuple of (plain, sgr, osc) - """ - - position = 0 - sgr: Optional[str] - osc: Optional[str] - for match in re_ansi.finditer(ansi_text): - start, end = match.span(0) - osc, sgr = match.groups() - if start > position: - yield _AnsiToken(ansi_text[position:start]) - if sgr: - if sgr == "(": - position = end + 1 - continue - if sgr.endswith("m"): - yield _AnsiToken("", sgr[1:-1], osc) - else: - yield _AnsiToken("", sgr, osc) - position = end - if position < len(ansi_text): - yield _AnsiToken(ansi_text[position:]) - - -SGR_STYLE_MAP = { - 1: "bold", - 2: "dim", - 3: "italic", - 4: "underline", - 5: "blink", - 6: "blink2", - 7: "reverse", - 8: "conceal", - 9: "strike", - 21: "underline2", - 22: "not dim not bold", - 23: "not italic", - 24: "not underline", - 25: "not blink", - 26: "not blink2", - 27: "not reverse", - 28: "not conceal", - 29: "not strike", - 30: "color(0)", - 31: "color(1)", - 32: "color(2)", - 33: "color(3)", - 34: "color(4)", - 35: "color(5)", - 36: "color(6)", - 37: "color(7)", - 39: "default", - 40: "on color(0)", - 41: "on color(1)", - 42: "on color(2)", - 43: "on color(3)", - 44: "on color(4)", - 45: "on color(5)", - 46: "on color(6)", - 47: "on color(7)", - 49: "on default", - 51: "frame", - 52: "encircle", - 53: "overline", - 54: "not frame not encircle", - 55: "not overline", - 90: "color(8)", - 91: "color(9)", - 92: "color(10)", - 93: "color(11)", - 94: "color(12)", - 95: "color(13)", - 96: "color(14)", - 97: "color(15)", - 100: "on color(8)", - 101: "on color(9)", - 102: "on color(10)", - 103: "on color(11)", - 104: "on color(12)", - 105: "on color(13)", - 106: "on color(14)", - 107: "on color(15)", -} - - -class AnsiDecoder: - """Translate ANSI code in to styled Text.""" - - def __init__(self) -> None: - self.style = Style.null() - - def decode(self, terminal_text: str) -> Iterable[Text]: - """Decode ANSI codes in an iterable of lines. - - Args: - lines (Iterable[str]): An iterable of lines of terminal output. - - Yields: - Text: Marked up Text. - """ - for line in terminal_text.splitlines(): - yield self.decode_line(line) - - def decode_line(self, line: str) -> Text: - """Decode a line containing ansi codes. - - Args: - line (str): A line of terminal output. - - Returns: - Text: A Text instance marked up according to ansi codes. - """ - from_ansi = Color.from_ansi - from_rgb = Color.from_rgb - _Style = Style - text = Text() - append = text.append - line = line.rsplit("\r", 1)[-1] - for plain_text, sgr, osc in _ansi_tokenize(line): - if plain_text: - append(plain_text, self.style or None) - elif osc is not None: - if osc.startswith("8;"): - _params, semicolon, link = osc[2:].partition(";") - if semicolon: - self.style = self.style.update_link(link or None) - elif sgr is not None: - # Translate in to semi-colon separated codes - # Ignore invalid codes, because we want to be lenient - codes = [ - min(255, int(_code) if _code else 0) - for _code in sgr.split(";") - if _code.isdigit() or _code == "" - ] - iter_codes = iter(codes) - for code in iter_codes: - if code == 0: - # reset - self.style = _Style.null() - elif code in SGR_STYLE_MAP: - # styles - self.style += _Style.parse(SGR_STYLE_MAP[code]) - elif code == 38: - #  Foreground - with suppress(StopIteration): - color_type = next(iter_codes) - if color_type == 5: - self.style += _Style.from_color( - from_ansi(next(iter_codes)) - ) - elif color_type == 2: - self.style += _Style.from_color( - from_rgb( - next(iter_codes), - next(iter_codes), - next(iter_codes), - ) - ) - elif code == 48: - # Background - with suppress(StopIteration): - color_type = next(iter_codes) - if color_type == 5: - self.style += _Style.from_color( - None, from_ansi(next(iter_codes)) - ) - elif color_type == 2: - self.style += _Style.from_color( - None, - from_rgb( - next(iter_codes), - next(iter_codes), - next(iter_codes), - ), - ) - - return text - - -if sys.platform != "win32" and __name__ == "__main__": # pragma: no cover - import io - import os - import pty - import sys - - decoder = AnsiDecoder() - - stdout = io.BytesIO() - - def read(fd: int) -> bytes: - data = os.read(fd, 1024) - stdout.write(data) - return data - - pty.spawn(sys.argv[1:], read) - - from .console import Console - - console = Console(record=True) - - stdout_result = stdout.getvalue().decode("utf-8") - print(stdout_result) - - for line in decoder.decode(stdout_result): - console.print(line) - - console.save_html("stdout.html") diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/rich/bar.py b/venv/lib/python3.11/site-packages/pip/_vendor/rich/bar.py deleted file mode 100644 index ed86a55..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/rich/bar.py +++ /dev/null @@ -1,94 +0,0 @@ -from typing import Optional, Union - -from .color import Color -from .console import Console, ConsoleOptions, RenderResult -from .jupyter import JupyterMixin -from .measure import Measurement -from .segment import Segment -from .style import Style - -# There are left-aligned characters for 1/8 to 7/8, but -# the right-aligned characters exist only for 1/8 and 4/8. -BEGIN_BLOCK_ELEMENTS = ["█", "█", "█", "▐", "▐", "▐", "▕", "▕"] -END_BLOCK_ELEMENTS = [" ", "▏", "▎", "▍", "▌", "▋", "▊", "▉"] -FULL_BLOCK = "█" - - -class Bar(JupyterMixin): - """Renders a solid block bar. - - Args: - size (float): Value for the end of the bar. - begin (float): Begin point (between 0 and size, inclusive). - end (float): End point (between 0 and size, inclusive). - width (int, optional): Width of the bar, or ``None`` for maximum width. Defaults to None. - color (Union[Color, str], optional): Color of the bar. Defaults to "default". - bgcolor (Union[Color, str], optional): Color of bar background. Defaults to "default". - """ - - def __init__( - self, - size: float, - begin: float, - end: float, - *, - width: Optional[int] = None, - color: Union[Color, str] = "default", - bgcolor: Union[Color, str] = "default", - ): - self.size = size - self.begin = max(begin, 0) - self.end = min(end, size) - self.width = width - self.style = Style(color=color, bgcolor=bgcolor) - - def __repr__(self) -> str: - return f"Bar({self.size}, {self.begin}, {self.end})" - - def __rich_console__( - self, console: Console, options: ConsoleOptions - ) -> RenderResult: - - width = min( - self.width if self.width is not None else options.max_width, - options.max_width, - ) - - if self.begin >= self.end: - yield Segment(" " * width, self.style) - yield Segment.line() - return - - prefix_complete_eights = int(width * 8 * self.begin / self.size) - prefix_bar_count = prefix_complete_eights // 8 - prefix_eights_count = prefix_complete_eights % 8 - - body_complete_eights = int(width * 8 * self.end / self.size) - body_bar_count = body_complete_eights // 8 - body_eights_count = body_complete_eights % 8 - - # When start and end fall into the same cell, we ideally should render - # a symbol that's "center-aligned", but there is no good symbol in Unicode. - # In this case, we fall back to right-aligned block symbol for simplicity. - - prefix = " " * prefix_bar_count - if prefix_eights_count: - prefix += BEGIN_BLOCK_ELEMENTS[prefix_eights_count] - - body = FULL_BLOCK * body_bar_count - if body_eights_count: - body += END_BLOCK_ELEMENTS[body_eights_count] - - suffix = " " * (width - len(body)) - - yield Segment(prefix + body[len(prefix) :] + suffix, self.style) - yield Segment.line() - - def __rich_measure__( - self, console: Console, options: ConsoleOptions - ) -> Measurement: - return ( - Measurement(self.width, self.width) - if self.width is not None - else Measurement(4, options.max_width) - ) diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/rich/box.py b/venv/lib/python3.11/site-packages/pip/_vendor/rich/box.py deleted file mode 100644 index 97d2a94..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/rich/box.py +++ /dev/null @@ -1,517 +0,0 @@ -import sys -from typing import TYPE_CHECKING, Iterable, List - -if sys.version_info >= (3, 8): - from typing import Literal -else: - from pip._vendor.typing_extensions import Literal # pragma: no cover - - -from ._loop import loop_last - -if TYPE_CHECKING: - from pip._vendor.rich.console import ConsoleOptions - - -class Box: - """Defines characters to render boxes. - - ┌─┬┐ top - │ ││ head - ├─┼┤ head_row - │ ││ mid - ├─┼┤ row - ├─┼┤ foot_row - │ ││ foot - └─┴┘ bottom - - Args: - box (str): Characters making up box. - ascii (bool, optional): True if this box uses ascii characters only. Default is False. - """ - - def __init__(self, box: str, *, ascii: bool = False) -> None: - self._box = box - self.ascii = ascii - line1, line2, line3, line4, line5, line6, line7, line8 = box.splitlines() - # top - self.top_left, self.top, self.top_divider, self.top_right = iter(line1) - # head - self.head_left, _, self.head_vertical, self.head_right = iter(line2) - # head_row - ( - self.head_row_left, - self.head_row_horizontal, - self.head_row_cross, - self.head_row_right, - ) = iter(line3) - - # mid - self.mid_left, _, self.mid_vertical, self.mid_right = iter(line4) - # row - self.row_left, self.row_horizontal, self.row_cross, self.row_right = iter(line5) - # foot_row - ( - self.foot_row_left, - self.foot_row_horizontal, - self.foot_row_cross, - self.foot_row_right, - ) = iter(line6) - # foot - self.foot_left, _, self.foot_vertical, self.foot_right = iter(line7) - # bottom - self.bottom_left, self.bottom, self.bottom_divider, self.bottom_right = iter( - line8 - ) - - def __repr__(self) -> str: - return "Box(...)" - - def __str__(self) -> str: - return self._box - - def substitute(self, options: "ConsoleOptions", safe: bool = True) -> "Box": - """Substitute this box for another if it won't render due to platform issues. - - Args: - options (ConsoleOptions): Console options used in rendering. - safe (bool, optional): Substitute this for another Box if there are known problems - displaying on the platform (currently only relevant on Windows). Default is True. - - Returns: - Box: A different Box or the same Box. - """ - box = self - if options.legacy_windows and safe: - box = LEGACY_WINDOWS_SUBSTITUTIONS.get(box, box) - if options.ascii_only and not box.ascii: - box = ASCII - return box - - def get_plain_headed_box(self) -> "Box": - """If this box uses special characters for the borders of the header, then - return the equivalent box that does not. - - Returns: - Box: The most similar Box that doesn't use header-specific box characters. - If the current Box already satisfies this criterion, then it's returned. - """ - return PLAIN_HEADED_SUBSTITUTIONS.get(self, self) - - def get_top(self, widths: Iterable[int]) -> str: - """Get the top of a simple box. - - Args: - widths (List[int]): Widths of columns. - - Returns: - str: A string of box characters. - """ - - parts: List[str] = [] - append = parts.append - append(self.top_left) - for last, width in loop_last(widths): - append(self.top * width) - if not last: - append(self.top_divider) - append(self.top_right) - return "".join(parts) - - def get_row( - self, - widths: Iterable[int], - level: Literal["head", "row", "foot", "mid"] = "row", - edge: bool = True, - ) -> str: - """Get the top of a simple box. - - Args: - width (List[int]): Widths of columns. - - Returns: - str: A string of box characters. - """ - if level == "head": - left = self.head_row_left - horizontal = self.head_row_horizontal - cross = self.head_row_cross - right = self.head_row_right - elif level == "row": - left = self.row_left - horizontal = self.row_horizontal - cross = self.row_cross - right = self.row_right - elif level == "mid": - left = self.mid_left - horizontal = " " - cross = self.mid_vertical - right = self.mid_right - elif level == "foot": - left = self.foot_row_left - horizontal = self.foot_row_horizontal - cross = self.foot_row_cross - right = self.foot_row_right - else: - raise ValueError("level must be 'head', 'row' or 'foot'") - - parts: List[str] = [] - append = parts.append - if edge: - append(left) - for last, width in loop_last(widths): - append(horizontal * width) - if not last: - append(cross) - if edge: - append(right) - return "".join(parts) - - def get_bottom(self, widths: Iterable[int]) -> str: - """Get the bottom of a simple box. - - Args: - widths (List[int]): Widths of columns. - - Returns: - str: A string of box characters. - """ - - parts: List[str] = [] - append = parts.append - append(self.bottom_left) - for last, width in loop_last(widths): - append(self.bottom * width) - if not last: - append(self.bottom_divider) - append(self.bottom_right) - return "".join(parts) - - -ASCII: Box = Box( - """\ -+--+ -| || -|-+| -| || -|-+| -|-+| -| || -+--+ -""", - ascii=True, -) - -ASCII2: Box = Box( - """\ -+-++ -| || -+-++ -| || -+-++ -+-++ -| || -+-++ -""", - ascii=True, -) - -ASCII_DOUBLE_HEAD: Box = Box( - """\ -+-++ -| || -+=++ -| || -+-++ -+-++ -| || -+-++ -""", - ascii=True, -) - -SQUARE: Box = Box( - """\ -┌─┬┐ -│ ││ -├─┼┤ -│ ││ -├─┼┤ -├─┼┤ -│ ││ -└─┴┘ -""" -) - -SQUARE_DOUBLE_HEAD: Box = Box( - """\ -┌─┬┐ -│ ││ -╞═╪╡ -│ ││ -├─┼┤ -├─┼┤ -│ ││ -└─┴┘ -""" -) - -MINIMAL: Box = Box( - """\ - ╷ - │ -╶─┼╴ - │ -╶─┼╴ -╶─┼╴ - │ - ╵ -""" -) - - -MINIMAL_HEAVY_HEAD: Box = Box( - """\ - ╷ - │ -╺━┿╸ - │ -╶─┼╴ -╶─┼╴ - │ - ╵ -""" -) - -MINIMAL_DOUBLE_HEAD: Box = Box( - """\ - ╷ - │ - ═╪ - │ - ─┼ - ─┼ - │ - ╵ -""" -) - - -SIMPLE: Box = Box( - """\ - - - ── - - - ── - - -""" -) - -SIMPLE_HEAD: Box = Box( - """\ - - - ── - - - - - -""" -) - - -SIMPLE_HEAVY: Box = Box( - """\ - - - ━━ - - - ━━ - - -""" -) - - -HORIZONTALS: Box = Box( - """\ - ── - - ── - - ── - ── - - ── -""" -) - -ROUNDED: Box = Box( - """\ -╭─┬╮ -│ ││ -├─┼┤ -│ ││ -├─┼┤ -├─┼┤ -│ ││ -╰─┴╯ -""" -) - -HEAVY: Box = Box( - """\ -┏━┳┓ -┃ ┃┃ -┣━╋┫ -┃ ┃┃ -┣━╋┫ -┣━╋┫ -┃ ┃┃ -┗━┻┛ -""" -) - -HEAVY_EDGE: Box = Box( - """\ -┏━┯┓ -┃ │┃ -┠─┼┨ -┃ │┃ -┠─┼┨ -┠─┼┨ -┃ │┃ -┗━┷┛ -""" -) - -HEAVY_HEAD: Box = Box( - """\ -┏━┳┓ -┃ ┃┃ -┡━╇┩ -│ ││ -├─┼┤ -├─┼┤ -│ ││ -└─┴┘ -""" -) - -DOUBLE: Box = Box( - """\ -╔═╦╗ -║ ║║ -╠═╬╣ -║ ║║ -╠═╬╣ -╠═╬╣ -║ ║║ -╚═╩╝ -""" -) - -DOUBLE_EDGE: Box = Box( - """\ -╔═╤╗ -║ │║ -╟─┼╢ -║ │║ -╟─┼╢ -╟─┼╢ -║ │║ -╚═╧╝ -""" -) - -MARKDOWN: Box = Box( - """\ - -| || -|-|| -| || -|-|| -|-|| -| || - -""", - ascii=True, -) - -# Map Boxes that don't render with raster fonts on to equivalent that do -LEGACY_WINDOWS_SUBSTITUTIONS = { - ROUNDED: SQUARE, - MINIMAL_HEAVY_HEAD: MINIMAL, - SIMPLE_HEAVY: SIMPLE, - HEAVY: SQUARE, - HEAVY_EDGE: SQUARE, - HEAVY_HEAD: SQUARE, -} - -# Map headed boxes to their headerless equivalents -PLAIN_HEADED_SUBSTITUTIONS = { - HEAVY_HEAD: SQUARE, - SQUARE_DOUBLE_HEAD: SQUARE, - MINIMAL_DOUBLE_HEAD: MINIMAL, - MINIMAL_HEAVY_HEAD: MINIMAL, - ASCII_DOUBLE_HEAD: ASCII2, -} - - -if __name__ == "__main__": # pragma: no cover - - from pip._vendor.rich.columns import Columns - from pip._vendor.rich.panel import Panel - - from . import box as box - from .console import Console - from .table import Table - from .text import Text - - console = Console(record=True) - - BOXES = [ - "ASCII", - "ASCII2", - "ASCII_DOUBLE_HEAD", - "SQUARE", - "SQUARE_DOUBLE_HEAD", - "MINIMAL", - "MINIMAL_HEAVY_HEAD", - "MINIMAL_DOUBLE_HEAD", - "SIMPLE", - "SIMPLE_HEAD", - "SIMPLE_HEAVY", - "HORIZONTALS", - "ROUNDED", - "HEAVY", - "HEAVY_EDGE", - "HEAVY_HEAD", - "DOUBLE", - "DOUBLE_EDGE", - "MARKDOWN", - ] - - console.print(Panel("[bold green]Box Constants", style="green"), justify="center") - console.print() - - columns = Columns(expand=True, padding=2) - for box_name in sorted(BOXES): - table = Table( - show_footer=True, style="dim", border_style="not dim", expand=True - ) - table.add_column("Header 1", "Footer 1") - table.add_column("Header 2", "Footer 2") - table.add_row("Cell", "Cell") - table.add_row("Cell", "Cell") - table.box = getattr(box, box_name) - table.title = Text(f"box.{box_name}", style="magenta") - columns.add_renderable(table) - console.print(columns) - - # console.save_svg("box.svg") diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/rich/cells.py b/venv/lib/python3.11/site-packages/pip/_vendor/rich/cells.py deleted file mode 100644 index 9354f9e..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/rich/cells.py +++ /dev/null @@ -1,154 +0,0 @@ -import re -from functools import lru_cache -from typing import Callable, List - -from ._cell_widths import CELL_WIDTHS - -# Regex to match sequence of the most common character ranges -_is_single_cell_widths = re.compile("^[\u0020-\u006f\u00a0\u02ff\u0370-\u0482]*$").match - - -@lru_cache(4096) -def cached_cell_len(text: str) -> int: - """Get the number of cells required to display text. - - This method always caches, which may use up a lot of memory. It is recommended to use - `cell_len` over this method. - - Args: - text (str): Text to display. - - Returns: - int: Get the number of cells required to display text. - """ - _get_size = get_character_cell_size - total_size = sum(_get_size(character) for character in text) - return total_size - - -def cell_len(text: str, _cell_len: Callable[[str], int] = cached_cell_len) -> int: - """Get the number of cells required to display text. - - Args: - text (str): Text to display. - - Returns: - int: Get the number of cells required to display text. - """ - if len(text) < 512: - return _cell_len(text) - _get_size = get_character_cell_size - total_size = sum(_get_size(character) for character in text) - return total_size - - -@lru_cache(maxsize=4096) -def get_character_cell_size(character: str) -> int: - """Get the cell size of a character. - - Args: - character (str): A single character. - - Returns: - int: Number of cells (0, 1 or 2) occupied by that character. - """ - return _get_codepoint_cell_size(ord(character)) - - -@lru_cache(maxsize=4096) -def _get_codepoint_cell_size(codepoint: int) -> int: - """Get the cell size of a character. - - Args: - codepoint (int): Codepoint of a character. - - Returns: - int: Number of cells (0, 1 or 2) occupied by that character. - """ - - _table = CELL_WIDTHS - lower_bound = 0 - upper_bound = len(_table) - 1 - index = (lower_bound + upper_bound) // 2 - while True: - start, end, width = _table[index] - if codepoint < start: - upper_bound = index - 1 - elif codepoint > end: - lower_bound = index + 1 - else: - return 0 if width == -1 else width - if upper_bound < lower_bound: - break - index = (lower_bound + upper_bound) // 2 - return 1 - - -def set_cell_size(text: str, total: int) -> str: - """Set the length of a string to fit within given number of cells.""" - - if _is_single_cell_widths(text): - size = len(text) - if size < total: - return text + " " * (total - size) - return text[:total] - - if total <= 0: - return "" - cell_size = cell_len(text) - if cell_size == total: - return text - if cell_size < total: - return text + " " * (total - cell_size) - - start = 0 - end = len(text) - - # Binary search until we find the right size - while True: - pos = (start + end) // 2 - before = text[: pos + 1] - before_len = cell_len(before) - if before_len == total + 1 and cell_len(before[-1]) == 2: - return before[:-1] + " " - if before_len == total: - return before - if before_len > total: - end = pos - else: - start = pos - - -# TODO: This is inefficient -# TODO: This might not work with CWJ type characters -def chop_cells(text: str, max_size: int, position: int = 0) -> List[str]: - """Break text in to equal (cell) length strings, returning the characters in reverse - order""" - _get_character_cell_size = get_character_cell_size - characters = [ - (character, _get_character_cell_size(character)) for character in text - ] - total_size = position - lines: List[List[str]] = [[]] - append = lines[-1].append - - for character, size in reversed(characters): - if total_size + size > max_size: - lines.append([character]) - append = lines[-1].append - total_size = size - else: - total_size += size - append(character) - - return ["".join(line) for line in lines] - - -if __name__ == "__main__": # pragma: no cover - - print(get_character_cell_size("😽")) - for line in chop_cells("""这是对亚洲语言支持的测试。面对模棱两可的想法，拒绝猜测的诱惑。""", 8): - print(line) - for n in range(80, 1, -1): - print(set_cell_size("""这是对亚洲语言支持的测试。面对模棱两可的想法，拒绝猜测的诱惑。""", n) + "|") - print("x" * n) diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/rich/color.py b/venv/lib/python3.11/site-packages/pip/_vendor/rich/color.py deleted file mode 100644 index dfe4559..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/rich/color.py +++ /dev/null @@ -1,622 +0,0 @@ -import platform -import re -from colorsys import rgb_to_hls -from enum import IntEnum -from functools import lru_cache -from typing import TYPE_CHECKING, NamedTuple, Optional, Tuple - -from ._palettes import EIGHT_BIT_PALETTE, STANDARD_PALETTE, WINDOWS_PALETTE -from .color_triplet import ColorTriplet -from .repr import Result, rich_repr -from .terminal_theme import DEFAULT_TERMINAL_THEME - -if TYPE_CHECKING: # pragma: no cover - from .terminal_theme import TerminalTheme - from .text import Text - - -WINDOWS = platform.system() == "Windows" - - -class ColorSystem(IntEnum): - """One of the 3 color system supported by terminals.""" - - STANDARD = 1 - EIGHT_BIT = 2 - TRUECOLOR = 3 - WINDOWS = 4 - - def __repr__(self) -> str: - return f"ColorSystem.{self.name}" - - def __str__(self) -> str: - return repr(self) - - -class ColorType(IntEnum): - """Type of color stored in Color class.""" - - DEFAULT = 0 - STANDARD = 1 - EIGHT_BIT = 2 - TRUECOLOR = 3 - WINDOWS = 4 - - def __repr__(self) -> str: - return f"ColorType.{self.name}" - - -ANSI_COLOR_NAMES = { - "black": 0, - "red": 1, - "green": 2, - "yellow": 3, - "blue": 4, - "magenta": 5, - "cyan": 6, - "white": 7, - "bright_black": 8, - "bright_red": 9, - "bright_green": 10, - "bright_yellow": 11, - "bright_blue": 12, - "bright_magenta": 13, - "bright_cyan": 14, - "bright_white": 15, - "grey0": 16, - "gray0": 16, - "navy_blue": 17, - "dark_blue": 18, - "blue3": 20, - "blue1": 21, - "dark_green": 22, - "deep_sky_blue4": 25, - "dodger_blue3": 26, - "dodger_blue2": 27, - "green4": 28, - "spring_green4": 29, - "turquoise4": 30, - "deep_sky_blue3": 32, - "dodger_blue1": 33, - "green3": 40, - "spring_green3": 41, - "dark_cyan": 36, - "light_sea_green": 37, - "deep_sky_blue2": 38, - "deep_sky_blue1": 39, - "spring_green2": 47, - "cyan3": 43, - "dark_turquoise": 44, - "turquoise2": 45, - "green1": 46, - "spring_green1": 48, - "medium_spring_green": 49, - "cyan2": 50, - "cyan1": 51, - "dark_red": 88, - "deep_pink4": 125, - "purple4": 55, - "purple3": 56, - "blue_violet": 57, - "orange4": 94, - "grey37": 59, - "gray37": 59, - "medium_purple4": 60, - "slate_blue3": 62, - "royal_blue1": 63, - "chartreuse4": 64, - "dark_sea_green4": 71, - "pale_turquoise4": 66, - "steel_blue": 67, - "steel_blue3": 68, - "cornflower_blue": 69, - "chartreuse3": 76, - "cadet_blue": 73, - "sky_blue3": 74, - "steel_blue1": 81, - "pale_green3": 114, - "sea_green3": 78, - "aquamarine3": 79, - "medium_turquoise": 80, - "chartreuse2": 112, - "sea_green2": 83, - "sea_green1": 85, - "aquamarine1": 122, - "dark_slate_gray2": 87, - "dark_magenta": 91, - "dark_violet": 128, - "purple": 129, - "light_pink4": 95, - "plum4": 96, - "medium_purple3": 98, - "slate_blue1": 99, - "yellow4": 106, - "wheat4": 101, - "grey53": 102, - "gray53": 102, - "light_slate_grey": 103, - "light_slate_gray": 103, - "medium_purple": 104, - "light_slate_blue": 105, - "dark_olive_green3": 149, - "dark_sea_green": 108, - "light_sky_blue3": 110, - "sky_blue2": 111, - "dark_sea_green3": 150, - "dark_slate_gray3": 116, - "sky_blue1": 117, - "chartreuse1": 118, - "light_green": 120, - "pale_green1": 156, - "dark_slate_gray1": 123, - "red3": 160, - "medium_violet_red": 126, - "magenta3": 164, - "dark_orange3": 166, - "indian_red": 167, - "hot_pink3": 168, - "medium_orchid3": 133, - "medium_orchid": 134, - "medium_purple2": 140, - "dark_goldenrod": 136, - "light_salmon3": 173, - "rosy_brown": 138, - "grey63": 139, - "gray63": 139, - "medium_purple1": 141, - "gold3": 178, - "dark_khaki": 143, - "navajo_white3": 144, - "grey69": 145, - "gray69": 145, - "light_steel_blue3": 146, - "light_steel_blue": 147, - "yellow3": 184, - "dark_sea_green2": 157, - "light_cyan3": 152, - "light_sky_blue1": 153, - "green_yellow": 154, - "dark_olive_green2": 155, - "dark_sea_green1": 193, - "pale_turquoise1": 159, - "deep_pink3": 162, - "magenta2": 200, - "hot_pink2": 169, - "orchid": 170, - "medium_orchid1": 207, - "orange3": 172, - "light_pink3": 174, - "pink3": 175, - "plum3": 176, - "violet": 177, - "light_goldenrod3": 179, - "tan": 180, - "misty_rose3": 181, - "thistle3": 182, - "plum2": 183, - "khaki3": 185, - "light_goldenrod2": 222, - "light_yellow3": 187, - "grey84": 188, - "gray84": 188, - "light_steel_blue1": 189, - "yellow2": 190, - "dark_olive_green1": 192, - "honeydew2": 194, - "light_cyan1": 195, - "red1": 196, - "deep_pink2": 197, - "deep_pink1": 199, - "magenta1": 201, - "orange_red1": 202, - "indian_red1": 204, - "hot_pink": 206, - "dark_orange": 208, - "salmon1": 209, - "light_coral": 210, - "pale_violet_red1": 211, - "orchid2": 212, - "orchid1": 213, - "orange1": 214, - "sandy_brown": 215, - "light_salmon1": 216, - "light_pink1": 217, - "pink1": 218, - "plum1": 219, - "gold1": 220, - "navajo_white1": 223, - "misty_rose1": 224, - "thistle1": 225, - "yellow1": 226, - "light_goldenrod1": 227, - "khaki1": 228, - "wheat1": 229, - "cornsilk1": 230, - "grey100": 231, - "gray100": 231, - "grey3": 232, - "gray3": 232, - "grey7": 233, - "gray7": 233, - "grey11": 234, - "gray11": 234, - "grey15": 235, - "gray15": 235, - "grey19": 236, - "gray19": 236, - "grey23": 237, - "gray23": 237, - "grey27": 238, - "gray27": 238, - "grey30": 239, - "gray30": 239, - "grey35": 240, - "gray35": 240, - "grey39": 241, - "gray39": 241, - "grey42": 242, - "gray42": 242, - "grey46": 243, - "gray46": 243, - "grey50": 244, - "gray50": 244, - "grey54": 245, - "gray54": 245, - "grey58": 246, - "gray58": 246, - "grey62": 247, - "gray62": 247, - "grey66": 248, - "gray66": 248, - "grey70": 249, - "gray70": 249, - "grey74": 250, - "gray74": 250, - "grey78": 251, - "gray78": 251, - "grey82": 252, - "gray82": 252, - "grey85": 253, - "gray85": 253, - "grey89": 254, - "gray89": 254, - "grey93": 255, - "gray93": 255, -} - - -class ColorParseError(Exception): - """The color could not be parsed.""" - - -RE_COLOR = re.compile( - r"""^ -\#([0-9a-f]{6})$| -color\(([0-9]{1,3})\)$| -rgb\(([\d\s,]+)\)$ -""", - re.VERBOSE, -) - - -@rich_repr -class Color(NamedTuple): - """Terminal color definition.""" - - name: str - """The name of the color (typically the input to Color.parse).""" - type: ColorType - """The type of the color.""" - number: Optional[int] = None - """The color number, if a standard color, or None.""" - triplet: Optional[ColorTriplet] = None - """A triplet of color components, if an RGB color.""" - - def __rich__(self) -> "Text": - """Displays the actual color if Rich printed.""" - from .style import Style - from .text import Text - - return Text.assemble( - f"", - ) - - def __rich_repr__(self) -> Result: - yield self.name - yield self.type - yield "number", self.number, None - yield "triplet", self.triplet, None - - @property - def system(self) -> ColorSystem: - """Get the native color system for this color.""" - if self.type == ColorType.DEFAULT: - return ColorSystem.STANDARD - return ColorSystem(int(self.type)) - - @property - def is_system_defined(self) -> bool: - """Check if the color is ultimately defined by the system.""" - return self.system not in (ColorSystem.EIGHT_BIT, ColorSystem.TRUECOLOR) - - @property - def is_default(self) -> bool: - """Check if the color is a default color.""" - return self.type == ColorType.DEFAULT - - def get_truecolor( - self, theme: Optional["TerminalTheme"] = None, foreground: bool = True - ) -> ColorTriplet: - """Get an equivalent color triplet for this color. - - Args: - theme (TerminalTheme, optional): Optional terminal theme, or None to use default. Defaults to None. - foreground (bool, optional): True for a foreground color, or False for background. Defaults to True. - - Returns: - ColorTriplet: A color triplet containing RGB components. - """ - - if theme is None: - theme = DEFAULT_TERMINAL_THEME - if self.type == ColorType.TRUECOLOR: - assert self.triplet is not None - return self.triplet - elif self.type == ColorType.EIGHT_BIT: - assert self.number is not None - return EIGHT_BIT_PALETTE[self.number] - elif self.type == ColorType.STANDARD: - assert self.number is not None - return theme.ansi_colors[self.number] - elif self.type == ColorType.WINDOWS: - assert self.number is not None - return WINDOWS_PALETTE[self.number] - else: # self.type == ColorType.DEFAULT: - assert self.number is None - return theme.foreground_color if foreground else theme.background_color - - @classmethod - def from_ansi(cls, number: int) -> "Color": - """Create a Color number from it's 8-bit ansi number. - - Args: - number (int): A number between 0-255 inclusive. - - Returns: - Color: A new Color instance. - """ - return cls( - name=f"color({number})", - type=(ColorType.STANDARD if number < 16 else ColorType.EIGHT_BIT), - number=number, - ) - - @classmethod - def from_triplet(cls, triplet: "ColorTriplet") -> "Color": - """Create a truecolor RGB color from a triplet of values. - - Args: - triplet (ColorTriplet): A color triplet containing red, green and blue components. - - Returns: - Color: A new color object. - """ - return cls(name=triplet.hex, type=ColorType.TRUECOLOR, triplet=triplet) - - @classmethod - def from_rgb(cls, red: float, green: float, blue: float) -> "Color": - """Create a truecolor from three color components in the range(0->255). - - Args: - red (float): Red component in range 0-255. - green (float): Green component in range 0-255. - blue (float): Blue component in range 0-255. - - Returns: - Color: A new color object. - """ - return cls.from_triplet(ColorTriplet(int(red), int(green), int(blue))) - - @classmethod - def default(cls) -> "Color": - """Get a Color instance representing the default color. - - Returns: - Color: Default color. - """ - return cls(name="default", type=ColorType.DEFAULT) - - @classmethod - @lru_cache(maxsize=1024) - def parse(cls, color: str) -> "Color": - """Parse a color definition.""" - original_color = color - color = color.lower().strip() - - if color == "default": - return cls(color, type=ColorType.DEFAULT) - - color_number = ANSI_COLOR_NAMES.get(color) - if color_number is not None: - return cls( - color, - type=(ColorType.STANDARD if color_number < 16 else ColorType.EIGHT_BIT), - number=color_number, - ) - - color_match = RE_COLOR.match(color) - if color_match is None: - raise ColorParseError(f"{original_color!r} is not a valid color") - - color_24, color_8, color_rgb = color_match.groups() - if color_24: - triplet = ColorTriplet( - int(color_24[0:2], 16), int(color_24[2:4], 16), int(color_24[4:6], 16) - ) - return cls(color, ColorType.TRUECOLOR, triplet=triplet) - - elif color_8: - number = int(color_8) - if number > 255: - raise ColorParseError(f"color number must be <= 255 in {color!r}") - return cls( - color, - type=(ColorType.STANDARD if number < 16 else ColorType.EIGHT_BIT), - number=number, - ) - - else: # color_rgb: - components = color_rgb.split(",") - if len(components) != 3: - raise ColorParseError( - f"expected three components in {original_color!r}" - ) - red, green, blue = components - triplet = ColorTriplet(int(red), int(green), int(blue)) - if not all(component <= 255 for component in triplet): - raise ColorParseError( - f"color components must be <= 255 in {original_color!r}" - ) - return cls(color, ColorType.TRUECOLOR, triplet=triplet) - - @lru_cache(maxsize=1024) - def get_ansi_codes(self, foreground: bool = True) -> Tuple[str, ...]: - """Get the ANSI escape codes for this color.""" - _type = self.type - if _type == ColorType.DEFAULT: - return ("39" if foreground else "49",) - - elif _type == ColorType.WINDOWS: - number = self.number - assert number is not None - fore, back = (30, 40) if number < 8 else (82, 92) - return (str(fore + number if foreground else back + number),) - - elif _type == ColorType.STANDARD: - number = self.number - assert number is not None - fore, back = (30, 40) if number < 8 else (82, 92) - return (str(fore + number if foreground else back + number),) - - elif _type == ColorType.EIGHT_BIT: - assert self.number is not None - return ("38" if foreground else "48", "5", str(self.number)) - - else: # self.standard == ColorStandard.TRUECOLOR: - assert self.triplet is not None - red, green, blue = self.triplet - return ("38" if foreground else "48", "2", str(red), str(green), str(blue)) - - @lru_cache(maxsize=1024) - def downgrade(self, system: ColorSystem) -> "Color": - """Downgrade a color system to a system with fewer colors.""" - - if self.type in (ColorType.DEFAULT, system): - return self - # Convert to 8-bit color from truecolor color - if system == ColorSystem.EIGHT_BIT and self.system == ColorSystem.TRUECOLOR: - assert self.triplet is not None - _h, l, s = rgb_to_hls(*self.triplet.normalized) - # If saturation is under 15% assume it is grayscale - if s < 0.15: - gray = round(l * 25.0) - if gray == 0: - color_number = 16 - elif gray == 25: - color_number = 231 - else: - color_number = 231 + gray - return Color(self.name, ColorType.EIGHT_BIT, number=color_number) - - red, green, blue = self.triplet - six_red = red / 95 if red < 95 else 1 + (red - 95) / 40 - six_green = green / 95 if green < 95 else 1 + (green - 95) / 40 - six_blue = blue / 95 if blue < 95 else 1 + (blue - 95) / 40 - - color_number = ( - 16 + 36 * round(six_red) + 6 * round(six_green) + round(six_blue) - ) - return Color(self.name, ColorType.EIGHT_BIT, number=color_number) - - # Convert to standard from truecolor or 8-bit - elif system == ColorSystem.STANDARD: - if self.system == ColorSystem.TRUECOLOR: - assert self.triplet is not None - triplet = self.triplet - else: # self.system == ColorSystem.EIGHT_BIT - assert self.number is not None - triplet = ColorTriplet(*EIGHT_BIT_PALETTE[self.number]) - - color_number = STANDARD_PALETTE.match(triplet) - return Color(self.name, ColorType.STANDARD, number=color_number) - - elif system == ColorSystem.WINDOWS: - if self.system == ColorSystem.TRUECOLOR: - assert self.triplet is not None - triplet = self.triplet - else: # self.system == ColorSystem.EIGHT_BIT - assert self.number is not None - if self.number < 16: - return Color(self.name, ColorType.WINDOWS, number=self.number) - triplet = ColorTriplet(*EIGHT_BIT_PALETTE[self.number]) - - color_number = WINDOWS_PALETTE.match(triplet) - return Color(self.name, ColorType.WINDOWS, number=color_number) - - return self - - -def parse_rgb_hex(hex_color: str) -> ColorTriplet: - """Parse six hex characters in to RGB triplet.""" - assert len(hex_color) == 6, "must be 6 characters" - color = ColorTriplet( - int(hex_color[0:2], 16), int(hex_color[2:4], 16), int(hex_color[4:6], 16) - ) - return color - - -def blend_rgb( - color1: ColorTriplet, color2: ColorTriplet, cross_fade: float = 0.5 -) -> ColorTriplet: - """Blend one RGB color in to another.""" - r1, g1, b1 = color1 - r2, g2, b2 = color2 - new_color = ColorTriplet( - int(r1 + (r2 - r1) * cross_fade), - int(g1 + (g2 - g1) * cross_fade), - int(b1 + (b2 - b1) * cross_fade), - ) - return new_color - - -if __name__ == "__main__": # pragma: no cover - - from .console import Console - from .table import Table - from .text import Text - - console = Console() - - table = Table(show_footer=False, show_edge=True) - table.add_column("Color", width=10, overflow="ellipsis") - table.add_column("Number", justify="right", style="yellow") - table.add_column("Name", style="green") - table.add_column("Hex", style="blue") - table.add_column("RGB", style="magenta") - - colors = sorted((v, k) for k, v in ANSI_COLOR_NAMES.items()) - for color_number, name in colors: - if "grey" in name: - continue - color_cell = Text(" " * 10, style=f"on {name}") - if color_number < 16: - table.add_row(color_cell, f"{color_number}", Text(f'"{name}"')) - else: - color = EIGHT_BIT_PALETTE[color_number] # type: ignore[has-type] - table.add_row( - color_cell, str(color_number), Text(f'"{name}"'), color.hex, color.rgb - ) - - console.print(table) diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/rich/color_triplet.py b/venv/lib/python3.11/site-packages/pip/_vendor/rich/color_triplet.py deleted file mode 100644 index 02cab32..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/rich/color_triplet.py +++ /dev/null @@ -1,38 +0,0 @@ -from typing import NamedTuple, Tuple - - -class ColorTriplet(NamedTuple): - """The red, green, and blue components of a color.""" - - red: int - """Red component in 0 to 255 range.""" - green: int - """Green component in 0 to 255 range.""" - blue: int - """Blue component in 0 to 255 range.""" - - @property - def hex(self) -> str: - """get the color triplet in CSS style.""" - red, green, blue = self - return f"#{red:02x}{green:02x}{blue:02x}" - - @property - def rgb(self) -> str: - """The color in RGB format. - - Returns: - str: An rgb color, e.g. ``"rgb(100,23,255)"``. - """ - red, green, blue = self - return f"rgb({red},{green},{blue})" - - @property - def normalized(self) -> Tuple[float, float, float]: - """Convert components into floats between 0 and 1. - - Returns: - Tuple[float, float, float]: A tuple of three normalized colour components. - """ - red, green, blue = self - return red / 255.0, green / 255.0, blue / 255.0 diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/rich/columns.py b/venv/lib/python3.11/site-packages/pip/_vendor/rich/columns.py deleted file mode 100644 index 669a3a7..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/rich/columns.py +++ /dev/null @@ -1,187 +0,0 @@ -from collections import defaultdict -from itertools import chain -from operator import itemgetter -from typing import Dict, Iterable, List, Optional, Tuple - -from .align import Align, AlignMethod -from .console import Console, ConsoleOptions, RenderableType, RenderResult -from .constrain import Constrain -from .measure import Measurement -from .padding import Padding, PaddingDimensions -from .table import Table -from .text import TextType -from .jupyter import JupyterMixin - - -class Columns(JupyterMixin): - """Display renderables in neat columns. - - Args: - renderables (Iterable[RenderableType]): Any number of Rich renderables (including str). - width (int, optional): The desired width of the columns, or None to auto detect. Defaults to None. - padding (PaddingDimensions, optional): Optional padding around cells. Defaults to (0, 1). - expand (bool, optional): Expand columns to full width. Defaults to False. - equal (bool, optional): Arrange in to equal sized columns. Defaults to False. - column_first (bool, optional): Align items from top to bottom (rather than left to right). Defaults to False. - right_to_left (bool, optional): Start column from right hand side. Defaults to False. - align (str, optional): Align value ("left", "right", or "center") or None for default. Defaults to None. - title (TextType, optional): Optional title for Columns. - """ - - def __init__( - self, - renderables: Optional[Iterable[RenderableType]] = None, - padding: PaddingDimensions = (0, 1), - *, - width: Optional[int] = None, - expand: bool = False, - equal: bool = False, - column_first: bool = False, - right_to_left: bool = False, - align: Optional[AlignMethod] = None, - title: Optional[TextType] = None, - ) -> None: - self.renderables = list(renderables or []) - self.width = width - self.padding = padding - self.expand = expand - self.equal = equal - self.column_first = column_first - self.right_to_left = right_to_left - self.align: Optional[AlignMethod] = align - self.title = title - - def add_renderable(self, renderable: RenderableType) -> None: - """Add a renderable to the columns. - - Args: - renderable (RenderableType): Any renderable object. - """ - self.renderables.append(renderable) - - def __rich_console__( - self, console: Console, options: ConsoleOptions - ) -> RenderResult: - render_str = console.render_str - renderables = [ - render_str(renderable) if isinstance(renderable, str) else renderable - for renderable in self.renderables - ] - if not renderables: - return - _top, right, _bottom, left = Padding.unpack(self.padding) - width_padding = max(left, right) - max_width = options.max_width - widths: Dict[int, int] = defaultdict(int) - column_count = len(renderables) - - get_measurement = Measurement.get - renderable_widths = [ - get_measurement(console, options, renderable).maximum - for renderable in renderables - ] - if self.equal: - renderable_widths = [max(renderable_widths)] * len(renderable_widths) - - def iter_renderables( - column_count: int, - ) -> Iterable[Tuple[int, Optional[RenderableType]]]: - item_count = len(renderables) - if self.column_first: - width_renderables = list(zip(renderable_widths, renderables)) - - column_lengths: List[int] = [item_count // column_count] * column_count - for col_no in range(item_count % column_count): - column_lengths[col_no] += 1 - - row_count = (item_count + column_count - 1) // column_count - cells = [[-1] * column_count for _ in range(row_count)] - row = col = 0 - for index in range(item_count): - cells[row][col] = index - column_lengths[col] -= 1 - if column_lengths[col]: - row += 1 - else: - col += 1 - row = 0 - for index in chain.from_iterable(cells): - if index == -1: - break - yield width_renderables[index] - else: - yield from zip(renderable_widths, renderables) - # Pad odd elements with spaces - if item_count % column_count: - for _ in range(column_count - (item_count % column_count)): - yield 0, None - - table = Table.grid(padding=self.padding, collapse_padding=True, pad_edge=False) - table.expand = self.expand - table.title = self.title - - if self.width is not None: - column_count = (max_width) // (self.width + width_padding) - for _ in range(column_count): - table.add_column(width=self.width) - else: - while column_count > 1: - widths.clear() - column_no = 0 - for renderable_width, _ in iter_renderables(column_count): - widths[column_no] = max(widths[column_no], renderable_width) - total_width = sum(widths.values()) + width_padding * ( - len(widths) - 1 - ) - if total_width > max_width: - column_count = len(widths) - 1 - break - else: - column_no = (column_no + 1) % column_count - else: - break - - get_renderable = itemgetter(1) - _renderables = [ - get_renderable(_renderable) - for _renderable in iter_renderables(column_count) - ] - if self.equal: - _renderables = [ - None - if renderable is None - else Constrain(renderable, renderable_widths[0]) - for renderable in _renderables - ] - if self.align: - align = self.align - _Align = Align - _renderables = [ - None if renderable is None else _Align(renderable, align) - for renderable in _renderables - ] - - right_to_left = self.right_to_left - add_row = table.add_row - for start in range(0, len(_renderables), column_count): - row = _renderables[start : start + column_count] - if right_to_left: - row = row[::-1] - add_row(*row) - yield table - - -if __name__ == "__main__": # pragma: no cover - import os - - console = Console() - - files = [f"{i} {s}" for i, s in enumerate(sorted(os.listdir()))] - columns = Columns(files, padding=(0, 1), expand=False, equal=False) - console.print(columns) - console.rule() - columns.column_first = True - console.print(columns) - columns.right_to_left = True - console.rule() - console.print(columns) diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/rich/console.py b/venv/lib/python3.11/site-packages/pip/_vendor/rich/console.py deleted file mode 100644 index e559cbb..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/rich/console.py +++ /dev/null @@ -1,2633 +0,0 @@ -import inspect -import os -import platform -import sys -import threading -import zlib -from abc import ABC, abstractmethod -from dataclasses import dataclass, field -from datetime import datetime -from functools import wraps -from getpass import getpass -from html import escape -from inspect import isclass -from itertools import islice -from math import ceil -from time import monotonic -from types import FrameType, ModuleType, TracebackType -from typing import ( - IO, - TYPE_CHECKING, - Any, - Callable, - Dict, - Iterable, - List, - Mapping, - NamedTuple, - Optional, - TextIO, - Tuple, - Type, - Union, - cast, -) - -from pip._vendor.rich._null_file import NULL_FILE - -if sys.version_info >= (3, 8): - from typing import Literal, Protocol, runtime_checkable -else: - from pip._vendor.typing_extensions import ( - Literal, - Protocol, - runtime_checkable, - ) # pragma: no cover - -from . import errors, themes -from ._emoji_replace import _emoji_replace -from ._export_format import CONSOLE_HTML_FORMAT, CONSOLE_SVG_FORMAT -from ._fileno import get_fileno -from ._log_render import FormatTimeCallable, LogRender -from .align import Align, AlignMethod -from .color import ColorSystem, blend_rgb -from .control import Control -from .emoji import EmojiVariant -from .highlighter import NullHighlighter, ReprHighlighter -from .markup import render as render_markup -from .measure import Measurement, measure_renderables -from .pager import Pager, SystemPager -from .pretty import Pretty, is_expandable -from .protocol import rich_cast -from .region import Region -from .scope import render_scope -from .screen import Screen -from .segment import Segment -from .style import Style, StyleType -from .styled import Styled -from .terminal_theme import DEFAULT_TERMINAL_THEME, SVG_EXPORT_THEME, TerminalTheme -from .text import Text, TextType -from .theme import Theme, ThemeStack - -if TYPE_CHECKING: - from ._windows import WindowsConsoleFeatures - from .live import Live - from .status import Status - -JUPYTER_DEFAULT_COLUMNS = 115 -JUPYTER_DEFAULT_LINES = 100 -WINDOWS = platform.system() == "Windows" - -HighlighterType = Callable[[Union[str, "Text"]], "Text"] -JustifyMethod = Literal["default", "left", "center", "right", "full"] -OverflowMethod = Literal["fold", "crop", "ellipsis", "ignore"] - - -class NoChange: - pass - - -NO_CHANGE = NoChange() - -try: - _STDIN_FILENO = sys.__stdin__.fileno() -except Exception: - _STDIN_FILENO = 0 -try: - _STDOUT_FILENO = sys.__stdout__.fileno() -except Exception: - _STDOUT_FILENO = 1 -try: - _STDERR_FILENO = sys.__stderr__.fileno() -except Exception: - _STDERR_FILENO = 2 - -_STD_STREAMS = (_STDIN_FILENO, _STDOUT_FILENO, _STDERR_FILENO) -_STD_STREAMS_OUTPUT = (_STDOUT_FILENO, _STDERR_FILENO) - - -_TERM_COLORS = { - "kitty": ColorSystem.EIGHT_BIT, - "256color": ColorSystem.EIGHT_BIT, - "16color": ColorSystem.STANDARD, -} - - -class ConsoleDimensions(NamedTuple): - """Size of the terminal.""" - - width: int - """The width of the console in 'cells'.""" - height: int - """The height of the console in lines.""" - - -@dataclass -class ConsoleOptions: - """Options for __rich_console__ method.""" - - size: ConsoleDimensions - """Size of console.""" - legacy_windows: bool - """legacy_windows: flag for legacy windows.""" - min_width: int - """Minimum width of renderable.""" - max_width: int - """Maximum width of renderable.""" - is_terminal: bool - """True if the target is a terminal, otherwise False.""" - encoding: str - """Encoding of terminal.""" - max_height: int - """Height of container (starts as terminal)""" - justify: Optional[JustifyMethod] = None - """Justify value override for renderable.""" - overflow: Optional[OverflowMethod] = None - """Overflow value override for renderable.""" - no_wrap: Optional[bool] = False - """Disable wrapping for text.""" - highlight: Optional[bool] = None - """Highlight override for render_str.""" - markup: Optional[bool] = None - """Enable markup when rendering strings.""" - height: Optional[int] = None - - @property - def ascii_only(self) -> bool: - """Check if renderables should use ascii only.""" - return not self.encoding.startswith("utf") - - def copy(self) -> "ConsoleOptions": - """Return a copy of the options. - - Returns: - ConsoleOptions: a copy of self. - """ - options: ConsoleOptions = ConsoleOptions.__new__(ConsoleOptions) - options.__dict__ = self.__dict__.copy() - return options - - def update( - self, - *, - width: Union[int, NoChange] = NO_CHANGE, - min_width: Union[int, NoChange] = NO_CHANGE, - max_width: Union[int, NoChange] = NO_CHANGE, - justify: Union[Optional[JustifyMethod], NoChange] = NO_CHANGE, - overflow: Union[Optional[OverflowMethod], NoChange] = NO_CHANGE, - no_wrap: Union[Optional[bool], NoChange] = NO_CHANGE, - highlight: Union[Optional[bool], NoChange] = NO_CHANGE, - markup: Union[Optional[bool], NoChange] = NO_CHANGE, - height: Union[Optional[int], NoChange] = NO_CHANGE, - ) -> "ConsoleOptions": - """Update values, return a copy.""" - options = self.copy() - if not isinstance(width, NoChange): - options.min_width = options.max_width = max(0, width) - if not isinstance(min_width, NoChange): - options.min_width = min_width - if not isinstance(max_width, NoChange): - options.max_width = max_width - if not isinstance(justify, NoChange): - options.justify = justify - if not isinstance(overflow, NoChange): - options.overflow = overflow - if not isinstance(no_wrap, NoChange): - options.no_wrap = no_wrap - if not isinstance(highlight, NoChange): - options.highlight = highlight - if not isinstance(markup, NoChange): - options.markup = markup - if not isinstance(height, NoChange): - if height is not None: - options.max_height = height - options.height = None if height is None else max(0, height) - return options - - def update_width(self, width: int) -> "ConsoleOptions": - """Update just the width, return a copy. - - Args: - width (int): New width (sets both min_width and max_width) - - Returns: - ~ConsoleOptions: New console options instance. - """ - options = self.copy() - options.min_width = options.max_width = max(0, width) - return options - - def update_height(self, height: int) -> "ConsoleOptions": - """Update the height, and return a copy. - - Args: - height (int): New height - - Returns: - ~ConsoleOptions: New Console options instance. - """ - options = self.copy() - options.max_height = options.height = height - return options - - def reset_height(self) -> "ConsoleOptions": - """Return a copy of the options with height set to ``None``. - - Returns: - ~ConsoleOptions: New console options instance. - """ - options = self.copy() - options.height = None - return options - - def update_dimensions(self, width: int, height: int) -> "ConsoleOptions": - """Update the width and height, and return a copy. - - Args: - width (int): New width (sets both min_width and max_width). - height (int): New height. - - Returns: - ~ConsoleOptions: New console options instance. - """ - options = self.copy() - options.min_width = options.max_width = max(0, width) - options.height = options.max_height = height - return options - - -@runtime_checkable -class RichCast(Protocol): - """An object that may be 'cast' to a console renderable.""" - - def __rich__( - self, - ) -> Union["ConsoleRenderable", "RichCast", str]: # pragma: no cover - ... - - -@runtime_checkable -class ConsoleRenderable(Protocol): - """An object that supports the console protocol.""" - - def __rich_console__( - self, console: "Console", options: "ConsoleOptions" - ) -> "RenderResult": # pragma: no cover - ... - - -# A type that may be rendered by Console. -RenderableType = Union[ConsoleRenderable, RichCast, str] - -# The result of calling a __rich_console__ method. -RenderResult = Iterable[Union[RenderableType, Segment]] - -_null_highlighter = NullHighlighter() - - -class CaptureError(Exception): - """An error in the Capture context manager.""" - - -class NewLine: - """A renderable to generate new line(s)""" - - def __init__(self, count: int = 1) -> None: - self.count = count - - def __rich_console__( - self, console: "Console", options: "ConsoleOptions" - ) -> Iterable[Segment]: - yield Segment("\n" * self.count) - - -class ScreenUpdate: - """Render a list of lines at a given offset.""" - - def __init__(self, lines: List[List[Segment]], x: int, y: int) -> None: - self._lines = lines - self.x = x - self.y = y - - def __rich_console__( - self, console: "Console", options: ConsoleOptions - ) -> RenderResult: - x = self.x - move_to = Control.move_to - for offset, line in enumerate(self._lines, self.y): - yield move_to(x, offset) - yield from line - - -class Capture: - """Context manager to capture the result of printing to the console. - See :meth:`~rich.console.Console.capture` for how to use. - - Args: - console (Console): A console instance to capture output. - """ - - def __init__(self, console: "Console") -> None: - self._console = console - self._result: Optional[str] = None - - def __enter__(self) -> "Capture": - self._console.begin_capture() - return self - - def __exit__( - self, - exc_type: Optional[Type[BaseException]], - exc_val: Optional[BaseException], - exc_tb: Optional[TracebackType], - ) -> None: - self._result = self._console.end_capture() - - def get(self) -> str: - """Get the result of the capture.""" - if self._result is None: - raise CaptureError( - "Capture result is not available until context manager exits." - ) - return self._result - - -class ThemeContext: - """A context manager to use a temporary theme. See :meth:`~rich.console.Console.use_theme` for usage.""" - - def __init__(self, console: "Console", theme: Theme, inherit: bool = True) -> None: - self.console = console - self.theme = theme - self.inherit = inherit - - def __enter__(self) -> "ThemeContext": - self.console.push_theme(self.theme) - return self - - def __exit__( - self, - exc_type: Optional[Type[BaseException]], - exc_val: Optional[BaseException], - exc_tb: Optional[TracebackType], - ) -> None: - self.console.pop_theme() - - -class PagerContext: - """A context manager that 'pages' content. See :meth:`~rich.console.Console.pager` for usage.""" - - def __init__( - self, - console: "Console", - pager: Optional[Pager] = None, - styles: bool = False, - links: bool = False, - ) -> None: - self._console = console - self.pager = SystemPager() if pager is None else pager - self.styles = styles - self.links = links - - def __enter__(self) -> "PagerContext": - self._console._enter_buffer() - return self - - def __exit__( - self, - exc_type: Optional[Type[BaseException]], - exc_val: Optional[BaseException], - exc_tb: Optional[TracebackType], - ) -> None: - if exc_type is None: - with self._console._lock: - buffer: List[Segment] = self._console._buffer[:] - del self._console._buffer[:] - segments: Iterable[Segment] = buffer - if not self.styles: - segments = Segment.strip_styles(segments) - elif not self.links: - segments = Segment.strip_links(segments) - content = self._console._render_buffer(segments) - self.pager.show(content) - self._console._exit_buffer() - - -class ScreenContext: - """A context manager that enables an alternative screen. See :meth:`~rich.console.Console.screen` for usage.""" - - def __init__( - self, console: "Console", hide_cursor: bool, style: StyleType = "" - ) -> None: - self.console = console - self.hide_cursor = hide_cursor - self.screen = Screen(style=style) - self._changed = False - - def update( - self, *renderables: RenderableType, style: Optional[StyleType] = None - ) -> None: - """Update the screen. - - Args: - renderable (RenderableType, optional): Optional renderable to replace current renderable, - or None for no change. Defaults to None. - style: (Style, optional): Replacement style, or None for no change. Defaults to None. - """ - if renderables: - self.screen.renderable = ( - Group(*renderables) if len(renderables) > 1 else renderables[0] - ) - if style is not None: - self.screen.style = style - self.console.print(self.screen, end="") - - def __enter__(self) -> "ScreenContext": - self._changed = self.console.set_alt_screen(True) - if self._changed and self.hide_cursor: - self.console.show_cursor(False) - return self - - def __exit__( - self, - exc_type: Optional[Type[BaseException]], - exc_val: Optional[BaseException], - exc_tb: Optional[TracebackType], - ) -> None: - if self._changed: - self.console.set_alt_screen(False) - if self.hide_cursor: - self.console.show_cursor(True) - - -class Group: - """Takes a group of renderables and returns a renderable object that renders the group. - - Args: - renderables (Iterable[RenderableType]): An iterable of renderable objects. - fit (bool, optional): Fit dimension of group to contents, or fill available space. Defaults to True. - """ - - def __init__(self, *renderables: "RenderableType", fit: bool = True) -> None: - self._renderables = renderables - self.fit = fit - self._render: Optional[List[RenderableType]] = None - - @property - def renderables(self) -> List["RenderableType"]: - if self._render is None: - self._render = list(self._renderables) - return self._render - - def __rich_measure__( - self, console: "Console", options: "ConsoleOptions" - ) -> "Measurement": - if self.fit: - return measure_renderables(console, options, self.renderables) - else: - return Measurement(options.max_width, options.max_width) - - def __rich_console__( - self, console: "Console", options: "ConsoleOptions" - ) -> RenderResult: - yield from self.renderables - - -def group(fit: bool = True) -> Callable[..., Callable[..., Group]]: - """A decorator that turns an iterable of renderables in to a group. - - Args: - fit (bool, optional): Fit dimension of group to contents, or fill available space. Defaults to True. - """ - - def decorator( - method: Callable[..., Iterable[RenderableType]] - ) -> Callable[..., Group]: - """Convert a method that returns an iterable of renderables in to a Group.""" - - @wraps(method) - def _replace(*args: Any, **kwargs: Any) -> Group: - renderables = method(*args, **kwargs) - return Group(*renderables, fit=fit) - - return _replace - - return decorator - - -def _is_jupyter() -> bool: # pragma: no cover - """Check if we're running in a Jupyter notebook.""" - try: - get_ipython # type: ignore[name-defined] - except NameError: - return False - ipython = get_ipython() # type: ignore[name-defined] - shell = ipython.__class__.__name__ - if ( - "google.colab" in str(ipython.__class__) - or os.getenv("DATABRICKS_RUNTIME_VERSION") - or shell == "ZMQInteractiveShell" - ): - return True # Jupyter notebook or qtconsole - elif shell == "TerminalInteractiveShell": - return False # Terminal running IPython - else: - return False # Other type (?) - - -COLOR_SYSTEMS = { - "standard": ColorSystem.STANDARD, - "256": ColorSystem.EIGHT_BIT, - "truecolor": ColorSystem.TRUECOLOR, - "windows": ColorSystem.WINDOWS, -} - -_COLOR_SYSTEMS_NAMES = {system: name for name, system in COLOR_SYSTEMS.items()} - - -@dataclass -class ConsoleThreadLocals(threading.local): - """Thread local values for Console context.""" - - theme_stack: ThemeStack - buffer: List[Segment] = field(default_factory=list) - buffer_index: int = 0 - - -class RenderHook(ABC): - """Provides hooks in to the render process.""" - - @abstractmethod - def process_renderables( - self, renderables: List[ConsoleRenderable] - ) -> List[ConsoleRenderable]: - """Called with a list of objects to render. - - This method can return a new list of renderables, or modify and return the same list. - - Args: - renderables (List[ConsoleRenderable]): A number of renderable objects. - - Returns: - List[ConsoleRenderable]: A replacement list of renderables. - """ - - -_windows_console_features: Optional["WindowsConsoleFeatures"] = None - - -def get_windows_console_features() -> "WindowsConsoleFeatures": # pragma: no cover - global _windows_console_features - if _windows_console_features is not None: - return _windows_console_features - from ._windows import get_windows_console_features - - _windows_console_features = get_windows_console_features() - return _windows_console_features - - -def detect_legacy_windows() -> bool: - """Detect legacy Windows.""" - return WINDOWS and not get_windows_console_features().vt - - -class Console: - """A high level console interface. - - Args: - color_system (str, optional): The color system supported by your terminal, - either ``"standard"``, ``"256"`` or ``"truecolor"``. Leave as ``"auto"`` to autodetect. - force_terminal (Optional[bool], optional): Enable/disable terminal control codes, or None to auto-detect terminal. Defaults to None. - force_jupyter (Optional[bool], optional): Enable/disable Jupyter rendering, or None to auto-detect Jupyter. Defaults to None. - force_interactive (Optional[bool], optional): Enable/disable interactive mode, or None to auto detect. Defaults to None. - soft_wrap (Optional[bool], optional): Set soft wrap default on print method. Defaults to False. - theme (Theme, optional): An optional style theme object, or ``None`` for default theme. - stderr (bool, optional): Use stderr rather than stdout if ``file`` is not specified. Defaults to False. - file (IO, optional): A file object where the console should write to. Defaults to stdout. - quiet (bool, Optional): Boolean to suppress all output. Defaults to False. - width (int, optional): The width of the terminal. Leave as default to auto-detect width. - height (int, optional): The height of the terminal. Leave as default to auto-detect height. - style (StyleType, optional): Style to apply to all output, or None for no style. Defaults to None. - no_color (Optional[bool], optional): Enabled no color mode, or None to auto detect. Defaults to None. - tab_size (int, optional): Number of spaces used to replace a tab character. Defaults to 8. - record (bool, optional): Boolean to enable recording of terminal output, - required to call :meth:`export_html`, :meth:`export_svg`, and :meth:`export_text`. Defaults to False. - markup (bool, optional): Boolean to enable :ref:`console_markup`. Defaults to True. - emoji (bool, optional): Enable emoji code. Defaults to True. - emoji_variant (str, optional): Optional emoji variant, either "text" or "emoji". Defaults to None. - highlight (bool, optional): Enable automatic highlighting. Defaults to True. - log_time (bool, optional): Boolean to enable logging of time by :meth:`log` methods. Defaults to True. - log_path (bool, optional): Boolean to enable the logging of the caller by :meth:`log`. Defaults to True. - log_time_format (Union[str, TimeFormatterCallable], optional): If ``log_time`` is enabled, either string for strftime or callable that formats the time. Defaults to "[%X] ". - highlighter (HighlighterType, optional): Default highlighter. - legacy_windows (bool, optional): Enable legacy Windows mode, or ``None`` to auto detect. Defaults to ``None``. - safe_box (bool, optional): Restrict box options that don't render on legacy Windows. - get_datetime (Callable[[], datetime], optional): Callable that gets the current time as a datetime.datetime object (used by Console.log), - or None for datetime.now. - get_time (Callable[[], time], optional): Callable that gets the current time in seconds, default uses time.monotonic. - """ - - _environ: Mapping[str, str] = os.environ - - def __init__( - self, - *, - color_system: Optional[ - Literal["auto", "standard", "256", "truecolor", "windows"] - ] = "auto", - force_terminal: Optional[bool] = None, - force_jupyter: Optional[bool] = None, - force_interactive: Optional[bool] = None, - soft_wrap: bool = False, - theme: Optional[Theme] = None, - stderr: bool = False, - file: Optional[IO[str]] = None, - quiet: bool = False, - width: Optional[int] = None, - height: Optional[int] = None, - style: Optional[StyleType] = None, - no_color: Optional[bool] = None, - tab_size: int = 8, - record: bool = False, - markup: bool = True, - emoji: bool = True, - emoji_variant: Optional[EmojiVariant] = None, - highlight: bool = True, - log_time: bool = True, - log_path: bool = True, - log_time_format: Union[str, FormatTimeCallable] = "[%X]", - highlighter: Optional["HighlighterType"] = ReprHighlighter(), - legacy_windows: Optional[bool] = None, - safe_box: bool = True, - get_datetime: Optional[Callable[[], datetime]] = None, - get_time: Optional[Callable[[], float]] = None, - _environ: Optional[Mapping[str, str]] = None, - ): - # Copy of os.environ allows us to replace it for testing - if _environ is not None: - self._environ = _environ - - self.is_jupyter = _is_jupyter() if force_jupyter is None else force_jupyter - if self.is_jupyter: - if width is None: - jupyter_columns = self._environ.get("JUPYTER_COLUMNS") - if jupyter_columns is not None and jupyter_columns.isdigit(): - width = int(jupyter_columns) - else: - width = JUPYTER_DEFAULT_COLUMNS - if height is None: - jupyter_lines = self._environ.get("JUPYTER_LINES") - if jupyter_lines is not None and jupyter_lines.isdigit(): - height = int(jupyter_lines) - else: - height = JUPYTER_DEFAULT_LINES - - self.tab_size = tab_size - self.record = record - self._markup = markup - self._emoji = emoji - self._emoji_variant: Optional[EmojiVariant] = emoji_variant - self._highlight = highlight - self.legacy_windows: bool = ( - (detect_legacy_windows() and not self.is_jupyter) - if legacy_windows is None - else legacy_windows - ) - - if width is None: - columns = self._environ.get("COLUMNS") - if columns is not None and columns.isdigit(): - width = int(columns) - self.legacy_windows - if height is None: - lines = self._environ.get("LINES") - if lines is not None and lines.isdigit(): - height = int(lines) - - self.soft_wrap = soft_wrap - self._width = width - self._height = height - - self._color_system: Optional[ColorSystem] - - self._force_terminal = None - if force_terminal is not None: - self._force_terminal = force_terminal - - self._file = file - self.quiet = quiet - self.stderr = stderr - - if color_system is None: - self._color_system = None - elif color_system == "auto": - self._color_system = self._detect_color_system() - else: - self._color_system = COLOR_SYSTEMS[color_system] - - self._lock = threading.RLock() - self._log_render = LogRender( - show_time=log_time, - show_path=log_path, - time_format=log_time_format, - ) - self.highlighter: HighlighterType = highlighter or _null_highlighter - self.safe_box = safe_box - self.get_datetime = get_datetime or datetime.now - self.get_time = get_time or monotonic - self.style = style - self.no_color = ( - no_color if no_color is not None else "NO_COLOR" in self._environ - ) - self.is_interactive = ( - (self.is_terminal and not self.is_dumb_terminal) - if force_interactive is None - else force_interactive - ) - - self._record_buffer_lock = threading.RLock() - self._thread_locals = ConsoleThreadLocals( - theme_stack=ThemeStack(themes.DEFAULT if theme is None else theme) - ) - self._record_buffer: List[Segment] = [] - self._render_hooks: List[RenderHook] = [] - self._live: Optional["Live"] = None - self._is_alt_screen = False - - def __repr__(self) -> str: - return f"" - - @property - def file(self) -> IO[str]: - """Get the file object to write to.""" - file = self._file or (sys.stderr if self.stderr else sys.stdout) - file = getattr(file, "rich_proxied_file", file) - if file is None: - file = NULL_FILE - return file - - @file.setter - def file(self, new_file: IO[str]) -> None: - """Set a new file object.""" - self._file = new_file - - @property - def _buffer(self) -> List[Segment]: - """Get a thread local buffer.""" - return self._thread_locals.buffer - - @property - def _buffer_index(self) -> int: - """Get a thread local buffer.""" - return self._thread_locals.buffer_index - - @_buffer_index.setter - def _buffer_index(self, value: int) -> None: - self._thread_locals.buffer_index = value - - @property - def _theme_stack(self) -> ThemeStack: - """Get the thread local theme stack.""" - return self._thread_locals.theme_stack - - def _detect_color_system(self) -> Optional[ColorSystem]: - """Detect color system from env vars.""" - if self.is_jupyter: - return ColorSystem.TRUECOLOR - if not self.is_terminal or self.is_dumb_terminal: - return None - if WINDOWS: # pragma: no cover - if self.legacy_windows: # pragma: no cover - return ColorSystem.WINDOWS - windows_console_features = get_windows_console_features() - return ( - ColorSystem.TRUECOLOR - if windows_console_features.truecolor - else ColorSystem.EIGHT_BIT - ) - else: - color_term = self._environ.get("COLORTERM", "").strip().lower() - if color_term in ("truecolor", "24bit"): - return ColorSystem.TRUECOLOR - term = self._environ.get("TERM", "").strip().lower() - _term_name, _hyphen, colors = term.rpartition("-") - color_system = _TERM_COLORS.get(colors, ColorSystem.STANDARD) - return color_system - - def _enter_buffer(self) -> None: - """Enter in to a buffer context, and buffer all output.""" - self._buffer_index += 1 - - def _exit_buffer(self) -> None: - """Leave buffer context, and render content if required.""" - self._buffer_index -= 1 - self._check_buffer() - - def set_live(self, live: "Live") -> None: - """Set Live instance. Used by Live context manager. - - Args: - live (Live): Live instance using this Console. - - Raises: - errors.LiveError: If this Console has a Live context currently active. - """ - with self._lock: - if self._live is not None: - raise errors.LiveError("Only one live display may be active at once") - self._live = live - - def clear_live(self) -> None: - """Clear the Live instance.""" - with self._lock: - self._live = None - - def push_render_hook(self, hook: RenderHook) -> None: - """Add a new render hook to the stack. - - Args: - hook (RenderHook): Render hook instance. - """ - with self._lock: - self._render_hooks.append(hook) - - def pop_render_hook(self) -> None: - """Pop the last renderhook from the stack.""" - with self._lock: - self._render_hooks.pop() - - def __enter__(self) -> "Console": - """Own context manager to enter buffer context.""" - self._enter_buffer() - return self - - def __exit__(self, exc_type: Any, exc_value: Any, traceback: Any) -> None: - """Exit buffer context.""" - self._exit_buffer() - - def begin_capture(self) -> None: - """Begin capturing console output. Call :meth:`end_capture` to exit capture mode and return output.""" - self._enter_buffer() - - def end_capture(self) -> str: - """End capture mode and return captured string. - - Returns: - str: Console output. - """ - render_result = self._render_buffer(self._buffer) - del self._buffer[:] - self._exit_buffer() - return render_result - - def push_theme(self, theme: Theme, *, inherit: bool = True) -> None: - """Push a new theme on to the top of the stack, replacing the styles from the previous theme. - Generally speaking, you should call :meth:`~rich.console.Console.use_theme` to get a context manager, rather - than calling this method directly. - - Args: - theme (Theme): A theme instance. - inherit (bool, optional): Inherit existing styles. Defaults to True. - """ - self._theme_stack.push_theme(theme, inherit=inherit) - - def pop_theme(self) -> None: - """Remove theme from top of stack, restoring previous theme.""" - self._theme_stack.pop_theme() - - def use_theme(self, theme: Theme, *, inherit: bool = True) -> ThemeContext: - """Use a different theme for the duration of the context manager. - - Args: - theme (Theme): Theme instance to user. - inherit (bool, optional): Inherit existing console styles. Defaults to True. - - Returns: - ThemeContext: [description] - """ - return ThemeContext(self, theme, inherit) - - @property - def color_system(self) -> Optional[str]: - """Get color system string. - - Returns: - Optional[str]: "standard", "256" or "truecolor". - """ - - if self._color_system is not None: - return _COLOR_SYSTEMS_NAMES[self._color_system] - else: - return None - - @property - def encoding(self) -> str: - """Get the encoding of the console file, e.g. ``"utf-8"``. - - Returns: - str: A standard encoding string. - """ - return (getattr(self.file, "encoding", "utf-8") or "utf-8").lower() - - @property - def is_terminal(self) -> bool: - """Check if the console is writing to a terminal. - - Returns: - bool: True if the console writing to a device capable of - understanding terminal codes, otherwise False. - """ - if self._force_terminal is not None: - return self._force_terminal - - if hasattr(sys.stdin, "__module__") and sys.stdin.__module__.startswith( - "idlelib" - ): - # Return False for Idle which claims to be a tty but can't handle ansi codes - return False - - if self.is_jupyter: - # return False for Jupyter, which may have FORCE_COLOR set - return False - - # If FORCE_COLOR env var has any value at all, we assume a terminal. - force_color = self._environ.get("FORCE_COLOR") - if force_color is not None: - self._force_terminal = True - return True - - isatty: Optional[Callable[[], bool]] = getattr(self.file, "isatty", None) - try: - return False if isatty is None else isatty() - except ValueError: - # in some situation (at the end of a pytest run for example) isatty() can raise - # ValueError: I/O operation on closed file - # return False because we aren't in a terminal anymore - return False - - @property - def is_dumb_terminal(self) -> bool: - """Detect dumb terminal. - - Returns: - bool: True if writing to a dumb terminal, otherwise False. - - """ - _term = self._environ.get("TERM", "") - is_dumb = _term.lower() in ("dumb", "unknown") - return self.is_terminal and is_dumb - - @property - def options(self) -> ConsoleOptions: - """Get default console options.""" - return ConsoleOptions( - max_height=self.size.height, - size=self.size, - legacy_windows=self.legacy_windows, - min_width=1, - max_width=self.width, - encoding=self.encoding, - is_terminal=self.is_terminal, - ) - - @property - def size(self) -> ConsoleDimensions: - """Get the size of the console. - - Returns: - ConsoleDimensions: A named tuple containing the dimensions. - """ - - if self._width is not None and self._height is not None: - return ConsoleDimensions(self._width - self.legacy_windows, self._height) - - if self.is_dumb_terminal: - return ConsoleDimensions(80, 25) - - width: Optional[int] = None - height: Optional[int] = None - - if WINDOWS: # pragma: no cover - try: - width, height = os.get_terminal_size() - except (AttributeError, ValueError, OSError): # Probably not a terminal - pass - else: - for file_descriptor in _STD_STREAMS: - try: - width, height = os.get_terminal_size(file_descriptor) - except (AttributeError, ValueError, OSError): - pass - else: - break - - columns = self._environ.get("COLUMNS") - if columns is not None and columns.isdigit(): - width = int(columns) - lines = self._environ.get("LINES") - if lines is not None and lines.isdigit(): - height = int(lines) - - # get_terminal_size can report 0, 0 if run from pseudo-terminal - width = width or 80 - height = height or 25 - return ConsoleDimensions( - width - self.legacy_windows if self._width is None else self._width, - height if self._height is None else self._height, - ) - - @size.setter - def size(self, new_size: Tuple[int, int]) -> None: - """Set a new size for the terminal. - - Args: - new_size (Tuple[int, int]): New width and height. - """ - width, height = new_size - self._width = width - self._height = height - - @property - def width(self) -> int: - """Get the width of the console. - - Returns: - int: The width (in characters) of the console. - """ - return self.size.width - - @width.setter - def width(self, width: int) -> None: - """Set width. - - Args: - width (int): New width. - """ - self._width = width - - @property - def height(self) -> int: - """Get the height of the console. - - Returns: - int: The height (in lines) of the console. - """ - return self.size.height - - @height.setter - def height(self, height: int) -> None: - """Set height. - - Args: - height (int): new height. - """ - self._height = height - - def bell(self) -> None: - """Play a 'bell' sound (if supported by the terminal).""" - self.control(Control.bell()) - - def capture(self) -> Capture: - """A context manager to *capture* the result of print() or log() in a string, - rather than writing it to the console. - - Example: - >>> from rich.console import Console - >>> console = Console() - >>> with console.capture() as capture: - ... console.print("[bold magenta]Hello World[/]") - >>> print(capture.get()) - - Returns: - Capture: Context manager with disables writing to the terminal. - """ - capture = Capture(self) - return capture - - def pager( - self, pager: Optional[Pager] = None, styles: bool = False, links: bool = False - ) -> PagerContext: - """A context manager to display anything printed within a "pager". The pager application - is defined by the system and will typically support at least pressing a key to scroll. - - Args: - pager (Pager, optional): A pager object, or None to use :class:`~rich.pager.SystemPager`. Defaults to None. - styles (bool, optional): Show styles in pager. Defaults to False. - links (bool, optional): Show links in pager. Defaults to False. - - Example: - >>> from rich.console import Console - >>> from rich.__main__ import make_test_card - >>> console = Console() - >>> with console.pager(): - console.print(make_test_card()) - - Returns: - PagerContext: A context manager. - """ - return PagerContext(self, pager=pager, styles=styles, links=links) - - def line(self, count: int = 1) -> None: - """Write new line(s). - - Args: - count (int, optional): Number of new lines. Defaults to 1. - """ - - assert count >= 0, "count must be >= 0" - self.print(NewLine(count)) - - def clear(self, home: bool = True) -> None: - """Clear the screen. - - Args: - home (bool, optional): Also move the cursor to 'home' position. Defaults to True. - """ - if home: - self.control(Control.clear(), Control.home()) - else: - self.control(Control.clear()) - - def status( - self, - status: RenderableType, - *, - spinner: str = "dots", - spinner_style: StyleType = "status.spinner", - speed: float = 1.0, - refresh_per_second: float = 12.5, - ) -> "Status": - """Display a status and spinner. - - Args: - status (RenderableType): A status renderable (str or Text typically). - spinner (str, optional): Name of spinner animation (see python -m rich.spinner). Defaults to "dots". - spinner_style (StyleType, optional): Style of spinner. Defaults to "status.spinner". - speed (float, optional): Speed factor for spinner animation. Defaults to 1.0. - refresh_per_second (float, optional): Number of refreshes per second. Defaults to 12.5. - - Returns: - Status: A Status object that may be used as a context manager. - """ - from .status import Status - - status_renderable = Status( - status, - console=self, - spinner=spinner, - spinner_style=spinner_style, - speed=speed, - refresh_per_second=refresh_per_second, - ) - return status_renderable - - def show_cursor(self, show: bool = True) -> bool: - """Show or hide the cursor. - - Args: - show (bool, optional): Set visibility of the cursor. - """ - if self.is_terminal: - self.control(Control.show_cursor(show)) - return True - return False - - def set_alt_screen(self, enable: bool = True) -> bool: - """Enables alternative screen mode. - - Note, if you enable this mode, you should ensure that is disabled before - the application exits. See :meth:`~rich.Console.screen` for a context manager - that handles this for you. - - Args: - enable (bool, optional): Enable (True) or disable (False) alternate screen. Defaults to True. - - Returns: - bool: True if the control codes were written. - - """ - changed = False - if self.is_terminal and not self.legacy_windows: - self.control(Control.alt_screen(enable)) - changed = True - self._is_alt_screen = enable - return changed - - @property - def is_alt_screen(self) -> bool: - """Check if the alt screen was enabled. - - Returns: - bool: True if the alt screen was enabled, otherwise False. - """ - return self._is_alt_screen - - def set_window_title(self, title: str) -> bool: - """Set the title of the console terminal window. - - Warning: There is no means within Rich of "resetting" the window title to its - previous value, meaning the title you set will persist even after your application - exits. - - ``fish`` shell resets the window title before and after each command by default, - negating this issue. Windows Terminal and command prompt will also reset the title for you. - Most other shells and terminals, however, do not do this. - - Some terminals may require configuration changes before you can set the title. - Some terminals may not support setting the title at all. - - Other software (including the terminal itself, the shell, custom prompts, plugins, etc.) - may also set the terminal window title. This could result in whatever value you write - using this method being overwritten. - - Args: - title (str): The new title of the terminal window. - - Returns: - bool: True if the control code to change the terminal title was - written, otherwise False. Note that a return value of True - does not guarantee that the window title has actually changed, - since the feature may be unsupported/disabled in some terminals. - """ - if self.is_terminal: - self.control(Control.title(title)) - return True - return False - - def screen( - self, hide_cursor: bool = True, style: Optional[StyleType] = None - ) -> "ScreenContext": - """Context manager to enable and disable 'alternative screen' mode. - - Args: - hide_cursor (bool, optional): Also hide the cursor. Defaults to False. - style (Style, optional): Optional style for screen. Defaults to None. - - Returns: - ~ScreenContext: Context which enables alternate screen on enter, and disables it on exit. - """ - return ScreenContext(self, hide_cursor=hide_cursor, style=style or "") - - def measure( - self, renderable: RenderableType, *, options: Optional[ConsoleOptions] = None - ) -> Measurement: - """Measure a renderable. Returns a :class:`~rich.measure.Measurement` object which contains - information regarding the number of characters required to print the renderable. - - Args: - renderable (RenderableType): Any renderable or string. - options (Optional[ConsoleOptions], optional): Options to use when measuring, or None - to use default options. Defaults to None. - - Returns: - Measurement: A measurement of the renderable. - """ - measurement = Measurement.get(self, options or self.options, renderable) - return measurement - - def render( - self, renderable: RenderableType, options: Optional[ConsoleOptions] = None - ) -> Iterable[Segment]: - """Render an object in to an iterable of `Segment` instances. - - This method contains the logic for rendering objects with the console protocol. - You are unlikely to need to use it directly, unless you are extending the library. - - Args: - renderable (RenderableType): An object supporting the console protocol, or - an object that may be converted to a string. - options (ConsoleOptions, optional): An options object, or None to use self.options. Defaults to None. - - Returns: - Iterable[Segment]: An iterable of segments that may be rendered. - """ - - _options = options or self.options - if _options.max_width < 1: - # No space to render anything. This prevents potential recursion errors. - return - render_iterable: RenderResult - - renderable = rich_cast(renderable) - if hasattr(renderable, "__rich_console__") and not isclass(renderable): - render_iterable = renderable.__rich_console__(self, _options) # type: ignore[union-attr] - elif isinstance(renderable, str): - text_renderable = self.render_str( - renderable, highlight=_options.highlight, markup=_options.markup - ) - render_iterable = text_renderable.__rich_console__(self, _options) - else: - raise errors.NotRenderableError( - f"Unable to render {renderable!r}; " - "A str, Segment or object with __rich_console__ method is required" - ) - - try: - iter_render = iter(render_iterable) - except TypeError: - raise errors.NotRenderableError( - f"object {render_iterable!r} is not renderable" - ) - _Segment = Segment - _options = _options.reset_height() - for render_output in iter_render: - if isinstance(render_output, _Segment): - yield render_output - else: - yield from self.render(render_output, _options) - - def render_lines( - self, - renderable: RenderableType, - options: Optional[ConsoleOptions] = None, - *, - style: Optional[Style] = None, - pad: bool = True, - new_lines: bool = False, - ) -> List[List[Segment]]: - """Render objects in to a list of lines. - - The output of render_lines is useful when further formatting of rendered console text - is required, such as the Panel class which draws a border around any renderable object. - - Args: - renderable (RenderableType): Any object renderable in the console. - options (Optional[ConsoleOptions], optional): Console options, or None to use self.options. Default to ``None``. - style (Style, optional): Optional style to apply to renderables. Defaults to ``None``. - pad (bool, optional): Pad lines shorter than render width. Defaults to ``True``. - new_lines (bool, optional): Include "\n" characters at end of lines. - - Returns: - List[List[Segment]]: A list of lines, where a line is a list of Segment objects. - """ - with self._lock: - render_options = options or self.options - _rendered = self.render(renderable, render_options) - if style: - _rendered = Segment.apply_style(_rendered, style) - - render_height = render_options.height - if render_height is not None: - render_height = max(0, render_height) - - lines = list( - islice( - Segment.split_and_crop_lines( - _rendered, - render_options.max_width, - include_new_lines=new_lines, - pad=pad, - style=style, - ), - None, - render_height, - ) - ) - if render_options.height is not None: - extra_lines = render_options.height - len(lines) - if extra_lines > 0: - pad_line = [ - [Segment(" " * render_options.max_width, style), Segment("\n")] - if new_lines - else [Segment(" " * render_options.max_width, style)] - ] - lines.extend(pad_line * extra_lines) - - return lines - - def render_str( - self, - text: str, - *, - style: Union[str, Style] = "", - justify: Optional[JustifyMethod] = None, - overflow: Optional[OverflowMethod] = None, - emoji: Optional[bool] = None, - markup: Optional[bool] = None, - highlight: Optional[bool] = None, - highlighter: Optional[HighlighterType] = None, - ) -> "Text": - """Convert a string to a Text instance. This is called automatically if - you print or log a string. - - Args: - text (str): Text to render. - style (Union[str, Style], optional): Style to apply to rendered text. - justify (str, optional): Justify method: "default", "left", "center", "full", or "right". Defaults to ``None``. - overflow (str, optional): Overflow method: "crop", "fold", or "ellipsis". Defaults to ``None``. - emoji (Optional[bool], optional): Enable emoji, or ``None`` to use Console default. - markup (Optional[bool], optional): Enable markup, or ``None`` to use Console default. - highlight (Optional[bool], optional): Enable highlighting, or ``None`` to use Console default. - highlighter (HighlighterType, optional): Optional highlighter to apply. - Returns: - ConsoleRenderable: Renderable object. - - """ - emoji_enabled = emoji or (emoji is None and self._emoji) - markup_enabled = markup or (markup is None and self._markup) - highlight_enabled = highlight or (highlight is None and self._highlight) - - if markup_enabled: - rich_text = render_markup( - text, - style=style, - emoji=emoji_enabled, - emoji_variant=self._emoji_variant, - ) - rich_text.justify = justify - rich_text.overflow = overflow - else: - rich_text = Text( - _emoji_replace(text, default_variant=self._emoji_variant) - if emoji_enabled - else text, - justify=justify, - overflow=overflow, - style=style, - ) - - _highlighter = (highlighter or self.highlighter) if highlight_enabled else None - if _highlighter is not None: - highlight_text = _highlighter(str(rich_text)) - highlight_text.copy_styles(rich_text) - return highlight_text - - return rich_text - - def get_style( - self, name: Union[str, Style], *, default: Optional[Union[Style, str]] = None - ) -> Style: - """Get a Style instance by its theme name or parse a definition. - - Args: - name (str): The name of a style or a style definition. - - Returns: - Style: A Style object. - - Raises: - MissingStyle: If no style could be parsed from name. - - """ - if isinstance(name, Style): - return name - - try: - style = self._theme_stack.get(name) - if style is None: - style = Style.parse(name) - return style.copy() if style.link else style - except errors.StyleSyntaxError as error: - if default is not None: - return self.get_style(default) - raise errors.MissingStyle( - f"Failed to get style {name!r}; {error}" - ) from None - - def _collect_renderables( - self, - objects: Iterable[Any], - sep: str, - end: str, - *, - justify: Optional[JustifyMethod] = None, - emoji: Optional[bool] = None, - markup: Optional[bool] = None, - highlight: Optional[bool] = None, - ) -> List[ConsoleRenderable]: - """Combine a number of renderables and text into one renderable. - - Args: - objects (Iterable[Any]): Anything that Rich can render. - sep (str): String to write between print data. - end (str): String to write at end of print data. - justify (str, optional): One of "left", "right", "center", or "full". Defaults to ``None``. - emoji (Optional[bool], optional): Enable emoji code, or ``None`` to use console default. - markup (Optional[bool], optional): Enable markup, or ``None`` to use console default. - highlight (Optional[bool], optional): Enable automatic highlighting, or ``None`` to use console default. - - Returns: - List[ConsoleRenderable]: A list of things to render. - """ - renderables: List[ConsoleRenderable] = [] - _append = renderables.append - text: List[Text] = [] - append_text = text.append - - append = _append - if justify in ("left", "center", "right"): - - def align_append(renderable: RenderableType) -> None: - _append(Align(renderable, cast(AlignMethod, justify))) - - append = align_append - - _highlighter: HighlighterType = _null_highlighter - if highlight or (highlight is None and self._highlight): - _highlighter = self.highlighter - - def check_text() -> None: - if text: - sep_text = Text(sep, justify=justify, end=end) - append(sep_text.join(text)) - text.clear() - - for renderable in objects: - renderable = rich_cast(renderable) - if isinstance(renderable, str): - append_text( - self.render_str( - renderable, emoji=emoji, markup=markup, highlighter=_highlighter - ) - ) - elif isinstance(renderable, Text): - append_text(renderable) - elif isinstance(renderable, ConsoleRenderable): - check_text() - append(renderable) - elif is_expandable(renderable): - check_text() - append(Pretty(renderable, highlighter=_highlighter)) - else: - append_text(_highlighter(str(renderable))) - - check_text() - - if self.style is not None: - style = self.get_style(self.style) - renderables = [Styled(renderable, style) for renderable in renderables] - - return renderables - - def rule( - self, - title: TextType = "", - *, - characters: str = "─", - style: Union[str, Style] = "rule.line", - align: AlignMethod = "center", - ) -> None: - """Draw a line with optional centered title. - - Args: - title (str, optional): Text to render over the rule. Defaults to "". - characters (str, optional): Character(s) to form the line. Defaults to "─". - style (str, optional): Style of line. Defaults to "rule.line". - align (str, optional): How to align the title, one of "left", "center", or "right". Defaults to "center". - """ - from .rule import Rule - - rule = Rule(title=title, characters=characters, style=style, align=align) - self.print(rule) - - def control(self, *control: Control) -> None: - """Insert non-printing control codes. - - Args: - control_codes (str): Control codes, such as those that may move the cursor. - """ - if not self.is_dumb_terminal: - with self: - self._buffer.extend(_control.segment for _control in control) - - def out( - self, - *objects: Any, - sep: str = " ", - end: str = "\n", - style: Optional[Union[str, Style]] = None, - highlight: Optional[bool] = None, - ) -> None: - """Output to the terminal. This is a low-level way of writing to the terminal which unlike - :meth:`~rich.console.Console.print` won't pretty print, wrap text, or apply markup, but will - optionally apply highlighting and a basic style. - - Args: - sep (str, optional): String to write between print data. Defaults to " ". - end (str, optional): String to write at end of print data. Defaults to "\\\\n". - style (Union[str, Style], optional): A style to apply to output. Defaults to None. - highlight (Optional[bool], optional): Enable automatic highlighting, or ``None`` to use - console default. Defaults to ``None``. - """ - raw_output: str = sep.join(str(_object) for _object in objects) - self.print( - raw_output, - style=style, - highlight=highlight, - emoji=False, - markup=False, - no_wrap=True, - overflow="ignore", - crop=False, - end=end, - ) - - def print( - self, - *objects: Any, - sep: str = " ", - end: str = "\n", - style: Optional[Union[str, Style]] = None, - justify: Optional[JustifyMethod] = None, - overflow: Optional[OverflowMethod] = None, - no_wrap: Optional[bool] = None, - emoji: Optional[bool] = None, - markup: Optional[bool] = None, - highlight: Optional[bool] = None, - width: Optional[int] = None, - height: Optional[int] = None, - crop: bool = True, - soft_wrap: Optional[bool] = None, - new_line_start: bool = False, - ) -> None: - """Print to the console. - - Args: - objects (positional args): Objects to log to the terminal. - sep (str, optional): String to write between print data. Defaults to " ". - end (str, optional): String to write at end of print data. Defaults to "\\\\n". - style (Union[str, Style], optional): A style to apply to output. Defaults to None. - justify (str, optional): Justify method: "default", "left", "right", "center", or "full". Defaults to ``None``. - overflow (str, optional): Overflow method: "ignore", "crop", "fold", or "ellipsis". Defaults to None. - no_wrap (Optional[bool], optional): Disable word wrapping. Defaults to None. - emoji (Optional[bool], optional): Enable emoji code, or ``None`` to use console default. Defaults to ``None``. - markup (Optional[bool], optional): Enable markup, or ``None`` to use console default. Defaults to ``None``. - highlight (Optional[bool], optional): Enable automatic highlighting, or ``None`` to use console default. Defaults to ``None``. - width (Optional[int], optional): Width of output, or ``None`` to auto-detect. Defaults to ``None``. - crop (Optional[bool], optional): Crop output to width of terminal. Defaults to True. - soft_wrap (bool, optional): Enable soft wrap mode which disables word wrapping and cropping of text or ``None`` for - Console default. Defaults to ``None``. - new_line_start (bool, False): Insert a new line at the start if the output contains more than one line. Defaults to ``False``. - """ - if not objects: - objects = (NewLine(),) - - if soft_wrap is None: - soft_wrap = self.soft_wrap - if soft_wrap: - if no_wrap is None: - no_wrap = True - if overflow is None: - overflow = "ignore" - crop = False - render_hooks = self._render_hooks[:] - with self: - renderables = self._collect_renderables( - objects, - sep, - end, - justify=justify, - emoji=emoji, - markup=markup, - highlight=highlight, - ) - for hook in render_hooks: - renderables = hook.process_renderables(renderables) - render_options = self.options.update( - justify=justify, - overflow=overflow, - width=min(width, self.width) if width is not None else NO_CHANGE, - height=height, - no_wrap=no_wrap, - markup=markup, - highlight=highlight, - ) - - new_segments: List[Segment] = [] - extend = new_segments.extend - render = self.render - if style is None: - for renderable in renderables: - extend(render(renderable, render_options)) - else: - for renderable in renderables: - extend( - Segment.apply_style( - render(renderable, render_options), self.get_style(style) - ) - ) - if new_line_start: - if ( - len("".join(segment.text for segment in new_segments).splitlines()) - > 1 - ): - new_segments.insert(0, Segment.line()) - if crop: - buffer_extend = self._buffer.extend - for line in Segment.split_and_crop_lines( - new_segments, self.width, pad=False - ): - buffer_extend(line) - else: - self._buffer.extend(new_segments) - - def print_json( - self, - json: Optional[str] = None, - *, - data: Any = None, - indent: Union[None, int, str] = 2, - highlight: bool = True, - skip_keys: bool = False, - ensure_ascii: bool = False, - check_circular: bool = True, - allow_nan: bool = True, - default: Optional[Callable[[Any], Any]] = None, - sort_keys: bool = False, - ) -> None: - """Pretty prints JSON. Output will be valid JSON. - - Args: - json (Optional[str]): A string containing JSON. - data (Any): If json is not supplied, then encode this data. - indent (Union[None, int, str], optional): Number of spaces to indent. Defaults to 2. - highlight (bool, optional): Enable highlighting of output: Defaults to True. - skip_keys (bool, optional): Skip keys not of a basic type. Defaults to False. - ensure_ascii (bool, optional): Escape all non-ascii characters. Defaults to False. - check_circular (bool, optional): Check for circular references. Defaults to True. - allow_nan (bool, optional): Allow NaN and Infinity values. Defaults to True. - default (Callable, optional): A callable that converts values that can not be encoded - in to something that can be JSON encoded. Defaults to None. - sort_keys (bool, optional): Sort dictionary keys. Defaults to False. - """ - from pip._vendor.rich.json import JSON - - if json is None: - json_renderable = JSON.from_data( - data, - indent=indent, - highlight=highlight, - skip_keys=skip_keys, - ensure_ascii=ensure_ascii, - check_circular=check_circular, - allow_nan=allow_nan, - default=default, - sort_keys=sort_keys, - ) - else: - if not isinstance(json, str): - raise TypeError( - f"json must be str. Did you mean print_json(data={json!r}) ?" - ) - json_renderable = JSON( - json, - indent=indent, - highlight=highlight, - skip_keys=skip_keys, - ensure_ascii=ensure_ascii, - check_circular=check_circular, - allow_nan=allow_nan, - default=default, - sort_keys=sort_keys, - ) - self.print(json_renderable, soft_wrap=True) - - def update_screen( - self, - renderable: RenderableType, - *, - region: Optional[Region] = None, - options: Optional[ConsoleOptions] = None, - ) -> None: - """Update the screen at a given offset. - - Args: - renderable (RenderableType): A Rich renderable. - region (Region, optional): Region of screen to update, or None for entire screen. Defaults to None. - x (int, optional): x offset. Defaults to 0. - y (int, optional): y offset. Defaults to 0. - - Raises: - errors.NoAltScreen: If the Console isn't in alt screen mode. - - """ - if not self.is_alt_screen: - raise errors.NoAltScreen("Alt screen must be enabled to call update_screen") - render_options = options or self.options - if region is None: - x = y = 0 - render_options = render_options.update_dimensions( - render_options.max_width, render_options.height or self.height - ) - else: - x, y, width, height = region - render_options = render_options.update_dimensions(width, height) - - lines = self.render_lines(renderable, options=render_options) - self.update_screen_lines(lines, x, y) - - def update_screen_lines( - self, lines: List[List[Segment]], x: int = 0, y: int = 0 - ) -> None: - """Update lines of the screen at a given offset. - - Args: - lines (List[List[Segment]]): Rendered lines (as produced by :meth:`~rich.Console.render_lines`). - x (int, optional): x offset (column no). Defaults to 0. - y (int, optional): y offset (column no). Defaults to 0. - - Raises: - errors.NoAltScreen: If the Console isn't in alt screen mode. - """ - if not self.is_alt_screen: - raise errors.NoAltScreen("Alt screen must be enabled to call update_screen") - screen_update = ScreenUpdate(lines, x, y) - segments = self.render(screen_update) - self._buffer.extend(segments) - self._check_buffer() - - def print_exception( - self, - *, - width: Optional[int] = 100, - extra_lines: int = 3, - theme: Optional[str] = None, - word_wrap: bool = False, - show_locals: bool = False, - suppress: Iterable[Union[str, ModuleType]] = (), - max_frames: int = 100, - ) -> None: - """Prints a rich render of the last exception and traceback. - - Args: - width (Optional[int], optional): Number of characters used to render code. Defaults to 100. - extra_lines (int, optional): Additional lines of code to render. Defaults to 3. - theme (str, optional): Override pygments theme used in traceback - word_wrap (bool, optional): Enable word wrapping of long lines. Defaults to False. - show_locals (bool, optional): Enable display of local variables. Defaults to False. - suppress (Iterable[Union[str, ModuleType]]): Optional sequence of modules or paths to exclude from traceback. - max_frames (int): Maximum number of frames to show in a traceback, 0 for no maximum. Defaults to 100. - """ - from .traceback import Traceback - - traceback = Traceback( - width=width, - extra_lines=extra_lines, - theme=theme, - word_wrap=word_wrap, - show_locals=show_locals, - suppress=suppress, - max_frames=max_frames, - ) - self.print(traceback) - - @staticmethod - def _caller_frame_info( - offset: int, - currentframe: Callable[[], Optional[FrameType]] = inspect.currentframe, - ) -> Tuple[str, int, Dict[str, Any]]: - """Get caller frame information. - - Args: - offset (int): the caller offset within the current frame stack. - currentframe (Callable[[], Optional[FrameType]], optional): the callable to use to - retrieve the current frame. Defaults to ``inspect.currentframe``. - - Returns: - Tuple[str, int, Dict[str, Any]]: A tuple containing the filename, the line number and - the dictionary of local variables associated with the caller frame. - - Raises: - RuntimeError: If the stack offset is invalid. - """ - # Ignore the frame of this local helper - offset += 1 - - frame = currentframe() - if frame is not None: - # Use the faster currentframe where implemented - while offset and frame is not None: - frame = frame.f_back - offset -= 1 - assert frame is not None - return frame.f_code.co_filename, frame.f_lineno, frame.f_locals - else: - # Fallback to the slower stack - frame_info = inspect.stack()[offset] - return frame_info.filename, frame_info.lineno, frame_info.frame.f_locals - - def log( - self, - *objects: Any, - sep: str = " ", - end: str = "\n", - style: Optional[Union[str, Style]] = None, - justify: Optional[JustifyMethod] = None, - emoji: Optional[bool] = None, - markup: Optional[bool] = None, - highlight: Optional[bool] = None, - log_locals: bool = False, - _stack_offset: int = 1, - ) -> None: - """Log rich content to the terminal. - - Args: - objects (positional args): Objects to log to the terminal. - sep (str, optional): String to write between print data. Defaults to " ". - end (str, optional): String to write at end of print data. Defaults to "\\\\n". - style (Union[str, Style], optional): A style to apply to output. Defaults to None. - justify (str, optional): One of "left", "right", "center", or "full". Defaults to ``None``. - overflow (str, optional): Overflow method: "crop", "fold", or "ellipsis". Defaults to None. - emoji (Optional[bool], optional): Enable emoji code, or ``None`` to use console default. Defaults to None. - markup (Optional[bool], optional): Enable markup, or ``None`` to use console default. Defaults to None. - highlight (Optional[bool], optional): Enable automatic highlighting, or ``None`` to use console default. Defaults to None. - log_locals (bool, optional): Boolean to enable logging of locals where ``log()`` - was called. Defaults to False. - _stack_offset (int, optional): Offset of caller from end of call stack. Defaults to 1. - """ - if not objects: - objects = (NewLine(),) - - render_hooks = self._render_hooks[:] - - with self: - renderables = self._collect_renderables( - objects, - sep, - end, - justify=justify, - emoji=emoji, - markup=markup, - highlight=highlight, - ) - if style is not None: - renderables = [Styled(renderable, style) for renderable in renderables] - - filename, line_no, locals = self._caller_frame_info(_stack_offset) - link_path = None if filename.startswith("<") else os.path.abspath(filename) - path = filename.rpartition(os.sep)[-1] - if log_locals: - locals_map = { - key: value - for key, value in locals.items() - if not key.startswith("__") - } - renderables.append(render_scope(locals_map, title="[i]locals")) - - renderables = [ - self._log_render( - self, - renderables, - log_time=self.get_datetime(), - path=path, - line_no=line_no, - link_path=link_path, - ) - ] - for hook in render_hooks: - renderables = hook.process_renderables(renderables) - new_segments: List[Segment] = [] - extend = new_segments.extend - render = self.render - render_options = self.options - for renderable in renderables: - extend(render(renderable, render_options)) - buffer_extend = self._buffer.extend - for line in Segment.split_and_crop_lines( - new_segments, self.width, pad=False - ): - buffer_extend(line) - - def _check_buffer(self) -> None: - """Check if the buffer may be rendered. Render it if it can (e.g. Console.quiet is False) - Rendering is supported on Windows, Unix and Jupyter environments. For - legacy Windows consoles, the win32 API is called directly. - This method will also record what it renders if recording is enabled via Console.record. - """ - if self.quiet: - del self._buffer[:] - return - with self._lock: - if self.record: - with self._record_buffer_lock: - self._record_buffer.extend(self._buffer[:]) - - if self._buffer_index == 0: - if self.is_jupyter: # pragma: no cover - from .jupyter import display - - display(self._buffer, self._render_buffer(self._buffer[:])) - del self._buffer[:] - else: - if WINDOWS: - use_legacy_windows_render = False - if self.legacy_windows: - fileno = get_fileno(self.file) - if fileno is not None: - use_legacy_windows_render = ( - fileno in _STD_STREAMS_OUTPUT - ) - - if use_legacy_windows_render: - from pip._vendor.rich._win32_console import LegacyWindowsTerm - from pip._vendor.rich._windows_renderer import legacy_windows_render - - buffer = self._buffer[:] - if self.no_color and self._color_system: - buffer = list(Segment.remove_color(buffer)) - - legacy_windows_render(buffer, LegacyWindowsTerm(self.file)) - else: - # Either a non-std stream on legacy Windows, or modern Windows. - text = self._render_buffer(self._buffer[:]) - # https://bugs.python.org/issue37871 - # https://github.com/python/cpython/issues/82052 - # We need to avoid writing more than 32Kb in a single write, due to the above bug - write = self.file.write - # Worse case scenario, every character is 4 bytes of utf-8 - MAX_WRITE = 32 * 1024 // 4 - try: - if len(text) <= MAX_WRITE: - write(text) - else: - batch: List[str] = [] - batch_append = batch.append - size = 0 - for line in text.splitlines(True): - if size + len(line) > MAX_WRITE and batch: - write("".join(batch)) - batch.clear() - size = 0 - batch_append(line) - size += len(line) - if batch: - write("".join(batch)) - batch.clear() - except UnicodeEncodeError as error: - error.reason = f"{error.reason}\n*** You may need to add PYTHONIOENCODING=utf-8 to your environment ***" - raise - else: - text = self._render_buffer(self._buffer[:]) - try: - self.file.write(text) - except UnicodeEncodeError as error: - error.reason = f"{error.reason}\n*** You may need to add PYTHONIOENCODING=utf-8 to your environment ***" - raise - - self.file.flush() - del self._buffer[:] - - def _render_buffer(self, buffer: Iterable[Segment]) -> str: - """Render buffered output, and clear buffer.""" - output: List[str] = [] - append = output.append - color_system = self._color_system - legacy_windows = self.legacy_windows - not_terminal = not self.is_terminal - if self.no_color and color_system: - buffer = Segment.remove_color(buffer) - for text, style, control in buffer: - if style: - append( - style.render( - text, - color_system=color_system, - legacy_windows=legacy_windows, - ) - ) - elif not (not_terminal and control): - append(text) - - rendered = "".join(output) - return rendered - - def input( - self, - prompt: TextType = "", - *, - markup: bool = True, - emoji: bool = True, - password: bool = False, - stream: Optional[TextIO] = None, - ) -> str: - """Displays a prompt and waits for input from the user. The prompt may contain color / style. - - It works in the same way as Python's builtin :func:`input` function and provides elaborate line editing and history features if Python's builtin :mod:`readline` module is previously loaded. - - Args: - prompt (Union[str, Text]): Text to render in the prompt. - markup (bool, optional): Enable console markup (requires a str prompt). Defaults to True. - emoji (bool, optional): Enable emoji (requires a str prompt). Defaults to True. - password: (bool, optional): Hide typed text. Defaults to False. - stream: (TextIO, optional): Optional file to read input from (rather than stdin). Defaults to None. - - Returns: - str: Text read from stdin. - """ - if prompt: - self.print(prompt, markup=markup, emoji=emoji, end="") - if password: - result = getpass("", stream=stream) - else: - if stream: - result = stream.readline() - else: - result = input() - return result - - def export_text(self, *, clear: bool = True, styles: bool = False) -> str: - """Generate text from console contents (requires record=True argument in constructor). - - Args: - clear (bool, optional): Clear record buffer after exporting. Defaults to ``True``. - styles (bool, optional): If ``True``, ansi escape codes will be included. ``False`` for plain text. - Defaults to ``False``. - - Returns: - str: String containing console contents. - - """ - assert ( - self.record - ), "To export console contents set record=True in the constructor or instance" - - with self._record_buffer_lock: - if styles: - text = "".join( - (style.render(text) if style else text) - for text, style, _ in self._record_buffer - ) - else: - text = "".join( - segment.text - for segment in self._record_buffer - if not segment.control - ) - if clear: - del self._record_buffer[:] - return text - - def save_text(self, path: str, *, clear: bool = True, styles: bool = False) -> None: - """Generate text from console and save to a given location (requires record=True argument in constructor). - - Args: - path (str): Path to write text files. - clear (bool, optional): Clear record buffer after exporting. Defaults to ``True``. - styles (bool, optional): If ``True``, ansi style codes will be included. ``False`` for plain text. - Defaults to ``False``. - - """ - text = self.export_text(clear=clear, styles=styles) - with open(path, "wt", encoding="utf-8") as write_file: - write_file.write(text) - - def export_html( - self, - *, - theme: Optional[TerminalTheme] = None, - clear: bool = True, - code_format: Optional[str] = None, - inline_styles: bool = False, - ) -> str: - """Generate HTML from console contents (requires record=True argument in constructor). - - Args: - theme (TerminalTheme, optional): TerminalTheme object containing console colors. - clear (bool, optional): Clear record buffer after exporting. Defaults to ``True``. - code_format (str, optional): Format string to render HTML. In addition to '{foreground}', - '{background}', and '{code}', should contain '{stylesheet}' if inline_styles is ``False``. - inline_styles (bool, optional): If ``True`` styles will be inlined in to spans, which makes files - larger but easier to cut and paste markup. If ``False``, styles will be embedded in a style tag. - Defaults to False. - - Returns: - str: String containing console contents as HTML. - """ - assert ( - self.record - ), "To export console contents set record=True in the constructor or instance" - fragments: List[str] = [] - append = fragments.append - _theme = theme or DEFAULT_TERMINAL_THEME - stylesheet = "" - - render_code_format = CONSOLE_HTML_FORMAT if code_format is None else code_format - - with self._record_buffer_lock: - if inline_styles: - for text, style, _ in Segment.filter_control( - Segment.simplify(self._record_buffer) - ): - text = escape(text) - if style: - rule = style.get_html_style(_theme) - if style.link: - text = f'{text}' - text = f'{text}' if rule else text - append(text) - else: - styles: Dict[str, int] = {} - for text, style, _ in Segment.filter_control( - Segment.simplify(self._record_buffer) - ): - text = escape(text) - if style: - rule = style.get_html_style(_theme) - style_number = styles.setdefault(rule, len(styles) + 1) - if style.link: - text = f'{text}' - else: - text = f'{text}' - append(text) - stylesheet_rules: List[str] = [] - stylesheet_append = stylesheet_rules.append - for style_rule, style_number in styles.items(): - if style_rule: - stylesheet_append(f".r{style_number} {{{style_rule}}}") - stylesheet = "\n".join(stylesheet_rules) - - rendered_code = render_code_format.format( - code="".join(fragments), - stylesheet=stylesheet, - foreground=_theme.foreground_color.hex, - background=_theme.background_color.hex, - ) - if clear: - del self._record_buffer[:] - return rendered_code - - def save_html( - self, - path: str, - *, - theme: Optional[TerminalTheme] = None, - clear: bool = True, - code_format: str = CONSOLE_HTML_FORMAT, - inline_styles: bool = False, - ) -> None: - """Generate HTML from console contents and write to a file (requires record=True argument in constructor). - - Args: - path (str): Path to write html file. - theme (TerminalTheme, optional): TerminalTheme object containing console colors. - clear (bool, optional): Clear record buffer after exporting. Defaults to ``True``. - code_format (str, optional): Format string to render HTML. In addition to '{foreground}', - '{background}', and '{code}', should contain '{stylesheet}' if inline_styles is ``False``. - inline_styles (bool, optional): If ``True`` styles will be inlined in to spans, which makes files - larger but easier to cut and paste markup. If ``False``, styles will be embedded in a style tag. - Defaults to False. - - """ - html = self.export_html( - theme=theme, - clear=clear, - code_format=code_format, - inline_styles=inline_styles, - ) - with open(path, "wt", encoding="utf-8") as write_file: - write_file.write(html) - - def export_svg( - self, - *, - title: str = "Rich", - theme: Optional[TerminalTheme] = None, - clear: bool = True, - code_format: str = CONSOLE_SVG_FORMAT, - font_aspect_ratio: float = 0.61, - unique_id: Optional[str] = None, - ) -> str: - """ - Generate an SVG from the console contents (requires record=True in Console constructor). - - Args: - title (str, optional): The title of the tab in the output image - theme (TerminalTheme, optional): The ``TerminalTheme`` object to use to style the terminal - clear (bool, optional): Clear record buffer after exporting. Defaults to ``True`` - code_format (str, optional): Format string used to generate the SVG. Rich will inject a number of variables - into the string in order to form the final SVG output. The default template used and the variables - injected by Rich can be found by inspecting the ``console.CONSOLE_SVG_FORMAT`` variable. - font_aspect_ratio (float, optional): The width to height ratio of the font used in the ``code_format`` - string. Defaults to 0.61, which is the width to height ratio of Fira Code (the default font). - If you aren't specifying a different font inside ``code_format``, you probably don't need this. - unique_id (str, optional): unique id that is used as the prefix for various elements (CSS styles, node - ids). If not set, this defaults to a computed value based on the recorded content. - """ - - from pip._vendor.rich.cells import cell_len - - style_cache: Dict[Style, str] = {} - - def get_svg_style(style: Style) -> str: - """Convert a Style to CSS rules for SVG.""" - if style in style_cache: - return style_cache[style] - css_rules = [] - color = ( - _theme.foreground_color - if (style.color is None or style.color.is_default) - else style.color.get_truecolor(_theme) - ) - bgcolor = ( - _theme.background_color - if (style.bgcolor is None or style.bgcolor.is_default) - else style.bgcolor.get_truecolor(_theme) - ) - if style.reverse: - color, bgcolor = bgcolor, color - if style.dim: - color = blend_rgb(color, bgcolor, 0.4) - css_rules.append(f"fill: {color.hex}") - if style.bold: - css_rules.append("font-weight: bold") - if style.italic: - css_rules.append("font-style: italic;") - if style.underline: - css_rules.append("text-decoration: underline;") - if style.strike: - css_rules.append("text-decoration: line-through;") - - css = ";".join(css_rules) - style_cache[style] = css - return css - - _theme = theme or SVG_EXPORT_THEME - - width = self.width - char_height = 20 - char_width = char_height * font_aspect_ratio - line_height = char_height * 1.22 - - margin_top = 1 - margin_right = 1 - margin_bottom = 1 - margin_left = 1 - - padding_top = 40 - padding_right = 8 - padding_bottom = 8 - padding_left = 8 - - padding_width = padding_left + padding_right - padding_height = padding_top + padding_bottom - margin_width = margin_left + margin_right - margin_height = margin_top + margin_bottom - - text_backgrounds: List[str] = [] - text_group: List[str] = [] - classes: Dict[str, int] = {} - style_no = 1 - - def escape_text(text: str) -> str: - """HTML escape text and replace spaces with nbsp.""" - return escape(text).replace(" ", " ") - - def make_tag( - name: str, content: Optional[str] = None, **attribs: object - ) -> str: - """Make a tag from name, content, and attributes.""" - - def stringify(value: object) -> str: - if isinstance(value, (float)): - return format(value, "g") - return str(value) - - tag_attribs = " ".join( - f'{k.lstrip("_").replace("_", "-")}="{stringify(v)}"' - for k, v in attribs.items() - ) - return ( - f"<{name} {tag_attribs}>{content}" - if content - else f"<{name} {tag_attribs}/>" - ) - - with self._record_buffer_lock: - segments = list(Segment.filter_control(self._record_buffer)) - if clear: - self._record_buffer.clear() - - if unique_id is None: - unique_id = "terminal-" + str( - zlib.adler32( - ("".join(repr(segment) for segment in segments)).encode( - "utf-8", - "ignore", - ) - + title.encode("utf-8", "ignore") - ) - ) - y = 0 - for y, line in enumerate(Segment.split_and_crop_lines(segments, length=width)): - x = 0 - for text, style, _control in line: - style = style or Style() - rules = get_svg_style(style) - if rules not in classes: - classes[rules] = style_no - style_no += 1 - class_name = f"r{classes[rules]}" - - if style.reverse: - has_background = True - background = ( - _theme.foreground_color.hex - if style.color is None - else style.color.get_truecolor(_theme).hex - ) - else: - bgcolor = style.bgcolor - has_background = bgcolor is not None and not bgcolor.is_default - background = ( - _theme.background_color.hex - if style.bgcolor is None - else style.bgcolor.get_truecolor(_theme).hex - ) - - text_length = cell_len(text) - if has_background: - text_backgrounds.append( - make_tag( - "rect", - fill=background, - x=x * char_width, - y=y * line_height + 1.5, - width=char_width * text_length, - height=line_height + 0.25, - shape_rendering="crispEdges", - ) - ) - - if text != " " * len(text): - text_group.append( - make_tag( - "text", - escape_text(text), - _class=f"{unique_id}-{class_name}", - x=x * char_width, - y=y * line_height + char_height, - textLength=char_width * len(text), - clip_path=f"url(#{unique_id}-line-{y})", - ) - ) - x += cell_len(text) - - line_offsets = [line_no * line_height + 1.5 for line_no in range(y)] - lines = "\n".join( - f""" - {make_tag("rect", x=0, y=offset, width=char_width * width, height=line_height + 0.25)} - """ - for line_no, offset in enumerate(line_offsets) - ) - - styles = "\n".join( - f".{unique_id}-r{rule_no} {{ {css} }}" for css, rule_no in classes.items() - ) - backgrounds = "".join(text_backgrounds) - matrix = "".join(text_group) - - terminal_width = ceil(width * char_width + padding_width) - terminal_height = (y + 1) * line_height + padding_height - chrome = make_tag( - "rect", - fill=_theme.background_color.hex, - stroke="rgba(255,255,255,0.35)", - stroke_width="1", - x=margin_left, - y=margin_top, - width=terminal_width, - height=terminal_height, - rx=8, - ) - - title_color = _theme.foreground_color.hex - if title: - chrome += make_tag( - "text", - escape_text(title), - _class=f"{unique_id}-title", - fill=title_color, - text_anchor="middle", - x=terminal_width // 2, - y=margin_top + char_height + 6, - ) - chrome += f""" - - - - - - """ - - svg = code_format.format( - unique_id=unique_id, - char_width=char_width, - char_height=char_height, - line_height=line_height, - terminal_width=char_width * width - 1, - terminal_height=(y + 1) * line_height - 1, - width=terminal_width + margin_width, - height=terminal_height + margin_height, - terminal_x=margin_left + padding_left, - terminal_y=margin_top + padding_top, - styles=styles, - chrome=chrome, - backgrounds=backgrounds, - matrix=matrix, - lines=lines, - ) - return svg - - def save_svg( - self, - path: str, - *, - title: str = "Rich", - theme: Optional[TerminalTheme] = None, - clear: bool = True, - code_format: str = CONSOLE_SVG_FORMAT, - font_aspect_ratio: float = 0.61, - unique_id: Optional[str] = None, - ) -> None: - """Generate an SVG file from the console contents (requires record=True in Console constructor). - - Args: - path (str): The path to write the SVG to. - title (str, optional): The title of the tab in the output image - theme (TerminalTheme, optional): The ``TerminalTheme`` object to use to style the terminal - clear (bool, optional): Clear record buffer after exporting. Defaults to ``True`` - code_format (str, optional): Format string used to generate the SVG. Rich will inject a number of variables - into the string in order to form the final SVG output. The default template used and the variables - injected by Rich can be found by inspecting the ``console.CONSOLE_SVG_FORMAT`` variable. - font_aspect_ratio (float, optional): The width to height ratio of the font used in the ``code_format`` - string. Defaults to 0.61, which is the width to height ratio of Fira Code (the default font). - If you aren't specifying a different font inside ``code_format``, you probably don't need this. - unique_id (str, optional): unique id that is used as the prefix for various elements (CSS styles, node - ids). If not set, this defaults to a computed value based on the recorded content. - """ - svg = self.export_svg( - title=title, - theme=theme, - clear=clear, - code_format=code_format, - font_aspect_ratio=font_aspect_ratio, - unique_id=unique_id, - ) - with open(path, "wt", encoding="utf-8") as write_file: - write_file.write(svg) - - -def _svg_hash(svg_main_code: str) -> str: - """Returns a unique hash for the given SVG main code. - - Args: - svg_main_code (str): The content we're going to inject in the SVG envelope. - - Returns: - str: a hash of the given content - """ - return str(zlib.adler32(svg_main_code.encode())) - - -if __name__ == "__main__": # pragma: no cover - console = Console(record=True) - - console.log( - "JSONRPC [i]request[/i]", - 5, - 1.3, - True, - False, - None, - { - "jsonrpc": "2.0", - "method": "subtract", - "params": {"minuend": 42, "subtrahend": 23}, - "id": 3, - }, - ) - - console.log("Hello, World!", "{'a': 1}", repr(console)) - - console.print( - { - "name": None, - "empty": [], - "quiz": { - "sport": { - "answered": True, - "q1": { - "question": "Which one is correct team name in NBA?", - "options": [ - "New York Bulls", - "Los Angeles Kings", - "Golden State Warriors", - "Huston Rocket", - ], - "answer": "Huston Rocket", - }, - }, - "maths": { - "answered": False, - "q1": { - "question": "5 + 7 = ?", - "options": [10, 11, 12, 13], - "answer": 12, - }, - "q2": { - "question": "12 - 8 = ?", - "options": [1, 2, 3, 4], - "answer": 4, - }, - }, - }, - } - ) diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/rich/constrain.py b/venv/lib/python3.11/site-packages/pip/_vendor/rich/constrain.py deleted file mode 100644 index 65fdf56..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/rich/constrain.py +++ /dev/null @@ -1,37 +0,0 @@ -from typing import Optional, TYPE_CHECKING - -from .jupyter import JupyterMixin -from .measure import Measurement - -if TYPE_CHECKING: - from .console import Console, ConsoleOptions, RenderableType, RenderResult - - -class Constrain(JupyterMixin): - """Constrain the width of a renderable to a given number of characters. - - Args: - renderable (RenderableType): A renderable object. - width (int, optional): The maximum width (in characters) to render. Defaults to 80. - """ - - def __init__(self, renderable: "RenderableType", width: Optional[int] = 80) -> None: - self.renderable = renderable - self.width = width - - def __rich_console__( - self, console: "Console", options: "ConsoleOptions" - ) -> "RenderResult": - if self.width is None: - yield self.renderable - else: - child_options = options.update_width(min(self.width, options.max_width)) - yield from console.render(self.renderable, child_options) - - def __rich_measure__( - self, console: "Console", options: "ConsoleOptions" - ) -> "Measurement": - if self.width is not None: - options = options.update_width(self.width) - measurement = Measurement.get(console, options, self.renderable) - return measurement diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/rich/containers.py b/venv/lib/python3.11/site-packages/pip/_vendor/rich/containers.py deleted file mode 100644 index e29cf36..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/rich/containers.py +++ /dev/null @@ -1,167 +0,0 @@ -from itertools import zip_longest -from typing import ( - Iterator, - Iterable, - List, - Optional, - Union, - overload, - TypeVar, - TYPE_CHECKING, -) - -if TYPE_CHECKING: - from .console import ( - Console, - ConsoleOptions, - JustifyMethod, - OverflowMethod, - RenderResult, - RenderableType, - ) - from .text import Text - -from .cells import cell_len -from .measure import Measurement - -T = TypeVar("T") - - -class Renderables: - """A list subclass which renders its contents to the console.""" - - def __init__( - self, renderables: Optional[Iterable["RenderableType"]] = None - ) -> None: - self._renderables: List["RenderableType"] = ( - list(renderables) if renderables is not None else [] - ) - - def __rich_console__( - self, console: "Console", options: "ConsoleOptions" - ) -> "RenderResult": - """Console render method to insert line-breaks.""" - yield from self._renderables - - def __rich_measure__( - self, console: "Console", options: "ConsoleOptions" - ) -> "Measurement": - dimensions = [ - Measurement.get(console, options, renderable) - for renderable in self._renderables - ] - if not dimensions: - return Measurement(1, 1) - _min = max(dimension.minimum for dimension in dimensions) - _max = max(dimension.maximum for dimension in dimensions) - return Measurement(_min, _max) - - def append(self, renderable: "RenderableType") -> None: - self._renderables.append(renderable) - - def __iter__(self) -> Iterable["RenderableType"]: - return iter(self._renderables) - - -class Lines: - """A list subclass which can render to the console.""" - - def __init__(self, lines: Iterable["Text"] = ()) -> None: - self._lines: List["Text"] = list(lines) - - def __repr__(self) -> str: - return f"Lines({self._lines!r})" - - def __iter__(self) -> Iterator["Text"]: - return iter(self._lines) - - @overload - def __getitem__(self, index: int) -> "Text": - ... - - @overload - def __getitem__(self, index: slice) -> List["Text"]: - ... - - def __getitem__(self, index: Union[slice, int]) -> Union["Text", List["Text"]]: - return self._lines[index] - - def __setitem__(self, index: int, value: "Text") -> "Lines": - self._lines[index] = value - return self - - def __len__(self) -> int: - return self._lines.__len__() - - def __rich_console__( - self, console: "Console", options: "ConsoleOptions" - ) -> "RenderResult": - """Console render method to insert line-breaks.""" - yield from self._lines - - def append(self, line: "Text") -> None: - self._lines.append(line) - - def extend(self, lines: Iterable["Text"]) -> None: - self._lines.extend(lines) - - def pop(self, index: int = -1) -> "Text": - return self._lines.pop(index) - - def justify( - self, - console: "Console", - width: int, - justify: "JustifyMethod" = "left", - overflow: "OverflowMethod" = "fold", - ) -> None: - """Justify and overflow text to a given width. - - Args: - console (Console): Console instance. - width (int): Number of characters per line. - justify (str, optional): Default justify method for text: "left", "center", "full" or "right". Defaults to "left". - overflow (str, optional): Default overflow for text: "crop", "fold", or "ellipsis". Defaults to "fold". - - """ - from .text import Text - - if justify == "left": - for line in self._lines: - line.truncate(width, overflow=overflow, pad=True) - elif justify == "center": - for line in self._lines: - line.rstrip() - line.truncate(width, overflow=overflow) - line.pad_left((width - cell_len(line.plain)) // 2) - line.pad_right(width - cell_len(line.plain)) - elif justify == "right": - for line in self._lines: - line.rstrip() - line.truncate(width, overflow=overflow) - line.pad_left(width - cell_len(line.plain)) - elif justify == "full": - for line_index, line in enumerate(self._lines): - if line_index == len(self._lines) - 1: - break - words = line.split(" ") - words_size = sum(cell_len(word.plain) for word in words) - num_spaces = len(words) - 1 - spaces = [1 for _ in range(num_spaces)] - index = 0 - if spaces: - while words_size + num_spaces < width: - spaces[len(spaces) - index - 1] += 1 - num_spaces += 1 - index = (index + 1) % len(spaces) - tokens: List[Text] = [] - for index, (word, next_word) in enumerate( - zip_longest(words, words[1:]) - ): - tokens.append(word) - if index < len(spaces): - style = word.get_style_at_offset(console, -1) - next_style = next_word.get_style_at_offset(console, 0) - space_style = style if style == next_style else line.style - tokens.append(Text(" " * spaces[index], style=space_style)) - self[line_index] = Text("").join(tokens) diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/rich/control.py b/venv/lib/python3.11/site-packages/pip/_vendor/rich/control.py deleted file mode 100644 index 88fcb92..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/rich/control.py +++ /dev/null @@ -1,225 +0,0 @@ -import sys -import time -from typing import TYPE_CHECKING, Callable, Dict, Iterable, List, Union - -if sys.version_info >= (3, 8): - from typing import Final -else: - from pip._vendor.typing_extensions import Final # pragma: no cover - -from .segment import ControlCode, ControlType, Segment - -if TYPE_CHECKING: - from .console import Console, ConsoleOptions, RenderResult - -STRIP_CONTROL_CODES: Final = [ - 7, # Bell - 8, # Backspace - 11, # Vertical tab - 12, # Form feed - 13, # Carriage return -] -_CONTROL_STRIP_TRANSLATE: Final = { - _codepoint: None for _codepoint in STRIP_CONTROL_CODES -} - -CONTROL_ESCAPE: Final = { - 7: "\\a", - 8: "\\b", - 11: "\\v", - 12: "\\f", - 13: "\\r", -} - -CONTROL_CODES_FORMAT: Dict[int, Callable[..., str]] = { - ControlType.BELL: lambda: "\x07", - ControlType.CARRIAGE_RETURN: lambda: "\r", - ControlType.HOME: lambda: "\x1b[H", - ControlType.CLEAR: lambda: "\x1b[2J", - ControlType.ENABLE_ALT_SCREEN: lambda: "\x1b[?1049h", - ControlType.DISABLE_ALT_SCREEN: lambda: "\x1b[?1049l", - ControlType.SHOW_CURSOR: lambda: "\x1b[?25h", - ControlType.HIDE_CURSOR: lambda: "\x1b[?25l", - ControlType.CURSOR_UP: lambda param: f"\x1b[{param}A", - ControlType.CURSOR_DOWN: lambda param: f"\x1b[{param}B", - ControlType.CURSOR_FORWARD: lambda param: f"\x1b[{param}C", - ControlType.CURSOR_BACKWARD: lambda param: f"\x1b[{param}D", - ControlType.CURSOR_MOVE_TO_COLUMN: lambda param: f"\x1b[{param+1}G", - ControlType.ERASE_IN_LINE: lambda param: f"\x1b[{param}K", - ControlType.CURSOR_MOVE_TO: lambda x, y: f"\x1b[{y+1};{x+1}H", - ControlType.SET_WINDOW_TITLE: lambda title: f"\x1b]0;{title}\x07", -} - - -class Control: - """A renderable that inserts a control code (non printable but may move cursor). - - Args: - *codes (str): Positional arguments are either a :class:`~rich.segment.ControlType` enum or a - tuple of ControlType and an integer parameter - """ - - __slots__ = ["segment"] - - def __init__(self, *codes: Union[ControlType, ControlCode]) -> None: - control_codes: List[ControlCode] = [ - (code,) if isinstance(code, ControlType) else code for code in codes - ] - _format_map = CONTROL_CODES_FORMAT - rendered_codes = "".join( - _format_map[code](*parameters) for code, *parameters in control_codes - ) - self.segment = Segment(rendered_codes, None, control_codes) - - @classmethod - def bell(cls) -> "Control": - """Ring the 'bell'.""" - return cls(ControlType.BELL) - - @classmethod - def home(cls) -> "Control": - """Move cursor to 'home' position.""" - return cls(ControlType.HOME) - - @classmethod - def move(cls, x: int = 0, y: int = 0) -> "Control": - """Move cursor relative to current position. - - Args: - x (int): X offset. - y (int): Y offset. - - Returns: - ~Control: Control object. - - """ - - def get_codes() -> Iterable[ControlCode]: - control = ControlType - if x: - yield ( - control.CURSOR_FORWARD if x > 0 else control.CURSOR_BACKWARD, - abs(x), - ) - if y: - yield ( - control.CURSOR_DOWN if y > 0 else control.CURSOR_UP, - abs(y), - ) - - control = cls(*get_codes()) - return control - - @classmethod - def move_to_column(cls, x: int, y: int = 0) -> "Control": - """Move to the given column, optionally add offset to row. - - Returns: - x (int): absolute x (column) - y (int): optional y offset (row) - - Returns: - ~Control: Control object. - """ - - return ( - cls( - (ControlType.CURSOR_MOVE_TO_COLUMN, x), - ( - ControlType.CURSOR_DOWN if y > 0 else ControlType.CURSOR_UP, - abs(y), - ), - ) - if y - else cls((ControlType.CURSOR_MOVE_TO_COLUMN, x)) - ) - - @classmethod - def move_to(cls, x: int, y: int) -> "Control": - """Move cursor to absolute position. - - Args: - x (int): x offset (column) - y (int): y offset (row) - - Returns: - ~Control: Control object. - """ - return cls((ControlType.CURSOR_MOVE_TO, x, y)) - - @classmethod - def clear(cls) -> "Control": - """Clear the screen.""" - return cls(ControlType.CLEAR) - - @classmethod - def show_cursor(cls, show: bool) -> "Control": - """Show or hide the cursor.""" - return cls(ControlType.SHOW_CURSOR if show else ControlType.HIDE_CURSOR) - - @classmethod - def alt_screen(cls, enable: bool) -> "Control": - """Enable or disable alt screen.""" - if enable: - return cls(ControlType.ENABLE_ALT_SCREEN, ControlType.HOME) - else: - return cls(ControlType.DISABLE_ALT_SCREEN) - - @classmethod - def title(cls, title: str) -> "Control": - """Set the terminal window title - - Args: - title (str): The new terminal window title - """ - return cls((ControlType.SET_WINDOW_TITLE, title)) - - def __str__(self) -> str: - return self.segment.text - - def __rich_console__( - self, console: "Console", options: "ConsoleOptions" - ) -> "RenderResult": - if self.segment.text: - yield self.segment - - -def strip_control_codes( - text: str, _translate_table: Dict[int, None] = _CONTROL_STRIP_TRANSLATE -) -> str: - """Remove control codes from text. - - Args: - text (str): A string possibly contain control codes. - - Returns: - str: String with control codes removed. - """ - return text.translate(_translate_table) - - -def escape_control_codes( - text: str, - _translate_table: Dict[int, str] = CONTROL_ESCAPE, -) -> str: - """Replace control codes with their "escaped" equivalent in the given text. - (e.g. "\b" becomes "\\b") - - Args: - text (str): A string possibly containing control codes. - - Returns: - str: String with control codes replaced with their escaped version. - """ - return text.translate(_translate_table) - - -if __name__ == "__main__": # pragma: no cover - from pip._vendor.rich.console import Console - - console = Console() - console.print("Look at the title of your terminal window ^") - # console.print(Control((ControlType.SET_WINDOW_TITLE, "Hello, world!"))) - for i in range(10): - console.set_window_title("🚀 Loading" + "." * i) - time.sleep(0.5) diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/rich/default_styles.py b/venv/lib/python3.11/site-packages/pip/_vendor/rich/default_styles.py deleted file mode 100644 index dca3719..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/rich/default_styles.py +++ /dev/null @@ -1,190 +0,0 @@ -from typing import Dict - -from .style import Style - -DEFAULT_STYLES: Dict[str, Style] = { - "none": Style.null(), - "reset": Style( - color="default", - bgcolor="default", - dim=False, - bold=False, - italic=False, - underline=False, - blink=False, - blink2=False, - reverse=False, - conceal=False, - strike=False, - ), - "dim": Style(dim=True), - "bright": Style(dim=False), - "bold": Style(bold=True), - "strong": Style(bold=True), - "code": Style(reverse=True, bold=True), - "italic": Style(italic=True), - "emphasize": Style(italic=True), - "underline": Style(underline=True), - "blink": Style(blink=True), - "blink2": Style(blink2=True), - "reverse": Style(reverse=True), - "strike": Style(strike=True), - "black": Style(color="black"), - "red": Style(color="red"), - "green": Style(color="green"), - "yellow": Style(color="yellow"), - "magenta": Style(color="magenta"), - "cyan": Style(color="cyan"), - "white": Style(color="white"), - "inspect.attr": Style(color="yellow", italic=True), - "inspect.attr.dunder": Style(color="yellow", italic=True, dim=True), - "inspect.callable": Style(bold=True, color="red"), - "inspect.async_def": Style(italic=True, color="bright_cyan"), - "inspect.def": Style(italic=True, color="bright_cyan"), - "inspect.class": Style(italic=True, color="bright_cyan"), - "inspect.error": Style(bold=True, color="red"), - "inspect.equals": Style(), - "inspect.help": Style(color="cyan"), - "inspect.doc": Style(dim=True), - "inspect.value.border": Style(color="green"), - "live.ellipsis": Style(bold=True, color="red"), - "layout.tree.row": Style(dim=False, color="red"), - "layout.tree.column": Style(dim=False, color="blue"), - "logging.keyword": Style(bold=True, color="yellow"), - "logging.level.notset": Style(dim=True), - "logging.level.debug": Style(color="green"), - "logging.level.info": Style(color="blue"), - "logging.level.warning": Style(color="red"), - "logging.level.error": Style(color="red", bold=True), - "logging.level.critical": Style(color="red", bold=True, reverse=True), - "log.level": Style.null(), - "log.time": Style(color="cyan", dim=True), - "log.message": Style.null(), - "log.path": Style(dim=True), - "repr.ellipsis": Style(color="yellow"), - "repr.indent": Style(color="green", dim=True), - "repr.error": Style(color="red", bold=True), - "repr.str": Style(color="green", italic=False, bold=False), - "repr.brace": Style(bold=True), - "repr.comma": Style(bold=True), - "repr.ipv4": Style(bold=True, color="bright_green"), - "repr.ipv6": Style(bold=True, color="bright_green"), - "repr.eui48": Style(bold=True, color="bright_green"), - "repr.eui64": Style(bold=True, color="bright_green"), - "repr.tag_start": Style(bold=True), - "repr.tag_name": Style(color="bright_magenta", bold=True), - "repr.tag_contents": Style(color="default"), - "repr.tag_end": Style(bold=True), - "repr.attrib_name": Style(color="yellow", italic=False), - "repr.attrib_equal": Style(bold=True), - "repr.attrib_value": Style(color="magenta", italic=False), - "repr.number": Style(color="cyan", bold=True, italic=False), - "repr.number_complex": Style(color="cyan", bold=True, italic=False), # same - "repr.bool_true": Style(color="bright_green", italic=True), - "repr.bool_false": Style(color="bright_red", italic=True), - "repr.none": Style(color="magenta", italic=True), - "repr.url": Style(underline=True, color="bright_blue", italic=False, bold=False), - "repr.uuid": Style(color="bright_yellow", bold=False), - "repr.call": Style(color="magenta", bold=True), - "repr.path": Style(color="magenta"), - "repr.filename": Style(color="bright_magenta"), - "rule.line": Style(color="bright_green"), - "rule.text": Style.null(), - "json.brace": Style(bold=True), - "json.bool_true": Style(color="bright_green", italic=True), - "json.bool_false": Style(color="bright_red", italic=True), - "json.null": Style(color="magenta", italic=True), - "json.number": Style(color="cyan", bold=True, italic=False), - "json.str": Style(color="green", italic=False, bold=False), - "json.key": Style(color="blue", bold=True), - "prompt": Style.null(), - "prompt.choices": Style(color="magenta", bold=True), - "prompt.default": Style(color="cyan", bold=True), - "prompt.invalid": Style(color="red"), - "prompt.invalid.choice": Style(color="red"), - "pretty": Style.null(), - "scope.border": Style(color="blue"), - "scope.key": Style(color="yellow", italic=True), - "scope.key.special": Style(color="yellow", italic=True, dim=True), - "scope.equals": Style(color="red"), - "table.header": Style(bold=True), - "table.footer": Style(bold=True), - "table.cell": Style.null(), - "table.title": Style(italic=True), - "table.caption": Style(italic=True, dim=True), - "traceback.error": Style(color="red", italic=True), - "traceback.border.syntax_error": Style(color="bright_red"), - "traceback.border": Style(color="red"), - "traceback.text": Style.null(), - "traceback.title": Style(color="red", bold=True), - "traceback.exc_type": Style(color="bright_red", bold=True), - "traceback.exc_value": Style.null(), - "traceback.offset": Style(color="bright_red", bold=True), - "bar.back": Style(color="grey23"), - "bar.complete": Style(color="rgb(249,38,114)"), - "bar.finished": Style(color="rgb(114,156,31)"), - "bar.pulse": Style(color="rgb(249,38,114)"), - "progress.description": Style.null(), - "progress.filesize": Style(color="green"), - "progress.filesize.total": Style(color="green"), - "progress.download": Style(color="green"), - "progress.elapsed": Style(color="yellow"), - "progress.percentage": Style(color="magenta"), - "progress.remaining": Style(color="cyan"), - "progress.data.speed": Style(color="red"), - "progress.spinner": Style(color="green"), - "status.spinner": Style(color="green"), - "tree": Style(), - "tree.line": Style(), - "markdown.paragraph": Style(), - "markdown.text": Style(), - "markdown.em": Style(italic=True), - "markdown.emph": Style(italic=True), # For commonmark backwards compatibility - "markdown.strong": Style(bold=True), - "markdown.code": Style(bold=True, color="cyan", bgcolor="black"), - "markdown.code_block": Style(color="cyan", bgcolor="black"), - "markdown.block_quote": Style(color="magenta"), - "markdown.list": Style(color="cyan"), - "markdown.item": Style(), - "markdown.item.bullet": Style(color="yellow", bold=True), - "markdown.item.number": Style(color="yellow", bold=True), - "markdown.hr": Style(color="yellow"), - "markdown.h1.border": Style(), - "markdown.h1": Style(bold=True), - "markdown.h2": Style(bold=True, underline=True), - "markdown.h3": Style(bold=True), - "markdown.h4": Style(bold=True, dim=True), - "markdown.h5": Style(underline=True), - "markdown.h6": Style(italic=True), - "markdown.h7": Style(italic=True, dim=True), - "markdown.link": Style(color="bright_blue"), - "markdown.link_url": Style(color="blue", underline=True), - "markdown.s": Style(strike=True), - "iso8601.date": Style(color="blue"), - "iso8601.time": Style(color="magenta"), - "iso8601.timezone": Style(color="yellow"), -} - - -if __name__ == "__main__": # pragma: no cover - import argparse - import io - - from pip._vendor.rich.console import Console - from pip._vendor.rich.table import Table - from pip._vendor.rich.text import Text - - parser = argparse.ArgumentParser() - parser.add_argument("--html", action="store_true", help="Export as HTML table") - args = parser.parse_args() - html: bool = args.html - console = Console(record=True, width=70, file=io.StringIO()) if html else Console() - - table = Table("Name", "Styling") - - for style_name, style in DEFAULT_STYLES.items(): - table.add_row(Text(style_name, style=style), str(style)) - - console.print(table) - if html: - print(console.export_html(inline_styles=True)) diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/rich/diagnose.py b/venv/lib/python3.11/site-packages/pip/_vendor/rich/diagnose.py deleted file mode 100644 index ad36183..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/rich/diagnose.py +++ /dev/null @@ -1,37 +0,0 @@ -import os -import platform - -from pip._vendor.rich import inspect -from pip._vendor.rich.console import Console, get_windows_console_features -from pip._vendor.rich.panel import Panel -from pip._vendor.rich.pretty import Pretty - - -def report() -> None: # pragma: no cover - """Print a report to the terminal with debugging information""" - console = Console() - inspect(console) - features = get_windows_console_features() - inspect(features) - - env_names = ( - "TERM", - "COLORTERM", - "CLICOLOR", - "NO_COLOR", - "TERM_PROGRAM", - "COLUMNS", - "LINES", - "JUPYTER_COLUMNS", - "JUPYTER_LINES", - "JPY_PARENT_PID", - "VSCODE_VERBOSE_LOGGING", - ) - env = {name: os.getenv(name) for name in env_names} - console.print(Panel.fit((Pretty(env)), title="[b]Environment Variables")) - - console.print(f'platform="{platform.system()}"') - - -if __name__ == "__main__": # pragma: no cover - report() diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/rich/emoji.py b/venv/lib/python3.11/site-packages/pip/_vendor/rich/emoji.py deleted file mode 100644 index 791f046..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/rich/emoji.py +++ /dev/null @@ -1,96 +0,0 @@ -import sys -from typing import TYPE_CHECKING, Optional, Union - -from .jupyter import JupyterMixin -from .segment import Segment -from .style import Style -from ._emoji_codes import EMOJI -from ._emoji_replace import _emoji_replace - -if sys.version_info >= (3, 8): - from typing import Literal -else: - from pip._vendor.typing_extensions import Literal # pragma: no cover - - -if TYPE_CHECKING: - from .console import Console, ConsoleOptions, RenderResult - - -EmojiVariant = Literal["emoji", "text"] - - -class NoEmoji(Exception): - """No emoji by that name.""" - - -class Emoji(JupyterMixin): - __slots__ = ["name", "style", "_char", "variant"] - - VARIANTS = {"text": "\uFE0E", "emoji": "\uFE0F"} - - def __init__( - self, - name: str, - style: Union[str, Style] = "none", - variant: Optional[EmojiVariant] = None, - ) -> None: - """A single emoji character. - - Args: - name (str): Name of emoji. - style (Union[str, Style], optional): Optional style. Defaults to None. - - Raises: - NoEmoji: If the emoji doesn't exist. - """ - self.name = name - self.style = style - self.variant = variant - try: - self._char = EMOJI[name] - except KeyError: - raise NoEmoji(f"No emoji called {name!r}") - if variant is not None: - self._char += self.VARIANTS.get(variant, "") - - @classmethod - def replace(cls, text: str) -> str: - """Replace emoji markup with corresponding unicode characters. - - Args: - text (str): A string with emojis codes, e.g. "Hello :smiley:!" - - Returns: - str: A string with emoji codes replaces with actual emoji. - """ - return _emoji_replace(text) - - def __repr__(self) -> str: - return f"" - - def __str__(self) -> str: - return self._char - - def __rich_console__( - self, console: "Console", options: "ConsoleOptions" - ) -> "RenderResult": - yield Segment(self._char, console.get_style(self.style)) - - -if __name__ == "__main__": # pragma: no cover - import sys - - from pip._vendor.rich.columns import Columns - from pip._vendor.rich.console import Console - - console = Console(record=True) - - columns = Columns( - (f":{name}: {name}" for name in sorted(EMOJI.keys()) if "\u200D" not in name), - column_first=True, - ) - - console.print(columns) - if len(sys.argv) > 1: - console.save_html(sys.argv[1]) diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/rich/errors.py b/venv/lib/python3.11/site-packages/pip/_vendor/rich/errors.py deleted file mode 100644 index 0bcbe53..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/rich/errors.py +++ /dev/null @@ -1,34 +0,0 @@ -class ConsoleError(Exception): - """An error in console operation.""" - - -class StyleError(Exception): - """An error in styles.""" - - -class StyleSyntaxError(ConsoleError): - """Style was badly formatted.""" - - -class MissingStyle(StyleError): - """No such style.""" - - -class StyleStackError(ConsoleError): - """Style stack is invalid.""" - - -class NotRenderableError(ConsoleError): - """Object is not renderable.""" - - -class MarkupError(ConsoleError): - """Markup was badly formatted.""" - - -class LiveError(ConsoleError): - """Error related to Live display.""" - - -class NoAltScreen(ConsoleError): - """Alt screen mode was required.""" diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/rich/file_proxy.py b/venv/lib/python3.11/site-packages/pip/_vendor/rich/file_proxy.py deleted file mode 100644 index 4b0b0da..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/rich/file_proxy.py +++ /dev/null @@ -1,57 +0,0 @@ -import io -from typing import IO, TYPE_CHECKING, Any, List - -from .ansi import AnsiDecoder -from .text import Text - -if TYPE_CHECKING: - from .console import Console - - -class FileProxy(io.TextIOBase): - """Wraps a file (e.g. sys.stdout) and redirects writes to a console.""" - - def __init__(self, console: "Console", file: IO[str]) -> None: - self.__console = console - self.__file = file - self.__buffer: List[str] = [] - self.__ansi_decoder = AnsiDecoder() - - @property - def rich_proxied_file(self) -> IO[str]: - """Get proxied file.""" - return self.__file - - def __getattr__(self, name: str) -> Any: - return getattr(self.__file, name) - - def write(self, text: str) -> int: - if not isinstance(text, str): - raise TypeError(f"write() argument must be str, not {type(text).__name__}") - buffer = self.__buffer - lines: List[str] = [] - while text: - line, new_line, text = text.partition("\n") - if new_line: - lines.append("".join(buffer) + line) - buffer.clear() - else: - buffer.append(line) - break - if lines: - console = self.__console - with console: - output = Text("\n").join( - self.__ansi_decoder.decode_line(line) for line in lines - ) - console.print(output) - return len(text) - - def flush(self) -> None: - output = "".join(self.__buffer) - if output: - self.__console.print(output) - del self.__buffer[:] - - def fileno(self) -> int: - return self.__file.fileno() diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/rich/filesize.py b/venv/lib/python3.11/site-packages/pip/_vendor/rich/filesize.py deleted file mode 100644 index 99f118e..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/rich/filesize.py +++ /dev/null @@ -1,89 +0,0 @@ -# coding: utf-8 -"""Functions for reporting filesizes. Borrowed from https://github.com/PyFilesystem/pyfilesystem2 - -The functions declared in this module should cover the different -use cases needed to generate a string representation of a file size -using several different units. Since there are many standards regarding -file size units, three different functions have been implemented. - -See Also: - * `Wikipedia: Binary prefix `_ - -""" - -__all__ = ["decimal"] - -from typing import Iterable, List, Optional, Tuple - - -def _to_str( - size: int, - suffixes: Iterable[str], - base: int, - *, - precision: Optional[int] = 1, - separator: Optional[str] = " ", -) -> str: - if size == 1: - return "1 byte" - elif size < base: - return "{:,} bytes".format(size) - - for i, suffix in enumerate(suffixes, 2): # noqa: B007 - unit = base**i - if size < unit: - break - return "{:,.{precision}f}{separator}{}".format( - (base * size / unit), - suffix, - precision=precision, - separator=separator, - ) - - -def pick_unit_and_suffix(size: int, suffixes: List[str], base: int) -> Tuple[int, str]: - """Pick a suffix and base for the given size.""" - for i, suffix in enumerate(suffixes): - unit = base**i - if size < unit * base: - break - return unit, suffix - - -def decimal( - size: int, - *, - precision: Optional[int] = 1, - separator: Optional[str] = " ", -) -> str: - """Convert a filesize in to a string (powers of 1000, SI prefixes). - - In this convention, ``1000 B = 1 kB``. - - This is typically the format used to advertise the storage - capacity of USB flash drives and the like (*256 MB* meaning - actually a storage capacity of more than *256 000 000 B*), - or used by **Mac OS X** since v10.6 to report file sizes. - - Arguments: - int (size): A file size. - int (precision): The number of decimal places to include (default = 1). - str (separator): The string to separate the value from the units (default = " "). - - Returns: - `str`: A string containing a abbreviated file size and units. - - Example: - >>> filesize.decimal(30000) - '30.0 kB' - >>> filesize.decimal(30000, precision=2, separator="") - '30.00kB' - - """ - return _to_str( - size, - ("kB", "MB", "GB", "TB", "PB", "EB", "ZB", "YB"), - 1000, - precision=precision, - separator=separator, - ) diff --git a/venv/lib/python3.11/site-packages/pip/_vendor/rich/highlighter.py b/venv/lib/python3.11/site-packages/pip/_vendor/rich/highlighter.py deleted file mode 100644 index c264679..0000000 --- a/venv/lib/python3.11/site-packages/pip/_vendor/rich/highlighter.py +++ /dev/null @@ -1,232 +0,0 @@ -import re -from abc import ABC, abstractmethod -from typing import List, Union - -from .text import Span, Text - - -def _combine_regex(*regexes: str) -> str: - """Combine a number of regexes in to a single regex. - - Returns: - str: New regex with all regexes ORed together. - """ - return "|".join(regexes) - - -class Highlighter(ABC): - """Abstract base class for highlighters.""" - - def __call__(self, text: Union[str, Text]) -> Text: - """Highlight a str or Text instance. - - Args: - text (Union[str, ~Text]): Text to highlight. - - Raises: - TypeError: If not called with text or str. - - Returns: - Text: A test instance with highlighting applied. - """ - if isinstance(text, str): - highlight_text = Text(text) - elif isinstance(text, Text): - highlight_text = text.copy() - else: - raise TypeError(f"str or Text instance required, not {text!r}") - self.highlight(highlight_text) - return highlight_text - - @abstractmethod - def highlight(self, text: Text) -> None: - """Apply highlighting in place to text. - - Args: - text (~Text): A text object highlight. - """ - - -class NullHighlighter(Highlighter): - """A highlighter object that doesn't highlight. - - May be used to disable highlighting entirely. - - """ - - def highlight(self, text: Text) -> None: - """Nothing to do""" - - -class RegexHighlighter(Highlighter): - """Applies highlighting from a list of regular expressions.""" - - highlights: List[str] = [] - base_style: str = "" - - def highlight(self, text: Text) -> None: - """Highlight :class:`rich.text.Text` using regular expressions. - - Args: - text (~Text): Text to highlighted. - - """ - - highlight_regex = text.highlight_regex - for re_highlight in self.highlights: - highlight_regex(re_highlight, style_prefix=self.base_style) - - -class ReprHighlighter(RegexHighlighter): - """Highlights the text typically produced from ``__repr__`` methods.""" - - base_style = "repr." - highlights = [ - r"(?P<)(?P[-\w.:|]*)(?P[\w\W]*)(?P>)", - r'(?P[\w_]{1,50})=(?P"?[\w_]+"?)?', - r"(?P[][{}()])", - _combine_regex( - r"(?P[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3})", - r"(?P([A-Fa-f0-9]{1,4}::?){1,7}[A-Fa-f0-9]{1,4})", - r"(?P(?:[0-9A-Fa-f]{1,2}-){7}[0-9A-Fa-f]{1,2}|(?:[0-9A-Fa-f]{1,2}:){7}[0-9A-Fa-f]{1,2}|(?:[0-9A-Fa-f]{4}\.){3}[0-9A-Fa-f]{4})", - r"(?P(?:[0-9A-Fa-f]{1,2}-){5}[0-9A-Fa-f]{1,2}|(?:[0-9A-Fa-f]{1,2}:){5}[0-9A-Fa-f]{1,2}|(?:[0-9A-Fa-f]{4}\.){2}[0-9A-Fa-f]{4})", - r"(?P[a-fA-F0-9]{8}-[a-fA-F0-9]{4}-[a-fA-F0-9]{4}-[a-fA-F0-9]{4}-[a-fA-F0-9]{12})", - r"(?P[\w.]*?)\(", - r"\b(?PTrue)\b|\b(?PFalse)\b|\b(?PNone)\b", - r"(?P\.\.\.)", - r"(?P(?(?\B(/[-\w._+]+)*\/)(?P[-\w._+]*)?", - r"(?b?'''.*?(?(file|https|http|ws|wss)://[-0-9a-zA-Z$_+!`(),.?/;:&=%#]*)", - ), - ] - - -class JSONHighlighter(RegexHighlighter): - """Highlights JSON""" - - # Captures the start and end of JSON strings, handling escaped quotes - JSON_STR = r"(?b?\".*?(?[\{\[\(\)\]\}])", - r"\b(?Ptrue)\b|\b(?Pfalse)\b|\b(?Pnull)\b", - r"(?P(? None: - super().highlight(text) - - # Additional work to handle highlighting JSON keys - plain = text.plain - append = text.spans.append - whitespace = self.JSON_WHITESPACE - for match in re.finditer(self.JSON_STR, plain): - start, end = match.span() - cursor = end - while cursor < len(plain): - char = plain[cursor] - cursor += 1 - if char == ":": - append(Span(start, end, "json.key")) - elif char in whitespace: - continue - break - - -class ISO8601Highlighter(RegexHighlighter): - """Highlights the ISO8601 date time strings. - Regex reference: https://www.oreilly.com/library/view/regular-expressions-cookbook/9781449327453/ch04s07.html - """ - - base_style = "iso8601." - highlights = [ - # - # Dates - # - # Calendar month (e.g. 2008-08). The hyphen is required - r"^(?P[0-9]{4})-(?P1[0-2]|0[1-9])$", - # Calendar date w/o hyphens (e.g. 20080830) - r"^(?P(?P[0-9]{4})(?P1[0-2]|0[1-9])(?P3[01]|0[1-9]|[12][0-9]))$", - # Ordinal date (e.g. 2008-243). The hyphen is optional - r"^(?P(?P[0-9]{4})-?(?P36[0-6]|3[0-5][0-9]|[12][0-9]{2}|0[1-9][0-9]|00[1-9]))$", - # - # Weeks - # - # Week of the year (e.g., 2008-W35). The hyphen is optional - r"^(?P(?P[0-9]{4})-?W(?P5[0-3]|[1-4][0-9]|0[1-9]))$", - # Week date (e.g., 2008-W35-6). The hyphens are optional - r"^(?P(?P[0-9]{4})-?W(?P5[0-3]|[1-4][0-9]|0[1-9])-?(?P[1-7]))$", - # - # Times - # - # Hours and minutes (e.g., 17:21). The colon is optional - r"^(?P